<?xml version="1.0" encoding="utf-8"?>
<posts>
  <row Id="1" PostTypeId="1" AcceptedAnswerId="15" CreationDate="2010-07-19T19:12:12.510" Score="14" ViewCount="357" Body="&lt;p&gt;How should I elicit prior distributions from experts when fitting a Bayesian model?&lt;/p&gt;&#xA;" OwnerUserId="8" LastActivityDate="2010-09-15T21:08:26.077" Title="Eliciting priors from experts" Tags="&lt;bayesian&gt;&lt;prior&gt;&lt;elicitation&gt;" AnswerCount="5" CommentCount="1" FavoriteCount="7" />
  <row Id="2" PostTypeId="1" AcceptedAnswerId="59" CreationDate="2010-07-19T19:12:57.157" Score="13" ViewCount="298" Body="&lt;p&gt;In many different statistical methods there is an &quot;assumption of normality&quot;.  What is &quot;normality&quot; and how do I know if there is normality?&lt;/p&gt;&#xA;" OwnerUserId="24" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-08-07T17:56:44.800" LastActivityDate="2011-04-10T14:30:50.133" Title="What is normality?" Tags="&lt;distributions&gt;&lt;normality&gt;" AnswerCount="4" CommentCount="1" FavoriteCount="1" />
  <row Id="3" PostTypeId="1" AcceptedAnswerId="5" CreationDate="2010-07-19T19:13:28.577" Score="31" ViewCount="947" Body="&lt;p&gt;What are some valuable Statistical Analysis open source projects available right now?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: as pointed out by Sharpie, valuable could mean helping you get things done faster or more cheaply.&lt;/p&gt;&#xA;" OwnerUserId="18" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-02-12T05:50:03.667" LastActivityDate="2011-06-02T05:25:33.257" CommunityOwnedDate="2010-07-19T19:13:28.577" Title="What are some valuable Statistical Analysis open source projects?" Tags="&lt;software&gt;&lt;open-source&gt;" AnswerCount="18" CommentCount="4" FavoriteCount="19" />
  <row Id="4" PostTypeId="1" AcceptedAnswerId="135" CreationDate="2010-07-19T19:13:31.617" Score="6" ViewCount="313" Body="&lt;p&gt;I have two groups of data.  Each with a different distribution of multiple variables.  I'm trying to determine if these two groups' distributions are different in a statistically significant way.  I have the data in both raw form and binned up in easier to deal with discrete categories with frequency counts in each.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What tests/procedures/methods should I use to determine whether or not these two groups are significantly different and how do I do that in SAS or R (or Orange)?&lt;/p&gt;&#xA;" OwnerUserId="23" LastActivityDate="2010-09-08T03:00:19.690" Title="Assessing the significance of differences in distributions" Tags="&lt;distributions&gt;&lt;statistical-significance&gt;" AnswerCount="5" CommentCount="2" FavoriteCount="1" />
  <row Id="5" PostTypeId="2" ParentId="3" CreationDate="2010-07-19T19:14:43.050" Score="61" ViewCount="" Body="&lt;p&gt;The R-project&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.r-project.org/&quot; rel=&quot;nofollow&quot;&gt;http://www.r-project.org/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;R is valuable and significant because it was the first widely-accepted Open-Source alternative to big-box packages.  It's mature, well supported, and a standard within many scientific communities.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.inside-r.org/why-use-r&quot; rel=&quot;nofollow&quot;&gt;Some reasons why it is useful and valuable&lt;/a&gt; &lt;/li&gt;&#xA;&lt;li&gt;There are some nice tutorials &lt;a href=&quot;http://gettinggeneticsdone.blogspot.com/search/label/ggplot2&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="23" LastEditorUserId="23" LastEditorDisplayName="" LastEditDate="2010-07-19T19:21:15.063" LastActivityDate="2010-07-19T19:21:15.063" CommunityOwnedDate="2010-07-19T19:14:43.050" CommentCount="3" />
  <row Id="6" PostTypeId="1" CreationDate="2010-07-19T19:14:44.080" Score="30" ViewCount="1594" Body="&lt;p&gt;Last year, I read a blog post from &lt;a href=&quot;http://anyall.org/&quot; rel=&quot;nofollow&quot;&gt;Bendan O'Connor&lt;/a&gt; entitled &lt;a href=&quot;http://anyall.org/blog/2008/12/statistics-vs-machine-learning-fight/&quot; rel=&quot;nofollow&quot;&gt;&quot;Statistics vs. Machine Learning, fight!&quot;&lt;/a&gt; that discussed some of the differences between the two fields.  &lt;a href=&quot;http://www.stat.columbia.edu/~cook/movabletype/archives/2008/12/machine-learnin.html&quot; rel=&quot;nofollow&quot;&gt;Andrew Gelman responded to favorably to this&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Simon Blomberg: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;From R's fortunes&#xA;  package: To paraphrase provocatively,&#xA;  'machine learning is statistics minus&#xA;  any checking of models and&#xA;  assumptions'.&#xA;  -- Brian D. Ripley (about the difference between machine learning&#xA;  and statistics) useR! 2004, Vienna&#xA;  (May 2004) :-) Season's Greetings!&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Andrew Gelman:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In that case, maybe we should get rid&#xA;  of checking of models and assumptions&#xA;  more often. Then maybe we'd be able to&#xA;  solve some of the problems that the&#xA;  machine learning people can solve but&#xA;  we can't!&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;There was also the &lt;a href=&quot;http://www.stat.osu.edu/~bli/dmsl/papers/Breiman.pdf&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;&quot;Statistical Modeling: The Two Cultures&quot;&lt;/strong&gt; paper&lt;/a&gt; by Leo Breiman in 2001 which argued that Statisticians rely too heavily on data modeling, and that machine learning techniques are making progress by instead relying on the &lt;em&gt;predictive accuracy&lt;/em&gt; of models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Has the Statistics field changed over the last decade in response to these critiques?  Do the &lt;em&gt;two cultures&lt;/em&gt; still exist or has Statistics grown to embrace machine learning techniques such as neural networks and support vector machines?&lt;/p&gt;&#xA;" OwnerUserId="5" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-10-28T10:39:42.043" LastActivityDate="2011-02-14T22:09:41.657" CommunityOwnedDate="2010-08-09T13:05:50.603" Title="The Two Cultures: statistics vs. machine learning?" Tags="&lt;machine-learning&gt;" AnswerCount="8" CommentCount="3" FavoriteCount="20" />
  <row Id="7" PostTypeId="1" AcceptedAnswerId="18" CreationDate="2010-07-19T19:15:59.303" Score="32" ViewCount="856" Body="&lt;p&gt;I've been working on a new method for analyzing and parsing datasets to identify and isolate subgroups of a population without foreknowledge of any subgroup's characteristics.  While the method works well enough with artificial data samples (i.e. datasets created specifically for the purpose of identifying and segregating subsets of the population), I'd like to try testing it with live data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I'm looking for is a freely available (i.e. non-confidential, non-proprietary) data source.  Preferably one containing bimodal or multimodal distributions or being obviously comprised of multiple subsets that cannot be easily pulled apart via traditional means.  Where would I go to find such information?&lt;/p&gt;&#xA;" OwnerUserId="38" LastEditorUserId="38" LastEditorDisplayName="" LastEditDate="2010-07-20T20:50:48.483" LastActivityDate="2011-03-02T13:19:47.840" CommunityOwnedDate="2010-07-20T20:50:48.483" Title="Locating freely available data samples" Tags="&lt;population&gt;&lt;dataset&gt;&lt;sample&gt;" AnswerCount="19" CommentCount="3" FavoriteCount="27" />
  <row Id="8" PostTypeId="1" CreationDate="2010-07-19T19:16:21.737" Score="0" ViewCount="117" Body="&lt;p&gt;Sorry, but the emptyness was a bit overwhelming. And this has been stuck in my head since it got asked at Area51!&lt;/p&gt;&#xA;" OwnerUserId="37" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-10-18T07:57:31.170" LastActivityDate="2010-10-18T07:57:31.170" ClosedDate="2010-07-19T20:19:46.577" Title="So how many staticians *does* it take to screw in a lightbulb?" Tags="&lt;humor&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="9" PostTypeId="2" ParentId="3" CreationDate="2010-07-19T19:16:27.553" Score="11" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://incanter.org/&quot; rel=&quot;nofollow&quot;&gt;Incanter&lt;/a&gt; is a Clojure-based, R-like platform (environment + libraries) for statistical computing and graphics. &lt;/p&gt;&#xA;" OwnerUserId="50" LastActivityDate="2010-07-19T19:16:27.553" CommunityOwnedDate="2010-07-19T19:16:27.553" CommentCount="3" />
  <row Id="10" PostTypeId="1" AcceptedAnswerId="153" CreationDate="2010-07-19T19:17:47.537" Score="5" ViewCount="1557" Body="&lt;p&gt;Many studies in the social sciences use Likert scales.  When is it appropriate to use Likert data as ordinal and when is it appropriate to use it as interval data?&lt;/p&gt;&#xA;" OwnerUserId="24" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-30T15:31:46.003" LastActivityDate="2011-03-30T18:37:24.253" Title="Under what conditions should Likert scales be used as ordinal or interval data?" Tags="&lt;scales&gt;&lt;measurement&gt;&lt;ordinal&gt;&lt;interval&gt;&lt;likert&gt;" AnswerCount="3" CommentCount="4" FavoriteCount="3" />
  <row Id="11" PostTypeId="1" AcceptedAnswerId="1201" CreationDate="2010-07-19T19:18:30.810" Score="2" ViewCount="104" Body="&lt;p&gt;Is there a good, modern treatment covering the various methods of multivariate interpolation, including which methodologies are typically best for particular types of problems? I'm interested in a solid statistical treatment including error estimates under various model assumptions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Inverse_distance_weighting&quot; rel=&quot;nofollow&quot;&gt;Shepard's method&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say we're sampling from a multivariate normal distribution with unknown parameters. What can we say about the standard error of the interpolated estimates?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was hoping for a pointer to a general survey addressing similar questions for the various types of multivariate interpolations in common use. &lt;/p&gt;&#xA;" OwnerUserId="34" LastEditorUserId="34" LastEditorDisplayName="" LastEditDate="2010-07-28T07:58:52.320" LastActivityDate="2010-08-03T21:50:09.007" Title="Multivariate Interpolation Approaches" Tags="&lt;multivariable&gt;&lt;interpolation&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="1" />
  <row Id="12" PostTypeId="2" ParentId="7" CreationDate="2010-07-19T19:18:41.370" Score="13" ViewCount="" Body="&lt;p&gt;See my response to &lt;a href=&quot;http://stackoverflow.com/questions/2252144/datasets-for-running-statistical-analysis-on/2252450#2252450&quot;&gt;&quot;Datasets for Running Statistical Analysis on&quot;&lt;/a&gt; in reference to datasets in R.&lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-07-19T19:18:41.370" />
  <row Id="13" PostTypeId="2" ParentId="6" CreationDate="2010-07-19T19:18:56.800" Score="9" ViewCount="" Body="&lt;p&gt;Machine Learning seems to have its basis in the pragmatic - a Practical observation or simulation of reality.  Even within statistics, mindless &quot;checking of models and assumptions&quot; can lead to discarding methods that are useful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, years ago, the very first commercially available (and working) Bankruptcy model implemented by the credit bureaus was created through a plain old linear regression model targeting a 0-1 outcome.  Technically, that's a bad approach, but practically, it worked.&lt;/p&gt;&#xA;" OwnerUserId="23" LastActivityDate="2010-07-19T19:18:56.800" CommentCount="3" />
  <row Id="14" PostTypeId="2" ParentId="3" CreationDate="2010-07-19T19:19:03.990" Score="6" ViewCount="" Body="&lt;p&gt;I second that Jay. Why is R valuable? Here's a short list of reasons. &lt;a href=&quot;http://www.inside-r.org/why-use-r&quot; rel=&quot;nofollow&quot;&gt;http://www.inside-r.org/why-use-r&lt;/a&gt;. Also check out &lt;a href=&quot;http://had.co.nz/ggplot2/&quot; rel=&quot;nofollow&quot;&gt;ggplot2&lt;/a&gt; - a very nice graphics package for R. Some nice tutorials &lt;a href=&quot;http://gettinggeneticsdone.blogspot.com/search/label/ggplot2&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="36" LastActivityDate="2010-07-19T19:19:03.990" CommunityOwnedDate="2010-07-19T19:19:03.990" CommentCount="1" />
  <row Id="15" PostTypeId="2" ParentId="1" CreationDate="2010-07-19T19:19:46.160" Score="9" ViewCount="" Body="&lt;p&gt;John Cook gives some interesting recommendations. Basically, get percentiles/quantiles (not means or obscure scale parameters!) from the experts, and fit them with the appropriate distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.johndcook.com/blog/2010/01/31/parameters-from-percentiles/&quot; rel=&quot;nofollow&quot;&gt;http://www.johndcook.com/blog/2010/01/31/parameters-from-percentiles/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="6" LastActivityDate="2010-07-19T19:19:46.160" />
  <row Id="16" PostTypeId="2" ParentId="3" CreationDate="2010-07-19T19:22:31.190" Score="13" ViewCount="" Body="&lt;p&gt;Two projects spring to mind:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.mrc-bsu.cam.ac.uk/bugs/&quot; rel=&quot;nofollow&quot;&gt;Bugs&lt;/a&gt; - taking (some of) the pain out of Bayesian statistics. It allows the user to focus more on the model and a bit less on MCMC.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.bioconductor.org/&quot; rel=&quot;nofollow&quot;&gt;Bioconductor&lt;/a&gt; - perhaps the most popular statistical tool in Bioinformatics. I know it's a R repository, but there are a large number of people who want to learn R, just for Bioconductor. The number of packages available for cutting edge analysis, make it second to none.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="8" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-07-19T20:43:02.683" LastActivityDate="2010-07-19T20:43:02.683" CommunityOwnedDate="2010-07-19T19:22:31.190" CommentCount="3" />
  <row Id="17" PostTypeId="1" AcceptedAnswerId="29" CreationDate="2010-07-19T19:24:12.187" Score="5" ViewCount="457" Body="&lt;p&gt;I have four competing models which I use to predict a binary outcome variable (say, employment status after graduating, 1 = employed, 0 = not-employed) for n subjects. A natural metric of model performance is hit rate which is the percentage of correct predictions for each one of the models. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems to me that I cannot use ANOVA in this setting as the data violates the assumptions underlying ANOVA. Is there an equivalent procedure I could use instead of ANOVA in the above setting to test for the hypothesis that all four models are equally effective?&lt;/p&gt;&#xA;" LastActivityDate="2010-07-19T19:28:15.640" Title="How can I adapt ANOVA for binary data?" Tags="&lt;anova&gt;" AnswerCount="1" />
  <row Id="18" PostTypeId="2" ParentId="7" CreationDate="2010-07-19T19:24:18.580" Score="14" ViewCount="" Body="&lt;p&gt;Also see the UCI machine learning Data Repository.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://archive.ics.uci.edu/ml/&quot; rel=&quot;nofollow&quot;&gt;http://archive.ics.uci.edu/ml/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="36" LastActivityDate="2010-07-19T19:24:18.580" CommentCount="1" />
  <row Id="19" PostTypeId="2" ParentId="7" CreationDate="2010-07-19T19:24:21.327" Score="9" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://www.gapminder.org/data/&quot; rel=&quot;nofollow&quot;&gt;Gapminder&lt;/a&gt; has a number (430 at the last look) of datasets, which may or may not be of use to you.&lt;/p&gt;&#xA;" OwnerUserId="55" LastActivityDate="2010-07-19T19:24:21.327" />
  <row Id="20" PostTypeId="2" ParentId="2" CreationDate="2010-07-19T19:24:35.803" Score="2" ViewCount="" Body="&lt;p&gt;The assumption of normality assumes your data is normally distributed (the bell curve, or gaussian distribution). You can check this by plotting the data or checking the measures for kurtosis (how sharp the peak is) and skewdness (?) (if more than half the data is on one side of the peak).&lt;/p&gt;&#xA;" OwnerUserId="37" LastActivityDate="2010-07-19T19:24:35.803" CommentCount="2" />
  <row Id="21" PostTypeId="1" CreationDate="2010-07-19T19:24:36.303" Score="4" ViewCount="133" Body="&lt;p&gt;What are some of the ways to forecast demographic census with some validation and calibration techniques?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some of the concerns:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Census blocks vary in sizes as rural&#xA;areas are a lot larger than condensed&#xA;urban areas. Is there a need to account for the area size difference?&lt;/li&gt;&#xA;&lt;li&gt;if let's say I have census data&#xA;dating back to 4 - 5 census periods,&#xA;how far can i forecast it into the&#xA;future?&lt;/li&gt;&#xA;&lt;li&gt;if some of the census zone change&#xA;lightly in boundaries, how can i&#xA;account for that change?&lt;/li&gt;&#xA;&lt;li&gt;What are the methods to validate&#xA;census forecasts? for example, if i&#xA;have data for existing 5 census&#xA;periods, should I model the first 3&#xA;and test it on the latter two? or is&#xA;there another way?&lt;/li&gt;&#xA;&lt;li&gt;what's the state of practice in&#xA;forecasting census data, and what are&#xA;some of the state of the art methods?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="59" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-30T21:14:45.273" LastActivityDate="2010-09-30T21:14:45.273" Title="Forecasting demographic census" Tags="&lt;forecasting&gt;&lt;population&gt;&lt;census&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="1" />
  <row Id="22" PostTypeId="1" CreationDate="2010-07-19T19:25:39.467" Score="34" ViewCount="1147" Body="&lt;p&gt;How would you describe in plain English the characteristics that distinguish Bayesian from Frequentist reasoning?&lt;/p&gt;&#xA;" OwnerUserId="66" LastEditorUserId="80" LastEditorDisplayName="" LastEditDate="2010-07-19T19:30:08.303" LastActivityDate="2010-12-04T06:54:30.117" Title="Bayesian and Frequentist reasoning in Plain English" Tags="&lt;bayesian&gt;&lt;frequentist&gt;" AnswerCount="8" CommentCount="0" FavoriteCount="23" />
  <row Id="23" PostTypeId="1" AcceptedAnswerId="91" CreationDate="2010-07-19T19:26:04.363" Score="6" ViewCount="229" Body="&lt;p&gt;How can I find the PDF (probability density function) of a distribution given the CDF (cumulative distribution function)?&lt;/p&gt;&#xA;" OwnerUserId="69" LastActivityDate="2010-07-20T08:52:27.083" Title="Finding the PDF given the CDF" Tags="&lt;distributions&gt;&lt;pdf&gt;&lt;cdf&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="24" PostTypeId="2" ParentId="3" CreationDate="2010-07-19T19:26:13.693" Score="12" ViewCount="" Body="&lt;p&gt;For doing a variety of MCMC tasks in Python, there's &lt;a href=&quot;http://code.google.com/p/pymc/&quot; rel=&quot;nofollow&quot;&gt;PyMC&lt;/a&gt;, which I've gotten quite a bit of use out of.  I haven't run across anything that I can do in BUGS that I can't do in PyMC, and the way you specify models and bring in data seems to be a lot more intuitive to me.&lt;/p&gt;&#xA;" OwnerUserId="61" LastActivityDate="2010-07-19T19:26:13.693" CommunityOwnedDate="2010-07-19T19:26:13.693" />
  <row Id="25" PostTypeId="1" AcceptedAnswerId="32" CreationDate="2010-07-19T19:27:13.503" Score="7" ViewCount="468" Body="&lt;p&gt;What modern tools (Windows-based) do you suggest for modeling financial time series?&lt;/p&gt;&#xA;" OwnerUserId="69" LastEditorUserId="69" LastEditorDisplayName="" LastEditDate="2010-07-26T23:38:29.430" LastActivityDate="2010-11-02T12:15:04.513" CommunityOwnedDate="2010-07-26T23:38:29.430" Title="Tools for modeling financial time series" Tags="&lt;modeling&gt;&lt;time-series&gt;&lt;finance&gt;&lt;software&gt;" AnswerCount="6" CommentCount="3" FavoriteCount="3" />
  <row Id="26" PostTypeId="1" AcceptedAnswerId="61" CreationDate="2010-07-19T19:27:43.860" Score="6" ViewCount="767" Body="&lt;p&gt;What is a standard deviation, how is it calculuated and what is its use in statistics?&lt;/p&gt;&#xA;" OwnerUserId="75" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-17T20:24:13.377" LastActivityDate="2011-01-15T16:08:28.623" Title="What is a standard deviation?" Tags="&lt;standard-deviation&gt;&lt;basic-concepts&gt;" AnswerCount="8" CommentCount="9" FavoriteCount="2" />
  <row Id="27" PostTypeId="2" ParentId="7" CreationDate="2010-07-19T19:28:12.753" Score="4" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://mathforum.org/workshops/sum96/data.collections/datalibrary/data.set6.html&quot; rel=&quot;nofollow&quot;&gt;http://mathforum.org/workshops/sum96/data.collections/datalibrary/data.set6.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="68" LastActivityDate="2010-07-19T19:28:12.753" />
  <row Id="28" PostTypeId="2" ParentId="3" CreationDate="2010-07-19T19:28:12.830" Score="6" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://www.gnu.org/software/gsl/&quot; rel=&quot;nofollow&quot;&gt;GSL&lt;/a&gt; for those of you who wish to program in C / C++ is a valuable resource as it provides several routines for random generators, linear algebra etc. While GSL is primarily available for Linux there are also ports for Windows. (See: &lt;a href=&quot;http://gladman.plushost.co.uk/oldsite/computing/gnu_scientific_library.php&quot; rel=&quot;nofollow&quot;&gt;http://gladman.plushost.co.uk/oldsite/computing/gnu_scientific_library.php&lt;/a&gt; and &lt;a href=&quot;http://david.geldreich.free.fr/dev.html&quot; rel=&quot;nofollow&quot;&gt;http://david.geldreich.free.fr/dev.html&lt;/a&gt;)&lt;/p&gt;&#xA;" LastActivityDate="2010-07-19T19:28:12.830" CommunityOwnedDate="2010-07-19T19:28:12.830" />
  <row Id="29" PostTypeId="2" ParentId="17" CreationDate="2010-07-19T19:28:15.640" Score="4" ViewCount="" Body="&lt;p&gt;Contingency table (chi-square). Also Logistic Regression is your friend - use dummy variables. &lt;/p&gt;&#xA;" OwnerUserId="36" LastActivityDate="2010-07-19T19:28:15.640" />
  <row Id="30" PostTypeId="1" AcceptedAnswerId="55" CreationDate="2010-07-19T19:28:34.220" Score="5" ViewCount="230" Body="&lt;p&gt;Which methods are used for testing random variate generation algorithms?&lt;/p&gt;&#xA;" OwnerUserId="69" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-08-25T14:12:54.547" LastActivityDate="2011-05-12T18:38:27.547" Title="Testing random variate generation algorithms" Tags="&lt;algorithms&gt;&lt;test&gt;&lt;random-variable&gt;&lt;random-generation&gt;" AnswerCount="5" CommentCount="1" FavoriteCount="1" />
  <row Id="31" PostTypeId="1" CreationDate="2010-07-19T19:28:44.903" Score="21" ViewCount="1733" Body="&lt;p&gt;After taking a statistics course and then trying to help fellow students, I noticed one subject that inspires much head-desk banging is interpreting the results of statical hypothesis tests.  It seems that students easily learn how to perform the calculations required by a given test but get hung up on interpreting the results.  Many computerized tools report test results in terms of &quot;p values&quot; or &quot;t values&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How would you explain the following points to college students taking their first course in statistics:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;What does a &quot;p-value&quot; mean in relation to the hypothesis being tested?  Are there cases when one should be looking for a high p-value or a low p-value?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;What is the relationship between a p-value and a t-value?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="13" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-08-07T17:56:25.423" LastActivityDate="2010-09-17T00:09:05.507" Title="What is the meaning of p values and t values in statistical tests?" Tags="&lt;hypothesis-testing&gt;&lt;t-test&gt;&lt;p-value&gt;" AnswerCount="7" FavoriteCount="8" />
  <row Id="32" PostTypeId="2" ParentId="25" CreationDate="2010-07-19T19:29:06.527" Score="9" ViewCount="" Body="&lt;p&gt;I recommend R (see &lt;a href=&quot;http://cran.r-project.org/web/views/TimeSeries.html&quot; rel=&quot;nofollow&quot;&gt;the time series view on CRAN&lt;/a&gt;).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some useful references:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://cran.r-project.org/doc/contrib/Farnsworth-EconometricsInR.pdf&quot; rel=&quot;nofollow&quot;&gt;Econometrics in R&lt;/a&gt;, by Grant Farnsworth&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://stackoverflow.com/questions/1714280/multivariate-time-series-modelling-in-r/1715488#1715488&quot;&gt;Multivariate time series modelling in R&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-07-19T19:29:06.527" />
  <row Id="33" PostTypeId="1" AcceptedAnswerId="49" CreationDate="2010-07-19T19:30:03.717" Score="4" ViewCount="285" Body="&lt;p&gt;What R packages should I install for seasonality analysis?&lt;/p&gt;&#xA;" OwnerUserId="69" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-16T06:56:44.040" LastActivityDate="2010-09-16T06:56:44.040" Title="R packages for seasonality analysis" Tags="&lt;r&gt;&lt;seasonality&gt;" AnswerCount="3" FavoriteCount="3" />
  <row Id="34" PostTypeId="2" ParentId="22" CreationDate="2010-07-19T19:30:07.117" Score="3" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://scienceblogs.com/goodmath/2008/04/schools_of_thought_in_probabil.php&quot; rel=&quot;nofollow&quot;&gt;Schools of thought in Probability Theory&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="79" LastActivityDate="2010-07-19T19:30:07.117" />
  <row Id="35" PostTypeId="1" AcceptedAnswerId="72" CreationDate="2010-07-19T19:30:30.390" Score="2" ViewCount="327" Body="&lt;p&gt;I have a data set that I'd expect to follow a Poisson distribution, but it is overdispersed by about 3-fold. At the present, I'm modelling this overdispersion using something like the following code in R.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;## assuming a median value of 1500&#xA;med = 1500&#xA;rawdist = rpois(1000000,med)&#xA;oDdist = rawDist + ((rawDist-med)*3)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Visually, this seems to fit my empirical data very well. If I'm happy with the fit, is there any reason that I should be doing something more complex, like using a &lt;a href=&quot;http://en.wikipedia.org/wiki/Overdispersion#Poisson&quot; rel=&quot;nofollow&quot;&gt;negative binomial distribution, as described here&lt;/a&gt;? (If so, any pointers or links on doing so would be much appreciated).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Oh, and I'm aware that this creates a slightly jagged distribution (due to the multiplication by three), but that shouldn't matter for my application.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update:&lt;/strong&gt;  For the sake of anyone else who searches and finds this question, here's a simple R function to model an overdispersed poisson using a negative binomial distribution. Set d to the desired mean/variance ratio:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;rpois.od&amp;lt;-function (n, lambda,d=1) {&#xA;  if (d==1)&#xA;    rpois(n, lambda)&#xA;  else&#xA;     rnbinom(n, size=(lambda/(d-1)), mu=lambda)&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(via the R mailing list: &lt;a href=&quot;https://stat.ethz.ch/pipermail/r-help/2002-June/022425.html&quot; rel=&quot;nofollow&quot;&gt;https://stat.ethz.ch/pipermail/r-help/2002-June/022425.html&lt;/a&gt;)&lt;/p&gt;&#xA;" OwnerUserId="54" LastEditorUserId="54" LastEditorDisplayName="" LastEditDate="2010-07-25T01:44:14.513" LastActivityDate="2010-07-25T01:44:14.513" Title="Modelling a Poisson distribution with overdispersion" Tags="&lt;distributions&gt;&lt;modeling&gt;&lt;poisson&gt;&lt;overdispersion&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="36" PostTypeId="1" CreationDate="2010-07-19T19:31:47.363" Score="13" ViewCount="1309" Body="&lt;p&gt;We all know the old saying &quot;Correlation does not mean causation&quot;. When I'm teaching I tend to use these standard examples to illustrate this point:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Number of storks and birth rate in Denmark;&lt;/li&gt;&#xA;&lt;li&gt;Number of priests in America and alcoholism&lt;/li&gt;&#xA;&lt;li&gt;In the start of the 20th century it was noted that there was a strong correlation between `Number of radios' and 'Number of people in Insane Asylums'&lt;/li&gt;&#xA;&lt;li&gt;and my favourite: &lt;a href=&quot;http://en.wikipedia.org/wiki/File%3aPiratesVsTemp_English.jpg&quot; rel=&quot;nofollow&quot;&gt;pirates cause global warming&lt;/a&gt; &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;However, I don't have any references for these examples and whilst amusing, they are obviously false.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone have any other good examples?&lt;/p&gt;&#xA;" OwnerUserId="8" LastActivityDate="2010-09-10T10:48:33.617" CommunityOwnedDate="2010-08-16T13:01:42.613" Title="Correlation does not mean causation" Tags="&lt;correlation&gt;&lt;teaching&gt;" AnswerCount="16" CommentCount="3" FavoriteCount="4" />
  <row Id="38" PostTypeId="2" ParentId="35" CreationDate="2010-07-19T19:32:28.063" Score="2" ViewCount="" Body="&lt;p&gt;If your mean value for the Poisson is 1500, then you're very close to a normal distribution; you might try using that as an approximation and then modelling the mean and variance separately.&lt;/p&gt;&#xA;" OwnerUserId="61" LastActivityDate="2010-07-19T19:32:28.063" CommentCount="2" />
  <row Id="39" PostTypeId="1" AcceptedAnswerId="721" CreationDate="2010-07-19T19:32:29.513" Score="4" ViewCount="121" Body="&lt;p&gt;I'm looking for worked out solutions using Bayesian and/or logit analysis similar to a workbook or an annal. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The worked out problems could be of any field; however, I'm interested in urban planning / transportation related fields. &lt;/p&gt;&#xA;" OwnerUserId="59" LastEditorUserId="190" LastEditorDisplayName="" LastEditDate="2010-07-27T04:57:47.780" LastActivityDate="2010-07-27T05:40:44.950" Title="Sample problems on logit modeling and Bayesian methods" Tags="&lt;modeling&gt;&lt;bayesian&gt;&lt;logit&gt;&lt;transportation&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="40" PostTypeId="1" AcceptedAnswerId="111" CreationDate="2010-07-19T19:32:47.750" Score="7" ViewCount="206" Body="&lt;p&gt;What algorithms are used in modern and good-quality random number generators? &lt;/p&gt;&#xA;" OwnerUserId="69" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-08-25T14:13:48.740" LastActivityDate="2010-08-25T14:13:48.740" Title="Pseudo-random number generation algorithms" Tags="&lt;algorithms&gt;&lt;random-variable&gt;&lt;random-generation&gt;" AnswerCount="3" CommentCount="1" FavoriteCount="2" />
  <row Id="41" PostTypeId="2" ParentId="26" CreationDate="2010-07-19T19:33:13.037" Score="5" ViewCount="" Body="&lt;p&gt;A quote from &lt;a href=&quot;http://en.wikipedia.org/wiki/Standard_deviation&quot; rel=&quot;nofollow&quot;&gt;wiki&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;It shows how much variation there is from the &quot;average&quot; (mean, or expected/budgeted value). A low standard deviation indicates that the data points tend to be very close to the mean, whereas high standard deviation indicates that the data is spread out over a large range of values.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="83" LastActivityDate="2010-07-19T19:33:13.037" />
  <row Id="42" PostTypeId="2" ParentId="3" CreationDate="2010-07-19T19:33:19.073" Score="10" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://www.cs.waikato.ac.nz/ml/weka&quot; rel=&quot;nofollow&quot;&gt;Weka&lt;/a&gt; for data mining - contains many classification and clustering algorithms in Java.&lt;/p&gt;&#xA;" OwnerUserId="80" LastActivityDate="2010-07-19T19:33:19.073" CommunityOwnedDate="2010-07-19T19:33:19.073" CommentCount="3" />
  <row Id="43" PostTypeId="2" ParentId="25" CreationDate="2010-07-19T19:33:37.497" Score="6" ViewCount="" Body="&lt;p&gt;R is great, but I wouldn't really call it &quot;windows based&quot; :) That's like saying the cmd prompt is windows based. I guess it is technically in a window...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;RapidMiner is far easier to use [1]. It's a free, open-source, multi-platform, GUI. Here's a video on time series forecasting:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://rapidminerresources.com/index.php?page=financial-time-series-modelling---part-1&quot; rel=&quot;nofollow&quot;&gt;http://rapidminerresources.com/index.php?page=financial-time-series-modelling---part-1&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, don't forget to read:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.forecastingprinciples.com/&quot; rel=&quot;nofollow&quot;&gt;http://www.forecastingprinciples.com/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[1] No, I don't work for them. &lt;/p&gt;&#xA;" OwnerUserId="74" LastEditorUserId="74" LastEditorDisplayName="" LastEditDate="2010-07-19T20:24:23.787" LastActivityDate="2010-07-19T20:24:23.787" CommentCount="4" />
  <row Id="44" PostTypeId="1" AcceptedAnswerId="89" CreationDate="2010-07-19T19:34:42.643" Score="6" ViewCount="200" Body="&lt;p&gt;How would you explain data visualization and why it is important to a layman?&lt;/p&gt;&#xA;" OwnerUserId="68" LastActivityDate="2010-11-23T20:42:01.350" Title="Explain data visualization" Tags="&lt;data-visualization&gt;" AnswerCount="5" CommentCount="1" FavoriteCount="2" />
  <row Id="45" PostTypeId="2" ParentId="40" CreationDate="2010-07-19T19:34:44.033" Score="4" ViewCount="" Body="&lt;p&gt;The &lt;a href=&quot;http://en.wikipedia.org/wiki/Mersenne_twister&quot; rel=&quot;nofollow&quot;&gt;Mersenne Twister&lt;/a&gt; is one I've come across and used before now.&lt;/p&gt;&#xA;" OwnerUserId="55" LastActivityDate="2010-07-19T19:34:44.033" CommentCount="1" />
  <row Id="46" PostTypeId="2" ParentId="26" CreationDate="2010-07-19T19:35:04.827" Score="0" ViewCount="" Body="&lt;p&gt;A standard deviation is the square root of the second central moment of a distribution. A central moment is the expected difference from the expected value of the distribution. A first central moment would usually be 0, so we define a second central moment as the expected value of the squared distance of a random variable from its expected value. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To put it on a scale that is more in line with the original observations, we take the square root of that second central moment and call it the standard deviation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Standard deviation is a property of a population. It measures how much average &quot;dispersion&quot; there is to that population. Are all the obsrvations clustered around the mean, or are they widely spread out? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To estimate the standard deviation of a population, we often calculate the standard deviation of a &quot;sample&quot; from that population. To do this, you take observations from that population, calculate a mean of those observations, and then calculate the square root of the average squared deviation from that &quot;sample mean&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To get an unbiased estimator of the variance, you don't actually calculate the average squared deviation from the sample mean, but instead, you divide by (N-1) where N is the number of observations in your sample. Note that this &quot;sample standard deviation&quot; is not an unbiased estimator of the standard deviation, but the square of the &quot;sample standard deviation&quot; is an unbiased estimator of the variance of the population. &lt;/p&gt;&#xA;" OwnerUserId="62" LastEditorUserId="62" LastEditorDisplayName="" LastEditDate="2010-07-20T02:13:12.733" LastActivityDate="2010-07-20T02:13:12.733" CommentCount="2" />
  <row Id="47" PostTypeId="1" AcceptedAnswerId="268" CreationDate="2010-07-19T19:36:12.140" Score="6" ViewCount="171" Body="&lt;p&gt;I have a dataset of 130k internet users characterized by 4 variables describing users' number of sessions, locations visited, avg data download and session time aggregated from four months of activity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Dataset is very heavy-tailed. For example third of users logged only once during four months, whereas six users had more than 1000 sessions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wanted to come up with a simple classification of users, preferably with indication of the most appropriate number of clusters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there anything you could recomend as a soultion?&lt;/p&gt;&#xA;" OwnerUserId="22" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-10-08T16:05:59.363" LastActivityDate="2010-11-23T18:59:58.170" Title="Clustering of large, heavy-tailed dataset" Tags="&lt;clustering&gt;&lt;large-data&gt;" AnswerCount="4" CommentCount="2" FavoriteCount="2" />
  <row Id="48" PostTypeId="2" ParentId="36" CreationDate="2010-07-19T19:36:20.173" Score="3" ViewCount="" Body="&lt;p&gt;Flip through Freakonomics for some great examples. Their bibliography is chock full of references.&lt;/p&gt;&#xA;" OwnerUserId="36" LastActivityDate="2010-07-19T19:36:20.173" CommunityOwnedDate="2010-08-16T13:01:42.613" CommentCount="1" />
  <row Id="49" PostTypeId="2" ParentId="33" CreationDate="2010-07-19T19:36:52.403" Score="4" ViewCount="" Body="&lt;p&gt;You don't need to install any packages because this is possible with base-R functions.  Have a look at &lt;a href=&quot;http://www.stat.ucl.ac.be/ISdidactique/Rhelp/library/ts/html/arima.html&quot; rel=&quot;nofollow&quot;&gt;the arima function&lt;/a&gt;.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a basic function of &lt;a href=&quot;http://en.wikipedia.org/wiki/Box%E2%80%93Jenkins&quot; rel=&quot;nofollow&quot;&gt;Box-Jenkins analysis&lt;/a&gt;, so you should consider reading one of the R time series text-books for an overview; my favorite is Shumway and Stoffer. &quot;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0387293175&quot; rel=&quot;nofollow&quot;&gt;Time Series Analysis and Its Applications: With R Examples&lt;/a&gt;&quot;.&lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-07-19T19:36:52.403" />
  <row Id="50" PostTypeId="1" AcceptedAnswerId="85" CreationDate="2010-07-19T19:37:31.873" Score="5" ViewCount="457" Body="&lt;p&gt;What do they mean when they say &quot;random variable&quot;? &lt;/p&gt;&#xA;" OwnerUserId="62" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-08-07T17:56:13.817" LastActivityDate="2010-09-21T00:23:43.417" Title="What is meant by a &quot;random variable&quot;?" Tags="&lt;random-variable&gt;" AnswerCount="5" CommentCount="3" FavoriteCount="2" />
  <row Id="51" PostTypeId="1" AcceptedAnswerId="64" CreationDate="2010-07-19T19:37:55.630" Score="1" ViewCount="59" Body="&lt;p&gt;Are there any objective methods of assessment or standardized tests available to measure the effectiveness of a software that does pattern recognition?&lt;/p&gt;&#xA;" OwnerUserId="68" LastActivityDate="2010-07-19T20:12:16.453" Title="Measuring the effectiveness of a pattern recognition software" Tags="&lt;pattern-recognition&gt;" AnswerCount="1" FavoriteCount="0" />
  <row Id="52" PostTypeId="2" ParentId="8" CreationDate="2010-07-19T19:39:06.033" Score="5" ViewCount="" Body="&lt;p&gt;Usually one (p=.042), but it also depends on power.&lt;/p&gt;&#xA;" OwnerUserId="36" LastActivityDate="2010-07-19T19:39:06.033" />
  <row Id="53" PostTypeId="1" AcceptedAnswerId="78" CreationDate="2010-07-19T19:39:08.483" Score="16" ViewCount="2016" Body="&lt;p&gt;What are the main differences between performing Principal Components Analysis on a correlation and covariance matrix? Do they give the same results?&lt;/p&gt;&#xA;" OwnerUserId="17" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-16T06:47:21.213" LastActivityDate="2010-11-06T21:56:17.333" Title="PCA on Correlation or Covariance?" Tags="&lt;pca&gt;" AnswerCount="3" FavoriteCount="5" />
  <row Id="54" PostTypeId="1" AcceptedAnswerId="65" CreationDate="2010-07-19T19:41:19.367" Score="6" ViewCount="232" Body="&lt;p&gt;As I understand UK Schools teach that the Standard Deviation is found using:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://upload.wikimedia.org/math/e/e/4/ee485814ab9e19908f2b39d5d70406d5.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;whereas US Schools teach:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://upload.wikimedia.org/math/8/3/a/83a7338b851dcaafaf5b64353af56596.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(at a basic level anyway).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This has caused a number of my students problems in the past as they have searched on the Internet, but found the wrong explanation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why the difference?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With simple datasets say 10 values, what degree of error will there be if the wrong method is applied (eg in an exam)?&lt;/p&gt;&#xA;" OwnerUserId="55" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-10-19T07:20:40.307" LastActivityDate="2010-10-19T07:20:40.307" Title="Why do US and UK Schools Teach Different methods of Calculating the Standard Deviation?" Tags="&lt;standard-deviation&gt;&lt;error&gt;&lt;unbiased-estimator&gt;" AnswerCount="4" CommentCount="4" />
  <row Id="55" PostTypeId="2" ParentId="30" CreationDate="2010-07-19T19:41:39.880" Score="5" ViewCount="" Body="&lt;p&gt;The &lt;a href=&quot;http://en.wikipedia.org/wiki/Diehard_tests&quot; rel=&quot;nofollow&quot;&gt;Diehard Test Suite&lt;/a&gt; is something close to a Golden Standard for testing random number generators. It includes a number of tests where a good random number generator should produce result distributed according to some know distribution against which the outcome using the tested generator can then be compared.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have to update this since I was not exactly right:&#xA;Diehard might still be used a lot, but it is no longer maintained and not state-of-the-art anymore. NIST has come up with &lt;a href=&quot;http://csrc.nist.gov/groups/ST/toolkit/rng/index.html&quot; rel=&quot;nofollow&quot;&gt;a set of improved tests&lt;/a&gt; since.&lt;/p&gt;&#xA;" OwnerUserId="56" LastEditorUserId="56" LastEditorDisplayName="" LastEditDate="2011-05-12T18:38:27.547" LastActivityDate="2011-05-12T18:38:27.547" />
  <row Id="56" PostTypeId="2" ParentId="22" CreationDate="2010-07-19T19:42:28.040" Score="17" ViewCount="" Body="&lt;p&gt;Here is how I would explain the basic difference to my grandma:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have misplaced my phone somewhere in the home. I can use the phone locator on the base of the instrument to locate the phone and when I press the phone locator the phone starts beeping.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Problem: Which area of my home should I search?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Frequentist Reasoning: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can hear the phone beeping. I also have a mental model which helps me identify the area from which the sound is coming from. Therefore, upon hearing the beep, I infer the area of my home I must search to locate the phone.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bayesian Reasoning:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can hear the phone beeping. Now, apart from a mental model which helps me identify the area from which the sound is coming from, I also know the locations where I have misplaced the phone in the past. So, I combine my inferences using the beeps and my prior information about the locations I have misplaced the phone in the past to identify an area I must search to locate the phone.&lt;/p&gt;&#xA;" LastActivityDate="2010-07-19T19:42:28.040" CommentCount="3" />
  <row Id="57" PostTypeId="2" ParentId="50" CreationDate="2010-07-19T19:42:34.670" Score="2" ViewCount="" Body="&lt;p&gt;From &lt;a href=&quot;http://en.wikipedia.org/wiki/Random_variable&quot; rel=&quot;nofollow&quot;&gt;Wikipedia&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In mathematics (especially probability&#xA;  theory and statistics), a random&#xA;  variable (or stochastic variable) is&#xA;  (in general) a measurable function &#xA;  that maps a probability space into a&#xA;  measurable space. Random variables&#xA;  mapping all possible outcomes of an&#xA;  event into the real numbers are&#xA;  frequently studied in elementary&#xA;  statistics and used in the sciences to&#xA;  make predictions based on data&#xA;  obtained from scientific experiments.&#xA;  In addition to scientific&#xA;  applications, random variables were&#xA;  developed for the analysis of games of&#xA;  chance and stochastic  events. The&#xA;  utility of random variables comes from&#xA;  their ability to capture only the&#xA;  mathematical properties necessary to&#xA;  answer probabilistic  questions.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;From &lt;a href=&quot;http://cnx.org/content/m13418/latest/&quot; rel=&quot;nofollow&quot;&gt;cnx.org&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;A random variable is a function, which assigns unique numerical values to all possible&#xA;  outcomes of a random experiment under fixed conditions. A random variable is not a &#xA;  variable but rather a function that maps events to numbers.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="69" LastEditorUserId="69" LastEditorDisplayName="" LastEditDate="2010-07-19T19:47:54.623" LastActivityDate="2010-07-19T19:47:54.623" CommentCount="3" />
  <row Id="58" PostTypeId="1" AcceptedAnswerId="71" CreationDate="2010-07-19T19:42:57.990" Score="6" ViewCount="408" Body="&lt;p&gt;What is the back-propagation algorithm and how does it work?&lt;/p&gt;&#xA;" OwnerUserId="68" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-29T00:38:41.203" LastActivityDate="2011-04-29T00:38:41.203" Title="Can someone please explain the back-propagation algorithm?" Tags="&lt;algorithms&gt;&lt;optimization&gt;&lt;neural-networks&gt;" AnswerCount="4" CommentCount="0" />
  <row Id="59" PostTypeId="2" ParentId="2" CreationDate="2010-07-19T19:43:20.423" Score="9" ViewCount="" Body="&lt;p&gt;The assumption of normality is just the supposition that the underlying &lt;a href=&quot;http://en.wikipedia.org/wiki/Random_variable&quot; rel=&quot;nofollow&quot;&gt;random variable&lt;/a&gt; of interest is distributed &lt;a href=&quot;http://en.wikipedia.org/wiki/Normal_distribution&quot; rel=&quot;nofollow&quot;&gt;normally&lt;/a&gt;, or approximately so.  Intuitively, normality may be understood as the result of the sum of a large number of independent random events.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More specifically, normal distributions are defined by the following function:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://upload.wikimedia.org/math/9/e/1/9e1e4a3af93c9680ba75669a0b69fbf6.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where μ and σ^2 are the mean and the variance, respectively, and which appears as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://upload.wikimedia.org/wikipedia/commons/thumb/7/74/Normal_Distribution_PDF.svg/300px-Normal_Distribution_PDF.svg.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This can be checked in &lt;a href=&quot;http://en.wikipedia.org/wiki/Normality_test&quot; rel=&quot;nofollow&quot;&gt;multiple ways&lt;/a&gt;, that may be more or less suited to your problem by its features, such as the size of n.  Basically, they all test for features expected if the distribution were normal (e.g. expected &lt;a href=&quot;http://en.wikipedia.org/wiki/Q-Q_plot&quot; rel=&quot;nofollow&quot;&gt;quantile distribution&lt;/a&gt;).&lt;/p&gt;&#xA;" OwnerUserId="39" LastEditorUserId="39" LastEditorDisplayName="" LastEditDate="2010-07-19T20:02:04.230" LastActivityDate="2010-07-19T20:02:04.230" CommentCount="1" />
  <row Id="60" PostTypeId="2" ParentId="47" CreationDate="2010-07-19T19:43:20.720" Score="1" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/K-means_algorithm&quot; rel=&quot;nofollow&quot;&gt;K-Means clustering&lt;/a&gt; should work well for this type of problem.  However, it does require that you specify the number of clusters in advance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given the nature of this data, however, you may be able to work with a &lt;a href=&quot;http://en.wikipedia.org/wiki/Cluster_analysis#Hierarchical_clustering&quot; rel=&quot;nofollow&quot;&gt;hierarchical clustering algorithm&lt;/a&gt; instead.  Since all 4 variables are most likely fairly highly correlated, you can most likely break out clusters, and stop when you reach a small enough distance between clusters.  This may be a much simpler approach in this specific case, and allows you to determine &quot;how many clusters&quot; by just stopping as soon as you've broken your set into fine enough clusters.&lt;/p&gt;&#xA;" OwnerUserId="41" LastActivityDate="2010-07-19T19:43:20.720" CommentCount="1" />
  <row Id="61" PostTypeId="2" ParentId="26" CreationDate="2010-07-19T19:44:35.037" Score="10" ViewCount="" Body="&lt;p&gt;Standard deviation is a number that represents a concept. The concept in this case is the &quot;spread&quot; or &quot;dispersion&quot; of the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think that it is confusing to some due to the use of the word &quot;standard&quot;, where some may mistakenly believe that there is some &quot;standard&quot;, &quot;ordinary&quot;, &quot;common&quot; deviation, whereas it should be read as &quot;standardized&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are other metrics for spread, including range and variance. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A distribution with a mean of 1 and a standard deviation of 2 has more &quot;spread&quot; than a distribution with a mean of 1 and a standard deviation of 1. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If there is no spread, then the standard deviation is zero. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Standard deviation is often used as it is in the same units as the mean, unlike variance. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is convenient to know that &lt;a href=&quot;http://en.wikipedia.org/wiki/Chebyshev%27s_inequality&quot; rel=&quot;nofollow&quot;&gt;at least 75%&lt;/a&gt; of the data points lie within 2 standard deviations of the mean (or &lt;a href=&quot;http://en.wikipedia.org/wiki/68-95-99.7_rule&quot; rel=&quot;nofollow&quot;&gt;around 95%&lt;/a&gt; if the distribution is Normal).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if the mean is 100, and the standard deviation is 15, then at least 75% of the values are between 70 and 130. &lt;/p&gt;&#xA;" OwnerUserId="74" LastEditorUserId="74" LastEditorDisplayName="" LastEditDate="2011-01-14T20:45:06.250" LastActivityDate="2011-01-14T20:45:06.250" />
  <row Id="62" PostTypeId="1" AcceptedAnswerId="80" CreationDate="2010-07-19T19:44:58.033" Score="3" ViewCount="64" Body="&lt;p&gt;With the recent FIFA world cup, I decided to have some fun and determine which months produced world cup football players. Turned out, most footballers in the 2010 world cup were born in the first half of the year.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Someone pointed out, that children born in the first half of the year had a physical advantage over others and hence &quot;survivorship bias&quot; was involved in the equation. Is this an accurate observation? Can someone please explain why he says that?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, when trying to understand the concept, I found most examples revolved around the financial sector. Are they any other everyday life examples explaining it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="58" LastEditorUserId="58" LastEditorDisplayName="" LastEditDate="2010-07-19T19:50:45.180" LastActivityDate="2010-07-20T16:19:51.750" Title="A case of survivorship bias?" Tags="&lt;statistical-bias&gt;" AnswerCount="2" />
  <row Id="63" PostTypeId="2" ParentId="36" CreationDate="2010-07-19T19:45:19.780" Score="13" ViewCount="" Body="&lt;p&gt;It might be useful to explain that &quot;causes&quot; is an asymmetric relation (X causes Y is different from Y causes X), whereas &quot;is correlated with&quot; is a symmetric relation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, homeless population and crime rate might be correlated, in that both tend to be high or low in the same locations.  It is equally valid to say that homelesss population is correlated with crime rate, or crime rate is correlated with homeless population.  To say that crime causes homelessness, or homeless populations cause crime are different statements.  And correlation does not imply that either is true.  For instance, the underlying cause could be a 3rd variable such as drug abuse, or unemployment.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The mathematics of statistics is not good at identifying underlying causes, which requires some other form of judgement.&lt;/p&gt;&#xA;" OwnerUserId="87" LastActivityDate="2010-07-19T19:45:19.780" CommunityOwnedDate="2010-08-16T13:01:42.613" CommentCount="2" />
  <row Id="64" PostTypeId="2" ParentId="51" CreationDate="2010-07-19T19:46:08.217" Score="3" ViewCount="" Body="&lt;p&gt;Yes, there are many methods.  You would need to specify which model you're using, because it can vary.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, Some models will be compared based on the &lt;a href=&quot;http://en.wikipedia.org/wiki/Akaike_information_criterion&quot; rel=&quot;nofollow&quot;&gt;AIC&lt;/a&gt; or &lt;a href=&quot;http://en.wikipedia.org/wiki/Bayesian_information_criterion&quot; rel=&quot;nofollow&quot;&gt;BIC&lt;/a&gt; criteria.  In other cases, one would look at the &lt;a href=&quot;http://en.wikipedia.org/wiki/Cross-validation_%28statistics%29&quot; rel=&quot;nofollow&quot;&gt;MSE from cross validation&lt;/a&gt; (as, for instance, with a support vector machine).&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;I recommend reading &lt;a href=&quot;http://research.microsoft.com/en-us/um/people/cmbishop/prml/&quot; rel=&quot;nofollow&quot;&gt;Pattern Recognition and Machine Learning&lt;/a&gt; by Christopher Bishop.&lt;/li&gt;&#xA;&lt;li&gt;This is also discussed in Chapter 5 on Credibility, and particularly section 5.5 &quot;Comparing data mining methods&quot; of &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0120884070&quot; rel=&quot;nofollow&quot;&gt;Data Mining: Practical Machine Learning Tools and Techniques&lt;/a&gt; by Witten and Frank (which discusses Weka in detail).&lt;/li&gt;&#xA;&lt;li&gt;Lastly, you should also have a look at &lt;a href=&quot;http://www-stat.stanford.edu/~tibs/ElemStatLearn/&quot; rel=&quot;nofollow&quot;&gt;The Elements of Statistical Learning&lt;/a&gt; by Hastie, Tibshirani and Friedman which is available for free online.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="5" LastEditorUserId="5" LastEditorDisplayName="" LastEditDate="2010-07-19T20:12:16.453" LastActivityDate="2010-07-19T20:12:16.453" />
  <row Id="65" PostTypeId="2" ParentId="54" CreationDate="2010-07-19T19:46:11.383" Score="11" ViewCount="" Body="&lt;p&gt;The first formula is the &lt;em&gt;population&lt;/em&gt; standard deviation and the second formula is the the &lt;em&gt;sample&lt;/em&gt; standard deviation. The second formula is also related to the unbiased estimator of the variance - see &lt;a href=&quot;http://en.wikipedia.org/wiki/Variance#Population_variance_and_sample_variance&quot; rel=&quot;nofollow&quot;&gt;wikipedia&lt;/a&gt; for further details.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suppose (here) in the UK they don't make the distinction between sample and population at high school. They certainly don't touch concepts such as biased estimators. &lt;/p&gt;&#xA;" OwnerUserId="8" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-07-19T20:44:53.427" LastActivityDate="2010-07-19T20:44:53.427" CommentCount="3" />
  <row Id="66" PostTypeId="2" ParentId="54" CreationDate="2010-07-19T19:46:11.993" Score="4" ViewCount="" Body="&lt;p&gt;This is &lt;a href=&quot;http://en.wikipedia.org/wiki/Bessel%27s_correction&quot; rel=&quot;nofollow&quot;&gt;Bessel's Correction&lt;/a&gt;.  The US version is showing the formula for the &lt;em&gt;sample standard deviation&lt;/em&gt;, where the UK version above is the &lt;em&gt;standard deviation of the sample&lt;/em&gt;.&lt;/p&gt;&#xA;" OwnerUserId="41" LastActivityDate="2010-07-19T19:46:11.993" />
  <row Id="67" PostTypeId="2" ParentId="44" CreationDate="2010-07-19T19:47:16.577" Score="1" ViewCount="" Body="&lt;p&gt;From &lt;a href=&quot;http://en.wikipedia.org/wiki/Data_visualization&quot; rel=&quot;nofollow&quot;&gt;Wikipedia&lt;/a&gt;: Data visualization is the study of the visual representation of data, meaning &quot;information which has been abstracted in some schematic form, including attributes or variables for the units of information&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data viz is important for visualizing trends in data, telling a story - See &lt;a href=&quot;http://www.edwardtufte.com/tufte/posters&quot; rel=&quot;nofollow&quot;&gt;Minard's map of Napoleon's march&lt;/a&gt; - possibly one of the best data graphics ever printed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also see any of Edward Tufte's books - especially &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0961392142&quot; rel=&quot;nofollow&quot;&gt;Visual Display of Quantitative Information.&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="36" LastActivityDate="2010-07-19T19:47:16.577" />
  <row Id="69" PostTypeId="2" ParentId="54" CreationDate="2010-07-19T19:47:49.587" Score="4" ViewCount="" Body="&lt;p&gt;Since N is the number of points in the data set, one could argue that by calculating the mean one has reduced the degree of freedom in the data set by one (since one introduced a dependency into the data set), so one should use N-1 when estimating the standard deviation from a data set for which one had to estimate the mean before.&lt;/p&gt;&#xA;" OwnerUserId="56" LastActivityDate="2010-07-19T19:47:49.587" />
  <row Id="70" PostTypeId="2" ParentId="7" CreationDate="2010-07-19T19:48:45.853" Score="9" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://data.worldbank.org/data-catalog&quot; rel=&quot;nofollow&quot;&gt;World Bank&lt;/a&gt; offers quite a lot of interesting data and has been recently very active in developing nice &lt;a href=&quot;http://data.worldbank.org/developers/api-overview&quot; rel=&quot;nofollow&quot;&gt;API&lt;/a&gt; for it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, &lt;a href=&quot;http://www.cs.purdue.edu/commugrate/data_access/all_data_sets.php&quot; rel=&quot;nofollow&quot;&gt;commugrate&lt;/a&gt; project has an interesting list avaialble.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For US health related data head for &lt;a href=&quot;http://www.healthindicators.gov/&quot; rel=&quot;nofollow&quot;&gt;Health Indicators Warehouse&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="22" LastEditorUserId="22" LastEditorDisplayName="" LastEditDate="2011-03-02T13:19:47.840" LastActivityDate="2011-03-02T13:19:47.840" />
  <row Id="71" PostTypeId="2" ParentId="58" CreationDate="2010-07-19T19:50:33.480" Score="2" ViewCount="" Body="&lt;p&gt;It's an algorithm for training feedforward multilayer neural networks (multilayer perceptrons). There are several nice java applets around the web that illustrate what's happening, like this one: &lt;a href=&quot;http://neuron.eng.wayne.edu/bpFunctionApprox/bpFunctionApprox.html&quot; rel=&quot;nofollow&quot;&gt;http://neuron.eng.wayne.edu/bpFunctionApprox/bpFunctionApprox.html&lt;/a&gt;. Also, &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0198538642&quot; rel=&quot;nofollow&quot;&gt;Bishop's book on NNs&lt;/a&gt; is the standard desk reference for anything to do with NNs.&lt;/p&gt;&#xA;" OwnerUserId="36" LastActivityDate="2010-07-19T19:50:33.480" />
  <row Id="72" PostTypeId="2" ParentId="35" CreationDate="2010-07-19T19:51:05.117" Score="4" ViewCount="" Body="&lt;p&gt;for overdispersed poisson, use the negative binomial, which allows you to parameterize the variance as a function of the mean precisely.  rnbinom(), etc. in R.&lt;/p&gt;&#xA;" OwnerUserId="96" LastActivityDate="2010-07-19T19:51:05.117" />
  <row Id="73" PostTypeId="1" AcceptedAnswerId="76" CreationDate="2010-07-19T19:51:32.587" Score="15" ViewCount="963" Body="&lt;p&gt;&lt;strong&gt;Duplicate thread:&lt;/strong&gt; &lt;a href=&quot;http://stats.stackexchange.com/questions/1676/i-just-installed-the-latest-version-of-r-what-packages-should-i-obtain&quot;&gt;I just installed the latest version of R. What packages should I obtain?&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are the &lt;a href=&quot;http://www.r-project.org/&quot; rel=&quot;nofollow&quot;&gt;R&lt;/a&gt; packages you couldn't imagine your daily work with data?&#xA;Please list both general and specific tools.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;UPDATE:&#xA;As for 24.10.10 &lt;code&gt;ggplot2&lt;/code&gt; seems to be the winer with 7 votes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other packages mentioned more than one are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;plyr&lt;/code&gt; - 4&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;RODBC&lt;/code&gt;, &lt;code&gt;RMySQL&lt;/code&gt; - 4&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;sqldf&lt;/code&gt; - 3&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;lattice&lt;/code&gt; - 2&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;zoo&lt;/code&gt; - 2&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;Hmisc/rms&lt;/code&gt; - 2&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;Rcurl&lt;/code&gt; - 2&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;XML&lt;/code&gt; - 2&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Thanks all for your answers!&lt;/p&gt;&#xA;" OwnerUserId="22" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-11T12:29:47.480" LastActivityDate="2011-05-11T12:33:30.563" CommunityOwnedDate="2010-07-19T20:07:49.007" Title="What R packages do you find most useful in your daily work?" Tags="&lt;r&gt;" AnswerCount="19" CommentCount="5" FavoriteCount="17" />
  <row Id="74" PostTypeId="2" ParentId="6" CreationDate="2010-07-19T19:51:34.287" Score="5" ViewCount="" Body="&lt;p&gt;In such a discussion, I always recall the famous Ken Thompson quote &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;When in doubt, use brute force.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In this case, machine learning is a salvation when the assumptions are hard to catch; or at least it is much better than guessing them wrong. &lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-07-19T19:51:34.287" />
  <row Id="75" PostTypeId="1" AcceptedAnswerId="94" CreationDate="2010-07-19T19:52:31.180" Score="4" ViewCount="130" Body="&lt;p&gt;I'm using &lt;a href=&quot;http://www.r-project.org/&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;R&lt;/strong&gt;&lt;/a&gt; and the manuals on the R site are really informative. However, I'd like to see some more examples and implementations with R which can help me develop my knowledge faster. Any suggestions?&lt;/p&gt;&#xA;" OwnerUserId="69" LastActivityDate="2010-07-21T03:12:51.570" CommunityOwnedDate="2010-10-19T11:13:59.663" Title="Where can I find useful R tutorials with various implementations?" Tags="&lt;r&gt;&lt;tutorial&gt;&lt;code&gt;" AnswerCount="5" CommentCount="4" FavoriteCount="3" />
  <row Id="76" PostTypeId="2" ParentId="73" CreationDate="2010-07-19T19:52:49.387" Score="18" ViewCount="" Body="&lt;p&gt;I use &lt;a href=&quot;http://cran.r-project.org/web/packages/plyr/index.html&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;plyr&lt;/strong&gt;&lt;/a&gt; and &lt;a href=&quot;http://cran.r-project.org/web/packages/ggplot2/index.html&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;ggplot2&lt;/strong&gt;&lt;/a&gt; the most on a daily basis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also rely heavily on time series packages; most especially, the &lt;a href=&quot;http://cran.r-project.org/web/packages/zoo/index.html&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;zoo&lt;/strong&gt;&lt;/a&gt; package.&lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-07-19T19:52:49.387" CommunityOwnedDate="2010-08-09T13:16:41.033" CommentCount="3" />
  <row Id="77" PostTypeId="2" ParentId="36" CreationDate="2010-07-19T19:54:03.253" Score="6" ViewCount="" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Sometimes correlation is enough. For example, in car insurance, male drivers are correlated with more accidents, so insurance companies charge them more. There is no way you could actually test this for causation. You cannot change the genders of the drivers experimentally. Google has made hundreds of billions of dollars not caring about causation. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;To find causation, you generally need experimental data, not observational data. Though, in economics, they often use observed &quot;shocks&quot; to the system to test for causation, like if a CEO dies suddenly and the stock price goes up, you can assume causation. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Correlation is a necessary but not sufficient condition for causation. To show causation requires a counter-factual.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="74" LastEditorUserId="74" LastEditorDisplayName="" LastEditDate="2010-09-09T18:16:59.853" LastActivityDate="2010-09-09T18:16:59.853" CommunityOwnedDate="2010-08-16T13:01:42.613" CommentCount="2" />
  <row Id="78" PostTypeId="2" ParentId="53" CreationDate="2010-07-19T19:54:38.710" Score="12" ViewCount="" Body="&lt;p&gt;You tend to use the covariance matrix when the variable scales are similar and the correlation matrix when variables are on different scales. Using the correlation matrix &lt;em&gt;standardises&lt;/em&gt; the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general they give different results. Especially when the scales are different.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As example, take a look a look at this R heptathlon data set. Some of the variables have an average value of about 1.8 (the high jump), whereas other variables (200m) are around 20.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(HSAUR)&#xA;# look at heptathlon data&#xA;heptathlon&#xA;&#xA;# correlations&#xA;round(cor(heptathlon[,-8]),2)   # ignoring &quot;score&quot; &#xA;# covariance&#xA;round(cov(heptathlon[,-8]),2)&#xA;&#xA;# PCA&#xA;# scale=T bases the PCA on the correlation matrix&#xA;hep.PC.cor = prcomp(heptathlon[,-8], scale=TRUE)&#xA;hep.PC.cov = prcomp(heptathlon[,-8], scale=FALSE)&#xA;&#xA;# PC scores per competitor&#xA;hep.scores.cor = predict(hep.PC.cor)&#xA;hep.scores.cov = predict(hep.PC.cov)&#xA;&#xA;# Plot of PC1 vs PC2&#xA;par(mfrow = c(2, 1))&#xA;plot(hep.scores.cov[,1],hep.scores.cov[,2],&#xA;     xlab=&quot;PC 1&quot;,ylab=&quot;PC 2&quot;, pch=NA, main=&quot;Covariance&quot;)&#xA;text(hep.scores.cov[,1],hep.scores.cov[,2],labels=1:25) &#xA;&#xA;plot(hep.scores.cor[,1],hep.scores.cor[,2],&#xA;     xlab=&quot;PC 1&quot;,ylab=&quot;PC 2&quot;, pch=NA, main=&quot;Correlation&quot;)&#xA;text(hep.scores.cor[,1],hep.scores.cor[,2],labels=1:25) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Notice that the outlying individuals (in &lt;em&gt;this&lt;/em&gt; data set) are outliers regardless of whether the covariance or correlation matrix is used.&lt;/p&gt;&#xA;" OwnerUserId="8" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-11-06T21:56:17.333" LastActivityDate="2010-11-06T21:56:17.333" CommentCount="2" />
  <row Id="79" PostTypeId="2" ParentId="54" CreationDate="2010-07-19T19:56:04.120" Score="3" ViewCount="" Body="&lt;p&gt;I am not sure this is purely a US vs. British issue. Here is a brief page I wrote &lt;a href=&quot;http://www.graphpad.com/faq/viewfaq.cfm?faq=1382&quot; rel=&quot;nofollow&quot;&gt;explaining the difference between using n vs. n-1 when computing a Standard Deviation&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="25" LastActivityDate="2010-07-19T19:56:04.120" CommentCount="1" />
  <row Id="80" PostTypeId="2" ParentId="62" CreationDate="2010-07-19T19:56:43.807" Score="7" ViewCount="" Body="&lt;p&gt;The basic idea behind this is that football clubs have an age cut-off when determining teams.  In the league my children participate in the age restrictions states that children born after July 31st are placed on the younger team.  This means that two children that are effectively the same age can be playing with two different age groups.  The child born July 31st will be playing on the older team and theoretically be the youngest and smallest on the team and in the league.  The child born on August 1st will be the oldest and largest child in the league and will be able to benefit from that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The survivorship bias comes because competitive leagues will select the best players for their teams.  The best players in childhood are often the older players since they have additional time for their bodies to mature.  This means that otherwise acceptable younger players are not selected simply because of their age.  Since they are not given the same opportunities as the older kids, they don’t develop the same skills and eventually drop out of competitive soccer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the cut-off for competitive soccer in enough countries is January 1st, that would support the phenomena you see.  A similar phenomena has been observed in several other sports including baseball and ice hockey.&lt;/p&gt;&#xA;" OwnerUserId="93" LastActivityDate="2010-07-19T19:56:43.807" CommentCount="2" />
  <row Id="81" PostTypeId="2" ParentId="73" CreationDate="2010-07-19T19:58:56.157" Score="6" ViewCount="" Body="&lt;p&gt;I use the &lt;strong&gt;&lt;a href=&quot;http://cran.r-project.org/web/packages/xtable/index.html&quot; rel=&quot;nofollow&quot;&gt;xtable&lt;/a&gt;&lt;/strong&gt; package. The xtable package turns tables produced by R (in particular, the tables displaying the anova results) into LaTeX tables, to be included in an article. &lt;/p&gt;&#xA;" OwnerUserId="69" LastActivityDate="2010-07-19T19:58:56.157" CommunityOwnedDate="2010-10-19T11:07:08.727" />
  <row Id="82" PostTypeId="2" ParentId="73" CreationDate="2010-07-19T20:01:38.803" Score="3" ViewCount="" Body="&lt;p&gt;I second the use of &lt;code&gt;plyr&lt;/code&gt; and &lt;code&gt;ggplot2&lt;/code&gt;.  I also use &lt;code&gt;maptools&lt;/code&gt; and various map related libaries.  I could not live without &lt;code&gt;RMySQL&lt;/code&gt; and &lt;code&gt;RpgSQL&lt;/code&gt; for retrieving data.  Also the &lt;code&gt;sqldf&lt;/code&gt; package so I don't have to switch mental gears when going from databases to dataframes. &lt;/p&gt;&#xA;" OwnerUserId="72" LastActivityDate="2010-07-19T20:01:38.803" CommunityOwnedDate="2010-10-19T11:07:08.727" />
  <row Id="83" PostTypeId="2" ParentId="75" CreationDate="2010-07-19T20:02:51.297" Score="1" ViewCount="" Body="&lt;p&gt;R is designed around ideas such as &quot;reproducible research&quot; and &quot;trustworthy software&quot;, as John Chambers says &lt;a href=&quot;http://books.google.com/books?id=UXneuOIvhEAC&amp;amp;printsec=frontcover&quot; rel=&quot;nofollow&quot;&gt;in his excellent book &quot;Software for Data Analysis: Programming with R&quot;&lt;/a&gt;.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of the best ways to learn R is to look at the wealth of source code that available on &lt;a href=&quot;http://cran.r-project.org/&quot; rel=&quot;nofollow&quot;&gt;CRAN&lt;/a&gt; (with 2461 packages and counting).  Simple &lt;code&gt;install.packages&lt;/code&gt;, load a &lt;code&gt;library()&lt;/code&gt;, and start browsing the code.&lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-07-19T20:02:51.297" CommunityOwnedDate="2010-10-19T11:13:59.663" />
  <row Id="84" PostTypeId="2" ParentId="44" CreationDate="2010-07-19T20:03:34.307" Score="4" ViewCount="" Body="&lt;p&gt;I would explain it to a layman as:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Data visualization is taking data, and making a picture out of it.  This allows you to easily see and understand relationships within the data much more easily than just looking at the numbers.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="41" LastActivityDate="2010-07-19T20:03:34.307" />
  <row Id="85" PostTypeId="2" ParentId="50" CreationDate="2010-07-19T20:08:00.927" Score="6" ViewCount="" Body="&lt;p&gt;A random variable is a variable whose value depends on unknown events.  We can summarize the unknown events as &quot;state&quot;, and then the random variable is a function of the state.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example:  Suppose we have three dice rolls (D1,D2,D3).  Then the state S=(D1,D2,D3).  One random variable X is the number of 5s. X=(D1==5?)+(D2==5?)+(D3==5?).   Another random variable Y is the sum of the dice rolls Y=D1+D2+D3.  &lt;/p&gt;&#xA;" OwnerUserId="87" LastEditorUserId="87" LastEditorDisplayName="" LastEditDate="2010-07-28T14:54:34.317" LastActivityDate="2010-07-28T14:54:34.317" />
  <row Id="86" PostTypeId="2" ParentId="50" CreationDate="2010-07-19T20:08:37.010" Score="4" ViewCount="" Body="&lt;p&gt;Unlike a regular variable, a random variable may not be substituted for a single, unchanging value.  Rather &lt;strong&gt;statistical properties&lt;/strong&gt; such as the &lt;strong&gt;distribution&lt;/strong&gt; of the random variable may be stated.  The distribution is a function that provides the probability the variable will take on a given value, or fall within a range given certain parameters such as the mean or standard deviation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Random variables may be classified as &lt;em&gt;discrete&lt;/em&gt; if the distribution describes values from a countable set, such as the integers.  The other classification for a random variable is &lt;em&gt;continuous&lt;/em&gt; and is used if the distribution covers values from an uncountable set such as the real numbers.&lt;/p&gt;&#xA;" OwnerUserId="13" LastEditorUserId="13" LastEditorDisplayName="" LastEditDate="2010-07-20T17:46:55.840" LastActivityDate="2010-07-20T17:46:55.840" CommentCount="4" />
  <row Id="89" PostTypeId="2" ParentId="44" CreationDate="2010-07-19T20:11:47.797" Score="2" ViewCount="" Body="&lt;p&gt;When I teach very basic statistics to Secondary School Students I talk about evolution and how we have evolved to spot patterns in pictures rather than lists of numbers and that data visualisation is one of the techniques we use to take advantage of this fact. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Plus I try to talk about recent news stories where statistical insight contradicts what the press is implying, making use of sites like &lt;a href=&quot;http://www.gapminder.org/&quot; rel=&quot;nofollow&quot;&gt;Gapminder&lt;/a&gt; to find the representation before choosing the story.&lt;/p&gt;&#xA;" OwnerUserId="55" LastActivityDate="2010-07-19T20:11:47.797" />
  <row Id="90" PostTypeId="2" ParentId="75" CreationDate="2010-07-19T20:12:24.130" Score="2" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://www.r-bloggers.com/&quot; rel=&quot;nofollow&quot;&gt;R bloggers&lt;/a&gt; has been steadily supplying me with a lot of good pragmatic content.&lt;br&gt;&#xA;From the author:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;R-Bloggers.com is a central hub (e.g: A blog aggregator) of content &#xA;collected from bloggers who write about R (in English). &#xA;The site will help R bloggers and users to connect and follow &#xA;the “R blogosphere”.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="22" LastActivityDate="2010-07-19T20:12:24.130" CommunityOwnedDate="2010-10-19T11:13:59.663" />
  <row Id="91" PostTypeId="2" ParentId="23" CreationDate="2010-07-19T20:15:54.823" Score="6" ViewCount="" Body="&lt;p&gt;As user28 said in comments above, the pdf is the first derivative of the cdf for a continuous random variable, and the difference for a discrete random variable. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the continuous case, wherever the cdf has a discontinuity the pdf has an atom.  Dirac delta &quot;functions&quot; can be used to represent these atoms.    &lt;/p&gt;&#xA;" OwnerUserId="87" LastEditorUserId="87" LastEditorDisplayName="" LastEditDate="2010-07-20T08:52:27.083" LastActivityDate="2010-07-20T08:52:27.083" />
  <row Id="93" PostTypeId="1" CreationDate="2010-07-19T20:17:07.643" Score="5" ViewCount="288" Body="&lt;p&gt;We're trying to use a Gaussian process to model h(t) -- the hazard function -- for a very small initial population, and then fit that using the available data.  While this gives us nice plots for credible sets for h(t) and so on, it unfortunately is also just pushing the inference problem from h(t) to the covariance function of our process.  Perhaps predictably, we have several reasonable and equally defensible guesses for this that all produce different result.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Has anyone run across any good approaches for addressing such a problem?  Gaussian-process related or otherwise?&lt;/p&gt;&#xA;" OwnerUserId="61" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-16T12:33:50.390" LastActivityDate="2010-09-16T12:33:50.390" Title="Robust nonparametric estimation of hazard/survival functions based on low count data" Tags="&lt;nonparametric&gt;&lt;survival&gt;&lt;hazard&gt;" AnswerCount="1" CommentCount="3" FavoriteCount="3" />
  <row Id="94" PostTypeId="2" ParentId="75" CreationDate="2010-07-19T20:18:24.520" Score="3" ViewCount="" Body="&lt;p&gt;Quick R site is basic, but quite nice for start &lt;a href=&quot;http://www.statmethods.net/index.html&quot; rel=&quot;nofollow&quot;&gt;http://www.statmethods.net/index.html&lt;/a&gt; . &lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-07-19T20:18:24.520" CommunityOwnedDate="2010-10-19T11:13:59.663" />
  <row Id="95" PostTypeId="1" CreationDate="2010-07-19T20:21:35.683" Score="5" ViewCount="174" Body="&lt;p&gt;I have been using various GARCH-based models to forecast volatility for various North American equities using historical daily data as inputs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Asymmetric GARCH models are often cited as a modification of the basic GARCH model to account for the 'leverage effect' i.e. volatility tends to increase more after a negative return than a similarly sized positive return.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What kind of a difference would you expect to see between a standard GARCH and an asymmetric GARCH forecast for a broad-based equity index like the S&amp;amp;P 500 or the NASDAQ-100?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is nothing particularly special about these two indices, but I think it is helpful to give something concrete to focus the discussion, as I am sure the effect would be different depending on the equities used.&lt;/p&gt;&#xA;" OwnerUserId="57" LastEditorUserId="154" LastEditorDisplayName="" LastEditDate="2010-07-20T06:14:40.363" LastActivityDate="2010-08-01T13:38:50.213" Title="How Large a Difference Can Be Expected Between Standard GARCH and Asymmetric GARCH Volatility Forecasts?" Tags="&lt;time-series&gt;&lt;garch&gt;&lt;volatility-forecasting&gt;&lt;finance&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="96" PostTypeId="2" ParentId="75" CreationDate="2010-07-19T20:22:44.807" Score="3" ViewCount="" Body="&lt;p&gt;Another great resource is &lt;a href=&quot;http://learnr.wordpress.com/&quot; rel=&quot;nofollow&quot;&gt;the LearnR blog&lt;/a&gt;, which went through an extensive study of visualizations with lattice and ggplot2.&lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-07-19T20:22:44.807" CommunityOwnedDate="2010-10-19T11:13:59.663" />
  <row Id="97" PostTypeId="1" AcceptedAnswerId="104" CreationDate="2010-07-19T20:23:22.603" Score="10" ViewCount="612" Body="&lt;p&gt;I have some &lt;a href=&quot;http://en.wikipedia.org/wiki/Ordinal_scale#Ordinal_scale&quot; rel=&quot;nofollow&quot;&gt;ordinal data&lt;/a&gt; gained from survey questions.  In my case they are &lt;a href=&quot;http://en.wikipedia.org/wiki/Likert_scale&quot; rel=&quot;nofollow&quot;&gt;Likert style&lt;/a&gt; responses (Strongly Disagree-Disagree-Neutral-Agree-Strongly Agree).  In my data they are coded as 1-5.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I  don't think means would mean much here, so what basic summary statistics are considered usefull?&lt;/p&gt;&#xA;" OwnerUserId="114" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-31T07:30:20.490" LastActivityDate="2011-03-31T07:30:20.490" Title="What are good basic statistics to use for ordinal data?" Tags="&lt;summary-statistics&gt;&lt;likert&gt;&lt;ordinal&gt;" AnswerCount="4" FavoriteCount="4" />
  <row Id="98" PostTypeId="2" ParentId="1" CreationDate="2010-07-19T20:23:57.330" Score="6" ViewCount="" Body="&lt;p&gt;Eliciting priors is a tricky business. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.stat.cmu.edu/tr/tr808/tr808.pdf&quot; rel=&quot;nofollow&quot;&gt;Statistical Methods for Eliciting Probability Distributions&lt;/a&gt; and &lt;a href=&quot;http://www.jeremy-oakley.staff.shef.ac.uk/Oakley_elicitation.pdf&quot; rel=&quot;nofollow&quot;&gt;Eliciting Probability Distributions&lt;/a&gt; are quite good practical guides for prior elicitation.  The process in both papers is outlined as follows:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;background and preparation;&lt;/li&gt;&#xA;&lt;li&gt;identifying and recruiting the expert(s);&lt;/li&gt;&#xA;&lt;li&gt;motivation and training the expert(s);&lt;/li&gt;&#xA;&lt;li&gt;structuring and decomposition (typically deciding precisely what variables should&#xA;be elicited, and how to elicit joint distributions in the multivariate case);&lt;/li&gt;&#xA;&lt;li&gt;the elicitation itself.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Of course, they also review how the elicitation results in information that may be fit to or otherwise define distributions (for instance, in the Bayesian context, &lt;a href=&quot;http://en.wikipedia.org/wiki/Beta_distribution&quot; rel=&quot;nofollow&quot;&gt;Beta distributions&lt;/a&gt;), but quite importantly, they also address common pitfalls in modeling expert knowledge (anchoring, narrow and small-tailed distributions, etc.).&lt;/p&gt;&#xA;" OwnerUserId="39" LastEditorUserId="39" LastEditorDisplayName="" LastEditDate="2010-07-19T20:56:56.330" LastActivityDate="2010-07-19T20:56:56.330" />
  <row Id="99" PostTypeId="2" ParentId="73" CreationDate="2010-07-19T20:25:08.793" Score="6" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://cran.r-project.org/web/packages/multicore/index.html&quot; rel=&quot;nofollow&quot;&gt;multicore&lt;/a&gt; is quite nice for tool for making faster scripts faster.&lt;br&gt;&#xA;&lt;a href=&quot;http://cran.r-project.org/web/packages/cacheSweave/index.html&quot; rel=&quot;nofollow&quot;&gt;cacheSweave&lt;/a&gt; saves a lot of time when using &lt;code&gt;Sweave&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-07-19T20:25:08.793" CommunityOwnedDate="2010-07-19T20:25:08.793" CommentCount="1" />
  <row Id="100" PostTypeId="1" AcceptedAnswerId="110" CreationDate="2010-07-19T20:30:23.320" Score="2" ViewCount="214" Body="&lt;p&gt;I'd like to see the answer with qualitative view on the problem, not just definition. Examples and analogous from other areas of applied math also would be good.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand, my question is silly, but I can't find good and intuitive introduction textbook on signal processing — if someone would suggest one, I will be happy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="117" LastActivityDate="2010-07-19T20:40:36.573" Title="What's the purpose of window function in spectral analysis?" Tags="&lt;signal-processing&gt;" AnswerCount="1" />
  <row Id="101" PostTypeId="2" ParentId="31" CreationDate="2010-07-19T20:32:08.183" Score="12" ViewCount="" Body="&lt;p&gt;Understanding p-value&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose, that you want to test the hypothesis that the average height of male students at your University is 5 ft 7 inches. You collect heights of 100 students selected at random and compute the sample mean (say it turns out to be 5 ft 9 inches). Using an appropriate formula/statistical routine you compute the p-value for your hypothesis and say it turns out to be 0.06. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to interpret p=0.06 appropriately, we should keep several things in mind:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;The first step under classical hypothesis testing is the assumption that the hypothesis under consideration is true. (In our context, we assume that the &lt;strong&gt;true&lt;/strong&gt; average height is 5 ft 7 inches.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Imagine doing the following calculation: Compute the probability that the sample mean is greater than 5 ft 9 inches assuming that our hypothesis is in fact correct (see point 1). &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In other words, we want to know P(Sample mean &gt;= 5 ft 9 inches | Given true value = 5 ft 7 inches). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The calculation in step 2 is what is called the p-value. Therefore, a p-value of 0.06 would mean that if we were to repeat our experiment many, many times (each time we select 100 students at random and compute the sample mean) then 6 times out of 100 we can expect to see a sample mean greater than 5 ft 9 inches.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given the above understanding, should we still retain our assumption that our hypothesis is true (see step 1). Well, a p=0.06 indicates that one of two things have happened: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(A) Either our hypothesis is correct and an extremely unlikely event has occurred (e.g., all 100 students are student athletes)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;or&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(B) Our assumption is incorrect and the sample we have obtained is not that unusual. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The traditional way to choose between (A) and (B) is to choose an arbitrary cut-off for p. We choose (A) if p &gt; 0.05 and (B) if p &amp;lt; 0.05.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: Need to run now. Will add about the t-test when I find some more time.&lt;/p&gt;&#xA;" LastActivityDate="2010-07-19T20:32:08.183" CommentCount="3" />
  <row Id="102" PostTypeId="2" ParentId="44" CreationDate="2010-07-19T20:32:14.283" Score="-1" ViewCount="" Body="&lt;p&gt;For me &lt;a href=&quot;http://nvac.pnl.gov/agenda.stm&quot; rel=&quot;nofollow&quot;&gt;Illuminating the Path&lt;/a&gt; report has been always good point of reference.&lt;br&gt;&#xA;For more recent overview you can also have a look at good &lt;a href=&quot;http://queue.acm.org/detail.cfm?id=1805128&quot; rel=&quot;nofollow&quot;&gt;article&lt;/a&gt; by Heer and colleagues.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But what would explain better than visualization itself?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/uFvje.jpg&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(&lt;a href=&quot;http://blog.ffctn.com/what-is-data-visualization&quot; rel=&quot;nofollow&quot;&gt;Source&lt;/a&gt;)&lt;/p&gt;&#xA;" OwnerUserId="22" LastEditorUserId="22" LastEditorDisplayName="" LastEditDate="2010-11-23T20:42:01.350" LastActivityDate="2010-11-23T20:42:01.350" />
  <row Id="103" PostTypeId="1" CreationDate="2010-07-19T20:33:26.713" Score="11" ViewCount="369" Body="&lt;p&gt;What is the best blog on data visualization?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm making this question a community wiki since it is highly subjective.  Please limit each answer to one link.&lt;/p&gt;&#xA;" OwnerUserId="5" LastEditorUserId="5" LastEditorDisplayName="" LastEditDate="2010-07-19T21:25:22.670" LastActivityDate="2011-03-16T23:59:43.900" CommunityOwnedDate="2010-07-19T20:33:26.713" Title="What is your favorite data visualization blog?" Tags="&lt;data-visualization&gt;&lt;blog&gt;&lt;best-of&gt;" AnswerCount="14" CommentCount="3" FavoriteCount="13" />
  <row Id="104" PostTypeId="2" ParentId="97" CreationDate="2010-07-19T20:34:46.820" Score="6" ViewCount="" Body="&lt;p&gt;A frequency table is a good place to start. You can do the count, and relative frequency for each level. Also, the total count, and number of missing values may be of use. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also use a contingency table to compare two variables at once. Can display using a mosaic plot too.&lt;/p&gt;&#xA;" OwnerUserId="74" LastActivityDate="2010-07-19T20:34:46.820" />
  <row Id="105" PostTypeId="2" ParentId="103" CreationDate="2010-07-19T20:35:34.510" Score="12" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://flowingdata.com/&quot; rel=&quot;nofollow&quot;&gt;http://flowingdata.com/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="46" LastActivityDate="2010-07-19T20:35:34.510" CommunityOwnedDate="2010-07-19T20:35:34.510" CommentCount="1" />
  <row Id="107" PostTypeId="2" ParentId="103" CreationDate="2010-07-19T20:36:01.200" Score="8" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://infosthetics.com/&quot; rel=&quot;nofollow&quot;&gt;http://infosthetics.com/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="46" LastActivityDate="2010-07-19T20:36:01.200" CommunityOwnedDate="2010-07-19T20:36:01.200" />
  <row Id="108" PostTypeId="2" ParentId="103" CreationDate="2010-07-19T20:36:47.800" Score="9" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://www.informationisbeautiful.net/&quot; rel=&quot;nofollow&quot;&gt;http://www.informationisbeautiful.net/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-07-19T20:36:47.800" CommunityOwnedDate="2010-07-19T20:36:47.800" CommentCount="1" />
  <row Id="109" PostTypeId="1" CreationDate="2010-07-19T20:37:21.667" Score="1" ViewCount="689" Body="&lt;p&gt;Following one-way ANOVA, there are many possible follow-up multiple comparison tests. Holm's test (or better, the Holm-Sidak) test has lots of power, but because it works in a stepwise manner, it cannot compute confidence intervals. Its advantage over the tests than can compute confidence intervals (Tukey, Dunnett) is that is has more power. But is it fair to say that the Holm method &lt;em&gt;always&lt;/em&gt; has more power than the methods of Tukey and Dunnet? Or does it depend...?&lt;/p&gt;&#xA;" OwnerUserId="25" LastActivityDate="2010-08-04T14:38:22.450" ClosedDate="2010-08-05T13:06:12.850" Title="Power of Holm's multiple comparison testing compared to others." Tags="&lt;multiple-comparisons&gt;&lt;power&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="110" PostTypeId="2" ParentId="100" CreationDate="2010-07-19T20:40:36.573" Score="3" ViewCount="" Body="&lt;p&gt;It depends on where you apply the window function.  If you do it in the time domain, it's because you only want to analyze the periodic behavior of the function in a short duration.  You do this when you don't believe that your data is from a stationary process.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you do it in the frequency domain, then you do it to isolate a specific set of frequencies for further analysis; you do this when you believe that (for instance) high-frequency components are spurious.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first three chapters of &quot;A Wavelet Tour of Signal Processing&quot; by Stephane Mallat have an excellent introduction to signal processing in general, and chapter 4 goes into a very good discussion of windowing and time-frequency representations in both continuous and discrete time, along with a few worked-out examples.&lt;/p&gt;&#xA;" OwnerUserId="61" LastActivityDate="2010-07-19T20:40:36.573" CommentCount="2" />
  <row Id="111" PostTypeId="2" ParentId="40" CreationDate="2010-07-19T20:41:24.917" Score="7" ViewCount="" Body="&lt;p&gt;In R, the default setting for random number generation are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;For U(0,1), use the Mersenne-Twister algorithm&lt;/li&gt;&#xA;&lt;li&gt;For Guassian numbers use  the numerical inversion of the standard normal distribution function. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;You can easily check this, viz.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; RNGkind()&#xA;[1] &quot;Mersenne-Twister&quot; &quot;Inversion&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It is possible to change the default generator to other PRNGs, such as Super-Duper,Wichmann-Hill, Marsaglia-Multicarry or even a user-supplied PRNG. See the ?RNGkind for further details. I have never needed to change the default PRNG.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;a href=&quot;http://www.gnu.org/software/gsl/manual/html_node/Random-number-environment-variables.html&quot; rel=&quot;nofollow&quot;&gt;C GSL&lt;/a&gt; library also uses the &lt;a href=&quot;http://www.gnu.org/software/gsl/manual/html_node/Random-number-generator-algorithms.html&quot; rel=&quot;nofollow&quot;&gt;Mersenne-Twister&lt;/a&gt; by default.&lt;/p&gt;&#xA;" OwnerUserId="8" LastActivityDate="2010-07-19T20:41:24.917" CommentCount="3" />
  <row Id="112" PostTypeId="2" ParentId="103" CreationDate="2010-07-19T20:52:27.263" Score="1" ViewCount="" Body="&lt;p&gt;I see all my favorite blogs have been listed. So I'll give you this one:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://ilovecharts.tumblr.com/&quot; rel=&quot;nofollow&quot;&gt;http://ilovecharts.tumblr.com/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;it's a bit light hearted.&lt;/p&gt;&#xA;" OwnerUserId="12" LastActivityDate="2010-07-19T20:52:27.263" CommunityOwnedDate="2010-07-19T20:52:27.263" />
  <row Id="113" PostTypeId="1" CreationDate="2010-07-19T20:54:23.200" Score="6" ViewCount="302" Body="&lt;p&gt;I have been looking into theoretical frameworks for method selection (note: not model selection) and have found very little systematic, mathematically-motivated work. By 'method selection', I mean a framework for distinguishing the appropriate (or better, optimal) method with respect to a problem, or problem type.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I have found is substantial, if piecemeal, work on particular methods and their tuning (i.e. prior selection in Bayesian methods), and method selection via bias selection (e.g. &lt;a href=&quot;http://portal.acm.org/citation.cfm?id=218546&quot; rel=&quot;nofollow&quot;&gt;Inductive Policy: The Pragmatics of Bias Selection&lt;/a&gt;). I may be unrealistic at this early stage of machine learning's development, but I was hoping to find something like what &lt;a href=&quot;ftp://ftp.sas.com/pub/neural/measurement.html&quot; rel=&quot;nofollow&quot;&gt;measurement theory&lt;/a&gt; does in prescribing admissible transformations and tests by scale type, only writ large in the arena of learning problems.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestions?&lt;/p&gt;&#xA;" OwnerUserId="39" LastEditorUserId="39" LastEditorDisplayName="" LastEditDate="2010-07-21T15:44:07.450" LastActivityDate="2010-10-08T23:57:02.170" Title="What are some good frameworks for method selection?" Tags="&lt;machine-learning&gt;&lt;methodology&gt;&lt;theory&gt;" AnswerCount="1" CommentCount="5" FavoriteCount="4" />
  <row Id="114" PostTypeId="1" CreationDate="2010-07-19T21:00:53.077" Score="14" ViewCount="394" Body="&lt;p&gt;What research level statistical blogs are there?&lt;/p&gt;&#xA;" OwnerUserId="8" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-11-23T09:10:44.523" LastActivityDate="2010-11-23T09:10:44.523" CommunityOwnedDate="2010-07-19T21:18:51.293" Title="What statistical blogs would you recommend?" Tags="&lt;blog&gt;" AnswerCount="7" CommentCount="1" FavoriteCount="8" />
  <row Id="115" PostTypeId="2" ParentId="103" CreationDate="2010-07-19T21:01:35.757" Score="1" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://datavis.tumblr.com/&quot; rel=&quot;nofollow&quot;&gt;http://datavis.tumblr.com/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="127" LastActivityDate="2010-07-19T21:01:35.757" CommunityOwnedDate="2010-07-19T21:01:35.757" />
  <row Id="116" PostTypeId="2" ParentId="114" CreationDate="2010-07-19T21:04:16.080" Score="5" ViewCount="" Body="&lt;p&gt;Cosma Shalizi's &lt;a href=&quot;http://www.cscs.umich.edu/~crshalizi/weblog/&quot; rel=&quot;nofollow&quot;&gt;blog&lt;/a&gt;, often talks about statistics, and is always interesting.&lt;/p&gt;&#xA;" OwnerUserId="72" LastActivityDate="2010-07-19T21:04:16.080" CommentCount="4" />
  <row Id="117" PostTypeId="2" ParentId="114" CreationDate="2010-07-19T21:04:24.283" Score="12" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://www.r-bloggers.com/&quot; rel=&quot;nofollow&quot;&gt;http://www.r-bloggers.com/&lt;/a&gt; is an aggregated blog from lots of blogs that talk about statistics using R, and the &lt;a href=&quot;http://search.twitter.com/search?q=%23rstats&quot; rel=&quot;nofollow&quot;&gt;#rstats&lt;/a&gt; hashtag on twitter is also helpful. I write quite a bit about &lt;a href=&quot;http://gettinggeneticsdone.blogspot.com/search/label/R&quot; rel=&quot;nofollow&quot;&gt;statistics and R in genetics research&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="36" LastEditorUserId="36" LastEditorDisplayName="" LastEditDate="2010-07-20T15:13:37.983" LastActivityDate="2010-07-20T15:13:37.983" CommentCount="0" />
  <row Id="118" PostTypeId="1" AcceptedAnswerId="151" CreationDate="2010-07-19T21:04:39.057" Score="36" ViewCount="2417" Body="&lt;p&gt;In the definition of standard deviation, why do we have to &lt;strong&gt;square&lt;/strong&gt; the difference from the mean to get the mean (E) and take the &lt;strong&gt;square root back&lt;/strong&gt; at the end? Can't we just simply take &lt;strong&gt;the absolute value&lt;/strong&gt; of the difference instead and get the expected value (mean) of those, and wouldn't that also show the variation of the data? The number is going to be different from square method (the absolute-value method will be smaller), but it should still show the spread of data. Anybody know why we take this square approach as a standard?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The definition of standard deviation:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\sigma = \sqrt{E\left[\left(X - \mu\right)^2\right]}.$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can't we just take the absolute value instead and still be a good measurement?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\sigma = E\left[|X - \mu|\right]$&#xD;&lt;/p&gt;&#xA;" OwnerUserId="83" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-11-24T21:37:59.540" LastActivityDate="2011-01-05T03:41:08.693" Title="Standard deviation: Why square the difference instead of taking the absolute value?" Tags="&lt;standard-deviation&gt;&lt;definition&gt;" AnswerCount="14" CommentCount="5" FavoriteCount="10" />
  <row Id="119" PostTypeId="2" ParentId="118" CreationDate="2010-07-19T21:11:44.797" Score="2" ViewCount="" Body="&lt;p&gt;There are many reasons; probably the main is that it is a natural parameter of normal distribution.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-07-19T21:11:44.797" CommentCount="1" />
  <row Id="120" PostTypeId="2" ParentId="118" CreationDate="2010-07-19T21:14:07.983" Score="13" ViewCount="" Body="&lt;p&gt;One way you can think of this is that standard deviation is similar to a &quot;distance from the mean&quot;.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Compare this to distances in euclidean space - this gives you the true distance, where what you suggested (which, btw, is the &lt;a href=&quot;http://en.wikipedia.org/wiki/Average_absolute_deviation&quot; rel=&quot;nofollow&quot;&gt;absolute deviation&lt;/a&gt;) is more like a &lt;a href=&quot;http://en.wikipedia.org/wiki/Manhattan_distance_transform&quot; rel=&quot;nofollow&quot;&gt;manhattan distance&lt;/a&gt; calculation.&lt;/p&gt;&#xA;" OwnerUserId="41" LastActivityDate="2010-07-19T21:14:07.983" CommentCount="1" />
  <row Id="121" PostTypeId="2" ParentId="118" CreationDate="2010-07-19T21:14:25.407" Score="29" ViewCount="" Body="&lt;p&gt;The squared difference has nicer mathematical properties; it's continuously differentiable (nice when you want to minimize it), it's a sufficient statistic for the Gaussian distribution, and it's (a version of) the L2 norm which comes in handy for proving convergence and so on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The mean absolute deviation (the absolute value notation you suggest) is also used as a measure of dispersion, but it's not as &quot;well-behaved&quot; as the squared error.&lt;/p&gt;&#xA;" OwnerUserId="61" LastActivityDate="2010-07-19T21:14:25.407" CommentCount="5" />
  <row Id="123" PostTypeId="2" ParentId="118" CreationDate="2010-07-19T21:15:20.917" Score="6" ViewCount="" Body="&lt;p&gt;Squaring the difference from the mean has a couple of reasons.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Variance is defined as the 2nd moment of the deviation (the R.V here is (x-$\mu$) ) and thus the square as moments are simply the expectations of higher powers of the random variable.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Having a square as opposed to the absolute value function gives a nice continuous and differentiable function (absolute value is not differentiable at 0) - which makes it the natural choice, especially in the context of estimation and regression analysis.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The squared formulation also naturally falls out of parameters of the Normal Distribution. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="130" LastActivityDate="2010-07-19T21:15:20.917" />
  <row Id="124" PostTypeId="1" CreationDate="2010-07-19T21:17:30.543" Score="12" ViewCount="311" Body="&lt;p&gt;I'm a programmer without statistical background, and I'm currently looking at different classification methods for a large number of different documents that I want to classify into pre-defined categories. I've been reading about kNN, SVM and NN. However, I have some trouble getting started. What resources do you recommend? I do know single variable and multi variable calculus quite well, so my math should be strong enough. I also own Bishop's book on Neural Networks, but it has proven to be a bit dense as an introduction.&lt;/p&gt;&#xA;" OwnerUserId="131" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-07-21T22:17:00.927" LastActivityDate="2010-11-10T20:50:22.887" Title="Statistical classification of text" Tags="&lt;classification&gt;&lt;information-retrieval&gt;&lt;text-mining&gt;" AnswerCount="6" FavoriteCount="5" />
  <row Id="125" PostTypeId="1" CreationDate="2010-07-19T21:18:12.713" Score="14" ViewCount="1441" Body="&lt;p&gt;Which is the best introductory textbook for Bayesian statistics?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One book per answer, please.&lt;/p&gt;&#xA;" OwnerUserId="5" LastEditorUserId="509" LastEditorDisplayName="" LastEditDate="2010-10-06T10:58:25.327" LastActivityDate="2011-03-13T00:58:39.420" CommunityOwnedDate="2010-07-19T21:18:12.713" Title="What is the best introductory Bayesian statistics textbook?" Tags="&lt;bayesian&gt;&lt;books&gt;&lt;best-of&gt;" AnswerCount="17" FavoriteCount="14" />
  <row Id="126" PostTypeId="2" ParentId="125" CreationDate="2010-07-19T21:19:43.570" Score="16" ViewCount="" Body="&lt;p&gt;My favorite is &lt;a href=&quot;http://www.amazon.com/exec/obidos/ISBN=158488388X/&quot; rel=&quot;nofollow&quot;&gt;&quot;Bayesian Data Analysis&quot;&lt;/a&gt; by Gelman, et al.&lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-07-19T21:19:43.570" CommunityOwnedDate="2010-07-19T21:19:43.570" CommentCount="1" />
  <row Id="127" PostTypeId="2" ParentId="125" CreationDate="2010-07-19T21:23:20.593" Score="7" ViewCount="" Body="&lt;p&gt;Another vote for Gelman et al., but a close second for me -- being of the learn-by-doing persuasion -- is &lt;a href=&quot;http://bayes.bgsu.edu/bcwr/&quot; rel=&quot;nofollow&quot;&gt;Bayesian Computation with R&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="61" LastActivityDate="2010-07-19T21:23:20.593" CommunityOwnedDate="2010-07-19T21:23:20.593" CommentCount="2" />
  <row Id="128" PostTypeId="1" AcceptedAnswerId="191" CreationDate="2010-07-19T21:23:57.973" Score="1" ViewCount="1142" Body="&lt;p&gt;In Plain English, how does one interpret a Bland-Altman plot?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are the advantages of using a Bland-Altman plot over other methods of comparing two different measurement methods?&lt;/p&gt;&#xA;" OwnerUserId="132" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-13T08:21:54.850" LastActivityDate="2011-04-13T08:21:54.850" Title="How does one interpret a Bland-Altman plot?" Tags="&lt;data-visualization&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="129" PostTypeId="2" ParentId="125" CreationDate="2010-07-19T21:24:58.567" Score="3" ViewCount="" Body="&lt;p&gt;I quite like &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1584885874&quot; rel=&quot;nofollow&quot;&gt;Markov Chain Monte Carlo: Stochastic Simulation for Bayesian Inference&lt;/a&gt; by Gamerman and Lopes.&lt;/p&gt;&#xA;" OwnerUserId="8" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-10-05T13:56:15.550" LastActivityDate="2010-10-05T13:56:15.550" CommunityOwnedDate="2010-07-19T21:24:58.567" />
  <row Id="130" PostTypeId="1" AcceptedAnswerId="131" CreationDate="2010-07-19T21:26:27.023" Score="12" ViewCount="989" Body="&lt;p&gt;I had a plan of learning R in the near future. Reading &lt;a href=&quot;http://stats.stackexchange.com/questions/3/what-are-some-valuable-statistical-analysis-open-source-projects&quot; rel=&quot;nofollow&quot;&gt;another question&lt;/a&gt; I found out about Clojure. Now I don't know what to do.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think a big &lt;strong&gt;advantage of R&lt;/strong&gt; for me is that some people in Economics use it, including one of my supervisors (though the other said: stay away from R!). One &lt;strong&gt;advantage of Clojure&lt;/strong&gt; is that it is Lisp-based, and as I have started learning Emacs and I am keen on writing my own customisations, it would be helpful (yeah, I know Clojure and Elisp are different dialects of Lisp, but they are both Lisp and thus similar I would imagine).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can't ask which one is better, because I know this is very personal, but could someone give me the advantages (or advantages) of Clojure x R, especially in practical terms? For example, which one should be easier to learn, which one is more flexible or more powerful, which one has more libraries, more support, more users, etc?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My intended use&lt;/strong&gt;: The bulk of my estimation should be done using Matlab, so I am not looking for anything too deep in terms of statistical analysis, but rather a software to substitute Excel for the initial data manipulation and visualisation, summary statistics and charting, but also some basic statistical analysis or the initial attempts at my estimation.&lt;/p&gt;&#xA;" OwnerUserId="90" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-08-07T18:00:44.807" LastActivityDate="2010-08-07T18:00:44.807" Title="Clojure versus R: advantages and disadvantages for data analysis" Tags="&lt;r&gt;" AnswerCount="4" CommentCount="10" FavoriteCount="5" />
  <row Id="131" PostTypeId="2" ParentId="130" CreationDate="2010-07-19T21:28:41.907" Score="10" ViewCount="" Body="&lt;p&gt;Let me start by saying that I love both languages: you can't go wrong with either, and they are certainly better than something like C++ or Java for doing data analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For basic data analysis I would suggest R (especially with plyr).  IMO, R is a little easier to learn than Clojure, although this isn't completely obvious since Clojure is based on Lisp and there are numerous fantastic Lisp resources available (such as &lt;a href=&quot;http://mitpress.mit.edu/sicp/&quot; rel=&quot;nofollow&quot;&gt;SICP&lt;/a&gt;).  There are less keywords in Clojure, but the libraries are much more difficult to install and work with.  Also, keep in mind that R (or S) is largely derived from Scheme, so you would benefit from Lisp knowledge when using it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The main advantage of R is the community on CRAN (over 2461  packages and counting).  Nothing will compare with this in the near future, not even a commercial application like matlab.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Clojure has the big advantage of running on the JVM which means that it can use any Java based library immediately.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would add that I gave &lt;a href=&quot;http://bit.ly/dhDZkp&quot; rel=&quot;nofollow&quot;&gt;a talk relating Clojure/Incanter to R&lt;/a&gt; a while ago, so you may find it of interest.  In my experience around creating this, Clojure was generally slower than R for simple operations.&lt;/p&gt;&#xA;" OwnerUserId="5" LastEditorUserId="5" LastEditorDisplayName="" LastEditDate="2010-07-26T14:06:09.937" LastActivityDate="2010-07-26T14:06:09.937" />
  <row Id="132" PostTypeId="2" ParentId="125" CreationDate="2010-07-19T21:29:37.040" Score="4" ViewCount="" Body="&lt;p&gt;Coming from non-statistical background I found &lt;a href=&quot;http://www.princeton.edu/~slynch/bayesbook/bookinfo.html&quot; rel=&quot;nofollow&quot;&gt;Introduction to Applied Bayesian Statistics and Estimation for Social Scientists&lt;/a&gt; quite informative and easy to follow.&lt;/p&gt;&#xA;" OwnerUserId="22" LastActivityDate="2010-07-19T21:29:37.040" CommunityOwnedDate="2010-07-19T21:29:37.040" />
  <row Id="133" PostTypeId="2" ParentId="4" CreationDate="2010-07-19T21:31:53.813" Score="0" ViewCount="" Body="&lt;p&gt;I don't know how to use SAS/R/Orange, but it sounds like the kind of test you need is a &lt;a href=&quot;http://en.wikipedia.org/wiki/Chi-square_test&quot; rel=&quot;nofollow&quot;&gt;chi-square test&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="139" LastActivityDate="2010-07-19T21:31:53.813" CommentCount="2" />
  <row Id="134" PostTypeId="1" CreationDate="2010-07-19T21:32:38.523" Score="8" ViewCount="733" Body="&lt;p&gt;On smaller window sizes, &lt;code&gt;n log n&lt;/code&gt; sorting might work. Are there any better algorithms to achieve this?&lt;/p&gt;&#xA;" OwnerUserId="138" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-08-03T12:14:50.543" LastActivityDate="2010-11-27T21:45:52.897" Title="Algorithms to compute the running median?" Tags="&lt;algorithms&gt;&lt;median&gt;" AnswerCount="5" CommentCount="6" FavoriteCount="5" />
  <row Id="135" PostTypeId="2" ParentId="4" CreationDate="2010-07-19T21:36:12.850" Score="5" ViewCount="" Body="&lt;p&gt;I believe that this calls for a &lt;a href=&quot;http://www.itl.nist.gov/div898/software/dataplot/refman1/auxillar/ks2samp.htm&quot; rel=&quot;nofollow&quot;&gt;two-sample Kolmogorov–Smirnov test&lt;/a&gt;, or the like.  The two-sample Kolmogorov–Smirnov test is based on comparing differences in the &lt;a href=&quot;http://en.wikipedia.org/wiki/Empirical_distribution_function&quot; rel=&quot;nofollow&quot;&gt;empirical distribution functions&lt;/a&gt; (ECDF) of two samples, meaning it is sensitive to both location and shape of the the two samples.  It also generalizes out to a multivariate form.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This test is found in various forms in different packages in R, so if you are basically proficient, all you have to do is install one of them (e.g. &lt;a href=&quot;http://cran.r-project.org/web/packages/fBasics/fBasics.pdf&quot; rel=&quot;nofollow&quot;&gt;fBasics&lt;/a&gt;), and run it on your sample data.&lt;/p&gt;&#xA;" OwnerUserId="39" LastEditorUserId="39" LastEditorDisplayName="" LastEditDate="2010-07-19T21:52:08.617" LastActivityDate="2010-07-19T21:52:08.617" CommentCount="3" />
  <row Id="137" PostTypeId="2" ParentId="124" CreationDate="2010-07-19T21:38:09.370" Score="9" ViewCount="" Body="&lt;p&gt;Do you have a college, university, or city library card? If so you may have free access to Books24x7.com and SpringerLink.com which are e-book repositories. If so, I recommend these books - they are highly rated on Amazon too:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Text Mining&quot; by Weiss&#xA;&lt;a href=&quot;http://www.springerlink.com/content/k46654/?p=b318fa02471245298ce4a9fba9104dd5&amp;amp;pi=0&quot; rel=&quot;nofollow&quot;&gt;http://www.springerlink.com/content/k46654/?p=b318fa02471245298ce4a9fba9104dd5&amp;amp;pi=0&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Text Mining Application Programming&quot;, by Konchady&#xA;&lt;a href=&quot;http://library.books24x7.com/toc.asp?bookid=26322&quot; rel=&quot;nofollow&quot;&gt;http://library.books24x7.com/toc.asp?bookid=26322&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For software, I recommend RapidMiner (with the text plugin), free and open-source. There is also GATE for advanced stuff. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is my &quot;text mining process&quot;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;* collect the documents (usually a web crawl)&#xA;      o [sample if too large]&#xA;      o timestamp&#xA;      o strip out markup&#xA;* tokenize&#xA;* break into characters, words, n-grams, or sliding windows&#xA;* stemming (aka lemmatization)&#xA;      o [include synonyms]&#xA;      o see porter or snowflake algorithm&#xA;      o pronouns and articles are usually bad predictors&#xA;* remove stopwords&#xA;* feature vectorization&#xA;      o binary (appears or doesn’t)&#xA;      o word count&#xA;      o relative frequency: tf-idf&#xA;      o information gain, chi square&#xA;      o [have a minimum value for inclusion]&#xA;* weighting&#xA;      o weight words at top of document higher?&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then you can start the work of classifying them. kNN, SVM, or Naive Bayes as appropriate. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can see my series of text mining videos &lt;a href=&quot;http://vancouverdata.blogspot.com/2010/11/text-analytics-with-rapidminer-loading.html&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="74" LastEditorUserId="74" LastEditorDisplayName="" LastEditDate="2010-11-10T20:50:22.887" LastActivityDate="2010-11-10T20:50:22.887" CommentCount="2" />
  <row Id="138" PostTypeId="1" CreationDate="2010-07-19T21:38:10.290" Score="18" ViewCount="1181" Body="&lt;p&gt;I'm looking to learn &lt;a href=&quot;http://en.wikipedia.org/wiki/R_%28programming_language%29&quot; rel=&quot;nofollow&quot;&gt;R&lt;/a&gt; on the cheap. What's the best free resource/book/tutorial for learning R?&lt;/p&gt;&#xA;" OwnerUserId="142" LastEditorUserId="509" LastEditorDisplayName="" LastEditDate="2010-08-08T12:46:01.343" LastActivityDate="2011-05-27T03:38:47.843" CommunityOwnedDate="2010-08-01T18:56:25.603" Title="Resources for learning R" Tags="&lt;r&gt;" AnswerCount="17" CommentCount="5" FavoriteCount="20" />
  <row Id="139" PostTypeId="2" ParentId="138" CreationDate="2010-07-19T21:39:17.220" Score="10" ViewCount="" Body="&lt;p&gt;If I had to choose one thing, make sure that you read &lt;a href=&quot;http://www.burns-stat.com/pages/Tutor/R_inferno.pdf&quot; rel=&quot;nofollow&quot;&gt;&quot;The R Inferno&quot;&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are many good resources on &lt;a href=&quot;http://www.r-project.org&quot; rel=&quot;nofollow&quot;&gt;the R homepage&lt;/a&gt;, but in particular, read &lt;a href=&quot;http://cran.r-project.org/doc/manuals/R-intro.pdf&quot; rel=&quot;nofollow&quot;&gt;&quot;An Introduction to R&quot;&lt;/a&gt; and &lt;a href=&quot;http://cran.r-project.org/doc/manuals/R-lang.pdf&quot; rel=&quot;nofollow&quot;&gt;&quot;The R Language Definition&quot;&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-07-19T21:39:17.220" CommunityOwnedDate="2010-08-01T18:56:25.603" />
  <row Id="140" PostTypeId="2" ParentId="138" CreationDate="2010-07-19T21:39:35.690" Score="4" ViewCount="" Body="&lt;p&gt;The official guides are pretty nice; check out &lt;a href=&quot;http://cran.r-project.org/manuals.html&quot; rel=&quot;nofollow&quot;&gt;http://cran.r-project.org/manuals.html&lt;/a&gt; . There is also a lot of contributed documentation there.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-07-19T21:39:35.690" CommunityOwnedDate="2010-08-01T18:56:25.603" />
  <row Id="141" PostTypeId="2" ParentId="103" CreationDate="2010-07-19T21:40:02.540" Score="2" ViewCount="" Body="&lt;p&gt;Light-hearted: &lt;a href=&quot;http://indexed.blogspot.com/&quot; rel=&quot;nofollow&quot;&gt;http://indexed.blogspot.com/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="142" LastActivityDate="2010-07-19T21:40:02.540" CommunityOwnedDate="2010-07-19T21:40:02.540" />
  <row Id="142" PostTypeId="2" ParentId="138" CreationDate="2010-07-19T21:42:57.987" Score="2" ViewCount="" Body="&lt;p&gt;After you learn the basics, I find the following sites very useful:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.r-bloggers.com/&quot; rel=&quot;nofollow&quot;&gt;R-bloggers&lt;/a&gt;. &lt;/li&gt;&#xA;&lt;li&gt;Subscribing to the &lt;a href=&quot;http://stackoverflow.com/questions/tagged/R&quot;&gt;Stack overflow R tag&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="8" LastActivityDate="2010-07-19T21:42:57.987" CommunityOwnedDate="2010-08-01T18:56:25.603" />
  <row Id="143" PostTypeId="2" ParentId="124" CreationDate="2010-07-19T21:48:28.567" Score="4" ViewCount="" Body="&lt;p&gt;Neural network may be to slow for a large number of documents (also this is now pretty much obsolete).&lt;br&gt;&#xA;And you may also check Random Forest among classifiers; it is quite fast, scales nice and does not need complex tuning.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-07-19T21:48:28.567" CommentCount="2" />
  <row Id="144" PostTypeId="2" ParentId="138" CreationDate="2010-07-19T21:48:52.670" Score="11" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://www.statmethods.net/index.html&quot; rel=&quot;nofollow&quot;&gt;Quick-R&lt;/a&gt; can be a good place to start.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A little bit data mining oriented &lt;a href=&quot;http://www.rdatamining.com&quot; rel=&quot;nofollow&quot;&gt;R and Data Mining&lt;/a&gt; resources: &lt;a href=&quot;http://www.rdatamining.com/docs/Rdatamining.pdf&quot; rel=&quot;nofollow&quot;&gt;Examples and Case Studies&lt;/a&gt; and &lt;a href=&quot;http://www.rdatamining.com/docs/R-refcard-data-mining.pdf&quot; rel=&quot;nofollow&quot;&gt;R Reference Card for Data Mining&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="22" LastEditorUserId="22" LastEditorDisplayName="" LastEditDate="2011-04-01T15:12:01.257" LastActivityDate="2011-04-01T15:12:01.257" CommunityOwnedDate="2010-08-01T18:56:25.603" />
  <row Id="145" PostTypeId="1" AcceptedAnswerId="147" CreationDate="2010-07-19T21:50:16.260" Score="2" ViewCount="356" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;Possible Duplicate:&lt;/strong&gt;&lt;br&gt;&#xA;  &lt;a href=&quot;http://stats.stackexchange.com/questions/7/locating-freely-available-data-samples&quot;&gt;Locating freely available data samples&lt;/a&gt;  &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&#xA;&#xA;&lt;p&gt;Where can I find freely accessible data sources?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm thinking of sites like&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www2.census.gov/census_2000/datasets/&quot; rel=&quot;nofollow&quot;&gt;http://www2.census.gov/census_2000/datasets/&lt;/a&gt;?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="138" LastEditorUserId="442" LastEditorDisplayName="" LastEditDate="2010-08-30T15:02:00.623" LastActivityDate="2010-08-30T15:02:00.623" CommunityOwnedDate="2010-07-19T21:50:44.433" ClosedDate="2010-07-20T00:21:48.007" Title="Free Dataset Resources?" Tags="&lt;dataset&gt;" AnswerCount="5" CommentCount="4" FavoriteCount="1" />
  <row Id="146" PostTypeId="1" AcceptedAnswerId="149" CreationDate="2010-07-19T21:52:51.707" Score="8" ViewCount="822" Body="&lt;p&gt;A while ago a user on R-help mailing list asked about the soundness of using PCA scores in a regression. The user is trying to use one PC scores to explain variation in another PC (see full discussion &lt;a href=&quot;http://r.789695.n4.nabble.com/PCA-and-Regression-td2280038.html&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;). The answer was that no, this is not sound because PC are orthogonal to each other. Can someone explain in a bit more detail why this is so?&lt;/p&gt;&#xA;" OwnerUserId="144" LastActivityDate="2010-09-10T18:14:48.727" Title="PCA scores in multiple regression" Tags="&lt;r&gt;&lt;pca&gt;&lt;scores&gt;&lt;regression&gt;" AnswerCount="5" CommentCount="3" FavoriteCount="5" />
  <row Id="147" PostTypeId="2" ParentId="145" CreationDate="2010-07-19T21:53:02.283" Score="3" ViewCount="" Body="&lt;p&gt;Amazon has free Public Data sets for use with EC2. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://aws.amazon.com/publicdatasets/&quot; rel=&quot;nofollow&quot;&gt;http://aws.amazon.com/publicdatasets/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's a list: &lt;a href=&quot;http://developer.amazonwebservices.com/connect/kbcategory.jspa?categoryID=243&quot; rel=&quot;nofollow&quot;&gt;http://developer.amazonwebservices.com/connect/kbcategory.jspa?categoryID=243&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="142" LastActivityDate="2010-07-19T21:53:02.283" CommunityOwnedDate="2010-07-19T21:53:02.283" />
  <row Id="148" PostTypeId="2" ParentId="145" CreationDate="2010-07-19T21:58:51.867" Score="3" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://infochimps.org/&quot; rel=&quot;nofollow&quot;&gt;http://infochimps.org/&lt;/a&gt; - is a good resource for free data sets.&lt;/p&gt;&#xA;" OwnerUserId="130" LastActivityDate="2010-07-19T21:58:51.867" CommunityOwnedDate="2010-07-19T21:58:51.867" />
  <row Id="149" PostTypeId="2" ParentId="146" CreationDate="2010-07-19T22:02:10.340" Score="5" ViewCount="" Body="&lt;p&gt;A principal component is a weighted linear combination of all your factors (X's).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;example: PCA1 = 0.1X1 + 0.3X2&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There will be one principal component for each factor (though in general a small number are selected). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The principal components are created such that they have zero correlation (are orthogonal), by design.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, principal component PCA1 should not explain any variation in principal component PCA2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a book, I recommend Multivariate Data Analysis by Hair (6th edition)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is also good: &lt;a href=&quot;http://www.statsoft.com/textbook/principal-components-factor-analysis/&quot; rel=&quot;nofollow&quot;&gt;http://www.statsoft.com/textbook/principal-components-factor-analysis/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="74" LastEditorUserId="74" LastEditorDisplayName="" LastEditDate="2010-09-10T17:57:34.477" LastActivityDate="2010-09-10T17:57:34.477" />
  <row Id="150" PostTypeId="2" ParentId="125" CreationDate="2010-07-19T22:13:29.830" Score="3" ViewCount="" Body="&lt;p&gt;For complete beginners, try William Briggs &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0557019907&quot; rel=&quot;nofollow&quot;&gt;Breaking the Law of Averages: Real-Life Probability and Statistics in Plain English&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="25" LastActivityDate="2010-07-19T22:13:29.830" CommunityOwnedDate="2010-07-19T22:13:29.830" CommentCount="1" />
  <row Id="151" PostTypeId="2" ParentId="118" CreationDate="2010-07-19T22:31:12.830" Score="19" ViewCount="" Body="&lt;p&gt;If the goal of the standard deviation is to summarise the spread of a symmetrical data set (i.e. in general how far each datum is from the mean), then we need a good method of defining how to measure that spread.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The benefits of squaring include:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Squaring always gives a positive&#xA;value, so the sum will not be zero.&lt;/li&gt;&#xA;&lt;li&gt;Squaring emphasizes larger differences - a feature that turns out to be both good and bad (think of the effect outliers have).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Squaring however does have a problem as a measure of spread and that is that the units are all squared, where as we'd might prefer the spread to be in the same units as the original data (think of squared pounds or squared dollars or squared apples). Hence the square root allows us to return to the original units.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suppose you could say that absolute difference assigns equal weight to the spread of data where as squaring emphasises the extremes. Technically though, as others have pointed out, squaring makes the algebra much easier to work with and offers properties that the absolute method does not (for example, the variance is equal to the expected value of the square of the distribution minus the square of the mean of the distribution)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;&lt;strong&gt;It's important to note&lt;/em&gt;&lt;/strong&gt; &lt;em&gt;&lt;strong&gt;however&lt;/em&gt;&lt;/strong&gt; that there's no reason you couldn't take the absolute difference if that is your preference on how you wish to view 'spread' (sort of how some people see 5% as some magical thresh hold for p-values, when in fact it's situation dependent). Indeed, there are in fact several competing methods for measuring spread.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My view is to use the squared values because I like to think of how it relates to the Pythagorean Theorem of Statistics: c = sqrt(a^2 + b^2)  ...this also helps me remember that when working with independent random variables, variances add, standard deviations don't. But that's just my personal subjective preference.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An much more indepth analysis can be read &lt;a href=&quot;http://www.leeds.ac.uk/educol/documents/00003759.htm&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="81" LastEditorUserId="81" LastEditorDisplayName="" LastEditDate="2010-07-20T14:56:51.450" LastActivityDate="2010-07-20T14:56:51.450" CommentCount="3" />
  <row Id="152" PostTypeId="1" AcceptedAnswerId="1087" CreationDate="2010-07-19T22:37:38.013" Score="4" ViewCount="169" Body="&lt;p&gt;Label switching (i.e., the posterior distribution is invariant to switching component labels) is a problematic issue when using MCMC to estimate mixture models. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Is there a standard (as in widely accepted) methodology to deal with the issue?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If there is no standard approach then what are the pros and cons of the leading approaches to solve the label switching problem?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-27T16:03:35.180" LastActivityDate="2011-03-27T16:03:35.180" Title="Is there a standard method to deal with label switching problem in MCMC estimation of mixture models?" Tags="&lt;bayesian&gt;&lt;mcmc&gt;&lt;mixture&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="153" PostTypeId="2" ParentId="10" CreationDate="2010-07-19T22:39:27.230" Score="3" ViewCount="" Body="&lt;p&gt;The simple answer is that Likert scales are always ordinal. The intervals between positions on the scale are monotonic but never so well-defined as to be numerically uniform increments.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That said, the distinction between ordinal and interval is based on the specific demands of the analysis being performed. Under special circumstances, you may be able to treat the responses as if they fell on an interval scale. To do this, typically the respondents need to be in close agreement regarding the meaning of the scale responses and the analysis (or the decisions made based on the analysis) should be relatively insensitive to problems that may arise.&lt;/p&gt;&#xA;" OwnerUserId="145" LastActivityDate="2010-07-19T22:39:27.230" CommentCount="1" />
  <row Id="154" PostTypeId="2" ParentId="1" CreationDate="2010-07-19T22:40:47.947" Score="14" ViewCount="" Body="&lt;p&gt;I am currently researching the &lt;em&gt;trial roulette method&lt;/em&gt; for my masters thesis as an elicitation technique. This is a graphical method that allows an expert to represent her subjective probability distribution for an uncertain quantity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Experts are given counters (or what one can think of as casino chips) representing equal densities whose total would sum up to 1 - for example 20 chips of probability = 0.05 each. They are then instructed to arrange them on a pre-printed grid, with bins representing result intervals. Each column would represent their belief of the probability of getting the corresponding bin result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example: A student is asked to predict the mark in a future exam. The&#xA;figure below shows a completed grid for the elicitation of&#xA;a subjective probability distribution. The horizontal axis of the&#xA;grid shows the possible bins (or mark intervals) that the student was&#xA;asked to consider. The numbers in top row record the number of chips&#xA;per bin. The completed grid (using a total of 20 chips) shows that the&#xA;student believes there is a 30% chance that the mark will be between&#xA;60 and 64.9.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://img641.imageshack.us/img641/4716/chipsbinscrisp.png&quot; alt=&quot;Eliciting priors from experts - Trial Roulette Method&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some reasons in favour of using this technique are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Many questions about the shape of the expert's subjective probability distribution can be answered without the need to pose a long series of questions to the expert - the statistician can simply read off density above or below any given point, or that between any two points. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;During the elicitation process, the experts can move around the chips if unsatisfied with the way they placed them initially - thus they can be sure of the final result to be submitted. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;It forces the expert to be coherent in the set of probabilities that are provided. If all the chips are used, the probabilities must sum to one.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Graphical methods seem to provide more accurate results, especially for participants with modest levels of statistical sophistication.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="108" LastEditorUserId="108" LastEditorDisplayName="" LastEditDate="2010-09-03T17:46:44.017" LastActivityDate="2010-09-03T17:46:44.017" CommentCount="2" />
  <row Id="155" PostTypeId="1" CreationDate="2010-07-19T22:43:50.967" Score="13" ViewCount="557" Body="&lt;p&gt;I really enjoy hearing simple explanations to complex problems. What is your favorite analogy or anecdote that explains a difficult statistical concept?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My favorite is &lt;a href=&quot;http://www-stat.wharton.upenn.edu/~steele/Courses/434/434Context/Co-integration/Murray93DrunkAndDog.pdf&quot; rel=&quot;nofollow&quot;&gt;Murray's&lt;/a&gt; explanation of cointegration using a drunkard and her dog. Murray explains how two random processes (a wandering drunk and her dog, Oliver) can have unit roots but still be related (cointegrated) since their joint first differences are stationary.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The drunk sets out from the bar, about to wander aimlessly in random-walk fashion. But&#xA;  periodically she intones &quot;Oliver, where are you?&quot;, and Oliver interrupts his aimless&#xA;  wandering to bark. He hears her; she hears him. He thinks, &quot;Oh, I can't let her get too far&#xA;  off; she'll lock me out.&quot; She thinks, &quot;Oh, I can't let him get too far off; he'll wake&#xA;  me up in the middle of the night with his barking.&quot; Each assesses how far&#xA;  away the other is and moves to partially close that gap.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="154" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-08-09T10:36:33.590" LastActivityDate="2010-09-14T18:11:45.067" Title="What is your favorite layman's explanation for a difficult statistical concept?" Tags="&lt;teaching&gt;" AnswerCount="8" CommentCount="2" FavoriteCount="7" />
  <row Id="156" PostTypeId="1" AcceptedAnswerId="198" CreationDate="2010-07-19T22:50:13.297" Score="1" ViewCount="102" Body="&lt;p&gt;I know this must be standard material, but I had difficulty in finding a proof in this form.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let e be a standard white Gaussian vector of size N.  Let all the other matrices in the following be constant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let v = X*y + e, where X is an N x L matrix and y is an N x 1 vector, and let&lt;/p&gt;&#xA;&#xA;&lt;p&gt;y_bar = (X^T*X)^(-1)X^T*v&lt;/p&gt;&#xA;&#xA;&lt;p&gt;e_bar = v - X*y_bar. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If c is any constant vector, J = N - rank(X), and &lt;/p&gt;&#xA;&#xA;&lt;p&gt;u = c^T*y_bar&lt;/p&gt;&#xA;&#xA;&lt;p&gt;s^2 = e_bar^T*e_bar*c^T*(X^T*X)^(-1)*c,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;then the random variable defined as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;t = u/sqrt(s^2/J)  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;follows a normalized student t distribution with J degrees of freedom.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would be grateful if you could provide an outline for its proof. &lt;/p&gt;&#xA;" OwnerUserId="148" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-07-29T18:34:37.847" LastActivityDate="2010-07-29T18:34:37.847" Title="How to get to a t variable from linear regression" Tags="&lt;regression&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="157" PostTypeId="2" ParentId="155" CreationDate="2010-07-19T22:52:22.730" Score="5" ViewCount="" Body="&lt;p&gt;Definitely the Monty Hall Problem. &lt;a href=&quot;http://en.wikipedia.org/wiki/Monty_Hall_problem&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Monty_Hall_problem&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="36" LastActivityDate="2010-07-19T22:52:22.730" CommentCount="1" />
  <row Id="159" PostTypeId="2" ParentId="103" CreationDate="2010-07-19T23:00:30.737" Score="4" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://junkcharts.typepad.com/&quot; rel=&quot;nofollow&quot;&gt;Junk Charts&lt;/a&gt; is always interesting and thought-provoking, usually providing both criticism of visualizations in the popular media and suggestions for improvements.&lt;/p&gt;&#xA;" OwnerUserId="145" LastActivityDate="2010-07-19T23:00:30.737" CommunityOwnedDate="2010-07-19T23:00:30.737" />
  <row Id="160" PostTypeId="2" ParentId="103" CreationDate="2010-07-19T23:06:43.987" Score="1" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://dataspora.com/blog/&quot; rel=&quot;nofollow&quot;&gt;http://dataspora.com/blog/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="158" LastActivityDate="2010-07-19T23:06:43.987" CommunityOwnedDate="2010-07-19T23:06:43.987" />
  <row Id="161" PostTypeId="1" CreationDate="2010-07-19T23:11:36.240" Score="8" ViewCount="196" Body="&lt;p&gt;Econometricians often talk about a time series being &lt;em&gt;integrated with order k, I(k)&lt;/em&gt;. &lt;em&gt;k&lt;/em&gt; being the minimum number of differences required to obtain a stationary time series.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What methods or statistical tests can be used to determine, given a level of confidence, the &lt;em&gt;order of integration&lt;/em&gt; of a time series?&lt;/p&gt;&#xA;" OwnerUserId="154" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-07-19T23:39:49.573" LastActivityDate="2010-07-20T11:14:41.487" Title="What methods can be used to determine the Order of Integration of a time series?" Tags="&lt;time-series&gt;" AnswerCount="2" />
  <row Id="162" PostTypeId="2" ParentId="155" CreationDate="2010-07-19T23:13:32.150" Score="6" ViewCount="" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;If you carved your distribution (histogram) out&#xA;of wood, and tried to balance it on&#xA;your finger, the balance point would&#xA;be the mean.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If you put a stick in the middle of&#xA;your scatter plot, and attached the&#xA;stick to each data point with a&#xA;spring, the resting point of the&#xA;stick would be your regression line.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="74" LastEditorUserId="74" LastEditorDisplayName="" LastEditDate="2010-09-09T05:40:47.083" LastActivityDate="2010-09-09T05:40:47.083" CommentCount="4" />
  <row Id="163" PostTypeId="2" ParentId="50" CreationDate="2010-07-19T23:17:53.423" Score="3" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Definition:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A random variable is a measurable function from a probability space into a measurable space  known as the state space.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Example:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lets say I roll a fair six-sided die, with outcomes being one of the following: 1, 2, 3, 4, 5, or 6.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Whichever number the die lands on is the number of free text-books I will give you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this case, the &lt;em&gt;final amount&lt;/em&gt; of free text books that I give you is the &lt;strong&gt;random variable&lt;/strong&gt; because its value is based on the outcome (1, 2, 3, 4, 5, or 6 free text books) of a random event (rolling the die) and is not known before I roll the die.&lt;/p&gt;&#xA;" OwnerUserId="81" LastEditorUserId="81" LastEditorDisplayName="" LastEditDate="2010-07-20T14:37:39.310" LastActivityDate="2010-07-20T14:37:39.310" CommentCount="4" />
  <row Id="164" PostTypeId="2" ParentId="145" CreationDate="2010-07-19T23:19:44.963" Score="1" ViewCount="" Body="&lt;p&gt;For governmental data:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;US: &lt;a href=&quot;http://www.data.gov/&quot; rel=&quot;nofollow&quot;&gt;http://www.data.gov/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;World: &lt;a href=&quot;http://www.guardian.co.uk/world-government-data&quot; rel=&quot;nofollow&quot;&gt;http://www.guardian.co.uk/world-government-data&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="158" LastActivityDate="2010-07-19T23:19:44.963" CommunityOwnedDate="2010-07-19T23:19:44.963" />
  <row Id="165" PostTypeId="1" AcceptedAnswerId="207" CreationDate="2010-07-19T23:21:05.320" Score="16" ViewCount="828" Body="&lt;p&gt;Maybe the concept, why it's used, and an example. Thanks!&lt;/p&gt;&#xA;" OwnerUserId="74" LastEditorUserId="61" LastEditorDisplayName="" LastEditDate="2010-07-20T03:01:29.000" LastActivityDate="2010-08-05T09:15:34.467" CommunityOwnedDate="2010-08-16T06:37:30.433" Title="How would you explain Markov Chain Monte Carlo (MCMC) to a layperson?" Tags="&lt;bayesian&gt;&lt;mcmc&gt;&lt;teaching&gt;" AnswerCount="5" CommentCount="3" FavoriteCount="8" />
  <row Id="166" PostTypeId="1" CreationDate="2010-07-19T23:21:35.430" Score="8" ViewCount="684" Body="&lt;p&gt;Australia is currently having an election and understandably the media reports new political poll results daily. In a country of 22 million what percentage of the population would need to be sampled to get a statistically valid result?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possible that using too large a sample could affect the results, or does statistical validity monotonically increase with sample size?&lt;/p&gt;&#xA;" OwnerUserId="154" LastEditorUserId="442" LastEditorDisplayName="" LastEditDate="2010-09-17T13:20:58.950" LastActivityDate="2010-09-17T13:20:58.950" Title="How do you decide the sample size when polling a large population?" Tags="&lt;sample-size&gt;&lt;polling&gt;" AnswerCount="3" FavoriteCount="2" />
  <row Id="167" PostTypeId="2" ParentId="146" CreationDate="2010-07-19T23:26:31.473" Score="6" ViewCount="" Body="&lt;p&gt;Principal components are orthogonal by definition, so any pair of PCs will have zero correlation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, PCA can be used in regression if there are a large number of explanatory variables. These can be reduced to a small number of principal components and used as predictors in a regression.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-07-19T23:26:31.473" CommentCount="4" />
  <row Id="168" PostTypeId="1" AcceptedAnswerId="179" CreationDate="2010-07-19T23:26:44.747" Score="9" ViewCount="264" Body="&lt;p&gt;For univariate kernel density estimators (KDE), I use Silverman's rule for calculating $h$:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{equation}&#xA;0.9 \min(sd, IQR/1.34)\times n^{-0.2}&#xA;\end{equation}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are the standard rules for multivariate KDE (assuming a Normal kernel).&lt;/p&gt;&#xA;" OwnerUserId="8" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-11-05T13:06:26.657" LastActivityDate="2011-03-02T12:11:19.977" Title="Choosing a bandwidth for kernel density estimators" Tags="&lt;smoothing&gt;&lt;kde&gt;&lt;kernel&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="4" />
  <row Id="169" PostTypeId="2" ParentId="145" CreationDate="2010-07-19T23:27:36.400" Score="2" ViewCount="" Body="&lt;p&gt;For time series data, try the &lt;a href=&quot;http://robjhyndman.com/TSDL&quot; rel=&quot;nofollow&quot;&gt;Time Series Data Library&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-07-19T23:27:36.400" CommunityOwnedDate="2010-07-19T23:27:36.400" />
  <row Id="170" PostTypeId="1" AcceptedAnswerId="174" CreationDate="2010-07-19T23:29:54.663" Score="23" ViewCount="750" Body="&lt;p&gt;Are there any free statistical textbooks available? &lt;/p&gt;&#xA;" OwnerUserId="8" LastEditorUserId="69" LastEditorDisplayName="" LastEditDate="2010-10-27T09:23:26.720" LastActivityDate="2010-10-27T09:23:26.720" CommunityOwnedDate="2010-07-21T22:02:27.833" Title="Free statistical textbooks" Tags="&lt;teaching&gt;&lt;books&gt;" AnswerCount="14" CommentCount="2" FavoriteCount="20" />
  <row Id="171" PostTypeId="2" ParentId="161" CreationDate="2010-07-19T23:32:30.337" Score="7" ViewCount="" Body="&lt;p&gt;There are a number of statistical tests (known as &quot;unit root tests&quot;) for dealing with this problem. The most popular is probably the &quot;Augmented Dickey-Fuller&quot; (ADF) test, although the Phillips-Perron (PP) test and the KPSS test are also widely used. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Both the ADF and PP tests are based on a null hypothesis of a unit root (i.e., an I(1) series). The KPSS test is based on a null hypothesis of stationarity (i.e., an I(0) series). Consequently, the KPSS test can give quite different results from the ADF or PP tests.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-07-19T23:32:30.337" />
  <row Id="172" PostTypeId="2" ParentId="166" CreationDate="2010-07-19T23:34:18.163" Score="6" ViewCount="" Body="&lt;p&gt;Sample size doesn't much depend on the population size, which is counter-intuitive to many.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most polling companies use 400 or 1000 people in their samples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a reason for this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A sample size of 400 will give you a confidence interval of +/-5% 19 times out of 20 (95%)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A sample size of 1000 will give you a confidence interval of +/-3% 19 times out of 20 (95%)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you are measuring a proportion near 50% anyways. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This calculator isn't bad:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.raosoft.com/samplesize.html&quot; rel=&quot;nofollow&quot;&gt;http://www.raosoft.com/samplesize.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="74" LastEditorUserId="74" LastEditorDisplayName="" LastEditDate="2010-09-11T18:48:28.197" LastActivityDate="2010-09-11T18:48:28.197" />
  <row Id="173" PostTypeId="1" CreationDate="2010-07-19T23:37:22.980" Score="11" ViewCount="538" Body="&lt;p&gt;I recently started working for a tuberculosis clinic.  We meet periodically to discuss the number of TB cases we're currently treating, the number of tests administered, etc.  I'd like to start modeling these counts so that we're not just guessing whether something is unusual or not.  Unfortunately, I've had very little training in time series, and most of my exposure has been to models for very continuous data (stock prices) or very large numbers of counts (influenza).  But we deal with 0-18 cases per month (mean 6.68, median 7, var 12.3), which are distributed like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://img827.imageshack.us/img827/1927/activetbcases.png&quot; alt=&quot;alt text&quot; title=&quot;Cases by month&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://img827.imageshack.us/img827/4348/tbcasedistribution.png&quot; alt=&quot;alt text&quot; title=&quot;Distribution of counts&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've found a few articles that address models like this, but I'd greatly appreciate hearing suggestions from you - both for approaches and for R packages that I could use to implement those approaches.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT:&lt;/strong&gt;  mbq's answer has forced me to think more carefully about what I'm asking here; I got too hung-up on the monthly counts and lost the actual focus of the question.  What I'd like to know is: does the (fairly visible) decline from, say, 2008 onward reflect a downward trend in the overall number of cases?  It looks to me like the number of cases monthly from 2001-2007 reflects a stable process; maybe some seasonality, but overall stable.  From 2008 through the present, it looks like that process is changing: the overall number of cases is declining, even though the monthly counts might wobble up and down due to randomness and seasonality.  How can I test if there's a real change in the process?  And if I can identify a decline, how could I use that trend and whatever seasonality there might be to estimate the number of cases we might see in the upcoming months?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Whew.  Thanks for bearing with me.&lt;/p&gt;&#xA;" OwnerUserId="71" LastEditorUserId="71" LastEditorDisplayName="" LastEditDate="2010-07-21T00:25:04.360" LastActivityDate="2011-05-18T14:52:56.227" Title="Time series for count data, with counts &lt; 20" Tags="&lt;r&gt;&lt;time-series&gt;&lt;poisson&gt;&lt;count-data&gt;&lt;epidemiology&gt;" AnswerCount="8" FavoriteCount="3" />
  <row Id="174" PostTypeId="2" ParentId="170" CreationDate="2010-07-19T23:37:43.807" Score="14" ViewCount="" Body="&lt;p&gt;The most widely used and probably the best of what is available is&#xA;&lt;a href=&quot;http://www.statsoft.com/textbook/&quot; rel=&quot;nofollow&quot;&gt;http://www.statsoft.com/textbook/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other online stats books include&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://davidmlane.com/hyperstat/&quot; rel=&quot;nofollow&quot;&gt;http://davidmlane.com/hyperstat/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://faculty.vassar.edu/lowry/webtext.html&quot; rel=&quot;nofollow&quot;&gt;http://faculty.vassar.edu/lowry/webtext.html&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.psychstat.missouristate.edu/multibook2/mlt.htm&quot; rel=&quot;nofollow&quot;&gt;http://www.psychstat.missouristate.edu/multibook2/mlt.htm&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://bookboon.com/uk/student/statistics&quot; rel=&quot;nofollow&quot;&gt;http://bookboon.com/uk/student/statistics&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.freebookcentre.net/SpecialCat/Free-Statistics-Books-Download.html&quot; rel=&quot;nofollow&quot;&gt;http://www.freebookcentre.net/SpecialCat/Free-Statistics-Books-Download.html&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-07-19T23:37:43.807" />
  <row Id="175" PostTypeId="1" CreationDate="2010-07-19T23:39:49.730" Score="12" ViewCount="1521" Body="&lt;p&gt;Often times a statistical analyst is handed a set dataset and asked to fit a model using a technique such as linear regression.  Very frequently the dataset is accompanied with a disclaimer similar to &quot;Oh yeah, we messed up collecting some of these data points -- do what you can&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This situation leads to regression fits that are heavily impacted by the presence of outliers that may be erroneous data. Given the following:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;It is dangerous from both a scientific and moral standpoint to throw out data for no reason other than it &quot;makes the fit look bad&quot;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;In real life, the people who collected the data are frequently not available to answer questions such as &quot;when generating this data set, which of the points did you mess up, exactly?&quot;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;What statistical tests or rules of thumb can be used as a basis for excluding outliers in linear regression analysis?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any special considerations for multilinear regression?&lt;/p&gt;&#xA;" OwnerUserId="13" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-08-13T12:59:06.957" LastActivityDate="2010-10-04T21:29:50.943" Title="How should outliers be dealt with in linear regression analysis?" Tags="&lt;regression&gt;&lt;outliers&gt;" AnswerCount="8" CommentCount="1" FavoriteCount="7" />
  <row Id="176" PostTypeId="2" ParentId="22" CreationDate="2010-07-19T23:40:01.007" Score="10" ViewCount="" Body="&lt;p&gt;Let us say a man rolls a six sided die and it has outcomes 1, 2, 3, 4, 5, or 6. Furthermore, he says that if it lands on a 3, he'll give you a free text book.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then informally:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;strong&gt;&lt;em&gt;Frequentist&lt;/em&gt;&lt;/strong&gt; would say that each outcome has an equal 1 in 6 chance of occurring.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;strong&gt;&lt;em&gt;Bayesian&lt;/em&gt;&lt;/strong&gt; however would say hang on a second, I know that man, he's David Blaine, a famous trickster! I have a feeling he's up to something. I'm going to say that there's only a 1% chance of it landing on a 3  &lt;em&gt;BUT&lt;/em&gt;  I'll re-evaluate that beliefe and change it the more times he rolls the die. If I see the other numbers come up equally often, then I'll iteratively increase the chance from 1% to something slightly higher, otherwise I'll reduce it even further. &lt;/p&gt;&#xA;" OwnerUserId="81" LastEditorUserId="81" LastEditorDisplayName="" LastEditDate="2010-07-20T08:46:03.027" LastActivityDate="2010-07-20T08:46:03.027" />
  <row Id="177" PostTypeId="2" ParentId="175" CreationDate="2010-07-19T23:45:44.677" Score="15" ViewCount="" Body="&lt;p&gt;Rather than exclude outliers, you can use a robust method of regression. In R, for example, the &lt;a href=&quot;http://sekhon.berkeley.edu/library/MASS/html/rlm.html&quot; rel=&quot;nofollow&quot;&gt;&lt;code&gt;rlm()&lt;/code&gt; function from the MASS package&lt;/a&gt; can be used instead of the &lt;code&gt;lm()&lt;/code&gt; function. The method of estimation can be tuned to be more or less robust to outliers.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-07-19T23:45:44.677" />
  <row Id="178" PostTypeId="2" ParentId="3" CreationDate="2010-07-19T23:48:50.943" Score="7" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://rapid-i.com/&quot; rel=&quot;nofollow&quot;&gt;RapidMiner&lt;/a&gt; for data mining&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://gate.ac.uk/&quot; rel=&quot;nofollow&quot;&gt;GATE&lt;/a&gt; for text mining&lt;/p&gt;&#xA;" OwnerUserId="74" LastActivityDate="2010-07-19T23:48:50.943" CommunityOwnedDate="2010-07-19T23:48:50.943" />
  <row Id="179" PostTypeId="2" ParentId="168" CreationDate="2010-07-19T23:59:29.487" Score="8" ViewCount="" Body="&lt;p&gt;For a univariate KDE, you are better off using something other than Silverman's rule which is based on a normal approximation. One excellent approach is the Sheather-Jones method, easily implemented in R; for example,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plot(density(precip, bw=&quot;SJ&quot;))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The situation for multivariate KDE is not so well studied, and the tools are not so mature. Rather than a bandwidth, you need a bandwidth matrix. To simplify the problem, most people assume a diagonal matrix, although this may not lead to the best results. The &lt;a href=&quot;http://cran.r-project.org/web/packages/ks/&quot; rel=&quot;nofollow&quot;&gt;ks package in R&lt;/a&gt; provides some very useful tools including allowing a full (not necessarily diagonal) bandwidth matrix.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-07-19T23:59:29.487" />
  <row Id="180" PostTypeId="2" ParentId="145" CreationDate="2010-07-20T00:06:20.580" Score="2" ViewCount="" Body="&lt;p&gt;I really like the &lt;a href=&quot;http://research.stlouisfed.org/fred2/&quot; rel=&quot;nofollow&quot;&gt;FRED&lt;/a&gt;, from the St. Louis Fed (economics data). You can chart the series or more than one series, you can do some transformations to your data and chart it, and the NBER recessions are shaded.&lt;/p&gt;&#xA;" OwnerUserId="90" LastActivityDate="2010-07-20T00:06:20.580" CommunityOwnedDate="2010-07-20T00:06:20.580" />
  <row Id="181" PostTypeId="1" AcceptedAnswerId="1097" CreationDate="2010-07-20T00:15:02.920" Score="10" ViewCount="1023" Body="&lt;p&gt;Is there a standard and accepted method for selecting the number of layers, and the number of nodes in each layer, in a FF NN? I'm interested in automated ways of building neural networks.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-09-03T08:40:44.130" Title="How to choose the number of hidden layers and nodes in a feedforward neural network?" Tags="&lt;neural-networks&gt;" AnswerCount="3" FavoriteCount="4" />
  <row Id="182" PostTypeId="2" ParentId="175" CreationDate="2010-07-20T00:15:47.393" Score="7" ViewCount="" Body="&lt;p&gt;Sometimes outliers are bad data, and should be excluded, such as typos. Sometimes they are Wayne Gretzky or Michael Jordan, and should be kept. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Outlier detection methods include:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Univariate -&gt; boxplot. outside of 1.5 times inter-quartile range is an outlier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bivariate -&gt; scatterplot with confidence ellipse. outside of, say, 95% confidence ellipse is an outlier. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Multivariate -&gt; Mahalanobis D2 distance&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Mark those observations as outliers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Run a logistic regression (on Y=IsOutlier) to see if there are any systematic patterns. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Remove ones that you can demonstrate they are not representative of any sub-population. &lt;/p&gt;&#xA;" OwnerUserId="74" LastEditorUserId="74" LastEditorDisplayName="" LastEditDate="2010-09-09T00:10:56.520" LastActivityDate="2010-09-09T00:10:56.520" />
  <row Id="183" PostTypeId="1" AcceptedAnswerId="518" CreationDate="2010-07-20T00:20:51.767" Score="2" ViewCount="122" Body="&lt;p&gt;I need to analyze the 100k MovieLens dataset for clustering with two algorithms of my choice, between the likes of k-means, agnes, diana, dbscan, and several others. What tools (like Rattle, or Weka) would be best suited to help me make some simple clustering analysis over this dataset?&lt;/p&gt;&#xA;" OwnerUserId="166" LastActivityDate="2010-07-22T16:33:43.240" Title="What tools could be used for applying clustering algorithms on MovieLens?" Tags="&lt;clustering&gt;" AnswerCount="2" />
  <row Id="184" PostTypeId="2" ParentId="33" CreationDate="2010-07-20T00:21:58.193" Score="3" ViewCount="" Body="&lt;p&gt;Try using the &lt;code&gt;stl()&lt;/code&gt; function for time series decomposition. It provides a very flexible method for extracting a seasonal component from a time series.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-07-20T00:21:58.193" />
  <row Id="185" PostTypeId="2" ParentId="124" CreationDate="2010-07-20T00:30:00.173" Score="7" ViewCount="" Body="&lt;p&gt;A great introductory text covering the topics you mentioned is &lt;a href=&quot;http://www.informationretrieval.org&quot; rel=&quot;nofollow&quot;&gt;Introduction to Information Retrieval&lt;/a&gt;, which is available online in full text for free.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://nlp.stanford.edu/IR-book/iir.jpg&quot; alt=&quot;Introduction to Information Retrieval&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="80" LastActivityDate="2010-07-20T00:30:00.173" CommentCount="2" />
  <row Id="186" PostTypeId="2" ParentId="130" CreationDate="2010-07-20T00:46:08.467" Score="2" ViewCount="" Body="&lt;p&gt;I'm using Clojure along with the excellent &lt;a href=&quot;http://incanter.org/&quot; rel=&quot;nofollow&quot;&gt;Incanter&lt;/a&gt; at work (summer intern; I get to do what I want for a bit). It's been (mostly!) a pleasure. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Getting situated with Emacs (sounds like you have that down) and setting up the environment used to be a pain. Now, though, it isn't too bad: &lt;a href=&quot;http://github.com/technomancy/leiningen&quot; rel=&quot;nofollow&quot;&gt;Leiningen&lt;/a&gt; it and you're set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have but minimal experience with R, but Incanter and Clojure, together, along with whatever Java libs you need should have you covered in many ways. Not only that, but you've got a full-blown, beautiful language to boot. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd take a chance with it–at the very least you'll have fun.&lt;/p&gt;&#xA;" OwnerUserId="169" LastActivityDate="2010-07-20T00:46:08.467" />
  <row Id="187" PostTypeId="2" ParentId="181" CreationDate="2010-07-20T00:47:45.310" Score="2" ViewCount="" Body="&lt;p&gt;As far as I know there is no way to select automatically the number of layers and neurons in each layer. But there are networks that can build automatically their topology, like EANN (Evolutionary Artificial Neural Networks, which use Genetic Algorithms to evolved the topology).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are several approaches, a more or less modern one that seemed to give good results was NEAT (Neuro Evolution of Augmented Topologies). You can get more info:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://nn.cs.utexas.edu/?neat&quot; rel=&quot;nofollow&quot;&gt;http://nn.cs.utexas.edu/?neat&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="119" LastActivityDate="2010-07-20T00:47:45.310" />
  <row Id="188" PostTypeId="2" ParentId="165" CreationDate="2010-07-20T00:52:13.287" Score="6" ViewCount="" Body="&lt;p&gt;I'd probably say something like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Anytime we want to talk about probabilities, we're really integrating a density.  In Bayesian analysis, a lot of the densities we come up with aren't analytically tractable: you can only integrate them -- if you can integrate them at all -- with a great deal of suffering.  So what we do instead is simulate the random variable a lot, and then figure out probabilities from our simulated random numbers.  If we want to know the probability that X is less than 10, we count the proportion of simulated random variable results less than 10 and use that as our estimate.  That's the &quot;Monte Carlo&quot; part, it's an estimate of probability based off of random numbers.  With enough simulated random numbers, the estimate is very good, but it's still inherently random.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;So why &quot;Markov Chain&quot;?  Because under certain technical conditions, you can generate a memoryless process (aka a Markovian one) that has the same limiting distribution as the random variable that you're trying to simulate.  You can iterate any of a number of different kinds of simulation processes that generate correlated random numbers (based only on the current value of those numbers), and you're guaranteed that once you pool enough of the results, you will end up with a pile of numbers that looks &quot;as if&quot; you had somehow managed to take independent samples from the complicated distribution you wanted to know about.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;So for example, if I want to estimate the probability that a standard normal random variable was less than 0.5, I could generate ten thousand independent realizations from a standard normal distribution and count up the number less than 0.5; say I got 6905 that were less than 10000; my estimate for P(Z&amp;lt;0.5) would be 0.6905, which isn't that far off from the actual value.  That'd be a Monte Carlo estimate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Now imagine I couldn't draw independent normal random variables, instead I'd start at 0, and then with every step add some uniform random number between -0.5 and 0.5 to my current value, and then decide based on a particular test whether I liked that new value or not; if I liked it, I'd use the new value as my current one, and if not, I'd reject it and stick with my old value.  Because I only look at the new and current values, this is a Markov chain.  If I set up the test to decide whether or not I keep the new value correctly (it'd be a random walk Metropolis-Hastings, and the details get a bit complex), then even though I never generate a single normal random variable, if I do this procedure for long enough, the list of numbers I get from the procedure will be distributed like a large number of draws from something that generates normal random variables.  This would give me a Markov Chain Monte Carlo simulation for a standard normal random variable.  If I used this to estimate probabilities, that would be a MCMC estimate.&quot;&lt;/p&gt;&#xA;" OwnerUserId="61" LastActivityDate="2010-07-20T00:52:13.287" CommunityOwnedDate="2010-08-16T06:37:30.433" CommentCount="3" />
  <row Id="189" PostTypeId="2" ParentId="23" CreationDate="2010-07-20T00:59:34.643" Score="3" ViewCount="" Body="&lt;p&gt;For a continuous random variable you can always approximate the pdf by calculating (CDF(x2) - CDF(x1))/(x2 - x1) where x1 and x2 are on either side of the point where you want to know the pdf and the distance |x2 - x1| is small.&lt;/p&gt;&#xA;" OwnerUserId="173" LastActivityDate="2010-07-20T00:59:34.643" CommentCount="2" />
  <row Id="190" PostTypeId="2" ParentId="170" CreationDate="2010-07-20T01:07:38.383" Score="3" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://www.sportsci.org/resource/stats/&quot; rel=&quot;nofollow&quot;&gt;A New View of Statistics&lt;/a&gt; by Will G. Hopkins is great!&lt;/p&gt;&#xA;" OwnerUserId="25" LastActivityDate="2010-07-20T01:07:38.383" />
  <row Id="191" PostTypeId="2" ParentId="128" CreationDate="2010-07-20T01:17:17.377" Score="2" ViewCount="" Body="&lt;p&gt;The Bland-Altman plot is more widely known as the &lt;strong&gt;Tukey Mean-Difference Plot&lt;/strong&gt; (one of many charts devised by John Tukey &lt;a href=&quot;http://en.wikipedia.org/wiki/John_Tukey&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/John_Tukey&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The idea is that x-axis is the mean of your two measurements, which is your best guess as to the &quot;correct&quot; result and the y-axis is the difference between the two measurement differences. The chart can then highlight certain types of anomalies in the measurements. For example, if one method always gives too high a result, then you'll get all of your points above or all below the zero line. It can also reveal, for example, that one method over-estimates high values and under-estimates low values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you see the points on the Bland-Altman plot scattered all over the place, above and below zero, then the suggests that there is no consistent bias of one approach versus the other (of course, there could be hidden biases that this plot does not show up).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Essentially, it is a good first step for exploring the data. Other techniques can be used to dig into more particular sorts of behaviour of the measurements.&lt;/p&gt;&#xA;" OwnerUserId="173" LastActivityDate="2010-07-20T01:17:17.377" />
  <row Id="192" PostTypeId="1" AcceptedAnswerId="293" CreationDate="2010-07-20T01:18:11.523" Score="3" ViewCount="625" Body="&lt;p&gt;I'm aware that this one is far from &lt;em&gt;yes or no&lt;/em&gt; question, but I'd like to know which techniques do you prefer in categorical data analysis - i.e. cross tabulation with two categorical variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've come up with: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&amp;chi;&lt;sup&gt;2&lt;/sup&gt; test - well, this is quite self-explanatory&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Fisher's exact test - when n &amp;lt; 40,&lt;/li&gt;&#xA;&lt;li&gt;Yates' continuity correction - when n &gt; 40,&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;Cramer's V - measure of association for tables which have more than &lt;em&gt;2 x 2&lt;/em&gt; cells,&lt;/li&gt;&#xA;&lt;li&gt;&amp;Phi; coefficient - measure of association for &lt;em&gt;2 x 2&lt;/em&gt; tables,&lt;/li&gt;&#xA;&lt;li&gt;contingency coefficient (C) - measure of association for &lt;em&gt;n x n&lt;/em&gt; tables,&lt;/li&gt;&#xA;&lt;li&gt;odds ratio - independence of two categorical variables,&lt;/li&gt;&#xA;&lt;li&gt;McNemar marginal homogeniety test,&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;And my question here is: Which statistical techniques for cross-tabulated data (two categorical variables) do you consider relevant (and why)?&lt;/p&gt;&#xA;" OwnerUserId="1356" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-03T15:54:57.267" LastActivityDate="2010-11-03T15:54:57.267" Title="Cross tabulation of two categorical variables: recommended techniques" Tags="&lt;categorical-data&gt;&lt;association-measure&gt;" AnswerCount="4" CommentCount="1" FavoriteCount="1" />
  <row Id="193" PostTypeId="2" ParentId="166" CreationDate="2010-07-20T01:45:12.020" Score="4" ViewCount="" Body="&lt;p&gt;Suppose that you want to know what percentage of people would vote for a particular candidate (say, pi. Note: by definition pi is between 0 and 100). You sample N voters at random to find out how they would vote and your survey of these N voters tells you that the percentage is p. So, you would like to establish a confidence interval for the true percentage. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you assume that p is normally distributed (an assumption that may or may not be justified depending on how 'big' N is) then your confidence interval for pi would be of the following form:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;CI = [ p - k * sd(p) , p + k * sd(p)]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where k is a constant that depends on the extent of confidence you want (i.e., 95% or 99% etc).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From a polling perspective, you want the width of your confidence interval to be 'low'. Usually, pollsters work with the margin of error which is basically one-half of the CI. In other words:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Margin of Error = k * sd(p). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is how we would go about calculating sd(p).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By definition, p = Sum X_i / N &lt;/p&gt;&#xA;&#xA;&lt;p&gt;where,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;X_i = 1 if voter i votes for candidate and 0 otherwise.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since, we sampled the voters at random, we could assume that X_i is a i.i.d bernoulli random variable. Therefore, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Variance(P) =  V ( Sum X_i / N) = [ Sum V(X_i) ] / N^2 = N pi (1-pi) / N^2 = pi  (1-pi) / N.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;sd(p) = sqrt ( pi * (1-pi) / N)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now to estimate margin of error we need to know pi which we do not know obviously. But, an inspection of the numerator suggests that the 'worst' estimate for sd(p) in the sense that we get the 'largest' standard deviation is when pi = 50. Therefore,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Worst possible sd(p) = sqrt (50 * 50 / N ) = 50 / sqrt(N)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, you see that the margin of error falls off exponentially with N and thus you really do not need very big samples to reduce your margin of error or in other words N need not be very large for you to obtain a narrow confidence interval.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, for a 95 % confidence interval (i.e., k= 1.96) and N = 1000, the confidence interval is: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;[p - 1.96 * 50/sqrt(1000), p + 1.96 * 50/sqrt(1000)] = [p - 3, p + 3] &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As we increase N the costs of polling go up linearly but the gains go down exponentially. That is the reason why pollsters usually cap N at 1000 as that gives them a reasonable error of margin under the worst possible assumption of pi = 50%.&lt;/p&gt;&#xA;" LastActivityDate="2010-07-20T01:45:12.020" />
  <row Id="194" PostTypeId="1" AcceptedAnswerId="200" CreationDate="2010-07-20T01:47:36.197" Score="5" ViewCount="217" Body="&lt;p&gt;I am sure that everyone who's trying to find patterns in historical stock market data or betting history would like to know about this. Given a huge sets of data, and thousands of random variables that may or may not affect it, it makes sense to ask any patterns that you extract out from the data are indeed true patterns, not statistical fluke.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A lot of patterns are only valid when they are tested in the samples. And even those that are patterns that are valid out of samples may cease to become valid when you apply it in the real world. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand that it is not possible to completely 100% make sure a pattern is valid all the time, but besides in and out of samples tests, are their any tests that could establish the validness of a pattern?&lt;/p&gt;&#xA;" OwnerUserId="175" LastActivityDate="2010-07-21T18:41:12.033" Title="Data Mining-- How to Tell Whether the Pattern Extracted is Meaningful?" Tags="&lt;data-mining&gt;" AnswerCount="3" CommentCount="2" />
  <row Id="195" PostTypeId="1" AcceptedAnswerId="2872" CreationDate="2010-07-20T02:01:05.727" Score="2" ViewCount="163" Body="&lt;p&gt;I am looking at fitting distributions to data (with a particular focus on the tail) and am leaning towards Anderson-Darling tests rather than Kolmogorov-Smirnov. What do you think are the relative merits of these or other tests for fit (e.g. Cramer-von Mises)?&lt;/p&gt;&#xA;" OwnerUserId="173" LastActivityDate="2010-09-20T00:29:57.047" Title="What do you think is the best goodness of fit test?" Tags="&lt;test&gt;&lt;hypothesis-testing&gt;&lt;fitting&gt;" AnswerCount="3" CommentCount="0" FavoriteCount="2" />
  <row Id="196" PostTypeId="1" AcceptedAnswerId="232" CreationDate="2010-07-20T02:17:24.800" Score="7" ViewCount="304" Body="&lt;p&gt;Besides &lt;a href=&quot;http://en.wikipedia.org/wiki/Gnuplot&quot; rel=&quot;nofollow&quot;&gt;gnuplot&lt;/a&gt; and &lt;a href=&quot;http://www.ggobi.org/&quot; rel=&quot;nofollow&quot;&gt;ggobi&lt;/a&gt;, what open source tools are people using for visualizing multi-dimensional data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Gnuplot is more or less a basic plotting package. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ggobi can do a number of nifty things, such as:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;animate data along a dimension or among discrete collections&lt;/li&gt;&#xA;&lt;li&gt;animate linear combinations varying the coefficients&lt;/li&gt;&#xA;&lt;li&gt;compute principal components and other transformations&lt;/li&gt;&#xA;&lt;li&gt;visualize and rotate 3 dimensional data clusters&lt;/li&gt;&#xA;&lt;li&gt;use colors to represent a different dimension&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;What other useful approaches are based in open source and thus freely reusable or customizable?&lt;/p&gt;&#xA;" OwnerUserId="87" LastEditorUserId="509" LastEditorDisplayName="" LastEditDate="2010-08-14T12:21:50.320" LastActivityDate="2010-08-16T06:19:27.897" CommunityOwnedDate="2010-07-20T02:35:32.883" Title="Open source tools for visualizing multi-dimensional data?" Tags="&lt;data-visualization&gt;&lt;open-source&gt;" AnswerCount="6" CommentCount="1" FavoriteCount="5" />
  <row Id="197" PostTypeId="2" ParentId="196" CreationDate="2010-07-20T02:24:38.993" Score="5" ViewCount="" Body="&lt;p&gt;How about R with &lt;a href=&quot;http://had.co.nz/ggplot2/&quot; rel=&quot;nofollow&quot;&gt;ggplot2&lt;/a&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other tools that I really like:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.processing.org/&quot; rel=&quot;nofollow&quot;&gt;Processing&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://prefuse.org/&quot; rel=&quot;nofollow&quot;&gt;Prefuse&lt;/a&gt;  &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://vis.stanford.edu/protovis/&quot; rel=&quot;nofollow&quot;&gt;Protovis&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="5" LastEditorUserId="5" LastEditorDisplayName="" LastEditDate="2010-07-20T02:42:01.603" LastActivityDate="2010-07-20T02:42:01.603" CommunityOwnedDate="2010-08-16T13:45:34.723" />
  <row Id="198" PostTypeId="2" ParentId="156" CreationDate="2010-07-20T02:32:43.653" Score="2" ViewCount="" Body="&lt;p&gt;Start with the distribution of y_bar, show that since v is normal, y_bar is multivariate normal and that consequently u must also be a multivariate normal; also show that the covariance matrix of y_bar is of the form sigma^2*(X^T * X)^-1 and thus -- if sigma^2 were known -- the variance of u would be sigma^2*c^T*(X^T*X)^(-1)*c.  Show that the distribution of e_bar^T * e_bar must be chi-squared and (&lt;em&gt;carefully&lt;/em&gt;) find the degrees of freedom.  Think about how what the operation e_bar^T*e_bar*c^T*(X^T*X)^(-1)*c must therefore produce, and what it's distribution and degrees of freedom are.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The result follows (almost) immediately from the definition of the t-distribution.&lt;/p&gt;&#xA;" OwnerUserId="61" LastActivityDate="2010-07-20T02:32:43.653" />
  <row Id="199" PostTypeId="2" ParentId="194" CreationDate="2010-07-20T02:32:53.093" Score="2" ViewCount="" Body="&lt;p&gt;You could try:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Bagging &lt;a href=&quot;http://en.m.wikipedia.org/wiki/Bootstrap_aggregating&quot; rel=&quot;nofollow&quot;&gt;http://en.m.wikipedia.org/wiki/Bootstrap_aggregating&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Boosting &lt;a href=&quot;http://en.m.wikipedia.org/wiki/Boosting&quot; rel=&quot;nofollow&quot;&gt;http://en.m.wikipedia.org/wiki/Boosting&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Cross validation &lt;a href=&quot;http://en.m.wikipedia.org/wiki/Cross-validation_(statistics&quot; rel=&quot;nofollow&quot;&gt;http://en.m.wikipedia.org/wiki/Cross-validation_(statistics&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-07-20T02:32:53.093" />
  <row Id="200" PostTypeId="2" ParentId="194" CreationDate="2010-07-20T02:48:45.177" Score="8" ViewCount="" Body="&lt;p&gt;If you want to know that a pattern is meaningful, you need to show what it actually &lt;em&gt;means&lt;/em&gt;. Statistical tests do not do this. Unless your data can be said to be in some sense &quot;complete&quot;, inferences draw from the data will always be provisional.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can increase your &lt;em&gt;confidence&lt;/em&gt; in the validity of a pattern by testing against more and more out of sample data, but that doesn't protect you from it turning out to be an artefact. The broader your range of out of sample data -- eg, in terms of how it is acquired and what sort of systematic confounding factors might exist within it -- the better the validation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ideally, though, you need to go beyond identifying patterns and come up with a persuasive theoretical framework that &lt;em&gt;explains&lt;/em&gt; the patterns you've found, and then test &lt;em&gt;that&lt;/em&gt; by other, independent means. (This is called &quot;science&quot;.)&lt;/p&gt;&#xA;" OwnerUserId="174" LastActivityDate="2010-07-20T02:48:45.177" CommentCount="3" />
  <row Id="201" PostTypeId="2" ParentId="7" CreationDate="2010-07-20T03:11:36.027" Score="5" ViewCount="" Body="&lt;p&gt;Start R and type &lt;code&gt;data()&lt;/code&gt;. This will show all datasets in the search path.&#xA;Many additional datasets are available in add-on packages.&#xA;For example, there are some interesting real-world social science datasets in the &lt;code&gt;AER&lt;/code&gt; package.&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-07-20T03:11:36.027" />
  <row Id="202" PostTypeId="2" ParentId="138" CreationDate="2010-07-20T03:13:22.953" Score="6" ViewCount="" Body="&lt;ul&gt;&#xA;&lt;li&gt;If you like learning through videos, I collated a &lt;a href=&quot;http://jeromyanglim.blogspot.com/2010/05/videos-on-data-analysis-with-r.html&quot; rel=&quot;nofollow&quot;&gt;list of R training videos&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;li&gt;I also prepared a &lt;a href=&quot;http://jeromyanglim.blogspot.com/2009/06/learning-r-for-researchers-in.html&quot; rel=&quot;nofollow&quot;&gt;general post on learning R&lt;/a&gt; with  suggestions on books, online manuals, blogs, videos, user interfaces, and more.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-27T03:38:47.843" LastActivityDate="2011-05-27T03:38:47.843" CommunityOwnedDate="2010-08-01T18:56:25.603" />
  <row Id="203" PostTypeId="1" AcceptedAnswerId="243" CreationDate="2010-07-20T03:31:45.820" Score="7" ViewCount="790" Body="&lt;p&gt;Following on from &lt;a href=&quot;http://stats.stackexchange.com/questions/10/under-what-conditions-should-likert-scales-be-used-as-ordinal-or-interval-data&quot; rel=&quot;nofollow&quot;&gt;this question&lt;/a&gt;:&#xA;Imagine that you want to test for differences in central tendency between two groups (e.g., males and females)&#xA;on a 5-point Likert item (e.g., satisfaction with life: Dissatisfied to Satisfied).&#xA;I think a t-test would be sufficiently accurate for most purposes,&#xA; but that a bootstrap test of differences between group means would often provide more accurate estimate of confidence intervals.&#xA;What statistical test would you use?&lt;/p&gt;&#xA;" OwnerUserId="183" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-03-30T18:36:01.337" LastActivityDate="2011-03-30T18:36:01.337" Title="Group differences on a five point Likert item" Tags="&lt;t-test&gt;&lt;scales&gt;&lt;ordinal&gt;&lt;likert&gt;&lt;interval&gt;" AnswerCount="4" CommentCount="2" FavoriteCount="1" />
  <row Id="204" PostTypeId="2" ParentId="196" CreationDate="2010-07-20T03:35:58.693" Score="4" ViewCount="" Body="&lt;p&gt;The lattice package in R&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-07-20T03:35:58.693" CommunityOwnedDate="2010-07-20T03:35:58.693" />
  <row Id="205" PostTypeId="1" AcceptedAnswerId="353" CreationDate="2010-07-20T03:51:24.050" Score="6" ViewCount="196" Body="&lt;p&gt;I'm curious about why we treat fitting GLMS as though they were some special optimization problem.  Are they?  It seems to me that they're just maximum likelihood, and that we write down the likelihood and then ... we maximize it!  So why do we use Fisher scoring instead of any of the myriad of optimization schemes that has been developed in the applied math literature? &lt;/p&gt;&#xA;" OwnerUserId="187" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-11-16T23:36:33.113" LastActivityDate="2010-11-16T23:36:33.113" Title="Why do we make a big fuss about using Fisher scoring when we fit a GLM?" Tags="&lt;glm&gt;&lt;optimization&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="206" PostTypeId="1" AcceptedAnswerId="209" CreationDate="2010-07-20T03:53:54.767" Score="2" ViewCount="416" Body="&lt;p&gt;What is the difference between discrete data and continuous data?&lt;/p&gt;&#xA;" OwnerUserId="188" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-27T03:35:28.263" LastActivityDate="2011-05-27T03:35:28.263" Title="What is the difference between discrete data and continuous data?" Tags="&lt;continuous-data&gt;&lt;discrete-data&gt;" AnswerCount="5" CommentCount="1" />
  <row Id="207" PostTypeId="2" ParentId="165" CreationDate="2010-07-20T04:00:14.387" Score="14" ViewCount="" Body="&lt;p&gt;First, we need to understand what is a markov chain. Consider the following &lt;a href=&quot;http://en.wikipedia.org/wiki/Examples_of_Markov_chains#A_very_simple_weather_model&quot; rel=&quot;nofollow&quot;&gt;weather&lt;/a&gt; example from Wikipedia. Suppose that weather on any given day can be classified into two states only: sunny and rainy. Based on past experience, we know the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Probability(Next day is sunny | Given today is rainy ) = 0.50&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since, the next day's weather is either sunny or rainy it follows that:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Probability(Next day is Rainy | Given today is rainy ) = 0.50 &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Similarly, let:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Probability(Next day is rainy | Given today is sunny ) = 0.10&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, it follows that:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Probability(Next day is sunny | Given today is sunny ) = 0.90&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The above four numbers can be compactly represented as a transition matrix which represents the probabilities of the weather moving from one state to another state as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;         S   R&#xA;P = S [ 0.9 0.1&#xA;    R   0.5 0.5]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;We might ask several questions whose answers follow:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Q1: If the weather is sunny today then what is the weather likely to be tomorrow?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A1: Since, we do not know what is going to happen for sure, the best we can say is that there is a 90% chance that it is likely to be sunny and 10% that it will be rainy. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Q2: What about two days from today?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A2: One day prediction: 90% sunny, 10% rainy. Therefore, two days from now:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First day it can be sunny and the next day also it can be sunny. Chances of this happening are: 0.9  0.9. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First day it can be rainy and second day it can be sunny. Chances of this happening are: 0.1 * 0.5&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, the probability that the weather will be sunny in two days is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Prob(Sunny two days from now) = 0.9  0.9 + 0.1  0.5 = 0.81 + 0.05 = 0.86 &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Similarly, the probability that it will be rainy is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Prob(Rainy two days from now) = 0.1 * 0.5 + 0.9 0.1 = 0.05 + 0.09 = 0.14&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you keep forecasting weather like this you will notice that eventually the nth day forecast where n is very large (say 30) settles to the following 'equilibrium' probabilities:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Prob(Sunny) = 0.833&#xA;Prob(Rainy) = 0.167&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words, your forecast for the nth day and the n+1th day remain the same. In addition, you can also check that the 'equilibrium' probabilities do not depend on the weather today. You would get the same forecast for the weather if you start of by assuming that the weather today is sunny or rainy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The above example will only work if the state transition probabilities satisfy several conditions which I will not discuss here. But, notice the following features of this 'nice' markov chain (nice = transition probabilities satisfy conditions):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Irrespective of the initial starting state we will eventually reach an equilibrium probability distribution of states.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Markov Chain Monte Carlo exploits the above feature as follows: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We want to generate random draws from a target distribution. We then identify a way to construct a 'nice' markov chain such that its equilibrium probability distribution is our target distribution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we can construct such a chain then we arbitrarily start from some point and iterate the markov chain many times (like how we forecasted the weather n times). Eventually, the draws we generate would appear as if they are coming from our target distribution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We then approximate the quantities of interest (e.g. mean) by taking the sample average of the draws after discarding a few initial draws which is the monte carlo component.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are several ways to construct 'nice' markov chains (e.g., gibbs sampler, Metropolis-Hastings algorithm).&lt;/p&gt;&#xA;" LastActivityDate="2010-07-21T16:00:43.450" CommunityOwnedDate="2010-08-16T06:37:30.433" />
  <row Id="208" PostTypeId="2" ParentId="206" CreationDate="2010-07-20T04:07:11.093" Score="4" ViewCount="" Body="&lt;p&gt;Temperatures are continuous. It can be 23 degrees, 23.1 degrees, 23.100004 degrees. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Gender is discrete. You can only be male or female (insert san francisco joke here). Something you would represent with a whole number like 0, 1, 2, etc&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The difference is important as many statistical and data mining algorithms can handle one type but not the other. For example in regular regression, the Y must be continuous. In logistic regression the Y is discrete. &lt;/p&gt;&#xA;" OwnerUserId="74" LastActivityDate="2010-07-20T04:07:11.093" />
  <row Id="209" PostTypeId="2" ParentId="206" CreationDate="2010-07-20T04:16:52.663" Score="12" ViewCount="" Body="&lt;p&gt;Discrete data can only take particular values. There may potentially be an infinite number of those values, but each is distinct and there's no grey area in between. Discrete data can be numeric -- like numbers of apples -- but it can also be categorical -- like red or blue, or male or female, or good or bad.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Continuous data are not restricted to defined separate values, but can occupy any value over a continuous range. Between any two continuous data values there may be an infinite number of others. Continuous data are always essentially numeric.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It sometimes makes sense to treat numeric data that is properly of one type as being of the other. For example, something like &lt;em&gt;height&lt;/em&gt; is continuous, but often we don't really care too much about tiny differences and instead group heights into a number of discrete &lt;strong&gt;bins&lt;/strong&gt;. Conversely, if we're counting large amounts of some discrete entity -- grains of rice, or termites, or pennies in the economy -- we may choose not to think of 2,000,006 and 2,000,008 as crucially different values but instead as nearby points on an approximate continuum.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It can also sometimes be useful to treat numeric data as categorical, eg: underweight, normal, obese. This is usually just another kind of binning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seldom makes sense to consider categorical data as continuous.&lt;/p&gt;&#xA;" OwnerUserId="174" LastEditorUserId="174" LastEditorDisplayName="" LastEditDate="2010-07-20T05:25:22.477" LastActivityDate="2010-07-20T05:25:22.477" CommentCount="3" />
  <row Id="210" PostTypeId="2" ParentId="206" CreationDate="2010-07-20T04:19:12.023" Score="5" ViewCount="" Body="&lt;p&gt;Data is always discrete. Given a sample of &lt;code&gt;n&lt;/code&gt; values on a variable, the maximum number of distinct values the variable can take is equal to  &lt;code&gt;n&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data on a variable are typically assumed to be drawn from a random variable.&#xA;The random variable is continuous over a range if there is an infinite number of possible values that the variable can take between any two different points in the range.&#xA;For example, height, weight, and time are typically assumed to be continuous.&#xA;Of course, any measurement of these variables will be finitely accurate and in some&#xA; sense discrete.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is useful to distinguish between ordered (i.e., ordinal), unordered (i.e., nominal),&lt;br&gt;&#xA;and binary discrete variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some introductory textbooks confuse a continuous variable with a numeric variable.&#xA;For example, a score on a computer game is discrete even though it is numeric.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some introductory textbooks confuse a ratio variable with continuous variables. A count variable is a ratio variable, but it is not continuous.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In actual practice, a variable is often treated as continuous when it can take on a sufficiently large number of different values.&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-07-20T04:19:12.023" />
  <row Id="211" PostTypeId="2" ParentId="138" CreationDate="2010-07-20T04:49:07.680" Score="3" ViewCount="" Body="&lt;p&gt;I have written a document that is freely available at my website and on CRAN. See the linked page:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.ms.unimelb.edu.au/~andrewpr/r-users/&quot; rel=&quot;nofollow&quot;&gt;icebreakeR&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The datasets that are used in the document are also linked from that page.  Feedback is welcome and appreciated!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Andrew&lt;/p&gt;&#xA;" OwnerUserId="187" LastActivityDate="2010-07-20T04:49:07.680" CommunityOwnedDate="2010-08-01T18:56:25.603" />
  <row Id="212" PostTypeId="1" AcceptedAnswerId="5001" CreationDate="2010-07-20T04:54:20.793" Score="3" ViewCount="167" Body="&lt;p&gt;I have 2 ASR (Automatic Speech Recognition) models, providing me with text transcriptions for my testdata. The error measure I use is Word Error Rate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What methods do I have to test for statistical significance of my new results?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;An example:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have an experiment with 10 speaker, all having 100 (the same) sentences, total 900 words per speaker. Method A has an WER (word error rate) of 19.0%, Method B 18.5%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I test whether Method B is significantly better?&lt;/p&gt;&#xA;" OwnerUserId="190" LastEditorUserId="190" LastEditorDisplayName="" LastEditDate="2010-07-21T06:19:29.143" LastActivityDate="2010-11-29T18:25:11.713" Title="What method to use to test Statistical Significance of ASR results" Tags="&lt;statistical-significance&gt;" AnswerCount="3" />
  <row Id="213" PostTypeId="1" AcceptedAnswerId="532" CreationDate="2010-07-20T05:02:33.793" Score="22" ViewCount="1018" Body="&lt;p&gt;Suppose I have a large set of multivariate data with at least three variables. How can I find the outliers? Pairwise scatterplots won't work as it is possible for an outlier to exist in 3 dimensions that is not an outlier in any of the 2 dimensional subspaces.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not thinking of a regression problem, but of true multivariate data. So answers involving robust regression or computing leverage are not helpful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One possibility would be to compute the principal component scores and look for an outlier in the bivariate scatterplot of the first two scores. Would that be guaranteed to work? Are there better approaches?&lt;/p&gt;&#xA;" OwnerUserId="159" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-07-20T05:28:56.073" LastActivityDate="2011-02-27T21:22:37.633" Title="What is the best way to identify outliers in multivariate data?" Tags="&lt;multivariable&gt;&lt;outliers&gt;" AnswerCount="14" CommentCount="2" FavoriteCount="16" />
  <row Id="214" PostTypeId="2" ParentId="170" CreationDate="2010-07-20T05:02:42.573" Score="1" ViewCount="" Body="&lt;p&gt;Some free Stats textbooks are also available &lt;a href=&quot;http://www.e-booksdirectory.com/mathematics.php&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="40" LastActivityDate="2010-07-20T05:02:42.573" />
  <row Id="215" PostTypeId="2" ParentId="195" CreationDate="2010-07-20T05:03:12.730" Score="1" ViewCount="" Body="&lt;p&gt;I'm not sure about these tests, so this answer may be off-topic.  Apologies if so.  But, are you sure that you want a test?  It really depends on what the purpose of the exercise is.  Why are you fitting the distributions to the data, and what will you do with the fitted distributions  afterward?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to know what distribution fits best just because you're interested, then a test may help.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, if you want to actually do something with the distribution, then you'd be better off developing a loss function based on your intentions, and using the distribution that gives you the most satisfactory value for the loss function.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It sounds to me from your description (particular focus on the tail) that you want to actually do something with the distribution.  If so, it's hard for me to imagine a situation where an existing test will provide better guidance than comparing the effects of the fitted distributions in situ, somehow.&lt;/p&gt;&#xA;" OwnerUserId="187" LastActivityDate="2010-07-20T05:03:12.730" CommentCount="2" />
  <row Id="216" PostTypeId="1" AcceptedAnswerId="217" CreationDate="2010-07-20T05:04:40.840" Score="8" ViewCount="240" Body="&lt;p&gt;Hello,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are some good visualization libraries for online use? Are they easy to use and is there good documentation?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="191" LastActivityDate="2010-07-20T08:45:07.490" Title="Web visualization libraries" Tags="&lt;data-visualization&gt;&lt;library&gt;&lt;protovis&gt;" AnswerCount="3" FavoriteCount="6" />
  <row Id="217" PostTypeId="2" ParentId="216" CreationDate="2010-07-20T05:10:08.383" Score="5" ViewCount="" Body="&lt;p&gt;IMO, &lt;strong&gt;&lt;a href=&quot;http://vis.stanford.edu/protovis/&quot; rel=&quot;nofollow&quot;&gt;Protovis&lt;/a&gt;&lt;/strong&gt; is the best and is very well documented and supported.  It is the basis for my &lt;a href=&quot;http://cran.r-project.org/web/packages/webvis/index.html&quot; rel=&quot;nofollow&quot;&gt;webvis&lt;/a&gt; R package.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;These are also very good, although they have more of a learning curve:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.processing.org/&quot; rel=&quot;nofollow&quot;&gt;Processing&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://prefuse.org/&quot; rel=&quot;nofollow&quot;&gt;Prefuse&lt;/a&gt;  &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="5" LastEditorUserId="5" LastEditorDisplayName="" LastEditDate="2010-07-20T05:15:51.977" LastActivityDate="2010-07-20T05:15:51.977" CommentCount="3" />
  <row Id="218" PostTypeId="2" ParentId="1" CreationDate="2010-07-20T05:13:21.963" Score="2" ViewCount="" Body="&lt;p&gt;This interesting question is the subject of some research in &lt;a href=&quot;http://www.acera.unimelb.edu.au/&quot; rel=&quot;nofollow&quot;&gt;ACERA&lt;/a&gt;.  The lead researcher is Andrew  Speirs-Bridge, and his work is eminently google-able :)&lt;/p&gt;&#xA;" OwnerUserId="187" LastActivityDate="2010-07-20T05:13:21.963" />
  <row Id="219" PostTypeId="2" ParentId="40" CreationDate="2010-07-20T05:21:14.590" Score="2" ViewCount="" Body="&lt;p&gt;The Xorshift PNG designed by George Marsaglia. Its period (2^128-1) is much shorter than the Mersenne-Twister but the algorithm is very simple to implement and lends itself to parallelization. Performs well on many-core architectures such as DSP chips and Nvidia's Tesla.&lt;/p&gt;&#xA;" OwnerUserId="154" LastEditorUserId="154" LastEditorDisplayName="" LastEditDate="2010-07-20T05:33:00.140" LastActivityDate="2010-07-20T05:33:00.140" CommentCount="3" />
  <row Id="220" PostTypeId="1" AcceptedAnswerId="221" CreationDate="2010-07-20T05:23:04.163" Score="6" ViewCount="295" Body="&lt;p&gt;If $X_1, ..., X_n$ are independent identically-distributed random variables, what can be said about the distribution of $\min(X_1, ..., X_n)$ in general?&lt;/p&gt;&#xA;" OwnerUserId="85" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-29T00:37:42.233" LastActivityDate="2011-04-29T00:37:42.233" Title="How is the minimum of a set of random variables distributed?" Tags="&lt;distributions&gt;&lt;random-variable&gt;&lt;minimum&gt;" AnswerCount="3" CommentCount="2" />
  <row Id="221" PostTypeId="2" ParentId="220" CreationDate="2010-07-20T05:35:48.807" Score="8" ViewCount="" Body="&lt;p&gt;If the cdf of $X_i$ is denoted by $F(x)$, then the cdf of the minimum is given by $[1-F(x)]^n$.&lt;/p&gt;&#xA;" OwnerUserId="159" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-04-28T06:51:33.960" LastActivityDate="2011-04-28T06:51:33.960" />
  <row Id="222" PostTypeId="1" CreationDate="2010-07-20T05:37:46.557" Score="9" ViewCount="1560" Body="&lt;p&gt;I know this is probably simplistic but what are Principal component scores?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This question originates from my attempt to understand this question &lt;a href=&quot;http://stats.stackexchange.com/questions/213/what-is-the-best-way-to-identify-outliers-in-multivariate-data&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="191" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-08-07T17:55:46.563" LastActivityDate="2010-08-07T17:55:46.563" Title="What are principal component scores?" Tags="&lt;pca&gt;&lt;scores&gt;" AnswerCount="6" CommentCount="1" FavoriteCount="7" />
  <row Id="223" PostTypeId="1" CreationDate="2010-07-20T05:54:15.017" Score="4" ViewCount="114" Body="&lt;p&gt;I have a friend who is an MD and wants to refresh his Statistics. So is there any recommended resource online (or offline) ? He did stats ~20 years ago.&lt;/p&gt;&#xA;" OwnerUserId="79" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-08-09T10:37:57.740" LastActivityDate="2010-08-09T10:37:57.740" Title="Intro to statistics for an MD?" Tags="&lt;online&gt;&lt;books&gt;" AnswerCount="4" CommentCount="4" FavoriteCount="3" />
  <row Id="224" PostTypeId="1" CreationDate="2010-07-20T06:03:59.930" Score="7" ViewCount="415" Body="&lt;p&gt;Which visualization libraries (plots, graphs, ...) would you suggest to use in a standalone application (Linux, .Net, Windows, whatever). Reasonable performance would be nice as well.&lt;/p&gt;&#xA;" OwnerUserId="128" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-27T03:36:21.040" LastActivityDate="2011-05-27T03:36:21.040" CommunityOwnedDate="2010-08-30T05:10:29.263" Title="Recommended visualization libraries for standalone applications" Tags="&lt;data-visualization&gt;&lt;software&gt;" AnswerCount="10" CommentCount="3" FavoriteCount="7" />
  <row Id="225" PostTypeId="1" AcceptedAnswerId="717" CreationDate="2010-07-20T06:07:37.007" Score="4" ViewCount="271" Body="&lt;p&gt;Why is the average of the highest value from 100 draws from a normal distribution different from the 98% percentile of the normal distribution?  It seems that by definition that they should be the same.  But...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Code in R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;NSIM &amp;lt;- 10000&#xA;x &amp;lt;- rep(NA,NSIM)&#xA;for (i in 1:NSIM)&#xA;{&#xA;    x[i] &amp;lt;- max(rnorm(100))&#xA;}&#xA;qnorm(.98)&#xA;qnorm(.99)&#xA;mean(x)&#xA;median(x)&#xA;hist(x)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I imagine that I'm misunderstanding something about what the maximum of a 100 draws from the normal distribution should be.  As is demonstrated by an unexpectedly asymetrical distribution of maximum values.&lt;/p&gt;&#xA;" OwnerUserId="196" LastEditorUserId="196" LastEditorDisplayName="" LastEditDate="2010-07-20T08:47:33.180" LastActivityDate="2010-07-28T01:06:56.963" Title="Why is the average of the highest value from 100 draws from a normal distribution different from the 98th percentile of the normal distribution?" Tags="&lt;r&gt;&lt;distributions&gt;&lt;maximum&gt;" AnswerCount="3" CommentCount="2" />
  <row Id="226" PostTypeId="2" ParentId="222" CreationDate="2010-07-20T06:23:21.623" Score="9" ViewCount="" Body="&lt;p&gt;Principal component analysis (PCA) is one popular approach analyzing variance when you are dealing with multivariate data. You have random variables X1, X2,...Xn which are all correlated (positively or negatively) to varying degrees, and you want to get a better understanding of what's going on. PCA can help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What PCA gives you is a change of variable into Y1, Y2,..., Yn (i.e. the same number of variables) which are linear combinations of the Xs. For example, you might have Y1 = 2.1 X1 - 1.76 X2 + 0.2 X3...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Ys the nice property that each of these have zero correlation with each other. Better still, you get them in decreasing order of variance. So, Y1 &quot;explains&quot; a big chunk of the variance of the original variables, Y2 a bit less and so on. Usually after the first few Ys, the variables become somewhat meaningless. The PCA score for any of the Xi is just it's coefficient in each of the Ys. In my earlier example, the score for X2 in the first principal component (Y1) is 1.76.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The way PCA does this magic is by computing eigenvectors of the covariance matrix.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To give a concrete example, imagine X1,...X10 are changes in 1 year, 2 year, ..., 10 year Treasury bond yields over some time period. When you compute PCA you generally find that the first component has scores for each bond of the same sign and about the same sign. This tells you that most of the variance in bond yields comes from everything moving the same way: &quot;parallel shifts&quot; up or down. The second component typically shows &quot;steepening&quot; and &quot;flattening&quot; of the curve and has opposite signs for X1 and X10.&lt;/p&gt;&#xA;" OwnerUserId="173" LastActivityDate="2010-07-20T06:23:21.623" CommentCount="2" />
  <row Id="227" PostTypeId="2" ParentId="222" CreationDate="2010-07-20T06:24:32.793" Score="2" ViewCount="" Body="&lt;p&gt;Let i be N rows and j be M columns. Suppose you linearize the combination of variables (columns):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Z_i1 = c_11×Y_i1 + c_12×Yi2 + ... + c_1M×YiM&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The above formula basically says to multiply row elements with a certain value c (loadings) and sum them by columns. Resulting values (Y values times the loading) are scores.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A principal component (PC) is a linear combination Z_1 = (Z_11, ..., Z_N1) (values by columns which are called scores). In essence, the PC should present the most important features of variables (columns). Ergo, you can extract as many PC as there are variables (or less).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An output from &lt;a href=&quot;http://cran.r-project.org/&quot; rel=&quot;nofollow&quot;&gt;R&lt;/a&gt; on PCA (a fake example) looks like this. PC1, PC2... are principal components 1, 2... The example below is showing only the first 8 principal components (out of 17). You can also extract other elements from PCA, like loadings and scores.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Importance of components:&#xA;                          PC1    PC2    PC3    PC4    PC5    PC6    PC7    PC8&#xA;Standard deviation     1.0889 1.0642 1.0550 1.0475 1.0387 1.0277 1.0169 1.0105&#xA;Proportion of Variance 0.0697 0.0666 0.0655 0.0645 0.0635 0.0621 0.0608 0.0601&#xA;Cumulative Proportion  0.0697 0.1364 0.2018 0.2664 0.3298 0.3920 0.4528 0.5129&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="144" LastEditorUserId="144" LastEditorDisplayName="" LastEditDate="2010-07-20T07:30:08.730" LastActivityDate="2010-07-20T07:30:08.730" CommentCount="1" />
  <row Id="228" PostTypeId="2" ParentId="216" CreationDate="2010-07-20T06:27:16.233" Score="2" ViewCount="" Body="&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://insideria.com/2009/12/28-rich-data-visualization-too.html&quot; rel=&quot;nofollow&quot;&gt;http://insideria.com/2009/12/28-rich-data-visualization-too.html&lt;/a&gt; 28 Rich Data Visualization Tools&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.rgraph.net/&quot; rel=&quot;nofollow&quot;&gt;http://www.rgraph.net/&lt;/a&gt; R graph&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://vis.stanford.edu/protovis/&quot; rel=&quot;nofollow&quot;&gt;http://vis.stanford.edu/protovis/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="10" LastActivityDate="2010-07-20T06:27:16.233" />
  <row Id="229" PostTypeId="2" ParentId="225" CreationDate="2010-07-20T06:28:47.557" Score="6" ViewCount="" Body="&lt;p&gt;The maximum does not have a normal distribution. Its cdf is Phi^100(x) where Phi(x) is the standard normal cdf. In general the moments of this distribution are tricky to obtain analytically. There is an ancient paper on this by &lt;a href=&quot;http://www.jstor.org/stable/2332087&quot; rel=&quot;nofollow&quot;&gt;Tippett (&lt;em&gt;Biometrika&lt;/em&gt;, 1925)&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-07-20T06:28:47.557" CommentCount="2" />
  <row Id="230" PostTypeId="2" ParentId="224" CreationDate="2010-07-20T06:35:21.287" Score="5" ViewCount="" Body="&lt;p&gt;You could have a look at Processing: &lt;a href=&quot;http://processing.org/&quot; rel=&quot;nofollow&quot;&gt;http://processing.org/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="173" LastActivityDate="2010-07-20T06:35:21.287" CommunityOwnedDate="2010-10-12T08:48:39.990" />
  <row Id="231" PostTypeId="2" ParentId="223" CreationDate="2010-07-20T06:41:13.570" Score="3" ViewCount="" Body="&lt;p&gt;This is one I've used successfully:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/013124941X&quot; rel=&quot;nofollow&quot;&gt;http://www.amazon.co.uk/Statistics-without-Psychology-Christine-Dancey/dp/013124941X&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I just stumlbed on this too, this might be useful:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www-users.york.ac.uk/~mb55/pubs/pbstnote.htm&quot; rel=&quot;nofollow&quot;&gt;http://www-users.york.ac.uk/~mb55/pubs/pbstnote.htm&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm sure I knew of a free pdf that some doctors I know use, but I can't seem to find it at the moment. I will try to dig it out.&lt;/p&gt;&#xA;" OwnerUserId="199" LastActivityDate="2010-07-20T06:41:13.570" />
  <row Id="232" PostTypeId="2" ParentId="196" CreationDate="2010-07-20T06:41:25.627" Score="6" ViewCount="" Body="&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://rosuda.org/mondrian/&quot; rel=&quot;nofollow&quot;&gt;Mondrian&lt;/a&gt;: Exploratory data analysis with focus on large data and databases.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://rosuda.org/iplots/&quot; rel=&quot;nofollow&quot;&gt;iPlots&lt;/a&gt;: a package for the R statistical environment which provides high interaction statistical graphics, written in Java.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="103" LastEditorUserId="103" LastEditorDisplayName="" LastEditDate="2010-07-20T07:26:11.410" LastActivityDate="2010-07-20T07:26:11.410" CommunityOwnedDate="2010-07-20T06:41:25.627" CommentCount="1" />
  <row Id="233" PostTypeId="2" ParentId="213" CreationDate="2010-07-20T06:45:55.213" Score="0" ViewCount="" Body="&lt;p&gt;Maybe I've got this wrapped around my head, but I would have thought a residuals plot would do this reasonably easily. Big residual= big outlier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think!?&lt;/p&gt;&#xA;" OwnerUserId="199" LastActivityDate="2010-07-20T06:45:55.213" CommentCount="2" />
  <row Id="234" PostTypeId="2" ParentId="222" CreationDate="2010-07-20T06:47:14.180" Score="2" ViewCount="" Body="&lt;p&gt;Say you have a cloud of N points in, say, 3D (which can be listed in a 100x3 array). Then, the principal components analysis (PCA) fits an arbitrarily oriented ellipsoid into the data. The principal component score is the length of the diameters of the ellipsoid. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the direction in which the diameter is large, the data varies a lot, while in the direction in which the diameter is small, the data varies litte. If you wanted to project N-d data into a 2-d scatter plot, you plot them along the two largest principal components, because with that approach you display most of the variance in the data.&lt;/p&gt;&#xA;" OwnerUserId="198" LastActivityDate="2010-07-20T06:47:14.180" CommentCount="1" />
  <row Id="235" PostTypeId="2" ParentId="216" CreationDate="2010-07-20T07:17:58.993" Score="4" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://raphaeljs.com/&quot; rel=&quot;nofollow&quot;&gt;RaphaelJS&lt;/a&gt; can do some pretty amazing stuff and it just got some major backing from &lt;a href=&quot;http://www.sencha.com/&quot; rel=&quot;nofollow&quot;&gt;Sencha&lt;/a&gt; (formerly ExtJS).  Raphael is pretty smart about browsers by using a VML backend for Internet Explorer and SVG for everything else.  However, the library is pretty low-level.  Fortunately, the author has started another project, &lt;a href=&quot;http://g.raphaeljs.com/&quot; rel=&quot;nofollow&quot;&gt;gRaphael&lt;/a&gt;, that focuses on drawing charts and graphs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;a href=&quot;http://simile.mit.edu/&quot; rel=&quot;nofollow&quot;&gt;MIT SIMILE Project&lt;/a&gt; also has some interesting JavaScript libraries:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.simile-widgets.org/timeplot/&quot; rel=&quot;nofollow&quot;&gt;Timeplot&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.simile-widgets.org/timeline/&quot; rel=&quot;nofollow&quot;&gt;Timeline&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;There is also a project to port Processing to JavaScript: &lt;a href=&quot;http://processingjs.org/&quot; rel=&quot;nofollow&quot;&gt;ProcessingJS&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://jmol.sourceforge.net/&quot; rel=&quot;nofollow&quot;&gt;Jmol&lt;/a&gt; is a Java applet for viewing chemical structures, but it is used as the display engine for 3D graphics in the &lt;a href=&quot;http://www.sagemath.org/&quot; rel=&quot;nofollow&quot;&gt;SAGE&lt;/a&gt; system, which has a completely browser-based GUI.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And for an open source alternative to Google Maps, there is the excellent &lt;a href=&quot;http://www.openlayers.org&quot; rel=&quot;nofollow&quot;&gt;OpenLayers&lt;/a&gt; JavaScript library which powers the frontend of the equally excellent &lt;a href=&quot;http://www.openstreetmap.org/&quot; rel=&quot;nofollow&quot;&gt;OpenStreetMap&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="13" LastEditorUserId="13" LastEditorDisplayName="" LastEditDate="2010-07-20T08:45:07.490" LastActivityDate="2010-07-20T08:45:07.490" />
  <row Id="236" PostTypeId="2" ParentId="224" CreationDate="2010-07-20T07:30:34.160" Score="0" ViewCount="" Body="&lt;p&gt;Unfortunately, it only runs on macs, but otherwise a great application (basically &lt;em&gt;Processing&lt;/em&gt; in python):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://nodebox.net/code/index.php/Home&quot; rel=&quot;nofollow&quot;&gt;http://nodebox.net/code/index.php/Home&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;NodeBox is a Mac OS X application that lets you create 2D visuals (static, animated or interactive) using Python programming code and export them as a PDF or a QuickTime movie. NodeBox is free and well-documented.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="138" LastEditorUserId="138" LastEditorDisplayName="" LastEditDate="2010-07-21T18:33:33.063" LastActivityDate="2010-07-21T18:33:33.063" CommunityOwnedDate="2010-10-12T08:48:39.990" />
  <row Id="237" PostTypeId="2" ParentId="173" CreationDate="2010-07-20T07:31:08.963" Score="1" ViewCount="" Body="&lt;p&gt;I'm going to leave the main question alone, because I think I will get it wrong (although I too analyse data for a healthcare provider, and to be honest, if I had these data, I would just analyse them using standard techniques and hope for the best, they look pretty okay to me).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for R packages, I have found the TSA library and it's accompanying &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0387759581&quot; rel=&quot;nofollow&quot;&gt;book&lt;/a&gt; very useful indeed. The &lt;code&gt;armasubsets&lt;/code&gt; command, particularly, I think is a great time saver.&lt;/p&gt;&#xA;" OwnerUserId="199" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-08-09T12:18:10.497" LastActivityDate="2010-08-09T12:18:10.497" />
  <row Id="238" PostTypeId="2" ParentId="224" CreationDate="2010-07-20T07:33:09.067" Score="8" ViewCount="" Body="&lt;p&gt;There is always lovely gnuplot:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.gnuplot.info/&quot; rel=&quot;nofollow&quot;&gt;http://www.gnuplot.info/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Gnuplot is a portable command-line driven graphing utility for linux, OS/2, MS Windows, OSX, VMS, and many other platforms. The source code is copyrighted but freely distributed (i.e., you don't have to pay for it). It was originally created to allow scientists and students to visualize mathematical functions and data interactively, but has grown to support many non-interactive uses such as web scripting. It is also used as a plotting engine by third-party applications like Octave. Gnuplot has been supported and under active development since 1986.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Gnuplot supports many types of plots in either 2D and 3D. It can draw using lines, points, boxes, contours, vector fields, surfaces, and various associated text. It also supports various specialized plot types. &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="138" LastActivityDate="2010-07-20T07:33:09.067" CommunityOwnedDate="2010-10-12T08:48:39.990" />
  <row Id="240" PostTypeId="2" ParentId="213" CreationDate="2010-07-20T07:55:30.107" Score="3" ViewCount="" Body="&lt;p&gt;My first response would be that if you can do multivariate regression on the data, then to use the residuals from that regression to spot outliers. (I know you said it's not a regression problem, so this might not help you, sorry !)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm copying some of this from a &lt;a href=&quot;http://stackoverflow.com/questions/1444306/how-to-use-outlier-tests-in-r-code/1444548#1444548&quot;&gt;Stackoverflow question I've previously answered&lt;/a&gt; which has some example &lt;a href=&quot;http://en.wikipedia.org/wiki/R_%28programming_language%29&quot; rel=&quot;nofollow&quot;&gt;R&lt;/a&gt; code&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, we'll create some data, and then taint it with an outlier;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; testout&amp;lt;-data.frame(X1=rnorm(50,mean=50,sd=10),X2=rnorm(50,mean=5,sd=1.5),Y=rnorm(50,mean=200,sd=25)) &#xA;&amp;gt; #Taint the Data &#xA;&amp;gt; testout$X1[10]&amp;lt;-5 &#xA;&amp;gt; testout$X2[10]&amp;lt;-5 &#xA;&amp;gt; testout$Y[10]&amp;lt;-530 &#xA;&#xA;&amp;gt; testout &#xA;         X1         X2        Y &#xA;1  44.20043  1.5259458 169.3296 &#xA;2  40.46721  5.8437076 200.9038 &#xA;3  48.20571  3.8243373 189.4652 &#xA;4  60.09808  4.6609190 177.5159 &#xA;5  50.23627  2.6193455 210.4360 &#xA;6  43.50972  5.8212863 203.8361 &#xA;7  44.95626  7.8368405 236.5821 &#xA;8  66.14391  3.6828843 171.9624 &#xA;9  45.53040  4.8311616 187.0553 &#xA;10  5.00000  5.0000000 530.0000 &#xA;11 64.71719  6.4007245 164.8052 &#xA;12 54.43665  7.8695891 192.8824 &#xA;13 45.78278  4.9921489 182.2957 &#xA;14 49.59998  4.7716099 146.3090 &#xA;&amp;lt;snip&amp;gt; &#xA;48 26.55487  5.8082497 189.7901 &#xA;49 45.28317  5.0219647 208.1318 &#xA;50 44.84145  3.6252663 251.5620 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It's often most usefull to examine the data graphically (you're brain is much better at spotting outliers than maths is)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; #Use Boxplot to Review the Data &#xA;&amp;gt; boxplot(testout$X1, ylab=&quot;X1&quot;) &#xA;&amp;gt; boxplot(testout$X2, ylab=&quot;X2&quot;) &#xA;&amp;gt; boxplot(testout$Y, ylab=&quot;Y&quot;) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can then use stats to calculate critical cut off values, here using the Lund Test (See Lund, R. E. 1975, &quot;Tables for An Approximate Test for Outliers in Linear Models&quot;, Technometrics, vol. 17, no. 4, pp. 473-476. and Prescott, P. 1975, &quot;An Approximate Test for Outliers in Linear Models&quot;, Technometrics, vol. 17, no. 1, pp. 129-132.)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; #Alternative approach using Lund Test &#xA;&amp;gt; lundcrit&amp;lt;-function(a, n, q) { &#xA;+ # Calculates a Critical value for Outlier Test according to Lund &#xA;+ # See Lund, R. E. 1975, &quot;Tables for An Approximate Test for Outliers in Linear Models&quot;, Technometrics, vol. 17, no. 4, pp. 473-476. &#xA;+ # and Prescott, P. 1975, &quot;An Approximate Test for Outliers in Linear Models&quot;, Technometrics, vol. 17, no. 1, pp. 129-132. &#xA;+ # a = alpha &#xA;+ # n = Number of data elements &#xA;+ # q = Number of independent Variables (including intercept) &#xA;+ F&amp;lt;-qf(c(1-(a/n)),df1=1,df2=n-q-1,lower.tail=TRUE) &#xA;+ crit&amp;lt;-((n-q)*F/(n-q-1+F))^0.5 &#xA;+ crit &#xA;+ } &#xA;&#xA;&amp;gt; testoutlm&amp;lt;-lm(Y~X1+X2,data=testout) &#xA;&#xA;&amp;gt; testout$fitted&amp;lt;-fitted(testoutlm) &#xA;&#xA;&amp;gt; testout$residual&amp;lt;-residuals(testoutlm) &#xA;&#xA;&amp;gt; testout$standardresid&amp;lt;-rstandard(testoutlm) &#xA;&#xA;&amp;gt; n&amp;lt;-nrow(testout) &#xA;&#xA;&amp;gt; q&amp;lt;-length(testoutlm$coefficients) &#xA;&#xA;&amp;gt; crit&amp;lt;-lundcrit(0.1,n,q) &#xA;&#xA;&amp;gt; testout$Ynew&amp;lt;-ifelse(testout$standardresid&amp;gt;crit,NA,testout$Y) &#xA;&#xA;&amp;gt; testout &#xA;         X1         X2        Y    newX1   fitted    residual standardresid &#xA;1  44.20043  1.5259458 169.3296 44.20043 209.8467 -40.5171222  -1.009507695 &#xA;2  40.46721  5.8437076 200.9038 40.46721 231.9221 -31.0183107  -0.747624895 &#xA;3  48.20571  3.8243373 189.4652 48.20571 203.4786 -14.0134646  -0.335955648 &#xA;4  60.09808  4.6609190 177.5159 60.09808 169.6108   7.9050960   0.190908291 &#xA;5  50.23627  2.6193455 210.4360 50.23627 194.3285  16.1075799   0.391537883 &#xA;6  43.50972  5.8212863 203.8361 43.50972 222.6667 -18.8306252  -0.452070155 &#xA;7  44.95626  7.8368405 236.5821 44.95626 223.3287  13.2534226   0.326339981 &#xA;8  66.14391  3.6828843 171.9624 66.14391 148.8870  23.0754677   0.568829360 &#xA;9  45.53040  4.8311616 187.0553 45.53040 214.0832 -27.0279262  -0.646090667 &#xA;10  5.00000  5.0000000 530.0000       NA 337.0535 192.9465135   5.714275585 &#xA;11 64.71719  6.4007245 164.8052 64.71719 159.9911   4.8141018   0.118618011 &#xA;12 54.43665  7.8695891 192.8824 54.43665 194.7454  -1.8630426  -0.046004311 &#xA;13 45.78278  4.9921489 182.2957 45.78278 213.7223 -31.4266180  -0.751115595 &#xA;14 49.59998  4.7716099 146.3090 49.59998 201.6296 -55.3205552  -1.321042392 &#xA;15 45.07720  4.2355525 192.9041 45.07720 213.9655 -21.0613819  -0.504406009 &#xA;16 62.27717  7.1518606 186.6482 62.27717 169.2455  17.4027250   0.430262983 &#xA;17 48.50446  3.0712422 228.3253 48.50446 200.6938  27.6314695   0.667366651 &#xA;18 65.49983  5.4609713 184.8983 65.49983 155.2768  29.6214506   0.726319931 &#xA;19 44.38387  4.9305222 213.9378 44.38387 217.7981  -3.8603382  -0.092354925 &#xA;20 43.52883  8.3777627 203.5657 43.52883 228.9961 -25.4303732  -0.634725264 &#xA;&amp;lt;snip&amp;gt; &#xA;49 45.28317  5.0219647 208.1318 45.28317 215.3075  -7.1756966  -0.171560291 &#xA;50 44.84145  3.6252663 251.5620 44.84145 213.1535  38.4084869   0.923804784 &#xA;       Ynew &#xA;1  169.3296 &#xA;2  200.9038 &#xA;3  189.4652 &#xA;4  177.5159 &#xA;5  210.4360 &#xA;6  203.8361 &#xA;7  236.5821 &#xA;8  171.9624 &#xA;9  187.0553 &#xA;10       NA &#xA;11 164.8052 &#xA;12 192.8824 &#xA;13 182.2957 &#xA;14 146.3090 &#xA;15 192.9041 &#xA;16 186.6482 &#xA;17 228.3253 &#xA;18 184.8983 &#xA;19 213.9378 &#xA;20 203.5657 &#xA;&amp;lt;snip&amp;gt; &#xA;49 208.1318 &#xA;50 251.5620 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Obviosuly there are other outlier tests than the Lund test (Grubbs springs to mind), but I'm not sure which are better suited to multivariate data.&lt;/p&gt;&#xA;" OwnerUserId="114" LastActivityDate="2010-07-20T07:55:30.107" CommentCount="1" />
  <row Id="241" PostTypeId="2" ParentId="213" CreationDate="2010-07-20T07:56:06.767" Score="4" ViewCount="" Body="&lt;p&gt;I'm not sure what you mean when you say you aren't thinking of a regression problem but of &quot;true multivariate data&quot;.  My initial response would be to calculate the Mahalanobis distance since it doesn't require that you specify a particular IV or DV, but at its core (as far as I understand it) it is related to a leverage statistic.&lt;/p&gt;&#xA;" OwnerUserId="196" LastActivityDate="2010-07-20T07:56:06.767" CommentCount="1" />
  <row Id="242" PostTypeId="1" AcceptedAnswerId="306" CreationDate="2010-07-20T07:56:16.297" Score="8" ViewCount="172" Body="&lt;p&gt;This is a bit of a flippant question, but I have a serious interest in the answer. I work in a psychiatric hospital and I have three years' of data, collected every day across each ward regarding the level of violence on that ward.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Clearly the model which fits these data is a time series model. I had to difference the scores in order to make them more normal. I fit an ARMA model with the differenced data, and the best fit I think was a model with one degree of differencing and first order auto-correlation at lag 2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is, what on earth can I use this model for? Time series always seems so useful in the textbooks when it's about hare populations and oil prices, but now I've done my own the result seems so abstract as to be completely opaque. The differenced scores correlate with each other at lag two, but I can't really advise everyone to be on high alert two days after a serious incident in all seriousness.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or can I?&lt;/p&gt;&#xA;" OwnerUserId="199" LastEditorUserId="5" LastEditorDisplayName="" LastEditDate="2010-07-20T15:53:21.083" LastActivityDate="2010-07-20T15:53:21.083" Title="Using time series analysis to analyze/predict violent behavior" Tags="&lt;time-series&gt;&lt;forecasting&gt;" AnswerCount="2" CommentCount="5" />
  <row Id="243" PostTypeId="2" ParentId="203" CreationDate="2010-07-20T08:05:04.060" Score="5" ViewCount="" Body="&lt;p&gt;Depending on the size of the dataset in question, a permutation test might be preferable to a bootstrap in that it may be able to provide an exact test of the hypothesis (and an exact CI).&lt;/p&gt;&#xA;" OwnerUserId="196" LastEditorUserId="196" LastEditorDisplayName="" LastEditDate="2010-07-20T09:09:01.443" LastActivityDate="2010-07-20T09:09:01.443" />
  <row Id="244" PostTypeId="2" ParentId="224" CreationDate="2010-07-20T08:13:31.763" Score="15" ViewCount="" Body="&lt;p&gt;The Visualization Tool Kit &lt;a href=&quot;http://www.vtk.org&quot; rel=&quot;nofollow&quot;&gt;VTK&lt;/a&gt; is pretty impressive for 3D visualizations of numerical data.  Unfortunately, it is also pretty low level.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://graphviz.org/&quot; rel=&quot;nofollow&quot;&gt;Graphviz&lt;/a&gt; is used pretty extensively for visualizing graphs and other tree-like data structures.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://igraph.sourceforge.net/&quot; rel=&quot;nofollow&quot;&gt;igraph&lt;/a&gt; can also be used for visualization of tree-like data structures.  Contains nice interfaces to scripting languages such as R and Python along with a stand-alone C library.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;a href=&quot;http://www.ncl.ucar.edu/&quot; rel=&quot;nofollow&quot;&gt;NCL&lt;/a&gt; (NCAR Command Language) library contains some pretty neat graphing routines- especially if you are looking at spatially distributed, multidimensional data such as wind fields.  Which makes sense as NCAR is the National Center for Atmospheric Research.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are willing to relax the executable requirement, or try a tool like &lt;a href=&quot;http://www.py2exe.org/&quot; rel=&quot;nofollow&quot;&gt;py2exe&lt;/a&gt;, there is the possibility of leveraging some neat Python libraries and applications such as:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://code.enthought.com/projects/mayavi/&quot; rel=&quot;nofollow&quot;&gt;MayaVi&lt;/a&gt;: A higher level front-end to VTK developed by Enthought.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://code.enthought.com/chaco/&quot; rel=&quot;nofollow&quot;&gt;Chaco&lt;/a&gt;: Another Enthought library focused on 2D graphs.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://matplotlib.sourceforge.net/&quot; rel=&quot;nofollow&quot;&gt;Matplotlib&lt;/a&gt;: Another 2D plotting library.  Has nice support for TeX-based mathematical annotation.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://sourceforge.net/projects/matplotlib/files/matplotlib-toolkits/&quot; rel=&quot;nofollow&quot;&gt;Basemap&lt;/a&gt;: An add-on to Matplotlib for drawing maps and displaying geographic data (&lt;a href=&quot;http://www.scipy.org/Cookbook/Matplotlib/Maps&quot; rel=&quot;nofollow&quot;&gt;sexy examples here&lt;/a&gt;).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If we were to bend the concept of &quot;standalone application&quot; even further to include PDF files, there are some neat graphics libraries available to LaTeX users:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://asymptote.sourceforge.net/&quot; rel=&quot;nofollow&quot;&gt;Asymptote&lt;/a&gt; can generate a variety of graphs, but its  crown jewel is definitely the ability to embed 3D graphs into PDF documents that can be manipulated (zoomed, rotated, animated, etc) by anyone using the Adobe Acrobat reader (&lt;a href=&quot;http://asymptote.sourceforge.net/gallery/3D%20graphs/helix.pdf&quot; rel=&quot;nofollow&quot;&gt;example&lt;/a&gt;).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://sourceforge.net/projects/pgf/&quot; rel=&quot;nofollow&quot;&gt;PGF/TikZ&lt;/a&gt; provides a wonderful vector drawing language to TeX documents.  The &lt;a href=&quot;http://tug.ctan.org/tex-archive/graphics/pgf/base/doc/generic/pgf/pgfmanual.pdf&quot; rel=&quot;nofollow&quot;&gt;manual&lt;/a&gt; is hands-down the most well-written, comprehensive and beautiful piece of documentation I have ever seen in an open source project.  &lt;a href=&quot;http://sourceforge.net/projects/pgfplots/&quot; rel=&quot;nofollow&quot;&gt;PGFPlots&lt;/a&gt; provides an abstraction layer for drawing plots.  A wondeful showcase can be found at &lt;a href=&quot;http://www.texample.net/tikz/examples/all/&quot; rel=&quot;nofollow&quot;&gt;TeXample&lt;/a&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://www.tug.org/PSTricks/main.cgi/&quot; rel=&quot;nofollow&quot;&gt;PSTricks&lt;/a&gt; served as an inspiration for TikZ and allows users to leverage the power of the PostScript language to create some neat graphics.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;And for kicks, there's &lt;a href=&quot;http://www.mps.mpg.de/dislin/&quot; rel=&quot;nofollow&quot;&gt;DISLIN&lt;/a&gt;, which has a native interface for &lt;code&gt;Fortran&lt;/code&gt;!  Not open source or free for commercial use though.&lt;/p&gt;&#xA;" OwnerUserId="13" LastEditorUserId="13" LastEditorDisplayName="" LastEditDate="2010-07-20T08:47:44.613" LastActivityDate="2010-07-20T08:47:44.613" CommunityOwnedDate="2010-10-12T08:48:39.990" CommentCount="4" />
  <row Id="245" PostTypeId="2" ParentId="222" CreationDate="2010-07-20T08:20:36.037" Score="1" ViewCount="" Body="&lt;p&gt;Principal component scores are a group of scores that are obtained following a Principle Components Analysis (PCA).  In PCA the relationships between a group of scores is analyzed such that an equal number of new &quot;imaginary&quot; variables (aka principle components) are created.  The first of these new imaginary variables is maximally correlated with all of the original group of variables.  The next is somewhat less correlated, and so forth until the point that if you used all of the principal components scores to predict any given variable from the initial group you would be able to explain all of its variance.  The way in which PCA proceeds is complex and has certain restrictions.  Among these is the restriction that the correlation between any two principal components (i.e. imaginary variables) is zero; thus it doesn't make sense to try to predict one principal component with another.&lt;/p&gt;&#xA;" OwnerUserId="196" LastActivityDate="2010-07-20T08:20:36.037" />
  <row Id="246" PostTypeId="2" ParentId="242" CreationDate="2010-07-20T08:28:52.710" Score="5" ViewCount="" Body="&lt;p&gt;You fitted the model to the differences, which means that you're describing the change in levels of violence. You get a lag of 2 days. A lag is indicative of the memory of the process. In other words, the change in levels of violence today has some dependency on the change in levels of violence in the last two days. For longer time-scales, the contribution of random influences becomes strong enough so that there is no clear link anymore.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is the auto-correlation positive? Then a change of levels of violence today suggests a similar change in levels of violence in two days. Is it negative? Then violence might stay higher for two days.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, you may want to have to control for confounding effects. For example, after a serious incident, people may be more likely to report minor incidents, but this &quot;sensitization&quot; would be going away after two days.&lt;/p&gt;&#xA;" OwnerUserId="198" LastActivityDate="2010-07-20T08:28:52.710" />
  <row Id="247" PostTypeId="2" ParentId="224" CreationDate="2010-07-20T08:36:36.217" Score="2" ViewCount="" Body="&lt;p&gt;Might be a bit narrow in scope, but if you're doing any work in Clojure on the JVM there's the excellent &lt;a href=&quot;http://incanter.org&quot; rel=&quot;nofollow&quot;&gt;Incanter&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Incanter is a Clojure-based, R-like platform for statistical computing and graphics.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="171" LastActivityDate="2010-07-20T08:36:36.217" CommunityOwnedDate="2010-10-12T08:48:39.990" CommentCount="1" />
  <row Id="248" PostTypeId="2" ParentId="196" CreationDate="2010-07-20T08:38:38.223" Score="3" ViewCount="" Body="&lt;p&gt;ggobi and the R links to Ggobi are really rather good for this.   There are simpler visualisations (iPlots is very nice, also interactive, as mentioned).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But it depends whether you are doing something more specialised.   For example TreeView lets you visualise the kind of cluster dendrograms you get out of microarrays.&lt;/p&gt;&#xA;" OwnerUserId="211" LastActivityDate="2010-07-20T08:38:38.223" CommunityOwnedDate="2010-07-20T08:38:38.223" />
  <row Id="249" PostTypeId="1" CreationDate="2010-07-20T08:49:13.050" Score="4" ViewCount="157" Body="&lt;p&gt;I have a set of &lt;em&gt;N&lt;/em&gt; bodies, which is a random sample from a population whose mean and variance I want to estimate. A property of each body is being measured m_i times (m_i&gt;1) and different for each body index i  identifies which body it is; the property is expected to be distributed around zero). I would like to describe the resulting measurement. Particularly I'm interested in average property value and in the variance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The average value is simple. First calculate the mean values for each body and then calculate the mean of means.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The variance is more tricky. There are two variances: the variance of measurement and the variance of property values. In order to have an idea on the confidence we have in any single measurement, we need to account for both the sources. Unfortunately, I can't think of a good method. It is obvious that putting all the numbers in a single pool and calculating the stdev of this pool isn't a good idea.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestion?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt;&#xA;Colin Gillespie suggests applying Random Effects Model. This model seems to be the right solution for my case, except for the fact that it is described (in Wikipedia) for the cases where each group (body in my case) is sampled equally (m_i is constant for all the bodies), which is not correct in my case&lt;/p&gt;&#xA;" OwnerUserId="213" LastEditorUserId="213" LastEditorDisplayName="" LastEditDate="2010-07-20T11:19:46.940" LastActivityDate="2010-07-20T11:19:46.940" Title="Variance components" Tags="&lt;standard-deviation&gt;&lt;variance&gt;&lt;anova&gt;&lt;random-effects-model&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="250" PostTypeId="2" ParentId="130" CreationDate="2010-07-20T08:53:15.833" Score="2" ViewCount="" Body="&lt;p&gt;Here's a data point: R has a &lt;a href=&quot;http://www.bytemining.com/2010/05/hitting-the-big-data-ceiling-in-r/&quot; rel=&quot;nofollow&quot;&gt;&quot;big data ceiling&quot;&lt;/a&gt;, useful to know if you plan on working with huge data sets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm unsure whether the same limitations apply to Clojure/Incanter, whether it outperforms R or is actually worse. I imagine the JVM can probably handle large datasets, especially if you manage to harness the power of Clojure's lazy features.&lt;/p&gt;&#xA;" OwnerUserId="171" LastActivityDate="2010-07-20T08:53:15.833" CommentCount="2" />
  <row Id="252" PostTypeId="2" ParentId="73" CreationDate="2010-07-20T09:13:29.427" Score="5" ViewCount="" Body="&lt;p&gt;For me personally, I use the following three packages the most, all available from the awesome &lt;a href=&quot;http://www.omegahat.org&quot; rel=&quot;nofollow&quot;&gt;Omega Project for Statistical Computing&lt;/a&gt; (I do not claim to be an expert, but for my purposes they are very easy to use):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;RCurl&lt;/strong&gt;: It has lots of options which allows access to websites that the default functions in base R would have difficulty with I think it's fair to say. It is an R-interface to the libcurl library, which has the &lt;em&gt;added&lt;/em&gt; benefit of a whole community outside of R developing it. Also available on &lt;a href=&quot;http://cran.r-project.org/web/packages/RCurl/index.html&quot; rel=&quot;nofollow&quot;&gt;CRAN&lt;/a&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;XML&lt;/strong&gt;: It is very forgiving of parsing malformed XML/HTML. It is an R-interface to the libxml2 library and again has the &lt;em&gt;added&lt;/em&gt; benefit of a whole community outside of R developing it Also available on &lt;a href=&quot;http://cran.r-project.org/web/packages/XML/index.html&quot; rel=&quot;nofollow&quot;&gt;CRAN&lt;/a&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;RJSONIO&lt;/strong&gt;: It allows one to parse the text returned from a json call and organise it into a list structure for further analysis.The competitor to this package is rjson but this one has the advantage of being vectorised, readily extensible through S3/S4, fast and scalable to large data.  &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="81" LastEditorUserId="81" LastEditorDisplayName="" LastEditDate="2010-07-28T10:02:05.600" LastActivityDate="2010-07-28T10:02:05.600" CommunityOwnedDate="2010-07-20T09:13:29.427" CommentCount="3" />
  <row Id="253" PostTypeId="2" ParentId="138" CreationDate="2010-07-20T09:26:05.113" Score="5" ViewCount="" Body="&lt;p&gt;If you're an economist/econometrician then Grant Farnworth's paper on using R is indispensable and is available on CRAN at:&#xA;&lt;a href=&quot;http://cran.r-project.org/doc/contrib/Farnsworth-EconometricsInR.pdf&quot; rel=&quot;nofollow&quot;&gt;http://cran.r-project.org/doc/contrib/Farnsworth-EconometricsInR.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="215" LastEditorUserId="215" LastEditorDisplayName="" LastEditDate="2010-07-20T11:37:25.383" LastActivityDate="2010-07-20T11:37:25.383" CommunityOwnedDate="2010-08-01T18:56:25.603" />
  <row Id="254" PostTypeId="2" ParentId="249" CreationDate="2010-07-20T09:30:18.797" Score="4" ViewCount="" Body="&lt;p&gt;I think if I understand your description correctly, you need to use a &lt;a href=&quot;http://en.wikipedia.org/wiki/Random_effects_model&quot; rel=&quot;nofollow&quot;&gt;linear mixed model&lt;/a&gt;. However, this maybe overkill, since these models are used to find differences between groups. For example, if you have two types of bodies and you wish to determine if they are different.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically, you have &lt;em&gt;between&lt;/em&gt; subject variation and &lt;em&gt;within&lt;/em&gt; subject variation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To fit these models in R, you can use the &lt;code&gt;lmer&lt;/code&gt; function from the &lt;code&gt;lme4&lt;/code&gt; library. So if I understand you correctly, your function will look something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#Load the R library&#xA;library(lme4)&#xA;&#xA;#data is a R data frame that contains your data&#xA;#measurement and Subject are variables&#xA;fm1 = lmer(measurement ~ (1|Subject), data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you are looking for differences between bodies, then it will look something like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;fm2 = lmer(measurement ~ body + (body|Subject), data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The command &lt;code&gt;summary(fm1)&lt;/code&gt; should give the values you are after.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are some resources that will help you get started:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://lme4.r-forge.r-project.org/&quot; rel=&quot;nofollow&quot;&gt;Documentation&lt;/a&gt; for the lme4 package&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://zoonek2.free.fr/UNIX/48_R/all.html&quot; rel=&quot;nofollow&quot;&gt;Statistics with R&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Most statistical software will be able to fit models of this type.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;BTW, the subject part is usually called the random effect. However, there a many different  views on what a random effect is. See Ch11.4 of &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0521867061&quot; rel=&quot;nofollow&quot;&gt;Data analysis using regression&lt;/a&gt; by Gelman and Hill for more details.&lt;/p&gt;&#xA;" OwnerUserId="8" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-07-20T10:50:24.037" LastActivityDate="2010-07-20T10:50:24.037" CommentCount="3" />
  <row Id="255" PostTypeId="2" ParentId="213" CreationDate="2010-07-20T09:30:51.697" Score="4" ViewCount="" Body="&lt;p&gt;It may be an overshoot, but you may train an unsupervised Random Forest on the data and use the object proximity measure to detect outliers. More details &lt;a href=&quot;http://www.stat.berkeley.edu/~breiman/RandomForests/cc_home.htm#outliers&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-07-20T09:30:51.697" CommentCount="1" />
  <row Id="256" PostTypeId="1" AcceptedAnswerId="265" CreationDate="2010-07-20T09:34:22.213" Score="4" ViewCount="241" Body="&lt;p&gt;What is the easiest way to understand boosting?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why doesn't it boost very weak classifiers &quot;to infinity&quot; (perfectness?) ?&lt;/p&gt;&#xA;" OwnerUserId="217" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-07-20T10:18:12.143" LastActivityDate="2011-01-18T18:44:34.727" Title="How boosting works?" Tags="&lt;machine-learning&gt;&lt;boosting&gt;" AnswerCount="3" FavoriteCount="3" />
  <row Id="257" PostTypeId="1" AcceptedAnswerId="262" CreationDate="2010-07-20T09:38:44.770" Score="7" ViewCount="850" Body="&lt;p&gt;We may assume that we have CSV file and we want a very basic line plot with several lines on one plot and a simple legend.&lt;/p&gt;&#xA;" OwnerUserId="217" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-08-17T05:27:15.433" LastActivityDate="2010-08-17T05:27:15.433" Title="What is the easiest way to create publication-quality plots under Linux?" Tags="&lt;data-visualization&gt;&lt;csv-file&gt;" AnswerCount="6" CommentCount="1" FavoriteCount="2" />
  <row Id="258" PostTypeId="1" CreationDate="2010-07-20T09:43:23.910" Score="7" ViewCount="133" Body="&lt;p&gt;Rules:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;one classifier per answer&lt;/li&gt;&#xA;&lt;li&gt;vote up if you agree &lt;/li&gt;&#xA;&lt;li&gt;downvote/remove duplicates.&lt;/li&gt;&#xA;&lt;li&gt;put your application in the comment&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="217" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-07-20T18:18:08.507" LastActivityDate="2011-03-30T11:35:21.860" CommunityOwnedDate="2010-07-20T09:43:23.910" Title="Poll: What is the best out-of-the-box 2-class classifier for your application?" Tags="&lt;machine-learning&gt;&lt;classification&gt;&lt;application&gt;" AnswerCount="7" FavoriteCount="2" />
  <row Id="259" PostTypeId="2" ParentId="258" CreationDate="2010-07-20T09:44:11.807" Score="5" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Support_vector_machine&quot; rel=&quot;nofollow&quot;&gt;Support vector machine&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="217" LastActivityDate="2010-07-20T09:44:11.807" CommunityOwnedDate="2010-07-20T09:44:11.807" CommentCount="5" />
  <row Id="260" PostTypeId="2" ParentId="258" CreationDate="2010-07-20T09:45:06.377" Score="4" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Random_forest&quot; rel=&quot;nofollow&quot;&gt;Random forest&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="217" LastActivityDate="2010-07-20T09:45:06.377" CommunityOwnedDate="2010-07-20T09:45:06.377" CommentCount="1" />
  <row Id="261" PostTypeId="2" ParentId="257" CreationDate="2010-07-20T09:45:18.933" Score="11" ViewCount="" Body="&lt;p&gt;It's hard to go past R for graphics. You could do what you want in 3 lines. For example, assuming the csv file has four columns:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x &amp;lt;- read.csv(&quot;file.csv&quot;)&#xA;matplot(x[,1],x[,2:4],type=&quot;l&quot;,col=1:3)&#xA;legend(&quot;topleft&quot;,legend=c(&quot;A&quot;,&quot;B&quot;,&quot;C&quot;),lty=1,col=1:3)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-07-20T09:45:18.933" CommentCount="4" />
  <row Id="262" PostTypeId="2" ParentId="257" CreationDate="2010-07-20T09:50:45.953" Score="12" ViewCount="" Body="&lt;p&gt;The easiest way is to use R &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Use &lt;code&gt;read.csv&lt;/code&gt; to enter the data into R, then use a combination of the &lt;code&gt;plot&lt;/code&gt; and &lt;code&gt;line&lt;/code&gt; commands&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want something really special, then look at the libraries &lt;a href=&quot;http://had.co.nz/ggplot2/&quot; rel=&quot;nofollow&quot;&gt;ggplot2&lt;/a&gt; or &lt;a href=&quot;http://lmdvr.r-forge.r-project.org/figures/figures.html&quot; rel=&quot;nofollow&quot;&gt;lattice&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In &lt;code&gt;ggplot2&lt;/code&gt; the following commands should get you started.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(ggplot2)&#xA;#You would use read.csv here&#xA;N = 10&#xA;d = data.frame(x=1:N,y1=runif(N),y2=rnorm(N), y3 = rnorm(N, 0.5))&#xA;p = ggplot(d)&#xA;&#xA;p = p+geom_line(aes(x, y1, colour=&quot;Type 1&quot;))&#xA;p = p+geom_line(aes(x, y2, colour=&quot;Type 2&quot;))&#xA;p = p+geom_line(aes(x, y3, colour=&quot;Type 3&quot;))&#xA;#Add points&#xA;p = p+geom_point(aes(x, y3, colour=&quot;Type 3&quot;))&#xA;print(p)   &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This would give you the following plot:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://img84.imageshack.us/img84/6393/tmpq.jpg&quot; alt=&quot;Line plot&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Saving plots in R&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Saving plots in R is straightforward:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#Look at ?jpeg to other different saving options&#xA;jpeg(&quot;figure.jpg&quot;)&#xA;print(p)#for ggplot2 graphics&#xA;dev.off()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Instead of &lt;code&gt;jpeg&lt;/code&gt;'s you can also save as a &lt;code&gt;pdf&lt;/code&gt; or postscript file:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#This example uses R base graphics&#xA;#Just change to print(p) for ggplot2&#xA;pdf(&quot;figure.pdf&quot;)&#xA;plot(d$x,y1, type=&quot;l&quot;)&#xA;lines(d$x, y2)&#xA;dev.off()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="8" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-07-20T14:48:45.743" LastActivityDate="2010-07-20T14:48:45.743" CommentCount="7" />
  <row Id="263" PostTypeId="2" ParentId="257" CreationDate="2010-07-20T09:53:11.583" Score="8" ViewCount="" Body="&lt;p&gt;My favorite tool is Python with &lt;a href=&quot;http://matplotlib.sourceforge.net/&quot; rel=&quot;nofollow&quot;&gt;mathplotlib&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The advantages:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Immediate export from the environment where I do my experiments in&lt;/li&gt;&#xA;&lt;li&gt;Support for the scipy/numpy data structures&lt;/li&gt;&#xA;&lt;li&gt;Familiar syntax/options (matlab background)&lt;/li&gt;&#xA;&lt;li&gt;Full latex support for labels/legends etc. So same typesetting as in the rest of your document!&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Specifically, for different file formats like svg and eps, use the format parameter of &lt;a href=&quot;http://matplotlib.sourceforge.net/api/pyplot_api.html#matplotlib.pyplot.savefig&quot; rel=&quot;nofollow&quot;&gt;savefig&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An example:&#xA;input.csv&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&quot;Line 1&quot;,0.5,0.8,1.0,0.9,0.9&#xA;&quot;Line 2&quot;,0.2,0.7,1.2,1.1,1.1&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import csv&#xA;import matplotlib.pyplot as plt&#xA;&#xA;legends = []&#xA;for row in csv.reader(open('input.csv')):&#xA;    legends.append(row[0])&#xA;    plt.plot(row[1:])&#xA;&#xA;plt.legend(legends)&#xA;plt.savefig(&quot;out.svg&quot;, format='svg')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="190" LastEditorUserId="190" LastEditorDisplayName="" LastEditDate="2010-07-20T14:14:17.293" LastActivityDate="2010-07-20T14:14:17.293" CommentCount="5" />
  <row Id="264" PostTypeId="2" ParentId="134" CreationDate="2010-07-20T09:59:00.690" Score="4" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://www.ligo.caltech.edu/docs/T/T030168-00.pdf&quot; rel=&quot;nofollow&quot;&gt;Here&lt;/a&gt; is an article describing one possible algorith. Source code included and a quite serious application (gravitational wave detection based on laser interferometry), so you can expect it to be well tested.&lt;/p&gt;&#xA;" OwnerUserId="217" LastActivityDate="2010-07-20T09:59:00.690" CommentCount="0" />
  <row Id="265" PostTypeId="2" ParentId="256" CreationDate="2010-07-20T10:05:48.050" Score="6" ViewCount="" Body="&lt;p&gt;In plain English: If your classifier misclassifies some data, train another copy of it mainly on this misclassified part with hope that it will discover something subtle. And then, as usual, iterate. On the way there are some voting schemes that allow to combine all those classifiers' predictions in sensible way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because sometimes it is impossible (the noise is just hiding some of the information, or it is not even present in the data); on the other hand, boosting too much may lead to overfitting.&lt;/p&gt;&#xA;" OwnerUserId="88" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-07-20T10:17:16.153" LastActivityDate="2010-07-20T10:17:16.153" />
  <row Id="266" PostTypeId="2" ParentId="134" CreationDate="2010-07-20T10:23:56.707" Score="3" ViewCount="" Body="&lt;p&gt;As you mentioned sorting would be &lt;code&gt;O(n·log n)&lt;/code&gt; for a window of length &lt;code&gt;n&lt;/code&gt;. Doing this moving adds another &lt;code&gt;l=vectorlength&lt;/code&gt; making the total cost &lt;code&gt;O(l·n·log n)&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The simplest way to push this is by keeping an ordered list of the last n elements in memory when moving from one window to the next one. As removing/inserting one element from/into an ordered list are both &lt;code&gt;O(n)&lt;/code&gt; this would result in costs of &lt;code&gt;O(l·n)&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Pseudocode:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;l = length(input)&#xA;aidvector = sort(input(1:n))&#xA;output(i) = aid(n/2)&#xA;for i = n+1:l&#xA;    remove input(i-n) from aidvector&#xA;    sort aid(n) into aidvector&#xA;    output(i) = aid(n/2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="128" LastActivityDate="2010-07-20T10:23:56.707" />
  <row Id="267" PostTypeId="1" AcceptedAnswerId="271" CreationDate="2010-07-20T10:35:55.960" Score="2" ViewCount="415" Body="&lt;p&gt;If I have two lists A and B, both of which are subsets of a much larger list C, how can I determine if the degree of overlap of A and B is greater than I would expect by chance?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should I just randomly select elements from C of the same lengths as lists A and B and determine that random overlap, and do this many times to determine some kind or empirical p-value? Is there a better way to test this?&lt;/p&gt;&#xA;" OwnerUserId="194" LastActivityDate="2010-07-20T11:10:42.150" Title="How do I calculate if the degree of overlap between two lists is significant?" Tags="&lt;statistical-significance&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="1" />
  <row Id="268" PostTypeId="2" ParentId="47" CreationDate="2010-07-20T11:01:06.313" Score="2" ViewCount="" Body="&lt;p&gt;You could of for a supervised self-organizing map (e.g. with &lt;a href=&quot;http://cran.r-project.org/web/packages/kohonen/index.html&quot; rel=&quot;nofollow&quot;&gt;kohonen&lt;/a&gt; package for R), and use the login frequency as dependent variable. That way, the clustering will focus on separating the frequent visitors from the rare visitors. By plotting the number of users on each map unit, you may get an idea in clusters present in your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because SOMs are non-linear mapping methods, this approach is particularly interesting for tailed data.&lt;/p&gt;&#xA;" OwnerUserId="107" LastActivityDate="2010-07-20T11:01:06.313" CommentCount="1" />
  <row Id="269" PostTypeId="1" AcceptedAnswerId="416" CreationDate="2010-07-20T11:07:42.403" Score="6" ViewCount="1250" Body="&lt;p&gt;What is the difference between a population and a sample? What common variables and statistics are used for each one, and how do those relate to each other? &lt;/p&gt;&#xA;" OwnerUserId="62" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-08-07T17:55:39.090" LastActivityDate="2010-08-07T17:55:39.090" Title="What is the difference between a population and a sample?" Tags="&lt;standard-deviation&gt;&lt;variance&gt;&lt;sample&gt;&lt;population&gt;" AnswerCount="3" FavoriteCount="2" />
  <row Id="270" PostTypeId="1" AcceptedAnswerId="279" CreationDate="2010-07-20T11:08:47.770" Score="5" ViewCount="235" Body="&lt;p&gt;Due to the factorial in a poisson distribution, it becomes unpractical to estimate poisson models (for example, using maximum likelihood) when the observations are large. So, for example, if I am trying to estimate a model to explain the number of suicides in a given year (only annual data are available), and say, there are thousands of suicides every year, is it wrong to express suicides in hundreds, so that 2998 would be 29.98 ~= 30? In other words, is it wrong to change the unit of measurement to make the data manageable? &lt;/p&gt;&#xA;" OwnerUserId="90" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-10-08T16:05:24.073" LastActivityDate="2010-10-08T16:05:24.073" Title="Poisson regression with large data: is it wrong to change the unit of measurement?" Tags="&lt;modeling&gt;&lt;poisson&gt;&lt;large-data&gt;" AnswerCount="4" />
  <row Id="271" PostTypeId="2" ParentId="267" CreationDate="2010-07-20T11:10:42.150" Score="4" ViewCount="" Body="&lt;p&gt;If I understand your question correctly, you need to use the &lt;a href=&quot;http://en.wikipedia.org/wiki/Hypergeometric_distribution&quot; rel=&quot;nofollow&quot;&gt;Hypergeometric distribution&lt;/a&gt;. This distribution is usually associated with urn models, i.e there are n balls in an urn, &lt;em&gt;y&lt;/em&gt; are painted red, and you draw &lt;em&gt;m&lt;/em&gt; balls from the urn. Then if &lt;em&gt;X&lt;/em&gt; is the number of balls in your sample of &lt;em&gt;m&lt;/em&gt; that are red, &lt;em&gt;X&lt;/em&gt; has a hyper-geometric distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For your specific example, let n_A, n_B and n_C denote the lengths of your three lists and let n_A_B denote the overlap between A and B. Then &lt;/p&gt;&#xA;&#xA;&lt;p&gt;n_A_B ~ HG(n_A, n_C, n_B)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To calculate a p-value, you could use this R command: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#Some example values&#xA;n_A = 100;n_B = 200; n_C = 500; n_A_B = 50&#xA;1-phyper(n_A_B, n_B, n_C-n_B, n_A)&#xA;[1] 0.008626697&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Word of caution. Remember multiple testing, i.e. if you have lots of &lt;em&gt;A&lt;/em&gt; and &lt;em&gt;B&lt;/em&gt; lists, then you will need to adjust your p-values with a correction. For the example the FDR or Bonferroni corrections.&lt;/p&gt;&#xA;" OwnerUserId="8" LastActivityDate="2010-07-20T11:10:42.150" />
  <row Id="272" PostTypeId="2" ParentId="161" CreationDate="2010-07-20T11:14:41.487" Score="2" ViewCount="" Body="&lt;p&gt;Also, for some elaborate discussion (including bashing of ADF / PP / KPSS :) you might want to have a look at the book by Maddala and Kim:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0521587824&quot; rel=&quot;nofollow&quot;&gt;http://www.amazon.com/Cointegration-Structural-Change-Themes-Econometrics/dp/0521587824&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Quite extensive and not very easy to read sometimes, but a useful reference.&lt;/p&gt;&#xA;" OwnerUserId="216" LastActivityDate="2010-07-20T11:14:41.487" />
  <row Id="273" PostTypeId="2" ParentId="25" CreationDate="2010-07-20T11:18:01.390" Score="1" ViewCount="" Body="&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Clearly R&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;RadidMiner is nice, but switching to thinking in terms of operators takes a moment&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Matlab / Octave&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If you describe a specific problem, I may be able to get more specific.&lt;/p&gt;&#xA;" OwnerUserId="216" LastActivityDate="2010-07-20T11:18:01.390" />
  <row Id="274" PostTypeId="2" ParentId="269" CreationDate="2010-07-20T11:21:59.493" Score="9" ViewCount="" Body="&lt;p&gt;The population is the whole set of values, or individuals, you are interested in. The sample is a subset of the population, and is the set of values you actually use in your estimation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, for example, if you want to know the average height of the residents of China, that is your population, ie, the population of China. The thing is, this is quite large a number, and you wouldn't be able to get data for everyone there. So you draw a sample, that is, you get some observations, or the height of some of the people in China (a subset of the population, the sample) and do your inference based on that. &lt;/p&gt;&#xA;" OwnerUserId="90" LastActivityDate="2010-07-20T11:21:59.493" CommentCount="3" />
  <row Id="275" PostTypeId="2" ParentId="270" CreationDate="2010-07-20T11:29:53.070" Score="3" ViewCount="" Body="&lt;p&gt;In case of Poisson it is bad, since counts are counts -- their unit is an unity. On the other hand, if you'd use some advanced software like R, its Poisson handling functions will be aware of such large numbers and would use some numerical tricks to handle them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Obviously I agree that normal approximation is another good approach.&lt;/p&gt;&#xA;" OwnerUserId="88" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-07-20T12:26:55.800" LastActivityDate="2010-07-20T12:26:55.800" />
  <row Id="276" PostTypeId="1" AcceptedAnswerId="553" CreationDate="2010-07-20T11:43:25.013" Score="6" ViewCount="192" Body="&lt;p&gt;Is there a rule-of thumb or even any way at all to tell how large a sample should be in order to estimate a model with a given number of parameters? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, for example, if I want to estimate a least-squares regression with 5 parameters, how large should the sample be? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does it matter what estimation technique you are using (e.g. maximum likelihood, least squares, GMM), or how many or what tests you are going to perform? Should the sample variability be taken into account when making the decision?&lt;/p&gt;&#xA;" OwnerUserId="90" LastActivityDate="2010-09-16T22:25:36.037" Title="How large should a sample be for a given estimation technique and parameters?" Tags="&lt;sample-size&gt;&lt;estimation&gt;&lt;least-squares&gt;&lt;maximum-likelihood&gt;" AnswerCount="4" />
  <row Id="277" PostTypeId="1" CreationDate="2010-07-20T11:49:02.490" Score="8" ViewCount="361" Body="&lt;p&gt;When would one prefer to use a Conditional Autoregressive model over a Simultaneous Autoregressive model when modelling autocorrelated geo-referenced areal data?&lt;/p&gt;&#xA;" OwnerUserId="215" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-26T08:51:44.677" LastActivityDate="2011-04-26T08:51:44.677" Title="Spatial statistics models -- CAR vs SAR" Tags="&lt;modeling&gt;&lt;spatial&gt;" AnswerCount="2" FavoriteCount="3" />
  <row Id="278" PostTypeId="1" CreationDate="2010-07-20T11:49:27.543" Score="6" ViewCount="71" Body="&lt;p&gt;When a non-hierarchical cluster analysis is carried out, the order of observations in the data file determine the clustering results, especially if the data set is small (i.e, 5000 observations). To deal with this problem I usually performed a random reorder of data observations. My problem is that if I replicate the analysis n times, the results obtained are different and sometimes these differences are great. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I deal with this problem? Maybe I could run the analysis several times and after consider that one observation belong to the group in which more times was assigned. Has someone a better approach to this problem?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Manuel Ramon&lt;/p&gt;&#xA;" OwnerUserId="221" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-17T20:36:37.643" LastActivityDate="2010-09-17T20:36:37.643" Title="How to deal with the effect of the order of observations in a non hierarchical cluster analysis?" Tags="&lt;clustering&gt;" AnswerCount="3" CommentCount="1" FavoriteCount="1" />
  <row Id="279" PostTypeId="2" ParentId="270" CreationDate="2010-07-20T11:54:15.197" Score="4" ViewCount="" Body="&lt;p&gt;When you're dealing with a Poisson distribution with large values of \lambda (its parameter), it is common to use a normal approximation to the Poisson distribution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As &lt;a href=&quot;http://www.stat.ucla.edu/~dinov/courses_students.dir/Applets.dir/NormalApprox2PoissonApplet.html&quot; rel=&quot;nofollow&quot;&gt;this site&lt;/a&gt; mentions, it's all right to use the normal approximation when \lambda gets over 20, and the approximation improves as \lambda gets even higher. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Poisson distribution is defined only over the state space consisting of the non-negative integers, so rescaling and rounding is going to introduce odd things into your data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using the normal approx. for large Poisson statistics is VERY common.&lt;/p&gt;&#xA;" OwnerUserId="62" LastActivityDate="2010-07-20T11:54:15.197" />
  <row Id="280" PostTypeId="2" ParentId="138" CreationDate="2010-07-20T11:54:50.530" Score="2" ViewCount="" Body="&lt;p&gt;The &lt;a href=&quot;http://www.r-project.org/&quot; rel=&quot;nofollow&quot;&gt;R project&lt;/a&gt; website has lots of manuals to start, and I suggest you the &lt;a href=&quot;http://r.789695.n4.nabble.com/&quot; rel=&quot;nofollow&quot;&gt;Nabble R forum&lt;/a&gt; and the &lt;a href=&quot;http://www.r-bloggers.com/&quot; rel=&quot;nofollow&quot;&gt;R-bloggers&lt;/a&gt; site as well. &lt;/p&gt;&#xA;" OwnerUserId="221" LastActivityDate="2010-07-20T11:54:50.530" CommunityOwnedDate="2010-08-01T18:56:25.603" />
  <row Id="281" PostTypeId="2" ParentId="73" CreationDate="2010-07-20T11:59:56.727" Score="3" ViewCount="" Body="&lt;p&gt;Day-to-day the most useful package must be &quot;foreign&quot; which has functions for reading and writing data for other statistical packages e.g. Stata, SPSS, Minitab, SAS, etc. Working in a field where R is not that commonplace means that this is a very important package.&lt;/p&gt;&#xA;" OwnerUserId="215" LastActivityDate="2010-07-20T11:59:56.727" CommunityOwnedDate="2010-07-20T11:59:56.727" />
  <row Id="282" PostTypeId="2" ParentId="222" CreationDate="2010-07-20T12:02:26.817" Score="8" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;First, lets define a score:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;John, Mike and Kate get the following percentages for exams in Maths, Science, English and Music as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;      Maths    Science    English    Music    &#xA;John  80        85          60       55  &#xA;Mike  90        85          70       45&#xA;Kate  95        80          40       50&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In this case there are 12 scores in total. Each &lt;strong&gt;score&lt;/strong&gt; represents the exam results for each person in a particular subject. So a score in this case is simply a representation of where a row and column intersect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Now lets informally define a Principal Component:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the table above, can you easily plot the data in a 2D graph? No, because there are four subjects (which means four variables), i.e.:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;You could plot two subjects in the exact same way you would with x &amp;amp; y co-ordinates in a 2D graph. &lt;/li&gt;&#xA;&lt;li&gt;You could even plot three subjects in the same way you would plot x, y &amp;amp; z in a 3D graph (though this is generally bad practice). &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;But how would you plot 4 subjects?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At the moment we have four variables which each represent just one subject. So a method around this might be to somehow combine the subjects into maybe just two new variables which we can then plot. This is known as &lt;em&gt;Multidimensional scaling&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Principal Component analysis is a form of multidimensional scaling. It is a linear transformation of the variables into a lower dimensional space which retain maximal amount of information about the variables. For example, this would mean we could look at the types of subjects each student is maybe more suited to.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A principle Component is therefore a combination of the original variables after a linear transformation. In &lt;strong&gt;R&lt;/strong&gt;, this is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;DF&amp;lt;-data.frame(Maths=c(80, 90, 95), Science=c(85, 85, 80), English=c(60, 70, 40), Music=c(55, 45, 50))&#xA;prcomp(DF, scale = FALSE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Which will give you something like this (first two Principle Components only for sake of simplicity):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;                PC1         PC2&#xA;Maths    0.27795606  0.76772853 &#xA;Science -0.17428077 -0.08162874 &#xA;English -0.94200929  0.19632732 &#xA;Music    0.07060547 -0.60447104 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;So what is a Principal Component Score?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's a score from the table at the end of this post.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The output from &lt;strong&gt;R&lt;/strong&gt; means we can now plot each persons score across all subjects in a 2D graph as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;      x                                       y&#xA;John 0.28*80 + -0.17*90 + -0.94*60 + 0.07*55  0.77*80 + -0.08*85 + 0.19*60 + -0.60*55 &#xA;Mike 0.28*90 + -0.17*85 + -0.94*70 + 0.07*45  0.77*90 + -0.08*85 + 0.19*70 + -0.60*45&#xA;Kate 0.28*95 + -0.17*80 + -0.94*40 + 0.07*50  0.77*95 + -0.08*80 + 0.19*40 + -0.60*50&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Which simplifies to:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;      x       y&#xA;John  -45.45  33.2&#xA;Mike  -51.9   48.8&#xA;Kate  -21.1   44.35&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;There are &lt;strong&gt;&lt;em&gt;six principle component scores&lt;/em&gt;&lt;/strong&gt; in the table above. You can now plot the scores in a 2D graph to get a sense of the type of subjects each student is perhaps more suited to.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT 1: Hmm, I probably could have thought up a better example, and there is more to it than what I've put here, but I hope you get the idea.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT 2: full credit to @drpaulbrewer for his comment in improving this answer.&lt;/p&gt;&#xA;" OwnerUserId="81" LastEditorUserId="81" LastEditorDisplayName="" LastEditDate="2010-08-01T09:30:24.463" LastActivityDate="2010-08-01T09:30:24.463" CommentCount="3" />
  <row Id="283" PostTypeId="1" AcceptedAnswerId="307" CreationDate="2010-07-20T12:09:08.457" Score="7" ViewCount="1344" Body="&lt;p&gt;What is meant when we say we have a saturated model?&lt;/p&gt;&#xA;" OwnerUserId="215" LastActivityDate="2010-07-27T13:09:54.817" Title="What is a &quot;saturated&quot; model?" Tags="&lt;modeling&gt;&lt;regression&gt;" AnswerCount="4" />
  <row Id="284" PostTypeId="2" ParentId="50" CreationDate="2010-07-20T12:25:30.920" Score="1" ViewCount="" Body="&lt;p&gt;A random variable, usually denoted X, is a variable where the outcome is uncertain. The observation of a particular outcome of this variable is called a realisation. More concretely, it is a function which maps a probability space into a measurable space, usually called a state space. Random variables are discrete (can take a number of distinct values) or continuous (can take an infinite number of values). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider the random variable X which is the total obtained when rolling two dice. It can take any of the values 2-12 (with equal probability given fair dice) and the outcome is uncertain until the dice are rolled. &lt;/p&gt;&#xA;" OwnerUserId="215" LastActivityDate="2010-07-20T12:25:30.920" />
  <row Id="285" PostTypeId="2" ParentId="47" CreationDate="2010-07-20T12:26:39.483" Score="0" ViewCount="" Body="&lt;p&gt;You might consider transforming (perhaps a log) the positively skewed variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If after exploring various clustering algorithms you find that the four variables simply reflect varying intensity levels of usage, you might think about a theoretically based classification. Presumably this classification is going to be used for a purpose and that purpose could drive meaningful cut points on one or more of the variables.&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-07-20T12:26:39.483" CommentCount="1" />
  <row Id="286" PostTypeId="2" ParentId="170" CreationDate="2010-07-20T12:28:32.410" Score="3" ViewCount="" Body="&lt;p&gt;There's a superb Probability book here:&#xA;&lt;a href=&quot;http://www.dartmouth.edu/~chance/teaching_aids/books_articles/probability_book/book.html&quot; rel=&quot;nofollow&quot;&gt;http://www.dartmouth.edu/~chance/teaching_aids/books_articles/probability_book/book.html&lt;/a&gt;&#xA;which you can also buy in hardcopy.;&lt;/p&gt;&#xA;" OwnerUserId="211" LastActivityDate="2010-07-20T12:28:32.410" />
  <row Id="287" PostTypeId="1" AcceptedAnswerId="506" CreationDate="2010-07-20T12:29:17.607" Score="3" ViewCount="169" Body="&lt;p&gt;Can someone explain to me the difference between method of moments and GMM (general method of moments), their relationship, and when should one or the other be used?&lt;/p&gt;&#xA;" OwnerUserId="90" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-07-30T11:38:17.293" LastActivityDate="2010-07-30T11:38:17.293" Title="What is the difference/relationship between method of moments and GMM?" Tags="&lt;estimation&gt;&lt;gmm&gt;&lt;method-of-moments&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="288" PostTypeId="1" CreationDate="2010-07-20T12:29:34.187" Score="5" ViewCount="352" Body="&lt;p&gt;Suppose that I culture cancer cells in &lt;em&gt;n&lt;/em&gt; different dishes &lt;em&gt;g₁&lt;/em&gt;, &lt;em&gt;g₂&lt;/em&gt;, … , &lt;em&gt;g&lt;sub&gt;n&lt;/sub&gt;&lt;/em&gt; and observe the number of cells &lt;em&gt;n&lt;sub&gt;i&lt;/sub&gt;&lt;/em&gt; in each dish that look different than normal.  The total number of cells in dish &lt;em&gt;g&lt;sub&gt;i&lt;/sub&gt;&lt;/em&gt; is &lt;em&gt;t&lt;sub&gt;i&lt;/sub&gt;&lt;/em&gt;.  There is individual differences between individual cells, but also differences between the populations in different dishes because each dish has a slightly different temperature, amount of liquid, and so on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I model this as a beta-binomial distribution: &lt;em&gt;n&lt;sub&gt;i&lt;/sub&gt;&lt;/em&gt; ~ Binomial(&lt;em&gt;p&lt;sub&gt;i&lt;/sub&gt;&lt;/em&gt;, &lt;em&gt;t&lt;sub&gt;i&lt;/sub&gt;&lt;/em&gt;) where &lt;em&gt;p&lt;sub&gt;i&lt;/sub&gt;&lt;/em&gt; ~ Beta(&lt;em&gt;α&lt;/em&gt;, &lt;em&gt;β&lt;/em&gt;).  Given a number of observations of &lt;em&gt;n&lt;sub&gt;i&lt;/sub&gt;&lt;/em&gt; and &lt;em&gt;t&lt;sub&gt;i&lt;/sub&gt;&lt;/em&gt;, how can I estimate &lt;em&gt;α&lt;/em&gt; and &lt;em&gt;β&lt;/em&gt;?&lt;/p&gt;&#xA;" OwnerUserId="220" LastActivityDate="2010-07-26T21:13:43.743" Title="Estimating beta-binomial distribution" Tags="&lt;estimation&gt;&lt;beta-binomial&gt;" AnswerCount="3" FavoriteCount="1" />
  <row Id="289" PostTypeId="2" ParentId="224" CreationDate="2010-07-20T12:32:41.060" Score="3" ViewCount="" Body="&lt;p&gt;For visualizing graphs in a Java/SWT environment, check out Zest: &lt;a href=&quot;http://eclipse.org/gef/zest&quot; rel=&quot;nofollow&quot;&gt;http://eclipse.org/gef/zest&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="80" LastActivityDate="2010-07-20T12:32:41.060" CommunityOwnedDate="2010-10-12T08:48:39.990" />
  <row Id="290" PostTypeId="1" AcceptedAnswerId="445" CreationDate="2010-07-20T12:33:30.577" Score="5" ViewCount="177" Body="&lt;p&gt;I know of Cameron and Trivedi's Microeconometrics Using Stata. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are other good text for learning the Stata? &lt;/p&gt;&#xA;" OwnerUserId="189" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-03-13T16:19:54.677" LastActivityDate="2011-03-17T00:16:14.973" Title="Resources for learning Stata" Tags="&lt;books&gt;&lt;stata&gt;" AnswerCount="6" CommentCount="1" FavoriteCount="2" />
  <row Id="291" PostTypeId="2" ParentId="7" CreationDate="2010-07-20T12:35:43.100" Score="6" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://lib.stat.cmu.edu/DASL/&quot; rel=&quot;nofollow&quot;&gt;http://lib.stat.cmu.edu/DASL/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="211" LastActivityDate="2010-07-20T12:35:43.100" />
  <row Id="292" PostTypeId="2" ParentId="276" CreationDate="2010-07-20T12:38:38.663" Score="2" ViewCount="" Body="&lt;p&gt;It should always be large enough! ;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All parameter estimates come with an estimate uncertainty, which is determined by the sample size. If you carry out a regression analysis, it helps to remind yourself that the &amp;Chi;&lt;sup&gt;2&lt;/sup&gt; distribution is constructed from the input data set. If your model had 5 parameters and you had 5 data points, you would only be able to calculate a single point of the &amp;Chi;&lt;sup&gt;2&lt;/sup&gt; distribution. Since you will need to minimize it, you could only pick that one point as a guess for the minimum, but would have to assign infinite errors to your estimated parameters. Having more data points would allow you to map the parameter space better leading to a better estimate of the minimum of the &amp;Chi;&lt;sup&gt;2&lt;/sup&gt; distribution and thus smaller estimator errors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would you be using a Maximum Likelihood estimator instead the situation would be similar: More data points leads to better estimate of the minimum.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for point variance, you would need to model this as well. Having more data points would make clustering of points around the &quot;true&quot; value more obvious (due to the Central Limit Theorem) and the danger of interpreting a large, chance flucuation as the true value for that point would go down. And as for any other parameter your estimate for the point variance would become more stable the more data points you have.&lt;/p&gt;&#xA;" OwnerUserId="56" LastActivityDate="2010-07-20T12:38:38.663" />
  <row Id="293" PostTypeId="2" ParentId="192" CreationDate="2010-07-20T12:43:02.087" Score="4" ViewCount="" Body="&lt;p&gt;I think you need to rework this question.   It all depends on the problem/data which has generated the cross-tab.   &lt;/p&gt;&#xA;" OwnerUserId="211" LastActivityDate="2010-07-20T12:43:02.087" CommentCount="1" />
  <row Id="294" PostTypeId="2" ParentId="53" CreationDate="2010-07-20T12:47:39.550" Score="12" ViewCount="" Body="&lt;p&gt;Bernard Flury, in his excellent book introducing multivariate analysis, described this as an anti-property of principal components.   It's actually worse than choosing between correlation or covariance.   If you changed the units (e.g. US style gallons, inches etc. and EU style litres, centimetres) you will get substantively different projections of the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The argument against automatically using correlation matrices is that it is quite a brutal way of standardising your data.   The problem with automatically using the covariance matrix, which is very apparent with that heptathalon data, is that the variables with the highest variance will dominate the first principal component (the variance maximising property).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the &quot;best&quot; method to use is based on a subjective choice, careful thought and some experience.&lt;/p&gt;&#xA;" OwnerUserId="211" LastActivityDate="2010-07-20T12:47:39.550" />
  <row Id="295" PostTypeId="2" ParentId="31" CreationDate="2010-07-20T12:52:55.920" Score="2" ViewCount="" Body="&lt;p&gt;A nice definition of p-value is &quot;the probability of observing a test statistic at least as large as the one calculated assuming the null hypothesis is true&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem with that is that it requires an understanding of &quot;test statistic&quot; and &quot;null hypothesis&quot;. But, that's easy to get across. If the null hypothesis is true, usually something like &quot;parameter from population A is equal to parameter from population B&quot;, and you calculate statistics to estimate those parameters, what is the probability of seeing a test statistic that says, &quot;they're this different&quot;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;E.g., If the coin is fair, what is the probability I'd see 60 heads out of 100 tosses? That's testing the null hypothesis, &quot;the coin is fair&quot;, or &quot;p = .5&quot; where p is the probability of heads.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The test statistic in that case would be the number of heads. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, I &lt;em&gt;assume&lt;/em&gt; that what you're calling &quot;t-value&quot; is a generic &quot;test statistic&quot;, not a value from a &quot;t distribution&quot;. They're not the same thing, and the term &quot;t-value&quot; isn't (necessarily) widely used and could be confusing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you're calling &quot;t-value&quot; is probably what I'm calling &quot;test statistic&quot;. In order to calculate a p-value (remember, it's just a probability) you need a distribution, and a value to plug into that distribution which will return a probability. Once you do that, the probability you return is your p-value. You can see that they are related because under the same distribution, different test-statistics are going to return different p-values. More extreme test-statistics will return lower p-values giving greater indication that the null hypothesis is false. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've ignored the issue of one-sided and two-sided p-values here.  &lt;/p&gt;&#xA;" OwnerUserId="62" LastActivityDate="2010-07-20T12:52:55.920" />
  <row Id="296" PostTypeId="2" ParentId="7" CreationDate="2010-07-20T12:54:28.273" Score="8" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://mlcomp.org/&quot; rel=&quot;nofollow&quot;&gt;MLComp&lt;/a&gt; has quite a few interesting datasets, and as a bonus your algorithm will get ranked if you upload it.&lt;/p&gt;&#xA;" OwnerUserId="127" LastActivityDate="2010-07-20T12:54:28.273" />
  <row Id="297" PostTypeId="2" ParentId="3" CreationDate="2010-07-20T13:08:42.907" Score="3" ViewCount="" Body="&lt;p&gt;I really enjoy working with &lt;a href=&quot;http://roofit.sourceforge.net/&quot; rel=&quot;nofollow&quot;&gt;RooFit&lt;/a&gt; for easy proper fitting of signal and background distributions and &lt;a href=&quot;http://tmva.sourceforge.net/&quot; rel=&quot;nofollow&quot;&gt;TMVA&lt;/a&gt; for quick principal component analyses and modelling of multivariate problems with some standard tools (like genetic algorithms and neural networks, also does BDTs). They are both part of the &lt;a href=&quot;http://root.cern.ch/drupal/&quot; rel=&quot;nofollow&quot;&gt;ROOT&lt;/a&gt; C++ libraries which have a pretty heavy bias towards particle physics problems though. &lt;/p&gt;&#xA;" OwnerUserId="56" LastActivityDate="2010-07-20T13:08:42.907" CommunityOwnedDate="2010-07-20T13:08:42.907" />
  <row Id="298" PostTypeId="1" CreationDate="2010-07-20T13:11:50.297" Score="10" ViewCount="2193" Body="&lt;p&gt;Am I looking for a better behaved distribution for the independent variable in question, or to reduce the effect of outliers, or something else?&lt;/p&gt;&#xA;" OwnerUserId="125" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-10-12T19:04:00.400" LastActivityDate="2010-10-12T21:33:05.457" Title="In linear regression, when is it appropriate to use the log of an independent variable instead of the actual values?" Tags="&lt;distributions&gt;&lt;regression&gt;&lt;data-transformation&gt;&lt;logarithm&gt;" AnswerCount="6" CommentCount="2" FavoriteCount="2" />
  <row Id="299" PostTypeId="2" ParentId="298" CreationDate="2010-07-20T13:16:29.303" Score="5" ViewCount="" Body="&lt;p&gt;One typically takes the log of an input variable to scale it and change the distribution (e.g. to make it normally distributed).  It cannot be done blindly however; you need to be careful when making any scaling to ensure that the results are still interpretable.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is discussed in most introductory statistics texts.  You can also read Andrew Gelman's paper on &lt;a href=&quot;http://www.stat.columbia.edu/~gelman/research/published/standardizing7.pdf&quot; rel=&quot;nofollow&quot;&gt;&quot;Scaling regression inputs by dividing by two standard deviations&quot;&lt;/a&gt; for a discussion on this.  He also has a very nice discussion on this at the beginning of &lt;a href=&quot;http://www.stat.columbia.edu/~gelman/arm/&quot; rel=&quot;nofollow&quot;&gt;&quot;Data Analysis Using Regression and Multilevel/Hierarchical Models&quot;&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Taking the log is not an appropriate method for dealing with bad data/outliers.&lt;/p&gt;&#xA;" OwnerUserId="5" LastEditorUserId="5" LastEditorDisplayName="" LastEditDate="2010-07-20T13:22:18.760" LastActivityDate="2010-07-20T13:22:18.760" />
  <row Id="300" PostTypeId="2" ParentId="278" CreationDate="2010-07-20T13:19:31.293" Score="4" ViewCount="" Body="&lt;p&gt;A &quot;right&quot; answer cannot depend on an arbitrary ordering of some method you are using.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You need to consider all possible orderings (or some representative sample) and estimate your parameters for every case. This will give you distributions for the parameters you are trying to estimate. Estimate the &quot;true&quot; parameter values from these distributions (this will also give you an estimate for your estimator error).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively use a method that doesn't introduce an ordering.&lt;/p&gt;&#xA;" OwnerUserId="56" LastActivityDate="2010-07-20T13:19:31.293" />
  <row Id="301" PostTypeId="2" ParentId="298" CreationDate="2010-07-20T13:22:40.320" Score="5" ViewCount="" Body="&lt;p&gt;You tend to take logs of the data when there is a problem with the residuals. For example, if you plot the residuals against a particular covariate and observe an increasing/decreasing pattern (a funnel shape), then a transformation may be appropriate. Non-random residuals usually indicate that your model assumptions are wrong, i.e. non-normal data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some data types automatically lend themselves to logarithmic transformations. For example, I usually take logs when dealing with concentrations or age. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Although transformations aren't primarily used to deal outliers, they do help since taking logs squashes your data.&lt;/p&gt;&#xA;" OwnerUserId="8" LastActivityDate="2010-07-20T13:22:40.320" CommentCount="5" />
  <row Id="302" PostTypeId="2" ParentId="276" CreationDate="2010-07-20T14:04:12.107" Score="1" ViewCount="" Body="&lt;p&gt;I've heard two rules of thumb in this regard.  One holds that so long as there are enough observations in the error term to evoke the central limit theorem, e.g. 20 or 30, you are fine.  The other holds that for each estimated slope one should have at least 20 or 30 observations.  The difference between using 20 or 30 as the target number is based on different thoughts regarding when there are enough observations to reasonably evoke the Central Limit Theorem. &lt;/p&gt;&#xA;" OwnerUserId="196" LastActivityDate="2010-07-20T14:04:12.107" CommentCount="2" />
  <row Id="303" PostTypeId="2" ParentId="288" CreationDate="2010-07-20T14:08:42.377" Score="2" ViewCount="" Body="&lt;p&gt;You have a hierarchical bayesian model. Brief details below:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Likelihood Function: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;f(n&lt;sub&gt;i&lt;/sub&gt; | p&lt;sub&gt;i&lt;/sub&gt;, t&lt;sub&gt;i&lt;/sub&gt;) = (t&lt;sub&gt;i&lt;/sub&gt; n&lt;sub&gt;i&lt;/sub&gt;) p&lt;sub&gt;i&lt;/sub&gt;&lt;sup&gt;n&lt;sub&gt;i&lt;/sub&gt;&lt;/sup&gt; (1-p&lt;sub&gt;i&lt;/sub&gt;)&lt;sup&gt;(t&lt;sub&gt;i&lt;/sub&gt; - n&lt;sub&gt;i&lt;/sub&gt;)&lt;/sup&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Priors on p&lt;sub&gt;i&lt;/sub&gt;, &amp;alpha;, &amp;beta;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;pi ~ Beta(&amp;alpha;, &amp;beta;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&amp;alpha; ~ N(&amp;alpha;_mean, &amp;alpha;_var) I(&amp;alpha; &gt; 0)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&amp;beta; ~ N(&amp;beta;_mean, &amp;beta;_var) I(&amp;beta; &gt; 0)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Posteriors are:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;p&lt;sub&gt;i&lt;/sub&gt; ~ Beta(&amp;alpha; + ni, &amp;beta; + ti-ni)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&amp;alpha; &amp;prop; I(&amp;alpha; &gt; 0) &amp;prod; p&lt;sub&gt;i&lt;/sub&gt;&lt;sup&gt;(&amp;alpha;-1)&lt;/sup&gt; exp(-(&amp;alpha;-&amp;alpha;_mean)&lt;sup&gt;2&lt;/sup&gt;) / (2 &amp;alpha;_var)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&amp;beta; &amp;prop; I(&amp;beta; &gt; 0) &amp;prod; (1-p&lt;sub&gt;i&lt;/sub&gt;)^(&amp;beta;-1) exp(-(&amp;beta;-&amp;beta;_mean)&lt;sup&gt;2&lt;/sup&gt;) / (2 &amp;beta;_var)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can then use a combination of Gibbs and Metropolis-Hastings to draw from the posterior distributions.&lt;/p&gt;&#xA;" LastActivityDate="2010-07-20T14:08:42.377" CommentCount="1" />
  <row Id="304" PostTypeId="2" ParentId="298" CreationDate="2010-07-20T14:13:50.103" Score="0" ViewCount="" Body="&lt;p&gt;Shane's point that taking the log to deal with bad data is well taken.  As is Colin's regarding the importance of normal residuals.  In practice I find that usually you can get normal residuals if the input and output variables are also relatively normal.  In practice this means eyeballing the distribution of the transformed and untransformed datasets and assuring oneself that they have become more normal and/or conducting tests of normality (e.g. Shapiro-Wilk or Kolmogorov-Smirnov tests) and determining whether the outcome is more normal.  Interpretablity and tradition are also important.  For example, in cognitive psychology log transforms of reaction time are often used, however, to me at least, the interpretation of a log RT is unclear.  Furthermore, one should be cautious using log transformed values as the shift in scale can change a main effect into an interaction and vice versa.&lt;/p&gt;&#xA;" OwnerUserId="196" LastActivityDate="2010-07-20T14:13:50.103" CommentCount="2" />
  <row Id="305" PostTypeId="1" AcceptedAnswerId="796" CreationDate="2010-07-20T14:19:41.383" Score="17" ViewCount="394" Body="&lt;p&gt;It seems like when the assumption of homogeneity of variance is met that the results from a Welch adjusted t-test and a standard t-test are approximately the same.  Why not simply always use the Welch adjusted t?&lt;/p&gt;&#xA;" OwnerUserId="196" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-07-26T13:05:38.663" LastActivityDate="2010-10-19T15:28:42.237" Title="When conducting a t-test why would one prefer to assume (or test for) equal variances rather than always use a Welch approximation of the df?" Tags="&lt;variance&gt;&lt;t-test&gt;&lt;homogeneity&gt;" AnswerCount="6" FavoriteCount="5" />
  <row Id="306" PostTypeId="2" ParentId="242" CreationDate="2010-07-20T14:21:37.820" Score="6" ViewCount="" Body="&lt;p&gt;The model that fits the data doesn't have to be a time series model; I would advise thinking outside the box a little.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have multiple variables (e.g. age, gender, diet, ethnicity, illness, medication) you can use these for a different model.  Maybe having certain patients in the same room is an important predictor?  Or perhaps it has to do with the attending staff?  Or consider using a multi-variate time series model (e.g. VECM) if you have other variables that you can use.  Look at the relationships between violence across patients: do certain patients act out together?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The time series model is useful if time has some important role in the behavior.  For instance, there might be a clustering of violence.  Look at the volatility clustering literature.  As @Jonas suggests, with a lag order of 2, you may need to be on higher alert on the day following a burst in violence.  But that doesn't help you prevent the first day: there may be other information that you can link into the analysis to actually &lt;em&gt;understand&lt;/em&gt; the cause of the violence, rather than simply forcasting it in a time series fashion.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lastly, as a technical suggestion: if you're using R for the analysis, you might have a look at &lt;a href=&quot;http://cran.r-project.org/web/packages/forecast/index.html&quot; rel=&quot;nofollow&quot;&gt;the forecast package&lt;/a&gt; from Rob Hyndman (the creator of this site).  This has many very nice features; see the paper &lt;a href=&quot;http://www.jstatsoft.org/v27/i03&quot; rel=&quot;nofollow&quot;&gt;&quot;Automatic Time Series Forecasting: The forecast Package for R&quot;&lt;/a&gt; in the Journal of Statistical Software.&lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-07-20T14:21:37.820" CommentCount="1" />
  <row Id="307" PostTypeId="2" ParentId="283" CreationDate="2010-07-20T14:23:30.843" Score="5" ViewCount="" Body="&lt;p&gt;A saturated model is one in which there are as many estimated parameters as data points. By definition, this will lead to a perfect fit, but will be of little use statistically, as you have no data left to estimate variance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if you have 6 data points and fit a 5th-order polynomial to the data, you would have a saturated model (one parameter for each of the 5 powers of your independant variable plus one for the constant term).&lt;/p&gt;&#xA;" OwnerUserId="229" LastActivityDate="2010-07-20T14:23:30.843" CommentCount="3" />
  <row Id="308" PostTypeId="2" ParentId="290" CreationDate="2010-07-20T14:38:07.360" Score="1" ViewCount="" Body="&lt;p&gt;There are couple of good links with introductory material at Princton Uni Library &lt;a href=&quot;http://libguides.princeton.edu/content.php?pid=27916&amp;amp;sid=459449&quot; rel=&quot;nofollow&quot;&gt;website&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="22" LastActivityDate="2010-07-20T14:38:07.360" />
  <row Id="309" PostTypeId="2" ParentId="305" CreationDate="2010-07-20T14:40:32.910" Score="2" ViewCount="" Body="&lt;p&gt;The fact that something more complex reduces to something less complex when some assumption is checked is not enough to throw the simpler method away. &lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-07-20T14:40:32.910" CommentCount="1" />
  <row Id="310" PostTypeId="2" ParentId="270" CreationDate="2010-07-20T14:41:33.560" Score="1" ViewCount="" Body="&lt;p&gt;You can simply ignore the 'factorial' when using maximum likelihood. Here is the reasoning for your suicides example. Let:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&amp;lambda; : Be the expected number of suicides per year&lt;/p&gt;&#xA;&#xA;&lt;p&gt;k&lt;sub&gt;i&lt;/sub&gt;: Be the number of suicides in year i.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then you would maximize the log-likelihood as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;LL = &amp;sum; ( k&lt;sub&gt;i&lt;/sub&gt; log(&amp;lambda;) - &amp;lambda; - k&lt;sub&gt;i&lt;/sub&gt;! )&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maximizing the above is equivalent to maximizing the following as k&lt;sub&gt;i&lt;/sub&gt;! is a constant :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;LL&lt;sup&gt;'&lt;/sup&gt; = &amp;sum; ( k&lt;sub&gt;i&lt;/sub&gt; log(&amp;lambda;) - &amp;lambda; )&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could explain why the factorial is an issue? Am I missing something?&lt;/p&gt;&#xA;" LastActivityDate="2010-07-20T14:41:33.560" CommentCount="5" />
  <row Id="311" PostTypeId="2" ParentId="220" CreationDate="2010-07-20T14:47:31.380" Score="1" ViewCount="" Body="&lt;p&gt;Rob Hyndman gave the easy exact answer for a fixed n.  If you're interested in asymptotic behavior for large n, this is handled in the field of &lt;a href=&quot;http://en.wikipedia.org/wiki/Extreme_value_theory&quot; rel=&quot;nofollow&quot;&gt;extreme value theory&lt;/a&gt;.  There is a small family of possible limiting distributions; see for example the first chapters of &lt;a href=&quot;http://books.google.com/books?id=3ZKmAAAAIAAJ&amp;amp;q=leadbetter+and+lindgren&amp;amp;dq=leadbetter+and+lindgren&amp;amp;hl=en&amp;amp;ei=z7ZFTPPFH9T-nAfa58HaAw&amp;amp;sa=X&amp;amp;oi=book_result&amp;amp;ct=result&amp;amp;resnum=1&amp;amp;ved=0CC0Q6AEwAA&quot; rel=&quot;nofollow&quot;&gt;this book&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="89" LastActivityDate="2010-07-20T14:47:31.380" CommentCount="1" />
  <row Id="312" PostTypeId="1" AcceptedAnswerId="318" CreationDate="2010-07-20T15:03:40.450" Score="2" ViewCount="55" Body="&lt;p&gt;Hi all, I'm a physics graduate who ended up doing infosec so most of the statistics I ever learned is useful for thermodynamics. I'm currently trying to think of a model for working out how many of a population of computers are infected with viruses, though I assume the maths works out the same way for real-world diseases so references in or answers relevant to that field would be welcome too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's what I've come up with so far:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;assume I know the total population of computers, N.&lt;/li&gt;&#xA;&lt;li&gt;I know the fraction D of computers that have virus-detection software (i.e. the amount of the population that is being screened)&lt;/li&gt;&#xA;&lt;li&gt;I know the fraction I of computers that have detection software &lt;em&gt;that has reported an infection&lt;/em&gt;&lt;/li&gt;&#xA;&lt;li&gt;I don't know, but can find out or estimate, the probability of Type I and II errors in the detection software.&lt;/li&gt;&#xA;&lt;li&gt;I don't (yet) care about the time evolution of the population.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;So where do I go from here? Would you model infection as a binomial distribution with probability like (I given D), or as a Poisson? Or is the distribution different?&lt;/p&gt;&#xA;" OwnerUserId="209" LastActivityDate="2010-07-20T15:23:53.653" Title="What approach could be used for modelling virus infections?" Tags="&lt;distributions&gt;&lt;modeling&gt;&lt;poisson&gt;&lt;binomial&gt;&lt;disease&gt;" AnswerCount="1" />
  <row Id="313" PostTypeId="2" ParentId="31" CreationDate="2010-07-20T15:10:15.333" Score="2" ViewCount="" Body="&lt;p&gt;Imagine you have a bag containing 900 black marbles and 100 white, i.e. 10% of the marbles are white. Now imagine you take 1 marble out, look at it and record its colour, take out another, record its colour etc.. and do this 100 times. At the end of this process you will have a number for white marbles which, ideally, we would expect to be 10, i.e. 10% of 100, but in actual fact may be 8, or 13 or whatever simply due to randomness. If you repeat this 100 marble withdrawal experiment many, many times and then plot a histogram of the number of white marbles drawn per experiment, you'll find you will have a Bell Curve centred about 10. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This represents your 10% hypothesis: with any bag containing 1000 marbles of which 10% are white, if you randomly take out 100 marbles you will find 10 white marbles in the selection, give or take 4 or so. The p-value is all about this &quot;give or take 4 or so.&quot; Let's say by referring to the Bell Curve created earlier you can determine that less than 5% of the time would you get 5 or fewer white marbles and another &amp;lt; 5% of the time accounts for 15 or more white marbles i.e. &gt; 90% of the time your 100 marble selection will contain between 6 to 14 white marbles inclusive.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now assuming someone plonks down a bag of 1000 marbles with an unknown number of white marbles in it, we have the tools to answer these questions&lt;/p&gt;&#xA;&#xA;&lt;p&gt;i) Are there fewer than 100 white marbles?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ii) Are there more than 100 white marbles?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;iii) Does the bag contain 100 white marbles?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Simply take out 100 marbles from the bag and count how many of this sample are white. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;a) If there are 6 to 14 whites in the sample you cannot reject the hypothesis that there are 100 white marbles in the bag and the corresponding p-values for 6 through 14 will be &gt; 0.05. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;b) If there are 5 or fewer whites in the sample you can reject the hypothesis that there are 100 white marbles in the bag and the corresponding p-values for 5 or fewer will be &amp;lt; 0.05. You would expect the bag to contain &amp;lt; 10% white marbles.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;c) If there are 15 or more whites in the sample you can reject the hypothesis that there are 100 white marbles in the bag and the corresponding p-values for 15 or more will be &amp;lt; 0.05. You would expect the bag to contain &gt; 10% white marbles.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;In response to Baltimark's comment&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given the example above, there is an approximately:-&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4.8% chance of getter 5 white balls or fewer&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1.85% chance of 4 or fewer&lt;/p&gt;&#xA;&#xA;&lt;p&gt;0.55% chance of 3 or fewer&lt;/p&gt;&#xA;&#xA;&lt;p&gt;0.1% chance of 2 or fewer &lt;/p&gt;&#xA;&#xA;&lt;p&gt;6.25% chance of 15 or more&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3.25% chance of 16 or more&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1.5% chance of 17 or more&lt;/p&gt;&#xA;&#xA;&lt;p&gt;0.65% chance of 18 or more&lt;/p&gt;&#xA;&#xA;&lt;p&gt;0.25% chance of 19 or more&lt;/p&gt;&#xA;&#xA;&lt;p&gt;0.1% chance of 20 or more&lt;/p&gt;&#xA;&#xA;&lt;p&gt;0.05% chance of 21 or more&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These numbers were estimated from an empirical distribution generated by a simple Monte Carlo routine run in R and the resultant quantiles of the sampling distribution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the purposes of answering the original question, suppose you draw 5 white balls, there is only an approximate 4.8% chance that if the 1000 marble bag really does contain 10% white balls you would pull out only 5 whites in a sample of 100. This equates to a p value &amp;lt; 0.05. You now have to choose between&lt;/p&gt;&#xA;&#xA;&lt;p&gt;i) There really are 10% white balls in the bag and I have just been &quot;unlucky&quot; to draw so few&lt;/p&gt;&#xA;&#xA;&lt;p&gt;or&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ii) I have drawn so few white balls that there can't really be 10% white balls (reject the hypothesis of 10% white balls)&lt;/p&gt;&#xA;" OwnerUserId="226" LastEditorUserId="226" LastEditorDisplayName="" LastEditDate="2010-07-21T00:33:56.993" LastActivityDate="2010-07-21T00:33:56.993" CommentCount="2" />
  <row Id="314" PostTypeId="2" ParentId="257" CreationDate="2010-07-20T15:10:34.927" Score="8" ViewCount="" Body="&lt;p&gt;R is definitely the answer.  I would just add to what Rob and Colin already said:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To improve the quality of your plots, you should consider using &lt;a href=&quot;http://cran.r-project.org/web/packages/Cairo/index.html&quot; rel=&quot;nofollow&quot;&gt;the &lt;strong&gt;Cairo&lt;/strong&gt; package&lt;/a&gt; for the output device.  That will greatly improve the &lt;em&gt;quality&lt;/em&gt; of the final graphics.  You simply call the function before plotting and it redirects to Cairo as the output device.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Cairo(600, 600, file=&quot;plot.png&quot;, type=&quot;png&quot;, bg=&quot;white&quot;)&#xA;plot(rnorm(4000),rnorm(4000),col=&quot;#ff000018&quot;,pch=19,cex=2) # semi-transparent red&#xA;dev.off() # creates a file &quot;plot.png&quot; with the above plot&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Lastly, in terms of putting it in a publication, that's the role that &lt;code&gt;Sweave&lt;/code&gt; plays.  It makes combining plots with your paper a trivial operation (and has the added benefit of leaving you with something that is &lt;em&gt;reproducible&lt;/em&gt; and understandable).  Use &lt;code&gt;cacheSweave&lt;/code&gt; if you have long-running computations.&lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-07-20T15:10:34.927" />
  <row Id="315" PostTypeId="2" ParentId="224" CreationDate="2010-07-20T15:15:57.377" Score="5" ViewCount="" Body="&lt;p&gt;If you can use R try &lt;a href=&quot;http://had.co.nz/ggplot2/&quot; rel=&quot;nofollow&quot;&gt;ggplot2&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="36" LastActivityDate="2010-07-20T15:15:57.377" CommunityOwnedDate="2010-10-12T08:48:39.990" />
  <row Id="316" PostTypeId="2" ParentId="73" CreationDate="2010-07-20T15:20:38.567" Score="6" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;ggplot2&lt;/strong&gt; - hands down best visualization for R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;RMySQL/RSQLite/RODBC&lt;/strong&gt; - for connecting to a databases&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;sqldf&lt;/strong&gt; - manipulate data.frames with SQL queries&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Hmisc/rms&lt;/strong&gt; - packages from Frank Harrell containing convenient miscellaneous functions and nice functions for regression analyses.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;GenABEL&lt;/strong&gt; - nice package for genome-wide association studies&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Rcmdr&lt;/strong&gt; - a decent GUI for R if you need one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also check out &lt;a href=&quot;http://crantastic.org/popcon&quot; rel=&quot;nofollow&quot;&gt;CRANtastic - this link&lt;/a&gt; has a list of the most popular R packages. Many on the top of the list have already been ment&lt;/p&gt;&#xA;" OwnerUserId="36" LastEditorUserId="36" LastEditorDisplayName="" LastEditDate="2010-07-20T15:28:42.347" LastActivityDate="2010-07-20T15:28:42.347" CommunityOwnedDate="2010-07-20T15:20:38.567" CommentCount="1" />
  <row Id="317" PostTypeId="2" ParentId="114" CreationDate="2010-07-20T15:21:01.857" Score="10" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://www.johndcook.com/blog/&quot; rel=&quot;nofollow&quot;&gt;The Endeavour&lt;/a&gt; sometimes features statistics posts. Otherwise it is mostly around the interplay of computer science and math.&lt;/p&gt;&#xA;" OwnerUserId="56" LastActivityDate="2010-07-20T15:21:01.857" CommunityOwnedDate="2010-07-20T15:21:01.857" />
  <row Id="318" PostTypeId="2" ParentId="312" CreationDate="2010-07-20T15:23:53.653" Score="4" ViewCount="" Body="&lt;p&gt;Computer virus propagation is structurally similar to infectious diseases propagation (vaccinations = anti-virus software, virus via email = getting a virus from someone etc).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Use the following links &lt;a href=&quot;http://en.wikipedia.org/wiki/Mathematical_modelling_in_epidemiology&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Mathematical_modelling_in_epidemiology&lt;/a&gt; and &lt;a href=&quot;http://en.wikipedia.org/wiki/Compartmental_models_in_epidemiology&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Compartmental_models_in_epidemiology&lt;/a&gt; as jumping points for your model.&lt;/p&gt;&#xA;" LastActivityDate="2010-07-20T15:23:53.653" />
  <row Id="319" PostTypeId="2" ParentId="31" CreationDate="2010-07-20T15:33:42.533" Score="8" ViewCount="" Body="&lt;p&gt;No amount of verbal explanation or calculations really helped me to understand &lt;em&gt;at a gut level&lt;/em&gt; what p-values were, but it really snapped into focus for me once I took a course that involved simulation.  That gave me the ability to actually &lt;em&gt;see&lt;/em&gt; data generated by the null hypothesis and to plot the means/etc. of simulated samples, then look at where my sample's statistic fell on that distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think the key advantage to this is that it lets students forget about the math and the test statistic distributions for a minute and focus on the concepts at hand.  Granted, it required that I learn &lt;em&gt;how&lt;/em&gt; to simulate that stuff, which will cause problems for an entirely different set of students.  But it worked for me, and I've used simulation countless times to help explain statistics to others with great success (e.g., &quot;This is what your data looks like; this is what a Poisson distribution looks like overlaid.  Are you SURE you want to do a Poisson regression?&quot;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This doesn't exactly answer the questions you posed, but for me, at least, it made them trivial.&lt;/p&gt;&#xA;" OwnerUserId="71" LastActivityDate="2010-07-20T15:33:42.533" CommentCount="2" />
  <row Id="320" PostTypeId="2" ParentId="283" CreationDate="2010-07-20T15:47:04.940" Score="5" ViewCount="" Body="&lt;p&gt;A saturated model is a model that is overparameterized to the point that it is basically just interpolating the data.  In some settings, such as image compression and reconstruction, this isn't necessarily a bad thing, but if you're trying to build a predictive model it's very problematic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In short, saturated models lead to extremely high-variance predictors that are being pushed around by the noise more than the actual data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a thought experiment, imagine you've got a saturated model, and there is noise in the data, then imagine fitting the model a few hundred times, each time with a different realization of the noise, and then predicting a new point.  You're likely to get radically different results each time, both for your fit and your prediction (and polynomial models are especially egregious in this regard); in other words the variance of the fit and the predictor are extremely high.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By contrast a model that is not saturated will (if constructed reasonably) give fits that are more consistent with each other even under different noise realization, and the variance of the predictor will also be reduced.&lt;/p&gt;&#xA;" OwnerUserId="61" LastActivityDate="2010-07-20T15:47:04.940" />
  <row Id="321" PostTypeId="1" CreationDate="2010-07-20T16:01:25.590" Score="4" ViewCount="194" Body="&lt;p&gt;There is a variant of boosting called &lt;a href=&quot;http://dx.doi.org/10.1214/aos/1016218223&quot; rel=&quot;nofollow&quot;&gt;gentleboost&lt;/a&gt;.  How does gentle boosting differ from the better-known &lt;a href=&quot;http://en.wikipedia.org/wiki/AdaBoost&quot; rel=&quot;nofollow&quot;&gt;AdaBoost&lt;/a&gt;?&lt;/p&gt;&#xA;" OwnerUserId="220" LastEditorUserId="220" LastEditorDisplayName="" LastEditDate="2010-07-20T19:25:13.293" LastActivityDate="2010-09-08T17:30:41.103" Title="How does gentle boosting differ from AdaBoost?" Tags="&lt;machine-learning&gt;&lt;boosting&gt;" AnswerCount="1" CommentCount="4" FavoriteCount="3" />
  <row Id="322" PostTypeId="1" CreationDate="2010-07-20T16:03:40.877" Score="7" ViewCount="271" Body="&lt;p&gt;I'm looking for a book or online resource that explains different kinds of entropy such as Sample Entropy and Shannon Entropy and their advantages and disadvantages.&#xA;Can someone point me in the right direction?&lt;/p&gt;&#xA;" OwnerUserId="237" LastActivityDate="2010-09-05T23:52:45.783" Title="Good introduction into different kinds of entropy" Tags="&lt;entropy&gt;" AnswerCount="6" CommentCount="0" FavoriteCount="2" />
  <row Id="323" PostTypeId="2" ParentId="114" CreationDate="2010-07-20T16:10:16.080" Score="6" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://www.stat.columbia.edu/~cook/movabletype/mlm/&quot; rel=&quot;nofollow&quot;&gt;Statistical Modeling, Causal Inference, and Social Science&lt;/a&gt; from Andrew Gelman is a good blog.&lt;/p&gt;&#xA;" OwnerUserId="237" LastActivityDate="2010-07-20T16:10:16.080" CommunityOwnedDate="2010-07-20T16:10:16.080" CommentCount="2" />
  <row Id="324" PostTypeId="2" ParentId="114" CreationDate="2010-07-20T16:12:08.700" Score="4" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://darrenjw.wordpress.com/&quot; rel=&quot;nofollow&quot;&gt;Darren Wilkinson's research blog&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="237" LastActivityDate="2010-07-20T16:12:08.700" CommunityOwnedDate="2010-07-20T16:12:08.700" />
  <row Id="325" PostTypeId="1" CreationDate="2010-07-20T16:12:43.443" Score="0" ViewCount="134" Body="&lt;p&gt;When is it reasonable to formulate and test a null hypothesis?&#xA;Could be the used assumptions explained in Bayesian terms?&#xA;Or should Bayesians use something else?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possible to compare it to the estimation&#xA;of the posterior probability: &lt;code&gt;P(h|data)&lt;/code&gt;?&lt;/p&gt;&#xA;" OwnerUserId="200" LastEditorUserId="200" LastEditorDisplayName="" LastEditDate="2010-07-21T08:55:00.207" LastActivityDate="2010-07-21T08:55:00.207" ClosedDate="2010-07-21T09:18:50.217" Title="When to use null hypothesis testing?" Tags="&lt;bayesian&gt;&lt;hypothesis-testing&gt;" CommentCount="3" />
  <row Id="326" PostTypeId="2" ParentId="114" CreationDate="2010-07-20T16:13:03.643" Score="4" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://xianblog.wordpress.com/&quot; rel=&quot;nofollow&quot;&gt;XI'AN'S OG&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="237" LastActivityDate="2010-07-20T16:13:03.643" CommunityOwnedDate="2010-07-20T16:13:03.643" CommentCount="1" />
  <row Id="327" PostTypeId="2" ParentId="62" CreationDate="2010-07-20T16:19:51.750" Score="1" ViewCount="" Body="&lt;p&gt;Malcom Gladewell analyses the problem in the book Outliers by analyzing Hockey Players.&lt;/p&gt;&#xA;" OwnerUserId="237" LastActivityDate="2010-07-20T16:19:51.750" />
  <row Id="328" PostTypeId="1" AcceptedAnswerId="329" CreationDate="2010-07-20T16:27:08.763" Score="5" ViewCount="518" Body="&lt;p&gt;I realize that the statistical analysis of financial data is a huge topic, but that is exactly why it is necessary for me to ask my question as I try to break into the world of financial analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As at this point I know next to nothing about the subject, the results of my google searches  are overwhelming.  Many of the matches advocate learning specialized tools or the R programming language. While I will learn these when they are necessary, I'm first interested in books, articles or any other resources that explain modern methods of statistical analysis specifically for financial data.  I assume there are a number of different wildly varied methods for analyzing data, so ideally I'm seeking an overview of the various methods that are practically applicable.  I'd like something that utilizes real world examples that a beginner is capable of grasping but that aren't overly simplistic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are some good resources for learning bout the statistical analysis of financial data?&lt;/p&gt;&#xA;" OwnerUserId="75" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-17T20:53:00.430" LastActivityDate="2010-10-02T17:17:44.863" Title="Resources for learning about the Statistical Analysis of Financial Data" Tags="&lt;books&gt;&lt;finance&gt;" AnswerCount="9" CommentCount="3" FavoriteCount="7" />
  <row Id="329" PostTypeId="2" ParentId="328" CreationDate="2010-07-20T16:31:26.140" Score="8" ViewCount="" Body="&lt;p&gt;You might start with this &lt;a href=&quot;http://oyc.yale.edu/economics/financial-markets/&quot; rel=&quot;nofollow&quot;&gt;series of lectures by Robert Shiller at Yale&lt;/a&gt;.  He gives a good overview of the field.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My favorite books on the subject:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;I strongly recommend starting with &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0387202706&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;Statistics and Finance&lt;/strong&gt;&lt;/a&gt;, by David Ruppert (&lt;a href=&quot;http://www.stat.tamu.edu/~ljin/Finance/stat689-R.htm&quot; rel=&quot;nofollow&quot;&gt;the R code for the book is available&lt;/a&gt;).  This is a great introduction and covers the basics of finance and statistics so it's appropriate as a first book.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0387279652&quot; rel=&quot;nofollow&quot;&gt;Modeling Financial Time Series with S-Plus&lt;/a&gt;, by Eric Zivot&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0471690740&quot; rel=&quot;nofollow&quot;&gt;Analysis of Financial Time Series&lt;/a&gt;, by Ruey Tsay&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0387759581&quot; rel=&quot;nofollow&quot;&gt;Time Series Analysis&lt;/a&gt;, by Jonathan D. Cryer&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Beyond that, you may want some general resources, and the &quot;bible&quot; of finance is &lt;a href=&quot;http://www.rotman.utoronto.ca/~hull/ofod/&quot; rel=&quot;nofollow&quot;&gt;Options, Futures, and Other Derivatives&lt;/a&gt; by John Hull.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lastly, in terms of some good general books, you might start with these two:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0393325350&quot; rel=&quot;nofollow&quot;&gt;A Random Walk Down Wall Street&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0471295639&quot; rel=&quot;nofollow&quot;&gt;Against the Gods: The Remarkable Story of Risk&lt;/a&gt; &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="5" LastEditorUserId="5" LastEditorDisplayName="" LastEditDate="2010-07-20T16:49:17.697" LastActivityDate="2010-07-20T16:49:17.697" />
  <row Id="331" PostTypeId="2" ParentId="305" CreationDate="2010-07-20T16:43:42.850" Score="5" ViewCount="" Body="&lt;p&gt;Because exact results are preferable to approximations, and avoid odd edge cases where the approximation may lead to a different result than the exact method.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Welch method isn't a quicker way to do any old t-test, it's a tractable approximation to an otherwise very hard problem: how to construct a t-test under unequal variances.  The equal-variance case is well-understood, simple, and exact, and therefore should always be used when possible.&lt;/p&gt;&#xA;" OwnerUserId="61" LastActivityDate="2010-07-20T16:43:42.850" />
  <row Id="333" PostTypeId="2" ParentId="328" CreationDate="2010-07-20T17:17:21.007" Score="3" ViewCount="" Body="&lt;p&gt;Ed Thorpe started the whole statistical arbitrage thing. He has a website, and some good articles.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://edwardothorp.com/&quot; rel=&quot;nofollow&quot;&gt;http://edwardothorp.com/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should also read Nassim Taleb's &quot;Fooled By Randomness&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, go on Google Scholar and read the top articles by Markowitz, Sharpe, Fama, Modigliani. If you don't have full access,  go to the nearest college and get a community library card. &lt;/p&gt;&#xA;" OwnerUserId="74" LastEditorUserId="74" LastEditorDisplayName="" LastEditDate="2010-08-13T18:01:09.700" LastActivityDate="2010-08-13T18:01:09.700" CommentCount="3" />
  <row Id="337" PostTypeId="2" ParentId="322" CreationDate="2010-07-20T17:20:41.267" Score="1" ViewCount="" Body="&lt;p&gt;The entropy is only one (as a concept) -- the amount of information needed to describe some system; there are only many its generalizations. Sample entropy is only some entropy-like descriptor used in heart rate analysis.&lt;/p&gt;&#xA;" OwnerUserId="88" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-07-20T17:36:23.330" LastActivityDate="2010-07-20T17:36:23.330" CommentCount="4" />
  <row Id="338" PostTypeId="2" ParentId="322" CreationDate="2010-07-20T17:22:01.153" Score="4" ViewCount="" Body="&lt;p&gt;Cover and Thomas's book &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0471062596&quot; rel=&quot;nofollow&quot;&gt;Elements of Information Theory&lt;/a&gt; is a good source on entropy and its applications, although I don't know that it addresses exactly the issues you have in mind.&lt;/p&gt;&#xA;" OwnerUserId="89" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-09-02T09:52:44.180" LastActivityDate="2010-09-02T09:52:44.180" CommentCount="2" />
  <row Id="339" PostTypeId="2" ParentId="26" CreationDate="2010-07-20T17:31:49.313" Score="1" ViewCount="" Body="&lt;p&gt;If the information required is the distribution of data about the mean, standard deviation comes in handy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The sum of the difference of each value from the mean is zero (obviously, since the value are evenly spread around the mean), hence we square each difference so as to convert negative values to positive, sum them across the population, and take their square root. This value is then divided by the number of samples (or, the size of the population). This gives the standard deviation.&lt;/p&gt;&#xA;" OwnerUserId="218" LastActivityDate="2010-07-20T17:31:49.313" />
  <row Id="340" PostTypeId="2" ParentId="269" CreationDate="2010-07-20T17:41:00.267" Score="3" ViewCount="" Body="&lt;p&gt;The population is everything in the group of study. For example, if you are studying the price of Apple's shares, it is the historical, current, and even all future stock prices. Or, if you run an egg factory, it is all the eggs made by the factory.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You don't always have to sample, and do statistical tests. If your population is your immediate living family, you don't need to sample, as the population is small. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sampling is popular for a variety of reasons:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;it is cheaper than a census (sampling the whole population)&lt;/li&gt;&#xA;&lt;li&gt;you don't have access to future data, so must sample the past&lt;/li&gt;&#xA;&lt;li&gt;you have to destroy some items by testing them, and don't want to destroy them all (say, eggs)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="74" LastActivityDate="2010-07-20T17:41:00.267" />
  <row Id="341" PostTypeId="1" AcceptedAnswerId="385" CreationDate="2010-07-20T18:12:29.550" Score="1" ViewCount="67" Body="&lt;p&gt;Do you think that unbalanced classes is a big problem for k-nearest neighbor? If so, do you know any smart way to handle this?&lt;/p&gt;&#xA;" OwnerUserId="88" LastEditorUserId="190" LastEditorDisplayName="" LastEditDate="2010-07-21T06:50:32.437" LastActivityDate="2010-07-21T06:50:32.437" Title="kNN and unbalanced classes" Tags="&lt;k-nearest-neighbour&gt;&lt;unbalanced-classes&gt;" AnswerCount="1" />
  <row Id="342" PostTypeId="2" ParentId="134" CreationDate="2010-07-20T18:13:44.820" Score="3" ViewCount="" Body="&lt;p&gt;If you're willing to tolerate an approximation, there are other methods. For example, one approximation is a value whose rank is within some (user specified) distance from the true median. For example, the median has (normalized) rank 0.5, and if you specify an error term of 10%, you'd want an answer that has rank between 0.45 and 0.55. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If such an answer is appropriate, then there are many solutions that can work on sliding windows of data. The basic idea is to maintain a sample of the data of a certain size (roughly 1/error term) and compute the median on this sample. It can be shown that with high probability, regardless of the nature of the input, the resulting median satisfies the properties I mentioned above.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, the main question is how to maintain a running sample of the data of a certain size, and there are many approaches for that, including the technique known as reservoir sampling. For example, this paper: &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.24.7136&quot; rel=&quot;nofollow&quot;&gt;http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.24.7136&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="139" LastActivityDate="2010-07-20T18:13:44.820" />
  <row Id="343" PostTypeId="2" ParentId="258" CreationDate="2010-07-20T18:16:17.513" Score="1" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/K-nearest_neighbor_algorithm&quot; rel=&quot;nofollow&quot;&gt;kNN&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-07-20T18:16:17.513" CommunityOwnedDate="2010-07-20T18:16:17.513" />
  <row Id="344" PostTypeId="2" ParentId="258" CreationDate="2010-07-20T18:17:25.903" Score="2" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Naive_bayes&quot; rel=&quot;nofollow&quot;&gt;Naive Bayes&lt;/a&gt; and &lt;a href=&quot;http://en.wikipedia.org/wiki/Random_naive_Bayes&quot; rel=&quot;nofollow&quot;&gt;Random Naive Bays&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-07-20T18:17:25.903" CommunityOwnedDate="2010-07-20T18:17:25.903" CommentCount="2" />
  <row Id="345" PostTypeId="2" ParentId="328" CreationDate="2010-07-20T19:08:50.867" Score="1" ViewCount="" Body="&lt;p&gt;Also good is &quot;Statistical Analysis of Financial Data in S-PLUS&quot; by Rene A. Carmona&lt;/p&gt;&#xA;" OwnerUserId="247" LastActivityDate="2010-07-20T19:08:50.867" />
  <row Id="346" PostTypeId="1" AcceptedAnswerId="428" CreationDate="2010-07-20T19:21:16.220" Score="12" ViewCount="501" Body="&lt;p&gt;I'm looking for a good algorithm (meaning minimal computation, minimal storage requirements) to estimate the median of a data set that is too large to store, such that each value can only be read once (unless you explicitly store that value). There are no bounds on the data that can be assumed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Approximations are fine, as long as the accuracy is known.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any pointers?&lt;/p&gt;&#xA;" OwnerUserId="247" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-10-08T16:06:48.257" LastActivityDate="2010-10-08T22:49:46.743" Title="What is a good algorithm for estimating the median of a huge read-once data set?" Tags="&lt;algorithms&gt;&lt;median&gt;&lt;large-data&gt;" AnswerCount="7" CommentCount="2" FavoriteCount="4" />
  <row Id="347" PostTypeId="2" ParentId="75" CreationDate="2010-07-20T19:50:57.100" Score="1" ViewCount="" Body="&lt;p&gt;I found this rather helpful: &lt;a href=&quot;http://cran.r-project.org/doc/contrib/Verzani-SimpleR.pdf&quot; rel=&quot;nofollow&quot;&gt;http://cran.r-project.org/doc/contrib/Verzani-SimpleR.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="15" LastActivityDate="2010-07-20T19:50:57.100" CommunityOwnedDate="2010-10-19T11:13:59.663" />
  <row Id="348" PostTypeId="2" ParentId="22" CreationDate="2010-07-20T19:54:13.473" Score="-1" ViewCount="" Body="&lt;p&gt;A male cat and a female cat are penned up in a steel chamber, along with enough food and water for 70 days.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A Frequentist would say the average gestation period for &lt;a href=&quot;http://en.wikipedia.org/wiki/Cat#Reproduction&quot; rel=&quot;nofollow&quot;&gt;felines&lt;/a&gt; is 66 days, the female was in heat when the cats were penned up, and once in heat she will mate repeatedly for 4 to 7 days.  Since there were likely many acts of propagation and enough subsequent time for gestation, the odds are, when the box is opened on day 70, there's a litter of newborn kittens.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A Bayesian would say, I heard some serious Marvin Gaye coming from the box on day 1 and then this morning I heard many kitten-like sounds coming from the box.  So without knowing much about cat reproduction, the odds are, when the box is opened on day 70, there's a litter of newborn kittens. &lt;/p&gt;&#xA;" OwnerUserId="24" LastEditorUserId="24" LastEditorDisplayName="" LastEditDate="2010-07-20T22:52:03.620" LastActivityDate="2010-07-20T22:52:03.620" CommentCount="1" />
  <row Id="349" PostTypeId="2" ParentId="346" CreationDate="2010-07-20T19:59:33.833" Score="4" ViewCount="" Body="&lt;p&gt;How about something like a binning procedure?  Assume (for illustration purposes) that you know that the values are between 1 and 1 million.  Set up N bins, of size S. So if S=10000, you'd have 100 bins, corresponding to values [1:10000, 10001:20000, ... , 990001:1000000] &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, step through the values. Instead of storing each value, just increment the counter in the appropriate bin. Using the midpoint of each bin as an estimate, you can make a reasonable approximation of the median. You can scale this to as fine or coarse of a resolution as you want by changing the size of the bins. You're limited only by how much memory you have.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since you don't know how big your values may get, just pick a bin size large enough that you aren't likely to run out of memory, using some quick back-of-the-envelope calculations. You might also store the bins sparsely, such that you only add a bin if it contains a value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The link ryfm provides gives an example of doing this, with the additional step of using the cumulative percentages to more accurately estimate the point within the median bin, instead of just using midpoints. This is a nice improvement.&lt;/p&gt;&#xA;" OwnerUserId="54" LastEditorUserId="54" LastEditorDisplayName="" LastEditDate="2010-07-20T22:40:50.903" LastActivityDate="2010-07-20T22:40:50.903" CommentCount="5" />
  <row Id="350" PostTypeId="2" ParentId="346" CreationDate="2010-07-20T20:12:26.983" Score="1" ViewCount="" Body="&lt;p&gt;You can try to find a median based on grouped frequency distribution, &lt;a href=&quot;http://www.statcan.gc.ca/edu/power-pouvoir/ch11/median-mediane/5214872-eng.htm&quot; rel=&quot;nofollow&quot;&gt;here is some details&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="178" LastActivityDate="2010-07-20T20:12:26.983" />
  <row Id="351" PostTypeId="2" ParentId="7" CreationDate="2010-07-20T21:05:35.517" Score="4" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://timetric.com&quot; rel=&quot;nofollow&quot;&gt;Timetric&lt;/a&gt; provides a web interface to data and provide a list of the &lt;a href=&quot;http://timetric.com/dataset/&quot; rel=&quot;nofollow&quot;&gt;publicly available data sets&lt;/a&gt; they use&lt;/p&gt;&#xA;" OwnerUserId="16" LastActivityDate="2010-07-20T21:05:35.517" CommunityOwnedDate="2010-07-20T21:05:35.517" />
  <row Id="352" PostTypeId="2" ParentId="346" CreationDate="2010-07-20T21:18:45.380" Score="1" ViewCount="" Body="&lt;p&gt;I've never had to do this, so this is just a suggestion.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I see two (other) possibilities. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Half data&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Load in half the data and sort&lt;/li&gt;&#xA;&lt;li&gt;Next read in the remaining values and compare against the your sorted list. &#xA;&lt;ol&gt;&#xA;&lt;li&gt;If the new value is larger, discard it.&lt;/li&gt;&#xA;&lt;li&gt;else put the value in the sorted list and removing the largest value from that list.&lt;/li&gt;&#xA;&lt;/ol&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Sampling distribution&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The other option, is to use an approximation involving the sampling distribution. If your data is Normal, then the standard error for moderate &lt;em&gt;n&lt;/em&gt; is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1.253 * sd / sqrt(n)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To determine the size of &lt;em&gt;n&lt;/em&gt; that you would be happy with, I ran a quick Monte-Carlo simulation in R&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;n = 10000&#xA;outside.ci.uni = 0&#xA;outside.ci.nor = 0&#xA;N=1000&#xA;for(i in 1:N){&#xA;  #Theoretical median is 0&#xA;  uni = runif(n, -10, 10)&#xA;  nor  = rnorm(n, 0, 10)&#xA;&#xA;  if(abs(median(uni)) &amp;gt; 1.96*1.253*sd(uni)/sqrt(n))&#xA;    outside.ci.uni = outside.ci.uni + 1&#xA;&#xA;  if(abs(median(nor)) &amp;gt; 1.96*1.253*sd(nor)/sqrt(n))&#xA;    outside.ci.nor = outside.ci.nor + 1&#xA;}&#xA;&#xA;outside.ci.uni/N&#xA;outside.ci.nor/N&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For n=10000, 15% of the uniform median estimates were outside the CI.&lt;/p&gt;&#xA;" OwnerUserId="8" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-07-21T15:29:25.620" LastActivityDate="2010-07-21T15:29:25.620" CommentCount="1" />
  <row Id="353" PostTypeId="2" ParentId="205" CreationDate="2010-07-20T22:11:54.673" Score="5" ViewCount="" Body="&lt;p&gt;Fisher's scoring is just a version of Newton's method that happens to be identified with GLMs, there's nothing particularly special about it, other than the fact that the Fisher's information matrix happens to be rather easy to find for random variables in the exponential family.  It also ties in to a lot of other math-stat material that tends to come up about the same time, and gives a nice geometric intuition about what exactly Fisher information means.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There's absolutely no reason I can think of not to use some other optimizer if you prefer, other than that you might have to code it by hand rather than use a pre-existing package.  I suspect that any strong emphasis on Fisher scoring is a combination of (in order of decreasing weight) pedagogy, ease-of-derivation, historical bias, and &quot;not-invented-here&quot; syndrome.  &lt;/p&gt;&#xA;" OwnerUserId="61" LastActivityDate="2010-07-20T22:11:54.673" CommentCount="1" />
  <row Id="354" PostTypeId="1" CreationDate="2010-07-20T22:26:26.083" Score="4" ViewCount="185" Body="&lt;p&gt;Why do we seek to minimize &lt;code&gt;x^2&lt;/code&gt; instead of minimizing &lt;code&gt;|x|^1.95&lt;/code&gt; or &lt;code&gt;|x|^2.05&lt;/code&gt;.&#xA;Are there reasons why the number should be exactly two or is it simply a convention that has the advantage of simplifying the math?&lt;/p&gt;&#xA;" OwnerUserId="237" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-28T12:04:24.103" LastActivityDate="2010-11-28T12:04:24.103" Title="Bias towards natural numbers in the case of least squares" Tags="&lt;standard-deviation&gt;&lt;least-squares&gt;" AnswerCount="4" FavoriteCount="1" />
  <row Id="355" PostTypeId="2" ParentId="328" CreationDate="2010-07-20T22:27:42.633" Score="1" ViewCount="" Body="&lt;p&gt;Check out &lt;a href=&quot;http://wilmott.com/&quot; rel=&quot;nofollow&quot;&gt;Wilmott.com&lt;/a&gt; as well.  It's oriented toward more advanced practitioners, but if I had to choose one person from whom to learn financial math, it would be Paul Wilmott.  Brilliant but grounded.&lt;/p&gt;&#xA;" OwnerUserId="158" LastActivityDate="2010-07-20T22:27:42.633" />
  <row Id="356" PostTypeId="2" ParentId="288" CreationDate="2010-07-20T22:31:45.140" Score="2" ViewCount="" Body="&lt;p&gt;To fit the model you can use &lt;a href=&quot;http://www-fis.iarc.fr/~martyn/software/jags/&quot; rel=&quot;nofollow&quot;&gt;JAGS&lt;/a&gt; or &lt;a href=&quot;http://www.mrc-bsu.cam.ac.uk/bugs/&quot; rel=&quot;nofollow&quot;&gt;Winbugs&lt;/a&gt;. In fact if you look at the week 3 of the lecture notes at Paul Hewson's &lt;a href=&quot;http://users.aims.ac.za/~paulhewson/&quot; rel=&quot;nofollow&quot;&gt;webpage&lt;/a&gt;, the rats JAGS example is a beta binomial model. He puts gamma priors on alpha and beta.&lt;/p&gt;&#xA;" OwnerUserId="8" LastActivityDate="2010-07-20T22:31:45.140" />
  <row Id="357" PostTypeId="2" ParentId="2" CreationDate="2010-07-20T23:07:48.713" Score="3" ViewCount="" Body="&lt;p&gt;You can't know whether there normality and that's why you have to make an assumption that's there.&#xA;You can only prove the absence of normality with statistic tests.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even worse, when you work with real world data it's almost certain that there isn't true normality in your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That means that your statistical test is always a bit biased. The question is whether you can live with it's bias.&#xA;To do that you have to understand your data and the kind of normality that your statistical tool assumes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's the reason why Frequentist tools are such as subjective as Bayesian tools. You can't determine based on the data that it's normally distributed. You have to assume normality.&lt;/p&gt;&#xA;" OwnerUserId="237" LastActivityDate="2010-07-20T23:07:48.713" />
  <row Id="358" PostTypeId="2" ParentId="354" CreationDate="2010-07-20T23:21:05.400" Score="3" ViewCount="" Body="&lt;p&gt;We try to minimize the variance that is left within descriptors. Why variance? Read &lt;a href=&quot;http://stats.stackexchange.com/questions/118/standard-deviation-why-square-the-difference-instead-of-taking-the-absolute-val&quot; rel=&quot;nofollow&quot;&gt;this question&lt;/a&gt;; this also comes together with the (mostly silent) assumption that errors are normally distributed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Extension:&lt;/strong&gt;&lt;br&gt;&#xA;Two additional arguments:  &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;For variances, we have this nice &quot;law&quot; that the sum of variances is equal to the variance of sum, for uncorrelated samples. If we assume that the error is not correlated with the case, minimizing residual of squares will work straightforward to maximizing explained variance, what is maybe a not-so-good but still popular quality measure.  &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If we assume normality of an error, least squares error estimator is a maximal likelihood one.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="88" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-07-21T09:31:29.867" LastActivityDate="2010-07-21T09:31:29.867" CommentCount="4" />
  <row Id="359" PostTypeId="1" AcceptedAnswerId="457" CreationDate="2010-07-20T23:28:49.693" Score="6" ViewCount="209" Body="&lt;p&gt;The Wald, Likelihood Ratio and Lagrange Multiplier tests in the context of maximum likelihood estimation are asymptotically equivalent. However, for small samples, they tend to diverge quite a bit, and in some cases they result in different conclusions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can they be ranked according to how likely they are to reject the null? What to do when the tests have conflicting answers? Can you just pick the one which gives the answer you want or is there a &quot;rule&quot; or &quot;guideline&quot; as to how to proceed?&lt;/p&gt;&#xA;" OwnerUserId="90" LastActivityDate="2010-09-20T02:40:27.090" Title="The trinity of tests in maximum likelihood: what to do when faced with contradicting conclusions?" Tags="&lt;hypothesis-testing&gt;&lt;maximum-likelihood&gt;" AnswerCount="3" FavoriteCount="2" />
  <row Id="360" PostTypeId="2" ParentId="173" CreationDate="2010-07-20T23:56:33.933" Score="3" ViewCount="" Body="&lt;p&gt;Does it really need some advanced model? Based on what I know about TB, in case there is no epidemy the infections are stochastic acts and so the count form month N shouldn't be correlated with count from month N-1. (You can check this assumption with autocorrelation). If so, analyzing just the distribution of monthly counts may be sufficient to decide if some count is significantly higher than normal.&lt;br&gt;&#xA;On the other hand you can look for correlations with some other variables, like season, travel traffic, or anything that you can imagine that may be correlated. If you would found something like this, it could be then used for some data normalization.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-07-20T23:56:33.933" CommentCount="2" />
  <row Id="361" PostTypeId="2" ParentId="258" CreationDate="2010-07-21T00:09:09.960" Score="7" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Logistic_regression&quot; rel=&quot;nofollow&quot;&gt;Logistic Regression&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;fast and perform well on most datasets&lt;/li&gt;&#xA;&lt;li&gt;almost no parameters to tune&lt;/li&gt;&#xA;&lt;li&gt;handles both discrete/continuous features&lt;/li&gt;&#xA;&lt;li&gt;model is easily interpretable&lt;/li&gt;&#xA;&lt;li&gt;(not really restricted to binary classifications)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="170" LastActivityDate="2010-07-21T00:09:09.960" CommunityOwnedDate="2010-07-21T00:09:09.960" />
  <row Id="362" PostTypeId="1" AcceptedAnswerId="429" CreationDate="2010-07-21T00:24:35.500" Score="7" ViewCount="1422" Body="&lt;p&gt;What is the difference between the Shapiro-Wilk test of normality and the Kolmogorov-Smirnov test of normality?  When will results from these two methods differ?&lt;/p&gt;&#xA;" OwnerUserId="196" LastActivityDate="2010-07-21T15:36:34.687" Title="What is the difference between the Shapiro-Wilk test of normality and the Kolmogorov-Smirnov test of normality?" Tags="&lt;distributions&gt;&lt;statistical-significance&gt;&lt;normality&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="363" PostTypeId="1" CreationDate="2010-07-21T00:44:08.597" Score="26" ViewCount="980" Body="&lt;p&gt;If you could go back in time and tell yourself to read a specific book at the beginning of your career as a statistician, which book would it be?&lt;/p&gt;&#xA;" OwnerUserId="74" LastActivityDate="2011-03-17T00:09:23.670" CommunityOwnedDate="2010-07-21T00:54:29.463" Title="What is the single most influential book every statistician should read?" Tags="&lt;books&gt;" AnswerCount="15" CommentCount="2" FavoriteCount="22" />
  <row Id="365" PostTypeId="2" ParentId="257" CreationDate="2010-07-21T01:04:08.710" Score="4" ViewCount="" Body="&lt;p&gt;Another option is &lt;a href=&quot;http://www.gnuplot.info/&quot; rel=&quot;nofollow&quot;&gt;Gnuplot&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="226" LastActivityDate="2010-07-21T01:04:08.710" />
  <row Id="366" PostTypeId="2" ParentId="175" CreationDate="2010-07-21T01:23:00.817" Score="1" ViewCount="" Body="&lt;p&gt;For a linear regression you could use a repeated median straight line fit.&lt;/p&gt;&#xA;" OwnerUserId="226" LastActivityDate="2010-07-21T01:23:00.817" />
  <row Id="367" PostTypeId="2" ParentId="363" CreationDate="2010-07-21T01:35:13.500" Score="8" ViewCount="" Body="&lt;p&gt;I am no statistician, and I haven't read that much on the topic, but perhaps &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0805071342&quot; rel=&quot;nofollow&quot;&gt;Lady Tasting Tea: How Statistics Revolutionized Science in the Twentieth Century&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;should be mentioned? It is no textbook, but still worth reading.&lt;/p&gt;&#xA;" OwnerUserId="90" LastActivityDate="2010-07-21T01:35:13.500" CommunityOwnedDate="2010-07-21T01:35:13.500" CommentCount="1" />
  <row Id="368" PostTypeId="2" ParentId="212" CreationDate="2010-07-21T03:01:00.020" Score="2" ViewCount="" Body="&lt;p&gt;Suppose that the text has N words and that you require that an ASR should correctly predict at least 95% of words in the text. You currently have the observed error rate for the two methods. You can perform two type of tests.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Test 1: Do the ASR models meet your criteria of 95% prediction?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Test 2: Are the two ASR models equally good in speech recognition?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could make different type of assumptions regarding the data generating mechanism for your ASR models. The simplest, although a bit naive, would assume that word detection of each word in the text is an iid bernoulli variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Under the above assumption you could do a test of proportions where you check if the error rate for each model is consistent with a true error rate of 5% (test 1) or a test of difference in proportions where you check if the error rates between the two models is the same (test 2). &lt;/p&gt;&#xA;" LastActivityDate="2010-07-21T03:01:00.020" CommentCount="2" />
  <row Id="369" PostTypeId="1" CreationDate="2010-07-21T03:47:17.577" Score="5" ViewCount="184" Body="&lt;p&gt;Say I've got a program that monitors a news feed and as I'm monitoring it I'd like to discover when a bunch of stories come out with a particular keyword in the title. Ideally I want to know when there are an unusual number of stories clustered around one another.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm entirely new to statistical analysis and I'm wondering how you would approach this problem. How do you select what variables to consider? What characteristics of the problem affect your choice of an algorithm? Then, what algorithm do you choose and why?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks, and if the problem needs clarification please let me know.&lt;/p&gt;&#xA;" OwnerUserId="191" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-17T20:53:12.380" LastActivityDate="2010-09-17T20:53:12.380" Title="Working through a clustering problem" Tags="&lt;clustering&gt;" AnswerCount="4" FavoriteCount="3" />
  <row Id="370" PostTypeId="2" ParentId="363" CreationDate="2010-07-21T03:48:55.563" Score="10" ViewCount="" Body="&lt;p&gt;Here are two to put on the list:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/096139210X&quot; rel=&quot;nofollow&quot;&gt;Tufte. The visual display of quantitative information&lt;/a&gt;&lt;br&gt;&#xA;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0201076160&quot; rel=&quot;nofollow&quot;&gt;Tukey. Exploratory data analysis&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-07-21T03:48:55.563" CommunityOwnedDate="2010-07-21T03:48:55.563" CommentCount="1" />
  <row Id="371" PostTypeId="2" ParentId="363" CreationDate="2010-07-21T04:19:00.347" Score="4" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0521592712&quot; rel=&quot;nofollow&quot;&gt;Probability Theory: The Logic of Science&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="34" LastActivityDate="2010-07-21T04:19:00.347" CommunityOwnedDate="2010-07-21T04:19:00.347" />
  <row Id="372" PostTypeId="1" AcceptedAnswerId="381" CreationDate="2010-07-21T04:26:07.550" Score="1" ViewCount="265" Body="&lt;p&gt;What topics in statistics are most useful/relevant to data mining?&lt;/p&gt;&#xA;" OwnerUserId="252" LastEditorUserId="509" LastEditorDisplayName="" LastEditDate="2010-08-12T13:24:12.943" LastActivityDate="2010-09-13T13:16:33.200" CommunityOwnedDate="2010-07-21T08:05:16.953" Title="What are the key statistical concepts that relate to data mining?" Tags="&lt;probability&gt;&lt;data-mining&gt;&lt;decision-trees&gt;" AnswerCount="2" CommentCount="5" FavoriteCount="2" />
  <row Id="373" PostTypeId="1" AcceptedAnswerId="378" CreationDate="2010-07-21T04:30:50.663" Score="10" ViewCount="350" Body="&lt;p&gt;From Wikipedia :&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Suppose you're on a game show, and&#xA;  you're given the choice of three&#xA;  doors: Behind one door is a car;&#xA;  behind the others, goats. You pick a&#xA;  door, say No. 1, and the host, who&#xA;  knows what's behind the doors, opens&#xA;  another door, say No. 3, which has a&#xA;  goat. He then says to you, &quot;Do you&#xA;  want to pick door No. 2?&quot; Is it to&#xA;  your advantage to switch your choice?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The answer is, of course, yes - but it's incredibly un-inituitive. What misunderstanding do most people have about probability that leads to us scratching our heads -- or better put; what general rule can we take away from this puzzle to better train our intuition in the future?&lt;/p&gt;&#xA;" OwnerUserId="252" LastActivityDate="2010-08-01T06:53:28.040" Title="The Monty Hall Problem - where does our intuition fail us?" Tags="&lt;probability&gt;&lt;games&gt;" AnswerCount="9" FavoriteCount="2" />
  <row Id="374" PostTypeId="1" AcceptedAnswerId="379" CreationDate="2010-07-21T04:32:42.520" Score="5" ViewCount="109" Body="&lt;p&gt;What are pivot tables, and how can they be helpful in analyzing data?&lt;/p&gt;&#xA;" OwnerUserId="252" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-17T20:52:22.490" LastActivityDate="2010-09-17T20:52:22.490" Title="What are pivot tables, and how can they be helpful in analyzing data?" Tags="&lt;multivariate-analysis&gt;" AnswerCount="1" />
  <row Id="375" PostTypeId="1" AcceptedAnswerId="474" CreationDate="2010-07-21T04:34:45.700" Score="5" ViewCount="208" Body="&lt;p&gt;What's a good way to test a series of numbers to see if they're random (or at least psuedo-random)? Is there a good statistical measure of randomness that can be used to determine how random a set is?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More importantly, how can one &lt;em&gt;prove&lt;/em&gt; a method of generating numbers is psuedo-random?&lt;/p&gt;&#xA;" OwnerUserId="252" LastEditorUserId="190" LastEditorDisplayName="" LastEditDate="2010-07-26T11:26:49.593" LastActivityDate="2010-07-26T20:12:15.570" Title="Testing (and proving) the randomness of numbers" Tags="&lt;random-generation&gt;&lt;proof&gt;&lt;randomness&gt;" AnswerCount="3" CommentCount="2" FavoriteCount="1" />
  <row Id="376" PostTypeId="2" ParentId="354" CreationDate="2010-07-21T04:48:50.457" Score="5" ViewCount="" Body="&lt;p&gt;There's no reason you couldn't try to minimize norms other than x^2, there have been entire books written on quantile regression, for instance, which is more or less minimizing |x| if you're working with the median.  It's just generally harder to do and, depending on the error model, may not give good estimators (depending on whether that means low-variance or unbiased or low MSE estimators in the context).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for why we prefer integer moments over real-number-valued moments, the main reason is likely that while integer powers of real numbers always result in real numbers, non-integer powers of negative real numbers create complex numbers, thus requiring the use of an absolute value.  In other words, while the 3rd moment of a real-valued random variable is real, the 3.2nd moment is not necessarily real, and so causes interpretation problems.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other than that...&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Analytical expressions for the integer moments of random variables are typically much easier to find than real-valued moments, be it by generating functions or some other method.  Methods to minimize them are thus easier to write.&lt;/li&gt;&#xA;&lt;li&gt;The use of integer moments leads to expressions that are more tractable than real-valued moments.&lt;/li&gt;&#xA;&lt;li&gt;I can't think of a compelling reason that (for instance) the 1.95th moment of the absolute value of X would provide better fitting properties than (for instance) the 2nd moment of X, although that could be interesting to investigate&lt;/li&gt;&#xA;&lt;li&gt;Specific to the L2 norm (or squared error), it can be written via dot products, which can lead to vast improvements in speed of computation.  It's also the only Lp space that's a Hilbert space, which is a nice feature to have.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="61" LastActivityDate="2010-07-21T04:48:50.457" />
  <row Id="377" PostTypeId="2" ParentId="373" CreationDate="2010-07-21T05:45:43.907" Score="4" ViewCount="" Body="&lt;p&gt;This doesn't give a general rule, but I think that one reason why it's a challenging puzzle is that our intuition doesn't handle conditional probability very well. There are plenty of &lt;a href=&quot;http://www.stubbornmule.net/2010/06/probability-paradoxes/&quot; rel=&quot;nofollow&quot;&gt;other probability puzzles that play on the same phenomenon&lt;/a&gt;. Since I'm linking to my blog, here's &lt;a href=&quot;http://www.stubbornmule.net/2010/06/monty-hall/&quot; rel=&quot;nofollow&quot;&gt;a post specifically on Monty Hall&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="173" LastActivityDate="2010-07-21T05:45:43.907" />
  <row Id="378" PostTypeId="2" ParentId="373" CreationDate="2010-07-21T05:54:25.527" Score="8" ViewCount="" Body="&lt;p&gt;Consider two simple variations of the problem: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;No doors are opened for the contestant. The host offers no help in picking a door. In this case it is obvious that the odds of picking the correct door are 1/3.&lt;/li&gt;&#xA;&lt;li&gt;Before the contestant is asked to venture a guess, the host opens a door and reveals a goat. After the host reveals a goat, the contestant has to pick the car from the two remaining doors. In this case it is obvious that the odds of picking the correct door is 1/2.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;For a contestant to know the probability of his door choice being correct, he has to know how many positive outcomes are available to him and divide that number by the amount of possible outcomes. Because of the two simple cases outlined above, it is very natural to think of all the possible outcomes available as the number of doors to choose from, and the amount of positive outcomes as the number of doors that conceal a car. Given this intuitive assumption, even if the host opens a door to reveal a goat &lt;em&gt;after&lt;/em&gt; the contestant makes a guess, the probability of either door containing a car remains 1/2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In reality, probability recognizes a set of possible outcomes larger than the three doors and it recognizes a set of positive outcomes that is larger than the singular door with the car. In the correct analysis of the problem, the host provides the contestant with new information making a new question to be addressed: what is the probability that my original guess is such that the new information provided by the host is sufficient to inform me of the correct door? In answering this question, the set of positive outcomes and the set of possible outcomes are not tangible doors and cars but rather abstract arrangements of the goats and car. The three possible outcomes are the three possible arrangements of two goats and one car behind three doors. The two positive outcomes are the two possible arrangements where the first guess of the contestant is false. In each of these two arrangements, the information given by the host (one of the two remaining doors is empty) is sufficient for the contestant to determine the door that conceals the car.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;In summation:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We have a tendency to look for a simple mapping between physical manifestations of our choices (the doors and the cars) and the number of possible outcomes and desired outcomes in a question of probability. This works fine in cases where no new information is provided to the contestant. However, if the contestant is provided with more information (ie one of the doors you didn't choose is certainly not a car), this mapping breaks down and the correct question to be asked is found to be more abstract.&lt;/p&gt;&#xA;" OwnerUserId="68" LastActivityDate="2010-07-21T05:54:25.527" />
  <row Id="379" PostTypeId="2" ParentId="374" CreationDate="2010-07-21T06:09:43.093" Score="3" ViewCount="" Body="&lt;p&gt;A pivot-table is a tool to dynamically show a slice and group multivariate data in tabular form.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, when we have the following data structure&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Region  Year  Product  Sales &#xA;US      2008  Phones   125 &#xA;EU      2008  Phones   352 &#xA;US      2008  Mouses   52 &#xA;EU      2008  Mouses   65 &#xA;US      2009  Phones   140 &#xA;EU      2009  Phones   320 &#xA;US      2009  Mouses   60 &#xA;EU      2009  Mouses   100&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;A pivot table can for example display a table with the sum of all products with in the rows the years and in the columns the regions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;All dimensions of the table can be switched easily. Also the data fields shown can be changed. This is called pivoting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The tool is useful in exploratory data analyses. Because it is a dynamic tool, it can be used to visually detect patterns and outliers etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most spreadsheet applications have support for this kind of tables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An image from wikipedia: &lt;img src=&quot;http://upload.wikimedia.org/wikipedia/commons/6/63/Pivottable-Pivoted.PNG&quot; alt=&quot;http://upload.wikimedia.org/wikipedia/commons/6/63/Pivottable-Pivoted.PNG&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="190" LastEditorUserId="190" LastEditorDisplayName="" LastEditDate="2010-07-21T08:00:10.447" LastActivityDate="2010-07-21T08:00:10.447" />
  <row Id="380" PostTypeId="2" ParentId="375" CreationDate="2010-07-21T06:15:36.623" Score="1" ViewCount="" Body="&lt;p&gt;You cannot prove, because it is impossible; you can only check if there are no any embarrassing autocorrelations or distribution disturbances, and indeed &lt;a href=&quot;http://en.wikipedia.org/wiki/Diehard_tests&quot; rel=&quot;nofollow&quot;&gt;Diehard&lt;/a&gt; is a standard for it. This is for statistics/physics, cryptographers will also mainly check (among other things) how hard is it to fit the generator to the data to obtain the future values.  &lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-07-21T06:15:36.623" />
  <row Id="381" PostTypeId="2" ParentId="372" CreationDate="2010-07-21T06:22:51.820" Score="5" ViewCount="" Body="&lt;p&gt;Understanding &lt;strong&gt;multivariate normal distribution&lt;/strong&gt; &lt;a href=&quot;http://en.wikipedia.org/wiki/Multivariate_normal_distribution&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Multivariate_normal_distribution&lt;/a&gt; is important. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The concept of &lt;strong&gt;correlation&lt;/strong&gt; and more generally (non linear) dependence is important. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Concentration of measure, asymptotic normality, convergence of random variables&lt;/strong&gt;.... how to make something from random to deterministic! &lt;a href=&quot;http://en.wikipedia.org/wiki/Convergence_of_random_variables&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Convergence_of_random_variables&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;maximum likelihood estimation&lt;/strong&gt; &lt;a href=&quot;http://en.wikipedia.org/wiki/Maximum_likelihood&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Maximum_likelihood&lt;/a&gt; and before that, statistical modeling :) and more generally minimum contrast estimation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;stationary process&lt;/strong&gt; &lt;a href=&quot;http://en.wikipedia.org/wiki/Stationary_process&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Stationary_process&lt;/a&gt; and more generally stationnarity assumption and ergodic property. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;as Peter said, the question is so broad ... that the answer couldn't be given in a post ... &lt;/p&gt;&#xA;" OwnerUserId="223" LastActivityDate="2010-07-21T06:22:51.820" CommunityOwnedDate="2010-08-12T13:30:08.187" />
  <row Id="382" PostTypeId="2" ParentId="124" CreationDate="2010-07-21T06:28:23.900" Score="1" ViewCount="" Body="&lt;p&gt;Firstly I can recommend you the book &lt;a href=&quot;http://nlp.stanford.edu/fsnlp/&quot; rel=&quot;nofollow&quot;&gt;Foundations of statistical natural language processing&lt;/a&gt; by Manning and Schütze.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The methods I would use are word-frequency distributions and ngram language models. The first works very well when you want to classify on topic and your topics are specific and expert (having keywords). Ngram modelling is the best way when you want to classify writing styles etc. &lt;/p&gt;&#xA;" OwnerUserId="190" LastActivityDate="2010-07-21T06:28:23.900" />
  <row Id="383" PostTypeId="2" ParentId="4" CreationDate="2010-07-21T06:28:43.477" Score="2" ViewCount="" Body="&lt;p&gt;You might be interested in applying relative distribution methods.  Call one group the reference group, and the other the comparison group.  In a way similar to constructing a probability-probability plot, you can construct a relative CDF/PDF, which is a ratio of the densities.  This relative density can be used for inference.  If the distributions are identical, you expect a uniform relative distribution.  There are tools, graphical and statistical, to explore and examine departures from uniformity.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A good starting point to get a better sense is &lt;a href=&quot;http://www.csss.washington.edu/Papers/wp27.pdf&quot; rel=&quot;nofollow&quot;&gt;Applying Relative Distrbution Methods in R&lt;/a&gt; and the &lt;a href=&quot;http://cran.r-project.org/web/packages/reldist/index.html&quot; rel=&quot;nofollow&quot;&gt;reldist&lt;/a&gt; package in R.  For details, you'll need to refer to the book, &lt;a href=&quot;http://csde.washington.edu/~handcock/RelDist/&quot; rel=&quot;nofollow&quot;&gt;Relative Distribution Methods in the Social Sciences&lt;/a&gt; by Handcock and Morris.  There's also a &lt;a href=&quot;http://www.jstor.org/pss/270964&quot; rel=&quot;nofollow&quot;&gt;paper&lt;/a&gt; by the authors covering the relevant techniques.&lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-07-21T06:28:43.477" />
  <row Id="384" PostTypeId="2" ParentId="369" CreationDate="2010-07-21T06:33:36.307" Score="1" ViewCount="" Body="&lt;p&gt;I would start with a frequency distribution. Collect for a big corpus the word-frequencies and select smartly the words that are keywords (not misspellings, with a very low frequency, and not stop words like &quot;and&quot;, &quot;or&quot;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then when you have a number of new feeds, compare their distribution with the distribution that you build from your training data. Look to the big differences in frequencies and select so the important keywords of that moment.&lt;/p&gt;&#xA;" OwnerUserId="190" LastActivityDate="2010-07-21T06:33:36.307" />
  <row Id="385" PostTypeId="2" ParentId="341" CreationDate="2010-07-21T06:39:38.230" Score="2" ViewCount="" Body="&lt;p&gt;In principal unbalanced classes are not a problem at all for the k-nearest neighbor algorithm. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because the algorithm is totally not influenced by the size of the class, it will not favor any on size. Try to run k-means with an obvious outlier and k+1 and you will see that most of the time the outlier will get it's own class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Off course, with hard datasets it is always advisable to run the algorithm multiple times. This to avoid trouble because of a bad initialization.&lt;/p&gt;&#xA;" OwnerUserId="190" LastActivityDate="2010-07-21T06:39:38.230" CommentCount="1" />
  <row Id="386" PostTypeId="2" ParentId="213" CreationDate="2010-07-21T06:46:11.897" Score="6" ViewCount="" Body="&lt;p&gt;I would do some sort of &quot;leave one out testing algorithm&quot; (n is the number of data):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;for i=1 to n&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;strong&gt;compute a density estimation of the data set obtained by throwing Xi away&lt;/strong&gt;. (This density estimate should be done with some assumption if the dimension is high, for example, a gaussian assumption for which the density estimate is easy: mean and covariance)&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Calculate the likelihood of Xi for the density estimated in step 1&lt;/strong&gt;. call it $L_i$. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;end for &lt;/p&gt;&#xA;&#xA;&lt;p&gt;sort the $L_i$ (for i=1,..,n) and use a multiple hypothesis testing procedure to say which are not good ... &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This will work if n is sufficiently large... you can also use &quot;leave k out strategy&quot; which can be more relevent when you have &quot;groups&quot; of outliers ...&lt;/p&gt;&#xA;" OwnerUserId="223" LastEditorUserId="223" LastEditorDisplayName="" LastEditDate="2010-08-04T17:47:33.707" LastActivityDate="2010-08-04T17:47:33.707" CommentCount="3" />
  <row Id="387" PostTypeId="2" ParentId="7" CreationDate="2010-07-21T07:04:26.103" Score="19" ViewCount="" Body="&lt;p&gt;The following list contains many data sets you may be interested:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.usnews.com/usnews/edu/college/rankings/rankindex_brief.php&quot; rel=&quot;nofollow&quot;&gt;America's Best Colleges - U.S. News &amp;amp; World Reports&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://factfinder.census.gov/servlet/BasicFactsServlet&quot; rel=&quot;nofollow&quot;&gt;American FactFinder&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.baseball1.com/&quot; rel=&quot;nofollow&quot;&gt;The Baseball Archive&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.albany.edu/sourcebook/&quot; rel=&quot;nofollow&quot;&gt;The Bureau of Justice Statistics&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.bls.gov/&quot; rel=&quot;nofollow&quot;&gt;The Bureau of Labor Statistics&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.bts.gov/&quot; rel=&quot;nofollow&quot;&gt;The Bureau of Transportation Statistics&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.census.gov/&quot; rel=&quot;nofollow&quot;&gt;The Census Bureau&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://lib.stat.cmu.edu/DASL/&quot; rel=&quot;nofollow&quot;&gt;Data and Story Library (DASL)&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.stat.ucla.edu/data/&quot; rel=&quot;nofollow&quot;&gt;Data Sets, UCLA Statistics Department&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.cvgs.k12.va.us/DIGSTATS/&quot; rel=&quot;nofollow&quot;&gt;DIG Stats&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.ers.usda.gov/Briefing/&quot; rel=&quot;nofollow&quot;&gt;Economic Research Service, US Department of Agriculture&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.eia.doe.gov/index.html&quot; rel=&quot;nofollow&quot;&gt;Energy Information Administration&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://epp.eurostat.ec.europa.eu/&quot; rel=&quot;nofollow&quot;&gt;Eurostat&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://exploringdata.cqu.edu.au/&quot; rel=&quot;nofollow&quot;&gt;Exploring Data&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.fedstats.gov/&quot; rel=&quot;nofollow&quot;&gt;FedStats&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.gallup.com/&quot; rel=&quot;nofollow&quot;&gt;The Gallop Organization&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.gtz.de/en/themen/umwelt-infrastruktur/transport/10285.htm&quot; rel=&quot;nofollow&quot;&gt;International Fuel Prices&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.amstat.org/publications/jse/&quot; rel=&quot;nofollow&quot;&gt;Journal of Statistics Education Data Archive&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.kentuckyderby.com/2003/derby_history/derby_statistics/&quot; rel=&quot;nofollow&quot;&gt;Kentucky Derby Race Statistics&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.ed.gov/NCES/&quot; rel=&quot;nofollow&quot;&gt;National Center for Education Statistics&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.cdc.gov/nchs/&quot; rel=&quot;nofollow&quot;&gt;National Center for Health Statistics&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.ncdc.noaa.gov/oa/ncdc.html&quot; rel=&quot;nofollow&quot;&gt;National Climatic Data Center&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.ngdc.noaa.gov/&quot; rel=&quot;nofollow&quot;&gt;National Geophysical Data Center&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.noaa.gov/&quot; rel=&quot;nofollow&quot;&gt;National Oceanic and Atmospheric Administration&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.amstat.org/sections/SIS/Sports%20Data%20Resources/&quot; rel=&quot;nofollow&quot;&gt;Sports Data Resources&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.statcan.ca/start.html&quot; rel=&quot;nofollow&quot;&gt;Statistics Canada&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://lib.stat.cmu.edu/datasets/&quot; rel=&quot;nofollow&quot;&gt;StatLib---Datasets Archive&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.statistics.gov.uk/&quot; rel=&quot;nofollow&quot;&gt;UK Government Statistical Service&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.un.org/cyberschoolbus/index.asp&quot; rel=&quot;nofollow&quot;&gt;United Nations: Cyber SchoolBus Resources&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="69" LastActivityDate="2010-07-21T07:04:26.103" CommunityOwnedDate="2010-07-21T07:04:26.103" />
  <row Id="388" PostTypeId="2" ParentId="363" CreationDate="2010-07-21T07:15:54.857" Score="4" ViewCount="" Body="&lt;p&gt;On the math/foundations side: Harald Cramér's &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0691005478&quot; rel=&quot;nofollow&quot;&gt;Mathematical Methods of Statistics&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-07-21T07:15:54.857" CommunityOwnedDate="2010-07-21T07:15:54.857" />
  <row Id="389" PostTypeId="2" ParentId="175" CreationDate="2010-07-21T07:51:34.243" Score="4" ViewCount="" Body="&lt;p&gt;I do think there is something to be said for just excluding the outliers. A regression line is supposed to summarise the data. Because of leverage you can have a situation where 1% of your data points affects the slope by 50%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's only dangerous from a moral and scientific point of view if you don't tell anybody that you excluded the outliers. As long as you point them out you can say:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;This regression line fits pretty well for most of the data. 1% of the time a value will come along that doesn't fit this trend, but hey, it's a crazy world, no system is perfect&quot;&lt;/p&gt;&#xA;" OwnerUserId="199" LastActivityDate="2010-07-21T07:51:34.243" />
  <row Id="390" PostTypeId="2" ParentId="363" CreationDate="2010-07-21T08:01:54.403" Score="3" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1587990717&quot; rel=&quot;nofollow&quot;&gt;Fooled By Randomness&lt;/a&gt; by Taleb&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Taleb is a professor at Columbia and an options trader. He made about $800 million dollars in 2008 betting against the market. He also wrote Black Swan. He discusses the absurdity of using the normal distribution to model markets, and philosophizes on our ability to use induction. &lt;/p&gt;&#xA;" OwnerUserId="74" LastEditorUserId="74" LastEditorDisplayName="" LastEditDate="2010-09-09T20:46:31.427" LastActivityDate="2010-09-09T20:46:31.427" CommunityOwnedDate="2010-07-21T08:01:54.403" />
  <row Id="392" PostTypeId="2" ParentId="257" CreationDate="2010-07-21T08:19:36.577" Score="4" ViewCount="" Body="&lt;p&gt;Take a look at the sample galleries for three popular visualization libraries:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://matplotlib.sourceforge.net/gallery.html&quot; rel=&quot;nofollow&quot;&gt;matplotlib gallery&lt;/a&gt; (Python)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://addictedtor.free.fr/graphiques/&quot; rel=&quot;nofollow&quot;&gt;R graph gallery&lt;/a&gt; (R) -- (also see &lt;a href=&quot;http://had.co.nz/ggplot2/&quot; rel=&quot;nofollow&quot;&gt;ggplot2&lt;/a&gt;, scroll down to reference)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://prefuse.org/gallery/&quot; rel=&quot;nofollow&quot;&gt;prefuse visualization gallery&lt;/a&gt; (Java)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;For the first two, you can even view the associated source code -- the simple stuff is simple, not many lines of code.  The prefuse case will have the requisite Java boilerplate code.  All three support a number of backends/devices/renderers (pdf, ps, png, etc).  All three are clearly capable of high quality graphics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think it pretty much boils down to which language are you most comfortable working in.  Go with that.&lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-07-21T08:19:36.577" />
  <row Id="393" PostTypeId="2" ParentId="363" CreationDate="2010-07-21T09:02:11.717" Score="10" ViewCount="" Body="&lt;p&gt;The book from hastie, Tibshirani and Friedman &lt;a href=&quot;http://www-stat.stanford.edu/~tibs/ElemStatLearn/&quot; rel=&quot;nofollow&quot;&gt;http://www-stat.stanford.edu/~tibs/ElemStatLearn/&lt;/a&gt; should be in any statistician's library ! &lt;/p&gt;&#xA;" OwnerUserId="223" LastActivityDate="2010-07-21T09:02:11.717" CommunityOwnedDate="2010-07-21T09:02:11.717" />
  <row Id="394" PostTypeId="2" ParentId="354" CreationDate="2010-07-21T09:53:40.083" Score="0" ViewCount="" Body="&lt;p&gt;My understanding is that wbecause we are trying to mimimise errors, we need to find a way of not getting ourselves in a situation where the sum of the negative difference in errors is equal to the sum of the positive difference in errors but we haven't found a good fit. We do this by squaring the sum of the difference in errors which means the negative and positive difference in errors both become positive (-1*-1 = 1). If we raised x to the power of anything other than a positve integer we wouldn't address this problem because the erros would not have the same sign, or if we raised to the power of something that isn't an integer we'd enter the realms of complex numbers. &lt;/p&gt;&#xA;" OwnerUserId="210" LastActivityDate="2010-07-21T09:53:40.083" />
  <row Id="395" PostTypeId="1" CreationDate="2010-07-21T10:13:25.813" Score="4" ViewCount="114" Body="&lt;p&gt;I have a data set where a series of measurements are being taken each week. In general the data set shows a +/- 1mm change each week with a mean measurement staying at about 0mm. In plotting the data this week it appears that some noticeable movement has occurred at two points and looking back at the data set, it is also possible that movement occurred last week as well. &#xA;What is the best way of looking at this data set to see how likely it is that the movements that have been seen are real movements rather than just some effect caused by the natural tolerance in the readings.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some more information on the data set. Measurements have been taken at 39 locations which should behave in a similar way although only some of the points may show signs of movement. At each point the readings have now been taken 10 times on a bi-weekly basis and up until the most recent set of readings the measurements were between -1mm and 1mm. The measurements can only be taken with mm accuracy so we only receive results to the nearest mm. The results for one of the points showing a movement is 0mm, 1mm, 0mm, -1mm, -1mm, 0mm, -1mm, -1mm, 1mm, 3mm. We are not looking for statistically significant information, just an indicator of what might have occurred. The reason is that if a measurement reaches 5mm in a subsequent week we have a problem and we'd like to be forewarned that this might occur. &lt;/p&gt;&#xA;" OwnerUserId="210" LastEditorUserId="556" LastEditorDisplayName="" LastEditDate="2011-02-09T11:30:29.170" LastActivityDate="2011-02-09T11:30:29.170" Title="How to tell if something happened in a data set which monitors a value over time" Tags="&lt;variance&gt;&lt;monitoring&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="1" />
  <row Id="396" PostTypeId="1" AcceptedAnswerId="399" CreationDate="2010-07-21T11:00:44.557" Score="17" ViewCount="662" Body="&lt;p&gt;I usually make my own idiosyncratic choices when preparing plots. However, I wonder if there are any best practices for generating plots. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: &lt;a href=&quot;http://stats.stackexchange.com/users/159/rob-hyndman&quot; rel=&quot;nofollow&quot;&gt;Rob's&lt;/a&gt; comment to an answer to this &lt;a href=&quot;http://stats.stackexchange.com/questions/257/what-is-the-easiest-way-to-create-publication-quality-plots-under-linux&quot; rel=&quot;nofollow&quot;&gt;question&lt;/a&gt; is very relevant here. &lt;/p&gt;&#xA;" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-08-17T05:27:03.063" LastActivityDate="2011-06-02T12:47:17.963" CommunityOwnedDate="2011-06-01T03:42:00.250" Title="What best practices should I follow when preparing plots?" Tags="&lt;data-visualization&gt;&lt;best-practices&gt;" AnswerCount="13" FavoriteCount="6" />
  <row Id="397" PostTypeId="2" ParentId="30" CreationDate="2010-07-21T11:11:47.037" Score="4" ViewCount="" Body="&lt;p&gt;Just to add a bit to honk's answer, the &lt;a href=&quot;http://en.wikipedia.org/wiki/Diehard_tests&quot; rel=&quot;nofollow&quot;&gt;Diehard Test Suite&lt;/a&gt; (developed by George Marsaglia) are the standard tests for PRNG.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There's a nice &lt;a href=&quot;http://www.phy.duke.edu/~rgb/General/dieharder.php&quot; rel=&quot;nofollow&quot;&gt;Diehard C library&lt;/a&gt; that gives you access to these tests. As well as the standard Diehard tests it also provides functions for a few other PRNG tests involving (amongst other things) checking bit order. There is also a facilty for testing the speed of the RNG and writing your own tests.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a R interface to the Dieharder library, called &lt;a href=&quot;http://dirk.eddelbuettel.com/code/rdieharder.html&quot; rel=&quot;nofollow&quot;&gt;RDieHarder&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(RDieHarder)&#xA;dhtest = dieharder(rng=&quot;randu&quot;, test=10, psamples=100, seed=12345)&#xA;print(dhtest)&#xA;&#xA;Diehard Count the 1s Test (byte)&#xA;&#xA;       data:  Created by RNG `randu' with seed=12345, &#xA;              sample of size 100 p-value &amp;lt; 2.2e-16&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This shows that the &lt;a href=&quot;http://en.wikipedia.org/wiki/RANDU&quot; rel=&quot;nofollow&quot;&gt;RANDU&lt;/a&gt; RNG generator fails the minimum-distance / 2dsphere test. &lt;/p&gt;&#xA;" OwnerUserId="8" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-09-28T22:20:14.603" LastActivityDate="2010-09-28T22:20:14.603" />
  <row Id="398" PostTypeId="2" ParentId="396" CreationDate="2010-07-21T11:16:55.077" Score="11" ViewCount="" Body="&lt;p&gt;We could stay here all day denoting best practices, but you should start by reading Tufte.  My primary recommendation:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Keep it simple.&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Often people try to load up their charts with information. But you should really just have one main idea that you're trying to convey and if someone doesn't get your message almost immediately, then you should rethink how you have presented it.  So don't start working on your chart until the message itself is clear. Occam's razor applies here too. &lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-07-21T11:16:55.077" CommunityOwnedDate="2011-06-01T03:42:00.250" CommentCount="1" />
  <row Id="399" PostTypeId="2" ParentId="396" CreationDate="2010-07-21T11:18:56.323" Score="12" ViewCount="" Body="&lt;p&gt;The Tufte principles are very good practices when preparing plots. See also his book &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0961392177&quot; rel=&quot;nofollow&quot;&gt;Beautiful Evidence&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The principles include:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Keep a high data-ink ratio&lt;/li&gt;&#xA;&lt;li&gt;Remove chart junk&lt;/li&gt;&#xA;&lt;li&gt;Give graphical element multiple functions&lt;/li&gt;&#xA;&lt;li&gt;Keep in mind the data density&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The term to search for is Information Visualization&lt;/p&gt;&#xA;" OwnerUserId="190" LastActivityDate="2010-07-21T11:18:56.323" CommunityOwnedDate="2011-06-01T03:42:00.250" CommentCount="1" />
  <row Id="400" PostTypeId="2" ParentId="396" CreationDate="2010-07-21T11:20:45.167" Score="4" ViewCount="" Body="&lt;p&gt;One rule of thumb that I don't always follow but which is on occasion useful is to take into account that it is likely that your plot will at some point in it's future be&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;sent by fax,&lt;/li&gt;&#xA;&lt;li&gt;photocopied, and/or&lt;/li&gt;&#xA;&lt;li&gt;reproduced in black-and-white.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You need to try and make your plots clear enough that even if they are imprecisely reproduced in the future, the information the plot is trying to convey is still legible. &lt;/p&gt;&#xA;" OwnerUserId="210" LastEditorUserId="3874" LastEditorDisplayName="" LastEditDate="2011-06-01T14:03:21.963" LastActivityDate="2011-06-01T14:03:21.963" CommunityOwnedDate="2011-06-01T03:42:00.250" CommentCount="1" />
  <row Id="401" PostTypeId="2" ParentId="369" CreationDate="2010-07-21T12:22:21.570" Score="2" ViewCount="" Body="&lt;p&gt;Hello there...&#xA;this problem you are asking about is known as test mining!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;there are a few things you need to consider.. for example in your question you mentioned using keywords in titles.. one may ask why not the text in the article rather than just the title? which brings me to the first consideration.... what data do you limit yourself to?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Secondly as the previous answer suggests using frequencies is a great start.... to take the analysis further you may start looking at what words occur frequently together! for example the word happy may occur very frequently... however if always accompanied with a &quot;not&quot; your conclusions would be very different!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a very nice Australian piece of software i have used in the past called Leximancer i would advise anybody nterested in text mining to have a look at their site and the examples they have.... from memory one of which analysed speaches by 2 US president candidant it makes for some very interesting reading!&lt;/p&gt;&#xA;" OwnerUserId="256" LastActivityDate="2010-07-21T12:22:21.570" />
  <row Id="402" PostTypeId="2" ParentId="173" CreationDate="2010-07-21T12:30:43.210" Score="3" ViewCount="" Body="&lt;p&gt;You might consider applying a &lt;a href=&quot;http://gunston.gmu.edu/708/frTukey.asp&quot; rel=&quot;nofollow&quot;&gt;Tukey Control chart&lt;/a&gt; to the data.&lt;/p&gt;&#xA;" OwnerUserId="226" LastActivityDate="2010-07-21T12:30:43.210" CommentCount="2" />
  <row Id="403" PostTypeId="2" ParentId="192" CreationDate="2010-07-21T12:33:43.903" Score="1" ViewCount="" Body="&lt;p&gt;I must agree.. there is no single best analysis!&#xA;not just in cross tabulations or analysis of categorical data but in any data analysis... and thank god for that!&#xA;if there was just a single best way to address these analyses well many of us would not have a job to start with... not to mention the loss of the thrill of the hunt!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;the joy of analysis is the unknown and the search for answers and evidence and how one question leads to another... that is what i love about statistics!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So back to the categorical data analysis... it really depends on what your doing. Are you looking to find how different variables affect each other as in drug tests for example we may look at treatment vs placebo crossed with disease and no disease... the question here is does treatment reduce disease.... chi square usually does well here (given a good sample size).&#xA;Another context ihad today was looking at missing value trends... i was looking to find if missing values in one categorical variable relate to another... in some cases i knew the result should be missing and yet there were observations that had values... a completely different context to the drug test!&lt;/p&gt;&#xA;" OwnerUserId="256" LastActivityDate="2010-07-21T12:33:43.903" />
  <row Id="404" PostTypeId="2" ParentId="173" CreationDate="2010-07-21T12:37:31.977" Score="4" ViewCount="" Body="&lt;p&gt;You might want to have a look at &lt;a href=&quot;http://cran.r-project.org/web/packages/strucchange/index.html&quot; rel=&quot;nofollow&quot;&gt;strucchange&lt;/a&gt;: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Testing, monitoring and dating structural changes in (linear) regression models. strucchange features tests/methods from the generalized fluctuation test framework as well as from the F test (Chow test) framework. This includes methods to fit, plot and test fluctuation processes (e.g., CUSUM, MOSUM, recursive/moving estimates) and F statistics, respectively. It is possible to monitor incoming data online using fluctuation processes. Finally, the breakpoints in regression models with structural changes can be estimated together with confidence intervals. Emphasis is always given to methods for visualizing the data.&quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;PS. Nice graphics ;)&lt;/p&gt;&#xA;" OwnerUserId="46" LastActivityDate="2010-07-21T12:37:31.977" CommentCount="1" />
  <row Id="405" PostTypeId="2" ParentId="395" CreationDate="2010-07-21T12:38:51.197" Score="1" ViewCount="" Body="&lt;p&gt;What kind of movement are we talking about?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could of course fit a distribution over your data and see whether the new weeks fit in this distribution or are in the tail of it (which means it is likely something significant, real that you are observing)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, more information from your side would be helpful. Maybe you could provide a part of the dataset?&lt;/p&gt;&#xA;" OwnerUserId="190" LastActivityDate="2010-07-21T12:38:51.197" CommentCount="1" />
  <row Id="406" PostTypeId="2" ParentId="6" CreationDate="2010-07-21T12:43:00.080" Score="7" ViewCount="" Body="&lt;p&gt;I disagree with this question as it suggests that machine learning and statistics are different or conflicting sciences.... when the opposite is true!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;machine learning makes extensive use of statistics... a quick survey of any Machine learning or data mining software package will reveal Clustering techniques such as k-means also found in statistics.... will also show dimension reduction techniques such as Principal components analysis also a statistical technique... even logistic regression yet another.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my view the main difference is that traditionally statistics was used to proove a pre conceived theory and usually the analysis was design around that principal theory. Where with data mining or machine learning the opposite approach is usually the norm in that we have the outcome we just want to find a way to predict it rather than ask the question or form the theory is this the outcome!&lt;/p&gt;&#xA;" OwnerUserId="256" LastActivityDate="2010-07-21T12:43:00.080" />
  <row Id="407" PostTypeId="2" ParentId="213" CreationDate="2010-07-21T12:59:00.577" Score="1" ViewCount="" Body="&lt;p&gt;One of the above answers touched in mahalanobis distances.... perhaps anpther step further and calculating simultaneous confidence intervals would help detect outliers!&lt;/p&gt;&#xA;" OwnerUserId="256" LastActivityDate="2010-07-21T12:59:00.577" />
  <row Id="408" PostTypeId="2" ParentId="396" CreationDate="2010-07-21T13:01:39.883" Score="6" ViewCount="" Body="&lt;p&gt;In addition to conveying a clear message I always try to remember the plotsmanship:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;font sizes for labels and legends should be big enough, preferably the same font size and font used in the final publication.&lt;/li&gt;&#xA;&lt;li&gt;linewidths should be big enough (1 pt lines tend to disappear if plots are shrunk only slightly). I try to go to linewidths of 3 to 5 pt.&lt;/li&gt;&#xA;&lt;li&gt;if plotting multiple datasets/curves with color make sure that they can be understood if printed in black-and-white, e.g. by using different symbols or linestyles in addition to color.&lt;/li&gt;&#xA;&lt;li&gt;always use a lossless (or close to lossless) format, e.g. a vector format like pdf, ps or svg or high resolution png or gif (jpeg doesn't work at all and was never designed for line art).&lt;/li&gt;&#xA;&lt;li&gt;prepare graphics in the final aspect ratio to be used in the publication. Changing the aspect ration later can give irritating font or symbol shapes.&lt;/li&gt;&#xA;&lt;li&gt;always remove useless clutter from the plotting program like unused histogram information, trend lines (hardly useful) or default titles.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I have configured my plotting software (matplotlib, ROOT or root2matplotlib) to do most of this right by default. Before I was using &lt;code&gt;gnuplot&lt;/code&gt; which needed extra care here.&lt;/p&gt;&#xA;" OwnerUserId="56" LastActivityDate="2010-07-21T13:01:39.883" CommunityOwnedDate="2011-06-01T03:42:00.250" />
  <row Id="409" PostTypeId="1" CreationDate="2010-07-21T13:11:33.167" Score="2" ViewCount="120" Body="&lt;p&gt;At the moment I use standard deviation of the mean to estimate uncertainty:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://upload.wikimedia.org/math/8/a/e/8ae657bf495238caf69d82d754e70ed8.png&quot; alt=&quot;definition = stdev over sq.root of N&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where &lt;em&gt;N&lt;/em&gt; is in hundreds and mean is a time series (monthly) mean. I&#xA;present it then like this: &lt;img src=&quot;http://rogercortesi.com/eqn/tempimagedir/eqn1898.png&quot; alt=&quot;mean plus-minus stdev about the mean&quot;&gt; for each element (month) in the (annual) time series.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this valid? Is this appropriate for time series?&lt;/p&gt;&#xA;" OwnerUserId="219" LastEditorUserId="219" LastEditorDisplayName="" LastEditDate="2010-07-21T13:40:27.607" LastActivityDate="2010-08-30T13:55:03.287" Title="How to approximate measurement uncertainty?" Tags="&lt;time-series&gt;&lt;standard-deviation&gt;&lt;mean&gt;&lt;uncertainty&gt;" AnswerCount="2" />
  <row Id="410" PostTypeId="2" ParentId="409" CreationDate="2010-07-21T13:20:53.847" Score="2" ViewCount="" Body="&lt;p&gt;The answer to this question depends a lot on how your measurement uncertainty arises. If it is due to to uncorrelated normally distributed fluctuations in your measurement your measurement outcomes will also be normally distributed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If this assumption is valid can be hard to prove, but plotting histograms of outcomes of independent measurement should give you a feeling for the shape (and size) of the uncertainty.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since the normal distribution works on a unlimited range of values, typical cases where measurements are not normally distributed are counting experiments where the number to be measured is small (e.g. less than 20) and/or fluctuations are large, or when the measured quantity is defined to be in a range, e.g. a fraction. One might still use normal approximation in a certain range though.&lt;/p&gt;&#xA;" OwnerUserId="56" LastActivityDate="2010-07-21T13:20:53.847" CommentCount="2" />
  <row Id="411" PostTypeId="1" CreationDate="2010-07-21T13:39:06.783" Score="18" ViewCount="1011" Body="&lt;p&gt;There are many ways to measure how similar two probability distributions are.  Among methods which are popular (in different circles) are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;the Kolmogorov distance: the sup-distance between the distribution functions;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;the Kantorovich-Rubinstein distance: the maximum difference between the expectations w.r.t. the two distributions of functions with Lipschitz constant 1, which also turns out to be the L^1 distance between the distribution functions;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;the bounded-Lipschitz distance: like the K-R distance but the functions are also required to have absolute value at most 1.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;These have different advantages and disadvantages.  Only convergence in the sense of 3. actually corresponds precisely to convergence in distribution; convergence in the sense of 1. or 2. is slightly stronger in general.  (In particular, if X_n=1/n with probability 1, then X_n converges to 0 in distribution, but not in the Kolmogorov distance. However, if the limit distribution is continuous then this pathology doesn't occur.) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the perspective of elementary probability or measure theory, 1. is very natural because it compares the probabilities of being in some set. A more sophisticated probabilistic perspective, on the other hand, tends to focus more on expectations than probabilities. Also, from the perspective of functional analysis, distances like 2. or 3. based on duality with some function space are very appealing, because there is a large set of mathematical tools for working with such things.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, my impression (correct me if I'm wrong!) is that in statistics, the Kolmogorov distance is the usually preferred way of measuring similarity of distributions.  I can guess one reason: if one of the distributions is discrete with finite support -- in particular, if it is the distribution of some real-world data -- then the Kolmogorov distance to a model distribution is easy to compute.  (The K-R distance would be slightly harder to compute, and the B-L distance would probably be impossible in practical terms.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my question (finally) is, are there other reasons, either practical or theoretical, to favor the Kolmogorov distance (or some other distance) for statistical purposes?&lt;/p&gt;&#xA;" OwnerUserId="89" LastEditorUserId="223" LastEditorDisplayName="" LastEditDate="2010-12-17T07:45:11.850" LastActivityDate="2010-12-17T22:42:22.920" Title="Motivation for Kolmogorov distance between distributions" Tags="&lt;distributions&gt;&lt;probability&gt;&lt;hypothesis-testing&gt;&lt;mathematical-statistics&gt;" AnswerCount="5" CommentCount="8" FavoriteCount="5" />
  <row Id="412" PostTypeId="2" ParentId="395" CreationDate="2010-07-21T13:40:02.007" Score="4" ViewCount="" Body="&lt;p&gt;I think you need look at statistical &lt;a href=&quot;http://en.wikipedia.org/wiki/Control_chart&quot; rel=&quot;nofollow&quot;&gt;control charts&lt;/a&gt;. The most common of which are cusum and Shewhart charts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically, data arrives sequentially and is tested against a number of rules. For example,&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Is the data far away from the cumulative mean - say 3 standard deviations &lt;/li&gt;&#xA;&lt;li&gt;Has the data been increasing for the last few points.&lt;/li&gt;&#xA;&lt;li&gt;Does the data alternate between positive and negative values.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In R you can use the &lt;a href=&quot;http://cran.r-project.org/web/packages/qcc/index.html&quot; rel=&quot;nofollow&quot;&gt;qcc&lt;/a&gt; package.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#Taken from the documentation&#xA;library(qcc)&#xA;data(orangejuice)&#xA;attach(orangejuice)&#xA;plot(qcc(D[trial], sizes=size[trial], type=&quot;p&quot;))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Gives the following plot, with possible problem points highlighted in red.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://img805.imageshack.us/img805/5858/tmp.jpg&quot; alt=&quot;control chart&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="8" LastActivityDate="2010-07-21T13:40:02.007" />
  <row Id="413" PostTypeId="2" ParentId="6" CreationDate="2010-07-21T13:48:15.687" Score="3" ViewCount="" Body="&lt;p&gt;I don't really know what the conceptual/historical difference between machine learning and statistic is but I am sure it is not that obvious... and I am not really interest in knowing if I am a machine learner or a statistician, I think 10 years after Breiman's paper, lots of people are both...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyway, I found  &lt;strong&gt;interesting the question about predictive accuracy of models&lt;/strong&gt;. We have to remember that it is not always possible to measure the accuracy of a model and more precisely we are most often implicitly making some modeling when measuring errors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For Example, mean absolute error in time series forecast is a mean over time and it measures the performance of a procedure to forecast the median with the assumption that performance is, in some sense, &lt;strong&gt;stationary&lt;/strong&gt; and shows some &lt;strong&gt;ergodic&lt;/strong&gt; property. If (for some reason) you need to forecast the mean temperature on earth for the next 50 years and if your modeling performs well for the last 50 years... it does not means that... &lt;/p&gt;&#xA;&#xA;&lt;p&gt;More generally, (if I remember, it is called no free lunch) you can't do anything without modeling... In addition, I think statistic is trying to find an answer to the question : &quot;is something significant or not &quot;, this is a very important question in science and can't be answered through a learning process. To state John Tukey (was he a statistician ?) :&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;em&gt;The combination of some data and an aching desire for an answer does not ensure that a reasonable answer can be extracted from a given body of data&lt;/em&gt; &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Hope this helps ! &lt;/p&gt;&#xA;" OwnerUserId="223" LastActivityDate="2010-07-21T13:48:15.687" />
  <row Id="414" PostTypeId="1" AcceptedAnswerId="439" CreationDate="2010-07-21T13:50:08.847" Score="11" ViewCount="845" Body="&lt;p&gt;What is a good introduction to statistics for a mathematician who is already well-versed in probability?  I have two distinct motivations for asking, which may well lead to different suggestions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;I'd like to better understand the statistics motivation behind many problems considered by probabilists.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I'd like to know how to better interpret the results of Monte Carlo simulations which I sometimes do to form mathematical conjectures.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I'm open to the possibility that the best way to go is not to look for something like &quot;Statistics for Probabilists&quot; and just go to a more introductory source.&lt;/p&gt;&#xA;" OwnerUserId="89" LastEditorUserId="509" LastEditorDisplayName="" LastEditDate="2010-08-13T17:20:39.927" LastActivityDate="2010-08-13T21:22:56.557" CommunityOwnedDate="2010-07-21T13:50:08.847" Title="Introduction to statistics for mathematicians" Tags="&lt;books&gt;" AnswerCount="6" FavoriteCount="7" />
  <row Id="415" PostTypeId="2" ParentId="414" CreationDate="2010-07-21T13:53:13.193" Score="3" ViewCount="" Body="&lt;p&gt;I think you should take a look to the similar post from mathoverflow at &lt;a href=&quot;http://mathoverflow.net/questions/31655/statistics-for-mathematicians/31665#31665&quot; rel=&quot;nofollow&quot;&gt;http://mathoverflow.net/questions/31655/statistics-for-mathematicians/31665#31665&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My answer to this post was Asymptotic statistics from Van der Vaart &lt;a href=&quot;http://www.cambridge.org/catalogue/catalogue.asp?isbn=9780521784504&quot; rel=&quot;nofollow&quot;&gt;http://www.cambridge.org/catalogue/catalogue.asp?isbn=9780521784504&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="223" LastActivityDate="2010-07-21T13:53:13.193" CommunityOwnedDate="2010-07-21T13:53:13.193" CommentCount="1" />
  <row Id="416" PostTypeId="2" ParentId="269" CreationDate="2010-07-21T14:00:03.147" Score="8" ViewCount="" Body="&lt;p&gt;The population is the set of entities under study. For example, the mean height of men. This is a hypothetical population because it includes all men that have lived, are alive and will live in the future. I like this example because it drives home the point that we, as analysts, choose the population that we wish to study. Typically it is impossible to survey/measure the entire population because not all members are observable (e.g. men who will exist in the future). If it is possible to enumerate the entire population it is often costly to do so and would take a great deal of time. In the example above we have a population &quot;men&quot; and a parameter of interest, their height.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead, we could take a subset of this population called a sample and use this sample to draw inferences about the population under study, given some conditions. Thus we could measure the mean height of men in a sample of the population which we call a statistic and use this to draw inferences about the parameter of interest in the population. It is an inference because there will be some uncertainty and inaccuracy involved in drawing conclusions about the population based upon a sample. This should be obvious - we have fewer members in our sample than our population therefore we have lost some information.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are many ways to select a sample and the study of this is called sampling theory. A commonly used method is called Simple Random Sampling (SRS). In SRS each member of the population has an equal probability of being included in the sample, hence the term &quot;random&quot;. There are many other sampling methods e.g. stratified sampling, cluster sampling, etc which all have their advantages and disadvantages.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is important to remember that the sample we draw from the population is only one from a large number of potential samples. If ten researchers were all studying the same population, drawing their own samples then they may obtain different answers. Returning to our earlier example, each of the ten researchers may come up with a different mean height of men i.e. the statistic in question (mean height) varies of sample to sample -- it has a distribution called a sampling distribution. We can use this distribution to understand the uncertainty in our estimate of the population parameter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The sampling distribution of the sample mean is known to be a normal distribution with a standard deviation equal to the sample standard deviation divided by the sample size. Because this could easily be confused with the standard deviation of the sample it more common to call the standard deviation of the sampling distribution the &lt;strong&gt;standard error&lt;/strong&gt;.&lt;/p&gt;&#xA;" OwnerUserId="215" LastActivityDate="2010-07-21T14:00:03.147" />
  <row Id="417" PostTypeId="2" ParentId="414" CreationDate="2010-07-21T14:01:46.887" Score="4" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://www.powells.com/biblio/61-9780691005478-1&quot; rel=&quot;nofollow&quot;&gt;Mathematical Methods of Statistics&lt;/a&gt;, Harald Cramér is really great if you're coming to Statistics from the mathematical side.  It's a bit dated, but still relevant for all the basic mathematical statistics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Two other noteworthy books come to mind for inference and estimation theory:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.powells.com/biblio/9780387985022&quot; rel=&quot;nofollow&quot;&gt;Theory of Point Estimation&lt;/a&gt;, E. L. Lehmann&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.powells.com/biblio/0387945466&quot; rel=&quot;nofollow&quot;&gt;Theory of Statistics&lt;/a&gt;, Schervish&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Not entirely sure if this is what you wanted, but you can check out the reviews and see if they meet your expectations.&lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-07-21T14:01:46.887" CommunityOwnedDate="2010-07-21T14:01:46.887" />
  <row Id="418" PostTypeId="1" CreationDate="2010-07-21T14:30:42.197" Score="5" ViewCount="198" Body="&lt;p&gt;Coming from the field of computer vision, I've often used the &lt;a href=&quot;http://en.wikipedia.org/wiki/RANSAC&quot; rel=&quot;nofollow&quot;&gt;RANSAC&lt;/a&gt; (Random Sample Consensus) method for fitting models to data with lots of outliers. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I've never seen it used by statisticians, and I've always been under the impression that it wasn't considered a &quot;statistically-sound&quot; method. Why is that so? It is random in nature, which makes it harder to analyze, but so are bootstrapping methods. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or is simply a case of academic silos not talking to one another?&lt;/p&gt;&#xA;" OwnerUserId="77" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-07-21T17:30:38.037" LastActivityDate="2010-07-21T17:30:38.037" Title="Why isn't RANSAC most widely used in statistics?" Tags="&lt;outliers&gt;&lt;bootstrap&gt;&lt;robust&gt;" AnswerCount="1" FavoriteCount="2" />
  <row Id="419" PostTypeId="2" ParentId="373" CreationDate="2010-07-21T14:43:31.513" Score="3" ViewCount="" Body="&lt;p&gt;I agree that students find this problem very difficult. The typical response I get is that after you've been shown a goat there's a 50:50 chance of getting the car so why does it matter? Students seem to divorce their first choice from the decision they're now being asked to make i.e. they view these two actions as independent. I then remind them that they were twice as likely to have chosen the wrong door initially hence why they're better off switching. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In recent years I've started actually playing the game in glass and it helps students to understand the problem much better. I use three cardboard toilet roll &quot;middles&quot; and in two of them are paper clips and in the third is a £5 note. &lt;/p&gt;&#xA;" OwnerUserId="215" LastActivityDate="2010-07-21T14:43:31.513" />
  <row Id="420" PostTypeId="2" ParentId="373" CreationDate="2010-07-21T14:55:06.430" Score="2" ViewCount="" Body="&lt;p&gt;I'd modify what Graham Cookson said slightly.  I think the really crucial thing that people overlook is not their first choice, but the &lt;em&gt;host's&lt;/em&gt; choice, and the assumption that the host made sure &lt;em&gt;not&lt;/em&gt; to reveal the car.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In fact, when I discuss this problem in a class, I present it in part as a case study in being clear on your assumptions. It is to your advantage to switch &lt;em&gt;if the host is making sure only to reveal a goat&lt;/em&gt;.  On the other hand, if the host picked randomly between doors 2 and 3, and happened to reveal a goat, then there is no advantage to switching.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Of course, the practical upshot is that if you don't know the host's strategy, you should switch anyway.)&lt;/p&gt;&#xA;" OwnerUserId="89" LastActivityDate="2010-07-21T14:55:06.430" />
  <row Id="421" PostTypeId="1" CreationDate="2010-07-21T15:01:21.127" Score="16" ViewCount="849" Body="&lt;p&gt;What book would you recommend for scientists who are not statisticians?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Clear delivery is most appreciated. As well as the explanation of the appropriate techniques and methods for typical tasks: time series analysis, presentation and aggregation of large data sets.&lt;/p&gt;&#xA;" OwnerUserId="219" LastEditorUserId="2669" LastEditorDisplayName="" LastEditDate="2011-05-04T18:19:29.047" LastActivityDate="2011-05-04T18:19:29.047" CommunityOwnedDate="2010-07-21T15:01:21.127" Title="What book would you recommend for non-statistician scientists?" Tags="&lt;books&gt;&lt;science&gt;" AnswerCount="13" FavoriteCount="12" />
  <row Id="422" PostTypeId="2" ParentId="421" CreationDate="2010-07-21T15:09:11.543" Score="0" ViewCount="" Body="&lt;p&gt;That'll depend very much on their background, but I found &quot;Statistics in a Nutshell&quot; to be pretty good.&lt;/p&gt;&#xA;" OwnerUserId="247" LastActivityDate="2010-07-21T15:09:11.543" CommunityOwnedDate="2010-07-21T15:09:11.543" CommentCount="1" />
  <row Id="423" PostTypeId="1" CreationDate="2010-07-21T15:13:21.493" Score="40" ViewCount="2885" Body="&lt;p&gt;This is one of my favorites:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://imgs.xkcd.com/comics/correlation.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One entry per answer. This is in the vein of the Stack Overflow question &lt;em&gt;&lt;a href=&quot;http://stackoverflow.com/questions/84556/whats-your-favorite-programmer-cartoon&quot;&gt;What’s your favorite “programmer” cartoon?&lt;/a&gt;&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S. Do not hotlink the cartoon without the site's permission please.&lt;/p&gt;&#xA;" OwnerUserId="5" LastEditorUserId="509" LastEditorDisplayName="" LastEditDate="2010-08-11T08:49:02.137" LastActivityDate="2011-05-06T12:13:40.790" CommunityOwnedDate="2010-07-21T15:13:21.493" Title="What is your favorite &quot;data analysis&quot; cartoon?" Tags="&lt;humor&gt;" AnswerCount="26" CommentCount="4" FavoriteCount="23" />
  <row Id="424" PostTypeId="2" ParentId="423" CreationDate="2010-07-21T15:21:33.883" Score="25" ViewCount="" Body="&lt;p&gt;Was XKCD, so time for Dilbert:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://dilbert.com/dyn/str_strip/000000000/00000000/0000000/000000/00000/2000/300/2318/2318.strip.print.gif&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-07-21T15:21:33.883" CommunityOwnedDate="2010-07-21T15:21:33.883" CommentCount="1" />
  <row Id="425" PostTypeId="2" ParentId="423" CreationDate="2010-07-21T15:23:53.183" Score="25" ViewCount="" Body="&lt;p&gt;One of my favorites from &lt;a href=&quot;http://www.xkcd.com&quot; rel=&quot;nofollow&quot;&gt;xckd&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Random Number&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://xkcd.com/221/&quot; rel=&quot;nofollow&quot;&gt;&lt;img src=&quot;http://imgs.xkcd.com/comics/random_number.png&quot; alt=&quot;alt text&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;RFC 1149.5 specifies 4 as the standard IEEE-vetted random number.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="13" LastEditorUserId="13" LastEditorDisplayName="" LastEditDate="2010-07-21T18:19:37.957" LastActivityDate="2010-07-21T18:19:37.957" CommunityOwnedDate="2010-07-21T15:23:53.183" />
  <row Id="427" PostTypeId="2" ParentId="31" CreationDate="2010-07-21T15:33:11.690" Score="6" ViewCount="" Body="&lt;p&gt;&lt;em&gt;Before touching this topic, I always make sure that students are happy moving between percentages, decimals, odds and fractions. If they are not completely happy with this then they can get confused very quickly.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I like to explain hypothesis testing for the first time (and therefore p-values and test statistics) through Fisher's classic tea experiment. I have several reasons for this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(i) I think working through an experiment and defining the terms as we go along makes more sense that just defining all of these terms to begin with.&#xA;(ii) You don't need to rely explicitly on probability distributions, areas under the curve, etc to get over the key points of hypothesis testing.&#xA;(iii) It explains this ridiculous notion of &quot;as or more extreme than those observed&quot; in a fairly sensible manner&#xA;(iv) I find students like to understand the history, origins and back story of what they are studying as it makes it more real than some abstract theories.&#xA;(v) It doesn't matter what discipline or subject the students come from, they can relate to the example of tea (N.B. Some international students have difficulty with this peculiarly British institution of tea with milk.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[Note: I originally got this idea from Dennis Lindley's wonderful article &quot;The Analysis of Experimental Data: The Appreciation of Tea &amp;amp; Wine&quot; in which he demonstrates why Bayesian methods are superior to classical methods.]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The back story is that Muriel Bristol visits Fisher one afternoon in the 1920's at Rothamsted Experimental Station for a cup of tea. When Fisher put the milk in last she complained saying that she could also tell whether the milk was poured first (or last) and that she preferred the former. To put this to the test he designed his classic tea experiment where Muriel is presented with a pair of tea cups and she must identify which one had the milk added first. This is repeated with six pairs of tea cups. Her choices are either Right (R) or Wrong (W) and her results are: RRRRRW.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose that Muriel is actually just guessing and has no ability to discriminate whatsoever. This is called the &lt;strong&gt;Null Hypothesis&lt;/strong&gt;. According to Fisher the purpose of the experiment is to discredit this null hypothesis. If Muriel is guessing she will identify the tea cup correctly with probability 0.5 on each turn and as they are independent the observed result has 0.5^6 = 0.016 (or 1/64). Fisher then argues that either:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(a) the null hypothesis (Muriel is guessing) is true and an event of small probability has occurred or,&#xA;(b) the null hypothesis is false and Muriel has discriminatory powers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The p-value (or probability value) is the probability of observing this outcome (RRRRRW) given the null hypothesis is true - it's the small probability referred to in (a), above. In this instance it's 0.016. Since events with small probabilities only occur rarely (by definition) situation (b) might be a more preferable explanation of what occurred than situation (a). When we reject the null hypothesis we're in fact accepting the opposite hypothesis which is we call the alternative hypothesis. In this example, Muriel has discriminatory powers is the alternative hypothesis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An important consideration is what do we class as a &quot;small&quot; probability? What's the cutoff point at which we're willing to say that an event is unlikely? The standard benchmark is 5% (0.05) and this is called the significance level. When the p-value is smaller than the significance level we reject the null hypothesis as being false and accept our alternative hypothesis. It is common parlance to claim a result is &quot;significant&quot; when the p-value is smaller than the significance level i.e. when the probability of what we observed occurring given the null hypothesis is true is smaller than our cutoff point. It is important to be clear that using 5% is completely subjective (as is using the other common significance levels of 1% and 10%). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Fisher realised that this doesn't work; every possible outcome with one wrong pair was equally suggestive of discriminatory powers. The relevant probability for situation (a), above, is therefore 6(0.5)^6 = 0.094 (or 6/64) which now is &lt;strong&gt;not significant&lt;/strong&gt; at a significance level of 5%. To overcome this Fisher argued that if 1 error in 6 is considered evidence of discriminatory powers then so is no errors i.e. outcomes that more strongly indicate discriminatory powers than the one observed should be included when calculating the p-value. This resulted in the following amendment to the reasoning, either:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(a) the null hypothesis (Muriel is guessing) is true and the probability of events as, or more, extreme than that observed is small, or&#xA;(b) the null hypothesis is false and Muriel has discriminatory powers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Back to our tea experiment and we find that the p-value under this set-up is 7(0.5)^6 = 0.109 which still is not significant at the 5% threshold. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I then get students to work with some other examples such as coin tossing to work out whether or not a coin is fair. This drills home the concepts of the null/alternative hypothesis, p-values and significance levels. We then move onto the case of a continuous variable and introduce the notion of a test-statistic. As we have already covered the normal distribution, standard normal distribution and the z-transformation in depth it's merely a matter of bolting together several concepts. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As well as calculating test-statistics, p-values and making a decision (significant/not significant) I get students to work through published papers in a fill in the missing blanks game. &lt;/p&gt;&#xA;" OwnerUserId="215" LastActivityDate="2010-07-21T15:33:11.690" />
  <row Id="428" PostTypeId="2" ParentId="346" CreationDate="2010-07-21T15:35:45.560" Score="1" ViewCount="" Body="&lt;p&gt;Could you group the data set into much smaller data sets (say 100 or 1000 or 10,000 data points) If you then calculated the median of each of the groups. If you did this with enough data sets you could plot something like the average of the results of each of the smaller sets and this woul, by running enough smaller data sets converge to an 'average' solution. &lt;/p&gt;&#xA;" OwnerUserId="210" LastActivityDate="2010-07-21T15:35:45.560" CommentCount="2" />
  <row Id="429" PostTypeId="2" ParentId="362" CreationDate="2010-07-21T15:36:34.687" Score="7" ViewCount="" Body="&lt;p&gt;Briefly stated, the Shapiro-Wilk test is a specific test for normality, whereas the method used by &lt;a href=&quot;http://www.itl.nist.gov/div898/handbook/eda/section3/eda35g.htm&quot; rel=&quot;nofollow&quot;&gt;Kolmogorov-Smirnov test&lt;/a&gt; is more general, but less powerful (meaning it correctly rejects the null hypothesis of normality less often). Both statistics take normality as the null and establishes a test statistic based on the sample, but how they do so is different from one another in ways that make them more or less sensitive to features of normal distributions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How exactly W (the test statistic for Shapiro-Wilk) is calculated is &lt;a href=&quot;http://www.itl.nist.gov/div898/software/dataplot/refman1/auxillar/wilkshap.htm&quot; rel=&quot;nofollow&quot;&gt;a bit involved&lt;/a&gt;, but conceptually, it involves arraying the sample values by size and measuring fit against expected means, variances and covariances.  These multiple comparisons against normality, as I understand it, give the test more power than the the Kolmogorov-Smirnov test, which is one way in which they may differ.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By contrast, the Kolmogorov-Smirnov test for normality is derived from a general approach for assessing goodness of fit by comparing the expected cumulative distribution against the empirical cumulative distribution, vis:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://www.itl.nist.gov/div898/handbook/eda/section3/gif/ecdf.gif&quot; alt=&quot;alt text&quot;&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As such, it is sensitive at the center of the distribution, and not the tails.  However, the K-S is test is convergent, in the sense that as n tends to infinity, the test converges to the true answer in probability (I believe that &lt;a href=&quot;http://en.wikipedia.org/wiki/Glivenko%E2%80%93Cantelli_theorem&quot; rel=&quot;nofollow&quot;&gt;Glivenko-Cantelli Theorem&lt;/a&gt; applies here, but someone may correct me). These are two more ways in which these two tests might differ in their evaluation of normality.&lt;/p&gt;&#xA;" OwnerUserId="39" LastActivityDate="2010-07-21T15:36:34.687" CommentCount="1" />
  <row Id="430" PostTypeId="2" ParentId="373" CreationDate="2010-07-21T15:37:17.370" Score="3" ViewCount="" Body="&lt;p&gt;I find that people find the solution more intuitive if you change it to 100 doors, close first, second, to 98 doors.  Similarly for 50 doors, etc.  &lt;/p&gt;&#xA;" OwnerUserId="251" LastEditorUserId="251" LastEditorDisplayName="" LastEditDate="2010-07-22T00:37:06.953" LastActivityDate="2010-07-22T00:37:06.953" CommentCount="2" />
  <row Id="431" PostTypeId="2" ParentId="363" CreationDate="2010-07-21T15:38:11.677" Score="3" ViewCount="" Body="&lt;p&gt;I think every statistician should read Stigler's &lt;em&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/067440341X&quot; rel=&quot;nofollow&quot;&gt;The History of Statistics: The Measurement of Uncertainty before 1900&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is beautifully written, thorough and it isn't a historian's perspective but a mathematician's, hence it doesn't avoid the technical details. &lt;/p&gt;&#xA;" OwnerUserId="215" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2011-02-20T02:34:07.180" LastActivityDate="2011-02-20T02:34:07.180" CommunityOwnedDate="2010-07-21T15:38:11.677" />
  <row Id="432" PostTypeId="2" ParentId="423" CreationDate="2010-07-21T15:38:46.170" Score="25" ViewCount="" Body="&lt;p&gt;My favourite Dilbert cartoon:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://dilbert.com/dyn/str_strip/000000000/00000000/0000000/000000/00000/5000/600/5651/5651.strip.gif&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="8" LastActivityDate="2010-07-21T15:38:46.170" CommunityOwnedDate="2010-07-21T15:38:46.170" CommentCount="1" />
  <row Id="433" PostTypeId="2" ParentId="423" CreationDate="2010-07-21T15:43:44.457" Score="14" ViewCount="" Body="&lt;p&gt;Here's &lt;a href=&quot;http://dilbert.com/strips/comic/2010-07-02/&quot; rel=&quot;nofollow&quot;&gt;another one from Dilbert&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://dilbert.com/dyn/str_strip/000000000/00000000/0000000/000000/90000/3000/400/93464/93464.strip.gif&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-07-21T15:43:44.457" CommunityOwnedDate="2010-07-21T15:43:44.457" />
  <row Id="434" PostTypeId="2" ParentId="22" CreationDate="2010-07-21T15:50:43.377" Score="11" ViewCount="" Body="&lt;p&gt;&lt;em&gt;Very&lt;/em&gt; crudely I would say that:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Frequentist:&lt;/strong&gt; Sampling is infinite and decision rules can be sharp. Data are a repeatable random sample - there is a frequency. Underlying parameters are fixed i.e. they remain constant during this repeatable sampling process.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Bayesian:&lt;/strong&gt;  Unknown quantities are treated probabilistically and the state of the world can always be updated. Data are observed from the realised sample. Parameters are unknown and described probabilistically. It is the data which are fixed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a brilliant &lt;a href=&quot;http://behind-the-enemy-lines.blogspot.com/2008/01/are-you-bayesian-or-frequentist-or.html&quot; rel=&quot;nofollow&quot;&gt;blog post&lt;/a&gt; which gives an indepth example of how a Bayesian and Frequentist would tackle the same problem. Why not answer the problem for yourself and then check?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem (taken from Panos Ipeirotis' blog):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You have a coin that when flipped ends up head with probability p and ends up tail with probability 1-p. (The value of p is unknown.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Trying to estimate p, you flip the coin 100 times. It ends up head 71 times.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then you have to decide on the following event: &quot;In the next two tosses we will get two heads in a row.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would you bet that the event will happen or that it will not happen?&lt;/p&gt;&#xA;" OwnerUserId="215" LastActivityDate="2010-07-21T15:50:43.377" CommentCount="0" />
  <row Id="435" PostTypeId="2" ParentId="423" CreationDate="2010-07-21T15:53:59.553" Score="27" ViewCount="" Body="&lt;p&gt;One more &lt;a href=&quot;http://dilbert.com/fast/2008-05-08/&quot; rel=&quot;nofollow&quot;&gt;Dilbert&lt;/a&gt;:&#xA;&lt;img src=&quot;http://dilbert.com/dyn/str_strip/000000000/00000000/0000000/000000/00000/5000/600/5652/5652.strip.print.gif&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-07-21T15:53:59.553" CommunityOwnedDate="2010-07-21T15:53:59.553" CommentCount="1" />
  <row Id="436" PostTypeId="2" ParentId="421" CreationDate="2010-07-21T16:04:18.627" Score="5" ViewCount="" Body="&lt;p&gt;The answer would most definitely depend on their discipline, the methods/techniques that they would like to learn and their existing mathematical/statistical abilities.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, economists/social scientists who want to learn about cutting edge empirical econometrics could read Angrist and Pischke's &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0691120358&quot; rel=&quot;nofollow&quot;&gt;Mostly Harmless Econometrics&lt;/a&gt;. This is a non-technical book covering the &quot;natural experimental revolution&quot; in economics. The book only presupposes that they know what regression is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I think the best book on applied regression is Gelman and Hill's &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/052168689X&quot; rel=&quot;nofollow&quot;&gt;Data Analysis Using Regression and Multilevel/Hierarchical Models&lt;/a&gt;. This covers basic regression, multilevel regression, and Bayesian methods in a clear and intuitive way. It would be good for any scientist with a basic background in statistics.&lt;/p&gt;&#xA;" OwnerUserId="215" LastEditorUserId="509" LastEditorDisplayName="" LastEditDate="2010-08-11T08:46:15.107" LastActivityDate="2010-08-11T08:46:15.107" CommunityOwnedDate="2010-07-21T16:04:18.627" />
  <row Id="437" PostTypeId="2" ParentId="414" CreationDate="2010-07-21T16:07:20.810" Score="1" ViewCount="" Body="&lt;p&gt;For you I would suggest:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Introduction to the Mathematical and Statistical Foundations of Econometrics by Herman J. Bierens, CUP. The word &quot;Introduction&quot; in the title is a sick joke for most PhD econometrics students.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Markov Chain Monte Carlo by Dani Gamerman, Chapman &amp;amp; Hall is also concise.&lt;/p&gt;&#xA;" OwnerUserId="215" LastActivityDate="2010-07-21T16:07:20.810" CommunityOwnedDate="2010-07-21T16:07:20.810" CommentCount="1" />
  <row Id="438" PostTypeId="2" ParentId="165" CreationDate="2010-07-21T16:42:43.820" Score="5" ViewCount="" Body="&lt;p&gt;OK here's my best attempt at an informal and crude explanation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A Markov Chain is a random process that has the property that the future depends only on the current state of the process and not the past i.e. it is memoryless. An example of a random process could be the stock exchange. An example of a Markov Chain would be a board game like Monopoly or Snakes and Ladders where your future position (after rolling the die) would depend only on where you started from before the roll, not any of your previous positions. A textbook example of a Markov Chain is the &quot;drunkard's walk&quot;. Imagine somebody who is drunk and can move only left or right by one pace. The drunk moves left or right with equal probability. This is a Markov Chain where the drunk's future/next position depends only upon where he is at present.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Monte Carlo methods are computational algorithms (simply sets of instructions) which randomly sample from some process under study. They are a way of estimating something which is too difficult or time consuming to find deterministically. They're basically a form of computer simulation of some mathematical or physical process. The Monte Carlo moniker comes from the analogy between a casino and random number generation. Returning to our board game example earlier, perhaps we want to know if some properties on the Monopoly board are visited more often than others. A Monte Carlo experiment would involve rolling the dice repeatedly and counting the number of times you land on each property. It can also be used for calculating numerical integrals. (Very informally, we can think of an integral as the area under the graph of some function.)  Monte Carlo integration works great on a high-dimensional functions by taking a random sample of points of the function and calculating some type of average at these various points. By increasing the sample size, the law of large numbers tells us we can increase the accuracy of our approximation by covering more and more of the function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These two concepts can be put together to solve some difficult problems in areas such as Bayesian inference, computational biology, etc where multi-dimensional integrals need to be calculated to solve common problems. The idea is to construct a Markov Chain which converges to the desired probability distribution after a number of steps. The state of the chain after a large number of steps is then used as a sample from the desired distribution and the process is repeated. There many different MCMC algorithms which use different techniques for generating the Markov Chain. Common ones include the Metropolis-Hastings and the Gibbs Sampler.&lt;/p&gt;&#xA;" OwnerUserId="215" LastActivityDate="2010-07-21T16:42:43.820" CommunityOwnedDate="2010-08-16T06:37:30.433" />
  <row Id="439" PostTypeId="2" ParentId="414" CreationDate="2010-07-21T16:50:35.697" Score="6" ViewCount="" Body="&lt;p&gt;As you said, it's not necessarily the case that a mathematician may want a rigorous book. Maybe the goal is to get some intuition of the concepts quickly, and then fill in the details. I recommend two books from CMU professors, both published by Springer: &quot;All of Statistics&quot; by Larry Wasserman is quick and informal. &quot;Theory of Statistics&quot; by Mark Schervish is rigorous and relatively complete. It has decision theory, finite sample, some asymptotics and sequential analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Added 7/28/10: There is one additional reference that is orthogonal to the other two: very rigorous, focused on learning theory, and short. It's by Smale (Steven Smale!) and Cucker, &quot;&lt;a href=&quot;http://www.ams.org/journals/bull/2002-39-01/S0273-0979-01-00923-5/home.html&quot; rel=&quot;nofollow&quot;&gt;On the Mathematical Foundations of Learning&lt;/a&gt;&quot;. Not easy read, but the best crash course on the theory.&lt;/p&gt;&#xA;" OwnerUserId="30" LastEditorUserId="30" LastEditorDisplayName="" LastEditDate="2010-07-29T02:46:37.897" LastActivityDate="2010-07-29T02:46:37.897" CommunityOwnedDate="2010-07-21T16:50:35.697" CommentCount="1" />
  <row Id="440" PostTypeId="2" ParentId="421" CreationDate="2010-07-21T16:53:35.677" Score="2" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0387790535&quot; rel=&quot;nofollow&quot;&gt;Peter Dalgaard's Introductory Statistics with R&lt;/a&gt; is a great book for some introductory statistics with a focus on the R software for data analysis.&lt;/p&gt;&#xA;" OwnerUserId="36" LastActivityDate="2010-07-21T16:53:35.677" CommunityOwnedDate="2010-07-21T16:53:35.677" />
  <row Id="441" PostTypeId="2" ParentId="290" CreationDate="2010-07-21T16:54:38.483" Score="5" ViewCount="" Body="&lt;p&gt;UCLA has the best free resources you'll find anywhere.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.ats.ucla.edu/stat/stata/&quot; rel=&quot;nofollow&quot;&gt;http://www.ats.ucla.edu/stat/stata/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="36" LastActivityDate="2010-07-21T16:54:38.483" />
  <row Id="442" PostTypeId="2" ParentId="363" CreationDate="2010-07-21T16:56:45.453" Score="4" ViewCount="" Body="&lt;p&gt;I say the &lt;a href=&quot;http://amazon.com/o/ASIN/0961392142/ref=nosim/gettgenedone-20&quot; rel=&quot;nofollow&quot;&gt;visual display of quantitative information&lt;/a&gt; by Tufte, and &lt;a href=&quot;http://amazon.com/o/ASIN/0060731338/ref=nosim/gettgenedone-20&quot; rel=&quot;nofollow&quot;&gt;Freakonomics&lt;/a&gt; for something fun.&lt;/p&gt;&#xA;" OwnerUserId="36" LastActivityDate="2010-07-21T16:56:45.453" CommunityOwnedDate="2010-07-21T16:56:45.453" />
  <row Id="443" PostTypeId="2" ParentId="36" CreationDate="2010-07-21T17:06:32.303" Score="4" ViewCount="" Body="&lt;p&gt;There's two aspects to this &lt;em&gt;post hoc ergo propter hoc&lt;/em&gt; problem that I like to cover: (i) reverse causality and (ii) endogeneity &lt;/p&gt;&#xA;&#xA;&lt;p&gt;An example of &quot;possible&quot; reverse causality:&#xA;Social drinking and earnings - drinkers earn more money according to Bethany L. Peters &amp;amp; Edward Stringham, 2006. &quot;No Booze? You May Lose: Why Drinkers Earn More Money Than Nondrinkers,&quot; Journal of Labor Research, Transaction Publishers, vol. 27(3), pages 411-421, June. Or do people who earn more money drink more either because they have a greater disposable income or due to stress? This is a great paper to discuss for all sorts of reasons including measurement error, response bias, causality, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An example of &quot;possible&quot; endogeneity:&#xA;The Mincer Equation explains log earnings by education, experience and experience squared. There is a long literature on this topic. Labour economists want to estimate the causal relationship of education on earnings but perhaps education is endogenous because &quot;ability&quot; could increase the amount of education an individual has (by lowering the cost of obtaining it) and could lead to an increase in earnings, irrespective of the level of education. A potential solution to this could be an instrumental variable. Angrist and Pischke's book, Mostly Harmless Econometrics covers this and relates topics in great detail and clarity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other silly examples that I have no support for include:&#xA;- Number of televisions per capita and the numbers of mortality rate. So let's send TVs to developing countries. Obviously both are endogenous to something like GDP.&#xA;- Number of shark attacks and ice cream sales. Both are endogenous to the temperature perhaps?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also like to tell the terrible joke about the lunatic and the spider. A lunatic is wandering the corridors of an asylum with a spider he's carrying in the palm of his hand. He sees the doctor and says, &quot;Look Doc, I can talk to spiders. Watch this. &quot;Spider, go left!&quot; The spider duly moves to the left. He continues, &quot;Spider, go right.&quot; The spider shuffles to the right of his palm. The doctor replies, &quot;Interesting, maybe we should talk about this in the next group session.&quot; The lunatic retorts, &quot;That's nothing Doc. Watch this.&quot; He pulls off each of the spider's legs one by one and then shouts, &quot;Spider, go left!&quot; The spider lies motionless on his palm and the lunatic turns to the doctor and concludes, &quot;If you pull off a spider's legs he'll go deaf.&quot; &lt;/p&gt;&#xA;" OwnerUserId="215" LastEditorUserId="215" LastEditorDisplayName="" LastEditDate="2010-07-23T10:08:40.817" LastActivityDate="2010-07-23T10:08:40.817" CommunityOwnedDate="2010-08-16T13:01:42.613" />
  <row Id="445" PostTypeId="2" ParentId="290" CreationDate="2010-07-21T17:13:33.117" Score="2" ViewCount="" Body="&lt;p&gt;The UCLA resource listed by Stephen Turner (above) are excellent if you just want to apply methods you're already familiar with using Stata.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're looking for textbooks which teach you statistics/econometrics while using Stata then these are solid recommendations (but it depends at what level you're looking at):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Introductory Methods&lt;/strong&gt;&#xA;An Introduction to Modern Econometrics Using Stata by Chris Baum&#xA;Introduction to Econometrics by Chris Dougherty&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Advanced/Specialised Methods&lt;/strong&gt;&#xA;Multilevel and Longitudinal Modeling Using Stata by Rabe-Hesketh and Skrondal&#xA;Regression Models for Categorical Dependent Variables Using Stata by Long and Freese&lt;/p&gt;&#xA;" OwnerUserId="215" LastActivityDate="2010-07-21T17:13:33.117" CommentCount="1" />
  <row Id="446" PostTypeId="2" ParentId="125" CreationDate="2010-07-21T17:17:45.780" Score="4" ViewCount="" Body="&lt;p&gt;The Gelman books are all excellent but not necessarily introductory in that they assume that you know some statistics already. Therefore they are an introduction to the Bayesian way of doing statistics rather than to statistics in general. I would still give them the thumbs up, however.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an introductory statistics/econometrics book which takes a Bayesian perspective, I would recommend Gary Koop's Bayesian Econometrics.&lt;/p&gt;&#xA;" OwnerUserId="215" LastActivityDate="2010-07-21T17:17:45.780" CommunityOwnedDate="2010-07-21T17:17:45.780" />
  <row Id="447" PostTypeId="2" ParentId="170" CreationDate="2010-07-21T17:26:14.377" Score="0" ViewCount="" Body="&lt;p&gt;I really like these two books by Daniel McFadden of Berkeley:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://elsa.berkeley.edu/users/mcfadden/e240a_sp98/e240a.html&quot; rel=&quot;nofollow&quot;&gt;http://elsa.berkeley.edu/users/mcfadden/e240a_sp98/e240a.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://elsa.berkeley.edu/users/mcfadden/e240b_f01/e240b.html&quot; rel=&quot;nofollow&quot;&gt;http://elsa.berkeley.edu/users/mcfadden/e240b_f01/e240b.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="215" LastActivityDate="2010-07-21T17:26:14.377" />
  <row Id="448" PostTypeId="2" ParentId="396" CreationDate="2010-07-21T17:26:15.483" Score="5" ViewCount="" Body="&lt;p&gt;Here are my guidelines, based on the most common errors I see (in addition to all the other good points mentioned)&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Use scatter graphs, not line plots, if element order is not relevant.&lt;/li&gt;&#xA;&lt;li&gt;When preparing plots that are meant to be compared, use the same scale factor for all of them. &lt;/li&gt;&#xA;&lt;li&gt;Even better - find a way to combine the data in a single graph (eg: boxplots are a better than several histograms to compare a large number of distributions).&lt;/li&gt;&#xA;&lt;li&gt;Do not forget to specify units&lt;/li&gt;&#xA;&lt;li&gt;Use a legend only if you must - it's generally clearer to label curves directly. &lt;/li&gt;&#xA;&lt;li&gt;If you must use a legend, move it inside the plot, in a blank area.&lt;/li&gt;&#xA;&lt;li&gt;For line graphs, aim for an aspect ratio which yields &lt;a href=&quot;http://processtrends.com/pg_data_vis_bank_to_45.htm&quot; rel=&quot;nofollow&quot;&gt;lines that are roughly at 45o with the page&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="77" LastActivityDate="2010-07-21T17:26:15.483" CommunityOwnedDate="2011-06-01T03:42:00.250" />
  <row Id="449" PostTypeId="2" ParentId="328" CreationDate="2010-07-21T17:28:57.803" Score="1" ViewCount="" Body="&lt;p&gt;More from an economics perspectives I think these two sets of lecture notes are very good:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://home.datacomm.ch/paulsoderlind/Courses/OldCourses/FinEcmtAll.pdf&quot; rel=&quot;nofollow&quot;&gt;http://home.datacomm.ch/paulsoderlind/Courses/OldCourses/FinEcmtAll.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://personal.lse.ac.uk/mele/files/fin_eco.pdf&quot; rel=&quot;nofollow&quot;&gt;http://personal.lse.ac.uk/mele/files/fin_eco.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first provides econometric methods for analysing financial data whereas the second provides the financial economics theory behind the models being applied. They're both MSc level texts.&lt;/p&gt;&#xA;" OwnerUserId="215" LastActivityDate="2010-07-21T17:28:57.803" />
  <row Id="450" PostTypeId="2" ParentId="418" CreationDate="2010-07-21T17:30:02.077" Score="3" ViewCount="" Body="&lt;p&gt;For us, it is just one example of a robust regression -- I believe it is used by statisticians also, but maybe not so wide because it has some better known alternatives.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-07-21T17:30:02.077" CommentCount="2" />
  <row Id="451" PostTypeId="2" ParentId="26" CreationDate="2010-07-21T17:38:08.583" Score="3" ViewCount="" Body="&lt;p&gt;When describing a variable we typically summarise it using two measures: a measure of centre and a measure of spread. Common measures of spread include the mean, median and mode. Common measure of spread include the variance and interquartile range.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The variance (represented by the Greek lowercase sigma) is commonly used when the mean is reported. The variance is the average squared deviation of variable. The deviation is calculated by subtracting the mean from each observation. This is squared because the sum would otherwise be zero and squaring removes this problem while maintaining the relative size of the deviations. The problem with using the variation as a measure of spread is that it is in squared units. For example if our variable of interest was height measured in inches then the variance would be reported in squared-inches which makes little sense. The standard deviation (represented by the Greek lowercase sigma raised to the power 2) is the square-root of the variance and returns the measure of spread to the original units. This is much more intuitive and is therefore more popular than the variance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When using the standard deviation, one has to be careful of outliers as they will skew the standard deviation (and the mean) as they are not resistant measures of spread. A simple example will illustrate this property. The mean of my terrible cricket batting scores of 13, 14, 16, 23, 26, 28, 33, 39, and 61 is 28.11. If we consider 61 to be an outlier and deleted it, the mean would be 24. &lt;/p&gt;&#xA;" OwnerUserId="215" LastActivityDate="2010-07-21T17:38:08.583" />
  <row Id="452" PostTypeId="1" AcceptedAnswerId="711" CreationDate="2010-07-21T17:45:01.570" Score="6" ViewCount="841" Body="&lt;p&gt;It has been suggested by Angrist and Pischke that Robust (i.e. robust to heteroskedasticity or unequal variances) Standard Errors are reported as a matter of course rather than testing for it. Two questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;What is impact on the standard errors of doing so when there is homoskedasticity?&lt;/li&gt;&#xA;&lt;li&gt;Does anybody actually do this in their work?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="215" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-08-13T13:35:57.700" LastActivityDate="2010-12-19T00:59:28.370" Title="Always Report Robust (White) Standard Errors?" Tags="&lt;regression&gt;&lt;error&gt;&lt;standard-error&gt;" AnswerCount="3" CommentCount="4" FavoriteCount="1" />
  <row Id="453" PostTypeId="2" ParentId="421" CreationDate="2010-07-21T17:54:05.170" Score="3" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/041587291X&quot; rel=&quot;nofollow&quot;&gt;Statistics in Plain English&lt;/a&gt; is pretty good.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4.5 on Amazon, 11 reviews.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Explains ANOVA pretty well too. &lt;/p&gt;&#xA;" OwnerUserId="74" LastActivityDate="2010-07-21T17:54:05.170" CommunityOwnedDate="2010-07-21T17:54:05.170" />
  <row Id="454" PostTypeId="2" ParentId="396" CreationDate="2010-07-21T18:11:59.113" Score="1" ViewCount="" Body="&lt;p&gt;It depends on the way in which the plots will be discussed. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, if I'm sending out plots for a group meeting that will be done with callers from different locations, I prefer putting them together in Powerpoint as opposed to Excel, so it's easier to flip around. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For one-on-one technical calls, I'll put something in excel so that the client be able to move a plot aside, and view the raw data. Or, I can enter p-values into cells along side regression coefficients, e.g. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Keep in mind: plots are cheap, especially for a slide show, or for emailing to a group. I'd rather make 10 clear plots that we can flip through than 5 plots where I try to put distinct cohorts (e.g. &quot;males and females&quot;) on the same graph. &lt;/p&gt;&#xA;" OwnerUserId="62" LastActivityDate="2010-07-21T18:11:59.113" CommunityOwnedDate="2011-06-01T03:42:00.250" />
  <row Id="455" PostTypeId="2" ParentId="423" CreationDate="2010-07-21T18:18:13.170" Score="12" ViewCount="" Body="&lt;p&gt;Allright, I think this one is hilarious- but let's see if it passes the Statistical Analysis Miller test.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Fermirotica&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://xkcd.com/563/&quot; rel=&quot;nofollow&quot;&gt;&lt;img src=&quot;http://imgs.xkcd.com/comics/fermirotica.png&quot; alt=&quot;I love how Google handles dimensional analysis.  Stats are ballpark and vary wildly by time of day and whether your mom is in town.&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;I love how Google handles dimensional analysis.  Stats are ballpark and vary wildly by time of day and whether your mom is in town.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="13" LastEditorUserId="237" LastEditorDisplayName="" LastEditDate="2010-07-23T15:27:06.463" LastActivityDate="2010-07-23T15:27:06.463" CommunityOwnedDate="2010-07-21T18:18:13.170" CommentCount="1" />
  <row Id="456" PostTypeId="2" ParentId="277" CreationDate="2010-07-21T18:26:02.873" Score="5" ViewCount="" Body="&lt;p&gt;As the &lt;a href=&quot;http://books.google.com/books?id=6q2lOfLnwkAC&amp;amp;dq=Encyclopedia+of+GIS&quot; rel=&quot;nofollow&quot;&gt;Encyclopedia of GIS&lt;/a&gt; states, the conditional autoregressive model (CAR) is appropriate for situations with first order dependency or relatively local spatial autocorrelation, and simultaneous autoregressive model (SAR) is  more suitable where there are second order dependency or a more global spatial autocorrelation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is made clear by the fact that CAR obeys the spatial version of the &lt;a href=&quot;http://en.wikipedia.org/wiki/Markov_property&quot; rel=&quot;nofollow&quot;&gt;Markov property&lt;/a&gt;, namely it assumes that the state of a particular area is influenced its neighbors and not neighbors of neighbors, etc. (i.e. it is spatially “memoryless”, instead of temporally), whereas SAR does not assume such.  This is due to the different ways in which they specify their variance-covariance matrixes.  So, when the spatial Markov property obtains, CAR provides a simpler way to model autocorrelated geo-referenced areal data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See &lt;a href=&quot;http://www.geog.ucsb.edu/~good/papers/387.pdf&quot; rel=&quot;nofollow&quot;&gt;Gis And Spatial Data Analysis: Converging Perspectives&lt;/a&gt; for more details.&lt;/p&gt;&#xA;" OwnerUserId="39" LastEditorUserId="39" LastEditorDisplayName="" LastEditDate="2010-07-27T19:11:57.330" LastActivityDate="2010-07-27T19:11:57.330" />
  <row Id="457" PostTypeId="2" ParentId="359" CreationDate="2010-07-21T18:28:27.423" Score="3" ViewCount="" Body="&lt;p&gt;I do not know the literature in the area well enough to offer a direct response. However, it seems to me that if the three tests differ then that is an indication that you need further research/data collection in order to definitively answer your question. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may also want to look at &lt;a href=&quot;http://scholar.google.com/scholar?q=small%20sample%20properties%20of%20wald%20likelihood%20ratio&amp;amp;um=1&amp;amp;ie=UTF-8&amp;amp;sa=N&amp;amp;hl=en&amp;amp;tab=ws&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; Google Scholar search&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update in response to your comment:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If collecting additional data is not possible then there is one workaround. Do a simulation which mirrors your data structure, sample size and your proposed model. You can set the parameters to some pre-specified values. Estimate the model using the data generated and then check which one of the three tests points you to the right model. Such a simulation would offer some guidance as to which test to use for your real data. Does that make sense?&lt;/p&gt;&#xA;" LastActivityDate="2010-07-22T01:21:24.837" CommentCount="3" />
  <row Id="458" PostTypeId="2" ParentId="194" CreationDate="2010-07-21T18:41:12.033" Score="1" ViewCount="" Body="&lt;p&gt;One way to test patterns in stock market data is discussed &lt;a href=&quot;http://www.evidencebasedta.com/&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;. A similar approach would be to randomise the stock market data and identify your patterns of interest, which would obviously be devoid of any meaning due to the deliberate randomising process. These randomly generated patterns and their returns would form your null hypothesis. By statistically comparing the pattern returns in the actual data with the returns from the null hypothesis randomised data patterns you may be able to distinguish patterns which actually have some meaning or predictive value. &lt;/p&gt;&#xA;" OwnerUserId="226" LastActivityDate="2010-07-21T18:41:12.033" />
  <row Id="459" PostTypeId="1" CreationDate="2010-07-21T19:59:48.020" Score="4" ViewCount="153" Body="&lt;p&gt;We're plotting time-series metrics in the context of network/server operations. The data has a 5-minute sample rate, and consists of things like CPU utilization, error rate, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We're adding a horizontal &quot;threshold&quot; line to the graphs, to visually indicate a value threshold above which people should worry/take notice. For example, in the CPU utilization example, perhaps the &quot;worry&quot; threshold is 75%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My team has some internal debate over what color this line should be:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Something like a bright red that clearly stands out from the background grid and data lines, and indicates this is a warning condition&lt;/li&gt;&#xA;&lt;li&gt;Something more subtle and definitely NOT red, since the &quot;ink&quot; for the line doesn't represent any actual data, and thus attention shouldn't be drawn to it unnecessarily.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Would appreciate guidance / best practices...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks,&#xA;Mike&lt;/p&gt;&#xA;" OwnerUserId="259" LastActivityDate="2010-08-04T06:51:12.213" Title="Good line color for &quot;threshold&quot; line in a time-series graph?" Tags="&lt;time-series&gt;&lt;data-visualization&gt;" AnswerCount="4" />
  <row Id="460" PostTypeId="2" ParentId="213" CreationDate="2010-07-21T20:03:40.367" Score="3" ViewCount="" Body="&lt;p&gt;For moderate dimensions, like 3, then some sort of kernel cross-validation technique as suggested elsewhere seems reasonable and is the best I can come up with.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For higher dimensions, I'm not sure that the problem is solvable; it lands pretty squarely into 'curse-of-dimensionality' territory.  The issue is that distance functions tend to converge to very large values very quickly as you increase dimensionality, including distances derived from distributions.  If you're defining an outlier as &quot;a point with a comparatively large distance function relative to the others&quot;, and all your distance functions are beginning to converge because you're in a high-dimensional space, well, you're in trouble.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Without some sort of distributional assumption that will let you turn it into a probabilistic classification problem, or at least some rotation that lets you separate your space into &quot;noise dimensions&quot; and &quot;informative dimensions&quot;, I think that the geometry of high-dimensional spaces is going to prohibit any easy -- or at least robust -- identification of outliers.&lt;/p&gt;&#xA;" OwnerUserId="61" LastActivityDate="2010-07-21T20:03:40.367" CommentCount="1" />
  <row Id="461" PostTypeId="2" ParentId="459" CreationDate="2010-07-21T20:07:32.057" Score="1" ViewCount="" Body="&lt;p&gt;To me, whether or not the line represents actual data seems irrelevant.  What's the point of the plot?  If it's so that somebody will do something when utilization crosses a threshold, the line marking the threshold had better be very visible.  If the point of the plot is to give an overview of utilization over time, then why include the line at all?  Just put the major gridlines of your plot at intervals that will coincide with your threshold (25% in your example), and let the reader figure it out.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;... y'all been reading too much Tufte.&lt;/p&gt;&#xA;" OwnerUserId="71" LastActivityDate="2010-07-21T20:07:32.057" CommentCount="3" />
  <row Id="462" PostTypeId="2" ParentId="396" CreationDate="2010-07-21T20:11:02.580" Score="4" ViewCount="" Body="&lt;p&gt;In the physics field there is a rule that the whole paper/report should be understandable only from quick look at the plots. So I would mainly advice that they should be self-explanatory.&lt;br&gt;&#xA;This also implies that you must always check whether your audience is familiar with some kind of plot -- I had once made a big mistake assuming that every scientist knows what boxplots are, and then waisted an hour to explain it.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-07-21T20:11:02.580" CommunityOwnedDate="2011-06-01T03:42:00.250" />
  <row Id="463" PostTypeId="2" ParentId="363" CreationDate="2010-07-21T20:29:12.783" Score="4" ViewCount="" Body="&lt;p&gt;In addition to &quot;The History of Statistics&quot; suggested by Graham, another Stigler book worth reading is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0674009797&quot; rel=&quot;nofollow&quot;&gt;Statistics on the Table: The History of Statistical Concepts and Methods&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="90" LastActivityDate="2010-07-21T20:29:12.783" CommunityOwnedDate="2010-07-21T20:29:12.783" />
  <row Id="464" PostTypeId="2" ParentId="459" CreationDate="2010-07-21T20:34:43.147" Score="2" ViewCount="" Body="&lt;p&gt;If this is about your &quot;Qnotifier&quot; I think that you should plot the threshold line in some darker gray so it is distinguishable but not disturbing. Then I would color the part of the  plot that reaches over the threshold in some alarmistic hue, like red. &lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-07-21T20:34:43.147" CommentCount="2" />
  <row Id="465" PostTypeId="2" ParentId="452" CreationDate="2010-07-21T20:45:08.177" Score="2" ViewCount="" Body="&lt;p&gt;I thought that the White Standard Error and the Standard Error computed in the &quot;normal&quot; way (eg, Hessian and/or OPG in the case of maximum likelihood) were asymptotically equivalent in the case of homoskedasticity? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Only if there is heteroskedasticity will the &quot;normal&quot; standard error be inappropriate, which means that the White Standard Error is appropriate with or without heteroskedasticity, that is, even when your model is homoskedastic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can't really talk about 2, but I don't see the why one wouldn't want to calculate the White SE and include in the results.&lt;/p&gt;&#xA;" OwnerUserId="90" LastActivityDate="2010-07-21T20:45:08.177" />
  <row Id="466" PostTypeId="2" ParentId="170" CreationDate="2010-07-21T20:53:54.800" Score="0" ViewCount="" Body="&lt;p&gt;Not Statistics specific, but a good resource is:  &lt;a href=&quot;http://www.reddit.com/r/mathbooks&quot; rel=&quot;nofollow&quot;&gt;http://www.reddit.com/r/mathbooks&lt;/a&gt;&#xA;Also, George Cain at Georgia Tech maintains a list of freely available maths texts that includes some statistical texts.  &lt;a href=&quot;http://people.math.gatech.edu/~cain/textbooks/onlinebooks.html&quot; rel=&quot;nofollow&quot;&gt;http://people.math.gatech.edu/~cain/textbooks/onlinebooks.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="115" LastActivityDate="2010-07-21T20:53:54.800" CommunityOwnedDate="2010-07-21T20:53:54.800" />
  <row Id="467" PostTypeId="2" ParentId="7" CreationDate="2010-07-21T21:04:03.950" Score="4" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://www.reddit.com/r/datasets&quot; rel=&quot;nofollow&quot;&gt;http://www.reddit.com/r/datasets&lt;/a&gt;  and also, &lt;a href=&quot;http://www.reddit.com/r/opendata&quot; rel=&quot;nofollow&quot;&gt;http://www.reddit.com/r/opendata&lt;/a&gt; both contain a constantly growing list of pointers to various datasets.&lt;/p&gt;&#xA;" OwnerUserId="115" LastActivityDate="2010-07-21T21:04:03.950" CommunityOwnedDate="2010-07-21T21:04:03.950" />
  <row Id="468" PostTypeId="2" ParentId="124" CreationDate="2010-07-21T22:05:04.773" Score="2" ViewCount="" Body="&lt;p&gt;If you're coming from the programming side, one option is to use the &lt;a href=&quot;http://www.nltrk.org&quot; rel=&quot;nofollow&quot;&gt;Natural Language Toolkit&lt;/a&gt; (NLTK) for Python.  There's an O'Reilly book, &lt;a href=&quot;http://www.nltk.org/book&quot; rel=&quot;nofollow&quot;&gt;available freely&lt;/a&gt;, which might be a less dense and more practical introduction to building classifiers for documents among other things.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're interested in beefing up on the statistical side, Roger Levy's book in progress, &lt;a href=&quot;http://idiom.ucsd.edu/~rlevy/textbook/text.html&quot; rel=&quot;nofollow&quot;&gt;Probabilistic Models in Study of Language&lt;/a&gt;, might not be bad to peruse.  It's written for cogsci/compsci grad students starting out with statistical NLP techniques.&lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-07-21T22:05:04.773" />
  <row Id="469" PostTypeId="2" ParentId="170" CreationDate="2010-07-21T22:13:19.353" Score="1" ViewCount="" Body="&lt;p&gt;The &lt;a href=&quot;http://fedc.wiwi.hu-berlin.de/xplore/ebooks/html/csa/&quot; rel=&quot;nofollow&quot;&gt;Handbook of Computation Statistics&lt;/a&gt; (Gentle, Härdle, Mori) is available online and quite good.  You'll also find a number of other books by Härdle on statistics in finance, nonparametrics, among other topics at the &lt;a href=&quot;http://fedc.wiwi.hu-berlin.de/xplore/ebooks/html/&quot; rel=&quot;nofollow&quot;&gt;same site&lt;/a&gt;.  &lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-07-21T22:13:19.353" CommunityOwnedDate="2010-07-21T22:13:19.353" />
  <row Id="470" PostTypeId="2" ParentId="170" CreationDate="2010-07-21T22:35:38.850" Score="11" ViewCount="" Body="&lt;p&gt;The Elements of Statistical Learning by Hastie, Tibshirani, and Friedman is a standard text for statistics and data mining, and is now free:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www-stat.stanford.edu/~tibs/ElemStatLearn/&quot; rel=&quot;nofollow&quot;&gt;http://www-stat.stanford.edu/~tibs/ElemStatLearn/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also Available &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0387848576&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="36" LastEditorUserId="218" LastEditorDisplayName="" LastEditDate="2010-07-24T11:51:57.310" LastActivityDate="2010-07-24T11:51:57.310" CommunityOwnedDate="2010-07-21T22:35:38.850" CommentCount="2" />
  <row Id="471" PostTypeId="2" ParentId="363" CreationDate="2010-07-21T22:57:58.260" Score="4" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0393310728&quot; rel=&quot;nofollow&quot;&gt;Darrell Huff -- How to Lie with Statistics&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="168" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2011-02-20T02:34:52.907" LastActivityDate="2011-02-20T02:34:52.907" CommunityOwnedDate="2010-07-21T22:57:58.260" CommentCount="1" />
  <row Id="472" PostTypeId="2" ParentId="170" CreationDate="2010-07-21T23:00:34.700" Score="1" ViewCount="" Body="&lt;p&gt;For getting into stochastic processes and SDEs, Tom Kurtz's &lt;a href=&quot;http://www.math.wisc.edu/~kurtz/m735.htm&quot; rel=&quot;nofollow&quot;&gt;lecture notes&lt;/a&gt; are hard to beat.  It starts with a decent review of probability and some convergence results, and then dives right into continuous time stochastic processes in fairly clear, comprehensible language.  In general it's one of the best books on the topic -- free or otherwise -- I've found.&lt;/p&gt;&#xA;" OwnerUserId="61" LastActivityDate="2010-07-21T23:00:34.700" CommunityOwnedDate="2010-07-21T23:00:34.700" />
  <row Id="473" PostTypeId="2" ParentId="369" CreationDate="2010-07-22T02:37:18.820" Score="2" ViewCount="" Body="&lt;p&gt;The traditional solution to this problem is to use the &lt;a href=&quot;http://dx.doi.org/10.1137/S0036144598347035&quot; rel=&quot;nofollow&quot;&gt;vector representation&lt;/a&gt; for the news stories and then cluster the vectors.  The vectors are arrays where each entry represents a word or word class.  The value associated to each word will be the &lt;a href=&quot;http://en.wikipedia.org/wiki/TF_IDF&quot; rel=&quot;nofollow&quot;&gt;tf-idf&lt;/a&gt; weight.  This value goes up the more frequent the word in the document and down the more frequent the word is in the whole collection of documents.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may think of the titles as the documents, but sticking to just the title for news stories may be a bit risky for clustering similar stories.  The problem is that by using word counts you are discarding all information on the order of the words. Longer texts compensate for that loss information by distinguishing documents by the vocabulary used (articles mentioning &lt;em&gt;finance&lt;/em&gt;, &lt;em&gt;money&lt;/em&gt;, ... are closer to each other than those mentioning &lt;em&gt;ergodic&lt;/em&gt;, &lt;em&gt;Poincare&lt;/em&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to stick to titles, one idea is to think of word pairs as the words you use in the vector representation.  So for the title &lt;em&gt;The eagle has landed&lt;/em&gt;, you would think of &lt;em&gt;the eagle&lt;/em&gt;, &lt;em&gt;eagle has&lt;/em&gt;, &lt;em&gt;has landed&lt;/em&gt;. as the &amp;ldquo;words.&amp;rdquo;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To discover when a cluster has become much bigger or different from the others you will need to develop a decision procedure.&lt;/p&gt;&#xA;" OwnerUserId="260" LastActivityDate="2010-07-22T02:37:18.820" />
  <row Id="474" PostTypeId="2" ParentId="375" CreationDate="2010-07-22T03:21:18.913" Score="4" ViewCount="" Body="&lt;p&gt;For testing the numbers produced by random number generators the &lt;a href=&quot;http://en.wikipedia.org/wiki/Diehard_tests&quot; rel=&quot;nofollow&quot;&gt;Diehard tests&lt;/a&gt; are a practical approach. But those tests seem kind of arbitrary and one is may be left wondering if more should be included or if there is any way to really check the randomness.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The best candidate for a definition of a random sequence seems to be the &lt;a href=&quot;http://en.wikipedia.org/wiki/Algorithmically_random_sequence&quot; rel=&quot;nofollow&quot;&gt;Martin-Löf randomness&lt;/a&gt;.   The main idea for this kind of randomness, is beautifully developed in &lt;a href=&quot;http://en.wikipedia.org/wiki/The_Art_of_Computer_Programming&quot; rel=&quot;nofollow&quot;&gt;Knuth, section 3.5&lt;/a&gt;, is to test for uniformity for all types of sub-sequences of the sequence of random numbers.  Getting that &lt;em&gt;all type of subsequences&lt;/em&gt; definition right turned out to be be really hard even when one uses notions of computability.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Diehard tests are just some of the possible subsequences one may consider and their failure would exclude Martin-Löf randomness.&lt;/p&gt;&#xA;" OwnerUserId="260" LastActivityDate="2010-07-22T03:21:18.913" />
  <row Id="475" PostTypeId="2" ParentId="224" CreationDate="2010-07-22T03:55:48.147" Score="3" ViewCount="" Body="&lt;p&gt;There is also &lt;a href=&quot;http://gephi.org/&quot; rel=&quot;nofollow&quot;&gt;Gephi&lt;/a&gt; for plotting social networks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(p.s: Here is how to &lt;a href=&quot;http://www.r-bloggers.com/data-preparation-for-social-network-analysis-using-r-and-gephi/&quot; rel=&quot;nofollow&quot;&gt;connect it with R&lt;/a&gt;)&lt;/p&gt;&#xA;" OwnerUserId="253" LastActivityDate="2010-07-22T03:55:48.147" CommunityOwnedDate="2010-10-12T08:48:39.990" />
  <row Id="476" PostTypeId="2" ParentId="359" CreationDate="2010-07-22T04:34:32.110" Score="4" ViewCount="" Body="&lt;p&gt;I won't give a definitive answer in terms of ranking the three.  Build 95% CIs around your parameters based on each, and if they're radically different, then your first step should be to dig deeper.  Transform your data (though the LR will be invariant), regularize your likelihood, etc.  In a pinch though, I would probably opt for the LR test and associated CI.  A rough argument follows.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The LR is invariant under the choice of parametrization (e.g. T versus logit(T)).  The Wald statistic assumes normality of (T - T0)/SE(T).  If this fails, your CI is bad.  The nice thing about the LR is that you don't need to find a transform f(T) to satisfy normality.  The 95% CI based on T will be the same.  Also, if your likelihood isn't quadratic, the Wald 95% CI, which is symmetric, can be kooky since it may prefer values with lower likelihood to those with higher likelihood.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another way to think about the LR is that it's using more information, loosely speaking, from the likelihood function.  The Wald is based on the MLE and the curvature of the likelihood at null.  The Score is based on the slope at null and curvature at null.  The LR evaluates the likelihood under the null, and the likelihood under the union of the null and alternative, and combines the two.  If you're forced to pick one, this may be intuitively satisfying for picking the LR.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Keep in mind that there are other reasons, such as convenience or computational, to opt for the Wald or Score.  The Wald is the simplest and, given a multivariate parameter, if you're testing for setting many individual ones to 0, there are convenient ways to approximate the likelihood.  Or if you want to add a variable at a time from some set, you may not want to maximize the likelihood for each new model, and the implementation of Score tests offers some convenience here.  The Wald and Score become attractive as your models and likelihood become unattractive.  (But I don't think this is what you were questioning, since you have all three available ...)&lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-07-22T04:34:32.110" />
  <row Id="477" PostTypeId="2" ParentId="223" CreationDate="2010-07-22T04:58:50.427" Score="1" ViewCount="" Body="&lt;p&gt;I assume your friend prefers something that's biostatistics oriented.  Glantz's &lt;a href=&quot;http://www.powells.com/biblio/62-9780071435093-1&quot; rel=&quot;nofollow&quot;&gt;Primer of Biostatistics&lt;/a&gt; is a small book, an easy and quick read, and tends to get rave reviews from a similar audience.  If an online reference works, I like Gerard Dallal's &lt;a href=&quot;http://www.tufts.edu/~gdallal/LHSP.HTM&quot; rel=&quot;nofollow&quot;&gt;Handbook of Statitical Practice&lt;/a&gt;, which may do the trick if he's just refreshing previous knowledge.&lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-07-22T04:58:50.427" />
  <row Id="478" PostTypeId="2" ParentId="421" CreationDate="2010-07-22T05:13:17.627" Score="5" ViewCount="" Body="&lt;p&gt;I'm going to assume some basic statistics knowledge and recommend:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://www.powells.com/biblio/62-9780071435093-1&quot; rel=&quot;nofollow&quot;&gt;The Statistical Sleuth&lt;/a&gt; (Ramsey, Schafer) which contain a good deal of mini case studies as they cover the basic statistical tools for data analysis.  &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://www.tufts.edu/~gdallal/LHSP.HTM&quot; rel=&quot;nofollow&quot;&gt;A First Course in Multivariate Statistics&lt;/a&gt; (Flury) which covers the essential statistics required for data mining and the like.  &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="251" LastEditorUserId="251" LastEditorDisplayName="" LastEditDate="2010-07-22T16:13:22.167" LastActivityDate="2010-07-22T16:13:22.167" CommunityOwnedDate="2010-07-22T05:13:17.627" CommentCount="1" />
  <row Id="479" PostTypeId="2" ParentId="459" CreationDate="2010-07-22T07:28:42.960" Score="4" ViewCount="" Body="&lt;p&gt;If it does not break your styleguide I would rather color the background of the plots red/(yellow/)green than just plotting a line. In my imagination this should make it pretty clear to a user that values are fine on green and to be checked on red. Just my 5&amp;#xa2;.&lt;/p&gt;&#xA;" OwnerUserId="128" LastActivityDate="2010-07-22T07:28:42.960" CommentCount="4" />
  <row Id="480" PostTypeId="1" AcceptedAnswerId="635" CreationDate="2010-07-22T08:58:36.800" Score="3" ViewCount="354" Body="&lt;p&gt;I am not an expert of random forest but I clearly understand that the key issue with random forest is the (random) tree generation. Can you explain me how the trees are generated? (i.e. What is the used distribution for tree generation?)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance ! &lt;/p&gt;&#xA;" OwnerUserId="223" LastEditorUserId="217" LastEditorDisplayName="" LastEditDate="2010-07-26T16:57:59.560" LastActivityDate="2010-07-26T17:50:21.780" Title="How does random forest generate the random forest" Tags="&lt;machine-learning&gt;&lt;r&gt;&lt;algorithms&gt;&lt;decision-trees&gt;&lt;random-forest&gt;" AnswerCount="2" />
  <row Id="481" PostTypeId="1" AcceptedAnswerId="482" CreationDate="2010-07-22T09:17:27.490" Score="4" ViewCount="202" Body="&lt;p&gt;Another question about time series from me.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a dataset which gives daily records of violent incidents in a psychiatric hospital over three years. With the help from my previous question I have been fiddling with it and am a bit happier about it now.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The thing I have now is that the daily series is very noisy. It fluctuates wildly, up and down, from 0 at times up to 20. Using loess plots and the forecast package (which I can highly recommend for novices like me) I just get a totally flat line, with massive confidence intervals from the forecast.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, aggregating weekly or monthly the data make a lot more sense. They sweep down from the start of the series, and then increase again in the middle. Loess plotting and the forecast package both produce something that looks a lot more meaningful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It does feel a bit like cheating though. Am I just preferring the aggregated versions because they look nice with no real validity to it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or would it be better to compute a moving average and use that as the basis? I'm afraid I don't understand the theory behind all this well enough to be confident about what is acceptable&lt;/p&gt;&#xA;" OwnerUserId="199" LastActivityDate="2010-07-23T08:15:23.430" Title="Is it valid to aggregate a time series to make it look more meaningful?" Tags="&lt;time-series&gt;&lt;forecasting&gt;" AnswerCount="3" />
  <row Id="482" PostTypeId="2" ParentId="481" CreationDate="2010-07-22T09:24:52.503" Score="4" ViewCount="" Body="&lt;p&gt;This totally depends on your time series and what effect you want to discover/proof etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An important thing here is, what kind of periods do you have in your data. Make a spectrum of you data and see what frequencies are common in you data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyway, you are not lying when you decide to display aggregated values. When you are looking to effects that are occurring over weeks (like, more violence in summer when it's hot weather) it is the right thing to do.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maybe you can also take a look at the Hilbert Huang Transform. This will give you Intrinsic Mode Functions that are very handy for visual analyses.&lt;/p&gt;&#xA;" OwnerUserId="190" LastActivityDate="2010-07-22T09:24:52.503" />
  <row Id="483" PostTypeId="2" ParentId="480" CreationDate="2010-07-22T09:53:52.400" Score="4" ViewCount="" Body="&lt;p&gt;The main idea is the bagging procedure, not making trees random. In detail, each tree is build on a sample of objects drawn with replacement from the original set; thus each tree has some objects that it haven't seen, what makes the whole ensemble more heterogeneous and thus better in generalizing.&lt;br&gt;&#xA;Furthermore, trees are being weekend in such a way that on the each split only M (or &lt;code&gt;mtry&lt;/code&gt;) randomly selected attributes are considered; M is usually a square root of the number of attributes in the set. This makes the trees less overfitted, since they are not pruned.&lt;br&gt;&#xA;You can find more details &lt;a href=&quot;http://www.stat.berkeley.edu/~breiman/RandomForests/cc_home.htm&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, there is a variant of RF called Extreme Random Forest, in which trees are made in more less random way (there is no optimization of splits) -- consult, I think, &lt;a href=&quot;http://www.springerlink.com/index/10.1007/s10994-006-6226-1&quot; rel=&quot;nofollow&quot;&gt;http://www.springerlink.com/index/10.1007/s10994-006-6226-1&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="88" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-07-22T10:04:38.557" LastActivityDate="2010-07-22T10:04:38.557" CommentCount="2" />
  <row Id="485" PostTypeId="1" AcceptedAnswerId="495" CreationDate="2010-07-22T10:08:10.067" Score="20" ViewCount="945" Body="&lt;p&gt;A question previously sought recommendations for &lt;a href=&quot;http://stats.stackexchange.com/questions/414/intro-to-statistics-for-mathematicians&quot; rel=&quot;nofollow&quot;&gt;textbooks on mathematical statistics&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone know of any good online &lt;strong&gt;video lectures&lt;/strong&gt; on &lt;strong&gt;mathematical statistics&lt;/strong&gt;?&#xA;The closest that I've found are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.youtube.com/watch?v=UzxYlbK2c7E&amp;amp;feature=PlayList&amp;amp;p=A89DCFA6ADACE599&amp;amp;index=0&quot; rel=&quot;nofollow&quot;&gt;Machine Learning&lt;/a&gt; &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://economistsview.typepad.com/economics421/&quot; rel=&quot;nofollow&quot;&gt;Econometrics&lt;/a&gt; &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE:&lt;/strong&gt; A number of the suggestions mentioned below are good statistics-101 type videos.&#xA;However, I'm specifically wondering whether there are any videos that provide a rigorous mathematical presentation of statistics.&#xA;i.e., videos that might accompany a course that use a textbook mentioned in this &lt;a href=&quot;http://mathoverflow.net/questions/31655/statistics-for-mathematicians&quot; rel=&quot;nofollow&quot;&gt;discussion on mathoverflow&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-07-22T14:06:03.173" LastActivityDate="2010-09-16T16:27:02.927" CommunityOwnedDate="2010-07-22T14:10:29.167" Title="Mathematical Statistics Videos" Tags="&lt;mathematics&gt;&lt;video&gt;&lt;mathematical-statistics&gt;" AnswerCount="12" CommentCount="3" FavoriteCount="17" />
  <row Id="486" PostTypeId="1" AcceptedAnswerId="720" CreationDate="2010-07-22T10:11:15.210" Score="7" ViewCount="2312" Body="&lt;p&gt;I have calculated AIC and AICc to compare two general linear mixed models; The AICs are positive with model 1 having a lower AIC than model 2.  However, the values for AICc are both negative (model 1 is still &amp;lt; model 2).  Is it valid to use and compare negative AICc values? &lt;/p&gt;&#xA;" OwnerUserId="266" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-01-30T18:09:40.193" LastActivityDate="2011-01-30T18:09:40.193" Title="Negative values for AICc (corrected Akaike Information Criterion)" Tags="&lt;mixed-model&gt;&lt;model-selection&gt;&lt;aic&gt;" AnswerCount="4" CommentCount="3" />
  <row Id="487" PostTypeId="2" ParentId="363" CreationDate="2010-07-22T10:15:28.313" Score="6" ViewCount="" Body="&lt;p&gt;Not a book, but I recently discovered an article by Jacob Cohen in American Psychologist entitled &quot;Things I have learned (so far).&quot;  It's available as a pdf &lt;a href=&quot;http://www.uvm.edu/~bbeckage/Teaching/DataAnalysis/AssignedPapers/Cohen_1990.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.  &lt;/p&gt;&#xA;" OwnerUserId="266" LastActivityDate="2010-07-22T10:15:28.313" CommunityOwnedDate="2010-07-22T10:15:28.313" CommentCount="1" />
  <row Id="488" PostTypeId="2" ParentId="421" CreationDate="2010-07-22T10:16:18.263" Score="4" ViewCount="" Body="&lt;p&gt;A lot of Social Science / Psychology students with minimal mathematical background like Andy Field's book: &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0761944524&quot; rel=&quot;nofollow&quot;&gt;Discovering Statistics Using SPSS&lt;/a&gt;. He also has a website that shares a &lt;a href=&quot;http://www.statisticshell.com/woodofsuicides.html&quot; rel=&quot;nofollow&quot;&gt;lot of material&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="183" LastEditorUserId="509" LastEditorDisplayName="" LastEditDate="2010-08-11T08:42:35.537" LastActivityDate="2010-08-11T08:42:35.537" CommunityOwnedDate="2010-07-22T10:16:18.263" />
  <row Id="489" PostTypeId="2" ParentId="396" CreationDate="2010-07-22T11:05:55.197" Score="2" ViewCount="" Body="&lt;p&gt;It also depends on where you wan't to publish your plots. You'll save yourself a lot of trouble by consulting the guide for authors before making any plots for a journal. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also save the plots in a format that is easy to modify or save the code you have used to create them. Chances are that you need to make corrections. &lt;/p&gt;&#xA;" OwnerUserId="214" LastActivityDate="2010-07-22T11:05:55.197" CommunityOwnedDate="2011-06-01T03:42:00.250" />
  <row Id="490" PostTypeId="1" AcceptedAnswerId="606" CreationDate="2010-07-22T11:10:29.417" Score="7" ViewCount="393" Body="&lt;p&gt;What are the &lt;strong&gt;variable/feature  selection that you prefer&lt;/strong&gt; for binary classification when there are many more variables/feature than observations in the learning set? The aim here is to discuss what is the feature selection procedure that reduces the best the classification error. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can &lt;strong&gt;fix notations&lt;/strong&gt; for homogeneity: for i=0,1,  let x1^i,...,xni^i be the learning set of observations from group i.   So n0+n1=n is the size of the learning set. We set p the number of features (i.e. the dimension of the feature space). If x is a vector of R^p, x[i] is the ith coordinate. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please give full references if you cannot give the details. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT (updated continuously):  Procedures proposed in the answers below &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Greedy forward selection&lt;/strong&gt; &lt;a href=&quot;http://stats.stackexchange.com/questions/490/variable-selection-procedure-for-binary-classification/497#497&quot;&gt;http://stats.stackexchange.com/questions/490/variable-selection-procedure-for-binary-classification/497#497&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Backward elimination&lt;/strong&gt; &lt;a href=&quot;http://stats.stackexchange.com/questions/490/variable-selection-procedure-for-binary-classification/501#501&quot;&gt;http://stats.stackexchange.com/questions/490/variable-selection-procedure-for-binary-classification/501#501&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Metropolis scanning / MCMC&lt;/strong&gt; &lt;a href=&quot;http://stats.stackexchange.com/questions/490/variable-selection-procedure-for-binary-classification/505#505&quot;&gt;http://stats.stackexchange.com/questions/490/variable-selection-procedure-for-binary-classification/505#505&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;penalized logistic regression&lt;/strong&gt; &lt;a href=&quot;http://stats.stackexchange.com/questions/490/variable-selection-procedure-for-binary-classification/606#606&quot;&gt;http://stats.stackexchange.com/questions/490/variable-selection-procedure-for-binary-classification/606#606&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;As this is community wiki there can be more discussion and update&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have one remark: in a certain sence, you all give a procedure that permit ordering of variables but not variable selection (you are quite evasive on how to select the number of features, I guess you all use cross validation?) Can you improve the answers in this direction? (as this is community wiki you don't need to be the answer writter to add an information about how to select the number of variables? I have openned a question in this direction here &lt;a href=&quot;http://stats.stackexchange.com/questions/880/cross-validation-to-select-the-number-of-used-variables-in-very-high-dimensional&quot;&gt;http://stats.stackexchange.com/questions/880/cross-validation-to-select-the-number-of-used-variables-in-very-high-dimensional&lt;/a&gt;)&lt;/p&gt;&#xA;" OwnerUserId="223" LastEditorUserId="223" LastEditorDisplayName="" LastEditDate="2010-12-17T08:00:29.153" LastActivityDate="2010-12-17T11:13:27.427" CommunityOwnedDate="2010-07-22T11:58:13.327" Title="Variable selection procedure for binary classification" Tags="&lt;machine-learning&gt;&lt;classification&gt;&lt;multiple-comparisons&gt;&lt;multivariate-analysis&gt;&lt;feature-selection&gt;" AnswerCount="6" CommentCount="2" FavoriteCount="3" />
  <row Id="491" PostTypeId="2" ParentId="421" CreationDate="2010-07-22T11:19:10.587" Score="1" ViewCount="" Body="&lt;p&gt;I recently found &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0131467573&quot; rel=&quot;nofollow&quot;&gt;Even You Can Learn Statistics&lt;/a&gt; to be pretty useful.&lt;/p&gt;&#xA;" OwnerUserId="268" LastActivityDate="2010-07-22T11:19:10.587" CommunityOwnedDate="2010-07-22T11:19:10.587" />
  <row Id="492" PostTypeId="1" CreationDate="2010-07-22T11:31:16.777" Score="4" ViewCount="234" Body="&lt;p&gt;I am proposing to try and find a trend in some very noisy long term data. The data is basically weekly measurements of something which moved about 5mm over a period of about 8 months. The data is to 1mm accuracey and is very noisy regularly changing +/-1 or 2mm in a week. We only have the data to the nearest mm. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We plan to use some basic signal processing with a fast fourier transform to separate out the noise from the raw data. The basic assumption is if we mirror our data set and add it to the end of our existing data set we can create a full wavelength of the data and therefore our data will show up in a fast fourier transform and we can hopefully then separate it out. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given that this sounds a little dubious to me, is this a method worth purusing or is the method of mirroring and appending our data set somehow fundamentally flawed? We are looking at other approaches such as using a low pass filter as well. &lt;/p&gt;&#xA;" OwnerUserId="210" LastEditorUserId="223" LastEditorDisplayName="" LastEditDate="2011-02-08T16:37:42.407" LastActivityDate="2011-02-08T16:37:42.407" Title="Dubious use of signal processing principles to identify a trend" Tags="&lt;data-mining&gt;&lt;signal-processing&gt;&lt;trend&gt;" AnswerCount="4" CommentCount="1" />
  <row Id="493" PostTypeId="2" ParentId="170" CreationDate="2010-07-22T11:33:47.037" Score="0" ViewCount="" Body="&lt;p&gt;Statsoft's &lt;a href=&quot;http://www.statsoft.com/textbook/&quot; rel=&quot;nofollow&quot;&gt;Electronic Statistics Handbook&lt;/a&gt; ('The only Internet Resource about Statistics Recommended by Encyclopedia Britannica') is worth checking out.&lt;/p&gt;&#xA;" OwnerUserId="268" LastActivityDate="2010-07-22T11:33:47.037" CommunityOwnedDate="2010-07-22T11:33:47.037" />
  <row Id="494" PostTypeId="2" ParentId="486" CreationDate="2010-07-22T11:42:42.967" Score="3" ViewCount="" Body="&lt;p&gt;Generally, it is assumed that AIC (and so AICc) is defined up to adding a constant, so the fact if it is negative or positive is not meaningful at all. So the answer is yes, it is valid.&lt;/p&gt;&#xA;" OwnerUserId="88" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-07-22T20:16:08.967" LastActivityDate="2010-07-22T20:16:08.967" CommentCount="2" />
  <row Id="495" PostTypeId="2" ParentId="485" CreationDate="2010-07-22T11:43:42.560" Score="4" ViewCount="" Body="&lt;p&gt;There is one called &lt;a href=&quot;http://www.academicearth.org/courses/math-and-proability-for-life-sciences&quot; rel=&quot;nofollow&quot;&gt;Math and probability for life sciences&lt;/a&gt;, but I haven't followed it so I can't tell you if its good or not.&lt;/p&gt;&#xA;" OwnerUserId="214" LastActivityDate="2010-07-22T11:43:42.560" CommentCount="2" />
  <row Id="496" PostTypeId="2" ParentId="492" CreationDate="2010-07-22T11:44:13.230" Score="3" ViewCount="" Body="&lt;p&gt;I think you can get some distortion on the pasting point as not all the underlying waves will connect very well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would suggest using a Hilbert Huang transform for this. Just do the split into intrinsic mode functions and see what is left over as residue when calculating them.&lt;/p&gt;&#xA;" OwnerUserId="190" LastActivityDate="2010-07-22T11:44:13.230" />
  <row Id="497" PostTypeId="2" ParentId="490" CreationDate="2010-07-22T11:53:51.010" Score="2" ViewCount="" Body="&lt;p&gt;Greedy forward selection.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The steps for this method are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Make sure you have a train and validation set&lt;/li&gt;&#xA;&lt;li&gt;Repeat the following&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Train a classifier with each single feature separately that is not selected yet and with all the previously selected features&lt;/li&gt;&#xA;&lt;li&gt;If the result improves, add the best performing feature, else stop procedure&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="190" LastActivityDate="2010-07-22T11:53:51.010" CommunityOwnedDate="2010-07-28T09:35:43.920" CommentCount="4" />
  <row Id="498" PostTypeId="1" CreationDate="2010-07-22T11:58:21.923" Score="1" ViewCount="262" Body="&lt;p&gt;Sometimes, I just want to do a copy &amp;amp; paste from the output window in SAS. I can highlight text with a mouse-drag, but only SOMETIMES does that get copied to the clipboard. It doesn't matter if I use &quot;CTRL-C&quot; or right click -&gt; copy, or edit -&gt; copy&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any other SAS users experience this, and do you know a workaround/option/technique that can fix it? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sometimes, I can fix it by clicking in another window, and coming back to the output window, but sometimes I just have to save the output window as a .lst and get the text from another editor. &lt;/p&gt;&#xA;" OwnerUserId="62" LastActivityDate="2011-05-04T02:39:22.600" Title="In PC SAS, how do you copy &amp; paste from the output window?" Tags="&lt;sas&gt;&lt;pc-sas&gt;" AnswerCount="4" CommentCount="11" />
  <row Id="499" PostTypeId="1" CreationDate="2010-07-22T12:06:28.817" Score="12" ViewCount="230" Body="&lt;p&gt;I've heard that when many regression model specifications (say, in OLS) are considered as possibilities for a dataset, this causes multiple comparison problems and the p-values and confidence intervals are no longer reliable. One extreme example of this is stepwise regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When can I use the data itself to help specify the model, and when is this not a valid approach? Do you always need to have a subject-matter-based theory to form  the model?&lt;/p&gt;&#xA;" OwnerUserId="267" LastEditorUserId="267" LastEditorDisplayName="" LastEditDate="2010-07-22T13:26:10.600" LastActivityDate="2010-10-19T01:50:06.413" Title="When can you use data-based criteria to specify a regression model?" Tags="&lt;regression&gt;&lt;frequentist&gt;&lt;multiple-comparisons&gt;" AnswerCount="5" FavoriteCount="1" />
  <row Id="500" PostTypeId="2" ParentId="499" CreationDate="2010-07-22T12:16:29.217" Score="2" ViewCount="" Body="&lt;p&gt;If I understand your question right, than the answer to your problem is to correct the p-values accordingly to the number of hypothesis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example Holm-Bonferoni corrections, where you sort the hypothesis (= your different models) by their p-value and reject those with a p samller than (desired p-value / index).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More about the topic can be found on &lt;a href=&quot;http://en.wikipedia.org/wiki/Multiple_comparisons&quot; rel=&quot;nofollow&quot;&gt;Wikipedia&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="190" LastActivityDate="2010-07-22T12:16:29.217" CommentCount="1" />
  <row Id="501" PostTypeId="2" ParentId="490" CreationDate="2010-07-22T12:17:20.900" Score="2" ViewCount="" Body="&lt;p&gt;Backward elimination.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Start with the full set, then iteratively train the classifier on the remaining features and remove the feature with the smallest importance, stop when the classifier error rapidly increases/becomes unacceptable high.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Importance can be even obtained by removing iteratively each feature and check the error increase or adapted from the classifier if it produces it (like in case of Random Forest). &lt;/p&gt;&#xA;" OwnerUserId="88" LastEditorUserId="223" LastEditorDisplayName="" LastEditDate="2010-07-22T13:05:25.687" LastActivityDate="2010-07-22T13:05:25.687" CommunityOwnedDate="2010-07-22T12:17:20.900" CommentCount="6" />
  <row Id="502" PostTypeId="2" ParentId="485" CreationDate="2010-07-22T12:25:42.333" Score="3" ViewCount="" Body="&lt;p&gt;I do not know at what level you want the videos to be but I have heard good things about Khan's Academy: &lt;a href=&quot;http://www.khanacademy.org/#Statistics&quot; rel=&quot;nofollow&quot;&gt;http://www.khanacademy.org/#Statistics&lt;/a&gt;&lt;/p&gt;&#xA;" LastActivityDate="2010-07-22T12:25:42.333" />
  <row Id="503" PostTypeId="2" ParentId="498" CreationDate="2010-07-22T12:25:48.810" Score="2" ViewCount="" Body="&lt;p&gt;I have been using SAS a long time and have never had an issue with highlighting results from the output window.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However since you are having an issue... there are alarge number of solutions!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Perhaps the most i like... and probably a good habit to get into is to output your results into datasets... or into excel spread sheets directly (using the ODS) you can also output directly into pdf, rtf with 2 lines of the simplest code you can imagine!&#xA;if your code produces alot of output and you only have one table you want to copy you can specify the name of the table and it alone will be output using the ODS output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;usually you just need to wrap your Procedure (like Proc Means for example)&#xA;with &lt;/p&gt;&#xA;&#xA;&lt;p&gt;ods PDF;&#xA;Proc Means Data = blah N NMISS MEAN STD;&#xA;class upto you;&#xA;var you name it;&#xA;run;&#xA;ods PDF close;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;of course there are many ways to get fancy with the way the output looks but that is a matter of trial and error and finding what you like or meets your needs.&lt;/p&gt;&#xA;" OwnerUserId="256" LastActivityDate="2010-07-22T12:25:48.810" CommentCount="1" />
  <row Id="504" PostTypeId="2" ParentId="485" CreationDate="2010-07-22T12:33:49.683" Score="3" ViewCount="" Body="&lt;p&gt;Many of the &lt;strong&gt;Berkeley&lt;/strong&gt; introductory statistics courses are available online (and on iTunes).  Here's an example: &lt;a href=&quot;http://webcast.berkeley.edu/course_details.php?seriesid=1906978493&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;Stats 2&lt;/strong&gt;&lt;/a&gt;.  You can &lt;a href=&quot;http://www.google.com/search?hl=en&amp;amp;rlz=1C1CHMP_en-USUS292US307&amp;amp;q=statistics+video+site%3awebcast.berkeley.edu+&amp;amp;aq=f&amp;amp;aqi=&amp;amp;aql=&amp;amp;oq=&amp;amp;gs_rfai=&quot; rel=&quot;nofollow&quot;&gt;find more here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-07-22T12:33:49.683" />
  <row Id="505" PostTypeId="2" ParentId="490" CreationDate="2010-07-22T12:42:13.893" Score="2" ViewCount="" Body="&lt;p&gt;Metropolis scanning / MCMC&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Select few features randomly for a&#xA;start, train classifier only on them&#xA;and obtain the error. &lt;/li&gt;&#xA;&lt;li&gt;Make some&#xA;random change to this working set --&#xA;either remove one feature, add&#xA;another at random or replace some&#xA;feature with one not being currently&#xA;used.&lt;/li&gt;&#xA;&lt;li&gt;Train new classifier and get&#xA;its error; store in &lt;code&gt;dE&lt;/code&gt; the difference&#xA;the error on the new set minus the error on the previous set. &lt;/li&gt;&#xA;&lt;li&gt;With probability &lt;code&gt;min(1;exp(-beta*dE))&lt;/code&gt; accept this change, otherwise reject it and try another random change.&lt;/li&gt;&#xA;&lt;li&gt;Repeat it for a long time and finally return the working set that has globally achieved the smallest error.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You may extend it with some wiser control of &lt;code&gt;beta&lt;/code&gt; parameter. Simpler way is to use simulated annealing when you increase &lt;code&gt;beta&lt;/code&gt; (lower the temperature in physical analogy) over the time to reduce fluctuations and drive the algorithm towards minimum. Harder is to use &lt;a href=&quot;http://en.wikipedia.org/wiki/Replica_exchange&quot; rel=&quot;nofollow&quot;&gt;replica exchange&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-07-22T12:42:13.893" CommunityOwnedDate="2010-07-22T12:42:13.893" />
  <row Id="506" PostTypeId="2" ParentId="287" CreationDate="2010-07-22T13:10:57.267" Score="5" ViewCount="" Body="&lt;p&gt;Both MOM and GMM are very general methods for estimating parameters of statistical models. GMM is - as the name suggests - a generalisation of MOM. It was developed by Lars Peter Hansen and first published in Econometrica [1]. As there are numerous textbooks on the subject (e.g. [2]) I presume you want a non-technical answer here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Traditional or Classical Method of Moments Estimator&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The MOM estimator is a consistent but inefficient estimator. Assume a vector of data y which were generated by a probability distribution indexed by a parameter vector theta with k elements. In the method of moments, theta is estimated by computing k sample moments of y, setting them equal to population moments derived from the assumed probability distribution, and solving for theta. For example, the population moment of mu is the expectation of y, whereas the sample moment of mu is the sample mean of y. You would repeat this for each of the k elements of theta. As sample moments are generally consistent estimators of population moments, theta-hat will be consistent for theta.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Generalised Method of Moments&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the example above, we had the same number of moment conditions as unknown parameters, so all we would have done is solved the k equations in k unknowns to obtain the parameter estimates. Hansen asked: What happens when we have more moment conditions than parameters as usually occurs in econometric models? How can we combine them optimally? That is the purpose of the GMM estimator. In GMM we estimate the parameter vector by minimising the sum of squares of the differences between the population moments and the sample moments, using the variance of the moments as a metric. This is the minimum variance estimator in the class of estimators that use these moment conditions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;[1] Hansen, L. P.  (1982): Large Sample Properties of Generalized Method of Moments Estimators, &lt;em&gt;Econometrica&lt;/em&gt;, 50, 1029-1054&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[2] Hall, A. R. (2005). &lt;em&gt;Generalized Method of Moments (Advanced Texts in Econometrics).&lt;/em&gt; Oxford University Press&lt;/p&gt;&#xA;" OwnerUserId="215" LastActivityDate="2010-07-22T13:10:57.267" />
  <row Id="507" PostTypeId="1" AcceptedAnswerId="560" CreationDate="2010-07-22T13:40:30.237" Score="8" ViewCount="394" Body="&lt;p&gt;What is your preferred method of checking for convergence when using Markov chain Monte Carlo for Bayesian inference, and why?&lt;/p&gt;&#xA;" OwnerUserId="215" LastActivityDate="2010-08-13T00:37:26.680" Title="What is the best method for checking convergence in MCMC?" Tags="&lt;bayesian&gt;&lt;mcmc&gt;" AnswerCount="3" FavoriteCount="3" />
  <row Id="508" PostTypeId="2" ParentId="396" CreationDate="2010-07-22T13:41:53.637" Score="1" ViewCount="" Body="&lt;p&gt;I would add that the choice of plot should reflect the type of statistical test used to analyse the data.  In other words, whatever characteristics of the data were used for analysis should be shown visually - so you would show means and standard errors if you used a t-test but boxplots if you used a Mann-Whitney test.&lt;/p&gt;&#xA;" OwnerUserId="266" LastActivityDate="2010-07-22T13:41:53.637" CommunityOwnedDate="2011-06-01T03:42:00.250" />
  <row Id="509" PostTypeId="2" ParentId="507" CreationDate="2010-07-22T13:43:18.233" Score="2" ViewCount="" Body="&lt;p&gt;I like to do trace plots primarily and sometimes I use the Gelman-Rubin convergence diagnostic.&lt;/p&gt;&#xA;" LastActivityDate="2010-07-22T13:43:18.233" />
  <row Id="510" PostTypeId="2" ParentId="423" CreationDate="2010-07-22T13:49:37.283" Score="12" ViewCount="" Body="&lt;p&gt;More about design and power than analysis, but I like this one&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://imgs.xkcd.com/comics/experimentation.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="266" LastActivityDate="2010-07-22T13:49:37.283" CommunityOwnedDate="2010-07-22T13:49:37.283" />
  <row Id="511" PostTypeId="2" ParentId="423" CreationDate="2010-07-22T14:32:20.363" Score="10" ViewCount="" Body="&lt;p&gt;From &lt;a href=&quot;http://xkcd.com/701/&quot; rel=&quot;nofollow&quot;&gt;xkcd&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://imgs.xkcd.com/comics/science_valentine.png&quot; alt=&quot;You don't use science to show that you are right, you use science to become right.&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If some people who really believe that everything should be scientifically tested would actually walk their talk than they this comic might even show an event that actually happens.&lt;/p&gt;&#xA;" OwnerUserId="237" LastActivityDate="2010-07-22T14:32:20.363" CommunityOwnedDate="2010-07-22T14:32:20.363" />
  <row Id="512" PostTypeId="2" ParentId="492" CreationDate="2010-07-22T14:52:54.510" Score="5" ViewCount="" Body="&lt;p&gt;If you want to filter the long term trend out using signal processing, why not just use a low-pass? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The simplest thing I can think of would be an exponential moving average.&lt;/p&gt;&#xA;" OwnerUserId="33" LastActivityDate="2010-07-22T14:52:54.510" CommentCount="2" />
  <row Id="513" PostTypeId="2" ParentId="276" CreationDate="2010-07-22T15:22:20.527" Score="2" ViewCount="" Body="&lt;p&gt;I like to use resampling: I repeat whatever method I used with a subsample of the data (say 80% or even 50% of the total).  By doing this with many different subsamples, I get a feel for how robust the estimates are.  For many estimation procedures this can be made into a real (meaning publishable) estimate of your errors.&lt;/p&gt;&#xA;" OwnerUserId="260" LastActivityDate="2010-07-22T15:22:20.527" />
  <row Id="514" PostTypeId="2" ParentId="26" CreationDate="2010-07-22T15:36:53.510" Score="1" ViewCount="" Body="&lt;p&gt;Here's how I would answer this question using a diagram.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say we weigh 30 cats and calculate the mean weight.  Then we produce a scatter plot, with weight on the y axis and cat identity on the x axis.  The mean weight can be drawn in as a horizontal line.  We can then draw in vertical lines which connect each data point to the mean line - these are the deviations of each data point from the mean, and we call them residuals.  Now, these residuals can be useful because they can tell us something about the spread of the data: if there are many big residuals, then cats vary a lot in mass.  Conversely, if the residuals are mainly small, then cats are fairly closely clustered around the average weight.  So if we could have some metric which tells us the &lt;em&gt;average&lt;/em&gt; length of a residual in this data set, this would be a handy way of denoting how much spread there is in the data.  The standard deviation is, effectively, the length of the average residual. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would follow on on from this by giving the calculation for  s.d., explaining why we square and then square root (I like Vaibhav's short and sweet explanation).  Then I would mention the problems of outliers, as Graham does in his last paragraph.&lt;/p&gt;&#xA;" OwnerUserId="266" LastActivityDate="2010-07-22T15:36:53.510" />
  <row Id="515" PostTypeId="2" ParentId="155" CreationDate="2010-07-22T15:49:54.270" Score="5" ViewCount="" Body="&lt;p&gt;1) A good demonstration of how &quot;random&quot; needs to be defined in order to work out probability of certain events:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the chance that a random line drawn across a circle will be longer than the radius?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The question totally depends how you draw your line. Possibilities which you can describe in a real-world way for a circle drawn on the ground might include:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Draw two random points inside the circle and draw a line through those. (see where two flies / stones fall...)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Choose a fixed point on the circumference, then a random one elsewhere in the circle and join those. (in effect this is laying a stick across the circle at a variable angle through a given point and a random one eg where stone falls)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Draw a diameter. Randomly choose a point along it and draw a perpendicular through that. (roll a stick along in a straight line so it rests across the circle)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is relatively easy to show someone who can do some geometry (but not necessarily stats) the answer to the question can vary quite widely (from about 2/3 to about 0.866 or so)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) A reverse-engineered coin-toss: toss it (say) ten times and write down the result. Work out the probabilty of this exact sequence (1/2^10). A tiny chance, but you just saw it happen with your own eyes!... Every sequence &lt;em&gt;might&lt;/em&gt; come up, including ten heads in a row, but it is hard for lay people to get their head round it. As an encore, try to convince them they have just as good a chance of winning the lottery with the numbers 1 through 6 as any other combination.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) Explaining why medical diagnosis may seem really flawed. A test for disease foo which is 99.9% accurate at identifying those who have it but .1% false-positively diagnoses those who don't really have it may seem to be wrong really so often when the prevalence of the disease is really low (eg 1 in 1000) but many patients are tested for it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is one that is best explained with real numbers - imagine 1 million people are tested, so 1000 have the disease, 999 are correctly identified, but 0.1% of 999,000 is 999 who are told they have it but don't. So half those who are told they have it actually do not, despite the high level of accuracy (99.9%) and low level of false posisitive (0.1%). A second (ideally different) test will then separate these groups out. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;[incidentally, I chose the numbers because they are easy to work with, of course they do not have to add up to 100% as the accuracy / fales positive rates are independent factors in the test]&lt;/p&gt;&#xA;" OwnerUserId="270" LastActivityDate="2010-07-22T15:49:54.270" CommentCount="1" />
  <row Id="516" PostTypeId="2" ParentId="486" CreationDate="2010-07-22T16:00:44.953" Score="2" ViewCount="" Body="&lt;p&gt;Yes it's valid to compare negative AICc values, in the same way as you would negative AIC values.  The correction factor in the AICc can become large with small sample size and relatively large number of parameters, and penalize heavier than the AIC.  So positive AIC values can correspond to negative AICc values.&lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-07-22T16:00:44.953" />
  <row Id="517" PostTypeId="1" AcceptedAnswerId="522" CreationDate="2010-07-22T16:21:06.090" Score="6" ViewCount="507" Body="&lt;p&gt;In the context of machine learning, what is the difference between unsupervised learning, supervised learning and semi-supervised learning?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And what are some of the main algorithmic approaches to look at?&lt;/p&gt;&#xA;" OwnerUserId="68" LastEditorUserId="68" LastEditorDisplayName="" LastEditDate="2010-07-22T16:44:09.063" LastActivityDate="2010-08-13T17:59:07.270" Title="Unsupervised, supervised and semi-supervised learning" Tags="&lt;machine-learning&gt;" AnswerCount="3" CommentCount="2" FavoriteCount="2" />
  <row Id="518" PostTypeId="2" ParentId="183" CreationDate="2010-07-22T16:24:10.190" Score="2" ViewCount="" Body="&lt;p&gt;Try &lt;a href=&quot;http://www.r-project.org/&quot; rel=&quot;nofollow&quot;&gt;R&lt;/a&gt;. &lt;a href=&quot;http://cran.at.r-project.org/web/views/Cluster.html&quot; rel=&quot;nofollow&quot;&gt;Here&lt;/a&gt; you have a list of clustering packages available.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-07-22T16:24:10.190" CommentCount="1" />
  <row Id="519" PostTypeId="2" ParentId="36" CreationDate="2010-07-22T16:32:57.967" Score="3" ViewCount="" Body="&lt;p&gt;I read (a long time ago) of an interesting example about a decline in birth rates (or fertility rates if you prefer that measure) especially in the US, starting in the early 1960's,  as nuclear weapons testing was at an all-time high (in 1961 the biggest nuclear bomb ever detonated was tested in the USSR). Rates continued to deline until towards the end of the twentieth century when most countried finally stopped doing this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can't find a reference which combines these figures now, but this Wikipedia article has figures on &lt;a href=&quot;http://en.wikipedia.org/wiki/Nuclear_weapons_testing&quot; rel=&quot;nofollow&quot;&gt;nuclear weapons test&lt;/a&gt; numbers by country.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, it might make better sense to look at the correlation of birth rate with the introduction and legalisation of the contraceptive pill 'coincidentally' starting in the early 1960s.  (In only some states first, then all states for married women only, then some for unmarried, then across the board), But even that could only be part of the cause; lots of other aspects of equality, economic changes and other factors play a significant part.&lt;/p&gt;&#xA;" OwnerUserId="270" LastActivityDate="2010-07-22T16:32:57.967" CommunityOwnedDate="2010-08-16T13:01:42.613" CommentCount="2" />
  <row Id="520" PostTypeId="2" ParentId="183" CreationDate="2010-07-22T16:33:43.240" Score="2" ViewCount="" Body="&lt;p&gt;Python will give you all the flexibility you need. With the NumPy and &lt;a href=&quot;http://docs.scipy.org/doc/scipy/reference/cluster.html&quot; rel=&quot;nofollow&quot;&gt;SciPy cluster module&lt;/a&gt; you have the tools you need, and the datatypes of NumPy give you a good insight in how much memory you will use.&lt;/p&gt;&#xA;" OwnerUserId="190" LastActivityDate="2010-07-22T16:33:43.240" CommentCount="5" />
  <row Id="521" PostTypeId="2" ParentId="517" CreationDate="2010-07-22T16:39:05.870" Score="3" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Unsupervised Learning&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unsupervised learning is when you have no labeled data available for training. Examples of this are often clustering methods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Supervised Learning&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this case your training data exists out of labeled data. The problem you solve here is often predicting the labels for data points without label.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Semi-Supervised Learning&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this case both labeled data and unlabeled data are used. This for example can be used in Deep belief networks, where some layers are learning the structure of the data (unsupervised) and one layer is used to make the classification (trained with supervised data)&lt;/p&gt;&#xA;" OwnerUserId="190" LastActivityDate="2010-07-22T16:39:05.870" />
  <row Id="522" PostTypeId="2" ParentId="517" CreationDate="2010-07-22T18:03:43.437" Score="4" ViewCount="" Body="&lt;p&gt;Generally, the problems of machine learning may be considered variations on function estimation for classification, prediction or modeling.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In &lt;strong&gt;supervised learning&lt;/strong&gt; one is furnished with input (x1, x2, . .,) and output (y1, y2, . .,) and are challenged with finding a function that approximates this behavior in a generalizable fashion.  The output could be a class label (in classification) or a real number (in regression)-- these are the &quot;supervision&quot; in supervised learning. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the case of &lt;strong&gt;unsupervised learning&lt;/strong&gt;, in the base case, you receives inputs x1, x2, . ., but neither target outputs, nor rewards from its environment are provided.  Based on the problem (classify, or predict) and your background knowledge of the space sampled, you may use various methods: density estimation (estimating some underlying PDF for prediction), k-means clustering (classifying unlabeled real valued data), k-modes clustering (classifying unlabeled categorical data), etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Semi-supervised learning&lt;/strong&gt; involves function estimation on labeled and unlabeled data.  This approach is motivated by the fact that labeled data is often costly to generate, whereas unlabeled data is generally not.  The challenge here mostly involves the technical question of how to treat data mixed in this fashion. See this &lt;a href=&quot;http://pages.cs.wisc.edu/~jerryzhu/pub/ssl_survey.pdf&quot; rel=&quot;nofollow&quot;&gt;Semi-Supervised Learning Literature Survey&lt;/a&gt; for more details on semi-supervised learning methods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In addition to these kinds of learning, there are others, such as &lt;strong&gt;reinforcement learning&lt;/strong&gt; whereby the learning method interacts with its environment by producing actions a1, a2, . . .. that produce rewards or punishments r1, r2, ...&lt;/p&gt;&#xA;" OwnerUserId="39" LastEditorUserId="39" LastEditorDisplayName="" LastEditDate="2010-07-22T18:22:28.563" LastActivityDate="2010-07-22T18:22:28.563" />
  <row Id="523" PostTypeId="2" ParentId="517" CreationDate="2010-07-22T18:16:08.797" Score="1" ViewCount="" Body="&lt;p&gt;I don't think that supervised/unsupervised is the best way to think about it. For basic data mining, it's better to think about what you are trying to do. There are four main tasks:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;prediction. if you are predicting a real number, it is called regression. if you are predicting a whole number or class, it is called classification.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;modeling. modeling is the same as prediction, but the model is comprehensible by humans. Neural networks and support vector machines work great, but do not produce comprehensible models [1]. decision trees and classic linear regression are examples of easy-to-understand models. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;similarity. if you are trying to find natural groups of attributes, it is called factor analysis. if you are trying to find natural groups of observations, it is called clustering.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;association. it's much like correlation, but for enormous binary datasets. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;[1] Apparently Goldman Sachs created tons of great neural networks for prediction, but then no one understood them, so they had to write other programs to try to explain the neural networks. &lt;/p&gt;&#xA;" OwnerUserId="74" LastEditorUserId="74" LastEditorDisplayName="" LastEditDate="2010-08-13T17:59:07.270" LastActivityDate="2010-08-13T17:59:07.270" />
  <row Id="524" PostTypeId="1" AcceptedAnswerId="525" CreationDate="2010-07-22T18:25:54.250" Score="7" ViewCount="128" Body="&lt;p&gt;Debugging MCMC programs is notoriously difficult. The difficulty arises because of several issues some of which are:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(a) Cyclic nature of the algorithm&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We iteratively draw parameters conditional on all other parameters. Thus, if a implementation is not working properly it is difficult to isolate the bug as the issue can be anywhere in the iterative sampler.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(b) The correct answer is not necessarily known. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We have no way to tell if we have achieved convergence. To some extent this can be mitigated by testing the code on simulated data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In light of the above issues, I was wondering if there is a standard technique that can be used to debug MCMC programs. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wanted to share the approach I use to debug my own programs. I, of course, do all of the things that PeterR mentioned. Apart from those, I perform the following tests using simulated data:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Start all parameters from true values and see if the sampler diverges too far from the true values.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I have flags for each parameter in my iterative sampler that determines whether I am drawing that parameter in the iterative sampler. For example, if a flag 'gen_param1' is set to true then I draw 'param1' from its full conditional in the iterative sampler. If this is set to false then 'param1' is set to its true value.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Once I finish writing up the sampler, I test the program using the following recipe:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Set the generate flag for one parameter to true and everything else to false and assess convergence with respect to true value.&lt;/li&gt;&#xA;&lt;li&gt;Set the generate flag for another parameter in conjunction with the first one and again assess convergence.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The above steps have been incredibly helpful to me.&lt;/p&gt;&#xA;" LastActivityDate="2010-07-25T20:08:35.907" Title="Is there a standard technique to debug MCMC programs?" Tags="&lt;mcmc&gt;&lt;code&gt;" AnswerCount="2" FavoriteCount="2" />
  <row Id="525" PostTypeId="2" ParentId="524" CreationDate="2010-07-22T18:33:47.900" Score="7" ViewCount="" Body="&lt;p&gt;Standard programming practice:&#xA;-when debugging run the simulation with fixed sources of randomness (i.e. same seed) so that any changes are due to code changes and not different random numbers&#xA;-try your code on a model (or several models) where the answer IS known&#xA;-adopt good programming habits so that you introduce fewer bugs &#xA;-think very hard &amp;amp; long about the answers you do get, whether they make sense, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wish you good luck, and plenty of coffee!&lt;/p&gt;&#xA;" OwnerUserId="247" LastActivityDate="2010-07-22T18:33:47.900" />
  <row Id="526" PostTypeId="1" AcceptedAnswerId="585" CreationDate="2010-07-22T19:17:41.870" Score="4" ViewCount="177" Body="&lt;p&gt;As you know, there are two popular types of cross-validation, K-fold and random subsampling (as described in &lt;a href=&quot;http://en.wikipedia.org/wiki/Cross-validation_%28statistics%29&quot; rel=&quot;nofollow&quot;&gt;Wikipedia&lt;/a&gt;). Nevertheless, I know that some researchers are making and publishing papers where something that is described as a K-fold CV is indeed a random subsampling one, so in practice you never know what is really in the article you're reading.&lt;br&gt;&#xA;Usually of course the difference is unnoticeable, and so goes my question -- can you think of an example when the result of one type is significantly different from another? &lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-07-24T22:54:13.430" Title="Does the cross validation implementation influence its results?" Tags="&lt;machine-learning&gt;&lt;cross-validation&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="527" PostTypeId="1" AcceptedAnswerId="578" CreationDate="2010-07-22T21:15:52.347" Score="3" ViewCount="258" Body="&lt;p&gt;I have two different analytical methods that can measure the concentration of a particular molecule in a matrix (for instance measure the amount of salt in water)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The two methods are different, and each has it's own error.  What ways exist to show the two methods are equivalent (or not).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm thinking that plotting the results from a number of samples measured by both methods on a scatter graph is a good first step, but are there any good statistical methods ?&lt;/p&gt;&#xA;" OwnerUserId="114" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-17T20:32:24.677" LastActivityDate="2010-09-19T12:20:31.163" Title="What ways are there to show two analytical methods are equivalent ?" Tags="&lt;hypothesis-testing&gt;" AnswerCount="4" CommentCount="2" FavoriteCount="1" />
  <row Id="528" PostTypeId="2" ParentId="36" CreationDate="2010-07-22T22:33:37.800" Score="1" ViewCount="" Body="&lt;p&gt;Teaching &quot;Correlation does not mean causation&quot; doesn't really help anyone because at the end of the day all deductive arguments are based in part on correlation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Human are very bad at learning not to do something. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The goal should rather be constructive: Always think about alternatives to your starting assumptions that might produce the same data.&lt;/p&gt;&#xA;" OwnerUserId="237" LastActivityDate="2010-07-22T22:33:37.800" CommunityOwnedDate="2010-08-16T13:01:42.613" />
  <row Id="529" PostTypeId="2" ParentId="527" CreationDate="2010-07-22T22:38:05.597" Score="0" ViewCount="" Body="&lt;p&gt;Your use of the phrase 'analytical methods' is a bit confusing to me. I am going to assume that by 'analytical methods' you mean some specific model/estimation strategy. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Broadly, speaking there are two types of metrics you could use to choose between estimators.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;In-sample Metrics&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Likelihood ratio / Wald test / Score test&lt;/li&gt;&#xA;&lt;li&gt;R&lt;sup&gt;2&lt;/sup&gt;&lt;/li&gt;&#xA;&lt;li&gt;In-sample Hit Rates (Percentage of correct predictions for sample data)&lt;/li&gt;&#xA;&lt;li&gt;(Lots of other metrics depending on model / estimation context)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Out-of-sample Metrics&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Out-of-sample Hit Rates (Percentage of correct predictions for out-of-sample data)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If the estimates are equivalent they would perform equally well on these metrics. You could also see if the estimates are not statistically different from one another (like the two-sample test of equality of means) but the methodology for this would depend on model and method specifics.&lt;/p&gt;&#xA;" LastActivityDate="2010-07-22T22:38:05.597" CommentCount="4" />
  <row Id="530" PostTypeId="2" ParentId="481" CreationDate="2010-07-22T23:52:24.403" Score="5" ViewCount="" Body="&lt;p&gt;It's very common in forecasting to aggregate data in order to increase the signal/noise ratio.  There are several papers on the effect of temporal aggregation on forecast accuracy in economics, for example. What you're probably seeing in the daily data is a weak signal that is being swamped by noise, whereas the weekly and monthly data are showing a stronger signal that is more visible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Whether you want to use temporal aggregation depends entirely on what your purpose is. If you need forecasts of daily incidents, then aggregation isn't going to be much use. If you are interested in exploring the effects of several covariates on the frequency of incidence, and all your data are available on a daily basis, then I would probably use the daily data as it will give a larger sample size and probably enable you to detect the effects more easily.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since you are using the forecast package, presumably you are interested in time series forecasting. So do you need daily forecasts, weekly forecasts or monthly forecasts? The answer will determine whether aggregation is appropriate for you.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-07-22T23:52:24.403" />
  <row Id="531" PostTypeId="2" ParentId="499" CreationDate="2010-07-23T00:04:48.980" Score="4" ViewCount="" Body="&lt;p&gt;I don't think it is possible to do Bonferoni or similar corrections to adjust for variable selection in regression because all the tests and steps involved in model selection are not independent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One approach is to formulate the model using one set of data, and do inference on a different set of data. This is done in forecasting all the time where we have a training set and a test set. It is not very common in other fields, probably because data are so precious that we want to use every single observation for model selection and for inference. However, as you note in your question, the downside is that the inference is actually misleading.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are many situations where a theory-based approach is impossible as there is no well-developed theory. In fact, I think this is much more common than the cases where theory suggests a model.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-07-23T00:04:48.980" />
  <row Id="532" PostTypeId="2" ParentId="213" CreationDate="2010-07-23T01:44:22.153" Score="8" ViewCount="" Body="&lt;p&gt;I think Robin Girard's answer would work pretty well for 3 and possibly 4 dimensions, but the curse of dimensionality would prevent it working beyond that. However, his suggestion led me to a related approach which is to apply the cross-validated kernel density estimate to the first three principal component scores. Then a very high-dimensional data set can still be handled ok.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In summary, for i=1 to n&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Compute a density estimate of the first three principal component scores obtained from the data set without Xi. &lt;/li&gt;&#xA;&lt;li&gt;Calculate the likelihood of Xi for the density estimated in step 1.&#xA;call it Li. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;end for&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sort the Li (for i=1,..,n) and the outliers are those with likelihood below some threshold. I'm not sure what would be a good threshold -- I'll leave that for whoever writes the paper on this! One possibility is to do a boxplot of the log(Li) values and see what outliers are detected at the negative end.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-07-23T01:44:22.153" CommentCount="3" />
  <row Id="533" PostTypeId="2" ParentId="155" CreationDate="2010-07-23T01:48:56.683" Score="2" ViewCount="" Body="&lt;p&gt;Along the lines of the mean as balance point, I like this view of the median as a balance point:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://statpics.blogspot.com/2009/08/pearl-balanced-median-necklace.html&quot; rel=&quot;nofollow&quot;&gt;A Pearl: a Balanced Median Necklace&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-07-23T01:48:56.683" />
  <row Id="534" PostTypeId="1" AcceptedAnswerId="538" CreationDate="2010-07-23T01:56:02.517" Score="23" ViewCount="2316" Body="&lt;p&gt;We all know the mantra &quot;correlation does not imply causation&quot; which is drummed into all first year statistics students. There are some nice examples &lt;a href=&quot;http://stats.stackexchange.com/questions/36/correlation-does-not-mean-causation&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; to illustrate the idea.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But sometimes correlation &lt;em&gt;does&lt;/em&gt; imply causation. The following example is taking from this &lt;a href=&quot;http://en.wikipedia.org/wiki/Correlation_does_not_imply_causation#Determining_causation&quot; rel=&quot;nofollow&quot;&gt;Wikipedia page&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;For example, one could run an experiment on identical twins who were known to consistently get the same grades on their tests. One twin is sent to study for six hours while the other is sent to the amusement park. If their test scores suddenly diverged by a large degree, this would be strong evidence that studying (or going to the amusement park) had a causal effect on test scores. In this case, correlation between studying and test scores would almost certainly imply causation.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Are there other situations where correlation implies causation?&lt;/p&gt;&#xA;" OwnerUserId="159" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-08-31T23:50:10.197" LastActivityDate="2010-10-02T17:02:16.690" Title="Under what conditions does correlation imply causation?" Tags="&lt;correlation&gt;&lt;causal-inference&gt;" AnswerCount="14" CommentCount="4" FavoriteCount="9" />
  <row Id="535" PostTypeId="2" ParentId="534" CreationDate="2010-07-23T02:00:12.007" Score="0" ViewCount="" Body="&lt;p&gt;Almost surely in a well designed experiment.  (Designed, of course, to elicit such a &lt;a href=&quot;http://18th.eserver.org/hume-enquiry.html#7&quot; rel=&quot;nofollow&quot;&gt;connexion&lt;/a&gt;.)&lt;/p&gt;&#xA;" OwnerUserId="251" LastEditorUserId="251" LastEditorDisplayName="" LastEditDate="2010-07-23T02:06:23.333" LastActivityDate="2010-07-23T02:06:23.333" />
  <row Id="537" PostTypeId="2" ParentId="534" CreationDate="2010-07-23T02:42:39.580" Score="7" ViewCount="" Body="&lt;p&gt;Your example is that of a &lt;a href=&quot;http://en.wikipedia.org/wiki/Controlled_experiment&quot; rel=&quot;nofollow&quot;&gt;controlled experiment&lt;/a&gt;. The only other context that I know of where a correlation can imply causation is that of a &lt;a href=&quot;http://en.wikipedia.org/wiki/Natural_experiment&quot; rel=&quot;nofollow&quot;&gt;natural experiment&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically, a natural experiment takes advantage of an assignment of some respondents to a treatment that happens naturally in the real world. Since assignment of respondents to treatment and control groups is not controlled by the experimenter the extent to which correlation would imply causation is perhaps weaker to some extent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See the wiki links for more information controlled / natural experiments.&lt;/p&gt;&#xA;" LastActivityDate="2010-07-23T02:42:39.580" />
  <row Id="538" PostTypeId="2" ParentId="534" CreationDate="2010-07-23T04:49:41.213" Score="14" ViewCount="" Body="&lt;p&gt;Correlation is not sufficient for causation.  One can get around the Wikipedia example by imagining that those twins always cheated in their tests by having a device that gives them the answers.  The twin that goes to the amusement park looses the device, hence the low grade.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A good way to get this stuff straight is to think of the structure of Bayesian network that may be generating the measured quantities, as done by  Pearl in his book &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/052189560X&quot; rel=&quot;nofollow&quot;&gt;&lt;em&gt;Causality&lt;/em&gt;&lt;/a&gt;.  His basic point is to look for hidden variables.  If there is a hidden variable that happens not to vary in the measured sample, then the correlation would not imply causation.  Expose all hidden variables and one has causation.&lt;/p&gt;&#xA;" OwnerUserId="260" LastEditorUserId="260" LastEditorDisplayName="" LastEditDate="2010-07-23T11:16:16.243" LastActivityDate="2010-07-23T11:16:16.243" CommentCount="5" />
  <row Id="539" PostTypeId="1" AcceptedAnswerId="4183" CreationDate="2010-07-23T06:17:10.517" Score="12" ViewCount="989" Body="&lt;p&gt;In answering &lt;a href=&quot;http://stats.stackexchange.com/questions/206/discrete-and-continuous&quot; rel=&quot;nofollow&quot;&gt;this question on discrete and continuous data&lt;/a&gt; I glibly asserted that it rarely makes sense to treat categorical data as continuous.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the face of it that seems self-evident, but intuition is often a poor guide for statistics, or at least mine is. So now I'm wondering: is it true? Or are there established analyses for which a transform from categorical data to some continuum is actually useful? Would it make a difference if the data were ordinal?&lt;/p&gt;&#xA;" OwnerUserId="174" LastActivityDate="2010-11-04T07:29:48.097" Title="Does it ever make sense to treat categorical data as continuous?" Tags="&lt;categorical-data&gt;&lt;continuous-data&gt;&lt;data-transformation&gt;" AnswerCount="4" CommentCount="2" FavoriteCount="4" />
  <row Id="540" PostTypeId="2" ParentId="534" CreationDate="2010-07-23T06:17:56.740" Score="5" ViewCount="" Body="&lt;p&gt;In my opinion the APA Statistical Task force summarised it quite well&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;''Inferring causality from nonrandomized&#xA;  designs is a risky enterprise.&#xA;  Researchers using nonrandomized&#xA;  designs have an extra obligation to&#xA;  explain the logic behind covariates&#xA;  included in their designs and to alert&#xA;  the reader to plausible rival&#xA;  hypotheses that might explain their&#xA;  results. Even in randomized&#xA;  experiments, attributing causal&#xA;  effects to any one aspect of the&#xA;  treatment condition requires support&#xA;  from additional experimentation.''&#xA;  - &lt;a href=&quot;http://www.loyola.edu/library/ref/articles/Wilkinson.pdf&quot; rel=&quot;nofollow&quot;&gt;APA Task Force&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-07-23T06:17:56.740" />
  <row Id="541" PostTypeId="2" ParentId="539" CreationDate="2010-07-23T06:22:04.313" Score="7" ViewCount="" Body="&lt;p&gt;If there are only two categories, then transforming them to (0,1) makes sense. In fact, this is commonly done where the resulting dummy variable is used in regression models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If there are more than two categories, then I think it only makes sense if the data are ordinal, and then only in very specific circumstances. For example, if I am doing regression and fit a nonparametric nonlinear function to the ordinal-cum-numeric variable, I think that is ok. But if I use linear regression, then I am making very strong assumptions about the relative difference between consecutive values of the ordinal variable, and I'm usually reluctant to do that.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-07-23T06:22:04.313" CommentCount="3" />
  <row Id="542" PostTypeId="2" ParentId="499" CreationDate="2010-07-23T06:49:38.953" Score="7" ViewCount="" Body="&lt;p&gt;In the social science context where I come from, the issue is whether you are interested in (a) prediction or (b) testing a focused research question.&#xA;If the purpose is prediction then data driven approaches are appropriate.&#xA;If the purpose is to examine a focused research question then it is important to consider which regression model specifically tests your question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if your task was to select a set of selection tests to predict job performance, the aim can in some sense be seen as one of maximising prediction of job performance.&#xA;Thus, data driven approaches would be useful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In contrast if you wanted to understand the relative role of personality variables and ability variables in influencing performance, then a specific model comparison approach might be more appropriate. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Typically when exploring focussed research questions the aim is to elucidate something about the underlying causal processes that are operating as opposed to developing a model with optimal prediction. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I'm in the process of developing models about process based on cross-sectional data I'd be wary about:&#xA;(a) including predictors that could theoretically be thought of as consequences of the outcome variable. E.g., a person's belief that they are a good performer is a good predictor of job performance, but it is likely that this is at least partially caused by the fact that they have observed their own performance.&#xA;(b) including a large number of predictors that are all reflective of the same underlying phenomena. E.g., including 20 items all measuring satisfaction with life in different ways.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, focused research questions rely a lot more on domain specific knowledge. &#xA;This probably goes some way to explaining why data driven approaches are less often used in the social sciences.&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-07-23T06:49:38.953" />
  <row Id="543" PostTypeId="2" ParentId="10" CreationDate="2010-07-23T07:13:03.533" Score="1" ViewCount="" Body="&lt;p&gt;In addition to what has already been said above about summated scales, I'd also mention that the issue can change when analysing data at the group-level. For example, if you were examining&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;life satisfaction of states or countries,&lt;/li&gt;&#xA;&lt;li&gt;job satisfaction of organisations or departments,&lt;/li&gt;&#xA;&lt;li&gt;student satisfaction in subjects.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In all these cases each aggregate measure (perhaps the mean) is based on many individual responses (e.g., n=50, 100, 1000, etc.). In these cases the original Likert item begins to take on properties that resemble an interval scale at the aggregate level.&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-07-23T07:13:03.533" />
  <row Id="544" PostTypeId="2" ParentId="539" CreationDate="2010-07-23T07:27:31.830" Score="5" ViewCount="" Body="&lt;p&gt;It is common practice to treat ordered categorical variables with many categories as continuous. Examples of this:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Number of items correct on a 100 item test&lt;/li&gt;&#xA;&lt;li&gt;A summated psychological scale (e.g., that is the mean of 10 items each on a five point scale)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;And by &quot;treating as continuous&quot; I mean including the variable in a model that assumes a continuous random variable (e.g., as a dependent variable in a linear regression). I suppose the issue is how many scale points are required for this to be a reasonable simplifying assumption.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A few other thoughts:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Polychoric correlations&lt;/strong&gt; attempt to model the relationship between two ordinal variables in terms of assumed latent continuous variables.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Optimal scaling&lt;/strong&gt; allows you to develop models where the scaling of a categorical variable is developed in a data driven way whilst respecting whatever scale constraints you impose (e.g., ordinality)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-07-23T07:27:31.830" CommentCount="2" />
  <row Id="545" PostTypeId="2" ParentId="481" CreationDate="2010-07-23T08:15:23.430" Score="1" ViewCount="" Body="&lt;p&gt;The problem (dilemma) you face appears to be the one of selecting an optimal (or otherwise good) sampling interval for revising your forecasts. To start with, see &lt;a href=&quot;http://books.google.com.au/books?id=XXFNW_QaJYgC&amp;amp;pg=PA42&quot; rel=&quot;nofollow&quot; title=&quot;chapter 3&quot;&gt;link text&lt;/a&gt; of Brown's famous book, which would also qualify as a good reference. It all boils down to &quot;balancing the risk of not noticing a change quickly against the inherent variability of the data and the cost of revising plans frequently&quot;. If you are not prepared to revise your forecast (and the decisions that motivated it) daily, you don't really need to use the (noisiest) daily data. An important point, often lost in the contemporary forecasting literature, is that forecasts are only necessary to assist with making a decision (unless one also knows how to derive fun from them).&lt;/p&gt;&#xA;" OwnerUserId="273" LastActivityDate="2010-07-23T08:15:23.430" />
  <row Id="546" PostTypeId="2" ParentId="423" CreationDate="2010-07-23T08:18:52.737" Score="10" ViewCount="" Body="&lt;p&gt;Another one from &lt;a href=&quot;http://xkcd.com/388/&quot; rel=&quot;nofollow&quot;&gt;xkcd&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://imgs.xkcd.com/comics/fuck_grapefruit.png&quot; alt=&quot;Coconuts are so far down to the left they couldn't be fit on the chart.  Ever spent half an hour trying to open a coconut with a rock?  Down with coconuts.&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="183" LastEditorUserId="509" LastEditorDisplayName="" LastEditDate="2010-08-11T09:20:21.947" LastActivityDate="2010-08-11T09:20:21.947" CommunityOwnedDate="2010-07-23T08:18:52.737" CommentCount="1" />
  <row Id="547" PostTypeId="2" ParentId="534" CreationDate="2010-07-23T08:33:04.260" Score="5" ViewCount="" Body="&lt;p&gt;There is also a problem with the opposite case, when lack of correlation is used as a proof for the lack of causation. This problem is nonlinearity; when looking at correlation people usually check Pearson, which is only a tip of an iceberg. &lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-07-23T08:33:04.260" />
  <row Id="548" PostTypeId="1" AcceptedAnswerId="559" CreationDate="2010-07-23T10:51:52.960" Score="3" ViewCount="161" Body="&lt;p&gt;In an answer to &lt;a href=&quot;http://stats.stackexchange.com/questions/539/does-it-ever-make-sense-to-treat-categorical-data-as-continuous&quot; rel=&quot;nofollow&quot;&gt;this question about treating categorical data as continuous&lt;/a&gt;, optimal scaling was mentioned. How does this method work and how is it applied?&lt;/p&gt;&#xA;" OwnerUserId="266" LastEditorUserId="190" LastEditorDisplayName="" LastEditDate="2010-07-23T11:06:32.657" LastActivityDate="2010-07-23T16:30:41.750" Title="How can I use optimal scaling to scale an ordinal categorical variable?" Tags="&lt;categorical-data&gt;&lt;data-transformation&gt;&lt;optimal-scaling&gt;" AnswerCount="1" />
  <row Id="549" PostTypeId="2" ParentId="539" CreationDate="2010-07-23T13:28:55.297" Score="2" ViewCount="" Body="&lt;p&gt;In an analysis of ranking by frequency, as with a Pareto chart and associated values (eg how many categories make up the top 80% of product faults)&lt;/p&gt;&#xA;" OwnerUserId="270" LastActivityDate="2010-07-23T13:28:55.297" />
  <row Id="550" PostTypeId="2" ParentId="155" CreationDate="2010-07-23T13:58:07.597" Score="6" ViewCount="" Body="&lt;p&gt;I have used the drunkard's walk before for random walk, and the drunk and her dog for cointegration; they're very helpful (partially because they're amusing).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of my favorite common examples is the &lt;a href=&quot;http://mathworld.wolfram.com/BirthdayProblem.html&quot; rel=&quot;nofollow&quot;&gt;Birthday Paradox&lt;/a&gt; (&lt;a href=&quot;http://en.wikipedia.org/wiki/Birthday_problem&quot; rel=&quot;nofollow&quot;&gt;wikipedia entry&lt;/a&gt;), which illustrates some important concepts of probability.  You can simulate this with a room full of people.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Incidentally, I strongly recommend Andrew Gelman's &lt;a href=&quot;http://www.stat.columbia.edu/~gelman/bag-of-tricks/&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;&quot;Teaching Statistics: A Bag of Tricks&quot;&lt;/strong&gt;&lt;/a&gt; for some examples of creative ways to teach statistical concepts (see the &lt;a href=&quot;http://www.stat.columbia.edu/~gelman/bag-of-tricks/contents.pdf&quot; rel=&quot;nofollow&quot;&gt;table of contents&lt;/a&gt;).  Also look at his paper about the course that he teaches on teaching statistics: &lt;a href=&quot;http://www.stat.columbia.edu/~gelman/research/published/teachcourse3.pdf&quot; rel=&quot;nofollow&quot;&gt;&quot;A Course on Teaching Statistics at the University Level&quot;&lt;/a&gt;.  And on &lt;a href=&quot;http://www.stat.columbia.edu/~gelman/research/published/teachingbayes.pdf&quot; rel=&quot;nofollow&quot;&gt;&quot;Teaching Bayes to Graduate Students in Political Science, Sociology,&#xA;Public Health, Education, Economics, ...&quot;&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For describing Bayesian methods, using an unfair coin and flipping it multiple times is a pretty common/effective approach.&lt;/p&gt;&#xA;" OwnerUserId="5" LastEditorUserId="5" LastEditorDisplayName="" LastEditDate="2010-07-23T16:25:03.033" LastActivityDate="2010-07-23T16:25:03.033" />
  <row Id="551" PostTypeId="2" ParentId="298" CreationDate="2010-07-23T14:43:14.180" Score="7" ViewCount="" Body="&lt;p&gt;I always tell students there are three reasons to transform a variable by taking the natural logarithm. The reason for logging the variable will determine whether you want to log the independent variable(s), dependent or both. To be clear throughout I'm talking about taking the natural logarithm. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Firstly, to improve model fit as other posters have noted. For instance if your residuals aren't normally distributed then taking the logarithm of a skewed variable may improve the fit by altering the scale and making the variable more &quot;normally&quot; distributed. For instance, earnings is truncated at zero and often exhibits positive skew. If the variable has negative skew you could firstly invert the variable before taking the logarithm. I'm thinking here particularly of Likert scales that are inputted as continuous variables. While this usually applies to the dependent variable you occasionally have problems with the residuals (e.g. heteroscedasticity) caused by an independent variable which can be sometimes corrected by taking the logarithm of that variable. For example when running a model that explained lecturer evaluations on a set of lecturer and class covariates the variable &quot;class size&quot; (i.e. the number of students in the lecture) had outliers which induced heteroscedasticity because the variance in the lecturer evaluations was smaller in larger cohorts than smaller cohorts. Logging the student variable would help, although in this example either calculating Robust Standard Errors or using Weighted Least Squares may make interpretation easier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second reason for logging one or more variables in the model is for interpretation. I call this convenience reason. If you log both your dependent (Y) and independent (X) variable(s) your regression coefficients (beta) will be elasticities and interpretation would go as follows: a 1% increase in X would lead to a ceteris paribus beta % increase in Y (on average). Logging only one side of the regression &quot;equation&quot; would lead to alternative interpretations as outlined below:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Y and X -- a one unit increase in X would lead to a beta increase/decrease in Y&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Log Y and Log X -- a 1% increase in X would lead to a beta % increase/decrease in Y &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Log Y and X -- a one unit increase in X would lead to a beta*100 % increase/decrease in Y&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Y and Log X -- a 1% increase in X would lead to a beta/100 increase/decrease in Y&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And finally there could be a theoretical reason for doing so. For example some models that we would like to estimate are multiplicative and therefore nonlinear. Taking logarithms allows these models to be estimated by linear regression. Good examples of this include the Cobb-Douglas production function in economics and the Mincer Equation in education. The Cobb-Douglas production function explains how inputs are converted into outputs:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Y = AL^alphaK^beta &lt;/p&gt;&#xA;&#xA;&lt;p&gt;where&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Y is the total production or output of some entity e.g. firm, farm, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A is the total factor productivity (the change in output not caused by the inputs e.g. by technology change or weather)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;L is the labour input&lt;/p&gt;&#xA;&#xA;&lt;p&gt;K is the capital input&lt;/p&gt;&#xA;&#xA;&lt;p&gt;alpha &amp;amp; beta are output elasticities.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Taking logarithms of this makes the function easy to estimate using OLS linear regression as such:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ln Y = A + alphaLn L + beta Ln K&lt;/p&gt;&#xA;" OwnerUserId="215" LastActivityDate="2010-07-23T14:43:14.180" CommentCount="1" />
  <row Id="552" PostTypeId="2" ParentId="486" CreationDate="2010-07-23T14:53:50.583" Score="5" ViewCount="" Body="&lt;p&gt;AIC = -2Ln(L)+ 2k &lt;/p&gt;&#xA;&#xA;&lt;p&gt;where L is the maximised value of Likelihood function for that model and k is the number of parameters in the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your example -2Ln(L)+ 2k &amp;lt;0 means that the log-likelihood at the maximum was &gt; 0&#xA;which means that the likelihood at the maximum was &gt; 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is no problem with a positive log-likelihood. It is a common misconception that the log-likelihood must be negative. If the likelihood is derived from a probability density it can quite reasonably exceed 1 which means that log-likelihood is positive, hence the deviance and the AIC are negative. This is what occurred in your model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you believe that comparing AICs is a good way to choose a model then it would still be the case that the (algebraically) lower AIC is preferred not the one with the lowest absolute AIC value. To reiterate you want the most negative number in your example.&lt;/p&gt;&#xA;" OwnerUserId="215" LastActivityDate="2010-07-23T14:53:50.583" />
  <row Id="553" PostTypeId="2" ParentId="276" CreationDate="2010-07-23T15:02:40.983" Score="4" ViewCount="" Body="&lt;p&gt;The trivial answer is that more data are always preferred to less data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem of small sample size is clear. In linear regression (OLS) technically you can fit a model such as OLS where n = k+1 but you will get rubbish out of it i.e. very large standard errors. There is a great paper by Arthur Goldberger called Micronumerocity on this topic which is summarized in chapter 23 of his book &lt;em&gt;A Course in Econometrics&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A common heuristic is that you should have 20 observations for every parameter you want to estimate. It is always a trade off between the size of your standard errors (and therefore significance testing) and the size of your sample. This is one reason some of us hate significance testing as you can get an incredibly small (relative) standard error with an enormous sample and therefore find pointless statistical significance on naive tests such as whether a regression coefficient is zero. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;While sample size is important the quality of your sample is more important e.g. whether the sample is generalisable to the population, is it a Simple Random Sample or some other appropriate sampling methodology (and have this been accounted for during analysis), is there measurement error, response bias, selection bias, etc.&lt;/p&gt;&#xA;" OwnerUserId="215" LastEditorUserId="215" LastEditorDisplayName="" LastEditDate="2010-07-23T15:20:15.537" LastActivityDate="2010-07-23T15:20:15.537" />
  <row Id="554" PostTypeId="2" ParentId="155" CreationDate="2010-07-23T15:09:02.370" Score="4" ViewCount="" Body="&lt;p&gt;I like to demonstrate sampling variation and essentially the Central Limit Theorem through an &quot;in-class&quot; exercise. Everybody in the class of say 100 students writes their age on a piece of paper. All pieces of paper are the same size and folded in the same fashion after I've calculated the average. This is the population and I calculate the average age. Then each student randomly selects 10 pieces of paper, writes down the ages and returns them to the bag. (S)he calculates the mean and passes the bag along to the next student. Eventually we have 100 samples of 10 students each estimating the population mean which we can describe through a histogram and some descriptive statistics. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We then repeat the demonstration this time using a set of 100 &quot;opinions&quot; that replicate some Yes/No question from recent polls e.g. If the (British General) election were called tomorrow would you consider voting for the British National Party. Students them sample 10 of these opinions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At the end we've demonstrated sampling variation, the Central Limit Theorem, etc with both continuous and binary data.&lt;/p&gt;&#xA;" OwnerUserId="215" LastActivityDate="2010-07-23T15:09:02.370" />
  <row Id="555" PostTypeId="1" AcceptedAnswerId="557" CreationDate="2010-07-23T15:17:56.770" Score="17" ViewCount="821" Body="&lt;p&gt;ANOVA is equivalent to linear regression with the use of suitable dummy variables. The conclusions remain the same irrespective of whether you use ANOVA or linear regression. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In light of their equivalence, is there any reason why ANOVA is used instead of linear regression? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: I am particularly interested in hearing about &lt;strong&gt;technical&lt;/strong&gt; reasons for the use of ANOVA instead of linear regression. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is one example using one-way ANOVA. Suppose, you want to know if the average height of male and females is the same. To test for your hypothesis you would collect data from a random   sample of male and females (say 30 each) and perform the ANOVA analysis (i.e., sum of squares for gender and error) to decide whether an effect exists.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could also use linear regression to test for this as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Define:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Gender = 1 if respondent is a male and 0 otherwise.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Height = Intercept + beta * Gender + error&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where&lt;/p&gt;&#xA;&#xA;&lt;p&gt;error ~ N(0,sigma^2)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then a test of whether beta = 0 is a an equivalent test for your hypothesis.&lt;/p&gt;&#xA;" LastActivityDate="2010-10-29T13:26:46.320" Title="Why is ANOVA taught / used as if it is a different research methodology compared to linear regression?" Tags="&lt;regression&gt;&lt;anova&gt;" AnswerCount="3" CommentCount="2" FavoriteCount="7" />
  <row Id="556" PostTypeId="2" ParentId="206" CreationDate="2010-07-23T15:18:34.053" Score="0" ViewCount="" Body="&lt;p&gt;Discrete data can take on only integer values whereas continuous data can take on any value. For instance the number of cancer patients treated by a hospital each year is discrete but your weight is continuous. Some data are continuous but measured in a discrete way e.g. your age. It is common to report your age as say, 31.&lt;/p&gt;&#xA;" OwnerUserId="215" LastActivityDate="2010-07-23T15:18:34.053" CommentCount="1" />
  <row Id="557" PostTypeId="2" ParentId="555" CreationDate="2010-07-23T15:35:55.653" Score="15" ViewCount="" Body="&lt;p&gt;As an economist, the analysis of variance (ANOVA) is taught and usually understood in relation to linear regression (e.g. in Arthur Goldberger's &lt;em&gt;A Course in Econometrics&lt;/em&gt;). Economists/Econometricians typically view ANOVA as uninteresting and prefer to move straight to regression models. From the perspective of linear (or even generalised linear) models, ANOVA assigns coefficients into batches, with each batch corresponding to a &quot;source of variation&quot; in ANOVA terminology.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Generally you can replicate the inferences you would obtain from ANOVA using regression but not always OLS regression. Multilevel models are needed for analysing hierarchical data structures such as &quot;split-plot designs,&quot; where between-group effects are compared to group-level errors, and within-group effects are compared to data-level errors. Gelman's paper [1] goes into great detail about this problem and effectively argues that ANOVA is an important statistical tool that should still be taught for it's own sake.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In particular Gelman argues that ANOVA is a way of understanding and structuring multilevel models. Therefore ANOVA is not an alternative to regression but as a tool for summarizing complex high-dimensional inferences and for exploratory data analysis. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Gelman is a well-respected statistician and some credence should be given to his view. However, almost all of the empirical work that I do would be equally well served by linear regression and so I firmly fall into the camp of viewing it as a little bit pointless. Some disciplines with complex study designs (e.g. psychology) may find ANOVA useful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[1] Gelman, A. (2005). Analysis of variance: why it is more important than ever (with discussion). &lt;em&gt;Annals of Statistics&lt;/em&gt; 33, 1–53.&lt;/p&gt;&#xA;" OwnerUserId="215" LastActivityDate="2010-07-23T15:35:55.653" CommentCount="4" />
  <row Id="558" PostTypeId="1" CreationDate="2010-07-23T16:15:33.137" Score="4" ViewCount="77" Body="&lt;p&gt;I am trying to get a global perspective on some of the essential ideas in machine learning, and I was wondering if there is a comprehensive treatment of the different notions of loss (squared, log, hinge, proxy, etc.).  I was thinking something along the lines of a more comprehensive, formal presentation of John Langford’s excellent post on &lt;a href=&quot;http://hunch.net/?p=269&quot; rel=&quot;nofollow&quot;&gt;Loss Function Semantics&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="39" LastActivityDate="2010-07-24T19:55:30.013" Title="Comprehensive overview of loss functions?" Tags="&lt;loss-functions&gt;" AnswerCount="3" FavoriteCount="1" />
  <row Id="559" PostTypeId="2" ParentId="548" CreationDate="2010-07-23T16:30:41.750" Score="4" ViewCount="" Body="&lt;p&gt;The general idea is that you should scale the categorical variable in such way that the resulting continuous variables will be just the most useful. So, it is always coupled with some regression or learning procedure and so the fitting of the model is accompanied by optimization (or trying various possibilities) of ordinal variables scaling.&lt;br&gt;&#xA;For some more practical issues, consult the docks of R &lt;a href=&quot;http://cran.r-project.org/web/packages/aspect/index.html&quot; rel=&quot;nofollow&quot;&gt;aspect&lt;/a&gt; and &lt;a href=&quot;http://cran.r-project.org/web/packages/homals/index.html&quot; rel=&quot;nofollow&quot;&gt;homals&lt;/a&gt; packages. &lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-07-23T16:30:41.750" />
  <row Id="560" PostTypeId="2" ParentId="507" CreationDate="2010-07-23T16:36:50.980" Score="5" ViewCount="" Body="&lt;p&gt;I use the Gelman-Rubin convergence diagnostic as well.  A potential problem with Gelman-Rubin is that it may mis-diagnose convergence if the shrink factor happens to be close to 1 by chance, in which case you can use a Gelman-Rubin-Brooks plot.  See the &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.70.2165&quot; rel=&quot;nofollow&quot;&gt;&quot;General Methods for Monitoring Convergence of Iterative Simulations&quot;&lt;/a&gt; paper for details.  This is supported in &lt;a href=&quot;http://cran.r-project.org/web/packages/coda/index.html&quot; rel=&quot;nofollow&quot;&gt;the &lt;strong&gt;coda&lt;/strong&gt; package&lt;/a&gt; in R (for &quot;Output analysis and diagnostics for Markov Chain Monte Carlo simulations&quot;).  &lt;code&gt;coda&lt;/code&gt; also includes other functions (such as the Geweke’s convergence diagnostic).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also have a look at &lt;a href=&quot;http://www.jstatsoft.org/v21/i11/paper&quot; rel=&quot;nofollow&quot;&gt;&quot;boa: An R Package for MCMC Output Convergence&#xA;Assessment and Posterior Inference&quot;&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-07-23T16:36:50.980" CommentCount="1" />
  <row Id="561" PostTypeId="2" ParentId="534" CreationDate="2010-07-23T16:49:47.407" Score="8" ViewCount="" Body="&lt;p&gt;At the heart of your question is the question &quot;when is a relationship causal?&quot; It doesn't just need to be correlation implying (or not) causation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A good book on this topic is called &lt;em&gt;Mostly Harmless Econometrics&lt;/em&gt; by Johua Angrist and Jorn-Steffen Pischke. They start from the experimental ideal where we are able to randomise the &quot;treatment&quot; under study in some fashion and then they move onto alternative methods for generating this randomisation in order to draw causal influences. This begins with the study of so called natural experiments. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of the first examples of a natural experiment being used to identify causal relationships is Angrist's 1989 paper on &lt;a href=&quot;http://www.irs.princeton.edu/pubs/pdfs/251.pdf&quot; rel=&quot;nofollow&quot;&gt;&quot;Lifetime Earnings and the Vietnam Era Draft Lottery.&quot;&lt;/a&gt; This paper attempts to estimate the effect of military service on lifetime earnings. A key problem with estimating any causal effect is that certain types of people may be more likely to enlist, which may bias any measurement of the relationship. Angrist uses the natural experiment created by the Vietnam draft lottery to effectively &quot;randomly assign&quot; the treatment &quot;military service&quot; to a group of men. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So when do we have a causality? Under experimental conditions. When do we get close? Under natural experiments. There are also other techniques that get us close to &quot;causality&quot; i.e. they are much better than simply using statistical control. They include regression discontinuity, difference-in-differences, etc.&lt;/p&gt;&#xA;" OwnerUserId="215" LastActivityDate="2010-07-23T16:49:47.407" />
  <row Id="562" PostTypeId="1" AcceptedAnswerId="566" CreationDate="2010-07-23T16:51:11.790" Score="6" ViewCount="217" Body="&lt;p&gt;This is a fairly general question:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have typically found that using multiple different models outperforms one model when trying to predict a time series out of sample.  Are there any good papers that demonstrate that the combination of models will outperform a single model?  Are there any best-practices around combining multiple models?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some references:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Hui Zoua, Yuhong Yang &lt;a href=&quot;http://www.stat.umn.edu/~hzou/Papers/after.pdf&quot; rel=&quot;nofollow&quot;&gt;&quot;Combining time series models for forecasting&quot;&lt;/a&gt; International Journal of Forecasting 20 (2004) 69– 84&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="5" LastEditorUserId="5" LastEditorDisplayName="" LastEditDate="2010-07-23T16:59:31.433" LastActivityDate="2010-08-04T07:39:52.463" Title="When to use multiple models for prediction?" Tags="&lt;modeling&gt;&lt;model-comparison&gt;" AnswerCount="4" CommentCount="1" FavoriteCount="5" />
  <row Id="563" PostTypeId="1" CreationDate="2010-07-23T16:53:20.117" Score="4" ViewCount="470" Body="&lt;p&gt;Instrumental variables are becoming increasingly common in applied economics and statistics. For the uninitiated, can we have some non-technical answers to the following questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;What is an instrumental variable?&lt;/li&gt;&#xA;&lt;li&gt;When would one want to employ an instrumental variable?&lt;/li&gt;&#xA;&lt;li&gt;How does one find or choose an instrumental variable?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="215" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-11-23T17:06:45.300" LastActivityDate="2010-11-23T17:06:45.300" Title="What is an instrumental variable?" Tags="&lt;regression&gt;&lt;econometrics&gt;&lt;instrumental-variables&gt;" AnswerCount="2" CommentCount="3" />
  <row Id="564" PostTypeId="1" CreationDate="2010-07-23T16:57:50.063" Score="1" ViewCount="355" Body="&lt;p&gt;Difference in differences has long been popular as a non-experimental tool, especially in economics. Can somebody please provide a clear and non-technical answer to the following questions about difference-in-differences.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is a difference-in-difference estimator?&#xA;Why is a difference-in-difference estimator any use?&#xA;Can we actually trust difference-in-difference estimates?&lt;/p&gt;&#xA;" OwnerUserId="215" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-11-23T17:06:13.883" LastActivityDate="2011-01-07T11:51:22.947" Title="What is difference-in-differences?" Tags="&lt;regression&gt;&lt;econometrics&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="565" PostTypeId="2" ParentId="562" CreationDate="2010-07-23T17:02:28.887" Score="2" ViewCount="" Body="&lt;p&gt;The most spectacular example is the &lt;a href=&quot;http://www.netflixprize.com/&quot; rel=&quot;nofollow&quot;&gt;Netflix challenge&lt;/a&gt;, which made really boosted blending popularity.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-07-23T17:02:28.887" CommentCount="1" />
  <row Id="566" PostTypeId="2" ParentId="562" CreationDate="2010-07-23T17:05:52.033" Score="3" ViewCount="" Body="&lt;p&gt;Sometimes this kind of models are called an ensemble. For example &lt;a href=&quot;http://wapedia.mobi/en/Machine_learning_ensemble&quot; rel=&quot;nofollow&quot;&gt;this page&lt;/a&gt; gives a nice overview how it works. Also the references mentioned there are very useful.&lt;/p&gt;&#xA;" OwnerUserId="190" LastActivityDate="2010-07-23T17:05:52.033" CommentCount="6" />
  <row Id="567" PostTypeId="2" ParentId="170" CreationDate="2010-07-23T17:11:35.127" Score="6" ViewCount="" Body="&lt;p&gt;I've often found the Engineering Statistics Handbook useful. It can be found &lt;a href=&quot;http://www.itl.nist.gov/div898/handbook/&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Although I've never read it myself, I hear &lt;a href=&quot;http://www.lulu.com/items/volume_68/8123000/8123594/3/print/IPSUR.pdf&quot; rel=&quot;nofollow&quot;&gt;Introduction to Probability and Statistics Using R&lt;/a&gt; is very good. It's a full ~400 page ebook (also available as an actual book). As a bonus, it also teaches you R, which of course you want to learn anyways.&lt;/p&gt;&#xA;" OwnerUserId="92" LastEditorUserId="92" LastEditorDisplayName="" LastEditDate="2010-07-30T19:07:26.457" LastActivityDate="2010-07-30T19:07:26.457" CommunityOwnedDate="2010-07-23T17:11:35.127" CommentCount="1" />
  <row Id="568" PostTypeId="2" ParentId="31" CreationDate="2010-07-23T17:29:50.730" Score="2" ViewCount="" Body="&lt;p&gt;In statistics you can never say something is absolutely certain, so statisticians use another approach to gauge whether a hypothesis is true or not. They try to reject all the other hypotheses that are not supported by the data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To do this, statistical tests have a null hypothesis and an alternate hypothesis. The p-value reported from a statistical test is the likelihood of the result given that the null hypothesis was correct. That's why we want small p-values. The smaller they are, the less likely the result would be if the null hypothesis was correct. If the p-value is small enough (ie,it is very unlikely for the result to have occurred if the null hypothesis was correct), then the null hypothesis is rejected.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this fashion, null hypotheses can be formulated and subsequently rejected. If the null hypothesis is rejected, you accept the alternate hypothesis as the best explanation. Just remember though that the alternate hypothesis is never certain, since the null hypothesis could have, by chance, generated the results.&lt;/p&gt;&#xA;" OwnerUserId="92" LastActivityDate="2010-07-23T17:29:50.730" />
  <row Id="569" PostTypeId="2" ParentId="562" CreationDate="2010-07-23T17:32:44.140" Score="2" ViewCount="" Body="&lt;p&gt;Following up on Peter's response on ensemble methods:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;This is covered in &lt;a href=&quot;http://www-stat.stanford.edu/~tibs/ElemStatLearn/&quot; rel=&quot;nofollow&quot;&gt;&quot;The Elements of Statistical Learning&quot;&lt;/a&gt; (see page 288, for example).&lt;/li&gt;&#xA;&lt;li&gt;Witten and Frank &lt;a href=&quot;http://www.cs.waikato.ac.nz/~ml/weka/book.html&quot; rel=&quot;nofollow&quot;&gt;&quot;Data Mining: Practical Machine Learning Tools and Techniques&quot;&lt;/a&gt; covers this in section 7.5, including a discussion of Bagging, Randomization, Boosting, Additive regression, Additive logistic regression, Option trees, Logistic model trees, and Stacking.&lt;/li&gt;&#xA;&lt;li&gt;This is covered in Chapter 14 of Christopher M. Bishop &lt;a href=&quot;http://research.microsoft.com/en-us/um/people/cmbishop/prml/&quot; rel=&quot;nofollow&quot;&gt;&quot;Pattern Recognition and Machine Learning&quot;&lt;/a&gt;, including Bayesian Model Averaging, Boosting, Committees, Tree-based Models, and Conditional Mixture Models.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-07-23T17:32:44.140" />
  <row Id="570" PostTypeId="1" AcceptedAnswerId="3798" CreationDate="2010-07-23T17:59:27.367" Score="7" ViewCount="318" Body="&lt;p&gt;I'm curious if there are graphical techniques particular, or more applicable, to structural equation modeling.  I guess this could fall into categories for exploratory tools for covariance analysis or graphical diagnostics for SEM model evaluation.  (I'm not really thinking of path/graph diagrams here.) &lt;/p&gt;&#xA;" OwnerUserId="251" LastEditorUserId="251" LastEditorDisplayName="" LastEditDate="2010-07-27T00:49:21.897" LastActivityDate="2010-10-20T11:31:09.770" Title="What graphical techniques are used in Structural Equation Modeling?" Tags="&lt;sem&gt;&lt;data-visualization&gt;" AnswerCount="3" CommentCount="2" FavoriteCount="2" />
  <row Id="571" PostTypeId="2" ParentId="534" CreationDate="2010-07-23T18:29:41.240" Score="4" ViewCount="" Body="&lt;p&gt;One useful sufficient condition for some definitions of causation:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Causation can be claimed when one of the correlated variables can be controlled (we can directly set its value) and correlation is still present. &lt;/p&gt;&#xA;" OwnerUserId="217" LastActivityDate="2010-07-23T18:29:41.240" CommentCount="1" />
  <row Id="572" PostTypeId="2" ParentId="555" CreationDate="2010-07-23T18:42:05.440" Score="5" ViewCount="" Body="&lt;p&gt;I think Graham's second paragraph gets at the heart of the matter.  I suspect it's not so much technical than historical, probably due to the influence of &quot;&lt;a href=&quot;http://en.wikipedia.org/wiki/Ronald_Fisher&quot; rel=&quot;nofollow&quot;&gt;Statistical Methods for Research Workers&lt;/a&gt;&quot;, and the ease of teaching/applying the tool for non-statisticans in experimental analysis involving discrete factors, rather than delving into model building and associated tools.  In statistics, ANOVA is usually taught as a special case of regression.  (I think this is similar to why biostatistics is filled with a myriad of eponymous &quot;tests&quot; rather than emphasizing model building.)&lt;/p&gt;&#xA;" OwnerUserId="251" LastEditorUserId="251" LastEditorDisplayName="" LastEditDate="2010-07-23T18:54:03.763" LastActivityDate="2010-07-23T18:54:03.763" />
  <row Id="573" PostTypeId="1" CreationDate="2010-07-23T19:45:45.270" Score="6" ViewCount="263" Body="&lt;p&gt;In &quot;&lt;a href=&quot;http://dx.doi.org/10.1145/1553374.1553453&quot; rel=&quot;nofollow&quot;&gt;Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations&lt;/a&gt;&quot; by Lee et. al.(&lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.149.802&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;PDF&lt;/a&gt;) Convolutional DBN's are proposed. Also the method is evaluated for image classification. This sounds logical, as there are natural local image features, like small corners and edges etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In &quot;&lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.154.380&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;Unsupervised feature learning for audio classification using convolutional deep belief networks&lt;/a&gt;&quot; by Lee et. al. this method is applied for audio in different types of classifications. Speaker identification, gender indentification, phone classification and also some music genre / artist classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can the convolutional part of this network be interpreted for audio, like it can be explained for images as edges?&lt;/p&gt;&#xA;" OwnerUserId="190" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-11-01T03:20:03.513" LastActivityDate="2010-11-01T03:29:42.770" Title="How to understand a convolutional deep belief network for audio classification" Tags="&lt;classification&gt;&lt;intuition&gt;&lt;unsupervised-learning&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="1" />
  <row Id="574" PostTypeId="2" ParentId="212" CreationDate="2010-07-23T20:07:51.003" Score="0" ViewCount="" Body="&lt;p&gt;Assuming there is some training involved, you may use some kind of cross-validation, or bootstrap of a train set.&lt;br&gt;&#xA;If not, stick to the Srikant solution. I would do it even simpler, just assuming that the number of error is Poisson distributed.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-07-23T20:07:51.003" CommentCount="2" />
  <row Id="575" PostTypeId="1" AcceptedAnswerId="762" CreationDate="2010-07-23T20:14:05.327" Score="8" ViewCount="769" Body="&lt;p&gt;What is the preferred method for for conducting post-hocs for within subjects tests?  I've seen published work where Tukey's HSD is employed but a review of Keppel and Maxwell &amp;amp; Delaney suggests that the likely violation of sphericity in these designs makes the error term incorrect and this approach problematic.  Maxwell &amp;amp; Delaney provide an approach to the problem in their book, but I've never seen it done that way in any stats package.  Is the approach they offer appropriate?  Would a Bonferoni or Sidak correction on multiple paired sample t-tests be reasonable?  An acceptable answer will provide general R code which can conduct post-hocs on a simple, multiple-way, and mixed designs as produced by the ezANOVA function in the ez package and appropriate citations that are likely to pass muster with reviewers.&lt;/p&gt;&#xA;" OwnerUserId="196" LastEditorUserId="196" LastEditorDisplayName="" LastEditDate="2010-07-27T05:20:08.040" LastActivityDate="2010-08-01T18:14:31.930" Title="Post-hocs for within subjects tests?" Tags="&lt;r&gt;&lt;post-hoc&gt;&lt;sphericity&gt;&lt;within-subjects&gt;" AnswerCount="5" FavoriteCount="4" />
  <row Id="576" PostTypeId="2" ParentId="564" CreationDate="2010-07-23T20:42:43.260" Score="2" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Difference_in_differences&quot; rel=&quot;nofollow&quot;&gt;Wikipedia has a decent entry on this subject&lt;/a&gt;, but why not just use linear regression allowing for interactions between your independent variables of interest? This seems more interpretable to me. Then you might read up on &lt;a href=&quot;http://books.google.com/books?id=fuq94a8C0ioC&amp;amp;lpg=PP1&amp;amp;dq=applied%20multiple%20regression%20correlation%20analysis%20for%20the%20behavioral%20sciences&amp;amp;pg=PA271#v=onepage&amp;amp;q=simple%20slopes&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;analysis of simple slopes (in the Cohen et al book free on Google Books)&lt;/a&gt; if your variables of interest are quantitative.&lt;/p&gt;&#xA;" OwnerUserId="36" LastActivityDate="2010-07-23T20:42:43.260" />
  <row Id="577" PostTypeId="1" AcceptedAnswerId="767" CreationDate="2010-07-23T20:49:12.340" Score="30" ViewCount="1809" Body="&lt;p&gt;The AIC and BIC are both methods of assessing model fit penalized for the number of estimated parameters.  As I understand it, BIC penalizes models more for free parameters than does AIC.  Beyond a preference based on the stringency of the criteria, are there any other reasons to prefer AIC over BIC or vice versa?&lt;/p&gt;&#xA;" OwnerUserId="196" LastEditorUserId="196" LastEditorDisplayName="" LastEditDate="2010-07-25T05:09:18.723" LastActivityDate="2011-05-13T14:06:44.390" Title="Is there any reason to prefer the AIC or BIC over the other?" Tags="&lt;modeling&gt;&lt;aic&gt;&lt;cross-validation&gt;&lt;bic&gt;&lt;model-selection&gt;" AnswerCount="9" CommentCount="0" FavoriteCount="15" />
  <row Id="578" PostTypeId="2" ParentId="527" CreationDate="2010-07-23T21:18:52.370" Score="2" ViewCount="" Body="&lt;p&gt;If you have no way of knowing the true concentration, the simplest approach would be a correlation.  A step beyond that might be to conduct a simple regression predicting the outcome on method 2 using method 1 (or vice versa). If the methods are identical the intercept should be 0; if the intercept is greater or less than 0 it would indicate the bias of one method relative to another.  The unstandardized slope should be near 1 if the methods on average produce results that are identical (after controlling for an upward or downward bias in the intercept).  The error in the unstandardized slope might serve as an index of the extent to which the two methods agree.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems to me that the difficulty with statistical methods here that you are seeking to affirm what is typically posed as a null hypothesis, that is, that there are no differences between the methods.  This isn't a death blow for using statistical methods so long as you don't need a p value and you can quantify what you mean by &quot;equivalent&quot; and can decide how much deviation the two methods can have from one another before you no longer consider them equivalent.  In the regression approach I detailed above, you might consider the methods equivalent if confidence interval around the slope estimate included 1 and the CI around the intercept included 0.&lt;/p&gt;&#xA;" OwnerUserId="196" LastActivityDate="2010-07-23T21:18:52.370" CommentCount="1" />
  <row Id="579" PostTypeId="2" ParentId="577" CreationDate="2010-07-23T21:23:18.427" Score="3" ViewCount="" Body="&lt;p&gt;Indeed the only difference is that BIC is AIC extended to take number of objects (samples) into account. I would say that while both are quite weak (in comparison to for instance cross-validation) it is better to use AIC, than more people will be familiar with the abbreviation -- indeed I have never seen a paper or a program where BIC would be used (still I admit that I'm biased to problems where such criteria simply don't work).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: AIC and BIC are equivalent to cross-validation provided two important asumptions -- when they are defined, so when the model is a maximum likelihood one and when you are only interested in model performance on a training data. In case of collapsing some data into some kind of consensus they are perfectly ok.&lt;br&gt;&#xA;In case of making a prediction machine for some real-world problem the first is false, since your training set represent only a scrap of information about the problem you are dealing with, so you just can't optimize your model; the second is false, because you expect that your model will handle the new data for witch you can't even expect that the training set will be representative. &#xA;And to this end CV was invented; to simulate the behavior of the model when confronted with an independent data. In case of model selection, CV gives you not only the quality approximate, but also quality approximation distribution, so it has this great advantage that it can say &quot;I don't know, whatever the new data will come, either of them can be better.&quot;  &lt;/p&gt;&#xA;" OwnerUserId="88" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-07-24T21:03:17.387" LastActivityDate="2010-07-24T21:03:17.387" CommentCount="11" />
  <row Id="581" PostTypeId="1" AcceptedAnswerId="588" CreationDate="2010-07-23T22:12:36.750" Score="1" ViewCount="456" Body="&lt;p&gt;I am currently using Viterbi training for an image segementation probelm.   I wanted to know what are the advantages/disadvantages of using the Baum-Welch algorithm instead of Viterbi training.   &lt;/p&gt;&#xA;" OwnerUserId="99" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-07-26T13:48:06.203" LastActivityDate="2010-07-26T13:48:06.203" Title="Differences between Baum Welch and Viterbi Training" Tags="&lt;machine-learning&gt;&lt;image-processing&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="582" PostTypeId="2" ParentId="577" CreationDate="2010-07-23T23:38:20.190" Score="4" ViewCount="" Body="&lt;p&gt;As you mentioned, AIC and BIC are methods to penalize models for having more regressor variables. A penalty function is used in these methods, which is a function of the number of parameters in the model. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;When applying AIC, the penalty function is &lt;em&gt;z(p)&lt;/em&gt; = 2 &lt;em&gt;p&lt;/em&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;When applying BIC, the penalty function is &lt;em&gt;z(p)&lt;/em&gt; = p ln(&lt;em&gt;n&lt;/em&gt;), which is based on interpreting the penalty as deriving from prior information (hence the name Bayesian Information Criterion).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;When &lt;em&gt;n&lt;/em&gt; is large the two models will produce quite different results. Then the BIC applies a much larger penalty for complex models, and hence will lead to simpler models than AIC. However, as stated in &lt;a href=&quot;http://en.wikipedia.org/wiki/Bayesian_information_criterion&quot; rel=&quot;nofollow&quot;&gt;Wikipedia on BIC&lt;/a&gt;: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;it should be noted that in many&#xA;  applications..., BIC simply reduces to&#xA;  maximum likelihood selection because&#xA;  the number of parameters is equal for&#xA;  the models of interest.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="108" LastEditorUserId="108" LastEditorDisplayName="" LastEditDate="2010-07-23T23:43:40.540" LastActivityDate="2010-07-23T23:43:40.540" CommentCount="1" />
  <row Id="583" PostTypeId="2" ParentId="577" CreationDate="2010-07-24T00:07:06.830" Score="13" ViewCount="" Body="&lt;p&gt;Though AIC and BIC are both &lt;a href=&quot;http://en.wikipedia.org/wiki/Maximum_likelihood&quot; rel=&quot;nofollow&quot;&gt;maximum likelihood estimate&lt;/a&gt; driven and penalize free parameters in an effort to combat overfitting, they do so in ways that result in significantly different behavior.  Lets look at one commonly presented version of the methods (which results form stipulating normally distributed errors and other well behaving assumptions):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;AIC&lt;/strong&gt; = -2*ln(likelihood) + 2*k,&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;and &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;BIC&lt;/strong&gt; = -2*ln(likelihood) + ln(N)*k,&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;where:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;k = model degrees of freedom&lt;/li&gt;&#xA;&lt;li&gt;N = number of observations&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The best model in the group compared is the one that minimizes these scores, in both cases.  Clearly, AIC does not depend directly on sample size.  Moreover, generally speaking, AIC presents the danger that it might overfit, whereas BIC presents the danger that it might underfit, simply in virtue of how they penalize free parameters (2*k in AIC; ln(N)*k in BIC). Diachronically, as data is introduced and the scores are recalculated, at relatively low N (7 and less) BIC is more tolerant of free parameters than AIC, but less tolerant at higher N (as the natural log of N overcomes 2).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Additionally, AIC is aimed at finding the best approximating model to the unknown data generating process (via minimizing expected estimated &lt;a href=&quot;http://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence&quot; rel=&quot;nofollow&quot;&gt;K-L divergence&lt;/a&gt;).  As such, it fails to converge in probability to the true model (assuming one is present in the group evaluated), whereas BIC does converge as n tends to infinity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, as in many methodological questions, which is to be preferred depends upon what you are trying to do, what other methods are available, and whether or not any of the features outlined (convergence, relative tolerance for free parameters, minimizing expected K-L divergence) speak to your goals.&lt;/p&gt;&#xA;" OwnerUserId="39" LastEditorUserId="39" LastEditorDisplayName="" LastEditDate="2011-04-29T18:30:11.667" LastActivityDate="2011-04-29T18:30:11.667" CommentCount="1" />
  <row Id="584" PostTypeId="2" ParentId="581" CreationDate="2010-07-24T00:44:43.103" Score="0" ViewCount="" Body="&lt;p&gt;Forward-backward is used when you want to count 'invisible things'. For example, when using E-M to improve a model via unsupervised data. I think that Petrov's paper is an example. In the technique I'm thinking of, you first train a model with annotated data with fairly coarse annotations (e.g. a tag for 'Verb'). Then you arbitrarily split the probability mass for that state in two slightly unequal quantities, and retrain, running forward-backward to maximize likelihood by redistributing mass between the two states.&lt;/p&gt;&#xA;" OwnerUserId="240" LastActivityDate="2010-07-24T00:44:43.103" />
  <row Id="585" PostTypeId="2" ParentId="526" CreationDate="2010-07-24T00:46:15.330" Score="3" ViewCount="" Body="&lt;p&gt;You can certainly get different results simply because you train on different examples. I very much doubt that there's an algorithm or problem domain where the results of the two would differ in some predictable way.&lt;/p&gt;&#xA;" OwnerUserId="240" LastActivityDate="2010-07-24T00:46:15.330" CommentCount="1" />
  <row Id="586" PostTypeId="2" ParentId="558" CreationDate="2010-07-24T01:41:28.650" Score="2" ViewCount="" Body="&lt;p&gt;The &lt;a href=&quot;http://yann.lecun.com/exdb/publis/#lecun-06&quot; rel=&quot;nofollow&quot;&gt;Tutorial on Energy-Based Learning&lt;/a&gt; by LeCun et al. might get you a good part of the way there.  They describe a number of loss functions and discuss what makes them &quot;good or bad&quot; for energy based models.&lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-07-24T01:41:28.650" />
  <row Id="587" PostTypeId="2" ParentId="577" CreationDate="2010-07-24T03:58:58.333" Score="12" ViewCount="" Body="&lt;p&gt;My quick explanation is&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;AIC is best for prediction as it is asymptotically equivalent to cross-validation.&lt;/li&gt;&#xA;&lt;li&gt;BIC is best for explanation as it is allows consistent estimation of the underlying data generating process.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-07-24T03:58:58.333" CommentCount="5" />
  <row Id="588" PostTypeId="2" ParentId="581" CreationDate="2010-07-24T04:22:01.467" Score="3" ViewCount="" Body="&lt;p&gt;The Baum-Welch algorithm and the Viterbi algorithm calculate different things.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you know the transition probabilities for the hidden part of your model, and the emission probabilities for the visible outputs of your model, then the Viterbi algorithm gives you the most likely &lt;em&gt;complete&lt;/em&gt; sequence of hidden states conditional on both your outputs and your model specification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Baum-Welch algorithm gives you both the mostl likely hidden transition probabilities as well as the most likely set of emission probabilities given only the observed states of the model (and, usually, an upper bound on the number of hidden states).  You also get the &quot;pointwise&quot; highest likelihood points in the hidden states, which is often slightly different from the single hidden sequence that is overall most likely.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you know your model and just want the latent states, then there is no reason to use the Baum-Welch algorithm.  If you don't know your model, then you can't be using the Viterbi algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edited to add: See Peter Smit's comment; there's some overlap/vagueness in nomenclature.  Some poking around led me to a chapter by Luis Javier Rodrıguez and Ines Torres in &quot;Pattern Recognition and Image Analysis&quot; (ISBN 978-3-540-40217-6, pp 845-857) which discusses the speed versus accuracy trade-offs of the two algorithms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Briefly, the Baum-Welch algorithm is essentially the Expectation-Maximization algorithm applied to a HMM; as a strict EM-type algorithm you're guaranteed to converge to at least a local maximum, and so for unimodal problems find the MLE.  It requires two passes over your data for each step, though, and the complexity gets very big in the length of the data and number of training samples.  However you do end up with the full conditional likelihood for your hidden parameters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Viterbi training algorithm (as opposed to the &quot;Viterbi algorithm&quot;) approximates the MLE to achieve a gain in speed at the cost of accuracy.  It segments the data and then applies the Viterbi algorithm (as I understood it) to get the most likely state sequence in the segment, then uses that most likely state sequence to re-estimate the hidden parameters.  This, unlike the Baum-Welch algorithm, doesn't give the full conditional likelihood of the hidden parameters, and so ends up reducing the accuracy while saving significant (the chapter reports 1 to 2 orders of magnitude) computational time.&lt;/p&gt;&#xA;" OwnerUserId="61" LastEditorUserId="61" LastEditorDisplayName="" LastEditDate="2010-07-24T19:31:34.640" LastActivityDate="2010-07-24T19:31:34.640" CommentCount="2" />
  <row Id="589" PostTypeId="2" ParentId="411" CreationDate="2010-07-24T04:54:53.943" Score="5" ViewCount="" Body="&lt;p&gt;Computational issues are the strongest argument I've heard one way or the other.  The single biggest advantage of the Kolmogorov distance is that it's very easy to compute analytically for pretty much any CDF.  Most other distance metrics don't have a closed-form expression except, sometimes, in the Gaussian case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Kolmogorov distance of a sample also has a known sampling distribution given the CDF (I don't think most other ones do), which ends up being related to the Wiener process.  This is the basis for the Kolmogorov-Smirnoff test for comparing a sample to a distribution or two samples to each other.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On a more functional-analysis note, the sup norm is nice in that (as you mention) it basically defines uniform convergence.  This leaves you with norm convergence implying pointwise convergence, and so you if you're clever about how you define your function sequences you can work within a RKHS and use all of the nice tools that that provides as well.&lt;/p&gt;&#xA;" OwnerUserId="61" LastActivityDate="2010-07-24T04:54:53.943" />
  <row Id="590" PostTypeId="1" AcceptedAnswerId="611" CreationDate="2010-07-24T09:09:14.243" Score="2" ViewCount="135" Body="&lt;p&gt;In some papers, for example in &lt;a href=&quot;http://www.jstor.org/stable/2239262&quot; rel=&quot;nofollow&quot;&gt;&quot;The Geometric Density with Unknown Location Parameter&quot;&lt;/a&gt; by Klotz, a Geometric Distribution is called a Geometric Density.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For me, this claim looks erroneous, however Klotz is a serious statistician and a professor in the field.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is, to what extend is it legitimate to call a Geometric Distribution a Geometric Density?&lt;/p&gt;&#xA;" OwnerUserId="190" LastEditorUserId="190" LastEditorDisplayName="" LastEditDate="2010-07-25T10:09:23.647" LastActivityDate="2010-08-03T19:39:42.083" Title="To what extent can we call a Geometric Distribution a Geometric Density" Tags="&lt;distributions&gt;&lt;discrete-data&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="591" PostTypeId="2" ParentId="558" CreationDate="2010-07-24T09:33:01.940" Score="1" ViewCount="" Body="&lt;p&gt;The loss function is given by the problem. It could be anything. For example, you could also penalize the used CPU time and space.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In reinforcement learning, the loss function is an unknown non-deterministic function.&#xA;You cannot redefine it without changing the problem.&lt;/p&gt;&#xA;" OwnerUserId="200" LastActivityDate="2010-07-24T09:33:01.940" />
  <row Id="592" PostTypeId="2" ParentId="421" CreationDate="2010-07-24T09:52:52.377" Score="2" ViewCount="" Body="&lt;p&gt;It is a bit old, but I have found Chris Chatfield's book,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0412253402&quot; rel=&quot;nofollow&quot;&gt;Statistics for Technology: A Course in Applied Technology&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;to be an excellent introduction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It was how I first learned about statistics from a conceptual point of view.&lt;/p&gt;&#xA;" OwnerUserId="57" LastEditorUserId="509" LastEditorDisplayName="" LastEditDate="2010-08-11T08:38:27.217" LastActivityDate="2010-08-11T08:38:27.217" CommunityOwnedDate="2010-07-24T09:52:52.377" CommentCount="2" />
  <row Id="593" PostTypeId="2" ParentId="575" CreationDate="2010-07-24T12:09:08.743" Score="3" ViewCount="" Body="&lt;p&gt;I recall some discussion on this in the past; I'm not aware of any implementation of Maxwell &amp;amp; Delaney's approach, although it shouldn't be too difficult to do.  Have a look at &quot;&lt;a href=&quot;http://gribblelab.org/2009/03/09/repeated-measures-anova-using-r/&quot; rel=&quot;nofollow&quot;&gt;Repeated Measures ANOVA using R&lt;/a&gt;&quot; which also shows one method of addressing the sphericity issue in &lt;a href=&quot;http://en.wikipedia.org/wiki/Tukey%27s_range_test&quot; rel=&quot;nofollow&quot;&gt;Tukey's HSD&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might also find &lt;a href=&quot;http://www.r-statistics.com/2010/02/post-hoc-analysis-for-friedmans-test-r-code/&quot; rel=&quot;nofollow&quot;&gt;this description of Friedman's test&lt;/a&gt; of interest.&lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-07-24T12:09:08.743" CommentCount="1" />
  <row Id="594" PostTypeId="1" AcceptedAnswerId="599" CreationDate="2010-07-24T19:25:59.147" Score="6" ViewCount="102" Body="&lt;p&gt;The E-M procedure appears, to the uninitiated, as more or less black magic. Estimate parameters of an HMM (for example) using supervised data. Then decode untagged data, using forward-backward to 'count' events as if the data were tagged, more or less. Why does this make the model better? I do know something about the math, but I keep wishing for some sort of mental picture of it.&lt;/p&gt;&#xA;" OwnerUserId="240" LastActivityDate="2010-07-24T20:23:06.997" Title="E-M, is there an intuitive explanation?" Tags="&lt;e-m&gt;&lt;intuition&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="1" />
  <row Id="595" PostTypeId="2" ParentId="492" CreationDate="2010-07-24T19:43:24.270" Score="4" ViewCount="" Body="&lt;p&gt;You could use the (fast :) ) discrete &lt;strong&gt;wavelet transform&lt;/strong&gt;. The package wavethresh under R will do all the work. &#xA;Anyway, I like the solution of @James because it is simple and seems to go straigh to the point.  &lt;/p&gt;&#xA;" OwnerUserId="223" LastEditorUserId="223" LastEditorDisplayName="" LastEditDate="2010-08-03T19:40:40.863" LastActivityDate="2010-08-03T19:40:40.863" CommentCount="5" />
  <row Id="596" PostTypeId="1" CreationDate="2010-07-24T19:50:48.730" Score="0" ViewCount="130" Body="&lt;p&gt;E-M provides a way to improve the estimation of a generative model with unannotated data. Is there anything out there that works the same way for discriminative models (e.g. perceptrons)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, consider averaged perceptron tagger. It would be handy to be able to throw the entire Gigaword through some process of unsupervised model improvement.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, I was pleasantly surprise to note that this site has the ambition of dealing with machine learning, but I'm learning by experiment what vocabulary is generic and what is very domain-specific. Apologies.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider a sequence classification problem, like part-of-speech tagging or named entity extraction. You can train a generative model (e.g. an HMM). Thats's a probability model, and you can apply E-M. However, the number of states grows prohibitive if you want to look at many features, and so the fashion tends toward things like CRFs (batch) or Perceptron (online).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, &lt;a href=&quot;http://docs.google.com/viewer?a=v&amp;amp;q=cache%3alFSlhiI9L5cJ%3awww.aclweb.org/anthology/E/E09/E09-1087.pdf+unsupervised+perceptron+training&amp;amp;hl=en&amp;amp;gl=us&amp;amp;pid=bl&amp;amp;srcid=ADGEESjBE8kls2rWhD_U1fL9lz0Wq_elscJ6orH7KNx0ryr0hh37jnzp53sOA-jzJrSI0gea3AwN4lLkq7jsNFbAUcooyDgyYRO2BSqZdpBH4x84KlzLlsccaMcQNbk-1zLvFDAIZVy8&amp;amp;sig=AHIEtbTx9N6TyrKV_7dza3xG9G_dhNxfVg&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; paper talks about unsupervised learning in for a perceptron POS tagger, but the details are that they add the output of several pre-existing taggers to the training set of their model.&lt;/p&gt;&#xA;" OwnerUserId="240" LastEditorUserId="240" LastEditorDisplayName="" LastEditDate="2010-07-26T00:32:30.473" LastActivityDate="2010-08-15T23:43:45.330" Title="Something like E-M for discriminative models?" Tags="&lt;machine-learning&gt;&lt;e-m&gt;" AnswerCount="3" CommentCount="4" />
  <row Id="597" PostTypeId="2" ParentId="558" CreationDate="2010-07-24T19:55:30.013" Score="2" ViewCount="" Body="&lt;p&gt;Well, there's &lt;a href=&quot;http://docs.google.com/viewer?a=v&amp;amp;q=cache%3acfjRh4vE38kJ%3abooks.nips.cc/papers/files/nips16/NIPS2003_LT21.pdf+passive-aggressive+machine+learning&amp;amp;hl=en&amp;amp;gl=us&amp;amp;pid=bl&amp;amp;srcid=ADGEEShVh5WierOOrVz8pTpfOPVUCm8HIqkUCm1rxCsUPm5UUapNONytwRMjMq63zcOye0L3djRjQCZW6YN_r12SvIEaWCtbx3b5La7EJ8nQuoWm2Jw8n8DuLEjS19K56fh8DZLOfrHu&amp;amp;sig=AHIEtbRBrb7Q7Lx6LSh_fKEFXyWaxW0RVw&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; and &lt;a href=&quot;http://portal.acm.org/citation.cfm?id=1248547.1248566&quot; rel=&quot;nofollow&quot;&gt;that&lt;/a&gt;. Two papers by Cramer and others discussing loss in the context of online learning algorithms.&lt;/p&gt;&#xA;" OwnerUserId="240" LastActivityDate="2010-07-24T19:55:30.013" />
  <row Id="598" PostTypeId="2" ParentId="3" CreationDate="2010-07-24T20:15:20.653" Score="5" ViewCount="" Body="&lt;p&gt;Few more on top of already mentioned:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.knime.org/&quot; rel=&quot;nofollow&quot;&gt;KNIME&lt;/a&gt; together with R, Python and Weka integration extensions for data mining   &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://rosuda.org/mondrian/&quot; rel=&quot;nofollow&quot;&gt;Mondrian&lt;/a&gt; for quick EDA     &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;And from spatial perspective:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://geodacenter.asu.edu/ogeoda&quot; rel=&quot;nofollow&quot;&gt;GeoDa&lt;/a&gt; for spatial EDA and clustering of areal data&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.satscan.org/&quot; rel=&quot;nofollow&quot;&gt;SaTScan&lt;/a&gt; for clustering of point data&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="22" LastActivityDate="2010-07-24T20:15:20.653" CommunityOwnedDate="2010-07-24T20:15:20.653" CommentCount="2" />
  <row Id="599" PostTypeId="2" ParentId="594" CreationDate="2010-07-24T20:23:06.997" Score="6" ViewCount="" Body="&lt;p&gt;Just to save some typing, call the observed data X, the missing data Z (e.g. the hidden states of the HMM), and the parameter vector we're trying to find Q (e.g. transition/emission probabilities).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The intuitive explanation is that we basically cheat, pretend for a moment we know Q so we can find a conditional distribution of Z that in turn lets us find the MLE for Q (ignoring for the moment the fact that we're basically making a circular argument), then admit that we cheated, put in our new, better value for Q, and do it all over again until we don't have to cheat anymore.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Slightly more technically, by pretending that we know the real value Q, we can pretend we know something about the conditional distribution of Z|{X,Q}, which lets us improve our estimate for Q, which we now pretend is the real value for Q so we can pretend we know something about the conditional distribution of Z|{X,Q}, which lets us improve our estimate for Q, which... and so on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even more technically, if we knew Z, we could maximize log(f(Q|X,Z)) and have the right answer.  The problem is that we don't know Z, and any estimate for Q must depend on it.   But if we want to find the best estimate (or distribution) for Z, then we need to know X and Q.  We're stuck in a chicken-and-egg situation if we want the unique maximizer analytically.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Our 'out' is that -- for any estimate of Q (call it Q_n) -- we can find the distribution of Z|{Q_n,X}, and so we can maximize our &lt;em&gt;expected&lt;/em&gt; joint log-likelihood of Q|{X,Z}, with respect to the conditional distribution of Z|{Q_n,X}.  This conditional distribution basically tells us how Z depends on the current value of Q given X, and lets us know how to change Q to increase our likelihood for both Q and Z at the same time for a particular value of Q (that we've called Q_n).  Once we've picked out a new Q_n+1, we have a different conditional distribution for Z|{Q_n+1, X} and so have to re-calculate the expectation.&lt;/p&gt;&#xA;" OwnerUserId="61" LastActivityDate="2010-07-24T20:23:06.997" />
  <row Id="600" PostTypeId="2" ParentId="411" CreationDate="2010-07-24T20:36:13.223" Score="6" ViewCount="" Body="&lt;p&gt;Mark,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;the main reason of which I am aware for the use of K-S is because it arises naturally from Glivenko-Cantelli theorems in univariate empirical processes. The one reference I'd recommend is A.W.van der Vaart &quot;Asymptotic Statistics&quot;, ch. 19. A more advanced monograph is &quot;Weak Convergence and Empirical Processes&quot; by Wellner and van der Vaart. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd add two quick notes:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;another measure of distance commonly used in univariate distributions is the Cramer-von Mises distance, which is an L^2 distance;&lt;/li&gt;&#xA;&lt;li&gt;in general vector spaces different distances are employed; the space of interest in many papers is polish. A very good introduction is Billingsley's &quot;Convergence of Probability Measures&quot;. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I apologize if I can't be more specific. I hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="30" LastActivityDate="2010-07-24T20:36:13.223" CommentCount="2" />
  <row Id="601" PostTypeId="2" ParentId="596" CreationDate="2010-07-24T21:11:14.777" Score="0" ViewCount="" Body="&lt;p&gt;Unsupervised learning?&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-07-24T21:11:14.777" CommentCount="2" />
  <row Id="602" PostTypeId="1" AcceptedAnswerId="1733" CreationDate="2010-07-24T22:29:40.557" Score="3" ViewCount="124" Body="&lt;p&gt;Do you know any good heuristics for finding optimal value of ν in case of ν-SVM classification? In this particular problem I have a radial basis kernel, if it helps.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-08-16T09:23:45.913" Title="Heuristics for optimizing ν-SVM?" Tags="&lt;machine-learning&gt;&lt;svm&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="603" PostTypeId="2" ParentId="526" CreationDate="2010-07-24T22:54:13.430" Score="3" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Usually of course the difference is&#xA;  unnoticeable, and so goes my question&#xA;  -- can you think of an example when the result of one type is&#xA;  significantly different from another?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I am not sure at all the difference is unnoticeable, and that only in ad hoc example it will be noticeable. Both cross-validation and bootstrapping (sub-sampling) methods depend critically on their design parameters, and this understanding is not complete yet. In general, results &lt;em&gt;within&lt;/em&gt; k-fold cross-validation depend critically on the number of folds, so you can expect always different results from what you would observe in sub-sampling.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Case in point: say that you have a true linear model with a fixed number of parameters. If you use k-fold cross-validation (with a given, fixed k), and let the number of observations go to infinity, k-fold cross validation will be asymptotically inconsistent for model selection, i.e., it will identify an incorrect model with probability greater than 0. This surprising result is due to Jun Shao, &quot;Linear Model Selection by Cross-Validation&quot;, &lt;em&gt;Journal of the American Statistical Association&lt;/em&gt;, &lt;strong&gt;88&lt;/strong&gt;, 486-494 (1993), but more papers can be found in this vein.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, respectable statistical papers specify the cross-validation protocol, exactly because results are not invariant. In the case where they choose a large number of folds for large datasets, they remark and try to correct for biases in model selection.&lt;/p&gt;&#xA;" OwnerUserId="30" LastActivityDate="2010-07-24T22:54:13.430" CommentCount="5" />
  <row Id="604" PostTypeId="1" CreationDate="2010-07-24T23:09:05.870" Score="7" ViewCount="326" Body="&lt;p&gt;I am puzzled by something I found using Linear Discriminant Analysis. Here is the problem - I first ran the Discriminant analysis using 20 or so independent variables to predict 5 segments. Among the outputs, I asked for the Predicted Segments, which are the same as the original segments for around 80% of the cases. Then I ran again the Discriminant Analysis with the same independent variables, but now trying to predict the Predicted Segments. I was expecting I would get 100% of correct classification rate, but that did not happen and I am not sure why. It seems to me that if the Discriminant Analysis cannot predict with 100% accuracy it own predicted segments then somehow it is not a optimum procedure since a rule exist that will get 100% accuracy. I am missing something?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note - This situation seems to be similar to that in Linear Regression Analysis. If you fit the model $y = a + bX + \text{error}$ and use the estimated equation with the same data you will get $\hat{y}$ [$= \hat{a} + \hat{b}X$]. Now if you estimate the model $\hat{y} = \hat{a} + \hat{b}X + \text{error}$, you will find the same $\hat{a}$ and $\hat{b}$ as before, no error, and R2 = 100% (perfect fit). I though this would also happen with Linear Discriminant Analysis, but it does not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note 2 - I run this test with Discriminant Analysis in SPSS.&lt;/p&gt;&#xA;" OwnerUserId="165" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-03T09:00:53.737" LastActivityDate="2011-05-03T09:00:53.737" Title="Why prediction of a predicted variable from a discriminant analysis is imperfect" Tags="&lt;regression&gt;&lt;discriminant-analysis&gt;" AnswerCount="2" CommentCount="6" FavoriteCount="2" />
  <row Id="605" PostTypeId="2" ParentId="604" CreationDate="2010-07-24T23:25:48.150" Score="1" ViewCount="" Body="&lt;p&gt;This is quite normal in case of machine learning -- it does not need to be optimal, it must be general. &lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-07-24T23:25:48.150" CommentCount="1" />
  <row Id="606" PostTypeId="2" ParentId="490" CreationDate="2010-07-25T02:37:06.947" Score="5" ViewCount="" Body="&lt;p&gt;A very popular approach is penalized logistic regression, in which one maximizes the sum of the log-likelihood and a penalization term consisting of the L1-norm (&quot;lasso&quot;), L2-norm (&quot;ridge&quot;), a combination of the two (&quot;elastic&quot;), or a penalty associated to groups of variables (&quot;group lasso&quot;). This approach has several advantages:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;It has strong theoretical properties, e.g., &lt;a href=&quot;http://www-stat.stanford.edu/~candes/papers/LassoPredict.pdf&quot; rel=&quot;nofollow&quot;&gt;see this paper by Candes &amp;amp; Plan&lt;/a&gt; and close connections to compressed sensing;&lt;/li&gt;&#xA;&lt;li&gt;It has accessible expositions, e.g., in &lt;a href=&quot;http://www-stat.stanford.edu/~tibs/ElemStatLearn/download.html&quot; rel=&quot;nofollow&quot;&gt;Elements of Statistical Learning&lt;/a&gt; by Friedman-Hastie-Tibshirani (available online);&lt;/li&gt;&#xA;&lt;li&gt;It has readily available software to fit models. R has the &lt;a href=&quot;http://cran.r-project.org/web/packages/glmnet/index.html&quot; rel=&quot;nofollow&quot;&gt;glmnet&lt;/a&gt; package which is very fast and works well with pretty large datasets. Python has &lt;a href=&quot;http://scikit-learn.sourceforge.net&quot; rel=&quot;nofollow&quot;&gt;scikit-learn&lt;/a&gt;, which includes L1- and L2-penalized logistic regression;&lt;/li&gt;&#xA;&lt;li&gt;It works very well in practice, as shown in many application papers in image recognition, signal processing, biometrics, and finance.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="30" LastActivityDate="2010-07-25T02:37:06.947" CommunityOwnedDate="2010-07-25T02:37:06.947" />
  <row Id="607" PostTypeId="2" ParentId="6" CreationDate="2010-07-25T03:29:41.460" Score="14" ViewCount="" Body="&lt;p&gt;I think the answer to your question is simply in the affirmative. Take any issue of Statistical Science, JAMA, Annals of Statistics of the past 10 years and you'll find papers on boosting, SVM, and neural networks, although this area is less active now. Statisticians have appropriated the work of Valiant and Vapnick, but on the other side, computer scientists have absorbed the work of Donoho and Talagrand. I don't think there is much difference in scope and methods anymore. I have never bought Breiman's argument that CS people were only interested in minimizing loss using whatever works. That view was heavily influenced by his participation in Neural Networks conferences and his consulting work; but PAC, SVMs, Boosting have all solid foundations. And today, unlike in 2001, Statistics is more concerned with finite-sample properties, algorithms and massive datasets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I think that there are still three important differences that are not going away soon. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Methodological Statistics papers are still overwhelmingly formal and deductive, whereas Machine Learning researchers are more tolerant of new approaches even if they don't come with a proof attached;&lt;/li&gt;&#xA;&lt;li&gt;The ML community primarily shares new results and publications in conferences and related proceedings, whereas statisticians use journal papers. This slows down progress in Statistics and identification of star researchers. John Langford a &lt;a href=&quot;http://hunch.net/?p=318&quot; rel=&quot;nofollow&quot;&gt;nice post&lt;/a&gt; on the subject from a while back;&lt;/li&gt;&#xA;&lt;li&gt;Statistics still covers areas that are (for now) of little concern to ML, such as survey design, sampling, industrial Statistics etc.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="30" LastActivityDate="2010-07-25T03:29:41.460" CommentCount="1" />
  <row Id="608" PostTypeId="1" AcceptedAnswerId="1203" CreationDate="2010-07-25T05:08:20.333" Score="15" ViewCount="453" Body="&lt;p&gt;In a &lt;a href=&quot;http://stats.stackexchange.com/questions/577/is-there-any-reason-to-prefer-the-aic-or-bic-over-the-other&quot;&gt;question&lt;/a&gt; elsewhere on this site, several answers mentioned that the AIC is equivalent to  leave-one-out (LOO) cross-validation and that the BIC is equivalent to K-fold cross validation.  Is there a way to empirically demonstrate this in R such that the techniques involved in LOO and K-fold are made clear and demonstrated to be equivalent to the AIC and BIC values?  Well commented code would be helpful in this regard.  In addition, in demonstrating the BIC please use the lme4 package.  See below for a sample dataset...&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(lme4) #for the BIC function&#xA;&#xA;generate.data &amp;lt;- function(seed)&#xA;{&#xA;    set.seed(seed) #Set a seed so the results are consistent (I hope)&#xA;    a &amp;lt;- rnorm(60) #predictor&#xA;    b &amp;lt;- rnorm(60) #predictor&#xA;    c &amp;lt;- rnorm(60) #predictor&#xA;    y &amp;lt;- rnorm(60)*3.5+a+b #the outcome is really a function of predictor a and b but not predictor c&#xA;    data &amp;lt;- data.frame(y,a,b,c) &#xA;    return(data)    &#xA;}&#xA;&#xA;data &amp;lt;- generate.data(76)&#xA;good.model &amp;lt;- lm(y ~ a+b,data=data)&#xA;bad.model &amp;lt;- lm(y ~ a+b+c,data=data)&#xA;AIC(good.model)&#xA;BIC(logLik(good.model))&#xA;AIC(bad.model)&#xA;BIC(logLik(bad.model))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Per earlier comments, below I have provided a list of seeds from 1 to 10000 in which AIC and BIC disagree.  This was done by a simple search through the available seeds, but if someone could provide a way to generate data which would tend to produce divergent answers from these two information criteria it may be particularly informative.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;notable.seeds &amp;lt;- read.csv(&quot;http://student.ucr.edu/~rpier001/res.csv&quot;)$seed&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As an aside, I thought about ordering these seeds by the extent to which the AIC and BIC disagree which I've tried quantifying as the sum of the absolute differences of the AIC and BIC.  For example, &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;AICDiff &amp;lt;- AIC(bad.model) - AIC(good.model) &#xA;BICDiff &amp;lt;- BIC(logLik(bad.model)) - BIC(logLik(good.model))&#xA;disagreement &amp;lt;- sum(abs(c(AICDiff,BICDiff)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where my disagreement metric only reasonably applies when the observations are notable.  For example,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;are.diff &amp;lt;- sum(sign(c(AICDiff,BICDiff)))&#xA;notable &amp;lt;- ifelse(are.diff == 0 &amp;amp; AICDiff != 0,TRUE,FALSE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However in cases where AIC and BIC disagreed, the calculated disagreement value was always the same (and is a function of sample size).  Looking back at how AIC and BIC are calculated I can see why this might be the case computationally, but I'm not sure why it would be the case conceptually.  If someone could elucidate that issue as well, I'd appreciate it.&lt;/p&gt;&#xA;" OwnerUserId="196" LastEditorUserId="196" LastEditorDisplayName="" LastEditDate="2010-08-03T22:18:43.423" LastActivityDate="2010-08-03T22:58:09.447" Title="How can one empirically demonstrate in R which cross-validation methods the AIC and BIC are equivalent to?" Tags="&lt;r&gt;&lt;aic&gt;&lt;cross-validation&gt;&lt;bic&gt;" AnswerCount="1" CommentCount="8" FavoriteCount="9" />
  <row Id="609" PostTypeId="2" ParentId="411" CreationDate="2010-07-25T07:18:44.263" Score="5" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;As a summary&lt;/strong&gt;, my answer is : if you have an explicit expression or can figure out some how what your distance is measuring (what &quot;differences&quot; it gives weigth to), then you can say what it is better for. An other complementary way to analyse and compare such test is the minimax theory. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;At the end some test will be good for some alternatives and some for others. For a given set of alternatives it is sometime possible to show if your test has optimal property in the worst case: this is the minimax theory. &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Some details&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hence You can tell about the properties of two different test by regarding the set of alternative for which they are minimax (if such alternative exist) i.e. (using the word of Donoho and Jin) by comparing their &quot;optimal detection boudary&quot; &lt;a href=&quot;http://projecteuclid.org/DPubS?service=UI&amp;amp;version=1.0&amp;amp;verb=Display&amp;amp;handle=euclid.aos/1085408492&quot; rel=&quot;nofollow&quot;&gt;http://projecteuclid.org/DPubS?service=UI&amp;amp;version=1.0&amp;amp;verb=Display&amp;amp;handle=euclid.aos/1085408492&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let me go distance by distance:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;KS distance is obtained calculating supremum of difference between empirical cdf and cdf. Being a suppremum it will be highly sensitive to local alternatives (local change in the cdf) but not with global change (at least using L2 distance between cdf would be less local (Am I openning open door ?)). However, the most important thing is that is uses the cdf. This implies an asymetry: you give more importance to the changes in the tail of your distribution.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Wassertein metric  (what you meant by Kantorovitch Rubinstein ? )  &lt;a href=&quot;http://en.wikipedia.org/wiki/Wasserstein_metric&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Wasserstein_metric&lt;/a&gt; is ubiquitous and hence hard to compare. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;For the particular case of W2 it has been uses in &lt;a href=&quot;http://projecteuclid.org/DPubS?service=UI&amp;amp;version=1.0&amp;amp;verb=Display&amp;amp;handle=euclid.aos/1017938923&quot; rel=&quot;nofollow&quot;&gt;http://projecteuclid.org/DPubS?service=UI&amp;amp;version=1.0&amp;amp;verb=Display&amp;amp;handle=euclid.aos/1017938923&lt;/a&gt;  and it is related to the L2 distance to inverse of cdf. My understanding is that it gives even more weight to the tails but I think you should read the paper to know more about it. &lt;/li&gt;&#xA;&lt;li&gt;For the case of the L1 distance between density function it will highly depend on how you estimate your dentity function from the data... but otherwise it seems to be a &quot;balanced test&quot; not giving importance to tails. &lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;To recall and extend the comment I made which complete the answer: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know you did not meant to be exhaustive but you could add Anderson darling statistic (see &lt;a href=&quot;http://en.wikipedia.org/wiki/Anderson%E2%80%93Darling_test&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Anderson%E2%80%93Darling_test&lt;/a&gt;). This made me remind of a paper fromo Jager and Wellner (see &lt;a href=&quot;http://projecteuclid.org/DPubS?service=UI&amp;amp;version=1.0&amp;amp;verb=Display&amp;amp;handle=euclid.aos/1194461721&quot; rel=&quot;nofollow&quot;&gt;http://projecteuclid.org/DPubS?service=UI&amp;amp;version=1.0&amp;amp;verb=Display&amp;amp;handle=euclid.aos/1194461721&lt;/a&gt;) which extands/generalises Anderson darling statistic (and include in particular higher criticism of Tukey). Higher criticism was already shown to be minimax for a wide range of alternatives and the same is done by Jager and Wellner for their extention. I don't think that minimax property has been shown for Kolmogorov test. Anyway, understanding for which type of alternative your test is minimax helps you to know where is its strength, so you should read the paper above.. &lt;/p&gt;&#xA;" OwnerUserId="223" LastEditorUserId="223" LastEditorDisplayName="" LastEditDate="2010-08-05T05:35:45.267" LastActivityDate="2010-08-05T05:35:45.267" CommentCount="2" />
  <row Id="610" PostTypeId="2" ParentId="492" CreationDate="2010-07-25T09:13:59.517" Score="4" ViewCount="" Body="&lt;p&gt;It sounds dodgy to me as the trend estimate will be biased near the point where you splice on the false data. An alternative approach is a nonparametric regression smoother such as loess or splines.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-07-25T09:13:59.517" />
  <row Id="611" PostTypeId="2" ParentId="590" CreationDate="2010-07-25T12:36:23.680" Score="4" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Heuristic answer:&lt;/strong&gt; Without much mathematic you could say that a continuous variable has a density with respect to the Lebesgue measure, and a discrete random variable has a density with respect to the counting measure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Developped answer:&lt;/strong&gt; The concept of density is much wider than you may think. A density of a probability measure $P$ can be defined with respect to a measure $\lambda$ that dominates $P$ by the Radon Nikodym Theorem (see &lt;a href=&quot;http://en.wikipedia.org/wiki/Radon%E2%80%93Nikodym_theorem&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Radon%E2%80%93Nikodym_theorem&lt;/a&gt;). Here density should be understood as a density with respect to the counting measure defined on the mentionned countable set. I agree that it is not extremely rigorous not to mention the reference when talking about a density (but who mention density  wrt lesbesgue measure?), but it pose no problem while reading the paper in question so .... &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Additional Annex Notes&#xA; I have seen a certain number of machine learning notes (I won't do delation) where  the reference measure is not the counting measure and we see things such as $P(X=x|Y=y)$ with X being a continuous variable (with a density wrt Lebesgues) (to apply Bayes principle and derive the Bayes rule). I guess people want to be pedagogic and do not want to bother students with technical details ;) My conclusion would be that even great mathematician can commit abuse of wording or of notation (it is not the case in the paper you mention because the authors of a paper in AoP may have mathematical background), it is not a problem as along as it is understood by everyone...  &lt;/p&gt;&#xA;" OwnerUserId="223" LastEditorUserId="223" LastEditorDisplayName="" LastEditDate="2010-08-03T19:39:42.083" LastActivityDate="2010-08-03T19:39:42.083" CommentCount="2" />
  <row Id="612" PostTypeId="1" AcceptedAnswerId="849" CreationDate="2010-07-25T14:31:31.773" Score="6" ViewCount="431" Body="&lt;p&gt;I have tried to reproduce some research (using PCA) from SPSS in R. In my experience, &lt;a href=&quot;http://www.personality-project.org/R/html/principal.html&quot; rel=&quot;nofollow&quot;&gt;&lt;code&gt;principal()&lt;/code&gt;&lt;/a&gt; function from package &lt;code&gt;psych&lt;/code&gt; was the only function that came close (or if my memory serves me right, dead on) to match the output. To match the same results as in SPSS, I had to use parameter &lt;code&gt;principal(..., rotate = &quot;varimax&quot;)&lt;/code&gt;. I have seen papers talk about how they did PCA, but based on the output of SPSS and use of rotation, it sounds more like Factory analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Question: Is PCA, even after rotation (using &lt;code&gt;varimax&lt;/code&gt;), still PCA? I was under the impression that this might be in fact Factor analysis... In case it's not, what details am I missing?&lt;/p&gt;&#xA;" OwnerUserId="144" LastEditorUserId="190" LastEditorDisplayName="" LastEditDate="2010-07-26T17:52:30.217" LastActivityDate="2010-07-27T19:27:45.063" Title="is psych::principal function still PCA when using rotation?" Tags="&lt;r&gt;&lt;pca&gt;&lt;spss&gt;&lt;factor-analysis&gt;" AnswerCount="3" CommentCount="2" FavoriteCount="4" />
  <row Id="613" PostTypeId="2" ParentId="130" CreationDate="2010-07-25T14:33:25.313" Score="3" ViewCount="" Body="&lt;p&gt;I have been a heavy R user for the past 6-7 years. As a language, it has several design limitations. Yet, for work in econometrics and in data analysis, I still wholeheartedly recommend it. It has a large number of packages that would be relevant to you for econometrics, time series, consumer choice modeling etc. and of course excellent visualization, good algebra and numerical libraries etc. I would not worry too much about data size limitations. Although R was not designed for &quot;big data&quot; (unlike, say, SAS) there are ways around it. The availability of packages is what makes the difference, really.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've only read Clojure's language specs, and it's beautiful and clean. It addresses in a natural way issues of parallelization and scale.  And if you have some basic java or OOP knowledge, you can benefit from the large number of high-quality java libraries.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The issue I have with Clojure is that is a recent one-man (R.Hickey) operation, therefore 1) very risky 2) very immature 3) with niche adoption. Great for enthusiasts, early adopters, CS/ML people who want to try new things. For a user who sees a language as a means to an end and who needs very robust code that can be shared code with others, established languages seem a safer choice. Just know who you are.&lt;/p&gt;&#xA;" OwnerUserId="30" LastActivityDate="2010-07-25T14:33:25.313" CommentCount="1" />
  <row Id="614" PostTypeId="1" CreationDate="2010-07-25T14:53:50.473" Score="9" ViewCount="338" Body="&lt;p&gt;There have been a few questions about statistical &lt;a href=&quot;http://stats.stackexchange.com/questions/tagged/textbook&quot; rel=&quot;nofollow&quot;&gt;textbooks&lt;/a&gt;, such as the question &lt;a href=&quot;http://stats.stackexchange.com/questions/170/free-statistical-textbooks&quot; rel=&quot;nofollow&quot;&gt;Free statistical textbooks&lt;/a&gt;. However, I am looking for textbooks that are Open Source, for example, having an &lt;a href=&quot;http://creativecommons.org/&quot; rel=&quot;nofollow&quot;&gt;Creative Commons&lt;/a&gt; license. The reason is that in course material in other domains, you still want to include some text about basic statistics. In this case, it would be interesting to reuse existing material, instead of rewriting that material.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, what Open Source textbooks on statistics (and perhaps machine learning) are available?&lt;/p&gt;&#xA;" OwnerUserId="107" LastEditorUserId="107" LastEditorDisplayName="" LastEditDate="2010-07-26T15:31:39.667" LastActivityDate="2010-07-29T18:59:01.417" CommunityOwnedDate="2010-07-26T15:31:39.667" Title="Open Source statistical textbooks?" Tags="&lt;books&gt;&lt;open-source&gt;" AnswerCount="9" CommentCount="6" FavoriteCount="7" />
  <row Id="615" PostTypeId="2" ParentId="612" CreationDate="2010-07-25T14:56:32.433" Score="3" ViewCount="" Body="&lt;p&gt;Thanks to the chaos in definitions of both they are effectively a synonyms. Don't believe words and look deep into the docks to find the equations.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-07-25T14:56:32.433" CommentCount="3" />
  <row Id="616" PostTypeId="2" ParentId="614" CreationDate="2010-07-25T15:25:13.873" Score="5" ViewCount="" Body="&lt;p&gt;The &lt;a href=&quot;http://en.wikibooks.org/wiki/Statistics&quot; rel=&quot;nofollow&quot;&gt;&quot;Statistics&quot;&lt;/a&gt; book on wikibooks&lt;/p&gt;&#xA;" OwnerUserId="5" LastEditorUserId="5" LastEditorDisplayName="" LastEditDate="2010-07-25T15:32:14.903" LastActivityDate="2010-07-25T15:32:14.903" CommunityOwnedDate="2010-07-26T15:37:25.283" />
  <row Id="617" PostTypeId="2" ParentId="614" CreationDate="2010-07-25T15:32:59.290" Score="6" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://www.opentextbook.org/2009/04/03/multivariate-statistics-with-r/&quot; rel=&quot;nofollow&quot;&gt;Multivariate statistics with R&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-07-25T15:32:59.290" CommunityOwnedDate="2010-07-26T15:37:10.527" />
  <row Id="618" PostTypeId="2" ParentId="36" CreationDate="2010-07-25T18:02:06.573" Score="4" ViewCount="" Body="&lt;p&gt;I've always liked this one:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/dxfWK.jpg&quot; alt=&quot;lemons vs deaths&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;source: &lt;a href=&quot;http://pubs.acs.org/doi/abs/10.1021/ci700332k&quot; rel=&quot;nofollow&quot;&gt;http://pubs.acs.org/doi/abs/10.1021/ci700332k&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="54" LastEditorUserId="54" LastEditorDisplayName="" LastEditDate="2010-07-25T18:10:35.387" LastActivityDate="2010-07-25T18:10:35.387" CommunityOwnedDate="2010-08-16T13:01:42.613" CommentCount="3" />
  <row Id="619" PostTypeId="2" ParentId="524" CreationDate="2010-07-25T20:08:35.907" Score="4" ViewCount="" Body="&lt;p&gt;I have a depressing and not-very-specific anecdote to share here. I spent some time as a co-worker of a statistical MT researcher. If you want to see a really big, complex, model, look no further.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;He was putting me through NLP bootcamp for his own amusement. I am, in general, the sort of programmer who lives and dies by the unit test and the debugger. As a young person at Symbolics, I was struck by the aphorism, 'programming is debugging an empty editor buffer.' (Sort of like training a perceptron model.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, I asked him, 'how do you test and debug this stuff.' He answer was, &quot;You get it right the first time. You think it through (in his case, often on paper) very carefully, and you code it very carefully. Because when you get it wrong, the chances of isolating the problem are very slim.&quot;&lt;/p&gt;&#xA;" OwnerUserId="240" LastActivityDate="2010-07-25T20:08:35.907" />
  <row Id="620" PostTypeId="2" ParentId="614" CreationDate="2010-07-25T22:04:16.257" Score="2" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikibooks.org/wiki/R_Programming&quot; rel=&quot;nofollow&quot;&gt;R programming wiki book&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="253" LastActivityDate="2010-07-25T22:04:16.257" CommunityOwnedDate="2010-07-26T15:52:05.660" CommentCount="1" />
  <row Id="621" PostTypeId="2" ParentId="614" CreationDate="2010-07-25T23:56:43.840" Score="3" ViewCount="" Body="&lt;p&gt;Some googling found &lt;a href=&quot;http://collegeopentextbooks.org/statisticsprobbooks.html&quot; rel=&quot;nofollow&quot;&gt;http://collegeopentextbooks.org/statisticsprobbooks.html&lt;/a&gt; . Still, be aware that most of CC-ed material is share-aliked (so you must also publish your work on CC) or at least attributed (so you must add info that certain part was copied and from whom). The same works with GFDL (both SA &amp;amp; A), it is even worse since in principle you should print it along with the document.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-07-25T23:56:43.840" CommunityOwnedDate="2010-07-26T15:48:09.047" CommentCount="1" />
  <row Id="622" PostTypeId="1" AcceptedAnswerId="628" CreationDate="2010-07-26T09:12:21.093" Score="10" ViewCount="611" Body="&lt;p&gt;I see these terms being used and I keep getting them mixed up. Is there a simple explanation of the differences between them?&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-07-31T00:34:24.843" Title="What is the difference between a partial likelihood, profile likelihood and marginal likelihood?" Tags="&lt;estimation&gt;&lt;maximum-likelihood&gt;" AnswerCount="2" FavoriteCount="3" />
  <row Id="623" PostTypeId="2" ParentId="175" CreationDate="2010-07-26T09:46:48.410" Score="2" ViewCount="" Body="&lt;p&gt;Garbage in, garbage out....&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Implicit in getting the full benefit of linear regression is that the noise follows a normal distribution.  Ideally you have mostly data and a little noise.... not mostly noise and a little data.  You can test for normality of residuals after the linear fit by looking at the residuals.  You can also filter input data before the linear fit for obvious, glaring errors. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are some types of noise in garbage input data that do not typically fit a normal distribution:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Digits missing or added with hand-entered data (off by a factor of 10 or more)&lt;/li&gt;&#xA;&lt;li&gt;Wrong or incorrectly converted units (grams vs kilos vs pounds; meters, feet, miles, km), possibly from merging multiple data sets (Note: The Mars Orbiter was thought to be lost in this way, so even NASA rocket scientists can make this mistake)&lt;/li&gt;&#xA;&lt;li&gt;Use of codes like 0, -1, -99999 or 99999 to mean something non-numeric like &quot;not applicable&quot; or &quot;column unavailable&quot; and just dumping this into a linear model along with valid data&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Writing a spec for what is &quot;valid data&quot; for each column can help you tag invalid data. For instance, a person's height in cm should be in a range, say, 100-300cm.  If you find 1.8 for height thats a typo, and while you can assume it was 1.8m and alter it to 180 -- I'd say it is usually safer to throw it out and best to document as much of the filtering as possible.  &lt;/p&gt;&#xA;" OwnerUserId="87" LastActivityDate="2010-07-26T09:46:48.410" />
  <row Id="624" PostTypeId="1" CreationDate="2010-07-26T11:53:11.027" Score="2" ViewCount="130" Body="&lt;p&gt;In engineering, we usually have Handbooks that pretty much dictate the state of the practice. These books are usually devoid of theory and focus on the applied methodology. Is there a forecasting Handbook out there? that solely focuses on the technique and not the background?&lt;/p&gt;&#xA;" OwnerUserId="59" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-30T21:25:14.480" LastActivityDate="2010-09-30T21:25:14.480" CommunityOwnedDate="2010-07-26T15:00:23.580" Title="Forecasting handbooks" Tags="&lt;forecasting&gt;" AnswerCount="1" CommentCount="0" FavoriteCount="2" />
  <row Id="625" PostTypeId="2" ParentId="624" CreationDate="2010-07-26T12:01:58.983" Score="4" ViewCount="" Body="&lt;p&gt;There are two that I know of:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0444513957&quot; rel=&quot;nofollow&quot;&gt;Handbook of economic forecasting&lt;/a&gt;. Relatively theoretical. Not for undergraduates. A narrow look at forecasting --- specifically about economic forecasting.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0792379306&quot; rel=&quot;nofollow&quot;&gt;Principles of forecasting&lt;/a&gt;. Simpler, broader. Widely used by forecasting practitioners. Often reflects the idiosyncratic opinions of the editor which are presented as established facts.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Alternatively, you could use an intro textbook. &lt;a href=&quot;http://robjhyndman.com/forecasting&quot; rel=&quot;nofollow&quot;&gt;My own textbook&lt;/a&gt; is often used as a sort of handbook by forecasting practitioners working in a business environment.&lt;/p&gt;&#xA;" OwnerUserId="159" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-07-26T12:20:12.410" LastActivityDate="2010-07-26T12:20:12.410" CommentCount="2" />
  <row Id="626" PostTypeId="2" ParentId="125" CreationDate="2010-07-26T12:38:40.610" Score="3" ViewCount="" Body="&lt;p&gt;&quot;Bayesian Core: A Practical Approach to Computational Bayesian Statistics&quot; by Marin and Robert, &lt;em&gt;Springer-Verlag&lt;/em&gt; (2007)&lt;/p&gt;&#xA;" OwnerUserId="30" LastActivityDate="2010-07-26T12:38:40.610" CommunityOwnedDate="2010-07-26T12:38:40.610" />
  <row Id="627" PostTypeId="2" ParentId="170" CreationDate="2010-07-26T12:53:22.357" Score="5" ViewCount="" Body="&lt;p&gt;I really like  &lt;a href=&quot;http://www.tufts.edu/~gdallal/LHSP.HTM&quot; rel=&quot;nofollow&quot;&gt;The Little Handbook of Statistical Practice&lt;/a&gt; by Gerard E. Dallal&lt;/p&gt;&#xA;" OwnerUserId="236" LastActivityDate="2010-07-26T12:53:22.357" CommunityOwnedDate="2010-07-26T12:53:22.357" />
  <row Id="628" PostTypeId="2" ParentId="622" CreationDate="2010-07-26T14:40:27.763" Score="8" ViewCount="" Body="&lt;p&gt;The likelihood function usually depends on many parameters. Depending on the application, we are usually interested in only a subset of these parameters. For example, in linear regression, interest typically lies in the slope coefficients and not on the error variance. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Denote the parameters we are interested in as &amp;beta; and the parameters that are not of primary interest as &amp;theta;. The standard way to approach the estimation problem is to maximize the likelihood function so that we obtain estimates of &amp;beta; and &amp;theta;. However, since the primary interest lies in &amp;beta; partial, profile and marginal likelihood offer alternative ways to estimate &amp;beta; without estimating &amp;theta;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to see the difference denote the standard likelihood by L(&amp;beta;, &amp;theta;|data). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Maximum Likelihood&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Find &amp;beta; and &amp;theta; that maximizes L(&amp;beta;, &amp;theta;|data).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Partial Likelihood&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we can write the likelihood function as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;L(&amp;beta;, &amp;theta;|data) = L1(&amp;beta;|data) L2(&amp;theta;|data)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then we simply maximize L1(&amp;beta;|data).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Profile Likelihood&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we can express &amp;theta; as a function of &amp;beta; then we replace &amp;theta; with the corresponding function. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say, &amp;theta; = g(&amp;beta;). Then, we maximize:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;L(&amp;beta;, g(&amp;beta;)|data)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Marginal Likelihood&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We integrate out &amp;theta; from the likelihood equation by exploiting the fact that we can identify the probability distribution of &amp;theta; conditional on &amp;beta;.&lt;/p&gt;&#xA;" LastActivityDate="2010-07-31T00:34:24.843" CommentCount="3" />
  <row Id="631" PostTypeId="1" AcceptedAnswerId="647" CreationDate="2010-07-26T16:10:45.817" Score="3" ViewCount="520" Body="&lt;p&gt;What is an estimator of standard deviation of standard deviation if normality of data can be assumed?&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-08-02T03:10:44.470" Title="Standard deviation of standard deviation" Tags="&lt;standard-deviation&gt;" AnswerCount="3" FavoriteCount="2" />
  <row Id="632" PostTypeId="2" ParentId="631" CreationDate="2010-07-26T16:23:25.073" Score="2" ViewCount="" Body="&lt;p&gt;Assume you observe $X1,\dots,Xn$ iid from a normal with mean zero and variance $\sigma^2$. The (empirical) standard deviation is the square root of the estimation $\hat{\sigma}^2$ of $\sigma^2$ (unbiased or not that is not the question). As an estimator (obtained with $X1,\dots,Xn$), $\hat{\sigma}$ has a variance that can be calculated theoretically. Maybe what you call the standard deviation of standard deviation is actually the square root of the variance of the  standard deviation, i.e. $\sqrt{E[(\sigma-\hat{\sigma})^2]}$?  It is not an estimator, it is a theoretical quantity (something like $\sigma/\sqrt{n}$ to be confirmed) that can be calculated explicitely !&lt;/p&gt;&#xA;" OwnerUserId="223" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-08-02T03:10:44.470" LastActivityDate="2010-08-02T03:10:44.470" CommentCount="4" />
  <row Id="633" PostTypeId="1" AcceptedAnswerId="634" CreationDate="2010-07-26T17:29:19.477" Score="4" ViewCount="168" Body="&lt;p&gt;I have a data set of about 3,000 field observations. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data collected is divided into 20 variables (real numbers), 30 boolean variables, and 10 or so look up variables and one &quot;answer&quot; variable&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We have about 20,000 objects in the field, and i'm trying to produce an &quot;answer&quot; for the 20,000 objects based on the 3,000 observations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are some of the available methods that incorporate booleans and look up tables?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;any suggestions on how i should proceed?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;the answer variable is a boolean as well&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT 2&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;a sample of the variable data:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Age of specimen&lt;/li&gt;&#xA;&lt;li&gt;length, area, volume&lt;/li&gt;&#xA;&lt;li&gt;time since last inspection&lt;/li&gt;&#xA;&lt;li&gt;height&lt;/li&gt;&#xA;&lt;li&gt;design life&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Lookup table&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;material type &lt;/li&gt;&#xA;&lt;li&gt;coating type&lt;/li&gt;&#xA;&lt;li&gt;design standard&lt;/li&gt;&#xA;&lt;li&gt;design effectiveness&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;a sample of the boolean&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;is it inspected?&lt;/li&gt;&#xA;&lt;li&gt;is it in bad shape&lt;/li&gt;&#xA;&lt;li&gt;does it need repairs soon&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;the answer variable which is my f(x) is:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;is it useable&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="59" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-17T19:07:16.110" LastActivityDate="2010-11-02T18:47:07.033" Title="Incorporating boolean data into analysis" Tags="&lt;modeling&gt;&lt;categorical-data&gt;&lt;model-selection&gt;&lt;binary-data&gt;" AnswerCount="4" FavoriteCount="1" />
  <row Id="634" PostTypeId="2" ParentId="633" CreationDate="2010-07-26T17:46:45.127" Score="7" ViewCount="" Body="&lt;p&gt;You are decribing &quot;&lt;a href=&quot;http://www.oswego.edu/~srp/stats/variable_types.htm&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;categorical&lt;/strong&gt; variables&lt;/a&gt;&quot; (represented in R a factors). These can be incorporated into almost any statistical model by being assigned levels.  You would need to give more detail about your particular problem in order to be advised on a particular method.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Edit&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the response variable has two possible outcomes, you might consider &lt;a href=&quot;http://en.m.wikipedia.org/wiki/Binomial_regression&quot; rel=&quot;nofollow&quot;&gt;binomial&lt;/a&gt; or &lt;a href=&quot;http://en.wikipedia.org/wiki/Logistic_regression&quot; rel=&quot;nofollow&quot;&gt;logistic&lt;/a&gt; regression. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: If you're not familiar with the different kinds of variables in statistics, I suggest reading the first few chapters of Andrew Gelman's &quot;&lt;a href=&quot;http://www.stat.columbia.edu/~gelman/arm/&quot; rel=&quot;nofollow&quot;&gt;Data Analysis Using Regression and Multilevel/Hierarchical Models&lt;/a&gt;&quot; which covers this in a very understandable manner.&lt;/p&gt;&#xA;" OwnerUserId="5" LastEditorUserId="5" LastEditorDisplayName="" LastEditDate="2010-07-26T18:25:45.280" LastActivityDate="2010-07-26T18:25:45.280" />
  <row Id="635" PostTypeId="2" ParentId="480" CreationDate="2010-07-26T17:50:21.780" Score="4" ViewCount="" Body="&lt;p&gt;Robin,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;implementations of RF differ slightly. I know that Salford Systems' &lt;a href=&quot;http://salford-systems.com/products/randomforests/overview.html&quot; rel=&quot;nofollow&quot;&gt;proprietary implementation&lt;/a&gt; is supposed to be better than the &lt;a href=&quot;http://cran.r-project.org/web/packages/randomForest/&quot; rel=&quot;nofollow&quot;&gt;vanilla one&lt;/a&gt; in R. A description of the algorithm is in &lt;a href=&quot;http://www-stat.stanford.edu/~tibs/ElemStatLearn/download.html&quot; rel=&quot;nofollow&quot;&gt;ESL by Friedman-Hastie-Tibshirani, 2nd ed, 3rd printing&lt;/a&gt;. An entire chapter (15th) is devoted to RF, and I find it actually clearer than the original paper. The tree construction algorithm is detailed on p.588; no need for me to reproduce it here, since the book is available online. I hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="30" LastActivityDate="2010-07-26T17:50:21.780" CommentCount="1" />
  <row Id="636" PostTypeId="2" ParentId="633" CreationDate="2010-07-26T17:50:37.240" Score="1" ViewCount="" Body="&lt;p&gt;Try Random Forest; from my experience it may perform well on such kind of data, and gives you a some additional interesting information, like variable importance and object similarity measure. &lt;/p&gt;&#xA;" OwnerUserId="88" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-07-26T18:21:26.337" LastActivityDate="2010-07-26T18:21:26.337" CommentCount="4" />
  <row Id="637" PostTypeId="2" ParentId="633" CreationDate="2010-07-26T18:22:33.917" Score="4" ViewCount="" Body="&lt;p&gt;It sounds like you are trying to predict your boolean response, yes?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is called classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Logistic Regression is the obvious choice here, but there are other methods too. You can't do traditional regression, because the response is not a real number.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The lookup variables are called nominals, and can be dealt with in regression by using &quot;dummy&quot; variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if your lookup variable is type=[steel, aluminum, plastic] (N=3), then your dummy variables would look like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;IsSteel = [1,0]&#xA;IsAlum = [1,0]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There would only be two (N-1) dummy variables, as IsSteel=0 AND IsAlum=0 represents &quot;IsPlastic&quot;=1&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But any good stats program should handle this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you need a book, I recommend Multivariate Data Analysis by Hair.&lt;/p&gt;&#xA;" OwnerUserId="74" LastActivityDate="2010-07-26T18:22:33.917" />
  <row Id="638" PostTypeId="1" AcceptedAnswerId="644" CreationDate="2010-07-26T18:24:35.737" Score="5" ViewCount="180" Body="&lt;p&gt;Provided a sample size S that I plan on using to forecast data. What are some of the ways to subdivide the data so that I use some of it to establish a model, and the remainder data to validate the model?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know there is no black and white answer to this but it would be interesting to know  some &quot;rules of thumb&quot; or usually used ratios. I know back at university, one of our professors used to say model on 60% and validate on 40%.&lt;/p&gt;&#xA;" OwnerUserId="59" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-16T06:56:07.427" LastActivityDate="2010-09-16T06:56:07.427" Title="How to calculate the ratio of &quot;data used in analysis&quot; and &quot;saved data for validation&quot; from a sample set?" Tags="&lt;machine-learning&gt;&lt;best-practices&gt;" AnswerCount="4" FavoriteCount="1" />
  <row Id="640" PostTypeId="2" ParentId="631" CreationDate="2010-07-26T18:40:54.907" Score="6" ViewCount="" Body="&lt;p&gt;I suppose that you are looking for the &lt;a href=&quot;http://en.wikipedia.org/wiki/Variance#Distribution_of_the_sample_variance&quot; rel=&quot;nofollow&quot;&gt;distribution of the sample variance&lt;/a&gt;.&lt;/p&gt;&#xA;" LastActivityDate="2010-07-26T18:40:54.907" />
  <row Id="641" PostTypeId="1" CreationDate="2010-07-26T19:20:53.477" Score="4" ViewCount="154" Body="&lt;p&gt;Sites like eMarketer offer general survey results about internet usage.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Who else has a big set of survey results, or regularly releases them?&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Preferably marketing research focused.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="74" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-16T12:42:31.387" LastActivityDate="2010-09-16T12:42:31.387" CommunityOwnedDate="2010-07-26T19:41:11.150" Title="Where is a good place to find survey results?" Tags="&lt;dataset&gt;&lt;survey&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="2" />
  <row Id="642" PostTypeId="2" ParentId="641" CreationDate="2010-07-26T19:23:19.133" Score="1" ViewCount="" Body="&lt;p&gt;government websites usually .... I use the &lt;a href=&quot;http://www.rita.dot.gov/&quot; rel=&quot;nofollow&quot;&gt;RITA&lt;/a&gt; a lot&lt;/p&gt;&#xA;" OwnerUserId="59" LastActivityDate="2010-07-26T19:23:19.133" />
  <row Id="643" PostTypeId="1" CreationDate="2010-07-26T19:26:37.037" Score="12" ViewCount="387" Body="&lt;p&gt;My father is a math enthusiast, but not interested in statistics much. It would be neat to &lt;em&gt;try&lt;/em&gt; to illustrate some of the wonderful bits of statistics, and the CLT is a prime candidate. How would you convey the mathematical beauty and impact of the central limit theorem to a non-statistician? &lt;/p&gt;&#xA;" OwnerUserId="7" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-10-19T06:42:19.517" LastActivityDate="2011-06-02T11:08:55.880" CommunityOwnedDate="2011-06-02T11:09:11.620" Title="How do you convey the beauty of the Central Limit Theorem to a non-statistician?" Tags="&lt;theory&gt;&lt;central-limit-theorem&gt;" AnswerCount="5" CommentCount="1" FavoriteCount="2" />
  <row Id="644" PostTypeId="2" ParentId="638" CreationDate="2010-07-26T19:27:49.840" Score="4" ViewCount="" Body="&lt;p&gt;Well as you said there is no black and white answer. I generally don't divide the data in 2 parts but use methods like k-fold cross validation instead. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In k-fold cross validation you divide your data randomly into k parts and fit your model on k-1 parts and test the errors on the left out part. You repeat the process k times leaving each part out of fitting one by one. You can take the mean error from each of the k iterations as an indication of the model error. This works really well if you want to compare the predictive power of different models. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One extreme form of k-fold cross validation is the generalized cross validation where you just leave out one data point for testing and fit the model to all the remaining points. Then repeat the process n times leaving out each data point one by one. I generally prefer k-fold cross validation over the generalized cross validation ... just a personal choice&lt;/p&gt;&#xA;" OwnerUserId="288" LastActivityDate="2010-07-26T19:27:49.840" CommentCount="1" />
  <row Id="645" PostTypeId="1" AcceptedAnswerId="680" CreationDate="2010-07-26T19:28:53.083" Score="5" ViewCount="563" Body="&lt;p&gt;Having just recently started teaching myself Machine Learning and Data Analysis I'm finding myself hitting a brick wall on the need for creating and querying large sets of data. I would like to take data I've been aggregating in my professional and personal life and analyze it but I'm uncertain of the best way to do the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) How should I be storing this data? Excel? SQL? ??&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) What is a good way for a beginner to begin trying to analyze this data? I am a professional computer programmer so the complexity is not in writing programs but more or less specific to the domain of data analysis. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: Apologies for my vagueness, when you first start learning about something it's hard to know what you don't know, ya know? ;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having said that, my aim is to apply this to two main topics:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Software team metrics (think Agile velocity, quantifying risk, likelihood of a successfully completed iteration given x number of story points)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Machine learning (ex. system exceptions have occurred in a given set of modules what is the likelihood that a module will throw an exception in the field, how much will that cost, what can the data tell me about key modules to improve that will get me the best bang for my buck, predict what portion of the system the user will want to use next in order to start loading data, etc)&lt;/p&gt;&#xA;" OwnerUserId="274" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-16T06:50:58.507" LastActivityDate="2011-01-25T22:04:35.493" Title="Best Way to Aggregate and Analyze Data" Tags="&lt;best-practices&gt;" AnswerCount="4" CommentCount="1" FavoriteCount="3" />
  <row Id="646" PostTypeId="2" ParentId="641" CreationDate="2010-07-26T19:31:36.057" Score="9" ViewCount="" Body="&lt;p&gt;The best place to find survey data related to the social sciences is the ICPSR data clearinghouse: &lt;a href=&quot;http://www.icpsr.umich.edu/icpsrweb/ICPSR/access/index.jsp&quot; rel=&quot;nofollow&quot;&gt;http://www.icpsr.umich.edu/icpsrweb/ICPSR/access/index.jsp&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, the 'survey' tag on Infochimps has many interesting and free data sets: &lt;a href=&quot;http://infochimps.org/tags/survey&quot; rel=&quot;nofollow&quot;&gt;http://infochimps.org/tags/survey&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="302" LastActivityDate="2010-07-26T19:31:36.057" CommentCount="1" />
  <row Id="647" PostTypeId="2" ParentId="631" CreationDate="2010-07-26T19:34:07.533" Score="0" ViewCount="" Body="&lt;p&gt;I think both the concept and the terminology of &quot;SD of SD&quot; is too slippery to tackle. But it is easier to think about the confidence interval of a SD. You compute the SD from a sample of data and want to compute a confidence interval that is 95% (or some other confidence level) likely to contain the true SD of the population from which the data were sampled.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.graphpad.com/faq/viewfaq.cfm?faq=1381&quot; rel=&quot;nofollow&quot; title=&quot;Here&quot;&gt;Detailed explanation&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.graphpad.com/quickcalcs/CISD1.cfm&quot; rel=&quot;nofollow&quot;&gt;Free web calculator&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="25" LastActivityDate="2010-07-26T19:34:07.533" CommentCount="1" />
  <row Id="648" PostTypeId="2" ParentId="643" CreationDate="2010-07-26T19:34:09.857" Score="7" ViewCount="" Body="&lt;p&gt;To fully appreciate the CLT, it should be seen.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hence the notion of the &lt;a href=&quot;http://en.wikipedia.org/wiki/Galton_board&quot; rel=&quot;nofollow&quot;&gt;bean machine&lt;/a&gt; and plenty of youtube &lt;a href=&quot;http://www.youtube.com/results?search_query=galton+board&amp;amp;aq=0&quot; rel=&quot;nofollow&quot;&gt;videos&lt;/a&gt; for illustration.&lt;/p&gt;&#xA;" OwnerUserId="68" LastActivityDate="2010-07-26T19:34:09.857" CommunityOwnedDate="2011-06-02T11:09:11.620" CommentCount="4" />
  <row Id="650" PostTypeId="2" ParentId="645" CreationDate="2010-07-26T19:34:52.007" Score="3" ViewCount="" Body="&lt;p&gt;Your question is so broad that the answer is: it depends. Still, to give some more useful answer I'll indicate what I think are common in Research.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Storing of data is very often done in text files. When doing statistical analyses you mostly work with a collection of one type of vectors. This can be seen as a table and written in csv format. The reason thins are often stored in plain-text, is because simply every tool can read them and it is easy to transform them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;About analyzing, this is a bit harder to be specific. If it is 2 dimensional, make a scatterplot. If it is high-dimensional, do PCA and see where the first principal components exist of to discover important variables. If you have time data, plot it. This is all so general that to be useful you have to really indicate better what your data is.&lt;/p&gt;&#xA;" OwnerUserId="190" LastActivityDate="2010-07-26T19:34:52.007" CommentCount="3" />
  <row Id="651" PostTypeId="2" ParentId="223" CreationDate="2010-07-26T19:39:05.273" Score="1" ViewCount="" Body="&lt;p&gt;My book, &lt;a href=&quot;http://www.intuitivebiostatistics.com&quot; rel=&quot;nofollow&quot;&gt;&lt;em&gt;Intuitive Biostatistics&lt;/em&gt;&lt;/a&gt;, is written partly from a medical point of view. It focusses on the practical parts of interpreting statistical results, with almost no math.&lt;/p&gt;&#xA;" OwnerUserId="25" LastActivityDate="2010-07-26T19:39:05.273" CommentCount="1" />
  <row Id="652" PostTypeId="1" AcceptedAnswerId="682" CreationDate="2010-07-26T19:39:49.377" Score="8" ViewCount="1180" Body="&lt;p&gt;I bought this book:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://rcm.amazon.com/e/cm?lt1=_blank&amp;amp;bc1=000000&amp;amp;IS2=1&amp;amp;bg1=FFFFFF&amp;amp;fc1=000000&amp;amp;lc1=0000FF&amp;amp;t=justibozon-20&amp;amp;o=1&amp;amp;p=8&amp;amp;l=as1&amp;amp;m=amazon&amp;amp;f=ifr&amp;amp;md=10FE9736YVPPT7A0FBG2&amp;amp;asins=0470539399&quot; rel=&quot;nofollow&quot;&gt;How to Measure Anything: Finding the Value of Intangibles in Business&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;and &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://rcm.amazon.com/e/cm?lt1=_blank&amp;amp;bc1=000000&amp;amp;IS2=1&amp;amp;bg1=FFFFFF&amp;amp;fc1=000000&amp;amp;lc1=0000FF&amp;amp;t=justibozon-20&amp;amp;o=1&amp;amp;p=8&amp;amp;l=as1&amp;amp;m=amazon&amp;amp;f=ifr&amp;amp;md=10FE9736YVPPT7A0FBG2&amp;amp;asins=0596153937&quot; rel=&quot;nofollow&quot;&gt;Head First Data Analysis: A Learner's Guide to Big Numbers, Statistics, and Good Decisions&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What other books would you recommend?&lt;/p&gt;&#xA;" OwnerUserId="274" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-10T18:26:46.800" LastActivityDate="2011-03-16T22:48:04.127" CommunityOwnedDate="2010-07-26T19:53:10.810" Title="Best books for an introduction to statistical data analysis?" Tags="&lt;machine-learning&gt;&lt;bayesian&gt;&lt;books&gt;" AnswerCount="12" CommentCount="12" FavoriteCount="11" />
  <row Id="653" PostTypeId="2" ParentId="638" CreationDate="2010-07-26T19:42:50.803" Score="3" ViewCount="" Body="&lt;p&gt;It really depends on the amount of data you have, the specific cost of methods and how exactly you want your result to be.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some examples:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have little data, you probably want to use cross-validation (k-fold, leave-one-out, etc.) Your model will probably not take much resources to train and test anyway. It are good ways to get the most out of your data&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You have a lot of data: you probably want to take a reasonably large test-set, ensuring that there will be little possibility that some strange samples will give to much variance to your results. How much data you should take? It depends completely on your data and model. In speech recognition for example, if you would take too much data (let's say 3000 sentences), your experiments would take days, as a realtime factor of 7-10 is common. If you would take too little, it is too much dependent on the speakers that you are choosing (which are not allowed in the training set).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Remember also, in a lot of cases it is good to have a validation/development set too!&lt;/p&gt;&#xA;" OwnerUserId="190" LastActivityDate="2010-07-26T19:42:50.803" />
  <row Id="654" PostTypeId="2" ParentId="638" CreationDate="2010-07-26T19:44:44.183" Score="4" ViewCount="" Body="&lt;p&gt;1:10 test:train ratio is popular because it looks round, 1:9 is popular because of 10-fold CV, 1:2 is popular because it is also round and reassembles bootstrap. Sometimes one gets a test from some data-specific criteria, for instance last year for testing, years before for training. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The general rule is such: the train must be large enough to so the accuracy won't drop significantly, and the test must be large enough to silence random fluctuations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Still I prefer CV, since it gives you also a distribution of error.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-07-26T19:44:44.183" />
  <row Id="655" PostTypeId="2" ParentId="652" CreationDate="2010-07-26T19:49:53.953" Score="1" ViewCount="" Body="&lt;p&gt;You might find useful this one: &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0387848576&quot; rel=&quot;nofollow&quot;&gt;The Elements of Statistical Learning: Data Mining, Inference, and Prediction&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;UPDATE #1:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This book might be useful as well: &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0596510497&quot; rel=&quot;nofollow&quot;&gt;O'Reilly: Statistics in a Nutshel&lt;/a&gt;l&lt;/p&gt;&#xA;" OwnerUserId="315" LastEditorUserId="315" LastEditorDisplayName="" LastEditDate="2010-08-03T09:09:40.580" LastActivityDate="2010-08-03T09:09:40.580" CommentCount="1" />
  <row Id="656" PostTypeId="2" ParentId="423" CreationDate="2010-07-26T19:51:45.370" Score="21" ViewCount="" Body="&lt;p&gt;&lt;img src=&quot;http://bp1.blogger.com/_x7QjiZypFj0/Rp9dcGTsBKI/AAAAAAAAAA0/VWwfWDv6nzM/s400/Outlier.jpg&quot; alt=&quot;Image at bp1.blogger.com.&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="25" LastEditorUserId="509" LastEditorDisplayName="" LastEditDate="2010-08-11T08:50:54.893" LastActivityDate="2010-08-11T08:50:54.893" CommunityOwnedDate="2010-07-26T19:51:45.370" CommentCount="1" />
  <row Id="657" PostTypeId="2" ParentId="138" CreationDate="2010-07-26T19:53:45.660" Score="2" ViewCount="" Body="&lt;p&gt;I liked these lectures: &lt;a href=&quot;http://www.youtube.com/results?search_query=statistical+aspects+of+data+mining&amp;amp;aq=0&quot; rel=&quot;nofollow&quot;&gt;Statistical Aspects of Data Mining&lt;/a&gt;. The lecturer is solving example problems using R.&lt;/p&gt;&#xA;" OwnerUserId="315" LastActivityDate="2010-07-26T19:53:45.660" CommunityOwnedDate="2010-08-01T18:56:25.603" />
  <row Id="658" PostTypeId="2" ParentId="643" CreationDate="2010-07-26T19:56:23.970" Score="9" ViewCount="" Body="&lt;p&gt;What I loved most with CLT is the cases when it is not applicable -- this gives me a hope that the life is a bit more interesting that Gauss curve suggests. So show him the Cauchy distribution.&lt;/p&gt;&#xA;" OwnerUserId="88" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-07-27T18:06:52.203" LastActivityDate="2010-07-27T18:06:52.203" CommunityOwnedDate="2011-06-02T11:09:11.620" CommentCount="9" />
  <row Id="659" PostTypeId="2" ParentId="222" CreationDate="2010-07-26T19:58:28.347" Score="2" ViewCount="" Body="&lt;p&gt;The principal components of a data matrix are the eigenvector-eigenvalue pairs of its variance-covariance matrix.  In essence, they are the decorrelated pieces of the variance.  Each one is a linear combination of the variables for an observation -- suppose you measure w, x, y,z on each of a bunch of subjects.  Your first PC might work out to be something like&lt;/p&gt;&#xA;&#xA;&lt;p&gt;0.5w + 4x + 5y - 1.5z&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The loadings (eigenvectors) here are (0.5, 4, 5, -1.5).  The score (eigenvalue) for each observation is the resulting value when you substitute in the observed (w, x, y, z) and compute the total.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This comes in handy when you project things onto their principal components (for, say, outlier detection) because you just plot the scores on each like you would any other data.  This can reveal a lot about your data if much of the variance is correlated (== in the first few PCs).&lt;/p&gt;&#xA;" OwnerUserId="317" LastActivityDate="2010-07-26T19:58:28.347" CommentCount="1" />
  <row Id="660" PostTypeId="1" CreationDate="2010-07-26T20:03:02.643" Score="6" ViewCount="182" Body="&lt;p&gt;I am collecting textual data surrounding press releases, blog posts, reviews, etc of certain companies' products and performance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Specifically, I am looking to see if there are correlations between certain &lt;em&gt;types&lt;/em&gt; and/or &lt;em&gt;sources&lt;/em&gt; of such &quot;textual&quot; content with market valuations of the companies' stock symbols.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Such &lt;em&gt;apparent&lt;/em&gt; correlations can be found by the human mind fairly quickly - but that is not scalable. How can I go about automating such analysis of disparate sources?&lt;/p&gt;&#xA;" OwnerUserId="292" LastActivityDate="2010-11-10T20:50:54.277" Title="Automating statistical correlation between &quot;texts&quot; and &quot;data&quot;" Tags="&lt;finance&gt;&lt;correlation&gt;&lt;text-mining&gt;" AnswerCount="2" CommentCount="3" FavoriteCount="3" />
  <row Id="661" PostTypeId="2" ParentId="155" CreationDate="2010-07-26T20:06:09.983" Score="3" ViewCount="" Body="&lt;p&gt;Sam Savage's book &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0471381977&quot; rel=&quot;nofollow&quot;&gt;Flaw of Averages&lt;/a&gt; is filled with good layman explanations of statistical concepts.  In particular, he has a good explanation of Jensen's inequality.  If the graph of your return on an investment is convex, i.e. it &quot;smiles at you&quot;, then randomness is in your favor: your average return is greater than your return at the average.&lt;/p&gt;&#xA;" OwnerUserId="319" LastActivityDate="2010-07-26T20:06:09.983" />
  <row Id="662" PostTypeId="2" ParentId="614" CreationDate="2010-07-26T20:08:09.760" Score="3" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://www.math.umass.edu/~lavine/Book/book.html&quot; rel=&quot;nofollow&quot;&gt;Introduction to Statistical Thought&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="319" LastActivityDate="2010-07-26T20:08:09.760" CommunityOwnedDate="2010-07-26T20:08:09.760" />
  <row Id="663" PostTypeId="2" ParentId="170" CreationDate="2010-07-26T20:09:20.320" Score="4" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://www.math.umass.edu/~lavine/Book/book.html&quot; rel=&quot;nofollow&quot;&gt;Introduction to Statistical Thought&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="319" LastActivityDate="2010-07-26T20:09:20.320" CommunityOwnedDate="2010-07-26T20:09:20.320" CommentCount="2" />
  <row Id="664" PostTypeId="2" ParentId="375" CreationDate="2010-07-26T20:12:15.570" Score="1" ViewCount="" Body="&lt;p&gt;It's seldom useful to conclude that something is &quot;random&quot; in the abstract.  More often you want to test whether it has a certain kind of random structure.  For example, you might want to test whether something has a uniform distribution, with all values in a certain range equally likely.  Or you might want to test whether something has a normal distribution, etc.  To test whether data has a particular distribution, you can use a goodness of fit test such as the chi square test or the Kolmogorov-Smirnov test.&lt;/p&gt;&#xA;" OwnerUserId="319" LastActivityDate="2010-07-26T20:12:15.570" />
  <row Id="665" PostTypeId="1" CreationDate="2010-07-26T20:17:17.680" Score="15" ViewCount="1075" Body="&lt;p&gt;What's the difference between probability and statistics, and why are they studied together?&lt;/p&gt;&#xA;" OwnerUserId="327" LastEditorUserId="2645" LastEditorDisplayName="" LastEditDate="2011-03-20T16:07:28.560" LastActivityDate="2011-03-20T16:07:28.560" Title="What's the difference between probability and statistics?" Tags="&lt;probability&gt;&lt;teaching&gt;&lt;mathematical-statistics&gt;" AnswerCount="15" FavoriteCount="6" />
  <row Id="666" PostTypeId="2" ParentId="165" CreationDate="2010-07-26T20:18:14.120" Score="2" ViewCount="" Body="&lt;p&gt;I wrote a computer program in Python to &quot;randomly&quot; generate music. I believe the timings and notes are based on row, row, row your boat ;D&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://justinbozonier.posterous.com/creating-music-procedurally-aka-my-computer-m&quot; rel=&quot;nofollow&quot;&gt;http://justinbozonier.posterous.com/creating-music-procedurally-aka-my-computer-m&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should be able to play the song.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also wrote another sample that used several very large books from Project Gutenbergs as a corpus to create &quot;random&quot; text passages that seem almost sensical (Emphasis on almost!).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://justinbozonier.posterous.com/markov-chains-in-python&quot; rel=&quot;nofollow&quot;&gt;http://justinbozonier.posterous.com/markov-chains-in-python&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="274" LastActivityDate="2010-07-26T20:18:14.120" CommunityOwnedDate="2010-08-16T06:37:30.433" CommentCount="3" />
  <row Id="667" PostTypeId="2" ParentId="665" CreationDate="2010-07-26T20:18:46.037" Score="4" ViewCount="" Body="&lt;p&gt;Probability is a pure science (math), statistics is about data. They are connected since probability forms some kind of fundament for statistics, providing basic ideas.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-07-26T20:18:46.037" CommentCount="2" />
  <row Id="668" PostTypeId="2" ParentId="138" CreationDate="2010-07-26T20:19:12.447" Score="2" ViewCount="" Body="&lt;p&gt;If you already know another programming language, &lt;a href=&quot;http://www.johndcook.com/R_language_for_programmers.html&quot; rel=&quot;nofollow&quot;&gt;these notes&lt;/a&gt; may help point out some of the ways R might surprise you.&lt;/p&gt;&#xA;" OwnerUserId="319" LastActivityDate="2010-07-26T20:19:12.447" CommunityOwnedDate="2010-08-01T18:56:25.603" />
  <row Id="669" PostTypeId="2" ParentId="421" CreationDate="2010-07-26T20:22:30.303" Score="3" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0471381977&quot; rel=&quot;nofollow&quot;&gt;The Flaw of Averages&lt;/a&gt; by Sam Savage.&lt;/p&gt;&#xA;" OwnerUserId="319" LastActivityDate="2010-07-26T20:22:30.303" CommunityOwnedDate="2010-07-26T20:22:30.303" />
  <row Id="671" PostTypeId="2" ParentId="138" CreationDate="2010-07-26T20:25:16.363" Score="4" ViewCount="" Body="&lt;p&gt;If you have experience in other languages, these &quot;R Rosetta Stone&quot; videos may be useful:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.vcasmo.com/video/drewconway/7183&quot; rel=&quot;nofollow&quot;&gt;Python&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.vcasmo.com/video/drewconway/7211&quot; rel=&quot;nofollow&quot;&gt;MATLAB&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.vcasmo.com/video/drewconway/7210&quot; rel=&quot;nofollow&quot;&gt;SQL&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;These are all included in the &lt;a href=&quot;http://jeromyanglim.blogspot.com/2010/05/videos-on-data-analysis-with-r.html&quot; rel=&quot;nofollow&quot;&gt;video list added by Jeromy&lt;/a&gt;, so big +1 for his list.&lt;/p&gt;&#xA;" OwnerUserId="302" LastActivityDate="2010-07-26T20:25:16.363" CommunityOwnedDate="2010-08-01T18:56:25.603" />
  <row Id="672" PostTypeId="1" AcceptedAnswerId="824" CreationDate="2010-07-26T20:30:36.507" Score="12" ViewCount="845" Body="&lt;p&gt;What are the main ideas, that is, concepts related to &lt;a href=&quot;http://en.wikipedia.org/wiki/Bayes%27_theorem&quot; rel=&quot;nofollow&quot;&gt;Bayes' theorem&lt;/a&gt;?&#xA;I am not asking for any derivations of complex mathematical notation.&lt;/p&gt;&#xA;" OwnerUserId="333" LastEditorUserId="509" LastEditorDisplayName="" LastEditDate="2011-02-02T19:00:52.413" LastActivityDate="2011-02-28T09:55:19.267" Title="What is Bayes' theorem all about?" Tags="&lt;probability&gt;&lt;bayesian&gt;&lt;theory&gt;" AnswerCount="8" CommentCount="2" FavoriteCount="4" />
  <row Id="673" PostTypeId="2" ParentId="665" CreationDate="2010-07-26T20:34:45.657" Score="5" ViewCount="" Body="&lt;p&gt;Table 3.1 of &lt;a href=&quot;http://www.intuitivebiostatistics.com&quot; rel=&quot;nofollow&quot;&gt;&lt;em&gt;Intuitive Biostatistics&lt;/em&gt;&lt;/a&gt; answers this question with the diagram shown below. Note that all the arrows point to the right for probability, and point to the left for statistics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PROBABILITY&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;General ---&gt; Specific&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Population ---&gt; Sample&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Model ---&gt; Data&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;STATISTICS&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;General &amp;lt;--- Specific&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Population &amp;lt;--- Sample&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Model &amp;lt;--- Data&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="25" LastActivityDate="2010-07-26T20:34:45.657" CommentCount="3" />
  <row Id="674" PostTypeId="2" ParentId="665" CreationDate="2010-07-26T20:45:36.923" Score="3" ViewCount="" Body="&lt;p&gt;Probability is about quantifying uncertainty whereas statistics is explaining the variation in some measure of interest (e.g., why do income levels vary?) that we observe in the real world. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We explain the variation by using some observable factors (e.g., gender, education level, age etc for the income example). However, since we cannot possibly take into account all possible factors that affect income, we leave any unexplained variation to random errors (which is where quantifying uncertainty comes in).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since, we attribute &quot;Variation = Effect of Observable Factors + Effect of Random Errors&quot; we need the tools provided by probability to account for the effect of random errors on the variation that we observe.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some examples follow:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Quantifying Uncertainty&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example 1: You roll a 6-sided dice. What is the probability of obtaining a 1?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example 2: What is the probability that the annual income of an adult person selected at random from the United States is less than $40,000?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Explaining Variation&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example 1: We observe that the annual income of a person varies. What factors explain the variation in a person's income? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Clearly, we cannot account for all factors. Thus, we attribute a person's income to some observable factors (e.g, education level, gender, age etc) and leave any remaining variation to uncertainty (or in the language of statistics: to random errors). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example 2: We observe that some consumers choose Tide most of the time they buy a detergent whereas some other consumers choose detergent brand xyz. What explains the variation in choice? We attribute the variation in choices to some observable factors such as price, brand name etc and leave any unexplained variation to random errors (or uncertainty).&lt;/p&gt;&#xA;" LastActivityDate="2010-07-26T21:01:25.187" CommentCount="2" />
  <row Id="675" PostTypeId="2" ParentId="665" CreationDate="2010-07-26T20:47:19.230" Score="24" ViewCount="" Body="&lt;p&gt;The short answer to this I've heard from Persi Diaconis is the following: the problems considered by probability and statistics are inverse to each other.  In probability theory we consider some underlying process which has some randomness or uncertainty modeled by random variables, and we figure out what happens.  In statistics we observe something that has happened, and try to figure out what underlying process would explain those observations.&lt;/p&gt;&#xA;" OwnerUserId="89" LastActivityDate="2010-07-26T20:47:19.230" CommentCount="5" />
  <row Id="676" PostTypeId="2" ParentId="36" CreationDate="2010-07-26T20:49:33.217" Score="0" ViewCount="" Body="&lt;p&gt;Well my Prof. used these in Introductory probability class:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Shoe size are correlated with reading ability&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Shark attack is correlated with sale of ice cream. &lt;/p&gt;&#xA;" OwnerUserId="288" LastActivityDate="2010-07-26T20:49:33.217" CommunityOwnedDate="2010-08-16T13:01:42.613" />
  <row Id="677" PostTypeId="2" ParentId="660" CreationDate="2010-07-26T20:56:22.360" Score="5" ViewCount="" Body="&lt;p&gt;Let's say you have some data like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Return Symbol News Text&#xA;-4% DELL   Centegra and Dell Services recognized with Outsourcing Center's...&#xA;7%   MSFT   Rising Service Revenues Benefit VMWare&#xA;1%   CSCO   Cisco Systems (CSCO) Receives 5 Star Strong Buy Rating From S&amp;amp;P&#xA;4%   GOOG   Summary Box: Google eyes more government deals&#xA;7%   AAPL   Sohu says 2nd-quarter net income rises 10 percent on higher...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You want to predict the return based on the text.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is called Text Mining. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you do ultimately is create an enormous matrix like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Return Centegra Rising Services Recognized...&#xA;-4%    0.23     0      0.11     0.34&#xA;7%     0        0.1    0.23     0&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;That has one column for every word, and one row for each return, and a weighted score for each word. The score is generated via TFIDF or SVD method. Then you run a regression and see if you can predict which words predict the return. You'll probably need to use PCA first. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Book: Fundamentals of Predictive Text Mining, Weiss&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Software: GATE or RapidMiner with Text Plugin&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should also do a search on Google Scholar and read up on the ins and outs. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can see my series of text mining videos &lt;a href=&quot;http://vancouverdata.blogspot.com/2010/11/text-analytics-with-rapidminer-loading.html&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="74" LastEditorUserId="74" LastEditorDisplayName="" LastEditDate="2010-11-10T20:50:54.277" LastActivityDate="2010-11-10T20:50:54.277" CommentCount="1" />
  <row Id="678" PostTypeId="2" ParentId="665" CreationDate="2010-07-26T21:00:00.617" Score="3" ViewCount="" Body="&lt;p&gt;The &lt;strong&gt;probability&lt;/strong&gt; of an event is its long-run relative frequency. So it's basically telling you the &lt;strong&gt;&lt;em&gt;chance&lt;/em&gt;&lt;/strong&gt; of, for example, getting a 'head' on the next flip of a coin, or getting a '3' on the next roll of a die.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A &lt;strong&gt;statistic&lt;/strong&gt; is any numerical measure computed from a sample of the population. For example, the sample mean. We use this as a statistic which estimates the population mean, which is a parameter. So basically it's giving you some kind of &lt;strong&gt;&lt;em&gt;summary&lt;/em&gt;&lt;/strong&gt; of a sample.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;em&gt;You can only get a statistic from a&#xA;sample, otherwise if you compute a&#xA;numerical measure on a population, it&#xA;is called a population parameter.&lt;/em&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="81" LastEditorUserId="81" LastEditorDisplayName="" LastEditDate="2010-07-26T21:05:06.097" LastActivityDate="2010-07-26T21:05:06.097" />
  <row Id="680" PostTypeId="2" ParentId="645" CreationDate="2010-07-26T21:11:25.677" Score="9" ViewCount="" Body="&lt;p&gt;If you have enormous data sets - ones that make Excel or Notepad load slowly, then a database is a good way to go. MS SQL Server Express is free, and it's easy to connect with JMP and other programs. You may want to sample in this case. You don't have to normalize the data in the database. Otherwise, CSV is sharing-friendly. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In terms of analysis, here are some starting points:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Describe one variable:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Histogram  &lt;/li&gt;&#xA;&lt;li&gt;Summary statistics (mean etc)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Describe relationship between variables:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Scatter Plot  &lt;/li&gt;&#xA;&lt;li&gt;Correlation  &lt;/li&gt;&#xA;&lt;li&gt;Mosaic plot for categorical  &lt;/li&gt;&#xA;&lt;li&gt;Contingency table for categorical  &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Predict a real number: regression&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;OLS regression or machine learning regression techniques&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;when the technique used to predict is understandable by humans, this is called modeling. For example, a neural network can make predictions, but is generally not understandable. You can use regression to find Key Performance Indicators too. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Predict a whole number: classification&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;logistic regression or machine learning techniques&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Put observations into &quot;natural&quot; groups: clustering&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Generally one finds &quot;similar&quot; observations by calculating the distance between them. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Put attributes into &quot;natural&quot; groups: factoring&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;And other matrix operations such as PCA, NMF&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Quantifying Risk = Standard Deviation, or proportion of times that &quot;bad things&quot; happen&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Likelihood of a successfully completed iteration given x number of story points = Logistic Regression&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good luck!&lt;/p&gt;&#xA;" OwnerUserId="74" LastEditorUserId="74" LastEditorDisplayName="" LastEditDate="2011-01-25T22:04:35.493" LastActivityDate="2011-01-25T22:04:35.493" CommentCount="2" />
  <row Id="681" PostTypeId="2" ParentId="288" CreationDate="2010-07-26T21:13:43.743" Score="2" ViewCount="" Body="&lt;p&gt;You don't necessarily have to go Bayesian on your model, plain maximum likelihood estimation works just fine (though has no explicit solution). Multiple R packages (eg. aod or VGAM) will fit the distribution for you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively, you can use the quasi-likelihood based overdispersed binomial model that does not assume a beta-binomial distribution, just adjusts for the overdispersion. The &lt;code&gt;glm&lt;/code&gt; function with the &lt;code&gt;quasibinomial&lt;/code&gt; family will fit this model in R.&lt;/p&gt;&#xA;" OwnerUserId="279" LastActivityDate="2010-07-26T21:13:43.743" />
  <row Id="682" PostTypeId="2" ParentId="652" CreationDate="2010-07-26T21:17:29.877" Score="3" ViewCount="" Body="&lt;p&gt;I didn't find &lt;em&gt;How To Measure Anything&lt;/em&gt;, nor &lt;em&gt;Head First&lt;/em&gt;, particularly good.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Statistics In Plain English&lt;/em&gt; is a good starter book.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you finish that, &lt;em&gt;Multivariate Data Analysis&lt;/em&gt; (Joseph Hair &lt;em&gt;et al.&lt;/em&gt;) is fantastic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good luck!&lt;/p&gt;&#xA;" OwnerUserId="74" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-16T22:48:04.127" LastActivityDate="2011-03-16T22:48:04.127" CommunityOwnedDate="2010-07-26T21:17:29.877" CommentCount="1" />
  <row Id="683" PostTypeId="2" ParentId="322" CreationDate="2010-07-26T21:34:14.130" Score="2" ViewCount="" Body="&lt;p&gt;If your interested in the mathematical statistic around entropy, you may consult this book &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.renyi.hu/~csiszar/Publications/Information_Theory_and_Statistics:_A_Tutorial.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.renyi.hu/~csiszar/Publications/Information_Theory_and_Statistics:_A_Tutorial.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;it is freely available ! &lt;/p&gt;&#xA;" OwnerUserId="223" LastEditorUserId="223" LastEditorDisplayName="" LastEditDate="2010-09-02T09:21:53.573" LastActivityDate="2010-09-02T09:21:53.573" />
  <row Id="684" PostTypeId="2" ParentId="672" CreationDate="2010-07-26T21:35:29.510" Score="4" ViewCount="" Body="&lt;p&gt;There are two schools of thought is Statistics: &lt;a href=&quot;http://stats.stackexchange.com/questions/22/bayesian-and-frequentist-reasoning-in-plain-english&quot;&gt;Frequentest and Bayesian&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bayes theorem is to do with the latter and can be seen as a way of understanding how the probability that a theory is true is affected by a new piece of evidence. This is known as conditional probability. You might want to look at &lt;a href=&quot;http://stattrek.com/Lesson1/Bayes.aspx&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; to get a handle on the math.&lt;/p&gt;&#xA;" OwnerUserId="81" LastEditorUserId="81" LastEditorDisplayName="" LastEditDate="2011-02-10T11:39:16.037" LastActivityDate="2011-02-10T11:39:16.037" CommentCount="3" />
  <row Id="685" PostTypeId="1" CreationDate="2010-07-26T21:40:20.870" Score="-2" ViewCount="186" Body="&lt;p&gt;Is there something about statistics that lends itself to this sort of saying, or is it just that people will say anything to support their case, and this includes citing irrelevant or incomplete statistics?&lt;/p&gt;&#xA;" OwnerUserId="327" LastActivityDate="2010-07-26T22:06:32.547" ClosedDate="2010-07-29T23:45:11.587" Title="Lies, Damn Lies and Statistics" Tags="&lt;theory&gt;" AnswerCount="2" CommentCount="5" />
  <row Id="686" PostTypeId="2" ParentId="485" CreationDate="2010-07-26T21:58:37.497" Score="2" ViewCount="" Body="&lt;p&gt;Check out the following links. I'm not sure what exactly are you looking for.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://videolectures.net/mlss08au_freitas_asm/&quot; rel=&quot;nofollow&quot;&gt;Monte Carlo Simulation for Statistical Inference&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://videolectures.net/mlss08au_smola_ksvm/&quot; rel=&quot;nofollow&quot;&gt;Kernel methods and Support Vector Machines&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://videolectures.net/epsrcws08_campbell_isvm/&quot; rel=&quot;nofollow&quot;&gt;Introduction to Support Vector Machines&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://academicearth.org/lectures/monte-carlo-simulations-application-to-lattice-models&quot; rel=&quot;nofollow&quot;&gt;Monte Carlo Simulations&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://freescienceonline.blogspot.com/2009_09_01_archive.html&quot; rel=&quot;nofollow&quot;&gt;Free Science and Video Lectures Online!&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://videolectures.net/Top/Computer_Science/Machine_Learning/&quot; rel=&quot;nofollow&quot;&gt;Video lectures on Machine Learning&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="339" LastActivityDate="2010-07-26T21:58:37.497" CommunityOwnedDate="2010-07-26T21:58:37.497" CommentCount="1" />
  <row Id="687" PostTypeId="2" ParentId="685" CreationDate="2010-07-26T22:00:01.037" Score="2" ViewCount="" Body="&lt;p&gt;Statistics is about inferring something about a population, and that  requires some level of interpretation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More intuitively, &quot;is the glass half full or half empty?&quot;. They both mean the same thing, but  may have a different effect on the person who hears it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I would say it's the interpretation aspect which is the problem&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S. There's an interesting article on the &lt;a href=&quot;http://www.bbc.co.uk/dna/h2g2/A1091350&quot; rel=&quot;nofollow&quot;&gt;BBC website&lt;/a&gt; which may be worth a read.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.P.S. If you meant this more generally, then there could be a case for saying that the frequentest approach to statistics can give a different result to the Bayesian approach.&lt;/p&gt;&#xA;" OwnerUserId="81" LastActivityDate="2010-07-26T22:00:01.037" CommentCount="1" />
  <row Id="688" PostTypeId="2" ParentId="685" CreationDate="2010-07-26T22:06:32.547" Score="0" ViewCount="" Body="&lt;p&gt;I think it is because &quot;normal&quot; people &lt;em&gt;think&lt;/em&gt; that they understand it.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-07-26T22:06:32.547" CommunityOwnedDate="2010-07-26T22:09:02.433" />
  <row Id="689" PostTypeId="2" ParentId="485" CreationDate="2010-07-26T22:12:58.017" Score="5" ViewCount="" Body="&lt;p&gt;See &lt;a href=&quot;http://jeromyanglim.blogspot.com/2010/05/videos-on-data-analysis-with-r.html?utm_source=feedburner&amp;amp;utm_medium=feed&amp;amp;utm_campaign=Feed%3a+jeromyanglim+%28Jeromy+Anglim%27s+Blog%3a+Psychology+and+Statistics%29&quot; rel=&quot;nofollow&quot;&gt;Videos on data analysis using R&lt;/a&gt; on Jeromy Anglim's blog.  There are many links at that page and he updates it.  He has another post with many links to videos on probability and statistics as well as linear algebra and calculus.&lt;/p&gt;&#xA;" LastActivityDate="2010-07-26T22:12:58.017" CommunityOwnedDate="2010-07-26T22:12:58.017" />
  <row Id="691" PostTypeId="2" ParentId="118" CreationDate="2010-07-26T22:22:21.440" Score="3" ViewCount="" Body="&lt;p&gt;Just so people know, there is a Math Overflow question on the same topic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://mathoverflow.net/questions/1048/why-is-it-so-cool-to-square-numbers-in-terms-of-finding-the-standard-deviation&quot; rel=&quot;nofollow&quot;&gt;Why-is-it-so-cool-to-square-numbers-in-terms-of-finding-the-standard-deviation&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The take away message is that using the square root of the variance leads to easier maths. A similar response is given by Rich and Reed above. &lt;/p&gt;&#xA;" OwnerUserId="352" LastActivityDate="2010-07-26T22:22:21.440" />
  <row Id="692" PostTypeId="1" AcceptedAnswerId="741" CreationDate="2010-07-26T22:29:48.407" Score="9" ViewCount="183" Body="&lt;p&gt;Oversimplifying a bit, I have about a million records that record the entry time and exit time of people in a system spanning about ten years.  Every record has an entry time, but not every record has an exit time.  The mean time in the system is ~1 year. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The missing exit times happen for two reasons:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The person has not left the system at the time the data was captured.&lt;/li&gt;&#xA;&lt;li&gt;The person's exit time was not recorded. This happens to say 50% of the records&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The questions of interest are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Are people spending less time in the system, and how much less time.&lt;/li&gt;&#xA;&lt;li&gt;Are more exit times being recorded, and how many.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;We can model this by saying that the probability that an exit gets recorded varies linearly with time, and that the time in the system has a Weibull whose parameters vary linearly with time.  We can then make a maximum likelihood estimate of the various parameters and eyeball the results and deem them plausible.  We chose the Weibull distribution because it seems to be used in measuring lifetimes and is fun to say as opposed to fitting the data better than say a gamma distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where should I look to get a clue as to how to do this correctly?  We are somewhat mathematically savvy, but not extremely statistically savvy.   &lt;/p&gt;&#xA;" OwnerUserId="72" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-16T12:35:10.570" LastActivityDate="2010-09-16T12:35:10.570" Title="How do I determine if a survival model with missing data is appropriate?" Tags="&lt;survival&gt;&lt;missing-data&gt;" AnswerCount="2" CommentCount="0" FavoriteCount="2" />
  <row Id="693" PostTypeId="2" ParentId="665" CreationDate="2010-07-26T22:48:51.740" Score="12" ViewCount="" Body="&lt;p&gt;I like the example of a jar of red and green jelly beans.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A probabilist starts by knowing the proportion of each and asks the probability of drawing a red jelly bean.  A statistician infers the proportion of red jelly beans by sampling from the jar.&lt;/p&gt;&#xA;" OwnerUserId="319" LastActivityDate="2010-07-26T22:48:51.740" CommentCount="1" />
  <row Id="694" PostTypeId="2" ParentId="423" CreationDate="2010-07-26T23:04:39.127" Score="19" ViewCount="" Body="&lt;p&gt;This isn't technically a cartoon, but close enough:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://www.austinthirdgen.org/upload/piechart.jpg&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;props to xkcd!&lt;/p&gt;&#xA;" OwnerUserId="74" LastEditorUserId="74" LastEditorDisplayName="" LastEditDate="2010-09-09T05:37:35.640" LastActivityDate="2010-09-09T05:37:35.640" CommunityOwnedDate="2010-07-26T23:04:39.127" CommentCount="1" />
  <row Id="695" PostTypeId="2" ParentId="652" CreationDate="2010-07-26T23:10:16.777" Score="2" ViewCount="" Body="&lt;p&gt;My favourite book on Statistics is is David William's &lt;a href=&quot;http://amzn.to/aRoxQq&quot; rel=&quot;nofollow&quot;&gt;Weighing the Odds&lt;/a&gt;. Davison's &lt;a href=&quot;http://amzn.to/90gOnD&quot; rel=&quot;nofollow&quot;&gt;Statistical Models&lt;/a&gt; is good too.&lt;/p&gt;&#xA;" OwnerUserId="173" LastActivityDate="2010-07-26T23:10:16.777" CommunityOwnedDate="2010-07-26T23:10:16.777" CommentCount="1" />
  <row Id="696" PostTypeId="2" ParentId="195" CreationDate="2010-07-26T23:22:39.377" Score="1" ViewCount="" Body="&lt;p&gt;I think that my question is subsumed by this more general discussion: &lt;a href=&quot;http://stats.stackexchange.com/questions/411/motivation-for-kolmogorov-distance-between-distributions&quot; rel=&quot;nofollow&quot;&gt;http://stats.stackexchange.com/questions/411/motivation-for-kolmogorov-distance-between-distributions&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="173" LastActivityDate="2010-07-26T23:22:39.377" />
  <row Id="697" PostTypeId="2" ParentId="660" CreationDate="2010-07-26T23:48:24.280" Score="1" ViewCount="" Body="&lt;p&gt;As per above, you need a set of articles and responses, and then you train eg. a Neural Net to them. RapidMiner will let you do this but there are many other tools out there that will let you do regressions of this size. Ideally your response variable will be consistent (ie % change after 1 hour exactly, or % change after 1 day exactly etc). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may also want to apply some sort of filtering or classification to your training variables ie the words in the article. This could be as simple as filtering some words (eg prepositions, pronouns) or more complex like using syntax to choose which words should go into the regression. Note that any filtering you do risks biasing the result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some folks at University of Arizona already made a system that does this - their paper is on acm here and you may find it interesting. &lt;a href=&quot;http://www.computer.org/portal/web/csdl/doi/10.1109/MC.2010.2&quot; rel=&quot;nofollow&quot;&gt;http://www.computer.org/portal/web/csdl/doi/10.1109/MC.2010.2&lt;/a&gt; (you'll need a subscription to access if you're not eg at university). The references may also help point you in the right direction.&lt;/p&gt;&#xA;" OwnerUserId="367" LastActivityDate="2010-07-26T23:48:24.280" />
  <row Id="699" PostTypeId="2" ParentId="346" CreationDate="2010-07-27T00:01:58.560" Score="2" ViewCount="" Body="&lt;p&gt;The &lt;a href=&quot;http://en.wikipedia.org/wiki/Selection_algorithm#Linear_general_selection_algorithm_-_Median_of_Medians_algorithm&quot; rel=&quot;nofollow&quot;&gt;Rivest-Tarjan-Selection algorithm&lt;/a&gt; (sometimes also called the median-of-medians algorithm) will let you compute the median element in linear-time without any sorting. For large data sets this is can be quite a bit faster than log-linear sorting. However, it won't solve your memory storage problem.&lt;/p&gt;&#xA;" OwnerUserId="352" LastActivityDate="2010-07-27T00:01:58.560" />
  <row Id="700" PostTypeId="2" ParentId="322" CreationDate="2010-07-27T00:21:32.637" Score="2" ViewCount="" Body="&lt;p&gt;These &lt;a href=&quot;http://www.statslab.cam.ac.uk/~yms/ICL0706.ps&quot; rel=&quot;nofollow&quot;&gt;lecture notes&lt;/a&gt; on information theory by O. Johnson contain a good introduction to different kinds of entropy.&lt;/p&gt;&#xA;" OwnerUserId="368" LastActivityDate="2010-07-27T00:21:32.637" />
  <row Id="701" PostTypeId="2" ParentId="118" CreationDate="2010-07-27T00:24:09.637" Score="1" ViewCount="" Body="&lt;p&gt;Because squares can allow use of many other mathematical operations or functions more easily than absolute values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example: squares can be integrated, differentiated, can be used in trigonometric, logarithmic and other functions, with ease.&lt;/p&gt;&#xA;" OwnerUserId="369" LastActivityDate="2010-07-27T00:24:09.637" />
  <row Id="702" PostTypeId="2" ParentId="692" CreationDate="2010-07-27T00:37:33.963" Score="2" ViewCount="" Body="&lt;p&gt;You could use the estimated model to predict the exit times for all the people in your system. You could then compare the estimated exit times with the actual exit times (where you have this data) and compute a metric such as &lt;a href=&quot;http://en.wikipedia.org/wiki/Root_mean_square_deviation&quot; rel=&quot;nofollow&quot;&gt;RMSE&lt;/a&gt; to assess how good your predictions are which will in turn give you a sense of model fit. See also this &lt;a href=&quot;http://www.statsoft.com/textbook/survival-failure-time-analysis/#dgoodness&quot; rel=&quot;nofollow&quot;&gt;link&lt;/a&gt;.&lt;/p&gt;&#xA;" LastActivityDate="2010-07-27T00:37:33.963" CommentCount="2" />
  <row Id="703" PostTypeId="2" ParentId="622" CreationDate="2010-07-27T00:47:25.470" Score="3" ViewCount="" Body="&lt;p&gt;All three are used when dealing with nuisance parameters in the completely specified likelihood function.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The marginal likelihood  is the primary method to eliminate nuisance parameters in theory.  It's a true likelihood function (i.e. it's proportional to the (marginal) probability of the observed data).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The partial likelihood is not a true likelihood in general.  However, in some cases it can be treated as a likelihood for asymptotic inference.  For example in Cox proportional hazards models, where it originated, we're interested in the observed rankings in the data (T1 &gt; T2 &gt; ..) without specifying the baseline hazard.  Efron showed that the partial likelihood loses little to no information for a variety of hazard functions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The profile likelihood is convenient when we have a multidimensional likelihood function and a single parameter of interest.  It's specified by replacing the nuisance S by its MLE at each fixed T (the parameter of interest), i.e. L(T) = L(T, S(T)).  This can work well in practice, though there is a potential bias in the MLE obtained in this way; the marginal likelihood corrects for this bias.&lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-07-27T00:47:25.470" />
  <row Id="704" PostTypeId="2" ParentId="346" CreationDate="2010-07-27T00:54:58.187" Score="0" ViewCount="" Body="&lt;p&gt;Here's an answer to the question asked on stackoverflow: &lt;a href=&quot;http://stackoverflow.com/questions/1058813/on-line-iterator-algorithms-for-estimating-statistical-median-mode-skewness/2144754#2144754&quot;&gt;http://stackoverflow.com/questions/1058813/on-line-iterator-algorithms-for-estimating-statistical-median-mode-skewness/2144754#2144754&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The iterative update median += eta * sgn(sample - median) sounds like it could be a way to go.&lt;/p&gt;&#xA;" LastActivityDate="2010-07-27T00:54:58.187" CommentCount="1" />
  <row Id="705" PostTypeId="2" ParentId="570" CreationDate="2010-07-27T01:07:48.393" Score="3" ViewCount="" Body="&lt;p&gt;This is a very interesting question. Suppose that we have a 2 dimensional covariance matrix (very unrealistic example for SEM but please bear with me). Then you can plot the iso-contours for the observed covariance matrix vis-a-vis the estimated covariance matrix to get a sense of model fit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, in reality you will a high-dimensional covariance matrix. In such a situation, you could probably do several 2 dimensional plots taking 2 variables at a time. Not the ideal solution but perhaps may help to some extent.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A slightly better method is to perform &lt;a href=&quot;http://en.wikipedia.org/wiki/Principal_component_analysis&quot; rel=&quot;nofollow&quot;&gt;Principal Component Analysis (PCA)&lt;/a&gt; on the observed covariance matrix. Save the projection matrix from the PCA analysis on the observed covariance matrix. Use this projection matrix to transform the estimated covariance matrix. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We then plot iso-contours for the two highest variances of the rotated observed covariance matrix vis-a-vis the estimated covariance matrix. Depending on how many plots we want to do we can take the second and the third highest variances etc. We start from the highest variances as we want to explain as much variation in our data as possible. &lt;/p&gt;&#xA;" LastActivityDate="2010-07-27T02:14:37.583" CommentCount="3" />
  <row Id="706" PostTypeId="2" ParentId="118" CreationDate="2010-07-27T01:51:15.673" Score="1" ViewCount="" Body="&lt;p&gt;Yet another reason (in addition to the excellent ones above) comes from Fisher himself, who showed that the standard deviation is more &quot;efficient&quot; than the absolute deviation. Here, efficient has to do with how much a statistic will fluctuate in value on different samplings from a population. If your population is normally distributed, the standard deviation of various samples from that population will, on average, tend to give you values that are pretty similar to each other, whereas the absolute deviation will give you numbers that spread out a bit more. Now, obviously this is in ideal circumstances, but this reason convinced a lot of people (along with the math being cleaner), so most people worked with standard deviations.&lt;/p&gt;&#xA;" OwnerUserId="378" LastActivityDate="2010-07-27T01:51:15.673" />
  <row Id="707" PostTypeId="2" ParentId="638" CreationDate="2010-07-27T02:19:56.727" Score="3" ViewCount="" Body="&lt;p&gt;As an extension on the k-fold answer, the &quot;usual&quot; choice of k is either 5 or 10. The leave-one-out method has a tendency to produce models that are too conservative. FYI, here is a reference on that fact: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Shao, J. (1993), Linear Model Selection by Cross-Validation, Journal of the American&#xA;Statistical Association, Vol. 88, No. 422, pp. 486-494&lt;/p&gt;&#xA;" OwnerUserId="188" LastActivityDate="2010-07-27T02:19:56.727" CommentCount="6" />
  <row Id="708" PostTypeId="2" ParentId="672" CreationDate="2010-07-27T02:31:38.587" Score="2" ViewCount="" Body="&lt;p&gt;Let me give you a very very intuitional insight.  Suppose you are tossing a coin 10 times and you get 8 heads and 2 tails. The question that would come to your mind is whether this coin is biased towards heads or not. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now if you go by conventional definitions or the frequentist approach of probability you might say that the coin is unbiased and this is an exceptional occurrence. Hence you would conclude that the possibility of getting a head next toss is also 50%. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But suppose you are a Bayesian.  You would actually think that since you have got exceptionally high number of heads, the coin has a bias towards the head side. There are methods to calculate this possible bias. You would calculate them and then when you toss the coin next time, you would definitely call a heads.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, Bayesian probability is about the belief that you develop based on the data you observe. I hope that was simple enough.&lt;/p&gt;&#xA;" OwnerUserId="387" LastEditorUserId="509" LastEditorDisplayName="" LastEditDate="2011-02-02T19:11:03.597" LastActivityDate="2011-02-02T19:11:03.597" />
  <row Id="709" PostTypeId="2" ParentId="563" CreationDate="2010-07-27T03:46:45.583" Score="4" ViewCount="" Body="&lt;p&gt;Hi Graham,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://cgibbons.berkeley.edu/Courses/ECON141_S10/IV.pdf&quot; rel=&quot;nofollow&quot;&gt;Here are some slides that I prepared for an econometrics course at UC Berkeley.&lt;/a&gt; I hope that you find them useful---I believe that they answer your questions and provide some examples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Charlie&lt;/p&gt;&#xA;" OwnerUserId="401" LastActivityDate="2010-07-27T03:46:45.583" CommentCount="1" />
  <row Id="710" PostTypeId="2" ParentId="612" CreationDate="2010-07-27T03:54:35.923" Score="2" ViewCount="" Body="&lt;p&gt;My understanding is that the distinction between PCA and Factor analysis primarily is in whether there is an error term. Thus PCA can, and will, faithfully represent the data whereas factor analysis is less faithful to the data it is trained on but attempts to represent underlying trends or communality in the data.  Under a standard approach PCA is not rotated, but it is mathematically possible to do so, so people do it from time to time.  I agree with the commenters in that the &quot;meaning&quot; of these methods is somewhat up for grabs and that it probably is wise to be sure the function you are using does what you intend - for example, as you note R has some functions that perform a different sort of PCA than users of SPSS are familiar with.&lt;/p&gt;&#xA;" OwnerUserId="196" LastActivityDate="2010-07-27T03:54:35.923" />
  <row Id="711" PostTypeId="2" ParentId="452" CreationDate="2010-07-27T03:54:51.710" Score="4" ViewCount="" Body="&lt;p&gt;Using robust standard errors has become common practice in economics. Robust standard errors are typically larger than non-robust (standard?) standard errors, so the practice can be viewed as an effort to be conservative. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In large samples (&lt;i&gt;e.g.,&lt;/i&gt; if you are working with Census data with millions of observations or data sets with &quot;just&quot; thousands of observations), heteroskedasticity tests will almost surely turn up positive, so this approach is appropriate. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another means for combating heteroskedasticity is weighted least squares, but this approach has become looked down upon because it changes the estimates for parameters, unlike the use of robust standard errors. If your weights are incorrect, your estimates are biased. If your weights are right, however, you get smaller (&quot;more efficient&quot;) standard errors than OLS with robust standard errors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Charlie&lt;/p&gt;&#xA;" OwnerUserId="401" LastActivityDate="2010-07-27T03:54:51.710" CommentCount="3" />
  <row Id="712" PostTypeId="1" CreationDate="2010-07-27T03:58:22.863" Score="6" ViewCount="209" Body="&lt;p&gt;Is anyone aware of good data anonymization software? Or perhaps a package for R that does data anonymization? Obviously not expecting uncrackable anonymization - just want to make it difficult. &lt;/p&gt;&#xA;" OwnerUserId="402" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-01T17:23:11.270" LastActivityDate="2011-05-01T17:23:11.270" Title="Data anonymization software" Tags="&lt;software&gt;&lt;anonymization&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="713" PostTypeId="2" ParentId="118" CreationDate="2010-07-27T04:04:15.027" Score="2" ViewCount="" Body="&lt;p&gt;Naturally you can describe dispersion of a distribution in any way meaningful (absolute deviation, quantiles, etc.). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One nice fact is that the variance is the second central moment, and every distribution is uniquely described by its moments if they exist.&#xA;Another nice fact is that the variance is much more tractable mathematically than any comparable metric.&#xA;Another fact is that the variance is one of two parameters of the normal distribution for the usual parametrization, and the normal distribution only has 2 non-zero central moments which are those two very parameters. Even for non-normal distributions it can be helpful to think in a normal framework.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As I see it, the reason the standard deviation exists as such is that in applications the square-root of the variance regularly appears (such as to standardize a random varianble), which necessitated a name for it.&lt;/p&gt;&#xA;" LastActivityDate="2010-07-27T04:04:15.027" />
  <row Id="714" PostTypeId="2" ParentId="73" CreationDate="2010-07-27T04:27:33.930" Score="2" ViewCount="" Body="&lt;p&gt;data.table is my favorite now! Very look forward to the new version with the more wishlist implemented. &lt;/p&gt;&#xA;" LastActivityDate="2010-07-27T04:27:33.930" CommunityOwnedDate="2010-07-27T04:27:33.930" />
  <row Id="715" PostTypeId="1" CreationDate="2010-07-27T04:42:25.210" Score="2" ViewCount="146" Body="&lt;p&gt;I am developing a multi-class perceptron algorithm and was wondering if there are any datasets that could be used to test a multi-class perceptron? - A dataset where the classes are linearly separable and have at least 100 or more instances for training?&lt;/p&gt;&#xA;" OwnerUserId="130" LastEditorUserId="442" LastEditorDisplayName="" LastEditDate="2010-08-30T15:13:35.773" LastActivityDate="2010-08-30T15:13:35.773" Title="Dataset for multi class perceptron" Tags="&lt;classification&gt;&lt;dataset&gt;&lt;multivariable&gt;" AnswerCount="1" />
  <row Id="717" PostTypeId="2" ParentId="225" CreationDate="2010-07-27T05:04:45.570" Score="1" ViewCount="" Body="&lt;p&gt;I asked about why there was a difference between the average of the maximum of 100 draws from a random normal distribution and the 98th percentile of the normal distribution.  The answer I received from Rob Hyndman was mostly acceptable, but too technically dense to accept without revision.  I was left wondering whether it was possible to provide an answer that explains in intuitively understandable plain language why these two values are not equal.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ultimately, my answer may be unsatisfyingly circular; but conceptually, the reason max(rnorm(100)) tends to be higher than qnorm(.98) is, in short, because on average the highest of 100 random normally distributed scores will on occasion exceed its expected value.  However this distortion is non-symmetrical, since when low scores are drawn they are unlikely to end up being the highest out of the 100 scores.  Each independent draw is a new chance to exceed the expected value, or to be ignored because the obtained value isn't the maximum of the 100 drawn values.  For a visual demonstration compare the histogram of the maximum of 20 values to the histogram of the maximum of 100 values, the difference in skew, especially in the tails, is stark.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I arrived at this answer indirectly while working through a related problem/question I had asked in the comments.  Specifically, if I found that someone's test scores were ranked in the 95th percentile, I'd expect that on average if I put them in a room with 99 other test takers that their rank would average out to be 95.  This turns out to be more or less the case (R code)...&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for (i in 1:NSIM)&#xA;{&#xA;    rank[i] &amp;lt;- seq(1,100)[order(c(qnorm(.95),rnorm(99)))==1]&#xA;}&#xA;summary(rank)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As an extension of that logic, I had likewise been expecting that if I took 100 people in a room and selected the person with 95th highest score, then took another 99 people and had them take the same test, that &lt;em&gt;on average&lt;/em&gt; the selected person would be ranked 95th in the new group. But this is not the case (R code)...&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for (i in 1:NSIM)&#xA;{&#xA;    testtakers &amp;lt;- rnorm(100)&#xA;    testtakers &amp;lt;- testtakers[order(testtakers)]&#xA;    testtakers &amp;lt;- testtakers[order(testtakers)]&#xA;    ranked95 &amp;lt;- testtakers[95]&#xA;    rank[i] &amp;lt;- seq(1,100)[order(c(ranked95,rnorm(99)))==1]&#xA;}&#xA;summary(rank)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What makes the first case different from the second case is that in the first case the individual's score places them at exactly the 95th percentile.  In the second case their score may turn out to be somewhat higher or lower than the true 95th percentile.  Since they can not possibly rank higher than 100, groups that produce a rank 95 score that is actually at the 99th percentile or higher can not offset (in terms of average rank) those cases where the rank 95 score is much lower than the true 90th percentile.  If you look at the histograms for the two rank vectors provided in this answer it is easy to see that there is a restriction of range in the upper ends that is a consequence of this process I have been describing.&lt;/p&gt;&#xA;" OwnerUserId="196" LastActivityDate="2010-07-27T05:04:45.570" />
  <row Id="718" PostTypeId="2" ParentId="575" CreationDate="2010-07-27T05:23:38.050" Score="4" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://www.uvm.edu/~dhowell/StatPages/More_Stuff/RepMeasMultComp/RepMeasMultComp.html&quot; rel=&quot;nofollow&quot;&gt;This article&lt;/a&gt; by David Howell explains the problems and several solutions.&lt;/p&gt;&#xA;" OwnerUserId="25" LastActivityDate="2010-07-27T05:23:38.050" CommentCount="1" />
  <row Id="720" PostTypeId="2" ParentId="486" CreationDate="2010-07-27T05:36:14.333" Score="5" ViewCount="" Body="&lt;p&gt;All that matters is the difference between two AIC (or, better, AICc) values, representing the fit to two models.  The actual value of the AIC (or AICc), and whether it is positive or negative, means nothing. If you simply changed the units the data are expressed in, the AIC (and AICc) would change dramatically. But the difference between the AIC of the two alternative models would not change at all.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bottom line: Ignore the actual value of AIC (or AICc) and whether it is positive or negative. Ignore also the ratio of two AIC (or AICc) values. Pay attention only to the difference.&lt;/p&gt;&#xA;" OwnerUserId="25" LastActivityDate="2010-07-27T05:36:14.333" CommentCount="4" />
  <row Id="721" PostTypeId="2" ParentId="39" CreationDate="2010-07-27T05:40:44.950" Score="4" ViewCount="" Body="&lt;p&gt;The &lt;a href=&quot;http://www.ats.ucla.edu/stat/&quot; rel=&quot;nofollow&quot;&gt;UCLA Statistical Computing&lt;/a&gt; site has a number of examples in various languages (SAS, R, etc).  In particular, see the following pages (look among the links titled logistic regression, categorical data analysis and generalized linear models):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.ats.ucla.edu/stat/dae/&quot; rel=&quot;nofollow&quot;&gt;Data Analysis Examples&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.ats.ucla.edu/stat/examples/default.htm&quot; rel=&quot;nofollow&quot;&gt;Textbook Examples&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-07-27T05:40:44.950" />
  <row Id="722" PostTypeId="2" ParentId="175" CreationDate="2010-07-27T05:41:11.750" Score="3" ViewCount="" Body="&lt;p&gt;I've published a method for identifying outliers in nonlinear regression, and it can be also used when fitting a linear model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;HJ Motulsky and RE Brown. &lt;a href=&quot;http://www.biomedcentral.com/1471-2105/7/123/abstract/&quot; rel=&quot;nofollow&quot;&gt;Detecting outliers when fitting data with nonlinear regression – a new method based on robust nonlinear regression and the false discovery rate&lt;/a&gt;. BMC Bioinformatics 2006, 7:123&lt;/p&gt;&#xA;" OwnerUserId="25" LastActivityDate="2010-07-27T05:41:11.750" />
  <row Id="723" PostTypeId="1" AcceptedAnswerId="758" CreationDate="2010-07-27T06:00:07.690" Score="5" ViewCount="447" Body="&lt;p&gt;I'm doing shopping cart analyses my dataset is set of transaction vectors, with the items the products being bought.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When applying k-means on the transactions, I will always get &lt;em&gt;some&lt;/em&gt; result. A random matrix would probably also show some clusters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a way to test whether the clustering I find is a significant one, or that is can be very well be a coincidence. If yes, how can I do it.&lt;/p&gt;&#xA;" OwnerUserId="190" LastActivityDate="2010-07-27T14:44:54.903" Title="How can I test whether my clustering of binary data is significant" Tags="&lt;clustering&gt;&lt;statistical-significance&gt;&lt;binary-data&gt;" AnswerCount="3" FavoriteCount="6" />
  <row Id="724" PostTypeId="2" ParentId="712" CreationDate="2010-07-27T06:01:10.010" Score="5" ViewCount="" Body="&lt;p&gt;The &lt;a href=&quot;http://sourceforge.net/projects/anony-toolkit/&quot; rel=&quot;nofollow&quot;&gt;Cornell Anonymization Tookit&lt;/a&gt; is open source.  Their &lt;a href=&quot;http://www.cs.cornell.edu/bigreddata/privacy/&quot; rel=&quot;nofollow&quot;&gt;research page&lt;/a&gt; has links to associated publications.&lt;/p&gt;&#xA;" OwnerUserId="251" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-13T10:35:33.393" LastActivityDate="2011-04-13T10:35:33.393" />
  <row Id="725" PostTypeId="1" AcceptedAnswerId="759" CreationDate="2010-07-27T06:10:43.347" Score="4" ViewCount="152" Body="&lt;p&gt;An &lt;strong&gt;hyperspectral image is&lt;/strong&gt; a multidimensional image with more than 200 spectral bands i.e. an image for which each pixel is a vector of dimension 200 (most often it is a sampled spectral curve that is encoutered in satellite imagery or medical imagery). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are the &lt;strong&gt;implemented package&lt;/strong&gt; (I am especially interested in R packages but if other free algorithms exist, I will try them) for &lt;strong&gt;frontier detection&lt;/strong&gt; and (unsupervised) &lt;strong&gt;segmentation&lt;/strong&gt; of this type of images?  &lt;/p&gt;&#xA;" OwnerUserId="223" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-17T10:22:25.583" LastActivityDate="2011-01-25T20:40:17.520" Title="Suggested R packages for frontier estimation or segmentation of hyperspectral images" Tags="&lt;machine-learning&gt;&lt;multivariate-analysis&gt;&lt;image-processing&gt;" AnswerCount="4" />
  <row Id="726" PostTypeId="1" CreationDate="2010-07-27T06:20:38.880" Score="44" ViewCount="4231" Body="&lt;p&gt;What is your favorite statistician quote? &#xA;This is community wiki, so please one quote per answer.  &lt;/p&gt;&#xA;" OwnerUserId="223" LastEditorUserId="223" LastEditorDisplayName="" LastEditDate="2010-12-17T08:01:29.653" LastActivityDate="2011-06-01T17:43:40.663" CommunityOwnedDate="2010-07-27T06:20:38.880" Title="Famous statistician quotes" Tags="&lt;big-list&gt;&lt;history&gt;" AnswerCount="99" FavoriteCount="33" />
  <row Id="727" PostTypeId="2" ParentId="726" CreationDate="2010-07-27T06:23:57.233" Score="26" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;The combination of some data and an&#xA;  aching desire for an answer does not&#xA;  ensure that a reasonable answer can be&#xA;  extracted from a given body of data&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Tukey&lt;/p&gt;&#xA;" OwnerUserId="223" LastActivityDate="2010-07-27T06:23:57.233" CommunityOwnedDate="2010-07-27T06:23:57.233" />
  <row Id="728" PostTypeId="2" ParentId="726" CreationDate="2010-07-27T06:26:04.483" Score="11" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;All we know about the world teaches us that the effects of A and B are always different---in some decimal place---for any A and B. Thus asking &quot;are the effects different?&quot; is foolish.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Tukey (again but this one is my favorite)&lt;/p&gt;&#xA;" OwnerUserId="223" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-07-27T06:42:07.557" LastActivityDate="2010-07-27T06:42:07.557" CommunityOwnedDate="2010-07-27T06:26:04.483" CommentCount="2" />
  <row Id="729" PostTypeId="2" ParentId="726" CreationDate="2010-07-27T06:36:26.867" Score="35" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;In God we trust. All others must bring&#xA;  data. &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;(W. Edwards Deming)&lt;/p&gt;&#xA;" OwnerUserId="159" LastEditorUserId="461" LastEditorDisplayName="" LastEditDate="2010-07-31T00:19:53.500" LastActivityDate="2010-07-31T00:19:53.500" CommunityOwnedDate="2010-07-27T06:36:26.867" CommentCount="3" />
  <row Id="730" PostTypeId="2" ParentId="726" CreationDate="2010-07-27T06:37:30.157" Score="60" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;All models are wrong, but some are useful. (George E. P. Box)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="159" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-07-28T06:55:15.623" LastActivityDate="2010-07-28T06:55:15.623" CommunityOwnedDate="2010-07-27T06:37:30.157" CommentCount="4" />
  <row Id="731" PostTypeId="2" ParentId="645" CreationDate="2010-07-27T06:45:31.107" Score="2" ViewCount="" Body="&lt;p&gt;If you're looking at system faults, you might be interested in the following paper employing machine learning techniques for fault diagnosis at eBay.  It may give you a sense of what kind of data to collect or how one team approached a specific problem in a similar domain.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.cs.berkeley.edu/~brewer/papers/icac2004_chen_diagnosis.pdf&quot; rel=&quot;nofollow&quot;&gt;Fault Diagnosis Using Decision Trees&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If you're just getting started, something like &lt;a href=&quot;http://rapid-i.com/content/view/181/190/&quot; rel=&quot;nofollow&quot;&gt;RapidMiner&lt;/a&gt; or &lt;a href=&quot;http://www.ailab.si/orange/&quot; rel=&quot;nofollow&quot;&gt;Orange&lt;/a&gt; might be a good software system to start playing with your data pretty quickly.  Both of them can access the data in a variety of formats (file csv, database, among others).  &lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-07-27T06:45:31.107" CommentCount="2" />
  <row Id="732" PostTypeId="2" ParentId="726" CreationDate="2010-07-27T06:51:24.730" Score="31" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Strange events permit themselves the&#xA;  luxury of occurring.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;-- &lt;a href=&quot;http://gutenberg.net.au/ebooks02/0200691.txt&quot; rel=&quot;nofollow&quot;&gt;Charlie Chan&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="251" LastEditorUserId="380" LastEditorDisplayName="" LastEditDate="2010-08-07T10:08:04.317" LastActivityDate="2010-08-07T10:08:04.317" CommunityOwnedDate="2010-07-27T06:51:24.730" CommentCount="3" />
  <row Id="733" PostTypeId="2" ParentId="723" CreationDate="2010-07-27T06:52:09.080" Score="2" ViewCount="" Body="&lt;p&gt;There is something like &lt;a href=&quot;http://en.wikipedia.org/wiki/Silhouette_%28clustering%29&quot; rel=&quot;nofollow&quot;&gt;silhouette&lt;/a&gt;, which to some extent defines statistic that determines the cluster quality (for instance it is used in optimizing k). Now a possible Monte Carlo would go as follows: you generate a lot of random dataset similar to your original (for instance by shuffling values between rows in each column), cluster and obtain a distribution of mean silhouette that then may be used to test significance of silhouette in real data. Still I admin that I have never tried this idea.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-07-27T06:52:09.080" CommentCount="2" />
  <row Id="734" PostTypeId="2" ParentId="715" CreationDate="2010-07-27T06:55:06.470" Score="2" ViewCount="" Body="&lt;p&gt;Maybe the good old &lt;a href=&quot;http://archive.ics.uci.edu/ml/datasets/Iris&quot; rel=&quot;nofollow&quot;&gt;iris&lt;/a&gt;? It suits your needs and is good for start. &lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-07-27T06:55:06.470" CommentCount="3" />
  <row Id="735" PostTypeId="2" ParentId="645" CreationDate="2010-07-27T07:19:13.377" Score="0" ViewCount="" Body="&lt;p&gt;The one thing &lt;a href=&quot;http://root.cern.ch&quot; rel=&quot;nofollow&quot;&gt;ROOT&lt;/a&gt; is really good at is storing enourmous amounts of data. ROOT is a C++ library used in particle physics; it also comes with Ruby and Python bindings, so you could use packages in these languages (e.g. NumPy or Scipy) to analyze the data when you find that ROOT offers to few possibilities out-of-the-box.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The ROOT fileformat can store trees or tuples, and entries can be read sequentially, so you do not need to keep all data in memory at the same time. This allows to analyze petabytes of data, something you wouldn't want to try with Excel or R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The ROOT I/O documentation can be reached from &lt;a href=&quot;http://root.cern.ch/drupal/content/root-files-1&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="56" LastActivityDate="2010-07-27T07:19:13.377" />
  <row Id="736" PostTypeId="2" ParentId="73" CreationDate="2010-07-27T07:24:19.963" Score="3" ViewCount="" Body="&lt;p&gt;zoo and xts are a must in my work!&lt;/p&gt;&#xA;" OwnerUserId="300" LastActivityDate="2010-07-27T07:24:19.963" CommunityOwnedDate="2010-07-27T07:24:19.963" />
  <row Id="737" PostTypeId="2" ParentId="726" CreationDate="2010-07-27T07:26:34.187" Score="20" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;There are no routine statistical&#xA;  questions, only questionable&#xA;  statistical routines.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;D.R. Cox&lt;/p&gt;&#xA;" LastEditorUserId="380" LastEditorDisplayName="" LastEditDate="2010-08-07T10:09:08.980" LastActivityDate="2010-08-07T10:09:08.980" CommunityOwnedDate="2010-07-27T07:26:34.187" CommentCount="1" />
  <row Id="738" PostTypeId="2" ParentId="726" CreationDate="2010-07-27T07:38:19.743" Score="24" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Say you were standing with one foot in the oven and one foot in an ice bucket.  According to the percentage people, you should be perfectly comfortable.  &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;-Bobby Bragan, 1963&lt;/p&gt;&#xA;" OwnerUserId="188" LastEditorUserId="795" LastEditorDisplayName="" LastEditDate="2010-12-03T04:00:23.143" LastActivityDate="2010-12-03T04:00:23.143" CommunityOwnedDate="2010-07-27T07:38:19.743" />
  <row Id="739" PostTypeId="2" ParentId="726" CreationDate="2010-07-27T07:41:56.333" Score="40" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;&quot;To call in the statistician after the experiment is done may be no more than asking him to perform a post-mortem examination: he may be able to say what the experiment died of.&quot;&lt;br&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;-- Ronald Fisher (1938)&lt;/p&gt;&#xA;" LastActivityDate="2010-07-29T07:26:45.607" CommunityOwnedDate="2010-07-27T07:41:56.333" CommentCount="1" />
  <row Id="740" PostTypeId="1" CreationDate="2010-07-27T07:54:02.530" Score="0" ViewCount="96" Body="&lt;p&gt;I'm currently doing some transitional modelling using a first order markov transitional model to see how patients transition between 2 pre-defined states e.g. Disability and No-disability. I understand that with a markov transitional model, the transition from one state to another at ANY time point (say T) is only dependent upon the previous state (T-1). The model therefore includes a variable that adjusts for a patients previous state.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've fitted the model for this and obtained the odds ratio (OR) for the transition at the time point of interest; this is fine and can be interpretted. As i mentioned before, the model adjusts for a patients previous state and this variable also has an (OR) in the model output however i am not sure how to interpret this odds ratio? Anyone know how this is interpretted?&lt;/p&gt;&#xA;" LastActivityDate="2010-10-25T09:33:57.093" ClosedDate="2010-10-25T10:38:48.963" Title="Interpretation of coefficients in a first order Markov Transitional model" Tags="&lt;modeling&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="741" PostTypeId="2" ParentId="692" CreationDate="2010-07-27T07:56:42.460" Score="4" ViewCount="" Body="&lt;p&gt;The basic way to see if your data is Weibull is to &lt;a href=&quot;http://www.itl.nist.gov/div898/handbook/eda/section3/weibplot.htm&quot; rel=&quot;nofollow&quot;&gt;plot&lt;/a&gt; the log of cumulative hazards versus log of times and see if a straight line might be a good fit.  The cumulative hazard can be found using the non-parametric Nelson-Aalen estimator.  There are similar graphical &lt;a href=&quot;http://www.weibull.com/hotwire/issue71/relbasics71.htm&quot; rel=&quot;nofollow&quot;&gt;diagnostics&lt;/a&gt; for Weibull regression if you fit your data with covariates and some references follow.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;a href=&quot;http://www.powells.com/biblio/72-9780387239187-0&quot; rel=&quot;nofollow&quot;&gt;Klein &amp;amp; Moeschberger&lt;/a&gt; text is pretty good and covers a lot of ground with model building/diagnostics for parametric and semi-parametric models (though mostly the latter).  If you're working in R, &lt;a href=&quot;http://www.powells.com/biblio/61-9780387987842-1&quot; rel=&quot;nofollow&quot;&gt;Theneau's book&lt;/a&gt; is pretty good (I believe he wrote the &lt;a href=&quot;http://cran.r-project.org/web/views/Survival.html&quot; rel=&quot;nofollow&quot;&gt;survival&lt;/a&gt; package).  It covers a lot of Cox PH and associated models, but I don't recall if it has much coverage of parametric models, like the one you're building.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;BTW, is this a million subjects each with one entry/exit or recurrent entry/exit events for some smaller pool of people? Are you conditioning your likelihood to account for the censoring mechanism?  &lt;/p&gt;&#xA;" OwnerUserId="251" LastEditorUserId="251" LastEditorDisplayName="" LastEditDate="2010-07-27T09:45:01.060" LastActivityDate="2010-07-27T09:45:01.060" CommentCount="1" />
  <row Id="742" PostTypeId="2" ParentId="740" CreationDate="2010-07-27T08:07:52.573" Score="0" ViewCount="" Body="&lt;p&gt;What do you want to analyze with this model? Currently as you have made it, you can insert a sequence and see whether it comes from this model or not (the odds ratio).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You probably want to predict how the patient develops further. For that you don't need the odds ratio, but basic inference. See also &lt;a href=&quot;http://stats.stackexchange.com/questions/165/how-would-you-explain-markov-chain-monte-carlo-mcmc-to-a-layperson&quot; rel=&quot;nofollow&quot;&gt;this question&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="190" LastActivityDate="2010-07-27T08:07:52.573" />
  <row Id="743" PostTypeId="1" AcceptedAnswerId="839" CreationDate="2010-07-27T08:13:26.197" Score="4" ViewCount="164" Body="&lt;p&gt;I was having a look round a few things yesturday and came across &lt;a href=&quot;http://en.wikipedia.org/wiki/Bayesian_search_theory&quot; rel=&quot;nofollow&quot;&gt;Bayesian Search Theory&lt;/a&gt;. Thinking about this theory led me to think about a problem I was working on a few years ago regarding geological interpretation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We were looking at the geology at one specific site and it was essentially made up from two different types of rocks. Boreholes had been drilled at different locations and showed differing amounts of the two different types of rocks at different levels in the ground along with different amounts of weathering of the rock. A number of geologists looked at the available data and all came up with different interpretations. It seems to me that Bayesian Search Theory could have been used in this case, particualrly where extra data was gathered with time, to give some indication of how likely the different interpretations were. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Has anyone encountered a case where Bayesian Search Theory has been used in this case. Is there a standard frameowrk for doing this? I would have thought this may be something that the oil industry may have a lot of research on because it would be applicable to the search for oil. &lt;/p&gt;&#xA;" OwnerUserId="210" LastActivityDate="2010-07-27T22:33:05.287" Title="Use of Bayesian Search Theory in geological interpretation" Tags="&lt;bayesian&gt;&lt;search-theory&gt;" AnswerCount="2" />
  <row Id="744" PostTypeId="2" ParentId="726" CreationDate="2010-07-27T08:42:23.037" Score="46" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;&quot;An approximate answer to the right problem is worth a good deal more than an exact answer to an approximate problem.&quot; -- John Tukey&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="319" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-07-28T06:55:37.570" LastActivityDate="2010-07-28T06:55:37.570" CommunityOwnedDate="2010-07-27T08:42:23.037" CommentCount="5" />
  <row Id="745" PostTypeId="2" ParentId="73" CreationDate="2010-07-27T08:42:25.907" Score="1" ViewCount="" Body="&lt;p&gt;I use &lt;/p&gt;&#xA;&#xA;&lt;p&gt;car, doBy, Epi, ggplot2, gregmisc (gdata, gmodels, gplots, gtools), Hmisc, plyr, RCurl, RDCOMClient, reshape, RODBC, TeachingDemos, XML.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;a lot.&lt;/p&gt;&#xA;" LastActivityDate="2010-07-27T08:42:25.907" CommunityOwnedDate="2010-07-27T08:42:25.907" />
  <row Id="746" PostTypeId="2" ParentId="726" CreationDate="2010-07-27T08:43:15.033" Score="3" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;efficiency = statistical efficiency x usage.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;-- John Tukey&lt;/p&gt;&#xA;" OwnerUserId="319" LastEditorUserId="795" LastEditorDisplayName="" LastEditDate="2010-12-03T04:03:20.240" LastActivityDate="2010-12-03T04:03:20.240" CommunityOwnedDate="2010-07-27T08:43:15.033" />
  <row Id="747" PostTypeId="2" ParentId="73" CreationDate="2010-07-27T09:00:42.587" Score="5" ViewCount="" Body="&lt;p&gt;Packages I often use are &lt;a href=&quot;http://cran.r-project.org/web/packages/raster/index.html&quot; rel=&quot;nofollow&quot;&gt;raster&lt;/a&gt;, &lt;a href=&quot;http://cran.r-project.org/web/packages/sp/index.html&quot; rel=&quot;nofollow&quot;&gt;sp&lt;/a&gt;, &lt;a href=&quot;http://cran.r-project.org/web/packages/spatstat/index.html&quot; rel=&quot;nofollow&quot;&gt;spatstat&lt;/a&gt;, &lt;a href=&quot;http://cran.r-project.org/package=vegan&quot; rel=&quot;nofollow&quot;&gt;vegan&lt;/a&gt; and &lt;a href=&quot;http://cran.r-project.org/web/packages/splancs/index.html&quot; rel=&quot;nofollow&quot;&gt;splancs&lt;/a&gt;. I sometimes use ggplot2, tcltk and lattice.&lt;/p&gt;&#xA;" OwnerUserId="144" LastActivityDate="2010-07-27T09:00:42.587" CommunityOwnedDate="2010-07-27T09:00:42.587" />
  <row Id="748" PostTypeId="2" ParentId="726" CreationDate="2010-07-27T09:09:25.517" Score="16" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;A big computer, a complex algorithm and a long time does not equal science.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;-- Robert Gentleman&lt;/p&gt;&#xA;" OwnerUserId="434" LastEditorUserId="795" LastEditorDisplayName="" LastEditDate="2010-10-02T17:09:45.723" LastActivityDate="2010-10-02T17:09:45.723" CommunityOwnedDate="2010-07-27T09:09:25.517" CommentCount="1" />
  <row Id="749" PostTypeId="2" ParentId="723" CreationDate="2010-07-27T09:10:20.163" Score="7" ViewCount="" Body="&lt;p&gt;Regarding shopping cart analysis, I think that the main objective is to individuate the most frequent combinations of products bought by the customers. The &lt;code&gt;association rules&lt;/code&gt; represent the most natural methodology here (indeed they were actually developed for this purpose). Analysing the combinations of products bought by the customers, and the number of times these combinations are repeated, leads to a rule of the type ‘if condition, then result’ with a corresponding interestingness measurement. You may also consider &lt;code&gt;Log-linear models&lt;/code&gt; in order to investigate the associations between the considered variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now as for clustering, here are some information that may come in handy:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At first consider &lt;code&gt;Variable clustering&lt;/code&gt;. Variable clustering is used for assessing collinearity, redundancy, and for separating variables into clusters that can be scored as a single variable, thus resulting in data reduction. Look for the &lt;code&gt;varclus&lt;/code&gt; function (package Hmisc in R)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assessment of the clusterwise stability: function &lt;code&gt;clusterboot&lt;/code&gt; {R package  fpc}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Distance based statistics for cluster validation: function &lt;code&gt;cluster.stats&lt;/code&gt; {R package  fpc}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As mbq have mentioned, use the silhouette widths for assessing the best number of clusters. Watch &lt;a href=&quot;http://www.google.com/codesearch/p?hl=en#sTQFIWS4uR8/afs/sipb/project/r-project/arch/sun4x_510/lib/R/library/cluster/R-ex/pam.object.R&amp;amp;q=lang%3ar%20%22optimal%20number%20of%20clusters%22&amp;amp;sa=N&amp;amp;cd=6&amp;amp;ct=rc&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt;. Regarding silhouette widths, see also the &lt;a href=&quot;http://finzi.psych.upenn.edu/R/library/optpart/html/optsil.html&quot; rel=&quot;nofollow&quot;&gt;optsil&lt;/a&gt; function. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Estimate the number of clusters in a data set via the &lt;a href=&quot;http://finzi.psych.upenn.edu/R/library/clusterSim/html/index.GAP.html&quot; rel=&quot;nofollow&quot;&gt;gap statistic&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For calculating Dissimilarity Indices and Distance Measures see &lt;a href=&quot;http://finzi.psych.upenn.edu/R/library/labdsv/html/dsvdis.html&quot; rel=&quot;nofollow&quot;&gt;dsvdis&lt;/a&gt; and &lt;a href=&quot;http://finzi.psych.upenn.edu/R/library/vegan/html/vegdist.html&quot; rel=&quot;nofollow&quot;&gt;vegdist&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EM clustering algorithm can decide how many clusters to create by cross validation, (if you can't specify apriori how many clusters to generate). &lt;em&gt;Although the EM algorithm is guaranteed to converge to a maximum, this is a local maximum and may not necessarily be the same as the global maximum. For a better chance of obtaining the global maximum, the whole procedure should be repeated several times, with different initial guesses for the parameter values. The overall log-likelihood figure can be used to compare the different final configurations obtained: just choose the largest of the local maxima&lt;/em&gt;.&#xA;You can find an implementation of the EM clusterer in the open-source project &lt;a href=&quot;http://www.cs.waikato.ac.nz/~ml/weka/&quot; rel=&quot;nofollow&quot;&gt;WEKA&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://zoonek2.free.fr/UNIX/48_R/06.html&quot; rel=&quot;nofollow&quot;&gt;This&lt;/a&gt; is also an interesting link.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also search &lt;a href=&quot;http://www.statsoft.com/textbook/cluster-analysis/#vfold&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; for &lt;code&gt;Finding the Right Number of Clusters in k-Means and EM Clustering: v-Fold Cross-Validation&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, you may explore clustering results using &lt;a href=&quot;http://had.co.nz/model-vis/&quot; rel=&quot;nofollow&quot;&gt;clusterfly&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="339" LastEditorUserId="339" LastEditorDisplayName="" LastEditDate="2010-07-27T14:44:54.903" LastActivityDate="2010-07-27T14:44:54.903" CommentCount="2" />
  <row Id="750" PostTypeId="2" ParentId="726" CreationDate="2010-07-27T09:16:48.383" Score="10" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;There are three kinds of lies: lies,&#xA;  damned lies, and statistics.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;-- &lt;a href=&quot;https://secure.wikimedia.org/wikipedia/en/wiki/Lies,_damned_lies,_and_statistics&quot; rel=&quot;nofollow&quot;&gt;probably: Charles Wentworth Dilke (1843–1911).&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="127" LastActivityDate="2010-07-27T09:16:48.383" CommunityOwnedDate="2010-07-27T09:16:48.383" CommentCount="5" />
  <row Id="751" PostTypeId="2" ParentId="726" CreationDate="2010-07-27T09:19:53.223" Score="3" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;The Median Isn't the Message&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;--&lt;a href=&quot;http://cancerguide.org/median_not_msg.html&quot; rel=&quot;nofollow&quot;&gt;Stephen Jay Gould&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="127" LastActivityDate="2010-07-27T09:19:53.223" CommunityOwnedDate="2010-07-27T09:19:53.223" />
  <row Id="752" PostTypeId="2" ParentId="726" CreationDate="2010-07-27T09:20:27.967" Score="21" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Figures don't lie, but liars do figure&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;--Mark Twain&lt;/p&gt;&#xA;" OwnerUserId="127" LastActivityDate="2010-07-27T09:20:27.967" CommunityOwnedDate="2010-07-27T09:20:27.967" CommentCount="5" />
  <row Id="753" PostTypeId="2" ParentId="726" CreationDate="2010-07-27T09:22:04.753" Score="35" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Statistics are like bikinis.  What&#xA;  they reveal is suggestive, but what&#xA;  they conceal is vital.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;--Aaron Levenstein&lt;/p&gt;&#xA;" OwnerUserId="127" LastActivityDate="2010-07-27T09:22:04.753" CommunityOwnedDate="2010-07-27T09:22:04.753" />
  <row Id="754" PostTypeId="2" ParentId="726" CreationDate="2010-07-27T09:25:00.340" Score="2" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;The mathematician, carried along on his flood of symbols, dealing apparently with purely formal thruths, may still reach results of endless importance for our description of physical universe&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;-- Karl Pearson&lt;/p&gt;&#xA;" OwnerUserId="223" LastEditorUserId="795" LastEditorDisplayName="" LastEditDate="2010-10-02T17:11:34.173" LastActivityDate="2010-10-02T17:11:34.173" CommunityOwnedDate="2010-07-27T09:25:00.340" CommentCount="1" />
  <row Id="755" PostTypeId="2" ParentId="726" CreationDate="2010-07-27T09:44:12.930" Score="19" ViewCount="" Body="&lt;p&gt;I don't know about famous, but the following is one of my favourites:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Conducting data analysis is like&#xA;  drinking a fine wine. It is important&#xA;  to swirl and sniff the wine, to unpack&#xA;  the complex bouquet and to appreciate&#xA;  the experience. Gulping the wine&#xA;  doesn’t work.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;-Daniel B. Wright (2003), see &lt;a href=&quot;http://www2.fiu.edu/~dwright/pdf/makefriends.pdf&quot; rel=&quot;nofollow&quot;&gt;PDF of Article&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-07-28T09:01:45.693" LastActivityDate="2010-07-28T09:01:45.693" CommunityOwnedDate="2010-07-27T09:44:12.930" CommentCount="1" />
  <row Id="757" PostTypeId="2" ParentId="73" CreationDate="2010-07-27T09:51:19.900" Score="2" ViewCount="" Body="&lt;p&gt;I find &lt;a href=&quot;http://cran.r-project.org/web/packages/lattice/index.html&quot; rel=&quot;nofollow&quot;&gt;lattice&lt;/a&gt; along with the companion book &quot;Lattice: Multivariate Data Visualization with R&quot; by Deepayan Sarkar invaluable.&lt;/p&gt;&#xA;" OwnerUserId="439" LastActivityDate="2010-07-27T09:51:19.900" CommunityOwnedDate="2010-07-27T09:51:19.900" />
  <row Id="758" PostTypeId="2" ParentId="723" CreationDate="2010-07-27T10:00:23.720" Score="5" ViewCount="" Body="&lt;p&gt;This is a method using Monte Carlo to show whether a result is correct.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Our Null Hypothesis H_0 is that our dataset does not have an interesting clustering. Our alternative hypothesis H_1 is that our dataset contains an interesting clustering.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hereby we think of interesting as, more interesting than the clustering structure of a random dataset with the same row and column margins. Of course other constraints could be chosen, but to loose constraints will make our result too general, and to narrow constraints will fix the clustering to much, therefore making our result insignificant automatically. The margins, as we will see, are a good choice because of the methods existing for randomizing with it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's define as our test statistic the clustering error (squared in-cluster distance), T from Π_0. The value for our original dataset is &lt;em&gt;t&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We don't know anything about this distribution, except that we can draw samples from it. Which makes it a good candidate for Monte Carlo.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now we draw &lt;em&gt;n&lt;/em&gt; (i.i.d) random samples from Π_0 and calculate the empirical &lt;em&gt;p&lt;/em&gt;-value with the formula p_emp = 1 / (n+1) * (Σ_i=1-n I(t_i &gt;= t)  + 1)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The random sampling can be done by swap randomization. In simple words, a square is searched with on two opposite corners an 1 and on the other two corners a 0. Then the corners are flipped. This is keeping the column and row margins. The procedure is repeated enough times until the dataset is randomized enough (this will take some experiments). More info about this can be found in &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.76.3286&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;Assessing Data Mining Results via Swap Randomization by Gionis et. al.&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One method to do this is defining the distribution of your data and taking the clustering error as test-statistic t.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if we consider all data sets with the same row and column margins as being our data distribution, than we can take n random matrices Xi from this distribution and calculate the clustering error for them. Then we can calculate the emperical p-value by the formula &lt;/p&gt;&#xA;" OwnerUserId="190" LastActivityDate="2010-07-27T10:00:23.720" />
  <row Id="759" PostTypeId="2" ParentId="725" CreationDate="2010-07-27T10:07:42.560" Score="3" ViewCount="" Body="&lt;p&gt;I am afraid there is no; during my little adventure with such data we have just converted it to a data frame form, added some extra attributes made from neighborhoods of pixels and used standard methods. Still, packages &lt;a href=&quot;http://cran.r-project.org/web/packages/ripa/index.html&quot; rel=&quot;nofollow&quot;&gt;ripa&lt;/a&gt; and &lt;a href=&quot;http://cran.r-project.org/web/packages/hyperSpec/index.html&quot; rel=&quot;nofollow&quot;&gt;hyperSpec&lt;/a&gt; might be useful.&lt;br&gt;&#xA;For other software, I've got an impression that most of sensible applications are commercial.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-07-27T10:07:42.560" CommentCount="2" />
  <row Id="760" PostTypeId="2" ParentId="575" CreationDate="2010-07-27T10:24:44.870" Score="8" ViewCount="" Body="&lt;p&gt;I am currently writing a paper in which I have the pleasure to conduct both between and within subjects comparisons. After discussion with my supervisor we decided to run &lt;em&gt;t&lt;/em&gt;-tests and use the pretty simple &lt;code&gt;Holm-Bonferroni method&lt;/code&gt; (&lt;a href=&quot;http://en.wikipedia.org/wiki/Holm%E2%80%93Bonferroni_method&quot; rel=&quot;nofollow&quot;&gt;wikipedia&lt;/a&gt;) for correcting for alpha error cumulation. It controls for familwise error rate but has a greater power than the ordinary Bonferroni procedure.&#xA;Procedure:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;You run the &lt;em&gt;t&lt;/em&gt;-tests for all comparisons you want to do.&lt;/li&gt;&#xA;&lt;li&gt;You order the &lt;em&gt;p&lt;/em&gt;-values according to their value.&lt;/li&gt;&#xA;&lt;li&gt;You test the smallest &lt;em&gt;p&lt;/em&gt;-value against &lt;em&gt;alpha&lt;/em&gt; / &lt;em&gt;k&lt;/em&gt;, the second smallest against &lt;em&gt;alpha&lt;/em&gt; /( &lt;em&gt;k&lt;/em&gt; - 1), and so forth until the first test turns out non-significant in this sequence of tests.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Cite Holm (1979) which can be downloaded via the link at &lt;a href=&quot;http://en.wikipedia.org/wiki/Holm%E2%80%93Bonferroni_method&quot; rel=&quot;nofollow&quot;&gt;wikipedia&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="442" LastEditorUserId="442" LastEditorDisplayName="" LastEditDate="2010-07-27T13:08:54.103" LastActivityDate="2010-07-27T13:08:54.103" />
  <row Id="761" PostTypeId="2" ParentId="665" CreationDate="2010-07-27T10:26:36.600" Score="5" ViewCount="" Body="&lt;p&gt;I like this from Steve Skienna's &lt;a href=&quot;http://books.google.com/books?id=UvWGgaE4ZA8C&amp;amp;lpg=PA86&amp;amp;ots=RXQyDddYcu&amp;amp;dq=In%20summary%2C%20probability%20theory%20enables%20us%20to%20find%20the%20consequences%20of%20a%20given%20ideal%20world%2C%20while%20statistical%20theory%20enables%20us%20to%20to%20measure%20the%20extent%20to%20which%20our%20world%20is%20ideal.&amp;amp;pg=PA86#v=onepage&amp;amp;q&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;Calculated Bets&lt;/a&gt; (see the link for complete discussion):&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In summary, probability theory enables us to find the consequences of a given ideal world, while statistical theory enables us to to measure the extent to which our world is ideal.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-07-27T10:26:36.600" />
  <row Id="762" PostTypeId="2" ParentId="575" CreationDate="2010-07-27T11:04:03.853" Score="3" ViewCount="" Body="&lt;p&gt;Have a look at the &lt;a href=&quot;http://cran.r-project.org/web/packages/multcomp/index.html&quot; rel=&quot;nofollow&quot;&gt;multcomp&lt;/a&gt;-package and its vignette &lt;a href=&quot;http://cran.r-project.org/web/packages/multcomp/vignettes/generalsiminf.pdf&quot; rel=&quot;nofollow&quot;&gt;Simultaneous Inference in General Parametric Models&lt;/a&gt;. I think it should do what wan't and the vignette has very good examples and extensive references.&lt;/p&gt;&#xA;" OwnerUserId="214" LastActivityDate="2010-07-27T11:04:03.853" />
  <row Id="763" PostTypeId="2" ParentId="562" CreationDate="2010-07-27T11:04:08.207" Score="4" ViewCount="" Body="&lt;p&gt;The final year of the NetFlix Prize competition (2009) seemed to me to have sharply changed the general community-wide presumption against combining multiple learning algorithms. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, my formal training (university courses) and later on-the-job oversight/mentoring taught us to avoid algorithm combination unless we had an explicit reason to do so--and &quot;to improve resolution of my current algorithm&quot;, wasn't really deemed a good reason. (Others might have a different experience--of course i'm inferring a community-wide view based solely on my own experience, though my experience in coding poorly-performing ML algorithms is substantial.) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Still, there were a few &quot;patterns&quot; in which combining algorithms in one way or another was accepted, and actually improved performance. For me, the most frequent example involved some ML algorithm configured in machine mode (assigning a class label to each data point) and in which there were more than two classes (usually many more). When for instance, using a supervised-learning algorithm  to resolve four classes, and we would see excellent separation &lt;em&gt;except for&lt;/em&gt; let's say Class III versus Class IV. So out of those six decision boundaries, only one resolved below the required threshold. Particularly when classes III and IV together accounted for a small percent of the data, adding an additional algorithm &lt;em&gt;optimized just on the resolution of those two classes&lt;/em&gt;, was a fairly common solution to this analytical problem type. (Usually that 'blind spot' was an inherent limitation of the primary algorithm--e.g., it was a linear classifier and the III/IV decision boundary was non-linear. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words, when we had a reliable algorithm suited to the processing environment (which was usually streaming data) and that performed within the spec except for a single blind spot that caused it to fail to resolve two (or more) classes that accounted for a small fraction of the data, then it was always better to 'bolt-on' another specialized algorithm to catch what the main algorithm was systematically missing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lastly, on this topic, i would like to recommend highly Chapter 17, &lt;em&gt;Combining Multiple Learners&lt;/em&gt;, in &lt;strong&gt;&lt;em&gt;&lt;a href=&quot;http://www.cmpe.boun.edu.tr/~ethem/i2ml2e/&quot; rel=&quot;nofollow&quot;&gt;Introduction to Machine Learning&lt;/a&gt;&lt;/em&gt;&lt;/strong&gt;, 2d, by Ethem Alpaydin, MIT Press, 2010. Note that this is the &lt;em&gt;second edition&lt;/em&gt; published a few months ago; the first edition was published in 2004 and i doubt it has the same coverage of this topic. (Actually i recommend the entire text, but that chapter in particular since it relates to Shane's Question.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In 25 pages, the author summarizes probably every ML algorithm-combination scheme whose utility has been demonstrated in the academic literature or practice--e.g., bagging, boosting, mixture of experts, stacked generalization, cascading, voting, error-correcdting,....&lt;/p&gt;&#xA;" OwnerUserId="438" LastEditorUserId="438" LastEditorDisplayName="" LastEditDate="2010-08-04T07:39:52.463" LastActivityDate="2010-08-04T07:39:52.463" CommentCount="2" />
  <row Id="764" PostTypeId="1" AcceptedAnswerId="772" CreationDate="2010-07-27T11:29:20.193" Score="9" ViewCount="381" Body="&lt;p&gt;I have commonly heard that LME models are more sound in the analysis of accuracy data (i.e., in psychology experiments), in that they can work with binomial and other non-normal distributions that traditional approaches (e.g., ANOVA) can't.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the mathematical basis of LME models that allow them to incorporate these other distributions, and what are some not-overly-technical papers describing this?&lt;/p&gt;&#xA;" OwnerUserId="445" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-08-07T17:52:12.747" LastActivityDate="2010-08-07T17:52:12.747" Title="Linear Mixed Effects Models" Tags="&lt;mixed-model&gt;" AnswerCount="3" FavoriteCount="9" />
  <row Id="765" PostTypeId="2" ParentId="283" CreationDate="2010-07-27T11:56:42.250" Score="3" ViewCount="" Body="&lt;p&gt;A model is saturated if and only if it has as many parameters as it has data points (observations). Or put otherwise, in non-saturated models the degrees of freedom are bigger than zero.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This basically means that this model is useless, because it does not describe the data more parsimoniously than the raw data does (and describing data parsimoniously is generally the idea behind using a model). Furthermore, saturated models can (but don't necessarily) provide a (useless) perfect fit because they just interpolate or iterate the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Take for example the mean as a model for some data. If you have only one data point (e.g., 5) using the mean (i.e., 5; note that the mean is a saturated model for only one data point) does not help at all. However if you already have two data points (e.g., 5 and 7) using the mean (i.e., 6) as a model provides you with a more  parsimonious description than the original data.&lt;/p&gt;&#xA;" OwnerUserId="442" LastEditorUserId="442" LastEditorDisplayName="" LastEditDate="2010-07-27T12:32:15.423" LastActivityDate="2010-07-27T12:32:15.423" />
  <row Id="766" PostTypeId="2" ParentId="270" CreationDate="2010-07-27T12:10:34.937" Score="2" ViewCount="" Body="&lt;p&gt;Most statistical packages have a function to calculate the natural logarithm of the factorial directly (e.g. the lfactorial() function in R, the lnfactorial() function in Stata). This allows you to include the constant term in the log-likelihood if you want.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-07-27T12:10:34.937" CommentCount="1" />
  <row Id="767" PostTypeId="2" ParentId="577" CreationDate="2010-07-27T12:31:57.280" Score="25" ViewCount="" Body="&lt;p&gt;Your question implies that AIC and BIC try to answer the same question, which is not true.&#xA;AIC tries to select the model that most adequately describes an unknown, high dimensional reality. This means that reality is never in the set of candidate models that are being considered.  On the contrary, BIC tries to find the TRUE model among the set of candidates. I find it quite odd the assumption that reality is instantiated in one of the model that the researchers built along the way. This is a real issue for BIC.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Nevertheless, there are a lot of researchers who say BIC is better than AIC, using model recovery simulations as an argument. These simulations consist of generating data from models A and B, and then fitting both datasets with the two models. Overfitting occurs when the wrong model fits the data better than the generating. The point of these simulations is to see how well AIC and BIC correct these overfits. Usually, the results point to the fact that AIC is too liberal and still frequently prefers a more complex, wrong model over a simpler, true model. At first glance these simulations seem to be really good arguments, but the problem with them is that they are meaningless for AIC. As I said before, AIC does not consider that any of the candidate models being tested is actually true. According to AIC, all models are approximations to reality, and reality should never have a low dimensionality. At least lower than some of the candidate models. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;my recommendation: use both AIC and BIC. Most of the times they will agree on the preferred model, when they dont, just report it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are unhappy with both AIC and BIC, and you have free time to invest, look up for Minimum Description Length (MDL), a totally different approach that overcomes the limitations of AIC and BIC. There are several measures stemming from MDL, like normalized maximum likelihood or the Fisher Information approximation. The problem with MDL is that its mathematically demanding and/or computationally intensive. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Still, if you wanna stick to simple solutions, a nice way for assessing model flexibility (especially when the number of parameters are equal, rendering AIC and BIC useless) is doing Parametric Bootstrap, which is quite easy to implement. here is a link to a paper on it:&#xA;&lt;a href=&quot;http://www.ejwagenmakers.com/2004/PBCM.pdf&quot; rel=&quot;nofollow&quot;&gt;link text&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;some people here advocate the use of cross-validation. I personally have used it, and dont have anything against it, but the issue with it is that the choice among the sample-cutting rule (leave-one-out, K-fold, etc) is an unprincipled one. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;cheers&lt;/p&gt;&#xA;" OwnerUserId="447" LastEditorUserId="447" LastEditorDisplayName="" LastEditDate="2010-07-27T12:39:36.703" LastActivityDate="2010-07-27T12:39:36.703" CommentCount="3" />
  <row Id="768" PostTypeId="2" ParentId="665" CreationDate="2010-07-27T12:36:47.457" Score="2" ViewCount="" Body="&lt;p&gt;In probability theory, we are given random variables X1, X2, ... in some way, and then we study their properties, i.e. calculate probability P{ X1 \in B1 }, study the convergence of X1, X2, ... etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In mathematical statistics, we are given n realizations of some random variable X, and set of distributions D; the problem is to find amongst distributions from D one which is most likely to generate the data we observed. &lt;/p&gt;&#xA;" LastActivityDate="2010-07-27T12:36:47.457" CommentCount="1" />
  <row Id="769" PostTypeId="1" CreationDate="2010-07-27T12:46:48.420" Score="0" ViewCount="131" Body="&lt;p&gt;When does data analysis cease to be statistics ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are the following examples all applications of statistics ?: computer vision, face recognition, compressed sensing, lossy data compression, signal processing.&lt;/p&gt;&#xA;" OwnerUserId="327" LastActivityDate="2010-07-27T16:00:22.087" Title="What types of data analysis do not count as statistics?" Tags="&lt;theory&gt;" AnswerCount="2" CommentCount="4" />
  <row Id="770" PostTypeId="1" CreationDate="2010-07-27T12:53:30.250" Score="5" ViewCount="223" Body="&lt;p&gt;During every machine learning tutorial you'll find, there is the common &quot;You will need to know x amount of stats before starting this tutorial&quot;.  As such, using your knowledge of stats, you will learn about machine learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is whether this can be reversed.  Can a computer science student learn statistics through studying machine learning algorithms?  Has this been tested, at all?  Are there examples where this is the case already?&lt;/p&gt;&#xA;" OwnerUserId="453" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-17T20:31:31.940" LastActivityDate="2011-02-14T23:04:02.267" Title="Is it possible to use machine learning as a method for learning stats, rather than vice-versa?" Tags="&lt;machine-learning&gt;&lt;teaching&gt;" AnswerCount="5" />
  <row Id="771" PostTypeId="2" ParentId="770" CreationDate="2010-07-27T13:02:58.047" Score="0" ViewCount="" Body="&lt;p&gt;I think that learning machine learning requires only an elementary subset of statistics; too much may be dangerous, since some intuitions are in conflict. Still, the answer to the question can it be reversed is no.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-07-27T13:02:58.047" />
  <row Id="772" PostTypeId="2" ParentId="764" CreationDate="2010-07-27T13:07:38.643" Score="7" ViewCount="" Body="&lt;p&gt;One major benefit of mixed-effects models is that they don't assume independence amongst observations, and there can be a correlated observations within a unit or cluster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is covered concisely in &quot;Modern Applied Statistics with S&quot; (MASS) in the first section of chapter 10 on &quot;Random and Mixed Effects&quot;.  V&amp;amp;R walk through an example with gasoline data comparing ANOVA and lme in that section, so it's a good overview.  The R function to be used in &lt;code&gt;lme&lt;/code&gt; in the &lt;code&gt;nlme&lt;/code&gt; package.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The model formulation is based on Laird and Ware (1982), so you can refer to that as a primary source although it's certainly not good for an introduction.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Laird, N.M. and Ware, J.H. (1982) &quot;Random-Effects Models for Longitudinal Data&quot;, Biometrics, 38, 963–974.&lt;/li&gt;&#xA;&lt;li&gt;Venables, W.N. and Ripley, B.D. (2002) &quot;&lt;a href=&quot;http://www.stats.ox.ac.uk/pub/MASS4/&quot; rel=&quot;nofollow&quot;&gt;Modern Applied Statistics with S&lt;/a&gt;&quot;, 4th Edition, Springer-Verlag.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You can also have a look at the &lt;a href=&quot;http://cran.r-project.org/doc/contrib/Fox-Companion/appendix-mixed-models.pdf&quot; rel=&quot;nofollow&quot;&gt;&quot;Linear Mixed Models&quot;&lt;/a&gt; (PDF) appendix to John Fox's &quot;An R and S-PLUS Companion to Applied Regression&quot;.  And &lt;a href=&quot;http://idiom.ucsd.edu/~rlevy/lign251/fall2007/lecture_14.pdf&quot; rel=&quot;nofollow&quot;&gt;this lecture by Roger Levy&lt;/a&gt; (PDF) discusses mixed effects models w.r.t. a multivariate normal distribution.&lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-07-27T13:07:38.643" CommentCount="1" />
  <row Id="773" PostTypeId="2" ParentId="283" CreationDate="2010-07-27T13:09:54.817" Score="3" ViewCount="" Body="&lt;p&gt;As everybody else said before, it means that you have as much parameters have you have data points. So, no goodness of fit testing. But this does not mean that &quot;by definition&quot;, the model can perfectly fit any data point. I can tell you by personal experience of working with some saturated models that could not predict specific data points. It is quite rare, but possible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another important issue is that saturated does not mean useless. For instance, in mathematical models of human cognition, model parameters are associated with specific cognitive processes that have a theoretical background. If a model is saturated, you can test its adequacy by doing focused experiments with manipulations that should affect only specific parameters. If the theoretical predictions match the observed differences (or lack of) in parameter estimates, then one can say that the model is valid. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;An example: Imagine for instance a model that has two sets of parameters, one for cognitive processing, and another for motor responses. Imagine now that you have an experiment with two conditions, one in which the participants ability to respond is impaired (they can only use one hand instead of two), and in the other condition there is no impairment. If the model is valid, differences in parameter estimates for both conditions should only occur for the motor response parameters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, be aware that even if one model is non-saturated, it might still be non-identifiable, which means that different combinations of parameter values produce the same result, which compromises any model fit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you wanna find more information on these issues in general, you might wanna take look at these papers:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bamber, D., &amp;amp; van Santen, J. P. H. (1985). How many parameters can a model have and still be testable? Journal of Mathematical Psychology, 29, 443-473. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bamber, D., &amp;amp; van Santen, J. P. H. (2000). How to Assess a Model's Testability and Identifiability. Journal of Mathematical Psychology, 44, 20-40. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;cheers&lt;/p&gt;&#xA;" OwnerUserId="447" LastActivityDate="2010-07-27T13:09:54.817" />
  <row Id="774" PostTypeId="2" ParentId="770" CreationDate="2010-07-27T13:14:19.160" Score="6" ViewCount="" Body="&lt;p&gt;I really wouldn't suggest using machine learning in order to learn statistics.  The mathematics employed in machine learning is often different because there's a real emphasis on the computational algorithm.  Even treatment of the same concept will be different.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A simple example of this would be to compare the treatment of linear regression between a basic statistics textbook and a machine learning textbook.  Most machine learning texts give a heavy treatment to concepts like &quot;&lt;a href=&quot;http://en.wikipedia.org/wiki/Gradient_descent&quot; rel=&quot;nofollow&quot;&gt;gradient descent&lt;/a&gt;&quot; and other optimzation techniques, while a statistics textbook will typically just cover &lt;a href=&quot;http://en.wikipedia.org/wiki/Ordinary_least_squares&quot; rel=&quot;nofollow&quot;&gt;ordinary least squares&lt;/a&gt; (if even that).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lastly, machine learning generally doesn't cover the same material when it comes to things like model comparison, sampling, etc.  So while some of the basic models are the same, the conceptual frameworks can be very different.&lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-07-27T13:14:19.160" />
  <row Id="775" PostTypeId="1" AcceptedAnswerId="777" CreationDate="2010-07-27T13:14:54.837" Score="4" ViewCount="305" Body="&lt;p&gt;What is the difference between operations research and statistical analysis?&lt;/p&gt;&#xA;" OwnerUserId="460" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-13T11:59:13.923" LastActivityDate="2011-04-13T11:59:13.923" Title="Operations research versus statistical analysis?" Tags="&lt;operations-research&gt;" AnswerCount="3" />
  <row Id="776" PostTypeId="2" ParentId="769" CreationDate="2010-07-27T13:25:13.050" Score="1" ViewCount="" Body="&lt;p&gt;This may be slightly controversial, but certainly most of the examples that you give are not statistics (some of these would more properly fall under machine learning):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Statistics usually covers the scenario where you are making inferences something when you only have a subset of the data (hence the use of things like &lt;a href=&quot;http://stats.stackexchange.com/questions/31/what-is-the-meaning-of-p-values-and-t-values-in-statistical-tests&quot; rel=&quot;nofollow&quot;&gt;the p-value&lt;/a&gt;).  The goal is to come to a deeper understanding of the underlying process that generates the data.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the examples that you provide, the goal is to perform some kind of action; whether the model itself is an accurate depiction of the underlying process is completely irrelevant so long as the predictions are accurate.  See &lt;a href=&quot;http://stats.stackexchange.com/questions/6/the-two-cultures-statistics-vs-machine-learning&quot; rel=&quot;nofollow&quot;&gt;my related question&lt;/a&gt; about the difference between machine learning and statistics.&lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-07-27T13:25:13.050" />
  <row Id="777" PostTypeId="2" ParentId="775" CreationDate="2010-07-27T13:35:21.463" Score="6" ViewCount="" Body="&lt;p&gt;Those are entire academic discplines so I do not think you can expect much more here than pointers to further, and more extensive, documentation as e.g. Wikipedia on &lt;a href=&quot;http://en.wikipedia.org/wiki/Operations_research&quot; rel=&quot;nofollow&quot;&gt;Operations Research&lt;/a&gt; and &lt;a href=&quot;http://en.wikipedia.org/wiki/Statistics&quot; rel=&quot;nofollow&quot;&gt;Statistics&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let me try a personal definition which may be grossly simplifying:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Operations Research is concerned with process modeling and optimisation&lt;/li&gt;&#xA;&lt;li&gt;Statistical Modeling is concerning with describing the so-called 'data generating process': find a model that describes something observed, and then do estimation, inference and possibly prediction.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="334" LastActivityDate="2010-07-27T13:35:21.463" />
  <row Id="778" PostTypeId="2" ParentId="672" CreationDate="2010-07-27T13:39:35.543" Score="5" ViewCount="" Body="&lt;p&gt;I'm sorry, but there seems to be some confusion here: &#xA;Bayes' theorem is not up for discussion of the neverending Bayesian-&lt;a href=&quot;http://en.wikipedia.org/wiki/Probability_interpretations#Frequentism&quot; rel=&quot;nofollow&quot;&gt;Frequentist&lt;/a&gt; debate. It is a theorem that is consistent with both schools of thought (given that it is consistent with Kolmogorov's probability axioms).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, Bayes' theorem is the core of Bayesian statistics, but the theorem itself is universal. The clash between frequentists and Bayesians mostly pertains to how prior distributions can be defined or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, if the question is about Bayes' theorem (and not Bayesian statistics):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bayes' theorem defines how one can calculate specific conditional probabilities. Imagine for instance that you know: the probability of somebody having symptom A, given that they have disease X p(A|X); the probability of somebody in general having disease X p(X); the probability of somebody in general having symptom A p(A). with these 3 pieces of information you can calculate the probability of somebody having disease X, given that they have sympotm A p(X|A).  &lt;/p&gt;&#xA;" OwnerUserId="447" LastEditorUserId="509" LastEditorDisplayName="" LastEditDate="2011-02-02T19:09:56.450" LastActivityDate="2011-02-02T19:09:56.450" CommentCount="1" />
  <row Id="779" PostTypeId="1" CreationDate="2010-07-27T13:49:41.073" Score="6" ViewCount="273" Body="&lt;p&gt;I've heard that a lot of quantities that occur in nature are normally distributed. This is typically justified using the central limit theorem, which says that when you average a large number of iid random variables, you get a normal distribution. So, for instance, a trait that is determined by the additive effect of a large number of genes may be approximately normally distributed since the gene values may behave roughly like iid random variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, what confuses me is that the property of being normally distributed is clearly not invariant under monotonic transformations. So, if there are two ways of measuring something that are related by a monotonic transformation, they are unlikely to both be normally distributed (unless that monotonic transformation is linear). For instance, we can measure the sizes of raindrops by diameter, by surface area, or by volume. Assuming similar shapes for all raindrops, the surface area is proportional to the square of the diameter, and the volume is proportional to the cube of the diameter. So all of these ways of measuring cannot be normally distributed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my question is whether the particular way of scaling (i.e., the particular choice of monotonic transformation) under which the distribution does become normal, must carry a physical significance. For instance, should heights be normally distributed or the square of height, or the logarithm of height, or the square root of height? Is there a way of answering that question by understanding the processes that affect height?&lt;/p&gt;&#xA;" OwnerUserId="463" LastActivityDate="2010-07-27T16:37:40.463" Title="Normal distribution and monotonic transformations" Tags="&lt;normality&gt;" AnswerCount="6" CommentCount="5" FavoriteCount="1" />
  <row Id="780" PostTypeId="2" ParentId="775" CreationDate="2010-07-27T13:55:18.623" Score="6" ViewCount="" Body="&lt;p&gt;Operations Research (OR), sometimes called &quot;Management Science&quot;, consists of three main topics, Optimization, Stochastic Processes, Process and Production Methodologies. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;OR uses statistical analysis in many contexts (for example discrete event simulations) but they should not be considered the same, additionally one of the main topics in OR is optimization (linear, and nonlinear) which can make it more clear why these two fields should be considered different &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is &lt;a href=&quot;http://www.or-exchange.com/questions&quot; rel=&quot;nofollow&quot;&gt;another exchange website for OR&lt;/a&gt; if you are interested &lt;/p&gt;&#xA;" OwnerUserId="172" LastEditorUserId="172" LastEditorDisplayName="" LastEditDate="2010-07-27T19:10:25.243" LastActivityDate="2010-07-27T19:10:25.243" CommentCount="1" />
  <row Id="781" PostTypeId="2" ParentId="712" CreationDate="2010-07-27T13:59:02.623" Score="2" ViewCount="" Body="&lt;p&gt;One approach would be to use Bloom filters. Check &lt;a href=&quot;http://www.uni-due.de/soziologie/schnell_forschung_safelink_software.php&quot; rel=&quot;nofollow&quot;&gt;SAFELINK&lt;/a&gt; project website for &lt;a href=&quot;http://www.uni-due.de/methods/bloom/index-en-bloom.html&quot; rel=&quot;nofollow&quot;&gt;programs&lt;/a&gt; in Java and Python. Paper explaining method is &lt;a href=&quot;http://dx.doi.org/10.1186/1472-6947-9-41&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is also an interesting approach to anaonymization of strings in the context of &lt;a href=&quot;http://en.wikipedia.org/wiki/Record_linkage&quot; rel=&quot;nofollow&quot;&gt;record linkage&lt;/a&gt; using n-grams developed by &lt;a href=&quot;http://datamining.anu.edu.au/projects/linkage.html&quot; rel=&quot;nofollow&quot;&gt;ANU Data Mining Group&lt;/a&gt;. The paper with description and sample Python code is available &lt;a href=&quot;http://www.biomedcentral.com/1472-6947/4/9/&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="22" LastEditorUserId="22" LastEditorDisplayName="" LastEditDate="2011-04-30T18:31:33.180" LastActivityDate="2011-04-30T18:31:33.180" />
  <row Id="782" PostTypeId="2" ParentId="779" CreationDate="2010-07-27T14:02:37.327" Score="2" ViewCount="" Body="&lt;p&gt;I think you missunderstood (half of) the use statistician make of the normal distribution but I really like your question. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't think it is a good idea to assume systematically normality and I  admit it is done sometime (maybe because the normal distribution is tractable, unimodal ...) without verification. Hence your remark about monotonic map is excellent ! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However the powerfull use of normality comes when you construct yourself new statistics such as the one that appears when you apply the empiriral counter part of expectation: &lt;strong&gt;the empirical mean&lt;/strong&gt;. Hence empirical mean and more generally smoothing is what makes normality appear everywhere...  &lt;/p&gt;&#xA;" OwnerUserId="223" LastEditorUserId="223" LastEditorDisplayName="" LastEditDate="2010-07-27T15:44:37.567" LastActivityDate="2010-07-27T15:44:37.567" />
  <row Id="783" PostTypeId="2" ParentId="726" CreationDate="2010-07-27T14:03:52.940" Score="11" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;&quot;... we want to underscore that,&#xA;  surely, God loves the .06 nearly as&#xA;  much as the .05. Can there be any&#xA;  doubt that God views the strength of&#xA;  evidence for or against the null as a&#xA;  fairly continuous function of the&#xA;  magnitude of p?&quot; (p.1277)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Rosnow, R. L., &amp;amp; Rosenthal, R. (1989). Statistical procedures and the justification of knowledge in psychological science. American Psychologist, 44(10), 1276-1284. &lt;a href=&quot;http://socrates.berkeley.edu/~maccoun/PP279_Rosnow.pdf&quot; rel=&quot;nofollow&quot;&gt;pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="442" LastEditorUserId="442" LastEditorDisplayName="" LastEditDate="2010-07-27T16:17:34.270" LastActivityDate="2010-07-27T16:17:34.270" CommunityOwnedDate="2010-07-27T14:03:52.940" CommentCount="3" />
  <row Id="784" PostTypeId="2" ParentId="614" CreationDate="2010-07-27T14:04:13.360" Score="2" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://mitpress.mit.edu/catalog/item/default.asp?ttype=2&amp;amp;tid=12156&quot; rel=&quot;nofollow&quot;&gt;Street-Fighting Mathematics&lt;/a&gt;. The Art of Educated Guessing and Opportunistic Problem Solving&lt;br&gt;&#xA;by Sanjoy Mahajan from MIT&lt;/p&gt;&#xA;" OwnerUserId="22" LastActivityDate="2010-07-27T14:04:13.360" CommunityOwnedDate="2010-07-27T14:04:13.360" />
  <row Id="785" PostTypeId="2" ParentId="726" CreationDate="2010-07-27T14:10:31.507" Score="5" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;The statistician cannot evade the responsibility for understanding the&#xA;  process he applies or recommends.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;-– Sir Ronald A. Fisher&lt;/p&gt;&#xA;" OwnerUserId="447" LastEditorUserId="795" LastEditorDisplayName="" LastEditDate="2010-12-03T04:02:46.950" LastActivityDate="2010-12-03T04:02:46.950" CommunityOwnedDate="2010-07-27T14:10:31.507" />
  <row Id="786" PostTypeId="2" ParentId="726" CreationDate="2010-07-27T14:14:10.623" Score="11" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;The death of one man is a tragedy. &#xA;  The death of millions is a statistic.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;~Joseph Stalin, comment to Churchill at Potsdam, 1945&lt;/p&gt;&#xA;" OwnerUserId="460" LastEditorUserId="509" LastEditorDisplayName="" LastEditDate="2010-08-06T23:23:57.000" LastActivityDate="2010-08-06T23:23:57.000" CommunityOwnedDate="2010-07-27T14:14:10.623" CommentCount="1" />
  <row Id="787" PostTypeId="2" ParentId="726" CreationDate="2010-07-27T14:20:42.650" Score="0" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Statistics are the triumph of the&#xA;  quantitative method, and the&#xA;  quantitative method is the victory of&#xA;  sterility and death.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;~ &lt;strong&gt;Hillaire Belloc&lt;/strong&gt; &lt;em&gt;in&lt;/em&gt; The Silence of the Sea&lt;/p&gt;&#xA;" OwnerUserId="460" LastActivityDate="2010-07-27T14:20:42.650" CommunityOwnedDate="2010-07-27T14:20:42.650" CommentCount="1" />
  <row Id="788" PostTypeId="2" ParentId="726" CreationDate="2010-07-27T14:25:39.673" Score="1" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;A witty statesman said, you might&#xA;  prove anything by figures.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;~ Thomas Carlyle, Chartism (1839) ch. 2&lt;/p&gt;&#xA;" OwnerUserId="460" LastActivityDate="2010-07-27T14:25:39.673" CommunityOwnedDate="2010-07-27T14:25:39.673" CommentCount="1" />
  <row Id="789" PostTypeId="2" ParentId="614" CreationDate="2010-07-27T14:38:34.183" Score="2" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://psy.otago.ac.nz/miller/index.htm#GLMBook&quot; rel=&quot;nofollow&quot;&gt;Statistical Analysis with the General Linear Model&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I covers basic linear models (ANOVA, ANCOVA, multiple regression). I can tell by personal experience that it is really really good book to get into the general framework of linear models, which are very useful in many advanced approaches (e.g., hierarchical modeling) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;cheers&lt;/p&gt;&#xA;" OwnerUserId="447" LastActivityDate="2010-07-27T14:38:34.183" CommunityOwnedDate="2010-07-27T14:38:34.183" CommentCount="1" />
  <row Id="790" PostTypeId="1" CreationDate="2010-07-27T14:46:49.640" Score="2" ViewCount="218" Body="&lt;p&gt;I am looking at a scientific paper in which a single measurement is calculated using a logarithmic mean&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;'triplicate spots were combined to&#xA;  produce one signal by taking the&#xA;  logarithmic mean of reliable spots'&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Why choose the log-mean?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are the authors making an assumption about the underlying distribution? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;That it is what...Log-Normal? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Have they just picked something they thought was reasonable... i.e. between a a mean and a geometric mean?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any thoughts?&lt;/p&gt;&#xA;" OwnerUserId="228" LastActivityDate="2010-07-27T17:19:27.563" Title="Why (or when) to use the log-mean?" Tags="&lt;distributions&gt;&lt;logarithm&gt;&lt;mean&gt;" AnswerCount="2" CommentCount="3" />
  <row Id="791" PostTypeId="2" ParentId="770" CreationDate="2010-07-27T14:52:19.113" Score="1" ViewCount="" Body="&lt;p&gt;I really dont think so, as there are fundamental aspects in statistics that are simply overlooked in machine learning. For instance, in statistics, when fitting a model to data, the discrpeancy function that is used (e.g., G^2, RMSEA) is essential because they have different statistical properties. In machine learning, it just doesnt matter, so it is not covered at all. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course one could argue that you could learn those things later, but IMHO, it is better to undestand and care about some issues, and possibly in the future not care about them, then the other way around. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;cheers&lt;/p&gt;&#xA;" OwnerUserId="447" LastActivityDate="2010-07-27T14:52:19.113" />
  <row Id="792" PostTypeId="2" ParentId="779" CreationDate="2010-07-27T15:03:28.010" Score="3" ViewCount="" Body="&lt;p&gt;Simply CLT (nor any other theorem) does not state that every quantity in the universe is normally distributed. Indeed, statisticians often use monotonic transformations to improve normality, so they could use their favorite tools.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-07-27T15:03:28.010" />
  <row Id="793" PostTypeId="2" ParentId="790" CreationDate="2010-07-27T15:04:58.067" Score="0" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Are the authors making an assumption&#xA;  about the underlying distribution?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;You are making an assumption whether you choose to use it or whether you choose against using it.&#xA;For Power Law distributions it usually makes sense to look at the logarithms.&lt;/p&gt;&#xA;" OwnerUserId="237" LastActivityDate="2010-07-27T15:04:58.067" />
  <row Id="794" PostTypeId="2" ParentId="726" CreationDate="2010-07-27T15:05:38.800" Score="12" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;The subjectivist (i.e. Bayesian)&#xA;  states his judgements, whereas the&#xA;  objectivist sweeps them under the&#xA;  carpet by calling assumptions&#xA;  knowledge, and he basks in the&#xA;  glorious objectivity of science.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I.J. Good&lt;/p&gt;&#xA;" LastActivityDate="2010-07-27T15:05:38.800" CommunityOwnedDate="2010-07-27T15:05:38.800" CommentCount="2" />
  <row Id="795" PostTypeId="1" CreationDate="2010-07-27T15:08:11.493" Score="-2" ViewCount="163" Body="&lt;p&gt;I have data compiled by someone else where score averages have been computed over time- averages range from 0-100. The original scores have negative values in many cases and the average would have been negative also, raw average ranges from -30 to 90.  How is this 'normalization' accomplished?&lt;br&gt;&#xA;Thanks&lt;/p&gt;&#xA;" OwnerUserId="474" LastActivityDate="2010-07-27T15:36:41.480" ClosedDate="2010-08-03T09:14:27.403" Title="Normalization of series" Tags="&lt;mean&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="796" PostTypeId="2" ParentId="305" CreationDate="2010-07-27T15:14:22.937" Score="12" ViewCount="" Body="&lt;p&gt;I would like to oppose the other two answers based on a paper (in German) by &lt;a href=&quot;http://www.psycontent.com/content/t5726m72644gq457/&quot; rel=&quot;nofollow&quot;&gt;Kubinger, Rasch and Moder (2009)&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;They argue, based on &quot;extensive&quot; simulations from distributions either meeting or not meeting the assumptions imposed by a t-test, (normality and homogenity of variance) that the welch-tests performs equally well when the assumptions are met (i.e., basically same probability of committing alpha and beta errors) but outperforms the t-test if the assumptions are not met, especially in terms of power. Therefore, they recommend to always use the welch-test if the sample size exceeds 30.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a meta-comment: For people interested in statistics (like me and probably most other here) an argument based on data (as mine) should at least count equally as arguments solely based on theoretical grounds (as the others here).&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update:&lt;/strong&gt;&lt;br&gt;&#xA;After thinking about this topic again, I found two further recommendations of which the newer one assists my point. Look at the original papers (which are both, at least for me, freely available) for the argumentations that lead to these recommendations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first recommendation comes from Graeme D. Ruxton in 2006: &quot;&lt;strong&gt;If you want to compare the central tendency of 2 populations based on samples of unrelated data, then the unequal variance t-test should always be used in preference to the Student's t-test or Mann–Whitney U test.&lt;/strong&gt;&quot;&lt;br&gt;&#xA;In:&lt;br&gt;&#xA;Ruxton, G.D., 2006. &lt;a href=&quot;http://beheco.oxfordjournals.org/content/17/4/688.full&quot; rel=&quot;nofollow&quot;&gt;The unequal variance t-test is an underused&#xA;alternative to Student’s t-test and the Mann–Whitney U test&lt;/a&gt;.&#xA;&lt;em&gt;Behav. Ecol&lt;/em&gt;. 17, 688–690.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second (older) recommendation is from Coombs et al. (1996, p. 148): &quot;&lt;strong&gt;In summary, the independent samples t test is generally acceptable in terms of controlling Type I error rates provided there are sufficiently large equal-sized samples, even when the equal population variance assumption is violated.&lt;/strong&gt; For unequal-sized samples, however, an alternative that does not assume equal population variances is preferable. Use the James second-order test when distributions are either short-tailed symmetric or normal. Promising alternatives include the Wilcox H and Yuen trimmed means tests, which provide broader control of Type I error rates than either the Welch test or the James test and have greater power when data are long-tailed.&quot; (emphasis added)&lt;br&gt;&#xA;In:&lt;br&gt;&#xA;Coombs WT, Algina J, Oltman D. 1996. &lt;a href=&quot;http://rer.sagepub.com/content/66/2/137.full.pdf&quot; rel=&quot;nofollow&quot;&gt;Univariate and multivariate omnibus hypothesis tests selected to control type I error rates when population variances are not necessarily equal&lt;/a&gt;. &lt;em&gt;Rev Educ Res&lt;/em&gt; 66:137–79.&lt;/p&gt;&#xA;" OwnerUserId="442" LastEditorUserId="442" LastEditorDisplayName="" LastEditDate="2010-10-19T15:28:42.237" LastActivityDate="2010-10-19T15:28:42.237" CommentCount="1" />
  <row Id="797" PostTypeId="2" ParentId="779" CreationDate="2010-07-27T15:19:43.983" Score="3" ViewCount="" Body="&lt;p&gt;Very good question. I feel that the answer depends on the whether you can identify the underlying process that gives rise to the measurement in question. If for example, you have evidence that height is a linear combination of several factors (e.g., height of parents, height of grandparents etc) then it would be natural to assume that height is normally distributed. On the other hand if you have evidence or perhaps even theory that the log of height is a linear combination of several variables (e.g., log parents heights, log of grandparents heights etc) then the log of height will be normally distributed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In most situations, we do not know the underlying process that drives the measurement of interest. Thus, we can do one of several things:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(a) If the empirical distribution of heights looks normal then we use a the normal density for further analysis which implicitly assumes that height is a linear combination of several variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(b) If the empirical distribution does not look normal then we can try some transformation as suggested by &lt;a href=&quot;http://stats.stackexchange.com/users/88/mbq&quot; rel=&quot;nofollow&quot;&gt;mbq&lt;/a&gt; (e.g. log(height)). In this case we implicitly assume that the transformed variable (i.e., log(height)) is a linear combination of several variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(c) If (a) or (b) do not help then we have to abandon the advantages that CLT and an assumption of normality give us and model the variable using some other distribution.&lt;/p&gt;&#xA;" LastActivityDate="2010-07-27T15:25:00.930" />
  <row Id="798" PostTypeId="1" AcceptedAnswerId="862" CreationDate="2010-07-27T15:21:48.417" Score="9" ViewCount="1643" Body="&lt;p&gt;I'm interested in finding as optimal of a method as I can for determining how many bins I should use in a histogram. My data should range from 30 to 350 objects at most, and in particular I'm trying to apply thresholding (like Otsu's method) where &quot;good&quot; objects, which I should have fewer of and should be more spread out, are separated from &quot;bad&quot; objects, which should be more dense in value. A concrete value would have a score of 1-10 for each object. I'd had 5-10 objects with scores 6-10, and 20-25 objects with scores 1-4. I'd like to find a histogram binning pattern that generally allows something like Otsu's method to threshold off the low scoring objects. However, in the implementation of Otsu's I've seen, the bin size was 256, and often I have many fewer data points that 256, which to me suggests that 256 is not a good bin number. With so few data, what approaches should I take to calculating the number of bins to use?&lt;/p&gt;&#xA;" OwnerUserId="476" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-10-06T14:32:02.417" LastActivityDate="2011-01-07T09:27:45.153" Title="Calculating optimal number of bins in a histogram for n, where n ranges from 30-350" Tags="&lt;rule-of-thumb&gt;&lt;histogram&gt;" AnswerCount="6" FavoriteCount="4" />
  <row Id="799" PostTypeId="2" ParentId="764" CreationDate="2010-07-27T15:26:34.537" Score="5" ViewCount="" Body="&lt;p&gt;The main advantage of LME for analysing accuracy data is the ability to account for a series of random effects. In psychology experiments, researchers usually aggregate items and/or participants. Not only are people different from each other, but items also differ (some words might be more distinctive or memorable, for instance). Ignoring these sources of variability usually leads to underestimations of accuracy (for instance lower d' values). Although the participant aggregation issue can somehow be dealt with  individual estimation, the item effects are still there, and are commonly larger than participant effects. LME not only allows you to tackle both random effects simultaneously, but also to add specificy additional predictor variables (age, education level, word length, etc.) to them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A really good reference for LMEs, especially focused in the fields of linguistics and experimental psychology, is &#xA;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0521709180&quot; rel=&quot;nofollow&quot;&gt;Analyzing Linguistic Data: A Practical Introduction to Statistics using R&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;cheers&lt;/p&gt;&#xA;" OwnerUserId="447" LastActivityDate="2010-07-27T15:26:34.537" CommentCount="2" />
  <row Id="800" PostTypeId="2" ParentId="795" CreationDate="2010-07-27T15:27:18.890" Score="1" ViewCount="" Body="&lt;p&gt;The question is a bit unclear. But, perhaps the normalization you are looking for is this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Normalized Score = 100 * (Raw Score - min(Raw_score)) / (max(Raw Score) - min(Raw Score))&lt;/p&gt;&#xA;" LastActivityDate="2010-07-27T15:27:18.890" />
  <row Id="801" PostTypeId="2" ParentId="798" CreationDate="2010-07-27T15:30:31.073" Score="1" ViewCount="" Body="&lt;p&gt;I'm not sure this counts as strictly good practise, but I tend to produce more than one histogram with different bin widths and pick the histogram which histgram to use based on which histgram fits the interpretation I'm trying to communicate best. Whilst this introduces some objectivity into the choice of histogram I justify it on the basis I have had much more time to understnad the data than the person I'm giving the histogram to so I need to give them a very concise message. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm also a big fan of presenting histograms with the same number of points in each bin rather than the same bin width. I usually find these represent the data far better then the constant bin width although they are mopre difficult to produce. &lt;/p&gt;&#xA;" OwnerUserId="210" LastActivityDate="2010-07-27T15:30:31.073" CommentCount="2" />
  <row Id="802" PostTypeId="2" ParentId="779" CreationDate="2010-07-27T15:35:14.460" Score="3" ViewCount="" Body="&lt;p&gt;The rescaling of a particular variable should, when possible, relate to some comprehensible scale for the reason that it helps make the resulting model interpretable.  However, the resulting transformation need not absolutely carry a physical significance.  Essentially you have to engage in a trade off between the violation of the normality assumption and the interpretability of your model.  What I like to do in these situations is have the original data, data transformed in a way that makes sense, and the data transformed in a way that is most normal.  If the data transformed in a way that makes sense is the same as the results when the data is transformed in a way that makes it most normal, I report it in a way that is interpretable with a side note that the results are the same in the case of the optimally transformed (and/or untransformed) data.  When the untransformed data is behaving particularly poorly, I conduct my analyses with the transformed data but do my best to report the results in untransformed units.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, I think you have a misconception in your statement that &quot;quantities that occur in nature are normally distributed&quot;.  This only holds true in cases where the value is &quot;determined by the additive effect of a large number&quot; of independent factors.  That is, means and sums are normally distributed regardless of the underlying distribution from which they draw, where as individual values are not expected to be normally distributed.  As was of example, individual draws from a binomial distribution do not look at all normal, but a distribution of the sums of 30 draws from a binomial distribution does look rather normal.&lt;/p&gt;&#xA;" OwnerUserId="196" LastActivityDate="2010-07-27T15:35:14.460" />
  <row Id="803" PostTypeId="2" ParentId="795" CreationDate="2010-07-27T15:36:41.480" Score="2" ViewCount="" Body="&lt;p&gt;Or more generically,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;Index = ( ((RawValue - Min(Raw))/(Max(Raw)-Min(Raw)) * (Max(Out)-Min(Out) ) + Min(Out)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where &lt;code&gt;Raw&lt;/code&gt; is the input vector, &lt;code&gt;Out&lt;/code&gt; is the output vector, and &lt;code&gt;RawValue&lt;/code&gt; is the value in question.  Srikant's answer is the same for an output range of 0 to 100.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To convert back, rearrange to get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;RawValue = (((Index - Min(Out))*(Max(Raw)-Min(Raw)) / (Max(Out)-Min(Out)) + Min(Raw)&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="257" LastActivityDate="2010-07-27T15:36:41.480" />
  <row Id="804" PostTypeId="2" ParentId="305" CreationDate="2010-07-27T15:43:51.857" Score="5" ViewCount="" Body="&lt;p&gt;of course, one could ditch both tests, and start using a Bayesian t-test (Savage-Dickey ratio test), which can account for unequal and unequal variances, and best of all, it allows for a quantification of evidence in favor of the null hypothesis (which means, no more of old &quot;failure to reject&quot; talk)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This test is very simple (and fast) to implement, and there is a paper that clearly explains to readers unfamiliar with Bayesian statistics how to use it, along with an R script. you basically can just insert your data send the commands to the R console:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.ruudwetzels.com/articles/Wetzelsetal2009_SDtest.pdf&quot; rel=&quot;nofollow&quot;&gt;Wetzels, R., Raaijmakers, J. G. W., Jakab, E., &amp;amp; Wagenmakers, E.-J. (2009). How to Quantify Support For and Against the Null Hypothesis: A Flexible WinBUGS Implementation of a Default Bayesian t-test.&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;there is also a tutorial for all this, with example data:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.ruudwetzels.com/index.php?src=SDtest&quot; rel=&quot;nofollow&quot;&gt;http://www.ruudwetzels.com/index.php?src=SDtest&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know this is not a direct response to what was asked, but I thought readers might enjoy having this nice alternative&lt;/p&gt;&#xA;&#xA;&lt;p&gt;cheers&lt;/p&gt;&#xA;" OwnerUserId="447" LastActivityDate="2010-07-27T15:43:51.857" CommentCount="1" />
  <row Id="805" PostTypeId="1" CreationDate="2010-07-27T15:43:55.057" Score="1" ViewCount="145" Body="&lt;p&gt;I am attempting to calculate the standard error (SE) for the positive predictive value (PPV), negative predictive value (NPV), and diagnostic odds ratio (DOR) that I have obtained using the rates of true positives, false positives, true negatives, and false negatives in a sample. I am able to get 95% CIs but not SE. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you!&lt;/p&gt;&#xA;" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-08-13T13:35:40.603" LastActivityDate="2010-08-13T13:35:40.603" Title="How do I calculate the SE for PPV, NPV, and DOR?" Tags="&lt;error&gt;&lt;standard-error&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="806" PostTypeId="1" AcceptedAnswerId="829" CreationDate="2010-07-27T15:56:38.297" Score="0" ViewCount="446" Body="&lt;p&gt;Lately, there have been numerous questions about &lt;a href=&quot;http://en.wikipedia.org/wiki/Normalization_%28statistics%29&quot; rel=&quot;nofollow&quot;&gt;normalization&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are some of the situations where you never ever ever should normalize your data, and what are the alternatives?&lt;/p&gt;&#xA;" OwnerUserId="59" LastActivityDate="2010-09-27T23:40:16.273" Title="When should normalization never be used?" Tags="&lt;normality&gt;" AnswerCount="3" CommentCount="1" />
  <row Id="807" PostTypeId="2" ParentId="769" CreationDate="2010-07-27T16:00:22.087" Score="0" ViewCount="" Body="&lt;p&gt;A statistic is any sample estimation of a population characteristic is it not?  Thus, I would suggest so long as inference from a characterized sample to a population is taking place what is occurring is, at least in part, statistics.  Under my definition machine learning would be a discipline that makes use of statistics.  For example, under many examples of machine learning it seems like part of what is happening is an attempt to take information provided to the system (a sample) in order to guess from which distribution (a population) it arose.&lt;/p&gt;&#xA;" OwnerUserId="196" LastActivityDate="2010-07-27T16:00:22.087" />
  <row Id="808" PostTypeId="2" ParentId="726" CreationDate="2010-07-27T16:04:42.707" Score="3" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;A statistical analysis, properly conducted, is a delicate dissection of uncertainties, a surgery of suppositions.   &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;-- M.J. Moroney&lt;/p&gt;&#xA;" OwnerUserId="482" LastEditorUserId="795" LastEditorDisplayName="" LastEditDate="2010-10-02T17:12:07.793" LastActivityDate="2010-10-02T17:12:07.793" CommunityOwnedDate="2010-07-27T16:04:42.707" CommentCount="2" />
  <row Id="809" PostTypeId="2" ParentId="726" CreationDate="2010-07-27T16:14:26.303" Score="0" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;An argument over the meaning of words&#xA;  is a matter of law, an argument&#xA;  grounded in empirical data and&#xA;  quantitative estimates is an argument&#xA;  about science.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;~ Razib Khan (though he is not a statistician or famous)&lt;/p&gt;&#xA;" LastActivityDate="2010-07-27T16:14:26.303" CommunityOwnedDate="2010-07-27T16:14:26.303" />
  <row Id="810" PostTypeId="2" ParentId="798" CreationDate="2010-07-27T16:17:43.867" Score="0" ViewCount="" Body="&lt;p&gt;If you use too few bins, the histogram doesn't really portray the data very well. If you have too many bins, you get a broken comb look, which also doesn't give a sense of the distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One solution is to create a graph that shows every value. Either a dot plot, or a cumulative frequency distribution, which doesn't require any bins.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to create a frequency distribution with equally spaced bins, you need to decide how many bins (or the width of each). The decision clearly depends on the number of values. If you have lots of values, your graph will look better and be more informative if you have lots of bins. &lt;a href=&quot;http://en.wikipedia.org/wiki/Histogram#Number_of_bins_and_width&quot; rel=&quot;nofollow&quot;&gt;This wikipedia page&lt;/a&gt; lists several methods for deciding bin width from the number of observations. The simplest method is to set the number of bins equal to the square root of the number of values you are binning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://web.mit.edu/hshimaza/www/res/histogram.html#WebApp&quot; rel=&quot;nofollow&quot;&gt;This page from Hideaki Shimazaki&lt;/a&gt; explains an alternative method. It is a bit more complicated to calculate, but seems to do a great job. The top part of the page is a Java app. Scroll past that to see the theory and explanation, then keep scrolling to find links to the papers that explain the method.&lt;/p&gt;&#xA;" OwnerUserId="25" LastActivityDate="2010-07-27T16:17:43.867" />
  <row Id="811" PostTypeId="2" ParentId="726" CreationDate="2010-07-27T16:18:28.013" Score="3" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;It is the mark of a truly intelligent&#xA;  person to be moved by statistics.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;George Bernard Shaw&lt;/p&gt;&#xA;" LastActivityDate="2010-07-27T16:18:28.013" CommunityOwnedDate="2010-07-27T16:18:28.013" CommentCount="1" />
  <row Id="812" PostTypeId="1" AcceptedAnswerId="904" CreationDate="2010-07-27T16:19:42.880" Score="6" ViewCount="350" Body="&lt;p&gt;There are a lot of references in the statistic literature to &quot;&lt;strong&gt;functional data&lt;/strong&gt;&quot; (i.e. data that are curves), and in parallel, to &quot;&lt;strong&gt;high dimensional data&lt;/strong&gt;&quot; (i.e. when data are high dimensional vectors). My question is about the difference between the two type of data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When talking about applied statistic methodologies that apply in case 1 can be understood as a rephrasing of methodologies from case 2 through a projection into a finite dimensional subspace of a space of functions, it can be polynomes, splines, wavelet, Fourier, .... and will translate the functional problem into a finite dimensional vectorial problem (since in applied mathematic everything comes to be finite at some point). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My question is:&lt;/strong&gt;  can we say that any statistical procedure that applies to functional data  can also be applied (almost directly) to high dimension data and that any procedure dedicated to high dimensional data can be (almost directly) applied to functional data ? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the answer is no, can you illustrate ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT/UPDATE with the help of Simon Byrne's answer:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;sparsity (S-sparse assumption, $l^p$ ball and weak $l^p$ ball for $p&amp;lt;1$) is used as a structural assumption in high dimensional statistical analysis. &lt;/li&gt;&#xA;&lt;li&gt;&quot;smoothness&quot; is used as a structural assumption in functional data analysis. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;On the other hand, inverse Fourier transform and inverse wavelet transform are transforming sparcity into smoothness, and smoothness is transformed into sparcity by wavelet and fourier transform. This make the critical difference mentionned by Simon not so critical?&lt;/p&gt;&#xA;" OwnerUserId="223" LastEditorUserId="223" LastEditorDisplayName="" LastEditDate="2011-02-10T08:14:49.580" LastActivityDate="2011-02-10T08:14:49.580" Title="What is the difference between functional data analysis and high dimensional data analysis" Tags="&lt;data-mining&gt;&lt;signal-processing&gt;&lt;curve-fitting&gt;&lt;wavelet&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="2" />
  <row Id="813" PostTypeId="2" ParentId="726" CreationDate="2010-07-27T16:23:12.707" Score="25" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Statistical thinking will one day be&#xA;  as necessary a qualification for&#xA;  efficient citizenship as the ability&#xA;  to read and write.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;--H.G. Wells&lt;/p&gt;&#xA;" LastActivityDate="2010-07-27T16:23:12.707" CommunityOwnedDate="2010-07-27T16:23:12.707" CommentCount="2" />
  <row Id="814" PostTypeId="2" ParentId="726" CreationDate="2010-07-27T16:26:53.157" Score="4" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;These days the statistician is often&#xA;  asked such questions as &quot;Are you a&#xA;  Bayesian?&quot; &quot;Are you a frequentist?&quot;&#xA;  &quot;Are you a data analyst?&quot; &quot;Are you a&#xA;  designer of experiments?&quot;. I will&#xA;  argue that the appropriate answer to&#xA;  ALL of these questions can be (and&#xA;  preferably should be) &quot;yes&quot;, and that&#xA;  we can see why this is so if we&#xA;  consider the scientific context for&#xA;  what statisticians do.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;--G.E.P. Box&lt;/p&gt;&#xA;" LastActivityDate="2010-07-27T16:26:53.157" CommunityOwnedDate="2010-07-27T16:26:53.157" />
  <row Id="815" PostTypeId="2" ParentId="779" CreationDate="2010-07-27T16:34:22.917" Score="3" ViewCount="" Body="&lt;p&gt;I must admit that I do not really understand your question:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;your raindrops example is not very satisfying since this is not illustrating the fact that the Gaussian behaviour comes from the &quot;average of a large number of iid random variables&quot;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;if the quantity $X$ that you are interested in is an average $\frac{Y_1+\ldots+Y_N}{N}$ that fluctuates around its mean in a Gaussian way, you can also expect that $\frac{f(Y_1)+\ldots+f(Y_N)}{N}$ has a Gaussian behaviour.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;if the fluctuation of $X$ around its mean are approximately Gaussian and small, then so are the fluctuation of $f(X)$ around its mean (by Taylor expansion)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;could you cite some true examples of (real life) Gaussian behaviour coming from averaging: this is not very common! Gaussian behaviour is often used in statistics as a first rough approximation because the computations are very tractable. As physicists uses the harmonic approximation, statisticians uses the Gaussian approximation.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="368" LastActivityDate="2010-07-27T16:34:22.917" CommentCount="2" />
  <row Id="816" PostTypeId="2" ParentId="806" CreationDate="2010-07-27T16:34:55.863" Score="2" ViewCount="" Body="&lt;p&gt;Of course one should never try to blindly normalize data if the data does not follow a (single) normal distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example one might want to rescale observables $X$ to all be normal with $(X-\mu)/\sigma$, but this can only work if the data is normal and if both $\mu$ and $\sigma$ are the same for all data points (e.g. $\sigma$ doesn't depend on $\mu$ in a particular $X$ range).&lt;/p&gt;&#xA;" OwnerUserId="56" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-09-27T23:40:16.273" LastActivityDate="2010-09-27T23:40:16.273" />
  <row Id="817" PostTypeId="2" ParentId="779" CreationDate="2010-07-27T16:37:40.463" Score="3" ViewCount="" Body="&lt;p&gt;Vipul, you're not being totally precise in your question. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;This is typically justified using the&#xA;  central limit theorem, which says that&#xA;  when you average a large number of iid&#xA;  random variables, you get a normal&#xA;  distribution.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I'm not entirely sure this is what you're saying, but keep in mind that the raindrops in your example are not iid random variables. The mean calculated by sampling a certain number of those raindrops is a random variables, and as the means are calculated using a large enough sample size, the distribution of that sample mean is normal. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The law of large numbers says that the value of that sample mean converges to the average value of the population (strong or weak depending on type of convergence). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The CLT says that the sample mean, call it XM(n), which is a random variable, has a distribution, say G(n). As n approaches infintity, that distribution is the normal distribution. CLT is all about &lt;em&gt;convergence in distribution&lt;/em&gt;, not a basic concept. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The observations you draw (diameter, area, volume) don't have to be normal at all. They probably won't be if you plot them. But, the sample mean from taking all three observations will have a normal distribution. And, the volume won't be the cube of the diameter, nor will the area be the square of the diameter. The square of the sums is not going to be the sum of the squares, unless you get oddly lucky. &lt;/p&gt;&#xA;" OwnerUserId="62" LastActivityDate="2010-07-27T16:37:40.463" />
  <row Id="818" PostTypeId="2" ParentId="73" CreationDate="2010-07-27T16:37:42.413" Score="1" ViewCount="" Body="&lt;p&gt;We mostly use:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;ggplot - for charts&lt;/li&gt;&#xA;&lt;li&gt;stats&lt;/li&gt;&#xA;&lt;li&gt;e1071 - for &lt;a href=&quot;http://en.wikipedia.org/wiki/Support_vector_machine&quot; rel=&quot;nofollow&quot;&gt;SVMs&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="480" LastEditorUserId="509" LastEditorDisplayName="" LastEditDate="2010-08-09T08:15:05.380" LastActivityDate="2010-08-09T08:15:05.380" CommunityOwnedDate="2010-07-27T16:37:42.413" CommentCount="1" />
  <row Id="819" PostTypeId="2" ParentId="805" CreationDate="2010-07-27T16:40:37.527" Score="2" ViewCount="" Body="&lt;p&gt;The PPV and NPV are proportions. You know the numerator and denominator, which both are positive integers, so you can calculate the proportion. Your goal, I presume, is to quantify how well you have determined those values. If your sample size is huge, then those values are likely to be very close to their true population values. If the sample size is small, then there is more random variability and those values are more likely to be far from their true value. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The confidence intervals you calculated tell you what you want to know. Given a bunch of assumptions, you can be 95% confident that the confidence interval includes the population value. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What about the standard error? The problem is that the uncertainty of a proportion is not symmetrical. The asymmetry is very noticeable when the proportion is far from 0.50 and the sample size is small. Since the confidence interval is not symmetrical around the observed proportion, trying to express this uncertainty as a single plus/minus standard error won't be very informative. It makes more sense to report the confidence interval. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you really want to calculate a standard error, even knowing it isn't very helpful, here is the formula to compute an approximate standard error from p (the proportion) and n (sample size; the denominator): &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://upload.wikimedia.org/math/2/8/5/285265cbedc09f97b3c98762a3433393.png&quot; alt=&quot;&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="25" LastActivityDate="2010-07-27T16:40:37.527" />
  <row Id="820" PostTypeId="1" AcceptedAnswerId="822" CreationDate="2010-07-27T16:42:39.070" Score="0" ViewCount="78" Body="&lt;p&gt;I am building a web application for used book trading and I am adding a feature to propose other book that would be interesting when they view an offer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently the data that I store are the following (they are updated each time someone visit an offer) :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ISBN (book of the offer)&lt;br&gt;&#xA;SessId (a unique id that everyone has when visiting the website)&lt;br&gt;&#xA;NumberOfVisit (The number of time someone has view an offer of that book)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also have access to some user update data which categorize the book by subject and course. It isn't necessarily up-to-date and precise, but it's nonetheless data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are the approach to list the most interesting books for a book ?&lt;/p&gt;&#xA;" OwnerUserId="488" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-08-07T17:52:01.997" LastActivityDate="2010-08-07T17:52:01.997" Title="How can I link item by relevance ?" Tags="&lt;algorithms&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="821" PostTypeId="2" ParentId="820" CreationDate="2010-07-27T16:46:45.040" Score="1" ViewCount="" Body="&lt;p&gt;You probably want to look at &lt;a href=&quot;http://en.wikipedia.org/wiki/Recommender_system&quot; rel=&quot;nofollow&quot;&gt;recommender systems&lt;/a&gt;. &lt;/p&gt;&#xA;" LastActivityDate="2010-07-27T16:46:45.040" />
  <row Id="822" PostTypeId="2" ParentId="820" CreationDate="2010-07-27T16:50:45.797" Score="2" ViewCount="" Body="&lt;p&gt;There are many,many ways to do this....I'd suggest googling for some search terms to get started, such as &quot;market basket analysis&quot;, or having  a look at Toby Segaran's &quot;Programming Collective Intelligence&quot; if you know python (even if you don't - it is pretty easy to understand).&lt;/p&gt;&#xA;" OwnerUserId="247" LastActivityDate="2010-07-27T16:50:45.797" CommentCount="1" />
  <row Id="823" PostTypeId="2" ParentId="726" CreationDate="2010-07-27T16:50:51.430" Score="-1" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Let’s be clear: the work of science&#xA;  has nothing whatever to do with&#xA;  consensus. Consensus is the business&#xA;  of politics. Science, on the contrary,&#xA;  requires only one investigator who&#xA;  happens to be right, which means that&#xA;  he or she has results that are&#xA;  verifiable by reference to the real&#xA;  world. In science consensus is&#xA;  irrelevant. What is relevant is&#xA;  reproducible results. The greatest&#xA;  scientists in history are great&#xA;  precisely because they broke with the&#xA;  consensus.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;There is no such thing as consensus&#xA;  science. If it’s consensus, it isn’t&#xA;  science. If it’s science, it isn’t&#xA;  consensus. Period.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Michael Crichton&lt;/p&gt;&#xA;" OwnerUserId="25" LastActivityDate="2010-07-27T16:50:51.430" CommunityOwnedDate="2010-07-27T16:50:51.430" CommentCount="7" />
  <row Id="824" PostTypeId="2" ParentId="672" CreationDate="2010-07-27T16:56:35.273" Score="8" ViewCount="" Body="&lt;p&gt;Bayes' theorem is a relatively simple, but fundamental result of probability theory that allows for the calculation of certain conditional probabilities.  Conditional probabilities are just those probabilities that reflect the influence of one event on the probability of another. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Simply put, in its most famous form, it states that the probability of a hypothesis given new data (&lt;strong&gt;P(H|D)&lt;/strong&gt;; called the posterior probability) is equal to the following equation: the probability of the observed data given the hypothesis (&lt;strong&gt;P(D|H)&lt;/strong&gt;; called the conditional probability), times the probability of the theory being true prior to new evidence (&lt;strong&gt;P(H)&lt;/strong&gt;; called the prior probability of H), divided by the probability of seeing that data, period (&lt;strong&gt;P(D&lt;/strong&gt;); called the marginal probability of D).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Formally, the equation looks like this:  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://upload.wikimedia.org/math/3/1/3/313f04bd948b0c467d53cfc822a0fc8a.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The significance of Bayes theorem is largely due to its proper use being a point of contention between schools of thought on probability.  To a subjective Bayesian (that interprets probability as being subjective degrees of belief) Bayes' theorem provides the cornerstone for theory testing, theory selection and other practices, by plugging their subjective probability judgments into the equation, and running with it.  To a frequentist (that interprets probability as &lt;a href=&quot;http://en.wikipedia.org/wiki/Frequency_%28statistics%29&quot; rel=&quot;nofollow&quot;&gt;limiting relative frequencies&lt;/a&gt;), this use of Bayes' theorem is an abuse, and they strive to instead use meaningful (non-subjective) priors (as do objective Bayesians under yet another interpretation of probability). &lt;/p&gt;&#xA;" OwnerUserId="39" LastEditorUserId="39" LastEditorDisplayName="" LastEditDate="2010-07-27T17:15:57.083" LastActivityDate="2010-07-27T17:15:57.083" CommentCount="1" />
  <row Id="825" PostTypeId="1" AcceptedAnswerId="827" CreationDate="2010-07-27T17:04:37.710" Score="13" ViewCount="630" Body="&lt;p&gt;I have R-scripts for reading large amounts of csv data from different files and then perform machine learning tasks such as svm for classification.&#xA;&lt;br&gt;&#xA;Are there any libraries for making use of multiple cores on the server for R.&#xA;&lt;br&gt;&#xA;or &#xA;&lt;br&gt;&#xA;What is most suitable way to achieve that?&lt;/p&gt;&#xA;" OwnerUserId="480" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-14T18:01:00.103" LastActivityDate="2010-10-14T18:01:00.103" Title="Any suggestions for making R code use multiple processors?" Tags="&lt;r&gt;&lt;parallel-computing&gt;&lt;multicore&gt;" AnswerCount="5" CommentCount="8" FavoriteCount="3" />
  <row Id="826" PostTypeId="2" ParentId="643" CreationDate="2010-07-27T17:06:17.613" Score="3" ViewCount="" Body="&lt;p&gt;I like to demonstrate sampling variation and essentially the Central Limit Theorem through an &quot;in-class&quot; exercise. Everybody in the class of say 100 students writes their age on a piece of paper. All pieces of paper are the same size and folded in the same fashion after I've calculated the average. This is the population and I calculate the average age. Then each student randomly selects 10 pieces of paper, writes down the ages and returns them to the bag. (S)he calculates the mean and passes the bag along to the next student. Eventually we have 100 samples of 10 students each estimating the population mean which we can describe through a histogram and some descriptive statistics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We then repeat the demonstration this time using a set of 100 &quot;opinions&quot; that replicate some Yes/No question from recent polls e.g. If the (British General) election were called tomorrow would you consider voting for the British National Party. Students them sample 10 of these opinions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At the end we've demonstrated sampling variation, the Central Limit Theorem, etc with both continuous and binary data.&lt;/p&gt;&#xA;" OwnerUserId="215" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-06-02T08:09:07.907" LastActivityDate="2011-06-02T08:09:07.907" CommunityOwnedDate="2011-06-02T11:09:11.620" />
  <row Id="827" PostTypeId="2" ParentId="825" CreationDate="2010-07-27T17:06:30.563" Score="8" ViewCount="" Body="&lt;p&gt;If it's on Linux, then the most straight-forward is &lt;a href=&quot;http://cran.r-project.org/web/packages/multicore/index.html&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;multicore&lt;/strong&gt;&lt;/a&gt;.  Beyond that, I suggest having a look at &lt;a href=&quot;http://www.stats.uwo.ca/faculty/yu/Rmpi/&quot; rel=&quot;nofollow&quot;&gt;MPI&lt;/a&gt; (especially with the &lt;a href=&quot;http://cran.r-project.org/web/packages/snow/index.html&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;snow&lt;/strong&gt;&lt;/a&gt; package).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More generally, have a look at:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The &lt;a href=&quot;http://cran.r-project.org/web/views/HighPerformanceComputing.html&quot; rel=&quot;nofollow&quot;&gt;High-Performance Computing view&lt;/a&gt; on CRAN.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://cran.r-project.org/web/views/HighPerformanceComputing.html&quot; rel=&quot;nofollow&quot;&gt;&quot;State of the Art in Parallel Computing with R&quot;&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Lastly, I recommend using the &lt;a href=&quot;http://cran.r-project.org/web/packages/foreach/index.html&quot; rel=&quot;nofollow&quot;&gt;foreach&lt;/a&gt; package to abstract away the parallel backend in your code.  That will make it more useful in the long run.&lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-07-27T17:06:30.563" CommentCount="6" />
  <row Id="828" PostTypeId="2" ParentId="825" CreationDate="2010-07-27T17:12:15.030" Score="3" ViewCount="" Body="&lt;p&gt;Shane is correct. Both &lt;a href=&quot;http://cran.r-project.org/package=multicore&quot; rel=&quot;nofollow&quot;&gt;multicore&lt;/a&gt; and &lt;a href=&quot;http://cran.r-project.org/package=Rmpi&quot; rel=&quot;nofollow&quot;&gt;Rmpi&lt;/a&gt; are winners.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Slightly broader coverage of the topic is in the &lt;a href=&quot;http://cran.r-project.org/web/views/HighPerformanceComputing.html&quot; rel=&quot;nofollow&quot;&gt;CRAN Task View on High-Performance Computing&lt;/a&gt;.  This also links to a fairly recent survey article on &lt;a href=&quot;http://www.jstatsoft.org/v31/i01/&quot; rel=&quot;nofollow&quot;&gt;Parallel Computing with R&lt;/a&gt; from JSS.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lastly, a few hands-on examples and tips are in the &lt;em&gt;Intro to HPC with R&lt;/em&gt; tutorial I give once in a while -- see my &lt;a href=&quot;http://dirk.eddelbuettel.com&quot; rel=&quot;nofollow&quot;&gt;presentations page&lt;/a&gt; for the most recent copy from last week at useR.&lt;/p&gt;&#xA;" OwnerUserId="334" LastActivityDate="2010-07-27T17:12:15.030" CommentCount="3" />
  <row Id="829" PostTypeId="2" ParentId="806" CreationDate="2010-07-27T17:15:37.367" Score="3" ViewCount="" Body="&lt;p&gt;Whether one can normalize a non-normal data set depends on the application.  For example, data normalization is required for many statistical tests (i.e. calculating a z-score, t-score, etc.)  Some tests are more prone to failure when normalizing non-normal data, while some are more resistant (&quot;robust&quot; tests).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One less-robust statistic is the mean, which is sensitive to outliers (i.e. non-normal data).  Alternatively, the median is less sensitive to outliers (and therefore more robust).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A great example of non-normal data when many statistics fail is bi-modally distributed data.  Because of this, it's always good practice to visualize your data as a frequency distribution (or even better, test for normality!)&lt;/p&gt;&#xA;" OwnerUserId="491" LastActivityDate="2010-07-27T17:15:37.367" />
  <row Id="830" PostTypeId="2" ParentId="825" CreationDate="2010-07-27T17:18:36.487" Score="3" ViewCount="" Body="&lt;p&gt;Both Shane and Dirk's responses  are spot on. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Nevertheless, you might wanna take a look at a commercial version of R, called &lt;a href=&quot;http://www.revolutionanalytics.com/&quot; rel=&quot;nofollow&quot;&gt;Revolution R&lt;/a&gt; which is built to deal with big datasets and run on multiple cores. This software is free for academics (which might be your case, I dont know)&lt;/p&gt;&#xA;" OwnerUserId="447" LastActivityDate="2010-07-27T17:18:36.487" CommentCount="2" />
  <row Id="831" PostTypeId="2" ParentId="790" CreationDate="2010-07-27T17:19:27.563" Score="1" ViewCount="" Body="&lt;p&gt;In the case of log-normally distributed data .... the geometric mean is a better measure of central tendancy than the arithmetic mean. I mean I would guess they look at the paper and have seen a log-normal distribution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Spots ... makes me think its referring to probes from a microarray .. in which case they do tend to do log-normally distributed (can't find the reference though sorry).&lt;/p&gt;&#xA;" OwnerUserId="494" LastActivityDate="2010-07-27T17:19:27.563" />
  <row Id="832" PostTypeId="2" ParentId="643" CreationDate="2010-07-27T17:37:35.300" Score="2" ViewCount="" Body="&lt;p&gt;If you use Stata, you can use the -clt- command that creates graphs of sampling distributions, see&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.ats.ucla.edu/stat/stata/ado/teach/clt.htm&quot; rel=&quot;nofollow&quot;&gt;http://www.ats.ucla.edu/stat/stata/ado/teach/clt.htm&lt;/a&gt;&lt;/p&gt;&#xA;" LastActivityDate="2010-07-27T17:37:35.300" CommunityOwnedDate="2011-06-02T11:09:11.620" CommentCount="1" />
  <row Id="833" PostTypeId="2" ParentId="812" CreationDate="2010-07-27T17:44:35.993" Score="8" ViewCount="" Body="&lt;p&gt;Yes and no. At the theoretical level, both cases can use similar techniques and frameworks (an excellent example being Gaussian process regression).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The critical difference is the assumptions used to prevent overfitting (regularization):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;In the functional case, there is usually some assumption of smoothness, in other words, values occurring close to each other should be similar in some systematic way. This leads to the use of techniques such as splines, loess, Gaussian processes, etc.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;In the high-dimensional case, there is usually an assumption of sparsity: that is, only a subset of the dimensions will have any signal. This leads to techniques aiming at identifying those dimensions (Lasso, LARS, slab-and-spike priors, etc.)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;UPDATE:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I didn't really think about wavelet/Fourier methods, but yes, the thresholding techniques used for such methods are aiming for sparsity in the projected space. Conversely, some high-dimensional techniques assume a projection on to a lower-dimensional manifold (e.g. principal component analysis), which is a type of smoothness assumption.&lt;/p&gt;&#xA;" OwnerUserId="495" LastEditorUserId="495" LastEditorDisplayName="" LastEditDate="2010-07-28T11:37:58.753" LastActivityDate="2010-07-28T11:37:58.753" CommentCount="1" />
  <row Id="834" PostTypeId="1" CreationDate="2010-07-27T17:46:25.223" Score="2" ViewCount="644" Body="&lt;p&gt;Hello all,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've heard that AIC can be used to choose among several models (which regressor to use).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But i would like to understand formally what it is in a kind of &quot;advanced undergraduated&quot; level,which i think would be something formal but with intuition arising from the formula.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And is it possible to implement AIC in stata with complex survey data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you very much in advance!!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regards!&lt;/p&gt;&#xA;" OwnerUserId="498" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-29T00:34:45.513" LastActivityDate="2011-04-29T00:34:45.513" Title="What is AIC? Looking for a formal but intuitive answer..." Tags="&lt;stata&gt;&lt;aic&gt;" AnswerCount="3" CommentCount="3" FavoriteCount="1" />
  <row Id="835" PostTypeId="2" ParentId="726" CreationDate="2010-07-27T17:57:48.807" Score="6" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://googleblog.blogspot.com/2009/02/from-height-of-this-place.html&quot; rel=&quot;nofollow&quot;&gt;Data is the sword of the 21st century, those who wield it well, the Samurai.&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="62" LastActivityDate="2010-07-27T17:57:48.807" CommunityOwnedDate="2010-07-27T17:57:48.807" CommentCount="2" />
  <row Id="836" PostTypeId="2" ParentId="834" CreationDate="2010-07-27T18:14:44.467" Score="2" ViewCount="" Body="&lt;p&gt;It is a heuristic, and as such, has been subjected to &lt;em&gt;extensive&lt;/em&gt; testing.  So when to trust it or not is not simple clear-cut and always-true decision.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At a rough approximation, it trades off goodness of fit and number of variables (&quot;degrees of freedom&quot;). Much more, as usual, &lt;a href=&quot;http://en.wikipedia.org/wiki/Akaike_information_criterion&quot; rel=&quot;nofollow&quot;&gt;at the Wikipedia article about AIC&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="334" LastActivityDate="2010-07-27T18:14:44.467" />
  <row Id="837" PostTypeId="1" AcceptedAnswerId="858" CreationDate="2010-07-27T18:25:40.277" Score="0" ViewCount="356" Body="&lt;p&gt;Hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am creating multiple logistic regression models using lrm from Harrell's Design package in R.  One model I would like to make is the model with no predictors.  eg I want to predict a constant c such that &lt;/p&gt;&#xA;&#xA;&lt;p&gt;logit(Y) ~ c&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know I how to compute c (divide the number of &quot;1&quot;s by the total), what I would like is to use lrm so I can manipulate it as a model in a consistent way with the other models I am making.  Is this possible, and if so how?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have tried so far&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(Design)&#xA;data(mtcars)&#xA;lrm(am ~ 1, data=mtcars)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which gives the error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Error in dimnames(stats) &amp;lt;- list(names(cof), c(&quot;Coef&quot;, &quot;S.E.&quot;, &quot;Wald Z&quot;,  :&#xA;    length of 'dimnames' [1] not equal to array extent&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and I have tried&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;lrm(am ~ ., data=mtcars)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But this uses all the predictors, rather then none of the predictors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance&lt;/p&gt;&#xA;" OwnerUserId="501" LastEditorUserId="501" LastEditorDisplayName="" LastEditDate="2010-07-28T05:20:42.387" LastActivityDate="2010-07-28T05:20:42.387" Title="R lrm model with no predictors" Tags="&lt;r&gt;&lt;logistic-regression&gt;" AnswerCount="2" />
  <row Id="838" PostTypeId="2" ParentId="834" CreationDate="2010-07-27T18:28:36.730" Score="2" ViewCount="" Body="&lt;p&gt;Basically one needs a loss function in order to optimize anything. AIC provides the loss function which when minimized gives a &quot;optimal&quot;* model which fits the given data. The AIC loss function (2k-2*log(L)) tries to formulate the bias variance trade off that every statistical modeler faces when fitting a model to finite set of data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words while fitting a model if you increase the number of parameters you will improve the log likelihood but will run into the danger of over fitting. The AIC penalizes for increasing the number of parameters thus minimizing the AIC selects the model where the improvement in log likelihood is not worth the penalty for increasing the number of parameters.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Note that when I say optimal model it is optimal in the sense that the model minimizes the AIC. There are other criteria (e.g. BIC) which may give other &quot;optimal&quot; models.  &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I don't have any experience with stata so cannot help you with the other part of the question. &lt;/p&gt;&#xA;" OwnerUserId="288" LastActivityDate="2010-07-27T18:28:36.730" />
  <row Id="839" PostTypeId="2" ParentId="743" CreationDate="2010-07-27T18:33:51.930" Score="4" ViewCount="" Body="&lt;p&gt;Though it is not generally labeled as Bayesian search theory, such methods are pretty widely used in oil exploration.  There are, however, important differences in the standard examples that drive different features of their respective modeling problems.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the case of lost vessel exploration (in Bayesian search theory), we are looking for a specific point on the sea floor (one elevation), with a  distributions modeling the likelihood of its resting location, and another distribution modeling the likelihood of finding the boat were it at that depth.  These distributions are then guide search, and are continuously updated through the results of the guided search.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Though similar, oil exploration is fraught with complicating features (multiple sampling depths, high sampling costs, variable yields, multiple geological indicators, drilling cost, etc.) that necessitate methods that go beyond what is considered in the prior example.  See &lt;a href=&quot;http://www.nek.lu.se/ryde/NordicEcont09/Papers/levitt.pdf&quot; rel=&quot;nofollow&quot;&gt;Learning through Oil and Gas Exploration&lt;/a&gt; for an overview of these complicating factors and a way to model them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, yes, it may be said that the oil exploration problem is different in magnitude, but not kind from lost vessel exploration, and thus similar methods may be fruitfully applied. Finally, a quick literature search reveals many different modeling approaches, which is not too surprising, given the complicated nature of the problem.&lt;/p&gt;&#xA;" OwnerUserId="39" LastEditorUserId="39" LastEditorDisplayName="" LastEditDate="2010-07-27T18:39:12.480" LastActivityDate="2010-07-27T18:39:12.480" />
  <row Id="840" PostTypeId="2" ParentId="73" CreationDate="2010-07-27T18:33:56.767" Score="1" ViewCount="" Body="&lt;p&gt;I work with both R and &lt;a href=&quot;http://en.wikipedia.org/wiki/MATLAB&quot; rel=&quot;nofollow&quot;&gt;MATLAB&lt;/a&gt; and I use &lt;a href=&quot;http://cran.r-project.org/web/packages/R.matlab/index.html&quot; rel=&quot;nofollow&quot;&gt;R.matlab&lt;/a&gt; a lot to transfer data between the two.&lt;/p&gt;&#xA;" OwnerUserId="288" LastEditorUserId="509" LastEditorDisplayName="" LastEditDate="2010-08-09T08:13:30.967" LastActivityDate="2010-08-09T08:13:30.967" CommunityOwnedDate="2010-07-27T18:33:56.767" />
  <row Id="841" PostTypeId="1" AcceptedAnswerId="844" CreationDate="2010-07-27T18:38:02.297" Score="11" ViewCount="319" Body="&lt;p&gt;I have $N$ paired observations ($X_i$, $Y_i$) drawn from a common unknown distribution, which has finite first and second moments, and is symmetric around the mean.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let $\sigma_X$ the standard deviation of $X$ (unconditional on $Y$), and $\sigma_Y$ the same for Y. I would like to test the hypothesis  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$H_0$: $\sigma_X = \sigma_Y$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$H_1$: $\sigma_X \neq \sigma_Y$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone know of such a test? I can assume in first analysis that the distribution is normal, although the general case is more interesting. I am looking for a closed-form solution. Bootstrap is always a last resort.&lt;/p&gt;&#xA;" OwnerUserId="30" LastEditorUserId="223" LastEditorDisplayName="" LastEditDate="2011-02-10T07:50:44.253" LastActivityDate="2011-03-13T20:31:13.040" Title="Comparing the variance of paired observations" Tags="&lt;distributions&gt;&lt;hypothesis-testing&gt;&lt;standard-deviation&gt;&lt;normal-distribution&gt;" AnswerCount="5" CommentCount="2" FavoriteCount="2" />
  <row Id="842" PostTypeId="2" ParentId="825" CreationDate="2010-07-27T18:39:32.153" Score="4" ViewCount="" Body="&lt;p&gt;I noticed that the previous answers lack some general HPC considerations.&lt;br&gt;&#xA;First of all, neither of those packages will enable you to run &lt;strong&gt;one&lt;/strong&gt; SVM in parallel. So what you can speed up is parameter optimization or cross-validation, still you must write your own functions for that. Or of course you may run the job for different datasets in parallel, if it is a case.&lt;br&gt;&#xA;The second issue is memory; if you want to spread calculation over a few physical computers, there is no free lunch and you must copy the data -- here you must consider if it makes sense to predistribute a copy of data across computers to save some communication. On the other hand if you wish to use multiple cores on one computer, than the multicore is especially appropriate because it enables all child processes to access the memory of the parent process, so you can save some time and a lot of memory space.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-07-27T18:39:32.153" CommentCount="1" />
  <row Id="843" PostTypeId="2" ParentId="798" CreationDate="2010-07-27T18:47:25.880" Score="0" ViewCount="" Body="&lt;p&gt;If I need to determine the number of bins programmatically I usually start out with a histogram that has way more bins than needed. Once the histogram is filled I then combine bins until I have enough entries per bin for the method I am using, e.g. if I want to model Poisson-uncertainties in a counting experiment with uncertainties from a normal distribution until I have more than something like 10 entries.&lt;/p&gt;&#xA;" OwnerUserId="56" LastActivityDate="2010-07-27T18:47:25.880" />
  <row Id="844" PostTypeId="2" ParentId="841" CreationDate="2010-07-27T18:48:54.927" Score="3" ViewCount="" Body="&lt;p&gt;You could use the fact that the &lt;a href=&quot;http://en.wikipedia.org/wiki/Variance#Distribution_of_the_sample_variance&quot; rel=&quot;nofollow&quot;&gt;distribution of the sample variance&lt;/a&gt; is a chi square distribution centered at the true variance. Under your null hypothesis, your test statistic would be the difference of two chi squared random variates centered at the same unknown true variance. I do not know whether the difference of two chi-squared random variates is an identifiable distribution but the above may help you to some extent.&lt;/p&gt;&#xA;" LastActivityDate="2010-07-27T18:48:54.927" CommentCount="1" />
  <row Id="845" PostTypeId="2" ParentId="841" CreationDate="2010-07-27T19:02:09.860" Score="4" ViewCount="" Body="&lt;p&gt;If you want to go down the non-parametric route you could always try the squared ranks test. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the unpaired case, the assumptions for this test (taken from &lt;a href=&quot;http://www.stat.wvu.edu/~xiawang/courses/stat551/hwk/hw14.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;) are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Both samples are random samples from their respective populations.&lt;/li&gt;&#xA;&lt;li&gt;In addition to independence within each sample there is mutual independence between&#xA;the two samples.&lt;/li&gt;&#xA;&lt;li&gt;The measurement scale is at least interval.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;These &lt;a href=&quot;http://www.stat.wvu.edu/~xiawang/courses/stat551/hwk/hw14.pdf&quot; rel=&quot;nofollow&quot;&gt;lecture notes&lt;/a&gt; describe the unpaired case in detail.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the paired case you will have to change this procedure slightly. Midway down &lt;a href=&quot;http://www.itl.nist.gov/div898/software/dataplot/refman1/auxillar/signrank.htm&quot; rel=&quot;nofollow&quot;&gt;this page&lt;/a&gt; should give you an idea of where to start.&lt;/p&gt;&#xA;" OwnerUserId="8" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-07-28T00:08:20.203" LastActivityDate="2010-07-28T00:08:20.203" CommentCount="4" />
  <row Id="846" PostTypeId="1" AcceptedAnswerId="851" CreationDate="2010-07-27T19:10:53.667" Score="1" ViewCount="241" Body="&lt;p&gt;Hi, I'm working on regression models in STATISTICA application and I need to know what is Fisher-Snedecor distribution for and how to analyze my regression model in this distribution. What the significance level means? What is v1 and v2? I need an explanation and little tutorial on real data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance!&lt;/p&gt;&#xA;" OwnerUserId="503" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-08-11T12:58:41.393" LastActivityDate="2010-08-11T12:58:41.393" Title="Regression output and Fisher-Snedecor distribution" Tags="&lt;regression&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="847" PostTypeId="1" CreationDate="2010-07-27T19:12:53.037" Score="0" ViewCount="79" Body="&lt;p&gt;GWO = Google Website Optimizer&lt;/p&gt;&#xA;&#xA;&lt;p&gt;GWO is a tool provided by Google to do &lt;a href=&quot;http://en.wikipedia.org/wiki/A/B_testing&quot; rel=&quot;nofollow&quot;&gt;A/B&lt;/a&gt; and &lt;a href=&quot;http://en.wikipedia.org/wiki/Multivariate_testing&quot; rel=&quot;nofollow&quot;&gt;MVT&lt;/a&gt; experiments on websites.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This has been an unanswered question for a long time so I thought I'd ask it here and see if I can get any help. Here is some documentation (clues) that Google has published about the statistics used:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.google.com/support/websiteoptimizer/bin/answer.py?hl=en&amp;amp;answer=61146#stats&quot; rel=&quot;nofollow&quot;&gt;All about statistics&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.google.com/support/websiteoptimizer/bin/answer.py?hl=en&amp;amp;answer=74818&quot; rel=&quot;nofollow&quot;&gt;Fractional versus Full Factorial analysis&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Example Data (&lt;a href=&quot;http://dl.dropbox.com/u/2511052/gwoTestResults/WebOptimizer-summary.pdf&quot; rel=&quot;nofollow&quot;&gt;summary pdf&lt;/a&gt;):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://dl.dropbox.com/u/2511052/gwoTestResults/WebsiteOptimizer.xml&quot; rel=&quot;nofollow&quot;&gt;XML&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://dl.dropbox.com/u/2511052/gwoTestResults/WebsiteOptimizer.csv&quot; rel=&quot;nofollow&quot;&gt;CSV&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I (as well as many others) are interested in discivering the math used by Google, becuase no one has been able to reproduce their calculations yet.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you in advance for any help that you can provide.&lt;/p&gt;&#xA;" OwnerUserId="500" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-16T07:12:11.257" LastActivityDate="2010-09-16T07:12:11.257" Title="What statistical model is used to calculate the test results for GWO?" Tags="&lt;for-retag&gt;" AnswerCount="1" CommentCount="3" FavoriteCount="1" />
  <row Id="848" PostTypeId="2" ParentId="805" CreationDate="2010-07-27T19:19:46.097" Score="0" ViewCount="" Body="&lt;p&gt;As an addition to Harvey's response if you have a small sample of data then there are alternatives to the standard error approximation formula shown above. I would recommend using software, one example would be to use the &lt;code&gt;binom.test&lt;/code&gt; function in R, to calculate these confidence intervals rather than doing it &quot;by hand&quot;.&lt;/p&gt;&#xA;" LastActivityDate="2010-07-27T19:19:46.097" />
  <row Id="849" PostTypeId="2" ParentId="612" CreationDate="2010-07-27T19:22:33.407" Score="9" ViewCount="" Body="&lt;p&gt;Principal Components Analysis (PCA) and Common Factor Analysis (CFA) are distinct methods.  Often, they produce similar results and PCA is used as the default extraction method in the SPSS Factor Analysis routines.  This undoubtedly results in a lot of confusion about the distinction between the two.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The bottom line is, these are two different models, conceptually.  In PCA, the components are actual orthogonal linear combinations that maximize the total variance.  In FA, the factors are linear combinations that maximize the shared portion of the variance--underlying &quot;latent constructs&quot;.  That's why FA is often called &quot;common factor analysis&quot;.  FA uses a variety of optimization routines and the result, unlike PCA, depends on the optimization routine used and starting points for those routines.  Simply there is not a single unique solution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In R, the factanal() function provides CFA with a maximum likelihood extraction.  So, you shouldn't expect it to reproduce an SPSS result which is based on a PCA extraction.  It's simply not the same model or logic.  I'm not sure if you would get the same result if you used SPSS's Maximum Likelihood extraction either as they may not use the same algorithm.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For better or for worse in R, you can, however, reproduce the mixed up &quot;factor analysis&quot; that SPSS provides as its default.  Here's the process in R.  With this code, I'm able to reproduce the SPSS Principal Component &quot;Factor Analysis&quot; result using this dataset.  (With the exception of the sign, which is indeterminant).  That result could also then be rotated using any of Rs available rotation methods.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Load the base dataset attitude to work with.&#xA;data(attitude)&#xA;# Compute eigenvalues and eigen vectors of the correlation matrix.&#xA;pfa.eigen&amp;lt;-eigen(cor(attitude))&#xA;# Print and note that eigen values are those produced by SPSS.&#xA;# Also note that SPSS will extract 2 components as eigen values &amp;gt; 1 = 2&#xA;pfa.eigen$values&#xA;# set a value for the number of factors (for clarity)&#xA;factors&amp;lt;-2&#xA;# Extract and transform two components.&#xA;pfa.eigen$vectors [ , 1:factors ]  %*% &#xA;+ diag ( sqrt (pfa.eigen$values [ 1:factors ] ),factors,factors )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="485" LastEditorUserId="485" LastEditorDisplayName="" LastEditDate="2010-07-27T19:27:45.063" LastActivityDate="2010-07-27T19:27:45.063" CommentCount="2" />
  <row Id="850" PostTypeId="2" ParentId="837" CreationDate="2010-07-27T19:28:38.133" Score="0" ViewCount="" Body="&lt;p&gt;Try creating a dummy variable with so many ones as the number of rows in mtcars and put it formula instead of one.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-07-27T19:28:38.133" CommentCount="1" />
  <row Id="851" PostTypeId="2" ParentId="846" CreationDate="2010-07-27T19:34:06.107" Score="4" ViewCount="" Body="&lt;p&gt;The Fisher Snedecor distribution is another name for the &lt;a href=&quot;http://en.wikipedia.org/wiki/F-distribution&quot; rel=&quot;nofollow&quot;&gt;F-distribution&lt;/a&gt;. The F-distribution comes from the ratio of two chi-squared variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In regression, you use the F-distribution in the &lt;a href=&quot;http://en.wikipedia.org/wiki/Analysis_of_variance&quot; rel=&quot;nofollow&quot;&gt;ANOVA&lt;/a&gt; table. This table gives you information about which variables/covariates should be in your model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The v1 and v2 are called the degrees of freedom. In simple linear regression,o.e.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Y = a + bx + epsilon&lt;/p&gt;&#xA;&#xA;&lt;p&gt;we have a single extra variable. So v1 = p = 1 and v2 = n-p-1 = n-2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't use STATISTICA, so I can't give you more detailed information.&lt;/p&gt;&#xA;" OwnerUserId="8" LastActivityDate="2010-07-27T19:34:06.107" />
  <row Id="852" PostTypeId="1" CreationDate="2010-07-27T19:52:26.970" Score="2" ViewCount="161" Body="&lt;p&gt;The coefficient on a logged explanatory variable when the dependent variable also is in log form is an elasticity (or the percentage change in the dependent variable if the explanatory variable changes by one percent).  Suppose I estimate a regression without logging the dependent variable but I use a log link in a General Linear Model (and family gaussian) while the explanatory variable remains in log form.  Is the coefficient on that explanatory variable still an elasticity?&lt;/p&gt;&#xA;" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-11-02T04:32:16.943" LastActivityDate="2010-11-02T04:32:16.943" Title="Elasticities Using GLM" Tags="&lt;econometrics&gt;&lt;glm&gt;" AnswerCount="1" />
  <row Id="853" PostTypeId="2" ParentId="847" CreationDate="2010-07-27T19:55:30.113" Score="1" ViewCount="" Body="&lt;p&gt;It sounds like they simply use Choice Modeling. You would use conditional logistic regression to analyze it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The people from JMP (SAS) have a nice (free) book about Design of Experiments and Choice Modeling.&lt;/p&gt;&#xA;" OwnerUserId="74" LastActivityDate="2010-07-27T19:55:30.113" />
  <row Id="854" PostTypeId="2" ParentId="192" CreationDate="2010-07-27T19:58:44.973" Score="2" ViewCount="" Body="&lt;p&gt;I would use &lt;strong&gt;Fisher's Exact Test&lt;/strong&gt;, even for large N. I wouldn't know why not. Any performance argument predates today's fast computers.&lt;/p&gt;&#xA;" OwnerUserId="506" LastActivityDate="2010-07-27T19:58:44.973" CommentCount="5" />
  <row Id="855" PostTypeId="2" ParentId="806" CreationDate="2010-07-27T20:05:20.027" Score="1" ViewCount="" Body="&lt;p&gt;I thought this was too obvious, until I saw this &lt;a href=&quot;http://stats.stackexchange.com/questions/795/normalization-of-series&quot; rel=&quot;nofollow&quot;&gt;question&lt;/a&gt;!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you normalise data, make sure you always have access to the raw data after normalisation. Of course, you could break this rule if you have a good reason, e.g. storage.&lt;/p&gt;&#xA;" OwnerUserId="8" LastActivityDate="2010-07-27T20:05:20.027" />
  <row Id="856" PostTypeId="1" AcceptedAnswerId="1306" CreationDate="2010-07-27T20:24:45.767" Score="7" ViewCount="361" Body="&lt;p&gt;Does anyone know of a variation of Fisher's Exact Test which takes weights into account? For instance &lt;a href=&quot;http://www.measuredhs.com/help/Datasets/sampling_weights.htm&quot; rel=&quot;nofollow&quot;&gt;sampling weights&lt;/a&gt;.&lt;br&gt;&#xA;So instead of the usual 2x2 cross table, every data point has a &quot;mass&quot; or &quot;size&quot; value weighing the point.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;A B weight&#xA;N N 1&#xA;N N 3&#xA;Y N 1&#xA;Y N 2&#xA;N Y 6&#xA;N Y 7&#xA;Y Y 1&#xA;Y Y 2&#xA;Y Y 3&#xA;Y Y 4&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Fisher's Exact Test then uses this 2x2 cross table:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;A\B  N  Y All&#xA; N   2  2   4&#xA; Y   2  4   6&#xA;All  4  6  10&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If we would take the weight as an 'actual' number of data points, this would result in:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;A\B  N  Y All&#xA; N   4 13  17&#xA; Y   3 10  13&#xA;All  7 23  30&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But that would result in much too high a confidence. One data point changing from N/Y to N/N would make a very large difference in the statistic.&lt;br&gt;&#xA;Plus, it wouldn't work if any weight contained fractions.&lt;/p&gt;&#xA;" OwnerUserId="506" LastEditorUserId="506" LastEditorDisplayName="" LastEditDate="2010-07-29T15:21:16.363" LastActivityDate="2011-04-24T08:04:05.637" Title="Fisher's Exact Test with weights?" Tags="&lt;hypothesis-testing&gt;" AnswerCount="3" FavoriteCount="1" />
  <row Id="857" PostTypeId="2" ParentId="775" CreationDate="2010-07-27T20:25:16.327" Score="2" ViewCount="" Body="&lt;p&gt;Operations Research began during wartime in the 1940s with scientists and others addressing problems in Radar operations, Anti-Submarine Warfare (ASW), and air operations. It is really a methodology to help decision makers choose a course of action by using an analytic framework that includes statistics, linear and non-linear programming, game theory, decision theory, etc. Statistics is one of many tools it uses.&lt;/p&gt;&#xA;" OwnerUserId="482" LastActivityDate="2010-07-27T20:25:16.327" />
  <row Id="858" PostTypeId="2" ParentId="837" CreationDate="2010-07-27T20:33:48.787" Score="0" ViewCount="" Body="&lt;p&gt;I'm not sure why this doesn't work with lrm.  However, R does Logistic Regression just fine with its own internal functions.  See GLM.  Here's your model, working...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;summary(glm(am~1, data = mtcars, family=binomial(link=logit)))&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, unless you need something that lrm() from design provides, then use GLM with the binomial logit link.&lt;/p&gt;&#xA;" OwnerUserId="485" LastActivityDate="2010-07-27T20:33:48.787" CommentCount="2" />
  <row Id="859" PostTypeId="1" AcceptedAnswerId="867" CreationDate="2010-07-27T21:07:20.127" Score="6" ViewCount="406" Body="&lt;p&gt;Sorry for the verbose background to this question:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Occasionally in investigations of animal behaviour, an experimenter is interested in the amount of time that a subject spends in different, pre-defined zones in a test apparatus. I've often seen this sort of data analyzed using ANOVA; however, I have never been entirely convinced of the validity of such analyses, given that ANOVA assumes the observations are independent, and they never actually are independent in these analyses (since more time spent in one zone means that less is spent in other zones!). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example,&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;D. R. Smith, C. D. Striplin, A. M.&#xA;  Geller, R. B. Mailman, J. Drago, C. P.&#xA;  Lawler, M. Gallagher, &lt;a href=&quot;http://www.sciencedirect.com/science/article/B6T0F-3THY168-F/2/007a74da6fbe693e297e50ec91dec141&quot; rel=&quot;nofollow&quot;&gt;Behavioural&#xA;  assessment of mice lacking D1A&#xA;  dopamine receptors&lt;/a&gt;, Neuroscience,&#xA;  Volume 86, Issue 1, 21 May 1998, Pages&#xA;  135-146&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In the above article, they reduce the degrees of freedom by 1 in order to compensate for the non-independence. However, I am not sure how such a manipulation can actually ameliorate this violation of ANOVA assumptions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Perhaps a chi-squared procedure might be more appropriate? What would you do to analyze data like this (preference for zones, based on time spent in zones)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="445" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-12-21T17:06:54.740" LastActivityDate="2010-12-21T17:06:54.740" Title="ANOVA with non-independent observations" Tags="&lt;anova&gt;" AnswerCount="4" FavoriteCount="2" />
  <row Id="860" PostTypeId="2" ParentId="834" CreationDate="2010-07-27T21:51:43.650" Score="4" ViewCount="" Body="&lt;p&gt;Let $f$ be your true distribution, and $g$ the family from which you are trying to fit your data. Then $\theta$, the maximum likelihood estimator of parameters of $g$, is a random variable. You could formulate model selection as finding the distribution family $g$ that minimizes the expected KL divergence between $f$ and $g(\theta)$, which can be written as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\text{Entropy}(f)-E_x E_y[\log(g(x|\theta(y)))]$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since you are minimizing over $g$, the Entropy($f$) term doesn't matter and you look for $g$ that maximizes $E_x E_y[\log(g(x|\theta(y)))]$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let $L(\theta(y)|y)$ be the likelihood of data $y$ according to $g(\theta)$. You could estimate $E_x E_y[\log(g(x|\theta(y)))]$ as $\log(L(\theta(y)|y))$ but that estimator is biased.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Akaike's showed that when $f$ belongs to family $g$ with dimension $k$, the following estimator is asymptotically unbiased&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\log(L(\theta(y)|y))-k$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Burnham has more details in this &lt;a href=&quot;http://faculty.washington.edu/skalski/classes/QERM597/papers_xtra/Burnham%20and%20Anderson.pdf&quot; rel=&quot;nofollow&quot;&gt;paper&lt;/a&gt;, also blog &lt;a href=&quot;http://www.emakalic.org/blog/?p=26&quot; rel=&quot;nofollow&quot;&gt;post&lt;/a&gt; by Enes Makalic has further explanation and references&lt;/p&gt;&#xA;" OwnerUserId="511" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-11-18T00:36:44.833" LastActivityDate="2010-11-18T00:36:44.833" />
  <row Id="861" PostTypeId="2" ParentId="743" CreationDate="2010-07-27T22:33:05.287" Score="3" ViewCount="" Body="&lt;p&gt;There is a free book on Geostatistical Mapping with R &lt;a href=&quot;http://spatial-analyst.net/book/&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; it might help your problem.&lt;/p&gt;&#xA;" OwnerUserId="481" LastActivityDate="2010-07-27T22:33:05.287" />
  <row Id="862" PostTypeId="2" ParentId="798" CreationDate="2010-07-28T00:23:22.107" Score="9" ViewCount="" Body="&lt;p&gt;The Freedman-Diaconis rule is very robust and works well in practice. The bin-width is set to $h=2*\text{IQR}*n^{-1/3}$. So the number of bins is (max-min)/$h$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In R, you can use &lt;code&gt;hist(x,breaks=&quot;FD&quot;)&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="159" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2011-01-07T09:27:45.153" LastActivityDate="2011-01-07T09:27:45.153" CommentCount="3" />
  <row Id="863" PostTypeId="2" ParentId="859" CreationDate="2010-07-28T00:29:41.487" Score="3" ViewCount="" Body="&lt;p&gt;Mike,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I agree that an ANOVA based on total time probably isn't the correct approach here.  Further, I'm not convinced that Chi Sqaure solves your problem.  Chi square will respect the idea that you can't be in two locations at the same time, but it doesn't address the problem that there are likely dependencies between time N and time N+1.  In regards to this second issue, I see some analogies between your situation and what people run into with eye and mouse tracking data. A multinomial model of some sort may serve your purposes well.  Unfortunately, the details of that type of model are beyond my expertise.  I'm sure some statistics book somewhere has a nice little primer on that topic, but off the top of my head I'd point you towards:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Barr D.J. (2008) Analyzing ‘visual world’ eyetracking data using multilevel logistic regression. Journal of Memory and Language, Special Issue: Emerging Data Analysis (59) pp 457-474&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://r-forge.r-project.org/projects/gmpm/&quot; rel=&quot;nofollow&quot;&gt;https://r-forge.r-project.org/projects/gmpm/&lt;/a&gt; is a non-parametric approach to the same issue being developed by Dr. Barr&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If anything, both of those sources should be more than complete because they get into how to analyze the time course of the position.&lt;/p&gt;&#xA;" OwnerUserId="196" LastActivityDate="2010-07-28T00:29:41.487" />
  <row Id="864" PostTypeId="2" ParentId="859" CreationDate="2010-07-28T01:00:22.877" Score="3" ViewCount="" Body="&lt;p&gt;I am going to suggest an answer that is very different from that of a traditional ANOVA. Let T be the total time that is available for an animal to spend in all the zones. You could define T as the total amount of waking time or some such. Suppose that you have J zones. Then by definition you have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sum T_j = T&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could normalize the above by dividing the lhs and the rhs by T and you get&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sum P_j = 1&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where P_j is the proportion of time that an animal spends in zone j.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now the question you have is if P_j is significantly different from 1 / J for all j.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could assume that P_j follows a &lt;a href=&quot;http://en.wikipedia.org/wiki/Dirichlet_distribution&quot; rel=&quot;nofollow&quot;&gt;dirichlet distribution&lt;/a&gt; and estimate two models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Null Model&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Set the parameters of the distribution such that P_j = 1 / J. (Setting the parameters of the distribution to 1 will do.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Alternative Model&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Set the parameters of the distribution to be a function of zone specific covariates. You could then estimate the model parameters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You would choose the alternative model if it outperforms the null model on some critera (e.g., likelihood ratio).&lt;/p&gt;&#xA;" LastActivityDate="2010-07-28T01:00:22.877" />
  <row Id="865" PostTypeId="2" ParentId="225" CreationDate="2010-07-28T01:06:56.963" Score="0" ViewCount="" Body="&lt;p&gt;Just to expand on Rob's answer a bit, suppose that we want to know the cumulative distribution function (CDF) of the highest value of N independent draws from a standard normal distribution, X1, ..., XN. Call this highest value Y1, the first order statistic. Then the CDF is:&lt;br&gt;&#xA;Pr(Y1 &amp;lt; x) = Pr(max(X1, ..., XN) &amp;lt; x) = Pr(X1 &amp;lt; x, ..., XN &amp;lt; X)&lt;br&gt;&#xA;= Pr(X1 &amp;lt; x) * ... Pr(XN &amp;lt; x) = Pr(X &amp;lt; x)^100, &lt;br&gt;&#xA;where the second line follows by independence of the draws.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rob uses the standard notation that Phi(x) is defined as Pr(X &amp;lt; x) for a standard normal---&lt;i&gt;i.e.,&lt;/i&gt; Phi(x) is the standard normal CDF.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The probability density function (PDF) of the first order statistic is just the derivative of the CDF with respect to X:&lt;br&gt;&#xA;Pr(X1 = x) = 100*Pr(X &amp;lt; x)^99 Pr(X = x),&lt;br&gt;&#xA;the CDF at x raised to 99 (N-1) times the PDF at x times 100.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope that helps,&lt;br&gt;&#xA;Charlie&lt;/p&gt;&#xA;" OwnerUserId="401" LastActivityDate="2010-07-28T01:06:56.963" />
  <row Id="866" PostTypeId="1" AcceptedAnswerId="876" CreationDate="2010-07-28T01:10:18.423" Score="9" ViewCount="695" Body="&lt;p&gt;Say I want to estimate a large number of parameters, and I want to penalize some of them because I believe they should have little effect compared to the others. How do I decide what penalization scheme to use? When is ridge regression more appropriate? When should I use lasso?&lt;/p&gt;&#xA;" OwnerUserId="455" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-08-09T12:25:13.010" LastActivityDate="2011-03-20T20:13:05.083" Title="When should I use lasso vs ridge?" Tags="&lt;regression&gt;&lt;lasso&gt;&lt;ridge-regression&gt;" AnswerCount="4" CommentCount="1" FavoriteCount="6" />
  <row Id="867" PostTypeId="2" ParentId="859" CreationDate="2010-07-28T01:17:57.410" Score="3" ViewCount="" Body="&lt;p&gt;(Caveat Emptor: I'm not an expert in this area)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you just want to talk about differences in time spent per location, then submitting the &quot;time-per-location&quot; data as counts in a multinomial mixed model (see the MCMCglmm package for R), using subject as a random effect, should do the trick. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to talk about differences in location preference &lt;em&gt;through&lt;/em&gt; time, then maybe bin time to reasonable intervals (maybe to the resolution of your timing device?), classify each interval according to the mouse's location at that time (eg. if 3 locations, each interval gets labelled either 1, 2, or 3), and again use a multinomial mixed effects model with subject as a random effect but this time add interval as a fixed effect (though possibly only after factorizing interval, which drops power but should help capture non-linearities through time).&lt;/p&gt;&#xA;" OwnerUserId="364" LastActivityDate="2010-07-28T01:17:57.410" />
  <row Id="868" PostTypeId="1" AcceptedAnswerId="917" CreationDate="2010-07-28T03:15:34.473" Score="3" ViewCount="333" Body="&lt;p&gt;I created a quick fun Excel Spreadsheet tonight to try and predict which video games I'll enjoy if I buy them. I'm wondering if this quick example makes sense from a Logistic Regression perspective and if I am computing all of the values correctly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unfortunately, if I did everything correctly I doubt I have much to look forward to on my XBOX or PS3 ;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I laid out a few categories and weighted them like so (Real spreadsheet lists twice as many or so):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;&#xA;4&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; 4&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;3&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;1&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Visually Stunning &amp;nbsp; Exhilirating&amp;nbsp; Artistic&amp;nbsp; Sporty&#xA;&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then I went through some games I have and rated them in each category (ratings of 0-4). I then set a separate cell to be the value of Beta_0 and tuned that until the resulting percentages all looked about right.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Next I entered in my expected ratings for the new games I &lt;strong&gt;was&lt;/strong&gt; looking forward to and got percentages for those.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example:&#xA;Beta_0 := -35&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;&#xA;4&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; 4&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;3&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;1&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Visually Stunning &amp;nbsp; Exhilirating&amp;nbsp; Artistic&amp;nbsp; Sporty&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;4&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;0&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;1&#xA;&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would be calculated as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P = 1 / [1 + e^(-35 + (4*4 + 4*4 + 3*0 + 1*1)]&#xA;P = 88.1%&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I were to automate the regression am I correct in thinking I'd be tuning Beta_0 to make it so the positive training examples come out high and the negative training examples come out low?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm completely new to this (just started today thanks to this site actually!) so please have no concern about bruising my ego, I'm eager to learn more.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="274" LastActivityDate="2011-01-07T01:55:24.827" Title="Training a Logistic Regression Model" Tags="&lt;logit&gt;&lt;logistic-regression&gt;" AnswerCount="4" CommentCount="1" FavoriteCount="2" />
  <row Id="869" PostTypeId="2" ParentId="868" CreationDate="2010-07-28T03:49:27.933" Score="2" ViewCount="" Body="&lt;p&gt;Usually in logistic regression you'd want &quot;successes&quot; to be 1 and &quot;failures&quot; to be 0, but so long as you are consistent in how you enter your data and interpret it, the coefficients don't really care.&lt;/p&gt;&#xA;" OwnerUserId="196" LastActivityDate="2010-07-28T03:49:27.933" CommentCount="1" />
  <row Id="870" PostTypeId="1" AcceptedAnswerId="956" CreationDate="2010-07-28T03:54:56.447" Score="5" ViewCount="1177" Body="&lt;p&gt;Given a list of p-values generated from independent tests, sorted in ascending order, one can use the &lt;a href=&quot;http://www.math.tau.ac.il/~ybenja/MyPapers/benjamini_hochberg1995.pdf&quot; rel=&quot;nofollow&quot;&gt;Benjamini-Hochberg procedure&lt;/a&gt; for &lt;a href=&quot;http://en.wikipedia.org/wiki/False_discovery_rate#Independent_tests&quot; rel=&quot;nofollow&quot;&gt;multiple testing correction&lt;/a&gt;. For each p-value, the Benjamini-Hochberg procedure allows you to calculate the False Discovery Rate (FDR) for each of the p-values. That is, at each &quot;position&quot; in the sorted list of p-values, it will tell you what proportion of those are likely to be false rejections of the null hypothesis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is, are these FDR values to be referred to as &quot;&lt;strong&gt;&lt;a href=&quot;http://projecteuclid.org/DPubS?service=UI&amp;amp;version=1.0&amp;amp;verb=Display&amp;amp;handle=euclid.aos/1074290335&quot; rel=&quot;nofollow&quot;&gt;q-values&lt;/a&gt;&lt;/strong&gt;&quot;, or as &quot;&lt;strong&gt;corrected p-values&lt;/strong&gt;&quot;, or as something else entirely?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT 2010-07-12:&lt;/strong&gt; I would like to more fully describe the correction procedure we are using. First, we sort the test results in increasing order by their un-corrected original p-value. Then, we iterate over the list, calculating what I have been interpreting as &quot;the FDR expected if we were to reject the null hypothesis for this and all tests prior in the list,&quot; using the B-H correction, with an alpha equal to the observed, un-corrected p-value for the respective iteration. We then take, as what we've been calling our &quot;q-value&quot;, the maximum of the previously corrected value (FDR at iteration i - 1) or the current value (at i), to preserve monotonicity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Below is some Python code which represents this procedure:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def calc_benjamini_hochberg_corrections(p_values, num_total_tests):&#xA;    &quot;&quot;&quot;&#xA;    Calculates the Benjamini-Hochberg correction for multiple hypothesis&#xA;    testing from a list of p-values *sorted in ascending order*.&#xA;&#xA;    See&#xA;    http://en.wikipedia.org/wiki/False_discovery_rate#Independent_tests&#xA;    for more detail on the theory behind the correction.&#xA;&#xA;    **NOTE:** This is a generator, not a function. It will yield values&#xA;    until all calculations have completed.&#xA;&#xA;    :Parameters:&#xA;    - `p_values`: a list or iterable of p-values sorted in ascending&#xA;      order&#xA;    - `num_total_tests`: the total number of tests (p-values)&#xA;&#xA;    &quot;&quot;&quot;&#xA;    prev_bh_value = 0&#xA;    for i, p_value in enumerate(p_values):&#xA;        bh_value = p_value * num_total_tests / (i + 1)&#xA;        # Sometimes this correction can give values greater than 1,&#xA;        # so we set those values at 1&#xA;        bh_value = min(bh_value, 1)&#xA;&#xA;        # To preserve monotonicity in the values, we take the&#xA;        # maximum of the previous value or this one, so that we&#xA;        # don't yield a value less than the previous.&#xA;        bh_value = max(bh_value, prev_bh_value)&#xA;        prev_bh_value = bh_value&#xA;        yield bh_value&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="520" LastEditorUserId="520" LastEditorDisplayName="" LastEditDate="2010-08-12T16:07:55.730" LastActivityDate="2010-08-12T16:07:55.730" Title="Multiple hypothesis testing correction with Benjamini-Hochberg, p-values or q-values?" Tags="&lt;hypothesis-testing&gt;" AnswerCount="1" CommentCount="4" FavoriteCount="2" />
  <row Id="871" PostTypeId="1" AcceptedAnswerId="875" CreationDate="2010-07-28T04:08:23.973" Score="10" ViewCount="491" Body="&lt;p&gt;I realize this is pedantic and trite, but as a researcher in a field outside of statistics, with limited formal education in statistics, I always wonder if I'm writing &quot;p-value&quot; correctly. Specifically:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Is the &quot;p&quot; supposed to be capitalized?&lt;/li&gt;&#xA;&lt;li&gt;Is the &quot;p&quot; supposed to be italicized? (Or in mathematical font, in TeX?)&lt;/li&gt;&#xA;&lt;li&gt;Is there supposed to be a hyphen between &quot;p&quot; and &quot;value&quot;?&lt;/li&gt;&#xA;&lt;li&gt;Alternatively, is there no &quot;proper&quot; way of writing &quot;p-value&quot; at all, and any dolt will understand what I mean if I just place &quot;p&quot; next to &quot;value&quot; in some permutation of these options?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="520" LastActivityDate="2010-07-29T19:39:02.233" Title="Correct formatting (capitalization, italicization, hyphenation) of &quot;p-value&quot;?" Tags="&lt;hypothesis-testing&gt;" AnswerCount="4" CommentCount="1" FavoriteCount="2" />
  <row Id="872" PostTypeId="2" ParentId="871" CreationDate="2010-07-28T04:21:06.723" Score="3" ViewCount="" Body="&lt;p&gt;This seems to be a style issue with different journals and publishers adopting different conventions (or allowing a mixed muddle of styles depending on authors' preferences). My own preference, for what it's worth, is p-value, hyphenated with no italics and no capitalization.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-07-28T04:21:06.723" />
  <row Id="873" PostTypeId="2" ParentId="871" CreationDate="2010-07-28T04:24:05.240" Score="0" ViewCount="" Body="&lt;p&gt;The &lt;a href=&quot;http://www.amstat.org/publications/chance/assets/style.pdf&quot; rel=&quot;nofollow&quot;&gt;ASA House Style&lt;/a&gt; seems to recommend italicizing the p with hyphen: &lt;em&gt;p&lt;/em&gt;-value.  A google scholar search shows &lt;a href=&quot;http://scholar.google.com/scholar?q=p+value&amp;amp;hl=en&amp;amp;btnG=Search&amp;amp;as_sdt=80001&amp;amp;as_sdtp=on&quot; rel=&quot;nofollow&quot;&gt;varied spellings&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-07-28T04:24:05.240" />
  <row Id="874" PostTypeId="2" ParentId="866" CreationDate="2010-07-28T04:26:17.297" Score="2" ViewCount="" Body="&lt;p&gt;Ridge or lasso are forms of regularized linear regressions.  The regularization can also be interpreted as prior in a maximum a posteriori estimation method.  Under this interpretation, the ridge and the lasso make different assumptions on the class of linear transformation they infer to relate input and output data.   In the ridge, the coefficients of the linear transformation are normal distributed and in the lasso they are  Laplace distributed.  In the lasso, this makes it easier for the coefficients to be zero and therefore easier to eliminate some of your input variable as not contributing to the output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are also some practical considerations.  The ridge is a bit easier to implement and faster to compute, which may matter depending on the type of data you have. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have both implemented, use subsets of your data to find the ridge and the lasso and compare how well they work on the left out data.  The errors should give you an idea of which to use.   &lt;/p&gt;&#xA;" OwnerUserId="260" LastActivityDate="2010-07-28T04:26:17.297" />
  <row Id="875" PostTypeId="2" ParentId="871" CreationDate="2010-07-28T04:29:37.107" Score="10" ViewCount="" Body="&lt;p&gt;There do not appear to be &quot;standards&quot;.  For example:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The &lt;a href=&quot;http://www.nature.com/nature/authors/gta/#a5.6&quot; rel=&quot;nofollow&quot;&gt;Nature style guide&lt;/a&gt; refers to &quot;P value&quot;&lt;/li&gt;&#xA;&lt;li&gt;This &lt;a href=&quot;http://my.ilstu.edu/~jhkahn/apastats.html&quot; rel=&quot;nofollow&quot;&gt;APA style guide&lt;/a&gt; refers to &quot;&lt;em&gt;p&lt;/em&gt; value&quot;&lt;/li&gt;&#xA;&lt;li&gt;The &lt;a href=&quot;http://bloodjournal.hematologylibrary.org/authors/stylecheckforfigs.dtl&quot; rel=&quot;nofollow&quot;&gt;Blood style guide&lt;/a&gt; says:&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Capitalize and italicize the &lt;em&gt;P&lt;/em&gt; that introduces a &lt;em&gt;P&lt;/em&gt; value&lt;/li&gt;&#xA;&lt;li&gt;Italicize the &lt;em&gt;p&lt;/em&gt; that represents the Spearman rank correlation test&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/P-value&quot; rel=&quot;nofollow&quot;&gt;Wikipedia&lt;/a&gt; uses &quot;p-value&quot;, but italicizes the &lt;em&gt;p&lt;/em&gt; if used by itself&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;My brief, unscientific survey suggests that the most common combination is lower-case, italicized &lt;em&gt;p&lt;/em&gt; without a hyphen.&lt;/p&gt;&#xA;" OwnerUserId="163" LastActivityDate="2010-07-28T04:29:37.107" />
  <row Id="876" PostTypeId="2" ParentId="866" CreationDate="2010-07-28T05:55:31.407" Score="10" ViewCount="" Body="&lt;p&gt;Keep in mind that ridge regression can't zero out coefficients; thus, you either end up including all the coefficients in the model, or none of them. In contrast, the LASSO does both parameter shrinkage and variable selection automatically. If some of your covariates are highly correlated, you may want to look at the Elastic Net [3] instead of the LASSO.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd personally recommend using the Non-negative Garotte (NNG) [1] as its consistent in terms of estimation and variable selection [2]. Unlike LASSO and ridge regression, NNG requires an initial estimate that is then shrunk towards the origin. In the original paper, Breiman recommends the least squares solution for the initial estimate (you may however want to start the search from a ridge regression solution and use something like GCV to select the penalty parameter).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In terms of available software, I've implemented the original NNG in MATLAB (based on Breiman's original FORTRAN code). You can download it from: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.emakalic.org/blog/wp-content/uploads/2010/04/nngarotte.zip&quot; rel=&quot;nofollow&quot;&gt;http://www.emakalic.org/blog/wp-content/uploads/2010/04/nngarotte.zip&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;BTW, if you prefer a Bayesian solution, check out [4,5].&lt;/p&gt;&#xA;&#xA;&lt;p&gt;References:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[1] Breiman, L. Better Subset Regression Using the Nonnegative Garrote Technometrics, 1995, 37, 373-384&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[2] Yuan, M. &amp;amp; Lin, Y. On the non-negative garrotte estimator Journal of the Royal Statistical Society (Series B), 2007, 69, 143-161&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[3] Zou, H. &amp;amp; Hastie, T. Regularization and variable selection via the elastic net Journal of the Royal Statistical Society (Series B), 2005, 67, 301-320&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[4] Park, T. &amp;amp; Casella, G. The Bayesian Lasso Journal of the American Statistical Association, 2008, 103, 681-686&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[5] Kyung, M.; Gill, J.; Ghosh, M. &amp;amp; Casella, G. Penalized Regression, Standard Errors, and Bayesian Lassos Bayesian Analysis, 2010, 5, 369-412&lt;/p&gt;&#xA;" OwnerUserId="530" LastActivityDate="2010-07-28T05:55:31.407" />
  <row Id="877" PostTypeId="1" CreationDate="2010-07-28T06:31:36.857" Score="4" ViewCount="233" Body="&lt;p&gt;Has anyone gone through some papers using Vector Error Correction Models in causality applications with more than one cointegration vectors, say two. I guess there will be more than one ECM terms. How to assess the endogeneity of the  left hand variables if t-stats on different ECM coefficients yield different (conflicting) results. For left hand side variables to be endogenous should we have both ECM coefficients negative and significant?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Javed Iqbal&lt;/p&gt;&#xA;" OwnerUserId="531" LastActivityDate="2010-07-28T06:31:36.857" Title="Time Series Econometrics: VECM with multiple cointegration vectors" Tags="&lt;econometrics&gt;" />
  <row Id="878" PostTypeId="2" ParentId="726" CreationDate="2010-07-28T06:49:12.123" Score="21" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Absence of evidence is not evidence of&#xA;  absence.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Carl Sagan&lt;/p&gt;&#xA;" LastEditorUserId="380" LastEditorDisplayName="" LastEditDate="2010-08-07T10:06:32.290" LastActivityDate="2010-08-07T10:06:32.290" CommunityOwnedDate="2010-07-28T06:49:12.123" CommentCount="2" />
  <row Id="879" PostTypeId="2" ParentId="798" CreationDate="2010-07-28T07:23:22.887" Score="3" ViewCount="" Body="&lt;p&gt;Maybe the paper &quot;&lt;a href=&quot;http://pubs.research.avayalabs.com/pdfs/ALR-2007-003-paper.pdf&quot; rel=&quot;nofollow&quot;&gt;Variations on the histogram&lt;/a&gt;&quot; by Denby and Mallows will be of interest:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;This new display which we term &quot;dhist&quot; (for diagonally-cut histogram) preserves the desirable features of both the equal-width hist and the equal-area hist. It will show tall narrow bins like the e-a hist when there are spikes in the data and will show isolated outliers just like the usual histogram.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;They also mention that code in R is available on request.&lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-07-28T07:23:22.887" />
  <row Id="880" PostTypeId="1" CreationDate="2010-07-28T08:15:40.827" Score="4" ViewCount="169" Body="&lt;p&gt;My question is about cross validation when there are many more variables than observations. To fix ideas, I propose to restrict to the classification framework in very high dimension (more features than observation).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Problem:&lt;/strong&gt; Assume that for each variable $i=1,\dots,p$ you have a measure of importance $T[i]$ than exactly measure the interest of feature $i$ for the classification problem. The problem of selecting a subset of feature to reduce optimally the classification error is then reduced to that of finding the number of features. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question:&lt;/strong&gt; What is the most efficient way to run cross validation in this case (cross validation scheme)? My question is not about how to write the code but on the version of cross validation to use when trying to find the number of selected feature (to minimize the classification error) but how to deal with the high dimension when doing cross validation (hence the problem above may be a bit like a 'toy problem' to discuss CV in high dimension). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Notations:&lt;/strong&gt;  $n$ is the size of the learning set, p the number of features (i.e. the dimension of the feature space).  By very high dimension I mean p&gt;&gt;n (for example $p=10000$ and $n=100$). &lt;/p&gt;&#xA;" OwnerUserId="223" LastEditorUserId="223" LastEditorDisplayName="" LastEditDate="2010-09-02T07:13:57.967" LastActivityDate="2010-09-02T07:13:57.967" Title="Cross validation in very high dimension (to select the number of used variables in very high dimensional classification)" Tags="&lt;machine-learning&gt;&lt;classification&gt;&lt;cross-validation&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="4" />
  <row Id="881" PostTypeId="1" AcceptedAnswerId="1189" CreationDate="2010-07-28T08:15:51.733" Score="5" ViewCount="165" Body="&lt;p&gt;Here's something I've wondered about for a while, but haven't been able to discover the correct terminology. Say you have a relatively complicated density function that you suspect might have a close approximation as a sum of (properly weighted) simpler density functions. Have such things been studied? I'm particularly interested in reading about any applications.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's one example I've found:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.soa.org/library/research/transactions-of-society-of-actuaries/1966/january/tsa66v18pt1n5211.pdf&quot; rel=&quot;nofollow&quot;&gt;Expansion of probability density functions as a sum of gamma densities with applications in risk theory&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="34" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-28T08:58:47.087" LastActivityDate="2011-03-28T08:58:47.087" Title="Series expansion of a density function" Tags="&lt;probability&gt;&lt;mixture&gt;&lt;pdf&gt;" AnswerCount="3" />
  <row Id="882" PostTypeId="2" ParentId="881" CreationDate="2010-07-28T08:21:47.367" Score="4" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Histogram density estimator&lt;/strong&gt; is estimating the density with a sum of &lt;strong&gt;piecewise functions&lt;/strong&gt; (density of a uniform).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;KDE&lt;/strong&gt; is using a sum of &lt;strong&gt;smooth function&lt;/strong&gt; (gaussian  is an example) (as long as they are positive they can be transformed into a density by normalization) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The use of &quot;&lt;strong&gt;mixture&lt;/strong&gt;&quot; in statistic is about convex combination of densities.&lt;/p&gt;&#xA;" OwnerUserId="223" LastEditorUserId="223" LastEditorDisplayName="" LastEditDate="2010-08-03T19:20:24.017" LastActivityDate="2010-08-03T19:20:24.017" />
  <row Id="883" PostTypeId="2" ParentId="881" CreationDate="2010-07-28T09:09:51.180" Score="3" ViewCount="" Body="&lt;p&gt;You can do this with &lt;strong&gt;mixture modeling&lt;/strong&gt;. There are a number of R packages on CRAN for doing this. Search for &quot;mixture&quot; at &lt;a href=&quot;http://cran.r-project.org/web/packages/&quot; rel=&quot;nofollow&quot;&gt;http://cran.r-project.org/web/packages/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-07-28T09:09:51.180" />
  <row Id="884" PostTypeId="1" CreationDate="2010-07-28T09:54:14.730" Score="16" ViewCount="471" Body="&lt;p&gt;I was at a talk a few months back where the speaker used the term 'degrees of freedom'. She briefly said something along the lines of it meaning the number of values used to form a statistic that are free to vary.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What does this mean? I'm specifically looking for an intuitive explanation. &lt;/p&gt;&#xA;" OwnerUserId="541" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-08-07T17:51:33.760" LastActivityDate="2010-08-07T17:51:33.760" Title="What are &quot;degrees of freedom&quot;?" Tags="&lt;degrees-of-freedom&gt;" AnswerCount="4" FavoriteCount="8" />
  <row Id="885" PostTypeId="2" ParentId="884" CreationDate="2010-07-28T10:26:32.273" Score="3" ViewCount="" Body="&lt;p&gt;Check out this &lt;a href=&quot;http://www.statsdirect.com/help/basics/degrees_of_freedom.htm&quot; rel=&quot;nofollow&quot;&gt;explanation&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="339" LastActivityDate="2010-07-28T10:26:32.273" />
  <row Id="886" PostTypeId="1" AcceptedAnswerId="934" CreationDate="2010-07-28T11:31:59.857" Score="5" ViewCount="358" Body="&lt;p&gt;The 'fundamental' idea of statistics for estimating parameters is &lt;a href=&quot;http://en.wikipedia.org/wiki/Maximum_likelihood&quot; rel=&quot;nofollow&quot;&gt;maximum likelihood&lt;/a&gt;. I am wondering what is the corresponding idea in machine learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Qn 1. Would it be fair to say that the 'fundamental' idea in machine learning for estimating parameters is: 'Loss Functions'&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[Note: It is my impression that machine learning algorithms often optimize a loss function and hence the above question.]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Qn 2: Is there any literature that attempts to bridge the gap between statistics and machine learning?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[Note: Perhaps, by way of relating loss functions to maximum likelihood. (e.g., OLS is equivalent to maximum likelihood for normally distributed errors etc)]&lt;/p&gt;&#xA;" LastActivityDate="2010-08-04T09:07:44.193" Title="What is the 'fundamental' idea of machine learning for estimating parameters?" Tags="&lt;machine-learning&gt;&lt;maximum-likelihood&gt;&lt;loss-functions&gt;" AnswerCount="5" CommentCount="4" FavoriteCount="3" />
  <row Id="887" PostTypeId="1" CreationDate="2010-07-28T12:02:21.063" Score="1" ViewCount="320" Body="&lt;p&gt;Suppose there is a very big (infinite?) population of normally distributed values with unknown mean and variance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose also that we have a sample, &lt;em&gt;S&lt;/em&gt;, of &lt;em&gt;n&lt;/em&gt; values from the entire population. We can calculate mean and standard deviation for this sample (we use &lt;em&gt;n-1&lt;/em&gt; for stdev calculation).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first and most important question is how is stdev(S) related to the standard deviation of the entire population?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An illustration for this issue is the second question:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose we have an additional number, &lt;em&gt;x&lt;/em&gt;, and we would like to test whether it is an vis-a-vis the general population. My intuitive approach is to calculate Z as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Z = (x - mean(S))/stdev(S)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and then test it against standard distribution if &lt;em&gt;n&gt;30&lt;/em&gt; or against t-distribution if &lt;em&gt;n&amp;lt;30&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, this approach doesn't account for &lt;em&gt;n&lt;/em&gt;, the size of the sample. What is the right way to solve this question provided there is only single sample &lt;em&gt;S&lt;/em&gt;?&lt;/p&gt;&#xA;" OwnerUserId="213" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-10-19T07:21:44.890" LastActivityDate="2010-10-19T07:21:44.890" Title="Basic question regarding variance and stdev of a sample" Tags="&lt;standard-deviation&gt;&lt;variance&gt;&lt;normality&gt;&lt;sample&gt;&lt;unbiased-estimator&gt;" AnswerCount="6" CommentCount="1" FavoriteCount="1" />
  <row Id="888" PostTypeId="2" ParentId="884" CreationDate="2010-07-28T12:09:00.147" Score="0" ViewCount="" Body="&lt;p&gt;Or see  &lt;a href=&quot;http://en.wikipedia.org/wiki/Degrees_of_freedom_%28statistics%29&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks Srikant. Fixed now.&lt;/p&gt;&#xA;" OwnerUserId="247" LastEditorUserId="247" LastEditorDisplayName="" LastEditDate="2010-07-28T13:11:23.420" LastActivityDate="2010-07-28T13:11:23.420" CommentCount="1" />
  <row Id="889" PostTypeId="2" ParentId="887" CreationDate="2010-07-28T12:12:22.993" Score="1" ViewCount="" Body="&lt;p&gt;My first answer was full of errors. Here is a corrected version:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The correct way to test is as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;z = (mean(S) - mu) / (stdev(S) / sqrt(n) )&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See: &lt;a href=&quot;http://en.wikipedia.org/wiki/Student%27s_t-test#Independent_one-sample_t-test&quot; rel=&quot;nofollow&quot;&gt;Student's t-test&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note the following:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;The sample size is accounted for when you divide the standard deviation by the square root of the sample size.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;You should also note that the z-test is for testing whether the true mean of the population is some particular value. It does not make sense to substitute x instead of mu in the above statistic.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" LastActivityDate="2010-07-28T12:20:09.950" />
  <row Id="890" PostTypeId="1" CreationDate="2010-07-28T12:13:03.647" Score="2" ViewCount="271" Body="&lt;p&gt;Hello, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am struggling a little bit at the moment with a question related to logistic regression. I have a model that predicts the occurrence of animal based on land cover with reference to forest. I am not grasping the concept of a reference class and struggle to extrapolate the model onto a new area. Any explanations or guidance towards papers, lecture notes etc would be highly appreciated. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks in advance &#xA;Mike&lt;/p&gt;&#xA;" LastActivityDate="2010-08-02T08:14:36.877" Title="Logistic Regression: Reference Category and Prediction" Tags="&lt;logistic-regression&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="893" PostTypeId="2" ParentId="884" CreationDate="2010-07-28T12:48:14.233" Score="7" ViewCount="" Body="&lt;p&gt;I really like first sentence from &#xA;&lt;a href=&quot;http://www.jerrydallal.com/LHSP/dof.htm&quot; rel=&quot;nofollow&quot;&gt;The Little Handbook of Statistical Practice. Degrees of Freedom Chapter&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;One of the questions an instrutor&#xA;  dreads most from a mathematically&#xA;  unsophisticated audience is, &quot;What&#xA;  exactly is degrees of freedom?&quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I think you can get really good understanding about degrees of freedom from reading this chapter.&lt;/p&gt;&#xA;" OwnerUserId="236" LastActivityDate="2010-07-28T12:48:14.233" CommentCount="2" />
  <row Id="894" PostTypeId="2" ParentId="884" CreationDate="2010-07-28T12:49:31.780" Score="8" ViewCount="" Body="&lt;p&gt;Or simply: the number of elements in a numerical array that you're allowed to change so that the value of the statistic remains unchanged.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# for instance if:&#xA;x + y + z = 10&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;you can change, for instance, &lt;em&gt;x&lt;/em&gt; and &lt;em&gt;y&lt;/em&gt; at random, but you cannot change &lt;em&gt;z&lt;/em&gt; (you can, but not at random, therefore you're not &lt;em&gt;free&lt;/em&gt; to change it - see Harvey's comment), 'cause you'll change the value of the statistic (&amp;Sigma; = 10). So, in this case df = 2.&lt;/p&gt;&#xA;" OwnerUserId="1356" LastEditorUserId="1356" LastEditorDisplayName="" LastEditDate="2010-07-28T17:34:31.507" LastActivityDate="2010-07-28T17:34:31.507" CommentCount="3" />
  <row Id="895" PostTypeId="2" ParentId="887" CreationDate="2010-07-28T12:50:26.427" Score="2" ViewCount="" Body="&lt;p&gt;I'm finding it rather tricky to see what you are asking:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;If you want to know whether the Var(S) is different from the population variance, then see this previous &lt;a href=&quot;http://stats.stackexchange.com/questions/841/comparing-the-variance-of-paired-observations/845#845&quot;&gt;answer&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;li&gt;If you want to determine whether the mean(S) and the mean(X) are the same, then look at &lt;a href=&quot;http://en.wikipedia.org/wiki/Student%27s_t-test#Independent_two-sample_t-test&quot; rel=&quot;nofollow&quot;&gt;Independent two-sample t-tests&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;li&gt;If you want to test whether mean(S) is equal to the population mean, then see @Srikant answer above, i.e. a &lt;a href=&quot;http://en.wikipedia.org/wiki/Student%27s_t-test#Independent_one-sample_t-test&quot; rel=&quot;nofollow&quot;&gt;one-sample t-test&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="8" LastActivityDate="2010-07-28T12:50:26.427" />
  <row Id="897" PostTypeId="1" AcceptedAnswerId="905" CreationDate="2010-07-28T13:32:32.843" Score="5" ViewCount="311" Body="&lt;p&gt;What is the difference between offline and &lt;a href=&quot;http://en.wikipedia.org/wiki/Online_machine_learning&quot; rel=&quot;nofollow&quot;&gt;online learning&lt;/a&gt;?  Is it just a matter of learning over the entire dataset (offline) vs. learning incrementally (one instance at a time)?  What are examples of algorithms used in both?&lt;/p&gt;&#xA;" OwnerUserId="284" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-08-07T17:50:51.657" LastActivityDate="2010-08-07T17:50:51.657" Title="Online vs offline learning?" Tags="&lt;machine-learning&gt;" AnswerCount="1" />
  <row Id="898" PostTypeId="1" AcceptedAnswerId="918" CreationDate="2010-07-28T13:49:07.733" Score="4" ViewCount="237" Body="&lt;p&gt;I am interested in tools/techniques that can be used for analysis of &lt;a href=&quot;http://en.wikipedia.org/wiki/Real-time_data&quot; rel=&quot;nofollow&quot;&gt;streaming data in &quot;real-time&quot;&lt;/a&gt;*, where latency is an issue.  The most common example of this is probably price data from a financial market, although it also occurs in other fields (e.g. finding trends on Twitter or in Google searches).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my experience, the most common software category for this is &lt;a href=&quot;http://en.wikipedia.org/wiki/Complex_event_processing&quot; rel=&quot;nofollow&quot;&gt;&quot;&lt;strong&gt;complex event processing&lt;/strong&gt;&quot;&lt;/a&gt;.  This includes commercial software such as &lt;a href=&quot;http://www.streambase.com/index.htm&quot; rel=&quot;nofollow&quot;&gt;Streambase&lt;/a&gt; and &lt;a href=&quot;http://www.sybase.com/products/financialservicessolutions/aleristreamingplatform&quot; rel=&quot;nofollow&quot;&gt;Aleri&lt;/a&gt; or open-source ones such as &lt;a href=&quot;http://esper.codehaus.org/&quot; rel=&quot;nofollow&quot;&gt;Esper&lt;/a&gt; or &lt;a href=&quot;http://telegraph.cs.berkeley.edu/&quot; rel=&quot;nofollow&quot;&gt;Telegraph&lt;/a&gt; (which was the basis for &lt;a href=&quot;http://www.truviso.com/&quot; rel=&quot;nofollow&quot;&gt;Truviso&lt;/a&gt;).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many existing models are not suited to this kind of analysis because they're too computationally expensive.  Are any models** specifically designed to deal with real-time data?  What tools can be used for this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;i&gt;&#xA;* By &quot;real-time&quot;, I mean &quot;analysis on data &lt;em&gt;as it is created&lt;/em&gt;&quot;.  So I do not mean &quot;data that has a time-based relevance&quot; (as in &lt;a href=&quot;http://www.hilarymason.com/blog/conference-web2-expo-sf/&quot; rel=&quot;nofollow&quot;&gt;this talk by Hilary Mason&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;** By &quot;model&quot;, I mean a mathematical abstraction that describe the behavior of an object of study (e.g. in terms of random variables and their associated probability distributions), either for description or forecasting.  This could be a machine learning or statistical model.&#xA;&lt;/i&gt;&lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-11-11T05:30:32.227" Title="Modeling of real-time streaming data?" Tags="&lt;modeling&gt;&lt;software&gt;&lt;real-time&gt;" AnswerCount="6" CommentCount="1" FavoriteCount="4" />
  <row Id="899" PostTypeId="1" CreationDate="2010-07-28T13:53:18.503" Score="6" ViewCount="195" Body="&lt;p&gt;I'm trying to separate two groups of values from a single dataset. I can assume that one of the population is normally distributed and is at least half the size of the sample. The values of the second one are both lower or higher than the values from the first one (distribution is unknown). What I'm trying to do is to find the upper and lower limits that would enclose the normally-distributed population from the other.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My assumption provide me with starting point:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;all point within the interquartile range of the sample are from the normally-distributed population.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I'm trying to test for outliers taking them from the rest of the sample until they don't fit into the 3 st.dev of the normally-distributed population. Which is not ideal, but seem to produce reasonable enough result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is my assumption statistically sound? What would be a better way to go about this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;p.s. please fix the tags someone.&lt;/p&gt;&#xA;" OwnerUserId="219" LastEditorUserId="1739" LastEditorDisplayName="" LastEditDate="2010-11-03T08:53:27.370" LastActivityDate="2010-11-03T08:53:27.370" Title="Separating two populations from the sample" Tags="&lt;dataset&gt;&lt;outliers&gt;&lt;em-algorithm&gt;" AnswerCount="3" CommentCount="3" />
  <row Id="900" PostTypeId="2" ParentId="898" CreationDate="2010-07-28T13:57:09.400" Score="1" ViewCount="" Body="&lt;p&gt;I am not sure how far this would be relevant to what you want to do but see the paper on adaptive question design called &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.120.8664&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;FASTPACE&lt;/a&gt;. The goal of the algorithm is to ask the next question from a survey respondent based on his/her previous questions and answers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data does not arrive as fast as stock prices but nevertheless latency is an issue as most survey respondents expect the next question to appear within a few seconds.&lt;/p&gt;&#xA;" LastActivityDate="2010-07-28T13:57:09.400" />
  <row Id="901" PostTypeId="2" ParentId="73" CreationDate="2010-07-28T13:59:49.157" Score="1" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;RODBC&lt;/strong&gt; for accessing data from databases, &lt;strong&gt;sqldf&lt;/strong&gt; for performing simple SQL queries on dataframes (although I am forcing myself to use native R commands), and &lt;strong&gt;ggplot2&lt;/strong&gt; and &lt;strong&gt;plyr&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="11" LastActivityDate="2010-07-28T13:59:49.157" CommunityOwnedDate="2010-07-28T13:59:49.157" />
  <row Id="902" PostTypeId="2" ParentId="898" CreationDate="2010-07-28T14:01:46.063" Score="2" ViewCount="" Body="&lt;p&gt;It is going to depend a lot on what exactly you are looking for, but start at &lt;a href=&quot;http://www.cs.rutgers.edu/~muthu/stream-1-1.ps&quot; rel=&quot;nofollow&quot;&gt;Data Streams: Algorithms and Application by Muthukrishnan &lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are many others that can be found by googling &quot;data stream algorithms&quot;, or following the references in the paper.&lt;/p&gt;&#xA;" OwnerUserId="247" LastEditorUserId="247" LastEditorDisplayName="" LastEditDate="2010-07-28T16:49:31.297" LastActivityDate="2010-07-28T16:49:31.297" CommentCount="1" />
  <row Id="903" PostTypeId="2" ParentId="485" CreationDate="2010-07-28T14:06:35.367" Score="2" ViewCount="" Body="&lt;p&gt;There is a series of Google Tech Talk videos called &lt;a href=&quot;http://video.google.com/videosearch?q=mease+stats+202&amp;amp;sitesearch=#&quot; rel=&quot;nofollow&quot;&gt;Stats 202 - Statistical Aspects of Data Mining&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="11" LastActivityDate="2010-07-28T14:06:35.367" CommunityOwnedDate="2010-07-28T14:06:35.367" />
  <row Id="904" PostTypeId="2" ParentId="812" CreationDate="2010-07-28T14:16:38.977" Score="7" ViewCount="" Body="&lt;p&gt;Functional Data often involves different question.  I've been reading Functional Data Analysis, Ramsey and Silverman, and they spend a lot of times discussing curve registration, warping functions, and estimating derivatives of curves.  These tend to be very different questions than those asked by people interested in studying high-dimensional data.&lt;/p&gt;&#xA;" OwnerUserId="549" LastActivityDate="2010-07-28T14:16:38.977" CommentCount="2" />
  <row Id="905" PostTypeId="2" ParentId="897" CreationDate="2010-07-28T14:37:17.337" Score="6" ViewCount="" Body="&lt;p&gt;Online learning means that you are doing it as the data comes in...offline means that you have a static dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So for online learning, you (typically) have more data, but you have time constraints.  Another wrinkle that can affect online learning is that your concepts might change through time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say you want to build a classifier to recognize spam.  You can acquire a large corpus of e-mail, label it, and train a classifier on it.  This would be offline learning.  Or, you can take all the e-mail coming into your system, and continuously update your classifier (labels may be a bit tricky...).  This would be online learning.&lt;/p&gt;&#xA;" OwnerUserId="549" LastActivityDate="2010-07-28T14:37:17.337" CommentCount="1" />
  <row Id="908" PostTypeId="2" ParentId="880" CreationDate="2010-07-28T14:58:10.173" Score="5" ViewCount="" Body="&lt;p&gt;You miss one important issue -- there is almost never such thing as T[i]. Think of a simple problem in which the sum of two attributes (of a similar amplitude) is important; if you'd remove one of them the importance of the other will suddenly drop. Also, big amount of irrelevant attributes is the accuracy of most classifiers, so along their ability to assess importance. Last but not least, stochastic algorithms will return stochastic results, and so even the T[i] ranking can be unstable. So in principle you should at least recalculate T[i] after each (or at least after each non trivially redundant) attribute is removed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Going back to the topic, the question which CV to choose is mostly problem dependent; with very small number of cases LOO may be the best choice because all other start to reduce to it; still small is rather n=10 not n=100. So I would just recommend random subsampling (which I use most) or K-fold (then with recreating splits on each step). Still, you should also collect not only mean but also the standard deviation of error estimates; this can be used to (approximately) judge which changes of mean are significant ans so help you decide when to cease the process.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-07-28T14:58:10.173" CommentCount="3" />
  <row Id="909" PostTypeId="2" ParentId="899" CreationDate="2010-07-28T15:24:38.010" Score="2" ViewCount="" Body="&lt;p&gt;This assumes that you don't even know if the second distribution is normal or not; I basically handle this uncertainty by focusing only on the normal distribution. This may or may not be the best approach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you can assume that the two populations are completely separated (i.e., all values from distribution A are less than all values from distribution B), then one approach is to use the optimize() function in R to search for the break-point that yields estimates of the mean and sd of the normal distribution that make the data most likely:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#generate completely separated data&#xA;a = rnorm(100)&#xA;b = rnorm(100,10)&#xA;while(!all(a&amp;lt;b)){&#xA;    a = rnorm(100)&#xA;    b = rnorm(100,10)&#xA;}&#xA;&#xA;#create a mix&#xA;mix = c(a,b)&#xA;&#xA;#&quot;forget&quot; the original distributions&#xA;rm(a)&#xA;rm(b)&#xA;&#xA;#try to find the break point between the distributions&#xA;break_point = optimize(&#xA;    f = function(x){&#xA;        data_from_a = mix[mix&amp;lt;x]&#xA;        likelihood = dnorm(data_from_a,mean(data_from_a),sd(data_from_a))&#xA;        SLL = sum(log(likelihood))&#xA;        return(SLL)&#xA;    }&#xA;    , interval = c(sort(mix)[2],max(mix))&#xA;    , maximum = TRUE&#xA;)$maximum&#xA;&#xA;#label the data&#xA;labelled_mix = data.frame(&#xA;    x = mix&#xA;    , source = ifelse(mix&amp;lt;break_point,'A','B')&#xA;)&#xA;print(labelled_mix)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you can't assume complete separation, then I think you'll have to assume some distribution for the second distribution and then use mixture modelling. Note that mixture modelling won't actually label the individual data points, but will give you the mixture proportion and estimates of the parameters of each distribution (eg. mean, sd, etc.).&lt;/p&gt;&#xA;" OwnerUserId="364" LastActivityDate="2010-07-28T15:24:38.010" CommentCount="1" />
  <row Id="910" PostTypeId="2" ParentId="886" CreationDate="2010-07-28T15:29:33.070" Score="2" ViewCount="" Body="&lt;p&gt;There is a trivial answer -- there is no parameter estimation in machine learning! We don't assume that our models are equivalent to some hidden background models; we treat both reality and the model as black boxes and we try to shake the model box (train in official terminology) so that its output will be similar to that of the reality box.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The concept of not only likelihood but the whole model selection based on the training data is replaced by optimizing the accuracy (whatever defined; in principle the goodness in desired use) on the unseen data; this allows to optimize both precision and recall in a coupled manner. This leads to the concept of an ability to generalize, which is achieved in different ways depending on the learner type.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The answer to the question two depends highly on definitions; still I think that the nonparametric statistics is something that connects the two. &lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-07-28T15:29:33.070" CommentCount="6" />
  <row Id="911" PostTypeId="2" ParentId="856" CreationDate="2010-07-28T15:32:38.310" Score="3" ViewCount="" Body="&lt;p&gt;Interesting question.  What do you mean by weight?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would be inclined to do a bootstrap...pick your favorite statistic (i.e. Fisher's Exact), and compute it on your data.  Then assign new cells to each instance according to your null hypothesis, and repeat the process 999 times.  This should give a pretty good empirical distribution for your test statistic under the null hypothesis, and allow easy computation of your p-value!&lt;/p&gt;&#xA;" OwnerUserId="549" LastActivityDate="2010-07-28T15:32:38.310" CommentCount="1" />
  <row Id="912" PostTypeId="2" ParentId="868" CreationDate="2010-07-28T15:37:01.803" Score="1" ViewCount="" Body="&lt;p&gt;The other issue is that you put in your data, and the algorithm learns the weights and the Beta_0 for you...I don't know if Excel can do logistic regression...if it doesn't, I'd be inclined to use R to learn your model (and predict future cases for you!).&lt;/p&gt;&#xA;" OwnerUserId="549" LastActivityDate="2010-07-28T15:37:01.803" />
  <row Id="913" PostTypeId="1" AcceptedAnswerId="916" CreationDate="2010-07-28T16:15:07.883" Score="3" ViewCount="328" Body="&lt;p&gt;Comparing two variables, I came up with the following chart. the x, y pairs represent independent observations of data on the field.  I've doen &lt;a href=&quot;http://en.wikipedia.org/wiki/Correlation_and_dependence&quot; rel=&quot;nofollow&quot;&gt;Pearson correlation&lt;/a&gt; on it and have found one of 0.6. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My end goal is to establish a relationship between y and x such that y = f(x).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What analaysis would you recommend to obtain some form ofa relationship between the two variables?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://koopics.com/ask_math_chart.jpg&quot; alt=&quot;Graph&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="59" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-16T07:04:38.580" LastActivityDate="2010-09-19T01:47:02.270" Title="Relationships between two variables" Tags="&lt;regression&gt;" AnswerCount="8" />
  <row Id="914" PostTypeId="2" ParentId="913" CreationDate="2010-07-28T16:21:41.483" Score="3" ViewCount="" Body="&lt;p&gt;What you are looking for is called regression; there are a lot of methods you can do it, both statistical and machine learning ones. If you want to find f, you must use statistics; in that case you must first assume that f is of some form, like f:y=a*x+b and then use some regression method to fit the parameters.&lt;br&gt;&#xA;The plot suggests there are a lot of outliers (elements that does not follow f(x)); you may need robust regression to get rid of them.&lt;/p&gt;&#xA;" OwnerUserId="88" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-07-28T16:31:04.797" LastActivityDate="2010-07-28T16:31:04.797" />
  <row Id="915" PostTypeId="2" ParentId="913" CreationDate="2010-07-28T16:24:38.283" Score="3" ViewCount="" Body="&lt;p&gt;And just eyeballing the data, you are probably going to want to transform the data, as (at least to me) it looks skewed. Looking at the histograms of the two variables should suggest which transforms may be beneficial.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As suggested by mbq, more text &lt;a href=&quot;http://en.wikipedia.org/wiki/Data_transformation_%28statistics%29&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="247" LastEditorUserId="247" LastEditorDisplayName="" LastEditDate="2010-07-28T16:48:05.950" LastActivityDate="2010-07-28T16:48:05.950" CommentCount="0" />
  <row Id="916" PostTypeId="2" ParentId="913" CreationDate="2010-07-28T16:31:53.143" Score="4" ViewCount="" Body="&lt;p&gt;Normality seems to be strongly violated at least by your y variable. I would log transform y to see if that cleans things up a bit. Then, fit a regression to log(y) ~ x.  The formula the regression will return will be of the form log(y) = \alpha +  \beta*x which you can transform back to the original scale by y = exp(\alpha +  \beta*x)&lt;/p&gt;&#xA;" OwnerUserId="287" LastActivityDate="2010-07-28T16:31:53.143" CommentCount="2" />
  <row Id="917" PostTypeId="2" ParentId="868" CreationDate="2010-07-28T17:01:12.903" Score="4" ViewCount="" Body="&lt;p&gt;Like drknexus said, for a logistic regression, your outcome measure needs to be 0 and 1. I'd go back and recode your outcome as 0 (didn't like it), or 1 (did like it). Then, abandon excel and load the data into R (it's really not as intimidating as it looks).  Your regression will look something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;glm(Liked ~ Visually.Stunning + Exhilarating + Artistic + Sporty, family = binomial, data = data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The regression will return betas for each feature in terms of log-odds. So, for every 1 point increase in &lt;code&gt;Artistic&lt;/code&gt;, for instance, you'll have a value for how much that increases or decreases the log-odds of your enjoyment. Most of the betas will be positive, unless you &lt;em&gt;dislike&lt;/em&gt; sporty games or something.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, you'll have to ask yourself some interesting questions. The assumption of the model is that the values on each of these scores affect your enjoyment &lt;em&gt;independently&lt;/em&gt;, which probably isn't true! A game that is very Visually.Stunning and Exhilarating is probably way better than you would expect given those component parts. And it's probably the case that if a game gets scores of 1 on all features except Sporty, which gets a 4, that high Sporty score is worth less than if the other scores were higher.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That is, many or all of your features probably &lt;em&gt;interact&lt;/em&gt;. To fit an accurate model, then, you'll want to add in these interactions. That formula would look like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;glm(Liked ~ Visually.Stunning * Exhilarating * Artistic * Sporty, family = binomial, data = data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, there are two points of difficulty here. First, you need to have more data to fit a good model with this many interactions than the pure independence model. Second, you risk overfitting, which means that the model will very accurately describe the original data, but will be less good at making accurate predictions for future data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Needless to say, some people spend all day fitting and refitting models like this one. &lt;/p&gt;&#xA;" OwnerUserId="287" LastActivityDate="2010-07-28T17:01:12.903" CommentCount="5" />
  <row Id="918" PostTypeId="2" ParentId="898" CreationDate="2010-07-28T17:04:38.547" Score="3" ViewCount="" Body="&lt;p&gt;This area roughly falls into two categories.  The first concerns stream processing and querying issues and associated models and algorithms.  The second is efficient algorithms and models for learning from data streams (or data stream mining).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's my impression that the CEP industry is connected to the first area.  For example, StreamBase originated from the &lt;a href=&quot;http://www.cs.brown.edu/research/aurora/&quot; rel=&quot;nofollow&quot;&gt;Aurora&lt;/a&gt; project at Brown/Brandeis/MIT.  A similar project was Widom's &lt;a href=&quot;http://infolab.stanford.edu/stream/&quot; rel=&quot;nofollow&quot;&gt;STREAM&lt;/a&gt; at Stanford.  Reviewing the publications at either of those projects' sites should help exploring the area.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A nice paper summarizing the research issues (in 2002) from the first area is &lt;em&gt;&lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.106.9846&quot; rel=&quot;nofollow&quot;&gt;Models and issues in data stream systems&lt;/a&gt;&lt;/em&gt; by Babcock et al.  In stream mining, I'd recommend starting with &lt;em&gt;&lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.80.798&quot; rel=&quot;nofollow&quot;&gt;Mining Data Streams: A Review&lt;/a&gt;&lt;/em&gt; by Gaber et al.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;BTW, I'm not sure exactly what you're interested in as far as specific models.  If it's stream mining and classification in particular, the &lt;a href=&quot;http://en.wikipedia.org/wiki/Incremental_decision_tree#VFDT&quot; rel=&quot;nofollow&quot;&gt;VFDT&lt;/a&gt; is a popular choice.  The two review papers (linked above) point to many other models and it's very contextual.&lt;/p&gt;&#xA;" OwnerUserId="251" LastEditorUserId="251" LastEditorDisplayName="" LastEditDate="2010-07-28T20:07:08.813" LastActivityDate="2010-07-28T20:07:08.813" CommentCount="2" />
  <row Id="919" PostTypeId="2" ParentId="890" CreationDate="2010-07-28T17:14:08.763" Score="1" ViewCount="" Body="&lt;p&gt;I'm not sure I exactly understand your question, but I'm assuming your confusion involves a categorical predictor in your model. When it comes to continuous variables in a regression, the coefficients for each predictor are weights for the value of the predictor to produce the predicted y value:e.g.  y = 2*x   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, with a categorical variable, weights are meaningless. What does 2*Male mean, or in your case, 2*forest.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, the coefficients returned for levels of categorical variables represent how different they are from some reference level. In experimental settings, your reference level would be the control group, and then you would get a coefficient for every treatment group, indicating what the size of the effect of each treatment was.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my own research, and I'm guessing in yours too, there isn't always a meaningful control category for the reference level. So, what I'd do is set the reference level to whatever category would make exposition of the comparisons easiest. Or, you could use different contrasts, like sum contrasts, but those have their own difficulties especially if you have sparse data for one or more categories.&lt;/p&gt;&#xA;" OwnerUserId="287" LastActivityDate="2010-07-28T17:14:08.763" />
  <row Id="920" PostTypeId="1" AcceptedAnswerId="940" CreationDate="2010-07-28T17:15:09.090" Score="3" ViewCount="233" Body="&lt;p&gt;I completed a Monte Carlo simulation that consisted of 1,000,000 individual simulations. The simulation returns a variable, p, that can be either 1 or 0. I then weight the simulations based on predefined criteria and calculate the probability of p. I also calculate a risk ratio using p:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Risk ratio = P(p|test case) /&#xA;  P(p|control case)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I had eight Monte Carlo runs, which consist of one control case and seven test cases. I need to know if the probabilities of p are statistical different compared to the other cases. I know I can use a multiple comparison test or nonparametric ANOVA to test individual variables, but how do I do this for probabilities? &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;For example are these two probabilities statistically different?:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Probabilities&lt;/em&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P(p|test #3) = 4.08 X 10^-5&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P(p|test #4) = 6.10 X 10^-5&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Risk Ratios&lt;/em&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Risk Ratio(test #3) = 0.089&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Risk Ratio (test #4) = 0.119 &lt;/p&gt;&#xA;" OwnerUserId="559" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-16T07:07:54.150" LastActivityDate="2010-09-16T07:07:54.150" Title="Test if probabilities are statistically different?" Tags="&lt;hypothesis-testing&gt;" AnswerCount="1" CommentCount="0" />
  <row Id="922" PostTypeId="2" ParentId="886" CreationDate="2010-07-28T17:28:47.880" Score="0" ViewCount="" Body="&lt;p&gt;I don't think there is a fundamental idea around parameter estimation in Machine Learning.  The ML crowd will happily maximize the likelihood or the posterior, as long as the algorithms are efficient and predict &quot;accurately&quot;.  The focus is on computation, and results from statistics are widely used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're looking for fundamental ideas in general, then in computational learning theory, &lt;a href=&quot;http://en.wikipedia.org/wiki/Probably_approximately_correct_learning&quot; rel=&quot;nofollow&quot;&gt;PAC&lt;/a&gt; is central; in statistical learning theory, &lt;a href=&quot;http://en.wikipedia.org/wiki/Structural_risk_minimization&quot; rel=&quot;nofollow&quot;&gt;structural risk miniminization&lt;/a&gt;; and there are other areas (for example, see the &lt;em&gt;&lt;a href=&quot;http://hunch.net/?p=612&quot; rel=&quot;nofollow&quot;&gt;Prediction Science&lt;/a&gt;&lt;/em&gt; post by John Langford).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On bridging statistics/ML, the divide seems exagerrated.  I liked gappy's &lt;a href=&quot;http://stats.stackexchange.com/questions/6/the-two-cultures-statistics-vs-machine-learning/607#607&quot;&gt;answer&lt;/a&gt; to the &quot;Two Cultures&quot; question.&lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-07-28T17:28:47.880" CommentCount="1" />
  <row Id="924" PostTypeId="1" AcceptedAnswerId="926" CreationDate="2010-07-28T17:36:24.920" Score="4" ViewCount="404" Body="&lt;p&gt;For 1,000,000 observations, I observed a discrete event, X, 3 times for the control group and 10 times for the test group. How do I determine for a large number of observations (1,000,000), if three is statistically different than ten?&lt;/p&gt;&#xA;" OwnerUserId="559" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-10-08T16:05:48.613" LastActivityDate="2010-10-08T16:05:48.613" Title="Determine if three is statistically different than ten for a very large number of observations (1,000,000)" Tags="&lt;hypothesis-testing&gt;&lt;large-data&gt;" AnswerCount="7" CommentCount="2" FavoriteCount="2" />
  <row Id="925" PostTypeId="2" ParentId="924" CreationDate="2010-07-28T17:40:56.223" Score="0" ViewCount="" Body="&lt;p&gt;I would be really surprised if you find the difference statistically significant. Having said that you may want to use a test for a difference of proportions (3 out of 1M vs 10 out of 1M).&lt;/p&gt;&#xA;" LastActivityDate="2010-07-28T17:40:56.223" />
  <row Id="926" PostTypeId="2" ParentId="924" CreationDate="2010-07-28T17:43:48.153" Score="5" ViewCount="" Body="&lt;p&gt;I think a simple chi-squared test will do the trick. Do you have 1,000,000 observations for both control and test? If so, your table of observations will be (in R code)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: Woops! Left off a zero!&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;m &amp;lt;- rbind(c(3, 1000000-3), c(10, 1000000-10))&#xA;#      [,1]   [,2] &#xA;# [1,]    3 999997&#xA;# [2,]   10 999990&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And chi-squared test will be&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;chisq.test(m)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Which returns chi-squared = 2.7692, df = 1, p-value = 0.0961, which is not statistically significant at the p &amp;lt; 0.05 level. I'd be surprised if these could be clinically significant anyway.&lt;/p&gt;&#xA;" OwnerUserId="287" LastEditorUserId="287" LastEditorDisplayName="" LastEditDate="2010-07-28T17:51:05.423" LastActivityDate="2010-07-28T17:51:05.423" CommentCount="6" />
  <row Id="927" PostTypeId="1" AcceptedAnswerId="937" CreationDate="2010-07-28T17:43:49.697" Score="12" ViewCount="243" Body="&lt;p&gt;What are some podcasts related to statistical analysis?  I've found some audio recordings of college lectures on ITunes U, but I'm not aware of any statistical podcasts.  The closest thing I'm aware of is an operations research podcast &lt;a href=&quot;http://www.scienceofbetter.org/podcast/&quot; rel=&quot;nofollow&quot;&gt;The Science of Better&lt;/a&gt;.  It touches on statistical issues, but it's not specifically a statistical show.&lt;/p&gt;&#xA;" OwnerUserId="319" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-13T08:20:38.937" LastActivityDate="2011-04-13T08:20:38.937" CommunityOwnedDate="2010-07-28T17:43:49.697" Title="Statistical podcasts" Tags="&lt;books&gt;" AnswerCount="5" FavoriteCount="9" />
  <row Id="928" PostTypeId="1" AcceptedAnswerId="932" CreationDate="2010-07-28T17:57:51.697" Score="2" ViewCount="270" Body="&lt;p&gt;This one is bothering me for a while, and a great dispute was held around it. In psychology (as well as in other social sciences), we deal with different ways of dealing with numbers :-) i.e. &lt;strong&gt;the levels of measurement&lt;/strong&gt;. It's also common practice in psychology to standardize some questionnaire, hence transform the data into percentile scores (in order to assess a respondent's position within the representative sample).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Long story short, if you have a variable that holds the data expressed in percentile scores, how should you treat it? As an ordinal, interval, or even ratio variable?!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's not ratio, cause there no real 0 (0&lt;sup&gt;th&lt;/sup&gt; percentile doesn't imply absence of measured property, but the variable's smallest value). I advocate the view that percentile scores are ordinal, since P&lt;sub&gt;70&lt;/sub&gt; - P&lt;sub&gt;50&lt;/sub&gt; is not equal to P&lt;sub&gt;50&lt;/sub&gt; - P&lt;sub&gt;30&lt;/sub&gt;, while the other side says it's interval. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please gentlemen, cut the cord. Ordinal or interval?&lt;/p&gt;&#xA;" OwnerUserId="1356" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-08-07T17:48:50.100" LastActivityDate="2010-08-17T20:28:20.243" Title="Measurement level of percentile scores" Tags="&lt;measurement&gt;" AnswerCount="3" CommentCount="3" />
  <row Id="929" PostTypeId="1" CreationDate="2010-07-28T17:59:42.600" Score="0" ViewCount="116" Body="&lt;p&gt;How comprehensive is the following book - What interpretations are missing?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Interpretations of Probability, Andrei Khrennikov, 2009, de Gruyter, ISBN 978-3-11-020748-4 &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.degruyter.com/cont/fb/ma/detailEn.cfm?isbn=9783110207484&amp;amp;sel=pi&quot; rel=&quot;nofollow&quot;&gt;http://www.degruyter.com/cont/fb/ma/detailEn.cfm?isbn=9783110207484&amp;amp;sel=pi&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Contents:http://www.degruyter.com/files/pdf/9783110207484Contents.pdf&lt;/p&gt;&#xA;" OwnerUserId="560" LastActivityDate="2010-07-28T19:14:49.470" ClosedDate="2010-07-29T23:43:25.867" Title="Probability Interpretations" Tags="&lt;probability&gt;" AnswerCount="1" CommentCount="4" FavoriteCount="1" />
  <row Id="930" PostTypeId="2" ParentId="928" CreationDate="2010-07-28T18:05:11.857" Score="0" ViewCount="" Body="&lt;p&gt;Continuous (interval); this is a method how to convert ordinal data to something that may have some distribution that makes sense.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-07-28T18:05:11.857" CommentCount="2" />
  <row Id="931" PostTypeId="2" ParentId="927" CreationDate="2010-07-28T18:17:46.900" Score="3" ViewCount="" Body="&lt;p&gt;You may be interested in the following link: &lt;a href=&quot;http://www.ats.ucla.edu/stat/seminars/&quot; rel=&quot;nofollow&quot;&gt;http://www.ats.ucla.edu/stat/seminars/&lt;/a&gt; where the UCLA Statistical Computing unit of the UCLA has &lt;em&gt;very nice&lt;/em&gt; screen-casts available. I have found them very useful in the past. They function essentially as lectures. Top-quality teaching.&lt;/p&gt;&#xA;" OwnerUserId="561" LastActivityDate="2010-07-28T18:17:46.900" CommunityOwnedDate="2010-07-28T18:17:46.900" CommentCount="2" />
  <row Id="932" PostTypeId="2" ParentId="928" CreationDate="2010-07-28T18:18:26.243" Score="2" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Background to understand my answer&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The critical property that distinguishes between ordinal and interval scale is whether we can take &lt;em&gt;ratio of differences&lt;/em&gt;. While you cannot take ratio of direct measures for either scale the ratio of differences is meaningful for interval but not ordinal (See: &lt;a href=&quot;http://en.wikipedia.org/wiki/Level_of_measurement#Interval_scale&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Level_of_measurement#Interval_scale&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Temperature is the classic example for an interval scale. Consider the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;80 f = 26.67 c&lt;/p&gt;&#xA;&#xA;&lt;p&gt;40 f = 4.44 c and&lt;/p&gt;&#xA;&#xA;&lt;p&gt;20 f = -6.67 c&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Differences between the first and the second is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;40 f and 22.23 c&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Difference between the second and the third is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;20 f and 11.11 c&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Notice that the ratio is the same irrespective of the scale on which we measure temperature. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A classic example of ordinal data is ranks. If three teams are ranked 1, 2 and 4 then a statement like so does not make sense: &quot;Team 1's difference in strength vis-a-vis team 2 is half of team 2's difference in strength relative to team 4.&quot; (Abuse of notation: I am using the same number as rank and team index).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Answer to your question&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is ratio of differences in percentiles meaningful? In other words, is the ratio of difference in percentiles invariant to the underlying scale? Consider, for example: (P&lt;sub&gt;70&lt;/sub&gt;-P&lt;sub&gt;50&lt;/sub&gt;) / (P&lt;sub&gt;50&lt;/sub&gt;-P&lt;sub&gt;30&lt;/sub&gt;)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose that these percentiles are based on an underlying score between 0-100 and we compute the above ratio. Clearly, we would obtain the same ratio of percentile differences under arbitrary linear transformation of the score (e.g., multiply all scores by 10 so that the range is between 0-1000 and compute the percentiles).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, my answer: &lt;strong&gt;Interval&lt;/strong&gt;&lt;/p&gt;&#xA;" LastActivityDate="2010-07-28T18:18:26.243" />
  <row Id="933" PostTypeId="2" ParentId="290" CreationDate="2010-07-28T18:21:10.107" Score="1" ViewCount="" Body="&lt;p&gt;I have done very well with reading the official documentation. It is well-written, sometimes injected with humour (!) and, if you're willing to spend the time to learn Stata properly, is an absolute goldmine.&lt;/p&gt;&#xA;" OwnerUserId="561" LastActivityDate="2010-07-28T18:21:10.107" />
  <row Id="934" PostTypeId="2" ParentId="886" CreationDate="2010-07-28T18:25:00.973" Score="4" ViewCount="" Body="&lt;p&gt;If statistics is all about maximizing likelihood, then machine learning is all about minimizing loss. Since you don't know the loss you will incur on future data, you minimize an approximation, ie empirical loss.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, if you have a prediction task and are evaluated by the number of misclassifications, you could train parameters so that resulting model produces the smallest number of misclassifications on the training data. &quot;Number of misclassifications&quot; (ie, 0-1 loss) is a hard loss function to work with because it's not differentiable, so you approximate it with a smooth &quot;surrogate&quot;. For instance, log loss is an upper bound on 0-1 loss, so you could minimize that instead, and this will turn out to be the same as maximizing conditional likelihood of the data. With parametric model this approach becomes equivalent to logistic regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a structured modeling task, and log-loss approximation of 0-1 loss, you get something different from maximum conditional likelihood, you will instead maximize &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.16.8122&quot; rel=&quot;nofollow&quot;&gt;product&lt;/a&gt; of (conditional) marginal likelihoods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To get better approximation of loss, people noticed that training model to minimize loss and using that loss as an estimate of future loss is an overly optimistic estimate. So for more accurate (true future loss) minimization they add a bias correction term to empirical loss and minimize that, this is known as structured risk minimization.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In practice, figuring out the right bias correction term may be too hard, so you add an expression &quot;in the spirit&quot; of the bias correction term, for instance, sum of squares of parameters. In the end, almost all parametric machine learning supervised classification approaches end up training the model to minimize the following&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sum_i L(m(x_i,w),y_i) + P(w)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where m is your model parametrized by vector w, i is taken over all datapoints {x_i,y_i}, L is some computationally nice approximation of your true loss and P(w) is some bias-correction/regularization term&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance if you x \in {-1,1}^d, y \in {-1,1}, a typical approach would be to let m(x)=sign(w.x), L(m(x),y)=-log(y x.w), P(w)=q w.w, and choose q by cross validation&lt;/p&gt;&#xA;" OwnerUserId="511" LastEditorUserId="511" LastEditorDisplayName="" LastEditDate="2010-07-28T18:35:23.073" LastActivityDate="2010-07-28T18:35:23.073" CommentCount="4" />
  <row Id="935" PostTypeId="2" ParentId="929" CreationDate="2010-07-28T18:36:09.920" Score="1" ViewCount="" Body="&lt;p&gt;Though quantum probability and negative probability models are quite interesting, this is hardly exhaustive of nonstandard models of probability.  There are for instance, &lt;a href=&quot;http://en.wikipedia.org/wiki/Imprecise_probability&quot; rel=&quot;nofollow&quot;&gt;imprecise probability models&lt;/a&gt;, and models that violate Kolmogorov's countable additivity axiom, and more.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an aside, the book may be more properly called 'Models of Probability'.  &lt;a href=&quot;http://plato.stanford.edu/entries/probability-interpret/&quot; rel=&quot;nofollow&quot;&gt;Interpretations of probability&lt;/a&gt;, generally involve characterizing the competing understandings of probability as logically prescribed values, limiting frequencies, propensities, subjective beliefs, etc.   Models, or axiomatizations can certainly be motivated by these understandings, but the problem of creating a variant system is different than arguing for a particular interpretation.&lt;/p&gt;&#xA;" OwnerUserId="39" LastEditorUserId="39" LastEditorDisplayName="" LastEditDate="2010-07-28T19:14:49.470" LastActivityDate="2010-07-28T19:14:49.470" CommentCount="1" />
  <row Id="936" PostTypeId="2" ParentId="927" CreationDate="2010-07-28T19:22:44.490" Score="2" ViewCount="" Body="&lt;p&gt;There is &lt;a href=&quot;http://www.econtalk.org/&quot; rel=&quot;nofollow&quot;&gt;econtalk&lt;/a&gt;, it is mostly about economics, but delves very often to issues of research, science, and statistics.&lt;/p&gt;&#xA;" OwnerUserId="253" LastActivityDate="2010-07-28T19:22:44.490" CommunityOwnedDate="2010-07-28T19:22:44.490" CommentCount="1" />
  <row Id="937" PostTypeId="2" ParentId="927" CreationDate="2010-07-28T19:34:00.560" Score="5" ViewCount="" Body="&lt;p&gt;BBC's &lt;a href=&quot;http://news.bbc.co.uk/2/hi/programmes/more_or_less/default.stm&quot; rel=&quot;nofollow&quot;&gt;More or Less&lt;/a&gt; is often concerned with numeracy and statistical literacy issues.  But it's not specifically about statistics.  Their &lt;a href=&quot;http://news.bbc.co.uk/2/hi/programmes/more_or_less/1628489.stm&quot; rel=&quot;nofollow&quot;&gt;About&lt;/a&gt; page has some background.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;More or Less is devoted to the powerful, sometimes beautiful, often abused but ever ubiquitous world of numbers.&#xA;    The programme was an idea born of the sense that numbers were the principal language of public argument.&#xA;    [...]&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-07-28T19:34:00.560" CommunityOwnedDate="2010-07-28T19:34:00.560" />
  <row Id="938" PostTypeId="2" ParentId="125" CreationDate="2010-07-28T20:43:32.873" Score="2" ViewCount="" Body="&lt;p&gt;Gelman et al. is well-regarded but explictly intended for a graduate course. If you don't have substantial prior coursework in statistics, it is largely a waste.&lt;/p&gt;&#xA;" LastActivityDate="2010-07-28T20:43:32.873" CommunityOwnedDate="2010-07-28T20:43:32.873" />
  <row Id="939" PostTypeId="1" AcceptedAnswerId="941" CreationDate="2010-07-28T20:43:33.327" Score="2" ViewCount="239" Body="&lt;p&gt;Is the Yates' correction for continuity used only for 2X2 matrices? &lt;/p&gt;&#xA;" OwnerUserId="559" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-10-20T21:10:16.807" LastActivityDate="2010-10-20T21:10:16.807" Title="Yates' correction for continuity only for 2X2?" Tags="&lt;contingency-tables&gt;&lt;yates-correction&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="1" />
  <row Id="940" PostTypeId="2" ParentId="920" CreationDate="2010-07-28T20:50:25.120" Score="5" ViewCount="" Body="&lt;p&gt;If you have 1,000,000 independent &quot;coin flips&quot; that can produce 1 with probabilty (prob) and 0 with probability (1-prob), then the number of 1's observed will follow a &lt;a href=&quot;http://en.wikipedia.org/wiki/Binomial_distribution&quot; rel=&quot;nofollow&quot;&gt;Binomial distribution&lt;/a&gt;.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tests of statistical significance are rejection tests, i.e. reject the hypothesis that the two parameters are equal if the probability that param2 is observed in test2 when the true value is param1 is less than a certain number, like 5%, 1%, or 0.1%.  These tests are typically constructed from the cumulative distribution function.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The cumulative distribution function for a binomial is ugly, but can be found in R and probably some other statistics packages as well.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But the good news is that with 1,000,000 cases you don't need to do that.... you would if you had a relatively small number of cases.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because you have 1,000,000 independent flips, the CDF of a normal distribution is a good approximation (the Law of Large Numbers plays a role here).  The mean and variance you need to use are the obvious ones, and are in the &lt;a href=&quot;http://en.wikipedia.org/wiki/Binomial_distribution#Normal_approximation&quot; rel=&quot;nofollow&quot;&gt;Binomial Wikipedia&lt;/a&gt; article... You are then comparing two normally distributed variables and can use all the standard tests you would use with normally distributed variables. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, if the true probability were 40*10^-6 then in 1,000,000 tests you would expect to see 40 +/- 6  positive cases. If the acceptance interval for a test is, for instance, 5 standard deviations wide on each side, then this would be compatible with both observations.  If it were just 3 std dev wide on each side, one case would fit and the other would be statistically different.  &lt;/p&gt;&#xA;" OwnerUserId="87" LastEditorUserId="87" LastEditorDisplayName="" LastEditDate="2010-07-28T21:19:40.940" LastActivityDate="2010-07-28T21:19:40.940" CommentCount="0" />
  <row Id="941" PostTypeId="2" ParentId="939" CreationDate="2010-07-28T21:12:03.197" Score="3" ViewCount="" Body="&lt;p&gt;It's derived for binomial/hypergeometric distributions, so it's applicable to 2x2 or 2x1 cases.&lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-07-28T21:12:03.197" />
  <row Id="942" PostTypeId="1" CreationDate="2010-07-28T21:13:23.387" Score="7" ViewCount="457" Body="&lt;p&gt;I've been beginning to work my way through &lt;a href=&quot;http://www.autonlab.org/tutorials/&quot; rel=&quot;nofollow&quot;&gt;Statistical Data Mining Tutorials by Andrew Moore&lt;/a&gt; (highly recommended for anyone else first venturing into this field).  I started by reading this &lt;a href=&quot;http://www.autonlab.org/tutorials/biosurv01.pdf&quot; rel=&quot;nofollow&quot;&gt;extremely interesting PDF entitled &quot;Introductory overview of time-series-based anomaly detection algorithms&quot;&lt;/a&gt; in which Moore traces through many of the techniques used in the creation of an algorithm to detect disease outbreaks.  Halfway through the slides, on page 27, he lists a number of other &quot;state of the art methods&quot; used to detect outbreaks.  The first one listed is &lt;strong&gt;wavelets&lt;/strong&gt;.  Wikipeida describes a wavelet as&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;a wave-like oscillation with an&#xA;  amplitude that starts out at zero,&#xA;  increases, and then decreases back to&#xA;  zero. It can typically be visualized&#xA;  as a &quot;brief oscillation&quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;but does not describe their application to statistics and my Google searches yield highly academic papers that assume a knowledge of how wavelets relate to statistics or full books on the subject.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like a basic understanding of how wavelets are applied to time-series anomaly detection, much in the way Moore illustrates the other techniques in his tutorial.  Can someone provide an explanation of how detection methods using wavelets work or a link to an understandable article on the matter?&lt;/p&gt;&#xA;" OwnerUserId="75" LastEditorUserId="223" LastEditorDisplayName="" LastEditDate="2011-02-08T16:48:14.277" LastActivityDate="2011-02-08T16:48:14.277" Title="Application of wavelets to time-series-based anomaly detection algorithms" Tags="&lt;time-series&gt;&lt;outliers&gt;&lt;signal-processing&gt;&lt;wavelet&gt;" AnswerCount="3" FavoriteCount="3" />
  <row Id="943" PostTypeId="2" ParentId="841" CreationDate="2010-07-28T21:31:14.630" Score="2" ViewCount="" Body="&lt;p&gt;If you can assume bivariate normality, then you can develop a likelihood-ratio test comparing the two possible covariance matrix structures. The unconstrained (H_a) maximum likelihood estimates are well known - just the sample covariance matrix, the constrained ones (H_0) can be derived by writing out the likelihood (and will probably be some sort of &quot;pooled&quot; estimate). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you don't want to derive the formulas, you can use SAS or R to fit a repeated measures model with unstructured and compound symmetry covariance structures and compare the likelihoods.&lt;/p&gt;&#xA;" OwnerUserId="279" LastActivityDate="2010-07-28T21:31:14.630" />
  <row Id="944" PostTypeId="1" AcceptedAnswerId="947" CreationDate="2010-07-28T21:45:46.273" Score="0" ViewCount="120" Body="&lt;p&gt;When I type a left paren or any quote in the R console, it automatically creates a matching one to the right of my cursor. I guess the idea is that I can just type the expression I want inside without having to worry about matching, but I find it annoying, and would rather just type it myself. How can I disable this feature?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using R 2.8.0 on OSX 10.5.8.&lt;/p&gt;&#xA;" LastActivityDate="2010-07-29T02:11:34.563" Title="How can I get R to stop autocompleting my quotes/parens?" Tags="&lt;r&gt;" AnswerCount="3" CommentCount="5" />
  <row Id="945" PostTypeId="2" ParentId="944" CreationDate="2010-07-28T21:49:13.417" Score="2" ViewCount="" Body="&lt;p&gt;Well either use a different IDE -- this is entirely a feature of the OS X app -- or try to configure the feature in question. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for IDEs / R environments, I'm rather happy with &lt;a href=&quot;http://ess.r-project.org&quot; rel=&quot;nofollow&quot;&gt;ESS&lt;/a&gt;  which works on every platform R works on. &lt;/p&gt;&#xA;" OwnerUserId="334" LastActivityDate="2010-07-28T21:49:13.417" />
  <row Id="946" PostTypeId="1" AcceptedAnswerId="963" CreationDate="2010-07-28T21:50:29.093" Score="2" ViewCount="164" Body="&lt;p&gt;Hi all,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;New to the site.  I am just getting started with R, and want to replicate a feature that is available in SPSS.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Simply, I build a &quot;Custom Table&quot; in SPSS with a single categorical variable in the column and many continous/scale variables in the rows (no interactions, just stacked on top of each other).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The table reports the means and valid N's for each column (summary statistics are in the rows), and select the option to generate significance tests for column means (each column against the others) using alpha .05 and adjust for unequal variances.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is my question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I replicate this in R?  What is my best option to build this table  and what tests are available that will get me to the same spot?  Since I am getting used to R, I am still trying to navigate around what is available.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks in advance!&lt;/p&gt;&#xA;" OwnerUserId="569" LastActivityDate="2010-07-29T08:22:43.180" Title="Column Means Significance Tests in R" Tags="&lt;r&gt;&lt;mean&gt;&lt;statistical-significance&gt;&lt;spss&gt;" AnswerCount="2" />
  <row Id="947" PostTypeId="2" ParentId="944" CreationDate="2010-07-28T21:51:15.083" Score="4" ViewCount="" Body="&lt;p&gt;On OSX, go to &lt;code&gt;R &amp;gt; Preferences &amp;gt; Editor&lt;/code&gt; and deselect &lt;code&gt;Match braces/quotes&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="287" LastActivityDate="2010-07-28T21:51:15.083" CommentCount="1" />
  <row Id="948" PostTypeId="2" ParentId="946" CreationDate="2010-07-28T21:54:16.503" Score="1" ViewCount="" Body="&lt;pre&gt;&lt;code&gt;summary(df)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Will give you 5 number summaries and counts of &lt;code&gt;NA&lt;/code&gt; for continuous variables, and counts for categorical variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for the significance tests, you'll have to do that by hand with &lt;code&gt;t.test()&lt;/code&gt; or &lt;code&gt;wilcox.test()&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="287" LastActivityDate="2010-07-28T21:54:16.503" CommentCount="2" />
  <row Id="949" PostTypeId="1" CreationDate="2010-07-28T21:59:02.693" Score="7" ViewCount="403" Body="&lt;p&gt;Take $x \in {0,1}^d$ and $y \in {0,1}$ and suppose we model the task of predicting y given x using logistic regression. When can logistic regression coefficients be written in closed form?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One example is when we use a saturated model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;IE, define $P(y|x) \propto \exp(\sum_i w_i f_i(x_i))$, where i indexes sets in the power-set of ${x_1,\ldots,x_d}$, and $f_i$ returns 1 if all variables in i'th set are 1, 0 otherwise. Then you can express each $w_i$ in this logistic regression model as a logarithm of a rational function of statistics of the data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there other interesting examples when closed form exists?&lt;/p&gt;&#xA;" OwnerUserId="511" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-08-30T09:48:48.037" LastActivityDate="2011-01-30T03:26:47.567" Title="When is logistic regression solved in closed form?" Tags="&lt;logistic-regression&gt;" AnswerCount="1" CommentCount="5" FavoriteCount="1" />
  <row Id="950" PostTypeId="2" ParentId="373" CreationDate="2010-07-28T23:28:44.003" Score="0" ViewCount="" Body="&lt;p&gt;One does not need to know about conditional probability or Bayes Theorem to figure out that it is best to switch your answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose you initially pick Door 1.   Then the probability of Door 1 being a winner is 1/3 and the probability of Doors 2 or 3 being a winner is 2/3.   If Door 2 is shown to be a loser by the host's choice then the probabilty that 2 or 3 is a winner is still 2/3.   But since Door 2 is a loser, Door 3 must have a 2/3 probability of being a winner.     &lt;/p&gt;&#xA;" OwnerUserId="99" LastActivityDate="2010-07-28T23:28:44.003" />
  <row Id="951" PostTypeId="1" CreationDate="2010-07-29T00:48:11.603" Score="4" ViewCount="188" Body="&lt;p&gt;What is the relationship between a Nonhomogeneous Poisson process and a process that has heavy tail distribution for its inter arrival times?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any pointer to a resource that can shed some light on this question would be hugely appreciated&lt;/p&gt;&#xA;" OwnerUserId="172" LastActivityDate="2010-07-29T18:39:45.200" Title="Nonhomogeneous Poisson and Heavy tail inter arrival time distribution" Tags="&lt;distributions&gt;&lt;poisson&gt;" AnswerCount="2" />
  <row Id="952" PostTypeId="1" CreationDate="2010-07-29T01:45:55.080" Score="1" ViewCount="100" Body="&lt;p&gt;Hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a survey of 16 questions, each with four possible responses. The purpose of the survey is to measure the respondent's propensity towards four categories (which we will denote A, B, C, D). Each of the four responses per question are representative of an aspect of the four categories, A, B, C, D.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The respondent rank orders each of the four responses (we will denote the first response by &quot;4&quot;, the second by &quot;3&quot;, etc).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To score the categories, we add the responses up based on the coding above. There are 16 x (4 + 3 + 2 + 1) = 160 total points. The sums for each category are computed, and the maximum score is deemed the respondent's dominant category.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore each survey looks like the following (in CSV format)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;question_num, A, B, C, D&lt;br /&gt;&#xA;1, 4, 3, 1, 2&lt;br /&gt;&#xA;2, 3, 4, 1, 2&lt;br /&gt;&#xA;3, 3, 4, 2, 1&lt;br /&gt;&#xA;4, 4, 3, 1, 2&lt;br /&gt;&#xA;5, 4, 3, 1, 2&lt;br /&gt;&#xA;6, 4, 3, 2, 1&lt;br /&gt;&#xA;7, 4, 3, 1, 2&lt;br /&gt;&#xA;.&lt;br /&gt;&#xA;.&lt;br /&gt;&#xA;.&lt;br /&gt;&#xA;16, 3, 4, 1, 2&lt;br /&gt;&#xA;sums, 64, 48, 24, 24&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have about 325 surveys completed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is that I want to look for possible redundancy in the survey so I can eliminate some questions easing the burden on the future respondents.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My first pass was to do a multi-logistic regression with the response as the dominant category (described above). Would this&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not sure PCA will reveal anything useful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any thoughts?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks,&lt;br /&gt;&#xA;Adam&lt;/p&gt;&#xA;" OwnerUserId="513" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-16T12:42:47.950" LastActivityDate="2010-09-16T12:42:47.950" Title="find survey redundancy" Tags="&lt;logistic-regression&gt;&lt;survey&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="953" PostTypeId="2" ParentId="913" CreationDate="2010-07-29T02:00:32.547" Score="1" ViewCount="" Body="&lt;p&gt;I agree with the suggestions about running a regression possibly with log(y) as the outcome variable or some other suitable transformation. I just wanted to add one comment, if you are reporting the bivariate association, you might prefer:&#xA;(a) to correlate log(x) and log(y),&#xA;(b) Spearman's rho, which correlates the ranks of the two variables.&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-07-29T02:00:32.547" />
  <row Id="954" PostTypeId="2" ParentId="944" CreationDate="2010-07-29T02:11:34.563" Score="2" ViewCount="" Body="&lt;p&gt;To follow on from Dirk's comment, if you don't like your current IDE, check out some of the existing discussion on R IDEs:&#xA;&lt;a href=&quot;http://stackoverflow.com/questions/1439059/best-ide-texteditor-for-r&quot;&gt;http://stackoverflow.com/questions/1439059/best-ide-texteditor-for-r&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-07-29T02:11:34.563" />
  <row Id="955" PostTypeId="1" CreationDate="2010-07-29T03:25:03.290" Score="0" ViewCount="242" Body="&lt;p&gt;Just wonder, is there any data analysis/ statistic/ data mining work that are available on freelance basis?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This could be subjective and argumentative, which is why I put it as CW. &lt;/p&gt;&#xA;" OwnerUserId="175" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-16T07:08:24.013" LastActivityDate="2010-09-16T07:08:24.013" ClosedDate="2010-07-29T12:38:50.240" Title="Data Analysis Work-- Is there Any Freelance Opportunity?" Tags="&lt;job&gt;" AnswerCount="1" CommentCount="6" />
  <row Id="956" PostTypeId="2" ParentId="870" CreationDate="2010-07-29T04:06:29.247" Score="5" ViewCount="" Body="&lt;p&gt;As Robin said, you've got the Benjamini-Hochberg method backwards. With that method, you set a value for Q (upper case Q; the maximum desired FDR) and it then sorts your comparisons into two piles. The goal is that no more than Q% of the comparisons in the &quot;discovery&quot; pile are false (and thus at least 100%-Q%) are true. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you computed a new value for each comparison, which is the value of Q at which that comparisons would just barely be considered a discovery, then those new values are q-values (lower case q; see the link to a paper by John Storey in the original question).&lt;/p&gt;&#xA;" OwnerUserId="25" LastActivityDate="2010-07-29T04:06:29.247" CommentCount="1" />
  <row Id="957" PostTypeId="2" ParentId="887" CreationDate="2010-07-29T04:15:44.310" Score="0" ViewCount="" Body="&lt;p&gt;I think you need to nail down the question you are asking, before you can compute an answer. I think this question is way too vague to answer: &quot;test whether it is an vis-a-vis the general population&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The only question I think you can answer is this one: If the new value came from the same population as the others, what is the chance that it will be so far (or further) from the sample mean? That is the question that your equation will begin to answer, although it is not quite right. Here is a corrected equation that includes n.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;t = (x - mean(S))/(stdev(S)/sqrt(n))&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Compute the corresponding P value (with n-1 degrees of freedom) and you've answered the question.&lt;/p&gt;&#xA;" OwnerUserId="25" LastActivityDate="2010-07-29T04:15:44.310" CommentCount="2" />
  <row Id="958" PostTypeId="2" ParentId="955" CreationDate="2010-07-29T05:55:55.163" Score="1" ViewCount="" Body="&lt;p&gt;The short answer is &quot;yes&quot;. Such work does exist.&#xA;For example, there is sometimes consulting work in and around universities.&#xA;Also, some companies wish to outsource data analysis and statistical activities.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, I found that word of mouth was a powerful tool. Once you build up a good reputation in a given community, additional requests for work will follow.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;see: &lt;a href=&quot;http://www.google.com/search?sourceid=chrome&amp;amp;ie=UTF-8&amp;amp;q=statistics+jobs&quot; rel=&quot;nofollow&quot;&gt;http://www.google.com/search?sourceid=chrome&amp;amp;ie=UTF-8&amp;amp;q=statistics+jobs&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-07-29T05:55:55.163" />
  <row Id="959" PostTypeId="2" ParentId="952" CreationDate="2010-07-29T06:01:01.627" Score="0" ViewCount="" Body="&lt;p&gt;I've never had to perform such analyses, but there is an academic literature on the factor analysis of ipsative tests that would be relevant:&#xA;e.g.,&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://deepblue.lib.umich.edu/bitstream/2027.42/68736/2/10.1177_004912418000900206.pdf&quot; rel=&quot;nofollow&quot;&gt;http://deepblue.lib.umich.edu/bitstream/2027.42/68736/2/10.1177_004912418000900206.pdf&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.informaworld.com/smpp/content~db=all~content=a785042624&quot; rel=&quot;nofollow&quot;&gt;http://www.informaworld.com/smpp/content~db=all~content=a785042624&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-07-29T06:01:01.627" />
  <row Id="960" PostTypeId="2" ParentId="952" CreationDate="2010-07-29T07:05:38.310" Score="2" ViewCount="" Body="&lt;p&gt;So restructure your data merging all user responses, so in such form:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Q1 Q2 Q3 ...&#xA;user1 rank for option1 for Q1, user1 rank for option1 for Q2, ...&#xA;user1 rank for option2, ...&#xA;...&#xA;user2 rank for option1, ...&#xA;...&#xA;user325 rank for option4, ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And then cluster the questions. I recommend agglomerative clustering, there it is easy to see what questions can be removed.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-07-29T07:05:38.310" />
  <row Id="961" PostTypeId="1" AcceptedAnswerId="971" CreationDate="2010-07-29T07:07:36.513" Score="7" ViewCount="536" Body="&lt;p&gt;Hi everyone,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a dataset made up of elements from three groups, let's call them G1, G2, and G3.&#xA;I analysed certain characteristics of these elements and divided them into 3 types of &quot;behaviour&quot; T1, T2, and T3 (I used cluster analysis to do that).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, now I have a 3 x 3 contingency table like this with the counts of elements in the three groups divided by type:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;      |    T1   |    T2   |    T3   |&#xA;------+---------+---------+---------+---&#xA;  G1  |   18    |   15    |   65    | &#xA;------+---------+---------+---------+---&#xA;  G2  |   20    |   10    |   70    |&#xA;------+---------+---------+---------+---&#xA;  G3  |   15    |   55    |   30    |&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, I can run a Fisher test on these data in R&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data &amp;lt;- matrix(c(18, 20, 15, 15, 10, 55, 65, 70, 30), nrow=3)&#xA;fisher.test(data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and I get&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   Fisher's Exact Test for Count Data&#xA;&#xA;data:  data &#xA;p-value = 9.028e-13&#xA;alternative hypothesis: two.sided     &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So my questions are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;is it correct to use Fisher test this way?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;how do I know who is different from who? Is there a post-hoc test I can use? Looking at the data I would say the 3&lt;sup&gt;rd&lt;/sup&gt; group has a different behaviour from the first two, how do I show that statistically? &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;someone pointed me to logit models: are they a viable option for this type of analysis?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;any other option to analyse this type of data?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Thank you a lot&lt;/p&gt;&#xA;&#xA;&lt;p&gt;nico&lt;/p&gt;&#xA;" OwnerUserId="582" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-16T06:54:07.353" LastActivityDate="2010-09-16T06:54:07.353" Title="Statistical test for n x m contingency tables" Tags="&lt;contingency-tables&gt;" AnswerCount="2" FavoriteCount="4" />
  <row Id="962" PostTypeId="2" ParentId="942" CreationDate="2010-07-29T07:10:39.967" Score="5" ViewCount="" Body="&lt;p&gt;Wavelet is usefull to detect singularities in a signal (see for example the paper &lt;a href=&quot;http://www.math.univ-toulouse.fr/~bigot/Site/Publications_files/Spectrometry.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; (see figure 3 for an illustration) and the references mentionned in the paper.  I guess singularities are sometime anomaly? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The idea here is that the Continuous wavelet transform &lt;strong&gt;(CWT)&lt;/strong&gt; has maxima lines that propagates along frequencies, the longuer the line is the higher is the singularity. See Figure 3 in the paper to see what I mean ! note that there is a free matlab code related to that paper, it should be &lt;a href=&quot;http://www.math.univ-toulouse.fr/~bigot/Site/Software.html&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Additionally, I can give you some heuristic about &lt;strong&gt;why&lt;/strong&gt; the DISRCETE (preceding example is about the continuous one) wavelet transform (&lt;strong&gt;DWT&lt;/strong&gt;) &lt;strong&gt;is interesting for a statistician&lt;/strong&gt; (excuse non exhaustivity) :   &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;There is a wide class of (realistic (Besov space)) signal that are transformed into a &lt;strong&gt;sparse&lt;/strong&gt; sequence by the wavelet transform. (&lt;strong&gt;compression property&lt;/strong&gt;)&lt;/li&gt;&#xA;&lt;li&gt;A wide class of (quasi stationary) process that are transform into a sequence with almost uncorrelated features (&lt;strong&gt;decorrelation property&lt;/strong&gt;)&lt;/li&gt;&#xA;&lt;li&gt;Wavelet coefficients contain are containing an information that is localized in &lt;strong&gt;time and in frequency&lt;/strong&gt; (at different scales). (multiscale property)&lt;/li&gt;&#xA;&lt;li&gt;Wavelet coefficients of a signal &lt;strong&gt;concentrate on its singularities&lt;/strong&gt;. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="223" LastEditorUserId="223" LastEditorDisplayName="" LastEditDate="2010-11-09T08:27:30.530" LastActivityDate="2010-11-09T08:27:30.530" CommentCount="1" />
  <row Id="963" PostTypeId="2" ParentId="946" CreationDate="2010-07-29T08:15:59.907" Score="3" ViewCount="" Body="&lt;p&gt;As I read in help for the &lt;code&gt;t.test&lt;/code&gt;, it is only applicable for the 2-sample tests. If you want to perform it to every combination of the columns of a matrix A, taken 2 at a time, you could do something like this (for the moment, I can't recall a better way)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;apply(combn(1:dim(A)[2],2),2,function(x) t.test(A[,x[1]],A[,x[2]])) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;or, if you just want the p.values&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;t(apply(combn(1:dim(A)[2],2),2,function(x) c(x[1],x[2],(t.test(A[,x[1]],A[,x[2]]))$p.value)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="339" LastEditorUserId="339" LastEditorDisplayName="" LastEditDate="2010-07-29T08:22:43.180" LastActivityDate="2010-07-29T08:22:43.180" CommentCount="2" />
  <row Id="964" PostTypeId="1" AcceptedAnswerId="965" CreationDate="2010-07-29T08:28:36.520" Score="2" ViewCount="226" Body="&lt;p&gt;So in R, for instance, this would be:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;my_ts_logged_diffed = diff(log(some_ts_object))&#xA;plot(my_ts_logged_diffed)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This seems to be part of every experienced analyst/forecaster analytical workflow--in particular, a &lt;em&gt;visual examination of the plotted data&lt;/em&gt;. What are they looking for--i.e., what useful information does this transformation help reveal?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Similarly, I have a pretty good selection of time series textbooks, tutorials, and the like; nearly all of them mention this analytical step, but none of them say why it's done (i am sure there's a good reason, and one that's apparently too obvious to even mention).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(i do indeed routinely rely on this transformation but only for the limited purpose of testing for a normal distribution (i think the test is called &lt;em&gt;Shapiro-Wilk&lt;/em&gt;). The application of the test just involves (assuming i am applying it correctly) comparing a couple of parameters (a 'W' parameter and the p-value) against a baseline--the Test doesn't appear to require plotting the data).&lt;/p&gt;&#xA;" OwnerUserId="438" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-07-31T00:48:08.403" LastActivityDate="2010-11-02T01:47:17.577" Title="What are analysts looking for when they plot a differenced, logged time series?" Tags="&lt;time-series&gt;&lt;data-transformation&gt;" AnswerCount="6" FavoriteCount="1" />
  <row Id="965" PostTypeId="2" ParentId="964" CreationDate="2010-07-29T08:46:34.730" Score="3" ViewCount="" Body="&lt;p&gt;Most growth/decay processes will at most change the moving quantity at an exponential rate. The differences of the logs of the quantity relate to the local slope &amp;lambda;, so for a underlying exponential growth or decay process it would be flat in t. Any deviation from flat gives you hints if and where there are switchovers between different purly exponential pieces; also for chaotic behavior you would expect more than linear growth or decay, i.e. a &quot;local &amp;lambda;&quot; curve that is not flat for small t-ranges.&lt;/p&gt;&#xA;" OwnerUserId="56" LastActivityDate="2010-07-29T08:46:34.730" CommentCount="1" />
  <row Id="966" PostTypeId="2" ParentId="964" CreationDate="2010-07-29T10:22:37.673" Score="2" ViewCount="" Body="&lt;p&gt;This is often used for a price to return transformation based on assuming continuously compounded returns. The Campbell, Lo, and MacKinlay book (Econometrics of Financial Markets, 1997) lays it out quite nicely:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Define r_t as the log of gross returns 1 + R_t:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;r_t := log(1 + R_t) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which is the same as the log of the previous prices&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;log(P_t / P_{t-1}) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which is the same as &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;p_t - p_{t-1}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which is what you asked about. &lt;/p&gt;&#xA;" OwnerUserId="334" LastActivityDate="2010-07-29T10:22:37.673" CommentCount="2" />
  <row Id="967" PostTypeId="2" ParentId="726" CreationDate="2010-07-29T10:50:53.227" Score="15" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;I keep saying that the sexy job in the next 10 years will be statisticians. And I'm not kidding.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Val Harian&lt;/p&gt;&#xA;" OwnerUserId="215" LastEditorUserId="795" LastEditorDisplayName="" LastEditDate="2010-10-02T17:09:21.573" LastActivityDate="2010-10-02T17:09:21.573" CommunityOwnedDate="2010-07-29T10:50:53.227" CommentCount="2" />
  <row Id="968" PostTypeId="2" ParentId="726" CreationDate="2010-07-29T10:57:18.647" Score="20" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;All generalizations are false,&#xA;  including this one.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Mark Twain&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="1356" LastActivityDate="2010-07-29T10:57:18.647" CommunityOwnedDate="2010-07-29T10:57:18.647" CommentCount="1" />
  <row Id="969" PostTypeId="2" ParentId="726" CreationDate="2010-07-29T11:25:13.733" Score="3" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;An ecologist is a statistician who likes to be outside. &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;-- apparently a good friend of &lt;a href=&quot;http://r.789695.n4.nabble.com/Why-software-fails-in-scientific-research-td1573062.html#a2275423&quot; rel=&quot;nofollow&quot;&gt;Murray Cooper&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="144" LastEditorUserId="795" LastEditorDisplayName="" LastEditDate="2010-12-03T04:04:09.337" LastActivityDate="2010-12-03T04:04:09.337" CommunityOwnedDate="2010-07-29T11:25:13.733" />
  <row Id="970" PostTypeId="2" ParentId="951" CreationDate="2010-07-29T11:41:41.960" Score="1" ViewCount="" Body="&lt;p&gt;In general there is none. A Poisson process has inter-arrival times that are exponentially distributed, which does not have heavy tails. &lt;/p&gt;&#xA;" OwnerUserId="247" LastActivityDate="2010-07-29T11:41:41.960" />
  <row Id="971" PostTypeId="2" ParentId="961" CreationDate="2010-07-29T11:41:49.370" Score="7" ViewCount="" Body="&lt;p&gt;At first I think that the Fisher test is used correctly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Count data are better handled using log-linear models (not logit, to ensure that the fitted values are bounded below). In R you can specify &lt;code&gt;family=poisson&lt;/code&gt; (which sets errors = Poisson and link = log). The log link ensures that all the fitted values are positive, while the Poisson errors take account of the fact that the data are integer and have variances that are equal to their means.&#xA;e.g. &lt;code&gt;glm(y~x,poisson)&lt;/code&gt; and the model is fitted with a log link and Poisson errors (to account for the non-normality). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In cases where there is overdispersion (the residual deviance should be equal to the residual degrees of freedom, if the Poisson errors assumption is appropriate), instead of using &lt;code&gt;quasipoisson&lt;/code&gt; as the error family, you could fit a negative binomial model.&#xA;(This involves the function &lt;code&gt;glm.nb&lt;/code&gt; from package &lt;code&gt;MASS&lt;/code&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case you could fit and compare models using using commands like the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;observed &amp;lt;- as.vector(data)&#xA;Ts&amp;lt;-factor(rep(c(&quot;T1&quot;,&quot;T2&quot;,&quot;T3&quot;),each=3))&#xA;Gs&amp;lt;-factor(rep(c(&quot;G1&quot;,&quot;G2&quot;,&quot;G3&quot;),3))&#xA;&#xA;model1&amp;lt;-glm(observed~Ts*Gs,poisson)&#xA;&#xA;#or and a model without the interaction terms&#xA;model2&amp;lt;-glm(observed~Ts+Gs,poisson)&#xA;&#xA;&#xA;#you can compare the two models using anova with a chi-squared test&#xA;anova(model1,model2,test=&quot;Chi&quot;)&#xA;summary(model1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Always make sure that your minimal model contains all the nuisance variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for how do we know who is different from who, there are some plots that may help you. R function &lt;code&gt;assocplot&lt;/code&gt; produces an association plot indicating deviations from independence of rows and columns in a two dimensional contingency table.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are the same data plotted as a mosaic plot&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mosaicplot(data, shade = TRUE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="339" LastActivityDate="2010-07-29T11:41:49.370" CommentCount="3" />
  <row Id="972" PostTypeId="2" ParentId="602" CreationDate="2010-07-29T11:43:54.703" Score="1" ViewCount="" Body="&lt;p&gt;There are no general heuristics, you should make a grid search, especially since the value of nu must be between 0-1.&lt;/p&gt;&#xA;" OwnerUserId="566" LastActivityDate="2010-07-29T11:43:54.703" />
  <row Id="973" PostTypeId="1" AcceptedAnswerId="1020" CreationDate="2010-07-29T12:02:28.347" Score="8" ViewCount="595" Body="&lt;p&gt;What are the &lt;strong&gt;freely available data set for classification with more than 1000 features&lt;/strong&gt; (or sample points if it contains curves)? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is already a community wiki about free data sets:&#xA;&lt;a href=&quot;http://stats.stackexchange.com/questions/7/locating-freely-available-data-samples/&quot;&gt;http://stats.stackexchange.com/questions/7/locating-freely-available-data-samples/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But here, it would be nice to have a &lt;strong&gt;more focused list that can be used more conveniently&lt;/strong&gt;, also I propose the following rules:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;One post &lt;strong&gt;per dataset&lt;/strong&gt; &lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;No&lt;/strong&gt; link to set of dataset &lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;each data set &lt;strong&gt;must&lt;/strong&gt; be associated with&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;a &lt;strong&gt;name&lt;/strong&gt; (to figure out what it is about)  and a link to the dataset (R datasets can be named with package name)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;the number of features (let say it is &lt;strong&gt;p&lt;/strong&gt;) the size of the dataset (let say it is &lt;strong&gt;n&lt;/strong&gt;) and the number of labels/class (let say it is &lt;strong&gt;k&lt;/strong&gt;)    &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;a typical &lt;strong&gt;error rate&lt;/strong&gt; from your experience (state the used algorithm in to words) or from the litterature (in this last case link the paper) &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="223" LastEditorUserId="223" LastEditorDisplayName="" LastEditDate="2010-12-21T13:04:43.820" LastActivityDate="2010-12-21T13:04:43.820" CommunityOwnedDate="2010-07-29T12:02:28.347" Title="Free data set for very high dimensional classification" Tags="&lt;machine-learning&gt;&lt;classification&gt;&lt;dataset&gt;&lt;large-data&gt;" AnswerCount="5" CommentCount="0" FavoriteCount="4" />
  <row Id="975" PostTypeId="2" ParentId="927" CreationDate="2010-07-29T12:44:50.820" Score="2" ViewCount="" Body="&lt;p&gt;Another good podcast is &lt;a href=&quot;http://www.bbc.co.uk/radio4/features/in-our-time/&quot; rel=&quot;nofollow&quot;&gt;In our time&lt;/a&gt; by the BBC. It's a weekly podcast (off air for the summer) that deals with topics in History, Religion and Science. I would say that about 1 in 12 podcasts deal with Mathematics and Statistics. Take a look at the podcast archive for &lt;a href=&quot;http://www.bbc.co.uk/radio4/features/in-our-time/archive/science&quot; rel=&quot;nofollow&quot;&gt;Science subjects&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="8" LastActivityDate="2010-07-29T12:44:50.820" CommunityOwnedDate="2010-07-29T12:44:50.820" />
  <row Id="977" PostTypeId="1" AcceptedAnswerId="978" CreationDate="2010-07-29T13:43:53.010" Score="2" ViewCount="125" Body="&lt;p&gt;When we are monitoring movements of structures we normally install monitoring points onto the structure before we do any work which might cause movement. This gives us chance to take a few readings before we start doing the work to 'baseline' the readings. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Quite often the data is quite variable (the variations in the reading can easily be between 10 and 20% of the fianl movement). The measurements are also often affected by the environment in which they are taken so one set of measurements taken on one project may not have the same accuracy as measurements on another project. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any statisitcal method, or rule of thumb that can be applied to say how many baseline readings need to be taken to give a certain accuracy before the first reading is taken? Are there any rules of humb that can be applied to this situation? &lt;/p&gt;&#xA;" OwnerUserId="210" LastActivityDate="2010-07-30T15:37:06.190" Title="How many measurements are needed to 'baseline' a measurement?" Tags="&lt;variance&gt;&lt;measurement&gt;" AnswerCount="3" />
  <row Id="978" PostTypeId="2" ParentId="977" CreationDate="2010-07-29T13:51:07.953" Score="3" ViewCount="" Body="&lt;p&gt;I think you should look at &lt;a href=&quot;http://en.wikipedia.org/wiki/Statistical_power&quot; rel=&quot;nofollow&quot;&gt;power calculations&lt;/a&gt;. These are often used to decide the sample size of survey or clinical trial. Taken from wikipedia:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;A priori power analysis is conducted&#xA;  prior to the research study, and is&#xA;  typically used to determine an&#xA;  appropriate sample size to achieve&#xA;  adequate power.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="8" LastActivityDate="2010-07-29T13:51:07.953" />
  <row Id="980" PostTypeId="1" CreationDate="2010-07-29T14:04:20.853" Score="3" ViewCount="375" Body="&lt;p&gt;I haven't studied statistics for over 10 years (and then just a basic course), so maybe my question is a bit hard to understand.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyway, what I want to do is reduce the number of data points in a series. The x-axis is number of milliseconds since start of measurement and the y-axis is the reading for that point. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Often there is thousands of data points, but I might only need a few hundreds. So my question is: How do I accurately reduce the number of data points?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the process called? (So I can google it)&#xA;Are there any prefered algorithms (I will implement it in C#)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope you got some clues. Sorry for my lack of proper terminology.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Edit: More details comes here:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The raw data I got is heart rate data, and in the form of number of milliseconds since last beat. Before plotting the data I calculate number of milliseconds from first sample, and the bpm (beats per minute) at each data point (60000/timesincelastbeat). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to visualize the data, i.e. plot it in a line graph. I want to reduce the number of points in the graph from thousands to some hundreds. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One option would be to calculate the average bpm for every second in the series, or maybe every 5 seconds or so. That would have been quite easy if I knew I would have at least one sample for each of those periods (seconds of 5-seconds-intervals).&lt;/p&gt;&#xA;" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-07-31T00:49:57.463" LastActivityDate="2010-08-16T17:20:25.063" Title="How do I reduce the number of data points in a series?" Tags="&lt;data-visualization&gt;" AnswerCount="6" CommentCount="6" FavoriteCount="2" />
  <row Id="981" PostTypeId="2" ParentId="980" CreationDate="2010-07-29T14:15:09.800" Score="5" ViewCount="" Body="&lt;p&gt;You have two problems: too many points and how to smooth over the remaining points.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Thinning your sample&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have too many observations arriving in real time, you could always use &lt;a href=&quot;http://en.wikipedia.org/wiki/Simple_random_sample&quot; rel=&quot;nofollow&quot;&gt;simple random sampling&lt;/a&gt; to thin your sample. Note, for this too be true, the number of points would have to be very large.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose you have &lt;em&gt;N&lt;/em&gt; points and you only want &lt;em&gt;n&lt;/em&gt; of them. Then generate &lt;em&gt;n&lt;/em&gt; random numbers from a discrete uniform &lt;em&gt;U(0, N-1)&lt;/em&gt; distribution. These would be the points you use.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to do this sequentially, i.e. at each point you decide to use it or not, then just accept a point with probability &lt;em&gt;p&lt;/em&gt;. So if you set &lt;em&gt;p=0.01&lt;/em&gt; you would accept (on average) 1 point in a hundred. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your data is unevenly spread and you only want to thin dense regions of points, then just make your thinning function a bit more sophisticated. For example, instead of &lt;em&gt;p&lt;/em&gt;, what about:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://mathurl.com/2vlkhjs.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where &lt;img src=&quot;http://mathurl.com/343luwd.png&quot; alt=&quot;alt text&quot;&gt; is a positive number and &lt;em&gt;t&lt;/em&gt; is the time since the last observation. If the time between two points is large, i.e. large &lt;em&gt;t&lt;/em&gt;, the probability of accepting a point will be one. Conversely, if two points are close together, the probability of accepting a point will be &lt;em&gt;1-p&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You will need to experiment with values of &lt;img src=&quot;http://mathurl.com/343luwd.png&quot; alt=&quot;alt text&quot;&gt; and &lt;em&gt;p&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Smoothing&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Possibly something like a simple moving average type scheme. Or you could go for something more advanced like a &lt;a href=&quot;http://en.wikipedia.org/wiki/Kernel_smoother&quot; rel=&quot;nofollow&quot;&gt;kernel smoother&lt;/a&gt; (as others suggested). You will need to be careful that you don't smooth too much, since I assume that a sudden drop should be picked up very quickly in your scenario. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There should be &lt;em&gt;C#&lt;/em&gt; libraries available for this sort of stuff.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Conclusion&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thin if necessary, then smooth.&lt;/p&gt;&#xA;" OwnerUserId="8" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-07-30T11:00:12.113" LastActivityDate="2010-07-30T11:00:12.113" CommentCount="11" />
  <row Id="982" PostTypeId="2" ParentId="980" CreationDate="2010-07-29T14:18:02.383" Score="5" ViewCount="" Body="&lt;p&gt;Well, I think the word you're looking for is &quot;sampling,&quot; but I'm not sure why you want to do it. Thousands of data points isn't very many. Or are you looking just to plot a smaller number of equally-spaced points? That's usually called &quot;binning.&quot; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is your goal to generate a visualization? In that case, you might want to keep the raw data, plot it as a scattergraph, then overlay some sort of central tendency (regression line, spline, whatever) to communicate whatever the takehome message ought to be.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or is your goal to numerically summarize the results in some way? In that case, you might want to explain your problem in more detail!&lt;/p&gt;&#xA;" OwnerUserId="6" LastActivityDate="2010-07-29T14:18:02.383" CommentCount="3" />
  <row Id="983" PostTypeId="2" ParentId="977" CreationDate="2010-07-29T14:26:53.593" Score="2" ViewCount="" Body="&lt;p&gt;It really depends on the amount of variance relative to the size of a measurement that you care about. If you need to be able to tell the difference between a mean of 2 and a mean of 0, and your data look like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;-4.4 3.8 -2.0 -5.1 0.2 7.1 0.9 -5.4 2.8 0.5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then you're going to need a lot more data! But if you only care if the average is less than or more than 10, then that much data is adequate. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;@cgillespie gives the technically correct response. You need to have some idea what size of effect you care about, as well as some idea how much variance your measurements have. If the equations of power analysis are more than you can deal with, you can always use random numbers in an Excel spreadsheet! Generate columns of random numbers with a normal distribution and various means and variances, then figure out whether the confidence intervals (2 * standard deviation / sqrt(N)) around the mean of each set of numbers include differences that you might are about. Do that a bunch of times. That'll give you a good idea of how many measurements you need, and it's not too hard to explain to others.&lt;/p&gt;&#xA;" OwnerUserId="6" LastActivityDate="2010-07-29T14:26:53.593" CommentCount="1" />
  <row Id="984" PostTypeId="2" ParentId="980" CreationDate="2010-07-29T14:29:16.240" Score="1" ViewCount="" Body="&lt;p&gt;You're not providing enough information.  Why do you want to reduce the data points.  A few thousand is nothing these days.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given that you want the same result each time you view the same data perhaps you want to simply bin averages.  You have variable spacing on your x-axis.  Maybe you're trying to make that consistent?  In that case you would set a bin width of perhaps 50 msec, or 100, and then average all the points in there.  Make the bin width as large as you need to reduce the data points to the size of the set you want.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's really a hard question to answer without a reason for why you're getting rid of data.&lt;/p&gt;&#xA;" OwnerUserId="601" LastActivityDate="2010-07-29T14:29:16.240" CommentCount="1" />
  <row Id="986" PostTypeId="2" ParentId="961" CreationDate="2010-07-29T15:20:56.393" Score="3" ViewCount="" Body="&lt;p&gt;You can use &lt;em&gt;multinom&lt;/em&gt; from nnet package for multinomial regression. &#xA;Post-hoc tests you can use &lt;em&gt;linearHypothesis&lt;/em&gt; from car package.&#xA;You can conduct test of independence using &lt;em&gt;linearHypothesis&lt;/em&gt; (Wald test) or &lt;em&gt;anova&lt;/em&gt; (LR test).&lt;/p&gt;&#xA;" OwnerUserId="419" LastActivityDate="2010-07-29T15:20:56.393" />
  <row Id="988" PostTypeId="2" ParentId="980" CreationDate="2010-07-29T15:40:49.490" Score="0" ViewCount="" Body="&lt;p&gt;From what you say, both your x's and y's are univariate. Go for smoothing:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.google.com/url?sa=t&amp;amp;source=web&amp;amp;cd=4&amp;amp;ved=0CCkQFjAD&amp;amp;url=http%3A%2F%2Fwww.stat.cmu.edu%2F~larry%2F%3Dstat707%2Fnotes5.pdf&amp;amp;ei=vKBRTLP3F4zeOImt-cwE&amp;amp;usg=AFQjCNHk8dIhMW76xNJCEQ6aF62R2EEWtg&amp;amp;sig2=wlEuNrgXtgeUJMby2rpBeQ&quot; rel=&quot;nofollow&quot;&gt;http://www.google.com/url?sa=t&amp;amp;source=web&amp;amp;cd=4&amp;amp;ved=0CCkQFjAD&amp;amp;url=http%3A%2F%2Fwww.stat.cmu.edu%2F~larry%2F%3Dstat707%2Fnotes5.pdf&amp;amp;ei=vKBRTLP3F4zeOImt-cwE&amp;amp;usg=AFQjCNHk8dIhMW76xNJCEQ6aF62R2EEWtg&amp;amp;sig2=wlEuNrgXtgeUJMby2rpBeQ&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="603" LastActivityDate="2010-07-29T15:40:49.490" />
  <row Id="990" PostTypeId="2" ParentId="726" CreationDate="2010-07-29T15:43:35.187" Score="1" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;You may be too vague to be wrong and&#xA;  that's really bad cause that's just&#xA;  obscuring the issue.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Bruce Sterling &lt;/p&gt;&#xA;" OwnerUserId="237" LastActivityDate="2010-07-29T15:43:35.187" CommunityOwnedDate="2010-07-29T15:43:35.187" CommentCount="1" />
  <row Id="991" PostTypeId="2" ParentId="964" CreationDate="2010-07-29T15:45:39.227" Score="1" ViewCount="" Body="&lt;p&gt;usually, you plot such a series to check the extend to which it exhibits heteroskedasticity.&#xA;Depending on the answer, you may have to model the residuals, even if you are only interested in the mean. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the top of my head, this article is a practical example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.google.com/url?sa=t&amp;amp;source=web&amp;amp;cd=1&amp;amp;ved=0CBIQFjAA&amp;amp;url=http%3A%2F%2Fdss.ucsd.edu%2F~jhamilto%2FJHamilton_Engle.pdf&amp;amp;ei=CKJRTN-uO4XeOO_LhL4E&amp;amp;usg=AFQjCNEP4dL3_uRf28371SnBS4lhhnYsSw&amp;amp;sig2=tJRjwUetH8XTHuijcXYmbg&quot; rel=&quot;nofollow&quot;&gt;http://www.google.com/url?sa=t&amp;amp;source=web&amp;amp;cd=1&amp;amp;ved=0CBIQFjAA&amp;amp;url=http%3A%2F%2Fdss.ucsd.edu%2F~jhamilto%2FJHamilton_Engle.pdf&amp;amp;ei=CKJRTN-uO4XeOO_LhL4E&amp;amp;usg=AFQjCNEP4dL3_uRf28371SnBS4lhhnYsSw&amp;amp;sig2=tJRjwUetH8XTHuijcXYmbg&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="603" LastActivityDate="2010-07-29T15:45:39.227" />
  <row Id="992" PostTypeId="2" ParentId="913" CreationDate="2010-07-29T15:50:24.717" Score="1" ViewCount="" Body="&lt;p&gt;Try a bivariate robust regression &#xA;(see &lt;a href=&quot;http://cran.r-project.org/web/packages/rrcov/vignettes/rrcov.pdf&quot; rel=&quot;nofollow&quot;&gt;http://cran.r-project.org/web/packages/rrcov/vignettes/rrcov.pdf&lt;/a&gt; for an intro).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your data points are all positive, you might want to try to regress log(y) on log(x).&#xA;Note that log() is &lt;em&gt;not&lt;/em&gt; a substitute for a robust regression, but it sometimes makes the results more interpretable.&lt;/p&gt;&#xA;" OwnerUserId="603" LastActivityDate="2010-07-29T15:50:24.717" />
  <row Id="993" PostTypeId="2" ParentId="924" CreationDate="2010-07-29T15:57:21.187" Score="6" ViewCount="" Body="&lt;p&gt;A (two-sided) Fisher's Exact test gives p-value = &lt;strong&gt;0.092284&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;function p = fexact(k, x, m, n)&#xA;%FEXACT Fisher's Exact test.&#xA;%   Y = FEXACT(K, X, M, N) calculates the P-value for Fisher's&#xA;%   Exact Test.&#xA;%   K, X, M and N must be nonnegative integer vectors of the same&#xA;%   length.  The following must also hold:&#xA;%   X &amp;lt;= N &amp;lt;= M, X &amp;lt;= K &amp;lt;= M and K + N - M &amp;lt;= X.  Here:&#xA;%   K is the number of items in the group,&#xA;%   X is the number of items in the group with the feature,&#xA;%   M is the total number of items,&#xA;%   N is the total number of items with the feature,&#xA;&#xA;if nargin &amp;lt; 4&#xA;   help(mfilename);&#xA;   return;&#xA;end&#xA;nr = length(k);&#xA;if nr ~= length(x) | nr ~= length(m) | nr ~= length(n)&#xA;   help(mfilename);&#xA;   return;&#xA;end&#xA;&#xA;na = nan;&#xA;v = na(ones(nr, 1));&#xA;mi = max(0, k + n - m);&#xA;ma = min(k, n);&#xA;&#xA;d = hygepdf(x, m, k, n) * (1 + 5.8e-11);&#xA;for i = 1:nr&#xA;  y = hygepdf(mi(i):ma(i), m(i), k(i), n(i));&#xA;  v(i) = sum(y(y &amp;lt;= d(i)));&#xA;end&#xA;p = max(min(v, 1), 0);&#xA;p(isnan(v)) = nan;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For your example, try &lt;code&gt;fexact(1e6, 3, 2e6, 13)&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="506" LastEditorUserId="506" LastEditorDisplayName="" LastEditDate="2010-07-29T16:28:09.103" LastActivityDate="2010-07-29T16:28:09.103" CommentCount="2" />
  <row Id="994" PostTypeId="2" ParentId="421" CreationDate="2010-07-29T16:00:17.130" Score="2" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0713999225&quot; rel=&quot;nofollow&quot;&gt;The Drunkard's Walk: How Randomness Rules Our Lives&lt;/a&gt; by Leonard Mlodinow is an excellent book for laypeople. Enjoyable and educational.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It might not be a textbook, but it makes you think about the world in the right way.&lt;/p&gt;&#xA;" LastActivityDate="2010-07-29T16:00:17.130" CommunityOwnedDate="2010-07-29T16:00:17.130" />
  <row Id="995" PostTypeId="2" ParentId="614" CreationDate="2010-07-29T16:02:18.907" Score="0" ViewCount="" Body="&lt;p&gt;&lt;em&gt;Collaborative Statistics&lt;/em&gt; is CC BY: &lt;a href=&quot;http://cnx.org/content/col10522/latest/&quot; rel=&quot;nofollow&quot;&gt;http://cnx.org/content/col10522/latest/&lt;/a&gt;&lt;/p&gt;&#xA;" LastActivityDate="2010-07-29T16:02:18.907" CommunityOwnedDate="2010-07-29T16:02:18.907" />
  <row Id="996" PostTypeId="2" ParentId="213" CreationDate="2010-07-29T16:13:41.440" Score="5" ViewCount="" Body="&lt;p&gt;Multivariate robust estimators were designed for this!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;multivariate S-estimator:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://cran.r-project.org/web/packages/rrcov/vignettes/rrcov.pdf&quot; rel=&quot;nofollow&quot;&gt;http://cran.r-project.org/web/packages/rrcov/vignettes/rrcov.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="603" LastActivityDate="2010-07-29T16:13:41.440" />
  <row Id="997" PostTypeId="2" ParentId="924" CreationDate="2010-07-29T16:50:11.297" Score="8" ViewCount="" Body="&lt;p&gt;The huge denominators throw off one's intuition. Since the sample sizes are identical, and the proportions low, the problem can be recast: 13 events occurred, and were expected (by null hypothesis) to occur equally in both groups. In fact the split was 3 in one group and 10 in the other. How rare is that? The binomial test answers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Enter this line into R:&#xA;binom.test(3,13,0.5,alternative=&quot;two.sided&quot;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The two-tail P value is 0.09229, identical to four digits to the results of Fisher's test. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Looked at that way, the results are not surprising. The problem is equivalent to this one: If you flipped a coin 13 times, how surprising would it be to see three or fewer, or ten or more, heads. One of those outcomes would occur 9.23% of the time. &lt;/p&gt;&#xA;" OwnerUserId="25" LastEditorUserId="25" LastEditorDisplayName="" LastEditDate="2010-08-22T01:43:00.833" LastActivityDate="2010-08-22T01:43:00.833" CommentCount="2" />
  <row Id="998" PostTypeId="2" ParentId="924" CreationDate="2010-07-29T18:06:31.570" Score="0" ViewCount="" Body="&lt;p&gt;In addition to the other answers:&#xA;If you have 1,000,000 observations and when your event comes up only a few times, you are likely to want to look at a lot of different events.&#xA;If you look at 100 different events you will run into problems if you work with p&amp;lt;0.05 as criteria for significance. &lt;/p&gt;&#xA;" OwnerUserId="237" LastActivityDate="2010-07-29T18:06:31.570" />
  <row Id="999" PostTypeId="2" ParentId="138" CreationDate="2010-07-29T18:33:43.370" Score="1" ViewCount="" Body="&lt;p&gt;If you are coming from a SAS or SPSS background, check out:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://sites.google.com/site/r4statistics/&quot; rel=&quot;nofollow&quot;&gt;http://sites.google.com/site/r4statistics/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is the companion site to the book, R for SAS and SPSS Users by Robert Muenchen and a free version of the book can be found here.&lt;/p&gt;&#xA;" LastActivityDate="2010-07-29T18:33:43.370" CommunityOwnedDate="2010-08-01T18:56:25.603" />
  <row Id="1000" PostTypeId="2" ParentId="951" CreationDate="2010-07-29T18:39:45.200" Score="3" ViewCount="" Body="&lt;p&gt;Well, if you have a point process that you try modeling as a Poisson process, and find it has heavy tails, there are several possibilities.  What are the key assumptions for a Poisson Process:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-There is a constant rate function&#xA;-Events are memoryless, that is P(E in (t,t+d)) is independent of t and when other events are.&#xA;-The waiting time until the next event is exponentially distributed (kinda what the previous two are saying)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, how can you violate these assumptions to get heavy tails?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-Non-constant rate function.  If the rate function switches between, say, two values, you'll have too many short wait-times, and too many long wait-times, given the overall rate function.  This can show itself as having heavy tails.&#xA;-The waiting time is not exponentially distributed.  In which case, you don't have a Poisson process.  You have some other sort of point process.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that in the extreme case, any point process can be modeled by a NHPP - put a delta function at each event, and set the rate to 0 elsewhere.  I think we can all agree that this is a poor model, having little predictive power.   So if you are interested in a NHPP, you'll want to think a bit about whether that is the right model, or whether you are overly-adjusting a model to fit your data.&lt;/p&gt;&#xA;" OwnerUserId="549" LastActivityDate="2010-07-29T18:39:45.200" />
  <row Id="1001" PostTypeId="1" AcceptedAnswerId="3078" CreationDate="2010-07-29T18:46:47.247" Score="2" ViewCount="503" Body="&lt;p&gt;Hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have distributions from two different data sets and I would like to&#xA;measure how similar their distributions (in terms of their bin&#xA;frequencies) are. In other words, I am not interested in the correlation of &#xA;data point sequences but rather in the their distributional properties with respect to similarity. Currently I can only observe a similarity in eye-balling which is not enough. I don't want to assume causality and I don't want to predict at this point. So, I assume that correlation is the way to go. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Spearman's Correlation Coefficient is used to compare non-normal data and since I don't know anything about the real underlying distribution in my data, I think it would be a save bet. I wonder if this measure can also be used to&#xA;compare distributional data rather than the data poitns that are&#xA;summarized in a distribution. Here the example code in R that exemplifies&#xA;what I would like to check:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;aNorm &amp;lt;- rnorm(1000000)&#xA;bNorm &amp;lt;- rnorm(1000000)&#xA;cUni &amp;lt;- runif(1000000)&#xA;ha &amp;lt;- hist(aNorm)&#xA;hb &amp;lt;- hist(bNorm)&#xA;hc &amp;lt;- hist(cUni)&#xA;print(ha$counts)&#xA;print(hb$counts)&#xA;print(hc$counts)&#xA;# relatively similar&#xA;n &amp;lt;- min(c(NROW(ha$counts),NROW(hb$counts)))&#xA;cor.test(ha$counts[1:n], hb$counts[1:n], method=&quot;spearman&quot;)&#xA;# quite different&#xA;n &amp;lt;- min(c(NROW(ha$counts),NROW(hc$counts)))&#xA;cor.test(ha$counts[1:n], hc$counts[1:n], method=&quot;spearman&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Does this make sense or am I violating some assumptions of the coefficient?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks,&#xA;R.&lt;/p&gt;&#xA;" OwnerUserId="608" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-25T17:04:20.617" LastActivityDate="2010-09-25T17:04:20.617" Title="Spearman's Correlation Coefficient to compare distributions?" Tags="&lt;distributions&gt;" AnswerCount="3" CommentCount="5" FavoriteCount="2" />
  <row Id="1002" PostTypeId="2" ParentId="213" CreationDate="2010-07-29T18:48:16.383" Score="4" ViewCount="" Body="&lt;p&gt;What about (hierarchical) cluster analysis?&lt;/p&gt;&#xA;" OwnerUserId="609" LastActivityDate="2010-07-29T18:48:16.383" />
  <row Id="1003" PostTypeId="2" ParentId="614" CreationDate="2010-07-29T18:59:01.417" Score="3" ViewCount="" Body="&lt;p&gt;Try IPSUR, Introduction to Probability and Statistics Using R. It's free in the GNU sense of the word. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://ipsur.r-forge.r-project.org/book/index.php&quot; rel=&quot;nofollow&quot;&gt;http://ipsur.r-forge.r-project.org/book/index.php&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's definitely open source - on the download page you can download the LaTeX source or the lyx source used to generate this. &lt;/p&gt;&#xA;" OwnerUserId="36" LastActivityDate="2010-07-29T18:59:01.417" CommunityOwnedDate="2010-07-29T18:59:01.417" />
  <row Id="1004" PostTypeId="2" ParentId="1001" CreationDate="2010-07-29T18:59:32.213" Score="9" ViewCount="" Body="&lt;p&gt;Rather use &lt;a href=&quot;http://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test&quot; rel=&quot;nofollow&quot;&gt;Kolmogorov–Smirnov test&lt;/a&gt;, which is exactly what you need. R function &lt;code&gt;ks.test&lt;/code&gt; implements it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also check &lt;a href=&quot;http://stats.stackexchange.com/questions/411/motivation-for-kolmogorov-distance-between-distributions&quot;&gt;this question&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="88" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-07-29T19:14:52.727" LastActivityDate="2010-07-29T19:14:52.727" CommentCount="6" />
  <row Id="1005" PostTypeId="2" ParentId="138" CreationDate="2010-07-29T19:01:51.677" Score="6" ViewCount="" Body="&lt;p&gt;Try IPSUR, Introduction to Probability and Statistics Using R. It's a free book, free in the GNU sense of the word. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://ipsur.r-forge.r-project.org/book/index.php&quot; rel=&quot;nofollow&quot;&gt;http://ipsur.r-forge.r-project.org/book/index.php&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's definitely open source - on the download page you can download the LaTeX source or the lyx source used to generate this. &lt;/p&gt;&#xA;" OwnerUserId="36" LastActivityDate="2010-07-29T19:01:51.677" CommunityOwnedDate="2010-08-01T18:56:25.603" CommentCount="1" />
  <row Id="1006" PostTypeId="2" ParentId="924" CreationDate="2010-07-29T19:22:17.873" Score="4" ViewCount="" Body="&lt;p&gt;In this case Poisson is good approximation for distribution for number of cases.&#xA;There is simple formula to approximate variance of log RR (delta method) .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;log RR = 10/3 = 1.2, &#xA;se log RR = sqrt(1/3+1/10) = 0.66, so 95%CI = (-0.09; 2.5)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is not significant difference at 0.05 level using two-sided test.&#xA;LR based Chi-square test for Poisson model gives p=0.046 and Wald test p=0.067.&#xA;This results are similar to Pearson Chi-square test without continuity correction (Chi2 with correction p=0.096).&#xA;Another possibility is chisq.test with option simulate.p.value=T, in this case p=0.092 (for 100 000 simulations).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this case test statistics is rather discrete, so Fisher test can be conservative.&#xA;There is some evidence that difference can be significant. Before final conclusion data collecting process should be taken into account.&lt;/p&gt;&#xA;" OwnerUserId="419" LastActivityDate="2010-07-29T19:22:17.873" />
  <row Id="1008" PostTypeId="2" ParentId="871" CreationDate="2010-07-29T19:39:02.233" Score="0" ViewCount="" Body="&lt;p&gt;P value from theoretical point of view is some realization of random variable.&#xA;There is some standard (in probability) to use upper case letters for random variables and lower case for realizations.&#xA;In table headers we should use P (maybe &lt;em&gt;italicize&lt;/em&gt;), in text together with its value p=0.0012 and in text describing for example methodology p-value .&lt;/p&gt;&#xA;" OwnerUserId="419" LastActivityDate="2010-07-29T19:39:02.233" />
  <row Id="1009" PostTypeId="2" ParentId="899" CreationDate="2010-07-29T20:09:37.277" Score="3" ViewCount="" Body="&lt;ol&gt;&#xA;&lt;li&gt;For data in IQR range you should use&#xA;truncated normal distribution (for&#xA;example R package gamlss.tr) to&#xA;estimate parameters of this&#xA;distribution.  &lt;/li&gt;&#xA;&lt;li&gt;Another approach is using mixture models with 2 or 3 components (distributions). You can fit   such models using gamlss.mx package (distributions from package gamlss.dist can be specified for&#xA;each component of mixture).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="419" LastActivityDate="2010-07-29T20:09:37.277" />
  <row Id="1010" PostTypeId="2" ParentId="899" CreationDate="2010-07-29T21:20:25.090" Score="6" ViewCount="" Body="&lt;p&gt;If I understand correctly, then you can just fit a mixture of two Normals to the data. There are lots of R packages that are available to do this. This example uses the &lt;a href=&quot;http://cran.r-project.org/web/packages/mixtools/index.html&quot; rel=&quot;nofollow&quot;&gt;mixtools&lt;/a&gt; package:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#Taken from the documentation&#xA;library(mixtools)&#xA;data(faithful)&#xA;attach(faithful)&#xA;&#xA;#Fit two Normals&#xA;wait1 = normalmixEM(waiting, lambda = 0.5)&#xA;plot(wait1, density=TRUE, loglik=FALSE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This gives:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://img294.imageshack.us/img294/4213/kernal.jpg&quot; alt=&quot;Mixture of two Normals&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The package also contains more sophisticated methods - check the documentation.&lt;/p&gt;&#xA;" OwnerUserId="8" LastActivityDate="2010-07-29T21:20:25.090" />
  <row Id="1011" PostTypeId="2" ParentId="964" CreationDate="2010-07-29T21:21:02.530" Score="2" ViewCount="" Body="&lt;p&gt;Essentially they're looking for the log of the fold-change from one time point to the next because intuitively, it's easier to think about log-fold changes visually than actual fold-changes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Log-fold changes make decreases and increases simply a difference in sign, so that a log-2-fold increase is the same distance as a log-2-fold decrease (i.e. |log 2| = |log 0.5|). In addition, many systems exhibit multiplicative effects for independent events, e.g. biological systems and economic systems, where two independent x-fold increases results in an x^2 fold increase, whereas on a log scale, this becomes an additive (and thus easier to see) 2*log(x) increase.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, &lt;code&gt;diff(log(x))&lt;/code&gt; is prettier than &lt;code&gt;x[-1]/x[-length(x)]&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="378" LastActivityDate="2010-07-29T21:21:02.530" CommentCount="1" />
  <row Id="1012" PostTypeId="1" AcceptedAnswerId="1014" CreationDate="2010-07-29T21:29:57.590" Score="1" ViewCount="205" Body="&lt;p&gt;A colleague wants to compare models that use either a Gaussian distribution or a uniform distribution and for other reasons needs the standard devation of these two distributions to be equal.  In R I can do a simulation...&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sd(runif(100000000))&#xA;sd(runif(100000000,min=0,max=2))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and see that the calculated standard deviation is likely to be ~.2887 * the range of the uniform distribution.  However, I was wondering if there was an equation that could yield the exact value, and if so, what that formula was.&lt;/p&gt;&#xA;" OwnerUserId="196" LastEditorUserId="196" LastEditorDisplayName="" LastEditDate="2010-08-12T07:19:30.733" LastActivityDate="2010-09-29T14:34:45.027" Title="What would the calculated value of the standard deviation of a uniform distribution be?" Tags="&lt;distributions&gt;&lt;uniform&gt;&lt;normal-distribution&gt;" AnswerCount="2" CommentCount="3" />
  <row Id="1013" PostTypeId="2" ParentId="1012" CreationDate="2010-07-29T21:47:00.497" Score="1" ViewCount="" Body="&lt;p&gt;The standard deviation of the continous uniform distribution on the interval [0,1] is 12&lt;sup&gt;-1/2&lt;/sup&gt;&amp;asymp;0.288675. The &lt;a href=&quot;http://en.wikipedia.org/wiki/Uniform_distribution_%28continuous%29&quot; rel=&quot;nofollow&quot;&gt;Wikipedia article&lt;/a&gt; lists of it's more properties.&lt;/p&gt;&#xA;" OwnerUserId="56" LastEditorUserId="56" LastEditorDisplayName="" LastEditDate="2010-09-29T14:34:45.027" LastActivityDate="2010-09-29T14:34:45.027" CommentCount="2" />
  <row Id="1014" PostTypeId="2" ParentId="1012" CreationDate="2010-07-29T21:54:24.130" Score="7" ViewCount="" Body="&lt;p&gt;In general, the standard deviation of a continous uniform distribution is (max - min) / sqrt(12).&lt;/p&gt;&#xA;" OwnerUserId="614" LastActivityDate="2010-07-29T21:54:24.130" />
  <row Id="1015" PostTypeId="1" AcceptedAnswerId="3667" CreationDate="2010-07-29T22:03:55.887" Score="6" ViewCount="336" Body="&lt;p&gt;I am trying to calculate the reliability in an elicitation exercise by analysing some test-retest questions given to the experts. The experts elicited a series of probability distributions which were then compared with the true value (found at a later date) by computing the standardized quadratic scores. These scores are the values that I am using to calculate the reliability between the test-retest results. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which reliability method would be appropriate here?  I was looking mostly at Pearson's correlation and Chronbach's alpha (and got some negative values using both methods) but I am not sure this is the right approach.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE:&lt;/strong&gt;&#xA;Background information&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data were collected from a number of students who were asked to predict their own actual exam mark in four chosen modules by giving a probability distribution of the marks. One module was then repeated at a later date (hence the test-retest exercise).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once the exam was taken, and the real results were available, the standardized quadratic scores were computed. These scores are proper scoring rules used to compare assessed probability distributions with the observed data which might be known at a later stage.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The probability score &lt;em&gt;Q&lt;/em&gt; is defined as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://img717.imageshack.us/img717/9424/chart2j.png&quot; alt=&quot;Quadratic score&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where &lt;em&gt;k&lt;/em&gt; is the total number of elicited probabilities and &lt;em&gt;j&lt;/em&gt; is the true outcome.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is which reliability method would be more appropriate when it comes to assessing the reliability between the scores of the repeated modules? I calculated Pearson's correlation and Chronbach's alpha (and got some negative values using both methods) but there might be a better approach.&lt;/p&gt;&#xA;" OwnerUserId="108" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-18T15:21:53.397" LastActivityDate="2010-11-01T16:32:27.973" Title="Reliability in Elicitation Exercise" Tags="&lt;psychometrics&gt;&lt;reliability&gt;&lt;elicitation&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="1" />
  <row Id="1016" PostTypeId="1" AcceptedAnswerId="1026" CreationDate="2010-07-29T22:04:32.143" Score="4" ViewCount="683" Body="&lt;p&gt;I've got a linear regression model with the sample and variable observations and I want to know:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Whether a specific variable is significant enough to remain included in the model.&lt;/li&gt;&#xA;&lt;li&gt;Whether another variable (with observations) ought to be included in the model.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Which statistics can help me out? How can get them most efficiently?&lt;/p&gt;&#xA;" OwnerUserId="614" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-08-09T10:57:45.670" LastActivityDate="2010-09-16T23:06:05.980" Title="Is a variable significant in a linear regression model?" Tags="&lt;regression&gt;" AnswerCount="5" FavoriteCount="1" />
  <row Id="1018" PostTypeId="2" ParentId="973" CreationDate="2010-07-29T22:30:13.867" Score="2" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://archive.ics.uci.edu/ml/datasets/Arcene&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;Arcene&lt;/strong&gt;&lt;/a&gt;&lt;br&gt;&#xA;n=900&lt;br&gt;&#xA;p=10000 (3k is artificially added noise)&lt;br&gt;&#xA;k=2 (~balanced)&lt;br&gt;&#xA;From &lt;a href=&quot;http://www.nipsfsc.ecs.soton.ac.uk/papers/NIPS2003-Datasets.pdf&quot; rel=&quot;nofollow&quot;&gt;NIPS2003&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="88" LastEditorUserId="190" LastEditorDisplayName="" LastEditDate="2010-07-30T18:00:06.813" LastActivityDate="2010-07-30T18:00:06.813" CommunityOwnedDate="2010-07-29T22:30:13.867" />
  <row Id="1019" PostTypeId="2" ParentId="973" CreationDate="2010-07-29T22:32:44.880" Score="2" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://archive.ics.uci.edu/ml/datasets/Dexter&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;Dexter&lt;/strong&gt;&lt;/a&gt;&lt;br&gt;&#xA;n=2600&lt;br&gt;&#xA;p=20000 (10k+53 is artificial noise)&lt;br&gt;&#xA;k=2 (balanced)&lt;br&gt;&#xA;From &lt;a href=&quot;http://www.nipsfsc.ecs.soton.ac.uk/papers/NIPS2003-Datasets.pdf&quot; rel=&quot;nofollow&quot;&gt;NIPS2003&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="88" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-07-29T22:41:53.950" LastActivityDate="2010-07-29T22:41:53.950" CommunityOwnedDate="2010-07-29T22:32:44.880" CommentCount="3" />
  <row Id="1020" PostTypeId="2" ParentId="973" CreationDate="2010-07-29T22:35:28.497" Score="2" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://archive.ics.uci.edu/ml/datasets/Dorothea&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;Dorothea&lt;/strong&gt;&lt;/a&gt;&lt;br&gt;&#xA;n=1950&lt;br&gt;&#xA;p=100000 (0.1M, half is artificially added noise)&lt;br&gt;&#xA;k=2 (~10x unbalanced)&lt;br&gt;&#xA;From &lt;a href=&quot;http://www.nipsfsc.ecs.soton.ac.uk/papers/NIPS2003-Datasets.pdf&quot; rel=&quot;nofollow&quot;&gt;NIPS2003&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="88" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-07-29T22:41:33.450" LastActivityDate="2010-07-29T22:41:33.450" CommunityOwnedDate="2010-07-29T22:35:28.497" CommentCount="0" />
  <row Id="1021" PostTypeId="2" ParentId="973" CreationDate="2010-07-29T22:38:21.253" Score="2" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://archive.ics.uci.edu/ml/datasets/Gisette&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;Gisette&lt;/strong&gt;&lt;/a&gt;&lt;br&gt;&#xA;n=13500&lt;br&gt;&#xA;p=5000 (half is artificially added noise)&lt;br&gt;&#xA;k=2 (balanced)&lt;br&gt;&#xA;From &lt;a href=&quot;http://www.nipsfsc.ecs.soton.ac.uk/papers/NIPS2003-Datasets.pdf&quot; rel=&quot;nofollow&quot;&gt;NIPS2003&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-07-29T22:38:21.253" CommunityOwnedDate="2010-07-29T22:38:21.253" />
  <row Id="1023" PostTypeId="1" CreationDate="2010-07-29T23:04:08.707" Score="5" ViewCount="130" Body="&lt;p&gt;Introductory, advanced, and even obscure, please.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Mostly to test myself. I like to make sure I know what the heck I'm talking about :)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="74" LastActivityDate="2010-08-17T20:09:56.013" CommunityOwnedDate="2010-07-30T01:01:24.180" Title="Where can I find good statistics quizzes?" Tags="&lt;teaching&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="2" />
  <row Id="1024" PostTypeId="2" ParentId="1016" CreationDate="2010-07-29T23:25:12.603" Score="3" ViewCount="" Body="&lt;p&gt;For part 1, you're looking for the &lt;a href=&quot;http://en.wikipedia.org/wiki/F-test#Regression_problems&quot; rel=&quot;nofollow&quot;&gt;F-test&lt;/a&gt;. Calculate your residual sum of squares from each model fit and calculate an F-statistic, which you can use to find p-values from either an F-distribution or some other null distribution that you generate yourself.&lt;/p&gt;&#xA;" OwnerUserId="378" LastActivityDate="2010-07-29T23:25:12.603" />
  <row Id="1025" PostTypeId="2" ParentId="980" CreationDate="2010-07-29T23:45:46.407" Score="3" ViewCount="" Body="&lt;p&gt;Calculating averages leads to a different dataset than simply reducing the number of data points.&#xA;If one heartbeat per minute is much faster than the other heart beats you will lose the signal through your smoothing process.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you summary 125-125-0-125-125 as 100 than the story that the data tells is different through your smoothing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sometimes the heart even skips beats and I believe that's an event that interesting for however wants to look at plotted heart rate data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would therefore propose that you calculate the distance between two points with a formula like &lt;code&gt;d=sqrt((time1-time2)^2 + (bpm1-bpm2))&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You set a minimum distance in your program.&#xA;Then you iterate through your data and after every point you delete all following points for which d is smaller than your minimum distance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As the unit of time and bpm isn't the same you might want to think about how you can find a way to scale the units meaningfully. To do this task right you should speak to the doctors who in the end have to interpret your graphs and ask them what information they consider to be essential.&lt;/p&gt;&#xA;" OwnerUserId="237" LastEditorUserId="237" LastEditorDisplayName="" LastEditDate="2010-07-31T02:09:30.503" LastActivityDate="2010-07-31T02:09:30.503" CommentCount="1" />
  <row Id="1026" PostTypeId="2" ParentId="1016" CreationDate="2010-07-30T00:00:15.037" Score="15" ViewCount="" Body="&lt;p&gt;Statistical significance is not usually a good basis for determining whether a variable should be included in a model. Statistical tests were designed to test hypotheses, not select variables. I know a lot of textbooks discuss variable selection using statistical tests, but this is generally a bad approach. See Harrell's book &lt;em&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0387952322&quot; rel=&quot;nofollow&quot;&gt;Regression Modelling Strategies&lt;/a&gt;&lt;/em&gt; for some of the reasons why. These days, variable selection based on the AIC (or something similar) is usually preferred.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-07-30T00:00:15.037" CommentCount="6" />
  <row Id="1027" PostTypeId="2" ParentId="1015" CreationDate="2010-07-30T00:44:09.457" Score="0" ViewCount="" Body="&lt;p&gt;You would use Cronbach alpha if you do not know the true value but if you do know the true value then it seems a bit pointless to use Cronbach alpha. The use of Pearson correlation also seems a bit odd as you do not actually have a paired set of values. I would suggest using something like the &lt;a href=&quot;http://en.wikipedia.org/wiki/Mean_squared_error&quot; rel=&quot;nofollow&quot;&gt;Mean Squared Error (MSE)&lt;/a&gt;. Suppose that you have N experts and that the  expected estimate for the expert i is given by $\hat{\theta_i}$ and your true value is $\theta$. Then,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$MSE = \frac{\sum_i (\hat{\theta_i} - \theta)^2}{N}$&lt;/p&gt;&#xA;" LastActivityDate="2010-07-30T00:44:09.457" CommentCount="3" />
  <row Id="1028" PostTypeId="1" CreationDate="2010-07-30T00:55:20.603" Score="5" ViewCount="660" Body="&lt;p&gt;I am comparing two distributions with KL divergence which returns me a non-standardized number that, according to what I read about this measure, is the amount of information that is required to transform one hypothesis into the other. I have two questions:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;a) Is there a way to quantify a KL divergence so that it has a more meaningful interpretation, e.g. like an effect size or a R^2? Any form of standardization?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;b) In R, when using KLdiv (flexmix package) one can set the 'esp' value (standard esp=1e-4) that sets all points smaller than esp to some standard in order to provide numerical stability. I have been playing with different esp values and, for my data set, I am getting an increasingly larger KL divergence the smaller a number I pick. What is going on? I would expect that the smaller the esp, the more reliable the results should be since they let more 'real values' become part of the statistic. No? I have to change the esp since it otherwise does not calculate the statistic but simply shows up as NA in the result table...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance,&#xA;Ampleforth &lt;/p&gt;&#xA;" OwnerUserId="608" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-29T00:29:23.193" LastActivityDate="2011-04-29T00:29:23.193" Title="Questions about KL divergence?" Tags="&lt;distributions&gt;&lt;kullback-leibler&gt;" AnswerCount="3" FavoriteCount="4" />
  <row Id="1029" PostTypeId="2" ParentId="1023" CreationDate="2010-07-30T02:42:38.523" Score="1" ViewCount="" Body="&lt;p&gt;I wrote a post compiling links of  Practice Questions for Statistics in Psychology (Undergraduate Level).&#xA;&lt;a href=&quot;http://jeromyanglim.blogspot.com/2009/12/practice-questions-for-statistics-in.html&quot; rel=&quot;nofollow&quot;&gt;http://jeromyanglim.blogspot.com/2009/12/practice-questions-for-statistics-in.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The questions would fall into the introductory category.&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-07-30T02:42:38.523" CommunityOwnedDate="2010-07-30T02:42:38.523" />
  <row Id="1030" PostTypeId="2" ParentId="1028" CreationDate="2010-07-30T03:53:08.320" Score="6" ViewCount="" Body="&lt;p&gt;The KL(p,q) divergence between distributions p(.) and q(.) has an intuitive information theoretic interpretation which you may find useful. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose we observe data x generated by some probability distribution p(.). A lower bound on the average codelength in bits required to state the data generated by p(.) is given by the entropy of p(.). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, since we don't know p(.) we choose another distribution, say, q(.) to encode (or describe, state) the data. The average codelength of data generated by p(.) and encoded using q(.) will necessarily be longer than if the true distribution p(.) was used for the coding. The KL divergence tells us about the inefficiencies of this alternative code. In other words, the KL divergence between p(.) and q(.) is the average number of &lt;strong&gt;extra&lt;/strong&gt; bits required to encode data generated by p(.) using coding distribution q(.). The KL divergence is non-negative and equal to zero iff the actual data generating distribution is used to encode the data.&lt;/p&gt;&#xA;" OwnerUserId="530" LastActivityDate="2010-07-30T03:53:08.320" />
  <row Id="1031" PostTypeId="2" ParentId="1028" CreationDate="2010-07-30T05:29:11.857" Score="5" ViewCount="" Body="&lt;p&gt;KL has a deep meaning when you visualize a set of &lt;strong&gt;dentities as a manifold&lt;/strong&gt; within the fisher metric tensor, it gives the geodesic distance between two &quot;close&quot; distributions. Formally: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ds^2=2KL(p(x, \theta ),p(x,\theta + d \theta))$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The following lines are here to explain with details what is meant by this las mathematical formulae. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Definition of the Fisher metric.&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider a parametrized family of probability distributions $D=(f(x, \theta ))$ (given by densities in $R^n$), where $x$ is a random variable and theta is a parameter in $R^p$. You may all knnow that the fisher information matrix $F=(F_{ij})$ is &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$F_{ij}=E[d(\log f(x,\theta))/d \theta_i d(\log f(x,\theta))/d \theta_j]$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With this notation $D$ is a riemannian manifold and $F(\theta)$ is a Riemannian metric tensor. (The interest of this metric is given by cramer Rao lower bound theorem)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may say ... OK mathematical abstraction but where is KL ? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is not mathematical abstraction, if $p=1$ you can really imagine your parametrized density as a curve (instead of a subset of a space of infinite dimension) and $F_{11}$ is connected to the curvature of that curve...&#xA;(see the seminal paper of Bradley Efron &lt;a href=&quot;http://projecteuclid.org/DPubS?service=UI&amp;amp;version=1.0&amp;amp;verb=Display&amp;amp;handle=euclid.aos/1176343282&quot; rel=&quot;nofollow&quot;&gt;http://projecteuclid.org/DPubS?service=UI&amp;amp;version=1.0&amp;amp;verb=Display&amp;amp;handle=euclid.aos/1176343282&lt;/a&gt;) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The geometric answer to part of point a/ in your question :&lt;/strong&gt; the squared distance $ds^2$ between two (close) distributions $p(x,\theta)$ and $p(x,\theta+d \theta)$ on the manifold (think of geodesic distance on earth of two points that are close, it is related to the curvature of the earth) is given by the quadratic form:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ds^2= \sum F_{ij} d \theta^i d \theta^j$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and it is known to be twice the Kullback Leibler Divergence:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ds^2=2KL(p(x, \theta ),p(x,\theta + d \theta))$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to learn more about that I suggest reading the paper from Amari&#xA;&lt;a href=&quot;http://projecteuclid.org/DPubS?service=UI&amp;amp;version=1.0&amp;amp;verb=Display&amp;amp;handle=euclid.aos/1176345779&quot; rel=&quot;nofollow&quot;&gt;http://projecteuclid.org/DPubS?service=UI&amp;amp;version=1.0&amp;amp;verb=Display&amp;amp;handle=euclid.aos/1176345779&lt;/a&gt;&#xA;(I think there is also a book from Amari about riemannian geometry in statistic but I don't remember the name) &lt;/p&gt;&#xA;" OwnerUserId="223" LastEditorUserId="223" LastEditorDisplayName="" LastEditDate="2010-08-04T17:27:55.000" LastActivityDate="2010-08-04T17:27:55.000" CommentCount="4" />
  <row Id="1032" PostTypeId="2" ParentId="414" CreationDate="2010-07-30T08:04:50.453" Score="1" ViewCount="" Body="&lt;p&gt;You will find many applications of Mathematical Statistics in 'Mathematical Statistics and Data Analysis' by John A. Rice. The 'Application Index' lists all applications discussed in the text. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Javed&lt;/p&gt;&#xA;" OwnerUserId="531" LastActivityDate="2010-07-30T08:04:50.453" CommunityOwnedDate="2010-07-30T08:04:50.453" />
  <row Id="1033" PostTypeId="2" ParentId="887" CreationDate="2010-07-30T09:49:02.353" Score="0" ViewCount="" Body="&lt;p&gt;&quot;how is stdev(S) related to the standard deviation of the entire population?&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't know if the &quot;Confidence Interval&quot; concept might be what you are looking for? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Stdev(S) is an Estimate of the standard deviation of the entire population. To see how good an estimate, confidence intervals could be computed, and these would be dependent on the sample size.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See for e.g., Simulation and the Monte Carlo Method, Rubinstein &amp;amp; Kroese.&lt;/p&gt;&#xA;" LastActivityDate="2010-07-30T09:49:02.353" />
  <row Id="1035" PostTypeId="2" ParentId="913" CreationDate="2010-07-30T09:54:10.323" Score="4" ViewCount="" Body="&lt;p&gt;Another solution to your problem (without transforming variables) is regression with error distribution other then &lt;strong&gt;Gaussian&lt;/strong&gt; for example &lt;strong&gt;Gamma&lt;/strong&gt; or &lt;strong&gt;skewed t-Student&lt;/strong&gt;.&#xA;Gamma is in GLM family, so there is a lot of software to fit model with this error distribution.&lt;/p&gt;&#xA;" OwnerUserId="419" LastActivityDate="2010-07-30T09:54:10.323" />
  <row Id="1036" PostTypeId="2" ParentId="125" CreationDate="2010-07-30T10:03:37.523" Score="4" ViewCount="" Body="&lt;p&gt;Sivia and Skilling, Data analysis: a Bayesian tutorial (2ed) 2006 246p 0198568320&#xA;&lt;a href=&quot;http://books.google.com/books?id=zN-yliq6eZ4C&amp;amp;dq=isbn:0198568320&amp;amp;source=gbs_navlinks_s&quot; rel=&quot;nofollow&quot;&gt;books.goo&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Statistics lectures have been a source&#xA;  of much bewilderment and frustration&#xA;  for generations of students. This book&#xA;  attempts to remedy the situation by&#xA;  expounding a logical and unified&#xA;  approach to the whole subject of data&#xA;  analysis. This text is intended as a&#xA;  tutorial guide for senior&#xA;  undergraduates and research students&#xA;  in science and engineering ...&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I don't know the other recommendations though.&lt;/p&gt;&#xA;" OwnerUserId="557" LastActivityDate="2010-07-30T10:03:37.523" CommunityOwnedDate="2010-07-30T10:03:37.523" CommentCount="1" />
  <row Id="1037" PostTypeId="2" ParentId="125" CreationDate="2010-07-30T10:14:14.863" Score="-1" ViewCount="" Body="&lt;p&gt;This might be worth a look for people working on certain types of problems:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Millán, E. et al., 2010. Bayesian Networks for Student Model Engineering. Computers &amp;amp; Education. Available at: &lt;a href=&quot;http://dx.doi.org/10.1016/j.compedu.2010.07.010&quot; rel=&quot;nofollow&quot;&gt;http://dx.doi.org/10.1016/j.compedu.2010.07.010&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Abstract: &#xA;Bayesian networks are a graphical modeling tool that has been proven very powerful in a variety of application contexts. The purpose of this paper is to provide education practitioners with the background and examples needed to understand Bayesian netwrosk and use them to design and implement student models. The student model is the key component of any adaptive tutoring system, as it stores all the information about the student (for example, knowledge, interest, learning styles, etc.) so the tutoring system can use this information to provide personalized instruction. Basic and advanced concepts and techniques are introduced and applied in the context of typical student modeling problems. A repertoire of models of varying complexity is discussed. To illustrate the proposed methodology a Bayesian Student Model for the Simplex algorithm is implemented. &lt;/p&gt;&#xA;" OwnerUserId="210" LastEditorUserId="509" LastEditorDisplayName="" LastEditDate="2010-08-05T20:30:27.337" LastActivityDate="2010-08-05T20:30:27.337" CommunityOwnedDate="2010-07-30T10:14:14.863" CommentCount="1" />
  <row Id="1038" PostTypeId="2" ParentId="652" CreationDate="2010-07-30T12:11:18.743" Score="1" ViewCount="" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;em&gt;&lt;strong&gt;Wilcox, Rand R.&lt;/em&gt;&lt;/strong&gt; - &lt;em&gt;BASIC STATISTICS - Understanding&#xA;Conventional Methods and Modern&#xA;Insights&lt;/em&gt;, Oxford University Press,&#xA;2009&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;em&gt;&lt;strong&gt;Hoff, Peter D.&lt;/em&gt;&lt;/strong&gt; - &lt;em&gt;A First Course in&#xA;Bayesian Statistical Methods&lt;/em&gt;,&#xA;Springer, 2009&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;em&gt;&lt;strong&gt;Dalgaard, Peter&lt;/em&gt;&lt;/strong&gt; - &lt;em&gt;Introductory&#xA;Statistics with R, Second Edition&lt;/em&gt;, Springer, 2008&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;also take a glance at &lt;a href=&quot;http://stackoverflow.com/questions/192369/books-for-learning-the-r-language/2270793#2270793&quot;&gt;this link&lt;/a&gt;, though it's R-specific, there are plenty of books that can guide you through basic statistical techniques.&lt;/p&gt;&#xA;" OwnerUserId="1356" LastActivityDate="2010-07-30T12:11:18.743" CommunityOwnedDate="2010-07-30T12:11:18.743" />
  <row Id="1039" PostTypeId="2" ParentId="125" CreationDate="2010-07-30T12:15:45.737" Score="3" ViewCount="" Body="&lt;p&gt;If you're looking for an elementary text, i.e. one that doesn't have a calculus prerequisite, there's Don Berry's &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0534234720&quot; rel=&quot;nofollow&quot;&gt;Statistics: A Bayesian Perspective&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="319" LastActivityDate="2010-07-30T12:15:45.737" CommunityOwnedDate="2010-07-30T12:15:45.737" />
  <row Id="1040" PostTypeId="1" AcceptedAnswerId="1041" CreationDate="2010-07-30T14:24:17.350" Score="1" ViewCount="363" Body="&lt;p&gt;Consider the following model &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Y_i = f(X_i) + e_i$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;from which we observe n iid data points $\left( X_i, Y_i \right)_{i=1}^n$. Suppose that $X_i \in \mathbb{R}^d$ is a $d$ dimensional feature vector. And suppose that a ordinary least squares estimate is fit to data, that is,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\hat \beta = {\rm arg} \min_{\beta \in \mathbb{R}^d} \sum_i (Y_i - \sum_j X_{ij} \beta_j)^2$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since a wrong model is estimated, what is the interpretation for the confidence interval around estimated coefficients? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;More generally, does it make sense to estimate confidence intervals around parameters in a misspecified model? And what does the confidence interval tell us in such a case?&lt;/p&gt;&#xA;" OwnerUserId="168" LastEditorUserId="168" LastEditorDisplayName="" LastEditDate="2010-07-30T15:15:21.327" LastActivityDate="2011-04-16T10:55:20.060" Title="What is the interpretation/meaning of confidence intervals in misspecified models?" Tags="&lt;modeling&gt;&lt;estimation&gt;&lt;model-selection&gt;&lt;confidence-interval&gt;" AnswerCount="3" CommentCount="3" FavoriteCount="1" />
  <row Id="1041" PostTypeId="2" ParentId="1040" CreationDate="2010-07-30T14:38:39.530" Score="2" ViewCount="" Body="&lt;p&gt;The confidence interval that you obtain is conditional on the model being correct and the interpretation is also conditional on the model being the correct one. If you &lt;strong&gt;know&lt;/strong&gt; that the model is incorrect then obviously you would not use it to compute the confidence interval. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In reality, you do not know the &lt;strong&gt;true&lt;/strong&gt; model and so you have no way to tell if you have a misspecified model (although you do have ways to assess misspecification, e.g., examine if residuals are normally distributed, diagnostic plots of fitted vs observed values etc). So, to my mind, the real question is if the model is misspecified, to what extent can you rely on confidence intervals as a way to assess where the true parameter is. I suspect that the answer is specific to the degree of misspecification that is coming from f(x) i.e., the degree to which f(x) departs from the assumptions of OLS.&lt;/p&gt;&#xA;" LastActivityDate="2010-07-30T14:38:39.530" CommentCount="1" />
  <row Id="1043" PostTypeId="2" ParentId="977" CreationDate="2010-07-30T15:37:06.190" Score="1" ViewCount="" Body="&lt;p&gt;OK, so your data is very expensive to get.  If you have some indication of the shape of the data then perhaps a bootstrap / bayesian / optimization (sticking in keywords :)) approach would work best.  See the optim command in R as an example.  You would need to know some things though.  For example, could we assume something like normality?  If so then fitting a small number of data points to the normal distribution will likely give you a much better estimate of your parameters than simple mean and sdev values.&lt;/p&gt;&#xA;" OwnerUserId="601" LastActivityDate="2010-07-30T15:37:06.190" />
  <row Id="1044" PostTypeId="2" ParentId="73" CreationDate="2010-07-30T15:43:28.517" Score="3" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Sweave&lt;/strong&gt; lets you embed R code in a LaTeX document.  The results of executing the code, and optionally the source code, become part of the final document.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So instead of, for example, pasting an image produced by R into a LaTeX file, you can paste the R &lt;em&gt;code&lt;/em&gt; into the file and keep everything in one place.&lt;/p&gt;&#xA;" OwnerUserId="319" LastActivityDate="2010-07-30T15:43:28.517" CommunityOwnedDate="2010-07-30T15:43:28.517" />
  <row Id="1045" PostTypeId="1" AcceptedAnswerId="1051" CreationDate="2010-07-30T16:06:54.980" Score="2" ViewCount="163" Body="&lt;p&gt;The wiki article on &lt;a href=&quot;http://en.wikipedia.org/wiki/Credible_interval&quot; rel=&quot;nofollow&quot;&gt;credible intervals&lt;/a&gt; has the following statement:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;credible intervals and confidence intervals treat &lt;a href=&quot;http://en.wikipedia.org/wiki/Nuisance_parameter&quot; rel=&quot;nofollow&quot;&gt;nuisance parameters&lt;/a&gt; in radically different ways.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;What is the &lt;strong&gt;radical&lt;/strong&gt; difference that the wiki talks about?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Credible intervals are based on the posterior distribution of the parameter and confidence interval is based on the maximum likelihood associated with the data generating process. It seems to me that how credible and confidence intervals are computed is &lt;strong&gt;not dependent&lt;/strong&gt; on whether the parameters are nuisance or not. So, I am a bit puzzled by this statement.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: I am aware of alternative approaches to dealing with nuisance parameters under frequentist inference but I think they are less common than standard maximum likelihood. (See this question on the difference between &lt;a href=&quot;http://stats.stackexchange.com/questions/622/what-is-the-difference-between-a-partial-likelihood-profile-likelihood-and-margi&quot;&gt;partial, profile and marginal likelihoods&lt;/a&gt;.)&lt;/p&gt;&#xA;" LastActivityDate="2010-07-30T23:53:35.657" Title="Is there a radical difference in how bayesian and frequentist approaches treat nuisance parameters?" Tags="&lt;confidence-interval&gt;&lt;credible-interval&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="2" />
  <row Id="1046" PostTypeId="2" ParentId="825" CreationDate="2010-07-30T16:58:42.397" Score="5" ViewCount="" Body="&lt;p&gt;If you are using GNU/Linux previous answers by Shane and Dirk are great.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you need a solution for windows, there is one in this post:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.r-statistics.com/2010/04/parallel-multicore-processing-with-r-on-windows/&quot; rel=&quot;nofollow&quot;&gt;Parallel Multicore Processing with R (on Windows)&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Although the package is not yet on CRAN. it can be downloaded from that link.&lt;/p&gt;&#xA;" OwnerUserId="253" LastActivityDate="2010-07-30T16:58:42.397" />
  <row Id="1047" PostTypeId="1" AcceptedAnswerId="1048" CreationDate="2010-07-30T17:00:57.573" Score="3" ViewCount="499" Body="&lt;p&gt;I'm comparing a sample and checking whether it distributes as some, discrete, distribution.  However, I'm not enterily sure that Kolmogorov-Smirnov applies. &lt;a href=&quot;http://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test&quot; rel=&quot;nofollow&quot;&gt;Wikipedia&lt;/a&gt; seems to imply it does not. If it does not, how can I test the sample's distribution?&lt;/p&gt;&#xA;" OwnerUserId="614" LastActivityDate="2010-07-30T17:10:09.663" Title="Is Kolmogorov-Smirnov test valid with discrete distributions?" Tags="&lt;hypothesis-testing&gt;" AnswerCount="1" />
  <row Id="1048" PostTypeId="2" ParentId="1047" CreationDate="2010-07-30T17:10:09.663" Score="3" ViewCount="" Body="&lt;p&gt;It does not apply to discrete distributions. See &lt;a href=&quot;http://www.itl.nist.gov/div898/handbook/eda/section3/eda35g.htm&quot; rel=&quot;nofollow&quot;&gt;http://www.itl.nist.gov/div898/handbook/eda/section3/eda35g.htm&lt;/a&gt; for example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any reason you can't use a chi-square goodness of fit test?&#xA;see &lt;a href=&quot;http://www.itl.nist.gov/div898/handbook/eda/section3/eda35f.htm&quot; rel=&quot;nofollow&quot;&gt;http://www.itl.nist.gov/div898/handbook/eda/section3/eda35f.htm&lt;/a&gt; for more info.&lt;/p&gt;&#xA;" OwnerUserId="247" LastActivityDate="2010-07-30T17:10:09.663" />
  <row Id="1049" PostTypeId="2" ParentId="138" CreationDate="2010-07-30T17:10:24.607" Score="5" ViewCount="" Body="&lt;p&gt;One resource is 'Some hints for the R beginner' at&#xA;&lt;a href=&quot;http://www.burns-stat.com/pages/Tutor/hints_R_begin.html&quot; rel=&quot;nofollow&quot;&gt;http://www.burns-stat.com/pages/Tutor/hints_R_begin.html&lt;/a&gt;&lt;/p&gt;&#xA;" LastActivityDate="2010-07-30T17:10:24.607" CommunityOwnedDate="2010-08-01T18:56:25.603" CommentCount="1" />
  <row Id="1050" PostTypeId="2" ParentId="170" CreationDate="2010-07-30T20:00:45.253" Score="3" ViewCount="" Body="&lt;p&gt;Here's a fresh one: &lt;a href=&quot;http://www.lulu.com/product/file-download/introduction-to-probability-and-statistics-using-r/12037733&quot; rel=&quot;nofollow&quot;&gt;Introduction to Probability and Statistics Using R &lt;/a&gt;. It's R-specific, though, but it's a great one. I haven't read it yet, but it seems fine so far...&lt;/p&gt;&#xA;" OwnerUserId="1356" LastActivityDate="2010-07-30T20:00:45.253" CommunityOwnedDate="2010-07-30T20:00:45.253" />
  <row Id="1051" PostTypeId="2" ParentId="1045" CreationDate="2010-07-30T21:15:43.290" Score="3" ViewCount="" Body="&lt;p&gt;The fundamental difference is that in maximum likelihood based methods we can't integrate the nuisance parameters out (because the likelihood function is not a PDF and doesn't obey probability laws).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In maximum likelihood methods, the ideal way to deal with nuisance parameters is through marginal/conditional likelihoods, but these are defined differently from the question you linked.  (There is a notion of an &lt;em&gt;integrated&lt;/em&gt; (marginal/conditional) likelihood function as in the linked question, but this is not strictly the marginal likelihood function.)  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say you have a parameter of interest, $\theta$, a nuisance parameter, $\lambda$.  Suppose a transformation of your data $X$ to $(Y, Z)$ exists such that either $Y$ or $Y|Z$ depends only on $\theta$.  If $Y$ depends on $\theta$, then the joint density can be written &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$f(Y, Z; \theta, \lambda) = f_{Y}(Y; \theta) f_{Z|Y}(Z|Y; \theta, \lambda)$.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the latter case, we have &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$f(Y, Z; \theta, \lambda) = f_{Y|Z}(Y|Z; \theta) f_{Z}(Z; \theta, \lambda)$.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In either case, the factor depending on $\theta$ alone is of interest.  In the former, it's the basis for the definition of the marginal likelihood and in the latter, the conditional likelihood.  The important point here is to isolate a component that depends on $\theta$ alone.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we can't find such a transformation, we look at other likelihood functions to eliminate the nuisance.  We usually start with a profile likelihood.  To eliminate bias in the MLE, we try to obtain approximations for marginal or conditional likelihoods, usually through a &quot;modified profile likelihood&quot; function (yet another likelihood function!).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are many details, but the short story is that the likelihood methods treat nuisance parameters quite differently than Bayesian methods.  In particular, the estimated likelihoods don't account for uncertainty in the nuisance.  Bayesian methods do account for it through the specification of a prior.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are arguments in favor of an integrated likelihood function and lead to something resembling the Bayesian framework.  If you're interested, I can dig up some references.&lt;/p&gt;&#xA;" OwnerUserId="251" LastEditorUserId="251" LastEditorDisplayName="" LastEditDate="2010-07-30T23:53:35.657" LastActivityDate="2010-07-30T23:53:35.657" CommentCount="6" />
  <row Id="1052" PostTypeId="1" CreationDate="2010-07-30T22:38:06.510" Score="3" ViewCount="407" Body="&lt;p&gt;my question particularly applies to network reconstruction&lt;/p&gt;&#xA;" LastActivityDate="2010-07-31T03:04:03.400" Title="What is the major difference between correlation and mutual information?" Tags="&lt;correlation&gt;&lt;mutual-information&gt;" AnswerCount="2" FavoriteCount="0" />
  <row Id="1053" PostTypeId="1" CreationDate="2010-07-31T00:51:52.653" Score="10" ViewCount="276" Body="&lt;p&gt;I am looking for a good book/tutorial to learn the survival analysis. I am also interested in references on doing survival analysis in R&lt;/p&gt;&#xA;" OwnerUserId="172" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-10-21T13:39:03.730" LastActivityDate="2010-10-22T16:15:02.407" CommunityOwnedDate="2010-09-16T12:35:43.497" Title="References for Survival Analysis" Tags="&lt;books&gt;&lt;survival&gt;&lt;big-list&gt;" AnswerCount="8" CommentCount="1" FavoriteCount="4" />
  <row Id="1054" PostTypeId="1" CreationDate="2010-07-31T00:59:11.343" Score="4" ViewCount="208" Body="&lt;p&gt;What is the equivalent command in R for the stcox command in STATA? &lt;/p&gt;&#xA;" OwnerUserId="172" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-16T12:34:34.087" LastActivityDate="2010-09-19T07:23:51.393" Title="R command for stcox in STATA" Tags="&lt;r&gt;&lt;survival&gt;&lt;stata&gt;" AnswerCount="1" />
  <row Id="1055" PostTypeId="2" ParentId="1052" CreationDate="2010-07-31T01:08:08.763" Score="5" ViewCount="" Body="&lt;p&gt;Correlation measures the &lt;em&gt;linear&lt;/em&gt; relationship (Pearson's correlation) or &lt;em&gt;monotonic&lt;/em&gt; relationship (Spearman's correlation) between two variables, X and Y. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Mutual information is more general and measures the reduction of uncertainty in Y after observing X. It is the KL distance between the joint density and the product of the individual densities. So MI can measure non-monotonic relationships and other more complicated relationships.&lt;/p&gt;&#xA;" OwnerUserId="159" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-07-31T03:04:03.400" LastActivityDate="2010-07-31T03:04:03.400" CommentCount="2" />
  <row Id="1056" PostTypeId="2" ParentId="1053" CreationDate="2010-07-31T01:41:53.177" Score="3" ViewCount="" Body="&lt;p&gt;I like:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.powells.com/biblio/72-9780387239187-0&quot; rel=&quot;nofollow&quot;&gt;Survival Analysis: Techniques for Censored and Truncated Data&lt;/a&gt; (Klein &amp;amp; Moeschberger)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.powells.com/biblio/72-9780387239187-0&quot; rel=&quot;nofollow&quot;&gt;Modeling Survival Data: Extending the Cox Model&lt;/a&gt; (Therneau)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The first does a good job of straddling theory and model building issues.  It's mostly focused on semi-parametric techniques, but there is reasonable coverage of parametric methods.  It doesn't really provide any R or other code examples, if that's what you're after.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second is heavy with modeling on the Cox PH side (as the title might indicate).  It's by the author of the &lt;a href=&quot;http://cran.r-project.org/web/views/Survival.html&quot; rel=&quot;nofollow&quot;&gt;survival&lt;/a&gt; package in R and there are plenty of R examples and mini-case studies.  I think both books complement each other, but I'd recommend the first for getting started.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A quick way to get started in R is David Diez's &lt;a href=&quot;http://www.statgrad.com/teac/surv/R_survival.pdf&quot; rel=&quot;nofollow&quot;&gt;guide&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-07-31T01:41:53.177" CommunityOwnedDate="2010-09-16T12:35:43.497" />
  <row Id="1057" PostTypeId="2" ParentId="1052" CreationDate="2010-07-31T02:05:44.990" Score="1" ViewCount="" Body="&lt;p&gt;To add to Rob's answer ... with respect to reverse engineering a network, MI may be preferred over correlation when you want to extract causal rather than associative links in your network.  Correlation networks are purely associative.  But for MI, you need more data and computing power.&lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-07-31T02:05:44.990" />
  <row Id="1058" PostTypeId="2" ParentId="1054" CreationDate="2010-07-31T02:08:06.560" Score="7" ViewCount="" Body="&lt;p&gt;In package &lt;a href=&quot;http://cran.r-project.org/web/packages/survival/index.html&quot; rel=&quot;nofollow&quot;&gt;survival&lt;/a&gt;, it's &lt;code&gt;coxph&lt;/code&gt;.  John Fox has a nice introduction to using coxph in R:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://cran.r-project.org/doc/contrib/Fox-Companion/appendix-cox-regression.pdf&quot; rel=&quot;nofollow&quot;&gt;Cox Proportional-Hazards Regression for Survival Data&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="251" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-19T07:23:51.393" LastActivityDate="2010-09-19T07:23:51.393" />
  <row Id="1059" PostTypeId="2" ParentId="726" CreationDate="2010-07-31T05:49:03.187" Score="5" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;&quot;Million to one chances crop up nine times out of ten.&quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;-Terry Pratchett&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-07-31T05:49:03.187" CommunityOwnedDate="2010-07-31T05:49:03.187" />
  <row Id="1060" PostTypeId="1" AcceptedAnswerId="1061" CreationDate="2010-07-31T09:39:47.323" Score="4" ViewCount="324" Body="&lt;p&gt;I'll use an example so that you can reproduce the results &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# mortality &#xA;mort = ts(scan(&quot;http://www.stat.pitt.edu/stoffer/tsa2/data/cmort.dat&quot;),start=1970, frequency=52)&#xA;&#xA;# temperature&#xA;temp = ts(scan(&quot;http://www.stat.pitt.edu/stoffer/tsa2/data/temp.dat&quot;), start=1970, frequency=52)&#xA;&#xA;#pollutant particulates&#xA;part = ts(scan(&quot;http://www.stat.pitt.edu/stoffer/tsa2/data/part.dat&quot;), start=1970, frequency=52)&#xA;&#xA;temp = temp-mean(temp)&#xA;temp2 = temp^2&#xA;trend = time(mort)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, fit a model for mortality data&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;fit = lm(mort ~ trend + temp + temp2 + part, na.action=NULL)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What I want now is to reproduce the result of the AIC command &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;AIC(fit)&#xA;[1] 3332.282&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;According to R's help file for AIC, AIC = -2 * log.likelihood + 2 * npar.&#xA;If I'm correct I think that log.likelihood is given using the following formula:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;n = length(mort)&#xA;RSS = anova(fit)[length(anova(fit)[,2]),2] # there must be better ways to get this, anyway&#xA;(log.likelihood &amp;lt;- -n/2*(log(2*pi)+log(RSS/n)+1))&#xA;&#xA; [1] -1660.135&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is approximately equal to&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;logLik(fit)&#xA;'log Lik.' -1660.141 (df=6)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As far as I can tell, the number of parameters in the model are 5 (how can I get this number programmatically ??). So AIC should be given by:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;-2 * log.likelihood + 2 * 5&#xA;[1] 3330.271&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Ooops, it seems like I should have used 6 instead of 5 as the number of parameters. What is wrong with those calculations? &lt;/p&gt;&#xA;" OwnerUserId="339" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-29T05:03:59.523" LastActivityDate="2011-04-29T05:03:59.523" Title="Why does AIC formula in R appear to use one extra parameter than expected?" Tags="&lt;r&gt;&lt;time-series&gt;&lt;modeling&gt;&lt;aic&gt;" AnswerCount="1" FavoriteCount="2" />
  <row Id="1061" PostTypeId="2" ParentId="1060" CreationDate="2010-07-31T10:04:04.443" Score="10" ViewCount="" Body="&lt;pre&gt;&lt;code&gt;&amp;gt; -2*logLik(fit)+2*(length(fit$coef)+1)&#xA;[1] 3332.282&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(you forgot; you have 6 parameter because $\sigma_{\epsilon}$ also has to be estimated!&amp;nbsp;&lt;/p&gt;&#xA;" OwnerUserId="603" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-29T01:10:38.227" LastActivityDate="2011-04-29T01:10:38.227" CommentCount="4" />
  <row Id="1062" PostTypeId="1" CreationDate="2010-07-31T14:19:13.850" Score="2" ViewCount="172" Body="&lt;p&gt;In general inference, why orthogonal parameters are useful, and why is it worth trying to find a new parametrization that makes the parameters orthogonal ? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have seen some textbook examples, not so many, and would be interested in more concrete examples and/or motivation.&lt;/p&gt;&#xA;" OwnerUserId="368" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-08-07T17:48:25.920" LastActivityDate="2010-10-12T16:05:37.587" Title="Orthogonal parametrization" Tags="&lt;multivariable&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="1" />
  <row Id="1063" PostTypeId="1" AcceptedAnswerId="1065" CreationDate="2010-07-31T14:29:15.543" Score="0" ViewCount="657" Body="&lt;p&gt;My stats has been self taught but a lot of material I read point to a dataset having mean 0 and standard deviation of 1&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If that is the case then:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Why is mean 0 and SD 1 a nice property to have&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Why does a random variable drawn from this sample equal 0.5?  The chance of drawing 0.001 is the same as 0.5 so this should be flat distribution...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) When people talk about Z Scores what do they actually mean here   &lt;/p&gt;&#xA;" OwnerUserId="353" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-08-07T17:48:15.717" LastActivityDate="2010-08-07T17:48:15.717" CommunityOwnedDate="2010-07-31T14:29:30.753" Title="Why Are Mean 0 And Standard Deviation 1 Distributions Always Used" Tags="&lt;probability&gt;" AnswerCount="2" />
  <row Id="1064" PostTypeId="2" ParentId="726" CreationDate="2010-07-31T15:36:12.410" Score="15" ViewCount="" Body="&lt;p&gt;A nice one I came about:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;I think it is much more interesting to&#xA;  live with uncertainty than to live with&#xA;  answers that might be wrong.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;By Richard Feynman (&lt;a href=&quot;http://www.youtube.com/watch?v=zeCHiUe1et0&amp;amp;feature=player_embedded#!&quot; rel=&quot;nofollow&quot;&gt;link&lt;/a&gt;)&lt;/p&gt;&#xA;" OwnerUserId="253" LastEditorUserId="74" LastEditorDisplayName="" LastEditDate="2010-08-17T17:33:06.267" LastActivityDate="2010-08-17T17:33:06.267" CommunityOwnedDate="2010-07-31T15:36:12.410" CommentCount="1" />
  <row Id="1065" PostTypeId="2" ParentId="1063" CreationDate="2010-07-31T15:46:05.483" Score="4" ViewCount="" Body="&lt;p&gt;1)  At the beginning the most useful answer is probably that mean of 0 and sd of 1 are mathematically convenient.  If you can work out the probabilities for a distribution with a mean of 0 and standard deviation of 1 you can work them out for any similar distribution of scores with a very simple equation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2)  I'm not following this question.  The mean of 0 and standard deviation of 1 usually applies to the standard normal distribution, often called the bell curve.  The most likely value is the mean and it falls off as you get farther away.  If you have a truly flat distribution then there is no value more likely than another.  Your question here is poorly formed.  Were you looking at questions about coin flips perhaps?  Look up binomial distribution and central limit theorem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3)  &quot;mean here&quot;?  Where?  The simple answer for z-scores is that they are your scores scaled as if your mean were 0 and standard deviation were 1.  Another way of thinking about it is that it takes an individual score as the number of standard deviations that score is from the mean.  The equation is calculating the (score - mean) / standard deviation.  The reasons you'd do that are quite varied but one is that in intro statistics courses you have tables of probabilities for different z-scores (see answer 1).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you looked up z-score first, even in wikipedia, you would have gotten pretty good answers.&lt;/p&gt;&#xA;" OwnerUserId="601" LastActivityDate="2010-07-31T15:46:05.483" CommunityOwnedDate="2010-07-31T15:46:05.483" CommentCount="1" />
  <row Id="1066" PostTypeId="1" AcceptedAnswerId="1068" CreationDate="2010-07-31T18:20:26.487" Score="1" ViewCount="157" Body="&lt;p&gt;My question is actually quite short, but I'll have to start by describing the context since I am not sure how to directly ask it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider the following &quot;game&quot;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We have a segment of length n (&quot;large segment&quot;) and m integers (&quot;lengths&quot;), all considerably smaller than n. For each of the m lengths we draw a random  sub-segment of its length on the large segment. For example, if the large segment is of size 1000 (i.e. 1..1000) and we are given lengths 20, 10, 50, than a possible solution would be: 31..50, 35..44, 921..970 (sub-segments of lengths 20, 10 and 50 respectively).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Notes:&#xA;1. This is just a toy example. We usually have many more lengths so there are many overlaps and each position in the large segment is covered by multiple sub-segments. &#xA;2. Remember that the lengths are given; only their mapping to the large segment is random.&#xA;3. Drawing a sub-segment of length k is done bu simply drawing a number from a uniform distribution over 1..n-k (a sub-segment of size k can start at position 1, 2, ... n-k).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, we conduct many simulations of the process an d record the data. We finally examine for each position the distribution of number of sub-segments covering this position. If we look at positions that are relatively far from the edges of the large segment, the distribution in each such position is normal, and all the distributions look the same.&#xA;The &quot;problem&quot; is that the positions at the ends do not look normal at all. This is not surprising, since, for example, if we are now drawing a sub-segment of length 10, the only way the very first position in the large segment will be covered is if we draw 1, whereas, for example, the 10th position will be covered if we draw 1,2,3,..10.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I am trying to figure out is what is the kind of distribution we see in the &quot;edge&quot; positions (it's not normal, but I think it usually looks like a normal distribution with its tail cut in one direction), and also how can I approximate this distribution density function from my simulations. For the &quot;center&quot; positions, I just estimate the mean and standard deviation and since I beleive the distributions there are normal - I can use the normal density function. This alos makes me think if I really need to treat the positions in a categorical way - &quot;near the edges&quot; and &quot;not near the edges&quot;, or whether there are actually the same in some sort (some generalization of the normal distribution?).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you, and sorry again for the length of the post.&lt;/p&gt;&#xA;" OwnerUserId="634" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-30T21:24:14.807" LastActivityDate="2010-09-30T21:24:14.807" Title="Approximating density function for a non-normal distribution" Tags="&lt;distributions&gt;&lt;normality&gt;" AnswerCount="1" CommentCount="5" />
  <row Id="1068" PostTypeId="2" ParentId="1066" CreationDate="2010-07-31T18:50:56.847" Score="4" ViewCount="" Body="&lt;p&gt;I'm not sure I understand your question exactly, but I assume you are looking for the probability mass at each point, where an event is defined as a subsegment covering a particular position.  If this is true, I believe you should be able to work out the exact probability mass function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For each subsegment of length K the distribution at a particular location between K and N-K+1 is uniform and proportional to K.  The distribution at the tails is stepwise increasing from 1 to K.  This can be seen by just working out one example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given multiple subsegments of different sizes, simply add these functions up.  You can then normalize everything so the weights sum to one if you want a proper distribution function.  Given a length and a list of subsegment lengths, this probability mass function should be easy to code up in the language of your choice.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are interested in the &lt;em&gt;number of segments&lt;/em&gt; covering any point, this is simply the sum of Bernoulli random variables with different p, defined at each point using the same pmf.  The simplest approach would be to enumerate the possibilities from 1 to K.&lt;/p&gt;&#xA;" OwnerUserId="493" LastEditorUserId="493" LastEditorDisplayName="" LastEditDate="2010-07-31T19:06:43.163" LastActivityDate="2010-07-31T19:06:43.163" CommentCount="8" />
  <row Id="1069" PostTypeId="2" ParentId="1062" CreationDate="2010-07-31T19:47:20.103" Score="3" ViewCount="" Body="&lt;p&gt;This is a good, if underspecified question.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Simply put, obtaining an &lt;a href=&quot;http://en.wikipedia.org/wiki/Orthogonal_coordinates&quot; rel=&quot;nofollow&quot;&gt;orthogonal&lt;/a&gt; parametrization allows for parameters of interest to be conveniently related to other parameters, particularly in establishing needed minimizations. Whether or not this is useful depends on what you are trying to do (in the case of some physics problems, for instance, orthogonal parametrization may obscure the symmetries of interest).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the case of statistical inference, orthogonal parametrization can allow the use of  statistics by way of minimization (or its dual) on orthogonal parameters.  For instance, &lt;a href=&quot;http://www.jstor.org/pss/2345476&quot; rel=&quot;nofollow&quot;&gt;Cox and Reid&lt;/a&gt; use the orthogonality of nuisance parameters (and their appropriately applied maximum likelihood estimates) to construct a generalization of a liklihood ratio statistic for a parameter of interest. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To see how orthogonality allows for this requires an understanding of the properties of commonly used mathematical spaces and the construction of estimators, which is essentially an issue of &lt;a href=&quot;http://en.wikipedia.org/wiki/Information_geometry&quot; rel=&quot;nofollow&quot;&gt;information geometry&lt;/a&gt;.  See &lt;a href=&quot;http://omega.albany.edu:8008/ignorance/zhu98.pdf&quot; rel=&quot;nofollow&quot;&gt;Information Geometry, Bayesian Inference, Ideal Estimates and Error Decomposition&lt;/a&gt; for a lucid, but technical description of orthogonality and its role in statistical inference.&lt;/p&gt;&#xA;" OwnerUserId="39" LastEditorUserId="39" LastEditorDisplayName="" LastEditDate="2010-10-12T16:05:37.587" LastActivityDate="2010-10-12T16:05:37.587" CommentCount="4" />
  <row Id="1070" PostTypeId="2" ParentId="1062" CreationDate="2010-07-31T20:17:03.217" Score="3" ViewCount="" Body="&lt;p&gt;In Maximum Likelihood, the term orthogonal parameters is used when you can achieve a clean factorization of a multi-parameter likelihood function.  Say your data have two parameters $\theta$ and $\lambda$.  If you can rewrite the joint likelihood:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$L(\theta, \lambda) = L_{1}(\theta) L_{2}(\lambda)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;then we call $\theta$ and $\lambda$ &lt;em&gt;orthogonal parameters&lt;/em&gt;.  The obvious case is when you have independence, but this is not necessary for the definition as long as factorization can be achieved.  Orthogonal parameters are desirable because, if $\theta$ is of interest, then you can perform inference using $L_{1}$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When we don't have orthogonal parameters, we try to find factorizations like&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$L(\theta, \lambda) = L_{1}(\theta) L_{2}(\theta, \lambda)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and perform inference using $L_1$.  In this case, we must argue that the information loss due to excluding $L_{2}$ is low.  This leads to the concept of &lt;a href=&quot;http://stats.stackexchange.com/questions/1045/is-there-a-radical-difference-in-how-bayesian-and-frequentist-approaches-treat-nu/1051#1051&quot;&gt;marginal likelihood&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="251" LastEditorUserId="251" LastEditorDisplayName="" LastEditDate="2010-08-01T04:31:00.753" LastActivityDate="2010-08-01T04:31:00.753" CommentCount="3" />
  <row Id="1071" PostTypeId="1" CreationDate="2010-07-31T20:35:00.723" Score="1" ViewCount="69" Body="&lt;p&gt;I have data counting the number of words that appear in text reviews, each review rated 1 to 5 stars. So for every word, I have the rating breakdown of reviews that contain that word.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to perform a beta-binomial analysis on each word by rating (n=4) but I also want the rating categories to be evenly weighted (currently 5-star reviews are 59% of total reviews). If I proportionally weight the numbers (e.g. multiply by total number of reviews and divide by total number of reviews with that star rating), can I still say meaningful things about the data?&lt;/p&gt;&#xA;" OwnerUserId="637" LastActivityDate="2010-07-31T20:35:00.723" Title="Effects of scale on the beta-binomial distribution" Tags="&lt;dataset&gt;&lt;beta-binomial&gt;" CommentCount="2" FavoriteCount="1" />
  <row Id="1072" PostTypeId="2" ParentId="1016" CreationDate="2010-07-31T21:22:55.277" Score="2" ViewCount="" Body="&lt;p&gt;I second Rob's comment. An increasingly prefered alternative is to include all your variables and shrink them towards 0. See Tibshirani, R. (1996). Regression shrinkage and selection via the lasso.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www-stat.stanford.edu/~tibs/lasso/lasso.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www-stat.stanford.edu/~tibs/lasso/lasso.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="603" LastActivityDate="2010-07-31T21:22:55.277" CommentCount="2" />
  <row Id="1073" PostTypeId="2" ParentId="373" CreationDate="2010-07-31T21:33:59.133" Score="3" ViewCount="" Body="&lt;p&gt;Our aim is to eliminate &lt;strong&gt;both&lt;/strong&gt; goats.  We do this by marking one goat ourselves.  The quizmaster is then forced to choose between revealing the car or the other goat. Revealing the car is out of the question, so the quizmaster will reveal and eliminate the one goat we did not know about.  We then switch to the remaining door, thereby eliminating the goat we marked with our first choice, and get the car.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This strategy only fails if we do not mark a goat, but the car instead.  But that is unlikely: there are two goats and only one car.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So we have a chance of 2 in 3 to win the car.&lt;/p&gt;&#xA;" OwnerUserId="638" LastActivityDate="2010-07-31T21:33:59.133" CommentCount="1" />
  <row Id="1074" PostTypeId="2" ParentId="373" CreationDate="2010-07-31T22:01:41.933" Score="1" ViewCount="" Body="&lt;p&gt;The lesson?  Reformulate the question, and search for a strategy instead of looking at the situation.  Turn the thing on its head, work backwards...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;People are generally bad at working with chance.  Animals typically fare better, once they discover that either A or B gives a higher payout &lt;em&gt;on average&lt;/em&gt;; they stick to the choice with the better average.  (don't have a reference ready - sorry.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first thing people are tempted to do when seeing a 80/20 distribution, is to spread their choices to match the pay-out: 80% on the better choice, and 20% on the other.&#xA;This will result in a pay-out of 68%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Again, there is a valid scenario for people to choose such a strategy: If the odds shift over time, there's a good reason for sending out a probe and try the choice with the lower chance of success.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An important part of mathematical statistics actually studies the behaviour of processes to determine whether they &lt;strong&gt;are&lt;/strong&gt; random or not. &lt;/p&gt;&#xA;" OwnerUserId="638" LastActivityDate="2010-07-31T22:01:41.933" />
  <row Id="1079" PostTypeId="2" ParentId="146" CreationDate="2010-08-01T06:26:34.787" Score="3" ViewCount="" Body="&lt;p&gt;Careful... just because the PCs are by construction orthogonal to each other does not mean that there is not a pattern or that one PC can not appear to &quot;explain&quot; something about the other PCs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider 3D data (X,Y,Z) describing a large number of points distributed evenly on the surface of an American football (it is an ellipsoid -- not a sphere --  for those who have never watched American football).   Imagine that the football is in an arbitrary configuration so that neither X nor Y nor Z is along the long axis of the football.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Principal components will place PC1 along the long axis of the football, the axis that describes the most variance in the data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For any point in the PC1 dimension along the long axis of the football, the planar slice represented by PC2 and PC3 should describe a circle and the radius of this circular slice depends on the PC1 dimension.  It is true that regressions of PC2 or PC3 on PC1 should give a zero coefficient globally, but not over smaller sections of the football.... and it is clear that a 2D graph of PC1 and PC2 would show an &quot;interesting&quot; limiting boundary that is two-valued, nonlinear, and symmetric.&lt;/p&gt;&#xA;" OwnerUserId="87" LastEditorUserId="87" LastEditorDisplayName="" LastEditDate="2010-08-01T23:22:03.657" LastActivityDate="2010-08-01T23:22:03.657" />
  <row Id="1080" PostTypeId="2" ParentId="373" CreationDate="2010-08-01T06:53:28.040" Score="0" ViewCount="" Body="&lt;p&gt;I think there are several things going on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For one, the setup implies more information then the solution takes into account. That it is a game show, and the host is asking us if we want to switch.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you assume the host does not want the show to spend extra money (which is reasonable), then you would assume he would try to convince you to change if you had the right door. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a common sense way of looking at the problem that can confuse people, however I do think the main issue is not understanding how the new choice is different then the first (which is more clear in the 100 door case).&lt;/p&gt;&#xA;" OwnerUserId="572" LastActivityDate="2010-08-01T06:53:28.040" />
  <row Id="1081" PostTypeId="1" AcceptedAnswerId="1106" CreationDate="2010-08-01T07:16:03.853" Score="3" ViewCount="228" Body="&lt;p&gt;I have been reading Zuur, Ieno and Smith (2007) Analyzing ecological data, and on page 262, they try to explain how nMDS (non-metric multidimensional scaling) algorithm works. As my background is in biology and not math or statistics per se, I'm having hard time understanding a few points and would ask you if you could elaborate on them. I'm reproducing the entire algorithm list for clarity, and I hope I'm not breaking any laws by doing so.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Choose a measure of association and calculate the distance matrix D.&lt;/li&gt;&#xA;&lt;li&gt;Specify m, the number of axes.&lt;/li&gt;&#xA;&lt;li&gt;Construct a starting configuration E. This can be done using PCoA.&lt;/li&gt;&#xA;&lt;li&gt;Regress the configuration on D: D_ij = (alpha) + (beta)E_ij + (epsilon)_ij.&lt;/li&gt;&#xA;&lt;li&gt;Measure the relationship between the m dimensional configuration and the real dinstances by fitting a non-parametric (monotonic) regression curve in the Shepard diagram. A monotonic regression is constrained to increase. If a parametric regression line is used, we obtain PCoA.&lt;/li&gt;&#xA;&lt;li&gt;The discrepancy from the fitted curve is called STRESS.&lt;/li&gt;&#xA;&lt;li&gt;Using non-linear optimization routines, obtain a new estimation of E and go from step 4 until convergence.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Questions:&#xA;In 4., we regress the configuration to D. Where do we use the estimated parameters (alpha), (beta) and (epsilon)? Are these used to measure distance from the regression (Shepard diagram) in this new configuration&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In regard to number 7, can you talk a little about non-linear optimisation routines? My internet search came up pretty much empty in terms of a layman's explanation. I'm interested in knowing what this routine tries to achieve (in nMDS). And I guess the next question depends on knowing these routines: what represents convergence? What converges to where?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone add &quot;nmds&quot; tag? I can't create new tags yet...&lt;/p&gt;&#xA;" OwnerUserId="144" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-30T21:23:34.420" LastActivityDate="2010-09-30T21:23:34.420" Title="Help me understand nMDS algorithm" Tags="&lt;nonparametric&gt;&lt;multidimensional-scaling&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="1" />
  <row Id="1082" PostTypeId="1" CreationDate="2010-08-01T11:48:48.670" Score="3" ViewCount="143" Body="&lt;p&gt;I am trying to assess the significance of the obtained MI matrix. The initial input was a array of 3000 genes by 45 timepoints. MI was computed resulting in a array of 3600 by 3600.  I am thus comparing my results to a shuffled matrix with the same dimensions. I permutate the columns 100 times, thus have 100 results for each element in the matrix. At this stage shall I take the mean for each value in the cell and then overall mean of the matrix MI values to estimate the threshold cutoff? Is taking the mean plus 3SD sensible? Ideally comparison of probability density function between my model and the random should show large discrepancy.&lt;/p&gt;&#xA;" LastActivityDate="2010-08-01T18:41:27.523" Title="How to define the significance threshold for mutual information in terms of probability of that value occurring in surrogate set?" Tags="&lt;correlation&gt;" AnswerCount="1" FavoriteCount="2" />
  <row Id="1083" PostTypeId="2" ParentId="95" CreationDate="2010-08-01T11:56:51.260" Score="2" ViewCount="" Body="&lt;p&gt;Generally, by not allowing for assymetry, you expect the effect of shocks to last longers: i.e. the half-life increases (the half life is the number of units of time, after a 1 S.D. shock to $\epsilon_{t-1}$ for $\hat{\sigma}&lt;em&gt;t|I&lt;/em&gt;{t-1}$ to come back to the its unconditional value.) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a code snipped that downloads stock data, fits (e)Garch and computes half lifes, in R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;install.packages(&quot;rgarch&quot;,repos=&quot;http://R-Forge.R-project.org&quot;)&#xA;install.packages(&quot;fGarch&quot;)&#xA;install.packages(&quot;fImport&quot;)&#xA;library(rgarch)&#xA;library(fImport)&#xA;library(fGarch)&#xA;d1&amp;lt;-yahooSeries(symbols=&quot;ibm&quot;,nDaysBack=1000,frequency=c(&quot;daily&quot;))[,4]&#xA;dprice1&amp;lt;-diff(log(as.numeric(d1[length(d1):1])))&#xA;spec1&amp;lt;-ugarchspec(variance.model=list(model=&quot;eGARCH&quot;,garchOrder=c(1,1)),mean.model=list(armaOrder=c(0,0),include.mean=T))&#xA;spec2&amp;lt;-ugarchspec(variance.model=list(model=&quot;fGARCH&quot;,submodel=&quot;GARCH&quot;,garchOrder=c(1,1)),mean.model=list(armaOrder=c(0,0),include.mean=T))&#xA;fit1&amp;lt;-ugarchfit(data=dprice1,spec=spec1)&#xA;fit2&amp;lt;-ugarchfit(data=dprice1,spec=spec2)&#xA;halflife(fit1)&#xA;halflife(fit2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The reason for this is that generally speaking, negative spells tend to be more persistent.&#xA;If you don't control for this, you will generally bias the $\beta$ (i.e. persistance parameters) downwards.&lt;/p&gt;&#xA;" OwnerUserId="603" LastEditorUserId="603" LastEditorDisplayName="" LastEditDate="2010-08-01T13:38:50.213" LastActivityDate="2010-08-01T13:38:50.213" />
  <row Id="1084" PostTypeId="1" AcceptedAnswerId="1086" CreationDate="2010-08-01T12:08:02.313" Score="4" ViewCount="264" Body="&lt;p&gt;I'm trying to visualize a set of data that represents human body mass over time, taken from (usually) daily weighings.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because body mass tends to fluctuate +/- 3 pounds based on hydration I would like to draw a strongly smoothed line graph to minimize the fluctuation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help on what the equation would look like is much appreciated, or even just some names/links to send me in the right direction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT:&lt;/strong&gt;&#xA;I need to code the visualization in Javascript, so I need understanding of the math involved, rather than a library that will do it for me.&lt;/p&gt;&#xA;" OwnerUserId="642" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-08-13T11:29:34.633" LastActivityDate="2010-08-13T11:29:34.633" Title="Equation to calculate a smooth line given an irregular time series?" Tags="&lt;data-visualization&gt;&lt;smoothing&gt;" AnswerCount="2" />
  <row Id="1085" PostTypeId="2" ParentId="1084" CreationDate="2010-08-01T12:19:10.810" Score="0" ViewCount="" Body="&lt;p&gt;Use smoothing splines. Look at page 16 of this document:&#xA;&lt;a href=&quot;http://cran.r-project.org/web/packages/quantreg/vignettes/rq.pdf&quot; rel=&quot;nofollow&quot;&gt;http://cran.r-project.org/web/packages/quantreg/vignettes/rq.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="603" LastActivityDate="2010-08-01T12:19:10.810" CommentCount="2" />
  <row Id="1086" PostTypeId="2" ParentId="1084" CreationDate="2010-08-01T12:29:28.977" Score="6" ViewCount="" Body="&lt;p&gt;There are several methods of estimating a smooth trend line. Some of the most popular are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Moving average smoother&lt;/li&gt;&#xA;&lt;li&gt;Local linear regression (loess being a popular robust implementation)&lt;/li&gt;&#xA;&lt;li&gt;Smoothing splines &lt;/li&gt;&#xA;&lt;li&gt;Regression splines (of various flavors).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;My preference is to use &lt;strong&gt;penalized regression splines&lt;/strong&gt; which you can fit using the &lt;a href=&quot;http://cran.r-project.org/web/packages/mgcv/&quot; rel=&quot;nofollow&quot;&gt;mgcv&lt;/a&gt; package in R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt;: since you want to code it yourself, you might find it simplest to start with a moving average smoother (for each day, just take an average of observations from the k days on either side with k to be selected so as to give the appropriate level of smoothness).&lt;/p&gt;&#xA;" OwnerUserId="159" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-08-01T12:48:13.933" LastActivityDate="2010-08-01T12:48:13.933" CommentCount="1" />
  <row Id="1087" PostTypeId="2" ParentId="152" CreationDate="2010-08-01T13:51:19.667" Score="5" ViewCount="" Body="&lt;p&gt;There is a nice and reasonably recent discussion of this problem here:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.icms.org.uk/downloads/mixtures/Robert.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.icms.org.uk/downloads/mixtures/Robert.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Essentially, there are several standard strategies, and each has pros and cons. The most obvious thing to do is to formulate the prior in such a way as to ensure there is only one posterior mode (eg. order the means of the mixuture components), but this turns out to have a strange effect on the posterior, and therefore isn't generally used. Next is to ignore the problem during sampling, and then post-process the output to re-label the components to keep the labels consistent. This is easy to implement and seems to work OK. The more sophisticated approaches re-label on-line, either by keeping a single mode, or deliberately randomly permuting the labels to ensure mixing over multiple modes. I quite like the latter approach, but this still leaves the problem of how to summarise the output meaningfully. However, I see that as a separate problem.&lt;/p&gt;&#xA;" OwnerUserId="643" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-10-05T12:20:32.797" LastActivityDate="2010-10-05T12:20:32.797" />
  <row Id="1088" PostTypeId="2" ParentId="114" CreationDate="2010-08-01T14:16:22.900" Score="7" ViewCount="" Body="&lt;p&gt;In addition to those already mentioned, I like Rob Hyndman's blog:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://robjhyndman.com/researchtips/&quot; rel=&quot;nofollow&quot;&gt;http://robjhyndman.com/researchtips/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I guess he's too modest to mention it himself! ;-)&lt;/p&gt;&#xA;" OwnerUserId="643" LastActivityDate="2010-08-01T14:16:22.900" CommunityOwnedDate="2010-08-01T14:16:22.900" />
  <row Id="1089" PostTypeId="2" ParentId="726" CreationDate="2010-08-01T14:21:18.177" Score="3" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;The true logic of this world is in the calculus of probabilities.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;-- James Clerk Maxwell&lt;/p&gt;&#xA;" OwnerUserId="643" LastEditorUserId="795" LastEditorDisplayName="" LastEditDate="2010-12-03T04:04:49.117" LastActivityDate="2010-12-03T04:04:49.117" CommunityOwnedDate="2010-08-01T14:21:18.177" />
  <row Id="1090" PostTypeId="2" ParentId="726" CreationDate="2010-08-01T14:25:28.323" Score="3" ViewCount="" Body="&lt;p&gt;A bit obscure this one, but a great quote about subjective probability:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;... There is no way, however, in which the individual can avoid the burden of responsibility for his own evaluations. The key cannot be found that will unlock the enchanted garden wherein, among the fairy-rings and the shrubs of magic wands, beneath the trees laden with monads and noumena, blossom forth the flowers of probabilitas realis. With these fabulous blooms safely in our button-holes we would be spared the necessity of forming opinions, and the heavy loads we bear upon our necks would be rendered superflous once and for all.&quot; - Bruno de Finetti, Theory of Probability, Vol 2&lt;/p&gt;&#xA;" OwnerUserId="643" LastActivityDate="2010-08-01T14:25:28.323" CommunityOwnedDate="2010-08-01T14:25:28.323" />
  <row Id="1091" PostTypeId="2" ParentId="575" CreationDate="2010-08-01T18:14:31.930" Score="2" ViewCount="" Body="&lt;p&gt;There are TWO options for the inferential F-tests In SPSS. &#xA;Multivariate does NOT assume sphericity, adn so makes use of a different pairwise correlation for each pair of variables. &#xA;The &quot;tests of within subjects effects&quot;, including any post hoc tests, assumes sphericity and makes some corrections for using a common correlation across all tests. These procedures are a legacy of the days when computation was expensive, and are a waste of time with modern computing facilities.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My recommendation is to take the omnibus MULTIVARIATE F for any repeated measures. Then follow up with post hoc pairwise t-test, or ANOVA with only 2 levels in each repeated measure comparison if there are also between subject factors. I would make the simple bon ferroni correction of dividing the alpha level by the number of tests.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also be sure to look at the effect size [available in the option dialogue]. Large effect sizes that are 'close' to significant may be more worthy of attention [and future experiments] than small, but significant effects.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A more sophisticated approach is available in SPSS procedure MIXED, and also in less user friendly [but free] packages such as R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Summary, in SPSSS, multivariate F followed by pairwise post hocs eith Bon Ferroniwith Bonferroni should be sufficient for most needs.&lt;/p&gt;&#xA;" LastActivityDate="2010-08-01T18:14:31.930" />
  <row Id="1092" PostTypeId="2" ParentId="1082" CreationDate="2010-08-01T18:41:27.523" Score="2" ViewCount="" Body="&lt;p&gt;Dawy et al. outline an algorithm in &lt;em&gt;&lt;a href=&quot;http://www.ece.iit.edu/~biitcomm/research/references/Zaher%20Dawy/Zaher%20Dawy%202006/Gene%20Mapping%20and%20Marker%20Clustering%20Using%20Shannon%92s%20Mutual%20Information.pdf&quot; rel=&quot;nofollow&quot;&gt;Gene mapping and marker clustering using Shannon's mutual information&lt;/a&gt;&lt;/em&gt; (2006).  If you're using R, you may prefer the &lt;a href=&quot;http://www.bioconductor.org/packages/2.6/bioc/vignettes/BUS/inst/doc/bus.pdf&quot; rel=&quot;nofollow&quot;&gt;BUS&lt;/a&gt; package in Bioconductor.  &lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-08-01T18:41:27.523" />
  <row Id="1093" PostTypeId="1" CreationDate="2010-08-01T22:31:15.863" Score="2" ViewCount="200" Body="&lt;p&gt;I'm looking for a distribution to model a vector of $k$ binary random variables, $X_1, \ldots, X_k$.  Suppose I have observed that $\sum_i X_i = n$.  In this case I do not want to treat them as independent Bernoulli random variables.  Instead, I would like something like the multinomial:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(X_1=x_1, \ldots, X_k=x_k) = f(x_1, \ldots, x_k; n, p_1, \ldots, p_k) = \frac{n!}{x_1! \cdots x_k!} \prod_{i=1}^k p_i^{x_i}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;but instead of the $x_i$ being nonnegative integers, I want them restricted to be either 0 or 1.  I have been trying to see if the &lt;a href=&quot;http://en.wikipedia.org/wiki/Hypergeometric_distribution&quot; rel=&quot;nofollow&quot;&gt;multivariate hypergeometric&lt;/a&gt; is appropriate, but I'm not sure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance for any advice.&lt;/p&gt;&#xA;" OwnerUserId="647" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-29T00:33:52.767" LastActivityDate="2011-04-29T00:33:52.767" Title="Density function for a multivariate Bernoulli-like distribution" Tags="&lt;distributions&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="1094" PostTypeId="2" ParentId="1093" CreationDate="2010-08-01T22:38:49.430" Score="2" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In light of your comments, here is an updated answer:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Approach 1: Difficult to implement/analyze&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider the simple case of $k$ = 3 and $n$ = 2. In other words you toss 3 coins (with probabilities $p_1$, $p_2$ and $p_3$). Then, the required mass function for the above case is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$p_1 p_2 (1-p_3) + p_1 (1-p_2) p_3  + (1-p_1) p_2 p_3$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The above reduces to the &lt;a href=&quot;http://en.wikipedia.org/wiki/Binomial_distribution&quot; rel=&quot;nofollow&quot;&gt;binomial&lt;/a&gt; if the probabilities $p_i$ are all identical. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the general case, you will have ${k \choose n}$ terms where each term is unique with a structure similar to the one above.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Approach 2: Easier to analyze/implement&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead of the above, you could model each $X_i$ as a &lt;a href=&quot;http://en.wikipedia.org/wiki/Bernoulli_trial&quot; rel=&quot;nofollow&quot;&gt;bernoulli variable&lt;/a&gt; with probability $p_i$. You could then assume that $p_i$ follows a &lt;a href=&quot;http://en.wikipedia.org/wiki/Dirichlet_distribution&quot; rel=&quot;nofollow&quot;&gt;dirichlet distribution&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You would then estimate the model parameters by constructing the posterior distribution for $p_i$ conditional on observing $n$ successes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strike&gt;If you can normalize by n and and assuming that treating them as probabilities/proportions makes sense in your context you can use the &lt;a href=&quot;http://en.wikipedia.org/wiki/Dirichlet_distribution&quot; rel=&quot;nofollow&quot;&gt;dirichlet distribution&lt;/a&gt;.&lt;/strike&gt;&lt;/p&gt;&#xA;" LastActivityDate="2010-08-02T02:01:37.240" CommentCount="11" />
  <row Id="1095" PostTypeId="1" AcceptedAnswerId="1721" CreationDate="2010-08-01T22:52:13.423" Score="0" ViewCount="96" Body="&lt;p&gt;This question concerns  an implementation of the topmoumoute natural gradient (tonga) algorithm as described in page 5 in the paper Le Roux et al 2007 &lt;a href=&quot;http://research.microsoft.com/pubs/64644/tonga.pdf&quot; rel=&quot;nofollow&quot;&gt;http://research.microsoft.com/pubs/64644/tonga.pdf&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand that the basic idea is to augment stochastic gradient ascent with the covariance of the stochastic gradient estimates. Basically, the natural gradient approach multiplies a stochastic gradient with the inverse of the covariance of the gradient estimates in order to weight each component of the gradient by the variance of this component. We prefer moving into directions that show less variance during the stochastic gradient estimates: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ng \propto C^{-1} g$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since updating and inverting the covariance in an online optimisation setting is costly, the authors describe a ``cheap'' approximate update algorithm as described on page 5 as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$C_t = \gamma \hat{C}_{t-1} + g_tg_t^T$ where  $C_{t-1}$ is the low rank approximation at time step t-1. Writing  $\hat{C}_{t} = X_tX_t^T$ with $X_t =\sqrt{\gamma} X_{t-1}\ \ g_t]$ they use an iterative update rule for the  gram matrix $X_t^T X_T = G_t = $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;($\gamma G_{t-1}$ &amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; $\sqrt{\gamma} X^T_{t-1}g_t$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\sqrt{\gamma} g^T_t X_{t-1}$ &amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; $g_t^Tg_t$)                                                                                &lt;/p&gt;&#xA;&#xA;&lt;p&gt;They then state ``To keep a low-rank estimate of  $\hat{C}_{t} = X_tX_t^T$, we can compute its eigendecomposition and keep only the first k eigenvectors. This can be made low cost using its relation to that of the Gram matrix&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$G_t= X_t^T X_T$:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$G_t = VDV^T$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$C_t = (X_tVD^{-\frac12})D(X_tVD^{-\frac12})^T$''&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because it's cheaper than updating and decomposing G at every step, they then suggest that you should update X for several steps using &#xA;$C_{t+b} = X_{t+b}X_{t+b}^T$ with $X_{t+b} = \left[\gamma U_t, \ \gamma^{\frac{b-1}{2}g_{t+1}},...\ \  \gamma^{-\frac12}g_{t+b-1}, \ \gamma^{\frac{t+b}{2}g_{t+b}}\right]$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can see why you can get $C_t$ from $G_t$ using the eigendecomposition. But I'm unsure about their update rule for X. The authors don't explain where U is coming from. I assume (by notation) this is the first k eigenvectors of $C_t$, correct? But if so, why would the formula for $X_t$ be a good approximation for $X_t$? When I implement this update rule, $X_t$ does not seem to be a good approximation of the ''real'' $X_t$ (that you would get from $X_t = [\sqrt{\gamma} X_{t-1}\ \ g_t]$) at all. So why should I then be able to get a good approximation of $C^{-1}$ from this (I don't)? The authors are also not quite clear about how they keep $G_t$ from growing (The size oft he matrix $G_t$ increases at each iterative update). I assume they replace $G_t$ by $\hat V\hat D\hat V^T$ with $\hat V$ and $\hat D$ being the first k components of the eigendecomposition?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So in summary:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;I tried implementing this update rule, but I'm not getting good results and am unsure my implementation is correct&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;why should the update rule for $X_t$ be reasonable? Is $U_t$ really the first k eigenvectors of $C_t$? (Clearly I cannot let $X_t$ and $G_t$ grow for each observed gradient $g_t$)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;This is far fetched, but has anyone implemented this low rank approximate update of the covariance before and has some code to share so I can compare it to my implementation?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;As a simple example if I simulate having 15 gradients in matlab like: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X = rand(15, 5);&#xA;c = cov(X);&#xA;e = eig(c);&#xA;%if the eigenvectos of C would give a good approximation of the original , this should give an appoximation of c, no?:&#xA;c_r = e'*e; %no resemblence to c&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So I'm quite certainly doing it wrong, I guess U might actually not be the eigenvectors of C, but then what is U?&#xA;Any suggestions or references would be  most welcome!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(sorry about the terrible layout, looks like only a subset of latex is supported, no arrays for matriced and embedded latex doesn't look particularly good, all the formulas are much more readable on page 5 of the referenced paper :)&#xA;Also, is this considered off-topic here? It's really more related to optimisation and machine-learning...)&lt;/p&gt;&#xA;" OwnerUserId="282" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-29T00:34:34.327" LastActivityDate="2011-04-29T00:34:34.327" Title="Tonga: low rank approximation of the natural gradient, question regarding Le Roux et al. 2007" Tags="&lt;algorithms&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="1097" PostTypeId="2" ParentId="181" CreationDate="2010-08-02T02:20:30.080" Score="6" ViewCount="" Body="&lt;p&gt;I realize this question has been answered, but i don't think the extant answer really engages the question beyond pointing to a link generally related to the question's subject matter. In particular, the link describes one technique for programmatic network configuration, but that is not a &quot;[a] standard and accepted method&quot; for network configuration.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to set NN architecture programmatically. By following a small set of clear rules, you can get a &quot;correct&quot; architecture (layer/node configuration), though of course it won't be optimal. Once this basic network is initialized, you can optimize the configuration during training using a number of ancillary algorithms; one family of these works by &quot;pruning&quot; nodes based on (small) values of the weight vector after a certain number of training epochs--in other words, eliminating 'unnecessary' nodes (more on this below).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So every NN (i.e., MLP) has three types of layers: input, hidden, and output.&lt;hr&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The Input Layer&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Simple--this layer configuration is completely and uniquely determined once you know the shape of your training data. Specifically, there's only one input layer, and the number of neurons comprising that layer is equal to the number of dimensions (columns) in your data. Some NN configurations add one additional node for a bias term.&lt;hr&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The Output Layer&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;NN's have only one of them. It's size (number of neurons) is completely determined by the chosen model configuration. Is your NN going running in 'Machine' Mode or 'Regression' Mode (the ML convention of using a term that is also used in statistics but assigning a different meaning to it is very confusing). Machine mode: returns a class label (Group I); Regression Mode returns a value (e.g., price). One node in the output layer for Regression Mode; for &#xA;Machine Mode, one node for each class label in your model.&lt;hr&gt;  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The Hidden Layers&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So those few rules set the number of layers and size (neurons/layer) for both the input and output layers. That leave the hidden layers. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How many hidden layers? Well if your data is linearly separable (which you often know by the time you begin coding a NN) then you don't need any hidden layers at all. Of course, you don't need an NN to resolve your data either, but it will still do the job. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Beyond that, as you probably know, there's a mountain of commentary on the question of hidden layer configuration in NNs (see the famous NN FAQ for an [excellent summary][1] of that commentary). One issue within this subject on which there is a consensus is the performance difference from adding additional hidden layers: the situations in which performance improves with a second (or third, etc.) hidden layer are very small. &lt;em&gt;One hidden layer is sufficient for the large majority of problems.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So what about size of the hidden layer(s)--how many neurons? There are some empirically-derived rules-of-thumb, of these, the most commonly relied on is '&lt;em&gt;the optimal size of the hidden layer is usually between the size of the input and size of the output layers&lt;/em&gt;'. Jeff Heaton, author of &quot;Introduction to Neural Networks in Java&quot; offers a few more, [here][2].&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In sum, for most problems, one could probably get decent performance (even without a second optimization step) by setting the hidden layer configuration using just two rules: (i) number of hidden layers equals one; and (ii) the number of neurons in that layer is the mean of the neurons in the input and output layers. &lt;hr&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Optimization of the Network Configuration&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;'Pruning' describes a set of techniques to trim network size (by nodes not layers) to improve computational performance and sometimes resolution performance. The gist of these techniques is removing nodes from the network during training by identifying those nodes which, if removed from the network, would not noticeably affect network performance (i.e., resolution of the data). (Even without using a formal pruning technique, you can get a rough idea of which nodes are not important by looking at your weight matrix after training; look weights very close to zero--it's the nodes on either end of those weights that are often removed during pruning.) Obviously, if you use a pruning algorithm during training then begin with a network configuration that is more likely to have excess (i.e., 'prunable') nodes--in other words, when deciding on a network architecture, err on the side of more neurons, if you add a pruning step.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Put another way, by applying a pruning algorithm to your network during training, you can approach optimal network configuration; whether you can do that in a single &quot;up-front&quot; (such as a genetic-algorithm-based algorithm) i don't know, though i do know that for now, this two-setp optimization is more common.&lt;/p&gt;&#xA;" OwnerUserId="438" LastActivityDate="2010-08-02T02:20:30.080" />
  <row Id="1098" PostTypeId="2" ParentId="1093" CreationDate="2010-08-02T04:04:13.047" Score="3" ViewCount="" Body="&lt;p&gt;The appropriate distribution is &lt;a href=&quot;http://en.wikipedia.org/wiki/Noncentral_hypergeometric_distributions&quot; rel=&quot;nofollow&quot;&gt;Wallenius's noncentral hypergeometric distribution&lt;/a&gt;.  Using an urn analogy, the problem is equivalent to picking $n$ of $k$ balls without replacement, where each ball is a different color: the parameters $p$ are analogous to the weights of picking a particular color.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem: it's not very convenient to work with, though there is an &lt;a href=&quot;http://cran.r-project.org/web/packages/BiasedUrn/BiasedUrn.pdf&quot; rel=&quot;nofollow&quot;&gt;R package&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="647" LastActivityDate="2010-08-02T04:04:13.047" CommentCount="1" />
  <row Id="1099" PostTypeId="1" AcceptedAnswerId="1108" CreationDate="2010-08-02T04:40:22.673" Score="3" ViewCount="457" Body="&lt;p&gt;I am working on disease infection data, and I am puzzled on whether to handle the data as &quot;categorical&quot; or &quot;continuous&quot;.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&quot;Infection Count&quot; &#xA;&lt;ul&gt;&#xA;&lt;li&gt;the number of infection cases found in a specific period of time, the count &#xA;is generated    from categorical data (i.e. no. of patient tagged as &quot;infected&quot;)&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&quot;Patient Bed Days&quot; &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;sum of total number of day stay in the ward by all patients in that ward, again, the count is generated from categorical data (i.e. no. of patient tagged as &quot;staying in that particular ward&quot;)&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&quot;infection per patient bed days&quot;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&quot;infection count&quot; / &quot;patient bed days&quot;&#xA;both were originally count data, but now becomes a rate&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;Question:&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Can I use Chi-Square here to assess whether the difference in &quot;infections per patient bed days&quot; is statistically significant or not?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;h3&gt;Updates&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;I have found that I can compare the incidence rate (or call it infection rate), but doing something like &quot;incidence rate difference&quot; (IRD) or &quot;incidence rate ratio&quot; (IRR). (I found it from &lt;a href=&quot;http://www.statsdirect.com/help/rates/incidence_rates.htm&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What is the difference between IRD and t-test? &lt;/li&gt;&#xA;&lt;li&gt;Is there any statistical test complementary for IRR?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="588" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-02-25T03:08:55.227" LastActivityDate="2011-02-25T21:34:57.103" Title="How to handle count data (categorical data), when it has been converted to a rate?" Tags="&lt;categorical-data&gt;&lt;count-data&gt;" AnswerCount="6" CommentCount="3" FavoriteCount="2" />
  <row Id="1100" PostTypeId="2" ParentId="886" CreationDate="2010-08-02T05:16:40.183" Score="6" ViewCount="" Body="&lt;p&gt;I will give an itemized answer. Can provide more citations on demand, although this is not really controversial.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Statistics is &lt;strong&gt;not&lt;/strong&gt; all about&#xA;maximizing (log)-likelihood. That's&#xA;anathema to principled bayesians who&#xA;just update their posteriors or&#xA;propagate their beliefs through an&#xA;appropriate model.&lt;/li&gt;&#xA;&lt;li&gt;A lot of statistics &lt;strong&gt;is&lt;/strong&gt; about loss&#xA;minimization. And so is a lot of&#xA;Machine Learning. Empirical loss&#xA;minimization has a different meaning&#xA;in ML. For a clear, narrative view,&#xA;check out Vapnik's &quot;The nature of&#xA;statistical learning&quot;&lt;/li&gt;&#xA;&lt;li&gt;Machine Learning is &lt;strong&gt;not&lt;/strong&gt; all about&#xA;loss minimization. First, because&#xA;there are a lot of bayesians in ML;&#xA;second, because a number of&#xA;applications in ML have to do with&#xA;temporal learning and approximate DP.&#xA;Sure, there is an objective function,&#xA;but it has a very different meaning&#xA;than in &quot;statistical&quot; learning.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I don't think there is a gap between the fields, just many different approaches, all overlapping to some degree. I don't feel the need to make them into systematic disciplines with well-defined differences and similarities, and given the speed at which they evolve, I think it's a doomed enterprise anyway.&lt;/p&gt;&#xA;" OwnerUserId="30" LastActivityDate="2010-08-02T05:16:40.183" />
  <row Id="1101" PostTypeId="2" ParentId="1053" CreationDate="2010-08-02T05:49:40.147" Score="0" ViewCount="" Body="&lt;p&gt;Survival Analysis: A Self-Learning Text &#xA;by Kleinbaum and Klein&lt;/p&gt;&#xA;&#xA;&lt;p&gt;is pretty good.  It depends on what you want.  This is more of a non-technical introduction.  It's focused on practical applications and minimizes the mathematics.  Pedegocially, it's also intended for learning outside of the classroom.&lt;/p&gt;&#xA;" OwnerUserId="485" LastActivityDate="2010-08-02T05:49:40.147" CommunityOwnedDate="2010-09-16T12:35:43.497" />
  <row Id="1102" PostTypeId="2" ParentId="890" CreationDate="2010-08-02T08:14:36.877" Score="0" ViewCount="" Body="&lt;p&gt;Thannks a lot for this very detailed answer. It does make perfect sense to me. If I have a model know. For example: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;y = - 2.8 - 1.2 * urban - 3 * forest&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;with the reference category grassland and I would like to predict the model for a new environment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I have a point with grassland, the probability of y would be like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;y = 1 / (1 + exp(2.8))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is that right?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And lets say for a point in in a forest&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;y = 1 / (1+ exp(-(-2.8 - 3*1)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Thanks a lot again&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Mike&lt;/p&gt;&#xA;" LastActivityDate="2010-08-02T08:14:36.877" CommentCount="2" />
  <row Id="1103" PostTypeId="2" ParentId="1063" CreationDate="2010-08-02T08:46:38.047" Score="1" ViewCount="" Body="&lt;p&gt;To start with what we're talking about here is the standard normal distribution, a normal distribution with a mean of 0 and a standard deviation of 1. The short-hand for a variable which is distributed as a standard normal distribution is Z. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are my answers to your questions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(1) I think there are two key reasons why standard normal distributions are attractive. Firstly, any normally distributed variable can be converted or transformed to a standard normal by subtracting its mean from each observation before dividing each observation by the standard deviation. This is called the Z-transformation or the creation of Z-scores. This is very handy especially in the days before computers. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you wanted to find out the probability of some event from your variable which is normally distributed with mean 65.6 with a standard deviation of 10.2 wouldn't that be a right pain in the backside without a computer? Let's say that this variable is the heights in inches of American women. And let's say that we're interested in finding out the probability that a woman randomly drawn from the population will be very tall - say over 75 inches tall. Well this is a bit of a pain to find out with a computer as I would have to carry around a table for every possible normal distribution with me. However, if I transform this to a Z-score I can use the one table to find out the probability, thus:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(x_i - x-bar)/sd_x = Z&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(75 - 65.6)/10.2 = 0.9215&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using the Z table I find that the cumulative probability P(z &amp;lt; Z) - 0.8212 and therefore the probability of finding a woman as tall or taller than 75 inches is 17.88%. We can do this with &lt;strong&gt;any&lt;/strong&gt; normally distributed variable and so this standard normal distribution is very handy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second reason why the standard normal distribution is used frequently is due to the interpretation is provides in terms of Z-scores. Each &quot;observation&quot; in a Z-transformed variable is how many standard deviations the original untransformed observation was from the mean. This is particularly handy for standardized tests where the raw or absolute performance is less important than the relative performance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(2) I don't follow you here. I think you may be confused as to what we mean by a cumulative distribution function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(3) Z-scores are the individual &quot;observations&quot; or datum in a variable which has been Z-transformed. Return to my example of the variable - height of American women in inches. One particular observation of which may be a tall woman of height 75 inches. The Z-score for this is the result of Z-transforming the variable as we did earlier: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(x_i - x-bar)/sd_x = Z&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(75 - 65.6)/10.2 = 0.9215&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Z-score in this case is 0.9215. The interpretation of the Z-score is that this particular woman is 0.9215 standard deviations taller than the mean height. A person who was 55.4 inches tall have a Z-score of 1 and would be 1 standard deviation below mean height. &lt;/p&gt;&#xA;" OwnerUserId="215" LastActivityDate="2010-08-02T08:46:38.047" CommunityOwnedDate="2010-08-02T08:46:38.047" />
  <row Id="1104" PostTypeId="2" ParentId="726" CreationDate="2010-08-02T08:58:27.830" Score="9" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;While the individual man is an insoluble puzzle, in the aggregate he becomes a mathematical certainty. You can, for example, never foretell what any one man will be up to, but you can say with precision what an average number will be up to. Individuals vary, but percentages remain constant. So says the statistician.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Arthur Conan Doyle&lt;/p&gt;&#xA;" OwnerUserId="17" LastActivityDate="2010-08-02T08:58:27.830" CommunityOwnedDate="2010-08-02T08:58:27.830" />
  <row Id="1106" PostTypeId="2" ParentId="1081" CreationDate="2010-08-02T09:58:34.953" Score="3" ViewCount="" Body="&lt;p&gt;Few opening remarks. In nMDS you have a matrix of dissimilarities $D_{ij}$ (not distances; for instance this can be a per cent of people that said in some poll that i&amp;amp;j are not similar). What you want to obtain is a set of points ($E=[X_i]$) representing objects on M-dim space; having it, you have the matrix of distances between objects in this space $d_{ij}$.&lt;br&gt;&#xA;nMDS tries to guess such $E$ that $d_{ij}$ has the same rank as $D_{ij}$; it is like connecting each object pair with spring the more strong the less dissimilar the pair is and then releasing the whole configuration -- after relaxation, the objects that has been connected using stronger springs will be nearer.&lt;br&gt;&#xA;Point 4 is something like overfitting regression. You have some approximation of objects position $E^a$, and so also approximated distances $d^a_{ij}$. Now you can do regression $d^a_{ij}$~$f(D_{ij})$ and using it count the distances that should be if the $D$ would be represented perfectly $d^r_{ij}=f(D_{ij})$.&lt;br&gt;&#xA;Still, because you cannot directly count $E$ from $d^r$ (this is the problem of nonlinear optimization here), you must somehow mutate $E$ so that the distances will approach $d^r$. The standard method here is to mimic physical analogy with springs and move objects which are connected with most extended springs (having largest $|d^a_{ij}-d^r_{ij}|$) towards themselves so that the potential energy (this STRESS) of the system will be minimized mostly.&lt;/p&gt;&#xA;" OwnerUserId="88" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-08-02T11:23:47.897" LastActivityDate="2010-08-02T11:23:47.897" />
  <row Id="1107" PostTypeId="2" ParentId="1099" CreationDate="2010-08-02T10:03:30.803" Score="1" ViewCount="" Body="&lt;p&gt;From a technical purist point of view, you cannot as your ratio &quot;infection per patient bed days&quot; is not a continuous variable. For example, an irrational value will never appear in your dataset. However, you can ignore this technical issue and do whatever tests that may be appropriate for your context. By way of analogy, incomes levels are discrete but almost everyone treats them as continuous.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By the way, it is not entirely clear why you want to do a chi-square but I am assuming there is some background context why that makes sense for you.&lt;/p&gt;&#xA;" LastActivityDate="2010-08-02T10:03:30.803" />
  <row Id="1108" PostTypeId="2" ParentId="1099" CreationDate="2010-08-02T10:55:30.890" Score="4" ViewCount="" Body="&lt;p&gt;For me it does not at all sound appropriate to use a chi-square test here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I guess what you wanna do is the following: You have different wards or treatments or whatever else kind of nominal variable (i.e., groups) that divides your data. For each of these groups you collected the &lt;em&gt;Infection Count&lt;/em&gt; and the &lt;em&gt;Patient Bed Days&lt;/em&gt; to calculate the &lt;em&gt;infection per patient bed days&lt;/em&gt;. Know you wanna check for differences between the groups, right?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If so, an analysis of variance (ANOVA, in case of more than two groups) or a t-test (in case of two groups) is probably appropriate given by the reasons in Srikant Vadali's post (and if the assumptions homogeneity of variances and comparable groups sizes are also met) and the &lt;code&gt;beginner&lt;/code&gt; tag should be added.&lt;/p&gt;&#xA;" OwnerUserId="442" LastActivityDate="2010-08-02T10:55:30.890" />
  <row Id="1109" PostTypeId="2" ParentId="1099" CreationDate="2010-08-02T11:24:35.680" Score="5" ViewCount="" Body="&lt;p&gt;I'm not quite sure what your data look like, or what your precise problem is, but I assume you have a table with the following headings and type:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;ward (categorical), infections (integer), patient-bed-days (integer or continuous).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;and you want to tell if the infection rate is statistically different for different wards?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One way of doing this is to use a Poisson model:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Infections ~ Poisson (Patient bed days * ward infection rate)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This can be achieved by using a Poisson glm, with log link function and the log of patient-bed-days in the offset. In R, the code would look something like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;glm(infections ~ ward + offset(log(patient-bed-days)), family=poisson())&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="495" LastActivityDate="2010-08-02T11:24:35.680" />
  <row Id="1110" PostTypeId="2" ParentId="146" CreationDate="2010-08-02T11:36:35.380" Score="3" ViewCount="" Body="&lt;p&gt;This is a bit off-topic, but here's some good stuff about PCA (mostly in R):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;b&gt;HTML&lt;/b&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.statsoft.com/textbook/principal-components-factor-analysis/?button=1&quot; rel=&quot;nofollow&quot;&gt;http://www.statsoft.com/textbook/principal-components-factor-analysis/?button=1&lt;/a&gt; &lt;br /&gt;&#xA;&lt;a href=&quot;http://ordination.okstate.edu/PCA.htm&quot; rel=&quot;nofollow&quot;&gt;http://ordination.okstate.edu/PCA.htm&lt;/a&gt;&lt;br /&gt;&#xA;&lt;a href=&quot;http://astrostatistics.psu.edu/datasets/R/MV.html&quot; rel=&quot;nofollow&quot;&gt;http://astrostatistics.psu.edu/datasets/R/MV.html&lt;/a&gt;&lt;br /&gt;&#xA;&lt;a href=&quot;http://www.statmethods.net/advstats/factor.html&quot; rel=&quot;nofollow&quot;&gt;http://www.statmethods.net/advstats/factor.html&lt;/a&gt;&lt;br /&gt;&lt;/p&gt;&#xA;&#xA;&lt;hr /&gt;&#xA;&#xA;&lt;p&gt;&lt;b&gt;PDF&lt;/b&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://cran.r-project.org/web/packages/HSAUR/vignettes/Ch_principal_components_analysis.pdf&quot; rel=&quot;nofollow&quot;&gt;http://cran.r-project.org/web/packages/HSAUR/vignettes/Ch_principal_components_analysis.pdf&lt;/a&gt;&lt;br /&gt;&#xA;&lt;a href=&quot;http://www.uga.edu/strata/software/pdf/pcaTutorial.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.uga.edu/strata/software/pdf/pcaTutorial.pdf&lt;/a&gt;&lt;br /&gt;&#xA;&lt;a href=&quot;http://www.cs.otago.ac.nz/cosc453/student_tutorials/principal_components.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.cs.otago.ac.nz/cosc453/student_tutorials/principal_components.pdf&lt;/a&gt;&lt;br /&gt;&#xA;&lt;a href=&quot;http://www.miislita.com/information-retrieval-tutorial/pca-spca-tutorial.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.miislita.com/information-retrieval-tutorial/pca-spca-tutorial.pdf&lt;/a&gt;&lt;br /&gt;&#xA;&lt;a href=&quot;http://www.unc.edu/~rcm/book/factor.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.unc.edu/~rcm/book/factor.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1356" LastActivityDate="2010-08-02T11:36:35.380" />
  <row Id="1111" PostTypeId="2" ParentId="1053" CreationDate="2010-08-02T13:53:47.847" Score="3" ViewCount="" Body="&lt;p&gt;For a very clear, succinct and applied approach, I highly recommend &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0521546737&quot; rel=&quot;nofollow&quot;&gt;Event History Modeling&lt;/a&gt; by Box-Steffenmeier and Jones&lt;/p&gt;&#xA;" OwnerUserId="302" LastActivityDate="2010-08-02T13:53:47.847" CommunityOwnedDate="2010-09-16T12:35:43.497" />
  <row Id="1112" PostTypeId="1" AcceptedAnswerId="1113" CreationDate="2010-08-02T14:38:55.070" Score="6" ViewCount="320" Body="&lt;p&gt;I want to represent a variable as a number between 0 and 1. The variable is a non-negative integer with no inherent bound. I map 0 to 0 but what can I map to 1 or numbers between 0 and 1?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I could use the history of that variable to provide the limits. This would mean I have to restate old statistics if the maximum increases. Do I have to do this or are there other tricks I should know about?&lt;/p&gt;&#xA;" OwnerUserId="652" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-17T20:29:37.823" LastActivityDate="2010-09-17T20:29:37.823" Title="How to represent an unbounded variable as number between 0 and 1" Tags="&lt;normalization&gt;" AnswerCount="7" CommentCount="8" FavoriteCount="3" />
  <row Id="1113" PostTypeId="2" ParentId="1112" CreationDate="2010-08-02T14:56:35.277" Score="8" ViewCount="" Body="&lt;p&gt;A very common trick to do so (e.g., in connectionist modeling) is to use the &lt;a href=&quot;http://en.wikipedia.org/wiki/Tanh&quot; rel=&quot;nofollow&quot;&gt;hyperbolic tangent tanh&lt;/a&gt; as the 'squashing function&quot;.&#xA;It automatically fits all numbers into the interval between -1 and 1. Which in your case restricts the range from 0 to 1.&#xA;In &lt;code&gt;r&lt;/code&gt; and &lt;code&gt;matlab&lt;/code&gt; you get it via &lt;code&gt;tanh()&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another squashing function is the logistic function (thanks to Simon for the name), provided by $ f(x) = 1 / (1 + e ^{-x} ) $, which restricts the range from 0 to 1 (with 0 mapped to .5). So you would have to multiply the result by 2 and subtract 1 to fit your data into the interval between 0 and 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is some simple R code which plots both functions (tanh in red, logistic in blue) so you can see how both squash:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x &amp;lt;- seq(0,20,0.001)&#xA;plot(x,tanh(x),pch=&quot;.&quot;, col=&quot;red&quot;, ylab=&quot;y&quot;)&#xA;points(x,(1 / (1 + exp(-x)))*2-1, pch=&quot;.&quot;,col=&quot;blue&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="442" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-08-03T00:57:51.577" LastActivityDate="2010-08-03T00:57:51.577" CommentCount="1" />
  <row Id="1114" PostTypeId="2" ParentId="1112" CreationDate="2010-08-02T15:20:11.690" Score="3" ViewCount="" Body="&lt;p&gt;Any sigmoid function will work:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The top half of the &lt;a href=&quot;http://en.wikipedia.org/wiki/Logistic_function&quot; rel=&quot;nofollow&quot;&gt;logistic function&lt;/a&gt; (multiply by 2, subtract 1)&lt;/li&gt;&#xA;&lt;li&gt;The &lt;a href=&quot;http://en.wikipedia.org/wiki/Error_function&quot; rel=&quot;nofollow&quot;&gt;error function&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;tanh, as suggested by Henrik.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="495" LastActivityDate="2010-08-02T15:20:11.690" CommentCount="1" />
  <row Id="1115" PostTypeId="1" CreationDate="2010-08-02T16:28:37.277" Score="-3" ViewCount="185" Body="&lt;p&gt;I have computed percent change from time1 to time2 for several variables, can I regress Pch_earnings=pch_prod, pch_price?&#xA;If I run a model with actual data and dummy for time1=1, time2=0, dummy is not stat. sign. but there are large changes. Thanks&lt;/p&gt;&#xA;" OwnerUserId="474" LastActivityDate="2010-08-02T20:17:03.580" ClosedDate="2010-08-03T07:58:26.837" Title="Linear Regression" Tags="&lt;regression&gt;" AnswerCount="3" CommentCount="2" />
  <row Id="1116" PostTypeId="2" ParentId="1053" CreationDate="2010-08-02T16:29:37.387" Score="2" ViewCount="" Body="&lt;p&gt;&quot;Survival analysis using SAS: a practical guide&quot; by Paul D. Allison provides a good guide to the connection between the math and SAS code - how to think about your information, how to code, how to interpret results. Even if you are using R, there will be parallels that could prove useful.&lt;/p&gt;&#xA;" LastActivityDate="2010-08-02T16:29:37.387" CommunityOwnedDate="2010-09-16T12:35:43.497" CommentCount="1" />
  <row Id="1117" PostTypeId="2" ParentId="1040" CreationDate="2010-08-02T16:43:44.403" Score="0" ViewCount="" Body="&lt;p&gt;The answer depends on the degree of misspecification and sample size. In small and moderate samples simplified model will fit (in most cases) better to data then the true model.&#xA;In moderate and large samples residuals don't have to be normal as due to CLT regression coefficients are normal anyway.&lt;/p&gt;&#xA;" OwnerUserId="419" LastActivityDate="2010-08-02T16:43:44.403" />
  <row Id="1118" PostTypeId="2" ParentId="1115" CreationDate="2010-08-02T16:43:49.333" Score="0" ViewCount="" Body="&lt;p&gt;You're likely to have problems with covariance - your model fails to meet the assumption of linear regression that the observations are independent, because a subject in your study will be correlated with itself between time 1 and time 2 (By the way, what are your observations?  One for each product type?)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might want to look into &quot;repeated measures&quot; methods.&lt;/p&gt;&#xA;" LastActivityDate="2010-08-02T16:43:49.333" CommentCount="3" />
  <row Id="1119" PostTypeId="2" ParentId="1112" CreationDate="2010-08-02T16:49:48.093" Score="1" ViewCount="" Body="&lt;p&gt;In addition to the good suggestions by Henrik and Simon Byrne, you could use f(x) = x/(x+1).  By way of comparison, the logistic function will exaggerate differences as x grows larger.  That is, the difference between f(x) and f(x+1) will be larger with the logistic function than with f(x) = x/(x+1).  You may or may not want that effect.&lt;/p&gt;&#xA;" LastActivityDate="2010-08-02T16:49:48.093" />
  <row Id="1120" PostTypeId="2" ParentId="138" CreationDate="2010-08-02T16:55:33.157" Score="1" ViewCount="" Body="&lt;p&gt;There are some very good learning materials here: &lt;a href=&quot;http://scc.stat.ucla.edu/mini-courses/materials-from-past-mini-courses/spring-2009-mini-course-materials/&quot; rel=&quot;nofollow&quot;&gt;http://scc.stat.ucla.edu/mini-courses/materials-from-past-mini-courses/spring-2009-mini-course-materials/&lt;/a&gt;&lt;/p&gt;&#xA;" LastActivityDate="2010-08-02T16:55:33.157" CommunityOwnedDate="2010-08-02T16:55:33.157" />
  <row Id="1121" PostTypeId="2" ParentId="1112" CreationDate="2010-08-02T16:57:33.557" Score="0" ViewCount="" Body="&lt;p&gt;There are two ways to implement this that I use commonly. I am always working with realtime data, so this assumes continuous input. Here's some pseudo-code:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Using a trainable minmax:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;define function peak:&lt;br /&gt;&#xA;// keeps the highest value it has received&lt;br /&gt;&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;define function trough:&lt;br /&gt;&#xA;// keeps the lowest value it has received&lt;br /&gt;&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;define function calibrate:&lt;br /&gt;&#xA;// toggles whether peak() and trough() are receiving values or not&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;define function scale:&lt;br /&gt;&#xA;// maps input range [trough.value() to peak.value()] to [0.0 to 1.0]&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This function requires that you either perform an initial training phase (by using calibrate()) or that you re-train either at certain intervals or according to certain conditions. For instance, imagine a function like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;define function outBounds (val, thresh):&lt;br /&gt;&#xA;if val &gt; (thresh*peak.value()) || val &amp;lt; (trough.value() / thresh):&lt;br /&gt;&#xA;calibrate()&lt;br /&gt;&#xA;// peak and trough are normally not receiving values, but if outBounds() receives a value that is more than 1.5 times the current peak or less than the current trough divided by 1.5, then calibrate() is called which allows the function to recalibrate automatically.&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Using an historical minmax:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;var arrayLength = 1000&lt;/code&gt;&lt;br /&gt;&#xA;&lt;code&gt;var histArray[arrayLength]&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;define historyArray(f):&lt;br /&gt;&#xA;histArray.pushFront(f) //adds f to the beginning of the array&lt;/code&gt;&lt;br/&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;define max(array):&lt;br /&gt;&#xA;// finds maximum element in histArray[]&lt;br /&gt;&#xA;return max&lt;/code&gt;&lt;br /&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;define min(array):&lt;br /&gt;&#xA;// finds minimum element in histArray[]&lt;br /&gt;&#xA;return min&lt;/code&gt;&lt;br /&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;define function scale:&lt;br /&gt;&#xA;// maps input range [min(histArray) to max(histArray)] to [0.0 to 1.0]&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;main()&lt;/code&gt;&lt;br /&gt;&#xA;&lt;code&gt;historyArray(histArray)&lt;/code&gt;&lt;br /&gt;&#xA;&lt;code&gt;scale(min(histArray), max(histArray), histArray[0])&lt;br /&gt;&#xA;// histArray[0] is the current element&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="162" LastEditorUserId="162" LastEditorDisplayName="" LastEditDate="2010-08-02T17:33:15.917" LastActivityDate="2010-08-02T17:33:15.917" CommunityOwnedDate="2010-08-02T16:57:33.557" CommentCount="1" />
  <row Id="1122" PostTypeId="2" ParentId="534" CreationDate="2010-08-02T17:05:49.053" Score="2" ViewCount="" Body="&lt;ol&gt;&#xA;&lt;li&gt;Almost always in randomized trials&lt;/li&gt;&#xA;&lt;li&gt;Almost always in observational study when someone measure all confouders (almost never)&lt;/li&gt;&#xA;&lt;li&gt;Sometimes when someone measure some counfounders (IC* algorithim of DAG discovery in Pearl's book Causality)&lt;/li&gt;&#xA;&lt;li&gt;In non gaussian linear models with two or more variables but not using correlation as measure of relationship (http://www.cs.helsinki.fi/group/neuroinf/lingam/)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Most of discovery algorithms are implemented in Tetrad IV (http://www.phil.cmu.edu/projects/tetrad/)&lt;/p&gt;&#xA;" OwnerUserId="419" LastActivityDate="2010-08-02T17:05:49.053" />
  <row Id="1123" PostTypeId="1" CreationDate="2010-08-02T17:08:08.113" Score="1" ViewCount="136" Body="&lt;p&gt;In many papers I see data representing a rate of success  (i.e a number between 0 and 1) modeled as a gaussian. This is clearly a sin (the range of variation of the gaussian is all of R),&#xA;but how bad is that sin? Under what assumptions would you say it is tolerable?&lt;/p&gt;&#xA;" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-08-03T18:49:43.080" LastActivityDate="2010-08-03T19:26:55.673" Title="Modeling success rate with gaussian distribution" Tags="&lt;distributions&gt;&lt;normality&gt;" AnswerCount="4" />
  <row Id="1124" PostTypeId="2" ParentId="1123" CreationDate="2010-08-02T17:15:04.730" Score="0" ViewCount="" Body="&lt;p&gt;Could you quote from &quot;many papers&quot; so that we would get some context?  Between &quot;Gaussian&quot; and &quot;number between 0 and 1&quot; I see slight conflict as the draws from a Gaussian are not bounded. Maybe you meant p-values?&lt;/p&gt;&#xA;" OwnerUserId="334" LastActivityDate="2010-08-02T17:15:04.730" />
  <row Id="1125" PostTypeId="2" ParentId="1123" CreationDate="2010-08-02T17:15:14.387" Score="1" ViewCount="" Body="&lt;p&gt;It depends on the data. While the normal distribution does span the real line do note that nearly 99% of the values are contained within 3 standard deviations of the mean. Thus, if the following conditions hold it may be a reasonable assumption:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(a) the data range is such that 99% of the data falls between [$\mu - 3\sigma,\mu+3\sigma]$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(b) the data is unimodal &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(c) the data 'passes' other relevant tests for &lt;a href=&quot;http://en.wikipedia.org/wiki/Normality_test&quot; rel=&quot;nofollow&quot;&gt;normality&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having said that some decision needs to be taken in the event that a draw from this distribution falls below 0 or above 1. Two ideas in such a situation: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(a) If draw &amp;lt; 0 set the draw to 0 and if draw &gt; 1 set the draw to 1 or&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(b) Model the distribution as truncated normal with the cut off points at 0 and 1.&lt;/p&gt;&#xA;" LastActivityDate="2010-08-02T17:15:14.387" />
  <row Id="1126" PostTypeId="1" CreationDate="2010-08-02T17:29:12.087" Score="9" ViewCount="208" Body="&lt;p&gt;Joshua Epstein wrote a paper titled &quot;Why Model?&quot; available at &lt;a href=&quot;http://www.santafe.edu/media/workingpapers/08-09-040.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.santafe.edu/media/workingpapers/08-09-040.pdf&lt;/a&gt; in which gives 16 reasons:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Explain (very distinct from predict)&lt;/li&gt;&#xA;&lt;li&gt;Guide data collection&lt;/li&gt;&#xA;&lt;li&gt;Illuminate core dynamics&lt;/li&gt;&#xA;&lt;li&gt;Suggest dynamical analogies&lt;/li&gt;&#xA;&lt;li&gt;Discover new questions&lt;/li&gt;&#xA;&lt;li&gt;Promote a scientific habit of mind&lt;/li&gt;&#xA;&lt;li&gt;Bound (bracket) outcomes to plausible ranges&lt;/li&gt;&#xA;&lt;li&gt;Illuminate core uncertainties.&lt;/li&gt;&#xA;&lt;li&gt;Offer crisis options in near-real time&lt;/li&gt;&#xA;&lt;li&gt;Demonstrate tradeoffs / suggest efficiencies&lt;/li&gt;&#xA;&lt;li&gt;Challenge the robustness of prevailing theory through perturbations&lt;/li&gt;&#xA;&lt;li&gt;Expose prevailing wisdom as incompatible with available data&lt;/li&gt;&#xA;&lt;li&gt;Train practitioners&lt;/li&gt;&#xA;&lt;li&gt;Discipline the policy dialogue&lt;/li&gt;&#xA;&lt;li&gt;Educate the general public&lt;/li&gt;&#xA;&lt;li&gt;Reveal the apparently simple (complex) to be complex (simple)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;(Epstein elaborates on many of the reasons in more detail in his paper.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to ask the community:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;are there are additional reasons that Epstein did not list?&lt;/li&gt;&#xA;&lt;li&gt;is there a more elegant way to conceptualize (a different grouping perhaps) these reasons?&lt;/li&gt;&#xA;&lt;li&gt;are any of Epstein's reasons flawed or incomplete?&lt;/li&gt;&#xA;&lt;li&gt;are their clearer elaborations of these reasons?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="660" LastActivityDate="2010-09-03T03:49:48.030" CommunityOwnedDate="2010-08-04T11:38:07.717" Title="Reasons besides prediction to build models?" Tags="&lt;modeling&gt;" AnswerCount="10" CommentCount="5" FavoriteCount="4" />
  <row Id="1127" PostTypeId="2" ParentId="1115" CreationDate="2010-08-02T17:55:23.633" Score="0" ViewCount="" Body="&lt;p&gt;You can do it, but I think that using percentages in a regression framework is likely to lead to a model that has little value.  I would try to generalize the model so that the percentage change is a special case, but that more complex behaviour is possible.&lt;/p&gt;&#xA;" OwnerUserId="187" LastActivityDate="2010-08-02T17:55:23.633" />
  <row Id="1128" PostTypeId="2" ParentId="1126" CreationDate="2010-08-02T17:57:15.127" Score="4" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Reason 17.  Write a paper.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Sort-of just kidding but not really.  There seems to be a bit of overlap between some of his points (eg 1, 5, 6, 12, 14).&lt;/p&gt;&#xA;" OwnerUserId="334" LastActivityDate="2010-08-02T17:57:15.127" CommunityOwnedDate="2010-09-03T06:27:41.573" CommentCount="2" />
  <row Id="1129" PostTypeId="2" ParentId="1016" CreationDate="2010-08-02T18:00:10.207" Score="1" ViewCount="" Body="&lt;p&gt;Another vote for Rob's answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are also some interesting ideas in the &quot;relative importance&quot; literature.  This work develops methods that seek to determine how much importance is associated with each of a number of candidate predictors.  There are Bayesian and Frequentist methods.  Check the &quot;relaimpo&quot; package in R for citations and code.&lt;/p&gt;&#xA;" OwnerUserId="187" LastActivityDate="2010-08-02T18:00:10.207" />
  <row Id="1130" PostTypeId="1" CreationDate="2010-08-02T18:44:15.063" Score="3" ViewCount="75" Body="&lt;p&gt;I have data for about 1 year, 100 obs, mult obs per subject, transactions occur on weekly basis but have 6-12 subjects per week, there is no order to this. There is a policy change in latter half of year, I want to model change in dep. var due to policy change as a dummy var: time1=0, time2=1. Is this a case for fxd effects estimation? The number of weeks per subject varies a lot and the number of weeks in time1 is greater than time2.  Computed means for time1, time2 and percent change=large change in dep. var, estimated linear model:&#xA;pay=X1 X2 Time(dummy). Dummy var is not stat signif. Any suggestions as to how to model this? Can I treat it as panel data? Thanks&lt;/p&gt;&#xA;" OwnerUserId="474" LastActivityDate="2010-08-02T23:31:46.130" Title="Regression-mult obs per subject" Tags="&lt;regression&gt;" AnswerCount="1" />
  <row Id="1131" PostTypeId="2" ParentId="1126" CreationDate="2010-08-02T18:52:56.510" Score="3" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Save money&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I build mathematical/statistical of cellular mechanisms. For example, how a particular protein affects cellular ageing. The role of the model is mainly prediction, but also to save money. It's far cheaper to employ a single modeller than (say) a few wet-lab biologists with the associated equipment costs. Of course modelling doesn't fully replace the experiment, it just aids the process.&lt;/p&gt;&#xA;" OwnerUserId="8" LastActivityDate="2010-08-02T18:52:56.510" CommunityOwnedDate="2010-09-03T06:27:41.573" />
  <row Id="1132" PostTypeId="2" ParentId="1126" CreationDate="2010-08-02T19:16:14.867" Score="1" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;For fun!&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I'm sure most statisticians/modellers do their job because they enjoy it. Getting paid to do something you enjoy is quite nice!&lt;/p&gt;&#xA;" OwnerUserId="8" LastActivityDate="2010-08-02T19:16:14.867" CommunityOwnedDate="2010-09-03T06:27:41.573" CommentCount="1" />
  <row Id="1133" PostTypeId="1" AcceptedAnswerId="1210" CreationDate="2010-08-02T19:19:42.860" Score="2" ViewCount="709" Body="&lt;p&gt;I have cross classified data in a 2 x 2 x 6 table. Let's call the dimensions &lt;code&gt;response&lt;/code&gt;, &lt;code&gt;A&lt;/code&gt; and &lt;code&gt;B&lt;/code&gt;. I fit a logistic regression to the data with the model &lt;code&gt;response ~ A * B&lt;/code&gt;. An analysis of deviance of that model says that both terms and their interaction are significant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, looking at the proportions of the data, it looks like only 2 or so levels of &lt;code&gt;B&lt;/code&gt; are responsible for these significant effects. I would like to test to see which levels are the culprits. Right now, my approach is to perform 6 chi-squared tests on 2 x 2 tables of &lt;code&gt;response ~ A&lt;/code&gt;, and then to adjust the p-values from those tests for multiple comparisons (using the Holm adjustment).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is whether there is a better approach to this problem. Is there a more principled modeling approach, or multiple chi-squared test comparison approach?&lt;/p&gt;&#xA;" OwnerUserId="287" LastActivityDate="2011-03-23T19:27:19.840" Title="Multiple Chi-Squared Tests" Tags="&lt;categorical-data&gt;&lt;logistic-regression&gt;&lt;multiple-comparisons&gt;&lt;chi-squared&gt;" AnswerCount="3" CommentCount="1" FavoriteCount="2" />
  <row Id="1134" PostTypeId="2" ParentId="1126" CreationDate="2010-08-02T19:28:31.653" Score="4" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;dimension reduction&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Sometimes there can be too much data, so forming an initial model allows for further analysis. &lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-08-02T19:28:31.653" CommunityOwnedDate="2010-09-03T06:27:41.573" />
  <row Id="1135" PostTypeId="2" ParentId="1126" CreationDate="2010-08-02T19:38:11.380" Score="2" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;regulation&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Government agencies require firms to provide reports using certain models. This provides for a degree of standardization in oversight. An example is the use of Value-at-Risk in the financial sector. &lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-08-02T19:38:11.380" CommunityOwnedDate="2010-09-03T06:27:41.573" />
  <row Id="1136" PostTypeId="2" ParentId="1126" CreationDate="2010-08-02T19:44:20.117" Score="0" ViewCount="" Body="&lt;p&gt;This is closely related to some of the others, but:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Eliminate human judgement&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Human decision making is subject to many different forces and biases. That means that you not only get different answers to the same question, but you can also end up with really suboptimal outcomes. Examples would be the over-confidence bias or anchoring.  &lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-08-02T19:44:20.117" CommunityOwnedDate="2010-09-03T06:27:41.573" CommentCount="2" />
  <row Id="1137" PostTypeId="2" ParentId="1115" CreationDate="2010-08-02T20:17:03.580" Score="0" ViewCount="" Body="&lt;p&gt;Regress the X1 value for time1 (and any other covariates you want) on the X1 variable for time 2 (your dependent variable).  Your regression model will look something like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;x1 time 2&quot; = &quot;x1 time1&quot; + x2 + x3 + x4 etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your regression coefficients for x2....xn will be the effect of changes of that variable on &quot;x1 time2&quot; controlling for &quot;x1 time 1&quot; (and everything else in the model).  Therefore you'll be able to see what is going on at time2 controlling for where things were at time 1.&lt;/p&gt;&#xA;" LastActivityDate="2010-08-02T20:17:03.580" />
  <row Id="1138" PostTypeId="2" ParentId="1126" CreationDate="2010-08-02T20:20:52.697" Score="0" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Repetitive problems that involve some form of benefit / cost&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In my field, we model the same set of variables in different locations, time frame, and magnitudes&lt;/p&gt;&#xA;" OwnerUserId="59" LastActivityDate="2010-08-02T20:20:52.697" CommunityOwnedDate="2010-09-03T06:27:41.573" />
  <row Id="1139" PostTypeId="2" ParentId="1095" CreationDate="2010-08-02T20:26:23.357" Score="0" ViewCount="" Body="&lt;p&gt;&quot;topmoumoute natural gradient&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Chapeau bas, messieurs.&lt;/p&gt;&#xA;" OwnerUserId="664" LastActivityDate="2010-08-02T20:26:23.357" />
  <row Id="1140" PostTypeId="2" ParentId="103" CreationDate="2010-08-02T20:31:22.017" Score="2" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://eagereyes.org&quot; rel=&quot;nofollow&quot;&gt;http://eagereyes.org&lt;/a&gt;&#xA;by Robert Kosara (~5 posts a month). This blog includes tutorials and discussion articles plus it has a great home page with lots of links to useful information.&lt;/p&gt;&#xA;" OwnerUserId="665" LastActivityDate="2010-08-02T20:31:22.017" CommunityOwnedDate="2010-08-02T20:31:22.017" />
  <row Id="1141" PostTypeId="2" ParentId="103" CreationDate="2010-08-02T20:35:41.460" Score="4" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://datavisualization.ch&quot; rel=&quot;nofollow&quot;&gt;http://datavisualization.ch&lt;/a&gt;&lt;br&gt; &#xA;by Benjamin Wiederkehr and others (~15 links a month). If you want heaps of links you can subscribe to their twitter feed twitter slash datavis (~5 links a day) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;ahhh... i'm a new member and so i can only post one link per post.&lt;/p&gt;&#xA;" OwnerUserId="665" LastEditorUserId="665" LastEditorDisplayName="" LastEditDate="2010-08-09T11:12:16.973" LastActivityDate="2010-08-09T11:12:16.973" CommunityOwnedDate="2010-08-02T20:35:41.460" />
  <row Id="1142" PostTypeId="1" CreationDate="2010-08-02T20:37:27.650" Score="11" ViewCount="964" Body="&lt;p&gt;I am working with a large amount of time series. These time series are basically network measurements coming every 10 minutes, and some of them are periodic (i.e. the bandwidth), while some other aren't (i.e. the amount of routing traffic).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like a simple algorithm for doing an online &quot;outlier detection&quot;. Basically, I want to keep in memory (or on disk) the whole historical data for each time series, and I want to detect any outlier in a live scenario (each time a new sample is captured). What is the best way to achieve these results?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm currently using a moving average in order to remove some noise, but then what next? Simple things like standard deviation, mad, ... against the whole data set doesn't work well (I can't assume the time series are stationary), and I would like something more &quot;accurate&quot;, ideally a black box like:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;double outlier_detection(double* vector, double value);&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where vector is the array of double containing the historical data, and the return value is the anomaly score for the new sample &quot;value&quot; .&lt;/p&gt;&#xA;" OwnerUserId="667" LastActivityDate="2010-11-03T00:21:10.310" Title="Simple algorithm for online outlier detection of a generic time series" Tags="&lt;time-series&gt;&lt;outliers&gt;&lt;mathematical-statistics&gt;&lt;real-time&gt;" AnswerCount="7" CommentCount="3" FavoriteCount="6" />
  <row Id="1144" PostTypeId="2" ParentId="1142" CreationDate="2010-08-02T20:48:01.103" Score="1" ViewCount="" Body="&lt;p&gt;You could use the standard deviation of the last N measurements (you have to pick a suitable N). A good anomaly score would be how many standard deviations a measurement is from the moving average.&lt;/p&gt;&#xA;" OwnerUserId="666" LastActivityDate="2010-08-02T20:48:01.103" CommentCount="2" />
  <row Id="1145" PostTypeId="2" ParentId="1123" CreationDate="2010-08-02T20:53:20.623" Score="1" ViewCount="" Body="&lt;p&gt;It's usually a small sin. In nature, most phenomena can't realistically receive any value in R, but we model them as if they could.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The greater sin is to assume that the rate of success is shaped like a normal distribution if it isn't.&lt;/p&gt;&#xA;" OwnerUserId="666" LastActivityDate="2010-08-02T20:53:20.623" />
  <row Id="1146" PostTypeId="2" ParentId="1126" CreationDate="2010-08-02T20:54:01.713" Score="1" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Control&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;A major aspect of the dynamic modelling literature is associated with control. This kind of work spans a lot of disciplines from politics/economics (see, e.g. Stafford Beer), biology (see e.g. N Weiner's 1948 work on Cybernetics) through to contemporary state space control theory (see for an intro Ljung 1999). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Control is kind of related to Epstein's 9 and 10, and Shane's answers about human judgement / regulation, but I thought it made sense to be explicit. Indeed, at the end of my engineering undergraduate career I would have given you a very concise response to the uses of modelling: control, inference and prediction. I guess inference, by which I mean filtering/smoothing/dimension-reduction etc, is maybe similar to Epstein's points 3 and 8. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course in my later years I wouldn't be so bold as to limit the purposes of modelling to control, inference and prediction. Maybe a fourth, covering many of Epsteins's points, should be &quot;coercion&quot; - the only way you should &quot;educate the public&quot; is to encourage us to make our own models... &lt;/p&gt;&#xA;" OwnerUserId="668" LastActivityDate="2010-08-02T20:54:01.713" CommunityOwnedDate="2010-09-03T06:27:41.573" CommentCount="1" />
  <row Id="1147" PostTypeId="2" ParentId="1142" CreationDate="2010-08-02T21:23:37.547" Score="3" ViewCount="" Body="&lt;p&gt;I am guessing sophisticated time series model will not work for you because of the time it takes to detect outliers using this methodology. Therefore, here is a workaround: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;First establish a baseline 'normal' traffic patterns for a year based on manual analysis of historical data which accounts for time of the day, weekday vs weekend, month of the year etc. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Use this baseline along with some simple mechanism (e.g., moving average suggested by Carlos) to detect outliers.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;You may also want to review the statistical &lt;a href=&quot;http://en.wikipedia.org/wiki/Statistical_process_control&quot; rel=&quot;nofollow&quot;&gt;process control literature&lt;/a&gt; for some ideas.&lt;/p&gt;&#xA;" LastActivityDate="2010-08-02T21:23:37.547" CommentCount="5" />
  <row Id="1148" PostTypeId="2" ParentId="1142" CreationDate="2010-08-02T21:50:27.147" Score="2" ViewCount="" Body="&lt;p&gt;Seasonally adjust the data such that a normal day looks closer to flat. You could take today's 5:00pm sample and subtract or divide out the average of the previous 30 days at 5:00pm. Then look past N standard deviations (measured using pre-adjusted data) for outliers. &#xA;This could be done separately for weekly and daily &quot;seasons.&quot;&lt;/p&gt;&#xA;" OwnerUserId="33" LastActivityDate="2010-08-02T21:50:27.147" CommentCount="4" />
  <row Id="1149" PostTypeId="1" AcceptedAnswerId="1150" CreationDate="2010-08-02T22:42:32.947" Score="16" ViewCount="501" Body="&lt;p&gt;The wiki discusses the problems that arise when &lt;a href=&quot;http://en.wikipedia.org/wiki/Multicollinearity&quot; rel=&quot;nofollow&quot;&gt;multicollinearity&lt;/a&gt; is an issue in linear regression. The basic problem is multicollinearity results in unstable parameter estimates which makes it very difficult to assess the effect of independent variables on dependent variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand the technical reasons behind the problems (may not be able to invert $X' X$, ill-conditioned $X' X$ etc) but I am searching for a more intuitive (perhaps geometric?) explanation for this issue.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a geometric or perhaps some other form of easily understandable explanation as to why multicollinearity is problematic in the context of linear regression?   &lt;/p&gt;&#xA;" LastActivityDate="2010-08-10T08:04:22.340" Title="Is there an intuitive explanation why multicollinearity is a problem in linear regression?" Tags="&lt;regression&gt;&lt;intuition&gt;&lt;multicollinearity&gt;" AnswerCount="6" CommentCount="1" FavoriteCount="3" />
  <row Id="1150" PostTypeId="2" ParentId="1149" CreationDate="2010-08-02T22:59:09.380" Score="14" ViewCount="" Body="&lt;p&gt;Consider the simplest case where $Y$ is regressed against $X$ and $Z$ and where $X$ and $Z$ are highly positively correlated. Then the effect of $X$ on $Y$ is hard to distinguish from the effect of $Z$ on $Y$ because any increase in $X$ tends to be associated with an increase in $Z$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another way to look at this is to consider the equation. If we write $Y = b_0 + b_1X + b_2Z + e$, then the coefficient $b_1$ is the increase in $Y$ for every unit increase in $X$ while holding $Z$ constant. But in practice, it is often impossible to hold $Z$ constant and the positive correlation between $X$ and $Z$ mean that a unit increase in $X$ is usually accompanied by some increase in $Z$ at the same time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A similar but more complicated explanation holds for other forms of multicollinearity.&lt;/p&gt;&#xA;" OwnerUserId="159" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-08-10T06:07:48.497" LastActivityDate="2010-08-10T06:07:48.497" CommentCount="1" />
  <row Id="1151" PostTypeId="2" ParentId="1149" CreationDate="2010-08-02T23:26:02.567" Score="2" ViewCount="" Body="&lt;p&gt;The geometric approach is to consider the least squares projection of $Y$ onto the subspace spanned by $X$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say you have a model:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$E[Y | X] = \beta_{1} X_{1} + \beta_{2} X_{2}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Our estimation space is the plane determined by the vectors $X_{1}$ and $X_{2}$ and the problem is to find coordinates corresponding to $(\beta_{1}, \beta_{2})$ which will describe the vector $\hat{Y}$, a least squares projection of $Y$ on to that plane.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now suppose $X_{1} = 2 X_{2}$, i.e. they're colinear.  Then, the subspace determined by $X_{1}$ and $X_{2}$ is just a line and we have only one degree of freedom.  So we can't determine two values $\beta_{1}$ and $\beta_{2}$ as we were asked.&lt;/p&gt;&#xA;" OwnerUserId="251" LastEditorUserId="251" LastEditorDisplayName="" LastEditDate="2010-08-03T02:34:37.810" LastActivityDate="2010-08-03T02:34:37.810" CommentCount="2" />
  <row Id="1152" PostTypeId="2" ParentId="1130" CreationDate="2010-08-02T23:31:46.130" Score="1" ViewCount="" Body="&lt;p&gt;If you estimate the policy change as a fixed effects estimation in the context of an OLS regression you'll over-estimate your degrees of freedom because of the repeated measures by subject.  If you do not think there is an overall trend of time (beyond the policy shift) then there is no reason to keep all of the observations, you could simply aggregate by subject for &quot;before policy change&quot; and &quot;after policy change&quot; and do a paired samples t-test.  Failing that - if you intend to add more predictors, like perhaps a linear effect of time, you might think about liner mixed effects regression, e.g. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;lmer(pay ~ time * policy + (1+time|SubjID))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;... might be a good place to start.&lt;/p&gt;&#xA;" OwnerUserId="196" LastActivityDate="2010-08-02T23:31:46.130" />
  <row Id="1153" PostTypeId="2" ParentId="1142" CreationDate="2010-08-03T00:54:56.310" Score="7" ViewCount="" Body="&lt;p&gt;Here is a simple R function that will find time series outliers (and optionally show them in a plot). It will handle seasonal and non-seasonal time series. The basic idea is to find robust estimates of the trend and seasonal components and subtract them. Then find outliers in the residuals. The test for residual outliers is the same as for the standard boxplot -- points greater than 1.5IQR above or below the upper and lower quartiles are assumed outliers. The number of IQRs above/below these thresholds is returned as an outlier &quot;score&quot;. So the score can be any positive number, and will be zero for non-outliers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I realise you are not implementing this in R, but I often find an R function a good place to start. Then the task is to translate this into whatever language is required.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;tsoutliers &amp;lt;- function(x,plot=FALSE)&#xA;{&#xA;    x &amp;lt;- as.ts(x)&#xA;    if(frequency(x)&amp;gt;1)&#xA;        resid &amp;lt;- stl(x,s.window=&quot;periodic&quot;)$time.series[,3]&#xA;    else&#xA;    {&#xA;        tt &amp;lt;- 1:length(x)&#xA;        resid &amp;lt;- residuals(loess(x ~ tt))&#xA;    }&#xA;    resid.q &amp;lt;- quantile(resid,prob=c(0.25,0.75))&#xA;    iqr &amp;lt;- diff(resid.q)&#xA;    limits &amp;lt;- resid.q + 1.5*iqr*c(-1,1)&#xA;    score &amp;lt;- abs(pmin((resid-limits[1])/iqr,0) + pmax((resid - limits[2])/iqr,0))&#xA;    if(plot)&#xA;    {&#xA;        plot(x)&#xA;        x2 &amp;lt;- ts(rep(NA,length(x)))&#xA;        x2[score&amp;gt;0] &amp;lt;- x[score&amp;gt;0]&#xA;        tsp(x2) &amp;lt;- tsp(x)&#xA;        points(x2,pch=19,col=&quot;red&quot;)&#xA;        return(invisible(score))&#xA;    }&#xA;    else&#xA;        return(score)&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-08-03T00:54:56.310" CommentCount="6" />
  <row Id="1154" PostTypeId="2" ParentId="1142" CreationDate="2010-08-03T00:56:33.157" Score="5" ViewCount="" Body="&lt;p&gt;If you're worried about assumptions with any particular approach, one approach is to train a number of learners on different signals, then use &lt;a href=&quot;http://en.wikipedia.org/wiki/Ensembles_of_classifiers&quot; rel=&quot;nofollow&quot;&gt;ensemble methods&lt;/a&gt; and aggregate over the &quot;votes&quot; from your learners to make the outlier classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;BTW, this might be worth reading or skimming since it references a few approaches to the problem.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://ir.lib.sfu.ca/retrieve/2372/etd1964.pdf&quot; rel=&quot;nofollow&quot;&gt;Online outlier detection over data streams&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="251" LastEditorUserId="251" LastEditorDisplayName="" LastEditDate="2010-08-03T01:49:22.570" LastActivityDate="2010-08-03T01:49:22.570" />
  <row Id="1155" PostTypeId="2" ParentId="1149" CreationDate="2010-08-03T02:20:32.477" Score="1" ViewCount="" Body="&lt;p&gt;If two regressors are perfectly correlated, their coefficients will be impossible to calculate; it's helpful to consider why they would be difficult to interpret &lt;em&gt;if we could calculate them&lt;/em&gt;.  In fact, this explains why it's difficult to interpret variables that are not perfectly correlated but that are also not truly independent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose that our dependent variable is the daily supply of fish in New York, and our independent variables include one for whether it rains on that day and one for the amount of bait purchased on that day.  What we don't realize when we collect our data is that every time it rains, fishermen purchase no bait, and every time it doesn't, they purchase a constant amount of bait.  So Bait and Rain are perfectly correlated, and when we run our regression, we can't calculate their coefficients.  In reality, Bait and Rain are probably not perfectly correlated, but we wouldn't want to include them both as regressors without somehow cleaning them of their endogeneity.&lt;/p&gt;&#xA;" OwnerUserId="672" LastActivityDate="2010-08-03T02:20:32.477" />
  <row Id="1156" PostTypeId="2" ParentId="1149" CreationDate="2010-08-03T02:28:37.437" Score="0" ViewCount="" Body="&lt;p&gt;My (very) layman intuition for this is that the OLS model needs a certain level of &quot;signal&quot; in the X variable in order to detect it gives a &quot;good&quot; predicting for Y.  If the same &quot;signal&quot; is spread over many X's (because they are correlated), then none of the correlated X's can give enough of a &quot;proof&quot; (statistical significance) that it is a real predictor.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The previous (wonderful) answers do a great work in explaining why that is the case.&lt;/p&gt;&#xA;" OwnerUserId="253" LastActivityDate="2010-08-03T02:28:37.437" />
  <row Id="1157" PostTypeId="2" ParentId="103" CreationDate="2010-08-03T02:45:53.883" Score="2" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://chartporn.org/&quot; rel=&quot;nofollow&quot;&gt;http://chartporn.org/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I find the blog name pretty humorous. Great dataviz.&lt;/p&gt;&#xA;" OwnerUserId="11" LastActivityDate="2010-08-03T02:45:53.883" CommunityOwnedDate="2010-08-03T02:45:53.883" />
  <row Id="1158" PostTypeId="2" ParentId="103" CreationDate="2010-08-03T02:49:32.377" Score="3" ViewCount="" Body="&lt;p&gt;It's not a blog, but Edward Tufte has an &lt;a href=&quot;http://www.edwardtufte.com/bboard/q-and-a?topic_id=1&quot; rel=&quot;nofollow&quot;&gt;interesting forum&lt;/a&gt; on information design including data visualization.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-08-03T02:49:32.377" CommunityOwnedDate="2010-08-03T02:49:32.377" />
  <row Id="1159" PostTypeId="2" ParentId="1133" CreationDate="2010-08-03T02:54:30.290" Score="1" ViewCount="" Body="&lt;p&gt;The unprincipled approach is to discard the disproportionate data, refit the model and see if logit/conditional odds ratios for response and A are very different (controlling for B).  This might tell you if there's cause for concern.  Pooling the levels of B is another approach.  On more principled lines, If you're worried about relative proportions inducing Simpson's paradox, then you can look into the conditional and marginal odds ratios for response/A and see if they reverse.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For avoiding multiple comparisons in particular, the only thing that occurs to me is to use a hierarchical model which accounts for random effects across levels.&lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-08-03T02:54:30.290" />
  <row Id="1160" PostTypeId="1" CreationDate="2010-08-03T03:02:57.650" Score="1" ViewCount="228" Body="&lt;p&gt;R allows us to put code to run in the beginning/end of a session.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What codes would you suggest putting there?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know of three interesting examples (although I don't have &quot;how to do them&quot; under my fingers here):&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Saving the &lt;a href=&quot;http://stat.ethz.ch/R-manual/R-devel/library/utils/html/savehistory.html&quot; rel=&quot;nofollow&quot;&gt;session history&lt;/a&gt; when closing R.&lt;/li&gt;&#xA;&lt;li&gt;Running a fortune() at the beginning of an R session.&lt;/li&gt;&#xA;&lt;li&gt;I was thinking of having an automated saving of the workspace.  But I didn't set on solving the issue of managing space (so there would always be X amount of space used for that backup)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Any more ideas? (or how you implement the above ideas)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;p.s: I am not sure if to put this here or on stackoverflow.  But I feel the people here are the right ones to ask.&lt;/p&gt;&#xA;" OwnerUserId="253" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-08-03T07:28:24.693" LastActivityDate="2010-09-07T20:59:46.453" CommunityOwnedDate="2010-08-03T03:02:57.650" Title="What code would you put before/after your R session?" Tags="&lt;r&gt;" AnswerCount="3" CommentCount="6" FavoriteCount="3" />
  <row Id="1161" PostTypeId="2" ParentId="1160" CreationDate="2010-08-03T03:49:10.593" Score="4" ViewCount="" Body="&lt;p&gt;Some information about how to implement this is provided at &lt;code&gt;help(.First)&lt;/code&gt; and &lt;code&gt;help(.Last)&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-08-03T03:49:10.593" CommunityOwnedDate="2010-08-03T03:49:10.593" />
  <row Id="1162" PostTypeId="2" ParentId="1126" CreationDate="2010-08-03T06:26:47.963" Score="1" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;To take (useful) action.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I'm paraphrasing someone else here, but suppose we built a system of public health around the model that infectious diseases are due to malevolent spirits that spread through contact.  The science of microbes may be an infinitely better model, but you could prevent a good number of contagions nonetheless.  (I think this was on reading a history of cybernetics, but I can't remember who made the point.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The point is that, along the lines of &quot;all models bad, some useful&quot;, we need to formulate models and refine them in order to undertake any useful actions with lasting consequences.  Otherwise, we might as well flip coins.&lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-08-03T06:26:47.963" CommunityOwnedDate="2010-09-03T06:27:41.573" />
  <row Id="1164" PostTypeId="1" AcceptedAnswerId="1170" CreationDate="2010-08-03T07:49:34.003" Score="20" ViewCount="1160" Body="&lt;p&gt;When solving business problems using data, it's common that at least one key assumption that under-pins classical statistics is invalid. The rest of the time, no one bothers to check those assumptions so you never actually know. For instance, that so many of the common web metrics are &quot;long-tailed&quot; (relative to the normal distribution) is, by now, so well documented that we take it for granted. Another example, online communities--even in communities with thousands of members, it's well-documented that by far the largest share of contribution to/participation in many of these community is attributable to a minuscule group of 'super-contributors.' (E.g., a few months ago, just after the SO API was made available in beta, a StackOverflow member published a brief analysis from data he collected through the API; his conclusion--less than one percent of the SO members account for most of the activity on SO (presumably asking questions, and answering them), another 1-2% accounted for the rest, and the overwhelming majority of the members do nothing).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So given the abundance of populations like this of interest to analysts, and given that classical models perform demonstrably poorly on these data, and given that robust and resistant methods have been around for a while (at least 20 years, i believe)--why are they not used more often? (I am also wondering why &lt;em&gt;I&lt;/em&gt; don't use them more often, but that's not really a question for Stats Exhange.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Yes i know that there are textbook chapters devoted entirely to robust statistics and i know there are (a few) R Packages ('robustbase' is the one i am familiar with and use), etc.--but given the obvious advantages of these techniques, they are often clearly the better tools for the job--why are they not used much more often? Shouldn't we expect to see robust (and resistant) statistics used far more often (perhaps even presumptively) compared with the classical analogs?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The only substantive (i.e., technical) explanation i have heard is that robust techniques (likewise for resistant methods) lack the power/sensitivity of classical techniques. I don't know if this is indeed true in some cases, but i do know it is not true in many cases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A final word of preemption: yes I know this question does not have a single demonstrably correct answer; very few questions on this Site do. Moreover, this question is a genuine inquiry; it's not a pretext to advance a point of view--i don't have a point of view here, just a question, to which i don't even have a hint of an answer at the moment.&lt;/p&gt;&#xA;" OwnerUserId="438" LastEditorUserId="438" LastEditorDisplayName="" LastEditDate="2010-08-03T09:36:03.060" LastActivityDate="2011-05-12T08:11:59.857" Title="Why Haven't Robust (and Resistant) Statistics Replaced Classical Techniques?" Tags="&lt;outliers&gt;&lt;model-selection&gt;&lt;nonparametric&gt;&lt;robust&gt;" AnswerCount="12" CommentCount="3" FavoriteCount="13" />
  <row Id="1165" PostTypeId="2" ParentId="1160" CreationDate="2010-08-03T08:27:13.243" Score="1" ViewCount="" Body="&lt;p&gt;On open, I set R options, load environment variables (eg. global variables, API keys) and open database connections, and then close those connections when exiting. With some of these things, I prefer to do them onLoad of my packages rather than per session. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regarding how to save your session, use the save command. &lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-08-03T08:27:13.243" CommunityOwnedDate="2010-08-03T08:27:13.243" />
  <row Id="1166" PostTypeId="2" ParentId="1164" CreationDate="2010-08-03T09:05:56.737" Score="6" ViewCount="" Body="&lt;p&gt;I Give an answer in two directions: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;things that are robust are not necessarily labeled robust. If you believe robustness against everything exists then you are naive. &lt;/li&gt;&#xA;&lt;li&gt;Statistical approaches that leave the problem of robustness appart are sometime not adapted to the real world but are often more valuable (as a concept) than an algorithm that looks like kitchening. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;developpment&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, I think there are a lot of good approaches in statistic (you will find them in R packages not necessarily with robust mentionned somewhere) which are naturally robust and tested on real data and the fact that you don't find algorithm with &quot;robust&quot; mentionned somewhere does not mean it is not robust. Anyway if you think being robust means being universal then you'll never find any robust procedure (no free lunch) you need to have some knowledge/expertise on the data you analyse in order to use adapted tool or to create an adapted model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, some approaches in statistic are not robust because they are dedicated to one single type of model. I think it is good sometime to work in a laboratory to try to understand things. It is also good to treat problem separatly to understand to what problem our solution is... this is how mathematician work. The example of Gaussian model elocant: is so much criticised because the gaussian assumption is never fulfilled but has bring 75% of the ideas used practically in statistic today. Do you really think all this is about writting paper to follow the publish or perish rule (which I don't like, I agree) ? &lt;/p&gt;&#xA;" OwnerUserId="223" LastEditorUserId="223" LastEditorDisplayName="" LastEditDate="2010-08-03T16:51:26.903" LastActivityDate="2010-08-03T16:51:26.903" />
  <row Id="1167" PostTypeId="1" AcceptedAnswerId="1168" CreationDate="2010-08-03T09:14:22.303" Score="1" ViewCount="663" Body="&lt;p&gt;Hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't understand why R gives me a warning about &quot;Longer object length is not a multiple of shorter object length&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have this object which is generated by doing an aggregate over an xts series giving the weekday median:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;u &amp;lt;- aggregate(d, list(Ukedag = format(index(d),&quot;%w&quot;)), median)&#xA;&#xA;1 314.0&#xA;2 282.5&#xA;3 270.0&#xA;4 267.0&#xA;5 240.5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then I try to apply this to my original xts series, which looks like this (only a lot longer)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;head(d)&#xA;2009-01-02 116&#xA;2009-01-05 256&#xA;2009-01-06 286&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Using:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;coredata(d) &amp;lt;- coredat(d) - u[format(index(d),&quot;%w&quot;)];&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Which results in a warning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The intent is to subtract the weekday mean. It appears to work despite the warning, but what should I worry about?&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Revised solution: Attempt 2&lt;/h2&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;apply.daily(d, function(x) coredata(x) - u[format(index(x), &quot;%w&quot;)] )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I did indeed have a serious error. This doesn't give any warnings and I tested it by doing:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;apply.daily(d, function(x) u[format(index(x), &quot;%w&quot;)] )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then checking some dates, and it appeared that is was in alignment with the calendar.&lt;/p&gt;&#xA;" OwnerUserId="671" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-08-03T18:34:40.903" LastActivityDate="2010-08-03T18:34:40.903" Title="Longer object length is not a multiple of shorter object length?" Tags="&lt;r&gt;" AnswerCount="1" CommentCount="5" />
  <row Id="1168" PostTypeId="2" ParentId="1167" CreationDate="2010-08-03T09:24:24.087" Score="4" ViewCount="" Body="&lt;p&gt;Yes, this is something that you should worry about. Check the length of your objects with nrow(). R can auto-replicate objects so that they're the same length if they differ, which means you might be performing operations on mismatched data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this case you have an obvious flaw in that your subtracting aggregated data from raw data. These will definitely be of different lengths. I suggest that you merge them as time series (using the dates), then locf(), the do your subtraction.  Otherwise merge them by truncating the original dates to the same interval as the aggregated series. Just be very careful that you don't drop observations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lastly, as some general advice as you get started: look at the result of your computations to see if they make sense. You might even pull them into a spreadsheet and replicate the results.     &lt;/p&gt;&#xA;" OwnerUserId="5" LastEditorUserId="5" LastEditorDisplayName="" LastEditDate="2010-08-03T09:30:53.010" LastActivityDate="2010-08-03T09:30:53.010" CommentCount="3" />
  <row Id="1169" PostTypeId="1" AcceptedAnswerId="1171" CreationDate="2010-08-03T12:20:09.890" Score="4" ViewCount="383" Body="&lt;p&gt;I'm looking to check my logic here. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say you measure a quantity in group A, and find the mean is 2 and your 95% confidence interval ranges from 1 to 3. Then you measure the same quantity in group B and find a mean of 4 with a 95% confidence interval that ranges from 3.5 to 4.5. Assuming that A &amp;amp; B are independent, what is the 95% confidence interval for the difference between the groups? Presumably you can compute this using standard t-statistics, but I'd like to know if it's also possible to compute an estimate based on the CI's alone.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I reason that the lower bound of the CI of the difference should be the minimum credible difference between A &amp;amp; B; that is, the lower bound of the interval for B (3.5) minus upper bound of the interval for A (3), which yields a lower bound for the difference of 0.5. Similarly, the upper bound of the CI of the difference should be the maximum credible difference between A &amp;amp; B; that is, the upper bound of the interval for B (4.5) minus lower bound of the interval for A (1), which yields a lower bound for the difference of 3.5. This reasoning thus yields a confidence interval for the difference that ranges from 0.5 to 3.5.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does that make sense, or is this a case where logic and statistics diverge?&lt;/p&gt;&#xA;" OwnerUserId="364" LastEditorUserId="364" LastEditorDisplayName="" LastEditDate="2010-08-13T00:56:41.777" LastActivityDate="2010-10-30T12:50:27.890" Title="CI for a difference based on independent CIs" Tags="&lt;confidence-interval&gt;" AnswerCount="1" FavoriteCount="3" />
  <row Id="1170" PostTypeId="2" ParentId="1164" CreationDate="2010-08-03T12:22:58.247" Score="21" ViewCount="" Body="&lt;p&gt;Researchers want small p-values, and you can get smaller p-values if you use methods that make stronger distributional assumptions. In other words, non-robust methods let you publish more papers. Of course more of these papers may be false positives, but a publication is a publication.  That's a cynical explanation, but it's sometimes valid.&lt;/p&gt;&#xA;" OwnerUserId="319" LastActivityDate="2010-08-03T12:22:58.247" CommentCount="4" />
  <row Id="1171" PostTypeId="2" ParentId="1169" CreationDate="2010-08-03T12:30:00.740" Score="10" ViewCount="" Body="&lt;p&gt;No, you can't compute a CI for the difference that way I'm afraid, for the same reason you can't use whether the CIs overlap to judge the statistical significance of the difference. See, for example, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;On Judging the Significance of Differences by Examining the Overlap Between Confidence Intervals&quot;&#xA;Nathaniel Schenker, Jane F Gentleman. The American Statistician. August 1, 2001, 55(3): 182-186. doi:10.1198/000313001317097960. &#xA;&lt;a href=&quot;http://pubs.amstat.org/doi/abs/10.1198/000313001317097960&quot; rel=&quot;nofollow&quot;&gt;http://pubs.amstat.org/doi/abs/10.1198/000313001317097960&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;or:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Overlapping confidence intervals or standard error intervals: What do they mean in terms of statistical significance?&#xA;Mark E. Payton, Matthew H. Greenstone, and Nathaniel Schenker. Journal of Insect Science 2003; 3: 34. &lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pmc/articles/PMC524673/&quot; rel=&quot;nofollow&quot;&gt;http://www.ncbi.nlm.nih.gov/pmc/articles/PMC524673/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The correct procedure requires you also know the sample sizes of both groups. You can then back-compute the two standard deviations from the CIs and use those to conduct a standard two-sample t-test, or to calculate a standard error of the difference and hence a CI for the difference.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-08-03T12:30:00.740" CommentCount="5" />
  <row Id="1172" PostTypeId="2" ParentId="563" CreationDate="2010-08-03T13:12:04.477" Score="8" ViewCount="" Body="&lt;p&gt;As a medical statistician with no previous knowledge of econom(etr)ics, I struggled to get to grips with instrumental variables as I often struggled to follow their examples and didn't understand their rather different terminology (e.g. 'endogeneity', 'reduced form', 'structural equation', 'omitted variables'). Here's a few references I found useful (the first should be freely available, but I'm afraid the others probably require a subscription):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Staiger D. Instrumental Variables. AcademyHealth Cyber Seminar in Health&#xA;Services Research Methods, March 2002.&#xA;&lt;a href=&quot;http://www.dartmouth.edu/~dstaiger/wpapers-Econ.htm&quot; rel=&quot;nofollow&quot;&gt;http://www.dartmouth.edu/~dstaiger/wpapers-Econ.htm&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Newhouse JP, McClellan M. Econometrics in Outcomes Research: The Use&#xA;of Instrumental Variables. Annual Review of Public Health 1998;19:17-34.&#xA;&lt;a href=&quot;http://dx.doi.org/10.1146/annurev.publhealth.19.1.17&quot; rel=&quot;nofollow&quot;&gt;http://dx.doi.org/10.1146/annurev.publhealth.19.1.17&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Greenland S. An introduction to instrumental variables for epidemiologists. International Journal of Epidemiology 2000;29:722-729. &lt;a href=&quot;http://dx.doi.org/10.1093/ije/29.4.722&quot; rel=&quot;nofollow&quot;&gt;http://dx.doi.org/10.1093/ije/29.4.722&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Zohoori N, Savitz DA.  Econometric approaches to epidemiologic data: Relating endogeneity and unobserved heterogeneity to confounding.  Annals of Epidemiology 1997;7:251-257. &lt;a href=&quot;http://dx.doi.org/10.1016/S1047-2797(97)00023-9&quot; rel=&quot;nofollow&quot;&gt;http://dx.doi.org/10.1016/S1047-2797(97)00023-9&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I'd also recommend chapter 4 of:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Angrist JD, Pischke JS. Mostly harmless econometrics: an empiricist's companion. Princeton, N.J: Princeton University Press, 2009. &lt;a href=&quot;http://www.mostlyharmlesseconometrics.com/&quot; rel=&quot;nofollow&quot;&gt;http://www.mostlyharmlesseconometrics.com/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-08-03T13:12:04.477" />
  <row Id="1173" PostTypeId="1" AcceptedAnswerId="1251" CreationDate="2010-08-03T13:36:38.303" Score="2" ViewCount="484" Body="&lt;p&gt;In my area of research, a popular way of displaying data is to use a combination of a bar chart with &quot;handle-bars&quot;. For example, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://img291.imageshack.us/img291/3387/tmpc.jpg&quot; alt=&quot;Example plot&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &quot;handle-bars&quot; alternate between standard errors and standard deviations depending on the author. Typically, the sample sizes for each &quot;bar&quot; are fairly small - around six.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These plots seem to be particularly popular in biological sciences - see the first few papers of &lt;a href=&quot;http://www.biomedcentral.com/bmcbiol/3&quot; rel=&quot;nofollow&quot;&gt;BMC Biology, vol 3&lt;/a&gt; for examples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So how would you present this data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Why I dislike these plots&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Personally I don't like these plots. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;When the sample size is  small, why not just display the individual data points. &lt;/li&gt;&#xA;&lt;li&gt;Is it the sd or the se that is being displayed? No-one agrees which to use. &lt;/li&gt;&#xA;&lt;li&gt;Why use bars at all. The data doesn't (usually) go from 0 but a first pass at the graph suggests it does.&lt;/li&gt;&#xA;&lt;li&gt;The graphs don't give an idea about range or sample size of the data.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;R script&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is the R code I used to generate the plot. That way you can (if you want) use the same data.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;                                        #Generate the data&#xA;set.seed(1)&#xA;names = c(&quot;A1&quot;, &quot;A2&quot;, &quot;A3&quot;, &quot;B1&quot;, &quot;B2&quot;, &quot;B3&quot;, &quot;C1&quot;, &quot;C2&quot;, &quot;C3&quot;)&#xA;prevs = c(38, 37, 31, 31, 29, 26, 40, 32, 39)&#xA;&#xA;n=6; se = numeric(length(prevs))&#xA;for(i in 1:length(prevs))&#xA;  se[i] = sd(rnorm(n, prevs, 15))/n&#xA;&#xA;                                        #Basic plot&#xA;par(fin=c(6,6), pin=c(6,6), mai=c(0.8,1.0,0.0,0.125), cex.axis=0.8)&#xA;barplot(prevs,space=c(0,0,0,3,0,0, 3,0,0), names.arg=NULL, horiz=FALSE,&#xA;        axes=FALSE, ylab=&quot;Percent&quot;, col=c(2,3,4), width=5, ylim=range(0,50))&#xA;&#xA;                                        #Add in the CIs&#xA;xx = c(2.5, 7.5, 12.5, 32.5, 37.5, 42.5,  62.5, 67.5, 72.5)&#xA;for (i in 1:length(prevs)) {&#xA;  lines(rep(xx[i], 2), c(prevs[i], prevs[i]+se[i]))&#xA;  lines(c(xx[i]+1/2, xx[i]-1/2), rep(prevs[i]+se[i], 2))&#xA;}&#xA;&#xA;                                        #Add the axis&#xA;axis(2, tick=TRUE, xaxp=c(0, 50, 5))&#xA;axis(1, at=xx+0.1, labels=names, font=1,&#xA;     tck=0, tcl=0, las=1, padj=0, col=0, cex=0.1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="8" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-08-03T20:42:45.293" LastActivityDate="2010-10-19T20:36:30.143" Title="Alternative graphics to &quot;handle bar&quot; plots" Tags="&lt;data-visualization&gt;" AnswerCount="7" CommentCount="3" FavoriteCount="3" />
  <row Id="1174" PostTypeId="1" AcceptedAnswerId="1212" CreationDate="2010-08-03T13:54:19.897" Score="4" ViewCount="727" Body="&lt;p&gt;I know of normality tests, but how do I test for &quot;Poisson-ness&quot;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have ~1000 sample of non-negative integers, which I suspect are taken from a Poisson distribution, and I would like to test that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Dave&lt;/p&gt;&#xA;" OwnerUserId="634" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-08-03T23:47:36.280" LastActivityDate="2010-08-04T15:24:11.960" Title="How can I test if given samples are taken from a Poisson distribution?" Tags="&lt;distributions&gt;&lt;hypothesis-testing&gt;&lt;poisson&gt;" AnswerCount="8" FavoriteCount="2" />
  <row Id="1175" PostTypeId="2" ParentId="1173" CreationDate="2010-08-03T13:55:22.063" Score="1" ViewCount="" Body="&lt;p&gt;If the data are &lt;em&gt;rates&lt;/em&gt;: that is number of successes divided by number of trials, then a very elegant method is a funnel plot. For example, see &lt;a href=&quot;http://qshc.bmj.com/content/11/4/390.2.full&quot; rel=&quot;nofollow&quot;&gt;http://qshc.bmj.com/content/11/4/390.2.full&lt;/a&gt; (apologies if the link requires a subscription--let me know and I'll find another).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It may be possible to adapt it to other types of data, but I haven't seen any examples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;UPDATE:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's a link to an example which doesn't require a subscription (and has a good explanation for how they might be used):&#xA;&lt;a href=&quot;http://understandinguncertainty.org/fertility&quot; rel=&quot;nofollow&quot;&gt;http://understandinguncertainty.org/fertility&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;They can be used for non-rate data, by simply plotting mean against standard error, however they may lose some of their simplicity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The wikipedia article is not great, as it only discusses their use in meta-analyses. I'd argue they could be useful in many other contexts.&lt;/p&gt;&#xA;" OwnerUserId="495" LastEditorUserId="495" LastEditorDisplayName="" LastEditDate="2010-08-03T15:07:17.600" LastActivityDate="2010-08-03T15:07:17.600" CommentCount="3" />
  <row Id="1176" PostTypeId="2" ParentId="1173" CreationDate="2010-08-03T13:59:03.973" Score="3" ViewCount="" Body="&lt;p&gt;Frank Harrell's (most excellent) keynote entitled &quot;Information Allergy&quot; at useR! last month showed alternatives to these: rather than hiding the raw data via the aggregation the bars provide, the raw data is also shown as dots (or points).  &quot;Why hide the data?&quot; was Frank's comment. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given alpa blending, that strikes as a most sensible suggestion (and the whole talk most full of good, and important, nuggets).&lt;/p&gt;&#xA;" OwnerUserId="334" LastActivityDate="2010-08-03T13:59:03.973" CommentCount="7" />
  <row Id="1177" PostTypeId="2" ParentId="1174" CreationDate="2010-08-03T14:14:54.700" Score="2" ViewCount="" Body="&lt;p&gt;I suppose the easiest way is just to do a chi-squared &lt;a href=&quot;http://en.wikipedia.org/wiki/Pearson%27s_chi-square_test&quot; rel=&quot;nofollow&quot;&gt;Goodness of fit&lt;/a&gt; test.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In fact here's nice &lt;a href=&quot;http://home.ubalt.edu/ntsbarsh/Business-stat/otherapplets/PoissonTest.htm&quot; rel=&quot;nofollow&quot;&gt;java applet&lt;/a&gt; that will do just that!&lt;/p&gt;&#xA;" OwnerUserId="8" LastActivityDate="2010-08-03T14:14:54.700" />
  <row Id="1178" PostTypeId="2" ParentId="1174" CreationDate="2010-08-03T14:21:43.390" Score="3" ViewCount="" Body="&lt;p&gt;You can use the dispersion (ratio of variance to the mean) as a test statistic, since the Poisson should give a dispersion of 1. &lt;a href=&quot;http://www.stats.uwo.ca/faculty/aim/2004/04-259/notes/DispersionTests.pdf&quot; rel=&quot;nofollow&quot;&gt;Here is a link&lt;/a&gt; to how to use it as a model test.&lt;/p&gt;&#xA;" OwnerUserId="378" LastActivityDate="2010-08-03T14:21:43.390" CommentCount="1" />
  <row Id="1179" PostTypeId="2" ParentId="1173" CreationDate="2010-08-03T14:26:33.223" Score="1" ViewCount="" Body="&lt;p&gt;I'm curious at to why you don't like these plots. I use them all the time. Without wanting to state the blooming obvious, they allow you to compare the means of different groups and see if their 95% CIs overlap (i.e., true mean likely to be different).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's important to get a balance of simplicity and information for different purposes, I guess. But when I use these plots I am saying- &quot;these two groups are different from each other in some important way&quot; [or not].&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Seems pretty great to me, but I'd be interested to hear counter-examples. I suppose implicit in the use of the plot is that the data do not have a bizzare distribution which renders the mean invalid or misleading.&lt;/p&gt;&#xA;" OwnerUserId="199" LastActivityDate="2010-08-03T14:26:33.223" CommentCount="1" />
  <row Id="1180" PostTypeId="2" ParentId="1174" CreationDate="2010-08-03T14:39:16.187" Score="1" ViewCount="" Body="&lt;p&gt;For a Poisson distribution, the mean equals the variance.  If your sample mean is very different from your sample variance, you probably don't have Poisson data.  The dispersion test also mentioned here is a formalization of that notion.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your variance is much larger than your mean, as is commonly the case, you might want to try a negative binomial distribution next.  &lt;/p&gt;&#xA;" OwnerUserId="319" LastActivityDate="2010-08-03T14:39:16.187" CommentCount="2" />
  <row Id="1181" PostTypeId="2" ParentId="1174" CreationDate="2010-08-03T14:52:44.720" Score="5" ViewCount="" Body="&lt;p&gt;Here is a sequence of R commands that may be helpful. Feel free to comment or edit if you spot any mistakes.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(1)&#xA;x.poi&amp;lt;-rpois(n=200,lambda=2.5) # a vector of random variables from the Poisson distr.&#xA;&#xA;hist(x.poi,main=&quot;Poisson distribution&quot;)&#xA;&#xA;lambda.est &amp;lt;- mean(x.poi) ## estimate of parameter lambda&#xA;(tab.os&amp;lt;-table(x.poi)) ## table with empirical frequencies&#xA;&#xA;&#xA;freq.os&amp;lt;-vector()&#xA;for(i in 1: length(tab.os)) freq.os[i]&amp;lt;-tab.os[[i]]  ## vector of emprical frequencies&#xA;&#xA;freq.ex&amp;lt;-(dpois(0:max(x.poi),lambda=lambda.est)*200) ## vector of fitted (expected) frequencies&#xA;&#xA;acc &amp;lt;- mean(abs(freq.os-trunc(freq.ex))) ## absolute goodness of fit index acc&#xA;acc/mean(freq.os)*100 ## relative (percent) goodness of fit index&#xA;&#xA;h &amp;lt;- hist(x.poi ,breaks=length(tab.os))&#xA;xhist &amp;lt;- c(min(h$breaks),h$breaks)&#xA;yhist &amp;lt;- c(0,h$density,0)&#xA;xfit &amp;lt;- min(x.poi):max(x.poi)&#xA;yfit &amp;lt;- dpois(xfit,lambda=lambda.est)&#xA;plot(xhist,yhist,type=&quot;s&quot;,ylim=c(0,max(yhist,yfit)), main=&quot;Poison density and histogram&quot;)&#xA;lines(xfit,yfit, col=&quot;red&quot;)&#xA;&#xA;#Perform the chi-square goodness of fit test &#xA;#In case of count data we can use goodfit() included in vcd package&#xA;library(vcd) ## loading vcd package&#xA;gf &amp;lt;- goodfit(x.poi,type= &quot;poisson&quot;,method= &quot;MinChisq&quot;)&#xA;summary(gf)&#xA;plot(gf,main=&quot;Count data vs Poisson distribution&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="339" LastActivityDate="2010-08-03T14:52:44.720" CommentCount="2" />
  <row Id="1182" PostTypeId="2" ParentId="1173" CreationDate="2010-08-03T15:08:16.867" Score="3" ViewCount="" Body="&lt;p&gt;From a psychological perspective, I advocate plotting the data plus your uncertainty about the data. Thus, in a plot like you show, I would never bother with extending the bars all the way to zero, which only serves to minimize the eye's ability to distinguish differences in the range of the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Additionally, I'm frankly anti-bargraph; bar graphs map two variables to the same aesthetic attribute (x-axis location), which can cause confusion. A better approach is to avoid redundant aesthetic mapping by mapping one variable to the x-axis and another variable to another aesthetic attribute (eg. point shape or color or both).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, in your plot above, you only include error bars above the value, which hinders one's ability to compare the intervals of uncertainty relative to bars above and below the value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's how I would plot the data (via the ggplot2 package). Note that I add lines connecting points in the same series; some argue that this is only appropriate when the series across which the lines are connected are numeric (as seems to be in this case), however as long as there is any reasonable ordinal relationship among the levels of the x-axis variable, I think connecting lines are useful for helping the eye associate points across the x-axis. This can become particularly useful for detecting interactions, which really stand out with lines.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(ggplot2)&#xA;a = data.frame(names,prevs,se)&#xA;a$let = substr(a$names,1,1)&#xA;a$num = substr(a$names,2,2)&#xA;ggplot(data = a)+&#xA;layer(&#xA;    geom = 'point'&#xA;    , mapping = aes(&#xA;        x = num&#xA;        , y = prevs&#xA;        , colour = let&#xA;        , shape = let&#xA;    )&#xA;)+&#xA;layer(&#xA;    geom = 'line'&#xA;    , mapping = aes(&#xA;        x = num&#xA;        , y = prevs&#xA;        , colour = let&#xA;        , linetype = let&#xA;        , group = let&#xA;    )    &#xA;)+&#xA;layer(&#xA;    geom = 'errorbar'&#xA;    , mapping = aes(&#xA;        x = num&#xA;        , ymin = prevs-se&#xA;        , ymax = prevs+se&#xA;        , colour = let&#xA;    )&#xA;    , alpha = .5&#xA;    , width = .5&#xA;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://img517.imageshack.us/img517/8002/tmpa.jpg&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="364" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-08-06T17:00:04.637" LastActivityDate="2010-08-06T17:00:04.637" CommentCount="5" />
  <row Id="1183" PostTypeId="2" ParentId="1173" CreationDate="2010-08-03T15:19:15.750" Score="2" ViewCount="" Body="&lt;p&gt;I would use boxplots here; clean, meaningful, nonparametric... Or &lt;a href=&quot;http://cran.r-project.org/web/packages/vioplot/index.html&quot; rel=&quot;nofollow&quot;&gt;vioplot&lt;/a&gt; if the distribution is more interesting. &lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-08-03T15:19:15.750" CommentCount="4" />
  <row Id="1184" PostTypeId="1" AcceptedAnswerId="1187" CreationDate="2010-08-03T16:26:48.983" Score="3" ViewCount="298" Body="&lt;p&gt;Could you recommend an introductory reference to index decomposition analysis, including&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;different methods (e.g. methods linked to the Laspeyre index and methods linked to the Divisa index)&lt;/li&gt;&#xA;&lt;li&gt;properties of decomposition methods which can be used to compare the different methods&lt;/li&gt;&#xA;&lt;li&gt;implementations of methods, e.g. in R&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;? Any hint appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(could not tag as index-decomposition due to missing reputation)&lt;/p&gt;&#xA;" OwnerUserId="573" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-13T08:20:20.310" LastActivityDate="2011-04-13T08:20:20.310" Title="Introduction to index decomposition analysis" Tags="&lt;index-decomposition&gt;" AnswerCount="1" />
  <row Id="1185" PostTypeId="2" ParentId="1164" CreationDate="2010-08-03T17:03:58.947" Score="14" ViewCount="" Body="&lt;p&gt;I would suggest that it's a lag in teaching. Most people either learn statistics at collage or University. If statistics is not you first degree and instead did a mathematics or computer science degree then you probably only cover the fundamental statistics modules: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Probability&lt;/li&gt;&#xA;&lt;li&gt;Hypothesis testing&lt;/li&gt;&#xA;&lt;li&gt;Regression&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;This means that when faced with a problem we try and use what we know to solve the problem. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Data isn't Normal - take logs.&lt;/li&gt;&#xA;&lt;li&gt;Data has annoying outliers - remove them.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Unless you stumble across something else, then it's difficult to do something better. It's really hard using Google to find something if you don't know what it's called!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think with all techniques, it will take awhile before the newer techniques filter down. How long did it take standard hypothesis test to be part of a standard statistics curriculum?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;BTW, with a statistics degree there will still be a lag in teaching - just a shorter one!&lt;/p&gt;&#xA;" OwnerUserId="8" LastActivityDate="2010-08-03T17:03:58.947" CommentCount="2" />
  <row Id="1186" PostTypeId="2" ParentId="1174" CreationDate="2010-08-03T17:17:31.300" Score="2" ViewCount="" Body="&lt;p&gt;You can draw a single figure in which the observed and expected frequencies are drawn side by side. If the distributions are very different and you also have a variance-mean ratio bigger than one, then a good candidate is the negative binomial. Read the section &lt;a href=&quot;http://books.google.com/books?id=8D4HVx0apZQC&amp;amp;lpg=PP1&amp;amp;dq=the%20r%20book&amp;amp;hl=el&amp;amp;pg=PA536#v=onepage&amp;amp;q&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;Frequency Distributions&lt;/a&gt; from &lt;code&gt;The R Book&lt;/code&gt;. It deals with a very similar problem.&lt;/p&gt;&#xA;" OwnerUserId="632" LastActivityDate="2010-08-03T17:17:31.300" />
  <row Id="1187" PostTypeId="2" ParentId="1184" CreationDate="2010-08-03T17:24:56.913" Score="2" ViewCount="" Body="&lt;p&gt;This thesis may be a starting place: &lt;a href=&quot;https://scholarbank.nus.edu.sg/bitstream/handle/10635/14229/GranelF.pdf?sequence=1&quot; rel=&quot;nofollow&quot;&gt;A Comparative Analysis of Index Decomposition Methods&lt;/a&gt;, by Frédéric Granel.  It should serve as a basic introduction to IDA and the Laspeyre index, but it does not include the Divisa index or any code implementing the methods.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For more detail, including the Divisa index, you might try &lt;a href=&quot;http://www.sciencedirect.com/science?_ob=ArticleURL&amp;amp;_udi=B6V2W-4WR2C4X-3&amp;amp;_user=10&amp;amp;_coverDate=11%2F30%2F2009&amp;amp;_rdoc=1&amp;amp;_fmt=high&amp;amp;_orig=search&amp;amp;_sort=d&amp;amp;_docanchor=&amp;amp;view=c&amp;amp;_searchStrId=1420130047&amp;amp;_rerunOrigin=google&amp;amp;_acct=C000050221&amp;amp;_version=1&amp;amp;_urlVersion=0&amp;amp;_userid=10&amp;amp;md5=5872e15ee97331e2e332f10a8cefb041&quot; rel=&quot;nofollow&quot;&gt;Properties and linkages of some index decomposition analysis methods&lt;/a&gt;. As for implementations in R, there seems to be no package for IDA specifically, but &lt;a href=&quot;http://cran.r-project.org/web/packages/micEcon/micEcon.pdf&quot; rel=&quot;nofollow&quot;&gt;micEcon&lt;/a&gt; and &lt;a href=&quot;http://cran.r-project.org/web/packages/micEconAids/micEconAids.pdf.&quot; rel=&quot;nofollow&quot;&gt;micEconAids&lt;/a&gt; have parts of it, by way of demand analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Best of luck.&lt;/p&gt;&#xA;" OwnerUserId="39" LastEditorUserId="39" LastEditorDisplayName="" LastEditDate="2010-08-03T17:30:07.610" LastActivityDate="2010-08-03T17:30:07.610" CommentCount="1" />
  <row Id="1188" PostTypeId="2" ParentId="1123" CreationDate="2010-08-03T19:26:55.673" Score="1" ViewCount="" Body="&lt;p&gt;Are you entirely sure that they're using the normal distribution directly?  It's very common to use transformed responses to model success rates, but this involves passing through a link function to move from a Gaussian random variable to a value in [0,1].  A commonly used transform is the probit one, which is just the inverse Gaussian CDF. (So you'd end up with something like $\Phi^{-1}(p) = X\beta + \sigma$, where $\Phi$ is the Gaussian CDF).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're actually using a normal distribution directly to model a result in [0,1], then it strikes me that the variance would have to be so small -- especially for p near 0 or 1 -- that you'd nearly always overfit the model.&lt;/p&gt;&#xA;" OwnerUserId="61" LastActivityDate="2010-08-03T19:26:55.673" />
  <row Id="1189" PostTypeId="2" ParentId="881" CreationDate="2010-08-03T19:37:30.837" Score="3" ViewCount="" Body="&lt;p&gt;You can also use Edgeworth series, if your random variable has a finite mean and variance, which expands the CDF of your random variable in terms of the Gaussian CDF.  At first glance it's not quite as tidy conceptually as using a mixture model, but the derivation is quite pretty and it gives you a closed form with very fast decay in the tail terms.&lt;/p&gt;&#xA;" OwnerUserId="61" LastActivityDate="2010-08-03T19:37:30.837" />
  <row Id="1190" PostTypeId="2" ParentId="1099" CreationDate="2010-08-03T19:51:33.140" Score="1" ViewCount="" Body="&lt;p&gt;Chi-square tests do not seem appropriate.  As others said, provided there are a reasonable number of different rates, you could treat the data as continuous and do regression or ANOVA.  You would then want to look at the distribution of the residuals.  &lt;/p&gt;&#xA;" OwnerUserId="686" LastActivityDate="2010-08-03T19:51:33.140" />
  <row Id="1191" PostTypeId="2" ParentId="1016" CreationDate="2010-08-03T19:57:01.447" Score="0" ViewCount="" Body="&lt;p&gt;I also like Rob's answer.  And, if you happen to use SAS rather than R, you can use PROC GLMSELECT for models that would be done with PROC GLM, although it works well for some other models, as well.  See&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Flom and Cassell &quot;Stopping Stepwise: Why Stepwise Selection Methods are Bad and What you Should Use&quot; presented at various groups, most recently, NESUG 2009&lt;/p&gt;&#xA;" OwnerUserId="686" LastActivityDate="2010-08-03T19:57:01.447" />
  <row Id="1192" PostTypeId="2" ParentId="1174" CreationDate="2010-08-03T20:00:30.287" Score="0" ViewCount="" Body="&lt;p&gt;Yet another way to test this is with a quantile quantile plot.  In R, there is qqplot.&lt;/p&gt;&#xA;" OwnerUserId="686" LastActivityDate="2010-08-03T20:00:30.287" />
  <row Id="1193" PostTypeId="2" ParentId="726" CreationDate="2010-08-03T20:07:57.437" Score="1" ViewCount="" Body="&lt;p&gt;A variation on the Fisher quotation given &lt;a href=&quot;http://stats.stackexchange.com/questions/726/famous-statistician-quotes/739#739&quot;&gt;here&lt;/a&gt; is&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Hiring a statistician after the data have been collected is like hiring a physician when your patient is in the morgue.  He may be able to tell you what went wrong, but he is unlikely to be able to fix it.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;But I heard this attributed to Box, not Fisher.&lt;/p&gt;&#xA;" OwnerUserId="686" LastEditorUserId="174" LastEditorDisplayName="" LastEditDate="2010-08-07T14:44:30.670" LastActivityDate="2010-08-07T14:44:30.670" CommunityOwnedDate="2010-08-03T20:07:57.437" CommentCount="1" />
  <row Id="1194" PostTypeId="1" CreationDate="2010-08-03T20:19:57.303" Score="11" ViewCount="491" Body="&lt;p&gt;This question has been bugging me for some time, and I was going to write a blog post about it. However, I think it is better left for discussion in this forum. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Back in April, I attended a talk at the UMD Math Department Statistics group seminar series called &quot;To Explain or To Predict?&quot;. The talk was given by &lt;a href=&quot;http://www.rhsmith.umd.edu/faculty/gshmueli/web/html/&quot; rel=&quot;nofollow&quot;&gt;Prof. Galit Shmueli&lt;/a&gt; who teaches at UMD's Smith Business School. Her talk was based on research she did for a paper titled &lt;a href=&quot;http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1112893&quot; rel=&quot;nofollow&quot;&gt;&quot;Predictive vs. Explanatory Modeling in IS Research&quot;&lt;/a&gt;, and a follow up working paper titled &lt;a href=&quot;http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1351252&quot; rel=&quot;nofollow&quot;&gt;&quot;To Explain or To Predict?&quot;&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Dr. Shmueli's argument is that the terms predictive and explanatory in a statistical modeling context have become conflated, and that statistical literature lacks a a thorough discussion of the differences. In the paper, she contrasts both and talks about their practical implications. I encourage you to read the papers. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The question I'd like to pose to the practitioner community is: have you ever fallen into the trap of using one when meaning to use the other? I certainly have. How do you know which one to use? How do you define a predictive excercise vs an exlanatory/descriptive one? It would be useful if you could talk about the specific application. &lt;/p&gt;&#xA;" OwnerUserId="11" LastEditorUserId="11" LastEditorDisplayName="" LastEditDate="2010-08-03T21:15:08.433" LastActivityDate="2011-03-15T02:51:12.593" CommunityOwnedDate="2010-08-04T22:36:49.690" Title="Practical thoughts on explanatory vs. predictive modeling" Tags="&lt;predictive-models&gt;" AnswerCount="5" CommentCount="12" FavoriteCount="4" />
  <row Id="1195" PostTypeId="1" CreationDate="2010-08-03T20:23:34.770" Score="5" ViewCount="76" Body="&lt;p&gt;Many income surveys (especially older ones) truncate key variables, such as household income, at some arbitrary point, to protect confidentiality. This point changes over time. This reduces inequality measures associated with the variable. I am interested in fitting a Pareto tail to the truncated distribution, replacing truncated values with imputed values to mimic the actual distribution. What's the best way to do this?&lt;/p&gt;&#xA;" OwnerUserId="687" LastActivityDate="2011-05-12T17:12:20.963" Title="How can I apply a Pareto tail to a truncated distribution?" Tags="&lt;pareto-distribution&gt;&lt;data-imputation&gt;" AnswerCount="2" />
  <row Id="1196" PostTypeId="2" ParentId="1164" CreationDate="2010-08-03T20:26:23.000" Score="9" ViewCount="" Body="&lt;p&gt;Statistics is a tool for non-statistical-minded researchers, and they just don't care.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I once tried to help with a Medicine article my ex-wife was co-authoring. I wrote several pages describing the data, what it suggested, why certain observations had been excluded from the study... and the lead researcher, a doctor, threw it all away and asked someone to compute a p-value, which is all she (and just about everyone who would read the article) cared about.&lt;/p&gt;&#xA;" OwnerUserId="666" LastActivityDate="2010-08-03T20:26:23.000" />
  <row Id="1197" PostTypeId="2" ParentId="1194" CreationDate="2010-08-03T20:39:20.213" Score="1" ViewCount="" Body="&lt;p&gt;I haven't read her work beyond the abstract of the linked paper, but my sense is that the distinction between &quot;explanation&quot; and &quot;prediction&quot; should be thrown away and replaced with the distinction between the aims of the practitioner, which are either &quot;causal&quot; or &quot;predictive&quot;. In general, I think &quot;explanation&quot; is such a vague word that it means nearly nothing. For example, is Hooke's Law explanatory or predictive? On the other end of the spectrum, are predictively accurate recommendation systems good causal models of explicit item ratings? I think we all share the intuition that the goal of science is explanation, while the goal of technology is prediction; and this intuition somehow gets lost in consideration of the tools we use, like supervised learning algorithms, that can be employed for both causal inference and predictive modeling, but are really purely mathematical devices that are not intrinsically linked to &quot;prediction&quot; or &quot;explanation&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having said all of that, maybe the only word that I would apply to a model is interpretable. Regressions are usually interpretable; neural nets with many layers are often not so. I think people sometimes naively assume that a model that is interpretable is providing causal information, while uninterpretable models only provide predictive information. This attitude seems simply confused to me.&lt;/p&gt;&#xA;" OwnerUserId="303" LastActivityDate="2010-08-03T20:39:20.213" CommunityOwnedDate="2010-08-04T22:36:49.690" />
  <row Id="1198" PostTypeId="2" ParentId="652" CreationDate="2010-08-03T21:19:22.083" Score="2" ViewCount="" Body="&lt;p&gt;This book is dynamite:&#xA;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0471093157&quot; rel=&quot;nofollow&quot;&gt;George E. P. Box, Statistics for Experimenters: An Introduction to Design, Data Analysis, and Model Building&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It starts from zero knowledge of Statistics but it doesn't insult the reader's intelligence. It's incredibly practical but with no loss of rigour; in fact, it underscores the danger of ignoring underlying assumptions (which are often false in real life) of common tests.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's out of print but it's very easy to find a copy. Follow the link for a few options.&lt;/p&gt;&#xA;" OwnerUserId="666" LastEditorUserId="666" LastEditorDisplayName="" LastEditDate="2010-08-03T21:39:31.527" LastActivityDate="2010-08-03T21:39:31.527" CommunityOwnedDate="2010-08-03T21:19:22.083" CommentCount="1" />
  <row Id="1199" PostTypeId="2" ParentId="1194" CreationDate="2010-08-03T21:30:22.343" Score="3" ViewCount="" Body="&lt;p&gt;as others have already said, the distinction is somewhat meaningless, except in so far as the aims of the researcher are concerned. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Brad Efron, one of the commentators on &lt;a href=&quot;http://www.stat.osu.edu/~bli/dmsl/papers/Breiman.pdf&quot; rel=&quot;nofollow&quot;&gt;The Two Cultures&lt;/a&gt; paper, made the following observation (as discussed &lt;a href=&quot;http://stats.stackexchange.com/questions/6/the-two-cultures-statistics-vs-machine-learning&quot;&gt;in my earlier question&lt;/a&gt;):&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Prediction by itself is&#xA;  only occasionally sufficient. The post&#xA;  office is happy with any method that&#xA;  predicts correct addresses from&#xA;  hand-written scrawls. Peter&#xA;  Gregory undertook his study for&#xA;  prediction purposes, but also to&#xA;  better understand the medical basis of&#xA;  hepatitis. Most statistical surveys&#xA;  have the identification of causal&#xA;  factors as their ultimate goal.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Certain fields (eg. Medicine) place a heavy weight on model fitting as explanatory process (the distribution, etc.), as a means to understanding the underlying process that generates the data. Other fields are less concerned with this, and will be happy with a &quot;black box&quot; model that has a very high predictive success. This can work its way into the model building process as well.  &lt;/p&gt;&#xA;" OwnerUserId="5" LastEditorUserId="5" LastEditorDisplayName="" LastEditDate="2010-08-03T21:46:47.963" LastActivityDate="2010-08-03T21:46:47.963" CommunityOwnedDate="2010-08-04T22:36:49.690" />
  <row Id="1200" PostTypeId="2" ParentId="1194" CreationDate="2010-08-03T21:32:40.813" Score="3" ViewCount="" Body="&lt;p&gt;I am still a bit unclear as to what the question is. Having said that, to my mind the fundamental difference between predictive and explanatory models is the difference in their focus.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Explanatory Models&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By definition explanatory models have as their primary focus the goal of explaining something in the real world. In most instances, we seek to offer simple and clean explanations. By simple I mean that we prefer parsimony (explain the phenomena with as few parameters as possible) and by clean I mean that we would like to make statements of the following form: &quot;the effect of changing $x$ by one unit changes $y$ by $\beta$ holding everything else constant&quot;. Given these goals of simple and clear explanations, explanatory models seek to penalize complex models (by using appropriate criteria such as AIC) and prefer to obtain orthogonal independent variables (either via controlled experiments or via suitable data transformations).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Predictive Models&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The goal of predictive models is to predict something. Thus, they tend to focus less on parsimony or simplicity but more on their ability to predict the dependent variable. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the above is somewhat of an artificial distinction as explanatory models can be used for prediction and sometimes predictive models can explain something.&lt;/p&gt;&#xA;" LastActivityDate="2010-08-03T21:32:40.813" CommunityOwnedDate="2010-08-04T22:36:49.690" />
  <row Id="1201" PostTypeId="2" ParentId="11" CreationDate="2010-08-03T21:50:09.007" Score="1" ViewCount="" Body="&lt;p&gt;Sorry, no quick answer. There are thick books dedicated to answering this question. Here's a 600-page long example: &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1441929185&quot; rel=&quot;nofollow&quot;&gt;Harrell's Regression Modeling Strategies&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="666" LastActivityDate="2010-08-03T21:50:09.007" />
  <row Id="1202" PostTypeId="1" CreationDate="2010-08-03T22:09:33.840" Score="1" ViewCount="153" Body="&lt;p&gt;The question in short: What methods can be used to quantify distributional relationships between data when the distribution is un-known?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now the longer story: I have a list of distributions and would like to rank them based on their similarity to a given base-line distribution. Correlation jumps into my mind in such a case and the Spearman correlation coefficient in particular given that it does not make any distributional assumptions. However, I would actually need to create the coefficient based on binned data (as this is done for histograms or densities) rather than the raw data and I don't know if this is actually a valid step or if I am just manufacturing data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words, if I have a 10000 point data set for each distribution, I would first create a binned distribution for each were each bin is of equal width and contains the frequencies of how many points each bin has. Just the way this is done for density plots or histograms. Each bin is on a discrete scale. The data is actually computer screen coordinate data and values are between 1 and 1024. Each pixel position could represent a bin (but larger bins are possible e.g. every 5 pixel being one bin). I would then compare the sequence of bins with each other rather than the raw data. The data set would look like this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;bins: 1 2 3 4 .... 1024&lt;br&gt;&#xA;dist#base:1 2 2 3 ..... 3&lt;br&gt;&#xA;dist#1: 1 4 5 5 3&lt;br&gt;&#xA;dist#2: 2 2 3 5 6&lt;br&gt;&#xA;...&lt;br&gt;&#xA;dist#1000: 1 2 4 6 6&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does this make sense? Are there better ways of doing that? Are there better statistical methods? The goal of all this is to first) test how close are distributions from measure A to measure B and second) if I can predict one, if the other is missing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Best,&#xA;Ampleforth&lt;/p&gt;&#xA;" OwnerUserId="608" LastActivityDate="2010-08-08T20:38:09.240" Title="Ranking distributional data by similarity" Tags="&lt;distributions&gt;" AnswerCount="3" />
  <row Id="1203" PostTypeId="2" ParentId="608" CreationDate="2010-08-03T22:23:31.813" Score="3" ViewCount="" Body="&lt;p&gt;In an attempt to partially answer my own question, I read &lt;a href=&quot;http://stats.stackexchange.com/questions/577/is-there-any-reason-to-prefer-the-aic-or-bic-over-the-other&quot;&gt;Wikipedia's&lt;/a&gt; description of leave-one-out cross validation &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;involves using a single observation&#xA;  from the original sample as the&#xA;  validation data, and the remaining&#xA;  observations as the training data.&#xA;  This is repeated such that each&#xA;  observation in the sample is used once&#xA;  as the validation data.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In R code, I suspect that that would mean something like this...&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;resid &amp;lt;- rep(NA, Nobs) &#xA;for (lcv in 1:Nobs)&#xA;    {&#xA;        data.loo &amp;lt;- data[-lcv,] #drop the data point that will be used for validation&#xA;        loo.model &amp;lt;- lm(y ~ a+b,data=data.loo) #construct a model without that data point&#xA;            resid[lcv] &amp;lt;- data[lcv,&quot;y&quot;] - (coef(loo.model)[1] + coef(loo.model)[2]*data[lcv,&quot;a&quot;]+coef(loo.model)[3]*data[lcv,&quot;b&quot;]) #compare the observed value to the value predicted by the loo model for each possible observation, and store that value&#xA;    }&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;... is supposed to yield values in resid that is related to the AIC.  In practice the sum of squared residuals from each iteration of the LOO loop detailed above is a good predictor of the AIC for the notable.seeds, r^2 = .9776.  But, &lt;a href=&quot;http://stats.stackexchange.com/questions/577/is-there-any-reason-to-prefer-the-aic-or-bic-over-the-other&quot;&gt;elsewhere&lt;/a&gt; a contributor suggested that LOO should be asymptotically equivalent to the AIC (at least for linear models), so I'm a little disappointed that r^2 isn't closer to 1.  Obviously this isn't really an answer - more like additional code to try to encourage someone to try to provide a better answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Addendum:  Since AIC and BIC for models of fixed sample size only vary by a constant, the correlation of BIC to squared residuals is the same as the correaltion of AIC to squared residuals, so the approach I took above appears to be fruitless.&lt;/p&gt;&#xA;" OwnerUserId="196" LastEditorUserId="196" LastEditorDisplayName="" LastEditDate="2010-08-03T22:58:09.447" LastActivityDate="2010-08-03T22:58:09.447" CommentCount="2" />
  <row Id="1204" PostTypeId="2" ParentId="1202" CreationDate="2010-08-03T22:25:14.913" Score="0" ViewCount="" Body="&lt;p&gt;Edit: I misunderstood the question at first. Your observations are actually paired. Sample1 Bin1 to Baseline Bin1 etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you could do is take the difference between sample and baseline for each bin, then use the Wilcoxon signed rank statistic on the differences.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Wilcoxon_signed-rank_test&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Wilcoxon_signed-rank_test&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;D is your sequence of differences, then &lt;/li&gt;&#xA;&lt;li&gt;R is the ranks of |D|, and &lt;/li&gt;&#xA;&lt;li&gt;psi = 0 if D &amp;lt; 0, 1 if D &gt; 0. &lt;/li&gt;&#xA;&lt;li&gt;W = Sum(psi*R)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In R&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;wilcox.test(sample1-baseline)$statistic&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="287" LastEditorUserId="287" LastEditorDisplayName="" LastEditDate="2010-08-03T22:39:31.763" LastActivityDate="2010-08-03T22:39:31.763" />
  <row Id="1205" PostTypeId="1" CreationDate="2010-08-03T22:56:48.420" Score="1" ViewCount="563" Body="&lt;p&gt;I want to perform a two-sample T-test to test for a difference between two independent samples which each sample abides by the assumptions of the T-test (each distribution can be assumed to be independent and identically distributed as Normal with equal variance). The only complication from the basic two-sample T-test is that the data is weighted. I am using weighted means and standard deviations, but weighted N's will artificially inflate the size of the sample, hence bias the result. Is it simply a case of replacing the weighted Ns with the unweighted Ns?&lt;/p&gt;&#xA;" OwnerUserId="366" LastActivityDate="2010-08-04T15:57:30.210" Title="Two-sample T-test with weighted data" Tags="&lt;t-test&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="1206" PostTypeId="2" ParentId="1194" CreationDate="2010-08-03T23:36:08.593" Score="5" ViewCount="" Body="&lt;p&gt;One practical issue that arises here is variable selection in modelling. A variable can be an important explanatory variable (e.g., is statistically significant) but may not be useful for predictive purposes (i.e., its inclusion in the model leads to worse predictive accuracy). I see this mistake almost every day in published papers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another difference is in the distinction between principal components analysis and factor analysis. PCA is often used in prediction, but is not so useful for explanation. FA involves the additional step of rotation which is done to improve interpretation (and hence explanation). There is a &lt;a href=&quot;http://blog.bzst.com/2010/08/pca-debate.html&quot; rel=&quot;nofollow&quot;&gt;nice post today on Galit Shmueli's blog about this&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Update: a third case arises in time series when a variable may be an important explanatory variable but it just isn't available for the future. For example, home loans may be strongly related to GDP but that isn't much use for predicting future home loans unless we also have good predictions of GDP.&lt;/p&gt;&#xA;" OwnerUserId="159" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-08-04T05:17:43.387" LastActivityDate="2010-08-04T05:17:43.387" CommunityOwnedDate="2010-08-04T22:36:49.690" CommentCount="4" />
  <row Id="1207" PostTypeId="1" CreationDate="2010-08-04T00:32:13.360" Score="5" ViewCount="419" Body="&lt;p&gt;This post is the continuation of another post related to a &lt;a href=&quot;http://stats.stackexchange.com/questions/1142/simple-algorithm-for-online-outlier-detection-of-a-generic-time-series&quot;&gt;generic method for outlier detection in time series&lt;/a&gt;.&#xA;Basically, at this point I'm interested in a robust way to discover the periodicity/seasonality of a generic time series affected by a lot of noise.&#xA;From a developer point of view, I would like a simple interface such as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;unsigned int discover_period(vector v);&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where v is the array containing the samples, and the return value is the period of the signal.&#xA;The main point is that, again, I can't make any assumption regarding the analyzed signal.&#xA;I already tried an approach based on the signal autocorrelation (detecting the peaks of a correlogram), but it's not robust as I would like.&lt;/p&gt;&#xA;" OwnerUserId="667" LastActivityDate="2010-08-10T18:41:10.997" Title="Period detection of a generic time series" Tags="&lt;time-series&gt;&lt;algorithms&gt;&lt;real-time&gt;" AnswerCount="4" FavoriteCount="6" />
  <row Id="1208" PostTypeId="2" ParentId="1173" CreationDate="2010-08-04T01:03:35.623" Score="0" ViewCount="" Body="&lt;p&gt;@Chris check this out about interpreting overlapping CIs &lt;a href=&quot;http://pubs.amstat.org/doi/abs/10.1198/000313001317097960&quot; rel=&quot;nofollow&quot;&gt;http://pubs.amstat.org/doi/abs/10.1198/000313001317097960&lt;/a&gt;&#xA;Also the original question is also around the confusion of using SE or SD interchangeably while they are two different things&lt;/p&gt;&#xA;" OwnerUserId="694" LastActivityDate="2010-08-04T01:03:35.623" />
  <row Id="1209" PostTypeId="2" ParentId="223" CreationDate="2010-08-04T01:07:14.210" Score="0" ViewCount="" Body="&lt;p&gt;I recommend &quot;an introduction to medical statistics&quot; by Martin Bland, Oxford University Press&lt;/p&gt;&#xA;" OwnerUserId="694" LastActivityDate="2010-08-04T01:07:14.210" />
  <row Id="1210" PostTypeId="2" ParentId="1133" CreationDate="2010-08-04T01:26:59.243" Score="5" ViewCount="" Body="&lt;p&gt;You should look into &quot;partitioning chi-squared&quot;.  This is similar in logic to performing post-hoc tests in ANOVA.  It will allow you to determine whether your significant overall test is primarily attributable to differences in particular categories or groups of categories.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A quick google turned up this presentation, which at the end discusses methods for partitioning chi-squared.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.ed.uiuc.edu/courses/EdPsy490AT/lectures/2way_chi-ha-online.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.ed.uiuc.edu/courses/EdPsy490AT/lectures/2way_chi-ha-online.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="485" LastActivityDate="2010-08-04T01:26:59.243" CommentCount="4" />
  <row Id="1211" PostTypeId="2" ParentId="770" CreationDate="2010-08-04T02:05:29.027" Score="2" ViewCount="" Body="&lt;p&gt;It's like that old joke. When asked for directions the philosopher said &quot;Well, if I wanted to go there, I wouldn't start from here ...&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While I think each &quot;culture&quot; should be open to learning from the other, they have different ways of looking at the world.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think the problem with learning statistics through studying machine learning algorithms is that, whilst ML algorithms start with statistical concepts, statistics doesn't start with algorithms, but probability models.&lt;/p&gt;&#xA;" OwnerUserId="521" LastActivityDate="2010-08-04T02:05:29.027" />
  <row Id="1212" PostTypeId="2" ParentId="1174" CreationDate="2010-08-04T03:23:21.247" Score="2" ViewCount="" Body="&lt;p&gt;First of all my advice is you must refrain from trying out a Poisson distribution just as it is to the data. I suggest you must first make a theory as to why should Poisson distribution fit a particular dataset or a phenomenon. Once you have established this, the next question is whether the distribution is homogeneous or not. this means whether all parts of the data are handled by the same poisson distribution or is their a variation in this based on some aspect like time or space. Once you have convinced of these aspects, try the following three tests &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;likelihood ratio test using a chi square variable &lt;/li&gt;&#xA;&lt;li&gt;use of conditional chi-square statistic, this test is called the poisson dispersion test or the variance test&lt;/li&gt;&#xA;&lt;li&gt;use of the neyman-scott statistic, this is based on a variance stabilizing transform of the poisson variable.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;search for these and you will find them easily on the net. &lt;/p&gt;&#xA;" OwnerUserId="387" LastActivityDate="2010-08-04T03:23:21.247" CommentCount="1" />
  <row Id="1213" PostTypeId="2" ParentId="138" CreationDate="2010-08-04T03:28:53.913" Score="11" ViewCount="" Body="&lt;p&gt;Some useful R links (find out the link that suits you):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://had.co.nz/plyr/plyr-intro-090510.pdf&quot; rel=&quot;nofollow&quot;&gt;http://had.co.nz/plyr/plyr-intro-090510.pdf&lt;/a&gt; for data manipulation&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://www.stats.ox.ac.uk/~ruth/RCourse/Rcourse3.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.stats.ox.ac.uk/~ruth/RCourse/Rcourse3.pdf&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://cran.r-project.org/doc/contrib/usingR.pdf&quot; rel=&quot;nofollow&quot;&gt;http://cran.r-project.org/doc/contrib/usingR.pdf&lt;/a&gt; for R basics&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://www.ats.ucla.edu/stat/r/dae/default.htm&quot; rel=&quot;nofollow&quot;&gt;http://www.ats.ucla.edu/stat/r/dae/default.htm&lt;/a&gt; with annotated outputs in R&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://cran.r-project.org/doc/contrib/Rossiter-RIntro-ITC.pdf&quot; rel=&quot;nofollow&quot;&gt;http://cran.r-project.org/doc/contrib/Rossiter-RIntro-ITC.pdf&lt;/a&gt; tutorial with info on plots&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://www.statmethods.net/stats/regression.html&quot; rel=&quot;nofollow&quot;&gt;http://www.statmethods.net/stats/regression.html&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://www.rmetrics.org/&quot; rel=&quot;nofollow&quot;&gt;http://www.rmetrics.org/&lt;/a&gt; provides an Open Source framework for Financial Analysis.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://www.econ.uiuc.edu/~econ472/e-Tutorial.html&quot; rel=&quot;nofollow&quot;&gt;http://www.econ.uiuc.edu/~econ472/e-Tutorial.html&lt;/a&gt; has lecture notes with R code&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;A brief guide to R and Economics &lt;a href=&quot;http://people.su.se/~ma/R_intro/R_intro.pdf&quot; rel=&quot;nofollow&quot;&gt;http://people.su.se/~ma/R_intro/R_intro.pdf&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://www.stat.pitt.edu/stoffer/tsa2/index.html&quot; rel=&quot;nofollow&quot;&gt;http://www.stat.pitt.edu/stoffer/tsa2/index.html&lt;/a&gt; has a good beginner’s tutorial for Time Series&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://www.quantmod.com/&quot; rel=&quot;nofollow&quot;&gt;http://www.quantmod.com/&lt;/a&gt; provides a great analysis and visualization framework for quantitative trading&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://www.wise.xmu.edu.cn/2007summerworkshop/download/Advanced%20Topics%20in%20Time%20Series%20Econometrics%20Using%20R1_ZongwuCAI.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.wise.xmu.edu.cn/2007summerworkshop/download/Advanced%20Topics%20in%20Time%20Series%20Econometrics%20Using%20R1_ZongwuCAI.pdf&lt;/a&gt; advanced time series in R&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Interesting time series packages in R &lt;a href=&quot;http://robjhyndman.com/software&quot; rel=&quot;nofollow&quot;&gt;http://robjhyndman.com/software&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;A Data Mining tool in R &lt;a href=&quot;http://rattle.togaware.com/&quot; rel=&quot;nofollow&quot;&gt;http://rattle.togaware.com/&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;An online e-book for Data Mining with R &lt;a href=&quot;http://www.liaad.up.pt/~ltorgo/DataMiningWithR/&quot; rel=&quot;nofollow&quot;&gt;http://www.liaad.up.pt/~ltorgo/DataMiningWithR/&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Advanced Statistics using R &lt;a href=&quot;http://www.statmethods.net/advstats/index.html&quot; rel=&quot;nofollow&quot;&gt;http://www.statmethods.net/advstats/index.html&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Guide to Credit Scoring using R &lt;a href=&quot;http://cran.r-project.org/doc/contrib/Sharma-CreditScoring.pdf&quot; rel=&quot;nofollow&quot;&gt;http://cran.r-project.org/doc/contrib/Sharma-CreditScoring.pdf&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://addictedtor.free.fr/graphiques/&quot; rel=&quot;nofollow&quot;&gt;http://addictedtor.free.fr/graphiques/&lt;/a&gt; is a graph gallery of R plots and charts with supporting code&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;A tutorial for Lattice &lt;a href=&quot;http://osiris.sunderland.ac.uk/~cs0her/Statistics/UsingLatticeGraphicsInR.htm&quot; rel=&quot;nofollow&quot;&gt;http://osiris.sunderland.ac.uk/~cs0her/Statistics/UsingLatticeGraphicsInR.htm&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Ggplot R graphics &lt;a href=&quot;http://had.co.nz/ggplot2/&quot; rel=&quot;nofollow&quot;&gt;http://had.co.nz/ggplot2/&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Ggplot Vs Lattice @ &lt;a href=&quot;http://had.co.nz/ggplot/vs-lattice.html&quot; rel=&quot;nofollow&quot;&gt;http://had.co.nz/ggplot/vs-lattice.html&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Multiple tutorials for using ggplot2 and Lattice &lt;a href=&quot;http://learnr.wordpress.com/tag/ggplot2/&quot; rel=&quot;nofollow&quot;&gt;http://learnr.wordpress.com/tag/ggplot2/&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Introduction to the Text Mining package in R &lt;a href=&quot;http://cran.r-project.org/web/packages/tm/vignettes/tm.pdf&quot; rel=&quot;nofollow&quot;&gt;http://cran.r-project.org/web/packages/tm/vignettes/tm.pdf&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Social Network Analysis &lt;a href=&quot;http://www.r-project.org/conferences/useR-2008/slides/Bojanowski.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.r-project.org/conferences/useR-2008/slides/Bojanowski.pdf&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Web Scraping in R &lt;a href=&quot;http://www.programmingr.com/content/webscraping-using-readlines-and-rcurl&quot; rel=&quot;nofollow&quot;&gt;http://www.programmingr.com/content/webscraping-using-readlines-and-rcurl&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://learnr.wordpress.com/2009/10/06/export-data-frames-to-multi-worksheet-excel-file/&quot; rel=&quot;nofollow&quot;&gt;http://learnr.wordpress.com/2009/10/06/export-data-frames-to-multi-worksheet-excel-file/&lt;/a&gt; to embed R data frames in Excel via multiple approaches.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://www.statconn.com/&quot; rel=&quot;nofollow&quot;&gt;http://www.statconn.com/&lt;/a&gt; provides a tool to make R usable from Excel&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Connect to MySQL from R &lt;a href=&quot;http://erikvold.com/blog/index.cfm/2008/8/20/how-to-connect-to-mysql-with-r-in-wndows-using-rmysql&quot; rel=&quot;nofollow&quot;&gt;http://erikvold.com/blog/index.cfm/2008/8/20/how-to-connect-to-mysql-with-r-in-wndows-using-rmysql&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://www.statmethods.net/input/importingdata.html&quot; rel=&quot;nofollow&quot;&gt;http://www.statmethods.net/input/importingdata.html&lt;/a&gt; provides info about pulling data from SAS, STATA, SPSS, etc.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Thematic Maps with R &lt;a href=&quot;http://stackoverflow.com/questions/1260965/developing-geographic-thematic-maps-with-r&quot;&gt;http://stackoverflow.com/questions/1260965/developing-geographic-thematic-maps-with-r&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://smartdatacollective.com/Home/22052&quot; rel=&quot;nofollow&quot;&gt;http://smartdatacollective.com/Home/22052&lt;/a&gt; for geographic maps in R&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Google Charts with R &lt;a href=&quot;http://www.iq.harvard.edu/blog/sss/archives/2008/04/google_charts_f_1.shtml&quot; rel=&quot;nofollow&quot;&gt;http://www.iq.harvard.edu/blog/sss/archives/2008/04/google_charts_f_1.shtml&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Introduction to using RGoogleMaps @ &lt;a href=&quot;http://cran.r-project.org/web/packages/RgoogleMaps/vignettes/RgoogleMaps-intro.pdf&quot; rel=&quot;nofollow&quot;&gt;http://cran.r-project.org/web/packages/RgoogleMaps/vignettes/RgoogleMaps-intro.pdf&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://www.stat.uni-muenchen.de/~leisch/Sweave/&quot; rel=&quot;nofollow&quot;&gt;http://www.stat.uni-muenchen.de/~leisch/Sweave/&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;R2HTML &lt;a href=&quot;http://www.feferraz.net/en/P/R2HTML&quot; rel=&quot;nofollow&quot;&gt;http://www.feferraz.net/en/P/R2HTML&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Poor Man GUI for R &lt;a href=&quot;http://wiener.math.csi.cuny.edu/pmg/&quot; rel=&quot;nofollow&quot;&gt;http://wiener.math.csi.cuny.edu/pmg/&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;R Commander is a robust GUI for R &lt;a href=&quot;http://socserv.mcmaster.ca/jfox/Misc/Rcmdr/installation-notes.html&quot; rel=&quot;nofollow&quot;&gt;http://socserv.mcmaster.ca/jfox/Misc/Rcmdr/installation-notes.html&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;JGR is a Java-based GUI for R &lt;a href=&quot;http://jgr.markushelbig.org/Screenshots.html&quot; rel=&quot;nofollow&quot;&gt;http://jgr.markushelbig.org/Screenshots.html&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Tinn-R makes for a good R editor &lt;a href=&quot;http://www.sciviews.org/Tinn-R/&quot; rel=&quot;nofollow&quot;&gt;http://www.sciviews.org/Tinn-R/&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;An Eclipse plugin for R @ &lt;a href=&quot;http://www.walware.de/goto/statet&quot; rel=&quot;nofollow&quot;&gt;http://www.walware.de/goto/statet&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Instructions to install StatET in Eclipse &lt;a href=&quot;http://www.splusbook.com/Rintro/R_Eclipse_StatET.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.splusbook.com/Rintro/R_Eclipse_StatET.pdf&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Komodo Edit R editor &lt;a href=&quot;http://www.sciviews.org/SciViews-K/index.html&quot; rel=&quot;nofollow&quot;&gt;http://www.sciviews.org/SciViews-K/index.html&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://www.omegahat.org/&quot; rel=&quot;nofollow&quot;&gt;http://www.omegahat.org/&lt;/a&gt; has a very interesting list of packages that is seriously worth a look&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Commercial versions of R @ &lt;a href=&quot;http://www.revolution-computing.com/&quot; rel=&quot;nofollow&quot;&gt;http://www.revolution-computing.com/&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;A very informative blog &lt;a href=&quot;http://blog.revolution-computing.com/&quot; rel=&quot;nofollow&quot;&gt;http://blog.revolution-computing.com/&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Red R for R tasks &lt;a href=&quot;http://code.google.com/p/r-orange/&quot; rel=&quot;nofollow&quot;&gt;http://code.google.com/p/r-orange/&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;KNIME for R &lt;a href=&quot;http://www.knime.org/introduction/screenshots&quot; rel=&quot;nofollow&quot;&gt;http://www.knime.org/introduction/screenshots&lt;/a&gt; and is worth a serious look.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="387" LastEditorUserId="509" LastEditorDisplayName="" LastEditDate="2010-08-08T12:51:01.410" LastActivityDate="2010-08-08T12:51:01.410" CommunityOwnedDate="2010-08-04T03:28:53.913" />
  <row Id="1214" PostTypeId="2" ParentId="1207" CreationDate="2010-08-04T05:41:02.697" Score="7" ViewCount="" Body="&lt;p&gt;If you really have no idea what the periodicity is, probably the best approach is to find the frequency corresponding to the maximum of the spectral density. However, the spectrum at low frequencies will be affected by trend, so you need to detrend the series first. The following R function should do the job for most series. It is far from perfect, but I've tested it on a few dozen examples and it seems to work ok. It will return 1 for data that have no strong periodicity, and the length of period otherwise.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update:&lt;/strong&gt; Version 2 of function. This is much faster and seems to be more robust.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;find.freq &amp;lt;- function(x)&#xA;{&#xA;    n &amp;lt;- length(x)&#xA;    spec &amp;lt;- spec.ar(c(x),plot=FALSE)&#xA;    if(max(spec$spec)&amp;gt;10) # Arbitrary threshold chosen by trial and error.&#xA;    {&#xA;        period &amp;lt;- round(1/spec$freq[which.max(spec$spec)])&#xA;        if(period==Inf) # Find next local maximum&#xA;        {&#xA;            j &amp;lt;- which(diff(spec$spec)&amp;gt;0)&#xA;            if(length(j)&amp;gt;0)&#xA;            {&#xA;                nextmax &amp;lt;- j[1] + which.max(spec$spec[j[1]:500])&#xA;                period &amp;lt;- round(1/spec$freq[nextmax])&#xA;            }&#xA;            else&#xA;                period &amp;lt;- 1&#xA;        }&#xA;    }&#xA;    else&#xA;        period &amp;lt;- 1&#xA;    return(period)&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="159" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-08-05T02:59:12.767" LastActivityDate="2010-08-05T02:59:12.767" CommentCount="6" />
  <row Id="1215" PostTypeId="1" AcceptedAnswerId="1220" CreationDate="2010-08-04T05:41:51.087" Score="1" ViewCount="73" Body="&lt;p&gt;Statistical analysis is not my field, so forgive me if this is a very stupid question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a list of sold items by size. Shoes in this case&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Size    Qty&#xA;35  2&#xA;36  1&#xA;37  4&#xA;38  4&#xA;39  32&#xA;40  17&#xA;41  23&#xA;42  57&#xA;43  95&#xA;44  90&#xA;45  98&#xA;46  33&#xA;47  16&#xA;48  4&#xA;total:  476&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have to tell the owner how much of every size to buy. The problem is, I can't say him.&#xA; - You should buy 95 shoes size 43 for every one of size 36...   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The usual practice is to buy the whole size curve and buy extras for the most selling sizes.&lt;br&gt;&#xA;This is about a year worth of data. &#xA;How should I present this information in an easy to understand way?&#xA;What I want to present is a general rule. Something like &quot;for every size curve, you should buy x additional shoes of size x&quot;.&lt;br&gt;&#xA;The idea would be to later apply this approach to other clothing items.&lt;/p&gt;&#xA;" OwnerUserId="698" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-16T07:06:32.820" LastActivityDate="2010-09-16T07:06:32.820" Title="Analysis of sells (what to buy)" Tags="&lt;data-visualization&gt;" AnswerCount="2" />
  <row Id="1216" PostTypeId="2" ParentId="1215" CreationDate="2010-08-04T06:12:22.350" Score="3" ViewCount="" Body="&lt;p&gt;I suggest he stocks $y_s$ pairs of shoes of size $s$ where $y_s$ is chosen so that the probability of running out of stock before the next delivery is set to some acceptable level (e.g., 5%).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems reasonable to assume $y_s$ is Poisson with rate $\lambda_s$. You can estimate $\lambda_s$ as the average sales of that size over the last few delivery periods. Then all that remains is to find the 95th percentile of the Poisson distribution with mean equal to the estimated rate for each size.&lt;/p&gt;&#xA;" OwnerUserId="159" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-08-04T08:13:41.147" LastActivityDate="2010-08-04T08:13:41.147" CommentCount="2" />
  <row Id="1217" PostTypeId="2" ParentId="1194" CreationDate="2010-08-04T06:16:41.453" Score="2" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Example:&lt;/strong&gt; A classic example that I have seen is in the context of predicting human performance.&#xA;Self-efficacy (i.e., the degree to which a person thinks that they can perform a task well) is often a strong predictor of task performance. Thus, if you put self-efficacy into a multiple regression along with other variables such as intelligence and degree of prior experience, you often find that self-efficacy is a strong predictor.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This has lead some researchers to suggest that self-efficacy causes task performance. And that effective interventions are those which focus on increasing a person's sense of self-efficacy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the alternative theoretical model sees self-efficacy largely as a consequence of task performance. I.e., If you are good, you'll know it. In this framework interventions should focus on increasing actual competence and not perceived competence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, including a variable like self-efficacy might increase prediction, but assuming you adopt the self-efficacy-as-consequence model, it should not be included as a predictor if the aim of the model is to elucidate causal processes influencing performance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This of course raises the issue of how to develop and validate a causal theoretical model. This clearly relies on multiple studies, ideally with some experimental manipulation, and a coherent argument about dynamic processes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Proximal versus distal&lt;/strong&gt;: I've seen similar issues when researchers are interested in the effects of distal and proximal causes. Proximal causes tend to predict better than distal causes. However, theoretical interest may be in understanding the ways in which distal and proximal causes operate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Variable selection issue&lt;/strong&gt;: Finally, a huge issue in social science research is the variable selection issue.&#xA;In any given study, there is an infinite number of variables that could have been measured&#xA; but weren't. Thus, interpretation of models need to consider the implications of this when making theoretical interpretations.&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-08-04T06:16:41.453" CommunityOwnedDate="2010-08-04T22:36:49.690" />
  <row Id="1218" PostTypeId="2" ParentId="459" CreationDate="2010-08-04T06:51:12.213" Score="0" ViewCount="" Body="&lt;p&gt;I would strongly enjoin you to avoid red as an indicator: there are many sorts of colour-deficiency that make this choice problematic (see eg &lt;a href=&quot;http://en.wikipedia.org/wiki/Color_blindness#Design_implications_of_color_blindness&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Color_blindness#Design_implications_of_color_blindness&lt;/a&gt; ).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The high-contrast option is I believe the best choice.&lt;/p&gt;&#xA;" LastActivityDate="2010-08-04T06:51:12.213" />
  <row Id="1219" PostTypeId="2" ParentId="764" CreationDate="2010-08-04T08:20:14.020" Score="3" ViewCount="" Body="&lt;p&gt;A very good article explaining the general approach of LMMs and their advantage over ANOVA is:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Baayen, R. H., Davidson, D. J., &amp;amp; Bates, D. M. (2008). &lt;a href=&quot;http://www.ualberta.ca/~baayen/publications/baayenDavidsonBates.pdf&quot; rel=&quot;nofollow&quot;&gt;Mixed-effects modeling with crossed random effects for subjects and items&lt;/a&gt;. &lt;em&gt;Journal of Memory and Language&lt;/em&gt;, &lt;em&gt;59&lt;/em&gt;, 390-412.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Linear mixed-effects models (LMMs) generalize regression models to have residual-like components, random effects, at the level of, e.g., people or items and not only at the level of individual observations. The models are very flexible, for instance allowing the modeling of varying slopes and intercepts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;LMMs work by using a likelihood function of some kind, the probability of your data given some parameter, and a method for maximizing this (Maximum Likelihood Estimation; MLE) by fiddling around with the parameters.  MLE is a very general technique allowing lots of different models, e.g., those for binary and count data, to be fitted to data, and is explained in a number of places, e.g.,&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Agresti, A. (2007). &lt;em&gt;An Introduction to Categorical Data Analysis (2nd Edition)&lt;/em&gt;. John Wiley &amp;amp; Sons.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;LMMs, however, can't deal with non-Gaussian data like binary data or counts; for that you need Generalized Linear Mixed-effects Models (GLMMs).  One way to understand these is first to look into GLMs; also see Agresti (2007).&lt;/p&gt;&#xA;" OwnerUserId="702" LastActivityDate="2010-08-04T08:20:14.020" />
  <row Id="1220" PostTypeId="2" ParentId="1215" CreationDate="2010-08-04T08:20:54.580" Score="2" ViewCount="" Body="&lt;p&gt;Following on from Rob's answer, you could estimate an acceptable level using a Normal approximation. So if an acceptable rate of running out is 5%, then use the following rule:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Qty + 1.644 * Qty^(0.5)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The value 1.644 comes from the &lt;a href=&quot;http://en.wikipedia.org/wiki/Normal_distribution&quot; rel=&quot;nofollow&quot;&gt;Normal&lt;/a&gt; or Gaussian distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other acceptable rates could be:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;1%: change 1.644 to 2.326&lt;/li&gt;&#xA;&lt;li&gt;10%: change 1.644 to 1.282&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Further Rationale&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rob suggested that you could model your data using a Poisson distribution. When the rate (or in your case Qty) is large (say 10), then the Normal distribution gives a good approximation. The following graph shows the Poisson 95th percentile (red) and the Normal approximation in green. As you can see they are fairly close. The raw data is shown as points.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://img821.imageshack.us/img821/3636/tmps.jpg&quot; alt=&quot;Normal approximation&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="8" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-08-04T08:33:07.973" LastActivityDate="2010-08-04T08:33:07.973" />
  <row Id="1221" PostTypeId="2" ParentId="534" CreationDate="2010-08-04T08:39:21.830" Score="6" ViewCount="" Body="&lt;p&gt;In the twins example it is not just the correlation that suggests causality, but also the associated information or prior knowledge. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose I add one further piece of information. Assume that the diligent twin spent 6 hours studying for a stats exam, but due to an unfortunate error the exam was in history. Would we still conclude the study was the cause of the superior performance?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Determining causality is as much a philosophical question as a scientific one, hence the tendency to invoke philosophers such as David Hume and Karl Popper when causality is discussed. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Not surprisingly medicine has made significant contributions to establishing causality through heuristics, such as Koch's postulates for establishing the causal relationship between microbes and disease. &#xA;These have been extended to &quot;molecular Koch's postulates&quot; required to show that a gene in a pathogen encodes a product that contributes to the disease caused by the pathogen. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unfortunately I can't post a hyperlinks supposedly beCAUSE I'm a new user (not true) and don't have enough &quot;reputation points&quot;. The real reason is anybody's guess.&lt;/p&gt;&#xA;" OwnerUserId="521" LastEditorUserId="521" LastEditorDisplayName="" LastEditDate="2010-08-04T08:44:43.833" LastActivityDate="2010-08-04T08:44:43.833" />
  <row Id="1222" PostTypeId="2" ParentId="886" CreationDate="2010-08-04T09:07:44.193" Score="4" ViewCount="" Body="&lt;p&gt;I can't post a comment (the appropriate place for this comment) as I don't have enough reputation, but the answer accepted as the best answer by the question owner misses the point.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;If statistics is all about maximizing likelihood, then machine learning is all about minimizing loss.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The likelihood is a loss function. Maximising likelihood is the same as minimising a loss function: the deviance, which is just -2 times the log-likelihood function. Similarly finding a least squares solution is about minimising the loss function describing the residual sum of squares. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Both ML and stats use algorithms to optimise the fit of some function (in the broadest terms) to data. Optimisation necessarily involves minimising some loss function.&lt;/p&gt;&#xA;" OwnerUserId="521" LastActivityDate="2010-08-04T09:07:44.193" CommentCount="4" />
  <row Id="1223" PostTypeId="1" CreationDate="2010-08-04T10:02:35.090" Score="6" ViewCount="664" Body="&lt;p&gt;I'm looking for some robust techniques to remove outliers and errors (whatever the cause) from financial time-series data (i.e. tickdata). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tick-by-tick financial time-series data is very messy. It contains huge (time) gaps when the exchange is closed, and make huge jumps when the exchange opens again. When the exchange is open, all kinds of factors introduce trades at price levels that are wrong (they did not occur) and/or not representative of the market (a spike because of an incorrectly entered bid or ask price for example). &lt;a href=&quot;http://www.tickdata.com/pdf/Tick_Data_Filtering_White_Paper.pdf&quot; rel=&quot;nofollow&quot;&gt;This paper by tickdata.com&lt;/a&gt; (PDF) does a good job of outlining the problem, but offers few concrete solutions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most papers I can find online that mention this problem either ignore it (the tickdata is assumed filtered) or include the filtering as part of some huge trading model which hides any useful filtering steps.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is anybody aware of more in-depth work in this area?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update:&lt;/strong&gt; &lt;a href=&quot;http://stats.stackexchange.com/questions/1142/simple-algorithm-for-online-outlier-detection-of-a-generic-time-series&quot;&gt;this questions&lt;/a&gt; seems similar on the surface but:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Financial time series is (at least at the tick level) non-periodic.&lt;/li&gt;&#xA;&lt;li&gt;The opening effect is a big issue because you can't simply use the last day's data as initialisation even though you'd really like to (because otherwise you have nothing). External events might cause the new day's opening to differ dramatically both in absolute level, and in volatility from the previous day.&lt;/li&gt;&#xA;&lt;li&gt;Wildly irregular frequency of incoming data. Near open and close of the day the amount of datapoints/second can be 10 times higher than the average during the day. The other question deals with regularly sampled data.&lt;/li&gt;&#xA;&lt;li&gt;The &quot;outliers&quot; in financial data exhibit some specific patterns that could be detected with specific techniques not applicable in other domains and I'm -in part- looking for those specific techniques.&lt;/li&gt;&#xA;&lt;li&gt;In more extreme cases (e.g. the flash crash) the outliers might amount to more than 75% of the data over longer intervals (&gt; 10 minutes). In addition, the (high) frequency of incoming data contains some information about the outlier aspect of the situation.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="127" LastEditorUserId="127" LastEditorDisplayName="" LastEditDate="2010-08-04T13:43:26.550" LastActivityDate="2010-08-04T15:51:53.803" Title="Robust outlier detection in financial timeseries" Tags="&lt;time-series&gt;&lt;outliers&gt;" AnswerCount="3" CommentCount="11" FavoriteCount="3" />
  <row Id="1224" PostTypeId="1" CreationDate="2010-08-04T10:38:03.740" Score="2" ViewCount="99" Body="&lt;p&gt;With data from two centres I want to account for potential heterogeneity or confounders between two centers. So the analysis will initially be stratified by clinical center and a chi square test performed with one degree of freedom.  Is this appropriate with just two centres? Or is there an alternative?&lt;/p&gt;&#xA;" OwnerUserId="712" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-18T21:55:09.667" LastActivityDate="2010-10-19T06:44:37.850" Title="Heterogeneity with two studies" Tags="&lt;hypothesis-testing&gt;&lt;clinical-trials&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="1225" PostTypeId="2" ParentId="652" CreationDate="2010-08-04T10:55:31.480" Score="1" ViewCount="" Body="&lt;p&gt;&lt;em&gt;Statistics as Principled Argument&lt;/em&gt; by Abelson is a good side book to learning statistics, particularly if your substantive field is in the social sciences.  It won't teach you how to do analysis, but it will teach you about statistical thinking.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I reviewed this book &lt;a href=&quot;http://www.associatedcontent.com/article/2225261/book_review_statistics_as_principled.html?cat=58&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="686" LastActivityDate="2010-08-04T10:55:31.480" CommunityOwnedDate="2010-08-04T10:55:31.480" />
  <row Id="1226" PostTypeId="2" ParentId="30" CreationDate="2010-08-04T11:48:13.410" Score="3" ViewCount="" Body="&lt;p&gt;There are two parts to testing a random number generator. If you're only concerned with testing a uniform generator, then yes, something like the DIEHARD test suite is a good idea.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But often you need to test a transformation of a uniform generator.  For example, you might use a uniform generator to create exponentially or normally distributed values.  You may have a high-quality uniform generator -- say you have a trusted implementation of a well-known algorithm such as Mersenne Twister -- but you need to test whether the transformed output has the right distribution.  In that case you need to do some sort of goodness of fit test such as Kolmogorov-Smirnov.  But for starters, you could verify that the sample mean and variance have the values you expect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most people don't -- and shouldn't -- write their own uniform random number generator from scratch.  It's hard to write a good generator and easy to fool yourself into thinking you've written a good one when you haven't.  For example, Donald Knuth tells the story in &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0201896842&quot; rel=&quot;nofollow&quot;&gt;TAOCP volume 2&lt;/a&gt; of a random number generator he wrote that turned out to be awful. But it's common for people to have to write their own code to produce random values from a new distribution. &lt;/p&gt;&#xA;" OwnerUserId="319" LastActivityDate="2010-08-04T11:48:13.410" />
  <row Id="1228" PostTypeId="1" AcceptedAnswerId="1273" CreationDate="2010-08-04T12:05:30.000" Score="7" ViewCount="164" Body="&lt;p&gt;I am using a control chart to try to work on some infection data, and will raise an alert if the infection is considered &quot;out of control&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Problems arrive when I come to a set of data where most of the time points have zero infection, with only a few occasions of one to two infections, but these already exceed the control limit of the chart, and raise an alert.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How should I work on the control chart if the data set is having very few positive infection counts?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="588" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-16T06:59:23.943" LastActivityDate="2010-09-16T06:59:23.943" Title="How to interpret a control chart containing a majority of zero values?" Tags="&lt;data-visualization&gt;&lt;control-chart&gt;" AnswerCount="5" CommentCount="2" FavoriteCount="1" />
  <row Id="1229" PostTypeId="2" ParentId="652" CreationDate="2010-08-04T12:32:08.780" Score="1" ViewCount="" Body="&lt;p&gt;The best intro in my eyes is the following one:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;David Howell - Statistical Methods for&#xA;  Psychology&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It is the BEST in making statistical concepts understandable for non mathematicians so that they get the math afterwards!&#xA;Unfortunately it is updated every year and, hence, pricey.&lt;/p&gt;&#xA;" OwnerUserId="442" LastActivityDate="2010-08-04T12:32:08.780" CommunityOwnedDate="2010-08-04T12:32:08.780" />
  <row Id="1230" PostTypeId="2" ParentId="652" CreationDate="2010-08-04T12:32:12.103" Score="3" ViewCount="" Body="&lt;p&gt;I am a big fan of &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0521671051&quot; rel=&quot;nofollow&quot;&gt;Statistical Models - Theory and Practice&lt;/a&gt; by David Friedman. It  succeeds remarkably well to introduce and motivate the different concepts of statistical modeling through concrete, and historically important problems (cholera in London, Yule on the causes of poverty, Political repression in the McCarty era..). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Friedman illustrates the principles of modeling, and the pitfalls. In some sense, the discussion shows how to think about the critical issues and is honest about the connection between the statistical models and the real world phenomena.&lt;/p&gt;&#xA;" OwnerUserId="358" LastActivityDate="2010-08-04T12:32:12.103" CommunityOwnedDate="2010-08-04T12:32:12.103" />
  <row Id="1231" PostTypeId="2" ParentId="1228" CreationDate="2010-08-04T12:52:05.560" Score="1" ViewCount="" Body="&lt;p&gt;You are asking quite a tricky question!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is outside my area of expertise, but I know that &lt;a href=&quot;http://www.mcs.open.ac.uk/People/c.p.farrington&quot; rel=&quot;nofollow&quot;&gt;Prof Farrington&lt;/a&gt; does some work on this problem. So I would look at a some of his papers and follow a few of his references. To get you started, this &lt;a href=&quot;http://stats-www.open.ac.uk/TechnicalReports/Cusum.pdf&quot; rel=&quot;nofollow&quot;&gt;report&lt;/a&gt; looks relevant.&lt;/p&gt;&#xA;" OwnerUserId="8" LastActivityDate="2010-08-04T12:52:05.560" />
  <row Id="1232" PostTypeId="2" ParentId="1228" CreationDate="2010-08-04T12:56:10.217" Score="1" ViewCount="" Body="&lt;p&gt;Would it make sense to plot the control chart based on an average of the weekly infections or another similar floating average? Would this then 'damp' out spikes due to daily high values whilst ensuring that changes in trends are picked up in a relatively timely manner. &lt;/p&gt;&#xA;" OwnerUserId="210" LastActivityDate="2010-08-04T12:56:10.217" />
  <row Id="1233" PostTypeId="2" ParentId="1228" CreationDate="2010-08-04T13:18:12.700" Score="1" ViewCount="" Body="&lt;p&gt;Perhaps, you can build an edge case in your routine/software to deal with the situation. If you detect several zeros in the dataset then you set a separate control for that particular situation. This is obviously a hack and not a principled solution but may serve your present needs till you can come up with something better. &lt;/p&gt;&#xA;" LastActivityDate="2010-08-04T13:18:12.700" CommentCount="1" />
  <row Id="1234" PostTypeId="2" ParentId="1223" CreationDate="2010-08-04T13:32:07.383" Score="1" ViewCount="" Body="&lt;p&gt;Convert your price data to (percentage) returns,&#xA;then remove all the points lying at more than 3 MAD from the median.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Mean_absolute_deviation&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Mean_absolute_deviation&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="603" LastActivityDate="2010-08-04T13:32:07.383" CommentCount="4" />
  <row Id="1235" PostTypeId="2" ParentId="1223" CreationDate="2010-08-04T13:47:36.963" Score="2" ViewCount="" Body="&lt;p&gt;I'll add some paper references when I'm back at a computer, but here are some simple suggestions:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Definitely start by working with returns. This is critical to deal with the irregular spacing where you can naturally get big price gaps (especially around weekends). Then you can apply a simple filter to remove returns well outside the norm (eg. vs a high number of standard deviations). The returns will adjust to the new absolute level so large real changes will result in the loss of only one tick. I suggest using a two-pass filter with returns taken from 1 step and &lt;em&gt;n&lt;/em&gt; steps to deal with clusters of outliers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Edit 1:&lt;/em&gt; Regarding the usage of prices rather than returns: asset prices tend to not be stationary, so IMO that can pose some additional challenges. To account for the irregularity and power law effects, I would advise some kind of adjustment if you want to include them in your filter. You can scale the price changes by the time interval or by volatility.  You can refer to the &quot;realized volatility&quot; literture for some discussion on this.  Also discussed in Dacorogna et. al.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To account for the changes in volatility, you might try basing your volatility calculation from the same time of the day over the past week (using the seasonality).  &lt;/p&gt;&#xA;" OwnerUserId="5" LastEditorUserId="5" LastEditorDisplayName="" LastEditDate="2010-08-04T15:45:37.880" LastActivityDate="2010-08-04T15:45:37.880" CommentCount="1" />
  <row Id="1236" PostTypeId="2" ParentId="652" CreationDate="2010-08-04T14:02:21.880" Score="1" ViewCount="" Body="&lt;p&gt;As a biologist, I found the Sokal and Rohlf text to be quite readable, despite its voluminous-ness. It's not so great as a quick reference, but does walk one through statistical theory.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;R. R. Sokal and F. J. Rohlf, Biometry the principles and practice of statistics in biological research, Third. (New York: W.J. Freeman and Company, 1995).&lt;/p&gt;&#xA;" OwnerUserId="124" LastActivityDate="2010-08-04T14:02:21.880" CommunityOwnedDate="2010-08-04T14:02:21.880" />
  <row Id="1237" PostTypeId="2" ParentId="109" CreationDate="2010-08-04T14:32:12.320" Score="1" ViewCount="" Body="&lt;p&gt;I usually like to run simulations to answer questions like this, but without confirmed details of the algorithm the question asker wants evaluated and no obvious implementation of the Holm-Sidak procedure available in R, that is not possible.  For my answer I eyeballed the code provided &lt;a href=&quot;http://www.mathworks.com/matlabcentral/fileexchange/12786&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.  Assuming that is the right procedure, and assuming the null hypothesis is that all group means are equal:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Feel free to correct me. I usually like to run simulations on such things. But, without the ability to readily do that, I can't check my answer. So, I might be entirely wrong here.  My answer is that usually the Holm-Sidak will demonstrate greater power, but that the answer in a strict sense is &quot;it depends&quot;.  Both methods use the pooled error term and assume homogeneity of variance, so there is no difference in the procedures there.  However, since Holm-Sidak adjusts in a stepwise manner early comparisons are more likely to pass a threshold of significance than later comparisons, especially since the freedom to assess later comparisons is dependent on the outcome of previous comparisons.  Thus, it seems likely that in situations where the differences in means between groups to be compared is roughly equal (and meets the Tukey HSD threshold for significance) and the number of groups is sufficiently large (purposely vague without a simulation), that Holm-Sidak will fail to reach significance for the later comparisons.  Thus, in these situations Tukey's HSD will have more power than Holm-Sidak.&lt;/p&gt;&#xA;" OwnerUserId="196" LastEditorUserId="196" LastEditorDisplayName="" LastEditDate="2010-08-04T14:38:22.450" LastActivityDate="2010-08-04T14:38:22.450" />
  <row Id="1238" PostTypeId="2" ParentId="30" CreationDate="2010-08-04T14:47:27.783" Score="1" ViewCount="" Body="&lt;p&gt;The &lt;a href=&quot;http://csrc.nist.gov/groups/ST/toolkit/rng/stats_tests.html&quot; rel=&quot;nofollow&quot;&gt;NIST publishes a list of statistical tests&lt;/a&gt; with a reference implementation in C.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is also &lt;a href=&quot;http://www.iro.umontreal.ca/~simardr/testu01/tu01.html&quot; rel=&quot;nofollow&quot;&gt;TestU01&lt;/a&gt; by some smart folks, including respected PRNG researcher Pierre L'Ecuyer. Again, there is a reference implementation in C.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As pointed out by other commenters, these are for testing the generation of pseudo random bits. If you transform these bits into a different random variable (e.g. Box-Muller transform from uniform to Normal), you'll need additional tests to confirm the correctness of the transform algorithm.&lt;/p&gt;&#xA;" OwnerUserId="729" LastActivityDate="2010-08-04T14:49:15.003" CommentCount="2" />
  <row Id="1239" PostTypeId="1" CreationDate="2010-08-04T14:55:17.533" Score="2" ViewCount="296" Body="&lt;p&gt;Hello, what Ruby gems are there that can perform data processing?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="191" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-05-11T13:27:31.860" LastActivityDate="2011-05-11T13:27:31.860" Title="Ruby statistical gem" Tags="&lt;ruby&gt;" AnswerCount="2" CommentCount="3" />
  <row Id="1240" PostTypeId="2" ParentId="30" CreationDate="2010-08-04T15:11:13.447" Score="2" ViewCount="" Body="&lt;p&gt;Small correction to Colin's post: the CRAN package &#xA;&lt;a href=&quot;http://cran.r-project.org/package=RDieHarder&quot; rel=&quot;nofollow&quot;&gt;RDieHarder&lt;/a&gt; is an interface to &#xA;&lt;a href=&quot;http://www.phy.duke.edu/~rgb/General/dieharder.php&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;DieHarder&lt;/strong&gt;&lt;/a&gt;, the Diehard rewrite / extension / overhaul done by &lt;a href=&quot;http://www.phy.duke.edu/~rgb/&quot; rel=&quot;nofollow&quot;&gt;Robert G. Brown&lt;/a&gt; (who kindly lists me as a coauthor based on my RDieHarder wrappers) with recent contribution by David Bauer.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Among other things, DieHarder includes the &lt;a href=&quot;http://csrc.nist.gov/groups/ST/toolkit/rng/stats_tests.html&quot; rel=&quot;nofollow&quot;&gt;NIST battery of tests&lt;/a&gt; mentioned in Mark's post as well as some new ones.  This is ongoing research and has been for a while.  I gave a talk at useR! 2007 about RDieHarder which you can get from &lt;a href=&quot;http://dirk.eddelbuettel.com/presentations.html&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="334" LastActivityDate="2010-08-04T15:11:13.447" />
  <row Id="1241" PostTypeId="1" CreationDate="2010-08-04T15:13:27.610" Score="5" ViewCount="342" Body="&lt;p&gt;When would you tend to use ROC curves over some other tests to determine the predictive ability of some measurement on an outcome?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When dealing with discrete outcomes (alive/dead, present/absent), what makes ROC curves more or less powerful than something like a chi-square?&lt;/p&gt;&#xA;" OwnerUserId="684" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-08-27T02:42:25.167" LastActivityDate="2010-08-27T08:22:12.153" Title="What do ROC curves tell you that traditional inference wouldn't?" Tags="&lt;regression&gt;&lt;roc&gt;" AnswerCount="4" CommentCount="2" FavoriteCount="1" />
  <row Id="1242" PostTypeId="2" ParentId="1239" CreationDate="2010-08-04T15:14:21.807" Score="2" ViewCount="" Body="&lt;p&gt;Srikant is probably right that this would get better answers at stack overflow, but here's a quick answer anyway:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Rubygsl&lt;/strong&gt; provides an interface to the GNU Scientific Library.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;RSRuby&lt;/strong&gt; provides a way to invoke R commands through Ruby&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="54" LastActivityDate="2010-08-04T15:14:21.807" />
  <row Id="1243" PostTypeId="2" ParentId="1174" CreationDate="2010-08-04T15:24:11.960" Score="0" ViewCount="" Body="&lt;p&gt;I think the main point is the one sidmaestro raises...does the experimental setup or data generation mechanism support the premise that the data might arise from a Poisson distribution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not a big fan of testing for distributional assumptions, since those tests typically aren't very useful. What seems more useful to me is to make distributional or model assumptions that are flexible and reasonably robust to deviations from the model, typically for purposes of inference. In my experience, it is not that common to see mean=variance, so often the negative binomial model seems more appropriate, and includes the Poisson as a special case. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another point that is important in going for distributional testing, if that's what you want to do, is to make sure that there aren't strata involved which would make your observed distribution a mixture of other distributions. Individual stratum-specific distributions might appear Poisson, but the observed mixture might not be. An analogous situation from regression only assumes that the &lt;strong&gt;conditional&lt;/strong&gt; distribution of Y|X  is normally distributed, and not really the distribution of Y itself.&lt;/p&gt;&#xA;" OwnerUserId="732" LastActivityDate="2010-08-04T15:24:11.960" CommentCount="2" />
  <row Id="1244" PostTypeId="2" ParentId="1149" CreationDate="2010-08-04T15:37:40.830" Score="1" ViewCount="" Body="&lt;p&gt;The way I think about this really is in terms of information. Say each of X1 and X2 has some information about Y. The more correlated X1 and X2 are with each other, the more the information content about Y from X1 and X2 are similar or overlapping, to the point that for perfectly correlated X1 and X2, it really is the same information content. If we now put X1 and X2 in the same (regression) model to explain Y, the model tries to &quot;apportion&quot; the information that (X1,X2) contains about Y to each of X1 and X2, in a somewhat arbitrary manner. There is no really good way to apportion this, since any split of the information still leads to keeping the total information from (X1,X2) in the model (for perfectly correlated X's, this really is a case of non-identifiability). This leads to unstable individual estimates for the individual coefficients of X1 and X2, though if you look at the predicted values b1X1+b2X2 over many runs and estimates of b1 and b2, these will be quite stable. &lt;/p&gt;&#xA;" OwnerUserId="732" LastActivityDate="2010-08-04T15:37:40.830" />
  <row Id="1245" PostTypeId="2" ParentId="1223" CreationDate="2010-08-04T15:51:53.803" Score="4" ViewCount="" Body="&lt;p&gt;The problem is definitely &lt;em&gt;hard&lt;/em&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Mechanical rules like the +/- &lt;em&gt;N1&lt;/em&gt; times standard deviations, or +/ &lt;em&gt;N2&lt;/em&gt; times MAD,  or +/- &lt;em&gt;N3&lt;/em&gt; IQR or ... &lt;em&gt;will&lt;/em&gt; fail because there are always some series that are different as for example:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;fixings like interbank rate may be constant for some time and then jump all of a sudden&lt;/li&gt;&#xA;&lt;li&gt;similarly for &lt;em&gt;e.g.&lt;/em&gt; certain foreign exchanges coming off a peg&lt;/li&gt;&#xA;&lt;li&gt;certain instrument are implicitly spreads; these may be near zero for periods and all of a sudden jump manifold&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Been there, done that, ... in a previous job.  You could try to bracket each series using arbitrage relations ships (&lt;em&gt;e.g.&lt;/em&gt; assuming USD/EUR and EUR/JPY are presumed good, you can work out bands around what USD/JPY should be; likewise for derivatives off an underlying etc pp.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Commercial data vendors expand some effort on this, and those of use who are clients of theirs know ... it still does not exclude errors.&lt;/p&gt;&#xA;" OwnerUserId="334" LastActivityDate="2010-08-04T15:51:53.803" CommentCount="3" />
  <row Id="1246" PostTypeId="2" ParentId="1241" CreationDate="2010-08-04T15:55:11.867" Score="4" ViewCount="" Body="&lt;p&gt;An ROC curve is used when the predictor is continuous and the outcome is discrete, so a chi-square test would not be applicable. In fact, ROC analysis is in some sense equivalent to the Mann-Whitney test: the area under the curve is P(X&gt;Y) which is the quantity being tested by the M-W test. However Mann-Whitney analysis does not emphasize selecting a cutoff, while that is the main point of the ROC analysis. Additionally, ROC curves are often used as just a visual display of the predictive ability of a covariate.&lt;/p&gt;&#xA;" OwnerUserId="279" LastActivityDate="2010-08-04T15:55:11.867" />
  <row Id="1247" PostTypeId="2" ParentId="1241" CreationDate="2010-08-04T15:56:39.150" Score="3" ViewCount="" Body="&lt;p&gt;The shortest answer is that traditional tests of signal detection only give you a single point on the ROC (receiver operating characteristic) while the curve allows you to see responses through a range of values.  It's possible that the criteria and d' shift as one shifts throughout the curve.  It's like the difference between a t-test generated by selecting two classes of predictor variables and two regression lines generated by looking at parametric manipulations of each predictor variable.&lt;/p&gt;&#xA;" OwnerUserId="601" LastActivityDate="2010-08-04T15:56:39.150" />
  <row Id="1248" PostTypeId="2" ParentId="1205" CreationDate="2010-08-04T15:57:30.210" Score="1" ViewCount="" Body="&lt;p&gt;Use regression methods. A simple linear regression with group coded as 0-1 (or 1-2, etc) is equivalent to a t-test, but regression software usually has the capability to incorporate weigths correctly.&lt;/p&gt;&#xA;" OwnerUserId="279" LastActivityDate="2010-08-04T15:57:30.210" />
  <row Id="1249" PostTypeId="1" CreationDate="2010-08-04T16:04:56.337" Score="5" ViewCount="267" Body="&lt;p&gt;How to find a non-trivial upper bound on $E[\exp(Z^2)]$ when $Z \sim {\rm Bin}(n, n^{-\beta})$ with $\beta \in (0,1)$? A trivial bound is obtained for substituting $Z$ with $n$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A background on this question. In the paper by Baraud, 2002 -- Non-asymptotic minimax rates of testing in signal detection, if one is to substitute the model in Eq. (1), by a random effects model, then the above quantity appears in the computation of a lower bound. &lt;/p&gt;&#xA;" OwnerUserId="168" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-29T00:25:44.380" LastActivityDate="2011-04-29T00:25:44.380" Title="Non-trivial bound for $E[\exp(Z^2)]$ when $Z \sim {\rm Bin}(n, n^{-\beta})$ with $\beta \in (0,1)$" Tags="&lt;probability&gt;&lt;binomial&gt;&lt;mathematical-statistics&gt;" AnswerCount="3" CommentCount="6" FavoriteCount="1" />
  <row Id="1251" PostTypeId="2" ParentId="1173" CreationDate="2010-08-04T16:29:26.283" Score="5" ViewCount="" Body="&lt;p&gt;Thanks for all you answers. For completeness I thought I should include what I usually do. I tend to do a combination of the suggestions given: dots, boxplots (when n is large), and se (or sd) ranges.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://img594.imageshack.us/img594/6883/tmpsc.jpg&quot; alt=&quot;Dot plot&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the dot plot, it is clear that data is far more spread out the &quot;handle bar&quot; plots suggest. In fact, there is a negative value in A3!&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;I've made this answer a CW so I don't gain rep&lt;/p&gt;&#xA;" OwnerUserId="8" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-10-19T20:36:30.143" LastActivityDate="2010-10-19T20:36:30.143" CommunityOwnedDate="2010-10-19T20:36:30.143" CommentCount="2" />
  <row Id="1252" PostTypeId="1" AcceptedAnswerId="1349" CreationDate="2010-08-04T16:39:49.250" Score="12" ViewCount="927" Body="&lt;p&gt;I was wondering if you know if a stastical model &quot;cheat sheet(s)&quot; that lists any or more information:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;When to use the model&lt;/li&gt;&#xA;&lt;li&gt;When not to use the model&lt;/li&gt;&#xA;&lt;li&gt;required and optional inputs&lt;/li&gt;&#xA;&lt;li&gt;expected outputs&lt;/li&gt;&#xA;&lt;li&gt;has the model been tested in different fields (policy, bio, engineering, manufacturing, etc)&lt;/li&gt;&#xA;&lt;li&gt;is it accepted in practice or research&lt;/li&gt;&#xA;&lt;li&gt;expected variation / accuracy / precision&lt;/li&gt;&#xA;&lt;li&gt;caveats&lt;/li&gt;&#xA;&lt;li&gt;scalability&lt;/li&gt;&#xA;&lt;li&gt;deprecated model, avoid or don't use&lt;/li&gt;&#xA;&lt;li&gt;etc ..&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I've seen hierarchies before on various websites, and some simplistic model cheat sheets in various textbooks; however, it'll be nice if there is a larger one that encompasses various types of models based on different types of analysis and theories.&lt;/p&gt;&#xA;" OwnerUserId="59" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-16T06:59:07.783" LastActivityDate="2011-05-18T08:10:16.340" Title="Statistical models cheat sheet" Tags="&lt;modeling&gt;&lt;best-practices&gt;" AnswerCount="4" FavoriteCount="12" />
  <row Id="1253" PostTypeId="1" CreationDate="2010-08-04T16:53:04.197" Score="4" ViewCount="99" Body="&lt;p&gt;I'm trying to compute item-item similarity using &lt;a href=&quot;http://en.wikipedia.org/wiki/Jaccard_index#Tanimoto_coefficient_.28extended_Jaccard_coefficient.29&quot; rel=&quot;nofollow&quot;&gt;Jaccard(specifically Tanimoto)&lt;/a&gt; on a large list of data in the format&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(userid, itemid)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;An item is considered as rated if i have a userid-itemid pair. I have about 800k users and 7900 items, and 3.57 million 'ratings'.  I've restricted my data to users who have rated at least n items(usually 10).  However, I'm wondering if I should place an upper limit on number of items rated.  When users rate 1000 or more items, each user generates 999000 pairwise-combinations of items to use in my calc, assuming the calculation&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;n! / (n-r)!&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Adding this much input data slows the calculating process down tremendously, even when the workload is distributed(using hadoop).  I'm thinking that the users who rate many, many items are not my core users and might be diluting my similarity calculations.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My gut tells me to limt the data to customers who have rated between 10 and 150-200 items but I'm not sure if there is a better way to statistically determine these boundaries.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are some more details about my source data's distribution.  Please feel free to enlighten me on any statistical terms that I might have butchered!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The distribution of my users' itemCounts:&#xA;&lt;img src=&quot;http://www.neilkodner.com/images/littlesnapper/itemsRated.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; summary(raw)&#xA;   itemsRated      &#xA; Min.   :   1.000  &#xA; 1st Qu.:   1.000  &#xA; Median :   1.000  &#xA; Mean   :   4.466  &#xA; 3rd Qu.:   3.000  &#xA; Max.   :2069.000  &#xA;&#xA;&amp;gt; sd(raw)&#xA;itemsRated &#xA;  16.46169 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If I limit my data to users who have rated at least 10 items:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; above10&amp;lt;-raw[raw$itemsRated&amp;gt;=10,]&#xA;&amp;gt; summary(above10)&#xA;   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. &#xA;  10.00   13.00   19.00   34.04   35.00 2069.00 &#xA;&amp;gt; sd(above10)&#xA;[1] 48.64679&#xA;&amp;gt; length(above10)&#xA;[1] 64764&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If I further limit my data to users who have rated between 10 and 150 items:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; above10less150&amp;lt;-above10[above10&amp;lt;=150]&#xA;&amp;gt; summary(above10less150)&#xA;   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. &#xA;  10.00   13.00   19.00   28.17   33.00  150.00 &#xA;&amp;gt; sd(above10less150)&#xA;[1] 24.32098&#xA;&amp;gt; length(above10less150)&#xA;[1] 63080&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Edit: I dont think this is an issue of outliers as much as the data is positively skewed. &lt;/p&gt;&#xA;" OwnerUserId="738" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-17T20:31:51.893" LastActivityDate="2010-09-17T20:31:51.893" Title="How to limit my input data for Jaccard item-item similarity calc" Tags="&lt;distributions&gt;&lt;data-mining&gt;" AnswerCount="1" />
  <row Id="1254" PostTypeId="2" ParentId="1252" CreationDate="2010-08-04T17:30:03.033" Score="5" ViewCount="" Body="&lt;p&gt;Do you mean a statistical analysis decision tree? (&lt;a href=&quot;http://www.google.com/images?um=1&amp;amp;hl=en&amp;amp;biw=1024&amp;amp;bih=628&amp;amp;tbs=isch:1&amp;amp;sa=1&amp;amp;q=statistical+analysis+decision+tree&amp;amp;aq=f&amp;amp;aqi=&amp;amp;aql=&amp;amp;oq=&amp;amp;gs_rfai=&quot; rel=&quot;nofollow&quot;&gt;google search&lt;/a&gt;), like this (only with extensions):&#xA;&lt;img src=&quot;http://www.processma.com/resource/images/htest.gif&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;BTW, notice that the chart in wrong in that the tests it offers for median are not for median but for rank...&lt;/p&gt;&#xA;" OwnerUserId="253" LastActivityDate="2010-08-04T17:30:03.033" CommentCount="1" />
  <row Id="1255" PostTypeId="2" ParentId="1253" CreationDate="2010-08-04T17:35:42.507" Score="1" ViewCount="" Body="&lt;p&gt;I'm confused: shouldn't you only need the 7900^2 item similarities, for which you use ratings from all users, which is still quite sparse?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I still think there's a more efficient way to do this, but maybe I'm just being dense. Specifically, consider item A and item B. For item A, generate a U-dimensional vector of 0's and 1's, where U is the number of users in your data set, and there's a 1 in dimension i if and only if user i rated item A. Do the same thing for item B. Then you can easily generate the AB, A and B terms for your equation from these vectors. Importantly, these vectors are very sparse, so they can produce a very small data set if encoded properly.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Iterate over the item ID's to generate their cross product: (ItemAID, ItemBID)&lt;/li&gt;&#xA;&lt;li&gt;Map this pair to this n-tuple: (ItemAID, ItemBID, ItemAVector, ItemBVector)&lt;/li&gt;&#xA;&lt;li&gt;Reduce this n-tuple to your similarity measure: (ItemAID,ItemBID,SimilarityMetric)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;If you set up a cache of the ItemXVector's at the start, this computation should be very fast.&lt;/p&gt;&#xA;" OwnerUserId="303" LastEditorUserId="303" LastEditorDisplayName="" LastEditDate="2010-08-04T18:26:22.043" LastActivityDate="2010-08-04T18:26:22.043" CommentCount="1" />
  <row Id="1256" PostTypeId="1" AcceptedAnswerId="1263" CreationDate="2010-08-04T17:39:29.803" Score="5" ViewCount="268" Body="&lt;p&gt;I have a colleague who calculates correlations in which one set of scores for a subject (e.g. 100 scores) is correlated with another set of scores for that same subject. The resulting correlation reflects the degree to which those sets of scores are associated for that subject.  He needs to do this for N subjects. Consider the following dataset:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ncol &amp;lt;- 100&#xA;nrow &amp;lt;- 100&#xA;x &amp;lt;- matrix(rnorm(ncol*nrow),nrow,ncol)&#xA;y &amp;lt;- matrix(rnorm(ncol*nrow),nrow,ncol)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The correct output vector of correlations would be:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;diag(cor(t(x),t(y)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is there a faster way to do this without using a multicore package in R?&lt;/p&gt;&#xA;" OwnerUserId="196" LastActivityDate="2010-08-05T10:00:43.613" Title="How can one speed up this correlation calculation in R without multicore?" Tags="&lt;r&gt;&lt;correlation&gt;&lt;efficiency&gt;" AnswerCount="3" CommentCount="4" FavoriteCount="2" />
  <row Id="1257" PostTypeId="1" AcceptedAnswerId="1312" CreationDate="2010-08-04T18:04:52.290" Score="1" ViewCount="56" Body="&lt;p&gt;Consider the following sequential, adaptive data generating process for $Y_1$, $Y_2$, $Y_3$. (By sequential I mean that we generate $Y_1$, $Y_2$, $Y_3$ in sequence and by adaptive I mean that $Y_3$ is generated depending on the observed values of $Y_1$ and $Y_2$.):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Y_1 = X_1\  \beta + \epsilon_1$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Y_2 = X_2\  \beta + \epsilon_2$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Y_3 = X_3\  \beta + \epsilon_3$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$&#xA;X_3 = &#xA;\begin{cases} &#xA; X_{31} &amp;amp; \mbox{if }Y_1 Y_2  \gt  0 \\ X_{32} &amp;amp; \mbox{if }Y_1 Y_2 \le 0&#xA;\end{cases}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$X_1$, $X_2$, $X_{31}$ and $X_{32}$ are all 1 x 2 vectors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\beta$ is a 2 x 1 vector&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\epsilon_i \sim N(0,\sigma^2)$ for $i$ = 1, 2, 3&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose we observe the following sequence: {$Y_1 = y_1,\ Y_2 = y_2,\ X_3 = X_{31},\ Y_3 = y_3$} and wish to estimate the parameters $\beta$ and $\sigma$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to write down the likelihood function note that we have four random variables: $Y_1$, $Y_2$, $X_3$ and $Y_3$. Therefore, the joint density of $Y_1$, $Y_2$, $X_3$ and $Y_3$ is given by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$f(Y_1, Y_2, X_3, Y_3 |-) = f(Y_1|-)\ f(Y_2|-)\ [\ f(Y_3|X_{31},-)\ P(X_3=X_{31}|-)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad + \ f(Y_3|X_{32},-)\ P(X_3={X_{32}}|-)\ ]$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Note: I am suppressing the dependency of the density on $\beta$ and $\sigma$.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since the likelihood conditions on the observed data and our sequence is such that $y_1 y_2 &gt;0$. Therefore, we have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$L(\beta,\ \sigma | X_1,\ X_2,\ X_{31}, y_1, y_2, y_3) = f(Y_1|-)\ f(Y_2|-)\ f(Y_3|X_{31},-)\ P(X_3=X_{31}) $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is the above the correct likelihood function for this data generating process?&lt;/p&gt;&#xA;" LastActivityDate="2010-08-05T16:04:34.100" Title="What is the correct likelihood function for an sequential, adaptive data generation process?" Tags="&lt;likelihood-function&gt;&lt;data-generating-process&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="1258" PostTypeId="2" ParentId="1164" CreationDate="2010-08-04T18:12:23.507" Score="5" ViewCount="" Body="&lt;p&gt;As someone who has learned a little bit of statistics for my own research, I'll guess that the reasons are pedagogical and inertial.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've observed within my own field that the order in which topics are taught reflects the history of the field. Those ideas which came first are taught first, and so on. For people who only dip into stats for cursory instruction, this means they'll learn classical stats first, and probably last. Then, even if they learn more, the classical stuff with stick with them better due to primacy effects.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, everyone knows what a two sample t-test is. Less than everyone knows what a Mann-Whitney or Wilcoxon Rank Sum test is. This means that I have to exert just a little bit of energy on explaining what my robust test is, versus not having to exert any with a classical test. Such conditions will obviously result in fewer people using robust methods than should.&lt;/p&gt;&#xA;" OwnerUserId="287" LastActivityDate="2010-08-04T18:12:23.507" />
  <row Id="1259" PostTypeId="2" ParentId="726" CreationDate="2010-08-04T18:29:57.000" Score="2" ViewCount="" Body="&lt;p&gt;All information looks like noise until you break the code.&#xA;—Hiro in Neal Stephenson's &lt;em&gt;Snow Crash&lt;/em&gt; (1992)&lt;/p&gt;&#xA;" OwnerUserId="743" LastActivityDate="2010-08-04T18:29:57.000" CommunityOwnedDate="2010-08-04T18:29:57.000" />
  <row Id="1260" PostTypeId="2" ParentId="1256" CreationDate="2010-08-04T18:35:22.340" Score="0" ViewCount="" Body="&lt;p&gt;It might be one of those cases where using a different &lt;a href=&quot;http://cran.r-project.org/bin/windows/base/rw-FAQ.html#Can-I-use-a-fast-BLAS_003f&quot; rel=&quot;nofollow&quot;&gt;BLAS engine&lt;/a&gt; would help.  But I am not sure of it - it needs testing (and depends on your machine)&lt;/p&gt;&#xA;" OwnerUserId="253" LastActivityDate="2010-08-04T18:35:22.340" />
  <row Id="1261" PostTypeId="1" AcceptedAnswerId="1282" CreationDate="2010-08-04T19:06:50.250" Score="3" ViewCount="126" Body="&lt;p&gt;Suppose I have a table of counts that look like this&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;            A    B     C&#xA;Success  1261  230  3514&#xA;Failure   381  161  4012&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have a hypothesis that there is some probability $p$ such that $P(Success_A) = p^i$, $P(Success_B) = p^j$ and $P(Success_C) = p^k$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there some way to produce estimates for $p$, $i$, $j$ and $k$? The idea I have is to iteratively try values for $p$ between 0 and 1, and values for $i$, $j$ and $k$ between 1 and 5. Given the column totals, I could produce expected values, then calculate $\chi^2$ or $G^2$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This would produce a best fit, but it wouldn't give any confidence interval for any of the values. It's also not particularly computationally efficient.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a side question, if I wanted to test the goodness of fit of a particular set of values for $i$, $j$ and $k$ (specifically 1, 2, and 3), once I've calculated $\chi^2$ or $G^2$, I'd want to calculate significance on the $\chi^2$ distribution with 1 degree of freedom, correct? This isn't a normal contingency table since relationship of each column to the others is fixed to a single value. Given $p$, $i$, $j$ and $k$, filling in a single value in a cell fixes what the values of the other cells must be. &lt;/p&gt;&#xA;" OwnerUserId="287" LastActivityDate="2010-08-05T10:19:24.293" Title="Fitting a fixed, exponential relationship between categories with categorical data." Tags="&lt;modeling&gt;&lt;categorical-data&gt;&lt;chi-squared&gt;" AnswerCount="2" CommentCount="4" />
  <row Id="1262" PostTypeId="2" ParentId="1261" CreationDate="2010-08-04T19:19:23.767" Score="2" ViewCount="" Body="&lt;p&gt;You could write the likelihood function like so:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$L(p,i,j,k|-) \propto (p^i)^{1261} (1-p^i)^{381} (p^j)^{230} (1-p^j)^{161} (p^k)^{3514} (1-p^k)^{4012}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maximize the above likelihood function to estimate your parameters. Constructing confidence intervals and hypothesis testing should be straight forward once you have the estimates.&lt;/p&gt;&#xA;" LastActivityDate="2010-08-05T10:19:24.293" />
  <row Id="1263" PostTypeId="2" ParentId="1256" CreationDate="2010-08-04T19:30:30.623" Score="3" ViewCount="" Body="&lt;p&gt;While making a call to &lt;code&gt;diag&lt;/code&gt; you throw out a lot of information, so you can save time by simply not calculating it. You code is equivalent to:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sapply(1:100,function(i) cor(x[i,],y[i,]))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Extended to reflect comments:&lt;/strong&gt; This code will be slower for small matrices since it does not use the full &quot;vectorization power&quot; of &lt;code&gt;cor&lt;/code&gt;. So, if you'd like to make fast calculations on small matrices, write it as a C chunk. If one would like to parallelize it (again, will be profitable only for large matrices), may use this code replacing &lt;code&gt;sapply&lt;/code&gt; with &lt;code&gt;mc.lapply&lt;/code&gt; or something like this.&lt;/p&gt;&#xA;" OwnerUserId="88" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-08-05T10:00:43.613" LastActivityDate="2010-08-05T10:00:43.613" CommentCount="7" />
  <row Id="1264" PostTypeId="2" ParentId="1241" CreationDate="2010-08-04T19:31:22.247" Score="7" ViewCount="" Body="&lt;p&gt;The ROC function (it is not necessarily a curve) allows you to assess the discrimination ability provided by a a specific statistical model (comprised of a predictor variable or a set of them). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A main consideration of ROCs is that model predictions do not only stem from the model's ability to discriminate/make predictions based on the evidence provided by predictor variables. Also operating is a response criteria that defines how much evidence is necessary for the model to predict a response, and what is the outcome of these responses. The value that is established for the response criteria will greatly influence the model predictions, and ultimately the type of mistakes that it will make.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider a generic model with predictor variables and a response criteria. This model is trying to predict the Presence of X,by responding Yes or No.&#xA;So you have the following confusion matrix:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;                                **X present               X absent**&#xA; **Model Predicts X Present**       Hit                   False Alarm&#xA;&#xA; **Model Predicts X Absent**      Miss                 Correct Rejection&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In this matrix, you only need to consider the proportion of Hits and the False Alarms (because the others can be derived from these, given that they have to some to 1). For each response criteria, you wil ave a different confusion matrix. The errors (Misses and False Alarms) are negatively related, which means that a response criteria that minimizes false alarms maximizes misses and vice-versa. The message is: there is no free lunch. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, in order to understand how well the model discriminates cases/makes predictions, independently of the response criteria established, you plot the Hits and False rates produced across the range of possible response criteria.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you get from this plot is the ROC function. The area under the function provides an unbiased, and non-parametric measure of the discrimination ability of the model. This measure is very important because it is free of any confounds that could have been produced by the response criteria.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A second important aspect, is that by analyzing the function, one can define what response criteria is better for your objectives. What types of errors you want to avoid, and what are errors are OK. For instance, consider an HIV test: it is a test that looks up some sort of evidence (in this case antibodies) and  makes a discrimination/prediction based on the comparison of the evidence with response criterion. This response criterion is usually set very low, so that you minimize Misses. Of course this will result in more False Alarms, which have a cost, but a cost that is negligible when compared to the Misses.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With ROCs you can assess some model's discrimination ability, independently of the response criteria, and also establish the optimal response criteria, given the needs and constraints of whatever that you are measuring.&#xA;Tests like hi-square cannot help at all in this because even if your testing if the predictions are at chance level, many different Hit-False Alarm pairs are consistent with chance level.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some frameworks, like signal detection theory, assume a priori that the evidence available for discrimination has specific distribuiton (e.g., normal distribution, or gamma distribution). When these assumptions hold (or are pretty close), some really nice measures are available that make your life easier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;hope this helps to elucidate you on the advantages of ROCs &lt;/p&gt;&#xA;" OwnerUserId="447" LastActivityDate="2010-08-04T19:31:22.247" CommentCount="1" />
  <row Id="1265" PostTypeId="2" ParentId="1256" CreationDate="2010-08-04T19:35:08.557" Score="2" ViewCount="" Body="&lt;p&gt;This really depends on the relative numbers of &quot;scores&quot; and &quot;subjects&quot;. The method you use calculates lots of cross-correlations which are not required. However, if there are relatively few &quot;subjects&quot; relative to &quot;scores&quot;, then this probably doesn't matter too much, and the method you suggest is probably as good as anything, as it uses a small number of efficient blas operations. However, if there are a large number of &quot;subjects&quot; relative to scores, then it may well be quicker to loop over the rows computing the correlation for each pair separately, using the code suggested by &quot;mbq&quot;. &lt;/p&gt;&#xA;" OwnerUserId="643" LastActivityDate="2010-08-04T19:35:08.557" CommentCount="1" />
  <row Id="1266" PostTypeId="1" AcceptedAnswerId="2338" CreationDate="2010-08-04T20:01:07.787" Score="1" ViewCount="1329" Body="&lt;p&gt;Hello all,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The following question is one of those holy grails for me for some time now, I hope someone might be able to offer a good advice.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wish to perform a non-parametric repeated measures multiway anova using R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have been doing some online searching and reading for some time, and so far was able to find solutions for only some of the cases: friedman test for one way nonparametric repeated measures anova, ordinal regression with {car} Anova function for multi way nonparametric anova, and so on.  The partial solutions is NOT what I am looking for in this question thread.  I have summarized my findings so far in a post I published some time ago (titled: &lt;a href=&quot;http://www.r-statistics.com/2010/04/repeated-measures-anova-with-r-tutorials/&quot; rel=&quot;nofollow&quot;&gt;Repeated measures ANOVA with R (functions and tutorials)&lt;/a&gt;, in case it would help anyone) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If what I read online is true, this task might be achieved using a mixed Ordinal Regression model (a.k.a: Proportional Odds Model).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found two packages that seems relevant, but couldn't find any vignette on the subject:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://cran.r-project.org/web/packages/repolr/&quot; rel=&quot;nofollow&quot;&gt;http://cran.r-project.org/web/packages/repolr/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://cran.r-project.org/web/packages/ordinal/&quot; rel=&quot;nofollow&quot;&gt;http://cran.r-project.org/web/packages/ordinal/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;So being new to the subject matter, I was hoping for some directions from people here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any tutorials/suggested-reading on the subject?  Even better, can someone suggest a simple example code for how to run and analyse this in R (e.g: &quot;non-parametric repeated measures multiway anova&quot;) ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for any help,&#xA;Tal&lt;/p&gt;&#xA;" OwnerUserId="253" LastActivityDate="2010-09-03T09:41:58.997" Title="A non-parametric repeated-measures multi-way Anova in R ?" Tags="&lt;r&gt;&lt;nonparametric&gt;&lt;anova&gt;" AnswerCount="3" FavoriteCount="1" />
  <row Id="1267" PostTypeId="2" ParentId="1164" CreationDate="2010-08-04T20:27:57.627" Score="-2" ViewCount="" Body="&lt;p&gt;10 year lag: you gotta remenber that there isn't much &#xA;valid/usefull stuff in robust statistics before 2000.&lt;/p&gt;&#xA;" OwnerUserId="603" LastActivityDate="2010-08-04T20:27:57.627" CommentCount="6" />
  <row Id="1268" PostTypeId="1" AcceptedAnswerId="1393" CreationDate="2010-08-04T20:51:04.053" Score="9" ViewCount="615" Body="&lt;p&gt;I am using Singular Value Decomposition as a dimensionality reduction technique.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given &lt;code&gt;N&lt;/code&gt; vectors of dimension &lt;code&gt;D&lt;/code&gt;, the idea is to represent the features in a transformed space of uncorrelated dimensions, which condenses most of the information of the data in the eigenvectors of this space in a decreasing order of importance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I am trying to apply this procedure to time series data. The problem is that not all the sequences have the same length, thus I cant really build the &lt;code&gt;num-by-dim&lt;/code&gt; matrix and apply SVD. My first thought was to pad the matrix with zeros by building a &lt;code&gt;num-by-maxDim&lt;/code&gt; matrix and filling the empty spaces with zeros, but I'm not so sure if that is the correct way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is how do you the SVD approach of dimensionality reduction to time series of different length? Alternatively are there any other similar methods of eigenspace representation usually used with time series?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Below is a piece of MATLAB code to illustrate the idea:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X = randn(100,4);                       % data matrix of size N-by-dim&#xA;&#xA;X0 = bsxfun(@minus, X, mean(X));        % standarize&#xA;[U S V] = svd(X0,0);                    % SVD&#xA;variances = diag(S).^2 / (size(X,1)-1); % variances along eigenvectors&#xA;&#xA;KEEP = 2;                               % number of dimensions to keep&#xA;newX = U(:,1:KEEP)*S(1:KEEP,1:KEEP);    % reduced and transformed data&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(I am coding mostly in MATLAB, but I'm comfortable enough to read R/Python/.. as well)&lt;/p&gt;&#xA;" OwnerUserId="170" LastEditorUserId="223" LastEditorDisplayName="" LastEditDate="2010-12-17T07:59:12.010" LastActivityDate="2010-12-17T07:59:12.010" Title="SVD dimensionality reduction for time series of different length" Tags="&lt;time-series&gt;&lt;machine-learning&gt;&lt;pca&gt;&lt;data-transformation&gt;&lt;multivariate-analysis&gt;" AnswerCount="5" CommentCount="4" FavoriteCount="4" />
  <row Id="1269" PostTypeId="2" ParentId="1268" CreationDate="2010-08-04T21:10:02.473" Score="2" ViewCount="" Body="&lt;p&gt;Filling with zero is bad. Try filling with resampling using observations from the past.  &lt;/p&gt;&#xA;" OwnerUserId="223" LastActivityDate="2010-08-04T21:10:02.473" CommentCount="1" />
  <row Id="1270" PostTypeId="1" AcceptedAnswerId="1276" CreationDate="2010-08-04T21:12:03.857" Score="3" ViewCount="165" Body="&lt;p&gt;If I have a (financial) time series, and I sample it with two different periods, at 5 and at 60 minute intervals, can I create an exponential moving average on the 5 minute sampled data which is the same as an exponential moving average on the 60 minute sampled data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Something like this:&lt;br&gt;&#xA;e1 = EMA(a1) applied on sampled_data(60 min)&lt;br&gt;&#xA;e2 = EMA(a2) applied on sampled_data(5 min)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;a1 and a2 are the smoothing factors of the exponential moving average (the period)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can I compute the a2 value for any a1 value, such that e1 = e2?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I say that e1 = e2 I mean that if I graph the values of the EMA computed from 5 min data on top of the 60 min data chart and EMA, the two EMAs should be superposed. This means that in between two data points for EMA(60 min) there will be 60/5=12 data points for EMA(5 min).&lt;/p&gt;&#xA;" OwnerUserId="749" LastActivityDate="2010-08-04T23:36:10.817" Title="Exponential moving averages of a time series with varying sampling" Tags="&lt;time-series&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="1271" PostTypeId="2" ParentId="1266" CreationDate="2010-08-04T21:16:37.787" Score="3" ViewCount="" Body="&lt;p&gt;When in doubt, bootstrap!  Really, I don't know of a canned procedure to handle such a scenario.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bootstrapping is a generally applicable way of generating some error parameters from the data at hand.  Rather than relying on the typical parametric assumptions, bootstrap procedures capitalize on the characteristics of the sample to generate an empirical distribution against which your sample estimates can be compared.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Google scholar is gold...it's been done before...at least once.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lunneborg, Clifford E.; Tousignant, James P.; &#xA;1985&#xA;&quot;Efron's Bootstrap with Application to the Repeated Measures Design.&quot;&#xA;Multivariate Behavioral Research; Apr85, Vol. 20 Issue 2, p161, 18p&lt;/p&gt;&#xA;" OwnerUserId="485" LastEditorUserId="485" LastEditorDisplayName="" LastEditDate="2010-08-04T21:27:49.517" LastActivityDate="2010-08-04T21:27:49.517" CommentCount="2" />
  <row Id="1272" PostTypeId="1" CreationDate="2010-08-04T21:30:19.030" Score="0" ViewCount="173" Body="&lt;p&gt;I have a set of data which consists of many different types (measurable, categorical)&#xA;For example:&#xA;name   measurable_attribute_1   categorical_attribute_1   measurable_attribute_2   categorical_attribute_2 ...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Number of attributes may grows quite quickly during my study: into my spreadsheet, I can as many new entries as attribute... I have about a hundred of entries in this classification scheme, about 70 attribute, so far, and I am at the beginning of my data collection.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to perform statistical analysis of this data set. For example, what are the common features of the entries that have a similar categorical_attribute and this range of values of measurable_attribute.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Well, I would like to generate relationships between attributes in order to create training images. &#xA;However, I am not sure of how to organize the data prior to classification. Even though, should I organize the data?... (referring to this &lt;a href=&quot;http://stats.stackexchange.com/questions/47/clustering-of-large-heavy-tailed-dataset&quot;&gt;question&lt;/a&gt;) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, I can hardly gather entries into classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I do not want to introduce any bias obviously.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am also quite new to statistical analysis (but eager to learn).&lt;/p&gt;&#xA;" LastActivityDate="2010-08-05T16:33:34.410" Title="How to organize a dataset with many attributes" Tags="&lt;algorithms&gt;&lt;categorical-data&gt;&lt;classification&gt;" AnswerCount="2" />
  <row Id="1273" PostTypeId="2" ParentId="1228" CreationDate="2010-08-04T21:54:28.677" Score="2" ViewCount="" Body="&lt;p&gt;Change the variable. Run a control chart for the &quot;time between infections&quot; variable. That way, instead of a discrete variable with a very small range of values, you have a continuous variable with an adequate range of values. If the interval between infections gets too small, the chart will give an &quot;out of control&quot; indication.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This procedure was recommended by Donald Wheeler in &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0945320531&quot; rel=&quot;nofollow&quot;&gt;Understanding Variation: The Key to Managing Chaos&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="666" LastEditorUserId="666" LastEditorDisplayName="" LastEditDate="2010-08-04T23:48:08.457" LastActivityDate="2010-08-04T23:48:08.457" CommentCount="2" />
  <row Id="1274" PostTypeId="1" AcceptedAnswerId="1277" CreationDate="2010-08-04T22:17:59.627" Score="4" ViewCount="154" Body="&lt;p&gt;[0,1,0,2,4,1,0,1,5,1,4,2,1,3,1,1,1,1,0,1,1,0,2,0,2,0,0,1,0,1,2,2,1,2,4,1,4,1,0,0,4,1,0,1,0,1,1,2,1,1,0,0]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the best way to convince myself that these data are correlated? that no univariate discrete distribution would approximate them well? that a time series model is necessary to better estimate the future distribution of counts?&lt;/p&gt;&#xA;" OwnerUserId="273" LastActivityDate="2010-08-07T08:40:07.287" Title="Best way to show these or similar count data are not independent?" Tags="&lt;correlation&gt;&lt;count-data&gt;" AnswerCount="3" CommentCount="3" FavoriteCount="1" />
  <row Id="1275" PostTypeId="2" ParentId="1268" CreationDate="2010-08-04T22:46:30.457" Score="1" ViewCount="" Body="&lt;p&gt;You could estimate univariate time series models for the 'short' series and extrapolate them into the future to 'align' all the series.&lt;/p&gt;&#xA;" LastActivityDate="2010-08-04T22:46:30.457" CommentCount="3" />
  <row Id="1276" PostTypeId="2" ParentId="1270" CreationDate="2010-08-04T23:36:10.817" Score="4" ViewCount="" Body="&lt;p&gt;The trivial and non-helpful answer is &quot;Yes, downsample your 5-minute data to 60-minute data.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More practically, without throwing out 90% of your data, the answer is generally &quot;No, unless you get extremely lucky with sampling your five-minute data.&quot;  You should get an answer that's &lt;em&gt;close&lt;/em&gt; (and under most noise models I suspect they'll be equal in expectation) just by rescaling your smooth by a factor of 12, but any source of randomness in your data is going to cause some difference in the two curves on a point-by-point basis.&lt;/p&gt;&#xA;" OwnerUserId="61" LastActivityDate="2010-08-04T23:36:10.817" />
  <row Id="1277" PostTypeId="2" ParentId="1274" CreationDate="2010-08-04T23:46:28.333" Score="4" ViewCount="" Body="&lt;p&gt;You could just plot the ACF and check if the first coefficient is inside the critical values. The critical values are ok for non-Gaussian time series (at least asymptotically).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively, fit a simple count time series model such as the INAR(1) and see if the coefficient is significantly different from zero.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-08-04T23:46:28.333" CommentCount="3" />
  <row Id="1278" PostTypeId="1" CreationDate="2010-08-05T01:24:22.440" Score="2" ViewCount="687" Body="&lt;p&gt;(This is part-2 of my long question, you can have a look at part-1 &lt;a href=&quot;http://stats.stackexchange.com/questions/1099/how-to-handle-count-data-categorical-data-when-it-has-been-converted-to-a-rate&quot;&gt;here&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am going to do a quasi-experiment, with measuring the base line of a sample (actually not quite a sample, but a ward, with high patient turn-over rate), and then we do a intervention, and measure the variables (i.e. infection rate) again.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I googled a bit and found that this is something called a single case experiment, and it was said that single case experiment doesn't have very solid statistics because you don't have the control, you can't conclude on the causality in a solid manner.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have googled a bit again and found that I can compare the incidence rate (or call it infection rate), but doing something like &quot;incidence rate difference&quot; (IRD) or &quot;incidence rate ratio&quot; (IRR). (I found it from &lt;a href=&quot;http://www.statsdirect.com/help/rates/incidence_rates.htm&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the difference between IRD and t-test? And is there any statistical test complementary for IRR?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But mostly importantly, is it appropriate for me to use this test (does it have a name?) for single case experiment? Because the patients in the ward keep changing, this is what I worried about.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks again!&lt;/p&gt;&#xA;" OwnerUserId="588" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-11-23T17:05:55.630" LastActivityDate="2010-11-23T17:05:55.630" Title="Is calculating Incidence Rate Difference/Ratio appropriate for single case experimental design?" Tags="&lt;epidemiology&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="1279" PostTypeId="2" ParentId="485" CreationDate="2010-08-05T02:36:08.460" Score="0" ViewCount="" Body="&lt;p&gt;One resource I found recently was the UCCS video archive.&#xA;It has a couple of archived video series for subjects called Mathematical Statistics I and  Mathematical Statistics II. It requires a free registration to access.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.uccs.edu/~math/vidarchive.html&quot; rel=&quot;nofollow&quot;&gt;http://www.uccs.edu/~math/vidarchive.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This videos for &lt;a href=&quot;http://cmes.uccs.edu/Fall2007/Math481/archive.php&quot; rel=&quot;nofollow&quot;&gt;Math 481 Mathematical Stats I - Fall 2007&lt;/a&gt; provide a good intro to mathematical statistics.&lt;/p&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-08-09T07:20:11.550" LastActivityDate="2010-08-09T07:20:11.550" CommunityOwnedDate="2010-08-05T02:36:08.460" />
  <row Id="1280" PostTypeId="2" ParentId="726" CreationDate="2010-08-05T02:42:11.413" Score="28" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;87% of statistics are made up on the spot&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;-Unknown&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/k2eEh.gif&quot; alt=&quot;Dilbert.com&quot;&gt;&#xA;&lt;a href=&quot;http://dilbert.com/strips/comic/2008-05-08/&quot; rel=&quot;nofollow&quot;&gt;Dilbert.com&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="553" LastEditorUserId="442" LastEditorDisplayName="" LastEditDate="2011-01-14T09:38:16.137" LastActivityDate="2011-01-14T09:38:16.137" CommunityOwnedDate="2010-08-05T02:42:11.413" CommentCount="2" />
  <row Id="1281" PostTypeId="2" ParentId="1272" CreationDate="2010-08-05T02:57:24.893" Score="0" ViewCount="" Body="&lt;p&gt;You could look at Exploratory Factor Analysis. It will tell you which attributes are the most similar to each other.&lt;/p&gt;&#xA;" OwnerUserId="74" LastActivityDate="2010-08-05T02:57:24.893" />
  <row Id="1282" PostTypeId="2" ParentId="1261" CreationDate="2010-08-05T03:06:36.013" Score="4" ViewCount="" Body="&lt;p&gt;Following up on my comment, this question would be very simple if i, j, and k were not restricted to be integers. The reason is as follows:  pA, pB, and pC denote the observed probability of success in the three groups. Then let p=pA, i=1, j=log(pB)/log(pA), and k=log(pC)/log(pA). These will easily satisfy the required conditions (except for j and k being between 1 and 5, but that looks like an ad-hoc simplifying assumption instead of a real constraint).&#xA;In fact, if you do this with the given data, you get j=2.009 and k=2.884 which I think prompted the original question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is even possible to get standard errors for these quantities (or rather their logarithm). Note that if pB = p^j, then log(-log(pB)) = log(j) + log(-log(p)), so one can use logistic regression with a complimentary log-log link for the number of failures (the complimentary log-log function is log(-log(1-x)) and this link is built in for most statitical software such as R or SAS). Then one could check whether the 95% CIs include integers, or perhaps run a likelihood-ratio (or other) test comparing the fit of the unrestricted model to one where j and k are rounded to the nearest integer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The above assumes that i=1. Something similar could probably be done for other integer i's (probably by having an offset of log(i) in the model - I have not thought it through).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the end, I want to note that you should make sure that your hypothesis is meaningful by itself, and did not come from playing with the data. Otherwise any statistical test is biased because you picked a form of the null hypothesis (out all the possible weird forms that you could have imagined) that is likely to fit.&lt;/p&gt;&#xA;" OwnerUserId="279" LastActivityDate="2010-08-05T03:06:36.013" CommentCount="2" />
  <row Id="1283" PostTypeId="2" ParentId="1274" CreationDate="2010-08-05T03:23:53.207" Score="0" ViewCount="" Body="&lt;p&gt;Immediately, a runs test comes to mind.  That is, if this is an ordered set and you are interested in knowing if the ordered observations are independent.  It tests whether the sequence is likely due to randomness rather than systematic variation.  Without more detail, that's the best I can come up with.&lt;/p&gt;&#xA;" OwnerUserId="485" LastActivityDate="2010-08-05T03:23:53.207" />
  <row Id="1284" PostTypeId="2" ParentId="1266" CreationDate="2010-08-05T03:43:12.873" Score="1" ViewCount="" Body="&lt;p&gt;Consider checking out &lt;a href=&quot;https://r-forge.r-project.org/projects/gmpm/&quot; rel=&quot;nofollow&quot;&gt;gmpm&lt;/a&gt;, but it is still in development.&lt;/p&gt;&#xA;" OwnerUserId="196" LastActivityDate="2010-08-05T03:43:12.873" CommentCount="1" />
  <row Id="1285" PostTypeId="2" ParentId="1272" CreationDate="2010-08-05T03:55:13.897" Score="1" ViewCount="" Body="&lt;p&gt;One option would involve using &lt;strong&gt;optimal scaling principal components analysis&lt;/strong&gt;.&#xA;The approach allows you to state your measurement assumptions about each variable (e.g., nominal, ordinal, numeric).&#xA;I've used it in SPSS: see the Categories Add-On module (i.e., Analyze - Dimension Reduction - Optimal Scaling).&#xA;I'm not sure, but the homals package in R may also implement the procedure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A quick Google ( &lt;a href=&quot;http://www.google.com/search?sourceid=chrome&amp;amp;ie=UTF-8&amp;amp;q=optimal+scaling+principal+components&quot; rel=&quot;nofollow&quot;&gt;http://www.google.com/search?sourceid=chrome&amp;amp;ie=UTF-8&amp;amp;q=optimal+scaling+principal+components&lt;/a&gt;) &#xA;revealed this reference:&#xA;&lt;a href=&quot;http://takane.brinkster.net/Yoshio/p009.pdf&quot; rel=&quot;nofollow&quot;&gt;http://takane.brinkster.net/Yoshio/p009.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-08-05T04:14:12.710" LastActivityDate="2010-08-05T04:14:12.710" CommentCount="1" />
  <row Id="1286" PostTypeId="1" AcceptedAnswerId="1335" CreationDate="2010-08-05T04:54:46.270" Score="3" ViewCount="606" Body="&lt;p&gt;Sometimes I want to do an exact test by examining all possible combinations of the data to build an empirical distribution against which I can test my observed differences between means.  To find the possible combinations I'd typically use the combn function.  The choose function can show me how many possible combinations there are.  It is very easy for the number of combinations to get so large that it is not possible to store the result of the combn function, e.g. combn(28,14) requires a 2.1 Gb vector.  So I tried writing an object that steped through the same logic as the combn function in order to provide the values off an imaginary &quot;stack&quot; one at a time.  However, this method (as I instantiated it) is easily 50 times slower than combn at reasonable combination sizes, leading me to think it will also be painfully slow for larger combination sizes.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a better algorithm for doing this sort of thing than the algorithm used in combn?Specifically is there a way to generate and pull the Nth possible combination without calculating through all previous combinations?&lt;/p&gt;&#xA;" OwnerUserId="196" LastEditorUserId="196" LastEditorDisplayName="" LastEditDate="2010-08-05T08:21:17.047" LastActivityDate="2010-08-06T01:40:54.227" Title="How can I obtain some of all possible combinations in R?" Tags="&lt;r&gt;&lt;nonparametric&gt;&lt;permutation&gt;&lt;combination&gt;" AnswerCount="2" CommentCount="7" FavoriteCount="1" />
  <row Id="1287" PostTypeId="2" ParentId="1252" CreationDate="2010-08-05T04:58:19.663" Score="5" ViewCount="" Body="&lt;p&gt;Reading &quot;Using Multivariate Statistics (4th Edition) Barbara G. Tabachnick&quot; &#xA;I found these decision trees based on major research question. I think they are quite useful. Following this link you'll find an extract of the book&#xA;&lt;a href=&quot;http://www.psychwiki.com/images/d/d8/TF2.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.psychwiki.com/images/d/d8/TF2.pdf&lt;/a&gt;&#xA;see pages 29 to 31&lt;/p&gt;&#xA;" OwnerUserId="694" LastActivityDate="2010-08-05T04:58:19.663" CommentCount="2" />
  <row Id="1288" PostTypeId="2" ParentId="165" CreationDate="2010-08-05T09:15:34.467" Score="4" ViewCount="" Body="&lt;p&gt;So there are plenty of answers here paraphrased from statistics/probability textbooks, Wikipedia, etc. I believe we have &quot;laypersons&quot; where i work; i think they are in the marketing department. If i ever have to explain anything technical to them, i apply the rule &quot;show don't tell.&quot; With that rule in mind, i would probably show them something like this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The idea here is to try to code an algorithm that i can teach to spell--not by learning all of the hundreds (thousands?) of rules like &lt;em&gt;When adding an ending to a word that ends with a silent e, drop the final e if the ending begins with a vowel&lt;/em&gt;. One reason that won't work is i don't know those rules (i'm not even sure the one i just recited is correct). Instead i am going to teach it to spell by showing it a bunch of correctly spelled words and letting it extract the rules from those words, which is more or less the essence of Machine Learning, regardless of the algorithm--pattern extraction and pattern recognition.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The success criterion is correctly spelling a word the algorithm has never seen before (i realize that can happen by pure chance, but that won't occur to the marketing guys, so i'll ignore--plus i am going to have the algorithm attempt to spell not one word, but a lot, so it's not likely we'll be deceived by a few lucky guesses).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An hour or so ago, i downloaded (as a plain text file) from the excellent Project Gutenberg Site, the Herman Hesse novel &lt;em&gt;Siddhartha&lt;/em&gt;. I'll use the words in this novel to teach the algorithm how to spell.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So i coded the algorithm below that scanned this novel, three letters at a time (each word has one additional character at the end, which is 'whitespace', or the end of the word). Three-letter sequences can tell you a lot--for instance, the letter 'q' is nearly always followed by 'u'; the sequence 'ty' usually occurs at the end of a word; z rarely does, and so forth. (Note: i could just as easily have fed it entire words in order to train it to speak in complete sentences--exactly the same idea, just a few tweaks to the code.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;None of this involves MCMC though, that happens after training, when we give the algorithm a few random letters (as a seed) and it begins forming 'words'. How does the algorithm build words? Imagine that it has the block 'qua'; what letter does it add next? During training, the algorithm constructed a massive l*etter-sequence frequency matrix* from all of the thousands of words in the novel. Somewhere in that matrix is the three-letter block 'qua' and the frequencies for the characters that could follow the sequence. The algorithm selects a letter based on those frequencies that could possibly follow it. So the letter that the algorithm selects next depends on--and solely on--the last three in its word-construction queue.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So that's a Markov Chain Monte Carlo algorithm. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think perhaps the best way to illustrate how it works is to show the results based on different levels of training. Training level is varied by changing the number of passes the algorithm makes though the novel--the more passes thorugh the greater the fidelity of its letter-sequence frequency matrices.  Below are the results--in the form of 100-character strings output by the algorithm--after training on the novel 'Siddharta'.&lt;hr&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;A single pass through the novel, &lt;em&gt;Siddhartha&lt;/em&gt;:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;em&gt;then whoicks ger wiff all mothany stand ar you livid theartim mudded&#xA;  sullintionexpraid his sible his&lt;/em&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;(Straight away, it's learned to speak almost perfect Welsh; i hadn't expected that.)&lt;hr&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;After two passes through the novel:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;em&gt;the ack wor prenskinith show wass an twor seened th notheady theatin land&#xA;  rhatingle was the ov there&lt;/em&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;After 10 passes:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;em&gt;despite but the should pray with ack now have water her dog lever pain feet&#xA;  each not the weak memory&lt;/em&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;hr&gt; &#xA;&#xA;&lt;p&gt;And here's the code (in Python, i'm nearly certain that this could be done in R using an MCMC package, of which there are several, in just 3-4 lines)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def create_words_string(raw_string) :&#xA;  &quot;&quot;&quot; in case i wanted to use training data in sentence/paragraph form; &#xA;      this function will parse a raw text string into a nice list of words;&#xA;      filtering: keep only words having  more than 3 letters and remove &#xA;      punctuation, etc.&#xA;  &quot;&quot;&quot;&#xA;  pattern = r'\b[A-Za-z]{3,}\b'&#xA;  pat_obj = re.compile(pattern)&#xA;  words = [ word.lower() for word in pat_obj.findall(raw_string) ]&#xA;  pattern = r'\b[vixlm]+\b'&#xA;  pat_obj = re.compile(pattern)&#xA;  return &quot; &quot;.join([ word for word in words if not pat_obj.search(word) ])&#xA;&#xA;def create_markov_dict(words_string):&#xA;  # initialize variables&#xA;  wb1, wb2, wb3 = &quot; &quot;, &quot; &quot;, &quot; &quot;&#xA;  l1, l2, l3 = wb1, wb2, wb3&#xA;  dx = {}&#xA;  for ch in words_string :&#xA;    dx.setdefault( (l1, l2, l3), [] ).append(ch)&#xA;    l1, l2, l3 = l2, l3, ch&#xA;  return dx&#xA;&#xA;def generate_newtext(markov_dict) :&#xA;  simulated_text = &quot;&quot;&#xA;  l1, l2, l3 = &quot; &quot;, &quot; &quot;, &quot; &quot;&#xA;  for c in range(100) :&#xA;    next_letter = sample( markov_dict[(l1, l2, l3)], 1)[0]&#xA;    simulated_text += next_letter&#xA;    l1, l2, l3 = l2, l3, next_letter&#xA;  return simulated_text&#xA;&#xA;if __name__==&quot;__main__&quot; :&#xA;  # n = number of passes through the training text&#xA;  n = 1&#xA;  q1 = create_words_string(n * raw_str)&#xA;  q2 = create_markov_dict(q1)&#xA;  q3 = generate_newtext(q2)&#xA;  print(q3)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="438" LastActivityDate="2010-08-05T09:15:34.467" CommunityOwnedDate="2010-08-16T06:37:30.433" />
  <row Id="1289" PostTypeId="1" CreationDate="2010-08-05T10:03:34.997" Score="5" ViewCount="374" Body="&lt;p&gt;I am having difficulties to select the right way to visualize data. Let's say we have &lt;strong&gt;bookstores&lt;/strong&gt; that sells &lt;strong&gt;books&lt;/strong&gt;, and every book has at least one &lt;strong&gt;category&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a bookstore, if we count all the categories of books, we acquire a histogram that shows the number of books that falls into a specific category for that bookstore.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to visualize the bookstore behavior, I want to see if they favor a category over other categories. I don't want to see if they are favoring sci-fi all together, but I want to see if they are treating every category equally or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have ~1M bookstores.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have thought of 4 methods:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Sample the data, show only 500 bookstore's histograms. Show them in 5 separate pages using 10x10 grid. Example of a 4x4 grid:&#xA;&lt;img src=&quot;http://i37.tinypic.com/2aes2uo.jpg2.&quot; alt=&quot;multiple histograms 1&quot;&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Same as #1. But this time sort x axis values according to their count desc, so if there is a favoring it will be seen easily.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Imagine putting the histograms in #2 together like a deck and showing them in 3D. Something like this: &lt;img src=&quot;http://www.statsoft.com/textbook/popups/popup110.gif&quot; alt=&quot;3D histogram&quot;&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Instead of using third axis suing color to represent colors, so using a heatmap (2D histogram): &lt;img src=&quot;http://i38.tinypic.com/1ptfrm.png&quot; alt=&quot;2D histogram&quot;&gt;&#xA;If generally bookstores prefer some categories to others it will be displayed as a nice gradient from left to right.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Do you have any other visualization ideas/tools to represent multiple histograms?&lt;/p&gt;&#xA;" OwnerUserId="760" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-08-09T10:43:44.003" LastActivityDate="2010-08-09T10:43:44.003" Title="Visualizing multiple &quot;histograms&quot;" Tags="&lt;data-visualization&gt;&lt;pca&gt;" AnswerCount="2" CommentCount="7" FavoriteCount="3" />
  <row Id="1290" PostTypeId="2" ParentId="1286" CreationDate="2010-08-05T10:17:47.227" Score="0" ViewCount="" Body="&lt;p&gt;Generating combinations is pretty easy, see for instance &lt;a href=&quot;http://compprog.wordpress.com/2007/10/17/generating-combinations-1/&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt;; write this code in R and then process each combination at a time it appears.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-08-05T10:17:47.227" CommentCount="3" />
  <row Id="1291" PostTypeId="2" ParentId="1289" CreationDate="2010-08-05T10:31:54.357" Score="9" ViewCount="" Body="&lt;p&gt;As you have found out there are no easy answers to your question!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I presume that you interested in finding strange or different book stores? If this is the case then you could try things like &lt;a href=&quot;http://stats.stackexchange.com/questions/53/pca-on-correlation-or-covariance/78#78&quot;&gt;PCA&lt;/a&gt; (see the wikipedia &lt;a href=&quot;http://en.wikipedia.org/wiki/Cluster_analysis&quot; rel=&quot;nofollow&quot;&gt;cluster analysis&lt;/a&gt; page for more details).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To give you an idea, consider this example. You have 26 bookshops (with names A, B,..Z). All bookshops are similar, except:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Shop Z sells only a few History books.&lt;/li&gt;&#xA;&lt;li&gt;Shops O-Y sell more romance books than average.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;A principal components plot highlights these shops for further investigation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's some sample R code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; d = data.frame(Romance = rpois(26, 50), Horror = rpois(26, 100), &#xA;               Science = rpois(26, 75), History = rpois(26, 125))&#xA;&amp;gt; rownames(d) = LETTERS&#xA;#Alter a few shops&#xA;&amp;gt; d[15:25,][1] = rpois(11,150)&#xA;&amp;gt; d[26,][4] = rpois(1, 10)&#xA;#look at the data&#xA;&amp;gt; head(d, 2)&#xA;       Romance Horror Science History&#xA; A      36    107      62     139&#xA; B      47     93      64     118&#xA;&amp;gt; books.PC.cov = prcomp(d)&#xA;&amp;gt; books.scores.cov = predict(books.PC.cov)&#xA;# Plot of PC1 vs PC2&#xA;&amp;gt; plot(books.scores.cov[,1],books.scores.cov[,2],&#xA;       xlab=&quot;PC 1&quot;,ylab=&quot;PC 2&quot;, pch=NA)&#xA;&amp;gt; text(books.scores.cov[,1],books.scores.cov[,2],labels=LETTERS)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This gives the following plot: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://img265.imageshack.us/img265/7263/tmplx.jpg&quot; alt=&quot;PCA plot&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Notice that:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Shop z is an outlying point.&lt;/li&gt;&#xA;&lt;li&gt;The others shops form two distinct groups.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Other possibilities&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could also look at &lt;a href=&quot;http://www.ggobi.org/&quot; rel=&quot;nofollow&quot;&gt;GGobi&lt;/a&gt;, I've never used it, but it looks interesting.&lt;/p&gt;&#xA;" OwnerUserId="8" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-08-05T12:58:40.073" LastActivityDate="2010-08-05T12:58:40.073" CommentCount="5" />
  <row Id="1292" PostTypeId="1" AcceptedAnswerId="1297" CreationDate="2010-08-05T10:42:44.327" Score="5" ViewCount="327" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Decision_tree&quot; rel=&quot;nofollow&quot;&gt;Decision trees&lt;/a&gt; seems to be a very understandable machine learning method.&#xA;Once created it can be easily inspected by a human which is a great advantage in some applications. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are the practical weak sides of Decision Trees?&lt;/p&gt;&#xA;" OwnerUserId="217" LastActivityDate="2010-08-05T13:31:59.900" Title="What is the weak side of decision trees?" Tags="&lt;machine-learning&gt;&lt;nonparametric&gt;&lt;decision-trees&gt;" AnswerCount="3" FavoriteCount="3" />
  <row Id="1293" PostTypeId="1" CreationDate="2010-08-05T10:47:12.837" Score="0" ViewCount="175" Body="&lt;p&gt;My girlfriend (B.B.A) is really interested in Actuarial science. She's looking at self teaching her self. She's good with basic math (Calculus 1 and 2) and stats&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are some of the essential sources she needs to read in order to learn and excel in the field?&lt;/p&gt;&#xA;" OwnerUserId="59" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-08-07T17:47:26.950" LastActivityDate="2010-08-07T17:47:26.950" CommunityOwnedDate="2010-08-05T14:32:53.243" Title="Essential reads for people interested in Actuary and Actuarial science" Tags="&lt;books&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="1294" PostTypeId="2" ParentId="1289" CreationDate="2010-08-05T11:22:54.870" Score="1" ViewCount="" Body="&lt;p&gt;I would suggest something that hasn't got a defined name (probably &quot;parallel plot&quot;) and looks like this:  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://rosuda.org/iPlots/plots/pcp.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically you plot all counts for all bookstores as points over categories listed on x axis and connect the results from each bookstore with a line. Still this may be too tangled for 1M lines, though. The concept comes from GGobi which was already mentioned by csgillespie.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-08-05T11:22:54.870" CommentCount="3" />
  <row Id="1295" PostTypeId="2" ParentId="1292" CreationDate="2010-08-05T11:58:42.133" Score="5" ViewCount="" Body="&lt;p&gt;One disadvantage is that all terms are assumed to interact. That is, you can't have two explanatory variables that behave independently. Every variable in the tree is forced to interact with every variable further up the tree. This is extremely inefficient if there are variables that have no or weak interactions.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-08-05T11:58:42.133" CommentCount="5" />
  <row Id="1296" PostTypeId="1" AcceptedAnswerId="1353" CreationDate="2010-08-05T11:59:25.080" Score="6" ViewCount="485" Body="&lt;p&gt;I have detector which will detect an event with some probability &lt;em&gt;p&lt;/em&gt;. If the detector says that an event occured, then that is always the case, so there are not false-positives. After I run it for some time, I get &lt;em&gt;k&lt;/em&gt; events detected. I would like to calculate what the total number of events that occured was, detected or otherwise, with some confidence, say 95%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So for example, let's say I get 13 events detected. I would like to be able to calculate that there were between 13 and 19 events with 95% confidence based on &lt;em&gt;p&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's what I've tried so far:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The probability of detecting &lt;em&gt;k&lt;/em&gt; events if there were &lt;em&gt;n&lt;/em&gt; total is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;binomial(n, k) * p^k * (1 - p)^(n - k)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The sum of that over &lt;em&gt;n&lt;/em&gt; from &lt;em&gt;k&lt;/em&gt; to infinity is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;1/p&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which means, that the probability of there being &lt;em&gt;n&lt;/em&gt; events total is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;f(n) = binomial(n, k) * p^(k + 1) * (1 - p)^(n - k)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So if I want to be 95% sure I should find the first partial sum &lt;code&gt;f(k) + f(k+1) + f(k+2) ... + f(k+m)&lt;/code&gt; which is at least 0.95 and the answer is &lt;code&gt;[k, k+m]&lt;/code&gt;. Is this the correct approach? Also is there a closed formula for the answer?&lt;/p&gt;&#xA;" OwnerUserId="762" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-29T00:26:13.040" LastActivityDate="2011-04-29T00:26:13.040" Title="How to find a confidence interval for the total number of events" Tags="&lt;probability&gt;&lt;confidence-interval&gt;" AnswerCount="4" FavoriteCount="1" />
  <row Id="1297" PostTypeId="2" ParentId="1292" CreationDate="2010-08-05T12:08:23.523" Score="8" ViewCount="" Body="&lt;p&gt;Here are a couple I can think of:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;They can be extremely sensitive to small perturbations in the data: a slight change can result in a drastically different tree.&lt;/li&gt;&#xA;&lt;li&gt;They can easily overfit. This can be negated by validation methods and pruning, but this is a grey area.&lt;/li&gt;&#xA;&lt;li&gt;They can have problems out-of-sample prediction (this is related to them being non-smooth).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Some of these are related to the problem of &lt;a href=&quot;http://en.wikipedia.org/wiki/Multicollinearity&quot; rel=&quot;nofollow&quot;&gt;multicollinearity&lt;/a&gt;: when two variables both explain the same thing, a decision tree will greedily choose the best one, whereas many other methods will use them both. Ensemble methods such as random forests can negate this to a certain extent, but you lose the ease of understanding.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However the biggest problem, from my point of view at least, is the lack of a principled probabilistic framework. Many other methods have things like confidence intervals, posterior distributions etc., which give us some idea of how good a model is. A decision tree is ultimately an ad hoc heuristic, which can still be very useful (they are excellent for finding the sources of bugs in data processing), but there is the danger of people treating the output as &quot;the&quot; correct model (from my experience, this happens a lot in marketing).&lt;/p&gt;&#xA;" OwnerUserId="495" LastEditorUserId="495" LastEditorDisplayName="" LastEditDate="2010-08-05T13:31:59.900" LastActivityDate="2010-08-05T13:31:59.900" CommentCount="3" />
  <row Id="1298" PostTypeId="2" ParentId="1296" CreationDate="2010-08-05T12:12:16.980" Score="2" ViewCount="" Body="&lt;p&gt;I think you misunderstood the purpose of confidence intervals. Confidence intervals allow you to assess where the true value of the parameter is located. So, in your case, you can construct a confidence interval for $p$. It does not make sense to construct an interval for the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having said that, once you have an estimate of $p$ you can calculate the probability that you will observe different realizations such as 14, 15 etc using the binomial pdf.&lt;/p&gt;&#xA;" LastActivityDate="2010-08-05T12:12:16.980" CommentCount="1" />
  <row Id="1299" PostTypeId="2" ParentId="1292" CreationDate="2010-08-05T12:47:45.727" Score="2" ViewCount="" Body="&lt;p&gt;My answer is directed to CART (the C 4.5/C 5 implementations) though i don't think are limited to it. My guess is that this is what the OP has in mind--it's usually what someone means when they say &quot;Decision Tree.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Limitations of Decision Trees&lt;/strong&gt;:&lt;hr&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Low-Performance&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By 'performance' i don't mean resolution, but &lt;em&gt;execution speed&lt;/em&gt;. The reason why it's poor is that you need to 'redraw the tree' every time you wish to update your CART model--data classified by an already-trained Tree, that you then want to add to the Tree (i.e., use as a training data point) requires that you start from over--training instances can not be added incrementally, as they can for most other supervised learning algorithms. Perhaps the best way to state this is that Decision Trees cannot be trained in online mode, rather only in batch mode. Obviously you won't notice this limitation if you don't update your classifier, but then i would expect that you see a drop in resolution.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is significant because for Multi-Layer Perceptrons for instance, once it's trained, then it can begin classifying data; that data can also be used to 'tune' the already-trained classifier, though with Decision Trees, you need to retrain with the entire data set (original data used in training plus any new instances).&lt;hr&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Poor Resolution on Data With Complex Relationships Among the Variables&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Decision Trees classify by step-wise assessment of a data point of unknown class, one node at time, starting at the root node and ending with a terminal node.  And at each node, only two possibilities are possible (left-right), hence there are some variable relationships that Decision Trees just can't learn.&lt;hr&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Practically Limited to Classification&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Decision Trees work best when they are trained to assign a data point to a class--preferably one of only a few possible classes. I don't believe i have ever had any success using a Decision Tree in regression mode (i.e., continuous output, such as price, or expected lifetime revenue). This is not a formal or inherent limitation but a practical one. Most of the time, Decision Trees are used for prediction of factors or discrete outcomes. &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Poor Resolution With Continuous Expectation Variables&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Again, in principle, it's ok to have independent variables like &quot;download time&quot; or &quot;number of days since previous online purchase&quot;--just change your splitting criterion to variance (it's usually Information Entropy or Gini Impurity for discrete variables) but in my experience Decision Trees rarely work well in these instance. Exceptions are cases like &quot;student's age&quot; which looks continuous but in practice the range of values is quite small (particularly if they are reported as integers). &lt;/p&gt;&#xA;" OwnerUserId="438" LastActivityDate="2010-08-05T12:47:45.727" />
  <row Id="1302" PostTypeId="2" ParentId="1293" CreationDate="2010-08-05T13:04:27.533" Score="1" ViewCount="" Body="&lt;p&gt;sources:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) statistics upto ANOVA&#xA;2) probability upto Central limit theorem&#xA;3) Basic programming and data analysis skills&#xA;4) Familiarity with business economics&#xA;5) Financial mathematics&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a verrry basic smattering of things that she needs to read.&lt;/p&gt;&#xA;" LastActivityDate="2010-08-05T13:04:27.533" CommunityOwnedDate="2010-08-05T14:32:53.243" />
  <row Id="1304" PostTypeId="2" ParentId="1296" CreationDate="2010-08-05T13:35:59.983" Score="3" ViewCount="" Body="&lt;p&gt;If you measure $k$ events and know your detection efficiency is $p$ you can automatically correct your measured result up to the &quot;true&quot; count $k_\mathrm{true} = k/p$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your question is then about finding the range of $k_\mathrm{true}$ where 95%&#xA;of the observations will fall. You can use the &lt;a href=&quot;http://arxiv.org/abs/physics/9711021&quot; rel=&quot;nofollow&quot;&gt;Feldman-Cousins method&lt;/a&gt; to estimate this interval. If you have access to &lt;a href=&quot;http://root.cern.ch/drupal/&quot; rel=&quot;nofollow&quot;&gt;ROOT&lt;/a&gt; there is a class to do this calculation for you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You would calculate the upper and lower limits with Feldman-Cousins from the&#xA;&lt;em&gt;uncorrected&lt;/em&gt; number of events $k$ and then scale them up to 100% with $1/p$.&#xA;This way the actual number of measurements determines your uncertainty, not&#xA;some scaled number that wasn't measured.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;{&#xA;gSystem-&amp;gt;Load(&quot;libPhysics&quot;);&#xA;&#xA;const double lvl = 0.95;&#xA;TFeldmanCousins f(lvl);&#xA;&#xA;const double p = 0.95;&#xA;const double k = 13;&#xA;const double k_true = k/p;&#xA;&#xA;const double k_bg = 0;&#xA;&#xA;const double upper = f.CalculateUperLimit(k, k_bg) / p;&#xA;const double lower = f.GetLowerLimit() / p;&#xA;&#xA;std::cout &amp;lt;&amp;lt; &quot;[&quot;&#xA;  lower &amp;lt;&amp;lt;&quot;...&quot;&amp;lt;&amp;lt;&#xA;  k_true &amp;lt;&amp;lt;&quot;...&quot;&amp;lt;&amp;lt;&#xA;  upper &amp;lt;&amp;lt;&#xA;  &quot;]&quot; &amp;lt;&amp;lt; std::endl;&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="56" LastEditorUserId="56" LastEditorDisplayName="" LastEditDate="2010-08-05T16:34:40.807" LastActivityDate="2010-08-05T16:34:40.807" CommentCount="2" />
  <row Id="1305" PostTypeId="1" CreationDate="2010-08-05T13:37:05.597" Score="1" ViewCount="65" Body="&lt;p&gt;Can anyone point towards applications, SQL algorithms or similar that automate the production of standarized rates/ratios (ie directly/indirectly age/sex etc.-standardized rates/ratios) with confidence intervals from numerator, denominator &amp;amp; reference data?  The process would ideally prevent inappropriate calculations, eg from small counts.&lt;/p&gt;&#xA;" OwnerUserId="713" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-16T07:09:45.213" LastActivityDate="2010-09-16T07:09:45.213" ClosedDate="2010-09-16T07:09:04.720" Title="Automation of standardization" Tags="&lt;confidence-interval&gt;&lt;statistical-significance&gt;&lt;software&gt;&lt;epidemiology&gt;" CommentCount="1" />
  <row Id="1306" PostTypeId="2" ParentId="856" CreationDate="2010-08-05T13:56:03.143" Score="6" ViewCount="" Body="&lt;p&gt;I have a suspicion that 'exact' tests and sampling weights are essentially incompatible concepts. I checked in Stata, which has good facilities for sample surveys and reasonable ones for exact tests, and its 8 possible test statistics for a crosstab with sample weights don't include any 'exact' tests such as Fisher's. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The relevant Stata manual entry (for &lt;em&gt;svy: tabulate twoway&lt;/em&gt;) advises using its default test in all cases. This default method is based on the usual Pearson's chi-squared statistic. To quote: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;To account for the survey design, the statistic is turned into an F statistic with noninteger degrees of freedom by using a second-order Rao and Scott (1981, 1984) correction&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Refs:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Rao, J. N. K., and A. J. Scott. 1981. The analysis of categorical data from complex sample surveys: Chi-squared tests for goodness of fit and independence in two-way tables. Journal of the American Statistical Association 76:221–230.&lt;/li&gt;&#xA;&lt;li&gt;Rao, J. N. K., and A. J. Scott. 1984. On chi-squared tests for multiway contingency tables with cell proportions estimated from survey data. Annals of Statistics 12: 46–60.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-08-05T13:56:03.143" />
  <row Id="1307" PostTypeId="2" ParentId="1293" CreationDate="2010-08-05T13:57:42.843" Score="1" ViewCount="" Body="&lt;p&gt;And for bedtime reading, &quot;Against the Gods: The Remarkable Story of Risk&quot; by Peter L. Bernstein. She'll find out how Lloyds of London started, among many other interesting bits. Highly recommended and only $13 from amazon.&lt;/p&gt;&#xA;" OwnerUserId="247" LastActivityDate="2010-08-05T13:57:42.843" CommunityOwnedDate="2010-08-05T14:32:53.243" />
  <row Id="1308" PostTypeId="1" CreationDate="2010-08-05T14:00:07.723" Score="11" ViewCount="660" Body="&lt;p&gt;In the traditional birthday paradox the question is &quot;what are the chances that two or more people in a group of n people share a birthday&quot;.  I'm stuck on a problem which is an extension of this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead of knowing the probability that two people share a birthday I need to extend the question to know what is the probability that x or more people share a birthday.  With x=2 you can do this by calculating the probability that no two people share a birthday and subtract that from 1, but I don't think I can extend this logic to larger numbers of x.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To further complicate this I also need a solution which will work for very large numbers for n (millions) and x (thousands).&lt;/p&gt;&#xA;" OwnerUserId="765" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-29T00:26:29.467" LastActivityDate="2011-04-29T00:26:29.467" Title="Extending the birthday paradox to more than 2 people" Tags="&lt;probability&gt;&lt;bioinformatics&gt;" AnswerCount="3" CommentCount="5" FavoriteCount="4" />
  <row Id="1309" PostTypeId="2" ParentId="1308" CreationDate="2010-08-05T14:10:26.907" Score="2" ViewCount="" Body="&lt;p&gt;It is always possible to solve this problem with a monte-carlo solution, although that's far from the most efficient.  Here's a simple example of the 2 person problem in R (from &lt;a href=&quot;http://www.meetup.com/nyhackr/calendar/10251302/?from=list&amp;amp;offset=0&quot; rel=&quot;nofollow&quot;&gt;a presentation I gave last year&lt;/a&gt;; I used this as an example of inefficient code), which could be easily adjusted to account for more than 2:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;birthday.paradox &amp;lt;- function(n.people, n.trials) {&#xA;    matches &amp;lt;- 0&#xA;    for (trial in 1:n.trials) {&#xA;        birthdays &amp;lt;- cbind(as.matrix(1:365), rep(0, 365))&#xA;        for (person in 1:n.people) {&#xA;            day &amp;lt;- sample(1:365, 1, replace = TRUE)&#xA;            if (birthdays[birthdays[, 1] == day, 2] == 1) {&#xA;                matches &amp;lt;- matches + 1&#xA;                break&#xA;            }&#xA;            birthdays[birthdays[, 1] == day, 2] &amp;lt;- 1&#xA;        }&#xA;        birthdays &amp;lt;- NULL&#xA;    }&#xA;    print(paste(&quot;Probability of birthday matches = &quot;, matches/n.trials))&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="5" LastEditorUserId="5" LastEditorDisplayName="" LastEditDate="2010-08-05T14:54:11.593" LastActivityDate="2010-08-05T14:54:11.593" CommentCount="3" />
  <row Id="1311" PostTypeId="2" ParentId="1308" CreationDate="2010-08-05T15:27:49.650" Score="1" ViewCount="" Body="&lt;p&gt;This is an attempt at a general solution. There may be some mistakes so use with caution!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First some notation:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(x,n)$ be the probability that $x$ or more people share a birthday among $n$ people,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(y|n)$ be the probability that &lt;em&gt;exactly&lt;/em&gt; $y$ people share a birthday among $n$ people.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Notes:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Abuse of notation as $P(.)$ is being used in two different ways.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;By definition $y$ cannot take the value of 1 as it does not make any sense and $y$ = 0 can be interpreted to mean that no one shares a common birthday.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Then the required probability is given by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(x,n) = 1 - P(0|n) - P(2|n) - P(3|n) .... - P(x-1|n)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(y|n) = {n \choose y} (\frac{365}{365})^y \ \prod_{k=1}^{k=n-y}(1 -\frac{k}{365})$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the logic: You need the probability that exactly $y$ people share a birthday. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Step 1: You can pick $y$ people in ${n \choose y}$ ways.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Step 2: Since they share a birthday it can be any of the 365 days in a year. So, we basically have 365 choices which gives us $(\frac{365}{365})^y$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Step 3: The remaining $n-y$ people should not share a birthday with the first $y$ people or with each other. This reasoning gives us $\prod_{k=1}^{k=n-y}(1 -\frac{k}{365})$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can check that for $x$ = 2 the above collapses to the standard birthday paradox solution.&lt;/p&gt;&#xA;" LastActivityDate="2010-08-05T15:34:26.243" CommentCount="6" />
  <row Id="1312" PostTypeId="2" ParentId="1257" CreationDate="2010-08-05T16:04:34.100" Score="0" ViewCount="" Body="&lt;p&gt;It depends on exactly what you mean - your notation is a bit ambiguous. If I understand your notation correctly, you don't actually have 4 independent random variables - $X_3$ is a deterministic function of $Y_1$ and $Y_2$, and so it shouldn't occur explicitly in the likelihood. $Y_3$ is a function of the deterministic node $X_3$, and hence is a function of $Y_1$ and $Y_2$ when you drop out the deterministic node to form the likelihood, which is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$f(Y_1)f(Y_2)f(Y_3|Y_1,Y_2)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;provided that $Y_1Y_2&gt;0$ (for consistency with data on $X_3$) and is zero otherwise. Is that what you meant?&lt;/p&gt;&#xA;" OwnerUserId="643" LastActivityDate="2010-08-05T16:04:34.100" CommentCount="3" />
  <row Id="1313" PostTypeId="2" ParentId="1278" CreationDate="2010-08-05T16:54:35.073" Score="2" ViewCount="" Body="&lt;p&gt;If you cannot collect data on a different ward where you don't do the intervention, your conclusions will be weak, because you cannot rule out other causes that act simultaneously (change in weather, season, epidemic of something, etc, etc). However if you observe a large effect, your study would still contribute an interesting piece of evidence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The rest of your questions are a bit confused. If your outcome is binary: infection yes/no in a bunch of patients (probably adjusted for length of stay so it becomes a rate?), then you could not even do a t-test, so there is no point in discussing its appropriateness. But in the sense that it looks at differences it is similar to a t-test when you have continuous outcomes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a test loosely called &quot;ratio t-test&quot;, which is a t-test conducted on log-transformed data that concentrates on ratios instead of differences. So it is in some sense the counterpart of IRR, however I don't think you could actually perform it, because you don't have a continuous outcome variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So pick either the IRD or IRR. The difference is usually more important from a public health point of view, while ratios tend to be more impressive especially for rare events.&lt;/p&gt;&#xA;" OwnerUserId="279" LastActivityDate="2010-08-05T16:54:35.073" />
  <row Id="1314" PostTypeId="1" CreationDate="2010-08-05T18:58:03.303" Score="0" ViewCount="154" Body="&lt;p&gt;I completed a BA and MA in political science at a large state school five years ago. Since then, I've developed skills in R, SQL, Python, Perl, and Matlab, and continue to learn new things every day. However, I don't have a firm grasp of statistics beyond basic linear regression, logistic regression, and time series. For this reason, I am considering going back to school to work on a BS in statistics. I'd like to start doing some data consulting and free lance analytics work, but believe the education would be necessary. I believe that I could go ahead and begin with the freeland work while i'm in school.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would a BS in statistics be beneficial given my career goals?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I should mention that I have a somewhat serious felony and finding employment is near impossible. Therefore, I'm planning on just being self-employed the rest of my life.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help is appreciated.&lt;/p&gt;&#xA;" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-08-07T18:00:21.283" LastActivityDate="2010-08-07T18:00:21.283" CommunityOwnedDate="2010-08-05T19:30:48.810" ClosedDate="2010-08-05T23:35:57.870" Title="What are some possible career options?" Tags="&lt;r&gt;&lt;careers&gt;" CommentCount="5" />
  <row Id="1315" PostTypeId="1" CreationDate="2010-08-05T19:26:34.060" Score="6" ViewCount="539" Body="&lt;p&gt;i've sampled a real world process, network ping times. The &quot;round-trip-time&quot; is measured in milliseconds. Results are plotted in a histogram:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i36.tinypic.com/2qtfp6f.jpg&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ping times have a minimum value, but a long upper tail.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;i want to know what statistical distribution this is, and how to estimate its parameters. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even though the distribution is not a normal distribution, i can still show what i am trying to achieve.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The normal distribution uses the function:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://upload.wikimedia.org/math/9/e/1/9e1e4a3af93c9680ba75669a0b69fbf6.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;with the two parameters&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;μ (mean)&lt;/li&gt;&#xA;&lt;li&gt;σ&lt;sup&gt;2 &lt;/sup&gt; (variance)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h2&gt;Parameter estimation&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;The formulas for estimating the two parameters are:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://upload.wikimedia.org/math/3/2/4/324081bc08a66fbeba64f43a3dbd5e1a.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Applying these formulas against the data i have in Excel, i get:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;μ = 10.9558 (mean)&lt;/li&gt;&#xA;&lt;li&gt;σ&lt;sup&gt;2 &lt;/sup&gt; = 67.4578 (variance)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;With these parameters i can plot the &quot;&lt;em&gt;normal&lt;/em&gt;&quot; distribution over top my sampled data:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i34.tinypic.com/2mqu3wl.jpg&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Obviously it's not a normal distribution. A normal distribution has an infinite top and bottom tail, and is symmetrical. This distribution is not symmetrical.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;ul&gt;&#xA;  &lt;li&gt;What principles would i apply, what&#xA;  flowchart, would i apply to determine&#xA;  what kind of distribution this is?&lt;/li&gt;&#xA;  &lt;li&gt;Given that the distribution has no negative tail, and long positive tail: what distributions match that? &lt;/li&gt;&#xA;  &lt;li&gt;Is there a reference that matches distributions to the observations you're taking?&lt;/li&gt;&#xA;  &lt;/ul&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;And cutting to the chase, what is the formula for this distribution, and what are the formulas to estimate its parameters?&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;i want to get the distribution so i can get the &quot;average&quot; value, as well as the &quot;spread&quot;:&#xA;&lt;img src=&quot;http://i38.tinypic.com/2i27sp4.jpg&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;i am actually plotting the histrogram in software, and i want to overlay the theoretical distribution:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i38.tinypic.com/nqpnck.jpg&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note:&lt;/strong&gt; Cross-posted from &lt;a href=&quot;http://math.stackexchange.com/questions/1648/how-do-i-figure-out-what-kind-of-distribution-this-is&quot;&gt;math.stackexchange.com&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Tags&lt;/strong&gt;: sampling, statistics, parameter-estimation, normal-distribution&lt;/p&gt;&#xA;" OwnerUserId="775" LastEditorUserId="775" LastEditorDisplayName="" LastEditDate="2010-08-05T19:37:00.280" LastActivityDate="2010-08-06T16:20:50.017" Title="How do i figure out what kind of distribution this is?" Tags="&lt;distributions&gt;&lt;normality&gt;&lt;sample&gt;&lt;sample-size&gt;" AnswerCount="8" CommentCount="7" />
  <row Id="1316" PostTypeId="2" ParentId="1315" CreationDate="2010-08-05T19:33:56.103" Score="3" ViewCount="" Body="&lt;p&gt;Try the &lt;a href=&quot;http://en.wikipedia.org/wiki/Gamma_distribution&quot; rel=&quot;nofollow&quot;&gt;gamma&lt;/a&gt; distribution which is parametrized as $x \sim Gamma(k,\theta)$. If you see these &lt;a href=&quot;http://en.wikipedia.org/wiki/File%3aGamma_distribution_pdf.svg&quot; rel=&quot;nofollow&quot;&gt;pdf plots&lt;/a&gt; of the gamma from the wiki you will see that there are some plots that look similar to what you have.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update- Estimation Process&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The estimation via &lt;a href=&quot;http://en.wikipedia.org/wiki/Gamma_distribution#Maximum_likelihood_estimation&quot; rel=&quot;nofollow&quot;&gt;maximum likelihood&lt;/a&gt; is tricky but possible. I imagine you can start with the approximate solution given by the wiki for &lt;a href=&quot;http://upload.wikimedia.org/math/3/2/0/3203c979914ef4e2f354617a7723df81.png&quot; rel=&quot;nofollow&quot;&gt;$\hat{\theta}$&lt;/a&gt; and &lt;a href=&quot;http://upload.wikimedia.org/math/8/8/1/881fb8c7ad01bb1b1505dcacf092b4e4.png&quot; rel=&quot;nofollow&quot;&gt;$\hat{k}$&lt;/a&gt; and if the plots look ok and if needed you can estimate $\hat{k}$ more accurately using the details in the wiki. &lt;/p&gt;&#xA;" LastActivityDate="2010-08-05T19:51:03.543" CommentCount="5" />
  <row Id="1317" PostTypeId="2" ParentId="726" CreationDate="2010-08-05T19:41:53.000" Score="4" ViewCount="" Body="&lt;p&gt;9 out of ten dentists think the 10th dentist is an idiot. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;No idea who said it.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="776" LastActivityDate="2010-08-05T19:41:53.000" CommunityOwnedDate="2010-08-05T19:41:53.000" CommentCount="1" />
  <row Id="1318" PostTypeId="2" ParentId="1315" CreationDate="2010-08-05T19:51:02.640" Score="6" ViewCount="" Body="&lt;p&gt;There is no reason to expect that any real world data set will fit a known distributional form...especially from such a known messy data source.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you want to do with the answers will largely indicate an approach. For example, if you want to know when the ping times have changed significantly, then trending the empirical distribution may be a way to go. If you want to identify outliers, other techniques may be more appropriate.&lt;/p&gt;&#xA;" OwnerUserId="247" LastActivityDate="2010-08-05T19:51:02.640" CommentCount="2" />
  <row Id="1319" PostTypeId="2" ParentId="1315" CreationDate="2010-08-05T20:13:15.683" Score="7" ViewCount="" Body="&lt;p&gt;Let me ask a more basic question:  what do you want to &lt;em&gt;do&lt;/em&gt; with this distributional information?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reason I ask is because it may well make more sense to approximate the distribution with some sort of kernel density estimator, rather than insist that it fit into one of the (possibly shifted) exponential family distributions.  You can answer almost all of the same sorts of questions that a standard distribution will let you answer, and you don't have to worry (as much) about whether you've selected the correct model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But if there's a fixed minimum time, and you must have some sort of compactly parameterized distribution to go with it, then just eyeballing it I'd subtract off the minimum and fit a gamma, like others have suggested.  &lt;/p&gt;&#xA;" OwnerUserId="61" LastActivityDate="2010-08-05T20:13:15.683" />
  <row Id="1320" PostTypeId="2" ParentId="726" CreationDate="2010-08-05T20:38:41.187" Score="1" ViewCount="" Body="&lt;p&gt;No statistican, but useful for the profession:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The perfect is the enemy of the good &#xA;  - Voltaire&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="778" LastActivityDate="2010-08-05T20:38:41.187" CommunityOwnedDate="2010-08-05T20:38:41.187" />
  <row Id="1321" PostTypeId="1" AcceptedAnswerId="1322" CreationDate="2010-08-05T21:12:50.063" Score="4" ViewCount="174" Body="&lt;p&gt;What would be the best way to display changes in two scalar variables (x,y) over time (z), in one visualization?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One idea that I had was to plot x and y both on the vertical axis, with z as the horizontal. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: I'll be using R and likely ggplot2&lt;/p&gt;&#xA;" OwnerUserId="776" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-11-30T16:43:14.843" LastActivityDate="2010-11-30T16:43:14.843" Title="Visualizing two scalar variables over time" Tags="&lt;r&gt;&lt;time-series&gt;&lt;data-visualization&gt;&lt;ggplot2&gt;" AnswerCount="3" CommentCount="2" FavoriteCount="1" />
  <row Id="1322" PostTypeId="2" ParentId="1321" CreationDate="2010-08-05T22:01:03.533" Score="4" ViewCount="" Body="&lt;p&gt;The other idea is to plot one series as x and the second as y -- the time dependency will be hidden,  but this plots shows correlations pretty well. (Yet time can be shown to some extent by connecting points chronologically; if the series are quite short and continuous it should be readable.)&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-08-05T22:01:03.533" CommentCount="2" />
  <row Id="1324" PostTypeId="1" AcceptedAnswerId="1326" CreationDate="2010-08-05T22:16:20.007" Score="3" ViewCount="88" Body="&lt;p&gt;The title is quite self-explanatory - I'd like to know if there's any other parametric technique apart from repeated-measures ANOVA, that can be utilized in order to compare several (more than 2) repeated measures?&lt;/p&gt;&#xA;" OwnerUserId="1356" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-16T07:04:59.787" LastActivityDate="2010-09-16T07:04:59.787" Title="Parametric techniques for n-related samples" Tags="&lt;repeated-measures&gt;" AnswerCount="1" />
  <row Id="1325" PostTypeId="2" ParentId="1315" CreationDate="2010-08-05T22:17:29.820" Score="6" ViewCount="" Body="&lt;p&gt;Weibull is sometimes used for modelling ping time. try a weibull distribution. To fit one in R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x&amp;lt;-rweibull(n=1000,shape=2,scale=100)&#xA;#generate a weibull (this should be your data).&#xA;hist(x)&#xA;#this is an histogram of your data.&#xA;library(survival)&#xA;a1&amp;lt;-survreg(Surv(x,rep(1,1000))~1,dist='weibull')&#xA;exp(a1$coef) #this is the ML estimate of the scale parameter&#xA;1/a1$scale     #this is the ML estimate of the shape parameter&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you're wondering for the goofy names (i.e. $scale to get the inverse of the shape)&#xA;it's because &quot;survreg&quot; uses another parametrization (i.e. it is parametrized in terms of the &quot;inverse weibull&quot; which is more comon in actuarial sciences).&lt;/p&gt;&#xA;" OwnerUserId="603" LastEditorUserId="603" LastEditorDisplayName="" LastEditDate="2010-08-06T16:20:50.017" LastActivityDate="2010-08-06T16:20:50.017" />
  <row Id="1326" PostTypeId="2" ParentId="1324" CreationDate="2010-08-05T22:35:46.630" Score="6" ViewCount="" Body="&lt;p&gt;Multilevel/hierarchical linear models can be used for this.  Essentially, each repetition of the measure is clustered within the individual; individuals can then be clustered within other hierarchies.  For me, at least, it's more intuitive than repeated-measures ANOVA.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The canonical text is &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/076191904X&quot; rel=&quot;nofollow&quot;&gt;Raudenbush and Bryk&lt;/a&gt;; I'm also really fond of &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/052168689X&quot; rel=&quot;nofollow&quot;&gt;Gelman and Hill&lt;/a&gt;.  &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.110.5130&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;Here's a tutorial I read some time ago&lt;/a&gt; - you may or may not find the tutorial itself useful (that's so often a matter of personal taste, training and experience), but the bibliography at the end is good.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I should note that Gelman and Hill doesn't have a ton on multilevel models specifically for repeated measures; I can't remember if that's the case or not for Raudenbush and Bryk.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: Found a book I was looking for - &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0195152964&quot; rel=&quot;nofollow&quot;&gt;Applied Longitudinal Data Analysis by Singer and Willett&lt;/a&gt; has (I believe) an explicit focus on multilevel models for repeated measures.  I haven't had a chance to read very far into it, but it might be worth looking into.&lt;/p&gt;&#xA;" OwnerUserId="71" LastEditorUserId="71" LastEditorDisplayName="" LastEditDate="2010-08-05T23:06:58.227" LastActivityDate="2010-08-05T23:06:58.227" CommentCount="2" />
  <row Id="1327" PostTypeId="2" ParentId="1315" CreationDate="2010-08-05T23:22:01.933" Score="2" ViewCount="" Body="&lt;p&gt;Looking at it I would say a skew-normal or possibly a binormal distribution may fit it well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In R you could use the &lt;code&gt;sn&lt;/code&gt; library to deal with skew-normal distribution and use &lt;code&gt;nls&lt;/code&gt; or &lt;code&gt;mle&lt;/code&gt; to do a non-linear least square or a maximum likelihood extimation fit of your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;===&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: rereading your question/comments I would add something more&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If what you're interested into is just drawing a pretty graph over the bars forget about distributions, who cares in the end if you're not doing anything with it. Just draw a B-spline over your data point and you're good.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, with this approach you avoid having to implement a MLE fit algorithm (or similar), and you're covered in the case of a distribution that is not skew-normal (or whatever you choose to draw) &lt;/p&gt;&#xA;" OwnerUserId="582" LastEditorUserId="582" LastEditorDisplayName="" LastEditDate="2010-08-06T06:19:32.700" LastActivityDate="2010-08-06T06:19:32.700" CommentCount="1" />
  <row Id="1328" PostTypeId="2" ParentId="726" CreationDate="2010-08-06T00:29:25.567" Score="3" ViewCount="" Body="&lt;p&gt;&quot;If you think that statistics has nothing to say about what you do or how you could do it better, then you are either wrong or in need of a more interesting job.&quot; - Stephen Senn (Dicing with Death: Chance, Risk and Health, Cambridge University Press, 2003)&lt;/p&gt;&#xA;" OwnerUserId="781" LastActivityDate="2010-08-06T00:29:25.567" CommunityOwnedDate="2010-08-06T00:29:25.567" />
  <row Id="1330" PostTypeId="2" ParentId="726" CreationDate="2010-08-06T01:08:05.617" Score="15" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;The best thing about being a statistician is that you get to play in everyone's backyard.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;-- John Tukey &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(This is MY favourite Tukey quote)&lt;/p&gt;&#xA;" OwnerUserId="521" LastEditorUserId="795" LastEditorDisplayName="" LastEditDate="2010-12-03T04:02:12.130" LastActivityDate="2010-12-03T04:02:12.130" CommunityOwnedDate="2010-08-06T01:08:05.617" CommentCount="1" />
  <row Id="1331" PostTypeId="2" ParentId="485" CreationDate="2010-08-06T01:15:52.747" Score="3" ViewCount="" Body="&lt;p&gt;There is a new resources forming these days for talks about R:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.r-bloggers.com/RUG/&quot; rel=&quot;nofollow&quot;&gt;http://www.r-bloggers.com/RUG/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Compiled by the organizers of &quot;R Users Groups&quot; around the world (right now, mainly around the States).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is a new project (just a few weeks old), but already got good content on it, and good people wanting to take part in it.&#xA;&lt;img src=&quot;http://www.r-bloggers.com/RUG/wp-content/uploads/2010/07/banner.jpg&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="253" LastActivityDate="2010-08-06T01:15:52.747" CommunityOwnedDate="2010-08-06T01:15:52.747" />
  <row Id="1332" PostTypeId="2" ParentId="726" CreationDate="2010-08-06T01:20:36.700" Score="4" ViewCount="" Body="&lt;p&gt;&quot;It is easy to lie with statistics. It is hard to tell the truth without statistics.&quot; - Andrejs Dunkels&lt;/p&gt;&#xA;" OwnerUserId="521" LastActivityDate="2010-08-06T01:20:36.700" CommunityOwnedDate="2010-08-06T01:20:36.700" CommentCount="3" />
  <row Id="1333" PostTypeId="2" ParentId="726" CreationDate="2010-08-06T01:26:54.020" Score="21" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Statisticians, like artists, have the bad habit of falling in love with their models.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;-- George Box&lt;/p&gt;&#xA;" OwnerUserId="521" LastEditorUserId="795" LastEditorDisplayName="" LastEditDate="2010-10-02T17:10:08.093" LastActivityDate="2010-10-02T17:10:08.093" CommunityOwnedDate="2010-08-06T01:26:54.020" />
  <row Id="1334" PostTypeId="2" ParentId="726" CreationDate="2010-08-06T01:29:56.867" Score="2" ViewCount="" Body="&lt;p&gt;&quot;New methods always look better than old ones. Neural nets are better than logistic regression, support vector machines are better than neural nets, etc.&quot; - Brad Efron&lt;/p&gt;&#xA;" OwnerUserId="521" LastActivityDate="2010-08-06T01:29:56.867" CommunityOwnedDate="2010-08-06T01:29:56.867" CommentCount="1" />
  <row Id="1335" PostTypeId="2" ParentId="1286" CreationDate="2010-08-06T01:40:54.227" Score="2" ViewCount="" Body="&lt;p&gt;If you wish to trade processing speed for memory (which I think you do), I would suggest the following algorithm:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Set up a loop from 1 to N Choose K, indexed by i&lt;/li&gt;&#xA;&lt;li&gt;Each i can be considered an index to a &lt;a href=&quot;http://msdn.microsoft.com/en-us/library/aa289166%28VS.71%29.aspx&quot; rel=&quot;nofollow&quot;&gt;combinadic&lt;/a&gt;, decode as such&lt;/li&gt;&#xA;&lt;li&gt;Use the combination to perform your test statistic, store the result, discard the combination&lt;/li&gt;&#xA;&lt;li&gt;Repeat&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;This will give you all N Choose K possible combinations without having to create them explicitly. I have code to do this in R if you'd like it (you can email me at mark dot m period fredrickson at-symbol gmail dot com).&lt;/p&gt;&#xA;" OwnerUserId="729" LastActivityDate="2010-08-06T01:40:54.227" CommentCount="2" />
  <row Id="1336" PostTypeId="2" ParentId="726" CreationDate="2010-08-06T01:43:46.623" Score="1" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;In the long run, we're all dead. &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;-- John Maynard Keynes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A reference to survival analysis?!&lt;/p&gt;&#xA;" OwnerUserId="521" LastEditorUserId="795" LastEditorDisplayName="" LastEditDate="2010-12-03T04:05:41.940" LastActivityDate="2010-12-03T04:05:41.940" CommunityOwnedDate="2010-08-06T01:43:46.623" CommentCount="1" />
  <row Id="1337" PostTypeId="1" CreationDate="2010-08-06T01:53:47.023" Score="5" ViewCount="1533" Body="&lt;p&gt;Well we've got favourite statistics quotes. What about statistics jokes?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, what's your favourite statistics joke?&lt;/p&gt;&#xA;" OwnerUserId="521" LastActivityDate="2010-10-23T01:35:01.097" CommunityOwnedDate="2010-08-06T02:30:56.857" Title="Statistics Jokes" Tags="&lt;humor&gt;" AnswerCount="16" CommentCount="2" FavoriteCount="6" />
  <row Id="1338" PostTypeId="2" ParentId="1337" CreationDate="2010-08-06T01:54:55.680" Score="6" ViewCount="" Body="&lt;p&gt;I thought I'd start the ball rolling with my favourite.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Being a statistician means never having to say you are certain.&quot;&lt;/p&gt;&#xA;" OwnerUserId="521" LastActivityDate="2010-08-06T01:54:55.680" CommunityOwnedDate="2010-08-06T02:30:56.857" CommentCount="3" />
  <row Id="1339" PostTypeId="2" ParentId="1207" CreationDate="2010-08-06T02:48:09.630" Score="2" ViewCount="" Body="&lt;p&gt;You may want to define what you want more clearly (to yourself, if not here). If what you're looking for is the most statistically significant stationary period contained in your noisy data, there's essentially two routes to take:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) compute a robust autocorrelation estimate, and take the maximum coefficient&lt;br&gt;&#xA;2) compute a robust power spectral density estimate, and take the maximum of the spectrum&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem with #2 is that for any noisy time series, you will get a large amount of power in low frequencies, making it difficult to distinguish. There are some techniques for resolving this problem (i.e. pre-whiten, then estimate the PSD), but if the true period from your data is long enough, automatic detection will be iffy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your best bet is probably to implement a robust autocorrelation routine such as can be found in chapter 8.6, 8.7 in &lt;em&gt;Robust Statistics - Theory and Methods&lt;/em&gt; by Maronna, Martin and Yohai. Searching Google for &quot;robust durbin-levinson&quot; will also yield some results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're just looking for a simple answer, I'm not sure that one exists. Period detection in time series can be complicated, and asking for an automated routine that can perform magic may be too much. &lt;/p&gt;&#xA;" OwnerUserId="781" LastActivityDate="2010-08-06T02:48:09.630" CommentCount="1" />
  <row Id="1340" PostTypeId="2" ParentId="1164" CreationDate="2010-08-06T03:06:41.747" Score="7" ViewCount="" Body="&lt;p&gt;Anyone trained in statistical data analysis at a reasonable level uses the &lt;em&gt;concepts&lt;/em&gt; of robust statistics on a regular basis. Most researchers know enough to look for serious outliers and data recording errors; the policy of removing suspect data points goes back well into the 19th century with Lord Rayleigh, G.G. Stokes, and others of their age. If the question is: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Why don't researchers use the more modern methods for computing location, scale, regression, etc. estimates?&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;then the answer is given above -- the methods have largely been developed in the last 25 years, say 1985 - 2010. The lag for learning new methods factors in, as well as inertia compounded by the 'myth' that there is nothing wrong with blindly using classical methods. John Tukey comments that &lt;em&gt;just which&#xA;robust/resistant methods you use is not important—what is important is that you&#xA;use some. It is perfectly proper to use both classical and robust/resistant methods&#xA;routinely, and only worry when they differ enough to matter. But when they &lt;strong&gt;differ&lt;/strong&gt;,&#xA;you should think &lt;strong&gt;hard&lt;/strong&gt;.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If instead, the question is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Why don't researchers stop and ask questions about their data, instead of blindly applying highly unstable estimates?&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;then the answer really comes down to training. There are far too many researchers who were never trained in statistics properly, summed up by the general reliance on p-values as the be-all and end-all of 'statistical significance'.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;@Kwak: Huber's estimates from the 1970s &lt;em&gt;are&lt;/em&gt; robust, in the classical sense of the word: they resist outliers. And redescending estimators actually date well before the 1980s: the Princeton robustness study (of 1971) included the bisquare estimate of location, a redescending estimate.&lt;/p&gt;&#xA;" OwnerUserId="781" LastEditorUserId="781" LastEditorDisplayName="" LastEditDate="2010-08-06T15:49:38.253" LastActivityDate="2010-08-06T15:49:38.253" CommentCount="4" />
  <row Id="1341" PostTypeId="2" ParentId="652" CreationDate="2010-08-06T03:14:53.140" Score="2" ViewCount="" Body="&lt;p&gt;The classic &quot;orange horror&quot; remains an excellent introduction: &lt;em&gt;Exploratory Data Analysis&lt;/em&gt; by John Tukey. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0201076160&quot; rel=&quot;nofollow&quot;&gt;http://www.amazon.com/Exploratory-Data-Analysis-John-Tukey/dp/0201076160&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="781" LastActivityDate="2010-08-06T03:14:53.140" CommunityOwnedDate="2010-08-06T03:14:53.140" />
  <row Id="1342" PostTypeId="2" ParentId="726" CreationDate="2010-08-06T03:17:17.273" Score="1" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Do not make things easy for yourself&#xA;  by speaking or thinking of data as if&#xA;  they were different from what they&#xA;  are; and do not go off from facing&#xA;  data as they are, to amuse your&#xA;  imagination by wishing they were&#xA;  different from what they are. Such&#xA;  wishing is pure waste of nerve force,&#xA;  weakens your intellectual power, and&#xA;  gets you into habits of mental&#xA;  confusion.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;--Mary Everest Boole&lt;/p&gt;&#xA;" LastActivityDate="2010-08-06T03:17:17.273" CommunityOwnedDate="2010-08-06T03:17:17.273" />
  <row Id="1343" PostTypeId="2" ParentId="1321" CreationDate="2010-08-06T03:22:13.263" Score="5" ViewCount="" Body="&lt;p&gt;I sometimes make the x-axis time and plot both scalar variables on the y-axis.&#xA;When the two scalar variables are on a different metric, I rescale one or both of the scalar variables so they can be displayed on the same plot.&#xA;I use things like colour and shape to discriminate the two scalar variables.&#xA;I've often used &lt;code&gt;xyplot&lt;/code&gt; from &lt;code&gt;lattice&lt;/code&gt; for this purpose. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's an example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(lattice)&#xA;xyplot(dv1 + dv2  ~ iv, data = x, col = c(&quot;black&quot;, &quot;red&quot;))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-08-06T03:22:13.263" />
  <row Id="1344" PostTypeId="2" ParentId="1315" CreationDate="2010-08-06T03:47:26.320" Score="3" ViewCount="" Body="&lt;p&gt;A simpler approach might be to transform the data. After transforming, it might be close to Gaussian. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One common way to do so is by taking the logarithm of all values. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My guess is that in this case the distribution of the reciprocal of the round-trip times will be more symmetrical and perhaps close to Gaussian. By taking the reciprocal, you are essentially tabulating velocities instead of times, so it still is easy to interpret the results (unlike logarithms or many transforms).&lt;/p&gt;&#xA;" OwnerUserId="25" LastActivityDate="2010-08-06T03:47:26.320" />
  <row Id="1345" PostTypeId="2" ParentId="1315" CreationDate="2010-08-06T04:11:29.997" Score="1" ViewCount="" Body="&lt;p&gt;Based on your comment &quot;Really i want to draw the mathematical curve that follows the distribution. Granted it might not be a known distribution; but i can't imagine that this hasn't been investigated before.&quot; I am providing a function that sort of fits.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Take a look at &lt;a href=&quot;http://reference.wolfram.com/mathematica/ref/ExtremeValueDistribution.html?q=ExtremeValueDistribution&amp;amp;lang=en&quot; rel=&quot;nofollow&quot;&gt;ExtremeValueDistribution&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I added an amplitude and made the two betas different. I figure your function's center is closer to 9.5 then 10.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;New function:&#xA;a E^(-E^(((-x + alpha)/b1)) + (-x + alpha)/b2)/((b1 + b2)/2)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;{alpha-&gt;9.5, b2 -&gt; 0.899093, a -&gt; 5822.2, b1 -&gt; 0.381825}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.wolframalpha.com/input/?i=plot+11193.8+E%5E%28-E%5E%281.66667+%2810+-+x%29%29+%2B+1.66667+%2810+-+x%29%29+%2Cx+0..16%2C+y+from+0+to+4500&quot; rel=&quot;nofollow&quot;&gt;Wolfram alpha&lt;/a&gt;:&#xA;plot 11193.8 E^(-E^(1.66667 (10 - x)) + 1.66667 (10 - x)) ,x 0..16, y from 0 to 4500&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some points around 10ms:&lt;br&gt;&#xA;{{9, 390.254}, {10, 3979.59}, {11, 1680.73}, {12, 562.838}}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tail does not fit perfectly though. The tail can be fit better if b2 is lower and the peak is chosen to be closer to 9.&lt;/p&gt;&#xA;" OwnerUserId="782" LastActivityDate="2010-08-06T04:11:29.997" />
  <row Id="1346" PostTypeId="2" ParentId="1337" CreationDate="2010-08-06T04:50:59.280" Score="12" ViewCount="" Body="&lt;p&gt;I saw this posted as a comment on here somewhere:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://xkcd.com/552/&quot; rel=&quot;nofollow&quot;&gt;http://xkcd.com/552/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://imgs.xkcd.com/comics/correlation.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A: I used to think correlation implied causation. Then I took a statistics class. Now I don't.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;B: Sounds like the class helped.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A: Well, maybe.&lt;/p&gt;&#xA;" OwnerUserId="287" LastEditorUserId="253" LastEditorDisplayName="" LastEditDate="2010-08-06T12:47:14.807" LastActivityDate="2010-08-06T12:47:14.807" CommunityOwnedDate="2010-08-06T04:50:59.280" CommentCount="1" />
  <row Id="1347" PostTypeId="2" ParentId="726" CreationDate="2010-08-06T05:15:22.583" Score="4" ViewCount="" Body="&lt;p&gt;&quot;Extraordinary claims demand extraordinary evidence.&quot; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Often attributed to Carl Sagan, but he was paraphrasing sceptic Marcello Truzzi. Doubtless the concept is even more ancient. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;David Hume said, &quot;A wise man, therefore, proportions his belief to the evidence&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One could argue this is not a quote about statistics. However, applied statistics is ultimately in the business of evaluating the quality of evidence for or against some proposition.&lt;/p&gt;&#xA;" OwnerUserId="521" LastActivityDate="2010-08-06T05:15:22.583" CommunityOwnedDate="2010-08-06T05:15:22.583" />
  <row Id="1348" PostTypeId="2" ParentId="652" CreationDate="2010-08-06T05:27:57.743" Score="1" ViewCount="" Body="&lt;p&gt;An old favourite of mine as an introduction to biostatistics is Armitage &amp;amp; Berry's (&amp;amp; now Matthew's):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Statistical Methods in Medical Research&lt;/em&gt; &lt;/p&gt;&#xA;" OwnerUserId="521" LastActivityDate="2010-08-06T05:27:57.743" CommunityOwnedDate="2010-08-06T05:27:57.743" />
  <row Id="1349" PostTypeId="2" ParentId="1252" CreationDate="2010-08-06T05:33:06.860" Score="6" ViewCount="" Body="&lt;p&gt;I have previously found UCLA's &quot;What statistical analysis should I use?&quot; to be helpful:&#xA;&lt;a href=&quot;http://www.ats.ucla.edu/stat/mult_pkg/whatstat/default.htm&quot; rel=&quot;nofollow&quot;&gt;http://www.ats.ucla.edu/stat/mult_pkg/whatstat/default.htm&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It also gives examples of how to do the analysis in SAS, Stata and SPSS - although not R!&lt;/p&gt;&#xA;" OwnerUserId="521" LastActivityDate="2010-08-06T05:33:06.860" CommentCount="1" />
  <row Id="1350" PostTypeId="1" AcceptedAnswerId="1407" CreationDate="2010-08-06T06:32:06.513" Score="1" ViewCount="208" Body="&lt;p&gt;I am working with a large data set (approximately 50K observations) and trying to running a Maximum likelihood estimation on 5 unknowns in Stata. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I encountered an error message of &quot;Numerical Overflow&quot;. How can I overcome this? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am trying to run a Stochastic Frontier analysis using the built in Stata command &quot;frontier&quot;. The dependent variable is log of output and the independent variable is log of intermediate inputs, capital, labour, and utlities. &lt;/p&gt;&#xA;" OwnerUserId="189" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-10-08T16:07:49.240" LastActivityDate="2010-10-08T16:07:49.240" Title="How to get around Numerical Overflow in Stata?" Tags="&lt;large-data&gt;&lt;stata&gt;&lt;computing&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="1351" PostTypeId="2" ParentId="1296" CreationDate="2010-08-06T07:44:34.903" Score="4" ViewCount="" Body="&lt;p&gt;Assuming you want to pick a distribution for n, p(n) you can apply Bayes law.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You know that the probability of k events occuring given that n have actually occured is governed by a binomial distribtion &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$p(k|n) = {n \choose k} p^k (1-p)^{(n-k)}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The thing you really want to know is the probability of n events having actually occured, given that you observed k. By Bayes lay:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$p(n|k) = \frac{p(k|n)p(n)}{p(k)}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By applying the theorem of total probability, we can write:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$p(n|k) = \frac{p(k|n)p(n)}{\sum_{n'} p(k|n')p(n')}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So without further information, about the distribution of $p(n)$ you can't really go any further. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, if you want to pick a distribution for $p(n)$ for which there is a value $n$ greater than which $p(n) = 0$, or sufficiently close to zero, then you can do a bit better. For example, assume that the distribution of $n$ is uniform in the range $[0,n_{max}]$. this case:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$p(n) = \frac{1}{n_{max}}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Bayesian formulation simplifies to:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$p(n|k) = \frac{p(k|n)}{\sum_{n'} p(k|n')}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for the final part of the problem, I agree that the best approach is to perform a cumulative summation over $p(n|k)$, to generate the cummulative probability distribution function, and iterate until the 0.95 limit is reached.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given that this question migrated from SO, toy sample code in python is attached below&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy.random&#xA;&#xA;p = 0.8&#xA;nmax = 200&#xA;&#xA;def factorial(n):&#xA;    if n == 0:&#xA;        return 1&#xA;    return reduce( lambda a,b : a*b, xrange(1,n+1), 1 )&#xA;&#xA;def ncr(n,r):&#xA;    return factorial(n) / (factorial(r) * factorial(n-r))&#xA;&#xA;def binomProbability(n, k, p):&#xA;    p1 = ncr(n,k)&#xA;    p2 = p**k&#xA;    p3 = (1-p)**(n-k)&#xA;    return p1*p2*p3&#xA;&#xA;def posterior( n, k, p ):&#xA;    def p_k_given_n( n, k ):&#xA;        return binomProbability(n, k, p)&#xA;    def p_n( n ):&#xA;        return 1./nmax&#xA;    def p_k( k ):&#xA;        return sum( [ p_n(nd)*p_k_given_n(nd,k) for nd in range(k,nmax) ] )&#xA;    return (p_k_given_n(n,k) * p_n(n)) / p_k(k)&#xA;&#xA;&#xA;observed_k   = 80&#xA;p_n_given_k  = [ posterior( n, observed_k, p ) for n in range(0,nmax) ]&#xA;cp_n_given_k = numpy.cumsum(p_n_given_k)&#xA;for n in xrange(0,nmax):&#xA;    print n, p_n_given_k[n], cp_n_given_k[n]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="789" LastActivityDate="2010-08-06T07:44:34.903" />
  <row Id="1352" PostTypeId="1" AcceptedAnswerId="1384" CreationDate="2010-08-06T08:06:13.193" Score="7" ViewCount="214" Body="&lt;p&gt;In an average (median?) conversation about statistics you will often find yourself discussing this or that method of analyzing this or that type of data. In my experience, careful study design with special thought with regards to the statistical analysis is often neglected (working in biology/ecology, this seems to be a prevailing occurrence). Statisticians often find themselves in a gridlock with insufficient (or outright wrong) collected data. To paraphrase Ronald Fisher, they are forced to do a post-mortem on the data, which often leads to weaker conclusions, if at all.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to know which references you use to construct a successful study design, preferably for a wide range of methods (e.g. &lt;em&gt;t&lt;/em&gt;-test, GLM, GAM, ordination techniques...) that helps you avoid pitfalls mentioned above.&lt;/p&gt;&#xA;" OwnerUserId="144" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-16T06:58:05.970" LastActivityDate="2011-03-13T12:26:51.003" CommunityOwnedDate="2010-08-06T08:06:13.193" Title="References for how to plan a study" Tags="&lt;experiment-design&gt;" AnswerCount="4" FavoriteCount="2" />
  <row Id="1353" PostTypeId="2" ParentId="1296" CreationDate="2010-08-06T09:01:29.013" Score="9" ViewCount="" Body="&lt;p&gt;I would choose to use the &lt;a href=&quot;http://www.math.ntu.edu.tw/~hchen/teaching/StatInference/notes/lecture16.pdf&quot; rel=&quot;nofollow&quot;&gt;negative binomial distribution&lt;/a&gt;, which returns the probability that there will be X failures before the k_th success, when the constant probability of a success is p.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using an example&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;k=17 # number of successes&#xA;p=.6 # constant probability of success&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;the mean and sd for the failures are given by&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mean.X &amp;lt;- k*(1-p)/p&#xA;sd.X &amp;lt;- sqrt(k*(1-p)/p^2) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The distribution of the failures X, will have approximately that shape&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plot(dnbinom(0:(mean.X + 3 * sd.X),k,p),type='l')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So, the number of failures will be (with 95% confidence) approximately between&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;qnbinom(.025,k,p)&#xA;[1] 4&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;qnbinom(.975,k,p)&#xA;[1] 21&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So you inerval would be [k+qnbinom(.025,k,p),k+qnbinom(.975,k,p)] (using the example's numbers [21,38] )&lt;/p&gt;&#xA;" OwnerUserId="339" LastEditorUserId="339" LastEditorDisplayName="" LastEditDate="2010-08-06T10:55:02.630" LastActivityDate="2010-08-06T10:55:02.630" />
  <row Id="1354" PostTypeId="2" ParentId="1350" CreationDate="2010-08-06T09:34:37.720" Score="1" ViewCount="" Body="&lt;p&gt;How many variables do you have? Are you using a built in command?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Five years ago I used mlogit (multinomial logit) in Stata 8  to model 60,000 observations and about 40 variables with no difficulty.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd recommend searching the Stata websites (http://www.stata.com/) and even asking technical support.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, here are two answers:&#xA;&lt;a href=&quot;http://www.stata.com/statalist/archive/2007-04/msg00390.html&quot; rel=&quot;nofollow&quot;&gt;http://www.stata.com/statalist/archive/2007-04/msg00390.html&lt;/a&gt;&#xA;http://statalist.1588530.n2.nabble.com/st-Reproducing-xtlogit-with-xtmelogit-td3418488.html&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Stata provide the best technical support of any stats software I know.&lt;/p&gt;&#xA;" OwnerUserId="521" LastActivityDate="2010-08-06T09:34:37.720" />
  <row Id="1355" PostTypeId="1" AcceptedAnswerId="1364" CreationDate="2010-08-06T09:48:14.730" Score="5" ViewCount="127" Body="&lt;p&gt;Hello,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to predict the results of a simple card game, to judge on average, how long a game will last.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My 'simple' game is;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Cards are dealt from a randomised&#xA;deck to n players (typically 2-4)&lt;/li&gt;&#xA;&lt;li&gt;Each player gets five cards&lt;/li&gt;&#xA;&lt;li&gt;The top&#xA;card from the deck is turned over&lt;/li&gt;&#xA;&lt;li&gt;Each player takes it in turns to&#xA;either place a card of the same face&#xA;value (i.e 1-10, J, Q, K, A), the&#xA;same suit (i.e Hearts, Diamonds,&#xA;Spades, Clubs) or any suit of magic&#xA;card (a jack) &lt;/li&gt;&#xA;&lt;li&gt;If the player can place&#xA;a card they do, otherwise they must&#xA;take a card from the deck &lt;/li&gt;&#xA;&lt;li&gt;Play&#xA;continues in turn until all but one&#xA;player has no cards left&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I'm guessing that I could write code to play a mythical game and report the result, then run that code thousands of times.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Has anyone done this ?  Can they suggest code that does a similar job (my favoured language is R, but anything would do) ?  Is there a better way ?&lt;/p&gt;&#xA;" OwnerUserId="114" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-08-06T10:01:52.910" LastActivityDate="2010-08-09T16:31:07.027" Title="How could I predict the results of a simple card game ?" Tags="&lt;r&gt;&lt;probability&gt;&lt;games&gt;" AnswerCount="1" CommentCount="7" FavoriteCount="3" />
  <row Id="1356" PostTypeId="2" ParentId="1337" CreationDate="2010-08-06T09:52:51.527" Score="8" ViewCount="" Body="&lt;p&gt;Two statisticians were traveling in an airplane from LA to New York.&#xA;About an hour into the flight, the pilot announced that they had lost&#xA;an engine, but don’t worry, there are three left. However, instead of 5&#xA;hours it would take 7 hours to get to New York. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A little later, he&#xA;announced that a second engine failed, and they still had two left,&#xA;but it would take 10 hours to get to New York. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Somewhat later, the&#xA;pilot again came on the intercom and announced that a third engine&#xA;had died. Never fear, he announced, because the plane could fly on a&#xA;single engine. However, it would now take 18 hours to get to New&#xA;York.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At this point, one statistician turned to the other and said, “Gee, I&#xA;hope we don’t lose that last engine, or we’ll be up here forever!”&lt;/p&gt;&#xA;" OwnerUserId="114" LastActivityDate="2010-08-06T09:52:51.527" CommunityOwnedDate="2010-08-06T09:52:51.527" CommentCount="1" />
  <row Id="1357" PostTypeId="1" CreationDate="2010-08-06T10:48:52.543" Score="1" ViewCount="132" Body="&lt;p&gt;I am trying to compare it to Euclidean distance and Pearson correlation&lt;/p&gt;&#xA;" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-29T00:26:49.170" LastActivityDate="2011-04-29T00:26:49.170" Title="Is mutual information invariant to scaling, i.e. multiplying all elements by a nonzero constant?" Tags="&lt;correlation&gt;&lt;mutual-information&gt;" AnswerCount="3" FavoriteCount="2" />
  <row Id="1358" PostTypeId="1" AcceptedAnswerId="1365" CreationDate="2010-08-06T10:57:18.820" Score="10" ViewCount="171" Body="&lt;p&gt;In circular statistics, the expectation value of a random variable $Z$ with values on the circle $S$ is defined as&#xA;$$&#xA;m_1(Z)=\int_S z P^Z(\theta)\textrm{d}\theta&#xA;$$&#xA;(see &lt;a href=&quot;http://en.wikipedia.org/wiki/Circular_statistics#Moments&quot; rel=&quot;nofollow&quot;&gt;wikipedia&lt;/a&gt;).&#xA;This is a very natural definition, as is the definition of the variance&#xA;$$&#xA;\mathrm{Var}(Z)=1-|m_1(Z)|.&#xA;$$&#xA;So we didn't need a second moment in order to define the variance!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Nonetheless, we define the higher moments&#xA;$$&#xA;m_n(Z)=\int_S z^n P^Z(\theta)\textrm{d}\theta.&#xA;$$&#xA;I admit that this looks rather natural as well at first sight, and very similar to the definition in linear statistics. But still I feel a little bit uncomfortable, and have the following&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Questions:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1.&#xA;&lt;strong&gt;What is measured by the higher moments&lt;/strong&gt; defined above (intuitively)? Which properties of the distribution can be characterized by their moments?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2.&#xA;In the computation of the higher moments we use multiplication of complex numbers, although we think of the values of our random variables merely as vectors in the  plane or as angles. I know that complex multiplication is essentially addition of angles in this case, but still:&#xA;&lt;strong&gt;Why is complex multiplication a meaningful operation for circular data?&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="650" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-29T00:27:53.567" LastActivityDate="2011-04-29T00:27:53.567" Title="Intuition for higher moments in circular statistics" Tags="&lt;mathematical-statistics&gt;&lt;moments&gt;&lt;intuition&gt;&lt;directional-statistics&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="1359" PostTypeId="2" ParentId="1352" CreationDate="2010-08-06T11:09:39.877" Score="2" ViewCount="" Body="&lt;p&gt;In general, I would say any book that has DOE (design of experiments) in the title would fit the bill (and there are MANY).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My rule of thumb for such resource would be to start with the &lt;a href=&quot;http://en.wikipedia.org/wiki/Design_of_experiments&quot; rel=&quot;nofollow&quot;&gt;wiki page&lt;/a&gt;, in particular to your question, notice the &lt;a href=&quot;http://en.wikipedia.org/wiki/Design_of_experiments#Principles_of_experimental_design.2C_following_Ronald_A._Fisher&quot; rel=&quot;nofollow&quot;&gt;Principles of experimental design, following Ronald A. Fisher&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But a more serious answer would be domain specific (clinical trial has a huge manual, but for a study on mice, you'd probably go with some other field-related book)&lt;/p&gt;&#xA;" OwnerUserId="253" LastActivityDate="2010-08-06T11:09:39.877" CommunityOwnedDate="2010-08-06T11:09:39.877" CommentCount="3" />
  <row Id="1360" PostTypeId="2" ParentId="726" CreationDate="2010-08-06T11:11:00.890" Score="4" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Those who ignore Statistics are condemned to reinvent it.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;-- Brad Efron&lt;/p&gt;&#xA;" OwnerUserId="778" LastEditorUserId="795" LastEditorDisplayName="" LastEditDate="2010-12-03T04:03:04.030" LastActivityDate="2010-12-03T04:03:04.030" CommunityOwnedDate="2010-08-06T11:11:00.890" CommentCount="2" />
  <row Id="1361" PostTypeId="2" ParentId="726" CreationDate="2010-08-06T11:15:17.060" Score="4" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;My thesis is simply this: probability does not exist.&#xA;  - Bruno de Finetti&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="778" LastActivityDate="2010-08-06T11:15:17.060" CommunityOwnedDate="2010-08-06T11:15:17.060" />
  <row Id="1362" PostTypeId="2" ParentId="1352" CreationDate="2010-08-06T11:17:09.273" Score="1" ViewCount="" Body="&lt;p&gt;My rule of thumb is &quot;repeat more than you think it's sufficient&quot;. &lt;/p&gt;&#xA;" OwnerUserId="88" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-08-06T15:23:24.753" LastActivityDate="2010-08-06T15:23:24.753" CommunityOwnedDate="2010-08-06T11:17:09.273" />
  <row Id="1363" PostTypeId="2" ParentId="726" CreationDate="2010-08-06T11:23:15.363" Score="1" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;...Statistics used as a catalyst to engineering creation will, I believe, always result in the fastest and most economical progress.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;--George Box 1992&lt;/p&gt;&#xA;" OwnerUserId="114" LastEditorUserId="795" LastEditorDisplayName="" LastEditDate="2010-12-03T04:03:46.730" LastActivityDate="2010-12-03T04:03:46.730" CommunityOwnedDate="2010-08-06T11:23:15.363" />
  <row Id="1364" PostTypeId="2" ParentId="1355" CreationDate="2010-08-06T12:13:05.263" Score="6" ViewCount="" Body="&lt;p&gt;The easiest way is just to simulate the game lots of times. The R code below simulates a single game.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;nplayers = 4&#xA;#Create an empty data frame to keep track&#xA;#of card number, suit and if it's magic&#xA;empty.hand = data.frame(number = numeric(52),&#xA;  suit = numeric(52),&#xA;  magic  = numeric(52))&#xA;&#xA;#A list of players who are in the game&#xA;players =list()&#xA;for(i in 1:nplayers)&#xA;  players[[i]] = empty.hand&#xA;&#xA;#Simulate shuffling the deck&#xA;deck = empty.hand&#xA;deck$number = rep(1:13, 4)&#xA;deck$suit = as.character(rep(c(&quot;H&quot;, &quot;C&quot;, &quot;S&quot;, &quot;D&quot;), each=13))&#xA;deck$magic = rep(c(0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0), each=4)&#xA;deck = deck[sample(1:52, 52),]&#xA;&#xA;#Deal out five cards per person&#xA;for(i in 1:length(players)){&#xA;  r = (5*i-4):(5*i)&#xA;  players[[i]][r,] = deck[r,]&#xA;}&#xA;&#xA;#Play the game&#xA;i = 5*length(players)+1&#xA;current = deck[i,]&#xA;while(i &amp;lt; 53){&#xA;  for(j in 1:length(players)){&#xA;    playersdeck = players[[j]]&#xA;    #Need to test for magic and suit also - left as an exercise!&#xA;    if(is.element(current$number, playersdeck$number)){&#xA;      #Update current card&#xA;      current = playersdeck[match(current$number,&#xA;        playersdeck$number),]&#xA;      #Remove card from players deck&#xA;      playersdeck[match(current$number, playersdeck$number),] = c(0,&#xA;                   0, 0)&#xA;    } else {&#xA;      #Add card to players deck&#xA;      playersdeck[i,] = deck[i,]&#xA;      i = i + 1&#xA;    }&#xA;    players[[j]] = playersdeck&#xA;    #Has someone won or have we run out of card&#xA;    if(sum(playersdeck$number) == 0 | i &amp;gt; 52){&#xA;      i = 53&#xA;      break&#xA;    }&#xA;  }&#xA;}&#xA;&#xA;#How many cards are left for each player&#xA;for(i in 1:length(players))&#xA;{&#xA;  cat(sum(players[[i]]$number !=0), &quot;\n&quot;) &#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Some comments&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;You will need to add a couple of lines for magic cards and suits, but data structure is already there. I presume you didn't want a complete solution? ;)&lt;/li&gt;&#xA;&lt;li&gt;To estimate the average game length, just place the above code in a function and call lots of times.&lt;/li&gt;&#xA;&lt;li&gt;Rather than dynamically increasing a vector when a player gets a card, I find it easier just to create a sparse data frame that is more than sufficient. In this case, each player has a data frame with 52 rows, which they will never fill (unless it's a 1 player game).&lt;/li&gt;&#xA;&lt;li&gt;There is a small element of strategy with this game. What should you do if you can play more than one card. For example, if 7H comes up, and you have in your hand 7S, 8H and the JC. All three of these cards are &quot;playable&quot;.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="8" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-08-09T16:31:07.027" LastActivityDate="2010-08-09T16:31:07.027" CommentCount="1" />
  <row Id="1365" PostTypeId="2" ParentId="1358" CreationDate="2010-08-06T12:38:35.230" Score="7" ViewCount="" Body="&lt;p&gt;The moments are the Fourier coefficients of the probability measure $P^Z$.  Suppose (for the sake of intuition) that $Z$ has a density.  Then the argument (angle from $1$ in the complex plane) of $Z$ has a density on $[0,2\pi)$, and the moments are the coefficients when that density is expanded in a Fourier series.  Thus the usual intuition about Fourier series applies -- these measure the strengths of frequencies in that density.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for your second question, I think you already gave the answer: &quot;complex multiplication is essentially addition of angles in this case&quot;.&lt;/p&gt;&#xA;" OwnerUserId="89" LastActivityDate="2010-08-06T12:38:35.230" CommentCount="3" />
  <row Id="1366" PostTypeId="2" ParentId="1315" CreationDate="2010-08-06T12:46:57.243" Score="2" ViewCount="" Body="&lt;p&gt;Another approach, that is more justified by network considerations, is to try to fit a sum of independent exponentials with different parameters. A reasonable assumption would be that each node in the path of the ping the delay would be an independent exponential, with different parameters. A reference to the distributional form of the sum of independent exponentials with differing parameters is &lt;a href=&quot;http://www.math.bme.hu/~balazs/sumexp.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.math.bme.hu/~balazs/sumexp.pdf&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should probably also look at the ping times vs the number of hops.&lt;/p&gt;&#xA;" OwnerUserId="247" LastActivityDate="2010-08-06T12:46:57.243" />
  <row Id="1367" PostTypeId="2" ParentId="1337" CreationDate="2010-08-06T13:52:50.747" Score="8" ViewCount="" Body="&lt;p&gt;One passed by Gary Ramseyer:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Statistics play an important role in genetics. For instance, statistics prove that numbers of offspring is an inherited trait. If your parent didn't have any kids, odds are you won't either.&lt;/p&gt;&#xA;" OwnerUserId="634" LastActivityDate="2010-08-06T13:52:50.747" CommunityOwnedDate="2010-08-06T13:52:50.747" />
  <row Id="1368" PostTypeId="2" ParentId="1337" CreationDate="2010-08-06T14:00:40.623" Score="1" ViewCount="" Body="&lt;p&gt;A statistic professor plans to travel to a conference by plane. When he passes the security check, they discover a bomb in his carry-on-baggage. Of course, he is hauled off immediately for interrogation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;I don't understand it!&quot; the interrogating officer exclaims. &quot;You're an accomplished professional, a caring family man, a pillar of your parish - and now you want to destroy that all by blowing up an airplane!&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Sorry&quot;, the professor interrupts him. &quot;I had never intended to blow up the plane.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;So, for what reason else did you try to bring a bomb on board?!&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Let me explain. Statistics shows that the probability of a bomb being on an airplane is 1/1000. That's quite high if you think about it - so high that I wouldn't have any peace of mind on a flight.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;And what does this have to do with you bringing a bomb on board of a plane?&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;You see, since the probability of one bomb being on my plane is 1/1000, the chance that there are two bombs is 1/1000000. If I already bring one, the chance of another bomb being around is actually 1/1000000, and I am much safer...&quot; &lt;/p&gt;&#xA;" LastActivityDate="2010-08-06T14:00:40.623" CommunityOwnedDate="2010-08-06T14:00:40.623" CommentCount="1" />
  <row Id="1369" PostTypeId="1" AcceptedAnswerId="1372" CreationDate="2010-08-06T14:01:57.110" Score="0" ViewCount="86" Body="&lt;p&gt;I have a given distance with a standard deviation. I have simulated now a few 100 distances and would like to draw from these distances a sample of 10-20 resembling the original distribution. Is there any standardized way of doing so? &lt;/p&gt;&#xA;" OwnerUserId="791" LastEditorUserId="791" LastEditorDisplayName="" LastEditDate="2010-08-06T14:14:37.210" LastActivityDate="2010-08-06T15:14:46.647" Title="Sampling according to a normal distribution" Tags="&lt;sample&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="1370" PostTypeId="2" ParentId="1357" CreationDate="2010-08-06T14:10:01.230" Score="5" ViewCount="" Body="&lt;p&gt;I think the answer is yes to your question. I will show this for the discrete case only and I think the basic idea carries over to the continuous case. MI is defined as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$I(X;Y) = \sum_{y\in Y}\sum_{x\in X}\Bigg(p(x,y) log(\frac{p(x,y)}{p(x)p(y)})\Bigg)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Define:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Z_x = \alpha X$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Z_y = \alpha Y$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, the question is: Does $I(Z_x;Z_y)$ equal $I(X;Y)$?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since scaling is a one-to-one transformation it must be that:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$p(z_x) = p(x)$, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$p(z_y) = p(y)$ and&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$p(z_x,z_y) = p(x,y)$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, the mutual information remains the same and hence the answer is to your question is yes.&lt;/p&gt;&#xA;" LastActivityDate="2010-08-06T14:10:01.230" CommentCount="1" />
  <row Id="1371" PostTypeId="2" ParentId="1337" CreationDate="2010-08-06T14:12:21.147" Score="11" ViewCount="" Body="&lt;p&gt;George Burns said that &quot;If you live to be one hundred, you've got it made. Very few people die past that age.&quot;&lt;/p&gt;&#xA;" OwnerUserId="666" LastActivityDate="2010-08-06T14:12:21.147" CommunityOwnedDate="2010-08-06T14:12:21.147" CommentCount="1" />
  <row Id="1372" PostTypeId="2" ParentId="1369" CreationDate="2010-08-06T15:14:46.647" Score="2" ViewCount="" Body="&lt;p&gt;You mean you want to draw 10-20 numbers from a normal distribution? In R, use &lt;code&gt;rnorm&lt;/code&gt; function; for a generic solution, see &lt;a href=&quot;http://en.wikipedia.org/wiki/Normal_distribution#Generating_values_from_normal_distribution&quot; rel=&quot;nofollow&quot;&gt;Wikipedia&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-08-06T15:14:46.647" CommentCount="1" />
  <row Id="1373" PostTypeId="2" ParentId="1357" CreationDate="2010-08-06T15:22:43.663" Score="1" ViewCount="" Body="&lt;p&gt;Intuitive explanation is such: multiplying by constant does not change information content of X and Y, so also their mutual information -- and thus it is invariant to scaling. Still Srikant gave you a strict proof of this fact.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-08-06T15:22:43.663" />
  <row Id="1374" PostTypeId="2" ParentId="1337" CreationDate="2010-08-06T16:10:48.470" Score="3" ViewCount="" Body="&lt;p&gt;How many statisticians does it take to change a light bulb?&lt;br&gt;&#xA;5–7, with p-value 0.01&lt;/p&gt;&#xA;" OwnerUserId="88" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-08-06T17:54:34.830" LastActivityDate="2010-08-06T17:54:34.830" CommunityOwnedDate="2010-08-06T16:10:48.470" CommentCount="3" />
  <row Id="1375" PostTypeId="2" ParentId="1337" CreationDate="2010-08-06T18:42:26.487" Score="1" ViewCount="" Body="&lt;p&gt;Here is a list of many fun statistics jokes (&lt;a href=&quot;http://www.btinternet.com/~se16/hgb/statjoke.htm&quot; rel=&quot;nofollow&quot;&gt;link&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are just a few:&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Did you hear the one about the statistician? Probably....&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;It is proven that the celebration of birthdays is healthy. Statistics show that those people who celebrate the most birthdays become the oldest. -- S. den Hartog, Ph D. Thesis Universtity of Groningen.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;A statistician is a person who draws a mathematically precise line from an unwarranted assumption to a foregone conclusion.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;The average statistician is just plain mean.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;And there is also the one from a TED talk:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;A friend asked my wife what I do.  She answered that I model.  Model what, she was asked - he models genes, she answered.&quot;&lt;/p&gt;&#xA;" OwnerUserId="253" LastEditorUserId="556" LastEditorDisplayName="" LastEditDate="2010-09-01T23:10:22.907" LastActivityDate="2010-09-01T23:10:22.907" CommunityOwnedDate="2010-08-06T18:42:26.487" />
  <row Id="1376" PostTypeId="1" AcceptedAnswerId="1397" CreationDate="2010-08-06T19:02:08.477" Score="4" ViewCount="158" Body="&lt;p&gt;I am looking for a robust version of Hotelling's $T^2$ test for the mean of a vector. As data, I have a $m\ \times\  n$ matrix, $X$, each row an i.i.d. sample of an $n$-dimensional RV, $x$. The null hypothesis I wish to test is $E[x] = \mu$, where $\mu$ is a fixed $n$-dimensional vector. The classical Hotelling test appears to be susceptible to non-normality in the distribution of $x$ (just as the 1-d analogue, the Student t-test is susceptible to skew and kurtosis). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;what is the state of the art robust version of this test? I am looking for something relatively fast and conceptually simple. There was a paper in COMPSTAT 2008 on the topic, but I do not have access to the proceedings. Any help? &lt;/p&gt;&#xA;" OwnerUserId="795" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-16T06:57:55.643" LastActivityDate="2010-09-16T06:57:55.643" Title="Robust version of Hotelling $T^2$ test" Tags="&lt;robust&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="1377" PostTypeId="2" ParentId="1337" CreationDate="2010-08-06T20:07:17.070" Score="1" ViewCount="" Body="&lt;p&gt;&quot;If you torture data enough it will confess&quot; one of my professors &lt;/p&gt;&#xA;" OwnerUserId="236" LastActivityDate="2010-08-06T20:07:17.070" CommunityOwnedDate="2010-08-06T20:07:17.070" CommentCount="1" />
  <row Id="1378" PostTypeId="1" CreationDate="2010-08-06T21:54:11.230" Score="5" ViewCount="116" Body="&lt;p&gt;I have a dataset that contains ~7,500 blood tests from ~2,500 individuals.  I'm trying to find out if variability in the blood tests increases or decreases with the time between two tests.  For example - I draw your blood for the baseline test, then immediately draw a second sample.  Six months later, I draw another sample.  One might expect the difference between the baseline and the immediate repeat tests to be smaller than the difference between the baseline and the six-month test.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each point on the plot below reflects the difference between two tests.  X is the number of days between two tests; Y is the size of the difference between the two tests.  As you can see, tests aren't evenly distributed along X - the study wasn't designed to address this question, really.  Because the points are so heavily stacked at the mean, I've included 95% (blue) and 99% (red) quantile lines, based on 28-day windows.  These are obviously pulled around by the more extreme points, but you get the idea.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://a.imageshack.us/img175/6595/diffsbydays.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It looks to me like the variability is fairly stable.  If anything, it's higher when the test is repeated within a short period - that's terribly counterintuitive.  How can I address this in a systematic way, accounting for varying n at each time point (and some periods with no tests at all)?  Your ideas are greatly appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just for reference, this is the distribution of the number of days between test and retest:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://a.imageshack.us/img697/6572/testsateachtimepoint.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="71" LastActivityDate="2010-08-17T06:29:42.563" Title="Estimating variability over time" Tags="&lt;repeated-measures&gt;&lt;variability&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="1" />
  <row Id="1379" PostTypeId="2" ParentId="1202" CreationDate="2010-08-06T22:10:44.290" Score="0" ViewCount="" Body="&lt;p&gt;you could compute a &lt;a href=&quot;http://en.wikipedia.org/wiki/Kolmogorov_Smirnov_Test&quot; rel=&quot;nofollow&quot; title=&quot;Kolmogorov Smirnov&quot;&gt;Kolmogorov-Smirnov&lt;/a&gt; statistic based on your binned data. This would work by first computing an empirical CDF based on your bins (just a cumulative sum with rescaling), then compute the $\infty$-norm of the differences.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't know R well enough to give you code in R, but can quote the Matlab very simply:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;%let base be the 1 x 1024 vector of binned observed data&#xA;%let dists be the 1000 x 1024 matrix of binned distributions to be checked&#xA;emp_base = cumsum(base,2);emp_base = emp_base ./ emp_base(end);  %cumsum, then normalize&#xA;emp_dists = cumsum(dists,2);&#xA;emp_dists = bsxfun(@rdivide,emp_dists,emp_dists(:,end));   %normalize for the top sum.&#xA;emp_diff = bsxfun(@minus,emp_base,emp_dists);   %subtract the cdfs; R does this transparently, IIRC&#xA;KS_stat = max(abs(emp_diff),[],2);   %take the maximum absolute difference along dim 2&#xA;%KS_stat is now a 1000 x 1 vector of the KS statistic. you can convert to a p-value as well.&#xA;%but you might as well just rank them.&#xA;[dum,sort_idx] = sort(KS_stat);  %Matlab does not have a builtin ranking; PITA!&#xA;dist_ranks = nan(numel(sort_idx),1);&#xA;dist_ranks(sort_idx) = (1:numel(sort_idx))';&#xA;%dist_ranks are now the ranks of each of the distributions (ignoring ties!)&#xA;%if you want the best match, it is indexed by sort_idx(1);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;the &lt;code&gt;bsxfun&lt;/code&gt; nonsense here is Matlab's way of doing proper vector recycling, which R (and numpy, IIRC) does transparently.&lt;/p&gt;&#xA;" OwnerUserId="795" LastEditorUserId="795" LastEditorDisplayName="" LastEditDate="2010-08-07T16:00:25.753" LastActivityDate="2010-08-07T16:00:25.753" />
  <row Id="1380" PostTypeId="1" AcceptedAnswerId="1636" CreationDate="2010-08-06T22:16:01.897" Score="1" ViewCount="276" Body="&lt;p&gt;(migrating from math overflow, where no answers were posted)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;suppose I have $K$ different methods for forecasting a binary random variable, which I test on independent sets of data, resulting in $K$ contingency tables of values $n_{ijk}$ for $i,j=1,2$ and $k=1,2,...,K$. How can I compare these methods based on the contingency tables?  The general case would be nice, but $K=2$ is also very interesting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can think of a few approaches:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;compute &lt;em&gt;some&lt;/em&gt; statistic on each of the tables, and compare those random variables (I'm not sure if this is a standard problem or not),&lt;/li&gt;&#xA;&lt;li&gt;something like &lt;a href=&quot;http://www.jstor.org/pss/2982447&quot; rel=&quot;nofollow&quot; title=&quot;On methods of comparing contingency tables&quot;&gt;Goodman's improvement of Stouffer's method&lt;/a&gt;, but I cannot access this paper, and was hoping for something a little more recent (more likely to have the latest-greatest, plus computer simulations).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;any ideas?&lt;/p&gt;&#xA;" OwnerUserId="795" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-30T21:20:58.353" LastActivityDate="2010-10-01T01:34:16.313" Title="Comparing multiple contingency tables, independent data" Tags="&lt;forecasting&gt;&lt;contingency-tables&gt;" AnswerCount="2" />
  <row Id="1381" PostTypeId="2" ParentId="1001" CreationDate="2010-08-06T22:34:27.163" Score="3" ViewCount="" Body="&lt;p&gt;The Baumgartner-Weiss-Schindler statistic appears to be the modern alternative to the K-S test, and appears to be more powerful in certain situations. A few links:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.jstor.org/stable/2533862&quot; rel=&quot;nofollow&quot;&gt;A Nonparametric Test for the General Two-Sample Problem&lt;/a&gt; (the original B.W.S. paper)&lt;/li&gt;&#xA;&lt;li&gt;M. Neuhauser, &lt;a href=&quot;http://www.springerlink.com/content/j7270477094575hn/&quot; rel=&quot;nofollow&quot;&gt;'Exact Tests Based on the Baumgartner-Weiss-Schindler Statistic--A Survey'&lt;/a&gt;, Statistical Papers, Vol 46 (2005), pp. 1-30. (perhaps not relevant to your large sample case...)&lt;/li&gt;&#xA;&lt;li&gt;H. Murakami, 'K-Sample Rank Test Based on Modified Baumgartner Statistic and its Power Comparison', J. Jpn. Comp. Statist. Vol 19 (2006), pp. 1-13.&lt;/li&gt;&#xA;&lt;li&gt;M. Neuhauser, 'One-Sided Two-Sample and Trend Tests Based on a Modified Baumgartner-Weiss-Schindler Statistic', J. Nonparametric Statistics, Vol 13 (2001) pp 729-739.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-08-06T22:34:27.163" />
  <row Id="1382" PostTypeId="2" ParentId="1337" CreationDate="2010-08-07T02:33:35.130" Score="5" ViewCount="" Body="&lt;p&gt;A statistics major was completely hung over the day of his final exam. It was a true/false test, so he decided to flip a coin for the answers. The statistics professor watched the student the entire two hours as he was flipping the coin … writing the answer … flipping the coin … writing the answer. At the end of the two hours, everyone else had left the final except for the one student. The professor walks up to his desk and interrupts the student, saying, “Listen, I have seen that you did not study for this statistics test, you didn’t even open the exam. If you are just flipping a coin for your answer, what is taking you so long?”&#xA;The student replies bitterly (as he is still flipping the coin), “Shhh! I am checking my answers!”&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've posted a few others on &lt;a href=&quot;http://robjhyndman.com/researchtips/statistical-jokes/&quot; rel=&quot;nofollow&quot;&gt;my blog&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-08-07T02:33:35.130" CommunityOwnedDate="2010-08-07T02:33:35.130" />
  <row Id="1383" PostTypeId="1" AcceptedAnswerId="1594" CreationDate="2010-08-07T03:47:26.330" Score="0" ViewCount="238" Body="&lt;p&gt;There's a lot of work done in statistics,&#xA;while state-of-art in lossless data compression is apparently this:&#xA;&lt;a href=&quot;http://mattmahoney.net/dc/dce.html#Section_4&quot; rel=&quot;nofollow&quot;&gt;http://mattmahoney.net/dc/dce.html#Section_4&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please suggest good methods/models applicable for data compression.&#xA;To be specific:&lt;br&gt;&#xA;1) How to estimate the probability of the next bit in a bit string?&lt;br&gt;&#xA;2) How to integrate predictions of different models?&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Update:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;you should include a better description of what data you&#xA;  want to compress...&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Why, I'm talking about universal compression obviously.&#xA;For data with known structure its really not a mathematical problem,&#xA;so there's no sense to discuss it here.&#xA;In other words, the first question is: given a string of bits, &#xA;what do we do to determine the probability of the next bit, as&#xA;precisely as possible?&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;otherwise we will have 10 different answers trying to&#xA;  summarize different part of the huge theory of compression&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I'd written quite a few statistical compressors, and I'm&#xA;not interested in that.&#xA;I'm asking how a statistician would approach this task,&#xA;detect correlations in given data, and compute a probability&#xA;estimation for the next bit.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In addition, the two point you give to be more specific&#xA;  are not detailed enough to be understood.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;What's not detailed in there? I'm even talking about bits,&#xA;not some vague &quot;symbols&quot;. I'd note though, that I'm talking&#xA;about &quot;probability of a bit&quot; because computing a probability&#xA;of bit==0 or bit==1 is a matter of convenience.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, I'm obviously not talking about some &quot;random data compression&quot;,&#xA;or methods with infinite complexity, like &quot;Kolmogorov compression&quot;.&#xA;Again, I want to know how a good statistician would approach this&#xA;problem, given a string of bits.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's an example, if you need one: hxxp://encode.ru/threads/482-Bit-guessing-game&lt;/p&gt;&#xA;" OwnerUserId="799" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-08-12T13:39:10.303" LastActivityDate="2010-08-14T12:10:58.203" Title="Suggest a method for statistical data compression" Tags="&lt;modeling&gt;&lt;compression&gt;" AnswerCount="2" CommentCount="3" />
  <row Id="1384" PostTypeId="2" ParentId="1352" CreationDate="2010-08-07T03:55:36.433" Score="4" ViewCount="" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;I agree with the point that statistics consultants are often brought in later on a project when it's too late to remedy design flaws. It's also true that many statistics books give scant attention to study design issues.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;You say you want designs &quot;preferably for a wide range of methods (e.g. t-test, GLM, GAM, ordination techniques...&quot;. I see designs as relatively independent of statistical method: e.g., experiments (between subjects and within subjects factors) versus observational studies; longitudinal versus cross-sectional; etc. There are also a lot of issues related to measurement, domain specific theoretical knowledge, and domain specific study design principles that need to be understood in order to design a good study.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;In terms of books, I'd be inclined to look at domain specific books. In psychology (where I'm from) this means books on psychometrics for measurement, a book on research methods, and a book on statistics, as well as a range of even more domain specific research method books. You might want to check out &lt;a href=&quot;http://www.socialresearchmethods.net/kb/index.php&quot; rel=&quot;nofollow&quot;&gt;Research Methods Knowledge Base&lt;/a&gt; for a free online resource for the social sciences.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Published journal articles are also a good guide to what is best practice in a particular domain. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-08-08T11:31:31.680" LastActivityDate="2010-08-08T11:31:31.680" CommunityOwnedDate="2010-08-07T03:55:36.433" />
  <row Id="1385" PostTypeId="1" CreationDate="2010-08-07T05:07:27.083" Score="8" ViewCount="300" Body="&lt;p&gt;My question is directed to techniques to deal with &lt;em&gt;incomplete&lt;/em&gt; data during the classifier/model training/fitting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, in a dataset w/ a few hundred rows, each row having let's say five dimensions and a class label as the last item, most data points will look like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[0.74, 0.39, 0.14, 0.33, 0.34, 0]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A few might look something like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;[0.21, 0.68, ?, 0.82, 0.58, 1]&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So it's those types of data points that are the focus of this Question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My initial reason for asking this question was a problem directly in front of me; however, before posting my Question, i thought it might be more useful if i re-phrased it so the answers would be useful to a larger portion of the Community.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a simple heuristic, let's divide these data-handling techniques based on when during the processing flow they are employed--before input to the classifier or during (i.e., the technique is inside the classifier).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The best example i can think of for the latter is the clever 'three-way branching' technique used in Decision Trees.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;No doubt, the former category is far larger. The techniques i am aware of all fall into one of the groups below. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;While recently reviewing my personal notes on &quot;missing data handling&quot; i noticed that i had quite an impressive list of techniques. I just maintain these notes for general peace of mind and in case a junior colleague asks me how to deal with missing data. In actual practice, i don't actually use any of them, except for the last one.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Imputation&lt;/strong&gt;: a broad rubric for a set of techniques which whose common&#xA;denominator (i believe) is that the&#xA;missing data is supplied directly by&#xA;the same data set--substitution&#xA;rather than estimation/prediction.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Reconstruction&lt;/strong&gt;: estimate the missing data points using an&#xA;auto-associative network (just a&#xA;neural network in which the sizes of&#xA;the input and output layers are&#xA;equal--in other words, the output&#xA;has the same dimension as the&#xA;input); the idea here is to train&#xA;this network on complete data, then&#xA;feed it incomplete patterns, and&#xA;read the missing values from the&#xA;output nodes.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Bootstrapping&lt;/strong&gt;: (no summary necessary i shouldn't think, given&#xA;it's use elsewhere in statistical&#xA;analysis).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Denial&lt;/strong&gt;: quietly remove the data points with missing/corrupt elements&#xA;from your training set and pretend&#xA;they never existed.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="438" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-16T06:47:49.967" LastActivityDate="2010-09-16T06:47:49.967" Title="Techniques for Handling Incomplete/Missing Data" Tags="&lt;missing-data&gt;" AnswerCount="3" CommentCount="1" FavoriteCount="6" />
  <row Id="1386" PostTypeId="1" AcceptedAnswerId="1390" CreationDate="2010-08-07T05:18:58.967" Score="6" ViewCount="571" Body="&lt;p&gt;I am trying to test the null $E[X] = 0$, against the local alternative $E[X] &gt; 0$, for a random variable $X$, subject to mild to medium skew and kurtosis of the random variable. Following suggestions by Wilcox in 'Introduction to Robust Estimation and Hypothesis Testing', I have looked at tests based on the trimmed mean, the median, as well as the M-estimator of location (Wilcox' &quot;one-step&quot; procedure). These robust tests do outperform the standard t-test, in terms of power, when testing with a distribution that is non-skewed, but leptokurtotic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, when testing with a distribution that is skewed, these one-sided tests are either far too liberal or far too conservative under the null hypothesis, depending on whether the distribution is left- or right-skewed, respectively. For example, with 1000 observations, the test based on the median will actually reject ~40% of the time, at the nominal 5% level. The reason for this is obvious: for skewed distributions, the median and the mean are rather different. However, in my application, I really need to test the mean, not the median, not the trimmed mean.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The question, then: is there a more robust version of the t-test that actually tests for the mean, but is impervious to skew and kurtosis? Ideally the procedure would work well in the no-skew, high-kurtosis case as well. The 'one-step' test is almost good enough, with the 'bend' parameter set relatively high, but it is less powerful than the trimmed mean tests when there is no skew, and has some troubles maintaining the nominal level of rejects under skew.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;background:&lt;/strong&gt; the reason I really care about the mean, and not the median, is that the test would be used in a financial application. For example, if you wanted to test whether a portfolio had positive expected log returns, the mean is actually appropriate because if you invest in the portfolio, you will experience all the returns (which is the mean times the number of samples), instead of $n$ duplicates of the median.  That is, I really care about the sum of $n$ draws from the R.V. $X$. &lt;/p&gt;&#xA;" OwnerUserId="795" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-16T07:16:08.673" LastActivityDate="2010-09-17T21:21:15.497" Title="Robust t-test for mean" Tags="&lt;hypothesis-testing&gt;&lt;t-test&gt;&lt;finance&gt;&lt;robust&gt;" AnswerCount="2" CommentCount="3" FavoriteCount="4" />
  <row Id="1387" PostTypeId="2" ParentId="1337" CreationDate="2010-08-07T06:03:39.740" Score="2" ViewCount="" Body="&lt;p&gt;there was the one about the two statisticians who tried to use grant money to pay for their bill at a strip club.  They were vindicated when it was explained they were performing a 'posterior analysis'.  (groan)&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-08-07T06:03:39.740" CommunityOwnedDate="2010-08-07T06:03:39.740" />
  <row Id="1388" PostTypeId="2" ParentId="1337" CreationDate="2010-08-07T07:15:20.137" Score="11" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;A statistician's wife had twins. He&#xA;  was delighted. He rang the minister&#xA;  who was also delighted. &quot;Bring them to&#xA;  church on Sunday and we'll baptize&#xA;  them,&quot; said the minister. &quot;No,&quot;&#xA;  replied the statistician. &quot;Baptize&#xA;  one. We'll keep the other as a&#xA;  control.&quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;STATS: The Magazine For Students of Statistics, Winter 1996, Number 15&lt;/em&gt;&lt;/p&gt;&#xA;" LastActivityDate="2010-08-07T07:15:20.137" CommunityOwnedDate="2010-08-07T07:15:20.137" CommentCount="1" />
  <row Id="1389" PostTypeId="1" AcceptedAnswerId="1392" CreationDate="2010-08-07T07:23:07.937" Score="4" ViewCount="175" Body="&lt;p&gt;I came across an error of numerical overflow when running a maximum likelihood estimation on a log-linear specification. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What does numerical overflow mean?&lt;/p&gt;&#xA;" OwnerUserId="189" LastActivityDate="2010-08-07T22:38:33.683" Title="What is numerical overflow?" Tags="&lt;estimation&gt;&lt;maximum-likelihood&gt;" AnswerCount="3" FavoriteCount="1" />
  <row Id="1390" PostTypeId="2" ParentId="1386" CreationDate="2010-08-07T07:23:55.127" Score="4" ViewCount="" Body="&lt;p&gt;Why are you looking at non-parametric tests?  Are the assumptions of the t-test violated?  Namely, ordinal or non-normal data and inconstant variances?  Of course, if your sample is large enough you can justify the parametric t-test with its greater power despite the lack of normality in the sample.  Likewise if your concern is unequal variances, there are corrections to the parametric test that yield accurate p-values (the Welch correction).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Otherwise, comparing your results to the t-test is not a good way to go about this, because the t-test results are biased when the assumptions are not met.  The Mann-Whitney U is an appropriate non-parametric alternative, if that's what you really need.  You only lose power if you are using the non-parametric test when you could justifiably use the t-test (because the assumptions are met).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And, just for some more background, go here...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.jerrydallal.com/LHSP/STUDENT.HTM&quot; rel=&quot;nofollow&quot;&gt;http://www.jerrydallal.com/LHSP/STUDENT.HTM&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="485" LastEditorUserId="485" LastEditorDisplayName="" LastEditDate="2010-08-07T07:30:04.990" LastActivityDate="2010-08-07T07:30:04.990" CommentCount="2" />
  <row Id="1391" PostTypeId="2" ParentId="1386" CreationDate="2010-08-07T07:34:44.433" Score="6" ViewCount="" Body="&lt;p&gt;I agree that if you want to actually test whether the group means are different (as opposed to testing differences between group medians or trimmed means, etc.), then you don't want to use a nonparametric test that tests a different hypothesis.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;In general p-values from a t-test tend to be fairly accurate given moderate departures of the assumption of normality of residuals.&#xA;Check out this applet to get an intuition on this robustness: &lt;a href=&quot;http://onlinestatbook.com/stat_sim/robustness/index.html&quot; rel=&quot;nofollow&quot;&gt;http://onlinestatbook.com/stat_sim/robustness/index.html&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If you're still concerned about the violation of the normality assumption, &#xA;you might want to &lt;strong&gt;bootstrap&lt;/strong&gt;.&#xA;e.g., &lt;a href=&quot;http://biostat.mc.vanderbilt.edu/wiki/pub/Main/JenniferThompson/ms_mtg_18oct07.pdf&quot; rel=&quot;nofollow&quot;&gt;http://biostat.mc.vanderbilt.edu/wiki/pub/Main/JenniferThompson/ms_mtg_18oct07.pdf&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;You could also &lt;strong&gt;transform&lt;/strong&gt; the skewed dependent variable to resolve issues with departures from normality.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-08-07T09:39:58.333" LastActivityDate="2010-08-07T09:39:58.333" CommentCount="4" />
  <row Id="1392" PostTypeId="2" ParentId="1389" CreationDate="2010-08-07T07:35:58.240" Score="6" ViewCount="" Body="&lt;p&gt;It means that the algorithm generated a variable that is greater than the maximum allowed for that type of variable. That is due to the fact that computers use a finite number of bits to represent numbers, so it is not possible to represent ANY number, but only a limited subset of them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The actual value depends on the type of variable and the architecture of the system.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why that happens during a MLE I'm not sure, my best call would be that you should change the starting parameters.&lt;/p&gt;&#xA;" OwnerUserId="582" LastActivityDate="2010-08-07T07:35:58.240" />
  <row Id="1393" PostTypeId="2" ParentId="1268" CreationDate="2010-08-07T08:12:09.650" Score="4" ViewCount="" Body="&lt;p&gt;There is a reasonably new area of research called &lt;em&gt;Matrix Completion&lt;/em&gt;, that probably does what you want. A really nice introduction is given in this &lt;a href=&quot;http://videolectures.net/mlss09us_candes_mccota/&quot; rel=&quot;nofollow&quot;&gt;lecture&lt;/a&gt; by Emmanuel Candes &lt;/p&gt;&#xA;" OwnerUserId="352" LastEditorUserId="352" LastEditorDisplayName="" LastEditDate="2010-08-07T09:29:45.423" LastActivityDate="2010-08-07T09:29:45.423" CommentCount="3" />
  <row Id="1394" PostTypeId="2" ParentId="1274" CreationDate="2010-08-07T08:40:07.287" Score="0" ViewCount="" Body="&lt;p&gt;There's a clear time trend (decreasing average count). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hence (unless you condition on that trend) they're dependent (values close together in time will be more alike than values further apart in time). &lt;/p&gt;&#xA;" OwnerUserId="805" LastActivityDate="2010-08-07T08:40:07.287" />
  <row Id="1395" PostTypeId="1" AcceptedAnswerId="1402" CreationDate="2010-08-07T13:53:16.873" Score="4" ViewCount="136" Body="&lt;p&gt;Can anyone recommend me an open source graphic library to create forest and funnel plots?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was aiming at using it on a Java desktop application.&lt;/p&gt;&#xA;" OwnerUserId="807" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-08-11T10:53:09.990" LastActivityDate="2010-08-11T10:53:09.990" Title="Libraries for forest and funnel plots" Tags="&lt;data-visualization&gt;&lt;funnel-plot&gt;&lt;java&gt;" AnswerCount="3" FavoriteCount="1" />
  <row Id="1396" PostTypeId="2" ParentId="1389" CreationDate="2010-08-07T14:36:40.593" Score="3" ViewCount="" Body="&lt;p&gt;You can probably avoid your overflow problems by working with the log of the likelihood function rather than the likelihood function itself.  Both have the same maximum.&lt;/p&gt;&#xA;" OwnerUserId="319" LastActivityDate="2010-08-07T14:36:40.593" CommentCount="1" />
  <row Id="1397" PostTypeId="2" ParentId="1376" CreationDate="2010-08-07T14:40:50.007" Score="2" ViewCount="" Body="&lt;p&gt;Sure: two answers&lt;/p&gt;&#xA;&#xA;&lt;p&gt;a) If by robustness, you mean robust to outliers, then run Hottelling's T-test using a robust estimation of scale/scatter: you will find all the explications and R code here:&#xA;&lt;a href=&quot;http://www.statsravingmad.com/blog/statistics/a-robust-hotelling-test/&quot; rel=&quot;nofollow&quot;&gt;http://www.statsravingmad.com/blog/statistics/a-robust-hotelling-test/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;b) if by robustness you mean optimal under large group of distributions, then you should go for a sign based T2 (ask if this what you want, by the tone of your question i think not).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: this is the paper you want;&#xA;Roelant, E., Van Aelst, S., and Willems, G. (2008), “Fast Bootstrap for Robust Hotelling Tests,” COMPSTAT 2008: Proceedings in Computational Statistics (P. Brito, Ed.) Heidelberg: Physika-Verlag, to appear.&lt;/p&gt;&#xA;" OwnerUserId="603" LastEditorUserId="603" LastEditorDisplayName="" LastEditDate="2010-08-07T15:39:38.530" LastActivityDate="2010-08-07T15:39:38.530" CommentCount="1" />
  <row Id="1398" PostTypeId="2" ParentId="1378" CreationDate="2010-08-07T15:24:01.240" Score="0" ViewCount="" Body="&lt;p&gt;There was an article on LFT (liver function tests) and how it seemed to vary in a cyclic basis (in months). I think it was in Annals of Internal Medicine. &lt;/p&gt;&#xA;" OwnerUserId="812" LastActivityDate="2010-08-07T15:24:01.240" />
  <row Id="1399" PostTypeId="1" AcceptedAnswerId="1401" CreationDate="2010-08-07T18:10:02.220" Score="5" ViewCount="266" Body="&lt;p&gt;I'm interested in obtaining a bootstrapped confidence interval on quantity X, when this quantity is measured 10 times in each of 10 individuals.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One approach is to obtain the mean per individual, then bootstrap the means (eg. resample the means with replacement).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another approach is to do the following on each iteration of the bootstrapping procedure: within each individual, resample that individual's 10 observations with replacement, then compute a new mean for that individual, and finally compute a new group mean. In this approach, each individual observed in the original data set always contribute to the group mean on each iteration of the bootstrap procedure. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, a third approach is to combine the above two approaches: resample individuals then resample within those individuals. This approach differs from the preceding approach in that it permits the same individual to contribute multiply to the group mean on each iteration, though because each contribution is generated via an independent resampling procedure, these contributions may be expected to vary slightly from eachother.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In practice, I find that these approaches yield different estimates for the confidence interval (ex. with one data set, I find that the third approach yields much larger confidence intervals than the first two approaches), so I'm curious what each might be interpreted to represent. &lt;/p&gt;&#xA;" OwnerUserId="364" LastEditorUserId="364" LastEditorDisplayName="" LastEditDate="2010-08-13T00:57:22.773" LastActivityDate="2010-08-23T20:39:21.760" Title="Obtaining and interpreting bootstrapped confidence intervals from hierarchical data" Tags="&lt;confidence-interval&gt;&lt;bootstrap&gt;" AnswerCount="2" FavoriteCount="2" />
  <row Id="1401" PostTypeId="2" ParentId="1399" CreationDate="2010-08-07T18:31:32.663" Score="5" ViewCount="" Body="&lt;p&gt;Your first approach is about a between S CI.  If you wanted to measure within S then that's the wrong approach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second approach would generate a within S CI that would only apply to those 10 individuals.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The last approach is the correct one for the within S CI. Any increases in the CI are because your CI is more representative of a CI that could be applied to the population instead of those 10 S's.&lt;/p&gt;&#xA;" OwnerUserId="601" LastEditorUserId="601" LastEditorDisplayName="" LastEditDate="2010-08-07T18:40:21.040" LastActivityDate="2010-08-07T18:40:21.040" />
  <row Id="1402" PostTypeId="2" ParentId="1395" CreationDate="2010-08-07T21:14:56.077" Score="3" ViewCount="" Body="&lt;p&gt;Well, i use &lt;strong&gt;&lt;a href=&quot;http://www.graphviz.org&quot; rel=&quot;nofollow&quot;&gt;graphviz&lt;/a&gt;&lt;/strong&gt;, which has Java bindings &lt;a href=&quot;http://www2.research.att.com/~john/Grappa/&quot; rel=&quot;nofollow&quot;&gt;(Grappa&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Although the dot language (graphviz's syntax) is simple, i prefer to use graphviz as a library through the excellent and production-stable python bindings, &lt;a href=&quot;http://networkx.lanl.gov/pygraphviz/&quot; rel=&quot;nofollow&quot;&gt;pygraphviz&lt;/a&gt;, and &lt;a href=&quot;http://networkx.lanl.gov/&quot; rel=&quot;nofollow&quot;&gt;networkx&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's the code for a simple 'funnel diagram' using those tools; it's not the most elaborate diagram, but it is complete--it initializes the graph object, creates all of the necessary components, styles them, renders the graph, and writes it to file.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import networkx as NX&#xA;import pygraphviz as PV&#xA;&#xA;G = PV.AGraph(strict=False, directed=True)     # initialize graph object&#xA;&#xA;# create graph components:&#xA;node_list = [&quot;Step1&quot;, &quot;Step2&quot;, &quot;Step3&quot;, &quot;Step4&quot;]&#xA;edge_list = [(&quot;Step1, Step2&quot;), (&quot;Step2&quot;, &quot;Step3&quot;), (&quot;Step3&quot;, &quot;Step4&quot;)]&#xA;G.add_nodes_from(node_list)&#xA;G.add_edge(&quot;Step1&quot;, &quot;Step2&quot;)&#xA;G.add_edge(&quot;Step2&quot;, &quot;Step3&quot;)&#xA;G.add_edge(&quot;Step3&quot;, &quot;Step4&quot;)&#xA;&#xA;# style them:&#xA;nak = &quot;fontname fontsize fontcolor shape style fill color size&quot;.split()&#xA;nav = &quot;Arial 11 white invtrapezium filled cornflowerblue cornflowerblue 1.4&quot;.split()&#xA;nas = dict(zip(nak, nav))&#xA;for k, v in nas.iteritems() :&#xA;    G.node_attr[k] = v&#xA;&#xA;eak = &quot;fontname fontsize fontcolor dir arrowhead arrowsize arrowtail&quot;.split()&#xA;eav = &quot;Arial 10 red4 forward normal 0.8 inv&quot;.split()&#xA;eas = dict(zip(eak, eav))&#xA;for k, v in eas.iteritems() :&#xA;    G.edge_attr[k] = v&#xA;&#xA;n1 = G.get_node(&quot;Step1&quot;)&#xA;n1.attr['fontsize'] = '11'&#xA;n1.attr['fontcolor'] = 'red4'&#xA;n1.attr['label'] = '1411'&#xA;n1.attr['shape'] = 'rectangle'&#xA;n1.attr['width'] = '1.4'&#xA;n1.attr['height'] = '0.05'&#xA;n1.attr['color'] = 'firebrick4'&#xA;n4 = G.get_node(&quot;Step4&quot;)&#xA;n4.attr['shape'] = 'rectangle'&#xA;&#xA;# it's simple to scale graph features to indicate 'flow' conditions, e.g., scale &#xA;# each container size based on how many items each holds in a given time snapshot:&#xA;# (instead of setting node attribute ('width') to a static quantity, you would&#xA;# just bind 'n1.attr['width']' to a variable such as 'total_from_container_1'&#xA;&#xA;n1 = G.get_node(&quot;Step2&quot;)&#xA;n1.attr['width'] = '2.4'&#xA;&#xA;# likewise, you can do the same with edgewidth (i.e., make the arrow thicker&#xA;# to indicate higher 'flow rate')&#xA;&#xA;e1 = G.get_edge(&quot;Step1&quot;, &quot;Step2&quot;)&#xA;e1.attr['label'] = '  1411'&#xA;e1.attr['penwidth'] = 2.6&#xA;&#xA;# and you can easily add labels to the nodes and edges to indicate e.g., quantities: &#xA;e1 = G.get_edge(&quot;Step2&quot;, &quot;Step3&quot;)&#xA;e1.attr['label'] = '  392'&#xA;&#xA;G.write(&quot;conv_fnl.dot&quot;)      # save the dot file&#xA;G.draw(&quot;conv_fnl.png&quot;)       # save the rendered diagram&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://a.imageshack.us/img148/390/convfunnel.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="438" LastEditorUserId="438" LastEditorDisplayName="" LastEditDate="2010-08-11T04:32:31.583" LastActivityDate="2010-08-11T04:32:31.583" />
  <row Id="1403" PostTypeId="2" ParentId="1389" CreationDate="2010-08-07T22:28:44.363" Score="0" ViewCount="" Body="&lt;p&gt;As stated by nico, numerical overflow is when computation finds a number that is too great for the limited number of bits allocated by software to store the number. For example, if your software uses 32 bits to store integers, then computing an integer that is greater than 2,147,483,648 (or smaller than -2,147,483,648) will cause overflow.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One common reason for numerical overflow is trying to divide by a very small (close to zero) number. If the absolute values of your numbers are not too large, Look at your data and try to figure out where you might be dividing by a very small number.&lt;/p&gt;&#xA;" OwnerUserId="666" LastEditorUserId="666" LastEditorDisplayName="" LastEditDate="2010-08-07T22:38:33.683" LastActivityDate="2010-08-07T22:38:33.683" />
  <row Id="1404" PostTypeId="2" ParentId="1385" CreationDate="2010-08-07T22:57:34.860" Score="3" ViewCount="" Body="&lt;p&gt;I gave this answer to &lt;a href=&quot;http://stats.stackexchange.com/questions/1268/svd-dimensionality-reduction-for-time-series-of-different-length/1393#1393&quot;&gt;another question&lt;/a&gt;, but it might apply here too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;There is a reasonably new area of research called &lt;em&gt;Matrix Completion&lt;/em&gt;, that probably does what you want. A really nice introduction is given in this &lt;a href=&quot;http://videolectures.net/mlss09us_candes_mccota/&quot; rel=&quot;nofollow&quot;&gt;lecture&lt;/a&gt; by Emmanuel Candes&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Essentially, if your dataset has low rank (or approximately low rank) i.e. you have 100 rows, but the actual matrix has some small rank, say 10 (or only 10 large singular values), then you can use Matrix Completion to fill the missing data.&lt;/p&gt;&#xA;" OwnerUserId="352" LastActivityDate="2010-08-07T22:57:34.860" />
  <row Id="1405" PostTypeId="1" CreationDate="2010-08-08T00:44:57.630" Score="1" ViewCount="643" Body="&lt;p&gt;I am attempting to compare two diagnostic odds ratios (DORs). I would like to know of a statistical test which will allow me to do this. Please help! Thank you!&lt;/p&gt;&#xA;" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-29T00:28:51.273" LastActivityDate="2011-04-29T00:28:51.273" Title="Statistical test for difference between two odds ratios?" Tags="&lt;hypothesis-testing&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="2" />
  <row Id="1406" PostTypeId="2" ParentId="726" CreationDate="2010-08-08T04:15:18.393" Score="9" ViewCount="" Body="&lt;p&gt;This is unlikely to be a popular quote, but anyway,&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;If your experiment needs statistics, you ought to have done a better experiment.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Ernest Rutherford&lt;/p&gt;&#xA;" OwnerUserId="352" LastActivityDate="2010-08-08T04:15:18.393" CommunityOwnedDate="2010-08-08T04:15:18.393" CommentCount="5" />
  <row Id="1407" PostTypeId="2" ParentId="1350" CreationDate="2010-08-08T04:50:18.187" Score="5" ViewCount="" Body="&lt;p&gt;After a day of searching, I found out that the issue was due to starting values. Thought I should just post the answer for future reference. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The frontier command in Stata obtains its starting values using method of moments. The initial values might have produced negative infinity for the log likelihood. To get around the problem I needed to specify the starting values myself, which were obtained from a linear regression. &lt;/p&gt;&#xA;" OwnerUserId="189" LastActivityDate="2010-08-08T04:50:18.187" />
  <row Id="1408" PostTypeId="2" ParentId="942" CreationDate="2010-08-08T05:55:39.797" Score="4" ViewCount="" Body="&lt;p&gt;Most commonly used and implemented discrete wavelet basis functions (as distinct from the CWT described in Robin's answer) have two nice properties that make them useful for anomaly detection:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;They're compactly supported.&lt;/li&gt;&#xA;&lt;li&gt;They act as band-pass filters with the pass-band determined by their support.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;What this means in practical terms is that your discrete wavelet decomposition looks at local changes in the signal across a variety of scales and frequency bands.  If you have (for instance) large-magnitude, high-frequency noise superimposed across a function that displays a low-magnitude shift over a longer period, the wavelet transform will efficiently separate these two scales and let you see the baseline shift that many other techniques will miss; a shift in this baseline can suggest a disease outbreak or some other change of interest.  In a lot of ways, you can treat the decomposition itself as a smoother (and there's been quite a bit of work done on efficient shrinkage for wavelet coefficients in nonparametric estimation, see e.g. pretty much anything on wavelets by Donoho).  Unlike pure frequency-based methods, the compact support means that they're capable of handling non-stationary data.  Unlike purely time-based methods, they allow for some frequency-based filtering.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In practical terms, to detect anomalies or change points, you would apply a discrete wavelet transform (probably the variant known either as the &quot;Maximum Overlap DWT&quot; or &quot;shift invariant DWT&quot;, depending on who you read) to the data, and look at the lower-frequency sets of coefficients to see if you have significant shifts in the baseline.  This will show you when a long-term change is occurring underneath any day-to-day noise.  Percival and Walden (see references below) derive a few tests for statistically significant coefficients that you could use to see if a shift like this is significant or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An excellent reference work for discrete wavelets is Percival and Walden, &quot;Wavelet Methods for Time Series Analysis&quot;.  A good introductory work is &quot;Introduction to wavelets and wavelet transforms, a primer&quot; by Burrus, Gopinath, and Guo.  If you're coming from an engineering background, then &quot;Elements of wavelets for engineers and scientists&quot; is a good introduction from a signal-processing point of view.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Edited to include Robin's comments)&lt;/p&gt;&#xA;" OwnerUserId="61" LastEditorUserId="61" LastEditorDisplayName="" LastEditDate="2010-08-08T19:22:55.600" LastActivityDate="2010-08-08T19:22:55.600" CommentCount="4" />
  <row Id="1409" PostTypeId="2" ParentId="1395" CreationDate="2010-08-08T11:30:18.780" Score="5" ViewCount="" Body="&lt;p&gt;The &lt;code&gt;rmeta&lt;/code&gt;  package in R can produce forest and funnel plots.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://cran.r-project.org/web/packages/rmeta/index.html&quot; rel=&quot;nofollow&quot;&gt;http://cran.r-project.org/web/packages/rmeta/index.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-08-08T11:30:18.780" />
  <row Id="1410" PostTypeId="2" ParentId="570" CreationDate="2010-08-08T14:28:10.037" Score="2" ViewCount="" Body="&lt;p&gt;I suppose you could do a multidimensional scaling of the correlation or covariance matrix. It's not exactly structural equation modelling, but it might highlight patterns and structure in the correlation or covariance matrix. This could then be formalised with an appropriate model.&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-08-08T14:28:10.037" CommentCount="1" />
  <row Id="1411" PostTypeId="2" ParentId="3" CreationDate="2010-08-08T14:33:24.690" Score="10" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://www.ggobi.org/&quot; rel=&quot;nofollow&quot;&gt;ggobi&lt;/a&gt; &quot;is an open source visualization program for exploring high-dimensional data.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Mat Kelcey has a good &lt;a href=&quot;http://matpalm.com/blog/2010/06/04/5-minute-ggobi-demo/&quot; rel=&quot;nofollow&quot;&gt;5 minute intro to ggobi&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-08-08T14:33:24.690" CommunityOwnedDate="2010-08-08T14:33:24.690" />
  <row Id="1412" PostTypeId="1" CreationDate="2010-08-08T18:10:40.850" Score="3" ViewCount="195" Body="&lt;p&gt;Background:  In some cognitive psychology research areas N-alternative forced choice tasks are common.  The most common of these is a two alternative forced choice (2AFC).  This usually takes the form of participants being given a stimulus and asked to make one of two judgement, e.g. the target stimuli is present/absent, the stimulus on the left is the same/different than the the one on the right, etc.  Designs in which the experimental data is from a 2AFC but there is only one data point per subject are rare, but do exist, e.g. some eye-witness identification research.  Since the dependent variable (correct/incorrect) is binary, these experiments are reasonable places to use logistic regression.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is this:  since chance performance is 50% in a 2AFC trial, is it still reasonable to use the standard logistic link function?  Specifically, the logistic function has a minimum value approaching 0% correct, but in practice participants in a 2AFC should be correct at least 50% of the time due to chance.  I imagine the following case in which it may present a problem:  an independent variable is assessing the difficulty of the discrimination (e.g. difficulty 1, easy - 5, hard; please note this is introduced in ordinal terms only for ease of comprehension - for the sake of this discussion consider this variable as being interval) - participants got 50% correct at 5 and 4, 75% correct at 3, 85% correct at 2, and 99% correct at 1.  Would using a standard logistic link function cause us to underestimate the slope? [I think so, but please correct me if I'm wrong, see below]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit:  Those who have answered my question so far have expressed that the way in which I set up the problem was unclear.  I'm providing the sample below to help clear things up.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(psyphy)&#xA;make.data &amp;lt;- function(zero,one)&#xA;    {&#xA;        return(c(rep(0,zero),rep(1,one)))&#xA;    }&#xA;center &amp;lt;- function(x) {return(scale(x,scale=FALSE))}&#xA;logit.data &amp;lt;- data.frame(Score=c(make.data(50,50),make.data(50,50),make.data(25,75),make.data(15,85),make.data(1,99)), Difficulty=rep(5:1,each=100))&#xA;logit.data$Difficulty2 &amp;lt;- center(logit.data$Difficulty)^2&#xA;standard &amp;lt;- glm(Score~center(Difficulty),data=logit.data,family=binomial) #standard link function&#xA;standard.2 &amp;lt;- glm(Score~center(Difficulty)+Difficulty2,data=logit.data,family=binomial) #standard link function, but better with a quadradic&#xA;revised.link &amp;lt;- glm(Score~center(Difficulty),data=logit.data,family=binomial(mafc.logit(2)))&#xA;AIC(base)&#xA;AIC(base.2)&#xA;AIC(revised.link)&#xA;coef(base)&#xA;coef(base.2)&#xA;coef(revised.link)&#xA;#plot&#xA;plot(diffs,plogis(coef(standard)[1] +coef(standard)[2]*center(diffs)),xlab=&quot;Difficulty&quot;,ylab=&quot;Pr(Correct)&quot;,ylim=c(0,1),col=&quot;blue&quot;,type=&quot;l&quot;);abline(.5,0,col=&quot;Orange&quot;);lines(diffs,plogis(coef(standard.2)[1]+coef(standard.2)[2]*center(diffs)+coef(standard.2)[3]*center(diffs)^2),col=&quot;Cyan&quot;);lines(diffs,(p2afc(coef(revised.link)[1]+coef(revised.link)[2]*center(diffs))),col=&quot;Green&quot;);lines(5:1,c(.55,.60,.75,.85,.99),col=&quot;Black&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/elxVG.jpg&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the above image the orange horizontal line marks 50% correct responses.  The jagged black line represents the data supplied to the estimation equation (note the values for 4 and 5 disappear behind the orange 50% marker).  The blue line is the equation produced by a standard logistic link.  Note that it estimates below 50% accuracy when discrimination is most difficult (5).  The cyan line is the standard logistic link with a quadratic term.  The green line is a non-standard link that takes into account that the data comes from a 2AFC experiment where performance is very unlikely to fall below 50%.&#xA;Note that the AIC for a model fit using a non-standard link function is superior to the standard logistic link function.  Also note that the slope for the standard equation is less than the slope for the standard equation with the quadratic term (which more accurately reflects the real data).  Thus, using a logistic function blindly on 2AFC data does (at least) appear to underestimate the slope.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a problem with my demonstration that means that I am not seeing what I think I am seeing?  If I'm correct, then what other consequences (if any) are there of using the generic logistic function with 2AFC data [presumably extensible to NAFC cases]?&lt;/p&gt;&#xA;" OwnerUserId="196" LastEditorUserId="196" LastEditorDisplayName="" LastEditDate="2010-11-20T07:29:05.783" LastActivityDate="2010-11-20T07:29:05.783" Title="Consequences of an improper link function in N alternative forced choice procedures (e.g. 2AFC)?" Tags="&lt;logistic-regression&gt;&lt;link-function&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="1" />
  <row Id="1413" PostTypeId="1" AcceptedAnswerId="1416" CreationDate="2010-08-08T18:14:35.537" Score="2" ViewCount="221" Body="&lt;p&gt;It seems like the current revision of lmer does not allow for custom link functions.  &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;If one needs to fit a logistic&#xA;linear mixed effect model with a&#xA;custom link function what options&#xA;are available in R?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If none - what options are available in other&#xA;statistics/programming packages?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Are there conceptual reasons lmer&#xA;does not have custom link functions,&#xA;or are the constraints purely&#xA;pragmatic/programmatic?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="196" LastActivityDate="2010-08-08T19:26:40.020" Title="Mixed regression models and custom link functions in R?" Tags="&lt;r&gt;&lt;regression&gt;&lt;mixed-model&gt;&lt;link-function&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="2" />
  <row Id="1414" PostTypeId="2" ParentId="1412" CreationDate="2010-08-08T18:27:45.137" Score="0" ViewCount="" Body="&lt;p&gt;&lt;em&gt;My question is this: since chance performance is 50% in a 2AFC trial, is it still reasonable to use the standard logistic link function?&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;yes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Think of it this way: suppose you fit a logistic regression where your $y$ variable takes value 1 if subject i has flue, 0 otherwise.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So long as neither $y_i=1$ nor $y_i=0$ are rare events, then flue incidence (i.e. $n^{-1}\sum_{i=1}^ny_i$) is not relevant, it will be absorbed by the intercept of your model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;but in practice participants in a 2AFC should be correct 50% of the time due to chance&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;if this statement is true and all your exogenous variables have been de-meaned, then, &#xA;you can expect your estimated constant to be $logit^{-1}(0.5)\approx0.05$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Best,&lt;/p&gt;&#xA;" OwnerUserId="603" LastEditorUserId="603" LastEditorDisplayName="" LastEditDate="2010-08-08T18:35:16.010" LastActivityDate="2010-08-08T18:35:16.010" CommentCount="7" />
  <row Id="1415" PostTypeId="2" ParentId="973" CreationDate="2010-08-08T19:02:55.487" Score="0" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Prostate&lt;/strong&gt; (gene expression array)&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;k=2 &lt;/li&gt;&#xA;&lt;li&gt;n=48+52  &lt;/li&gt;&#xA;&lt;li&gt;p=6033&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Available via (among other) R package &lt;a href=&quot;http://cran.r-project.org/web/packages/spls/&quot; rel=&quot;nofollow&quot;&gt;spls&lt;/a&gt; &#xA;name of the dataset: prostate&lt;/p&gt;&#xA;&#xA;&lt;p&gt;error rate =  3/102 (see &lt;a href=&quot;http://www.stat.wisc.edu/%7Ekeles/Papers/C_SPLS.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;) also I think there are paper which show 1/102 error rate. I would say this is an easy test case. &lt;/p&gt;&#xA;" OwnerUserId="223" LastEditorUserId="223" LastEditorDisplayName="" LastEditDate="2010-11-09T08:29:57.363" LastActivityDate="2010-11-09T08:29:57.363" CommunityOwnedDate="2010-08-08T19:02:55.487" />
  <row Id="1416" PostTypeId="2" ParentId="1413" CreationDate="2010-08-08T19:26:40.020" Score="2" ViewCount="" Body="&lt;p&gt;Douglas Bates addressed this on the sig-ME list a while back:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://stat.ethz.ch/pipermail/r-sig-mixed-models/2008q4/001507.html&quot; rel=&quot;nofollow&quot;&gt;using glmer with user-defined link function&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I'm not aware of significant changes since, but his recommendation (using a quasi family with specified link and variance) might be of use.  Hopefully this addresses your first and third questions.  I'm not aware of other packages - sorry.&lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-08-08T19:26:40.020" />
  <row Id="1417" PostTypeId="2" ParentId="485" CreationDate="2010-08-08T19:59:34.217" Score="1" ViewCount="" Body="&lt;p&gt;This site from Ecole normal Supérieure de Paris contains a lot of very interesting video &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.diffusion.ens.fr/index.php?res=themes&amp;amp;idtheme=30&quot; rel=&quot;nofollow&quot;&gt;http://www.diffusion.ens.fr/index.php?res=themes&amp;amp;idtheme=30&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I greatly encourage you to visit this site !! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Among other you will find there all video presentation from the conference &quot;Mathematical Foundations of Learning Theory&quot; that held in 2006. &lt;/p&gt;&#xA;" OwnerUserId="223" LastActivityDate="2010-08-08T19:59:34.217" CommunityOwnedDate="2010-08-08T19:59:34.217" CommentCount="0" />
  <row Id="1418" PostTypeId="2" ParentId="1195" CreationDate="2010-08-08T19:59:40.893" Score="4" ViewCount="" Body="&lt;p&gt;The following paper describes a couple of approaches for imputing right censored data in the same domain (i.e. topcoded wage data).  They use a truncated normal distribution and describe a single imputation model assuming homoscedasticity, and a multiple imputation model assuming heterscedasticity.  Also a second paper of interest, where a generalized beta distribution is assumed, might be closer to what you want.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://ideas.repec.org/p/iab/iabdpa/200844.html&quot; rel=&quot;nofollow&quot;&gt;Multiple Imputation Approaches for Right-Censored Wages in the German IAB-Employment Register&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1431352&quot; rel=&quot;nofollow&quot;&gt;Measuring Inequality Using Censored Data: A Multiple Imputation Approach&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-08-08T19:59:40.893" />
  <row Id="1419" PostTypeId="2" ParentId="485" CreationDate="2010-08-08T20:22:58.713" Score="1" ViewCount="" Body="&lt;p&gt;The folks at SLAC put videos of their lecture series online.  Given that their audience is mostly physicists, they tend to be fairly mathematical.  &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www-group.slac.stanford.edu/sluo/lectures/lectureseries.html&quot; rel=&quot;nofollow&quot;&gt;SLUO Lecture Series&lt;/a&gt; (see the &quot;Stat&quot; links)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-08-08T20:22:58.713" CommunityOwnedDate="2010-08-08T20:22:58.713" CommentCount="1" />
  <row Id="1420" PostTypeId="2" ParentId="1202" CreationDate="2010-08-08T20:38:09.240" Score="1" ViewCount="" Body="&lt;p&gt;I think relative distribution methods are a good candidate for the question you pose.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since you're comparing data based on binning, this is very similar to the method of constructing a probability-probability plot.  Taking it a step further, you can can actually construct a relative CDF/PDF for two distributions based on empirical data.  From there, you can apply graphical and statistical techniques to explore the relative differences and perform inference on the relative distribution.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Handcock and Morris have an &lt;a href=&quot;http://csde.washington.edu/~handcock/RelDist/&quot; rel=&quot;nofollow&quot;&gt;interesting book&lt;/a&gt; devoted to this topic and there's the &lt;a href=&quot;http://cran.r-project.org/web/packages/reldist/index.html&quot; rel=&quot;nofollow&quot;&gt;reldist package&lt;/a&gt; in R available for applying these methods.  The following might be worth skimming to see if this is of any interest:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.jstor.org/pss/270964&quot; rel=&quot;nofollow&quot;&gt;Relative Distribution Methods&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.csss.washington.edu/Papers/wp27.pdf&quot; rel=&quot;nofollow&quot;&gt;Applying Relative Distribution Methods in R&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-08-08T20:38:09.240" />
  <row Id="1421" PostTypeId="2" ParentId="1412" CreationDate="2010-08-08T21:57:57.290" Score="1" ViewCount="" Body="&lt;p&gt;I don't see how the question in your example is sensible.  The slope of the values is the slope of the values. Using a logistic link function then you get the slope of the logit of the values.  There's no under or overestimating.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The more interesting case in your (our) field is that of interactions in accuracy.  You might want to read &lt;a href=&quot;http://www.sciencedirect.com/science?_ob=ArticleURL&amp;amp;_udi=B6WK4-4RH8SFS-1&amp;amp;_user=10&amp;amp;_coverDate=11%2F30%2F2008&amp;amp;_rdoc=1&amp;amp;_fmt=high&amp;amp;_orig=search&amp;amp;_sort=d&amp;amp;_docanchor=&amp;amp;view=c&amp;amp;_searchStrId=1424726112&amp;amp;_rerunOrigin=google&amp;amp;_acct=C000050221&amp;amp;_version=1&amp;amp;_urlVersion=0&amp;amp;_userid=10&amp;amp;md5=47db7bad266e5fff07e6fc8c0ceeac42&quot; rel=&quot;nofollow&quot;&gt;Dixon (2008)&lt;/a&gt; as one of the more recent papers on this problem.  It also addresses many of your fundamental concerns.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, in cognitive and perceptual psychology a logit link function is better than any other standard link.  If you want to know the true effects of your independent variables, (i.e. whether they interact or are additive, whether they are linear or curvilinear) then you would need to know better the true underlying model.  Since you probably don't know that logistic regression is probably better than almost anything else and vastly better than just analyzing meaned accuracy scores.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The primary consequence of doing this is contradicting other findings where mean accuracy scores were put into an ANOVA or regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;* EDIT*&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now that you've added some data it looks like you're trying to model a floor effect that you shouldn't be.  At some point the task becomes impossible.  It looks like that already happened at your level 4 difficulty.  Modelling level 5 is useless.  What if you had a level 6 or 7 difficulty?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It looks like a logistic will fit points 1-4 pretty well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And, you should be looking at residuals to assess fit, not just the curves overlaid.&lt;/p&gt;&#xA;" OwnerUserId="601" LastEditorUserId="601" LastEditorDisplayName="" LastEditDate="2010-08-12T07:32:48.517" LastActivityDate="2010-08-12T07:32:48.517" CommentCount="10" />
  <row Id="1423" PostTypeId="2" ParentId="726" CreationDate="2010-08-08T23:05:17.700" Score="19" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Prediction is very difficult, especially about the future.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;-- Niels Bohr&lt;/p&gt;&#xA;" OwnerUserId="749" LastActivityDate="2010-08-08T23:05:17.700" CommunityOwnedDate="2010-08-08T23:05:17.700" CommentCount="1" />
  <row Id="1424" PostTypeId="1" AcceptedAnswerId="1425" CreationDate="2010-08-09T00:00:13.560" Score="2" ViewCount="315" Body="&lt;p&gt;This definitely sounds like a homework, but I assure you that it's not. You're probably familiar with the &lt;a href=&quot;http://en.wikipedia.org/wiki/Risk_%28game%29&quot; rel=&quot;nofollow&quot;&gt;Risk&lt;/a&gt; game. Now, friend of mine rolled 3 aces in one hand. I reckon that probability of such event is C(n,k) = (n-1+k)!/[(n-1)!k!], so that's 8!/(5!*3!) = 56, so the prob. is 1/56. Am I correct?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Problem starts here: he rolled 3 aces in his 2nd attack, so he asked me: now, tell me 'bout the odds, you do statistics! And I must admit that I'm still stunned by his tremendous attacks (he lost 3 tanks both times).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is the probability of such two consecutive events 1/56&lt;sup&gt;2&lt;/sup&gt;?&lt;/p&gt;&#xA;" OwnerUserId="1356" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-29T00:29:08.950" LastActivityDate="2011-04-29T00:29:08.950" Title="The &quot;Risk&quot; game dice problem" Tags="&lt;probability&gt;&lt;games&gt;&lt;combination&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="1425" PostTypeId="2" ParentId="1424" CreationDate="2010-08-09T00:37:45.090" Score="7" ViewCount="" Body="&lt;p&gt;You should not do a calculation of probability for an event deemed surprising &lt;em&gt;post hoc&lt;/em&gt; as if it were an event specified before it was rolled (observed).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's very difficult to to do a proper calculation of &lt;em&gt;post hoc&lt;/em&gt; probability, because what &lt;em&gt;other&lt;/em&gt; events would have been deemed at least as surprising depends on what the context is. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would three ones twice in a row at an earlier or later stage of the game have been as surprising? Would &lt;em&gt;you&lt;/em&gt; rolling three ones have been as surprising as him rolling them? Would three sixes be as surprising as three ones? and so on... What is the totality of all the events would have been surprising enough to generate a post like this one?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Even if it were legitimate to do the calculation &lt;em&gt;as if it were a pre-specified event&lt;/em&gt;, it looks like you have that calculation incorrect. Specifically, the probability (&lt;em&gt;for an event specified before the roll&lt;/em&gt;) of taking three dice and rolling (1,1,1) is (1/6)^3 = 1/216, because the three rolls are independent, not 1/56, and the probability of doing it twice &lt;em&gt;out of a total of two rolls&lt;/em&gt; is the square of that - but neither the condition of being pre-specified nor the &quot;out of two rolls&quot; hold)&lt;/p&gt;&#xA;" OwnerUserId="805" LastEditorUserId="805" LastEditorDisplayName="" LastEditDate="2010-08-10T01:46:18.860" LastActivityDate="2010-08-10T01:46:18.860" CommentCount="3" />
  <row Id="1426" PostTypeId="2" ParentId="726" CreationDate="2010-08-09T01:44:41.210" Score="7" ViewCount="" Body="&lt;p&gt;&quot;It's easy to lie with statistics; it is easier to lie without them.&quot; -- Frederick Mosteller&lt;/p&gt;&#xA;" OwnerUserId="319" LastActivityDate="2010-08-09T01:44:41.210" CommunityOwnedDate="2010-08-09T01:44:41.210" />
  <row Id="1427" PostTypeId="1" AcceptedAnswerId="1428" CreationDate="2010-08-09T03:57:50.097" Score="2" ViewCount="205" Body="&lt;p&gt;Hello all,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm exploring the use of changepoint detection or other methods (am slowly becoming aware of wavelet transformation, etc. but have tons to learn in this area) to identify key shifts in health care performance patterns over time. However, many of the metrics I'm seeking to analyze (e.g., health care quality metrics) are both generally calculated and more reasonably interpreted as rolling-12 month aggregates. For example, it's important to me to track on a monthly basis the proportion of patients who are up-to-date on a certain lab test within the 12-month period ending that month, but I'm not particularly concerned with how many of these tests occurred specifically in August. So it's something sort of like having a moving average to work with as a raw starting point.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That said, there are also reasons why this rolling-12 aggregation does not result in a stationary process either.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My thought was to account for the data structure and seasonality by modeling it as a function of a 1-month and 12-month lag. Is this the proper way to think about this data? Is there anything else or a better approach I should be doing/considering? Again, my general goal is surveillance of the general trend as well as breaks -- so if it affects the answer, I'm looking at this in the context of using the R strucchange package, CUSUM statistics, or some other approach to identify good and bad anomalies.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Shelby&lt;/p&gt;&#xA;" OwnerUserId="394" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-04-14T14:58:25.980" LastActivityDate="2011-04-14T14:58:25.980" Title="Time-series data pre-aggregated into non-stationary rolling 12-month periods: are there special considerations for modeling?" Tags="&lt;time-series&gt;&lt;modeling&gt;&lt;change-point&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="1428" PostTypeId="2" ParentId="1427" CreationDate="2010-08-09T04:05:27.520" Score="4" ViewCount="" Body="&lt;p&gt;The 12-month rolling aggregation will remove seasonality which makes the task easier. For non-seasonal time series, the methods in the &lt;a href=&quot;http://cran.r-project.org/package=strucchange&quot; rel=&quot;nofollow&quot;&gt;strucchange&lt;/a&gt; package for R are excellent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For seasonal time series, you might look at the BFAST (Breaks For Additive Seasonal and Trend) method which is implemented in the &lt;a href=&quot;http://cran.r-project.org/package=bfast&quot; rel=&quot;nofollow&quot;&gt;bfast&lt;/a&gt; package for R. This method involves applying strucchange to the trend and seasonal components obtained from a decomposition of the data (applied iteratively to allow for the breaks discovered). You could apply bfast on the original data (without the 12-month aggregation).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Neither of these methods requires stationarity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would think that a direct modelling approach such as the one you propose would be less capable of finding general breaks due to the additional assumptions being made.&lt;/p&gt;&#xA;" OwnerUserId="159" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-08-09T04:13:46.500" LastActivityDate="2010-08-09T04:13:46.500" CommentCount="3" />
  <row Id="1429" PostTypeId="2" ParentId="1268" CreationDate="2010-08-09T04:22:42.533" Score="1" ViewCount="" Body="&lt;p&gt;I'm somewhat confused by your example code, as it seems you drop the &lt;code&gt;V&lt;/code&gt; variable from the computation of &lt;code&gt;newX&lt;/code&gt;. Are you looking to model &lt;code&gt;X&lt;/code&gt; as a reduced rank product, or are you interested in a reduced column space of &lt;code&gt;X&lt;/code&gt;? in the latter case, I think an EM-PCA approach would work. you can find matlab code under the title &lt;a href=&quot;http://lear.inrialpes.fr/~verbeek/software.php&quot; rel=&quot;nofollow&quot;&gt;Probabilistic PCA with missing values&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;hth,&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-08-09T04:22:42.533" CommentCount="1" />
  <row Id="1430" PostTypeId="1" AcceptedAnswerId="1431" CreationDate="2010-08-09T06:50:42.803" Score="4" ViewCount="424" Body="&lt;p&gt;Does anybody have a nice example of a stochastic process that is 2nd-order stationary, but is not strictly stationary?&lt;/p&gt;&#xA;" OwnerUserId="352" LastEditorUserId="223" LastEditorDisplayName="" LastEditDate="2010-08-09T06:57:37.530" LastActivityDate="2010-08-09T08:29:24.173" Title="Example of a 2nd order stationary, but not strictly stationary process" Tags="&lt;random-variable&gt;&lt;stochastic-processes&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="1431" PostTypeId="2" ParentId="1430" CreationDate="2010-08-09T07:05:20.680" Score="3" ViewCount="" Body="&lt;p&gt;Take any process $(X_t)_t$ with independent components that has a constant first and second moment and put a varying third moment. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is second order stationnary because $E[ X_t X_{t+h} ]=0$ and it is not strictly stationnary &#xA;because $P( X_t \geq x_t, X_{t+1} \geq  x_{t+1})$ depends upon $t$&lt;/p&gt;&#xA;" OwnerUserId="223" LastEditorUserId="223" LastEditorDisplayName="" LastEditDate="2010-08-09T08:29:24.173" LastActivityDate="2010-08-09T08:29:24.173" CommentCount="6" />
  <row Id="1432" PostTypeId="1" AcceptedAnswerId="1435" CreationDate="2010-08-09T07:32:32.767" Score="8" ViewCount="991" Body="&lt;p&gt;In answering &lt;a href=&quot;http://stats.stackexchange.com/questions/1412/consequences-of-an-improper-link-function-in-n-alternative-forced-choice-procedur&quot;&gt;this&lt;/a&gt; question John Christie suggested that the fit of logistic regression models should be assessed by evaluating the residuals.  I'm familiar with how to interpret residuals in OLS, they are in the same scale as the DV and very clearly the difference between y and the y predicted by the model.  However for logistic regression, in the past I've typically just examined estimates of model fit, e.g. AIC, because I wasn't sure what a residual would mean for a logistic regression.  After looking into R's help files a little bit I see that in R there are five types of glm residuals available, c(&quot;deviance&quot;, &quot;pearson&quot;, &quot;working&quot;,&quot;response&quot;, &quot;partial&quot;).  The help file refers to Davison, A. C. and Snell, E. J. (1991) Residuals and diagnostics. In: Statistical Theory and Modelling. In Honour of Sir David Cox, FRS, eds. Hinkley, D. V., Reid, N. and Snell, E. J., Chapman &amp;amp; Hall, of which I do not have a copy.  Is there a short way to describe how to interpret each of these types?  In a logistic context will sum of squared residuals provide a meaningful measure of model fit or is one better off with an Information Criterion?&lt;/p&gt;&#xA;" OwnerUserId="196" LastActivityDate="2010-11-01T14:03:54.977" Title="What do the residuals in a logistic regression mean?" Tags="&lt;r&gt;&lt;logistic-regression&gt;&lt;aic&gt;&lt;error&gt;" AnswerCount="4" CommentCount="3" FavoriteCount="1" />
  <row Id="1433" PostTypeId="2" ParentId="6" CreationDate="2010-08-09T10:12:35.817" Score="6" ViewCount="" Body="&lt;p&gt;What enforces more separation than there should be is each discipline's lexicon. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are many instances where ML uses one term and Statistics uses a different term--but both refer to the same thing--fine, you would expect that, and it doesn't cause any permanent confusion (e.g., features/attributes versus expectation variables, or neural network/MLP versus project-pursuit).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What's more troublesome, is the number of key terms in ML that are also used in Statistics, though in Statistics the meaning is completely different (homonyms, i suppose).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A few examples:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;&lt;strong&gt;Kernel Function&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In ML, kernel functions are used in classifiers (e.g., SVM) and of course in kernel machines. The term refers to a simple function (&lt;em&gt;cosine, sigmoidal, rbf, polynomial&lt;/em&gt;) to map data not linearly separable to a new input space, then applying a linear model in that new space (instead of using a non-linear model to begin with).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In statistics, a kernel function is weighting function used in density estimation to smooth the density curve.&lt;br&gt;&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;&lt;strong&gt;Regression&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In ML, predictive algorithms are run in either of two modes, 'machine' mode or 'regression' mode. 'Machine' refers to an algorithm that returns a class label--a classifier; 'Regression' mode returns continuous output--e.g., price or expected value. (Rarely do the algorithms have different names based on mode, though i sometimes see terms like 'SVR' and 'SVM' used in the same paper, to indicate/distinguish a support vector algorithm configured to return a class label versus one returning a continuous variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In Statistics, 'regression', AFAIK, if you are attempting to model the relationship between two more variables, you are doing regression analysis. It doesn't matter whether the output is a continuous variable or a class label (e.g., logistic regression).&lt;br&gt;&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;&lt;strong&gt;Bias&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In ML, the &lt;em&gt;bias&lt;/em&gt; term in the algorithm is conceptually identical to the &lt;em&gt;intercept&lt;/em&gt; term used by statisticians in regression modeling.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In Statistics,...well, i'm not sure but it appears to be used as a synonym for 'error' in some contexts (e.g., estimator error is referred to always as 'bias' rather than 'error.')&lt;/p&gt;&#xA;" OwnerUserId="438" LastActivityDate="2010-08-09T10:12:35.817" CommentCount="1" />
  <row Id="1434" PostTypeId="2" ParentId="421" CreationDate="2010-08-09T10:23:41.673" Score="1" ViewCount="" Body="&lt;p&gt;&quot;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0824718178&quot; rel=&quot;nofollow&quot;&gt;How to Tell the Liars from the Statisticians&lt;/a&gt;&quot; by Hooke. I am fond of its way of explaining the concepts of statistics to laypersons.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for explaining the motivations of statisticians, &quot;The Lady Tasting Tea&quot; is good reading.&lt;/p&gt;&#xA;" OwnerUserId="830" LastEditorUserId="509" LastEditorDisplayName="" LastEditDate="2010-08-11T08:37:11.667" LastActivityDate="2010-08-11T08:37:11.667" CommunityOwnedDate="2010-08-09T10:23:41.673" />
  <row Id="1435" PostTypeId="2" ParentId="1432" CreationDate="2010-08-09T10:26:35.913" Score="8" ViewCount="" Body="&lt;p&gt;The easiest residuals to understand are the deviance residuals as when squared these sum to -2 times the log-likelihood. In its simplest terms logistic regression can be understood in terms of fitting the function p = inv-logit(X*beta) for known X in such a way as to minimise the total deviance, which is the sum of squared deviance residuals of all the data points.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The (squared) deviance of each data point is equal to (-2 times) the logarithm of the difference between its predicted probability (inv-logit(X*beta))  and the complement of its actual value (1 for a control; a 0 for a case) in absolute terms. A perfect fit of a point (which never occurs) gives a deviance of zero as log(1) is zero. A poorly fitting point has a large residual deviance as -2 times the log of a very small value is a large number. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Doing logistic regression is akin to finding a beta value such that the sum of squared deviance residuals is minimised. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This can be illustrated with a plot, but I don't know how to upload one.&lt;/p&gt;&#xA;" OwnerUserId="521" LastEditorUserId="521" LastEditorDisplayName="" LastEditDate="2010-08-10T01:02:43.707" LastActivityDate="2010-08-10T01:02:43.707" CommentCount="2" />
  <row Id="1436" PostTypeId="2" ParentId="1337" CreationDate="2010-08-09T10:29:38.373" Score="4" ViewCount="" Body="&lt;p&gt;A mathematician, a physicist and a statistician went hunting for deer. When they chanced upon one buck lounging about, the mathematician fired first, missing the buck's nose by a few inches. The physicist then tried his hand, and missed the tail by a wee bit. The statistician started jumping up and down saying &quot;We got him! We got him!&quot;&lt;/p&gt;&#xA;" OwnerUserId="830" LastActivityDate="2010-08-09T10:29:38.373" CommunityOwnedDate="2010-08-09T10:29:38.373" />
  <row Id="1437" PostTypeId="1" CreationDate="2010-08-09T10:37:59.673" Score="1" ViewCount="173" Body="&lt;p&gt;Do you think it can be used instead of k means? I obtained a correlation with the first 2 components as they carry over 90% of the weight. Would you agree on the technique?&lt;/p&gt;&#xA;" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-08-09T10:59:13.217" LastActivityDate="2010-08-09T12:45:05.153" Title="Can Principal Component Analysis be used alone to infer major patterns within data instead of k means clustering?" Tags="&lt;pca&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="1438" PostTypeId="2" ParentId="1437" CreationDate="2010-08-09T10:47:18.197" Score="1" ViewCount="" Body="&lt;p&gt;I think it depends on your data set and what you want to do with it. If you look at my answer to this &lt;a href=&quot;http://stats.stackexchange.com/questions/1289/visualizing-multiple-histograms/1291#1291&quot;&gt;question&lt;/a&gt;, you will see that it indicates groups/differences. However, it certainly doesn't &lt;em&gt;prove&lt;/em&gt; differences - it just gives you an idea of where differences may lie.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How long would it take you run a quick k-means analysis on your data? When I have a large multivariate data set, I try many different techniques to get a handle on it.&lt;/p&gt;&#xA;" OwnerUserId="8" LastActivityDate="2010-08-09T10:47:18.197" CommentCount="1" />
  <row Id="1439" PostTypeId="2" ParentId="103" CreationDate="2010-08-09T11:14:34.503" Score="1" ViewCount="" Body="&lt;p&gt;I can't pick just one :)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Check out this great blog post by flowingdata:&lt;/strong&gt; &lt;a href=&quot;http://flowingdata.com/2009/05/06/37-data-ish-blogs-you-should-know-about/&quot; rel=&quot;nofollow&quot;&gt;37 Data-ish blogs you should know about&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="665" LastActivityDate="2010-08-09T11:14:34.503" CommunityOwnedDate="2010-08-09T11:14:34.503" />
  <row Id="1440" PostTypeId="2" ParentId="1437" CreationDate="2010-08-09T12:45:05.153" Score="0" ViewCount="" Body="&lt;p&gt;It's possible that my background in psychological research is disguising some understanding of the broader application of PCA and K-means, but I'd say the following:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;PCA&lt;/strong&gt; is used to reduce a set of variables to a smaller number of dimensions&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;k-means&lt;/strong&gt; is used to group cases.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;For &lt;strong&gt;example&lt;/strong&gt;, take a study of 1000 participants who have completed 10 different ability tests (verbal, mathematics, spatial, etc.).&#xA;I could use PCA (or factor analysis) to group tests in order to identify the main underlying dimensions.&#xA;I could use k-means to identify types of cases. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a side point, I often find one approach is &lt;a href=&quot;http://jeromyanglim.blogspot.com/2009/09/cluster-analysis-and-single-dominant.html&quot; rel=&quot;nofollow&quot;&gt;theoretically more interesting&lt;/a&gt;. If the cluster analysis is just grouping in terms of high and low on the first principal component, then I find PCA to be the more meaningful analysis, and the whole concept of clusters as an arbitrary dichotomisation (or categorisation) of a continuous variable.&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-08-09T12:45:05.153" />
  <row Id="1441" PostTypeId="1" CreationDate="2010-08-09T12:50:55.720" Score="3" ViewCount="224" Body="&lt;p&gt;This question follows from &lt;a href=&quot;http://stats.stackexchange.com/questions/1430/example-of-a-2nd-order-stationary-but-not-strictly-stationary-process&quot;&gt;my previous question&lt;/a&gt; , where Robin answered the question in the case of weak stationary processes.  Here, I am asking a similar question for (strong?) stationary processes.  I'll define what this means (the definition can also be found &lt;a href=&quot;http://en.wikipedia.org/wiki/Stationary_process&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let $X(t)$ be a stochastic process. We say that $X(t)$ is Nth-order stationary if, for every $t_1, t_2, \dots, t_N$ we have that the joint cumulative density functions&#xA;$$F_{X(t_1),X(t_2),\dots,X(t_N)} = F_{X(t_1 + \tau),X(t_2 + \tau),\dots,X(t_N + \tau)}$$&#xA;for all $\tau$. This is quite a strong condition, it says that the joint statistics don't change at all as time shifts. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, a 1st order stationary process is such that $F_{X(t_1)} = F_{X(t_2)}$ for all $t_1$ and $t_2$. That is, the $X(t)$ are all identically distributed. It is quite easy to see that a 1st order stationary process need not be 2nd order stationary.  Simply assign a correlation structure to say $X(t)$, $X(t+1)$, $X(t+2)$ that &lt;em&gt;does not&lt;/em&gt; correspond to a (symmetric) Toeplitz matrix. That is, in vector form, the covariance matrix of $[ X(t), X(t+1), X(t+3)]$ could be given as&#xA;$$\left[\begin{array}{cc} &#xA;\sigma^2 &amp;amp; a &amp;amp; b \newline &#xA;a &amp;amp; \sigma^2 &amp;amp; c \newline&#xA;b &amp;amp; c&amp;amp; \sigma^2&#xA;\end{array}\right]$$&#xA;for $a,b,c$ distinct.  This is now not 2nd order stationary because $E[X(t)X(t+1)] = a$ and, time shifting by 1 we have $E[X(t+1)X(t+2)] = c \neq a$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a similar way (presumably) a process that is 1st and 2nd order stationary need not be 3rd order stationary and this leads to my question:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Does somebody have a nice example of a stochastic process that is both 1st and 2nd order stationary, but not 3rd order stationary?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="352" LastActivityDate="2010-08-09T12:50:55.720" ClosedDate="2010-10-14T09:00:42.667" Title="Example of a stochastic process that is 1st and 2nd order stationary, but not strictly stationary (Round 2)" Tags="&lt;stochastic-processes&gt;" CommentCount="2" FavoriteCount="1" />
  <row Id="1442" PostTypeId="2" ParentId="173" CreationDate="2010-08-09T13:36:43.330" Score="7" ViewCount="" Body="&lt;p&gt;To assess the historical trend, I'd use a gam with trend and seasonal components. For example&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(mgcv)&#xA;require(forecast)&#xA;x &amp;lt;- ts(rpois(100,1+sin(seq(0,3*pi,l=100))),f=12)&#xA;tt &amp;lt;- 1:100&#xA;season &amp;lt;- seasonaldummy(x)&#xA;fit &amp;lt;- gam(x ~ s(tt,k=5) + season, family=&quot;poisson&quot;)&#xA;plot(fit)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then &lt;code&gt;summary(fit)&lt;/code&gt; will give you a test of significance of the change in trend and the plot will give you some confidence intervals. The assumptions here are that the observations are independent and the conditional distribution is Poisson. Because the mean is allowed to change smoothly over time, these are not particularly strong assumptions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To forecast is more difficult as you need to project the trend into the future. If you are willing to accept a linear extrapolation of the trend at the end of the data (which is certainly dodgy but probably ok for a few months), then use&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;fcast &amp;lt;- predict(fit,se.fit=TRUE,&#xA;               newdata=list(tt=101:112,season=seasonaldummyf(x,h=12)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To see the forecasts on the same graph:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plot(x,xlim=c(0,10.5))&#xA;lines(ts(exp(fcast$fit),f=12,s=112/12),col=2)&#xA;lines(ts(exp(fcast$fit-2*fcast$se),f=12,s=112/12),col=2,lty=2)&#xA;lines(ts(exp(fcast$fit+2*fcast$se),f=12,s=112/12),col=2,lty=2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can spot the unusual months by looking for outliers in the (deviance) residuals of the fit.&lt;/p&gt;&#xA;" OwnerUserId="159" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-08-13T13:01:56.437" LastActivityDate="2010-08-13T13:01:56.437" />
  <row Id="1443" PostTypeId="2" ParentId="6" CreationDate="2010-08-09T13:51:29.007" Score="1" ViewCount="" Body="&lt;p&gt;Ideally one should have a thorough knowledge of both statsitics and machine learning before attempting to answer his question. I am very much a neophyte to ML, so forgive me if wat I say is naive.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have limited experience in SVMs and regression trees. What strikes me as lacking in ML from a stats point of view is a well developed concept of inference. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Inference in ML seems to boil down almost exclusively to the predictice accuracy, as measured by (for example) mean classification error (MCE), or balanced error rate (BER) or similar. ML is in the very good habit of dividing data randomly (usually 2:1) into a training set and a test set. Models are fit using the training set and performance (MCE, BER etc) is assessed using the test set. This is an excellent practice and is only slowly making its way into mainstream statistics. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;ML also makes heavy use of resampling methods (especially cross-validation), whose origins appear to be in statistics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, ML seems to lack a fully developed concept of inference - beyond predictive accuracy. This has two results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) There does not seem to be an appreciation that any prediction (parameter estimation etc.) is subject to a random error and perhaps systemmatics error (bias). Statisticians will accept that this is an inevitable part of prediction and will try and estimate the error. Statistical techniques will try and find an estimate that has minimum bias and random error. Their techniques are usually driven by a model of the data process, but not always (eg. Bootstrap).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) There does not seem to be a deep understanding in ML of the limits of applying a model to new data to a new sample from the same population (in spite of what I said earlier about the training-test data set approach). Various statistical techniques, among them cross validation and penalty terms applied to likelihood-based methods, guide statisticians in the trade-off between parsimony and model complexity. Such guidelines in ML seem much more ad hoc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've seen several papers in ML where cross validation is used to optimise a fitting of many models on a training dataset - producing better and better fit as the model complexity increases. There appears little appreciation that the tiny gains in accuracy are not worth the extra complexity and this naturally leads to over-fitting. Then all these optimised models are applied to the test set as a check on predictive performance and to prevent overfitting. Two things have been forgotten (above). The predictive performance will have a stochastic component. Secondly multiple tests against a test set will again result in over-fitting. The &quot;best&quot; model will be choisen by the ML practitioner without a full appreciation he/she has cherry picked from one realisation of many possible outomes of this experiment. The best of several tested models will almost certainly not reflect the true performance on new data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any my 2 cents worth. We have much to learn from each other.&lt;/p&gt;&#xA;" OwnerUserId="521" LastEditorUserId="521" LastEditorDisplayName="" LastEditDate="2010-08-09T13:56:39.603" LastActivityDate="2010-08-09T13:56:39.603" CommunityOwnedDate="2010-08-09T13:51:29.007" CommentCount="1" />
  <row Id="1444" PostTypeId="1" AcceptedAnswerId="1446" CreationDate="2010-08-09T13:57:51.753" Score="15" ViewCount="1104" Body="&lt;p&gt;If I have highly skewed positive data I often take logs. But what should I do with highly skewed non-negative data that include zeros? I have seen two transformations used:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;log(x+1) which has the neat feature that 0 maps to 0.&lt;/li&gt;&#xA;&lt;li&gt;log(x+c) where c is either estimated or set to be some very small positive value.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Are there any other approaches? Are there any good reasons to prefer one approach over the others?&lt;/p&gt;&#xA;" OwnerUserId="159" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-29T00:20:08.787" LastActivityDate="2011-04-29T00:20:08.787" Title="How should I transform non-negative data including zeros?" Tags="&lt;data-transformation&gt;&lt;large-data&gt;" AnswerCount="7" CommentCount="2" FavoriteCount="3" />
  <row Id="1445" PostTypeId="2" ParentId="1444" CreationDate="2010-08-09T14:05:50.187" Score="2" ViewCount="" Body="&lt;p&gt;I assume you have continuous data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the data include &lt;strong&gt;zeroS&lt;/strong&gt; this means you have a spike on zero which may be due to some particular aspect of your data. It appears for example in wind energy, wind below 2m/s produce zero power (it is called cut in) and wind over (something around) 25m/5 also produce zero power (for security reason, it is called cut off).  While the distribution of produced wind energy seems continuous there is a spike in zero. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My solution:&lt;/strong&gt;  In this case, I suggest to treat the zeros separatly by working with a mixture of the spike in zero and the model you planned to use for the part of the distribution that is continuous (wrt lebesgue). &lt;/p&gt;&#xA;" OwnerUserId="223" LastActivityDate="2010-08-09T14:05:50.187" />
  <row Id="1446" PostTypeId="2" ParentId="1444" CreationDate="2010-08-09T14:22:11.460" Score="10" ViewCount="" Body="&lt;p&gt;It seems to me that the most appropriate choice of transformation is contingent on the model and the context. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The '0' point can arise from several different reasons each of which may have to be treated differently:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Truncation (as in Robin's example): Use appropriate models (e.g., mixtures, survival models etc)&lt;/li&gt;&#xA;&lt;li&gt;Missing data: Impute data / Drop observations if appropriate.&lt;/li&gt;&#xA;&lt;li&gt;Natural zero point (e.g., income levels; an unemployed person has zero income): Transform as needed&lt;/li&gt;&#xA;&lt;li&gt;Sensitivity of measuring instrument: Perhaps, add a small amount to data?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I am not really offering an answer as I suspect there is no universal, 'correct' transformation when you have zeros.&lt;/p&gt;&#xA;" LastActivityDate="2010-08-09T14:22:11.460" CommentCount="1" />
  <row Id="1447" PostTypeId="1" CreationDate="2010-08-09T14:52:42.430" Score="8" ViewCount="739" Body="&lt;p&gt;I want to fully grasp the notion of r^2 describing the amount of variation b/w variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Every web explanation is a bit mechanical/obtuse.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to &quot;get&quot; the concept, not just mechanically use the numbers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;eg: Hours studied vs. test score&lt;/p&gt;&#xA;&#xA;&lt;p&gt;r = .8&lt;/p&gt;&#xA;&#xA;&lt;p&gt;r^2 = .64&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, what does this mean?&#xA;64% of the variability of test scores can be explained by hours?&#xA;How do we know that just by squaring?   Thanks for any clarification.&lt;/p&gt;&#xA;" OwnerUserId="832" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-29T00:20:40.423" LastActivityDate="2011-04-29T00:20:40.423" Title="r-squared: I have never fully grasped the interpretation" Tags="&lt;regression&gt;" AnswerCount="4" CommentCount="1" FavoriteCount="7" />
  <row Id="1448" PostTypeId="2" ParentId="1447" CreationDate="2010-08-09T15:09:34.683" Score="0" ViewCount="" Body="&lt;p&gt;A mathematical demonstration of the relationship between the two is here: &lt;a href=&quot;http://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient#Pearson.27s_correlation_and_least_squares_regression_analysis&quot; rel=&quot;nofollow&quot;&gt;Pearson's correlation and least squares regression analysis&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strike&gt;I am not sure if there is a geometric or any other intuition that can be offered apart from the math but if I can think of one I will update this answer.&lt;/strike&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update: Geometric Intuition&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a geometric intuition I came up with. Suppose that you have two variables $x$ and $y$ which are mean centered. (Assuming mean centered lets us ignore the intercept which simplifies the geometrical intuition a bit.) Let us first consider the geometry of linear regression. In linear regression, we model $y$ as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$y = x\  \beta + \epsilon$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider the situation when we have two observations from the above data generating process given by the pairs ($y_1,y_2$) and ($x_1,x_2$). We can view them as vectors in two-dimensional space as shown in the figure below:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://a.imageshack.us/img202/669/linearregression1.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, in terms of the above geometry, our goal is to find a $\beta$ such that the vector $x\ \beta$ is the closest possible to the vector $y$. Note that different choices of $\beta$ scale $x$ appropriately. Let $\hat{\beta}$ be the value of $\beta$ that is our best possible approximation of $y$ and denote $\hat{y} = x\ \hat{\beta}$. Thus,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$y = \hat{y} + \hat{\epsilon}$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;From a geometrical perspective we have three vectors. $y$, $\hat{y}$ and $\hat{\epsilon}$. A little thought suggests that we must choose $\hat{\beta}$ such that three vectors look like the one below:&#xA;&lt;img src=&quot;http://a.imageshack.us/img19/9524/intuitionlinearregressi.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words, we need to choose $\beta$ such that the angle between $x\ \beta$ and $\hat{\epsilon}$ is 90&lt;sup&gt;0&lt;/sup&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, how much variation in $y$ have we explained with this projection of $y$ onto the vector $x$. Since the data is mean centered the variance in $y$ is equals ($y_1^2+y_2^2$) which is the square of the distance between the point represented by the point $y$ and the origin. The variation in $\hat{y}$ is similarly the distance from the point $\hat{y}$ and the origin and so on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By the Pythagorean theorem, we have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$y^2 = \hat{y}^2 + \hat{\epsilon}^2$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, the proportion of the variance explained by $x$ is $\frac{\hat{y}^2}{y^2}$. Notice also that $cos(\theta) = \frac{\hat{y}}{y}$.  and the wiki tells us that the &lt;a href=&quot;http://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient#Geometric_interpretation&quot; rel=&quot;nofollow&quot;&gt;geometrical interpretation of correlation&lt;/a&gt; is that correlation equals the cosine of the angle between the mean-centered vectors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, we have the required relationship:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Correlation)&lt;sup&gt;2&lt;/sup&gt; = Proportion of variation in $y$ explained by $x$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope that helps.&lt;/p&gt;&#xA;" LastActivityDate="2010-08-09T19:58:51.157" CommentCount="2" />
  <row Id="1449" PostTypeId="2" ParentId="726" CreationDate="2010-08-09T15:36:26.553" Score="4" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;My greatest concern was what to call&#xA;  it. I thought of calling it&#xA;  'information,' but the word was overly&#xA;  used, so I decided to call it&#xA;  'uncertainty.' When I discussed it&#xA;  with John von Neumann, he had a better&#xA;  idea. Von Neumann told me, 'You should&#xA;  call it entropy, for two reasons. In&#xA;  the first place your uncertainty&#xA;  function has been used in statistical&#xA;  mechanics under that name, so it&#xA;  already has a name. In the second&#xA;  place, and more important, no one&#xA;  really knows what entropy really is,&#xA;  so in a debate you will always have&#xA;  the advantage.'&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Claude Elwood Shannon&lt;/p&gt;&#xA;" OwnerUserId="223" LastActivityDate="2010-08-09T15:36:26.553" CommunityOwnedDate="2010-08-09T15:36:26.553" />
  <row Id="1450" PostTypeId="2" ParentId="1447" CreationDate="2010-08-09T15:44:50.163" Score="8" ViewCount="" Body="&lt;p&gt;Start with the basic idea of variation.  Your beginning model is the sum of the squared deviations from the mean.  The R^2 value is the proportion of that variation that is accounted for by using an alternative model.  For example, R-squared tells you how much of the variation in Y you can get rid of by summing up the squared distances from a regression line, rather than the mean.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think this is made perfectly clear if we consider that regression problem and imagine that plotted out.  Imagine a typical scatterplot where you have a predictor X along the horizontal axis and a response Y along the vertical axis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The mean is a horizontal line on the plot where Y is constant.  The Y variation is the sum of squared differences between the mean of Y and each individual data point.  It's the distance between the mean line and every individual point squared and added up.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also calculate another measure of variability after you have the regression line from the model.  This is the difference between each Y point and the regression line.  Rather than each (Y - the mean) squared we get (Y - the point on the regression line) squared.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the regression line is anything but horizontal, we're going to get less total distance when we use that instead of the mean--that is there is less unexplained variation.  The ratio between the extra variation explained and the original variation is your R^2.  It's the proportion of the original variation in your response that is explained by fitting that regression line.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See the (quick and dirty) image below...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://img217.imageshack.us/img217/5536/rsquared.png&quot; alt=&quot;http://img217.imageshack.us/img217/5536/rsquared.png&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Uploaded with &lt;a href=&quot;http://imageshack.us&quot; rel=&quot;nofollow&quot;&gt;ImageShack.us&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is some R code for a basic graph with mean and the regression line plotted to help visualize, but without some helpful notes:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data(trees)&#xA;plot((trees$Volume~trees$Girth))&#xA;abline(lm(trees$Volume~trees$Girth))&#xA;abline(lm(trees$Volume~1))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="485" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-08-10T04:00:36.160" LastActivityDate="2010-08-10T04:00:36.160" CommentCount="6" />
  <row Id="1451" PostTypeId="2" ParentId="942" CreationDate="2010-08-09T15:51:47.087" Score="4" ViewCount="" Body="&lt;p&gt;The list in the presentation that you reference seems fairly arbitrary to me, and the technique that would be used will really depend on the specific problem.  You will note however that it also includes &lt;a href=&quot;http://en.wikipedia.org/wiki/Kalman_filter&quot; rel=&quot;nofollow&quot;&gt;Kalman filters&lt;/a&gt;, so I suspect that the intended usage is as a filtering technique.  Wavelet transforms generally fall under the subject of &lt;a href=&quot;http://en.wikipedia.org/wiki/Category%3aSignal_processing&quot; rel=&quot;nofollow&quot;&gt;signal processing&lt;/a&gt;, and will often be used as a pre-processing step with very noisy data.  An example is the &quot;&lt;a href=&quot;http://portal.acm.org/citation.cfm?id=1343118.1343331&quot; rel=&quot;nofollow&quot;&gt;Multi-scale anomaly detection&lt;/a&gt;&quot; paper by Chen and Zhan (see below).  The approach would be to run an analysis on the different spectrum rather than on the original noisy series.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Wavelets are often compared to a continuous-time fourier transform, although they have the benefit of being localized in both time and frequency.  Wavelets can be used both for signal compression and also for smoothing (wavelet shrinkage).  Ultimately, it could make sense to apply a further statistical after the wavelet transform has been applied (by looking in at the auto-correlation function for instance).  One further aspect of wavelets that could be useful for anomaly detection is the effect of localization: namely, a discontinuity will only influence the wavelet that is near it (unlike a fourier transform).  One application of this is to finding locally stationary time series (using an LSW).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.stats.bris.ac.uk/~magpn/&quot; rel=&quot;nofollow&quot;&gt;Guy Nason&lt;/a&gt; has a nice book that I would recommend if you want to delve further into the practical statistical application: &quot;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0387759603&quot; rel=&quot;nofollow&quot;&gt;Wavelet Methods in Statistics with R&lt;/a&gt;&quot;.  This is specifically targeting the application of wavelets to statistical analysis, and he provides many real world examples along with all the code (using the &lt;a href=&quot;http://cran.r-project.org/web/packages/wavethresh/index.html&quot; rel=&quot;nofollow&quot;&gt;wavethresh package&lt;/a&gt;).  Nason's book does not address &quot;anomaly detection&quot; specifically, although it does do an admiral job of providing a general overview.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lastly, &lt;a href=&quot;http://en.wikipedia.org/wiki/Wavelet&quot; rel=&quot;nofollow&quot;&gt;the wikipedia article&lt;/a&gt; does provide many good introductory references, so it is worth going through it in detail.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Xiao-yun Chen, Yan-yan Zhan &lt;a href=&quot;http://portal.acm.org/citation.cfm?id=1343118.1343331&quot; rel=&quot;nofollow&quot;&gt;&quot;Multi-scale anomaly detection algorithm based on infrequent pattern of time series&quot;&lt;/a&gt; Journal of Computational and Applied Mathematics Volume 214 ,  Issue 1  (April 2008) &lt;/li&gt;&#xA;&lt;li&gt;G.P. Nason &quot;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0387759603&quot; rel=&quot;nofollow&quot;&gt;Wavelet Methods in Statistics with R&lt;/a&gt;&quot; Springer, 2008&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;[As a side note: if you are looking for a good modern technique for change point detection, I would suggest trying a HMM before spending too much time with wavelet methods, unless you have good reason to be using wavelets in your particular field.  This is based on my personal experience.  There are of course many other nonlinear models that could be considered, so it really depends on your specific problem.]&lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-08-09T15:51:47.087" />
  <row Id="1452" PostTypeId="2" ParentId="1444" CreationDate="2010-08-09T16:43:48.870" Score="5" ViewCount="" Body="&lt;p&gt;The log transforms with shifts are special cases of the &lt;a href=&quot;http://en.wikipedia.org/wiki/Box-Cox_transformation&quot; rel=&quot;nofollow&quot;&gt;Box-Cox transformations&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$y(\lambda_{1}, \lambda_{2}) = &#xA;\begin{cases} &#xA; \frac {(y+\lambda_{2})^{\lambda_1} - 1} {\lambda_{1}} &amp;amp; \mbox{when } \lambda_{1} \neq 0 \\ \log (y + \lambda_{2}) &amp;amp; \mbox{when } \lambda_{1} = 0&#xA;\end{cases}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These are the extended form for negative values, but also applicable to data containing zeros.  Box and Cox (1964) presents an algorithm to find appropriate values for the $\lambda$'s using maximum likelihood.  This gives you the ultimate transformation.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A reason to prefer Box-Cox transformations is that they're developed to ensure assumptions for the linear model.  There's some work done to show that even if your data cannot be transformed to normality, then the estimated $\lambda$ still lead to a symmetric distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not sure how well this addresses your data, since it could be that $\lambda = (0, 1)$ which is just the log transform you mentioned, but it may be worth estimating the requried $\lambda$'s to see if another transformation is appropriate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In R, the &lt;a href=&quot;http://www.stat.ucl.ac.be/ISdidactique/Rhelp/library/geoR/html/boxcox.fit.html&quot; rel=&quot;nofollow&quot;&gt;&lt;code&gt;boxcox.fit&lt;/code&gt;&lt;/a&gt; function in package &lt;code&gt;geoR&lt;/code&gt; will compute the parameters for you.&lt;/p&gt;&#xA;" OwnerUserId="251" LastEditorUserId="251" LastEditorDisplayName="" LastEditDate="2010-08-10T01:59:44.693" LastActivityDate="2010-08-10T01:59:44.693" CommentCount="7" />
  <row Id="1453" PostTypeId="2" ParentId="1447" CreationDate="2010-08-09T16:49:45.147" Score="3" ViewCount="" Body="&lt;p&gt;The &lt;a href=&quot;http://onlinestatbook.com/stat_sim/reg_by_eye/index.html&quot; rel=&quot;nofollow&quot;&gt;Regression By Eye&lt;/a&gt; applet could be of use if you're trying to develop some intuition.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It lets you generate data then guess a value for &lt;em&gt;R&lt;/em&gt;, which you can then compare with the actual value.  &lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-08-09T16:49:45.147" />
  <row Id="1454" PostTypeId="1" AcceptedAnswerId="1457" CreationDate="2010-08-09T17:43:33.807" Score="1" ViewCount="36" Body="&lt;p&gt;I have a data set that contains two types of points. The first type of points come from an N(0,1) distribution. The second type of points come from an N(m,v) distribution for some real m and some positive, real v. The objective is to classify each point as type 1 or type 2, and to identify m &amp;amp; v. We have no apriori information about m &amp;amp; v. Any ideas?&lt;/p&gt;&#xA;" OwnerUserId="247" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-29T00:21:42.403" LastActivityDate="2011-04-29T00:21:42.403" Title="How to identify points and an unknown distribution in a two type clustering problem?" Tags="&lt;clustering&gt;&lt;mixture&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="1455" PostTypeId="1" AcceptedAnswerId="1456" CreationDate="2010-08-09T17:47:12.227" Score="4" ViewCount="423" Body="&lt;p&gt;I have a bunch of articles presenting &quot;OR&quot; with a- 95% CI (confidence intervals).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to estimate from the articles the P value for the observed OR.  For that, I need an assumption regarding the OR distribution.  What distribution can I safely assume/use?&lt;/p&gt;&#xA;" OwnerUserId="253" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-29T00:22:12.450" LastActivityDate="2011-04-29T00:22:12.450" Title="What is the distribution of OR (odds ratio)?" Tags="&lt;distributions&gt;&lt;odds-ratio&gt;" AnswerCount="4" />
  <row Id="1456" PostTypeId="2" ParentId="1455" CreationDate="2010-08-09T18:00:24.507" Score="3" ViewCount="" Body="&lt;p&gt;The log odds ratio has a Normal asymptotic distribution :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\log(\hat{OR}) \sim N(\log(OR), \sigma_{\log(OR)}^2)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;with $\sigma$ estimated from the contingency table.  See, for example, page 6 of the notes:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://faculty.business.utsa.edu/rtripath/7853/Chapter14/Lecture14_2.pdf&quot; rel=&quot;nofollow&quot;&gt;Asymptotic Theory for Parametric Models&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="251" LastEditorUserId="251" LastEditorDisplayName="" LastEditDate="2010-08-13T06:44:20.513" LastActivityDate="2010-08-13T06:44:20.513" CommentCount="3" />
  <row Id="1457" PostTypeId="2" ParentId="1454" CreationDate="2010-08-09T18:03:31.257" Score="1" ViewCount="" Body="&lt;p&gt;You could use a &lt;a href=&quot;http://en.wikipedia.org/wiki/Mixture_model&quot; rel=&quot;nofollow&quot;&gt;mixture model&lt;/a&gt; to separate out the components. The data generating process  can be represented as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$z_i$: be the type (1 or 2) for the $i^{th}$ observation,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$y_i$ be the $i^{th}$ observation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then you have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$f(y_i|z_i=1) \sim N(0,1)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$f(y_i|z_i=2) \sim N(m,v)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(z_i=1) = \pi$ and&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(z_i=2) = 1-\pi$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The likelihood function is given by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$L(m,v,\pi|-) = \sum_{y_i} \pi f(y_i|z_i=1) + (1-\pi) f(y_i|z_i=2)$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can then use either &lt;a href=&quot;http://en.wikipedia.org/wiki/Mixture_model#Common_approaches_for_estimation_in_mixture_models&quot; rel=&quot;nofollow&quot;&gt;EM or MCMC&lt;/a&gt; to estimate for the model parameters.&lt;/p&gt;&#xA;" LastActivityDate="2010-08-09T18:36:17.977" CommentCount="1" />
  <row Id="1458" PostTypeId="1" CreationDate="2010-08-09T18:03:54.360" Score="9" ViewCount="439" Body="&lt;p&gt;I find it hard to understand what really is the issue with &lt;em&gt;multiple comparisons&lt;/em&gt;.  With a simple analogy, it is said that a person who will make many decisions will make many mistakes. So very conservative precaution is applied, like Bonferroni correction, so as to make the probability that, this person will make any mistake at all, as low as possible. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But why do we care about whether the person has made any mistake at all among all decisions he/she has made, rather than the &lt;em&gt;percentage&lt;/em&gt; of the wrong decisions?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let me try to explain what confuses me with another analogy.  Suppose there are two judges, one is 60 years old, and the other is 20 years old.  Then Bonferroni correction tells the one which is 20 years old to be as conservative as possible, in deciding for execution, because he  will work for many more years as a judge, will make many more decisions, so he has to be careful.  But the one at 60 years old will possibly retire soon, will make fewer decisions, so he can be more careless compared to the other.  But actually, both judges should be equally careful or conservative, regardless of the total number of decisions they will make.  I think this analogy more or less translates to the real problems where Bonferroni correction is applied, which I find counterintuitive. &lt;/p&gt;&#xA;" OwnerUserId="148" LastEditorUserId="223" LastEditorDisplayName="" LastEditDate="2010-12-17T07:48:12.923" LastActivityDate="2010-12-17T07:48:12.923" Title="Why is multiple comparison a problem?" Tags="&lt;hypothesis-testing&gt;&lt;multiple-comparisons&gt;" AnswerCount="4" CommentCount="1" FavoriteCount="6" />
  <row Id="1459" PostTypeId="1" CreationDate="2010-08-09T18:10:39.633" Score="2" ViewCount="259" Body="&lt;p&gt;I am trying to build a time series regression forecasting model for an outcome variable, in dollar amount, in terms of other predictors/input variables and autocorrelated errors.  This kind of model is also called dynamic regression model.  I need to learn how to identify transfer functions for each predictor and would love to hear from you about ways to do just that. &lt;/p&gt;&#xA;" OwnerUserId="833" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-08-13T13:04:38.100" LastActivityDate="2011-05-06T13:17:11.287" Title="How to identify transfer functions in a time series regression forecasting model?" Tags="&lt;time-series&gt;&lt;forecasting&gt;&lt;dynamic-regression&gt;" AnswerCount="3" FavoriteCount="1" />
  <row Id="1460" PostTypeId="1" AcceptedAnswerId="1994" CreationDate="2010-08-09T18:16:21.917" Score="0" ViewCount="247" Body="&lt;p&gt;I am trying to figure out the best transformation of my consumption variable. I am running a probit regression to look at whether or not a household enrolls in health insurance.  Consumption per capita is an independent variable and in my current model I use both consumption and consumption squared (two separate variables) to show that consumption increases but with diminishing returns. This makes for fairly straightforward interpretation. However, using the log of consumption is a slightly better fit because it normalizes the distribution and contributes a bit more to the overall R2 for the model but it is more difficult to interpret. Which would you suggest I use - log of consumption or consumption plus the quadratic function? My research is focused on health economics so I'm not sure what the preference is in that discipline. Any insight would be much appreciated. Thank you!&lt;/p&gt;&#xA;" OwnerUserId="834" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-16T06:56:30.077" LastActivityDate="2010-09-16T14:17:36.323" Title="Ideal transformation for consumption variable in a probit model" Tags="&lt;data-transformation&gt;&lt;econometrics&gt;" AnswerCount="4" />
  <row Id="1461" PostTypeId="2" ParentId="1460" CreationDate="2010-08-09T18:27:59.960" Score="0" ViewCount="" Body="&lt;p&gt;It seems to me that you already have a 'partial' statistics answer (better $R^2$ as to the decision as to what to choose: log vs quadratic). You could use other data-driven metrics (e.g., out-of-sample hit rates, whether the parameters are reasonable etc) to judge which model structure is 'better'. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: By the way, are these two models consistent with economic theory? Just asking as I do not know this area. Another way to select a model is to check its consistency with theory.&lt;/p&gt;&#xA;" LastActivityDate="2010-08-09T18:27:59.960" CommentCount="1" />
  <row Id="1462" PostTypeId="1" CreationDate="2010-08-09T18:53:30.313" Score="1" ViewCount="651" Body="&lt;p&gt;Let's say I want to make a football simulator based on real-life data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say I have a player who averages 5.3 yards per carry with a SD of 1.7 yards.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd like to generate a random variable that simulates the next few plays.&#xA;eg:  5.7, 4.9, 5.3, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What stats terms to I need to look up to pursue this idea?  Density function? The normal curve estimates what boundaries the data generally fall within, but how do I translate that into simulation of subsequent data points?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for any guidance!&lt;/p&gt;&#xA;" OwnerUserId="832" LastActivityDate="2010-08-13T16:15:41.417" Title="Using Std.Dev and Mean to generate hypothetical/additional data points?" Tags="&lt;standard-deviation&gt;" AnswerCount="6" CommentCount="1" />
  <row Id="1463" PostTypeId="2" ParentId="1458" CreationDate="2010-08-09T18:55:55.957" Score="11" ViewCount="" Body="&lt;p&gt;You've stated something that is a classic counter argument to Bonferroni corrections.  Shouldn't I adjust my alpha criterion based on every test I will ever make?  This kind of ad absurdum implication is why some people do not believe in Bonferroni style corrections at all.  Sometimes the kind of data one deals with in their career is such that this is not an issue.  For judges who make one, or very few decisions on each new piece of evidence this is a very valid argument.  But what about the judge with 20 defendants and who is basing their judgment on a single large set of data (e.g. war tribunals)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You're ignoring the kicks at the can part of the argument.  Generally scientists are looking for something — a p-value less than alpha.  Every attempt to find one is another kick at the can.  One will eventually find one if one takes enough shots at it.  Therefore, they should be penalized for doing that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The way you harmonize these two arguments is to realize they are both true.  The simplest solution is to consider testing of differences within a single dataset as a kicks at the can kind of problem but that expanding the scope of correction outside that would be a slippery slope.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a genuinely difficult problem in a number of fields, notably FMRI where there are thousands of data points being compared and there are bound to be some come up as significant by chance.  Given that the field has been historically very exploratory one has to do something to correct for the fact that hundreds of areas of the brain will look significant purely by chance.  Therefore, many methods of adjustment of criterion have been developed in that field.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, in some fields one might at most be looking at 3 to 5 levels of a variable and always just test every combination if a significant ANOVA occurs.  This is known to have some problems (type 1 errors) but it's not particularly terrible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It depends on your point of view.  The FMRI researcher recognizes a real need for a criterion shift.  The person looking at a small ANOVA may feel that there's clearly something there from the legitimate test and a bunch of t-tests are simply uncovering what that is.  The proper conservative point of view on the multiple comparisons is to always do something about them but only based on a single dataset.  Any new data resets the criterion... unless you're a Bayesian...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And remember, it all gets worked out in the replications.&lt;/p&gt;&#xA;" OwnerUserId="601" LastActivityDate="2010-08-09T18:55:55.957" CommentCount="1" />
  <row Id="1464" PostTypeId="2" ParentId="1462" CreationDate="2010-08-09T19:02:57.657" Score="2" ViewCount="" Body="&lt;p&gt;You need a random number generator for the standard normal.  Either you can supply mean and standard deviation as arguments to the function, or you simply scale yourself by multiplying with the latter for the variability and adding the former for for the central location.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a quick example of the former approach:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; set.seed(42)&#xA;&amp;gt; x &amp;lt;- rnorm(1000, 5.3, 1.7)        # 1000 draws of N(5.5, 1.7)&#xA;&amp;gt; print(c(mean=mean(x), sd=sd(x)))&#xA;  mean     sd &#xA;5.2561 1.7043 &#xA;&amp;gt; &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="334" LastActivityDate="2010-08-09T19:02:57.657" />
  <row Id="1465" PostTypeId="2" ParentId="1460" CreationDate="2010-08-09T19:13:45.493" Score="2" ViewCount="" Body="&lt;p&gt;I am not sure I understand your interpretation: a log-transformed predictor would imply that the effect is increasing with diminishing returns, while a quadratic function as a predictor would imply the existance of a peak in the effect (for ax^2+bx+c the peak is at -b/(2a)). I would assume the latter is less realistic in your context, and is also not supported by the better R-squared for the log-transformed predictor.&lt;/p&gt;&#xA;" OwnerUserId="279" LastActivityDate="2010-08-09T19:13:45.493" CommentCount="1" />
  <row Id="1466" PostTypeId="2" ParentId="1462" CreationDate="2010-08-09T19:13:48.053" Score="3" ViewCount="" Body="&lt;p&gt;If you want a realistic simulation you need to find a distribution that describes the real process good enough (a model).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When a real player makes a move he will on average (e.g.) throw &lt;code&gt;X&lt;/code&gt; yards, with a standard deviation of &lt;code&gt;Y&lt;/code&gt;. This does however not mean that the distribution of throws is a normal distribution. You should plot the (histogrammed) &lt;code&gt;throw&lt;/code&gt; distribution and plot your &lt;em&gt;fit&lt;/em&gt; (a Normal distribution with mean &lt;code&gt;X&lt;/code&gt; and &amp;sigma;=&lt;code&gt;Y&lt;/code&gt;) and determine if it fits good enough. If not find a distribution that describes the real data better.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you have that down you need to generate random numbers from the distribution you determined.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you data is complete enough you could maybe avoid having to create a complete model. You would create a &lt;a href=&quot;http://en.wikipedia.org/wiki/Frequency_distribution&quot; rel=&quot;nofollow&quot;&gt;frequency distribution&lt;/a&gt; of your data (a histogram) and then do &lt;a href=&quot;http://en.wikipedia.org/wiki/Rejection_sampling&quot; rel=&quot;nofollow&quot;&gt;rejection sampling&lt;/a&gt; directly from it.&lt;/p&gt;&#xA;" OwnerUserId="56" LastEditorUserId="56" LastEditorDisplayName="" LastEditDate="2010-08-10T06:53:15.300" LastActivityDate="2010-08-10T06:53:15.300" CommentCount="2" />
  <row Id="1467" PostTypeId="2" ParentId="1462" CreationDate="2010-08-09T19:53:30.360" Score="6" ViewCount="" Body="&lt;p&gt;Of course you can use rnorm() in R, but it may be easier to understand how drawing from a pdf works by using the &lt;a href=&quot;http://en.wikipedia.org/wiki/Normal_distribution#Generating_values_from_normal_distribution&quot; rel=&quot;nofollow&quot;&gt;probability integral transform&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically, once we specify the structure of the pdf, we can transform this into a cdf (empirically, to ignore what the equation is), and because the values of the cdf have unique values from 0 to 1, we can back-calculate a draw from the original pdf by matching random draws from 0 to 1, with the cdf.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This way, you only need to have a RNG from 0 to 1, and the function of the pdf, and you're set. Here is the R code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x &amp;lt;- seq(-4, 4, len = 1000)&#xA;f &amp;lt;- function(x, mu = 0, sigma = 1) {&#xA;  out &amp;lt;- 1 / sqrt(2*pi*sigma^2) * exp(-(x - mu)^2 / (2*sigma^2))&#xA;  out&#xA;}&#xA;&#xA;x.ecdf &amp;lt;- cumsum(f(x)) / sum(f(x))&#xA;&#xA;out &amp;lt;- vector()&#xA;y &amp;lt;- runif(100)&#xA;for (i in 1:length(y)) {&#xA;  out[i] &amp;lt;- which((y[i] - x.ecdf)^2 == min((y[i] - x.ecdf)^2))&#xA;}&#xA;&#xA;par(mfrow = c(1,2))&#xA;plot(x, x.ecdf)&#xA;hist(x[out], breaks = 20)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://probabilitynotes.files.wordpress.com/2010/08/rnormish.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="291" LastEditorUserId="291" LastEditorDisplayName="" LastEditDate="2010-08-09T20:28:40.977" LastActivityDate="2010-08-09T20:28:40.977" CommentCount="1" />
  <row Id="1468" PostTypeId="2" ParentId="1458" CreationDate="2010-08-09T21:18:01.033" Score="4" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;To fix ideas:&lt;/strong&gt; I will take the case when you obverse,  $n$ independent random variables $(X_i)_{i=1,\dots,n}$ such that for $i=1,\dots,n$ $X_i$ is drawn from $\mathcal{N}(\theta_i,1)$. I assume that you want to know which one have non zero mean, formally you want to test:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$H_{0i} : \theta_i=0$ Vs $H_{1i} : \theta_i\neq 0$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Definition of a threshold:&lt;/strong&gt; You have $n$ decisions to make and you may have different aim. For a given test $i$ you are certainly going to choose a threshold $\tau_i$ and decide not to accept $H_{0i}$ if $|X_i|&gt;\tau_i$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Different options:&lt;/strong&gt; You have to choose the thresholds $\tau_i$ and for that you have two &lt;strong&gt;options&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;choose the &lt;strong&gt;same threshold&lt;/strong&gt; for everyone&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;to choose  &lt;strong&gt;a different threshold&lt;/strong&gt; for everyone (most often a datawise threshold, see below). &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Different aims:&lt;/strong&gt; These options can be driven for different &lt;strong&gt;aims&lt;/strong&gt; such as&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Controling the probability to reject wrongly $H_{0i}$ for one or more than one $i$.&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Controlling the expectation of the false alarm ratio (or False Discovery Rate) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What ever is your aim at the end, it is a good idea to use a datawise threshold. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My answer to your question:&lt;/strong&gt; your intuition is related to the main heuristic for choosing a datawise threshold. It is the following (at the origin of Holm's procedure which is more powerfull than Bonferoni): &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Imagine you have already taken a decision for the $p$ lowest $|X_{i}|$ and the decision is to accept $H_{0i}$  for all of them. Then you only have to make $n-p$ comparisons and you haven't taken any risk to reject $H_{0i}$ wrongly ! Since you haven't used your budget, you may take a little more risk for the remaining test and choose a larger threshold. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;In the case of your judges:&lt;/strong&gt; I assume (and I guess you should do the same) that both judge have the same budgets of false accusation for their life.  The 60 years old judge may be less conservative if, in the past, he did not accuse anyone ! But if he already made a lot of accusation he will be more conservative and maybe even more than the youndest judge. &lt;/p&gt;&#xA;" OwnerUserId="223" LastEditorUserId="223" LastEditorDisplayName="" LastEditDate="2010-08-10T05:10:38.933" LastActivityDate="2010-08-10T05:10:38.933" CommentCount="2" />
  <row Id="1469" PostTypeId="1" AcceptedAnswerId="2084" CreationDate="2010-08-09T21:55:08.393" Score="3" ViewCount="123" Body="&lt;p&gt;Can someone recommend a text with derivations of classical estimator efficiency results? I'm particularly interested in likelihood and pseudo-likelihood estimators for multi-variate discrete models&lt;/p&gt;&#xA;" OwnerUserId="511" LastActivityDate="2010-08-25T00:46:17.397" Title="A Primer on Estimator Efficiency?" Tags="&lt;estimation&gt;&lt;efficiency&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="1" />
  <row Id="1470" PostTypeId="2" ParentId="1458" CreationDate="2010-08-09T22:18:22.997" Score="6" ViewCount="" Body="&lt;p&gt;Related to the comment earlier, what the fMRI researcher should remember is that clinically-important outcomes are what matter, not the density shift of a single pixel on a fMRI of the brain. If it doesn't result in a clinical improvement/detriment, it doesn't matter. That is one way of reducing the concern about multiple comparisons.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See also:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Bauer, P. (1991). Multiple testing in clinical trials. Stat Med, 10(6), 871-89; discussion 889-90.&lt;/li&gt;&#xA;&lt;li&gt;Proschan, M. A. &amp;amp; Waclawiw, M. A. (2000). Practical guidelines for multiplicity adjustment in clinical trials. Control Clin Trials, 21(6), 527-39.&lt;/li&gt;&#xA;&lt;li&gt;Rothman, K. J. (1990). No adjustments are needed for multiple comparisons. Epidemiology (Cambridge, Mass.), 1(1), 43-6.&lt;/li&gt;&#xA;&lt;li&gt;Perneger, T. V. (1998). What's wrong with bonferroni adjustments. BMJ (Clinical Research Ed.), 316(7139), 1236-8.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="561" LastActivityDate="2010-08-09T22:18:22.997" CommentCount="5" />
  <row Id="1471" PostTypeId="1" AcceptedAnswerId="1473" CreationDate="2010-08-09T22:25:11.207" Score="2" ViewCount="269" Body="&lt;p&gt;I need to analyze with R the data from a medical survey (with 100+ coded columns) that comes in a CSV. I will use &lt;a href=&quot;http://rattle.togaware.com/&quot; rel=&quot;nofollow&quot;&gt;rattle&lt;/a&gt; for some initial analysis but behind the scenes it's still R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I &lt;em&gt;read.csv()&lt;/em&gt; the file, columns with numerical codes are treated as numerical data. I'm aware I could create categorical columns from them with &lt;em&gt;factor()&lt;/em&gt; but doing it for 100+ columns is a pain. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope there is a better way to tell R to import the columns directly as factors. Or to at least to convert them in place afterwards.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you!&lt;/p&gt;&#xA;" OwnerUserId="840" LastActivityDate="2010-08-10T00:25:34.777" Title="Is it possible to directly read CSV columns as categorical data ?" Tags="&lt;r&gt;&lt;categorical-data&gt;&lt;data-transformation&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="1" />
  <row Id="1472" PostTypeId="2" ParentId="1459" CreationDate="2010-08-09T22:27:53.513" Score="4" ViewCount="" Body="&lt;p&gt;The classic approach, described in &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0470272848&quot; rel=&quot;nofollow&quot;&gt;Box, Jenkins &amp;amp; Reinsell (4th ed, 2008)&lt;/a&gt; involves looking at the cross-correlation function and the various auto-correlation functions, and making a lot of subjective decisions about the orders and lags for the various terms. The approach works ok for a single predictor, but is not really suitable for multiple predictors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An alternative approach, described in &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0471615285&quot; rel=&quot;nofollow&quot;&gt;Pankratz (1991)&lt;/a&gt;, involves fitting lagged regressions with AR errors and determining the appropriate rational lag structure from the fitted coefficients (also a relatively subjective process). Then refitting the entire model with the supposed lag structures and extracting the residuals. The order of the ARMA error process is determined from these residuals (using AIC for example). Then the final model is re-estimated. This approach works well for multiple predictors, and is considerably simpler to apply than the classic approach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wish I could say there was this neat automated procedure that did it all for you, but I can't. At least not yet.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-08-09T22:27:53.513" CommentCount="2" />
  <row Id="1473" PostTypeId="2" ParentId="1471" CreationDate="2010-08-09T22:31:23.070" Score="4" ViewCount="" Body="&lt;p&gt;You can use the &lt;code&gt;colClasses&lt;/code&gt; argument to specify the classes of your data columns.  For example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data &amp;lt;- read.csv('foo.csv', colClasses=c('numeric', 'factor', 'factor'))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;will assign numeric to the first column, factor to the second and third.  Since you have so many columns, a shortcut might be:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data &amp;lt;- read.csv('foo.csv', colClasses=c('numeric', rep('factor', 37), 'character'))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;or some such variation (i.e. assign numeric to first column, factor to next 37 columns, then character to the last one).&lt;/p&gt;&#xA;" OwnerUserId="251" LastEditorUserId="251" LastEditorDisplayName="" LastEditDate="2010-08-10T00:25:34.777" LastActivityDate="2010-08-10T00:25:34.777" />
  <row Id="1474" PostTypeId="2" ParentId="1471" CreationDate="2010-08-09T22:33:37.870" Score="2" ViewCount="" Body="&lt;p&gt;or just do it after you read the data&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dat &amp;lt;- read.csv(&quot;kdfjdkf&quot;)&#xA;apply(dat, 2, factor)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strike&gt;though this type of Q is probably more fit for &lt;a href=&quot;http://stackoverflow.com/questions/tagged/r&quot;&gt;Stack Overflow&lt;/a&gt;&lt;/strike&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;edit&lt;/strong&gt;: see below.&lt;/p&gt;&#xA;" OwnerUserId="291" LastEditorUserId="291" LastEditorDisplayName="" LastEditDate="2010-08-09T23:48:44.260" LastActivityDate="2010-08-09T23:48:44.260" CommentCount="3" />
  <row Id="1475" PostTypeId="1" AcceptedAnswerId="1491" CreationDate="2010-08-09T22:33:40.163" Score="8" ViewCount="367" Body="&lt;p&gt;I want to cluster ~22000 points. Many clustering algorithms work better with higher quality initial guesses. What tools exist that can give me a good idea of the rough shape of the data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I do want to be able to choose my own distance metric, so a program I can feed a list of pairwise distances to would be just fine. I would like to be able to do something like highlight a region or cluster on the display and get a list of which data points are in that area.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Free software preferred, but I do already have SAS and MATLAB. &lt;/p&gt;&#xA;" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-13T20:32:11.607" LastActivityDate="2011-01-20T15:30:01.123" Title="Visualization software for clustering" Tags="&lt;data-visualization&gt;&lt;clustering&gt;&lt;software&gt;" AnswerCount="7" FavoriteCount="3" />
  <row Id="1476" PostTypeId="2" ParentId="1458" CreationDate="2010-08-09T23:39:55.867" Score="10" ViewCount="" Body="&lt;p&gt;Well-respected statisticians have taken a wide variety of positions on multiple comparisons.  It's a subtle subject. If someone thinks it's simple, I'd wonder how much they've thought about it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's an interesting Bayesian perspective on multiple testing from Andrew Gelman: &lt;a href=&quot;http://www.stat.columbia.edu/~cook/movabletype/archives/2008/03/why_i_dont_usua_1.html&quot; rel=&quot;nofollow&quot;&gt;Why we don't (usually) worry about multiple comparisons&lt;/a&gt;.  &lt;/p&gt;&#xA;" OwnerUserId="319" LastActivityDate="2010-08-09T23:39:55.867" CommentCount="1" />
  <row Id="1477" PostTypeId="2" ParentId="1385" CreationDate="2010-08-10T00:32:44.947" Score="1" ViewCount="" Body="&lt;p&gt;If you have a reasonable hunch about the data generating process that is responsible for the data in question then you could use bayesian ideas to estimate the missing data. Under the bayesian approach you would simply assume that the missing data are also random variables and construct the posterior for the missing data conditional on the observed data. The posterior means would then be used as a substitute for the missing data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The use of bayesian models may qualify as imputation under a broad sense of the term but I thought of mentioning it as it did not appear on your list.&lt;/p&gt;&#xA;" LastActivityDate="2010-08-10T00:32:44.947" />
  <row Id="1478" PostTypeId="1" AcceptedAnswerId="1479" CreationDate="2010-08-10T00:33:26.960" Score="2" ViewCount="233" Body="&lt;p&gt;I am trying to write unit tests for a whole mess of statistics code. Some of the unit tests take the form: generate a sample following a null hypothesis, use code to get a p-value under that null, repeat hundreds of times, then look at all the p-values: if they are reasonably uniform, then the code passes. I usually check if the ratio of p-values &amp;lt; $\alpha$ is near $\alpha$ for $\alpha = 0.01, 0.05, 0.1$. But I am usually also interested in whether the p-values output deviate from uniformity. I usually test this with the Anderson-Darling test. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is where I have a circularity problem: how can I unit test my Anderson-Darling code? I can easily feed it uniformly generated variables, and get a p-value, repeat hundreds of times, but then I just have a bunch of p-values. I can q-q plot them, but I'm more interested in an automatic unit test I can run. What are some basic sanity checks I can implement automatically? there is the naive check of ratio of p-values &amp;lt; $\alpha$ noted above. I can also implement a Kolmogorov-Smirnov test. What else can I easily check for?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(I realize this question may seem hopelessly pedantic or naive or subject to infinite regress...)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;edit&lt;/strong&gt; some additional ideas: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;test the code on $\frac{i}{n}$ for $i = 1,2,...,n$, for different values of $n$. Presumably I can compute, by hand, the p-value for the A-D test in this case.&lt;/li&gt;&#xA;&lt;li&gt;compute $n$ p-values by feeding many uniform samples to the code $n$ times, then regress the order statistics of the p-values, $p_{(i)}$ vs $i/n$, to get $p_{(i)} = \beta_1 i/n + \beta_0$ and test the null $\beta_1 = 1, \beta_0 = 0$. Presumably I can simplify this test by hand in such a way that inspection reveals it to be correct.&lt;/li&gt;&#xA;&lt;li&gt;make sure the code is invariant with respect to permutation of the input. (duh)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="795" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-30T21:21:50.583" LastActivityDate="2010-09-30T21:21:50.583" Title="Testing implementation of Anderson-Darling test for uniform RV" Tags="&lt;test&gt;&lt;code&gt;&lt;uniform&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="1479" PostTypeId="2" ParentId="1478" CreationDate="2010-08-10T00:41:01.463" Score="1" ViewCount="" Body="&lt;p&gt;You could test your Anderson-Darling code using data that is generated from an external library. However, you then run into the issue of how to test/trust the external library. At some point you have to trust that well established libraries are error free and that their output can be relied on. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you have the Anderson-Dalring code tested against data generated from an external library the circularity will be broken and you can rely on your own code if it passes the Anderson-Darling tests. The same will hold for the K-S (I presume Kolmogorov–Smirnov) test.&lt;/p&gt;&#xA;" LastActivityDate="2010-08-10T00:41:01.463" CommentCount="2" />
  <row Id="1480" PostTypeId="2" ParentId="1385" CreationDate="2010-08-10T00:43:08.527" Score="0" ViewCount="" Body="&lt;p&gt;I might be a little unorthodox here, but what the heck. &lt;strong&gt;&lt;em&gt;Please note:&lt;/em&gt; this line of thought comes from my own philosophy for classification, which is that I use it when my purpose is squarely on pure prediction -- not explanation, conceptual coherence, etc. Thus, what I'm saying here contradicts how I'd approach building a regression model.&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Different classification approaches vary in their capability to handle missing data, and depending on some other factors^, I might just try #5: use a classifier that won't choke on those NAs. Part of the decision to go that route might also include thinking about how likely a similar proportion of NAs are to occur in the future data to which you'll be applying the model. If NAs for certain variables are going to be par for the course, then it might make sense to just roll with them (i.e., don't build a predictive model that assumes more informative data than what you'll actually have, or you'll be kidding yourself about how predictive it really is going to be). In fact, if I'm not convinced that NAs are missing at random, I'd be inclined to recode a new variable (or a new level if it's missing in a categorical variable) just to see if the missingness itself is predictive.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I had a good reason to use a classifier that did not take missing data very well, then my approach would be #1 (multiple imputation), seeking to find a classification model that behaved similarly well across imputed data sets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;^Including: how much missingness you have in your predictors, whether there are systematic patterns (if there are, it would be worth taking a closer look and thinking through the implications for your analysis), and how much data you have to work with overall.&lt;/p&gt;&#xA;" OwnerUserId="394" LastActivityDate="2010-08-10T00:43:08.527" />
  <row Id="1481" PostTypeId="2" ParentId="1462" CreationDate="2010-08-10T01:20:52.150" Score="1" ViewCount="" Body="&lt;p&gt;If you have all the relevant data rather than just summary data such as mean, SD etc. you could create your own distribution model from the real life data you have. Sort the data (y values) from lowest to highest and equally space them between 0 and 1 (x values). Then solve to find the coefficients of an nth order polynomial curve fit to the data (or several in piecewise fashion over parts of the data if necessary). Once you have these, it would be a simple matter to use a uniform random number generator to generate an x value between 0 and 1, and to plug this value into the polynomial equation to get a random y value that would approximate a draw from your distribution.&lt;/p&gt;&#xA;" OwnerUserId="226" LastActivityDate="2010-08-10T01:20:52.150" />
  <row Id="1482" PostTypeId="2" ParentId="1475" CreationDate="2010-08-10T02:54:09.990" Score="1" ViewCount="" Body="&lt;p&gt;Take a look at &lt;a href=&quot;http://bonsai.hgc.jp/~mdehoon/software/cluster/&quot; rel=&quot;nofollow&quot;&gt;Cluster 3.0&lt;/a&gt;.  I'm not sure if it will do all you want, but it's pretty well documented and lets you choose from a few distance metrics.  The visualization piece is through a separate program called &lt;a href=&quot;http://jtreeview.sourceforge.net/&quot; rel=&quot;nofollow&quot;&gt;Java TreeView&lt;/a&gt; (&lt;a href=&quot;http://sourceforge.net/dbimage.php?id=20379&quot; rel=&quot;nofollow&quot;&gt;screenshot&lt;/a&gt;).&lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-08-10T02:54:09.990" CommentCount="1" />
  <row Id="1483" PostTypeId="2" ParentId="1405" CreationDate="2010-08-10T04:19:05.090" Score="4" ViewCount="" Body="&lt;p&gt;Assuming the odds ratios are independent, you can proceed as you would in general with any estimate, only you have to look at the log odds.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Take the difference of the log odds, $\delta$.  The standard error of $\delta$ is $\sqrt{SE_{1}^2 + SE_{2}^2}$.  Then you can obtain a p-value for the ratio $z = \delta/SE(\delta)$ from the standard normal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;UPDATE&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The standard error of $\log OR$ is the square root of the sum of the reciprocals of the frequencies:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$SE(\log OR) = \sqrt{ {1 \over n_1} + {1 \over n_2} + {1 \over n_3} + {1 \over n_4} }$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case, each $n_i$ correspond to TP, FP, TN, FN.  &lt;/p&gt;&#xA;" OwnerUserId="251" LastEditorUserId="251" LastEditorDisplayName="" LastEditDate="2010-08-17T18:56:31.063" LastActivityDate="2010-08-17T18:56:31.063" CommentCount="4" />
  <row Id="1484" PostTypeId="2" ParentId="1475" CreationDate="2010-08-10T06:19:14.900" Score="5" ViewCount="" Body="&lt;p&gt;Exploring clustering results in high dimensions can be done in &lt;a href=&quot;http://www.r-project.org/&quot; rel=&quot;nofollow&quot;&gt;R&lt;/a&gt; using the packages &lt;a href=&quot;http://had.co.nz/model-vis/&quot; rel=&quot;nofollow&quot;&gt;clusterfly&lt;/a&gt; and &lt;a href=&quot;http://cran.r-project.org/web/packages/gcExplorer/index.html&quot; rel=&quot;nofollow&quot;&gt;gcExplorer&lt;/a&gt;. Look for more &lt;a href=&quot;http://cran.r-project.org/web/views/Cluster.html&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="339" LastActivityDate="2010-08-10T06:19:14.900" CommentCount="1" />
  <row Id="1485" PostTypeId="1" AcceptedAnswerId="1732" CreationDate="2010-08-10T06:41:06.993" Score="4" ViewCount="252" Body="&lt;p&gt;Hello,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a particular application I was in need of machine learning (I know the things I studied in my undergraduate course). I used Support Vector Machines and got the problem solved. Its working fine.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I need to improve the system. Problems here are&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;I get additional training examples every week. Right now the system starts training freshly with updated examples (old examples + new examples). I want to make it incremental learning. Using previous knowledge (instead of previous examples) with new examples to get new model (knowledge)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Right my training examples has 3 classes. So, every training example is fitted into one of these 3 classes. I want functionality of &quot;Unknown&quot; class. Anything that doesn't fit these 3 classes must be marked as &quot;unknown&quot;. But I can't treat &quot;Unknown&quot; as a new class and provide examples for this too.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Assuming, the &quot;unknown&quot; class is implemented. When class is &quot;unknown&quot; the user of the application inputs the what he thinks the class might be. Now, I need to incorporate the user input into the learning. I've no idea about how to do this too. Would it make any difference if the user inputs a new class (i.e.. a class that is not already in the training set)?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Do I need to choose a new algorithm or Support Vector Machines can do this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: I'm using libsvm implementation for SVM.&lt;/p&gt;&#xA;" OwnerUserId="851" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-08-16T12:54:51.460" LastActivityDate="2010-08-16T12:54:51.460" Title="Few machine learning problems" Tags="&lt;machine-learning&gt;&lt;svm&gt;" AnswerCount="2" FavoriteCount="2" />
  <row Id="1486" PostTypeId="2" ParentId="97" CreationDate="2010-08-10T06:42:10.703" Score="4" ViewCount="" Body="&lt;p&gt;Conventional practice is to use the non-parametric statistics &lt;strong&gt;&lt;em&gt;rank sum&lt;/em&gt;&lt;/strong&gt; and &lt;strong&gt;&lt;em&gt;mean rank&lt;/em&gt;&lt;/strong&gt; to describe ordinal data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's how they work:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Rank Sum&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;assign a rank to each member in each&#xA;group;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;e.g., suppose you are looking at goals for each&#xA;player on two opposing football&#xA;teams then rank each member on&#xA;&lt;em&gt;both&lt;/em&gt; teams from first to last;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;calculate rank sum by adding the ranks &lt;em&gt;per group&lt;/em&gt;;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;the magnitude of the rank sum tells&#xA;you how close together the ranks are&#xA;for each group&#xA;&lt;br&gt;&lt;br&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Mean Rank&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;M/R is a more sophisticated statistic than R/S because it compensates for unequal sizes in the groups you are comparing. Hence, in addition to the steps above, you divide each sum by the number of members in the group.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you have these two statistics, you can, for instance, z-test the rank sum to see if the&#xA;difference between the two groups is statistically significant (i believe that's known as the &lt;em&gt;Wilcixon rank sum test&lt;/em&gt;, which is interchangeable, i.e., functionally equivalent to the Mann-Whitney U test).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;R Functions for these statistics (the ones i know about, anyway):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;wilcox.test&lt;/strong&gt; in the standard R installation&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;meanranks&lt;/strong&gt; in the &lt;em&gt;cranks&lt;/em&gt; Package&lt;/p&gt;&#xA;" OwnerUserId="438" LastActivityDate="2010-08-10T06:42:10.703" />
  <row Id="1487" PostTypeId="1" AcceptedAnswerId="1489" CreationDate="2010-08-10T06:55:21.597" Score="1" ViewCount="119" Body="&lt;p&gt;We have performed a microarray screening of about 200 samples. In each sample we measure about 100 different variables. For technical reasons the screening of these 200 samples was divided into two batches with a couple of weeks interval between them. When all the data has been collected, I have performed principle component analysis (PCA) on the 200 x 100 table. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When we look on linear projection of first 4 components (responsible for ~70% of the variability), we see a clear division between the two experiment batches. An illustration to what I have can be seen here: &lt;a href=&quot;http://img594.imageshack.us/img594/3687/pca.png&quot; rel=&quot;nofollow&quot;&gt;http://img594.imageshack.us/img594/3687/pca.png&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are the accepted techniques to address this issue?&lt;/p&gt;&#xA;" OwnerUserId="213" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-09-16T09:41:11.077" LastActivityDate="2010-09-16T10:02:46.633" Title="Correcting experiment results" Tags="&lt;pca&gt;&lt;experiment-design&gt;&lt;normalization&gt;&lt;microarray&gt;" AnswerCount="3" />
  <row Id="1488" PostTypeId="2" ParentId="1475" CreationDate="2010-08-10T06:59:13.557" Score="0" ViewCount="" Body="&lt;p&gt;Weka is an open source program for data mining (wirtten and extensible in Java), &lt;a href=&quot;http://www.ailab.si/orange/&quot; rel=&quot;nofollow&quot;&gt;Orange&lt;/a&gt; is an open source program and library for data mining and machine learning (written in Python). They both allow convenient and efficient visual exploration of multidimensional data&lt;/p&gt;&#xA;" OwnerUserId="213" LastActivityDate="2010-08-10T06:59:13.557" CommentCount="2" />
  <row Id="1489" PostTypeId="2" ParentId="1487" CreationDate="2010-08-10T07:02:57.383" Score="1" ViewCount="" Body="&lt;p&gt;I would think that the first step would be to examine the component loadings and the actual variables to see if you can identify why the two batches yielded discernible differences. Depending on the reasons for the differences you may or may not be able to use a statistical control to &quot;correct&quot; the results.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, if you have every reason to believe samples were randomly assigned to batches, that the your testing methods have not reached floors or ceilings in their ability to assess the variables of interest, and the actual scores themselves are not of interest but only their relative position, perhaps you could standardize scores along each variable by batch.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit:  It looks like csgillespie understands your research area and has provided you some good links.  All I was suggesting was that for each batch you could calculate a Z score for each observed variable.  This would have the effect of eliminating batch differences since for each variable in each batch the mean would be the same (0) and the standard deviation would be the same (1).&lt;/p&gt;&#xA;" OwnerUserId="196" LastEditorUserId="196" LastEditorDisplayName="" LastEditDate="2010-08-10T15:31:41.193" LastActivityDate="2010-08-10T15:31:41.193" CommentCount="1" />
  <row Id="1490" PostTypeId="2" ParentId="1459" CreationDate="2010-08-10T07:03:22.983" Score="1" ViewCount="" Body="&lt;p&gt;Let me suggest you the &lt;a href=&quot;http://www.stat.pitt.edu/stoffer/tsa2/R_time_series_quick_fix.htm&quot; rel=&quot;nofollow&quot;&gt;R time series tutorial&lt;/a&gt;. It does not provide deep theoretical knowledge, but it does give you a nice introduction. Also, googling for &quot;r time series&quot; gives you a lot of very interesting links&lt;/p&gt;&#xA;" OwnerUserId="213" LastActivityDate="2010-08-10T07:03:22.983" />
  <row Id="1491" PostTypeId="2" ParentId="1475" CreationDate="2010-08-10T07:15:35.670" Score="9" ViewCount="" Body="&lt;p&gt;GGobi (http://www.ggobi.org/), along with the R package rggobi, is perfectly suited to this task. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;See the related presentation for examples: &lt;a href=&quot;http://www.ggobi.org/book/2007-infovis/05-clustering.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.ggobi.org/book/2007-infovis/05-clustering.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-08-10T07:15:35.670" CommentCount="3" />
  <row Id="1492" PostTypeId="2" ParentId="1149" CreationDate="2010-08-10T08:04:22.340" Score="1" ViewCount="" Body="&lt;p&gt;I was eating sushi once and thought that it might make a good intuitive demonstration of ill-conditioned problems. Suppose you wanted to show someone a plane using two sticks touching at their bases.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You'd probably hold the sticks orthogonal to each other.  The effect of any kind of shakiness of your hands on the plane causes it to wobble a little around the what you were hoping to show people, but after watching you for a while they get a good idea of what plane you were intending to demonstrate.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But let's say you bring the sticks' ends closer together and watch the effect of your hands shaking. The plane it forms will pitch far more wildly.  Your audience will have to watch longer to get a good idea of what plane the you are trying to demonstrate.&lt;/p&gt;&#xA;" OwnerUserId="167" LastActivityDate="2010-08-10T08:04:22.340" />
  <row Id="1493" PostTypeId="1" CreationDate="2010-08-10T08:45:44.720" Score="4" ViewCount="219" Body="&lt;p&gt;$\chi^n_k=\sum_{i=1}^kx_i^n$ where $x_i$ are Gaussian variables and $n&gt;2$?&lt;/p&gt;&#xA;" OwnerUserId="852" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-29T00:22:23.557" LastActivityDate="2011-04-29T00:22:23.557" Title="What is the distribution of $\chi^n_k$?" Tags="&lt;distributions&gt;&lt;probability&gt;&lt;stochastic-processes&gt;" AnswerCount="2" CommentCount="3" FavoriteCount="1" />
  <row Id="1494" PostTypeId="2" ParentId="1487" CreationDate="2010-08-10T08:50:49.683" Score="3" ViewCount="" Body="&lt;p&gt;How did you normalise your microarray data? Standard ways are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Robust Multichip Average (&lt;a href=&quot;http://rss.acs.unt.edu/Rdoc/library/affy/html/rma.html&quot; rel=&quot;nofollow&quot;&gt;RMA&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;li&gt;Genechip RMA - this can be a bit slow for lots of samples.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;This &lt;a href=&quot;http://www.ogic.ca/projects/SCNcourse/course_units/unit1/lecture/Introduction%20to%20Affymetrix%20Microarrays.ppt&quot; rel=&quot;nofollow&quot;&gt;presentation&lt;/a&gt; gives a good overview of the two techniques.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;There are two R microarray tutorial papers which may also help:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;A microarray analysis for differential gene expression in the soybean genome using Bioconductor and R (&lt;a href=&quot;http://dx.doi.org/10.1093/bib/bbm043&quot; rel=&quot;nofollow&quot;&gt;link to paper&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;li&gt;Analysing yeast time course microarray data using Bioconductor (&lt;a href=&quot;http://www.mas.ncl.ac.uk/~ncsg3/microarray/&quot; rel=&quot;nofollow&quot;&gt;link to paper&lt;/a&gt;).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Both these papers provide the data and R commands.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Competing interest: I'm a author on the second paper.&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="8" LastActivityDate="2010-08-10T08:50:49.683" />
  <row Id="1495" PostTypeId="1" AcceptedAnswerId="1504" CreationDate="2010-08-10T09:02:02.700" Score="8" ViewCount="139" Body="&lt;p&gt;Does anyone know of research which investigates the effectiveness (understandability?) of different visualization techniques? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, how quickly do people understand one form of visualization over another? Does interactivity with the visualization help people recall the data? Anything along those lines. An example of visualizations might be: scatter plots, graphs, timelines, maps, interactive interfaces (like Parallel Coordinates) etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm particularly interested in research within a lay-person population.&lt;/p&gt;&#xA;" OwnerUserId="665" LastEditorUserId="665" LastEditorDisplayName="" LastEditDate="2010-08-10T09:09:30.093" LastActivityDate="2010-08-10T15:33:16.217" Title="Cognitive processing/interpretation of data visualizations techniques" Tags="&lt;data-visualization&gt;" AnswerCount="5" FavoriteCount="4" />
  <row Id="1496" PostTypeId="2" ParentId="1444" CreationDate="2010-08-10T09:29:15.230" Score="3" ViewCount="" Body="&lt;p&gt;I'm presuming that zero != missing data, as that's an entirely different question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When thinking about how to handle zero's in multiple linear regression, I tend to consider how many zeros do we actually have?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Only a couple of zeros&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I have a single zero in a reasonably large data set, I tend to:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Remove the point, take logs and fit the model&lt;/li&gt;&#xA;&lt;li&gt;Add a small $c$ to the point, take logs and fit the model&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Does the model fit change? What about the parameter values. If the model is fairly robust to the removal of the point, I'll go for quick and dirty approach of adding $c$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could make this procedure a bit less crude and use the boxcox method with shifts described in ars' answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Large number of zeros&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If my data set contains a large number of zeros, then this suggests that simple linear regression isn't the best tool for the job. Instead I would use something like mixture modelling (as suggested by Srikant and Robin).&lt;/p&gt;&#xA;" OwnerUserId="8" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-08-10T10:08:18.710" LastActivityDate="2010-08-10T10:08:18.710" />
  <row Id="1497" PostTypeId="2" ParentId="1495" CreationDate="2010-08-10T10:24:28.057" Score="6" ViewCount="" Body="&lt;p&gt;Cleveland reports on a lot of this research in his 1994 book &lt;em&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0963488414&quot; rel=&quot;nofollow&quot;&gt;The Elements of Graphing Data&lt;/a&gt;&lt;/em&gt; (2nd ed). It is very readable and extremely useful.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-08-10T10:24:28.057" CommentCount="1" />
  <row Id="1498" PostTypeId="2" ParentId="1142" CreationDate="2010-08-10T10:54:56.550" Score="1" ViewCount="" Body="&lt;p&gt;what I do is group the measurements by hour and day of week and compare standard deviations of that. Still doesn't correct for things like holidays and summer/winter seasonality but its correct most of the time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The downside is that you really need to collect a year or so of data to have enough so that stddev starts making sense.&lt;/p&gt;&#xA;" OwnerUserId="94" LastActivityDate="2010-08-10T10:54:56.550" CommentCount="1" />
  <row Id="1499" PostTypeId="2" ParentId="1495" CreationDate="2010-08-10T10:58:52.050" Score="3" ViewCount="" Body="&lt;p&gt;Couple of thoughts:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;I think Bertin's &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1589482611&quot; rel=&quot;nofollow&quot;&gt;Semiology of Graphics&lt;/a&gt; is a classic position in this area.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;As &lt;a href=&quot;http://stats.stackexchange.com/users/159/rob-hyndman&quot;&gt;Rob&lt;/a&gt; pointed out, Cleveland has done some interesting work in the area, example &lt;a href=&quot;https://secure.cs.uvic.ca/twiki/pub/Research/Chisel/ComputationalAestheticsProject/cleveland.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Some &lt;a href=&quot;http://www.perceptualedge.com/examples.php&quot; rel=&quot;nofollow&quot;&gt;examples&lt;/a&gt; of poor design from Stephen Few. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Recently I stumbled upon interesting [and quite provocative, especially for Tufte / Cleveland fanatics as myself ;] piece of research from Human-Computer Interaction Lab at the University of Saskatchewan about &lt;a href=&quot;http://hci.usask.ca/uploads/173-pap0297-bateman.pdf&quot; rel=&quot;nofollow&quot;&gt;usefulness of junk&lt;/a&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="22" LastActivityDate="2010-08-10T10:58:52.050" CommentCount="2" />
  <row Id="1500" PostTypeId="2" ParentId="1475" CreationDate="2010-08-10T11:06:46.970" Score="2" ViewCount="" Body="&lt;p&gt;I've had good experience with &lt;a href=&quot;http://www.knime.org/&quot; rel=&quot;nofollow&quot;&gt;KNIME&lt;/a&gt; during one of my project. It 's an excellent solution for quick exploratory mining and graphing. On top of that it provides R and Weka modules seamless integration.  &lt;/p&gt;&#xA;" OwnerUserId="22" LastActivityDate="2010-08-10T11:06:46.970" CommentCount="1" />
  <row Id="1501" PostTypeId="1" AcceptedAnswerId="1503" CreationDate="2010-08-10T13:47:10.963" Score="2" ViewCount="123" Body="&lt;p&gt;Hello All,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm interested in the process of testing or validating a particular implementation of a statistical method, and what datasets and/or published analysis exist that could be used to do this in practice.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, if I write an al-gore-rythm to implement a simple linear regression, I might feed in soem numbers and check the result looks good, or I might feed numbers into my code and some other system and compare.  In some cases, people seem to have already done this and then publish the numbers and results which could be defined as reference data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To start off, the best one I know is the &lt;a href=&quot;http://www.itl.nist.gov/div898/strd/index.html&quot; rel=&quot;nofollow&quot;&gt;NIST Statistical Reference Datasets&lt;/a&gt; page that publishes a wide range of datasets and calculations that covers areas such as Analysis of Variance, Linaear Regression, Markov Chain and Monte Carlo simulation and Non-Linear regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any other good / notable ones out there.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: I reworded to make clear that I'm not just looking for open datasets, but I'm interested in datasets and solutions to specific statistical problems that could be used to test an implementation of a technique.&lt;/p&gt;&#xA;" OwnerUserId="114" LastEditorUserId="114" LastEditorDisplayName="" LastEditDate="2010-08-11T14:06:20.203" LastActivityDate="2010-08-11T14:06:20.203" CommunityOwnedDate="2010-08-10T13:47:10.963" Title="What resources/methods exist for testing/validation or evaluation of Statistical Methods" Tags="&lt;dataset&gt;&lt;validation&gt;&lt;testing&gt;" AnswerCount="1" CommentCount="8" />
  <row Id="1502" PostTypeId="2" ParentId="1475" CreationDate="2010-08-10T14:02:22.253" Score="1" ViewCount="" Body="&lt;p&gt;GGobi does look interesting for this. Another approach could be to treat your similarity/inverse distance matrices as network adjacency matrices and feeding that into a network analysis routine (e.g., either igraph in R or perhaps Pajek). With this approach I would experiment with cutting the cutting the node distances into a binary tie at various cutpoints.&lt;/p&gt;&#xA;" OwnerUserId="394" LastActivityDate="2010-08-10T14:02:22.253" CommentCount="4" />
  <row Id="1503" PostTypeId="2" ParentId="1501" CreationDate="2010-08-10T14:59:06.187" Score="1" ViewCount="" Body="&lt;p&gt;See the stackoverflow question on this subject: &lt;a href=&quot;http://stackoverflow.com/questions/2252144/datasets-for-running-statistical-analysis-on/&quot;&gt;Datasets for Running Statistical Analysis on&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would reiterate &lt;a href=&quot;http://stackoverflow.com/questions/2252144/datasets-for-running-statistical-analysis-on/2252450#2252450&quot;&gt;my answer&lt;/a&gt;, that R contains (in packages) many of the canonical datasets for specific statistical problems.&lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-08-10T14:59:06.187" CommunityOwnedDate="2010-08-10T14:59:06.187" CommentCount="1" />
  <row Id="1504" PostTypeId="2" ParentId="1495" CreationDate="2010-08-10T15:11:18.593" Score="4" ViewCount="" Body="&lt;p&gt;This subject matter is often discussed under the discipline of HCI (&lt;a href=&quot;http://en.wikipedia.org/wiki/Human%E2%80%93computer_interaction&quot; rel=&quot;nofollow&quot;&gt;human-computer interaction&lt;/a&gt;) which has &lt;a href=&quot;http://www.informaworld.com/smpp/title~db=all~content=t775653648&quot; rel=&quot;nofollow&quot;&gt;it's own journal&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a lot of great work being done on this at Stanford under &lt;a href=&quot;http://hci.stanford.edu/jheer/&quot; rel=&quot;nofollow&quot;&gt;Jeffrey Heer&lt;/a&gt; (the creator of &lt;a href=&quot;http://vis.stanford.edu/protovis/&quot; rel=&quot;nofollow&quot;&gt;protovis&lt;/a&gt; and &lt;a href=&quot;http://prefuse.org/&quot; rel=&quot;nofollow&quot;&gt;prefuse&lt;/a&gt; amongst other things) and in the Stanford &lt;a href=&quot;http://hci.stanford.edu/&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;Human-Computer Interaction Group&lt;/strong&gt;&lt;/a&gt;.  As an example, read the &lt;a href=&quot;http://hci.stanford.edu/publications/2009/heer-horizon-chi09.pdf&quot; rel=&quot;nofollow&quot;&gt;&quot;Sizing the Horizon: The Effects of Chart Size and Layering on the Graphical Perception of Time Series Visualizations&quot;&lt;/a&gt; paper.  The material on the &lt;a href=&quot;http://hci.stanford.edu/courses/cs147/2010/&quot; rel=&quot;nofollow&quot;&gt;CS147: Introduction to Human-Computer Interaction Design&lt;/a&gt; and &lt;a href=&quot;https://graphics.stanford.edu/wikis/cs448b-09-fall&quot; rel=&quot;nofollow&quot;&gt;cs448b Data Visualization&lt;/a&gt; homepages may also be of interest.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also look at the projects list in &lt;a href=&quot;http://www.hcii.cmu.edu/research/projects&quot; rel=&quot;nofollow&quot;&gt;the CMU HCI Institute&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="5" LastEditorUserId="5" LastEditorDisplayName="" LastEditDate="2010-08-10T15:25:14.560" LastActivityDate="2010-08-10T15:25:14.560" />
  <row Id="1505" PostTypeId="2" ParentId="1495" CreationDate="2010-08-10T15:29:25.843" Score="0" ViewCount="" Body="&lt;p&gt;Stephen Kosslyn studies human visual processing, and has written a book called &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0195306627&quot; rel=&quot;nofollow&quot;&gt;Graph Design for the Eye and Mind&lt;/a&gt;. There's useful stuff in there, but he also suggests funny things sometimes. For example, he suggests truncating the y-axis on bar graphs at some point, so that they don't really start at 0, which seems a to be a deep sin to me.&lt;/p&gt;&#xA;" OwnerUserId="287" LastActivityDate="2010-08-10T15:29:25.843" />
  <row Id="1506" PostTypeId="2" ParentId="1495" CreationDate="2010-08-10T15:33:16.217" Score="1" ViewCount="" Body="&lt;p&gt;Specifically on color and perception, I liked the papers below by Bergman, Rogowitz and Treinish.  &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.research.ibm.com/people/l/lloydt/color/color.HTM&quot; rel=&quot;nofollow&quot;&gt;Why Should Engineers and Scientists Be Worried About Color?&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.research.ibm.com/dx/proceedings/pravda/index.htm&quot; rel=&quot;nofollow&quot;&gt;A Rule-based Tool for Assisting Colormap Selection&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.research.ibm.com/dx/proceedings/pravda/truevis.htm&quot; rel=&quot;nofollow&quot;&gt;How NOT to Lie with Visualization&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.research.ibm.com/people/l/lloydt/&quot; rel=&quot;nofollow&quot;&gt;Lloyd Treinish's home page&lt;/a&gt;  (for links to related work)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-08-10T15:33:16.217" />
  <row Id="1507" PostTypeId="1" CreationDate="2010-08-10T16:03:57.047" Score="0" ViewCount="222" Body="&lt;p&gt;Q:  Is my approach correct?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Event:  You toss 5 coins at once. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A student of mine claimed he got 4T &amp;amp; 1H in 39 out of 40 trials (!!)  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I decided to calc the odds of this...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, P(4T &amp;amp; 1H) = 5C4 * (1/2)^4 * (1/2)^1 = .16&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I did this 2 ways:&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;1) Binomial Probability&lt;/p&gt;&#xA;&#xA;&lt;p&gt;n = 40&lt;/p&gt;&#xA;&#xA;&lt;p&gt;r = 39&lt;/p&gt;&#xA;&#xA;&lt;p&gt;p = .16&lt;/p&gt;&#xA;&#xA;&lt;p&gt;q = .84&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P(Exactly 39) = 40 C 39 * (.16)^39 * (.84)^1 = 0%&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;2) Binomial Distribution:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;n = 40&lt;/p&gt;&#xA;&#xA;&lt;p&gt;r = 39 (or more)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;p = .16&lt;/p&gt;&#xA;&#xA;&lt;p&gt;q = .84&lt;/p&gt;&#xA;&#xA;&lt;p&gt;E(X) = u = np = (.16)(40) = 6.4&lt;/p&gt;&#xA;&#xA;&lt;p&gt;SD = SQRT(npq) = 3.16&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Z(39) = (observed - expected) / SD = (39 - 6.4) / 3.16 = 10.3&lt;/p&gt;&#xA;&#xA;&lt;p&gt;p = P( Z &gt; 10.3) = 0%&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Conclusion:  The odds of getting 4T &amp;amp; 1H in 39 out of 40 trials is negligible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Student was on drugs at the time.&lt;/p&gt;&#xA;" OwnerUserId="832" LastActivityDate="2010-08-11T14:06:35.990" Title="Example of using binomial distribution" Tags="&lt;binomial&gt;" AnswerCount="4" CommentCount="10" />
  <row Id="1508" PostTypeId="2" ParentId="1507" CreationDate="2010-08-10T16:23:37.183" Score="0" ViewCount="" Body="&lt;p&gt;&lt;strike&gt;Technically, your case 1 and 2 calculations are not correct as they are not independent trials. You are tossing the same 5 coins 40 times. So, those events are dependent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you ignore the above issue then the above seems ok.&lt;/strike&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On some more reflection I think you can ignore the issue of dependency. Here is my reasoning: The probability of observing 4T &amp;amp; 1H is 0.16. In your case 1 and case 2 calculations you are using this probability across all 40 trials which implicitly accounts for the dependence in the trials. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another way to think about the issue is: If you observe 4T and 1H in the first trial what can you say about the probability that you would observe 4T and 1H in the second. It clearly equals 0.16 and thus there is no dependency. Knowledge of the outcome of one trial does not give us any additional information about the events that are likely to happen in the subsequent trial. Thus, the trials are independent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think the calculation is fine as it stands.&lt;/p&gt;&#xA;" LastActivityDate="2010-08-10T17:08:25.473" />
  <row Id="1509" PostTypeId="2" ParentId="1207" CreationDate="2010-08-10T17:29:28.280" Score="1" ViewCount="" Body="&lt;p&gt;You could use the Hilbert Transformation from DSP theory to measure the instantaneous frequency of your data. The site &lt;a href=&quot;http://ta-lib.org/&quot; rel=&quot;nofollow&quot;&gt;http://ta-lib.org/&lt;/a&gt; has open source code for measuring the dominant cycle period of financial data; the relevant function is called HT_DCPERIOD; you might be able to use this or adapt the code to your purposes. &lt;/p&gt;&#xA;" OwnerUserId="226" LastActivityDate="2010-08-10T17:29:28.280" CommentCount="1" />
  <row Id="1510" PostTypeId="2" ParentId="1478" CreationDate="2010-08-10T17:51:38.350" Score="1" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;I can q-q plot them, but I'm more interested in an automatic unit test I can run.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;If a visual inspection of a q-q plot would suffice, then you could calculate an entropy measure such as the &lt;a href=&quot;http://en.wikipedia.org/wiki/Gini_coefficient&quot; rel=&quot;nofollow&quot;&gt;Gini coefficient&lt;/a&gt; and accept the test after allowing for some tolerance for deviation from the 45 degree line.  For more on comparing distributions in general, you could look to the &lt;a href=&quot;http://cran.r-project.org/web/packages/reldist/index.html&quot; rel=&quot;nofollow&quot;&gt;reldist&lt;/a&gt; package in R for ideas.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Though not directly applicable, I think you'll also find the method presented by Cook, Gelman &amp;amp; Rubin interesting:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.stat.columbia.edu/~gelman/research/published/Cook_Software_Validation.pdf&quot; rel=&quot;nofollow&quot;&gt;Validation of Software for Bayesian Models Using Posterior Quantiles&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-08-10T17:51:38.350" />
  <row Id="1511" PostTypeId="2" ParentId="1507" CreationDate="2010-08-10T18:25:52.837" Score="4" ViewCount="" Body="&lt;p&gt;The probability of observing 4 heads and 1 tail 39 times out of 40 after observing 4 heads and 1 tail 39 times out of 40 is 1.0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;:)&lt;/p&gt;&#xA;" OwnerUserId="601" LastActivityDate="2010-08-10T18:25:52.837" CommentCount="6" />
  <row Id="1512" PostTypeId="2" ParentId="1207" CreationDate="2010-08-10T18:41:10.997" Score="3" ViewCount="" Body="&lt;p&gt;If you expect the process to be stationary -- the periodicity/seasonality will not change over time -- then something like a Chi-square periodogram (see e.g. Sokolove and Bushell, 1978) might be a good choice.  It's commonly used in analysis of circadian data which can have extremely large amounts of noise in it, but is expected to have very stable periodicities.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This approach makes no assumption about the shape of the waveform (other than that it is consistent from cycle to cycle), but does require that any noise be of constant mean and uncorrelated to the signal.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;chisq.pd &amp;lt;- function(x, min.period, max.period) {&#xA;N &amp;lt;- length(x)&#xA;variances = NULL&#xA;periods = seq(min.pd, max.pd)&#xA;rowlist = NULL&#xA;for(lc in periods){&#xA;    ncol = lc&#xA;    nrow = floor(N/ncol)&#xA;    rowlist = c(rowlist, nrow)&#xA;    x.trunc = x[1:(ncol*nrow)]&#xA;    x.reshape = t(array(x.trunc, c(ncol, nrow)))&#xA;    variances = c(variances, var(colMeans(x.reshape)))&#xA;}&#xA;Qp = (rowlist * periods * variances) / var(x)&#xA;df = periods - 1&#xA;pvals = 1-pchisq(Qp, df)&#xA;pass.periods = periods[pvals&amp;lt;alpha]&#xA;pass.pvals = pvals[pvals&amp;lt;alpha]&#xA;#return(cbind(pass.periods, pass.pvals))&#xA;return(cbind(periods[pvals==min(pvals)], pvals[pvals==min(pvals)]))&#xA;}&#xA;&#xA;x = cos( (2*pi/37) * (1:1000))+rnorm(1000)&#xA;chisq.pd(x, 2, 72, .05)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The last two lines are just an example, showing that it can identify the period of a pure trigonometric function, even with lots of additive noise.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As written, the last argument (alpha) in the call is superfluous, the function simply returns the 'best' period it can find; uncomment the first 'return' statement and comment out the second to have it return a list of all periods significant at the level alpha.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This function doesn't do any sort of sanity checking to make sure that you've put in identifiable periods, nor does it (can it) work with fractional periods, nor is there any sort of multiple comparison control built in if you decide to look at multiple periods.  But other than that it should be reasonably robust.&lt;/p&gt;&#xA;" OwnerUserId="61" LastActivityDate="2010-08-10T18:41:10.997" />
  <row Id="1513" PostTypeId="2" ParentId="1462" CreationDate="2010-08-10T20:20:04.203" Score="1" ViewCount="" Body="&lt;p&gt;if you decide to generate your distribution from the data you have observed, your model will never spit out a &quot;tail value&quot;, ie, something outside the range of what you have observed.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;your example data: &#xA;average 5.3 yards per carry with a SD of 1.7 yards&lt;/p&gt;&#xA;&#xA;&lt;p&gt;will have a max and a min, say 10 and 2.  in that case, your calculated distribution will not have any weight in the tails, and will be unable to generate a value of 11 or 1.  maybe this is ok, but it prevents you from ever generating one of those super-human events that everyone loves to watch.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;the functional form for the standard normal has tails that go to infinity, so if you assume the distribution is normal, you will be able to generate simulated values (at very small probabilities) that are higher or lower than your observed data.&lt;/p&gt;&#xA;" OwnerUserId="125" LastActivityDate="2010-08-10T20:20:04.203" />
  <row Id="1514" PostTypeId="2" ParentId="1444" CreationDate="2010-08-10T20:48:11.953" Score="2" ViewCount="" Body="&lt;p&gt;If you want something quick and dirty why not use the square root? &lt;/p&gt;&#xA;" OwnerUserId="856" LastActivityDate="2010-08-10T20:48:11.953" />
  <row Id="1515" PostTypeId="2" ParentId="1493" CreationDate="2010-08-10T21:34:55.513" Score="2" ViewCount="" Body="&lt;p&gt;well, as a bound, if $n$ is even, $\chi_k^n$ will be bounded from below by a Chi-square, and $(\chi_k^n)^{1/n}$ should be bounded from above by the maximum of $k$ half-normals, or thereabouts. &lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-08-10T21:34:55.513" />
  <row Id="1516" PostTypeId="2" ParentId="118" CreationDate="2010-08-10T22:34:01.363" Score="1" ViewCount="" Body="&lt;p&gt;The &lt;em&gt;reason&lt;/em&gt; that we calculate standard deviation instead of absolute error is that we are &lt;strong&gt;assuming error to be normally distributed&lt;/strong&gt;.  It's a part of the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose you were measuring very small lengths with a ruler, then standard deviation is a bad metric for error because you know you will never accidentally measure a negative length.  A better metric would be one to help fit a Gamma distribution to your error:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$E(\log(x)) - \log(E(x))$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Like st. dev., this is also non-negative and differentiable, but it is a better error statistic for this problem.&lt;/p&gt;&#xA;" OwnerUserId="858" LastEditorUserId="858" LastEditorDisplayName="" LastEditDate="2010-08-10T23:30:32.997" LastActivityDate="2010-08-10T23:30:32.997" CommentCount="1" />
  <row Id="1517" PostTypeId="1" AcceptedAnswerId="1518" CreationDate="2010-08-10T22:39:06.193" Score="2" ViewCount="213" Body="&lt;p&gt;I am a database developer working for a sales and manufacturing business.  I am mostly ignorant about statistics.  We need useful metrics.  Our managers are tuned to accounting, and the vagaries of production tend to confound us.  We do very little measuring of our production, and what we have is poorly formed.  I should note we are a &quot;job shop&quot;, not a &quot;flow shop&quot; -- we do a lot of engineer-to-order work, so the usual MRP standards are often hard to apply.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some tradition business metrics are known to us; for example &quot;inventory turn-over rate.&quot;  But we are unable to convert those to useful information.  I believe our inability to qualify data statistically is a big reason why.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course we perform averaging all the time.  Rolling averages (smoothing data over time using a 3-week rolling average, for example) is a helpful extension.  Recently I discovered how to apply standard deviation to labor costing with wonderful benefits.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now that I understand (A) averaging, (B) rolling averages, and (C) standard deviation, what are the next useful functions or functions I should seek to learn?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would love to have your insights on &quot;business intelligence,&quot; I mean defining and using metrics.  But it's the use of stats to get from raw data to usable information that I'm really after.  No matter, give me whatever you've got.  &lt;/p&gt;&#xA;" OwnerUserId="857" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-17T20:25:29.290" LastActivityDate="2010-09-17T20:25:29.290" CommunityOwnedDate="2010-08-10T22:39:06.193" Title="Useful statistical functions for business -- for use by a newbie" Tags="&lt;business-intelligence&gt;" AnswerCount="3" CommentCount="3" FavoriteCount="2" />
  <row Id="1518" PostTypeId="2" ParentId="1517" CreationDate="2010-08-10T23:36:25.607" Score="2" ViewCount="" Body="&lt;p&gt;There are lots of generic tools. You should probably start from these:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Tests to &lt;a href=&quot;http://en.wikipedia.org/wiki/Comparing_means&quot; rel=&quot;nofollow&quot;&gt;compare means&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Linear_regression&quot; rel=&quot;nofollow&quot;&gt;Linear Regression&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Foundational items to understand statistics:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Probability_distribution&quot; rel=&quot;nofollow&quot;&gt;Probability Distribution&lt;/a&gt; and &lt;a href=&quot;http://en.wikipedia.org/wiki/List_of_probability_distributions&quot; rel=&quot;nofollow&quot;&gt;List of Probability Distributions&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;This should probably keep you busy depending on how much background you already have.&lt;/p&gt;&#xA;" LastActivityDate="2010-08-10T23:36:25.607" CommunityOwnedDate="2010-08-10T23:36:25.607" CommentCount="3" />
  <row Id="1519" PostTypeId="1" CreationDate="2010-08-11T02:33:03.993" Score="3" ViewCount="700" Body="&lt;p&gt;In (most of) the analytical chemistry literature, the standard test for detecting outliers in univariate data (e.g. a sequence of measurements of some parameter) is Dixon's Q test. Invariably, all the procedures listed in the textbooks have you compute some quantity from the data to be compared with a tabular value. By hand, this is not much of a concern; however I am planning to write a computer program for Dixon Q, and just caching values strikes me as inelegant. Which brings me to my first question:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;How are the tabular values for Dixon Q generated?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Now, I have already looked into this &lt;a href=&quot;http://pubs.acs.org/doi/pdf/10.1021/ac00002a010&quot; rel=&quot;nofollow&quot;&gt;article&lt;/a&gt;, but I'm of the feeling that this is a bit of cheating, in that the author merely constructs a spline that passes through the tabular values generated by Dixon. I have the feeling that a special function (e.g. error function or incomplete beta/gamma) will be needed somewhere, but at least I have algorithms for those.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now for my second question: ISO seems to be slowly recommending Grubbs's test over Dixon Q nowadays, but judging from the textbooks it has yet to catch on. This on the other hand was relatively easy to implement since it only involves computing the inverse of the CDF of Student t. Now for my second question:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Why would I want to use Grubbs's instead of Dixon's?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;On the obvious front in my case, the algorithm is &quot;neater&quot;, but I suspect there are deeper reasons. Can anyone care to enlighten me?&lt;/p&gt;&#xA;" OwnerUserId="830" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-08-11T10:09:50.527" LastActivityDate="2010-08-20T19:47:54.470" Title="On univariate outlier tests (or: Dixon Q versus Grubbs)" Tags="&lt;outliers&gt;&lt;test&gt;" AnswerCount="1" FavoriteCount="2" />
  <row Id="1520" PostTypeId="1" AcceptedAnswerId="1537" CreationDate="2010-08-11T05:17:23.593" Score="6" ViewCount="368" Body="&lt;p&gt;Came across an interesting problem today. You are given a coin and x money, you double money if you get heads and lose half if tails on any toss.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;What is the expected value of your money in n tries&lt;/li&gt;&#xA;&lt;li&gt;What is the probability of getting more than expected value in (1)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;This is how I approached it. The probability of heads and tails is same (1/2). Expected value after first toss = 1/2(2*x) + 1/2(1/2*x) = 5x/4 So expected value is 5x/4 after first toss. Similarly repeating second toss expectation on 5x/4, Expected value after second toss = 1/2(2*5x/4) + 1/2(1/2*5x/4) = 25x/16&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So you get a sequence of expected values: 5x/4, 25x/16, 125x/64, ...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After n tries, your expected value should be 5^n/4^n * x.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If n is large enough, your expected value should approach the mean of the distribution. So probability that value is greater than expected value should be 0.5. I am not sure about this one.&lt;/p&gt;&#xA;" OwnerUserId="862" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-29T00:22:46.177" LastActivityDate="2011-04-29T00:22:46.177" Title="The expected value of random variable on tosses of a coin" Tags="&lt;probability&gt;&lt;stochastic-processes&gt;&lt;games&gt;" AnswerCount="3" CommentCount="1" FavoriteCount="2" />
  <row Id="1521" PostTypeId="1" AcceptedAnswerId="1532" CreationDate="2010-08-11T05:31:50.527" Score="6" ViewCount="627" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;Possible Duplicate:&lt;/strong&gt;&lt;br&gt;&#xA;  &lt;a href=&quot;http://stats.stackexchange.com/questions/6/the-two-cultures-statistics-vs-machine-learning&quot;&gt;The Two Cultures: statistics vs. machine learning?&lt;/a&gt;  &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&#xA;&#xA;&lt;p&gt;What is the difference between data mining and statistical analysis?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For some background, my statistical education has been, I think, rather traditional.  A specific question is posited, research is designed, and data are collected and analyzed to offer some insight on that question.  As a result, I've always been skeptical of what I considered &quot;data dredging&quot;--looking for patterns in a large dataset and using these patterns to draw conclusions.  I tend to associate the latter with data-mining and have always considered this somewhat unprincipled--along with things like algorithmic variable selection routines.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Nonetheless, there is a large and growing literature on data mining.  Often I see this label referring to specific techniques--clustering, tree-based classification, etc.  Yet, at least from my perspective, these techniques can be &quot;set loose&quot; on a set of data or used in a structured way to address a question.  I'd call the former data mining and the latter statistical analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I work in academic administration and have been asked to do some &quot;data mining&quot; to identify issues and opportunities.  Consistent with my background, my first questions were--what do you want to learn and what are the things that you think contribute to issue.  From their response, it was clear that me and the person asking the question had different ideas on the nature and value of data mining.  Perhaps my take akin to the difference between supervised and unsupervised learning?&lt;/p&gt;&#xA;" OwnerUserId="485" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-16T06:45:54.530" LastActivityDate="2010-09-16T06:45:54.530" Title="Data Mining and Statistical Analysis" Tags="&lt;data-mining&gt;" AnswerCount="6" CommentCount="2" FavoriteCount="6" />
  <row Id="1522" PostTypeId="2" ParentId="1521" CreationDate="2010-08-11T06:25:03.157" Score="2" ViewCount="" Body="&lt;p&gt;I previously wrote a post where I made a few observations comparing data mining to psychology. I think these observations may capture some of the differences you are identifying:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&quot;Data mining seems more concerned with prediction using observed variables than with understanding the causal system of latent variables; psychology is typically more concerned with the causal system of latent variables.&lt;/li&gt;&#xA;&lt;li&gt;Data mining typically involves massive datasets (e.g. 10,000 + rows) collected for a purpose other than the purpose of the data mining. Psychological datasets are typically small (e.g., less than 1,000 or 100 rows) and collected explicitly to explore a research question.&lt;/li&gt;&#xA;&lt;li&gt;Psychological analysis typically involves testing specific models. Automated model development approaches tend not to be theoretically interesting.&quot; - &lt;a href=&quot;http://jeromyanglim.blogspot.com/2009/10/data-mining-and-r.html&quot; rel=&quot;nofollow&quot;&gt;Data Mining and R&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-08-11T06:25:03.157" CommentCount="1" />
  <row Id="1523" PostTypeId="2" ParentId="1521" CreationDate="2010-08-11T06:29:22.680" Score="4" ViewCount="" Body="&lt;p&gt;This might be a duplicate, but I am going to answer it anyways.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data mining is statistics, with some minor differences. You can think of it as re-branding statistics, because statisticians are kinda weird. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is often associated with Computational Statistics, ie only stuff you can do with a computer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data miners &lt;em&gt;stole&lt;/em&gt; a significant proportion of multivariate statistics and called it their own. Check the table of contents of any 1990s multivariate book and compare it to a new data mining book. Very similar.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Statistics is associated with testing hypotheses and with model building, whereas Data Mining is more associated with prediction and classification, regardless of whether there is an understandable model.&lt;/p&gt;&#xA;" OwnerUserId="74" LastEditorUserId="74" LastEditorDisplayName="" LastEditDate="2010-08-11T17:23:20.277" LastActivityDate="2010-08-11T17:23:20.277" CommentCount="5" />
  <row Id="1524" PostTypeId="2" ParentId="1517" CreationDate="2010-08-11T06:42:21.157" Score="4" ViewCount="" Body="&lt;p&gt;I would suggest, if you can afford the time, to follow two online (taped) courses, one in probability and another one in statistics. I think it is the best way to get some basic knowledge that will help you move forward.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://academicearth.org/courses/math-and-proability-for-life-sciences&quot; rel=&quot;nofollow&quot;&gt;Probability course&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://video.google.com/videoplay?docid=-3474013489970580510&amp;amp;hl=en&amp;amp;emb=1#&quot; rel=&quot;nofollow&quot;&gt;Statistics course&lt;/a&gt; (This is lesson one. It goes up to &lt;a href=&quot;http://video.google.com/videoplay?docid=3835401745697888723&amp;amp;ei=coO0S9aOOJLF-Qb9kJnnBg&amp;amp;q=Statistics+401%3A+Lesson+64&amp;amp;hl=en&amp;amp;view=3#&quot; rel=&quot;nofollow&quot;&gt;Lesson 64&lt;/a&gt;. Unfortunately, I haven't found the links gathered together in one page. I think also that lessons 21,26 and 39 are missing but I don't remember having any problem whatsoever in following the course). You can also find the handouts for this class posted &lt;a href=&quot;http://www.public.iastate.edu/~pcaragea/S401F07/Handouts07.html&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.  &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="339" LastEditorUserId="339" LastEditorDisplayName="" LastEditDate="2010-08-11T07:22:46.477" LastActivityDate="2010-08-11T07:22:46.477" CommunityOwnedDate="2010-08-11T06:42:21.157" />
  <row Id="1525" PostTypeId="1" AcceptedAnswerId="4850" CreationDate="2010-08-11T07:24:02.763" Score="6" ViewCount="1122" Body="&lt;p&gt;Say I have eaten hamburgers every Tuesday for years. You could say that I eat hamburgers 14% of the time, or that the probability of me eating a hamburger in a given week is 14%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are the main differences between probabilities and proportions?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is a probability an expected proportion?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are probabilities uncertain and proportions are guaranteed?&lt;/p&gt;&#xA;" OwnerUserId="74" LastEditorUserId="74" LastEditorDisplayName="" LastEditDate="2010-08-16T19:03:53.470" LastActivityDate="2011-01-13T04:13:48.283" Title="What's the difference between a probability and a proportion?" Tags="&lt;probability&gt;" AnswerCount="5" CommentCount="4" FavoriteCount="1" />
  <row Id="1526" PostTypeId="2" ParentId="1521" CreationDate="2010-08-11T07:27:27.137" Score="2" ViewCount="" Body="&lt;p&gt;I don't think the distinction you make is really related to the difference between data mining and statistical analysis. You are talking about the difference between exploratory analysis and modelling-prediction approach. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think the tradition of statisic is build with all steps : &#xA; exploratory analysis, then modeling, then estimation, then testing, then forecasting/infering. Statistician do exploratory analysis to figure out what the data looks like (function summary under R !) &#xA;I guess datamining is less structured and could be identified with exploratory analysis. However it uses techniques from statistics that are from estimation, forecasting, classification ....    &lt;/p&gt;&#xA;" OwnerUserId="223" LastActivityDate="2010-08-11T07:27:27.137" CommentCount="1" />
  <row Id="1527" PostTypeId="2" ParentId="1525" CreationDate="2010-08-11T07:46:40.850" Score="13" ViewCount="" Body="&lt;p&gt;If you flip a fair coin 10 times and it comes up heads 3 times, the &lt;strong&gt;proportion&lt;/strong&gt; of heads is .30 but the &lt;strong&gt;probability&lt;/strong&gt; of a head on any one flip is .50.&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-08-11T07:46:40.850" CommentCount="5" />
  <row Id="1528" PostTypeId="2" ParentId="1525" CreationDate="2010-08-11T07:50:04.400" Score="3" ViewCount="" Body="&lt;p&gt;A proportion implies it is a guaranteed event, whereas a probability is not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you eat hamburgers 14% of the time, in a given (4-week) month (or over whatever interval you based your proportion on), you must have eaten 4 hamburgers; whereas with probability there is a possibility of having eaten no hamburgers at all or perhaps eaten a hamburger everyday.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Probability is a measure of uncertainty, whereas proportion is a measure of certainty.&lt;/p&gt;&#xA;" OwnerUserId="17" LastActivityDate="2010-08-11T07:50:04.400" />
  <row Id="1529" PostTypeId="1" CreationDate="2010-08-11T10:09:24.163" Score="4" ViewCount="132" Body="&lt;p&gt;When carrying out OLS multiple linear regression, rather than plot the residuals against fitted values,  I plot the (internal) Studentized residuals against fitted values (ditto for covariates). These residuals are defined as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{equation}&#xA;e^*_i = \frac{e_i}{\sqrt{s^2 (1-h_{ii})}}&#xA;\end{equation}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $e_i$ is the residual and $h_{ii}$ are the diagonal elements of the hat matrix. To get these studentized residuals in R, you can use the &lt;code&gt;rstandard&lt;/code&gt; command. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What type of residuals to do people routinely use in this context? For example, do you just stick with $e_i$ or do you use jackknife residuals, or something else entirely.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: I'm not that interested in papers that define a new type of residual that no-one ever uses.&lt;/p&gt;&#xA;" OwnerUserId="8" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-08-11T13:06:46.477" LastActivityDate="2010-08-11T13:06:46.477" Title="What type of post-fit analysis of residuals do you use" Tags="&lt;regression&gt;&lt;residuals&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="1530" PostTypeId="2" ParentId="1395" CreationDate="2010-08-11T10:29:48.033" Score="2" ViewCount="" Body="&lt;p&gt;In addition to the rmeta package there is also the meta package in R, which produce publication quality plots.&lt;/p&gt;&#xA;" OwnerUserId="609" LastActivityDate="2010-08-11T10:29:48.033" />
  <row Id="1531" PostTypeId="1" AcceptedAnswerId="2233" CreationDate="2010-08-11T10:30:31.527" Score="2" ViewCount="1754" Body="&lt;p&gt;I am working on some MRSA data and need to calculate the relative risk of a group of hospitals compared with the remaining hospital.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My colleagues throws me an excel with a formula inside to calculate the &quot;exact confidence interval of relative risk&quot;, I can do the calculation without difficulties, but I have no idea on how and why this formula is used for do such calculation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have attached the excel file &lt;a href=&quot;http://www.speedfile.org/987618&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; for your reference.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone show me a reference on the rationale of the calculation? Article from textbooks will be fine to me. Thanks!&lt;/p&gt;&#xA;" OwnerUserId="588" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-29T00:22:59.733" LastActivityDate="2011-04-29T00:22:59.733" Title="How to calculate the &quot;exact confidence interval&quot; for relative risk?" Tags="&lt;confidence-interval&gt;&lt;epidemiology&gt;&lt;relative-risk&gt;" AnswerCount="3" CommentCount="1" FavoriteCount="0" />
  <row Id="1532" PostTypeId="2" ParentId="1521" CreationDate="2010-08-11T10:36:15.907" Score="6" ViewCount="" Body="&lt;p&gt;Jerome Friedman wrote a paper a while back: &lt;a href=&quot;http://www-stat.stanford.edu/~jhf/ftp/dm-stat.pdf&quot; rel=&quot;nofollow&quot;&gt;&lt;em&gt;Data Mining and Statistics: What's the Connection?&lt;/em&gt;&lt;/a&gt;, which I think you'll find interesting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data mining was a largely commercial concern and driven by business needs (coupled with the &quot;need&quot; for vendors to sell software and hardware systems to businesses).  One thing Friedman noted was that all the &quot;features&quot; being hyped originated outside of statistics -- from algorithms and methods like neural nets to GUI driven data analysis -- and none of the traditional statistical offerings seemed to be a part of any of these systems (regression, hypothesis testing, etc).  &quot;Our core methodology has largely been ignored.&quot;  It was also sold as user driven along the lines of what you noted: here's my data, here's my &quot;business question&quot;, give me an answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think Friedman was trying to provoke.  He didn't think data mining had serious intellectual underpinnings where methodology was concerned, but that this would change and statisticians ought to play a part rather than ignoring it.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My own impression is that this has more or less happened.  The lines have been blurred.   Statisticians now publish in data mining journals.  Data miners these days seem to have some sort of statistical training.  While data mining packages still don't hype generalized linear models, logistic regression is well known among the analysts -- in addition to clustering and neural nets.  Optimal experimental design may not be part of the data mining core, but the software can be coaxed to spit out p-values.  Progress!&lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-08-11T10:36:15.907" CommentCount="2" />
  <row Id="1533" PostTypeId="2" ParentId="1531" CreationDate="2010-08-11T11:53:51.930" Score="3" ViewCount="" Body="&lt;p&gt;This seems to be Fisher's Exact Test for Count Data.&#xA;You can reproduce the results in R by giving:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data &amp;lt;- matrix(c(678,4450547,63,2509451),2,2)&#xA;fisher.test(data)&#xA;&#xA;data:  data &#xA;p-value &amp;lt; 2.2e-16&#xA;alternative hypothesis: true odds ratio is not equal to 1 &#xA;95 percent confidence interval:&#xA; 4.682723 7.986867 &#xA;sample estimates:&#xA;odds ratio &#xA;  6.068817 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="339" LastActivityDate="2010-08-11T11:53:51.930" />
  <row Id="1534" PostTypeId="1" CreationDate="2010-08-11T12:12:35.573" Score="1" ViewCount="63" Body="&lt;p&gt;I've been asked to give some advice for some clinicians who are comparing two different methods of blood pressure measurement.  I suggested to them that we should proceed with a two-one-sided-test technique to determine equivalence of the two techniques.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unfortunately I have now learned that the clinicians have multiple measurements of blood pressure by each of the two methods and that the blood pressure can be quite variable within each patient during the period of observation (they are theatre cases).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possible to use some multiple regression technique to perform an equivalence test?  Can I simply use confidence intervals to determine variability between the two techniques whilst accounting for inter-patient variability if I use the patients as factors in the regression model?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sorry it's an amateur question, but despite the Masters degree, I still feel like quite the amateur!&lt;/p&gt;&#xA;" OwnerUserId="867" LastEditorUserId="5" LastEditorDisplayName="" LastEditDate="2010-08-11T13:09:20.933" LastActivityDate="2010-08-11T14:05:25.087" Title="Best method for comparing multiple ranging measures" Tags="&lt;multiple-comparisons&gt;&lt;equivalence&gt;" AnswerCount="1" />
  <row Id="1535" PostTypeId="2" ParentId="1529" CreationDate="2010-08-11T12:16:45.457" Score="3" ViewCount="" Body="&lt;p&gt;Your question strikes me as oddly worded. You don't &quot;choose&quot; residuals. Those are given / implied by a procedure / analysis.  The underlying model implies certain (abstract) error terms, and the analysis produces residuals.  In that sense you choose the &lt;em&gt;analysis&lt;/em&gt; not the &lt;em&gt;residuals&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maybe rephrasing to &quot;what type of post-fit analysis of residuals do you use&quot; may be better? You seem to be leaning that way asking the question in the context of a OLS model. And in that case, looking at influential observations (&quot;hat matrix&quot;) is a good method.&lt;/p&gt;&#xA;" OwnerUserId="334" LastActivityDate="2010-08-11T12:16:45.457" CommentCount="1" />
  <row Id="1536" PostTypeId="1" CreationDate="2010-08-11T12:51:51.797" Score="0" ViewCount="227" Body="&lt;p&gt;Does mutual information discriminate against fold change differences?&lt;/p&gt;&#xA;" LastActivityDate="2010-08-11T18:55:03.780" Title="What are the advantages of using mutual information over Pearson correlation in network inference?" Tags="&lt;correlation&gt;&lt;mutual-information&gt;" AnswerCount="1" CommentCount="0" />
  <row Id="1537" PostTypeId="2" ParentId="1520" CreationDate="2010-08-11T12:56:09.933" Score="6" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;If n is large enough, your expected&#xA;  value should approach the mean of the&#xA;  distribution.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Yes that's correct. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;So probability that value is greater&#xA;  than expected value should be 0.5.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This would only be correct if the distribution is symmetric - which in your game isn't the case. You can see this easily if you think about what the median value of your winnings should be after $n$ throws.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;You can think of your problem as a &lt;a href=&quot;http://en.wikipedia.org/wiki/Random_walk&quot; rel=&quot;nofollow&quot;&gt;random walk&lt;/a&gt;. A basic one-dimensional random walk is a walk on the integer real line, where at each point we move $\pm 1$ with probability $p$. This is exactly what you have &lt;strong&gt;if&lt;/strong&gt; we ignore the doubling/halving of money and set $p=0.5$. All we have to do is remap your coordinate system to this example. Let $x$ be your initial starting pot. Then we remap in the following way:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x*2^{-2} = -2&#xA;x*2^{-1} = -1 &#xA;  x = 0&#xA; x*2 = 1  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;i.e. $2^k x=k$. Let $S_n$ denote how much money we have made from the game after $n$ turns, then&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{equation}&#xA;Pr(S_n = 2^k x) = 2^{-n} \binom{n}{(n+k)/2}&#xA;\end{equation}&#xA;for $n \ge (n+k)/2 \ge 0$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When $(n+k)$ isn't a multiple of 2, then $Pr(S_n)=0$. To understand this, assume that we begin with £10. After $n=1$ turns, the only possible values are £5 or £20, i.e. $k=-1$ or $k=1$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The above result is a standard result from Random walks. Google random walks for more info. Also from random walk theory, we can calculate the median return to be $x$, which is not the same as the expected value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note:&lt;/strong&gt; I have assumed that you can always half your money. For example, 1pence, 0.5pence,  0.25pence are all allowed. If you remove this assumption, then you have a random walk with an absorbing wall. &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;For completeness&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's a quick simulation in R of your process:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#Simulate 10 throws with a starting amount of x=money=10&#xA;#n=10&#xA;simulate = function(){&#xA;  #money won/lost in a single game&#xA;  money = 10&#xA;  for(i in 1:10){&#xA;    if(runif(1) &amp;lt; 0.5)&#xA;      money = money/2&#xA;    else&#xA;      money = 2*money&#xA;  }&#xA;  return(money)&#xA;}&#xA;&#xA;#The Money vector keeps track of all the games&#xA;#N is the number of games we play&#xA;N = 1000&#xA;Money = numeric(N)&#xA;for(i in 1:N)&#xA;  Money[i]= simulate()&#xA;&#xA;mean(Money);median(Money)&#xA;#Probabilities&#xA;#Simulated&#xA;table(Money)/1000&#xA;#Exact&#xA;2^{-10}*choose(10,10/2)&#xA;&#xA;#Plot the simulations&#xA;plot(Money)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="8" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-09-10T08:41:42.590" LastActivityDate="2010-09-10T08:41:42.590" CommentCount="3" />
  <row Id="1538" PostTypeId="1" AcceptedAnswerId="1539" CreationDate="2010-08-11T13:15:24.397" Score="9" ViewCount="305" Body="&lt;p&gt;The Kolmogorov–Smirnov distribution is known from the &lt;a href=&quot;http://en.wikipedia.org/wiki/Kolmogorov_Smirnov&quot; rel=&quot;nofollow&quot;&gt;Kolmogorov–Smirnov test&lt;/a&gt;. However, it is also the distribution of the supremum of the Brownian bridge.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since this is far from obvious (to me), I would like to ask you for an intuitive explanation of this coincidence. References are also welcome.&lt;/p&gt;&#xA;" OwnerUserId="650" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-29T00:23:29.750" LastActivityDate="2011-04-29T00:23:29.750" Title="Why does the supremum of the Brownian bridge have the Kolmogorov–Smirnov distribution?" Tags="&lt;distributions&gt;&lt;hypothesis-testing&gt;&lt;mathematical-statistics&gt;&lt;stochastic-processes&gt;" AnswerCount="2" />
  <row Id="1539" PostTypeId="2" ParentId="1538" CreationDate="2010-08-11T13:34:07.253" Score="8" ViewCount="" Body="&lt;p&gt;$\sqrt{n}\sup_x|F_n-F|=\sup_x|\frac{1}{\sqrt{n}}\sum_{i=1}^nZ_i(x)| $ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $Z_i(x)=1_{X_i\leq x}-E[1_{X_i\leq x}]$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;by CLT you have &#xA;$G_n=\frac{1}{\sqrt{n}}\sum_{i=1}^nZ_i(x)\rightarrow \mathcal{N}(0,F(x)(1-F(x)))$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;this is the intuition... &lt;/p&gt;&#xA;&#xA;&lt;p&gt;brownian bridge $B(t)$ has variance $t(1-t)$ &lt;a href=&quot;http://en.wikipedia.org/wiki/Brownian_bridge&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Brownian_bridge&lt;/a&gt; replace $t$ by $F(x)$. This is for one $x$...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You also need to check the covariance and hence it still is easy to show (CLT) that &#xA;for ($x_1,\dots,x_k$)&#xA;$(G_n(x_1),\dots,G_n(x_k))\rightarrow (B_1,\dots,B_k)$ where $(B_1,\dots,B_k)$ is $\mathcal{N}(0,\Sigma)$ with $\Sigma=(\sigma_{ij})$, $\sigma_{ij}=\min(F(x_i),F(x_j))-F(x_i)F(x_j)$.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;strong&gt;difficult&lt;/strong&gt; part is to show that the distribution of the suppremum of the limit is the supremum of the distribution of the limit...  Understanding why this happens requires some empirical process theory, reading books such as van der Waart and Welner (not easy). The name of the Theorem is Donsker Theorem &lt;a href=&quot;http://en.wikipedia.org/wiki/Donsker%27s_theorem&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Donsker%27s_theorem&lt;/a&gt; ... &lt;/p&gt;&#xA;" OwnerUserId="223" LastEditorUserId="223" LastEditorDisplayName="" LastEditDate="2010-08-11T17:10:44.323" LastActivityDate="2010-08-11T17:10:44.323" CommentCount="4" />
  <row Id="1540" PostTypeId="1" AcceptedAnswerId="1543" CreationDate="2010-08-11T13:38:19.647" Score="2" ViewCount="746" Body="&lt;p&gt;Assume you draw a uniformly distributed random number between 0 and 1 &lt;em&gt;n&lt;/em&gt; times.  How would one go about calculating the expected &lt;em&gt;minimum&lt;/em&gt; number drawn after n trials?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In addition, how would one go about calculating a confidence interval to state that the minimum number drawn is in the interval [a,b] with m% confidence?&lt;/p&gt;&#xA;" OwnerUserId="870" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-29T00:24:01.403" LastActivityDate="2011-04-29T00:24:01.403" Title="What is the expected MINIMUM value drawn from a uniform distribution between 0 and 1 after n trials?" Tags="&lt;uniform&gt;&lt;order-statistics&gt;&lt;minimum&gt;" AnswerCount="3" />
  <row Id="1541" PostTypeId="2" ParentId="1507" CreationDate="2010-08-11T13:47:37.173" Score="1" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;It must be indicative of something&#xA;  besides the redistribution of wealth.  &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Heads.  &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;A weaker man might be moved to&#xA;  re-examine his faith, for nothing else&#xA;  at least in the law of probability...  &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Heads.   &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Consider. One, probability is a&#xA;  factor which operates within natural&#xA;  forces. Two, probability is not&#xA;  operating as a factor. Three, we are&#xA;  now held within um... sub or&#xA;  supernatural forces. Discuss!  &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;What?  &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Look at it this way. If six monkeys...&#xA;  If six monkeys... The law of averages,&#xA;  if I have got this right means... that&#xA;  if six monkeys were thrown up in the&#xA;  air long enough... they would land on&#xA;  their tails about as often as they&#xA;  would land on their...  &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Heads, getting&#xA;  a bit of a bore, isn't it?&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;&amp;ndash; Tom Stoppard &lt;em&gt;Rosencrantz and Guildenstern are Dead&lt;/em&gt; (1966)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;As John Christie pointed out, no matter how unlikely the student's result was, you can't infer anything from a single trial. A clever student might well have tried this gambit knowing it could not be refuted, in which case I might be inclined to commend her. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Incidentally, Rosencrantz (or Guildenstern) tossed at least 157 consecutive heads and it was nothing to write home about.&lt;/p&gt;&#xA;" OwnerUserId="869" LastActivityDate="2010-08-11T13:47:37.173" />
  <row Id="1542" PostTypeId="1" CreationDate="2010-08-11T13:48:20.340" Score="1" ViewCount="186" Body="&lt;p&gt;i'm not sure how to google for this as i am not very familiar with time series analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;i have 500 websites, and i am measuring the number of visitors to each website each day.  at some point, i turn on SEO (search engine optimization) for each of the websites.  this happens on different days for different websites.  the distribution of visitors by account for any given day has a long tail to the right.  SEO may not have an immediate effect; it might take a few days/weeks to really start to see some results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;i want to measure something like the &quot;average&quot; lift in the number of visitors, but an average is probably not going to do the trick because of the mix of websites.  (a daily/weekly/monthly trend curve would be really cool, but the average problem will exist there, too)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;i can probably average the number of visitors per day for any given account, but i can't do it across accounts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;do i simply need to segment the websites into &quot;number of visitor&quot; groups?  what other kinds of approaches should i read about?&lt;/p&gt;&#xA;" OwnerUserId="125" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-30T21:21:25.593" LastActivityDate="2010-09-30T21:21:25.593" Title="Analysis of multiple time series" Tags="&lt;time-series&gt;" AnswerCount="1" CommentCount="0" />
  <row Id="1543" PostTypeId="2" ParentId="1540" CreationDate="2010-08-11T13:53:56.707" Score="4" ViewCount="" Body="&lt;p&gt;You are looking for &lt;a href=&quot;http://en.wikipedia.org/wiki/Order_statistic&quot; rel=&quot;nofollow&quot;&gt;order statistics&lt;/a&gt;. The wiki indicates that the distribution of the minimum draw from a uniform distribution between 0 and 1 after $n$ trials is a beta distribution (I have not checked it for correctness which you should probably do.). Specifically, let $U_{(1)}$ be the minimum order statistic. Then:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$U_{(1)} \sim B(1,n)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, the mean is $\frac{1}{1+n}$. You can use the beta distribution to identify $a$ and $b$ such that &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Prob(a \le U_{(1)} \le b) = 0.95$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;By the way, the use of the term confidence interval is not appropriate in this context as you are not performing inference.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Calculating $a$ and $b$ such that $Prob(a \le U_{(1)} \le b) = 0.95$ is not straightforward. There are several possible ways in which you can calculate $a$ and $b$. One approach is to center the interval around the mean. In this approach, you would set:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$a = \mu - \delta$ and &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$b = \mu + \delta$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\mu = \frac{1}{1+n}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You would then calculate $\delta$ such that the required probability is 0.95. Do note that under this approach you may not be able to identify a symmetric interval around the mean for high $n$ but this is just my hunch.&lt;/p&gt;&#xA;" LastActivityDate="2010-08-11T15:10:55.150" />
  <row Id="1544" PostTypeId="2" ParentId="1534" CreationDate="2010-08-11T14:05:25.087" Score="1" ViewCount="" Body="&lt;p&gt;It looks to me like &lt;a href=&quot;http://cran.r-project.org/doc/contrib/Fox-Companion/appendix-mixed-models.pdf&quot; rel=&quot;nofollow&quot;&gt;mixed effects or multi-level modelling&lt;/a&gt; is what you want here.&lt;/p&gt;&#xA;" OwnerUserId="601" LastActivityDate="2010-08-11T14:05:25.087" CommentCount="2" />
  <row Id="1545" PostTypeId="2" ParentId="1507" CreationDate="2010-08-11T14:06:35.990" Score="3" ViewCount="" Body="&lt;p&gt;How about a simulation based approach?  Here's some R code to generate 100000 students each trying the 40 tosses.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;theSum = c()&#xA;for (i in 1:100000) {&#xA;  coin1 = rbinom(40,1,.5)&#xA;  coin2 = rbinom(40,1,.5)&#xA;  coin3 = rbinom(40,1,.5)&#xA;  coin4 = rbinom(40,1,.5)&#xA;  coin5 = rbinom(40,1,.5)&#xA;  theSum[i] = sum(coin1+coin2+coin3+coin4+coin5 == 1)&#xA;}&#xA;&#xA;summary(theSum)&#xA;hist(theSum, xlim = c(0,40), freq = F, main = &quot;&quot;, xlab = &quot;&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The range of times the HTTTT combination occurred (in any order): 0-18 (out of 40), with a mean of around 6.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Below: a histogram of the 100000 attempts and how many times the magical combination occurred.  You'd have to be very lucky indeed to get it 39 times out of 40 with fair coins.  But stranger things have happened by chance (e.g., our evolution).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://img80.imageshack.us/img80/9268/coinflips.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="702" LastActivityDate="2010-08-11T14:06:35.990" />
  <row Id="1546" PostTypeId="2" ParentId="1542" CreationDate="2010-08-11T14:10:21.613" Score="5" ViewCount="" Body="&lt;p&gt;Any study should start out with some conception of a goal.  Are you interested in measuring the impact of your SEO?  Or are you trying to model the visitor behavior?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It isn't clear to me that you're making use of the &quot;time series&quot; aspect of this data.  Are you also interested in the time of day or day of week of visits, for instance?  Or visits around specific events?  You could just as easily divide your # of visits/day into two groups -- with/without SEO -- and eliminate time.  This would then be a categorical variable in your data.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A basic next step could be to run a logistic regression to see the impact of SEO on your site traffic.  In R, this could look something like this (where &quot;seo&quot; is a &lt;code&gt;factor&lt;/code&gt;):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;site.traffic.lg &amp;lt;- glm(num.visits ~ seo, family=binomial, data=your.data)&#xA;summary(site.traffic.lg)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you want to use the fact that this is across a number of different kinds of sites, you could include this by adding it into the formula as another variable.&lt;/p&gt;&#xA;" OwnerUserId="5" LastEditorUserId="5" LastEditorDisplayName="" LastEditDate="2010-08-11T14:16:42.220" LastActivityDate="2010-08-11T14:16:42.220" CommentCount="1" />
  <row Id="1547" PostTypeId="2" ParentId="1521" CreationDate="2010-08-11T14:37:06.930" Score="5" ViewCount="" Body="&lt;p&gt;Data mining is categorized as either Descriptive or Predictive. Descriptive data mining is to search massive data sets and discover the locations of unexpected structures or relationships, patterns, trends, clusters, and outliers in the data. On the other hand, Predictive is to build models and procedures for regression, classification, pattern recognition, or machine learning tasks, and assess the predictive accuracy of those models and procedures when applied to fresh data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The mechanism used to search for patterns or structure in high-dimensional data might be manual or automated; searching might require interactively querying a database management system, or it might entail using visualization software to spot anomalies in the data. In machine-learning terms, descriptive data mining is known as unsupervised learning, whereas predictive data mining is known as supervised learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most of the methods used in data mining are related to methods developed in statistics and machine learning. Foremost among those methods are the general topics of regression, classification, clustering, and visualization. Because of the enormous sizes of the data sets, many applications of data mining focus on dimensionality-reduction techniques (e.g., variable selection) and situations in which high-dimensional data are suspected of lying&#xA;on lower-dimensional hyperplanes. Recent attention has been directed to methods of identifying high-dimensional data lying on nonlinear surfaces or manifolds.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are also situations in data mining when &lt;em&gt;statistical inference&lt;/em&gt; — in its classical sense — either has no meaning or is of dubious validity: the former occurs when we have the &lt;em&gt;entire population&lt;/em&gt; to search for answers, and the latter occurs when a data set is a “convenience” sample rather than being a random sample drawn from some large population. When data are collected through time (e.g., retail transactions, stock-market transactions, patient records, weather records), sampling also may not make sense; the time-ordering of the observations is crucial to understanding the phenomenon generating the data, and to treat the observations as independent when they may be highly correlated will provide biased results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The central components of data mining are — in addition to statistical theory and methods&#xA;— computing and computational efficiency, automatic data processing, dynamic and interactive data visualization techniques, and algorithm development.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of the most important issues in data mining is the computational problem of &lt;em&gt;scalability&lt;/em&gt;. Algorithms developed for computing standard exploratory and confirmatory statistical methods were designed to be fast and computationally efficient when applied to small and medium-sized data sets; yet, it has been shown that most of these algorithms are not up to the challenge of handling huge data sets. As data sets grow, many existing&#xA;algorithms demonstrate a tendency to slow down dramatically (or even grind to a halt).&lt;/p&gt;&#xA;" OwnerUserId="339" LastActivityDate="2010-08-11T14:37:06.930" />
  <row Id="1548" PostTypeId="2" ParentId="1538" CreationDate="2010-08-11T15:02:36.447" Score="3" ViewCount="" Body="&lt;p&gt;For Kolmogorov-Smirnov, consider the null hypothesis.  It says that a sample is drawn from a particular distribution.  So if you construct the empirical distribution function for $n$ samples $f(x) = \frac{1}{n} \sum_i \chi_{(-\infty, X_i]}(x)$, in the limit of infinite data, it will converge to the underlying distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For finite information, it will be off.  If one of the measurements is $q$, then at $x=q$ the empirical distribution function takes a step up.  We can look at it as a random walk which is constrained to begin and end on the true distribution function.  Once you know that, you go ransack the literature for the huge amount of information known about random walks to find out what the largest expected deviation of such a walk is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can do the same trick with any $p$-norm of the difference between the empirical and underlying distribution functions.  For $p=2$, it's called the Cramer-von Mises test.  I don't know the set of all such tests for arbitrary real, positive $p$ form a complete class of any kind, but it might be an interesting thing to look at.&lt;/p&gt;&#xA;" OwnerUserId="873" LastActivityDate="2010-08-11T15:02:36.447" />
  <row Id="1549" PostTypeId="2" ParentId="1540" CreationDate="2010-08-11T15:16:55.993" Score="1" ViewCount="" Body="&lt;p&gt;As Srikant suggests, you need to look at &lt;a href=&quot;http://en.wikipedia.org/wiki/Order_statistic&quot; rel=&quot;nofollow&quot;&gt;order statistics&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To add to Srikant's answer, you can simulate this process easily in R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;n = 10&#xA;N = 1000;sims = numeric(N)&#xA;for(i in 1:N)&#xA;  sims[i] = min(runif(n))&#xA;&#xA;hist(sims, freq=FALSE)&#xA;x = seq(0,1,0.01)&#xA;lines(x, dbeta(x, 1, n), col=2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To get&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://img441.imageshack.us/img441/6826/tmpe.jpg&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Slight digression&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This question is related to one of my favourite statistics problems, the &lt;a href=&quot;http://en.wikipedia.org/wiki/German_tank_problem&quot; rel=&quot;nofollow&quot;&gt;German tank problem&lt;/a&gt;. This problem is about the maximum of uniform distributions, and can be summarised as:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Suppose one is an Allied intelligence&#xA;  analyst during World War II, and one&#xA;  has some serial numbers of captured&#xA;  German tanks. Further, assume that the&#xA;  tanks are numbered sequentially from 1&#xA;  to N. How does one estimate the total&#xA;  number of tanks?&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Taken from wikipedia&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Check out the wikipedia page for more details.&lt;/p&gt;&#xA;" OwnerUserId="8" LastActivityDate="2010-08-11T15:16:55.993" />
  <row Id="1550" PostTypeId="2" ParentId="1469" CreationDate="2010-08-11T15:20:19.787" Score="0" ViewCount="" Body="&lt;p&gt;&quot;Efficient&quot; usually just means that in the class of estimators considered and for a given loss function, you choose one which is optimal.  If you look up &quot;admissible&quot; instead, you'll find a huge amount of information, though it's a slightly weaker criterion.&lt;/p&gt;&#xA;" OwnerUserId="873" LastActivityDate="2010-08-11T15:20:19.787" />
  <row Id="1551" PostTypeId="2" ParentId="1521" CreationDate="2010-08-11T15:20:38.757" Score="5" ViewCount="" Body="&lt;p&gt;The difference between statistics and data mining is largely a historical one, since they came from different traditions: statistics and computer science.  Data mining grew in parallel out of work in the area of artificial intelligence and statistics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Section 1.4 from &lt;a href=&quot;http://www.cs.waikato.ac.nz/~ml/weka/book.html&quot; rel=&quot;nofollow&quot;&gt;Witten &amp;amp; Frank&lt;/a&gt; summarizes my viewpoint so I'm going to quote it at length:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;What's the difference between machine&#xA;  learning and statistics?  Cynics,&#xA;  looking wryly at the explosion of&#xA;  commercial interest (and hype) in this&#xA;  area, equate data mining to statistics&#xA;  plus marketing.  In truth, you should&#xA;  not look for a dividing line between&#xA;  machine learning and statistics&#xA;  because there is a continuum--and a&#xA;  multidimensional one at that--of data&#xA;  analysis techniques.  Some derive from&#xA;  the skills taught in standard&#xA;  statistics courses, and others are&#xA;  more closely associated with the kind&#xA;  of machine learning that has arisen&#xA;  out of computer science. &#xA;  Historically, the two sides have had&#xA;  rather different traditions.  If&#xA;  forced to point to a single difference&#xA;  of emphasis, it might be that&#xA;  statistics has been more concerned&#xA;  with testing hypotheses, whereas&#xA;  machine learning has been more&#xA;  concerned with formulating the process&#xA;  of generalization as a search through&#xA;  possible hypotheses...&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;In the past,&#xA;  very similar methods have developed in&#xA;  parallel in machine learning and&#xA;  statistics...&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;But now the two&#xA;  perspectives have converged.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;N.B.1 IMO, data mining and machine learning are &lt;em&gt;very&lt;/em&gt; closely related terms.  In one sense, machine learning techniques are used in data mining.  I regularly see these terms as interchangeable, and in so far as they are different, they usually go together.  I would suggest looking through &lt;a href=&quot;http://stats.stackexchange.com/questions/6/the-two-cultures-statistics-vs-machine-learning&quot;&gt;&quot;The Two Cultures&quot; paper&lt;/a&gt; as well as the other threads from my original question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;N.B.2 The term &quot;data mining&quot; can have a negative connotation when used colloquially to mean letting some algorithm loose on the data without any conceptual understanding.  The sense is that data mining will lead to spurious results and over-fitting.  I typically avoid using the term when talking to non-experts as a result, and instead use machine learning or statistical learning as a synonym.&lt;/p&gt;&#xA;" OwnerUserId="5" LastEditorUserId="5" LastEditorDisplayName="" LastEditDate="2010-08-11T15:42:21.273" LastActivityDate="2010-08-11T15:42:21.273" CommentCount="4" />
  <row Id="1552" PostTypeId="2" ParentId="1337" CreationDate="2010-08-11T15:56:33.000" Score="5" ViewCount="" Body="&lt;p&gt;A statistician confidently tried to cross a river that was 1 meter deep on average. He drowned.&lt;/p&gt;&#xA;" LastActivityDate="2010-08-11T15:56:33.000" CommunityOwnedDate="2010-08-11T15:56:33.000" />
  <row Id="1553" PostTypeId="2" ParentId="1536" CreationDate="2010-08-11T18:55:03.780" Score="2" ViewCount="" Body="&lt;p&gt;See this question for the differences/advantages of using mutual information versus Pearson correlation or Spearman's rank:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://stats.stackexchange.com/questions/1052/what-is-the-major-difference-between-correlation-and-mutual-information&quot;&gt;What is the major difference between correlation and mutual information?&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Does mutual information discriminate against fold change differences?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;If that's the variation measure you use for your correlation values, then that is assumed for the hypothesized network.&lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-08-11T18:55:03.780" />
  <row Id="1554" PostTypeId="2" ParentId="1540" CreationDate="2010-08-11T19:52:29.877" Score="0" ViewCount="" Body="&lt;p&gt;following @Srikant, one can compute the CDF of the beta distribution, and find conditions on $a, b$ such that the interval $[a,b]$ contains the minimum of $n$ draws of a uniform with 95% probability. The condition is: $(1-a)^n - (1-b)^n = 0.95$. One attractive choice would then be the interval $[0,1 - 0.05^{1/n}]$. This is also the smallest interval with the desired property.&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-08-11T19:52:29.877" />
  <row Id="1555" PostTypeId="1" AcceptedAnswerId="1608" CreationDate="2010-08-11T20:31:25.710" Score="3" ViewCount="314" Body="&lt;p&gt;I have the following frequency table:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;35  0   4   3   7   6   5   4&#xA;39  1   9   6   7   7   6   8&#xA;36  0   7   10  11  11  10  16&#xA;41  0   9   8   8   7   6   7&#xA;41  0   8   9   10  9   12  11&#xA;55  2   12  9   11  12  11  13&#xA;55  1   10  10  11  10  12  11&#xA;47  1   14  8   12  15  12  12&#xA;45  1   10  11  10  10  9   18&#xA;56  0   13  16  12  12  12  11&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The Kruskal-Wallis ANOVA test returns:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Source      SS      df     MS     Chi-sq   Prob&amp;gt;Chi-sq  &#xA;Columns  25306.8    7   3615.26   47.16    5.18783e-008  &#xA;Error     17083.2   72   237.27&#xA;Total     42390     79&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;According to a multiple comparison of mean ranks:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Six groups of mean significantly different from group 1 (column 1)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Six groups of mean significantly different from group 2 (column 2)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Now the Kruskal-Wallis and multiple comparison tests make sense, however the Chi Square Test returns a chi square value of 31.377 and a p-value of 0.9997, which leads us to accept the null hypothesis that the frequencies are independent. I understand that an assumption of ANOVA is independence, but...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to see test if the frequencies are statistically independent, was the Kruskal-Wallis and multiple comparison tests the correct methodology? Note: I am not trying to be subjective, but for a given set of frequencies, how do you test that the differences between groups are significant?&lt;/p&gt;&#xA;" OwnerUserId="559" LastActivityDate="2010-08-12T17:54:22.010" Title="Test if differences between frequencies is significant" Tags="&lt;multiple-comparisons&gt;&lt;statistical-significance&gt;&lt;anova&gt;&lt;chi-squared&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="1556" PostTypeId="1" AcceptedAnswerId="1558" CreationDate="2010-08-11T20:55:41.393" Score="5" ViewCount="232" Body="&lt;p&gt;What is the difference between having something statistically significant (such as a difference between two numbers) and stating if a group of numbers are independent or dependent. &lt;/p&gt;&#xA;" OwnerUserId="559" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-03-21T07:59:09.203" LastActivityDate="2011-03-21T07:59:09.203" Title="Statistically Significant vs Independent/Dependent" Tags="&lt;statistical-significance&gt;&lt;independence&gt;" AnswerCount="2" />
  <row Id="1557" PostTypeId="1" AcceptedAnswerId="1560" CreationDate="2010-08-11T20:57:31.327" Score="5" ViewCount="194" Body="&lt;p&gt;I am taken by the idea of James-Stein shrinkage (i.e. that a nonlinear function of a single observation of a vector of possibly independent normals can be a better estimator of the means of the random variables, where 'better' is measured by squared error). However, I have never seen it in applied work. Clearly I am not well enough read. Are there any classic examples of where James-Stein has improved estimation in an applied setting? If not, is this kind of shrinkage just an intellectual curiosity?&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-09-27T16:35:22.000" Title="James-Stein shrinkage 'in the wild'?" Tags="&lt;estimation&gt;&lt;error&gt;&lt;application&gt;" AnswerCount="5" FavoriteCount="2" />
  <row Id="1558" PostTypeId="2" ParentId="1556" CreationDate="2010-08-11T21:12:16.003" Score="7" ViewCount="" Body="&lt;p&gt;Significance in a independent samples t test just means that the probability of sampling a mean difference as extreme as the mean mean difference you sampled is less than .05 if the null were true.  This is totally unrelated to dependent/independent.  dependent means the observations in the groups are connected somehow A) they are the same person taking the same test a second time, B) people in each group are matched on some pre-test variable, C) people in the two groups are related (i.e. family).  Independent means the observations in the two groups are independent (i.e. the observations in group 1 are independent of the observations in group 2).. &lt;/p&gt;&#xA;" OwnerUserId="880" LastActivityDate="2010-08-11T21:12:16.003" />
  <row Id="1559" PostTypeId="2" ParentId="1525" CreationDate="2010-08-11T21:19:03.187" Score="-1" ViewCount="" Body="&lt;p&gt;I don't know if there is a difference, but probabilities are not % they range from 0 to 1. I mean if you multiply a probability by 100 you get %.  If your question is what's the difference between probability and % then this would be my answer, but this is not your question.  The definition of probability assumes an infinite number of sampling experiments, so then we can never truly get probability because we can never truly conduct an infinite number of sampling experiments.  &lt;/p&gt;&#xA;" OwnerUserId="880" LastActivityDate="2010-08-11T21:19:03.187" CommentCount="3" />
  <row Id="1560" PostTypeId="2" ParentId="1557" CreationDate="2010-08-11T21:20:47.680" Score="4" ViewCount="" Body="&lt;p&gt;James-Stein estimator is not widely used but it has inspired soft thresholding, hard thresholding which is really widely used. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Wavelet shrinkage estimation (see R package wavethresh) is used a lot in signal processing, shrunken centroid (package pamr under R) for classication is used for DNA micro array, there are a lot of examples of practical efficiency of shrinkage...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For theoretical purpose, see the section of candes's review about shrinkage estimation (p20-&gt; James stein and the section after after that one deals with soft and hard thresholding):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.161.8881&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.161.8881&amp;amp;rep=rep1&amp;amp;type=pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT from the comments:&lt;/strong&gt; why is JS shrinkage less used than Soft/hard Thresh ? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;James Stein is more difficult to manipulate (practically and theoretically) and to understand intuitively than hard thresholding but the why question is a good question! &lt;/p&gt;&#xA;" OwnerUserId="223" LastEditorUserId="223" LastEditorDisplayName="" LastEditDate="2010-09-27T05:01:01.460" LastActivityDate="2010-09-27T05:01:01.460" CommentCount="2" />
  <row Id="1561" PostTypeId="1" CreationDate="2010-08-11T21:28:10.213" Score="1" ViewCount="169" Body="&lt;p&gt;i have a different sample size &lt;code&gt;n&lt;/code&gt; for every month&lt;/p&gt;&#xA;&#xA;&lt;p&gt;for example i have &lt;code&gt;13890&lt;/code&gt; one month then &lt;code&gt;17756&lt;/code&gt;, then &lt;code&gt;21425&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;the data every month for example &lt;code&gt;13890&lt;/code&gt; is broken down into:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;48 chairs, 12 tables, 2 couches etc...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;next month we have similar metrics like&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;3 chairs, 23 tables, 4 couches etc..&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;i would like to know how i am doing with the furniture in relation to the total per month&lt;/p&gt;&#xA;" OwnerUserId="876" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-08-20T05:54:14.797" LastActivityDate="2010-08-20T05:54:14.797" Title="Normalizing data" Tags="&lt;proportion&gt;" AnswerCount="1" FavoriteCount="0" />
  <row Id="1562" PostTypeId="1" CreationDate="2010-08-11T21:49:08.720" Score="3" ViewCount="222" Body="&lt;p&gt;Hello,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a question on subject chi-square test for independence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have, for example, two events A and B. If chi square test is not passed: is A dependent on B (A|B) or B on A (B|A)? Or does be valid both? (A|B and B|A).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you in advance.&lt;/p&gt;&#xA;" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-08-11T23:36:13.873" LastActivityDate="2010-10-30T14:56:29.207" Title="What dependence is implied by a chi square test for independence?" Tags="&lt;statistical-significance&gt;&lt;chi-squared&gt;" AnswerCount="4" />
  <row Id="1563" PostTypeId="2" ParentId="124" CreationDate="2010-08-11T22:10:15.810" Score="0" ViewCount="" Body="&lt;p&gt;Naive Bayes is usually the starting point for text classification, here's an &lt;a href=&quot;http://www.drdobbs.com/184406064;jsessionid=A35OB1KFVVLTTQE1GHPCKH4ATMY32JVN&quot; rel=&quot;nofollow&quot;&gt;article&lt;/a&gt; from Dr. Dobbs on how to implement one. It's also often the ending point for text classification because it's so efficient and parallelizes well, SpamAssassin and POPFile use it.&lt;/p&gt;&#xA;" OwnerUserId="511" LastEditorUserId="511" LastEditorDisplayName="" LastEditDate="2010-08-11T22:17:23.683" LastActivityDate="2010-08-11T22:17:23.683" />
  <row Id="1564" PostTypeId="1" CreationDate="2010-08-11T22:14:39.753" Score="3" ViewCount="394" Body="&lt;p&gt;Hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could you inform me please, how can I calculate conditioned probability of several events?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;for example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P (A | B, C, D) - ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know, that:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P (A | B) = P (A intersection B) / P (B)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But, unfortunately, I can't find any formula if an event A depends on several variables. Thanks in advance.&lt;/p&gt;&#xA;" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-29T00:24:17.987" LastActivityDate="2011-04-29T00:24:17.987" Title="How can I calculate the conditional probability of several events?" Tags="&lt;conditional-probability&gt;" AnswerCount="3" FavoriteCount="1" />
  <row Id="1565" PostTypeId="2" ParentId="1562" CreationDate="2010-08-11T22:17:04.820" Score="0" ViewCount="" Body="&lt;p&gt;A chi-square test is not motivated by your description thus far.  Give many more details.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Chi-square tests for independence are used to see if the probability or counts of each kind of event in a given variable are independent of other variables.&lt;/p&gt;&#xA;" OwnerUserId="601" LastActivityDate="2010-08-11T22:17:04.820" />
  <row Id="1566" PostTypeId="2" ParentId="1485" CreationDate="2010-08-11T22:48:19.673" Score="2" ViewCount="" Body="&lt;p&gt;I'm not really sure if an implementation exists to address all your needs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For (1), you can use any of the online implementations of SVM such as Pegasos or LASVM. If you want something simpler, you may use Perceptron or Kernel Perceptron. Basically, in all these algorithms, given an already learned weight vector (say w0), you can update w0 incrementally, given a fresh set of new examples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For (2) and (3), I'm not sure if the above approaches would straightaway allow but you can probably borrow some ideas from the literature dealing with unknown classes. I'd suggest taking a look at &lt;a href=&quot;http://homepage.tudelft.nl/a9p19/papers/prl_08_reject.pdf&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="881" LastActivityDate="2010-08-11T22:48:19.673" />
  <row Id="1567" PostTypeId="2" ParentId="1561" CreationDate="2010-08-11T22:49:25.943" Score="2" ViewCount="" Body="&lt;p&gt;Do you mean you want the percentage of the n in that month that belongs to each furniture category?  If so, can't you take the N for each month and divide all of the values from that month by N?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, your first case would be:&#xA;0.0034557235 0.0008639309 0.0001439885&#xA;and your second case (where N = 17756) would be &#xA;0.0001689570 0.0012953368 0.0002252760.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or do you want a comparison of your observed values to your expected values?  If that is the case you can construct a table with furniture type as columns and months as rows.  For each cell you can take the (sum of values in the row to which it belongs) * (sum of values in the column to which it belongs) and divide by the total number of values you have.  That will give you the expected value for the cell.  If you subtract that from your initial value it will tell you how many more or less of each type of furniture you had than expected given the type of furniture it is and the month in which you made your observation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, consider this source data&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;Month  Chairs  Tables  Couches  Other   Total&#xA;1      48      12      2        13828   13890&#xA;2      3       24      4        17725   17756&#xA;Total  51      36      6        31553   31646&#xA;48 chairs, 12 tables, 2 couches&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It would be calculated like so (with the ?? marking values that still need to be calculated)...&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;Month  Chairs                 Tables  Couches  Other   Total&#xA;1      (51*13890)/31646=22.38     ??       ??     ??   13890&#xA;2      (51*17756)/31646=28.61     ??       ??     ??   17756&#xA;Total  51                         ??       ??     ??   31646&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Letting you know that in Month 1 there were 48-22.38=25.62 more chairs than expected and that in Month 2 there were 3-28.61=-25.61 more chairs than expected - but to make more sense we can flip the sign and the terminology and say there were 25.61 fewer chairs than expected.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For more details consider looking &lt;a href=&quot;http://davidmlane.com/hyperstat/B143466.html&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="196" LastEditorUserId="196" LastEditorDisplayName="" LastEditDate="2010-08-12T15:21:12.473" LastActivityDate="2010-08-12T15:21:12.473" />
  <row Id="1568" PostTypeId="2" ParentId="1564" CreationDate="2010-08-11T22:49:28.470" Score="4" ViewCount="" Body="&lt;p&gt;Take the intersection of B,C and D call it U. Then perform P(A|U).&lt;/p&gt;&#xA;" OwnerUserId="572" LastActivityDate="2010-08-11T22:49:28.470" />
  <row Id="1569" PostTypeId="2" ParentId="1028" CreationDate="2010-08-11T23:09:01.983" Score="3" ViewCount="" Body="&lt;p&gt;Suppose you are given n IID samples generated by either p or by q. You want to identify which distribution generated them. Take as null hypothesis that they were generated by q. Let a indicate probability of Type I error, mistakenly rejecting the null hypothesis, and b indicate probability of Type II error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then for large n, probability of Type I error is at least&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\exp(-n \text{KL}(p,q))$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words, for an &quot;optimal&quot; decision procedure, probability of Type I falls at most by a factor of exp(KL(p,q)) with each datapoint. Type II error falls by factor of $\exp(\text{KL}(q,p))$ at most.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For arbitrary n, a and b are related as follows&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$b \log \frac{b}{1-a}+(1-b)\log \frac{1-b}{a}  \le n \text{KL}(p,q)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$a \log \frac{a}{1-b}+(1-a)\log \frac{1-a}{b}  \le n \text{KL}(q,p)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we express the bound above as the lower bound on a in terms of b and KL and decrease b to 0, result &lt;a href=&quot;http://yaroslavvb.com/upload/lower-bounds.png&quot; rel=&quot;nofollow&quot;&gt;seems&lt;/a&gt; to approach the &quot;exp(-n KL(q,p))&quot; bound even for small n&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More details on page 10 &lt;a href=&quot;http://arxiv.org/abs/adap-org/9601001&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;, and pages 74-77 of Kullback's &quot;Information Theory and Statistics&quot; (1978).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a side note, this interpretation can be used to &lt;a href=&quot;http://www.pnas.org/content/97/21/11170.abstract&quot; rel=&quot;nofollow&quot;&gt;motivate&lt;/a&gt; Fisher Information metric, since for any pair of distributions p,q at Fisher's distance k from each other (small k) you need the same number of observations to to tell them apart&lt;/p&gt;&#xA;" OwnerUserId="511" LastEditorUserId="511" LastEditorDisplayName="" LastEditDate="2011-02-03T03:22:11.430" LastActivityDate="2011-02-03T03:22:11.430" CommentCount="4" />
  <row Id="1570" PostTypeId="2" ParentId="1562" CreationDate="2010-08-11T23:33:59.727" Score="5" ViewCount="" Body="&lt;p&gt;I assume A and B are both random variables taking discrete values and you are thinking of a chi-squared test on the two-way frequency table formed by the counts of observations on the two variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In that case, a significant result indicates both directions of dependence: A|B and B|A.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you think about Bayes' theorem, it is clear that one always implies the other:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P(A|B) = P(B|A) P(A) / P(B)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So P(A|B) = P(A) if and only if P(B|A)=P(B).&lt;/p&gt;&#xA;" OwnerUserId="159" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-08-13T06:32:21.300" LastActivityDate="2010-08-13T06:32:21.300" CommentCount="2" />
  <row Id="1571" PostTypeId="1" AcceptedAnswerId="1573" CreationDate="2010-08-11T23:47:41.980" Score="4" ViewCount="380" Body="&lt;p&gt;I am trying to recreate (in R) a frequentist hypothesis testing in Bayesian from, by calculating Bayes factors of the null (H0) and alternative (H1) models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The model is simply a simple linear regression that tries to detect a trend in global temp. data from 1995 to 2009 (&lt;a href=&quot;http://www.cru.uea.ac.uk/cru/data/temperature/hadcrut3gl.txt&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;). Therefore, H0 is no trend (i.e. slope = 0), or similary, the H0 model is a linear model with only the intercept. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I calculated the &lt;code&gt;lm()&lt;/code&gt; of both models to arrive at negative log likelihood values that are significantly different. The p-value for the H1 lm() model is 0.0877.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also calculated this in a Bayesian way by using &lt;a href=&quot;http://cran.r-project.org/web/packages/MCMCpack/index.html&quot; rel=&quot;nofollow&quot;&gt;MCMCpack&lt;/a&gt;, and I get negative log likelihood values that are &lt;strong&gt;super duper uber&lt;/strong&gt; different. Log likelihood values of 13.7 and 4.3 are about a 10000 fold difference in their likelihood ratios (where &lt;a href=&quot;http://en.wikipedia.org/wiki/Bayes_factor&quot; rel=&quot;nofollow&quot;&gt;&gt;100 is considered to be &quot;decisive&quot;&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The means and sds of the estimates are very similar, so why am I getting such different likelihood values? (particularly for the Bayesian H0 model) I feel like there is a gap in my understanding on marginal likelihoods, but I can't pinpoint the problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(MCMCpack)&#xA;&#xA;## data: http://www.cru.uea.ac.uk/cru/data/temperature/hadcrut3gl.txt&#xA;&#xA;head(hadcru, 2)&#xA;##  Year      1      2      3      4      5      6      7      8      9     10&#xA;## 1 1850 -0.691 -0.357 -0.816 -0.586 -0.385 -0.311 -0.237 -0.340 -0.510 -0.504&#xA;## 2 1851 -0.345 -0.394 -0.503 -0.480 -0.391 -0.264 -0.279 -0.175 -0.211 -0.123&#xA;##       11     12    Avg&#xA;## 1 -0.259 -0.318 -0.443&#xA;## 2 -0.141 -0.151 -0.288&#xA;&#xA;hadcru.lm &amp;lt;- lm(Avg ~ 1 + Year, data = subset(hadcru, (Year &amp;lt;= 2009 &amp;amp; Year &amp;gt;= 1995)))&#xA;hadcru.lm.zero &amp;lt;- lm(Avg ~ 1, data = subset(hadcru, (Year &amp;lt;= 2009 &amp;amp; Year &amp;gt;= 1995)))&#xA;&#xA;hadcru.mcmc &amp;lt;- MCMCregress(Avg ~ 1 + Year, data = subset(hadcru, (Year &amp;lt;= 2009 &amp;amp; Year &amp;gt;= 1995)), thin = 100, mcmc = 100000, b0 = c(-20, 0), B0 = c(.00001, .00001), marginal = &quot;Laplace&quot;)&#xA;hadcru.mcmc.zero &amp;lt;- MCMCregress(Avg ~ 1, data = subset(hadcru, (Year &amp;lt;= 2009 &amp;amp; Year &amp;gt;= 1995)), thin = 100, mcmc = 100000, b0 = c(0), B0 = c(.00001), marginal = &quot;Laplace&quot;)&#xA;&#xA;-logLik(hadcru.lm)&#xA;## 'log Lik.' -14.55338 (df=3)&#xA;-logLik(hadcru.lm.zero)&#xA;## 'log Lik.' -12.80723 (df=2)&#xA;&#xA;attr(hadcru.mcmc, &quot;logmarglike&quot;)&#xA;##           [,1]&#xA;## [1,] -13.65188&#xA;attr(hadcru.mcmc.zero, &quot;logmarglike&quot;)&#xA;##           [,1]&#xA;## [1,] -4.310564&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://www.skepticalscience.com/images/HadCRUT_1995_2009.gif&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="291" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-08-18T17:19:16.027" LastActivityDate="2010-08-18T17:19:16.027" Title="Recreating traditional null hypothesis testing with Bayesian methods" Tags="&lt;r&gt;&lt;bayesian&gt;&lt;maximum-likelihood&gt;&lt;mcmc&gt;" AnswerCount="3" />
  <row Id="1572" PostTypeId="2" ParentId="1571" CreationDate="2010-08-12T00:43:27.360" Score="1" ViewCount="" Body="&lt;p&gt;I do not know the packages you are using or their internal working but perhaps the choice of priors matter? Perhaps, you should consider using different prior structures to see how sensitive the mcmc marginal likelihoods are to your choice of priors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In particular, I suspect that the mcmc and the traditional likelihoods are likely to converge better as the priors become more diffuse. Note that in mcmc the marginal likelihoods are computed by integrating out the likelihood function with respect to the priors. Thus, I have a feeling that the 'diffuseness' of the priors may matter (could be wrong on this issue but worth checking out).&lt;/p&gt;&#xA;" LastActivityDate="2010-08-12T00:54:34.060" CommentCount="2" />
  <row Id="1573" PostTypeId="2" ParentId="1571" CreationDate="2010-08-12T01:11:05.383" Score="4" ViewCount="" Body="&lt;p&gt;When you're computing Bayes factors, the priors matter. The influence of the priors can persist even if you have a large amount of data.  When you're doing posterior inference, the effect of the prior goes away as you collect more data, but not so with Bayes factors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, you'll get faster convergence if your null and alternative priors have disjoint support.  Details &lt;a href=&quot;http://www.bepress.com/mdandersonbiostat/paper47/&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="319" LastActivityDate="2010-08-12T01:11:05.383" CommentCount="8" />
  <row Id="1576" PostTypeId="1" AcceptedAnswerId="1579" CreationDate="2010-08-12T03:46:05.987" Score="11" ViewCount="2022" Body="&lt;p&gt;It seems that a number of the statistical packages that I use seem to wrap these two concepts together. However, I'm wondering if there are different assumptions or data 'formalities' that must be true to use one over the other. A real example would be incredibly useful. &lt;/p&gt;&#xA;" OwnerUserId="776" LastActivityDate="2010-10-23T15:11:36.053" Title="What are the differences between Factor Analysis and Principal Component Analysis" Tags="&lt;pca&gt;&lt;factor-analysis&gt;" AnswerCount="7" CommentCount="0" FavoriteCount="9" />
  <row Id="1577" PostTypeId="2" ParentId="1576" CreationDate="2010-08-12T03:55:36.327" Score="3" ViewCount="" Body="&lt;p&gt;There are numerous suggested definitions on the web. Here is one from a &lt;a href=&quot;http://alumni.media.mit.edu/~tpminka/statlearn/glossary/glossary.html&quot; rel=&quot;nofollow&quot;&gt;on-line glossary on statistical learning&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Principal Component Analysis&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Constructing new features which are&#xA;  the principal components of a data&#xA;  set. The principal components are&#xA;  random variables of maximal variance&#xA;  constructed from linear combinations&#xA;  of the input features. Equivalently,&#xA;  they are the projections onto the&#xA;  principal component axes, which are&#xA;  lines that minimize the average&#xA;  squared distance to each point in the&#xA;  data set. To ensure uniqueness, all of&#xA;  the principal component axes must be&#xA;  orthogonal. PCA is a&#xA;  maximum-likelihood technique for&#xA;  linear regression in the presence of&#xA;  Gaussian noise on both inputs and&#xA;  outputs. In some cases, PCA&#xA;  corresponds to a Fourier transform,&#xA;  such as the DCT used in JPEG image&#xA;  compression. See &quot;Eigenfaces for&#xA;  recognition&quot; (Turk&amp;amp;Pentland, J&#xA;  Cognitive Neuroscience 3(1), 1991),&#xA;  Bishop, &quot;Probabilistic Principal&#xA;  Component Analysis&quot;, and &quot;Automatic&#xA;  choice of dimensionality for PCA&quot;.choice of dimensionality for PCA&quot;.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Factor analysis&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;A generalization of PCA which is based&#xA;  explicitly on maximum-likelihood. Like&#xA;  PCA, each data point is assumed to&#xA;  arise from sampling a point in a&#xA;  subspace and then perturbing it with&#xA;  full-dimensional Gaussian noise. The&#xA;  difference is that factor analysis&#xA;  allows the noise to have an arbitrary&#xA;  diagonal covariance matrix, while PCA&#xA;  assumes the noise is spherical. In&#xA;  addition to estimating the subspace,&#xA;  factor analysis estimates the noise&#xA;  covariance matrix. See &quot;The EM&#xA;  Algorithm for Mixtures of Factor&#xA;  Analyzers&quot;.choice of dimensionality&#xA;  for PCA&quot;.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="334" LastActivityDate="2010-08-12T03:55:36.327" />
  <row Id="1578" PostTypeId="2" ParentId="1576" CreationDate="2010-08-12T04:29:50.107" Score="3" ViewCount="" Body="&lt;p&gt;See my response in this thread.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://stats.stackexchange.com/questions/612/is-psychprincipal-function-still-pca-when-using-rotation&quot;&gt;http://stats.stackexchange.com/questions/612/is-psychprincipal-function-still-pca-when-using-rotation&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="485" LastActivityDate="2010-08-12T04:29:50.107" />
  <row Id="1579" PostTypeId="2" ParentId="1576" CreationDate="2010-08-12T04:44:12.307" Score="10" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Principal components analysis&lt;/strong&gt; involves extracting linear composites of observed variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Factor analysis&lt;/strong&gt; is based on a formal model predicting observed variables from theoretical latent factors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In psychology these two techniques are often applied in the construction of multi-scale tests&#xA; to determine which items load on which scales.&#xA;They typically yield similar substantive conclusions (for a discussion see Comrey (1988) Factor-Analytic Methods of Scale Development in Personality and Clinical Psychology).&#xA;This helps to explain why some statistics packages seem to bundle them together.&#xA;I have also seen situations where &quot;principal component analysis&quot; is incorrectly labelled &quot;factor analysis&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In terms of a &lt;strong&gt;simple rule of thumb&lt;/strong&gt;, I'd suggest that you:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Run factor analysis if you  assume or wish to test a theoretical model of latent factors causing observed variables.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Run principal components analysis If you want to simply reduce your correlated observed variables to a smaller set of important independent composite variables.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-08-12T04:44:12.307" CommentCount="1" />
  <row Id="1580" PostTypeId="1" AcceptedAnswerId="1582" CreationDate="2010-08-12T05:03:22.717" Score="6" ViewCount="658" Body="&lt;p&gt;Imagine you,&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Run a linear regression with four numeric predictors (IV1, ..., IV4)&lt;/li&gt;&#xA;&lt;li&gt;When only IV1 is included as a predictor the standardised beta is &lt;code&gt;+.20&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;When you also include IV2 to IV4 the sign of the standardised regression coefficient of IV1 flips to &lt;code&gt;-.25&lt;/code&gt; (i.e., it's become negative).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;This gives rise to a few questions:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;With regards to terminology, do you call this a &quot;supressor effect&quot;?&lt;/li&gt;&#xA;&lt;li&gt;What strategies would you use to explain and understand this effect?&lt;/li&gt;&#xA;&lt;li&gt;Do you have any examples of such effects in practice and how did you explain and understand these effects?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-08-13T13:40:33.383" LastActivityDate="2010-08-13T13:40:33.383" Title="Regression coefficients that flip sign after including other predictors" Tags="&lt;regression&gt;&lt;predictor&gt;" AnswerCount="3" FavoriteCount="2" />
  <row Id="1581" PostTypeId="2" ParentId="1580" CreationDate="2010-08-12T05:31:08.567" Score="6" ViewCount="" Body="&lt;p&gt;I believe effects like these are frequently caused by colinearity (see &lt;a href=&quot;http://stats.stackexchange.com/questions/1149/is-there-an-intuitive-explanation-why-multicollinearity-is-a-problem-in-linear-re&quot;&gt;this question&lt;/a&gt;). I think the book on multilevel modeling by Gelman and Hill talks about it. The problem is that &lt;code&gt;IV1&lt;/code&gt; is correlated with one or more of the other predictors, and when they are all included in the model, their estimation becomes erratic. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the coefficient flipping is due to colinearity, then it's not really interesting to report, because it's not due to the relationship between your predictors to the outcome, but really due to the relationship between predictors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I've seen suggested to resolve this problem is residualization. First, you fit a model for &lt;code&gt;IV2 ~ IV1&lt;/code&gt;, then take the residuals of that model as &lt;code&gt;rIV2&lt;/code&gt;. If all of your variables are correlated, you should really residualize all of them. You may choose do to so like this&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;rIV2 &amp;lt;- resid(IV2 ~ IV1)&#xA;rIV3 &amp;lt;- resid(IV3 ~ IV1 + rIV2)&#xA;rIV4 &amp;lt;- resid(IV4 ~ IV1 + rIV2 + rIV3)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, fit the final model with&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;DV ~ IV1 + rIV2 + rIV3 + rIV4&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, the coefficient for &lt;code&gt;rIV2&lt;/code&gt; represents the independent effect of &lt;code&gt;IV2&lt;/code&gt; given its correlation with &lt;code&gt;IV1&lt;/code&gt;. I've heard you won't get the same result if you residualized in a different order, and that choosing the residualization order is really a judgment call within your research. &lt;/p&gt;&#xA;" OwnerUserId="287" LastActivityDate="2010-08-12T05:31:08.567" CommentCount="3" />
  <row Id="1582" PostTypeId="2" ParentId="1580" CreationDate="2010-08-12T06:39:13.927" Score="5" ViewCount="" Body="&lt;p&gt;Multicollinearity is the usual suspect as JoFrhwld mentioned.  Basically, if your variables are positively correlated, then the coefficients will be negatively correlated, which can lead to a wrong sign on one of the coefficients.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One check would be to perform a principal components regression or ridge regression.  This reduces the dimensionality of the regression space, handling the multicollinearity.  You end up with biased estimates but a possibly lower MSE and corrected signs.  Whether you go with those particular results or not, it's a good diagnostic check.  If you still get sign changes, it may be theoretically interesting.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;UPDATE&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Following from the comment in John Christie's answer, this might be interesting.  Reversal in association (magnitude or direction) are examples of Simpson's Paradox, Lord's Paradox and Suppression Effects.  The differences essentially relate to the type of variable.  It's more useful to understand the underlying phenomenon rather than think in terms of a particular &quot;paradox&quot; or effect.  For a causal perspective, the paper below does a good job of explaining why and I'll quote at length their introduction and conclusion to whet your appetite.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2266743/&quot; rel=&quot;nofollow&quot;&gt;The role of causal reasoning in understanding Simpson's paradox, Lord's paradox, and the suppression effect: covariate selection in the analysis of observational studies&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Tu et al present an analysis of the equivalence of three paradoxes, concluding that all three simply reiterate the unsurprising change in the association of any two variables when a third variable is statistically controlled for. I call this unsurprising because reversal or change in magnitude is common in conditional analysis. To avoid either, we must avoid conditional analysis altogether. What is it about Simpson's and Lord's paradoxes or the suppression effect, beyond their pointing out the obvious, that attracts the intermittent and sometimes alarmist interests seen in the literature?&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;[...]&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;In conclusion, it cannot be overemphasized that although Simpson's and related paradoxes reveal the perils of using statistical criteria to guide causal analysis, they hold neither the explanations of the phenomenon they purport to depict nor the pointers on how to avoid them. The explanations and solutions lie in causal reasoning which relies on background knowledge, not statistical criteria. It is high time we stopped treating misinterpreted signs and symptoms ('paradoxes'), and got on with the business of handling the disease ('causality'). We should rightly turn our attention to the perennial problem of covariate selection for causal analysis using non-experimental data.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="251" LastEditorUserId="251" LastEditorDisplayName="" LastEditDate="2010-08-12T17:31:16.623" LastActivityDate="2010-08-12T17:31:16.623" CommentCount="3" />
  <row Id="1583" PostTypeId="1" AcceptedAnswerId="1589" CreationDate="2010-08-12T06:46:44.597" Score="2" ViewCount="110" Body="&lt;p&gt;Contrary to &lt;a href=&quot;http://stats.stackexchange.com/questions/1412/consequences-of-an-improper-link-function-in-n-alternative-forced-choice-procedur&quot;&gt;some here&lt;/a&gt;, others (e.g. &lt;a href=&quot;https://stat.ethz.ch/pipermail/r-help/2006-December/122353.html&quot; rel=&quot;nofollow&quot;&gt;Brian Ripley&lt;/a&gt;, the authors of &lt;a href=&quot;http://cran.r-project.org/web/packages/sensR/&quot; rel=&quot;nofollow&quot;&gt;sensR&lt;/a&gt;, and the authors of &lt;a href=&quot;http://cran.r-project.org/web/packages/psyphy/index.html&quot; rel=&quot;nofollow&quot;&gt;psyphy&lt;/a&gt;) appear to think that using a standard binomial link function when analyzing two alternative forced choice data in which the minimum expected proportion correct is .5 is incorrect.  However, their approach as to what the link function should be varies.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;1.The sensR library uses:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;function (mu) &#xA;{&#xA;    tres &amp;lt;- mu&#xA;    for (i in 1:length(mu)) {&#xA;        if (mu[i] &amp;gt; 0.5) &#xA;            tres[i] &amp;lt;- sqrt(2) * qnorm(mu[i])&#xA;        if (mu[i] &amp;lt;= 0.5) &#xA;            tres[i] &amp;lt;- 0&#xA;    }&#xA;    tres&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;2.The psyphy library uses:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;function (mu) &#xA;{&#xA;    m &amp;lt;- 2&#xA;    mu &amp;lt;- pmax(mu, 1/m + .Machine$double.eps)&#xA;    qlogis((m * mu - 1)/(m - 1))&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;3.&lt;a href=&quot;https://stat.ethz.ch/pipermail/r-help/2006-December/122351.html&quot; rel=&quot;nofollow&quot;&gt;Gabriel Baud-Bovy&lt;/a&gt; implicitly recommends (1+exp(x)/(1+exp(x)))/2.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The approach selected seems like it may have some consequences for the result.  Is there a &quot;correct&quot; link function to be using with these sorts of problems, or so long as the link, inverse link, mu.eta, and variance functions all agree is everything going to be all right?  Is there a single source material that provides any authoritative guidance on this issue?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Following John's advice I plotted these functions...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://psychlab2.ucr.edu/~russell/ForInternet/2AFCFunctionPlots.jpg&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The black line is a standard logistic function.  The red line is the function from sensR.  The blue line is from psyphy and the cyan line is from Gabriel Baud-Bovy, but given the oddness of the shape it provides, perhaps I misinterpreted him.  The psyphy function line looks like what I'd expect a logistic function to look like in a psychophysics 2AFC experiment.&lt;/p&gt;&#xA;" OwnerUserId="196" LastEditorUserId="196" LastEditorDisplayName="" LastEditDate="2010-08-12T15:18:27.963" LastActivityDate="2010-08-12T15:18:27.963" Title="Appropriate link function for 2AFC data?" Tags="&lt;logistic-regression&gt;&lt;logit&gt;&lt;link-function&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="1584" PostTypeId="2" ParentId="1576" CreationDate="2010-08-12T06:49:58.880" Score="4" ViewCount="" Body="&lt;p&gt;Differences between factor analysis and principal components analysis are:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;• In factor analysis there is a structured model and some assumptions. In this respect it is a statistical technique which does not apply to principal components analysis which is a purely mathematical transformation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;• The aim of principal components analysis is to explain the variance while factor analysis explains the covariance between the variables. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of the biggest reasons for the confusion between the two, has to do with the fact that one of the factor extraction methods in Factor Analysis is called &quot;method of principal components&quot;. However, it's one thing to use PCA and another thing to use the &lt;em&gt;method of principal components&lt;/em&gt; in FA. The names maybe similar, but there are significant differences. The former is an independent analytical method while the latter is merely a tool for factor extraction.&lt;/p&gt;&#xA;" OwnerUserId="339" LastActivityDate="2010-08-12T06:49:58.880" />
  <row Id="1585" PostTypeId="2" ParentId="1557" CreationDate="2010-08-12T07:27:47.233" Score="4" ViewCount="" Body="&lt;p&gt;Ridge regression is a form of shrinkage. See &lt;a href=&quot;http://www.jstor.org/pss/1268284&quot; rel=&quot;nofollow&quot;&gt;Draper &amp;amp; Van Nostrand (1979)&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Shrinkage has also proved useful in estimating seasonal factors for time series. See &lt;a href=&quot;http://www.forecasters.org/ijf/journal-issue/273/article/5847&quot; rel=&quot;nofollow&quot;&gt;Miller and Williams (IJF, 2003)&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-08-12T07:27:47.233" CommentCount="1" />
  <row Id="1586" PostTypeId="2" ParentId="1580" CreationDate="2010-08-12T07:30:47.867" Score="5" ViewCount="" Body="&lt;p&gt;See &lt;a href=&quot;http://en.wikipedia.org/wiki/Simpson%27s_paradox&quot; rel=&quot;nofollow&quot;&gt;Simpson's Paradox&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="601" LastActivityDate="2010-08-12T07:30:47.867" CommentCount="2" />
  <row Id="1587" PostTypeId="2" ParentId="1571" CreationDate="2010-08-12T07:45:05.587" Score="4" ViewCount="" Body="&lt;p&gt;I'm note sure I follow the R-code as I have only used R once or twice, but it looks to me as if you are comparing the marginal likelihood of a model with only an intercept and no slope (hadcru.mcmc.zero) and the marginal likelihood of a model with a slope and an intercept (hadcru.mcmc).  However, while hadcru.mcmc.zero seems to be the correct model for H0, hadcru.mcmc does not seem to me to correctly represent H1 as there is nothing as far as I can see that constrains the slope to be positive.  Is the something in the prior for the slope that makes it strictly positive (I don't know enough about MCMC in R to know)?  If not, that may be where your problem lies as the marginal likelihood would then have a component representing the likelihood of the data for all of then egative values of the slop permitted under the prior (and 0) as well as the positive.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is debatable whether the H0 for this question should be that the slope is exactly zero, nobody would believe that to be plausible a-priori.  Perhaps a test using the Bayes factor for a model where the slope is strictly positive (H1) against a model where it is zero or negative (H0).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;HTH (and I am not just confusing things)&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2010-08-12T07:45:05.587" CommentCount="2" />
  <row Id="1588" PostTypeId="2" ParentId="1557" CreationDate="2010-08-12T08:11:37.910" Score="3" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://strimmerlab.org/index.html&quot; rel=&quot;nofollow&quot;&gt;Korbinian Strimmer&lt;/a&gt; uses the James-Stein estimator for &lt;a href=&quot;http://jmlr.csail.mit.edu/papers/v10/hausser09a.html&quot; rel=&quot;nofollow&quot;&gt;infering gene networks&lt;/a&gt;. I've used his R packages a few times and it seems to provide a very good and quick answer.&lt;/p&gt;&#xA;" OwnerUserId="8" LastActivityDate="2010-08-12T08:11:37.910" CommentCount="1" />
  <row Id="1589" PostTypeId="2" ParentId="1583" CreationDate="2010-08-12T08:37:45.023" Score="3" ViewCount="" Body="&lt;p&gt;It doesn't just seem like it will have consequences, it will have large consequences.  Fit that second function to the data you put in your last question on this.  It goes dramatically negative as it approaches 0.5.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Perhaps more importantly, you also need to consider what the different equations mean for how one interprets the functioning of the mind.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is no known function that is just best for all 2AFC*.  Such a function would be tantamount to proving a universal law of the operations of the mind.  You have to model your data if you want the very best fit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;*OK, some models like splines will just fit most anything but you'd have to justify why you have all the extra parameters theoretically.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(ASIDE: you were opposed to clipping when difficulty achieved maximum (or minimum).  Consider, if you were modelling a robotic arm at the maximum point of travel you would just clip the results at the maximum point of travel (something your first equation does).  Just because you didn't know what that point was before you found it doesn't mean anything.  You found it when performance reached chance.)&lt;/p&gt;&#xA;" OwnerUserId="601" LastActivityDate="2010-08-12T08:37:45.023" CommentCount="3" />
  <row Id="1590" PostTypeId="1" AcceptedAnswerId="1596" CreationDate="2010-08-12T08:50:26.350" Score="-1" ViewCount="94" Body="&lt;p&gt;If one has an r value of 0.60, can one state that an increase in one variable is 60% likely to mean an increase in the other variable?&lt;/p&gt;&#xA;" OwnerUserId="888" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-08-12T08:56:54.663" LastActivityDate="2010-08-12T10:53:26.677" Title="Specific question about interpretation of r" Tags="&lt;correlation&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="1593" PostTypeId="2" ParentId="1564" CreationDate="2010-08-12T10:19:39.270" Score="2" ViewCount="" Body="&lt;p&gt;Another approach would be:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;P(A| B, C, D) = P(A, B, C, D)/P(B, C, D)&#xA;              = P(B| A, C, D).P(A, C, D)/P(B, C, D)&#xA;              = P(B| A, C, D).P(C| A, D).P(A, D)/{P(C| B, D).P(B, D)}&#xA;              = P(B| A, C, D).P(C| A, D).P(D| A).P(A)/{P(C| B, D).P(D| B).P(B)}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Note the similarity to:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;      P(A| B) = P(A, B)/P(B)&#xA;              = P(B| A).P(A)/P(B)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And there are many equivalent forms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Taking U = (B, C, D) gives:&#xA;    P(A| B, C, D) = P(A, U)/P(U)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;P(A| B, C, D) = P(A, U)/P(U)&#xA;              = P(U| A).P(A)/P(U)&#xA;              = P(B, C, D| A).P(A)/P(B, C, D)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm sure they're equivalent, but do you want the joint probability of B, C &amp;amp; D given A?&lt;/p&gt;&#xA;" OwnerUserId="521" LastEditorUserId="521" LastEditorDisplayName="" LastEditDate="2010-08-13T06:44:05.597" LastActivityDate="2010-08-13T06:44:05.597" />
  <row Id="1594" PostTypeId="2" ParentId="1383" CreationDate="2010-08-12T10:20:31.550" Score="1" ViewCount="" Body="&lt;p&gt;Interesting question!  All statistical models can be viewed as performing lossy data compression.  For instance simple linear regression with one predictor replaces $N$ points (where $N$ can be massive, e.g., in the 1000s) with two parameters: a slope and intercept.  The parameters may then be used to reconstruct the data, with degree of success depending on how good the original fit was.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your specific example concerns predicting binary time series data (Bernoulli distributed data, which is a specific case of the binomial distribution).  Binary data can encode a lot: coin flips, pictures, sounds, the digits of $\pi$, statistical programming languages...  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you can imagine, and as a quick search around Google will confirm, there are a lot of statistical models which could apply to binary data.   One is logistic regression, or (to express the same model in a more general framework) a Generalized Linear Model with a binomial distribution and a logit link function.  The function fit is of the following form:&#xA;$\mbox{logit}[P(Y)] = \beta X + \epsilon$, where $X$ (predictors), $Y$ (probability of a 1), and $\epsilon$ (residuals) are vectors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Okay.  Now a little demonstration. Suppose data are generated so that the probability of a 1 correlates with the sine of time (represented as black points in the graph below).  You don't know this, however.  You get data for time points from 0 to 359 (blue points).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://img196.imageshack.us/img196/589/cointimepredict2.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With the available data points, I fitted the function $\mbox{logit}[P(Y)] = \beta_0 + \beta_1 t + \beta_2 t^2 + \beta_3 t^3$, which popped out as $\mbox{logit}[P(Y)] = -0.2  -30.9 t  -3.1 t^2 + 22.2 t^3$.  (The probability predictions are plotted in red.)  It's a good fit to the data (between 0 and 359).  However as you can see, when extrapolating, it does a rather poor job: beyond a certain point it says &quot;just guess 1!&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Take-home message: to do the analysis correctly, you need to have a some idea of the likely processes generating the data.  If I knew a sine process were doing the job, then I'd be able to do a wonderful job predicting.  Thinking about this is where a statistician would start.  The appropriate model is always going to be domain specific, which is why, for example, compression techniques working well for images don't automatically apply to sounds.&lt;/p&gt;&#xA;" OwnerUserId="702" LastEditorUserId="702" LastEditorDisplayName="" LastEditDate="2010-08-14T11:17:52.653" LastActivityDate="2010-08-14T11:17:52.653" CommentCount="4" />
  <row Id="1595" PostTypeId="1" AcceptedAnswerId="1632" CreationDate="2010-08-12T10:46:45.407" Score="19" ViewCount="1858" Body="&lt;p&gt;Lots of people use a main tool like Excel or another spreadsheet, SPSS, STATA, or R for their statistics needs. They might turn to some specific package for very special needs, but a lot of things can be done with a simple spreadsheet or a general stats package or stats programming environment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've always liked Python as a programming language, and for simple needs, it's easy to write a short program that calculates what I need. Matplotlib allows me to plot it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Has anyone switched completely from, say R, to Python? R (or any other statistics package) has a lot of functionality specific to statistics, and it has data structures that allow you to think about the statistics you want to perform and less about the internal representation of your data. Python (or some other dynamic language) has the benefit of allowing me to program in a familiar, high-level language, and it lets me programmatically interact with real-world systems in which the data resides or from which I can take measurements. But I haven't found any Python package that would allow me to express things with &quot;statistical terminology&quot; – from simple descriptive statistics to more complicated multivariate methods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What can you recommend if I wanted to use Python as a &quot;statistics workbench&quot; to replace R, SPSS, etc.?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What would I gain and lose, based on your experience?&lt;/p&gt;&#xA;" OwnerUserId="890" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-08-12T10:47:58.527" LastActivityDate="2011-02-06T11:39:46.467" CommunityOwnedDate="2010-08-27T15:07:18.913" Title="Python as a statistics workbench" Tags="&lt;r&gt;&lt;spss&gt;&lt;stata&gt;&lt;python&gt;" AnswerCount="9" CommentCount="2" FavoriteCount="18" />
  <row Id="1596" PostTypeId="2" ParentId="1590" CreationDate="2010-08-12T10:53:26.677" Score="6" ViewCount="" Body="&lt;p&gt;No, your interpretation is incorrect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Common interpretations of a correlation between X and Y equal to .60 include:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;X explains the following proportion of variance in Y: $0.60^2=.36$ . I.e., 36% of variance. &lt;/li&gt;&#xA;&lt;li&gt;A value one standard deviation larger on X is associated with a value .60 of a standard deviation larger on Y.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Both these statements are bidirectional. I.e., you could switch the words X and Y in the above two statements and the statements would still be true.&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-08-12T10:53:26.677" />
  <row Id="1597" PostTypeId="2" ParentId="1595" CreationDate="2010-08-12T10:58:37.137" Score="7" ViewCount="" Body="&lt;p&gt;The following StackOverflow discussions might be useful&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://stackoverflow.com/questions/1177019/what-can-be-done-in-r-that-cant-be-done-with-python-numpy-scipy&quot;&gt;R versus Python&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://stackoverflow.com/questions/1944261/when-to-choose-r-vs-scipy&quot;&gt;SciPy versus R&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://stackoverflow.com/questions/2264974/psychology-researcher-wants-to-learn-new-language&quot;&gt;Psychology research choosing between R, Python, and Matlab&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-08-12T10:58:37.137" CommunityOwnedDate="2010-08-27T15:07:18.913" />
  <row Id="1598" PostTypeId="2" ParentId="1595" CreationDate="2010-08-12T10:59:17.060" Score="11" ViewCount="" Body="&lt;p&gt;I don't think there's any argument that the range of statistical packages in &lt;a href=&quot;http://cran.r-project.org/&quot; rel=&quot;nofollow&quot;&gt;cran&lt;/a&gt; and &lt;a href=&quot;http://bioconductor.org/&quot; rel=&quot;nofollow&quot;&gt;Bioconductor&lt;/a&gt; far exceed anything on offer from other languages, however, that isn't the only think to consider.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my research, I use R when I can but sometimes R is just to slow. For example, a large MCMC run. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Recently, I combined python and C to tackle this problem. Brief summary: fitting a large stochastic population model with ~60 parameters and inferring around 150 latent states using MCMC. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Read in the data in python&lt;/li&gt;&#xA;&lt;li&gt;Construct the C data structures in python using &lt;a href=&quot;http://python.net/crew/theller/ctypes/&quot; rel=&quot;nofollow&quot;&gt;ctypes&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;li&gt;Using a python &lt;code&gt;for&lt;/code&gt; loop, call the C functions that updated parameters and calculated the likelihood.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;A quick calculation showed that the programme spent 95% in C functions. However, I didn't have to write painful C code to read in data or construct C data structures.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;I know there's also &lt;a href=&quot;http://rpy.sourceforge.net/&quot; rel=&quot;nofollow&quot;&gt;rpy&lt;/a&gt;, where python can call R functions. This can be useful, but if you're &quot;just&quot; doing statistics then I would use R.&lt;/p&gt;&#xA;" OwnerUserId="8" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-08-12T13:06:09.067" LastActivityDate="2010-08-12T13:06:09.067" CommunityOwnedDate="2010-08-27T15:07:18.913" CommentCount="4" />
  <row Id="1599" PostTypeId="2" ParentId="118" CreationDate="2010-08-12T12:00:55.580" Score="2" ViewCount="" Body="&lt;p&gt;I think the contrast between using absolute deviations and squared deviations becomes clearer once you move beyond a single variable and think about linear regression. There's a nice discussion at &lt;a href=&quot;http://en.wikipedia.org/wiki/Least_absolute_deviations&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Least_absolute_deviations&lt;/a&gt;, particularly the section &quot;Contrasting Least Squares with Least Absolute Deviations&quot; , which links to some student exercises with a neat set of applets at &lt;a href=&quot;http://www.math.wpi.edu/Course_Materials/SAS/lablets/7.3/73_choices.html&quot; rel=&quot;nofollow&quot;&gt;http://www.math.wpi.edu/Course_Materials/SAS/lablets/7.3/73_choices.html&lt;/a&gt; .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To summarise, least absolute deviations is more robust to outliers than ordinary least squares, but it can be unstable (small change in even a single datum can give big change in fitted line) and doesn't always have a unique solution - there can be a whole range of fitted lines. Also least absolute deviations requires iterative methods, while ordinary least squares has a simple closed-form solution, though that's not such a big deal now as it was in the days of Gauss and Legendre, of course.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-08-12T12:00:55.580" />
  <row Id="1600" PostTypeId="2" ParentId="1595" CreationDate="2010-08-12T12:43:41.063" Score="8" ViewCount="" Body="&lt;p&gt;One benefit of moving to Python is the possibility to do more work in one language.  Python is a reasonable choice for number crunching, writing web sites, administrative scripting, etc. So if you do your statistics in Python, you wouldn't have to switch languages to do other programming tasks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Update: On January 26, 2011 Microsoft Research announced &lt;a href=&quot;http://blogs.msdn.com/b/the_blog_of_sho/archive/2011/01/26/introducing-sho.aspx&quot; rel=&quot;nofollow&quot;&gt;Sho&lt;/a&gt;, a new Python-based environment for data analysis. I haven't had a chance to try it yet, but it sounds like an interesting possibility if want to run Python and also interact with .NET libraries.&lt;/p&gt;&#xA;" OwnerUserId="319" LastEditorUserId="319" LastEditorDisplayName="" LastEditDate="2011-02-06T11:39:46.467" LastActivityDate="2011-02-06T11:39:46.467" CommunityOwnedDate="2010-08-27T15:07:18.913" CommentCount="1" />
  <row Id="1601" PostTypeId="1" AcceptedAnswerId="1607" CreationDate="2010-08-12T13:20:13.170" Score="4" ViewCount="617" Body="&lt;p&gt;In the analysis of test scores (e.g., in Education or Psychology), common analysis techniques often assume that data are normally distributed. However, perhaps more often than not, scores tend to deviate sometimes wildly from normal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am familiar with some basic normalizing transformations, like:&#xA;square roots, logarithms, reciprocal transformations for reducing positive skew,&#xA;reflected versions of the above for reducing negative skew,&#xA;squaring for leptokurtic distributions. I have heard of arcsine transformations and power transformations, though I am not really knowledgeable about them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, I am curious as to what other transformations are commonly used by analysts?&lt;/p&gt;&#xA;" OwnerUserId="445" LastActivityDate="2010-08-13T15:48:29.310" Title="Normalizing Transformation Options" Tags="&lt;normality&gt;&lt;data-transformation&gt;" AnswerCount="4" FavoriteCount="3" />
  <row Id="1602" PostTypeId="2" ParentId="22" CreationDate="2010-08-12T14:53:36.410" Score="10" ViewCount="" Body="&lt;p&gt;Tongue firmly in cheek:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A Bayesian defines a &quot;probability&quot; in exactly the same way that most non-statisticians do - namely an indication of the plausibility of a proposition or a situation.  If you ask him a question, he will give you a direct answer assigning probabilities describing the plausibilities of the possible outcomes for the particular situation (and state his prior assumptions).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A Frequentist is someone that believes probabilies represent long run frequencies with which events ocurr; if needs be, he will invent a fictitious population from which your particular situation could be considered a random sample so that he can meaningfully talk about long run frequencies.  If you ask him a question about a particular situation, he will not give a direct answer, but instead make a statement about this (possibly imaginary) population.  Many non-frequentists statisticians will be easily confused by the answer and interpret it as Bayesian probability about the particular situation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, it is important to note that most Frequentist methods have a Bayesian equivalent that in most circumstances will give essentially the same result, the difference is largely a matter of philosophy, and in practice it is a matter of &quot;horses for courses&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you may have guessed, I am a Bayesian and an engineer. ;o)&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2010-08-12T14:53:36.410" CommentCount="1" />
  <row Id="1603" PostTypeId="2" ParentId="1249" CreationDate="2010-08-12T15:11:07.163" Score="1" ViewCount="" Body="&lt;p&gt;This answer is inspired by shabbychef's answer using the median. By definition:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$E[exp(Z^2)] = \sum_{z=1}^{z=n} exp(z^2) P(z;n,n^{-\beta})$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;where,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(z;n,n^{-\beta})$ is the binomial probability.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Denote the &lt;a href=&quot;http://en.wikipedia.org/wiki/Binomial_distribution#Mode_and_median&quot; rel=&quot;nofollow&quot;&gt;mode of this binomial distribution&lt;/a&gt; by: $m(n,n^{-\beta})$. Thus, by definition we have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(z;n,n^{-\beta}) \le P(m(n,n^{-\beta});n,n^{-\beta}) \ \ \forall z$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\bar{P} = P(m(n,n^{-\beta});n,n^{-\beta})$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$E[exp(Z^2] \le \sum_{z=1}^{z=n} exp(z^2) \bar{P}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This upper bound is a function of $n$ and $\beta$ as desired. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hopefully, this in the right track unlike my previous attempt.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strike&gt;This approach is technically not ok as $Z$ is a discrete variable but can be justified if we take the normal approximation to the binomial.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not sure to what extent this is a better bound then the trivial bound but here is one approach. Take the &lt;a href=&quot;http://en.wikipedia.org/wiki/Taylor_series&quot; rel=&quot;nofollow&quot;&gt;taylor series expansion&lt;/a&gt; of $exp(z^2)$ and ignoring terms higher than the second term, you get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\int e^{z^2} f(z) dz &amp;lt; \int (1 + z^2) f(z) dz$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\int (1 + z^2) f(z) dz = 1 + \int z^2 f(z) dz$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But, we know that:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\int z^2 f(z) dz = Var(z) + E(z)^2$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Substituting for the variance and mean of the &lt;a href=&quot;http://en.wikipedia.org/wiki/Binomial_distribution&quot; rel=&quot;nofollow&quot;&gt;binomial distribution&lt;/a&gt; and simplifying, we get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\int e^{z^2} f(z) dz &amp;lt; 1 +  n^{1-\beta} (1-n^{-\beta} + n^{1-\beta})$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: Please check the math as I corrected one error.&#xA;&lt;/strike&gt;&lt;/p&gt;&#xA;" LastActivityDate="2010-08-12T17:08:46.207" CommentCount="3" />
  <row Id="1604" PostTypeId="1" AcceptedAnswerId="1609" CreationDate="2010-08-12T15:16:55.543" Score="4" ViewCount="142" Body="&lt;p&gt;Can someone provide me with a book or online reference on how to construct smoothing splines with cross-validation? I have a programming and undergraduate level mathematics background.  I would also appreciate an overview of whether this is smoothing technique is a good one for smoothing data and whether there are any disadvantages of which a non-statistician needs to be aware.&lt;/p&gt;&#xA;" OwnerUserId="847" LastEditorUserId="847" LastEditorDisplayName="" LastEditDate="2010-08-12T20:57:22.143" LastActivityDate="2010-08-13T02:46:22.033" Title="Constructing smoothing splines with cross-validation" Tags="&lt;cross-validation&gt;&lt;smoothing&gt;&lt;splines&gt;" AnswerCount="1" />
  <row Id="1605" PostTypeId="2" ParentId="1601" CreationDate="2010-08-12T16:03:50.133" Score="0" ViewCount="" Body="&lt;p&gt;A simple option is to use sums of scores instead of the scores themselves. The sum of distributions tends to normality. For example, in Education you could add a student's scores over a series of tests.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another option, of course, is to use techniques that do not assume normality, which are underestimated and underused.&lt;/p&gt;&#xA;" OwnerUserId="666" LastActivityDate="2010-08-12T16:03:50.133" CommentCount="2" />
  <row Id="1607" PostTypeId="2" ParentId="1601" CreationDate="2010-08-12T16:54:18.413" Score="2" ViewCount="" Body="&lt;p&gt;The &lt;a href=&quot;http://en.wikipedia.org/wiki/Box-Cox_transformation&quot; rel=&quot;nofollow&quot;&gt;Box-Cox&lt;/a&gt; transformation includes many of the ones you cited.  See this answer for some details:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://stats.stackexchange.com/questions/1444/how-should-i-transform-non-negative-data-including-zeros/1452#1452&quot;&gt;How should I transform non-negative data including zeros?&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;UPDATE: These &lt;a href=&quot;http://www.stat.uconn.edu/~studentjournal/index_files/pengfi_s05.pdf&quot; rel=&quot;nofollow&quot;&gt;slides&lt;/a&gt; provide a pretty good overview of Box-Cox transformations.&lt;/p&gt;&#xA;" OwnerUserId="251" LastEditorUserId="251" LastEditorDisplayName="" LastEditDate="2010-08-13T05:49:53.540" LastActivityDate="2010-08-13T05:49:53.540" CommentCount="2" />
  <row Id="1608" PostTypeId="2" ParentId="1555" CreationDate="2010-08-12T17:54:22.010" Score="1" ViewCount="" Body="&lt;p&gt;Based on your additional explanation in the comments, it appears that you have 8 groups (each corresponding to a column) and a continuous outcome variable that you grouped into 10 bins (each bin corresponding to a row). Note that it also implies that the rows are ordered with later rows implying larger values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First of all, if you do have the underlying continuous variable, then do not bin it - just use Kruskall-Wallis or ANOVA to compare the groups.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming that the binning is unavoidable, you can still use a Kruskall-Wallis test, but not on the frequencies as you have apparently done it. Your current KW inference just tells you that you have &lt;em&gt;more data&lt;/em&gt; in some groups as compared to others. The actual observations in this case are the row numbers (1 through 10), and the values in the table are just the frequencies of occurrences. Most statistical software has an option of specifying these as &quot;weights&quot; or &quot;frequencies&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The chi-square test can be used on the frequencies, however if the rows are ordered it might have much lower power compared to the Kruskall-Wallis test to actually detect differences, since it completely ignores the ordering of the rows. Thus even though its results are valid, I would not recommend using these due to the loss of power.&lt;/p&gt;&#xA;" OwnerUserId="279" LastActivityDate="2010-08-12T17:54:22.010" CommentCount="2" />
  <row Id="1609" PostTypeId="2" ParentId="1604" CreationDate="2010-08-12T19:52:35.463" Score="5" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0824793374&quot; rel=&quot;nofollow&quot;&gt;&lt;em&gt;Nonparametric Regression and Spline Smoothing&lt;/em&gt;&lt;/a&gt; by Eubank is a good book.  You probably want to start with Chapters 2 and 5 which cover goodness of fit and the theory and construction of smoothing splines.  I've heard good things about &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1584884746&quot; rel=&quot;nofollow&quot;&gt;&lt;em&gt;Generalized Additive Models: An Introduction with R&lt;/em&gt;&lt;/a&gt;, which might be better if you're looking for examples in R.  For a quick introduction, a google search turns up a course on &lt;a href=&quot;http://www.stat.osu.edu/~yklee/763/note.html&quot; rel=&quot;nofollow&quot;&gt;Nonparametric function estimation&lt;/a&gt; where you can peruse the slides and see examples in R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The general problem with splines is overfitting your data, but this is where cross validation comes in.  &lt;/p&gt;&#xA;" OwnerUserId="251" LastEditorUserId="251" LastEditorDisplayName="" LastEditDate="2010-08-13T02:46:22.033" LastActivityDate="2010-08-13T02:46:22.033" CommentCount="2" />
  <row Id="1610" PostTypeId="1" AcceptedAnswerId="1616" CreationDate="2010-08-12T19:55:02.167" Score="12" ViewCount="802" Body="&lt;p&gt;I'm not a statistician by education, I'm a software engineer. Yet statistics comes up a lot. In fact, questions specifically about Type I and Type II error are coming up a lot in the course of my studying for the Certified Software Development Associate exam (mathematics and statistics are 10% of the exam). I'm having trouble always coming up with the right definitions for Type I and Type II error - although I'm memorizing them now (and can remember them most of the time), I really don't want to freeze up on this exam trying to remember what the difference is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know that Type I Error is a false positive, or when you reject the null hypothesis and it's actually true and a Type II error is a false negative, or when you accept the null hypothesis and it's actually false.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there an easy way to remember what the difference is, such as a mnemonic? How do professional statisticians do it - is it just something that they know from using or discussing it often?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Side Note: This question can probably use some better tags. One that I wanted to create was &quot;terminology&quot;, but I don't have enough reputation to do it. If someone could add that, it would be great. Thanks.)&lt;/p&gt;&#xA;" OwnerUserId="110" LastActivityDate="2011-04-11T14:31:06.300" Title="Is there a way to remember the definitions of Type I and Type II Errors?" Tags="&lt;type-i-errors&gt;&lt;type-ii-errors&gt;&lt;terminology&gt;" AnswerCount="12" CommentCount="10" FavoriteCount="2" />
  <row Id="1611" PostTypeId="1" CreationDate="2010-08-12T20:09:33.550" Score="12" ViewCount="528" Body="&lt;p&gt;As an outsider, it appears that there are two competing views on how one should perform statistical inference. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are the two different methods both considered valid by working statisticians? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is choosing one considered more of a philosophical question? Or is the current situation considered problematic and attempts are being made to somehow unify the different approaches?&lt;/p&gt;&#xA;" OwnerUserId="572" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-06T19:12:41.113" LastActivityDate="2010-10-08T07:17:49.240" CommunityOwnedDate="2010-08-12T20:20:11.653" Title="Do working statisticians care about the difference between frequentist and bayesian inference?" Tags="&lt;bayesian&gt;&lt;frequentist&gt;" AnswerCount="5" CommentCount="0" FavoriteCount="2" />
  <row Id="1612" PostTypeId="2" ParentId="1610" CreationDate="2010-08-12T20:10:13.347" Score="2" ViewCount="" Body="&lt;p&gt;I used to think of it in terms of the usual &lt;a href=&quot;http://intuitor.com/statistics/T1T2Errors.html&quot; rel=&quot;nofollow&quot;&gt;picture&lt;/a&gt; of two Normal distributions (or bell curves).  Going left to right, distribution 1 is the Null, and the distribution 2 is the Alternative.  Type I (erroneously) rejects the first (Null) and Type II &quot;rejects&quot; the second (Alternative).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Now you just need to remember that you're not actually rejecting the alternative, but erroneously accepting (or failing to reject) the Null  --  i.e. restate everything in the form of the Null.  Hey, it worked for me!)&lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-08-12T20:10:13.347" CommentCount="1" />
  <row Id="1613" PostTypeId="2" ParentId="1611" CreationDate="2010-08-12T20:25:12.937" Score="1" ViewCount="" Body="&lt;p&gt;While this is subjective, I would say:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is called the bayesian/frequentist &quot;&lt;strong&gt;debate&lt;/strong&gt;&quot; for a reason.  There is a clear philosophical difference between the two approaches.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But as with most things, it's a spectrum.  Some people are very much in one camp or the other and completely reject the alternative.  Most people probably fall somewhere in the middle.  I myself would use either method depending on the circumstances.&lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-08-12T20:25:12.937" CommunityOwnedDate="2010-08-12T20:25:12.937" />
  <row Id="1614" PostTypeId="2" ParentId="1610" CreationDate="2010-08-12T20:27:49.107" Score="6" ViewCount="" Body="&lt;p&gt;Here's a handy way that happens to have some truth to it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Young scientists commit Type-I because they want to find effects and jump the gun while old scientist commit Type-II because they refuse to change their beliefs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(someone comment in a funnier version of that :) )&lt;/p&gt;&#xA;" OwnerUserId="601" LastActivityDate="2010-08-12T20:27:49.107" CommentCount="2" />
  <row Id="1615" PostTypeId="2" ParentId="1611" CreationDate="2010-08-12T20:51:36.930" Score="4" ViewCount="" Body="&lt;p&gt;Adding to what Shane says, I think the continuum comprises:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Firm philosophical standing in the Bayes camp&lt;/li&gt;&#xA;&lt;li&gt;Both considered valid, with one approach more or less preferable for a given problem&lt;/li&gt;&#xA;&lt;li&gt;I'd use a Bayesian approach (at all or more often) but I don't have the time.&lt;/li&gt;&#xA;&lt;li&gt;Firm philosophical standing in the frequentist camp&lt;/li&gt;&#xA;&lt;li&gt;I do it like I learned in class. What's Bayes?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;And yes, I know working statisticians and analysts at all of these points. Most of the time I'm living at #3, striving to spend more time at #2.&lt;/p&gt;&#xA;" OwnerUserId="394" LastActivityDate="2010-08-12T20:51:36.930" CommunityOwnedDate="2010-08-12T20:51:36.930" />
  <row Id="1616" PostTypeId="2" ParentId="1610" CreationDate="2010-08-12T20:52:04.730" Score="21" ViewCount="" Body="&lt;p&gt;Since type two means &quot;False negative&quot; or sort of &quot;false false&quot;, I remember it as the number of falses. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Type I: &quot;I &lt;strong&gt;falsely&lt;/strong&gt; think hypothesis is true&quot; (one false)&lt;/li&gt;&#xA;&lt;li&gt;Type II: &quot;I &lt;strong&gt;falsely&lt;/strong&gt; think hypothesis is &lt;strong&gt;false&lt;/strong&gt;&quot; (two falses)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="900" LastActivityDate="2010-08-12T20:52:04.730" CommentCount="6" />
  <row Id="1617" PostTypeId="2" ParentId="1610" CreationDate="2010-08-12T21:21:19.430" Score="1" ViewCount="" Body="&lt;p&gt;I remember it by thinking: What's the first thing I do when I do a null-hypothesis significance test? I set the criterion for the probability that I will make a false rejection. Thus, type 1 is this criterion and type 2 is the other probability of interest: the probability that I will fail to reject the null when the null is false. So, 1=first probability I set, 2=the other one.&lt;/p&gt;&#xA;" OwnerUserId="364" LastActivityDate="2010-08-12T21:21:19.430" />
  <row Id="1618" PostTypeId="2" ParentId="1611" CreationDate="2010-08-12T21:48:24.977" Score="2" ViewCount="" Body="&lt;p&gt;I would imagine that in applied fields the divide is not paid that much attention as researchers/practitioners tend to be pragmatic in applied works. You choose the tool that works given the context. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the debate is alive and well among those who care about the philosophical issues underlying these two approaches. See for example the following blog posts of &lt;a href=&quot;http://stat.columbia.edu/~gelman&quot; rel=&quot;nofollow&quot;&gt;Andrew Gelman&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.stat.columbia.edu/~cook/movabletype/archives/2010/06/hey_dude_ya_don.html&quot; rel=&quot;nofollow&quot;&gt;Ya don't know Bayes, Jack&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.stat.columbia.edu/~cook/movabletype/archives/2010/06/philosophy_and.html&quot; rel=&quot;nofollow&quot;&gt;Philosophy and the practice of Bayesian statistics&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" LastActivityDate="2010-08-12T21:48:24.977" CommunityOwnedDate="2010-08-12T21:48:24.977" CommentCount="1" />
  <row Id="1619" PostTypeId="2" ParentId="1610" CreationDate="2010-08-12T23:02:11.910" Score="2" ViewCount="" Body="&lt;p&gt;I use the &quot;judicial&quot; approach for remembering the difference between type I and type II: a judge committing a type I error sends an innocent man to jail, while a judge committing a type II error lets a guilty man walk free.&lt;/p&gt;&#xA;" OwnerUserId="830" LastActivityDate="2010-08-12T23:02:11.910" CommentCount="3" />
  <row Id="1620" PostTypeId="2" ParentId="1610" CreationDate="2010-08-12T23:38:47.220" Score="6" ViewCount="" Body="&lt;p&gt;I was talking to a friend of mine about this and he kicked me a link to &lt;a href=&quot;http://en.wikipedia.org/wiki/Type_I_and_type_II_errors&quot; rel=&quot;nofollow&quot;&gt;the Wikipedia article on type I and type II errors&lt;/a&gt;, where they apparently now provide a (somewhat unhelpful, in my opinion) mnemonic. I did, however, want to add it here just for the sake of completion. Although I didn't think it helped me, it might help someone else:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;For those experiencing difficulty&#xA;  correctly identifying the two error&#xA;  types, the following mnemonic is based&#xA;  on the fact that (a) an &quot;error&quot; is&#xA;  false, and (b) the Initial letters of&#xA;  &quot;Positive&quot; and &quot;Negative&quot; are written&#xA;  with a different number of vertical&#xA;  lines:&lt;/p&gt;&#xA;  &#xA;  &lt;ul&gt;&#xA;  &lt;li&gt;A Type I error is a false POSITIVE; and P has a single vertical line.&lt;/li&gt;&#xA;  &lt;li&gt;A Type II error is a false NEGATIVE; and N has two vertical lines.&lt;/li&gt;&#xA;  &lt;/ul&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;With this, you need to remember that a false positive means rejecting a true null hypothesis and a false negative is failing to reject a false null hypothesis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is by no means the best answer here, but I did want to throw it out there in the event someone finds this question and this can help them.&lt;/p&gt;&#xA;" OwnerUserId="110" LastActivityDate="2010-08-12T23:38:47.220" />
  <row Id="1621" PostTypeId="1" CreationDate="2010-08-12T23:53:12.230" Score="4" ViewCount="177" Body="&lt;p&gt;I came across a &lt;a href=&quot;http://www.citeulike.org/user/yaroslavvb/tag/information-geometry&quot; rel=&quot;nofollow&quot;&gt;large&lt;/a&gt; body of literature which advocates using Fisher's Information metric as a natural local metric in the space of probability distributions and then integrating over it to define distances and volumes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But are these &quot;integrated&quot; quantities actually useful for anything? I found no theoretical justifications and very few practical applications. One is Guy Lebanon's &lt;a href=&quot;http://www.cs.cmu.edu/~lafferty/pub/hyperplane.pdf&quot; rel=&quot;nofollow&quot;&gt; work&lt;/a&gt; where he uses &quot;Fisher's distance&quot; to classify documents and another one is Rodriguez' &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.68.9259&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;ABC of Model Selection…&lt;/a&gt; where &quot;Fisher's volume&quot; is used for model selection. Apparently, using &quot;information volume&quot; gives &quot;orders of magnitude&quot; improvement over AIC and BIC for model selection, but I haven't seen any follow up on that work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A theoretical justification might be to have a generalization bound which uses this measure of distance or volume and is better than bounds derived from MDL or asymptotic arguments, or a method relying on one of these quantities that's provably better in some reasonably practical situation, are there any results of this kind?&lt;/p&gt;&#xA;" OwnerUserId="511" LastEditorUserId="511" LastEditorDisplayName="" LastEditDate="2010-08-16T00:49:12.953" LastActivityDate="2010-10-17T20:12:38.687" Title="Using information geometry to define distances and volumes…useful?" Tags="&lt;model-selection&gt;&lt;information-geometry&gt;" AnswerCount="2" CommentCount="3" FavoriteCount="2" />
  <row Id="1622" PostTypeId="1" AcceptedAnswerId="1635" CreationDate="2010-08-13T00:18:34.380" Score="1" ViewCount="91" Body="&lt;p&gt;One estimate of the 'quality' of a portfolio of stocks is the &lt;em&gt;Sharpe ratio&lt;/em&gt;, which is defined as the mean of the returns divided by the standard deviation of the returns (modulo adjustments for risk free rate, etc). The sample Sharpe ratio is the sample mean divided by the sample standard deviation. Up to a constant factor ($\sqrt{n}$, where $n$ is the number of observations), this is distributed as a (possibly non-central) $t$-statistic. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there known techniques for comparing &lt;strike&gt;the mean of&lt;/strike&gt; independent variables distributed as non-central $t$-statistics? &lt;strike&gt;Of course, there are non-parametric tests of mean, but is there something specific to the case of noncentral $t$?&lt;/strike&gt; (I'm not sure what I meant by that.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;edit&lt;/strong&gt;: the original question is somewhat ambiguous (well, it's actually not what I want). Is there a way to test the null hypothesis: population Sharpe ratio of $X$ equals population Sharpe ratio of $Y$, given independent collections of observations drawn from $X$ and $Y$? Here Sharpe ratio is mean divided by standard deviation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;edit&lt;/strong&gt;: given $n_x, n_y$ observations of $X, Y$, construct sample means, standard deviations, to get sample Sharpe ratios: $\hat{S}_x = \frac{\hat{\mu}_x}{\hat{\sigma}_x}, \hat{S}_y = \frac{\hat{\mu}_y}{\hat{\sigma}_y}$. Then $t_x = \sqrt{n_x}\hat{S}_x$, and $t_y = \sqrt{n_y}\hat{S}_y$ are distributed as non-central t-statistics with noncentrality parameters $\sqrt{n_x}S_x$ and $\sqrt{n_y}S_y$, where $S_x, S_y$ are the population Sharpe ratios of $X, Y$. Given these independent observations, I wish to test the null hypothesis $H_0: S_x = S_y$. In one form of the problem, one only has the summary statistics $n_x, n_y, \hat{\mu}_x, \hat{\mu}_y, \hat{\sigma}_x, \hat{\sigma}_y$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For large sample sizes, $t_x, t_y$ are approximately normal, I believe but the small sample size case is also of interest (funds often quote performance based on monthly returns).&lt;/p&gt;&#xA;" OwnerUserId="795" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-30T21:20:25.937" LastActivityDate="2010-09-30T21:20:25.937" Title="Comparing 2 independent non-central t statistics" Tags="&lt;finance&gt;&lt;test&gt;&lt;mean&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="1623" PostTypeId="2" ParentId="507" CreationDate="2010-08-13T00:37:26.680" Score="2" ViewCount="" Body="&lt;p&gt;Rather than using the Gelman-Rubin statistic, which is a nice aid but not perfect (as with all convergence diagnostics), I simply use the same idea and plot the results for a visual graphical assessment. In almost all cases I have considered (which is a very large number), graphing the trace plots of multiple MCMC chains started from widely varied starting positions is sufficient to show or assess whether the same posterior is being converged to or not, in each case. I use this method to: 1. Whether the MCMC chain (ever) converges; 2.Assess how long I should set the burn-in period; and 3. to calculate Gelman's R statistic (see Gelman, Carlin, Stern and Rubin, Bayesian Data Analysis) to measure the efficiency and speed of mixing in the MCMC sampler. Efficiency and convergence are slightly different issues: e.g. you can have convergence with very low efficiency (i.e. thus requiring long chains to converge). I have used this graphical method to successfully diagnose (and later correct) lack of convergence problems in specific and general situations.&lt;/p&gt;&#xA;" LastActivityDate="2010-08-13T00:37:26.680" />
  <row Id="1624" PostTypeId="1" AcceptedAnswerId="1633" CreationDate="2010-08-13T00:48:15.973" Score="3" ViewCount="162" Body="&lt;p&gt;How are the values in a z-table computed? Is it possible to compute confidence without looking up a z-table?&lt;/p&gt;&#xA;" OwnerUserId="850" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-08-13T08:35:39.487" LastActivityDate="2010-08-14T03:56:28.753" Title="Based on z-score, is it possible to compute confidence without looking at a z-table?" Tags="&lt;confidence-interval&gt;&lt;z-statistic&gt;" AnswerCount="4" />
  <row Id="1625" PostTypeId="2" ParentId="1557" CreationDate="2010-08-13T01:08:39.567" Score="2" ViewCount="" Body="&lt;p&gt;See also &lt;a href=&quot;http://www.jstor.org/pss/1270447&quot; rel=&quot;nofollow&quot;&gt;Jennrich, RJ, Oman, SD &quot;How much does Stein estimation help in multiple linear regression?&quot; &lt;em&gt;Technometrics&lt;/em&gt;, &lt;strong&gt;28&lt;/strong&gt;, 113-121, 1986.&lt;/a&gt;&lt;/p&gt;&#xA;" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-08-13T01:26:17.920" LastActivityDate="2010-08-13T01:26:17.920" />
  <row Id="1626" PostTypeId="2" ParentId="1624" CreationDate="2010-08-13T01:10:05.260" Score="0" ViewCount="" Body="&lt;p&gt;I don't know how to post formulas here, so search google on &quot;normal distribution function&quot; or see, e.g., &lt;a href=&quot;http://www.rapidtables.com/math/probability/normal_distribution.htm#normal&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="394" LastActivityDate="2010-08-13T01:10:05.260" />
  <row Id="1627" PostTypeId="2" ParentId="396" CreationDate="2010-08-13T01:30:32.060" Score="4" ViewCount="" Body="&lt;p&gt;Take a look at the R graphics library, ggplot2. Details are at the web page &lt;a href=&quot;http://had.co.nz/ggplot2/&quot; rel=&quot;nofollow&quot;&gt;http://had.co.nz/ggplot2/&lt;/a&gt; This package generates very good default plots, that follow the Tufte principles, Cleveland's guidelines and Ihaka's color package.&lt;/p&gt;&#xA;" LastActivityDate="2010-08-13T01:30:32.060" CommunityOwnedDate="2011-06-01T03:42:00.250" />
  <row Id="1628" PostTypeId="2" ParentId="1610" CreationDate="2010-08-13T01:38:42.727" Score="4" ViewCount="" Body="&lt;p&gt;Based on the principle of &lt;a href=&quot;http://en.wikipedia.org/wiki/Occam%27s_razor&quot; rel=&quot;nofollow&quot;&gt;Occam's razor&lt;/a&gt;, Type I errors (rejecting the null hypothesis when it is true) are &lt;strong&gt;&quot;arguably&quot;&lt;/strong&gt; worse than Type II errors (not rejecting the null hypothesis when it is false).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you believe such an argument:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Type I errors are of primary concern&lt;/li&gt;&#xA;&lt;li&gt;Type II errors are of secondary concern&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Note: I'm not endorsing this value judgement, but it does help me remember Type I from Type II.&lt;/p&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-08-13T01:48:32.673" LastActivityDate="2010-08-13T01:48:32.673" />
  <row Id="1629" PostTypeId="2" ParentId="1601" CreationDate="2010-08-13T02:18:09.137" Score="3" ViewCount="" Body="&lt;p&gt;The &lt;strong&gt;first step&lt;/strong&gt; should be to &lt;strong&gt;ask why&lt;/strong&gt; your variables are non-normally distributed. This can be illuminating. &#xA;Common findings from my experience:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Ability tests (e.g., exams, intelligence tests, admission tests) tend to be negatively skewed when there are ceiling effects and positively skewed when there are floor effects. Both findings suggest that the difficulty level of the test is not optimised for the sample, either being too easy or too difficult to optimally differentiate ability. It also implies that the latent variable of interest could still be normally distributed, but that the structure of the test is inducing a skew in the measured variable.&lt;/li&gt;&#xA;&lt;li&gt;Ability tests often have outliers in terms of low scorers. In short there are many ways to do poorly on a test. In particular this can sometimes be seen on exams where there are a small percentage of students where some combination of lack of aptitude and lack of effort have combined to create very low test scores. This implies that the latent variable of interest probably has a few outliers.&lt;/li&gt;&#xA;&lt;li&gt;In relation to self-report tests (e.g., personality, attitude tests, etc.) skew often occurs when the sample is inherently high on the scale (e.g., distributions of life satisfaction are negatively skewed because most people are satisfied) or when the scale has been optimised for a sample different to the one the test is being applied to (e.g., applying a clinical measure of depression to a non-clinical sample).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;This first step may suggest design modifications to the test. If you are aware of these issues ahead of time, you can even design your test to avoid them, if you see them as problematic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;strong&gt;second step&lt;/strong&gt; is to &lt;strong&gt;decide what to do&lt;/strong&gt; in the situation where you have non-normal data.&#xA;Note transformations are but one possible strategy.&#xA;I'd reiterate the general advice from a &lt;a href=&quot;http://stats.stackexchange.com/questions/1386/a-robust-t-test-for-the-mean/1391#1391&quot;&gt;previous answer regarding non-normality&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Many procedures that assume normality of residuals are &lt;strong&gt;robust&lt;/strong&gt; to modest violations of normality of residuals&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Bootstrapping&lt;/strong&gt; is generally a good strategy&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Transformations&lt;/strong&gt; are another good strategy. Note that from my experience the kinds of mild skew that commonly occur with ability and self-report psychological tests can usually be fairly readily transformed to a  distribution approximating normality using a log, sqrt, or inverse transformation (or the reversed equivalent).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-08-13T02:18:09.137" />
  <row Id="1630" PostTypeId="2" ParentId="1444" CreationDate="2010-08-13T04:27:15.563" Score="7" ViewCount="" Body="&lt;p&gt;No-one mentioned the inverse hyperbolic sine transformation. So for completeness I'm adding it here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is an alternative to the Box-Cox transformations and is defined by&#xA;\begin{equation}&#xA;f(y,\theta) = \text{sinh}^{-1}(\theta y)/\theta = \log[\theta y + (\theta^2y^2+1)^{1/2}]/\theta,&#xA;\end{equation}&#xA;where $\theta&gt;0$. For any value of $\theta$, zero maps to zero. There is also a two parameter version allowing a shift, just as with the two-parameter BC transformation. &lt;a href=&quot;http://www.jstor.org/pss/2288929&quot; rel=&quot;nofollow&quot;&gt;Burbidge, Magee and Robb (1988)&lt;/a&gt; discuss the IHS transformation including estimation of $\theta$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The IHS transformation works with data defined on the whole real line including negative values and zeros. For large values of $y$ it behaves like a log transformation, regardless of the value of $\theta$ (except 0). The limiting case as $\theta\rightarrow0$ gives $f(y,\theta)\rightarrow y$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It looks to me like the IHS transformation should be a lot better known than it is.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-08-13T04:27:15.563" />
  <row Id="1631" PostTypeId="2" ParentId="1624" CreationDate="2010-08-13T05:17:37.523" Score="3" ViewCount="" Body="&lt;p&gt;A z-table gives you values of the cumulative distribution function for the standard (i.e., mean = 0, standard deviation = 1) normal distribution. The integral needs to be estimated numerically. See &lt;a href=&quot;http://en.wikipedia.org/wiki/Normal_distribution#Cumulative_distribution_function&quot; rel=&quot;nofollow&quot;&gt;here for a discussion&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, to answer your question literally, you could use some other lookup system other than a table, such as the &lt;code&gt;pnorm&lt;/code&gt; &lt;a href=&quot;http://stat.ethz.ch/R-manual/R-devel/library/stats/html/Normal.html&quot; rel=&quot;nofollow&quot;&gt;function&lt;/a&gt; in R.&#xA;Example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; pnorm(2)&#xA;[1] 0.9772499&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Help for &lt;code&gt;pnorm&lt;/code&gt; also provides the following reference on how to calculate the cdf of a normal distribution:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Cody, W. D. (1993) Algorithm 715: SPECFUN – A portable FORTRAN package of special function routines and test drivers. ACM Transactions on Mathematical Software 19, 22–32.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-08-13T08:22:26.757" LastActivityDate="2010-08-13T08:22:26.757" />
  <row Id="1632" PostTypeId="2" ParentId="1595" CreationDate="2010-08-13T05:30:09.033" Score="18" ViewCount="" Body="&lt;p&gt;It's hard to ignore the wealth of statistical packages available in R/CRAN.  That said, I spend a &lt;em&gt;lot&lt;/em&gt; of time in Python land and would never dissuade anyone from having as much fun as I do.  :)  Here are some libraries/links you might find useful for statistical work.  &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://scipy.org&quot; rel=&quot;nofollow&quot;&gt;NumPy/Scipy&lt;/a&gt; You probably know about these already.  But let me point out the &lt;a href=&quot;http://scipy.org/Cookbook&quot; rel=&quot;nofollow&quot;&gt;Cookbook&lt;/a&gt; where you can read about many statistical facilities already available and the &lt;a href=&quot;http://www.scipy.org/Numpy_Example_List&quot; rel=&quot;nofollow&quot;&gt;Example List&lt;/a&gt; which is a great reference for functions (including data manipulation and other operations).  Another handy reference is John Cook's &lt;a href=&quot;http://www.johndcook.com/distributions_scipy.html&quot; rel=&quot;nofollow&quot;&gt;Distributions in Scipy&lt;/a&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://code.google.com/p/pandas/&quot; rel=&quot;nofollow&quot;&gt;pandas&lt;/a&gt; This is a really nice library for working with statistical data -- tabular data, time series, panel data.  Includes many builtin functions for data summaries, grouping/aggregation, pivoting.  Also has a statistics/econometrics library.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://pypi.python.org/pypi/la/0.1.0&quot; rel=&quot;nofollow&quot;&gt;larry&lt;/a&gt;  Labeled array that plays nice with NumPy.  Provides statistical functions not present in NumPy and good for data manipulation.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://code.google.com/p/python-statlib/&quot; rel=&quot;nofollow&quot;&gt;python-statlib&lt;/a&gt; A fairly recent effort which combined a number of scattered statistics libraries.  Useful for basic and descriptive statistics if you're not using NumPy or pandas.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://statsmodels.sourceforge.net/&quot; rel=&quot;nofollow&quot;&gt;statsmodels&lt;/a&gt; Statistical modeling: Linear models, GLMs, among others.  &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://scikits.appspot.com/scikits&quot; rel=&quot;nofollow&quot;&gt;scikits&lt;/a&gt;  Statistical and scientific computing packages -- notably smoothing, optimization and machine learning.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://code.google.com/p/pymc/&quot; rel=&quot;nofollow&quot;&gt;PyMC&lt;/a&gt; For your Bayesian/MCMC/hierarchical modeling needs. Highly recommended.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://www.pymix.org/pymix/index.php?n=PyMix.Home&quot; rel=&quot;nofollow&quot;&gt;PyMix&lt;/a&gt; Mixture models.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If speed becomes a problem, consider &lt;a href=&quot;http://deeplearning.net/software/theano/&quot; rel=&quot;nofollow&quot;&gt;Theano&lt;/a&gt; -- used with good success by the deep learning people.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There's plenty of other stuff out there, but this is what I find the most useful along the lines you mentioned.&lt;/p&gt;&#xA;" OwnerUserId="251" LastEditorUserId="251" LastEditorDisplayName="" LastEditDate="2010-08-13T05:37:13.030" LastActivityDate="2010-08-13T05:37:13.030" CommunityOwnedDate="2010-08-27T15:07:18.913" CommentCount="1" />
  <row Id="1633" PostTypeId="2" ParentId="1624" CreationDate="2010-08-13T05:43:29.447" Score="3" ViewCount="" Body="&lt;p&gt;The simplest answer is that one either uses numerical quadrature techniques (Simpson's, for instance) on the PDF, or approximations such as those found in &lt;a href=&quot;http://www.math.ucla.edu/~cbm/aands//page_932.htm&quot; rel=&quot;nofollow&quot;&gt;Abramowitz and Stegun&lt;/a&gt;. Personally however, since one merely needs a few digits of accuracy for the &quot;error function&quot; that is involved in computing z-scores, you might be interested the simple approximations given &lt;a href=&quot;http://homepages.physik.uni-muenchen.de/~Winitzki/erf-approx.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; by S. Winitzki.&lt;/p&gt;&#xA;" OwnerUserId="830" LastActivityDate="2010-08-13T05:43:29.447" />
  <row Id="1634" PostTypeId="2" ParentId="1562" CreationDate="2010-08-13T05:51:00.667" Score="4" ViewCount="" Body="&lt;p&gt;There are two definitions of statistical independence:&#xA;1) P(A,B)=P(A)*P(B) &amp;lt;=&gt; 2) P(A|B)=P(A) &amp;lt;=&gt; 2a) P(B|A)=P(B).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(&amp;lt;=&gt; means if and only if)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So to answer your question: both are valid.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Pearson Chi-square test of independence is motivated by definition 1), logistic regression and multinomial regression are motivated by definition 2) of independence.&lt;/p&gt;&#xA;" OwnerUserId="419" LastActivityDate="2010-08-13T05:51:00.667" />
  <row Id="1635" PostTypeId="2" ParentId="1622" CreationDate="2010-08-13T05:51:02.383" Score="1" ViewCount="" Body="&lt;p&gt;I'm probably missing something important, but why does the fact that your observed variable is a Sharpe ratio change the statistic you would use to test the difference in Sharpe ratios? I understand that they are already distributed like 2 independent non-central t statistics, but what forces you to treat them that way?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Presumably the central limit theorem would hold even for Sharpe ratios and as such you should be able to apply a parametric test of mean differences, e.g. independent-samples Z.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;More importantly, if your data is financial data wouldn't it be better to treat these as paired samples paired by the times at which they were observed?&lt;/p&gt;&#xA;" OwnerUserId="196" LastActivityDate="2010-08-13T05:51:02.383" CommentCount="1" />
  <row Id="1636" PostTypeId="2" ParentId="1380" CreationDate="2010-08-13T06:09:24.530" Score="1" ViewCount="" Body="&lt;p&gt;I may be a little unclear about the question.  But here would be my solution computing some &quot;statistic on each of the tables&quot; and comparing those values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your contingency tables are like a binomial effect size display (BESD), with clear YES/NO predictions being provided by each of your K methods you'll have a number of tables like this...&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;                 Reality&#xA;             +    - &#xA;Pred  +     70   30 100&#xA;Pred  -     30   70 100&#xA;            100 100 200&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I believe you can find the difference between the success rates, e.g. 70/100 – 30/100 = 40/100 = .40, this value can be considered as being equivalent to an effect size r for each of your K methods.  As a proof of concept I've included equivalent R code...&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x &amp;lt;- rep(c(1,0),each=100)&#xA;y &amp;lt;- c(rep(1,70),rep(0,30),rep(1,30),rep(0,70))&#xA;cor(x,y)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can then compare them using Fisher's Z' transformation for r in the standard way, e.g. &lt;a href=&quot;http://www.fon.hum.uva.nl/Service/Statistics/Two_Correlations.html&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.  To deal with situations where K is greater than 2 one may want to apply some familywise error correction to the Z' tests, but the exact one selected I leave open for another debate.  P.S. I might be remembering incorrectly, but I think you can find more details in Essentials of Behavioral Research: Methods and Data Analysis by Rosenthal &amp;amp; Rosnow, 2007, Ch 11&lt;/p&gt;&#xA;" OwnerUserId="196" LastEditorUserId="196" LastEditorDisplayName="" LastEditDate="2010-08-13T06:24:50.753" LastActivityDate="2010-08-13T06:24:50.753" CommentCount="2" />
  <row Id="1637" PostTypeId="1" AcceptedAnswerId="1639" CreationDate="2010-08-13T09:41:13.160" Score="11" ViewCount="516" Body="&lt;p&gt;I'm sure I've got this completely wrapped round my head, but I just can't figure it out.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The t-test compares two normal distributions using the Z distribution. That's why there's an assumption of normality in the DATA.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ANOVA is equivalent to linear regression with dummy variables, and uses sums of squares, just like OLS. That's why there's an assumption of normality of RESIDUALS.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's taken me several years, but I think I've finally grasped those basic facts. So why is it that the t-test is equivalent to ANOVA with two groups? How can they be equivalent if they don't even assume the same things about the data?&lt;/p&gt;&#xA;" OwnerUserId="199" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-08-13T10:00:01.717" LastActivityDate="2011-02-04T01:29:59.663" Title="If the t-test and the ANOVA for two groups are equivalent, why aren't their assumptions equivalent?" Tags="&lt;distributions&gt;&lt;regression&gt;&lt;normality&gt;&lt;t-test&gt;&lt;anova&gt;" AnswerCount="4" CommentCount="1" FavoriteCount="4" />
  <row Id="1638" PostTypeId="2" ParentId="1610" CreationDate="2010-08-13T09:50:51.067" Score="4" ViewCount="" Body="&lt;p&gt;Hurrah, a question non-technical enough so as I can answer it!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Type one is a con&quot; [rhyming]- i.e. fools you into thinking that a difference exists when it doesn't. Always works for me.&lt;/p&gt;&#xA;" OwnerUserId="199" LastActivityDate="2010-08-13T09:50:51.067" />
  <row Id="1639" PostTypeId="2" ParentId="1637" CreationDate="2010-08-13T09:52:44.700" Score="11" ViewCount="" Body="&lt;p&gt;The t-test with two groups assumes that each group is normally distributed with the same variance (although the means may differ under the alternative hypothesis). That is equivalent to a regression with a dummy variable as the regression allows the mean of each group to differ but not the variance. Hence the residuals (equal to the data with the group means subtracted) have the same distribution --- that is, they are normally distributed with zero mean.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A t-test with unequal variances is not equivalent to a one-way ANOVA.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-08-13T09:52:44.700" CommentCount="2" />
  <row Id="1641" PostTypeId="2" ParentId="672" CreationDate="2010-08-13T11:42:08.157" Score="3" ViewCount="" Body="&lt;p&gt;Bayes' theorem is a way to rotate a conditional probability $P(A|B)$ to another conditional probability $P(B|A)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A stumbling block for some is the meaning of $P(B|A)$.  This is a way to reduce the space of possible events by considering only those events where $A$ definitely happens (or is true).  So for instance the probability that a thrown, fair, dice lands showing six, $P(\mbox{dice lands six})$, is 1/6, however the probability that a dice lands six given that it landed an even number, $P(\mbox{dice lands six}|\mbox{dice lands even})$, is 1/3.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can derive Bayes' theorem yourself as follows.  Start with the ratio definition of a conditional probability:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(B|A) = \frac{P(AB)}{P(A)}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $P(AB)$ is the joint probability of $A$ and $B$ and $P(A)$ is the marginal probability of $A$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently the formula makes no reference to $P(A|B)$, so let's write down the definition of this too:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(A|B) = \frac{P(BA)}{P(B)}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The little trick for making this work is seeing that $P(AB) = P(BA)$ (since a Boolean algebra is underneath all of this, you can easily prove this with a truth table by showing $AB = BA$), so we can write:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(A|B) = \frac{P(AB)}{P(B)}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now to slot this into the formula for $P(B|A)$, just rewrite the formula above so $P(AB)$ is on the left:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(AB) = P(A|B)P(B)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and hey presto:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(B|A) = \frac{P(A|B)P(B)}{P(A)}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for what the point is to rotating a conditional probability in this way, consider the common example of trying to infer the probability that someone has a disease given that they have a symptom, i.e., we &lt;em&gt;know&lt;/em&gt; that they have a symptom - we can just see it - but we cannot be certain whether they have a disease and have to infer it.  I'll start with the formula and work back.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(\mbox{disease}|\mbox{symptom}) = \frac{P(\mbox{symptom}|\mbox{disease})P(\mbox{disease})}{P(\mbox{symptom})}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So to work it out, you need to know the prior probability of the symptom, the prior probability of the disease (i.e., how common or rare are the symptom and disease) and also the probability that someone has a symptom given we know someone has a disease (e.g., via expensive time consuming lab tests).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It can get a lot more complicated than this, e.g., if you have multiple diseases and symptoms, but the idea is the same.  Even more generally, Bayes' theorem often makes an appearance if you have a probability theory of relationships between causes (e.g., diseases) and effects (e.g., symptoms) and you need to reason backwards (e.g., you see some symptoms from which you want to infer the underlying disease).&lt;/p&gt;&#xA;" OwnerUserId="702" LastActivityDate="2010-08-13T11:42:08.157" />
  <row Id="1642" PostTypeId="2" ParentId="1610" CreationDate="2010-08-13T12:22:25.313" Score="1" ViewCount="" Body="&lt;p&gt;You could reject the idea entirely.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some authors (Andrew Gelman is one) are shifting to discussing Type S (sign) and Type M (magnitude) errors.  You can infer the wrong effect direction (e.g., you believe the treatment group does better but actually does worse) or the wrong magnitude (e.g., you find a massive effect where there is only a tiny, or essentially no effect, or vice versa).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See more at &lt;a href=&quot;http://www.stat.columbia.edu/~cook/movabletype/archives/2004/12/type_1_type_2_t.html&quot; rel=&quot;nofollow&quot;&gt;Gelman's blog&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="702" LastActivityDate="2010-08-13T12:22:25.313" CommentCount="2" />
  <row Id="1643" PostTypeId="2" ParentId="1637" CreationDate="2010-08-13T12:24:11.597" Score="5" ViewCount="" Body="&lt;p&gt;I totally agree with Rob's answer, but let me put it another way (using wikipedia):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Analysis_of_variance#Assumptions_of_ANOVA&quot; rel=&quot;nofollow&quot;&gt;Assumptions ANOVA&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Independence of cases – this is an assumption of the model that simplifies the statistical analysis.&lt;/li&gt;&#xA;&lt;li&gt;Normality – the distributions of the residuals are normal.&lt;/li&gt;&#xA;&lt;li&gt;Equality (or &quot;homogeneity&quot;) of variances, called homoscedasticity &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Student%27s_t-test#Assumptions&quot; rel=&quot;nofollow&quot;&gt;Assumptions t-test&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Each of the two populations being compared should follow a normal distribution ...&lt;/li&gt;&#xA;&lt;li&gt;... the two populations being compared should have the same variance ...&lt;/li&gt;&#xA;&lt;li&gt;The data used to carry out the test should be sampled independently from the two populations being compared.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Hence, I would refute the question, as they obviously have the same assumptions (although in a different order :-) ).&lt;/p&gt;&#xA;" OwnerUserId="442" LastActivityDate="2010-08-13T12:24:11.597" />
  <row Id="1644" PostTypeId="2" ParentId="1611" CreationDate="2010-08-13T12:31:09.510" Score="7" ViewCount="" Body="&lt;p&gt;I don't think it matters very much, as long as the interpretation of the results is performed within the same framework as the analysis.  The main problem with frequentist statistics is that there is a natural tendency to treat the p-value of a frequentist significance test as if it was a Bayesian a-posteriori probability that the null hypothesis is true (and hence 1-p is the probability that the alternative hypothesis is true), or treating a frequentist confidence interval as a Bayesian credible interval (and hence assuming there is a 95% probability that the true value lies within a 95% confidence interval for the particular sample of data we have).  These sorts of interpretation are natural as it would be the direct answer to the question we would naturally want to ask.  It is a trade-off between whether the subjective element of the Bayesian approach (which is itself debatable, see e.g. Jaynes book) is sufficiently abhorrent that it is worth making do with an indirect answer to the key question (and vice versa).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As long as the form of the answer is acceptable, and we can agree on the assumptions made, then there is no reason to prefer one over the other - it is a matter of horses for courses.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm still a Bayesian though ;o)&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2010-08-13T12:31:09.510" CommunityOwnedDate="2010-08-13T12:31:09.510" CommentCount="1" />
  <row Id="1645" PostTypeId="1" AcceptedAnswerId="1648" CreationDate="2010-08-13T12:42:30.220" Score="6" ViewCount="896" Body="&lt;p&gt;Title says enough... so far, I've been using Shapiro-Wilk's statistic in order to test normality assumptions in small samples. Could you, please, recommend another technique?&lt;/p&gt;&#xA;" OwnerUserId="1356" LastActivityDate="2010-11-06T14:06:09.300" Title="Appropriate normality tests for small samples" Tags="&lt;normality&gt;&lt;test&gt;&lt;small-sample&gt;" AnswerCount="5" FavoriteCount="3" />
  <row Id="1646" PostTypeId="1" AcceptedAnswerId="2524" CreationDate="2010-08-13T12:44:23.913" Score="7" ViewCount="332" Body="&lt;p&gt;Imagine that:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;You have a sample of 1000 teams each with 10 members.&lt;/li&gt;&#xA;&lt;li&gt;You measured team functioning by asking each team member how well they think their team is functioning using a reliable multi-item numeric scale.&lt;/li&gt;&#xA;&lt;li&gt;You want to describe the extent to which the measure of team effectiveness is a property of the team member's idiosyncratic belief or a property of a shared belief about the team.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In this and related situations (e.g., aggregating to organisations), many researchers report the intraclass correlation (e.g., Table 1 in &lt;a href=&quot;http://www.krannert.purdue.edu/faculty/campionm/Relations_Between_Work.pdf&quot; rel=&quot;nofollow&quot;&gt;Campion &amp;amp;  Medsker, 1993&lt;/a&gt;).&#xA;Thus, my questions are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;What descriptive labels would you attach to different values of the intra-class correlation? I.e., the aim is to actually relate the values of the intra-class correlation to qualitative language such as: &quot;When the intraclass correlation is greater than x, it suggests that the attitudes are modestly/moderately/strongly shared across team members.&quot;&lt;/li&gt;&#xA;&lt;li&gt;Do you think the intraclass correlation is the appropriate statistic or would you use a different strategy?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-08-13T12:50:59.220" LastActivityDate="2010-09-10T11:28:41.017" Title="Intraclass correlation and aggregation" Tags="&lt;correlation&gt;&lt;intraclass-correlation&gt;&lt;aggregation&gt;&lt;interpretation&gt;&lt;effect-size&gt;" AnswerCount="2" CommentCount="3" FavoriteCount="2" />
  <row Id="1647" PostTypeId="2" ParentId="1645" CreationDate="2010-08-13T12:47:16.497" Score="6" ViewCount="" Body="&lt;p&gt;There is a whole &lt;a href=&quot;http://en.wikipedia.org/wiki/Category%3aNormality_tests&quot; rel=&quot;nofollow&quot;&gt;Wikipedia category on normality tests&lt;/a&gt; including:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;the &lt;a href=&quot;http://en.wikipedia.org/wiki/Anderson%E2%80%93Darling_test&quot; rel=&quot;nofollow&quot;&gt;Anderson-Darling test&lt;/a&gt;, popular amongst statisticians; and&lt;/li&gt;&#xA;&lt;li&gt;the &lt;a href=&quot;http://en.wikipedia.org/wiki/Jarque%E2%80%93Bera_test&quot; rel=&quot;nofollow&quot;&gt;Jarque-Bera test&lt;/a&gt;, popular amongst econometricians.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I think A-D is probably the best of them.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-08-13T12:47:16.497" CommentCount="1" />
  <row Id="1648" PostTypeId="2" ParentId="1645" CreationDate="2010-08-13T13:32:27.913" Score="10" ViewCount="" Body="&lt;p&gt;The &lt;a href=&quot;http://cran.r-project.org/web/packages/fBasics/index.html&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;fBasics&lt;/strong&gt;&lt;/a&gt; package in R (part of &lt;a href=&quot;https://www.rmetrics.org/&quot; rel=&quot;nofollow&quot;&gt;Rmetrics&lt;/a&gt;) includes &lt;a href=&quot;http://hosho.ees.hokudai.ac.jp/~kubo/Rdoc/library/fBasics/html/NormalityTests.html&quot; rel=&quot;nofollow&quot;&gt;several normality tests&lt;/a&gt;, covering many of the popular &lt;a href=&quot;http://en.wikipedia.org/wiki/Normality_test#Frequentist_tests&quot; rel=&quot;nofollow&quot;&gt;frequentist tests&lt;/a&gt; -- Kolmogorov-Smirnov, Shapiro-Wilk's, Jarque–Bera, and D'Agostino -- along with a wrapper for the normality tests in the &lt;a href=&quot;http://cran.r-project.org/web/packages/nortest/index.html&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;nortest&lt;/strong&gt;&lt;/a&gt; package -- Anderson–Darling, Cramer–von Mises, Lilliefors (Kolmogorov-Smirnov), Pearson chi–square, and Shapiro–Francia.  The package documentation also provides all the important references.  Here is a demo that shows how to use the &lt;a href=&quot;http://duncanjg.files.wordpress.com/2008/12/ksdemo3.pdf&quot; rel=&quot;nofollow&quot;&gt;tests from nortest&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One approach, if you have the time, is to use more than one test and check for agreement.  The tests vary in a number of ways, so it isn't entirely straightforward to choose &quot;the best&quot;.  What do other researchers in your field use?  This can vary and it may be best to stick with the accepted methods so that others will accept your work.  I frequently use the Jarque-Bera test, partly for that reason, and Anderson–Darling for comparison.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can look at &lt;a href=&quot;http://interstat.statjournals.net/YEAR/2002/articles/0201001.pdf&quot; rel=&quot;nofollow&quot;&gt;&quot;Comparison of Tests for Univariate Normality&quot;&lt;/a&gt; (Seier 2002) and &lt;a href=&quot;http://www.informaworld.com/smpp/content~db=all~content=a759350109&quot; rel=&quot;nofollow&quot;&gt;&quot;A comparison of various tests of normality&quot;&lt;/a&gt; (Yazici; Yolacan 2007) for a comparison and discussion of the issues.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's also trivial to test these methods for comparison in R, thanks to all the &lt;a href=&quot;http://cran.r-project.org/doc/manuals/R-intro.html#Probability-distributions&quot; rel=&quot;nofollow&quot;&gt;distribution functions&lt;/a&gt;.  Here's a simple example with simulated data (I won't print out the results to save space), although a more full exposition would be required:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(fBasics); library(ggplot2)&#xA;set.seed(1)&#xA;&#xA;# normal distribution&#xA;x1 &amp;lt;- rnorm(1e+06)   &#xA;x1.samp &amp;lt;- sample(x1, 200)&#xA;qplot(x1.samp, geom=&quot;histogram&quot;)&#xA;jbTest(x1.samp)&#xA;adTest(x1.samp)&#xA;&#xA;# cauchy distribution&#xA;x2 &amp;lt;- rcauchy(1e+06)&#xA;x2.samp &amp;lt;- sample(x2, 200)&#xA;qplot(x2.samp, geom=&quot;histogram&quot;)&#xA;jbTest(x2.samp)&#xA;adTest(x2.samp)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Once you have the results from the various tests over different distributions, you can compare which were the most effective.  For instance, the p-value for the Jarque-Bera test above returned 0.276 for the normal distribution (accepting) and &amp;lt; 2.2e-16 for the cauchy (rejecting the null hypothesis).&lt;/p&gt;&#xA;" OwnerUserId="5" LastEditorUserId="5" LastEditorDisplayName="" LastEditDate="2010-08-18T01:11:18.693" LastActivityDate="2010-08-18T01:11:18.693" CommentCount="1" />
  <row Id="1649" PostTypeId="1" AcceptedAnswerId="1650" CreationDate="2010-08-13T13:58:38.567" Score="4" ViewCount="292" Body="&lt;p&gt;Say I observe two groups of 10 people, measuring some quantity 100 times in each person. There will presumably be some variability across these 100 measures in each person. Can I use mixed effects analysis to assess whether this within-person variability is, on average, different between the two groups? For example, using traditional statistics, I could compute the standard deviation (SD) within each person then submit these SDs to an anova comparing the groups, but I wonder if this two-stage process can be replaced by a single mixed effects model, consequently obtaining the various advantages of mixed effects modelling (shrinkage, accounting for different numbers of observations per person, etc) as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To be clear, here is R code depicting the scenario described above and the SD/anova-based approach:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(1)&#xA;&#xA;group_A_base_sd = 1&#xA;group_B_base_sd = 2&#xA;within_group_sd_of_sds = .1&#xA;&#xA;n_per_group = 10&#xA;obs_per_id = 100&#xA;&#xA;temp = data.frame(&#xA;    id = 1:(n_per_group*2)&#xA;    , group = rep(c('A','B'))&#xA;)&#xA;&#xA;#generate example data&#xA;library(plyr) #to avoid loops (for coding convenience only)&#xA;obs_data = ddply(&#xA;    .data = temp&#xA;    , .variables = .(id,group)&#xA;    , .fun = function(x){&#xA;         #generate a unique sd for this individual&#xA;         # based on their group's sd plus some&#xA;         # within-group variability&#xA;         id_sd = ifelse(&#xA;              x$group=='A'&#xA;              , rnorm(&#xA;                   1&#xA;                   , group_A_base_sd&#xA;                   , within_group_sd_of_sds&#xA;              )&#xA;              , rnorm(&#xA;                   1&#xA;                   , group_B_base_sd&#xA;                   , within_group_sd_of_sds&#xA;              )&#xA;         )&#xA;         #generate data points with the above generated&#xA;         # variability&#xA;         to_return = data.frame(&#xA;              obs_num = 1:obs_per_id&#xA;              , measurement = rnorm(obs_per_id,0,id_sd)&#xA;         )&#xA;         return(to_return)&#xA;    }&#xA;)&#xA;&#xA;#first step of an anova-based approach:&#xA;# compute SDs within each Ss&#xA;obs_sds = ddply(&#xA;    .data = obs_data&#xA;    , .variables = .(id,group)&#xA;    , .fun = function(x){&#xA;         to_return = data.frame(&#xA;              obs_sd = sd(x$measurement)&#xA;         )&#xA;    }&#xA;)&#xA;&#xA;#second step of an anova-based approach:&#xA;# compute the anova on the SDs&#xA;summary(&#xA;    aov(&#xA;         formula = obs_sd~group&#xA;         , data = obs_sds&#xA;    )&#xA;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="364" LastActivityDate="2010-08-17T06:26:59.823" Title="Using mixed effects modelling to estimate and compare variability" Tags="&lt;variance&gt;&lt;mixed-model&gt;" AnswerCount="2" FavoriteCount="2" />
  <row Id="1650" PostTypeId="2" ParentId="1649" CreationDate="2010-08-13T14:17:15.793" Score="3" ViewCount="" Body="&lt;p&gt;You can structure the model along the following lines. Let,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$j = 1, 2$ be the two groups and&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$i$ index the individuals in the two groups. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then your model is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$y_{ij} \sim N(\mu_j,\sigma_j^2)$  $\forall \ i, j$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\sigma_j^2 \sim IG(v,1)$  $\forall \ j$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Note: $IG(.)$ is the inverse gamma distribution.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Priors&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\mu_j \sim N(\bar{\mu},\sigma_{\mu}^2)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$v \sim IG(\bar{v},1)$ is the prior for $v$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The above structure will let you shrink the error variances ($\sigma_j^2$) appropriately. You can then evaluate whether the within group variability is different by looking at the credible intervals associated with the group variabilities.&lt;/p&gt;&#xA;" LastActivityDate="2010-08-13T14:17:15.793" />
  <row Id="1651" PostTypeId="1" CreationDate="2010-08-13T14:28:02.520" Score="5" ViewCount="330" Body="&lt;p&gt;I need to fit $Y_{ij} \sim NegBin(m_{ij},k)$, i.e. a negative binomial distribution to  count data. However, the data I have observed are censored - I know the value of $y_{ij}$, but it could be more than that value. The log-likelihood is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{equation}&#xA;ll = \sum_{i=1}^n w_i (c_i \log(P(Y_{ij}=y_{ij}|X_{ij})) + (1- c_i) \log(1- \sum_{k=1}^32 P(Y_{ij} = k|X_{ij})))&#xA;\end{equation}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $X_{ij}$ represent the design matrix (with the covariates of interest), $w_i$ is the weight for each observation, $y_{ij}$ is the response variable and $P(Y_{ij}=y_{ij}|X_{ij})$ is the negative binomial distribution where the $m_{ij}=exp(X_{ij} \beta)$ and $\alpha$ is the over-dispersion parameter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone know of an R package to tackle this problem?&lt;/p&gt;&#xA;" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-08-17T13:21:27.277" LastActivityDate="2010-08-17T13:21:27.277" Title="How to fit a negative binomial distribution in R while incorporating censoring" Tags="&lt;r&gt;&lt;censoring&gt;&lt;negative-binomial&gt;" AnswerCount="2" CommentCount="3" />
  <row Id="1652" PostTypeId="2" ParentId="534" CreationDate="2010-08-13T15:21:55.373" Score="4" ViewCount="" Body="&lt;p&gt;Correlation alone never implies causation.  It's that simple.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But it's very rare to have only a correlation between two variables.  Often you also know something about what those variables are and a theory, or theories, suggesting why there might be a causal relationship between the variables.  If not, then we bother checking for a correlation?  (However people mining massive correlation matrices for significant results often have no casual theory - otherwise, why bother mining.  A counterargument to that is that often some exploration is needed to get ideas for casual theories. And so on and so on...)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A response to the common criticism &quot;Yeah, but that's just a correlation: it doesn't imply causation&quot;:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;For a casual relationship, correlation is necessary.  A repeated failure to find a correlation would be bad news indeed.&lt;/li&gt;&#xA;&lt;li&gt;I didn't just give you a correlation.&lt;/li&gt;&#xA;&lt;li&gt;Then go on to explain possible causal mechanisms explaining the correlation...&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="702" LastActivityDate="2010-08-13T15:21:55.373" />
  <row Id="1653" PostTypeId="2" ParentId="1637" CreationDate="2010-08-13T15:24:20.490" Score="1" ViewCount="" Body="&lt;p&gt;The t-test simply a special case of the F-test where only two groups are being compared.  The result of either will be exactly the same in terms of the p-value and there is a simple relationship between the F and t statistics as well.  F = t^2.  The two tests are algebraically equivalent and their assumptions are the same.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In fact, these equivalences extend to the whole class of ANOVAs, t-tests, and linear regression models.  The t-test is a special case of ANOVA.  ANOVA is a special case of regression.  All of these procedures are subsumed under the General Linear Model and share the same assumptions.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Independence of observations.&lt;/li&gt;&#xA;&lt;li&gt;Normality of residuals = normality in each group in the special case.&lt;/li&gt;&#xA;&lt;li&gt;Equal of variances of residuals = equal variances across groups in the special case.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;You might think of it as normality in the data, but you are checking for normality in each group--which is actually the same as checking for normality in the residuals when the only predictor in the model is an indicator of group.  Likewise with equal variances.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just as an aside, R does not have seperate routines for ANOVA.  The anova functions in R are just wrappers to the lm() function--the same thing that is used to fit linear regression models--packaged a little differently to provide what is typically found in an ANOVA summary rather than a regression summary.&lt;/p&gt;&#xA;" OwnerUserId="485" LastEditorUserId="485" LastEditorDisplayName="" LastEditDate="2010-08-13T15:42:56.300" LastActivityDate="2010-08-13T15:42:56.300" CommentCount="6" />
  <row Id="1654" PostTypeId="2" ParentId="173" CreationDate="2010-08-13T15:31:27.857" Score="3" ViewCount="" Body="&lt;p&gt;You may try to model your data using a Dynamic Generalized Linear Model (DGLM). In R, you can fit this kind of models using packages sspir and KFAS. In a sense, this is similar to the gam approach suggested by Rob, except that instead of assuming that the log mean of the Poisson observations be a smooth function of time, it assumes that it follows a stochastic dynamics.&lt;/p&gt;&#xA;" LastActivityDate="2010-08-13T15:31:27.857" CommentCount="1" />
  <row Id="1655" PostTypeId="2" ParentId="1601" CreationDate="2010-08-13T15:48:29.310" Score="2" ViewCount="" Body="&lt;p&gt;John Tukey systematically discusses transformations in his book on EDA.  In addition to the Box-Cox family (affinely scaled power transformations) he defines a family of &quot;folded&quot; transformations for proportions (essentially powers of x/(1-x)) and &quot;started&quot; counts (adding a positive offset to counted data before transforming them).  The folded transformations, which essentially generalize the logit, are especially useful for test scores.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a completely different vein, Johnson &amp;amp; Kotz in their books on distributions offer many transformations intended to convert test statistics to approximate normality (or to some other target distribution), such as the cube-root transformation for chi-square.  This material is a great source of ideas for useful transformations when you anticipate your data will follow some specific distribution.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-08-13T15:48:29.310" />
  <row Id="1656" PostTypeId="2" ParentId="1645" CreationDate="2010-08-13T15:54:37.857" Score="1" ViewCount="" Body="&lt;p&gt;For completeness, econometricians also like the Kiefer and Salmon test from their 1983 paper in Economics Letters -- it sums 'normalized' expressions of skewness and kurtosis which is then chi-square distributed.  I have an old C++ version I wrote during grad school I could translate into R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Edit:&lt;/em&gt; And &lt;a href=&quot;http://econ.la.psu.edu/~hbierens/NORMTEST.PDF&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; is recent paper by Bierens (re-)deriving Jarque-Bera and Kiefer-Salmon.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Edit 2:&lt;/em&gt; I looked over the old code, and it seems that it really is the same test between Jarque-Bera and Kiefer-Salmon. &lt;/p&gt;&#xA;" OwnerUserId="334" LastEditorUserId="334" LastEditorDisplayName="" LastEditDate="2010-08-13T16:54:43.857" LastActivityDate="2010-08-13T16:54:43.857" />
  <row Id="1657" PostTypeId="2" ParentId="726" CreationDate="2010-08-13T16:04:08.990" Score="4" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://www.causeweb.org/&quot; rel=&quot;nofollow&quot;&gt;CauseWeb&lt;/a&gt; has a collection of statistics quotations.  Many have already been repeated here, but it has plenty that haven't yet been quoted, such as&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&quot;The only statistics you can trust are those you falsified yourself.&quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;(Falsely attributed to Sir Winston Churchill.)  For the rest, follow the CauseWeb links to Resources-&gt;Fun-&gt;Quote.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-08-17T22:11:08.277" LastActivityDate="2010-08-17T22:11:08.277" CommunityOwnedDate="2010-08-13T16:04:08.990" />
  <row Id="1658" PostTypeId="2" ParentId="726" CreationDate="2010-08-13T16:11:20.950" Score="2" ViewCount="" Body="&lt;p&gt;&quot;I cannot conceal the fact here that in the [application of probability theory], I foresee many things happening which can cause one to be badly mistaken if he does not proceed cautiously.&quot;,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bernoulli (1713) (via ET Jaynes)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;A statistician is someone who knows what to assume to be Gaussian&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Dikran Marsupial (2009) (not famous yet ;o).&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2010-08-13T16:11:20.950" CommunityOwnedDate="2010-08-13T16:11:20.950" CommentCount="2" />
  <row Id="1659" PostTypeId="2" ParentId="1462" CreationDate="2010-08-13T16:15:41.417" Score="0" ViewCount="" Body="&lt;p&gt;A player's yardage is unlikely to be anywhere near normally distributed.  If it were, your guy averaging 5.3 give or take 1.7 yards would almost never lose yards or gain more than 11 yards on any play in the entire season.  Gone is the excitement of the game, to be replaced by some statistical mediocrity.  If football were played like this, a team's chances of making a set of downs would be almost certain; there would almost never be a loss of downs; and the game would simply be determined by who won the initial coin flip and got on the field first.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why not just draw a value at random from a list of the player's recent gains (and losses)?  It's fairly easy to program: you just have to generate a uniformly distributed integer to index into an array of the gains.  It doesn't require any kind of statistical model--no need to fit anything.  It can account for change in the player's ability over time (just by selecting which time period you will use to draw the data from).  And it's obviously driven by &quot;real-live data.&quot;&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-08-13T16:15:41.417" />
  <row Id="1660" PostTypeId="1" AcceptedAnswerId="1665" CreationDate="2010-08-13T16:43:45.133" Score="5" ViewCount="207" Body="&lt;p&gt;I need to define what a test of independence is, without the use of heavily statistic terms.&lt;/p&gt;&#xA;" OwnerUserId="559" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-17T20:25:42.563" LastActivityDate="2010-09-17T20:25:42.563" Title="What is a test of independence" Tags="&lt;basic-concepts&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="1661" PostTypeId="1" CreationDate="2010-08-13T17:21:37.057" Score="2" ViewCount="243" Body="&lt;p&gt;X1 is wing length, X2 is tail length for 45 male and 45 female bugs.&lt;br&gt;&#xA;Which 2-sample univariate t-test should I use? &#xA;My thought was to use Hotelling's T-square?&lt;br&gt;&#xA;But Hotelling's is multi-variate not univariate. Now, I'm not sure...  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas?&lt;/p&gt;&#xA;" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-08-15T08:47:32.430" LastActivityDate="2010-08-17T18:31:59.927" Title="Which 2-sample univariate t-test to use?" Tags="&lt;t-test&gt;" AnswerCount="3" CommentCount="1" FavoriteCount="1" />
  <row Id="1662" PostTypeId="2" ParentId="1661" CreationDate="2010-08-13T17:39:02.347" Score="1" ViewCount="" Body="&lt;p&gt;While your questions is not clear (which means do you want to compare?) you can consult the wiki: &lt;a href=&quot;http://en.wikipedia.org/wiki/Comparing_means&quot; rel=&quot;nofollow&quot;&gt;Comparing Means&lt;/a&gt; to decide what to do.&lt;/p&gt;&#xA;" LastActivityDate="2010-08-13T17:39:02.347" CommentCount="3" />
  <row Id="1663" PostTypeId="2" ParentId="1660" CreationDate="2010-08-13T18:27:56.683" Score="1" ViewCount="" Body="&lt;p&gt;Why don't you take the definition of wikipedia. It's quite short und doesn't use heavily statistic terms.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;A test of independence assesses whether paired observations on two variables, expressed in a contingency table, are independent of each other – for example, whether people from different regions differ in the frequency with which they report that they support a political candidate.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="927" LastActivityDate="2010-08-13T18:27:56.683" />
  <row Id="1664" PostTypeId="2" ParentId="1661" CreationDate="2010-08-13T19:29:26.220" Score="0" ViewCount="" Body="&lt;p&gt;As others have said, you need to clarify your question. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I'm guessing that you want to determine if wing length or tail length differ between male and female bugs. In this case I would just do a couple of &lt;a href=&quot;http://en.wikipedia.org/wiki/Student%27s_t-test#Independent_two-sample_t-test&quot; rel=&quot;nofollow&quot;&gt;two sample t-tests&lt;/a&gt;. So for wing length you would have the following hypothesis:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;H_0: wing length does not depend on gender&#xA;H_1: wing length differs by gender.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You would have something similar for tail length.&lt;/p&gt;&#xA;" OwnerUserId="8" LastActivityDate="2010-08-13T19:29:26.220" CommentCount="4" />
  <row Id="1665" PostTypeId="2" ParentId="1660" CreationDate="2010-08-13T19:39:43.713" Score="1" ViewCount="" Body="&lt;p&gt;I would start by defining what you mean by independence. For example, &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;If two variables are independent this&#xA;  means that knowing the value of one&#xA;  variable does not tell you anything&#xA;  about the value of the other variable.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Then I would describe the test:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;To test for independence we construct&#xA;  a table of values that we would expect&#xA;  to see if the variables were&#xA;  independent. If we observed something&#xA;  &quot;very&quot; different from these expected&#xA;  values, we would conclude that the&#xA;  variables are unlikely to be independent.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="8" LastActivityDate="2010-08-13T19:39:43.713" CommentCount="1" />
  <row Id="1666" PostTypeId="2" ParentId="414" CreationDate="2010-08-13T21:22:56.557" Score="2" ViewCount="" Body="&lt;p&gt;I loved the Freedman, Pisani, Purves &lt;em&gt;Statistics&lt;/em&gt; text because it is extremely &lt;em&gt;non&lt;/em&gt;-mathematical.  As a mathematician you will find it to be such a clear guide to the statistical concepts that you will be able to develop all the mathematical theory as an exercise: that's a rewarding thing to do.  (The first edition of this text was my initiation to statistics after I completed a PhD in pure mathematics and I still enjoy re-reading it.)&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-08-13T21:22:56.557" CommunityOwnedDate="2010-08-13T21:22:56.557" />
  <row Id="1667" PostTypeId="1" CreationDate="2010-08-13T22:29:08.050" Score="4" ViewCount="129" Body="&lt;p&gt;As an engineer, I'm interested in topics such as designing experiments that are statistically valid, quality control, process control, reliability, and cost control. I took a course in engineering statistics, but unfortunately neither the book nor the professor were that good. I did OK in the course, but I'm interested in learning more about these topics and how to apply them to engineering problems. I would prefer a general book that covered as many of these topics as possible - great depth is not needed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think that I can learn a lot about improving my abilities by looking at how all engineering disciplines use statistics, so I'm not looking for any particular engineering field.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can the Statistical Analysis community recommend books that I can use to learn more about applying statistics to engineering problems?&lt;/p&gt;&#xA;" OwnerUserId="110" LastEditorUserId="110" LastEditorDisplayName="" LastEditDate="2010-08-14T01:21:36.430" LastActivityDate="2010-08-14T01:32:29.947" CommunityOwnedDate="2010-08-13T22:29:08.050" Title="What books provide an overview of engineering statistics?" Tags="&lt;books&gt;&lt;engineering-statistics&gt;" AnswerCount="3" FavoriteCount="1" />
  <row Id="1668" PostTypeId="1" CreationDate="2010-08-13T22:35:50.853" Score="10" ViewCount="462" Body="&lt;p&gt;As a software engineer, I'm interested in topics such as statistical algorithms, data mining, machine learning, Bayesian networks, classification algorithms, neural networks, Markov chains, Monte Carlo methods, and random number generation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I personally haven't had the pleasure of working hands-on with any of these techniques, but I have had to work with software that, under the hood, employed them and would like to know more about them, at a high level. I'm looking for books that cover a great breadth - great depth is not necessary at this point. I think that I can learn a lot about software development if I can understand the mathematical foundations behind the algorithms and techniques that are employed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can the Statistical Analysis community recommend books that I can use to learn more about implementing various statistical elements in software?&lt;/p&gt;&#xA;" OwnerUserId="110" LastEditorUserId="110" LastEditorDisplayName="" LastEditDate="2010-08-14T01:21:07.197" LastActivityDate="2010-09-03T19:38:45.930" CommunityOwnedDate="2010-08-13T22:35:50.853" Title="What books provide an overview of computational statistics as it applies to computer science?" Tags="&lt;books&gt;&lt;computational-statistics&gt;&lt;computing&gt;" AnswerCount="9" FavoriteCount="6" />
  <row Id="1669" PostTypeId="2" ParentId="1667" CreationDate="2010-08-13T23:58:11.623" Score="1" ViewCount="" Body="&lt;p&gt;NIST/SEMATECH e-Handbook of Statistical Methods is a good start. Free and online:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.itl.nist.gov/div898/handbook/&quot; rel=&quot;nofollow&quot;&gt;http://www.itl.nist.gov/div898/handbook/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="74" LastActivityDate="2010-08-13T23:58:11.623" CommunityOwnedDate="2010-08-13T23:58:11.623" CommentCount="1" />
  <row Id="1670" PostTypeId="2" ParentId="1667" CreationDate="2010-08-14T00:58:21.737" Score="1" ViewCount="" Body="&lt;p&gt;When I took the Engineering Statistics course I mentioned in the question, the assigned textbook wasn't very helpful. Instead, I used &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0495107573&quot; rel=&quot;nofollow&quot;&gt;Probability and Statistics for Engineers and Scientists - Anthony Hayter&lt;/a&gt; to get through the course. It didn't cover everything in the same order and depth of the course, but it was sufficient to get me through the material and get a passing grade.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Topics covered include probability theory, random variables, discrete and continuous probability distributions, normal distributions, descriptive statistics, statistical estimation and sampling distributions, population means, discrete data analysis, ANOVA, linear regression, nonlinear regression, multifactor experimental design and analysis, nonparametric statistical analysis, quality control methods, and reliability analysis. Unfortunately, the course only covered the first 11 chapters and occasionally in more depth then this book went into.&lt;/p&gt;&#xA;" OwnerUserId="110" LastActivityDate="2010-08-14T00:58:21.737" CommunityOwnedDate="2010-08-14T00:58:21.737" />
  <row Id="1671" PostTypeId="2" ParentId="1668" CreationDate="2010-08-14T01:02:31.637" Score="1" ViewCount="" Body="&lt;p&gt;I picked up a copy of &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1584886412&quot; rel=&quot;nofollow&quot;&gt;Probability and Statistics for Computer Scientists - Michael Baron&lt;/a&gt; on sale with another statistics book (I honestly bought it because of the name - I wanted a book that would take some kind of look at statistics from a computer science perspective, even if it wasn't perfect). I haven't had a chance to read it or work any problems in it yet, but it seems like a solid book.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The preface of the book says that it's for upper level undergraduate students and beginning graduate students, and I would agree with this. Some understanding of probability and statistics are necessary to grasp the contents of this book.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Topics include probability, discrete random variables, continuous distributions, Monte Carlo methods, stochastic processes, queuing systems, statistical inference, and regression.&lt;/p&gt;&#xA;" OwnerUserId="110" LastActivityDate="2010-08-14T01:02:31.637" CommunityOwnedDate="2010-08-14T01:02:31.637" />
  <row Id="1672" PostTypeId="2" ParentId="1668" CreationDate="2010-08-14T01:09:33.820" Score="1" ViewCount="" Body="&lt;p&gt;Although it's not specifically computational statistics, &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1584885394&quot; rel=&quot;nofollow&quot;&gt;A Handbook of Statistical Analyses Using R - Brian S. Everitt and Torsten Hothorn&lt;/a&gt; covers a lot of topics that I've seen covered in basic and intermediate statistics books - inference, ANOVA, linear regression, logistic regression, density estimation, recursive partitioning, principal component analysis, and cluster analysis - using the R language. This might be of interest to those interested in programming.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, unlike other books, the emphasis is on using the R language to carry out these statistical functions. Other books I've seen use combinations of algebra and calculus to demonstrate statistics. This book actually focuses on how to analyze data using the R language. And to make it even more useful, the data sets the authors use are in CRAN - the R Repository.&lt;/p&gt;&#xA;" OwnerUserId="110" LastActivityDate="2010-08-14T01:09:33.820" CommunityOwnedDate="2010-08-14T01:09:33.820" />
  <row Id="1673" PostTypeId="2" ParentId="1668" CreationDate="2010-08-14T01:16:16.037" Score="1" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1584885459&quot; rel=&quot;nofollow&quot;&gt;Statistical Computing with R - Maria L. Rizzo&lt;/a&gt; covers a lot of the topics in Probability and Statistics for Computer Scientists - basic probability and statistics, random variables, Bayesian statistics, Markov chains, visualization of multivariate data, Monte Carlo methods, Permutation tests, probability density estimation, and numerical methods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The equations and formulas used are presented both as mathematical formulas as well as in R code. I would say that a basic knowledge of probability, statistics, calculus, and maybe discrete mathematics would be advisable for anyone who wants to read this book. A programming background would also be helpful, but there are some references for the R language, operators, and syntax.&lt;/p&gt;&#xA;" OwnerUserId="110" LastActivityDate="2010-08-14T01:16:16.037" CommunityOwnedDate="2010-08-14T01:16:16.037" />
  <row Id="1674" PostTypeId="1" AcceptedAnswerId="1693" CreationDate="2010-08-14T01:31:43.443" Score="3" ViewCount="128" Body="&lt;p&gt;On page 331 of &quot;Elements of Information Theory&quot; (1991), author says that while entropy is related to the volume of the typical set, Fisher information is related to the surface area of the typical set, but I can't find anything more on this...can anyone explain this connection?&lt;/p&gt;&#xA;" OwnerUserId="511" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-17T20:26:19.617" LastActivityDate="2010-09-17T20:26:19.617" Title="Fisher information and the &quot;surface area of the typical set&quot;" Tags="&lt;information-theory&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="1" />
  <row Id="1675" PostTypeId="2" ParentId="1667" CreationDate="2010-08-14T01:32:29.947" Score="1" ViewCount="" Body="&lt;p&gt;The book &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/053873518X&quot; rel=&quot;nofollow&quot;&gt;Statistical Methods for Engineers - Geoffrey Vining&lt;/a&gt; is used in my university's Engineering Statistics course. However, I do not recommend this book. When I took the course, I ended up not being able to learn from the professor, so I was using this book to teach myself the material. It went along with the course in terms of content and depth, but I found the examples presented to be confusing and not as clear as they could have been. If you have a strong statistics background to begin with, it might be a suitable book, but this was the first statistics course that I had taken (and the only one required). There were no errors with the book - the examples and solutions were all correct.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are an engineer with a preexisting background in statistics, perhaps it might be worth it to visit your local library and check it out first before you buy it - it might work better for you than it did for me.&lt;/p&gt;&#xA;" OwnerUserId="110" LastActivityDate="2010-08-14T01:32:29.947" CommunityOwnedDate="2010-08-14T01:32:29.947" />
  <row Id="1676" PostTypeId="1" CreationDate="2010-08-14T01:39:23.253" Score="4" ViewCount="330" Body="&lt;p&gt;&lt;strong&gt;Duplicate thread:&lt;/strong&gt; &lt;a href=&quot;http://stats.stackexchange.com/questions/73/what-r-packages-do-you-find-most-useful-in-your-daily-work&quot;&gt;What R packages do you find most useful in your daily work?&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any R packages that are just plain good to have, regardless of the type of work you are doing? If so, what are these packages? If not, what packages do you find the most useful?&lt;/p&gt;&#xA;" OwnerUserId="110" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-05-11T13:28:51.157" LastActivityDate="2011-05-11T13:28:51.157" CommunityOwnedDate="2010-08-14T09:41:44.997" Title="I just installed the latest version of R. What packages should I obtain?" Tags="&lt;r&gt;" AnswerCount="10" CommentCount="9" FavoriteCount="1" />
  <row Id="1677" PostTypeId="2" ParentId="1668" CreationDate="2010-08-14T01:41:18.490" Score="2" ViewCount="" Body="&lt;p&gt;You might want to read the extremely popular question on Stack Overflow on &#xA;&lt;a href=&quot;http://stackoverflow.com/questions/2039904/what-statistics-should-a-programmer-or-computer-scientist-know&quot;&gt;what statistics a programmer or computer scientist should know&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-08-14T01:41:18.490" CommunityOwnedDate="2010-08-14T01:41:18.490" CommentCount="1" />
  <row Id="1678" PostTypeId="2" ParentId="1676" CreationDate="2010-08-14T01:46:57.587" Score="5" ViewCount="" Body="&lt;p&gt;I imagine graphics and data manipulation are two things that are useful no matter what you are doing. Thus, I'd recommend:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;ggplot2 (great graphics)&lt;/li&gt;&#xA;&lt;li&gt;lattice (great graphics)&lt;/li&gt;&#xA;&lt;li&gt;plyr (useful for data manipulation)&lt;/li&gt;&#xA;&lt;li&gt;Hmisc (good for descriptive statistics and much more)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-08-14T01:46:57.587" CommentCount="1" />
  <row Id="1679" PostTypeId="2" ParentId="1624" CreationDate="2010-08-14T03:56:28.753" Score="0" ViewCount="" Body="&lt;p&gt;Excel has pretty good functions for the normal distribution, so long as you aren't way out in the tail (where they fall apart). &lt;/p&gt;&#xA;" LastActivityDate="2010-08-14T03:56:28.753" />
  <row Id="1680" PostTypeId="2" ParentId="1676" CreationDate="2010-08-14T04:58:27.553" Score="2" ViewCount="" Body="&lt;p&gt;This is definitely a question that doesn't have &quot;an answer&quot;.  It is completely dependent on what you want to do.  That aside, I'll share the packages that I install as a standard with an R update...&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;install.packages(c(&quot;car&quot;,&quot;gregmisc&quot;,&quot;xtable&quot;,&quot;Design&quot;,&quot;Hmisc&quot;,&quot;psych&quot;,&#xA;                        &quot;CCA&quot;, &quot;fda&quot;, &quot;zoo&quot;, &quot;fields&quot;,&#xA;                      &quot;catspec&quot;,&quot;sem&quot;,&quot;multilevel&quot;,&quot;Deducer&quot;,&quot;RQDA&quot;))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and leave it to you to investigate those packages and see if they are valuable to you.&lt;/p&gt;&#xA;" OwnerUserId="485" LastActivityDate="2010-08-14T04:58:27.553" />
  <row Id="1681" PostTypeId="1" AcceptedAnswerId="2081" CreationDate="2010-08-14T07:16:00.080" Score="3" ViewCount="152" Body="&lt;p&gt;Chernoff bound (for &lt;a href=&quot;http://en.wikipedia.org/wiki/Chernoff_bound#Theorem_for_additive_form_.28absolute_error.29&quot; rel=&quot;nofollow&quot;&gt;absolute error&lt;/a&gt;) gives a bound on probability of large deviation in terms of sample size and amount of deviation, but it doesn't seem possible to rewrite it to give an explicit bound on the amount of deviation. So, what is good way to bound the largest deviation from the mean in terms of sample size and probability of that deviation?&lt;/p&gt;&#xA;" OwnerUserId="511" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-29T00:15:43.423" LastActivityDate="2011-04-29T00:15:43.423" Title="Chernoff-like bound for largest allowed deviation?" Tags="&lt;probability&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="1682" PostTypeId="2" ParentId="31" CreationDate="2010-08-14T07:52:35.467" Score="1" ViewCount="" Body="&lt;p&gt;What the p-value doesn't tell you is how likely it is that the null hypothesis is true.  Under the conventional (Fisher) significance testing framework we first compute the likelihood of observing the data assuming the null hypothesis is true, this is the p-value.  It seems intuitively reasonable then to assume the null hypothesis is probably false if the data are sufficiently unlikely to be observed under the null hypothesis.  This is entirely reasonable.  Statisticians tranditionally use a threshold and &quot;reject the null hypothesis at the 95% significance level&quot; if (1 - p) &gt; 0.95; however this is just a convention that has proven reasonable in practice - it doesn't mean that there is less than 5% probability that the null hypothesis is false (and therefore a 95% probability that the alternative hypothesis is true).  One reason that we can't say this is that we have not looked at the alternative hypothesis yet.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Imaging a function f() that maps the p-value onto the probability that the alternative hypothesis is true.  It would be reasonable to assert that this function is strictly decreasing (such that the more likely the observations under the null hypothesis, the less likely the alternative hypothesis is true), and that it gives values between 0 and 1 (as it gives an estimate of probability).  However, that is all that we know about f(), so while there is a relationship between p and the probability that the alternative hypothesis is true, it is uncalibrated.  This means we cannot use the p-value to make quantitative statements about the plausibility of the nulll and alternatve hypotheses.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Caveat lector: It isn't really within the frequentist framework to speak of the probability that a hypothesis is true, as it isn't a random variable - it is either true or it isn't.  So where I have talked of the probability of the truth of a hypothesis I have implicitly moved to a Bayesian interpretation.  It is incorrect to mix Bayesian and frequentist, however there is always a temptation to do so as what we really want is an quantative indication of the relative plausibility/probability of the hypotheses.  But this is not what the p-value provides.&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2010-08-14T07:52:35.467" />
  <row Id="1683" PostTypeId="2" ParentId="596" CreationDate="2010-08-14T08:12:41.753" Score="1" ViewCount="" Body="&lt;p&gt;I think semi-supervised methods may be what you are looking for, there is quite a lot of litterature on this in Machine learning.  There is a good &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0262514125&quot; rel=&quot;nofollow&quot;&gt;book&lt;/a&gt; on this topic, which gives a good idea of recent developments in this area.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An E.M. algorithm for logistic regression (a discriminative model) is easily implemented as follows:  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(i) Train a LR model using only the labelled data.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(ii) Use the LR model to assign labels to the unlabelled data.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(iii) Train a new LR model using both the labelled and unlabelled data (using the predicted labels).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(iv) repeat (ii) and (iii) until convergence is reached (i.e. none of the predicted labels for the unlabelled examples change).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This works quite well for some problems (e.g. text classification), not so well in others.  EM also works well with naieve Bayes.  McLachlan's excellent &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0471691151&quot; rel=&quot;nofollow&quot;&gt;book&lt;/a&gt; on discriminant analysis also has some material on basic algorithms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;HTH&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2010-08-14T08:12:41.753" />
  <row Id="1684" PostTypeId="2" ParentId="1676" CreationDate="2010-08-14T08:51:32.627" Score="2" ViewCount="" Body="&lt;p&gt;If you are working with Latex, I recommend &lt;strong&gt;TikZ Device&lt;/strong&gt; for outputting nice, Latex-formatted (like PSTricks) graphics. The output you get is text-based Latex code, which can be embedded with include(filename) into any figure environment. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Pros: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Same font in graphics as in your text&lt;/li&gt;&#xA;&lt;li&gt;Professional look&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Cons:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Takes longer to compile than PNG or PDF&lt;/li&gt;&#xA;&lt;li&gt;for very complex R graphics, there are could be some display errors&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/Sharpie/RTikZDevice&quot; rel=&quot;nofollow&quot;&gt;https://github.com/Sharpie/RTikZDevice&lt;/a&gt; - Project, Packages available from CRAN and R-Forge&lt;/p&gt;&#xA;" OwnerUserId="939" LastEditorUserId="13" LastEditorDisplayName="" LastEditDate="2011-05-10T22:15:06.783" LastActivityDate="2011-05-10T22:15:06.783" CommentCount="2" />
  <row Id="1685" PostTypeId="2" ParentId="1383" CreationDate="2010-08-14T12:10:58.203" Score="0" ViewCount="" Body="&lt;p&gt;So, here's the example: &lt;a href=&quot;http://nishi.dreamhosters.com/u/book1bwt_de.txt&quot; rel=&quot;nofollow&quot;&gt;http://nishi.dreamhosters.com/u/book1bwt_de.txt&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Its a list of choices between 'd' and 'e' in coding of BWT output of a plaintext book.&#xA;(This is a practical task, think bzip2).&#xA;For the reference, current result is (reasonably good, but not really the best possible) &#xA;6087 bytes = 48686 bits (ie log-likelihood) for that string of 99054 bits&lt;/p&gt;&#xA;" OwnerUserId="799" LastActivityDate="2010-08-14T12:10:58.203" />
  <row Id="1686" PostTypeId="2" ParentId="1676" CreationDate="2010-08-14T13:06:45.767" Score="6" ViewCount="" Body="&lt;p&gt;In a narrow sense, R Core has a recommendation: the &quot;recommended&quot; packages.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Everything else depends on your data analysis tasks at hand, and I'd recommend the &lt;a href=&quot;http://cran.r-project.org/web/views&quot; rel=&quot;nofollow&quot;&gt;Task Views&lt;/a&gt; at CRAN.&lt;/p&gt;&#xA;" OwnerUserId="334" LastActivityDate="2010-08-14T13:06:45.767" CommunityOwnedDate="2010-08-14T13:06:45.767" CommentCount="1" />
  <row Id="1687" PostTypeId="1" AcceptedAnswerId="1762" CreationDate="2010-08-14T14:00:16.297" Score="2" ViewCount="142" Body="&lt;p&gt;The problem I’m trying to solve is “How do I figure out how much gunpowder should I put into a cartridge so that I can give myself a good probability of making the minimum power factor?”&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I compete in USPSA/IPSC which requires that a competitors rounds make a minimum power factor. Power Factor is computed to be the FLOOR(average bullet velocity * bullet weight) / 1000) where velocity is in feet per second, and bullet weight is in grains. Note the use of FLOOR. No rounding is done. Only the integral part of the computation is used. The higher the power factor, the higher the felt recoil and harder it is to quickly do follow up shots. Since the sport is about firing shots as quickly and as accurately as possible, the lower the recoil the better.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Different divisions within the sport have different power factor floors, but the particular division I compete in has a minimum of 165 Power Factor. The bullets I use are 180 gn bullets, and vary by about +/- 0.2 gn and is normally distributed. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What makes this an interesting problem (and move it out of my meager stats and probability skills) is the testing procedure during a major match. Random sample of 8 rounds are collected. Of the 8 rounds, one is taken apart and the bullet is weighed for use in the formula above. Next, 3 rounds are fired and the average velocity is used. If the resulting power factor is below the minimum, then another 3 rounds are fired. The average of the 3 fastest velocities from the 6 rounds fired is now used to compute the power factor. If the resulting power factor is still below the floor, then the shooter has the option of having the last round taken apart and weighed or the last round fired. If the bullet is taken apart and it is heavier than the first bullet, the heavier weight is used to compute the power factor. If the last round is fired, the average of the 3 fastest velocities from the 7 rounds fired is used to compute the power factor.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To add spice to this problem, not all chronographs used to measure bullet velocities are created equal. The chronograph industry acknowledges that there can be as much as +/- 4% variance between chronographs of different brands. Even more interesting is that the rules allow for the same chronograph used for a particular match to have +/- 4% variance over the duration of a match. I don’t know if either of these 4% variances are normally distributed or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With my own chronograph, I test batches of a particular gunpowder load to get the average velocity and standard deviation. After statistical analyses of many different batches, I’ve confirmed that this data is normally distributed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The way I’m currently determining my minimum load is by finding the load the gives me 165 &amp;lt; FLOOR( target * 179.9 / 1000) where target =  (average velocity - standard deviation) * fudge factor. For fudge factor, I’ve unscientifically chosen 1.04. The 0.04 is the 4% variance between chronographs, but ignores the day-to-day allowable variance. I chose to just subtract just 1 standard deviation because it’ll only be 16% of the time that one bullet will be below the floor. In my mind, the probability of all the first 3 bullets all going below the floor is 0.16^3 which less than half a percent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My specific questions are: Am I going about computing the target the right way? Should my fudge factor include another 4% for the day-to-day variance allowed? Is the 1 standard deviation too much or too little? How should I write the formula for my target? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit:&#xA;After Srikant's initial response below let me add a couple of focusing questions and notes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand figuring out the error due to my measurements. Not much problem there unless I get really sloppy with quality control or maintenance. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My grasp of probabilities is weak so please bear with me as I ask about computing probabilities:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) One of the key issues I need to deal with is figuring out how to correctly compute the probability for the testing process around the 7 rounds. It's pretty straight forward to me for the case of the first 3 rounds: (probability that the round is below the power flaw)^3. How do I account for the next 3 rounds and the last round?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can see computing the probabilities using combinations of bullets above or below the floor, but it's not quite a binary above or below. Let's assume that 6 rounds have been fired, and the average of the highest 3 rounds is 164.9. If the last round has at least 165.2, then the average of the highest 3 rounds out of 7 will be 165.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) The other issue I need to deal with is figuring out how to account for the 4% variance between my chrono and the match chrono, and how the match chrono is allowed to drift by 4% from day to day. Do I just assume the worst case and make sure that I'm at least 8% above 165 -- that is my rounds are shooting at least 179 power factor? Or do I try to assume some kind of normal distribution over the two 4% variances?&lt;/p&gt;&#xA;" OwnerUserId="937" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-29T00:18:36.073" LastActivityDate="2011-04-29T00:18:36.073" Title="Applied statistics to find the minimum load for power factor floor" Tags="&lt;probability&gt;&lt;standard-deviation&gt;" AnswerCount="2" />
  <row Id="1688" PostTypeId="2" ParentId="1676" CreationDate="2010-08-14T14:06:52.273" Score="4" ViewCount="" Body="&lt;p&gt;You can get user reviews of packages on &lt;a href=&quot;http://crantastic.org/reviews&quot; rel=&quot;nofollow&quot;&gt;crantastic&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="573" LastActivityDate="2010-08-14T14:06:52.273" CommunityOwnedDate="2010-08-14T14:06:52.273" />
  <row Id="1689" PostTypeId="2" ParentId="36" CreationDate="2010-08-14T17:42:41.177" Score="1" ViewCount="" Body="&lt;p&gt;Sperm count in males in Slovene villages and the number of bears (also in Slovenia) show a negative correlation. Some people find this very worrying. I'll try and get the study that did this.&lt;/p&gt;&#xA;" OwnerUserId="144" LastActivityDate="2010-08-14T17:42:41.177" CommunityOwnedDate="2010-08-16T13:01:42.613" />
  <row Id="1690" PostTypeId="2" ParentId="36" CreationDate="2010-08-14T17:59:20.630" Score="1" ViewCount="" Body="&lt;p&gt;The standard citation pointing out the correlation between the number of newborn babies and breeding-pairs of Stalks in West Germany is &lt;a href=&quot;http://nature.com/nature/journal/v332/n6164/abs/332495a0.html&quot; rel=&quot;nofollow&quot;&gt;&lt;em&gt;A new parameter for sex education&lt;/em&gt;, Nature 332, 495 (07 April 1988); doi:10.1038/332495a0&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="942" LastActivityDate="2010-08-14T17:59:20.630" CommunityOwnedDate="2010-08-16T13:01:42.613" />
  <row Id="1691" PostTypeId="2" ParentId="1687" CreationDate="2010-08-14T18:32:47.380" Score="0" ViewCount="" Body="&lt;p&gt;You have a complex statistical problem and a complete analysis would be too long. However, I will suggest one idea that may perhaps help you to some extent. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You have already performed some calibration tests to assess the mean and standard deviation of the velocity of a bullet. However, I suspect that either your testing or your calculations or both are sub-optimal for the reasons I mention below.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The crucial point is this: The measured velocity of a bullet is a random variable that depends on three factors: (a) amount of gunpowder in the bullet, (b) random factors due to imperfections of the gun, and (c) errors because of chronograph inaccuracy. In other words, you can model the measured velocity of a bullet as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$v_m = v(gp) + \epsilon_g + \epsilon_c$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$v_m$ is the measured velocity,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$v(gp)$ is the true velocity of the bullet given that the bullet has $gp$ amount of gunpowder,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\epsilon_g \sim N(0,\sigma_g^2)$ is the random error that arises due to imperfections of the gun and&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\epsilon_c \sim N(0,\sigma_c^2)$ is the error induced by the chronograph.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think assuming that the errors are normally distributed with a mean zero and a finite variance is reasonable. Thus, we have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$E(v_m) = v(gp)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Var(v_m) = \sigma_g^2 + \sigma_c^2$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The goal of testing is to get a sense of the average velocity and the standard deviation of the velocity of a bullet. You could simply compute the mean of the observed velocities to get a sense of the mean as $E(v_m) = v(g)$ but you cannot use the standard deviation of the observed velocities for setting your target as the the errors induced by the chronograph and the errors due to the imperfections of the gun are confounded.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, you need to design your test to disentangle the variation induced by the chronograph and the intrinsic variation because of other factors. One way to achieve the above disentanglement is to perform a test as follows:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Start test 1 and choose a certain amount of gunpowder that you will fill up each bullet for this test.&lt;/li&gt;&#xA;&lt;li&gt;Load the bullet with the amount of gunpowder you chose for this test in step 1.&lt;/li&gt;&#xA;&lt;li&gt;Measure the velocity of the bullet.&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Repeat steps 2 and 3 for a certain number of bullets.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Perform several tests along the above lines ensuring that you choose a different amount of gunpowder for each test.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Let:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$i$ index the bullets,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$j$ index the test.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then you have a set of velocities indexed $v_{ij}$. Then, you can calculate the variance of the velocity (and hence the standard deviation) as follows:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Calculate the variance of &lt;strong&gt;all&lt;/strong&gt; velocity measurements. Denote this number by $S_v^2$.&lt;/li&gt;&#xA;&lt;li&gt;Calculate the variance of velocities for each test separately. Denote each test specific variance by $S_j^2$. Note that each of these variance measures is a measure of how much variation is induced because of the chronograph as the amount of gunpowder is the same for each test.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Then a measure of the variance of the velocity that you can attribute to the gun itself would be:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$S_g^2 = S_v^2 - \frac {\sum_j S_j^2}{J}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you know $S_g$ you can use your existing formula and there is no need to worry about the additional variation induced in your measurements because of your own chronograph as that has been 'taken care of' in our analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope that helps to some extent.&lt;/p&gt;&#xA;" LastActivityDate="2010-08-14T18:32:47.380" CommentCount="1" />
  <row Id="1692" PostTypeId="2" ParentId="36" CreationDate="2010-08-14T19:50:51.313" Score="2" ViewCount="" Body="&lt;p&gt;Although it's more of an illustration of the problem of multiple comparisons, it is also a good example of misattributed causation:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.bmj.com/cgi/content/abstract/337/dec17_2/a2768&quot; rel=&quot;nofollow&quot;&gt;Rugby (the religion of Wales) and its influence on the Catholic church: should Pope Benedict XVI be worried?&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&quot;every time Wales win the rugby grand slam, a Pope dies, except for 1978 when Wales were really good, and two Popes died.&quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="495" LastActivityDate="2010-08-14T19:50:51.313" CommunityOwnedDate="2010-08-16T13:01:42.613" />
  <row Id="1693" PostTypeId="2" ParentId="1674" CreationDate="2010-08-14T21:05:33.317" Score="1" ViewCount="" Body="&lt;p&gt;UPDATE&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tough crowd.  :)  For a concise account of connecting the trace of the Fisher matrix to surface area, please see section 4 (&quot;Isoperimetric Inequalities&quot;) in the paper below.  The crucial part is establishing the relation between differential entropy and the trace of the Fisher matrix, which the authors prove in the appendix.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://statistics.stanford.edu/~ckirby/techreports/NSF/COV%20NSF%2048.pdf&quot; rel=&quot;nofollow&quot;&gt;On the similarity of the entropy power inequality and the Brunn-Minkowski inequality&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;The basic intuition is through the isoperimetric inequality for the surface area of a sphere maximizing the volume.  We can arrive at a similar relationship concerning the trace of the Fisher information matrix and the entropy w.r.t the Gaussian.  The following may be helpful.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.35.7743&quot; rel=&quot;nofollow&quot;&gt;Information Theoretic Inequalities for Contoured Probability Distributions&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="251" LastEditorUserId="251" LastEditorDisplayName="" LastEditDate="2010-08-17T04:25:48.237" LastActivityDate="2010-08-17T04:25:48.237" CommentCount="1" />
  <row Id="1694" PostTypeId="2" ParentId="534" CreationDate="2010-08-14T21:19:58.003" Score="5" ViewCount="" Body="&lt;p&gt;Sir Austin Bradford Hill's President's Address to the Royal Society of Medicine (&lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1898525/?tool=pubmed&quot; rel=&quot;nofollow&quot;&gt;The Environment and Disease: Association or Causation?&lt;/a&gt;) explains nine criteria which help to judge whether there is a causal relationship between two correlated or associated variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;They are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Strength of the association&lt;/li&gt;&#xA;&lt;li&gt;Consistency: &quot;has it been repeatedly&#xA;observed by different persons, in&#xA;different places, cirumstances and&#xA;times?&quot; &lt;/li&gt;&#xA;&lt;li&gt;Specificity  &lt;/li&gt;&#xA;&lt;li&gt;Temporality: &quot;which is the cart and&#xA;which is the horse?&quot; - the cause&#xA;must precede the effect &lt;/li&gt;&#xA;&lt;li&gt;Biological gradient (dose-response&#xA;curve) - in what way does the&#xA;magnitude of the effect depended&#xA;upon the magnitude of the (suspected) causal variable? &lt;/li&gt;&#xA;&lt;li&gt;Plausibility - is there a likely&#xA;explanation for causation?  &lt;/li&gt;&#xA;&lt;li&gt;Coherance - would causation&#xA;contradict other established facts?&lt;/li&gt;&#xA;&lt;li&gt;Experiment - does experimental&#xA;manipulation of the (suspected)&#xA;causal variable affect the&#xA;(suspected) dependent variable &lt;/li&gt;&#xA;&lt;li&gt;Analogy - have we encountered&#xA;similar causal relationships in the&#xA;past?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="942" LastActivityDate="2010-08-14T21:19:58.003" />
  <row Id="1695" PostTypeId="2" ParentId="1676" CreationDate="2010-08-14T21:52:11.520" Score="4" ViewCount="" Body="&lt;p&gt;I would suggest using some of the packages provided by &lt;a href=&quot;http://www.revolutionanalytics.com/&quot; rel=&quot;nofollow&quot;&gt;revolution R&lt;/a&gt;. In particular, I quite like the:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://cran.r-project.org/web/packages/multicore/index.html&quot; rel=&quot;nofollow&quot;&gt;multicore&lt;/a&gt; package for parallel computing using shared memory processors&lt;/li&gt;&#xA;&lt;li&gt;there optimized packages for &lt;a href=&quot;http://www.revolutionanalytics.com/why-revolution-r/benchmarks.php&quot; rel=&quot;nofollow&quot;&gt;matrices&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="8" LastActivityDate="2010-08-14T21:52:11.520" CommunityOwnedDate="2010-08-14T21:52:11.520" />
  <row Id="1696" PostTypeId="2" ParentId="44" CreationDate="2010-08-14T21:55:32.190" Score="3" ViewCount="" Body="&lt;p&gt;I would show them the raw data of &lt;a href=&quot;http://en.wikipedia.org/wiki/Anscombe%27s_quartet&quot; rel=&quot;nofollow&quot;&gt;Anscombe's Quartet&lt;/a&gt; (&lt;a href=&quot;http://jstor.org/stable/pdfplus/2682899.pdf&quot; rel=&quot;nofollow&quot;&gt;JSTOR link to the paper&lt;/a&gt;) in a big table, alongside another table showing the Mean &amp;amp; Variance of x and y, the correlation coefficient, and the equation of the linear regression line. Ask them to explain the differences between each of the 4 datasets. They will be confused.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then show them 4 graphs. They will be enlightened.  &lt;/p&gt;&#xA;" OwnerUserId="942" LastActivityDate="2010-08-14T21:55:32.190" CommentCount="1" />
  <row Id="1697" PostTypeId="2" ParentId="1676" CreationDate="2010-08-15T00:31:42.537" Score="1" ViewCount="" Body="&lt;p&gt;Jeromy mentioned my first pick: Lattice.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also have found the &lt;code&gt;doBy&lt;/code&gt; package and its &lt;code&gt;summaryBy&lt;/code&gt; function to be insanely useful.  They extend &lt;code&gt;aggregate&lt;/code&gt; with a formula syntax that lets you aggregate multiple funtions simulatenously in non-trivial ways.  Great if you want, say, mean, std. dev., and length.&lt;/p&gt;&#xA;" OwnerUserId="389" LastActivityDate="2010-08-15T00:31:42.537" CommunityOwnedDate="2010-08-15T00:31:42.537" />
  <row Id="1698" PostTypeId="2" ParentId="534" CreationDate="2010-08-15T00:35:01.157" Score="1" ViewCount="" Body="&lt;p&gt;A related question might be -- under what conditions can you reliably extract causal relations from data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A 2008 NIPS &lt;a href=&quot;http://jmlr.csail.mit.edu/proceedings/papers/v6/&quot; rel=&quot;nofollow&quot;&gt;workshop&lt;/a&gt; try to address that question empirically. One of the tasks was to infer the direction of causality from observations of pairs of variables where one variable was known to cause another, and the best method was able to correctly extract causal direction 80% of the time.&lt;/p&gt;&#xA;" OwnerUserId="511" LastEditorUserId="511" LastEditorDisplayName="" LastEditDate="2010-08-15T00:40:38.993" LastActivityDate="2010-08-15T00:40:38.993" />
  <row Id="1699" PostTypeId="1" AcceptedAnswerId="1706" CreationDate="2010-08-15T03:10:05.873" Score="4" ViewCount="399" Body="&lt;p&gt;I am trying to solve for an efficient portfolio in R. How do I translate my constraints for a tangency point for 2 risky asset portfolio, and a given risk free rate to R solve.QP function? So basically I have the following equations:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;w = weight of the first risky asset&#xA;R1 = mean return of the first risky asset&#xA;R2 = mean return of the second risky asset&#xA;sd1 = sdev of first risky asset&#xA;sd2 = sdev of second risky asset&#xA;corr = correlation between two risky assets&#xA;rf = risk free rate&#xA;Return of portfolio, R = R2*(1-w)+R1*w&#xA;Standard Dev of portfolio, SD = sqrt((sd1*w)^2+(sd2*(1-w))^2+2*w*(1-w)*corr*sd1*sd2)&#xA;&#xA;Now I need to maximize R-rf while minimizing SD (that is maximize my sharpe). &#xA;Let sigma be covariance matrix. So my function to minimize is W^T*sigma*W where W is&#xA;the weights vector. Now simulataneously I need to maximize the excess return (R-rf)&#xA;and W^T*1=1. I don't know how to express that in the constraints function.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am confused how to express these constraints as expected by &lt;a href=&quot;http://pbil.univ-lyon1.fr/library/quadprog/html/solve.QP.html&quot; rel=&quot;nofollow&quot;&gt;http://pbil.univ-lyon1.fr/library/quadprog/html/solve.QP.html&lt;/a&gt; . If you could also point me to a solved derivation of the final formula, that would be helpful as well as I am unable to get to the final formula.&lt;/p&gt;&#xA;" OwnerUserId="862" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-30T21:19:46.560" LastActivityDate="2010-09-30T21:19:46.560" Title="Tangency portfolio in R" Tags="&lt;r&gt;&lt;finance&gt;&lt;maximum&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="1700" PostTypeId="2" ParentId="278" CreationDate="2010-08-15T05:29:00.713" Score="4" ViewCount="" Body="&lt;p&gt;What you're discovering is a degree of instability in either the algorithm or the data itself. The approach termed 'consensus' or 'ensemble' clustering is a way of dealing with the problem. The problem there is: given a collection of clusterings, find a &quot;consensus&quot; clustering that is in some sense the &quot;average&quot; of the clusterings. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There's a fair bit of work on this topic, and a good place to start is the &lt;a href=&quot;http://strehl.com/download/strehl-jmlr02.pdf&quot; rel=&quot;nofollow&quot;&gt;clustering ensembles paper by Strehl and Ghosh&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="139" LastActivityDate="2010-08-15T05:29:00.713" />
  <row Id="1701" PostTypeId="2" ParentId="278" CreationDate="2010-08-15T06:13:47.870" Score="1" ViewCount="" Body="&lt;p&gt;Which flat-clustering algorithm are you using? It might also be the case that the different results are because maybe it's not your data but your algorithm itself is non-deterministic (e.g., using K-means with random initialization, or using a model-based clustering with EM or MCMC for inference with random initialization)?&lt;/p&gt;&#xA;" OwnerUserId="881" LastActivityDate="2010-08-15T06:13:47.870" CommentCount="1" />
  <row Id="1702" PostTypeId="2" ParentId="1668" CreationDate="2010-08-15T08:59:45.287" Score="2" ViewCount="" Body="&lt;p&gt;You've mentioned some ML techniques, so two quite nice books (quite because unfortunately my favorite is in Polish):&lt;br&gt;&#xA;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1420067184&quot; rel=&quot;nofollow&quot;&gt;http://www.amazon.com/Machine-Learning-Algorithmic-Perspective-Recognition/dp/1420067184&lt;/a&gt;&lt;br&gt;&#xA;&lt;a href=&quot;http://ai.stanford.edu/~nilsson/mlbook.html&quot; rel=&quot;nofollow&quot;&gt;http://ai.stanford.edu/~nilsson/mlbook.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For numeric stuff like random number generation:&lt;br&gt;&#xA;&lt;a href=&quot;http://www.nr.com/&quot; rel=&quot;nofollow&quot;&gt;http://www.nr.com/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-08-15T08:59:45.287" CommunityOwnedDate="2010-08-15T08:59:45.287" CommentCount="1" />
  <row Id="1706" PostTypeId="2" ParentId="1699" CreationDate="2010-08-15T12:30:19.470" Score="2" ViewCount="" Body="&lt;p&gt;I haven't looked at your code yet, but here are two pointers:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Rmetrics has the tangencyPortfolio function in the fPortfolio package: &lt;a href=&quot;http://help.rmetrics.org/fPortfolio/html/class-fPORTFOLIO.html&quot; rel=&quot;nofollow&quot;&gt;http://help.rmetrics.org/fPortfolio/html/class-fPORTFOLIO.html&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Here is a solution from David Ruppert's &quot;Statistics and Finance&quot; book: &lt;a href=&quot;http://www.stat.tamu.edu/~ljin/Finance/chapter5/Fig5_9.txt&quot; rel=&quot;nofollow&quot;&gt;http://www.stat.tamu.edu/~ljin/Finance/chapter5/Fig5_9.txt&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-08-15T12:30:19.470" CommentCount="1" />
  <row Id="1708" PostTypeId="1" CreationDate="2010-08-15T14:10:53.410" Score="2" ViewCount="246" Body="&lt;p&gt;I have weights of SNP variation (output through Eigenstrat program) for each SNP for the three main PCs. I wish to reduce my list of SNPs to those that show maximum differentiation between the three PCs. Can anyone help me with which statistical method to use to do this. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;say, if each PC describes the magnitude of variation, how can I find mutually exclusive rows or rows that define the strongest differenciator of each PC.&#xA;e.g:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;SNPNam PC1-wt PC 2-wt PC3-wt&lt;br&gt;&#xA;  SNP_1 -1.489 -0.029 -0.507&lt;br&gt;&#xA;  SNP_2 -1.446 -0.816 0.661&lt;br&gt;&#xA;  SNP_3 -1.416 0.338 1.631&lt;br&gt;&#xA;  SNP_4 -1.392 -1.452 0.062&lt;br&gt;&#xA;  SNP_5 -1.278 0.362 -1.006&lt;br&gt;&#xA;  SNP_6 -1.21 0.514 0.144&lt;br&gt;&#xA;  SNP_7 -1.119 -0.633 0.163&lt;br&gt;&#xA;  SNP_8 -1.112 -0.193 -0.256&lt;br&gt;&#xA;  SNP_9 -1.054 1.081 -1.519&lt;br&gt;&#xA;  SNP_10 -0.936 -1.052 -0.419&lt;br&gt;&#xA;  SNP_11 -0.861 0.381 -0.207&lt;br&gt;&#xA;  SNP_12 -0.662 0.852 -0.211&lt;br&gt;&#xA;  SNP_13 -0.503 -1.602 0.585&lt;br&gt;&#xA;  SNP_14 -0.417 0.529 -1.003&lt;br&gt;&#xA;  SNP_15 0.101 -0.85 -0.258&lt;br&gt;&#xA;  SNP_16 0.198 -0.435 -1.599&lt;br&gt;&#xA;  SNP_17 0.588 -1.292 -1.257&lt;br&gt;&#xA;  SNP_18 1.167 0.891 1.106&lt;br&gt;&#xA;  SNP_19 1.35 0.036 0.729&lt;br&gt;&#xA;  SNP_20 1.532 1.599 0.499  &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Any help regarding which test to perform and how to (which package) is very much appreciated.&lt;/p&gt;&#xA;" OwnerUserId="952" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-03-25T20:32:46.083" LastActivityDate="2011-03-25T20:32:46.083" Title="Variation in PCA weights" Tags="&lt;pca&gt;&lt;genetics&gt;" AnswerCount="1" CommentCount="3" FavoriteCount="1" />
  <row Id="1709" PostTypeId="1" AcceptedAnswerId="1717" CreationDate="2010-08-15T14:30:23.597" Score="4" ViewCount="252" Body="&lt;p&gt;I have collected positional data. To visualize the data, I'd like to draw a 'typical' outcome of an experiment. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data comes from a few hundred experiments, where I identify a variable number of objects at different positions relative to the origin in 2D. Thus, I can calculate the average number of objects, as well as estimate the empirical distribution of the objects. A plot of the 'typical' outcome would then have the average (or possibly mode) number of objects, say, 5. What I'm not sure about is where to position these 5 objects.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To simplify the problem, assume that the data follows a 2D normal distribution. If I were just to randomly draw 5 points from the distribution, I might get one point at [3,3], which would be a very rare outcome, and would thus not reflect the 'typical', or 'average' outcome. However, just drawing 5 points at [0,0] would also not make sense - even though [0,0] is the average position of the objects, 5 overlapping points are not an 'average' outcome of the process, either.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words, how can I get a 'likely' draw from a distribution?&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It looks like I should mention &lt;em&gt;why&lt;/em&gt; I don't want to use the usual methods (like a 2D smoothed histogram, or plotting all the many points) to look at the 2D distribution. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The objects (which are vesicles (i.e. little spheres) inside cells) vary in number, size and position (distribution of the distance from the cell center, amount of clustering). I would like to display all these features in one graph. Since there are several hundred cells containing many vesicles each, it is not very useful to combine them all in a single plot. I am well aware that I could use a multipanel graph showing the distributions of all parameters, but this would be a lot less intuitive. &lt;/li&gt;&#xA;&lt;li&gt;I would like to show a 'typical' cell that shows all the salient features that characterize a specific phenotype. This way, if I want to image a particular phenotype in a mixed population, I know what kind of cell I'm looking for.&lt;/li&gt;&#xA;&lt;li&gt;I think such a plot would be a cool way to display a lot of information at once, and I just want to try.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Maybe it would be clearer If I said that I want to simulate a likely experimental result based on my measurements?&lt;/p&gt;&#xA;" OwnerUserId="198" LastEditorUserId="198" LastEditorDisplayName="" LastEditDate="2010-08-15T20:04:03.057" LastActivityDate="2010-08-16T10:40:58.773" Title="How to draw a probable outcome from a distribution?" Tags="&lt;distributions&gt;&lt;data-visualization&gt;" AnswerCount="4" CommentCount="5" />
  <row Id="1710" PostTypeId="2" ParentId="36" CreationDate="2010-08-15T15:33:31.153" Score="1" ViewCount="" Body="&lt;p&gt;I've recently been to a conference and one of the speakers gave this very interesting example (although the point was to illustrate something else):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Americans and English eat a lot of fat food. There is a high rate of cardiovascular diseases in US and UK.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;French eat a lot of fat food, but they have a low(er) rate of cardiovascular diseases.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Americans and English drink a lot of alcohol. There is a high rate of cardiovascular diseases in US and UK.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Italians drink a lot of alcohol but, again, they have a low(er) rate of cardiovascular diseases.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The conclusion? Eat and drink what you want.&#xA;And you have a higher chance of getting a heart attack if you speak English!&lt;/p&gt;&#xA;" OwnerUserId="582" LastActivityDate="2010-08-15T15:33:31.153" CommunityOwnedDate="2010-08-16T13:01:42.613" CommentCount="1" />
  <row Id="1711" PostTypeId="2" ParentId="1709" CreationDate="2010-08-15T15:47:36.260" Score="0" ViewCount="" Body="&lt;p&gt;One thing that you could do is to plot the position of all your experiments in the 2D plane, one point for each object, maybe colored by experiment (if you have a lot of experiments you may just plot a random subset of them).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If there is a pattern in the position of the objects it should emerge when doing this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, depending on what you are measuring, maybe is not the absolute position that counts but the relative position of the objects. In that case you could rotate the positions around the origin so that for each experiment the first point always lies, for instance, on the x axis.&lt;/p&gt;&#xA;" OwnerUserId="582" LastActivityDate="2010-08-15T15:47:36.260" />
  <row Id="1712" PostTypeId="2" ParentId="1709" CreationDate="2010-08-15T16:05:02.070" Score="0" ViewCount="" Body="&lt;p&gt;Maybe you could use a &lt;a href=&quot;http://addictedtor.free.fr/graphiques/RGraphGallery.php?graph=139&quot; rel=&quot;nofollow&quot;&gt;smoothed scatterplot&lt;/a&gt;? It is an analogy to kernel density approximation, but in 2D.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-08-15T16:05:02.070" />
  <row Id="1713" PostTypeId="1" AcceptedAnswerId="1738" CreationDate="2010-08-15T17:01:49.620" Score="5" ViewCount="364" Body="&lt;p&gt;For some measurements, the results of an analysis are appropriately presented on the transformed scale. In most of the cases, however, it's desirable to present the results on the original scale of measurement (otherwise your work is more or less worthless). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, in case of log-transformed data, a problem with interpretation on the original scale arises because the mean of the logged values is not the log of the mean. Taking the antilogarithm of the estimate of the mean on the log scale does not give an estimate of the mean on the original scale.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If, however, the log-transformed data have symmetric distributions, the following&#xA;relationships hold (since the log preserves ordering):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Mean[log (Y)] = Median[log (Y)] = log[Median (Y)]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(the antilogarithm of the mean of the log values is the median on the original scale of measurements).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I only can make inferences about the difference (or the ratio) of the medians on the original scale of measurement.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Two-sample t-tests and confidence intervals are most reliable if the populations are roughly normal with approximately standard deviations, so we may be tempted to use the &lt;code&gt;Box-Cox&lt;/code&gt; transformation for the normality assumption to hold (I also think that it is a variance stabilizing transformation too). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, if we apply t-tools to &lt;code&gt;Box-Cox&lt;/code&gt; transformed data , we will get inferences about the difference in means of the transformed data. How can we interpret those on the original scale of measurement? (The mean of the transformed values is not the transformed mean). In other words, taking the inverse transform of the estimate of the mean, on the transformed scale, does not give an estimate of the mean on the original scale.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can I also make inferences only about the medians in this case? Is there a transformation that will allow me to go back to the means (on the original scale) ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;This question was initially posted as a comment &lt;a href=&quot;http://stats.stackexchange.com/questions/1601/normalizing-transformation-options/1607#1607&quot;&gt;here&lt;/a&gt;&lt;/em&gt; &lt;/p&gt;&#xA;" OwnerUserId="339" LastActivityDate="2010-08-16T13:42:04.937" Title="Express answers in terms of original units, in Box-Cox transformed data." Tags="&lt;data-transformation&gt;&lt;confidence-interval&gt;&lt;t-test&gt;&lt;interpretation&gt;" AnswerCount="3" FavoriteCount="2" />
  <row Id="1714" PostTypeId="2" ParentId="196" CreationDate="2010-08-15T17:26:03.560" Score="1" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://astrophysics.arc.nasa.gov/~pgazis/viewpoints.htm&quot; rel=&quot;nofollow&quot;&gt;Viewpoints&lt;/a&gt; is useful for multi-variate data sets.&lt;/p&gt;&#xA;" OwnerUserId="957" LastActivityDate="2010-08-15T17:26:03.560" CommunityOwnedDate="2010-08-15T17:26:03.560" CommentCount="1" />
  <row Id="1715" PostTypeId="2" ParentId="726" CreationDate="2010-08-15T18:36:58.477" Score="2" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;To understand God's Thoughts&#xA;  we must study statistics&#xA;  for these are the measure&#xA;  of His purpose.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;--Florence Nightingale&lt;/p&gt;&#xA;" LastEditorUserId="795" LastEditorDisplayName="" LastEditDate="2010-12-03T04:05:19.333" LastActivityDate="2010-12-03T04:05:19.333" CommunityOwnedDate="2010-08-15T18:36:58.477" />
  <row Id="1716" PostTypeId="2" ParentId="1709" CreationDate="2010-08-15T18:48:27.970" Score="1" ViewCount="" Body="&lt;p&gt;To summarise (please correct me if I'm wrong):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;You have a set of points for a number of parameters/states.&lt;/li&gt;&#xA;&lt;li&gt;The points provide a joint distribution of the parameters states&lt;/li&gt;&#xA;&lt;li&gt;You want to simulate from a model using some typical states.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The problem you have is that you can't write down a nice closed form density.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To tackle this problem you should use a particle filter. Suppose your model of a cell was this simple ODE:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{equation}&#xA;\frac{dX(t)}{dt} = \lambda X(t)&#xA;\end{equation}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and your data consists of values of $\lambda$ and $X(0)$. Put this data in a matrix with two columns and $n$ rows, where $n$ is the number of points. Then&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Choose a &lt;strong&gt;row&lt;/strong&gt; at random, to get a particular values of $\lambda$ and $X(0)$&lt;/li&gt;&#xA;&lt;li&gt;Optional step: perturb your parameters with noise.&lt;/li&gt;&#xA;&lt;li&gt;Simulate from your model, in this case the ODE.&lt;/li&gt;&#xA;&lt;li&gt;Repeat as necessary.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The key point is that step 1 is draw from the joint density of the $\lambda$ and $X(0)$. &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;This answer could be way off if I've misinterpreted what you mean about simulating from the model. Please correct me if I'm wrong.&lt;/p&gt;&#xA;" OwnerUserId="8" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-08-16T09:31:44.377" LastActivityDate="2010-08-16T09:31:44.377" CommentCount="1" />
  <row Id="1717" PostTypeId="2" ParentId="1709" CreationDate="2010-08-15T19:14:50.803" Score="2" ViewCount="" Body="&lt;p&gt;I also think that it's not clear what you want. But if you want a set of &lt;em&gt;deterministically&lt;/em&gt; chosen points, so that they preserve the moments of the initial distribution, you can use the &lt;em&gt;sigma point&lt;/em&gt; selection method that applies to the &lt;a href=&quot;http://en.wikipedia.org/wiki/UKF#Unscented_Kalman_filter&quot; rel=&quot;nofollow&quot;&gt;unscented Kalman filter&lt;/a&gt;.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say that you want to select $2L+1$ points that fulfill those requirements. Then proceed in the following way:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\mathcal{X}_0=\overline{x} \qquad w_0=\frac{\kappa}{L+\kappa} \qquad i=0$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\mathcal{X}_i=\overline{x}+\left(\sqrt{(\:L+\kappa\:)\:\mathbf{P}_x}\right)_i \qquad w_i=\frac{1}{2(L+\kappa)} \qquad i=1, \dots,L$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\mathcal{X}_i=\overline{x}-\left(\sqrt{(\:L+\kappa\:)\:\mathbf{P}_x}\right)_i \qquad w_i=\frac{1}{2(L+\kappa)} \qquad i=L+1, \dots,2L$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $w_i$ the weight of the i-th point,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\kappa=3-L$ (in case of Normally distributed data),&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and $\left(\sqrt{(\:L+\kappa\:)\mathbf{P}_x}\right)_i$ is the i-th row (or column)* of the &lt;em&gt;matrix square root&lt;/em&gt; of the weighted covariance $(\:L+\kappa\:)\:\mathbf{P}_x$ matrix (usually given by the &lt;a href=&quot;http://en.wikipedia.org/wiki/Cholesky_decomposition&quot; rel=&quot;nofollow&quot;&gt;Cholesky decomposition&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;* If the &lt;em&gt;matrix square root&lt;/em&gt; $\mathbf{A}$ gives the original by giving $\mathbf{A}^T\mathbf{A}$, then use the &lt;em&gt;rows&lt;/em&gt; of $\mathbf{A}$. If it gives the original by giving $\mathbf{A}\mathbf{A}^T$, then use the &lt;em&gt;columns&lt;/em&gt; of $\mathbf{A}$. The result of the matlab function &lt;a href=&quot;http://www.mathworks.com/access/helpdesk/help/techdoc/ref/chol.html&quot; rel=&quot;nofollow&quot;&gt;chol()&lt;/a&gt; falls into the first category.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a simple example using R&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x &amp;lt;- rnorm(1000,5,2.5)&#xA;y &amp;lt;- rnorm(1000,2,1)&#xA;&#xA;P &amp;lt;- cov(cbind(x,y))&#xA;V0 &amp;lt;- c(mean(x),mean(y))&#xA;n &amp;lt;- 2;k &amp;lt;- 1&#xA;A &amp;lt;- chol((n+k)*P) # matrix square root&#xA;&#xA;points &amp;lt;- as.data.frame(sapply(1:(2*n),function(i) if (i&amp;lt;=n) A[i,] + V0 else -A[i-n,] + V0))&#xA;attach(points)&#xA;&#xA;#mean (equals V0)&#xA;1/(2*(n+k))*(V1+V2+V3+V4) + k/(n+k)*V0&#xA;#covariance (equals P)&#xA;1/(2*(n+k)) * ((V1-V0) %*% t(V1-V0) + (V2-V0) %*% t(V2-V0) + (V3-V0) %*% t(V3-V0) + (V4-V0) %*% t(V4-V0))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="339" LastEditorUserId="339" LastEditorDisplayName="" LastEditDate="2010-08-16T10:40:58.773" LastActivityDate="2010-08-16T10:40:58.773" CommentCount="2" />
  <row Id="1718" PostTypeId="2" ParentId="1308" CreationDate="2010-08-15T22:03:07.387" Score="3" ViewCount="" Body="&lt;p&gt;This is a counting problem: there are &lt;em&gt;b^n&lt;/em&gt; possible birthday assignments for &lt;em&gt;n&lt;/em&gt; people.  Of those, let &lt;em&gt;q(k; n, b)&lt;/em&gt; be the number of assignments for which no birthday is shared by more than &lt;em&gt;k&lt;/em&gt; people but at least one birthday actually is shared by &lt;em&gt;k&lt;/em&gt; people.  The probability we seek can be found by summing the &lt;em&gt;q(k;n,b)&lt;/em&gt; for appropriate values of &lt;em&gt;k&lt;/em&gt; and multiplying the result by &lt;em&gt;b^(-n)&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These counts can be found exactly for values of &lt;em&gt;n&lt;/em&gt; less than several hundred.  However, they will not follow any straightforward formula:  &lt;strong&gt;we have to consider the patterns of ways in which birthdays can be assigned&lt;/strong&gt;.  I will illustrate this in lieu of providing a general demonstration.  Let &lt;em&gt;n&lt;/em&gt; = 4 (this is the smallest interesting situation).  The possibilities are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Each person has a unique birthday; the code is {4}.&lt;/li&gt;&#xA;&lt;li&gt;Exactly two people share a birthday; the code is {2,1}.&lt;/li&gt;&#xA;&lt;li&gt;Two people have one birthday and the other two have another; the code is {0,2}.&lt;/li&gt;&#xA;&lt;li&gt;Three people share a birthday; the code is {1,0,1}.&lt;/li&gt;&#xA;&lt;li&gt;Four people share a birthday; the code is {0,0,0,1}.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Generally, the code {&lt;em&gt;a&lt;/em&gt;[1], &lt;em&gt;a&lt;/em&gt;[2], ...} is a tuple of counts whose &lt;em&gt;k&lt;/em&gt; th element stipulates how many distinct birthdates are shared by exactly &lt;em&gt;k&lt;/em&gt; people.  Thus, in particular,&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;em&gt;1 a[1] + 2a[2] + ... + k a[k] + ... = n&lt;/em&gt;.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Note, even in this simple case, that there are two ways in which the maximum of two people per birthday is attained: one with the code {0,2} and another with the code {2,1}.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can directly count the number of possible birthday assignments corresponding to any given code.  This number is the product of three terms.  One is a multinomial coefficient; it counts the number of ways of partitioning &lt;em&gt;n&lt;/em&gt; people into &lt;em&gt;a[1]&lt;/em&gt; groups of 1, &lt;em&gt;a[2]&lt;/em&gt; groups of 2, and so on.  Because the sequence of groups does not matter, we have to divide this multinomial coefficient by &lt;em&gt;a&lt;/em&gt;[1]!&lt;em&gt;a&lt;/em&gt;[2]!... ; its reciprocal is the second term.  Finally, line up the groups and assign them each a birthday: there are &lt;em&gt;b&lt;/em&gt; candidates for the first group, &lt;em&gt;b&lt;/em&gt;-1 for the second, and so on.  These values have to be multiplied together, forming the third term.  It is equal to the &quot;factorial product&quot; &lt;em&gt;b^((a[1]+a[2]+...))&lt;/em&gt; where &lt;em&gt;b^((m))&lt;/em&gt; means &lt;em&gt;b(b-1)...(b-m+1)&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is an obvious and fairly simple recursion relating the count for a pattern &lt;em&gt;{a[1], ..., a[k]}&lt;/em&gt; to the count for the pattern &lt;em&gt;{a[1], ..., a[k-1]}&lt;/em&gt;.  This enables rapid calculation of the counts for modest values of &lt;em&gt;n&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I doubt there is a closed form formula for  &lt;em&gt;q(k; n, b)&lt;/em&gt;, which is obtained by summing the counts for all partitions of &lt;em&gt;n&lt;/em&gt; whose maximum term equals &lt;em&gt;k&lt;/em&gt;.  Let me offer some examples:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With &lt;strong&gt;&lt;em&gt;b&lt;/em&gt; = 5&lt;/strong&gt; (five possible birthdays) and &lt;strong&gt;&lt;em&gt;n&lt;/em&gt; = 4&lt;/strong&gt; (four people), we obtain&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;em&gt;q&lt;/em&gt;(1) = &lt;em&gt;q&lt;/em&gt;(1;4,5) = 120&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;&lt;em&gt;q&lt;/em&gt;(2) = 360 + 60 = 420&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;&lt;em&gt;q&lt;/em&gt;(3) = 80&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;&lt;em&gt;q&lt;/em&gt;(4) = 5.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Whence, for example, the chance that three or more people out of four share the same &quot;birthday&quot; (out of 5 possible dates) equals (80 + 5)/625 = 0.136.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As another example, take &lt;strong&gt;&lt;em&gt;b&lt;/em&gt; = 365&lt;/strong&gt; and &lt;strong&gt;&lt;em&gt;n&lt;/em&gt; = 23&lt;/strong&gt;.  Here are the values of &lt;em&gt;q&lt;/em&gt;( &lt;em&gt;k&lt;/em&gt;;23,365) for the smallest &lt;em&gt;k&lt;/em&gt; (to six sig figs only):&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;em&gt;k&lt;/em&gt;=1: 0.49270&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;&lt;em&gt;k&lt;/em&gt;=2: 0.494592&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;&lt;em&gt;k&lt;/em&gt;=3: 0.0125308&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;&lt;em&gt;k&lt;/em&gt;=4: 0.000172844&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;&lt;em&gt;k&lt;/em&gt;=5: 1.80449E-6&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;&lt;em&gt;k&lt;/em&gt;=6: 1.48722E-8&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;&lt;em&gt;k&lt;/em&gt;=7: 9.92255E-11&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;&lt;em&gt;k&lt;/em&gt;=8: 5.45195E-13&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Using this technique, we can readily compute that there is about a 50% chance of (at least) a three-way birthday collision among 87 people, a 50% chance of a four-way collision among 187, and a 50% chance of a five-way collision among 310 people.  That last calculation starts taking a few seconds (in Mathematica, anyway) because the number of partitions to consider starts getting large.  For substantially larger &lt;em&gt;n&lt;/em&gt; we need an approximation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One approximation is obtained by means of the Poisson distribution with expectation &lt;em&gt;n/b&lt;/em&gt;, because we can view a birthday assignment as arising from &lt;em&gt;b&lt;/em&gt; almost (but not quite) independent Poisson variables each with expectation &lt;em&gt;n/b&lt;/em&gt;: the variable for any given possible birthday describes how many of the &lt;em&gt;n&lt;/em&gt; people have that birthday.  The distribution of the maximum is therefore approximately &lt;em&gt;F(k)^b&lt;/em&gt; where &lt;em&gt;F&lt;/em&gt; is the Poisson CDF.  This is not a rigorous argument, so let's do a little testing.  The approximation for &lt;strong&gt;&lt;em&gt;n&lt;/em&gt; = 23&lt;/strong&gt;, &lt;strong&gt;&lt;em&gt;b&lt;/em&gt; = 365&lt;/strong&gt; gives&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;em&gt;k&lt;/em&gt;=1: 0.498783&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;&lt;em&gt;k&lt;/em&gt;=2: 0.496803&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;&lt;em&gt;k&lt;/em&gt;=3: 0.014187&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;&lt;em&gt;k&lt;/em&gt;=4: 0.000225115&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;By comparing with the preceding you can see that the relative probabilities can be poor when they are small, but the absolute probabilities are reasonably well approximated to about 0.5%.  Testing with a wide range of &lt;em&gt;n&lt;/em&gt; and &lt;em&gt;b&lt;/em&gt; suggests the approximation is usually about this good.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To wrap up, let's consider the original question: take &lt;strong&gt;&lt;em&gt;n&lt;/em&gt; = 10,000&lt;/strong&gt; (the number of observations) and &lt;strong&gt;&lt;em&gt;b&lt;/em&gt; = 1,000,000&lt;/strong&gt; (the number of possible &quot;structures,&quot; approximately).  The approximate distribution for the maximum number of &quot;shared birthdays&quot; is&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;em&gt;k&lt;/em&gt;=1: 0&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;&lt;em&gt;k&lt;/em&gt;=2: 0.8475+&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;&lt;em&gt;k&lt;/em&gt;=3: 0.1520+&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;&lt;em&gt;k&lt;/em&gt;=4: .0004+&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;&lt;em&gt;k&lt;/em&gt;&gt;4: &amp;lt; 1E-6&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;(This is a fast calculation.)  Clearly, observing one structure 10 times out of 10,000 would be highly significant.  Because &lt;em&gt;n&lt;/em&gt; and &lt;em&gt;b&lt;/em&gt; are both large, I expect the approximation to work quite well here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Incidentally, as Shane intimated, simulations can provide useful checks.  A Mathematica simulation is created with a function like&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;simulate[n_, b_] :=  Max[Last[Transpose[Tally[RandomInteger[{0, b - 1}, n]]]]];&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;which is then iterated and summarized, as in this example which runs 10,000 iterations of the &lt;strong&gt;&lt;em&gt;n&lt;/em&gt; = 10,000&lt;/strong&gt;, &lt;strong&gt;&lt;em&gt;b&lt;/em&gt; = 1,000,000&lt;/strong&gt; case:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;Tally[Table[simulate[10000, 1000000], {n, 1, 10000}]] // TableForm&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Its output is&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;2 8503&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;3 1493&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;4 4&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;These frequencies closely agree with those predicted by the Poisson approximation.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-08-15T22:03:07.387" />
  <row Id="1719" PostTypeId="1" CreationDate="2010-08-15T22:51:09.993" Score="4" ViewCount="152" Body="&lt;p&gt;Well, I'm an engineer by day. Although most of my work revolves around modeling, we generally do pretty basic stuff. An &quot;Advanced&quot; model would be a monte carlo simulation validated using R2 tests. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently, in my field, there is a lot of research using Logistic and bayesian analysis. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is, which courses would you recommend someone to take from &lt;a href=&quot;http://ocw.mit.edu/courses/mathematics/&quot; rel=&quot;nofollow&quot;&gt;MIT's open course site&lt;/a&gt; or any other sites, for someone who learns best by video/audio first, and reading second?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What i'd like to learn are the following:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Be able to understand the models and when to employ them&lt;/li&gt;&#xA;&lt;li&gt;able to take in field data (which is generated once and cannot be regenerated) and design and perform experiments&lt;/li&gt;&#xA;&lt;li&gt;Able to understand the results, look at them, and figure out if something is off, &quot;show stopper&quot; or &quot;outliers&quot;, or if everything is fine and dandy&lt;/li&gt;&#xA;&lt;li&gt;Be able to validate and calibrate the model, to actual &quot;As-built&quot; results&lt;/li&gt;&#xA;&lt;li&gt;Be able to forecast the results using appropriate sensitivity analysis&lt;/li&gt;&#xA;&lt;li&gt;be able to forecast / &quot;plug&quot; missing data&lt;/li&gt;&#xA;&lt;li&gt;be able to write journal papers related to my field&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;my field in a nutshell is: transportation demand modeling for passenger vehicles, using either the generic four step model, or socio economic activity/tour based models such as PECAS or urbansim&lt;/p&gt;&#xA;" OwnerUserId="59" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-08-15T23:22:39.817" LastActivityDate="2010-08-16T07:14:37.997" CommunityOwnedDate="2010-08-16T12:03:55.803" Title="Video/Audio online material for getting into Bayesian analysis and logistic-regressions" Tags="&lt;bayesian&gt;&lt;logistic-regression&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="4" />
  <row Id="1720" PostTypeId="2" ParentId="1713" CreationDate="2010-08-15T23:35:52.447" Score="4" ViewCount="" Body="&lt;p&gt;If the Box-Cox transformation yields a symmetric distribution, then the mean of the transformed data is back-transformed to the median on the original scale. This is true for any monotonic transformation, including the Box-Cox transformations, the IHS transformations, etc. So inferences about the means on the transformed data correspond to inferences about the median on the original scale.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As the original data were skewed (or you wouldn't have used a Box-Cox transformation in the first place), why do you want inferences about the means? I would have thought working with medians would make more sense in this situation. I don't understand why this is seen as a &quot;problem with interpretation on the original scale&quot;.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-08-15T23:35:52.447" CommentCount="4" />
  <row Id="1721" PostTypeId="2" ParentId="1095" CreationDate="2010-08-15T23:39:20.410" Score="0" ViewCount="" Body="&lt;p&gt;So it turns out the first assumption was actually correct:  U is indeed the first k eigenvectors of C, that we calculate from G by means of the eigendecompposition $(X_tVD^{-\frac12})D^{1/2} = X_tV$.  &lt;/p&gt;&#xA;" OwnerUserId="282" LastEditorUserId="282" LastEditorDisplayName="" LastEditDate="2010-08-16T21:27:59.097" LastActivityDate="2010-08-16T21:27:59.097" />
  <row Id="1722" PostTypeId="2" ParentId="596" CreationDate="2010-08-15T23:43:45.330" Score="2" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.120.3681&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;This paper&lt;/a&gt; is very interesting, has good results, and is easy to apply to discriminative models. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;And the term is semi-supervised learning, not unsupervised learning.&lt;/p&gt;&#xA;" OwnerUserId="959" LastActivityDate="2010-08-15T23:43:45.330" />
  <row Id="1723" PostTypeId="2" ParentId="1645" CreationDate="2010-08-15T23:59:19.030" Score="4" ViewCount="" Body="&lt;p&gt;For normality, actual Shapiro-Wilk has good power in fairly small samples. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The main competitor in studies that I have seen is the more general Anderson-Darling, which does fairly well, but I wouldn't say it was better. If you can clarify what alternatives interest you, possibly a better statistic would be more obvious. [edit: if you estimate parameters, the A-D test should be adjusted for that.]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[I strongly recommend against considering Jarque-Bera in small samples (which probably better known as Bowman-Shenton in statistical circles - they studied the small sample distribution). The asymptotic joint distribution of skewness and kurtosis is &lt;em&gt;nothing like&lt;/em&gt; the small-sample distribution - in the same way a banana doesn't look much like an orange. It also has very low power against some interesting alternatives - for example it is powerless to pick up a symmetric &lt;em&gt;bimodal&lt;/em&gt; distribution that has kurtosis close to that of a normal distribution.]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Frequently people test goodness of fit for what turn out to be not-particularly-good reasons, or they're answering a question other than the one that they actually want to answer. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, you almost certainly already know your data aren't really normal (not exactly), so there's no point in trying to answer a question you know the answer to - and the hypothesis test &lt;em&gt;doesn't actually answer it anyway&lt;/em&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given you know you don't have exact normality already, your hypothesis test of normality is really giving you an answer to a question closer to &quot;is my sample size large enough to pick up the amount of non-normality that I have&quot;, while the real question you're interested in answering is usually closer to &quot;what is the &lt;em&gt;impact&lt;/em&gt; of this non-normality on these other things I'm interested in?&quot;. The hypothesis test is measuring sample size, while the question you're interested in answering is not very dependent on sample size.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are times when testing of normality makes some sense, but those situations almost never occur with small samples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why are you testing normality?&lt;/p&gt;&#xA;" OwnerUserId="805" LastEditorUserId="805" LastEditorDisplayName="" LastEditDate="2010-08-18T00:52:39.357" LastActivityDate="2010-08-18T00:52:39.357" CommentCount="4" />
  <row Id="1724" PostTypeId="2" ParentId="1713" CreationDate="2010-08-16T00:30:59.557" Score="3" ViewCount="" Body="&lt;p&gt;If you want to do inference about means on the original scale, you could consider using inference that doesn't use a normality assumption. Take care, however. Simply plugging through a straight comparison of means via say resampling (either permutation tests or bootstrapping) when the two samples have different variances may be a problem if your analysis assumes the variances to be equal (and equal variances on the transformed scale will be difference variances on the original scale if the means differ). Such techniques don't avoid the necessity to think about what you're doing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another approach to consider if you're more interested in estimation or prediction than testing is to use a Taylor expansion of the transformed variables to compute the approximate mean and variance after transforming back - where in the usual Taylor expansion you'd write f(x+h), you now write t[mu + (Y-mu)] where Y is a random variable with mean mu and variance sigma-squared, which you're about to transform back using t(). If you take expectations, the second term drops out, and people usually take the first and third terms (where the third represents an approximation to the bias in just transforming the mean), and if you take the variance of the expansion to the second term, the first term and the first covariance terms drop out - because t(mu) is a constant - leaving you with a single-term approximation for the variance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The easiest case is when you have normality on the log-scale, and hence a lognormal on the original scale. If your variance is known (which happens very rarely at best), you can cosntruct lognormal CIs and PIs on the original scale, and you can give a predicted mean from the mean of the distribution of the relevant quantity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're estimating both mean and variance on the log-scale, you can construct log-t intervals (prediction intervals for an observation, say), but your original-scale &lt;em&gt;log-t doesn't have any moments&lt;/em&gt;. So the mean of a prediction just doesn't exist. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You need to think very carefully about precisely what question you're trying to answer.&lt;/p&gt;&#xA;" OwnerUserId="805" LastEditorUserId="805" LastEditorDisplayName="" LastEditDate="2010-08-16T00:36:15.757" LastActivityDate="2010-08-16T00:36:15.757" />
  <row Id="1725" PostTypeId="2" ParentId="1719" CreationDate="2010-08-16T02:38:20.447" Score="1" ViewCount="" Body="&lt;p&gt;I've only had a little look at this lecture series on Machine Learning, but it looks good.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://academicearth.org/courses/machine-learning&quot; rel=&quot;nofollow&quot;&gt;http://academicearth.org/courses/machine-learning&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://freevideolectures.com/Course/2257/Machine-Learning/11&quot; rel=&quot;nofollow&quot;&gt;Lecture 11&lt;/a&gt; covers Bayesian Statistics and Regularization.&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-08-16T02:38:20.447" CommentCount="1" />
  <row Id="1726" PostTypeId="2" ParentId="726" CreationDate="2010-08-16T05:45:29.743" Score="15" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Do not trust any statistics you did not fake yourself.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;-- Winston Churchill&lt;/p&gt;&#xA;" OwnerUserId="128" LastEditorUserId="795" LastEditorDisplayName="" LastEditDate="2010-12-03T04:01:19.320" LastActivityDate="2010-12-03T04:01:19.320" CommunityOwnedDate="2010-08-16T05:45:29.743" CommentCount="1" />
  <row Id="1727" PostTypeId="2" ParentId="196" CreationDate="2010-08-16T06:19:27.897" Score="0" ViewCount="" Body="&lt;p&gt;Python's &lt;a href=&quot;http://matplotlib.sourceforge.net/&quot; rel=&quot;nofollow&quot;&gt;matplotlib&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="961" LastActivityDate="2010-08-16T06:19:27.897" CommunityOwnedDate="2010-08-16T06:19:27.897" />
  <row Id="1728" PostTypeId="2" ParentId="1719" CreationDate="2010-08-16T07:09:25.387" Score="1" ViewCount="" Body="&lt;p&gt;I would go straight to &lt;a href=&quot;http://videolectures.net/Top/#o=top&amp;amp;t=vl&quot; rel=&quot;nofollow&quot;&gt;VideoLectures.net&lt;/a&gt;. This is by far the best source--whether free or paid--i have found for very-high quality (both w/r/t the video quality and w/r/t the presentation content) video lectures and tutorials on statistics, forecasting, and machine learning. The target audience for these video lectures ranges from beginner (some lectures are specifically tagged as &quot;tutorials&quot;) to expert; most of them seem to be somewhere in the middle. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;All of the lectures and tutorials are taught to highly experienced professionals and academics, and in many instances, the lecturer is the leading authority on the topic he/she is lecturing on. The site is also 100% free.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The one disadvantage is that you cannot download the lectures and store them in e.g., itunes; however, nearly every lectures has a set of slides which you can download (or, conveniently, you can view them online as you watch the presentation).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;YouTube might have more, but even if you search Y/T through a specific channel, i am sure the signal-to-noise ratio is far higher--on VideoLectures.net, every lecture i've viewed has been outstanding and if you scan the viewer reviews, you'll find that's the consensus opinion towards the entire collection.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A few that i've watched and that i can recommend highly:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;em&gt;Basics of Probability and Statistics&lt;/em&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;em&gt;Introduction to Machine Learning&lt;/em&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;em&gt;Gaussian Process Basics&lt;/em&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;em&gt;Graphical Models&lt;/em&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;em&gt;k-Nearest Neighbor Models&lt;/em&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="438" LastEditorUserId="438" LastEditorDisplayName="" LastEditDate="2010-08-16T07:14:37.997" LastActivityDate="2010-08-16T07:14:37.997" />
  <row Id="1729" PostTypeId="1" AcceptedAnswerId="1730" CreationDate="2010-08-16T08:11:26.317" Score="4" ViewCount="341" Body="&lt;p&gt;Dear list, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am trying to format the output from pairwise.t.test into LaTeX, but have not found a way of doing this. Has anyone got any suggestions? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT:&lt;/strong&gt; As this is a one-time only report where I do need to customize the variable names, and row-/column headings, I was hoping to avoid using Sweave.&lt;/p&gt;&#xA;" OwnerUserId="913" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-08-16T12:58:55.583" LastActivityDate="2010-08-16T12:58:55.583" Title="Export/format output from pairwise.t.test to LaTeX" Tags="&lt;r&gt;&lt;t-test&gt;&lt;post-hoc&gt;&lt;latex&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="1" />
  <row Id="1730" PostTypeId="2" ParentId="1729" CreationDate="2010-08-16T08:27:13.563" Score="4" ViewCount="" Body="&lt;p&gt;Does this help?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; library(xtable)&#xA;&amp;gt; attach(airquality)&#xA;&amp;gt; res &amp;lt;- pairwise.t.test(Ozone, Month)&#xA;&amp;gt; xtable(res$p.value, caption=res$method)&#xA;% latex table generated in R 2.9.2 by xtable 1.5-6 package&#xA;% Mon Aug 16 04:24:21 2010&#xA;\begin{table}[ht]&#xA;\begin{center}&#xA;\begin{tabular}{rrrrr}&#xA;  \hline&#xA; &amp;amp; 5 &amp;amp; 6 &amp;amp; 7 &amp;amp; 8 \\ &#xA;  \hline&#xA;6 &amp;amp; 1.00 &amp;amp;  &amp;amp;  &amp;amp;  \\ &#xA;  7 &amp;amp; 0.00 &amp;amp; 0.05 &amp;amp;  &amp;amp;  \\ &#xA;  8 &amp;amp; 0.00 &amp;amp; 0.05 &amp;amp; 1.00 &amp;amp;  \\ &#xA;  9 &amp;amp; 1.00 &amp;amp; 1.00 &amp;amp; 0.00 &amp;amp; 0.00 \\ &#xA;   \hline&#xA;\end{tabular}&#xA;\caption{t tests with pooled SD}&#xA;\end{center}&#xA;\end{table}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-08-16T08:27:13.563" CommentCount="1" />
  <row Id="1731" PostTypeId="2" ParentId="1729" CreationDate="2010-08-16T08:28:16.523" Score="2" ViewCount="" Body="&lt;p&gt;One way that you can do the above task without using Sweave is as follows: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Write a &lt;strong&gt;custom export function&lt;/strong&gt; from R that takes the pairwise.t.test object and outputs latex code.&lt;/li&gt;&#xA;&lt;li&gt;Use &lt;code&gt;\input{...}&lt;/code&gt; in your LaTeX document to &lt;strong&gt;input&lt;/strong&gt; this file into your LaTeX document.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;With regards to step 1, there are many functions that are useful. Here are a few:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;paste()&lt;/code&gt; is useful for concatenating elements including latex elements with R objects&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;formatC()&lt;/code&gt; is useful for formatting numbers&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;write()&lt;/code&gt; is useful for exporting the combined LaTeX/R character vector from R to a text file&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;There are also a variety of LaTeX table generating functions (e.g., see the &lt;code&gt;xtable&lt;/code&gt; package)&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-08-16T08:28:16.523" CommentCount="1" />
  <row Id="1732" PostTypeId="2" ParentId="1485" CreationDate="2010-08-16T09:07:50.237" Score="2" ViewCount="" Body="&lt;p&gt;For (1), as ebony1 suggests, there are several incremental or on-line SVM algorithms you could try, the only thing I would mention is that the hyper-parameters (regularisation and kernel parameters) may also need tuning as you go along as well, and there are fewer algorithmic tricks to help with that.  The regularisation parameter will almost certainly benefit from tuning, because as the amount of training data increases, the less regularisation that is normally required.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For (2) you could try fitting a one-class SVM to the training data, which would at least tell you if the data were consistent with the classes you do know about, and then classify as &quot;unknown&quot; if the output of the one-class SVM was sufficiently low.  IIRC libsvm has an implementation of one-class SVM.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For (3) if you just use 1-v-1 class component SVMs, then you just need to make three new SVMs, one for each of the unknown-v-known class combinations, and there is no need to retrain the others.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;HTH&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2010-08-16T09:07:50.237" />
  <row Id="1733" PostTypeId="2" ParentId="602" CreationDate="2010-08-16T09:13:42.133" Score="3" ViewCount="" Body="&lt;p&gt;For optimisation, you don't need to perform a grid search; a Nelder-Mead simplex(fminsearch in MATLAB) approach is just as effective and generally much faster, especially if you have a lot of hyper-parameters to tune.  Alternatively you can use gradient descent optimisation - if your implementation doesn't provide gradient information, you can always estimate it by finite differences (as fminunc in MATLAB does).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Span bound is a good criterion to optimise, as it is fast, but good old cross-validation is hard to beat (but use a continuous statistic such as the squared hinge loss).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;HTH&lt;/p&gt;&#xA;&#xA;&lt;p&gt;n.b. nu needs to lie in [0,1], however this is not a problem, just re-parameterise as theta = logit(nu), and then optimise theta instead of nu.  You can then use more or less any numerical optimisation technique you like, e.g. Nelder-Mead simplex, gradient descent, local search, genetic algorithms...&lt;/p&gt;&#xA;" OwnerUserId="887" LastEditorUserId="887" LastEditorDisplayName="" LastEditDate="2010-08-16T09:23:45.913" LastActivityDate="2010-08-16T09:23:45.913" />
  <row Id="1734" PostTypeId="2" ParentId="36" CreationDate="2010-08-16T11:00:11.680" Score="0" ViewCount="" Body="&lt;p&gt;As a generalization of 'pirates cause global warming': Pick any two quantities which are (monotonically) increasing or decreasing with time and you should see some correlation.&lt;/p&gt;&#xA;" OwnerUserId="961" LastActivityDate="2010-08-16T11:00:11.680" CommunityOwnedDate="2010-08-16T13:01:42.613" />
  <row Id="1735" PostTypeId="1" CreationDate="2010-08-16T11:36:42.863" Score="2" ViewCount="459" Body="&lt;p&gt;I am regressing two butterfly richness variables (summer and winter) &#xA;against a set of environmental variables separately. &#xA;(variables with continuous numbers)&#xA;Environmental variables are identitcal in each model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the summer model,&#xA;the weight rank of coefficients is&#xA;temp &gt; prec &gt; ndvi.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The weight rank in winter is &#xA;temp &gt; ndvi &gt; prec.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As it is almost implausible to compare the coefficients directly, &#xA;pls advise any advanced method other than regression &#xA;to discriminate such coefficient rank between seasons, &#xA;such as canonical correlation analysis (unsure if it is suitable here)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The spatial info here&#xA;richness and environmental variables comprising 2000 grid (continuous distribution) spanning from 100 E to 130 E longitude, 18 to 25 N latitude.&lt;/p&gt;&#xA;" OwnerUserId="962" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-24T14:01:07.313" LastActivityDate="2010-09-25T16:17:28.180" Title="Method to compare variable coefficient in two regression models" Tags="&lt;regression&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="1" />
  <row Id="1736" PostTypeId="1" CreationDate="2010-08-16T12:10:29.567" Score="5" ViewCount="306" Body="&lt;p&gt;I am wondering if there are any packages for python that is capable of performing survival analysis. I have been using the survival package in R but would like to port my work to python. &lt;/p&gt;&#xA;" OwnerUserId="172" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-16T12:33:26.227" LastActivityDate="2010-09-16T18:58:26.863" Title="Survival Analysis tools in Python" Tags="&lt;survival&gt;&lt;python&gt;" AnswerCount="3" CommentCount="1" FavoriteCount="2" />
  <row Id="1737" PostTypeId="1" AcceptedAnswerId="1745" CreationDate="2010-08-16T13:23:52.747" Score="7" ViewCount="463" Body="&lt;p&gt;I want to calculate a summary of a variable in a data.frame for each unique combination of factors in the data.frame. Should I use plyr to do this? I am ok with using loops as opposed to apply() ; so just finding out each unique combination would be enough.&lt;/p&gt;&#xA;" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-28T20:28:59.777" LastActivityDate="2011-04-28T22:36:25.187" Title="How to find all unique combinations of factors in a data.frame in R?" Tags="&lt;r&gt;&lt;categorical-data&gt;&lt;aggregate&gt;&lt;plyr&gt;" AnswerCount="6" CommentCount="1" FavoriteCount="3" />
  <row Id="1738" PostTypeId="2" ParentId="1713" CreationDate="2010-08-16T13:42:04.937" Score="4" ViewCount="" Body="&lt;p&gt;If you want inferences specifically about the mean of the original variable, then don't use Box-Cox transformation. IMO Box-Cox transformations are most useful when the transformed variable has its own interpretation, and the Box-Cox transformation only helps you to find the right scale for analysis - this turns out to be the case surprisingly often. Two unexpected exponents that I found this way were 1/3 (when the response variable was bladder volume) and -1 (when the response variable was breaths per minute).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The log-transformation is probably the only exception to this. The mean on the log-scale corresponds to the geometric mean in the original scale, which is at least a well-defined quantity.&lt;/p&gt;&#xA;" OwnerUserId="279" LastActivityDate="2010-08-16T13:42:04.937" />
  <row Id="1739" PostTypeId="2" ParentId="1737" CreationDate="2010-08-16T13:46:53.893" Score="8" ViewCount="" Body="&lt;p&gt;See &lt;code&gt;aggregate&lt;/code&gt; and &lt;code&gt;by&lt;/code&gt;. For example, from the help file for &lt;code&gt;aggregate&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;## Compute the averages according to region and the occurrence of more&#xA;## than 130 days of frost.&#xA;aggregate(state.x77,&#xA;      list(Region = state.region,&#xA;           Cold = state.x77[,&quot;Frost&quot;] &amp;gt; 130),&#xA;      mean)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="279" LastActivityDate="2010-08-16T13:46:53.893" CommentCount="2" />
  <row Id="1740" PostTypeId="2" ParentId="1737" CreationDate="2010-08-16T14:01:14.360" Score="3" ViewCount="" Body="&lt;p&gt;Here's the plyr solution, which has the advantage of returning multiple summary stats and producing a progress bar for long computes:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(ez) #for a data set&#xA;data(ANT)&#xA;cell_stats = ddply(&#xA;    .data = ANT #use the ANT data&#xA;    , .variables = .(cue,flanker) #uses each combination of cue and flanker&#xA;    , .fun = function(x){ #apply this function to each combin. of cue &amp;amp; flanker&#xA;        to_return = data.frame(&#xA;            , acc = mean(x$acc)&#xA;            , mrt = mean(x$rt[x$acc==1])&#xA;        )&#xA;        return(to_return)&#xA;    }&#xA;    , .progress = 'text'&#xA;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="364" LastActivityDate="2010-08-16T14:01:14.360" CommentCount="1" />
  <row Id="1741" PostTypeId="2" ParentId="1736" CreationDate="2010-08-16T14:05:44.193" Score="7" ViewCount="" Body="&lt;p&gt;AFAIK, there aren't any survival analysis packages in python. As mbq comments above, the only route available would be to &lt;a href=&quot;http://rpy.sourceforge.net/&quot; rel=&quot;nofollow&quot;&gt;Rpy&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even if there were a pure python package available, I would be very careful in using it, in particular I would look at:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;How often does it get updated.&lt;/li&gt;&#xA;&lt;li&gt;Does it have a large user base?&lt;/li&gt;&#xA;&lt;li&gt;Does it have advanced techniques?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;One of the benefits of R, is that these standard packages get a massive amount of testing and user feed back. When dealing with real data, unexpected edge cases can creep in.&lt;/p&gt;&#xA;" OwnerUserId="8" LastActivityDate="2010-08-16T14:05:44.193" />
  <row Id="1742" PostTypeId="2" ParentId="1668" CreationDate="2010-08-16T14:38:47.827" Score="4" ViewCount="" Body="&lt;p&gt;I'd suggest Christopher Bishop's &quot;Pattern Recognition and Machine Learning&quot;. You can see some of it, including a sample chapter, at &lt;a href=&quot;http://research.microsoft.com/en-us/um/people/cmbishop/PRML/index.htm&quot; rel=&quot;nofollow&quot;&gt;http://research.microsoft.com/en-us/um/people/cmbishop/PRML/index.htm&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="247" LastActivityDate="2010-08-16T14:38:47.827" CommunityOwnedDate="2010-08-16T14:38:47.827" />
  <row Id="1743" PostTypeId="2" ParentId="36" CreationDate="2010-08-16T15:29:24.607" Score="0" ViewCount="" Body="&lt;p&gt;A correlation on its own can &lt;b&gt;never&lt;/b&gt; establish a causal link.  &lt;a href=&quot;http://en.wikipedia.org/wiki/David_Hume&quot; rel=&quot;nofollow&quot;&gt;David Hume&lt;/a&gt; (1771-1776) argued quite effectively that we can not obtain &lt;b&gt;certain&lt;/b&gt; knowlege of cauasality by purely empirical means.  Kant attempted to address this, the Wikipedia page for &lt;a href=&quot;http://en.wikipedia.org/wiki/Kant&quot; rel=&quot;nofollow&quot;&gt;Kant&lt;/a&gt; seems to sum it up quite nicely:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;Kant believed himself to be creating a compromise between the empiricists and the rationalists. The empiricists believed that knowledge is acquired through experience alone, but the rationalists maintained that such knowledge is open to Cartesian doubt and that reason alone provides us with knowledge. Kant argues, however, that using reason without applying it to experience will only lead to illusions, while experience will be purely subjective without first being subsumed under pure reason.&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In otherwords, Hume tells us that we can never know a causal relationship exists just by observing a correlation, but Kant suggests that we may be able to use our reason to distinguish between correlations that do imply a causal link from those who don't.  I don't think Hume would have disagreed, as long as Kant were writing in terms of plausibility rather than certain knowledge.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In short, a correlation provides circumstantial evidence implying a causal link, but the weight of the evidence depends greatly on the particular circumstances involved, and we can never be absolutely sure.  The ability to predict the effects of interventions is one way to gain confidence (we can't prove anything, but we can disprove by observational evidence, so we have then at least attempted to falsify the theory of a causal link).  Having a simple model that explains why we should observed a correlation that also explains other forms of evidence is another way we can apply our reasoning as Kant suggests.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Caveat emptor: It is entirely possible I have misunderstood the  philosophy, however it remains the case that a correlation can never provide proof of a causal link.&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2010-08-16T15:29:24.607" CommunityOwnedDate="2010-08-16T15:29:24.607" />
  <row Id="1744" PostTypeId="2" ParentId="1737" CreationDate="2010-08-16T15:32:04.733" Score="0" ViewCount="" Body="&lt;p&gt;In addition to other suggestions you may find the &lt;code&gt;describe.by()&lt;/code&gt; function in the &lt;code&gt;psych&lt;/code&gt; package useful.&#xA;It can be used to show summary statistics on numeric variables across levels of a factor variable.&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-08-16T15:32:04.733" />
  <row Id="1745" PostTypeId="2" ParentId="1737" CreationDate="2010-08-16T15:46:32.153" Score="5" ViewCount="" Body="&lt;p&gt;While I think &lt;code&gt;aggregate&lt;/code&gt; is probably the solution you are seeking, if you are want to create an explicit list of all possible factor combinations, &lt;code&gt;expand.grid&lt;/code&gt; will do that for you. e.g.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; expand.grid(height = seq(60, 80, 5), weight = seq(100, 300, 50),&#xA;             sex = c(&quot;Male&quot;,&quot;Female&quot;))&#xA;       height weight    sex&#xA;1      60    100   Male&#xA;2      65    100   Male&#xA;... &#xA;30     80    100 Female&#xA;31     60    150 Female&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You could then loop over each row in the resulting data frame to pull out records from your original data.&lt;/p&gt;&#xA;" OwnerUserId="729" LastActivityDate="2010-08-16T15:46:32.153" />
  <row Id="1746" PostTypeId="2" ParentId="1736" CreationDate="2010-08-16T16:30:54.480" Score="3" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://sourceforge.net/projects/python-asurv/&quot; rel=&quot;nofollow&quot;&gt;python-asurv&lt;/a&gt; is an effort to port the &lt;a href=&quot;http://www.astrostatistics.psu.edu/statcodes/sc_censor.html&quot; rel=&quot;nofollow&quot;&gt;asurv&lt;/a&gt; software for survival methods in astronomy.   Might be worth keeping an eye on, but cgillespie is right about the things to watch out for: it has a &lt;em&gt;long&lt;/em&gt; way to go and development doesn't seem active.  (AFAICT only one method exists and even completed, the package may be lacking for, say, biostatisticians.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You're probably better off using &lt;a href=&quot;http://cran.r-project.org/web/packages/survival/index.html&quot; rel=&quot;nofollow&quot;&gt;survival&lt;/a&gt; package in R from Python through something like &lt;a href=&quot;http://rpy.sourceforge.net/rpy2.html&quot; rel=&quot;nofollow&quot;&gt;RPy&lt;/a&gt; or &lt;a href=&quot;http://www.jstatsoft.org/v35/c02&quot; rel=&quot;nofollow&quot;&gt;PypeR&lt;/a&gt;.  I haven't had any problems doing this myself.&lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-08-16T16:30:54.480" />
  <row Id="1747" PostTypeId="2" ParentId="1517" CreationDate="2010-08-16T17:13:58.303" Score="0" ViewCount="" Body="&lt;p&gt;2 great answers so far and I promise I'll accept one of them.  But I wanted to post this link as an answer too:&#xA;&lt;a href=&quot;http://www.statsoft.com/textbook/elementary-concepts-in-statistics/#Two%20basic%20features%20of%20every%20relation%20between%20variables&quot; rel=&quot;nofollow&quot;&gt;summary of how to compare two numbers&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Yes, for me merely comparing two numbers had some questions, so &lt;strong&gt;that&lt;/strong&gt; shows you the level I'm at.  While it won't be relevant to most, here is the VBA function I made based on that info ... well, not based that info exactly, I needed &lt;a href=&quot;http://wiki.answers.com/Q/How_do_you_determine_Percent_of_change_between_two_numbers&quot; rel=&quot;nofollow&quot;&gt;something simpler&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Private Function fxDifference(sng_A As Single, sng_B As Single) As Variant&#xA;    ''When calling this function, sng_B must not be zero (Error div/zero)&#xA;    fxDifference = (sng_A - sng_B)&#xA;    fxDifference = fxDifference / sng_B&#xA;    fxDifference = Round(fxDifference * 100, 1)&#xA;End Function&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I suppose when I said 'generic' tools, I meant 'really, really basic' ...&lt;/p&gt;&#xA;" OwnerUserId="857" LastActivityDate="2010-08-16T17:13:58.303" CommunityOwnedDate="2010-08-16T17:13:58.303" CommentCount="1" />
  <row Id="1748" PostTypeId="2" ParentId="980" CreationDate="2010-08-16T17:20:25.063" Score="0" ViewCount="" Body="&lt;p&gt;If BPM is staying the same over many samples (or changing infinitesimally in a way you aren't concerned about) you can truncate your data to a significant digit that you actually care about and then do Run Length Encoding.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, in R this data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;has this output&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;rle(data)&#xA;Run Length Encoding&#xA;  lengths: int [1:3] 10 15 15&#xA;  values : num [1:3] 0 1 2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="196" LastActivityDate="2010-08-16T17:20:25.063" />
  <row Id="1749" PostTypeId="1" AcceptedAnswerId="1751" CreationDate="2010-08-16T17:27:28.967" Score="5" ViewCount="93" Body="&lt;p&gt;I'm uncertain whether I should be able to intuit the answer to my question from a question that has &lt;a href=&quot;http://stats.stackexchange.com/questions/1123/modeling-success-rate-with-gaussian-distribution&quot;&gt;already been asked&lt;/a&gt; but I can't, so I am asking the question anyway.  Thus, I am looking for a clear easy to understand answer.  A &lt;a href=&quot;http://www.dailymail.co.uk/news/article-1303442/Pregancy-The-average-couple-sex-104-times-conceive.html&quot; rel=&quot;nofollow&quot;&gt;recent newspaper article&lt;/a&gt; reported that on average couples were able to conceive a child after 104 reproductive acts.  Assuming indpendant binomial trials, that means for each act there was a 1/104 probability of success.  I can do a quick simulation to show myself what the quantiles for this distribution look like, e.g. in R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;NSIM &amp;lt;- 10000&#xA;trialsuntilsuccess &amp;lt;- function(N=10000,Pr=1/104)&#xA;{&#xA;    return(min(seq(1:N)[rbinom(N,1,Pr)==1]))&#xA;}&#xA;res &amp;lt;- rep(NA,NSIM)&#xA;for (i in 1:NSIM)&#xA;{&#xA;    res[i] &amp;lt;- trialsuntilsuccess()&#xA;    if (i %% 10  == 0) {cat(i,&quot;\r&quot;); flush.console()}&#xA;}&#xA;quantile(res,c(.025,.975))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But it seems like there should be some simple equation or approximation that could be applied, perhaps a probit or poisson?  Any advice on how to get the quantiles without running a simulation?  Bonus points for providing a way to do the relevant calculations in R.&lt;/p&gt;&#xA;" OwnerUserId="196" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-08-16T19:05:00.580" LastActivityDate="2010-08-16T19:05:00.580" Title="How can you approximate the number of trials to success given a particular Pr(Success)?" Tags="&lt;probability&gt;&lt;binomial&gt;&lt;negative-binomial&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="1750" PostTypeId="2" ParentId="1737" CreationDate="2010-08-16T17:31:42.483" Score="0" ViewCount="" Body="&lt;p&gt;In library(doBy) there is also the summaryBy() function, e.g.&#xA;summaryBy(DV1 + DV2 ~ Height+Weight+Sex,data=my.data)&lt;/p&gt;&#xA;" OwnerUserId="196" LastActivityDate="2010-08-16T17:31:42.483" />
  <row Id="1751" PostTypeId="2" ParentId="1749" CreationDate="2010-08-16T17:43:49.463" Score="3" ViewCount="" Body="&lt;p&gt;If I understand your question correctly you want to compute the quantiles for the &quot;No of failures before the first success&quot; given that $p=\frac{1}{104}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The distribution you should be looking at is the &lt;a href=&quot;http://en.wikipedia.org/wiki/Negative_binomial_distribution&quot; rel=&quot;nofollow&quot;&gt;negative binomial distribution&lt;/a&gt;. The wiki discusses the negative binomial as:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In probability theory and statistics, the negative binomial distribution is a discrete probability distribution of the number of successes in a sequence of Bernoulli trials before a specified (non-random) number r of failures occurs.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Just invert the interpretation of success and failures with a setting of r=1 would accomplish what you want. The distribution with r=1 is also called the &lt;a href=&quot;http://en.wikipedia.org/wiki/Geometric_distribution&quot; rel=&quot;nofollow&quot;&gt;geometric distribution&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could then use the discrete distribution to compute the quantiles.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: I do not know R.&lt;/p&gt;&#xA;" LastActivityDate="2010-08-16T17:43:49.463" CommentCount="1" />
  <row Id="1752" PostTypeId="2" ParentId="1447" CreationDate="2010-08-16T17:50:47.190" Score="0" ViewCount="" Body="&lt;p&gt;Are you also asking what it means interpretively to use the phrase(s) &quot;explained by&quot; or &quot;accounted for&quot; in terms of causality with regression?&lt;/p&gt;&#xA;" LastActivityDate="2010-08-16T17:50:47.190" />
  <row Id="1753" PostTypeId="1" AcceptedAnswerId="1754" CreationDate="2010-08-16T18:01:43.127" Score="1" ViewCount="413" Body="&lt;p&gt;I came across an example where standard deviation was being plotted on a Cartesian plot (standard 2D with X and Y axes.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This seems like a valid thing to do but in this case the example only had a single line running across the graph to &quot;indicate&quot; standard deviation. This to me seems not very useful, possibly dangerous and misleading. Don't you need three lines plotted to properly visualize standard deviation on a graph? Thusly:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The mean&lt;/li&gt;&#xA;&lt;li&gt;The mean plus one standard deviation value&lt;/li&gt;&#xA;&lt;li&gt;The mean minus one standard deviation value&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;P.S. I am a software developer working on a data visualization package so please take my use of stats terminology with a grain of salt. Any corrections and feedback will be sincerely appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Rephrasing the question:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I had a set of five data points to plot on a cartesian plane:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X:    10    20    30    35    50&#xA;Y:    20    40    5     55    10&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For this sample data set (the Y values) the mean is 20 and the stdev is ~21.036 (x values plotted along the X axis and y values plotted along the Y axis.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What would a proper plotting of the mean and the stdev on top of the X/Y data set look like?&lt;/p&gt;&#xA;" OwnerUserId="968" LastEditorUserId="968" LastEditorDisplayName="" LastEditDate="2010-08-16T19:36:49.437" LastActivityDate="2010-08-16T20:03:34.477" Title="Visualizing standard deviation on a Cartesian plot" Tags="&lt;data-visualization&gt;&lt;standard-deviation&gt;" AnswerCount="2" CommentCount="5" FavoriteCount="1" />
  <row Id="1754" PostTypeId="2" ParentId="1753" CreationDate="2010-08-16T19:59:01.507" Score="4" ViewCount="" Body="&lt;p&gt;Probably a line for the mean and a line for +/- twice the standard deviation.  That would be the &quot;default&quot; plot for that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That said, I think you may be missing the point of the plot with the single line for the standard deviation.  If what you're trying to represent is change in the variability of Y over X (i.e., heteroscedasticity), then a line plotting SD over X might work.  It really does depend on the data and the questions that you're trying to ask.  There just isn't a set of rules that you can follow to produce good plots every time, and in general the more automated the plotting system gets, the more useless I find it.&lt;/p&gt;&#xA;" OwnerUserId="71" LastActivityDate="2010-08-16T19:59:01.507" CommentCount="3" />
  <row Id="1755" PostTypeId="2" ParentId="1753" CreationDate="2010-08-16T20:03:34.477" Score="2" ViewCount="" Body="&lt;p&gt;What about plotting the point with error bars, say mean +/- sd. Here's what your example data would look like:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/Zqgbc.png&quot; alt=&quot;dot-plot with error bars&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Here's the R code I used to generate the plot:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(ggplot2)&#xA;df = data.frame(values=c(10, 20, 30, 35, 50, 20, 40, 5, 55, 10),&#xA;  type=rep(c(&quot;X&quot;, &quot;Y&quot;),  each=5))&#xA;&#xA;means = tapply(df$values, df$type, mean)&#xA;sds = tapply(df$values, df$type, sd)&#xA;df_summary = data.frame(means, sds, type=c(&quot;X&quot;, &quot;Y&quot;))&#xA;&#xA;g = ggplot(data=df_summary, aes(y=means, x=type)) +&#xA;  geom_point(data=df,aes(y=values, x=type), col=2) +&#xA;  geom_errorbar(aes(ymax = means + sd, ymin=means - sd)) +&#xA;  ylab(&quot;Values&quot;)&#xA;&#xA;g&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="8" LastActivityDate="2010-08-16T20:03:34.477" CommentCount="1" />
  <row Id="1756" PostTypeId="2" ParentId="841" CreationDate="2010-08-16T20:12:03.067" Score="2" ViewCount="" Body="&lt;p&gt;The most naive approach I can think of is to regress $Y_i$ vs $X_i$ as $Y_i \sim \hat{m}X_i + \hat{b}$, then perform a $t$-test on the hypothesis $m = 1$. See &lt;a href=&quot;http://en.wikipedia.org/wiki/T-test#Slope_of_a_regression_line&quot; rel=&quot;nofollow&quot;&gt;t-test for regression slope&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A less naive approach is the Morgan-Pitman test. Let $U_i = X_i - Y_i, V_i = X_i + Y_i,$ then perform a test of the Pearson Correlation coefficient of $U_i$ vs $V_i$. (One can do this simply using the &lt;a href=&quot;http://en.wikipedia.org/wiki/Fisher_transformation&quot; rel=&quot;nofollow&quot;&gt;Fisher R-Z transform&lt;/a&gt;, which gives the confidence intervals around the sample Pearson coefficient, or via a bootstrap.) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are using R, and don't want to have to code everything yourself, I would use &lt;code&gt;bootdpci&lt;/code&gt; from Wilcox' Robust Stats package, WRS. (see &lt;a href=&quot;http://www-rcf.usc.edu/~rwilcox/&quot; rel=&quot;nofollow&quot;&gt;Wilcox' page&lt;/a&gt;.)&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-08-16T20:12:03.067" />
  <row Id="1757" PostTypeId="1" CreationDate="2010-08-16T23:36:20.887" Score="4" ViewCount="206" Body="&lt;p&gt;I am taking the time to learn how to analyze networks and want to test if there are differences between two networks over time.  Since I am new to R and networks in general, I am hoping to get some help how to compare and analyze network graphs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Simply, my dataset will contain information on flows between two geographic units (think Census migration data).  I want to test if the flows, generally, are different with time. Since I am just starting out, I am not even sure if I am phrasing my question correctly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have created a few basic graphs and generated some very basic &quot;summary statistics&quot; on a graph in isolation in R before, so I understand how to get up and running, but I am not really sure where to go from here.&lt;/p&gt;&#xA;" OwnerUserId="569" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-08-17T07:37:41.443" LastActivityDate="2010-08-17T07:51:41.177" Title="Significant Difference between two network graphs" Tags="&lt;r&gt;&lt;networks&gt;" AnswerCount="2" FavoriteCount="2" />
  <row Id="1758" PostTypeId="2" ParentId="1681" CreationDate="2010-08-17T00:50:11.027" Score="0" ViewCount="" Body="&lt;p&gt;I will try and answer your question for only one of the bounds. Chernoff bounds are given by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Pr[\frac{1}{m} \sum_{i=1}^{i=m} X_i \ge p+\epsilon] \le e^{-D(p+\epsilon||p)m}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$D(.)$ is the &lt;a href=&quot;http://en.wikipedia.org/wiki/Kullback-Leibler_divergence&quot; rel=&quot;nofollow&quot;&gt;Kullback-Leibler divergence&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the sake of convenience I will denote the rhs of the above inequality by $r(p,\epsilon,m)$. Thus, we have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Pr[\frac{1}{m} \sum_{i=1}^{i=m} X_i \ge p+\epsilon] \le r(p,\epsilon,m)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The above can be re-written as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Pr[ \sum_{i=1}^{i=m} X_i \ge m\ (p+\epsilon)] \le r(p,\epsilon,m)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we let: $Y = \sum_{i=1}^{i=m} X_i$ then we know that:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Y \sim Binomial(m,p)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$p = Prob(X_i=1)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given the above it follows that,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Pr[ \sum_{i=1}^{i=m} X_i \ge m\ (p+\epsilon)] = Pr[ Y_i \ge m\ (p+\epsilon)]$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Pr[ Y_i \ge m\ (p+\epsilon)] = \sum_{k=\lceil m\ (p+\epsilon) \rceil}^{k=m} {m \choose k} p^k (1-p)^{m-k}$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, it follows that:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\sum_{k=\lceil p+\epsilon \rceil}^{k=m} {m \choose k} p^k (1-p)^{m-k} \le r(p,\epsilon,m)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not sure if we can simplify the above to express $\epsilon$ as a function of $m$ but it may be of some help. Alternatively, you may want to explore the use of the normal approximation to the binomial.&lt;/p&gt;&#xA;" LastActivityDate="2010-08-17T00:50:11.027" CommentCount="4" />
  <row Id="1759" PostTypeId="2" ParentId="1757" CreationDate="2010-08-17T01:11:36.063" Score="1" ViewCount="" Body="&lt;p&gt;I am not sure I will be able to provide a complete answer but here is how I would start. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Step 1: Model the data generating process for the flows through the network&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, you may want to model the flows from one point to another point in the network as a &lt;a href=&quot;http://en.wikipedia.org/wiki/Poisson_distribution&quot; rel=&quot;nofollow&quot;&gt;poisson distribution&lt;/a&gt;. The poission distribution is used to model arrivals in a system over time and thus may work well for network flows. Depending on network complexity and your needs you can model each path such that the arrival rate for each path is either different or the same (See the $\lambda$ parameter for the poisson distribution.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Step 2: Identify a testing strategy which would let you ascertain the strength of evidence for your null model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The challenge in this step is two-fold. First, you have to define what you mean when you say that the network flows in a network at different times is the same. Are you talking about throughput? or Are you talking about the flows across each one of the paths?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second challenge is that once you have solved the above issue, you need to find out a way to test your null hypothesis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an example: Suppose you want to check that the flows across each path are the same and that your are modeling the path flows for the network by a single parameter (i.e., $\lambda$ is identical for all paths). Thus, your null hypothesis would assume that $\lambda$ does not change with time. This how you would go about testing your null hypothesis:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Null Hypothesis is True&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Pool all network flows across time and estimate a common $\lambda$ using &lt;a href=&quot;http://en.wikipedia.org/wiki/Poisson_distribution#Maximum_likelihood&quot; rel=&quot;nofollow&quot;&gt;maximum likelihood estimation&lt;/a&gt; for the poisson distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Null Hypothesis is not true&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Estimate $\lambda$ for each time period separately so that you get two different values  (one for each one of the time periods).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can then select the model (pooled or separate) that fits the data better on the basis of a criteria such as the &lt;a href=&quot;http://en.wikipedia.org/wiki/Likelihood-ratio_test&quot; rel=&quot;nofollow&quot;&gt;likelihood ratio&lt;/a&gt;.&lt;/p&gt;&#xA;" LastActivityDate="2010-08-17T01:11:36.063" />
  <row Id="1760" PostTypeId="2" ParentId="726" CreationDate="2010-08-17T02:01:44.427" Score="7" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;He uses statistics like a drunken man uses a lamp post, more for support than illumination.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;-- Andrew Lang&lt;/p&gt;&#xA;" OwnerUserId="74" LastEditorUserId="795" LastEditorDisplayName="" LastEditDate="2010-10-02T17:11:05.487" LastActivityDate="2010-10-02T17:11:05.487" CommunityOwnedDate="2010-08-17T02:01:44.427" />
  <row Id="1761" PostTypeId="1" CreationDate="2010-08-17T02:52:02.637" Score="11" ViewCount="496" Body="&lt;p&gt;There was already &lt;a href=&quot;http://stats.stackexchange.com/questions/485/mathematical-statistics-videos&quot;&gt;a request for Mathematical Statistics Videos&lt;/a&gt;,  but it explicitly asked from people for&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;videos that provide a rigorous&#xA;  mathematical presentation of&#xA;  statistics. i.e., videos that might&#xA;  accompany a course that use a textbook&#xA;  mentioned in this discussion on...&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So at the same time I am wondering, &lt;strong&gt;what recommendation do you have for stat/prob - 101  - video courses?&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="253" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-17T20:24:33.893" LastActivityDate="2010-09-18T02:43:41.650" CommunityOwnedDate="2010-08-17T02:52:02.637" Title="Statistics/Probability Videos for Beginners" Tags="&lt;books&gt;" AnswerCount="3" FavoriteCount="11" />
  <row Id="1762" PostTypeId="2" ParentId="1687" CreationDate="2010-08-17T03:09:29.367" Score="0" ViewCount="" Body="&lt;p&gt;I doubt that you can get an analytical solution to this problem. About the only step that is doable is the probability that the average of the first three velocities is under some cutoff and it is &lt;em&gt;not&lt;/em&gt; (probability the one velocity is under the cutoff)^3 as you stated. If $V_i \sim N (m, s^2)$, then $\bar{V}_3 \sim N (m, s^2/3)$, so the probability that $\bar{V}_3$ is less than $k$ standard deviations above the mean is $P(\bar{V}_3 &amp;lt; m + k s ) = \Phi(k/\sqrt{3})$ where $\Phi$ is the normal distribution function.  For example, for $k=0$: $P(\bar{V}_3 &amp;lt; m) = 0.5$, for $k=1$: $P(\bar{V}_3 &amp;lt; m + s) = 0.718$, etc.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The calculation involving the next three rounds is complicated by two things: it should somehow be conditional on the fact that the average was too low already, and then should only use the three highest values. While it might work out to be some triple integral, I don't think you want to use such a formula. After that the fact that you have a choice whether the bullet should be weighed or fired also probably depends on the previous values, and things get even more complicated depending on your strategy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even the simple multiplication of a random bullet weight and one random velocity is not as innocuous as it seems: the product will not be normal (or any other &quot;regular&quot; distribution). Its standard deviation can be approximated via standard &lt;a href=&quot;http://en.wikipedia.org/wiki/Propagation_of_uncertainty#Example_formulas&quot; rel=&quot;nofollow&quot;&gt;error propagation&lt;/a&gt;, but calculating probabilities of falling below/above some cutoff is not straightforward.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In summary, I think you should write a little simulation program - you will get better answers faster. Note that for the chronometer you will have to define more exactly what does +/-4% mean.&lt;/p&gt;&#xA;" OwnerUserId="279" LastEditorUserId="279" LastEditorDisplayName="" LastEditDate="2010-08-17T03:36:03.013" LastActivityDate="2010-08-17T03:36:03.013" CommentCount="4" />
  <row Id="1763" PostTypeId="2" ParentId="1761" CreationDate="2010-08-17T03:47:24.627" Score="10" ViewCount="" Body="&lt;p&gt;I think a number of the suggestions put forward on the &lt;a href=&quot;http://stats.stackexchange.com/questions/485/mathematical-statistics-videos&quot;&gt;mathematical statistics video question&lt;/a&gt; probably fall in the stats 101 category:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.khanacademy.org/#Statistics&quot; rel=&quot;nofollow&quot;&gt;http://www.khanacademy.org/#Statistics&lt;/a&gt;: series of short videos on introductory statistics&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.khanacademy.org/#Probability&quot; rel=&quot;nofollow&quot;&gt;http://www.khanacademy.org/#Probability&lt;/a&gt;: series of short videos on introductory probability&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.academicearth.org/courses/math-and-proability-for-life-sciences&quot; rel=&quot;nofollow&quot;&gt;Math and probability for life sciences&lt;/a&gt;: A whole university course introducing statistics and probability.&lt;/li&gt;&#xA;&lt;li&gt;I also have a &lt;a href=&quot;http://jeromyanglim.blogspot.com/2009/05/online-mathematics-video-courses-for.html&quot; rel=&quot;nofollow&quot;&gt;list of maths and statistics videos&lt;/a&gt; with a few others.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The Stat 579 Videos are also quite good:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://connect.extension.iastate.edu/p31588910/&quot; rel=&quot;nofollow&quot;&gt;http://connect.extension.iastate.edu/p31588910/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://connect.extension.iastate.edu/p45341752/&quot; rel=&quot;nofollow&quot;&gt;http://connect.extension.iastate.edu/p45341752/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://connect.extension.iastate.edu/p39131597/&quot; rel=&quot;nofollow&quot;&gt;http://connect.extension.iastate.edu/p39131597/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://connect.extension.iastate.edu/p21949344/&quot; rel=&quot;nofollow&quot;&gt;http://connect.extension.iastate.edu/p21949344/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://connect.extension.iastate.edu/p77697317/&quot; rel=&quot;nofollow&quot;&gt;http://connect.extension.iastate.edu/p77697317/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://connect.extension.iastate.edu/p62985380/&quot; rel=&quot;nofollow&quot;&gt;http://connect.extension.iastate.edu/p62985380/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://connect.extension.iastate.edu/p77697317/&quot; rel=&quot;nofollow&quot;&gt;http://connect.extension.iastate.edu/p77697317/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://connect.extension.iastate.edu/p54242569/&quot; rel=&quot;nofollow&quot;&gt;http://connect.extension.iastate.edu/p54242569/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://connect.extension.iastate.edu/p95967207/&quot; rel=&quot;nofollow&quot;&gt;http://connect.extension.iastate.edu/p95967207/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://connect.extension.iastate.edu/p80966639/&quot; rel=&quot;nofollow&quot;&gt;http://connect.extension.iastate.edu/p80966639/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://connect.extension.iastate.edu/p51894198/&quot; rel=&quot;nofollow&quot;&gt;http://connect.extension.iastate.edu/p51894198/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://connect.extension.iastate.edu/p39993940/&quot; rel=&quot;nofollow&quot;&gt;http://connect.extension.iastate.edu/p39993940/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-08-22T15:34:38.040" LastActivityDate="2010-08-22T15:34:38.040" CommunityOwnedDate="2010-08-17T03:47:24.627" CommentCount="6" />
  <row Id="1764" PostTypeId="1" AcceptedAnswerId="1792" CreationDate="2010-08-17T05:22:31.340" Score="10" ViewCount="333" Body="&lt;p&gt;Users are often tempted to break axis values to present data of different orders of magnitude on the same graph (see &lt;a href=&quot;http://peltiertech.com/Excel/Charts/BrokenYAxis.html&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;). While this may be convenient it's not always the preferred way of displaying the data (can be misleading at best). What are alternative ways of displaying the data that are different in several orders of magnitude?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can think of two ways, to log-transform the data or use lattice plots. What are other options?&lt;/p&gt;&#xA;" OwnerUserId="144" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-08-17T05:25:15.750" LastActivityDate="2010-08-17T17:35:21.270" Title="What are alternatives to broken axes?" Tags="&lt;data-visualization&gt;&lt;logarithm&gt;" AnswerCount="3" CommentCount="5" FavoriteCount="2" />
  <row Id="1765" PostTypeId="2" ParentId="1764" CreationDate="2010-08-17T06:22:29.067" Score="6" ViewCount="" Body="&lt;p&gt;Maybe it can be classified as lattice, but I'll try; plot all the bars scaled to the highest in one panel and put another panel showing zoom on lower ones. I used this technique once in case of a scatterplot, and the result was quite nice.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-08-17T06:22:29.067" />
  <row Id="1766" PostTypeId="2" ParentId="1649" CreationDate="2010-08-17T06:26:59.823" Score="1" ViewCount="" Body="&lt;p&gt;You can calculate SD for each person and its standard error (for example using bootstrap). Then you can use &lt;strong&gt;rmeta&lt;/strong&gt; package to do analysis. I think you should use some transformation of SD for example log (or maybe better log of variance). &lt;/p&gt;&#xA;" OwnerUserId="419" LastActivityDate="2010-08-17T06:26:59.823" />
  <row Id="1767" PostTypeId="2" ParentId="1378" CreationDate="2010-08-17T06:29:42.563" Score="1" ViewCount="" Body="&lt;p&gt;From your description I can't see any reason to distinguish the &quot;baseline test&quot; from the immediately drawn &quot;second sample&quot;. They are simply 2 baseline measurements and the variance (at baseline) can be calculated on that basis. It would be better plotting the mean of the two baseline measurements versus the third &quot;six month&quot; sample.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is with the 6 month sample. As only one sample is taken at this point there is no way of estimating the &quot;variability&quot; at this point, or rather separating sampling variation from longitudinal (real) change in TB reading.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we consider this a longitudinal data analysis problem we would probably choose the a random intercept (baseline TB) and a random slope (to fit the 6 month TB). The sampling variability would be estimated from the two baseline measurements and the slope from the third 6 month measure. We can't estimate the variability at 6 months without strong distributional assumptions on the chnage over those six months, such as assuming no change.&lt;/p&gt;&#xA;" OwnerUserId="521" LastActivityDate="2010-08-17T06:29:42.563" />
  <row Id="1769" PostTypeId="2" ParentId="1761" CreationDate="2010-08-17T07:28:37.813" Score="4" ViewCount="" Body="&lt;p&gt;I would suggest &lt;a href=&quot;http://video.google.com/videoplay?docid=-3474013489970580510&amp;amp;hl=en&amp;amp;emb=1#&quot; rel=&quot;nofollow&quot;&gt;Statistics 401&lt;/a&gt; (from Iowa State University). &#xA;This is the link for Lesson 1. There are 64 lectures in total. (Here is the &lt;a href=&quot;http://video.google.com/videoplay?docid=3835401745697888723&amp;amp;ei=coO0S9aOOJLF-Qb9kJnnBg&amp;amp;q=Statistics+401%3A+Lesson+64&amp;amp;hl=en&amp;amp;view=3#&quot; rel=&quot;nofollow&quot;&gt;last one&lt;/a&gt;). Unfortunately, I haven't found the links gathered together in one page. I think also that lessons 21,26 and 39 are missing. However, I don't remember having any problem in following the course). You can also find the handouts for this class posted &lt;a href=&quot;http://www.public.iastate.edu/~pcaragea/S401F07/Handouts07.html&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some other introductory courses in Statistics are:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://webcast.berkeley.edu/course_details_new.php?seriesid=2009-B-87384&amp;amp;semesterid=2009-B&quot; rel=&quot;nofollow&quot;&gt;Introductory Probability and Statistics for Business&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://videolectures.net/bootcamp07_keller_bss/&quot; rel=&quot;nofollow&quot;&gt;Basics of probability and statistics&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://sofia.fhda.edu/gallery/statistics/resources.html&quot; rel=&quot;nofollow&quot;&gt;Lesson Videos for Statistics&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www-group.slac.stanford.edu/sluo/lectures/Stat2007_lectures.htm&quot; rel=&quot;nofollow&quot;&gt;2007 SLUO Lectures on Statistics&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="339" LastEditorUserId="339" LastEditorDisplayName="" LastEditDate="2010-08-17T07:36:52.710" LastActivityDate="2010-08-17T07:36:52.710" CommunityOwnedDate="2010-08-17T07:28:37.813" />
  <row Id="1771" PostTypeId="2" ParentId="1757" CreationDate="2010-08-17T07:51:41.177" Score="3" ViewCount="" Body="&lt;p&gt;I agree with Srikant, you need to model your process. You mentioned that you had already created some networks in R, what model did you assume?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The way I would tackle this problem, is to form a mathematical model, say an ODE model. For example, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{equation}&#xA;\frac{dX_i(t)}{dt} = \lambda X_{i-1}(t) -\mu X_{i+1}(t)&#xA;\end{equation}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $X_i$ depends on the population at geographic unit $i$. Since you are interested in differences in time, your parameters $\lambda$ may also depend on $t$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can fit both models simultaneously and determine if the rates are different.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You problem &lt;strong&gt;isn't&lt;/strong&gt; easy and I don't think there's an simple solution to it.&lt;/p&gt;&#xA;" OwnerUserId="8" LastActivityDate="2010-08-17T07:51:41.177" CommentCount="1" />
  <row Id="1772" PostTypeId="2" ParentId="726" CreationDate="2010-08-17T08:29:54.487" Score="2" ViewCount="" Body="&lt;p&gt;It's not really &lt;em&gt;about&lt;/em&gt; statistics, but I think it applies to statistics:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;It is a capital mistake to theorize before one has data. Insensibly one begins to twist facts to suit theories, instead of theories to suit facts. &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Arthur Conan Doyle&lt;/p&gt;&#xA;" OwnerUserId="956" LastActivityDate="2010-08-17T08:29:54.487" CommunityOwnedDate="2010-08-17T08:29:54.487" CommentCount="1" />
  <row Id="1773" PostTypeId="1" AcceptedAnswerId="1775" CreationDate="2010-08-17T09:11:30.343" Score="5" ViewCount="263" Body="&lt;p&gt;Precision is defined as:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;p = true positives / (true positives + false positives)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is it correct that, as &lt;code&gt;true positives&lt;/code&gt; and &lt;code&gt;false positives&lt;/code&gt; approach 0, the precision approaches 1?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Same question for recall:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;r = true positives / (true positives + false negatives)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am currently implementing a statistical test where I need to calculate these values, and sometimes it happens that the denominator is 0, and I am wondering which value to return for this case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S.: Excuse the inappropriate tag, I wanted to use &lt;code&gt;recall&lt;/code&gt;, &lt;code&gt;precision&lt;/code&gt; and &lt;code&gt;limit&lt;/code&gt;, but I cannot create new Tags yet.&lt;/p&gt;&#xA;" OwnerUserId="977" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-16T07:01:47.313" LastActivityDate="2010-09-16T07:01:47.313" Title="What are correct values for precision and recall in edge cases?" Tags="&lt;precision-recall&gt;" AnswerCount="3" CommentCount="3" FavoriteCount="1" />
  <row Id="1774" PostTypeId="2" ParentId="1773" CreationDate="2010-08-17T09:21:16.873" Score="1" ViewCount="" Body="&lt;p&gt;That would depend on what you mean by &quot;approach 0&quot;. If false positives and false negatives both approach zero at a faster rate than true positives, then yes to both questions. But otherwise, not necessarily.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-08-17T09:21:16.873" CommentCount="1" />
  <row Id="1775" PostTypeId="2" ParentId="1773" CreationDate="2010-08-17T09:45:46.663" Score="4" ViewCount="" Body="&lt;p&gt;Given a confusion matrix:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; predicted&#xA; ---------&#xA;| TP | FN |&#xA; ---------   actual&#xA;| FP | TN |&#xA; ---------&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;we know that:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Precision = TP / (TP + FP)&#xA;Recall = TP / (TP + FN)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Lets consider the cases where the denominator is zero:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;TP+FN=0 : means that there were no positive cases in the input data&lt;/li&gt;&#xA;&lt;li&gt;TP+FP=0 : means that all instances were predicted as negative&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="170" LastActivityDate="2010-08-17T09:45:46.663" CommentCount="1" />
  <row Id="1776" PostTypeId="1" CreationDate="2010-08-17T11:23:18.773" Score="-1" ViewCount="125" Body="&lt;p&gt;The purpose to run regressions for butterfly richness again 5 environmental variables is to show the importance rank of the independent variables mainly by AIC.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In non-full models, they reveal that variable A tends to be more influential than the others by delta AIC.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, in the full model, the regression coefficient of variable A is slightly second to that of variable B. (R-square of the full model is 1.43)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The conflicting outcomes (non-full model by AIC and full model by slope) seems to make it difficult to ascertain that variable A is the variable mostly weighted.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please kindly suggest which criterion should be relied on for the specified purpose or any further test should be carried out.&#xA;Thank you.&lt;/p&gt;&#xA;" OwnerUserId="962" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-08-17T11:36:26.987" LastActivityDate="2010-08-17T11:56:05.983" Title="Shall I trust AIC (non-full model) or slope (full model)?" Tags="&lt;regression&gt;&lt;aic&gt;&lt;feature-selection&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="1777" PostTypeId="2" ParentId="1776" CreationDate="2010-08-17T11:35:36.490" Score="0" ViewCount="" Body="&lt;p&gt;Check out machine learning methods, for instance variable importance scores produced by random forest.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-08-17T11:35:36.490" />
  <row Id="1778" PostTypeId="2" ParentId="1776" CreationDate="2010-08-17T11:56:05.983" Score="3" ViewCount="" Body="&lt;p&gt;R-Squared can't be 1.43...  and other errors make your question hard to interpret.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's a sort of generic response that might eventually lead to an answer.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The AIC score tells you how good the model is similar to R-squared but penalizes it based on how many components are in the model.  You can theoretically always get a better fit with more elements to the model, and R-squared reflects that, but at some point adding more explanatory variables doesn't increase the model accuracy as much as it needlessly increases complexity.  Therefore, if you add a factor and the AIC goes down instead of up, that's because you're not adding enough explanatory power to make up for the increased complexity of the model and you should err on the side of parsimony (i.e. remove that factor).&lt;/p&gt;&#xA;" OwnerUserId="601" LastActivityDate="2010-08-17T11:56:05.983" CommentCount="1" />
  <row Id="1779" PostTypeId="2" ParentId="1651" CreationDate="2010-08-17T12:27:00.557" Score="2" ViewCount="" Body="&lt;p&gt;You can try gamlss.cens package. &lt;/p&gt;&#xA;" OwnerUserId="419" LastActivityDate="2010-08-17T12:27:00.557" />
  <row Id="1780" PostTypeId="1" AcceptedAnswerId="1785" CreationDate="2010-08-17T13:10:09.610" Score="5" ViewCount="119" Body="&lt;p&gt;Hello,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(I'm a bit outside my comfort zone, so apologies if this is badly worded, or off-topic)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a bibliographic database, containign details of about 1200 different papers, books, web sites etc, all with various details, including keywords and an abstract.  I want to somehow analyse this database and produce some graphics showing the correlations between different keywords. (like &quot;drug&quot; is often present with either &quot;pharmacology&quot; or &quot;assay&quot;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ideally this would be in R, but general advise would also be welcome. (I've seen &lt;a href=&quot;http://stats.stackexchange.com/questions/124/statistical-classification-of-text&quot;&gt;this&lt;/a&gt; question/answer which piqued my interest, and this &lt;a href=&quot;http://www2.warwick.ac.uk/fac/sci/moac/students/peter_cock/r/heatmap/&quot; rel=&quot;nofollow&quot;&gt;heatmap graphic&lt;/a&gt; also seem related)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My database could be in bibtex, or could be converted to plain text.&lt;/p&gt;&#xA;" OwnerUserId="114" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-08-17T15:05:28.463" LastActivityDate="2010-11-10T20:53:29.547" Title="How to start an analysis of keywords from a bibliography and detect correlations?" Tags="&lt;r&gt;&lt;text-mining&gt;" AnswerCount="5" FavoriteCount="2" />
  <row Id="1781" PostTypeId="1" CreationDate="2010-08-17T13:10:29.173" Score="7" ViewCount="294" Body="&lt;p&gt;Chemical analyses of environmental samples are often censored below at reporting limits or various detection/quantitation limits.  The latter can vary, usually in proportion to the values of other variables.  For example, a sample with a high concentration of one compound might need to be diluted for analysis, resulting in proportional inflation of the censoring limits for all other compounds analyzed at the same time in that sample.  As another example, sometimes the presence of a compound can alter the response of the test to other compounds (a &quot;matrix interference&quot;); when this is detected by the laboratory, it will inflate its reporting limits accordingly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am seeking a practical way to estimate the entire variance-covariance matrix for such datasets, especially when many of the compounds experience more than 50% censoring, which is often the case.  A conventional distributional model is that the logarithms of the (true) concentrations are multinormally distributed, and this appears to fit well in practice, so a solution for this situation would be useful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(By &quot;practical&quot; I mean a method that can reliably be coded in at least one generally available software environment like R, Python, SAS, etc., in a way that executes quickly enough to support iterative recalculations such as occur in multiple imputation, and which is reasonably stable [which is why I am reluctant to explore a BUGS implementation, although Bayesian solutions in general are welcome].)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks in advance for your thoughts on this matter.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-10-19T07:19:29.530" LastActivityDate="2010-12-29T00:43:51.397" Title="Unbiased estimation of covariance matrix for multiply censored data" Tags="&lt;correlation&gt;&lt;estimation&gt;&lt;censoring&gt;&lt;variance-covariance&gt;&lt;unbiased-estimator&gt;" AnswerCount="1" CommentCount="6" />
  <row Id="1782" PostTypeId="2" ParentId="1651" CreationDate="2010-08-17T13:18:59.637" Score="2" ViewCount="" Body="&lt;p&gt;Another R package that seems to do what you want, is &lt;a href=&quot;http://cran.r-project.org/web/packages/pscl/&quot; rel=&quot;nofollow&quot;&gt;pscal&lt;/a&gt;. The associated &lt;a href=&quot;http://cran.r-project.org/web/packages/pscl/vignettes/countreg.pdf&quot; rel=&quot;nofollow&quot;&gt;vignette&lt;/a&gt; has lots of examples.&lt;/p&gt;&#xA;" OwnerUserId="8" LastActivityDate="2010-08-17T13:18:59.637" CommentCount="1" />
  <row Id="1783" PostTypeId="2" ParentId="1773" CreationDate="2010-08-17T13:20:07.530" Score="2" ViewCount="" Body="&lt;p&gt;I am familiar with different terminology. What you call precision I would positive predictive value (PPV). And what you call recall I would call sensitivity (Sens). :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Receiver_operating_characteristic&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Receiver_operating_characteristic&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the case of sensitivity (recall), if the denominator is zero (as Amro points out), there are NO positive cases, so the classification is meaningless. (That does not stop either TP or FN being zero, which would result in a limiting sensitivity of 1 or 0. These points are respectively at the top right and bottom left hand corners of the ROC curve - TPR = 1 and TPR = 0.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The limit of PPV is meaningful though. It is possible for the test cut-off to be set so high (or low) so that all cases are predicted as negative. This is at the origin of the ROC curve. The limiting value of the PPV just before the cutoff reaches the origin can be estimated by considering the final segment of the ROC curve just before the origin. (This may be better to model as ROC curves are notoriously noisy.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example if there are 100 actual positives and 100 actual negatives and the final segnemt of the ROC curve approaches from TPR = 0.08, FPR = 0.02, then the limiting PPV would be PPR ~ 0.08*100/(0.08*100 + 0.02*100) = 8/10 = 0.8 i.e 80% probability of being a true positive.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In practice each sample is represented by a segment on the ROC curve - horizontal for an actual negative and vertical for an actual positive. One could estimate the limiting PPV by the very last segment before the origin, but that would give an estimated limiting PPV of 1, 0 or 0.5, depending on whether the last sample was a true positive, a false positive (actual negative) or made of an equal TP and FP. A modelling approach would be better, perhaps assuming the data are binormal - a common assumption, eg:&#xA;&lt;a href=&quot;http://mdm.sagepub.com/content/8/3/197.short&quot; rel=&quot;nofollow&quot;&gt;http://mdm.sagepub.com/content/8/3/197.short&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="521" LastActivityDate="2010-08-17T13:20:07.530" />
  <row Id="1784" PostTypeId="2" ParentId="1764" CreationDate="2010-08-17T13:30:03.980" Score="3" ViewCount="" Body="&lt;p&gt;Some additional ideas:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(1) You needn't confine yourself to a logarithmic transformation.  Search this site for the &quot;data-transformation&quot; tag, for example.  Some data lend themselves well to certain transformations like a root or a logit.  (Such transformations--even logs--are usually to be avoided when publishing graphics for a non-technical audience.  On the other hand, they can be excellent tools for seeing patterns in data.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(2) You can borrow a standard cartographic technique of insetting a detail of a chart within or next to your chart.  Specifically, you would plot the extreme values by themselves on one chart and all (or the) rest of the data on another with a more limited axis range, then graphically arrange the two along with indications (visual and/or written) of the relationship between them. Think of a map of the US in which Alaska and Hawaii are inset at different scales.  (This won't work with all kinds of charts, but could be effective with the bar charts in your illustration.) [I see this is similar to mbq's recent answer.]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(3) You can show the broken plot side-by-side with the same plot on unbroken axes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(4) In the case of your bar chart example, choose a suitable (perhaps hugely stretched) vertical axis and provide a panning utility.  [This is more of a trick than a genuinely useful technique, IMHO, but it might be useful in some special cases.]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(5) Select a different schema to display the data.  Instead of a bar chart that uses length to represent values, choose a chart in which the areas of symbols represent the values, for example.  [Obviously trade-offs are involved here.]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your choice of technique will likely depend on the purpose of the plot: plots created for data exploration often differ from plots for general audiences, for example.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-08-17T13:30:03.980" />
  <row Id="1785" PostTypeId="2" ParentId="1780" CreationDate="2010-08-17T13:52:29.993" Score="0" ViewCount="" Body="&lt;p&gt;I'm also outside my area of expertise, but assuming that you want to use R, here are a few thoughts.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;There is a &lt;a href=&quot;http://cran.r-project.org/web/packages/bibtex/bibtex.pdf&quot; rel=&quot;nofollow&quot;&gt;bibtex package in R&lt;/a&gt; for importing bibtex files.&lt;/li&gt;&#xA;&lt;li&gt;Various &lt;a href=&quot;http://www.statmethods.net/management/functions.html&quot; rel=&quot;nofollow&quot;&gt;character functions&lt;/a&gt; could be used to extract the key words.&lt;/li&gt;&#xA;&lt;li&gt;The data sounds a little like a &lt;a href=&quot;http://www.faculty.ucr.edu/~hanneman/nettext/C17_Two_mode.html&quot; rel=&quot;nofollow&quot;&gt;two-mode network&lt;/a&gt;, which might mean packages like &lt;code&gt;sna&lt;/code&gt; and &lt;code&gt;igraph&lt;/code&gt; are useful.&lt;/li&gt;&#xA;&lt;li&gt;Plots of 2d multidimensional scaling can also also be useful in visualising similarities  (e.g., based on co-occurrence or some other measure) between words (&lt;a href=&quot;http://www.statmethods.net/advstats/mds.html&quot; rel=&quot;nofollow&quot;&gt;here's a tutorial&lt;/a&gt;).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-08-17T13:52:29.993" />
  <row Id="1786" PostTypeId="2" ParentId="1780" CreationDate="2010-08-17T14:01:10.333" Score="0" ViewCount="" Body="&lt;p&gt;You may want to take a look at the &lt;a href=&quot;http://en.wikipedia.org/wiki/Phi_coefficient&quot; rel=&quot;nofollow&quot;&gt;phi coefficient&lt;/a&gt; which is a measure of association for &lt;a href=&quot;http://en.wikipedia.org/wiki/Level_of_measurement#Nominal_scale&quot; rel=&quot;nofollow&quot;&gt;nominal variables&lt;/a&gt;. &lt;/p&gt;&#xA;" LastActivityDate="2010-08-17T14:01:10.333" />
  <row Id="1787" PostTypeId="1" CreationDate="2010-08-17T15:56:39.067" Score="4" ViewCount="413" Body="&lt;p&gt;Hi all,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am fitting a GLM model (in R), and would like to get an estimation of the variability of the coefficients estimated by the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I understand it correctly the method to use in such a case is bootstraping (not, cross validation).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Am I correct that an easy way to do this is by using the boot command from the boot package, then output the coefficients at each simulation, and at the end calculate their var?  Or is there something I might be missing? &lt;/p&gt;&#xA;" OwnerUserId="253" LastActivityDate="2010-08-18T16:38:48.443" Title="Using bootstrap for glm coefficients variance estimation (in R)" Tags="&lt;r&gt;&lt;confidence-interval&gt;&lt;variance&gt;&lt;glm&gt;&lt;bootstrap&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="1" />
  <row Id="1788" PostTypeId="2" ParentId="726" CreationDate="2010-08-17T16:17:00.330" Score="3" ViewCount="" Body="&lt;p&gt;From a user on MedStats (Google Group):&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;... no scientific worker has a fixed level of significance at which &#xA;  from year to year, and in all circumstances, he rejects hypotheses; he &#xA;  rather gives his mind to each particular case in the light of his &#xA;  evidence and his ideas.&quot; - Sir Ronald A. Fisher (1956)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="561" LastActivityDate="2010-08-17T16:17:00.330" CommunityOwnedDate="2010-08-17T16:17:00.330" />
  <row Id="1789" PostTypeId="2" ParentId="1780" CreationDate="2010-08-17T16:27:30.400" Score="4" ViewCount="" Body="&lt;p&gt;Hi, so you have a document x keyword matrix which basically represents a bipartite graph (or two-mode network depending on your cultural background) with edges between documents and tags. If you're not interested in individual documents - as I understand you -, you can create a network of keywords by counting the number of cooccurrences between each keyword. Simply plotting this graph might already give you a neat idea of what this data looks like. You can further tweak the visualization if you, e.g., scale the size of the keywords by the number of total occurrences, or (in case you have a lot of keywords) introduce a minimum number of total occurrences for a keyword to appear in the first place. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a tool, I can only recommend &lt;a href=&quot;http://www.graphviz.org/&quot; rel=&quot;nofollow&quot;&gt;GraphViz&lt;/a&gt; which allows you to specify graphs like&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;keyword1 -- keyword2&#xA;keyword1 -- keyword3&#xA;keyword1[label=&quot;statistics&quot;, fontsize=...]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and &quot;compile&quot; them into pngs, pdfs, whatever, yielding very nice results (particularly if you play a bit with the font settings).&lt;/p&gt;&#xA;" OwnerUserId="979" LastActivityDate="2010-08-17T16:27:30.400" />
  <row Id="1790" PostTypeId="1" AcceptedAnswerId="1791" CreationDate="2010-08-17T16:39:24.803" Score="4" ViewCount="135" Body="&lt;p&gt;Dear all,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm trying to compare several methods by their performance on a set of synthetic data samples. For each method, I obtain a performance value between 0 and 1 for each of those samples. Then I plot a graph with the average performance per method&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem now is that the achievable quality per sample varies strongly between different samples (if you wonder why, I generate random graphs and evaluate community detection methods on them, and sometimes strange things happen, like elements from the same community get disconnected due to sparseness etc). So showing error bars based on standard deviation or standard error tend to get really large. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Imagine one method yields [1, 0.5, 1], and the other one (one the same three samples) [0.5, 0.25, 0.5]. Which measure can I apply to *de*emphasize the between-sample variance in the series and emphasize the fact that method 1 always outperforms method 2? Or, to put it differently, how can I test whether method 1 is significantly better than method 2 without being mislead by the different range of the indiviual datapoints? (Also note that I typically have more than two methods to compare, this is just for the example)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks,&#xA;Nic&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt;&#xA;One thing I've done is to count, for each method, how many times its performance is within 95% of the top performance. The picture pretty much speaks in favor of sample-based variance, not robust vs less robust methods. However, I'm still uncertain about how to generate a statistically valid statement from that..?&lt;/p&gt;&#xA;" OwnerUserId="979" LastEditorUserId="979" LastEditorDisplayName="" LastEditDate="2010-08-18T10:17:34.133" LastActivityDate="2010-08-18T10:17:34.133" Title="Meaningful deviation measure with strongly varying datapoints" Tags="&lt;standard-deviation&gt;&lt;statistical-significance&gt;&lt;standard-error&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="1791" PostTypeId="2" ParentId="1790" CreationDate="2010-08-17T16:49:24.107" Score="2" ViewCount="" Body="&lt;p&gt;You need to use some paired test, maybe paired t-test or a sign test is the distribution is really weired.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-08-17T16:49:24.107" CommentCount="4" />
  <row Id="1792" PostTypeId="2" ParentId="1764" CreationDate="2010-08-17T17:35:21.270" Score="6" ViewCount="" Body="&lt;p&gt;I am very &lt;a href=&quot;http://www.graphpad.com/faq/viewfaq.cfm?faq=1477&quot; rel=&quot;nofollow&quot;&gt;wary of using logarithmic axes on bar graphs&lt;/a&gt;. The problem is that you have to choose a starting point of the axis, and this is almost always arbitrary. You can choose to make two bars have very different heights, or almost the same height, merely by changing the minimum value on the axis. These three graphs all plot the same data:&#xA;&lt;img src=&quot;http://i.imgur.com/RzVdD.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An alternative to discontinuous axes, that no one has mentioned yet,is to simply show a table of values. In many cases, tables are easier to understand than graphs.&lt;/p&gt;&#xA;" OwnerUserId="25" LastActivityDate="2010-08-17T17:35:21.270" />
  <row Id="1794" PostTypeId="2" ParentId="1661" CreationDate="2010-08-17T18:31:59.927" Score="1" ViewCount="" Body="&lt;p&gt;From my point of view, when there are two explanatory variables and both have just two levels, we have the famous two-by-two contingency table. Fisher’s exact test can take such a matrix as its sole argument. Alternatively you can use Pearson’s chi-squared test. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your null hypothesis is not the 25:25:25:25 distribution across the four categories (i.e. say it's 9:3:3:1), you'll have to calculate the expected frequencies explicitly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then perform the chi-squared test (in R) like that:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;chisq.test(observed,p=c(9,3,3,1),rescale.p=TRUE)&#xA;# rescale.p is needed because the probabilities do not sum to 1.0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="339" LastActivityDate="2010-08-17T18:31:59.927" />
  <row Id="1795" PostTypeId="2" ParentId="1444" CreationDate="2010-08-17T18:48:40.177" Score="4" ViewCount="" Body="&lt;p&gt;A useful approach when the variable is used as an independent factor in regression is to replace it by two variables: one is a binary indicator of whether it is zero and the other is the value of the original variable or a re-expression of it, such as its logarithm.  This technique is discussed in Hosmer &amp;amp; Lemeshow's book on logistic regression (and in other places, I'm sure).  Truncated probability plots of the positive part of the original variable are useful for identifying an appropriate re-expression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When the variable is the dependent one in a linear model, censored regression (like Tobit) can be useful, again obviating the need to produce a started logarithm.  This technique is common among econometricians.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-08-17T18:48:40.177" />
  <row Id="1796" PostTypeId="2" ParentId="1790" CreationDate="2010-08-17T19:08:05.987" Score="1" ViewCount="" Body="&lt;p&gt;I am not at all sure if ignoring the performance spread is a good idea. Ideally, you would want a method to be both reliable (i.e., have low spread) and be valid (i.e., give a performance measure of close to 1). Consider the following two output measures: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Method 1. [0.80, 0.60] &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Method 2. [0.71, 0.69].&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unlike your example, there is no method that clearly dominates and in fact both methods perform equally well on average. Thus you may want to choose the one that is more reliable (i.e., has lower spread).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you accept the above reasoning then your null hypothesis should be:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{\mu_1}{\sigma_1} = \frac{\mu_2}{\sigma_2}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The above is analagous to the &lt;a href=&quot;http://en.wikipedia.org/wiki/Sharpe_ratio&quot; rel=&quot;nofollow&quot;&gt;Sharpe ratio&lt;/a&gt; from finance and I am sure there is an extensive financial literature which discusses how to test hypothesis like the above and its extensions to more than 2 groups. Unfortunately, I am not well read up on that literature to help you.&lt;/p&gt;&#xA;" LastActivityDate="2010-08-17T19:08:05.987" CommentCount="3" />
  <row Id="1797" PostTypeId="1" CreationDate="2010-08-17T19:15:32.930" Score="7" ViewCount="521" Body="&lt;p&gt;Hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was looking at this &lt;a href=&quot;http://glmm.wikidot.com/faq&quot; rel=&quot;nofollow&quot;&gt;page&lt;/a&gt; and noticed the methods for confidence intervals for lme and lmer in R.  For those who don't know R, those are functions for generating mixed effects or multi-level models.  If I have fixed effects in something like a repeated measures design what would a confidence interval around the predicted value (similar to mean) mean?  I can understand that for an effect you can have a reasonable confidence interval but it seems to me a confidence interval around a predicted mean in such designs seems to be impossible.  It could either be very large to acknowledge the fact that the random variable contributes to uncertainty in the estimate, but in that case it wouldn't be useful at all in an inferential sense comparing across values.  Or, it would have to be small enough to use inferentially but useless as an estimate of the quality of the mean (predicted) value that you could find in the population.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Am I missing something here or is my analysis of the situation correct?... [and probably a justification for why it isn't implemented in lmer (but easy to get in SAS). :)]&lt;/p&gt;&#xA;" OwnerUserId="601" LastActivityDate="2010-08-18T19:16:05.037" Title="What would a confidence interval around a predicted value from a mixed effects model mean?" Tags="&lt;r&gt;&lt;confidence-interval&gt;&lt;mixed-model&gt;&lt;repeated-measures&gt;&lt;sas&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="3" />
  <row Id="1798" PostTypeId="2" ParentId="1787" CreationDate="2010-08-17T19:30:14.080" Score="1" ViewCount="" Body="&lt;p&gt;Yes, you are right. What boot does is that it just generates new training sets by drawing with replacement from the original set. So about 2/3 of the original objects are present in each of the new sets, still the size is the same, so it does not influence model building.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-08-17T19:30:14.080" />
  <row Id="1799" PostTypeId="1" CreationDate="2010-08-17T20:00:59.700" Score="6" ViewCount="283" Body="&lt;p&gt;I am seeking recommendations and/or best practices for analyzing non-independent data. In particular, I am curious about non-independent data that does not reflect typical repeated-measures time-based data in which data for the same question(s) or stimuli are collected at different time points. Rather the data collected is elicited from similar (but not identical) questions or stimuli that are known to be related. A specific example follows.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have pain perception data for two groups (group A, group B) that can be further divided by gender (i.e., A-women; A-men; B-women; B-men). The pain perception dependent variables include 3 thermal threshold tests (i.e., detection, pain, tolerance) and pain magnitude estimates for a range of specific temperatures (temp X1, X2, X3, X4, X5). These pain perception variables are collected for both heat and cold stimuli (e.g., heat pain tolerance; cold pain tolerance; heat temp X1; cold temp X1). It should be expected (and indeed it is observed) that the various pain perception variables for a given individual are not independent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The purpose of the analysis is to look for between group differences based on group membership (group A vs. group B) and sex (women vs men). It is desirable to find interactions between group membership and sex. It is also desirable to find interactions between specific pain perception variables (or types of variables; i.e., &quot;cold stimuli&quot;) and group membership and/or sex. I have tried running a series of repeated-measures ANOVAs separately for each of the different groupings of pain perception variables (i.e., heat stimuli threshold tests; cold stimuli threshold tests; heat pain magnitude estimates; cold pain magnitude estimates); however this solution does not feel optimal or adequate. My specific questions are:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) is it appropriate to analyze data elicited from related (but not identical) questions/stimuli using repeated measures?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Is it appropriate to analyze chunks of the data (e.g., cold pain magnitude estimates; cold stimuli threshold tests) separately?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Would a different strategy (such as multilevel analysis / profile analysis / or some type of multivariate repeated measures ANOVA) be more appropriate?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) General recommendations and/or best practices for analyzing data such as this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you for your feedback and input&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Patrick Welch&lt;/p&gt;&#xA;&#xA;&lt;p&gt;note: I already searched the site for related questions (i.e., &lt;a href=&quot;http://stats.stackexchange.com/questions/859/anova-with-non-independent-observations&quot;&gt;http://stats.stackexchange.com/questions/859/anova-with-non-independent-observations&lt;/a&gt; ; &lt;a href=&quot;http://stats.stackexchange.com/questions/1324/parametric-techniques-for-n-related-samples&quot;&gt;http://stats.stackexchange.com/questions/1324/parametric-techniques-for-n-related-samples&lt;/a&gt;) but believe the current question to be different enough to warrant unique consideration.&lt;/p&gt;&#xA;" OwnerUserId="835" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-16T06:34:25.647" LastActivityDate="2010-09-16T06:34:25.647" Title="Recommendations - or best practices - for analyzing non-independent data. Specific example relating to pain perception data provided." Tags="&lt;non-independent&gt;" AnswerCount="3" FavoriteCount="4" />
  <row Id="1800" PostTypeId="2" ParentId="1023" CreationDate="2010-08-17T20:09:56.013" Score="1" ViewCount="" Body="&lt;p&gt;Tests and thousands of sample questions are available on the ARTIST (&quot;Assessment Resource Tools for Improving Statistical Thinking&quot;) site, &lt;a href=&quot;https://app.gen.umn.edu/artist/tests/index.html&quot; rel=&quot;nofollow&quot;&gt;https://app.gen.umn.edu/artist/tests/index.html&lt;/a&gt; .  Most are appropriate for an intro stats course.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-08-17T20:09:56.013" CommunityOwnedDate="2010-08-17T20:09:56.013" />
  <row Id="1801" PostTypeId="2" ParentId="928" CreationDate="2010-08-17T20:28:20.243" Score="1" ViewCount="" Body="&lt;p&gt;John Tukey strongly and cogently argued for a &lt;em&gt;proportion&lt;/em&gt; type of measurement in his book on &lt;em&gt;EDA&lt;/em&gt;.  One thing that makes proportions special and different from the classical (and long outmoded) &quot;nominal, ordinal, interval, ratio&quot; taxonomy is that frequently they enjoy an obvious symmetry: A proportion can be thought of as the average of a binary (0/1) indicator variable.  Because it should not make any meaningful difference to recode the indicator, &lt;em&gt;the data analysis should remain essentially unchanged when you re-express the proportion as its complement.&lt;/em&gt;  Specifically, recoding 0--&gt;1 and 1--&gt;0 changes the original proportion &lt;em&gt;p&lt;/em&gt; to 1-&lt;em&gt;p&lt;/em&gt;.  For example, it should make no difference to talk about 60% of people voting &quot;yes&quot; or 40% voting &quot;no&quot; in a referendum; the two numbers 0.6 and 0.4 represent exactly the same thing.  Thus, statistics, tests, decisions, summaries, etc., should  give the same results (&lt;em&gt;mutatis mutandis&lt;/em&gt;) regardless of which form of expression is used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Accordingly, Tukey used re-expressions of proportions, and analyses based on those re-expressions, that are (almost) invariant under the conversion &lt;em&gt;p&lt;/em&gt; &amp;lt;--&gt; 1-&lt;em&gt;p&lt;/em&gt;.  They are of the form &lt;em&gt;f&lt;/em&gt; (&lt;em&gt;p&lt;/em&gt;) +- &lt;em&gt;f&lt;/em&gt; (1-&lt;em&gt;p&lt;/em&gt;) for various functions &lt;em&gt;f&lt;/em&gt;.  (Taking the minus sign is usually best because it continues to distinguish between &lt;em&gt;p&lt;/em&gt; and 1-&lt;em&gt;p&lt;/em&gt;: only their signs differ when re-expressed.)  When scaled so that the differential change near &lt;em&gt;p&lt;/em&gt; = 0.5 equals 1, he called these the &quot;folded&quot; values.  Among them are the folded logarithm (&quot;flog&quot;), proportional to ln(&lt;em&gt;p&lt;/em&gt;) - ln(1-&lt;em&gt;p&lt;/em&gt;) = ln(&lt;em&gt;p&lt;/em&gt;/(1-&lt;em&gt;p&lt;/em&gt;)) = logit(&lt;em&gt;p&lt;/em&gt;), and the folded root (&quot;froot&quot;), proportional to Sqrt(&lt;em&gt;p&lt;/em&gt;) - Sqrt(1-&lt;em&gt;p&lt;/em&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A mathematical exposition of this topic is less convincing than seeing the statistics in action, so I recommend reading chapter 17 of EDA and studying the examples therein.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In sum, then, I am suggesting that the question itself is too limiting and that one should be open to possibilities that go beyond those suggested by the classical taxonomy of variables.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-08-17T20:28:20.243" />
  <row Id="1802" PostTypeId="2" ParentId="887" CreationDate="2010-08-17T20:58:12.667" Score="1" ViewCount="" Body="&lt;p&gt;The &lt;strong&gt;second question&lt;/strong&gt; seems to ask for a prediction interval for one future observation.  Such an interval is readily calculated under the assumptions that (a) the future observation is from the same distribution and (b) is independent of the previous sample.  When the underlying distribution is Normal, we just have to erect an interval around the difference of two Gaussian random variables.  Note that the interval will be &lt;em&gt;wider&lt;/em&gt; than suggested by a naive application of a t-test or z-test, because it has to accommodate the variance of the future value, too.  This rules out all the answers I have seen posted so far, so I guess I had better quote one explicitly.  Hahn &amp;amp; Meeker's formula for the endpoints of this prediction interval is&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;m +- t * Sqrt(1 + 1/n) * s&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where &lt;em&gt;m&lt;/em&gt; is the sample mean, &lt;em&gt;t&lt;/em&gt; is an appropriate two-sided critical value of Student's t (for &lt;em&gt;n&lt;/em&gt;-1 df), &lt;em&gt;s&lt;/em&gt; is the sample standard deviation, and &lt;em&gt;n&lt;/em&gt; is the sample size.  Note in particular the factor of Sqrt(1+1/&lt;em&gt;n&lt;/em&gt;) instead of Sqrt(1/&lt;em&gt;n&lt;/em&gt;).  That's a big difference!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This interval is used like any other interval: the requested test simply examines whether the new value lies within the prediction interval.  If so, the new value is consistent with the sample; if not, we reject the hypothesis that it was independently drawn from the same distribution as the sample.  Generalizations from one future value to &lt;em&gt;k&lt;/em&gt; future values or to the mean (or max or min) of &lt;em&gt;k&lt;/em&gt; future values, etc., exist.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a extensive literature on prediction intervals especially in a regression context.  Any decent regression textbook will have formulas.  You could begin with the Wikipedia entry ;-).  Hahn &amp;amp; Meeker's &lt;em&gt;Statistical Intervals&lt;/em&gt; is still in print and is an accessible read.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;strong&gt;first question&lt;/strong&gt; has an an answer that is so routine nobody seems yet to have given it here (although some of the links provide details).  For completeness, then, I will close by remarking that when the population has approximately a Normal distribution, the sample standard deviation is distributed as the square root of a scaled chi-square variate of &lt;em&gt;n&lt;/em&gt;-1 df whose expectation is the population variance.  That means (roughly) we expect the sample sd to be close to the population sd and the difference between the two usually will be on the order of Sqrt(&lt;em&gt;n&lt;/em&gt;-1) times the population sd.  Unlike parallel statements for the sample mean (which invoke the CLT), this statement relies fairly strongly on the assumption of a Normal population.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-08-17T20:58:12.667" />
  <row Id="1803" PostTypeId="2" ParentId="1780" CreationDate="2010-08-17T21:25:25.937" Score="0" ViewCount="" Body="&lt;p&gt;You could try to employ the &lt;a href=&quot;http://www-users.cs.umn.edu/~kumar/dmbook/ch6.pdf&quot; rel=&quot;nofollow&quot;&gt;theory&lt;/a&gt; and &lt;a href=&quot;http://cran.r-project.org/web/packages/arules/vignettes/arules.pdf&quot; rel=&quot;nofollow&quot;&gt;praxis&lt;/a&gt; of association analysis or market basket analysis to your problem (just read &quot;items&quot; as &quot;keywords&quot; / &quot;cited reference&quot; and &quot;market basket&quot; as &quot;journal article&quot;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Disclaimer - this is just an idea, I did not do anything like that myself. Just my 2Cents.&lt;/p&gt;&#xA;" OwnerUserId="573" LastActivityDate="2010-08-17T21:25:25.937" />
  <row Id="1804" PostTypeId="2" ParentId="1268" CreationDate="2010-08-17T22:49:17.710" Score="1" ViewCount="" Body="&lt;p&gt;Just a thought: you might not need the full SVD for your problem.  Let &lt;strong&gt;M&lt;/strong&gt; = &lt;strong&gt;U S V*&lt;/strong&gt; be the SVD of your &lt;em&gt;d&lt;/em&gt; by &lt;em&gt;n&lt;/em&gt; matrix (&lt;em&gt;i.e.&lt;/em&gt;, the time series are the columns).  To achieve the dimension reduction you'll be using the matrices &lt;strong&gt;V&lt;/strong&gt; and &lt;strong&gt;S&lt;/strong&gt;.  You can find them by diagonalizing &lt;strong&gt;M* M&lt;/strong&gt; = &lt;strong&gt;V (S*S) V*&lt;/strong&gt;.  However, because you are missing some values, you cannot compute &lt;strong&gt;M* M&lt;/strong&gt;.  Nevertheless, you can estimate it.  Its entries are sums of products of columns of &lt;strong&gt;M&lt;/strong&gt;.  When computing any of the SSPs, ignore pairs involving missing values.  Rescale each product to account for the missing values: that is, whenever a SSP involves &lt;em&gt;n-k&lt;/em&gt; pairs, rescale it by &lt;em&gt;n/(n-k).&lt;/em&gt;  This procedure is a &quot;reasonable&quot; estimator of &lt;strong&gt;M* M&lt;/strong&gt; and you can proceed from there.  If you want to get fancier, maybe multiple imputation techniques or &lt;em&gt;Matrix Completion&lt;/em&gt; will help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(This can be carried out in many statistical packages by computing a pairwise covariance matrix of the transposed dataset and applying PCA or factor analysis to it.)&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-08-17T22:49:17.710" CommentCount="2" />
  <row Id="1805" PostTypeId="1" AcceptedAnswerId="1806" CreationDate="2010-08-17T23:42:12.133" Score="8" ViewCount="938" Body="&lt;p&gt;I was taught to only apply Fisher's Exact Test in contingency tables that were 2x2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Questions:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Did Fisher himself ever envision this test to be used in tables larger than 2x2 (I am aware of the tale of him devising the test while trying to guess &lt;a href=&quot;http://en.wikipedia.org/wiki/Fisher%27s_exact_test&quot; rel=&quot;nofollow&quot;&gt;whether an old woman could tell if milk was added to tea or tea was added to milk&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Stata allows me to use Fisher's exact test to any contingency table. Is this valid?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) Is it preferable to use FET when expected cell counts in a contingency table are &amp;lt; 5?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks!&lt;/p&gt;&#xA;" OwnerUserId="561" LastActivityDate="2010-09-20T16:49:00.773" Title="Fisher's Exact Test in contingency tables larger than 2x2" Tags="&lt;contingency-tables&gt;" AnswerCount="2" FavoriteCount="5" />
  <row Id="1806" PostTypeId="2" ParentId="1805" CreationDate="2010-08-18T00:10:56.507" Score="7" ViewCount="" Body="&lt;p&gt;The only problem with applying Fisher's exact test to tables larger than 2x2  is that the calculations become much more difficult to do. The 2x2 version is the only one which is even feasible by hand, and so I doubt that Fisher ever imagined the test in larger tables because the computations would have been beyond anything he would have envisaged.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Nevertheless, the test can be applied to any mxn table and some software including Stata and SPSS provide the facility. Even so, the calculation is often approximated using a Monte Carlo approach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Yes, if the expected cell counts are small, it is better to use an exact test as the chi-squared test is no longer a good approximation in such cases.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-08-18T00:10:56.507" CommentCount="2" />
  <row Id="1807" PostTypeId="1" CreationDate="2010-08-18T01:39:31.013" Score="6" ViewCount="451" Body="&lt;p&gt;Student's $t$-test requires the sample standard deviation $s$. However, how do I compute for $s$ when only the sample size and sample average are known?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if sample size is $49$ and sample average is $112$, I will then attempt to create a list of $49$ identical samples with values of $112$ each. Expectedly, the sample standard deviation is $0$. This will create a divide-by-zero problem in the $t$ test.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;ADDITIONAL DATA:&lt;/strong&gt;&lt;br&gt;&#xA;The average income of ACME North Factory workers is $\$200$. It is reported that a random sample of $49$ workers in ACME South Factory had an annual income of $\$112$. Is this difference statistically significant?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Am I correct in saying that the population mean is $\$200$?&lt;/p&gt;&#xA;" OwnerUserId="850" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-08-18T06:22:24.297" LastActivityDate="2010-08-19T17:09:15.637" Title="How to perform Student's t-test having only sample size, sample average and population average are known?" Tags="&lt;standard-deviation&gt;&lt;t-test&gt;" AnswerCount="5" CommentCount="0" FavoriteCount="3" />
  <row Id="1809" PostTypeId="2" ParentId="1807" CreationDate="2010-08-18T02:08:34.550" Score="0" ViewCount="" Body="&lt;p&gt;What problem are you trying to solve? It would help us help you if you told us more.&lt;/p&gt;&#xA;" OwnerUserId="561" LastActivityDate="2010-08-18T02:08:34.550" CommentCount="3" />
  <row Id="1810" PostTypeId="2" ParentId="1807" CreationDate="2010-08-18T02:11:10.360" Score="0" ViewCount="" Body="&lt;p&gt;I presume you are referring to a one sample t test. Its goal is to compare the mean of your sample with a hypothetical mean. It then computes (assuming your population is Gaussian) a P value that answers this question: If the population mean really was the hypothetical value, how unlikely would it be to draw a sample whose mean is as far from that value (or further) than you observed? Of course, the answer to that question depends on sample size. But it also depends on variability. If your data have a huge amount of scatter, they are consistent with a broad range of population means. If your data are really tight, they are consistent with a smaller range of population means. &lt;/p&gt;&#xA;" OwnerUserId="25" LastEditorUserId="25" LastEditorDisplayName="" LastEditDate="2010-08-18T06:34:34.960" LastActivityDate="2010-08-18T06:34:34.960" />
  <row Id="1811" PostTypeId="2" ParentId="1807" CreationDate="2010-08-18T05:40:49.263" Score="6" ViewCount="" Body="&lt;p&gt;This does look to be a slightly contrived question. 49 is an exact square of 7. The value of a t-distribution with 48 DoF for a two-sided test of p&amp;lt;0.05 is very nearly 2 (2.01).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We reject the null hypothesis of equality of means if |sample_mean - popn_mean| &gt; 2*StdError, i.e. 200-112 &gt; 2*SE so SE &amp;lt; 44, i.e. SD &amp;lt; 7*44 = 308.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It would be impossible to get a normal distribution with a mean of 112 with a standard deviation of 308 (or more) without negative wages. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given wages are bounded below, they are likely to be skew, so assuming a log-normal distribution would be more appropriate, but it would still require highly variable wages to avoid a p&amp;lt;0.05 on a t-test.&lt;/p&gt;&#xA;" OwnerUserId="521" LastActivityDate="2010-08-18T05:40:49.263" CommentCount="0" />
  <row Id="1812" PostTypeId="1" AcceptedAnswerId="2043" CreationDate="2010-08-18T07:36:59.733" Score="5" ViewCount="352" Body="&lt;p&gt;One of the most important issues in using factor analysis is its interpretation. Factor analysis often uses factor rotation to enhance its interpretation. After a satisfactory rotation, the rotated factor loading matrix &lt;strong&gt;L'&lt;/strong&gt; will have the same ability to represent the correlation matrix and it can be used as the factor loading matrix, instead of the unrotated matrix &lt;strong&gt;L&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The purpose of rotation is to make the rotated factor loading matrix have some desirable properties. One of the methods used is to rotate the factor loading matrix such that the rotated matrix will have a &lt;strong&gt;simple structure&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;L. L. Thurstone introduced the Principle of Simple Structure, as a general guide for factor rotation:&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Simple Structure Criteria:&lt;/h2&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Each row of the factor matrix should contain at least one zero&lt;/li&gt;&#xA;&lt;li&gt;If there are m common factors, each column of the factor matrix should have at least m zeros&lt;/li&gt;&#xA;&lt;li&gt;For every pair of columns in the factor matrix, there should be several variables for which entries approach zero in the one column but not in the other&lt;/li&gt;&#xA;&lt;li&gt;For every pair of columns in the factor matrix, a large proportion of the variables should have entries approaching zero in both columns when there are four or more factors&lt;/li&gt;&#xA;&lt;li&gt;For every pair of columns in the factor matrix, there should be only a small number of variables with nonzero entries in both columns&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The ideal simple structure is such that:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;each item has a high, or meaningful, loading on one factor only and&lt;/li&gt;&#xA;&lt;li&gt;each factor have high, or meaningful, loadings for only some of the items.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The problem is that, trying several combinations of rotation methods along with the parameters that each one accepts (especially for oblique ones), the number of candidate matrices increases and it is very difficult to see which one better meets the above criteria.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I first faced that problem I realized that I was unable to select the best match by merely 'looking' at them, and that I needed an algorithm to help me decide. Under the stress of project's deadlines, the most I could do was to write the following code in MATLAB, which accepts one rotation matrix at a time and returns (under some assumptions) whether each criterion is met or not.&#xA;A new version (If I would ever tried to upgrade it) would accept a 3d matrix (a set of 2d matrices) as an argument, and the algorithm should return the one that better fits the above criteria.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How would you extract an algorithm out of those criteria? I am just asking for your opinions (I also think that there's been criticism over the usefulness of the method by itself) and perhaps better approaches to the rotation matrix selection problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, I would like to know what software do you prefer to perform FA. If it's R, what package do you use? (I must admit that if I had to do FA, I would turn to SPSS again). If someone wants to provide some code, I would prefer R or MATLAB. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S. The above &lt;a href=&quot;http://books.google.gr/books?id=5Jyaa2LQWbQC&amp;amp;pg=PA132&amp;amp;lpg=PA132&amp;amp;dq=%22The+criteria+of+simple+structure+was+originally+proposed%22&amp;amp;source=bl&amp;amp;ots=OXsCeU0Nzi&amp;amp;sig=qfiLMUyXsgwPunWBgNdSEeBNm34&amp;amp;hl=el&amp;amp;ei=huZeS5nkApTE4gaC67HsCw&amp;amp;sa=X&amp;amp;oi=book_result&amp;amp;ct=result&amp;amp;resnum=1&amp;amp;ved=0CAcQ6AEwAA#v=onepage&amp;amp;q=%22The%20criteria%20of%20simple%20structure%20was%20originally%20proposed%22&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;Simple Structure Criteria formulation&lt;/a&gt; can be found in the book &lt;em&gt;&quot;Making Sense of Factor Analysis&quot;&lt;/em&gt; by PETT, M., LACKEY, N., SULLIVAN, J.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S.2 (from the same book): &lt;em&gt;&quot;A test of successful factor analysis is the extent to which it can reproduce the original corr matrix. If you also used oblique solutions, among all select the one that generated the greatest number of highest and lowest factor loadings.&quot;&lt;/em&gt; &#xA;This sounds like another constraint that the algorithm could use.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S.3 This question has also been asked &lt;a href=&quot;http://stackoverflow.com/questions/2139703/fa-choosing-rotation-matrix-based-on-simple-structure-criteria&quot;&gt;here&lt;/a&gt;. However, I think it fits better on this site.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;function [] = simple_structure_criteria (my_pattern_table)&#xA;%Simple Structure Criteria&#xA;%Making Sense of Factor Analysis, page 132&#xA;&#xA;disp(' ');&#xA;disp('Simple Structure Criteria (Thurstone):');&#xA;disp('1. Each row of the factor matrix should contain at least one zero');&#xA;disp( '2. If there are m common factors, each column of the factor matrix should have at least m zeros');&#xA;disp( '3. For every pair of columns in the factor matrix, there should be several variables for which entries approach zero in the one column but not in the other');&#xA;disp( '4. For every pair of columns in the factor matrix, a large proportion of the variables should have entries approaching zero in both columns when there are four or more factors');&#xA;disp( '5. For every pair of columns in the factor matrix, there should be only a small number of variables with nonzero entries in both columns');&#xA;disp(' ');&#xA;disp( '(additional by Pedhazur and Schmelkin) The ideal simple structure is such that:');&#xA;disp( '6. Each item has a high, or meaningful, loading on one factor only and');&#xA;disp( '7. Each factor have high, or meaningful, loadings for only some of the items.');&#xA;&#xA;disp('')&#xA;disp('Start checking...')&#xA;&#xA;%test matrix&#xA;%ct=[76,78,16,7;19,29,10,13;2,6,7,8];&#xA;%test it by giving: simple_structure_criteria (ct)&#xA;&#xA;ct=abs(my_pattern_table);&#xA;&#xA;items=size(ct,1);&#xA;factors=size(ct,2);&#xA;my_zero = 0.1;&#xA;approach_zero = 0.2;&#xA;several = floor(items / 3);&#xA;small_number = ceil(items / 4);&#xA;large_proportion = 0.30;&#xA;meaningful = 0.4;&#xA;some_bottom = 2;&#xA;some_top = floor(items / 2);&#xA;&#xA;% CRITERION 1&#xA;disp(' ');&#xA;disp('CRITERION 1');&#xA;for i = 1 : 1 : items&#xA;    count = 0;&#xA;    for j = 1 : 1 : factors&#xA;        if (ct(i,j) &amp;lt; my_zero)&#xA;            count = count + 1;&#xA;            break&#xA;        end&#xA;    end&#xA;    if (count == 0)&#xA;        disp(['Criterion 1 is NOT MET for item ' num2str(i)])&#xA;    end&#xA;end&#xA;&#xA;&#xA;% CRITERION 2&#xA;disp(' ');&#xA;disp('CRITERION 2');&#xA;for j = 1 : 1 : factors &#xA;    m=0;&#xA;    for i = 1 : 1 : items&#xA;        if (ct(i,j) &amp;lt; my_zero)&#xA;            m = m + 1;&#xA;        end&#xA;    end&#xA;    if (m &amp;lt; factors)&#xA;        disp(['Criterion 2 is NOT MET for factor ' num2str(j) '. m = ' num2str(m)]);&#xA;    end&#xA;end&#xA;&#xA;% CRITERION 3&#xA;disp(' ');&#xA;disp('CRITERION 3');&#xA;for c1 = 1 : 1 : factors - 1&#xA;    for c2 = c1 + 1 : 1 : factors&#xA;        test_several = 0;&#xA;        for i = 1 : 1 : items&#xA;            if ( (ct(i,c1)&amp;gt;my_zero &amp;amp;&amp;amp; ct(i,c2)&amp;lt;my_zero) || (ct(i,c1)&amp;lt;my_zero &amp;amp;&amp;amp; ct(i,c2)&amp;gt;my_zero) ) % approach zero in one but not in the other&#xA;                test_several = test_several + 1;&#xA;            end&#xA;        end&#xA;        disp(['several = ' num2str(test_several) ' for factors ' num2str(c1) ' and ' num2str(c2)]);&#xA;        if (test_several &amp;lt; several)&#xA;            disp(['Criterion 3 is NOT MET for factors ' num2str(c1) ' and ' num2str(c2)]);&#xA;        end&#xA;    end&#xA;end&#xA;&#xA;% CRITERION 4&#xA;disp(' ');&#xA;disp('CRITERION 4');&#xA;if (factors &amp;gt; 3)&#xA;    for c1 = 1 : 1 : factors - 1&#xA;        for c2 = c1 + 1 : 1 : factors&#xA;            test_several = 0;&#xA;            for i = 1 : 1 : items&#xA;                if (ct(i,c1)&amp;lt;approach_zero &amp;amp;&amp;amp; ct(i,c2)&amp;lt;approach_zero) % approach zero in both&#xA;                    test_several = test_several + 1;&#xA;                end&#xA;            end&#xA;            disp(['large proportion = ' num2str((test_several / items)*100) '% for factors ' num2str(c1) ' and ' num2str(c2)]);&#xA;            if ((test_several / items) &amp;lt; large_proportion)&#xA;                pr = sprintf('%4.2g',  (test_several / items) * 100 );&#xA;                disp(['Criterion 4 is NOT MET for factors ' num2str(c1) ' and ' num2str(c2) '. Proportion is ' pr '%']);&#xA;            end&#xA;        end&#xA;    end&#xA;end&#xA;&#xA;% CRITERION 5&#xA;disp(' ');&#xA;disp('CRITERION 5');&#xA;for c1 = 1 : 1 : factors - 1&#xA;    for c2 = c1 + 1 : 1 : factors&#xA;        test_number = 0;&#xA;        for i = 1 : 1 : items&#xA;            if (ct(i,c1)&amp;gt;approach_zero &amp;amp;&amp;amp; ct(i,c2)&amp;gt;approach_zero) % approach zero in both&#xA;                test_number = test_number + 1;&#xA;            end&#xA;        end&#xA;        disp(['small number = ' num2str(test_number) ' for factors ' num2str(c1) ' and ' num2str(c2)]);&#xA;        if (test_number &amp;gt; small_number)&#xA;            disp(['Criterion 5 is NOT MET for factors ' num2str(c1) ' and ' num2str(c2)]);&#xA;        end&#xA;    end&#xA;end&#xA;&#xA;% CRITERION 6&#xA;disp(' ');&#xA;disp('CRITERION 6');&#xA;for i = 1 : 1 : items&#xA;    count = 0;&#xA;    for j = 1 : 1 : factors&#xA;        if (ct(i,j) &amp;gt; meaningful)&#xA;            count = count + 1;&#xA;        end&#xA;    end&#xA;    if (count == 0 || count &amp;gt; 1)&#xA;        disp(['Criterion 6 is NOT MET for item ' num2str(i)])&#xA;    end&#xA;end&#xA;&#xA;% CRITERION 7&#xA;disp(' ');&#xA;disp('CRITERION 7');&#xA;for j = 1 : 1 : factors &#xA;    m=0;&#xA;    for i = 1 : 1 : items&#xA;        if (ct(i,j) &amp;gt; meaningful)&#xA;            m = m + 1;&#xA;        end&#xA;    end&#xA;    disp(['some items = ' num2str(m) ' for factor ' num2str(j)]);&#xA;    if (m &amp;lt; some_bottom || m &amp;gt; some_top)&#xA;        disp(['Criterion 7 is NOT MET for factor ' num2str(j)]);&#xA;    end&#xA;end&#xA;disp('')&#xA;disp('Checking completed.')&#xA;return&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="339" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-01-22T11:35:40.123" LastActivityDate="2011-01-22T11:35:40.123" Title="FA: Choosing Rotation matrix, based on &quot;Simple Structure Criteria&quot;" Tags="&lt;r&gt;&lt;algorithms&gt;&lt;factor-analysis&gt;&lt;psychometrics&gt;&lt;matlab&gt;" AnswerCount="4" FavoriteCount="4" />
  <row Id="1813" PostTypeId="1" CreationDate="2010-08-18T08:29:52.133" Score="6" ViewCount="166" Body="&lt;p&gt;I have two implementations of a genetic algorithm which are supposed to behave equivalently. However due to technical restrictions which cannot be resolved their output is not exactly the same, given the same input.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Still I'd like to show that there is no significant performance difference.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have 20 runs with the same configuration for each of the two algorithms, using different initial random number seeds. For each run &lt;strong&gt;and generation&lt;/strong&gt; the &lt;strike&gt;minimum error&lt;/strike&gt; &lt;strong&gt;fitness of the best individual in the population&lt;/strong&gt; was recorded. &lt;em&gt;The algorithm employs an elite-preserving mechanism, so the fitness of the best individual is monotonically decreasing.&lt;/em&gt; A run consists of 1000 generations, so I have 1000 values per run. I cannot get more data, as the calculations are very expensive.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which test should I employ? An easy way would probably be to only compare the error in the final generations (again, which test would I use here)? But one might also think about comparing the convergence behaviour in general.&lt;/p&gt;&#xA;" OwnerUserId="986" LastEditorUserId="986" LastEditorDisplayName="" LastEditDate="2010-08-18T15:25:32.860" LastActivityDate="2010-08-18T15:25:32.860" Title="Comparing two genetic algorithms" Tags="&lt;hypothesis-testing&gt;&lt;genetic-algorithms&gt;" AnswerCount="2" CommentCount="3" FavoriteCount="3" />
  <row Id="1814" PostTypeId="2" ParentId="1812" CreationDate="2010-08-18T08:37:21.273" Score="2" ViewCount="" Body="&lt;p&gt;Great Question. This is not really an answer, but just a few thoughts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In most of the applications where I have used factor analysis, permitting correlated factors makes more theoretical sense. I tend to rely on the proxmax rotation method. I used to do this in SPSS and now I use the &lt;code&gt;factanal&lt;/code&gt; function in R. &lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-08-18T08:37:21.273" />
  <row Id="1815" PostTypeId="1" CreationDate="2010-08-18T08:54:23.650" Score="7" ViewCount="374" Body="&lt;p&gt;What are the panel's recommendations for books on design of experiments?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ideally, books should be still in print or available electronically, although that may not always be feasible. If you feel moved to add a few words on what's so good about the book that would be great too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, aim for one book per answer so that voting can help sort the suggestions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Community Wiki, please edit the question if you can make it better!)&lt;/p&gt;&#xA;" OwnerUserId="174" LastActivityDate="2010-09-19T09:34:20.480" CommunityOwnedDate="2010-08-18T08:54:23.650" Title="Recommended books on experiment design?" Tags="&lt;books&gt;&lt;experiment-design&gt;" AnswerCount="10" CommentCount="4" FavoriteCount="4" />
  <row Id="1816" PostTypeId="2" ParentId="1813" CreationDate="2010-08-18T08:58:46.820" Score="5" ViewCount="" Body="&lt;p&gt;Testing stochastic algorithms can be rather tricky!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I work in systems biology and there are many stochastic simulators available to use to simulate a model. Testing these simulators is tricky since any two realizations from a single model will be typically different.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the &lt;a href=&quot;http://code.google.com/p/dsmts/&quot; rel=&quot;nofollow&quot;&gt;dsmts&lt;/a&gt; we have calculated (analytically) the expected value and variance of a particular model. We then perform a hypothesis test to determine if a simulator differs from the truth. Section 3 of the &lt;a href=&quot;http://code.google.com/p/dsmts/downloads/detail?name=dsmts-userguide31.pdf&quot; rel=&quot;nofollow&quot;&gt;userguide&lt;/a&gt; gives the details. Essentially we do a t-test for the mean values and a chi-squared test for variances.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case, you are comparing two simulators so you should just use a two-sampled t-test instead.&lt;/p&gt;&#xA;" OwnerUserId="8" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-08-18T12:23:43.077" LastActivityDate="2010-08-18T12:23:43.077" CommentCount="5" />
  <row Id="1817" PostTypeId="2" ParentId="1815" CreationDate="2010-08-18T10:40:25.813" Score="3" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0470128666&quot; rel=&quot;nofollow&quot;&gt;Montgomery's Design and Analysis of Experiments&lt;/a&gt; is a classic and highly regarded text:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are interested in experimental design in a particular field (eg. clinical trials) other more specialised texts may be appropriate.&lt;/p&gt;&#xA;" OwnerUserId="521" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-08-18T13:07:11.413" LastActivityDate="2010-08-18T13:07:11.413" CommunityOwnedDate="2010-08-18T10:40:25.813" CommentCount="2" />
  <row Id="1818" PostTypeId="1" AcceptedAnswerId="1823" CreationDate="2010-08-18T10:56:21.970" Score="6" ViewCount="1063" Body="&lt;p&gt;I need some help about repeated measurement ANOVA.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We are investigating the effect of some intervention on reducing blood stream infection (BSI) rate in some wards. We plan to capture the BSI rate information at a monthly basis, 12 months without intervention first, then 12 months with intervention.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We are thinking of doing either time-series or repeated measurement ANOVA, I prefer the later one before I don't have much idea to do on the first one (extra question: too little time points, right?), but then here come another problem, how many wards do we need to show that there is really a statistically significant effect of intervention on BSI rate?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think I'll do two ANOVA, one for &quot;before intervention&quot;, one for &quot;during intervention&quot;, and I suppose that the ANOVA &quot;before intervention&quot; should not have a significant F-ratio test.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I consider the term &quot;sample size&quot; two-dimensionally, either the number of wards, or the number of repeated measurements.&lt;/p&gt;&#xA;" OwnerUserId="588" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-08-18T11:03:13.940" LastActivityDate="2010-08-18T11:52:36.243" Title="How to determine the sample size needed for repeated measurement ANOVA?" Tags="&lt;anova&gt;&lt;repeated-measures&gt;&lt;sample-size&gt;&lt;power&gt;" AnswerCount="2" FavoriteCount="2" />
  <row Id="1819" PostTypeId="2" ParentId="1815" CreationDate="2010-08-18T10:56:44.153" Score="0" ViewCount="" Body="&lt;p&gt;If you're interested in pharmaceutical trials, two books I recommend:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Statistical Issues in Drug Development by Stephen Senn (&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0471974889&quot; rel=&quot;nofollow&quot;&gt;Amazon link&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;li&gt;Cross-over Trials in Clinical Research by Stephen Senn (&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0471934933&quot; rel=&quot;nofollow&quot;&gt;Amazon link&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="561" LastActivityDate="2010-08-18T10:56:44.153" CommunityOwnedDate="2010-08-18T10:56:44.153" />
  <row Id="1820" PostTypeId="2" ParentId="1818" CreationDate="2010-08-18T11:03:02.863" Score="1" ViewCount="" Body="&lt;p&gt;You need to look at &lt;a href=&quot;http://en.wikipedia.org/wiki/Statistical_power&quot; rel=&quot;nofollow&quot;&gt;power calculations&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Searching for &quot;power calculations for repeated measurements&quot; in Google provides a good starting point. The &lt;a href=&quot;http://statpages.org/&quot; rel=&quot;nofollow&quot;&gt;first hit&lt;/a&gt; seems to give some good pointers.&lt;/p&gt;&#xA;" OwnerUserId="8" LastActivityDate="2010-08-18T11:03:02.863" />
  <row Id="1821" PostTypeId="2" ParentId="887" CreationDate="2010-08-18T11:09:28.313" Score="0" ViewCount="" Body="&lt;p&gt;1) The standard deviation of the sample (stdev(S)) is an unbiased estimate of the standard deviation of the population.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Given we have estimated both the population mean and variance we need to take this into account when we evaluate whether a new observation x is a member of this population.&#xA;We don't use Z = (x - mean(S))/stdev(S), but rather:&#xA; t = (x - mean(S))/(stdev(S)*sqrt(1 + 1/n)), where n is the sample size of the first sample. We the compare t with a t-distribution with n-1 degrees of freedom to give a p-value. See here:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Prediction_interval#Unknown_mean.2C_unknown_variance&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Prediction_interval#Unknown_mean.2C_unknown_variance&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This accounts for the sample size both in the divisor (sqrt(1 + 1/n)) and in the degrees of freedom of the t-distribution.&lt;/p&gt;&#xA;" OwnerUserId="521" LastActivityDate="2010-08-18T11:09:28.313" CommentCount="2" />
  <row Id="1822" PostTypeId="1" AcceptedAnswerId="1827" CreationDate="2010-08-18T11:37:10.547" Score="3" ViewCount="199" Body="&lt;p&gt;How do i test/verify if i can analyze my panel like dataset by simply pooling the individual series?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a dataset structured as a panel. Now I am wondering if i can simply&#xA;pool the individual series and estimate it via OLS or if I have to use another estimation technique.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Any R hints and references are highly welcomed.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="283" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-16T12:41:57.140" LastActivityDate="2010-09-16T12:41:57.140" Title="Test for Poolability of Individual Data Series" Tags="&lt;survey&gt;" AnswerCount="3" FavoriteCount="2" />
  <row Id="1823" PostTypeId="2" ParentId="1818" CreationDate="2010-08-18T11:40:44.177" Score="5" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;How to perform power analysis on repeated measures ANOVA?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.psycho.uni-duesseldorf.de/abteilungen/aap/gpower3/&quot; rel=&quot;nofollow&quot;&gt;G*Power 3&lt;/a&gt; is free software that provides a user-friendly GUI interface for performing power calculations.&#xA;It supports power calculations for repeated measures ANOVA.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What is the appropriate analysis for your design?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are a range of points related to what you have mentioned:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;More time points will give a clearer indication&#xA;of how the effect, if any, of your intervention operates over time. Thus, if the improvements decay over time or get greater, more time points will give a clearer sense of these patterns, both on average, and at an individual-level.&lt;/li&gt;&#xA;&lt;li&gt;If you have 12 time points or more, I'd look at multilevel modelling, particularly if you are expecting any missing observations. You are unlikely to be interested in  whether there is an effect of time. Rather you are likely to be interested in various specific effects (e.g., changes pre and post intervention; perhaps a linear or quadratic improvement effect post-intervention). You could also look at using planned contrasts on top of repeated measures ANOVA. &lt;a href=&quot;http://www.ats.ucla.edu/stat/examples/alda.htm&quot; rel=&quot;nofollow&quot;&gt;Applied Longitudinal Data Analysis: Modeling Change and Event Occurrence&lt;/a&gt; is a good starting point to learn about multilevel modelling of repeated measures data.&lt;/li&gt;&#xA;&lt;li&gt;The number of time points over and above pre- and post- wont do much to increase your   power to detect the effect of your intervention. More time points will increase your reliability of measurement, and it might ensure that you capture the time period where the effect applies, but probably the bigger issue will be the sample size in the two conditions.&lt;/li&gt;&#xA;&lt;li&gt;Assuming you are truly randomly allocating cases to conditions, the populations are by definition equal on the dependent variable, and one could argue that a significance test of baseline differences is meaningless. That said, researchers often still do it, and I suppose it does provide some evidence that random allocation has actually occurred.&lt;/li&gt;&#xA;&lt;li&gt;There is a fair amount of debate about the best way to test the effect of an intervention in a pre-post-interention-control design. A few options include: (a) the condition * time interaction; (b) the effect of condition but just at post intervention; (c) an ANCOVA looking at the effect of condition, controlling for pre, with post as the DV.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-08-18T11:52:36.243" LastActivityDate="2010-08-18T11:52:36.243" CommentCount="3" />
  <row Id="1824" PostTypeId="2" ParentId="1813" CreationDate="2010-08-18T12:00:36.353" Score="3" ViewCount="" Body="&lt;p&gt;Maybe you could measure the average difference between two runs of the same algorithm to the average difference between two runs from different algorithms. Doesn't solve the problem of how to measure that difference, but might be a more tractable problem. And the individual values of the time series would feed into the difference calculation instead of having to be treated as individual datapoints to be evaluated against each other (I also don't think that the particular difference at the nth step is what you really want to make statements about).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt;&#xA;Concerning details - well which features of the time series are you interested in, beyond the final error? I guess you actually got three different questions to solve: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;What constitues similarity for you, ie what do you mean when you say you don't believe the two methods are different?&lt;/li&gt;&#xA;&lt;li&gt;How do you quantify it - can be answered after 1, and &lt;/li&gt;&#xA;&lt;li&gt;How can you test for significant differences between your two methods?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;All I was saying in the first post was that the answer to (1) probably doesn't consider the individual differences at each of the 1000 generations. And that I'd advise coming up with a scalar value for either each time series or at least similarity between time series. Only then you get to the actual statistics question (which I know least about of all three points, but I was advised to use a paired t-test in a similar question I just asked, when having a scalar value per element).&lt;/p&gt;&#xA;" OwnerUserId="979" LastEditorUserId="979" LastEditorDisplayName="" LastEditDate="2010-08-18T14:09:05.340" LastActivityDate="2010-08-18T14:09:05.340" CommentCount="1" />
  <row Id="1825" PostTypeId="2" ParentId="1799" CreationDate="2010-08-18T12:51:41.617" Score="2" ViewCount="" Body="&lt;p&gt;If temperature levels X1...X5 are specific degree values, I'm not sure how temperature and stimulus (hot/cold) can be completely crossed. I presume then that &quot;temperature&quot; consists of 5 ordinal categories, ranging from &quot;likely to cause minimal discomfort&quot; to &quot;likely to cause maximum discomfort permissible by my research ethics board&quot;, which then permits crossing with hot/cold direction. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If this is the case, you are correct that the 3 response variables might not be expected to be independent, so you might increase your power (relative to 3 seperate analyses) if you do a multivariate test. Frankly, I must admit that my first inclination would be to simply do three separate analyses, one for each response variable, but that's because I'm not very knowledgeable with regards to the mechanics of multivariate tests. That said, given the likely dependence between the 3 response variables, significant results across 3 separate anovas might be most reasonably considered as manifestations of this dependence rather than 3 truly independent sets of phenomena.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am pretty sure that you &lt;em&gt;shouldn't&lt;/em&gt; lump all 3 response variables into a single univariate analysis by adding &quot;response variable type&quot; as a predictor variable; this approach could run into trouble if the scales and variances of your response variables are very different. Presumably, a multivariate analysis has features that take such differences into account. (However, I wonder if a mixed effects analysis might also be able to handle such differences? I'm new to mixed effects, but suspect I it might...)&lt;/p&gt;&#xA;" OwnerUserId="364" LastActivityDate="2010-08-18T12:51:41.617" CommentCount="1" />
  <row Id="1826" PostTypeId="1" AcceptedAnswerId="1830" CreationDate="2010-08-18T13:11:19.190" Score="11" ViewCount="409" Body="&lt;p&gt;How would you describe &lt;a href=&quot;http://en.wikipedia.org/wiki/Cross-validation_%28statistics%29&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;cross-validation&lt;/strong&gt;&lt;/a&gt; to someone without a data analysis background?&lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-08-19T23:49:54.203" Title="Cross-Validation in plain english?" Tags="&lt;cross-validation&gt;" AnswerCount="5" CommentCount="1" FavoriteCount="2" />
  <row Id="1827" PostTypeId="2" ParentId="1822" CreationDate="2010-08-18T13:15:34.937" Score="4" ViewCount="" Body="&lt;p&gt;You can fit a hierarchical bayesian (HB) model without pooling and do an ordinary OLS by pooling the data and compare the models in terms of model fit, hold-out predictions etc to evaluate whether pooling outperforms the HB model. The model very briefly will look like so:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Model&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$y_i \sim N(X\ \beta_i,\sigma^2\ I)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\beta_i \sim N(\bar{\beta},\Sigma)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Priors&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\bar{\beta} \sim N(\bar{\bar{\beta}},\Sigma_0)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\Sigma \sim IW(R,d)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\sigma^2 \sim IG(sp,sc)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While I do not use R, I do know that there are packages that will do the above for you. Someone more knowledgeable about R can perhaps help you out. &lt;/p&gt;&#xA;" LastActivityDate="2010-08-18T13:25:06.397" CommentCount="1" />
  <row Id="1828" PostTypeId="2" ParentId="1826" CreationDate="2010-08-18T13:20:28.593" Score="10" ViewCount="" Body="&lt;p&gt;I think that this is best described with &lt;a href=&quot;http://genome.tugraz.at/proclassify/help/pages/XV.html&quot; rel=&quot;nofollow&quot;&gt;the following picture&lt;/a&gt; (in this case showing k-fold cross-validation):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/YWgro.gif&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cross-validation is a technique used to protect against overfitting in a predictive model, particularly in a case where the amount of data may be limited.  In cross-validation, you make a fixed number of folds (or partitions) of the data, run the analysis on each fold, and then average the overall error estimate.&lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-08-18T13:20:28.593" CommentCount="3" />
  <row Id="1829" PostTypeId="1" AcceptedAnswerId="1845" CreationDate="2010-08-18T13:30:31.750" Score="7" ViewCount="460" Body="&lt;p&gt;I usually hear about &quot;ordinary least squares&quot;.  Is that the most widely used algorithm used for linear regression?  Are there reasons to use a different one?&lt;/p&gt;&#xA;" OwnerUserId="988" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-08-18T15:25:08.287" LastActivityDate="2010-08-19T06:51:15.437" Title="What algorithm is used in linear regression?" Tags="&lt;regression&gt;&lt;least-squares&gt;" AnswerCount="5" FavoriteCount="2" />
  <row Id="1830" PostTypeId="2" ParentId="1826" CreationDate="2010-08-18T13:38:29.843" Score="10" ViewCount="" Body="&lt;p&gt;Consider the following situation: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;I want to catch the subway to go to my office. My plan is to take my car, park at the subway and then take the train to go to my office. My goal is to catch the train at 8.15 am every day so that I can reach my office on time. I need to decide the following: (a) the time at which I need to leave from my home and (b) the route I will take to drive to the station.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In the above example, I have two parameters (i.e., time of departure from home and route to take to the station) and I need to choose these parameters such that I reach the station by 8.15 am. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to solve the above problem I may try out different sets of 'parameters' (i.e., different combination of times of departure and route) on MWF to see which combination is the 'best' one. The idea is that once I have identified the best combination I can use it every day so that I achieve my objective.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Problem of Overfitting&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem with the above approach is that I may overfit which essentially means that the best combination I identify may in some sense may be unique to Mon, Wed and Fridays and that combination may not work for Tue and Thu. Overfitting may happen if in my search for the best combination of times and routes I exploit some aspect of the traffic situation on MWF which does not occur on Tue and Thu.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;One Solution to Overfitting: Cross-Validation&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cross-validation is one solution to overfitting. The idea is that once we have identified our best combination of parameters (in our case time and route) we test the performance of that set of parameters in a different context. Therefore, we may want to test on Tue and Thu as well to ensure that our choices work for those days as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Extending the analogy to statistics&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In statistics, we have a similar issue. We often use a limited set of data to estimate the unknown parameters we do not know. If we overfit then our parameter estimates will work very well for the existing data but not as well for when we use them in another context. Thus, cross-validation helps in avoiding the above issue of overfitting by proving us some reassurance that the parameter estimates are not unique to the data we used to estimate them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, cross validation is not perfect. Going back to our example of the subway, it can happen that even after cross-validation, our best choice of parameters may not work one month down the line because of various issues (e.g., construction, traffic volume changes over time etc).&lt;/p&gt;&#xA;" LastActivityDate="2010-08-18T13:38:29.843" CommentCount="5" />
  <row Id="1831" PostTypeId="2" ParentId="1822" CreationDate="2010-08-18T13:42:13.063" Score="3" ViewCount="" Body="&lt;p&gt;Srikant is right.  The book you want is &lt;a href=&quot;http://www.stat.columbia.edu/~gelman/arm/&quot; rel=&quot;nofollow&quot;&gt;&quot;Data Analysis Using Regression and Multilevel/Hierarchical Models&quot;&lt;/a&gt; by Gelman and Hill, all &lt;a href=&quot;http://www.stat.columbia.edu/~gelman/arm/software/&quot; rel=&quot;nofollow&quot;&gt;the R code from the book&lt;/a&gt;, and the associated &lt;a href=&quot;http://cran.r-project.org/web/packages/arm/&quot; rel=&quot;nofollow&quot;&gt;arm package in R&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-08-18T13:42:13.063" CommentCount="2" />
  <row Id="1832" PostTypeId="2" ParentId="1815" CreationDate="2010-08-18T13:50:28.787" Score="4" ViewCount="" Body="&lt;p&gt;for me, the best book around is by George Box:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0471718130&quot; rel=&quot;nofollow&quot;&gt;Statistics for Experimenters: Design, Innovation, and Discovery&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;of course the book by Maxwell and Delaney is also pretty good:&#xA;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0805837183&quot; rel=&quot;nofollow&quot;&gt;Designing Experiments and Analyzing Data: A Model Comparison Perspective, Second Edition&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I personally prefer the first, but they are both top quality. They are a little bit expensive, but you can definitely find a cheap earlier edition for sale.&lt;/p&gt;&#xA;" OwnerUserId="447" LastActivityDate="2010-08-18T13:50:28.787" CommunityOwnedDate="2010-08-18T13:50:28.787" CommentCount="1" />
  <row Id="1833" PostTypeId="2" ParentId="1829" CreationDate="2010-08-18T14:01:06.170" Score="5" ViewCount="" Body="&lt;p&gt;The wiki link: &lt;a href=&quot;http://en.wikipedia.org/wiki/Linear_regression#Estimation_methods&quot; rel=&quot;nofollow&quot;&gt;Estimation Methods for Linear Regression&lt;/a&gt; gives a fairly comprehensive list of estimation methods including OLS and the contexts in which alternative estimation methods are used.&lt;/p&gt;&#xA;" LastActivityDate="2010-08-18T14:01:06.170" />
  <row Id="1834" PostTypeId="2" ParentId="1829" CreationDate="2010-08-18T14:01:19.650" Score="2" ViewCount="" Body="&lt;p&gt;It is easy to get confused between definitions and terminology.  Both terms are used, sometimes interchangeably.  A quick lookup on Wikipedia should help:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Ordinary_least_squares&quot; rel=&quot;nofollow&quot;&gt;ordinary least squares&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Linear_regression_model&quot; rel=&quot;nofollow&quot;&gt;lnear regression&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Ordinary Least Squares (OLS) is a method used to fit linear regression models. Because of the demonstrable consistency and efficiency (under supplementary assumptions) of the OLS method, it is the dominant approach.  See the articles for further leads.&lt;/p&gt;&#xA;" OwnerUserId="334" LastActivityDate="2010-08-18T14:01:19.650" CommentCount="2" />
  <row Id="1835" PostTypeId="2" ParentId="1822" CreationDate="2010-08-18T14:13:25.737" Score="2" ViewCount="" Body="&lt;p&gt;The only further comment I would make is that the approach need not be Bayesian and the model need not be a mixed or random effects model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the simplest case if you had two series in x the mean model may be:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;y = b01 + I*b02 + b11*x + I.b12*x&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where I indicates a sample from the 2nd series. An omnibus F-test can be used to determine whether the additional parameters are required to maintain distinct series (Ho: b02 = b12 = 0). &#xA;&lt;a href=&quot;http://en.wikipedia.org/wiki/F_test&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/F_test&lt;/a&gt;&#xA;This can be extended to more series, but it soon becomes more efficient to use a mixed or random effects model. &lt;/p&gt;&#xA;" OwnerUserId="521" LastActivityDate="2010-08-18T14:13:25.737" />
  <row Id="1836" PostTypeId="2" ParentId="1807" CreationDate="2010-08-18T14:33:06.530" Score="6" ViewCount="" Body="&lt;p&gt;This may surprise many, but to solve this problem you don't necessarily need to estimate &lt;em&gt;s&lt;/em&gt;.  In fact, you don't need to know &lt;em&gt;anything&lt;/em&gt; about the spread of the data (although that would be helpful, of course).  For instance, Wall, Boen, and Tweedie in a 2001 article describe how to find a finite confidence interval for the mean of any unimodal distribution based on a &lt;em&gt;single&lt;/em&gt; draw.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the present case, we have some basis to view the sample mean of 112 as a draw from an approximately normal distribution (namely, the sampling distribution of the average of a simple random sample of 49 salaries).  We are implicitly assuming there are a fairly large number of factory workers and that their salary distribution is not so skewed or multimodal as to render the central limit theorem inoperable.  Then a conservative 90% CI for the mean extends upwards to&lt;/p&gt;&#xA;&#xA;&lt;p&gt;112 + 5.84|112|,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;clearly covering the true mean of 200.  (See Wall &lt;em&gt;et al&lt;/em&gt; formula 3.)  Given the limited information available and the assumptions made here, we therefore cannot conclude that 112 differs &quot;significantly&quot; from 200.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Reference: &lt;a href=&quot;http://www.jstor.org/stable/2685995&quot; rel=&quot;nofollow&quot;&gt;&quot;An Effective Confidence Interval for the Mean With Samples of Size One and Two.&quot;  The American Statistician, May 2001, Vol. 55, No. 2: pp. 102-105.&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-08-19T00:30:34.967" LastActivityDate="2010-08-19T00:30:34.967" CommentCount="5" />
  <row Id="1837" PostTypeId="2" ParentId="1321" CreationDate="2010-08-18T14:47:31.707" Score="2" ViewCount="" Body="&lt;p&gt;A method that can be very effective--one I have found extremely useful--is to sort the data by time and draw a connected X,Y scatterplot.  (That is, successive points are connected by line segments or a spline.)  This much is straightforward in almost any statistical plotting package.  If the result is too confusing, add graphical indications of directionality.  Depending on the density of the points and their pattern of temporal evolution, options include using arrows on the line segments or otherwise applying a graduated color or thickness to the segments to indicate their times.  You can even dispense with the connecting lines and just color or size the points to indicate time: that works better when there are many points on the plot.  In addition to displaying the bivariate relationship among the data in a conventional form, this method supplies a clear visual indication of temporally local correlations, changes that run counter to the prevailing correlation, etc.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-08-18T14:47:31.707" />
  <row Id="1838" PostTypeId="1" AcceptedAnswerId="1843" CreationDate="2010-08-18T14:55:55.133" Score="6" ViewCount="543" Body="&lt;p&gt;Greetings,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently I'm doing the following in R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(zoo)&#xA;data &amp;lt;- read.csv(file=&quot;summary.csv&quot;,sep=&quot;,&quot;,head=TRUE)&#xA;cum  = zoo(data$dcomp, as.Date(data$date))&#xA;data = zoo(data$compressed, as.Date(data$date))&#xA;data &amp;lt;- aggregate(data, identity, tail, 1)&#xA;cum  &amp;lt;- aggregate(cum, identity, sum, 1)&#xA;days = seq(start(data), end(data), &quot;day&quot;)&#xA;data2 = na.locf(merge(data, zoo(,days)))&#xA;&#xA;plot(data2,xlab='',ylab='compressed bytes',col=rgb(0.18,0.34,0.55))&#xA;lines(cum,type=&quot;h&quot;,col=rgb(0,0.5,0))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Snip of summary.csv:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;date,revision,file,lines,nclass,nattr,nrel,bytes,compressed,diff,dcomp&#xA;2007-07-25,16,model.xml,96,11,22,5,4035,991,0,0&#xA;2007-07-27,17,model.xml,115,16,26,6,4740,1056,53,777&#xA;2007-08-09,18,model.xml,106,16,26,7,4966,1136,47,761&#xA;2007-08-10,19,model.xml,106,16,26,7,4968,1150,4,202&#xA;2007-09-06,81,model.xml,111,16,26,7,5110,1167,13,258&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The last two lines plot the information I need, and the result resembles the following:&#xA;&lt;img src=&quot;http://i.imgur.com/ech9l.png&quot; alt=&quot;alt text&quot;&gt;&#xA;Blue line is the entropy in bytes of the artifact I'm interested. Green lines represent the entropy of the changes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, in this graph, it works well because there isn't a huge difference in scales. But I have other graphs where the green lines become so small one cannot see.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The solution I was looking for, involved two things:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;To move the green vertical lines to a second graph, just below the first one, with its own y axis, but shared x axis.&lt;/li&gt;&#xA;&lt;li&gt;To provide it a logarithmic scale, since I'm more interested in the &quot;magnitude&quot;, than in the specific values.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S. If someone can also tell me how could I put &quot;minor ticks&quot; in the x scale referring to the months, I appreciate :-) If these are too much questions for a single post, I can divide them further.&lt;/p&gt;&#xA;" OwnerUserId="990" LastEditorUserId="990" LastEditorDisplayName="" LastEditDate="2010-08-18T15:39:27.513" LastActivityDate="2010-08-19T09:59:28.810" Title="How do I vertically stack two graphs with the same x scale, but a different y scale in R?" Tags="&lt;r&gt;&lt;time-series&gt;&lt;data-visualization&gt;&lt;entropy&gt;" AnswerCount="2" CommentCount="0" />
  <row Id="1839" PostTypeId="2" ParentId="1829" CreationDate="2010-08-18T14:57:00.733" Score="2" ViewCount="" Body="&lt;p&gt;I tend to think of 'least squares' as a criterion for defining the best fitting regression line (i.e., that which makes the sum of 'squared' residuals 'least') and the 'algorithm' in this context as the set of steps used to determine the regression coefficients that satisfy that criterion. This distinction suggests that it is possible to have  different algorithms that would satisfy the same criterion. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd be curious to know whether others make this distinction and what terminology they use.&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-08-18T14:57:00.733" CommentCount="1" />
  <row Id="1840" PostTypeId="2" ParentId="1826" CreationDate="2010-08-18T15:14:02.023" Score="5" ViewCount="" Body="&lt;p&gt;Let's say you investigate some process; you've gathered some data describing it and you have build a model (either statistical or ML, doesn't matter). But now, how to judge if it is ok? Probably it fits suspiciously good to the data it was build on, so no-one will believe that your model is so splendid that you think.&lt;br&gt;&#xA;First idea is to separate a subset of your data and use it to test the model build by your method on the rest of data. Now the result is definitely overfitting-free, nevertheless (especially for small sets) you could have been (un)lucky and draw (less)more simpler cases to test, making it (harder)easier to predict... Also your accuracy/error/goodness estimate is useless for model comparison/optimization, since you probably know nothing about its distribution.&lt;br&gt;&#xA;When in doubt, use brute force, so just replicate the above process, gather few estimates of accuracy/error/goodness and average them -- and so you obtain cross validation. Among better estimate you will also get a histogram, so you will be able to approximate distribution or perform some non-parametric tests.&lt;br&gt;&#xA;And this is it; the details of test-train splitting are the reason for different CV types, still except of rare cases and small strength differences they are rather equivalent. Indeed it is a huge advantage, because it makes it a bulletproof-fair method; it is very hard to cheat it.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-08-18T15:14:02.023" CommentCount="2" />
  <row Id="1841" PostTypeId="1" AcceptedAnswerId="2057" CreationDate="2010-08-18T15:19:44.807" Score="6" ViewCount="280" Body="&lt;p&gt;Disclaimer: I'm a software engineer, not a statistician, so please forgive any blunt error :-)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a set of time-series &quot;curves&quot;, each measuring the entropy of a given artifact. Now, I'm standing over the following premises (please criticize them as you see fit):&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;In order to approximate the upper bounds of the Kolmogorov complexity $K(s)$, of a string $s$, one can simply compress the string $s$ with some method, implement the corresponding decompressor in the chosen language, concatenate the decompressor to the compressed string, and measure the resulting string's length.&lt;/li&gt;&#xA;&lt;li&gt;For this purpose, I've used the &lt;em&gt;bzip2&lt;/em&gt; application, setting its compression level to the supported maximum (-9).&lt;/li&gt;&#xA;&lt;li&gt;If one is only interested in a time-series analysis of a set of evolving strings, calculating the compressed &lt;em&gt;deltas&lt;/em&gt; is enough to present a relative measure of entropy between any two strings (at least that's my interpretation after reading Cilibrasi05).&lt;/li&gt;&#xA;&lt;li&gt;For that, I used the &lt;em&gt;diff&lt;/em&gt; unix tool, with the (--minimal) parameter, again followed by a &lt;em&gt;bzip2&lt;/em&gt; compression, with the aforementioned settings.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I'm doing this to analyze the evolution of the entropy in a software artifact (code, model, whatever). I'm &lt;em&gt;not&lt;/em&gt; worried with the absolute values, but with the &lt;em&gt;relative&lt;/em&gt; increase (or decrease) in entropy. Now here comes the problem:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;I've done this for a set of 6 artifacts, which ought to belong to the same population, but I don't know how to provide statistical evidence of that (the corresponding of doing a two-tailed t-test of two samples).&lt;/li&gt;&#xA;&lt;li&gt;One of the artifacts evolution &lt;em&gt;should&lt;/em&gt; be different from all the others. We're talking something like an exponential v.s. sub-linear growth. How do I provide statistical evidence of that?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Again, the disclaimer of being a software engineer. Although I would appreciate every academic reference (papers, books, etc.) you could handle, I'm looking for something pragmatic that I can use in the next few days, like a script in R, or something in SPSS.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S. I'm sorry for asking for a &lt;em&gt;recipe&lt;/em&gt;, instead of a theoretical explanation.&lt;/p&gt;&#xA;" OwnerUserId="990" LastEditorUserId="990" LastEditorDisplayName="" LastEditDate="2010-08-18T16:00:37.460" LastActivityDate="2010-08-25T13:52:20.933" Title="Hypothesis testing that one time-series of a measure of entropy doesn't belong to a population" Tags="&lt;r&gt;&lt;time-series&gt;&lt;hypothesis-testing&gt;&lt;spss&gt;&lt;entropy&gt;" AnswerCount="2" />
  <row Id="1842" PostTypeId="2" ParentId="1838" CreationDate="2010-08-18T15:25:59.757" Score="9" ViewCount="" Body="&lt;p&gt;I think you can get what you want using &lt;code&gt;ggplot2&lt;/code&gt;. Using the code below, I can produce:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/qNcAk.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Obviously things like line colours can be changed to what ever you want. On the x-axis I specified major lines on years and minor lines on months.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(ggplot2)&#xA;t = as.Date(0:1000, origin=&quot;2008-01-01&quot;)  &#xA;y1 = rexp(1001)&#xA;y2 = cumsum(y1)&#xA;df = data.frame(t=t, values=c(y2,y1), type=rep(c(&quot;Bytes&quot;, &quot;Changes&quot;), each=1001))&#xA;&#xA;g = ggplot(data=df, aes(x=t, y=values)) +&#xA;  geom_line() +&#xA;  facet_grid(type ~ ., scales=&quot;free&quot;) +&#xA;  scale_y_continuous(trans=&quot;log10&quot;) +&#xA;  scale_x_date(major=&quot;years&quot;, minor=&quot;months&quot;) +&#xA;  ylab(&quot;Log values&quot;)&#xA;g&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="8" LastActivityDate="2010-08-18T15:25:59.757" CommentCount="4" />
  <row Id="1843" PostTypeId="2" ParentId="1838" CreationDate="2010-08-18T15:43:03.723" Score="8" ViewCount="" Body="&lt;p&gt;You can use &lt;code&gt;par(new=T)&lt;/code&gt; to plot into the same graph using two different y-axes! Thsi should also solve your problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Next you will find a simple example that plots two random normal variables, one on mean 0 the other one on mean 100 (both &lt;em&gt;sd&lt;/em&gt; s = 1) in the same plot. The first one in red on the left y-axis, the second one in blue on the right y-axis. Then, axis labels are added.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here you go:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x &amp;lt;- 1:10&#xA;y1 &amp;lt;- rnorm(10)&#xA;y2 &amp;lt;- rnorm(10)+100&#xA;&#xA;plot(x,y1,pch=0,type=&quot;b&quot;,col=&quot;red&quot;,yaxt=&quot;n&quot;,ylim=c(-8,2))&#xA;par(new=T)&#xA;plot(x,y2,pch=1,type=&quot;b&quot;,col=&quot;blue&quot;,yaxt=&quot;n&quot;,ylim=c(98,105))&#xA;&#xA;axis(side=2)&#xA;axis(side=4)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;looks like this then (remember red on left axis, blue on right axis): &lt;img src=&quot;http://i.imgur.com/311Vo.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE:&lt;/strong&gt;&lt;br&gt;&#xA;Based on comments I produced an updated version of my graph. Now I dig a little deeper into base graph functionality using &lt;code&gt;par(mar=c(a,b,c,d))&lt;/code&gt; to create a bigger margin around the graph (needed for right axis label), &lt;code&gt;mtext&lt;/code&gt; to show the axis labels and and advanced use of the &lt;code&gt;axis&lt;/code&gt; function:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x &amp;lt;- 1:100&#xA;y1 &amp;lt;- rnorm(100)&#xA;y2 &amp;lt;- rnorm(100)+100&#xA;&#xA;par(mar=c(5,5,5,5))&#xA;&#xA;plot(x,y1,pch=0,type=&quot;b&quot;,col=&quot;red&quot;,yaxt=&quot;n&quot;,ylim=c(-8,2),ylab=&quot;&quot;)&#xA;axis(side=2, at=c(-2,0,2))&#xA;mtext(&quot;red line&quot;, side = 2, line=2.5, at=0)&#xA;&#xA;par(new=T)&#xA;plot(x,y2,pch=1,type=&quot;b&quot;,col=&quot;blue&quot;,yaxt=&quot;n&quot;,ylim=c(98,108), ylab=&quot;&quot;)&#xA;axis(side=4, at=c(98,100,102), labels=c(&quot;98%&quot;,&quot;100%&quot;,&quot;102%&quot;))&#xA;mtext(&quot;blue line&quot;, side=4, line=2.5, at=100)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/14zqJ.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you see it is pretty straight forward. You can define the position of your data with &lt;code&gt;ylim&lt;/code&gt; in the &lt;code&gt;plot&lt;/code&gt; function, then use &lt;code&gt;at&lt;/code&gt; in the &lt;code&gt;axis&lt;/code&gt; function to select which axis ticks you wanna see. Furthermore, you can even provide the labels for the axis ticks (pretty useful for nominal x-axis) via &lt;code&gt;labels&lt;/code&gt; in the &lt;code&gt;axis&lt;/code&gt; function (done here on the right axis). To add axis labels, use &lt;code&gt;mtext&lt;/code&gt; with &lt;code&gt;at&lt;/code&gt; for vertical positioning (&lt;code&gt;line&lt;/code&gt; for horizontal positioning).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Make sure to check &lt;code&gt;?plot&lt;/code&gt;, &lt;code&gt;?par&lt;/code&gt;, &lt;code&gt;?axis&lt;/code&gt;, and &lt;code&gt;?mtext&lt;/code&gt; for further info.&lt;br&gt;&#xA;Great web resources are: &lt;a href=&quot;http://www.statmethods.net/index.html&quot; rel=&quot;nofollow&quot;&gt;Quick-R&lt;/a&gt; for Graphs: &lt;a href=&quot;http://www.statmethods.net/advgraphs/parameters.html&quot; rel=&quot;nofollow&quot;&gt;1&lt;/a&gt;, &lt;a href=&quot;http://www.statmethods.net/graphs/index.html&quot; rel=&quot;nofollow&quot;&gt;2&lt;/a&gt;, and &lt;a href=&quot;http://www.statmethods.net/advgraphs/index.html&quot; rel=&quot;nofollow&quot;&gt;3&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="442" LastEditorUserId="442" LastEditorDisplayName="" LastEditDate="2010-08-19T09:59:28.810" LastActivityDate="2010-08-19T09:59:28.810" CommentCount="5" />
  <row Id="1844" PostTypeId="1" AcceptedAnswerId="2053" CreationDate="2010-08-18T15:47:15.627" Score="7" ViewCount="465" Body="&lt;p&gt;One way to summarize the comparison of two survival curves is to compute the hazard ratio (HR). There are (at least) two methods to compute this value. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Logrank method. As part of the Kaplan-Meier calculations, compute the number of observed events (deaths, usually) in each group (Oa, and Ob), and the number of expected events assuming a null hypothesis of no difference in survival (Ea and Eb). The hazard ratio then is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;HR= (Oa/Ea)/(Ob/Eb)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Mantel-Haenszel method. First compute V, which is the sum of the hypergeometric variances at each time point. Then compute the hazard ratio as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;HR= exp[(Oa-Ea)/V]&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I got both these equations from chapter 3 of  Machin, Cheung and Parmar, &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0470870400&quot; rel=&quot;nofollow&quot;&gt;Survival Analysis&lt;/a&gt;. That book states that the two methods usually give very similar methods, and indeed that is the case with the example in the book. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Someone sent me an example where the two methods differ by a factor of three. In this particular example, it is obvious that the logrank estimate is sensible, and the Mantel-Haenszel estimate is far off. My question is if anyone has any general advice for when it is best to choose the logrank estimate of the hazard ratio, and when it is best to choose the Mantel-Haenszel estimate? Does it have to do with sample size? Number of ties? Ratio of sample sizes?&lt;/p&gt;&#xA;" OwnerUserId="25" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-16T06:46:18.790" LastActivityDate="2010-09-16T06:46:18.790" Title="What are the pros and cons of using the logrank vs. the Mantel-Haenszel method for computing the Hazard Ratio in survival analysis?" Tags="&lt;survival&gt;&lt;hazard&gt;" AnswerCount="4" CommentCount="7" FavoriteCount="3" />
  <row Id="1845" PostTypeId="2" ParentId="1829" CreationDate="2010-08-18T16:19:28.490" Score="6" ViewCount="" Body="&lt;p&gt;Regarding the question in the title, about what is the algorithm that is used:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a linear algebra perspective, the linear regression algorithm is the way to solve a linear system $\mathbf{A}x=b$ with more equations than unknowns. In most of the cases there is no solution to this problem. And this is because the vector $b$ doesn't belong to the column space of $\mathbf{A}$, $C(\mathbf{A})$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;code&gt;best straight line&lt;/code&gt; is the one that makes the overall error $e=\mathbf{A}x-b$ as small as it takes. And is convenient to think as small to be the squared length, $\lVert e \rVert^2$, because it's non negative, and it equals 0 only when $b\in C(\mathbf{A})$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Projecting (orthogonally) the vector $b$ to the nearest point in the column space of $\mathbf{A}$ gives the vector $b^*$ that solves the system (it's components lie on the best straight line) with the minimum error. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\mathbf{A}^T\mathbf{A}\hat{x}=\mathbf{A}^Tb \Rightarrow \hat{x}=(\mathbf{A}^T\mathbf{A})^{-1}\mathbf{A}^Tb$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and the projected vector $b^*$ is given by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$b^*=\mathbf{A}\hat{x}=\mathbf{A}(\mathbf{A}^T\mathbf{A})^{-1}\mathbf{A}^Tb$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Perhaps the least squares method is not &lt;em&gt;exclusively&lt;/em&gt; used because that &lt;code&gt;squaring&lt;/code&gt;  overcompensates for outliers. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let me give a simple example in R, that solves the regression problem using this algorithm:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;reg.data &amp;lt;- read.table(textConnection(&quot;&#xA;   b      x&#xA;  12      0&#xA;  10      1&#xA;   8      2&#xA;  11      3&#xA;   6      4&#xA;   7      5&#xA;   2      6&#xA;   3      7&#xA;   3      8 &quot;), header = T)&#xA;&#xA;attach(reg.data)&#xA;&#xA;A &amp;lt;- model.matrix(b~x)&#xA;&#xA;# intercept and slope&#xA;inv(t(A) %*% A) %*% t(A) %*% b&#xA;&#xA;# fitted values - the projected vector b in the C(A)&#xA;A %*% inv(t(A) %*%A ) %*% t(A) %*% b&#xA;&#xA;# The projection is easier if the orthogonal matrix Q is used, &#xA;# because t(Q)%*%Q = I&#xA;Q &amp;lt;- qr.Q(qr(A))&#xA;R &amp;lt;- qr.R(qr(A))&#xA;&#xA;# intercept and slope &#xA;best.line &amp;lt;- inv(R) %*% t(Q) %*% b&#xA;&#xA;# fitted values &#xA;Q %*% t(Q) %*% b&#xA;&#xA;plot(x,b,pch=16)&#xA;abline(best.line[1],best.line[2])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="339" LastActivityDate="2010-08-18T16:19:28.490" />
  <row Id="1846" PostTypeId="2" ParentId="1376" CreationDate="2010-08-18T16:28:57.907" Score="1" ViewCount="" Body="&lt;p&gt;Some robust alernatives are discussed in &lt;a href=&quot;http://dx.doi.org/10.1080/02664760050076434&quot; rel=&quot;nofollow&quot;&gt;A class of robust stepwise alternativese to Hotelling's T 2 tests&lt;/a&gt;, which deals with trimmed means of the marginals of residuals produced by stepwise regression, and in &lt;a href=&quot;http://dx.doi.org/10.1080/02664760902810813&quot; rel=&quot;nofollow&quot;&gt;A comparison of robust alternatives to Hoteslling's T^2 control chart&lt;/a&gt;, which outlines some robust alternatives based on MVE, MCD, RMCD and trimmed means.&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-08-18T16:28:57.907" />
  <row Id="1847" PostTypeId="2" ParentId="1787" CreationDate="2010-08-18T16:38:48.443" Score="2" ViewCount="" Body="&lt;p&gt;Have you thought about using simulate in the arm package?  Gelman &amp;amp; Hill have some nice chapters on this in their book.&lt;/p&gt;&#xA;" OwnerUserId="101" LastActivityDate="2010-08-18T16:38:48.443" CommentCount="1" />
  <row Id="1848" PostTypeId="1" AcceptedAnswerId="1849" CreationDate="2010-08-18T16:43:11.870" Score="4" ViewCount="304" Body="&lt;p&gt;I'm implementing a rating system to be used on my website, and I think the Bayesian average is the best way to go about it. Every item will be rated in six different categories by the users. I don't want items with only one high rating to shoot to the top though, which is why I want to implement a Bayesian system.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the formula:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Bayesian Rating = ( (avg_num_votes * avg_rating) + (this_num_votes * this_rating) ) / (avg_num_votes + this_num_votes)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Because the items will be rated in 6 different categories, should I use the average of the sums of those categories as &quot;this_rating&quot; for the Bayesian system? For instance, take one item with two ratings (scale of 0-5):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Rating 1:&#xA;  Category A: 3&#xA;  Category B: 1&#xA;  Category C: 2&#xA;  Category D: 4&#xA;  Category E: 5&#xA;  Category F: 3&#xA;  Sum: 18&#xA;&#xA;Rating 2:&#xA;  Category A: 2&#xA;  Category B: 3&#xA;  Category C: 3&#xA;  Category D: 5&#xA;  Category E: 0&#xA;  Category F: 1&#xA;  Sum: 14&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Should &quot;this_rating&quot; be simply the average of the sums listed above? Is my thinking correct, or should a Bayesian system be implemented for each category as well (or is that overthinking it)?&lt;/p&gt;&#xA;" OwnerUserId="991" LastActivityDate="2010-08-18T16:58:10.540" Title="Bayesian rating system with multiple categories for each rating" Tags="&lt;bayesian&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="3" />
  <row Id="1849" PostTypeId="2" ParentId="1848" CreationDate="2010-08-18T16:58:10.540" Score="4" ViewCount="" Body="&lt;p&gt;It depends on whether you want to wind up only with a cumulative rating of each object, or category-specific rating. Having a separate system in each category sounds more realistic, but your particular context might suggest otherwise. You could even do both a category-specific and overall rating!&lt;/p&gt;&#xA;" OwnerUserId="279" LastActivityDate="2010-08-18T16:58:10.540" CommentCount="2" />
  <row Id="1850" PostTypeId="1" AcceptedAnswerId="1930" CreationDate="2010-08-18T17:35:17.280" Score="4" ViewCount="1350" Body="&lt;p&gt;For an effect size analysis, I am noticing that there are differences between Cohen's d, Hedges's g and Hedges' g*. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Are these three metrics normally very similar?  &lt;/li&gt;&#xA;&lt;li&gt;What would be a case where they would produce different results? &lt;/li&gt;&#xA;&lt;li&gt;Also is it a matter of preference which I use or report with?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="559" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-08-19T08:02:01.917" LastActivityDate="2010-08-19T20:52:10.220" Title="Difference between Cohen's d and Hedges' g for effect size metrics" Tags="&lt;effect-size&gt;&lt;cohens-d&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="1851" PostTypeId="1" CreationDate="2010-08-18T17:59:23.647" Score="1" ViewCount="68" Body="&lt;p&gt;I have a large set of records with each record containing $N$ variables $v_i$ (say $N=20$).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can transform these variables $v_i$ to be in the range [0,1] and am able to use a simple loss function $L = \sum_i v_i^2$. After that I know that there are two large clusters of points with small $L$ and large $L$. There are also few points in between these clusters. I can successfully assign records to these clusters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But what &lt;code&gt;R&lt;/code&gt; functionality would I use to get a parametric representation (in the variables $v_i$) of the surface of the &quot;large $L$ cluster&quot;, preferably as a function of $L$?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(I know I could train a neural network on my data to give me a similar answer, but I would like to stay away from that for this application.)&lt;/p&gt;&#xA;" OwnerUserId="56" LastActivityDate="2010-08-18T17:59:23.647" Title="Find parametric representation of cluster surface" Tags="&lt;r&gt;&lt;clustering&gt;" CommentCount="3" />
  <row Id="1852" PostTypeId="2" ParentId="1807" CreationDate="2010-08-18T18:21:12.870" Score="1" ViewCount="" Body="&lt;p&gt;Suppose there are 999 workers at ACME north factory each making a wage of 112, and 1 CEO making 88112. The population mean salary is $\mu = 0.999 * 112 + 0.001 * 88112 = 200.$ The probability of drawing the CEO from a sample of 49 people at the factory is $49 / 1000 &amp;lt; 0.05$ (this is from the hypergeometric distribution), thus with 95% confidence, your &lt;strike&gt;population&lt;/strike&gt; sample mean will be 112. In fact, by adjusting the ratio of workers/CEOs, and the salary of the CEO, we can make it arbitrarily unlikely that a sample of 49 employees will draw a CEO, while fixing the population mean at 200, and the sample mean at 112. Thus, without making &lt;em&gt;some&lt;/em&gt; assumptions about the underlying distribution, you cannot draw any inference about the population mean.&lt;/p&gt;&#xA;" OwnerUserId="795" LastEditorUserId="795" LastEditorDisplayName="" LastEditDate="2010-08-19T17:09:15.637" LastActivityDate="2010-08-19T17:09:15.637" CommentCount="3" />
  <row Id="1853" PostTypeId="1" AcceptedAnswerId="1953" CreationDate="2010-08-18T18:49:48.227" Score="5" ViewCount="146" Body="&lt;p&gt;What tests are available for testing two independent samples for the null hypothesis that they come from populations with the same skew? There is a classical 1-sample test for whether the skew equals a fixed number (the test involves the 6th sample moment!); is there a straightforward translation to a 2-sample test? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there techniques which don't involve very high moments of the data? (I am anticipating an answer of the form 'bootstrap it': are bootstrap techniques known to be appropriate for this problem?)&lt;/p&gt;&#xA;" OwnerUserId="795" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-30T21:15:49.263" LastActivityDate="2010-09-30T21:15:49.263" Title="Testing two independent samples for null of same skew?" Tags="&lt;distributions&gt;&lt;hypothesis-testing&gt;&lt;bootstrap&gt;&lt;test&gt;&lt;moments&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="1854" PostTypeId="2" ParentId="1826" CreationDate="2010-08-18T19:09:37.983" Score="4" ViewCount="" Body="&lt;p&gt;&quot;Avoid learning your training data by heart by making sure the trained model performs well on independent data.&quot;&lt;/p&gt;&#xA;" OwnerUserId="961" LastActivityDate="2010-08-18T19:09:37.983" />
  <row Id="1855" PostTypeId="2" ParentId="1797" CreationDate="2010-08-18T19:16:05.037" Score="2" ViewCount="" Body="&lt;p&gt;It has the same meaning as any other confidence interval: under the assumption that the model is correct, if the experiment and procedure is repeated over and over, 95% of the time the true value of the quantity of interest will lie within the interval. In this case, the quantity of interest is the expected value of the response variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is probably easiest to explain this in the context of a linear model (mixed models are just an extension of this, so the same ideas apply):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The usual assumption is that:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$y_i = X_{i1} \beta_1 + X_{i2} \beta_2 + \ldots X_{ip} \beta_p + \epsilon $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $y_i$ is the response, $X_{ij}$'s are the covariates, $\beta_j$'s are the parameters, and $\epsilon$ is the error term which has mean zero. The quantity of interest is then:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$E[y_i] = X_{i1} \beta_1 + X_{i2} \beta_2 + \ldots X_{ip} \beta_p $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;which is a linear function of the (unknown) parameters, since the covariates are known (and fixed). Since we know the sampling distribution of the parameter vector, we can easily calculate the sampling distribution (and hence the confidence interval) of this quantity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So why would you want to know it? I guess if you're doing out-of-sample prediction, it could tell you how good your forecast is expected to be (though you'd need to take into account model uncertainty).&lt;/p&gt;&#xA;" OwnerUserId="495" LastActivityDate="2010-08-18T19:16:05.037" CommentCount="2" />
  <row Id="1856" PostTypeId="1" AcceptedAnswerId="2888" CreationDate="2010-08-18T20:36:59.617" Score="6" ViewCount="369" Body="&lt;p&gt;What do you think about applying machine learning techniques, like Random Forests or penalized regression (with L1 or L2 penalty, or a combination thereof) in small sample clinical studies when the objective is to isolate interesting predictors in a classification context? It is not a question about model selection, nor am I asking about how to find optimal estimates of variable effect/importance. I don't plan to do strong inference but just to use multivariate modeling, hence avoiding testing each predictor against the outcome of interest one at a time, and taking their interrelationships into account.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was just wondering if such an approach was already applied in this particular extreme case, say 20-30 subjects with data on 10-15 categorical or continuous variables. It is not exactly the $n\ll p$ case and I think the problem here is related to the number of classes we try to explain (which are often not well balanced), and the (very) small n. I am aware of the huge literature on this topic in the context of bioinformatics, but I didn't find any reference related to biomedical studies with psychometrically measured phenotypes (e.g. throughout neuropsychological questionnaires).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any hint or pointers to relevant papers?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am open to any other solutions for analyzing this kind of data, e.g. C4.5 algorithm or its derivatives, association rules methods, and any data mining techniques for supervised or semi-supervised classification.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-19T10:57:54.697" LastActivityDate="2010-10-01T15:56:38.640" Title="Application of machine learning techniques in small sample clinical studies" Tags="&lt;machine-learning&gt;&lt;feature-selection&gt;" AnswerCount="4" CommentCount="2" FavoriteCount="2" />
  <row Id="1857" PostTypeId="2" ParentId="1856" CreationDate="2010-08-18T20:51:32.280" Score="3" ViewCount="" Body="&lt;p&gt;One common rule of thumb is to have at least 10 times the number of training data instances (not to speak of any test/validation data, etc.) as there are adjustable parameters in the classifier.  Keep in mind that you have a problem wherein you need to not only have &lt;em&gt;adequate&lt;/em&gt; data but also &lt;em&gt;representative&lt;/em&gt; data.  In the end, there is no systematic rule because there are so many variables when making this decision.  As Hastie, Tibshirani, &#xA;and Friedman say in &lt;a href=&quot;http://www-stat.stanford.edu/~tibs/ElemStatLearn/&quot; rel=&quot;nofollow&quot;&gt;The Elements of Statistical Learning&lt;/a&gt; (see Chapter 7):&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;it is too difficult to give a general&#xA;  rule on how much training data is&#xA;  enough; among other things, this&#xA;  depends on the signal-to-noise ratio&#xA;  of the underlying function, and the&#xA;  complexity of the models being fit to&#xA;  the data.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;If you are new to this field, I recommend reading this short &lt;a href=&quot;http://users.rowan.edu/~polikar/RESEARCH/PUBLICATIONS/wiley06.pdf&quot; rel=&quot;nofollow&quot;&gt;&quot;Pattern Recognition&quot;&lt;/a&gt; paper from the Encyclopedia of Biomedical Engineering which gives a brief summary of some of the data issues.&lt;/p&gt;&#xA;" OwnerUserId="5" LastEditorUserId="5" LastEditorDisplayName="" LastEditDate="2010-08-18T21:23:54.503" LastActivityDate="2010-08-18T21:23:54.503" CommentCount="2" />
  <row Id="1858" PostTypeId="2" ParentId="1856" CreationDate="2010-08-18T21:28:17.517" Score="2" ViewCount="" Body="&lt;p&gt;I can assure you that RF would work in that case and its importance measure would be pretty insightful (because there will be no large tail of misleading unimportant attributes like in standard (n &amp;lt;&amp;lt; p)s). I can't recall now any paper dealing with similar problem, but I'll look for it.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-08-18T21:28:17.517" CommentCount="5" />
  <row Id="1859" PostTypeId="1" CreationDate="2010-08-18T22:07:56.910" Score="0" ViewCount="143" Body="&lt;p&gt;Hello,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to reduce a contingency table. The problem is that I only know that my variables are a subset of 4 bytes numbers. However, I don't know in which range the numbers occur. How can I define categories ? Can I add collocated rows/columns to reduce contingency table? Thans in advance.&lt;/p&gt;&#xA;" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-18T22:01:14.847" LastActivityDate="2010-11-04T07:38:07.023" ClosedDate="2011-04-11T01:45:27.487" Title="Contingency table" Tags="&lt;contingency-tables&gt;" CommentCount="1" />
  <row Id="1860" PostTypeId="1" AcceptedAnswerId="1861" CreationDate="2010-08-18T22:58:05.027" Score="4" ViewCount="86" Body="&lt;p&gt;I am studying a population of individuals who all begin with a measureable score of interest (ranging from -2 to 2) [call it &quot;old&quot;], then they all undergo a change to a new score (also ranging from -2 to 2) [&quot;new&quot;]. Thus all the variation is in the change (which can be positive or negative), and there are also a variety of predictors that help to explain variation in the amount of change.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My initial model is simply:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;change = a + bx + e&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where x is my vector of predictors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But now I'm concerned that some of these predictors could be correlated with the baseline (old) score. Is this, then, a better specification?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;change = a + bx + old +  e&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Or perhaps&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;new = a + bx + old + e&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="78" LastActivityDate="2010-08-19T12:59:30.587" Title="Regression specification choices" Tags="&lt;regression&gt;" AnswerCount="2" />
  <row Id="1861" PostTypeId="2" ParentId="1860" CreationDate="2010-08-18T23:17:46.677" Score="2" ViewCount="" Body="&lt;p&gt;You are right, version 1 is not acceptable. The second or third options (as long as &lt;code&gt;old&lt;/code&gt; has a coefficient that will be estimated) are both OK, and in fact equivalent with respect to estimates for &lt;code&gt;a&lt;/code&gt; and &lt;code&gt;b&lt;/code&gt;. This can be seen if you replace &lt;code&gt;change&lt;/code&gt; with &lt;code&gt;new-old&lt;/code&gt; in the second equation, and solve it for &lt;code&gt;new&lt;/code&gt;. All that happens is that the coefficient of &lt;code&gt;old&lt;/code&gt; is increased by 1 as compared to the third equation. Other statistics such as R^2 will change, of course, as they are decomposing a different variability.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note, however, that you have a different problem as well. If your scores are restricted to a -2 to 2 range, somebody with &lt;code&gt;old=-2&lt;/code&gt; cannot possibly get worse, and similarly for &lt;code&gt;old=2&lt;/code&gt; you can't get any better. Such a range restriction is usually not modeled well by a linear regression.&lt;/p&gt;&#xA;" OwnerUserId="279" LastEditorUserId="279" LastEditorDisplayName="" LastEditDate="2010-08-19T12:59:30.587" LastActivityDate="2010-08-19T12:59:30.587" />
  <row Id="1862" PostTypeId="1" AcceptedAnswerId="1869" CreationDate="2010-08-19T00:31:44.013" Score="5" ViewCount="298" Body="&lt;p&gt;I've been struggling with the following problem with hopefully is an easy one for statisticians (I'm a programmer with some exposure to statistics).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to summarize the responses to a survey (for management). The survey has 100+ questions, grouped in different areas (with about 5 to 10 questions per area). All answers are categorical (on an ordinal scale, they are like &quot;not at all&quot;, &quot;rarely&quot; ... &quot;daily or more frequently&quot;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Management would like to get a summary for each area and this is my problem: &lt;strong&gt;how to aggregate categorical answers within the related question?&lt;/strong&gt;. The questions are too many to make a graph or even a lattice plot for each area. I favor a visual approach if possible, compared to, say, tables with numbers (alas, they won't read them).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The only thing I can come up with is to count the number of answers in each area, then plot the histogram.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any thing else available for categorical data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I use R, but not sure if it's relevant, I feel this is more of a general statistics question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for any suggestions.&lt;/p&gt;&#xA;" OwnerUserId="840" LastActivityDate="2010-08-19T17:28:18.883" Title="How to summarize categorical data?" Tags="&lt;data-transformation&gt;&lt;categorical-data&gt;" AnswerCount="3" CommentCount="2" />
  <row Id="1863" PostTypeId="1" AcceptedAnswerId="1871" CreationDate="2010-08-19T00:50:20.120" Score="4" ViewCount="292" Body="&lt;p&gt;I ran a within subjects repeated measures experiment, where the independent variable had 3 levels. The dependent variable is a measure of correctness and is recorded as either correct / incorrect. Time taken to provide an answer was also recorded.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A within subjects repeated measures ANOVA is used to establish whether there is significant differences in correctness (DV) between the 3 levels of the IV, there is significant. Now, I'd like to analyze whether there is significant differences in the time taken to provide the answers when the answers are 1) correct, and 2) incorrect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My problem is: Across the levels there are different numbers of correct / incorrect answers, e.g. level 1 has 67 correct answers, level 2 has 30, level 3 has 25. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I compare the time take taken for all correct answers across the 3 levels? I think this means its unbalanced? Can I do 3 one way ANOVAS to do a pairwise comparison, while adjusting p downwards to account for each comparison?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="993" LastActivityDate="2010-08-21T07:21:06.203" Title="Might be an unbalanced within subjects repeated measures?" Tags="&lt;variance&gt;&lt;repeated-measures&gt;&lt;unbalanced-classes&gt;&lt;within-subjects&gt;" AnswerCount="3" FavoriteCount="2" />
  <row Id="1864" PostTypeId="2" ParentId="1856" CreationDate="2010-08-19T00:59:56.543" Score="2" ViewCount="" Body="&lt;p&gt;I would have very little confidence in the generalisability of results of an exploratory analysis with 15 predictors and a sample size of 20.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The confidence intervals of parameter estimates would be large. E.g., the 95% confidence interval on r = .30 with n = 20 is -0.17 to 0.66 .&lt;/li&gt;&#xA;&lt;li&gt;Issues tend to be compounded when you have multiple predictors used in an exploratory and data driven way.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In such circumstances, my advice would generally be to limit analyses to bivariate relationships.&#xA;If you take a bayesian perspective, then I'd say that your prior expectations are equally if not more important than the data.&lt;/p&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-08-19T05:53:41.183" LastActivityDate="2010-08-19T05:53:41.183" />
  <row Id="1865" PostTypeId="1" CreationDate="2010-08-19T01:15:27.043" Score="5" ViewCount="119" Body="&lt;p&gt;Standard deck has 52 cards, 26 Red and 26 Black. A run is a maximum contiguous block of cards, which has the same color.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Eg.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;(R,B,R,B,...,R,B) has 52 runs.&lt;/li&gt;&#xA;&lt;li&gt;(R,R,R,...,R,B,B,B,...,B) has 2 runs.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;What is the expected number of runs in a shuffled deck of cards?&lt;/p&gt;&#xA;" OwnerUserId="994" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-19T16:20:20.837" LastActivityDate="2010-09-19T16:20:20.837" Title="What is the expected number of runs of same color in a standard deck of cards?" Tags="&lt;probability&gt;&lt;games&gt;" AnswerCount="1" CommentCount="3" FavoriteCount="2" />
  <row Id="1866" PostTypeId="1" AcceptedAnswerId="1918" CreationDate="2010-08-19T02:10:15.867" Score="5" ViewCount="234" Body="&lt;p&gt;Following to the recent questions we had &lt;a href=&quot;http://stats.stackexchange.com/questions/1818/how-to-determine-the-sample-size-needed-for-repeated-measurement-anova/1823#1823&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was hopping to know if anyone had come across or can share &lt;strong&gt;R code for performing a custom power analysis based on simulation for a linear model?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Later I would obviously like to extend it to more complex models, but lm seems to right place to start. Thanks.&lt;/p&gt;&#xA;" OwnerUserId="253" LastActivityDate="2010-08-19T17:13:01.420" Title="How to simulate a custom power analysis of an lm model (using R)" Tags="&lt;r&gt;&lt;power&gt;&lt;power-analysis&gt;" AnswerCount="2" FavoriteCount="2" />
  <row Id="1867" PostTypeId="2" ParentId="1863" CreationDate="2010-08-19T02:35:58.207" Score="1" ViewCount="" Body="&lt;p&gt;So this is a one way repeated measures Anova - with the &quot;Y&quot; being time till answer was given, and the first factor having 3 levels (each subject having three of them).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think the easiest way for doing this would be to take the mean response time for each subject for each of the three levels (which will results in 3 numbers per subject).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And then run a friedman test on that (there is also a &lt;a href=&quot;http://www.r-statistics.com/2010/02/post-hoc-analysis-for-friedmans-test-r-code/&quot; rel=&quot;nofollow&quot;&gt;post hoc friedman test in R&lt;/a&gt;, in case you would want that - I assume you would)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The downside of this is that this assumes, in a sense, that your estimation of the three means (a mean for each of the three levels, per subject), are the same where in fact they are not.  You have more variability in your estimation of level 3 then of level 1. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Realistically, I would ignore that.  Theoretically, I hope someone here can offer a better solution so both of us would be able to learn :)&lt;/p&gt;&#xA;" OwnerUserId="253" LastActivityDate="2010-08-19T02:35:58.207" />
  <row Id="1868" PostTypeId="2" ParentId="1860" CreationDate="2010-08-19T03:03:39.437" Score="2" ViewCount="" Body="&lt;p&gt;A few references that you might find useful:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Edwards (2001) has a nice article called &lt;a href=&quot;http://public.kenan-flagler.unc.edu/faculty/edwardsj/Edwards01b.pdf&quot; rel=&quot;nofollow&quot;&gt;Ten Difference Score Myths&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;li&gt;I have a &lt;a href=&quot;http://jeromyanglim.blogspot.com/2009/09/difference-scores-are-they-okay-to-use.html&quot; rel=&quot;nofollow&quot;&gt;post with some general points on change scores&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-08-19T03:03:39.437" CommentCount="2" />
  <row Id="1869" PostTypeId="2" ParentId="1862" CreationDate="2010-08-19T03:15:34.427" Score="6" ViewCount="" Body="&lt;p&gt;You really need to figure out what is the question that you are trying to answer- or what question is management most interested in. Then you can select the survey questions that are most relevant to your problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Without knowing anything about your problem or dataset, here are some generic solutions:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Visually represent the answers as clusters. My favorite is by either using dendrograms or just plotting on an xy axis (Google &quot;cluster analysis r&quot; and go to the first result by statmethods.net)&lt;/li&gt;&#xA;&lt;li&gt;Rank the questions from greatest to least &quot;daily or more frequently&quot; responses. This is an example that may not exactly work for you but perhaps it will inspire you &lt;a href=&quot;http://www.programmingr.com/content/building-scoring-and-ranking-systems-r&quot; rel=&quot;nofollow&quot;&gt;http://www.programmingr.com/content/building-scoring-and-ranking-systems-r&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Crosstabs: if for example, you have a question &quot;How often do you come in late for work?&quot; and &quot;How often do you use Facebook?,&quot; by crosstabbing the two questions you can find out the percentage of people who rarely do both, or who do both everyday.(Google &quot;r frequency crosstabs&quot; or go to the aforementioned statmethods.net)&lt;/li&gt;&#xA;&lt;li&gt;Correlograms. I don't have any experience with these but I saw it also on the statmethods.net website. Basically you find which questions have the highest correlation and then create a table. You may find this useful although it looks kind of &quot;busy.&quot;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="995" LastEditorUserId="995" LastEditorDisplayName="" LastEditDate="2010-08-19T03:21:26.100" LastActivityDate="2010-08-19T03:21:26.100" CommentCount="1" />
  <row Id="1870" PostTypeId="1" AcceptedAnswerId="1879" CreationDate="2010-08-19T04:35:15.690" Score="3" ViewCount="200" Body="&lt;p&gt;The question is in the header, but I would extend the context a bit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Next semester I am due to be a TA in a course in statistics, where I would need to help sociology students learn to use SPSS.  I don't know SPSS, yet, and would like to learn how to use it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was thinking of taking a simple dataset, and start reviewing it with methods I know, thus starting to map out where are methods I know of.  And once finished, to try and explore more options.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone propose other/better strategies to master a new statistical GUI? (in my case SPSS, but it could apply to many other GUI's)&lt;/p&gt;&#xA;" OwnerUserId="253" LastActivityDate="2010-12-12T02:13:30.870" CommunityOwnedDate="2010-08-19T04:35:15.690" Title="How to learn how to use a new statistical GUI?" Tags="&lt;spss&gt;&lt;gui&gt;&lt;learning&gt;" AnswerCount="4" FavoriteCount="1" />
  <row Id="1871" PostTypeId="2" ParentId="1863" CreationDate="2010-08-19T04:35:45.863" Score="3" ViewCount="" Body="&lt;p&gt;It's not imbalanced because your repeated measures should be averaged across such subgroups within subject beforehand.  The only thing imbalanced is the quality of the estimates of your means.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just as you aggregated your accuracies to get a percentage correct and do your ANOVA in the first place you average your latencies as well.  Each participant provides 6 values, therefore it is not imbalanced.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most likely though... the ANOVA was not the best analysis in the first place.  You should probably be using mixed-effect modelling.  For the initial test of the accuracies you'd use mixed effects logistic regression.  For the second one you propose it would be a 3-levels x 2-correctnesses analysis of the latencies.  Both would have subjects as a random effect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In addition it's often best to do some sort of normality correction on the times like a log or -1/T correction.  This is less of a concern in ANOVA because you aggregate across a number of means first and that often ameliorates the skew of latencies through the central limit theorem.  You could check with a boxcox analysis to see what fits best.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On a more important note though... what are you expecting to find?  Is this just exploratory?  What would it mean to have different latencies in the correct and incorrect groups and what would it mean for them to interact?  Unless you are fully modelling the relationship between accuracy and speed in your experiment, or you have a full model that you are testing, then you are probably wasting your time.  A latency with an incorrect response means that someone did something other than what you wanted them to... and it could be anything.  That's why people almost always only work with the latencies to the correct responses. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(these two types of responses also often have very different distributions with incorrect much flatter because they disproportionately make up both the short and long latencies)&lt;/p&gt;&#xA;" OwnerUserId="601" LastEditorUserId="601" LastEditorDisplayName="" LastEditDate="2010-08-21T07:21:06.203" LastActivityDate="2010-08-21T07:21:06.203" CommentCount="2" />
  <row Id="1872" PostTypeId="2" ParentId="1870" CreationDate="2010-08-19T04:47:46.457" Score="3" ViewCount="" Body="&lt;p&gt;Since you are pretty well versed in R, get a copy of Muenchen's &quot;&lt;a href=&quot;http://www.springer.com/statistics/computanional+statistics/book/978-0-387-09417-5&quot; rel=&quot;nofollow&quot;&gt;R for SAS and SPSS Users&lt;/a&gt;&quot; (Springer, 2009) and work backwards.&lt;/p&gt;&#xA;" OwnerUserId="597" LastActivityDate="2010-08-19T04:47:46.457" CommunityOwnedDate="2010-08-19T04:47:46.457" CommentCount="1" />
  <row Id="1873" PostTypeId="1" CreationDate="2010-08-19T04:50:25.533" Score="2" ViewCount="133" Body="&lt;p&gt;I posted this on mathoverflow, but they sent me here. This question relates to a problem I had at work a while ago, doing a little data mining at a car rental company. Names changed, of course. I'm using Oracle DBMS if it matters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There was a flight of steps out the front of our building. It had a dodgy step on it, on which people often stub their toes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I had records for everyone who works in the building, detailing how many times they climbed these steps and how many of these times they stubbed their toes on the dodgy step. There's a total of 3000 stair-climbing incidents and 1000 toe-stubbing incidents.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Jack climbed the steps 15 times and stubbed his toes 7 times, which is 2 more than you'd expect. What's the probability that this is just random, vs the probability that Joe is actually clumsy?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm pretty sure from half-remembered statistics 1 that its something to do with chi-squared, but beats me where to go from there.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, we actually had several flights of steps, each with different rates of toe stubbing and instep bashing. How would I combine the stats from those to get a more accurate better likelihood of Joe being clumsy? We can assume that there's no systematic bias in respect of more clumsy people being inclined to use certain flights of steps.&lt;/p&gt;&#xA;" OwnerUserId="997" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-16T06:49:06.127" LastActivityDate="2010-09-16T06:49:06.127" Title="Based on my data, is Jack likely to be clumsy?" Tags="&lt;hypothesis-testing&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="1" />
  <row Id="1874" PostTypeId="1" CreationDate="2010-08-19T04:53:32.270" Score="4" ViewCount="62" Body="&lt;p&gt;I'm looking to construct a 3-D surface of a part of the brain based on 2-D contours from cross-sectional slices from multiple angles. Once I get this shape, I want to &quot;fit&quot; it to another set of contours via rescaling.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm aspiring to do this in the context of an MCMC analysis (So as to be able to make inferences, so it would be very nice if I could easily compute the volume of the rescaled surface, and the minimum distance between a given point of the distance. (Accurate approximations are fine). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What would be a good image reconstruction algorithm that allows for volume and distance to be quickly calculated?&lt;/p&gt;&#xA;" OwnerUserId="996" LastActivityDate="2010-10-12T18:11:52.460" Title="Parametric Surface Reconstruction from Contours with Quick Rescaling" Tags="&lt;bayesian&gt;&lt;mcmc&gt;&lt;fitting&gt;&lt;optimal-scaling&gt;&lt;interpolation&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="1875" PostTypeId="1" AcceptedAnswerId="1903" CreationDate="2010-08-19T05:56:06.483" Score="15" ViewCount="409" Body="&lt;p&gt;A question which bothered me for some time, which I don't know how to address:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Every day, my weatherman gives a percentage chance of rain (let's assume its calculated to 9000 digits and he has never repeated a number). Every subsequent day, it either rains or does not rain.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have years of data - pct chance vs rain or not. &lt;em&gt;Given this weatherman's history&lt;/em&gt;, if he says tonight that tomorrow's chance of rain is X, then what's my best guess as to what the chance of rain really is?&lt;/p&gt;&#xA;" OwnerUserId="997" LastEditorUserId="223" LastEditorDisplayName="" LastEditDate="2010-12-20T07:28:59.943" LastActivityDate="2011-01-25T18:19:47.763" Title="Is my weatherman accurate?" Tags="&lt;hypothesis-testing&gt;&lt;forecasting&gt;&lt;verification&gt;" AnswerCount="5" CommentCount="1" FavoriteCount="4" />
  <row Id="1876" PostTypeId="2" ParentId="1862" CreationDate="2010-08-19T06:13:02.550" Score="3" ViewCount="" Body="&lt;p&gt;Standard options include:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;getting the mean for items within a scale (e.g., if the scale is 1 to 5, the mean will be 1 to 5)&lt;/li&gt;&#xA;&lt;li&gt;converting each item to a binary measure (e.g., if item &gt;= 3, then 1, else 0) and then taking the mean of this binary response&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Given that you are aggregating over items and over large samples of people in the organisation, both options above (i.e., the mean of 1 to 5 or the mean of percentage above a point) will be reliable at the organisational-level (&lt;a href=&quot;http://jeromyanglim.blogspot.com/2009/10/job-satisfaction-measurement-scales.html&quot; rel=&quot;nofollow&quot;&gt;see here for further discussion&lt;/a&gt;). Thus, either of the above options are basically communicating the same information. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general I wouldn't be worried about the fact that items are categorical. By the time you create scales by aggregating over items and then aggregate over your sample of respondents, the scale will be a close approximation to a continuous scale.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Management may find one metric easier to interpret. When  I get Quality of Teaching scores (i.e., the average student satisfaction score of say 100 students) , it is the average on a 1 to 5 scale and that's fine. Over the years after seeing my own scores from year to year and also seeing some norms for the university I've developed a frame of reference of what different values mean.&#xA; However, management sometimes prefers to think about the percentage endorsing a statement, or the percentage of positive responses even when it is in a sense the mean percentage.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The main challenge is to give some tangible frame of reference for the scores. Management will want to know &lt;strong&gt;what the numbers actually mean&lt;/strong&gt;. For example, if the mean response for a scale is 4.2, What does that mean? Is it good? Is it bad? Is it just okay?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are using the survey over multiple years or in different organisations, then you can start to develop some norms. Access to norms is one reason organisations often get an external survey provider or use a standard survey.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may also wish to run a factor analysis to validate that the assignment of items to scales is empirically justifiable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In terms of a visual approach, you can have a simple line or bar graph with the scale type on the x-axis and the score on the y-axis. If you have normative data, you could add that also. &lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-08-19T06:13:02.550" />
  <row Id="1877" PostTypeId="2" ParentId="1873" CreationDate="2010-08-19T06:16:25.087" Score="3" ViewCount="" Body="&lt;pre&gt;&lt;code&gt;chisq.test(c(15,7),p=c(3000,1000),rescale.p=TRUE)&#xA;&#xA;    Chi-squared test for given probabilities&#xA;&#xA;data:  c(15, 7)&#xA;X-squared = 0.5455, df = 1, p-value = 0.4602&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;There is not enough evidence against the Null hypothesis (that is just a random incident).&#xA;A difference from the expected value as big as or bigger than the one observed will arise by chance alone in more than 46% of cases and is clearly not statistically significant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The chi-squared value is&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sum((c(15,7) - 22*c(3000,1000)/4000)^2 / (22*c(3000,1000)/4000))&#xA;[1] 0.5454545&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and the p-value comes from the right-hand tail of the cumulative probability function of&#xA;the chi-squared distribution 1-pchisq with 1 degrees of freedom (2 comparisons -1 for&#xA;contingency; the total count must be 22)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1-pchisq(0.5454545,1)&#xA;[1] 0.460181&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;exactly as we obtained using the built-in chisq.test function, above.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively, you could carry out a binomial test:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;binom.test(c(15,7),p=3/4)&#xA;&#xA;        Exact binomial test&#xA;&#xA;data:  c(15, 7)&#xA;number of successes = 15, number of trials = 22, p-value = 0.463&#xA;alternative hypothesis: true probability of success is not equal to 0.75&#xA;95 percent confidence interval:&#xA; 0.4512756 0.8613535&#xA;sample estimates:&#xA;probability of success&#xA;             0.6818182&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can see that the 95% confidence interval for the proportion of successes (0.45, 0.86)&#xA;contains 0.75, so there is no evidence against a 3:1 success ratio in these data. The p value is slightly different than it was in the chi-squared test, but the interpretation is exactly the same.&lt;/p&gt;&#xA;" OwnerUserId="339" LastEditorUserId="339" LastEditorDisplayName="" LastEditDate="2010-08-19T06:29:38.990" LastActivityDate="2010-08-19T06:29:38.990" CommentCount="1" />
  <row Id="1878" PostTypeId="2" ParentId="1875" CreationDate="2010-08-19T06:22:09.973" Score="11" ViewCount="" Body="&lt;p&gt;Comparison of probability forecast for binary event (or discrete Random Variable) can be done upon the &lt;strong&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Brier_score&quot; rel=&quot;nofollow&quot;&gt;Brier score&lt;/a&gt;&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;but you can also use &lt;strong&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Receiver_operating_characteristic&quot; rel=&quot;nofollow&quot;&gt;ROC curve&lt;/a&gt;&lt;/strong&gt;  since any probability forecast of this type can be transformed into a dicrimination procedure with a varying threshold &#xA;Indeed you can say &quot;it will rain&quot; if your probability is greater than $\tau$ and evaluate the missed, false discovery,true discovery and  true negatives for different values of $\tau$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should take a look at how the European center for medium range weather forecast (&lt;strong&gt;&lt;a href=&quot;http://www.ecmwf.int/products/forecasts/guide/The_verification_of_ECMWF_forecasts.html&quot; rel=&quot;nofollow&quot;&gt;ECMWF does&lt;/a&gt;&lt;/strong&gt;) .&lt;/p&gt;&#xA;" OwnerUserId="223" LastEditorUserId="223" LastEditorDisplayName="" LastEditDate="2011-01-25T18:19:47.763" LastActivityDate="2011-01-25T18:19:47.763" />
  <row Id="1879" PostTypeId="2" ParentId="1870" CreationDate="2010-08-19T06:28:05.217" Score="6" ViewCount="" Body="&lt;p&gt;As someone who made the shift the other way from SPSS to R, I'd say that SPSS is relatively simple and intuitive relative to R. The menus and dialog boxes guide you through the process. Of course this means that it is also fairly easy to run analyses that don't make sense. And the GUI  leads to less flexible analyses and tedious button pressing especially for repetitive analyses.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, your approach of taking a dataset and just playing around might be sufficient.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There's plenty of how-to books out there, such as:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Discovering Statistics Using SPSS&lt;/li&gt;&#xA;&lt;li&gt;SPSS Survival Manual&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;There's also plenty of websites offering tutorials: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.statisticshell.com/woodofsuicides.html&quot; rel=&quot;nofollow&quot;&gt;Andy Field&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://jeromyanglim.blogspot.com/2009/09/teaching-resources.html&quot; rel=&quot;nofollow&quot;&gt;My old notes&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.ats.ucla.edu/stat/spss/&quot; rel=&quot;nofollow&quot;&gt;UCLA on SPSS&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I'd also recommend that if you are teaching students about SPSS, you encourage them to use syntax. Using SPSS syntax is not as good as using technologies like &lt;a href=&quot;http://www.r-bloggers.com/getting-started-with-sweave-r-latex-eclipse-statet-texlipse/&quot; rel=&quot;nofollow&quot;&gt;R and Sweave&lt;/a&gt;.&#xA;However, using syntax is much better than just pressing menus and buttons in an ad hoc way and then wondering later what you've actually done.&#xA;I wrote a post listing tips for using &lt;a href=&quot;http://jeromyanglim.blogspot.com/2009/10/introduction-to-spss-syntax-advice-for.html&quot; rel=&quot;nofollow&quot;&gt;SPSS syntax in order to approximate reproducible research with SPSS&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-08-19T06:28:05.217" CommunityOwnedDate="2010-08-19T06:28:05.217" CommentCount="2" />
  <row Id="1880" PostTypeId="2" ParentId="1866" CreationDate="2010-08-19T06:35:02.147" Score="1" ViewCount="" Body="&lt;p&gt;Here are a few sources of simulation code in R. I'm not sure if any specifically address linear models, but perhaps they provide enough of an example to get the gist:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Benjamin Bolker has written a great book &lt;a href=&quot;http://www.math.mcmaster.ca/~bolker/emdbook/&quot; rel=&quot;nofollow&quot;&gt;Ecological Data and Models with R&lt;/a&gt;. An early draft of the whole book along with Sweave code is available online. &lt;a href=&quot;http://www.math.mcmaster.ca/~bolker/emdbook/chap5A.pdf&quot; rel=&quot;nofollow&quot;&gt;Chapter 5&lt;/a&gt; addresses power analysis and simulation.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;There's another couple of examples of simulation at the following sites:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.personality-project.org/R/r.datageneration.html&quot; rel=&quot;nofollow&quot;&gt;http://www.personality-project.org/R/r.datageneration.html&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://psy-ed.wikidot.com/simulation&quot; rel=&quot;nofollow&quot;&gt;http://psy-ed.wikidot.com/simulation&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-08-19T06:35:02.147" />
  <row Id="1881" PostTypeId="1" AcceptedAnswerId="1901" CreationDate="2010-08-19T06:42:15.767" Score="3" ViewCount="288" Body="&lt;p&gt;Hello everyone,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like an advice on a analysis method I am using, to know if it it statistically sound.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have measured two point processes $T^1 = t^1_1, t^1_2, ..., t^1_n$ and $T^2 = t^2_1, t^2_2, ..., t^2_m$ and I want to determine if the events in $T^1$ are somehow correlated to the events in $T^2$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of the methods that I have found in the literature is that of constructing a cross-correlation histogram: for each $t^1_n$ we find the delay to all the events of $T^2$ that fall in a given window of time (before and after $t^1_n$), and then we construct an histogram of all these delays.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the two processes are not correlated I would expect a flat histogram, as the probability of having an event in $T^2$ after (or before) an event in $T^1$ is equal at all delays. On the other hand if there is a peak in the histogram, this suggests that the two point process are somehow influencing each other (or, at least, have some common input).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, this is nice and good, but how do I determine whether the histograms do have a peak (I have to say that for my particular set of data they're clearly flat, but still it would be nice to have a statistical way of confirming that)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, here what I've done: I've repeated the process of generating the histogram for several (1000) times keeping $T^1$ as it is and using a &quot;shuffled&quot; version of $T^2$.&#xA;To shuffle $T^2$ I calculate the intervals between all the events, shuffle them and sum them to reconstitute a new point process. In R I simply do this with:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;times2.swp &amp;lt;- cumsum(sample(diff(times2)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So, I end up with 1000 new histogram, that show me the density of events in $T^{2*}$ compared to $T^1$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For each bin of these histogram (they're all binned in the same way) I calculate the density of 95% of the histogram. In other words I'm saying, for instance: at time delay 5 ms, in 95% of the shuffled point processes there is a probability x of finding an event in $T^{2*}$ after an event in $T^1$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would then take this 95% value for all of the time delays and use it as some &quot;confidence limit&quot; (probably this is not the correct term) so that anything that goes over this limit in the original histogram can be considered a &quot;true peak&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question 1&lt;/strong&gt;: is this method statistically correct? If not how would you tackle this problem?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question 2&lt;/strong&gt;: another thing that I want to see is whether there is a &quot;longer&quot; type of correlation of my data. For instance there may be similar changes in the rate of events in the two point processes (note that they may have quite different rates), but I'm not sure how to do that. I thought of creating an &quot;envelope&quot; of each point process using some sort of smoothing kernel and then performing a cross-correlation analysis of the two envelopes. Could you suggest any other possible type of analysis?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you and sorry for this very long question.&lt;/p&gt;&#xA;" OwnerUserId="582" LastActivityDate="2010-08-19T13:07:22.083" Title="Analysis of cross correlation between point-processes" Tags="&lt;point-process&gt;&lt;cross-correlation&gt;" AnswerCount="1" FavoriteCount="3" />
  <row Id="1882" PostTypeId="2" ParentId="1829" CreationDate="2010-08-19T06:42:28.403" Score="6" ViewCount="" Body="&lt;p&gt;To answer the letter of the question, &quot;ordinary least squares&quot; is not an algorithm; rather it is a type of problem in computational linear algebra, of which linear regression is one example. Usually one has data $(x_1,y_1)\dots(x_m,y_m)$ and a tentative function to fit the data against, of the form $f(x)=c_1 f_1(x)+\dots+c_n f_n(x)$. The $f_j(x)$ are called basis functions and can be anything from monomials $x^j$ to trigonometric functions (e.g. $\sin(jx)$) and exponential functions ($\exp(-jx)$). The term &quot;linear&quot; in &quot;linear regression&quot; here does not refer to the basis functions but to the coefficients $c_j$, in that taking the partial derivative with respect to any of the $c_j$ gives you the factor multiplying $c_j$; that is, $f_j(x)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One now has an m-by-n rectangular matrix $\textbf{A}$ (&quot;design matrix&quot;) that (usually) has more rows than columns, and each entry is of the form $f_j(x_i)$, i being the row index and j being the column index. OLS is now the task of finding the vector $\textbf{c}=c_1\dots c_n)$ that minimizes the quantity $\sqrt{\sum_{j=1}^{m}\left(y_j-f(x_j)\right)^2}$ (in matrix notation, $\|\textbf{A}\textbf{c}-\textbf{y}\|_2$ ; here, $\textbf{y}$ is usually called the &quot;response vector&quot;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are at least three methods used in practice for computing least-squares solutions: the normal equations, QR decomposition, and singular value decomposition. In brief, they are ways to transform the matrix $\textbf{A}$ into a product of matrices that are easily manipulated to solve for the vector $\textbf{c}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;gd047 already showed the method of normal equations in his answer; one just solves the n-by-n set of linear equations&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\textbf{A}^T\textbf{A}\textbf{c}=\textbf{A}^T\textbf{y}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;for $\textbf{c}$. Due to the fact that the matrix $\textbf{A}^T\textbf{A}$ is symmetric positive (semi)definite, the usual method used for this is Cholesky decomposition, which factors $\textbf{A}^T\textbf{A}$ into the form $\textbf{G}\textbf{G}^T$, $\textbf{G}$ a lower triangular matrix. The problem with this approach, despite the advantage of being able to compress the m-by-n design matrix into a (usually) much smaller n-by-n matrix, is that this operation is prone to loss of significant figures (this has something to do with the &quot;condition number&quot; of the design matrix).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A slightly better way is QR decomposition, which directly works with the design matrix. It factors $\textbf{A}$ as $\textbf{A}=\textbf{Q}\textbf{R}$, where $\textbf{Q}$ is an orthogonal matrix (the transpose of the matrix is the inverse) and $\textbf{R}$ is upper triangular. $\textbf{c}$ is subsequently computed as $\textbf{R}^{-1}\textbf{Q}^T\textbf{y}$. For reasons I won't get into (just see any numerical linear algebra text), this has better numerical properties than the method of normal equations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, the most expensive, yet safest, way of solving OLS is the singular value decomposition (SVD). This time, $\textbf{A}$ is factored as $\textbf{A}=\mathbf{U}\Sigma\textbf{V}^T$, where $\textbf{U}$ and $\textbf{V}$ are both orthogonal, and $\mathbf{\Sigma}$ is a diagonal matrix, whose diagonal entries are termed &quot;singular values&quot;. The power of this decomposition lies in the diagnostic ability granted to you by the singular values, in that if one sees one or more tiny singular values, then it is likely that you have chosen a not entirely independent basis set, thus necessitating a reformulation of your functional form.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is merely a sketch of these three algorithms; any good book on computational statistics and numerical linear algebra should be able to give you more relevant details.&lt;/p&gt;&#xA;" OwnerUserId="830" LastEditorUserId="830" LastEditorDisplayName="" LastEditDate="2010-08-19T06:51:15.437" LastActivityDate="2010-08-19T06:51:15.437" CommentCount="1" />
  <row Id="1883" PostTypeId="1" CreationDate="2010-08-19T07:00:53.840" Score="3" ViewCount="508" Body="&lt;p&gt;Let's say &lt;a href=&quot;http://chat.meta.stackoverflow.com/rooms/170/stats&quot; rel=&quot;nofollow&quot;&gt;chat&lt;/a&gt; is a better place for such discussions. --mbq&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What areas of statistics have been substantially revolutionised in the last 50 years? For example, about 40 years ago, Akaike with colleagues revolutionised the area of statistical model discrimination. About 10 years ago, Hyndman with colleagues revolutionised the area of exponential smoothing. About XX years ago, ... How do I possibly continue the list, with years and names please? By statistics I mean its all four types from Bartholomew's 1995 presidential address, Chambers's greater and lesser statistics together, as featuring in Hand's recent presidential address on 'Modern statistics' and so on - anything professionally relevant.&lt;/p&gt;&#xA;" OwnerUserId="273" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-17T20:21:24.043" LastActivityDate="2010-09-17T20:21:24.043" CommunityOwnedDate="2010-08-19T07:05:16.077" ClosedDate="2010-08-19T15:10:48.170" Title="Revolutions in statistics for the last 50 years?" Tags="&lt;for-retag&gt;" AnswerCount="11" CommentCount="10" FavoriteCount="3" />
  <row Id="1884" PostTypeId="2" ParentId="1865" CreationDate="2010-08-19T08:34:51.493" Score="3" ViewCount="" Body="&lt;p&gt;Suppose $X_n$ denotes the color of the $n$th card in the shuffled deck.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then note that the last card always denotes the end of a run. Other ends of runs are characterized by $X_n\ne X_{n+1}$ which indicates a run ending at $n$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that $P(X_n\ne X_{n+1})=26/51$ (since once you fix a card, you can choose another card from remaining 51 out of which 26 will have a different color).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So summing up the indicators $X_n\ne X_{n+1}$ we get the number of runs - &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\#runs=1+\sum_{n=1}^{51}\mathbb{I}_{X_n\ne X_{n+1}}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So $E(\#runs)=1+\sum_{n=1}^{51}P(X_n\ne X_{n+1})=1+\sum_{n=1}^{51}26/51=27$.&lt;/p&gt;&#xA;" OwnerUserId="994" LastActivityDate="2010-08-19T08:34:51.493" CommentCount="5" />
  <row Id="1885" PostTypeId="2" ParentId="1850" CreationDate="2010-08-19T08:50:44.340" Score="4" ViewCount="" Body="&lt;p&gt;To my understanding, Hedges's g is a somewhat more accurate version of Cohen's d (with pooled SD) in that we add a correction factor for small sample. Both measures generally agree when the homoscedasticity assumption is not violated, but we may found situations where this is not the case, see e.g. McGrath &amp;amp; Meyer, &lt;em&gt;Psychological Methods&lt;/em&gt; 2006, &lt;strong&gt;11(4)&lt;/strong&gt;: 386-401 (&lt;a href=&quot;http://www.bobmcgrath.org/Pubs/When_effect_sizes_disagree.pdf&quot; rel=&quot;nofollow&quot;&gt;pdf&lt;/a&gt;). Other papers are listed at the end of my reply.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I generally found that in almost every psychological or biomedical studies, this is the Cohen's d that is reported; this probably stands from the well-known rule of thumb for interpreting its magnitude (Cohen, 1988). I don't know about any recent paper considering Hedges's g (or Cliff delta as a non-parametric alternative). Bruce Thompson has a &lt;a href=&quot;http://people.cehd.tamu.edu/~bthompson/apaeffec.htm&quot; rel=&quot;nofollow&quot;&gt;revised version&lt;/a&gt; of the APA section on effect size.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Googling about Monte Carlo studies around effect size measures, I found this paper which might be interesting (I only read the abstract and the simulation setup): &lt;a href=&quot;http://www.coedu.usf.edu/main/departments/me/documents/cohen.pdf&quot; rel=&quot;nofollow&quot;&gt;Robust Confidence Intervals for Effect Sizes: A Comparative Study of Cohen’s d and Cliff’s Delta Under Non-normality and Heterogeneous Variances&lt;/a&gt; (pdf).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;About your 2nd comment, the &lt;code&gt;MBESS&lt;/code&gt; R package includes various utilities for ES calculation (e.g., &lt;code&gt;smd&lt;/code&gt; and related functions).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Other references&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Zakzanis, K.K. (2001). Statistics to tell the truth, the whole truth, and nothing but the truth: Formulae, illustrative numerical examples, and heuristic interpretation of effect size analyses for neuropsychological researchers. &lt;em&gt;Archives of Clinical Neuropsychology&lt;/em&gt;, 16(7), 653-667. (&lt;a href=&quot;http://www.sciencedirect.com/science?_ob=ArticleURL&amp;amp;_udi=B6VDJ-43TFJ8B-4&amp;amp;_user=10&amp;amp;_coverDate=10%2F31%2F2001&amp;amp;_rdoc=1&amp;amp;_fmt=high&amp;amp;_orig=search&amp;amp;_sort=d&amp;amp;_docanchor=&amp;amp;view=c&amp;amp;_searchStrId=1435354249&amp;amp;_rerunOrigin=google&amp;amp;_acct=C000050221&amp;amp;_version=1&amp;amp;_urlVersion=0&amp;amp;_userid=10&amp;amp;md5=12f831e4b738bda3a217d40d6cfd329f&quot; rel=&quot;nofollow&quot;&gt;pdf&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;li&gt;Durlak, J.A. (2009). How to Select, Calculate, and Interpret Effect Sizes. &lt;em&gt;Journal of Pediatric Psychology&lt;/em&gt; (&lt;a href=&quot;http://jpepsy.oxfordjournals.org/cgi/content/full/jsp004v1&quot; rel=&quot;nofollow&quot;&gt;pdf&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-08-19T08:50:44.340" />
  <row Id="1887" PostTypeId="2" ParentId="10" CreationDate="2010-08-19T10:00:00.370" Score="4" ViewCount="" Body="&lt;p&gt;Maybe too late but I add my answer anyway...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It depends on what you intend to do with your data: If you are interested in showing that scores differ when considering different group of participants (gender, country, etc.), you may treat your scores as numeric values, provided they fulfill usual assumptions about variance (or shape) and sample size. If you are rather interested in highlighting how response patterns vary across subgroups, then you should consider item scores as discrete choice among a set of answer options and look for log-linear modeling, ordinal logistic regression, item-response models or any other statistical model that allows to cope with polytomous items.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a rule of thumb, one generally considers that having 12 distinct points on a scale is sufficient to approximate an interval scale (for interpretation purpose). Likert items may be regarded as true ordinal scale, but they are often used as numeric and we can compute their mean or SD. This is often done in attitude surveys, although it is wise to report both mean/SD and % of response in, e.g. the two highest categories.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When using summated scale scores (i.e., we add up score on each item to compute a &quot;total score&quot;), usual statistics may be applied, but you have to keep in mind that you are now working with a latent variable so the underlying construct should make sense! In psychometrics, we generally check that (1) unidimensionnality of the scale holds, (2) scale reliability is sufficient. When comparing two such scale scores (for two different instruments), we might even consider using attenuated correlation measures instead of classical Pearson correlation coefficient. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Classical textbooks include:&lt;br&gt;&#xA;1. Nunnally, J.C. and Bernstein, I.H. (1994). &lt;em&gt;Psychometric Theory&lt;/em&gt; (3rd ed.). McGraw-Hill Series in Psychology.&lt;br&gt;&#xA;2. Streiner, D.L. and Norman, G.R. (2008). &lt;em&gt;Health Measurement Scales. A practical guide to their development and use&lt;/em&gt; (4th ed.). Oxford.&lt;br&gt;&#xA;3. Rao, C.R. and Sinharay, S., Eds. (2007). &lt;em&gt;Handbook of Statistics, Vol. 26: Psychometrics&lt;/em&gt;. Elsevier Science B.V.&lt;br&gt;&#xA;4. Dunn, G. (2000). &lt;em&gt;Statistics in Psychiatry&lt;/em&gt;. Hodder Arnold.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may also have a look at &lt;a href=&quot;http://www.ipn.uni-kiel.de/aktuell/buecher/rostbuch/inhalt.htm&quot; rel=&quot;nofollow&quot;&gt;Applications of latent trait and latent class models in the social sciences&lt;/a&gt;, from Rost &amp;amp; Langeheine, and W. Revelle's website on &lt;a href=&quot;http://www.personality-project.org/&quot; rel=&quot;nofollow&quot;&gt;personality research&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When validating a psychometric scale, it is important to look at so-called ceiling/floor effects (large asymmetry resulting from participants scoring at the lowest/highest response category), which may seriously impact on any statistics computed when treating them as numeric variable (e.g., country aggregation, t-test). This raises specific issues in cross-cultural studies since it is known that overall response distribution in attitude or health surveys differ from one country to the other (e.g. chinese people vs. those coming from western countries tend to highlight specific response pattern, the former having generally more extreme scores at the item level, see e.g. Song, X.-Y. (2007) Analysis of multisample structural equation models with applications to Quality of Life data, in &lt;em&gt;Handbook of Latent Variable and Related Models&lt;/em&gt;, Lee, S.-Y. (Ed.), pp 279-302, North-Holland).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More generally, you should look at the psychometric-related literature which makes extensive use of Likert items if you are interested with measurement issue. Various statistical models have been developed and are currently headed under the Item Response Theory framework.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-03-30T18:37:24.253" LastActivityDate="2011-03-30T18:37:24.253" />
  <row Id="1888" PostTypeId="2" ParentId="203" CreationDate="2010-08-19T10:19:35.723" Score="2" ViewCount="" Body="&lt;p&gt;Clason &amp;amp; Dormody discussed the issue of statistical testing for Lickert items (&lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.113.2197&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;Analyzing data measured by individual Likert-type items&lt;/a&gt;). I think that a bootstraped test is ok when the two distributions look similar (bell shaped and equal variance). However, a test for categorical data (e.g. trend or Fisher test, or ordinal logistic regression) would be interesting too since it allows to check for response distribution across the item categories, see Agresti's book on &lt;em&gt;Categorical Data Analysis&lt;/em&gt; (Chapter 7 on &lt;em&gt;Logit models for multinomial responses&lt;/em&gt;). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Aside from this, you can imagine situations where the t-test or any other non-parametric tests would fail if the response distribution is strongly imbalanced between the two groups. For example, if all people from group A answer 1 or 5 (in equally proportion) whereas all people in group B answer 3, then you end up with identical within-group mean and the test is not meaningful at all, though in this case the homoscedasticity assumption is largely violated.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-08-19T10:19:35.723" CommentCount="3" />
  <row Id="1889" PostTypeId="2" ParentId="1883" CreationDate="2010-08-19T10:29:27.773" Score="7" ViewCount="" Body="&lt;p&gt;Efron's work on the &lt;a href=&quot;http://en.wikipedia.org/wiki/Bootstrapping_%28statistics%29&quot; rel=&quot;nofollow&quot;&gt;Bootstrap&lt;/a&gt; comes to mind.&lt;/p&gt;&#xA;" OwnerUserId="334" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-08-19T10:44:00.393" LastActivityDate="2010-08-19T10:44:00.393" CommunityOwnedDate="2010-08-19T10:29:27.773" CommentCount="2" />
  <row Id="1890" PostTypeId="2" ParentId="1883" CreationDate="2010-08-19T10:45:32.653" Score="1" ViewCount="" Body="&lt;p&gt;The creation of this site ;-)&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-08-19T10:45:32.653" CommunityOwnedDate="2010-08-19T10:45:32.653" CommentCount="1" />
  <row Id="1891" PostTypeId="2" ParentId="1883" CreationDate="2010-08-19T11:03:39.603" Score="3" ViewCount="" Body="&lt;p&gt;Generalized linear models due to the recently deceased John Nelder and Robert Wedderburn.&lt;/p&gt;&#xA;" OwnerUserId="521" LastActivityDate="2010-08-19T11:03:39.603" CommunityOwnedDate="2010-08-19T11:03:39.603" />
  <row Id="1892" PostTypeId="2" ParentId="1883" CreationDate="2010-08-19T11:06:22.780" Score="3" ViewCount="" Body="&lt;ul&gt;&#xA;&lt;li&gt;Revolution 1: &lt;strong&gt;S&lt;/strong&gt; (&lt;a href=&quot;http://www.acm.org/announcements/ss99.html&quot; rel=&quot;nofollow&quot;&gt;ACM Software Systems Award&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;li&gt;Revolution 2: &lt;strong&gt;R&lt;/strong&gt; (&lt;a href=&quot;http://cran.r-project.org/doc/html/interface98-paper/paper.html&quot; rel=&quot;nofollow&quot;&gt;Ross Ihaka (1998)&lt;/a&gt; on the history of R to that point)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-08-19T11:06:22.780" CommunityOwnedDate="2010-08-19T11:06:22.780" CommentCount="5" />
  <row Id="1893" PostTypeId="2" ParentId="1883" CreationDate="2010-08-19T11:21:55.827" Score="7" ViewCount="" Body="&lt;p&gt;The application of Bayesian statistics with Monte Carlo methods. &lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-08-19T11:21:55.827" CommunityOwnedDate="2010-08-19T11:21:55.827" />
  <row Id="1894" PostTypeId="2" ParentId="1883" CreationDate="2010-08-19T11:22:51.240" Score="5" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Ensemble_learning&quot; rel=&quot;nofollow&quot;&gt;Ensemble methods&lt;/a&gt; like boosting, bagging, ... etc are another potential candidate.&lt;/p&gt;&#xA;" OwnerUserId="334" LastActivityDate="2010-08-19T11:22:51.240" CommunityOwnedDate="2010-08-19T11:22:51.240" />
  <row Id="1895" PostTypeId="1" CreationDate="2010-08-19T11:33:01.067" Score="2" ViewCount="86" Body="&lt;p&gt;I have a software benchmark which is quite noisy. I am trying to for the bugs which are causing the noise, and I need to be able to measure it somehow.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The benchmark is comprised of a number of subbenchmarks, for example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&quot;3d-cube&quot;: 31.56884765625,&#xA;&quot;3d-morph&quot;: 21.89599609375,&#xA;&quot;3d-raytrace&quot;: 51.802978515625,&#xA;&quot;access-binary-trees&quot;: 15.09521484375,&#xA;&quot;access-fannkuch&quot;: 45.578857421875,&#xA;&quot;access-nbody&quot;: 8.651123046875,&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The times are in milliseconds. The times typically vary between runs. For example, on my machine, the &quot;3d-cube&quot; benchmark tends to take around 35ms, but I've seen it go as high as 44ms, and 31ms (above) is uncharacteristically low.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My aim is to change the benchmark so that minor improvements to the run-time can be visible in a benchmark result. What I need is a number that tells me whether I have reduced the &quot;variability&quot; of the benchmark.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;My own solution&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;I run it the benchmark 1000 times, the took the sum of the differences between each subbenchmark's mean and its actual run-times. In pseudo-code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;v = 0&#xA;for s in subbenchmarks:&#xA;  x = mean of all iterations of s&#xA;  for i in iteration&#xA;    v += absolute_value(results[s][i] - x)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm sure this isn't statistically valid (having asked someone), but what is a &quot;correct&quot; way of measuring this &quot;variability&quot; so that I can reduce it.&lt;/p&gt;&#xA;" OwnerUserId="1001" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-08-19T11:59:53.297" LastActivityDate="2010-08-19T12:53:59.380" Title="Determining the &quot;variability&quot; of a benchmark" Tags="&lt;variance&gt;&lt;software&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="1896" PostTypeId="2" ParentId="1895" CreationDate="2010-08-19T11:58:09.057" Score="2" ViewCount="" Body="&lt;p&gt;I guess that your method is the one described &lt;a href=&quot;http://en.wikipedia.org/wiki/Absolute_deviation&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;, and it's apparently valid. You could also have used the &lt;a href=&quot;http://en.wikipedia.org/wiki/Standard_deviation&quot; rel=&quot;nofollow&quot;&gt;standard deviation&lt;/a&gt; as a measure of variability (which according to the article, it's not as &lt;a href=&quot;http://en.wikipedia.org/wiki/Robust_statistics&quot; rel=&quot;nofollow&quot;&gt;robust&lt;/a&gt; as your absolute deviation)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Check out &lt;a href=&quot;http://en.wikipedia.org/wiki/Statistical_dispersion&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt;, for other measures of statistical dispersion.&lt;/p&gt;&#xA;" OwnerUserId="339" LastActivityDate="2010-08-19T11:58:09.057" CommentCount="4" />
  <row Id="1897" PostTypeId="2" ParentId="1895" CreationDate="2010-08-19T12:29:30.023" Score="2" ViewCount="" Body="&lt;p&gt;As gd047 mentioned, the standard way of measuring variability is to use the &lt;a href=&quot;http://en.wikipedia.org/wiki/Variance&quot; rel=&quot;nofollow&quot;&gt;variance&lt;/a&gt;. So your pseudo-code will be:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;vnew = vector of length subbenchmarks&#xA;for s in subbenchmarks:&#xA;  vnew[i] = variance(s)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now the problem is, even if  you don't change your code, &lt;code&gt;vnew&lt;/code&gt; will be different for each run - there is noise. To determine if a change is significant, we need to perform a &lt;a href=&quot;http://en.wikipedia.org/wiki/Statistical_hypothesis_testing&quot; rel=&quot;nofollow&quot;&gt;hypothesis test&lt;/a&gt;, i.e. can the change be explained as random variation or is likely that something has changed. A quick and dirty rule would be:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{equation}&#xA;Y_i = \sqrt{n/2} (\frac{vnew_i}{vold_i} -1) \sim N(0,1)&#xA;\end{equation}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This means any values of $Y_i &amp;lt; -1.96$ (at a 5% significance level) can be considered significant, i.e. an improvement. However, I would probably increase this to -3 or -4. This would test for improvement in &lt;strong&gt;individual&lt;/strong&gt; benchmarks. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to combine all your benchmarks into a single test, then let&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{equation}&#xA;\bar Y = \frac{1}{n} \sum Y_i&#xA;\end{equation}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So &lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{equation}&#xA;\sqrt{n} \bar Y \sim N(0, 1)&#xA;\end{equation}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hence, an appropriate test would be to consider values of $\bar Y &amp;lt; 1.96$ to indicate an improvement. &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the benchmarks aren't Normal, then I would try working with log(benchmarks). It also depends on what you want to do. I read your question as &quot;You would like a good rule of thumb&quot;. In this case, taking logs is probably OK.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Further details of the mathematical reasoning are found at Section 3.2 of this &lt;a href=&quot;http://dsmts.googlecode.com/files/dsmts-userguide31.pdf&quot; rel=&quot;nofollow&quot;&gt;document&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;li&gt;I've made a approximation by assuming that v_old represents the true underlying variance.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="8" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-08-19T12:53:59.380" LastActivityDate="2010-08-19T12:53:59.380" CommentCount="1" />
  <row Id="1898" PostTypeId="2" ParentId="1883" CreationDate="2010-08-19T12:45:55.700" Score="3" ViewCount="" Body="&lt;p&gt;There was a great discussion on metaoptimize called &quot;&lt;a href=&quot;http://metaoptimize.com/qa/questions/867/most-influential-ideas-1995-2005&quot; rel=&quot;nofollow&quot;&gt;Most Influential Ideas 1995 - 2005&lt;/a&gt;&quot; &#xA;Which holds a great collection of ideas.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The one I mentioned there, and will repeat here, is the &quot;revolution&quot; in the concept of multiple comparisons, specifically the shift from using FWE to FDR methods, for testing very many hypothesis (like in micro array or fMRI and so on)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is one of the first articles that introduced this notion to the scientific community: &lt;a href=&quot;http://www.math.tau.ac.il/~ybenja/MyPapers/benjamini_hochberg1995.pdf&quot; rel=&quot;nofollow&quot;&gt;Benjamini, Yoav; Hochberg, Yosef (1995). &quot;Controlling the false discovery rate: a practical and powerful approach to multiple testing&quot;. Journal of the Royal Statistical Society&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="253" LastActivityDate="2010-08-19T12:45:55.700" CommunityOwnedDate="2010-08-19T12:45:55.700" CommentCount="2" />
  <row Id="1899" PostTypeId="2" ParentId="1883" CreationDate="2010-08-19T12:50:42.303" Score="2" ViewCount="" Body="&lt;p&gt;Cox proportional hazards survival analysis:&#xA;&lt;a href=&quot;http://en.wikipedia.org/wiki/Cox_proportional_hazards_model&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Cox_proportional_hazards_model&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="521" LastActivityDate="2010-08-19T12:50:42.303" CommunityOwnedDate="2010-08-19T12:50:42.303" />
  <row Id="1900" PostTypeId="2" ParentId="1883" CreationDate="2010-08-19T13:01:46.790" Score="6" ViewCount="" Body="&lt;p&gt;John Tukey's truly strange idea: exploratory data analysis.&#xA;&lt;a href=&quot;http://en.wikipedia.org/wiki/Exploratory_data_analysis&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Exploratory_data_analysis&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="521" LastActivityDate="2010-08-19T13:01:46.790" CommunityOwnedDate="2010-08-19T13:01:46.790" />
  <row Id="1901" PostTypeId="2" ParentId="1881" CreationDate="2010-08-19T13:07:22.083" Score="2" ViewCount="" Body="&lt;p&gt;A standard method to analyze this problem in two or more dimensions is &lt;strong&gt;Ripley's (cross) K function&lt;/strong&gt;, but there's no reason not to use it in one dimension, too.  (A Google search does a good job of digging up references.)  Essentially, it plots the CDF of all distances between points in the two realizations rather than a histogram approximation to the PDF of those distances.  (A variant, the L function, plots the difference between K and the null distribution for two uniform uncorrelated processes.)  This neatly sidesteps most of the issues you are confronting with the need to choose bins, to smooth, etc.  Confidence bands for K are typically created through simulation.  This is easy to do in R.  Many spatial stats packages for R can be used directly or readily adapted to this 1D case.  Roger Bivand's &lt;a href=&quot;http://cran.r-project.org/web/views/Spatial.html&quot; rel=&quot;nofollow&quot;&gt;overview page&lt;/a&gt; on CRAN lists these packages: refer to the section on &quot;Point Pattern Analysis.&quot;&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-08-19T13:07:22.083" CommentCount="1" />
  <row Id="1902" PostTypeId="2" ParentId="1883" CreationDate="2010-08-19T13:09:50.597" Score="2" ViewCount="" Body="&lt;p&gt;The Box-Jenkins approach to time-series modelling: ARIMA models etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Box-Jenkins&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Box-Jenkins&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="521" LastActivityDate="2010-08-19T13:09:50.597" CommunityOwnedDate="2010-08-19T13:09:50.597" />
  <row Id="1903" PostTypeId="2" ParentId="1875" CreationDate="2010-08-19T13:21:56.153" Score="6" ViewCount="" Body="&lt;p&gt;In effect you are thinking of a model in which the &lt;em&gt;true&lt;/em&gt; chance of rain, &lt;em&gt;p&lt;/em&gt;, is a function of the &lt;em&gt;predicted&lt;/em&gt; chance &lt;em&gt;q&lt;/em&gt;: &lt;em&gt;p&lt;/em&gt; = &lt;em&gt;p(q&lt;/em&gt;).  Each time a prediction is made, you observe one realization of a Bernoulli variate having probability &lt;em&gt;p(q)&lt;/em&gt; of success.  This is a classic logistic regression setup if you are willing to model the true chance as a linear combination of basis functions &lt;em&gt;f1&lt;/em&gt;, &lt;em&gt;f2&lt;/em&gt;, ..., &lt;em&gt;fk&lt;/em&gt;; that is, the model says&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Logit(&lt;em&gt;p&lt;/em&gt;) = &lt;em&gt;b0&lt;/em&gt; + &lt;em&gt;b1 f1(q)&lt;/em&gt; + &lt;em&gt;b2 f2(q)&lt;/em&gt; + ... + &lt;em&gt;bk fk(q)&lt;/em&gt; + &lt;em&gt;e&lt;/em&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;with iid errors &lt;em&gt;e&lt;/em&gt;.  If you're agnostic about the form of the relationship (although if the weatherman is any good &lt;em&gt;p(q) - q&lt;/em&gt; should be reasonably small), consider using a set of splines for the basis.  The output, as usual, consists of estimates of the coefficients and an estimate of the variance of &lt;em&gt;e&lt;/em&gt;.  Given any future prediction &lt;em&gt;q&lt;/em&gt;, just plug the value into the model with the estimated coefficients to obtain an answer to your question (and use the variance of &lt;em&gt;e&lt;/em&gt; to construct a prediction interval around that answer if you like).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This framework is flexible enough to include other factors, such as the possibility of changes in the quality of predictions over time.  It also lets you test hypotheses, such as whether &lt;em&gt;p&lt;/em&gt; = &lt;em&gt;q&lt;/em&gt; (which is what the weatherman implicitly claims).&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-08-19T13:21:56.153" CommentCount="1" />
  <row Id="1904" PostTypeId="1" CreationDate="2010-08-19T13:25:53.413" Score="7" ViewCount="227" Body="&lt;p&gt;What are the most significant annual Statistics conferences?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rules:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;One conference per answer&lt;/li&gt;&#xA;&lt;li&gt;Include a link to the conference &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-10-05T15:08:21.867" CommunityOwnedDate="2010-08-19T13:25:53.413" Title="Statistics conferences?" Tags="&lt;conferences&gt;" AnswerCount="8" FavoriteCount="2" />
  <row Id="1905" PostTypeId="2" ParentId="1904" CreationDate="2010-08-19T13:30:01.783" Score="1" ViewCount="" Body="&lt;p&gt;Shameless plug: &lt;a href=&quot;http://www.RinFinance.com&quot; rel=&quot;nofollow&quot;&gt;R/Finance&lt;/a&gt; which relevant for its intersection of domain-specifics as well as tools, and so far well received by participants of the 2009 and 2010 conference. .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Disclaimer: I am one of the organizers.&lt;/p&gt;&#xA;" OwnerUserId="334" LastActivityDate="2010-08-19T13:30:01.783" CommunityOwnedDate="2010-08-19T13:30:01.783" />
  <row Id="1906" PostTypeId="1" CreationDate="2010-08-19T13:37:35.557" Score="7" ViewCount="145" Body="&lt;p&gt;What are the most significant annual Data Mining conferences?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rules:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;One conference per answer&lt;/li&gt;&#xA;&lt;li&gt;Include a link to the conference &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-12-15T17:28:47.470" CommunityOwnedDate="2010-08-19T13:44:40.703" Title="Data mining conferences?" Tags="&lt;conferences&gt;" AnswerCount="5" FavoriteCount="1" />
  <row Id="1907" PostTypeId="2" ParentId="1906" CreationDate="2010-08-19T13:40:35.720" Score="2" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://www.sigkdd.org/conferences.php&quot; rel=&quot;nofollow&quot;&gt;KDD&lt;/a&gt; (ACM Special Interest Group on Knowledge Discovery and Data Mining) &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.sigkdd.org/kdd2010/&quot; rel=&quot;nofollow&quot;&gt;KDD 2010&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-08-19T13:40:35.720" CommunityOwnedDate="2010-08-19T13:45:00.000" />
  <row Id="1908" PostTypeId="1" CreationDate="2010-08-19T13:45:36.037" Score="3" ViewCount="179" Body="&lt;p&gt;What are the most significant annual Machine Learning conferences?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rules:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;One conference per answer&lt;/li&gt;&#xA;&lt;li&gt;Include a link to the conference &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="5" LastEditorUserId="877" LastEditorDisplayName="" LastEditDate="2010-08-23T15:27:33.560" LastActivityDate="2010-09-10T23:50:38.213" CommunityOwnedDate="2010-08-19T13:45:36.037" Title="Machine Learning conferences?" Tags="&lt;machine-learning&gt;&lt;conferences&gt;" AnswerCount="10" FavoriteCount="2" />
  <row Id="1909" PostTypeId="2" ParentId="1904" CreationDate="2010-08-19T13:46:41.337" Score="4" ViewCount="" Body="&lt;p&gt;UseR!&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.r-project.org/conferences.html&quot; rel=&quot;nofollow&quot;&gt;List of previous and upcoming R conferences on r-project&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Related Links:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.warwick.ac.uk/statsdept/useR-2011/&quot; rel=&quot;nofollow&quot;&gt;2011: University of Warwick, Coventry, UK&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.r-bloggers.com/RUG/?p=180&quot; rel=&quot;nofollow&quot;&gt;Videos of some keynote speakers from 2010&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-08-20T03:23:15.937" LastActivityDate="2010-08-20T03:23:15.937" CommunityOwnedDate="2010-08-19T13:46:41.337" CommentCount="2" />
  <row Id="1910" PostTypeId="2" ParentId="1908" CreationDate="2010-08-19T13:47:24.127" Score="2" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/ICML&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;ICML&lt;/strong&gt;&lt;/a&gt; (International Conference on Machine Learning)&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://icml2010.haifa.il.ibm.com/&quot; rel=&quot;nofollow&quot;&gt;ICML 2010&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-08-19T13:47:24.127" CommunityOwnedDate="2010-08-19T13:47:24.127" />
  <row Id="1911" PostTypeId="2" ParentId="1883" CreationDate="2010-08-19T14:13:27.077" Score="5" ViewCount="" Body="&lt;p&gt;In 1960 most people doing statistics were calculating with a four-function manual calculator or a slide rule or by hand; mainframe computers were just beginning to run some programs in Algol and Fortran; graphical output devices were rare and crude.  Because of these limitations, Bayesian analysis was considered formidably difficult due to the calculations required.  Databases were managed on punch cards and computer tape drives limited to a few megabytes.  Statistical education focused initially on learning formulas for t-testing and ANOVA.  Statistical practice usually did not go beyond such routine hypothesis testing (although some brilliant minds had just begun to exploit computers for deeper analysis, as exemplified by Mosteller &amp;amp; Wallace's book on the Federalist papers, for instance).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I recounted this well-known history as a reminder that &lt;em&gt;all&lt;/em&gt; of statistics has undergone a revolution due to the rise and spread of computing power during this last half century, a revolution that has made possible almost every other innovation in statistics during that time (with the notable exception of Tukey's pencil-and-paper EDA methods, as Thylacoleo has already observed).&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-08-19T14:13:27.077" CommunityOwnedDate="2010-08-19T14:13:27.077" />
  <row Id="1912" PostTypeId="1" AcceptedAnswerId="1959" CreationDate="2010-08-19T14:15:28.693" Score="3" ViewCount="178" Body="&lt;p&gt;&lt;a href=&quot;http://gking.harvard.edu/&quot; rel=&quot;nofollow&quot;&gt;Gary King&lt;/a&gt; made the &lt;a href=&quot;http://twitter.com/kinggary/status/21513150698&quot; rel=&quot;nofollow&quot;&gt;following statement on Twitter&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;scale invariance sounds cool but is&#xA;  usually statisticians shirking&#xA;  responsibility &amp;amp; losing power by&#xA;  neglecting subject matter info&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;What is an example of this phenomena, where &lt;a href=&quot;http://www.statistics.com/resources/glossary/s/scaleinv.php&quot; rel=&quot;nofollow&quot;&gt;scale invariance&lt;/a&gt; causes a loss of power?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Edit:&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://twitter.com/kinggary/status/21624260888&quot; rel=&quot;nofollow&quot;&gt;Gary responded&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;not [statistical] power, but scale invariance&#xA;  loses the power that can be extracted&#xA;  from knowledge of the substance&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Now this makes more sense.  How can scale invariance cause a loss of explanatory power from the resulting analysis?&lt;/p&gt;&#xA;" OwnerUserId="5" LastEditorUserId="5" LastEditorDisplayName="" LastEditDate="2010-08-20T15:19:17.757" LastActivityDate="2010-08-27T15:21:05.230" Title="Why can scale invariance cause a loss of explanatory power?" Tags="&lt;scale-invariance&gt;" AnswerCount="3" CommentCount="7" />
  <row Id="1913" PostTypeId="2" ParentId="1904" CreationDate="2010-08-19T14:38:51.867" Score="4" ViewCount="" Body="&lt;p&gt;In terms of overall breadth, I would say that the ASA/IMS Joint Statistical Meetings are the most significant.  Next year, the statisticians are taking their talents to South Beach...or &lt;a href=&quot;http://www.amstat.org/meetings/jsm/2011/index.cfm&quot; rel=&quot;nofollow&quot;&gt;Miami Beach&lt;/a&gt; is more correct.  I just couldn't help to use that line from Lebron James' infamous press conference.  Having said that, I prefer smaller conferences like the UseR! conferences, ICORS (robust statistics), etc.&lt;/p&gt;&#xA;" LastActivityDate="2010-08-19T14:38:51.867" CommunityOwnedDate="2010-08-19T14:38:51.867" />
  <row Id="1914" PostTypeId="1" CreationDate="2010-08-19T15:13:49.887" Score="3" ViewCount="102" Body="&lt;p&gt;I doing spatial bayesian data analysis, I am assuming a no-nugget exponential covariance. I have tried a variety of priors for the sills and range parameters (gamma, inverse gamma etc.) , unfortunately the convergence diagonstics are typically horrible. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am wondering how to figure out the poor mixing I observe, is there something I can do to make the MCMC chain behave better?&lt;/p&gt;&#xA;" OwnerUserId="1004" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-18T22:00:39.013" LastActivityDate="2010-09-18T22:00:39.013" Title="Choice for priors for exponential spatial covariance" Tags="&lt;bayesian&gt;&lt;mcmc&gt;&lt;spatial&gt;" AnswerCount="1" />
  <row Id="1915" PostTypeId="1" AcceptedAnswerId="1917" CreationDate="2010-08-19T16:09:23.430" Score="5" ViewCount="394" Body="&lt;p&gt;Hello,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am interested in running Newman's &lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1482622/&quot; rel=&quot;nofollow&quot;&gt;modularity clustering&lt;/a&gt; algorithm on a large graph. If you can point me to a library (or R package, etc) that implements it I would be most grateful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;best ~lara&lt;/p&gt;&#xA;" OwnerUserId="1007" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-08-20T08:27:54.520" LastActivityDate="2010-08-20T08:27:54.520" Title="Newman's modularity clustering for graphs" Tags="&lt;clustering&gt;&lt;networks&gt;&lt;data-visualization&gt;&lt;partitioning&gt;&lt;igraph&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="1916" PostTypeId="2" ParentId="1914" CreationDate="2010-08-19T16:31:49.917" Score="1" ViewCount="" Body="&lt;p&gt;Diggle and Ribeiro discuss this in their book (&quot;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0387329072&quot; rel=&quot;nofollow&quot;&gt;Model-based Geostatistics&lt;/a&gt;&quot;): see section 5.4.2.  They quote some research suggesting that re-parameterization might help a little.  For an exponential model (a Matern model with kappa = 1/2) this research suggests using the equivalent of log(sill/range) and log(range).  Diggle and Ribeiro themselves recommend a profile likelihood method to investigate the log-likelihood surface.  Their software is implemented in the R package &lt;a href=&quot;http://gbi.agrsci.dk/~ofch/geoRglm/&quot; rel=&quot;nofollow&quot;&gt;geoRglm&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Have you looked at an experimental variogram to check that a zero nugget and an exponential shape are appropriate?&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-08-20T08:27:30.700" LastActivityDate="2010-08-20T08:27:30.700" CommentCount="2" />
  <row Id="1917" PostTypeId="2" ParentId="1915" CreationDate="2010-08-19T16:59:14.233" Score="4" ViewCount="" Body="&lt;p&gt;The &lt;a href=&quot;http://cran.r-project.org/web/packages/igraph/index.html&quot; rel=&quot;nofollow&quot;&gt;igraph&lt;/a&gt; library implements some algorithms for community structure based on Newman's optimization of modularity.  You can consult the &lt;a href=&quot;http://cran.r-project.org/web/packages/igraph/igraph.pdf&quot; rel=&quot;nofollow&quot;&gt;reference manual&lt;/a&gt; for details and citations.&lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-08-19T16:59:14.233" />
  <row Id="1918" PostTypeId="2" ParentId="1866" CreationDate="2010-08-19T17:13:01.420" Score="1" ViewCount="" Body="&lt;p&gt;I'm not sure you need simulation for a simple regression model.  For example, see the paper &lt;a href=&quot;http://www.bobwheeler.com/statistics/Papers/PortablePower.PDF&quot; rel=&quot;nofollow&quot;&gt;Portable Power&lt;/a&gt;.  For more complex models, specifically mixed effects, the &lt;a href=&quot;http://cran.r-project.org/web/packages/pamm/index.html&quot; rel=&quot;nofollow&quot;&gt;pamm&lt;/a&gt; package in R performs power analyses through simulations.  Also see Todd Jobe's &lt;a href=&quot;http://www.unc.edu/~toddjobe/blog/2009/09/power-analysis-for-mixed-effect-models.html&quot; rel=&quot;nofollow&quot;&gt;post&lt;/a&gt; which has R code for simulation.&lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-08-19T17:13:01.420" />
  <row Id="1919" PostTypeId="2" ParentId="1904" CreationDate="2010-08-19T17:14:39.123" Score="0" ViewCount="" Body="&lt;p&gt;Not a &quot;statistics&quot; conference in the technical sense, but Predictive Analytics World is a case study conference on how companies are using predictive and other analytics in theis businesses.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.predictiveanalyticsworld.com/&quot; rel=&quot;nofollow&quot;&gt;Predictive Analytics World&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="11" LastActivityDate="2010-08-19T17:14:39.123" CommunityOwnedDate="2010-08-19T17:14:39.123" CommentCount="2" />
  <row Id="1920" PostTypeId="2" ParentId="1904" CreationDate="2010-08-19T17:19:52.763" Score="0" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://www.kdd.org/kdd2010/index.shtml&quot; rel=&quot;nofollow&quot;&gt;ACM SIGKDD 2010&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://kdd.org/kdd/2011/&quot; rel=&quot;nofollow&quot;&gt;KDD 2011 in San Diego&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="11" LastActivityDate="2010-08-19T17:19:52.763" CommunityOwnedDate="2010-08-19T17:19:52.763" CommentCount="1" />
  <row Id="1921" PostTypeId="2" ParentId="1863" CreationDate="2010-08-19T17:24:19.183" Score="4" ViewCount="" Body="&lt;p&gt;I just want to emphasize the importance of &lt;em&gt;not&lt;/em&gt; analyzing accuracies on the proportion scale. While lamentably pervasive across a number of disciplines, this practice can yield frankly incorrect conclusions. See: &lt;a href=&quot;http://dx.doi.org/10.1016/j.jml.2007.11.004&quot; rel=&quot;nofollow&quot;&gt;http://dx.doi.org/10.1016/j.jml.2007.11.004&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As John Christie notes, the best way to approach analysis of accuracy data is a mixed effects model using the binomial link and participants as a random effect, eg:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#R code&#xA;library(lme4)&#xA;fit = lmer(&#xA;    formula = acc ~ my_IV + (1|participant)&#xA;    , family = 'binomial'&#xA;    , data = my_data&#xA;)&#xA;print(fit)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Note that &quot;my_data&quot; should be the raw, trial-by-trial data such that &quot;acc&quot; is either 1 for accurate trials or 0 for inaccurate trials. That is, data should &lt;em&gt;not&lt;/em&gt; be aggregated to proportions before analysis.&lt;/p&gt;&#xA;" OwnerUserId="364" LastEditorUserId="364" LastEditorDisplayName="" LastEditDate="2010-08-19T17:43:34.613" LastActivityDate="2010-08-19T17:43:34.613" CommentCount="1" />
  <row Id="1922" PostTypeId="2" ParentId="1862" CreationDate="2010-08-19T17:28:18.883" Score="5" ViewCount="" Body="&lt;p&gt;There's a nice paper on visualization techniques you might use by Michael Friendly:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.math.yorku.ca/SCS/vcd/vcdstory.pdf&quot; rel=&quot;nofollow&quot;&gt;Visualizing Categorical Data: Data, Stories, and Pictures&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;(Actually, there's a whole &lt;a href=&quot;http://books.google.com/books?id=eG0phz62f1cC&amp;amp;lpg=PP1&amp;amp;dq=michael%20friendly%20visualizing%20categorical%20data&amp;amp;pg=PP1#v=onepage&amp;amp;q&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;book&lt;/a&gt; devoted to this by the same author.)  The &lt;a href=&quot;http://cran.r-project.org/web/packages/vcd/index.html&quot; rel=&quot;nofollow&quot;&gt;vcd&lt;/a&gt; package in R implements many of these techniques.&lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-08-19T17:28:18.883" CommentCount="1" />
  <row Id="1923" PostTypeId="1" AcceptedAnswerId="2509" CreationDate="2010-08-19T17:56:58.550" Score="4" ViewCount="208" Body="&lt;p&gt;Suppose instead of maximizing likelihood I maximize some other function g. Like likelihood, this function decomposes over x's (ie, g({x1,x2})=g({x1})g({x2}), and &quot;maximum-g&quot; estimator is consistent. How do I compute asymptotic variance of this estimator?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;b&gt;Update 8/24/10&lt;/b&gt;: Percy Liang goes through derivation of asymptotic variance in a similar setting in &lt;a href=&quot;http://www.eecs.berkeley.edu/~pliang/papers/asymptotics-icml2008.pdf&quot; rel=&quot;nofollow&quot;&gt;An asymptotic analysis of generative, discriminative, and pseudolikelihood estimators.&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;b&gt;Update 9/14/10&lt;/b&gt;: Most &lt;a href=&quot;http://yaroslavvb.com/upload/efficiency-vandervaart.djvu&quot; rel=&quot;nofollow&quot;&gt;useful theorem&lt;/a&gt; seems to be from Van der Vaart's &quot;Asymptotic Statistics&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Under some general regularity conditions, distribution of this estimator approaches normal centered around its expected value $\theta_0$ with covariance matrix&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{\ddot{g}(\theta_0)^{-1} E[\dot{g}\dot{g}^T] \ddot{g}(\theta_0)^{-1}}{n}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $\ddot{g}$ is a matrix of second derivatives, $\dot{g}$ is gradient, $n$ is number of samples&lt;/p&gt;&#xA;" OwnerUserId="511" LastEditorUserId="511" LastEditorDisplayName="" LastEditDate="2010-09-15T05:54:47.290" LastActivityDate="2010-09-15T05:54:47.290" Title="How to compute efficiency?" Tags="&lt;estimation&gt;&lt;efficiency&gt;&lt;asymptotics&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="1924" PostTypeId="2" ParentId="1923" CreationDate="2010-08-19T18:29:24.990" Score="1" ViewCount="" Body="&lt;p&gt;The consistency and asymptotic normality of the maximum likelihood estimator is demonstrated using some regularity conditions on the likelihood function. The wiki link on &lt;a href=&quot;http://en.wikipedia.org/wiki/Maximum_likelihood#Consistency&quot; rel=&quot;nofollow&quot;&gt;consistency&lt;/a&gt; and &lt;a href=&quot;http://en.wikipedia.org/wiki/Maximum_likelihood#Asymptotic_normality&quot; rel=&quot;nofollow&quot;&gt;asymptotic normality&lt;/a&gt; has the conditions necessary to prove these properties. The conditions at the wiki may be stronger than what you need as they are used to prove asymptotic normality whereas you simply want to compute the variance of the estimator.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am guessing that if your function satisfies the same conditions then the proof will carry over to your function as well. If not then we need to know one or both of the following: (a) the specific condition that $g(.)$ does not satisfy from the list at the wiki and (b) the specifics of $g(.)$ to give a better answer to your question.&lt;/p&gt;&#xA;" LastActivityDate="2010-08-19T18:29:24.990" CommentCount="7" />
  <row Id="1925" PostTypeId="2" ParentId="203" CreationDate="2010-08-19T18:30:39.777" Score="0" ViewCount="" Body="&lt;p&gt;Proportional odds ratio model is better then t-test for Likert item scale.&lt;/p&gt;&#xA;" OwnerUserId="419" LastActivityDate="2010-08-19T18:30:39.777" CommentCount="1" />
  <row Id="1926" PostTypeId="2" ParentId="1904" CreationDate="2010-08-19T18:43:16.163" Score="2" ViewCount="" Body="&lt;p&gt;For biostatistics the largest US conferences are the meetings of the local sections of the International Biometrics Society (IBS):&lt;br&gt;&#xA; - ENAR for the Eastern region: &lt;a href=&quot;http://www.enar.org/meetings.cfm&quot; rel=&quot;nofollow&quot;&gt;http://www.enar.org/meetings.cfm&lt;/a&gt;&lt;br&gt;&#xA; - WNAR for the Western region: &lt;a href=&quot;http://www.wnar.org/&quot; rel=&quot;nofollow&quot;&gt;http://www.wnar.org/&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of these ENAR is by far larger.&lt;/p&gt;&#xA;" OwnerUserId="279" LastActivityDate="2010-08-19T18:43:16.163" CommunityOwnedDate="2010-08-19T18:43:16.163" />
  <row Id="1927" PostTypeId="1" CreationDate="2010-08-19T19:36:01.337" Score="15" ViewCount="403" Body="&lt;p&gt;If so, what?&#xA;If not, why not?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a sample on the line, the median minimizes the total absolute deviation. It would seem natural to extend the definition to R2, etc., but I've never seen it. But then, I've been out in left field for a long time.&lt;/p&gt;&#xA;" OwnerUserId="1011" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-08-20T08:25:44.230" LastActivityDate="2010-10-19T06:16:41.847" Title="Is there an accepted definition for the median of a sample on the plane, or higher ordered spaces?" Tags="&lt;multivariable&gt;&lt;median&gt;" AnswerCount="7" FavoriteCount="1" />
  <row Id="1928" PostTypeId="2" ParentId="1927" CreationDate="2010-08-19T19:48:16.567" Score="9" ViewCount="" Body="&lt;p&gt;I'm not sure there is one accepted definition for a multivariate median.  The one I'm familiar with is &lt;a href=&quot;http://cgm.cs.mcgill.ca/~athens/Geometric-Estimators/oja.html&quot; rel=&quot;nofollow&quot;&gt;Oja's median point&lt;/a&gt;, which minimizes the sum of volumes of simplices formed over subsets of points.  (See the link for a technical definition.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Update: The site referenced for the Oja definition above also has a nice paper covering a number of definitions of a multivariate median:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://cgm.cs.mcgill.ca/~athens/Papers/depth.pdf&quot; rel=&quot;nofollow&quot;&gt;Geometric Measures of Data Depth&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="251" LastEditorUserId="251" LastEditorDisplayName="" LastEditDate="2010-08-19T19:58:31.783" LastActivityDate="2010-08-19T19:58:31.783" CommentCount="1" />
  <row Id="1929" PostTypeId="2" ParentId="1927" CreationDate="2010-08-19T19:53:51.467" Score="0" ViewCount="" Body="&lt;p&gt;I do not know if any such definition exists but I will try and extend the &lt;a href=&quot;http://en.wikipedia.org/wiki/Median#An_optimality_property&quot; rel=&quot;nofollow&quot;&gt;standard definition of the median&lt;/a&gt; to $R^2$. I will use the following notation:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$X$, $Y$: the random variables associated with the two dimensions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$m_x$, $m_y$: the corresponding medians.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$f(x,y)$: the joint pdf for our random variables&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To extend the definition of the median to $R^2$, we choose $m_x$ and $m_y$ to minimize the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$E(|(x,y) - (m_x,m_y)|$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem now is that we need a definition for what we mean by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$|(x,y) - (m_x,m_y)|$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The above is in a sense a distance metric and several possible candidate definitions are possible. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Euclidean_metric&quot; rel=&quot;nofollow&quot;&gt;Eucliedan Metric&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$|(x,y) - (m_x,m_y)| = \sqrt{(x-m_x)^2 + (y-m_y)^2}$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Computing the median under the euclidean metric will require computing the expectation of the above with respect to the joint density $f(x,y)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Taxicab_geometry&quot; rel=&quot;nofollow&quot;&gt;Taxicab Metric&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$|(x,y) - (m_x,m_y)| = |x-m_x| + |y-m_y|$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Computing the median in the case of the taxicab metric involves computing the median of $X$ and $Y$ separately as the metric is separable in $x$ and $y$. &lt;/p&gt;&#xA;" LastActivityDate="2010-08-19T19:53:51.467" CommentCount="8" />
  <row Id="1930" PostTypeId="2" ParentId="1850" CreationDate="2010-08-19T20:52:10.220" Score="6" ViewCount="" Body="&lt;p&gt;Both Cohen's d and Hedges' g pool variances on the assumption of equal population variances, but g pools using n - 1 for each sample instead of n, which provides a better estimate, especially the smaller the sample sizes.  Both d and g are somewhat positively biased, but only negligibly for moderate or larger sample sizes.  The bias is reduced using g*. The d by Glass does not assume equal variances, so it uses the sd of a control group or baseline comparison group as the standardizer for the difference between the two means.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These effect sizes and Cliff's and other nonparametric effect sizes are discussed in detail in my book:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Grissom, R. J., &amp;amp; Kim, J, J. (2005). Effect sizes for research: A broad practical approach. Mahwah, NJ: Erlbaum.&lt;/p&gt;&#xA;" OwnerUserId="1013" LastActivityDate="2010-08-19T20:52:10.220" />
  <row Id="1931" PostTypeId="2" ParentId="1927" CreationDate="2010-08-19T20:58:59.157" Score="7" ViewCount="" Body="&lt;p&gt;There are distinct ways to generalize the concept of median to higher dimensions.  One not yet mentioned, but which was proposed long ago, is to construct a convex hull, peel it away, and iterate for as long as you can: what's left in the last hull is a set of points that are all candidates to be &quot;medians.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://srab.cancer.gov/headbang/&quot; rel=&quot;nofollow&quot;&gt;&quot;Head-banging&quot;&lt;/a&gt; is another more recent attempt (c. 1980) to construct a robust center to a 2D point cloud.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The principal reason why there are multiple distinct generalizations and no one obvious solution is that R1 can be ordered but R2, R3, ... cannot be.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-08-19T20:58:59.157" CommentCount="6" />
  <row Id="1932" PostTypeId="2" ParentId="1908" CreationDate="2010-08-19T21:44:04.807" Score="2" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://nips.cc&quot; rel=&quot;nofollow&quot;&gt;NIPS (Neural Information Processing Systems)&lt;/a&gt;. It's actually an intersection of machine learning, and application areas such as speech/language, vision, neuro-science, and other related areas.&lt;/p&gt;&#xA;" OwnerUserId="881" LastActivityDate="2010-08-19T21:44:04.807" CommunityOwnedDate="2010-08-19T21:44:04.807" />
  <row Id="1933" PostTypeId="2" ParentId="1826" CreationDate="2010-08-19T21:50:48.220" Score="1" ViewCount="" Body="&lt;p&gt;Since you don't have access to the test data at the time of training, and you want your model to do well on the unseen test data, you &quot;pretend&quot; that you have access to some test data by repeatedly subsampling a small part of your training data, hold out this set while training the model, and then treating the held out set as a proxy to the test data (and choose model parameters that give best performance on the held out data). You hope that by randomly sampling various subsets from the training data, you might make them look like the test data (in the average behavior sense), and therefore the learned model parameters will be good for the test data as well (i.e., your model generalizes well for unseen data).&lt;/p&gt;&#xA;" OwnerUserId="881" LastEditorUserId="881" LastEditorDisplayName="" LastEditDate="2010-08-19T23:49:54.203" LastActivityDate="2010-08-19T23:49:54.203" />
  <row Id="1934" PostTypeId="2" ParentId="1228" CreationDate="2010-08-19T22:08:44.573" Score="1" ViewCount="" Body="&lt;p&gt;Thomas Ryan (&quot;Statistical Methods for Quality Improvement&quot;, Wiley, 1989) describes several procedures.  He tends to try to reduce all control charting to the Normal case, so his procedures are not as creative as they could be, but he claims they work pretty well.  One is to treat the values as Binomial data and use the ArcSin transformation, then run standard CUSUM charts.  Another is to view the values as Poisson data and use the square root transformation, then again run a CUSUM chart.  For these approaches, which are intended for process quality control, you're supposed to know the number of potentially exposed individuals during each period.  If you don't, you probably have to go with the Poisson model.  Given that the infections are rare, the square root transformation sets your upper control limit a tiny bit above (u/2)^2 where typically u = 3 (corresponding to the usual 3-SD UCL in a Normal chart), whence any count of Ceiling((3/2)^2) = 3 or greater would trigger an out-of-control condition.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One wonders whether control charting is the correct conceptual model for your problem, though.  You're not really running any kind of quality control process here: you probably know, on scientific grounds, when the infection rate is alarming.  You might know, as a hypothetical example, that fewer than ten infections over a week-long period is rarely a harbinger of an outbreak.  Why not set your upper limit on this kind of basis rather than employing an almost useless statistical limit?&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-08-19T22:08:44.573" CommentCount="1" />
  <row Id="1935" PostTypeId="1" CreationDate="2010-08-20T00:10:33.740" Score="6" ViewCount="339" Body="&lt;p&gt;Despite several attempts at reading about bootstrapping, I seem to always hit a brick wall. I wonder if anyone can give a reasonably non-technical definition of bootstrapping?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know it is not possible in this forum to provide enough detail to enable me to fully understand it, but a gentle push in the right direction with the main goal and mechanism of bootstrapping would be &lt;em&gt;much&lt;/em&gt; appreciated! Thanks.&lt;/p&gt;&#xA;" OwnerUserId="561" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-17T20:23:23.940" LastActivityDate="2010-09-17T20:23:23.940" Title="Whither bootstrapping - can someone provide a simple explanation to get me started?" Tags="&lt;nonparametric&gt;&lt;bootstrap&gt;" AnswerCount="7" FavoriteCount="1" />
  <row Id="1936" PostTypeId="2" ParentId="1908" CreationDate="2010-08-20T00:16:09.237" Score="1" ViewCount="" Body="&lt;p&gt;One of the only machine learning conferences for those in Australia and New Zealand is:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://ai10.cis.unisa.edu.au/&quot; rel=&quot;nofollow&quot;&gt;23rd Australasian Joint Conference on Artificial Intelligence&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;It's held in Adelaide this year.&lt;/p&gt;&#xA;" OwnerUserId="530" LastActivityDate="2010-08-20T00:16:09.237" CommunityOwnedDate="2010-08-20T00:16:09.237" />
  <row Id="1937" PostTypeId="2" ParentId="1935" CreationDate="2010-08-20T00:20:49.803" Score="1" ViewCount="" Body="&lt;p&gt;The wiki on &lt;a href=&quot;http://en.wikipedia.org/wiki/Bootstrapping_%28statistics%29&quot; rel=&quot;nofollow&quot;&gt;bootstrapping&lt;/a&gt; gives the following description:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Bootstrapping allows one to gather many alternative versions of the single statistic that would ordinarily be calculated from one sample. For example, assume we are interested in the height of people worldwide. As we cannot measure all the population, we sample only a small part of it. From that sample only one value of a statistic can be obtained, i.e one mean, or one standard deviation etc., and hence we don't see how much that statistic varies. When using bootstrapping, we randomly extract a new sample of n heights out of the N sampled data, where each person can be selected at most t times. By doing this several times, we create a large number of datasets that we might have seen and compute the statistic for each of these datasets. Thus we get an estimate of the distribution of the statistic. The key to the strategy is to create alternative versions of data that &quot;we might have seen&quot;.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I will provide more detail if you can clarify what part of the above description you do not understand.&lt;/p&gt;&#xA;" LastActivityDate="2010-08-20T00:20:49.803" />
  <row Id="1938" PostTypeId="2" ParentId="1904" CreationDate="2010-08-20T00:23:03.980" Score="1" ViewCount="" Body="&lt;p&gt;The main regular conference in Australia is the &quot;Australian Statistics Conference&quot;, held every second year. The next one is &lt;a href=&quot;http://www.promaco.com.au/2010/asc/&quot; rel=&quot;nofollow&quot;&gt;ASC 2010&lt;/a&gt;, to be held in Western Australia in December.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-08-20T00:23:03.980" CommunityOwnedDate="2010-08-20T00:23:03.980" />
  <row Id="1939" PostTypeId="2" ParentId="1935" CreationDate="2010-08-20T00:35:28.433" Score="2" ViewCount="" Body="&lt;p&gt;The Wikipedia entry on Bootstrapping is actually very good:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Bootstrapping_%28statistics%29&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Bootstrapping_%28statistics%29&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The most common reason bootstrapping is applied is when the form of the underlying distribution from which a sample is taken is unknown. Traditionally statisticians assume a normal distribution (for very good reasons related to the central limit theorem), but statistics (such as the standard deviation, confidence intervals, power calculations etc) estimated via normal distribution theory are only strictly valid if the underlying population distribution is normal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By repeatedly re-sampling the sample itself, bootstrapping enables estimates that are distribution independent. Traditionally each &quot;resample&quot; of the original sample randomly selects the same number of observations as in the original sample. However these are selected with replacement. If the sample has N observations, each bootstrap resample will have N observations, with many of the original sample repeated and many excluded. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The parameter of interest (eg. odds ratio etc) can then be estimated from each bootstrapped sample. Repeating the bootstrap say 1000 times allows an estimate of the &quot;median&quot; and 95% confidence interval on the statistic (eg odds ratio) by selecting the 2.5th, 50th and 97.5th percentile.&lt;/p&gt;&#xA;" OwnerUserId="521" LastActivityDate="2010-08-20T00:35:28.433" />
  <row Id="1940" PostTypeId="2" ParentId="1935" CreationDate="2010-08-20T01:08:12.280" Score="5" ViewCount="" Body="&lt;p&gt;The American Scientist recently had a nice article by Cosma Shalizi on &lt;a href=&quot;http://www.americanscientist.org/issues/pub/2010/3/the-bootstrap/1&quot; rel=&quot;nofollow&quot;&gt;the bootstrap&lt;/a&gt; which is fairly easy reading and gives you the essentials to grasp the concept.  &lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-08-20T01:08:12.280" CommentCount="1" />
  <row Id="1941" PostTypeId="2" ParentId="1781" CreationDate="2010-08-20T01:47:20.863" Score="2" ViewCount="" Body="&lt;p&gt;I have not full internalized the issue of matrix interference but here is one approach. Let:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Y$ be a vector that represents the concentration of all the target compounds in the undiluted sample.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Z$ be the corresponding vector in the diluted sample.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$d$ be the dilution factor i.e., the sample is diluted $d$:1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Our model is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Y \sim N(\mu,\Sigma)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Z = \frac{Y}{d} + \epsilon$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\epsilon \sim N(0,\sigma^2\ I)$ represents the error due to dilution errors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, it follows that:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Z \sim N(\frac{\mu}{d}, \Sigma + \sigma^2\ I)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Denote the above distribution of $Z$ by $f_Z(.)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let $O$ be the observed concentrations and $\tau$ represent the test instrument's threshold below which it cannot detect a compound. Then, for the $i^{th}$ compound we have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$O_i = Z_i I(Z_i &gt; \tau) + 0 I(Z_i \le \tau)$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Without loss of generality let the first $k$ compounds be such that they are below the threshold. Then the likelihood function can be written as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$L(O_1, ... O_k, O_{k+1},...O_n |- ) = [\prod_{i=1}^{i=k}{Pr(Z_i \le \tau)}]  [\prod_{i=k+1}^{i=n}{f(O_i |-)}]$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;where&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$f(O_i |-) = \int_{j\neq i}{f_Z(O_i|-) I(O_i &gt; \tau)}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Estimation is then a matter of using either maximum likelihood or bayesian ideas. I am not sure how tractable the above is but I hope it gives you some ideas.&lt;/p&gt;&#xA;" LastActivityDate="2010-08-20T01:47:20.863" CommentCount="1" />
  <row Id="1942" PostTypeId="1" CreationDate="2010-08-20T02:25:32.253" Score="1" ViewCount="103" Body="&lt;p&gt;What are the most significant annual conferences focusing on quantitative methods in psychology?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This could include but is not limited to psychometrics, mathematical psychology, and statistical methods in psychology.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rules:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;One conference per answer&lt;/li&gt;&#xA;&lt;li&gt;Include a link to the conference&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-08-22T03:05:50.557" CommunityOwnedDate="2010-08-20T02:25:32.253" Title="Quantitative methods and statistics conferences in psychology?" Tags="&lt;conferences&gt;&lt;psychometrics&gt;&lt;psychology&gt;" AnswerCount="3" />
  <row Id="1943" PostTypeId="2" ParentId="1844" CreationDate="2010-08-20T06:46:21.083" Score="3" ViewCount="" Body="&lt;p&gt;If I'm not mistaken, the log-rank estimator you reference is also known as the Pike estimator.  I believe it's generally recommended for HR &amp;lt; 3 because it exhibits less bias in that range.  The following paper may be of interest (note that the paper refers to it as O/E):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.jstor.org/pss/2530564&quot; rel=&quot;nofollow&quot;&gt;Estimation of the Proportional Hazard in Two-Treatment-Group Clinical Trials&lt;/a&gt; (Bernstein, Anderson, Pike)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;[...] The O/E method is biased but, within the range of values of the ratio of the hazard rates of interest in clinical trials, it is more efficient in terms of mean square error than either CML or the Mantel-Haenszel method for all but the largest trials. The Mantel-Haenszel method is minimally biased, gives answers very close to those obtained using CML, and may be used to provide satisfactory approximate confidence intervals.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-08-20T06:46:21.083" CommentCount="3" />
  <row Id="1944" PostTypeId="1" AcceptedAnswerId="1945" CreationDate="2010-08-20T08:31:16.120" Score="2" ViewCount="137" Body="&lt;p&gt;This may be a stupid question but... is there a specific name for normalizing some data so that it has mean=0 and sd=1? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or do I just say &quot;data was normalized to have mean=0 and sd=1&quot;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;thanks&#xA;nico&lt;/p&gt;&#xA;" OwnerUserId="582" LastEditorUserId="582" LastEditorDisplayName="" LastEditDate="2010-08-20T11:03:54.197" LastActivityDate="2010-08-20T11:03:54.197" Title="What is the name of this normalization?" Tags="&lt;terminology&gt;&lt;normalization&gt;" AnswerCount="2" CommentCount="4" FavoriteCount="1" />
  <row Id="1945" PostTypeId="2" ParentId="1944" CreationDate="2010-08-20T08:42:39.977" Score="6" ViewCount="" Body="&lt;p&gt;The quantity $z = \frac{X - \mu}{\sigma}$ is a &lt;a href=&quot;http://en.wikipedia.org/wiki/Standard_score&quot; rel=&quot;nofollow&quot;&gt;standard score&lt;/a&gt;.  So, standardization is a common way to refer to it.  &lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-08-20T08:42:39.977" CommentCount="2" />
  <row Id="1946" PostTypeId="2" ParentId="1944" CreationDate="2010-08-20T08:45:26.740" Score="3" ViewCount="" Body="&lt;p&gt;I think it is just called z-score.&lt;/p&gt;&#xA;" OwnerUserId="88" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-08-20T09:24:29.320" LastActivityDate="2010-08-20T09:24:29.320" CommentCount="4" />
  <row Id="1947" PostTypeId="2" ParentId="1927" CreationDate="2010-08-20T09:52:03.863" Score="12" ViewCount="" Body="&lt;p&gt;As &lt;strong&gt;@Ars&lt;/strong&gt; said there are no accepted definition (and this is a good point). There are general alternatives families of ways to generalize quantiles on $\mathbb{R}^d$, I think the most significant are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://projecteuclid.org/DPubS?service=UI&amp;amp;version=1.0&amp;amp;verb=Display&amp;amp;handle=euclid.aos/1176348670&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;Generalize quantile process&lt;/strong&gt;&lt;/a&gt; Let $P_n(A)$ be the empirical measure (=the proportion of observations in $A$). Then, with $\mathbb{A}$ a well chosen subset of the Borel sets in $\mathbb{R}^d$ and $\lambda$ a real valued measure,&#xA;you can define the empirical quantile function:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$U_n(t)=\inf (\lambda(A) : P_n(A)\geq t A\in\mathbb{A})$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose you can find one $A_{t}$ that gives you the minimum. Then the set (or an element of the set) $A_{1/2-\epsilon}\cap A_{1/2+\epsilon}$ gives you the median when $\epsilon$ is made small enough. The definition of the median is recovered when using $\mathbb{A}=(]-\infty,x] x\in\mathbb{R})$  and $\lambda(]-\infty,x])=x$. &lt;strong&gt;Ars&lt;/strong&gt; answer falls into that framework I guess...  &lt;strong&gt;tukey's half space location&lt;/strong&gt; may be obtained using $\mathbb{A}(a)=( H_{x}=(t\in \mathbb{R}^d :\; \langle a, t \rangle \leq x ) $ and  $\lambda(H_{x})=x$  (with $x\in \mathbb{R}$, $a\in\mathbb{R}^d$).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://projecteuclid.org/DPubS?service=UI&amp;amp;version=1.0&amp;amp;verb=Display&amp;amp;handle=euclid.aos/1031833659&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;variational definition and M-estimation&lt;/strong&gt;&lt;/a&gt;&#xA;The idea here is that the  $\alpha$-quantile $Q_{\alpha}$ of a random variable $Y$ in $\mathbb{R}$ can be defined through a variational equality. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;The most common definition is using the &lt;strong&gt;quantile regression function&lt;/strong&gt;   $\rho_{\alpha}$ (also known as pinball loss, guess why ? )  $Q_{\alpha}=arg\inf_{x\in \mathbb{R}}\mathbb{E}[\rho_{\alpha}(Y-x)]$. The case $\alpha=1/2$ gives $\rho_{1/2}(y)=|y|$ and you can generalize that to higher dimension using $l^1$ distances as done in &lt;strong&gt;@Srikant Answer&lt;/strong&gt;. This is theoretical median but gives you empirical median if you replace expectation by empirical expectation (mean).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;But &lt;a href=&quot;http://projecteuclid.org/DPubS?service=UI&amp;amp;version=1.0&amp;amp;verb=Display&amp;amp;handle=euclid.aos/1031833659&quot; rel=&quot;nofollow&quot;&gt;Kolshinskii&lt;/a&gt; proposes to use  Legendre-Fenchel transform: since $Q_{\alpha}=Arg\sup_s (s\alpha-f(s))$&#xA;where $f(s)=\frac{1}{2}\mathbb{E} [|s-Y|-|Y|+s]$ for $s\in \mathbb{R}$.&#xA;He gives a lot of deep reasons for that (see the paper ;)). Generalizing this to higher dimensions require working with a vectorial $\alpha$ and replacing $s\alpha$ by $\langle s,\alpha\rangle$ but you can take $\alpha=(1/2,\dots,1/2)$. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.ams.org/mathscinet-getitem?mr=55:26&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;Partial ordering&lt;/strong&gt;&lt;/a&gt; You can generalize the definition of quantiles in $\mathbb{R}^d$ as soon as you can create a partial order (with equivalence classes). &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Obviously there are bridges between the different formulations. They are not all obvious...&lt;/p&gt;&#xA;" OwnerUserId="223" LastEditorUserId="223" LastEditorDisplayName="" LastEditDate="2010-10-19T06:16:41.847" LastActivityDate="2010-10-19T06:16:41.847" CommunityOwnedDate="2010-10-19T06:16:41.847" CommentCount="1" />
  <row Id="1948" PostTypeId="2" ParentId="1927" CreationDate="2010-08-20T10:05:35.663" Score="3" ViewCount="" Body="&lt;p&gt;A definition that comes close to it, for unimodal distributions, is the tukey halfspace median &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://cgm.cs.mcgill.ca/~athens/Geometric-Estimators/halfspace.html&quot; rel=&quot;nofollow&quot;&gt;http://cgm.cs.mcgill.ca/~athens/Geometric-Estimators/halfspace.html&lt;/a&gt;&#xA;http://www.isical.ac.in/~statmath/html/publication/Tukey_tech_rep.pdf&lt;/p&gt;&#xA;" OwnerUserId="603" LastActivityDate="2010-08-20T10:05:35.663" />
  <row Id="1949" PostTypeId="2" ParentId="1519" CreationDate="2010-08-20T10:19:39.907" Score="3" ViewCount="" Body="&lt;p&gt;Really, these have been discredited long ago. For univariate Outliers, the optimal (most efficent) filter is median+/-$\delta \times$ MAD, or better yet (if you have access to R)&#xA;median+/-$\delta \times$ Qn (so you don't assume the underlying distribution to be symmetric),  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Qn estimator is implemented in package robustbase.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rousseeuw, P.J. and Croux, C. (1993) Alternatives to the Median&#xA;     Absolute Deviation, &lt;em&gt;Journal of the American Statistical&#xA;     Association&lt;/em&gt; *88*, 1273-1283.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Response to comment:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Two levels. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A) Philosophical.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Both the Dixon and Grub tests are only able to detect a particular type of (isolated) outliers. For the last 20-30 years the concept of outliers has involved unto &quot;any observation that departs from the main body of the data&quot;. Without further specification of what the particular departure is. This characterization-free approach renders the idea of building tests to detect outliers void. The emphasize shifted to the concept of estimators (a classical example of which is the median) that retain there values (i.e. are insensitive) even for large rate of contamination by outliers -such estimator is then said to be robust-  and the question of detecting outliers becomes void.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;B) Weakness,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can see that the Grub and Dixon tests easily break down: one can easily generated contaminated data that would pass either test like a bliss (i.e. without breaking the null).&#xA;This is particularly obvious in the Grubb test, because outliers will break down the mean and s.d. used in the construction of the test stat. It's less obvious in the Dixon, until one learns that order statistics are not robust to outliers either.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think you will find more explanation of these facts in papers oriented towards the general non-statistician audience such as the one cited above (I can also think of the Fast-Mcd paper by Rousseeuw). If you consult any recent book/intro to robust analysis, you will notice that neither Grubb nor Dixon are mentioned. &lt;/p&gt;&#xA;" OwnerUserId="603" LastEditorUserId="603" LastEditorDisplayName="" LastEditDate="2010-08-20T19:47:54.470" LastActivityDate="2010-08-20T19:47:54.470" CommentCount="1" />
  <row Id="1950" PostTypeId="2" ParentId="1935" CreationDate="2010-08-20T11:16:27.923" Score="1" ViewCount="" Body="&lt;p&gt;Bootstrap is essentially a simulation of repeating experiment; let's say you have a box with balls an want to obtain an average size of a ball -- so you draw some of them, measure and take a mean. Now you want to repeat it to get the distribution, for instance to get a standard deviation -- but you found out that someone stole the box.&lt;br&gt;&#xA;What can be done now is to use what you have -- this one series of measurements. The idea is to put the balls to the new box and simulate the original experiment by drawing the same number of balls with replacement -- both to have same sample size and some variability. Now this can be replicated many times to get a series of means which can be finally used to approximate the mean distribution. &lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-08-20T11:16:27.923" />
  <row Id="1951" PostTypeId="2" ParentId="1942" CreationDate="2010-08-20T12:28:09.067" Score="2" ViewCount="" Body="&lt;p&gt;The annual meeting of the Society for Computers in Psychology often features content on quantitative methods:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://sites.google.com/site/scipws/&quot; rel=&quot;nofollow&quot;&gt;http://sites.google.com/site/scipws/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="364" LastActivityDate="2010-08-20T12:28:09.067" CommunityOwnedDate="2010-08-20T12:28:09.067" />
  <row Id="1952" PostTypeId="2" ParentId="1935" CreationDate="2010-08-20T12:52:31.310" Score="2" ViewCount="" Body="&lt;p&gt;I like to think of it as follows: If you obtain a random sample data set from a population, then presumably that sample will have characteristics that roughly match that of the source population. So, if you're interested in obtaining confidence intervals on on a particular feature of the distribution, its skewness for example, you can treat the sample as a pseudo-population from which you can obtain many sets of random pseudo-samples, computing the value of the feature of interest in each. The assumption that the original sample roughly matches the population also means that you can obtain the pseudo-samples by sampling from the pseudo-population &quot;with replacement&quot; (eg. you sample a value, record it, then put it back; thus each value has a chance of being observed multiple times.). Sampling with replacement means that the computed value of the feature of interest will vary from pseudo-sample to pseudo-sample, yielding a distribution of values from which you can compute, say, the 2.5th and 97.5th percentiles to obtain the 95% confidence interval for the value of the feature of interest.&lt;/p&gt;&#xA;" OwnerUserId="364" LastActivityDate="2010-08-20T12:52:31.310" />
  <row Id="1953" PostTypeId="2" ParentId="1853" CreationDate="2010-08-20T13:42:01.620" Score="2" ViewCount="" Body="&lt;p&gt;L-moments might be useful here?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/L-moment&quot; rel=&quot;nofollow&quot;&gt;Wikipedia article&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.research.ibm.com/people/h/hosking/lmoments.html&quot; rel=&quot;nofollow&quot;&gt;The L-moments page (Jonathan R.M. Hosking, IBM Research)&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;They provide quantities analogous to conventional moments such as skewness and kurtosis, called the l-skewness and l-kurtosis. These have the advantage that they don't require calculation of high moments as they are computed from linear combinations of the data and defined as linear combinations of expected values of order statistics. This also means they are less sensitive to outliers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I believe you only need second-order moments to calculate their sample variances, which presumably you'd need for your test. Also their asymptotic distribution converges to a normal distribution much faster than conventional moments.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems the expressions for their sample variances get quite complicated (Elamir and Seheult 2004&lt;/a&gt;), but i know they've been programmed in downloadable packages for both R and Stata (available from their standard repositories), and maybe in other packages too for all i know. As your samples are independent once you've got the estimates and standard errors you could just plug them into a two-sample z-test if your sample sizes are &quot;large enough&quot; (Elamir and Seheult report some limited simulations that appear to show that 100 isn't large enough, but not what is). Or you could bootstrap the difference in l-skewness. The above properties suggest that may perform considerably better than bootstrapping based on the conventional skewness.&lt;/p&gt;&#xA;" OwnerUserId="449" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-08-20T13:53:11.397" LastActivityDate="2010-08-20T13:53:11.397" />
  <row Id="1954" PostTypeId="2" ParentId="1844" CreationDate="2010-08-20T13:54:04.360" Score="2" ViewCount="" Body="&lt;p&gt;There are actually several more methods and the choice often depends on whether you are most interested in looking for early differences, later differences or - as for the log-rank test &amp;amp; the Mantel-Haenszel test - give equal weight to all time points.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To the question at hand. The log-rank test is in fact a form of the Mantel-Haenszel test applied to survival data. The Mantel-Haenszel test is usually used to test for independence in stratified contingency tables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we try to apply the MH test to survival data, we can start by assuming that events at each failure time are independent. We then stratify by failure time. We use the MH methods for by making each failure time a strata. Not surprisingly they often give the same result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The exception occcurs when more than one event occurs simultaneously - multiple deaths at exactly the same time point. I can't remember how the treatment then differs. I think the log-rank test averages over the possible orderings of the tied failure times. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the log-rank test is the MH test for survival data and can deal with ties. I've never used the MH test for survival data.&lt;/p&gt;&#xA;" OwnerUserId="521" LastActivityDate="2010-08-20T13:54:04.360" />
  <row Id="1955" PostTypeId="1" CreationDate="2010-08-20T13:59:34.987" Score="4" ViewCount="115" Body="&lt;p&gt;&lt;em&gt;(Prompted to some extent by the answers already given by Shane and Srikant, I've rewritten this to try to clarify what I'm getting at, if only to myself.)&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose we have several similar systems, each with behaviour that approximates a continuous time Markov process. That is, there are some number of discrete states the system can be in and associated probabilities of transitioning from one state to another at any instant, depending solely on the current state. For now, consider the processes to be &lt;em&gt;stationary&lt;/em&gt;, ie the transition probabilities do not change over time, and unaffected by seasonality or other external considerations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unfortunately, we cannot measure the state of any system directly, but have instead to measure a proxy quantity, which varies with state but is &lt;em&gt;not&lt;/em&gt; discrete and is subject to various sources of noise and error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The principal question is this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Q1: Given two data sequences produced independently from two such systems, how can we decide whether the underlying Markov processes are the same?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, it &lt;em&gt;may&lt;/em&gt; be that the best way to approach this is as two separate problems:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Convert the imperfect proxy sequence into an idealised time series of (categorical) states&lt;/li&gt;&#xA;&lt;li&gt;Determine whether the state sequences correspond&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;On the other hand, such a separation &lt;em&gt;might&lt;/em&gt; involve discarding some information from the data in step 1 (eg, about its variability) that would be useful in step 2. Which leads to:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Q2: Does it make sense to decompose the problem in this way or is it better to compare the proxy data directly?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If such a decomposition does make sense, that opens up a whole other issue about how to do the idealisation, but that's definitely a question for another day.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Shane, below, mentions goodness-of-fit and distributional tests such as Anderson-Darling, and that seems like a promising approach. But I'd like to check I'm understanding the idea correctly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given sufficient samples in a sequence, we would expect the proportion of time spent in each state to tend to the stationary distribution. So one could test the distributions of occupancies in the two sequences for similarity. (I have the vague sense a two-sample Kolmogorov-Smirnov might suit for this, but please set me right about that.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The thing is, I'm not sure how good this can be as evidence. If the distributions are very different, that seems like a reasonable strike against the underlying processes being the same, but what if they're very similar? Can we draw conclusions in that direction?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Q3: Does a good fit of occupancy distributions tell us anything useful?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems like there could be an infinite number of processes that will tend to the same stationary distribution. I &lt;em&gt;think&lt;/em&gt; this is very unlikely in practice, and that different systems will tend to have distinctly different behaviour, but still it's worth considering.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, we will often have a model of the underlying process that we are looking for, although it may not be perfect. So we could compare each sequence with the expected behaviour of the model, instead of with each other. We may also have more than two sequences to test.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Q4: Is it better to compare multiple sequences to a single model, even an approximate one, or are we better off comparing data directly?&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="174" LastEditorUserId="174" LastEditorDisplayName="" LastEditDate="2010-08-22T14:07:19.200" LastActivityDate="2010-08-22T14:07:19.200" Title="Comparing noisy data sequences to estimate the likelihood of them being produced by different instances of an identical Markov process" Tags="&lt;time-series&gt;&lt;markov-process&gt;&lt;goodness-of-fit&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="1956" PostTypeId="2" ParentId="1955" CreationDate="2010-08-20T14:08:11.677" Score="3" ViewCount="" Body="&lt;p&gt;You can perhaps use a &lt;a href=&quot;http://en.wikipedia.org/wiki/Hidden_Markov_model&quot; rel=&quot;nofollow&quot;&gt;hidden markov model&lt;/a&gt; (HMM). I know that there is a R package that estimates HMMs but cannot recall its name right now.&lt;/p&gt;&#xA;" LastActivityDate="2010-08-20T14:08:11.677" CommentCount="1" />
  <row Id="1957" PostTypeId="2" ParentId="1935" CreationDate="2010-08-20T14:37:06.270" Score="3" ViewCount="" Body="&lt;p&gt;&lt;em&gt;Very&lt;/em&gt; broadly: the intuition, as well as the origin of the name (&quot;pulling oneself up by the bootstraps&quot;), derive from the observation that in using properties of a sample to draw inferences about a population (the &quot;inverse&quot; problem of statistical inference), we expect to err.  To find out the nature of that error, treat the sample itself as a population in its own right and study how your inferential procedure works when you draw samples from &lt;em&gt;it.&lt;/em&gt;  That's a &quot;forward&quot; problem: you know all about your sample-&lt;em&gt;qua&lt;/em&gt;-population and don't have to guess anything about it.  Your study will suggest (a) the extent to which your inferential procedure may be biased and (b) the size and nature of the statistical error of your procedure.  So, use this information to adjust your original estimates.  In many (but definitely not all) situations, the adjusted bias is asymptotically much lower.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One insight provided by this schematic description is that bootstrapping does not &lt;em&gt;require&lt;/em&gt; simulation or repeated subsampling: those just happen to be omnibus, computationally tractable ways to study any kind of statistical procedure when the population is known.  There exist plenty of bootstrap estimates that can be computed mathematically.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This answer owes much to Peter Hall's book &quot;The Bootstrap and Edgeworth Expansion&quot; (Springer 1992), especially his description of the &quot;Main Principle&quot; of bootstrapping.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-08-20T14:37:06.270" CommentCount="1" />
  <row Id="1958" PostTypeId="2" ParentId="1955" CreationDate="2010-08-20T15:35:33.223" Score="3" ViewCount="" Body="&lt;p&gt;A few thoughts:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Can you not just use a goodness-of-fit test?  Choose a distribution and compare both samples.  Or use a qqplot.  You may want to do this with returns (i.e. changes) instead of the original series, since this is often easier to model.  There are also relative distribution functions (see, for instance, the &lt;a href=&quot;http://cran.r-project.org/web/packages/reldist/index.html&quot; rel=&quot;nofollow&quot;&gt;reldist package&lt;/a&gt;).&lt;/li&gt;&#xA;&lt;li&gt;You could look at whether the two series are cointegrated (use &lt;a href=&quot;http://en.wikipedia.org/wiki/Johansen_test&quot; rel=&quot;nofollow&quot;&gt;the Johansen test&lt;/a&gt;).  This is available in the &lt;a href=&quot;http://cran.r-project.org/web/packages/urca/index.html&quot; rel=&quot;nofollow&quot;&gt;urca package&lt;/a&gt; (and related book).&lt;/li&gt;&#xA;&lt;li&gt;There many multivariate time series models such as &lt;a href=&quot;http://en.wikipedia.org/wiki/Vector_autoregression&quot; rel=&quot;nofollow&quot;&gt;VAR&lt;/a&gt; that could be applied to model the dependencies (see the &lt;a href=&quot;http://cran.r-project.org/web/packages/vars/index.html&quot; rel=&quot;nofollow&quot;&gt;vars package&lt;/a&gt;).&lt;/li&gt;&#xA;&lt;li&gt;You could trying using a &lt;a href=&quot;http://en.wikipedia.org/wiki/Copula_%28statistics%29&quot; rel=&quot;nofollow&quot;&gt;copula&lt;/a&gt;, which is used for dependence modeling, and is available in &lt;a href=&quot;http://cran.r-project.org/web/packages/copula/index.html&quot; rel=&quot;nofollow&quot;&gt;the copula package&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;If the noise is serious concern, then try using a filter on the data before analyzing it.&lt;/p&gt;&#xA;" OwnerUserId="5" LastEditorUserId="5" LastEditorDisplayName="" LastEditDate="2010-08-20T15:50:42.683" LastActivityDate="2010-08-20T15:50:42.683" CommentCount="2" />
  <row Id="1959" PostTypeId="2" ParentId="1912" CreationDate="2010-08-20T16:52:39.990" Score="2" ViewCount="" Body="&lt;p&gt;I still feel negatively about what seems to be a gratuitous insult on King's part but I can see where he might be coming from.  &quot;Scale-invariance&quot; is a restriction on a statistical procedure.  Thus, limiting our choice of procedures to scale-invariant ones (or to linear ones or to unbiased ones or minimax ones, etc.) potentially excludes procedures that might perform better.  Whether this is actually the case or not depends.  In many situations, data are reported in units that are essentially independent of what is being studied.  It shouldn't matter whether you measure distances in angstroms or parsecs, for example.  In this context, any procedure that is &lt;em&gt;not&lt;/em&gt; scale invariant is therefore an &lt;em&gt;arbitrary&lt;/em&gt; one--and arbitrariness is not a positive attribute in this field.  In other situations, though, there is a natural scale.  The most obvious of these concern counted data.  A procedure that treats counted data as if they were measurements on a continuous scale (e.g., using OLS for a counted response) is potentially inferior to other available procedures and may be (likely is, I suspect) inadmissible in the decision-theoretic sense.  This can be a tricky and subtle point because it's not always obvious when we have counted data.  One example I'm familiar with concerns many chemical or radioactivity measurements, which ultimately originate as counts on some machine.  Said counts get converted by the laboratory into a concentration or activity that forever after is treated as a real number.  (However, attempts to exploit this fact in the chemometrics literature have not yielded superior statistical procedures.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just to stave off one possible misunderstanding: I wouldn't view a selection of an informative prior for a scale parameter (in a Bayesian analysis) as a scale-dependent procedure.  Such a prior obviously favors some ranges of values over others, but does not affect the scale invariance of the procedure itself.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-08-27T15:21:05.230" LastActivityDate="2010-08-27T15:21:05.230" CommentCount="3" />
  <row Id="1960" PostTypeId="2" ParentId="1906" CreationDate="2010-08-20T16:54:44.217" Score="1" ViewCount="" Body="&lt;p&gt;M2010 - 13th Annual Data Mining Conference &lt;a href=&quot;http://www.sas.com/m2010&quot; rel=&quot;nofollow&quot;&gt;http://www.sas.com/m2010&lt;/a&gt;&lt;/p&gt;&#xA;" LastActivityDate="2010-08-20T16:54:44.217" CommunityOwnedDate="2010-08-20T16:54:44.217" />
  <row Id="1961" PostTypeId="1" AcceptedAnswerId="1981" CreationDate="2010-08-20T17:15:43.887" Score="2" ViewCount="246" Body="&lt;p&gt;When one wants to compute the correlation of two vectors of a continues variables, one uses pearson (or spearman) correlation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;But what should (can) one use for the case of two vectors with 2 (or 3) ordered levels only?  Is spearman enough, or does it require another method?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I remember coming across someone who once claimed to me that OR (odds ratio) is more fitting for such situations (for 2 by 2 tables, where order has no meaning), is this true?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an example R code, for allowing of answers relating to the same example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(10)&#xA;&#xA;x2 &amp;lt;- sample(c(-1,1),50, T)&#xA;x3 &amp;lt;- sample(c(-1:1),50, T)&#xA;y3 &amp;lt;- sample(c(-1:1),50, T)&#xA;y2 &amp;lt;- sample(c(-1,1),50, T)&#xA;&#xA;cor(x3,y3, method = c(&quot;spearman&quot;))&#xA;cor(x2,y2, method = c(&quot;spearman&quot;))&#xA;cor(x3,y2, method = c(&quot;spearman&quot;))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;p.s: for the 2 by 2 case, I followed from the comments that categorical &quot;measures of association&quot; is the term to look for.  However, Part of the time I am comparing 2 on 3 tables, where on the factor with 3 levels there is order - so I would like to take use of that information.&lt;/p&gt;&#xA;" OwnerUserId="253" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-25T07:49:22.363" LastActivityDate="2010-09-25T07:49:22.363" Title="Can you use normal correlation for vectors with only 2 (or 3), ordered, levels?" Tags="&lt;r&gt;&lt;correlation&gt;&lt;association-measure&gt;&lt;ordered-variables&gt;" AnswerCount="3" CommentCount="3" />
  <row Id="1962" PostTypeId="2" ParentId="1961" CreationDate="2010-08-20T17:38:51.850" Score="1" ViewCount="" Body="&lt;p&gt;The OR is a good measure of association, but sometimes people prefer a correlation coefficient for interpretation because it has a [-1, 1] scale.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For binary variables, the Phi statistic provides Pearson's correlation (see Jeromy's comment).  Cramer's V is applicable when you have more than 2x2 cases.  For details, see the following references:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Effect_size&quot; rel=&quot;nofollow&quot;&gt;Effect size&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://faculty.chass.ncsu.edu/garson/PA765/correl.htm&quot; rel=&quot;nofollow&quot;&gt;Correlation&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://edpsych.ed.sc.edu/seaman/edrm711/questions/categorical.htm&quot; rel=&quot;nofollow&quot;&gt;Cramer's V&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I've never used any of these, so hopefully someone will jump in and say if there are good reasons for preferring them.&lt;/p&gt;&#xA;" OwnerUserId="251" LastEditorUserId="251" LastEditorDisplayName="" LastEditDate="2010-08-21T05:49:42.250" LastActivityDate="2010-08-21T05:49:42.250" CommentCount="3" />
  <row Id="1963" PostTypeId="1" AcceptedAnswerId="1976" CreationDate="2010-08-20T18:06:04.580" Score="3" ViewCount="211" Body="&lt;p&gt;A (non-statistician) colleague has been encountering meta-analyses in papers he reviews for medical journals and is looking for a good introductory level treatment so he can educate himself.  Any recommendations?  Favorites?  Books, monographs, nontechnical survey articles would all be fine.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Yes, he's familiar with the Wikipedia entry and other stuff readily accessible by a Google search, such as &lt;a href=&quot;http://www.jerrydallal.com/LHSP/meta.htm&quot; rel=&quot;nofollow&quot;&gt;Jerry Dallal's nice little article&lt;/a&gt;.)&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="5" LastEditorDisplayName="" LastEditDate="2010-08-20T18:22:35.203" LastActivityDate="2010-09-01T12:21:52.743" CommunityOwnedDate="2010-08-20T18:16:10.147" Title="Looking for good introductory treatment of meta-analysis" Tags="&lt;modeling&gt;&lt;meta-analysis&gt;" AnswerCount="3" CommentCount="2" FavoriteCount="1" />
  <row Id="1964" PostTypeId="1" CreationDate="2010-08-20T18:34:22.683" Score="7" ViewCount="242" Body="&lt;p&gt;Hi, I am doing some Kernel density estimation, with a weighted points set (ie., each sample has a weight which is not necessary one), in N dimensions. Also, these samples are just in a metric space (ie., we can define a distance between them) but nothing else. For example, we cannot determine the mean of the sample points, nor the standard deviation, nor scale one variable compared to another. The Kernel is just affected by this distance, and the weight of each sample: f(x) = 1./(sum_weights) * sum(weight_i/h * Kernel(distance(x,x_i)/h))&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this context, I am trying to find a robust estimation for the kernel bandwidth 'h', possibly spatially varying, and preferably which gives an exact reconstruction on the training dataset x_i. If necessary, we could assume that the function is relatively smooth.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried using the distance to the first or second nearest neighbor but it gives quite bad results. I tried with leave-one-out optimization, but I have difficulties finding a good measure to optimize for in this context in N-d, so it finds very bad estimates, especially for the training samples themselves. I cannot use the greedy estimate based on the normal assumption since I cannot compute the standard deviation. I found references using covariance matrices to get anisotropic kernels, but again, it wouldn't hold in this space...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Someone has an idea or a reference ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you very much in advance!&lt;/p&gt;&#xA;" OwnerUserId="1025" LastActivityDate="2011-01-17T17:18:11.883" Title="Kernel bandwidth in Kernel density estimation" Tags="&lt;smoothing&gt;&lt;kde&gt;&lt;kernel&gt;&lt;pdf&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="1965" PostTypeId="2" ParentId="1249" CreationDate="2010-08-20T19:39:20.197" Score="2" ViewCount="" Body="&lt;p&gt;(I will delete my other non-answer, the edit of which had this nugget in it)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;No.&lt;/strong&gt; Asymptotically, the 'trivial' upper bound is the least upper bound. To see this, Let $P_n = P(Z = n)$. Trivially, $E[\exp{(Z^2)}] \ge P_n \exp{(n^2)} = L$, where $L$ is the lower bound of interest. Since $Z$ is binomial, we have $P_n = {n\choose n} (n^{-\beta})^n (1-n^{-\beta})^0 = n^{-n\beta}$. Then $\log{L} = -\beta n \log{n} + n^2$. It is easily shown that this is $\Omega{(n^2)}$, and thus, $L$ is $\Omega{(\exp{(n^2)})}$. Thus the trivial upper bound is, asymptotically, the least upper bound, i.e. $E[\exp{(Z^2)}] \in \Theta{(\exp{(n^2)})}$.&lt;/p&gt;&#xA;" OwnerUserId="795" LastEditorUserId="795" LastEditorDisplayName="" LastEditDate="2010-08-23T16:37:56.203" LastActivityDate="2010-08-23T16:37:56.203" CommentCount="6" />
  <row Id="1966" PostTypeId="1" CreationDate="2010-08-20T20:10:24.127" Score="6" ViewCount="168" Body="&lt;p&gt;What non-/semiparametric methods to estimate a probability density from a data sample are you using ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Please do not include more than one method per answer) &lt;/p&gt;&#xA;" OwnerUserId="961" LastActivityDate="2011-01-22T00:50:41.847" CommunityOwnedDate="2010-08-20T20:55:10.917" Title="Density estimation methods ?" Tags="&lt;probability&gt;&lt;estimation&gt;&lt;nonparametric&gt;" AnswerCount="5" CommentCount="1" FavoriteCount="2" />
  <row Id="1967" PostTypeId="2" ParentId="1249" CreationDate="2010-08-20T20:37:23.543" Score="2" ViewCount="" Body="&lt;p&gt;Numerical experiments (for 2 &amp;lt;= n &amp;lt;= 4000 and all values of beta) indicate the estimate n^2 - (n Ln(n))*beta exceeds the &lt;em&gt;logarithm&lt;/em&gt; of the expectation by an amount on the order of beta*Exp(-n).  The error appears to increase monotonically in beta for each n.  This should provide some useful clues about how to proceed (for those with the time and interest).  In particular, an upper bound for the expectation exists of the form Exp(n^2 - (n Ln(n))*beta + C*Exp(-n)*beta) with C &amp;lt;&amp;lt; 1. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After staring at the summation expression for the expectation, it became evident where the nLn(n)*beta term comes from: break each binomial coefficient Comb(n,k) into its sum Comb(n-1,k) + Comb(n-1,k-1) and write Exp(k^2) = Exp((k-1)^2)*Exp(2k-1).  This decomposes the expectation e(n,beta) into the sum of two parts, one of which looks like the expectation e(n-1,beta) and the other of which is messy (because each term is multiplied by Exp(2k-1)) but can be bounded above by replacing all those exponential terms by their obvious upper bound Exp(2n-1).  (This is not too bad, because the last term with the highest exponent strongly dominates the entire sum.)  This gives a recursive inequality for the expectation,&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;e(n,beta) &amp;lt;= (n^-beta * Exp(2n-1) + 1 - n^-beta) * e(n-1,beta)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Doing this n times creates a polynomial whose highest term is in fact Exp(n^2)*n^(-n*beta), with the remaining terms decreasing fairly rapidly.  At this point any reasonable bound on the remainder will produce an improved bound for the expectation essentially of the form suggested by the numerical experiments.  At this point you have to decide how hard you want to work to obtain a tighter upper bound; the numerical experiments suggest this additional work is not going to pay off unless you're interested in the smallest values of n.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-08-20T21:38:08.627" LastActivityDate="2010-08-20T21:38:08.627" />
  <row Id="1968" PostTypeId="2" ParentId="498" CreationDate="2010-08-20T20:45:24.150" Score="0" ViewCount="" Body="&lt;p&gt;I haven't had the problem you're talking about for a while but I used to.  It might be that when you're selecting the output that you want you release your mouse drag somewhere off of the output (ie you're selecting a large region and you start at the bottom right and drag to the upper left and let go of the mouse somewhere in the explorer area).  I know when I did that it wouldn't go through with the copy.  I needed to release the drag inside of the output window to allow a copy to go through.  Like I said the version I'm using now doesn't have this problem so I can't actually test this but I believe that's how I fixed that problem before.&lt;/p&gt;&#xA;" OwnerUserId="1028" LastActivityDate="2010-08-20T20:45:24.150" />
  <row Id="1969" PostTypeId="2" ParentId="1966" CreationDate="2010-08-20T20:51:48.023" Score="2" ViewCount="" Body="&lt;p&gt;I use Silverman's Adaptive Kernel Density estimator. see e.g &lt;a href=&quot;http://bm2.genes.nig.ac.jp/RGM2/R_current/library/quantreg/man/akj.html&quot; rel=&quot;nofollow&quot;&gt;&lt;code&gt;akj&lt;/code&gt; help page&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-08-20T20:51:48.023" CommunityOwnedDate="2010-08-20T21:03:32.960" />
  <row Id="1970" PostTypeId="1" AcceptedAnswerId="1989" CreationDate="2010-08-20T21:30:13.107" Score="5" ViewCount="133" Body="&lt;p&gt;What are good statistical journals with quick turnaround (fast review cycle), suitable for short notes in mathematical statistics and preferably with open access. An example is Statistics &amp;amp; Probability Letters, however, that journal only has sponsored open access.&lt;/p&gt;&#xA;" OwnerUserId="168" LastActivityDate="2010-08-21T21:19:39.280" CommunityOwnedDate="2010-08-20T21:30:13.107" Title="What is a statistical journal with quick turnaround?" Tags="&lt;open-source&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="1971" PostTypeId="2" ParentId="1908" CreationDate="2010-08-20T21:52:06.853" Score="1" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://www.aistats.org/&quot; rel=&quot;nofollow&quot;&gt;AISTATS&lt;/a&gt; -- Conference on Artificial Intelligence and Statistics&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Similar flavor of papers to NIPS, although papers may be of slightly lower quality. It is much smaller than ICML or NIPS, which allows people to have deeper interactions.&lt;/p&gt;&#xA;" OwnerUserId="168" LastActivityDate="2010-08-20T21:52:06.853" CommunityOwnedDate="2010-08-20T21:52:06.853" />
  <row Id="1972" PostTypeId="1" AcceptedAnswerId="1974" CreationDate="2010-08-20T22:12:36.160" Score="4" ViewCount="145" Body="&lt;p&gt;Hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a bunch of variables organized into 10 different levels of a grouping factor. I'm doing some ANCOVA on particular variables and also plotting the data using boxplots. I'd like to add 84% confidence intervals to all the groups (since non-overlapping 84% CIs indicate a significant difference at alpha .05 - at least for two groups). I can do all this quite easily in R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is - should I be applying a &quot;family-wise&quot; 84% CIs to all the groups? In other words, just as one would devalue an alpha level by the number of groups to obtain a family-wise alpha, should I inflate the CI a reciprocal amount to achieve a family-wise interval? This seems reasonable to me, but I haven't seen this discussed in the literature.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If alpha were CI were interchangeable for two or more groups the the family-wise 84% CI would be 99.5%, but i've read that alpha and CI are only interchangable for 1-sample situations. If this is the case, how would I go about calculating the family-wise confidence intervals for 10 (or any number) groups?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any advice would be welcome.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;best,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Steve&lt;/p&gt;&#xA;" OwnerUserId="1029" LastActivityDate="2010-08-21T17:13:07.467" Title="Family-wise confidence intervals" Tags="&lt;confidence-interval&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="1973" PostTypeId="2" ParentId="1966" CreationDate="2010-08-20T22:35:55.070" Score="2" ViewCount="" Body="&lt;p&gt;Half-space depth a.k.a. bag-plots.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.r-project.org/user-2006/Slides/Mizera.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.r-project.org/user-2006/Slides/Mizera.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="603" LastActivityDate="2010-08-20T22:35:55.070" CommunityOwnedDate="2010-08-20T22:35:55.070" />
  <row Id="1974" PostTypeId="2" ParentId="1972" CreationDate="2010-08-20T22:40:51.053" Score="1" ViewCount="" Body="&lt;p&gt;Hi Steve,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It sounds a reasonable solution &lt;strong&gt;if this is what important for you to present in the plot&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What this will give you  (besides many questions, in case you are working with people who like statistics less then you), is a CI that is applicable to your situation which requires correction for multiple hypothesis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What this won't give you, is the ability to compare difference between groups based on the CI.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regarding the computation of the CI, you could use the p.adjust with something like simes which will still keep your FWE (family wise error), but will give you a wider window.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As to why you didn't find people writing about this, that is a good question, I don't know.&lt;/p&gt;&#xA;" OwnerUserId="253" LastActivityDate="2010-08-20T22:40:51.053" />
  <row Id="1975" PostTypeId="2" ParentId="1961" CreationDate="2010-08-20T22:45:24.010" Score="2" ViewCount="" Body="&lt;p&gt;If you have more than two levels, you can use (M)CA:&#xA;&lt;a href=&quot;http://en.wikipedia.org/wiki/Correspondence_analysis&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Correspondence_analysis&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="603" LastEditorUserId="603" LastEditorDisplayName="" LastEditDate="2010-08-21T10:48:49.690" LastActivityDate="2010-08-21T10:48:49.690" />
  <row Id="1976" PostTypeId="2" ParentId="1963" CreationDate="2010-08-20T23:58:49.150" Score="3" ViewCount="" Body="&lt;p&gt;I have two suggestions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Systematic Reviews in Health Care: Meta-Analysis in Context (&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/072791488X&quot; rel=&quot;nofollow&quot;&gt;Amazon link&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;li&gt;Introduction to Meta-Analysis (Statistics in Practice) (&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0470057246&quot; rel=&quot;nofollow&quot;&gt;Amazon link&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Both books are very good, including introductory information as well as detailed information about how to actually perform meta-analyses.&lt;/p&gt;&#xA;" OwnerUserId="561" LastActivityDate="2010-08-20T23:58:49.150" CommunityOwnedDate="2010-08-20T23:58:49.150" />
  <row Id="1977" PostTypeId="2" ParentId="1966" CreationDate="2010-08-21T00:00:26.493" Score="2" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Dirichlet_process&quot; rel=&quot;nofollow&quot;&gt;Dirichlet Process&lt;/a&gt; mixture models can be very flexible nonparametric Bayesian approach for density modeling, and can also be used as building blocks in more complex models. They are essentially an infinite generalization of parametric Gaussian mixture models and don't require specifying in advance the number of components in the mixture.&lt;/p&gt;&#xA;" OwnerUserId="881" LastActivityDate="2010-08-21T00:00:26.493" CommunityOwnedDate="2010-08-21T00:00:26.493" />
  <row Id="1978" PostTypeId="2" ParentId="1966" CreationDate="2010-08-21T00:04:13.697" Score="2" ViewCount="" Body="&lt;p&gt;Gaussian Processes can also be another nonparametric Bayesian approach for density estimation. See this &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.144.3937&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;Gaussian Process Density Sampler&lt;/a&gt; paper.&lt;/p&gt;&#xA;" OwnerUserId="881" LastActivityDate="2010-08-21T00:04:13.697" CommunityOwnedDate="2010-08-21T00:04:13.697" />
  <row Id="1979" PostTypeId="2" ParentId="1963" CreationDate="2010-08-21T04:42:39.203" Score="3" ViewCount="" Body="&lt;p&gt;I wrote a post a while back on &lt;a href=&quot;http://jeromyanglim.blogspot.com/2009/12/meta-analysis-tips-resources-and.html&quot; rel=&quot;nofollow&quot;&gt;getting started with meta analysis&lt;/a&gt; with:&#xA;(a) tips on getting started,&#xA;(b) links to online introductory texts, &#xA;and (c) links to free software for meta analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Specifically, you might want to read &lt;a href=&quot;http://www.stat-help.com/meta.pdf&quot; rel=&quot;nofollow&quot;&gt;James DeCoster's notes&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-08-21T04:42:39.203" CommunityOwnedDate="2010-08-21T04:42:39.203" />
  <row Id="1980" PostTypeId="1" CreationDate="2010-08-21T04:58:11.600" Score="19" ViewCount="744" Body="&lt;p&gt;&lt;strong&gt;The Question:&lt;/strong&gt; Are there any good examples of &lt;a href=&quot;http://reproducibleresearch.net/index.php/Main_Page&quot; rel=&quot;nofollow&quot;&gt;reproducible research&lt;/a&gt; using R that are freely available online?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Ideal Example:&lt;/strong&gt; &#xA;Specifically, ideal examples would provide:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The raw data (and ideally meta data explaining the data),&lt;/li&gt;&#xA;&lt;li&gt;All R code including data import, processing, analyses, and output generation,&lt;/li&gt;&#xA;&lt;li&gt;Sweave or some other approach for linking the final output to the final document,&lt;/li&gt;&#xA;&lt;li&gt;All in a format that is easily downloadable and compilable on a reader's computer.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Ideally, the example would be a journal article or a thesis where the emphasis is on an actual applied topic as opposed to a statistical teaching example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Reasons for interest:&lt;/strong&gt;&#xA;I'm particularly interested in applied topics in journal articles and theses, because in these situations, several additional issues arise:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Issues arise related to data cleaning and processing&lt;/li&gt;&#xA;&lt;li&gt;Issues arise related to managing metadata&lt;/li&gt;&#xA;&lt;li&gt;Journals and theses often have style guide expectations regarding the appearance and formatting of tables and figures&lt;/li&gt;&#xA;&lt;li&gt;Many journals and theses often have a wide range of analyses which raise issues regarding workflow (i.e., how to sequence analyses) and processing time (e.g., issues of caching analyses, etc.).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Seeing complete working examples could provide good instructional material for researchers starting out with reproducible research.&lt;/p&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-08-21T07:53:32.330" LastActivityDate="2011-01-01T10:35:17.683" Title="Complete substantive examples of reproducible research using R" Tags="&lt;r&gt;&lt;latex&gt;&lt;reproducible-research&gt;&lt;sweave&gt;" AnswerCount="10" FavoriteCount="11" />
  <row Id="1981" PostTypeId="2" ParentId="1961" CreationDate="2010-08-21T05:30:40.953" Score="3" ViewCount="" Body="&lt;p&gt;A few thoughts: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;There are many different binary-binary and ordinal-ordinal measures of association.&#xA;SPSS provides names and algorithms for many of them under &lt;a href=&quot;ftp://ftp.spss.com/pub/spss/statistics/spss/algorithms/proximit.pdf&quot; rel=&quot;nofollow&quot;&gt;proximities&lt;/a&gt;&#xA;and &lt;a href=&quot;http://support.spss.com/productsext/spss/documentation/statistics/algorithms/14.0/crosstabs.pdf&quot; rel=&quot;nofollow&quot;&gt;crosstabs&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;li&gt;I'm also intrigued by  &lt;a href=&quot;http://jeromyanglim.blogspot.com/2009/09/tetrachoric-correlations-overview-and.html&quot; rel=&quot;nofollow&quot;&gt;tetrachoric&lt;/a&gt; (binary-binary) and &lt;a href=&quot;http://www.john-uebersax.com/stat/tetra.htm&quot; rel=&quot;nofollow&quot;&gt;polychoric&lt;/a&gt; (ordinal-ordinal) correlations that&#xA;aim to estimate the correlation between theorised latent continuous variables.&lt;/li&gt;&#xA;&lt;li&gt;You can use Pearson's correlation. However, it is not always the most meaningful metric of association. Also, confidence intervals and p-values that assume continuous normal variables wont be perfectly accurate.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-08-21T05:30:40.953" />
  <row Id="1982" PostTypeId="2" ParentId="1980" CreationDate="2010-08-21T05:47:09.693" Score="3" ViewCount="" Body="&lt;p&gt;Charles Geyer's &lt;a href=&quot;http://www.stat.umn.edu/~charlie/Sweave/#exam&quot; rel=&quot;nofollow&quot;&gt;page on Sweave&lt;/a&gt; has an example from a thesis, which meets some of your requirements (the raw data is simply from an R package, but the R/sweave code and final PDF are available):&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;A paper on the theory in Yun Ju Sung's thesis, Monte Carlo Likelihood Inference for Missing Data Models (preprint) contained computing examples. Every number in the paper and every plot was taken (by cut-and-paste, I must admit) from a &quot;supplementary materials&quot; document done in Sweave.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;(The &lt;a href=&quot;http://www.stat.umn.edu/geyer/bernor/library/bernor/doc/examples.Rnw&quot; rel=&quot;nofollow&quot;&gt;source file&lt;/a&gt; is linked under the &quot;Supplementary Materials for a Paper&quot; section.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know I've come across at least one R example browsing the &lt;a href=&quot;http://www.stat.umn.edu/geyer/bernor/library/bernor/doc/examples.Rnw&quot; rel=&quot;nofollow&quot;&gt;ReproducibleResearch.net material&lt;/a&gt; page before, but unfortunately didn't bookmark it.  &lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-08-21T05:47:09.693" CommentCount="1" />
  <row Id="1983" PostTypeId="2" ParentId="1980" CreationDate="2010-08-21T07:15:43.827" Score="2" ViewCount="" Body="&lt;p&gt;I have found good ones in the past and will post once I dig them up, but some quick general suggestions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;You may be able to find some interesting examples by searching google with keywords and ext:rnw (which will search for files with the sweave extension). Here's &lt;a HREF=&quot;http://www.google.com/m/search?oe=UTF-8&amp;amp;client=safari&amp;amp;hl=en&amp;amp;aq=f&amp;amp;oq=&amp;amp;aqi=-k0d0t0&amp;amp;fkt=1178&amp;amp;fsdt=6703&amp;amp;q=ext%3Arnw+paper&quot; rel=&quot;nofollow&quot;&gt;an example search&lt;/a&gt;.  This is the third result from my search: &lt;a href=&quot;http://www.ne.su.se/paper/araietal_source.Rnw&quot; rel=&quot;nofollow&quot;&gt;http://www.ne.su.se/paper/araietal_source.Rnw&lt;/a&gt;. Here's another example from my search: &lt;a href=&quot;http://www.stat.umn.edu/geyer/gdor/&quot; rel=&quot;nofollow&quot;&gt;http://www.stat.umn.edu/geyer/gdor/&lt;/a&gt;. &lt;/li&gt;&#xA;&lt;li&gt;Many R packages have interesting vignettes which essentially amount to the same thing. An example: &lt;a href=&quot;https://r-forge.r-project.org/scm/viewvc.php/paper/maxLik.Rnw&quot; rel=&quot;nofollow&quot;&gt;https://r-forge.r-project.org/scm/viewvc.php/paper/maxLik.Rnw&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="5" LastEditorUserId="5" LastEditorDisplayName="" LastEditDate="2010-08-21T07:36:31.883" LastActivityDate="2010-08-21T07:36:31.883" />
  <row Id="1984" PostTypeId="2" ParentId="1980" CreationDate="2010-08-21T07:30:48.057" Score="2" ViewCount="" Body="&lt;p&gt;Also look at &lt;a href=&quot;http://www.jstatsoft.org/&quot; rel=&quot;nofollow&quot;&gt;Journal Of Statistical Software&lt;/a&gt;; they encourage making papers in Sweave.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-08-21T07:30:48.057" CommentCount="2" />
  <row Id="1985" PostTypeId="2" ParentId="1980" CreationDate="2010-08-21T14:03:03.573" Score="6" ViewCount="" Body="&lt;p&gt;Frank Harrell has been beating the drum on reproducible research and reports for many, many years.  You could start &#xA;&lt;a href=&quot;http://biostat.mc.vanderbilt.edu/wiki/Main/StatReport&quot; rel=&quot;nofollow&quot;&gt;at this wiki page&lt;/a&gt; which lists plenty of other resources, including published research and also covers Charles Geyer's page.&lt;/p&gt;&#xA;" OwnerUserId="334" LastActivityDate="2010-08-21T14:03:03.573" />
  <row Id="1986" PostTypeId="2" ParentId="1972" CreationDate="2010-08-21T17:13:07.467" Score="0" ViewCount="" Body="&lt;p&gt;For nearly equal sample sizes you can translate &lt;em&gt;Tukey's HSD&lt;/em&gt; (Google it) into a set of individual CIs.  For unequal sample sizes your approach may be doomed, because all pairwise comparisons cannot be reduced to pairwise comparisons of intervals: check out the literature on the &lt;em&gt;Tukey-Kramer Method&lt;/em&gt; for details.  (I know Stata and SAS both do these computations; contributed package DTK does it in R.)&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-08-21T17:13:07.467" CommentCount="1" />
  <row Id="1987" PostTypeId="2" ParentId="1460" CreationDate="2010-08-21T17:59:43.010" Score="2" ViewCount="" Body="&lt;p&gt;You can also use the HST (mentioned here)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://stats.stackexchange.com/questions/1444/how-should-i-transform-non-negative-data-including-zeros/1630#1630&quot;&gt;http://stats.stackexchange.com/questions/1444/how-should-i-transform-non-negative-data-including-zeros/1630#1630&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If $x$ is p.c. consumption, create a variable $x'=x-\bar{x}$ (i.e. de-mean x).&#xA;Then use $f(x',\theta=1)$ as your explanatory variable (where $f()$ is the inverse hyperbolic sin transform).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;for positive values of x' (i.e. people who consume more than the average) &#xA;$f(x',\theta=1)$ behaves as $\log(x')$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;for negative values of $x'$ (i.e. people who consume less than the average) &#xA;$f(x',\theta=1)$ behaves as $-\log(-x')$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;($f(x',\theta=1)$ looks like a big 'S', passing by the origin).&lt;/p&gt;&#xA;" OwnerUserId="603" LastEditorUserId="603" LastEditorDisplayName="" LastEditDate="2010-09-16T14:17:36.323" LastActivityDate="2010-09-16T14:17:36.323" />
  <row Id="1988" PostTypeId="2" ParentId="1841" CreationDate="2010-08-21T20:33:14.980" Score="0" ViewCount="" Body="&lt;p&gt;You can do it all on Excel.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Plotting the six time series should give you a hint of the shapes of the curves. Let's say that, as you mentioned, five of the curves look like they're exponential and the sixth looks like it grows sub-linearly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Insert a trendline for each curve. If you are right, five of them will provide the best fit (as measured by r squared) with an exponential trendline, while the sixth will be best fitted to a logarithmic trendline.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This may sound non-deterministic, but if all six values of r squared are close to 1 you can be pretty confident of your result.&lt;/p&gt;&#xA;" OwnerUserId="666" LastActivityDate="2010-08-21T20:33:14.980" CommentCount="1" />
  <row Id="1989" PostTypeId="2" ParentId="1970" CreationDate="2010-08-21T21:19:39.280" Score="2" ViewCount="" Body="&lt;p&gt;Maybe &lt;a href=&quot;http://www.i-journals.org/ss/&quot; rel=&quot;nofollow&quot;&gt;Statistics Surveys&lt;/a&gt; (but I think they are seeking review more than short note), &lt;a href=&quot;http://www3.stat.sinica.edu.tw/statistica/&quot; rel=&quot;nofollow&quot;&gt;Statistica Sinica&lt;/a&gt;, or the &lt;a href=&quot;http://www.imstat.org/ejs/&quot; rel=&quot;nofollow&quot;&gt;Electronic Journal of Statistics&lt;/a&gt;. They are not as quoted as SPL, but I hope this may help.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-08-21T21:19:39.280" CommunityOwnedDate="2010-08-21T21:19:39.280" />
  <row Id="1990" PostTypeId="2" ParentId="1942" CreationDate="2010-08-21T21:24:35.643" Score="3" ViewCount="" Body="&lt;p&gt;The &lt;a href=&quot;http://www.eam-online.org/&quot; rel=&quot;nofollow&quot;&gt;European Association of Methodology&lt;/a&gt; has a meeting turning around statistics and psychometrics for applied research in social, educational and psychological science every two years. The latest was held in &lt;a href=&quot;http://www.iqb.hu-berlin.de/veranst/EAM-SMABS&quot; rel=&quot;nofollow&quot;&gt;Postdam&lt;/a&gt; two months ago.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-08-21T21:24:35.643" CommunityOwnedDate="2010-08-21T21:24:35.643" />
  <row Id="1991" PostTypeId="2" ParentId="1912" CreationDate="2010-08-21T21:34:42.003" Score="2" ViewCount="" Body="&lt;p&gt;We have a tendency to crunch data according to pre-established algorithms and methods, and forget that &quot;data&quot; is actually information about the real world. I recall as a child in school solving a second-degree equation where the teacher had stated that the answer represented the length of a pencil. Some students actually reported that the answer was &quot;one inch plus or minus two inches&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Before you plug your data into any software, you should first get to &lt;em&gt;really&lt;/em&gt; know and understand it, which you can only accomplish if you keep the subject matter in mind. That's the only way you can spot any quirky data points (such as a pencil measuring -1 inch) or determine which scales make sense in the real world.&lt;/p&gt;&#xA;" OwnerUserId="666" LastActivityDate="2010-08-21T21:34:42.003" CommentCount="3" />
  <row Id="1992" PostTypeId="2" ParentId="1908" CreationDate="2010-08-21T21:36:07.480" Score="0" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://www.aaai.org/Conferences/conferences.php&quot; rel=&quot;nofollow&quot;&gt;AAAI (in Atlanta this year)&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1033" LastActivityDate="2010-08-21T21:36:07.480" CommunityOwnedDate="2010-08-21T21:36:07.480" />
  <row Id="1993" PostTypeId="2" ParentId="1980" CreationDate="2010-08-21T21:59:56.010" Score="6" ViewCount="" Body="&lt;p&gt;We wrote a paper explaining how to use R/Bioconductor when analysing microarray data. The paper was written in Sweave and all the code used to generate the graphs is included as supplementary material.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Gillespie, C. S., Lei, G., Boys, R. J., Greenall, A. J., Wilkinson, D. J., 2010. &lt;a href=&quot;http://www.mas.ncl.ac.uk/~ncsg3/microarray/&quot; rel=&quot;nofollow&quot;&gt;Analysing yeast time course microarray data using BioConductor: a case study using yeast2 Affymetrix arrays&lt;/a&gt; BMC Research Notes,  3:81.&lt;/p&gt;&#xA;" OwnerUserId="8" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-09-02T09:45:08.890" LastActivityDate="2010-09-02T09:45:08.890" />
  <row Id="1994" PostTypeId="2" ParentId="1460" CreationDate="2010-08-21T22:50:51.183" Score="1" ViewCount="" Body="&lt;p&gt;Using variables in logs is actually quite common in economics, since the estimated coefficients can be interpreted as sensitivities to relative changes in RHS variables (or elasticities, if both LHS and RHS variables are in logs). For example, say that you have model &lt;em&gt;y = b ln(x)&lt;/em&gt;, and &lt;em&gt;x&lt;/em&gt; changes to &lt;em&gt;x(1+r)&lt;/em&gt;. Then you can use the approximation $ln(1+t) \approx t$ to see how &lt;em&gt;y&lt;/em&gt; changes:&#xA;$$y = b \ln(x(1+r)) = b \ln(x) + b \ln(1+r) \approx b \ln(x) + b r.$$&#xA;So if &lt;em&gt;r&lt;/em&gt; is 0.01 (&lt;em&gt;x&lt;/em&gt; increases by 1%), &lt;em&gt;y&lt;/em&gt; increases by &lt;em&gt;b r = 0.01 b&lt;/em&gt; (of course, this works only for small &lt;em&gt;r&lt;/em&gt;). In case of your probit model, if coefficient for log-consumption is &lt;em&gt;b&lt;/em&gt;, it can be interpreted so that increase in consumption by 1% would increase probability of enrollment by &lt;em&gt;b&lt;/em&gt; %.&lt;/p&gt;&#xA;" OwnerUserId="1034" LastActivityDate="2010-08-21T22:50:51.183" />
  <row Id="1995" PostTypeId="1" CreationDate="2010-08-22T00:22:33.757" Score="11" ViewCount="508" Body="&lt;p&gt;Under which conditions should someone consider using multilevel/hierarchical analysis as opposed to more basic/traditional analyses (e.g., ANOVA, OLS regression, etc.)? Are there any situations in which this could be considered mandatory? Are there situations in which using multilevel/hierarchical analysis is inappropriate? Finally, what are some good resources for beginners to learn multilevel/hierarchical analysis?&lt;/p&gt;&#xA;" OwnerUserId="835" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-17T20:34:17.710" LastActivityDate="2011-05-22T04:40:26.447" Title="Under what conditions should one use multilevel/hierarchical analysis?" Tags="&lt;multilevel-analysis&gt;" AnswerCount="7" FavoriteCount="7" />
  <row Id="1996" PostTypeId="2" ParentId="726" CreationDate="2010-08-22T00:34:17.460" Score="5" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;It ain’t what you don’t know that gets&#xA;  you into trouble. It’s what you know&#xA;  for sure that just ain’t so.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Mark Twain (okay, so he's not a statistician)&lt;/p&gt;&#xA;" OwnerUserId="74" LastEditorUserId="74" LastEditorDisplayName="" LastEditDate="2010-12-05T05:32:57.497" LastActivityDate="2010-12-05T05:32:57.497" CommunityOwnedDate="2010-08-22T00:34:17.460" CommentCount="1" />
  <row Id="1997" PostTypeId="2" ParentId="1995" CreationDate="2010-08-22T00:40:31.490" Score="9" ViewCount="" Body="&lt;p&gt;When the structure of your data is naturally hierarchical or nested, multilevel modeling is a good candidate.  More generally, it's one method to model interactions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A natural example is when your data is from an organized structure such as country, state, districts, where you want to examine effects at those levels.  Another example where you can fit such a structure is is longitudinal analysis, where you have repeated measurements from many subjects over time (e.g. some biological response to a drug dose).  One level of your model assumes a group mean response for all subjects over time.  Another level of your model then allows for perturbations (random effects) from the group mean, to model individual differences.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A popular and good book to start with is Gelman's &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/052168689X&quot; rel=&quot;nofollow&quot;&gt;&lt;em&gt;Data Analysis Using Regression and Multilevel/Hierachical Models&lt;/em&gt;&lt;/a&gt;.  &lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-08-22T00:40:31.490" CommentCount="3" />
  <row Id="1998" PostTypeId="1" CreationDate="2010-08-22T00:53:28.873" Score="4" ViewCount="176" Body="&lt;p&gt;Miller and Chapman (2001) argue that it is absolutely inappropriate to control for non-independent covariates that are related to both the independent and dependent variables in an observational (non-randomized) study - even though this is routinely done in the social sciences. How problematic is it to do so? How is the best way to deal with this problem? If you routinely control for non-independent covariates in an observational study in your own research, how do you justify it? Finally, is this a fight worth picking when arguing methodology with ones colleagues (i.e., does it really matter)?&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Thanks&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Miller, G. A., &amp;amp; Chapman, J. P. (2001). Misunderstanding analysis of covariance. Journal&#xA;of Abnormal Psychology, 110, 40-48. - &lt;a href=&quot;http://mres.gmu.edu/pmwiki/uploads/Main/ancova.pdf&quot; rel=&quot;nofollow&quot;&gt;http://mres.gmu.edu/pmwiki/uploads/Main/ancova.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="835" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-16T06:42:57.313" LastActivityDate="2010-09-16T06:42:57.313" Title="How problematic is it to control for non-independent covariates in an observational (i.e., non-randomized) study?" Tags="&lt;non-independent&gt;" AnswerCount="5" CommentCount="1" FavoriteCount="3" />
  <row Id="1999" PostTypeId="2" ParentId="1995" CreationDate="2010-08-22T01:04:54.670" Score="2" ViewCount="" Body="&lt;p&gt;Generally, speaking a hierarchical bayesian (HB) analysis will lead to efficient and stable individual level estimates unless your data is such that individual level effects are completely homogeneous (an unrealistic scenario). The efficiency and stable parameter estimates of HB models becomes really important when you have sparse data (e.g., less no of obs than the no of parameters at the individual level) and when you want to estimate individual level estimates.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, HB models are not always easy to estimate. Therefore, while HB analysis usually trumps non-HB analysis you have to weigh the relative costs vs benefits based on your past experience and your current priorities in terms of time and cost.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having said that if you are not interested in individual level estimates then you can simply estimate an aggregate level model but even in these contexts estimating aggregating models via HB using individual level estimates may make a lot of sense. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In summary, fitting HB models is the recommended approach as long as you have the time and the patience to fit them. You can then use aggregate models as a benchmark to assess the performance of your HB model.&lt;/p&gt;&#xA;" LastActivityDate="2010-08-22T01:04:54.670" CommentCount="2" />
  <row Id="2000" PostTypeId="2" ParentId="1998" CreationDate="2010-08-22T01:22:31.077" Score="3" ViewCount="" Body="&lt;p&gt;I read the first page of their paper and so I may have misunderstood their point but it seems to me that they are basically discussing the problem of including multi-collinear independent variables in the analysis. The example they take of age and grade illustrates this idea as they state that:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Age is so intimately associated with grade in school that&#xA;  removal of variance in basketball ability associated with age would&#xA;  remove considerable (perhaps nearly all) variance in basketball&#xA;  ability associated with grade&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;ANCOVA is linear regression with the levels represented as dummy variables and the covariates also appearing as independent variables in the regression equation. Thus, unless I have misunderstood their point (which is quite possible as I have not read their paper completely) it seems they are saying 'do not include dependent covariates' which is equivalent to stating avoid multi-collinear variables.&lt;/p&gt;&#xA;" LastActivityDate="2010-08-22T01:22:31.077" />
  <row Id="2001" PostTypeId="2" ParentId="1998" CreationDate="2010-08-22T02:10:22.197" Score="3" ViewCount="" Body="&lt;p&gt;It is as problematic as the degree of correlation.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The irony is that you wouldn't bother controlling if there weren't some expected correlation with one of the variables.  And, if you expect your independent variable to affect your dependent then it's necessarily somewhat correlated with both.  However, if it's highly correlated them perhaps you shouldn't be controlling for it since it's tantamount to controlling out the actual independent or dependent variable.&lt;/p&gt;&#xA;" OwnerUserId="601" LastActivityDate="2010-08-22T02:10:22.197" />
  <row Id="2002" PostTypeId="1" CreationDate="2010-08-22T02:24:24.947" Score="5" ViewCount="308" Body="&lt;p&gt;Hi. I am running LOESS regression models in R, and I want to compare the outputs of 12 different models with varying sample sizes. I can describe the actual models in more details if it helps with answering the question. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are the sample sizes: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Fastballs vs RHH 2008-09: 2002&#xA;Fastballs vs LHH 2008-09: 2209&#xA;Fastballs vs RHH 2010: 527 &#xA;Fastballs vs LHH 2010: 449&#xA;&#xA;Changeups vs RHH 2008-09: 365&#xA;Changeups vs LHH 2008-09: 824&#xA;Changeups vs RHH 2010: 201&#xA;Changeups vs LHH 2010: 330&#xA;&#xA;Curveballs vs RHH 2008-09: 488&#xA;Curveballs vs LHH 2008-09: 483&#xA;Curveballs vs RHH 2010: 213&#xA;Curveballs vs LHH 2010: 162&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The LOESS regression model is a surface fit, where the X location and the Y location of each baseball pitch is used to predict sw, swinging strike probability. However, I'd like to compare between all 12 of these models, but setting the same span (i.e. span = 0.5) will bear different results since there is such a wide range of sample sizes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My basic question is how do you determine the span of your model? A higher span smooths out the fit more, while a lower span captures more trends but introduces statistical noise if there is too little data. I use a higher span for smaller sample sizes and a lower span for larger sample sizes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What should I do? What's a good rule of thumb when setting span for LOESS regression models in R? Thanks in advance!&lt;/p&gt;&#xA;" LastActivityDate="2010-08-22T03:14:30.497" Title="How do I decide what span to use in LOESS regression in R?" Tags="&lt;r&gt;&lt;regression&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="1" />
  <row Id="2003" PostTypeId="2" ParentId="2002" CreationDate="2010-08-22T02:46:05.200" Score="4" ViewCount="" Body="&lt;p&gt;I suggest checking out generalized additive models (GAM, see the mgcv package in R). I'm just learning about them myself, but they seem to automatically figure out how much &quot;wiggly-ness&quot; is justified by the data. I also see that you're dealing with binomial data (strike vs not a strike), so be sure to analyze the raw data (i.e. don't aggregate to proportions, use the raw pitch-by-pitch data) and use family='binomial' (assuming that you're going to use R). If you have information about what individual pitchers and hitters are contributing to the data, you can probably increase your power by doing a generalized additive mixed model (GAMM, see the gamm4 package in R) and specifying pitcher and hitter as random effects (and again, setting family='binomial'). Finally, you probably want to allow for an interaction between the smooths of X &amp;amp; Y, but I've never tried this myself so I don't know how to go about that. A gamm4 model without the X*Y interaction would look like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;fit = gamm4(&#xA;    formula = strike ~ s(X) + s(Y) + pitch_type*batter_handedness + (1|pitcher) + (1|batter)&#xA;    , data = my_data&#xA;    , family = 'binomial'&#xA;)&#xA;summary(fit$gam)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Come to think of it, you probably want to let the smooths vary within each level of pitch type and batter handedness. This makes the problem more difficult as I've not yet found out how to let the smooths vary by multiple variables in a way that subsequently produces meaninful analytic tests (&lt;a href=&quot;https://stat.ethz.ch/pipermail/r-sig-mixed-models/2010q3/004170.html&quot; rel=&quot;nofollow&quot;&gt;see my queries to the R-SIG-Mixed-Models list&lt;/a&gt;). You could try:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;my_data$dummy = factor(paste(my_data$pitch_type,my_data$batter_handedness))&#xA;fit = gamm4(&#xA;    formula = strike ~ s(X,by=dummy) + s(Y,by=dummy) + pitch_type*batter_handedness + (1|pitcher) + (1|batter)&#xA;    , data = my_data&#xA;    , family = 'binomial'&#xA;)&#xA;summary(fit$gam)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But this won't give meaningful tests of the smooths. In attempting to solve this problem myself, I've used bootstrap resampling where on each iteration I obtain the model predictions for the full data space then compute the bootstap 95% CIs for each point in the space and any effects I care to compute.&lt;/p&gt;&#xA;" OwnerUserId="364" LastEditorUserId="364" LastEditorDisplayName="" LastEditDate="2010-08-22T03:14:30.497" LastActivityDate="2010-08-22T03:14:30.497" />
  <row Id="2004" PostTypeId="2" ParentId="1995" CreationDate="2010-08-22T02:48:20.607" Score="8" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://www.cmm.bristol.ac.uk/learning-training/index.shtml&quot; rel=&quot;nofollow&quot;&gt;The Centre for Multilevel Modelling&lt;/a&gt; has some good free online tutorials for multi-level modeling, and they have software tutorials for fitting models in both their MLwiN software and STATA.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Take this as heresy, because I have not read more than a chapter in the book, but Hierarchical linear models: applications and data analysis methods By Stephen W. Raudenbush, Anthony S. Bryk comes highly recommended. I also swore there was a book on multi level modeling using R software in the Springer Use R! series, but I can't seem to find it at the moment (I thought it was written by the same people who wrote the A Beginner’s Guide to R book). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;edit: The book on using R for multi-level models is &lt;a href=&quot;http://www.springer.com/life+sciences/ecology/book/978-0-387-87457-9&quot; rel=&quot;nofollow&quot;&gt;Mixed Effects Models and Extensions in Ecology with R by Zuur, A.F., Ieno, E.N., Walker, N., Saveliev, A.A., Smith, G.M.&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;good luck&lt;/p&gt;&#xA;" OwnerUserId="1036" LastEditorUserId="1036" LastEditorDisplayName="" LastEditDate="2010-08-23T16:26:02.707" LastActivityDate="2010-08-23T16:26:02.707" />
  <row Id="2005" PostTypeId="2" ParentId="1942" CreationDate="2010-08-22T03:05:50.557" Score="2" ViewCount="" Body="&lt;p&gt;You're probably already aware of it, but the Society for Mathematical Psychology has an annual conference, MathPsych, which is attached to CogSci (generally happnens in the same city either before or after) and blends statistical methodology and Psychological modeling.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;They do a pretty good job getting big names to come present, it's pretty cutting edge.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2010 conference site: &lt;a href=&quot;http://www.mathpsych.org/conferences/2010/&quot; rel=&quot;nofollow&quot;&gt;http://www.mathpsych.org/conferences/2010/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1038" LastActivityDate="2010-08-22T03:05:50.557" CommunityOwnedDate="2010-08-22T03:05:50.557" />
  <row Id="2006" PostTypeId="2" ParentId="1998" CreationDate="2010-08-22T03:11:06.507" Score="2" ViewCount="" Body="&lt;p&gt;As I see it, there are two basic problems with observational studies that &quot;control for&quot; a number of independent variables.  1) You have the problem of missing explanatory variables and thus model misspecification.  2) You have the problem of multiple correlated independent variables--a problem that does not exist in (well) designed experiments--and the fact that regression coefficients and ANCOVA tests of covariates are based on partials, making them difficult to interpret.  The first is intrinsic to the nature of observational research and is addressed in scientific context and the process of competitive elaboration.  The latter is an issue of education and relies on a clear understanding of regression and ANCOVA models and exactly what those coefficients represent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With respect to the first issue, it is easy enough to demonstrate that if all of the influences on some dependent variable are known and included in a model, statistical methods of control are effective and produce good predictions and estimates of effects for individual variables.  The problem in the &quot;soft sciences&quot; is that all of the relevant influences are rarely included or even known and thus the models are poorly specified and difficult to interpret.  Yet, many worthwhile problems exist in these domains.  The answeres simply lack certainty.  The beauty of the scientific process is that it is self corrective and models are questioned, elaborated, and refined.  The alternative is to suggest that we cannot investigate these issues scientifically when we can't design experiments.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second issue is a technical issue in the nature of ANCOVA and regression models.  Analysts need to be clear about what these coefficients and tests represent.  Correlations among the independent variables influence regression coefficients and ANCOVA tests.  They are tests of partials.  These models take out the variance in a given independent variable and the dependent variable that are associated with all of the other variables in the model and then examine the relationship in those residuals.  As a result, the individual coefficients and tests are very difficult to interpret outside of the context of a clear conceptual understanding of the entire set of variables included and their interrelationships.  This, however, produces NO problems for prediction--just be cautious about interpreting specific tests and coefficients.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;A side note:&lt;/strong&gt;  The latter issue is related to a problem discussed previously in this forum on the reversing of regression signs--e.g., from negative to positive--when other predictors are introduced into a model.  In the presence of correlated predictors and without a clear understanding of the multiple and complex relationships among the entire set of predictors, there is no reason to EXPECT a (by nature partial) regression coefficient to have a particular sign.  When there is strong theory and a clear understanding of those interrelationships, such sign &quot;reversals&quot; can be enlightening and theoretically useful.  Though, given the complexity of many social science problems sufficient understanding would not be common, I would expect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Disclaimer:&lt;/strong&gt;  I'm a sociologist and public policy analyst by training.&lt;/p&gt;&#xA;" OwnerUserId="485" LastEditorUserId="485" LastEditorDisplayName="" LastEditDate="2010-08-22T16:22:23.530" LastActivityDate="2010-08-22T16:22:23.530" />
  <row Id="2007" PostTypeId="1" AcceptedAnswerId="2026" CreationDate="2010-08-22T03:24:28.493" Score="8" ViewCount="325" Body="&lt;p&gt;Although I was trained as an engineer, I find that I'm becoming more interested in data mining.  Right now I'm trying to investigate the field further.  In particular, I would like to understand the different categories of software tools that exist and which tools are notable in each category and why. (Note that I didn't say the &quot;best&quot; tools, just the notable ones lest we start a flame war.)  Especially make note of the tools that are open-source and freely available - although don't take this to mean that I'm only interested in open-source and free.&lt;/p&gt;&#xA;" OwnerUserId="1026" LastActivityDate="2010-08-25T22:55:22.173" CommunityOwnedDate="2010-08-22T09:47:37.533" Title="A survey of data-mining software tools." Tags="&lt;data-mining&gt;" AnswerCount="6" CommentCount="4" FavoriteCount="4" />
  <row Id="2008" PostTypeId="1" CreationDate="2010-08-22T04:16:03.970" Score="1" ViewCount="107" Body="&lt;p&gt;Hi all,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am looking through articles citing an article I'm reading - and I wish to judge how much that citing article is &quot;important&quot; or &quot;good&quot; by itself.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;one way of knowing would be if I had known how &quot;distinguished&quot; that journal was.  Which leads me to my question:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What measures are there (and where can I find them) for a journal &quot;importance&quot; or &quot;impact&quot;?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(I know of Impact Factor score.  But I wonder if there are other such measures, and which are more applicable to statistical journals)&lt;/p&gt;&#xA;" OwnerUserId="253" LastActivityDate="2010-08-22T10:47:03.253" CommunityOwnedDate="2010-08-22T04:16:03.970" Title="Measures of publication &quot;importance&quot; in statistics ?" Tags="&lt;journals&gt;" AnswerCount="3" CommentCount="2" />
  <row Id="2010" PostTypeId="1" AcceptedAnswerId="2019" CreationDate="2010-08-22T04:51:46.790" Score="0" ViewCount="108" Body="&lt;p&gt;If &lt;i&gt;x1, x2,..&lt;/i&gt; are identically distributed random variables with given mean, and N is a random variable &gt;= 0 and N is independent of x1,x2,... If &lt;code&gt;y=x_1+x_2+...x_n&lt;/code&gt;. How do we find &lt;code&gt;E(y|N=n)&lt;/code&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;&#xA;E(y|N=n) = sum_Y(y*P(Y=y|N=n)) = sum_Y(y*P(Y=y,N=n)/P(N=n))&#xA;&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not sure how to count/compute &lt;code&gt;P(Y=y,N=n)&lt;/code&gt;. How do compute it?&lt;/p&gt;&#xA;" OwnerUserId="862" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-18T21:59:55.723" LastActivityDate="2010-09-19T16:41:11.350" Title="Conditional expectation given number of outcomes" Tags="&lt;probability&gt;&lt;conditional-probability&gt;" AnswerCount="2" />
  <row Id="2012" PostTypeId="2" ParentId="2008" CreationDate="2010-08-22T06:37:15.933" Score="2" ViewCount="" Body="&lt;p&gt;this might be a good place to start:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.eigenfactor.org/whyeigenfactor.htm&quot; rel=&quot;nofollow&quot;&gt;http://www.eigenfactor.org/whyeigenfactor.htm&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="74" LastActivityDate="2010-08-22T06:37:15.933" CommunityOwnedDate="2010-08-22T06:37:15.933" />
  <row Id="2013" PostTypeId="2" ParentId="2007" CreationDate="2010-08-22T07:27:32.010" Score="5" ViewCount="" Body="&lt;p&gt;Have a look at &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.cs.waikato.ac.nz/ml/weka/&quot; rel=&quot;nofollow&quot;&gt;Weka&lt;/a&gt; (java)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.ailab.si/orange/&quot; rel=&quot;nofollow&quot;&gt;Orange&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;the open-source &lt;a href=&quot;http://www.r-project.org&quot; rel=&quot;nofollow&quot;&gt;R&lt;/a&gt; statistical software (Check the &lt;a href=&quot;http://cran.r-project.org/web/views/MachineLearning.html&quot; rel=&quot;nofollow&quot;&gt;Machine Learning&lt;/a&gt; taskview)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;and the &lt;a href=&quot;http://archive.ics.uci.edu/ml/&quot; rel=&quot;nofollow&quot;&gt;UCI Machine Learning Repository&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-08-22T07:27:32.010" CommunityOwnedDate="2010-08-22T09:47:37.533" CommentCount="5" />
  <row Id="2016" PostTypeId="2" ParentId="2008" CreationDate="2010-08-22T10:02:58.580" Score="2" ViewCount="" Body="&lt;p&gt;Anne-Wil Harzing has a useful site with some free software called &lt;a href=&quot;http://www.harzing.com/pop.htm&quot; rel=&quot;nofollow&quot;&gt;Publish or Perish&lt;/a&gt;.&#xA;The site discusses a number of journal, article, and author impact factor metrics.&#xA;The software uses Google Scholar to calculate citation based impact factor metrics.&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-08-22T10:02:58.580" CommunityOwnedDate="2010-08-22T10:02:58.580" />
  <row Id="2017" PostTypeId="2" ParentId="2007" CreationDate="2010-08-22T10:07:23.333" Score="3" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://rattle.togaware.com/&quot; rel=&quot;nofollow&quot;&gt;Rattle&lt;/a&gt; is a data mining GUI that provides a front end to a wide range of R packages.&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-08-22T10:07:23.333" CommunityOwnedDate="2010-08-22T10:07:23.333" />
  <row Id="2018" PostTypeId="2" ParentId="1998" CreationDate="2010-08-22T10:19:09.027" Score="1" ViewCount="" Body="&lt;p&gt;Some of the matching tools developed by Gary King and colleagues look promising:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://gking.harvard.edu/stats.shtml&quot; rel=&quot;nofollow&quot;&gt;Software&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.vcasmo.com/video/drewconway/8105&quot; rel=&quot;nofollow&quot;&gt;Video providing a tutorial in R&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-08-22T10:19:09.027" />
  <row Id="2019" PostTypeId="2" ParentId="2010" CreationDate="2010-08-22T10:29:46.923" Score="1" ViewCount="" Body="&lt;p&gt;You can use the fact that expectation is &lt;a href=&quot;http://en.wikipedia.org/wiki/Expected_value#Linearity&quot; rel=&quot;nofollow&quot;&gt;linear&lt;/a&gt; and compute $E(Y|N=n)$. The fact that $N$ is a random variable does not matter as you are computing the expected value of $Y$ conditional on a specific value of $N$. &lt;/p&gt;&#xA;" LastActivityDate="2010-08-22T10:29:46.923" />
  <row Id="2020" PostTypeId="2" ParentId="1935" CreationDate="2010-08-22T10:30:13.163" Score="0" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;This is the essence of bootstrapping:&#xA;  taking different samples of your data,&#xA;  getting a statistic for each sample&#xA;  (e.g., the mean, median, correlation,&#xA;  regression coefficient, etc.), and&#xA;  using the variability in the statistic&#xA;  across samples to indicate something&#xA;  about the standard error and&#xA;  confidence intervals for the&#xA;  statistic.&#xA;  - &lt;a href=&quot;http://jeromyanglim.blogspot.com/2009/05/bootstrapping.html&quot; rel=&quot;nofollow&quot;&gt;Bootstrapping and the boot package in R&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-08-22T10:30:13.163" />
  <row Id="2021" PostTypeId="2" ParentId="2008" CreationDate="2010-08-22T10:47:03.253" Score="1" ViewCount="" Body="&lt;p&gt;There's a nice website: &lt;a href=&quot;http://www.arnetminer.org/page/conference-rank/html/Journal.html&quot; rel=&quot;nofollow&quot;&gt;http://www.arnetminer.org/page/conference-rank/html/Journal.html&lt;/a&gt; - however, it only contains computer science conferences. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's an interesting question because publication venue is an extremely important factor for the reception of an article - most of these concepts however live inside the heads of the people working in the field; all these metrics imo just try to mimic those concepts. Although having argued in a comment to your question in favor of metrics, actually looking (even superficially) at as many articles as possible from different sources should let you develop an intuition about good and bad publication venues, which is, on the long run, probably the most efficient tool you can get.&lt;/p&gt;&#xA;" OwnerUserId="979" LastActivityDate="2010-08-22T10:47:03.253" CommunityOwnedDate="2010-08-22T10:47:03.253" />
  <row Id="2022" PostTypeId="1" CreationDate="2010-08-22T11:24:41.663" Score="4" ViewCount="116" Body="&lt;p&gt;Assume:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;A previous study looking at the relationship between $X$ and $Y$ obtained a correlation of $r = 0.50$ using a sample of $n = 100$. The raw data is not available.&lt;/li&gt;&#xA;&lt;li&gt;The current study also looking at the relationship between $X$ and $Y$ obtained a correlation of $r = 0.45$ with $n = 50$.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;How would you do the following tasks:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Give your best estimate of the population correlation between $X$ and $Y$ assuming the two studies are estimating the same correlation.&lt;/li&gt;&#xA;&lt;li&gt;Give your best estimate of the population correlation between $X$ and $Y$ in the current study&#xA;assuming that the first study is slightly different than the current study (e.g., it used a different measurement procedure, a different type of sample, etc.). Of course the weight given to the previous study would depend on perceived similarity with the second study. Thus, are there standard ways of quantifying similarity between studies in such calculations?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-08-22T22:56:57.690" Title="Estimating population correlation based on current data and a previous study" Tags="&lt;bayesian&gt;&lt;correlation&gt;&lt;meta-analysis&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="2023" PostTypeId="2" ParentId="2022" CreationDate="2010-08-22T13:52:09.993" Score="2" ViewCount="" Body="&lt;p&gt;The article on &lt;a href=&quot;http://www.jstor.org/stable/20150454&quot; rel=&quot;nofollow&quot;&gt;Combinative Properties of Correlation Coefficients&lt;/a&gt; may have the answer to your question.&lt;/p&gt;&#xA;" LastActivityDate="2010-08-22T13:52:09.993" CommentCount="1" />
  <row Id="2024" PostTypeId="1" AcceptedAnswerId="2027" CreationDate="2010-08-22T14:18:56.710" Score="2" ViewCount="159" Body="&lt;p&gt;Dear list,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a very basic questions however I have not been able to find an answer.&#xA;When you plot your regular bar plot with your standard errors of the means for comparison do you plot the standard errors and means from the raw data ?&lt;br&gt;&#xA;Or do you plot the ones predicted by the model you are fitting. Thank you in advance for your help. I usually use the latter please advice.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alfred&lt;/p&gt;&#xA;" LastActivityDate="2010-08-22T15:54:54.607" Title="Plotting standard errors" Tags="&lt;confidence-interval&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="2026" PostTypeId="2" ParentId="2007" CreationDate="2010-08-22T15:08:47.093" Score="3" ViewCount="" Body="&lt;p&gt;This is probably the most comprehensive list you'll find: &lt;a href=&quot;http://mloss.org/software/&quot; rel=&quot;nofollow&quot;&gt;mloss.org&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="635" LastActivityDate="2010-08-22T15:08:47.093" CommunityOwnedDate="2010-08-22T15:08:47.093" />
  <row Id="2027" PostTypeId="2" ParentId="2024" CreationDate="2010-08-22T15:33:16.123" Score="4" ViewCount="" Body="&lt;p&gt;The list of things to say here...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As Tai's question suggests, it's hard to directly answer your question without information on the actual model.  Nevertheless, it's usually good to present data reflective of the model.  Typically that is the means with t-tests or ANOVAs.  With something else it's probably close to that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How does a standard error on a graph work for comparison?  Are you going to be putting the N on the graph and the multiplying factor needed for comparisons?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How many data points do you have?  You could probably just put up the entire data set, with a line indicating your predicted value and some kind of measure of variability around it.  Perhaps even an overlayed boxplot.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The measure of variability should reflect what you want to say about the data.  If you just want people to compare values then std. err. isn't a very good idea because it's dependent upon n and requires some value of multiplication greater than 2 (i.e. bars have to not overlap by some amount for significant effects).  Instead, put up 0.5*LSD bars (comparison bars) (about an 84% confidence interval, or a 0.5 * 95%CI * sqrt(2)).  Those bars would show significant differences at the point of bar overlap.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or, you could be wanting to represent how well you estimated the predicted values.  In that case a more convenient confidence interval (about 95%) would be best or even the std. err would be ok.  If you want to reflect the estimated variability of the population you put up the standard deviation.&lt;/p&gt;&#xA;" OwnerUserId="601" LastEditorUserId="601" LastEditorDisplayName="" LastEditDate="2010-08-22T15:54:54.607" LastActivityDate="2010-08-22T15:54:54.607" CommentCount="2" />
  <row Id="2029" PostTypeId="2" ParentId="2007" CreationDate="2010-08-22T18:15:09.023" Score="2" ViewCount="" Body="&lt;p&gt;Have a look at &lt;a href=&quot;http://www.knime.org/&quot; rel=&quot;nofollow&quot;&gt;KNIME&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Very easy to learn. With lots of scope for further progress. Integrates nicely with Weka and R.&lt;/p&gt;&#xA;" OwnerUserId="22" LastActivityDate="2010-08-22T18:15:09.023" CommunityOwnedDate="2010-08-22T18:15:09.023" />
  <row Id="2030" PostTypeId="2" ParentId="2007" CreationDate="2010-08-22T19:24:35.670" Score="2" ViewCount="" Body="&lt;p&gt;From the popularity perspective, this paper (2008) surveys &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.144.5575&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;top 10 algorithms in data mining&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="881" LastActivityDate="2010-08-22T19:24:35.670" CommunityOwnedDate="2010-08-22T19:24:35.670" />
  <row Id="2031" PostTypeId="2" ParentId="203" CreationDate="2010-08-22T19:33:49.787" Score="2" ViewCount="" Body="&lt;p&gt;IMHO you cannot use a t-test for Likert scales. The Likert scale is ordinal and &quot;knows&quot; only about relations of values of a variable: e.g. &quot;totally dissatisfied&quot; is worse than &quot;somehow dissatisfied&quot;. A t-test on the other hand needs to calculate means and more and thus needs interval data. You can map Likert scale scores to interval data (&quot;totally dissatisfied&quot; is 1 and so on) but nobody guarantees that &quot;totally dissatisfied&quot; is the same distance to &quot;somehow dissatisfied&quot; as &quot;somehow dissatisfied&quot; is from &quot;neither nor&quot;. By the way: what is the difference between &quot;totally dissatisfied&quot; and &quot;somehow dissatisfied&quot;? So in the end, you'd do a t-test on the coded values of your ordinal data but that just doesn't make any sense. &lt;/p&gt;&#xA;" OwnerUserId="1048" LastActivityDate="2010-08-22T19:33:49.787" CommentCount="1" />
  <row Id="2032" PostTypeId="1" CreationDate="2010-08-22T19:49:20.753" Score="1" ViewCount="335" Body="&lt;p&gt;What learning material would you suggest for a CS person / novice statistician / novice mathematician to get into predictive analytics?&lt;/p&gt;&#xA;" OwnerUserId="1049" LastActivityDate="2010-10-25T22:33:58.293" CommunityOwnedDate="2010-08-23T13:39:23.147" Title="Recommend some books/articles/guides to enter predictive analytics?" Tags="&lt;books&gt;&lt;predictive-models&gt;" AnswerCount="2" CommentCount="5" FavoriteCount="1" />
  <row Id="2033" PostTypeId="2" ParentId="1493" CreationDate="2010-08-22T21:41:03.547" Score="0" ViewCount="" Body="&lt;p&gt;&lt;pre&gt;&lt;code&gt;&#xA;set.seed(10); x &amp;lt;- matrix(rnorm(15000), ncol=15)&#xA;plot(density(rowSums(x^2)), col=2, xlim=c(-100, 100))&#xA;for (i in 3:10) {&#xA;   lines(density(rowSums(x^i)), col=i)&#xA;}&#xA;&lt;/pre&gt;&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can give us a plot.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't have the perfect theoretical results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another thing is, what's will it be if $n$ is not an integer?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, what is $(-3.5)^{3.4}$?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Oh, maybe you're looking for this &lt;a href=&quot;http://biomet.oxfordjournals.org/cgi/pdf_extract/46/3-4/296&quot; rel=&quot;nofollow&quot;&gt;article&lt;/a&gt;?&lt;/p&gt;&#xA;" OwnerUserId="1043" LastEditorUserId="1043" LastEditorDisplayName="" LastEditDate="2010-08-23T02:22:47.720" LastActivityDate="2010-08-23T02:22:47.720" />
  <row Id="2034" PostTypeId="2" ParentId="2022" CreationDate="2010-08-22T22:56:57.690" Score="1" ViewCount="" Body="&lt;p&gt;You can't hope to combine correlations with any legitimacy unless you also know the means and variances of the X's and Y's in each case, as well as their counts (&lt;em&gt;n&lt;/em&gt;) and correlations (&lt;em&gt;r&lt;/em&gt;).  The article Srikant Vadali refers to (Jack Dunlap, 1937) starts off by making exactly this assumption.  (It's easy to construct examples with the given values of your &lt;em&gt;n&lt;/em&gt; and &lt;em&gt;r&lt;/em&gt; statistics where the combined value of &lt;em&gt;r&lt;/em&gt; is arbitrarily close to +1 or -1 or anything in between.)  Having these full second-moment statistics is crucial in the case of question (2) where one should expect there to be some systematic differences between the two studies.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-08-22T22:56:57.690" CommentCount="5" />
  <row Id="2035" PostTypeId="1" AcceptedAnswerId="2036" CreationDate="2010-08-23T03:12:13.720" Score="4" ViewCount="207" Body="&lt;p&gt;If $X_i\sim\Gamma(\alpha_i,\beta_i)$ for $1\leq i\leq n$, let &#xA;$Y = \sum_{i=1}^n c_iX_i$ where $c_i$ are positive real numbers.&#xA;Assume all the parameters $\alpha_i$'s and $\beta_i$'s are all known, what is $Y$'s distribution ?&lt;/p&gt;&#xA;" OwnerUserId="1043" LastActivityDate="2011-05-20T08:17:10.563" Title="The distribution of the linear combination of Gamma random variables" Tags="&lt;distributions&gt;" AnswerCount="1" CommentCount="0" FavoriteCount="1" />
  <row Id="2036" PostTypeId="2" ParentId="2035" CreationDate="2010-08-23T06:07:35.467" Score="5" ViewCount="" Body="&lt;p&gt;See Theorem 1 given in &lt;a href=&quot;http://www.ism.ac.jp/editsec/aism/pdf/037_3_0541.pdf&quot; rel=&quot;nofollow&quot;&gt;Moschopoulos&lt;/a&gt; (1985) for the distribution of a sum of independent gamma variables.  You can extend this result using the &lt;a href=&quot;http://en.wikipedia.org/wiki/Gamma_distribution#Scaling&quot; rel=&quot;nofollow&quot;&gt;scaling property&lt;/a&gt; for linear combinations.&lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-08-23T06:07:35.467" CommentCount="9" />
  <row Id="2037" PostTypeId="1" CreationDate="2010-08-23T09:34:51.700" Score="8" ViewCount="229" Body="&lt;p&gt;Suppose I have a black box that generates data following a normal distribution with mean m and standard deviation s. Suppose, however, that whenever it outputs a value &amp;lt; 0 it does not record anything (can't even tell that it's outputted such a value). We have a truncated gaussian distribution without a spike.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I estimate these parameters?&lt;/p&gt;&#xA;" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-11-06T18:55:09.787" LastActivityDate="2010-11-06T18:55:09.787" Title="Estimating mean and st dev of a truncated gaussian curve without spike" Tags="&lt;distributions&gt;&lt;estimation&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="3" />
  <row Id="2038" PostTypeId="1" AcceptedAnswerId="2040" CreationDate="2010-08-23T09:48:44.153" Score="2" ViewCount="990" Body="&lt;p&gt;Hi,&#xA;I am reviewing PCA concepts. I came across this nice tutorial - &lt;a href=&quot;http://cran.r-project.org/web/packages/HSAUR/vignettes/Ch_principal_components_analysis.pdf&quot; rel=&quot;nofollow&quot;&gt;http://cran.r-project.org/web/packages/HSAUR/vignettes/Ch_principal_components_analysis.pdf&lt;/a&gt; on how to do PCA in R language. I don't understand the interpretation of Figure 13.3. So I am plotting first eigenvector vs the second eigenvector. What does that mean? Suppose eigenvalue corresponding to first eigenvector explains 60% of variation in data set and second eigenvalue-eigenvector explain 20% of variation. What does it mean to plot these against each other?&lt;/p&gt;&#xA;" OwnerUserId="862" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-11T15:03:32.613" LastActivityDate="2010-10-11T15:03:32.613" Title="Interpretation of biplots in principal components analysis in R" Tags="&lt;r&gt;&lt;data-visualization&gt;&lt;pca&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="4" />
  <row Id="2039" PostTypeId="2" ParentId="2038" CreationDate="2010-08-23T10:22:08.937" Score="1" ViewCount="" Body="&lt;p&gt;The plot is showing:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;the score of each case (i.e., athlete) on the first two principal components&lt;/li&gt;&#xA;&lt;li&gt;the loading of each variable (i.e., each sporting event) on the first two principal components.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The left and bottom axes are showing the loadings; the top and right axes are showing principal component scores.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general it assumes that two components explain a sufficient amount of the variance to provide&#xA; a meaningful visual representation of the structure of cases and variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can look to see which events are close together in the space. Where this applies, this may suggest that athletes who are good at one event are likely also to be good at the other proximal events. Alternatively you can use the plot to see which events are distant. For example, javelin appears to be bit of an outlier and a major event defining the second principal component. Perhaps a different kind of athlete is good at javelin than is good at most of the other events. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, more could be said about substantive interpretation.&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-08-23T10:22:08.937" />
  <row Id="2040" PostTypeId="2" ParentId="2038" CreationDate="2010-08-23T10:23:50.200" Score="4" ViewCount="" Body="&lt;p&gt;PCA is one of the many ways to analyse the structure of a given correlation matrix. By construction, the first principal axis is the one which maximizes the variance (reflected by its eigenvalue) when data are projected onto a line (which stands for a direction in the p-space, assuming you have p variables) and the second one is orthogonal to it, and still maximizes the remaining variance. This is the reason why using the first two axes should yield the better approximation of the original variables space (say, a matrix X of dim n x p) when it is projected onto a plane.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Principal components are just linear combinations of the original variables. Therefore, plotting individual factor scores (defined as Xu, where u is the vector of loadings of any principal component) may help to highlight groups of homogeneous individuals, for example, or to interpret one's overall scoring when considering all variables at the same time. In other words, this is a way to summarize one's location with respect to his value on the p variables, or a combination thereof. In your case, Fig. 13.3 in HSAUR shows that Joyner-Kersee (Jy-K) has a high (negative) score on the 1st axis, suggesting he performed overall quite good on all events. The same line of reasonning applies for interpreting the second axis. I take a very short look at the figure so I will not go into details and my interpretation is certainly superficial. I assume that you will find further information in the HSAUR textbook. Here it is worth noting that both variables and individuals are shown on the same diagram (this is called a &lt;em&gt;biplot&lt;/em&gt;), which helps to interpret the factorial axes while looking at individuals' location. Usually, we plot the variables into a so-called correlation circle (where the angle formed by any two variables, represented here as vectors, reflects their actual pairwise correlation, since r(x1,x2)=cos&lt;sup&gt;2&lt;/sup&gt;(x1,x2)).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think, however, you'd better start reading some introductory book on multivariate analysis to get deep insight into PCA-based methods. For example, B.S. Everitt wrote an excellent textbook on this topic, &lt;em&gt;An R and S-Plus&lt;sup&gt;®&lt;/sup&gt; Companion to Multivariate Analysis&lt;/em&gt;, and you can check the &lt;a href=&quot;http://biostatistics.iop.kcl.ac.uk/publications/everitt/&quot; rel=&quot;nofollow&quot;&gt;companion website&lt;/a&gt; for illustration. There are other great R packages for applied multivariate data analysis, like &lt;a href=&quot;http://cran.r-project.org/web/packages/ade4/index.html&quot; rel=&quot;nofollow&quot;&gt;ade4&lt;/a&gt; and &lt;a href=&quot;http://cran.r-project.org/web/packages/FactoMineR/index.html&quot; rel=&quot;nofollow&quot;&gt;FactoMineR&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-08-23T10:29:54.623" LastActivityDate="2010-08-23T10:29:54.623" />
  <row Id="2041" PostTypeId="2" ParentId="2037" CreationDate="2010-08-23T10:46:36.267" Score="5" ViewCount="" Body="&lt;p&gt;The model for your data would be:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$y_i \sim N(\mu,\sigma^2) I(y_i &gt; 0)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, the density function is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$f(y_i|-) = \frac{exp(-\frac{(y_i-\mu)^2}{2 \sigma^2})}{\sqrt{2 \pi \sigma}\ (1 - \phi(-\frac{\mu}{\sigma}))}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\phi(.)$ is the standard normal cdf.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can then estimate the parameters $\mu$ and $\sigma$ using either maximum likelihood or bayesian methods.&lt;/p&gt;&#xA;" LastActivityDate="2010-08-23T10:46:36.267" />
  <row Id="2042" PostTypeId="2" ParentId="213" CreationDate="2010-08-23T10:48:27.980" Score="7" ViewCount="" Body="&lt;p&gt;Have a look at the &lt;a href=&quot;http://cran.r-project.org/web/packages/mvoutlier/index.html&quot; rel=&quot;nofollow&quot;&gt;mvoutlier&lt;/a&gt; package which relies on ordered robust mahalanobis distances, as suggested by @drknexus.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-08-23T10:48:27.980" />
  <row Id="2043" PostTypeId="2" ParentId="1812" CreationDate="2010-08-23T11:18:49.273" Score="5" ViewCount="" Body="&lt;p&gt;The R &lt;a href=&quot;http://cran.r-project.org/web/packages/psych/index.html&quot; rel=&quot;nofollow&quot;&gt;psych&lt;/a&gt; package includes various routines to apply Factor Analysis (whether it be PCA-, ML- or FA-based), but see my short review on &lt;a href=&quot;http://crantastic.org/packages/psych&quot; rel=&quot;nofollow&quot;&gt;crantastic&lt;/a&gt;. Most of the usual rotation techniques are available, as well as algorithm relying on simple structure criteria; you might want to have a look at W. Revelle's paper on this topic, &lt;a href=&quot;http://personality-project.org/revelle/publications/vss.pdf&quot; rel=&quot;nofollow&quot;&gt;Very Simple Structure: An Alternative Procedure For Estimating The Optimal Number Of Interpretable Factors&lt;/a&gt; (MBR 1979 (14)) and the &lt;code&gt;VSS()&lt;/code&gt; function. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many authors are using orthogonal rotation (VARIMAX), considering loadings higher than, say 0.3 or 0.4 (which amounts to 9 or 16% of variance explained by the factor), as it provides simpler structures for interpretation and scoring purpose (e.g., in quality of life research); others (e.g. Cattell, 1978; Kline, 1979) would recommend oblique rotations since &quot;in the real world, it is not unreasonable to think that factors, as important determiners of behavior, would be correlated&quot; (I'm quoting Kline, &lt;em&gt;Intelligence. The Psychometric View&lt;/em&gt;, 1991, p. 19). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To my knowledge, researchers generally start with FA (or PCA), using a scree-plot together with simulated data (parallel analysis) to help choosing the right number of factors. I often found that item cluster analysis and VSS nicely complement such an approach. When one is interested in second-order factors, or to carry on with SEM-based methods, then obviously you need to use oblique rotation and factor out the resulting correlation matrix.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other packages/software:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://lavaan.ugent.be/&quot; rel=&quot;nofollow&quot;&gt;lavaan&lt;/a&gt;, for latent variable analysis in R;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://openmx.psyc.virginia.edu/&quot; rel=&quot;nofollow&quot;&gt;OpenMx&lt;/a&gt; based on &lt;a href=&quot;http://www.vcu.edu/mx/&quot; rel=&quot;nofollow&quot;&gt;Mx&lt;/a&gt;, a general purpose software including a matrix algebra interpreter and numerical optimizer for structural equation modeling.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;References&lt;/strong&gt;&lt;br&gt;&#xA; 1. Cattell, R.B. (1978). The scientific use of factor analysis in behavioural and life sciences. New York, Plenum.&lt;br&gt;&#xA; 2. Kline, P. (1979). Psychometrics and Psychology. London, Academic Press.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-08-23T11:18:49.273" />
  <row Id="2044" PostTypeId="2" ParentId="726" CreationDate="2010-08-23T13:47:21.180" Score="15" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;If you torture the data enough, nature will always confess.  &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;--Ronald Coase (quoted from Coase, R. H. 1982. How should economists chose? American Enterprise Institute, Washington, D. C.).  I think most who hear this quote misunderstand its profound message against data dredging.&lt;/p&gt;&#xA;" LastEditorUserId="795" LastEditorDisplayName="" LastEditDate="2010-12-03T04:01:49.103" LastActivityDate="2010-12-03T04:01:49.103" CommunityOwnedDate="2010-08-23T13:47:21.180" />
  <row Id="2045" PostTypeId="2" ParentId="2037" CreationDate="2010-08-23T14:14:25.510" Score="2" ViewCount="" Body="&lt;p&gt;As Srikant Vadali has suggested, Cohen and Hald solved this problem using ML (with a Newton-Raphson root finder) around 1950.  Another paper is Max Halperin's &quot;Estimation in the Truncated Normal Distribution&quot; available on &lt;a href=&quot;http://www.jstor.org/pss/2281315&quot; rel=&quot;nofollow&quot;&gt;JSTOR&lt;/a&gt; (for those with access).  Googling &quot;truncated gaussian estimation&quot; produces lots of useful-looking hits.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-10-10T18:41:06.207" LastActivityDate="2010-10-10T18:41:06.207" />
  <row Id="2046" PostTypeId="2" ParentId="726" CreationDate="2010-08-23T14:14:57.677" Score="3" ViewCount="" Body="&lt;p&gt;'Figures fool when fools figure'.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Henry Oliver Lancaster&lt;/p&gt;&#xA;" LastActivityDate="2010-08-23T14:14:57.677" CommunityOwnedDate="2010-08-23T14:14:57.677" />
  <row Id="2048" PostTypeId="2" ParentId="1908" CreationDate="2010-08-23T14:53:02.577" Score="0" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://www.dice.ucl.ac.be/esann/&quot; rel=&quot;nofollow&quot;&gt;European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="976" LastActivityDate="2010-08-23T14:53:02.577" CommunityOwnedDate="2010-08-23T14:53:02.577" />
  <row Id="2049" PostTypeId="2" ParentId="1908" CreationDate="2010-08-23T14:55:35.160" Score="0" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://www.wcci2010.org/topics/ijcnn-2010&quot; rel=&quot;nofollow&quot;&gt;IEEE World Congress on Computational Intelligence&lt;/a&gt;. Note that it is the link for 2010 conference.&lt;/p&gt;&#xA;" OwnerUserId="976" LastActivityDate="2010-08-23T14:55:35.160" CommunityOwnedDate="2010-08-23T14:55:35.160" />
  <row Id="2050" PostTypeId="2" ParentId="1908" CreationDate="2010-08-23T14:58:22.317" Score="0" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://delab.csd.auth.gr/icann2010/&quot; rel=&quot;nofollow&quot;&gt;International Conference on Artificial Neural Networks&lt;/a&gt;. Note that the link is for the 2010 conference.&lt;/p&gt;&#xA;" OwnerUserId="976" LastActivityDate="2010-08-23T14:58:22.317" CommunityOwnedDate="2010-08-23T14:58:22.317" />
  <row Id="2051" PostTypeId="2" ParentId="1908" CreationDate="2010-08-23T15:35:18.580" Score="0" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://aimedicine.info/aime/&quot; rel=&quot;nofollow&quot;&gt;Artificial Intelligence In Medicine&lt;/a&gt; (AIME), odd years starting from 1985.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-08-23T15:35:18.580" CommunityOwnedDate="2010-08-23T15:35:18.580" />
  <row Id="2053" PostTypeId="2" ParentId="1844" CreationDate="2010-08-23T18:03:15.097" Score="2" ViewCount="" Body="&lt;p&gt;I think I figured out the answer (to my own question). If the assumption of proportional hazards is true, the two methods give similar estimates of the hazard ratio. The discrepancy I found in one particular example, I now think, is due to the fact that that assumption is dubious. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the assumption of proportional hazards is true, then a graph of log(time) vs. log(-log(St)) (where St is the proportional survival at time t) should show two parallel lines. Below is the graph created from the problem data set. It seems far from linear. If the assumption of proportional hazards is not valid, then the concept of a hazard ratio is meaningless, and so it doesn't matter which method is used to compute the hazard ratio.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/cgsYv.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wonder if the discrepancy between the logrank and Mantel-Haenszel estimates of the hazard ratio can be used as a method to test the assumption of proportional hazards?&lt;/p&gt;&#xA;" OwnerUserId="25" LastEditorUserId="25" LastEditorDisplayName="" LastEditDate="2010-08-23T21:10:58.187" LastActivityDate="2010-08-23T21:10:58.187" />
  <row Id="2054" PostTypeId="2" ParentId="1927" CreationDate="2010-08-23T18:34:05.990" Score="3" ViewCount="" Body="&lt;p&gt;The Tukey halfspace median can be extended to &gt;2 dimensions using DEEPLOC, an algorithm due to Struyf and Rousseeuw; see &lt;A HREF=&quot;http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.37.2052&quot; rel=&quot;nofollow&quot;&gt;here&lt;/A&gt; for details.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The algorithm is used to approximate the point of greatest depth efficiently; naive methods which attempt to determine this exactly usually run afoul of (the computational version of) &quot;the curse of dimensionality&quot;, where the runtime required to calculate a statistic grows exponentially with the number of dimensions of the space.&lt;/p&gt;&#xA;" OwnerUserId="1068" LastActivityDate="2010-08-23T18:34:05.990" />
  <row Id="2055" PostTypeId="2" ParentId="1799" CreationDate="2010-08-23T19:47:39.243" Score="2" ViewCount="" Body="&lt;p&gt;As to your suggestion of using Multi-level models in this and the other thread, I see no benefit of approaching your analysis in this manner over repeated measures ANOVA. MLM are simply an extension of OLS regression, and offer an explicit framework to model group level (often referred to as &quot;contextual&quot;) effects on lower level estimates. I would guess from your example you have no specific &quot;contexts&quot; besides that measures are repeated within individuals, and this is accounted for within the repeated ANOVA design (and your not interested in measuring effects of specific individuals anyway, only to control for this non-independence). Neither groups nor gender with what your description says are &quot;contexts&quot; in this sense, they can only be direct effects (or at least you can only observe if they have direct effects).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;MLM just complicates things in this example IMO. It doesn't directly solve your problem that several dependent measures are non-independent, you can't measure any group level characteristic on your outcome because you only have two groups, and your hierarchy is cross classified and has three levels, (an observation is nested within only 1 individual, but the gender and group nestings are not mutually exclusive). All of the things you listed as purposes of the project can be accomplished using repeated ANOVA simply including group, gender, and group*gender interaction effects into the models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is beyond the scope of your question, but I would never agree that gender is a group observations are nested within. &lt;/p&gt;&#xA;" OwnerUserId="1036" LastEditorUserId="1036" LastEditorDisplayName="" LastEditDate="2010-08-24T01:49:46.973" LastActivityDate="2010-08-24T01:49:46.973" />
  <row Id="2056" PostTypeId="2" ParentId="1399" CreationDate="2010-08-23T20:39:21.760" Score="4" ViewCount="" Body="&lt;p&gt;According to Davison and Hinckley (&quot;Bootstrap methods and their application&quot;, 1997, Section 3.8), the third algorithm is conservative.  They advocate a fourth approach: simply resampling the subjects.&lt;/p&gt;&#xA;" OwnerUserId="187" LastActivityDate="2010-08-23T20:39:21.760" CommentCount="3" />
  <row Id="2057" PostTypeId="2" ParentId="1841" CreationDate="2010-08-23T22:04:33.300" Score="4" ViewCount="" Body="&lt;p&gt;I have a relatively simple solution to propose, Hugo.   Because you're forthright about not being a statistician (often a plus ;-) but obviously can handle technical language, I'll take some pains to be technically clear but avoid statistical jargon.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's start by checking my understanding: you have six series of data (t[j,i], h[j,i]), 1 &amp;lt;= j &amp;lt;= 6, 1 &amp;lt;= i &amp;lt;= n[j], where t[j,i] is the time you measured the entropy h[j,i] for artifact j and n[j] is the number of observations made of artifact j.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We may as well assume t[j,i] &amp;lt;= t[j,i+1] is always the case, but it sounds like you cannot necessarily assume that t[1,i] = ... = t[6,i] for all i (synchronous measurements) or even that t[j,i+1] - t[j,i] is a constant for any given j (equal time increments). We might as well also suppose j=1 designates your special artifact.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We do need a model for the data.  &quot;Exponential&quot; versus &quot;sublinear&quot; covers a lot of ground, suggesting we should adopt a very broad (non-parametric) model for the behavior of the curves.  One thing that simply distinguishes these two forms of evolution is that the increments h[j,i+1] - h[j,i] in the exponential case will be increasing whereas for concave sublinear growth the increments will decreasing.  Specifically, the &lt;em&gt;increments of the increments&lt;/em&gt;,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;d2[j,i] = h[j,i+1] - 2*h[j,i+1] + h[j,i], 1 &amp;lt;= i &amp;lt;= n[j]-2,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;will either tend to be positive (for artifact 1) or negative (for the others).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A big question concerns the nature of variation: the observed entropies might not exactly fit along any nice curve; they might oscillate, seemingly at random, around some ideal curve.  Because you don't want to do any statistical modeling, we aren't going to learn much about the nature of this variation, but let's hope that the amount of variation for any given artifact j is typically about the same size for all times t[j,i].  This lets us write each entropy in the form&lt;/p&gt;&#xA;&#xA;&lt;p&gt;h[j,i] = y[j,i] + e[j,i]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where y[j,i] is the &quot;true&quot; entropy for artifact j at time t[j,i] and e[j,i] is the difference between the observed entropy h[j,i] and the true entropy.  It might be reasonable, as a first cut at this problem, to hope that the e[j,i] act randomly and appear to be statistically independent of each other and of the y[j,i] and t[j,i].&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This setup and these assumptions imply that the set of second increments for artifact j, {d2[j,i] | 1 &amp;lt;= i &amp;lt;= n[j]-2}, will not necessarily be entirely positive or entirely negative, but that each such set should look like a bunch of (potentially different) positive or negative numbers plus some fluctuation:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;d2[j,i] = (y[j,i+2] - 2*y[j,i+1] + y[j,i]) + (e[j,i+2] - 2*e[j,i+1] + e[j,i]).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We're still not in a classic probability context, but we're close if we (incorrectly, but perhaps not fatally) treat the correct second increments (y[j,i+2] - 2*y[j,i+1] + y[j,i]) as if they were numbers drawn randomly from some box.  In the case of artifact 1 your hope is that this is a box of all positive numbers; for the other artifacts, your hope is that it is a box of all negative numbers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;At this point we can apply some standard machinery for hypothesis testing&lt;/strong&gt;.  The null hypothesis is that the true second increments are all (or most of them) negative; the alternative hypothesis covers all the other 2^6-1 possibilities concerning the signs of the six batches of second increments.  This suggests &lt;strong&gt;running a t-test separately for each collection of actual second increments to compare them against zero&lt;/strong&gt;.  (A non-parametric equivalent, such as a sign test, would be fine, too.)  Use a Bonferroni correction with these planned multiple comparisons; that is, if you want to test at a level of &lt;em&gt;alpha&lt;/em&gt; (e.g., 5%) to attain a desired &quot;probability value,&quot; use the &lt;em&gt;alpha&lt;/em&gt;/6 critical value for the test.  This can readily be done even in a spreadsheet if you like.  It's fast and straightforward.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This approach is not going to be the best one because among all those that could be conceived: it's one of the less powerful and it still makes some assumptions (such as independence of the errors); but if it works--that is, if you find the second increments for j=1 to be significantly above 0 and all the others to be significantly below 0--then it will have done its job.  If this is not the case, your expectations might still be correct, but it would take a greater statistical modeling effort to analyze the data.  (The next phase, if needed, might be to look at the runs of increments for each artifact to see whether there's evidence that &lt;em&gt;eventually&lt;/em&gt; each curve becomes exponential or sublinear.  It should also involve a deeper analysis of the nature of variation in the data.)&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-08-25T13:52:20.933" LastActivityDate="2010-08-25T13:52:20.933" CommentCount="2" />
  <row Id="2058" PostTypeId="2" ParentId="213" CreationDate="2010-08-23T22:41:05.283" Score="4" ViewCount="" Body="&lt;p&gt;You can find candidates for &quot;outliers&quot; among the support points of the minimum volume bounding ellipsoid.  (Efficient algorithms to find these points in fairly high dimensions, both exactly and approximately, were invented in a spate of papers in the 1970's because this problem is intimately connected with a question in experimental design.)&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-08-23T22:41:05.283" />
  <row Id="2059" PostTypeId="1" CreationDate="2010-08-23T23:21:15.567" Score="4" ViewCount="193" Body="&lt;p&gt;It seems that MLE (via EM) is widely used in machine learning / statistics to learn the parameters of a mixture of Gaussians. I'm assuming we're given random samples from the mixture. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is: Are there any proven &lt;em&gt;quantitative&lt;/em&gt; bounds on the error in terms of the number of samples (and perhaps the parameters of the Gaussian)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, what is the runtime required to estimate the parameters up to a certain error?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ideally, these bounds would not assume that we start in a local neighborhood of the optimal solution or any such thing. (If EM is not the method of choice and there is a better way of doing it, please point this out, as well.)&lt;/p&gt;&#xA;" OwnerUserId="1072" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-27T16:01:57.583" LastActivityDate="2011-03-27T16:01:57.583" Title="Learning parameters of a mixture of Gaussian using MLE" Tags="&lt;estimation&gt;&lt;normal-distribution&gt;&lt;em-algorithm&gt;&lt;mixture&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="1" />
  <row Id="2060" PostTypeId="2" ParentId="2059" CreationDate="2010-08-24T00:27:18.560" Score="2" ViewCount="" Body="&lt;p&gt;EM essentially solves the maximum likelihood problem and therefore has the same properties w.r.t. sample sizes. EM for Gaussian mixture models is known to converge asymptotically to a local maximum and exhibits first order convergence (see &lt;a href=&quot;http://dspace.mit.edu/bitstream/handle/1721.1/7195/AIM-1520.pdf;jsessionid=DEB41C50ECD9DA8161DC80F7057BDCBE?sequence=2&quot; rel=&quot;nofollow&quot;&gt;this paper&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;BTW, there are some results which quantify how good the EM solution is in terms of the parameters of the data distribution. See &lt;a href=&quot;http://www.cs.caltech.edu/~schulman/Papers/em-uaif.pdf&quot; rel=&quot;nofollow&quot;&gt;this paper&lt;/a&gt; which shows that the goodness depends on the separation of mixture components (measured by variances). A lot of papers have analyzed mixture models using this criteria.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you don't want to use EM for mixture models, you can take a &lt;a href=&quot;http://research.microsoft.com/en-us/um/people/cmbishop/downloads/Bishop-robust-mixture-Neurocomputing-04.pdf&quot; rel=&quot;nofollow&quot;&gt;fully Bayesian approach&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="881" LastEditorUserId="881" LastEditorDisplayName="" LastEditDate="2010-08-24T01:24:08.150" LastActivityDate="2010-08-24T01:24:08.150" CommentCount="1" />
  <row Id="2061" PostTypeId="1" CreationDate="2010-08-24T02:48:08.290" Score="4" ViewCount="354" Body="&lt;p&gt;When lecturing statistics, it can be useful to incorporate the occasional short video.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My first thoughts included: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Animations and visualisations of statistical concepts&lt;/li&gt;&#xA;&lt;li&gt;Stories regarding the application of a particular technique&lt;/li&gt;&#xA;&lt;li&gt;Humorous videos that have relevance to a statistical idea &lt;/li&gt;&#xA;&lt;li&gt;Interviews with a statistician or a researcher who uses statistics&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Are there any videos that you use when teaching statistics or that you think would be useful?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please provide:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Link to the video online&lt;/li&gt;&#xA;&lt;li&gt;Description of content&lt;/li&gt;&#xA;&lt;li&gt;To what statistical topic the video might relate&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-09-17T08:49:22.640" LastActivityDate="2010-10-14T10:31:18.257" CommunityOwnedDate="2010-08-25T08:02:51.643" Title="Short online videos that could assist a lecture on statistics" Tags="&lt;teaching&gt;&lt;video&gt;" AnswerCount="5" CommentCount="1" FavoriteCount="1" />
  <row Id="2062" PostTypeId="2" ParentId="2061" CreationDate="2010-08-24T02:57:23.123" Score="5" ViewCount="" Body="&lt;p&gt;Link:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://animation.yihui.name/&quot; rel=&quot;nofollow&quot;&gt;http://animation.yihui.name/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A big list of animation &quot;clips&quot; (gif's or other formats), through the use of the &quot;&lt;a href=&quot;http://cran.r-project.org/web/packages/animation/index.html&quot; rel=&quot;nofollow&quot;&gt;animation&lt;/a&gt;&quot; package (R).  Including the following topics:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Topics&lt;/strong&gt;(in each of them there are 1 or more animation, by topic) &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Theory of Probability&lt;/li&gt;&#xA;&lt;li&gt;Mathematical Statistics&lt;/li&gt;&#xA;&lt;li&gt;Sampling Survey&lt;/li&gt;&#xA;&lt;li&gt;Linear Models&lt;/li&gt;&#xA;&lt;li&gt;Multivariate Statistics&lt;/li&gt;&#xA;&lt;li&gt;Nonparametric Statistics (no videos yet)&lt;/li&gt;&#xA;&lt;li&gt;Computational Statistics&lt;/li&gt;&#xA;&lt;li&gt;Time Series Analysis&lt;/li&gt;&#xA;&lt;li&gt;Data Mining, Machine Learning&lt;/li&gt;&#xA;&lt;li&gt;Statistical Data Analysis&lt;/li&gt;&#xA;&lt;li&gt;R Graphics&lt;/li&gt;&#xA;&lt;li&gt;Package ''animation''&lt;/li&gt;&#xA;&lt;li&gt;Dynamic Graphics&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Through the use of the package, the animations can be reproduced in various formats (such as gif, aws, and others)&lt;/p&gt;&#xA;" OwnerUserId="253" LastActivityDate="2010-08-24T02:57:23.123" CommunityOwnedDate="2010-08-25T08:02:51.643" CommentCount="1" />
  <row Id="2063" PostTypeId="2" ParentId="1873" CreationDate="2010-08-24T06:06:54.557" Score="0" ViewCount="" Body="&lt;p&gt;Perhaps I'm misunderstanding the OP, but shouldn't the test be for 8 successes (of not stubbing the fellow's toe) and 7 failures, for a total of 15 trials? And shouldn't the comparison be to a probability of 2/3 (2000 incidents of not stubbing a toe relative to the 1000 observed ascents)? This, of course, is taking the true population probability of stubbing a toe to be 1/3. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One way to permit both quantities to be random is to bootstrap. If stubbing is 1 and not stubbing is 0, draw (3000-15) observations with replacement from the total pool. Note that we exclude Joe from the total pool in order to compare him to others. Sum the observations (i.e., count the number of toe stubs) and divide by the total number of observations, 3000-15. Draw 15 observations from Joe's sample, sum, and divide by 15. Subtract Joe's proportion of stubs from the population proportion. Simulate many (1000, 10000 maybe) times, look at the 2.5 and 97.5 percentiles (for a 95% test against the null hypothesis that Joe's stub rate is the same as that of the population; a different level or one-sided test could also be used as desired) and see whether 0 is in that interval. If it is, then you cannot reject the null hypothesis that Joe is as clumsy as everyone else.&lt;/p&gt;&#xA;" OwnerUserId="401" LastActivityDate="2010-08-24T06:06:54.557" />
  <row Id="2064" PostTypeId="2" ParentId="1998" CreationDate="2010-08-24T06:38:43.993" Score="2" ViewCount="" Body="&lt;p&gt;In the social sciences, we often call this issue &quot;post treatment bias.&quot; If you are considering the effect of some treatment (your independent variable), including variables that arise after treatment (in a causal sense), then your estimate of the treatment effect can be biased. If you include these variables, then you are, in some sense, controlling for the impact of treatment. If treatment T causes outcome Y and other variable A and A causes Y, then controlling for A ignores the impact that T has on Y via A. This bias can be positive or negative.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the social sciences, this can be especially difficult because A might cause T, which feeds back on A, and A and T both cause Y. For example, high GDP can lead to high levels of democratization (our treatment), which leads to higher GDP, and higher GDP and higher democratization both lead to less government corruption, say. Since GDP causes democratization, if we don't control for it, then we have an endogeneity issue or &quot;omitted variables bias.&quot; But if we do control for GDP, we have post treatment bias. Other than use randomized trials when we can, there is little else that we can do to steer our ship between Scylla and Charybdis. Gary King talks about these issues as his nomination for Harvard's &quot;Hardest Unsolved Problems in the Social Sciences&quot; initiative &lt;a href=&quot;http://www.slideshare.net/HardProblemsSS/gary-king-hard-unsolved-problem-in-social-sciencepost-treatment-bias&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="401" LastActivityDate="2010-08-24T06:38:43.993" />
  <row Id="2065" PostTypeId="2" ParentId="1844" CreationDate="2010-08-24T06:49:15.703" Score="1" ViewCount="" Body="&lt;p&gt;I thought I'd stumbled across a web site and reference that deals exactly with this question:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.graphpad.com/faq/viewfaq.cfm?faq=1226&quot; rel=&quot;nofollow&quot;&gt;http://www.graphpad.com/faq/viewfaq.cfm?faq=1226&lt;/a&gt;&#xA;Start from &quot;The two methods compared&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The site references the Berstein paper ars linked (above):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.jstor.org/stable/2530564?seq=1&quot; rel=&quot;nofollow&quot;&gt;http://www.jstor.org/stable/2530564?seq=1&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The site summarises Berstein et al's results nicely, so I'll quote it:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The two usually give identical (or&#xA;  nearly identical) results. But the&#xA;  results can differ when several&#xA;  subjects die at the same time or when&#xA;  the hazard ratio is far from 1.0.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Bernsetin and colleagues analyzed&#xA;  simulated data with both methods (1).&#xA;  In all their simulations, the&#xA;  assumption of proportional hazards was&#xA;  true. The two methods gave very&#xA;  similar values.  The logrank method&#xA;  (which they refer to as the O/E&#xA;  method) reports values that are closer&#xA;  to 1.0 than the true Hazard Ratio,&#xA;  especially when the hazard ratio is&#xA;  large or the sample size is large.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;When there are ties, both methods are&#xA;  less accurate. The logrank methods&#xA;  tend to report hazard ratios that are&#xA;  even closer to 1.0 (so the reported&#xA;  hazard ratio is too small when the&#xA;  hazard ratio is greater than 1.0, and&#xA;  too large when the hazard ratio is&#xA;  less than 1.0). The Mantel-Haenszel&#xA;  method, in contrast, reports hazard&#xA;  ratios that are further from 1.0  (so&#xA;  the reported hazard ratio is too large&#xA;  when the hazard ratio is greater than&#xA;  1.0, and too small when the hazard ratio is less than 1.0). &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;They did not test the two methods with&#xA;  data simulated where the assumption of&#xA;  proportional hazards is not true. I&#xA;  have seen one data set where the two&#xA;  estimate of HR were very different (by&#xA;  a factor of three), and the assumption&#xA;  of proportional hazards was dubious&#xA;  for those data. It seems that the&#xA;  Mantel-Haenszel method gives more&#xA;  weight to differences in the hazard at&#xA;  late time points, while the logrank&#xA;  method gives equal weight everywhere&#xA;  (but I have not explored this in&#xA;  detail).  If you see very different HR&#xA;  values with the two methods, think&#xA;  about whether the assumption of&#xA;  proportional hazards is reasonable. If&#xA;  that assumption is not reasonable,&#xA;  then of course the entire concept of a&#xA;  single hazard ratio describing the&#xA;  entire curve is not meaningful&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The site also refer to the dataset in which &quot;the two estimate of HR were very different (by a factor of three)&quot;, and suggest that the PH assumption is a key consideration. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then I thought, &quot;Who authored the site?&quot; After a little searching I found it was Harvey Motulsky. So Harvey I've managed to reference you in answering your own question. You've become the authority!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is the &quot;problem dataset&quot; a publicly available dataset?&lt;/p&gt;&#xA;" OwnerUserId="521" LastActivityDate="2010-08-24T06:49:15.703" CommentCount="2" />
  <row Id="2066" PostTypeId="1" AcceptedAnswerId="2074" CreationDate="2010-08-24T09:48:08.093" Score="8" ViewCount="564" Body="&lt;p&gt;Is there a numerically stable way to calculate values of a &lt;a href=&quot;http://en.wikipedia.org/wiki/Beta_distribution&quot; rel=&quot;nofollow&quot;&gt;beta distribution&lt;/a&gt; for large integer alpha, beta (e.g. alpha,beta &gt; 1000000)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Actually, I only need a 99% confidence interval around the mode, if that somehow makes the problem easier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Add&lt;/strong&gt;: I'm sorry, my question wasn't as clearly stated as I thought it was. What I want to do is this: I have a machine that inspects products on a conveyor belt. Some fraction of these products is rejected by the machine. Now if the machine operator changes some inspection setting, I want to show him/her the estimated reject rate and some hint about how reliable the current estimate is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I thought I treat the actual reject rate as a random variable X, and calculate the probability distribution for that random variable based on the number of rejected objects N and accepted objects M. If I assume a uniform prior distribution for X, this is a beta distribution depending on N and M. I can either display this distribution to the user directly or find an interval [l,r] so that the actual reject rate is in this interval with p &gt;= 0.99 (using shabbychef's terminology) and display this interval. For small M, N (i.e. immediately after the parameter change), I can calculate the distribution directly and approximate the interval [l,r]. But for large M,N, this naive approach leads to underflow errors, because x^N*(1-x)^M is to small to be represented as a double precision float.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I guess my best bet is to use my naive beta-distribution for small M,N and switch to a normal distribution with the same mean and variance as soon as M,N exceed some threshold. Does that make sense?&lt;/p&gt;&#xA;" OwnerUserId="956" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-10-06T14:29:31.523" LastActivityDate="2010-10-06T14:29:31.523" Title="How can I (numerically) approximate values for a beta distribution with large alpha &amp; beta" Tags="&lt;confidence-interval&gt;&lt;algorithms&gt;&lt;beta&gt;" AnswerCount="3" CommentCount="3" FavoriteCount="1" />
  <row Id="2067" PostTypeId="1" AcceptedAnswerId="2138" CreationDate="2010-08-24T10:05:42.533" Score="8" ViewCount="530" Body="&lt;p&gt;I am currently finishing a paper and stumbled upon &lt;a href=&quot;http://stats.stackexchange.com/questions/2024/plotting-standard-errors&quot;&gt;this question from yesterday&lt;/a&gt; which led me to pose the same question to myself. Is it better to provide my graph with the actual standard error from the data or the one estimated from my ANOVA?&lt;br&gt;&#xA;As the question from yesterday was rather unspecific and mine is pretty specific I thought it would be appropriate to pose this follow-up question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Details:&lt;/strong&gt;&lt;br&gt;&#xA;I have run an experiment in some cognitive psychology domain (conditional reasoning) comparing two groups (inductive and deductive instructions, i.e., a between-subjects manipulation) with two within-subjects manipulations (problem type and content of the problem, each with two factor levels).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The results look like this (left panel with SE-estimates from the ANOVA Output, right panel with SEs estimated from the data):&#xA;&lt;img src=&quot;http://i.imgur.com/fHmrK.png&quot; alt=&quot;alt text&quot;&gt;&lt;br&gt;&#xA;Note that the different lines represent the two different groups (i.e., the between-subjects manipulation) and the within-subjects manipulations are plotted on the x-axis (i.e., the 2x2 factor levels).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the text I provide the respective results of the ANOVA and even planned comparisons for the critical cross-over interaction in the middle. &lt;strong&gt;The SEs are there to give the reader some hint about the variability of the data.&lt;/strong&gt; I prefer SEs over standard deviations and confidence intervals as it is not common to plot SDs and there are severe problems when comparing within- and between-subjects CIs (as the same surely applys for SEs, it is not so common to falsely infer significant differences from them).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To repeat my question: &lt;strong&gt;Is it better to plot the SEs estimated from the ANOVA or should I plot the SEs estimated from the raw data?&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update:&lt;/strong&gt;&lt;br&gt;&#xA;I think I should be a little bit clearer in what the estimated SEs are. The ANOVA Output in SPSS gives me &lt;code&gt;estimated marginal means&lt;/code&gt; with corresponding SEs and CIs. This is what is plotted in the left graph. As far as I understand this, they should be the SDs of the residuals. But, when saving the residuals their SDs are not somehow near the estimated SEs. So a secondary (potentially SPSS specific) question would be:&lt;br&gt;&#xA;&lt;strong&gt;What are these SEs?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE 2:&lt;/strong&gt; I finally managed to write a R-function that should be able to make a plot as I finally liked it (see my accepted answer) on its own. If anybody has time, I would really appreciate if you could have a look at it. &lt;a href=&quot;http://www.psychologie.uni-freiburg.de/Members/singmann/R/rm.plot&quot; rel=&quot;nofollow&quot;&gt;Here it is.&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="442" LastEditorUserId="442" LastEditorDisplayName="" LastEditDate="2010-12-09T12:28:44.597" LastActivityDate="2010-12-09T12:30:33.363" Title="Follow up: In a mixed within-between ANOVA plot estimated SEs or actual SEs?" Tags="&lt;data-visualization&gt;&lt;anova&gt;&lt;mixed-model&gt;&lt;standard-error&gt;" AnswerCount="4" CommentCount="6" FavoriteCount="5" />
  <row Id="2068" PostTypeId="2" ParentId="2067" CreationDate="2010-08-24T10:35:37.933" Score="6" ViewCount="" Body="&lt;p&gt;You will not find a single reasonable error bar for inferential purposes with this type of experimental design.  This is an old problem with no clear solution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems impossible to have the estimate SE's you have here.  There are two main kinds of error in such a design, the between and within S error.  They are usually very different from one another and not comparable.  There just really is no good single error bar to represent your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One might argue that the raw SEs or SDs from the data are most important in a descriptive rather than inferential sense.  They either tell about the quality of the central tendency estimate (SE) or the variability of the data (SD). However, even then it's somewhat disingenuous because the thing you're testing and measuring within S is not that raw value but rather the effect of the within S variable.  Therefore, reporting variability of the raw values is either meaningless or misleading with respect to within S effects.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have typically endorsed no error bars on such graphs and adjacent effects graphs indicating the variability of the effects.  One might have CI's on that graph that are perfectly reasonable.  See Masson &amp;amp; Loftus (2003) for examples of the effects graphs.  Simply eliminate their ((pretty much completely useless) error bars around the mean values they show and just use the effect error bars.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For your study I'd first replot the data as the 2 x 2 x 2 design it is (2-panel 2x2) and then plot immediately adjacent a graph with confidence intervals of the validity, plausibility, instruction, and interaction effects.  Put SDs and SEs for the instruction groups in a table or in the text.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(waiting for expected mixed effects analysis response ;) )&lt;/p&gt;&#xA;&#xA;&lt;p&gt;UPDATE:  OK, after editing it's clear the only thing you want is an SE to be used to show the quality of the estimate of the value.  In that case use your model values.  Both values are based on a model and there is no 'true' value in your sample.  Use the ones from the model you applied to your data.  BUT, make sure you warn readers in the figure caption that these SEs have no inferential value whatsoever for your within S effects or interactions.&lt;/p&gt;&#xA;" OwnerUserId="601" LastEditorUserId="601" LastEditorDisplayName="" LastEditDate="2010-08-24T12:12:36.473" LastActivityDate="2010-08-24T12:12:36.473" CommentCount="1" />
  <row Id="2069" PostTypeId="2" ParentId="2067" CreationDate="2010-08-24T11:21:58.323" Score="6" ViewCount="" Body="&lt;p&gt;This looks like a very nice experiment, so congratulations!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I agree with John Christie, it is a mixed model, but provided it can be properely specified in an ANOVA design (&amp;amp; is balanced) I don't see why it can't be so formulated. Two factor within and 1-factor between subjects, but the between subjects factor (inductive/deductive) clearly interacts (modifies) the within-subjects effects. I assume the plotted means are from the ANOVA model (LHS) and so the model is correctly specified. Well done -  this is non-trivial! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some points:&#xA;1) The &quot;estimated&quot; vs &quot;actual&quot; &quot;error&quot; is a false dichotomy. Both assume an underlying model and make estimates on that basis. If the model is reasonable, I would argue it is better to use the model-based estimates (they are based on the pooling of larger samples). But as James mentions, the errors differ depending on the comparison you are making, so no simple representation is possible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) I would prefer to see box-plots or individual data points plotted (if there are not too many), perhaps with some sideways jitter, so points with the same value can be distinguished.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Box_plot&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Box_plot&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) If you must plot an estimate of the error of the mean, never plot SDs - they are an estimate of the standard deviation of the sample and relate to population variablility, not a statistical comparison of means. It is generally preferable to plot 95% confidence intervals rather than SEs, but not in this case (see 1 and John's point)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4) The one issue with this data that concerns me is the assumption of uniform variance is probably violated as the &quot;MP Valid and Plausible&quot; data are clearly constrained by the 100% limit, especially for the deductive people. I'm tossing up in my own mind how important this issue is. Moving to a mixed-effects logit (binomial probability) is probably the ideal solution, but it's a hard ask. It might be best to let others answer.&lt;/p&gt;&#xA;" OwnerUserId="521" LastEditorUserId="521" LastEditorDisplayName="" LastEditDate="2010-08-24T11:39:14.513" LastActivityDate="2010-08-24T11:39:14.513" CommentCount="4" />
  <row Id="2070" PostTypeId="2" ParentId="2067" CreationDate="2010-08-24T11:32:58.983" Score="1" ViewCount="" Body="&lt;p&gt;Lately I've been using mixed effects analysis, and in attempting to develop an accompanying visual data analysis approach I've been using bootstrapping (&lt;a href=&quot;https://stat.ethz.ch/pipermail/r-sig-mixed-models/2010q3/004369.html&quot; rel=&quot;nofollow&quot;&gt;see my description here&lt;/a&gt;), which yields confidence intervals that are not susceptible to the within-versus-between troubles of conventional CIs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, I would avoid mapping multiple variables to the same visual aesthetic, as you have done in the graph above; you have 3 variables (MP/AC, valid/invalid, plausible/implausible) mapped to the x-axis, which makes it rather difficult to parse the design and patterns. I would suggest instead mapping, say, MP/AC to the x-axis, valid/invalid to facet columns, and plausible/implausible to facet rows. Check out ggplot2 in R to easily achieve this, eg:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(ggplot2)&#xA;ggplot(&#xA;    data = my_data&#xA;    , mapping = aes(&#xA;        y = mean_endorsement&#xA;        , x = mp_ac&#xA;        , linetype = deductive_inductive&#xA;        , shape = deductive_inductive&#xA;)+&#xA;geom_point()+&#xA;geom_line()+&#xA;facet_grid(&#xA;    plausible_implausible ~ valid_invalid&#xA;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="364" LastActivityDate="2010-08-24T11:32:58.983" CommentCount="2" />
  <row Id="2071" PostTypeId="2" ParentId="2066" CreationDate="2010-08-24T12:38:01.413" Score="2" ViewCount="" Body="&lt;p&gt;A quick graphical experiment suggests that the beta distribution looks very like a normal distribution when alpha and beta are both very large. By googling &quot;beta distribution limit normal&quot; i found &lt;a href=&quot;http://nrich.maths.org/discus/messages/117730/143065.html?1200700623&quot; rel=&quot;nofollow&quot;&gt;http://nrich.maths.org/discus/messages/117730/143065.html?1200700623&lt;/a&gt;, which gives a handwaving 'proof'.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The wikipedia page for the beta distribution gives its mean, mode (v close to mean for large alpha and beta) and variance, so you could use a normal distribution with the same mean &amp;amp; variance to get an approximation. Whether it's a good enough approximation for your purposes depends on what your purposes are.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-08-24T12:38:01.413" CommentCount="3" />
  <row Id="2072" PostTypeId="1" CreationDate="2010-08-24T13:07:33.350" Score="9" ViewCount="189" Body="&lt;p&gt;It isn't clear to me how to calculate cointegration with irregular time series (ideally using &lt;a href=&quot;http://en.wikipedia.org/wiki/Johansen_test&quot; rel=&quot;nofollow&quot;&gt;the Johansen test&lt;/a&gt; with VECM).  My initial thought would be to regularize the series and interpolate missing values, although that may bias the estimation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any literature on this subject?&lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-08-30T17:42:31.317" Title="Does a cointegration model exist for irregularly spaced time series?" Tags="&lt;time-series&gt;&lt;cointegration&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="1" />
  <row Id="2073" PostTypeId="2" ParentId="2066" CreationDate="2010-08-24T15:06:00.023" Score="2" ViewCount="" Body="&lt;p&gt;I am going to infer you want an interval $[l,r]$ such that the probability that a random draw from the Beta RV is in the interval with probability 0.99, with bonus points for $l$ and $r$ being symmetric around the mode. By &lt;a href=&quot;http://en.wikipedia.org/wiki/Gauss%27s_inequality&quot; rel=&quot;nofollow&quot;&gt;Gauss' Inequality&lt;/a&gt; or the Vysochanskii-Petunin inequality, you can construct intervals that contain the interval $[l,r]$, and would be fairly decent approximations. For sufficiently large $\alpha, \beta$, you will have numerical underflow issues in even &lt;em&gt;representing&lt;/em&gt; $l$ and $r$ as distinct numbers, so this route may be good enough.&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-08-24T15:06:00.023" CommentCount="5" />
  <row Id="2074" PostTypeId="2" ParentId="2066" CreationDate="2010-08-24T15:43:19.670" Score="6" ViewCount="" Body="&lt;p&gt;A Normal approximation works extremely well, &lt;em&gt;especially&lt;/em&gt; in the tails.  Use a mean of alpha/(alpha+beta) and a variance of alpha*beta/((alpha+beta)^2*(1+alpha+beta)).  For example, the absolute relative error in the tail probability in a tough situation (where skewness might be of concern) such as alpha = 10^6, beta = 10^8 peaks around 0.00026 and is less than 0.00006 when you're more than 1 SD from the mean.  (This is &lt;em&gt;not&lt;/em&gt; because beta is so large: with alpha = beta = 10^6, the absolute relative errors are bounded by 0.0000001.)  Thus, this approximation is excellent for essentially any purpose involving 99% intervals.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In light of the edits to the question, note that one does not compute beta integrals by actually integrating the integrand: of course you'll get underflows (although they don't really matter, because they don't contribute appreciably to the integral).  There are many, many ways to compute the integral or approximate it, as documented in Johnson &amp;amp; Kotz (Distributions in Statistics).  An online calculator is found at &lt;a href=&quot;http://www.danielsoper.com/statcalc/calc37.aspx&quot; rel=&quot;nofollow&quot;&gt;http://www.danielsoper.com/statcalc/calc37.aspx&lt;/a&gt; .  You actually need the inverse of this integral.  Some methods to compute the inverse are documented on the Mathematica site at &lt;a href=&quot;http://functions.wolfram.com/GammaBetaErf/InverseBetaRegularized/&quot; rel=&quot;nofollow&quot;&gt;http://functions.wolfram.com/GammaBetaErf/InverseBetaRegularized/&lt;/a&gt; .  Code is provided in Numerical Recipes (www.nr.com).  A really nice online calculator is the Wolfram Alpha site (www.wolframalpha.com ): enter &quot;inverse beta regularized (.005, 1000000, 1000001)&quot; for the left endpoint and &quot;inverse beta regularized (.995, 1000000, 1000001)&quot; for the right endpoint (alpha=1000000, beta=1000001, 99% interval).&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-08-24T17:16:41.757" LastActivityDate="2010-08-24T17:16:41.757" CommentCount="1" />
  <row Id="2075" PostTypeId="2" ParentId="927" CreationDate="2010-08-24T16:17:53.823" Score="1" ViewCount="" Body="&lt;p&gt;I also just realized the &lt;a href=&quot;http://itunes.apple.com/us/podcast/freakonomics-radio/id354668519&quot; rel=&quot;nofollow&quot;&gt;freakonomics has a podcast&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="253" LastActivityDate="2010-08-24T16:17:53.823" CommunityOwnedDate="2010-08-24T16:17:53.823" />
  <row Id="2076" PostTypeId="2" ParentId="924" CreationDate="2010-08-24T17:08:00.637" Score="0" ViewCount="" Body="&lt;p&gt;If you wanted to check non-parametrically for significance, you could bootstrap the confidence intervals on the ratio, or you could do a permutation test on the two classes. For example, to do the bootstrap, create two arrays: one with 3 ones and 999,997 zeros, and one with 10 ones and 999,990 zeros. Then draw with replacement a sample of 1m items from the first population and a sample of 1m items from the second population. The ratio we're interested in is the ratio of &quot;hits&quot; in the first group to the ratio of &quot;hits&quot; in the second group, or: (proportion of ones in the first sample) / (proportion of ones in the second sample). We do this 1000 times. I don't have matlab handy but here's the R code to do it:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# generate the test data to sample from&#xA;v1 &amp;lt;- c(rep(1,3),rep(0,999997))&#xA;v2 &amp;lt;- c(rep(1,10),rep(0,999990))&#xA;# set up the vectors that will hold our proportions&#xA;t1 &amp;lt;- vector()&#xA;t2 &amp;lt;- vector()&#xA;# loop 1000 times each time sample with replacement from the test data and&#xA;# record the proportion of 1's from each sample&#xA;# note: this step takes a few minutes. There are ways to write it such that&#xA;# it will go faster in R (applies), but it's more obvious what's going on this way:&#xA;for(i in 1:1000) {&#xA;   t1[i] &amp;lt;- length(which(sample(v1,1000000,replace=TRUE)==1)) / 1000000&#xA;   t2[i] &amp;lt;- length(which(sample(v2,1000000,replace=TRUE)==1)) / 1000000&#xA;}&#xA;# what was the ratio of the proportion of 1's between each group for each random draw?&#xA;ratios &amp;lt;- t1 / t2&#xA;# grab the 95% confidence interval over the bootstrapped samples&#xA;quantile(ratios,c(.05,.95))&#xA;# and the 99% confidence interval&#xA;quantile(ratios,c(.01,.99))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The output is:&#xA;    5%       95% &#xA;    0.0000000 0.8333333 &#xA;and:&#xA;    1%  99% &#xA;    0.00 1.25 &#xA;Since the 95% confidence interval doesn't overlap the null hypothesis (1), but the 99% confidence interval does, I believe that it would be correct to say that this is significant at an alpha of .05 but not at .01. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another way to look at it is with a permutation test to estimate the distribution of ratios given the null hypothesis. In this case you'd mix the two samples together and randomly divide it into two 1,000,000 item groups. Then you'd see what the distribution of ratios under the null hypothesis looks like, and your empirical p-value is how extreme the true ratio is given this distribution of null ratios. Again, the R code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# generate the test data to sample from&#xA;v1 &amp;lt;- c(rep(1,3),rep(0,999997))&#xA;v2 &amp;lt;- c(rep(1,10),rep(0,999990))&#xA;v3 &amp;lt;- c(v1,v2)&#xA;&#xA;# vectors to hold the null hypothesis ratios&#xA;t1 &amp;lt;- vector()&#xA;t2 &amp;lt;- vector()&#xA;&#xA;# loop 1000 times; each time randomly divide the samples&#xA;# into 2 groups and see what those two random groups' proportions are&#xA;for(i in 1:1000) {&#xA;  idxs &amp;lt;- sample(1:2000000,1000000,replace=FALSE)&#xA;  s1 &amp;lt;- v3[idxs]&#xA;  s2 &amp;lt;- v3[-idxs]&#xA;  t1[i] &amp;lt;- length(which(s1==1)) / 1000000&#xA;  t2[i] &amp;lt;- length(which(s2==1)) / 1000000&#xA;}&#xA;&#xA;# vector of the ratios&#xA;ratios &amp;lt;- t1 / t2&#xA;&#xA;# take a look at the distribution&#xA;plot(density(ratios))&#xA;&#xA;# calculate the sampled ratio of proportions&#xA;sample.ratio &amp;lt;- ((3/1000000)/(10/1000000))&#xA;# where does this fall on the distribution of null proportions?&#xA;plot(abline(v=sample.ratio))&#xA;&#xA;# this ratio (r+1)/(n+1) gives the p-value of the true sample&#xA;(length(which(ratios &amp;lt;= sample.ratio)) + 1) / (1001)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The output is ~ .0412 (of course this will vary run to run since it's based on random draws). So again, you could potentially call this significant at the .05 value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I should issue the caveats: it depends too on how your data was collected and the type of study, and I'm just a grad student so don't take my word as gold. If anyone has any criticism of my methods I'd love to hear them since I'm doing this stuff for my own work as well and I'd love to find out the methods are flawed here rather than in peer review. For more stuff like this check out Efron &amp;amp; Tibshirani 1993, or chapter 14 of Introduction to the Practice of Statistics by David Moore (a good general textbook for practitioners).&lt;/p&gt;&#xA;" OwnerUserId="1076" LastActivityDate="2010-08-24T17:08:00.637" CommentCount="1" />
  <row Id="2077" PostTypeId="1" AcceptedAnswerId="2080" CreationDate="2010-08-24T18:40:42.633" Score="14" ViewCount="714" Body="&lt;p&gt;Besides taking differences, what are other techniques for making a non-stationary time series, stationary?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ordinarily one refers to a series as &quot;&lt;a href=&quot;http://en.wikipedia.org/wiki/Order_of_integration&quot; rel=&quot;nofollow&quot;&gt;integrated of order p&lt;/a&gt;&quot; if it can be made stationary through a lag operator $(1-L)^P X_t$.&lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2011-03-03T19:41:27.587" Title="How to make a time series stationary?" Tags="&lt;time-series&gt;&lt;stationarity&gt;" AnswerCount="5" FavoriteCount="3" />
  <row Id="2078" PostTypeId="2" ParentId="1927" CreationDate="2010-08-24T19:25:14.850" Score="1" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Geometric_median&quot; rel=&quot;nofollow&quot;&gt;Geometric median&lt;/a&gt; is the point with the smallest average euclidian distance from the samples&lt;/p&gt;&#xA;" OwnerUserId="511" LastActivityDate="2010-08-24T19:25:14.850" />
  <row Id="2079" PostTypeId="1" CreationDate="2010-08-24T20:09:45.430" Score="1" ViewCount="45" Body="&lt;p&gt;There seems to be a large body of applied research where distribution q is picked to minimize &lt;a href=&quot;http://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence&quot; rel=&quot;nofollow&quot;&gt;KL(q,p)&lt;/a&gt; where p is empirical distribution. Are there theoretical reasons to prefer this estimator? For instance, a theoretical reason to prefer MLE for estimating mean of a one-dimensional Gaussian with mean between -1 and 1 because it's admissible, and also as sample size goes to infinity, it minimizes maximum risk and Bayes risk.&lt;/p&gt;&#xA;" OwnerUserId="511" LastActivityDate="2010-08-24T20:09:45.430" Title="What's good about I-projections?" Tags="&lt;estimation&gt;" />
  <row Id="2080" PostTypeId="2" ParentId="2077" CreationDate="2010-08-24T21:27:09.350" Score="6" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;De-trending&lt;/strong&gt; is fundamental.  This includes regressing against covariates other than time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Seasonal adjustment&lt;/strong&gt; is a version of taking differences but could be construed as a separate technique.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Transformation&lt;/strong&gt; of the data implicitly converts a difference operator into something else; e.g., differences of the logarithms are actually ratios.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some &lt;strong&gt;EDA smoothing&lt;/strong&gt; techniques (such as removing a moving median) could be construed as non-parametric ways of detrending.  They were used as such by Tukey in his book on EDA.  Tukey continued by detrending the residuals and iterating this process for as long as necessary (until he achieved residuals that appeared stationary and symmetrically distributed around zero).&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-08-24T21:27:09.350" />
  <row Id="2081" PostTypeId="2" ParentId="1681" CreationDate="2010-08-24T22:19:51.203" Score="1" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;in that notation, I'm looking for epsilon as a function of m &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;If I have not misunderstood, this indicates you want to solve an equation in q (= p + epsilon) of the form&lt;/p&gt;&#xA;&#xA;&lt;p&gt;q ln(p/q) + (1-q) ln((1-p)/(1-q)) = D(q||p) = y; y &amp;lt; 0,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;given p and y.  (y = ln(p)/m reveals the m-dependence.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is correct that we do not have a name for the solution, but that does not mean it cannot be found, and fairly easily at that.  A little algebra converts this equation into one of the form&lt;/p&gt;&#xA;&#xA;&lt;p&gt;H(q) = alpha + beta*q where&lt;/p&gt;&#xA;&#xA;&lt;p&gt;H(q) = -q ln(q) - (1-q) ln(1-q) and alpha and beta depend only on y (&quot;probability of deviation&quot;) and m (&quot;sample size&quot;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Geometrically this asks for the intersections of a concave downward curve and a line; the curve has endpoints at (0,0) and (1,0) and is symmetric in the unit interval.  There will therefore be up to two solutions.  Newton-Raphson ought to converge rapidly after an initial set of bisection steps finds a point between 0 and the left root and another between the right root and 1 (assume either exist).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you need them, theoretical properties of the solution(s) could be readily derived from the definitions of H, alpha, and beta.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-08-24T22:19:51.203" CommentCount="1" />
  <row Id="2082" PostTypeId="2" ParentId="2077" CreationDate="2010-08-24T22:38:15.263" Score="3" ViewCount="" Body="&lt;p&gt;Difference with another series. i.e. Brent oil prices are not stationary,&#xA;but the spread brent-light sweet crude is. A more risky proposition for &#xA;forecasting is to bet on the existence of a co integration relationship&#xA;with another time series.&lt;/p&gt;&#xA;" OwnerUserId="603" LastActivityDate="2010-08-24T22:38:15.263" />
  <row Id="2083" PostTypeId="2" ParentId="33" CreationDate="2010-08-24T22:46:09.307" Score="0" ViewCount="" Body="&lt;p&gt;Is the OP asking for a way to find the seasonality of a series? In my opinion, that is a quite complex question, and none of the above answers (I think) address that. Please correct me if I am worng, either about the question or the answer.&lt;/p&gt;&#xA;" OwnerUserId="528" LastActivityDate="2010-08-24T22:46:09.307" />
  <row Id="2084" PostTypeId="2" ParentId="1469" CreationDate="2010-08-25T00:46:17.397" Score="1" ViewCount="" Body="&lt;p&gt;&quot;Asymptotic Statistics&quot; by A. W. van der Vaart seems to serve the purpose&lt;/p&gt;&#xA;" OwnerUserId="511" LastActivityDate="2010-08-25T00:46:17.397" />
  <row Id="2085" PostTypeId="1" AcceptedAnswerId="2115" CreationDate="2010-08-25T01:57:42.433" Score="5" ViewCount="551" Body="&lt;p&gt;I am modelling a nested mixed-effects model with just the intercept in the random part, of the form:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;fit4&amp;lt;-lme(fixed = Stdquad~factor(LayoutN)+factor(nCHIPS.fixed), random = ~1|Class.Ordered/student)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When trying to check the assumption on the independent, identically distributed (iid) random effects in SPlus using &lt;code&gt;ranef()&lt;/code&gt;, I keep getting the error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Problem in sort.list: ordering not defined for mode &quot;list&quot;: sort.list(x, partial)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I suspect the nesting is the problem since if I remove &lt;em&gt;student&lt;/em&gt;, the plots come up fine. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to know how I can check that the random effects are iid and are independent for different groups using S-Plus (or R since its very similar).&lt;/p&gt;&#xA;" OwnerUserId="108" LastEditorUserId="108" LastEditorDisplayName="" LastEditDate="2010-08-25T09:53:10.710" LastActivityDate="2010-08-25T19:45:27.547" Title="Checking assumptions for random effects in nested mixed-effects models in R / S-Plus" Tags="&lt;r&gt;&lt;mixed-model&gt;&lt;error&gt;&lt;random-effects-model&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="4" />
  <row Id="2086" PostTypeId="1" AcceptedAnswerId="2087" CreationDate="2010-08-25T03:16:23.030" Score="5" ViewCount="462" Body="&lt;p&gt;I'm looking for correlations between the answers to different questions in a survey (&quot;umm, let's see if answers to question 11 correlate with those of question 78&quot;). All answers are categorical (most of them range from &quot;very unhappy&quot; to &quot;very happy&quot;), but a few have a different set of answers. Most of them can be considered ordinal so let's consider this  case here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since I don't have access to a commercial statistics program, I must use R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried &lt;a href=&quot;http://rattle.togaware.com/&quot; rel=&quot;nofollow&quot;&gt;Rattle&lt;/a&gt; (a freeware data mining package for R, very nifty) but unfortunately it doesn't support categorical data. One hack I could use is to import in R the coded version of the survey which has numbers (1..5) instead of &quot;very unhappy&quot; ... &quot;happy&quot; and let Rattle believe they are numerical data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was thinking to do a scatter plot and have the dot size proportional to the number of numbers for each pair. After some googling I found &lt;a href=&quot;http://www.r-statistics.com/2010/04/correlation-scatter-plot-matrix-for-ordered-categorical-data/&quot; rel=&quot;nofollow&quot;&gt;http://www.r-statistics.com/2010/04/correlation-scatter-plot-matrix-for-ordered-categorical-data/&lt;/a&gt; but it seems very complicated (to me).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not a statistician (but a programmer) but have had some reading in the matter and, if I understand correctly, &lt;em&gt;Spearman's rho&lt;/em&gt; would be appropriate here. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the short version of the question for those in a hurry: &lt;strong&gt;is there a way to quickly plot &#xA;Spearman's rho in R&lt;/strong&gt;? A plot is preferable to a matrix of numbers because it's easier to eye ball and also can be included in materials.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you in advance. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS I pondered for a while whether to post this on the main SO site or here. After searching both sites for R correlation, I felt this site is better suited for the question.&lt;/p&gt;&#xA;" OwnerUserId="840" LastEditorUserId="840" LastEditorDisplayName="" LastEditDate="2010-08-25T04:05:22.290" LastActivityDate="2010-08-25T04:50:17.533" Title="Quickly evaluate (visually) correlations between ordered categorical data in R?" Tags="&lt;r&gt;&lt;correlation&gt;&lt;categorical-data&gt;&lt;data-visualization&gt;" AnswerCount="2" CommentCount="4" FavoriteCount="4" />
  <row Id="2087" PostTypeId="2" ParentId="2086" CreationDate="2010-08-25T03:32:04.410" Score="11" ViewCount="" Body="&lt;p&gt;Hello wishihadabettername,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another good visualization of correlation is offered by the &lt;a href=&quot;http://cran.r-project.org/web/packages/corrplot/index.html&quot; rel=&quot;nofollow&quot;&gt;corrplot&lt;/a&gt; package, giving you things like this:&#xA;&lt;img src=&quot;http://i.imgur.com/IsxzL.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is a great package.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also have a look at the &lt;a href=&quot;http://stats.stackexchange.com/questions/1961/can-you-use-normal-correlation-for-vectors-with-only-2-or-3-ordered-levels&quot;&gt;answer here&lt;/a&gt;, it might be good for you to know.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lastly, if you have suggestions how the code on the post you referred to could be simpler - please let me know.&lt;/p&gt;&#xA;" OwnerUserId="253" LastActivityDate="2010-08-25T03:32:04.410" CommentCount="2" />
  <row Id="2088" PostTypeId="2" ParentId="2086" CreationDate="2010-08-25T04:50:17.533" Score="2" ViewCount="" Body="&lt;p&gt;A couple of additional plotting ideas are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://addictedtor.free.fr/graphiques/RGraphGallery.php?graph=59&quot; rel=&quot;nofollow&quot;&gt;Sunflower plot&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Scatter plot with a jitter using &lt;a href=&quot;http://stat.ethz.ch/R-manual/R-patched/library/base/html/jitter.html&quot; rel=&quot;nofollow&quot;&gt;base graphics&lt;/a&gt; or &lt;a href=&quot;http://had.co.nz/ggplot2/geom_jitter.html&quot; rel=&quot;nofollow&quot;&gt;ggplot2&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-08-25T04:50:17.533" CommentCount="3" />
  <row Id="2089" PostTypeId="2" ParentId="1595" CreationDate="2010-08-25T05:19:46.727" Score="12" ViewCount="" Body="&lt;p&gt;First, let me say I agree with John D Cook's answer: Python is not a Domain Specific Language like R, and accordingly, there is a lot more you'll be able to do with it further down the road. Of course, R being a DSL means that the latest algorithms published in JASA will almost certainly be in R. If you are doing mostly ad hoc work and want to experiment with the latest lasso regression technique, say, R is hard to beat. If you are doing more production analytical work, integrating with existing software and environments, and concerned about speed, extensibility and maintainability, Python will serve you much better. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second, ars gave a great answer with good links. Here are a few more packages that I view as essential to analytical work in Python:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://matplotlib.sourceforge.net/gallery.html&quot; rel=&quot;nofollow&quot;&gt;matplotlib&lt;/a&gt; for beautiful, publication quality graphics. &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://ipython.scipy.org/moin/&quot; rel=&quot;nofollow&quot;&gt;IPython&lt;/a&gt; for an enhanced, interactive Python console. Importantly, IPython provides a powerful framework for interactive, parallel computing in Python. &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.cython.org/&quot; rel=&quot;nofollow&quot;&gt;Cython&lt;/a&gt; for easily writing C extensions in Python. This package lets you take a chunk of computationally intensive Python code and easily convert it to a C extension. You'll then be able to load the C extension like any other Python module but the code will run very fast since it is in C.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.vni.com/campaigns/pyimslstudioeval/&quot; rel=&quot;nofollow&quot;&gt;PyIMSL Studio&lt;/a&gt; for a collection of hundreds of mathemaical and statistical algorithms that are thoroughly documented and supported. You can call the exact same algorithms from Python and C, with nearly the same API and you'll get the same results. Full disclosure: I work on this product, but I also use it a lot. &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://pypi.python.org/pypi/xlrd&quot; rel=&quot;nofollow&quot;&gt;xlrd&lt;/a&gt; for reading in Excel files easily.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If you want a more MATLAB-like interactive IDE/console, check out &lt;a href=&quot;http://code.google.com/p/spyderlib/&quot; rel=&quot;nofollow&quot;&gt;Spyder&lt;/a&gt;, or the &lt;a href=&quot;http://pydev.org/&quot; rel=&quot;nofollow&quot;&gt;PyDev&lt;/a&gt; plugin for &lt;a href=&quot;http://www.eclipse.org/&quot; rel=&quot;nofollow&quot;&gt;Eclipse&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1080" LastEditorUserId="1080" LastEditorDisplayName="" LastEditDate="2010-08-25T17:42:00.907" LastActivityDate="2010-08-25T17:42:00.907" CommunityOwnedDate="2010-08-27T15:07:18.913" CommentCount="2" />
  <row Id="2090" PostTypeId="2" ParentId="2061" CreationDate="2010-08-25T05:38:05.883" Score="3" ViewCount="" Body="&lt;p&gt;ASA Sections on: Statistical Computing Statistical Graphics has a video library:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://stat-graphics.org/movies/&quot; rel=&quot;nofollow&quot;&gt;http://stat-graphics.org/movies/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;It contains a large number of interesting videos relevant to statistical computing and graphics. The videos go back as far as the 1960s.&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-08-25T05:38:05.883" CommunityOwnedDate="2010-08-25T08:02:51.643" CommentCount="1" />
  <row Id="2091" PostTypeId="1" AcceptedAnswerId="2103" CreationDate="2010-08-25T08:22:06.220" Score="3" ViewCount="606" Body="&lt;p&gt;X is uniform random variable in [0,1] and Y=1-X. How do I calculate the distribution function F(X,Y)? I can see that Y is also uniformly distributed and can draw the intervals. But I am unable to compute the distribution function when x+y&gt;1 &amp;amp; x,y in [0,1]. How do I compute it for this specific case?&lt;/p&gt;&#xA;" OwnerUserId="862" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-10-14T02:31:58.780" LastActivityDate="2010-10-14T14:44:02.327" Title="Computing probability distribution function for uniform random variables and Y=1-X" Tags="&lt;distributions&gt;&lt;probability&gt;&lt;copula&gt;" AnswerCount="2" CommentCount="0" FavoriteCount="1" />
  <row Id="2092" PostTypeId="1" AcceptedAnswerId="2094" CreationDate="2010-08-25T08:33:14.610" Score="3" ViewCount="1741" Body="&lt;p&gt;The waiting times for poisson distribution is an exponential distribution with parameter lambda. But I don't understand it. Poisson models the number of arrivals per unit of time for example. How is this related to exponential distribution? Lets say probability of k arrivals in a unit of time is P(k) (modeled by poisson) and probability of k+1 is P(k+1), how does exponential distribution model the waiting time between them?&lt;/p&gt;&#xA;" OwnerUserId="862" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-18T21:58:05.727" LastActivityDate="2010-09-18T21:58:05.727" Title="Relationship between poisson and exponential distribution" Tags="&lt;distributions&gt;&lt;poisson&gt;&lt;exponential&gt;" AnswerCount="2" />
  <row Id="2093" PostTypeId="1" AcceptedAnswerId="2097" CreationDate="2010-08-25T09:32:22.513" Score="0" ViewCount="289" Body="&lt;p&gt;I've implemented the &lt;a href=&quot;http://en.wikipedia.org/wiki/Marsaglia_polar_method&quot; rel=&quot;nofollow&quot;&gt;Marsaglia polar method&lt;/a&gt; to generate random numbers that are normally distributed. Unfortunately the code as shown on this &lt;a href=&quot;http://www.taygeta.com/random/gaussian.html&quot; rel=&quot;nofollow&quot;&gt;website&lt;/a&gt; returns numbers that are not within the range [0, 1).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;     double x1, y1, w;&#xA;&#xA;     do {&#xA;             x1 = 2.0 * ranf() - 1.0;&#xA;             y1 = 2.0 * ranf() - 1.0;&#xA;             w = x1 * x1 + y1 * y1;&#xA;     } while ( w &amp;gt;= 1.0 || w == 0 );&#xA;&#xA;     w = sqrt( (-2.0 * ln( w ) ) / w );&#xA;     return x1 * w;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where ranf() returns values [0, 1).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if (x1, y1) was (-0.43458512585358, -0.07521858582050478), this returns a value of -1.7830255550765148. Obviously not a value within my expected range.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've seen some implementations that multiplies this by the standard deviation and adds the mean. But if I want to get numbers that range from [0, 1), what should I use as my standard deviation and mean? currently, I'm using a standard deviation of 1/8.125 ~= 0.121212... and 0.5 as the mean, but I've only stumbled on this by experimentation. What is the official way for me to get this appropriately ranged? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To answer the question in the comments below: I want to generate normal random variables within the range 0 and 1. Obviously I'd like the mean to be at 0.5 and values no be distributed around this mean.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More for my enlightenment, will the polar method really return numbers over the entire real line? If so, then it should be easy for me to scale the result down to [0, 1), assuming I don't hit floating point underflow due to the division to scale things down.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;New edit:&#xA;What I'm trying to do is take Aniko's advise in his answer to my other &lt;a href=&quot;http://stats.stackexchange.com/questions/1687/applied-statistics-to-find-the-minimum-load-for-power-factor-floor&quot;&gt;Power Factor Problem question&lt;/a&gt;. So my first thought was that simulate my distribution of velocities, was to implement my on GaussianRandom class by deriving from &lt;a href=&quot;http://msdn.microsoft.com/en-us/library/ts6se2ek.aspx&quot; rel=&quot;nofollow&quot;&gt;Random&lt;/a&gt; and overriding &lt;a href=&quot;http://msdn.microsoft.com/en-us/library/system.random.sample.aspx&quot; rel=&quot;nofollow&quot;&gt;Random.Sample()&lt;/a&gt; and the 3 other methods as recommended by MSDN. Since Random.Sample() is suppose to return a double in the range [0, 1), then I've hit this stumbling block.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've come to the same conclusion as mbq's answer after staring at the ln(x) graph where x is in (0, 1] and sqrt(x) where x is in (0, 2].&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From a practical/programming standpoint, I can't override Random.Sample(), because later when I actually call GaussianRandom.NextDouble(), there is no way for me to pass in the actual standard deviations for my velocities. It looks like I'll have to use the Marsaglia polar method in the raw instead of wrapped within the Random class. My &quot;cartridge&quot; generator will have to &lt;code&gt;return x1 * w * stdDev + mean&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="937" LastEditorUserId="937" LastEditorDisplayName="" LastEditDate="2010-08-25T20:49:17.453" LastActivityDate="2010-08-25T21:15:38.573" Title="How to make Marsaglia polar method return values [0, 1) ?" Tags="&lt;random-generation&gt;&lt;normal-distribution&gt;" AnswerCount="4" CommentCount="7" />
  <row Id="2094" PostTypeId="2" ParentId="2092" CreationDate="2010-08-25T09:43:51.450" Score="4" ViewCount="" Body="&lt;p&gt;I will use the following notation to be as consistent as possible with the wiki (in case you want to go back and forth between my answer and the wiki definitions for the &lt;a href=&quot;http://en.wikipedia.org/wiki/Poisson_distribution&quot; rel=&quot;nofollow&quot;&gt;poisson&lt;/a&gt; and &lt;a href=&quot;http://en.wikipedia.org/wiki/Exponential_distribution&quot; rel=&quot;nofollow&quot;&gt;exponential&lt;/a&gt;.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$N_t$: the number of arrivals during time period $t$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$X_t$: the time it takes for one additional arrival to arrive assuming that someone arrived at time $t$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By definition, the following conditions are equivalent:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ (X_t &gt; x) \equiv (N_t = N_{t+x})$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The event on the left captures the event that no one has arrived in the time interval $[t,t+x]$ which implies that our count of the number of arrivals at time $t+x$ is identical to the count at time $t$ which is the event on the right.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, it follows that:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(X_t &amp;lt; x) = 1 - P(X_t &gt; x)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using the equivalence of the two events that we described above, we can re-write the above as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(X_t &amp;lt; x) = 1 - P(N_{t+x} - N_t = 0)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(N_{t+x} - N_t = 0) = P(N_x = 0)$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using the poisson pmf the above simplifies to:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(N_{t+x} - N_t = 0) = e^{-\lambda x}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Substituting in our original eqn, we have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(X_t &amp;lt; x) = 1 - e^{-\lambda x}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The above is the cdf of a exponential pdf.&lt;/p&gt;&#xA;" LastActivityDate="2010-08-25T09:43:51.450" CommentCount="1" />
  <row Id="2095" PostTypeId="2" ParentId="2091" CreationDate="2010-08-25T10:01:31.670" Score="3" ViewCount="" Body="&lt;p&gt;$F(x,y) = P(X \le x, Y \le y)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using the fact that $Y= 1-X$ and after simplifying, the above can be re-written as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$F(x,y) = P(1-y \le X \le x)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since, $X$ is a uniform between 0 and 1, it follows that:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$F(x,y) = x+y-1$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The above is a bit sloppy as I did not specify the domain of the cdf where it is non-zero appropriately. Consider:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$F(x,y) = P(1-y \le X \le x)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$F(x,y) = x+y-1$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Two points about the above cdf:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;For the above cdf to be non-zero it must be the case that:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$x &gt; 1-y$&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Also, note that the cdf attains the value of 1 when $x+y=2$.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Thus, the correct way to represent the cdf is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$F(x,y) = 0$ if $x+y \le 1$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$F(x,y) = x+y-1$ if $1 \le x+y \le 2$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$F(x,y) = 1$ otherwise&lt;/p&gt;&#xA;" LastActivityDate="2010-10-14T14:44:02.327" CommentCount="2" />
  <row Id="2096" PostTypeId="2" ParentId="2092" CreationDate="2010-08-25T10:58:52.970" Score="4" ViewCount="" Body="&lt;p&gt;For a Poisson process, hits occur at random independent of the past, but with a known long term average rate $\lambda$ of hits per unit time. The Poisson distribution would let us find the probability of getting some particular number of hits.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, instead of looking at the number of hits, we look at the random variable $L$ (for Lifetime), the time you have to wait for the first hit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The probability that the waiting time is more than a given time value is&#xA;$P(L \gt t)  = P(\text{no hits in time t})=\frac{\Lambda^0e^{-\Lambda}}{0!}=e^{-\lambda t}$ (by the Poisson distribution, where $\Lambda = \lambda t$).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(L \le t) = 1 - e^{-\lambda t}$ (the cumulative distribution function). &#xA;We can get the density function by taking the derivative of this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$f(x) = &#xA;\begin{cases} &#xA; \lambda e^{-\lambda t} &amp;amp; \mbox{for } t \ge 0 \\ 0 &amp;amp; \mbox{for } t \lt 0&#xA;\end{cases}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any random variable that has a density function like this is said to be exponentially distributed.&lt;/p&gt;&#xA;" OwnerUserId="339" LastActivityDate="2010-08-25T10:58:52.970" />
  <row Id="2097" PostTypeId="2" ParentId="2093" CreationDate="2010-08-25T11:01:38.983" Score="3" ViewCount="" Body="&lt;p&gt;It is just impossible, since normal distribution has nonzero probability for the whole $R$. Of course because of the numerical issues it would be not the case for Marsaglia method (this obviously means that it is not creating normally distributed numbers), yet this numerical range is so huge that it is nonsense to scale it to [0,1]. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: So, as long as I understand, you want this number to be in [0,1) to overload .net Random; you don't need to! Write your own class, say NormalRandom which will hold one Random object and will use its output to generate (using Marsaglia) normal random numbers from $-\infty$ to $\infty$ and just use them. Or rather use something already done; Google found &lt;a href=&quot;http://www.codeproject.com/KB/recipes/Random.aspx&quot; rel=&quot;nofollow&quot;&gt;http://www.codeproject.com/KB/recipes/Random.aspx&lt;/a&gt; .&lt;br&gt;&#xA;On margin, remember that when it is said that something which cannot be negative is &quot;normally distributed&quot; (for instance mass) it does mean that it is not from normal but from &quot;almost-normal&quot; distribution which looks pretty normal, but still has one tail cut out (nevertheless simple rejection of negative outputs from generator is a fair way to deal with it).   &lt;/p&gt;&#xA;" OwnerUserId="88" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-08-25T21:15:38.573" LastActivityDate="2010-08-25T21:15:38.573" CommentCount="3" />
  <row Id="2098" PostTypeId="2" ParentId="2093" CreationDate="2010-08-25T12:25:39.347" Score="1" ViewCount="" Body="&lt;p&gt;Based on the update you want to generate truncated normal random variables. Scaling based on the maximum and minimum will 'destroy' the normal distribution properties of the draws. Instead, you should consider using one of the two methods described below:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Rejection_sampling&quot; rel=&quot;nofollow&quot;&gt;Rejection Sampling&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You draw from the standard normal and then accept the draw only if it is between 0 and 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Inverse_transform_sampling&quot; rel=&quot;nofollow&quot;&gt;Inverse Transform Sampling&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\phi(x)$ be the standard normal cdf.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then a standard normal draw that is restricted to lie between 0 and 1 is given by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$x = \phi^{-1}(\phi(0) + (\phi(1) - \phi(0)) U )$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$U$ is a uniform random draw between 0 and 1.&lt;/p&gt;&#xA;" LastActivityDate="2010-08-25T12:25:39.347" CommentCount="6" />
  <row Id="2099" PostTypeId="1" AcceptedAnswerId="2100" CreationDate="2010-08-25T13:10:22.887" Score="5" ViewCount="317" Body="&lt;p&gt;Is there any free desktop software for &lt;a href=&quot;http://en.wikipedia.org/wiki/Treemapping&quot; rel=&quot;nofollow&quot;&gt;treemapping&lt;/a&gt;? &lt;a href=&quot;http://en.wikipedia.org/wiki/List_of_treemapping_software&quot; rel=&quot;nofollow&quot;&gt;Every one I've found&lt;/a&gt; appears to have a commercial license. &lt;a href=&quot;http://manyeyes.alphaworks.ibm.com/manyeyes/&quot; rel=&quot;nofollow&quot;&gt;Many Eyes&lt;/a&gt; is free and has treemapping features but what I'm really looking for is downloadable desktop software.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="666" LastActivityDate="2010-08-25T16:31:08.917" Title="Free treemapping software" Tags="&lt;data-visualization&gt;&lt;software&gt;" AnswerCount="3" FavoriteCount="1" />
  <row Id="2100" PostTypeId="2" ParentId="2099" CreationDate="2010-08-25T13:15:48.387" Score="3" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://flowingdata.com/2010/02/11/an-easy-way-to-make-a-treemap/&quot; rel=&quot;nofollow&quot;&gt;Flowing Data has a tutorial&lt;/a&gt; on how to use the &lt;code&gt;map.market&lt;/code&gt; function in the &lt;code&gt;portfolio&lt;/code&gt; package in R.&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-08-25T13:15:48.387" CommentCount="1" />
  <row Id="2101" PostTypeId="2" ParentId="2099" CreationDate="2010-08-25T13:31:32.787" Score="4" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Protovis&lt;/strong&gt; has an &lt;a href=&quot;http://vis.stanford.edu/protovis/ex/treemap.html&quot; rel=&quot;nofollow&quot;&gt;nice treemap layout&lt;/a&gt;.  I could try to add this into &lt;a href=&quot;http://cran.r-project.org/web/packages/webvis/index.html&quot; rel=&quot;nofollow&quot;&gt;webvis&lt;/a&gt; if you want to create it from R, but it isn't currently an option.&lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-08-25T13:31:32.787" />
  <row Id="2102" PostTypeId="2" ParentId="2093" CreationDate="2010-08-25T13:36:22.150" Score="2" ViewCount="" Body="&lt;p&gt;Another possible interpretation is that you are confusing &quot;normal&quot; and &quot;uniform&quot;.  A uniform variate will have a mean of 0.5 and be evenly distributed on [0, 1).  If this interpretation is correct, your code simplifies to&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;return ranf()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-08-25T13:36:22.150" CommentCount="2" />
  <row Id="2103" PostTypeId="2" ParentId="2091" CreationDate="2010-08-25T14:08:50.823" Score="3" ViewCount="" Body="&lt;p&gt;I'm interpreting your question as concerning the cdf, which by definition is a function $F$ for which&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$F(x,y) - F(x,0) - F(0,y) + F(0,0) = \Pr(X \le x, Y \le y) \text{;}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$F(0,0) = 0.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For $x + y \lt 1$, the right hand side is zero and the left hand side becomes a statement that $F$ is a bilinear function, implying (in conjunction with some of the initial values specified below) that the graph of $F$ is part of a plane.  For $x + y \gt 1$, the assumption of uniform distributions implies $F$ must be increasing at a unit rate in both $x$ and $y$, whence the graph of $F$ in this region is a part of a plane of the form $x + y = \text{constant}$.  From the evident restrictions&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$F(x, 0) = F(0, y) = 0,$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$0 \le x  \le 1, 0  \le y  \le 1,$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;it is geometrically obvious that the first piece of the graph must lie in the xy plane and the second piece must intersect the first along the line segment $x + y = 1, 0  \le x  \le 1$.  The full solution therefore is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$F(x,y) = 0 \quad \text{if} \quad x + y  \le 1$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$F(x,y) = x + y - 1 \quad \text{if} \quad x + y \gt 1.$$&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;This result is the &lt;a href=&quot;http://en.wikipedia.org/wiki/Copula_%28statistics%29&quot; rel=&quot;nofollow&quot;&gt;Fréchet–Hoeffding minimum copula&lt;/a&gt; $W(x,y)$.  Generally, a &lt;em&gt;copula&lt;/em&gt; expresses a multivariate distribution after the marginal variables have been subjected to a probability integral transformation; that is, the marginals have been made uniform.  All 2D copulas must have values between this minimum $W$ and the maximum copula $M(x,y) = \min(x,y)$.  $W$ expresses maximum anticorrelation between the variables while $M$ expresses maximum correlation between them.  Follow the link (a Wikipedia article) for Mathematica plots of these copulas.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-10-14T02:31:24.530" LastActivityDate="2010-10-14T02:31:24.530" CommentCount="6" />
  <row Id="2104" PostTypeId="1" AcceptedAnswerId="2106" CreationDate="2010-08-25T14:17:44.987" Score="4" ViewCount="645" Body="&lt;p&gt;Thank you for reading. I am trying to get sphericity values for a purely within subject design. I have been unable to use ezANOVA, or Anova(). Anova works if I add a between subject factor, but I have been unable to get sphericity for a purely within subject design. Any advice?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I already read the R newsletter, fox chapter appendix, EZanova, and whatever I could find online.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My original ANOVA&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;anova(aov(resp ~ sucrose*citral, random =~1 | subject, data = p12bl, subset = exps==1)) &#xA;anova(aov(resp ~ sucrose*citral, random =~1 | subject/sucrose*citral, data = p12bl, subset = exps==1))&#xA;&#xA;&amp;gt; str(subset(p12bl, exps==1))&#xA;'data.frame':   192 obs. of  12 variables:&#xA; $ exps     : int  1 1 1 1 1 1 1 1 1 1 ...&#xA; $ Order    : int  1 1 1 1 1 1 1 1 1 1 ...&#xA; $ threshold: Factor w/ 2 levels &quot; Suprathreshold&quot;,..: 1 1 1 1 1 1 1 1 1 1 ...&#xA; $ SET      : Factor w/ 2 levels &quot; A&quot;,&quot; B&quot;: 1 1 1 1 1 1 1 1 1 1 ...&#xA; $ subject  : Factor w/ 16 levels &quot;1&quot;,&quot;2&quot;,&quot;3&quot;,&quot;4&quot;,..: 1 2 3 4 5 6 7 8 9 10 ...&#xA; $ stim     : chr  &quot;S1C1&quot; &quot;S1C1&quot; &quot;S1C1&quot; &quot;S1C1&quot; ...&#xA; $ resp     : num  6.01 5.63 0 2.57 6.81 ...&#xA; $ id       : int  1 2 3 4 5 6 7 8 9 10 ...&#xA; $ X1       : Factor w/ 1 level &quot;S&quot;: 1 1 1 1 1 1 1 1 1 1 ...&#xA; $ sucrose  : Factor w/ 4 levels &quot;1&quot;,&quot;2&quot;,&quot;3&quot;,&quot;4&quot;: 1 1 1 1 1 1 1 1 1 1 ...&#xA; $ X3       : Factor w/ 1 level &quot;C&quot;: 1 1 1 1 1 1 1 1 1 1 ...&#xA; $ citral   : Factor w/ 4 levels &quot;1&quot;,&quot;2&quot;,&quot;3&quot;,&quot;4&quot;: 1 1 1 1 1 1 1 1 1 1 ...&#xA;&#xA;subset(p12b,exps==1)&#xA;   exps Order       threshold SET observ S1C1 S1C2 S1C3 S1C4 S2C1 S2C2 S2C3 S2C4 S3C1 S3C2 S3C3 S3C4 S4C1 S4C2 S4C3 S4C4&#xA;1     1     1  Suprathreshold   A      1  6.0  7.1  7.5  8.6 15.0 15.4 15.0 13.1 16.9   13 13.1 16.5   24   16   21   20&#xA;2     1     1  Suprathreshold   A      2  5.6  0.8  4.0  5.6  5.6 11.3 12.9 14.5 18.5   15 12.9 14.5   24   26   29   28&#xA;3     1     1  Suprathreshold   A      3  0.0  0.0  1.7  0.0  5.0  8.4  8.4  5.0 11.7   20 18.5 16.8   29   37   37   30&#xA;4     1     1  Suprathreshold   A      4  2.6  3.3  9.1 16.3  5.4 10.0  9.6 16.8 13.5   12 22.2 23.1   19   20   22   23&#xA;5     1     1  Suprathreshold   A      5  6.8  5.3 15.4 14.5 11.5  8.3 14.5 14.2  8.9   17 11.2 15.1   24   23   19   19&#xA;6     1     1  Suprathreshold   A      6  2.6  2.8  2.6  5.2 13.4 15.6 13.7 13.0 13.7   15 16.0 18.9   22   24   25   25&#xA;7     1     1  Suprathreshold   A      7  1.3  5.8 10.2  9.8 11.9 12.3 17.7 16.7 11.4   19 19.2 21.1   16   19   18   19&#xA;8     1     1  Suprathreshold   A      8  2.0  5.6  3.9  2.0  4.9  5.2  7.5  4.9 20.2   21  8.2  9.5   30   26   32   45&#xA;9     1     1  Suprathreshold   A      9  9.4 11.3 11.7 12.1 14.7 13.8 12.6 14.9 15.2   15 15.9 13.9   17   18   15   18&#xA;10    1     1  Suprathreshold   A     10  4.5 17.8 18.5 21.6  5.8 10.9 17.0 20.2  6.6   10 17.8 18.7   12   12   16   19&#xA;11    1     1  Suprathreshold   A     11  9.8 13.0 16.1 18.0 10.5 11.6 15.4 17.3 10.1   14 15.2 16.7   13   15   15   17&#xA;12    1     1  Suprathreshold   A     12  9.6 10.4 13.3 11.3 12.1 12.6 13.6 13.6 14.9   16 15.1 16.3   16   18   18   17&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Sample output&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ezANOVA( data = subset(p12bl, exps==1)  , dv= .(resp), sid = .(observ), within = .(sucrose,citral), between = NULL, collapse_within = FALSE)&#xA;Note: model has only an intercept; equivalent type-III tests substituted.&#xA;$ANOVA&#xA;          Effect DFn DFd  SSn  SSd     F       p p&amp;lt;.05   pes&#xA;1        sucrose   3  33 4953 3263 16.70 9.0e-07     * 0.603&#xA;2         citral   3  33  410  553  8.16 3.3e-04     * 0.426&#xA;3 sucrose:citral   9  99   56  791  0.77 6.4e-01       0.066&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Warning messages:&#xA;1: You have removed one or more Ss from the analysis. Refactoring &quot;observ&quot; for ANOVA. &#xA;2: Too few Ss for Anova(), reverting to aov(). See &quot;Warning&quot; section of the help on ezANOVA. &lt;/p&gt;&#xA;" OwnerUserId="1084" LastEditorUserId="1084" LastEditorDisplayName="" LastEditDate="2010-08-27T18:36:36.293" LastActivityDate="2010-09-07T15:44:17.460" Title="How to get Sphericity in R for a nested within subject design?" Tags="&lt;r&gt;&lt;anova&gt;&lt;sphericity&gt;" AnswerCount="4" CommentCount="2" FavoriteCount="3" />
  <row Id="2105" PostTypeId="2" ParentId="2093" CreationDate="2010-08-25T14:43:24.377" Score="1" ViewCount="" Body="&lt;p&gt;You also might want to consider that perhaps you &lt;em&gt;don't&lt;/em&gt; want a normally distributed variable on [0,1) but rather a random variable with a bell-shaped distribution naturally restricted to the [0,1) interval. If so, consider the &lt;a href=&quot;http://en.wikipedia.org/wiki/Beta_distribution&quot; rel=&quot;nofollow&quot;&gt;Beta distribution&lt;/a&gt; or the &lt;a href=&quot;http://www.johndcook.com/blog/2009/11/24/kumaraswamy-distribution/&quot; rel=&quot;nofollow&quot;&gt;Kumaraswamy distribution&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="279" LastActivityDate="2010-08-25T14:43:24.377" CommentCount="1" />
  <row Id="2106" PostTypeId="2" ParentId="2104" CreationDate="2010-08-25T15:22:37.273" Score="5" ViewCount="" Body="&lt;p&gt;Try:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(ez)&#xA;ezANOVA(data=subset(p12bl, exps==1),&#xA;  within=.(sucrose, citral),&#xA;  wid=.(subject),&#xA;  dv=.(resp)&#xA;  )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and the equivalent aov command, minus sphericity etc, is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;aov(resp ~ sucrose*citral + Error(subject/(sucrose*citral)), &#xA;    data= subset(p12bl, exps==1))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here's the equivalent using Anova from car directly:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(car)&#xA;df1&amp;lt;-read.table(&quot;clipboard&quot;, header=T) #From copying data in the question above&#xA;sucrose&amp;lt;-factor(rep(c(1:4), each=4))&#xA;citral&amp;lt;-factor(rep(c(1:4), 4))&#xA;idata&amp;lt;-data.frame(sucrose,citral)&#xA;&#xA;mod&amp;lt;-lm(cbind(S1C1, S1C2, S1C3, S1C4, S2C1, S2C2, S2C3, S2C4, &#xA;        S3C1, S3C2, S3C3, S3C4, S4C1, S4C2, S4C3, S4C4)~1, data=df1)&#xA;av.mod&amp;lt;-Anova(mod, idata=idata, idesign=~sucrose*citral)&#xA;summary(av.mod)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="966" LastEditorUserId="966" LastEditorDisplayName="" LastEditDate="2010-08-31T15:25:02.000" LastActivityDate="2010-08-31T15:25:02.000" CommentCount="11" />
  <row Id="2107" PostTypeId="2" ParentId="2099" CreationDate="2010-08-25T16:31:08.917" Score="1" ViewCount="" Body="&lt;p&gt;While I haven't used either of these, I have these two projects bookmarked:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.randelshofer.ch/treeviz/index.html&quot; rel=&quot;nofollow&quot;&gt;Treeviz&lt;/a&gt; &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://jtreemap.sourceforge.net/&quot; rel=&quot;nofollow&quot;&gt;JTreeMap&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Both are implemented in Java, are free and open source (MIT and Apache 2.0 licenses, respectively). Both appear to be actually libraries, but seem to come with an example application.&lt;/p&gt;&#xA;" OwnerUserId="705" LastActivityDate="2010-08-25T16:31:08.917" />
  <row Id="2108" PostTypeId="1" CreationDate="2010-08-25T16:44:21.330" Score="4" ViewCount="177" Body="&lt;p&gt;Hi,&#xA;  I have a web service, which returns a dataset, and I would like to query it from the [R] statistical package. I'd like to know how (or if) this can be done and also how to load the first datatable, of this dataset, into memory.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: It is an XML web service&lt;/p&gt;&#xA;" OwnerUserId="428" LastEditorUserId="428" LastEditorDisplayName="" LastEditDate="2010-08-26T09:52:27.760" LastActivityDate="2010-08-26T12:36:48.317" Title="Using [R] to connect to a Web Service" Tags="&lt;r&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="1" />
  <row Id="2110" PostTypeId="2" ParentId="2108" CreationDate="2010-08-25T16:49:37.560" Score="3" ViewCount="" Body="&lt;p&gt;If your webservice amounts to calling a URL and getting back XML or json, then you can just use the &lt;code&gt;XML&lt;/code&gt; or &lt;code&gt;rjson&lt;/code&gt; packages directly for this.  Possibly with &lt;code&gt;RCurl&lt;/code&gt; for more elaborate session handling.  See &lt;a href=&quot;http://www.omegahat.org/RCurl/RCurlJSS.pdf&quot; rel=&quot;nofollow&quot;&gt;the &lt;code&gt;RCurl&lt;/code&gt; paper for an example&lt;/a&gt;.  You can also look at &lt;a href=&quot;http://code.google.com/p/overflowr/&quot; rel=&quot;nofollow&quot;&gt;our &lt;code&gt;overflowr&lt;/code&gt; package&lt;/a&gt; for an example with &lt;code&gt;rjson&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Edit&lt;/em&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using &lt;code&gt;RCurl&lt;/code&gt; is not necessary unless you have a complicated request.  Otherwise, R can natively handle pulling in data over http.  Just pass the url into the function (e.g. read.table).  A simple example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;URL &amp;lt;- &quot;http://ichart.finance.yahoo.com/table.csv?s=SPY&quot;&#xA;dat &amp;lt;- read.csv(URL)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In your case, since your data is XML, you can use the &lt;code&gt;readLines&lt;/code&gt; function in the same manner.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Have a look at the &lt;code&gt;getNYTCongress&lt;/code&gt; function in &lt;a href=&quot;http://code.google.com/p/nytr/source/browse/trunk/R/nyt.congress.R&quot; rel=&quot;nofollow&quot;&gt;the &lt;code&gt;nytR&lt;/code&gt; package&lt;/a&gt; for an example of reading an XML webservice without &lt;code&gt;RCurl&lt;/code&gt;.  Also look at &lt;a href=&quot;http://stackoverflow.com/questions/1395528/scraping-html-tables-into-r-data-frames-using-the-xml-package&quot;&gt;this question on StackOverflow&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="5" LastEditorUserId="5" LastEditorDisplayName="" LastEditDate="2010-08-26T12:36:48.317" LastActivityDate="2010-08-26T12:36:48.317" CommentCount="2" />
  <row Id="2111" PostTypeId="1" CreationDate="2010-08-25T18:15:55.980" Score="6" ViewCount="454" Body="&lt;p&gt;For the selection of predictors in multivariate linear regression with $p$ suitable predictors, what methods are available to find an 'optimal' subset of the predictors without explicitly testing all $2^p$ subsets? In 'Applied Survival Analysis,' Hosmer &amp;amp; Lemeshow make reference to Kuk's method, but I cannot find the original paper. Can anyone describe this method, or, even better, a more modern technique? One may assume normally distributed errors.&lt;/p&gt;&#xA;" OwnerUserId="795" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-08-25T22:57:35.883" LastActivityDate="2010-09-16T23:32:16.287" Title="Computing best subset of predictors for linear regression" Tags="&lt;modeling&gt;&lt;regression&gt;&lt;multivariable&gt;&lt;model-selection&gt;&lt;feature-selection&gt;" AnswerCount="2" CommentCount="4" FavoriteCount="3" />
  <row Id="2112" PostTypeId="2" ParentId="2104" CreationDate="2010-08-25T18:30:01.223" Score="6" ViewCount="" Body="&lt;p&gt;Did you try the &lt;code&gt;car&lt;/code&gt; package, from John Fox? It includes the function &lt;code&gt;Anova()&lt;/code&gt; which is very useful when working with experimental designs. It should give you corrected p-value following Greenhouse-Geisser correction and Huynh-Feldt correction. I can post a quick R example if you wonder how to use it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, there is a nice tutorial on the use of R with repeated measurements and mixed-effects model for &lt;a href=&quot;http://www.psych.upenn.edu/~baron/rpsych/rpsych.html&quot; rel=&quot;nofollow&quot;&gt;psychology experiments and questionnaires&lt;/a&gt;; see Section 6.10 about sphericity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a sidenote, the Mauchly's Test of Sphericity is available in &lt;code&gt;mauchly.test()&lt;/code&gt;, but it doesn't work with &lt;code&gt;aov&lt;/code&gt; object if I remembered correctly. The &lt;a href=&quot;http://cran.r-project.org/doc/Rnews/Rnews_2007-2.pdf&quot; rel=&quot;nofollow&quot;&gt;R Newsletter&lt;/a&gt; from October 2007 includes a brief description of this topic.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-08-25T18:30:01.223" CommentCount="9" />
  <row Id="2113" PostTypeId="2" ParentId="2032" CreationDate="2010-08-25T18:32:36.633" Score="1" ViewCount="" Body="&lt;p&gt;Go to www.vaultanalytics.com/books&lt;/p&gt;&#xA;&#xA;&lt;p&gt;They have written a book on what predictive models are, when to use what tests/models, and how to create them in Excel.  I'm using it every day in my job.  I think it's extremely useful.&lt;/p&gt;&#xA;" LastActivityDate="2010-08-25T18:32:36.633" CommunityOwnedDate="2010-08-25T18:32:36.633" />
  <row Id="2114" PostTypeId="2" ParentId="2111" CreationDate="2010-08-25T18:37:15.187" Score="6" ViewCount="" Body="&lt;p&gt;I've never heard of Kuk's method, but the hot topic these days is L1 minimisation. The rationale being that if you use a penalty term of the absolute value of the regression coefficients, the unimportant ones should go to zero.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These techniques have some funny names: Lasso, LARS, Dantzig selector. You can read the papers, but a good place to start is with &lt;a href=&quot;http://www-stat.stanford.edu/~tibs/ElemStatLearn/&quot; rel=&quot;nofollow&quot;&gt;Elements of Statistical Learning&lt;/a&gt;, Chapter 3.&lt;/p&gt;&#xA;" OwnerUserId="495" LastActivityDate="2010-08-25T18:37:15.187" CommentCount="4" />
  <row Id="2115" PostTypeId="2" ParentId="2085" CreationDate="2010-08-25T19:45:27.547" Score="4" ViewCount="" Body="&lt;p&gt;It seems you are using the &lt;code&gt;nlme&lt;/code&gt; package. Maybe it would be worth trying R and the &lt;code&gt;lme4&lt;/code&gt; instead, although it is not fully comparable wrt. syntax or function call. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case, I would suggest to specify the &lt;code&gt;level&lt;/code&gt; when you called &lt;code&gt;ranef()&lt;/code&gt;, see &lt;code&gt;?ranef.lme&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;   level: an optional vector of positive integers giving the levels of&#xA;          grouping to be used in extracting the random effects from an&#xA;          object with multiple nested grouping levels. Defaults to all&#xA;          levels of grouping.&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is also present in the official &lt;a href=&quot;http://ftp.chg.ru/pub/math/NLME/UGuide.pdf&quot; rel=&quot;nofollow&quot;&gt;documentation&lt;/a&gt; for NLME 3.0 (e.g., p. 17).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Check out Douglas Bates's neat handouts on GLMM. He is also writing a textbook entitled &lt;em&gt;lme4: Mixed-effects modeling with R&lt;/em&gt;. All are available on &lt;a href=&quot;http://lme4.R-forge.R-project.org/&quot; rel=&quot;nofollow&quot;&gt;R-forge&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-08-25T19:45:27.547" CommentCount="1" />
  <row Id="2116" PostTypeId="1" CreationDate="2010-08-25T20:09:26.997" Score="5" ViewCount="445" Body="&lt;p&gt;As someone who needs statistical knowledge but is not a formally trained statistician, I'd find it helpful to have a flowchart (or some kind of decision tree) to help me choose the correct approach to solve a particular problem (eg. &quot;do you need this and know that and that and consider data to be normally distributed? Use technique X. If data is not normal, use Y or Z&quot;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After some &lt;a href=&quot;http://www.google.ca/#hl=en&amp;amp;q=flowchart+to+select+statistical+test&quot; rel=&quot;nofollow&quot;&gt;googling&lt;/a&gt;, I've seen several attempts of various coverage and quality (some not available at the moment). I've also seen similar flowcharts in statistics textbooks I've consulted in libraries.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A bonus would be an interactive site that, besides just having a chart, would provide extra info (such as assumptions) and would point to how to perform those techniques in popular stat packages. &quot;Need to do ANOVA in R? You need package X and here's a tutorial&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm asking as a community wiki question in the hope there are better resources I couldn't find. Since statistics is a large subject, I think such a flowchart would be suitable for techniques that can be approached by someone who has beginner or intermediate-level knowledge. Anything more complicated would need someone with formal training.&lt;/p&gt;&#xA;" OwnerUserId="840" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-16T06:44:49.510" LastActivityDate="2010-09-16T06:44:49.510" CommunityOwnedDate="2010-08-25T20:09:26.997" Title="Flowcharts to help selecting the proper analysis technique and test?" Tags="&lt;best-practices&gt;" AnswerCount="2" FavoriteCount="2" />
  <row Id="2117" PostTypeId="2" ParentId="2116" CreationDate="2010-08-25T20:17:24.127" Score="5" ViewCount="" Body="&lt;p&gt;These are not really interactive flowcharts, but maybe this could be useful: (1) &lt;a href=&quot;http://j.mp/cmakYq&quot; rel=&quot;nofollow&quot;&gt;http://j.mp/cmakYq&lt;/a&gt;, (2) &lt;a href=&quot;http://j.mp/aaxUsz&quot; rel=&quot;nofollow&quot;&gt;http://j.mp/aaxUsz&lt;/a&gt;, and (3) &lt;a href=&quot;http://j.mp/bDMyAR&quot; rel=&quot;nofollow&quot;&gt;http://j.mp/bDMyAR&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-08-25T20:17:24.127" CommunityOwnedDate="2010-08-25T20:17:24.127" CommentCount="1" />
  <row Id="2118" PostTypeId="2" ParentId="2116" CreationDate="2010-08-25T21:24:22.967" Score="3" ViewCount="" Body="&lt;p&gt;You can look at the solution given on the question &lt;a href=&quot;http://stats.stackexchange.com/questions/1252/statistical-models-cheat-sheet&quot;&gt;&quot;Statistical models cheat sheet&quot;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="253" LastActivityDate="2010-08-25T21:24:22.967" CommunityOwnedDate="2010-08-25T21:24:22.967" />
  <row Id="2119" PostTypeId="1" AcceptedAnswerId="2123" CreationDate="2010-08-25T21:56:01.473" Score="1" ViewCount="292" Body="&lt;p&gt;Some guys told me that it's appropriate to use Wald-Wolfowitz Runs Test as a normality test (like Shapiro-Wilk's or Kolmogorov-Smirnov...). Do you think this is good way to test normality assumptions?&lt;/p&gt;&#xA;" OwnerUserId="1356" LastActivityDate="2010-08-25T22:54:03.987" Title="Wald-Wolfowitz Runs Test for Normality Assumptions Testing" Tags="&lt;normality&gt;&lt;test&gt;&lt;nonparametric&gt;" AnswerCount="1" />
  <row Id="2120" PostTypeId="2" ParentId="1912" CreationDate="2010-08-25T21:58:52.960" Score="1" ViewCount="" Body="&lt;p&gt;Let me take a stab at this...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I understand the original sentiment and subsequent response, I think what Gary King is getting at is that noticing a scale invariance effect on your data is a very gross understanding of the phenomena of whats going on.  While this 'birds eye view' of whatever phenomena you are observing could be insightful, one might gloss over useful information at the microscopic level.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This example might not be the best, but consider Conway's 'Game of Life'.  This is completely determined, as in it is a deterministic system.  Consider looking at some statistic of this system, cluster longevity, say, for some appropriate definition of cluster.  For arguments sake lets say this follows a power law (I don't know if it does or not, but just for this example, lets say it does).  This gives a gross high level description of the system but you've washed all the details of how gliders race across the board, how they collide to give glider guns and other useful information that you might be able to use to determine some specifics about your system.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not sure this is the best example or even if I've gotten the gist of what Gary King was trying to say, but thats my 2 cents.&lt;/p&gt;&#xA;" LastActivityDate="2010-08-25T21:58:52.960" />
  <row Id="2121" PostTypeId="1" CreationDate="2010-08-25T22:34:45.943" Score="3" ViewCount="349" Body="&lt;p&gt;I am using ridge regression on highly multicollinear data. Using OLS I get large standard errors on the coefficients due to the multicollinearity. I know ridge regression is a way to deal with this problem, but in all the implementations of ridge regression that I've looked at, there are no standard errors reported for the coefficients. I would like some way of estimating how much the ridge regression is helping by seeing how much it is decreasing the standard errors of specific coefficients. Is there some way to estimate them in ridge regression?&lt;/p&gt;&#xA;" OwnerUserId="1090" LastActivityDate="2010-11-01T16:24:39.703" Title="How can I estimate coefficient standard errors when using ridge regression?" Tags="&lt;standard-error&gt;&lt;ridge-regression&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="2123" PostTypeId="2" ParentId="2119" CreationDate="2010-08-25T22:54:03.987" Score="3" ViewCount="" Body="&lt;p&gt;This is not a great idea. The Wald-Wolfowitz test can detect certain departures from being independently distributed (it detects whether a series of observations is 'too bunchy' or 'too  jittery' with respect to the distribution of positive and negative values), which have nothing to do with normality. It is not sensitive to the shape of the empirical distribution: it will 'pass' Cauchy, shifted uniform, $t$, polluted normal, etc. random variates.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Moreover, the WW test is sensitive to the ordering of the observations. A true test for normality should be invariant to permutations.&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-08-25T22:54:03.987" CommentCount="1" />
  <row Id="2124" PostTypeId="2" ParentId="2007" CreationDate="2010-08-25T22:55:22.173" Score="2" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://rapid-i.com/content/view/181/190/lang,en/&quot; rel=&quot;nofollow&quot;&gt;RapidMiner&lt;/a&gt; (Java) [open source]&lt;/p&gt;&#xA;" OwnerUserId="170" LastActivityDate="2010-08-25T22:55:22.173" CommunityOwnedDate="2010-08-25T22:55:22.173" />
  <row Id="2125" PostTypeId="1" AcceptedAnswerId="2128" CreationDate="2010-08-25T23:53:00.417" Score="5" ViewCount="2879" Body="&lt;p&gt;Is it cheating if I know the answer, but ask it anyways?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I figure it's good for people to know :)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In particular, I am referring to the Pearson product-moment correlation coefficient&lt;/p&gt;&#xA;" OwnerUserId="74" LastEditorUserId="74" LastEditorDisplayName="" LastEditDate="2010-08-26T21:06:25.317" LastActivityDate="2010-10-22T09:51:55.363" Title="What's the difference between correlation and simple linear regression?" Tags="&lt;correlation&gt;&lt;regression&gt;" AnswerCount="4" CommentCount="2" FavoriteCount="4" />
  <row Id="2126" PostTypeId="2" ParentId="2121" CreationDate="2010-08-26T00:05:08.597" Score="2" ViewCount="" Body="&lt;p&gt;Assuming that the data generating process follows the standard assumptions behind OLS the standard errors for ridge regression is given by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ \sigma^2 (A^T A + \Gamma^T \Gamma)^{-1} A^T A (A^T A + \Gamma^T \Gamma)^{-1}$  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The notation above follows the wiki notation for &lt;a href=&quot;http://en.wikipedia.org/wiki/Tikhonov_regularization&quot; rel=&quot;nofollow&quot;&gt;ridge regression&lt;/a&gt;. Specifically,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$A$ is the covraiate matrix,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\sigma^2$ is the error variance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\Gamma$ is the Tikhonov matrix chosen suitably in ridge regression.&lt;/p&gt;&#xA;" LastActivityDate="2010-08-26T00:30:18.583" CommentCount="1" />
  <row Id="2127" PostTypeId="2" ParentId="2111" CreationDate="2010-08-26T00:56:53.163" Score="1" ViewCount="" Body="&lt;p&gt;This is a huge topic. As previously mentioned, Hastie, Tibshirani and Friedman give a good intro in Ch3 of Elements of Statistical Learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A few points.&#xA;1) What do you mean by &quot;best&quot; or &quot;optimal&quot;? What is best in one sense may not be best in another. Two common criteria are predictive accuracy (predicting the outcome variable) and producing unbiased estimators of the coefficients. Some methods, such as Lasso &amp;amp; Ridge Regression inevitably produce biased coefficient estimators.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) The phrase &quot;best subsets&quot; itself can be used in two separate senses. Generally to refer to the best subset among all predictors which optimises some model building criteria. More specifically it can refer to Furnival and Wilson's efficient algorithm for finding that subset among moderate (~50) numbers of linear predictors (Regressions by Leaps and Bounds. Technometrics, Vol. 16, No. 4 (Nov., 1974), pp. 499-51)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.jstor.org/stable/1267601&quot; rel=&quot;nofollow&quot;&gt;http://www.jstor.org/stable/1267601&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="521" LastActivityDate="2010-08-26T00:56:53.163" CommentCount="1" />
  <row Id="2128" PostTypeId="2" ParentId="2125" CreationDate="2010-08-26T02:48:15.233" Score="11" ViewCount="" Body="&lt;p&gt;What's the difference between the correlation between $X$ and $Y$ and a linear regression predicting $Y$ from $X$?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, some &lt;strong&gt;similarities&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;the standardised regression coefficient is the same as Pearson's correlation coefficient&lt;/li&gt;&#xA;&lt;li&gt;The square of Pearson's correlation coefficient is the same as the $R^2$ in  simple linear regression&lt;/li&gt;&#xA;&lt;li&gt;Neither simple linear regression nor correlation answer questions of causality directly. This point is important, because I've met people that think that simple regression can magically allow an inference that X causes Y.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Second, some &lt;strong&gt;differences&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The regression equation (i.e., $a + bX$) can be used to make predictions on $Y$ based on values of $X$&lt;/li&gt;&#xA;&lt;li&gt;While correlation typically refers to the linear relationship, it can refer to other forms of dependence, such as polynomial or truly nonlinear relationships&lt;/li&gt;&#xA;&lt;li&gt;While correlation typically refers to Pearson's correlation coefficient, there are other types of correlation, such as Spearman's.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-08-27T02:27:34.790" LastActivityDate="2010-08-27T02:27:34.790" CommentCount="0" />
  <row Id="2129" PostTypeId="2" ParentId="2125" CreationDate="2010-08-26T03:21:45.833" Score="4" ViewCount="" Body="&lt;p&gt;Here is an answer I &lt;a href=&quot;http://www.graphpad.com/faq/viewfaq.cfm?faq=1141&quot; rel=&quot;nofollow&quot;&gt;posted on the graphpad.com website&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Correlation and linear regression are not the same. Consider these differences:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Correlation quantifies the degree to which two variables are related. Correlation does not fit a line through the data. &lt;/li&gt;&#xA;&lt;li&gt;With correlation you don't have to think about cause and effect. You simply quantify how well two variables relate to each other. With regression, you do have to think about cause and effect as the regression line is determined as the best way to predict Y from X.&lt;/li&gt;&#xA;&lt;li&gt;With correlation, it doesn't matter which of the two variables you call &quot;X&quot; and which you call &quot;Y&quot;. You'll get the same correlation coefficient if you swap the two. With linear regression, the decision of which variable you call &quot;X&quot; and which you call &quot;Y&quot; matters a lot, as you'll get a different best-fit line if you swap the two. The line that best predicts Y from X is not the same as the line that predicts X from Y (unless you have perfect data with no scatter.)&lt;/li&gt;&#xA;&lt;li&gt;Correlation is almost always used when you measure both variables. It rarely is appropriate when one variable is something you experimentally manipulate. With linear regression, the X variable is usually something you experimentally manipulate (time, concentration...) and the Y variable is something you measure.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="25" LastActivityDate="2010-08-26T03:21:45.833" CommentCount="0" />
  <row Id="2130" PostTypeId="1" AcceptedAnswerId="2139" CreationDate="2010-08-26T04:33:09.090" Score="1" ViewCount="180" Body="&lt;p&gt;Hi, the following exponential expansion form:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ exp\left[\sum_{k=1}^\infty \gamma_k x^k\right] = \sum_{j=0}^\infty\delta_j x^j $$&#xA;where $\gamma_k$'s are known, and $\delta_0=1$, &#xA;$$\delta_{j+1} = \frac{1}{j+1}\sum_{i=1}^{j+1}i\gamma_i\delta_{j+1-i}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone know how to prove it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="1043" LastActivityDate="2010-08-28T01:39:46.570" Title="How to prove the exponential expansion recursive form?" Tags="&lt;mathematical-statistics&gt;" AnswerCount="2" />
  <row Id="2131" PostTypeId="1" CreationDate="2010-08-26T04:45:31.653" Score="4" ViewCount="261" Body="&lt;p&gt;for eg. if I have a class variable credit scoring with two classes good and bad, where #(good) = 700 and #(bad)= 300. I do not want to shorten my data. which technique I should use? I was using SVM but it is giving all bad as good in predictions.&lt;/p&gt;&#xA;" OwnerUserId="861" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-08-26T07:02:24.003" LastActivityDate="2010-09-03T09:12:37.553" Title="For a classification problem if class variable has unequal distribution which technique we should use?" Tags="&lt;classification&gt;&lt;binary-data&gt;" AnswerCount="3" CommentCount="2" FavoriteCount="3" />
  <row Id="2132" PostTypeId="2" ParentId="2125" CreationDate="2010-08-26T06:37:23.230" Score="4" ViewCount="" Body="&lt;p&gt;In the single predictor case of linear regression, the standardized slope has the same value as the correlation coefficient.  The advantage of the linear regression is that the relationship can be described in such a way that you can predict (based on the relationship between the two variables) the score on the predicted variable given any particular value of the predictor variable.  In particular one piece of information a linear regression gives you that a correlation does not is the intercept, the value on the predicted variable when the predictor is 0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In short - they produce identical results computationally, but there are more elements which are capable of interpretation in the simple linear regression.  If you are interested in simply characterizing the magnitude of the relationship between two variables, use correlation - if you are interested in predicting or explaining your results in terms of particular values you probably want regression.&lt;/p&gt;&#xA;" OwnerUserId="196" LastActivityDate="2010-08-26T06:37:23.230" />
  <row Id="2133" PostTypeId="2" ParentId="2131" CreationDate="2010-08-26T07:04:58.167" Score="3" ViewCount="" Body="&lt;p&gt;Your class sample sizes do not seem so unbalanced since you have 30% of observations in your minority class. Logistic regression should be well performing in your case. Depending on the number of predictors that enter your model, you may consider some kind of penalization for parameters estimation, like ridge (L2) or lasso (L1). For an overview of problems with very unbalanced class, see Cramer (1999), The Statistician, 48: 85-94 (&lt;a href=&quot;http://www.tinbergen.nl/discussionpapers/98085.pdf&quot; rel=&quot;nofollow&quot;&gt;PDF&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not familiar with credit scoring techniques, but I found some papers that suggest that you could use SVM with weighted classes, e.g. &lt;a href=&quot;http://www.springerlink.com/content/uwx80880224n25w3/&quot; rel=&quot;nofollow&quot;&gt;Support Vector Machines for Credit Scoring: Extension to Non Standard Cases&lt;/a&gt;. As an alternative, you can look at &lt;a href=&quot;http://mpra.ub.uni-muenchen.de/8156/1/MPRA_paper_8156.pdf&quot; rel=&quot;nofollow&quot;&gt;boosting&lt;/a&gt; methods with CART, or Random Forests (in the latter case, it is possible to adapt the sampling strategy so that each class is represented when constructing the classification trees). The paper by Novak and LaDue discuss the pros and cons of &lt;a href=&quot;http://ageconsearch.umn.edu/bitstream/15129/1/31010109.pdf&quot; rel=&quot;nofollow&quot;&gt;GLM vs Recursive partitioning&lt;/a&gt;. I also found this article, &lt;a href=&quot;http://fic.wharton.upenn.edu/fic/handpaper.pdf&quot; rel=&quot;nofollow&quot;&gt;Scorecard construction with unbalanced class sizes&lt;/a&gt; by Hand and Vinciotti.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-08-26T07:04:58.167" />
  <row Id="2134" PostTypeId="1" CreationDate="2010-08-26T08:03:41.620" Score="3" ViewCount="397" Body="&lt;p&gt;I'm looking for an R package for estimating the coefficients of logit models with individual fixed-effect (individual intercept) using Chamberlain's 1980 estimator. It is often known as Chamberlain's fixed-effect logit estimator.&lt;br&gt;&#xA;It's a classic esitimator when dealing with binary outcome panel data (at least in econometrics), but I just don't find anything related to it in the CRAN...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any clue ?&lt;/p&gt;&#xA;" OwnerUserId="1091" LastActivityDate="2010-08-26T08:59:08.873" Title="R package for fixed-effect logistic regression" Tags="&lt;r&gt;&lt;logistic-regression&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="2135" PostTypeId="2" ParentId="2131" CreationDate="2010-08-26T08:45:35.657" Score="5" ViewCount="" Body="&lt;p&gt;A popular approach towards solving class imbalance problems is to bias the classifier so that it pays more attention to the positive instances. This can be done, for instance, by increasing the penalty associated with misclassifying the positive class relative to the negative class. Another approach is to preprocess the data by oversampling the majority class or undersampling the minority class in order to create a balanced dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, in your case, class imbalancing don't seem to be a problem. Perhaps it is a matter of parameter tuning, since finding the optimal parameters for an SVM classifier can be a rather tedious process. There are two parameters for e.g. in an RBF kernel: $C$ and $\gamma$. It is not known beforehand which $C$ and $\gamma$ are best for a given problem; consequently some kind of model selection (parameter search) must be done.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the data preprocessing phase, remember that SVM requires that each data instance is represented as a vector of real numbers. Hence, if there are categorical attributes, it's recommended to convert them into numeric data, using m numbers to represent an m-category attribute (or replacing it with m new binary variables).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, scaling the variables before applying SVM is crucial, in order to avoid attributes in greater numeric ranges dominating those in smaller numeric ranges.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Check out &lt;a href=&quot;http://www.csie.ntu.edu.tw/~cjlin/papers/guide/guide.pdf&quot; rel=&quot;nofollow&quot;&gt;this paper&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're working in R, check out the &lt;a href=&quot;http://www.stat.ucl.ac.be/ISdidactique/Rhelp/library/e1071/html/tune.html&quot; rel=&quot;nofollow&quot;&gt;tune&lt;/a&gt; function (package e1071) to tune hyperparameters using a grid search over supplied parameter ranges. Then, using  &lt;a href=&quot;http://bm2.genes.nig.ac.jp/RGM2/R_current/library/e1071/man/plot.tune.html&quot; rel=&quot;nofollow&quot;&gt;plot.tune&lt;/a&gt;, you can see visually which set of values gives the smaller error rate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a shortcut around the time-consuming parameter search. There is an R package called &quot;svmpath&quot; which computes the entire regularization path for a 2-class SVM classifier in one go. Here is &lt;a href=&quot;http://jmlr.csail.mit.edu/papers/volume5/hastie04a/hastie04a.pdf&quot; rel=&quot;nofollow&quot;&gt;a link to the paper&lt;/a&gt; that describes what it's doing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S. You may also find this paper interesting: &lt;a href=&quot;http://www.google.gr/url?sa=t&amp;amp;source=web&amp;amp;cd=3&amp;amp;ved=0CCwQFjAC&amp;amp;url=http%3A%2F%2Fciteseerx.ist.psu.edu%2Fviewdoc%2Fdownload%3Fdoi%3D10.1.1.29.3039%26rep%3Drep1%26type%3Dpdf&amp;amp;ei=Byh2TMbJM4XN4Abf942yBg&amp;amp;usg=AFQjCNG9_KErIloNqZKPUMEFkjRpEP2Tyg&amp;amp;sig2=pMx4yZ01d2-pT-kt4Wp-gw&quot; rel=&quot;nofollow&quot;&gt;Obtaining calibrated probability estimates&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="339" LastActivityDate="2010-08-26T08:45:35.657" CommentCount="1" />
  <row Id="2136" PostTypeId="2" ParentId="2134" CreationDate="2010-08-26T08:59:08.873" Score="3" ViewCount="" Body="&lt;p&gt;Conditional logistic regression (I assume that this is what you refered to when talking about Chamberlain's estimator) is available through &lt;code&gt;clogit()&lt;/code&gt; in the &lt;a href=&quot;http://cran.r-project.org/web/packages/survival/index.html&quot; rel=&quot;nofollow&quot;&gt;survival&lt;/a&gt; package. I also found this page which contains R code to estimate &lt;a href=&quot;http://course.wilkes.edu/Merrill/stories/storyReader%2424&quot; rel=&quot;nofollow&quot;&gt;conditional logit parameters&lt;/a&gt;. The &lt;a href=&quot;http://cran.r-project.org/web/packages/survey/index.html&quot; rel=&quot;nofollow&quot;&gt;survey&lt;/a&gt; package also includes a lot of wrapper function for GLM and Survival model in the case of complex sampling, but I didn't look at.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Try also to look at &lt;code&gt;logit.mixed&lt;/code&gt; in the &lt;a href=&quot;http://cran.r-project.org/web/packages/Zelig/index.html&quot; rel=&quot;nofollow&quot;&gt;Zelig&lt;/a&gt; package, or directly use the &lt;a href=&quot;http://cran.r-project.org/web/packages/lme4/index.html&quot; rel=&quot;nofollow&quot;&gt;lme4&lt;/a&gt; package which provide methods for mixed-effects models with binomial link (see &lt;code&gt;lmer&lt;/code&gt; or &lt;code&gt;glmer&lt;/code&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Did you take a look at &lt;a href=&quot;http://cran.r-project.org/doc/contrib/Farnsworth-EconometricsInR.pdf&quot; rel=&quot;nofollow&quot;&gt;Econometrics in R&lt;/a&gt;, from Grant V. Farnsworth? It seems to provide a gentle overview of applied econometrics in R (with which I am not familiar).&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-08-26T08:59:08.873" CommentCount="3" />
  <row Id="2137" PostTypeId="2" ParentId="2121" CreationDate="2010-08-26T09:36:21.003" Score="2" ViewCount="" Body="&lt;p&gt;I think boostrap would the best option to obtain robust SEs. This was done in some applied work using shrinkage methods, e.g. &lt;a href=&quot;http://www.biomedcentral.com/1753-6561/3/S7/S61&quot; rel=&quot;nofollow&quot;&gt;Analysis of North American Rheumatoid Arthritis Consortium data using a penalized logistic regression approach&lt;/a&gt; (BMC Proceedings 2009).&#xA;There is also a nice paper from Casella on SE computation with penalized model, &lt;a href=&quot;http://j.mp/c3c4tC&quot; rel=&quot;nofollow&quot;&gt;Penalized Regression, Standard Errors, and Bayesian Lassos&lt;/a&gt; (Bayesian Analysis 2010 5(2)). But they are more concerned with &lt;em&gt;lasso&lt;/em&gt; and &lt;em&gt;elasticnet&lt;/em&gt; penalization.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I always thought of ridge regression as a way to get better predictions than standard OLS, where the model is generally not parcimonious. For variable selection, the &lt;em&gt;lasso&lt;/em&gt; or &lt;em&gt;elasticnet&lt;/em&gt; criteria are more appropriate, but then it is difficult to apply a bootstrap procedure (since selected variables would change from one sample to the other, and even in the inner $k$-fold loop used to optimize the $\ell_1$/$\ell_2$ parameters); this is not the case with ridge regression, since you always consider all variables. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have no idea about R packages that would give this information. It doesn't seem to be available in the &lt;a href=&quot;http://cran.r-project.org/web/packages/glmnet/index.html&quot; rel=&quot;nofollow&quot;&gt;glmnet&lt;/a&gt; package (see Friedman's paper in JSS, &lt;a href=&quot;http://www.jstatsoft.org/v33/i01/paper&quot; rel=&quot;nofollow&quot;&gt;Regularization Paths for Generalized Linear Models via Coordinate Descent&lt;/a&gt;).&#xA;However, Jelle Goeman who authored the &lt;a href=&quot;http://cran.r-project.org/web/packages/penalized/index.html&quot; rel=&quot;nofollow&quot;&gt;penalized&lt;/a&gt; package discuss this point too. Cannot find the original PDF on the web, so I simply quote his words:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;It is a very natural question to ask&#xA;  for standard errors of regression&#xA;  coefficients or other estimated&#xA;  quantities. In principle such standard&#xA;  errors can easily be calculated, e.g.&#xA;  using the bootstrap.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Still, this&#xA;  package deliberately does not provide&#xA;  them. The reason for this is that&#xA;  standard errors are not very&#xA;  meaningful for strongly biased&#xA;  estimates such as arise from penalized&#xA;  estimation methods. Penalized&#xA;  estimation is a procedure that reduces&#xA;  the variance of estimators by&#xA;  introducing substantial bias. The bias&#xA;  of each estimator is therefore a major&#xA;  component of its mean squared error,&#xA;  whereas its variance may contribute&#xA;  only a small part. &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Unfortunately, in&#xA;  most applications of penalized&#xA;  regression it is impossible to obtain&#xA;  a sufficiently precise estimate of the&#xA;  bias. Any bootstrap-based cal-&#xA;  culations can only give an assessment&#xA;  of the variance of the estimates.&#xA;  Reliable estimates of the bias are&#xA;  only available if reliable unbiased&#xA;  estimates are available, which is&#xA;  typically not the case in situations&#xA;  in which penalized estimates are used.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Reporting a standard error of a&#xA;  penalized estimate therefore tells&#xA;  only part of the story. It can give a&#xA;  mistaken impression of great&#xA;  precision, completely ignoring the&#xA;  inaccuracy caused by the bias. It is&#xA;  certainly a mistake to make confidence&#xA;  statements that are only based on an&#xA;  assessment of the variance of the&#xA;  estimates, such as bootstrap-based&#xA;  confidence intervals do.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-01T16:24:39.703" LastActivityDate="2010-11-01T16:24:39.703" />
  <row Id="2138" PostTypeId="2" ParentId="2067" CreationDate="2010-08-26T10:15:37.873" Score="4" ViewCount="" Body="&lt;p&gt;As a consequence of the inspiring answers and discussion to my question I constructed the following plots that do not rely on any model based parameters, but present the underlying data.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reasons are that independent of whatever kind of standard-error I may choose, the standard-error is a model based parameter. So, why not present the underlying data and thereby transmit more information?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furthermore, if choosing the SE from the ANOVA, two problems arise for my specific problems.&lt;br&gt;&#xA;First (at least for me) it is somehow unclear what the SEs from &lt;code&gt;SPSS&lt;/code&gt; ANOVA Output actually are (&lt;a href=&quot;http://www.analysisfactor.com/statchat/why-report-estimated-marginal-means-in-spss-glm/&quot; rel=&quot;nofollow&quot;&gt;see also this discussion, in the comments&lt;/a&gt;). They are somehow related to the MSE but how exactly I don't know.&lt;br&gt;&#xA;Second, they are only reasonable when the underlying assumptions are met. However, as the following plots show, the assumptions of homogeneity of variance is clearly violated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The plots with boxplots:&#xA;&lt;img src=&quot;http://i.imgur.com/cnMz3.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The plots with all data points:&#xA;&lt;img src=&quot;http://i.imgur.com/Fbt4J.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that the two groups are dislocated a little to the left or the right: deductive to the left, inductive to the right. &#xA;The means are still plotted in black and the data or boxplots in the background in grey. The differences between the plots on the left and on the right are if the means are dislocated the same as the points or boxplots or if they are presented centrally.&lt;br&gt;&#xA;Sorry for the nonoptimal quality of the graphs and the missing x-axis labels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The question that remains is, which one of the above plots is the one to choose now. I have to think about it and ask the other author of our paper. But right now, I prefer the &quot;points with means dislocated&quot;. And I still would be very interested in comments.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update:&lt;/strong&gt; After some programming I finally managed to write a R-function to automatically create a plot like points with means dislocated. &lt;a href=&quot;http://www.psychologie.uni-freiburg.de/Members/singmann/R/rm.plot&quot; rel=&quot;nofollow&quot;&gt;Check it out (and send me comments)&lt;/a&gt;!&lt;/p&gt;&#xA;" OwnerUserId="442" LastEditorUserId="442" LastEditorDisplayName="" LastEditDate="2010-12-09T12:30:33.363" LastActivityDate="2010-12-09T12:30:33.363" CommentCount="2" />
  <row Id="2139" PostTypeId="2" ParentId="2130" CreationDate="2010-08-26T10:36:17.803" Score="1" ViewCount="" Body="&lt;p&gt;A brute-force approach would use &lt;a href=&quot;http://en.wikipedia.org/wiki/Taylor_series&quot; rel=&quot;nofollow&quot;&gt;taylor series&lt;/a&gt; expansion of $exp(.)$ at 0 and group terms appropriately to demonstrate the relationship. I have not attempted to do so myself but an inspection of the first few terms does indicate that the proposed relationship holds. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Perhaps, there is a more elegant approach?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$exp\left[\gamma_1 x^1\right] = 1 + \frac{\gamma_1 x^1}{1!} + \frac{(\gamma_1 x^1)^2}{2!} + ...$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$exp\left[\gamma_2 x^2\right] = 1 + \frac{\gamma_2 x^2}{1!} + \frac{(\gamma_2 x^2)^2}{2!} + ...$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$exp\left[\gamma_3 x^3\right] = 1 + \frac{\gamma_3 x^3}{1!} + \frac{(\gamma_3 x^3)^2}{2!} + ...$$&lt;/p&gt;&#xA;" LastActivityDate="2010-08-26T10:36:17.803" CommentCount="0" />
  <row Id="2140" PostTypeId="1" AcceptedAnswerId="2141" CreationDate="2010-08-26T11:42:13.020" Score="3" ViewCount="107" Body="&lt;p&gt;I'm investgating the repeatability of an experimental process (actually measuring lift in a wind tunnel), which can be measured using a single number, Q. However, I only have limited access to the tunnel so the number of experiments that I can run is restricted.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I have 5 experiments with the tunnel in one configuration:  Q1 = [1.18, -0.41, -0.66, 0.98, 0.1]&#xA;and five in another: Q2 = [-0.36 -0.73 -1.47 0.15 -0.31]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How confident can I be that the repeatability of the tunnel in the second configuration is better than the first? i.e. the standard deviation of the data from the first configuration is 0.81 and from the second configuration is 0.60, but these values will clearly change if I do additional experiments. Can I somehow calculate the standard deviation of the standard deviation?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any advice on how I could calculate this would be appreciated and more generally how I can determine when I have enough experiments to say something like I am 95% confident that configuration 2 has better repeatability than configuration 1.&lt;/p&gt;&#xA;" OwnerUserId="1094" LastEditorUserId="279" LastEditorDisplayName="" LastEditDate="2010-08-26T13:07:19.183" LastActivityDate="2010-08-29T14:41:44.697" Title="How do I compare the repeatability of two sets of experiments with a low number of samples?" Tags="&lt;hypothesis-testing&gt;&lt;variability&gt;" AnswerCount="3" FavoriteCount="1" />
  <row Id="2141" PostTypeId="2" ParentId="2140" CreationDate="2010-08-26T13:06:34.753" Score="2" ViewCount="" Body="&lt;p&gt;Look up the two-sample F-test for variances. With equal sample sizes Q, this test calculates the ratio of the two variances, and then compares the value to the F-distribution with (Q-1, Q-1) degrees of freedom. A main assumption is normality, but that might be OK for random measurement error like this. &#xA;With your values, however the difference is not statistically significant (p=0.29). In general, the standard error of the standard deviation is large unless you have lots of samples, and thus detecting differences in variability (repeatability) is difficult.&lt;/p&gt;&#xA;" OwnerUserId="279" LastActivityDate="2010-08-26T13:06:34.753" CommentCount="2" />
  <row Id="2142" PostTypeId="1" CreationDate="2010-08-26T13:07:47.100" Score="4" ViewCount="1297" Body="&lt;p&gt;When performing linear regression, it is often useful to do a transformation such as log-transformation for the dependent variable to achieve better normal distribution conformation. Often it is also useful to inspect beta's from the regression to better assess the effect size/real relevance of the results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This raises the problem that when using e.g. log transformation, the effect sizes will be in log scale, and I've been told that because of non-linearity of the used scale, back-transforming these beta's will result in non-meaningful values that do not have any real world usage. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This far we have usually performed linear regression with transformed variables to inspect the significance and then linear regression with the original non-transformed variables to determine the effect size.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a right/better way of doing this? For the most part we work with clinical data, so a real life example would be to determine how a certain exposure affects continues variables such as height, weight or some laboratory measurement, and we would like to conclude something like &quot;exposure A had the effect of increasing weight by 2 kg&quot;.&lt;/p&gt;&#xA;" LastActivityDate="2010-11-02T17:21:52.077" Title="Linear regression effect sizes when using transformed variables" Tags="&lt;regression&gt;&lt;data-transformation&gt;&lt;effect-size&gt;" AnswerCount="3" FavoriteCount="2" />
  <row Id="2143" PostTypeId="2" ParentId="2142" CreationDate="2010-08-26T13:30:32.093" Score="2" ViewCount="" Body="&lt;p&gt;SHORT ANSWER: Absolutely correct, the back transformation of the beta value is meaningless.  However, you can report the non-linearity as something like.  &quot;If you weigh 100kg then eating two pieces of cake a day will increase your weight by approximately 2kg in one week.  However, if you weigh 200kg your weight would increase 2.5kg.  See figure 1 for a depiction of this non-linear relationship (figure 1 being a fit of the curve over the raw data).&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;LONG ANSWER:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The meaningfulness of the back transformed value varies but when properly done it usually has some meaning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have a regression of natural log values on two x predictors with a beta of 0.13, and an intercept of 7.0, then the back transformation of 0.13 (1.14) is pretty much meaningless.  That is correct.  However, the back transformation of 7.13 is going to be a value that can be interpreted with some meaning.  You could then subtract out the back transformation of 7.0 and be left with a remainder value that is your effect in a meaningful scale (152.2).  If you want to look at any predicted value you would need to first calculate it all out in log values and then back-transform.  This would have to be done separately for every predicted value and result in a curve if graphed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is often reasonable to do if your transformation has a relatively small effect on your data.  Log transformation of reaction times are one kind of value that can be back transformed.  When it's done correctly you'll find that the values seem close to median values doing simple calculations on the raw data.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even then though one must be careful with interactions and non-interactions.  The relative values vary across the scale.  The analysis was sensitive to the log value while the back transformed values may show different patterns that make interactions seem like they shouldn't be there or vice versa.  In other words, you can back transform things that make small changes to the data as long as you're careful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some changes, like logistic transform of probability, can have quite massive impacts, especially near the end of the scale.  An example of a place you should never back transform is interaction plots near the high or low end of probability.&lt;/p&gt;&#xA;" OwnerUserId="601" LastEditorUserId="601" LastEditorDisplayName="" LastEditDate="2010-08-26T13:44:01.287" LastActivityDate="2010-08-26T13:44:01.287" />
  <row Id="2144" PostTypeId="2" ParentId="2140" CreationDate="2010-08-26T13:49:10.053" Score="2" ViewCount="" Body="&lt;p&gt;If you want a non-parametric approach that doesn't assume normality, you could obtain distributions of SDs for each configuration using bootstrapping, then compare these distributions. Comments below the answer to &lt;a href=&quot;http://stats.stackexchange.com/questions/1169/ci-for-a-difference-based-on-independent-cis&quot;&gt;this question&lt;/a&gt; suggest that non-overlapping 84% confidence intervals would indicate a difference in the SDs at an alpha of .05. Alternatively, on each iteration of the bootstrap you could find the difference between the two pseudo-estimates of the SD then compare the distribution of this difference to zero. Below is R code achieving both approaches; both suggest that you have not observed a difference that should be considered real.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;a = c(1.18, -0.41, -0.66, 0.98, 0.1)&#xA;b = c(-0.36, -0.73, -1.47, 0.15, -0.31)&#xA;library(plyr)&#xA;iterations = 1e4&#xA;sds = ldply(&#xA;    .data = 1:iterations&#xA;    , .fun = function(x){&#xA;        a_sd = sd(sample(a,replace=T))&#xA;        b_sd = sd(sample(b,replace=T))&#xA;        to_return = data.frame(&#xA;            a_sd = a_sd&#xA;            , b_sd = b_sd&#xA;            , diff = a_sd - b_sd&#xA;        )&#xA;        return(to_return)&#xA;    }&#xA;    , .progress = 'text'&#xA;)&#xA;&#xA;quantile(sds$a_sd,c(.08,.92))&#xA;quantile(sds$b_sd,c(.08,.92))&#xA;#substantial overlap&#xA;&#xA;quantile(sds$diff,c(.025,.975))&#xA;#includes zero&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="364" LastEditorUserId="364" LastEditorDisplayName="" LastEditDate="2010-08-26T13:56:57.967" LastActivityDate="2010-08-26T13:56:57.967" />
  <row Id="2145" PostTypeId="2" ParentId="2130" CreationDate="2010-08-26T14:43:02.220" Score="4" ViewCount="" Body="&lt;p&gt;Srikant asks for a &quot;more elegant approach.&quot;  Perhaps the following will respond to this challenge.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let the argument of the exponential be $f(x)$ (so its power series coefficients are the gammas) and let the right hand side be $g(x)$ (with deltas as its coefficients), so that by definition&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$g(x) = exp(f(x))$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Differentiating both sides and replacing $exp(f)$ with $g$ yields&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$g' = f' * g$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Writing this out as power series gives the desired result: the delta comes from $g'$ while the convolution of the gammas and deltas comes from $f' * g$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You don't have to worry about convergence (and the whole machinery of Taylor series), by the way: all these calculations can be performed in the ring of formal power series.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-08-28T01:39:46.570" LastActivityDate="2010-08-28T01:39:46.570" CommentCount="3" />
  <row Id="2146" PostTypeId="1" CreationDate="2010-08-26T17:26:23.483" Score="2" ViewCount="94" Body="&lt;p&gt;hi,&#xA; I have to compare pairs of audio strems as 1d time series. Looking at the aligned trajectories I need to either cluster them together or assume they arise from independent generators. &#xA; I remember hearing of Dirichlet processes being applied. Could try to defined a hidden Markov model on the structure, not sure yet how the emission probabilities would work out, but it must be possible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;any recommendations?&#xA;Best&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDITS- set of 1D data time series s in {1...N}, another set of trajectories q in {1...N} all over a discrete time domain t=0...t=T under a uniform interval. Task, determine whether under certain assumptions (eg. iid, normality etc), is the generator under these assumptions a valid null hypothesis to have generated the two categories of trajectories s and q? Or is the null hypothesis abandoned for a more complex model?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So do we use an autoregressive model? a Gaussian process would be hard to interpret the inverse covariance. Some parametric model would seem to be valid. Don't know... any ideas?&lt;/p&gt;&#xA;" OwnerUserId="1098" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-16T06:47:00.887" LastActivityDate="2010-09-16T06:47:00.887" Title="For data similar to audio, how to determine if their are 1 or 2 categories?" Tags="&lt;machine-learning&gt;&lt;time-series&gt;&lt;clustering&gt;" CommentCount="9" FavoriteCount="3" />
  <row Id="2147" PostTypeId="2" ParentId="1142" CreationDate="2010-08-26T18:02:43.510" Score="1" ViewCount="" Body="&lt;p&gt;A good solution will have several ingredients, including:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Use a resistant, moving window smooth to remove nonstationarity.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Re-express the original data so that the residuals with respect to the smooth are approximately symmetrically distributed.  Given the nature of your data, it's likely that their square roots or logarithms would give symmetric residuals.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Apply control chart methods, or at least control chart thinking, to the residuals.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;As far as that last one goes, control chart thinking shows that &quot;conventional&quot; thresholds like 2 SD or 1.5 times the IQR beyond the quartiles work poorly because they trigger too many false out-of-control signals.  People usually use 3 SD in control chart work, whence 2.5 (or even 3) times the IQR beyond the quartiles would be a good starting point.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have more or less outlined the nature of Rob Hyndman's solution while adding to it two major points: the potential need to re-express the data and the wisdom of being more conservative in signaling an outlier.  I'm not sure that Loess is good for an online detector, though, because it doesn't work well at the endpoints.  You might instead use something as simple as a moving median filter (as in Tukey's resistant smoothing).  If outliers don't come in bursts, you can use a narrow window (5 data points, perhaps, which will break down only with a burst of 3 or more outliers within a group of 5).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you have performed the analysis to determine a good re-expression of the data, it's unlikely you'll need to change the re-expression.  Therefore, your online detector really only needs to reference the most recent values (the latest window) because it won't use the earlier data at all.  If you have really long time series you could go further to analyze autocorrelation and seasonality (such as recurring daily or weekly fluctuations) to improve the procedure.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-11-03T00:21:10.310" LastActivityDate="2010-11-03T00:21:10.310" />
  <row Id="2148" PostTypeId="2" ParentId="2142" CreationDate="2010-08-26T20:43:29.787" Score="2" ViewCount="" Body="&lt;p&gt;I would suggest that transformations aren't important to get a normal distribution for your errors. Normality isn't a necessary assumption. If you have &quot;enough&quot; data, the central limit theorem kicks in and your standard estimates become asymptotically normal. Alternatively, you can use bootstrapping as a non-parametric means to estimate the standard errors. (Homoskedasticity, a common variance for the observations across units, is required for your standard errors to be right; robust options permit heteroskedasticity).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead, transformations help to ensure that a linear model is appropriate. To give a sense of this, let's consider how we can interpret the coefficients in transformed models:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;outcome is units, predictors is units: A one unit change in the predictor leads to a beta unit change in the outcome.&lt;/li&gt;&#xA;&lt;li&gt;outcome in units, predictor in log units: A one percent change in the predictor leads to a beta/100 unit change in the outcome.&lt;/li&gt;&#xA;&lt;li&gt;outcome in log units, predictor in units: A one unit change in the predictor leads to a beta x 100% change in the outcome.&lt;/li&gt;&#xA;&lt;li&gt;outcome in log units, predictor in log units: A one percent change in the predictor leads to a beta percent change in the outcome.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If transformations are necessary to have your model make sense (i.e., for linearity to hold), then the estimate from this model should be used for inference. An estimate from a model that you don't believe isn't very helpful. The interpretations above can be quite useful in understanding the estimates from a transformed model and can often be more relevant to the question at hand. For example, economists like the log-log formulation because the interpretation of beta is an elasticity, an important measure in economics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd add that the back transformation doesn't work because the expectation of a function is not the function of the expectation; the log of the expected value of beta is not the expected value of the log of beta. Hence, your estimator is not unbiased. This throws off standard errors, too.&lt;/p&gt;&#xA;" OwnerUserId="401" LastActivityDate="2010-08-26T20:43:29.787" />
  <row Id="2149" PostTypeId="1" AcceptedAnswerId="2153" CreationDate="2010-08-26T21:00:11.910" Score="10" ViewCount="499" Body="&lt;p&gt;A &lt;a href=&quot;http://en.wikipedia.org/wiki/Particle_filter&quot; rel=&quot;nofollow&quot;&gt;particle filter&lt;/a&gt; and &lt;a href=&quot;http://en.wikipedia.org/wiki/Kalman_filter&quot; rel=&quot;nofollow&quot;&gt;Kalman filter&lt;/a&gt; are both &lt;a href=&quot;http://en.wikipedia.org/wiki/Recursive_Bayesian_estimation&quot; rel=&quot;nofollow&quot;&gt;recursive Bayesian estimators&lt;/a&gt;.  I often encounter Kalman filters in my field, but very rarely see the usage of a particle filter.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When would one be used over the other?&lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-08-27T06:56:54.697" Title="What is the difference between a particle filter (sequential Monte Carlo) and a Kalman filter?" Tags="&lt;bayesian&gt;&lt;particle-filter&gt;&lt;kalman-filter&gt;" AnswerCount="1" FavoriteCount="5" />
  <row Id="2150" PostTypeId="2" ParentId="2032" CreationDate="2010-08-26T21:24:56.327" Score="5" ViewCount="" Body="&lt;p&gt;There's no need to call it Predictive Analytics :) It already has &lt;strong&gt;two&lt;/strong&gt; names: statistics, and data mining. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Beginner Stats Book: Statistics in Plain English&lt;br&gt;&#xA;Advanced Stats Book: Multivariate Analysis, by Hair&lt;br&gt;&#xA;Data Mining Book: I still haven't found a great one, but Data Mining by Witten is okay.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Don't get too confused by all the details. There are only so many things you can accomplish in general:  &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;predict a real number (regression)&lt;/li&gt;&#xA;&lt;li&gt;predict a whole number (classification)&lt;/li&gt;&#xA;&lt;li&gt;modeling (same as the above two, but the model is understandable by humans)&lt;/li&gt;&#xA;&lt;li&gt;group similar observations (clustering)&lt;/li&gt;&#xA;&lt;li&gt;group similar factors (factor analysis)&lt;/li&gt;&#xA;&lt;li&gt;describe a single factor&lt;/li&gt;&#xA;&lt;li&gt;describe the relationship between multiple factors (correlation, association, etc)&lt;/li&gt;&#xA;&lt;li&gt;determine if a population value is different from another, based on a sample&lt;/li&gt;&#xA;&lt;li&gt;design experiments and calculate sample size&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;good luck!&lt;/p&gt;&#xA;" OwnerUserId="74" LastActivityDate="2010-08-26T21:24:56.327" CommunityOwnedDate="2010-08-26T21:24:56.327" />
  <row Id="2151" PostTypeId="1" CreationDate="2010-08-27T01:56:42.663" Score="3" ViewCount="662" Body="&lt;p&gt;In other words, instead of having a two class problem I am dealing with 4 classes and still would like to assess performance using AUC.&lt;/p&gt;&#xA;" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-08-27T02:41:15.557" LastActivityDate="2010-08-27T13:22:46.397" Title="How to plot ROC curves in multiclass classification?" Tags="&lt;classification&gt;&lt;roc&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="2152" PostTypeId="2" ParentId="2151" CreationDate="2010-08-27T02:58:26.190" Score="2" ViewCount="" Body="&lt;p&gt;While the math is beyond me this general review article has some references you will likely be interested in, and has a brief description of multi-class ROC graphs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An introduction to ROC analysis by Tom Fawcett&#xA;Pattern Recognition Letters&#xA;&lt;a href=&quot;http://dx.doi.org/10.1016/j.patrec.2005.10.010&quot; rel=&quot;nofollow&quot;&gt;Volume 27, Issue 8, June 2006, Pages 861-874&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Link to &lt;a href=&quot;http://www.google.gr/url?sa=t&amp;amp;source=web&amp;amp;cd=1&amp;amp;ved=0CB8QFjAA&amp;amp;url=http%3A%2F%2Fciteseerx.ist.psu.edu%2Fviewdoc%2Fdownload%3Fdoi%3D10.1.1.98.4088%26rep%3Drep1%26type%3Dpdf&amp;amp;ei=nlx3TIq-LJCk4Abn_Li3Bg&amp;amp;usg=AFQjCNHO-_yjWAJrRVnJms7MbcqaJkd8eg&amp;amp;sig2=sqERq2v68UvOhJDMviOklg&quot; rel=&quot;nofollow&quot;&gt;pdf&lt;/a&gt; as provided by gd047- thanks&lt;/p&gt;&#xA;" OwnerUserId="1036" LastEditorUserId="1036" LastEditorDisplayName="" LastEditDate="2010-08-27T13:22:46.397" LastActivityDate="2010-08-27T13:22:46.397" CommentCount="3" />
  <row Id="2153" PostTypeId="2" ParentId="2149" CreationDate="2010-08-27T06:23:29.647" Score="14" ViewCount="" Body="&lt;p&gt;In a linear system with Gaussian noise, the Kalman filter is optimal. In a system that is nonlinear, the Kalman filter can be used for state estimation, but the particle filter may give better results at the price of additional computational effort. In a system that has non-Gaussian noise, the Kalman filter is the optimal &lt;em&gt;linear&lt;/em&gt; filter, but again the particle filter may perform better. The &lt;a href=&quot;http://en.wikipedia.org/wiki/Kalman_filter#Unscented_Kalman_filter&quot; rel=&quot;nofollow&quot;&gt;unscented Kalman filter&lt;/a&gt; (UKF) provides a balance between the low computational effort of the Kalman filter and the high performance of the particle filter. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The particle filter has some similarities with the UKF in that it transforms a set of points via known nonlinear equations and combines the results to estimate the mean and covariance of the state. However, in the particle filter the points are chosen randomly, whereas in the UKF the points are chosen on the basis of a specific algorithm &lt;strong&gt;*&lt;/strong&gt;. Because of this, the number of points used in a particle filter generally needs to be much greater than the number of points in a UKF. Another difference between the two filters is that the estimation error in a UKF does not converge to zero in any sense, but the estimation error in a particle filter does converge to zero as the number of particles (and hence the computational effort) approaches infinity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;*&lt;/strong&gt; The unscented transformation is a method for calculating the statistics of a random variable which undergoes a nonlinear transformation and uses the intuition (which also applies to the particle filter) that it is easier to approximate a probability distribution than it is to approximate an arbitrary nonlinear function or transformation. See also &lt;a href=&quot;http://stats.stackexchange.com/questions/1709/how-to-draw-a-probable-outcome-from-a-distribution/1717#1717&quot;&gt;this&lt;/a&gt; as an example of how the points are chosen in UKF. &lt;/p&gt;&#xA;" OwnerUserId="339" LastEditorUserId="339" LastEditorDisplayName="" LastEditDate="2010-08-27T06:56:54.697" LastActivityDate="2010-08-27T06:56:54.697" />
  <row Id="2154" PostTypeId="1" AcceptedAnswerId="2157" CreationDate="2010-08-27T07:32:57.340" Score="1" ViewCount="91" Body="&lt;p&gt;(I know it is not quite appropriate to quote it as &quot;part 2&quot;, but since the question has been dormant for quite a while, I hope by doing this will rise peoples' attention again, you may have a look of part I &lt;a href=&quot;http://stats.stackexchange.com/questions/1228/how-to-interpret-a-control-chart-containing-a-majority-of-zero-values&quot;&gt;here&lt;/a&gt;.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have come across an article online talking about the case similar to mine, that most of the time the case count is zero, make sometimes when the case number increases to one, it already shoots above the control level and consider the case as &quot;out of control&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since c-chart will be easier for my bosses to read and interpret, I wonder if the method is sound, or did anyone have some more official reference on this method? (I have googled quite a while but I can find nothing)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The article can be found &lt;a href=&quot;http://www.spcforexcel.com/small-sample-case-for-c-and-u-control-charts&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An to further my question, I want to ask one more thing: for the assumption of c-chart that the case count needs to follow a Poisson distribution, is it applicable to all lambda (i.e. mean of case count)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks again.&lt;/p&gt;&#xA;" OwnerUserId="588" LastActivityDate="2010-08-27T14:12:33.627" Title="How to interpret a control chart containing a majority of zero values? (Part 2)" Tags="&lt;control-chart&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="2155" PostTypeId="2" ParentId="2151" CreationDate="2010-08-27T08:12:12.627" Score="4" ViewCount="" Body="&lt;p&gt;It seems you are looking for multi-class ROC analysis, which is a kind of multi-objective optimization covered in a &lt;a href=&quot;http://www.cs.bris.ac.uk/~flach/ICML04tutorial/&quot; rel=&quot;nofollow&quot;&gt;tutorial&lt;/a&gt; at ICML'04. As in several multi-class problem, the idea is generally to carry out pairwise comparison (one class vs. all other classes, one class vs. another class, see (1) or the &lt;em&gt;Elements of Statistical Learning&lt;/em&gt;), and there is a recent paper by Landgrebe and Duin on that topic, &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.108.3250&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;Approximating the multiclass ROC by pairwise analysis&lt;/a&gt;, Pattern Recognition Letters 2007 28: 1747-1758. Now, for visualization purpose, I've seen some papers some time ago, most of them turning around &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.14.2427&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;volume under the ROC surface&lt;/a&gt; (VUS) or &lt;a href=&quot;http://cskku.kkh.go.th/homework/ke/paper/visualization%20and%20analysis%20of%20classifiers%20performance%20in%20multi-class%20medical%20data.pdf&quot; rel=&quot;nofollow&quot;&gt;Cobweb diagram&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't know, however, if there exists an R implementation of these methods, although I think the &lt;code&gt;stars()&lt;/code&gt; function might be used for cobweb plot. I just ran across a Matlab toolbox that seems to offer multi-class ROC analysis, &lt;a href=&quot;http://doc.prsdstudio.com/latest/guide/ROC_analysis.html&quot; rel=&quot;nofollow&quot;&gt;PRSD Studio&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other papers that may also be useful as a first start for visualization/computation:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://users.dsic.upv.es/~flip/ROCML2005/papers/fieldsend2CRC.pdf&quot; rel=&quot;nofollow&quot;&gt;Visualisation of multi-class ROC surfaces&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://homepage.tudelft.nl/a9p19/papers/prasa_06_vuc.pdf&quot; rel=&quot;nofollow&quot;&gt;A simplified extension of the Area under the ROC to the multiclass domain&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;References:&lt;/strong&gt;&lt;br&gt;&#xA;1. Allwein, E.L., Schapire, R.E. and Singer, Y. (2000). Reducing multiclass to binary: A unifying approach for margin classifiers. &lt;em&gt;Journal of Machine Learning Research&lt;/em&gt;, &lt;strong&gt;1&lt;/strong&gt;:113–141.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-08-27T08:12:12.627" CommentCount="1" />
  <row Id="2156" PostTypeId="2" ParentId="1241" CreationDate="2010-08-27T08:22:12.153" Score="2" ViewCount="" Body="&lt;p&gt;In case you're interested in further references, an extensive list of papers is available on K.H. Zou's website, &lt;a href=&quot;http://www.spl.harvard.edu/archive/spl-pre2007/pages/ppl/zou/roc.html&quot; rel=&quot;nofollow&quot;&gt;Receiver Operating Characteristic (ROC) Literature Research&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ROC curves are also used when one is interested in comparing different classifiers performance, with wide applications in biomedical research and bioinformatics.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-08-27T08:22:12.153" />
  <row Id="2157" PostTypeId="2" ParentId="2154" CreationDate="2010-08-27T09:27:37.370" Score="1" ViewCount="" Body="&lt;p&gt;C-charts basically works like this: you collect all your observations (this is an approximation of getting only observations from &quot;normal&quot; periods) of a case count, then fit Poisson distribution to it (so here you apply all Poisson assumptions, mainly that each case occurance is independent to the others), and finally test all case counts with H0 that they are just from this fitted distribution.&#xA;So, if you have majority of zeros, it will just drive $\lambda$ of fit near zero; for $\lambda&amp;lt;0.01$ even one case will be something strange (on $p$-value=1%).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT:&#xA;In the article you've linked the whole fitting is done just by taking mean as $\lambda$, while testing just by $3\sigma$ criterion based on the fact that Poisson's variance is also $\lambda$. Still it is sufficient for this case.&lt;/p&gt;&#xA;" OwnerUserId="88" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-08-27T13:37:10.033" LastActivityDate="2010-08-27T13:37:10.033" CommentCount="3" />
  <row Id="2158" PostTypeId="2" ParentId="1815" CreationDate="2010-08-27T09:36:24.300" Score="0" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0471699462&quot; rel=&quot;nofollow&quot;&gt;Experiments: Planning, Analysis and Optimization&lt;/a&gt; by Wu &amp;amp; Hamada.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm only a couple of chapters in, so not yet in a position to recommend confidently, but so far it looks like a good graduate text, reasonably detailed, comprehensive and up-to-date. Has more of a &quot;no nonsense&quot; feel than the Montgomery.&lt;/p&gt;&#xA;" OwnerUserId="174" LastActivityDate="2010-08-27T09:36:24.300" CommunityOwnedDate="2010-08-27T09:36:24.300" />
  <row Id="2159" PostTypeId="2" ParentId="1595" CreationDate="2010-08-27T10:10:35.860" Score="3" ViewCount="" Body="&lt;p&gt;I use Python for statistical analysis and forecasting.  As mentioned by others above, Numpy and Matplotlib are good workhorses.  I also use ReportLab for producing PDF output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm currently looking at both Resolver and Pyspread which are Excel-like spreadsheet applications which are based on Python.  Resolver is a commercial product but &lt;a href=&quot;http://pyspread.sourceforge.net/&quot; rel=&quot;nofollow&quot;&gt;Pyspread&lt;/a&gt; is still open-source. (Apologies, I'm limited to only one link)&lt;/p&gt;&#xA;" OwnerUserId="1105" LastActivityDate="2010-08-27T10:10:35.860" CommunityOwnedDate="2010-08-27T15:07:18.913" CommentCount="1" />
  <row Id="2160" PostTypeId="2" ParentId="1815" CreationDate="2010-08-27T11:15:23.930" Score="0" ViewCount="" Body="&lt;p&gt;Not really a book but a gentle introduction on DoE in R: &lt;a href=&quot;http://cran.r-project.org/doc/contrib/Vikneswaran-ED_companion.pdf&quot; rel=&quot;nofollow&quot;&gt;An R companion to Experimental Design&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-08-27T11:15:23.930" CommunityOwnedDate="2010-08-27T11:15:23.930" />
  <row Id="2162" PostTypeId="1" AcceptedAnswerId="2164" CreationDate="2010-08-27T14:30:36.417" Score="6" ViewCount="273" Body="&lt;p&gt;Thank you for reading. I find that I am unable to draw lines/points or add text outside of the plot area. If I draw a line from inside the plot (within the axis) area to the area of the xlab main title areas only the part within the plot shows. In a multiple plot graph (mfrow) the line/point will only be drawn inside the last active plot.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plot(0);&#xA;l=locator(2,type='n');&#xA;lines(l, type='l'); &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Any solution to this problem? Thanks!&lt;/p&gt;&#xA;" OwnerUserId="1084" LastActivityDate="2010-08-27T14:48:36.610" Title="R - How to draw a line or add a text outside of the plot area." Tags="&lt;r&gt;&lt;data-visualization&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="2163" PostTypeId="2" ParentId="1815" CreationDate="2010-08-27T14:38:12.360" Score="0" ViewCount="" Body="&lt;p&gt;Ronald Fisher's &lt;a href=&quot;http://en.wikipedia.org/wiki/The_Design_of_Experiments&quot; rel=&quot;nofollow&quot;&gt;The Design of Experiments&lt;/a&gt; (link is Wikipedia rather than Amazon since it is long out of print) is interesting for historical context. The book is often credited as founding the whole field, and certainly did a lot to promote things like blocking, randomisation and factorial design, though things have moved on a bit since.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a period document it's quite fascinating, but it's also maddening. In the absence of a common terminology and notation, a lot of time is spent painstakingly explaining things in what now seems comically-stilted English. If you had to use it as a reference to look up how to calculate something you'd probably gnaw your own leg off. But the &lt;em&gt;terribly polite&lt;/em&gt; hatchet job on some of Galton's analysis is entertaining.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(I know, I know -- how the readers of tomorrow will laugh at the archaisms of today's scientific literature...)&lt;/p&gt;&#xA;" OwnerUserId="174" LastActivityDate="2010-08-27T14:38:12.360" CommunityOwnedDate="2010-08-27T14:38:12.360" />
  <row Id="2164" PostTypeId="2" ParentId="2162" CreationDate="2010-08-27T14:48:36.610" Score="7" ViewCount="" Body="&lt;p&gt;The xpd parameter controls where you can draw. Check the current value with par()$xpd and then try setting par(xpd=NA).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the par help:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; ‘xpd’ A logical value or ‘NA’.  If ‘FALSE’, all plotting is&#xA;      clipped to the plot region, if ‘TRUE’, all plotting is&#xA;      clipped to the figure region, and if ‘NA’, all plotting is&#xA;      clipped to the device region. See also ‘clip’.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1076" LastActivityDate="2010-08-27T14:48:36.610" CommentCount="3" />
  <row Id="2165" PostTypeId="2" ParentId="53" CreationDate="2010-08-27T16:09:10.220" Score="3" ViewCount="" Body="&lt;p&gt;A late reply, but you may find VERY useful handouts on multivariate data analysis &quot;à la française&quot; on the &lt;a href=&quot;http://pbil.univ-lyon1.fr/R/enseignement.html&quot; rel=&quot;nofollow&quot;&gt;Bioinformatics department&lt;/a&gt; of Lyon. These come from the authors of the R &lt;a href=&quot;http://cran.r-project.org/web/packages/ade4/index.html&quot; rel=&quot;nofollow&quot;&gt;ade4&lt;/a&gt; package. It is in french, though.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-08-27T16:09:10.220" />
  <row Id="2166" PostTypeId="2" ParentId="213" CreationDate="2010-08-27T16:44:31.610" Score="4" ViewCount="" Body="&lt;p&gt;I'm not aware that anyone is doing this, but I generally like to try &lt;a href=&quot;http://en.wikipedia.org/wiki/Dimensionality_reduction&quot; rel=&quot;nofollow&quot;&gt;dimensionality reduction&lt;/a&gt; when I have a problem like this.  You might look into a method from manifold learning or &lt;a href=&quot;http://en.wikipedia.org/wiki/Nonlinear_dimensionality_reduction&quot; rel=&quot;nofollow&quot;&gt;non-linear dimensionality reduction&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An example would be a &lt;a href=&quot;http://en.wikipedia.org/wiki/Kohonen_maps&quot; rel=&quot;nofollow&quot;&gt;Kohonen map&lt;/a&gt;.  A good reference for R is &lt;a href=&quot;http://www.jstatsoft.org/v21/i05/paper&quot; rel=&quot;nofollow&quot;&gt;&quot;Self- and Super-organizing Maps in R: The kohonen Package&quot;&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-08-27T16:44:31.610" />
  <row Id="2167" PostTypeId="1" AcceptedAnswerId="2168" CreationDate="2010-08-27T17:00:11.247" Score="5" ViewCount="334" Body="&lt;p&gt;The &lt;a href=&quot;http://en.wikipedia.org/wiki/Kernel_trick&quot; rel=&quot;nofollow&quot;&gt;kernel trick&lt;/a&gt; is used in several machine learning models (e.g. &lt;a href=&quot;http://en.wikipedia.org/wiki/Support_vector_machine&quot; rel=&quot;nofollow&quot;&gt;SVM&lt;/a&gt;).  It was first introduced in the &quot;Theoretical foundations of the potential function method in pattern recognition learning&quot; paper in 1964.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The wikipedia definition says that it is &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;a method for using a linear classifier&#xA;  algorithm to solve a non-linear&#xA;  problem by mapping the original&#xA;  non-linear observations into a&#xA;  higher-dimensional space, where the&#xA;  linear classifier is subsequently&#xA;  used; this makes a linear&#xA;  classification in the new space&#xA;  equivalent to non-linear&#xA;  classification in the original space.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;One example of a linear model that has been extended to non-linear problems is the &lt;a href=&quot;http://en.wikipedia.org/wiki/Kernel_principal_component_analysis&quot; rel=&quot;nofollow&quot;&gt;kernel PCA&lt;/a&gt;.  Can the kernel trick be applied to any linear model, or does it have certain restrictions?&lt;/p&gt;&#xA;" OwnerUserId="5" LastEditorUserId="5" LastEditorDisplayName="" LastEditDate="2010-08-27T18:03:23.793" LastActivityDate="2010-08-28T07:52:54.063" Title="Applying the &quot;kernel trick&quot; to linear methods?" Tags="&lt;machine-learning&gt;&lt;kernel&gt;" AnswerCount="2" CommentCount="3" />
  <row Id="2168" PostTypeId="2" ParentId="2167" CreationDate="2010-08-27T17:16:04.163" Score="5" ViewCount="" Body="&lt;p&gt;The kernel trick can only be applied to linear models where the examples in the problem formulation appear as dot products (Support Vector Machines, PCA, etc).&lt;/p&gt;&#xA;" OwnerUserId="881" LastActivityDate="2010-08-27T17:16:04.163" CommentCount="11" />
  <row Id="2169" PostTypeId="1" AcceptedAnswerId="2288" CreationDate="2010-08-27T18:23:40.470" Score="4" ViewCount="131" Body="&lt;p&gt;I am trying to compute the standard error of the sample &lt;a href=&quot;http://en.wikipedia.org/wiki/Spectral_risk_measure&quot; rel=&quot;nofollow&quot;&gt;spectral risk measure&lt;/a&gt;, which is used as a metric for portfolio risk. Briefly, a sample spectral risk measure is defined as &#xA;$q = \sum_i w_i x_{(i)}$, where $x_{(i)}$ are the sample order statistics, and $w_i$ is a sequence of monotonically non-increasing non-negative weights that sum to $1$.  I would like to compute the standard error of $q$ (preferrably not via bootstrap). I don't know much about L-estimators, but it looks to me like $q$ is a kind of L-estimator (but with extra restrictions imposed on the weights $w_i$), so this should probably be an easily solved problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;edit&lt;/strong&gt;: per @srikant's question, I should note that the weights $w_i$ are chosen &lt;em&gt;a priori&lt;/em&gt; by the user, and should be considered independent from the samples $x$.&lt;/p&gt;&#xA;" OwnerUserId="795" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-04T15:56:08.473" LastActivityDate="2010-11-04T16:51:47.097" Title="How to compute the standard error of an L-estimator?" Tags="&lt;estimation&gt;&lt;finance&gt;&lt;standard-error&gt;" AnswerCount="2" CommentCount="5" />
  <row Id="2170" PostTypeId="1" AcceptedAnswerId="2177" CreationDate="2010-08-27T20:17:34.687" Score="5" ViewCount="288" Body="&lt;p&gt;Say some previous findings identified a curvilinear effect of X on Y, (specifically that X had a positive effect on Y, and that X^2 had a negative effect). You want to see if the same holds for your entirely different sample (although everything else between studies, constructs/measures, are exactly the same). Neither the previous study nor my study are experimental (so I am not manipulating X, merely observing it).  There is no explicit theoretical reason why a curvilinear effect would occur.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I will give some examples of how I would currently go about seeing if this is true, but I would like some input and peoples opinions on what they believe would be a preferrable method, if any of my suggestions are innapropriate, and of course if their are other alternatives.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Examples:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Simply examine the bivariate scatterplot and fit some type of smoothing line to the mean of Y over bins of X (e.g. LOESS). Although confounding could be an issue, if a curvilinear effect exists their will likely be some evidence in its distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Examine the &lt;a href=&quot;http://en.wikipedia.org/wiki/Partial_regression_plot&quot; rel=&quot;nofollow&quot;&gt;partial regression plot&lt;/a&gt; or other visualization techniques to identify the effect of X on Y independent of other confounding variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) Use some type of model selection criteria (e.g. BIC), and determine whether a model including X^2 is preferred over a model without X^2&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4) Include a model with X^2 and see if X^2 has a statistically significant regression coefficient.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Like I said any other suggestions are welcome as well. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: In this context, my main concern is to identify if the effect of X on Y is best represented in a similar manner to the previous study. While their could be other substantively interesting points to compare between studies (like the magnitude of the effect of X on Y), this is not my main concern.&lt;/p&gt;&#xA;" OwnerUserId="1036" LastEditorUserId="1036" LastEditorDisplayName="" LastEditDate="2010-08-30T12:27:26.667" LastActivityDate="2010-09-03T17:57:15.193" Title="Preferred method for identifying curvilinear effect in multi-variable regression framework" Tags="&lt;modeling&gt;&lt;regression&gt;&lt;methodology&gt;" AnswerCount="2" FavoriteCount="2" />
  <row Id="2171" PostTypeId="1" AcceptedAnswerId="2172" CreationDate="2010-08-27T22:00:32.020" Score="11" ViewCount="358" Body="&lt;p&gt;I'm interested in learning how to create the type of visualizations you see at &lt;a href=&quot;http://flowingdata.com&quot; rel=&quot;nofollow&quot;&gt;http://flowingdata.com&lt;/a&gt; and informationisbeautiful. EDIT: Meaning, visualizations that are interesting in of themselves -- kinda like the NY Times graphics, as opposed to a quick something for a report.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What kinds of tools are used to create these -- is it mostly a lot of Adobe Illustrator/Photoshop? What are good resources (books, websites, etc.) to learn how to use these tools for data visualization in particular?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know &lt;em&gt;what&lt;/em&gt; I want visualizations to look like (and I'm familiar with design principles, e.g., from Tufte's books), but I have no idea &lt;em&gt;how&lt;/em&gt; to create them.&lt;/p&gt;&#xA;" OwnerUserId="1106" LastEditorUserId="1106" LastEditorDisplayName="" LastEditDate="2010-08-28T04:31:24.107" LastActivityDate="2010-11-17T02:44:28.000" CommunityOwnedDate="2010-08-28T04:09:07.887" Title="Resources for learning to create data visualizations?" Tags="&lt;data-visualization&gt;" AnswerCount="5" CommentCount="1" FavoriteCount="11" />
  <row Id="2172" PostTypeId="2" ParentId="2171" CreationDate="2010-08-27T22:50:35.313" Score="13" ViewCount="" Body="&lt;p&gt;Flowing data regularly discusses the tools that he uses.  See, for instance:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://flowingdata.com/2008/10/20/40-essential-tools-and-resources-to-visualize-data/&quot; rel=&quot;nofollow&quot;&gt;40 Essential Tools and Resources to Visualize Data&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://flowingdata.com/2009/09/03/what-visualization-toolsoftware-should-you-use-getting-started/&quot; rel=&quot;nofollow&quot;&gt;What Visualization Tool/Software Should You Use? – Getting Started&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;He also shows in great detail how he makes graphics on occasion, such as:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://flowingdata.com/2009/11/12/how-to-make-a-us-county-thematic-map-using-free-tools/&quot; rel=&quot;nofollow&quot;&gt;How to Make a US County Thematic Map Using Free Tools&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://flowingdata.com/2008/12/16/how-to-make-a-graph-in-adobe-illustrator/&quot; rel=&quot;nofollow&quot;&gt;How to Make a Graph in Adobe Illustrator&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://flowingdata.com/2010/01/21/how-to-make-a-heatmap-a-quick-and-easy-solution/&quot; rel=&quot;nofollow&quot;&gt;How to Make a Heatmap – a Quick and Easy Solution&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;There are also other questions on this site:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://stats.stackexchange.com/questions/224/visualization-libraries&quot;&gt;http://stats.stackexchange.com/questions/224/visualization-libraries&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://stats.stackexchange.com/questions/216/web-visualization-libraries&quot;&gt;http://stats.stackexchange.com/questions/216/web-visualization-libraries&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;IMO, try:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;R and ggplot2: this is a &lt;a href=&quot;http://vimeo.com/13454730&quot; rel=&quot;nofollow&quot;&gt;good introductory video&lt;/a&gt;, but the ggplot2 website has lots of resources.&lt;/li&gt;&#xA;&lt;li&gt;Processing: plenty of &lt;a href=&quot;http://processing.org/learning/&quot; rel=&quot;nofollow&quot;&gt;good tutorials on the homepage&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;li&gt;Protovis: also a plethora of &lt;a href=&quot;http://vis.stanford.edu/protovis/ex/&quot; rel=&quot;nofollow&quot;&gt;great examples on the homepage&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;You can use Adobe afterwards to clean these up.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also look at the R &lt;code&gt;webvis&lt;/code&gt; package, although it isn't as complete as &lt;code&gt;ggplot2&lt;/code&gt;.  From R, you can run this command to see the Playfair's Wheat example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;install.packages(&quot;webvis&quot;)&#xA;library(webvis)&#xA;demo(&quot;playfairs.wheat&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Lastly, my favorite commercial applications for interactive visualization are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.tableausoftware.com/&quot; rel=&quot;nofollow&quot;&gt;Tableau&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://spotfire.tibco.com/&quot; rel=&quot;nofollow&quot;&gt;Spotfire&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.qlikview.com/&quot; rel=&quot;nofollow&quot;&gt;Qlikview&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="5" LastEditorUserId="5" LastEditorDisplayName="" LastEditDate="2010-08-28T00:06:51.733" LastActivityDate="2010-08-28T00:06:51.733" CommunityOwnedDate="2010-08-28T07:10:15.533" CommentCount="3" />
  <row Id="2173" PostTypeId="2" ParentId="2171" CreationDate="2010-08-27T23:47:09.177" Score="1" ViewCount="" Body="&lt;p&gt;You'll spend a lot of time getting up to speed with R. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;RapidMiner is free and open source and graphical, and has plenty of good visualizations, and you can export them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have money to spare, or are a university staff/student then JMP is also very freaking nice. It can make some very pretty graphs, very very easily. Can export to flash or PNG or PDF or what have you. &lt;/p&gt;&#xA;" OwnerUserId="74" LastActivityDate="2010-08-27T23:47:09.177" CommunityOwnedDate="2010-08-28T07:11:23.257" CommentCount="2" />
  <row Id="2174" PostTypeId="2" ParentId="2171" CreationDate="2010-08-28T00:26:17.800" Score="3" ViewCount="" Body="&lt;p&gt;Already mentioned processing has a nice set of books available. See: &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0262182629&quot; rel=&quot;nofollow&quot;&gt;1&lt;/a&gt;, &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/144937980X&quot; rel=&quot;nofollow&quot;&gt;2&lt;/a&gt;, &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0123736021&quot; rel=&quot;nofollow&quot;&gt;3&lt;/a&gt;, &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/159059617X&quot; rel=&quot;nofollow&quot;&gt;4&lt;/a&gt;, &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0470375485&quot; rel=&quot;nofollow&quot;&gt;5&lt;/a&gt;, &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0596514557&quot; rel=&quot;nofollow&quot;&gt;6&lt;/a&gt;, &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1568817169&quot; rel=&quot;nofollow&quot;&gt;7&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You will find lots of stuff on the web to help you start with R. As next step then ggplot2 has excellent web &lt;a href=&quot;http://had.co.nz/ggplot2/&quot; rel=&quot;nofollow&quot;&gt;documentation&lt;/a&gt;. I also found Hadley's &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0387981403&quot; rel=&quot;nofollow&quot;&gt;book&lt;/a&gt; very helpful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Python might be another way to go. Especially with tools like:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://matplotlib.sourceforge.net/&quot; rel=&quot;nofollow&quot;&gt;matplotlib&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://networkx.lanl.gov/&quot; rel=&quot;nofollow&quot;&gt;NetworkX&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://cneurocvs.rmki.kfki.hu/igraph/&quot; rel=&quot;nofollow&quot;&gt;igraph&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://code.enthought.com/chaco/&quot; rel=&quot;nofollow&quot;&gt;Chaco&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://code.enthought.com/projects/mayavi/&quot; rel=&quot;nofollow&quot;&gt;Mayavi&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;All projects are well documented on the web. You might also consider peeking into &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1430218436&quot; rel=&quot;nofollow&quot;&gt;some&lt;/a&gt; &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1847197906&quot; rel=&quot;nofollow&quot;&gt;books&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lastly, &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0387329064&quot; rel=&quot;nofollow&quot;&gt;Graphics of Large Datasets&lt;/a&gt; book could be also some help.&lt;/p&gt;&#xA;" OwnerUserId="22" LastActivityDate="2010-08-28T00:26:17.800" CommunityOwnedDate="2010-08-28T07:10:59.003" CommentCount="2" />
  <row Id="2175" PostTypeId="7" CreationDate="2010-08-28T01:20:33.947" Score="0" ViewCount="0" Body="&lt;p&gt;&lt;strong&gt;CrossValidated&lt;/strong&gt; is for statisticians, data miners, and anyone else doing data analysis or interested in it as a discipline. If you have a question about&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;statistical analysis&lt;/strong&gt;, applied or theoretical&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;designing experiments&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;collecting data&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;data mining&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;machine learning&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;visualizing data&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;probability theory&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;mathematical statistics&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;li&gt;statistical and data-driven &lt;strong&gt;computing&lt;/strong&gt; (e.g., questions about R, SAS, SPSS, Stata and Minitab)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;then you're in the right place to ask your question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Homework&lt;/strong&gt; questions are welcome.  &lt;em&gt;Please mark them with the &lt;a href=&quot;http://stats.stackexchange.com/questions/tagged/homework&quot;&gt;homework&lt;/a&gt; tag&lt;/em&gt;.  They get &lt;a href=&quot;http://meta.stackoverflow.com/questions/10811/how-to-ask-and-answer-homework-questions/10812#10812&quot;&gt;somewhat special treatment&lt;/a&gt;, because ultimately you benefit most by finding the solution &lt;em&gt;yourself.&lt;/em&gt;  The community will try to provide &lt;a href=&quot;http://meta.stats.stackexchange.com/q/12/919&quot;&gt;guidance, hints, and useful links&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;There are certain subjects that will probably get better responses on our sister sites&lt;/em&gt;. If your question is about&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Programming&lt;/strong&gt;, ask on &lt;a href=&quot;http://stackoverflow.com&quot;&gt;Stack Overflow&lt;/a&gt;.  If the language is statistically oriented (such as &lt;strong&gt;R&lt;/strong&gt; or &lt;strong&gt;Stata&lt;/strong&gt;), then decide based on the nature of your question: if it needs &lt;em&gt;statistical expertise&lt;/em&gt; to understand or answer, ask it here; if it's about an &lt;em&gt;algorithm&lt;/em&gt; or details of the &lt;em&gt;language&lt;/em&gt;, &lt;a href=&quot;http://meta.stats.stackexchange.com/questions/793/internet-support-for-statistics-software&quot;&gt;ask elsewhere&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Mathematics&lt;/strong&gt;, ask on &lt;a href=&quot;http://math.stackexchange.com&quot;&gt;math.stackexchange.com&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Bugs in software&lt;/strong&gt;, ask the people who produced the software&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="-1" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-04-07T06:50:24.433" LastActivityDate="2011-04-07T06:50:24.433" />
  <row Id="2176" PostTypeId="2" ParentId="2104" CreationDate="2010-08-28T02:32:13.413" Score="3" ViewCount="" Body="&lt;p&gt;I generally recommend avoiding these types of sphericity tests altogether by using modern mixed modeling methods.  If you are not working with few subjects this will give you a great deal of flexibility in modeling an appropriate covariance structure, freeing you from the strict assumption of sphericity when necessary.  I infer from the &lt;code&gt;str&lt;/code&gt; output that you have 16 subjects with 12 observations each (I assume balance b/c you areusing clasical method-of-moments tools) which should be enough data to fit a mixed model with structured covariance matrices via (restricted) maximum likelihood.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Without being close to your data I can't offer specific model recommendations, but a place to start in R would be to replace &lt;code&gt;aov&lt;/code&gt; in your model specifications with &lt;code&gt;lme&lt;/code&gt; (after &lt;code&gt;library(nlme)&lt;/code&gt;). The reason this will work is that you have mistakenly provided an &lt;code&gt;nlme&lt;/code&gt;-style random argument to &lt;code&gt;aov&lt;/code&gt; (when, as @Matt Albrecht pointed out, an &lt;code&gt;Error&lt;/code&gt; term would have been approriate).  In nlme, with the random argument set to &lt;code&gt;~ 1|&amp;lt;your grouping structure&amp;gt;&lt;/code&gt; &#xA;and no &lt;code&gt;correlation&lt;/code&gt; or &lt;code&gt;weight&lt;/code&gt; arguments, you are specifying a random intercept for each group, implying the response covariance within groups is &lt;code&gt;ZGZ' + R = 1G1'+ \sigma^2 I&lt;/code&gt; ==&gt; compound symmetry with between-group variance off-diagonal and between-group variance + within-group variance on the diagonal ==&gt; a spherical structure.  From there you can begin to explore (e.g. using the built-in graphical methods), model, and test (e.g. comparing information criteria or using LRTs for nested models) the various forms of non-sphericity. Some of the tools for the modeling component are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Using the &lt;code&gt;weights&lt;/code&gt; argument to model non-constant variance (diagonals) within or between groups (e.g. error variance changes between sucrose levels).&lt;/li&gt;&#xA;&lt;li&gt;Using the &lt;code&gt;correlation&lt;/code&gt; argument to model non-constant covariance (off-diagonals) within groups (e.g. a structure within a group where residual errors that are closer together in time (e.g. AR1 structure) or space (e.g. Spherical structure) are more similar).&lt;/li&gt;&#xA;&lt;li&gt;Modeling random slopes by adding terms to the LHS of the &lt;code&gt;|&lt;/code&gt; in the &lt;code&gt;random&lt;/code&gt; formula.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Though the process can be complex with many potential pitfalls, I believe it will lead you to think more about the data generating mechanism, and when combined with careful graphical checks (I recommed &lt;code&gt;lattice&lt;/code&gt; b/c &lt;code&gt;nlme&lt;/code&gt; has excellent lattice-based plotting methods -- but &lt;code&gt;ggplot&lt;/code&gt; works well too) you are likely to have not only a better scientific understanding of the process, but also less biased and more efficient estimators with which to draw inferences.&lt;/p&gt;&#xA;" OwnerUserId="1107" LastActivityDate="2010-08-28T02:32:13.413" CommentCount="1" />
  <row Id="2177" PostTypeId="2" ParentId="2170" CreationDate="2010-08-28T04:34:44.940" Score="6" ViewCount="" Body="&lt;p&gt;It sounds as though you are interested in formal inference and for that method 4 is best.  Add X^2 to a model containing terms you wish to control for and conduct a test to assess the streght of evidence for the quadratic term given the terms in the model.  Note however that &quot;absence of evidence is not evidence of absence&quot; and statistical power will come into play (this is of interest if you fail to reject or the CI contains zero).  You willof course also want to perform diagnostics of model assumptions prior to drawing conclusions.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Methods 1 and 2 are excellent exploratory tools and I would encourage exploring the relationship in as many meaningful ways as you wish (since you know &lt;em&gt;a priori&lt;/em&gt; what formal test you will conduct -- a test of the quadratic term -- this will not lead to data-driven hypothesis testing).  Other methods of exploration include plotting a fitted LOESS smoother or spline to the (possibly partial) relationships, fitting smoothers or parametric fits within subsets of the data (eg using conditioning plots), a 3d scatterplot with a fitted surface (particularly if you include continuous interactions), etc.  These plots will not only help you understand the data better but can also be used as part of a less formal case for/against the quadratic (keeping in mind that humans are excellent at spotting trends in noise).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not sure what model selection methods you refer to in 3, but generally automated model selection and testing do not mix.  If you are referring to using information criteria (AICc, BIC, ...) note that the theory behind these is based on prediction rather than testing.  So, number 4 is the most rigorous way to test the quadratic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, 2 comments on terminology:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;'multivariate' models are those whose response is a matrix, and 'multivariable' models are those with a vector response and multiple terms on the RHS.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Partial_residual_plot&quot; rel=&quot;nofollow&quot;&gt;Partial residual plots&lt;/a&gt; differ from partial regression plots..&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="1107" LastActivityDate="2010-08-28T04:34:44.940" CommentCount="2" />
  <row Id="2178" PostTypeId="2" ParentId="2167" CreationDate="2010-08-28T07:52:54.063" Score="2" ViewCount="" Body="&lt;p&gt;Two further references from &lt;a href=&quot;http://www.kyb.mpg.de/~bs&quot; rel=&quot;nofollow&quot;&gt;B. Schölkopf&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Schölkopf, B. and Smola, A.J. (2002). &lt;a href=&quot;http://mitpress.mit.edu/catalog/item/default.asp?ttype=2&amp;amp;tid=8684&quot; rel=&quot;nofollow&quot;&gt;Learning with kernels&lt;/a&gt;. The MIT Press.&lt;/li&gt;&#xA;&lt;li&gt;Schölkopf, B., Tsuda, K., and Vert, J.-P. (2004). &lt;a href=&quot;http://mitpress.mit.edu/catalog/item/default.asp?ttype=2&amp;amp;tid=10338&quot; rel=&quot;nofollow&quot;&gt;Kernel methods in computational biology&lt;/a&gt;. The MIT Press.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;and a website dedicated to &lt;a href=&quot;http://www.kernel-machines.org/&quot; rel=&quot;nofollow&quot;&gt;kernel machines&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-08-28T07:52:54.063" />
  <row Id="2179" PostTypeId="1" AcceptedAnswerId="2180" CreationDate="2010-08-28T13:34:42.963" Score="6" ViewCount="305" Body="&lt;p&gt;How to obtain a variable (attribute) importance using SVM? &lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-09-02T18:07:50.913" Title="Variable importance from SVM" Tags="&lt;machine-learning&gt;&lt;feature-selection&gt;&lt;svm&gt;" AnswerCount="2" FavoriteCount="4" />
  <row Id="2180" PostTypeId="2" ParentId="2179" CreationDate="2010-08-28T14:36:05.907" Score="3" ViewCount="" Body="&lt;p&gt;If you use l-1 penalty on the weight vector, it does automatic feature selection as the weights corresponding to irrelevant attributes are automatically set to zero. See &lt;a href=&quot;http://books.nips.cc/papers/files/nips16/NIPS2003_AA07.pdf&quot; rel=&quot;nofollow&quot;&gt;this paper&lt;/a&gt;. The (absolute) magnitude of each non-zero weights can give an idea about the importance of the corresponding attribute.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also look at &lt;a href=&quot;http://jmlr.csail.mit.edu/papers/volume3/rakotomamonjy03a/rakotomamonjy03a.pdf&quot; rel=&quot;nofollow&quot;&gt;this paper&lt;/a&gt; which uses criteria derived from SVMs to guide the attribute selection.&lt;/p&gt;&#xA;" OwnerUserId="881" LastActivityDate="2010-08-28T14:36:05.907" CommentCount="2" />
  <row Id="2181" PostTypeId="1" CreationDate="2010-08-28T17:07:59.760" Score="8" ViewCount="685" Body="&lt;p&gt;I'm interested in getting some books about multivariate analysis, and need your recommendations. Free books are always welcome, but if you know about some great non-free MVA book, please, state it.&lt;/p&gt;&#xA;" OwnerUserId="1356" LastActivityDate="2011-05-07T04:00:18.503" CommunityOwnedDate="2010-08-28T17:07:59.760" Title="Book recommendations for multivariate analysis" Tags="&lt;books&gt;&lt;multivariate-analysis&gt;" AnswerCount="10" CommentCount="2" FavoriteCount="5" />
  <row Id="2182" PostTypeId="1" CreationDate="2010-08-28T22:34:01.353" Score="7" ViewCount="322" Body="&lt;p&gt;I need help explaining, and citing basic statistics texts, papers or other references, why it is generally incorrect to use the margin of error (MOE) statistic reported in polling to naively declare a statistical tie.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An example:&#xA;Candidate A leads Candidate B in a poll, 39 - 31 percent, 4.5 percent margin-of-error for 500 surveyed voters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My friend reasons like so:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Because of the intricacies of statistical modeling, the margin of error means that A's true support could be as low as 34.5 percent and B's could be as high as 35.5 percent. Therefore, A and B are actually in a statistical dead heat.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;All help appreciated in clearly articulating the flaw my friend's reasoning.  I've tried to explain that it is incorrect to naively reject the hypothesis &quot;A leads B&quot; if $p_A-p_B &amp;lt; 2MOE$.    &lt;/p&gt;&#xA;" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-12-20T06:57:35.117" LastActivityDate="2010-12-22T06:32:25.093" Title="Can you explain why statistical tie is not naively when p_1-p_2 &lt; 2 MOE ?" Tags="&lt;polling&gt;" AnswerCount="4" CommentCount="0" />
  <row Id="2183" PostTypeId="2" ParentId="2182" CreationDate="2010-08-28T23:20:51.237" Score="5" ViewCount="" Body="&lt;p&gt;My first attempt at an answer was flawed (see below for the flawed answer). The reason it is flawed is that the margin of error (MOE) that is reported applies to a candidate's polling percentage but &lt;em&gt;not&lt;/em&gt; to the difference of the percentages. My second attempt explicitly addresses the question posed by the OP a bit better.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Second Attempt&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The OP's friend reasons as follows:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Construct the confidence interval for Candidate A and Candidate B separately using the given MOE.&lt;/li&gt;&#xA;&lt;li&gt;If they overlap we have a statistical dead hear and if they do not then A is currently leading B.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The main issue here is that the first step is invalid. Constructing confidence intervals independently for the two candidates is not a valid step because the polling percentages for the two candidates are dependent random variables. In other words, a voter who decides not to vote for A may potentially decide to vote for B instead. Thus, the correct way to assess if the lead is significant or not is to construct a confidence interval for the difference. See the wiki as to how to compute the standard error for the &lt;a href=&quot;http://en.wikipedia.org/wiki/Margin_of_error#Comparing_percentages&quot; rel=&quot;nofollow&quot;&gt;difference of polling percentages&lt;/a&gt; under some assumptions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Flawed answer below&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my opinion the 'correct' way to think of the polling result is as follows:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In a survey of 500 voters the chances that we will see a difference in lead as high as 8% is greater than 5%.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Whether you believe that 'A leads B' or 'A ties B' is then dependent on the extent to which you are willing to accept 5% as your cut-off criteria.&lt;/p&gt;&#xA;" LastActivityDate="2010-08-29T02:31:19.680" CommentCount="2" />
  <row Id="2184" PostTypeId="2" ParentId="2182" CreationDate="2010-08-29T00:12:35.500" Score="3" ViewCount="" Body="&lt;p&gt;Not only is that a bad way to term things but that's not even a statistical dead heat.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The MOE is just one side of a confidence interval and you seem to be applying it correct in one aspect.  Candidate B's score is unlikely to be higher than 35.5%.  Since it's usually a 95% confidence interval then Candidate B's score will be higher than 35.5% less than 5% of the time.  Not impossible, just usually considered unlikely.  On the other hand, a score of say 34% also has a cutoff probability.  It's just not reported.  It's probably something lower than random chance, we just don't know what it is from the MOE alone.  Therefore, it's completely incorrect to say that scores with the MOE cutoff are all just randomly equally likely.  Which is the implication of the assertion of statistical dead heat.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Not only that, you don't use overlapping confidence intervals that way.  If you really wanted to only say that Candidate A was going to win if the probability of Candidate B winning was less than 5% then Candidate A is definitely in the lead.  The lead is 8%.  The confidence interval of that subtraction score is not double the confidence interval of the individual scores but only sqrt(2) times those scores.  That's because finding the difference between the values would only double the variance.  The confidence interval is based on a sqrt of the variance therefore combining them is the average (4.5) * sqrt(2).  Since the MOE of your 8% lead is approximately 6.4% then Candidate A is in the lead.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an aside, MOE's are very conservative and based on the 50% choice value. The formula is sqrt(0.25/n) * 2.  There is a formula for calculating standard errors of difference scores that we could use as well.  We would apply that using the found values rather than the 50% cutoff and that still gives us a significant lead for Candidate A (7.5% MOE).  I believe that, given the questioners comment, and the proximity of that cutoff to the hypothetical one selected, that that was probably what they were looking for.  I like my answer better though because, while not the most correct mathematically in this instance, it is the generally correct way to compare data where confidence intervals are reported for individual values and gives a framework for thinking about why the CI doesn't double.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any introduction to both confidence intervals and to power would be helpful here.  Even the wikipedia article on MOE looks pretty good.&lt;/p&gt;&#xA;" OwnerUserId="601" LastEditorUserId="601" LastEditorDisplayName="" LastEditDate="2010-08-29T00:32:54.227" LastActivityDate="2010-08-29T00:32:54.227" />
  <row Id="2185" PostTypeId="2" ParentId="665" CreationDate="2010-08-29T01:35:44.567" Score="4" ViewCount="" Body="&lt;p&gt;Similar to what Mark said, Statistics was historically called &lt;strong&gt;Inverse Probability&lt;/strong&gt;, since statistics tries to infer the causes of an event given the observations, while probability tends to be the other way around.&lt;/p&gt;&#xA;" OwnerUserId="1106" LastActivityDate="2010-08-29T01:35:44.567" />
  <row Id="2186" PostTypeId="2" ParentId="2181" CreationDate="2010-08-29T08:44:54.437" Score="5" ViewCount="" Body="&lt;p&gt;Off the top of my head, I would say that the following general purpose books are rather interesting as a first start:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Izenman, J. &lt;a href=&quot;http://www.springer.com/statistics/statistical+theory+and+methods/book/978-0-387-78188-4&quot; rel=&quot;nofollow&quot;&gt;Modern Multivariate Statistical Techniques: Regression, Classification, and Manifold Learning&lt;/a&gt;. Springer. &lt;a href=&quot;http://astro.temple.edu/~alan/MMST/&quot; rel=&quot;nofollow&quot;&gt;companion website&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Tinsley, H. and Brown, S. (2000). &lt;a href=&quot;http://www.sciencedirect.com/science/book/9780126913606&quot; rel=&quot;nofollow&quot;&gt;Handbook of Applied Multivariate Statistics and Mathematical Modeling&lt;/a&gt;. Academic Press.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;There is also many applied textbook, like&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Everitt, B.S. (2005). &lt;a href=&quot;http://www.springer.com/statistics/social+sciences+%26+law/book/978-1-85233-882-4&quot; rel=&quot;nofollow&quot;&gt;An R and S-Plus® Companion to Multivariate Analysis&lt;/a&gt;. Springer. &lt;a href=&quot;http://biostatistics.iop.kcl.ac.uk/publications/everitt/&quot; rel=&quot;nofollow&quot;&gt;companion website&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;It is difficult to suggest you specific books as there are many ones that are domain-specific (e.g. social sciences, machine learning, categorical data, biomedical data).&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-08-29T08:44:54.437" CommunityOwnedDate="2010-08-29T08:44:54.437" CommentCount="2" />
  <row Id="2187" PostTypeId="2" ParentId="2181" CreationDate="2010-08-29T08:58:43.240" Score="4" ViewCount="" Body="&lt;p&gt;Here are some of my books on that field (in alphabetical order). &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;AFIFI, A., CLARK, V. Computer-Aided&#xA;Multivariate Analysis. CHAPMAN &amp;amp; HALL, 2000&lt;/li&gt;&#xA;&lt;li&gt;AGRESTI, A. Categorical Data Analysis. WILEY, 2002&lt;/li&gt;&#xA;&lt;li&gt;HAIR, Multivariate Data Analysis. 6th Ed.&lt;/li&gt;&#xA;&lt;li&gt;ΗÄRDLE, W., SIMAR, L. Applied Multivariate Statistical Analysis. SPRINGER, 2007.&lt;/li&gt;&#xA;&lt;li&gt;HARLOW, L. The Essence of Multivariate Thinking. LAWRENCE ERLBAUM ASSOCIATES, INC., 2005&lt;/li&gt;&#xA;&lt;li&gt;GELMAN, A., HILL, J. Data Analysis&#xA;Using Regression and&#xA;Multilevel/Hierarchical Models.&#xA;CAMBRIDGE UNIVERSITY PRESS, 2007.&lt;/li&gt;&#xA;&lt;li&gt;IZENMAN, A. J. Modern Multivariate Statistical Techniques. SPRINGER, 2008 &lt;/li&gt;&#xA;&lt;li&gt;RENCHER, A. Methods of Multivariate analysis. SECOND ED., WILEY-INTERSCIENCE, 2007&lt;/li&gt;&#xA;&lt;li&gt;TABACHNICK B., FIDELL, L. Using Multivariate Statistics. 5th Ed. Pearson Education. Inc, 2007.&lt;/li&gt;&#xA;&lt;li&gt;TIMM, N. Applied Multivariate Analysis. SPRINGER, 2002&lt;/li&gt;&#xA;&lt;li&gt;YANG, K., TREWN, J. Multivariate Statistical Methods in Quality Management. MCGRAW-HILL, 2004&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="339" LastActivityDate="2010-08-29T08:58:43.240" CommunityOwnedDate="2010-08-29T08:58:43.240" CommentCount="4" />
  <row Id="2188" PostTypeId="2" ParentId="490" CreationDate="2010-08-29T12:04:51.210" Score="3" ViewCount="" Body="&lt;p&gt;I have a slight preference for &lt;a href=&quot;http://www.stat.berkeley.edu/~breiman/RandomForests/&quot; rel=&quot;nofollow&quot;&gt;Random Forests&lt;/a&gt; by Leo Breiman &amp;amp; Adele Cutleer for several reasons:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;it allows to cope with categorical and continuous predictors, as well as unbalanced class sample size;&lt;/li&gt;&#xA;&lt;li&gt;as an ensemble/embedded method, cross-validation is embedded and allows to estimate a generalization error;&lt;/li&gt;&#xA;&lt;li&gt;it is relatively insensible to its tuning parameters (% of variables selected for growing a tree, # of trees built);&lt;/li&gt;&#xA;&lt;li&gt;it provides an original measure of variable importance and is able to uncover complex interactions between variables (although this may lead to hard to read results).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Some authors argued that it performed as well as penalized SVM or Gradient Boosting Machines (see, e.g. Cutler et al., 2009, for the latter point).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A complete coverage of its applications or advantages may be off the topic, so I suggest the &lt;a href=&quot;http://www-stat.stanford.edu/~tibs/ElemStatLearn/&quot; rel=&quot;nofollow&quot;&gt;Elements of Statistical Learning&lt;/a&gt; from Hastie et al. (chap. 15) and Sayes et al. (2007) for further readings.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Last but not least, it has a nice implementation in R, with the &lt;a href=&quot;http://cran.r-project.org/web/packages/randomForest/index.html&quot; rel=&quot;nofollow&quot;&gt;randomForest&lt;/a&gt; package. Other R packages also extend or use it, e.g. &lt;a href=&quot;http://cran.r-project.org/web/packages/party/index.html&quot; rel=&quot;nofollow&quot;&gt;party&lt;/a&gt; and &lt;a href=&quot;http://cran.r-project.org/web/packages/caret/index.html&quot; rel=&quot;nofollow&quot;&gt;caret&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;References:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cutler, A., Cutler, D.R., and Stevens, J.R. (2009). Tree-Based Methods, in &lt;em&gt;High-Dimensional Data Analysis in Cancer Research&lt;/em&gt;, Li, X. and Xu, R. (eds.), pp. 83-101, Springer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Saeys, Y., Inza, I., and Larrañaga, P. (2007). A review of feature selection techniques in bioinformatics. &lt;em&gt;Bioinformatics&lt;/em&gt;, &lt;strong&gt;23(19)&lt;/strong&gt;: 2507-2517.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-08-29T12:04:51.210" CommunityOwnedDate="2010-08-29T12:04:51.210" />
  <row Id="2189" PostTypeId="2" ParentId="2140" CreationDate="2010-08-29T14:07:55.913" Score="1" ViewCount="" Body="&lt;p&gt;I'm not sure if such a (non-parametric) permutation procedure could be applied here. Anyways, here is my idea:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;a &amp;lt;- c(1.18, -0.41, -0.66, 0.98, 0.1)&#xA;b &amp;lt;- c(-0.36, -0.73, -1.47, 0.15, -0.31)&#xA;total &amp;lt;- c(a,b)&#xA;first &amp;lt;- combn(total,length(a))&#xA;second &amp;lt;- apply(first,2,function(z) total[is.na(pmatch(total,z))])&#xA;var.ratio &amp;lt;- apply(first,2,var) / apply(second,2,var)&#xA;# the first element of var.ratio is the one that I'm interested in&#xA;(p.value &amp;lt;- length(var.ratio[var.ratio &amp;gt;= var.ratio[1]]) / length(var.ratio))&#xA;[1] 0.3055556&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="339" LastEditorUserId="339" LastEditorDisplayName="" LastEditDate="2010-08-29T14:41:44.697" LastActivityDate="2010-08-29T14:41:44.697" />
  <row Id="2190" PostTypeId="1" CreationDate="2010-08-29T16:26:44.937" Score="1" ViewCount="98" Body="&lt;p&gt;I have a great prediction yet I am unsure how to uncover how the results were generated?&lt;/p&gt;&#xA;" LastEditorUserId="71" LastEditorDisplayName="" LastEditDate="2010-08-29T18:45:28.027" LastActivityDate="2010-09-28T19:53:38.997" Title="How to reconstruct ensemble of trees from random forest?" Tags="&lt;classification&gt;&lt;random-forest&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="1" />
  <row Id="2191" PostTypeId="2" ParentId="2181" CreationDate="2010-08-29T17:15:22.733" Score="3" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0534349749&quot; rel=&quot;nofollow&quot;&gt;Analyzing Multivariate Data&lt;/a&gt; by James Lattin, J Douglas Carroll and Paul E Green.&lt;/p&gt;&#xA;" OwnerUserId="174" LastActivityDate="2010-08-29T17:15:22.733" CommunityOwnedDate="2010-08-29T17:15:22.733" />
  <row Id="2192" PostTypeId="2" ParentId="224" CreationDate="2010-08-29T17:49:09.767" Score="0" ViewCount="" Body="&lt;p&gt;I've used &lt;a href=&quot;http://zedgraph.org/&quot; rel=&quot;nofollow&quot;&gt;ZedGraph&lt;/a&gt; for .NET. It's open source, and supports all common 2D chart types. &lt;/p&gt;&#xA;" OwnerUserId="956" LastActivityDate="2010-08-29T17:49:09.767" CommunityOwnedDate="2010-10-12T08:48:39.990" />
  <row Id="2193" PostTypeId="2" ParentId="2190" CreationDate="2010-08-29T19:47:13.183" Score="1" ViewCount="" Body="&lt;p&gt;From the trees attributed to each class's output you can do a tree search on the similarities. You could do it manually, but that would be as tedious as examining the weights on a Neural network. So you want to find the overlaps in the decision tree structures. This can look for various features depending upon the problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Eg. you can see if there is statistically more preference for certain nodes to be parents of other nodes giving a hierarchical structure. You just search all nodes and find if there is more than average having certain children. Do a distribution test to show its beyond a random chance to have consistently those children and that reveals structure of the problem you analyzed. &#xA;Best.&lt;/p&gt;&#xA;" OwnerUserId="1098" LastActivityDate="2010-08-29T19:47:13.183" CommentCount="3" />
  <row Id="2194" PostTypeId="2" ParentId="2171" CreationDate="2010-08-29T19:54:09.083" Score="0" ViewCount="" Body="&lt;p&gt;R is great, but it is not that &lt;strong&gt;R&lt;/strong&gt; is difficult to learn it's that the documentation is impossible to search for any other name like Rq would be great. So when you got a problem, searching for a solution is a nightmare, and the documentation is not great either. Matlab or Octave will be great. And to get those plots in R or Matlab would be very very tedious.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;IMHO post processing visuals is the best route. Alot of them from flowing data are put through Adobe Illustrator or Gimp. It is faster. Once you get the structure of the plot, then change details in an editor. Using R as an editor does not give you the flexibility you want. You will find yourself searching for new packages all the time.&lt;/p&gt;&#xA;" OwnerUserId="1098" LastActivityDate="2010-08-29T19:54:09.083" CommunityOwnedDate="2010-08-29T19:54:09.083" />
  <row Id="2195" PostTypeId="2" ParentId="21" CreationDate="2010-08-29T20:01:34.300" Score="2" ViewCount="" Body="&lt;p&gt;I don't know about the first point. But for the second one, autoregressive (AR) functions could be simple. I would really chose a parametric method against a non-parametric one. The forecasting in AR is straight forward. And consensus data has lots of samples for each period so you can get robust parameter estimates at each time. And for the association x_n-1 to x_n function is simply a smoothed interpolation of any choice. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Changes in zones, well based on empirical data or prior belief?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;State of the art in consenus? Those methods are arcane. They iterate over generations! Aeons. You could use Gaussian processes which would be quite advanced methodology for these problems. But most in the field stick to older methods given more 'tuning'.&#xA;Best.&lt;/p&gt;&#xA;" OwnerUserId="1098" LastActivityDate="2010-08-29T20:01:34.300" />
  <row Id="2196" PostTypeId="2" ParentId="369" CreationDate="2010-08-29T20:42:33.390" Score="3" ViewCount="" Body="&lt;p&gt;You can try &lt;em&gt;Latent Semantic Analysis&lt;/em&gt;, which basically provides a way to represent in a reduced space your news feeds and any term (in your case, keyword appearing in the title). As it relies on Singular Value Decomposition, I suppose you may then be able to check if there exists a particular association between those two attributes. I know this is used to find documents matching a specific set of criteria, as in information retrieval, or to construct a tree reflecting terms similarity (like a dictionary) based on a large corpus (which here plays the role of the concept space).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See for a gentle introduction &lt;a href=&quot;http://lsa.colorado.edu/papers/dp1.LSAintro.pdf&quot; rel=&quot;nofollow&quot;&gt;An Introduction to Latent Semantic Analysis&lt;/a&gt;, by Landauer et al.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Moreover, there is an R package that implements this technique, namely &lt;a href=&quot;http://cran.r-project.org/web/packages/lsa/index.html&quot; rel=&quot;nofollow&quot;&gt;lsa&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-08-29T20:42:33.390" />
  <row Id="2197" PostTypeId="1" CreationDate="2010-08-29T22:50:57.957" Score="7" ViewCount="209" Body="&lt;p&gt;Let's say I have a dataset with 1000 observations in 10 variables, &quot;A&quot; through &quot;J.&quot; I have 1000 responses/measures for each of the first 8 variables, through &quot;H,&quot; but only the first 500 observations for &quot;I&quot; are not missing, and only the last 500 observations for &quot;J&quot; are not missing -- there are no observations for which I have measures of both of the last two variables, I and J.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, if I calculate (pairwise) correlations, I have a full correlation matrix, with only the correlation between I and J missing. Let's say I want to run Principal Component Analysis, or some other such scaling procedure on this correlation matrix.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I think I would like to do is: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Randomly generate (perhaps from some distribution on [-1, 1], or perhaps via sampling from existing values in the rest of the correlation matrix) an &quot;invented&quot; correlation between I and J.&lt;/li&gt;&#xA;&lt;li&gt;Put that in the correlation matrix.&lt;/li&gt;&#xA;&lt;li&gt;Run PCA on the correlation matrix with this invented value.&lt;/li&gt;&#xA;&lt;li&gt;Repeat steps 1 - 3 some large number of times.&lt;/li&gt;&#xA;&lt;li&gt;Assess the collective results of this large number of PCAs, looking at the mean and variance of the loadings, scores, eigenvalues, etc., based on the &quot;pseudo-bootstrapped&quot; iterations.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Is there a better way to handle (a) missing value(s) in the correlation matrix?&lt;/li&gt;&#xA;&lt;li&gt;Is there any precedent for replacing such (a) missing value(s) with random invented values? If so, what is it called?&lt;/li&gt;&#xA;&lt;li&gt;Is this related to the bootstrap?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Thanks a lot, in advance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit:&#xA;Question 4. Is this a defensible approach to imputation?&lt;/p&gt;&#xA;" OwnerUserId="1117" LastEditorUserId="1117" LastEditorDisplayName="" LastEditDate="2010-08-29T23:18:38.910" LastActivityDate="2010-10-10T19:36:49.337" Title="Precedent for Bootstrap-like procedure with &quot;invented&quot; data?" Tags="&lt;correlation&gt;&lt;pca&gt;&lt;bootstrap&gt;" AnswerCount="3" FavoriteCount="1" />
  <row Id="2198" PostTypeId="1" AcceptedAnswerId="2202" CreationDate="2010-08-30T00:54:36.273" Score="3" ViewCount="280" Body="&lt;p&gt;If given probability of $A$ is $a$ and probability of $B$ is $b$, how do I find min/max probability of intersection? Max value of intersection would be $\min(a,b)$, how do I find the min? &lt;/p&gt;&#xA;" OwnerUserId="862" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-08-30T06:25:16.763" LastActivityDate="2010-08-30T06:25:16.763" Title="Find range of possible values for probability of intersection given individual probabilities" Tags="&lt;probability&gt;" AnswerCount="2" />
  <row Id="2199" PostTypeId="2" ParentId="2181" CreationDate="2010-08-30T01:22:20.390" Score="2" ViewCount="" Body="&lt;p&gt;Tabachnick is the most cited on Google Scholar&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hair (6th ed) has the most ratings (with a score above 4.5) on Amazon&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I recommend Hair, as I've read it, and it is written in plain language. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are a student or staff at a university, then I would see if your school has an account with SpringerLink, as the Hardle book is on there for free. &lt;/p&gt;&#xA;" OwnerUserId="74" LastActivityDate="2010-08-30T01:22:20.390" CommunityOwnedDate="2010-08-30T01:22:20.390" CommentCount="1" />
  <row Id="2200" PostTypeId="2" ParentId="2198" CreationDate="2010-08-30T02:51:15.950" Score="0" ViewCount="" Body="&lt;p&gt;The min is the smaller of two values: $\min(a,b) = a$ if $a &amp;lt; b$ and $b$ otherwise. Though I do not think this is what you are asking for...&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-08-30T02:51:15.950" CommentCount="6" />
  <row Id="2201" PostTypeId="2" ParentId="2197" CreationDate="2010-08-30T03:31:36.610" Score="5" ViewCount="" Body="&lt;p&gt;An alternative approach would be to impute the missing raw data using a missing data replacement procedure. You could then run the PCA on the correlation matrix that resulted from the imputed dataset (see also &lt;a href=&quot;http://www.stat.psu.edu/~jls/mifaq.html&quot; rel=&quot;nofollow&quot;&gt;multiple imputation&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are a few links on missing data imputation in R:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.stat.columbia.edu/~gelman/arm/missing.pdf&quot; rel=&quot;nofollow&quot;&gt;Gelman on missing data imputation&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.statmethods.net/input/missingdata.html&quot; rel=&quot;nofollow&quot;&gt;Quick-R&lt;/a&gt; has links to R packages such as &lt;code&gt;Amelia II&lt;/code&gt;, &lt;code&gt;Mice&lt;/code&gt; and &lt;code&gt;mitools&lt;/code&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-08-30T03:31:36.610" CommentCount="2" />
  <row Id="2202" PostTypeId="2" ParentId="2198" CreationDate="2010-08-30T03:59:15.940" Score="3" ViewCount="" Body="&lt;p&gt;if $a+b \le 1$, then presumably one can find disjoint sets $A$ and $B$ with ${\rm P}A = a$ and&#xA;${\rm P}B = b$. so in this case, the min is 0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;if $a+b &gt; 1$, we get a smallest intersection by choosing $B$ to contain all of $A^C$, which has probability $1-a$ and then adding to that a piece of $A$ to bring ${\rm P}B$ up to $b$. so the piece of $A$ added in has to have probability $b - (1-a) = a+b - 1$. this last quantity is then the min probability for the intersection when $a+b &gt; 1$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;so in any case. the min is $(a+b-1)^+$.&lt;/p&gt;&#xA;" OwnerUserId="1112" LastActivityDate="2010-08-30T03:59:15.940" CommentCount="1" />
  <row Id="2203" PostTypeId="2" ParentId="2181" CreationDate="2010-08-30T04:55:48.270" Score="4" ViewCount="" Body="&lt;p&gt;JOHNSON R., WICHERN D., &lt;a href=&quot;http://www.pearsonhighered.com/educator/academic/product/0,3110,0131877151,00.html&quot; rel=&quot;nofollow&quot;&gt;Applied Multivariate Statistical Analysis&lt;/a&gt;, is what we used in our undergraduate Multivariate class at UC Davis, and it does a pretty good job (though it's a bit pricey).&lt;/p&gt;&#xA;" OwnerUserId="1118" LastActivityDate="2010-08-30T04:55:48.270" CommunityOwnedDate="2010-08-30T04:55:48.270" />
  <row Id="2204" PostTypeId="2" ParentId="2072" CreationDate="2010-08-30T09:45:59.520" Score="4" ViewCount="" Body="&lt;p&gt;You could start with the following references:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://dx.doi.org/10.1016/S0304-4076%2898%2900025-6&quot; rel=&quot;nofollow&quot;&gt;Comte (1999)&lt;/a&gt; &quot;Discrete and continuous time cointegration&quot;, &lt;em&gt;Journal of Econometrics&lt;/em&gt;.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www-m4.ma.tum.de/Diplarb/DA-Ferstl.pdf&quot; rel=&quot;nofollow&quot;&gt;Ferstl (2009)&lt;/a&gt; &quot;Cointegration in discrete and continuous time&quot;. Thesis.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scholar.google.com.au/scholar?cites=2228464727485746115&quot; rel=&quot;nofollow&quot;&gt;Citations of Comte&lt;/a&gt; may also be useful.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-08-30T09:45:59.520" />
  <row Id="2205" PostTypeId="2" ParentId="2181" CreationDate="2010-08-30T11:48:13.960" Score="2" ViewCount="" Body="&lt;p&gt;Hastie, T., Tibshirani, R. and Friedman, J.: &quot;The Elements of Statistical Learning: Data Mining, Inference, and Prediction.&quot;, Springer (&lt;a href=&quot;http://www-stat.stanford.edu/~tibs/ElemStatLearn/&quot; rel=&quot;nofollow&quot;&gt;book home page&lt;/a&gt;)&lt;/p&gt;&#xA;" OwnerUserId="961" LastActivityDate="2010-08-30T11:48:13.960" CommunityOwnedDate="2010-08-30T11:48:13.960" />
  <row Id="2206" PostTypeId="2" ParentId="2197" CreationDate="2010-08-30T11:49:00.233" Score="5" ViewCount="" Body="&lt;ol&gt;&#xA;&lt;li&gt;I don't know.&lt;/li&gt;&#xA;&lt;li&gt;What you've shown is a legitimate Monte Carlo simulation &lt;/li&gt;&#xA;&lt;li&gt;Bootstrap is also a Monte Carlo method, but it is more about estimating distributions.&lt;/li&gt;&#xA;&lt;li&gt;In general yes, especially if imputation is giving poor results. In special cases when imputation works great, no. In simple words, it will be as good as strongly you are convinced that you cannot say more about I&amp;amp;J correlation that it is in -1..1.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="88" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-10T19:36:49.337" LastActivityDate="2010-10-10T19:36:49.337" CommentCount="1" />
  <row Id="2207" PostTypeId="2" ParentId="354" CreationDate="2010-08-30T13:44:51.723" Score="3" ViewCount="" Body="&lt;p&gt;In ordinary least squares, the solution to (A'A)^(-1) x = A'b minimizes squared error loss, and is the maximum likelihood solution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, largely because the math was easy in this historic case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But generally people minimize many different &lt;a href=&quot;http://en.wikipedia.org/wiki/Loss_function&quot; rel=&quot;nofollow&quot;&gt;loss functions&lt;/a&gt;, such as exponential, logistic, cauchy, laplace, huber, etc.  These more exotic loss functions generally require a lot of computational resources, and don't have closed form solutions (in general), so they're only starting to become more popular now.&lt;/p&gt;&#xA;" OwnerUserId="1119" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-28T11:58:46.793" LastActivityDate="2010-11-28T11:58:46.793" CommentCount="3" />
  <row Id="2208" PostTypeId="2" ParentId="409" CreationDate="2010-08-30T13:55:03.287" Score="0" ViewCount="" Body="&lt;p&gt;The reason the above works for uncertainty of the mean is because of the central limit theorem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As long as the central limit theorem holds for your application, so will the above.&lt;/p&gt;&#xA;" OwnerUserId="1119" LastActivityDate="2010-08-30T13:55:03.287" />
  <row Id="2209" PostTypeId="2" ParentId="125" CreationDate="2010-08-30T14:00:17.647" Score="2" ViewCount="" Body="&lt;p&gt;Its focus isn't strictly on Bayesian statistics, so it lacks some methodology, but David MacKay's Information Theory, Inference, and Learning Algorithms made me intuitively grasp Bayesian statistics better than others - most do the how quite nicely, but I felt MacKay explained why better.&lt;/p&gt;&#xA;" OwnerUserId="1119" LastEditorUserId="1119" LastEditorDisplayName="" LastEditDate="2010-09-09T06:16:29.723" LastActivityDate="2010-09-09T06:16:29.723" CommunityOwnedDate="2010-08-30T14:00:17.647" CommentCount="2" />
  <row Id="2210" PostTypeId="2" ParentId="1164" CreationDate="2010-08-30T14:11:06.037" Score="3" ViewCount="" Body="&lt;p&gt;While they're not mutually exclusive, I think the growing popularity of Bayesian statistics is part of it.  Bayesian statistics can achieve a lot of the same goals through priors and model averaging, and tend to be a bit more robust in practice.&lt;/p&gt;&#xA;" OwnerUserId="1119" LastActivityDate="2010-08-30T14:11:06.037" />
  <row Id="2212" PostTypeId="2" ParentId="224" CreationDate="2010-08-30T14:19:24.787" Score="3" ViewCount="" Body="&lt;p&gt;For javascript protovis (http://vis.stanford.edu/protovis/) is very nice.&lt;/p&gt;&#xA;" OwnerUserId="1119" LastActivityDate="2010-08-30T14:19:24.787" CommunityOwnedDate="2010-08-30T14:19:24.787" />
  <row Id="2213" PostTypeId="1" AcceptedAnswerId="2218" CreationDate="2010-08-30T15:33:28.180" Score="7" ViewCount="340" Body="&lt;p&gt;What is the difference between a &lt;a href=&quot;http://en.wikipedia.org/wiki/Feedforward_neural_network&quot; rel=&quot;nofollow&quot;&gt;feed-forward&lt;/a&gt; and &lt;a href=&quot;http://en.wikipedia.org/wiki/Recurrent_neural_networks&quot; rel=&quot;nofollow&quot;&gt;recurrent&lt;/a&gt; neural network?  Why would you use one over the other?  Do other network topologies exist?  &lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2011-02-27T16:29:58.647" Title="Feed-forward and recurrent neural networks?" Tags="&lt;machine-learning&gt;&lt;neural-networks&gt;&lt;topologies&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="2" />
  <row Id="2214" PostTypeId="1" CreationDate="2010-08-30T16:36:55.427" Score="5" ViewCount="126" Body="&lt;p&gt;One of the purported uses of L-estimators is the ability to 'robustly' estimate the parameters of a random variable drawn from a given class. One of the downsides of using &lt;a href=&quot;http://en.wikipedia.org/wiki/Stable_distribution&quot; rel=&quot;nofollow&quot;&gt;Levy $\alpha$-stable distributions&lt;/a&gt; is that it is difficult to estimate the parameters given a sample of observations drawn from the class. Has there been any work in estimating parameters of a Levy RV using L-estimators? There is an obvious difficulty in the fact that the PDF and CDF of the Levy distribution do not have a closed form, but perhaps this could be overcome by some trickery. Any hints?&lt;/p&gt;&#xA;" OwnerUserId="795" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-10-19T06:25:46.190" LastActivityDate="2010-10-19T09:24:21.443" Title="Estimating parameters of sum-stable RV via L-estimators" Tags="&lt;distributions&gt;&lt;estimation&gt;&lt;robust&gt;&lt;parametric&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="2215" PostTypeId="2" ParentId="2197" CreationDate="2010-08-30T16:54:55.467" Score="5" ViewCount="" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;I think we need to know more about the nature of the data to make recommendations on how to deal with the missing values. An exploratory task that jumps out to me is to look at the behavior of variables A through H when I is present, versus A through H when J is present. Is there anything interesting to take into account for subsequent modeling? Instead of resampling a descriptive statistic, like correlation, I would consider resampling the data itself. For example, you could use the bootstrap to create 500 new (I,J) pairs based upon the 500 values you actually have for these variables. But, again, the exploratory work may inform a resampling scheme beyond a &quot;naive&quot;, IID approach.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;In general, as others have noted, filling in missing data goes by &quot;imputation&quot; and there are different techniques depending on the context. For example, in one setting I might simply use a median value, or a spline fit, but for a missing data point in a time series I might impute with a value generated from an ARMA time series model.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Your outlined solution would be &quot;bootstrapping&quot; if you resample from the observed data. I think of &lt;em&gt;Monte Carlo&lt;/em&gt; as any method that uses probabilistic sampling of data as input into a computation. When the sampling is from a non-parametric or parametric distribution that you use to model how the data was generated, I still call it Monte Carlo. But, when the sampling is done from an empirical distribution (i.e., the observed data itself, not a model of the data generating process) I call it bootstrapping.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="1080" LastActivityDate="2010-08-30T16:54:55.467" CommentCount="1" />
  <row Id="2216" PostTypeId="2" ParentId="2072" CreationDate="2010-08-30T17:42:31.317" Score="2" ViewCount="" Body="&lt;p&gt;Although it may only be of little help, the problem you present to me is synonymous with the &quot;&lt;a href=&quot;http://dx.doi.org/10.1198/016214502760047140&quot; rel=&quot;nofollow&quot;&gt;Change of Support&lt;/a&gt;&quot; problem encountered when using areal units. Although this work just presents a framework for what you describe as &quot;reglarize and interpolate&quot; using a method referred to as &quot;kriging&quot;. I don't think any of this work will help answer your question of whether estimating your missing values in the series in such a manner will bias error correction estimates, although if some of your samples are in clustered time intervals for both series you may be able to check for yourself. You may also be interested in the technique of &quot;co-kriging&quot; from this field, which uses information from one source to estimate the value for another (if your interested I would suggest you check out the work being done by &lt;a href=&quot;http://sites.google.com/site/goovaertspierre/pierregoovaertswebsite&quot; rel=&quot;nofollow&quot;&gt;Pierre Goovaerts&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Again I'm not sure how helpful this will be though. It may be substantially simpler to just use current time-series forecasting techniques to estimate your missing data. It won't help you decide what to estimate either.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good luck, and keep the thread updated if you find any pertinent material. I would be interested, and you would think with the proliferation of data sources online this would become an pertinent issue for at least some research projects.&lt;/p&gt;&#xA;" OwnerUserId="1036" LastActivityDate="2010-08-30T17:42:31.317" />
  <row Id="2217" PostTypeId="2" ParentId="423" CreationDate="2010-08-30T18:02:22.737" Score="24" ViewCount="" Body="&lt;p&gt;Another from XKCD:&#xA;&lt;img src=&quot;http://i.imgur.com/3ngU8.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Mentioned &lt;a href=&quot;http://www.stat.columbia.edu/~cook/movabletype/archives/2009/02/cartoon.html&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;http://www.cerebralmastication.com/2009/02/box-plot-vs-violin-plot-in-r/&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-08-30T18:02:22.737" CommunityOwnedDate="2010-08-30T18:02:22.737" />
  <row Id="2218" PostTypeId="2" ParentId="2213" CreationDate="2010-08-30T18:23:24.283" Score="9" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.multil.jpg&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;Feed-forward&lt;/strong&gt;&lt;/a&gt; ANNs allow signals to travel one way only: from input to output. There are no feedback (loops); &lt;em&gt;i.e.&lt;/em&gt;, the output of any layer does not affect that same layer. Feed-forward ANNs tend to be straightforward networks that associate inputs with outputs. They are extensively used in pattern recognition. This type of organisation is also referred to as bottom-up or top-down.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://www.cs.iusb.edu/~danav/teach/c463/forward_nn.gif&quot; alt=&quot;A feed-forward network&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.neural2.jpg&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;Feedback&lt;/strong&gt;&lt;/a&gt; (or recurrent or interactive) networks can have signals traveling in both directions by introducing loops in the network. Feedback networks are powerful and can get extremely complicated. Computations derived from earlier input are fed back into the network, which gives them a kind of memory. Feedback networks are dynamic; their 'state' is changing continuously until they reach an equilibrium point. They remain at the equilibrium point until the input changes and a new equilibrium needs to be found.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://www.cs.iusb.edu/~danav/teach/c463/rec_nn.gif&quot; alt=&quot;feedback architecture&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Feedforward neural networks are ideally suitable for modeling relationships between a set of predictor or input variables and one or more response or output variables. In other words, they are appropriate for any functional mapping problem where we want to know how a number of input variables affect the output variable. The multilayer feedforward neural networks, also called &lt;a href=&quot;http://en.wikipedia.org/wiki/Multilayer_perceptron&quot; rel=&quot;nofollow&quot;&gt;multi-layer perceptrons&lt;/a&gt; (MLP), are the most widely studied and used neural network model in practice.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an example of feedback network, I can recall &lt;a href=&quot;http://en.wikipedia.org/wiki/Hopfield_net&quot; rel=&quot;nofollow&quot;&gt;Hopfield’s network&lt;/a&gt;. The main use of Hopfield’s network is as associative memory. An associative memory is a device which accepts an input pattern and generates an output as the stored pattern which is most closely associated with the input. The function of the associate memory is to recall the corresponding stored pattern, and then produce a clear version of the pattern at the output. Hopfield networks are typically used for those problems with binary pattern vectors and the input pattern may be a noisy version of one of the stored patterns. In the Hopfield network, the stored patterns are encoded as the weights of the network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Kohonen’s self-organizing maps&lt;/strong&gt; (SOM) represent another neural network type that is markedly different from the feedforward multilayer networks. Unlike training in the feedforward MLP, the SOM training or learning is often called unsupervised because there are no known target outputs associated with each input pattern in SOM and during the training process, the SOM processes the input patterns and learns to cluster or segment the data through adjustment of weights (that makes it an important neural network model for dimension reduction and data clustering). A two-dimensional map is typically created in such a way that the orders of the interrelationships among inputs are preserved. The number and composition of clusters can be visually determined based on the output distribution generated by the training process. With only input variables in the training sample, SOM aims to learn or discover the underlying structure of the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(The diagrams are from Dana Vrajitoru's &lt;a href=&quot;http://www.cs.iusb.edu/~danav/teach/c463/12_nn.html&quot; rel=&quot;nofollow&quot;&gt;C463 / B551 Artificial Intelligence web site&lt;/a&gt;.)&lt;/p&gt;&#xA;" OwnerUserId="339" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-02-27T16:08:43.827" LastActivityDate="2011-02-27T16:08:43.827" />
  <row Id="2219" PostTypeId="1" AcceptedAnswerId="2221" CreationDate="2010-08-30T18:42:30.753" Score="3" ViewCount="420" Body="&lt;p&gt;&lt;strong&gt;General question:&lt;/strong&gt; Given a dartboard of unit radius, what's the probability that a dart randomly lands within a circle of radius 1/3 centered inside the dartboard?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Standard answer:&lt;/strong&gt; The dart is thrown such that it hits each point with equal likelihood. The probability that it lands within the inner circle is the ratio of the areas of the two circles, which is 1/9.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Other formulation:&lt;/strong&gt; Suppose that the dart is thrown directly at the center of the dartboard, but wind comes from a random direction. For any wind direction, the velocity of the wind pushes the dart some distance from the center of the dartboard to its perimeter, with each distance being equally likely. Each wind vector is equally likely and, for each vector, each distance is equally likely. The probability of being less than 1/3 units from the center of the dartboard is 1/3.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The definition of randomness is different for each formulation. In the standard answer, a random vector is chosen from the set ${(x,y)\colon \; x^2+y^2 \leq 1}$ and we ask the probability that $x^2 + y^2 \leq \frac{1}{9}$. In the other formulation, a random vector is chosen ${(x,y)\colon \; x^2+y^2 = 1}$ and, on this vector, a distance $d$ is chosen uniformly on $[0,1]$. We ask the probability that $d\leq\frac{1}{3}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand the math of solving this problem, but I don't understand intuitively why these different conceptions of randomness give two different answers. It seems that both are valid means for answering the general question. Intuitively, why do they give different answers?&lt;/p&gt;&#xA;" OwnerUserId="401" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-08-31T13:43:39.357" LastActivityDate="2010-08-31T13:43:39.357" Title="Two answers to the dartboard problem" Tags="&lt;probability&gt;&lt;games&gt;" AnswerCount="3" />
  <row Id="2220" PostTypeId="1" AcceptedAnswerId="2222" CreationDate="2010-08-30T18:47:48.040" Score="6" ViewCount="601" Body="&lt;p&gt;Permutation tests are significance tests based on permutation resamples drawn at random from the original data. Permutation resamples are drawn without replacement, in contrast to bootstrap samples, which are drawn with replacement. Here is &lt;a href=&quot;http://stackoverflow.com/questions/2449226/randomized-experiments-in-r&quot;&gt;an example I did in R&lt;/a&gt; of a simple permutation test. (Your comments are welcome)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Permutation tests have great advantages. They do not require specific population shapes such as normality. They apply to a variety of statistics, not just to statistics that have a simple distribution under the null hypothesis. They can give very accurate p-values, regardless of the shape and size of the population (if enough permutations are used). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have also read that it is often useful to give a confidence interval along with a test, which is created using bootstrap resampling rather than permutation resampling.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could you explain (or just give the R code) how a confidence interval is constructed (i.e. for the difference between the means of the two samples in the above example) ? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After some googling I found &lt;a href=&quot;http://bcs.whfreeman.com/ips5e/content/cat_080/pdf/moore14.pdf&quot; rel=&quot;nofollow&quot;&gt;this interesting reading&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="339" LastEditorUserId="339" LastEditorDisplayName="" LastEditDate="2010-08-31T09:46:54.943" LastActivityDate="2010-09-01T08:54:35.887" Title="How do we create a confidence interval for the parameter of a permutation test?" Tags="&lt;confidence-interval&gt;&lt;bootstrap&gt;&lt;permutation&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="2221" PostTypeId="2" ParentId="2219" CreationDate="2010-08-30T18:53:38.447" Score="2" ViewCount="" Body="&lt;p&gt;Intuitively, imagine modeling the second formulation as follows: randomly select an angle to the $x$-axis, calling it $\theta$, then model the location of the dart as falling uniformly in a very thin rectangle along the line $y = (\tan\theta) x$.  Approximately, the dart is in the inner circle with probability $1/3$. However, when you consider the collection of all such thin rectangles (draw them, say), you will see that they have more overlapping area near the center of the dartboard, and less overlap towards the perimeter of the dartboard. This will be more obvious as you draw the rectangles larger and larger (though the approximation will be worse). As you make the rectangles thinner, the approximation gets better, but the same principle applies: you are putting more area around the center of the circle, which increases the probability of hitting the inner circle. &lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-08-30T18:53:38.447" CommentCount="3" />
  <row Id="2222" PostTypeId="2" ParentId="2220" CreationDate="2010-08-30T19:49:10.880" Score="6" ViewCount="" Body="&lt;p&gt;It's OK to use permutation resampling.  It really depends on a number of factors.  If your permutations are a relatively low number then your estimation of your confidence interval is not so great with permutations.  Your permutations are in somewhat of a gray area and probably are fine.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The only difference from your prior code is that you'd generate your samples randomly instead of with permutations.  And, you'd generate more of them, let's say 1000 for example.  Get the difference scores for your 1000 replications of your experiment.  Take the cutoffs for the middle 950 (95%).  That's your confidence interval.  It falls directly from the bootstrap.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You've already done most of this in your example.  dif.treat is 462 items long.  Therefore, you need the lower 2.5% and upper 2.5% cut offs (about 11 items in on each end).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using your code from before... &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;y &amp;lt;- sort(dif.treat)&#xA;ci.lo &amp;lt;- y[11]&#xA;ci.hi &amp;lt;- y[462-11]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Off hand I'd say that 462 is a little low but you'll find a bootstrap to 10,000 comes out with scores that are little different (likely closer to the mean).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thought I'd also add in some simple code requiring the boot library (based on your prior code).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;diff &amp;lt;- function(x,i) mean(x[i[6:11]]) - mean(x[i[1:5]])&#xA;b &amp;lt;- boot(total, diff, R = 1000)&#xA;boot.ci(b)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="601" LastEditorUserId="601" LastEditorDisplayName="" LastEditDate="2010-08-30T20:24:38.670" LastActivityDate="2010-08-30T20:24:38.670" CommentCount="6" />
  <row Id="2223" PostTypeId="1" CreationDate="2010-08-30T19:54:28.290" Score="1" ViewCount="88" Body="&lt;p&gt;I would like to know if anyone could recommend a book that deals more with the practical issues around conducting a meta-analysis?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanking you in advance&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Andrew Vitiello &lt;/p&gt;&#xA;" OwnerUserId="431" LastActivityDate="2010-08-30T20:33:07.587" CommunityOwnedDate="2010-08-31T13:03:46.473" Title="Books covering how to conduct a meta-anlysis" Tags="&lt;meta-analysis&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="2224" PostTypeId="2" ParentId="2223" CreationDate="2010-08-30T20:33:07.587" Score="2" ViewCount="" Body="&lt;p&gt;I asked this question last week and obtained two excellent answers.  The question is readily accessible through links on your &quot;meta-analysis&quot; tag.  Here's the URL:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://stats.stackexchange.com/questions/1963/looking-for-good-introductory-treatment-of-meta-analysis&quot;&gt;http://stats.stackexchange.com/questions/1963/looking-for-good-introductory-treatment-of-meta-analysis&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-08-30T20:33:07.587" CommunityOwnedDate="2010-08-31T13:03:46.473" CommentCount="4" />
  <row Id="2225" PostTypeId="2" ParentId="2219" CreationDate="2010-08-30T21:07:57.217" Score="1" ViewCount="" Body="&lt;p&gt;Think of the board as a filter -- it just converts the positions on board into an id of a field that dart hit. So that the output will be only a deterministically converted input -- and thus it is obvious that different realization of throwing darts will result in distribution of results.&lt;br&gt;&#xA;The paradox itself is purely linguistic -- &quot;random throwing&quot; seems ok, while the true is that it misses crucial information about how the throwing is realized.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-08-30T21:07:57.217" />
  <row Id="2226" PostTypeId="2" ParentId="2219" CreationDate="2010-08-30T22:37:36.583" Score="3" ViewCount="" Body="&lt;p&gt;It seems to me that the fundamental issue is that the two scenarios assume different data generating process for the position of a dart which results in different probabilities. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first situation's data generating process looks like so: (a) Pick a $x \in U[-1,1]$ and (b) Pick a $y$ uniformly subject to the constraint that $x^2+y^2 \le 1$. Then the required probability is $P(x^2 + y^2  \le \frac{1}{9})= \frac{1}{9}$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second situation's data generating process is as described in the question: (a) Pick an angle $\theta \in [0,2\pi]$ and (b) Pick a point on the diameter that is at an angle $\theta$ to the x-axis. Under this data generating process the required probability is $\frac{1}{3}$ as mentioned in the question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As articulated by mbq, the issue is that the phrase 'randomly lands on the dartboard' is not precise enough as it leaves the meaning of 'random' ambiguous. This is similar to asking what is the probability of coin landing heads on a random toss. The answer can be 0.5 if we assume that the coin is a fair coin but it can be anything else (say, 0.8) if the coin is biased towards heads.&lt;/p&gt;&#xA;" LastActivityDate="2010-08-30T22:37:36.583" />
  <row Id="2227" PostTypeId="2" ParentId="423" CreationDate="2010-08-30T23:04:11.447" Score="18" ViewCount="" Body="&lt;p&gt;There is this one on Bayesian learning:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/64Pe4.jpg&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="881" LastActivityDate="2010-08-30T23:04:11.447" CommunityOwnedDate="2010-08-30T23:04:11.447" CommentCount="2" />
  <row Id="2228" PostTypeId="2" ParentId="2104" CreationDate="2010-08-31T00:03:12.787" Score="2" ViewCount="" Body="&lt;p&gt;ez has now been updated to version 2.0. Among other improvements, the bug that caused it to fail to work for this example has been fixed.&lt;/p&gt;&#xA;" OwnerUserId="364" LastActivityDate="2010-08-31T00:03:12.787" CommentCount="10" />
  <row Id="2229" PostTypeId="2" ParentId="1531" CreationDate="2010-08-31T00:32:34.883" Score="3" ViewCount="" Body="&lt;p&gt;There is no single exact confidence interval for the ratio of two proportions. Generally speaking, an exact 95% confidence interval is any interval-generating procedure that guarantees at least 95% coverage of the true ratio, irrespective of the values of the underlying proportions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An interval formed by the Fisher Exact Test is probably overly conservative -- in that it has MORE than 95% coverage for most values of the parameters. It's not wrong but it's also wider than it has to be.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The interval used by the StatXact software with the default settings would be a better choice here -- I believe it uses some variety of Chan interval (i.e. an extremum-searching interval using the Berger-Boos procedure and a standardized statistic), but would need to check the manual to be sure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you ask for the &quot;how and why&quot; -- does this answer your question? I think we could certainly expound further about the definition of confidence intervals and how to construct one from scratch if that's what you were looking for. Or does it do the trick just to say that this is a Fisher Exact Test-based interval, one (but not the only and not the most powerful) of the confidence intervals that guarantees its coverage unconditionally?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Footnote: Some authors reserve the word &quot;exact&quot; to apply only to intervals and tests where false-positives are controlled at exactly alpha, instead of merely bounded by alpha. Taken in this sense, there simply isn't a deterministic exact confidence interval for the ratio of two proportions, period. All of the deterministic intervals are necessarily approximate. Of course, even so some intervals and tests do unconditionally control Type I error and some don't.)&lt;/p&gt;&#xA;" OwnerUserId="1122" LastEditorUserId="1122" LastEditorDisplayName="" LastEditDate="2010-08-31T02:24:01.933" LastActivityDate="2010-08-31T02:24:01.933" />
  <row Id="2230" PostTypeId="1" AcceptedAnswerId="2232" CreationDate="2010-08-31T03:57:21.193" Score="7" ViewCount="1123" Body="&lt;p&gt;I've never really grokked the difference between these two measures of convergence. (Or, in fact, any of the different types of convergence, but I mention these two in particular because of the Weak and Strong Laws of Large Numbers.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sure, I can quote the definition of each and give an example where they differ, but I still don't quite get it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What's a good way to understand the difference? Why is the difference important? Is there a particularly memorable example where they differ?&lt;/p&gt;&#xA;" OwnerUserId="1106" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-08-31T08:21:26.447" LastActivityDate="2011-05-20T12:06:18.283" Title="Convergence in probability vs. almost sure convergence" Tags="&lt;probability&gt;&lt;random-variable&gt;" AnswerCount="4" FavoriteCount="3" />
  <row Id="2231" PostTypeId="2" ParentId="2230" CreationDate="2010-08-31T04:39:45.463" Score="3" ViewCount="" Body="&lt;p&gt;I understand it as follows,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Convergence in probability&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The probability that the sequence of random variables equals the target value is asymptotically decreasing and approaches 0 but never actually attains 0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Almost Sure Convergence&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The sequence of random variables will equal the target value asymptotically but you cannot predict at what point it will happen.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Almost sure convergence is a stronger condition on the behavior of a sequence of random variables because it states that &quot;something will definitely happen&quot; (we just don't know when). In contrast, convergence in probability states that &quot;while something is likely to happen&quot; the likelihood of &quot;something &lt;em&gt;not&lt;/em&gt; happening&quot; decreases asymptotically but never actually reaches 0. (something $\equiv$ a sequence of random variables converging to a particular value). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;a href=&quot;http://en.wikipedia.org/wiki/Convergence_of_random_variables&quot; rel=&quot;nofollow&quot;&gt;wiki&lt;/a&gt; has some examples of both which should help clarify the above (in particular see the example of the archer in the context of convergence in prob and the example of the charity in the context of almost sure convergence).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From a practical standpoint, convergence in probability is enough as we do not particularly care about very unlikely events. As an example, consistency of an estimator is essentially convergence in probability. Thus, when using a consistent estimate, we implicitly acknowledge the fact that in large samples there is a very small probability that our estimate is far from the true value. We live with this 'defect' of convergence in probability as we know that asymptotically the probability of the estimator being far from the truth is vanishingly small.&lt;/p&gt;&#xA;" LastActivityDate="2010-08-31T04:39:45.463" />
  <row Id="2232" PostTypeId="2" ParentId="2230" CreationDate="2010-08-31T06:53:32.093" Score="11" ViewCount="" Body="&lt;p&gt;From my point of view the difference is important, but largely for philosophical reasons.  Assume you have some device, that improves with time.  So, every time you use the device the probability of it failing is less than before.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Convergence in probability says that the chance of failure goes to zero as the number of usages goes to infinity.  So, after using the device a large number of times, you can be very confident of it working correctly, it still might fail, it's just very unlikely.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Convergence almost surely is a bit stronger.  It says that the total number of failures is &lt;em&gt;finite&lt;/em&gt;. That is, if you count the number of failures as the number of usages goes to infinity, you will get a finite number.  The impact of this is as follows:  As you use the device more and more, you will, after some finite number of usages, exhaust all failures.  From then on the device will work &lt;em&gt;perfectly&lt;/em&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As Srikant points out, you don't actually know when you have exhausted all failures, so from a purely practical point of view, there is not much difference between the two modes of convergence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, personally I am very glad that, for example, the strong law of large numbers exists, as opposed to just the weak law.  Because now, a scientific experiment to obtain, say, the speed of light, is justified in taking averages.  At least in theory, after obtaining enough data, you can get arbitrarily close to the true speed of light.  There wont be any failures (however improbable) in the averaging process.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let me clarify what I mean by ''failures (however improbable) in the averaging process''.  Choose some $\delta &gt; 0$ arbitrarily small. You obtain $n$ estimates $X_1,X_2,\dots,X_n$ of the speed of light (or some other quantity) that has some `true' value, say $\mu$. You compute the average &#xA;$$S_n = \frac{1}{n}\sum_{k=1}^n X_k.$$&#xA;As we obtain more data ($n$ increases) we can compute $S_n$ for each $n = 1,2,\dots$.  The weak law says (under some assumptions about the $X_n$) that the probability&#xA;$$P(|S_n - \mu| &gt; \delta) \rightarrow 0$$&#xA;as $n$ goes to $\infty$. The strong law says that the number of times that $|S_n - \mu|$ is larger than $\delta$ is finite (with probability 1).  That is, if we define the indicator function $I(|S_n - \mu| &gt; \delta)$ that returns one when $|S_n - \mu| &gt; \delta$ and zero otherwise, then &#xA;$$\sum_{n=1}^{\infty}I(|S_n - \mu| &gt; \delta)$$&#xA;converges. This gives you considerable confidence in the value of $S_n$, because it guarantees (i.e. with probability 1) the existence of some finite $n_0$ such that $|S_n - \mu| &amp;lt; \delta$ for all $n &gt; n_0$ (i.e. the average never &lt;em&gt;fails&lt;/em&gt; for $n &gt; n_0$).  Note that the weak law gives no such guarantee.&lt;/p&gt;&#xA;" OwnerUserId="352" LastEditorUserId="352" LastEditorDisplayName="" LastEditDate="2010-10-02T04:41:28.330" LastActivityDate="2010-10-02T04:41:28.330" CommentCount="1" />
  <row Id="2233" PostTypeId="2" ParentId="1531" CreationDate="2010-08-31T07:47:14.557" Score="5" ViewCount="" Body="&lt;p&gt;Check out the R &lt;a href=&quot;http://cran.r-project.org/web/packages/Epi/index.html&quot; rel=&quot;nofollow&quot;&gt;Epi&lt;/a&gt; and &lt;a href=&quot;http://cran.r-project.org/web/packages/epitools/index.html&quot; rel=&quot;nofollow&quot;&gt;epitools&lt;/a&gt; packages, which include many functions for computing exact and approximate CIs/p-values for various measures of association found in epidemiological studies, including relative risk (RR). I know there is also &lt;a href=&quot;http://cran.r-project.org/web/packages/PropCIs&quot; rel=&quot;nofollow&quot;&gt;PropCIs&lt;/a&gt;, but I never tried it. Bootstraping is also an option, but generally these are exact or approximated CIs that are provided in epidemiological papers, although most of the explanatory studies rely on GLM, and thus make use of odds-ratio (OR) instead of RR (although, wrongly it is often the RR that is interpreted because it is easier to understand, but this is another story).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also check your results with online calculator, like on &lt;a href=&quot;http://statpages.org/ctab2x2.html&quot; rel=&quot;nofollow&quot;&gt;statpages.org&lt;/a&gt;, or &lt;a href=&quot;http://www.phsim.man.ac.uk/risk/&quot; rel=&quot;nofollow&quot;&gt;Relative Risk and Risk Difference Confidence Intervals&lt;/a&gt;. The latter explains how computations are done.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By &quot;exact&quot; tests, we generally mean tests/CIs not relying on an asymptotic distribution, like the chi-square or standard normal; e.g. in the case of an RR, an 95% CI may be approximated as&#xA;$\exp\left[ \log(\text{rr}) - 1.96\sqrt{\text{Var}\big(\log(\text{rr})\big)} \right], \exp\left[ \log(\text{rr}) + 1.96\sqrt{\text{Var}\big(\log(\text{rr})\big)} \right]$, &#xA;where $\text{Var}\big(\log(\text{rr})\big)=1/a - 1/(a+b) + 1/c - 1/(c+d)$ (assuming a 2-way cross-classification table, with $a$, $b$, $c$, and $d$ denoting cell frequencies). The explanations given by @Keith are, however, very insightful. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For more details on the calculation of CIs in epidemiology, I would suggest to look at Rothman and Greenland's textbook, &lt;a href=&quot;http://www.lww.com/product/?978-0-7817-5564-1&quot; rel=&quot;nofollow&quot;&gt;Modern Epidemiology&lt;/a&gt; (now in it's 3rd edition), &lt;a href=&quot;http://www.wiley.com/remtitle.cgi?0471526290&quot; rel=&quot;nofollow&quot;&gt;Statistical Methods for Rates and Proportions&lt;/a&gt;, from Fleiss et al., or &lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1637917/&quot; rel=&quot;nofollow&quot;&gt;Statistical analyses of the relative risk&lt;/a&gt;, from J.J. Gart (1979). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You will generally get similar results with &lt;code&gt;fisher.test()&lt;/code&gt;, as pointed by @gd047, although in this case this function will provide you with a 95% CI for the odds-ratio (which in the case of a disease with low prevalence will be very close to the RR).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Notes:&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;I didn't check your Excel file, for the reason advocated by @csgillespie.&lt;/li&gt;&#xA;&lt;li&gt;Michael E Dewey provides an interesting summary of &lt;a href=&quot;http://www.zen103156.zen.co.uk/rr.pdf&quot; rel=&quot;nofollow&quot;&gt;confidence intervals for risk ratios&lt;/a&gt;, from a digest of posts on the R mailing-list.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-08-31T11:14:57.893" LastActivityDate="2010-08-31T11:14:57.893" />
  <row Id="2234" PostTypeId="1" CreationDate="2010-08-31T10:02:07.947" Score="11" ViewCount="793" Body="&lt;p&gt;I would like as many algorithms that perform the same task as logistic regression.  That is  algorithms/models that can give a prediction to a binary response (Y) with some explanatory variable (X).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would be glad if after you name the algorithm, if you would also show how to implement it in R.  Here is a code that can be updated with other models:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(55)&#xA;n &amp;lt;- 100&#xA;x &amp;lt;- c(rnorm(n), 1+rnorm(n))&#xA;y &amp;lt;- c(rep(0,n), rep(1,n))&#xA;r &amp;lt;- glm(y~x, family=binomial)&#xA;plot(y~x)&#xA;abline(lm(y~x),col='red',lty=2)&#xA;xx &amp;lt;- seq(min(x), max(x), length=100)&#xA;yy &amp;lt;- predict(r, data.frame(x=xx), type='response')&#xA;lines(xx,yy, col='blue', lwd=5, lty=2)&#xA;title(main='Logistic regression with the &quot;glm&quot; function')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="253" LastEditorUserId="5" LastEditorDisplayName="" LastEditDate="2010-08-31T13:06:01.117" LastActivityDate="2010-12-31T14:56:21.053" CommunityOwnedDate="2010-08-31T10:02:07.947" Title="Alternatives to logistic regression ? (in R)" Tags="&lt;r&gt;&lt;regression&gt;&lt;logistic-regression&gt;&lt;classification&gt;&lt;predictive-models&gt;" AnswerCount="4" CommentCount="4" FavoriteCount="10" />
  <row Id="2235" PostTypeId="2" ParentId="2234" CreationDate="2010-08-31T10:13:21.270" Score="9" ViewCount="" Body="&lt;p&gt;Popular right now are randomForest and gbm (called MART or Gradient Boosting in machine learning literature), rpart for simple trees.  Also popular is bayesglm, which uses MAP with priors for regularization.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;install.packages(c(&quot;randomForest&quot;, &quot;gbm&quot;, &quot;rpart&quot;, &quot;arm&quot;))&#xA;&#xA;library(randomForest)&#xA;library(gbm)&#xA;library(rpart)&#xA;library(arm)&#xA;&#xA;r1 &amp;lt;- randomForest(y~x)&#xA;r2 &amp;lt;- gbm(y~x)&#xA;r3 &amp;lt;- rpart(y~x)&#xA;r4 &amp;lt;- bayesglm(y ~ x, family=binomial)&#xA;&#xA;yy1 &amp;lt;- predict(r1, data.frame(x=xx))&#xA;yy2 &amp;lt;- predict(r2, data.frame(x=xx))&#xA;yy3 &amp;lt;- predict(r3, data.frame(x=xx))&#xA;yy4 &amp;lt;- predict(r4, data.frame(x=xx), type=&quot;response&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1119" LastActivityDate="2010-08-31T10:13:21.270" CommunityOwnedDate="2010-08-31T10:13:21.270" />
  <row Id="2236" PostTypeId="2" ParentId="2234" CreationDate="2010-08-31T13:02:43.247" Score="5" ViewCount="" Body="&lt;p&gt;I agree with Joe, and would add:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any classification method could in principle be used, although it will depend on the data/situation.  For instance, you could also use a SVM, possibly with the popular C-SVM model.  Here's an example from kernlab using a radial basis kernel function:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(kernlab)&#xA;x &amp;lt;- rbind(matrix(rnorm(120),,2),matrix(rnorm(120,mean=3),,2))&#xA;y &amp;lt;- matrix(c(rep(1,60),rep(-1,60)))&#xA;&#xA;svp &amp;lt;- ksvm(x,y,type=&quot;C-svc&quot;)&#xA;plot(svp,data=x)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-08-31T13:02:43.247" CommunityOwnedDate="2010-08-31T13:02:43.247" CommentCount="1" />
  <row Id="2237" PostTypeId="1" CreationDate="2010-08-31T14:26:41.777" Score="1" ViewCount="149" Body="&lt;p&gt;Hi!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is based on the &quot;&lt;em&gt;forecast&lt;/em&gt;&quot;  package for R used in &lt;em&gt;Forecasting with Exponential Smoothing. The State Space Approach&lt;/em&gt; - Hyndman et al. 2008. I am using the &quot;&lt;em&gt;ets&lt;/em&gt;&quot; function to estimate the parameters of a model. Is there a way to obtain standard errors for the estimates of the smoothing parameters? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lavinia&lt;/p&gt;&#xA;" OwnerUserId="443" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-10-03T03:49:18.430" LastActivityDate="2010-10-03T03:49:18.430" Title="Standard errors for estimates of smoothing parameters" Tags="&lt;time-series&gt;&lt;forecasting&gt;" AnswerCount="2" />
  <row Id="2238" PostTypeId="2" ParentId="2237" CreationDate="2010-08-31T14:30:58.947" Score="2" ViewCount="" Body="&lt;p&gt;Not all methods lead to analytic expressions (preferably based on proper asymptotic results) that provides this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But the bootstrap allows you to approximate this via simulation. In essence, you generate (lots of) surrogate 'fake' data sets, employ your estimator on each of these and then use the population of your estimates to make inferences.  However, doing bootstrapping in a time series context has its own challenges...&lt;/p&gt;&#xA;" OwnerUserId="334" LastActivityDate="2010-08-31T14:30:58.947" CommentCount="3" />
  <row Id="2239" PostTypeId="2" ParentId="2234" CreationDate="2010-08-31T15:30:46.797" Score="8" ViewCount="" Body="&lt;p&gt;Actually, that depends on what you want to obtain. If you perform logistic regression only for the predictions, you can use any supervised classification method suited for your data. Another possibility : discriminant analysis ( lda() and qda() from package MASS)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;r &amp;lt;- lda(y~x) # use qda() for quadratic discriminant analysis&#xA;&#xA;xx &amp;lt;- seq(min(x), max(x), length=100)&#xA;pred &amp;lt;- predict(r, data.frame(x=xx), type='response')&#xA;yy &amp;lt;- pred$posterior[,2]&#xA;&#xA;color &amp;lt;- c(&quot;red&quot;,&quot;blue&quot;)&#xA;&#xA;plot(y~x,pch=19,col=color[pred$class])&#xA;abline(lm(y~x),col='red',lty=2)&#xA;lines(xx,yy, col='blue', lwd=5, lty=2)&#xA;title(main='lda implementation')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;On the other hand, if you need confidence intervals around your predictions or standard errors on your estimates, most classification algorithms ain't going to help you. You could use generalized additive (mixed) models, for which a number of packages are available. I often use the mgcv package of Simon Wood. Generalized additive models allow more flexibility than logistic regression, as you can use splines for modelling your predictors.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(55)&#xA;require(mgcv)&#xA;n &amp;lt;- 100&#xA;x1 &amp;lt;- c(rnorm(n), 1+rnorm(n))&#xA;x2 &amp;lt;- sqrt(c(rnorm(n,4),rnorm(n,6)))&#xA;y &amp;lt;- c(rep(0,n), rep(1,n))&#xA;r &amp;lt;- gam(y~s(x1)+s(x2),family=binomial)&#xA;&#xA;xx &amp;lt;- seq(min(x1), max(x1), length=100)&#xA;xxx &amp;lt;- seq(min(x2), max(x2), length=100)&#xA;yy &amp;lt;- predict(r, data.frame(x1=xx,x2=xxx), type='response')&#xA;&#xA;color=c(&quot;red&quot;,&quot;blue&quot;)&#xA;clustering &amp;lt;- ifelse(r$fitted.values &amp;lt; 0.5,1,2)&#xA;&#xA;plot(y~x1,pch=19,col=color[clustering])&#xA;abline(lm(y~x1),col='red',lty=2)&#xA;lines(xx,yy, col='blue', lwd=5, lty=2)&#xA;title(main='gam implementation')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;There's a whole lot more to do :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;op &amp;lt;- par(mfrow=c(2,1))&#xA;plot(r,all.terms=T)&#xA;par(op)&#xA;&#xA;summary(r)&#xA;anova(r)&#xA;&#xA;r2 &amp;lt;- gam(y~s(x1),family=binomial)&#xA;anova(r,r2,test=&quot;Chisq&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd recommend the &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1584884746&quot; rel=&quot;nofollow&quot;&gt;book of Simon Wood about Generalized Additive Models&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1124" LastActivityDate="2010-08-31T15:30:46.797" CommunityOwnedDate="2010-08-31T15:30:46.797" />
  <row Id="2240" PostTypeId="2" ParentId="2220" CreationDate="2010-08-31T15:55:59.283" Score="4" ViewCount="" Body="&lt;p&gt;As a permutation test is an &lt;strong&gt;exact&lt;/strong&gt; test, giving you an exact p-value. Bootstrapping a permutation test doesn't make sense. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Next to that, determining a confidence interval around a test statistic doesn't make sense either, as it is calculated based on your sample and not an estimate. You determine confidence intervals around estimates like means and the likes, but not around test statistics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Permutation tests should not be used on datasets that are so big you can't calculate all possible permutations any more. If that's the case, use a bootstrap procedure to determine the cut-off for the test statistic you use. But again, this has little to do with a 95% confidence interval. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;An example : I use here the classic T-statistic, but use a simple approach to bootstrapping for calculation of the empirical distribution of my statistic. Based on that, I calculate an empirical p-value :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x &amp;lt;- c(11.4,25.3,29.9,16.5,21.1)&#xA;y &amp;lt;- c(23.7,26.6,28.5,14.2,17.9,24.3)&#xA;&#xA;t.sample &amp;lt;- t.test(x,y)$statistic&#xA;t.dist &amp;lt;- apply(&#xA;      replicate(1000,sample(c(x,y),11,replace=F)),2,&#xA;      function(x){t.test(x[1:5],x[6:11])$statistic})&#xA;&#xA;# two sided testing&#xA;center &amp;lt;- mean(t.dist)&#xA;t.sample &amp;lt;-abs(t.sample-center)&#xA;t.dist &amp;lt;- abs(t.dist - center)&#xA;p.value &amp;lt;- sum( t.sample &amp;lt; t.dist ) / length(t.dist)&#xA;p.value&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Take into account that this 2-sided testing only works for symmetrical distributions. Non-symmetrical distributions are typically only tested one-sided.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;OK, I misunderstood the question. If you want to calculate a confidence interval on the estimate of the difference, you can use the code mentioned &lt;a href=&quot;http://stackoverflow.com/questions/3615718/bootstrapping-to-compare-two-groups/3615880#3615880&quot;&gt;here&lt;/a&gt; for bootstrapping within each sample. Mind you, this is a biased estimate: generally this gives a CI that is too small. Also see the example given there as a reason why you have to use a different approach for the confidence interval and the p-value.&lt;/p&gt;&#xA;" OwnerUserId="1124" LastEditorUserId="1124" LastEditorDisplayName="" LastEditDate="2010-09-01T08:54:35.887" LastActivityDate="2010-09-01T08:54:35.887" CommentCount="4" />
  <row Id="2241" PostTypeId="2" ParentId="423" CreationDate="2010-08-31T15:58:42.227" Score="6" ViewCount="" Body="&lt;p&gt;I found this &lt;a href=&quot;http://www.erlang-factory.com/upload/presentations/282/neo4j-is-not-erlang-but-i-still-heart-you-2010-06-10.pdf&quot; rel=&quot;nofollow&quot;&gt;from a NoSQL presentation&lt;/a&gt;, but the cartoon can be found directly at&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://browsertoolkit.com/fault-tolerance.png&quot; rel=&quot;nofollow&quot;&gt;http://browsertoolkit.com/fault-tolerance.png&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/MV8HX.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="1080" LastActivityDate="2010-08-31T15:58:42.227" CommunityOwnedDate="2010-08-31T15:58:42.227" />
  <row Id="2242" PostTypeId="2" ParentId="2182" CreationDate="2010-08-31T16:32:21.467" Score="1" ViewCount="" Body="&lt;p&gt;This answer is incorrect, as the true value A is not a random variable and hence P(A &amp;lt; some value) is meaningless. I tried to formalize the idea that overlapping confidence intervals doesn't mean that both samples are likely to be  drawn from the same population. My bad.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;WRONG ANSWER&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;More mathematically : Assuming the&#xA;  scores values are independent (which&#xA;  they aren't, but with the right&#xA;  setting it should be close), the&#xA;  chance that A has as low as 34.5 and B&#xA;  as high as 35.5 equals&#xA;  p(A&amp;lt;=34.5)*p(B&gt;=35.5) = 0.000625.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="1124" LastEditorUserId="1124" LastEditorDisplayName="" LastEditDate="2010-09-03T23:51:38.993" LastActivityDate="2010-09-03T23:51:38.993" CommentCount="3" />
  <row Id="2243" PostTypeId="2" ParentId="423" CreationDate="2010-08-31T17:41:55.473" Score="3" ViewCount="" Body="&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/tuNFK.jpg&quot; alt=&quot;Bush and Gorbachev in a statistical golf cart&quot;&gt; My favorite was created by Emanuel Parzen, appearing in &lt;a href=&quot;http://www.ima.umn.edu/preprints/July90Series/663.pdf&quot; rel=&quot;nofollow&quot;&gt;IMA preprint 663&lt;/a&gt;, but this illustrates my degenerate sense of humor. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Gorbachev says to Bush: &quot;that's a very nice golfcart, Mr. President. Can it change how statistics is practiced?&quot; etc. hahahah.&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-08-31T17:41:55.473" CommunityOwnedDate="2010-08-31T17:41:55.473" />
  <row Id="2244" PostTypeId="1" AcceptedAnswerId="2246" CreationDate="2010-08-31T18:56:09.437" Score="3" ViewCount="611" Body="&lt;p&gt;Hi&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the best package to to do some survival analysis and plots in R? I have tried some tutorials but I couldn't find a definite answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;TIA&lt;/p&gt;&#xA;" OwnerUserId="1088" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-16T12:33:07.553" LastActivityDate="2010-09-16T12:33:07.553" Title="Kaplan-Meier, survival analysis and plots in R" Tags="&lt;r&gt;&lt;data-visualization&gt;&lt;survival&gt;" AnswerCount="3" />
  <row Id="2245" PostTypeId="1" AcceptedAnswerId="2251" CreationDate="2010-08-31T19:13:04.883" Score="11" ViewCount="510" Body="&lt;p&gt;In his 1984 paper &lt;a href=&quot;http://www-unix.oit.umass.edu/~stanek/pdffiles/causal-holland.pdf&quot; rel=&quot;nofollow&quot;&gt;&quot;Statistics and Causal Inference&quot;&lt;/a&gt;, Paul Holland raised one of the most fundamental questions in statistics:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;What can a statistical model say about&#xA;  causation?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This led to his motto:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;NO CAUSATION WITHOUT MANIPULATION&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;which emphasized the importance of restrictions around experiments that consider causation. &#xA;Andrew Gelman makes &lt;a href=&quot;http://www.stat.columbia.edu/~cook/movabletype/archives/2010/08/no_understandin.html&quot; rel=&quot;nofollow&quot;&gt;a similar point&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&quot;To find out what happens when you change something, it is necessary to change it.&quot;...There are things you learn from perturbing a system that you'll never find out from any amount of passive observation.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;His ideas are summarized in &lt;a href=&quot;http://www.stat.columbia.edu/~gelman/research/published/causalreview4.pdf&quot; rel=&quot;nofollow&quot;&gt;this article&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What considerations should be made when making a causal inference from a statistical model?&lt;/p&gt;&#xA;" OwnerUserId="5" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-16T06:32:59.970" LastActivityDate="2010-09-17T18:22:02.003" Title="Statistics and causal inference?" Tags="&lt;causal-inference&gt;" AnswerCount="3" CommentCount="1" FavoriteCount="8" />
  <row Id="2246" PostTypeId="2" ParentId="2244" CreationDate="2010-08-31T19:41:55.037" Score="4" ViewCount="" Body="&lt;p&gt;Try CRAN Task View: &lt;a href=&quot;http://cran.at.r-project.org/web/views/Survival.html&quot; rel=&quot;nofollow&quot;&gt;http://cran.at.r-project.org/web/views/Survival.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-08-31T19:41:55.037" />
  <row Id="2247" PostTypeId="2" ParentId="2244" CreationDate="2010-08-31T19:48:40.107" Score="6" ViewCount="" Body="&lt;p&gt;I think that it's fair to say that the &lt;a href=&quot;http://cran.r-project.org/web/packages/survival/&quot; rel=&quot;nofollow&quot;&gt;survival&lt;/a&gt; package is the &quot;recommended&quot; package in general, as it's included in base R (i.e. does not need to be installed separately).  There are many good tutorials online for this.  But you need to be more specific to get a more specific answer.&lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-08-31T19:48:40.107" CommentCount="1" />
  <row Id="2248" PostTypeId="1" CreationDate="2010-08-31T20:57:50.950" Score="4" ViewCount="929" Body="&lt;p&gt;I have a series of observations that fall into bins (or &quot;scores&quot;); that is, the data can be 0, 1, 2, 3 or 4.  There are two groups of such data, control and treated.  I know the number of individuals with each score for each group.  What is the best way to determine whether these groups are different or not?  A colleague suggested just arranging the data as individual data points with the given score, and doing the analysis on those two columns of data.  Since there are ten individuals per group, this is not difficult, but I do not believe that I am getting a valid answer.  Thanks for your help.&lt;/p&gt;&#xA;" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-08-31T21:49:50.293" LastActivityDate="2010-09-19T23:57:17.357" Title="What is the proper way to analyze discrete data?" Tags="&lt;nonparametric&gt;&lt;statistical-significance&gt;&lt;discrete-data&gt;&lt;scales&gt;" AnswerCount="3" />
  <row Id="2249" PostTypeId="2" ParentId="2248" CreationDate="2010-08-31T21:33:24.810" Score="4" ViewCount="" Body="&lt;p&gt;Three things come to mind:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Contingency table analysis using Fisher's exact test or Chi Square (but will only tell you that somewhere in the table there is a difference that is significant. You'd have to visualize your data or do post-hoc tests to know where this difference is.) Not my preferred solution.&lt;/li&gt;&#xA;&lt;li&gt;A non-parametric method such the Mann Whitney test. This will rank all of your scores within each group. A good method, but may be underpowered.&lt;/li&gt;&#xA;&lt;li&gt;A parametric method (such as a t test). Disadvantage is that the assumptions of this method may be violated, especially with such a small sample. Also, the difference between 0 and 1 is not likely to be the same (depending on what you're measuring) as the difference between 3 and 4. The good news is that the t test is relatively robust to the assumptions you are supposed to ensure are true before using the test. However, as I said, the sample size is fairly small.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The best bet may be the Mann Whitney test.&lt;/p&gt;&#xA;" OwnerUserId="561" LastEditorUserId="561" LastEditorDisplayName="" LastEditDate="2010-09-01T01:57:27.007" LastActivityDate="2010-09-01T01:57:27.007" CommentCount="3" />
  <row Id="2250" PostTypeId="2" ParentId="2248" CreationDate="2010-08-31T21:49:30.223" Score="4" ViewCount="" Body="&lt;p&gt;What you are looking for seems to be a test for comparing two groups where observations are kind of ordinal data. In this case, I would suggest to apply a trend test to see if there are any differences between the CTL and TRT group. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using a t-test would not acknowledge the fact your data are discrete, and the Gaussian assumption may be seriously violated if scores distribution isn't symmetric as is often the case with Likert scores (such as the ones you seem to report). Don't know if these data come from a case-control study or not, but you might also apply rank-based method as suggested by @propfol: If it is not a matched design, the Wilcoxon-Mann-Whitney test (&lt;code&gt;wilcox.test()&lt;/code&gt; in R) is fine, and ask for an exact p-value although you may encounter problem with tied observations. The efficiency of the WMW test is $3/\pi$ with respect to the t-test if normality holds but it may even be better otherwise, I seem to remember.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given your sample size, you may also consider applying a permutation test (see the &lt;a href=&quot;http://cran.r-project.org/web/packages/perm/index.html&quot; rel=&quot;nofollow&quot;&gt;perm&lt;/a&gt; or &lt;a href=&quot;http://cran.r-project.org/web/packages/coin/index.html&quot; rel=&quot;nofollow&quot;&gt;coin&lt;/a&gt; R packages).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Check also those related questions:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://stats.stackexchange.com/questions/203/group-differences-on-a-five-point-likert-item/1888#1888&quot;&gt;Group differences on a five point Likert item&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://stats.stackexchange.com/questions/10/under-what-conditions-should-likert-scales-be-used-as-ordinal-or-interval-data/1887#1887&quot;&gt;Under what conditions should Likert scales be used as ordinal or interval data?&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-01T14:32:30.520" LastActivityDate="2010-09-01T14:32:30.520" CommentCount="2" />
  <row Id="2251" PostTypeId="2" ParentId="2245" CreationDate="2010-08-31T23:26:46.003" Score="7" ViewCount="" Body="&lt;p&gt;This is a broad question, but given the Box, Hunter and Hunter quote is true I think what it comes down to is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) The quality of the experimental design&#xA;    -  randomization, sample sizes, control of confounders,...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) The quality of the implementation of the design.&#xA;    - adherance to protocol, measurement error, data handling, ...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) The quality of the model to accurately reflect the design.&#xA;    - blocking structures are accurately represented, proper degrees of freedom are associated with effects, estimators are unbiased, ...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At the risk of stating the obvious I'll try to hit on the key points of each:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(1) is a large sub-field of statistics, but in it's most basic form I think it comes down to the fact that when making causal inference we ideally start with identical units that are monitored in identical environments other than being assigned to a treatment.  Any systematic differences between groups after assigment are then logically attributable to the treatment (we can infer cause).  But, the world isn't that nice and units differ prior to treatment and evironments during experiments are not perfectly controlled.  So we &quot;control what we can and randomize what we can't&quot;, which helps to insure that there won't be systematic bias due to the confounders that we controlled or randomized.  One problem is that experiments tend to be difficult (to impossible) and expensive and a large variety of designs have been developed to efficiently extract as much information as possible in as carefully controlled a setting as possible, given the costs.  Some of these are quite rigorous (e.g. in medicine the double-blind, randomized, placebo-controlled trial) and others less so (e.g. various forms of 'quasi-experiments').  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(2) is also a big issue and one that statisticians generally don't think about...though we should.  In applied statistical work I can recall incidences where 'effects' found in the data were spurious results of inconsistency of data collection or handling.  I also wonder how often information on true causal effects of interest is lost due to these issues (I believe students in the applied sciences generally have little-to-no training about ways that data can become corrupted - but I'm getting off topic here...)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(3) is another large technical subject, and another necessary step in objective causal inference.  To a certain degree this is taken care of because the design crowd develop designs and models together (since inference from a model is the goal, the attributes of the estimators drive design). But this only gets us so far because in the 'real world' we end up analysing experimental data from non-textbook designs and then we have to think hard about things like the appropriate controls and how they should enter the model and what associated degrees of freedom should be and whether assumptions are met if if not how to adjust of violations and how robust the estimators are to any remaining violations and...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyway, hopefully some of the above helps in thinking about considerations in making causal inference from a model.  Did I forget anything big?&lt;/p&gt;&#xA;" OwnerUserId="1107" LastEditorUserId="1107" LastEditorDisplayName="" LastEditDate="2010-09-01T00:08:44.693" LastActivityDate="2010-09-01T00:08:44.693" CommentCount="2" />
  <row Id="2252" PostTypeId="2" ParentId="2230" CreationDate="2010-09-01T00:00:38.753" Score="3" ViewCount="" Body="&lt;p&gt;If you enjoy visual explanations, there was a nice &lt;a href=&quot;http://dx.doi.org/doi:10.1198/tas.2009.0032&quot; rel=&quot;nofollow&quot;&gt;'Teacher's Corner' article&lt;/a&gt; on this subject in the American Statistician (cite below).  As a bonus, the authors included an &lt;a href=&quot;http://www.biostatisticien.eu/ConvergenceConcepts/&quot; rel=&quot;nofollow&quot;&gt;R package&lt;/a&gt; to facilitate learning.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;@article{lafaye09,&#xA;  title={Understanding Convergence Concepts: A Visual-Minded and Graphical Simulation-Based Approach},&#xA;  author={Lafaye de Micheaux, P. and Liquet, B.},&#xA;  journal={The American Statistician},&#xA;  volume={63},&#xA;  number={2},&#xA;  pages={173--178},&#xA;  year={2009},&#xA;  publisher={ASA}&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1107" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-09-01T00:23:54.000" LastActivityDate="2010-09-01T00:23:54.000" />
  <row Id="2253" PostTypeId="2" ParentId="2248" CreationDate="2010-09-01T00:27:21.270" Score="3" ViewCount="" Body="&lt;p&gt;This question is a little unusual because the nature of &quot;different&quot; is unspecified.  This response is formulated in the spirit of trying to detect as many kinds of differences as possible, not just changes of location (&quot;trend&quot;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One approach that might have more power than most, while remaining agnostic about the relative magnitudes represented by the five groups (e.g., adopting a multinomial model) yet retaining the ordering of the groups, is a Kolmogorov-Smirnov test: as a test statistic use the size of the largest deviation between the two empirical cdfs.  This is easy and quick to compute and it would also be easy to bootstrap a p-value by pooling the two sets of results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Specifically, let the count in bin $j$ for group $i$ be $k_{ij}$.  Then the empirical cdf for group $i$ is essentially the vector $\left( 0 = m_{i0}, m_{i1}, \ldots, m_{i5}=n_i \right) / n_i$ where $m_{i,j} = m_{i-1,j} + k_{ij}, 1 \le i \le 5$.  The test statistic is the sup norm of the difference of these two vectors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Critical values ($\alpha = 0.05$) with two groups of ten individuals are going to be around 0.2 - 0.4, with the higher values occurring when the 20 values are spread evenly between the two extremes.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-09-01T00:27:21.270" CommentCount="2" />
  <row Id="2254" PostTypeId="5" CreationDate="2010-08-30T10:31:13.000" Score="0" ViewCount="0" Body="&lt;p&gt;&lt;a href=&quot;http://www.r-project.org&quot; rel=&quot;nofollow&quot;&gt;R&lt;/a&gt; is available on a wide variety of UNIX platforms, Windows and MacOS, and can be downloaded from &lt;a href=&quot;http://cran.r-project.org/&quot; rel=&quot;nofollow&quot;&gt;CRAN&lt;/a&gt;. It is an implementation of the S programming language combined with lexical scoping semantics inspired by Scheme. R was created by Ross Ihaka and Robert Gentleman and is now developed by the R Development Core Team. It is easily extended through a packaging system on &lt;a href=&quot;http://cran.r-project.org/&quot; rel=&quot;nofollow&quot;&gt;CRAN&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="-1" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2011-01-10T23:21:33.097" LastActivityDate="2011-01-10T23:21:33.097" />
  <row Id="2255" PostTypeId="2" ParentId="93" CreationDate="2010-09-01T03:35:33.823" Score="2" ViewCount="" Body="&lt;p&gt;This is probably a stupid answer (I am new here), but if you want to estimate the hazard function from observations of an initial population that slowly died away (i.e. had events and then were censored), isn't that what the Nelson-Aalen estimator was built to do?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We could have another conversation about the reliability of the available classical confidence intervals -- my understanding is that there basically do not exist functioning exact confidence intervals that guarantee their coverage even over small sample sizes, since such an interval would need to work over all distributions of censoring time. (Maybe the problem is simpler when individuals are always censored exactly after their first event.) And mapping out the coverage of an approximate interval precisely would take work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But if you just need a point estimate, the Nelson-Aalen estimator seems to do the trick. (It's a lot like the Kaplan-Meier estimate for the survival function...)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to calculate an a posteriori distribution on a whole family of possible hazard functions, and your prior is that they are drawn from the Gaussian processes with certain statistics, can you explain further what the difficulty is? If there isn't agreement on the covariance matrix, then that needs to be part of the prior -- that the covariance matrix is drawn from some distribution. You're not going to get around having to state a prior if the goal is a posterior.&lt;/p&gt;&#xA;" OwnerUserId="1122" LastActivityDate="2010-09-01T03:35:33.823" />
  <row Id="2256" PostTypeId="1" AcceptedAnswerId="2257" CreationDate="2010-09-01T05:55:30.923" Score="2" ViewCount="291" Body="&lt;p&gt;I have a dataset forwhich i have performed an mds and visualized the results using scatterplot3d library. However i would like to see the names of the points on the 3d plot. How do i accomplish that? Each column belongs to a certain group i would like to see which points belong to which groups on the 3dplot.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#generate a distance matrix of the data&#xA;d &amp;lt;- dist(data)&#xA;&#xA;#perform the MDS on  3 dimensions and include a Goodness-of-fit (GOF)&#xA;&#xA;fit.mds &amp;lt;- cmdscale(d,eig=TRUE, k=3) # k is the number of dimensions; 3 in this case&#xA;&#xA;#Assign names x,y,z to the result vectors (dimension numbers)&#xA;x &amp;lt;- fit.mds$points[,1]&#xA;y &amp;lt;- fit.mds$points[,2]&#xA;z &amp;lt;- fit.mds$points[,3]&#xA;&#xA;plot3d &amp;lt;- scatterplot3d(x,y,z,highlight.3d=TRUE,xlab=&quot;&quot;,ylab=&quot;&quot;,pch=16,main=&quot;Multidimensional Scaling 3-D Plot&quot;,col.axis=&quot;blue&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="201" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-18T21:56:35.090" LastActivityDate="2010-09-18T21:56:35.090" Title="Adding labels to points using mds and scatter3d package with R" Tags="&lt;r&gt;&lt;multidimensional-scaling&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="2257" PostTypeId="2" ParentId="2256" CreationDate="2010-09-01T06:11:00.007" Score="2" ViewCount="" Body="&lt;p&gt;Basically, what you need is to store your &lt;code&gt;scatterplot3d&lt;/code&gt; in a variable and reuse it like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x &amp;lt;- replicate(10,rnorm(100))&#xA;x.mds &amp;lt;- cmdscale(dist(x), eig=TRUE, k=3)&#xA;s3d &amp;lt;- scatterplot3d(x.mds$points[,1:3])&#xA;text(s3d$xyz.convert(0,0,0), labels=&quot;Origin&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Replace the coordinates and text by whatever you want to draw. You can also use a color vector to highlight the groups of interest.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;code&gt;R.basic&lt;/code&gt; package, from H&lt;a href=&quot;http://www.braju.com/R/&quot; rel=&quot;nofollow&quot;&gt;enrik Bengtsson&lt;/a&gt;, seems to provide additional facilities to customize 3D plots, but I never tried it.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-09-01T06:11:00.007" CommentCount="2" />
  <row Id="2258" PostTypeId="1" AcceptedAnswerId="2279" CreationDate="2010-09-01T07:09:34.870" Score="1" ViewCount="128" Body="&lt;p&gt;Given a function mapping between two sample spaces $S_1$ and $S_2$, if $S_2$,$F_2$ is measurable how do I show that preimage of $S_2$,$F_2$ in $S_1$ is measurable set?&lt;/p&gt;&#xA;" OwnerUserId="862" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-09-02T00:32:21.437" LastActivityDate="2010-09-02T00:32:21.437" Title="How to prove that preimage of measurable space is measurable?" Tags="&lt;probability&gt;" AnswerCount="1" CommentCount="6" />
  <row Id="2259" PostTypeId="1" AcceptedAnswerId="2265" CreationDate="2010-09-01T07:23:52.913" Score="2" ViewCount="269" Body="&lt;p&gt;Imagine that &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;responses were collected on a 20 item scale which was designed to measure 4 factors with 5 items on each scale. &lt;/li&gt;&#xA;&lt;li&gt;participants were drawn from two groups (Group 1) and (Group 2) with sample size $n_1 = 150$ and $n_2 = 150$.&lt;/li&gt;&#xA;&lt;li&gt;a researcher wanted to assess the factor structure of the scale&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Common scenarios that I see in my consulting:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Group 1 are first year psychology students and Group 2 is sampled from the general community&lt;/li&gt;&#xA;&lt;li&gt;Group 1 are sampled at one period of time and Group 2 is sampled several years later&lt;/li&gt;&#xA;&lt;li&gt;Group 1 is a normal population and Group 2 is a clinical population&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Under what circumstances would it be appropriate to collapse across groups?&lt;/li&gt;&#xA;&lt;li&gt;How would these circumstances be assessed?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My initial Thoughts:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My own initial thoughts were as follows:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;strong&gt;Theoretical assessment&lt;/strong&gt;: assess the degree to which the two groups were sampled or measured in ways that would alter the means, sds, or correlation between the items&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Empirical assessment:&lt;/strong&gt; Examine differences between means, sds, and intercorrelations on the scales and optionally on other relevant variables (e.g., demographics); perform a &lt;a href=&quot;http://www.biostat.umn.edu/~melanie/PH7435/DATA/MultipleGroupCFAinAMOS.pdf&quot; rel=&quot;nofollow&quot;&gt;two-group confirmatory factor analysis&lt;/a&gt; to assess the consistency of the factor structure across groups.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Essentially, if the empirical evidence suggests that the groups are similar and the theoretical assessment suggests that they are similar, then it should be reasonable to combine.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Conclusion&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Does the approach above seem reasonable?&lt;/li&gt;&#xA;&lt;li&gt;Do you have alternative strategies?&lt;/li&gt;&#xA;&lt;li&gt;Are there any references that provide recommendations or examples regarding best practice in this situation?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-09-01T11:01:01.643" Title="When is it acceptable to collapse across groups when performing a factor analysis?" Tags="&lt;factor-analysis&gt;&lt;sampling&gt;" AnswerCount="3" FavoriteCount="1" />
  <row Id="2260" PostTypeId="2" ParentId="2259" CreationDate="2010-09-01T07:55:30.543" Score="1" ViewCount="" Body="&lt;p&gt;It might be a little fly by night, but your theory may suggest whether the two groups have the same factor structure or not.  If your theory suggests they do, and there is no reason to doubt the theory, I'd suggest you could go right ahead and trust that they have the same factor structure.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your empirical assessment would probably be a good route to go just to spot check the theoretical assessment as whether they are likely to share the same structure.  However, I don't intuitively see why mean differences between items would imply they have a different underlying factor structure.  It seems to me that might just suggest that one group has higher or lower scores on a given factor.&lt;/p&gt;&#xA;" OwnerUserId="196" LastActivityDate="2010-09-01T07:55:30.543" CommentCount="2" />
  <row Id="2261" PostTypeId="2" ParentId="2259" CreationDate="2010-09-01T08:44:54.663" Score="0" ViewCount="" Body="&lt;p&gt;The approach you mention seems reasonable, but you'd have to take into account that you cannot see the total dataset as a single population. So theoretically, you should use any kind of method that can take differences between those groups into account, similar to using &quot;group&quot; as a random term in an ANOVA or GLM approach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An alternative for empirical evaluation would be to check formally whether an effect of group can be found on the answers. To do that, you could create a binary dataset with following columns :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;yes/no  -  item  -  participant  -  group&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With this you can use item as a random term, participant nested in group and test the fixed effect of group using e.g. a glm with a logit link. You can just ignore participant too if you lose too many df.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is an approximation of the truth, but if the effect of group is significant, I wouldn't collapse the dataset.&lt;/p&gt;&#xA;" OwnerUserId="1124" LastActivityDate="2010-09-01T08:44:54.663" CommentCount="4" />
  <row Id="2262" PostTypeId="1" AcceptedAnswerId="2263" CreationDate="2010-09-01T09:46:35.167" Score="0" ViewCount="593" Body="&lt;p&gt;Hello. I am currently into a situation that i don't really know how to solve by myself.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to calculate the AUC of each peak and then compare these areas in relation to each other. The problem is that the peaks are not completely separated and the only information i got is the mean and the SD of each peak.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone know how to do this? Any hint or guess would already be really cool.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="1133" LastActivityDate="2010-09-01T14:52:06.747" Title="Area Under Curve (AUC) - given peak mean and standard deviation (SD)" Tags="&lt;normal-distribution&gt;" AnswerCount="3" CommentCount="4" />
  <row Id="2263" PostTypeId="2" ParentId="2262" CreationDate="2010-09-01T10:10:54.380" Score="2" ViewCount="" Body="&lt;p&gt;That really depends on the form and the height of the curve. If you assume the curves are all gaussian and you know the heights, then you can calculate the area under the curve by using the normal density function. In R this would become:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;heights &amp;lt;- 1&#xA;avg &amp;lt;- 3&#xA;sdev &amp;lt;- 2&#xA;&#xA;AUC &amp;lt;- heights/dnorm(avg,avg,sd) # the density function at the mean&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As the value of the density function at the mean is only determined by the sd, this information suffices for calculation of the AUC, given the assumptions are correct. If all heights are the same, the AUC is proportional to the sd only. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Without information about the shape of the curve and the heights, you simply cannot calculate the AUC as far as I know.&lt;/p&gt;&#xA;" OwnerUserId="1124" LastActivityDate="2010-09-01T10:10:54.380" CommentCount="10" />
  <row Id="2264" PostTypeId="1" CreationDate="2010-09-01T10:20:34.940" Score="3" ViewCount="250" Body="&lt;p&gt;I've got product ratings for a few thousand products. The number of ratings for each product varies from zero to about fifty. I want to find the expected value of product rating for each product. If there are lots of ratings for the product I'd expect the ev to be the average of the ratings for the product, but if there are only a few I'd expect the ev to be closer to the average of all ratings. How do I calculate the true ev? Please be gentle: I'm no statistician or mathematician. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit 1: Joris's answer below maintains I can't calculate ev because by definition that means I must have the entire population. In that case please can you tell me how to calculate the quantity that is similar to ev in spirit, does not require the entire population, and can make use of prior information.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit 2: I would expect that if each product's ratings have low variance ratings, or if there is a very high variance between different products' ratings, then the measured ratings are more significant.&lt;/p&gt;&#xA;" OwnerUserId="1134" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-17T20:22:34.877" LastActivityDate="2010-09-17T20:22:34.877" Title="Expected value of small sample" Tags="&lt;expected-value&gt;" AnswerCount="4" CommentCount="3" />
  <row Id="2265" PostTypeId="2" ParentId="2259" CreationDate="2010-09-01T11:01:01.643" Score="3" ViewCount="" Body="&lt;p&gt;There seems to be two cases to consider, depending on whether your scale was already validated using standard psychometric methods (from classical test or item response theory). In what follows, I will consider the first case where I assume preliminary studies have demonstrated construct validity and scores reliability for your scale.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this case, there is no formal need to apply exploratory factor analysis, unless you want to examine the pattern matrix within each group (but I generally do it, just to ensure that there are no items that unexpectedly highlight low factor loading or cross-load onto different factors); in order to be able to pool all your data, you need to use a multi-group factor analysis (hence, a confirmatory approach as you suggest), which basically amount to add extra parameters for testing a group effect on factor loading (1st order model) or factor correlation (2nd order model, if this makes sense) which would impact measurement invariance across subgroups of respondents. This can be done using &lt;a href=&quot;http://www.statmodel.com/&quot; rel=&quot;nofollow&quot;&gt;Mplus&lt;/a&gt; (see the discussion about CFA &lt;a href=&quot;http://www.statmodel.com/discussion/messages/9/9.html&quot; rel=&quot;nofollow&quot;&gt;there&lt;/a&gt;) or &lt;a href=&quot;http://www.vcu.edu/mx/&quot; rel=&quot;nofollow&quot;&gt;Mx&lt;/a&gt; (e.g. &lt;a href=&quot;http://www.eric.ed.gov:80/ERICWebPortal/search/detailmini.jsp?_nfpb=true&amp;amp;_&amp;amp;ERICExtSearch_SearchValue_0=EJ857035&amp;amp;ERICExtSearch_SearchType_0=no&amp;amp;accno=EJ857035&quot; rel=&quot;nofollow&quot;&gt;Conor et al.&lt;/a&gt;, 2009), not sure about &lt;a href=&quot;http://www.spss.com/amos/&quot; rel=&quot;nofollow&quot;&gt;Amos&lt;/a&gt; as it seems to be restricted to simple factor structure. The Mx software has been redesigned to work within the R environment, &lt;a href=&quot;http://openmx.psyc.virginia.edu/&quot; rel=&quot;nofollow&quot;&gt;OpenMx&lt;/a&gt;. The wiki is well responding so you can ask questions if you encounter difficulties with it. There is also a more recent package, &lt;a href=&quot;http://lavaan.ugent.be/&quot; rel=&quot;nofollow&quot;&gt;lavaan&lt;/a&gt;, which appears to be a promising package for SEMs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatives models coming from IRT may also be considered, including a Latent Regression Rasch Model (for each scale separately, see De Boeck and Wilson, 2004), or a Multivariate Mixture Rasch Model (von Davier and Carstensen, 2007). You can take a look at &lt;a href=&quot;http://www.jstatsoft.org/index.php?vol=20&quot; rel=&quot;nofollow&quot;&gt;Volume 20&lt;/a&gt; of the &lt;a href=&quot;http://www.jstatsoft.org/&quot; rel=&quot;nofollow&quot;&gt;Journal of Statistical Software&lt;/a&gt;, entirely devoted to psychometrics in R, for further information about IRT modeling with R.&#xA;You may be able to reach similar tests using Structural Equation Modeling, though. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If factor structure proves to be equivalent across the two groups, then you can aggregate the scores (on your four summated scales) and report your statistics as usual. &#xA;However, it is always a challenging task to use CFA since not rejecting H0 does by no mean allow you to check that your postulated theoretical model is correct in the true world, but just that there is no reason to reject it on statistical grounds; on the other hand, rejecting the null would lead to accept the alternative, which is generally left unspecified, unless you apply sequential testing of nested models. Anyway, this is the way we go in cross-cultural settings, especially when we want to assess whether a given questionnaire (e.g., on Patients Reported Outcomes) measures what it purports to do whatever the population it is administered to.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, regarding the apparent differences between the two groups -- one is drawn from a population of students, the other is a clinical sample, assessed at a later date -- it depends very much on your own considerations: Does mixing of these two samples makes sense from the literature surrounding the questionnaire used (esp., it should have shown temporal stability and applicability in a wide population), do you plan to generalize your findings over a larger population (obviously, you gain power by increasing sample size). At first sight, I would say that you need to ensure that both groups are comparable with respect to the characteristics thought to influence one's score on this questionnaire (e.g., gender, age, SES, biomedical history, etc.), and this can be done using classical statistics for two-groups comparison (on raw scores). It is worth noting that in clinical studies, we face the reverse situation: We usually want to show that scores differ between different clinical subgroups (or between treated and naive patients), which is often refered to as &lt;em&gt;know-group validity&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Reference:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;De Boeck, P. and Wilson, M. (2004). &lt;em&gt;Explanatory Item Response Models. A Generalized Linear and Nonlinear Approach&lt;/em&gt;. Springer.&lt;/li&gt;&#xA;&lt;li&gt;von Davier, M. and Carstensen, C.H. (2007). &lt;em&gt;Multivariate and Mixture Distribution Rasch Models&lt;/em&gt;. Springer.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-09-01T11:01:01.643" CommentCount="2" />
  <row Id="2266" PostTypeId="2" ParentId="2264" CreationDate="2010-09-01T11:26:03.903" Score="3" ViewCount="" Body="&lt;p&gt;The true expected value cannot be calculated. You can estimate it using the mean of the ratings for each product, and get an idea about the position by calculating the 95% confidence interval (CI) on the mean.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is done by&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$CI \approx avg \pm 2 * \frac{SD}{\sqrt{n}}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;with n being the number of ratings, SD the standard deviation and avg the average. More correct would be to use the T-distribution, where you use the 2.5% and 97.5% quantile of the T-distribution with degrees of freedom equal to number of observations minus one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$CI = avg \pm T_{(p=0.975,df=n-1)} * \frac{SD}{\sqrt{n}}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For 10 ratings, $T_{(p=0.975,df=n-1)}$ is 2.26. For 50 ratings, it is 2.01.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There's a chance of 95% this confidence interval contains the true expected value. You assume here that the distribution of the average is normal. If you have a very low amount of ratings, the SD can be estimated wrongly. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In that case, you could estimate an &quot;overall&quot; standard deviation on the scoring, and use that to calculate the CI. But keep in mind that this way you assume that the standard deviation is the same for every product. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In extremis, you could resort to bootstrapping to calculate the CI for every product. This will increase the calculation time substantially, and won't be adding any value for products with enough ratings. &lt;/p&gt;&#xA;" OwnerUserId="1124" LastEditorUserId="1124" LastEditorDisplayName="" LastEditDate="2010-09-01T15:39:17.250" LastActivityDate="2010-09-01T15:39:17.250" CommentCount="10" />
  <row Id="2267" PostTypeId="2" ParentId="1676" CreationDate="2010-09-01T11:31:31.833" Score="2" ViewCount="" Body="&lt;p&gt;You can also take a look at &lt;a href=&quot;http://cran.r-project.org/web/views/&quot; rel=&quot;nofollow&quot;&gt;Task views&lt;/a&gt; on CRAN and see if something suit your needs. I agree with @Jeromy for these must-have packages (for data manipulation and plotting).&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-09-01T11:31:31.833" CommunityOwnedDate="2010-09-01T11:31:31.833" />
  <row Id="2268" PostTypeId="2" ParentId="2264" CreationDate="2010-09-01T11:57:18.720" Score="0" ViewCount="" Body="&lt;p&gt;I haven't looked into it much, but this article on &lt;a href=&quot;http://www.thebroth.com/blog/118/bayesian-rating&quot; rel=&quot;nofollow&quot;&gt;Bayesian rating systems&lt;/a&gt; looks interesting.&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-09-01T11:57:18.720" CommentCount="2" />
  <row Id="2269" PostTypeId="1" CreationDate="2010-09-01T12:07:28.173" Score="2" ViewCount="252" Body="&lt;p&gt;How can I access tables created in SAS Enterprise Guide Client into SAS Enterprise Miner Client?&lt;/p&gt;&#xA;" OwnerUserId="1135" LastActivityDate="2011-05-31T18:52:33.753" Title="Access tables created in SAS Enterprise Guide Client into SAS Enterprise Miner Client?" Tags="&lt;sas&gt;" AnswerCount="1" CommentCount="8" FavoriteCount="1" />
  <row Id="2270" PostTypeId="2" ParentId="1963" CreationDate="2010-09-01T12:21:52.743" Score="3" ViewCount="" Body="&lt;p&gt;I'll add an independent recommendation for Jeromy's blog post, and second the suggestions of James DeCoster's notes and the Borenstein textbook (propofols' no. 2).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At risk of indulging in self-promotion, I recently published a methods paper entitled &lt;a href=&quot;http://onlinelibrary.wiley.com/doi/10.1111/j.2041-210X.2010.00056.x/abstract&quot; rel=&quot;nofollow&quot;&gt;Getting Started with Meta-analysis&lt;/a&gt;.  It's aimed at ecologists and evolutionary biologists, so the examples are taken from these fields, but I hope it will be useful for those working in other areas. &lt;/p&gt;&#xA;" OwnerUserId="266" LastActivityDate="2010-09-01T12:21:52.743" CommunityOwnedDate="2010-09-01T12:21:52.743" CommentCount="1" />
  <row Id="2271" PostTypeId="1" AcceptedAnswerId="2311" CreationDate="2010-09-01T13:39:53.860" Score="6" ViewCount="701" Body="&lt;p&gt;I just came by the following plot:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/kQ6ke.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And wondered how can it be done in R?  (or other softwares)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update 10.03.11&lt;/strong&gt;: Thank you everyone who participated in answering this question - you gave wonderful solutions!  I've compiled all the solution presented here (as well as some others I've came by online) in &lt;a href=&quot;http://www.r-statistics.com/2011/03/beeswarm-boxplot-and-plotting-it-with-r/&quot; rel=&quot;nofollow&quot;&gt;a post on my blog&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="253" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-10T16:10:49.130" LastActivityDate="2011-03-10T16:10:49.130" Title="How to plot a violin scatter boxplot (in R)?" Tags="&lt;r&gt;&lt;data-visualization&gt;" AnswerCount="4" CommentCount="12" FavoriteCount="2" />
  <row Id="2272" PostTypeId="1" AcceptedAnswerId="2287" CreationDate="2010-09-01T13:53:07.183" Score="14" ViewCount="1244" Body="&lt;p&gt;Joris and Srikant's exchange &lt;a href=&quot;http://stats.stackexchange.com/questions/2182/can-you-explaining-why-statistical-tie-is-not-naively-when-p-1-p-2-2-moe/2242#2242&quot;&gt;here&lt;/a&gt; got me wondering (again) if my internal explanations for the the difference between confidence intervals and credible intervals were the correct ones.  How you would explain the difference?&lt;/p&gt;&#xA;" OwnerUserId="71" LastActivityDate="2010-09-16T09:35:44.257" Title="What's the difference between a confidence interval and a credible interval?" Tags="&lt;bayesian&gt;&lt;confidence-interval&gt;&lt;frequentist&gt;&lt;credible-interval&gt;" AnswerCount="5" CommentCount="2" FavoriteCount="7" />
  <row Id="2273" PostTypeId="2" ParentId="2271" CreationDate="2010-09-01T13:54:11.347" Score="2" ViewCount="" Body="&lt;p&gt;Try the &lt;a href=&quot;http://cran.r-project.org/web/packages/vioplot/index.html&quot; rel=&quot;nofollow&quot;&gt;vioplot&lt;/a&gt; package:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(vioplot)&#xA;vioplot(rnorm(100))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(with awful default color ;-)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is also wvioplot() in the &lt;a href=&quot;http://cran.r-project.org/web/packages/wvioplot/index.html&quot; rel=&quot;nofollow&quot;&gt;wvioplot&lt;/a&gt; package, for weighted violin plot, and &lt;a href=&quot;http://www.jstatsoft.org/v28/c01/paper&quot; rel=&quot;nofollow&quot;&gt;beanplot&lt;/a&gt;, which combines violin and rug plots. They are also available through the &lt;a href=&quot;http://cran.r-project.org/web/packages/lattice/index.html&quot; rel=&quot;nofollow&quot;&gt;lattice&lt;/a&gt; package, see &lt;code&gt;?panel.violin&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-01T14:45:42.040" LastActivityDate="2010-09-01T14:45:42.040" CommentCount="7" />
  <row Id="2274" PostTypeId="2" ParentId="2262" CreationDate="2010-09-01T14:09:09.870" Score="1" ViewCount="" Body="&lt;p&gt;Given how your plot looks like, I would suggest rather to fit a mixture of gaussians and get their respective densities. Look at the &lt;a href=&quot;http://cran.r-project.org/web/packages/mclust/index.html&quot; rel=&quot;nofollow&quot;&gt;mclust&lt;/a&gt; package; basically this is refered to model-based clustering (you are seeking groups of points belonging to a given distribution, that is to be estimated, whose location parameter -- but also shape -- varies along a common dimension). A full explanation of MClust is available &lt;a href=&quot;http://www.google.fr/url?sa=t&amp;amp;source=web&amp;amp;cd=5&amp;amp;ved=0CD0QFjAE&amp;amp;url=http%3A%2F%2Fciteseerx.ist.psu.edu%2Fviewdoc%2Fdownload%3Fdoi%3D10.1.1.156.6814%26rep%3Drep1%26type%3Dpdf&amp;amp;rct=j&amp;amp;q=Mclust%201D%20mixture%20of%20gaussian&amp;amp;ei=Cl1-TI_YAovOswb3zIiUCQ&amp;amp;usg=AFQjCNHRwRIYnWEvDbdMqBj6s2LmpCSouw&amp;amp;sig2=gkbsyHKQMi9KiontFQ0E1g&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems the &lt;a href=&quot;http://cran.r-project.org/web/packages/delt/index.html&quot; rel=&quot;nofollow&quot;&gt;delt&lt;/a&gt; package offers an alternative way to fit 1D data with a mixture of gaussians, but I didn't get into details. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyway, I think this is the best way to get automatic estimates and avoid cutting your x-scale at arbitrary locations.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-09-01T14:09:09.870" CommentCount="1" />
  <row Id="2275" PostTypeId="1" AcceptedAnswerId="2285" CreationDate="2010-09-01T14:29:02.513" Score="2" ViewCount="120" Body="&lt;p&gt;I do not study statistics but engineering, but this is a statistics question, and I hope you can lead me to what I need to learn to solve this problem.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have this situation where I calculate probabilities of 1000's of things happening in like 30 days.  If in 30 days I see what actually happened, how can I test to see how accurately I predicted?  These calculations result in probabilities and in actual values (ft).  What is the method for doing this?&#xA;Thanks,&#xA;CP&lt;/p&gt;&#xA;" OwnerUserId="1137" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-09-02T00:12:45.650" LastActivityDate="2010-09-02T00:12:45.650" Title="How can I determine accuracy of past probability calculations?" Tags="&lt;probability&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="1" />
  <row Id="2277" PostTypeId="2" ParentId="2262" CreationDate="2010-09-01T14:52:06.747" Score="1" ViewCount="" Body="&lt;p&gt;It is critical to know how the peak heights and sds were calculated.  (I take &quot;mean&quot; in the question to be a mistaken way of referring to a height.  Without the heights, the problem is hopeless; it would be like requesting a formula for the area of a rectangle given only its width and location.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One would expect, as Joris Meys' answer and its commentary suggest, that the area could be estimated as a sum of Gaussians.  Actually, we don't need to assume a Gaussian shape; almost any standard (preferably unimodal, continuous) shape will do, because the area will be proportional to the peak height (a y-scale factor) and the sd (an x-scale factor), whence the total estimated area ought to be a constant times the sum of height*SD and the relative contribution of each peak will equal its height*SD divided by this sum.  But this all assumes the heights and sds were fit to the curve with such an application in mind.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I realize there are many problems with such a formula, but let's not get carried away by all the detail in the example graph: the problem as posed says that the &quot;means&quot; and SDs are the &lt;em&gt;only&lt;/em&gt; information available.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-09-01T14:52:06.747" CommentCount="4" />
  <row Id="2278" PostTypeId="2" ParentId="423" CreationDate="2010-09-01T15:02:00.683" Score="12" ViewCount="" Body="&lt;p&gt;And another one from xkcd.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Title: &lt;strong&gt;Self-Description&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/T1Hep.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The mouseover text:  &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The contents of any one panel are&#xA;  dependent on the contents of every&#xA;  panel including itself. The graph of&#xA;  panel dependencies is complete and&#xA;  bidirectional, and each node has a&#xA;  loop. The mouseover text has two&#xA;  hundred and forty-two characters.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="442" LastActivityDate="2010-09-01T15:02:00.683" CommunityOwnedDate="2010-09-01T15:02:00.683" />
  <row Id="2279" PostTypeId="2" ParentId="2258" CreationDate="2010-09-01T15:07:38.893" Score="1" ViewCount="" Body="&lt;p&gt;Strictly speaking, this is trivial: the preimage of $(S_2, F_2)$ is all of $S_1$ (by definition), which is measurable (by definition).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Perhaps you want to conclude that the preimage of any &lt;em&gt;measurable subset&lt;/em&gt; of $S_2$ is measurable: that is a nice property of a function.  However, this conclusion is not true in general, either.  For example, let $(S_1, F_1)$ contain a nonmeasurable set and let $(S_2, F_2)$ contain two disjoint measurable atoms.  Map every element of the nonmeasurable set to one of the atoms and map every element of the complement of the nonmeasurable set to the other atom.  The preimage of the first atom is not measurable, whence this map is not measurable.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-09-01T15:07:38.893" CommentCount="1" />
  <row Id="2280" PostTypeId="2" ParentId="2264" CreationDate="2010-09-01T16:01:33.643" Score="4" ViewCount="" Body="&lt;p&gt;Incorporating a prior is one way to 'make up' for small samples.  Another is to use a mixed model, with an intercept for the mean structure and a random intercept for each product.  The estimate of the population mean plus the predicted random effect (BLUP) then offers a form of shrinkage, where values for products with less information are shrunk more toward the overall sample mean than those based on more information.  This method is common in, for example, Small Area Estimation in survey sampling.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit:  The R code might look like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(nlme)&#xA;f &amp;lt;- lme(score ~ 1, data = yourData, random = ~1|product)&#xA;p &amp;lt;- predict(f)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you go this route the assumptions are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;independent, normal errors with expected value 0 and constant variance for all observations&lt;/li&gt;&#xA;&lt;li&gt;normal random effects with expected value 0&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Violations of these can generally be modeled, but of course with that comes added complexity...&lt;/p&gt;&#xA;" OwnerUserId="1107" LastEditorUserId="1107" LastEditorDisplayName="" LastEditDate="2010-09-01T23:12:40.680" LastActivityDate="2010-09-01T23:12:40.680" CommentCount="4" />
  <row Id="2281" PostTypeId="2" ParentId="2272" CreationDate="2010-09-01T16:01:43.313" Score="10" ViewCount="" Body="&lt;p&gt;My understanding is as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Background&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose that you have some data $x$ and you are trying to estimate $\theta$. You have a data generating process that describes how $x$ is generated conditional on $\theta$. In other words you know the distribution of $x$ (say, $f(x|\theta)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Inference Problem&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your inference problem is: What values of $\theta$ are reasonable given the observed data $x$ ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Confidence Intervals&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Confidence intervals are a classical answer to the above problem. In this approach, you assume that there is &lt;em&gt;true, fixed&lt;/em&gt; value of $\theta$. Given this assumption, you use the data $x$ to get to an estimate of $\theta$ (say, $\hat{\theta}$). Once you have your estimate you want to assess where the true value is in relation to your estimate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Notice that under this approach the true value is &lt;em&gt;not&lt;/em&gt; a random variable. It is a fixed but unknown quantity. In contrast, your estimate &lt;em&gt;is&lt;/em&gt; a random variable as it depends on your data $x$ which was generated from your data generating process. Thus, you realize that you get different estimates each time you repeat your study.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The above understanding leads to the following methodology to assess where the true parameter is in relation to your estimate. Define an interval, $I \equiv [lb(x), ub(x)]$ with the following property:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(\theta \in I) = 0.95$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An interval constructed like the above is what is called a confidence interval. Since, the true value is unknown but fixed, the true value is either in the interval or outside the interval. The confidence interval then is a statement about the likelihood that the interval we obtain actually has the true parameter value. Thus, the probability statement is about the interval (i.e., the chances that interval which has the true value or not) rather than about the location of the true parameter value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this paradigm, it is meaningless to speak about the probability that a true value is less than or greater than some value as the true value is &lt;em&gt;not&lt;/em&gt; a random variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Credible Intervals&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In contrast to the classical approach, in the bayesian approach we assume that the true value is a random variable. Thus, we capture the our uncertainty about the true parameter value by a imposing a prior distribution on the true parameter vector (say $f(\theta)$).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using bayes theorem, we construct the posterior distribution for the parameter vector by blending the prior and the data we have (briefly the posterior is $f(\theta|-) \propto f(\theta) f(x|\theta)$).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We then arrive at a point estimate using the posterior distribution (e.g., use the mean of the posterior distribution). However, since under this paradigm, the true parameter vector is a random variable, we also want to know the extent of uncertainty we have in our point estimate. Thus, we construct an interval such that the following holds:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(l(\theta) \le {\theta} \le ub(\theta)) = 0.95$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The above is a credible interval. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Summary&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Credible intervals capture our current uncertainty in the location of the parameter values and thus can be interpreted as probabilistic statement about the parameter. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In contrast, confidence intervals capture the uncertainty about the interval we have obtained (i.e., whether it contains the true value or not). Thus, they cannot be interpreted as a probabilistic statement about the true parameter values.&lt;/p&gt;&#xA;" LastActivityDate="2010-09-01T16:01:43.313" CommentCount="15" />
  <row Id="2282" PostTypeId="1" AcceptedAnswerId="2283" CreationDate="2010-09-01T16:04:03.513" Score="0" ViewCount="90" Body="&lt;p&gt;Following &lt;a href=&quot;http://stats.stackexchange.com/questions/1676/i-just-installed-the-latest-version-of-r-what-packages-should-i-obtain&quot;&gt;this question&lt;/a&gt;, I wish to have some way of counting how many times I am using a package in my daily work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a function/package to do that?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In case there isn't, how would you construct such a capability?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The way I would do that is by changing it so that at the end of any R session, a log file of the commands would be saved to some location.  On that file, I would grep out all instances of &quot;library&quot; and &quot;require&quot;.  Then save the results into a file (with some time stamp).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lastly, I might want some function to (once in X time) send this file to a remote FTP location - so that other R users could analyse the results.  (is there a way to do that with R ?!)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If someone wants to try and construct such a machanism - I'd be glad to help by providing a relevant FTP account and by spreading the word on it on &lt;a href=&quot;http://R-bloggers.com&quot; rel=&quot;nofollow&quot;&gt;&quot;R bloggers&quot;&lt;/a&gt; for the good of the community.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;p.s (mainly for Shane): I wasn't sure if this question should go on stackoverflow or here.  If this type of question wasn't debated yet on the meta.stat - it should be.  If it was, I'll be glad to know what the conclusion of that discussion was. &lt;/p&gt;&#xA;" OwnerUserId="253" LastActivityDate="2010-09-01T16:13:47.113" ClosedDate="2010-09-01T16:59:06.390" Title="Counting how many times a package has been loaded in R ?" Tags="&lt;r&gt;&lt;recommendations&gt;" AnswerCount="1" CommentCount="10" />
  <row Id="2283" PostTypeId="2" ParentId="2282" CreationDate="2010-09-01T16:13:47.113" Score="1" ViewCount="" Body="&lt;p&gt;Overload &lt;code&gt;library()&lt;/code&gt; and &lt;code&gt;require()&lt;/code&gt; so that they report what they do (whichever way: append to a text file, say) and have those replacement functions loaded first at startup.&lt;/p&gt;&#xA;" OwnerUserId="334" LastActivityDate="2010-09-01T16:13:47.113" />
  <row Id="2284" PostTypeId="2" ParentId="2275" CreationDate="2010-09-01T16:49:30.997" Score="3" ViewCount="" Body="&lt;p&gt;In their classic book on the &lt;em&gt;Federalist&lt;/em&gt; papers, Mosteller and Wallace argue for a log penalty function: you penalize yourself $-\log(p)$ when you predict an event with probability $p$ and it occurs; the penalty for it not occurring equals $-\log(1-p)$.  Thus, the penalty is high when whatever happens is unexpected according to your prediction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Their argument in favor of this function rests on a simple natural criterion: &quot;the penalty function should encourage the prediction of the correct probabilities if they are known.&quot;  Assuming the total penalty is summed over all predictions and there will be three or more of them, M&amp;amp;W claim that the log penalty function is the &lt;em&gt;only&lt;/em&gt; one (up to affine transformation) for which the &quot;expected penalty is minimized over all predictions&quot; by the correct probabilities.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Following this, then, a good test for you to use is to track your accumulated log penalties.  If, after a long time (or by means of some independent oracle), you obtain accurate estimates of what the probabilities actually were, you can compare your penalty with the minimum possible one.  The average of that difference measures your long-run predictive performance (the lower the better).  This is an excellent way to compare two or more competing predictors, too.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-09-01T16:49:30.997" CommentCount="1" />
  <row Id="2285" PostTypeId="2" ParentId="2275" CreationDate="2010-09-01T17:30:32.223" Score="4" ViewCount="" Body="&lt;p&gt;What you're looking for are called &lt;strong&gt;Scoring Rules&lt;/strong&gt;, which are ways of evaluating probabilistic forecasts. They were invented in the 1950s by weather forecasters, and there has been a been a bit of work on them in the statistics community, but I don't know of any books on the topic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One thing you could do would be to bin the forecasts by probability range (e.g.: 0-5%, 5%-10%, etc.) and look at how many predicted events in that range occurred (if there are 40 events in the 0-5% range, and 20 occur, then your might have problems). If the events are independent, then you could compare these numbers to a binomial distribution.&lt;/p&gt;&#xA;" OwnerUserId="495" LastActivityDate="2010-09-01T17:30:32.223" CommentCount="3" />
  <row Id="2286" PostTypeId="2" ParentId="1337" CreationDate="2010-09-01T18:23:56.687" Score="0" ViewCount="" Body="&lt;p&gt;Here's a groaner:&#xA;Q: What do you call 100 statisticians at a tea party?&#xA;A: A Z-Party.&lt;/p&gt;&#xA;" OwnerUserId="1118" LastActivityDate="2010-09-01T18:23:56.687" CommunityOwnedDate="2010-09-01T18:23:56.687" />
  <row Id="2287" PostTypeId="2" ParentId="2272" CreationDate="2010-09-01T18:46:23.463" Score="11" ViewCount="" Body="&lt;p&gt;I agree completely with Srikant's explanation. To give a more heuristic spin on it:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Classical approaches generally posit that the world is one way (e.g., a parameter has one particular true value), and try to conduct experiments whose resulting conclusion -- no matter the true value of the parameter -- will be correct with at least some minimum probability.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a result, to express uncertainty in our knowledge after an experiment, the frequentist approach uses a &quot;confidence interval&quot; -- a range of values designed to include the true value of the parameter with some minimum probability, say 95%. A frequentist will design the experiment and 95% confidence interval procedure so that out of every 100 experiments run start to finish, at least 95 of the resulting confidence intervals will be expected to include the true value of the parameter. The other 5 might be slightly wrong, or they might be complete nonsense -- formally speaking that's ok as far as the approach is concerned, as long as 95 out of 100 inferences are correct. (Of course we would prefer them to be slightly wrong, not total nonsense.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bayesian approaches formulate the problem differently. Instead of saying the parameter has one true value, a Bayesian method says the value is chosen from some probability distribution -- known as the prior probability distribution. This &quot;prior&quot; might be known (imagine trying to estimate the size of a truck, if we know the overall distribution of truck sizes from the DMV) or it might be an assumption drawn out of thin air. The Bayesian inference is simpler -- we collect some data, and then calculate the probability of different values of the parameter GIVEN the data. This new probability distribution is called the &quot;a posteriori probability&quot; or simply the &quot;posterior.&quot; Bayesian approaches can summarize their uncertainty by giving a range of values on the posterior probability distribution that includes 95% of the probability -- this is called a &quot;95% credibility interval.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A Bayesian partisan might criticize the frequentist confidence interval like this: &quot;So what if 95 out of 100 experiments yield a confidence interval that includes the true value? I don't care about 99 experiments I DIDN'T DO; I care about this experiment I DID DO. Your rule allows 5 out of the 100 to be complete nonsense [negative values, impossible values] as long as the other 95 are correct; that's ridiculous.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A frequentist die-hard might criticize the Bayesian credibility interval like this: &quot;So what if 95% of the posterior probability is included in this range? What if the true value is, say, 0.37? If it is, then your method, run start to finish, will be WRONG 75% of the time. Your response is, 'Oh well, that's ok because according to the prior it's very rare that the value is 0.37,' and that may be so, but I want a method that works for ANY possible value of the parameter. I don't care about 99 values of the parameter that IT DOESN'T HAVE; I care about the one true value IT DOES HAVE. Oh also, by the way, your answers are only correct if the prior is correct. If you just pull it out of thin air because it feels right, you can be way off.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a sense both of these partisans are correct in their criticisms of each others' methods, but I would urge you to think mathematically about the distinction -- as Srikant explains.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a talk on this subject that goes into more detail, including an extended example of where confidence intervals and credibility intervals diverge; see &lt;a href=&quot;http://groups.csail.mit.edu/mac/users/gjs/6.945/readings/winstein-inference-slides.pdf&quot; rel=&quot;nofollow&quot;&gt;http://groups.csail.mit.edu/mac/users/gjs/6.945/readings/winstein-inference-slides.pdf&lt;/a&gt; .&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Here's an extended example from that talk that shows the difference precisely in a discrete example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I was a child my mother used to occasionally surprise me by ordering a jar of chocolate-chip cookies to be delivered by mail. The delivery company stocked four different kinds of cookie jars -- type A, type B, type C, and type D, and they were all on the same truck and you were never sure what type you would get. Each jar had exactly 100 cookies, but the feature that distinguished the different cookie jars was their respective distributions of chocolate chips per cookie. If you reached into a jar and took out a single cookie uniformly at random, these are the probability distributions you would get on the number of chips:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/yXQQQ.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A type-A cookie jar, for example, has 70 cookies with two chips each, and no cookies with four chips or more! A type-D cookie jar has 70 cookies with one chip each. Notice how each vertical column is a probability mass function -- the conditional probability of the number of chips you'd get, given that the jar = A, or B, or C, or D, and each column sums to 100.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I used to love to play a game as soon as the deliveryman dropped off my new cookie jar. I'd pull one single cookie at random from the jar, count the chips on the cookie, and try to express my uncertainty -- at the 70% level -- of which jars it could be. Thus it's the identity of the jar (A, B, C or D) that is the &lt;strong&gt;value of the parameter&lt;/strong&gt; being estimated. The number of chips (0, 1, 2, 3 or 4) is the &lt;strong&gt;outcome&lt;/strong&gt; or the observation or the sample.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Originally I played this game using a frequentist, 70% confidence interval. Such an interval needs to make sure that &lt;strong&gt;no matter&lt;/strong&gt; the true value of the parameter, meaning no matter which cookie jar I got, the interval would cover that true value with at least 70% probability.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An interval, of course, is a function that relates an outcome (a row) to a set of values of the parameter (a set of columns). But to &lt;em&gt;construct&lt;/em&gt; the confidence interval and guarantee 70% coverage, we need to work &quot;vertically&quot; -- looking at each column in turn, and making sure that 70% of the probability mass function is covered so that 70% of the time, that column's identity will be part of the interval that results. Remember that it's the vertical columns that form a p.m.f.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So after doing that procedure, I ended up with these intervals:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/tHJv1.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if the number of chips on the cookie I draw is 1, my confidence interval will be {B,C,D}. If the number is 4, my confidence interval will be {B,C}. Notice that since each column sums to 70% or greater, then no matter which column we are truly in (no matter which jar the deliveryman dropped off), the interval resulting from this procedure will include the correct jar with at least 70% probability.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Notice also that the procedure I followed in constructing the intervals had some discretion. In the column for type-B, I could have just as easily made sure that the intervals that included B would be 0,1,2,3 instead of 1,2,3,4. That would have resulted in 79% coverage for type-B (17+20+22+20), still meeting the lower bound of 70%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My sister Bayesia thought this approach was crazy, though. &quot;You have to consider the deliverman as part of the system,&quot; she said. &quot;Let's treat the identity of the jar as a random variable itself, and let's &lt;em&gt;assume&lt;/em&gt; that the deliverman chooses among them uniformly -- meaning he has all four on his truck, and when he gets to our house he picks one at random, each with uniform probability.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;With that assumption, now let's look at the joint probabilities of the whole event -- the jar type &lt;strong&gt;and&lt;/strong&gt; the number of chips you draw from your first cookie,&quot; she said, drawing the following table:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/zo4wH.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Notice that the whole table is now a probability mass function -- meaning the whole table sums to 100%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Ok,&quot; I said, &quot;where are you headed with this?&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;You've been looking at the conditional probability of the number of chips, given the jar,&quot; said Bayesia. &quot;That's all wrong! What you really care about is the conditional probability of which jar it is, given the number of chips on the cookie! Your 70% interval should simply include the list jars that, in total, have 70% probability of being the true jar. Isn't that a lot simpler and more intuitive?&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Sure, but how do we calculate that?&quot; I asked.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Let's say we &lt;strong&gt;know&lt;/strong&gt; that you got 3 chips. Then we can ignore all the other rows in the table, and simply treat that row as a probability mass function. We'll need to scale up the probabilities proportionately so each row sums to 100, though.&quot; She did:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/vvwDC.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Notice how each row is now a p.m.f., and sums to 100%. We've flipped the conditional probability from what you started with -- now it's the probability of the man having dropped off a certain jar, given the number of chips on the first cookie.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Interesting,&quot; I said. &quot;So now we just circle enough jars in each row to get up to 70% probability?&quot; We did just that, making these credibility intervals:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/u8qUn.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each interval includes a set of jars that, &lt;em&gt;a posteriori&lt;/em&gt;, sum to 70% probability of being the true jar.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Well, hang on,&quot; I said. &quot;I'm not convinced. Let's put the two kinds of intervals side-by-side and compare them for coverage and, assuming that the deliveryman picks each kind of jar with equal probability, credibility.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here they are:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Confidence intervals:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/MTLp7.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Credibility intervals:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/OUOrh.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;See how crazy your confidence intervals are?&quot; said Bayesia. &quot;You don't even have a sensible answer when you draw a cookie with zero chips! You just say it's the empty interval. But that's obviously wrong -- it has to be one of the four types of jars. How can you live with yourself, stating an interval at the end of the day when you &lt;strong&gt;know the interval is wrong?&lt;/strong&gt; And ditto when you pull a cookie with 3 chips -- your interval is only correct 59% of the time. Calling this a '70%' confidence interval is bullshit.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Well, hey,&quot; I replied. &quot;It provides 70% coverage no matter which jar it is. That's a lot more than you can say about your credibility intervals. What if the jar is type B? Then your interval will be wrong 40% of the time, and only correct 60% of the time. How can you call that a 70% interval if it's only right 60% of the time? PLUS we had to make this assumption that the deliveryman behaves uniformly and selects each type of jar at random. Where did that come from? What if it's wrong? You haven't talked to him; you haven't interviewed him. Yet all your statements of &lt;em&gt;a posteriori&lt;/em&gt; probability rest on this statement about his behavior. I didn't have to make any such assumptions, and my interval meets its criterion even in the worst case.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;It's true that my credibility interval does perform poorly on type-B jars,&quot; Bayesia said. &quot;But so what? Type B jars happen only 25% of the time. It's balanced out by my fantastic coverage of type A and D jars!&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;It's true that my confidence interval does perform poorly when I've drawn a cookie with zero chips,&quot; I said. &quot;But so what? Chipless cookies happen, at most, 27% of the time in the worst case (a type-D jar). I can afford to give nonsense for this outcome because no jar will result in a wrong answer more than 30% of the time.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;The column sums matter,&quot; I said.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;The row sums matter,&quot; Bayesia said.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;I can see we're at an impasse,&quot; I said. &quot;We're both correct in the mathematical statements we're making, but we disagree about the appropriate way to quantify uncertainty.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;That's true,&quot; said my sister. &quot;Want a cookie?&quot;&lt;/p&gt;&#xA;" OwnerUserId="1122" LastEditorUserId="1122" LastEditorDisplayName="" LastEditDate="2010-09-04T05:28:46.313" LastActivityDate="2010-09-04T05:28:46.313" CommentCount="4" />
  <row Id="2288" PostTypeId="2" ParentId="2169" CreationDate="2010-09-01T19:58:58.197" Score="0" ViewCount="" Body="&lt;p&gt;It looks like I am probably stuck with a bootstrap. One interesting possibility here is to compute the 'exact bootstrap covariance', as outlined by &lt;a href=&quot;http://onlinelibrary.wiley.com/doi/10.1111/1467-9868.00221/abstract&quot; rel=&quot;nofollow&quot;&gt;Hutson &amp;amp; Ernst&lt;/a&gt;. Presumably the bootstrap covariance gives a good estimate of the standard error, asymptotically. However, the approach of Hutson &amp;amp; Ernst requires computation of the covariance of each pair of order statistics, and so this method is quadratic in the number of samples. Maybe I should just stick with the bootstrap!&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-09-01T19:58:58.197" />
  <row Id="2289" PostTypeId="2" ParentId="2244" CreationDate="2010-09-01T20:42:40.850" Score="2" ViewCount="" Body="&lt;p&gt;In an article in &lt;em&gt;The American Statistician&lt;/em&gt;, Wolkewitz &lt;em&gt;et al.&lt;/em&gt; use packages &lt;em&gt;Epi&lt;/em&gt;, &lt;em&gt;mvna&lt;/em&gt;, and &lt;em&gt;survival&lt;/em&gt;.  See &lt;em&gt;Two Pitfalls in Survival Analyses of Time-Dependent Exposure: A Case Study in a Cohort of Oscar Nominees&lt;/em&gt;, v. 64 no. 3 (August 2010) pp 205-211.  This exposition introduces multistate survival models and focuses on the use of a &quot;Lexis diagram&quot; to assess possible forms of bias.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-09-01T20:42:40.850" />
  <row Id="2290" PostTypeId="1" AcceptedAnswerId="2310" CreationDate="2010-09-01T21:06:34.860" Score="5" ViewCount="666" Body="&lt;p&gt;This is a follow-up to the &lt;a href=&quot;http://stats.stackexchange.com/questions/1818/how-to-determine-the-sample-size-needed-for-repeated-measurement-anova&quot;&gt;repeated measures sample size&lt;/a&gt; question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am planning a repeated measures experiment. We record energy usage for 12 months, then give (a randomly assigned) half of the customers continuous information about their energy usage (perform the treatment), and record their energy usage for another 12 months. A similar study performed in the past showed a 5% reduction in energy usage.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to estimate the required sample size using $\alpha=0.05, \beta=0.1$. G*Power 3 has a tool for repeated measures power analysis. However, it requires two inputs that I am not entirely familiar with:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;$\lambda$ - the noncentrality parameter (How do I estimate this?)&lt;/li&gt;&#xA;&lt;li&gt;$f$ - the effect size (I believe that this is the square root of Cohen's $f^2$)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;According to Wikipedia's effect size page:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Cohen's $f^2= {R^2_{AB} - R^2_A \over 1 - R^2_{AB}}$ where $R^2_A$ is the variance accounted for by a set of one or more independent variables $A$, and $R^2_{AB}$ is the combined variance accounted for by $A$ and another set of one or more independent variables $B$.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;However, my expected 5% change in energy consumption does not tell me how much variability will be explained. Is there any way to make this conversion?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you know of a way to do this power analysis in R, I would love to hear it. I am planning to simulate some data and try using lmer from the lme4 package.&lt;/p&gt;&#xA;" OwnerUserId="743" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-09-02T12:15:16.680" LastActivityDate="2010-09-02T12:27:52.250" Title="Determination of effect size for a repeated measures ANOVA power analysis." Tags="&lt;r&gt;&lt;repeated-measures&gt;&lt;power-analysis&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="2291" PostTypeId="1" AcceptedAnswerId="2302" CreationDate="2010-09-01T23:57:06.760" Score="5" ViewCount="290" Body="&lt;p&gt;I'm working on a small (200M) corpus of text, which I want to explore with some cluster analysis. What books or articles on that subject would you recommend?  &lt;/p&gt;&#xA;" OwnerUserId="138" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-17T20:23:04.700" LastActivityDate="2010-10-05T23:09:37.403" CommunityOwnedDate="2010-09-02T09:35:37.073" Title="Recommended books or articles as introduction to Cluster Analysis?" Tags="&lt;machine-learning&gt;&lt;books&gt;&lt;clustering&gt;" AnswerCount="4" FavoriteCount="1" />
  <row Id="2292" PostTypeId="2" ParentId="2237" CreationDate="2010-09-02T00:30:48.977" Score="0" ViewCount="" Body="&lt;p&gt;The ets() function uses maximum likelihood estimation. So it would be possible to obtain standard errors based on the Hessian matrix in the usual way. However, in forecasting, the value of the model parameters is usually of very limited interest -- what we care about are the forecasts and their variances. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can't think of a situation where you might want a confidence interval for a smoothing parameter, for example. What could you do with the information that the &quot;true&quot; value of alpha (whatever that means) lies between 0.2 and 0.4? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consequently, I have not included the calculation of the standard errors of parameters in the package.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-09-02T00:30:48.977" CommentCount="3" />
  <row Id="2293" PostTypeId="2" ParentId="1906" CreationDate="2010-09-02T01:10:36.043" Score="3" ViewCount="" Body="&lt;p&gt;NIPS:  &lt;a href=&quot;http://nips.cc/&quot; rel=&quot;nofollow&quot;&gt;http://nips.cc/&lt;/a&gt;&lt;/p&gt;&#xA;" LastActivityDate="2010-09-02T01:10:36.043" CommunityOwnedDate="2010-09-02T01:10:36.043" CommentCount="1" />
  <row Id="2294" PostTypeId="1" CreationDate="2010-09-02T01:14:21.507" Score="2" ViewCount="83" Body="&lt;p&gt;I have a four-state, discrete time Markov process with time-dependent transition matrices such that after a given time T the matrices become constant. The idea is people in a program leaving the program in a variety of ways. Everyone starts in state 1, and states 2, 3 and 4 are absorbing, but state 4 represents the fairly small percentage of people who are 'lost in the system' - in other words state 4 represents our ignorance of what happens to people rather than a genuine outcome.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to use lumping to put those in state 4 in with those in state 1 and run this as a three-state system, and compare this with the naive approach of running this as a four-state system then apportioning those who are asymptotically in state 4 into states 2 and 3 according to their relative proportions. (in other words, p_2/(p_2 + p_3) of those in state 4 go into state 2 after the system is run to infinite time and similar for state 3)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From some rough scribblings it doesn't seem that these two methods give the same results, so it would be good to get an idea on the error involved. To this end, here's my question:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can I have pointers to the literature on lumping in Markov chains (or related) that would apply - even roughly - in this example? Or otherwise some words of advice on how to approach this.&lt;/p&gt;&#xA;" OwnerUserId="1144" LastActivityDate="2010-09-02T01:14:21.507" Title="Lumping in Markov process with absorbing states" Tags="&lt;modeling&gt;&lt;asymptotics&gt;&lt;markov-process&gt;" CommentCount="2" />
  <row Id="2296" PostTypeId="1" AcceptedAnswerId="2297" CreationDate="2010-09-02T02:00:45.543" Score="3" ViewCount="224" Body="&lt;p&gt;I am interested in fitting a factor analysis-like model on asset returns or other similar latent variable models. What are good papers to read on this topic? I am particularly interested in how to handle the fact that a factor analysis model is identical under a sign change for the &quot;factor loadings&quot;.&lt;/p&gt;&#xA;" OwnerUserId="1146" LastActivityDate="2011-02-03T20:06:33.490" Title="Papers on Bayesian factor analysis?" Tags="&lt;bayesian&gt;&lt;pca&gt;&lt;factor-analysis&gt;" AnswerCount="5" FavoriteCount="2" />
  <row Id="2297" PostTypeId="2" ParentId="2296" CreationDate="2010-09-02T02:19:37.283" Score="3" ViewCount="" Body="&lt;p&gt;Some references to help you out. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Tipping, M. E. &amp;amp; Bishop, C. M.&#xA;Probabilistic principal component&#xA;analysis Journal of the Royal&#xA;Statistical Society (Series B),&#xA;1999, 21, 611-622&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Tom Minka. Automatic choice of&#xA;dimensionality for PCA. NIPS 2000&#xA;url:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://research.microsoft.com/en-us/um/people/minka/papers/pca/&quot; rel=&quot;nofollow&quot;&gt;http://research.microsoft.com/en-us/um/people/minka/papers/pca/&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;Šmídl, V. &amp;amp; Quinn, A. On Bayesian&#xA;principal component analysis&#xA;Computational Statistics &amp;amp; Data&#xA;Analysis, 2007, 51, 4101-4123&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;If you are familiar with information theoretic model selection (MML, MDL, etc.), I highly recommend checking out:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Wallace, C. S. &amp;amp; Freeman, P. R.&#xA;Single-Factor Analysis by Minimum&#xA;Message Length Estimation Journal of&#xA;the Royal Statistical Society&#xA;(Series B), 1992, 54, 195-209&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;C. S. Wallace. Multiple Factor&#xA;Analysis by MML Estimation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.allisons.org/ll/Images/People/Wallace/Multi-Factor/&quot; rel=&quot;nofollow&quot;&gt;http://www.allisons.org/ll/Images/People/Wallace/Multi-Factor/&lt;/a&gt;&lt;br&gt;&#xA;Tech report:&#xA;&lt;a href=&quot;http://www.allisons.org/ll/Images/People/Wallace/Multi-Factor/TR95.218.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.allisons.org/ll/Images/People/Wallace/Multi-Factor/TR95.218.pdf&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="530" LastActivityDate="2010-09-02T02:19:37.283" CommentCount="1" />
  <row Id="2298" PostTypeId="1" AcceptedAnswerId="2319" CreationDate="2010-09-02T03:32:30.773" Score="5" ViewCount="169" Body="&lt;p&gt;This is somewhat vague, but suppose you have a black box function $f(x_1,x_2,\ldots,x_k)$, for which you have code, and you are interested in the behaviour of $f$ when the $x_i$ are i.i.d. standard Gaussian random variables. What are some good ways to visualize this function? To make it easier, we may assume that $k$ is smallish, say less than 10. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One particular relationship of interest is how $f$ varies with one of the input, say $x_i$. An easy way to visualize this relationship would be to sample the function for fixed values of $x_i$ while varying the other input (either in a structured way, or randomly, say), then box-plotting, which could show how the mean trend is affected by $x_i$, but also whether the scatter is affected (i.e. heteroskedasticity). However, interaction between $x_i$ and the levels of the other input might be masked by this approach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I am looking for is somewhat open-ended. I do not have a particular hypothesis that I am testing, but rather am looking for new ways of visualizing the response which might reveal peculiarities of the function.&lt;/p&gt;&#xA;" OwnerUserId="795" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-02T07:50:54.603" LastActivityDate="2010-09-02T21:56:32.277" CommunityOwnedDate="2010-09-02T07:18:38.477" Title="Visualization of a multivariate function" Tags="&lt;data-visualization&gt;&lt;computing&gt;" AnswerCount="3" CommentCount="9" FavoriteCount="1" />
  <row Id="2299" PostTypeId="1" AcceptedAnswerId="2303" CreationDate="2010-09-02T04:01:56.450" Score="15" ViewCount="217" Body="&lt;p&gt;What broad methods are there to detect fraud, anomalies, fudging, etc. in scientific works produced by a third party? (I was motivated to ask this by the recent &lt;a href=&quot;http://en.wikipedia.org/wiki/Marc_Hauser#Scientific_misconduct&quot; rel=&quot;nofollow&quot;&gt;Marc Hauser affair&lt;/a&gt;.) Usually for election and accounting fraud, some variant of &lt;a href=&quot;http://en.wikipedia.org/wiki/Benfords_law&quot; rel=&quot;nofollow&quot;&gt;Benford's Law&lt;/a&gt; is cited. I am not sure how this could be applied to &lt;em&gt;e.g.&lt;/em&gt; the Marc Hauser case, because Benford's Law requires numbers to be approximately log uniform. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a concrete example, suppose a paper cited the p-values for a large number of statistical tests. Could one transform these to log uniformity, then apply Benford's Law? It seems like there would be all kinds of problems with this approach (&lt;em&gt;e.g.&lt;/em&gt; some of the null hypotheses might legitimately be false, the statistical code might give p-values which are only approximately correct, the tests might only give p-values which are uniform under the null asymptotically, etc.)&lt;/p&gt;&#xA;" OwnerUserId="795" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-18T21:53:38.603" LastActivityDate="2010-09-26T22:53:23.677" Title="Statistical forensics: Benford and beyond." Tags="&lt;meta-analysis&gt;&lt;fraud&gt;" AnswerCount="2" FavoriteCount="4" />
  <row Id="2300" PostTypeId="2" ParentId="2296" CreationDate="2010-09-02T04:13:24.343" Score="1" ViewCount="" Body="&lt;p&gt;A decent overview of factor analysis is &lt;a href=&quot;http://www.wiley.com/WileyCDA/WileyTitle/productCd-0470711108.html&quot; rel=&quot;nofollow&quot;&gt;Latent Variable Methods and Factor Analysis&lt;/a&gt; by Bartholomew and Knott. They write about the interpretation of latent factors. This book is not as algorithmically-oriented as I would like, but their description of &lt;em&gt;e.g.&lt;/em&gt; partial factor analysis is decent.&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-09-02T04:13:24.343" CommentCount="1" />
  <row Id="2301" PostTypeId="2" ParentId="2291" CreationDate="2010-09-02T04:23:58.130" Score="1" ViewCount="" Body="&lt;p&gt;Cluster Analysis by Brian S. Everitt is a nice book length applied treatment of Cluster Analysis.&lt;/p&gt;&#xA;" OwnerUserId="485" LastActivityDate="2010-09-02T04:23:58.130" CommunityOwnedDate="2010-09-02T09:35:37.073" />
  <row Id="2302" PostTypeId="2" ParentId="2291" CreationDate="2010-09-02T05:24:13.440" Score="2" ViewCount="" Body="&lt;p&gt;This chapter of &lt;a HREF=&quot;http://www-users.cs.umn.edu/~kumar/dmbook/ch8.pdf&quot; rel=&quot;nofollow&quot;&gt;Introduction to Data Mining&lt;/a&gt; is available online and gives a nice overview. &lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-09-02T05:24:13.440" CommunityOwnedDate="2010-09-02T09:35:37.073" />
  <row Id="2303" PostTypeId="2" ParentId="2299" CreationDate="2010-09-02T05:45:28.647" Score="10" ViewCount="" Body="&lt;p&gt;Great Question!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the scientific context there are various kinds of problematic reporting and problematic behaviour:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Fraud&lt;/strong&gt;: I'd define fraud as a deliberate intention on the part of the author or analyst to misrepresent the results and where the misrepresentation is of a sufficiently grave nature. The main example being complete fabrication of raw data or summary statistics.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Error&lt;/strong&gt;: Data analysts can make errors at many phases of data analysis from data entry, to data manipulation, to analyses, to reporting, to interpretation.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Inappropriate behaviour&lt;/strong&gt;: There are many forms of inappropriate behaviour. In general, it can be summarised by an orientation which seeks to confirm a particular position rather than search for the truth. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Common examples of inappropriate behaviour include: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Examining a series of possible dependent variables and only reporting the one that is statistically significant&lt;/li&gt;&#xA;&lt;li&gt;Not mentioning important violations of assumptions&lt;/li&gt;&#xA;&lt;li&gt;Performing data manipulations and outlier removal procedures without mentioning it, particularly where these procedures are both inappropriate and chosen purely to make the results look better&lt;/li&gt;&#xA;&lt;li&gt;Presenting a model as confirmatory which is actually exploratory&lt;/li&gt;&#xA;&lt;li&gt;Omitting important results that go against the desired argument&lt;/li&gt;&#xA;&lt;li&gt;Choosing a statistical test solely on the basis that it makes the results look better&lt;/li&gt;&#xA;&lt;li&gt;Running a series of five or ten under-powered studies where only one is statistically significant (perhaps at p = .04) and then reporting the study without mention of the other studies&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In general, I'd hypothesise that &lt;strong&gt;incompetence&lt;/strong&gt; is related to all three forms of problematic behaviour. A researcher who does not understand how to do good science but otherwise wants to be successful will have a greater incentive to misrepresent their results, and is less likely to respect the principles of ethical data analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The above distinctions have implications for detection of problematic behaviour.&#xA;For example, if you manage to discern that a set of reported results are wrong, it still needs to be ascertained as to whether the results arose from fraud, error or inappropriate behaviour. Also, I'd assume that various forms of inappropriate behaviour are far more common than fraud.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With regards to detecting problematic behaviour, I think it is largely a skill that comes from experience working with &lt;strong&gt;data&lt;/strong&gt;, working with a &lt;strong&gt;topic&lt;/strong&gt;, and working with &lt;strong&gt;researchers&lt;/strong&gt;. All of these experiences strengthen your expectations about what data should look like. Thus, major deviations from expectations start the process of searching for an explanation. Experience with researchers gives you a sense of the kinds of inappropriate behaviour which are more or less common. In combination this leads to the generation of hypotheses. For example, if I read a journal article and I am surprised with the results, the study is underpowered, and the nature of the writing suggests that the author is set on making a point, I generate the hypothesis that the results perhaps should not be trusted.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Other Resources&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Robert P. Abelson &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0805805281&quot; rel=&quot;nofollow&quot;&gt;Statistics as a Principled Argument&lt;/a&gt; has a chapter titled &quot;On Suspecting Fishiness&quot; &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-09-02T05:45:28.647" />
  <row Id="2304" PostTypeId="2" ParentId="2298" CreationDate="2010-09-02T06:33:11.917" Score="3" ViewCount="" Body="&lt;p&gt;Just a thought, although I've never tried it.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;you could obtain a large number of values from the function across different parameter values&lt;/li&gt;&#xA;&lt;li&gt;take a tour of the resulting data in &lt;a href=&quot;http://www.ggobi.org/&quot; rel=&quot;nofollow&quot;&gt;ggobi&lt;/a&gt; (check out &lt;a href=&quot;http://vimeo.com/12292239&quot; rel=&quot;nofollow&quot;&gt;Mat Kelcey's video&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-09-02T06:33:11.917" CommunityOwnedDate="2010-09-02T07:18:38.477" CommentCount="3" />
  <row Id="2305" PostTypeId="2" ParentId="2291" CreationDate="2010-09-02T10:25:32.673" Score="4" ViewCount="" Body="&lt;p&gt;It may be worth looking at M.W. Berry's books: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;em&gt;Survey of Text Mining I: Clustering, Classification, and Retrieval&lt;/em&gt; (2003)&lt;/li&gt;&#xA;&lt;li&gt;&lt;em&gt;Survey of Text Mining II: Clustering, Classification, and Retrieval&lt;/em&gt; (2008)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;They consist of series of applied and review papers. The latest seems to be available as PDF at the following address: &lt;a href=&quot;http://bit.ly/deNeiy&quot; rel=&quot;nofollow&quot;&gt;http://bit.ly/deNeiy&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are few links related to CA as applied to text mining: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.cscjournals.org/csc/manuscript/Journals/IJCSS/volume4/Issue2/IJCSS-271.pdf&quot; rel=&quot;nofollow&quot;&gt;Document Topic Generation in Text Mining by Using Cluster Analysis with EROCK&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.google.fr/url?sa=t&amp;amp;source=web&amp;amp;cd=8&amp;amp;ved=0CFoQFjAH&amp;amp;url=http%3A%2F%2Fciteseerx.ist.psu.edu%2Fviewdoc%2Fdownload%3Fdoi%3D10.1.1.21.8862%26rep%3Drep1%26type%3Dpdf&amp;amp;ei=P3h_TJXfAdT34AbIuN3TCw&amp;amp;usg=AFQjCNHaVKyo45HtjNKiEPvFnPuvD2MvmQ&amp;amp;sig2=vPQ1BmIz8vb16q1Oysi8_g&quot; rel=&quot;nofollow&quot;&gt;An Approach to Text Mining using Information Extraction&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You can also look at &lt;em&gt;Latent Semantic Analysis&lt;/em&gt;, but see my response there: &lt;a href=&quot;http://stats.stackexchange.com/questions/369/working-through-a-clustering-problem/2196#2196&quot;&gt;Working through a clustering problem&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-09-02T10:25:32.673" CommunityOwnedDate="2010-09-02T10:25:32.673" />
  <row Id="2306" PostTypeId="1" AcceptedAnswerId="2307" CreationDate="2010-09-02T10:25:42.330" Score="10" ViewCount="535" Body="&lt;p&gt;I am getting a bit confused about feature selection and machine learning&#xA;and I was wondering if you could help me out.  I have a microarray dataset that is&#xA;classified into two groups and has 1000s of features.  My aim is to get a small number of genes (my features) (10-20) in a signature that I will in theory be able to apply to&#xA;other datasets to optimally classify those samples.  As I do not have that many samples (&amp;lt;100), I am not using a test and training set but using Leave-one-out cross-validation to help&#xA;determine the robustness.  I have read that one should perform feature selection for each split of the samples i.e.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Select one sample as the test set&lt;/li&gt;&#xA;&lt;li&gt;On the remaining samples perform feature selection&lt;/li&gt;&#xA;&lt;li&gt;Apply machine learning algorithm to remaining samples using the features selected&lt;/li&gt;&#xA;&lt;li&gt;Test whether the test set is correctly classified&lt;/li&gt;&#xA;&lt;li&gt;Go to 1.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;If you do this, you might get different genes each time, so how do you&#xA;get your &quot;final&quot; optimal gene classifier? i.e. what is step 6.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I mean by optimal is the collection of genes that any further studies&#xA;should use.  For example, say I have a cancer/normal dataset and I want&#xA;to find the top 10 genes that will classify the tumour type according to&#xA;an SVM.  I would like to know the set of genes plus SVM parameters that&#xA;could be used in further experiments to see if it could be used as a&#xA;diagnostic test.&lt;/p&gt;&#xA;" OwnerUserId="1150" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-10T17:31:47.207" LastActivityDate="2010-11-06T20:49:43.557" Title="Feature selection for &quot;final&quot; model when performing cross-validation in machine learning" Tags="&lt;machine-learning&gt;&lt;classification&gt;&lt;feature-selection&gt;&lt;genetics&gt;" AnswerCount="5" CommentCount="2" FavoriteCount="7" />
  <row Id="2307" PostTypeId="2" ParentId="2306" CreationDate="2010-09-02T10:46:12.320" Score="8" ViewCount="" Body="&lt;p&gt;This is a very good question that I faced myself when working with SNPs data... And I didn't find any obvious answer through the literature. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Whether you use LOO or K-fold CV, you'll end up with different features since the cross-validation iteration must be the most outer loop, as you said. You can think of some kind of voting scheme which would rate the n-vectors of features you got from your LOO-CV (can't remember the paper but it is worth checking the work of &lt;a href=&quot;http://www.imbi.uni-freiburg.de/biom/index.php?showEmployee=binderh&quot; rel=&quot;nofollow&quot;&gt;Harald Binder&lt;/a&gt; or &lt;a href=&quot;http://www.lri.fr/~antoine/Papers/papers.html&quot; rel=&quot;nofollow&quot;&gt;Antoine Cornuéjols&lt;/a&gt;). In the absence of a new test sample, what is usually done is to re-apply the ML algorithm to the whole sample once you have found its optimal cross-validated parameters. But proceeding this way, you cannot ensure that there is no overfitting (since the sample was already used for model optimization).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or, alternatively, you can use embedded methods which provide you with features ranking through a measure of variable importance, e.g. like in &lt;a href=&quot;http://www.stat.berkeley.edu/~breiman/RandomForests/&quot; rel=&quot;nofollow&quot;&gt;Random Forests&lt;/a&gt; (RF). As cross-validation is included in RFs, you don't have to worry about the $n\ll p$ case or curse of dimensionality. Here are nice papers of their applications in gene expression studies:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Cutler, A., Cutler, D.R., and Stevens, J.R. (2009). Tree-Based Methods, in &lt;em&gt;High-Dimensional Data Analysis in Cancer Research&lt;/em&gt;, Li, X. and Xu, R. (eds.), pp. 83-101, Springer.&lt;/li&gt;&#xA;&lt;li&gt;Saeys, Y., Inza, I., and Larrañaga, P. (2007). A review of feature selection techniques in bioinformatics. &lt;em&gt;Bioinformatics&lt;/em&gt;, &lt;strong&gt;23(19)&lt;/strong&gt;: 2507-2517.&lt;/li&gt;&#xA;&lt;li&gt;Díaz-Uriarte, R., Alvarez de Andrés, S. (2006). Gene selection and classification of microarray data using random forest. &lt;em&gt;BMC Bioinformatics&lt;/em&gt;, &lt;strong&gt;7&lt;/strong&gt;:3.&lt;/li&gt;&#xA;&lt;li&gt;Diaz-Uriarte, R. (2007). GeneSrF and varSelRF: a web-based tool and R package for gene selection and classification using random forest. &lt;em&gt;BMC Bioinformatics&lt;/em&gt;, &lt;strong&gt;8&lt;/strong&gt;: 328&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Since you are talking of SVM, you can look for &lt;em&gt;penalized SVM&lt;/em&gt;.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-06T20:49:43.557" LastActivityDate="2010-11-06T20:49:43.557" CommentCount="4" />
  <row Id="2308" PostTypeId="2" ParentId="1980" CreationDate="2010-09-02T11:15:56.443" Score="4" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://bioinformatics.mdanderson.org/Supplements/ReproRsch-Chemo/&quot; rel=&quot;nofollow&quot;&gt;Irreproducibility of NCI60 Predictors of Chemotherapy&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a reproducible analysis showing the lack of reproducibility of a paper that has been in the news.  A clinical trial based on the false conclusions of the irreproducible paper was suspended, re-instated, suspended again, ... It's a good example of reproducible analysis in the news.  &lt;/p&gt;&#xA;" OwnerUserId="319" LastEditorUserId="319" LastEditorDisplayName="" LastEditDate="2010-09-02T14:57:59.267" LastActivityDate="2010-09-02T14:57:59.267" />
  <row Id="2309" PostTypeId="2" ParentId="2306" CreationDate="2010-09-02T11:29:47.973" Score="4" ViewCount="" Body="&lt;p&gt;To add to chl: When using support vector machines, a highly recommended penalization method is the elastic net. This method will shrink coefficients towards zero, and in theory retains the most stable coefficients in the model. Initially it was used in a regression framework, but it is easily extended for use with support vector machines.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.math.mtu.edu/~shuzhang/MA5761/model_selection2.pdf&quot; rel=&quot;nofollow&quot;&gt;The original publication&lt;/a&gt; : Zou and Hastie (2005) : Regularization and variable selection via the elastic net. J.R.Statist.Soc. B, 67-2,pp.301-320&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.springerlink.com/content/517605v67gg75338/fulltext.pdf&quot; rel=&quot;nofollow&quot;&gt;Elastic net for SVM&lt;/a&gt; : Zhu &amp;amp; Zou (2007): Variable Selection for the Support Vector Machine : Trends in Neural Computation, chapter 2 (Editors: Chen and Wang) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.sciencedirect.com/science?_ob=ArticleURL&amp;amp;_udi=B8H0V-50M1N7Y-2&amp;amp;_user=794998&amp;amp;_coverDate=07%2F31%2F2010&amp;amp;_rdoc=1&amp;amp;_fmt=high&amp;amp;_orig=search&amp;amp;_origin=search&amp;amp;_sort=d&amp;amp;_docanchor=&amp;amp;view=c&amp;amp;_searchStrId=1448587725&amp;amp;_rerunOrigin=google&amp;amp;_acct=C000043466&amp;amp;_version=1&amp;amp;_urlVersion=0&amp;amp;_userid=794998&amp;amp;md5=7e56402b32c944bc33c9373013f60ae8&amp;amp;searchtype=a&quot; rel=&quot;nofollow&quot;&gt;improvements on the elastic net&lt;/a&gt; Jun-Tao and Ying-Min(2010): An Improved Elastic Net for Cancer Classification and Gene Selection : Acta Automatica Sinica, 36-7,pp.976-981&lt;/p&gt;&#xA;" OwnerUserId="1124" LastActivityDate="2010-09-02T11:29:47.973" />
  <row Id="2310" PostTypeId="2" ParentId="2290" CreationDate="2010-09-02T12:27:52.250" Score="1" ViewCount="" Body="&lt;p&gt;Assuming you are going to average the first 12 months to form a baseline measure and the second 12 months to form as a follow-up measure, your problem reduces to a repeated measures t-test.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;G*Power&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might want to check out the following menu in G*Power 3:&#xA;&lt;code&gt;Tests - Means - Two Dependent Groups (matched pairs)&lt;/code&gt;.&#xA; Use A priori, $\alpha=.05$, Power = 0.90.&#xA;Use the &lt;code&gt;Determine&lt;/code&gt; button to determine effect size. This requires that you can estimate  time 1 and 2 means, sds, and correlation between time points.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;R&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.ats.ucla.edu/stat/R/dae/t_test_power3.htm&quot; rel=&quot;nofollow&quot;&gt;UCLA has a tutorial&lt;/a&gt; on doing a power analysis on a repeated measures t-test using R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Side point&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a side point, you might want to consider having a control group.&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-09-02T12:27:52.250" />
  <row Id="2311" PostTypeId="2" ParentId="2271" CreationDate="2010-09-02T12:46:29.553" Score="8" ViewCount="" Body="&lt;p&gt;Make.Funny.Plot does more or less what I think it should do. To be adapted according to your own needs, and might be optimized a bit, but this should be a nice start.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Make.Funny.Plot &amp;lt;- function(x){&#xA;    unique.vals &amp;lt;- length(unique(x))&#xA;    N &amp;lt;- length(x)&#xA;    N.val &amp;lt;- min(N/20,unique.vals)&#xA;&#xA;    if(unique.vals&amp;gt;N.val){&#xA;      x &amp;lt;- ave(x,cut(x,N.val),FUN=min)&#xA;      x &amp;lt;- signif(x,4)&#xA;    }&#xA;    # construct the outline of the plot&#xA;    outline &amp;lt;- as.vector(table(x))&#xA;    outline &amp;lt;- outline/max(outline)&#xA;&#xA;    # determine some correction to make the V shape,&#xA;    # based on the range&#xA;    y.corr &amp;lt;- diff(range(x))*0.05&#xA;&#xA;    # Get the unique values&#xA;    yval &amp;lt;- sort(unique(x))&#xA;&#xA;    plot(c(-1,1),c(min(yval),max(yval)),&#xA;        type=&quot;n&quot;,xaxt=&quot;n&quot;,xlab=&quot;&quot;)&#xA;&#xA;    for(i in 1:length(yval)){&#xA;        n &amp;lt;- sum(x==yval[i])&#xA;        x.plot &amp;lt;- seq(-outline[i],outline[i],length=n)&#xA;        y.plot &amp;lt;- yval[i]+abs(x.plot)*y.corr&#xA;        points(x.plot,y.plot,pch=19,cex=0.5)&#xA;    }&#xA;}&#xA;&#xA;N &amp;lt;- 500&#xA;x &amp;lt;- rpois(N,4)+abs(rnorm(N))&#xA;Make.Funny.Plot(x)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;EDIT : corrected so it always works.&lt;/p&gt;&#xA;" OwnerUserId="1124" LastEditorUserId="1124" LastEditorDisplayName="" LastEditDate="2010-09-02T13:17:51.240" LastActivityDate="2010-09-02T13:17:51.240" CommentCount="10" />
  <row Id="2312" PostTypeId="2" ParentId="2291" CreationDate="2010-09-02T13:10:52.193" Score="1" ViewCount="" Body="&lt;p&gt;Not specifically about text-mining, but I quite liked &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1584883669&quot; rel=&quot;nofollow&quot;&gt;&quot;Exploratory Data Analysis with MATLAB&quot;&lt;/a&gt; by Martinez and Martinez.&lt;/p&gt;&#xA;" OwnerUserId="582" LastActivityDate="2010-09-02T13:10:52.193" CommunityOwnedDate="2010-09-02T13:10:52.193" />
  <row Id="2313" PostTypeId="2" ParentId="2271" CreationDate="2010-09-02T14:05:11.540" Score="4" ViewCount="" Body="&lt;p&gt;I have come up with the code similar to Joris, still I think this is more than a stem plot; here I mean that they y value in each series is a absolute value of a distance to the in-bin mean, and x value is more about whether the value is lower or higher than mean.&lt;br&gt;&#xA;Example code (sometimes throws warnings but works):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;px&amp;lt;-function(x,N=40,...){&#xA;x&amp;lt;-sort(x);&#xA;&#xA;#Cutting in bins&#xA;cut(x,N)-&amp;gt;p;&#xA;&#xA;#Calculate the means over bins&#xA;sapply(levels(p),function(i) mean(x[p==i]))-&amp;gt;meansl;&#xA;means&amp;lt;-meansl[p];&#xA;&#xA;#Calculate the mins over bins&#xA;sapply(levels(p),function(i) min(x[p==i]))-&amp;gt;minl;&#xA;mins&amp;lt;-minl[p];&#xA;&#xA;#Each dot is one value.&#xA;#X is an order of a value inside bin, moved so that the values lower than bin mean go below 0&#xA;X&amp;lt;-rep(0,length(x));&#xA;for(e in levels(p)) X[p==e]&amp;lt;-(1:sum(p==e))-1-sum((x-means)[p==e]&amp;lt;0);&#xA;#Y is a bin minum + absolute value of a difference between value and its bin mean&#xA;plot(X,mins+abs(x-means),pch=19,cex=0.5,...);&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-09-02T14:05:11.540" CommentCount="2" />
  <row Id="2314" PostTypeId="2" ParentId="2306" CreationDate="2010-09-02T15:56:37.563" Score="2" ViewCount="" Body="&lt;p&gt;As step 6 (or 0) you run the feature detection algorithm on the entire data set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The logic is the following: you have to think of cross-validation as a method for finding out the properties of the procedure you are using to select the features. It answers the question: &quot;if I have some data and perform this procedure, then what is the error rate for classifying a new sample?&quot;. Once you know the answer, you can use the procedure (feature selection + classification rule development) on the entire data set. People like leave-one-out because the predictive properties usually depend on the sample size, and $n-1$ is usually close enough to $n$ not to matter much.&lt;/p&gt;&#xA;" OwnerUserId="279" LastActivityDate="2010-09-02T15:56:37.563" CommentCount="4" />
  <row Id="2315" PostTypeId="1" CreationDate="2010-09-02T16:12:52.183" Score="3" ViewCount="186" Body="&lt;p&gt;A previous user asked &lt;a href=&quot;http://stats.stackexchange.com/questions/1266/a-non-parametric-repeated-measures-multi-way-anova-in-r&quot;&gt;this question&lt;/a&gt; specifically for R.  I'd like to know what, if any, other software can do this.&lt;/p&gt;&#xA;" OwnerUserId="132" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-09-03T08:10:45.787" LastActivityDate="2010-09-03T17:53:09.490" Title="What software allows non-parametric repeated-measures multi-way Anova?" Tags="&lt;anova&gt;&lt;nonparametric&gt;&lt;software&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="2316" PostTypeId="2" ParentId="2264" CreationDate="2010-09-02T16:58:19.700" Score="0" ViewCount="" Body="&lt;p&gt;Ha! I've answered my own question. Simon Funk figured this out for the Netflix challenge &lt;a href=&quot;http://sifter.org/~simon/journal/20061211.html&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;. See the paragraph commencing &quot;However, even this isn't quite as simple as it appears&quot;. But I'm having difficulty proving it algebraically: maybe you guys would like to take that on. &lt;/p&gt;&#xA;" OwnerUserId="1134" LastActivityDate="2010-09-02T16:58:19.700" CommentCount="7" />
  <row Id="2317" PostTypeId="2" ParentId="2306" CreationDate="2010-09-02T17:53:16.097" Score="5" ViewCount="" Body="&lt;p&gt;In principle:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Make your predictions using a single model trained on the entire dataset (so there is only one set of features).  The cross-validation is only used to estimate the predictive performance of the single model trained on the whole datset.  It is VITAL in using cross-validation that in each fold you repeat the entire procedure used to fit the primary model, as otherwise you can end up with a substantial optimistic bias in performance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To see why this happens, consider a binary classification problem with 1000 binary features but only 100 patterns, where the targets and features are all purely random, so there is no statistical relationship between the inputs and the targets whatsoever.  If we train a primary model on the full dataset, we can always achieve zero error on the training set as there are more features than patterns.  We can even find a subset of &quot;informative&quot; features (that happen to be correllated to the target by chance).  I we then adopt the naughty approach and perform cross-validation using only those features, we will get an estimate of performance that is better than random guessing.  The reason is that in each fold of the cross-validation procedure there is some information about the held-out patterns used for testing as the features were chosen because they were good for predicting the targets, all of them, including those held out.  Of course the true error rate will be 0.5.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we adopt the proper procedure, and perform feature selection in each fold, there is no longer any information about the target of the held out pattern in the choice of features used in that fold.  If you use the proper procedure, in this case, you will get an error rate of about 0.5 (although it will vary a bit for difference realisations of the dataset).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good papers to read are:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Christophe Ambroise, Geoffrey J. McLachlan, &quot;Selection bias in gene extraction on the basis of microarray gene-expression data&quot;, PNAS &lt;a href=&quot;http://www.pnas.org/content/99/10/6562.abstract&quot; rel=&quot;nofollow&quot;&gt;http://www.pnas.org/content/99/10/6562.abstract&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;which is highly relevant to the OP and&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Gavin C. Cawley, Nicola L. C. Talbot, &quot;On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation&quot;, JMLR 11(Jul):2079−2107, 2010 &lt;a href=&quot;http://jmlr.csail.mit.edu/papers/v11/cawley10a.html&quot; rel=&quot;nofollow&quot;&gt;http://jmlr.csail.mit.edu/papers/v11/cawley10a.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;which demonstrates that the same thing can easily ocurr in model selection (e.g. tuning the hyper-parameters of an SVM, which also need to be repeated in each iteration of the CV procedure).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In practice:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would recommend using Bagging, and using the out-of-bag error for estimating performance.    You will get a committee model using many features, but that is actually a good thing.  If you only use a single model, it will be likely that you will over-fit the feature selection criterion, and end up with a model that gives poorer predictions than a model that uses a larger number of features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alan Millers book on subset selection in regression (Chapman and Hall monographs on statistics and applied probability, volume 95) gives the good bit of advice (page 221) that if predictive performance is the most important thing, then don't do any feature selection, just use ridge regression instead.  And that is in a book on subset selection!!! ;o)&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2010-09-02T17:53:16.097" />
  <row Id="2318" PostTypeId="2" ParentId="2179" CreationDate="2010-09-02T18:07:50.913" Score="2" ViewCount="" Body="&lt;p&gt;Isabelle Guyon, André Elisseeff, &quot;An Introduction to Variable and Feature Selection&quot;, JMLR, 3(Mar):1157-1182, 2003. &lt;a href=&quot;http://jmlr.csail.mit.edu/papers/v3/guyon03a.html&quot; rel=&quot;nofollow&quot;&gt;http://jmlr.csail.mit.edu/papers/v3/guyon03a.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;is well worth reading, it will give a good overview of approaches and issues.  The one thing I would add is that feature selection doesn't necessarily improve predictive performance, and can easily make it worse (beacuse it is easy to over-fit the feature selection criterion).  One of the advantages of (especially linear) SVMs is that they work well with large numbers of features (providing you tune the regularisation parameter properly), so there is often no need if you are only interested in prediction.&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2010-09-02T18:07:50.913" CommentCount="3" />
  <row Id="2319" PostTypeId="2" ParentId="2298" CreationDate="2010-09-02T18:56:34.177" Score="2" ViewCount="" Body="&lt;p&gt;Given that you are at the initial, exploratory stages of the analysis I would start simple. Consider sampling your inputs using a &lt;a href=&quot;http://en.wikipedia.org/wiki/Latin_hypercube_sampling&quot; rel=&quot;nofollow&quot;&gt;Latin Hypercube&lt;/a&gt; strategy. Then, a tornado chart can be used to get a quick assessment of the multiple,one-way sensitivities f() has to the various input variables. Here is an example chart (from &lt;a href=&quot;http://www.add-ins.com/sensitivity_chart_creator.htm&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/DQCby.gif&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This chart is not that interesting, but an interpretation would be &quot;NPV is most sensitive to Shipments, all other things being equal. But, the sensitivity is mostly on the upside, which is good. The Escalation variable induces sensitivity into NPV, but what looks to be skewed negatively a bit...&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could do something similar for Mean(f) on the X-axis as well as Var(f)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given what you find from some first glance visualizations like this, you could then slice and dice more and focus on specific variables or relationships between variables. Maybe you can revisit this thread in coming months and post the visualizations you found useful :)&lt;/p&gt;&#xA;" OwnerUserId="1080" LastActivityDate="2010-09-02T18:56:34.177" CommunityOwnedDate="2010-09-02T18:56:34.177" />
  <row Id="2320" PostTypeId="2" ParentId="2315" CreationDate="2010-09-02T19:22:14.867" Score="1" ViewCount="" Body="&lt;p&gt;This question was updated with a link to the previous question, at which point I realized that my response originally posted here pointing to the ez package in R was better left at the previous question.&lt;/p&gt;&#xA;" OwnerUserId="364" LastEditorUserId="364" LastEditorDisplayName="" LastEditDate="2010-09-03T17:53:09.490" LastActivityDate="2010-09-03T17:53:09.490" />
  <row Id="2321" PostTypeId="2" ParentId="2298" CreationDate="2010-09-02T21:56:32.277" Score="0" ViewCount="" Body="&lt;p&gt;You could apply some sort of dimensionality reduction technique like principal components and plot the value of the function as you vary the first, second, third etc. principal components, holding all others fixed.  This would show you how the function varies in the directions of the maximal variance of the inputs.&lt;/p&gt;&#xA;" LastActivityDate="2010-09-02T21:56:32.277" CommunityOwnedDate="2010-09-02T21:56:32.277" CommentCount="1" />
  <row Id="2322" PostTypeId="2" ParentId="2306" CreationDate="2010-09-02T23:05:55.530" Score="0" ViewCount="" Body="&lt;p&gt;I'm not sure about classification problems, but in the case of feature selection for regression problems, Jun Shao showed &lt;a href=&quot;http://www.jstor.org/pss/2290328&quot; rel=&quot;nofollow&quot;&gt;that Leave-One-Out CV is asymptotically inconsistent&lt;/a&gt;, i.e. the probability of selecting the proper subset of features does not converge to 1 as the number of samples increases. From a practical point of view, Shao recommends a Monte-Carlo cross-validation, or leave-many-out procedure.&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-09-02T23:05:55.530" CommentCount="2" />
  <row Id="2323" PostTypeId="1" CreationDate="2010-09-03T00:34:44.053" Score="1" ViewCount="51" Body="&lt;p&gt;I think that dynamic pricing algorithms (used in aviation and ticketing industry) is very statistical based, anyone here has experience with those algorithms with references for it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regards&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Camilo&lt;/p&gt;&#xA;" OwnerUserId="1167" LastActivityDate="2010-09-03T02:58:41.277" Title="What are good references for dynamic pricing?" Tags="&lt;time-series&gt;" AnswerCount="1" FavoriteCount="2" />
  <row Id="2324" PostTypeId="2" ParentId="2323" CreationDate="2010-09-03T02:15:18.283" Score="1" ViewCount="" Body="&lt;p&gt;This article is highly cited:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Yield Management at American Airlines&quot; (&lt;a href=&quot;http://www2.haut.edu.cn/jpkc2006/orsite/c/case/c28.pdf&quot; rel=&quot;nofollow&quot;&gt;Free PDF&lt;/a&gt; or  &lt;a href=&quot;http://www.jstor.org/stable/25061571&quot; rel=&quot;nofollow&quot;&gt;JSTOR&lt;/a&gt;)&lt;/p&gt;&#xA;" OwnerUserId="74" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-09-03T02:58:41.277" LastActivityDate="2010-09-03T02:58:41.277" CommentCount="1" />
  <row Id="2325" PostTypeId="2" ParentId="1432" CreationDate="2010-09-03T02:27:00.000" Score="2" ViewCount="" Body="&lt;h2&gt;On Pearsons residuals,&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;The Pearson residual is the difference between the observed and estimated probabilities divided by the binomial standard deviation of the estimated probability. Therefore standardizing the residuals.&#xA;For large samples the standardized residuals should have a normal distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From Menard, Scott (2002). Applied logistic regression analysis, 2nd Edition. Thousand Oaks, CA: Sage Publications. Series: Quantitative Applications in the Social Sciences, No. 106. First ed., 1995. See Chapter 4.4&lt;/p&gt;&#xA;" OwnerUserId="694" LastActivityDate="2010-09-03T02:27:00.000" />
  <row Id="2326" PostTypeId="1" AcceptedAnswerId="2366" CreationDate="2010-09-03T03:25:40.293" Score="1" ViewCount="138" Body="&lt;p&gt;Hello,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am looking at setting up an experiment concerning a hobby of mine, basically measuring a variety of parameters 'before' and 'after' and see which one, if any, gives the most reliable prediction of a final parameter i.e. do they have a linear relationship, etc.  The object being to save some time and effort later &lt;em&gt;not&lt;/em&gt; sorting by the parameters that have little actual bearing on the final results, and to see if some of the more tedious sorting methods are actually as useful as thought.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The experiment is an extension of one I did for my final paper in my stats class a couple years ago:  investigating the relationship between case weight &amp;amp; volume vs. muzzle velocity for centerfire rifle cases as used in modern target competition (in which I'm fairly active).  For that paper I had to spend a fair amount of 'time' demonstrating various methods that didn't really pertain to what I wanted to know, but had to cover anyway to get an 'A' ;)  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For this go-around... I'm looking at taking one hundred pieces of brass cases from one box, one lot.  These are the 'non-consumable' parts, as they can and do get re-used and reloaded multiple times.  Everything else - bullet, powder, primer - get shot down the barrel and cannot be re-used/re-measured.  Specifically, the cases can and do become 'fire-formed' to the interior dimensions of the chamber under extreme pressure (50-70k psi).  In my original experiment, I found a significant, but not especially strong, correlation between the initial case weight and the muzzle velocity... of the first firing.  Case &lt;em&gt;volume&lt;/em&gt;, on the other hand, which should be very strongly related to MV... was not, theoretically because the 'virgin' cases don't necessarily 'fit' the internal dimensions of the chamber and a certain amount of the energy generated is expended not just in heat, but in squashing the brass against the case wall during firing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So again, I'm looking at taking 100pcs of brass, weighing them straight out of the box, taking various measurements (case wall thickness inside near the case web, case neck thickness near the mouth, weight, volume, etc.) in their 'virgin' untouched state, then performing several routine case prep steps (trim to length, chamfer/debur the mouth, debur the flash hole, uniform primer pocket) and repeat the measurements.  Then load and fire the rounds as uniformly as possible - bullets sorted for maximum consistency, powder weighed on a milligram-capable analytic lab scale, etc. while controlling the rate of fire so as to regulate temperature rise along the barrel and measuring the muzzle velocity via a chronograph approx. 15 feet down range.  Then I plan to completely clean the cases inside and out to remove any buildup of carbon or powder residue, and repeat the measurements and then load and fire again and then clean and measure one last time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Part of what I want to see is how the distributions of some of the measurements change as the cases are prepped, then again as they are fired and formed to the chamber.  After that... I want to see how much difference is there &lt;em&gt;really&lt;/em&gt; between 'virgin' cases and fire-formed cases in terms of muzzle velocity, and finally... of all the tedious measurement steps mentioned above, which ones actually give a reasonable indication of consistent MV so I can successfully cull suspect pieces of brass during preliminary sorting rather than suffer lost points when using them on target in competition.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The single biggest problem in my mind, at least on the surface, is the chronograph.  Getting anything approaching credible numbers for accuracy or % error from the vendors is somewhere between difficult and impossible.  Given the nature of the device, it is extremely difficult to test on a consumer level - every round through it may (or may not) be just slightly different, so determining how much of the variation displayed on its screen is really the ammunition and how much is the variability of the instrument itself... has me scratching my head, to say the least.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As to why I was kind of vague before about the details of the experiment... well, sometimes people get kinda weird as soon as they realize something involves GUNS and won't touch it with a 10 ft pole regardless of how reasonable it may be.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Monte&lt;/p&gt;&#xA;" OwnerUserId="1114" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-16T06:35:12.133" LastActivityDate="2010-09-16T06:35:12.133" Title="Setting up experiment for statistical analysis" Tags="&lt;r&gt;&lt;experiment-design&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="2327" PostTypeId="2" ParentId="1126" CreationDate="2010-09-03T03:49:48.030" Score="1" ViewCount="" Body="&lt;p&gt;In my opinion 16 are too many reasons, too fine of a specification and sort of overlap at times. Instead I would personally streamline into  broad groups. We can classify study objectives in 3 main categories: single hypothesis testing, exploratory study and to predict.&lt;/p&gt;&#xA;" OwnerUserId="694" LastActivityDate="2010-09-03T03:49:48.030" CommunityOwnedDate="2010-09-03T03:49:48.030" />
  <row Id="2328" PostTypeId="1" CreationDate="2010-09-03T04:53:37.240" Score="2" ViewCount="181" Body="&lt;p&gt;What should be the ratio of number of observations and number of variables?&#xA;How to detect overfitting in the neural network model and what are the ways to avoid overfitting?&#xA;If I want to perform classification with Neural Network, should the classes  have equal frequency?&#xA;Please help me out.&lt;/p&gt;&#xA;" OwnerUserId="861" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-09-03T04:59:10.847" LastActivityDate="2010-09-03T08:15:52.293" Title="How to perform Neural Network modelling effectively?" Tags="&lt;neural-networks&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="3" />
  <row Id="2329" PostTypeId="2" ParentId="534" CreationDate="2010-09-03T06:03:54.097" Score="0" ViewCount="" Body="&lt;p&gt;Suppose we think the factor A is the cause of the phenomenon B. Then we try to vary it to see whether B changes. &#xA;If B doesn't change and if we can assume that everything else unchanged, strong evidence that A is not the cause of B. &#xA;If B does change, we can't conclude that A is the cause because the change of A might have caused a change in the actual causation C, which made B change. &lt;/p&gt;&#xA;" LastActivityDate="2010-09-03T06:03:54.097" CommentCount="1" />
  <row Id="2330" PostTypeId="2" ParentId="852" CreationDate="2010-09-03T06:56:22.773" Score="5" ViewCount="" Body="&lt;p&gt;In the usual case with a log variable, the model is&#xA;\begin{align}&#xA;\log(y) &amp;amp;= a + b\log(x) + \varepsilon\newline&#xA;\text{or}\quad y &amp;amp;= e^a x^b e^\varepsilon,&#xA;\end{align}&#xA;where $\varepsilon\sim\text{N}(0,\sigma^2)$ and $b$ is the elasticity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the situation you mention, &#xA;\begin{align}&#xA;y &amp;amp;= \exp[a + b\log(x)] + \varepsilon \newline&#xA;\text{or}\quad y &amp;amp;= e^ax^b + \varepsilon,&#xA;\end{align}&#xA;where $\varepsilon\sim\text{N}(0,\sigma^2)$. So, ignoring the error, the parameter $b$ is playing the same role in both models and is an elasticity in both cases. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is different is the assumption on the error distribution. In the first case, the error is assumed to be logNormal and multiplicative while in the second case the error is assumed to be Normal and additive.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-09-03T06:56:22.773" />
  <row Id="2331" PostTypeId="4" CreationDate="2010-09-03T07:25:03.153" Score="0" ViewCount="0" Body="R is an open source programming language and software environment for statistical computing and graphics." OwnerUserId="-1" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2011-01-10T23:21:33.160" LastActivityDate="2011-01-10T23:21:33.160" />
  <row Id="2332" PostTypeId="2" ParentId="2328" CreationDate="2010-09-03T08:03:15.777" Score="4" ViewCount="" Body="&lt;p&gt;The advice I would give is as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(i) Exaust the possibilities of linear models (e.g. logistic regression) before going on to neural nets, especially if you have many features and not too many observations.  For many problems ANN do not out-perform simple linear classifiers, and the only way to find out if your problem is in this category is to try it and see.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(ii) Investigate kernel methods (e.g. SVM, kernel logistic regression), Gaussian process models first.  In both cases over-fitting is effectively controlled by tuning a small number of hyper-parameters.  For kernel methods this is often performed by cross-validation, for Gaussian process models this is performed by maximising the marginal likelihood (also known as the Bayesian &quot;evidence&quot; for the model).  I have found it is much easier to get a reasonable model using these methods than with neural networks, as the means of avoiding over-fitting is so much more straightforward.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(iii) If you really want to use a neural network, start with a (regularised) radial basis function network, rather than a feedforward MLP type network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(iv) If yo do use an MLP, then use regularisation.  If you do, it will be less sensitive to choices about architecture, such as optimising the number of hidden units.  Instead, all you have to do is choose a good value for the regularisation parameter.  MacKay's Bayesian &quot;evidence framework&quot; provides a good method for setting the regularisation parameter.  If you use regularisation, then the number of observations and number of variables becomes much less of an issue.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To detect over-fitting, simply perform cross-validation to test generalisation performance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for classes having equal frequencies, the thing to remember is that if you train a model will a balanced training set, but the classes are not balanced in the operational data, then the model is very likely to under-predict the minority class.  If you use a probabilistic classifier such as logistic regression or a neural network, you can always correct the estimated probabilities to account for that after training.  If yoour dataset is very imbalanced, I would recommend differential weighting of patterns from the positive and negative classes, with the weighting factors selected by cross-validation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, when the classes are very unbalanced, it is normally the case that false-negative and false-positive errors have difference costs (e.g. in medical screening tests a false-negative is much worse than a false-positive).  So often all you need to do is include the misclassification costs into the error function used to train the network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are a MATLAB user (like me) I can strongly recommend the NETLAB software (Ian Nabney and Chris Bishop) or the software that goes with the book Gaussian Proccess for Machine Learning by Rasmussen and Williams.  I can else strongly reccommend the book &quot;Neural networks for pattern recognition&quot; by Chris Bishop for anyone starting out in neural nets.  It is a brilliant book, and covers the material with great clarity and the minimum level of maths required to really understand what you are doing, and most of it is implemented in the NETLAB software (which may also run under Octave).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;HTH&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S. The best way of modelling with a neural net is probably to use a Bayesian approach based on Hybrid Monte Carlo (HMC), as developed by Radford Neal.  In general problems start in modelling when you try and optimise some parameters and you end up over-fitting.  The best solution is to never optimise anything and marginalise (integrate) over parameters instead.  Sadly this integration can't be performed analytically, so you need to use sampling based approaches instead.  However, this is (a) computationally expensive and (b) a bit of a &quot;black art&quot; and required deep understanding and experience.&lt;/p&gt;&#xA;" OwnerUserId="887" LastEditorUserId="887" LastEditorDisplayName="" LastEditDate="2010-09-03T08:15:52.293" LastActivityDate="2010-09-03T08:15:52.293" />
  <row Id="2333" PostTypeId="2" ParentId="58" CreationDate="2010-09-03T08:28:20.503" Score="4" ViewCount="" Body="&lt;p&gt;Back-propogation is a way of working out the derivative of the error function with respect to the weights, so that the model can be trained by gradient descent optimisation methods - it is basically just the application of the &quot;chain rule&quot;.  There isn't really much more to it than that, so if you are comfortable with calculus that is basically the best way to look at it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are not comfortable with calculus, a better way would be to say that we know how badly the output units are doing because we have a desired output with which to compare the actual output.  However we don't have a desired output for the hidden units, so what do we do?  The back-propagation rule is basically a way of speading out the blame for the error of the output units onto the hidden units.  The more influence a hidden unit has on a particular output unit, the more blame it gets for the error.  The total blame associated with a hidden unit then give an indication of how much the input-to-hidden layer weights need changing.  The two things that govern how much blame is passed back is the weight connecting the hidden and output layer weights (obviously) and the output of the hidden unit (if it is shouting rather than whispering it is likely to have a larger influence).  The rest is just the mathematical niceties that turn that intuition into the derivative of the training criterion.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd also recommend Bishops book for a proper answer! ;o)&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2010-09-03T08:28:20.503" />
  <row Id="2334" PostTypeId="2" ParentId="181" CreationDate="2010-09-03T08:40:44.130" Score="2" ViewCount="" Body="&lt;p&gt;I am working on an empirical study of this at the moment (approching a processor-century of simulations on our HPC facility!).  My advice would be to use a &quot;large&quot; network and regularisation, if you use regularisation then the network architecture becomes less important (provided it is large enough to represent the underlying function we want to capture), but you do need to tune the regularisation parameter properly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of the problems with architecture selection is that it is a discrete, rather than continuous, control of the complexity of the model, and therefore can be a bit of a blunt instrument, especially when the ideal complexity is low.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, this is all subject to the &quot;no free lunch&quot; theorems, while regularisation is effective in most cases, there will always be cases where architecture selection works better, and the only way to find out if that is true of the problem at hand is to try both approaches and cross-validate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I were to build an automated neural network builder, I would use Radford Neal's Hybrid Monte Carlo (HMC) sampling-based Bayesian approach, and use a large network and integrate over the weights rather than optimise the weights of a single network.  However that is computationally expensive and a bit of a &quot;black art&quot;, but the results Prof. Neal achieves suggests it is worth it!&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2010-09-03T08:40:44.130" />
  <row Id="2335" PostTypeId="1" AcceptedAnswerId="2342" CreationDate="2010-09-03T08:53:27.510" Score="7" ViewCount="295" Body="&lt;p&gt;I'm trying to interpret the following type of logistic model:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mdl &amp;lt;- glm(c(suc,fail) ~ fac1 + fac2, data=df, family=binomial)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is the output of &lt;code&gt;predict(mdl)&lt;/code&gt; the expected odds of success for each data point? Is there a simple way to tabulate the odds for each factor level of the model, rather than all the data points?&lt;/p&gt;&#xA;" OwnerUserId="229" LastEditorUserId="229" LastEditorDisplayName="" LastEditDate="2010-09-03T09:23:26.890" LastActivityDate="2010-09-03T15:16:08.050" Title="Output of logistic model in R" Tags="&lt;r&gt;&lt;logistic-regression&gt;&lt;glm&gt;" AnswerCount="1" CommentCount="3" FavoriteCount="5" />
  <row Id="2336" PostTypeId="2" ParentId="2131" CreationDate="2010-09-03T09:12:37.553" Score="0" ViewCount="" Body="&lt;p&gt;I would advise using a different value of the regularisation parameter C for examples of the positive class and examples of the negative class (many SVM packages support this, and in any case it is easily implemented).  Then use e.g. cross-validation to find good values of the two regularisation parameters.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It can be shown that this is asypmtotically equivalent re-sampling the data in a ratio determined by C+ and C- (so there is no advantage in re-sampling rather than re-weighting, they come to the same thing in the end and weights can be continuous, rather than discrete, so it gives finer control).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Don't simply choose C+ and C- to give a 50-50 weighting to positive and negative patterns though, as the stength of the effect of the &quot;imbalances classes&quot; problem will vary from dataset to dataset, so the strength of the optimal re-weighting cannot be determined a-priori.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also remember that false-positive and false-negative costs may be different, and the problem may resolve itself if these are included in determining C+ and C-.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is also worth bearing in mind, that for some problems the Bayes optimal decision rule will assign all patterns to a single class and ignore the other, so it isn't necessarily a bad thing - it may just mean that the density of patterns of one class is everywhere below the density of patterns of the other class.&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2010-09-03T09:12:37.553" />
  <row Id="2337" PostTypeId="1" AcceptedAnswerId="2340" CreationDate="2010-09-03T09:38:35.287" Score="2" ViewCount="315" Body="&lt;p&gt;I have a data-set consisting of N p-dimensional observations (all quantitative variables). I want to apply a hierarchical clustering algorithm to those data. As explained on page 505 in &lt;a href=&quot;http://www-stat.stanford.edu/~tibs/ElemStatLearn/&quot; rel=&quot;nofollow&quot;&gt;Elements of Statistical Learning&lt;/a&gt;, when using weighted average to combine the distances of the individual variables, it is often desirable (I have found no clues to the contrary in my scenario), to set the weights for each variable such that all variables have equal influence on the distance of the observations (formulas can be found in the book). The problem is, that I use &lt;a href=&quot;http://docs.scipy.org/doc/scipy/reference/spatial.distance.html&quot; rel=&quot;nofollow&quot;&gt;scipy.spatial.distance&lt;/a&gt; for calculating the distances, and this does no let me specify any weights. My question is, if I standardize my observations (multiply each dimension with 1/average of that dimension), does that solve the problem?&lt;/p&gt;&#xA;" OwnerUserId="977" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-16T06:35:32.243" LastActivityDate="2010-09-16T06:35:32.243" Title="How to standardize a data-set" Tags="&lt;standarization&gt;" AnswerCount="1" CommentCount="4" />
  <row Id="2338" PostTypeId="2" ParentId="1266" CreationDate="2010-09-03T09:41:58.997" Score="1" ViewCount="" Body="&lt;p&gt;The &lt;a href=&quot;http://cran.r-project.org/web/packages/ez/index.html&quot; rel=&quot;nofollow&quot;&gt;ez&lt;/a&gt; package, of which I am the author, has a function called ezPerm() which computes a permutation test, but probably doesn't do interactions properly (the documentation admits as much). The latest version has a function called ezBoot(), which lets you do bootstrap resampling that takes into account repeated measures (by resampling subjects, then within subjects), either using traditional cell means as the prediction statistic or using mixed effects modelling to make predictions for each cell in the design. I'm still not sure how &quot;non-parametric&quot; the bootstrap CIs from mixed effects model predictions are; my intuition is that they might reasonably be considered non-parametric, but my confidence in this area is low given that I'm still learning about mixed effects models. &lt;/p&gt;&#xA;" OwnerUserId="364" LastActivityDate="2010-09-03T09:41:58.997" CommentCount="1" />
  <row Id="2340" PostTypeId="2" ParentId="2337" CreationDate="2010-09-03T11:04:29.063" Score="1" ViewCount="" Body="&lt;p&gt;I'd just say: be careful with that. Standarization is needed only when some variable(s) dominates the dissimilarity score just because it is expressed in &quot;smaller units&quot;; let's say that you have a variable that is truly equal for all elements, but there is some, very small variability due to the measurement error. Now if you'd normalize this value, you'll make those errors an important factor for clustering.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-09-03T11:04:29.063" CommentCount="1" />
  <row Id="2342" PostTypeId="2" ParentId="2335" CreationDate="2010-09-03T13:21:10.640" Score="10" ViewCount="" Body="&lt;p&gt;The help pages for&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;predict.glm&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;state: &quot;Thus for a default binomial model the default predictions are of log-odds         (probabilities on logit scale) and ‘type = &quot;response&quot;’ gives the predicted probabilities&quot;. So, &lt;code&gt;predict(mdl)&lt;/code&gt; returns the log(odds), and using &quot;type = &quot;response&quot; returns the predicted probabilities. You might find this toy example instructive:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; y &amp;lt;- c(0,0,0,1,1,1,1,1,1,1)&#xA;&amp;gt; prop.table(table(y))&#xA;y&#xA;  0   1 &#xA;0.3 0.7 &#xA;&amp;gt; glm.y &amp;lt;- glm(y~1, family = &quot;binomial&quot;)&#xA;&amp;gt; ## predicted log(odds)&#xA;&amp;gt; predict(glm.y)&#xA;        1         2         3         4         5         6         7         8 &#xA;0.8472979 0.8472979 0.8472979 0.8472979 0.8472979 0.8472979 0.8472979 0.8472979 &#xA;        9        10 &#xA;0.8472979 0.8472979 &#xA;&amp;gt; ## predicted probabilities (p = odds/(1+odds))&#xA;&amp;gt; exp(predict(glm.y))/(1+exp(predict(glm.y)))&#xA;  1   2   3   4   5   6   7   8   9  10 &#xA;0.7 0.7 0.7 0.7 0.7 0.7 0.7 0.7 0.7 0.7 &#xA;&amp;gt; predict(glm.y, type = &quot;response&quot;)&#xA;  1   2   3   4   5   6   7   8   9  10 &#xA;0.7 0.7 0.7 0.7 0.7 0.7 0.7 0.7 0.7 0.7 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Regarding your second question, you might want to check out the effects-package &lt;a href=&quot;http://socserv.socsci.mcmaster.ca/jfox/Misc/effects/index.html&quot; rel=&quot;nofollow&quot;&gt;http://socserv.socsci.mcmaster.ca/jfox/Misc/effects/index.html&lt;/a&gt; by John Fox; see also his JSS article &quot;Effect Displays in R for Generalised Linear Models&quot; (pp. 8-10).&lt;/p&gt;&#xA;" OwnerUserId="307" LastEditorUserId="307" LastEditorDisplayName="" LastEditDate="2010-09-03T15:16:08.050" LastActivityDate="2010-09-03T15:16:08.050" CommentCount="1" />
  <row Id="2343" PostTypeId="1" CreationDate="2010-09-03T13:21:55.897" Score="3" ViewCount="177" Body="&lt;p&gt;Hello ! I am trying to develop some algorithm to compute probabilities in multi-type branching trees, and I doubt I am doing this right...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let us consider a multi-type branching process with two types, denoted by 0 and 1. The process starts in state 0 with probability 1, so that the root vertex of any tree generated by this process has state 0. A vertex in state 0 generates two vertices in state 1 with probability 1. A vertex in state 1 generates either two vertices in state 0 (with probability 0.5) or one vertex in state 0 and one vertex in state 1 (with probability 0.5).&#xA;We denote those three possible transitions by:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;A: 0 -&gt; 1 1 (1.)&lt;/li&gt;&#xA;&lt;li&gt;B: 1 -&gt; 0 0 (0.5)&lt;/li&gt;&#xA;&lt;li&gt;C: 1 -&gt; 0 1 (0.5)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;(their probabilities are between brackets).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the sequel, a &quot;tree&quot; will refer to an unordered tree, &lt;em&gt;i.e.&lt;/em&gt; two isomorphic ordered trees such that the tree isomorphism preserves the labels will be considered as a same mathematical object (following the principle in Chi (2004), p. 1993, paragraph 3 - see link at the end of the post). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now let us consider the particular labeled tree of height three, composed by one root vertex in state 0 with &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;one first child in state 1 that itself has one child in state 0 and one child in state 1&lt;/li&gt;&#xA;&lt;li&gt;one second child in state 1 that itself has two children in state 0&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;This tree should be depicted in some file I am not allowed to post, so I should draw it in ascii mode with limited guarantee on the result:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;           0  &#xA;           |  &#xA;     -------------  &#xA;     |           |  &#xA;     1           1&#xA;     |           |  &#xA; ---------   ---------  &#xA; |       |   |       |  &#xA; 0       1   0       0  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I would like to compute the probability that this tree was generated by the above multi-type branching process after 2 generation steps (the generation of the root vertex does not count as a step).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using equation 2 p. 1994 in Chi (2004), this probability should be 0.25, since each transition A, B and C is applied once. However, each possible tree of height 3 among the three trees generated by this process has probability 0.25, and the sum of the probabilities is 0.75 instead of 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another possibility is to consider that in this given tree, the set of children (0, 0) of vertex 1 may have been generated by any vertex in state 1 (and same principle for the set of children (0, 1)), so that the tree probability is in fact 0.25+0.25 = 0.5. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, how to compute the probability that a given tree &lt;em&gt;t&lt;/em&gt; of height &lt;em&gt;n&lt;/em&gt; was generated by a multi-type branching process after &lt;em&gt;n-1&lt;/em&gt; generation steps ? Can equation 2 p. 1994 in Chi (2004) be used ? Or do we have to compute the number of trees that are isomorphic to &lt;em&gt;t&lt;/em&gt; in some sense ? Or do we have to give up the idea that isomorphic trees are equivalent representations of same object ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for your help ! &#xA;JB&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ref.&lt;br&gt;&#xA;Z. Chi. &lt;a href=&quot;http://projecteuclid.org/DPubS?service=UI&amp;amp;version=1.0&amp;amp;verb=Display&amp;amp;handle=euclid.aoap/1099674086&quot; rel=&quot;nofollow&quot;&gt;Limit laws of estimators for critical multi-type Galton–Watson processes.&lt;/a&gt; Ann. Appl. Probab. Volume 14, Number 4 (2004), 1992-2015. &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;As a beginning of answer to the suggestion of Aniko, I think the following equivalence relation could lead to equivalence classes of trees such that formula (2) could be applied. It consists in adding constraints to the usual notion of isomorphism (by &quot;isomorphism&quot;, I mean &quot;an isomorphism preserving labels&quot;). I denote by $I(v)$ the label of vertex $v$, and &#xA;$t_v$ the complete subtree rooted in $v$. A complete subtree rooted in $v$ is the subgraph induced by the descendants of $v$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The property $P(t)$ is defined inductively by: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(t)$ is satisfied iif $t$ has a single vertex, or if for every pair ${v, v'}$ of children of the root of $t$ such that $I(v)$ = $I(v')$, $t_v$ and $t_{v'}$ are isomorphic and they satisfy $P(t_{v'})$ and $P(t_{v})$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The two trees $t$ and $t'$ are in the same equivalence class iif [they are isomorphic, and property $P$ is satisfied for $t$ and $t'$] or [$t=t'$].&lt;/p&gt;&#xA;" OwnerUserId="1185" LastEditorUserId="1185" LastEditorDisplayName="" LastEditDate="2010-09-10T13:33:36.397" LastActivityDate="2010-11-22T16:36:15.373" Title="Trees generated by multi-type branching processes in n steps" Tags="&lt;probability&gt;&lt;algorithms&gt;&lt;stochastic-processes&gt;" AnswerCount="2" FavoriteCount="2" />
  <row Id="2344" PostTypeId="1" AcceptedAnswerId="2346" CreationDate="2010-09-03T13:50:51.707" Score="8" ViewCount="780" Body="&lt;p&gt;I am using the random forest algorithm as a robust classifier of two groups in a microarray study with 1000s of features.  &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What is the best way to present the random forest so that there is enough information to make it&#xA;reproducible in a paper?&lt;/li&gt;&#xA;&lt;li&gt;Is there a plot method in R to actually plot the tree, if there are a small number of features?&lt;/li&gt;&#xA;&lt;li&gt;Is the OOB estimate of error rate the best statistic to quote?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Many thanks&lt;/p&gt;&#xA;" OwnerUserId="1150" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-10T17:33:57.390" LastActivityDate="2010-10-10T17:33:57.390" Title="Best way to present a random forest in a publication?" Tags="&lt;r&gt;&lt;machine-learning&gt;&lt;classification&gt;&lt;random-forest&gt;&lt;microarray&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="7" />
  <row Id="2345" PostTypeId="2" ParentId="2343" CreationDate="2010-09-03T14:19:30.763" Score="3" ViewCount="" Body="&lt;p&gt;I think you explained well why the probability of the given tree is 0.5 if its topology does not count. Looking at the formula (2) superficially, I find it hard to imagine the definition of isomorphism under which it would work (only leaves can be rearranged?), though perhaps the trick is in finding the right definition of the counting function f.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the general case I would try to write a recursive formula, and I think the binomial coefficient should pop uo in it.&lt;/p&gt;&#xA;" OwnerUserId="279" LastActivityDate="2010-09-03T14:19:30.763" CommentCount="3" />
  <row Id="2346" PostTypeId="2" ParentId="2344" CreationDate="2010-09-03T14:32:35.293" Score="7" ViewCount="" Body="&lt;p&gt;Regarding making it reproducible, the best way is to &lt;em&gt;provide reproducible research&lt;/em&gt; (i.e. code and data) along with the paper.  Make it available on your website, or on a hosting site (like github).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regarding visualization, Leo Breiman has done some interesting work on this (see &lt;a href=&quot;http://www.stat.berkeley.edu/~breiman/RandomForests/&quot; rel=&quot;nofollow&quot;&gt;his homepage&lt;/a&gt;, in particular the &lt;a href=&quot;http://www.stat.berkeley.edu/~breiman/RandomForests/cc_graphics.htm&quot; rel=&quot;nofollow&quot;&gt;section on graphics&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But if you're using R, then the &lt;code&gt;randomForest&lt;/code&gt; package has some useful functions:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data(mtcars)&#xA;mtcars.rf &amp;lt;- randomForest(mpg ~ ., data=mtcars, ntree=1000, keep.forest=FALSE,&#xA;                           importance=TRUE)&#xA;plot(mtcars.rf, log=&quot;y&quot;)&#xA;varImpPlot(mtcars.rf)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(1)&#xA;data(iris)&#xA;iris.rf &amp;lt;- randomForest(Species ~ ., iris, proximity=TRUE,&#xA;                        keep.forest=FALSE)&#xA;MDSplot(iris.rf, iris$Species)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm not aware of a simple way to actually plot a tree, but you can use the &lt;code&gt;getTree&lt;/code&gt; function to retrieve the tree and plot that separately.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;getTree(randomForest(iris[,-5], iris[,5], ntree=10), 3, labelVar=TRUE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The &lt;a href=&quot;http://www.statistik.uni-dortmund.de/useR-2008/slides/Strobl+Zeileis.pdf&quot; rel=&quot;nofollow&quot;&gt;Strobl/Zeileis presentation on &quot;Why and how to use random forest variable importance measures (and how you shouldn’t)&quot;&lt;/a&gt; has examples of trees which must have been produced in this way.  This &lt;a href=&quot;http://www.statmethods.net/advstats/cart.html&quot; rel=&quot;nofollow&quot;&gt;blog post on tree models&lt;/a&gt; has some nice examples of CART tree plots which you can use for example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As @chl commented, a single tree isn't especially meaningful in this context, so short of using it to explain what a random forest is, I wouldn't include this in a paper.&lt;/p&gt;&#xA;" OwnerUserId="5" LastEditorUserId="5" LastEditorDisplayName="" LastEditDate="2010-09-03T15:30:56.143" LastActivityDate="2010-09-03T15:30:56.143" CommentCount="2" />
  <row Id="2347" PostTypeId="2" ParentId="3" CreationDate="2010-09-03T14:42:15.677" Score="7" ViewCount="" Body="&lt;p&gt;There are also those projects initiated by the FSF or redistributed under GNU General Public License, like:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.gnu.org/software/pspp/&quot; rel=&quot;nofollow&quot;&gt;PSPP&lt;/a&gt;, which aims to be a free alternative to SPSS&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://gretl.sourceforge.net/&quot; rel=&quot;nofollow&quot;&gt;GRETL&lt;/a&gt;, mostly dedicated to regression and econometrics&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;There is even applications that were released just as a companion software for a textbook, like &lt;a href=&quot;http://www.jmulti.de/&quot; rel=&quot;nofollow&quot;&gt;JMulTi&lt;/a&gt;, but are still in use by few people.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am still playing with &lt;a href=&quot;http://www.stat.uiowa.edu/~luke/xls/xlsinfo/&quot; rel=&quot;nofollow&quot;&gt;xlispstat&lt;/a&gt;, from time to time, although Lisp has been largely superseded by R (see Jan de Leeuw's overview on &lt;a href=&quot;http://www.jstatsoft.org/v13/i07&quot; rel=&quot;nofollow&quot;&gt;Lisp vs. R&lt;/a&gt; in the &lt;em&gt;Journal of Statistical Software&lt;/em&gt;). Interestingly, one of the cofounders of the R language, Ross Ihaka, argued on the contrary that the future of statistical software is... Lisp: &lt;a href=&quot;http://www.stat.auckland.ac.nz/%7Eihaka/downloads/Compstat-2008.pdf&quot; rel=&quot;nofollow&quot;&gt;Back to the Future: Lisp as a Base for a Statistical Computing System&lt;/a&gt;. @Alex already pointed to the Clojure-based statistical environment &lt;a href=&quot;http://incanter.org/&quot; rel=&quot;nofollow&quot;&gt;Incanter&lt;/a&gt;, so maybe we will see a revival of Lisp-based software in the near future? :-)&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-09-03T14:42:15.677" CommunityOwnedDate="2010-09-03T14:42:15.677" />
  <row Id="2348" PostTypeId="1" CreationDate="2010-09-03T15:02:58.163" Score="9" ViewCount="340" Body="&lt;p&gt;I was reading &lt;a href=&quot;http://xianblog.wordpress.com/2010/09/02/random-dive-mh&quot; rel=&quot;nofollow&quot;&gt;Christian Robert's Blog&lt;/a&gt; today and quite liked the new Metropolis-Hastings algorithm he was discussing. It seemed simple and easy to implement.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Whenever I code up MCMC, I tend to stick with very basic MH algorithms, such as independent moves or random walks on the log scale.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What MH algorithms do people routinely use? In particular:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Why do you use them? &lt;/li&gt;&#xA;&lt;li&gt;In some sense you must think that they are optimal - after all you use them routinely! So how do you judge optimality: ease-of-coding, convergence, ... &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I'm particularly interested in what is used in practice, i.e. when you code up your own schemes.&lt;/p&gt;&#xA;" OwnerUserId="8" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-09-04T10:49:24.537" LastActivityDate="2010-10-05T12:11:53.590" Title="Metropolis-Hastings algorithms used in practice" Tags="&lt;mcmc&gt;&lt;metropolis-hastings&gt;" AnswerCount="4" CommentCount="3" FavoriteCount="2" />
  <row Id="2349" PostTypeId="2" ParentId="2269" CreationDate="2010-09-03T15:51:04.217" Score="1" ViewCount="" Body="&lt;p&gt;Obviously there is probably a better solution available than what I'm about to say - especially since these are both SAS products.  However but I think it bares saying that if all else fails, when your data is in a tabular structure, you can almost always export the data as a deliminated text file (e.g. csv) and import it into the new program as a deliminated text file.  I don't think I've seen a program that works with tablular data structures yet that doesn't provide these options.&lt;/p&gt;&#xA;" OwnerUserId="196" LastActivityDate="2010-09-03T15:51:04.217" />
  <row Id="2350" PostTypeId="1" AcceptedAnswerId="2359" CreationDate="2010-09-03T15:55:37.100" Score="5" ViewCount="241" Body="&lt;p&gt;I am applying a random forest algorithm as a classifier on a microarray dataset which are split into two known groups with 1000s of features.  After the initial run I look at the importance of the features and run the tree algorithm again with the 5, 10 and 20 most important features.  &lt;strong&gt;I find that for all features, top 10 and 20 that the OOB estimate of error rate is 1.19% where as for the top 5 features it is 0%.&lt;/strong&gt;  This seems counter-intuitive to me, so I was wondering whether you could explain whether I am missing something or I am using the wrong metric.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I an using the randomForest package in R with ntree=1000, nodesize=1 and mtry=sqrt(n)&lt;/p&gt;&#xA;" OwnerUserId="1150" LastActivityDate="2010-09-03T19:00:15.123" Title="Why does the random forest OOB estimate of error improve when the number of features selected are decreased?" Tags="&lt;r&gt;&lt;machine-learning&gt;&lt;classification&gt;&lt;random-forest&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="2" />
  <row Id="2351" PostTypeId="2" ParentId="2348" CreationDate="2010-09-03T16:00:07.917" Score="2" ViewCount="" Body="&lt;p&gt;Hybrid Monte Carlo is the standard algorithm used for neural networks.  Gibbs sampling for Gaussian process classification (when not using a deterministic approximation instead). &lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2010-09-03T16:00:07.917" />
  <row Id="2352" PostTypeId="1" CreationDate="2010-09-03T16:15:14.543" Score="7" ViewCount="278" Body="&lt;p&gt;In his paper &lt;a href=&quot;http://www.jstor.org/pss/2290328&quot; rel=&quot;nofollow&quot;&gt;Linear Model Selection by Cross-Validation&lt;/a&gt;, Jun Shao shows that for the problem of variable selection in multivariate linear regression, the method of leave-one-out cross validation (LOOCV) is 'asymptotically inconsistent'. In plain English, it tends to select models with too many variables. In a simulation study, Shao shows that even for as few as 40 observations, LOOCV can underperform other cross-validation techniques.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This paper is somewhat controversial, and somewhat ignored (10 years after its publication, my chemometrics colleagues had never heard of it and were happily using LOOCV for variable selection...). There is also a belief (I am guilty of this), that it's results extend somewhat beyond the original limited scope.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The question, then: how far do these results extend? Are they applicable to the following problems?&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Variable selection for logistic regression/GLM?&lt;/li&gt;&#xA;&lt;li&gt;Variable selection for Fisher LDA classification?&lt;/li&gt;&#xA;&lt;li&gt;Variable selection using SVM with finite (or infinite) kernel space?&lt;/li&gt;&#xA;&lt;li&gt;Comparison of models in classification, say SVM using different kernels?&lt;/li&gt;&#xA;&lt;li&gt;Comparison of models in linear regression, say comparing MLR to Ridge Regression?&lt;/li&gt;&#xA;&lt;li&gt;etc.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-09-14T02:33:16.480" Title="When are Shao's results on leave-one-out cross-validation applicable?" Tags="&lt;classification&gt;&lt;model-selection&gt;&lt;cross-validation&gt;" AnswerCount="3" CommentCount="1" FavoriteCount="3" />
  <row Id="2353" PostTypeId="2" ParentId="3" CreationDate="2010-09-03T16:27:56.423" Score="10" ViewCount="" Body="&lt;p&gt;This may get downvoted to oblivion, but I happily used the Matlab clone &lt;a href=&quot;http://www.gnu.org/software/octave/&quot; rel=&quot;nofollow&quot;&gt;Octave&lt;/a&gt; for many years. There are fairly good libraries in octave forge for generation of random variables from different distributions, statistical tests, etc, though clearly it is dwarfed by R. One possible advantage over R is that Matlab/octave is the &lt;em&gt;lingua franca&lt;/em&gt; among numerical analysts, optimization researchers, and some subset of applied mathematicians (at least when I was in school), whereas nobody in my department, to my knowledge, used R. my loss. learn both if possible!&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-09-03T16:27:56.423" CommunityOwnedDate="2010-09-03T16:27:56.423" CommentCount="2" />
  <row Id="2354" PostTypeId="2" ParentId="2170" CreationDate="2010-09-03T17:57:15.193" Score="1" ViewCount="" Body="&lt;p&gt;Thanks again for the response, and any other responses in the future will be much appreciated. I think I personally prefer using exploratory tools to identify the relationships, especially since the original researcher did not give any real reason why a curvilinear relationship would exist theoretically. Although exploration would identify if the relationship was not properly identified (e.g. if it should have been a cubed polynomial term instead of squared), this seems unlikely given theres no reason why it would have a curve in it to begin with.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The attached image is a plot of several of their finals models, and I have plotted the expected value of Y given their X and X^2 coefficients reported, holding everything else in the models constant. If their results are representative, I should observe similar findings in my sample controlling for other confounders. It also is enlightening just to plot their findings, as I see the squared term dominates in several of the models, and so for all practical purposes it has a negative relationship that reaches the bottom of realistic Y values fairly quickly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/8IQ64.jpg&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="1036" LastActivityDate="2010-09-03T17:57:15.193" />
  <row Id="2355" PostTypeId="2" ParentId="2352" CreationDate="2010-09-03T18:14:13.153" Score="4" ViewCount="" Body="&lt;p&gt;I would say: everywhere, but I haven't seen a strict proof of it. The intuition behind is such that when doing CV one must hold a balance between train large enough to build sensible model and test large enough so it would be a sensible benchmark.&lt;br&gt;&#xA;When dealing with thousands of pretty homogeneous objects, picking one is connected with risk that it is pretty similar to other object that was left in the set -- and then the results would be too optimistic.&lt;br&gt;&#xA;On the other hand, in case of few objects there will be no vital difference between LOO and k-fold; $10/10$ is just $1$ and we can't do anything with it.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-09-03T18:14:13.153" CommentCount="3" />
  <row Id="2356" PostTypeId="1" CreationDate="2010-09-03T18:23:44.087" Score="16" ViewCount="918" Body="&lt;p&gt;A recent question on the difference between confidence and credible intervals led me to start re-reading Edwin Jaynes' article on that topic:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Jaynes, E. T., 1976. `Confidence Intervals vs Bayesian Intervals,' in Foundations of Probability Theory, Statistical Inference, and Statistical Theories of Science, W. L. Harper and C. A. Hooker (eds.), D. Reidel, Dordrecht, p. 175; (&lt;a href=&quot;http://bayes.wustl.edu/etj/articles/confidence.pdf&quot; rel=&quot;nofollow&quot;&gt;pdf&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the abstract, Jaynes writes:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;...we exhibit the Bayesian and orthodox solutions to six common statistical problems involving confidence intervals (including significance tests based on the same reasoning).  In every case, we find the situation is exactly the opposite, i.e. the Bayesian method is easier to apply and yields the same or better results.  Indeed, the orthodox results are satisfactory only when they agree closely (or exactly) with the Bayesian results.  NO CONTRARY EXAMPLE HAS YET BEEN PRODUCED&quot;. (the SHOUTING being mine not Jaynes')&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The paper was published in 1976, so perhaps things have moved on, so my question is, are there examples where the frequentist confidence interval is clearly superior to the Bayesian credible interval (as per the challenge implicitly made by Jaynes).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Examples based on incorrect prior assumptions are not acceptable as they say nothing about the internal consistency of the different approaches.&lt;/p&gt;&#xA;" OwnerUserId="887" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-29T00:53:39.100" LastActivityDate="2011-04-29T00:53:39.100" Title="Are there any examples where Bayesian credible intervals are obviously inferior to frequentist confidence intervals" Tags="&lt;bayesian&gt;&lt;confidence-interval&gt;" AnswerCount="5" CommentCount="12" FavoriteCount="3" />
  <row Id="2357" PostTypeId="2" ParentId="1668" CreationDate="2010-09-03T18:51:15.067" Score="1" ViewCount="" Body="&lt;p&gt;As a computer engineer coming to data analysis myself, a really readable book that covers things from a pretty unintimidating &amp;amp; readable perspective (at the cost of not covering as much as any of the other books suggested here) was Programming Collective Intelligence by Toby Segaran. I found it a lot more approachable than, for example, Bishop's book, which a great reference but goes into more depth that you probably want at a first pass. On amazon: &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0596529325&quot; rel=&quot;nofollow&quot;&gt;http://www.amazon.com/Programming-Collective-Intelligence-Building-Applications/dp/0596529325&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1076" LastActivityDate="2010-09-03T18:51:15.067" CommunityOwnedDate="2010-09-03T18:51:15.067" />
  <row Id="2358" PostTypeId="1" CreationDate="2010-09-03T18:54:17.230" Score="5" ViewCount="625" Body="&lt;p&gt;Are multiple and multivariate regression really different? What &lt;em&gt;is&lt;/em&gt; a variate anyways? :)&lt;/p&gt;&#xA;" OwnerUserId="74" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-12T17:52:27.880" LastActivityDate="2010-09-19T09:32:42.730" Title="Explain the difference between multiple regression and multivariate regression, with minimal use of symbols/math." Tags="&lt;regression&gt;&lt;multivariate-analysis&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="1" />
  <row Id="2359" PostTypeId="2" ParentId="2350" CreationDate="2010-09-03T19:00:15.123" Score="4" ViewCount="" Body="&lt;p&gt;This is feature selection overfit and this is pretty known -- see &lt;a href=&quot;http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=124442&amp;amp;tool=pmcentrez&amp;amp;rendertype=abstract&quot; rel=&quot;nofollow&quot;&gt;Ambroise &amp;amp; McLachlan 2002&lt;/a&gt;. &#xA;The problem is based on the facts that RF is too smart and number of objects is too small. In the latter case, it is generally pretty easy to randomly create attribute that may have good correlation with the decision. And when the number of attributes is large, you may be certain that some of totally irrelevant ones will be a very good predictors, even enough to form a cluster that will be able to recreate the decision in 100%, especially when the huge flexibility of RF is considered. And so, it becomes obvious that when instructed to find the best possible subset of attributes, the FS procedure finds this cluster.&lt;br&gt;&#xA;One solution (CV) is given in A&amp;amp;McL, you can also test our approach to the topic, the &lt;a href=&quot;http://cran.r-project.org/web/packages/Boruta/index.html&quot; rel=&quot;nofollow&quot;&gt;Boruta algorithm&lt;/a&gt;, which basically extends the set with &quot;shadow attributes&quot; made to be random by design and compares their RF importance to this obtained for real attributes to judge which of them are indeed random and can be removed; this is replicated many times to be significant. Boruta is rather intended to a bit different task, but as far as my tests showed, the resulting set is free of the FS overfit problem.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-09-03T19:00:15.123" />
  <row Id="2360" PostTypeId="2" ParentId="2358" CreationDate="2010-09-03T19:03:07.687" Score="8" ViewCount="" Body="&lt;p&gt;Very quickly, I would say: 'multiple' applies to the number of predictors that enter the model (or equivalently the design matrix) with a single outcome (Y response), while 'multivariate' refers to a matrix of response vectors. Cannot remember the author who starts its introductory section on multivariate modeling with that consideration, but I think it is Brian Everitt in his textbook &lt;a href=&quot;http://www.springer.com/statistics/social+sciences+%26+law/book/978-1-85233-882-4&quot; rel=&quot;nofollow&quot;&gt;An R and S-Plus Companion to Multivariate Analysis&lt;/a&gt;. For a thorough discussion about this, I would suggest to look at his latest book, &lt;a href=&quot;http://www.crcpress.com/product/isbn/9781439807699?refpage=http%3A//www.crcpress.com/ecommerce_product/browse_book_categories.jsf&amp;amp;refpn=category&amp;amp;refpv=STA12A&quot; rel=&quot;nofollow&quot;&gt;Multivariable Modeling and Multivariate Analysis for the Behavioral Sciences&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For 'variate', I would say this is a common way to refer to any random variable that follows a known or hypothesized distribution, e.g. we speak of gaussian variates $X_i$ as a series of observations drawn from a normal distribution (with parameters $\mu$ and $\sigma^2$). In probabilistic terms, we said that these are some random &lt;em&gt;realizations&lt;/em&gt; of X, with mathematical expectation $\mu$, and about 95% of them are expected to lie on the range $[\mu-2\sigma;\mu+2\sigma]$ .&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-19T09:32:42.730" LastActivityDate="2010-09-19T09:32:42.730" />
  <row Id="2361" PostTypeId="2" ParentId="1668" CreationDate="2010-09-03T19:22:30.447" Score="2" ViewCount="" Body="&lt;p&gt;Here is a very nice book from James E. Gentle, &lt;a href=&quot;http://www.springer.com/statistics/computanional+statistics/book/978-0-387-98143-7&quot; rel=&quot;nofollow&quot;&gt;Computational Statistics&lt;/a&gt; (Springer, 2009), which covers both computational and statistical aspects of data analysis. Gentle also authored other great books, check his publications.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another great book is the &lt;a href=&quot;http://www.springer.com/statistics/computanional+statistics/book/978-3-540-40464-4&quot; rel=&quot;nofollow&quot;&gt;Handbook of Computational Statistics&lt;/a&gt;, from Gentle et al. (Springer, 2004); it is circulating as PDF somewhere on the web, so just try looking at it on Google.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-09-03T19:22:30.447" CommunityOwnedDate="2010-09-03T19:22:30.447" />
  <row Id="2362" PostTypeId="2" ParentId="2344" CreationDate="2010-09-03T19:22:56.617" Score="4" ViewCount="" Body="&lt;ol&gt;&#xA;&lt;li&gt;As Shane wrote; make it reproducible research + include random seeds, because RF is stochastic.&lt;/li&gt;&#xA;&lt;li&gt;First of all, plotting single trees forming RF is nonsense; this is an ensemble classifier, it makes sense only as a whole. But even plotting the whole forest is nonsense -- it is a black-box classifier, so it is not intended to explain the data with its structure, rather to replicate the original process. Instead, make some of plots Shane suggested.&lt;/li&gt;&#xA;&lt;li&gt;In practice, OOB is a very good error approximation; yet this is not a widely accepted fact, so for publication it is better to also make a CV to confirm it. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-09-03T19:22:56.617" CommentCount="7" />
  <row Id="2363" PostTypeId="2" ParentId="2358" CreationDate="2010-09-03T19:27:20.703" Score="5" ViewCount="" Body="&lt;p&gt;Here are two closely related examples which illustrate the ideas. The examples are somewhat US centric but the ideas can be extrapolated to other countries.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Example 1&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose that a university wishes to refine its admission criteria so that they admit 'better' students. Also, suppose that a student's grade Point Average (GPA) is what the university wishes to use as a performance metric for students. They have several criteria in mind such as high school GPA (HSGPA), SAT scores (SAT), Gender etc and would like to know which one of these criteria matter as far as GPA is concerned.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Solution: Multiple Regression&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the above context, there is one dependent variable (GPA) and you have multiple independent variables (HSGPA, SAT, Gender etc). You want to find out which one of the independent variables are good predictors for your dependent variable. You would use multiple regression to make this assessment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Example 2&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead of the above situation, suppose the admissions office wants to track student performance across time and wishes to determine which one of their criteria drives student performance across time. In other words, they have GPA scores for the four years that a student stays in school (say, GPA1, GPA2, GPA3, GPA4) and they want to know which one of the independent variables predict GPA scores better on a year-by-year basis. The admissions office hopes to find that the &lt;em&gt;same&lt;/em&gt; independent variables predict performance across all four years so that their choice of admissions criteria ensures that student performance is consistently high across all four years.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Solution: Multivariate Regression&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In example 2, we have multiple dependent variables (i.e., GPA1, GPA2, GPA3, GPA4) and multiple independent variables. In such a situation, you would use multivariate regression.&lt;/p&gt;&#xA;" LastActivityDate="2010-09-03T19:27:20.703" />
  <row Id="2364" PostTypeId="2" ParentId="1668" CreationDate="2010-09-03T19:38:45.930" Score="1" ViewCount="" Body="&lt;p&gt;CRAN has several good examples of books pertaining to statistical programming. Some of them will not pertain to machine learning and MCMC, but each entry is annotated, so you should have a rough idea of what each book contains to dive a bit further.&#xA;&lt;a href=&quot;http://www.r-project.org/doc/bib/R-books.html&quot; rel=&quot;nofollow&quot;&gt;http://www.r-project.org/doc/bib/R-books.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1118" LastActivityDate="2010-09-03T19:38:45.930" CommunityOwnedDate="2010-09-03T19:38:45.930" />
  <row Id="2365" PostTypeId="2" ParentId="2356" CreationDate="2010-09-03T20:24:52.440" Score="4" ViewCount="" Body="&lt;p&gt;The problem starts with your sentence :&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Examples based on incorrect prior&#xA;  assumptions are not acceptable as they&#xA;  say nothing about the internal&#xA;  consistency of the different&#xA;  approaches.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Yeah well, how do you know your prior is correct? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Take the case of Bayesian inference in phylogeny. The probability of at least one change is related to evolutionary time (branch length t) by the formula &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P=1-e^{-\frac{4}{3}ut}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;with u being the rate of substitution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now you want to make a model of the evolution, based on comparison of DNA sequences. In essence, you try to estimate a tree in which you try to model the amount of change between the DNA sequences as close as possible. The P above is the chance of at least one change on a given branch. Evolutionary models describe the chances of change between any two nucleotides, and from these evolutionary models the estimation function is derived, either with p as a parameter or with t as a parameter. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You have no sensible knowledge and you chose a flat prior for p. This inherently implies an exponentially decreasing prior for t.  (It becomes even more problematic if you want to set a flat prior on t. The implied prior on p is strongly dependent on where you cut off the range of t.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In theory, t can be infinite, but when you allow an infinite range, the area under its density function equals infinity as well, so you have to define a truncation point for the prior.  Now when you chose the truncation point sufficiently large, it is not difficult to prove that both ends of the credible interval rise, and at a certain point the true value is not contained in the credible interval any more. Unless you have a very good idea about the prior, Bayesian methods are not guaranteed to be equal to or superior to other methods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ref: Joseph Felsenstein : Inferring Phylogenies, chapter 18&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On a side note, I'm getting sick of that Bayesian/Frequentist quarrel. They're both different frameworks, and neither is the Absolute Truth. The classical examples pro Bayesian methods invariantly come from probability calculation, and not one frequentist will contradict them. The classical argument against Bayesian methods invariantly involve the arbitrary choice of a prior. And sensible priors are definitely possible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It all boils down to the correct use of either method at the right time. I've seen very few arguments/comparisons where both methods were applied correctly. Assumptions of any method are very much underrated and far too often ignored.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT : to clarify, the problem lies in the fact that the estimate based on p differs from the estimate based on t in the Bayesian framework when working with uninformative priors (which is in a number of cases the only possible solution). This is not true in the ML framework for phylogenetic inference. It is not a matter of a wrong prior, it is inherent to the method.&lt;/p&gt;&#xA;" OwnerUserId="1124" LastEditorUserId="1124" LastEditorDisplayName="" LastEditDate="2010-09-03T23:00:59.537" LastActivityDate="2010-09-03T23:00:59.537" CommentCount="41" />
  <row Id="2366" PostTypeId="2" ParentId="2326" CreationDate="2010-09-03T21:16:13.070" Score="2" ViewCount="" Body="&lt;p&gt;Actually, people get paid big money for statistical guidance through experiments... If you're not too sure about it, I'd also advise to consult a statistician. An internet forum is not the best aid for complex analyses. Much of what is possible depends on the structure within the dataset: How are the variables distributed, how is the correlation structure,...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I'll take a shot at it.&#xA;So you have $m$ measurements at 4 different times, being: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;virgin state&lt;/li&gt;&#xA;&lt;li&gt;after preparation&lt;/li&gt;&#xA;&lt;li&gt;after first round (shot in &quot;virgin state&quot;)&lt;/li&gt;&#xA;&lt;li&gt;after second round (shot in &quot;non-virgin state&quot;)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Your response variable is $MV$. Let's assume the error of the chronograph is random and normally distributed, this makes life a lot more easy. Let's also assume that for all other measurements the assumptions of parametric models apply, i.e. that every measurement consists of $n$ indipendent and identically distributed random variables, being your $n$ tested cases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A rather basic approach would involve &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;using a Kolgomorov-Smirnov test to compare distributions (think about correction for multitesting)&lt;/li&gt;&#xA;&lt;li&gt;a model selection procedure to determine the best predictors for $MV$ using either set&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;So in theory, you could use a GEE with &quot;round&quot; as a factor, indicating whether the measurement was made in the first round or in the second round, thus allowing for different intercepts and coefficients for first and second round that can be compared and tested. You have to specify &quot;case&quot; as a random factor in the GEE to get correct estimates of your SE. Basically, if the main effect &quot;round&quot; is significant, there is a difference before and after in average $MV$. If the interaction &quot;round&quot;:measurement is significant, the impact of the measurement is different between the two rounds. Now it boils down to use any sensible criterium to select the best model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Without seeing the data, it's impossible to say whether this approach is actually valid. You will have to do some descriptive and exploratory analysis to really understand the underlying distributions and correlations. Otherwise you can never know whether you violated assumptions of any method.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On a sidenote: Give this dataset to 10 statisticians, and chance is big you get as many different models. You should keep in mind whether or not the model also makes sense in real life. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;my 2 cents&lt;/p&gt;&#xA;" OwnerUserId="1124" LastActivityDate="2010-09-03T21:16:13.070" />
  <row Id="2367" PostTypeId="2" ParentId="2352" CreationDate="2010-09-04T03:12:13.830" Score="4" ViewCount="" Body="&lt;p&gt;You need to specify the purpose of the model before you can say whether Shao's results are applicable. For example, if the purpose is prediction, then LOOCV makes good sense and the inconsistency of variable selection is not a problem. On the other hand, if the purpose is to identify the important variables and explain how they affect the response variable, then Shao's results are obviously important and LOOCV is not appropriate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The AIC is asymptotically LOOCV and BIC is asymptotically equivalent to a leave-$v$-out CV where $v=n[1-1/(\log(n)-1)]$ --- the BIC result for linear models only. So the BIC gives consistent model selection. Therefore a short-hand summary of Shao's result is that AIC is useful for prediction but BIC is useful for explanation.&lt;/p&gt;&#xA;" OwnerUserId="159" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-09-14T02:33:16.480" LastActivityDate="2010-09-14T02:33:16.480" CommentCount="12" />
  <row Id="2368" PostTypeId="2" ParentId="2296" CreationDate="2010-09-04T03:58:10.850" Score="2" ViewCount="" Body="&lt;p&gt;You should take a look at some of the nonparametric Bayesian approaches (see &lt;a href=&quot;http://hal3.name/docs/daume08ihfrm.pdf&quot; rel=&quot;nofollow&quot;&gt;this paper&lt;/a&gt; and &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.133.2649&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;this paper&lt;/a&gt;) to factor analysis which do not assume the number of factors to be known; the first one can also model the case where the factors have a dependency structure among them.&lt;/p&gt;&#xA;" OwnerUserId="881" LastActivityDate="2010-09-04T03:58:10.850" />
  <row Id="2369" PostTypeId="2" ParentId="2356" CreationDate="2010-09-04T04:22:51.527" Score="3" ViewCount="" Body="&lt;p&gt;Frequentist confidence intervals bound the rate of false positives (Type I errors), and guarantee their coverage will be bounded below by the confidence parameter, even in the worst case. Bayesian credibility intervals don't.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So if the thing you care about is false positives and you need to bound them, confidence intervals are the the approach that you'll want to use.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, let's say you have an evil king with a court of 100 courtiers and courtesans and he wants to play a cruel statistical game with them. The king has a bag of a trillion fair coins, plus one unfair coin whose heads probability is 10%. He's going to perform the following game. First, he'll draw a coin uniformly at random from the bag.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then the coin will be passed around a room of 100 people and each one will be forced to do an experiment on it, privately, and then each person will state a 95% uncertainty interval on what they think the coin's heads probability is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anybody who gives an interval that represents a false positive -- i.e. an interval that doesn't cover the true value of the heads probability -- will be beheaded.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we wanted to express the /a posteriori/ probability distribution function of the coin's weight, then of course a credibility interval is what does that. The answer will always be the interval [0.5, 0.5] irrespective of outcome. Even if you flip zero heads or one head, you'll still say [0.5, 0.5] because it's a heck of a lot more probable that the king drew a fair coin and you had a 1/1024 day getting ten heads in a row, than that the king drew the unfair coin.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So this is not a good idea for the courtiers and courtesans to use! Because when the unfair coin is drawn, the whole room (all 100 people) will be wrong and they'll all get beheaded.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this world where the most important thing is false positives, what we need is an absolute guarantee that the rate of false positives will be less than 5%, no matter which coin is drawn. Then we need to use a confidence interval, like Blyth-Still-Casella or Clopper-Pearson, that works and provides at least 95% coverage &lt;strong&gt;&lt;em&gt;irrespective of the true value of the parameter, even in the worst case&lt;/em&gt;&lt;/strong&gt;. If everybody uses this method instead, then no matter which coin is drawn, at the end of the day we can guarantee that the expected number of wrong people will be no more than five.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the point is: if your criterion requires bounding false positives (or equivalently, guaranteeing coverage), you gotta go with a confidence interval. That's what they do. Credibility intervals may be a more intuitive way of expressing uncertainty, they may perform pretty well from a frequentist analysis, but they are not going to provide the guaranteed bound on false positives you'll get when you go asking for it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Of course if you also care about false negatives, you'll need a method that makes guarantees about those too...)&lt;/p&gt;&#xA;" OwnerUserId="1122" LastEditorUserId="1122" LastEditorDisplayName="" LastEditDate="2010-09-04T05:26:37.343" LastActivityDate="2010-09-04T05:26:37.343" CommentCount="12" />
  <row Id="2370" PostTypeId="1" CreationDate="2010-09-04T05:19:51.860" Score="4" ViewCount="90" Body="&lt;p&gt;I am trying fit a Cox Proportional Hazard model with coxph with a cmoputer of 12GB of RAM. It keeps running out of memory. Is there a biglm equivalent of coxph?&lt;/p&gt;&#xA;" OwnerUserId="1126" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-10-08T16:07:41.500" LastActivityDate="2010-10-08T16:07:41.500" Title="Is there a biglm equivalent for coxph?" Tags="&lt;r&gt;&lt;survival&gt;&lt;large-data&gt;" AnswerCount="1" FavoriteCount="2" />
  <row Id="2371" PostTypeId="2" ParentId="2370" CreationDate="2010-09-04T07:17:51.167" Score="4" ViewCount="" Body="&lt;p&gt;Maybe take a look at the &lt;a href=&quot;http://cran.r-project.org/web/packages/DatABEL/&quot; rel=&quot;nofollow&quot;&gt;DatABEL&lt;/a&gt; package. I know it is used in genomic studies with large data that may be stored on the HD instead of RAM. From what I read in the help file, you can then apply different kind of model, including survival model.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-04T11:18:53.117" LastActivityDate="2010-09-04T11:18:53.117" CommentCount="1" />
  <row Id="2372" PostTypeId="2" ParentId="2272" CreationDate="2010-09-04T10:22:19.790" Score="1" ViewCount="" Body="&lt;p&gt;I disagree with Srikant's answer on one fundamental point. Srikant stated this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Inference Problem:&#xA;Your inference problem is: What values of θ are reasonable given the observed data x?&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In fact this is the BAYESIAN INFERENCE PROBLEM. In Bayesian statistics we seek to calculate P(θ| x)  i.e the probability of the parameter value given the observed data (sample).  The CREDIBLE INTERVAL is an interval of θ that has a 95% chance (or other) of containing the true value of θ given the several assumptions underlying the problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The FREQUENTIST INFERENCE PROBLEM is this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are the observed data x reasonable given the hypothesised values of θ? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In frequentist statistics we seek to calculate P(x| θ) i.e the probability of observing the data (sample) given the hypothesised parameter value(s). The CONFIDENCE INTERVAL (perhaps a misnomer) is interpreted as: if the experiment that generated the random sample x were repeated many times, 95% (or other) of such intervals constructed from those random samples would contain the true value of the parameter. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Mess with your head? That's the problem with frequentist statistics and the main thing Bayesian statistics has going for it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As Sikrant points out, P(θ| x) and  P(x| θ) are related as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P(θ| x) = P(θ)P(x| θ)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where P(θ) is our prior probability; P(x| θ) is the probability of the data conditional on that prior and P(θ| x) is the posterior probability. The prior P(θ) is inherently subjective, but that is the price of knowledge about the Universe - in a very profound sense.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The other parts of both Sikrant's and Keith's answers are excellent.&lt;/p&gt;&#xA;" OwnerUserId="521" LastEditorUserId="521" LastEditorDisplayName="" LastEditDate="2010-09-04T10:28:57.360" LastActivityDate="2010-09-04T10:28:57.360" CommentCount="2" />
  <row Id="2373" PostTypeId="2" ParentId="2272" CreationDate="2010-09-04T11:07:44.643" Score="1" ViewCount="" Body="&lt;p&gt;As I understand it: A credible interval is a statement of the range of values for the statistic of interest that remain plausible given the particular sample of data that we have actually observed.  A confidence interval is a statement of the frequency with which the true value lies in the confidence interval when the experiment is repeated a large number of times, each time with a different sample of data from the same underlying population.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Normally the question we want to answer is &quot;what values of the statistic are consistent with the observed data&quot;, and the credible interval gives a direct answer to that question - the true value of the statistic lies in a 95% credible interval with probability 95%.  The confidence interval does not give a direct answer to this question; it is not correct to assert that the probability that the true value of the statistic lies within the 95% confidence interval is 95% (unless it happens to coincide with the credible interval).  However this is a very common misinterpretation of a frequentist confidence interval as it the interpretation that would be a direct answer to the question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The paper by Jayne's that I discuss in another question gives a good example of this (example #5), were a perfectly correct confidence interval is constructed, where the particular sample of data on which it is based rules out any possibility of the true value of the statistic being in the 95% confidence interval!  This is only a problem if the confidence interval is incorrectly interpreted as a statment of plausible values of the statistic on the basis of the particular sample we have observed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At the end of the day, it is a matter of &quot;horses for courses&quot;, and which interval is best depends on the question you want answered - just choose the method that directly answers that question.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suspect confidence intervals are more useful when analysing [desgined] repeatable experiments (as that is just the assumption underlying the confidence interval), and credible intervals better when analysing observational data, but that is just an opinion (I use both sorts of intervals in my own work, but wouldn't describe myself as an expert in either).&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2010-09-04T11:07:44.643" CommentCount="1" />
  <row Id="2374" PostTypeId="1" AcceptedAnswerId="2375" CreationDate="2010-09-04T11:15:48.317" Score="5" ViewCount="499" Body="&lt;p&gt;I used to analyse items from a psychometric point of view. But now I am trying to analyse other types of questions on motivation and other topics. These questions are all on likert scales. My initial thought was to use factor analysis, because the questions are hypothesised to reflect some underlying dimensions.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;But is factor analysis appropriate?&lt;/li&gt;&#xA;&lt;li&gt;Is it necessary to validate each question regarding its dimensionality ? &lt;/li&gt;&#xA;&lt;li&gt;Is there a problem with performing factor analysis on likert items?&lt;/li&gt;&#xA;&lt;li&gt;Are there any good papers and methods on how to conduct factor analysis on likert and other categorical items?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="1154" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-03-31T14:01:52.980" LastActivityDate="2011-03-31T14:01:52.980" Title="Factor analysis of questionnaires composed of Likert items" Tags="&lt;factor-analysis&gt;&lt;psychometrics&gt;&lt;likert&gt;&lt;psychology&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="3" />
  <row Id="2375" PostTypeId="2" ParentId="2374" CreationDate="2010-09-04T12:21:17.757" Score="8" ViewCount="" Body="&lt;p&gt;From what I've seen so far, FA is used for attitude items as it is for other kind of rating scales. The problem arising from the metric used (that is, &quot;are Likert scales really to be treated as numeric scales?&quot; is a long-standing debate, but providing you check for the bell-shaped response distribution you may handle them as continuous measurements, otherwise check for non-linear FA models or &lt;em&gt;optimal scaling&lt;/em&gt;) may be handled by polytmomous IRT models, like the Graded Response, Rating Scale, or Partial Credit Model. The latter two may be used as a rough check of whether the threshold distances, as used in Likert-type items, are a characteristic of the response format (RSM) or of the particular item (PCM).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regarding your second point, it is known, for example, that response distributions in attitude or health surveys differ from one country to the other (e.g. chinese people tend to highlight 'extreme' response patterns compared to those coming from western countries, see e.g. Song, X.-Y. (2007) Analysis of multisample structural equation models with applications to Quality of Life data, in &lt;em&gt;Handbook of Latent Variable and Related Models&lt;/em&gt;, Lee, S.-Y. (Ed.), pp 279-302, North-Holland). Some methods to handle such situation off the top of my head:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;use of log-linear models (marginal approach) to highlight strong between-groups imbalance at the item level (coefficients are then interpreted as relative risks instead of odds);&lt;/li&gt;&#xA;&lt;li&gt;the multi-sample SEM method from Song cited above (Don't know if they do further work on that approach, though).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Now, the point is that most of these approaches focus at the item level (ceiling/floor effect, decreased reliability, bad item fit statistics, etc.), but when one is interested in how people deviate from what would be expected from an ideal set of observers/respondents, I think we must focus on person fit indices instead.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Such $\chi^2$ statistics are readily available for IRT models, like INFIT or OUTFIT mean square, but generally they apply on the whole questionnaire. Moreover, since estimation of items parameters rely in part on persons parameters (e.g., in the marginal likelihood framework, we assume a gaussian distribution), the presence of outlying individuals may lead to potentially biased estimates and poor model fit. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As proposed by Eid and Zickar (2007), combining a latent class model (to isolate group of respondents, e.g. those always answering on the extreme categories vs. the others) and an IRT model (to estimate item parameters and persons locations on the latent trait in both groups) appears a nice solution. Other modeling strategies are described in their paper (e.g. HYBRID model, see also Holden and Book, 2009).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Likewise, &lt;a href=&quot;http://www.psychology.gatech.edu/unfolding/publications.html&quot; rel=&quot;nofollow&quot;&gt;unfolding models&lt;/a&gt; may be used to cope with &lt;em&gt;response style&lt;/em&gt;, which is defined as a consistent and content-independent pattern of response category (e.g. tendency to agree with all statements). In the social sciences or psychological literature, this is know as Extreme Response Style (ERS). References (1–3) may be useful to get an idea on how it manifests and how it may be measured.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a short list of papers that may help to progress on this subject:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Hamilton, D.L. (1968). &lt;a href=&quot;http://www.psych.umn.edu/faculty/waller/classes/meas08/Readings/Hamilton1968.pdf&quot; rel=&quot;nofollow&quot;&gt;Personality attributes associated with extreme response style&lt;/a&gt;. &lt;em&gt;Psychological Bulletin&lt;/em&gt;, &lt;strong&gt;69(3)&lt;/strong&gt;: 192–203.&lt;/li&gt;&#xA;&lt;li&gt;Greanleaf, E.A. (1992). Measuring extreme response style. &lt;em&gt;Public Opinion Quaterly&lt;/em&gt;, &lt;strong&gt;56(3)&lt;/strong&gt;: 328-351.&lt;/li&gt;&#xA;&lt;li&gt;de Jong, M.G., Steenkamp, J.-B.E.M., Fox, J.-P., and Baumgartner, H. (2008). Using Item Response Theory to Measure Extreme Response Style in Marketing Research: A Global Investigation. &lt;em&gt;Journal of marketing research&lt;/em&gt;, &lt;strong&gt;45(1)&lt;/strong&gt;: 104-115.&lt;/li&gt;&#xA;&lt;li&gt;Morren, M., Gelissen, J., and Vermunt, J.K. (2009). &lt;a href=&quot;http://spitswww.uvt.nl/~vermunt/morren2009.pdf&quot; rel=&quot;nofollow&quot;&gt;Dealing with extreme response style in cross-cultural research: A restricted latent class factor analysis approach&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Moors, G. (2003). Diagnosing Response Style Behavior by Means of a Latent-Class Factor Approach. Socio-Demographic Correlates of Gender Role Attitudes and Perceptions of Ethnic Discrimination Reexamined. &lt;em&gt;Quality &amp;amp; Quantity&lt;/em&gt;, 37(3), 277-302.&lt;/li&gt;&#xA;&lt;li&gt;de Jong, M.G. Steenkamp J.B., Fox, J.-P., and Baumgartner, H. (2008). Item Response Theory to Measure Extreme Response Style in Marketing Research: A Global Investigation. &lt;em&gt;Journal of Marketing Research&lt;/em&gt;, 45(1), 104-115.&lt;/li&gt;&#xA;&lt;li&gt;Javaras, K.N. and Ripley, B.D. (2007). An “Unfolding” Latent Variable Model for Likert Attitude Data. &lt;em&gt;JASA&lt;/em&gt;, 102(478): 454-463.&lt;/li&gt;&#xA;&lt;li&gt;slides from Moustaki, Knott and Mavridis, &lt;a href=&quot;http://www.rcec.nl/Publicaties/Downloads%2025ste%20IRT%20workshop/Irini%20Moustaki.pdf&quot; rel=&quot;nofollow&quot;&gt;Methods for detecting outliers in latent variable models&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Eid, M. and Zickar, M.J. (2007). Detecting response styles and faking in personality and organizational assessments by Mixed Rasch Models. In von Davier, M. and Carstensen, C.H. (Eds.), &lt;em&gt;Multivariate and Mixture Distribution Rasch Models&lt;/em&gt;, pp. 255–270, Springer.&lt;/li&gt;&#xA;&lt;li&gt;Holden, R.R. and Book, A.S. (2009). Using hybrid Rasch-latent class modeling to improve the detection of fakers on a personality inventory. &lt;em&gt;Personality and Individual Differences&lt;/em&gt;, &lt;strong&gt;47(3)&lt;/strong&gt;: 185-190.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-07T10:18:19.340" LastActivityDate="2010-09-07T10:18:19.340" CommentCount="2" />
  <row Id="2376" PostTypeId="2" ParentId="2348" CreationDate="2010-09-04T21:25:49.863" Score="1" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Metropolis%E2%80%93Hastings_algorithm&quot; rel=&quot;nofollow&quot;&gt;MH sampling&lt;/a&gt; is used when it's difficult to sample from the target distribution (e.g., when the prior isn't &lt;a href=&quot;http://en.wikipedia.org/wiki/Conjugate_prior&quot; rel=&quot;nofollow&quot;&gt;conjugate&lt;/a&gt; to the likelihood). So you use a proposal distribution to generate samples and accept/reject them based on the acceptance probability. The &lt;a href=&quot;http://en.wikipedia.org/wiki/Gibbs_sampling&quot; rel=&quot;nofollow&quot;&gt;Gibbs sampling&lt;/a&gt; algorithm is a particular instance of MH where the proposals are &lt;em&gt;always&lt;/em&gt; accepted. Gibbs sampling is one of the most commonly used algorithm due to its simplicity but it may not always to possible to apply, in which case one resorts to MH based on accept/reject proposals.&lt;/p&gt;&#xA;" OwnerUserId="881" LastActivityDate="2010-09-04T21:25:49.863" />
  <row Id="2377" PostTypeId="1" AcceptedAnswerId="2392" CreationDate="2010-09-04T23:00:04.117" Score="7" ViewCount="524" Body="&lt;p&gt;I am curious if there is a transform which alters the skew of a random variable without affecting the kurtosis. This would be analogous to how an affine transform of a RV affects the mean and variance, but not the skew and kurtosis (partly because the skew and kurtosis are defined to be invariant to changes in scale). Is this a known problem?&lt;/p&gt;&#xA;" OwnerUserId="795" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-09-08T08:09:20.603" LastActivityDate="2011-05-20T09:56:36.523" Title="A transform to change skew without affecting kurtosis?" Tags="&lt;data-transformation&gt;&lt;random-variable&gt;&lt;moments&gt;" AnswerCount="4" CommentCount="3" />
  <row Id="2378" PostTypeId="2" ParentId="2348" CreationDate="2010-09-05T03:23:37.750" Score="0" ViewCount="" Body="&lt;p&gt;In physics, statistical physics in particular, Metropolis-type algorithm(s) are used extensively. There are really countless variants of these, and the new ones are being actively developed. It's much too broad topic to give any sort of expanation here, so if you're interested you can start e.g. from &lt;a href=&quot;http://mcwa.csi.cuny.edu/umass/lectures.html&quot; rel=&quot;nofollow&quot;&gt;these lecture notes&lt;/a&gt; or from the ALPS library webpage (http://alps.comp-phys.org/mediawiki).&lt;/p&gt;&#xA;" OwnerUserId="1197" LastActivityDate="2010-09-05T03:23:37.750" CommentCount="1" />
  <row Id="2379" PostTypeId="1" AcceptedAnswerId="2415" CreationDate="2010-09-05T04:16:29.570" Score="21" ViewCount="447" Body="&lt;p&gt;Mathematics has its famous &lt;a href=&quot;http://en.wikipedia.org/wiki/Millennium_Prize_Problems&quot; rel=&quot;nofollow&quot;&gt;Millennium Problems&lt;/a&gt; (and, historically, &lt;a href=&quot;http://en.wikipedia.org/wiki/Hilbert%27s_problems&quot; rel=&quot;nofollow&quot;&gt;Hilbert's 23&lt;/a&gt;), questions that helped to shape the direction of the field.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have little idea, though, what the Riemann Hypotheses and P vs. NP's of statistics would be.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, what are the overarching open questions in statistics?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edited to add:&lt;/strong&gt;&#xA;As an example of the general spirit (if not quite specificity) of answer I'm looking for, I found a &quot;Hilbert's 23&quot;-inspired lecture by David Donoho at a &quot;Math Challenges of the 21st Century&quot; conference: &lt;a href=&quot;http://www-stat.stanford.edu/~donoho/Lectures/AMS2000/AMS2000.html&quot; rel=&quot;nofollow&quot;&gt;High-Dimensional Data Analysis: The Curses and Blessings of Dimensionality&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So a potential answer could talk about big data and why it's important, the types of statistical challenges high-dimensional data poses, and methods that need to be developed or questions that need to be answered in order to help solve the problem.&lt;/p&gt;&#xA;" OwnerUserId="1106" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-17T20:20:33.770" LastActivityDate="2010-09-17T20:20:33.770" CommunityOwnedDate="2010-09-05T04:16:29.570" Title="What are the 'big problems' in statistics?" Tags="&lt;for-retag&gt;" AnswerCount="6" CommentCount="1" FavoriteCount="6" />
  <row Id="2380" PostTypeId="2" ParentId="2379" CreationDate="2010-09-05T05:23:25.660" Score="3" ViewCount="" Body="&lt;p&gt;As an example of the general spirit (if not quite specificity) of answer I'm looking for, I found a &quot;Hilbert's 23&quot;-inspired lecture by David Donoho at a &quot;Math Challenges of the 21st Century&quot; conference:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www-stat.stanford.edu/~donoho/Lectures/AMS2000/AMS2000.html&quot; rel=&quot;nofollow&quot;&gt;High-Dimensional Data Analysis: The Curses and Blessings of Dimensionality&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1106" LastEditorUserId="1106" LastEditorDisplayName="" LastEditDate="2010-09-05T05:36:49.673" LastActivityDate="2010-09-05T05:36:49.673" CommunityOwnedDate="2010-09-05T05:23:25.660" CommentCount="1" />
  <row Id="2381" PostTypeId="2" ParentId="322" CreationDate="2010-09-05T06:49:22.653" Score="0" ViewCount="" Body="&lt;p&gt;Jaynes &lt;a href=&quot;http://omega.albany.edu:8008/ETJ-PDF/cc11g.pdf&quot; rel=&quot;nofollow&quot;&gt;shows&lt;/a&gt; how to derive Shannon's entropy from basic principles in his &lt;a href=&quot;http://omega.albany.edu:8008/JaynesBookPdf.html&quot; rel=&quot;nofollow&quot;&gt;book&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One idea is that if you approximate $n!$ by $n^n$, entropy is the rewriting of the following quantity&#xA;$$\frac{1}{n}\log \frac{n!}{(n p_1)!\cdots (n p_d)!}$$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The quantity inside the log is the number of different length n observation sequences over $d$ outcomes that are matched by distribution $p$, so it's a kind of a measure of explanatory power of the distribution.&lt;/p&gt;&#xA;" OwnerUserId="511" LastActivityDate="2010-09-05T06:49:22.653" />
  <row Id="2382" PostTypeId="2" ParentId="2379" CreationDate="2010-09-05T08:36:31.083" Score="1" ViewCount="" Body="&lt;p&gt;Mathoverflow has a similar question about &lt;a href=&quot;http://mathoverflow.net/questions/37151/what-are-the-big-problems-in-probability-theory&quot; rel=&quot;nofollow&quot;&gt;big problems in probability theory&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It would appear from that page that the biggest questions are to do with self avoiding random walks and percolations.&lt;/p&gt;&#xA;" OwnerUserId="352" LastEditorUserId="352" LastEditorDisplayName="" LastEditDate="2010-09-05T08:59:37.680" LastActivityDate="2010-09-05T08:59:37.680" CommunityOwnedDate="2010-09-05T08:36:31.083" CommentCount="1" />
  <row Id="2383" PostTypeId="2" ParentId="7" CreationDate="2010-09-05T09:22:57.927" Score="2" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://sites.google.com/site/munaga71/Home/datasetlinks&quot; rel=&quot;nofollow&quot;&gt;Here's another list&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="976" LastActivityDate="2010-09-05T09:22:57.927" CommunityOwnedDate="2010-09-05T09:22:57.927" />
  <row Id="2384" PostTypeId="1" AcceptedAnswerId="2387" CreationDate="2010-09-05T11:15:57.607" Score="1" ViewCount="161" Body="&lt;p&gt;Say I have a series of forecasts and observations like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;      EntityF   EntityO&#xA;2004  120       125&#xA;2006  166       173&#xA;2008  150       167   &#xA;2010  152       -&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And assume that the (i) entity is the same and (ii) the forecasting methodology is constant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd like to &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Produce a meaningful metric of the forecasting error.&lt;/li&gt;&#xA;&lt;li&gt;Be able to predict the current forecast (2010) error based on 1.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="722" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-16T06:33:13.167" LastActivityDate="2010-09-16T06:33:13.167" Title="Testing prediction time series against real data" Tags="&lt;time-series&gt;&lt;forecasting&gt;" AnswerCount="1" />
  <row Id="2385" PostTypeId="1" AcceptedAnswerId="2389" CreationDate="2010-09-05T11:37:47.670" Score="6" ViewCount="205" Body="&lt;p&gt;I have a very large data set which I would like to summarise in as small a space as possible, preferably one side of A4.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data are from a customer satisfaction survey and are Likert-type scales, 5 scales for each work area, with 190 work areas in total. I would also like to represent the response rate on the visualisation somehow, because response rates are very variable and I want management to look at these as well as the actual scores.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If necessary I don't mind somehow reducing the 5 scales down to one (using factor analysis or some such thing). One or two sides of A4 to go to the senior management team who are of course very busy with lots of other things and decidedly non-technical. Use of colour is no problem, in fact would probably be seen as a boon.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's just occurred to me that representing the order of the work areas, rather than their absolute value, would be OK, but again I don't want to lose the response rate information.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this question isn't too vague, any ideas gratefully received. I am using R and anticipate that this work will involve my learning ggplot2, which I have not as yet got around to.&lt;/p&gt;&#xA;" OwnerUserId="199" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-10-08T16:06:54.717" LastActivityDate="2010-10-08T16:06:54.717" Title="Data visualisation- summarise 190 means and response rates" Tags="&lt;data-visualization&gt;&lt;large-data&gt;" AnswerCount="3" CommentCount="3" FavoriteCount="1" />
  <row Id="2386" PostTypeId="2" ParentId="2379" CreationDate="2010-09-05T11:43:35.673" Score="0" ViewCount="" Body="&lt;p&gt;My answer would be the struggle between frequentist and Bayesian statistics. When people ask you which you &quot;believe in&quot;, this is not good! Especially for a scientific discipline.&lt;/p&gt;&#xA;" OwnerUserId="561" LastActivityDate="2010-09-05T11:43:35.673" CommunityOwnedDate="2010-09-05T11:43:35.673" CommentCount="2" />
  <row Id="2387" PostTypeId="2" ParentId="2384" CreationDate="2010-09-05T12:50:11.763" Score="4" ViewCount="" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;You could use Mean Absolute Error&#xA;(mean of $|F-O|$) or Mean Squared&#xA;Error (mean of $(F-O)^2$)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If your forecast method is unbiased, then the best estimate of a future forecast error is 0 and the variance of the forecast error can be estimated by the MSE.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-09-05T12:50:11.763" />
  <row Id="2388" PostTypeId="2" ParentId="2385" CreationDate="2010-09-05T13:22:42.097" Score="2" ViewCount="" Body="&lt;p&gt;I would suggest you check out either box-plots (if you have an intro text to R, box plots always seem to be one of the first plots they use), or you can plot the means of each group on the Y axis and use the X-axis to represent each of your 190 work areas (and then maybe put error bars representing a confidence interval for the estimate of the mean).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can plot each of the likert scales next to each other, and use a different color to represent the means, and as long as you choose distinct colors and the same order for your likert scales across work areas people will be able to distinguish them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I personally would only plot the scales next to each other if they are expected to have some sort of relationship with each other (if scale A is high I might expect scale B to be low). If they are not you could panel the charts on top of each other (check out the lattice package in R, and &lt;a href=&quot;http://processtrends.com/pg_charts_stacked_transformed_data.htm&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; is what I think is a good example with sample R code), and so you only need to label one X-axis (this also allows you to use different Y-axis scales if the scales are not easily plotted on all the same Y-levels, although by your description this doesn't seem to be the case). You could also include response rate as one of the panels (maybe represented as a bar).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is difficult with 190 different groups is you will have trouble distinguishing different work groups unless you highlight specific groups, but any chart with all of the groups will be excellent to examine overall trends (and maybe spot outliers). Also if your work groups have no logical ordering or higher order groupings the orientation on the axis will be arbitrary. You could order according to values on one of the scales (or according to response rate).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also I am personally learning R at the moment, and I would highly suggest you check out the &lt;a href=&quot;http://www.springer.com/series/6991?detailsPage=titles&quot; rel=&quot;nofollow&quot;&gt;Use R!&lt;/a&gt; series by Springer. The book A Beginner's Guide to R is one of the best intro texts I have encountered, and they have books on ggplot2 and the lattice packages that would likely help you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an end if you post some examples of plots and code to make them some more of the R savy crowd on the forum will likely be able to give you suggestions. When you do finish come back and post your results! &#xA;HTH and good luck.&lt;/p&gt;&#xA;" OwnerUserId="1036" LastActivityDate="2010-09-05T13:22:42.097" CommentCount="2" />
  <row Id="2389" PostTypeId="2" ParentId="2385" CreationDate="2010-09-05T15:12:53.097" Score="5" ViewCount="" Body="&lt;p&gt;I find a heatmap to be one of the most effective ways of summarizing large amounts of multi-dimensional data in a confined space.  The LearnR blog has &lt;a href=&quot;http://learnr.wordpress.com/2010/01/26/ggplot2-quick-heatmap-plotting/&quot; rel=&quot;nofollow&quot;&gt;a nice example&lt;/a&gt; of creating one in ggplot2.&lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-09-05T15:12:53.097" />
  <row Id="2390" PostTypeId="1" AcceptedAnswerId="2394" CreationDate="2010-09-05T15:59:43.343" Score="5" ViewCount="390" Body="&lt;p&gt;I'm familiar with what the 2nd moment (variance) indicates as well as what the 3rd moment (skewness) indicates.  I know that on a histogram the 4th moment (kurtosis) indicates the &quot;peeked-ness&quot; of the data.  My question asks what are the practical implications/interpretations of a kurtotic distribution.  I'm asking this because I haven't found a case yet where I thought the 4th moment was theoretically interesting and interpretable.  I understand that such interpretations/implications are likely to be dataset specific, so I am looking for descriptions of example datasets where the kurtosis of the distribution was theoretically interesting and interpretable.&lt;/p&gt;&#xA;" OwnerUserId="196" LastActivityDate="2010-09-06T02:58:30.540" Title="What practical implications/interpretations are there of a kurtotic distribution?" Tags="&lt;definition&gt;&lt;interpretation&gt;&lt;kurtosis&gt;" AnswerCount="3" FavoriteCount="1" />
  <row Id="2391" PostTypeId="1" AcceptedAnswerId="2424" CreationDate="2010-09-05T16:35:45.123" Score="3" ViewCount="747" Body="&lt;p&gt;Suppose that I have three populations with four, mutually exclusive characteristics.  I take random samples from each population and construct a crosstab or frequency table for the characteristics that I am measuring.  Am I correct in saying that:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) If I wanted to test whether there is any relationship between the populations and the characteristics (e.g. whether one population has a higher frequency of one of the characteristics), I should run a chi-squared test and see whether the result is significant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) If the chi-squared test is significant, it only shows me that there is some relationship between the populations and characteristics, but not how they are related.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) Furthermore, not all of the characteristics need to be related to the population.  For example, if the different populations have significantly different distributions of characteristics A and B, but not of C and D, then the chi-squared test may still come back as significant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4) If I wanted to measure whether or not a specific characteristic is affected by the population, then I can run a test for equal proportions (I have seen this called a z-test, or as prop.test() in R) on just that characteristic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words, is it appropriate to use the prop.test() to more accurately determine the nature of a relationship between two sets of categories when the chi-squared test says that there is a significant relationship?&lt;/p&gt;&#xA;" OwnerUserId="1195" LastActivityDate="2010-09-07T15:12:36.180" Title="What is the relationship between a chi square test and test of equal proportions?" Tags="&lt;contingency-tables&gt;&lt;chi-squared&gt;&lt;proportion&gt;" AnswerCount="2" />
  <row Id="2392" PostTypeId="2" ParentId="2377" CreationDate="2010-09-05T17:07:00.207" Score="2" ViewCount="" Body="&lt;p&gt;My answer is the beginnings of a total hack, but I am not aware of any established way to do what you ask.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My first step would be to rank order your dataset you can find the proportional position in your dataset and then transform it to a normal distribution, this method was used in Reynolds &amp;amp; Hewitt, 1996.  See sample R code below in PROCMiracle.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once the distribution is normal, then the problem has been turned on its head - a matter of adjusting kurtosis but not skew.  A google search suggested that one could follow the procedures of John &amp;amp; Draper, 1980 to adjust the kurtosis but not the skew - but I could not replicate that result.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My attempts to develop a crude spreading/narrowing function that takes the input (normalized) value and adds or subtracts a value from it proportional to the position of the variable on the normal scale does result in a monotonic adjustment, but in practice tends to create a bimodal distribution though one that has the desired skewness and kurtosis values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I realize this is not a complete answer, but I thought it might provide a step in the right direction.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;PROCMiracle &amp;lt;- function(datasource,normalrank=&quot;BLOM&quot;)&#xA;  {&#xA;     switch(normalrank,&#xA;      &quot;BLOM&quot; = {&#xA;                  rmod &amp;lt;- -3/8&#xA;                  nmod &amp;lt;- 1/4&#xA;                },&#xA;      &quot;TUKEY&quot; = {&#xA;                  rmod &amp;lt;- -1/3&#xA;                  nmod &amp;lt;- 1/3&#xA;                },&#xA;      &quot;VW&quot; ={&#xA;                  rmod &amp;lt;- 0&#xA;                  nmod &amp;lt;- 1&#xA;            },&#xA;      &quot;NONE&quot; = {&#xA;                  rmod &amp;lt;- 0&#xA;                  nmod &amp;lt;- 0&#xA;                }&#xA;    )&#xA;    print(&quot;This may be doing something strange with NA values!  Beware!&quot;)&#xA;    return(scale(qnorm((rank(datasource)+rmod)/(length(datasource)+nmod))))&#xA;  }&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="196" LastActivityDate="2010-09-05T17:07:00.207" CommentCount="3" />
  <row Id="2394" PostTypeId="2" ParentId="2390" CreationDate="2010-09-05T18:55:00.790" Score="9" ViewCount="" Body="&lt;p&gt;The kurtosis also indicates the &quot;fat tailedness&quot; of the distribution. A distribution with high kurtosis will have many extreme events (events far away from the center) and many &quot;typical&quot; events (events near the center). A distribution with low kurtosis will have events a moderate distance from the center. &#xA;This picture may help:  &lt;a href=&quot;http://mvpprograms.com/help/images/KurtosisPict.jpg&quot; rel=&quot;nofollow&quot;&gt;http://mvpprograms.com/help/images/KurtosisPict.jpg&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1146" LastActivityDate="2010-09-05T18:55:00.790" CommentCount="2" />
  <row Id="2395" PostTypeId="2" ParentId="2379" CreationDate="2010-09-05T19:18:58.237" Score="2" ViewCount="" Body="&lt;p&gt;You might check out Harvard's &lt;a href=&quot;http://socialscience.fas.harvard.edu/hardproblems&quot; rel=&quot;nofollow&quot;&gt;&quot;Hard Problems in the Social Sciences' colloquium&lt;/a&gt; held earlier this year. Several of these talks offer issues in the use of statistics and modeling in the social sciences.&lt;/p&gt;&#xA;" OwnerUserId="401" LastActivityDate="2010-09-05T19:18:58.237" CommunityOwnedDate="2010-09-05T19:18:58.237" />
  <row Id="2396" PostTypeId="2" ParentId="2379" CreationDate="2010-09-05T19:19:03.197" Score="3" ViewCount="" Body="&lt;p&gt;I'm not sure how big they are, but there is a &lt;a href=&quot;http://en.wikipedia.org/wiki/Unsolved_problems_in_Statistics&quot; rel=&quot;nofollow&quot;&gt;Wikipedia page&lt;/a&gt; for unsolved problems in statistics.&lt;/p&gt;&#xA;" OwnerUserId="196" LastActivityDate="2010-09-05T19:19:03.197" CommunityOwnedDate="2010-09-05T19:19:03.197" />
  <row Id="2397" PostTypeId="1" AcceptedAnswerId="2408" CreationDate="2010-09-05T19:52:53.743" Score="4" ViewCount="509" Body="&lt;p&gt;I'm looking for the limiting distribution of multinomial distribution over d outcomes. IE, the distribution of the following&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\lim_{n\to \infty} n^{-\frac{1}{2}} \mathbf{X_n}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $\mathbf{X_n}$ is a vector value random variable with density $f_n(\mathbf{x})$ for $\mathbf{x}$ such that $\sum_i x_i=n$, $x_i\in \mathbb{Z}, x_i\ge 0$ and 0 for all other $\mathbf{x}$, where&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$f_{n}(\mathbf{x})=\frac{n!}{x_1\cdots x_d} p_1^{x_1}\cdots p_d^{x_d}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found one form in Larry Wasserman's &quot;All of Statistics&quot; Theorem 14.6, &lt;a href=&quot;http://yaroslavvb.com/upload/wasserman-multinomial.pdf&quot; rel=&quot;nofollow&quot;&gt;page 237&lt;/a&gt; but for limiting distribution it gives Normal with a singular covariance matrix, so I'm not sure how to normalize that. You could project the random vector into (d-1)-dimensional space to make covariance matrix full-rank, but what projection to use?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;b&gt;Update 11/5&lt;/b&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ray Koopman has a nice &lt;a href=&quot;http://groups.google.com/group/sci.stat.math/browse_thread/thread/add687f2a741c5f6/1178f6b77c8b07c7?q=singular+author:koopman#1178f6b77c8b07c7&quot; rel=&quot;nofollow&quot;&gt;summary&lt;/a&gt; of the problem of singular Gaussian. Basically, singular covariance matrix represents perfect correlation between variables, which is not possible to represent with a Gaussian. However, one could get a Gaussian distribution for the conditional density, conditioned on the fact that the value of random vector is valid (components add up to $n$ in the case above). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The difference for the conditional Gaussian, is that inverse is replaced with pseudo-inverse, and normalization factor uses &quot;product of non-zero eigenvalues&quot; instead of &quot;product of all eigenvalues&quot;. Ian Frisce gives &lt;a href=&quot;http://fedc.wiwi.hu-berlin.de/xplore/tutorials/mvahtmlnode34.html&quot; rel=&quot;nofollow&quot;&gt;link&lt;/a&gt; with some details.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There's also a way to express normalization factor of conditional Gaussian without referring to eigenvalues, &#xA;&lt;a href=&quot;http://math.stackexchange.com/questions/4106/normalization-factor-for-restricted-density&quot;&gt;here&lt;/a&gt;'s a derivation&lt;/p&gt;&#xA;" OwnerUserId="511" LastEditorUserId="511" LastEditorDisplayName="" LastEditDate="2010-11-06T02:01:08.523" LastActivityDate="2010-11-06T02:01:08.523" Title="Asymptotic distribution of multinomial" Tags="&lt;asymptotics&gt;&lt;multinomial&gt;" AnswerCount="4" CommentCount="5" />
  <row Id="2398" PostTypeId="2" ParentId="2397" CreationDate="2010-09-05T22:04:14.960" Score="1" ViewCount="" Body="&lt;p&gt;It looks to me like Wasserman's covariance matrix is singular, to see, multiply it by a vector of $d$ ones, i.e. $[1,1,1,\dots,1]^\prime$ of length $d$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Multinomial_distribution&quot; rel=&quot;nofollow&quot;&gt;Wikipedia&lt;/a&gt; gives the same covariance matrix anyway. If we restrict ourselves to just a binomial distribution then the standard central limit theorem tells us that the binomial distribution (after appropriate scaling) converges to the normal as $n$ gets big (see &lt;a href=&quot;http://en.wikipedia.org/wiki/Binomial_distribution#Normal_approximation&quot; rel=&quot;nofollow&quot;&gt;wikipedia again&lt;/a&gt;). Applying similar ideas you should be able to show that an appropriately scaled mulinomial is going to converge in distribution to the multivariate normal, i.e. each marginal distribution is just a binomial and converges to the normal distribution, and the variance between them is known.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, I am very confident you will find that the distribution of&#xA;$$\frac{X_n - np}{\sqrt{n}}$$&#xA;converges to the multivariate normal with zero mean and covariance &#xA;$$\frac{C}{n}$$&#xA;where $C$ is the covariance matrix of the multinomial in question and $p$ is the vector of probabilities $[p_1,\dots,p_d]$.&lt;/p&gt;&#xA;" OwnerUserId="352" LastEditorUserId="352" LastEditorDisplayName="" LastEditDate="2010-09-05T22:20:30.703" LastActivityDate="2010-09-05T22:20:30.703" CommentCount="3" />
  <row Id="2399" PostTypeId="2" ParentId="2390" CreationDate="2010-09-05T22:41:39.633" Score="0" ViewCount="" Body="&lt;p&gt;There is the &lt;a href=&quot;http://en.wikipedia.org/wiki/Kurtosis_risk&quot; rel=&quot;nofollow&quot;&gt;Kurtosis risk&lt;/a&gt; which isn't explained fantastically well at that link. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, measures of normality (or deviation therefrom) are crucial if you are using analyses that assume normality. For example, the standard workhorse Pearson-&lt;em&gt;r&lt;/em&gt; correlation coefficient is severely sensitive to outliers and becomes essentially invalid as excess kurtosis deviates from 0. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;em&gt;K²&lt;/em&gt; test is often used to check a distribution for normality and incorporates the sample kurtosis as a factor.&lt;/p&gt;&#xA;" OwnerUserId="869" LastActivityDate="2010-09-05T22:41:39.633" />
  <row Id="2400" PostTypeId="2" ParentId="322" CreationDate="2010-09-05T23:52:45.783" Score="0" ViewCount="" Body="&lt;p&gt;Grünwald and Dawid's paper &lt;em&gt;&lt;a href=&quot;http://projecteuclid.org/euclid.aos/1091626173&quot; rel=&quot;nofollow&quot;&gt;Game theory, maximum entropy, minimum discrepancy and robust Bayesian decision theory&lt;/a&gt;&lt;/em&gt; discuss generalisations of the traditional notion of entropy. Given a loss, its associated entropy function is the mapping from a distribution to the minimal achievable expected loss for that distribution. The usual entropy function is the generalised entropy associated with the log loss. Other choices of losses yield different entropy such as the Rényi entropy.&lt;/p&gt;&#xA;" OwnerUserId="1201" LastActivityDate="2010-09-05T23:52:45.783" CommentCount="3" />
  <row Id="2401" PostTypeId="1" AcceptedAnswerId="2409" CreationDate="2010-09-06T00:58:21.607" Score="5" ViewCount="330" Body="&lt;p&gt;Hi folks,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've never liked how people typically analyze data from Likert scales as if error were continuous &amp;amp; gaussian when there are reasonable expectations that these assumptions are violated at least at the extremes of the scales. What do you think of the following alternative:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the response takes value k on an n-point scale, expand that data to n trials, k of which have the value 1 and n-k of which have the value 0. Thus, we're treating response on a likert scale as if it is the overt aggregate of a covert series of binomial trials (in fact, from a cognitive science perspective, this is actually an appealing model for the mechanisms involved in such decision making scenarios). With the expanded data, you can now use a mixed effects model specifying respondent as a random effect (also question as a random effect if you have multiple questions) and using the binomial link function to specify the error distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone see any assumption violations or other detrimental aspects of this approach?&lt;/p&gt;&#xA;" OwnerUserId="364" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-03-30T18:43:17.430" LastActivityDate="2011-03-30T18:43:17.430" Title="Is it appropriate to treat n-point Likert scale data as n trials from a binomial process?" Tags="&lt;binomial&gt;&lt;psychometrics&gt;&lt;likert&gt;&lt;psychology&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="1" />
  <row Id="2402" PostTypeId="2" ParentId="2401" CreationDate="2010-09-06T01:15:20.940" Score="5" ViewCount="" Body="&lt;p&gt;If you really wish to abandon the assumption of interval level data for likert scales I would suggest that you assume the data to be a ordered logit or probit instead. Likert scales usually measure strength of response and hence higher values should indicate a stronger response on the underlying item of interest. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose that you have a $H$ item scale and that $S$ represents the unobserved strength of response on the item of interest. Then you can assume the following response model:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$y = 1$ if $S \le \alpha_1$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$y = h\ $  if $\alpha_{h-1} &amp;lt; S\  \le \alpha_h$ for $h = 2, 3, ..H-1$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$y = H\ $  if $\alpha_{H-1} &amp;lt; S  &amp;lt;\  \infty$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming that $S$ follows a normal distribution with an unknown mean and variance would give a ordered probit model.&lt;/p&gt;&#xA;" LastActivityDate="2010-09-06T01:21:13.017" />
  <row Id="2403" PostTypeId="2" ParentId="2377" CreationDate="2010-09-06T02:32:10.617" Score="1" ViewCount="" Body="&lt;p&gt;Another possible interesting technique has come to mind, though this doesn't quite answer the question, is to transform a sample to have a fixed sample L-skew and sample L-kurtosis (as well as a fixed mean and L-scale). These four constraints are linear in the order statistics. To keep the transform monotonic on a sample of $n$ observations would then require another $n-1$ equations. This could then be posed as a quadratic optimization problem: minimize the $\ell_2$ norm between the sample order statistics and the transformed version subject to the given constraints. This is a kind of wacky approach, though. In the original question, I was looking for something more basic and fundamental. I also was implicitly looking for a technique which could be applied to individual observations, independent of having an entire cohort of samples.&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-09-06T02:32:10.617" />
  <row Id="2404" PostTypeId="2" ParentId="2390" CreationDate="2010-09-06T02:58:30.540" Score="2" ViewCount="" Body="&lt;p&gt;I seem to remember that the median has a smaller standard error than the mean when the samples are drawn from a leptokurtic distribution, but the mean has a smaller standard error when the distribution is platykurtic. I think I read this in one of Wilcox' books. Thus the kurtosis may dictate which kinds of locational tests one uses..&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-09-06T02:58:30.540" />
  <row Id="2405" PostTypeId="1" AcceptedAnswerId="2406" CreationDate="2010-09-06T04:14:20.773" Score="4" ViewCount="153" Body="&lt;p&gt;Suppose that I have a market research survey with the question &quot;What brand of television are you planning on buying&quot; and then also have a choice that is &quot;I don't plan on buying a television.&quot;  Respondents may choose more than one brand, but they may not make any other choices if they select that they do not plan on buying a television.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;If I wanted to do an analysis of television brands among only those people who said they were going to buy a television, can I remove all of the respondents that said they would not buy a television, then recalculate the proportions for each brand with the new base number of observations?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If this skews the demographic distribution of my sample, is that okay? (e.g. the full sample has 30% Caucasians and 10% Asians, but the sample of people who say they will buy televisions is 30% Asians and 10% Caucasians.  It seems like this should be okay, since we would not expect the two populations (TV buyers and people in general) to be the same.&lt;/p&gt;&#xA;" OwnerUserId="1195" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-16T12:43:13.977" LastActivityDate="2010-09-16T12:43:13.977" Title="Survey questionnaire rebasing after removal of a selection" Tags="&lt;survey&gt;" AnswerCount="1" />
  <row Id="2406" PostTypeId="2" ParentId="2405" CreationDate="2010-09-06T06:45:09.040" Score="4" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;The short answer:&lt;/strong&gt; what you propose to do sounds reasonable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It often occurs in survey research that a question only applies to a subset of the population.&#xA;In such situations you typically want to say:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&quot;Of American/Australian/French/etc. adults where this question is applicable, $X\%$ believes/intends to do/thinks/etc. ...&quot;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Representative:&lt;/strong&gt; As you say, in survey research you are often concerned with your findings being representative of a target population. If we assume that your initial sample is representative, then taking just the subset that intend to buy a television will be a representative sample of your population that intends to buy a television. This is true even if the subset that intends to buy a television is different to the subset that does not intend to buy a television in terms of demographics or some other factor in addition&#xA;to television purchasing intention.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In addition, you may want to produce cross-tabulations of purchase intention with demographics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Split Question&lt;/strong&gt;: I suppose you also need to think about what is the best split-question. Consumers vary in the degree to which they are intending to purchase a television and the degree to which they have thought about brands. Thus, you need to think about your research question. For example, you might ask the question: &quot;if you were going to purchase a television, which brand would you consider?&quot; This would be good for assessing longer term  brand purchasing intentions. Your structure focuses on the subset of consumers who are planning to purhcase.&lt;/p&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-09-06T08:06:05.930" LastActivityDate="2010-09-06T08:06:05.930" CommentCount="4" />
  <row Id="2407" PostTypeId="2" ParentId="2385" CreationDate="2010-09-06T07:48:01.307" Score="2" ViewCount="" Body="&lt;p&gt;To give you a few more things to look at:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Principal components - look at some previous answers about &lt;a href=&quot;http://stats.stackexchange.com/questions/tagged/pca&quot;&gt;PC&lt;/a&gt;. In particular, this &lt;a href=&quot;http://stats.stackexchange.com/questions/1289/visualizing-multiple-histograms&quot;&gt;answer&lt;/a&gt; may be helpful.&lt;/li&gt;&#xA;&lt;li&gt;Cluster analysis. This &lt;a href=&quot;http://www.statmethods.net/advstats/cluster.html&quot; rel=&quot;nofollow&quot;&gt;page&lt;/a&gt; gives quite a nice overall in R.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I would recommend trying as many things as possible and see what comes out. Once you have your data in R in a reasonable format, it shouldn't take too long to try these things.&lt;/p&gt;&#xA;" OwnerUserId="8" LastActivityDate="2010-09-06T07:48:01.307" />
  <row Id="2408" PostTypeId="2" ParentId="2397" CreationDate="2010-09-06T10:36:39.710" Score="3" ViewCount="" Body="&lt;p&gt;The covariance is still non-negative definite (so is a valid &lt;a href=&quot;http://en.wikipedia.org/wiki/Multivariate_normal_distribution&quot; rel=&quot;nofollow&quot;&gt;multivariate normal distribution&lt;/a&gt;), but &lt;em&gt;not&lt;/em&gt; positive definite: what this means is that (at least) one element of the random vector is a linear combination of the others.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a result, any draw from this distribution will always lie on a subspace of $R^d$. As a consequence, this means it is not possible to define a density function (as the distribution is concentrated on the subspace:  think of the way a univariate normal will concentrate at the mean if the variance is zero).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, as suggested by Robby McKilliam, in this case you can drop the last element of the random vector. The covariance matrix of this reduced vector will be the original matrix, with the last column and row dropped, which will now be positive definite, and will have a density (this trick will work in other cases, but you have to be careful which element you drop, and you may need to drop more than one). &lt;/p&gt;&#xA;" OwnerUserId="495" LastEditorUserId="495" LastEditorDisplayName="" LastEditDate="2010-09-06T11:28:12.913" LastActivityDate="2010-09-06T11:28:12.913" CommentCount="5" />
  <row Id="2409" PostTypeId="2" ParentId="2401" CreationDate="2010-09-06T10:43:15.640" Score="5" ViewCount="" Body="&lt;p&gt;I don't know of any articles related to your question in the psychometric litterature. It seems to me that ordered logistic models allowing for random effect components can handle this situation pretty well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I agree with @Srikant and think that a proportional odds model or an ordered probit model (depending on the link function you choose) might better reflect the intrinisc coding of Likert items, and their typical use as rating scales in opinion/attitude survey or questionnaire. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other alternatives are: (1) use of adjacent instead of proportional or cumulative categories (where there is a connection with log-linear models); (2) use of item-response models like the partial-credit model or the rating scale model (as was mentioned in my response on &lt;a href=&quot;http://stats.stackexchange.com/questions/2374/likert-scales-analysis/2375#2375&quot;&gt;Likert scales analysis&lt;/a&gt;). The latter case is comparable to a mixed-effects approach, with subjects treated as random effects, and is readily available in the SAS system (e.g., &lt;a href=&quot;http://brm.psychonomic-journals.org/content/34/2/151.full.pdf&quot; rel=&quot;nofollow&quot;&gt;Fitting mixed-effects models for repeated ordinal outcomes with the NLMIXED procedure&lt;/a&gt;) or R (see &lt;a href=&quot;http://www.jstatsoft.org/v20&quot; rel=&quot;nofollow&quot;&gt;vol. 20&lt;/a&gt; of the &lt;em&gt;Journal of Statistical Software&lt;/em&gt;). You might also be interested in the discussion provided by John Linacre about &lt;a href=&quot;http://www.winsteps.com/a/linacre-optimizing-category.pdf&quot; rel=&quot;nofollow&quot;&gt;Optimizing Rating Scale Category Effectiveness&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The following papers may also be useful:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Wu, C-H (2007). &lt;a href=&quot;http://www.m-hikari.com/ams/ams-password-2007/ams-password57-60-2007/wuchienhoAMS57-60-2007.pdf&quot; rel=&quot;nofollow&quot;&gt;An Empirical Study on the Transformation of Likert-scale Data to Numerical Scores&lt;/a&gt;. &lt;em&gt;Applied Mathematical Sciences&lt;/em&gt;, &lt;strong&gt;1(58)&lt;/strong&gt;: 2851-2862.&lt;/li&gt;&#xA;&lt;li&gt;Rost, J and and Luo, G (1997). &lt;a href=&quot;http://ipn.uni-kiel.de/aktuell/buecher/rostbuch/c26.pdf&quot; rel=&quot;nofollow&quot;&gt;An Application of a Rasch-Based Unfolding Model to a Questionnaire on Adolescent Centrism&lt;/a&gt;. In Rost, J and Langeheine, R (Eds.), &lt;em&gt;Applications of latent trait and latent class models in the social sciences&lt;/em&gt;, New York: Waxmann.&lt;/li&gt;&#xA;&lt;li&gt;Lubke, G and Muthen, B (2004). &lt;a href=&quot;http://gseis.ucla.edu/faculty/muthen/Likart.pdf&quot; rel=&quot;nofollow&quot;&gt;Factor-analyzing Likert-scale data under the assumption of multivariate normality complicates a meaningful comparison of observed groups or latent classes&lt;/a&gt;. &lt;em&gt;Structural Equation Modeling&lt;/em&gt;, &lt;strong&gt;11&lt;/strong&gt;: 514-534. &lt;/li&gt;&#xA;&lt;li&gt;Nering, ML and Ostini, R (2010). &lt;em&gt;Handbook of Polytomous Item Response Theory Models&lt;/em&gt;. Routledge Academic&lt;/li&gt;&#xA;&lt;li&gt;Bender R and Grouven U (1998). Using binary logistic regression models for ordinal data with non-proportional odds. &lt;em&gt;Journal of Clinical Epidemiology&lt;/em&gt;, &lt;strong&gt;51(10)&lt;/strong&gt;: 809-816. (Cannot find the pdf but this one is available, &lt;a href=&quot;http://www.rbsd.de/PDF/olr_mr.pdf&quot; rel=&quot;nofollow&quot;&gt;Ordinal logistic regression in medical research&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-06T11:41:43.773" LastActivityDate="2010-09-06T11:41:43.773" />
  <row Id="2410" PostTypeId="1" AcceptedAnswerId="2411" CreationDate="2010-09-06T14:59:09.557" Score="5" ViewCount="175" Body="&lt;p&gt;A colleague in my office said to me today &quot;Tree models aren't good because they get caught by extreme observations&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A search here resulted in &lt;a href=&quot;http://stats.stackexchange.com/questions/1292/what-is-the-weak-side-of-decision-trees&quot;&gt;this thread&lt;/a&gt; that basically supports the claim.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which leads me to the question - under what situation can a CART model be robust, and how is that shown?&lt;/p&gt;&#xA;" OwnerUserId="253" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2011-01-27T10:43:05.190" LastActivityDate="2011-01-27T10:43:05.190" Title="Can CART models be made robust?" Tags="&lt;regression&gt;&lt;classification&gt;&lt;robust&gt;&lt;decision-trees&gt;&lt;cart&gt;" AnswerCount="2" FavoriteCount="4" />
  <row Id="2411" PostTypeId="2" ParentId="2410" CreationDate="2010-09-06T15:20:06.863" Score="9" ViewCount="" Body="&lt;p&gt;No, not in their present forms.&#xA;The problem is that convex loss functions cannot be made to be robust to contamination by outliers (this is a well known fact since the 70's but keeps being rediscovered periodically, see for instance this paper for one recent such re-discovery):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.cs.columbia.edu/~rocco/Public/mlj9.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.cs.columbia.edu/~rocco/Public/mlj9.pdf&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, in the case of regression trees, the fact that CART uses marginals (or alternatively univariate projections) can be used: &#xA;one can think of a version of CART where the s.d. criterion is replaced by a more&#xA;robust counterpart (MAD or better yet, Qn estimator).&lt;/p&gt;&#xA;" OwnerUserId="603" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-09-07T03:23:26.940" LastActivityDate="2010-09-07T03:23:26.940" CommentCount="8" />
  <row Id="2412" PostTypeId="2" ParentId="2410" CreationDate="2010-09-06T15:20:57.673" Score="4" ViewCount="" Body="&lt;p&gt;You might consider using &lt;a href=&quot;http://en.wikipedia.org/wiki/Leo_Breiman&quot; rel=&quot;nofollow&quot;&gt;Breiman's&lt;/a&gt; bagging or &lt;a href=&quot;http://en.wikipedia.org/wiki/Random_forest&quot; rel=&quot;nofollow&quot;&gt;random forests&lt;/a&gt;.  One good reference is &lt;a href=&quot;http://www.springerlink.com/content/l4780124w2874025/&quot; rel=&quot;nofollow&quot;&gt;Breiman &quot;Bagging Predictors&quot;&lt;/a&gt; (1996).  Also summarized in Clifton Sutton's &lt;a href=&quot;http://mason.gmu.edu/~csutton/vt6.pdf&quot; rel=&quot;nofollow&quot;&gt;&quot;Classification and Regression Trees, Bagging, and Boosting&quot;&lt;/a&gt; in the Handbook of Statistics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also see &lt;a href=&quot;http://cran.r-project.org/doc/Rnews/Rnews_2002-3.pdf&quot; rel=&quot;nofollow&quot;&gt;Andy Liaw and Matthew Wiener R News discussion&lt;/a&gt; of the randomForest package.&lt;/p&gt;&#xA;" OwnerUserId="5" LastEditorUserId="5" LastEditorDisplayName="" LastEditDate="2010-09-06T15:30:27.687" LastActivityDate="2010-09-06T15:30:27.687" CommentCount="3" />
  <row Id="2413" PostTypeId="2" ParentId="328" CreationDate="2010-09-06T16:26:44.860" Score="4" ViewCount="" Body="&lt;p&gt;You should check out &lt;a href=&quot;http://area51.stackexchange.com/proposals/117/quantitative-finance?referrer=b3Z9BBygZU6P1xPZSakPmQ2&quot;&gt;http://area51.stackexchange.com/proposals/117/quantitative-finance?referrer=b3Z9BBygZU6P1xPZSakPmQ2&lt;/a&gt;, they are trying to start one on stackexhange.com&lt;/p&gt;&#xA;" OwnerUserId="1137" LastActivityDate="2010-09-06T16:26:44.860" CommentCount="1" />
  <row Id="2414" PostTypeId="2" ParentId="2377" CreationDate="2010-09-06T17:03:56.570" Score="0" ViewCount="" Body="&lt;p&gt;Please forgive me if I have misunderstood the question, but it seems that almost any nonlinear monotonic function will do, provided you allow yourself to follow up its application by re-standardizing the result to match the kurtosis of the original RV.  The only limitation is that if the re-expression is too &quot;strong,&quot; it can produce an RV with no kurtosis at all.  There's nothing special about this idea and it amounts almost to a trick: if you don't care about preserving the variance, you can achieve any kurtosis you want by means of a scale change, so you can exploit all the familiar ways of transforming variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A little more explicitly: let $f$ be a real-valued monotonic function of the real line and let $X$ be a random variable with finite nonzero kurtosis $\kappa$.  That implies $X$ has finite first through third moments, too, whence it has a finite fourth moment $\mu_4$ and finite first moment $\mu_1$.  Define $Y = f \left( (X - \mu_1) / \mu_4^{1/4} \right)$.  If $Y$ has finite fourth moment, then among the family $Z_\lambda = \left( Y - \nu_1 \right) \lambda + \nu_1$ there will be a unique member $Z_\lambda$ with $\lambda \gt 0$ and having kurtosis equal to $\kappa$.  Provided $f$ is nonlinear over the essential support of $X$, the skewness of $Y$ will differ from that of $X$.  Members of the Box-Cox family (apart from the trivial case of the identity function) will work, for instance.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-09-06T17:03:56.570" CommentCount="4" />
  <row Id="2415" PostTypeId="2" ParentId="2379" CreationDate="2010-09-06T17:27:01.890" Score="13" ViewCount="" Body="&lt;p&gt;A &lt;em&gt;big&lt;/em&gt; question should involve key issues of statistical methodology or, because statistics is entirely about applications, it should concern how statistics is used with problems important to society.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This characterization suggests the following should be included in any consideration of big problems:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;How best to conduct drug trials&lt;/strong&gt;.  Currently, classical hypothesis testing requires many formal phases of study.  In later (confirmatory) phases, the economic and ethical issues loom large.  Can we do better?  Do we have to put hundreds or thousands of sick people into control groups and keep them there until the end of a study, for example, or can we find better ways to identify treatments that really work and deliver them to members of the trial (and others) sooner?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Coping with scientific publication bias&lt;/strong&gt;.  Negative results are published much less simply because they just don't attain a magic p-value.  All branches of science need to find better ways to bring scientifically &lt;em&gt;important,&lt;/em&gt; not just statistically &lt;em&gt;significant,&lt;/em&gt; results to light.  (The multiple comparisons problem and coping with high-dimensional data are subcategories of this problem.)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Probing the limits of statistical methods&lt;/strong&gt; and their &lt;strong&gt;interfaces with machine learning and machine cognition&lt;/strong&gt;.  Inevitable advances in computing technology will make true AI accessible in our lifetimes.  How are we going to program artificial brains?  What role might statistical thinking and statistical learning have in creating these advances?  How can statisticians help in thinking about artificial cognition, artificial learning, in exploring their limitations, and making advances?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Developing better ways to analyze geospatial data&lt;/strong&gt;.  It is often claimed that the majority, or vast majority, of databases contain locational references.  Soon many people and devices will be located in real time with GPS and cell phone technologies.  Statistical methods to analyze and exploit spatial data are really just in their infancy (and seem to be relegated to GIS and spatial software which is typically used by non-statisticians).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-09-06T17:27:01.890" CommunityOwnedDate="2010-09-06T17:27:01.890" CommentCount="3" />
  <row Id="2416" PostTypeId="5" CreationDate="2010-09-06T18:24:38.507" Score="0" ViewCount="0" Body="&lt;p&gt;&quot;Mixed models&quot; refers to a class of models that are variously known as: mixed-effects models, multilevel models, hierarchical linear models,... This class of models was developed to account for correlation that may occur within nested data. A classic example is the estimation of test scores of students: if test scores are correlated within classes, schools, districts, etc., mixed models allow the modeler to simultaneously estimate the differences between individual students and between the groups to which they belong (with the possibility of including covariates at all levels).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;StatsExchangers often recommend the following resources for learning more about mixed models: Venables and Ripley (2002), Laird and Ware (1982), Baayen (2008), Raudenbush and Bryk (2001), Gelman and Hill (2006), and Singer and Willett (2003).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Mixed models are available in the following statistical packages: R (lme4 and nlme), SAS (PROC MIXED), MLwiN, Stata (xtreg and xtmixed), and HLM.&lt;/p&gt;&#xA;" OwnerUserId="-1" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-24T14:13:42.103" LastActivityDate="2010-09-24T14:13:42.103" />
  <row Id="2417" PostTypeId="4" CreationDate="2010-09-06T18:24:38.507" Score="0" ViewCount="0" Body="&quot;Mixed models&quot; refers to a class of models developed to account for correlation that may occur within nested data. A classic example is the estimation of test scores of students: if test scores are correlated within classes, schools, districts, etc., mixed models allow the modeler to simultaneously estimate the differences between individual students and between the groups to which they belong." OwnerUserId="-1" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-24T14:13:42.197" LastActivityDate="2010-09-24T14:13:42.197" />
  <row Id="2419" PostTypeId="1" AcceptedAnswerId="2421" CreationDate="2010-09-06T19:00:03.070" Score="3" ViewCount="667" Body="&lt;p&gt;Is there a a good python library for training boosted decision trees ?&lt;/p&gt;&#xA;" OwnerUserId="961" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-10T07:41:15.820" LastActivityDate="2010-12-23T05:18:33.843" Title="Boosted Decision Trees in python?" Tags="&lt;python&gt;&lt;decision-trees&gt;&lt;boosting&gt;" AnswerCount="6" CommentCount="6" FavoriteCount="4" />
  <row Id="2420" PostTypeId="1" CreationDate="2010-09-06T19:00:42.580" Score="1" ViewCount="179" Body="&lt;p&gt;I have some data values of type date/time (the last date that a resource was accessed) and I wish to chart this data on the y-axis against  the different categories of resource on the x-axis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What would be a sensible type of chart for displaying this sort of data ?  For example is a histogram / bar chart satisfactory or is there a different graph type more applicable ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: Are bar charts for only displaying quantities - can dates be considered quantities ?&lt;/p&gt;&#xA;" OwnerUserId="414" LastEditorUserId="414" LastEditorDisplayName="" LastEditDate="2010-09-07T07:49:42.283" LastActivityDate="2010-09-07T08:21:20.630" Title="Date/Time data on the y-axis" Tags="&lt;data-visualization&gt;" AnswerCount="2" CommentCount="6" />
  <row Id="2421" PostTypeId="2" ParentId="2419" CreationDate="2010-09-06T19:28:19.213" Score="4" ViewCount="" Body="&lt;p&gt;My first look would be at &lt;a href=&quot;http://www.ailab.si/orange/&quot; rel=&quot;nofollow&quot;&gt;Orange&lt;/a&gt;, which is a fully-featured app for ML, with a backend in Python. See e.g. &lt;a href=&quot;http://www.ailab.si/orange/doc/modules/orngEnsemble.htm&quot; rel=&quot;nofollow&quot;&gt;orngEnsemble&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other promising projects are &lt;a href=&quot;https://mlpy.fbk.eu/&quot; rel=&quot;nofollow&quot;&gt;mlpy&lt;/a&gt; and the &lt;a href=&quot;http://scikit-learn.sourceforge.net/&quot; rel=&quot;nofollow&quot;&gt;scikit.learn&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know that &lt;a href=&quot;http://pycv.sharkdolphin.com/&quot; rel=&quot;nofollow&quot;&gt;PyCV&lt;/a&gt; include several boosting procedures, but apparently not for CART. &#xA;Take also a look at &lt;a href=&quot;http://sourceforge.net/projects/mlboost/&quot; rel=&quot;nofollow&quot;&gt;MLboost&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-06T19:35:13.690" LastActivityDate="2010-09-06T19:35:13.690" CommentCount="3" />
  <row Id="2422" PostTypeId="2" ParentId="2420" CreationDate="2010-09-06T20:25:58.380" Score="6" ViewCount="" Body="&lt;p&gt;I presume that you have only a few resources that you are interested in? If so, then histograms are fine, or you could also try box-plots:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#Some R code&#xA;#Create random dates for resource A &amp;amp; resource B&#xA;dates.a = as.Date(rnorm(100, 200, 100), origin=&quot;2008-01-01&quot;)&#xA;dates.b = as.Date(rnorm(100, 300, 50), origin=&quot;2008-01-01&quot;)&#xA;&#xA;df = data.frame(dates = c(dates.a, dates.b),&#xA;  type=rep(c(&quot;A&quot;, &quot;B&quot;), each=100))&#xA;&#xA;#Plot the two resource together using boxplots&#xA;boxplot(df$dates~df$type)&#xA;#See ?hist for histograms.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This gives:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/jKlde.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="8" LastActivityDate="2010-09-06T20:25:58.380" />
  <row Id="2423" PostTypeId="1" AcceptedAnswerId="2426" CreationDate="2010-09-06T20:28:53.090" Score="5" ViewCount="293" Body="&lt;p&gt;Can anyone recommend some books that are considered to be standard references for classical (frequentist) statistics? IE, fairly comprehensive, and also, been around for a while so that typos and mistakes in formulas had a chance to be checked and corrected&lt;/p&gt;&#xA;" OwnerUserId="511" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-09-08T06:40:30.013" LastActivityDate="2010-09-08T06:40:30.013" CommunityOwnedDate="2010-09-06T23:56:59.877" Title="Standard reference for classical mathematical statistics?" Tags="&lt;books&gt;&lt;mathematical-statistics&gt;" AnswerCount="5" CommentCount="5" />
  <row Id="2424" PostTypeId="2" ParentId="2391" CreationDate="2010-09-06T20:39:55.953" Score="3" ViewCount="" Body="&lt;p&gt;Very short answer: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The chi-Square test (&lt;code&gt;chisq.test()&lt;/code&gt; in R) compares the observed frequencies in each category of a contingency table with the expected frequencies (computed as the product of the marginal frequencies). It is used to determine whether the deviations between the observed and the expected counts are too large to be attributed to chance. Departure from independence is easily checked by inspecting residuals (try &lt;code&gt;?mosaicplot&lt;/code&gt; or &lt;code&gt;?assocplot&lt;/code&gt;, but also look at the &lt;code&gt;vcd&lt;/code&gt; package). Use &lt;code&gt;fisher.test()&lt;/code&gt; for an exact test (relying on the hypergeometric distribution).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;code&gt;prop.test()&lt;/code&gt; function in R allows to test whether proportions are comparable between groups or does not differ from theoretical probabilities. It is refered to as a $z$-test because the statistics looks like&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;z=(f_1-f_2)/\sqrt{\hat p\big(\frac{1}{n_1}+\frac{1}{n_2}\big)}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\hat p=(p_1+p_2)/(n_1+n_2)$, and indices $(1,2)$ refers to the first and second line of your table.&#xA;In a two-way contingency table where $H_0:\; p_1=p_2$, this should yield comparable results to the ordinary $\chi^2$ test:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; tab &amp;lt;- matrix(c(100, 80, 20, 10), ncol = 2)&#xA;&amp;gt; chisq.test(tab)&#xA;&#xA;    Pearson's Chi-squared test with Yates' continuity correction&#xA;&#xA;data:  tab &#xA;X-squared = 0.8823, df = 1, p-value = 0.3476&#xA;&#xA;&amp;gt; prop.test(tab)&#xA;&#xA;    2-sample test for equality of proportions with continuity correction&#xA;&#xA;data:  tab &#xA;X-squared = 0.8823, df = 1, p-value = 0.3476&#xA;alternative hypothesis: two.sided &#xA;95 percent confidence interval:&#xA; -0.15834617  0.04723506 &#xA;sample estimates:&#xA;   prop 1    prop 2 &#xA;0.8333333 0.8888889 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For analysis of discrete data with R, I highly recommend &lt;a href=&quot;https://home.comcast.net/~lthompson221/Splusdiscrete2.pdf&quot; rel=&quot;nofollow&quot;&gt;R (and S-PLUS) Manual to Accompany Agresti’s Categorical Data Analysis (2002)&lt;/a&gt;, from Laura Thompson.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-09-06T20:39:55.953" />
  <row Id="2425" PostTypeId="2" ParentId="2423" CreationDate="2010-09-06T21:47:23.083" Score="4" ViewCount="" Body="&lt;p&gt;I have found &lt;em&gt;Statistical Inference&lt;/em&gt; by Casella and Berger to be a relatively comprehensive introduction.&lt;/p&gt;&#xA;" OwnerUserId="743" LastActivityDate="2010-09-06T21:47:23.083" />
  <row Id="2426" PostTypeId="2" ParentId="2423" CreationDate="2010-09-06T22:05:28.123" Score="1" ViewCount="" Body="&lt;p&gt;E. L. Lehman, &lt;em&gt;Theory of Point Estimation,&lt;/em&gt; 1983, and its companion book, &lt;em&gt;Testing Statistical Hypotheses.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(NB: The latest edition of TPE, coauthored with George Casella, has not been getting good reviews on Amazon, but the original is still a classic.)&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-09-06T22:05:28.123" />
  <row Id="2427" PostTypeId="1" AcceptedAnswerId="2442" CreationDate="2010-09-06T22:44:39.583" Score="4" ViewCount="360" Body="&lt;p&gt;I would like to solve &lt;a href=&quot;http://projecteuler.net/index.php?section=problems&amp;amp;id=213&quot; rel=&quot;nofollow&quot;&gt;Project Euler 213&lt;/a&gt; but don't know where to start because I'm a layperson in the field of Statistics, notice that an accurate answer is required so the Monte Carlo method won't work. Could you recommend some statistics topics for me to read on? Please do not post the solution here.&lt;/p&gt;&#xA;" OwnerUserId="18" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-16T06:32:09.567" LastActivityDate="2010-09-16T06:32:09.567" Title="How should one approch Project Euler problem 213" Tags="&lt;markov-process&gt;&lt;monte-carlo&gt;" AnswerCount="6" CommentCount="4" />
  <row Id="2428" PostTypeId="2" ParentId="2427" CreationDate="2010-09-06T23:09:16.800" Score="4" ViewCount="" Body="&lt;p&gt;I suspect that some knowledge of discrete-time &lt;a href=&quot;http://en.wikipedia.org/wiki/Markov_chain&quot; rel=&quot;nofollow&quot;&gt;Markov chains&lt;/a&gt; could prove useful.&lt;/p&gt;&#xA;" OwnerUserId="495" LastActivityDate="2010-09-06T23:09:16.800" />
  <row Id="2429" PostTypeId="2" ParentId="2423" CreationDate="2010-09-07T02:04:47.937" Score="1" ViewCount="" Body="&lt;p&gt;I'd recommend &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0387945466&quot; rel=&quot;nofollow&quot;&gt;Theory of Statistics&lt;/a&gt; by Mark Schervish.&lt;/p&gt;&#xA;" OwnerUserId="881" LastActivityDate="2010-09-07T02:04:47.937" CommunityOwnedDate="2010-09-07T02:04:47.937" CommentCount="1" />
  <row Id="2430" PostTypeId="1" CreationDate="2010-09-07T03:12:09.853" Score="3" ViewCount="188" Body="&lt;p&gt;I am trying to identify approximate 3% of the population for some characteristic feature. Standard decision tree or logistic regression gives too many false positives. Is there a chance that rules based classifier can improve performance? I would like to get approx 75% of recall with 95% of precission (i.e. FalsePositives &amp;lt;= 5% of Positives) &lt;/p&gt;&#xA;" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-07T08:05:20.140" LastActivityDate="2010-09-07T22:25:29.387" Title="When does rules based classifier outperforms decision trees?" Tags="&lt;machine-learning&gt;&lt;classification&gt;" AnswerCount="3" CommentCount="2" FavoriteCount="1" />
  <row Id="2431" PostTypeId="2" ParentId="2423" CreationDate="2010-09-07T03:19:29.143" Score="2" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0387402721&quot; rel=&quot;nofollow&quot;&gt;&lt;em&gt;All of Statistics&lt;/em&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-09-07T03:19:29.143" CommunityOwnedDate="2010-09-07T03:19:29.143" CommentCount="5" />
  <row Id="2432" PostTypeId="1" AcceptedAnswerId="2445" CreationDate="2010-09-07T03:24:59.747" Score="7" ViewCount="372" Body="&lt;ul&gt;&#xA;&lt;li&gt;Is there a modelling technique like &lt;a href=&quot;http://en.wikipedia.org/wiki/Local_regression&quot; rel=&quot;nofollow&quot;&gt;LOESS&lt;/a&gt; that allows for zero, one, or more discontinuities, where the timing of the discontinuities are not known apriori?&lt;/li&gt;&#xA;&lt;li&gt;If a technique exists, is there an existing implementation in R?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastEditorUserId="223" LastEditorDisplayName="" LastEditDate="2010-12-17T07:47:31.553" LastActivityDate="2011-02-17T17:10:30.397" Title="LOESS that allows discontinuities" Tags="&lt;r&gt;&lt;regression&gt;&lt;curve-fitting&gt;&lt;discontinuity&gt;&lt;loess&gt;" AnswerCount="4" CommentCount="3" FavoriteCount="2" />
  <row Id="2433" PostTypeId="2" ParentId="2432" CreationDate="2010-09-07T05:12:15.380" Score="5" ViewCount="" Body="&lt;p&gt;Here are some methods and associated R packages to solve this problem&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Wavelet &lt;strong&gt;thresolding estimation&lt;/strong&gt; in regression allows for discontonuities. You may use the package wavethresh in R.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A lot of &lt;strong&gt;tree based methods&lt;/strong&gt; (not far from the idea of wavelet) are usefull when you have disconitnuities. Hence package treethresh, package tree !&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the familly of &quot;&lt;strong&gt;local maximum likelihood&lt;/strong&gt;&quot; methods... among others: &#xA;Work of Pozhel and Spokoiny: Adaptive weights Smoothing (package aws) &#xA;Work by Catherine Loader:  package locfit&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I guess any &lt;strong&gt;kernel smoother&lt;/strong&gt; with &lt;strong&gt;locally varying bandwidth&lt;/strong&gt; makes the point but I don't know R package for that. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;note: I don't really get what is the difference between LOESS and regression... is it the idea that in LOESS alrgorithms should be &quot;on line&quot;  ? &lt;/p&gt;&#xA;" OwnerUserId="223" LastEditorUserId="223" LastEditorDisplayName="" LastEditDate="2011-02-17T17:10:30.397" LastActivityDate="2011-02-17T17:10:30.397" CommentCount="1" />
  <row Id="2434" PostTypeId="2" ParentId="2427" CreationDate="2010-09-07T07:18:25.347" Score="2" ViewCount="" Body="&lt;p&gt;Could you not iterate through the probabilities of occupation of the cells for each flea. That is, flea k is initially in cell (i(k),j(k)) with probability 1. After 1 iteration, he has probability 1/4 in each of the 4 adjacent cells (assuming he's not on an edge or in a corner). Then the next iteration, each of those quarters gets &quot;smeared&quot; in turn. After 50 iterations you have a matrix of occupation probabilities for flea k. Repeat over all 900 fleas (if you take advantage of symmetries this reduces by nearly a factor of 8) and add the probabilities (you don't need to store all of them at once, just the current flea's matrix (hmm, unless you are very clever, you may want an additional working matrix) and the current sum of matrices). It looks to me like there are lots of ways to speed this up here and there.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This involves no simulation at all. However, it does involve quite a lot of computation; it should not be very hard to work out the simulation size required to give the answers to somewhat better than 6 dp accuracy with high probability and figure out which approach will be faster. I expect this approach would beat simulation by some margin.&lt;/p&gt;&#xA;" OwnerUserId="805" LastEditorUserId="805" LastEditorDisplayName="" LastEditDate="2010-09-07T07:23:41.247" LastActivityDate="2010-09-07T07:23:41.247" CommentCount="4" />
  <row Id="2435" PostTypeId="2" ParentId="2430" CreationDate="2010-09-07T08:04:24.807" Score="3" ViewCount="" Body="&lt;p&gt;Probably you just have unbalanced classes (3% to 97%, if I understood well) -- try balancing them (get this 3% of true ones and about equal number of false ones) and check the classifier build on this case. If you are worried that you have thrown out most of your data, iterate it few times and connect them with some simple blender, like voting. (More complex blenders will also suffer from unbalanced classes). You should also check some better classifiers than a single tree or logistic regression -- like SVM or Random Forest.&lt;br&gt;&#xA;Of course you can also use some classifier immune to imbalance problem, like kNN or as you say some rule-based approach.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-09-07T08:04:24.807" />
  <row Id="2436" PostTypeId="2" ParentId="2427" CreationDate="2010-09-07T08:20:14.377" Score="1" ViewCount="" Body="&lt;p&gt;If you really want to do this by Monte Carlo, just write a program that simulates the process and run it huge number of times counting the number of empty cells -- nothing more complicated.&lt;br&gt;&#xA;But I agree with Geln_b that this might be simple enough to be done analytically -- this 6 digits requirement suggests that.&lt;/p&gt;&#xA;" OwnerUserId="88" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-07T14:22:30.850" LastActivityDate="2010-09-07T14:22:30.850" />
  <row Id="2437" PostTypeId="2" ParentId="2420" CreationDate="2010-09-07T08:21:20.630" Score="0" ViewCount="" Body="&lt;p&gt;My personal preference would be for box plots.  If the distributions of date/time in one or more categories are skewed, then box plots would definitely be more informative than bars.&lt;/p&gt;&#xA;" OwnerUserId="266" LastActivityDate="2010-09-07T08:21:20.630" />
  <row Id="2438" PostTypeId="2" ParentId="7" CreationDate="2010-09-07T08:58:26.777" Score="5" ViewCount="" Body="&lt;p&gt;NIST provides a &lt;a href=&quot;http://www.itl.nist.gov/div898/strd/general/dataarchive.html&quot; rel=&quot;nofollow&quot;&gt;Reference Dataset archive&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="830" LastActivityDate="2010-09-07T08:58:26.777" CommunityOwnedDate="2010-09-07T08:58:26.777" />
  <row Id="2439" PostTypeId="1" AcceptedAnswerId="2440" CreationDate="2010-09-07T11:48:49.003" Score="7" ViewCount="206" Body="&lt;p&gt;A cursory search reveals that &lt;a href=&quot;http://en.wikipedia.org/wiki/Latin_square&quot; rel=&quot;nofollow&quot;&gt;Latin squares&lt;/a&gt; are fairly extensively used in the design of experiments.  During my PhD, I have studied various theoretical properties of Latin squares (from a combinatorics point-of-view), but do not have a deep understanding of what is it about Latin squares that make them particularly well-suited to experimental design.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand that Latin squares are good at allowing statisticians to efficiently study situations where there are two factors which vary in different &quot;directions&quot;.  But, I'm also fairly confident there would be many other techniques that could be used.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;What is it, in particular, about Latin squares that make them so well suited for the design of experiments, that other designs do not have?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Moreover, there are zillions of Latin squares to choose from, so which Latin square do you choose?  I understand that choosing one at random is important, but there would still be some Latin squares that would be less suited to running experiments than others (e.g. the Cayley table of a cyclic group).  This raises the following question.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Which properties of Latin squares are desirable and which properties of Latin squares are undesirable for experimental design?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="386" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-09-07T17:32:38.397" LastActivityDate="2010-09-07T17:32:38.397" Title="Desirable and undesirable properties of Latin squares in experiments?" Tags="&lt;experiment-design&gt;" AnswerCount="1" CommentCount="0" FavoriteCount="1" />
  <row Id="2440" PostTypeId="2" ParentId="2439" CreationDate="2010-09-07T13:03:36.843" Score="7" ViewCount="" Body="&lt;p&gt;Imagine:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;you were interested in the effect of &lt;strong&gt;word type&lt;/strong&gt; (nouns, adjectives, adverbs, and verbs) on &lt;strong&gt;recall&lt;/strong&gt;. &lt;/li&gt;&#xA;&lt;li&gt;you wanted to include word type as a within-subjects factor (i.e., all participants were exposed to all conditions)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Such a design would raise the issue of carry over effects. I.e., the order of the conditions may effect the dependent variable recall. For example, participants might get better at recalling words with practice. Thus, if the conditions were always presented in the same order, then the effect or order would be confounded with the effect of condition (i.e., word type).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Latin Squares is one of several strategies for dealing with order effects.&#xA;A Latin Squares design could involve assigning participants to one of four separate orderings (i.e., a between subjects condition called order):&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;nouns adjectives adverbs verbs&lt;/li&gt;&#xA;&lt;li&gt;adjectives adverbs verbs nouns &lt;/li&gt;&#xA;&lt;li&gt;adverbs verbs nouns adjectives &lt;/li&gt;&#xA;&lt;li&gt;verbs nouns adjectives adverbs &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Thus, the Latin Squares design only entails a subset of possible orderings, and to some extent the effect of order can be estimated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a &lt;a href=&quot;http://jeromyanglim.blogspot.com/2008/11/carryover-effects-in-repeated-measures.html&quot; rel=&quot;nofollow&quot;&gt;blog post&lt;/a&gt; I suggest the following simple rules of thumb:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&quot;If order is the focus of the analysis (e.g., skill acquisition looking at effects of practice), then don't worry about order effects&lt;/li&gt;&#xA;&lt;li&gt;If order effects are very strong, it may be better to stick to between subjects designs&lt;/li&gt;&#xA;&lt;li&gt;if order effects are small or moderate or unknown, typical design strategy depends on the number of levels of the within-subjects factor of interest.&#xA;&lt;ul&gt;&#xA;&lt;li&gt;If there are few levels (e.g., 2,3,4 perhaps), present all orders (counterbalance)&lt;/li&gt;&#xA;&lt;li&gt;If there are more levels (e.g., 4+ perhaps), adopt a latin squares approach or randomise ordering&quot;&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;To specifically answer your question, Latin Squares designs allow you to get the statistical power benefits of a within-subjects design while, potentially at least, minimising the main problem of within subjects designs: i.e., order effects.&lt;/p&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-09-07T13:47:53.373" LastActivityDate="2010-09-07T13:47:53.373" />
  <row Id="2441" PostTypeId="2" ParentId="2427" CreationDate="2010-09-07T14:37:45.337" Score="4" ViewCount="" Body="&lt;p&gt;An analytical approach may be tedious and I have not thought through the intricacies but here is an approach that you may want to consider. Since you are interested in the expected number of cells that are empty after 50 rings you need to define a markov chain over the &quot;No of the fleas in a cell&quot; rather than the position of a flea (See Glen_b's &lt;a href=&quot;http://stats.stackexchange.com/questions/2427/how-should-one-approch-project-euler-problem-213/2434#2434&quot;&gt;answer&lt;/a&gt; which models the position of a flea as a markov chain. As pointed out by Andy in the comments to that answer that approach may not get what you want.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Specifically, let:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$n_{ij}(t)$ be the number of fleas in a cell in row $i$ and column $j$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then the markov chain starts with the following state:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$n_{ij}(0) =1$ for all $i$ and $j$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since, fleas move to one of four adjacent cells, the state of a cell changes depending on how many fleas are in the target cell and how many fleas are there in the four adjacent cells and the probability that they will move to that cell. Using this observation, you can write the state transition probabilities for each cell as a function of the state of that cell and the state of the adjacent cells. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you wish I can expand the answer further but this along with a basic introduction to markov chains should get you started.&lt;/p&gt;&#xA;" LastActivityDate="2010-09-07T14:37:45.337" CommentCount="4" />
  <row Id="2442" PostTypeId="2" ParentId="2427" CreationDate="2010-09-07T14:51:35.883" Score="6" ViewCount="" Body="&lt;p&gt;You're right; Monte Carlo is impracticable.  (In a naive simulation--that is, one that exactly reproduces the problem situation without any simplifications--each iteration would involve 900 flea moves.  A crude estimate of the proportion of empty cells is $1/e$, implying the variance of the Monte-Carlo estimate after $N$ such iterations is approximately $1/N 1/e (1 - 1/e) = 0.2325\ldots /N$.  To pin down the answer to six decimal places, you would need to estimate it to within 5.E-7 and, to achieve a confidence of 95+% (say), you would have to approximately halve that precision to 2.5E-7.  Solving $\sqrt(0.2325/N) \lt 2.5E-7$ gives $N &gt; 4E12$, approximately.  That would be around 3.6E15 flea moves, each taking several ticks of a CPU.  With one modern CPU available you will need a full year of (highly efficient) computing.  And I have somewhat incorrectly and overoptimistically assumed the answer is given as a proportion instead of a count: as a count, it will need three more significant figures, entailing a million fold increase in computation...  Can you wait a long time?)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as an analytical solution goes, some simplifications are available.  (These can be used to shorten a Monte Carlo computation, too.)  The expected number of empty cells is the sum of the probabilities of emptiness over all the cells.  To find this, you could compute the probability distribution of occupancy numbers of each cell.  Those distributions are obtained by summing over the (independent!) contributions from each flea.  This reduces your problem to finding the number of paths of length 50 along a 30 by 30 grid between any given pair of cells on that grid (one is the flea's origin and the other is a cell for which you want to calculate the probability of the flea's occupancy).&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-09-07T14:51:35.883" CommentCount="1" />
  <row Id="2443" PostTypeId="2" ParentId="2391" CreationDate="2010-09-07T15:12:36.180" Score="4" ViewCount="" Body="&lt;p&gt;Hi hgcrpd, a chi-square test for equality of two proportions is exactly the same thing as a $z$-test. The chi-square distribution with one degree of freedom is just that of a normal deviate, squared. You're basically just repeating the chi-square test on a subset of the contingency table. (This is why chl gets the exact same $p$-value with both tests.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem of doing the chi-square test globally first and then diving down to do more tests on subsets is you won't necessarily preserve your alpha -- that is, you won't control false positives to be less than 5% (or whatever $\alpha$) across the whole experiment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think if you want to do this properly in the classical paradigm, you need to identify your hypotheses at the outset (which proportions to compare), collect the data, and then test the hypotheses such that the total threshold for significance of each test sums to $\alpha$. Unless you can prove &lt;em&gt;a priori&lt;/em&gt; that there's some correlation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The most powerful test for equality of proportions is called Barnard's test for superiority.&lt;/p&gt;&#xA;" OwnerUserId="1122" LastActivityDate="2010-09-07T15:12:36.180" CommentCount="1" />
  <row Id="2445" PostTypeId="2" ParentId="2432" CreationDate="2010-09-07T15:45:24.470" Score="10" ViewCount="" Body="&lt;p&gt;It sounds like you want to perform multiple changepoint detection followed by independent smoothing within each segment.  (Detection can be online or not, but your application is not likely to be online.)  There's a lot of literature on this; Internet searches are fruitful.  &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;DA Stephens wrote a useful introduction to Bayesian changepoint detection in 1994 (App. Stat. 43 #1 pp 159-178: &lt;a href=&quot;http://www.jstor.org/pss/2986119&quot; rel=&quot;nofollow&quot;&gt;JSTOR&lt;/a&gt;).  &lt;/li&gt;&#xA;&lt;li&gt;More recently Paul Fearnhead has been doing nice work (e.g., &lt;em&gt;Exact and efficient Bayesian inference for multiple changepoint problems&lt;/em&gt;, Stat Comput (2006) 16: 203-213: &lt;a href=&quot;http://eprints.lancs.ac.uk/8189/1/PScpt2.pdf&quot; rel=&quot;nofollow&quot;&gt;Free PDF&lt;/a&gt;).  &lt;/li&gt;&#xA;&lt;li&gt;A recursive algorithm exists, based on a beautiful analysis by D Barry &amp;amp; JA Hartigan&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;em&gt;Product Partition Models for Change Point Models,&lt;/em&gt; Ann. Stat. 20:260-279: &lt;a href=&quot;http://www.jstor.org/pss/2242159&quot; rel=&quot;nofollow&quot;&gt;JSTOR&lt;/a&gt;; &lt;/li&gt;&#xA;&lt;li&gt;&lt;em&gt;A Bayesian Analysis for Change Point Problems,&lt;/em&gt; JASA 88:309-319: &lt;a href=&quot;http://www.jstor.org/pss/2290726&quot; rel=&quot;nofollow&quot;&gt;JSTOR&lt;/a&gt;.  &lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;One implementation of the Barry &amp;amp; Hartigan algorithm is documented in O. Seidou &amp;amp; TBMJ Ourda, &lt;em&gt;Recursion-based Multiple Changepoint Detection in Multivariate Linear Regression and Application to River Streamflows,&lt;/em&gt; Water Res. Res., 2006: &lt;a href=&quot;http://www.oifii.org/do-org/dod/inrs-ete_ouarda-taha_2006WR005021.pdf&quot; rel=&quot;nofollow&quot;&gt;Free PDF&lt;/a&gt;.  &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I haven't looked hard for any R implementations (I had coded one in Mathematica a while ago) but would appreciate a reference if you do find one.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-09-08T22:31:40.630" LastActivityDate="2010-09-08T22:31:40.630" CommentCount="3" />
  <row Id="2446" PostTypeId="1" CreationDate="2010-09-07T16:01:51.877" Score="7" ViewCount="296" Body="&lt;p&gt;In order to correlate or compare means of two dependent variables. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my case, I need to correlate individual (e.g. subjects=30) slope values from different conditions (e.g. conditions=4), and each slope value summarizes the relation between the dependent variable (e.g. measured 4 times in each level of the independent variable) and the independent variable (e.g. 5 levels).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How to correct the df of the comparison to reflect the fact that each data point (slope value) summarizes many measurements?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: I am not asking how to do a regression between slope values. I already did regression in a within subject design, minimized euclidean distance regression etc.&lt;/p&gt;&#xA;" OwnerUserId="1084" LastEditorUserId="1084" LastEditorDisplayName="" LastEditDate="2010-09-15T17:04:17.370" LastActivityDate="2010-09-21T20:33:56.847" Title="How to choose df for comparisons between summary statistics (e.g. slope values) ?" Tags="&lt;correlation&gt;&lt;regression&gt;&lt;statistical-significance&gt;&lt;degrees-of-freedom&gt;" AnswerCount="3" CommentCount="5" FavoriteCount="2" />
  <row Id="2447" PostTypeId="2" ParentId="2432" CreationDate="2010-09-07T16:03:44.463" Score="5" ViewCount="" Body="&lt;p&gt;do it with koencker's broken line regression, see page 18 of this vignette&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://cran.r-project.org/web/packages/quantreg/vignettes/rq.pdf&quot; rel=&quot;nofollow&quot;&gt;http://cran.r-project.org/web/packages/quantreg/vignettes/rq.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In response to Whuber last comment:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This estimator is defined like this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$x\in\mathbb{R}$, $x_{(i)}\geq x_{(i-1)}\;\forall i$,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$e_i:=y_{i}-\beta_{i}x_{(i)}-\beta_0$,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$z^+=\max(z,0)$, $z^-=\max(-z,0)$, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\tau \in (0,1)$, $\lambda\geq 0$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\underset{\beta\in\mathbb{R}^n|\tau, \lambda}{\min.} \sum_{i=1}^{n} \tau e_i^++\sum_{i=1}^{n}(1-\tau)e_i^-+\lambda\sum_{i=2}^{n}|\beta_{i}-\beta_{i-1}|$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\tau$ gives the desired quantile (i.e. in the example, $\tau=0.9$). $\lambda$ directs the number of breakpoint: for $\lambda$ large this estimator shrinks to no break point (corresponding to the classicla linear quantile regression estimator).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Quantile Smoothing Splines&#xA;Roger Koenker, Pin Ng, Stephen Portnoy&#xA;Biometrika, Vol. 81, No. 4 (Dec., 1994), pp. 673-680&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: there is a open acess working paper with the same name by the same others but it's not the same thing.&lt;/p&gt;&#xA;" OwnerUserId="603" LastEditorUserId="603" LastEditorDisplayName="" LastEditDate="2010-09-10T17:36:27.873" LastActivityDate="2010-09-10T17:36:27.873" CommentCount="10" />
  <row Id="2448" PostTypeId="2" ParentId="2430" CreationDate="2010-09-07T16:09:53.610" Score="4" ViewCount="" Body="&lt;p&gt;This is only valid for the logit: you can use another link function (complementary log-log or cloglog in short). This is a variation of the classical logit function that allows for assymetry (when one tail of the link function does not go to 0 at the same speed as the other tail goes to 1). I had a very good experience fitting one of these to a database with about 1% of 'ones'. &#xA;this is a good starting point reference:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.springerlink.com/content/cfakldm2renaj34m/&quot; rel=&quot;nofollow&quot;&gt;http://www.springerlink.com/content/cfakldm2renaj34m/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These can be fitted using the zelig package in R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;edit: there is a good comparaison with a logit link here:&#xA;&lt;a href=&quot;http://rss.acs.unt.edu/Rdoc/library/VGAM/html/cloglog.html&quot; rel=&quot;nofollow&quot;&gt;http://rss.acs.unt.edu/Rdoc/library/VGAM/html/cloglog.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="603" LastEditorUserId="603" LastEditorDisplayName="" LastEditDate="2010-09-07T22:25:29.387" LastActivityDate="2010-09-07T22:25:29.387" CommentCount="1" />
  <row Id="2449" PostTypeId="2" ParentId="2430" CreationDate="2010-09-07T16:45:12.883" Score="2" ViewCount="" Body="&lt;p&gt;One strategy would be to use margin based methods with uneven margins (see &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.130.4424&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;this paper&lt;/a&gt;). Or you can use &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.155.1168&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;active learning&lt;/a&gt; to provide the learner with more balanced classes. Besides this, actually there are a number of other ways too that you can use to deal with imbalanced dataset. See &lt;a href=&quot;http://www.ece.stevens-tech.edu/~hhe/PDFfiles/ImbalancedLearning.pdf&quot; rel=&quot;nofollow&quot;&gt;this survey paper&lt;/a&gt; which discusses a number of techniques such as resampling, cost-sensitive approaches, active learning, etc (and evaluation methods).&lt;/p&gt;&#xA;" OwnerUserId="881" LastEditorUserId="881" LastEditorDisplayName="" LastEditDate="2010-09-07T17:14:37.033" LastActivityDate="2010-09-07T17:14:37.033" />
  <row Id="2450" PostTypeId="2" ParentId="2423" CreationDate="2010-09-07T17:02:12.730" Score="1" ViewCount="" Body="&lt;p&gt;A comprehensive and authoratative reference is Kendall's &lt;em&gt;Advanced Theory of Statistics&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Volume 1 &lt;em&gt;Distribution Theory&lt;/em&gt;  &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Volume 2A &lt;em&gt;Classical Inference and Linear Models&lt;/em&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;There is also a Volume 2B but it is &lt;em&gt;Bayesian Inference&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other than those, I agree the Casella and Berger is an excellent reference at the graduate level, and suggest Bain and Engelhardt's &lt;em&gt;Introduction to Probability and Mathematical Statistics&lt;/em&gt; for upper-level undergraduates.&lt;/p&gt;&#xA;" OwnerUserId="1107" LastActivityDate="2010-09-07T17:02:12.730" CommunityOwnedDate="2010-09-07T17:02:12.730" />
  <row Id="2451" PostTypeId="2" ParentId="1160" CreationDate="2010-09-07T20:59:46.453" Score="0" ViewCount="" Body="&lt;p&gt;date() (so i know how long i have been working)&lt;/p&gt;&#xA;" OwnerUserId="603" LastActivityDate="2010-09-07T20:59:46.453" CommunityOwnedDate="2010-09-07T20:59:46.453" />
  <row Id="2452" PostTypeId="2" ParentId="1964" CreationDate="2010-09-07T21:40:03.113" Score="2" ViewCount="" Body="&lt;p&gt;One place to start would be Silverman's &lt;a href=&quot;http://nedwww.ipac.caltech.edu/level5/March02/Silverman/Silver2_5.html&quot; rel=&quot;nofollow&quot;&gt;nearest-neighbor estimator&lt;/a&gt;, but to add in the weights somehow. (I am not sure exactly what your weights are for here.) The nearest neighbor method can evidently be formulated in terms of distances. I believe your first and second nearest neighbor method are versions of the nearest-neighbor method, but without a kernel function, and with a small value of $k$.&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-09-07T21:40:03.113" />
  <row Id="2453" PostTypeId="2" ParentId="4" CreationDate="2010-09-07T23:43:17.103" Score="4" ViewCount="" Body="&lt;p&gt;I'm going to ask the consultant's dumb question.  Why do you want to know if these distributions are different in a statistically significant way?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it that the data that you are using are representative samples from populations or processes, and you want to assess the evidence that those populations or processes differ?  If so, then a statistical test is right for you.  But this seems like a strange question to me.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or, are you interested in whether you really need to behave as though those populations or processes are different, regardless of the truth?  Then you will be better off determining a loss function, ideally one that returns units that are meaningful to you, and predicting the expected loss when you (a) treat the populations as different, and (b) treat them as the same.  Or you can choose some quantile of the loss distribution if you want to adopt a more or less conservative position.&lt;/p&gt;&#xA;" OwnerUserId="187" LastActivityDate="2010-09-07T23:43:17.103" />
  <row Id="2454" PostTypeId="2" ParentId="2432" CreationDate="2010-09-07T23:47:38.133" Score="2" ViewCount="" Body="&lt;p&gt;It should be possible to code a solution in R using the non-linear regression function nls, b splines (the bs function in the spline package, for example) and the ifelse function.  &lt;/p&gt;&#xA;" OwnerUserId="187" LastActivityDate="2010-09-07T23:47:38.133" CommentCount="1" />
  <row Id="2455" PostTypeId="1" AcceptedAnswerId="2456" CreationDate="2010-09-08T00:31:39.353" Score="4" ViewCount="398" Body="&lt;p&gt;Age pyramid looks like this:&#xA;&lt;img src=&quot;http://i.imgur.com/GtBSK.png&quot; alt=&quot;alt text&quot;&gt;&lt;br&gt;&#xA;I would like to make something similar, namely a 2 barplots (not histograms) with same categories, rotated vertically and extending to both sides as in pyramid.&lt;br&gt;&#xA;Is it a simple way to do this in R?&lt;br&gt;&#xA;It would be also nice to control the colour of each bar.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-09-08T00:45:25.083" Title="How to make age pyramid like plot in R?" Tags="&lt;r&gt;&lt;data-visualization&gt;" AnswerCount="1" FavoriteCount="3" />
  <row Id="2456" PostTypeId="2" ParentId="2455" CreationDate="2010-09-08T00:39:29.500" Score="9" ViewCount="" Body="&lt;p&gt;You can do this with &lt;a href=&quot;http://rss.acs.unt.edu/Rdoc/library/plotrix/html/pyramid.plot.html&quot; rel=&quot;nofollow&quot;&gt;the &lt;code&gt;pyramid.plot()&lt;/code&gt; function&lt;/a&gt; from the &lt;code&gt;plotrix&lt;/code&gt; package.  Here's an example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(plotrix)&#xA; xy.pop&amp;lt;-c(3.2,3.5,3.6,3.6,3.5,3.5,3.9,3.7,3.9,3.5,3.2,2.8,2.2,1.8,&#xA;  1.5,1.3,0.7,0.4)&#xA; xx.pop&amp;lt;-c(3.2,3.4,3.5,3.5,3.5,3.7,4,3.8,3.9,3.6,3.2,2.5,2,1.7,1.5,&#xA;  1.3,1,0.8)&#xA; agelabels&amp;lt;-c(&quot;0-4&quot;,&quot;5-9&quot;,&quot;10-14&quot;,&quot;15-19&quot;,&quot;20-24&quot;,&quot;25-29&quot;,&quot;30-34&quot;,&#xA;  &quot;35-39&quot;,&quot;40-44&quot;,&quot;45-49&quot;,&quot;50-54&quot;,&quot;55-59&quot;,&quot;60-64&quot;,&quot;65-69&quot;,&quot;70-74&quot;,&#xA;  &quot;75-79&quot;,&quot;80-44&quot;,&quot;85+&quot;)&#xA; mcol&amp;lt;-color.gradient(c(0,0,0.5,1),c(0,0,0.5,1),c(1,1,0.5,1),18)&#xA; fcol&amp;lt;-color.gradient(c(1,1,0.5,1),c(0.5,0.5,0.5,1),c(0.5,0.5,0.5,1),18)&#xA; par(mar=pyramid.plot(xy.pop,xx.pop,labels=agelabels,&#xA;  main=&quot;Australian population pyramid 2002&quot;,lxcol=mcol,rxcol=fcol,&#xA;  gap=0.5,show.values=TRUE))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Which ends up looking like this:&#xA;&lt;img src=&quot;http://i.imgur.com/EKGpi.jpg&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="5" LastEditorUserId="5" LastEditorDisplayName="" LastEditDate="2010-09-08T00:45:25.083" LastActivityDate="2010-09-08T00:45:25.083" CommentCount="3" />
  <row Id="2457" PostTypeId="1" AcceptedAnswerId="2463" CreationDate="2010-09-08T01:57:18.193" Score="5" ViewCount="355" Body="&lt;p&gt;I would just like someone to confirm my understanding or if I'm missing something.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The definition of a markov process says the next step depends on the current state only and no past states. So, let's say we had a state space of a,b,c,d and we go from a-&gt;b-&gt;c-&gt;d. That means that the transition to d could only depend on the fact that we were in c.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, is it true that you could just make the model more complex and kind of &quot;get around&quot; this limitation? In other words, if your state space were now aa, ab, ac, ad, ba, bb, bc, bd, ca, cb, cc, cd, da, db, dc, dd, meaning that your new state space becomes the previous state combined with the current state, then the above transition would be *a-&gt;ab-&gt;bc-&gt;cd and so the transition to cd (equivalent in the previous model to d) is now &quot;dependent&quot; on a state which, if modeled differently, is a previous state (I refer to it as a sub-state below).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Am I correct in that one can make it &quot;depend on previous states (sub-state)&quot; (I know technically it doesn't in the new model since the sub-state is no longer a real state) maintain the markov property by expanding the state space as I did? So, one could in effect create a markov process that could depend on any number of previous sub-states.&lt;/p&gt;&#xA;" OwnerUserId="1208" LastActivityDate="2010-09-08T02:54:16.440" Title="Markov Process about only depending on previous state" Tags="&lt;markov-process&gt;" AnswerCount="2" FavoriteCount="2" />
  <row Id="2458" PostTypeId="1" CreationDate="2010-09-08T02:08:55.547" Score="1" ViewCount="730" Body="&lt;p&gt;I'm a new user to R, sorry if my question is too basic.&#xA;I've installed  the newest version of R on windows 7 and as a practice, i tried to open a .txt file using &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;students&amp;lt;-read.table(&quot;students.txt&quot;,header=T,sep=&quot;\t&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;but there's always an error message as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Error in file(file, &quot;rt&quot;) : cannot open the connection&#xA;In addition: Warning message:&#xA;In file(file, &quot;rt&quot;) :&#xA;  cannot open file 'students.txt': No such file or directory&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I've already changed the directory to where the students.txt resides; but it just doesn't work. Please help!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you very much in advance if anyone can help me solve the problem.&lt;/p&gt;&#xA;" OwnerUserId="1209" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-08T09:02:02.780" LastActivityDate="2010-09-08T12:21:02.107" Title="How to import files into R?" Tags="&lt;r&gt;&lt;error&gt;" AnswerCount="3" CommentCount="1" FavoriteCount="1" />
  <row Id="2459" PostTypeId="2" ParentId="2427" CreationDate="2010-09-08T02:11:18.783" Score="1" ViewCount="" Body="&lt;p&gt;if you are going to go the numerical route, a simple observation: the problem appears to be subject to red-black parity (a flea on a red square always moves to a black square, and vice-versa). This can help reduce your problem size by a half (just consider two moves at a time, and only look at fleas on the red squares, say.) &lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-09-08T02:11:18.783" CommentCount="3" />
  <row Id="2460" PostTypeId="2" ParentId="2458" CreationDate="2010-09-08T02:12:15.320" Score="5" ViewCount="" Body="&lt;p&gt;The error message is saying that it cannot find the students.txt file.  You should try two things:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Call &lt;code&gt;getwd()&lt;/code&gt; and confirm that the file is in the directory that is returned.&lt;/li&gt;&#xA;&lt;li&gt;Try specifying the full path to the file in your &lt;code&gt;read.table()&lt;/code&gt; call.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Look at &lt;code&gt;help(&quot;read.table&quot;)&lt;/code&gt; for more detail on the function.  You may, for instance, want to change the default to &lt;code&gt;stringsAsFactors=FALSE&lt;/code&gt;, depending on your application.&lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-09-08T02:12:15.320" CommentCount="2" />
  <row Id="2461" PostTypeId="2" ParentId="2458" CreationDate="2010-09-08T02:17:18.087" Score="6" ViewCount="" Body="&lt;p&gt;There is &lt;a href=&quot;http://cran.r-project.org/doc/manuals/R-data.html&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;an entire R manual devoted to data import / export questions&lt;/strong&gt;&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, if you fail to specify file name and path correctly, use the &lt;code&gt;file.choose()&lt;/code&gt; function instead which allows you to point, click and shoot:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;students&amp;lt;-read.table(file.choose(),header=T,sep=&quot;\t&quot;) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="334" LastEditorUserId="334" LastEditorDisplayName="" LastEditDate="2010-09-08T12:21:02.107" LastActivityDate="2010-09-08T12:21:02.107" CommentCount="2" />
  <row Id="2462" PostTypeId="2" ParentId="2457" CreationDate="2010-09-08T02:17:56.713" Score="4" ViewCount="" Body="&lt;p&gt;&lt;em&gt;The definition of a markov process says the next step depends on the current state only and no past states.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That is the markov property and it defines a &lt;strong&gt;first order&lt;/strong&gt; MC, which is very tractable mathematically and quiet easy to present/explain. Of course you could have &lt;strong&gt;$n^{th}$ order MC&lt;/strong&gt; (where the next state depends on the current and the past $n-1$ states) as well as variable order MC's (when the length of the memory is fixed but depends on the previous state).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$n^{th}$ order MC's retain the explicit formulation for the distribution of the stationnary state, but as you pointed out, the size of the state matrix growths with $n$ such that an unrestricted $n^{th}$ order MC with $k$ states has $O(k^{2n})$ entry in its state matrix.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may want to have a look at recent papers such as &lt;em&gt;Higher-order multivariate Markov chains and their applications&lt;/em&gt; as this field is advancing quiet fast.&lt;/p&gt;&#xA;" OwnerUserId="603" LastEditorUserId="603" LastEditorDisplayName="" LastEditDate="2010-09-08T02:54:16.440" LastActivityDate="2010-09-08T02:54:16.440" />
  <row Id="2463" PostTypeId="2" ParentId="2457" CreationDate="2010-09-08T02:20:47.227" Score="8" ViewCount="" Body="&lt;p&gt;Technically, both the processes you describe are markov chains. The difference is that the first one is a first order markov chain whereas the second one is a second order markov chain. And yes, you can transform a second order markov chain to a first order markov chain by a suitable change in state space definition. Let me explain via an example. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose that we want to model the weather as a stochastic process and suppose that on any given day the weather can be rainy, sunny or cloudy. Let $W_t$ be the weather in any particular day and let us denote the possible states by the symbols $R$ (for rainy), $S$ for (sunny) and $C$ (for cloudy).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;First Order Markov Chain&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(W_t = w | W_{t-1}, W_{t-2},W_{t-3} ..) = P(W_t = w | W_{t-1})$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Second Order Markov Chain&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(W_t = w | W_{t-1}, W_{t-2},W_{t-3} ..) = P(W_t = w | W_{t-1},W_{t-2})$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second order markov chain can be transformed into a first order markov chain be re-defining the state space as follows. Define: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Z_{t-1,t}$ as the weather on two consecutive days. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words, the state space can take one of the following values: $RR$, $RC$, $RS$, $CR$, $CC$, $CS$, $SR$, $SC$ and $SS$. With this re-defined state space we have the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(Z_{t-1,t} = z_{t-1,t} | Z_{t-2,t-1}, Z_{t-3,t-2}, ..) = P(Z_{t-1,t} =  z_{t-1,t} | Z_{t-2,t-1})$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The above is clearly a first order markov chain on the re-defined state space. The one difference from the second order markov chain is that your redefined markov chain needs to be specified with two initial starting states i.e., the chain must be started with some assumptions about the weather on day 1 and on day 2. &lt;/p&gt;&#xA;" LastActivityDate="2010-09-08T02:28:25.460" CommentCount="1" />
  <row Id="2464" PostTypeId="2" ParentId="4" CreationDate="2010-09-08T03:00:19.690" Score="0" ViewCount="" Body="&lt;p&gt;One measure of the difference between two distribution is the &quot;maximum mean discrepancy&quot; criteria, which basically measures the difference between the empirical means of the samples from the two distributions in a Reproducing Kernel Hilbert Space (RKHS). See this paper &lt;a href=&quot;http://arxiv.org/PS_cache/arxiv/pdf/0805/0805.2368v1.pdf&quot; rel=&quot;nofollow&quot;&gt;&quot;A kernel method for the two sample problem&quot;&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="881" LastActivityDate="2010-09-08T03:00:19.690" />
  <row Id="2465" PostTypeId="2" ParentId="2446" CreationDate="2010-09-08T03:34:47.097" Score="2" ViewCount="" Body="&lt;p&gt;Here's how I have understood your question:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;you have two groups of participants&lt;/li&gt;&#xA;&lt;li&gt;Five observations per participant&lt;/li&gt;&#xA;&lt;li&gt;Based on the five observations, you can extract a single summary statistic (e.g., if the five observations were performance over five time points, the summary statistic might be the slope of the regression line predicting performance from time)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;General points&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;If you want to test whether there are differences between groups on the summary statistic, you can do a standard t-test with standard degrees of freedom.&lt;/li&gt;&#xA;&lt;li&gt;Having ore observations per individual will increase the reliability with which you measure the summary statistic.&lt;/li&gt;&#xA;&lt;li&gt;Greater reliability of measurement means larger expected group differences and thus greater statistical power (see &lt;a href=&quot;http://en.wikipedia.org/wiki/Correction_for_attenuation&quot; rel=&quot;nofollow&quot;&gt;reliability attenuation&lt;/a&gt;). &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Very similar points could be made if instead of having two groups you had a numeric variable measured once on each participant, such as age, and you wanted to correlate this with your summary statistic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are many ways to measure something on a set of participants. You just happened to have applied an algorithm (e.g., a linear regression leading to a slope) to a set of observations to derive your measure.&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-09-08T03:34:47.097" CommentCount="1" />
  <row Id="2466" PostTypeId="1" AcceptedAnswerId="2482" CreationDate="2010-09-08T04:44:53.493" Score="9" ViewCount="328" Body="&lt;p&gt;Say I have a population of 50 million unique things, and I take 10 million samples (with replacement)... The first graph is I've attached shows how many times I sample the same &quot;thing&quot;, which is relatively rare as the population is larger than my sample.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However if my population is only 10 million things, and I take 10 million samples, as the second graph shows I will more often sample the same thing repeated times.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is - from my frequency table of observations (the data in the bar charts) is it possible to get an estimate of the original population size when it is an unknown? And it would be great if you could provide a pointer to how to go about this in R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/XOEyE.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="1210" LastEditorUserId="521" LastEditorDisplayName="" LastEditDate="2010-09-15T04:09:47.407" LastActivityDate="2010-09-15T04:09:47.407" Title="Estimate the size of a population being sampled by the number of repeat observations" Tags="&lt;r&gt;&lt;sampling&gt;&lt;e-m&gt;" AnswerCount="3" FavoriteCount="2" />
  <row Id="2467" PostTypeId="1" AcceptedAnswerId="2473" CreationDate="2010-09-08T04:49:55.197" Score="3" ViewCount="331" Body="&lt;p&gt;I have analysed several dimensions in a survey. Each part of the survey represents a theoretical dimension and is analysed with factorial analysis. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to use scores from factor analysis to do a classification. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;The first factors represents a  large part of the variance. Can I keep only first factor or do I need to retain all factors?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;After factor analysis i did a PROMAX rotation which is an oblique rotation. How should i use the output from the PROMAX rotation? If i have taken account of that how do i compute distance with factors correlations matrix ?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="1154" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-09-08T05:51:11.067" LastActivityDate="2010-10-13T19:20:16.370" Title="Classification after factor analysis" Tags="&lt;classification&gt;&lt;clustering&gt;&lt;factor-analysis&gt;&lt;psychometrics&gt;" AnswerCount="3" CommentCount="3" FavoriteCount="2" />
  <row Id="2468" PostTypeId="2" ParentId="2466" CreationDate="2010-09-08T05:03:32.267" Score="5" ViewCount="" Body="&lt;p&gt;You can estimate via a binomial distribution. If there are $n$ draws, with replacement, from $k$ objects (with $k$ unknown), the probability of an object being drawn once in a single draw is $P = \frac{1}{k}$. Think of this as a coinflip now. The probability of exactly $m$ heads (i.e. $m$ duplicates) from $n$ trials is ${n \choose m} P^m (1-P)^{n-m}$.  Multiply this by $n$ to get the expected number of times observed (your plot). For large $n$ it can be a little hairy to back out $k$ from the data, but for small $m$, you can probably do fine assuming the $(1-P)$ term is equal to $1$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;edit&lt;/strong&gt;: one possible way to fix the numerical problems is to look at the ratios of counts. That is, if $P_m$ is the probability of drawing $m$ heads, then $P_{m} / P_{m+1}$ is equal to $(k-1)\frac{m+1}{n-m}$. Then look at the ratios of counts of duplicates in your data to get multiple estimates of $k$, then take the median or mean.&lt;/p&gt;&#xA;" OwnerUserId="795" LastEditorUserId="795" LastEditorDisplayName="" LastEditDate="2010-09-08T16:55:59.350" LastActivityDate="2010-09-08T16:55:59.350" />
  <row Id="2469" PostTypeId="1" AcceptedAnswerId="2472" CreationDate="2010-09-08T05:41:48.627" Score="4" ViewCount="591" Body="&lt;p&gt;&lt;strong&gt;Question:&lt;/strong&gt;&#xA;What is a good method for conducting post hoc tests of differences between group means after adjusting for the effect of a covariate?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Prototypical example:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Four groups, 30 participants per group (e.g., four different clinical psychology populations)&lt;/li&gt;&#xA;&lt;li&gt;Dependent Variable is numeric (e.g., intelligence scores)&lt;/li&gt;&#xA;&lt;li&gt;Covariate is numeric (e.g., index of socioeconomic status)&lt;/li&gt;&#xA;&lt;li&gt;Research questions concern whether any pair of groups are significantly different on the dependent variable after controlling for the covariate&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Related Questions&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What is the preferred method?&lt;/li&gt;&#xA;&lt;li&gt;What implementations are available in R?&lt;/li&gt;&#xA;&lt;li&gt;Are there any general references on how a covariate changes procedures for conducting post hoc tests?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-09-08T07:17:02.653" Title="Post hoc tests in ANCOVA" Tags="&lt;anova&gt;&lt;multiple-comparisons&gt;&lt;ancova&gt;" AnswerCount="2" FavoriteCount="3" />
  <row Id="2470" PostTypeId="2" ParentId="2469" CreationDate="2010-09-08T07:00:04.273" Score="1" ViewCount="" Body="&lt;p&gt;Combining simple methods that you can easily access from R and general principles you could use Tukey's HSD simply enough.  The error term from the ANCOVA will provide the error term for the confidence intervals.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In R code that would be...&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;n &amp;lt;- 30; k &amp;lt;- 4&#xA;y &amp;lt;- rnorm(n*k)&#xA;a &amp;lt;- factor(rep(1:k, n))&#xA;cov &amp;lt;- y + rnorm(n*k)&#xA;m &amp;lt;- aov(y ~ cov + a)&#xA;TukeyHSD(m)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(ignore the error in the result, it just means the covariate wasn't assessed, which is what you want)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That gives narrower confidence intervals than you get if you run the model without the cov... as expected.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any post hoc technique that relies on the residuals from the model for the error variance could easily be used.&lt;/p&gt;&#xA;" OwnerUserId="601" LastEditorUserId="601" LastEditorDisplayName="" LastEditDate="2010-09-08T07:05:16.717" LastActivityDate="2010-09-08T07:05:16.717" />
  <row Id="2471" PostTypeId="2" ParentId="2466" CreationDate="2010-09-08T07:09:54.603" Score="6" ViewCount="" Body="&lt;p&gt;This sounds like a form of 'mark and recapture' aka 'capture-recapture', a well-known technique in ecology (and some other fields such as epidemiology). Not my area but &lt;a href=&quot;http://en.wikipedia.org/wiki/Mark_and_recapture&quot; rel=&quot;nofollow&quot;&gt;the Wikipedia article on mark and recapture&lt;/a&gt; looks reasonable, though your situation is not the one to which the Lincoln–Petersen method explained there applies. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think shabbychef is one the right track for your situation, but using the Poisson distribution to approximate the binomial would probably make things a bit simpler and should be a very good approximation if the population size is very large, as in your examples. I think getting an explicit expression for the maximum likelihood estimate of the population size should then be pretty straightforward (see e.g. &lt;a href=&quot;http://en.wikipedia.org/wiki/Poisson_distribution#Maximum_likelihood&quot; rel=&quot;nofollow&quot;&gt;Wikipedia again&lt;/a&gt;), though i don't have time to work out the details right now.&lt;/p&gt;&#xA;" OwnerUserId="449" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-09-08T11:29:12.837" LastActivityDate="2010-09-08T11:29:12.837" CommentCount="1" />
  <row Id="2472" PostTypeId="2" ParentId="2469" CreationDate="2010-09-08T07:17:02.653" Score="4" ViewCount="" Body="&lt;p&gt;Multiple testing following ANCOVA, or more generally any GLM, but the comparisons now focus on the adjusted group/treatment or marginal means (i.e. what the scores would be if groups did not differ on the covariate of interest). To my knowledge, Tukey HSD and Scheffé tests are used. Both are quite conservative and will tend to bound type I error rate. The latter is preferred in case of unequal sample size in each group. I seem to remember that some people also use Sidak correction on specific contrasts (when it is of interest of course) as it is less conservative than the Bonferroni correction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Such tests are readily available in the R &lt;code&gt;multcomp&lt;/code&gt; package (see &lt;code&gt;?glht&lt;/code&gt;). The accompagnying vignette include example of use in the case of a simple linear model (section 2), but it can be extended to any other model form. Other examples can be found in the &lt;code&gt;HH&lt;/code&gt; packages (see &lt;code&gt;?MMC&lt;/code&gt;). Several MCP and resampling procedures (recommended for strong inferences, but it relies on a different approach to the correction for Type I error rate inflation) are also available in the &lt;code&gt;multtest&lt;/code&gt; package, through &lt;a href=&quot;http://www.bioconductor.org&quot; rel=&quot;nofollow&quot;&gt;Bioconductor&lt;/a&gt;, see refs (3–4). The definitive reference to multiple comparison is the book from the same authors: Dudoit, S. and van der Laan, M.J., &lt;em&gt;Multiple Testing Procedures with Applications to Genomics&lt;/em&gt; (Springer, 2008).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Reference 2 explained the difference between MCP in the general case (ANOVA, working with unadjusted means) vs. ANCOVA. There are also several papers that I can't remember actually, but I will look at them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other useful references:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Westfall, P.H. (1997). Multiple Testing of General Contrasts Using Logical Contraints and Correlations. &lt;em&gt;JASA&lt;/em&gt; &lt;strong&gt;92&lt;/strong&gt;: 299-306.&lt;/li&gt;&#xA;&lt;li&gt;Westfall, P.H. and Young, S.S. (1993) &lt;em&gt;Resampling Based Multiple Testing, Examples and Methods for p-Value Adjustment&lt;/em&gt;. John Wiley and Sons: New York.&lt;/li&gt;&#xA;&lt;li&gt;Pollard, K.S., Dudoit, S., and van der Laan, M.J. (2004). &lt;a href=&quot;http://works.bepress.com/sandrine_dudoit/18&quot; rel=&quot;nofollow&quot;&gt;Multiple Testing Procedures: R multtest Package and Applications to Genomics&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;li&gt;Taylor, S.L. Lang, D.T., and Pollard, K.S. (2007). &lt;a href=&quot;http://cran.r-project.org/doc/Rnews/Rnews_2007-3.pdf&quot; rel=&quot;nofollow&quot;&gt;Improvements to the multiple testing package multtest&lt;/a&gt;. &lt;em&gt;R News&lt;/em&gt; &lt;strong&gt;7(3)&lt;/strong&gt;: 52-55. &lt;/li&gt;&#xA;&lt;li&gt;Bretz, F., Genz, A., and Hothorn, L.A. (2001). On the numerical availability of multiple comparison procedures. &lt;em&gt;Biometrical Journal&lt;/em&gt;, &lt;strong&gt;43(5)&lt;/strong&gt;: 645–656.&lt;/li&gt;&#xA;&lt;li&gt;Hothorn, T., Bretz, F., and Westfall, P. (2008). &lt;a href=&quot;http://epub.ub.uni-muenchen.de/2120/1/tr019.pdf&quot; rel=&quot;nofollow&quot;&gt;Simultaneous Inference in General Parametric Models&lt;/a&gt;. Department of Statistics: Technical Reports, Nr. 19.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The first two are referenced in SAS PROC related to MCP.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-09-08T07:17:02.653" CommentCount="1" />
  <row Id="2473" PostTypeId="2" ParentId="2467" CreationDate="2010-09-08T07:36:36.987" Score="4" ViewCount="" Body="&lt;p&gt;One solution to your 1. question is to use cross-validation. You compute classification accuracy for models with different number of components and then pick one with the highest classification accuracy. You can check the references below:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.bepress.com/sagmb/vol3/iss1/art33/&quot; rel=&quot;nofollow&quot;&gt;PLS Dimension Reduction for Classification with Microarray Data&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.sciencedirect.com/science?_ob=ArticleURL&amp;amp;_udi=B6V03-4Y34W95-2&amp;amp;_user=10&amp;amp;_coverDate=07%2F31%2F2010&amp;amp;_rdoc=1&amp;amp;_fmt=high&amp;amp;_orig=search&amp;amp;_origin=search&amp;amp;_sort=d&amp;amp;_docanchor=&amp;amp;view=c&amp;amp;_searchStrId=1454239636&amp;amp;_rerunOrigin=google&amp;amp;_acct=C000050221&amp;amp;_version=1&amp;amp;_urlVersion=0&amp;amp;_userid=10&amp;amp;md5=e2279e3a8f1e8a43932e99beadcdd90b&amp;amp;searchtype=a&quot; rel=&quot;nofollow&quot;&gt;Rasch-based high-dimensionality data reduction and class prediction with applications to microarray gene expression data&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To my experience factor rotation does not improve classification accuracy. Please report your results.&lt;/p&gt;&#xA;" OwnerUserId="609" LastActivityDate="2010-09-08T07:36:36.987" CommentCount="5" />
  <row Id="2474" PostTypeId="2" ParentId="2467" CreationDate="2010-09-08T07:42:53.730" Score="2" ViewCount="" Body="&lt;p&gt;One approach that side-steps cross-validation to determine the optimal number of factors is to use the nonparametric Bayesian approaches for factor analysis. These approaches let the number of factors to be unbounded and eventually decided by the data. See &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.142.4273&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;this paper&lt;/a&gt; that uses such an approach for classification based on factor analysis.&lt;/p&gt;&#xA;" OwnerUserId="881" LastActivityDate="2010-09-08T07:42:53.730" CommentCount="1" />
  <row Id="2475" PostTypeId="2" ParentId="2458" CreationDate="2010-09-08T08:13:59.550" Score="0" ViewCount="" Body="&lt;p&gt;I am running R 2.11.0 on Windows 7 and the function for me for the file chooser is &lt;code&gt;file.choose()&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So your code would look like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;students &amp;lt;- read.table(file.choose(), header=T, sep=&quot;\t&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1195" LastActivityDate="2010-09-08T08:13:59.550" CommentCount="1" />
  <row Id="2476" PostTypeId="1" AcceptedAnswerId="2478" CreationDate="2010-09-08T10:23:59.423" Score="5" ViewCount="421" Body="&lt;p&gt;To cluster (text) documents you need a way of measuring similarity between pairs of documents.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Two alternatives are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Compare documents as term vectors using &lt;strong&gt;&lt;a href=&quot;http://nlp.stanford.edu/IR-book/html/htmledition/dot-products-1.html#8077&quot; rel=&quot;nofollow&quot;&gt;Cosine Similarity&lt;/a&gt;&lt;/strong&gt; - and TF/IDF as the weightings for terms.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Compare each documents probability distribution using f-divergence e.g. &lt;strong&gt;&lt;a href=&quot;http://nlp.stanford.edu/IR-book/html/htmledition/extended-language-modeling-approaches-1.html#15544&quot; rel=&quot;nofollow&quot;&gt;Kullback-Leibler divergence&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Is there any intuitive reason to prefer one method to the other (assuming average document sizes of 100 terms)?&lt;/p&gt;&#xA;" OwnerUserId="1212" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-17T20:20:49.300" LastActivityDate="2010-09-17T20:20:49.300" Title="Measuring Document Similarity" Tags="&lt;information-retrieval&gt;" AnswerCount="1" FavoriteCount="3" />
  <row Id="2477" PostTypeId="2" ParentId="125" CreationDate="2010-09-08T10:28:33.510" Score="3" ViewCount="" Body="&lt;p&gt;I have read some parts of &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0387922997&quot; rel=&quot;nofollow&quot;&gt;A First Course in Bayesian Statistical Methods&lt;/a&gt;, and I found it easy to follow. (Example R-code is provided throughout the text)&lt;/p&gt;&#xA;" OwnerUserId="339" LastActivityDate="2010-09-08T10:28:33.510" CommunityOwnedDate="2010-09-08T10:28:33.510" />
  <row Id="2478" PostTypeId="2" ParentId="2476" CreationDate="2010-09-08T10:49:23.317" Score="6" ViewCount="" Body="&lt;p&gt;For text documents, the feature vectors can be very high dimensional and sparse under any of the standard representations (bag of words or TF-IDF etc). Measuring distances directly under such a representation may not be reliable since it is a known fact that in very high dimensions, distance between any two points starts to look the same. One way to deal with this is to reduce the data dimensionality by using &lt;a href=&quot;http://en.wikipedia.org/wiki/Principal_component_analysis&quot; rel=&quot;nofollow&quot;&gt;PCA&lt;/a&gt; or LSA (&lt;a href=&quot;http://en.wikipedia.org/wiki/Latent_semantic_analysis&quot; rel=&quot;nofollow&quot;&gt;Latent Semantic Analysis&lt;/a&gt;; also known as &lt;a href=&quot;http://en.wikipedia.org/wiki/Latent_semantic_indexing&quot; rel=&quot;nofollow&quot;&gt;Latent Semantic Indexing&lt;/a&gt;) and then measure the distances in the new space. Using something like LSA over PCA is advantageous since it can give a meaningful representation in terms of &quot;semantic concepts&quot;, apart from measuring distances in a lower dimensional space.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Comparing documents based on the probability distributions is usually done by first computing the topic distribution of each document (using something like &lt;a href=&quot;http://en.wikipedia.org/wiki/Latent_Dirichlet_allocation&quot; rel=&quot;nofollow&quot;&gt;Latent Dirichlet Allocation&lt;/a&gt;), and then computing some sort of divergence (e.g., KL divergence) between the topic distributions of pair of documents. In a way, it's actually kind of similar to doing LSA first and then measuring distances in the LSA space using KL-divergence between the vectors (instead of cosine similarity).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;KL-divergence is a distance measure for comparing distributions so it may be preferable if the document representation is in terms of some distribution (which is often actually the case -- e.g., documents represented as distribution over topics, as in LDA). Also note that under such a representation, the entries in the feature vector would sum to one (since you are basically treating the document as a &lt;em&gt;distribution&lt;/em&gt; over topics or semantic concepts).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also see a &lt;a href=&quot;http://metaoptimize.com/qa/questions/1595/how-to-compute-the-semantic-distance-between-two-text-documents&quot; rel=&quot;nofollow&quot;&gt;related thread here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="881" LastEditorUserId="881" LastEditorDisplayName="" LastEditDate="2010-09-08T10:58:25.990" LastActivityDate="2010-09-08T10:58:25.990" CommentCount="6" />
  <row Id="2481" PostTypeId="1" AcceptedAnswerId="2495" CreationDate="2010-09-08T11:33:12.393" Score="3" ViewCount="249" Body="&lt;p&gt;I have found the term &quot;asymptotic power of a statistical test&quot; only related to the Kolmogorov-Smirnov test (to be precise: asyptotic power = 1). What does this term acctually mean? In my opinion it should be someting like this: &quot;if the alternative hypothesis is true, than for every significance level alpha there exists a sample size n that the selected test would reject the null hypothesis&quot;. Is &quot;my&quot; definition correct? According to &quot;my defintion&quot; the majority of classical tests (t-tset, ...) should have the asymptotic power 1, not only KS test. Am I right? ;)&lt;/p&gt;&#xA;" OwnerUserId="1215" LastActivityDate="2010-09-11T00:47:08.027" Title="Asymptotic power" Tags="&lt;hypothesis-testing&gt;&lt;statistical-significance&gt;&lt;power&gt;" AnswerCount="3" CommentCount="2" />
  <row Id="2482" PostTypeId="2" ParentId="2466" CreationDate="2010-09-08T11:44:25.410" Score="8" ViewCount="" Body="&lt;p&gt;Hi Aaron, How's the Garvan?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is we don't know how many zero counts are observed. We have to estimate this. A classic statistical procedure for situations like this is the Expectation-Maximisation algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A simple example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assume we draw from an unknown population (of 1,000,000) with a poisson constant of 0.2.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;counts &amp;lt;- rpois(1000000, 0.2)&#xA;table(counts)&#xA;&#xA;     0      1      2      3      4      5&#xA;818501 164042  16281   1111     62      3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But we don't observe the zero counts. Instead we observe this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;table &amp;lt;- c(&quot;0&quot;=0, table(counts)[2:6])&#xA;&#xA;table&#xA;&#xA;     0      1      2      3      4      5&#xA;     0 164042  16281   1111     62      3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Possible frequencies observed&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;k &amp;lt;- c(&quot;0&quot;=0, &quot;1&quot;=1, &quot;2&quot;=2, &quot;3&quot;=3, &quot;4&quot;=4, &quot;5&quot;=5)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Initialise mean of Poisson distribution - just take a guess (we know it's 0.2 here).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;lambda &amp;lt;- 1 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Expectation - Poisson Distribution&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;P_k &amp;lt;- lambda^k*exp(-lambda)/factorial(k)&#xA;P_k&#xA;              0           1           2           3           4           5&#xA;0.367879441 0.367879441 0.183939721 0.061313240 0.015328310 0.003065662  &#xA;n0 &amp;lt;- sum(table[2:6])/(1 - P_k[1]) - sum(table[2:6])&#xA;&#xA;&#xA;n0&#xA;       0&#xA;105628.2     &#xA;table[1] &amp;lt;-  105628.2&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Maximisation&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;lambda_MLE &amp;lt;- (1/sum(table))*(sum(table*k))        &#xA;lambda_MLE        &#xA;[1] 0.697252        &#xA;lambda &amp;lt;- lambda_MLE&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Second iteration&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;P_k &amp;lt;- lambda^k*exp(-lambda)/factorial(k)        &#xA;n0 &amp;lt;- sum(table[2:6])/(1 - P_k[1]) - sum(table[2:6])       &#xA;table[1] &amp;lt;-  n0 &#xA;lambda &amp;lt;- (1/sum(table))*(sum(table*k))&#xA;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; population lambda_MLE&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;[1,]   361517.1  0.5537774&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Now iterate until convergence:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for (i in 1:200) {  &#xA;P_k &amp;lt;- lambda^k*exp(-lambda)/factorial(k)  &#xA;n0 &amp;lt;- sum(table[2:6])/(1 - P_k[1]) - sum(table[2:6])&#xA;table[1] &amp;lt;-  n0&#xA;lambda &amp;lt;- (1/sum(table))*(sum(table*k))&#xA;}&#xA;cbind( population = sum(table), lambda_MLE)&#xA;     population lambda_MLE&#xA;[1,]    1003774  0.1994473&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Our population estimate is 1003774 and our poisson rate is estimated at 0.1994473 - this is the estimated proportion of the population sampled. The main problem you will have in the typical biological problems you are dealing with is assumption that the poisson rate is a constant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sorry for the long-winded post - this wiki is not really suitable for R code.&lt;/p&gt;&#xA;" OwnerUserId="521" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-08T13:35:34.850" LastActivityDate="2010-09-08T13:35:34.850" CommentCount="3" />
  <row Id="2483" PostTypeId="1" AcceptedAnswerId="2539" CreationDate="2010-09-08T11:49:25.087" Score="2" ViewCount="210" Body="&lt;p&gt;&lt;strong&gt;[Edit]&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In light of the feedback I have received, I have added some more information below, to help clarify what I am seeking to 'model'.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Additionally, I have reformualted the problem from a &quot;dog barking warning system&quot; to something else which hopefully, has less ambiguity. Instead, I will repose the problem as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lets assume that my neighbour is a &quot;mad scientist&quot;, who claims to have invented a &quot;terrorist event&quot; forecasting machine. The machine has three colored bulbs - one of which will be illuminated, depending on the severity of a terrorist event forecasted by the machine.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The light bulbs have the following interpretation:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;No light bulbs illumintaed means there is no percived threat&lt;/li&gt;&#xA;&lt;li&gt;Green bulb illuminated means there is a level 1 threat imminent&lt;/li&gt;&#xA;&lt;li&gt;Orange bulb illuminated there is a level 2 threat imminent&lt;/li&gt;&#xA;&lt;li&gt;Red bulb illuminated there is a level 3 threat imminent&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;To avoid getting too pedantic, lets assume for the sake of argument that the following terms are defined and agreed upon:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Level 1 terrorist event&lt;/li&gt;&#xA;&lt;li&gt;Level 2 terrorist event&lt;/li&gt;&#xA;&lt;li&gt;Level 3 terrorist event&lt;/li&gt;&#xA;&lt;li&gt;&quot;imminent&quot; terrorist event&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;What I am trying to find out, is if there is a way I can design an experiment that can help me say with a degree of confidence, whether the scientists claims are statistically significant or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;if the claims are found to be statistically significant, then I would like to be able to add a CI (confidence interval) to the claim. So I can say something like - if the orange bulb is illuminated, then a level 2 terrorist event will occur withing a x% CI.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having said that - IIRC, CI are only meaningful for N~ RV.? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an aside, I was thinking that something akin to Fischer's &lt;a href=&quot;http://en.wikipedia.org/wiki/Lady_tasting_tea&quot; rel=&quot;nofollow&quot;&gt;tea experiment&lt;/a&gt; (or running a bernoulli test would be usefull, but my stats 'fu is not what it used to be ...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The purpose of such a model (asssuming that the scientist machine really does work), is to act as a decision support system - i.e. decisions can be taken on the output of the machine - IF it can be depended upon.&lt;/p&gt;&#xA;" OwnerUserId="1216" LastEditorUserId="1216" LastEditorDisplayName="" LastEditDate="2010-09-10T13:59:33.863" LastActivityDate="2010-09-10T18:25:12.273" Title="How to model and test a decision support system (e.g. a terrorist warning system)?" Tags="&lt;modeling&gt;&lt;model-selection&gt;&lt;best-practices&gt;&lt;experiment-design&gt;" AnswerCount="3" CommentCount="3" />
  <row Id="2486" PostTypeId="2" ParentId="2481" CreationDate="2010-09-08T12:54:35.677" Score="2" ViewCount="" Body="&lt;p&gt;As I understood it, the asymptotic power is the hypothetical power when the effect size goes to zero and the sample size to infinity. Basically it should be 0 or 1, indicating whether the test cannot or can distinguish an arbitrary small deviation from the null hypothesis when the sample size is sufficiently large. &lt;/p&gt;&#xA;" OwnerUserId="1124" LastActivityDate="2010-09-08T12:54:35.677" CommentCount="1" />
  <row Id="2487" PostTypeId="2" ParentId="2481" CreationDate="2010-09-08T13:43:43.877" Score="1" ViewCount="" Body="&lt;p&gt;Yes, you are right. I would only replace &quot;there exists a sample size n that the selected test would reject the null hypothesis&quot; with &quot;for every e&gt;0 there exists a sample size n_0 such that the probability to reject the null hypothesis is greater than 1-e for all n&gt;n_0&quot;.&lt;/p&gt;&#xA;" OwnerUserId="1219" LastActivityDate="2010-09-08T13:43:43.877" />
  <row Id="2489" PostTypeId="2" ParentId="2483" CreationDate="2010-09-08T15:43:01.230" Score="3" ViewCount="" Body="&lt;p&gt;I have thought about this question a while and have come to the following rather vague answer to a, in my eyes, rather vague question. Despite the asker's wish I don't use the word model as I didn't get it into my thinking on this problem. Sorry for that.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;What is your hypothesis?&lt;/strong&gt;&lt;br&gt;&#xA;I see three different possible interpretations of the hypothesis:&lt;br&gt;&#xA;(a) The one implied by the question: If the dog barks one time, then it rains. (...) This hypotheses would be false if and only if the dog barks one time and it is currently not raining (i.e., &lt;a href=&quot;http://en.wikipedia.org/wiki/Material_conditional&quot; rel=&quot;nofollow&quot;&gt;a truth functional material implication&lt;/a&gt;)&lt;br&gt;&#xA;(b) The one that seems more plausible to me: If it rains, then the dog barks exactly one time. (...) This hypotheses would be false if and only if it is raining and the dog does not bark exactly one time&lt;br&gt;&#xA;(c) The stochastic hypotheses: Out of a population of dog's the neighbor's dog is a pretty reliable weather indicator compared to the other dogs and barks according to the mentioned scheme.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;How to test your hypotheses?&lt;/strong&gt;&lt;br&gt;&#xA;(a) &amp;amp; (b) Verifying these general claims is pretty difficult. It is like asking if it is really true that the sun always rises in the morning. There are pretty good arguments from philosophers of science as &lt;a href=&quot;http://en.wikipedia.org/wiki/Karl_Popper&quot; rel=&quot;nofollow&quot;&gt;Karl Popper&lt;/a&gt; and &lt;a href=&quot;http://en.wikipedia.org/wiki/Imre_Lakatos&quot; rel=&quot;nofollow&quot;&gt;the like&lt;/a&gt; that we can never proof claims like these but only try to falsify them and treat them as true as long as they haven't been falsified. In our example one could do two things: Firstly we could simply survey the weather conditions and the dog's behavior until we get bored and accept the hypotheses as true or see any pairing that is inconsistent with our hypotheses. Secondly, create an environment in which you can control the weather conditions (i.e., run an experiment) take the dog into this environment and proceed with surveying until boredom/falsification. Note however that in case of any falsification one could always argue that e.g., this rain was not really a rain but rather a light drizzle and so on. See &lt;a href=&quot;http://en.wikipedia.org/wiki/Imre_Lakatos&quot; rel=&quot;nofollow&quot;&gt;Lakatos&lt;/a&gt; and also further the &lt;a href=&quot;http://en.wikipedia.org/wiki/Duhem%E2%80%93Quine_thesis&quot; rel=&quot;nofollow&quot;&gt;Duhem-Quine Thesis&lt;/a&gt;. To come back to my example: at some point humanity accepted that the sun always rises in the morning although we can never be sure of it!&lt;br&gt;&#xA;(c) To test this claim I would do the following: I would take the neighbor's dog and some random dog from the same population and bring them, separately, to the above mentioned environment where you can control the weather. There you have four different stimuli (S1 = no rain, S2 = rain, ...) which I would present to each dog in a random order and record the reaction to each stimuli. Note that you need to repeat each stimulus several times (e.g., 30 times). Then you could enter the data with stimuli as the subject, the dog as the between-subjects manipulations and the stimuli as the within-subjects manipulations into an Analysis of Variance (ANOVA). Your hypotheses would probably predict an interaction of dog with stimuli. If this would occur you could run tests testing for: the random dog which should have equal barking for all four stimuli, and the neighbor's dog which should stick to the norm (e.g., one sample t-tests against a value). &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Note that your test of hypotheses (c) would be easier and more 'traditional' if you would suspect that there exist two populations of dogs, one weather forecasting and one weather blind population of dogs. Then you could sample some (e.g., 30) from each population and aggregate the data from each stimuli. Dogs would be the subjects, population the between-subject manipulation and stimuli the within-subject manipulation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Two caveats: I don't know if the described way is the usual one for testing for single outliers . I don't know if there exist more common methods to test for deviance from a norm.&lt;/p&gt;&#xA;" OwnerUserId="442" LastEditorUserId="442" LastEditorDisplayName="" LastEditDate="2010-09-08T15:49:37.677" LastActivityDate="2010-09-08T15:49:37.677" />
  <row Id="2490" PostTypeId="2" ParentId="2483" CreationDate="2010-09-08T16:31:59.693" Score="1" ViewCount="" Body="&lt;p&gt;Count sucesses an failures when the barking patterns occur and get a contingency table. Then do a Chi squared test against 50/50 occurences (ie null hypothesis is that barking is not correlated with getting the correct weather pattern).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, there is also the other side of this to consider: what barking pattern precedes a particular weather type.&lt;/p&gt;&#xA;" OwnerUserId="229" LastActivityDate="2010-09-08T16:31:59.693" />
  <row Id="2491" PostTypeId="2" ParentId="321" CreationDate="2010-09-08T17:30:41.103" Score="2" ViewCount="" Body="&lt;p&gt;The second paper you cite seems to contain the answer to your question. To recap; mathematically, the main difference is in the shape of the loss function being used. Friedman, Hastie, and Tibshirani's loss function being easier to optimize at each iteration. – &lt;/p&gt;&#xA;" OwnerUserId="603" LastActivityDate="2010-09-08T17:30:41.103" />
  <row Id="2492" PostTypeId="1" AcceptedAnswerId="2498" CreationDate="2010-09-08T17:47:21.820" Score="10" ViewCount="730" Body="&lt;p&gt;A former colleague once argued to me as follows: &quot;we usually apply normality tests to the results of processes that, under the null, generate random variables that are only &lt;em&gt;asymptotically&lt;/em&gt; or &lt;em&gt;nearly&lt;/em&gt; normal (with the 'asymptotically' part dependent on some quantity which we cannot make large); In the era of cheap memory, big data, and fast processors, normality tests should &lt;em&gt;always&lt;/em&gt; reject the null of normal distribution for large (though not insanely large) samples. And so, perversely, normality tests should only be used for small samples, when they presumably have lower power and less control over type I rate.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this a valid argument? Is this a well-known argument? Are there well known tests for a 'fuzzier' null hypothesis than normality?&lt;/p&gt;&#xA;" OwnerUserId="795" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-09-10T16:14:32.013" LastActivityDate="2010-09-18T02:32:42.093" CommunityOwnedDate="2010-09-08T17:47:21.820" Title="Normality Testing: 'Essentially Useless?'" Tags="&lt;hypothesis-testing&gt;&lt;normality&gt;&lt;philosophical&gt;" AnswerCount="4" CommentCount="6" FavoriteCount="7" />
  <row Id="2493" PostTypeId="1" AcceptedAnswerId="2497" CreationDate="2010-09-08T19:48:42.110" Score="9" ViewCount="160" Body="&lt;p&gt;I'm looking for the appropriate theoretical framework or speciality to help me deal with understanding how to deal with the errors that the GPS system has - especially when dealing with routes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Fundamentally, I'm looking for the requirements on the data and any algorithms to use to be able to establish the length of a trail. The answer needs to be trustworthy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A friend of mine was the race director of a race which was billed as 160km but the Garmin watches everybody has makes it more like 190km+. It caused quite some grief at the finish line, let me tell you!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my friend went back to the course with various GPS devices in order to remap it and the results are interesting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using a handheld Garmin Oregon 300 she got 33.7km for one leg. For the same leg on a wrist watch Garmin Forerunner 310xt it came out to 38.3km.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I got the data from the Oregon it was obvious that it was only recording data every 90 seconds or so. The Forerunner does it every couple of seconds.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I plotted the data from the Oregon I could see that it got confused by some switchbacks and put a line straight through them and a curve was made a little less.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I muse that the difference in the recording frequency is much of the explanation. i.e. by recording every couple of seconds the Forerunner is closer to the real route. However, there will be an amount of error because of the way GPS works. If the points recorded are spread around the real route randomly (because of the error) then the total distance will be larger than the real route. (A wiggle line going either side of a straight line is longer than the straight line).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, my questions: 1. Are there any techniques I can use on a single dataset to reduce the error in a valid way? 2. Does my theory about the difference in recording frequency hold water? 3. If I have multiple recordings of the same routes are there any valid techniques to combine them to get closer to the real route?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As I say, I don't really know what to search for to find any useful science about this. I'm looking for ways to establish just how long a given piece of trail is and it is very important to people. An extra 30km in a race is an extra 5+ hours we weren't expecting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As requested here is some sample data:&#xA;&lt;img src=&quot;http://i.imgur.com/GZGCh.jpg&quot; alt=&quot;Summary image&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://s3.amazonaws.com/trailhunger-stackexchange/trapper-lake-detailed.csv&quot; rel=&quot;nofollow&quot;&gt;Detailed high-frequency sample data&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://s3.amazonaws.com/trailhunger-stackexchange/trapper-lake-low-sample.csv&quot; rel=&quot;nofollow&quot;&gt;Low-frequency sample data&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for any advice you can give.&lt;/p&gt;&#xA;" OwnerUserId="1227" LastEditorUserId="1227" LastEditorDisplayName="" LastEditDate="2010-09-09T02:50:02.270" LastActivityDate="2010-09-09T02:50:02.270" Title="Managing error with GPS routes (theoretical framework?)" Tags="&lt;error&gt;&lt;sampling&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="3" />
  <row Id="2495" PostTypeId="2" ParentId="2481" CreationDate="2010-09-08T20:23:12.640" Score="2" ViewCount="" Body="&lt;p&gt;The definition above (a fixed alternative, sample size going to infinity) is more precisely related to the &lt;em&gt;consistency&lt;/em&gt; (or not) of a hypothesis test.  That is, a test is consistent against a fixed alternative if the power function approaches 1 at that alternative.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Asymptotic power is something different.  As Joris remarked, with &lt;strong&gt;asymptotic power&lt;/strong&gt; the alternatives $\theta_n$ are changing, are converging to the null value $\theta_0$ (on the order of $\sqrt n$, say) while the sample size marches to infinity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Under some regularity conditions (for example, the test statistic has a monotone likelihood ratio, is asymptotically normal, has asymptotic variance $\tau$ continuous in $\theta$, yada yada yada) if $\sqrt n(\theta_n - \theta_0)$ goes to $\delta$ then the power function goes to $\Phi(\delta/\tau - z_\alpha)$, where  $\Phi$ is the standard normal CDF.  This last quantity is called the asymptotic power of just such a test.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See Lehmann's $\underline{\mbox{Elements of Large Sample Theory}}$ for discussion and worked out examples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By the way, yes, the majority of classical tests are consistent.&lt;/p&gt;&#xA;" OwnerUserId="1108" LastEditorUserId="1108" LastEditorDisplayName="" LastEditDate="2010-09-11T00:47:08.027" LastActivityDate="2010-09-11T00:47:08.027" CommentCount="2" />
  <row Id="2496" PostTypeId="2" ParentId="2446" CreationDate="2010-09-08T21:21:58.237" Score="2" ViewCount="" Body="&lt;p&gt;This a very simple multi-level (a.k.a. hierarchical) model. Douglas Bate his currently working on a book on the subject (draft avalaible here: &lt;a href=&quot;http://lme4.r-forge.r-project.org/book/&quot; rel=&quot;nofollow&quot;&gt;http://lme4.r-forge.r-project.org/book/&lt;/a&gt;). While there are many books on this subject, Doug's has the added benefit of being designed arround the 'lme' R package, a very handy pakage designed to fit such model. I think it is best for you to go and read the first chapter of that book as well as practice the exemples provided there inside R. You can always come back with more specific questions.&lt;/p&gt;&#xA;" OwnerUserId="603" LastEditorUserId="603" LastEditorDisplayName="" LastEditDate="2010-09-08T21:27:01.180" LastActivityDate="2010-09-08T21:27:01.180" CommentCount="6" />
  <row Id="2497" PostTypeId="2" ParentId="2493" CreationDate="2010-09-08T22:10:12.760" Score="6" ViewCount="" Body="&lt;p&gt;This is a well studied problem in geospatial science--you can find discussions of it on GIS forums.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, note that the wiggles do not necessarily increase the route's length, because many of them actually cut inside curves.  (I have evaluated this by having an entire classroom of students digitize the same path and then I compared the paths.)  There really is a lot of cancellation.  Also, we can expect that readings taken just a few seconds apart will have strongly, positively, correlated errors.  Thus the measured path should wiggle only gradually around the true path.  Even large departures don't affect the length much.  For example, if you deviate by (say) 5 meters laterally in the middle of a 100 m straight stretch, your estimate of the length only goes up to $2 \sqrt{50^2 + 5^2} = 100.5$, a 0.5% error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is difficult to compare two arbitrary paths in an objective way.  One of the better methods, I believe, is a form of bootstrapping: subsample (or otherwise generalize) the most detailed path you have.  Plot its length as a function of the amount of subsampling.  If you express the subsampling as a typical vertex-to-vertex distance, you can extrapolate a fit to a zero distance, which can provide an excellent estimate of the path length.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With multiple recordings, you can create a 2D kernel smooth of each, sum the smooths, and subject that to a topographic analysis to look for a &quot;ridgeline.&quot;  You won't get a single connected line, usually, but you can often patch the ridges together into a continuous path.  People have used this method to average hurricane tracks, for example.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-09-08T22:10:12.760" CommentCount="3" />
  <row Id="2498" PostTypeId="2" ParentId="2492" CreationDate="2010-09-08T22:23:19.163" Score="16" ViewCount="" Body="&lt;p&gt;It's not an argument. It is a (a bit strongly stated) fact that formal normality tests always reject on the huge sample sizes we work with today. It's even easy to proof that when n gets large, even the smallest deviation from perfect normality will lead to a significant result. And as every dataset has some degree of randomness...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let me illustrate with the Shapiro-Wilks test. If you construct an almost-normal distribution and do a small simulation, you get more or less following results in R :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x &amp;lt;- replicate(100,{ # generates 100 different tests on each distribution&#xA;  c(&#xA;    shapiro.test(rnorm(10)+c(1,0,2,0,1))$p.value,&#xA;    shapiro.test(rnorm(100)+c(1,0,2,0,1))$p.value,&#xA;    shapiro.test(rnorm(1000)+c(1,0,2,0,1))$p.value,&#xA;    shapiro.test(rnorm(5000)+c(1,0,2,0,1))$p.value&#xA;    )&#xA;  } # rnorm gives a random draw from the normal distribution&#xA;)&#xA;rownames(x)&amp;lt;-c(&quot;n10&quot;,&quot;n100&quot;,&quot;n1000&quot;,&quot;n5000&quot;)&#xA;&#xA;rowMeans(x&amp;lt;0.05) # the proportion of significant deviations&#xA;    n10  n100 n1000 n5000 &#xA;    0.04  0.04  0.20  0.87 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;so in 87% of the cases, the last distribution is not seen any more as a normal distribution. Yet, if you see the qq plots, you would never ever decide on a deviation from normality. Below you see as an example the qq-plots for one set of random samples &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/Rg5nU.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;with p-values&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; n10  n100 n1000 n5000 &#xA;0.760 0.681 0.164 0.007 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1124" LastEditorUserId="1124" LastEditorDisplayName="" LastEditDate="2010-09-08T22:29:39.747" LastActivityDate="2010-09-08T22:29:39.747" CommunityOwnedDate="2010-09-08T22:23:19.163" CommentCount="8" />
  <row Id="2499" PostTypeId="1" AcceptedAnswerId="2506" CreationDate="2010-09-09T00:15:07.943" Score="5" ViewCount="494" Body="&lt;p&gt;there are several distinct usages:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;kernel density estimation&lt;br&gt;&#xA;kernel trick&lt;br&gt;&#xA;kernel smoothing&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please explain what the &quot;kernel&quot; in them means, in plain English, in your own words&lt;/p&gt;&#xA;" OwnerUserId="74" LastEditorUserId="74" LastEditorDisplayName="" LastEditDate="2010-09-09T03:23:23.090" LastActivityDate="2010-09-11T01:37:36.063" Title="Explain what a kernel is, in plain English" Tags="&lt;kernel&gt;" AnswerCount="3" CommentCount="2" FavoriteCount="8" />
  <row Id="2500" PostTypeId="2" ParentId="2499" CreationDate="2010-09-09T01:21:55.677" Score="1" ViewCount="" Body="&lt;p&gt;There appear to be at least two different meanings of &quot;kernel&quot;: one more commonly used in statistics; the other in machine learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In statistics &quot;kernel&quot; is most commonly used to refer to kernel density estimation and kernel smoothing.&#xA;&lt;a href=&quot;http://en.wikipedia.org/wiki/Kernel_density_estimation&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Kernel_density_estimation&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Kernel_smoothing&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Kernel_smoothing&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A straightforward explanation of kernels in density estimation can be found here:&#xA;&lt;a href=&quot;http://school.maths.uwa.edu.au/~duongt/seminars/intro2kde/&quot; rel=&quot;nofollow&quot;&gt;http://school.maths.uwa.edu.au/~duongt/seminars/intro2kde/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In machine learning &quot;kernel&quot; is usually used to refer to the &quot;kernel trick&quot;, a method of using a linear classifier to solve a non-linear problem &quot;by mapping the original non-linear observations into a higher-dimensional space&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Kernel_trick&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Kernel_trick&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A simple visualisation might be to imaging that all of class 0 are with radius r of the origin in an x, y plane (class 0: x^2 + y^2 &amp;lt; r^2); and all of class 1 are beyond radius r in that plane (class 1: x^2 + y^2 &gt; r^2). No linear separator is possible, but clearly a circle of radius r will perfectly separate the data. We can transform the data into three dimensional space by calculating three new variables x^2, y^2 and sqrt(2)*x*y. The two classes will now be separable by a plane in this 3 dimensional space. The equation of that optimally separating hyperplane where z1 = x^2, z2 = y^2 and z3 = sqrt(2)*x*y is z1 +  z2 = 1, and in this case omits z3. (If the circle is off-set from the origin, the optimal separating hyperplane will vary in z3 as well.) The kernel is the mapping function which calculates the value of the 2-dimensional data in 3-dimensional space.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are other uses of &quot;kernels&quot; in mathematics, but these seem to be the main ones in statistics:&#xA;&lt;a href=&quot;http://en.wikipedia.org/wiki/Kernel_%28mathematics%29&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Kernel_%28mathematics%29&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="521" LastEditorUserId="521" LastEditorDisplayName="" LastEditDate="2010-09-09T02:13:33.667" LastActivityDate="2010-09-09T02:13:33.667" CommentCount="1" />
  <row Id="2501" PostTypeId="2" ParentId="2492" CreationDate="2010-09-09T02:35:31.153" Score="6" ViewCount="" Body="&lt;p&gt;When thinking about whether normality testing is 'essentially useless', one first has to think about what it is supposed to be useful for. Many people (well... at least, many scientists) misunderstand the question the normality test answers. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The question normality tests answer: Is there convincing evidence of any deviation from the Gaussian ideal? With moderately large real data sets, the answer is almost always yes.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The question scientists often expect the normality test to answer: Do the data deviate enough  from the Gaussian ideal to &quot;forbid&quot; use of a test that assumes a Gaussian distribution? Scientists often want the normality test to be the referee that decides when to abandon conventional (ANOVA, etc.) tests and instead analyze transformed data or use a rank-based nonparametric test or a resampling or bootstrap approach. For this purpose, normality tests are not very useful.&lt;/p&gt;&#xA;" OwnerUserId="25" LastEditorUserId="25" LastEditorDisplayName="" LastEditDate="2010-09-09T21:37:22.197" LastActivityDate="2010-09-09T21:37:22.197" CommunityOwnedDate="2010-09-09T02:35:31.153" CommentCount="1" />
  <row Id="2504" PostTypeId="1" AcceptedAnswerId="2511" CreationDate="2010-09-09T06:01:05.300" Score="7" ViewCount="217" Body="&lt;p&gt;Is it possible to test for finiteness (or existence) of the variance of a random variable given a sample? As a null, either {the variance exists and is finite} or {the variance does not exist/is infinite} would be acceptable. Philosophically (and computationally), this seems very strange because there should be no difference between a population without finite variance, and one with a very very large variance (say &gt; $10^{400}$), so I am not hopeful this problem can be solved.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One approach that had been suggested to me was via the Central Limit Theorem: assuming the samples are i.i.d., and the population has finite mean, one could check, somehow, whether the sample mean has the right standard error with increasing sample size. I'm not sure I believe this method would work, though. (In particular, I don't see how to make it into a proper test.)&lt;/p&gt;&#xA;" OwnerUserId="795" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-03T13:01:24.287" LastActivityDate="2011-04-04T01:20:36.233" Title="Test for finite variance?" Tags="&lt;hypothesis-testing&gt;&lt;variance&gt;&lt;central-limit-theorem&gt;&lt;philosophical&gt;" AnswerCount="4" />
  <row Id="2506" PostTypeId="2" ParentId="2499" CreationDate="2010-09-09T06:09:23.923" Score="3" ViewCount="" Body="&lt;p&gt;In both statistics (kernel density estimation or kernel smoothing) and machine learning (kernel methods) literature, kernel is used as a measure of similarity. In particular, the kernel function k(x,.) defines the distribution of similarities of points around a given point x. k(x,y) denotes the similarity of point x with another given point y.&lt;/p&gt;&#xA;" OwnerUserId="881" LastEditorUserId="881" LastEditorDisplayName="" LastEditDate="2010-09-09T06:53:49.670" LastActivityDate="2010-09-09T06:53:49.670" CommentCount="2" />
  <row Id="2507" PostTypeId="2" ParentId="2492" CreationDate="2010-09-09T08:59:38.777" Score="3" ViewCount="" Body="&lt;p&gt;Let me add one small thing:&lt;br&gt;&#xA;&lt;strong&gt;Performing a normality test without taking its alpha-error into account heightens your overall probability of performing an alpha-error.&lt;/strong&gt;  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You shall never forget that each additional test does this as long as you don't control for alpha-error accumulation. Hence, another good reason to dismiss normality testing.&lt;/p&gt;&#xA;" OwnerUserId="442" LastActivityDate="2010-09-09T08:59:38.777" CommunityOwnedDate="2010-09-09T08:59:38.777" CommentCount="3" />
  <row Id="2509" PostTypeId="2" ParentId="1923" CreationDate="2010-09-09T12:02:48.057" Score="3" ViewCount="" Body="&lt;p&gt;I think the standard solution goes as follows.  I'll just do the scalar case, the multi parameter case is similar. Your objective function is $g_N(p,X_1,\dots,X_N)$&#xA;where $p$ is the parameter you want to estimate and $X_1,\dots,X_N$ are the observed random variables.  For notational simplicity I will just write the objective function as $g(p)$ from now on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We need some assumptions. Firstly I'll assume that you have already shown that the maximiser of $g$ is a consistent estimator (this actually tends to be the hardest part!).  So, if the `true' value of the parameter is $p_0$ and the estimator is&#xA;$$ \hat{p} =  \arg\max_{p} g(p) $$&#xA;then we have that $\hat{p} \rightarrow p_0$ almost surely as $N \rightarrow \infty$. Our second assumption is that $g$ is twice differentiable in a neighbourhood about $p_0$ (you can sometimes get away without this assumption, but the solution becomes more problem dependent). In view of strong consistency we can and will assume that $\hat{p}$ is inside this neighbourhood.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Denote by $g'$ and $g''$ the first and second derivatives of $g$ with respect to $p$. Then&#xA;$$ g'(p_0) - g'(\hat{p}) = (p_0 - \hat{p})g''(\bar{p})$$&#xA;where $\bar{p}$ lies between $\hat{p}$ and $p_0$. Now because $\hat{p}$ maximises $g$ we have $g'(\hat{p}) = 0$ so&#xA;$$(p_0 - \hat{p}) = \frac{g'(p_0)}{g''(\bar{p})}$$&#xA;and because $\hat{p} \rightarrow p_0$ almost surely then $\bar{p} \rightarrow p_0$ almost surely so $g''(\bar{p}) \rightarrow g''(p_0)$ almost surely and&#xA;$$(p_0 - \hat{p}) \rightarrow \frac{g'(p_0)}{g''(p_0)}$$&#xA;almost surely. So, in order to describe the distribution of $p_0 - \hat{p}$, i.e. the estimators central limit theorem, you need to find the distribution of $\frac{g'(p_0)}{g''(p_0)}$.  This now becomes problem dependent. However, note that you are garanteed to have $E[g'(p_0)] = 0$ otherwise you would never have been able to prove consistency in the first place.&lt;/p&gt;&#xA;" OwnerUserId="352" LastActivityDate="2010-09-09T12:02:48.057" CommentCount="3" />
  <row Id="2510" PostTypeId="1" AcceptedAnswerId="2521" CreationDate="2010-09-09T12:33:53.397" Score="2" ViewCount="245" Body="&lt;p&gt;For the more mathematically minded,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;we have $x \in \mathbb{R}^2$ and the function $h(x)$ defined as: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$h(x)=\alpha_1x_1^2+\alpha_2x_2^2+\alpha_3x_1+\alpha_4x_2+\alpha_5x_1x_2+\alpha_6$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and the vector of alpha's is &lt;strong&gt;known&lt;/strong&gt; and further guaranteed to be such that $h(x)$ is a general&#xA; elliptic paraboloid (i.e. a convex function). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, define $g(x)=(h(x))^+$  where $(z)^+$ is the positive part of $z\in\mathbb{R}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The questions follows: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Is there a way to globally approximate $g(x)$ by a polynomial ? If so what it is?&lt;/li&gt;&#xA;&lt;li&gt;Can this approach (the answer to the previous sub-question) be extended to $x\in\mathbb{R}^p$ with $p$ moderatly large ?&lt;/li&gt;&#xA;&lt;li&gt;Would the problem be any easier if we were to assume $\alpha_5=0$ ?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Following Whuber's comment: is it possible to find a polynomial approximation to $g(x)$ that would be better than $h(x)$? in the event that many approximations solution for this problem exist, what are they ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can obviously solve $argmin.\int_{\mathbb{R}^2}(g(x)-\hat{g}(x))^2 dx$. I'm wondering if there are explicit, known solutions to this problem (i.e. polynomial series for example of which i know close to nothing).&lt;/p&gt;&#xA;" OwnerUserId="603" LastEditorUserId="603" LastEditorDisplayName="" LastEditDate="2010-09-09T19:21:26.227" LastActivityDate="2010-09-09T20:15:54.960" Title="Can we approximate this function by a polynomial?" Tags="&lt;approximation&gt;" AnswerCount="2" CommentCount="5" />
  <row Id="2511" PostTypeId="2" ParentId="2504" CreationDate="2010-09-09T12:57:35.380" Score="5" ViewCount="" Body="&lt;p&gt;No, this is not possible, because a finite sample of size $n$ cannot reliably distinguish between, say, a normal population and a normal population contaminated by a $1/N$ amount of a Cauchy distribution where $N$ &gt;&gt; $n$.  (Of course the former has finite variance and the latter has infinite variance.)  Thus any fully nonparametric test will have arbitrarily low power against such alternatives.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-09-09T12:57:35.380" CommentCount="5" />
  <row Id="2513" PostTypeId="1" AcceptedAnswerId="2530" CreationDate="2010-09-09T15:04:28.853" Score="6" ViewCount="389" Body="&lt;p&gt;I have 2560 paired observations from an experiment in which participants provided two ratings for a set of objects, at two different points in time.  Half of the objects in the set had the value of an attribute A changed between the two time points, half did not.  Of the objects that were changed in each participant's set, half went from A' to A'' and half from A'' to A'. (i.e. all participants experienced both orders). My main hypothesis is that changing this attribute from A' to A'' leads on average to a higher rating, and this is indeed supported by the data.  I am also interested in determining whether the magnitude and perhaps direction of this effect depends on the A' rating.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the purposes of this question, I am considering only those instances where A was changed (1280 pairs of obs).  The following GLMM&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(A'' rating - A' rating) = participant + order + A' rating&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where A' rating is a covariate and participant and order are categorical variables, leads to the conclusion that there is a significant positive correlation between A' rating and effect of changing to A'' and that this correlation is &amp;lt;1, such that objects with a low A' rating have their rating increased by changing to A'' but that objects with a high A' rating actually get rated lower when changed to A''.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to test whether this is simply due to regression to the mean.  To this end, I have followed &lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pubmed/16475086&quot; rel=&quot;nofollow&quot;&gt;Kelly and Price&lt;/a&gt; in using Pitman's test of equality of variances for paired samples and would appreciate some feedback on whether I've done the right thing. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is what I did, following the suggestion of a colleague:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) calculated the SD of A'' ratings $(SD_1)$ and the SD of A' ratings $(SD_2)$&lt;br&gt;&#xA;2) regressed A'' rating on A' rating and recorded the correlation $r$.&lt;br&gt;&#xA;3) calculated T as $T=\frac{\sqrt{(n-2)} [(SD_1/SD_2)-(SD_2/SD_1)]}{2 \sqrt{(1-r^2)}}$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The 2-tailed p value of T (Student's t dist with 1280-2 DF) is 0.07, i.e. at alpha=0.05 there is no significant difference between the variances for the two sets of ratings and thus no effect of A' rating on rating difference beyond regression to the mean.  (We can argue about 2-t vs. 1-t p values later).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I now plan to adjust my difference scores to account for this and re-do the GLMM outlined above, as outlined by Kelly &amp;amp; Price.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you've got this far through the detail, then firstly well done, and secondly, can you tell me if 1) my colleague's suggestion was sensible and 2) if it was, have I implemented it correctly? I have a couple of concerns/apparent grey areas but I'd be interested to hear what others have to say first.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you.&lt;/p&gt;&#xA;" OwnerUserId="266" LastEditorUserId="266" LastEditorDisplayName="" LastEditDate="2010-09-10T11:50:47.273" LastActivityDate="2010-09-28T02:27:31.433" Title="Pitman's test of equality of variance and testing for regression to the mean: am I doing the right thing?" Tags="&lt;regression&gt;&lt;standard-deviation&gt;&lt;variance&gt;" AnswerCount="2" CommentCount="3" FavoriteCount="4" />
  <row Id="2516" PostTypeId="1" AcceptedAnswerId="2519" CreationDate="2010-09-09T18:21:30.200" Score="22" ViewCount="969" Body="&lt;p&gt;In a &lt;a href=&quot;http://magazine.amstat.org/blog/2010/09/01/statrevolution/&quot; rel=&quot;nofollow&quot;&gt;recent article&lt;/a&gt; of &lt;em&gt;Amstat News&lt;/em&gt;, the authors (Mark van der Laan and Sherri Rose) stated that &quot;We know that for large enough sample sizes, every study—including ones in which the null hypothesis of no effect is true — will declare a statistically significant effect.&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Well, I for one didn't know that. Is this true? Does it mean that hypothesis testing is worthless for large data sets?&lt;/p&gt;&#xA;" OwnerUserId="666" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-10-08T16:04:46.717" LastActivityDate="2010-10-08T16:04:46.717" Title="Are large data sets inappropriate for hypothesis testing?" Tags="&lt;hypothesis-testing&gt;&lt;sample-size&gt;&lt;dataset&gt;&lt;large-data&gt;" AnswerCount="9" CommentCount="2" FavoriteCount="8" />
  <row Id="2517" PostTypeId="2" ParentId="2516" CreationDate="2010-09-09T18:42:48.590" Score="1" ViewCount="" Body="&lt;p&gt;I think what they mean is that one often makes an assumption about the probability density of the null hypothesis which has a 'simple' form but does not correspond to the true probability density. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now with small data sets, you might not have enough sensitivity to see this effect but with a large enough data set you will reject the null hypothesis and conclude that there is a new effect instead of concluding that your assumption about the null hypothesis is wrong.&lt;/p&gt;&#xA;" OwnerUserId="961" LastActivityDate="2010-09-09T18:42:48.590" CommentCount="1" />
  <row Id="2518" PostTypeId="2" ParentId="2516" CreationDate="2010-09-09T18:55:04.400" Score="3" ViewCount="" Body="&lt;p&gt;In a certain sense, [all] &lt;strong&gt;many&lt;/strong&gt; null hypothesis are [always] false (the group of people living in houses with odd numbers does never &lt;em&gt;exactly&lt;/em&gt; earn the same on average as the group of people living in houses with even numbers). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the frequentist framework, the question that is asked is whether the difference in income between the two group is larger than $T_{\alpha}n^{-0.5}$ (where $T_{\alpha}$ is the $\alpha$ quantile of the distribution of the test statistic under the null). Obviously, for $n$ growing without bounds, this band becomes increasingly easy to break through.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is not a defect of statistical tests. Simply a consequence of the fact that without further information (a prior) we have that a large number of small inconsistencies with the null have to be taken as evidence against the null. No matter how trivial these inconsistencies turn out to be.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In large studies, it becomes then interesting to re-frame the issue as a bayesian test, i.e. ask oneself (for instance), what is $\hat{P}(|\bar{\mu}_1-\bar{\mu}_2|^2&gt;\eta|\eta, X)$. &lt;/p&gt;&#xA;" OwnerUserId="603" LastEditorUserId="603" LastEditorDisplayName="" LastEditDate="2010-09-09T19:55:14.900" LastActivityDate="2010-09-09T19:55:14.900" CommentCount="5" />
  <row Id="2519" PostTypeId="2" ParentId="2516" CreationDate="2010-09-09T18:59:36.507" Score="15" ViewCount="" Body="&lt;p&gt;It is not true.  If the sample size is large enough and the null hypothesis is true it will not be rejected.  Nevertheless, it's possible that all nulls are false and therefore all will be rejected if N is high enough.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is true is that trivially small effects are found with very large sample sizes.  That in no way suggests you shouldn't have such large samples sizes.  What it means is that the way you interpret your finding is dependent upon the effect size and sensitivity of the test.  If you have a very small effect size and highly sensitive test you have to recognize that the statistically significant finding may not be meaningful or useful.&lt;/p&gt;&#xA;" OwnerUserId="601" LastActivityDate="2010-09-09T18:59:36.507" CommentCount="1" />
  <row Id="2520" PostTypeId="2" ParentId="2510" CreationDate="2010-09-09T19:00:45.007" Score="1" ViewCount="" Body="&lt;p&gt;Note that $h(x)$ itself is a polynomial in $x_1$ and $x_2$ and $g(x)$ is what I would call a &lt;em&gt;truncated polynomial&lt;/em&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The region where $h(x)$ is negative, $g(x)$ will be zero. You can approximate this region where $h(x)$ is flat by a non-constant polynomial (non-constant because you also need to approximate the region where $h(x) &gt; 0$) but this will always be a 'wiggly' approximation (at least for a finite number of terms in the approximating polynomial).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's not clear to me what you want to achieve, $g(x)$ actually has a quite simple analytical form and maybe for your problem you just have to consider the two regions $h(x) &amp;lt; 0$ and $h(x) \ge 0$ separately.&lt;/p&gt;&#xA;" OwnerUserId="961" LastActivityDate="2010-09-09T19:00:45.007" CommentCount="1" />
  <row Id="2521" PostTypeId="2" ParentId="2510" CreationDate="2010-09-09T19:27:32.943" Score="5" ViewCount="" Body="&lt;p&gt;The $L^2$ distance between $g$ and a polynomial approximation will be finite if and only if the polynomial approximation behaves asymptotically like $g$, which means it behaves asymptotically like $h$, which implies it must equal $h$.  Therefore $h$ is the unique $L^2$ approximator.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(1) Yes; the global approximator to $g$ is $h$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(2) Yes; the same reasoning holds for all finite $p$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(3) No; the value of $\alpha_5$ makes no difference.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Follow-up question (i): No, you cannot do better than $h$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Follow-up question (ii) [how many approximations]: Not applicable due to the answer to (i).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can get a considerably better set of answers if you're willing to limit the domain of $g$ to a compact [measurable] subset.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-09-09T20:15:54.960" LastActivityDate="2010-09-09T20:15:54.960" CommentCount="4" />
  <row Id="2522" PostTypeId="2" ParentId="2516" CreationDate="2010-09-09T19:42:26.763" Score="3" ViewCount="" Body="&lt;p&gt;I agree with the answers that have appeared, but would like to add that perhaps the question could be redirected.  Whether to test a hypothesis or not is a research question that ought, at least in general, be independent of how much data one has.  If you really need to test a hypothesis, do so, and don't be afraid of your ability to detect small effects.  But first ask whether that's part of your research objectives.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now for some quibbles:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Some null hypotheses are absolutely true by construction.  When you're testing a pseudorandom number generator for equidistribution, for instance, and that PRG is truly equidistributed (which would be a mathematical theorem), then the null holds.  Probably most of you can think of more interesting real-world examples arising from randomization in experiments where the treatment really does have no effect.  (I would hold out the entire literature on esp as an example. ;-)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;In a situation where a &quot;simple&quot; null is tested against a &quot;compound&quot; alternative, as in classic t-tests or z-tests, it typically takes a sample size proportional to $1/\epsilon^2$ to detect an effect size of $\epsilon$.  There's a practical upper bound to this in any study, implying there's a practical lower bound on a detectable effect size.  So, as a theoretical matter der Laan and Rose are correct, but we should take care in applying their conclusion.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-09-09T19:42:26.763" />
  <row Id="2524" PostTypeId="2" ParentId="1646" CreationDate="2010-09-09T20:39:15.037" Score="4" ViewCount="" Body="&lt;p&gt;I think (1) is not a statistical question but a subject-area one. E.g., in the described example it would be up to those who study group psychology to determine appropriate language for the strength of ICCs.  This is analogous to a Pearson correlation -- what constitutes 'strong' differs depending on whether one is working in, for example, sociology or physics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(2) is to an extent also subject-area specific -- it depends on what researchers are aiming to measure and describe.  But from a statistical point of view ICC is a reasonable metric for within-team relatedness.  However I agree with Mike that when you say you'd like to &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&quot;describe the extent to which&#xA;  the measure of team effectiveness is a&#xA;  property of the team member's&#xA;  idiosyncratic belief or a property of&#xA;  a shared belief about the team&quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;then it is probably more appropriate to use variance components in their raw form than to convert them into an ICC.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To clarify, think of the ICC as calculated within a mixed model.  For a single-level mixed model with random group-level intercepts $b_i \sim N(0, \sigma^2_b)$ and within-group errors $\epsilon_{ij} \stackrel{\mathrm{iid}}{\sim} N(0, \sigma^2)$, $\sigma^2_b$ describes the amount of variation between teams and $\sigma^2$ describes variation within teams.  Then, for a single team, we get a response covariance matrix of $\sigma^2 \mathbf{I} + \sigma^2_b \mathbf{1}\mathbf{1}'$ which when converted to a correlation matrix is $\frac{\sigma^2}{\sigma^2 + \sigma^2_b} \mathbf{I} + \frac{\sigma^2_b}{\sigma^2 + \sigma^2_b} \mathbf{1}\mathbf{1}'$.  So, $\frac{\sigma^2_b}{\sigma^2 + \sigma^2_b} = \mathrm{ICC}$ describes the level of correlation between effectiveness responses within a team, but it sounds as though you may be more interested in $\sigma^2$ and $\sigma^2_b$, or perhaps $\frac{\sigma^2}{\sigma^2_b}$.&lt;/p&gt;&#xA;" OwnerUserId="1107" LastActivityDate="2010-09-09T20:39:15.037" />
  <row Id="2525" PostTypeId="2" ParentId="363" CreationDate="2010-09-09T22:52:29.493" Score="3" ViewCount="" Body="&lt;p&gt;Long ago, Jack Kiefer's little monograph &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0387964207&quot; rel=&quot;nofollow&quot;&gt;&quot;Introduction to Statistical Inference&quot;&lt;/a&gt; peeled away the mystery of a great deal of classical statistics and helped me get started with the rest of the literature.  I still refer to it and warmly recommend it to strong students in second-year stats courses.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2011-02-20T02:32:14.440" LastActivityDate="2011-02-20T02:32:14.440" CommunityOwnedDate="2010-09-09T22:52:29.493" CommentCount="2" />
  <row Id="2526" PostTypeId="2" ParentId="1520" CreationDate="2010-09-09T23:09:59.063" Score="-1" ViewCount="" Body="&lt;p&gt;You're right about the expectation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You actually also have the right answer to the probability of getting more than your original stake back, although not the right proof. Consider, instead of the raw amount of money you have, its base-2 logarithm.  This turns out to be the number of times you've doubled your money, minus the number of times you've halved it.  This is the sum $S_n$ of $n$ independent random variables, each equal to $+1$ or $-1$ with probability $1/2$. The probability that you want is the probability that this is positive. If $n$ is odd, then by symmetry it's exactly $1/2$; if $n$ is even (call it $2k$) then it's $1/2$ minus half the probability that $S_n = 0$. But $P(S_{2k} = 0) = {2k \choose k}/2^{2k}$, which approaches $0$ as $k \to \infty$.&lt;/p&gt;&#xA;" OwnerUserId="98" LastActivityDate="2010-09-09T23:09:59.063" CommentCount="1" />
  <row Id="2528" PostTypeId="2" ParentId="2516" CreationDate="2010-09-10T03:51:12.823" Score="3" ViewCount="" Body="&lt;p&gt;A (frequentist) hypothesis test, precisely, address the question of the probability of the observed data or something more extreme would be likely assuming the null hypothesis is true.  This interpretation is indifferent to sample size.  That interpretation is valid whether the sample is of size 5 or 1,000,000.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An important caveat is that the test is only relevant to sampling errors.  Any errors of measurement, sampling problems,coverage, data entry errors, etc are outside of the scope of sampling error.  As sample size increases, non-sampling errors become more influential as small departures can produce significant departures from the random sampling model.  As a result, tests of significance become less useful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is in no way an indictment of significance testing.  However, we need to be careful about our attributions.  A result may be statistically significant.  However, we need to be cautious about how we make attributions when sample size is large.  Is that difference due to sampling error (which the test assumes) or is it the result of any of a number of possible non-sampling errors that could influence the test statistic?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another consideration with large samples is the practical significance of a result.  A significant test might suggest (even if we can rule out non-sampling error) a difference that is trivial in a practical sense.  Even if that result is unlikely given the sampling model, is it significant in the context of the problem?  Given a large enough sample, a difference in a few dollars might be enough to produce a result that is statistically significant when comparing income among two groups.  Is this important in any meaningful sense?  Statistical significance is no replacement for good judgment and subject matter knowledge.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an aside, the null is neither true nor false.  It is a model.  It is an assumption.  We assume the null is true and assess our sample in terms of that assumption.  If our sample would be unlikely given this assumption, we place more trust in our alternative.  To question whether or not a null is ever true in practice is a misunderstanding of the logic of significance testing.&lt;/p&gt;&#xA;" OwnerUserId="485" LastEditorUserId="485" LastEditorDisplayName="" LastEditDate="2010-09-10T04:16:54.423" LastActivityDate="2010-09-10T04:16:54.423" />
  <row Id="2529" PostTypeId="2" ParentId="36" CreationDate="2010-09-10T07:40:27.233" Score="1" ViewCount="" Body="&lt;p&gt;The more fire engines sent to a fire, the bigger the damage.&lt;/p&gt;&#xA;" OwnerUserId="1048" LastActivityDate="2010-09-10T07:40:27.233" CommunityOwnedDate="2010-09-10T07:40:27.233" />
  <row Id="2530" PostTypeId="2" ParentId="2513" CreationDate="2010-09-10T09:52:48.877" Score="1" ViewCount="" Body="&lt;p&gt;As far as I know, the Pitman test is formulated as :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$F=\frac{SD_2}{SD_1} ~with~ SD_2 &gt; SD_1$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$T=\frac{(F-1)\sqrt{n-2}}{2\sqrt{F(1-r^2)}} $$ with $r$ the correlation between the scores in sample 1 and sample 2. This is not equivalent to the formula you use and mentioned in the paper. I'm not positive about my formula either, I got it from a course somewhere (alas no reference...)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Apart from that, it might be interesting to take a look at an alternative approach to dealing with regression to the mean. I found the &lt;a href=&quot;http://ije.oxfordjournals.org/content/34/1/215.full&quot; rel=&quot;nofollow&quot;&gt;tutorial paper of Barnett et al&lt;/a&gt; on regression to the mean very enlightening.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now let's get back a moment to the 2-sided versus 1-sided p-values. Regardless of the formula you use, the sign of T is only dependent on the order of the SD's. (In fact, how I know the pitman test, T is always positive.) Hence, the underlying distribution is -as far as I'm concerned- not the T distribution but half the T distribution, meaning you have to put the cutoff at $T_{0.975, df}$, but the related p-value is originating from one tail only. This is equivalent the standard F test for comparing variances.&lt;/p&gt;&#xA;" OwnerUserId="1124" LastActivityDate="2010-09-10T09:52:48.877" CommentCount="1" />
  <row Id="2531" PostTypeId="2" ParentId="2516" CreationDate="2010-09-10T10:03:13.960" Score="1" ViewCount="" Body="&lt;p&gt;Hypothesis testing for large data should the desired level of difference into account, rather than whether there is a difference or not. You're not interested in the H0 that the estimate is exactly 0. A general approach would be to test whether the difference between the null hypothesis and the observed value is larger than a given cut-off value. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;An simple example with the T-test:&#xA;You can make following assumptions for big sample sizes, given you have equal sample sizes and standard deviations in both groups, and $\bar{X_1} &gt; \bar{X_2}$ :&#xA;$$T=\frac{\bar{X1}-\bar{X2}-\delta}{\sqrt{\frac{S^2}{n}}}+\frac{\delta}{\sqrt{\frac{S^2}{n}}} \approx N(\frac{\delta}{\sqrt{\frac{S^2}{n}}},1)$$&#xA;hence&#xA;$$T=\frac{\bar{X1}-\bar{X2}}{\sqrt{\frac{S^2}{n}}} \approx N(\frac{\delta}{\sqrt{\frac{S^2}{n}}},1)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;as your null hypothesis $H_0:\bar{X1}-\bar{X2} = \delta $ implies:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{\bar{X1}-\bar{X2}-\delta}{\sqrt{\frac{S^2}{n}}}\approx N(0,1)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This you can easily use to test for a significant and relevant difference. In R you can make use of the noncentrality parameter of the T distributions to generalize this result for smaller sample sizes as well. You should take into account that this is a one-sided test, the alternative $H_A$ is $\bar{X1}-\bar{X2} &gt; \delta $.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mod.test &amp;lt;- function(x1,x2,dif,...){&#xA;    avg.x1 &amp;lt;- mean(x1)&#xA;    avg.x2 &amp;lt;- mean(x2)&#xA;    sd.x1 &amp;lt;- sd(x1)&#xA;    sd.x2 &amp;lt;- sd(x2)&#xA;&#xA;    sd.comb &amp;lt;- sqrt((sd.x1^2+sd.x2^2)/2)&#xA;    n &amp;lt;- length(x1)&#xA;    t.val &amp;lt;- (abs(avg.x1-avg.x2))*sqrt(n)/sd.comb&#xA;    ncp &amp;lt;- (dif*sqrt(n)/sd.comb)&#xA;    p.val &amp;lt;- pt(t.val,n-1,ncp=ncp,lower.tail=FALSE)&#xA;    return(p.val)&#xA;}&#xA;&#xA;n &amp;lt;- 5000&#xA;&#xA;test1 &amp;lt;- replicate(100,&#xA;  t.test(rnorm(n),rnorm(n,0.05))$p.value)&#xA;table(test1&amp;lt;0.05)&#xA;test2 &amp;lt;- replicate(100,&#xA;  t.test(rnorm(n),rnorm(n,0.5))$p.value)&#xA;table(test2&amp;lt;0.05)&#xA;&#xA;test3 &amp;lt;- replicate(100,&#xA;   mod.test(rnorm(n),rnorm(n,0.05),dif=0.3))&#xA;table(test3&amp;lt;0.05)&#xA;&#xA;test4 &amp;lt;- replicate(100,&#xA;   mod.test(rnorm(n),rnorm(n,0.5),dif=0.3))&#xA;table(test4&amp;lt;0.05)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Which gives :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; table(test1&amp;lt;0.05)&#xA;FALSE  TRUE &#xA;   24    76 &#xA;&#xA;&amp;gt; table(test2&amp;lt;0.05)&#xA;TRUE &#xA; 100 &#xA;&#xA;&amp;gt; table(test3&amp;lt;0.05)&#xA;FALSE &#xA;  100 &#xA;&#xA;&amp;gt; table(test4&amp;lt;0.05)&#xA;TRUE &#xA; 100 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1124" LastEditorUserId="1124" LastEditorDisplayName="" LastEditDate="2010-09-10T13:45:11.413" LastActivityDate="2010-09-10T13:45:11.413" CommentCount="2" />
  <row Id="2532" PostTypeId="2" ParentId="36" CreationDate="2010-09-10T10:48:33.617" Score="1" ViewCount="" Body="&lt;p&gt;when $x_t$; $y_t$ are stationary time series, then correlation between $y_t$ and $x_{t-1}$ implies causation of $y_t$ by $x_{t-1}$. For some reason it's not mentioned here.&lt;/p&gt;&#xA;" OwnerUserId="603" LastActivityDate="2010-09-10T10:48:33.617" CommunityOwnedDate="2010-09-10T10:48:33.617" />
  <row Id="2533" PostTypeId="2" ParentId="26" CreationDate="2010-09-10T10:48:46.980" Score="-1" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://en.lmgtfy.com/?q=standard+deviation&quot; rel=&quot;nofollow&quot;&gt;You can find the answer by clicking on this sentence.&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="442" LastActivityDate="2010-09-10T10:48:46.980" CommentCount="1" />
  <row Id="2535" PostTypeId="2" ParentId="1646" CreationDate="2010-09-10T11:28:41.017" Score="0" ViewCount="" Body="&lt;p&gt;1) With correlations, you can never really give sensible cut-offs, but the general rules of the normal correlation apply I'd say. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Regarding the appropriateness of the ICC : depending on the data, the ICC is equivalent to an F-test (see eg &lt;a href=&quot;http://www.jstor.org/stable/2533395&quot; rel=&quot;nofollow&quot;&gt;Commenges &amp;amp; Jacqmin, 1994&lt;/a&gt; and &lt;a href=&quot;http://worldcatlibraries.org/registry/gateway/?issn=0033-3123&amp;amp;aulast=Kistner&amp;amp;aufirst=Emily&amp;amp;date=2004&amp;amp;atitle=Exact%20Distributions%20of%20Intraclass%20Correlation%20and%20Cronbachs%20Alpha%20with%20Gaussian%20Data%20and%20General%20Covariance&amp;amp;title=Psychometrika&amp;amp;volume=69&amp;amp;issue=3&amp;amp;spage=459?&quot; rel=&quot;nofollow&quot;&gt;Kistner &amp;amp; Muller, 2004&lt;/a&gt;). So in essence, the mixed model framework can tell you at least as much about your hypothesis, and allows for simultaneously testing more hypotheses than the ICC. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cronbach's $\alpha$ is also directly related to the ICC, and another measure that is (was?) often reported, albeit in the context of agreement between items within a group. This approach comes from psychological questionnaires, where a cut-off of 0.7 is rather often used to determine whether the questions really group into the studied factors.&lt;/p&gt;&#xA;" OwnerUserId="1124" LastActivityDate="2010-09-10T11:28:41.017" CommentCount="1" />
  <row Id="2536" PostTypeId="2" ParentId="534" CreationDate="2010-09-10T12:33:09.340" Score="6" ViewCount="" Body="&lt;p&gt;I'll just add some additional comments about causality as viewed from an &lt;em&gt;epidemiological perspective&lt;/em&gt;. Most of these arguments are taken from &lt;a href=&quot;http://occmed.oxfordjournals.org/content/56/6/434.2.full&quot; rel=&quot;nofollow&quot;&gt;Practical Psychiatric Epidemiology&lt;/a&gt;, by Prince et al. (2003).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Causation, or &lt;em&gt;causality interpretation&lt;/em&gt;, are by far the most difficult aspects of epidemiological research. &lt;a href=&quot;http://en.wikipedia.org/wiki/Cohort_study&quot; rel=&quot;nofollow&quot;&gt;Cohort&lt;/a&gt; and &lt;a href=&quot;http://en.wikipedia.org/wiki/Cross-sectional_study&quot; rel=&quot;nofollow&quot;&gt;cross-sectional&lt;/a&gt; studies might both lead to confoundig effects for example. Quoting S. Menard (&lt;em&gt;Longitudinal Research&lt;/em&gt;, Sage University Paper 76, 1991), H.B. Asher in &lt;em&gt;Causal Modeling&lt;/em&gt; (Sage, 1976) initially proposed the following set of criteria to be fulfilled:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The phenomena or variables in question must covary, as indicated for example by differences between experimental and control groups or by nonzero correlation between the two variables.&lt;/li&gt;&#xA;&lt;li&gt;The relationship must not be attributable to any other variable or set of variables, i.e., it must not be spurious, but must persist even when other variables are controlled, as indicated for example by successful randomization in an experimental design (no difference between experimental and control groups prior to treatment) or by a nonzero partial correlation between two variables with other variable held constant.&lt;/li&gt;&#xA;&lt;li&gt;The supposed cause must precede or be simultnaeous with the supposed effect in time, as indicated by the change in the cause occuring no later than the associated change in the effect.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;While the first two criteria can easily be checked using a cross-sectional or time-ordered cross-sectional study, the latter can only be assessed with longitudinal data, except for biological or genetic characteristics for which temporal order can be assume without longitudinal data. Of course, the situation becomes more complex in case of a non-recursive causal relationship.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also like the following illustration (Chapter 13, in the aforementioned reference) which summarizes the approach promulgated by Hill (1965) which includes 9 different criteria related to causation effect, as also cited by @James. The original article was indeed entitled &quot;The environment and disease: association or causation?&quot; (&lt;a href=&quot;http://epiville.ccnmtl.columbia.edu/assets/pdfs/Hill_1965.pdf&quot; rel=&quot;nofollow&quot;&gt;PDF version&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/BNQEa.png&quot; alt=&quot;Hill1965&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, Chapter 2 of Rothman's most famous book, &lt;em&gt;Modern Epidemiology&lt;/em&gt; (1998, Lippincott Williams &amp;amp; Wilkins, 2nd Edition), offers a very complete discussion around causation and causal inference, both from a statistical and philosophical perspective.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd like to add the following references (roughly taken from an online course in epidemiology) are also very interesting:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Swaen, G and van Amelsvoort, L (2009). &lt;a href=&quot;http://www.uc.pt/en/fmuc/phdhs/Courses/Epidemiology/A_weight_of_evidence_approach_to_causal_inference.pdf&quot; rel=&quot;nofollow&quot;&gt;A weight of evidence approach to causal inference&lt;/a&gt;. &lt;em&gt;Journal of Clinical Epidemiology&lt;/em&gt;, &lt;em&gt;62&lt;/em&gt;, 270-277.&lt;/li&gt;&#xA;&lt;li&gt;Botti, C, Comba, P, Forastiere, F, and Settimi, L (1996). &lt;a href=&quot;http://www.uc.pt/en/fmuc/phdhs/Courses/Epidemiology/Causal_inference_in_environmental_epidemiology._the_role_of_implicit_values.pdf&quot; rel=&quot;nofollow&quot;&gt;Causal inference in environmental epidemiology. the role of implicit values&lt;/a&gt;. &lt;em&gt;The Science of the Total Environment&lt;/em&gt;, &lt;em&gt;184&lt;/em&gt;, 97-101.&lt;/li&gt;&#xA;&lt;li&gt;Weed, DL (2002). &lt;a href=&quot;http://www.uc.pt/en/fmuc/phdhs/Courses/Epidemiology/Environmental_epidemiology._Basics_and_proof_of_cause.effect.pdf&quot; rel=&quot;nofollow&quot;&gt;Environmental epidemiology. Basics and proof of cause effect&lt;/a&gt;. &lt;em&gt;Toxicology&lt;/em&gt;, &lt;em&gt;181-182&lt;/em&gt;, 399-403.&lt;/li&gt;&#xA;&lt;li&gt;Franco, EL, Correa, P, Santella, RM, Wu, X, Goodman, SN, and Petersen, GM (2004). &lt;a href=&quot;http://www.uc.pt/en/fmuc/phdhs/Courses/Epidemiology/Role_and_limitations_of_epidemiology_in_establishing_a_causal_association.pdf&quot; rel=&quot;nofollow&quot;&gt;Role and limitations of epidemiology in establishing a causal association&lt;/a&gt;. &lt;em&gt;Seminars in Cancer Biology&lt;/em&gt;, &lt;em&gt;14&lt;/em&gt;, 413–426.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Finally, this review offers a larger perspective on causal modeling, &lt;a href=&quot;http://www.i-journals.org/ss/viewarticle.php?id=57&quot; rel=&quot;nofollow&quot;&gt;Causal inference in statistics: An overview&lt;/a&gt; (J Pearl, SS 2009 (3)).&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-02T17:02:16.690" LastActivityDate="2010-10-02T17:02:16.690" />
  <row Id="2537" PostTypeId="1" AcceptedAnswerId="2538" CreationDate="2010-09-10T12:51:35.787" Score="2" ViewCount="208" Body="&lt;p&gt;Say we have 5 items, and people are asked which item they like. Multiple answers are possible, but also no answer is possible. The people are categorized according to factors like gender, age, and so on. One possible approach to analyze the differences between genders, age groups and so on is making use of the &lt;a href=&quot;http://en.wikipedia.org/wiki/Generalized_estimating_equation&quot; rel=&quot;nofollow&quot;&gt;Generalized Estimating Equations&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I thus construct a dataset with a binary variable indicating whether the item was liked or not, and as predictor variables I have the items, the person id, the age,... i.e. :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Liked   Item   Person ...&#xA;0       1      1&#xA;1       2      1&#xA;0       3      1&#xA;0       4      1&#xA;1       5      1&#xA;1       1      2&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then I apply a model with following form :&#xA;$$Liked = Item + Gender + Item*Gender + Age + Item*Age + ... $$&#xA;with Person as random factor (called id in some applications) and a logit link function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I like to give confidence intervals on the conditional fractions, i.e. the confidence interval of the fractions males and females that liked a particular item, corrected for age differences and the likes. I know I could use the estimated coefficients to get the results I want, but I'm a bit lost in how to do that. I can calculate the estimated odds, but I'm not sure on the standard error (SE) on those odds based on the SE of the coefficients. I'm not sure on how to deal with the random component of the variance for example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Any pointers on how to calculate that SE from the SE of the coefficients?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Any alternatives for an approach? I've been thinking about mixed models, but a colleague directed me to GEE as more appropriate for these data. Your ideas?&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Edit : for practical pointers, I'm using &lt;a href=&quot;http://cran.r-project.org/web/packages/geepack/index.html&quot; rel=&quot;nofollow&quot;&gt;geepack&lt;/a&gt; in R for this. I tried &lt;code&gt;effect()&lt;/code&gt;, but the option &lt;code&gt;se.fit=T&lt;/code&gt; is not implemented. In any case, that would give the SE for every observation, which is not what I'm interested in.&lt;/p&gt;&#xA;" OwnerUserId="1124" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-01T11:02:10.263" LastActivityDate="2010-11-01T11:02:10.263" Title="Confidence intervals on differences in choices in a GEE framework: methods and alternatives?" Tags="&lt;confidence-interval&gt;&lt;estimation&gt;&lt;mixed-model&gt;&lt;psychometrics&gt;&lt;random-effects-model&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="2538" PostTypeId="2" ParentId="2537" CreationDate="2010-09-10T14:46:40.440" Score="5" ViewCount="" Body="&lt;p&gt;Well, the &lt;a href=&quot;http://cran.r-project.org/web/packages/gee/index.html&quot; rel=&quot;nofollow&quot;&gt;gee&lt;/a&gt; package includes facilities for fitting GEE and &lt;code&gt;gee()&lt;/code&gt; return asymptotic and robust SE. I never used the &lt;a href=&quot;http://cran.r-project.org/web/packages/geepack/index.html&quot; rel=&quot;nofollow&quot;&gt;geepack&lt;/a&gt; package. From what I saw in the online example, output seems to resemble more or less that of &lt;code&gt;gee&lt;/code&gt;. To compute $100(1-\alpha)$ CIs for your main effects (e.g. gender), why not use the robust SE (in the following I will assume it is extracted from, say &lt;code&gt;summary(gee.fit)&lt;/code&gt;, and stored in a variable &lt;code&gt;rob.se&lt;/code&gt;)? I suppose that&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;exp(coef(gee.fit)[&quot;gender&quot;]+c(-1,1)*rob.se*qnorm(0.975))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;should yield 95% CIs expressed on the odds scale.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, in fact I rarely use GEE except when I am working with binary endpoints in longitudinal studies, because it's easy to pass or estimate a given working correlation matrix. In the case you summarize here, I would rather rely on an IRT model for dichotomous items (see the &lt;a href=&quot;http://cran.r-project.org/web/views/Psychometrics.html&quot; rel=&quot;nofollow&quot;&gt;psychometrics&lt;/a&gt; task view), or (it is quite the same in fact) a mixed-effects GLM such as the one that is proposed in the &lt;a href=&quot;http://cran.r-project.org/web/packages/lme4/index.html&quot; rel=&quot;nofollow&quot;&gt;lme4&lt;/a&gt; package, from Doug Bates. For study like yours, as you said, subjects will be considered as random effects, and your other covariates enter the model as fixed effects; the response is the 0/1 rating on each item (which enter the model as well). Then you will get 95% CI for fixed effects, either from the SE computed as &lt;code&gt;sqrt(diag(vcov(glmm.fit)))&lt;/code&gt; or as read in &lt;code&gt;summary(glmm.fit)&lt;/code&gt;, or using &lt;code&gt;confint()&lt;/code&gt; together with an &lt;code&gt;lmList&lt;/code&gt; object. Doug Bates gave nice illustrations in the following two paper/handout:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.jstatsoft.org/v20/i02&quot; rel=&quot;nofollow&quot;&gt;Estimating the Multilevel Rasch Model: With the lme4 Package&lt;/a&gt; (JSS, 2007 20(2))&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.stat.wisc.edu/~bates/UseR2008/WorkshopD.pdf&quot; rel=&quot;nofollow&quot;&gt;Item Response Models as GLMMs&lt;/a&gt; (from a workshop at UseR 2008)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;There is also a discussion about profiling &lt;code&gt;lmer&lt;/code&gt; fits (based on &lt;em&gt;profile deviance&lt;/em&gt;) to investigate variability in fixed effects, but I didn't investigate that point. I think it is still in section 1.5 of Doug's &lt;a href=&quot;http://lme4.r-forge.r-project.org/book/&quot; rel=&quot;nofollow&quot;&gt;draft on mixed models&lt;/a&gt;. There are a lot of discussion about computing SE and CI for GLMM as implemented in the &lt;code&gt;lme4&lt;/code&gt; package (whose interface differs from the previous &lt;code&gt;nlme&lt;/code&gt; package), so that you will easily find other interesting threads after googling about that. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's not clear to me why GEE would have to be preferred in this particular case. Maybe, look at the R translation of Agresti's book by Laura Thompson, &lt;a href=&quot;https://home.comcast.net/~lthompson221/Splusdiscrete2.pdf&quot; rel=&quot;nofollow&quot;&gt;R (and S-PLUS) Manual to Accompany Agresti's Categorical Data&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;hr /&gt;&#xA;&lt;strong&gt;Update:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I just realized that the above solution would only work if you're interested in getting a confidence interval for the gender effect alone. If it is the interaction item*gender that is of concern, you have to model it explicitly in the GLMM (my second reference on Bates's has an example on how to do it with &lt;code&gt;lmer&lt;/code&gt;). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another solution is to use an explanatory IRT model, where you explicitly acknowledge the potential effect of person covariates, like gender or age, and consider fitting them within a Rasch model, for example. This is called a Latent Regression Rasch Model, and is fully described in de Boeck and Wilson's book, &lt;em&gt;Explanatory item response models: a generalized linear and nonlinear approach&lt;/em&gt; (Springer, 2004), which you can read online on &lt;a href=&quot;http://books.google.fr/books?id=pDeLy5L14mAC&amp;amp;pg=PA66&amp;amp;lpg=PA66&amp;amp;dq=latent+regression+Rasch+model&amp;amp;source=bl&amp;amp;ots=Ivct9snqQO&amp;amp;sig=okSBDj9TAe5gZHIcMDHOQxvAbZU&amp;amp;hl=fr&amp;amp;ei=q0yKTPlCgYDgBq-CrdoK&amp;amp;sa=X&amp;amp;oi=book_result&amp;amp;ct=result&amp;amp;resnum=7&amp;amp;ved=0CEMQ6AEwBg#v=onepage&amp;amp;q=latent%20regression%20Rasch%20model&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;Google books&lt;/a&gt; (section 2.4). There are some facilities to fit this kind of model in Stata (see &lt;a href=&quot;http://www.gllamm.org/aggression.html&quot; rel=&quot;nofollow&quot;&gt;there&lt;/a&gt;). In R, we can mimic such model with a mixed-effects approach; a toy example would look something like&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;lmer(response ~ 0 + Age + Sex + item + (Sex|id), data=df, binomial)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;if I remember correctly. I'm not sure whether the &lt;a href=&quot;http://cran.r-project.org/web/packages/eRm/index.html&quot; rel=&quot;nofollow&quot;&gt;eRm&lt;/a&gt; allows to easily incorporate person covariates (because we need to construct a specific design matrix), but it may be worth checking out since it provides 95% CIs too.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-10T15:37:00.850" LastActivityDate="2010-09-10T15:37:00.850" CommentCount="5" />
  <row Id="2539" PostTypeId="2" ParentId="2483" CreationDate="2010-09-10T16:30:12.310" Score="2" ViewCount="" Body="&lt;p&gt;I am going to give a simple answer in response to your edit. If my answer does not meet your real needs please include it in the comments and I will change/complexify it a bit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$S$ be the severity of the event of interest with higher numbers representing more severity&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$W$ be the warning generated by the system.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The range of values that these variables can take is assumed to be from 1 to $n$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Either via an experiment or via an observational study you have a set of observations about actual events and the warnings generated by the system. Thus, you can calculate the following empirical probabilities:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(W=w | S=s)$ which is the probability that the system generates warning $w$ when the event is $s$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From a decision making perspective, you really want to know:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(S=s | W=w)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words, you want to know the probability of different events occurring in the near future given that the system has given you a specific warning. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can compute the required probabilities using &lt;a href=&quot;http://en.wikipedia.org/wiki/Bayes_theorem&quot; rel=&quot;nofollow&quot;&gt;Bayes theorem&lt;/a&gt;. Briefly, the calculation is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(S=s | W=w) = \frac{P(W=w | S=s) P(S=s)}{\sum_s(P(W=w |S=s)P(S=s))}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From your study you know:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(W=w | S=s)$ = Proportion of time the system generated warning $w$ when the event was $s$,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to compute the required probability, $P(S=s | W=w)$ you need to have some sense of $P(S=s)$ which you can estimate or guess based on previous experience.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can then assess the accuracy of the device by looking at the following probabilities: $P(S=s|W=s)$.&lt;/p&gt;&#xA;" LastActivityDate="2010-09-10T18:25:12.273" CommentCount="10" />
  <row Id="2540" PostTypeId="2" ParentId="1520" CreationDate="2010-09-10T16:32:41.367" Score="1" ViewCount="" Body="&lt;p&gt;Let $S_k$ be the wealth after $k$ plays of this game, where we assume $S_0 = 1.$ The temptation here is to take $X_k = \log{S_k}$, and study $X_k$ as a symmetric random walk, with innovations of size $\pm \log{2}$. This, as it turns out, will be fine for the second question, but not the first. A bit of work will show that, asymptotically we have $X_k \sim \mathcal{N}(0,k(\log{2})^2)$. From this you &lt;strong&gt;cannot&lt;/strong&gt; conclude that $S_k$ is asymptotically log normally distributed with $\mu = 0, \sigma = \log{2}\sqrt{k}.$ The log operation does not commute with the limit.  If it did, you would get the expected value of $S_k$ as $\exp{(k \log{2}\log{2}/2)}$, which is nearly correct, but not quite.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, this method is just fine for finding quantiles of $S_k$, and other questions of probability, like question (2). We have $S_k \ge (\frac{5}{4})^k \Leftrightarrow X_k \ge k \log{(5/4)} \Leftrightarrow X_k / \sqrt{k}\log{2} \ge \sqrt{k}\log{(5/4})/\log{2}.$ The quantity on the lefthand side of the last inequality is, asymptotically, a standard normal, and so the probability that $S_k$ exceeds its mean approaches $1 - \Phi{(\sqrt{k}\log{(5/4)}/\log{2})},$ where $\Phi$ is the CDF of the standard normal. This approaches zero fairly quickly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Matlab code to check this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;top_k = 512;&#xA;nsamps = 8192;&#xA;innovs = log(2) * cumsum(sign(randn(top_k,nsamps)),1);&#xA;s_k = exp(innovs);&#xA;k_vals = (1:top_k)';&#xA;mean_v = (5/4) .^ k_vals;&#xA;exceed = bsxfun(@ge,s_k,mean_v);&#xA;prob_g = mean(double(exceed),2);&#xA;&#xA;%theoretical value&#xA;%(can you believe matlab doesn't come with normal cdf function!?)&#xA;nrmcdf = @(x)((1 + erf(x / sqrt(2)))/2);&#xA;p_thry = 1 - nrmcdf(sqrt(k_vals) * log(5/4) / log(2));&#xA;&#xA;loglog(k_vals,prob_g,'b-',k_vals,p_thry,'r-');&#xA;legend('empirical probability','theoretical probability');&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;the graph produced:&#xA;&lt;img src=&quot;http://i.imgur.com/r2U31.jpg&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="795" LastEditorUserId="795" LastEditorDisplayName="" LastEditDate="2010-09-11T21:21:57.313" LastActivityDate="2010-09-11T21:21:57.313" />
  <row Id="2541" PostTypeId="1" AcceptedAnswerId="2542" CreationDate="2010-09-10T17:07:39.130" Score="5" ViewCount="609" Body="&lt;p&gt;I have read/heard many times that the sample size of at least 30 units is considered as &quot;large sample&quot; (normality assumptions of means usually approximately holds due to the CLT, ...).  Therefore, in my experiments, I usually generate samples of 30 units. Can you please give me some reference which should be cited when using sample size 30?&lt;/p&gt;&#xA;" OwnerUserId="1215" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-10T18:21:56.803" LastActivityDate="2010-09-10T19:48:16.560" Title="Is there a reference that suggest using 30 as a large enough sample size?" Tags="&lt;sample-size&gt;&lt;rule-of-thumb&gt;" AnswerCount="4" CommentCount="1" FavoriteCount="4" />
  <row Id="2542" PostTypeId="2" ParentId="2541" CreationDate="2010-09-10T17:44:58.943" Score="6" ViewCount="" Body="&lt;p&gt;The choice of &lt;em&gt;n&lt;/em&gt; = 30 for a boundary between small and large samples is a rule of thumb, only. There is a large number of books that quote (around) this value, for example, Hogg and Tanis' &lt;em&gt;Probability and Statistical Inference&lt;/em&gt; (7e) says &quot;greater than 25 or 30&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That said, the story told to me was that the only reason 30 was regarded as a good boundary was because it made for pretty Student's &lt;em&gt;t&lt;/em&gt; tables in the back of textbooks to fit nicely on one page.  That, and the critical values (between Student's &lt;em&gt;t&lt;/em&gt; and Normal) are only off by approximately up to 0.25, anyway, from df = 30 to df = infinity.  For hand computation the difference didn't really matter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Nowadays it is easy to compute critical values for all sorts of things to 15 decimal places.  On top of that we have resampling and permutation methods for which we aren't even restricted to parametric population distributions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In practice I never rely on &lt;em&gt;n&lt;/em&gt; = 30.  Plot the data.  Superimpose a normal distribution, if you like.  Visually assess whether a normal approximation is appropriate (and ask whether an approximation is even really needed).  If generating samples for research and an approximation is obligatory, generate &lt;em&gt;enough&lt;/em&gt; of a sample size to make the approximation as close as desired (or as close as computationally feasible).&lt;/p&gt;&#xA;" OwnerUserId="1108" LastActivityDate="2010-09-10T17:44:58.943" CommentCount="2" />
  <row Id="2543" PostTypeId="2" ParentId="146" CreationDate="2010-09-10T18:14:48.727" Score="3" ViewCount="" Body="&lt;p&gt;If your data is high dimensional and noisy, and you don't have a large number of sample, you run into the danger of overfitting. In such cases, it does make sense to use PCA (which can capture a dominant part of data variance; orthogonality isn't an issue) or factor analysis (which can find the true explanatory variables underlying the data) to reduce data dimensionality and then train a regression model with them. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For factor analysis based approaches, see this paper &lt;a href=&quot;http://ftp.isds.duke.edu/WorkingPapers/02-12.pdf&quot; rel=&quot;nofollow&quot;&gt;Bayesian Factor Regression Model&lt;/a&gt;, and a &lt;a href=&quot;http://www.umiacs.umd.edu/~hal/docs/daume08ihfrm.pdf&quot; rel=&quot;nofollow&quot;&gt;nonparametric Bayesian version of this model&lt;/a&gt; that does not assume that you &lt;em&gt;a priori&lt;/em&gt;  know the &quot;true&quot; number of relevant factors (or principal components in case of PCA).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd add that in many cases, supervised dimensionality reduction (e.g., &lt;a href=&quot;http://en.wikipedia.org/wiki/Linear_discriminant_analysis&quot; rel=&quot;nofollow&quot;&gt;Fisher Discriminant Analysis&lt;/a&gt;) can give improvements over simple PCA or FA based approaches, because you can make use of the label information while doing dimensionality reduction.&lt;/p&gt;&#xA;" OwnerUserId="881" LastActivityDate="2010-09-10T18:14:48.727" />
  <row Id="2544" PostTypeId="2" ParentId="2541" CreationDate="2010-09-10T18:41:42.803" Score="3" ViewCount="" Body="&lt;p&gt;IMO, it all depends on what you want to use your sample for.  Two &quot;silly&quot; examples to illustrate what I mean: If you need to estimate a mean, 30 observations is more than enough.  If you need to estimate a linear regression with 100 predictors, 30 observations will not be close to enough.&lt;/p&gt;&#xA;" OwnerUserId="1204" LastActivityDate="2010-09-10T18:41:42.803" />
  <row Id="2545" PostTypeId="2" ParentId="2541" CreationDate="2010-09-10T19:05:05.227" Score="3" ViewCount="" Body="&lt;p&gt;Mostly arbitrary rule of thumb. This statement depends on a number of factor to be true. For example on the distribution of the data. If the data comes from a Cauchy for example, even 30^3 observations are not enough to estimate the mean (in that case even an &lt;em&gt;infinite&lt;/em&gt; number of observations would not be enough to cause $\bar{\mu}^{(n)}$ to converge). This number (30) is also false if the values you draw are not independent from one another (again, you may have that there are no convergence at all, regardless of sample size).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More generally, the CLT needs essentially two pilars to hold:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;That the random variables are $&lt;a href=&quot;http://en.wikipedia.org/wiki/Independent_and_identically_distributed_random_variables&quot; rel=&quot;nofollow&quot;&gt;iid&lt;/a&gt;$: that you can re-order your observations without loosing any information.&lt;/li&gt;&#xA;&lt;li&gt;That the r.v. come from a distribution with finite second moments: meaning that the classical estimators of mean and s.d. tend to converge as sample size increases.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;(Both these condition can be somewhat weakened, but the differences are largely of theoretical nature)&lt;/p&gt;&#xA;" OwnerUserId="603" LastEditorUserId="603" LastEditorDisplayName="" LastEditDate="2010-09-10T19:48:16.560" LastActivityDate="2010-09-10T19:48:16.560" CommentCount="2" />
  <row Id="2546" PostTypeId="2" ParentId="2541" CreationDate="2010-09-10T19:42:16.590" Score="8" ViewCount="" Body="&lt;p&gt;Actually, the &quot;magic number&quot; 30 is a fallacy. See &lt;a href=&quot;http://www.uvm.edu/~bbeckage/Teaching/DataAnalysis/AssignedPapers/Cohen_1990.pdf&quot; rel=&quot;nofollow&quot;&gt;Jacob's Cohen's delightful paper&lt;/a&gt;. This myth is his first example of how &quot;some things you learn aren't so&quot;.&lt;/p&gt;&#xA;" OwnerUserId="666" LastActivityDate="2010-09-10T19:42:16.590" CommentCount="1" />
  <row Id="2547" PostTypeId="1" AcceptedAnswerId="2550" CreationDate="2010-09-10T20:26:56.997" Score="11" ViewCount="1162" Body="&lt;ul&gt;&#xA;&lt;li&gt;If you look at &lt;a href=&quot;http://www.wolframalpha.com/input/?i=world+population&quot; rel=&quot;nofollow&quot;&gt;Wolfram Alpha&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/x0ILQ.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Or this Wikipedia page &lt;a href=&quot;http://en.wikipedia.org/wiki/List_of_countries_by_median_age&quot; rel=&quot;nofollow&quot;&gt;List of countries by median age&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/QBtjT.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Clearly &lt;a href=&quot;http://en.wikipedia.org/wiki/Median&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;median&lt;/strong&gt;&lt;/a&gt; seems to be the statistic of choice when it comes to ages.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not able to explain to myself why &lt;a href=&quot;http://en.wikipedia.org/wiki/Arithmetic_mean&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;arithmetic mean&lt;/strong&gt;&lt;/a&gt; would be a worse statistic. Why is it so?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;sub&gt;Originally posted &lt;a href=&quot;http://math.stackexchange.com/questions/4380/why-is-median-age-a-better-statistic-than-mean-age&quot;&gt;here&lt;/a&gt; because I did not know this site existed.&lt;/sub&gt;&lt;/p&gt;&#xA;" OwnerUserId="285" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-15T22:40:02.183" LastActivityDate="2011-03-27T05:39:30.573" Title="Why is median age a better statistic than mean age?" Tags="&lt;mean&gt;&lt;median&gt;" AnswerCount="8" CommentCount="2" FavoriteCount="3" />
  <row Id="2548" PostTypeId="2" ParentId="2547" CreationDate="2010-09-10T20:48:49.707" Score="7" ViewCount="" Body="&lt;p&gt;John gave you a good answer on the sister site.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One aspect he didn't mention explicitly is robustness: median as a measure of &lt;em&gt;central location&lt;/em&gt; does better than the mean as it has a higher breakdown point (of 50%) whereas the mean has a very low one of 0 (see wikipedia for details). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Intuitively, it means that individual bad observations do not skew the median whereas they do for the mean.  &lt;/p&gt;&#xA;" OwnerUserId="334" LastActivityDate="2010-09-10T20:48:49.707" CommentCount="3" />
  <row Id="2549" PostTypeId="2" ParentId="2547" CreationDate="2010-09-10T21:33:58.017" Score="2" ViewCount="" Body="&lt;p&gt;John's answer on math.stackexchange can be viewed as the following: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;When you have a skewed distribution the median may be a better summary statistic than the mean.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Note that when he says that there are more infants than adults he essentially is suggesting that the age distribution is a skewed distribution.&lt;/p&gt;&#xA;" LastActivityDate="2010-09-10T21:33:58.017" CommentCount="3" />
  <row Id="2550" PostTypeId="2" ParentId="2547" CreationDate="2010-09-10T22:17:24.443" Score="12" ViewCount="" Body="&lt;p&gt;Statistics does not provide a good answer to this question, IMO.  A mean is ok to use, too, and is relevant in mortality studies for example.  But ages are not as easy to measure as you might think: older people, illiterate people, and people in some third-world countries tend to round their ages to a multiple of 5 or 10, for instance.  The median is more resistant to such errors than the mean.  Moreover, median ages are typically 20 - 40, but people can live to 100 and more (an increasing and noticeable proportion of the population of modern countries now lives beyond 100).  People of such age have 1.5 to 4 times the influence on the mean than they do on the median compared to very young people.  Thus, the median is a bit more up-to-date statistic concerning a country's age distribution and is a little more independent of mortality rates and life expectancy than the mean is.  Finally, the median gives us a slightly better picture of what the age distribution itself looks like: when you see a median of 35, for example, you know that half the population is older than 35 and you can infer some things about birth rates, ages of parents, and so on; but if the &lt;em&gt;mean&lt;/em&gt; is 35, you can't say as much, because that 35 could be influenced by a large population bulge at age 70, for example, or perhaps a population gap in some age range due to an old war or epidemic.  Thus, for &lt;em&gt;demographic,&lt;/em&gt; not &lt;em&gt;statistical,&lt;/em&gt; reasons, a median appears more worthy of the role of an omnibus value for summarizing the ages of relatively large populations of people.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-09-13T20:26:11.393" LastActivityDate="2010-09-13T20:26:11.393" CommentCount="2" />
  <row Id="2551" PostTypeId="2" ParentId="726" CreationDate="2010-09-10T23:05:50.640" Score="2" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;There is no free hunch.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;-- Robert Abelson&lt;/p&gt;&#xA;" OwnerUserId="666" LastActivityDate="2010-09-10T23:05:50.640" CommunityOwnedDate="2010-09-10T23:05:50.640" />
  <row Id="2552" PostTypeId="2" ParentId="1908" CreationDate="2010-09-10T23:50:38.213" Score="0" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://www.ecmlpkdd2010.org/&quot; rel=&quot;nofollow&quot;&gt;European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD)&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To see the type of papers presented at the conference see the videos of the last confenece on &lt;a href=&quot;http://videolectures.net/ecmlpkdd09_bled/&quot; rel=&quot;nofollow&quot;&gt;videolectures.net&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1215" LastActivityDate="2010-09-10T23:50:38.213" CommunityOwnedDate="2010-09-10T23:50:38.213" />
  <row Id="2553" PostTypeId="2" ParentId="2547" CreationDate="2010-09-11T00:30:30.430" Score="2" ViewCount="" Body="&lt;p&gt;Why is an axe better than a hatchet?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That's similar to your question.  They just mean and do different things.  If one is talking about medians then the story they are trying to convey, the model they are trying to apply to the data, is different than one with means.&lt;/p&gt;&#xA;" OwnerUserId="601" LastActivityDate="2010-09-11T00:30:30.430" />
  <row Id="2555" PostTypeId="1" AcceptedAnswerId="2558" CreationDate="2010-09-11T01:13:44.640" Score="2" ViewCount="136" Body="&lt;p&gt;In &lt;em&gt;&quot;On Gaussian-like Densities of Order Greater than Two&quot;&lt;/em&gt; (Willett, P. Thomas, J. B., 1987), section II, the author state:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\mathcal{N}(x,y,\rho)=\phi(x)\phi(y)\sum_{n=0}^{\infty}\rho^nH_n(x)H_n(y)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;$\phi(.)$ is the unit normal density,&lt;/li&gt;&#xA;&lt;li&gt;$H_n(.)$ is the $n^{th}$ Hermite polynomial,&lt;/li&gt;&#xA;&lt;li&gt;$\mathcal{N}(x,y,\rho)$ is the bivariate Gaussian density with correlation $\rho$.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I tried to repeat this (approximately, that is up to the 10th Hermite polynomial) in R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(PolynomF)&#xA;x&amp;lt;-polynom()&#xA;H&amp;lt;-polylist(1,x);for(n in 2:10) H[[n+1]]&amp;lt;-x*H[[n]]-(n-1)*H[[n-1]]&#xA;Hp&amp;lt;-as.function(H)&#xA;#The Hermite polynomial of order 10&#xA;rho&amp;lt;-0.7&#xA;R&amp;lt;-c();for(n in 0:10) R[[n+1]]&amp;lt;-rho^n&#xA;&#xA;HPA&amp;lt;-function(z) prod(dnorm(z))*sum(Hp(z[1])*Hp(z[2])*R)&#xA;&#xA;z&amp;lt;-runif(2)&#xA;HPA(z)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Which is completely off the mark (i.e. for $z=(0.65,0.63)$ i get 1194). What am i doing wrong ?&lt;/p&gt;&#xA;" OwnerUserId="603" LastEditorUserId="603" LastEditorDisplayName="" LastEditDate="2010-09-11T01:39:26.823" LastActivityDate="2010-09-11T04:44:51.973" Title="Issues using Hermite approximation to bivariate Gaussian in R" Tags="&lt;approximation&gt;" AnswerCount="1" />
  <row Id="2556" PostTypeId="2" ParentId="2499" CreationDate="2010-09-11T01:37:36.063" Score="1" ViewCount="" Body="&lt;p&gt;Following up Thylacoleo's example using the circle to explain the kernel trick (i don't have enough  reputation to add a comment directly to his answer)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Was there a simple typo in the equation for the separating hyperplane? and it should be z1 + z2 = r^2, instead of z1 + z2 = 1? Or do I misunderstand? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I agree its a nice simple example to illustrate the concept. Thanks. Though the definition of z3 still seems a bit of a mystery, but apparently it doesn't matter for the example centered at the origin.&lt;/p&gt;&#xA;" OwnerUserId="1246" LastActivityDate="2010-09-11T01:37:36.063" CommentCount="1" />
  <row Id="2558" PostTypeId="2" ParentId="2555" CreationDate="2010-09-11T04:44:51.973" Score="5" ViewCount="" Body="&lt;p&gt;It would appear that you neglected to normalize your Hermite polynomials.  Try this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(PolynomF)&#xA;x &amp;lt;- polynom()&#xA;H &amp;lt;- polylist(1,x); for(n in 2:10) H[[n+1]] &amp;lt;- x*H[[n]] - (n-1)*H[[n-1]]&#xA;for(n in 1:11) H[[n]] &amp;lt;- H[[n]]*exp(-lgamma(n)/2)&#xA;Hp &amp;lt;- as.function(H)&#xA;#The (normalized) Hermite polynomial of order 10&#xA;rho &amp;lt;- 0.7&#xA;R&amp;lt;-c(); for(n in 0:10) R[[n+1]] &amp;lt;- rho^n&#xA;&#xA;HPA &amp;lt;- function(z) prod(dnorm(z))*sum(Hp(z[1])*Hp(z[2])*R)&#xA;&#xA;set.seed(1)&#xA;z &amp;lt;- runif(2)&#xA;HPA(z)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now have a look at what it should be:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(mvtnorm)&#xA;sigma &amp;lt;- matrix(c(1, rho, rho, 1), ncol = 2)&#xA;dmvnorm(z, sigma = sigma)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which is only off in the thousandths place on my machine. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, you may be interested in the &lt;em&gt;orthopolynom&lt;/em&gt; package for another way to generate normalized Hermite polynomials.&lt;/p&gt;&#xA;" OwnerUserId="1108" LastActivityDate="2010-09-11T04:44:51.973" CommentCount="7" />
  <row Id="2559" PostTypeId="1" AcceptedAnswerId="2570" CreationDate="2010-09-11T07:52:28.570" Score="5" ViewCount="178" Body="&lt;p&gt;The following paper describes an implementation of R in parallel on a graphics processing unit (GPU).&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Buckner et al., The gputools package enables GPU computing in R, BIOINFORMATICS, Vol. 26 no. 1 2010, pages 134–135&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In the experimental section, on a 4-core computer, they compare the performance of the program running on the GPU with the performance without the GPU.  The write the following:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;We chose to use a single thread of the R environment in our test, since this is the way that most users interact with R.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So, the authors find the baseline by running their experiments using a single core (running in serial).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But, the experimental conditions for the GPU side is unclear (to me).  When using a GPU, for efficiency we should simultaneously make use of the CPUs.  If the authors used the remaining CPUs in the computer (which would be sensible to do in an optimised algorithm), then the speedup would be based on additional CPUs as well as GPUs over the baseline (and thus be artificially inflated by a factor slightly less than 4).&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;How should this experiment be interpreted?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In particular, I would like to know if my above interpretation is the correct one, and if so, what does this experiment actually tell us.&lt;/p&gt;&#xA;" OwnerUserId="386" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-09-11T08:45:18.317" LastActivityDate="2010-09-16T12:27:58.123" Title="Gputools for R: how to interpret the experimental procedure?" Tags="&lt;r&gt;&lt;gpu&gt;&lt;parallel-computing&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="2560" PostTypeId="2" ParentId="2559" CreationDate="2010-09-11T09:10:29.047" Score="3" ViewCount="" Body="&lt;p&gt;There is fundamental difference between parallel computing in CPUs and GPUs. Essentially, the &lt;a href=&quot;http://en.wikipedia.org/wiki/Central_processing_unit&quot; rel=&quot;nofollow&quot;&gt;CPU&lt;/a&gt; has been designed to do clever things on behalf of the programmer. For example, &lt;a href=&quot;http://en.wikipedia.org/wiki/Central_processing_unit#Instruction_level_parallelism&quot; rel=&quot;nofollow&quot;&gt;Instruction level parallelism&lt;/a&gt;. The &lt;a href=&quot;http://en.wikipedia.org/wiki/Graphics_processing_unit&quot; rel=&quot;nofollow&quot;&gt;GPU&lt;/a&gt; on the other hand, stips away this useful stuff and instead contains many more cores. It's a trade off between the processor helping you out and giving you more cores. Therefore, to use the GPU effectively, you need to submit as many threads (as memory allows as possible). The reason for this is because the GPU doesn't do any clever scheduling. So when it requests data for one thread, you want to have another one in the thread queue waiting to take over.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Example&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose you have a &lt;code&gt;for&lt;/code&gt; loop that you want to make parallel:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#f(i) does not depend on f(j)&#xA;#for any j != i&#xA;for(i in 1:100000)&#xA;    w[i] = f(i)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can submit N=1000000 threads (spread over the number cores) to the GPU. Now you may think that you could let n threads be done on the multi-core CPU, but:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;There's quite a lot of extra programming baggage for at most little gain. GPU programming is hard (at least I think so), so combing it with multi-core CPUs is something you want to avoid.&lt;/li&gt;&#xA;&lt;li&gt;The f(i) that you submit to the GPU tends to be a very simple function, say multiplying two elements of a matrix together. &lt;/li&gt;&#xA;&lt;li&gt;You will get a time penalty if you use the GPU and CPU together since they both have to ask each other if they are finished.&lt;/li&gt;&#xA;&lt;li&gt;By reducing the number of threads used on the GPU, you could easily be reducing efficiency, i.e. it takes the same amount of time to do N-n operations as it does for N operations!&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Of course there are situations when you may want to use both the GPU and CPU, but typically you don't use them for the same operation. &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Unfortunately, I don't have access to the paper at the moment. It's on my (long) list of things to read. So the above is more a general discussion on CPU and GPUs. I'll try and read it in the next day or two.&lt;/p&gt;&#xA;" OwnerUserId="8" LastActivityDate="2010-09-11T09:10:29.047" CommentCount="1" />
  <row Id="2561" PostTypeId="2" ParentId="726" CreationDate="2010-09-11T10:26:16.900" Score="5" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;At their best, graphics are instruments for reasoning.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Edward Tufte,  &lt;a href=&quot;http://www.edwardtufte.com&quot; rel=&quot;nofollow&quot;&gt;www.edwardtufte.com&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-11T21:12:24.913" LastActivityDate="2010-09-11T21:12:24.913" CommunityOwnedDate="2010-09-11T10:26:16.900" CommentCount="2" />
  <row Id="2562" PostTypeId="2" ParentId="726" CreationDate="2010-09-11T10:29:14.257" Score="4" ViewCount="" Body="&lt;p&gt;May I add this one, because I like Jan's contributions to psychometrics and statistics...&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Causal interpretation of the results&#xA;  of regression analysis of&#xA;  observational data is a risky&#xA;  business. The responsibility rests&#xA;  entirely on the shoulders of the&#xA;  researcher, because the shoulders of&#xA;  the statistical technique cannot carry&#xA;  such strong inferences.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Jan de Leeuw, &lt;a href=&quot;http://gifi.stat.ucla.edu/&quot; rel=&quot;nofollow&quot;&gt;homepage&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-09-11T10:29:14.257" CommunityOwnedDate="2010-09-11T10:29:14.257" />
  <row Id="2563" PostTypeId="1" CreationDate="2010-09-11T13:15:03.890" Score="3" ViewCount="134" Body="&lt;p&gt;What are some of the best ranking algorithms with inputs as up and down votes?&lt;/p&gt;&#xA;" OwnerUserId="1248" LastActivityDate="2010-09-11T18:51:18.540" Title="What are some of the best ranking algorithms with inputs as up and down votes?" Tags="&lt;algorithms&gt;" AnswerCount="1" CommentCount="5" />
  <row Id="2564" PostTypeId="2" ParentId="726" CreationDate="2010-09-11T13:50:33.270" Score="2" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;We are drowning in information and starving for knowledge.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Rutherford D. Roger&lt;/p&gt;&#xA;" OwnerUserId="961" LastActivityDate="2010-09-11T13:50:33.270" CommunityOwnedDate="2010-09-11T13:50:33.270" />
  <row Id="2565" PostTypeId="2" ParentId="2547" CreationDate="2010-09-11T14:20:43.827" Score="3" ViewCount="" Body="&lt;p&gt;Here's my answer first posted on math.stackexchange:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Median is what many people actually have in mind when they say &quot;mean.&quot; It's easier to interpret the median: half the population is above this age and half are below. Mean is a little more subtle.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;People look for symmetry and sometimes impose symmetry when it isn't there. The age distribution in a population is far from symmetric, so the mean could be misleading. Age distributions are something like a pyramid. Lots of children, not many elderly. (Or at least that's how it is in a sort of steady state. In the US, the post-WWII baby boom generation has distorted this distribution as they age.  Some people have called this &quot;squaring the pyramid&quot; because the boomers have made the top of the pyramid wider than it has been in the past.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With an asymmetrical distribution, it may be better to report the median because it is a symmetrical statistic. The median is symmetrical even if the sampling distribution isn't.&lt;/p&gt;&#xA;" OwnerUserId="319" LastActivityDate="2010-09-11T14:20:43.827" />
  <row Id="2566" PostTypeId="2" ParentId="726" CreationDate="2010-09-11T15:18:26.880" Score="6" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;…the statistician knows…that in nature&#xA;  there never was a normal distribution,&#xA;  there never was a straight line, yet&#xA;  with normal and linear assumptions,&#xA;  known to be false, he can often derive&#xA;  results which match, to a useful&#xA;  approximation, those found in the real&#xA;  world.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;George Box (JASA, 1976, Vol. 71, 791-799)&lt;/p&gt;&#xA;" OwnerUserId="1107" LastActivityDate="2010-09-11T15:18:26.880" CommunityOwnedDate="2010-09-11T15:18:26.880" />
  <row Id="2567" PostTypeId="2" ParentId="726" CreationDate="2010-09-11T15:27:50.037" Score="1" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;The business of the statistician is to&#xA;  catalyze the scientific learning&#xA;  process.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;George Box&lt;/p&gt;&#xA;" OwnerUserId="1107" LastActivityDate="2010-09-11T15:27:50.037" CommunityOwnedDate="2010-09-11T15:27:50.037" />
  <row Id="2569" PostTypeId="2" ParentId="726" CreationDate="2010-09-11T15:37:55.277" Score="2" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;We statisticians, as a police of&#xA;  science (a label some dislike but I am&#xA;  proud of...), have&#xA;  the fundamental duty of helping others&#xA;  to engage in statistical thinking as a&#xA;  necessary step of scientific inquiry&#xA;  and evidence-based policy formulation.&#xA;  In order to truly fulfill this task,&#xA;  we must constantly firm up and deepen&#xA;  our own foundation, and resist the&#xA;  temptation of competing for “methods&#xA;  and results” without pondering deeply&#xA;  whether we are helping others or&#xA;  actually harming them by effectively&#xA;  encouraging more false discoveries or&#xA;  misguided policies. Otherwise, we&#xA;  indeed can lose our identity, no&#xA;  matter how much we are desired or&#xA;  feared now.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Xiao-Li Meng&lt;/p&gt;&#xA;" OwnerUserId="1107" LastActivityDate="2010-09-11T15:37:55.277" CommunityOwnedDate="2010-09-11T15:37:55.277" CommentCount="1" />
  <row Id="2570" PostTypeId="2" ParentId="2559" CreationDate="2010-09-11T16:05:23.110" Score="7" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;But, the experimental conditions for the GPU side is unclear (to me). When using a GPU, for efficiency we should simultaneously make use of the CPUs.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;That is not generally true, and in particular is not true for the &lt;a href=&quot;http://cran.r-project.org/package=gputools&quot; rel=&quot;nofollow&quot;&gt;gputools&lt;/a&gt; R package which offers an 'everything to the GPU' mode with new functions &lt;code&gt;gpuMatMult()&lt;/code&gt;, &lt;code&gt;gpuQr()&lt;/code&gt;, &lt;code&gt;gpuCor()&lt;/code&gt; etc.  In other words, it offers you new functions that shift the computations completely to the GPU.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But your intuition is good. There should be a mixed mode with hybrid operations between the GPU and the CPU -- and the &lt;a href=&quot;http://icl.cs.utk.edu/magma/&quot; rel=&quot;nofollow&quot;&gt;Magma&lt;/a&gt; library aims to offer just that.  Better still, the &lt;a href=&quot;http://cran.r-project.org/package=magma&quot; rel=&quot;nofollow&quot;&gt;magma&lt;/a&gt; R package brings this to R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Moreover, I have a benchmarking paper / vignette / small package almost completed that compares these as well as several BLAS such as Atlas, Goto and the MKL. I'll update this entry with a URL in a couple of days.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Edit on 16 Sep&lt;/em&gt;: The paper I mentioned is now out and on CRAN with its own package &lt;a href=&quot;http://cran.r-project.org/package=gcbd&quot; rel=&quot;nofollow&quot;&gt;gcbd&lt;/a&gt;; I and I wrote a &lt;a href=&quot;http://dirk.eddelbuettel.com/blog/2010/09/15#gcbd_0.2.2&quot; rel=&quot;nofollow&quot;&gt;brief blog entry&lt;/a&gt; about it  too.&lt;/p&gt;&#xA;" OwnerUserId="334" LastEditorUserId="334" LastEditorDisplayName="" LastEditDate="2010-09-16T12:27:58.123" LastActivityDate="2010-09-16T12:27:58.123" />
  <row Id="2571" PostTypeId="2" ParentId="2563" CreationDate="2010-09-11T18:51:18.540" Score="3" ViewCount="" Body="&lt;p&gt;David MacIver had a nice post on the algorithm he used for ranking programming languages based on relative rankings from different users for various factors:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.drmaciver.com/2010/05/rank-aggregation-in-the-right-tool/&quot; rel=&quot;nofollow&quot;&gt;Rank Aggregation in The Right Tool&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;It's based the &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.28.8702&quot; rel=&quot;nofollow&quot;&gt;&lt;em&gt;Rank Aggregation Methods for the Web&lt;/em&gt;&lt;/a&gt; paper by Dwork et al.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want something similar to this site, then &lt;a href=&quot;http://www.reddit.com/&quot; rel=&quot;nofollow&quot;&gt;Reddit&lt;/a&gt;'s code is open source.  There was some discussion of their ranking algorithm on Hacker News a while back:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://news.ycombinator.com/item?id=231168&quot; rel=&quot;nofollow&quot;&gt;Reddit's Ranking Algorithm&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;For a quick and simple take on &quot;hotness&quot;, consider Evan Miller's algorithm with time decay:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.evanmiller.org/rank-hotness-with-newtons-law-of-cooling.html&quot; rel=&quot;nofollow&quot;&gt;Rank Hotness With Newton's Law of Cooling&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-09-11T18:51:18.540" />
  <row Id="2572" PostTypeId="1" AcceptedAnswerId="2605" CreationDate="2010-09-11T19:05:38.680" Score="2" ViewCount="217" Body="&lt;p&gt;Could you suggest good articles / books / online materials on correlation stress testing?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Correlation stress testing: say, we have a positive semidefinite correlation matrix. We might want to explore the impact on the results if some or all elements of the matrix are increased or a decrease by a given value/s. Of course, there are limits to stress testing due to the requirement of the correlation matrix to remain positive semidefinite.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you in advance for your suggestions!&lt;/p&gt;&#xA;" OwnerUserId="1250" LastActivityDate="2010-09-13T10:36:05.453" Title="Correlation stress testing" Tags="&lt;correlation&gt;" AnswerCount="2" />
  <row Id="2573" PostTypeId="1" AcceptedAnswerId="2621" CreationDate="2010-09-11T19:36:38.390" Score="3" ViewCount="691" Body="&lt;p&gt;If we want to visibly see the distribution of a continuous data, which one among histogram and pdf should be used?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are the differences, not formula wise, between histogram and pdf?&lt;/p&gt;&#xA;" OwnerUserId="1102" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-10-06T14:33:18.633" LastActivityDate="2010-10-06T14:33:18.633" Title="Difference between histogram and pdf?" Tags="&lt;distributions&gt;&lt;pdf&gt;&lt;histogram&gt;" AnswerCount="3" CommentCount="3" />
  <row Id="2574" PostTypeId="2" ParentId="2573" CreationDate="2010-09-11T19:40:11.540" Score="5" ViewCount="" Body="&lt;p&gt;A histogram is pre-computer age estimate of a density.  A density estimate is an alternative. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;These days we use both, and there is a rich literature about which defaults one should use.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A pdf, on the other hand, is a closed-form expression for a &lt;em&gt;given&lt;/em&gt; distribution. That is different from describing your dataset with an &lt;em&gt;estimated&lt;/em&gt; density or histogram.&lt;/p&gt;&#xA;" OwnerUserId="334" LastActivityDate="2010-09-11T19:40:11.540" CommentCount="5" />
  <row Id="2575" PostTypeId="2" ParentId="2573" CreationDate="2010-09-11T20:00:52.923" Score="2" ViewCount="" Body="&lt;p&gt;There's no hard and fast rule here.  If you know the density of your population, then a PDF is better.  On the other hand, often we deal with samples and a histogram might convey some information that an estimated density covers up.  For example, Andrew Gelman makes this point:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;a href=&quot;http://www.stat.columbia.edu/~cook/movabletype/archives/2009/10/variations_on_t.html&quot; rel=&quot;nofollow&quot;&gt;Variations on the histogram&lt;/a&gt;&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;A key benefit of a histogram is that, as a plot of raw data, it contains the seeds of its own error assessment. Or, to put it another way, the jaggedness of a slightly undersmoothed histogram performs a useful service by visually indicating sampling variability. That's why, if you look at the histograms in my books and published articles, I just about always use lots of bins. I also almost never like those kernel density estimates that people sometimes use to display one-dimensional distributions. I'd rather see the histogram and know where the data are.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-09-11T20:00:52.923" CommentCount="3" />
  <row Id="2576" PostTypeId="1" CreationDate="2010-09-11T20:20:34.950" Score="6" ViewCount="266" Body="&lt;p&gt;I'm working with the results of a survey which has multiple questions. All answers (in this case) are categorical and ordinal (such as very unhappy, unhappy, neutral, happy, very happy).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm looking for a way to sort the questions from those with &quot;worst results&quot; to those with &quot;best results&quot;. Getting the extremes is somewhat easy visually. If I plot the distribution of answers for each question, I can identify which questions have lots of 'good' answers (distribution is negatively skewed) or those with lots of 'bad' answers (positively skewed histogram). So &lt;em&gt;picking&lt;/em&gt; the extremes is easy but this is also dependent on the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Quantitatively however, I don't know what to do. Since the answers are on an ordinal, but not an interval scale, I don't know how to calculate an aggregate number for each question. Perhaps giving a numerical value to each category (such as -2, -1, 0, 1 or 2) and summing up the results might work if there's nothing better, but I do realize that mathematically this is not accurate as this is not an interval scale.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Oh, I'm not a statistician, just a programmer. I hope there is a reasonable option to this, I can imagine it's a fairly common question with categorical data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS I use R in case there is something built-in.&lt;/p&gt;&#xA;" OwnerUserId="840" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-04T15:49:14.910" LastActivityDate="2010-11-04T20:10:51.363" Title="How to rank the results of questions with categorical answers?" Tags="&lt;categorical-data&gt;&lt;scales&gt;&lt;survey&gt;" AnswerCount="5" FavoriteCount="1" />
  <row Id="2577" PostTypeId="2" ParentId="2576" CreationDate="2010-09-11T20:38:07.827" Score="1" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;If I plot the distribution of answers for each question, I can identify which questions have lots of 'good' answers (distribution is negatively skewed) or those with lots of 'bad' answers (positively skewed histogram). So picking the extremes is easy but this is also dependent on the data.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Is an absolute ranking necessary?  Like you point out, things may be fuzzier in the middle, so is it relevant to your investigation to distinguish between rank 8 and 9 (or whatever) based on some scoring method?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One approach would be to continue with what you stated above -- look at the distributions and categorize questions based on proportions of good/ok/bad based on the data.  You might start with a mosaic plot (with questions as factors) to explore your data.  This may help reveal criteria for collapsing questions into groups.  Instead of piecemeal rankings, they get classified into categories (e.g. what might have been ranks 1-5 become category 1, etc).  &lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-09-11T20:38:07.827" CommentCount="2" />
  <row Id="2578" PostTypeId="2" ParentId="2576" CreationDate="2010-09-11T21:08:54.290" Score="3" ViewCount="" Body="&lt;p&gt;Recoding your data with numerical values seems ok, provided the assumption of an ordinal scale holds. This is often the case for Likert-type item, but see these related questions:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://stats.stackexchange.com/questions/2401/is-it-appropriate-to-treat-n-point-likert-scale-data-as-n-trials-from-a-binomial/&quot;&gt;Is it appropriate to treat n-point Likert scale data as n trials from a binomial process?&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://stats.stackexchange.com/questions/10/under-what-conditions-should-likert-scales-be-used-as-ordinal-or-interval-data/&quot;&gt;Under what conditions should Likert scales be used as ordinal or interval data?&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;When validating a questionnaire, we often provide usual numerical summaries (mean $\pm$ sd, range, quartiles) to highlight ceiling/floor effect, that is higher response rate in the extreme range of the scale. Dotplots are also great tool to summarize such data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is just for visualization/summary purpose. If you want to get into more statistical stuff, you can use proportional odds model or ordinal logistic regression, for ordinal items, and multinomial regression, for discrete ones.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-04T15:51:20.853" LastActivityDate="2010-11-04T15:51:20.853" />
  <row Id="2579" PostTypeId="1" CreationDate="2010-09-11T23:08:48.143" Score="0" ViewCount="299" Body="&lt;p&gt;good day,&#xA;I have a question related to the time fixed effects when using Swamy Random coefficient model, as you may know in the fixed and random effect context, we use a set of  time dummies to estimate the time fixed effects, and i would like if we can use the same approch whith swamy???&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any hint would be highly appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ama&lt;/p&gt;&#xA;" OwnerUserId="1251" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-16T12:46:48.100" LastActivityDate="2010-09-16T12:46:48.100" Title="Swamy Random Coefficient Model and time fixed effects" Tags="&lt;econometrics&gt;&lt;panel-data&gt;" AnswerCount="1" />
  <row Id="2580" PostTypeId="1" AcceptedAnswerId="2606" CreationDate="2010-09-11T23:12:23.770" Score="5" ViewCount="212" Body="&lt;p&gt;I have data with a double peak that I'm trying to model, and there's enough overlap between the peaks that I can't treat them independently. A histogram of the data might look something like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/EuTFp.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've created two models for this: one uses two Poisson distributions, and the other uses two negative binomial distributions (to account for overdispersion). What's the appropriate way to tell which model fits the data more accurately?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My initial thought is that I could use a Kolmogorov-Smirnov test to compare each model to the data, then do a likelihood ratio test to see if one is a significantly better fit.  Does this make sense?  If so, I'm not exactly sure how to perform the likelihood  ratio test. Is chi-squared appropriate, and how many degrees of freedom do I have?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If it helps, some (very simplified) R code for the models might look something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;## inital data points&#xA;a &amp;lt;- read.table(&quot;data&quot;)&#xA;&#xA;#create model data&#xA;model.pois = c(rpois(1000000,200),rpois(500000,250))&#xA;model.nb = c(rnbinom(1000000,200,0.5),rnbinom(500000,275,0.5)&#xA;&#xA;#Kolmogorov-Smirnov test&#xA;#use ks.boot, since it's count data that may contain duplicate values&#xA;kpois = ks.boot(model.pois,a)&#xA;knb = ks.boot(model.nb,a)&#xA;&#xA;#here's where I'd do some sort of likelihood ratio test&#xA;# . . .&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Edit:  Here's an image that may explain the data and the distributions I'm fitting better.  It's totally clear from the visualization that the second model (using the negative binomial dist to account for overdispersion) is a better fit. I'd like to show this quantitatively, though.&#xA;&lt;img src=&quot;http://i.imgur.com/NE6Vq.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(red - data, green - model)&lt;/p&gt;&#xA;" OwnerUserId="54" LastEditorUserId="54" LastEditorDisplayName="" LastEditDate="2010-09-12T11:35:01.417" LastActivityDate="2011-05-12T19:02:32.260" Title="Measuring goodness-of-fit in a model that combines two distributions" Tags="&lt;modeling&gt;&lt;likelihood-ratio&gt;&lt;kolmogorov-smirnov&gt;" AnswerCount="3" CommentCount="4" FavoriteCount="2" />
  <row Id="2581" PostTypeId="2" ParentId="2572" CreationDate="2010-09-11T23:57:36.683" Score="0" ViewCount="" Body="&lt;p&gt;The answer may depend on why you want to do stress testing but here is one partial idea:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can do a &lt;a href=&quot;http://en.wikipedia.org/wiki/Eigendecomposition_of_a_matrix&quot; rel=&quot;nofollow&quot;&gt;eigendecompostion&lt;/a&gt; of your current matrix, The eigenvectors are aligned with the axis of the ellipsoid corresponding to the correlation matrix and the eigenvalues are a measure of how &quot;long&quot; the ellipsoid is along the corresponding vector. So one way to perform stress testing is to vary the eigenvalues so that the shape of the ellipsoid changes from the original one.&lt;/p&gt;&#xA;" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-09-12T02:11:56.117" LastActivityDate="2010-09-12T02:11:56.117" CommentCount="1" />
  <row Id="2582" PostTypeId="2" ParentId="2576" CreationDate="2010-09-12T02:48:34.503" Score="3" ViewCount="" Body="&lt;p&gt;If all your questions have the same response scale and they are standard Likert items, scaling the item 1,2,3,4,5 and taking the mean is generally fine.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can investigate the robustness of the rank ordering by experimenting with different scaling procedures (e.g., 0, 0, 0, 1, 1 is common where you want to assess the percentage happy or very happy; or agreeing or strongly agreeing). From my experience, such variants in scaling will give you almost identical question orderings. You could also explore optimal scaling principal components or some form of polytomous IRT approach is you wanted to be sophisticated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A table with three columns would be fine: rank, item text, mean. You could also do the same thing with question on the x axis and mean on the y axis.&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-09-12T02:48:34.503" />
  <row Id="2583" PostTypeId="2" ParentId="2352" CreationDate="2010-09-12T03:25:19.597" Score="0" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;This paper is somewhat controversial, and somewhat ignored&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Not really, it's well regarded where the theory of model selection is concerned, though it's certainly misinterpreted.  The real issue is how relevant it is to the practice of modeling in the wild.  Suppose you perform the simulations for the cases you propose to investigate and determine that LOOCV is indeed inconsistent.  The only reason you'd get that is because you already knew the &quot;true&quot; model and could hence determine that the probability of recovering the &quot;true&quot; model does not converge to 1.  For modeling in the wild, how often is this true (that the phenomena are described by linear models and the &quot;true&quot; model is a subset of those in consideration)?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Shao's paper is certainly interesting for advancing the theoretical framework.  It even provides some clarity: if the &quot;true&quot; model is indeed under consideration, then we have the consistency results to hang our hats on.  But I'm not sure how interesting actual simulations for the cases you describe would be.  This is largely why most books like EOSL don't focus as much on Shao's result, but instead on prediction/generalization error as a criterion for model selection.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT:  The very short answer to your question is: Shao's results are applicable when you're performing least squares estimation, quadratic loss function.  Not any wider.  (I think there was an interesting paper by Yang (2005?) which investigated whether you could have consistency and efficiency, with a negative answer.)&lt;/p&gt;&#xA;" OwnerUserId="251" LastEditorUserId="251" LastEditorDisplayName="" LastEditDate="2010-09-12T04:19:37.793" LastActivityDate="2010-09-12T04:19:37.793" CommentCount="3" />
  <row Id="2584" PostTypeId="2" ParentId="2547" CreationDate="2010-09-12T05:01:58.287" Score="3" ViewCount="" Body="&lt;p&gt;For a concrete example, consider the mean ages for the Congo (DRC) and Japan.  One is devastated by civil war, the other is well developed with an ageing population.  The mean isn't terribly interesting for an apples to apples comparison.  On the other hand, the median can be informative as a measure of central tendency since by definition we have half above, half below.  The wikipedia article on &lt;a href=&quot;http://en.wikipedia.org/wiki/Population_pyramid&quot; rel=&quot;nofollow&quot;&gt;Population Pyramid&lt;/a&gt; might be enlightening (see the sections on youth bulge, ageing populations).&lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-09-12T05:01:58.287" />
  <row Id="2585" PostTypeId="1" CreationDate="2010-09-12T05:04:04.770" Score="1" ViewCount="124" Body="&lt;p&gt;I have a large sample of market data - that is the prices and amount of goods being sold by vendors at specific, but inconsistent, points in time. Assuming I am a buyer and the quality of the goods are constant across the entire market (consumable goods by one manufacturer, let's say), how would you graph this data? Keep in mind that outliers may exist (fire sales or inflated asking prices).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Seeing as I'm only concerned with the lowest-priced goods at a specific moment in time, a market average wouldn't do much for me unless I was buying most of the market's supply. I considered a box plot, but that also tends to be mean/median-biased. A simple line/area/bar graph of the lowest prices over time doesn't seem to give enough information, especially if there isn't much volume available at the lowest price.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;*I'm not a stats guy, so go easy on me!&lt;/p&gt;&#xA;" OwnerUserId="1253" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-16T06:31:51.467" LastActivityDate="2010-09-18T21:46:28.497" CommunityOwnedDate="2010-09-12T05:04:04.770" Title="Graphing low-end prices of goods in a market over time" Tags="&lt;time-series&gt;&lt;data-visualization&gt;" AnswerCount="2" CommentCount="3" />
  <row Id="2586" PostTypeId="2" ParentId="2585" CreationDate="2010-09-12T08:12:21.863" Score="2" ViewCount="" Body="&lt;p&gt;You could create a scatterplot with time on the x-axis and price on the y-axis.&#xA;You could make the size of the points proportional to the amount sold (or some function of amount sold such as log of amount sold). You could add a line that passed through the minimum values or some estimate of the minimum, where the data was unavailable. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are trying to model the minimum price at any given point in time, it seems like you would need to make some assumptions about the duration of any particular price. If you knew the minimum price for each time period you could graph just that data. If there are time points with missing data, you could use some form of missing data imputation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Local quantile regression might also be of interest.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might also need to consider issues of invalid data: e.g., very low prices that only last a short period of time or have a minimal quantity of stock that for what ever reason should not be interpolated to future time points.&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-09-12T08:12:21.863" CommunityOwnedDate="2010-09-12T08:12:21.863" />
  <row Id="2587" PostTypeId="1" AcceptedAnswerId="2595" CreationDate="2010-09-12T11:22:28.790" Score="2" ViewCount="92" Body="&lt;p&gt;Are the two synonymous? The reason I am asking is that I have a paper that calculates a certain p-value as the upper tail of the hypergeometric distribution:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\Sigma_{k}^{m} = \frac{\binom{m}{k}\binom{N-m}{n-k}}{\binom{N}{n}}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to implement their proposal using Python. &lt;a href=&quot;http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.hypergeom.html#scipy.stats.hypergeom&quot; rel=&quot;nofollow&quot;&gt;Scipy offers a hypergeometric distribution&lt;/a&gt; and it can calculate the survival function of that distribution from a given point. So I am wondering, are the two the same? Can I just use that function?&lt;/p&gt;&#xA;" OwnerUserId="977" LastActivityDate="2010-09-12T17:00:58.643" Title="Is the survival function the same as the upper tail?" Tags="&lt;distributions&gt;" AnswerCount="2" />
  <row Id="2588" PostTypeId="2" ParentId="2587" CreationDate="2010-09-12T11:29:59.453" Score="0" ViewCount="" Body="&lt;p&gt;Yes it is the same as upper tail. See: &lt;a href=&quot;http://en.wikipedia.org/wiki/Survival_function&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Survival_function&lt;/a&gt;&lt;/p&gt;&#xA;" LastActivityDate="2010-09-12T11:29:59.453" CommentCount="2" />
  <row Id="2589" PostTypeId="2" ParentId="2576" CreationDate="2010-09-12T13:42:31.060" Score="2" ViewCount="" Body="&lt;p&gt;A small additional point to the answers already given:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Without assuming your ordinal data is interval, you can compare any convenient quantiles - e.g. medians.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or, when comparing X vs Y which are both ordered categorical, you can estimate something like P(Y&gt;X) - P(X&gt;Y) or P(Y&gt;X) + 0.5 * P(Y=X) (etc.), where you estimate probabilities by proportions of course.&lt;/p&gt;&#xA;" OwnerUserId="805" LastActivityDate="2010-09-12T13:42:31.060" />
  <row Id="2591" PostTypeId="1" AcceptedAnswerId="2593" CreationDate="2010-09-12T15:12:02.687" Score="1" ViewCount="548" Body="&lt;p&gt;If one is testing assumption of homoscedasticity, parametric (Bartlett Test of Homogeneity of Variances, &lt;code&gt;bartlett.test&lt;/code&gt;) and non-parametric (Figner-Killeen Test of Homogeneity of Variances, &lt;code&gt;fligner.test&lt;/code&gt;) tests are available. How to tell which kind to use? Should this depend on e.g. normality of the data?&lt;/p&gt;&#xA;" OwnerUserId="144" LastActivityDate="2010-09-12T16:18:40.850" Title="When to use (non)parametric test of homoscedasticity assumption?" Tags="&lt;r&gt;&lt;homogeneity&gt;" AnswerCount="1" />
  <row Id="2592" PostTypeId="1" AcceptedAnswerId="2596" CreationDate="2010-09-12T16:11:22.923" Score="2" ViewCount="345" Body="&lt;p&gt;I want to project a vector onto space transformed by PCA.&#xA;I've calculated PCA in R language using &lt;code&gt;prcomp&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I should be able to multiply my vector by matrix of rotation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should principal components in this matrix be arranged in rows or columns?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Vector that I want to project:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;attr1 attr2 ... &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Rotation matrix:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;PC1 PC2 PC3 ... &#xA;... ... ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1260" LastEditorUserId="1260" LastEditorDisplayName="" LastEditDate="2010-09-12T16:23:55.990" LastActivityDate="2010-09-12T17:03:16.110" Title="How to project a vector onto matrix of rotation in PCA" Tags="&lt;r&gt;&lt;pca&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="2593" PostTypeId="2" ParentId="2591" CreationDate="2010-09-12T16:18:40.850" Score="2" ViewCount="" Body="&lt;p&gt;It seems that the FK test is to be prefered in case of strong departure from the normality (to which the Bartlett test is sensible). Quoting the on-line help, &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The Fligner-Killeen (median) test has&#xA;  been determined in a simulation study&#xA;  as one of the many tests for&#xA;  homogeneity of variances which is most&#xA;  robust against departures from&#xA;  normality, see Conover, Johnson &amp;amp;&#xA;  Johnson (1981).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Generally speaking, the Levene test works well in the ANOVA framework, providing there are small to moderate deviations from the normality. In this case, it outperfoms the Bartlett test. If the distribution are nearly normal, however, the Bartlett test is better. I've also heard of the Brown–Forsythe test as a non-parametric alternative to the Levene test. Basically, it relies on either the median or the trimmed mean (as compared to the mean in the Levene test). According to Brown and Forsythe (1974), a test based on the mean provided the best power for symmetric distributions with moderate tails.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In conclusion, I would say that if there is strong evidence of departure from the normality (as seen e.g., with the help of a Q-Q plot), then use a non-parametric test (FK or BF test); otherwise, use Levene or Bartlett test.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There was also a small discussion about this test for small and large samples in the R Journal, last year, &lt;a href=&quot;http://journal.r-project.org/archive/2009-2/RJournal_2009-2_Coeurjolly~et~al.pdf&quot; rel=&quot;nofollow&quot;&gt;asympTest: A Simple R Package for Classical Parametric Statistical Tests and Confidence Intervals in Large Samples&lt;/a&gt;. It seems that the FK test is also available through the &lt;code&gt;coin&lt;/code&gt; interface for permutation tests, see the &lt;a href=&quot;http://cran.r-project.org/web/packages/coin/vignettes/coin.pdf&quot; rel=&quot;nofollow&quot;&gt;vignette&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;References&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Brown, M. B. and Forsythe, A. B. (1974). Robust Tests for Equality of Variances. &lt;em&gt;JASA&lt;/em&gt;, &lt;em&gt;69&lt;/em&gt;, 364-367.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-09-12T16:18:40.850" />
  <row Id="2594" PostTypeId="2" ParentId="2592" CreationDate="2010-09-12T16:24:34.503" Score="1" ViewCount="" Body="&lt;p&gt;I believe that the eigenvectors (i.e., the principal components) should be arranged as columns. &lt;/p&gt;&#xA;" LastActivityDate="2010-09-12T16:24:34.503" />
  <row Id="2595" PostTypeId="2" ParentId="2587" CreationDate="2010-09-12T17:00:58.643" Score="3" ViewCount="" Body="&lt;p&gt;Yes.  By definition, the survival function is $S(X = x) = Pr(X &gt; x) = 1 - F(X = x)$ where $F$ is the CDF.&lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-09-12T17:00:58.643" />
  <row Id="2596" PostTypeId="2" ParentId="2592" CreationDate="2010-09-12T17:03:16.110" Score="3" ViewCount="" Body="&lt;p&gt;Well, @Srikant already gave you the right answer since the rotation (or loadings) matrix contains eigenvectors arranged column-wise, so that you just have to multiply (using &lt;code&gt;%*%&lt;/code&gt;) your vector or matrix of new data with e.g. &lt;code&gt;prcomp(X)$rotation&lt;/code&gt;. Be careful, however, with any extra centering or scaling parameters that were applied when computing PCA EVs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In R, you may also find useful the &lt;code&gt;predict()&lt;/code&gt; function, see &lt;code&gt;?predict.prcomp&lt;/code&gt;. BTW, you can check how projection of new data is implemented by simply entering:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;getS3method(&quot;predict&quot;, &quot;prcomp&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-09-12T17:03:16.110" />
  <row Id="2597" PostTypeId="1" AcceptedAnswerId="2599" CreationDate="2010-09-12T19:49:25.443" Score="7" ViewCount="408" Body="&lt;p&gt;I have found extensive literature proposing all sorts of criteria (e.g. &lt;a href=&quot;http://vohweb.chem.ucla.edu/voh/classes%5Cwinter08%5C160BID48%5CNumClusters.pdf&quot; rel=&quot;nofollow&quot;&gt;Glenn et al. 1985&lt;/a&gt;(pdf) and &lt;a href=&quot;http://www.cc.gatech.edu/~hpark/papers/cluster_JOGO.pdf&quot; rel=&quot;nofollow&quot;&gt;Jung et al. 2002&lt;/a&gt;(pdf)). However, most of these are not that easy to implement (at least from my perspective). I am using &lt;a href=&quot;http://docs.scipy.org/doc/scipy/reference/cluster.hierarchy.html#function-reference&quot; rel=&quot;nofollow&quot;&gt;scipy.cluster.hierarchy&lt;/a&gt; to obtain a cluster hierarchy, and I am now trying to decide how to form flat clusters from that. My goal is to discover common patterns in my observations, so I have no reference to compare the obtained clustering to. Can anyone suggest a pragmatic solution? &lt;/p&gt;&#xA;" OwnerUserId="977" LastActivityDate="2010-09-13T08:26:14.417" Title="What stop-criteria for agglomerative hierarchical clustering are used in practice?" Tags="&lt;clustering&gt;" AnswerCount="3" FavoriteCount="3" />
  <row Id="2598" PostTypeId="1" CreationDate="2010-09-12T20:18:16.740" Score="1" ViewCount="93" Body="&lt;p&gt;Let's pretend there are 10 doors in front of me. Behind one of them, there is a treasure and nothing behind all the others. Intuitively, I can easily determine I have 1 chance out of 10 the pick open the right door (the one with the treasure). I repeat this 5 times (i.e. all door closed, open one door). What are my chance to find the treasure the five times?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In brief, I am wondering how to determine the probably that I find the threasure 5 times in row.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thansk&lt;/p&gt;&#xA;" OwnerUserId="64" LastActivityDate="2010-09-13T12:38:17.643" Title="Probability to find the treasure five times in row" Tags="&lt;probability&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="2599" PostTypeId="2" ParentId="2597" CreationDate="2010-09-12T20:26:40.637" Score="6" ViewCount="" Body="&lt;p&gt;The following Wikipedia entry actually does a pretty good job of explaining the most popular and relatively simple methods:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Determining_the_number_of_clusters_in_a_data_set&quot; rel=&quot;nofollow&quot;&gt;Determining the number of clusters in a data set&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The &lt;a href=&quot;http://en.wikipedia.org/wiki/Determining_the_number_of_clusters_in_a_data_set#The_Elbow_Method&quot; rel=&quot;nofollow&quot;&gt;Elbow Method&lt;/a&gt; heuristic described there is probably the most popular due to its simple explanation (amount of variance explained by number of clusters) coupled with the visual check.  The &lt;a href=&quot;http://en.wikipedia.org/wiki/Determining_the_number_of_clusters_in_a_data_set#An_Information_Theoretic_Approach.5B6.5D&quot; rel=&quot;nofollow&quot;&gt;information theoretic method&lt;/a&gt; isn't hard to implement either and the page has some pseudocode you could use to start.  The latter is analagous to a penalized likelihood based on model complexity as in the well known information criteria such as AIC, BIC, etc.&lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-09-12T20:26:40.637" CommentCount="2" />
  <row Id="2600" PostTypeId="5" CreationDate="2010-09-12T20:54:55.157" Score="0" ViewCount="0" Body="&lt;p&gt;Psychometrics has evolved as a subfield of psychology to become the science of measurement of unobservable individual characteristics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unlike psychophysics which mostly focus on perceptual attributes, psychometrics aims at providing reproducible methods for assessing high-level or cognitive skills through the scientific development and evaluation of reliable and valid mental tests or questionnaires. Common traits of interest are knowledge, attitudes, personality, and abilities.&#xA;Psychometric techniques developed for assessing intelligence or proficiency are now used in political science, social sciences, or health surveys.&#xA;The &lt;a href=&quot;http://www.psychometrika.org/&quot; rel=&quot;nofollow&quot;&gt;&lt;em&gt;Psychometrika&lt;/em&gt;&lt;/a&gt; journal certainly reflects the current state of the art in this field, but other reviews now offer well-acknowledged articles on multivariate/multilevel data analysis, structural equation, or other latent variable models. Recommended books are Nunally and Bernstein (1994), Kline (1998), Boomsma et al. (2001), De Boeck and Wilson (2004), Skrondal and Rabe-Hesketh (2004), and Rao and Sinharay (2007).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many R packages provide dedicated functions for applied psychometrics and item analysis, including exploratory and confirmatory factor analysis, multidimensional scaling, and other methods belonging to Classical Test Theory, Item Response Theory, and Structural Equation Modeling (See the &lt;a href=&quot;http://cran.r-project.org/web/views/Psychometrics.html&quot; rel=&quot;nofollow&quot;&gt;Psychometrics&lt;/a&gt; task view and the &lt;a href=&quot;http://www.jstatsoft.org/v20&quot; rel=&quot;nofollow&quot;&gt;Journal of Statistical Software&lt;/a&gt;). For Matlab users, there is the &lt;a href=&quot;http://ppw.kuleuven.be/okp/software/irtm/&quot; rel=&quot;nofollow&quot;&gt;IRTm&lt;/a&gt; toolbox. For Stata users, there are &lt;a href=&quot;http://www.gllamm.org/&quot; rel=&quot;nofollow&quot;&gt;gllamm&lt;/a&gt; and &lt;a href=&quot;http://www.stata-journal.com/article.html?article=st0119&quot; rel=&quot;nofollow&quot;&gt;raschtest&lt;/a&gt;, among others (Check out the &lt;a href=&quot;http://www.freeirt.org/&quot; rel=&quot;nofollow&quot;&gt;FreeIRT&lt;/a&gt; project).&lt;/p&gt;&#xA;" OwnerUserId="-1" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-26T10:18:59.927" LastActivityDate="2010-09-26T10:18:59.927" />
  <row Id="2601" PostTypeId="4" CreationDate="2010-09-12T20:54:55.157" Score="0" ViewCount="0" Body="Psychometrics has evolved as a subfield of psychology to become the science of measurement of unobservable individual characteristics." OwnerUserId="-1" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-12T20:54:55.157" LastActivityDate="2010-09-12T20:54:55.157" />
  <row Id="2602" PostTypeId="1" AcceptedAnswerId="2607" CreationDate="2010-09-12T21:38:14.563" Score="3" ViewCount="451" Body="&lt;p&gt;I often find parts of financial correlation matrices not statistically significantly different from zero. Sometimes, these correlations have a tangible effect on results - low correlations lead to high diversification benefit. Also, the sign of correlations can skew the results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would you consider it appropriate to make these correlations equal to zero to at least get rid of the directionality and, if necessary, fix the negative eigenvalues that might arise due to such corrections?&lt;/p&gt;&#xA;" OwnerUserId="1250" LastActivityDate="2010-09-17T21:30:52.620" Title="How to deal with correlations not statistically significantly different from zero?" Tags="&lt;correlation&gt;&lt;statistical-significance&gt;" AnswerCount="2" />
  <row Id="2603" PostTypeId="2" ParentId="2598" CreationDate="2010-09-12T21:54:02.223" Score="3" ViewCount="" Body="&lt;p&gt;Is the location of the treasure changed after each attempt? Wouldn't it be $(10\%)^5$?&lt;/p&gt;&#xA;" OwnerUserId="1250" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-13T12:38:17.643" LastActivityDate="2010-09-13T12:38:17.643" CommentCount="2" />
  <row Id="2604" PostTypeId="2" ParentId="2580" CreationDate="2010-09-12T23:31:55.703" Score="6" ViewCount="" Body="&lt;p&gt;You can't compare them directly since the Negative Binomial has more parameters. Indeed the Poisson is &quot;nested&quot; within the Negative Binomial in the sense that it's a limiting case, so the NegBin will &lt;em&gt;always&lt;/em&gt; fit better than the Poisson. However, that makes it possible to consider something like a likelihood ratio test but the fact that the Poisson is at the boundary of the parameter space for the negative binomial may affect the distribution of the test statistic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In any case, even if the difference in number of parameters wasn't a problem, you can't do K-S tests directly because you have &lt;em&gt;estimated&lt;/em&gt; parameters, and K-S is specifically for the case where all parameters are specified. Your idea of using the bootstrap deals with this issue, but not the first one (difference in number of parameters)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd also be considering smooth tests of goodness of fit (e.g. see Rayner and Best's book), which, for example, can lead to a partition the chi-square goodness of fit test into components of interest (measuring deviations from the Poisson model in this case) - taken out to say fourth order or sixth order, this should lead to a test with good power for the NegBin alternative.&lt;/p&gt;&#xA;" OwnerUserId="805" LastActivityDate="2010-09-12T23:31:55.703" CommentCount="5" />
  <row Id="2605" PostTypeId="2" ParentId="2572" CreationDate="2010-09-13T00:13:30.363" Score="3" ViewCount="" Body="&lt;p&gt;&lt;em&gt;Of course, there are limits to stress testing due to the requirement of the correlation matrix to remain positive semidefinite.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;finding these limits is, imho, the critical part of what you are trying to do.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The answer to this problem depends on how many of the correlation coefficient you want to stress test simultaneously.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;if you want to stress test &lt;em&gt;only one&lt;/em&gt; coeficient at a time (say the entry $a_{ij},i\neq j$ of your correlation matrix), then you can use the &lt;a href=&quot;http://en.wikipedia.org/wiki/Diagonally_dominant_matrix&quot; rel=&quot;nofollow&quot;&gt;Gershgorin&lt;/a&gt; theorem to place explicit bounds of the values that $a_{ij}$ can take. Let $a_{ij}^+$ and $a_{ij}^-$ be these bounds. Then you can stress test by computing you measure of risk for a grid of values in $(a_{ij}^-,a_{ij}^+)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to stress test &lt;em&gt;more than one&lt;/em&gt; (say $k&gt;1$) coefficient simultaneously, then there is no closed form solution for the bounds on these correlations coefficients. An exact solution for this problem exists however (i.e. the $k$-dimensional ellipse inside which your $k$ correlation coefficient are allowed to reside)  but this requires a higher level of mathematical sophistication. If this is the case you are interested in, let me know in the comments.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Exact solution for $k&gt;1$: this is can be recasted as an &lt;a href=&quot;http://stanford.edu/~boyd/papers/sdp.html&quot; rel=&quot;nofollow&quot;&gt;SDP&lt;/a&gt; problem. &#xA;say $a=(a_1,...,a_k)$ are $k$ correlation coefficients you want to vary and $w=(w_1,...,w_k)$ are strictly positive numbers and $C(a)$ is the $p$ by $p$ correlation matrix with $p\geq k$. Then, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\underset{a|w}{\min.}\; a'w$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$s.t. C(a)\in \mathbb{R}_{++}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\mathbb{R}_{++}$ is the set of all SDP symmetric matrices. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Imposing this constraint requires SDP programing. It can be shown that (via the Shur complement of $C(a)$) this is equivalent to imposing $p$ linear inequality and $p$ quadratic inequality on the values of $a$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, we know the set of all solutions is a $k$ dimensional ellipse. Such ellipse is defined by $k+1$ points on it's boundary. Each solution to this SDP (corresponding to a given vector $w$) will be one point on the boundary of this ellipse.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, each run of the minimization problem has time complexity $O(p^3)$ where $p$ is the number of assets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;there is a nice packages to solve SDP in &lt;a href=&quot;http://cvxr.com/cvx/&quot; rel=&quot;nofollow&quot;&gt;Matlab&lt;/a&gt;. I think there is also one in R, but  last i checked it was not as nice.&lt;/p&gt;&#xA;" OwnerUserId="603" LastEditorUserId="603" LastEditorDisplayName="" LastEditDate="2010-09-13T10:36:05.453" LastActivityDate="2010-09-13T10:36:05.453" CommentCount="14" />
  <row Id="2606" PostTypeId="2" ParentId="2580" CreationDate="2010-09-13T00:45:03.360" Score="2" ViewCount="" Body="&lt;p&gt;You can use a metric such as &lt;a href=&quot;http://en.wikipedia.org/wiki/Mean_squared_error&quot; rel=&quot;nofollow&quot;&gt;Mean Squared Error&lt;/a&gt; between actual vs predicted values to compare the two models.&lt;/p&gt;&#xA;" LastActivityDate="2010-09-13T00:45:03.360" CommentCount="1" />
  <row Id="2607" PostTypeId="2" ParentId="2602" CreationDate="2010-09-13T00:56:45.050" Score="2" ViewCount="" Body="&lt;p&gt;&lt;em&gt;I often find parts of financial correlation matrices not statistically significantly different from zero&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might want to be careful with that. First of all, statistically significant and large enough to be meaningful are not the same &lt;a href=&quot;http://stats.stackexchange.com/questions/2516/are-large-data-sets-inappropriate-for-hypothesis-testing/&quot;&gt;thing&lt;/a&gt;. Second, be careful that the test you use to assess the significance of the correlation coefficients do not assume the underlying data to be Gaussian because if you are dealing with financial series, they are most likely fat-tailed (and this is a case where it would make a difference, i.e. render your test void). Third, testing whether a single correlation coefficient is statistically significant is not the same as testing whether some of your correlation coefficients are statistically &lt;a href=&quot;http://en.wikipedia.org/wiki/Multiple_comparisons&quot; rel=&quot;nofollow&quot;&gt;significant&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Also, the sign of correlations can skew the results.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strike&gt;Presumably, not if the correlations are small to begin with (i.e. if you only fudge with small correlation coefficients this should not be a concern).&lt;/strike&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;see Edward's comment below.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;fix the negative eigenvalues that might arise due to such corrections?&lt;/em&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strike&gt;&lt;em&gt;decreasing&lt;/em&gt; the absolute value of a correlation coefficient (as you consider doing) will never make a PSD matrix become indefinite (&amp;lt;=&gt; it will not change the sign of any of your eigenvalues).&lt;/strike&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;see Whuber's comment below.&lt;/p&gt;&#xA;" OwnerUserId="603" LastEditorUserId="603" LastEditorDisplayName="" LastEditDate="2010-09-13T16:19:52.300" LastActivityDate="2010-09-13T16:19:52.300" CommentCount="6" />
  <row Id="2608" PostTypeId="2" ParentId="2602" CreationDate="2010-09-13T01:48:17.720" Score="1" ViewCount="" Body="&lt;p&gt;I am not sure if this is feasible in your context but one thing you can do to avoid these issues is to use bayesian estimation and to compute the expected values of different investing decisions based on the posterior distribution of the parameters. &lt;/p&gt;&#xA;" LastActivityDate="2010-09-13T01:48:17.720" />
  <row Id="2609" PostTypeId="2" ParentId="2597" CreationDate="2010-09-13T05:22:06.293" Score="2" ViewCount="" Body="&lt;p&gt;I recently became fund of the &lt;a href=&quot;http://www.r-statistics.com/2010/06/clustergram-visualization-and-diagnostics-for-cluster-analysis-r-code/&quot; rel=&quot;nofollow&quot;&gt;clustergram visualization&lt;/a&gt; method (implemented in R).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I use it for an extra method to assess a &quot;good&quot; number of clusters.  Extending it to other clustering methods is not so hard (I actually did it, just didn't get to publish the code)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/7hOv0.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="253" LastActivityDate="2010-09-13T05:22:06.293" />
  <row Id="2610" PostTypeId="2" ParentId="2597" CreationDate="2010-09-13T08:26:14.417" Score="5" ViewCount="" Body="&lt;p&gt;It is rather difficult to provide a clear-cut solution about how to choose the &quot;best&quot; number of clusters in your data, whatever the clustering method you use, because Cluster Analysis seeks to isolate groups of statistical units (whether it be individuals or variables) for exploratory or descriptive purpose, essentially. Hence, you also have to interpret the output of your clustering scheme and several cluster solutions may be equally interesting. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, regarding usual statistical criteria used to decide when to stop to aggregate data, as pointed by @ars most are &lt;em&gt;visual-guided criteria&lt;/em&gt;, including the analysis of the dendrogram or the inspection of clusters profiles, also called &lt;a href=&quot;http://en.wikipedia.org/wiki/Silhouette_%28clustering%29&quot; rel=&quot;nofollow&quot;&gt;silhouette&lt;/a&gt; plots (Rousseeuw, 1987). Several &lt;em&gt;numerical criteria&lt;/em&gt;, also known as validity indices, were also proposed, e.g. Dunn’s validity index, Davies-Bouldin valid- ity index, C index, Hubert’s gamma, to name a few. Hierarchical clustering is often run together with k-means (in fact, several instances of k-means since it is a stochastic algorithm), so that it add support to the clustering solutions found. I don't know if all of this stuff is readily available in Python, but a huge amount of methods is available in R (see the &lt;a href=&quot;http://cran.r-project.org/web/views/Cluster.html&quot; rel=&quot;nofollow&quot;&gt;Cluster&lt;/a&gt; task view, already cited by @mbq for a related question, &lt;a href=&quot;http://stats.stackexchange.com/questions/183/what-tools-could-be-used-for-applying-clustering-algorithms-on-movielens/518#518&quot;&gt;What tools could be used for applying clustering algorithms on MovieLens?&lt;/a&gt;). Other approaches include &lt;a href=&quot;http://en.wikipedia.org/wiki/Fuzzy_clustering&quot; rel=&quot;nofollow&quot;&gt;fuzzy clustering&lt;/a&gt; and &lt;a href=&quot;http://www.stat.washington.edu/mclust/&quot; rel=&quot;nofollow&quot;&gt;model-based clustering&lt;/a&gt; (also called &lt;em&gt;latent trait analysis&lt;/em&gt;, in the psychometric community) if you seek more robust way to choose the number of clusters in your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;BTW, I just came across this webpage, &lt;a href=&quot;http://code.google.com/p/scipy-cluster/&quot; rel=&quot;nofollow&quot;&gt;scipy-cluster&lt;/a&gt;, which is &lt;em&gt;an extension to Scipy for generating, visualizing, and analyzing hierarchical clusters&lt;/em&gt;. Maybe it includes other functionalities? I've also heard of &lt;a href=&quot;http://pychem.sourceforge.net/&quot; rel=&quot;nofollow&quot;&gt;PyChem&lt;/a&gt; which offers pretty good stuff for multivariate analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The following reference may also be helpful:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Steinley, D., &amp;amp; Brusco, M. J. (2008). Selection of variables in cluster analysis: An empirical comparison of eight procedures. &lt;em&gt;Psychometrika&lt;/em&gt;, &lt;em&gt;73&lt;/em&gt;, 125-144.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-09-13T08:26:14.417" CommentCount="2" />
  <row Id="2611" PostTypeId="1" CreationDate="2010-09-13T11:47:27.230" Score="15" ViewCount="498" Body="&lt;p&gt;The logic of multiple imputation (MI) is to impute the missing values not once but several (typically M=5) times, resulting in M completed datasets. The M completed datasets are then analyzed with complete-data methods upon which the M estimates and their standard errors are combined using Rubin's formulas to obtain the &quot;overall&quot; estimate and its standard error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Great so far, but i'm not sure how to apply this recipe when variance components of a mixed-effects model are concerned. The sampling distribution of a variance component is asymmetrical - therefore the corresponding confidence interval can't be given in the typical &quot;estimate ± 1.96*se(estimate)&quot; form. For this reason the R packages lme4 and nlme don't even provide the standard errors of the variance components, but only provide confidence intervals.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can therefore perform MI on a dataset and then get M confidence intervals per variance component after fitting the same mixed-effect model on the M completed datasets. The question is how to combine these M intervals into one &quot;overall&quot; confidence interval.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I guess this should be possible - the authors of an article (yucel &amp;amp; demirtas (2010) Impact of non-normal random effects on inference by MI) seem to have done it, but they don't explain exactly how.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any tips would be much obliged!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cheers, Rok&lt;/p&gt;&#xA;" OwnerUserId="1266" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-13T12:20:07.773" LastActivityDate="2010-10-06T14:31:47.217" Title="How to combine confidence intervals for a variance component of a mixed-effects model when using multiple imputation" Tags="&lt;modeling&gt;&lt;confidence-interval&gt;&lt;mixed-model&gt;&lt;data-imputation&gt;" AnswerCount="3" CommentCount="5" FavoriteCount="3" />
  <row Id="2612" PostTypeId="2" ParentId="2611" CreationDate="2010-09-13T12:57:32.943" Score="5" ViewCount="" Body="&lt;p&gt;This is a great question! Not sure this is a full answer, however, I drop these few lines in case it helps.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems that Yucel and Demirtas (2010) refer to an older paper published in the JCGS, &lt;a href=&quot;http://sci2s.ugr.es/keel/pdf/specific/articulo/schafer_yucel02.pdf&quot; rel=&quot;nofollow&quot;&gt;Computational strategies for multivariate linear mixed-effects models with missing values&lt;/a&gt;, which uses an hybrid EM/Fisher scoring approach for producing likelihood-based estimates of the VCs. It has been implemented in the R package &lt;a href=&quot;http://cran.r-project.org/web/packages/mlmmm/index.html&quot; rel=&quot;nofollow&quot;&gt;mlmmm&lt;/a&gt;. I don't know, however, if it produces CIs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Otherwise, I would definitely check the &lt;a href=&quot;http://www.mrc-bsu.cam.ac.uk/bugs/winbugs/contents.shtml&quot; rel=&quot;nofollow&quot;&gt;WinBUGS&lt;/a&gt; program, which is largely used for multilevel models, including those with missing data. I seem to remember it will only works if your MV are in the response variable, not in the covariates because we generally have to specify the full conditional distributions (if MV are present in the independent variables, it means that we must give a prior to the missing Xs, and that will be considered as a parameter to be estimated by WinBUGS...). It seems to apply to R as well, if I refer to the following thread on r-sig-mixed, &lt;a href=&quot;https://stat.ethz.ch/pipermail/r-sig-mixed-models/2008q3/001192.html&quot; rel=&quot;nofollow&quot;&gt;missing data in lme, lmer, PROC MIXED&lt;/a&gt;. Also, it may be worth looking at the &lt;a href=&quot;http://www.cmm.bristol.ac.uk/MLwiN/&quot; rel=&quot;nofollow&quot;&gt;MLwiN&lt;/a&gt; software.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-13T14:14:58.480" LastActivityDate="2010-09-13T14:14:58.480" CommentCount="3" />
  <row Id="2613" PostTypeId="1" AcceptedAnswerId="2618" CreationDate="2010-09-13T13:14:51.620" Score="6" ViewCount="258" Body="&lt;p&gt;I am advising a small medical study (two groups, treatment is a dummy variable), i.e. a 2x2 contingency table. I'm comparing the value of the &lt;a href=&quot;http://en.wikipedia.org/wiki/Pearson%27s_chi-square_test&quot; rel=&quot;nofollow&quot;&gt;Pearson's $\chi^2$ test&lt;/a&gt; and a non parametric competitor &lt;a href=&quot;http://en.wikipedia.org/wiki/McNemar%27s_test&quot; rel=&quot;nofollow&quot;&gt;McNemar's $\chi^2$ test&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The answer brings an additional questions: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;they have matched each case in group1 with 4 controls in group2 (matched according to all the variables they deem important, such as sex, age, hemisf) except treatment. Setting aside the question of whether the matching is well done (i.e. whether all important variables have indeed been identified), doesn't the fact that to each case corresponds &lt;strong&gt;four&lt;/strong&gt; controls (i.e. not one) artificially inflate the significance of the results? (they still have to send me part of the data, this is why the table below does not reflect this 4 to 1 ratio).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I obtain the following (very different) results (n=116) (R CrossTable() function).:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;     [,1] [,2]&#xA;[1,]   39    9&#xA;[2,]   49   19&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Statistics for All Table Factors&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Pearson's Chi-squared test with Yates' continuity correction&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Chi^2 =  0.844691     d.f. =  1     p =  0.3580586 &lt;/p&gt;&#xA;&#xA;&lt;h2&gt;McNemar's Chi-squared test with continuity correction&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Chi^2 =  26.22414     d.f. =  1     p =  3.039988e-07 &lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Fisher's Exact Test for Count Data&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Sample estimate odds ratio:  1.672924 &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternative hypothesis: true odds ratio is not equal to 1&#xA;p =  0.279528 &#xA;95% confidence interval:  0.6356085 4.692326 &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The McNemar is the approximate version, but the exact version gives the same conclusions (strong rejection of the null). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is: how can i understand such a large difference between $\chi^2$ and McNemar ?&lt;/p&gt;&#xA;" OwnerUserId="603" LastEditorUserId="603" LastEditorDisplayName="" LastEditDate="2010-09-13T16:28:22.580" LastActivityDate="2010-09-13T16:28:22.580" Title="How to explain such a big difference between parametric and non parametric test (and other questions)?" Tags="&lt;hypothesis-testing&gt;&lt;nonparametric&gt;" AnswerCount="1" CommentCount="4" FavoriteCount="3" />
  <row Id="2614" PostTypeId="2" ParentId="372" CreationDate="2010-09-13T13:16:33.200" Score="1" ViewCount="" Body="&lt;p&gt;Understanding the causes of over-fitting, and how it can be avoided, is absoluetly  vital in datamining.  If you mine for statistical associations in data, you will always be able to find them (even if the data are purely random) but that doesn't mean they are of any predictive value.  The more you mine, the more likely you are to dig up a spurious rule.  I know that may sound rather obvious, but that doesn't mean that spurious rules never get presented to clients! ;o)&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2010-09-13T13:16:33.200" CommunityOwnedDate="2010-09-13T13:16:33.200" />
  <row Id="2615" PostTypeId="1" AcceptedAnswerId="2664" CreationDate="2010-09-13T13:35:11.777" Score="4" ViewCount="228" Body="&lt;p&gt;Suppose I have some initial correlation matrix. I want to stress each correlation in the matrix by the same constant simultaneously (except the diagonal; lets call this global parallel stress since it affects the entire matrix by the same constant at the same time). I start with adding 0.01% to each correlation and check if the matrix is still PSD. I continue increasing the stress levels by small increments. Eventually I encounter a stress level above which the matrix would no longer be PSD. Let's call this the upper stress boundary. I repeat the same procedure for negative stresses and find the lower stress boundary.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My general observation was that the minimum eigenvalue initial correlation matrix is roughly equal to the upper stress boundary. &lt;strong&gt;Is there an analytical solution for this problem?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Generally, I am more interested in the upper stress boundary. I stress the entire matrix at the same time because it is a simple way to reduce diversification benefit across all market variables. However, from a theoretical point of view it would be interesting to find out how to analytically calculate both lower and upper stress boundaries.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: please see a previous thread, which might be relevant here (some brilliant idea's from kwak): &lt;a href=&quot;http://stats.stackexchange.com/questions/2572/correlation-stress-testing&quot;&gt;http://stats.stackexchange.com/questions/2572/correlation-stress-testing&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1250" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-17T20:52:41.447" LastActivityDate="2010-09-20T01:50:20.380" Title="Analytical solutions to limits of correlation stress testing" Tags="&lt;correlation&gt;" AnswerCount="4" CommentCount="5" />
  <row Id="2616" PostTypeId="2" ParentId="485" CreationDate="2010-09-13T14:07:11.150" Score="0" ViewCount="" Body="&lt;p&gt;I just came across this website, &lt;a href=&quot;http://www.censusatschool.org.nz/2009/informal-inference/&quot; rel=&quot;nofollow&quot;&gt;CensusAtSchool -- Informal inference&lt;/a&gt;. Maybe worth looking at the videos and handouts...&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-09-13T14:07:11.150" CommunityOwnedDate="2010-09-13T14:07:11.150" />
  <row Id="2617" PostTypeId="1" CreationDate="2010-09-13T14:07:17.403" Score="4" ViewCount="645" Body="&lt;p&gt;I'm trying to take a normal distribution of points, and force them to become a uniform distribution. I've had little success on &lt;a href=&quot;http://stackoverflow.com/questions/3696747/generation-of-uniformly-distributed-random-noise&quot;&gt;S.O.&lt;/a&gt;, so I thought I'd ask here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically, I have a hash function which takes an &lt;code&gt;X&lt;/code&gt;, &lt;code&gt;Y&lt;/code&gt;, and seed value and generates numbers in a normal distribution with a mean of 0. These numbers vary only slightly from the numbers on all sides of them. When mapped out with the &lt;code&gt;X&lt;/code&gt; and &lt;code&gt;Y&lt;/code&gt; and the hashed value being &lt;code&gt;Z&lt;/code&gt;, it creates a terrain map. The map has some large peaks far past 1 and -1, but the middle 50% of the values lie between (-0.4,0.4). I'm trying to smooth this map out so that it retains its shape, but has a more-or-less uniform (rather than normal) distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As I said, the middle 50% of the values lie between (-0.4,0.4). The theoretical bounds for the hash function is (-2.25,2.25), though after generating a billion samples, the range of numbers found were about (-1.75,1.75).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think I need to take the above information to determine the best fit normal distribution, then use it to transform each value. As I found out on S.O., it's a really tough problem for me to explain. Hopefully someone here can at least point me in the right direction explaining it, or understands what I'm trying to do.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;The probability densities of my distribution, in blue, and &lt;code&gt;Normal(0,.72)&lt;/code&gt;, in red: [-3.8,3.8]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/AAGtA.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Their cumulative probabilities: [-3.8,3.8]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/7euTm.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;The probability density after inverse probability transform: [0,1]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/cJyta.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;And the cumulative probability compared to &lt;code&gt;Uniform&lt;/code&gt;: [0,1]&#xA;&lt;img src=&quot;http://i.imgur.com/CHiQo.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="1267" LastEditorUserId="1267" LastEditorDisplayName="" LastEditDate="2010-09-15T19:41:45.263" LastActivityDate="2010-09-16T05:33:58.557" Title="How can I determine the best fit normal distribution from this information?" Tags="&lt;distributions&gt;&lt;algorithms&gt;&lt;data-transformation&gt;" AnswerCount="5" CommentCount="14" FavoriteCount="2" />
  <row Id="2618" PostTypeId="2" ParentId="2613" CreationDate="2010-09-13T14:58:53.460" Score="9" ViewCount="" Body="&lt;p&gt;Because the null hypothesis your McNemar test tests, is not the same as the one tested by the $\chi^2$ test. The McNemar test actually tests whether the probability of 1-2 equals that of 2-1 (say first number is the row and second the column). If you switch columns, then you get a completely different outcome. The $\chi^2$ test just tests whether the frequencies can be calculated from the marginal frequencies, which means both categorical variables are independent. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To illustrate in R :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; x &amp;lt;- matrix(c(39,49,9,19),ncol=2)&#xA;&#xA;&amp;gt; y &amp;lt;-x[,2:1]&#xA;&#xA;&amp;gt; x&#xA;     [,1] [,2]&#xA;[1,]   39    9&#xA;[2,]   49   19&#xA;&#xA;&amp;gt; y&#xA;     [,1] [,2]&#xA;[1,]    9   39&#xA;[2,]   19   49&#xA;&#xA;&amp;gt; mcnemar.test(x)&#xA;&#xA;        McNemar's Chi-squared test with continuity correction&#xA;&#xA;data:  x &#xA;McNemar's chi-squared = 26.2241, df = 1, p-value = 3.04e-07&#xA;&#xA;&#xA;&amp;gt; mcnemar.test(y)&#xA;&#xA;        McNemar's Chi-squared test with continuity correction&#xA;&#xA;data:  y &#xA;McNemar's chi-squared = 6.2241, df = 1, p-value = 0.01260&#xA;&#xA;&#xA;&amp;gt; chisq.test(x)&#xA;&#xA;        Pearson's Chi-squared test with Yates' continuity correction&#xA;&#xA;data:  x &#xA;X-squared = 0.8447, df = 1, p-value = 0.3581&#xA;&#xA;&#xA;&amp;gt; chisq.test(y)&#xA;&#xA;        Pearson's Chi-squared test with Yates' continuity correction&#xA;&#xA;data:  y &#xA;X-squared = 0.8447, df = 1, p-value = 0.3581&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It is obvious the result of the McNemar test is completely different depending on which column comes first, whereas the $\chi^2$ test gives exactly the same outcome. Now why is this one non-significant? Well, take a look at the expected values :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; m1 &amp;lt;- margin.table(x,1)/116&#xA;&#xA;&amp;gt; m2 &amp;lt;- margin.table(x,2)/116&#xA;&#xA;&amp;gt; outer(m1,m2)*116&#xA;         [,1]     [,2]&#xA;[1,] 36.41379 11.58621&#xA;[2,] 51.58621 16.41379&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Pretty close to the table you have.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So both tests are not disagreeing at all. The $\chi^2$ rightfully concludes that both variables are independent, i.e. the counts in one variable are not influenced by the other and vice versa, and the McNemar test rightfully concludes that the probability of being first row-second column (0.07) is not the same as being second row-first column (0.42). &lt;/p&gt;&#xA;" OwnerUserId="1124" LastActivityDate="2010-09-13T14:58:53.460" />
  <row Id="2619" PostTypeId="1" AcceptedAnswerId="2622" CreationDate="2010-09-13T14:59:18.623" Score="3" ViewCount="964" Body="&lt;p&gt;I have three variables:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;distance (continuous, variable range negative infinity to positive infinity)&lt;/li&gt;&#xA;&lt;li&gt;isLand (discrete categorical/ Boolean, variable range 1 or 0)&lt;/li&gt;&#xA;&lt;li&gt;occupants (discrete categorical, variable range 0-7)  &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I want to answer the following statistical questions:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;How to I compare distributions that have both categorical and continuous variable. For example, I like to determine if the data distribution of distance vs occupants varies depending on the value of isLand.&lt;/li&gt;&#xA;&lt;li&gt;Given two of the three variables, can I predict the third using some equation?&lt;/li&gt;&#xA;&lt;li&gt;How can I determine independence with more than two variables?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="559" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-16T06:31:27.430" LastActivityDate="2010-09-16T06:31:27.430" Title="Continuous and Categorical variable data analysis" Tags="&lt;categorical-data&gt;&lt;continuous-data&gt;" AnswerCount="2" CommentCount="4" />
  <row Id="2620" PostTypeId="2" ParentId="2617" CreationDate="2010-09-13T15:12:28.100" Score="3" ViewCount="" Body="&lt;p&gt;The &lt;a href=&quot;http://en.wikipedia.org/wiki/Probability_integral_transform&quot; rel=&quot;nofollow&quot;&gt;inverse probability transform&lt;/a&gt; seems to be what you want.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To get a better uniform fit, the CDF $F(x)$ in the inverse probabiliy transform needs to be the actual distribution generating the data.  If this isn't actually normal, then that could account for the relatively poor fit.  You might try a &lt;a href=&quot;http://en.wikipedia.org/wiki/Qq_plot&quot; rel=&quot;nofollow&quot;&gt;Q-Q plot&lt;/a&gt; or &lt;a href=&quot;http://stats.stackexchange.com/questions/1645/appropriate-normality-tests-for-small-samples&quot;&gt;normality test&lt;/a&gt; first.  Or your data may very well be normal, just with some noise, in which case you can obtain a best fit.  I would suggest trying in order:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Standardization&quot; rel=&quot;nofollow&quot;&gt;Standardize&lt;/a&gt; your data to N(0, 1) and see if the transformation improves.&lt;/li&gt;&#xA;&lt;li&gt;After a normality test, &lt;a href=&quot;http://www.bigre.ulb.ac.be/Users/jvanheld/statistics_bioinformatics/practicals/microarray_fitting.html&quot; rel=&quot;nofollow&quot;&gt;fit a normal distribution&lt;/a&gt; to your data, then apply the transformation.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="251" LastEditorUserId="251" LastEditorDisplayName="" LastEditDate="2010-09-13T19:54:36.780" LastActivityDate="2010-09-13T19:54:36.780" CommentCount="2" />
  <row Id="2621" PostTypeId="2" ParentId="2573" CreationDate="2010-09-13T15:14:47.923" Score="2" ViewCount="" Body="&lt;p&gt;To clarify Dirks point :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say your data is a sample of a normal distribution. You could construct the following plot:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/i4NXE.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The red line is the empirical density estimate, the blue line is the theoretical pdf of the underlying normal distribution. Note that the histogram is expressed in densities and not in frequencies here. This is done for plotting purposes, in general frequencies are used in histograms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So to answer your question : you use the empirical distribution (i.e. the histogram) if you want to describe your sample, and the pdf if you want to describe the hypothesized underlying distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Plot is generated by following code in R :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x &amp;lt;- rnorm(100)&#xA;y &amp;lt;- seq(-4,4,length.out=200)&#xA;&#xA;hist(x,freq=F,ylim=c(0,0.5))&#xA;lines(density(x),col=&quot;red&quot;,lwd=2)&#xA;lines(y,dnorm(y),col=&quot;blue&quot;,lwd=2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1124" LastEditorUserId="1124" LastEditorDisplayName="" LastEditDate="2010-09-13T19:52:04.797" LastActivityDate="2010-09-13T19:52:04.797" CommentCount="1" />
  <row Id="2622" PostTypeId="2" ParentId="2619" CreationDate="2010-09-13T16:18:06.563" Score="1" ViewCount="" Body="&lt;p&gt;I would recommend reading about logistic or log-linear models in particular, and methods of categorical data analysis in general.  The notes on the following course are pretty good for a start: &lt;a href=&quot;http://www.stat.psu.edu/online/courses/stat504/&quot; rel=&quot;nofollow&quot;&gt;Analysis of Discrete Data&lt;/a&gt;.  The &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0471226181&quot; rel=&quot;nofollow&quot;&gt;textbook&lt;/a&gt; by Agresti is quite good.  You might also consider &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1441917411&quot; rel=&quot;nofollow&quot;&gt;Kleinbaum&lt;/a&gt; for a quick start.&lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-09-13T16:18:06.563" CommentCount="8" />
  <row Id="2623" PostTypeId="1" AcceptedAnswerId="2627" CreationDate="2010-09-13T16:56:42.960" Score="5" ViewCount="267" Body="&lt;p&gt;Does the autocorrelation function have any meaning with a non-stationary time series?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The time series is generally assumed to be stationary before autocorrelation is used for Box and Jenkins modeling purposes.&lt;/p&gt;&#xA;" OwnerUserId="284" LastEditorUserId="5" LastEditorDisplayName="" LastEditDate="2010-09-13T18:37:51.263" LastActivityDate="2010-09-13T20:00:10.513" Title="Autocorrelation in the presence of non-stationarity?" Tags="&lt;time-series&gt;&lt;autocorrelation&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="2624" PostTypeId="2" ParentId="2615" CreationDate="2010-09-13T17:27:13.523" Score="3" ViewCount="" Body="&lt;p&gt;This is not a complete answer but it was too long to fit in as a comment and hopefully gives you some ideas.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I am about to say is not really a proof but more of an intuitive sketch as to why the minimum eigenvalue may matter as far as the upper stress boundary is concerned.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Every correlation matrix can be decomposed into a set of eigenvectors and a corresponding set of eigenvalues. The eigenvectors correspond to the axis of the ellipsoid associated with the correlation matrix and the eigenvalues give us a 'sense' of the length of the axis associated with the corresponding eigenvector.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, when you stress a correlation matrix you are essentially performing one or more of the following two operations simultaneously: 1. You are rotating the eigenvectors and 2. You are changing the eigenvalues. Notice that rotating eigenvectors simply changes the orientation of the ellipsoid but if the eigenvalues remain positive then we have a psd correlation matrix. However, as eigenvalues drop the ellipsoid shrinks in the direction of the corresponding eigenvector and eventually as the eigenvalue reaches zero the ellipsoid collapses completely in that direction resulting in a matrix that is not psd.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you stress the correlation matrix there is less room along the axis that is shortest (i.e., the axis which has the lowest eigenvalue) to shrink and thus the minimum eigenvalue has a special role to play in stress testing. &lt;/p&gt;&#xA;" LastActivityDate="2010-09-13T18:23:25.273" />
  <row Id="2625" PostTypeId="2" ParentId="2617" CreationDate="2010-09-13T17:37:30.807" Score="4" ViewCount="" Body="&lt;p&gt;Interpreting your question as asking for the &quot;inverse probability transform,&quot; as ars has indicated and you have confirmed in comments to the question, gives a simple solution: perform a sort to rank all the $z$ values in ascending order from $1$ to $N$ (the amount of data), then convert each $z$ into $2 Rank(z)/(N+1) - 1$.  If you want to be really careful, scan over the ranked $z$'s to look for ties and assign each value within a group of tied values the average of their ranks.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-09-13T17:37:30.807" CommentCount="2" />
  <row Id="2626" PostTypeId="2" ParentId="2623" CreationDate="2010-09-13T17:44:20.733" Score="5" ViewCount="" Body="&lt;p&gt;In its alternative form as a variogram, the rate at which the function grows with large lags is roughly the square of the average trend.  This can sometimes be a useful way to decide whether you have adequately removed any trends.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(You can think of the variogram as the squared correlation multiplied by an appropriate variance and flipped upside down.)&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-09-13T17:44:20.733" />
  <row Id="2627" PostTypeId="2" ParentId="2623" CreationDate="2010-09-13T17:55:10.243" Score="7" ViewCount="" Body="&lt;p&gt;@whuber gave a nice answer.  I would just add, that you can simulate this very easily in R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;op &amp;lt;- par(mfrow = c(2,2), mar = .5 + c(0,0,0,0))&#xA;&#xA;N &amp;lt;- 500&#xA;# Simulate a Gaussian noise process&#xA;y1 &amp;lt;- rnorm(N)&#xA;# Turn it into integrated noise (a random walk)&#xA;y2 &amp;lt;- cumsum(y1)&#xA;&#xA;plot(ts(y1), xlab=&quot;&quot;, ylab=&quot;&quot;, main=&quot;&quot;, axes=F); box()&#xA;plot(ts(y2), xlab=&quot;&quot;, ylab=&quot;&quot;, main=&quot;&quot;, axes=F); box()&#xA;acf(y1, xlab=&quot;&quot;, ylab=&quot;&quot;, main=&quot;&quot;, axes=F); box()&#xA;acf(y2, xlab=&quot;&quot;, ylab=&quot;&quot;, main=&quot;&quot;, axes=F); box()&#xA;&#xA;par(op)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Which ends up looking somewhat like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/5UZLJ.jpg&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So you can easily see that the ACF function trails off slowly to zero in the case of a non-stationary series.  The rate of decline is some measure of the trend, as @whuber mentioned, although this isn't the best tool to use for that kind of analysis.&lt;/p&gt;&#xA;" OwnerUserId="5" LastEditorUserId="5" LastEditorDisplayName="" LastEditDate="2010-09-13T20:00:10.513" LastActivityDate="2010-09-13T20:00:10.513" CommentCount="1" />
  <row Id="2628" PostTypeId="1" AcceptedAnswerId="2629" CreationDate="2010-09-13T18:35:23.810" Score="8" ViewCount="541" Body="&lt;p&gt;Imagine you have to do reporting on the numbers of candidates who yearly take a given test. It seems rather difficult to infer the observed % of success, for instance, on a wider population due to the specifity of the target population. So you may consider that these data represent the whole population. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are results of tests indicating that the proportions of males and females are different really correct? Does a test comparing observed and theoretical proportions appear to be a correct one, since you consider a whole population (and not a sample)?&lt;/p&gt;&#xA;" OwnerUserId="1154" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-09-14T07:59:54.997" LastActivityDate="2010-09-14T18:15:20.813" Title="Statistical inference when the sample &quot;is&quot; the population" Tags="&lt;hypothesis-testing&gt;&lt;test&gt;&lt;population&gt;&lt;sampling&gt;" AnswerCount="4" CommentCount="1" />
  <row Id="2629" PostTypeId="2" ParentId="2628" CreationDate="2010-09-13T19:30:09.810" Score="11" ViewCount="" Body="&lt;p&gt;There may be varying opinions on this, but I would treat the population data as a sample and assume a hypothetical population, then make inferences in the usual way.  One way to think about this is that there is an underlying data generating process responsible for the collected data, the &quot;population&quot; distribution.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your particular case, this might make even more sense since you will have cohorts in the future.  Then your population is really cohorts who take the test even in the future.  In this way, you could account for time based variations if you have data for more than a year, or try to account for latent factors through your error model.  In short, you can develop richer models with greater explanatory power.&lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-09-13T19:30:09.810" CommentCount="4" />
  <row Id="2630" PostTypeId="2" ParentId="2628" CreationDate="2010-09-13T20:12:57.343" Score="5" ViewCount="" Body="&lt;p&gt;Actually, if you're really positive you have the whole population, there's even no need to go into statistics. Then you know exactly how big the difference is, and there is no reason whatsoever to test it any more. A classical mistake is using statistical significance as &quot;relevant&quot; significance. If you sampled the population, the difference is what it is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, if you reformulate your hypothesis, then the candidates can be seen as a sample of possible candidates, which would allow for statistical testing. In this case, you'd test in general whether male and female differ on the test at hand.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As ars said, you can use tests of multiple years and add time as a random factor. But if your interest really is in the differences between these candidates on this particular test, you cannot use the generalization and testing is senseless. &lt;/p&gt;&#xA;" OwnerUserId="1124" LastEditorUserId="1124" LastEditorDisplayName="" LastEditDate="2010-09-13T21:52:22.893" LastActivityDate="2010-09-13T21:52:22.893" />
  <row Id="2631" PostTypeId="1" AcceptedAnswerId="2633" CreationDate="2010-09-13T20:20:30.583" Score="4" ViewCount="91" Body="&lt;p&gt;This is a followup to &lt;a href=&quot;http://stats.stackexchange.com/q/2597/977&quot;&gt;this&lt;/a&gt; question. I am currently trying to implement the C-Index in order to find a near-optimal number of clusters from a hierarchy of clusters. I do this by calculating the C-Index for every step of the (agglomerative) hierarchical clustering. The problem is that the C-Index is minimal (0 to be exact) for very degenerated clusterings. Consider this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$c = \frac{S-S_{min}}{S_{max}-S_{min}}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this case $S$ is the sum of all distances between pairs of observations in the same cluster over all clusters. Let $n$ be the number of these pairs. $S_{min}$ and $S_{max}$ are the sums of $n$ lowest/highest distances across all pairs of observations. In the first step of the hierarchical clustering, the two closest observations (minimal distance) are merged into a cluster. Let $d$ be the distance between these observations. Now there is one pair of observations in the same cluster, so $n=1$ (all other clusters are singletons). Consequently $S=d$. The problem is that $S_{min}$ also equals $d$, because $d$ is the smallest distance (that is why the observations where merged first). So for this case, the C-Index is always 0. It stays 0 as long as only singleton clusters are merged. This means the optimal clustering according the C-Index would always consist of a bunch of clusters containing two observations, and the rest singletons. Does this mean that the C-Index is not applicable to hierarchical clustering? Am I doing something wrong? I have searched a lot, but could not find any suitable explanation. Can someone refer me to some resource that is freely available on the internet? Or, if not, at least a book I may try to get at my universities library?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance!&lt;/p&gt;&#xA;" OwnerUserId="977" LastActivityDate="2010-09-13T21:04:03.930" Title="Can someone explain the C-Index in the context of hierarchical clustering?" Tags="&lt;clustering&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="2632" PostTypeId="2" ParentId="2617" CreationDate="2010-09-13T20:53:17.000" Score="3" ViewCount="" Body="&lt;p&gt;OK, your hash function is generating numbers according to some distribution. You're thinking of it as normal, but it's really not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There's a simple way to convert between any univariate distribution and a uniform distribution. That is to use the cumulative distribution function CDF, which is a simple monotonic function, S-shaped, going from y = 0 to y = 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So to convert a number x from your hash distribution into a uniform number y, simply take y = CDF(x). To convert from uniform y to hash number x, just invert the CDF function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To get the CDF function, just make a table lookup. Generate a large number N of your hash function numbers, put them in a big array, and sort them in ascending order. That is your table. Then to calculate CDF(x), just lookup x in the table, by binary search. Then its index i in the table tells what y is, by y = i/N. (Actually, I'm cheating a bit. You will be more accurate if you do interpolation between two adjacent entries.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to invert the CDF function, just take your uniform number y, and get i = N*y. That gives you an index in the table, where you find x. Of course, you should interpolate, but if N is big enough, and you're not really fussy about accuracy, you don't really need to.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S. I'm glossing over some details, like what to do at the very ends of the table, or what if your table contains duplicate values, but this should get you going.&lt;/p&gt;&#xA;" OwnerUserId="1270" LastEditorUserId="1270" LastEditorDisplayName="" LastEditDate="2010-09-13T21:01:50.457" LastActivityDate="2010-09-13T21:01:50.457" CommentCount="3" />
  <row Id="2633" PostTypeId="2" ParentId="2631" CreationDate="2010-09-13T21:04:03.930" Score="2" ViewCount="" Body="&lt;p&gt;This may be one of the cases where there's more art than science to clustering.  I would suggest that you let your clustering algorithm run for a short time before letting the C-Index calculations kick in.  &quot;Short time&quot; may be after processing a few pairs, just when it starts to exceed 0, or some other heuristic.  (After all you don't expect to stop at 1 or 2 clusters, otherwise a different separation algorithm may have been deployed.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a book recommendation, I can suggest:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://books.google.com/books?id=htZzDGlCnQYC&quot; rel=&quot;nofollow&quot;&gt;Cluster Analysis&lt;/a&gt; by Brian Everitt, Sabine Landau, Morven Leese&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You can scan/search the available contents on google books to see if it might meet your needs.  It's worked as a reference for me in the past.&lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-09-13T21:04:03.930" CommentCount="1" />
  <row Id="2634" PostTypeId="2" ParentId="2619" CreationDate="2010-09-13T23:32:13.713" Score="0" ViewCount="" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;To examine the relationship between a continuous and categorical factor, a good start is to use side-by-side box plots, continuous on the left, categorical on the bottom. Are the means different? Use ANOVA to check. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;To examine the relationship between categorical factors, a good start is to use a mosaic plot, as well as a contingency table. You could group first then make separate plots.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;To predict occupants, ordinal logistic regression is probably the best way to go.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;To predict isLand, (binomial) logistic regression should do the trick.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;To predict distance, OLS regression will work. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="74" LastActivityDate="2010-09-13T23:32:13.713" />
  <row Id="2635" PostTypeId="1" CreationDate="2010-09-14T00:20:44.673" Score="0" ViewCount="436" Body="&lt;p&gt;I need to update the failure rate (given as deterministic) based on new rate of faiulre about the same system (it is a determinsitc one too). I read about the conjugate prior and Gamma distribution as a conjugate for Posission process. Also, I can equate the mean value of Gamma dist. (beat/alpha) to the new rate (as a mean value) but I do not have any other information such as st.dev, C.O.V, 90% percintile value,...etc. Is there a magic way to manuplate that and find parameters for the prior Gamma hence I get the posterior which Gamma too?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="1272" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-11-04T06:24:49.677" LastActivityDate="2010-11-04T10:14:48.267" Title="Conjugate prior for a Gamma distribution" Tags="&lt;bayesian&gt;" AnswerCount="2" CommentCount="3" FavoriteCount="1" />
  <row Id="2636" PostTypeId="2" ParentId="2547" CreationDate="2010-09-14T00:23:26.010" Score="1" ViewCount="" Body="&lt;p&gt;You're getting good answers here, but let me just add my 2 cents. I work in pharmacometrics, which deals in things like blood volume, elimination rate, base level of drug effect, maximum drug effect, and parameters like that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We make a distinction between variables that can take on any value plus or minus, versus values that can only be positive. An example of a variable that can take on any value, plus or minus, would be drug effect, which could be positive, zero, or negative. An example of a variable that can only realistically be positive is blood volume or drug elimination rate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We model these things with distributions that are typically either normal or lognormal, normal for the any-valued ones, and lognormal for the only-positive ones. A lognormal number is the number E taken to the power of a normally distributed number, and that is why it can only be positive.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a normally distributed variable, the median, mean, and mode are the same number, so it doesn't matter which you use. However, for a lognormally distributed variable, the mean is larger than both the median and the mode, so it is not really very useful. In fact, the median is where the underlying normal has its mean, so it is a much more attractive measure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since age (presumably) can never be negative, a lognormal distribution is probably a better description of it than normal, so median (E to the mean of the underlying normal) is more useful.&lt;/p&gt;&#xA;" OwnerUserId="1270" LastActivityDate="2010-09-14T00:23:26.010" CommentCount="3" />
  <row Id="2637" PostTypeId="2" ParentId="166" CreationDate="2010-09-14T00:55:16.510" Score="2" ViewCount="" Body="&lt;p&gt;As a rough generalization, any time you sample a fraction of the people in a population, you're going to get a different answer than if you sample the same number again (but possibly different people).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So if you want to find out how many people in Australia are &gt;= 30 years old, and if the true fraction (God told us) just happened to be precisely 0.4, and if we ask 100 people, the average number we can expect to say they are &gt;= 30 is 100 x 0.4 = 40, and the standard deviation of that number is +/- sqrt(100 * 0.4 * 0.6) = sqrt(24) ~ 4.9 or 4.9% (Binomial distribution).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since that square root is in there, when the sample size goes up by 100 times, the standard deviation goes up by 10 times. So in general, to reduce the uncertainty of a measurement like this by a factor of 10, you need to sample 100 times as many people. So if you ask 100 x 100 = 10000 people, the standard deviation would go up to 49 or, as a percent, down to 0.49%.&lt;/p&gt;&#xA;" OwnerUserId="1270" LastActivityDate="2010-09-14T00:55:16.510" />
  <row Id="2638" PostTypeId="2" ParentId="155" CreationDate="2010-09-14T01:02:43.727" Score="2" ViewCount="" Body="&lt;p&gt;Fun question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Someone found out I work in biostatistics, and they asked me (basically) &quot;Isn't statistics just a way of lying?&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Which brings back the Mark Twain quote about Lies, Damn Lies, and Statistics.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried to explain that statistics allows us to say with 100 percent precision that, given assumptions, and given data, that the probability of such-and-so was exactly such-and-such.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;She wasn't impressed.&lt;/p&gt;&#xA;" OwnerUserId="1270" LastEditorUserId="1270" LastEditorDisplayName="" LastEditDate="2010-09-14T18:11:45.067" LastActivityDate="2010-09-14T18:11:45.067" />
  <row Id="2639" PostTypeId="1" CreationDate="2010-09-14T01:11:32.053" Score="1" ViewCount="355" Body="&lt;p&gt;I have production (q) values from 4 different methods stored in the 4 matrices. Each of the 4 matrices contains q values from a different method as:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Matrix_1 = 1 row x 20 column &#xA;&#xA;Matrix_2 = 100 rows x 20 columns &#xA;&#xA;Matrix_3 = 100 rows x 20 columns &#xA;&#xA;Matrix_4 = 100 rows x 20 columns &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The number of columns indicate the number of years. 1 row would contain the production values corresponding to the 20 years. Other 99 rows for matrix 2, 3 and 4 are just the different realizations (or simulation runs). So basically the other 99 rows for matrix 2,3 and 4 are repeat cases (but not with exact values because of random numbers).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider &lt;code&gt;Matrix_1&lt;/code&gt; as the reference truth (or base case ). Now I want to compare the other 3 matrices  with &lt;code&gt;Matrix_1&lt;/code&gt; to see which one among those three matrices (each with 100 repeats) compares best, or closely imitates, with &lt;code&gt;Matrix_1&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can this be done in Matlab? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know, manually, that we use confidence interval (CI) by plotting the &lt;code&gt;mean of Matrix_1&lt;/code&gt;, and drawing each distribution of &lt;code&gt;mean of Matrix_2&lt;/code&gt;, &lt;code&gt;mean of Matrix_3&lt;/code&gt; and &lt;code&gt;mean of Matrix_4&lt;/code&gt;. The largest CI among matrix 2, 3 and 4 which contains the reference truth (or &lt;code&gt;mean of Matrix_1&lt;/code&gt;) will be the answer. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mean of Matrix_1 = (1 row x 1 column)&#xA;&#xA;mean of Matrix_2 = (100 rows x 1 column)&#xA;&#xA;mean of Matrix_3 = (100 rows x 1 column)&#xA;&#xA;mean of Matrix_4 = (100 rows x 1 column)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I hope the question is clear and relevant to SA. Otherwise please feel free to edit/suggest anything in question. Thanks!&lt;/p&gt;&#xA;" OwnerUserId="1102" LastEditorUserId="1102" LastEditorDisplayName="" LastEditDate="2010-09-28T19:05:54.930" LastActivityDate="2010-09-28T19:19:41.043" Title="How to compare different distributions with reference truth value in Matlab?" Tags="&lt;distributions&gt;&lt;confidence-interval&gt;&lt;multiple-comparisons&gt;&lt;matlab&gt;" AnswerCount="2" CommentCount="6" FavoriteCount="0" />
  <row Id="2641" PostTypeId="1" AcceptedAnswerId="2647" CreationDate="2010-09-14T03:24:01.207" Score="14" ViewCount="1217" Body="&lt;p&gt;The &lt;a href=&quot;http://en.wikipedia.org/wiki/Likelihood_function&quot; rel=&quot;nofollow&quot;&gt;wikipedia page&lt;/a&gt; claims that likelihood and probability are distinct concepts.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In non-technical parlance, &quot;likelihood&quot; is usually a synonym for &quot;probability,&quot; but in statistical usage there is a clear distinction in perspective: the number that is the probability of some observed outcomes given a set of parameter values is regarded as the likelihood of the set of parameter values given the observed outcomes. &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Can someone give a more down-to-earth description of what this means?  In addition, some examples of how &quot;probability&quot; and &quot;likelihood&quot; disagree would be nice.&lt;/p&gt;&#xA;" OwnerUserId="386" LastActivityDate="2010-09-15T00:21:43.453" Title="What is the difference between &quot;likelihood&quot; and &quot;probability&quot;?" Tags="&lt;probability&gt;&lt;likelihood-function&gt;" AnswerCount="6" CommentCount="3" FavoriteCount="7" />
  <row Id="2642" PostTypeId="2" ParentId="2641" CreationDate="2010-09-14T03:44:11.397" Score="3" ViewCount="" Body="&lt;p&gt;If I have a fair coin (parameter value) then the probability that it will come up heads is 0.5.  If I flip a coin 100 times and it comes up heads 52 times then it has a high likelihood of being fair (the numeric value of likelihood potentially taking a number of forms).&lt;/p&gt;&#xA;" OwnerUserId="601" LastActivityDate="2010-09-14T03:44:11.397" />
  <row Id="2644" PostTypeId="2" ParentId="2617" CreationDate="2010-09-14T04:40:27.327" Score="1" ViewCount="" Body="&lt;p&gt;Why does your transformation need to be a smooth deformation? Just take the binary representation of each number in 64-bit two's complement fixed point, or 64-bit IEEE floating point, and throw that into SHA-1. Boom, instant hash with uniform distribution of the outputs (assuming no duplicates).&lt;/p&gt;&#xA;" OwnerUserId="1122" LastActivityDate="2010-09-14T04:40:27.327" CommentCount="1" />
  <row Id="2645" PostTypeId="2" ParentId="2641" CreationDate="2010-09-14T05:16:17.883" Score="10" ViewCount="" Body="&lt;p&gt;I'll give you the perspective from the view of Likelihood Theory which originated with &lt;a href=&quot;http://en.wikipedia.org/wiki/Ronald_Fisher&quot; rel=&quot;nofollow&quot;&gt;Fisher&lt;/a&gt; -- and is the basis for the statistical definition in the cited Wikipedia article.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose you have random variates $X$ which arise from a parameterized distribution $F(X; \theta)$, where $\theta$ is the parameter characterizing $F$.  Then the probability of $X = x$ would be: $P(X = x) = F(x; \theta)$, with known $\theta$.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;More often, you have data $X$ and $\theta$ is unknown.  Given the assumed model $F$, the likelihood is defined as the probability of observed data as a function of $\theta$: $L(\theta) = P(\theta; X = x)$.  Note that $X$ is known, but $\theta$ is unknown; in fact the motivation for defining the likelihood is to determine the parameter of the distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Although it seems like we have simply re-written the probability function, a key consequences of this is that the likelihood function does &lt;em&gt;not&lt;/em&gt; obey the laws of probability (for example, it's not bound to the [0, 1] interval).  However, the likelihood function is proportional to the probabiilty of the observed data.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This concept of likelihood actually leads to a different school of thought, &quot;likelihoodists&quot; (distinct from frequentist and bayesian) and you can google to search for all the various historical debates.  The cornerstone is the &lt;a href=&quot;http://en.wikipedia.org/wiki/Likelihood_principle&quot; rel=&quot;nofollow&quot;&gt;Likelihood Principle&lt;/a&gt; which essentially says that we can perform inference directly from the likelihood function (neither Bayesians nor frequentists accept this since it is not probability based inference).  These days a lot of what is taught as &quot;frequentist&quot; in schools is actually an amalgam of frequentist and likelihood thinking.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For deeper insight, a nice start and historical reference is Edwards' &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0801844436&quot; rel=&quot;nofollow&quot;&gt;Likelihood&lt;/a&gt;.  For a modern take, I'd recommend Richard Royall's wonderful monograph, &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0412044110&quot; rel=&quot;nofollow&quot;&gt;Statistical Evidence: A Likelihood Paradigm&lt;/a&gt;.  &lt;/p&gt;&#xA;" OwnerUserId="251" LastEditorUserId="251" LastEditorDisplayName="" LastEditDate="2010-09-14T07:10:18.027" LastActivityDate="2010-09-14T07:10:18.027" CommentCount="1" />
  <row Id="2646" PostTypeId="2" ParentId="2641" CreationDate="2010-09-14T06:04:27.093" Score="1" ViewCount="" Body="&lt;p&gt;Suppose you have a coin with probability $p$ to land heads and $(1-p)$ to land tails. Let $x=1$ indicate heads and $x=0$ indicate tails. Define $f$ as follows&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$f(x,p)=p^x (1-p)^{1-x}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$f(x,2/3)$ is probability of x given $p=0.5$, $f(1,p)$ is likelihood of $p$ given $x=1$. Basically likelihood vs. probability tells you which parameter of density is considered to be the variable&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://yaroslavvb.com/upload/likelihood-vs-probability.png&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="511" LastActivityDate="2010-09-14T06:04:27.093" />
  <row Id="2647" PostTypeId="2" ParentId="2641" CreationDate="2010-09-14T06:08:02.933" Score="7" ViewCount="" Body="&lt;p&gt;The answer depends on whether you are dealing with discrete or continuous random variables. So, I will split my answer accordingly. I will assume that you want some technical details and not necessarily an explanation in plain english. If my assumption is not correct please let me know and I will revise my answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Discrete Random Variables&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose that you have a stochastic process that takes discrete values (e.g., outcomes of tossing a coin 10 times, number of customers who arrive at a store in 10 minutes etc). In such cases, we can calculate the probability of observing a particular set of outcomes by making suitable assumptions about the underlying stochastic process (e.g., probability of coin landing heads is $p$ and that coin tosses are independent).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Denote the observed outcomes by $O$ and the set of parameters that describe the stochastic process as $\theta$. Thus, when we speak of probability we want to calculate $P(O|\theta)$. In other words, given specific values for $\theta$, $P(O|\theta)$ is the probability that we would observe the outcomes represented by $O$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, when we model a real life stochastic process, we often do not know $\theta$. We simply observe $O$ and the goal then is to arrive at an estimate for $\theta$ that would be a plausible choice given the observed outcomes $O$. We know that given a value of $\theta$ the probability of observing $O$ is $P(O|\theta)$. Thus, a 'natural' estimation process is to choose that value of $\theta$ that would maximize the probability that we would actually observe $O$. In other words, we find the parameter values $\theta$ that maximize the following function:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$L(\theta|O) = P(O|\theta)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$L(\theta|O)$ is called as the likelihood function. Notice that by definition the likelihood function is conditioned on the observed $\theta$ and that it is a function of the unknown parameters $\theta$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Continuous Random Variables&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the continuous case the situation is similar with one important difference. We can no longer talk about the probability that we observed $O$ given $\theta$ as in the continuous case $P(O|\theta) = 0$. Without getting into technicalities, the basic idea is as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Denote the probability density function (pdf) associated with the outcomes $O$ as: $f(O|\theta)$. Thus, in the continuous case we estimate $\theta$ given observed outcomes $O$ by maximizing the following function:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$L(\theta|O) = f(O|\theta)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this situation, we cannot technically assert that we are finding the parameter value that maximizes the probability that we observe $O$ as we maximize the pdf associated with the observed outcomes $O$.&lt;/p&gt;&#xA;" LastActivityDate="2010-09-14T06:08:02.933" CommentCount="4" />
  <row Id="2648" PostTypeId="1" CreationDate="2010-09-14T08:43:15.420" Score="5" ViewCount="136" Body="&lt;p&gt;Imagine&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;having a dependent variable $Y$ that is a proportion (i.e., the proportion of observations made at the given time point that satisfy a condition, where each time point involves 50 to 250 observations)&lt;/li&gt;&#xA;&lt;li&gt;$Y$ is measured at a series of time time points $X$, where $X = 1, 2, 3, ...$, typically to around 400.&lt;/li&gt;&#xA;&lt;li&gt;At initial time points, $Y$ typically equals zero or close to zero&lt;/li&gt;&#xA;&lt;li&gt;After an extended period of time $Y$ typically equals one or close to one&lt;/li&gt;&#xA;&lt;li&gt;At some point in between a transition occurs where values of $Y$ increase&lt;/li&gt;&#xA;&lt;li&gt;Throughout there is considerable time point to time point variability and given that Y is a proportion, the distribution of errors is not normal. Note also that the values of zero and one are common.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Properties of the data vary across studies, such as:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;the initial value of $Y$&lt;/li&gt;&#xA;&lt;li&gt;the time point when the value of $Y$ starts to increase&lt;/li&gt;&#xA;&lt;li&gt;the duration of transition from values mainly around 0 to mainly around 1&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Questions&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What would be a good modelling approach to such data?&lt;/li&gt;&#xA;&lt;li&gt;How could the onset of the transition from values close to zero to values close to one be detected, especially given the non-normal errors?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-09-14T11:02:10.090" LastActivityDate="2010-09-14T15:28:33.867" Title="Predicting proportions from time with a discontinuity" Tags="&lt;time-series&gt;&lt;proportion&gt;&lt;discontinuity&gt;" AnswerCount="3" CommentCount="3" FavoriteCount="1" />
  <row Id="2649" PostTypeId="2" ParentId="2641" CreationDate="2010-09-14T08:45:32.590" Score="3" ViewCount="" Body="&lt;p&gt;I'll try and minimise the mathematics in my explanation as there are some good mathematical explanations already.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As Robin Girand points out the difference between probability and likelihood is closely related to the &lt;a href=&quot;http://stats.stackexchange.com/questions/665/whats-the-difference-between-probability-and-statistics/675&quot;&gt;difference between probability and statistics&lt;/a&gt;. In a sense probability and statistics concern themselves with problems that are opposite or inverse to one another. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider a coin toss. (My answer will be similar to &lt;a href=&quot;http://en.wikipedia.org/wiki/Likelihood_function#Example_1&quot; rel=&quot;nofollow&quot;&gt;Example 1 on Wikipedia&lt;/a&gt;.) If we know the coin is fair (p=0.5) a typical probability question is: What is the probability of getting two heads in a row. The answer is P(HH) = P(H)*P(H) = 0.5*0.5 = 0.25.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A typical statistical question is: Is the coin fair? To answer this we need to ask: To what extent does our sample support the our hypothesis that P(H) = P(T) = 0.5? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first point to note is that the direction of the question has reversed. In probability we start with an assumed parameter (P(head)) and estimate the probability of a given sample (two heads in a row). In statistics we start with the observation (two heads in a row) and make INFERENCE about our parameter (p = P(H) = 1- P(T) = 1 - q).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/File%3aLikelihoodFunctionAfterHH.png&quot; rel=&quot;nofollow&quot;&gt;Example 1 on Wikipedia&lt;/a&gt; shows us that the maximum likelihood estimate of P(H) after 2 heads in a row is p_MLE = 1. But the data in no way rule out the the true parameter value p(H) = 0.5 (let's not concern ourselves with the details at the moment). Indeed only very small values of p(H) and particularly p(H)=0 can be reasonably eliminated after n = 2 (two throws of the coin). After the &lt;a href=&quot;http://en.wikipedia.org/wiki/File%3aLikelihoodFunctionAfterHHT.png&quot; rel=&quot;nofollow&quot;&gt;third throw&lt;/a&gt; we can now eliminate the possibility that P(H) = 1.0 (i.e. it is not a two-headed coin), but most values in between can be &lt;strong&gt;reasonably supported by the data&lt;/strong&gt;. (An exact binomial 95% confidence interval for p(H) is 0.094 to 0.992.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After 100 coin tosses and (say) 70 heads, we now have a reasonable basis for the suspicion that the coin is not in fact fair. An exact 95% CI on p(H) is now 0.600 to 0.787 and the probability of observing a result as extreme as  70 or more heads (or tails) from 100 tosses given p(H) = 0.5 is 0.0000785.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Although I have not explicitly used likelihood calculations this example captures the concept of likelihood: &lt;strong&gt;Likelihood is a measure of the extent to which a sample provides support for particular values of a parameter in a parametric model&lt;/strong&gt;. &lt;/p&gt;&#xA;" OwnerUserId="521" LastEditorUserId="521" LastEditorDisplayName="" LastEditDate="2010-09-15T00:21:43.453" LastActivityDate="2010-09-15T00:21:43.453" CommentCount="3" />
  <row Id="2650" PostTypeId="1" AcceptedAnswerId="2651" CreationDate="2010-09-14T09:05:22.790" Score="2" ViewCount="278" Body="&lt;p&gt;Good day,&lt;br&gt;&#xA;I'm working with a panel dataset, I've used many models,&#xA;homogeneous (fixed effect, pooled ols and Driscoll and Kraay)&#xA;heterogeneous (swamy random coefficients)  and  would like to do a&#xA;post-estimation to select the model that best fit  my regression.&#xA;Is there any method, command that may allow me to do this?&#xA;Any hint will be highly appreciated.&lt;br&gt;&#xA;Ama&lt;/p&gt;&#xA;" OwnerUserId="1251" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-14T13:04:19.760" LastActivityDate="2010-09-14T13:04:19.760" Title="Panel data and selection models issue" Tags="&lt;model-selection&gt;&lt;econometrics&gt;&lt;panel-data&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="2651" PostTypeId="2" ParentId="2650" CreationDate="2010-09-14T10:02:54.613" Score="1" ViewCount="" Body="&lt;p&gt;According to my experience, model selection is not so much about seeking the model that best fits the regression. The first question to ask is : which model reflects my experimental design the best? The second question would then be : which model reflects the covariance structure in my data the best?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Only then you can start worrying about which model fits the data the best. There are a number of approaches to evaluate and comppare non-nested models. A naive approach would be to do a cross-validation and compare those results. One could eg do a leave-one-out crossvalidation, get the SS values for each run, and treat these SS values as sample data for model comparison. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There's a whole set of literature on comparing non-nested models, but that's always within the same framework. There's little possibilities for comparing models originating from different frameworks other than working with some general loss function.&lt;/p&gt;&#xA;" OwnerUserId="1124" LastActivityDate="2010-09-14T10:02:54.613" CommentCount="4" />
  <row Id="2652" PostTypeId="2" ParentId="2648" CreationDate="2010-09-14T11:59:07.703" Score="2" ViewCount="" Body="&lt;p&gt;Ignoring the &quot;change point&quot; your description suggests to me a (non-linear) mixed effects model of the following form:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;g(E(Yi)) = Xi*beta + Zi*U&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Where The betas are fixed effects, the U's are random effects, g(E(yi)) is the (logit) link to a binomial mean.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This will deal with logitudinal correlation of data and the non-Guassian distribution issues.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This must be coupled with some form of change point model, probably a Hidden Markov Model (HMM). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Hidden_markov_model&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Hidden_markov_model&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It may be necessary to set-up the model as a Directed Acyclic Graph (DAG) in MCMC format, or even specify it fully in a Bayesian framework using software such as WinBUGS.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Directed_Acyclic_Graph&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Directed_Acyclic_Graph&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/MCMC&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/MCMC&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/WinBUGS&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/WinBUGS&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="521" LastEditorUserId="521" LastEditorDisplayName="" LastEditDate="2010-09-14T12:07:16.107" LastActivityDate="2010-09-14T12:07:16.107" CommentCount="1" />
  <row Id="2653" PostTypeId="2" ParentId="2579" CreationDate="2010-09-14T13:02:36.380" Score="4" ViewCount="" Body="&lt;p&gt;I think you use Stata, given your other post about &lt;a href=&quot;http://stats.stackexchange.com/questions/2650/panel-data-and-selection-models-issue&quot;&gt;Panel data and selection models issue&lt;/a&gt;. Did you look at the following paper, &lt;a href=&quot;http://www.stata-journal.com/sjpdf.html?articlenum=st0046&quot; rel=&quot;nofollow&quot;&gt;From the help desk: Swamy’s random-coefficients model&lt;/a&gt; from the Stata Journal (2003 3(3))? It seems that the command &lt;code&gt;xtrchh2&lt;/code&gt; (available through &lt;code&gt;findit xtrchh&lt;/code&gt; in Stata command line) includes an option about time, but I'm afraid it only allow to estimate the panel-specific coefficients. Looking around, I only found this article, &lt;a href=&quot;http://www.stata-journal.com/article.html?article=st0084&quot; rel=&quot;nofollow&quot;&gt;Estimation and testing of fixed-effect panel-data systems&lt;/a&gt; (SJ 2005 5(2)), but it doesn't seem to address your question.&#xA;So maybe it is better to use the &lt;a href=&quot;http://www.stata.com/help.cgi?xtreg&quot; rel=&quot;nofollow&quot;&gt;xtreg&lt;/a&gt; command directly.&#xA;If you have more than one random coefficient, then it may be better to &lt;a href=&quot;http://www.gllamm.org/&quot; rel=&quot;nofollow&quot;&gt;gllamm&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Otherwise, I would suggest trying the &lt;a href=&quot;http://cran.r-project.org/web/packages/plm/index.html&quot; rel=&quot;nofollow&quot;&gt;plm&lt;/a&gt; R package (it has a lot of dependencies, but it mainly relies on the &lt;code&gt;nlme&lt;/code&gt; and &lt;code&gt;survival&lt;/code&gt; packages). The &lt;code&gt;effect&lt;/code&gt; parameter that is passed to &lt;code&gt;plm()&lt;/code&gt; seems to return individual, time or both (for balanced design) kind of effects; there's also a function names &lt;code&gt;plstest()&lt;/code&gt;. I'm not a specialist of econometrics, I only used it for clinical trials in the past, but quoting the online help, it seems you will be able to get fixed effects for your time covariate (expressed as deviations from the overall mean or as deviations from the first value of the index):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(plm)&#xA;data(&quot;Grunfeld&quot;, package = &quot;plm&quot;)&#xA;gi &amp;lt;- plm(inv ~ value + capital, data = Grunfeld,&#xA;          model = &quot;within&quot;, effect = &quot;twoways&quot;)&#xA;summary(gi)&#xA;fixef(gi,effect = &quot;time&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where the data looks like (or see the plot below to get a rough idea):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    firm year   inv  value capital&#xA;  1    1 1935 317.6 3078.5     2.8&#xA;  2    1 1936 391.8 4661.7    52.6&#xA;  3    1 1937 410.6 5387.1   156.9&#xA;...&#xA;198   10 1952  6.00  74.42    9.93&#xA;199   10 1953  6.53  63.51   11.68&#xA;200   10 1954  5.12  58.12   14.33&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For more information, check the accompagnying vignette or this paper, &lt;a href=&quot;http://www.jstatsoft.org/v27/i02/paper&quot; rel=&quot;nofollow&quot;&gt;Panel Data Econometrics in R: The plm Package&lt;/a&gt;, published in the JSS (2008 27(2)).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/6V8wi.png&quot; alt=&quot;Grunfeld&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-14T13:28:24.557" LastActivityDate="2010-09-14T13:28:24.557" CommentCount="1" />
  <row Id="2654" PostTypeId="2" ParentId="2648" CreationDate="2010-09-14T13:05:45.937" Score="3" ViewCount="" Body="&lt;p&gt;Sounds to me that Y(X) is a &lt;a href=&quot;http://en.wikipedia.org/wiki/Sigmoid_function&quot; rel=&quot;nofollow&quot;&gt;sigmiodal&lt;/a&gt; process. Thus logistic regression should be suitable for this data. If you model this in R with:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;glm(Y~X,family=binomial)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;you will find that the &quot;sharpness&quot; of the transition is determined by the magnitude of the X coefficient, and the point of transition (technically the mid-point) is at the ratio of the intercept coefficient to the X coefficient times -1. I made an image to illustrate this but cannot seem to upload it for some reason.&lt;/p&gt;&#xA;" OwnerUserId="229" LastActivityDate="2010-09-14T13:05:45.937" CommentCount="5" />
  <row Id="2657" PostTypeId="2" ParentId="2628" CreationDate="2010-09-14T14:58:42.770" Score="0" ViewCount="" Body="&lt;p&gt;If you consider whatever it is that you are measuring to be a random process, then yes statistical tests are relevant. Take for example, flipping a coin 10 times to see if it is fair. You get 6 heads and 4 tails - what do you conclude?&lt;/p&gt;&#xA;" OwnerUserId="229" LastActivityDate="2010-09-14T14:58:42.770" CommentCount="2" />
  <row Id="2658" PostTypeId="2" ParentId="2648" CreationDate="2010-09-14T15:28:33.867" Score="2" ViewCount="" Body="&lt;p&gt;James' approach looks good: each observation, according to your description, might have a Binomial(n[i], p[i]) distribution where n[i] is known and--to be fully general--p[i] is a completely unknown function that rises from near 0 to near 1 as i increases.  A logistic regression (GLM with binomial response and logistic link) against X[i]==i alone as the explanatory variable might even work.  If it's a poor fit, you can introduce additional terms, such as higher powers or (better yet, given the nonparametric spirit) splines.  This readily allows for incorporating any covariates into the model, too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In effect, what appears to be an abrupt change in the response might really just be a natural linear (or nearly linear) progression of logit(p) on which is superimposed Binomial variability.  It is this possibility that leads me slightly away from the direction indicated by Thylacoleo, whose approach clearly is valid and likely to be effective.  I'm just suspecting (hoping?) that your situation might be amenable to this somewhat simpler analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A complicating possibility concerns the possible autocorrelation of the responses, but that would need to be investigated only if the logistic residuals look strongly over- or under-dispersed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a matter of EDA, you could smooth successive observations in a natural way and plot their &lt;em&gt;logits&lt;/em&gt; against i.  For instance, to smooth observations y[i] and y[i+1] you would construct (n[i]*y[i] + n[i+1]*y[i+1])/(n[i] + n[i+1]), effectively pooling two successive batches; longer smoothing windows can be constructed the same way.  (This would automatically cancel out any negative short-term temporal correlation, too.)  The fit to the smooth wouldn't be quite right--it would be less steep than appropriate--but it would suggest choices for the general form of the covariates (i.e., functions of i) to use in the regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is, of course, only one of many possible models.  For example, the observations at each time point might be a binomial mixture, which would allow both for overdispersion and another way of getting nonlinear fits on the logit scale.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-09-14T15:28:33.867" />
  <row Id="2659" PostTypeId="2" ParentId="2641" CreationDate="2010-09-14T15:45:09.347" Score="4" ViewCount="" Body="&lt;p&gt;This is the kind of question that just about everybody is going to answer and I would expect all the answers to be good.  But you're a mathematician, Douglas, so let me offer a mathematical reply.  A statistical model has to connect two distinct conceptual entities: &lt;strong&gt;data&lt;/strong&gt;, which are elements $x$ of some set (such as a vector space), and a possible quantitative &lt;strong&gt;model&lt;/strong&gt; of the data behavior.  Models are usually represented by points $\theta$ on a finite dimensional manifold, a manifold with boundary, or a function space (the latter is termed a &quot;non-parametric&quot; problem).  The data $x$ are connected to the possible models $\theta$ by means of a function $\Lambda(x, \theta)$.  For any given $\theta$, $\Lambda(x, \theta)$ is intended to be the probability (or probability density) of $x$.  For any given $x$, on the other hand, $\Lambda(x, \theta)$ can be viewed as a function of $\theta$ and is usually assumed to have certain nice properties, such as being continuously second differentiable.  The intention to view $\Lambda$ in this way and to invoke these assumptions is announced by calling $\Lambda$ the &quot;likelihood.&quot;  It's quite like the distinction between variables and parameters in a differential equation: sometimes we want to study the solution (i.e., we focus on the variables as the argument) and sometimes we want to study how the solution varies with the parameters.  The main distinction is that in statistics we rarely need to study the simultaneous variation of both sets of arguments; there is no statistical object that naturally corresponds to changing both the data $x$ and the model parameters $\theta$.  That's why you hear more about this dichotomy than you would in analogous mathematical settings.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-09-14T15:45:09.347" />
  <row Id="2660" PostTypeId="2" ParentId="2639" CreationDate="2010-09-14T16:43:26.823" Score="1" ViewCount="" Body="&lt;p&gt;This is two questions; (1) has good standard explanations available anywhere, so I'll address (2).  You can compare distributions using a Kolmogorov-Smirnov statistic: it is the maximum difference between their empirical distribution functions.  (The value of the edf at a number $x$ is the proportion of data less than or equal to $x$.)  Its distribution depends on the base distribution and on the size of the sample distribution, so in this case about the only option is to find the distribution of the KS statistic through simulation: draw several thousand (at least) samples of size 100 from the reference distribution in a way that emulates how batches B, C, and D were originally created.  Compute the KS statistic for each sample: this is your simulation, which you can summarize with a histogram.  Also compute the KS statistic for B, C, and D.  Where they are situated with respect to the simulation results informs you about how close B, C, and D are to the reference.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;NB: This answer responds to the question as originally posed:  &quot;1) Explain the concept of CI in brief.  2) How do I use CI to find which one among &lt;code&gt;B, C or D&lt;/code&gt; imitates or is close to the &lt;code&gt;A&lt;/code&gt;?&quot;&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-09-28T19:19:41.043" LastActivityDate="2010-09-28T19:19:41.043" CommentCount="2" />
  <row Id="2661" PostTypeId="1" AcceptedAnswerId="2665" CreationDate="2010-09-14T17:18:10.680" Score="4" ViewCount="934" Body="&lt;p&gt;I am doing some basic fitting of data and exploring different fits. I understand that the residual is the difference between the sample and the estimated function value. The norm of the residuals is a measure of the deviation between the correlation and the data.  A lower norm signifies a better fit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose a cubic fit has a norm of residuals of 0.85655 and a linear fit has a norm of residuals of 0.89182. What if the norms are 0.17113 and 0.24916? Is the difference between these two significant? Is a norm of residual less than 1 considered good? If not what is generally regarded as &quot;acceptable&quot; norm of residual. &lt;/p&gt;&#xA;" OwnerUserId="559" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-14T17:12:45.643" LastActivityDate="2010-10-14T17:12:45.643" Title="Difference between Norm of Residuals and what is a &quot;good&quot; Norm of Residual" Tags="&lt;regression&gt;&lt;residuals&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="1" />
  <row Id="2662" PostTypeId="1" CreationDate="2010-09-14T18:13:16.813" Score="0" ViewCount="181" Body="&lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have asked the mods to lock this thread. I will post a question which will be a poll on what specific project question we want to investigate as part of the first project.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;End of update&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[&lt;strong&gt;Important Meta Note&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please do not vote-up this question as this question is a housekeeping type of question designed to move the polystats project to the next phase of the project. However, do vote up answers depending on their quality. I will add dummy edits at some regular frequency to push this question to the top from time to time. You may also wish to see this meta thread on the issue of &lt;a href=&quot;http://meta.stats.stackexchange.com/q/428/28&quot;&gt;rep and polystats project&lt;/a&gt;.]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The consensus (based on the votes/answers) to the first question &lt;a href=&quot;http://stats.stackexchange.com/q/2512/28&quot;&gt;What should be our first Polystats Project&lt;/a&gt; seems to be indicate that our first project would use Stackexchange data to answer several type of questions. The following summarizes the project ideas of that thread:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Are there reputation effects in SE communities and to what extent are reputation effects contingent on various variables such as size of the community, number of views per day, no of questions per day etc?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Quantify the quality of answers and questions (what makes a question/answer a good one?)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Classify how the communities differ.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Is there any evidence for herd behavior? (i.e., Is the probability of a question/answer being voted up/down dependent on the existing up/down totals?)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The above is my interpretation of the suggestions made in that thread. While we can debate the merits of the above questions on various dimensions (e.g., which ones are interesting, doable etc), I would like the focus of this question to be on our data collection strategy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently, there exist three main SE sites (Stackoverflow, Superuser and ServerFault) and a whole bunch of beta sites including our site (I am not counting meta.SE and SE 1.0 sites). It seems to me that the beta sites are not well established enough for us to answer any of the questions we have in a reasonable way. Thus, I propose that we restrict attention to Stackoverflow, Superuser and Server Fault.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the amount of data that is available on these sites is staggering. The following is a snapshot of the current statistics for these three sites:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/pD8jR.pngBlockquote&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Clearly, we cannot download all the data to work with and need some form of sampling strategy. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given the above background, my questions are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Do you agree with the restriction to Stackoverflow, Superuser and Serverfault?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;What data sampling strategy would you recommend that we follow?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Do you think that all of us should work with the same dataset (to ensure that we can reproduce each other's results/bugs)? If so, where/how should we store it?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;What variables do you like to see in the data dump? The following shows the data that we can potentially download using the API/data-dump provided by SE. Table names are in capital letters with corresponding fields below the table name.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;blockquote&gt;&#xA;    &lt;p&gt;&#xA;    POSTS&#xA;    Id&#xA;    PostTypeId&#xA;    AcceptedAnswerId&#xA;    CreationDate&#xA;    Score&#xA;    ViewCount&#xA;    Body&#xA;    OwnerUserId&#xA;    LastEditorUserId&#xA;    LastEditDate&#xA;    LastActivityDate&#xA;    Title&#xA;    Tags&#xA;    AnswerCount&#xA;    CommentCount&#xA;    FavoriteCount&#xA;    ClosedDate&#xA;    ParentId&#xA;    CommunityOwnedDate&#xA;    USERS&#xA;    Id&#xA;    Reputation&#xA;    EmailHash&#xA;    CreationDate&#xA;    DisplayName&#xA;    LastAccessDate&#xA;    WebsiteUrl&#xA;    Location&#xA;    Age&#xA;    AboutMe&#xA;    Views&#xA;    UpVotes&#xA;    DownVotes&#xA;    COMMENTS&#xA;    Id&#xA;    PostId&#xA;    Score&#xA;    Text&#xA;    CreationDate&#xA;    UserId&#xA;    BADGES&#xA;    Id&#xA;    UserId&#xA;    Name&#xA;    Date&#xA;    POSTTAGS&#xA;    PostId&#xA;    TagId&#xA;    TAGS&#xA;    Id&#xA;    TagName&#xA;    VOTES&#xA;    Id&#xA;    PostId&#xA;    VoteTypeId&#xA;    CreationDate&#xA;    BountyAmount&#xA;    UserId&#xA;    VoteTypes&#xA;    Id&#xA;    Name&#xA;    &lt;/p&gt;&#xA;  &lt;/blockquote&gt;&#xA;&lt;/blockquote&gt;&#xA;" LastActivityDate="2010-09-19T22:46:17.927" Title="What should be the data collection strategy for our first Polystats Project?" Tags="&lt;polystats-project-1&gt;" AnswerCount="3" CommentCount="4" FavoriteCount="1" />
  <row Id="2663" PostTypeId="2" ParentId="2628" CreationDate="2010-09-14T18:15:20.813" Score="2" ViewCount="" Body="&lt;p&gt;Traditionally, statistical inference is taught in the context of probability samples and the nature of sampling error.  This model is the basis for the test of significance.  However, there are other ways to model systematic departures from chance and it turns out that our parametric (sampling based) tests tend to be good approximations of these alternatives.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Parametric tests of hypotheses rely on sampling theory to produce estimates of likely error.  If a sample of a given size is taken from a population, knowledge of the systematic&#xA;nature of sampling makes testing and confidence intervals meaningful.  With a population, sampling theory is simply not relevant and tests are not meaningful in the traditional sense.  Inference is useless, there is nothing to infer to, there is just the thing...the&#xA;parameter itself.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some get around this by appealing to super-populations that the current census represents.  I find these appeals unconvincing--parametric tests are premised on probability sampling and its characteristics.  A population at a given time may be a sample of a larger population over time and place.  However, I don't see any way that one could legitimately argue that this is a random (or more generally any form form of a probability) sample.  Without a probability sample, sampling theory and the&#xA;traditional logic of testing simply do not apply.  You may just as well test on the basis of a convenience sample.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Clearly, to accept testing when using a population, we need to dispense with the basis of those tests in sampling procedures.  One way to do this is to recognize the close connection between our sample-theoretic tests--such as t, Z, and F--and randomization procedures.  Randomization tests are based on the sample at hand.  If I collect&#xA;data on the income of males and females, the probability model and the basis for our estimates of error are repeated random allocations of the actual data values.  I could compare observed differences across groups to a distribution based on this randomization. (We do this all the time in experiments, by the way, where the random sampling from a population model is rarely appropriate).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, it turns out that sample-theoretic tests are often good approximations of randomization tests.  So, ultimately, I think tests from populations are useful and meaningful within this framework and can help to distinguish systematic from chance variation--just like with sample-based tests.  The logic used to get there is a little different, but it doesn't have much affect on the practical meaning and use of tests.  Of course, it might be better to just use randomization and permutation tests directly given they are easily available with all our modern computing power.&lt;/p&gt;&#xA;" OwnerUserId="485" LastActivityDate="2010-09-14T18:15:20.813" CommentCount="5" />
  <row Id="2664" PostTypeId="2" ParentId="2615" CreationDate="2010-09-14T19:18:26.480" Score="0" ViewCount="" Body="&lt;p&gt;As kwak has pointed out, my question was answered on another forum:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.or-exchange.com/questions/695/analytical-solutions-to-limits-of-correlation-stress-testing&quot; rel=&quot;nofollow&quot;&gt;http://www.or-exchange.com/questions/695/analytical-solutions-to-limits-of-correlation-stress-testing&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I did several quick calculations and the suggested analytical solution is consistent with the numerical ones. I still need to check the proof.&lt;/p&gt;&#xA;" OwnerUserId="1250" LastActivityDate="2010-09-14T19:18:26.480" CommentCount="2" />
  <row Id="2665" PostTypeId="2" ParentId="2661" CreationDate="2010-09-14T20:21:34.290" Score="10" ViewCount="" Body="&lt;p&gt;So, I would recommend using standard method for comparing nested models. In your case, you consider two alternative models, the cubic fit being the more &quot;complex&quot; one. An F- or $\chi^2$-test tells you whether the residual sum of squares or deviance significantly decrease when you add further terms. It is very like comparing a model including only the intercept (in this case, you have residual variance only) vs. another one which include one meaningful predictor: does this added predictor account for a sufficient part of the variance in the response? In your case, it amounts to say: Modeling a cubic relationship between X and Y decreases the unexplained variance (equivalently, the $R^2$ will increase), and thus provide a better fit to the data compared to a linear fit. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is often used as a &lt;em&gt;test of linearity&lt;/em&gt; between the response variable and the predictor, and this is the reason why Frank Harrell advocates the use of &lt;a href=&quot;http://lib.stat.cmu.edu/S/Harrell/help/Design/html/Design.trans.html&quot; rel=&quot;nofollow&quot;&gt;restricted cubic spline&lt;/a&gt; instead of assuming a strict linear relationship between Y and the continuous Xs (e.g. age).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The following example comes from a book I was reading some months ago (&lt;a href=&quot;http://www.springer.com/biomed/cancer/book/978-0-387-69763-5&quot; rel=&quot;nofollow&quot;&gt;High-dimensional data analysis in cancer research&lt;/a&gt;, Chap. 3, p. 45), but it may well serves as an illustration. The idea is just to fit different kind of models to a simulated data set, which clearly highlights a non-linear relationship between the response variable and the predictor. The true generative model is shown in black. The other colors are for different models (restricted cubic spline, B-spline close to yours, and CV smoothed spline).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(rms)&#xA;library(splines)&#xA;set.seed(101)&#xA;f &amp;lt;- function(x) sin(sqrt(2*pi*x))&#xA;n &amp;lt;- 1000&#xA;x &amp;lt;- runif(n, 0, 2*pi)&#xA;sigma &amp;lt;- rnorm(n, 0, 0.25)&#xA;y &amp;lt;- f(x) + sigma&#xA;plot(x, y, cex=.4)&#xA;curve(f, 0, 6, lty=2, add=TRUE)&#xA;# linear fit&#xA;lm00 &amp;lt;- lm(y~x)&#xA;# restricted cubic spline, 3 knots (2 Df)&#xA;lm0 &amp;lt;- lm(y~rcs(x,3))&#xA;lines(seq(0,6,length=1000), &#xA;      predict(lm0,data.frame(x=seq(0,6,length=1000))),&#xA;      col=&quot;red&quot;)&#xA;# use B-spline and a single knot at x=1.13 (4 Df)&#xA;lm1 &amp;lt;- lm(y~bs(x, knots=1.13))&#xA;lines(seq(0,6,length=1000),&#xA;      predict(lm1,data.frame(x=seq(0,6,length=1000))), &#xA;      col=&quot;green&quot;)&#xA;# cross-validated smoothed spline (approx. 20 Df)&#xA;xy.spl &amp;lt;- smooth.spline(x, y, cv=TRUE)&#xA;lines(xy.spl, col=&quot;blue&quot;)&#xA;legend(&quot;bottomleft&quot;, c(&quot;f(x)&quot;,&quot;RCS {rms}&quot;,&quot;BS {splines}&quot;,&quot;SS {stats}&quot;), &#xA;       col=1:4, lty=c(2,rep(1,3)),bty=&quot;n&quot;, cex=.6)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/9Ykjp.png&quot; alt=&quot;exmaple&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, suppose you want to compare the linear fit (&lt;code&gt;lm00&lt;/code&gt;) and model relying on B-spline (&lt;code&gt;lm1&lt;/code&gt;), you just have to do an F-test to see that the latter provides a better fit:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; anova(lm00, lm1)&#xA;Analysis of Variance Table&#xA;&#xA;Model 1: y ~ x&#xA;Model 2: y ~ bs(x, knots = 1.13)&#xA;  Res.Df     RSS Df Sum of Sq      F    Pr(&amp;gt;F)    &#xA;1    998 309.248                                  &#xA;2    995  63.926  3    245.32 1272.8 &amp;lt; 2.2e-16 ***&#xA;---&#xA;Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Likewise, it is quite usual to compare &lt;a href=&quot;http://en.wikipedia.org/wiki/Generalized_linear_model&quot; rel=&quot;nofollow&quot;&gt;GLM&lt;/a&gt; with &lt;a href=&quot;http://en.wikipedia.org/wiki/Generalized_additive_model&quot; rel=&quot;nofollow&quot;&gt;GAM&lt;/a&gt; based on the results of a $\chi^2$-test.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-08T07:22:07.983" LastActivityDate="2010-10-08T07:22:07.983" />
  <row Id="2666" PostTypeId="2" ParentId="2662" CreationDate="2010-09-14T20:38:15.233" Score="1" ViewCount="" Body="&lt;p&gt;My thoughts:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;I agree that we should either (a) restrict to those sites or else (b) do everything but them.  The latter option would give more variety in the communities so I think that it could be more interesting, and it would give a picture of communities at much different stages of development and in completely different fields.  So we can get a nice picture of community development and decay.  &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I don't think that we should necessarily need to sample initially based on doing step 1.  People can then engage in dimension reduction or sampling exercises as part of the actual analysis if there is too much data, although I'm optimistic that we can make available all of the data given the restriction from 1.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I'm happy to act as the &quot;data tsar&quot; to get that started (or someone else can), and post a complete dataset in a readable format (or at least provide a script to get it).  That being said, I think the data processing part of this task could be part of what someone's value add in this whole project.  So long as everyone has access to the same data (which they do already), then whether everyone &lt;em&gt;uses&lt;/em&gt; the same data isn't especially important in my view.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Clearly what data one uses will depend on what question they're trying to address.&lt;br&gt;&#xA;I myself am interested in describing how the communities differ.  For that, I would probably start with basic descriptive statistics about questions, users, votes (I'm somewhat less interested in the comments and badges tables), look at some of the distributions (I'm sure that most voting patterns follow a power law, for instance).  Maybe we can use a gini coefficient to classify the inequality of the various communities.  And I would also want to look at social network relationships (e.g. connectedness)?  Is there a stagnation effect as communities grow older (we seem to casually observed this)?  Do new users behave differently than old users?  Needless to say, there are a very large number of interesting questions that can be asked of this data.  &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Alternatively, to answer the &quot;what makes a good question/answer&quot;, &lt;a href=&quot;http://garr.posterous.com/how-to-create-a-guaranteed-killer-ted-talk-wi&quot; rel=&quot;nofollow&quot;&gt;we can do something like this&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-09-14T21:53:11.493" />
  <row Id="2667" PostTypeId="1" AcceptedAnswerId="2674" CreationDate="2010-09-14T21:43:41.883" Score="4" ViewCount="269" Body="&lt;p&gt;When given samples of a discrete random variable, the entropy of the distribution may be estimated by $- \sum \hat{P_i} \log{\hat{P_i}}$, where $\hat{P_i}$ is the sample estimate of the frequency of the $i$th value. (this is up to a constant determined by the base of the log.) This estimate should not be applied to observations from a continuous distribution, at least naively, because it would yield a value which depends only on the sample size.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.87.5281&quot; rel=&quot;nofollow&quot;&gt;Beirlant &lt;em&gt;et al&lt;/em&gt;&lt;/a&gt; describe a number of approaches for the continuous problem, including estimates based on empirical CDF, nearest neighbor distances and the $m$-spacing estimate, which is given by $$\frac{1}{n}\sum_{i=1}^{n-m}\log{(\frac{n}{m}(X_{(i+m)} - X_{(i)}))}$$, where $X_{(i)}$ is the $i$th order statistic of the sample, and $m$ varies in a certain way with $n$. It is not clear how this estimate is to be computed in the presence of ties, &lt;em&gt;i.e.&lt;/em&gt; it does not appear to be applicable to discrete distributions. (a naive correction for ties (drop terms which have $\log{0}$) appears to give an estimator which does not depend on the relative frequency of the classes, only their values, which seems wrong.) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The question: is there an 'all-purpose' estimator which can deal with both discrete and continuous (or even mixed) distributions?&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-09-15T09:34:46.873" Title="All-Purpose Sample Entropy" Tags="&lt;discrete-data&gt;&lt;continuous-data&gt;&lt;entropy&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="2668" PostTypeId="2" ParentId="665" CreationDate="2010-09-14T21:56:16.450" Score="4" ViewCount="" Body="&lt;p&gt;Probability is the embrace of uncertainty, while statistics is an empirical, ravenous pursuit of the truth (damned liars excluded, of course).&lt;/p&gt;&#xA;" OwnerUserId="1108" LastActivityDate="2010-09-14T21:56:16.450" CommentCount="1" />
  <row Id="2670" PostTypeId="1" CreationDate="2010-09-15T00:45:11.290" Score="3" ViewCount="324" Body="&lt;p&gt;There are 100 people in a room.&#xA;What are the odds at least 1 person is born in January?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the best way to calculate this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I used Binomial Distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$E(x) = np = 100 \frac{1}{12} = 8.3$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$std.dev. = \sqrt{npq} = 2.76$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Z(\text{1 person}) = (1 - 8.3) / 2.76 = -2.64$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$p = P(Z&amp;lt;-2.64) = .004$ or $.4%$ odds that less than 1 person is born in Jan&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can a confidence interval calc apply here?  If so, how?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(With 95% certainty, the number of people born in Jan is b/w x &amp;amp; y)&lt;/p&gt;&#xA;" OwnerUserId="1279" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-16T12:29:04.900" LastActivityDate="2010-09-16T12:29:04.900" Title="Odds at least 1 person is born in January?" Tags="&lt;probability&gt;&lt;confidence-interval&gt;&lt;binomial&gt;" AnswerCount="3" CommentCount="2" FavoriteCount="1" />
  <row Id="2671" PostTypeId="2" ParentId="2670" CreationDate="2010-09-15T01:07:21.113" Score="9" ViewCount="" Body="&lt;p&gt;Do you mean &lt;a href=&quot;http://en.wikipedia.org/wiki/Odds#Gambling_odds_versus_probabilities&quot; rel=&quot;nofollow&quot;&gt;odds or probability&lt;/a&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;p(at least 1 person born in January) = 1 - p(no one born in January)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming Independence:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;p(no one born in January) = P(person 1 not born in Jan)&#xA;                            *P(person 2 not born in Jan)&#xA;                            *...P(Person 100 not born in Jan)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Assume P(not born in Jan) = (365-31)/365&lt;/p&gt;&#xA;&#xA;&lt;p&gt;p(no one born in January) = (334/365)^100 = 0.00013975&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, p(at least 1 person born in January)  = 0.99986025&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a binomial calculation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A few points.&lt;br&gt;&#xA;1) Your question is about &lt;a href=&quot;http://stats.stackexchange.com/questions/665/whats-the-difference-between-probability-and-statistics/675&quot;&gt;probability it is not about statistics&lt;/a&gt;, so it is not meaningful to ask for a confidence interval on a probability calculation. A confidence interval is a measure of uncertainty about a parameter estimate. By framing the question as you have you enable a direct calculation of the parameter value (p = 334/365).&lt;br&gt;&#xA;2) Not withstanding comment (1) you have attempted to use a normal approximation to the binomial distribution for your calculation. This will produce a serious error when p is close to 0 or 1 (as here) and is not necessary as exact binomial solutions are trivial in this case.&lt;br&gt;&#xA;3) Your calculation of the mean or expected value (and Std. dev.) of the binomial distribution is correct. We can use this and the &lt;a href=&quot;http://en.wikipedia.org/wiki/Binomial_distribution&quot; rel=&quot;nofollow&quot;&gt;binomial distribution&lt;/a&gt; to estimate the probability of seeing a range of results.  I will use p(born Jan) = 31/365 ~ 0.085.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We expect to see 8.5 people born in January in a class of 100 people. Of course this is impossible. But the probability of seeing 9 people born in January is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\begin{align} &#xA;p(K = k|n = 100)&amp;amp;= \binom{n}{k} * p^k*(1-p)^{(n-k)} \\&#xA;&amp;amp;= \binom{100}{9} * (0.085)^9(1 - 0.085)^{(100-9)}  \\&#xA;&amp;amp;= 1.902*10^{12}*(2.316*10^{-10})*(0.00030855)  \\&#xA;&amp;amp;= 0.13591726  \\&#xA;\end{align}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $\binom{n}{k}$ is the &lt;a href=&quot;http://en.wikipedia.org/wiki/Binomial_coefficient&quot; rel=&quot;nofollow&quot;&gt;binomial coefficient&lt;/a&gt;. (Can someone direct me to how I do maths formulas?)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Similarly the probability of seeing 8 people born in January is: 0.14315621. These two outcomes cover nearly 28% of the probability.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can construct an approximately symmetrical bound on the mean of 0.085 by using the cumulative binomial distribution functions. I'll use one in Stata, but I think Excel has them. It turns out the probability of seeing 3 or fewer January birthdays is 2.53% and the probability of seeing 15 or more January birthdays is 2.19%. So the probability of seeing 4, 5, 6, 7, 8, 9, 10, 11, 12, 13 or 14 birthdays is 95.3% which is a reasonable estimate of a 95% &quot;coverage probability&quot;, although &lt;a href=&quot;http://en.wikipedia.org/wiki/Coverage_probability&quot; rel=&quot;nofollow&quot;&gt;coverage probability&lt;/a&gt; has a slightly different meaning in statistics.&lt;/p&gt;&#xA;" OwnerUserId="521" LastEditorUserId="521" LastEditorDisplayName="" LastEditDate="2010-09-15T04:34:47.393" LastActivityDate="2010-09-15T04:34:47.393" CommentCount="2" />
  <row Id="2672" PostTypeId="2" ParentId="2670" CreationDate="2010-09-15T06:30:41.257" Score="3" ViewCount="" Body="&lt;p&gt;I will shamelessly propose using simulation (in R) to answer your questions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;rbinom(n = 100000, size = 100, prob = 1/12)&lt;/code&gt; draws 100 trials from the binomial distribution 100,000 times and returns a vector of the number of 'successes' (i.e., January births, using the naive 1/12 probability).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should be able to &lt;strong&gt;approximate&lt;/strong&gt; whatever you want from the results.&lt;/p&gt;&#xA;" OwnerUserId="71" LastActivityDate="2010-09-15T06:30:41.257" />
  <row Id="2673" PostTypeId="2" ParentId="2670" CreationDate="2010-09-15T06:49:48.500" Score="6" ViewCount="" Body="&lt;p&gt;Just to add few comments to Thylacoleo excellent answer. From the CDC’s National Vital Statistics System &lt;a href=&quot;http://www.cdc.gov/nchs/data/nvsr/nvsr54/nvsr54_02.pdf&quot; rel=&quot;nofollow&quot;&gt;recent report&lt;/a&gt;, we can estimate the probability of being born on January 1st with a bit more precision. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Month&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From page 50 of the report, we the probability of being born in a certain month:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#R code&#xA;#Month totals - page 50&#xA;m_totals = c(329803, 307248, 336920, 330106, 346754, 337425, &#xA;             364226, 360103, 359644, 354048, 320094, 343579)&#xA;&#xA;m_probs = m_totals/sum(m_totals)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This gives the probability of being born in January as 0.0806 which is a bit lower that 31/365=0.085. For info, Feb and Nov have probabilities around 0.076(ish). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Day of the week&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For fun, (and since it was on the next page), from page 51 of the report we get the probability of being born on a certain day:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#Starting on a Sunday - page 51&#xA;&amp;gt; d_totals = c(7563, 11733, 13001, 12598, 12514, 12396, 8605)&#xA;&amp;gt; d_probs = d_totals/sum(d_totals)&#xA;&amp;gt; round(d_probs,4)&#xA;[1] 0.0965 0.1496 0.1658 0.1607 0.1596 0.1581 0.1097&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Looks like Sunday is a popular day!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Notes&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;This &lt;a href=&quot;http://themetricsystem.rjmetrics.com/2009/09/14/what-are-the-odds-debunking-the-090909-babies/&quot; rel=&quot;nofollow&quot;&gt;page&lt;/a&gt; pointed me in the direction of the CDC report.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="8" LastActivityDate="2010-09-15T06:49:48.500" CommentCount="3" />
  <row Id="2674" PostTypeId="2" ParentId="2667" CreationDate="2010-09-15T09:34:46.873" Score="3" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Entropy is entropy with respect to a measure&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As noticed in the answer to this question &lt;a href=&quot;http://mathoverflow.net/questions/33088/entropy-of-a-general-prob-measure/33090#33090&quot; rel=&quot;nofollow&quot;&gt;http://mathoverflow.net/questions/33088/entropy-of-a-general-prob-measure/33090#33090&lt;/a&gt; , entropy is only defined with respect to a given  measure. For example discrete entropy is entropy with respect to counting measure. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Sample entropy should be an estimate of a predefined entropy.&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think the idea of sample entropy can be generalised to any type of entropy but you need to know which entropy you are trying to estimate before estimating it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Example of entropy with respect to counting+lebesgues&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if you are trying to estimate the entropy with respect to the sum of the lebesgues measure on $[0,1]$ and the counting measure on ${0,1}$ a good estimate certainly (my intuition) is a sum of the two estimates you mention in your question (with $i=0,1$ in the first sum).   &lt;/p&gt;&#xA;" OwnerUserId="223" LastActivityDate="2010-09-15T09:34:46.873" CommentCount="1" />
  <row Id="2675" PostTypeId="1" AcceptedAnswerId="2701" CreationDate="2010-09-15T10:49:45.213" Score="3" ViewCount="300" Body="&lt;p&gt;Hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We want to work out a value for &quot;Average time in service&quot; for our users.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, as we have many subscribers still active, how can we do that? If we just look at average time of deactivated subscribers, that won't show the picture.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there some &quot;correct&quot; way to do this?  We know our churn rate by month, so maybe we need to work out from those still on the service, when they're likely to leave? and use that?&lt;/p&gt;&#xA;" OwnerUserId="1284" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-05-24T15:30:25.013" LastActivityDate="2011-05-24T15:30:25.013" Title="How to calculate average time in service" Tags="&lt;survival&gt;" AnswerCount="2" CommentCount="3" FavoriteCount="3" />
  <row Id="2678" PostTypeId="2" ParentId="2675" CreationDate="2010-09-15T11:50:29.130" Score="5" ViewCount="" Body="&lt;p&gt;Sounds like &lt;a href=&quot;http://en.wikipedia.org/wiki/Time_to_event_analysis&quot; rel=&quot;nofollow&quot;&gt;Survival analysis&lt;/a&gt; to me. The still active subscribers have censored times.  &lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-09-15T11:50:29.130" CommentCount="2" />
  <row Id="2679" PostTypeId="1" AcceptedAnswerId="2680" CreationDate="2010-09-15T14:18:53.813" Score="3" ViewCount="264" Body="&lt;p&gt;Car analogy:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assume the traffic (number of cars per hour) on a road has a Poisson distribution, and the time between cars has the matching exponential distribution. If the chance of each car being red is independent from both time and the color of other cars, will the number of red cars per hour also have a Poisson distribution? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I strongly suspect so. Furthermore I expect E(red cars)=p(red) * E(cars), and because of the Poisson distribution σ(red cars) = p(red) * σ(cars). But how would I (dis)prove this?&lt;/p&gt;&#xA;" OwnerUserId="1287" LastActivityDate="2010-09-18T05:07:02.720" Title="Is a random sample of a Poisson distribution also Poisson distributed?" Tags="&lt;poisson&gt;" AnswerCount="4" />
  <row Id="2680" PostTypeId="2" ParentId="2679" CreationDate="2010-09-15T14:28:02.933" Score="5" ViewCount="" Body="&lt;p&gt;The answer to your first question is yes. If the sum of two independent variables is poisson then the individual variables are also poisson. See &lt;a href=&quot;http://en.wikipedia.org/wiki/Raikov%27s_theorem&quot; rel=&quot;nofollow&quot;&gt;Raikov's Theorem&lt;/a&gt; &lt;/p&gt;&#xA;" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-09-15T18:15:34.653" LastActivityDate="2010-09-15T18:15:34.653" CommentCount="9" />
  <row Id="2681" PostTypeId="2" ParentId="2679" CreationDate="2010-09-15T14:34:29.747" Score="4" ViewCount="" Body="&lt;p&gt;What you have described is a standard and important result from the theory of stochastic processes. Pick up any good book in the library on those, flip to the &quot;Poisson Processes&quot; chapter, and look for &quot;decomposition&quot; or sometimes &quot;thinning&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Two of my favorites are &lt;em&gt;Introduction to Stochastic Processes&lt;/em&gt; by Cinlar and &lt;em&gt;Essentials of Stochastic Processes&lt;/em&gt; by Durrett.&lt;/p&gt;&#xA;" OwnerUserId="1108" LastActivityDate="2010-09-15T14:34:29.747" CommentCount="4" />
  <row Id="2682" PostTypeId="2" ParentId="2679" CreationDate="2010-09-15T14:39:11.517" Score="0" ViewCount="" Body="&lt;p&gt;I think you need additional assumptions to draw this conclusion.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Imagine this: the cars on the road are originally unpainted, but just before you observe them a demon decides what color they will become: red or some other color (of which there could be many).  The demon does not have a clock (to ensure that whatever he does is independent of time), but he does have a fair coin.  Ordinarily he flips the coin to determine whether the car will be red or some other color.  However, his can of red paint can color ten cars and needs to be used as quickly as possible.  So, if the coin decides red, the demon just goes ahead and paints the next ten cars red.  Once the paint is exhausted, he goes back to flipping the coin.  Other colors are painted in the same fashion.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Obviously the car count per unit time is unaffected by this procedure, but I hope it's just as obvious that the count of red cars per unit time is not a Poisson process, because it will be overdispersed: there will be too many high counts, due to the temporal clumping of red cars, and too many low counts, due to the temporal clumping of non-red cars.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It could be that this scenario violates your sense of &quot;the chance of red being independent of the color of the other cars,&quot; but it's difficult to know exactly what this statement means, since it's open to various interpretations.  In the present case, the probability that the next car to be observed is red, given that the previous car was non-red (&quot;the other&quot; colors), is independent of the color of the previous car.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-09-15T14:39:11.517" CommentCount="2" />
  <row Id="2684" PostTypeId="1" AcceptedAnswerId="2740" CreationDate="2010-09-15T16:40:39.463" Score="2" ViewCount="935" Body="&lt;p&gt;Hi folks,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using g*power to calculate the a priori power for a 2 group repeated measures anova with a continuous dependent variable with 3 time points.  I think that higher values for the &quot;correlation among rep measures&quot; should indicate a better test (i.e., high test-retest correltaion) and thus yield better power and a lower total sample size; however the opposite seems to be true:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Test family: F tests; Statistical test: ANOVA: repeated measures, between factors&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Effect size f: 0.25&#xA;alpha error probability: 0.05&#xA;Power: 0.8&#xA;Number of groups: 2&#xA;Number of measurements: 3&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Correlation among rep measures: 0.8 (a reasonable value for these neuropsychological assessments of cognitive function) -&gt; total sample size: 112&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Correlation among rep measures: 0.5 (a poor test would have a value this low) -&gt; 86&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What am I not understanding?  Thanks!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Henry&lt;/p&gt;&#xA;" OwnerUserId="1290" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-16T12:27:41.330" LastActivityDate="2010-09-17T12:56:54.857" Title="How does &quot;correlation among repeated measures&quot; work for repeated measures power analysis in g*power?" Tags="&lt;hypothesis-testing&gt;&lt;anova&gt;" AnswerCount="4" />
  <row Id="2685" PostTypeId="2" ParentId="1799" CreationDate="2010-09-15T16:53:28.813" Score="2" ViewCount="" Body="&lt;p&gt;I don't think that repeated measures is appropriate here - as I understand your protocol, the five temperatures (either hot or cold) are not repeated measures as they occur at different temperatures.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, it does feel like you need some data reduction across the five levels in each temperature group.  You might be well-served to consider some exploratory analysis to decide what the general shape of the relationship of pain perception to ordinal temperature input is across the hot and cold groups; then reduce that to a linear slope or exponential function; then model that single variable as your dependent variable with gender and A/B as factors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Latent growth curve modeling might also be appropriate, but that is beyond me.  &lt;/p&gt;&#xA;" OwnerUserId="1290" LastActivityDate="2010-09-15T16:53:28.813" />
  <row Id="2686" PostTypeId="1" CreationDate="2010-09-15T18:42:29.020" Score="6" ViewCount="201" Body="&lt;p&gt;Help me here, please. Perhaps before even giving me an answer you may need to help me ask the question. I have never learned about time series analysis and do not know if that is indeed what I need. I have never learned about time smoothed averages and do not know if that is indeed what I need. My statistics background: I have 12 credits in biostatistics (multiple linear regression, multiple logistic regression, survival analysis, multi-factorial anova but never repeated measures anova). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;So please look at my scenarios below. What are the buzzwords I should be searching for and can you suggest a resource to learn what I need to learn?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to look at several different data sets for totally different purposes but common to all of them is that there are dates as one variable. So a couple of examples spring to mind: clinical productivity over time (as in how many surgeries or how many office visits) or electric bill over time (as in money paid to electricity company per month). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For both of the above the near universal way to do it is to create a spreadsheet of month or quarter in one column and in the other column would be something such as electricity payment or number of patients seen in the clinic. However, counting per month leads to a lot of noise that has no meaning. For instance, if I usually pay the electricity bill on the 28th of every month but on one occasion I forget and so I only pay it 5 days later on the 3rd of the next month then one month will appear as if there was zero expense and the next month will show ginormous expense. Since one has the actual dates of payment why would one purposefully throw away the very granular data by boxing it into expenses by calendar month. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Similarly if I am out of town for 6 days at a conference then that month will appear to be very unproductive and if those 6 days fell near the end of the month, the next month will be uncharacteristicaly busy since there will be a whole waiting list of people who wanted to see me but had to wait till I returned. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then of course there are the obvious seasonal variations. Air conditioners use a lot of electricity so obviously one has to adjust for summer heat. Billions of children are referred to me for recurrent acute otitis media in the winter and hardly any in the summer and early fall. No child of school age gets scheduled for elective surgery in the first 6 weeks that schools return following the long summer vacation. Seasonality is just one independent variable that affects the dependent variable. There must be other independent variables some of which can be guessed and others that are not known.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A whole bunch of different issues crop up when looking at enrollment in a longstanding clinical study.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What branch of statistics lets us look at this over time by simply looking at events and their actual dates but without creating artificial boxes (months/quarters/years) that do not really exist. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I thought of making the weighted average count for any event. For instance number of patients seen this week is equal to 0.5*nr seen this week + 0.25*nr seen last week + 0.25*nr seen next week. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to learn more about this. What buzzwords should I be searching for? &lt;/p&gt;&#xA;" OwnerUserId="104" LastActivityDate="2010-09-24T17:57:25.143" Title="Understanding productivity or expenses over time without falling victim to stochastic interruptions" Tags="&lt;time-series&gt;&lt;finance&gt;&lt;count-data&gt;" AnswerCount="4" CommentCount="3" FavoriteCount="4" />
  <row Id="2687" PostTypeId="2" ParentId="2684" CreationDate="2010-09-15T19:03:14.480" Score="0" ViewCount="" Body="&lt;p&gt;Higher correlation among repeated measures means that each repeat is adding less information to the first measurement. If the correlation were 1, the repeats are no extra information to the original measurement, and the sample size could be calculated ignoring the repeats. If the correlation were 0, each repeated measurement of a subject adds the same amount of information as an extra subject, so the sample size could be calculated treating every observation as independent.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-09-15T19:03:14.480" CommentCount="1" />
  <row Id="2688" PostTypeId="1" CreationDate="2010-09-15T19:20:45.563" Score="4" ViewCount="274" Body="&lt;p&gt;I have a dataset where I need to do linear regression. Unfortunately there is a problem with heteroscedasticity. I´ve rerun the analysis using robust regression with the HC3 estimator for the variance and also done bootstrapping with the bootcov function in Hmisc for R. The results are quite close. What is generally recommended? &lt;/p&gt;&#xA;" OwnerUserId="1291" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-09-27T19:43:36.460" LastActivityDate="2010-09-27T19:43:36.460" Title="Robust linear regression or bootstrap" Tags="&lt;regression&gt;&lt;bootstrap&gt;&lt;heteroscedasticity&gt;" AnswerCount="2" CommentCount="4" FavoriteCount="1" />
  <row Id="2689" PostTypeId="1" AcceptedAnswerId="2707" CreationDate="2010-09-15T19:54:28.173" Score="4" ViewCount="258" Body="&lt;p&gt;I'm doing this only for learning purposes. I've no intentions of reversing the methods of IMDB.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I asked myself I owned IMDB or similar website. How would I compute the movie rating?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All I can think of is &lt;em&gt;Arithmetic Mean&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a movie data provided below computation would be &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;(38591*10 + 27994*9 + 32732*8 + 17864*7 + 7361*6 + 2965*5 + 1562*4 + 1073*3 + 891*2 + 3401*1) / 134434 = 8.17055953&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/bWeRr.jpg&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;My rating 8.17055953 which is close to IMDB's Arithmetic Mean. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Whats wrong with my score? Why is it not ideal (because IMDB didn't use it)?&lt;/li&gt;&#xA;&lt;li&gt;If you had to compute. How would you have done it? What factors you would consider?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not asking about the ranking mechanism for the Top/Bottom listing (that is shown here : &lt;a href=&quot;http://www.imdb.com/chart/top&quot; rel=&quot;nofollow&quot;&gt;http://www.imdb.com/chart/top&lt;/a&gt;). This question is about how IMDb turns the arithmetic mean into the weighted mean. Take &quot;Manos: The Hands of Fate&quot; — Unweighted mean 2.1, weighted mean 1.5. (Or &quot;The Godfather&quot;, another example of ballot stuffing.) &lt;/p&gt;&#xA;" OwnerUserId="851" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-15T22:39:07.443" LastActivityDate="2010-09-17T02:08:35.420" Title="How would YOU compute IMDB movie rating?" Tags="&lt;rating&gt;" AnswerCount="3" FavoriteCount="2" />
  <row Id="2690" PostTypeId="2" ParentId="2684" CreationDate="2010-09-15T20:02:18.693" Score="1" ViewCount="" Body="&lt;p&gt;I would guess the explanation is something like this: The higher the correlation between the time points, the lower the proportion of unshared variance. That is, the lower the proportion of variance that can contain the effect (i.e., the unshared variance). Given that in this unshared variance the ratio of effect to non-effect is the same (i.e., your effect size) the total amount of variance that contains the effect is smaller the higher the correlation between the time points. Hence, the higher the correlation the more power (i.e., sample size) needed to find the expected effect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I think there is another issue in your question you have to think about. What does the effect size stand for? I am not a user of g-power, but i guess it is the effect size for the interaction of group by time points. In this case my explanation should apply but you should ask yourself if this is the effect you are interrested in.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furthermore, you shouldn't confuse test-retest reliability with correlation between time-points. Generally you want a high test-retest reliability, but in most cases you want a small correlation between time points but an effect between the time point (i.e., rather low correlation).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this is all clear, currently in a hurry, but can get extend/clarify tomorrow.&lt;/p&gt;&#xA;" OwnerUserId="442" LastActivityDate="2010-09-15T20:02:18.693" />
  <row Id="2691" PostTypeId="1" CreationDate="2010-09-15T20:05:55.993" Score="34" ViewCount="3629" Body="&lt;p&gt;In today's pattern recognition class my professor talked about PCA, eigenvectors &amp;amp; eigenvalues. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I got the mathematics of it. If I'm asked to find eigenvalues etc. I'll do it correctly like a machine. But I didn't &lt;strong&gt;understand&lt;/strong&gt; it. I didn't get the purpose of it. I didn't get the feel of it.  I strongly believe in &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;you do not really understand something unless you can explain it to your grandmother -- Albert Einstein&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Well, I can't explain these concepts to a layman or grandma.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Why PCA, eigenvectors &amp;amp; eigenvalues? What was the &lt;em&gt;need&lt;/em&gt; for these concepts?&lt;/li&gt;&#xA;&lt;li&gt;How would you explain these to a layman?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="851" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-16T07:13:27.197" LastActivityDate="2011-04-13T01:55:06.840" Title="Making sense of principal component analysis, eigenvectors &amp; eigenvalues" Tags="&lt;pca&gt;" AnswerCount="15" CommentCount="2" FavoriteCount="21" />
  <row Id="2692" PostTypeId="2" ParentId="2689" CreationDate="2010-09-15T20:42:28.683" Score="6" ViewCount="" Body="&lt;p&gt;Partial answer.  See the help page entitled: &lt;a href=&quot;http://www.imdb.com/help/show_leaf?voteaverage&quot; rel=&quot;nofollow&quot;&gt;The vote average for film &quot;X&quot; should be Y! Why are you displaying another rating?&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In short, IMDb uses:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;a complex voter weighting system to make sure&#xA;  that the final rating is&#xA;  representative of the general voting&#xA;  population and not subject to over&#xA;  influence from individuals who are not&#xA;  regular participants in the poll.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Also note that:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In order to avoid leaving the scheme open to abuse, [IMDb does not] disclose the exact methods used.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="938" LastActivityDate="2010-09-15T20:42:28.683" />
  <row Id="2693" PostTypeId="2" ParentId="2688" CreationDate="2010-09-15T20:45:01.947" Score="2" ViewCount="" Body="&lt;p&gt;In economics, the Eicker-White or &quot;robust&quot; standard errors are typically reported. Bootstrapping (unfortunately, I'd say) is less common. I'd say that the robust estimates are the standard version.&lt;/p&gt;&#xA;" OwnerUserId="401" LastActivityDate="2010-09-15T20:45:01.947" />
  <row Id="2694" PostTypeId="2" ParentId="2691" CreationDate="2010-09-15T20:54:17.370" Score="2" ViewCount="" Body="&lt;p&gt;Eigens are mathematical concept used for implementing PCA, so they are out of discussion.&#xA;PCA is based on the idea that the multivariate data is so hard to interpret not because the reality is complex, but because we have measured wrong variables, and the reality is trivial; namely there are only linear relations involved.&lt;br&gt;&#xA;To this end we use some math-magick to expose this structure, and voilà.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is pure wishful thinking, but it often works because of the Taylor expansion.&lt;/p&gt;&#xA;" OwnerUserId="88" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-15T22:33:05.543" LastActivityDate="2010-09-15T22:33:05.543" />
  <row Id="2695" PostTypeId="2" ParentId="2691" CreationDate="2010-09-15T21:07:58.990" Score="6" ViewCount="" Body="&lt;p&gt;Alright, I'll give this a try. A few months back I dug through a good amount of literature to find an intuitive explanation I could explain to a non-statistician. I found the derivations that use Lagrange multipliers the most intuitive. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say we have high dimension data - say 30 measurements made on an insect. The bugs have different genotypes and slightly different physical features in some of these dimensions, but with such high dimension data it's hard to tell which insects belong to which group.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PCA is a technique to reduce dimension by: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Taking linear combinations of the original variables. &lt;/li&gt;&#xA;&lt;li&gt;Each linear combination explains the most variance in the data it can.&lt;/li&gt;&#xA;&lt;li&gt;Each linear combination is uncorrelated with the others&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Or, in mathematical terms:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;For Y_j = a_j' x (linear combination for jth component)&lt;/li&gt;&#xA;&lt;li&gt;For k &gt; j, V(Y_k) &amp;lt; V(Y_j) (first components explain more variation)&lt;/li&gt;&#xA;&lt;li&gt;a_k' a_j = 0 (orthogonality)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Finding linear combinations that satisfy these constraints leads us to eigenvalues. Why?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I recommend checking out the book &lt;em&gt;An Introduction to Multivariate Data Analysis&lt;/em&gt; for the full derivation (p. 50), but the basic idea is successive optimizations problems (maximizing variance) constrained such that a'a = 1 for coefficients a (to prevent the case when variance could be infinite) and constrained to make sure the coefficients are orthogonal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This leads to optimization with Lagrange multipliers, which in turn reveals why eigenvalues are used. I am too lazy to type it out (sorry!) but, &lt;a href=&quot;http://web.ipac.caltech.edu/staff/fmasci/home/statistics_refs/PrincipalComponentAnalysis.pdf&quot; rel=&quot;nofollow&quot;&gt;this PDF&lt;/a&gt; goes through the proof pretty well from this point. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would never try to explain this to my grandmother, but if I had to talk generally about dimension reduction techniques, I'd point to this trivial projection example (not PCA). Suppose you have a Calder mobile that is very complex. Some points in 3-d space close to each other, others aren't. If we hung this mobile from the ceiling and shined light on it from one angle, we get a projection onto a lower dimension plane (a 2-d wall). Now, if this mobile is mainly wide in one direction, but skinny in the other direction, we can rotate it to get projections that differ in usefulness. Intuitively, a skinny shape in one dimension projected on a wall is less useful - all the shadows overlap and don't give us much information. However, if we rotate it so the light shines on the wide side, we get a better picture of the reduced dimension data - points are more spread out. This is often what we want. I think my grandmother could understand that :-)&lt;/p&gt;&#xA;" OwnerUserId="7" LastActivityDate="2010-09-15T21:07:58.990" CommentCount="3" />
  <row Id="2696" PostTypeId="2" ParentId="1" CreationDate="2010-09-15T21:08:26.077" Score="3" ViewCount="" Body="&lt;p&gt;I'd recommend letting the subject expert specify the &lt;em&gt;mean&lt;/em&gt; or &lt;em&gt;mode&lt;/em&gt; of the prior but I'd feel free to adjust what they give as a &lt;em&gt;scale&lt;/em&gt;.  Most people are not very good at quantifying variance.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;And I would definitely not let the expert determine the distribution family, in particular the tail thickness. For example, suppose you need a symmetric distribution for a prior.  No one can specify their subjective belief so finely as to distinguish a normal distribution from, say, a Student-t distribution with 5 degrees of freedom.  But in some contexts the t(5) prior is much more robust than the normal prior.  In short, I think the choice of tail thickness is a technical statistical matter, not a matter of quantifying expert opinion.&lt;/p&gt;&#xA;" OwnerUserId="319" LastActivityDate="2010-09-15T21:08:26.077" />
  <row Id="2697" PostTypeId="2" ParentId="2691" CreationDate="2010-09-15T21:14:37.067" Score="16" ViewCount="" Body="&lt;p&gt;Hmm, here goes for a completely non-mathematical take on PCA...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Imagine you have just opened a cider shop.  You have 50 varieties of cider and you want to work out how to allocate them onto shelves, so that similar-tasting ciders are put on the same shelf.  There are lots of different tastes and textures in cider - sweetness, tartness, bitterness, yeastiness, fruitiness, clarity, fizziness etc etc.  So what you need to do to put the bottles into categories is answer two questions:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) What qualities are most important for identifying groups of ciders? e.g. does classifying based on sweetness make it easier to cluster your ciders into similar-tasting groups than classifying based on fruitiness?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Can we reduce our list of variables by combining some of them? e.g. is there actually a variable that is some combination of &quot;yeastiness and clarity and fizziness&quot; and which makes a really good scale for classifying varieties?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is essentially what PCA does.  Principal components are variables that usefully explain variation in a data set - in this case, that usefully differentiate between groups.  Each principal component is one of your original explanatory variables, or a combination of some of your original explanatory variables. &lt;/p&gt;&#xA;" OwnerUserId="266" LastActivityDate="2010-09-15T21:14:37.067" CommentCount="3" />
  <row Id="2698" PostTypeId="2" ParentId="2691" CreationDate="2010-09-15T21:33:12.360" Score="14" ViewCount="" Body="&lt;p&gt;Let's do (2) first.  PCA fits an ellipsoid to the data.  An ellipsoid is a multidimensional generalization of distorted spherical shapes like eggs, cigars, and pancakes.  These are all neatly described by the directions and lengths of their principal (semi-)axes, such as the axis of the cigar or egg or the plane of the pancake.  No matter how the ellipsoid is turned, the eigenvectors point in those principal directions and the eigenvalues give you the lengths.  The smallest eigenvalues correspond to the thinnest directions having the least variation, so ignoring them (which collapses them flat) loses relatively little information: that's PCA.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(1) Apart from simplification (above), we have needs for pithy description, visualization, and insight.  Being able to reduce dimensions is a &lt;em&gt;good&lt;/em&gt; thing: it makes it easier to describe the data and, if we're lucky to reduce them to three or less, lets us draw a picture.  Sometimes we can even find useful ways to interpret the combinations of data represented by the coordinates in the picture, which can afford insight into the joint behavior of the variables.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-09-15T21:38:51.857" LastActivityDate="2010-09-15T21:38:51.857" CommentCount="4" />
  <row Id="2699" PostTypeId="2" ParentId="2689" CreationDate="2010-09-15T21:39:44.023" Score="4" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Whats wrong with my score? Why is it not ideal (because IMDB didn't&#xA;  use it)?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;If the score was only for your use, then nothing is wrong with your calculation. &#xA;However, IMDB try to make it difficult for people to obviously influence the final score.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;If you had to compute. How would you have done it? What factors you&#xA;  would consider?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Here are a few factors that you could consider (but will be unable to check):&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The final score may be weighted according to how many votes have been cast.&lt;/li&gt;&#xA;&lt;li&gt;Votes may be weighted by a time variable. For example, votes cast last year are less important than votes cast today.&lt;/li&gt;&#xA;&lt;li&gt;Votes cast by users who have voted for other movies have more weight, i,e. a reputation coefficient.&lt;/li&gt;&#xA;&lt;li&gt;Perhaps they incorporate data from other sites.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="8" LastActivityDate="2010-09-15T21:39:44.023" CommentCount="2" />
  <row Id="2700" PostTypeId="2" ParentId="2691" CreationDate="2010-09-15T21:42:36.200" Score="21" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://www.cs.otago.ac.nz/cosc453/student_tutorials/principal_components.pdf&quot; rel=&quot;nofollow&quot;&gt;This manuscript&lt;/a&gt; really helped me grok PCA. I think it's still too complex for explaining to your grandmother, but it's not bad. You should skip first few bits on calculating eigens, etc. Jump down to the example in chapter 3 and look at the graphs. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have some examples where I worked through some toy examples so I could understand PCA vs. OLS linear regression. I'll try to dig those up and post them as well. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;edit:&lt;/strong&gt;&#xA;You didn't really ask about the difference between Ordinary Least Squares (OLS) and PCA but since I dug up my notes I did a &lt;a href=&quot;http://www.cerebralmastication.com/2010/09/principal-component-analysis-pca-vs-ordinary-least-squares-ols-a-visual-explination/&quot; rel=&quot;nofollow&quot;&gt;blog post about it&lt;/a&gt;. The very short version is OLS of y ~ x minimizes error perpendicular to the independent axis like this (yellow lines are examples of two errors):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/ozqBY.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you were to regress x ~ y (as opposed to y ~ x in the first example) it would minimize error like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/pHbub.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and PCA effectively minimizes error orthogonal to the model itself, like so:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/bOl5N.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More importantly, as others have said, in a situation where you have a WHOLE BUNCH of independent variables, PCA helps you figure out which ones matter the most. The examples above just help visualize what the first principal component looks like in a really simple case. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my blog post I have the R code for creating the above graphs and for calculating the first principal component. It might be worth playing with to build your intuition around PCA. I tend to now really &lt;em&gt;own&lt;/em&gt; something until I write code that reproduces it. &lt;/p&gt;&#xA;" OwnerUserId="29" LastEditorUserId="29" LastEditorDisplayName="" LastEditDate="2010-09-16T17:27:35.140" LastActivityDate="2010-09-16T17:27:35.140" CommentCount="4" />
  <row Id="2701" PostTypeId="2" ParentId="2675" CreationDate="2010-09-16T00:20:19.813" Score="1" ViewCount="" Body="&lt;p&gt;onestop is right, you're looking to do survival analysis. In general, you can use a nonparametric &lt;a href=&quot;http://en.wikipedia.org/wiki/Kaplan-Meier_estimator&quot; rel=&quot;nofollow&quot;&gt;Kaplan-Meier estimator&lt;/a&gt; to plot the survival curve and then derive the &quot;average time in service&quot;.  The area underneath the survival curve works if you don't have any censored data (i.e. subscribers who are still active).  Michael Berry had a nice and clear explanation of this in a blog post for a business scenario similar to yours:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://blog.data-miners.com/2010/06/why-is-area-under-survival-curve-equal.html&quot; rel=&quot;nofollow&quot;&gt;Why is the area under the survival curve equal to the average tenure?&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In your case, where you have censored data, the median time (0.5 quantile) is available as the &quot;average time in service&quot; -- as noted by @onestop.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Harvey Motulsky's book has a nice discussion of this:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.graphpad.com/www/book/survive.htm&quot; rel=&quot;nofollow&quot;&gt;Survival Curves&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="251" LastEditorUserId="251" LastEditorDisplayName="" LastEditDate="2010-09-16T16:23:10.463" LastActivityDate="2010-09-16T16:23:10.463" CommentCount="4" />
  <row Id="2702" PostTypeId="2" ParentId="665" CreationDate="2010-09-16T00:54:49.907" Score="3" ViewCount="" Body="&lt;p&gt;Probability answers questions about what &lt;strong&gt;will&lt;/strong&gt; happen, statistics answers questions about what &lt;strong&gt;did&lt;/strong&gt; happen.&lt;/p&gt;&#xA;" OwnerUserId="274" LastActivityDate="2010-09-16T00:54:49.907" />
  <row Id="2703" PostTypeId="2" ParentId="2691" CreationDate="2010-09-16T02:04:43.360" Score="2" ViewCount="" Body="&lt;p&gt;I might be a bad person to answer this because I'm the proverbial grandmother who has had the concept explained to me and not much more, but here goes:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose you have a population. A large portion of the population is dropping dead of heart attacks. You are trying to figure out what causes the heart attacks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You have two pieces of data: height and weight.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, it's clear that there's SOME relationship between weight and heart attacks, but the correlation isn't really strong. There are some heavy people who have a lot of heart attacks, but some don't.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, you do a PCA, and it tells you that weight divided by height ('body mass') is a much more likely predictor of heart attacks then either weight or height, because, lo and behold, the &quot;reality&quot; is that it's body mass that causes the heart attacks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Essentially, you do PCA because you are measuring a bunch of things and you don't really know if those are really the principle components or if there's some deeper underlying component that you didn't measure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[Please feel free to edit this if it's completely off base. I really don't understand the concept any more deeply than this].&lt;/p&gt;&#xA;" OwnerUserId="239" LastActivityDate="2010-09-16T02:04:43.360" CommunityOwnedDate="2010-09-16T02:05:54.657" CommentCount="1" />
  <row Id="2704" PostTypeId="2" ParentId="2691" CreationDate="2010-09-16T02:18:11.883" Score="1" ViewCount="" Body="&lt;p&gt;I had imagined a lengthy demo with a bunch of graphs and explanations when I stumbled across  &lt;a href=&quot;http://www.youtube.com/watch?v=BfTMmoDFXyE&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1108" LastActivityDate="2010-09-16T02:18:11.883" CommentCount="1" />
  <row Id="2705" PostTypeId="2" ParentId="2684" CreationDate="2010-09-16T03:26:35.713" Score="1" ViewCount="" Body="&lt;ul&gt;&#xA;&lt;li&gt;All else being equal, bigger correlations in a repeated measures design means more power&lt;/li&gt;&#xA;&lt;li&gt;A larger correlation means that more of the variance in the dependent variable is explained by systematic effects of cases/individuals&lt;/li&gt;&#xA;&lt;li&gt;Prior research or previous datasets are useful for estimating the correlation. Estimates of test-retest reliability of measures is a starting point. Interestingly, I've found that greater changes in the means (i.e., the hypothesis of interest) is associated with slightly lower correlations than would be otherwise be the case. Thus, as a very rough rule of thumb, you might want to reduce your estimate of the correlation a little bit from the published test-retest reliability estimates, perhaps by .1 or .2. Also, you might want to tweak the test-retest reliability estimate if the time between measurements is different to the published test-retest reliability estimates (i.e., increasing it if the time frame is shorter, and increasing it if it is longer).&lt;/li&gt;&#xA;&lt;li&gt;Yes, more reliable tests will generally lead to larger test-retest reliability and thus, stronger correlations between levels of the repeated measures factor and more power. They should also lead to larger observed effect sizes (i.e., cohen's ds) because there is less attenuation for low reliability. Of course, theoretically the more you are interested in traits rather than states, the harder it will be to actually create meaningful changes.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-09-16T03:26:35.713" />
  <row Id="2706" PostTypeId="2" ParentId="2691" CreationDate="2010-09-16T04:10:44.910" Score="1" ViewCount="" Body="&lt;p&gt;Basically PCA finds new variables which are linear combinations of the original variables such that in the new space, the data has fewer dimensions.  Think of a data set consisting of the points in 3 dimensions on the surface of a flat plate held up at an angle.  In the original x, y, z axes you need 3 dimensions to represent the data, but with the right linear transformation, you only need 2.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically what @Joel said, but only linear combinations of the input variables.&lt;/p&gt;&#xA;" OwnerUserId="1303" LastActivityDate="2010-09-16T04:10:44.910" />
  <row Id="2707" PostTypeId="2" ParentId="2689" CreationDate="2010-09-16T05:11:05.977" Score="5" ViewCount="" Body="&lt;p&gt;First, &lt;strong&gt;define the theoretical construct&lt;/strong&gt; of interest. There are many ways that a rating can be defined:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What is the theoretical target population? The entire world, English speakers, people who visit IMDB, people who have seen the movie in question?&lt;/li&gt;&#xA;&lt;li&gt;What is the target time frame? It is the rating of the movie now or averaged over its release time.&lt;/li&gt;&#xA;&lt;li&gt;Is it a democratic rating or an expert rating? Some people are more knowledgeable about the worth of movies. Some people are better able to differentiate a good from a bad movie. Some people are more consistent in their ratings over time. Should ratings from people who are &quot;better&quot; at rating movies be given more worth. This relates to a philosophical question of aesthetics and the meaning of intersubjective goodness.&lt;/li&gt;&#xA;&lt;li&gt;Assuming you could get honest ratings from the entire target population in the entire time frame weighted or not by expertise, what is the mapping between these ratings and the composite rating? This could be the arithmetic mean. Alternatively, there are many other ways of combining individual ratings. For example, you could use an interpolated median. Some alternatives would have minimal effect on the rank order of films, but would have a major influence on the absolute value of the rating.&lt;/li&gt;&#xA;&lt;li&gt;Is the number of people interested in the movie relevant to the rating?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Second, use all the available information to &lt;strong&gt;estimate the theoretical construct&lt;/strong&gt;.&#xA;This is where the issues discussed by others would be important.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;the role of demographic adjustments would depend on your definition of the target population&lt;/li&gt;&#xA;&lt;li&gt;a weight for trust could be incorporated. Many indicators could be used: &#xA;&lt;ul&gt;&#xA;&lt;li&gt;the number of previous ratings (more ratings would suggest someone who is engaged in the site more)&lt;/li&gt;&#xA;&lt;li&gt;the degree to which previous ratings are consistent with other raters or at least a subset of raters (greater consistency would suggest thoughtful and honest responding; &lt;/li&gt;&#xA;&lt;li&gt;the degree to which responses are distributed over an extended period of time (this would suggest that the person is less likely to be attempting to game the system)&lt;/li&gt;&#xA;&lt;li&gt;degree of engagement with the site in general: e.g., accessing the site, contributing to discussion boards (more engagement, more trust)&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;as mentioned by @csgillespie you could weight more recent votes greater if you wanted to estimate current attitudes to the film&lt;/li&gt;&#xA;&lt;li&gt;You could weight for expertise in ratings. This would be correlated with trust ratings, but there is a difference.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Third, &lt;strong&gt;validate and monitor&lt;/strong&gt; the estimation process using external trusted data sources.&lt;/p&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-09-17T02:08:35.420" LastActivityDate="2010-09-17T02:08:35.420" />
  <row Id="2708" PostTypeId="2" ParentId="2691" CreationDate="2010-09-16T05:15:42.383" Score="2" ViewCount="" Body="&lt;p&gt;I view PCA as a geometric tool. If you are given a bunch of points in 3-space which are pretty much all on a straight line, and you want to figure out the equation of that line, you get it via PCA (take the first component). If you have a bunch of points in 3-space which are mostly planar, and want to discover the equation of that plane, do it via PCA (take the least significant component vector and that should be normal to the plane). &lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-09-16T05:15:42.383" />
  <row Id="2709" PostTypeId="2" ParentId="2617" CreationDate="2010-09-16T05:33:58.557" Score="0" ViewCount="" Body="&lt;p&gt;To recap, you have a function $f(x,y)$ which is continuous, and pseudo-random uniform at integer $(x,y)$, but nearly normal over the plane. You wish to make $f$ pseudo-random uniform over the whole plane. Suppose you could somehow make $f$ parametrized by some parameter $\lambda$, say by using $\theta$ to twiddle the pseudo-random uniform number generated at the lattice points. Then if you took $g(x,y) = \sum_{i} f(x,y,\lambda_i)$ for a whole bunch of $\lambda_i$ values, $g(x,y)$ would look much more normal, via the Central Limit Theorem.  You could then invert the normal to get a uniform field. (I was inspired by Mike Dunlavey's comment on adding 12 uniforms to get a normal).&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-09-16T05:33:58.557" />
  <row Id="2710" PostTypeId="2" ParentId="2688" CreationDate="2010-09-16T06:36:14.457" Score="3" ViewCount="" Body="&lt;p&gt;You could use generalized least squares, such as the gls() function from the nlme package, which allows you to specify a variance function using the weight argument.  &lt;/p&gt;&#xA;" OwnerUserId="187" LastActivityDate="2010-09-16T06:36:14.457" />
  <row Id="2711" PostTypeId="1" CreationDate="2010-09-16T08:01:38.570" Score="4" ViewCount="93" Body="&lt;p&gt;I'm trying to calculate how good are my measurements in machine learning! Let's say that I have five choices, and that error is 4, 2, 0.002, 3, 6. Naturally, I will pick third one for the hit, but I would like to say following:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;I'm X% certain that hit is third pick  &lt;/li&gt;&#xA;&lt;li&gt;I'm Y% certain that hit is first (last) pick&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Of course, X&gt;&gt;Y but I would like to know if is there some kind of statistics metric for this. I can always introduce my metric, but I'm interested is there some already known, so that I can avoid my explanation and just refer to particular metric?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The closest one is 'confidence interval', but that gives me the probability that my results are in some interval.&lt;/p&gt;&#xA;" OwnerUserId="1313" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-09-17T11:04:45.970" LastActivityDate="2010-10-17T11:45:39.267" Title="How good is my error?" Tags="&lt;machine-learning&gt;&lt;model-selection&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="2712" PostTypeId="2" ParentId="2711" CreationDate="2010-09-16T09:24:21.157" Score="1" ViewCount="" Body="&lt;p&gt;I would recommend cross-validation; then, from all iterations, you can get not only average, but also an approximation of distribution of error, and you can use it to estimate the confidence interval.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-09-16T09:24:21.157" CommentCount="5" />
  <row Id="2713" PostTypeId="2" ParentId="2272" CreationDate="2010-09-16T09:35:44.257" Score="1" ViewCount="" Body="&lt;p&gt;Hello,&#xA;The answers provided before are very helpful and detailed. Here is my $0.25.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Confidence interval (CI) is a concept based on the classical definition of probability (also called the &quot;Frequentist definition&quot;) that probability is like proportion and is based on the axiomatic system of Kolmogrov (and others). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Credibile intervals (HPD) can be considered to have its roots in decision theory, based on the works of Wald and de Finetti (and extended a lot by others).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As people in this thread have done a great job in giving examples and the difference of hypotheses in the Bayesian and frequentist case, I will just stress on a few important points.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;CIs are based on the fact that&#xA;inference MUST be made on all&#xA;possible repetitions of an&#xA;experiment that can be seen and NOT&#xA;only on the observed data where as&#xA;HPDs are based ENTIRELY on the&#xA;observed data (and obv. our prior&#xA;assumptions).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;In general CIs are NOT coherent (will be explained later) where as HPDs are coherent(due to their roots in decision theory). Coherence (as I would explain to my grand mom) means: given a betting problem on a parameter value, if a classical statistician (frequentist) bets on CI and a bayesian bets on HPDs, the frequentist IS BOUND to lose (excluding the trivial case when HPD = CI). In short, if you want to summarize the findings of your experiement as a probability based on the data, the probability HAS to be a posterior probability (based on a prior). There is a theorem (cf Heath and Sudderth, Annals of Statistics, 1978) which (roughly) states: Assignment of probability to &lt;code&gt;\theta&lt;/code&gt; based on data will not make a sure loser if and only if it is obtained in a bayesian way.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;As CIs don't condition on the observed data (also called &quot;Conditionality Principle&quot; CP), there can be paradoxical examples. Fisher was a big supporter of CP and also found a lot of paradoxical examples when this was NOT followed (as in the case of CI). This is the reason why he used p-values for inference, as opposed to CI. In his view p-values were based on the observed data (much can be said about p-values, but that is not the focus here). Two of the very famous paradoxical examples are: (4 and 5)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Cox's example (Annals of Math. Stat., 1958):&#xA;X_i ~ N(mu, sigma^2) (iid) for i=1..n and we want to estimate mu. n is NOT fixed and is chosen by tossing a coin. If coin toss results in H, 2 is chosen o/w 1000 is chosen. The &quot;common sense&quot; estimate - sample mean is an unbiased estimate with a variance of 0.5sigma^2+0.0005sigma^2. What do we use as the variance of sample mean when n = 1000? Isn't it better (or sensible) to use the variance of sample mean estimator as 0.001sigma^2 (conditional variance) instead of the actual variance of the estimator, which is HUGE!! (0.5sigma^2+0.0005sigma^2). This is a simple illustration of CP when we use the variance as 0.001sigma^2 when n=1000. n stand alone has no importance or no information for mu and sigma (ie n is ancillary for them) but GIVEN its value, you know a lot about the &quot;quality of data&quot;. This directly relates to CI as they involve the variance which should not be conditioned on n, ie we will end up using the larger variance, hence over conservative.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Welch's example:&#xA;This example works for any n, but we will take n=2 for simplicity.&#xA;X_1, X_2 ~ U(theta - 1/2, theta +1/2) (iid), theta belongs to the Real line. This implies X_1 - theta ~ U(-1/2, 1/2) (iid). (X_1 + X_2) /2(xbar) - theta (not that this is NOT a statistic) has a distribution independent of theta. We can choose c &gt; 0 s.t. Prob_theta{-c &amp;lt;= xbar - theta &amp;lt;= c} = 1- alpha (~99%), implying (xbar - c, xbar + c) is the 99% CI of theta. The interpretation of this CI is: if we sample repeatedly, we will get different xbar and 99% (at least) times it will contain true theta, BUT (the elephant in the room) for a GIVEN data, we DON'T know the probability that CI will contain true theta. Now, consider the following data: X_1 =0 and X_2=1, as |X_1 - X_2|=1, we know FOR SURE that the interval (X_, X_2) contains theta (one possible criticism, P{|X_1 - X_2|=1} = 0, but we can handle it mathematically and I won't discuss it). This example also illustrates the concept of coherence beautifully. If you are a classical statistician, you will definitely bet on the 99% CI without looking at the value of |X_1 - X_2|(assuming you are true to your profession). However, a bayesian will bet on the CI only if the value of |X_1 - X_2| is close to 1. If we condition on |X_1 - X_2|, the interval is coherent and the player won't be a sure loser any longer (similar to the theorem by Heath and Sudderth).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Fisher had a recommendation for such problems - use CP. For the Welch's example, Fisher suggested to condition of X_2-X_1. As we see, X_2-X_1 is ancillary for theta, but it provides information about theta. If X_2-X_1 is SMALL, there is not a lot of information about theta in the data. If X_2-X_1 is LARGE, there is a lot of information about theta in the data. Fisher extended the strategy of conditioning on the ancillary statistic to a general theory called &lt;em&gt;Fiducial Inference&lt;/em&gt; (also called his greatest failure, cf Zabell, Stat. Sci. 1992), but it didn't become popular due to lack of generality and flexibility. Fisher was trying to find a way different from both the classical statistics (of Neyman School) and the bayesian school (hence the famous adage from Savage: &quot;Fisher wanted to make a Bayesian omelette (ie using CP) without breaking the Bayesian eggs&quot;). Folklore (no proof) says: Fisher in his debates attacked Neyman (for Type I and Type II error and CI) by calling him a &lt;em&gt;Quality Control guy&lt;/em&gt; rather than a &lt;em&gt;Scientist&lt;/em&gt;, as Neyman's methods didn't condition on the observed data, instead looked at all possible repetitions.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Statisticians also want to use Sufficiency Principle (SP) in addition to the CP. But SP and CP together imply the Likelihood Principle (LP) (cf Birnbaum, JASA, 1962) ie given CP and SP, one must ignore the sample space and look at the likelihood function only. Thus, we only need to look at the given data and &lt;em&gt;NOT&lt;/em&gt; at the whole sample space (looking at whole sample space is in a way similar to repeated sampling). This has led to concept like Observed Fisher Information (cf. Efron and Hinkley, AS, 1978) which measure the information about the data from a frequentist perspective. The amount of information in the data is a bayesian concept (and hence related to HPD), instead of CI.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Kiefer did some foundational work on CI in the late 1970s, but his extensions haven't become popular. A good source of reference is Berger (&quot;Could Fisher, Neyman and Jefferys agree about testing of hypotheses&quot;, Stat Sci, 2003).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;h2&gt;Summary:&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;(As pointed out by Srikant and others)&lt;br&gt;&#xA;CIs can't be interpreted as probability and they don't tell anything about the unkown parameter GIVEN the observed data. CIs are statements about repeated experiments.&lt;br&gt;&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;HPDs are probabilistic intervals based on the posterior distribution of the unknown parameter and have a probability based interpretation based on the given data. &lt;br&gt;&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Frequentist property (repeated sampling) property is a desirable property and HPDs (with appropriate priors) and CI both have them. HPDs condition on the given data also in answering the questions about the unknown parameter &lt;br&gt; &lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Objective NOT Subjective) Bayesians agree with the classical statisticians that there is a single TRUE value of the parameter. However, they both differ in the way they make inference about this true parameter.&lt;br&gt;&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bayesian HPDs give us a good way of conditioning on data, but if they fail to agree with the frequentist properties of CI they are not very useful (analalogy: a person who uses HPDs (with some prior) without a good frequentist property, is bound to be doomed like a carpenter who only cares about the hammer and forgets the screw driver) &lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At last, I have seen people in this thread (comments by Dr. Joris: &quot;...assumptions involved imply a diffuse prior, i.e. a complete lack of knowledge about the true parameter.&quot;) talking about lack of knowledge about the true parameter being equivalent to using a diffuse prior. I DONT know if I can agree with the statement (Dr. Keith agrees with me). For example, in the basic linear models case, some distributions can be obtained by using a uniform prior (which some people called diffuse), BUT it DOESN'T mean that uniform distribution can be regarded as a LOW INFORMATION PRIOR. In general, NON-INFORMATIVE(Objective) prior doesn't mean it has low information about the parameter.&lt;br&gt;&#xA;&lt;br&gt; &lt;br&gt; &lt;br&gt;&#xA;&lt;em&gt;Note:&lt;/em&gt; A lot of these points are based on the lectures by one of the prominent bayesians. I am still a student and could have misunderstood him in some way. Also, I could not figure how to insert mathematical equations in my comments. Please accept my apologies in advance.&lt;/p&gt;&#xA;" OwnerUserId="1307" LastActivityDate="2010-09-16T09:35:44.257" />
  <row Id="2714" PostTypeId="2" ParentId="1487" CreationDate="2010-09-16T10:02:46.633" Score="0" ViewCount="" Body="&lt;p&gt;Hello,&#xA;Using PCA only to analyze your gene expression data can lead to (possibly) wrong conclusions. The fundamental assumption behind PCA is &lt;em&gt;equal genewise variation&lt;/em&gt; and this may NOT be justified almost always. The links to the paper to analyze time course data by csgillespie point to better ways.&lt;/p&gt;&#xA;" OwnerUserId="1307" LastActivityDate="2010-09-16T10:02:46.633" />
  <row Id="2715" PostTypeId="1" CreationDate="2010-09-16T10:21:36.837" Score="26" ViewCount="1077" Body="&lt;p&gt;I like G van Belle's book on &lt;a href=&quot;http://vanbelle.org/&quot; rel=&quot;nofollow&quot;&gt;Statistical Rules of Thumb&lt;/a&gt;, and to a lesser extent &lt;a href=&quot;http://eu.wiley.com/WileyCDA/WileyTitle/productCd-0470457988.html&quot; rel=&quot;nofollow&quot;&gt;Common Errors in Statistics (and How to Avoid Them)&lt;/a&gt; from Philipp I Good. They address common pitfalls when interpreting results from experimental and observational studies and provide practical recommendations for statistical inference, or exploratory data analysis. But I feel that &quot;modern&quot; guidelines are somewhat lacking, especially with the ever growing use of computational and robust statistics in various fields, or the introduction of techniques from the machine learning community in, e.g. clinical biostatistics or genetic epidemiology.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Apart from computational tricks or common pitfalls in data visualization which could be addressed elsewhere, I would like to ask: &lt;strong&gt;What are the top rules of thumb you would recommend for efficient data analysis?&lt;/strong&gt; (&lt;em&gt;one rule per answer, please&lt;/em&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am thinking of guidelines that you might provide to a colleague, a researcher without strong background in statistical modeling, or a student in intermediary to advanced course. This might pertain to various stages of data analysis, e.g. sampling strategies, feature selection or model building, model comparison, post-estimation, etc.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-10T09:16:28.193" LastActivityDate="2011-04-10T14:26:46.080" CommunityOwnedDate="2010-09-16T10:21:36.837" Title="Rules of thumb for &quot;modern&quot; statistics" Tags="&lt;modeling&gt;&lt;best-practices&gt;&lt;rule-of-thumb&gt;&lt;eda&gt;" AnswerCount="14" CommentCount="8" FavoriteCount="20" />
  <row Id="2716" PostTypeId="2" ParentId="2691" CreationDate="2010-09-16T10:23:42.837" Score="3" ViewCount="" Body="&lt;p&gt;OK, a totally non-math answer:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have a bunch of variables on a bunch of subjects and you want to reduce it to a smaller number of variables on those same subjects, while losing as little information as possible, then PCA is one tool to do this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It differs from factor analysis, although they often give similar results, in that FA tries to recover a small number of  latent variables from a larger number of observed variables that are believed to be related to the latent variables.&lt;/p&gt;&#xA;" OwnerUserId="686" LastActivityDate="2010-09-16T10:23:42.837" CommentCount="1" />
  <row Id="2717" PostTypeId="1" AcceptedAnswerId="2721" CreationDate="2010-09-16T11:47:15.633" Score="7" ViewCount="569" Body="&lt;p&gt;Hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a (symmetric) matrix &lt;code&gt;M&lt;/code&gt; that represents the distance between each pair of nodes. For example,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;    A   B   C   D   E   F   G   H   I   J   K   L&#xA;A   0  20  20  20  40  60  60  60 100 120 120 120&#xA;B  20   0  20  20  60  80  80  80 120 140 140 140&#xA;C  20  20   0  20  60  80  80  80 120 140 140 140&#xA;D  20  20  20   0  60  80  80  80 120 140 140 140&#xA;E  40  60  60  60   0  20  20  20  60  80  80  80&#xA;F  60  80  80  80  20   0  20  20  40  60  60  60&#xA;G  60  80  80  80  20  20   0  20  60  80  80  80&#xA;H  60  80  80  80  20  20  20   0  60  80  80  80&#xA;I 100 120 120 120  60  40  60  60   0  20  20  20&#xA;J 120 140 140 140  80  60  80  80  20   0  20  20&#xA;K 120 140 140 140  80  60  80  80  20  20   0  20&#xA;L 120 140 140 140  80  60  80  80  20  20  20   0&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is there any method to extract clusters from &lt;code&gt;M&lt;/code&gt; (if needed, the number of clusters can be fixed), such that each cluster contains nodes with small distances between them. In the example, the clusters would be &lt;code&gt;(A, B, C, D)&lt;/code&gt;, &lt;code&gt;(E, F, G, H)&lt;/code&gt; and &lt;code&gt;(I, J, K, L)&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've already tried UPGMA and &lt;code&gt;k&lt;/code&gt;-means but the resulting clusters are very bad.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The distances are the average steps a random walker would take to go from node &lt;code&gt;A&lt;/code&gt; to node &lt;code&gt;B&lt;/code&gt; (&lt;code&gt;!= A&lt;/code&gt;) and go back to node &lt;code&gt;A&lt;/code&gt;. It's guaranteed that &lt;code&gt;M^1/2&lt;/code&gt; is a metric. To run &lt;code&gt;k&lt;/code&gt;-means, I don't use the centroid. I define the distance between node &lt;code&gt;n&lt;/code&gt; cluster &lt;code&gt;c&lt;/code&gt; as the average distance between &lt;code&gt;n&lt;/code&gt; and all nodes in &lt;code&gt;c&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks a lot :)&lt;/p&gt;&#xA;" OwnerUserId="1316" LastEditorUserId="1316" LastEditorDisplayName="" LastEditDate="2010-09-16T12:15:23.647" LastActivityDate="2010-11-07T07:18:14.823" Title="Clustering with a distance matrix" Tags="&lt;clustering&gt;" AnswerCount="3" CommentCount="2" FavoriteCount="4" />
  <row Id="2718" PostTypeId="2" ParentId="2715" CreationDate="2010-09-16T12:08:07.737" Score="13" ViewCount="" Body="&lt;h2&gt;There is no free lunch&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Great part of statistical failures is created by clicking a big shiny button called &quot;Calculate significance&quot; without taking into account its burden of hidden assumptions. &lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Repeat&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Even if a single call to random generator is involved, one may have luck or bad luck and so get to the wrong conclusions.&lt;/p&gt;&#xA;" OwnerUserId="88" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-16T12:18:23.643" LastActivityDate="2010-09-16T12:18:23.643" CommunityOwnedDate="2010-09-16T12:08:07.737" />
  <row Id="2719" PostTypeId="2" ParentId="2715" CreationDate="2010-09-16T12:15:45.783" Score="11" ViewCount="" Body="&lt;p&gt;There can be a long list but to mention a few: (in no specific order)&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;P-value is NOT probability. Specifically, it is not the probability of committing Type I error. Similarly, CIs have no probabilistic interpretation for the given data. They are applicable for repeated experiments.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Problem related to variance dominate bias most the time in practice, so a biased estimate with small variance is better than an unbiased estimate with large variance (most of the time).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Model fitting is an iterative process. Before analyzing the data understand the source of data and possible models that fit or dont fit the description. Also, try model any design issues in your model. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Use the visualization tools, look at the data (for possible abnormalities, obvious trends etc etc to understand the data) before analyzing it. Use the visualization methods (if possible) to see how the model fits to that data.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Last but not the least, use statistical software for what they are made for (to make your task of computation easier), they are not a substitute for human thinking.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="1307" LastEditorUserId="1307" LastEditorDisplayName="" LastEditDate="2010-09-16T12:22:52.473" LastActivityDate="2010-09-16T12:22:52.473" CommunityOwnedDate="2010-09-16T12:15:45.783" CommentCount="7" />
  <row Id="2720" PostTypeId="2" ParentId="1815" CreationDate="2010-09-16T12:31:45.183" Score="1" ViewCount="" Body="&lt;p&gt;I am surprise no one mentioned: Statistical Design by George Casella&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://books.google.com/books?id=sqnbSUtryVAC&amp;amp;printsec=frontcover&amp;amp;dq=statistical++design+casella+book&amp;amp;source=bl&amp;amp;ots=634WifKYf5&amp;amp;sig=nfzDyBldgGgNFTc8rl8i8XYLfp8&amp;amp;hl=en&amp;amp;ei=sQ2STNnXG4Oenwf06_nVCA&amp;amp;sa=X&amp;amp;oi=book_result&amp;amp;ct=result&amp;amp;resnum=9&amp;amp;ved=0CD8Q6AEwCA#v=onepage&amp;amp;q=statistical%20%20design%20casella%20book&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;Google Books Link&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1307" LastActivityDate="2010-09-16T12:31:45.183" CommunityOwnedDate="2010-09-16T12:31:45.183" />
  <row Id="2721" PostTypeId="2" ParentId="2717" CreationDate="2010-09-16T12:36:49.757" Score="6" ViewCount="" Body="&lt;p&gt;There are a number of options.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;k-medoids clustering&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;First, you could try partitioning around medoids (pam) instead of using k-means clustering. This one is more robust, and could give better results. Van der Laan reworked the algorithm. If you're going to implement it yourself, his &lt;a href=&quot;http://www.bepress.com/ucbbiostat/paper105/&quot; rel=&quot;nofollow&quot;&gt;article&lt;/a&gt; is worth a read.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a specific k-medoids clustering algorithm for large datasets. The algorithm is called Clara in R, and is described in chapter 3 of  &lt;em&gt;Finding Groups in Data: An Introduction to Cluster Analysis.&lt;/em&gt; by Kaufman, L and Rousseeuw, PJ (1990).&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;hierarchical clustering&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Instead of UPGMA, you could try some other hierarchical clustering options. First of all, when you use hierarchical clustering, be sure you define the partitioning method properly. This partitioning method is essentially how the distances between observations and clusters are calculated. I mostly use Ward's method or complete linkage, but other options might be the choice for you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Don't know if you tried it yet, but the single linkage method or neighbour joining is often preferred above UPGMA in phylogenetic applications. If you didn't try it yet, you could give it a shot as well, as it often gives remarkably good results.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;In R you can take a look at the package &lt;em&gt;cluster&lt;/em&gt;. All described algorithms are implemented there. See ?pam, ?clara, ?hclust, ... Check also the different implementation of the algorithm in ?kmeans. Sometimes chosing another algorithm can improve the clustering substantially.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;EDIT : Just thought about something: If you work with graphs and nodes and the likes, you should take a look at the markov clustering algorithm as well. That one is used for example in grouping sequences based on blast similarities, and performs incredibly well. It can do the clustering for you, or give you some ideas on how to solve the research problem you're focusing on. Without knowing anything about it in fact, I guess his results are definitely worth looking at. If I may say so, I still consider this method of Stijn van Dongen one of the nicest results in clustering I've ever encountered.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.micans.org/mcl/&quot; rel=&quot;nofollow&quot;&gt;http://www.micans.org/mcl/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1124" LastEditorUserId="1124" LastEditorDisplayName="" LastEditDate="2010-09-16T12:44:13.560" LastActivityDate="2010-09-16T12:44:13.560" />
  <row Id="2722" PostTypeId="2" ParentId="1815" CreationDate="2010-09-16T12:45:22.580" Score="0" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://books.google.co.uk/books?id=CaFZPbCllrMC&quot; rel=&quot;nofollow&quot;&gt;The Design of Experiments: Statistical Principles for Practical Applications&lt;/a&gt; by Roger Mead. Examples are drawn from agriculture and biology, so probably most appropriate if you're interested in one of those fields. Rather expensive for a 600-page paperback but you can probably find it second-hand.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-09-16T12:45:22.580" CommunityOwnedDate="2010-09-16T12:45:22.580" />
  <row Id="2723" PostTypeId="2" ParentId="2715" CreationDate="2010-09-16T12:57:26.753" Score="14" ViewCount="" Body="&lt;p&gt;Don't forget to do some basic data checking before you start the analysis. In particular, look at a scatter plot of every variable you intend to analyse against ID number, date / time of data collection or similar. The eye can often pick up patterns that reveal problems when summary statistics don't show anything unusual. And if you're going to use a log or other transformation for analysis, also use it for the plot.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-09-16T12:57:26.753" CommunityOwnedDate="2010-09-16T12:57:26.753" CommentCount="2" />
  <row Id="2724" PostTypeId="2" ParentId="2715" CreationDate="2010-09-16T13:13:10.500" Score="9" ViewCount="" Body="&lt;p&gt;One thing I tell my students is to produce an appropriate graph for every p-value. e.g., a scatterplot if they test correlation, side-by-side boxplots if they do a one-way ANOVA, etc.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-09-16T13:13:10.500" CommunityOwnedDate="2010-09-16T13:13:10.500" />
  <row Id="2725" PostTypeId="2" ParentId="2662" CreationDate="2010-09-16T13:13:42.507" Score="1" ViewCount="" Body="&lt;p&gt;I agree with Shane for everything but I will put in my responses as well. I think it will be useful though because I have seperate interests in the project than Shane. I am solely interested in the question &quot;Are there repuation effects in SE communities&quot; (not because the other things aren't interesting, simply because that is how I personally like to pursue topics in a simple, counter-factual framework).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1- Restricting to Stackoverflow, Superuser and Serverfault is fine to answer my question, but if people want to consider a wider array of topics it is benificial to have a broader range of communities (its necessary to examine between community variation). It also doesn't hinder my quest to only consider every other community than these three as Shane suggests (I don't have any theoretical reasons to expect differences between reputation effects in different communities apriori). So I concur with Shane's suggestion of every other community besides Stackoverflow, Superuser and Serverfault.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2 - I agree with Shane as well, and I think more data is better at this point. If people need to reduce the data to either make it more manageable or address particular questions they can do it on their own. If we had more refined goals I wouldn't disagree with a sampling strategy, but I don't want to limit potential projects before we get started.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3 - Again I agree with Shane, and I think it will be simpler to manage if we all work with the same data dump. This will allow easier replication and I think will make trouble shooting easier to handle (although if people use different sampling schemes this will become more difficult). I could volunteer some space either in my Dropbox account or google docs and publish the link (as long as people are fine with delimited text I don't see any space problems on my end). I suppose in the future we may want to think about how to store results/code in a manageable and retrievable fashion.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4 - I'll come back to this when I take a closer look at the variables and what information is contained. Although we may want to include everything just so it does not limit anyones ability to answer different questions at this point.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sounds like a really cool project, and I hope it gets some really good user input from the community. I am really interested to see how different people approach the subject. I think I may use this as personal motivation to learn some more R so I can disseminate results.&lt;/p&gt;&#xA;" OwnerUserId="1036" LastActivityDate="2010-09-16T13:13:42.507" />
  <row Id="2726" PostTypeId="1" AcceptedAnswerId="2735" CreationDate="2010-09-16T13:46:18.170" Score="3" ViewCount="66" Body="&lt;p&gt;Is there a way to have R plot to an in-memory object or connection, rather than a named file?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to have a plotting server create many graphs without ever going to a file.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;a href=&quot;http://cran.r-project.org/web/packages/Cairo/index.html&quot; rel=&quot;nofollow&quot;&gt;Cairo&lt;/a&gt; package documents use of a connection, but it doesn't seem to work.  What I would like to do is something like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(Cairo)&#xA;plot.to.var &amp;lt;- function(data) {&#xA;   tc = textConnection(&quot;output&quot;, &quot;w&quot;)&#xA;   CairoPDF(tc)&#xA;   plot(data)&#xA;   dev.off()&#xA;   tc.close()&#xA;   output&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When I do this, CairoPDF mentions a connection patch I can't find a reference to, that will allow me to do this, even though the documentation shows this working.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have no particular desire to use Cairo, merely saw that the documentation mentioned this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas?&lt;/p&gt;&#xA;" OwnerUserId="1119" LastActivityDate="2010-09-16T16:46:17.977" Title="R plots to a connection?" Tags="&lt;r&gt;&lt;data-visualization&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="2727" PostTypeId="2" ParentId="2717" CreationDate="2010-09-16T13:59:10.410" Score="5" ViewCount="" Body="&lt;p&gt;One way to highlight clusters on your distance matrix is by way of &lt;a href=&quot;http://en.wikipedia.org/wiki/Multidimensional_scaling&quot;&gt;Multidimensional scaling&lt;/a&gt;. When projecting individuals (here what you call your nodes) in an 2D-space, it provides a comparable solution to PCA. This is unsupervised, so you won't be able to specify a priori the number of clusters, but I think it may help to quickly summarize a given distance or similarity matrix.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is what you would get with your data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;tmp &amp;lt;- matrix(c(0,20,20,20,40,60,60,60,100,120,120,120,&#xA;                20,0,20,20,60,80,80,80,120,140,140,140,&#xA;                20,20,0,20,60,80,80,80,120,140,140,140,&#xA;                20,20,20,0,60,80,80,80,120,140,140,140,&#xA;                40,60,60,60,0,20,20,20,60,80,80,80,&#xA;                60,80,80,80,20,0,20,20,40,60,60,60,&#xA;                60,80,80,80,20,20,0,20,60,80,80,80,&#xA;                60,80,80,80,20,20,20,0,60,80,80,80,&#xA;                100,120,120,120,60,40,60,60,0,20,20,20,&#xA;                120,140,140,140,80,60,80,80,20,0,20,20,&#xA;                120,140,140,140,80,60,80,80,20,20,0,20,&#xA;                120,140,140,140,80,60,80,80,20,20,20,0),&#xA;              nr=12, dimnames=list(LETTERS[1:12], LETTERS[1:12]))&#xA;d &amp;lt;- as.dist(tmp)&#xA;mds.coor &amp;lt;- cmdscale(d)&#xA;plot(mds.coor[,1], mds.coor[,2], type=&quot;n&quot;, xlab=&quot;&quot;, ylab=&quot;&quot;)&#xA;text(jitter(mds.coor[,1]), jitter(mds.coor[,2]),&#xA;     rownames(mds.coor), cex=0.8)&#xA;abline(h=0,v=0,col=&quot;gray75&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/oA1vV.png&quot; alt=&quot;mds&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I added a small jittering on the x and y coordinates to allow distinguishing cases. Replace &lt;code&gt;tmp&lt;/code&gt; by &lt;code&gt;1-tmp&lt;/code&gt; if you'd prefer working with dissimilarities, but this yields essentially the same picture. However, here is the hierarchical clustering solution, with &lt;em&gt;single&lt;/em&gt; agglomeration criteria:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plot(hclust(dist(1-tmp), method=&quot;single&quot;))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/mOKiI.png&quot; alt=&quot;hc&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might further refine the selection of clusters based on the dendrogram, or more robust methods, see e.g. this related question: &lt;a href=&quot;http://stats.stackexchange.com/questions/2597/what-stop-criteria-for-agglomerative-hierarchical-clustering-are-used-in-practice&quot;&gt;What stop-criteria for agglomerative hierarchical clustering are used in practice?&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-09-16T13:59:10.410" />
  <row Id="2728" PostTypeId="1" AcceptedAnswerId="2738" CreationDate="2010-09-16T15:40:06.907" Score="4" ViewCount="252" Body="&lt;p&gt;Hello all,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am clustering a dataset using the pam command (from {cluster} package), and I wish to decide on the number of clusters to use.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was able to implement The_Elbow_Method in R (&lt;a href=&quot;http://en.wikipedia.org/wiki/Determining_the_number_of_clusters_in_a_data_set#The_Elbow_Method&quot; rel=&quot;nofollow&quot;&gt;see wiki&lt;/a&gt;) for doing that.  But that doesn't provide me with any solid criteria (like AIC, &lt;a href=&quot;http://en.wikipedia.org/wiki/Determining_the_number_of_clusters_in_a_data_set#Information_Criterion_Approach&quot; rel=&quot;nofollow&quot;&gt;for example&lt;/a&gt;) for decision.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I came by the {clValid} package which looks promising, but I wanted to know if there are &lt;strong&gt;any other R solutions (you know of) for choosing the number of clusters for pam?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's some dummy code if someone wants to show examples:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data(iris)&#xA;head(iris)&#xA;require(cluster)&#xA;pam(iris[,1:4], 3)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="253" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-16T16:20:44.223" LastActivityDate="2010-11-18T20:58:25.817" Title="Algorithm for choosing the number of clusters when using pam in R?" Tags="&lt;r&gt;&lt;clustering&gt;&lt;model-selection&gt;" AnswerCount="4" FavoriteCount="1" />
  <row Id="2729" PostTypeId="2" ParentId="2728" CreationDate="2010-09-16T15:43:10.977" Score="2" ViewCount="" Body="&lt;p&gt;What about &lt;a href=&quot;http://bm2.genes.nig.ac.jp/RGM2/R_current/library/cluster/html/silhouette.html&quot; rel=&quot;nofollow&quot;&gt;silhouette&lt;/a&gt;? &lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-09-16T15:43:10.977" CommentCount="5" />
  <row Id="2730" PostTypeId="1" AcceptedAnswerId="2778" CreationDate="2010-09-16T15:53:31.090" Score="9" ViewCount="596" Body="&lt;p&gt;I am running experiments for a paper and am looking for an interesting book/website to understand properly how anova and ancova work. I have a good math background so I don't necessarily need a vulgarized explanation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd also like to know how to determine when to use anova versus ancova.&lt;/p&gt;&#xA;" OwnerUserId="1320" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-20T06:49:02.970" LastActivityDate="2010-10-20T11:56:39.077" Title="Good resource to understand anova and ancova?" Tags="&lt;books&gt;&lt;anova&gt;&lt;ancova&gt;" AnswerCount="6" FavoriteCount="4" />
  <row Id="2731" PostTypeId="2" ParentId="2728" CreationDate="2010-09-16T16:02:54.897" Score="2" ViewCount="" Body="&lt;p&gt;The &lt;a href=&quot;http://cran.r-project.org/web/packages/fpc/index.html&quot; rel=&quot;nofollow&quot;&gt;fpc package&lt;/a&gt; provides a few clustering statistics.  If you're looking for information criteria in particular, the &lt;code&gt;cluster.stats&lt;/code&gt; method provides an information based distance.  For mixture models based on clustering, the BIC is available.&lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-09-16T16:02:54.897" CommentCount="1" />
  <row Id="2732" PostTypeId="2" ParentId="2730" CreationDate="2010-09-16T16:06:13.050" Score="10" ViewCount="" Body="&lt;p&gt;So, in addition to this paper, &lt;a href=&quot;http://mres.gmu.edu/pmwiki/uploads/Main/ancova.pdf&quot; rel=&quot;nofollow&quot;&gt;Misunderstanding Analysis of Covariance&lt;/a&gt;, which enumerates common pitfalls when using ANCOVA, I would recommend starting with:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Frank Harrell's &lt;a href=&quot;http://biostat.mc.vanderbilt.edu/wiki/Main/FrankHarrell&quot; rel=&quot;nofollow&quot;&gt;homepage&lt;/a&gt;, especially his handout on &lt;a href=&quot;http://biostat.mc.vanderbilt.edu/wiki/Main/RmS&quot; rel=&quot;nofollow&quot;&gt;Regression Modeling Strategies&lt;/a&gt; and &lt;a href=&quot;http://biostat.mc.vanderbilt.edu/wiki/pub/Main/BioMod/notes.pdf&quot; rel=&quot;nofollow&quot;&gt;Biostatistical Modeling&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;John Fox's &lt;a href=&quot;http://socserv.mcmaster.ca/jfox/&quot; rel=&quot;nofollow&quot;&gt;homepage&lt;/a&gt; includes great material on &lt;a href=&quot;http://en.wikipedia.org/wiki/Linear_model&quot; rel=&quot;nofollow&quot;&gt;Linear Model&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://cran.r-project.org/doc/contrib/Faraway-PRA.pdf&quot; rel=&quot;nofollow&quot;&gt;Practical Regression and Anova using R&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;This is mostly R-oriented material, but I feel you might better catch the idea if you start playing a little bit with these models on toy examples or real datasets (and R is great for that).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for a good book, I would recommend &lt;a href=&quot;http://eu.wiley.com/WileyCDA/WileyTitle/productCd-EHEP000137.html&quot; rel=&quot;nofollow&quot;&gt;&lt;em&gt;Design and Analysis of Experiments&lt;/em&gt;&lt;/a&gt; by Montgomery (now in its 7th ed.); ANCOVA is described in chapter 15. &lt;a href=&quot;http://www.springer.com/statistics/statistical+theory+and+methods/book/978-0-387-95361-8&quot; rel=&quot;nofollow&quot;&gt;&lt;em&gt;Plane Answers to Complex Questions&lt;/em&gt;&lt;/a&gt; by Christensen is an excellent book on the theory of linear model (ANCOVA in chapter 9); it assumes a good mathematical background. Any biostatistical textbook should cover both topics, but I like &lt;a href=&quot;http://www.pearsonhighered.com/educator/product/Biostatistical-Analysis/9780131008465.page&quot; rel=&quot;nofollow&quot;&gt;&lt;em&gt;Biostatistical Analysis&lt;/em&gt;&lt;/a&gt; by Zar (ANCOVA in chapter 12), mainly because this was one of my first textbook.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And finally, H. Baayen's textbook is very complete, &lt;a href=&quot;http://magnuson.psy.uconn.edu/mirman/R/Baayen.pdf?bcsi_scan_D99544420D78AF92=0&amp;amp;bcsi_scan_filename=Baayen.pdf&quot; rel=&quot;nofollow&quot;&gt;Practical Data Analysis for the Language Sciences with R&lt;/a&gt;. Although it focus on linguistic data, it includes a very comprehensive treatment of the Linear Model and mixed-effects models.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-20T11:56:39.077" LastActivityDate="2010-10-20T11:56:39.077" CommentCount="2" />
  <row Id="2733" PostTypeId="2" ParentId="485" CreationDate="2010-09-16T16:27:02.927" Score="1" ViewCount="" Body="&lt;p&gt;There are a bunch of helpful video tutorials on basic statistics &amp;amp; data mining with R and Weka at SentimentMining.net.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://sentimentmining.net/StatisticsVideos/&quot; rel=&quot;nofollow&quot;&gt;http://sentimentmining.net/StatisticsVideos/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="653" LastActivityDate="2010-09-16T16:27:02.927" CommunityOwnedDate="2010-09-16T16:27:02.927" CommentCount="1" />
  <row Id="2734" PostTypeId="2" ParentId="2730" CreationDate="2010-09-16T16:37:35.490" Score="4" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0470510242&quot; rel=&quot;nofollow&quot;&gt;The R book&lt;/a&gt; does a good job on that. You can see that it dedicates one chapter to each one of those methods (11 and 12). If you are new to R, this is a great book to start with.&lt;/p&gt;&#xA;" OwnerUserId="339" LastEditorUserId="339" LastEditorDisplayName="" LastEditDate="2010-09-16T16:48:14.703" LastActivityDate="2010-09-16T16:48:14.703" />
  <row Id="2735" PostTypeId="2" ParentId="2726" CreationDate="2010-09-16T16:46:17.977" Score="2" ViewCount="" Body="&lt;p&gt;I don't think so; R does not have binary memory buffers. Cairo feature you mentioned needs recompilation of both R and package; trying to do this with plain R does not throw errors, but nothing is written neither to textConnection nor socket.&lt;br&gt;&#xA;So I think the best idea will be to use ramdisk; if you still want to try the patch, see &lt;a href=&quot;http://rwiki.sciviews.org/doku.php?id=developers%3ar_connections_api&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-09-16T16:46:17.977" CommentCount="1" />
  <row Id="2736" PostTypeId="2" ParentId="1736" CreationDate="2010-09-16T16:53:28.610" Score="2" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://www.vni.com/campaigns/pyimslstudioeval/&quot; rel=&quot;nofollow&quot;&gt;PyIMSL&lt;/a&gt; contains a handful of routines for survival analyses. It is Free As In Beer for noncommercial use, fully supported otherwise. From the documentation in the Statistics User Guide...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Computes Kaplan-Meier estimates of survival &#xA;probabilties:  kaplanMeierEstimates()&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Analyzes survival and reliability data using Cox’s&#xA;proportional hazards model: propHazardsGenLin()&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Analyzes survival data using the generalized &#xA;linear model:  survivalGlm() &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Estimates using various parametric modes:  survivalEstimates()&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Estimates a reliability hazard function using a&#xA;nonparametric approach:  nonparamHazardRate()&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Produces population and cohort life tables:  lifeTables()&lt;/p&gt;&#xA;" OwnerUserId="1080" LastEditorUserId="1080" LastEditorDisplayName="" LastEditDate="2010-09-16T18:58:26.863" LastActivityDate="2010-09-16T18:58:26.863" />
  <row Id="2737" PostTypeId="2" ParentId="2686" CreationDate="2010-09-16T17:21:56.977" Score="1" ViewCount="" Body="&lt;p&gt;&lt;strike&gt;I have heard of 'time-based boxcar' functions which might solve your problem. A time-based boxcar sum of 'window size' $\Delta t$ is defined at time $t$ to be the sum of all values between $t - \Delta t$ and $t$. This will be subject to discontinuities which you may or may not want. If you want older values to be downweighted, you can employ a simple or exponential moving average within your time based window.&lt;/strike&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;edit:&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I interpret the question as follows: suppose some events occur at times $t_i$&#xA;with magnitudes $x_i$. (for example, $x_i$ might be the amount of a bill paid.)&#xA;Find some function $f(t)$ which estimates the &lt;em&gt;sum&lt;/em&gt; of the magnitudes of the&#xA;$x_i$ for times &quot;near&quot; $t$.  For one of the examples posed by the OP, $f(t)$&#xA;would represent &quot;how much one was paying for electricity&quot; around time $t$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Similar to this problem is that of estimating the &quot;average&quot; value around&#xA;time $t$. For example: &lt;a href=&quot;http://en.wikipedia.org/wiki/Regression_analysis&quot; rel=&quot;nofollow&quot;&gt;regression&lt;/a&gt;, &lt;a href=&quot;http://en.wikipedia.org/wiki/Interpolation&quot; rel=&quot;nofollow&quot;&gt;interpolation&lt;/a&gt; (not&#xA;usually applied to noisy data), and &lt;a href=&quot;http://en.wikipedia.org/wiki/Filter_%28signal_processing%29&quot; rel=&quot;nofollow&quot;&gt;filtering&lt;/a&gt;. You could spend a lifetime&#xA;studying just one of these three problems.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A seemingly unrelated problem, statistical in nature, is &lt;a href=&quot;http://en.wikipedia.org/wiki/Density_estimation&quot; rel=&quot;nofollow&quot;&gt;Density&#xA;Estimation&lt;/a&gt;.  Here the goal is, given observations of magnitudes $y_i$&#xA;generated by some process, to estimate, roughly, the probability of that&#xA;process generating an event of magnitude $y$. One approach to density&#xA;estimation is via a &lt;a href=&quot;http://en.wikipedia.org/wiki/Kernel_density_estimation&quot; rel=&quot;nofollow&quot;&gt;kernel function&lt;/a&gt;. My suggestion is to abuse the kernel&#xA;approach for this problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let $w(t)$ be a function such that $w(t) \ge 0$ for all $t$, $w(0) = 1$&#xA;(ordinary kernels do not all share this property), and $w'(t) \le 0$. Let $h$&#xA;be the &lt;em&gt;bandwidth&lt;/em&gt;, which controls how much influence each data point has.&#xA;Given data $t_i, x_i$, define the sum estimate by&#xA;$$f(t) = \sum_{i=1}^n x_i w(|t - t_i|/h).$$&#xA;Some possible values of the function $w(t)$ are as follows:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;a uniform (or 'boxcar') kernel: $w(t) = 1$ for $t \le 1$ and $0$ otherwise;&lt;/li&gt;&#xA;&lt;li&gt;a triangular kernel: $w(t) = \max{(0,1-t)}$;&lt;/li&gt;&#xA;&lt;li&gt;a quadratic kernel: $w(t) = \max{(0,1-t^2)}$;&lt;/li&gt;&#xA;&lt;li&gt;a tricube kernel: $w(t) = \max{(0,(1-t^2)^3)}$;&lt;/li&gt;&#xA;&lt;li&gt;a Gaussian kernel: $w(t) = \exp{(-t^2 / 2)}$;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I call these kernels, but they are off by a constant factor here and there; see also a &lt;a href=&quot;http://en.wikipedia.org/wiki/Kernel_%28statistics%29&quot; rel=&quot;nofollow&quot;&gt;comprehensive list of kernels&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some example code in Matlab:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;%%kernels&#xA;ker0 = @(t)(max(0,ceil(1-t))); %uniform&#xA;ker1 = @(t)(max(0,1-t)); %triangular&#xA;ker2 = @(t)(max(0,1-t.^2)); %quadratic&#xA;ker3 = @(t)(max(0,(1-t.^2).^3)); %tricube&#xA;ker4 = @(t)(exp(-0.5 * t.^2)); %Gaussian&#xA;&#xA;%%compute f(t) given x_i,t_i,kernel,h&#xA;ff = @(x_i,t_i,t,kerf,h)(sum(x_i .* kerf(abs(t - t_i) / h)));&#xA;&#xA;%%some sample data: irregular electric bills&#xA;sdata = [&#xA;datenum(2009,12,30),141.73;...&#xA;datenum(2010,01,25),100.45;...&#xA;datenum(2010,02,23),98.34;...&#xA;datenum(2010,03,30),83.92;...&#xA;datenum(2010,05,01),56.21;...       %late this month;&#xA;datenum(2010,05,22),47.33;...       &#xA;datenum(2010,06,14),62.84;...       &#xA;datenum(2010,07,30),83.34;...       &#xA;datenum(2010,09,10),93.34;...       %really late this month&#xA;datenum(2010,09,22),78.34;...&#xA;datenum(2010,10,22),93.25;...&#xA;datenum(2010,11,14),83.39;...       %early this month;&#xA;datenum(2010,12,30),133.82];&#xA;&#xA;%%some irregular observation times at which to sample the filtered version;&#xA;t_obs  = sort(datenum(2009,12,01) + 400 * rand(1,400));&#xA;&#xA;t_i = sdata(:,1);x_i = sdata(:,2);&#xA;&#xA;%%compute f(t) for each of the kernel functions;&#xA;h   = 60;    %bandwidth of 60 days;&#xA;&#xA;fx0 = arrayfun(@(t)(ff(x_i,t_i,t,ker0,h)),t_obs);&#xA;fx1 = arrayfun(@(t)(ff(x_i,t_i,t,ker1,h)),t_obs);&#xA;fx2 = arrayfun(@(t)(ff(x_i,t_i,t,ker2,h)),t_obs);&#xA;fx3 = arrayfun(@(t)(ff(x_i,t_i,t,ker3,h)),t_obs);&#xA;fx4 = arrayfun(@(t)(ff(x_i,t_i,t,ker4,0.5*h)),t_obs);   %!!use smaller bandwidth&#xA;&#xA;%%plot them;&#xA;lhand = plot(t_i,x_i,'--rs',t_obs,fx0,'m-+',t_obs,fx1,'b-+',t_obs,fx2,'k-+',...&#xA;t_obs,fx3,'g-+',t_obs,fx4,'c-+');&#xA;set(lhand(1),'MarkerSize',12);&#xA;set(lhand(2:end),'MarkerSize',4);&#xA;datetick(gca());&#xA;legend(lhand,{'data','uniform','triangular','quadratic','tricube','gaussian'});&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The plot shows the use of a few kernels on some sample electric bill data. &lt;img src=&quot;http://i.imgur.com/pLGm5.jpg&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note&#xA;that the uniform kernel is subject to the 'stochastic shocks' which the OP is&#xA;trying to avoid. The tricube and Gaussian kernels give much smoother&#xA;approximations. If this approach is acceptable, one only has to choose the&#xA;kernel and the bandwidth (in general that is a hard problem, but given some&#xA;domain knowledge, and some code-test-recode loops, it should not be too&#xA;difficult.)&lt;/p&gt;&#xA;" OwnerUserId="795" LastEditorUserId="795" LastEditorDisplayName="" LastEditDate="2010-09-24T04:24:59.887" LastActivityDate="2010-09-24T04:24:59.887" CommentCount="3" />
  <row Id="2738" PostTypeId="2" ParentId="2728" CreationDate="2010-09-16T18:21:20.133" Score="1" ViewCount="" Body="&lt;p&gt;You may find an &lt;a href=&quot;http://stats.stackexchange.com/questions/723/how-can-i-test-whether-my-clustering-of-binary-data-is-significant/749#749&quot;&gt;answer to a similar question&lt;/a&gt; useful. I have also used clValid but, as I recall, it was rather slow (at least for relatively large datasets).&lt;/p&gt;&#xA;" OwnerUserId="339" LastActivityDate="2010-09-16T18:21:20.133" CommentCount="2" />
  <row Id="2739" PostTypeId="1" AcceptedAnswerId="2741" CreationDate="2010-09-16T18:36:13.733" Score="3" ViewCount="87" Body="&lt;p&gt;Can someone give a concise, layman's explanation of an &quot;If-Then&quot; rule (as in rule-based systems).  I am finding this term used frequently without anyone really defining it properly.&lt;/p&gt;&#xA;" OwnerUserId="1121" LastActivityDate="2010-09-16T19:27:07.387" Title="What is an &quot;If-Then&quot; rule?" Tags="&lt;machine-learning&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="2740" PostTypeId="2" ParentId="2684" CreationDate="2010-09-16T18:41:33.953" Score="2" ViewCount="" Body="&lt;p&gt;Higher correlation within subject gets you more power when the test being done is a differencing, equivalent to a paired t-test.  The standard deviation used in calculating effect size is multiplied by sqrt(1-r).  The standard deviation for difference scores (for a one-sample test) is SD*sqrt(2-2r).  This also applies where it is appropriate to model using random intercepts, because that means a chunk of variability is being accounted for by individual random effects, and not being counted as error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The opposite happens when the test is strictly between subjects.  There you are in the world of increasing your effective sample size (as indicated by &quot;Onestop&quot; earlier).  By repeating measurements, you increase your sample by the degree to which the observations are uncorrelated.  The standard deviation for an average of independent (uncorrelated) numbers drawn from the same distribution is the standard error: SD*sqrt(1/N).  For averages of correlated numbers, it is roughly SD*sqrt(1/N +r(N-1)/N).  Note that as correlation approaches 1, you get back the original SD, and as it approaches 0, SD approaches the SE for independent observations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Paul&lt;/p&gt;&#xA;" OwnerUserId="1324" LastEditorUserId="1324" LastEditorDisplayName="" LastEditDate="2010-09-17T12:56:54.857" LastActivityDate="2010-09-17T12:56:54.857" CommentCount="1" />
  <row Id="2741" PostTypeId="2" ParentId="2739" CreationDate="2010-09-16T19:20:07.583" Score="5" ViewCount="" Body="&lt;p&gt;It is just a simple classifier; so simple that it is better to explain it by example. Let's say you have a 3 class classification problem and information system with 4 continous predictors $X1,\ldots, X4$. Now you can define simple rules like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;IF X1&amp;lt;3.45 THEN Class1&#xA;IF X3&amp;gt;7.2 THEN Class2&#xA;IF X2&amp;lt;2.11 THEN Class1&#xA;IF X2&amp;lt;1.2 THEN Class3&#xA; ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and this is it; the final decision is produced by voting. Structures like this can be trained by various methods, probably the best known are &lt;a href=&quot;http://en.wikipedia.org/wiki/Rough_sets&quot; rel=&quot;nofollow&quot;&gt;rough sets&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="88" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-16T19:27:07.387" LastActivityDate="2010-09-16T19:27:07.387" CommentCount="2" />
  <row Id="2742" PostTypeId="1" AcceptedAnswerId="2747" CreationDate="2010-09-16T19:28:10.130" Score="2" ViewCount="652" Body="&lt;p&gt;Several statistical packages, such as SAS, SPSS, and R, allow you to perform some kind of factor rotation following a PCA. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Why is a rotation necessary after a PCA?&lt;/li&gt;&#xA;&lt;li&gt;Why would you apply an oblique rotation after a PCA given that the aim of PCA is to produce orthogonal dimensions?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="1154" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-09-17T06:40:34.453" LastActivityDate="2010-09-17T06:40:34.453" Title="On the use of oblique rotation after PCA" Tags="&lt;pca&gt;&lt;factor-analysis&gt;" AnswerCount="3" CommentCount="1" />
  <row Id="2743" PostTypeId="1" CreationDate="2010-09-16T19:43:55.820" Score="3" ViewCount="448" Body="&lt;p&gt;I am attempting to calculate Cohen's d and then pool those estimates into a summary effect size. Can anyone help? (STATA or SPSS software owned).&lt;/p&gt;&#xA;" LastEditorUserId="442" LastEditorDisplayName="" LastEditDate="2010-09-17T08:14:48.507" LastActivityDate="2010-09-17T08:14:48.507" Title="How to use STATA to pool Cohen's d?" Tags="&lt;spss&gt;&lt;meta-analysis&gt;&lt;stata&gt;&lt;effect-size&gt;&lt;cohens-d&gt;" AnswerCount="2" FavoriteCount="0" />
  <row Id="2744" PostTypeId="2" ParentId="2662" CreationDate="2010-09-16T19:50:38.853" Score="1" ViewCount="" Body="&lt;p&gt;First, a meta point- We probably should not have moved to this phase without finalizing the project question. So, I will wait a few days for any more opinions to come in and then get this thread locked. The next question will focus on which one of the four ideas we should work on our first project. Ideally, all of us should commit to working on the most voted idea (even if it is not our personal favorite) to maximize participation and to divide-and-conquer the project. After all, the point of Polystats Projects = 'massively collaborative data analysis'.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are my answers to my four questions from the top:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Depending on the project selected we should use data from the SO, SU and SF OR the data from the beta sites OR perhaps all sites. I think the answer to this is dependent on the question we select.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;IMO, if we select SO, SU and SF we definitely need a sampling strategy. If we want to maximize contribution then it would make sense to work with 'small' datasets as the community will have different computing limitations (e.g., some folks may not have the RAM to load all the data in memory etc). We want to be as inclusive as possible and hence large datasets are a negative as far as this issue is concerned.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Perhaps, we can do some sort of &lt;a href=&quot;http://en.wikipedia.org/wiki/Stratified_sampling&quot; rel=&quot;nofollow&quot;&gt;stratified sampling&lt;/a&gt; with the strata being: time period, No of votes for qns during a time period (high, medium, low etc), rep of users during a time period (again high, medium, low etc).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Yes, we should work with the same dataset. Not only does this help with reproducibility but it would also ensure that we are on the same page. One advantage here is that if we use different methods (machine learning, stats, data mining etc) to investigate the same question then we can attribute any differences in conclusions to the methods used rather than any idiosyncrasies of the data we used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some form of delimited, text file is ideal (perhaps, csv). Storage is much less of an issue as long as it is a stable link.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;What data we need to extract is dependent on the questions we seek to investigate. So, I will defer answering this question.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" LastActivityDate="2010-09-16T19:50:38.853" CommunityOwnedDate="2010-09-16T19:50:38.853" />
  <row Id="2745" PostTypeId="2" ParentId="2742" CreationDate="2010-09-16T19:58:55.770" Score="2" ViewCount="" Body="&lt;p&gt;The problem with orthogonal dimensions is that the components can be uninterpretable. Thus, while oblique rotation (i.e., nonorthogonal dimensions) is technically less satisfying such a rotation sometimes enhances interpretablity of the resulting components. &lt;/p&gt;&#xA;" LastActivityDate="2010-09-16T19:58:55.770" />
  <row Id="2746" PostTypeId="1" AcceptedAnswerId="2750" CreationDate="2010-09-16T20:39:00.603" Score="5" ViewCount="494" Body="&lt;p&gt;I would like to be able to efficiently generate PSD (positive semi definite) correlation matrices. My method slows down dramatically as I increase the size of matrices to be generated.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Could you suggest any efficient solutions? If you are aware of any examples in Matlab, I would be very thankful&lt;/li&gt;&#xA;&lt;li&gt;When generating a PSD correlation matrix how would you pick the parameters to describe matrices to be generated? An average correlation, standard deviation of correlations, eigenvalues?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="1250" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-09-17T08:47:32.237" LastActivityDate="2010-09-24T09:19:30.617" Title="How to efficiently generate Positive Semi Definite correlation matrices?" Tags="&lt;correlation&gt;&lt;data-generating-process&gt;" AnswerCount="6" CommentCount="4" FavoriteCount="1" />
  <row Id="2747" PostTypeId="2" ParentId="2742" CreationDate="2010-09-16T20:45:22.383" Score="2" ViewCount="" Body="&lt;p&gt;I think there are different opinions or views about PCA, but basically we often think of it as either a &lt;em&gt;reduction technique&lt;/em&gt; (you reduce your features space to a smaller one, often much more &quot;readable&quot; providing you take care of properly centering/standardizing the data when it is needed) or a way to construct &lt;em&gt;latent factors&lt;/em&gt; or dimensions that account for a significant part of the inter-individual dispersion (here, the &quot;individuals&quot; stand for the statistical units on which data are collected; this may be country, people, etc.). In both case, we construct linear combinations of the original variables that account for the maximum of variance (when projected on the principal axis), subject to a constraint of orthogonality between any two principal components. Now, what has been described is purely algebrical or mathematical and we don't think of it as a (generating) model, contrary to what is done in the factor analysis tradition where we include an error term to account for some kind of measurement error. I also like the introduction given by William Revelle in his forthcoming handbook on &lt;a href=&quot;http://personality-project.org/r/book/&quot; rel=&quot;nofollow&quot;&gt;applied psychometrics using R&lt;/a&gt; (Chapter 6), if we want to analyze the structure of a correlation matrix, then&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The first [approach, PCA] is a model&#xA;  that approximates the correlation&#xA;  matrix in terms of the product of&#xA;  components where each component is a&#xA;  weighted linear sum of the variables,&#xA;  the second model [factor analysis] is&#xA;  also an approximation of the&#xA;  correlation matrix by the product of&#xA;  two factors, but the factors in this&#xA;  are seen as causes rather than as&#xA;  consequences of the variables.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In other words, with PCA you are expressing each component (factor) as a linear combination of the variables whereas in FA these are the variables that are expressed as a linear combination of the factors. It is well acknowledged that both methods will generally yield quite similar results (see e.g. Harman, 1976 or Catell, 1978), especially in the &quot;ideal&quot; case where we have a large number of individuals and a good ratio factor:variables (typically varying between 2 and 10 depending on the authors you consider!). This is because, by estimating the diagonals in the correlation matrix (as is done in FA, and these elements are known as the communalities), the error variance is eliminated from the factor matrix. This is the reason why PCA is often used as a way to uncover latent factors or psychological constructs in place of FA developed in the last century. But, as we go on this way, we often want to reach an easier interpretation of the resulting factor structure (or the so-called pattern matrix). And then comes the useful trick of rotating the factorial axis so that we maximize loadings of variables on specific factor, or equivalently reach a &quot;simple structure&quot;. Using orthogonal rotation (e.g. VARIMAX), we preserve the independence of the factors. With oblique rotation (e.g. OBLIMIN, PROMAX), we break it and factors are allowed to correlate. This has been largely debated in the literature, and has lead some authors (not psychometricians, but statisticians in the early 1960's) to conclude that FA is an unfair approach due to the fact that researchers might seek the factor solution that is the more convenient to interpret. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But the point is that rotation methods were originally developed in the context of the FA approach and are now routinely used with PCA. I don't think this contradicts the algorithmic computation of the principal components: You can rotate your factorial axes the way you want, provided you keep in mind that once correlated (by oblique rotation) the interpretation of the factorial space becomes less obvious.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;PCA is routinely used when developing new questionnaires, although FA is probably a better approach in this case because we are trying to extract meaningful factors that take into account measurement errors and whose relationships might be studied on their own (e.g. by factoring out the resulting pattern matrix, we get a second-order factor model). But PCA is also used for checking the factorial structure of already validated ones. Researchers don't really matter about FA vs. PCA when they have, say 500 representative subjects who are asked to rate a 60-item questionnaire tackling five dmensions (this is the case of the &lt;a href=&quot;http://en.wikipedia.org/wiki/Revised_NEO_Personality_Inventory&quot; rel=&quot;nofollow&quot;&gt;NEO-FFI&lt;/a&gt;, for example), and I think they are right because in this case we aren't very much interested in identifying a generating or conceptual model (the term &quot;representative&quot; is used here to alleviate the issue of &lt;em&gt;measurement invariance&lt;/em&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, about the choice of rotation method and why some authors argue against the strict use of orthogonal rotation, I would like to quote Paul Kline, as I did in response to the following question, &lt;a href=&quot;http://stats.stackexchange.com/q/1812/930&quot;&gt;FA: Choosing Rotation matrix, based on “Simple Structure Criteria”&lt;/a&gt;, &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;(...) in the real world, it is not&#xA;  unreasonable to think that factors, as&#xA;  important determiners of behavior,&#xA;  would be correlated. -- P. Kline,&#xA;  &lt;em&gt;Intelligence. The Psychometric View&lt;/em&gt;, 1991, p. 19&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I would thus conclude that, depending on the objective of your study (do you want to highlight the main patterns of your correlation matrix &lt;em&gt;or&lt;/em&gt; do you seek to provide a sensible interpretation of the underlying mechanisms that may have cause you to observe such a correlation matrix), you are up to choose the method that is the most appropriate: This doesn't have to do with the construction of linear combinations, but merely on the way you want to interpret the resulting factorial space.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;References&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Harman, H.H. (1976). &lt;em&gt;Modern Factor Analysis&lt;/em&gt;. Chicago, University of Chicago Press.&lt;/li&gt;&#xA;&lt;li&gt;Cattell, R.B. (1978). &lt;em&gt;The Scientific Use of Factor Analysis&lt;/em&gt;. New York, Plenum.&lt;/li&gt;&#xA;&lt;li&gt;Kline, P. (1991). &lt;em&gt;Intelligence. The Psychometric View&lt;/em&gt;. Routledge.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-16T21:05:45.597" LastActivityDate="2010-09-16T21:05:45.597" />
  <row Id="2748" PostTypeId="1" CreationDate="2010-09-16T20:46:35.240" Score="1" ViewCount="959" Body="&lt;p&gt;I am attempting to find a program that will let me conduct Cox regression on my matched case-control dataset. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please assist.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;p.s. I have STATA, SPSS, and MedCalc&lt;/p&gt;&#xA;" LastEditorUserId="521" LastEditorDisplayName="" LastEditDate="2010-09-18T09:50:10.163" LastActivityDate="2010-10-02T08:15:01.607" Title="How to conduct conditional Cox regression for matched case-control study?" Tags="&lt;regression&gt;&lt;survival&gt;&lt;frailty&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="1" />
  <row Id="2749" PostTypeId="2" ParentId="2748" CreationDate="2010-09-16T20:56:40.193" Score="1" ViewCount="" Body="&lt;p&gt;If you are using Stata, you can just look at the &lt;code&gt;stcox&lt;/code&gt; command. Examples are available from &lt;a href=&quot;http://www.stata.com/capabilities/survivalsession.html&quot; rel=&quot;nofollow&quot;&gt;Stata&lt;/a&gt; or &lt;a href=&quot;http://www.ats.ucla.edu/stat/stata/topics/survival.htm&quot; rel=&quot;nofollow&quot;&gt;UCLA&lt;/a&gt; website.&#xA;Also, take a look at &lt;a href=&quot;http://www.stata-journal.com/sjpdf.html?articlenum=st0070&quot; rel=&quot;nofollow&quot;&gt;Analysis of matched cohort data&lt;/a&gt; from the &lt;em&gt;Stata Journal&lt;/em&gt; (2004 4(3)).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Under R, you can use the &lt;code&gt;coxph()&lt;/code&gt; function from the &lt;code&gt;survival&lt;/code&gt; library.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-09-16T20:56:40.193" CommentCount="2" />
  <row Id="2750" PostTypeId="2" ParentId="2746" CreationDate="2010-09-16T21:05:15.777" Score="5" ViewCount="" Body="&lt;p&gt;You can do it backward: every matrix $C \in \mathbb{R}_{++}^p$   (the set of all symmetric $p \times p$ PSD matrices) can be decomposed as &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$C=O^{T}DO$ where $O$ is an orthonormal matrix&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To get $O$, first generate a random basis $(v_1,...,v_p)$ (where $v_i$ are random vectors, typically in $(-1,1)$). From there, use the Gram-Schmidt orthogonalization process to get $(u_1,....,u_p)=O$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$R$ has a number of packages that can do the G-S orthogonalization of a random basis efficiently, that is even for large dimensions, for example the 'far' package. Although you will find the G-S algorithm on wiki, it's probably better not to re-invent the wheel and go for a matlab implementation (one surely exists, i just can't recommend any).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, $D$ is a diagonal matrices whose elements are all positive (this is, again, easy to generate: generate $p$ random numbers, square them, sort them and place them unto the diagonal of a identity $p$ by $p$ matrix).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for part 'b' of your question, all the characteristics of $PSD$ matrices are defined by there eigenvalues so you can't get more control than that.&lt;/p&gt;&#xA;" OwnerUserId="603" LastEditorUserId="603" LastEditorDisplayName="" LastEditDate="2010-09-16T21:29:47.540" LastActivityDate="2010-09-16T21:29:47.540" />
  <row Id="2751" PostTypeId="2" ParentId="2743" CreationDate="2010-09-16T21:09:15.640" Score="3" ViewCount="" Body="&lt;p&gt;In Stata:&#xA;1) Download the user-written package -&lt;a href=&quot;http://ideas.repec.org/c/boc/bocode/s456798.html&quot; rel=&quot;nofollow&quot;&gt;metan&lt;/a&gt;- from the &lt;a href=&quot;http://ideas.repec.org/s/boc/bocode.html&quot; rel=&quot;nofollow&quot;&gt;SSC software library&lt;/a&gt;:&lt;br/&gt;&#xA;&lt;code&gt;. ssc install metan&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Assuming you have numbers, means and SDs for two groups, the syntax is of the form:&lt;br/&gt;&#xA;&lt;code&gt;. metan n1 mean1 sd1 n0 mean0 sd0&lt;/code&gt;&lt;br/&gt;&#xA;(Cohen's d is the default method for continuous data)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This gives a fixed-effect meta-analysis. Add the 'random' option for a random-effects meta-analysis, i.e.&lt;br/&gt;&#xA;&lt;code&gt;. metan n1 mean1 sd1 n0 mean0 sd0, random&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-metan- has many other options which control various aspects of the output and forest plot.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-09-16T21:09:15.640" CommentCount="2" />
  <row Id="2752" PostTypeId="2" ParentId="2743" CreationDate="2010-09-16T21:10:09.010" Score="3" ViewCount="" Body="&lt;p&gt;For SPSS, look at :&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.mathkb.com/Uwe/Forum.aspx/stat-consult/1201/Effect-Size-in-SPSS&quot; rel=&quot;nofollow&quot;&gt;www.mathkb.com/Uwe/Forum.aspx/stat-consult/1201/Effect-Size-in-SPSS&lt;/a&gt; &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.spsstools.net/Syntax/T-Test/StandardizedEffectsSize.txt&quot; rel=&quot;nofollow&quot;&gt;www.spsstools.net/Syntax/T-Test/StandardizedEffectsSize.txt&lt;/a&gt; (maybe better organized)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;For Stata, I used &lt;a href=&quot;http://ideas.repec.org/c/boc/bocode/s456738.html&quot; rel=&quot;nofollow&quot;&gt;SIZEFX: Stata module to compute effect size correlations&lt;/a&gt; (&lt;code&gt;findit sizefx&lt;/code&gt; at Stata command prompt), but &lt;code&gt;metan&lt;/code&gt; as suggested by @onestop is probably more featured.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-09-16T21:10:09.010" CommentCount="1" />
  <row Id="2753" PostTypeId="2" ParentId="2715" CreationDate="2010-09-16T21:32:16.190" Score="7" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Question your data.&lt;/strong&gt; In the modern era of cheap RAM, we often work on large amounts of data. One 'fat-finger' error or 'lost decimal place' can easily dominate an analysis. Without some basic sanity checking, (or plotting the data, as suggested by others here) one can waste a lot of time. This also suggests using some basic techniques for 'robustness' to outliers.&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-09-16T21:32:16.190" CommunityOwnedDate="2010-09-16T21:32:16.190" />
  <row Id="2755" PostTypeId="2" ParentId="2715" CreationDate="2010-09-16T22:36:18.300" Score="5" ViewCount="" Body="&lt;p&gt;For data organization/management, ensure that when you generate new variables in the dataset (for example, calculating body mass index from height and weight), the original variables are never deleted. A non-destructive approach is best from a reproducibility perspective. You never know when you might mis-enter a command and subsequently need to redo your variable generation. Without the original variables, you will lose a lot of time!&lt;/p&gt;&#xA;" OwnerUserId="561" LastActivityDate="2010-09-16T22:36:18.300" CommunityOwnedDate="2010-09-16T22:36:18.300" />
  <row Id="2756" PostTypeId="2" ParentId="411" CreationDate="2010-09-16T22:38:46.897" Score="2" ViewCount="" Body="&lt;p&gt;I can't give you additional reasons to use the Kolmogorov-Smirnov test.  But, I can give you an important reason not to use it.  It does not fit the tail of the distribution well.  In this regard, a superior distribution fitting test is Anderson-Darling.  As a second best, the Chi Square test is pretty good.  Both are deemed much superior to the K-S test in this regard.&lt;/p&gt;&#xA;" OwnerUserId="1329" LastActivityDate="2010-09-16T22:38:46.897" />
  <row Id="2757" PostTypeId="2" ParentId="2715" CreationDate="2010-09-16T22:39:16.767" Score="7" ViewCount="" Body="&lt;p&gt;Use software that shows the chain of programming logic from the raw data through to the final analyses/results. Avoid software like Excel where one user can make an undetectable error in one cell, that only manual checking  will pick up.&lt;/p&gt;&#xA;" LastActivityDate="2010-09-16T22:39:16.767" CommunityOwnedDate="2010-09-16T22:39:16.767" />
  <row Id="2761" PostTypeId="2" ParentId="305" CreationDate="2010-09-16T23:52:59.773" Score="1" ViewCount="" Body="&lt;p&gt;I would take the opposite view here.  Why bother with the Welch test when the standard unpaired student t test gives you nearly identical results.  I studied this issue a while back and I explored a range of scenarios in an attempt to break down the t test and favor the Welch test.  To do so I used sample sizes up to 5 times greater for one group vs the other.  And, I explored variances up to 25 times greater for one group vs the other.  And, it really did not make any material difference.  The unpaired t test still generated a range of p values that were nearly identical to the Welch test.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can see my work at the following link and focus especially on slide 5 and 6. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.slideshare.net/gaetanlion/unpaired-t-test-family&quot; rel=&quot;nofollow&quot;&gt;http://www.slideshare.net/gaetanlion/unpaired-t-test-family&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1329" LastActivityDate="2010-09-16T23:52:59.773" CommentCount="2" />
  <row Id="2763" PostTypeId="2" ParentId="2746" CreationDate="2010-09-17T00:14:46.137" Score="5" ViewCount="" Body="&lt;p&gt;An even simpler characterization is that for real matrix $A$, $A^TA$ is positive semidefinite. To see why this is the case, one only has to prove that $y^T (A^TA) y \ge 0$ for all vectors $y$ (of the right size, of course). This is trivial: $y^T (A^TA) y = (Ay)^T Ay = ||Ay||$ which is nonnegative. So in Matlab, simply try&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;A = randn(m,n);   %here n is the desired size of the final matrix, and m &amp;gt; n&#xA;X = A' * A;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Depending on the application, this may not give you the distribution of eigenvalues you want; Kwak's answer is much better in that regard. The eigenvalues of &lt;code&gt;X&lt;/code&gt; produced by this code snippet should follow the Marchenko-Pastur distribution.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For simulating the correlation matrices of stocks, say, you may want a slightly different approach:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;k = 7;      % # of latent dimensions;&#xA;n = 100;    % # of stocks;&#xA;A = 0.01 * randn(k,n);  % 'hedgeable risk'&#xA;D = diag(0.001 * randn(n,1));   % 'idiosyncratic risk'&#xA;X = A'*A + D;&#xA;ascii_hist(eig(X));    % this is my own function, you do a hist(eig(X));&#xA;-Inf &amp;lt;= x &amp;lt;  -0.001 : **************** (17)&#xA;-0.001 &amp;lt;= x &amp;lt;   0.001 : ************************************************** (53)&#xA; 0.001 &amp;lt;= x &amp;lt;   0.002 : ******************** (21)&#xA; 0.002 &amp;lt;= x &amp;lt;   0.004 : ** (2)&#xA; 0.004 &amp;lt;= x &amp;lt;   0.005 :  (0)&#xA; 0.005 &amp;lt;= x &amp;lt;   0.007 : * (1)&#xA; 0.007 &amp;lt;= x &amp;lt;   0.008 : * (1)&#xA; 0.008 &amp;lt;= x &amp;lt;   0.009 : *** (3)&#xA; 0.009 &amp;lt;= x &amp;lt;   0.011 : * (1)&#xA; 0.011 &amp;lt;= x &amp;lt;     Inf : * (1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-09-17T00:14:46.137" CommentCount="2" />
  <row Id="2764" PostTypeId="1" CreationDate="2010-09-17T00:30:13.537" Score="1" ViewCount="251" Body="&lt;p&gt;I am attempting to calculate Orwin's (1983) modification of Rosenthal's (1979) Fail-safe N for my meta-analysis of Odds Ratios. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, all the equations I am finding are using Cohen's d, which I cannot calculate (I don't have two groups). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have STATA, SPSS, and MedCalc at my disposal.&lt;/p&gt;&#xA;" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-17T20:18:34.320" LastActivityDate="2010-09-17T20:18:34.320" Title="Calculating Orwin's (1983) modified Fail-safe N in a meta-analysis with Odds Ratio as summary statistic?" Tags="&lt;meta-analysis&gt;&lt;cohens-d&gt;&lt;odds-ratio&gt;" AnswerCount="2" />
  <row Id="2765" PostTypeId="2" ParentId="2730" CreationDate="2010-09-17T00:41:03.327" Score="5" ViewCount="" Body="&lt;p&gt;In my line of work, I've found this one to be quite useful: &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0495597848&quot; rel=&quot;nofollow&quot;&gt;Statistical Methods for Psychology (Howell, 2009)&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1334" LastActivityDate="2010-09-17T00:41:03.327" CommentCount="2" />
  <row Id="2766" PostTypeId="2" ParentId="2639" CreationDate="2010-09-17T02:40:40.300" Score="0" ViewCount="" Body="&lt;p&gt;It's not clear why you must have a confidence interval. As @whuber pointed out, there are better ways to compare distributions. You are losing some information by looking only at the mean. However, if you must, you might want to compute a confidence interval on &lt;code&gt;k1_mean&lt;/code&gt; based on the 40000 observations. See &lt;a href=&quot;http://en.wikipedia.org/wiki/Standard_error_of_the_mean#Assumptions_and_usage&quot; rel=&quot;nofollow&quot;&gt;Wikipedia&lt;/a&gt; for the simple explanation of how to compute this. From there, you could just count the number of the 100 values of &lt;code&gt;k2_mean&lt;/code&gt; that fall into a confidence interval on &lt;code&gt;k1_mean&lt;/code&gt;, and repeat for the third and fourth methods as well. Again, this isn't the best way to compare distributions.&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-09-17T02:40:40.300" CommentCount="2" />
  <row Id="2767" PostTypeId="2" ParentId="2516" CreationDate="2010-09-17T04:11:52.573" Score="1" ViewCount="" Body="&lt;p&gt;Hypothesis testing traditionally focused on p values to derive statistical significance when alpha is less than 0.05 has a major weakness.  And, that is that with a large enough sample size any experiment can eventually reject the null hypothesis and detect trivially small differences that turn out to be statistically significant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is the reason why drug companies structure clinical trials to obtain FDA approval with very large samples.  The large sample will reduce the standard error to close to zero.  This in turn will artificially boost the t stat and commensurately lower the p value to close to 0%. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I gather within scientific communities that are not corrupted by economic incentives and related conflict of interest hypothesis testing is moving away from any p value measurements towards Effect Size measurements.  This is because the unit of statistical distance or differentiation in Effect Size analysis is the standard deviation instead of the standard error.  And, the standard deviation is completely independent from the sample size.  The standard error on the other hand is totally dependent from the sample size. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, anyone who is skeptical of hypothesis testing reaching statistically significant results based on large samples and p value related methodologies is right to be skeptical.  They should rerun the analysis using the same data but using instead Effect Size statistical tests.  And, then observe if the Effect Size is deemed material or not.  By doing so, you could observe that a bunch of differences that are statistically significant are associated with Effect Size that are immaterial.  That's what clinical trial researchers sometimes mean when a result is statistically significant but not &quot;clinically significant.&quot;  They mean by that one treatment may be better than placebo, but the difference is so marginal that it would make no difference to the patient within a clinical context.    &lt;/p&gt;&#xA;" OwnerUserId="1329" LastActivityDate="2010-09-17T04:11:52.573" />
  <row Id="2768" PostTypeId="1" CreationDate="2010-09-17T04:36:00.473" Score="1" ViewCount="261" Body="&lt;p&gt;I was asked such a question as &quot;Did you do any consistency check in your daily work?&quot; during a phone interview for a Biostatistician position. I don't know what to answer. Any information is appreciated.&lt;/p&gt;&#xA;" OwnerUserId="1336" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-09-17T11:44:38.123" LastActivityDate="2010-09-17T11:44:38.123" Title="What is a consistency check?" Tags="&lt;validation&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="2769" PostTypeId="2" ParentId="2742" CreationDate="2010-09-17T05:01:12.770" Score="1" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Basic Points&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Rotation can make interpretation of components clearer&lt;/li&gt;&#xA;&lt;li&gt;Oblique rotation often makes more theoretical sense. I.e., Observed variables can be explained in terms of a smaller number of correlated components.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Example&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;10 tests all measuring ability with some measuring verbal and some measuring spatial ability. All tests are intercorrelated, but intercorrelations within verbal or within spatial tests are greater than across test type. A parsimonious PCA might involve two correlated components, a verbal and a spatial. Theory and research suggests that these two abilities are correlated. Thus, an oblique rotation makes theoretical sense.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-09-17T05:01:12.770" />
  <row Id="2770" PostTypeId="1" AcceptedAnswerId="2773" CreationDate="2010-09-17T05:05:49.097" Score="5" ViewCount="139" Body="&lt;p&gt;I'm an undergraduate statistics student looking for a good treatment of clinical trials analysis. The text should cover the fundamentals of experimental design, blocking, power analysis, latin squares design, and cluster randomization designs, among other topics. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have an undergraduate knowledge of mathematical statistics and real analysis, but if there's a fantastic text that requires a bit higher level of statistics or analysis, I can work up to it.&lt;/p&gt;&#xA;" OwnerUserId="1118" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-09-17T06:36:30.523" LastActivityDate="2011-05-11T23:19:08.927" Title="Good text on Clinical Trials?" Tags="&lt;books&gt;&lt;teaching&gt;&lt;clinical-trials&gt;" AnswerCount="4" />
  <row Id="2771" PostTypeId="2" ParentId="2516" CreationDate="2010-09-17T05:49:34.810" Score="3" ViewCount="" Body="&lt;p&gt;The short answer is &quot;no&quot;. Research on hypothesis testing in the asymptotic regime of infinite observations and multiple hypotheses has been very, very active in the past 15-20 years, because of microarray data and financial data applications. The long answer is in the course page of Stat 329, &quot;Large-Scale Simultaneous Inference&quot;, taught in 2010 by Brad Efron. A &lt;a href=&quot;http://www-stat.stanford.edu/~omkar/329/chapter2.pdf&quot; rel=&quot;nofollow&quot;&gt;full chapter&lt;/a&gt; is devoted to large-scale hypothesis testing.&lt;/p&gt;&#xA;" OwnerUserId="30" LastActivityDate="2010-09-17T05:49:34.810" />
  <row Id="2772" PostTypeId="1" CreationDate="2010-09-17T06:11:36.290" Score="7" ViewCount="380" Body="&lt;p&gt;In two papers in &lt;a href=&quot;http://dx.doi.org/10.1016/0304-405X%2886%2990027-9&quot; rel=&quot;nofollow&quot;&gt;1986&lt;/a&gt; and &lt;a href=&quot;http://dx.doi.org/10.1016/0304-405X%2888%2990062-1&quot; rel=&quot;nofollow&quot;&gt;1988&lt;/a&gt;, Connor and Korajczyk proposed an approach to modeling asset returns. Since these time series have usually more assets than time period observations, they proposed to perform a PCA on cross-sectional covariances of asset returns. They call this method Asymptotic Principal Component Analysis (APCA, which is rather confusing, since the audience thinks immediately of asymptotic properties of PCA). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have worked out the equations, and the two approaches seem numerically equivalent. The asymptotics of course differ, since convergence is proved for $N \rightarrow \infty$ rather than $T \rightarrow \infty$. My question is: has anyone used APCA and compared to PCA? Are there concrete differences? If so, which ones?&lt;/p&gt;&#xA;" OwnerUserId="30" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-09-17T10:58:42.450" LastActivityDate="2011-05-20T01:48:06.953" Title="What is the difference between Principal Component Analysis and Asymptotic Principal Component Analysis" Tags="&lt;pca&gt;&lt;econometrics&gt;" AnswerCount="1" CommentCount="3" FavoriteCount="2" />
  <row Id="2773" PostTypeId="2" ParentId="2770" CreationDate="2010-09-17T06:32:58.730" Score="4" ViewCount="" Body="&lt;p&gt;I would definitively recommend &lt;a href=&quot;http://books.google.fr/books?id=HXMUEjZ4vyAC&amp;amp;dq=Design+and+Analysis+of+Clinical+Trials%3a+Concepts+and+Methodologies&amp;amp;printsec=frontcover&amp;amp;source=bn&amp;amp;hl=fr&amp;amp;ei=YQuTTPG-KseLswbt9JH5CQ&amp;amp;sa=X&amp;amp;oi=book_result&amp;amp;ct=result&amp;amp;resnum=4&amp;amp;ved=0CCwQ6AEwAw&quot; rel=&quot;nofollow&quot;&gt;Design and Analysis of Clinical Trials: Concepts and Methodologies&lt;/a&gt; which seems actually the most complete one given your request.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://eu.wiley.com/WileyCDA/WileyTitle/productCd-0470018771.html&quot; rel=&quot;nofollow&quot;&gt;Statistical Issues in Drug Development&lt;/a&gt; also covers a broad range of concepts but is less oriented toward design of experiment. &lt;a href=&quot;http://books.google.fr/books?id=LEDJ-6bPa8AC&amp;amp;printsec=frontcover&amp;amp;dq=Statistics+Applied+to+Clinical+Trials&amp;amp;source=bl&amp;amp;ots=LwzvnasZuk&amp;amp;sig=OkDTIEnTaaONeDG5Ik1XVOhIeNM&amp;amp;hl=fr&amp;amp;ei=yAiTTNHoN8LEswbg2_D4CQ&amp;amp;sa=X&amp;amp;oi=book_result&amp;amp;ct=result&amp;amp;resnum=2&amp;amp;ved=0CCQQ6AEwAQ&quot; rel=&quot;nofollow&quot;&gt;Statistics Applied to Clinical Trials&lt;/a&gt; includes more technical stuff, but mainly focus on crossover trials and applications in epidemiology. And there is always the most famous Rothman, &lt;a href=&quot;http://books.google.fr/books?id=Z3vjT9ALxHUC&amp;amp;dq=rothman+modern+epidemiology&amp;amp;printsec=frontcover&amp;amp;source=bn&amp;amp;hl=fr&amp;amp;ei=MAmTTMF2zcuzBve20fgJ&amp;amp;sa=X&amp;amp;oi=book_result&amp;amp;ct=result&amp;amp;resnum=4&amp;amp;ved=0CDMQ6AEwAw&quot; rel=&quot;nofollow&quot;&gt;Modern Epidemiology&lt;/a&gt;, which provides valuable help for interpretation and examples of application of clinical biostatistics. Finally, T&lt;a href=&quot;http://eu.wiley.com/WileyCDA/WileyTitle/productCd-0471349917.html&quot; rel=&quot;nofollow&quot;&gt;he Design and Analysis of Clinical Experiments&lt;/a&gt; is more centered onto the analysis of specific experimental settings, but it does not address other points and is a bit older.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-09-17T06:32:58.730" />
  <row Id="2774" PostTypeId="2" ParentId="2768" CreationDate="2010-09-17T06:43:26.070" Score="3" ViewCount="" Body="&lt;p&gt;I suppose this has to do with some form of Quality Control about &lt;em&gt;data integrity&lt;/em&gt;, and more specifically that you regularly check that your working database isn't corrupted (due to error during transfer, copy, or after an update or a sanity check). This may also mean ensuring that your intermediate computation are double-checked (either manually or through additional code or macros in your statistical software).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other information may be found here: the ICH E6 (R1) reference guide about &lt;a href=&quot;http://www.ema.europa.eu/docs/en_GB/document_library/Scientific_guideline/2009/09/WC500002874.pdf&quot; rel=&quot;nofollow&quot;&gt;Guideline for Good Clinical Practice&lt;/a&gt; from the EMEA, &lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2213906/&quot; rel=&quot;nofollow&quot;&gt;Guidelines on Good Clinical Laboratory Practice&lt;/a&gt;, or &lt;a href=&quot;http://www.nia.nih.gov/ResearchInformation/CTtoolbox/&quot; rel=&quot;nofollow&quot;&gt;Clinical Research Study Investigator's Toolbox&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-17T06:49:06.947" LastActivityDate="2010-09-17T06:49:06.947" />
  <row Id="2775" PostTypeId="2" ParentId="2746" CreationDate="2010-09-17T06:52:22.090" Score="2" ViewCount="" Body="&lt;p&gt;The simplest method is the one above, which is a simulation of a random dataset and the computation of the &lt;a href=&quot;http://en.wikipedia.org/wiki/Gramian_matrix&quot; rel=&quot;nofollow&quot;&gt;Gramian&lt;/a&gt;. A word of caution: The resulting matrix will not be uniformly random, in that its decomposition, say $U^TSU$ will have rotations not distributed according to the Haar Measure. If you want to have &quot;uniformly distributed&quot; PSD matrices then you can use any of the approaches described &lt;a href=&quot;http://en.wikipedia.org/wiki/Rotation_matrix#Uniform_random_rotation_matrices&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="30" LastEditorUserId="30" LastEditorDisplayName="" LastEditDate="2010-09-17T14:39:00.353" LastActivityDate="2010-09-17T14:39:00.353" CommentCount="3" />
  <row Id="2776" PostTypeId="2" ParentId="2764" CreationDate="2010-09-17T07:10:43.227" Score="0" ViewCount="" Body="&lt;p&gt;So, perhaps check these additional resources: &lt;a href=&quot;http://j.mp/d8znoP&quot; rel=&quot;nofollow&quot;&gt;http://j.mp/d8znoP&lt;/a&gt; for SPSS. Don't know about Stata.&#xA;There is some R code about fail-safe N in the following handout: &lt;a href=&quot;http://www.biostat.uzh.ch/teaching/phd/doktorandenseminar/schroedle.pdf&quot; rel=&quot;nofollow&quot;&gt;Tests for funnel plot asymmetry and failsafe N&lt;/a&gt;, but I didn't check on the &lt;a href=&quot;http://www.metaanalysis.com&quot; rel=&quot;nofollow&quot;&gt;www.metaanalysis.com&lt;/a&gt; website. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Otherwise, &lt;a href=&quot;http://www.clintools.com/&quot; rel=&quot;nofollow&quot;&gt;ClinTools Software&lt;/a&gt; may be an option (I hope the demo version let you do some computation on real data), or better the &lt;a href=&quot;http://www.biomedcentral.com/1471-2288/6/50&quot; rel=&quot;nofollow&quot;&gt;MIX&lt;/a&gt; software.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-09-17T07:10:43.227" />
  <row Id="2777" PostTypeId="1" AcceptedAnswerId="2781" CreationDate="2010-09-17T07:12:27.057" Score="6" ViewCount="430" Body="&lt;p&gt;&lt;strong&gt;Context&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Imagine you had a longitudinal study which measured a dependent variable (DV) once a week for 20 weeks on 200 participants. Although I'm interested in general, typical DVs that I'm thinking of include job performance following hire or various well-being measures following a clinical psychology intervention.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know that multilevel modelling can be used to model the relationship between time and the DV. You can also allow coefficients (e.g., intercepts, slopes, etc.) to vary between individuals and estimate the particular values for participants. But what if when visually inspecting the data you find that the relationship between time and the DV is any one of the following:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;different in functional form (perhaps some are linear and others are exponential or some have a discontinuity)&lt;/li&gt;&#xA;&lt;li&gt;different in error variance (some individuals are more volatile from one time point to the next)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Questions&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What would be a good way to approach modelling data like this? &lt;/li&gt;&#xA;&lt;li&gt;Specifically, what approaches are good at identifying different types of relationships, and categorising individuals with regards to their type?&lt;/li&gt;&#xA;&lt;li&gt;What implementations exist in R for such analyses?&lt;/li&gt;&#xA;&lt;li&gt;Are there any references on how to do this: textbook or actual application?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-09-19T09:58:00.853" Title="Modelling longitudinal data where the effect of time varies in functional form between individuals" Tags="&lt;repeated-measures&gt;&lt;random-effects-model&gt;&lt;latent-class&gt;" AnswerCount="3" FavoriteCount="3" />
  <row Id="2778" PostTypeId="2" ParentId="2730" CreationDate="2010-09-17T08:43:52.880" Score="6" ViewCount="" Body="&lt;p&gt;The classics I think are Winer and Kirk, both cover essentially only ANOVA and ANCOVA. You can probably get used copys for cheap (e.g., I own a Winer second edition from 71 bought via AMAZON for less than 10$):&lt;br&gt;&#xA;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0070709823&quot; rel=&quot;nofollow&quot;&gt;Winer - Statistical Principles In Experimental Design&lt;/a&gt;&lt;br&gt;&#xA;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0534250920&quot; rel=&quot;nofollow&quot;&gt;Kirk - Experimental Design&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A more contemporary book is the one by Maxwell &amp;amp; Delaney. Besides ANOVA and ANCOVA it covers other methods, e.g., multivariate and multilevel:&lt;br&gt;&#xA;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0805837183&quot; rel=&quot;nofollow&quot;&gt;Maxwell &amp;amp; Delaney - Designing Experiments and Analyzing Data: A Model Comparison Perspective&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Perhaps it is the best to go with this last one. It is pretty good.&lt;/p&gt;&#xA;" OwnerUserId="442" LastActivityDate="2010-09-17T08:43:52.880" CommentCount="1" />
  <row Id="2779" PostTypeId="2" ParentId="2711" CreationDate="2010-09-17T08:44:09.753" Score="1" ViewCount="" Body="&lt;p&gt;For the problem above, I have used a really simple metric.&#xA;I wanted to asses how good is my hit!&#xA;If I have, for example, errors 4, 2, 0.002, 3, 6 then I choose h1=0.002 as hit, and h2=2 as the closest error.&#xA;X/h1 + X/h2 = 100% =&gt; X/h1 and X/h2 is %.&lt;/p&gt;&#xA;" OwnerUserId="1313" LastActivityDate="2010-09-17T08:44:09.753" />
  <row Id="2780" PostTypeId="2" ParentId="2764" CreationDate="2010-09-17T08:57:59.967" Score="1" ViewCount="" Body="&lt;p&gt;Normally you always find ways to convert effect size. For example you can calculate $r$ from $d$ and back. So you surely will be able to converts odds ratio to Cohen's $d$.&lt;br&gt;&#xA;One book I usually found a good resource for stuff like that is the one by Rosenthal &amp;amp; Rosnow. I think it was this one:&lt;br&gt;&#xA;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0073531960&quot; rel=&quot;nofollow&quot;&gt;Rosenthal &amp;amp; Rosnow - Essentials of Behavioral Research: Methods and Data Analysis&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But in your case you should take a look at his paper, it will probably solve your problem (but I haven't looked at it):&lt;br&gt;&#xA;&lt;a href=&quot;http://www.citeulike.org/group/3472/article/2833123&quot; rel=&quot;nofollow&quot;&gt;Chinn, S. (2000). A simple method for converting an odds ratio to effect size for use in meta-analysis. Statistic in Medicine, 19, 3127-3131.&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="442" LastActivityDate="2010-09-17T08:57:59.967" CommentCount="1" />
  <row Id="2781" PostTypeId="2" ParentId="2777" CreationDate="2010-09-17T08:59:36.203" Score="5" ViewCount="" Body="&lt;p&gt;I would suggest to look at the following three directions:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;em&gt;longitudinal clustering&lt;/em&gt;: this is unsupervised, but you use k-means approach relying on the Calinsky criterion for assessing quality of the partitioning (package &lt;a href=&quot;http://cran.r-project.org/web/packages/kml/index.html&quot; rel=&quot;nofollow&quot;&gt;kml&lt;/a&gt;, and references included in the online help); so basically, it won't help identifying specific shape for individual time course, but just separate homogeneous evolution profile&lt;/li&gt;&#xA;&lt;li&gt;some kind of &lt;em&gt;latent growth curve&lt;/em&gt; accounting for heteroscedasticity: my best guess would be to look at the extensive references around &lt;a href=&quot;http://www.statmodel.com/&quot; rel=&quot;nofollow&quot;&gt;MPlus&lt;/a&gt; software, especially the FAQ and mailing. I've also heard of random effect multiplicative heteroscedastic model (try googling around those keywords). I find these papers (&lt;a href=&quot;http://web.mit.edu/polisci/research/apss/Prior.PI%20Development.MIT.pdf&quot; rel=&quot;nofollow&quot;&gt;1&lt;/a&gt;, &lt;a href=&quot;http://www.laser.uni-erlangen.de/papers/paper/98.pdf&quot; rel=&quot;nofollow&quot;&gt;2&lt;/a&gt;) interesting, but I didn't look at them in details. I will update with references on neuropsychological assessment once back to my office.&lt;/li&gt;&#xA;&lt;li&gt;&lt;em&gt;functional PCA&lt;/em&gt; (&lt;a href=&quot;http://cran.r-project.org/web/packages/fpca/index.html&quot; rel=&quot;nofollow&quot;&gt;fpca&lt;/a&gt; package) but it may be worth looking at &lt;a href=&quot;http://www.stats.ox.ac.uk/~silverma/fdacasebook/&quot; rel=&quot;nofollow&quot;&gt;functional data analysis&lt;/a&gt; &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Other references (just browsed on the fly):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Willett &amp;amp; Bull (2004), &lt;a href=&quot;http://www.ucs.louisiana.edu/~rmm2440/Willett_and_Bub_2004.pdf&quot; rel=&quot;nofollow&quot;&gt;Latent Growth Curve Analysis&lt;/a&gt; -- the authors use LGC on non-linear reading trajectories&lt;/li&gt;&#xA;&lt;li&gt;Welch (2007), &lt;a href=&quot;http://etd.library.pitt.edu/ETD/available/etd-07312007-173433/unrestricted/WelchGregDissertation_080707.pdf&quot; rel=&quot;nofollow&quot;&gt;Model Fit and Interpretation of Non-Linear Latent Growth Curve Models&lt;/a&gt; -- a PhD on modeling non-linear change in the context of latent growth modeling&lt;/li&gt;&#xA;&lt;li&gt;Berkey CS, Laird NM (1986). &lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pubmed/3707041&quot; rel=&quot;nofollow&quot;&gt;Nonlinear growth curve analysis: estimating the population parameters&lt;/a&gt;. Ann Hum Biol. 1986 Mar-Apr;13(2):111-28&lt;/li&gt;&#xA;&lt;li&gt;Rice (2003), &lt;a href=&quot;http://www.google.fr/url?sa=t&amp;amp;source=web&amp;amp;cd=8&amp;amp;ved=0CFsQFjAH&amp;amp;url=http%3A%2F%2Fciteseerx.ist.psu.edu%2Fviewdoc%2Fdownload%3Fdoi%3D10.1.1.6.8240%26rep%3Drep1%26type%3Dpdf&amp;amp;rct=j&amp;amp;q=time%20varying%20individual%20trajectories%20functional%20data%20analysis&amp;amp;ei=6W-TTPSPE4uQ4Qbgtei4BA&amp;amp;usg=AFQjCNGQwrXuydM3-zfnSU51Am5IrQdc3A&amp;amp;sig2=VclEUPG0CLoiYTI2GAJbfQ&amp;amp;cad=rja&quot; rel=&quot;nofollow&quot;&gt;Functional and Longitudinal Data Analysis: Perspectives on Smoothing&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Wu, Fan and Müller (2007). &lt;a href=&quot;http://orfe.princeton.edu/~jqfan/papers/07/WuFanMueller.pdf&quot; rel=&quot;nofollow&quot;&gt;Varying-Coefficient Functional Linear Regression&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-17T13:52:39.000" LastActivityDate="2010-09-17T13:52:39.000" CommentCount="5" />
  <row Id="2782" PostTypeId="2" ParentId="2715" CreationDate="2010-09-17T09:40:03.130" Score="9" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;If you're deciding between two ways of analysing your data, try it both ways and see if it  makes a difference.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is useful in many contexts:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;To transform or not transform&lt;/li&gt;&#xA;&lt;li&gt;Non-parametric or parameteric test&lt;/li&gt;&#xA;&lt;li&gt;Spearman's or Pearson's correlation&lt;/li&gt;&#xA;&lt;li&gt;PCA or factor analysis&lt;/li&gt;&#xA;&lt;li&gt;Whether to use the arithmetic mean or a robust estimate of the mean&lt;/li&gt;&#xA;&lt;li&gt;Whether to include a covariate or not&lt;/li&gt;&#xA;&lt;li&gt;Whether to use list-wise deletion, pair-wise deletion, imputation, or some other method of missing values replacement&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;This shouldn't absolve one from thinking through the issue, but it at least gives a sense of the degree to which substantive findings are robust to the choice.&lt;/p&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-09-19T10:13:15.170" LastActivityDate="2010-09-19T10:13:15.170" CommunityOwnedDate="2010-09-17T09:40:03.130" CommentCount="4" />
  <row Id="2783" PostTypeId="2" ParentId="2746" CreationDate="2010-09-17T11:10:21.900" Score="3" ViewCount="" Body="&lt;p&gt;You haven't specified a distribution for the matrices. Two common ones are the Wishart and inverse Wishart distributions. The &lt;a href=&quot;http://en.wikipedia.org/wiki/Wishart_distribution#Bartlett_decomposition&quot; rel=&quot;nofollow&quot;&gt;Bartlett decomposition&lt;/a&gt; gives a Cholesky factorisation of a random Wishart matrix (which can also be efficiently solved to obtain a random inverse Wishart matrix). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In fact, the Cholesky space is a convenient way to generate other types of random PSD matrices, as you only have to ensure that the diagonal is non-negative.&lt;/p&gt;&#xA;" OwnerUserId="495" LastActivityDate="2010-09-17T11:10:21.900" CommentCount="4" />
  <row Id="2784" PostTypeId="2" ParentId="2061" CreationDate="2010-09-17T11:11:41.793" Score="2" ViewCount="" Body="&lt;p&gt;In the category (3) of humorous videos, check out &lt;a href=&quot;http://video.google.com/videoplay?docid=489221653835413043#&quot; rel=&quot;nofollow&quot;&gt;'Statz rappers'&lt;/a&gt;; general interest.  (Pretty funny even to older people ;-).)&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-09-17T11:11:41.793" CommunityOwnedDate="2010-09-17T11:11:41.793" />
  <row Id="2785" PostTypeId="2" ParentId="2768" CreationDate="2010-09-17T11:41:19.193" Score="5" ViewCount="" Body="&lt;p&gt;To chl's list, which focuses on frank data processing errors, I would add checks for subtler errors to address the following questions and issues (given in no particular order and certainly incomplete):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(1) Assuming database integrity, are the data reasonable?  Do they roughly conform with expectations or conventional models, or would they surprise someone familiar with similar data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(2) Are the data internally consistent?  For example, if one field is supposed to be the sum of two others, is it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(3) How complete are the data?  Are they what were specified during the data collection planning phase?  Are there any extra data that were not planned for?  If so, why are they there?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(4) Most analyses implicitly or explicitly model the data in a parsimonious way and include the possibility of variation from the general description.  Each such model suggests its own particular way to identify outliers--the data that deviate remarkably from the general description.  Were attempts made to identify and understand outliers at each stage of exploration and analysis?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(5) In many cases it is possible for the analyst to introduce additional data into the analysis for quality checking and insight.  For example, many data sets in the natural and social sciences as well as business include (at least implicitly) location information: identifiers of Census regions; names of countries, states, counties; customer zip codes; and so on.  Even if--perhaps especially if--spatial correlation is not an element of the EDA or modeling, the analyst can join the data to geographic representations of the locations and map them to look for patterns and outliers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(6) One of the most insidious errors that can creep into an analysis is loss of data.  When extracting fields, summarizing data, reformatting datasets, etc., if one or two items are dropped from a large dataset often there will be nothing to flag it.  But occasionally something important is lost, to one's extreme embarrassment if it is ever discovered.  Simple checks--such as comparing before and after counts and totals of data--need to occur routinely to guard against such things.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(7) Another insidious error is associated with type conversion in digital computing.  For example, recently I had to construct a key (for matching two data files) out of a floating point field.  The software (Stata) imported the field as a single precision float in one file but, for whatever reason, as a double precision float in another file.  Most of the time the values matched but, in a few cases due to different rounding, they did not.  Some data were lost as a result.  I caught this only due to the application of (6).  In general, it pays to check for consistency of field data types: ints vs. floats, lengths of strings, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(8) If a spreadsheet is ever used at &lt;em&gt;any&lt;/em&gt; stage of analysis, expect the worst.  The problem is that even a stray keystroke can invisibly corrupt the data.  When the results are critical, it pays to keep going back and forth--export to the spreadsheet, do the analysis, import back, and systematically compare--to make sure nothing untoward happened.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(9) Whenever a database is updated, it is worthwhile to pause and perform systematic, complete comparisons with the old one to make sure nothing was lost, changed, or corrupted in the process.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(10) At a higher level, whenever an estimate is performed (such as a regression, PCA, whatever), it can be worthwhile to perform it using a different technique to check for sensitivities or even possible errors in the code.  E.g., follow an OLS regression by some form of robust regression and compare the coefficients.  For important results, it can be comforting to obtain the answers using two (or more) different software platforms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Perhaps the best kind of general &quot;consistency check&quot; anyone can perform is to graph everything, early and often.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-09-17T11:41:19.193" />
  <row Id="2786" PostTypeId="2" ParentId="2746" CreationDate="2010-09-17T12:12:34.027" Score="4" ViewCount="" Body="&lt;p&gt;As a variation on kwak's answer: generate a diagonal matrix $\mathbf{D}$ with random nonnegative eigenvalues from a distribution of your choice, and then perform a similarity transformation $\mathbf{A}=\mathbf{Q}\mathbf{D}\mathbf{Q}^T$ with $\mathbf{Q}$ a &lt;a href=&quot;http://dx.doi.org/10.1137/0717034&quot; rel=&quot;nofollow&quot;&gt;Haar-distributed pseudorandom orthogonal matrix&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="830" LastActivityDate="2010-09-17T12:12:34.027" CommentCount="5" />
  <row Id="2787" PostTypeId="1" AcceptedAnswerId="2790" CreationDate="2010-09-17T12:52:07.833" Score="1" ViewCount="79" Body="&lt;p&gt;It was recommended to me on the &lt;a href=&quot;http://ui.stackexchange.com/questions/1543/whats-recommended-for-the-number-of-users-to-use-in-qualitative-and-quantitative&quot;&gt;User Interface Stack Exchange&lt;/a&gt; that I also ask this question here. So....&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm currently rebuilding our entire intranet from scratch, mostly because the tech behind is out-dated and it has been proved that a lot of information is difficult to find.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Though that is beside the point, what I am wondering is what would be the optimum amount of users to use for qualitative and quantitative testing with a userbase of around 1000 users?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a general rule of thumb for both based on the total number of users you have? or is it just say 5 for qualitative and 10 for quantitative?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What would be the best approach?&lt;/p&gt;&#xA;" OwnerUserId="593" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-02-23T01:02:50.827" LastActivityDate="2011-02-23T01:02:50.827" Title="What's recommended for the number of users to use in qualitative and quantitative usability studies?" Tags="&lt;sample-size&gt;&lt;testing&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="2788" PostTypeId="1" AcceptedAnswerId="2789" CreationDate="2010-09-17T12:54:10.173" Score="3" ViewCount="203" Body="&lt;p&gt;I hope you all won't mind a basic question. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We are examining the effects of a compound at various concentration on the behaviour of an organism. The compound is administered once at the beginning of the time course.  Observations are made 1/minute for 2 hours.  Each concentration is applied to 12 individuals.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we repeat this experiment with different individuals, what is the best way to combine the 2 experiments? The issue that we're having is that although the distribution and shape of the data from each experiment is very similar, the plot of the medians by time is usually significantly shifted up or down between exp1 and exp2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It will no doubt be very evident that I'm unskilled in statistics. If this question needs more information to be answerable, let me know.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks,&#xA;Jen&lt;/p&gt;&#xA;" OwnerUserId="1079" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-17T13:42:48.570" LastActivityDate="2010-09-17T13:50:02.507" Title="Combining repeated experiments into one dataset" Tags="&lt;multilevel-analysis&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="2789" PostTypeId="2" ParentId="2788" CreationDate="2010-09-17T13:14:32.587" Score="5" ViewCount="" Body="&lt;p&gt;Just add &quot;experiment&quot; as an effect to your model, that should account for the shift between experiments and let you gain the power of increased N across experiments to detect effects of concentration and time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In R, if using ANOVA and treating time as a factor (e.g. not numeric), then do:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(ez)&#xA;&#xA;ezANOVA(&#xA;    data = my_data&#xA;    , dv = .(my_dv)&#xA;    , wid = .(individual)&#xA;    , within = .(time)&#xA;    , between = .(concentration,experiment)&#xA;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, this:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;treats experiment as a fixed effect, whereas it might more reasonably be considered a random effect (thanks &lt;a href=&quot;http://stats.stackexchange.com/users/442/henrik&quot;&gt;Henrik&lt;/a&gt;!)&lt;/li&gt;&#xA;&lt;li&gt;treats time as non-continuous&lt;/li&gt;&#xA;&lt;li&gt;assumes sphericity across the levels of time&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;An approach that solves all three issues is to employ a mixed effects model. If you think that the effect of time is linear, then leave time as a numeric variable and do:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(lme4)&#xA;&#xA;lmer(&#xA;    data = my_data&#xA;    , formula = my_dv ~ time*concentration+(1|individual)+(1|experiment)&#xA;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you don't think time is linear, you could convert it to a factor and repeat the above, or use generalized additive mixed modelling:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(gamm4)&#xA;fit &amp;lt;- gamm4(&#xA;    data = my_data&#xA;    , formula = my_dv ~ time+concentration+s(time,by=concentration,bs='tp')&#xA;    , random = ~ (1|individual) + (1|experiment)&#xA;)&#xA;print(fit$gam)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;That assumes that experiment only shifts the time function, but lets concentration change the shape of the time function. I have a hard time figuring out how to visualise the results from single gamm4 fits, so I usually obtain the fitted model's predictions across the fixed-effects space then bootstrap (in your case, sampling individuals with replacement within each experiment) confidence intervals around these predictions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, all of the above assume that residuals are gaussian; if you're dealing with anything different (eg. binomial data), then you need to change the &quot;family&quot; arguments of lmer and gamm4 (ezANOVA can't do anything but gaussian).&lt;/p&gt;&#xA;" OwnerUserId="364" LastEditorUserId="364" LastEditorDisplayName="" LastEditDate="2010-09-17T13:46:22.303" LastActivityDate="2010-09-17T13:46:22.303" CommentCount="2" />
  <row Id="2790" PostTypeId="2" ParentId="2787" CreationDate="2010-09-17T13:27:27.217" Score="3" ViewCount="" Body="&lt;p&gt;Not an exact answer to your question (which is kind of not understandable enough for conrete answers, see Srikant's comment), but a more general one:&lt;br&gt;&#xA;If you want to do statistical testing on your data it is advisable to make sure that your sample conforms to the &lt;strong&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Central_limit_theorem&quot; rel=&quot;nofollow&quot;&gt;central limit theorem&lt;/a&gt;&lt;/strong&gt;. That is, the distribution of sample means from which your sample is drawn is normal (i.e., bell-shaped). Then you can use a lot of nice methods (as long as the other assumptions are met) on your data that try to estimate population parameters from your sample. &lt;strong&gt;A rule of thumb about the central limit theorem is that you need at least 30 participants (or 20, see &lt;a href=&quot;http://stats.stackexchange.com/questions/276/how-large-should-a-sample-be-for-a-given-estimation-technique-and-parameters&quot;&gt;here&lt;/a&gt;)&lt;/strong&gt;.&lt;br&gt;&#xA;My advice: If you think that 10 could be enough, use at least 20.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For qualitative testing, I would guess, just use participants until you get a picture of what the issues are. Ask a couple of people to test your stuff and ask them to tell you their impression. If they agree to some degree, there you go. I think 5 is a good starting point (caveat: this is a non-theoretically answer, based on my stomach, and I just had lunch).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are interested in polls from your sample, &lt;a href=&quot;http://stats.stackexchange.com/questions/166/how-do-you-decide-the-sample-size-when-polling-a-large-population&quot;&gt;this discussion&lt;/a&gt; could be useful.&lt;/p&gt;&#xA;" OwnerUserId="442" LastActivityDate="2010-09-17T13:27:27.217" />
  <row Id="2791" PostTypeId="2" ParentId="2770" CreationDate="2010-09-17T13:36:22.340" Score="2" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/019928511X&quot; rel=&quot;nofollow&quot;&gt;Experimental Design for the Life Sciences&lt;/a&gt;, by Ruxton &amp;amp; Colegrave, is a nice book and is aimed primarily at undergraduates.&lt;/p&gt;&#xA;" OwnerUserId="266" LastActivityDate="2010-09-17T13:36:22.340" />
  <row Id="2792" PostTypeId="2" ParentId="2788" CreationDate="2010-09-17T13:39:43.130" Score="2" ViewCount="" Body="&lt;p&gt;I think a way in which you could analyze the two experiments together is by defining a &lt;a href=&quot;http://en.wikipedia.org/wiki/Hierarchical_linear_modeling&quot; rel=&quot;nofollow&quot;&gt;multilevel/hierarchical model&lt;/a&gt;. The individuals are nested within each experiment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The standard for this approach is &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/052168689X&quot; rel=&quot;nofollow&quot;&gt;Gelman's book&lt;/a&gt; (I think).&lt;br&gt;&#xA;The &lt;a href=&quot;http://www.sciencedirect.com/science?_ob=PublicationURL&amp;amp;_tockey=%23TOC%236896%232008%23999409995%23701947%23FLA%23&amp;amp;_cdi=6896&amp;amp;_pubType=J&amp;amp;view=c&amp;amp;_auth=y&amp;amp;_acct=C000054038&amp;amp;_version=1&amp;amp;_urlVersion=0&amp;amp;_userid=1634520&amp;amp;md5=f69a3f5826f12784ec9f11fa8c4210f0&quot; rel=&quot;nofollow&quot;&gt;Journal of Memory and Language had a special issue on analyzing data in 2008&lt;/a&gt; which covered hierarchical models and even some examples in &lt;code&gt;R&lt;/code&gt;.&lt;br&gt;&#xA;Other's here can probably provide good web-resources as well.&lt;/p&gt;&#xA;" OwnerUserId="442" LastEditorUserId="442" LastEditorDisplayName="" LastEditDate="2010-09-17T13:50:02.507" LastActivityDate="2010-09-17T13:50:02.507" CommentCount="2" />
  <row Id="2793" PostTypeId="2" ParentId="2516" CreationDate="2010-09-17T15:26:45.237" Score="0" ViewCount="" Body="&lt;p&gt;One simple point not made directly in another answer is that it's simply not true that &quot;all null hypotheses are false.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The simple hypothesis that a physical coin has heads probability exactly equal to 0.5, ok, that is false.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But the compound hypothesis that a physical coin has heads probability greater than 0.499 and less than 0.501 may be true. If so, no hypothesis test -- no matter how many coin flips go into it -- is going to be able to reject this hypothesis with a probability greater than $\alpha$ (the tests's bound on false positives).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The medical industry tests &quot;non-inferiority&quot; hypotheses all the time, for this reason -- e.g. a new cancer drug has to show that its patients' probability of progression-free survival isn't less than 3 percentage points lower than an existing drug's, at some confidence level (the $\alpha$, usually 0.05).&lt;/p&gt;&#xA;" OwnerUserId="1122" LastActivityDate="2010-09-17T15:26:45.237" />
  <row Id="2794" PostTypeId="1" AcceptedAnswerId="2798" CreationDate="2010-09-17T15:44:08.690" Score="4" ViewCount="601" Body="&lt;p&gt;I know the significance of the slope of a LMS linear regression can be calculated using the r2 coefficient of determination and looking up the appropriate value in an F table. However, I was thinking of perhaps making this &quot;more robust&quot; by replacing the LMS linear regression with a repeated median straight line fit, and perhaps even replacing the average value used to calculate r2 with the median value of the data. Is there any reason why this would NOT be a valid approach? Maybe the values in an F table are predicated on using LMS and averages, for example?&lt;/p&gt;&#xA;" OwnerUserId="226" LastActivityDate="2010-09-21T16:43:01.597" Title="Significance of the slope of a straight line fit" Tags="&lt;regression&gt;&lt;statistical-significance&gt;&lt;robust&gt;" AnswerCount="3" FavoriteCount="1" />
  <row Id="2796" PostTypeId="2" ParentId="1621" CreationDate="2010-09-17T16:31:46.073" Score="0" ViewCount="" Body="&lt;p&gt;The most well know argument is that the fisher metric, being invariant to coordinate transforms, can be used to formulate an uninformed prior (Jeffreys prior). Not sure I buy it!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Less well known, is that sometimes these &quot;integrated quantities&quot; turn out to be divergences and such, one may argue that the fisher distances generate a generalised set of divergences (and properties thereof).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But still, I'm yet to find a good intuitive description of the fisher information and the quantities it generates. Please tell me if you find one.&lt;/p&gt;&#xA;" OwnerUserId="1342" LastActivityDate="2010-09-17T16:31:46.073" CommentCount="1" />
  <row Id="2797" PostTypeId="2" ParentId="2245" CreationDate="2010-09-17T16:49:17.907" Score="2" ViewCount="" Body="&lt;p&gt;In addition to the excellent answer above, there is a statistical method that can get you closer to demonstrating causality.  It is Granger Causality that demonstrates that one independent variable occurring before a dependent variable has a causal effect or not.  I introduce this method in an easy to follow presentation at the following link: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.slideshare.net/gaetanlion/granger-causality-presentation&quot; rel=&quot;nofollow&quot;&gt;http://www.slideshare.net/gaetanlion/granger-causality-presentation&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also apply this method to testing competing macroeconomic theories:&#xA;&lt;a href=&quot;http://www.slideshare.net/gaetanlion/economic-theory-testing-presentation&quot; rel=&quot;nofollow&quot;&gt;http://www.slideshare.net/gaetanlion/economic-theory-testing-presentation&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Be aware that this method is not perfect.  It just confirms that certain events occur before others and that those events appear to have a consistent directional relationship.  This seems to entail true causality but it is not always the case.  The rooster morning call does not cause the sun to rise.       &lt;/p&gt;&#xA;" OwnerUserId="1329" LastActivityDate="2010-09-17T16:49:17.907" />
  <row Id="2798" PostTypeId="2" ParentId="2794" CreationDate="2010-09-17T17:37:00.617" Score="3" ViewCount="" Body="&lt;p&gt;No, F tests are based on the assumption that lowest sum of residual squares is optimal. It does not hold in case of robust regression, where the criterion is different.&lt;br&gt;&#xA;For instance, effectively one may consider robust regression as least squares on data stripped from outliers; using $r^2$ on all data in this case adds non-culpable penalty for high residuals of outliers.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-09-17T17:37:00.617" />
  <row Id="2799" PostTypeId="2" ParentId="2245" CreationDate="2010-09-17T18:22:02.003" Score="1" ViewCount="" Body="&lt;p&gt;Read this:  &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0395615569&quot; rel=&quot;nofollow&quot;&gt;Experimental and Quasi-Experimental Designs for Generalized Causal Inference (Shadish, Cook, and Campbell, (2002)&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1334" LastActivityDate="2010-09-17T18:22:02.003" CommentCount="1" />
  <row Id="2801" PostTypeId="2" ParentId="2794" CreationDate="2010-09-17T18:36:17.920" Score="3" ViewCount="" Body="&lt;p&gt;No need to reinvent the wheel. There is an alternative, robust, R^2 measure with very good statistical properties:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.google.com/url?sa=t&amp;amp;source=web&amp;amp;cd=3&amp;amp;ved=0CBsQFjAC&amp;amp;url=http%3A%2F%2Fwww.hec.unige.ch%2Fwww%2Fdms%2Fhec_en%2Fvictoriafeser%2Frecherche%2Fpaper-rob-R2.pdf&amp;amp;rct=j&amp;amp;q=robust%20r-2%20dehon&amp;amp;ei=MbSTTIvWMsWYOJHrnIgJ&amp;amp;usg=AFQjCNE_NtnSZK618rQc0bph5VlJGK7mkQ&amp;amp;sig2=HSRGJYQSPJHoHVaYLWkbCQ&amp;amp;cad=rja&quot; rel=&quot;nofollow&quot;&gt;A robust coefficient of determination for regression, O Renauda&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit:&#xA;*Is there any reason why this would NOT be a valid approach? * For one this does not make your method any more robust. There is a large &lt;a href=&quot;http://en.wikipedia.org/wiki/Robust_statistics#Measures_of_robustness&quot; rel=&quot;nofollow&quot;&gt;literature&lt;/a&gt; on this issue, and fortunatly, good tools have been designed to adress these points. &lt;/p&gt;&#xA;" OwnerUserId="603" LastEditorUserId="603" LastEditorDisplayName="" LastEditDate="2010-09-21T16:43:01.597" LastActivityDate="2010-09-21T16:43:01.597" />
  <row Id="2802" PostTypeId="2" ParentId="2794" CreationDate="2010-09-17T18:57:23.110" Score="1" ViewCount="" Body="&lt;p&gt;I would simply use the standard regression output to evaluate the significance of the slope coefficient.  I mean by that looking at the coefficient itself, its standard error, t stat (# of standard errors = Coefficient/Standard error), p value, and confidence interval.  The p value directly addresses the statistical significance of the slope or coefficient you have in mind. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;R Square of the model tells how well the model explains the dependent variable, or how well the model fits the dependent variable. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The p value of each coefficient tells you how statistically significant those coefficients are.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Very often you can have a model with a high R Square, but that includes one variable with a coefficient that is not statistically significant (its p value is too high).  In such a case, it suggests your model would be nearly as good if you took that variable out.  By the way, you should really focus on the Adjusted R Square instead of R Square.  The Adjusted R Square correctly penalizes the model for having more variable and potentially over-fitting the data with independent variables that are not so relevant.  &lt;/p&gt;&#xA;" OwnerUserId="1329" LastActivityDate="2010-09-17T18:57:23.110" />
  <row Id="2805" PostTypeId="2" ParentId="2077" CreationDate="2010-09-17T21:44:07.557" Score="3" ViewCount="" Body="&lt;p&gt;I still think using the % change from one period to the next is the best way to render a non-stationary variable stationary as you first suggest.  A transformation such as a log works reasonably well (it flattens the non-stationary quality; but does not eliminate it entirely).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The third way is to deseasonalize and de-trend the data simultaneously in one single linear regression.  One independent variable would be trend (or time): 1, 2, 3, ... to how many time period you have.  And, the other variable would be a categorical variable with 11 different categories (for 11 out of the 12 months).  Then, using the resulting coefficient from this regression you can simultaneously detrend and de-seasonalize the data.  You will see your whole data set essentially flattened.  The remaining differences between periods will reflect changes independent from both growth trend and season. &lt;/p&gt;&#xA;" OwnerUserId="1329" LastActivityDate="2010-09-17T21:44:07.557" CommentCount="2" />
  <row Id="2806" PostTypeId="1" CreationDate="2010-09-18T02:08:24.590" Score="4" ViewCount="586" Body="&lt;p&gt;I previously asked this on StackOverflow, but it seems like it might be more appropriate here, given that it didn't get any answers on SO.  It's kind of at the intersection between statistics and programming.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to write some code to do PCA (Principal Component Analysis). I've browsed through the well-known algorithms and implemented &lt;a href=&quot;http://en.wikipedia.org/wiki/Principal_component_analysis#Computing_principal_components_with_expectation_maximization&quot; rel=&quot;nofollow&quot;&gt;this one&lt;/a&gt;, which as far as I can tell is equivalent to the NIPALS algorithm.  It works well for finding the first 2-3 principal components, but then seems to become very slow to converge (on the order of hundreds to thousands of iterations). Here are the details of what I need:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;The algorithm must be efficient when dealing with huge numbers of features (order 10,000 to 20,000) and sample sizes on the order of a few hundred.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;It must be reasonably implementable without a decent linear algebra/matrix library, as the target language is D, which doesn't have one yet, and even if it did, I would prefer not to add it as a dependency to the project in question.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;As a side note, on the same dataset R seems to find all principal components very fast, but it uses singular value decomposition, which is not something I want to code myself.&lt;/p&gt;&#xA;" OwnerUserId="1347" LastActivityDate="2010-11-21T09:56:02.610" Title="Best PCA Algorithm For Huge Number of Features?" Tags="&lt;pca&gt;&lt;algorithms&gt;&lt;code&gt;" AnswerCount="5" CommentCount="5" FavoriteCount="2" />
  <row Id="2807" PostTypeId="2" ParentId="2492" CreationDate="2010-09-18T02:32:42.093" Score="3" ViewCount="" Body="&lt;p&gt;IMHO normality tests are absolutely useless for the following reasons:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;On small samples, there's a good chance that the true distribution of the population is substantially non-normal, but the normality test isn't powerful to pick it up.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;On large samples, things like the T-test and ANOVA are pretty robust to non-normality.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The whole idea of a normally distributed population is just a convenient mathematical approximation anyhow.  None of the quantities typically dealt with statistically could plausibly have distributions with a support of all real numbers.  For example, people can't have a negative height.  Something can't have negative mass or more mass than there is in the universe.  Therefore, it's safe to say that &lt;strong&gt;nothing&lt;/strong&gt; is &lt;strong&gt;exactly&lt;/strong&gt; normally distributed in the real world.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="1347" LastActivityDate="2010-09-18T02:32:42.093" CommunityOwnedDate="2010-09-18T02:32:42.093" CommentCount="2" />
  <row Id="2808" PostTypeId="2" ParentId="1761" CreationDate="2010-09-18T02:43:41.650" Score="2" ViewCount="" Body="&lt;p&gt;GD047 linked to a UC Berkeley video on statistics, and I had another one. It's similar to MIT's Open Courseware (maybe a little lower quality). This one is for the Intro Statistics and Probability Class. It unfortunately only has 9 episodes, but it's free :).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://webcast.berkeley.edu/course_details_new.php?seriesid=2010-D-87392&amp;amp;semesterid=2010-D&quot; rel=&quot;nofollow&quot;&gt;UC Berkeley STATS 20 Webcast&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1118" LastActivityDate="2010-09-18T02:43:41.650" CommunityOwnedDate="2010-09-18T02:43:41.650" />
  <row Id="2809" PostTypeId="2" ParentId="2777" CreationDate="2010-09-18T02:56:41.683" Score="4" ViewCount="" Body="&lt;p&gt;I'd recommend taking a look at a couple of papers by Heping Zhang using adaptive splines for modeling longitudinal data:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.jstor.org/pss/1390725&quot; rel=&quot;nofollow&quot;&gt;Multivariate adaptive splines for analysis of longitudinal data&lt;/a&gt; (&lt;a href=&quot;http://statistics.stanford.edu/~ckirby/techreports/BIO/BIO%20182.pdf&quot; rel=&quot;nofollow&quot;&gt;Free PDF&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pubmed/14746441&quot; rel=&quot;nofollow&quot;&gt;Mixed effects multivariate adaptive splines model for the analysis of longitudinal and growth curve data&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In addition, see the &lt;a href=&quot;http://c2s2.yale.edu/software/masal/&quot; rel=&quot;nofollow&quot;&gt;MASAL&lt;/a&gt; page for software including an R package.&lt;/p&gt;&#xA;" OwnerUserId="251" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-09-19T09:58:00.853" LastActivityDate="2010-09-19T09:58:00.853" CommentCount="2" />
  <row Id="2810" PostTypeId="2" ParentId="2806" CreationDate="2010-09-18T03:04:07.327" Score="0" ViewCount="" Body="&lt;p&gt;See Sam Roweis' paper, &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.16.3330&quot; rel=&quot;nofollow&quot;&gt;&lt;em&gt;EM Algorithms for PCA and SPCA&lt;/em&gt;&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-09-18T03:04:07.327" CommentCount="2" />
  <row Id="2811" PostTypeId="2" ParentId="2777" CreationDate="2010-09-18T03:45:56.633" Score="3" ViewCount="" Body="&lt;p&gt;It looks to me like &lt;a href=&quot;http://dx.doi.org/10.1007/s10940-007-9036-0&quot; rel=&quot;nofollow&quot;&gt;Growth Mixture Models&lt;/a&gt; might have potential to allow you to examine your error variance. (&lt;a href=&quot;http://statmodel2.com/download/kreutermuthen2006_34.pdf&quot; rel=&quot;nofollow&quot;&gt;PDF&lt;/a&gt; here). (I'm not sure what multiplicative heteroscedastic models are, but I will definitely have to check them out).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Latent group based trajectory models have become really popular lately in criminology. But many people simply take for granted that groups actually exist, and some &lt;a href=&quot;http://dx.doi.org/10.1111/j.1745-9125.2010.00185.x&quot; rel=&quot;nofollow&quot;&gt;astute research&lt;/a&gt; has pointed out that you will find groups even in random data. Also to note Nagin's group based modelling approach does not allow you to assess your error (and honestly I have never seen a model that would look anything like a discontinuity).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Although it would be difficult with 20 time points, for exploratory purposes creating simple heuristics to identify patterns could be helpful (e.g. always low or always high, coefficient of variation). I'm envisioning sparklines in a spreadsheet or parallel coordinates plots but I doubt they would be helpful (I honestly have not ever seen a parallel coordinate plot that is very enlightening).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good luck&lt;/p&gt;&#xA;" OwnerUserId="1036" LastActivityDate="2010-09-18T03:45:56.633" CommentCount="4" />
  <row Id="2812" PostTypeId="2" ParentId="2806" CreationDate="2010-09-18T04:08:12.430" Score="0" ViewCount="" Body="&lt;p&gt;It sounds like maybe you want to use the &lt;a href=&quot;http://en.wikipedia.org/wiki/Lanczos_iteration&quot; rel=&quot;nofollow&quot;&gt;Lanczos Algorithm&lt;/a&gt;. Failing that, you might want to consult &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0801854148&quot; rel=&quot;nofollow&quot;&gt;Golub &amp;amp; Van Loan.&lt;/a&gt; I once coded a SVD algorithm (in SML, of all languages) from their text, and it worked reasonably well.&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-09-18T04:08:12.430" />
  <row Id="2813" PostTypeId="2" ParentId="2806" CreationDate="2010-09-18T04:25:12.827" Score="1" ViewCount="" Body="&lt;p&gt;I'd suggest trying &lt;a href=&quot;http://en.wikipedia.org/wiki/Kernel_PCA&quot; rel=&quot;nofollow&quot;&gt;kernel PCA&lt;/a&gt; which has a time/space complexity dependent on the number of examples (N) rather than number of features (P), which I think would be more suitable in your setting (P&gt;&gt;N)). Kernel PCA basically works with NxN kernel matrix (matrix of similarities between the data points), rather than the PxP covariance matrix which can be hard to deal with for large P. Another good thing about kernel PCA is that it can learn non-linear projections as well if you use it with a suitable kernel. See &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.128.7613&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;this paper on kernel PCA&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="881" LastActivityDate="2010-09-18T04:25:12.827" />
  <row Id="2814" PostTypeId="2" ParentId="2615" CreationDate="2010-09-18T05:03:55.393" Score="0" ViewCount="" Body="&lt;p&gt;The proof should be rather simple. Let $C$ be your correlation matrix (it has all ones on the diagonal). You multiply each element on the off diagonal by $(1+k)$. This is equivalent to computing the matrix $\hat{C} = (1+k) C - k \operatorname{diag}(C) = (1+k)C - k I,$ where $\operatorname{diag}$ is the diagonal part of a matrix, which in the case of $C$ is $I$, the identity matrix.  It is well known that the eigenvalues of a matrix commute with a polynomial applied to the matrix, and we are computing a polynomial function of $C$, with polynomial function $f(x) = (1+k)x - k$. Thus if $\lambda$ is an eigenvalue of $C$, then $f(\lambda) = (1+k)\lambda - k$ is an eigenvalue of $\hat{C} = f(C)$. Setting $f(\lambda) \le 0$ for $\lambda$ an eigenvalue of $C$ gives the desired condition on $k$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that depending on whether $k$ is positive or negative, you will want to check either $f(\lambda_1)$ or $f(\lambda_n)$, where $\lambda_1, \lambda_n$ are the smallest and largest eigenvalue of $C$. Unless $C$ is the identity matrix, you will have $\lambda_1 \lt 1 \lt \lambda_n$. This is the case because the sum of the eigenvalues equals $n$, the size of the matrix $C$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The commutivity of eigenvalues and polynomials is easy to check, actually. First if $x, \lambda$ are eigenvector, eigenvalue of matrix $A$, show that $x, c\lambda^j$ are eigenvector, eigenvalue of $c A^j$. This holds for integer $j$, even negative integers or zero.  Then show that if $x, \lambda_1$ are eigenvector, -value of $A$ and $x, \lambda_2$ are eigenvector, -value of $B$, then $x, \lambda_1 + \lambda_2$ are eigenvector, -value of $A + B$. From there, one can easily show that $x, c_0 + c_1 \lambda + c_2 \lambda^2 + \ldots c_n \lambda^n$ are eigenvector, -value of the matrix $c_0 I + c_1 A + c_2 A^2 + \ldots c_n A^n$. &lt;/p&gt;&#xA;" OwnerUserId="795" LastEditorUserId="795" LastEditorDisplayName="" LastEditDate="2010-09-18T20:55:20.910" LastActivityDate="2010-09-18T20:55:20.910" CommentCount="4" />
  <row Id="2815" PostTypeId="2" ParentId="2679" CreationDate="2010-09-18T05:07:02.720" Score="0" ViewCount="" Body="&lt;p&gt;you could model the traffic flow of red cars as a compound poisson distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;suppose we look at the number N of cars that pass in a fixed time [8 hours, perhaps].&#xA;N should have a poisson distribution, according to your assumptions. let $\lambda$ be the poisson parameter for N. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;suppose all of the cars are colorless to begin with. as each car passes, we flip a coin for which P(heads) = $p$. [presumably $p$ will reflect the proportion of red cars in the population.] each time the coin lands 'heads' the car is colored red.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;then if M denotes the number of red cars seen in the time period involved, M is also poisson with parameter $p\lambda$. [easy homework exercise, or see feller's book: intro to probability theory$\cdots$, vol 1, wiley.]&lt;/p&gt;&#xA;" OwnerUserId="1112" LastActivityDate="2010-09-18T05:07:02.720" />
  <row Id="2816" PostTypeId="2" ParentId="2770" CreationDate="2010-09-18T08:51:04.823" Score="3" ViewCount="" Body="&lt;p&gt;It's difficult to go past Piantadosi's &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0471727814&quot; rel=&quot;nofollow&quot;&gt;Clinical Trials: A Methodologic Perspective&lt;/a&gt; at least as a starting point. It is quite comprehensive and covers the topics you want and much more - including ethics, history, reporting, fraud, meta-analyses, randomisation, phases, suopping rules etc. - at an advanced undergraduate to early post-grad level.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The few times I was frustrated with it (the 1st edition anyway) where when it lacked sufficient detail - eg the derivations for sample size calculations. A comprehensive starting point!&lt;/p&gt;&#xA;" OwnerUserId="521" LastEditorUserId="521" LastEditorDisplayName="" LastEditDate="2010-09-18T08:56:35.737" LastActivityDate="2010-09-18T08:56:35.737" />
  <row Id="2817" PostTypeId="2" ParentId="2806" CreationDate="2010-09-18T09:26:40.507" Score="0" ViewCount="" Body="&lt;p&gt;I seem to recall that it is possible to perform PCA by computing the eigen-decomposition of X^TX rather than XX^T and then transform to get the PCs.  However I can't remember the details off-hand, but it is in Jolliffe's (excellent) book and I'll look it up when I am next at work.  I'd transliterate the linear algebra routines from e.g. Numerical Methods in C, rather than use any other algorithm.&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2010-09-18T09:26:40.507" CommentCount="1" />
  <row Id="2818" PostTypeId="2" ParentId="2748" CreationDate="2010-09-18T09:49:20.960" Score="2" ViewCount="" Body="&lt;p&gt;Half the battle with many questions is understanding the terminology. Matching implies within group (or within pair) correlation. Under appropriate circumstances matching can be dealt with paired t-tests, conditional logistic regression or mixed effects models. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In survival analysis (or time to event analysis), within group correlation is known as shared  &lt;a href=&quot;http://www.demogr.mpg.de/papers/working/wp-2003-032.pdf&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;FRAILTY&lt;/strong&gt;&lt;/a&gt;. &lt;a href=&quot;http://fmwww.bc.edu/RePEc/usug2001/uk7.pdf&quot; rel=&quot;nofollow&quot;&gt;Stata&lt;/a&gt; covers frailty pretty thoroughly. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;An example of the use of Stata for Cox PH with frailty on the UCLA website from Hosmer &amp;amp; Lemeshow's book is &lt;a href=&quot;http://www.ats.ucla.edu/stat/stata/examples/asa2/chap9.htm&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://user.demogr.mpg.de/jwv/pdf/Vaupel-MPS-5-1995-2.pdf&quot; rel=&quot;nofollow&quot;&gt;Here&lt;/a&gt; is a paper discussing the concept of shared frailty in twins.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An example is provided with Stata 11. See the 2nd last example &lt;a href=&quot;http://www.stata.com/help.cgi?stcox&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;. I'll copy from the STATA SURVIVAL ANALYSIS AND EPIDEMIOLOGICAL TABLES REFERENCE MANUAL [ST] (pp141-142).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;webuse catheter, clear&#xA;list in 1/10&#xA;&#xA;     +----------------------------------------------------------------+&#xA;     | patient   time   infect    age   female   _st   _d    _t   _t0 |&#xA;     |----------------------------------------------------------------|&#xA;  1. |       1     16        1     28        0     1    1    16     0 |&#xA;  2. |       1      8        1     28        0     1    1     8     0 |&#xA;  3. |       2     13        0     48        1     1    0    13     0 |&#xA;  4. |       2     23        1     48        1     1    1    23     0 |&#xA;  5. |       3     22        1     32        0     1    1    22     0 |&#xA;     |----------------------------------------------------------------|&#xA;  6. |       3     28        1     32        0     1    1    28     0 |&#xA;  7. |       4    318        1   31.5        1     1    1   318     0 |&#xA;  8. |       4    447        1   31.5        1     1    1   447     0 |&#xA;  9. |       5     30        1     10        0     1    1    30     0 |&#xA; 10. |       5     12        1     10        0     1    1    12     0 |&#xA;     +----------------------------------------------------------------+&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&quot;Consider the data from a study of 38 kidney dialysis patients, as described in McGilchrist and&#xA;Aisbett (1991). The study is concerned with the prevalence of infection at the catheter insertion point.&#xA;Two recurrence times (in days) are measured for each patient, and each recorded time is the time&#xA;from initial insertion (onset of risk) to infection or censoring.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each patient (patient) has two recurrence times (time) recorded, with each catheter insertion&#xA;resulting in either infection (infect==1) or right-censoring (infect==0). Among the covariates&#xA;measured are age and sex (female==1 if female, female==0 if male).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;stset time, fail(infect)&#xA;stcox age female, shared(patient)&#xA;Cox regression --&#xA;         Breslow method for ties                Number of obs      =        76&#xA;         Gamma shared frailty                   Number of groups   =        38&#xA;Group variable: patient&#xA;&#xA;No. of subjects =           76                  Obs per group: min =         2&#xA;No. of failures =           58                                 avg =         2&#xA;Time at risk    =         7424                                 max =         2&#xA;&#xA;                                                Wald chi2(2)       =     11.66&#xA;Log likelihood  =   -181.97453                  Prob &amp;gt; chi2        =    0.0029&#xA;&#xA;------------------------------------------------------------------------------&#xA;          _t | Haz. Ratio   Std. Err.      z    P&amp;gt;|z|     [95% Conf. Interval]&#xA;-------------+----------------------------------------------------------------&#xA;         age |   1.006202   .0120965     0.51   0.607     .9827701    1.030192&#xA;      female |   .2068678    .095708    -3.41   0.001     .0835376    .5122756&#xA;-------------+----------------------------------------------------------------&#xA;       theta |   .4754497   .2673108&#xA;------------------------------------------------------------------------------&#xA;Likelihood-ratio test of theta=0: chibar2(01) =     6.27 Prob&amp;gt;=chibar2 = 0.006&#xA;&#xA;Note: standard errors of hazard ratios are conditional on theta.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;From the output, we obtain $\hat{\theta}$ = 0.475, and given the standard error of $\hat{\theta}$ and likelihood-ratio test of H0:$\theta$ = 0, we find a significant frailty effect, meaning that the correlation within patient cannot be ignored.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Me again.&#xA;This is similar to a twin study - two measurements in the same unit (i.e. two from one patient or one from each twin)&#xA;If we ignore frailty the estimates change, especially the sex effect:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;stcox age female&#xA;&#xA;Cox regression -- Breslow method for ties&#xA;&#xA;No. of subjects =           76                     Number of obs   =        76&#xA;No. of failures =           58&#xA;Time at risk    =         7424&#xA;                                                   LR chi2(2)      =      6.67&#xA;Log likelihood  =   -185.10993                     Prob &amp;gt; chi2     =    0.0355&#xA;&#xA;------------------------------------------------------------------------------&#xA;          _t | Haz. Ratio   Std. Err.      z    P&amp;gt;|z|     [95% Conf. Interval]&#xA;-------------+----------------------------------------------------------------&#xA;         age |   1.002245   .0091153     0.25   0.805     .9845377    1.020271&#xA;      female |   .4499194   .1340786    -2.68   0.007     .2508832    .8068592&#xA;------------------------------------------------------------------------------&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Dummy variable model in response to Andy W's question (below):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;gen pnum = patient -  floor(_n/2)&#xA;list in 1/10&#xA;&#xA;     +-----------------------------------------------------------------------+&#xA;     | patient   time   infect    age   female   pnum   _st   _d    _t   _t0 |&#xA;     |-----------------------------------------------------------------------|&#xA;  1. |       1     16        1     28        0      1     1    1    16     0 |&#xA;  2. |       1      8        1     28        0      0     1    1     8     0 |&#xA;  3. |       2     13        0     48        1      1     1    0    13     0 |&#xA;  4. |       2     23        1     48        1      0     1    1    23     0 |&#xA;  5. |       3     22        1     32        0      1     1    1    22     0 |&#xA;     |-----------------------------------------------------------------------|&#xA;  6. |       3     28        1     32        0      0     1    1    28     0 |&#xA;  7. |       4    318        1   31.5        1      1     1    1   318     0 |&#xA;  8. |       4    447        1   31.5        1      0     1    1   447     0 |&#xA;  9. |       5     30        1     10        0      1     1    1    30     0 |&#xA; 10. |       5     12        1     10        0      0     1    1    12     0 |&#xA;     +-----------------------------------------------------------------------+&#xA;&#xA;stcox age female pnum&#xA;&#xA;Cox regression -- Breslow method for ties&#xA;&#xA;No. of subjects =           76                     Number of obs   =        76&#xA;No. of failures =           58&#xA;Time at risk    =         7424&#xA;                                                   LR chi2(3)      =      7.63&#xA;Log likelihood  =   -184.63204                     Prob &amp;gt; chi2     =    0.0543&#xA;&#xA;------------------------------------------------------------------------------&#xA;          _t | Haz. Ratio   Std. Err.      z    P&amp;gt;|z|     [95% Conf. Interval]&#xA;-------------+----------------------------------------------------------------&#xA;         age |   1.001207   .0091269     0.13   0.895     .9834779    1.019257&#xA;      female |   .4259506   .1295227    -2.81   0.005     .2347073    .7730221&#xA;        pnum |   .7639054   .2111721    -0.97   0.330     .4443606    1.313238&#xA;------------------------------------------------------------------------------&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The coefficients are different and the evidence of a sex effect has disappeared.&lt;/p&gt;&#xA;" OwnerUserId="521" LastEditorUserId="521" LastEditorDisplayName="" LastEditDate="2010-10-02T08:15:01.607" LastActivityDate="2010-10-02T08:15:01.607" CommentCount="7" />
  <row Id="2819" PostTypeId="1" AcceptedAnswerId="2822" CreationDate="2010-09-18T11:21:46.510" Score="6" ViewCount="766" Body="&lt;p&gt;I have p values from a lot of tests and would like to know whether there is actually something significant after correcting for multiple testing. The complication: my tests are not independent. The method I am thinking about (a variant of Fisher's Product Method, Zaykin et al., &lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2569904/&quot; rel=&quot;nofollow&quot;&gt;Genet Epidemiol&lt;/a&gt;, 2002) needs the correlation between the p values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to estimate this correlation, I am currently thinking about bootstrapping cases, running the analyses and correlating the resulting vectors of p values. Does anyone have a better idea? Or even a better idea for my original problem (correcting for multiple testing in correlated tests)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Background: I am logistically regressing whether or not my subjects are suffering from a particular disease on the interaction between their genotype (AA, Aa or aa) and a covariate. However, the genotype is actually a lot (30-250) of Single Nucleotide Polymorphisms (SNPs), which are certainly not independent but in Linkage Disequilibrium.&lt;/p&gt;&#xA;" OwnerUserId="1352" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-10-09T20:11:09.663" LastActivityDate="2010-10-09T20:11:09.663" Title="Correcting p values for multiple tests where tests are correlated (genetics)" Tags="&lt;correlation&gt;&lt;multiple-comparisons&gt;&lt;statistical-significance&gt;&lt;genetics&gt;" AnswerCount="2" />
  <row Id="2820" PostTypeId="2" ParentId="2819" CreationDate="2010-09-18T12:03:38.633" Score="1" ViewCount="" Body="&lt;p&gt;Hello Stephan,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using a method like bonferroni is fine, the problem is that if you have many tests you are not likely to find many &quot;discoveries&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can go with the FDR approach for dependent tests (see &lt;a href=&quot;http://en.wikipedia.org/wiki/False_discovery_rate#Dependent_tests&quot; rel=&quot;nofollow&quot;&gt;here for details&lt;/a&gt;) the problem is that I am not sure if you can say upfront if your correlations are all positive ones.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In R you can do simple FDR with p.adjust.  For more complex things I would take a look at &lt;a href=&quot;http://cran.r-project.org/web/packages/multcomp/index.html&quot; rel=&quot;nofollow&quot;&gt;multcomp&lt;/a&gt;, but I didn't go through it to see for solutions in cases of dependencies.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good luck.&lt;/p&gt;&#xA;" OwnerUserId="253" LastActivityDate="2010-09-18T12:03:38.633" CommentCount="4" />
  <row Id="2821" PostTypeId="2" ParentId="328" CreationDate="2010-09-18T12:21:00.450" Score="0" ViewCount="" Body="&lt;p&gt;Oren, it is useful to define what aspects of finance you intend to tackle. Statistics is a tool when seen from the econometrics perspective (in terms of assessing the plausibility of a proposed model / theory) or can be the first or primary line of attack when seen from the machine learning side - that is you go low on the domain knowledge and rely more on constructing a feature space and applying algorithms. (However, the task of constructing a useful feature space is dependent on deep domain knowledge).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To get a handle on the theoretical aspects of finance - I'd recommend say:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Investment Science by Luenberger&#xA;Mathematical Techniques in Finance: Tools for Incomplete Markets by Ales Cerny (perhaps a later read)&#xA;Asset Pricing by Cochrane&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To learn how to apply statistics / econometrics coupled with the theory:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Introductory Econometrics for Finance by Chris Brooks&#xA;Asset Price Dynamics, Volatility, and Prediction by Stephen Taylor&#xA;Econometrics of Financial Markets by Campbell, MacKinlay and Lo&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The books recommended above, ones by David Ruppert, Eric Zivot, Ruey Tsay are useful, however, I would recommend Chris Brooks' &amp;amp; Ruppert's texts first, followed by Taylor's.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Paul Soderlind's notes, and Kevin Sheppard's notes (both available online) are quite good.&lt;/p&gt;&#xA;" OwnerUserId="1349" LastActivityDate="2010-09-18T12:21:00.450" />
  <row Id="2822" PostTypeId="2" ParentId="2819" CreationDate="2010-09-18T14:32:08.507" Score="12" ViewCount="" Body="&lt;p&gt;This is actually a hot topic in Genomewide analysis studies (GWAS)! I am not sure the method you are thinking of is the most appropriate in this context. Pooling of p-values was described by some authors, but in a different context (replication studies or meta-analysis, see e.g. (1) for a recent review). Combining SNP p-values by Fisher's method is generally used when one wants to derive an unique p-value for a given gene; this allows to work at the gene level, and reduce the amount of dimensionality of subsequent testing, but as you said the non-independence between markers (arising from spatial colocation or linkage disiquilibrium, LD) introduce a bias. More powerful alternatives rely on resampling procedures, for example the use of maxT statistics for combining p-value and working at the gene level or when one is interested in pathway-based approaches, see e.g. (2) (§2.4 p. 93 provides details on their approach).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My main concerns with bootstraping (with replacement) would be that you are introducing an artificial form of relatedness, or in other words you create virtual twins, hence altering Hardy-Weinberg equilibrium (but also minimum allele frequency and call rate). This would not be the case with a permutation approach where you permute individual labels and keep the genotyping data as is. Usually, the &lt;a href=&quot;http://pngu.mgh.harvard.edu/~purcell/plink/&quot; rel=&quot;nofollow&quot;&gt;plink&lt;/a&gt; software can give you raw and permuted p-values, although it uses (by default) an adaptive testing strategy with a sliding window that allows to stop running all permutations (say 1000 per SNP) if it appears that the SNP under consideration is not &quot;interesting&quot;; it also has option for computing maxT, see the &lt;a href=&quot;http://pngu.mgh.harvard.edu/~purcell/plink/perm.shtml&quot; rel=&quot;nofollow&quot;&gt;online help&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But given the low number of SNPs you are considering, I would suggest relying on FDR-based or maxT tests as implemented in the &lt;a href=&quot;http://cran.r-project.org/web/packages/multtest/index.html&quot; rel=&quot;nofollow&quot;&gt;multtest&lt;/a&gt; R package (see &lt;code&gt;mt.maxT&lt;/code&gt;), but the definitive guide to resampling strategies for genomic application is &lt;a href=&quot;http://www.springer.com/978-0-387-49316-9&quot; rel=&quot;nofollow&quot;&gt;Multiple Testing Procedures with Applications to Genomics&lt;/a&gt;, from Dudoit &amp;amp; van der Laan (Springer, 2008). See also Andrea Foulkes's book on &lt;a href=&quot;http://www.jstatsoft.org/v31/b02/paper&quot; rel=&quot;nofollow&quot;&gt;genetics with R&lt;/a&gt;, which is reviewed in the JSS. She has great material on multiple testing procedures.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Further Notes&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many authors have pointed to the fact that simple multiple testing correcting methods such as the Bonferroni or Sidak are too stringent for adjusting the results for the individual SNPs. Moreover, neither of these methods take into account the correlation that exists between SNPs due to LD which tags the genetic variation across gene regions. Other alternative have bee proposed, like a derivative of Holm's method for multiple comparison (3), Hidden Markov Model (4), conditional or positive FDR (5) or derivative thereof (6), to name a few. So-called gap statistics or sliding window have been proved successful in some case, but you'll find a good review in (7) and (8). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've also heard of methods that make effective use of the haplotype structure or LD, e.g. (9), but I never used them. They seem, however, more related to estimating the correlation between markers, not p-value as you meant. But in fact, you might better think in terms of the dependency structure between successive test statistics, than between correlated p-values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;References&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Cantor, RM, Lange, K and Sinsheimer, JS. &lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2801749/&quot; rel=&quot;nofollow&quot;&gt;Prioritizing GWAS Results: A Review of Statistical Methods and Recommendations for Their Application&lt;/a&gt;. Am J Hum Genet. 2010 86(1): 6–22.&lt;/li&gt;&#xA;&lt;li&gt;Corley, RP, Zeiger, JS, Crowley, T et al. &lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pubmed/18384978&quot; rel=&quot;nofollow&quot;&gt;Association of candidate genes with antisocial drug dependence in adolescents&lt;/a&gt;. Drug and Alcohol Dependence 2008 96: 90–98.&lt;/li&gt;&#xA;&lt;li&gt;Dalmasso, C, Génin, E and Trégouet DA. &lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2535720/&quot; rel=&quot;nofollow&quot;&gt;A Weighted-Holm Procedure Accounting for Allele Frequencies in Genomewide Association Studies&lt;/a&gt;. Genetics 2008 180(1): 697–702.&lt;/li&gt;&#xA;&lt;li&gt;Wei, Z, Sun, W, Wang, K, and Hakonarson, H. &lt;a href=&quot;http://www4.stat.ncsu.edu/~sun/Papers/HMM-GWAS.pdf&quot; rel=&quot;nofollow&quot;&gt;Multiple Testing in Genome-Wide Association Studies via Hidden Markov Models&lt;/a&gt;. Bioinformatics 2009 25(21): 2802-2808.&lt;/li&gt;&#xA;&lt;li&gt;Broberg, P. &lt;a href=&quot;http://www.biomedcentral.com/1471-2105/6/199#B23&quot; rel=&quot;nofollow&quot;&gt;A comparative review of estimates of the proportion unchanged genes and the false discovery rate&lt;/a&gt;. BMC Bioinformatics 2005 6: 199.&lt;/li&gt;&#xA;&lt;li&gt;Need, AC, Ge, D, Weale, ME, et a. &lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2631150/&quot; rel=&quot;nofollow&quot;&gt;A Genome-Wide Investigation of SNPs and CNVs in Schizophrenia&lt;/a&gt;. PLoS Genet. 2009 5(2): e1000373.&lt;/li&gt;&#xA;&lt;li&gt;Han, B, Kang, HM, and Eskin, E. &lt;a href=&quot;http://www.plosgenetics.org/article/info%3Adoi%2F10.1371%2Fjournal.pgen.1000456&quot; rel=&quot;nofollow&quot;&gt;Rapid and Accurate Multiple Testing Correction and Power Estimation for Millions of Correlated Markers&lt;/a&gt;. PLoS Genetics 2009&lt;/li&gt;&#xA;&lt;li&gt;Liang, Y and Kelemen, A. &lt;a href=&quot;http://www.i-journals.org/ss/include/getdoc.php?id=107&amp;amp;article=26&amp;amp;mode=pdf&quot; rel=&quot;nofollow&quot;&gt;Statistical advances and challenges for analyzing correlated high dimensional snp data in genomic study for complex diseases&lt;/a&gt;. Statistics Surveys 2008 2 :43–60. -- the best recent review ever&lt;/li&gt;&#xA;&lt;li&gt;Nyholt, DR. &lt;a href=&quot;https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1181954/&quot; rel=&quot;nofollow&quot;&gt;A Simple Correction for Multiple Testing for Single-Nucleotide Polymorphisms in Linkage Disequilibrium with Each Other&lt;/a&gt;. Am J Hum Genet. 2004 74(4): 765–769.&lt;/li&gt;&#xA;&lt;li&gt;Nicodemus, KK, Liu, W, Chase, GA, Tsai, Y-Y, and Fallin, MD. &lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1866703/&quot; rel=&quot;nofollow&quot;&gt;Comparison of type I error for multiple test corrections in large single-nucleotide polymorphism studies using principal components versus haplotype blocking algorithms&lt;/a&gt;. BMC Genetics 2005; 6(Suppl 1): S78.&lt;/li&gt;&#xA;&lt;li&gt;Peng, Q, Zhao, J, and Xue, F. &lt;a href=&quot;http://www.biomedcentral.com/1471-2156/11/6&quot; rel=&quot;nofollow&quot;&gt;PCA-based bootstrap confidence interval tests for gene-disease association involving multiple SNPs&lt;/a&gt;. BMC Genetics 2010, 11:6&lt;/li&gt;&#xA;&lt;li&gt;Li, M, Romero, R, Fu, WJ, and Cui, Y (2010). &lt;a href=&quot;http://www.biomedcentral.com/1471-2156/11/79&quot; rel=&quot;nofollow&quot;&gt;Mapping Haplotype-haplotype Interactions with Adaptive LASSO&lt;/a&gt;. BMC Genetics 2010, 11:79 -- although not directly related to the question, it covers haplotype-based analysis/epistatic effect&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-06T11:23:22.413" LastActivityDate="2010-10-06T11:23:22.413" CommentCount="7" />
  <row Id="2823" PostTypeId="1" CreationDate="2010-09-18T16:14:02.140" Score="4" ViewCount="160" Body="&lt;p&gt;I've just been given a stack of polling data to analyse. Some of the questions are obviously leading or present subtle incentives (for the poller or polled) for specific answers. Of other questions I'm not so sure but I have some doubts. I'm also starting to question other factors about how the poll was conducted (environment, privacy, etc.). I'd like to be able to present the analysis with the unreliable data discounted according to some recognised standard.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are actionable rules or methods for assessing whether results are junk based on human factors in data collection?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd gladly accept specific criteria, or links to resources such as books or web pages on the subject.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As I'm not a professional statistician, even being told what terms are used for this field of study and it's concerns would help a lot (though maybe best added as comments).&lt;/p&gt;&#xA;" OwnerUserId="1343" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-09-19T09:30:23.640" LastActivityDate="2010-09-19T09:30:23.640" CommunityOwnedDate="2010-09-18T16:14:21.843" Title="What are good resources/criteria for judging human bias in data collection?" Tags="&lt;books&gt;&lt;theory&gt;&lt;survey&gt;" AnswerCount="2" />
  <row Id="2824" PostTypeId="1" AcceptedAnswerId="2853" CreationDate="2010-09-18T17:42:06.037" Score="8" ViewCount="473" Body="&lt;p&gt;A few months ago I posted a question about homoscedascity tests in R on SO, and Ian Fellows answered that (I'll paraphrase his answer very loosely) &quot;homoscedascity tests are not a good tool when testing the goodness of fit of your model. With small samples, you don't have enough power to detect departures from homoscedascity, while with big samples you have &quot;plenty of power&quot;, so you are more likely to screen even trivial departures from equality&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;His great answer came as a slap in my face. I used to check normality and homoscedascity assumptions each time I ran ANOVA. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now you lads hop in: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is, in your opinion, best practice when checking ANOVA assumptions?&lt;/p&gt;&#xA;" OwnerUserId="1356" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-09-19T09:42:19.610" LastActivityDate="2010-09-19T14:44:17.667" Title="Checking ANOVA assumptions" Tags="&lt;hypothesis-testing&gt;&lt;anova&gt;&lt;homogeneity&gt;&lt;goodness-of-fit&gt;" AnswerCount="4" FavoriteCount="3" />
  <row Id="2825" PostTypeId="2" ParentId="2715" CreationDate="2010-09-18T18:15:49.220" Score="12" ViewCount="" Body="&lt;p&gt;Keep your analysis reproducible. A reviewer or your boss or someone else will eventually ask you how exactly you arrived at your result - probably six months or more after you did the analysis. You will &lt;em&gt;not&lt;/em&gt; remember how you cleaned the data, what analysis you did, why you chose the specific model you used... And reconstructing all this is a pain.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Corollary: use R, put comments in your analysis scripts and keep them.&lt;/p&gt;&#xA;" OwnerUserId="1352" LastActivityDate="2010-09-18T18:15:49.220" CommunityOwnedDate="2010-09-18T18:15:49.220" CommentCount="1" />
  <row Id="2826" PostTypeId="2" ParentId="2823" CreationDate="2010-09-18T18:33:13.560" Score="1" ViewCount="" Body="&lt;p&gt;Possibly &lt;a href=&quot;http://en.wikipedia.org/wiki/Benford%27s_law&quot; rel=&quot;nofollow&quot;&gt;Benford's Law&lt;/a&gt; might help. Check the Application section on the wiki&lt;/p&gt;&#xA;" OwnerUserId="364" LastEditorUserId="795" LastEditorDisplayName="" LastEditDate="2010-09-18T20:43:30.903" LastActivityDate="2010-09-18T20:43:30.903" CommunityOwnedDate="2010-09-18T18:33:13.560" CommentCount="1" />
  <row Id="2827" PostTypeId="2" ParentId="2715" CreationDate="2010-09-18T19:49:37.593" Score="3" ViewCount="" Body="&lt;p&gt;For histograms, a good rule of thumb for &lt;strong&gt;number of bins in a histogram&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;&lt;strong&gt;square root of the number of data points&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="438" LastActivityDate="2010-09-18T19:49:37.593" CommunityOwnedDate="2010-09-18T19:49:37.593" />
  <row Id="2828" PostTypeId="1" AcceptedAnswerId="2890" CreationDate="2010-09-18T20:20:10.293" Score="12" ViewCount="360" Body="&lt;p&gt;How can we compare complexity of two models with the same number of parameters?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;b&gt;Edit 09/19&lt;/b&gt;: To clarify, model complexity is a measure of how hard it is to learn from limited data. When two models fit existing data equally well, a model with lower complexity will give lower error on future data. When approximations are used, this may technically not always be true, but that's OK if it tends to be true in practice. Various approximations give different complexity measures&lt;/p&gt;&#xA;" OwnerUserId="511" LastEditorUserId="511" LastEditorDisplayName="" LastEditDate="2010-09-19T22:06:26.193" LastActivityDate="2010-12-16T20:16:41.043" Title="Measures of model complexity" Tags="&lt;model-selection&gt;" AnswerCount="6" CommentCount="7" FavoriteCount="6" />
  <row Id="2829" PostTypeId="2" ParentId="527" CreationDate="2010-09-18T20:41:05.413" Score="1" ViewCount="" Body="&lt;p&gt;I agree with @drnexus. In addition, I might recommend a Morgan-Pitman test for the equality of variances of the two methods. This would tell you if one method has more variance than the other. This in itself might not be a bad thing because presumably the two tests have different bias-variance tradeoffs (for example, one test might always say 50% (biased, but no variance) while the other is unbiased but very noisy). Some domain knowledge might be helpful here in determining how much tradeoff you want of your method. Of course, as noted by others, having a 'gold standard' would be much preferred. &lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-09-18T20:41:05.413" />
  <row Id="2830" PostTypeId="2" ParentId="2828" CreationDate="2010-09-18T20:41:57.250" Score="2" ViewCount="" Body="&lt;p&gt;I think it would depend on the actual model fitting procedure.  For a generally applicable measure, you might consider &lt;em&gt;Generalized Degrees of Freedom&lt;/em&gt; described in &lt;a href=&quot;http://www.jstor.org/pss/2669609&quot; rel=&quot;nofollow&quot;&gt;Ye 1998&lt;/a&gt; -- essentially the sensitivity of change of model estimates to perturbation of observations -- which works quite well as a measure of model complexity.&lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-09-18T20:41:57.250" CommentCount="4" />
  <row Id="2831" PostTypeId="2" ParentId="2715" CreationDate="2010-09-18T21:07:17.487" Score="9" ViewCount="" Body="&lt;p&gt;One rule per answer ;-)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Talk to the statistician &lt;em&gt;before&lt;/em&gt; conducting the study. If possible, before applying for the grant. Help him/her understand the problem you are studying, get his/her input on how to analyze the data you are about to collect and think about what that means for your study design and data requirements. Perhaps the stats guy/gal suggests doing a hierarchical model to account for who diagnosed the patients - then you need to track who diagnosed whom. Sounds trivial, but it's far better to think about this before you collect data (and fail to collect something crucial) than afterwards.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On a related note: do a power analysis before starting. Nothing is as frustrating as not having budgeted for a sufficiently large sample size. In thinking about what effect size you are expecting, remember publication bias - the effect size you are going to find will probably be smaller than what you expected given the (biased) literature.&lt;/p&gt;&#xA;" OwnerUserId="1352" LastActivityDate="2010-09-18T21:07:17.487" CommunityOwnedDate="2010-09-18T21:07:17.487" />
  <row Id="2832" PostTypeId="2" ParentId="1815" CreationDate="2010-09-18T21:07:17.843" Score="1" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://ukcatalogue.oup.com/product/9780199285112.do&quot; rel=&quot;nofollow&quot;&gt;Experimental Design for the Life Sciences&lt;/a&gt;, by Ruxton &amp;amp; Colegrave. Aimed primarily at undergraduates.&lt;/p&gt;&#xA;" OwnerUserId="266" LastActivityDate="2010-09-18T21:07:17.843" CommunityOwnedDate="2010-09-18T21:07:17.843" />
  <row Id="2833" PostTypeId="2" ParentId="2824" CreationDate="2010-09-18T21:27:13.180" Score="6" ViewCount="" Body="&lt;p&gt;A couple of graphs will usually be much more enlightening than the p value from a test of normality or homoskedasticity. Plot observed dependent variables against independent variables. Plot observations against fits. Plot residuals against independent variables. Investigate anything that looks strange on these plots. If something does not look strange, I would not worry about a significant test of an assumption.&lt;/p&gt;&#xA;" OwnerUserId="1352" LastActivityDate="2010-09-18T21:27:13.180" CommentCount="4" />
  <row Id="2834" PostTypeId="2" ParentId="527" CreationDate="2010-09-18T21:29:59.203" Score="5" ViewCount="" Body="&lt;p&gt;The simple correlation approach isn't the right way to analyze results from method comparison studies. There are (at least) two highly recommended books on this topic that I referenced at the end (1,2). Briefly stated, when comparing measurement methods we usually expect that (a) our conclusions should not depend on the particular sample used for the comparison, and (b) measurement error associated to the particular measurement instrument should be accounted for. This precludes any method based on correlations, and we shall turn our attention to variance components or mixed-effects models that allow to reflect the systematic effect of item (here, item stands for individual or sample on which data are collected), which results from (a).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case, you have single measurements collected using two different methods (I assume that none of them might be considered as a gold standard) and the very basic thing to do is to plot the differences ($X_1-X_2$) versus the means ($(X_1+X_2)/2$); this is called a Bland-Altman plot. It will allow you to check if (1) the variations between the two set of measurements are constant and (2) the variance of the difference is constant across the range of observed values. Basically, this is just a 45° rotation of a simple scatterplot of $X_1$ vs. $X_2$, and its interpretation is close to a plot of fitted &lt;em&gt;vs.&lt;/em&gt; residuals values used in linear regression. Then,&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;if the difference is constant (&lt;em&gt;constant bias&lt;/em&gt;), you can compute the limit of agreement (see (3))&lt;/li&gt;&#xA;&lt;li&gt;if the difference is not constant across the range of measurement, you can fit a linear regression model between the two methods (choose the one you want as predictor)&lt;/li&gt;&#xA;&lt;li&gt;if the variance of the differences is not constant, try to find a suitable transformation that makes the relationship linear with constant variance&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Other details may be found in (2), chapter 4.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;References&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Dunn, G (2004). &lt;em&gt;Design and Analysis of Reliability Studies&lt;/em&gt;. Arnold. See the review in the &lt;a href=&quot;http://ije.oxfordjournals.org/content/34/2/499.1.full&quot; rel=&quot;nofollow&quot;&gt;&lt;em&gt;International Journal of Epidemiology&lt;/em&gt;&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;li&gt;Carstensen, B (2010). &lt;em&gt;Comparing clinical measurement methods&lt;/em&gt;. Wiley. See the &lt;a href=&quot;http://staff.pubhealth.ku.dk/~bxc/MethComp/&quot; rel=&quot;nofollow&quot;&gt;companion website&lt;/a&gt;, including R code.&lt;/li&gt;&#xA;&lt;li&gt;The original article from Bland and Altman, &lt;a href=&quot;http://www-users.york.ac.uk/~mb55/meas/ba.htm&quot; rel=&quot;nofollow&quot;&gt;Statistical methods for assessing agreement between two methods of clinical measurement&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;li&gt;Carstensen, B (2004). &lt;a href=&quot;http://biostatistics.oxfordjournals.org/content/5/3/399.full.pdf&quot; rel=&quot;nofollow&quot;&gt;Comparing and predicting between several methods of measurement&lt;/a&gt;. &lt;em&gt;Biostatistics&lt;/em&gt;, &lt;em&gt;5(3)&lt;/em&gt;, 399–413.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-19T12:20:31.163" LastActivityDate="2010-09-19T12:20:31.163" CommentCount="5" />
  <row Id="2835" PostTypeId="2" ParentId="2585" CreationDate="2010-09-18T21:46:28.497" Score="2" ViewCount="" Body="&lt;p&gt;A slight variation on Jeromy's theme: time on the horizontal axis, price on the vertical axis. Plot multiple lines: one connecting the minimum prices, one connecting the 10% quantiles of prices, one connecting the 25% quantiles of prices. Plot these lines in varying shades of gray: large amounts of product available at that price translate into a black line, small amounts into an almost white line.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I like scaling things like these to the interval 0-100, then using the colors named &quot;gray0&quot; to &quot;gray100&quot; in R:&#xA;&lt;a href=&quot;http://research.stowers-institute.org/efg/R/Color/Chart/ColorChart.pdf&quot; rel=&quot;nofollow&quot;&gt;http://research.stowers-institute.org/efg/R/Color/Chart/ColorChart.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, this kind of relies on your data not being too irregularly spaced in time. If this is an issue, Jeromy's idea of local quantile regression is appealing. You may be able to tweak the &lt;a href=&quot;http://stat.ethz.ch/CRAN/web/packages/quantreg/index.html&quot; rel=&quot;nofollow&quot;&gt;quantreg&lt;/a&gt; package in R to help you here.&lt;/p&gt;&#xA;" OwnerUserId="1352" LastActivityDate="2010-09-18T21:46:28.497" CommunityOwnedDate="2010-09-18T21:46:28.497" CommentCount="4" />
  <row Id="2836" PostTypeId="2" ParentId="2828" CreationDate="2010-09-18T21:50:53.100" Score="2" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Minimum_description_length&quot; rel=&quot;nofollow&quot;&gt;Minimum Description Length&lt;/a&gt; may be an avenue worth pursuing.&lt;/p&gt;&#xA;" OwnerUserId="1352" LastActivityDate="2010-09-18T21:50:53.100" CommentCount="1" />
  <row Id="2837" PostTypeId="2" ParentId="113" CreationDate="2010-09-19T00:27:18.983" Score="1" ViewCount="" Body="&lt;p&gt;John, I am not sure my suggestion may be of help.  But, in any case the book &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0199730067&quot; rel=&quot;nofollow&quot;&gt;Intuitive Biostatistics&lt;/a&gt; by Harvey Motulsky may be of assistance.  Chapter 37 'Choosing a Test' has a pretty good table on page 298 that tells you given the nature of the data set and problem you are addressing what statistical method you should use.  Amazon lets you search through this book.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good luck. &lt;/p&gt;&#xA;" OwnerUserId="1329" LastEditorUserId="1329" LastEditorDisplayName="" LastEditDate="2010-10-08T23:57:02.170" LastActivityDate="2010-10-08T23:57:02.170" />
  <row Id="2838" PostTypeId="2" ParentId="913" CreationDate="2010-09-19T00:52:52.930" Score="1" ViewCount="" Body="&lt;p&gt;Many have already made excellent suggestions regarding transforming the variables and using robust regression methods.  But, when looking at the scatter plot, I observe two separate data sets.  One set has a very strong linear relationship where the correlation is a lot higher than the overall 0.6.  And, visually it looks like Y = 0.13X.  So, when X = 15,000 Y is around 2,000 or so.  Thus, a regression line with a similar slope would fit the vast majority of the data points really well.  Then, you have a second data set of 300 datapoints that are wild outliers that are random.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would focus on those 300 outliers.  Can you explain them?  Are there reasons why they are so far off the regression line?  Are those datapoints a fractional % of your whole data set?  Are they material events you need to keep for your study?  Or can you afford to take them out?  If you can take them out, you may have a pretty strong regression with a high R Square.  You just would have to accept that in a few percentage of the time things go wild and your regression model will be off.  But, that's the truth of any model you built. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have to keep those 300 outliers in your overall data set, they will materially affect your regression.  And, you will end up with a regression model that does not fit well the majority of your data point.  And, it won't fit the outliers either because they are random and won't fit any regression line.             &lt;/p&gt;&#xA;" OwnerUserId="1329" LastActivityDate="2010-09-19T00:52:52.930" />
  <row Id="2839" PostTypeId="2" ParentId="913" CreationDate="2010-09-19T01:47:02.270" Score="0" ViewCount="" Body="&lt;p&gt;Like the others have said, some sort of transformation is recommended. Your data seems highly clustered, and could be roughly linear, but it's difficult to tell with all the other points around it.&#xA;Others have suggested trying a log transformation, but it might also be a good idea to try a &lt;a href=&quot;http://en.wikipedia.org/wiki/Box-Cox_transformation&quot; rel=&quot;nofollow&quot;&gt;Box-Cox Transformation&lt;/a&gt;. If the resulting exponent it tells you to multiply by is 0, then a log transform is the best. All software packages that I know of allow you to do Box-Cox. In R, it's in the MASS package. Here's some information about that:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://stat.ethz.ch/R-manual/R-devel/library/MASS/html/boxcox.html&quot; rel=&quot;nofollow&quot;&gt;Doing Box-Cox Transformations in R&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That's not going to give a you a perfectly linear fit, but it'll probably make the interpretation of your data a little easier.&lt;/p&gt;&#xA;" OwnerUserId="1118" LastActivityDate="2010-09-19T01:47:02.270" />
  <row Id="2840" PostTypeId="2" ParentId="2824" CreationDate="2010-09-19T02:09:59.820" Score="2" ViewCount="" Body="&lt;p&gt;QQ Plots are pretty good ways to detect non-normality. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For homoscedasticity, try Levene's test or a Browne-Forsythe test. Both are similar, though BF is a little more robust. They are less sensitive to non-normality than Bartlett's test, but even still, I've found them not to be the most reliable with small sample sizes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Qq_plot&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Qq_plot&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Brown&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Brown&lt;/a&gt;–Forsythe_test&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Levene%27s_test&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Levene%27s_test&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1118" LastActivityDate="2010-09-19T02:09:59.820" />
  <row Id="2841" PostTypeId="2" ParentId="2823" CreationDate="2010-09-19T03:24:26.727" Score="3" ViewCount="" Body="&lt;p&gt;In regards to the leading questions, here are several options of how I would attempt to investigate if your suspicions are true;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1 - Conduct your own experiment. One of your conditions will be to mimic the leading questions in the prior surveys, the other condition will be a survey constructed with functionally similar questions and answers but without the leading question(s). Randomly allocate surveys, and differences in answer distributions can be attributable to the different questions (+ sampling error).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2 - Determine if any other surveys have functionally similar questions but are not leading, and look at the distribution of answers for those surveys. The only thing you have to worry about here is the differences in the sample characteristics between your unfair surveys and the fair surveys that could account for some of the observed differences in the answers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3 - Identify constructs between questions in your unfair survey. If leading questions within that construct have low correlations with other fair items it could be taken as evidence that the question is not returning the information it should.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm sure their is some psychological/psychometrics literature on how to ask questions (or about other items such as &quot;environment&quot;). I'm sure it would do you some good to review some of that work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good luck&lt;/p&gt;&#xA;" OwnerUserId="1036" LastActivityDate="2010-09-19T03:24:26.727" CommunityOwnedDate="2010-09-19T03:24:26.727" />
  <row Id="2842" PostTypeId="2" ParentId="2824" CreationDate="2010-09-19T04:34:21.487" Score="3" ViewCount="" Body="&lt;p&gt;The are some very good web guides to checking the assumptions of ANOVA &amp;amp; what to do if the fail. &lt;a href=&quot;http://quality-control-plan.com/StatGuide/oneway_anova_ass_viol.htm&quot; rel=&quot;nofollow&quot;&gt;Here&lt;/a&gt; is one. &lt;a href=&quot;http://homepage.mac.com/bradthiessen/stats/m301/4a.pdf&quot; rel=&quot;nofollow&quot;&gt;This&lt;/a&gt; is another.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Essentially your eye is the best judge, so do some &lt;a href=&quot;http://en.wikipedia.org/wiki/Exploratory_data_analysis&quot; rel=&quot;nofollow&quot;&gt;exploratory data analysis&lt;/a&gt;. That means plot the data - histograms and box plots are a good way to assess normality and homoscedascity. And remember ANOVA is robust to minor violations of these.&lt;/p&gt;&#xA;" OwnerUserId="521" LastActivityDate="2010-09-19T04:34:21.487" />
  <row Id="2843" PostTypeId="2" ParentId="328" CreationDate="2010-09-19T05:41:03.267" Score="0" ViewCount="" Body="&lt;p&gt;I like &lt;a href=&quot;http://books.google.com/books?id=Qc8KWWtUokcC&amp;amp;lpg=PR1&amp;amp;dq=risk%20and%20asset%20allocation%20meucci&amp;amp;pg=PR1#v=onepage&amp;amp;q&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;Risk and Asset Allocation&lt;/a&gt; by A. Meucci. This book is a bit more advanced than Ruppert's book, but still very user-friendly.&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-09-19T05:41:03.267" />
  <row Id="2844" PostTypeId="1" AcceptedAnswerId="2847" CreationDate="2010-09-19T09:21:07.803" Score="4" ViewCount="272" Body="&lt;p&gt;Example code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(pc.cr &amp;lt;- princomp(USArrests))  # inappropriate&#xA;summary(pc.cr)&#xA;loadings(pc.cr)  ## note that blank entries are small but not zero&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am getting different outputs from each, and I am not sure I understand what the difference is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the output:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; summary(pc.cr)&#xA;Importance of components:&#xA;                           Comp.1      Comp.2      Comp.3       Comp.4&#xA;Standard deviation     82.8908472 14.06956001 6.424204055 2.4578367034&#xA;Proportion of Variance  0.9655342  0.02781734 0.005799535 0.0008489079&#xA;Cumulative Proportion   0.9655342  0.99335156 0.999151092 1.0000000000&#xA;&#xA;&#xA;&amp;gt; loadings(pc.cr)  ## note that blank entries are small but not zero&#xA;&#xA;...&#xA;&#xA;               Comp.1 Comp.2 Comp.3 Comp.4&#xA;SS loadings      1.00   1.00   1.00   1.00&#xA;Proportion Var   0.25   0.25   0.25   0.25&#xA;Cumulative Var   0.25   0.50   0.75   1.00&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;p.s: how can I get access to the table created by summary(pc.cr)?? (I can't seem to find it in str)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="253" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-19T10:52:02.287" LastActivityDate="2010-09-19T11:40:21.617" Title="What is the difference between summary and loadings for princomp?" Tags="&lt;r&gt;&lt;pca&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="2845" PostTypeId="2" ParentId="1815" CreationDate="2010-09-19T09:24:48.077" Score="0" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0824778812&quot; rel=&quot;nofollow&quot;&gt;Experimental Design in Biotechnology&lt;/a&gt; by Perry D. Haaland, ed Marcel Dekker. &lt;/p&gt;&#xA;" LastActivityDate="2010-09-19T09:34:20.480" CommunityOwnedDate="2010-09-19T09:24:48.077" />
  <row Id="2846" PostTypeId="1" AcceptedAnswerId="2850" CreationDate="2010-09-19T09:32:28.610" Score="8" ViewCount="416" Body="&lt;p&gt;I have read and seen a lot of Parallel coordinates plots. Can someone answer the following set of questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;What are parallel coordinates plots (PCP) in simple words, so that a layman can understand?&lt;/li&gt;&#xA;&lt;li&gt;A mathematical explanation with some intuition if possible&lt;/li&gt;&#xA;&lt;li&gt;When are PCP useful and when to use them?&lt;/li&gt;&#xA;&lt;li&gt;When are PCP &lt;em&gt;not&lt;/em&gt; useful and when they should be avoided?&lt;/li&gt;&#xA;&lt;li&gt;Possible advantages and disadvantages of PCP&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="1307" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-09-19T09:37:17.790" LastActivityDate="2010-09-19T17:57:05.217" Title="An easy explanation for the parallel coordinates plot" Tags="&lt;r&gt;&lt;data-visualization&gt;" AnswerCount="3" CommentCount="3" FavoriteCount="1" />
  <row Id="2847" PostTypeId="2" ParentId="2844" CreationDate="2010-09-19T10:45:31.837" Score="3" ViewCount="" Body="&lt;p&gt;The first output is the correct and most useful one. Calling &lt;code&gt;loadings()&lt;/code&gt; on your object just returns a summary where the SS are always equal to 1, hence the % variance is just the SS loadings divided by the number of variables. It makes sense only when using Factor Analysis (like in &lt;code&gt;factanal&lt;/code&gt;). I never use &lt;code&gt;princomp&lt;/code&gt; or its SVD-based alternative (&lt;code&gt;prcomp&lt;/code&gt;), and I prefer the &lt;a href=&quot;http://cran.r-project.org/web/packages/FactoMineR/index.html&quot; rel=&quot;nofollow&quot;&gt;FactoMineR&lt;/a&gt; or &lt;a href=&quot;http://cran.r-project.org/web/packages/ade4/index.html&quot; rel=&quot;nofollow&quot;&gt;ade4&lt;/a&gt; package which are by far more powerful!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;About your second question, the &lt;code&gt;summary()&lt;/code&gt; function just returns the SD for each component (&lt;code&gt;pc.cr$sdev&lt;/code&gt; in your case), and the rest of the table seems to be computed afterwards (through the &lt;code&gt;print&lt;/code&gt; or &lt;code&gt;show&lt;/code&gt; method, I didn't investigate this in details).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; getS3method(&quot;summary&quot;,&quot;princomp&quot;)&#xA;function (object, loadings = FALSE, cutoff = 0.1, ...)&#xA;{&#xA;    object$cutoff &amp;lt;- cutoff&#xA;    object$print.loadings &amp;lt;- loadings&#xA;    class(object) &amp;lt;- &quot;summary.princomp&quot;&#xA;    object&#xA;}&#xA;&amp;lt;environment: namespace:stats&amp;gt;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What &lt;code&gt;princomp()&lt;/code&gt; itself does may be viewed using &lt;code&gt;getAnywhere(&quot;princomp.default&quot;)&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-19T11:40:21.617" LastActivityDate="2010-09-19T11:40:21.617" CommentCount="2" />
  <row Id="2848" PostTypeId="2" ParentId="1708" CreationDate="2010-09-19T11:24:58.573" Score="2" ViewCount="" Body="&lt;p&gt;It looks like you are referring to eigenanalysis for SNPs data and the article from Nick Patterson, &lt;a href=&quot;http://www.plosgenetics.org/article/info%3adoi/10.1371/journal.pgen.0020190&quot; rel=&quot;nofollow&quot;&gt;Population Structure and Eigenanalysis&lt;/a&gt; (PLoS Genetics 2006), where the first component explains the largest variance on allele frequency wrt. potential stratification in the sample (due to ethnicity or, more generally, ancestry). So I wonder why you want to consider all three first components, unless they appear to be significant from their expected distribution according to &lt;a href=&quot;http://j.mp/cHQnxw&quot; rel=&quot;nofollow&quot;&gt;TW distribution&lt;/a&gt;. Anyway, in R you can isolate the most informative SNPs (i.e. those that are at the extreme of the successive principal axes) with the &lt;code&gt;apply()&lt;/code&gt; function, working on row, e.g.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;apply(snp.df, 1, function(x) any(abs(x)&amp;gt;threshold))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where &lt;code&gt;snp.df&lt;/code&gt; stands for the data you show and which is stored either as a &lt;code&gt;data.frame&lt;/code&gt; or &lt;code&gt;matrix&lt;/code&gt; under R, and &lt;code&gt;threshold&lt;/code&gt; is the value you want to consider (this can be Mean $\pm$ 6 SD, as in Price et al. &lt;em&gt;Nature Genetics&lt;/em&gt; 2007 38(8): 904, or whatever value you want).&#xA;You may also implement the iterative PCA yourself.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, the TW test can be implemented as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;##' Test for the largest eigenvalue in a gaussian covariance matrix&#xA;##' &#xA;##' This function computes the test statistic and associated p-value&#xA;##' for a Wishart matrix focused on individuals (Tracy-Widom distribution).&#xA;##' &#xA;##' @param C a rectangular matrix of bi-allelic markers (columns) indexed&#xA;##'        on m individuals. Caution: genotype should be in {0,1,2}.&#xA;##' @return test statistic and tabulated p-value&#xA;##' @reference \cite{Johnstone:2001}&#xA;##' @seealso The RMTstat package provides other interesting functions to&#xA;##'          deal with Wishart matrices.&#xA;##' @example&#xA;##' X &amp;lt;- replicate(100,sample(0:2,20,rep=T))&#xA;tw.test &amp;lt;- function(C) {&#xA;  m &amp;lt;- nrow(C)  # individuals&#xA;  n &amp;lt;- ncol(C)  # markers&#xA;  # compute M&#xA;  C &amp;lt;- scale(C, scale=F)&#xA;  pj &amp;lt;- attr(C,&quot;scaled:center&quot;)/2&#xA;  M &amp;lt;- C/sqrt(pj*(1-pj))&#xA;  # compute X=MM'&#xA;  X &amp;lt;- M %*% t(M)&#xA;  ev &amp;lt;- sort(svd(X)$d, decr=T)[1:(m-1)]&#xA;  nprime &amp;lt;- ((m+1)*sum(ev)^2)/(((m-1)*sum(ev^2))-sum(ev)^2)&#xA;  l &amp;lt;- (m-1)*ev[1]/sum(ev)&#xA;  # normalize l and compute test statistic&#xA;  num &amp;lt;- (sqrt(nprime-1)+sqrt(m))&#xA;  mu &amp;lt;- num^2/nprime&#xA;  sigma &amp;lt;- num/nprime*(1/sqrt(nprime-1)+1/sqrt(m))^(1/3)&#xA;  l &amp;lt;- (l-mu)/sigma&#xA;  # estimate associated p-value&#xA;  if (require(RMTstat)) pv &amp;lt;- ptw(l, lower.tail=F)&#xA;  else pv &amp;lt;- NA&#xA;  return(list(stat=l, pval=pv))&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-26T20:55:20.310" LastActivityDate="2010-09-26T20:55:20.310" CommentCount="1" />
  <row Id="2849" PostTypeId="1" CreationDate="2010-09-19T12:46:28.310" Score="2" ViewCount="235" Body="&lt;p&gt;I have a static panel data model with small T (T=5) that makes it impossible for me to use granger causality as it requires a long time span.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my question: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Is there any alternative solution to test for causation even in a  small T context?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Any hint will be highely appreciated!&lt;/p&gt;&#xA;" OwnerUserId="1251" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-10T07:40:11.763" LastActivityDate="2010-11-10T07:40:11.763" Title="How to test for causation in a static panel data model with small t?" Tags="&lt;econometrics&gt;&lt;causal-inference&gt;&lt;panel-data&gt;" AnswerCount="1" CommentCount="5" />
  <row Id="2850" PostTypeId="2" ParentId="2846" CreationDate="2010-09-19T12:57:26.100" Score="3" ViewCount="" Body="&lt;p&gt;It seems to me that the main function of PCP is to highlight homogeneous groups of individuals, or conversely (in the dual space, by analogy with PCA) specific patterns of association on different variables. It produces an effective graphical summary of a multivariate data set, when there are not too much variables. Variables are automatically scaled to a fixed range (typically, 0–1) which is equivalent to working with standardized variables (to prevent the influence of one variable onto the others due to scaling issue), but for very high-dimensional data set (# of variables &gt; 10), you definitely have to look at other displays, like &lt;a href=&quot;http://had.co.nz/ggplot/plot-templates.html&quot; rel=&quot;nofollow&quot;&gt;fluctuation plot&lt;/a&gt; or &lt;a href=&quot;http://en.wikipedia.org/wiki/Heat_map&quot; rel=&quot;nofollow&quot;&gt;heatmap&lt;/a&gt; as used in microarray studies.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It helps answering questions like: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;are there any consistent pattern of individual scores that may be explained by specific class membership (e.g. gender difference)?&lt;/li&gt;&#xA;&lt;li&gt;are there any systematic covariation between scores observed on two or more variables (e.g. low scores observed on variable $X_1$ is always associated to high scores on $X_2$)?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In the following plot of the &lt;a href=&quot;http://en.wikipedia.org/wiki/Iris_flower_data_set&quot; rel=&quot;nofollow&quot;&gt;Iris data&lt;/a&gt;, it is clearly seen that species (here shown in different colors) show very discriminant profiles when considering petal length and width, or that &lt;em&gt;Iris setosa&lt;/em&gt; (blue) are more homogeneous with respect to their petal length (i.e. their variance is lower), for example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/xKvQv.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can even use it as a backend to classification or dimension reduction techniques, like PCA. Most often, when performing a PCA, in addition to reducing the features space you also want to highlight clusters of individuals (e.g. are there individuals who systematically score higher on some combination of the variables); this is usually down by applying some kind of hierarchical clustering on the factor scores and highlighting the resulting cluster membership on the factorial space (see the &lt;a href=&quot;http://cran.r-project.org/web/packages/FactoClass/index.html&quot; rel=&quot;nofollow&quot;&gt;FactoClass&lt;/a&gt; R package). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is also used in clustergrams (&lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.133.1405&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;Visualizing non-hierarchical and hierarchical cluster analyses&lt;/a&gt;) which aims at examining how cluster allocation evolves when increasing the number of clusters (see also, &lt;a href=&quot;http://stats.stackexchange.com/questions/2597/what-stop-criteria-for-agglomerative-hierarchical-clustering-are-used-in-practice/2609#2609&quot;&gt;What stop-criteria for agglomerative hierarchical clustering are used in practice?&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Such displays are also useful when linked to usual scatterplots (which by construction are restricted to 2D-relationships), this is called &lt;em&gt;brushing&lt;/em&gt; and it is available in the &lt;a href=&quot;http://www.ggobi.org/&quot; rel=&quot;nofollow&quot;&gt;GGobi&lt;/a&gt; data visualization system, or the &lt;a href=&quot;http://stats.math.uni-augsburg.de/Mondrian/&quot; rel=&quot;nofollow&quot;&gt;Mondrian&lt;/a&gt; software.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-19T16:55:34.567" LastActivityDate="2010-09-19T16:55:34.567" />
  <row Id="2851" PostTypeId="2" ParentId="2446" CreationDate="2010-09-19T13:21:39.003" Score="2" ViewCount="" Body="&lt;p&gt;Concerning your more specific question (i.e. how many degrees of freedom): the question is how many replicates do you have. Look at the early pages of chapter 19 of &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0470510242&quot; rel=&quot;nofollow&quot;&gt;the R book&lt;/a&gt; for examples and guidelines for such accounting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We could do the accounting here but i don't understand the design of your experiment (probably due to difference in vocabulary, it could be easier if you explained it in formal (i.e. math) script with care to define the indices).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might also want to Check the following paper &#xA;Hurlbert, S.H. (1984) Pseudoreplication and the design of ecological field experiments. Ecological Monographs, 54, 187–211.&lt;/p&gt;&#xA;" OwnerUserId="603" LastEditorUserId="603" LastEditorDisplayName="" LastEditDate="2010-09-21T20:33:56.847" LastActivityDate="2010-09-21T20:33:56.847" CommentCount="2" />
  <row Id="2852" PostTypeId="1" CreationDate="2010-09-19T14:02:26.617" Score="1" ViewCount="308" Body="&lt;p&gt;The biological data is listed as following: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   V1    V2    V3    V4    V5    V6&#xA;0.064 0.014 0.016 0.012 0.013 0.023&#xA;0.056 0.000 0.000 0.008 0.010 0.000&#xA;0.042 0.014 0.024 0.008 0.017 0.023&#xA;0.031 0.014 0.016 0.008 0.013 0.023&#xA;0.068 0.000 0.008 0.004 0.020 0.000&#xA;0.081 0.000 0.000 0.004 0.010 0.000&#xA;0.060 0.014 0.016 0.006 0.010 0.023&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;or you can download the data from &lt;a href=&quot;http://www.mediafire.com/?6yp9l9m47jv433a&quot; rel=&quot;nofollow&quot;&gt;http://www.mediafire.com/?6yp9l9m47jv433a&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;A&amp;lt;- dat[,1] &#xA;B&amp;lt;- dat[,2:6]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to compare the difference between the first column to other columns of the data.Because only dat[,2] and dat[,6] not subject to normal distribute,I used wilcox.test instead of t.test function to caculate in R. But the warning messages rised up,such as &quot;In wilcox.test.default(A, B[, 1]) : cannot compute exact p-value with ties&quot;. Could you give me some suggestions? Thank you.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;wilcox.test(A,B[,1])&#xA;&#xA;Wilcoxon rank sum test with continuity correction&#xA;&#xA;data: A and B[, 1] W = 49, p-value = 0.00184 alternative hypothesis: true location shift is not equal to 0&#xA;&#xA;Warning message: In wilcox.test.default(A, B[, 1]) : cannot compute exact p-value with ties&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-20T06:33:04.833" LastActivityDate="2010-09-20T11:29:59.137" Title="How to analyze these data?" Tags="&lt;r&gt;&lt;hypothesis-testing&gt;" AnswerCount="3" CommentCount="4" FavoriteCount="1" />
  <row Id="2853" PostTypeId="2" ParentId="2824" CreationDate="2010-09-19T14:44:17.667" Score="5" ViewCount="" Body="&lt;p&gt;In applied settings it is typically more important to know whether any  violation of assumptions is problematic for inference. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assumption tests based on significance tests are rarely of interest in large samples, because most inferential tests are robust to mild violations of assumptions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of the nice features of graphical assessments of assumptions is that they focus attention on the degree of violation and not the statistical significance of any violation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, it's also possible to focus on numeric summaries of your data which quantify the degree of violation of assumptions and not the statistical significance (e.g., skewness values, kurtosis values, ratio of largest to smallest group variances, etc.). You can also get standard errors or confidence intervals on these values, which will get smaller with larger samples. This perspective is consistent with the general idea that statistical significance is not equivalent to practical importance.&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-09-19T14:44:17.667" CommentCount="1" />
  <row Id="2854" PostTypeId="1" AcceptedAnswerId="5646" CreationDate="2010-09-19T15:00:04.550" Score="4" ViewCount="461" Body="&lt;p&gt;Dear all, &#xA;I was encouraged to ask this question here as well as on stackoverflow and would be very appreciative of any answers...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Due to hetereoscedasticity I'm doing bootstrapped linear regression (appeals more to me than robust regression). I'd like to create a plot along the lines of what I've done in the script here. However the &lt;code&gt;fill=int&lt;/code&gt; is not right since &lt;code&gt;int&lt;/code&gt; should (I believe) be calculated using a bivariate normal distribution. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Any idea how I could do that in this setting? &lt;/li&gt;&#xA;&lt;li&gt;Also is there a way for &lt;code&gt;bootcov&lt;/code&gt; to return bias-corrected percentiles?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;sample script:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(ggplot2) &#xA;library(Hmisc) &#xA;library(Design) # for ols()&#xA;&#xA;o&amp;lt;-data.frame(value=rnorm(10,20,5),&#xA;              bc=rnorm(1000,60,50),&#xA;              age=rnorm(1000,50,20),&#xA;              ai=as.factor(round(runif(1000,0,4),0)),&#xA;              Gs=as.factor(round(runif(1000,0,6),0))) &#xA;&#xA;reg.s&amp;lt;-function(x){      &#xA;    ols(value~as.numeric(bc)+as.numeric(age),data=x,x=T,y=T)-&amp;gt;temp &#xA;    bootcov(temp,B=1000,coef.reps=T)-&amp;gt;t2 &#xA;&#xA;    return(t2) &#xA;    } &#xA;&#xA;dlply(o,.(ai,Gs),function(x) reg.s(x))-&amp;gt;b.list &#xA;llply(b.list,function(x) x[[&quot;boot.Coef&quot;]])-&amp;gt;b2 &#xA;&#xA;ks&amp;lt;-llply(names(b2),function(x){ &#xA;    s&amp;lt;-data.frame(b2[[x]]) &#xA;    s$ai&amp;lt;-x &#xA;    return(s) &#xA;    }) &#xA;&#xA;&#xA;ks3&amp;lt;-do.call(rbind,ks) &#xA;ks3$ai2&amp;lt;-with(ks3,substring(ai,1,1)) &#xA;&#xA;ks3$gc2&amp;lt;-sapply(strsplit(as.character(ks3$ai), &quot;\\.&quot;), &quot;[[&quot;, 2) &#xA;&#xA;&#xA;k&amp;lt;-ks3 &#xA;j&amp;lt;-dlply(k,.(ai2,gc2),function(x){ &#xA;    i1&amp;lt;-quantile(x$Intercept,probs=c(0.025,0.975))[1] &#xA;    i2&amp;lt;-quantile(x$Intercept,probs=c(0.025,0.975))[2] &#xA;&#xA;    j1&amp;lt;-quantile(x$bc,probs=c(0.025,0.975))[1] &#xA;    j2&amp;lt;-quantile(x$bc,probs=c(0.025,0.975))[2] &#xA;&#xA;    o&amp;lt;-x$Intercept&amp;gt;i1 &amp;amp; x$Intercept&amp;lt;i2 &#xA;&#xA;    p&amp;lt;-x$bc&amp;gt;j1 &amp;amp; x$bc&amp;lt;j2 &#xA;&#xA;    h&amp;lt;-o &amp;amp; p &#xA;    return(h) &#xA;    }) &#xA;&#xA;m&amp;lt;-melt(j) &#xA;ks3$int&amp;lt;-m[,1]   &#xA;&#xA;ggplot(ks3,aes(x=bc,y=Intercept,fill=int)) +&#xA;  geom_point(,alpha=0.3,size=1,shape=21) +&#xA;  facet_grid(gc2~ai2,scales = &quot;free_y&quot;)+theme_bw()-&amp;gt;plott &#xA;plott&amp;lt;-plott+opts(panel.grid.minor=theme_blank(),panel.grid.major=theme_blank()) &#xA;plott&amp;lt;-plott+geom_vline(x=0,color=&quot;red&quot;) &#xA;plott+xlab(&quot;BC coefficient&quot;)+ylab(&quot;Intercept&quot;) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1291" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-12-19T17:06:45.113" LastActivityDate="2010-12-24T11:45:34.523" Title="Calculating probability for bivariate normal distributions based on bootstrapped regression coefficients" Tags="&lt;r&gt;&lt;bootstrap&gt;&lt;heteroscedasticity&gt;&lt;ggplot2&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="2855" PostTypeId="2" ParentId="2010" CreationDate="2010-09-19T16:41:11.350" Score="1" ViewCount="" Body="&lt;p&gt;Almost all statistics implicitly condition on N. We treat N as a constant that can come out from the expression $\mathbb{E}\left[\frac{1}{N}\sum_{i=1}^{N}{x_i}\right]$, for example. For that to be appropriate, N has to be a fixed value, which we get by conditioning. Without conditioning on N, as you said, we'd need to know the distribution of N, which is kind of strange to think about (&quot;what are my chances of selecting each value of N for my study?&quot;). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is important to assume that the $x_i$ are independent of N, however. This might be violated if, for example, a study has $100,000 to spend on its subjects and will spend it all no matter how many people are involved. If you are looking at the impact of this spending, then the spending-per-person declines as the number of people increases and N is correlated with your treatment (i.e., spending-per-person).&lt;/p&gt;&#xA;" OwnerUserId="401" LastActivityDate="2010-09-19T16:41:11.350" />
  <row Id="2856" PostTypeId="2" ParentId="2846" CreationDate="2010-09-19T17:10:15.873" Score="3" ViewCount="" Body="&lt;p&gt;In regards to questions 3, 4, and 5 I would suggest you check out this work&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://dx.doi.org/10.1057/palgrave.ivs.9500166&quot; rel=&quot;nofollow&quot;&gt;Perceiving patterns in parallel coordinates: determining thresholds for identification of relationships&#xA;by: Jimmy Johansson, Camilla Forsell, Mats Lind, Matthew Cooper in &#xA;&lt;em&gt;Information Visualization, Vol. 7, No. 2.&lt;/em&gt; (2008), pp. 152-162.&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To sum up their findings people are ok at identifying the direction of the slope of the relationship between each node, but aren't that good at identifying the strength of the relationship or the degree of the slope. They give suggested levels of noise in which people can still decipher the relationship in the article. Unfortunately the article does not discuss identifying subgroups via color like chl demonstrates.&lt;/p&gt;&#xA;" OwnerUserId="1036" LastEditorUserId="1036" LastEditorDisplayName="" LastEditDate="2010-09-19T17:28:17.197" LastActivityDate="2010-09-19T17:28:17.197" CommentCount="2" />
  <row Id="2857" PostTypeId="2" ParentId="2852" CreationDate="2010-09-19T17:36:12.660" Score="1" ViewCount="" Body="&lt;p&gt;For most of your variables (e.g. &lt;code&gt;V2&lt;/code&gt;), some observations have identical values, hence the warning message thrown by R: unique ranks cannot be computed for all observations, and there are ties, precluding the computation of an exact p-value. For your variable named &lt;code&gt;V2&lt;/code&gt;, there are in fact only two distinct values (out of 7), so I am very puzzled by the approach you took to analysis your data. With such a high number of tied data, I would not trust any Wilcoxon test. Moreover, in most non-parametric tests we assume that the sampled populations are symmetric and have the same dispersion or shape, which is hardly verifiable in your case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, I think a permutation test would be more appropriate in your case, see e.g. &lt;code&gt;permTS&lt;/code&gt; (&lt;a href=&quot;http://cran.r-project.org/web/packages/perm/index.html&quot; rel=&quot;nofollow&quot;&gt;perm&lt;/a&gt;), &lt;code&gt;pperm&lt;/code&gt; (&lt;a href=&quot;http://cran.r-project.org/web/packages/exactRankTests/index.html&quot; rel=&quot;nofollow&quot;&gt;exactRankTests&lt;/a&gt;), or the &lt;a href=&quot;http://cran.r-project.org/web/packages/coin/index.html&quot; rel=&quot;nofollow&quot;&gt;coin&lt;/a&gt; package.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-09-19T17:36:12.660" />
  <row Id="2858" PostTypeId="2" ParentId="2846" CreationDate="2010-09-19T17:57:05.217" Score="3" ViewCount="" Body="&lt;p&gt;Please visit &lt;a href=&quot;http://www.cs.tau.ac.il/~aiisreal/&quot; rel=&quot;nofollow&quot;&gt;http://www.cs.tau.ac.il/~aiisreal/&lt;/a&gt; and also look at the new book&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Parallel Coordinates - This book is about visualization, systematically incorporating&#xA;the fantastic human pattern recognition into the problem-solving process...&#xA;www.springer.com/math/cse/book/978-0-387-21507-5.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In Ch. 10 there are lots of real examples with multivariate data showing how parallel coordinates (abbr. ||-cs) can be used. It is also worth learning some of the math to visualize and work with multivariate/multidimensional relations (surfaces) and not just point sets. It is fun seeing and working with the analogues of familiar objects in many dimensions i.e. Moebius strip, convex sets and more. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In short ||-cs are a multidimensional coordinate system where the axes are parallel to each other allowing for lots of axes to be seen. The methodology has been applied to Conflict resolution algorithms in Air Traffic Control, Computer Vision, Process Control and Decision Support.&lt;/p&gt;&#xA;" OwnerUserId="1366" LastActivityDate="2010-09-19T17:57:05.217" />
  <row Id="2859" PostTypeId="2" ParentId="2852" CreationDate="2010-09-19T18:11:38.943" Score="4" ViewCount="" Body="&lt;p&gt;Sometimes a formal statistical test is overkill.  Row by row, the entries in the first column are the largest.  Draw a picture to make this apparent: side-by-side boxplots or dotplots would work nicely.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Although this is a &lt;em&gt;post-hoc&lt;/em&gt; comparison, if the initial intent had been to compare the first column against the rest for a shift in distribution, the most extreme characterizations would be that either all maxima or all minima occur in the first column (a two-sided test).  The chance of this occurring by chance, if all columns contained values drawn at random from a common distribution, would be $2 (\frac{1}{6})^7$ = about 0.0007%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In fact, the first two contains the largest 7 of the 42 values.  Again, &lt;em&gt;ex post facto&lt;/em&gt;, the chance of such an extreme ordering occurring equals $\frac{2}{42 \choose 7}$ = about 0.000007%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These results indicate that any reasonably powerful test you choose to conduct will conclude there's a highly significant difference.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In any event, You don't need a p-value; you need to characterize &lt;em&gt;how&lt;/em&gt; large the difference is (the right way to do this depends on what the data mean) and you need to seek an explanation for the difference.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-09-19T18:11:38.943" CommentCount="3" />
  <row Id="2860" PostTypeId="1" CreationDate="2010-09-19T18:53:35.410" Score="3" ViewCount="362" Body="&lt;p&gt;We know that the projection matrix learned by PCA can be applied to out-of-sample data points to get their low-dimensional embedding. However, how reliable are these embeddings expected to be, as compared to the embedding obtained from PCA with these out-of-sample points combined with the original data? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider this hypothetical pseudo out-of-sample setting: Let's say I have 1000 data points and I want to do PCA on them. Can I instead do PCA on maybe just 500 of them (in order to have some computational savings) and then use the learned projection matrix to get the embedding of the rest of the points as well (by treating them as out-of-sample data)?&lt;/p&gt;&#xA;" OwnerUserId="881" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-09-21T12:01:56.170" LastActivityDate="2010-09-22T00:35:28.130" Title="PCA on out-of-sample data" Tags="&lt;machine-learning&gt;&lt;pca&gt;&lt;dimensionality-reduction&gt;" AnswerCount="4" FavoriteCount="3" />
  <row Id="2861" PostTypeId="2" ParentId="2615" CreationDate="2010-09-19T20:09:26.417" Score="0" ViewCount="" Body="&lt;p&gt;I am not sure what the real question is, but suppose instead of changing &lt;em&gt;every&lt;/em&gt; non-diagonal element, you changed just 2 (to keep the resulting matrix symmetric). That is let $\hat{C}$ be $C$ with $\hat{C_{i,j}} = C_{i,j} + \Delta C / 2= \hat{C_{j,i}},$ for some choice of $i,j$ with $i \ne j$. (alternatively, imagine $\Delta C$ is added to $C_{i,j}$ only, and so $\hat{C}$ is no longer symmetric.) I will consider the question &quot;how small can $\Delta C$ be before $\hat{C}$ is no longer PSD?&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This question is easily solved as well, but the answer is not enlightening in my view. Let $\lambda_k, x^{(k)}$ be eigenvalue and eigenvector of $C$, where the eigenvector has unit norm. $\hat{C}$ is no longer PSD if there is index $k$ such that $\lambda_k + \Delta C x^{(k)}_i x^{(k)}_j &amp;lt; 0$. This can only hold if $x^{(k)}_i x^{(k)}_j &amp;lt; 0$, in which case we have $\Delta C &gt; -\lambda_k / x^{(k)}_i x^{(k)}_j$.  Compute the RHS for each $k$ for which the negativity condition holds and take the minimum, and that gives you the sufficient condition on $\Delta C.$&lt;/p&gt;&#xA;" OwnerUserId="795" LastEditorUserId="795" LastEditorDisplayName="" LastEditDate="2010-09-20T01:50:20.380" LastActivityDate="2010-09-20T01:50:20.380" />
  <row Id="2862" PostTypeId="1" CreationDate="2010-09-19T21:30:35.003" Score="4" ViewCount="237" Body="&lt;p&gt;Can someone suggest a topic or some reasonably narrow area in financial time series analysis (e.g. statistical, machine learning, etc.) which can make a good topic for a master thesis? By 'good' I mean that it's perspective i.e. can be extended to PhD-level studies, but still manageable in 3-4 month time for an applied math master student. Thank you. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: Thank you for the feedback. The field is indeed very broad and well developed that's why I would like to have some input from the community. Let's narrow down to high frequency financial time series or market microstructure (but if you come up with ideas in different areas I would be most grateful to hear it).&lt;/p&gt;&#xA;" OwnerUserId="1075" LastEditorUserId="1075" LastEditorDisplayName="" LastEditDate="2010-09-20T10:16:36.393" LastActivityDate="2010-09-20T16:52:43.323" CommunityOwnedDate="2010-09-20T10:15:14.007" Title="What is a good topic on financial time series analysis for master thesis?" Tags="&lt;time-series&gt;&lt;finance&gt;" AnswerCount="1" CommentCount="4" FavoriteCount="1" />
  <row Id="2863" PostTypeId="1" AcceptedAnswerId="2864" CreationDate="2010-09-19T22:35:50.730" Score="2" ViewCount="522" Body="&lt;p&gt;I want to run an item-total correlation on a 19-item questionnaire (some of the questions should get reverse-scored).  My question is: do I reverse score the items PRIOR to running the item-total correlation (in order to eliminate any variables that do not correlate with the total at &gt;.40)?  Additionally, should the items be reverse-scored prior to running a factor analysis?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" LastActivityDate="2010-09-20T04:07:18.163" Title="Do I reverse score items before running reliability analyses (item-total correlation) and factor analysis?" Tags="&lt;correlation&gt;&lt;factor-analysis&gt;&lt;reliability&gt;" AnswerCount="2" />
  <row Id="2864" PostTypeId="2" ParentId="2863" CreationDate="2010-09-19T22:56:29.170" Score="3" ViewCount="" Body="&lt;p&gt;Yes, you should reverse score all items as needed to ensure that a particular score means the same thing on all items. You should do this for all types of analysis. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, you have 'propensity to shoplift' measured via 3 items on a scale of 1 to 5 (where 1 is low propensity to shoplift and 5 is high). Suppose that you reversed item 1 on the on the survey so that 1 is high and 5 is low. Then you should reverse the score for item one so that 5 means the same thing across all three items (i.e., 5 is high propensity to shoplift).&lt;/p&gt;&#xA;" LastActivityDate="2010-09-19T22:56:29.170" />
  <row Id="2867" PostTypeId="1" AcceptedAnswerId="2868" CreationDate="2010-09-20T00:20:26.977" Score="2" ViewCount="142" Body="&lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have requested this thread to be locked.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;End of update&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am listing the four answers that came up in the thread &quot;&lt;a href=&quot;http://stats.stackexchange.com/q/2512/28&quot;&gt;What should be our first Polystats Project?&lt;/a&gt;&quot; as separate answers in this question. Please vote up or downvote the answers depending on which one question you think we should consider for our first project. BTW, this question is CW so that downvotes and upvotes have no rep consequences.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A few points:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;We all should contribute to the project even if our favorite project question is not chosen by the community.&lt;/li&gt;&#xA;&lt;li&gt;I will keep the poll question open for 3 days (i.e., till 22nd September, 00 hours UTC) at which time I will ask the mods to lock the question.&lt;/li&gt;&#xA;&lt;li&gt;If there is a tie I will flip a coin to choose the specific question that we will look at.  &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Previous Questions reg Polystats-Project-1:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Question 1: &lt;a href=&quot;http://stats.stackexchange.com/q/2512/28&quot;&gt;What should be our first Polystats Project?&lt;/a&gt;&lt;br&gt;&#xA;Question 2: &lt;a href=&quot;http://stats.stackexchange.com/q/2662/28&quot;&gt;What should be the data collection strategy for our first Polystats Project?&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Relevant Meta Thread:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://meta.stats.stackexchange.com/q/419/28&quot;&gt;How about starting a Polystats Project?&lt;/a&gt;&lt;/p&gt;&#xA;" LastActivityDate="2010-09-23T00:07:53.833" CommunityOwnedDate="2010-09-20T00:20:26.977" Title="Which one of the following questions should we address for our first Polystats Project?" Tags="&lt;polystats-project-1&gt;" AnswerCount="8" FavoriteCount="1" />
  <row Id="2868" PostTypeId="2" ParentId="2867" CreationDate="2010-09-20T00:20:54.603" Score="3" ViewCount="" Body="&lt;p&gt;Are there reputation effects in SE communities and to what extent are reputation effects contingent on various variables such as size of the community, number of views per day, no of questions per day etc?&lt;/p&gt;&#xA;" LastActivityDate="2010-09-20T00:20:54.603" CommunityOwnedDate="2010-09-20T00:20:54.603" />
  <row Id="2869" PostTypeId="2" ParentId="2867" CreationDate="2010-09-20T00:21:22.840" Score="1" ViewCount="" Body="&lt;p&gt;Quantify the quality of answers and questions (what makes a question/answer a good one?)&lt;/p&gt;&#xA;" LastActivityDate="2010-09-20T00:21:22.840" CommunityOwnedDate="2010-09-20T00:21:22.840" />
  <row Id="2870" PostTypeId="2" ParentId="2867" CreationDate="2010-09-20T00:21:47.970" Score="2" ViewCount="" Body="&lt;p&gt;Classify how the communities differ.&lt;/p&gt;&#xA;" LastActivityDate="2010-09-20T00:21:47.970" CommunityOwnedDate="2010-09-20T00:21:47.970" />
  <row Id="2871" PostTypeId="2" ParentId="2867" CreationDate="2010-09-20T00:22:16.237" Score="1" ViewCount="" Body="&lt;p&gt;Is there any evidence for herd behavior? (i.e., Is the probability of a question/answer being voted up/down dependent on the existing up/down totals?)&lt;/p&gt;&#xA;" LastActivityDate="2010-09-20T00:22:16.237" CommunityOwnedDate="2010-09-20T00:22:16.237" />
  <row Id="2872" PostTypeId="2" ParentId="195" CreationDate="2010-09-20T00:29:57.047" Score="1" ViewCount="" Body="&lt;p&gt;I have been told many times that the Anderson Darling (AD) test is much better than the Kolmogorov-Smirnov (KS) one because AD does a better job at fitting the tails of the distribution.  KS is only good at fitting the mid-range of the distribution; but, is not better than AD even in this regard.  I think the main advantage of the KS test is its very intuitive visual interpretation (fitting of the respective cumulative distributions).  Because of the KS easy visual and intuitive interpretation it has become dominant in certain specialties such as credit scoring models within the financial service industry.  But, more visually intuitive does not mean better. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When using Monte Carlo simulation models that automatically fit a statistical distribution to a data set; their respective software manuals typically recommend leaning more on the AD than the KS test for the reason mentioned above (fits the tails better).       &lt;/p&gt;&#xA;" OwnerUserId="1329" LastActivityDate="2010-09-20T00:29:57.047" />
  <row Id="2873" PostTypeId="2" ParentId="2860" CreationDate="2010-09-20T00:38:48.693" Score="1" ViewCount="" Body="&lt;p&gt;I have never done this but my intuition suggests that the answer would depend to the extent to which the covariance matrix for the 500 data points is 'different' from the out-of-sample data. If the out-of-sample covariance matrix is very different then clearly the projection matrix of those points would be different than the projection matrix that emerges from the in-sample data. Thus, to the extent that the covariance matrix for the in-sample and out-of-sample data is 'similar' the results should be about the same.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The above intuition suggests that you should carefully select the 500 in-sample points so that the resulting covariance matrices are as identical as possible for the in-sample and the out-of-sample.&lt;/p&gt;&#xA;" LastActivityDate="2010-09-20T00:38:48.693" />
  <row Id="2875" PostTypeId="1" CreationDate="2010-09-20T01:19:26.897" Score="1" ViewCount="69" Body="&lt;p&gt;Hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My friend and I are working on a project on distributed datastructures. We were wondering how much is nearest neighbor information used in modern recommendation systems and whether it would be worthwhile to work on a distributed datastructure (say a kd-tree) for that purpose.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="250" LastActivityDate="2010-09-20T01:19:26.897" Title="Nearest neighbor information for recommendation engines" Tags="&lt;k-nearest-neighbour&gt;&lt;recommendations&gt;" FavoriteCount="1" />
  <row Id="2876" PostTypeId="2" ParentId="359" CreationDate="2010-09-20T01:56:02.970" Score="0" ViewCount="" Body="&lt;p&gt;is this, perhaps, just a case where the asymptotic approximations to the [null] distributions of one or more of the three test statistics are not so good? perhaps the results differ because the actual sizes of the tests are different? how large are your sample sizes?&lt;/p&gt;&#xA;" OwnerUserId="1112" LastEditorUserId="1112" LastEditorDisplayName="" LastEditDate="2010-09-20T02:40:27.090" LastActivityDate="2010-09-20T02:40:27.090" />
  <row Id="2877" PostTypeId="2" ParentId="2860" CreationDate="2010-09-20T03:38:03.447" Score="2" ViewCount="" Body="&lt;p&gt;This isn't unlike a model selection problem where the goal is to arrive at something close to the &quot;true dimensionality&quot; of the data.  You could try a cross validation approach, say 5-fold CV with your 500 data points.  This will give you a reasonable metric of generalization error for out-of-sample data.  The following paper has a nice survey and review of related methods:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.springerlink.com/content/vv7h068052v71335/&quot; rel=&quot;nofollow&quot;&gt;Cross-validation methods in principal component analysis: A comparison&lt;/a&gt; (Diana, Tommassi, 2002)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-09-20T03:38:03.447" />
  <row Id="2878" PostTypeId="2" ParentId="2863" CreationDate="2010-09-20T04:07:18.163" Score="3" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Reliability Analysis&lt;/strong&gt;: Yes, you should reverse score the reversed items.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Factor Analysis&lt;/strong&gt;: It does not matter so much. Eigenvalues and associated indices (e.g., variance explained by factors, rules of thumb regarding number of factors to extract, etc.) should be the same. The sign of factor loadings will flip based on whether you reverse reversed items.&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-09-20T04:07:18.163" />
  <row Id="2879" PostTypeId="2" ParentId="2867" CreationDate="2010-09-20T05:10:45.747" Score="0" ViewCount="" Body="&lt;p&gt;Can we predict how long a user will remain active on the site? (nice Survival Analysis exercise.)&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-09-20T05:10:45.747" CommunityOwnedDate="2010-09-20T05:10:45.747" />
  <row Id="2880" PostTypeId="2" ParentId="2867" CreationDate="2010-09-20T05:12:02.297" Score="0" ViewCount="" Body="&lt;p&gt;Can we cluster the tags somehow, or construct a similarity between them? (with an eye towards identifying redundant tags, perhaps.)&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-09-20T05:12:02.297" CommunityOwnedDate="2010-09-20T05:12:02.297" />
  <row Id="2881" PostTypeId="2" ParentId="2867" CreationDate="2010-09-20T05:17:01.317" Score="0" ViewCount="" Body="&lt;p&gt;Create a better metric than '% accept rate' to identify users who 'give back to the community'.&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-09-20T05:17:01.317" CommunityOwnedDate="2010-09-20T05:17:01.317" />
  <row Id="2882" PostTypeId="2" ParentId="2867" CreationDate="2010-09-20T05:23:55.793" Score="0" ViewCount="" Body="&lt;p&gt;Does rep really correspond to expertise? (I guess this is similar to the other question, &quot;are there rep effects?&quot; or &quot;snowball effect&quot;, but one possible outcome of this research would be a new way to compute rep.)&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-09-20T05:23:55.793" CommunityOwnedDate="2010-09-20T05:23:55.793" />
  <row Id="2883" PostTypeId="2" ParentId="2061" CreationDate="2010-09-20T06:05:15.013" Score="3" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://www.xtranormal.com/watch/6878253/&quot; rel=&quot;nofollow&quot;&gt;BIOSTATISTICS VS. LAB RESEARCH&lt;/a&gt;:&#xA;A funny/sad video on statistics consulting.&lt;/p&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-09-20T10:34:39.257" LastActivityDate="2010-09-20T10:34:39.257" CommunityOwnedDate="2010-09-20T06:05:15.013" />
  <row Id="2884" PostTypeId="2" ParentId="2852" CreationDate="2010-09-20T06:14:15.227" Score="0" ViewCount="" Body="&lt;p&gt;Thank you very much, chl, whuber and Gaetan Lion. But do you think is there any problem that if I change to caculate the differene among the data using Kruskal-Wallis test instead of comparing the difference between the first column with other columns?&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;kruskal.test(as.list(Data))&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    Kruskal-Wallis rank sum test&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;data:  as.list(Data) &#xA;Kruskal-Wallis chi-squared = 19.9149, df = 5, p-value = 0.001297&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;kruskal.test(as.list(Data[,2:6]))&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    Kruskal-Wallis rank sum test&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;data:  as.list(Data[, 2:6]) &#xA;Kruskal-Wallis chi-squared = 3.8242, df = 4, p-value = 0.4303&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The result also shows the 1st column has great difference between the other columns. Is it that right?&lt;/p&gt;&#xA;" LastActivityDate="2010-09-20T11:29:59.137" CommentCount="3" />
  <row Id="2885" PostTypeId="2" ParentId="2828" CreationDate="2010-09-20T06:20:50.417" Score="2" ViewCount="" Body="&lt;p&gt;Minimum Description Length (MDL) and Minimum Message Length (MML) are certainly worth checking out.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as MDL is concerned, a simple paper that illustrates the Normalized Maximum Likelihood (NML) procedure as well as the asymptotic approximation is:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;S. de Rooij &amp;amp; P. Grünwald. An empirical&#xA;  study of minimum description length&#xA;  model selection with infinite&#xA;  parametric complexity.  Journal of&#xA;  Mathematical Psychology, 2006, 50,&#xA;  180-192&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Here, they look at the model complexity of a Geometric vs. a Poisson distribution. An excellent (free) tutorial on MDL can be found &lt;a href=&quot;http://www.cwi.nl/~pdg/ftp/mdlintro.pdf&quot; rel=&quot;nofollow&quot; title=&quot;here&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively, a paper on the complexity of the exponential distribution examined with both MML and MDL can be found &lt;a href=&quot;http://ieeexplore.ieee.org/Xplore/login.jsp?url=http%3A%2F%2Fieeexplore.ieee.org%2Fiel5%2F18%2F5075865%2F05075876.pdf%3Farnumber%3D5075876&amp;amp;authDecision=-203&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;. Unfortunately, there is no up-to-date tutorial on MML, but the &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/038723795X&quot; rel=&quot;nofollow&quot;&gt;book&lt;/a&gt; is an excellent reference, and highly recommended.&lt;/p&gt;&#xA;" OwnerUserId="530" LastActivityDate="2010-09-20T06:20:50.417" CommentCount="2" />
  <row Id="2886" PostTypeId="1" AcceptedAnswerId="2887" CreationDate="2010-09-20T06:44:20.220" Score="5" ViewCount="338" Body="&lt;p&gt;As title, I am thinking of merging both into &quot;missing data&quot;, which is to name it as NA in R. Since I don't see it will make much sense (or even any sense), to separate the &quot;don't know&quot; row out and to compare the information with other rows.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it OK for me to do so? Thanks!&lt;/p&gt;&#xA;" OwnerUserId="588" LastActivityDate="2010-09-20T07:39:17.980" Title="How will you deal with &quot;don't know&quot; and &quot;missing data&quot; in survey data?" Tags="&lt;survey&gt;&lt;missing-data&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="2887" PostTypeId="2" ParentId="2886" CreationDate="2010-09-20T07:08:01.583" Score="6" ViewCount="" Body="&lt;p&gt;Well, you should also considered that &quot;don't know&quot; is at least some kind of answer, whereas non-response is a purely missing value. Now, we often allow for &quot;don't know&quot; response in survey just to avoid forcing people to provide a response anyway (which might bias the results). For example, in the National Health and Nutrition Examination Survey, they are coded differently but subsequently discarded from the analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could try analyzing the data both ways: (1) treating &quot;don't know response&quot; as specific response category and handling all responses set with some kind of multivariate data analysis (e.g. &lt;a href=&quot;http://en.wikipedia.org/wiki/Multiple_correspondence_analysis&quot; rel=&quot;nofollow&quot;&gt;multiple correspondence analysis&lt;/a&gt; or multiple factor analysis for mixed data, see the &lt;a href=&quot;http://cran.r-project.org/web/packages/FactoMineR/index.html&quot; rel=&quot;nofollow&quot;&gt;FactoMineR&lt;/a&gt; package), and (2) if it doesn't bring any evidence of distortion on items distribution, just merge it with missing values. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For (2), I would also suggest you to check that &quot;don't know&quot; and MV are at least missing at random (MAR), or that they are not specific of one respondents group (e.g. male/female, age class, SES, etc.).&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-09-20T07:08:01.583" />
  <row Id="2888" PostTypeId="2" ParentId="1856" CreationDate="2010-09-20T07:29:51.983" Score="3" ViewCount="" Body="&lt;p&gt;I haven't seen this used in outside of bioinformatics/machine learning either, but maybe you can be the first one :)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a good representative of small sample method method from bioinformatics, logistic regression with L1 regularization can give a good fit when number of parameters is exponential in the number of observations, non-asymptotic confidence intervals can be crafted using Chernoff-type inequalities (ie, &lt;a href=&quot;http://www2.research.att.com/~phillips/pdf/colt_2004.pdf&quot; rel=&quot;nofollow&quot;&gt;Dudik, (2004)&lt;/a&gt; for example). Trevor Hastie has done some work applying these methods to identifying gene interactions. In the paper below, he uses it to identify significant effects from a model with 310,637 adjustable parameters fit to a sample of 2200 observations&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Genome-wide association analysis by lasso penalized logistic regression.&quot;&#xA;Authors: Hastie, T; Sobel, E; Wu, T. T; Chen, Y. F; Lange, K&#xA;Bioinformatics Vol: 25 Issue: 6 ISSN: 1367-4803 Date: 03/2009 Pages: 714 - 721&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Related presentation by Victoria Stodden (&lt;a href=&quot;http://www.stanford.edu/~vcs/talks/MicrosoftMay082008.pdf&quot; rel=&quot;nofollow&quot;&gt;Model Selection with Many More Variables than Observations &lt;/a&gt;)&lt;/p&gt;&#xA;" OwnerUserId="511" LastEditorUserId="511" LastEditorDisplayName="" LastEditDate="2010-10-01T15:56:38.640" LastActivityDate="2010-10-01T15:56:38.640" CommentCount="3" />
  <row Id="2889" PostTypeId="2" ParentId="2886" CreationDate="2010-09-20T07:31:33.540" Score="1" ViewCount="" Body="&lt;p&gt;It depends on the type of question/response in your survey. If they are like &quot;I like&quot;, &quot;I dislike&quot;, &quot;Don't know&quot;, chl answers partially to your question. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first solution is chl's answer. You have to check if &quot;Don't know&quot; doesn't hide anything. You have to analyse separately these values to see if it highlights a specific profile of respondents.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not about imputation but... &quot;Frenchy&quot; software do it for MCA, ... often considering MAR assumption. It supposes that these answers are randomly distributed (you pick randomly another modality of response).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also use a more sophisticated approach : if &quot;Like&quot; is at 30% and &quot;Dislike&quot; at 70% you pick an uniform random number distributed on (0,1) and choose &quot;Like&quot; if your number is at or below 0.3. If you pick a number between 0.3 and 1 you choose &quot;Dislike&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A more modern approach is Multiplie imputation (see MI PROC in SAS and mice package in R). Imputation is very efficient... But it can't recreate atypic profiles...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're working in educational testing or if you need to compute a score, let me know I will complete this answer about scores estimation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ref:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Multiple Imputation for Nonreponse in survey, Rubin (1987). Wiley.&#xA;mice package: &lt;a href=&quot;http://cran.r-project.org/web/packages/mice/index.html&quot; rel=&quot;nofollow&quot;&gt;http://cran.r-project.org/web/packages/mice/index.html&lt;/a&gt;&#xA;Survey Methodology, Robert M. Groves, Floyd J. Fowler &amp;amp; al. Wiley.&lt;/p&gt;&#xA;" OwnerUserId="1154" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-20T07:39:17.980" LastActivityDate="2010-09-20T07:39:17.980" />
  <row Id="2890" PostTypeId="2" ParentId="2828" CreationDate="2010-09-20T08:40:00.997" Score="6" ViewCount="" Body="&lt;p&gt;Besides the various measures of &lt;strong&gt;Minimum Description Length&lt;/strong&gt; (e.g., normalized maximum likelihood, Fisher Information approximation), there are two other methods worth to mention:  &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Parametric Bootstrap&lt;/strong&gt;. It's a lot easier to implement than the demanding MDL measures. A nice paper is by Wagenmaker and colleagues:&lt;br&gt;&#xA;Wagenmakers, E.-J., Ratcliff, R., Gomez, P., &amp;amp; Iverson, G. J. (2004). &lt;a href=&quot;http://www.ejwagenmakers.com/2004/PBCM.pdf&quot; rel=&quot;nofollow&quot;&gt;Assessing model mimicry using the parametric bootstrap&lt;/a&gt;. &lt;em&gt;Journal of Mathematical Psychology&lt;/em&gt;, 48, 28-50.&lt;br&gt;&#xA;The abstract:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;We present a general sampling&#xA;  procedure to quantify model mimicry,&#xA;  defined as the ability of a model to&#xA;  account for data generated by a&#xA;  competing model. This sampling&#xA;  procedure, called the parametric&#xA;  bootstrap cross-fitting method (PBCM;&#xA;  cf. Williams (J. R. Statist. Soc. B 32&#xA;  (1970) 350; Biometrics 26 (1970) 23)),&#xA;  generates distributions of differences&#xA;  in goodness-of-fit expected under each&#xA;  of the competing models. In the data&#xA;  informed version of the PBCM, the&#xA;  generating models have specific&#xA;  parameter values obtained by fitting&#xA;  the experimental data under&#xA;  consideration. The data informed&#xA;  difference distributions can be&#xA;  compared to the observed difference in&#xA;  goodness-of-fit to allow a&#xA;  quantification of model adequacy. In&#xA;  the data uninformed version of the&#xA;  PBCM, the generating models have a&#xA;  relatively broad range of parameter&#xA;  values based on prior knowledge.&#xA;  Application of both the data informed&#xA;  and the data uninformed PBCM is&#xA;  illustrated with several examples.  &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update: Assessing model mimicry in plain English.&lt;/strong&gt; You take one of the two competing models and randomly pick a set of parameters for that model (either data informed or not). Then, you produce data from this model with the picked set of parameters. Next, you let both models fit the produced data and check which of the two candidate models gives the better fit. If both models are equally flexible or complex, the model from which you produced the data should  give a better fit. However, if the other model is more complex, it could give a better fit, although the data was produced from the other model. You repeat this several times with both models (i.e., let both models produce data and look which of the two fits better). The model that &quot;overfits&quot; the data produced by the other model is the more complex one.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Cross-Validation&lt;/strong&gt;: It is also quite easy to implement. &lt;a href=&quot;http://stats.stackexchange.com/questions/1826/cross-validation-in-plain-english&quot;&gt;See the answers to this question&lt;/a&gt;. However, note that the issue with it is that the choice among the sample-cutting rule (leave-one-out, K-fold, etc) is an unprincipled one. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="442" LastEditorUserId="442" LastEditorDisplayName="" LastEditDate="2010-09-21T08:45:45.537" LastActivityDate="2010-09-21T08:45:45.537" CommentCount="2" />
  <row Id="2891" PostTypeId="1" AcceptedAnswerId="5695" CreationDate="2010-09-20T09:07:06.277" Score="1" ViewCount="98" Body="&lt;p&gt;I'm looking for a simple way to store ratios.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a time component, I must store the average ratio between two behavior. For example the number of people that turn left compared to the number of people that turn right.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have to detect unusual behavior (people that turn right abnormally).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How should I mathematically compare the average ratio against the analyzed ratio, and how should I display the difference on a graph ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks a lot in advance.&lt;/p&gt;&#xA;" LastEditorUserId="1739" LastEditorDisplayName="" LastEditDate="2010-12-22T14:22:27.143" LastActivityDate="2010-12-22T14:22:27.143" Title="Best value to store ratio data and compare it to time period average" Tags="&lt;data-visualization&gt;&lt;multiple-comparisons&gt;&lt;count-data&gt;&lt;logit&gt;&lt;proportion&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="2892" PostTypeId="1" AcceptedAnswerId="2905" CreationDate="2010-09-20T10:26:08.910" Score="3" ViewCount="498" Body="&lt;p&gt;What is your intuition / interpretation of a distribution of eigenvalues of a correlation matrix? I tend to hear that usually 3 largest eigenvalues are the most important, while those close to zero are noise. Also, I've seen a few research papers investigating how naturally occuring eigenvalue distributions differ from those calculated from random correlation matrices (again, distinguising noise from signal).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please feel free to elaborate on your insights.&lt;/p&gt;&#xA;" OwnerUserId="1250" LastActivityDate="2010-09-21T11:59:13.977" CommunityOwnedDate="2010-09-21T00:56:34.300" Title="Intuition / interpretation of a distribution of eigenvalues of a correlation matrix?" Tags="&lt;distributions&gt;&lt;correlation&gt;" AnswerCount="3" CommentCount="2" FavoriteCount="1" />
  <row Id="2893" PostTypeId="1" AcceptedAnswerId="2897" CreationDate="2010-09-20T10:56:57.297" Score="7" ViewCount="344" Body="&lt;p&gt;It is usual to use second, third and fourth moments of a distribution to describe certain properties. Do partial moments or moments higher than the fourth describe any useful properties of a distribution?&lt;/p&gt;&#xA;" OwnerUserId="1250" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-09-23T22:08:32.743" LastActivityDate="2011-03-24T18:26:30.237" Title="Moments of a distribution - any use for partial or higher moments?" Tags="&lt;distributions&gt;&lt;moments&gt;" AnswerCount="3" FavoriteCount="2" />
  <row Id="2894" PostTypeId="1" CreationDate="2010-09-20T13:24:09.147" Score="8" ViewCount="133" Body="&lt;p&gt;I am trying to estimate the mean of a more-or-less Gaussian distribution via sampling. I have no prior knowledge about its mean or its variance. Each sample is expensive to obtain. How do I dynamically decide how many samples I need to get a certain level of confidence/accuracy? Alternatively, how do I know when I can stop taking samples?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All the answers to questions like this that I can find seem to presume some knowledge of the variance, but I need to discover that along the way as well. Other are geared towards taking polls, and it's not clear to me (beginner that I am) how that generalizes -- my mean isn't w/in [0,1], etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think this is probably a simple question with a well known answer, but my Google-fu is failing me. Even just telling me what to search for would be helpful.&lt;/p&gt;&#xA;" OwnerUserId="1376" LastEditorUserId="1376" LastEditorDisplayName="" LastEditDate="2010-09-20T13:29:12.717" LastActivityDate="2010-09-20T15:51:22.327" CommunityOwnedDate="2010-09-20T13:25:02.783" Title="Dynamic calculation of number of samples required to estimate the mean" Tags="&lt;estimation&gt;&lt;sample-size&gt;" AnswerCount="2" CommentCount="8" FavoriteCount="1" />
  <row Id="2895" PostTypeId="2" ParentId="665" CreationDate="2010-09-20T13:59:30.777" Score="2" ViewCount="" Body="&lt;p&gt;Probability studies, well, how probable events are. You intuitively know what probability is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Statistics is the study of data: showing it (using tools such as charts), summarizing it (using means and standard deviations etc.), reaching conclusions about the world from which that data was drawn (fitting lines to data etc.), and -- this is key -- quantifying how sure we can be about our conclusions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to quantify how sure we can be about our conclusions we need to use Probability. Let's say you have last year's data about rainfall in the region where you live and where I live. Last year it rained an average of 1/4 inch per week where you live, and 3/8 inch where I live. So we can say that rainfall in my region is on average 50% greater than where you live, right? Not so fast, Sparky. It could be a coincidence: maybe it just happened to rain a lot last year where I live. We can use Probability to estimate how confident we can be in our conclusion that my home is 50% soggier than yours.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So basically you can say that Probability is the mathematical foundation for the Theory of Statistics.&lt;/p&gt;&#xA;" OwnerUserId="666" LastActivityDate="2010-09-20T13:59:30.777" />
  <row Id="2896" PostTypeId="2" ParentId="2894" CreationDate="2010-09-20T15:19:18.547" Score="0" ViewCount="" Body="&lt;p&gt;You would normally want at least 30 to invoke central limit theorem (though this is somewhat arbitrary). Unlike in the case with polls etc, which are modelled using the binomial distribution, you can not determine a sample size beforehand which guarantees a level of accuracy with a Gaussian process - it depends on what residuals you get which determine the standard error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It should be noted that if you have a robust sampling strategy, you can get much more accurate results than with a much larger sample size with a poor strategy.&lt;/p&gt;&#xA;" OwnerUserId="229" LastEditorUserId="229" LastEditorDisplayName="" LastEditDate="2010-09-20T15:44:52.563" LastActivityDate="2010-09-20T15:44:52.563" CommunityOwnedDate="2010-09-20T15:19:18.547" CommentCount="2" />
  <row Id="2897" PostTypeId="2" ParentId="2893" CreationDate="2010-09-20T15:22:46.340" Score="4" ViewCount="" Body="&lt;p&gt;Aside from special properties of a few numbers (e.g., 2), the only real reason to single out integer moments as opposed to fractional moments is convenience.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Higher moments can be used to understand tail behavior.  For example, a centered random variable $X$ with variance 1 has subgaussian tails (i.e. $\mathbb{P}(|X| &amp;gt; t) &amp;lt; C e^{-ct^2}$ for some constants $c,C &gt; 0$) if and only if $\mathbb{E}|X|^p \le (A \sqrt{p})^p$ for every $p\ge 1$ and some constant $A &amp;gt; 0$.&lt;/p&gt;&#xA;" OwnerUserId="89" LastEditorUserId="89" LastEditorDisplayName="" LastEditDate="2011-03-24T18:26:30.237" LastActivityDate="2011-03-24T18:26:30.237" CommentCount="4" />
  <row Id="2898" PostTypeId="2" ParentId="2894" CreationDate="2010-09-20T15:51:22.327" Score="2" ViewCount="" Body="&lt;p&gt;You need to search for 'Bayesian adaptive designs'. The basic idea is as follows:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;You initialize the prior for the parameters of interest.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Before any data collection your priors would be diffuse. As additional data comes in you  re-set the prior to be the posterior that corresponds to the 'prior + data till that point in time'. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Collect data.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Compute the posterior based on data + priors. The posterior is then used as the prior in step 1 if you actually collect additional data.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Assess whether your stopping criteria are met&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Stopping criteria could include something like the 95% credible interval should not be bigger than $\pm \epsilon$ units for the parameters of interest. You could also have more formal loss functions associated with the parameters of interest and compute expected loss with respect to the posterior distribution for the parameter of interest.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;You then repeat steps 1, 2 and 3 till your stopping criteria from step 4 are met.&lt;/p&gt;&#xA;" LastActivityDate="2010-09-20T15:51:22.327" CommunityOwnedDate="2010-09-20T15:51:22.327" />
  <row Id="2899" PostTypeId="2" ParentId="2893" CreationDate="2010-09-20T15:54:47.243" Score="5" ViewCount="" Body="&lt;p&gt;I get suspicious when I hear people ask about third and fourth moments.  There are two common errors people often have in mind when they bring up the topic. I'm not saying that you are necessarily making these mistakes, but they do come up often.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, it sounds like they implicitly believe that distributions can be boiled down to four numbers; they suspect that just two numbers is not enough, but three or four should be plenty.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second, it sounds like hearkening back to the moment-matching approach to statistics that has largely lost out to maximum likelihood methods in contemporary statistics. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Update: I expanded this answer into a &lt;a href=&quot;http://www.johndcook.com/blog/2010/09/20/skewness-andkurtosis/&quot;&gt;blog post&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="319" LastEditorUserId="319" LastEditorDisplayName="" LastEditDate="2010-09-20T17:31:39.423" LastActivityDate="2010-09-20T17:31:39.423" />
  <row Id="2900" PostTypeId="2" ParentId="1805" CreationDate="2010-09-20T16:15:58.623" Score="1" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://mathworld.wolfram.com/FishersExactTest.html&quot; rel=&quot;nofollow&quot;&gt;This page in MathWorld&lt;/a&gt;&#xA;explains how the calculations work. It points out that the test can be defined in a variety of ways: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;To compute the P-value of the test,&#xA;  the tables must be ordered by some&#xA;  criterion that measures dependence,&#xA;  and those tables that represent equal&#xA;  or greater deviation from independence&#xA;  than the observed table are the ones&#xA;  whose probabilities are added&#xA;  together. There are a variety of&#xA;  criteria that can be used to measure&#xA;  dependence. &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I have not been able to find other articles or texts that explain how this is done with tables larger than 2x2. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.quantitativeskills.com/sisa/statistics/five2hlp.htm&quot; rel=&quot;nofollow&quot;&gt;This calculator&lt;/a&gt;&#xA; computes the exact Fisher's test for tables with 2 columns and up to 5 rows. The criterion it uses is the hypergeometric probability of each table. The overall P value is the sum of the hypergeometric probability of all tables with the same marginal totals whose probabilities are less than or equal to the probability computed from the actual data.&lt;/p&gt;&#xA;" OwnerUserId="25" LastEditorUserId="25" LastEditorDisplayName="" LastEditDate="2010-09-20T16:49:00.773" LastActivityDate="2010-09-20T16:49:00.773" CommentCount="0" />
  <row Id="2901" PostTypeId="2" ParentId="2893" CreationDate="2010-09-20T16:42:10.380" Score="1" ViewCount="" Body="&lt;p&gt;One example of use (interpretation is a better qualifier) of a higher moment: the fifth moment of a univariate distribution measures the asymmetry of its tails.&lt;/p&gt;&#xA;" OwnerUserId="603" LastEditorUserId="603" LastEditorDisplayName="" LastEditDate="2010-09-20T20:07:31.340" LastActivityDate="2010-09-20T20:07:31.340" CommentCount="9" />
  <row Id="2902" PostTypeId="2" ParentId="2862" CreationDate="2010-09-20T16:52:43.323" Score="2" ViewCount="" Body="&lt;p&gt;Natural experiments are good working material for MA-thesis (less issue statistically speaking, quicker path to results,...).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A good one recently in finance deals with short selling ban: in many countries in the euro-zone, in 2008, short sales were banned on some banks/financial corporations in some countries (Greece, France, Belgium, Germany) while still allowed in others. You could try to find some banks that are similar in many characteristic (size, volatility in the years leading to 2008, type of assets) and see to what extend there behavior (volatility) was different during the crisis conditional to being in a no-short sale country or not. &lt;/p&gt;&#xA;" OwnerUserId="603" LastActivityDate="2010-09-20T16:52:43.323" CommunityOwnedDate="2010-09-20T16:52:43.323" CommentCount="2" />
  <row Id="2903" PostTypeId="2" ParentId="2730" CreationDate="2010-09-20T16:54:56.517" Score="4" ViewCount="" Body="&lt;p&gt;Gelman has a good discussion paper on &lt;a href=&quot;http://projecteuclid.org/euclid.aos/1112967698&quot; rel=&quot;nofollow&quot;&gt;ANOVA&lt;/a&gt;&#xA;&lt;em&gt;Analysis of variance—why it is more important than ever&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="603" LastActivityDate="2010-09-20T16:54:56.517" />
  <row Id="2904" PostTypeId="1" CreationDate="2010-09-20T17:45:15.683" Score="4" ViewCount="269" Body="&lt;p&gt;Hi there,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am attempting to estimate a model of the following form:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;W = alphaH * H + alphaM * M + alphaL * L + X * beta&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where &lt;code&gt;H, M, L&lt;/code&gt; are indicators for a discrete choice variable, and &lt;code&gt;beta&lt;/code&gt; is something like 35-dimensional. Because we believe our data/model has endogeneity issues, we have expanded the model to&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;W  = alphaH * H' + alphaM * M' + alphaL * L' + X * beta&#xA;H  = Z * betaH&#xA;M  = Z * betaM&#xA;L  = Z * betaL&#xA;H' = 1( H = max(H,M,L) )&#xA;M' = 1( M = max(H,M,L) )&#xA;L' = 1( L = max(H,M,L) )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where &lt;code&gt;Z&lt;/code&gt; are instruments, and &lt;code&gt;betaH, betaM, betaL&lt;/code&gt; are parameters to be estimated. This &quot;subregression&quot; corresponds to a latent utility-based choice model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We have been able to estimate the second-stage model (estimates of &lt;code&gt;H, M, L&lt;/code&gt;, implying &lt;code&gt;H', M', L'&lt;/code&gt;) in Stata using the &lt;code&gt;mvprobit&lt;/code&gt; command, but can't figure out how to estimate the entire model in one fell swoop. To work around this, we wrote some code in MATLAB to estimate the model using simulated maximum likelihood, but MATLAB is choking on local minima (maxima in this problem, but MATLAB will only minimize the negative...), of which there are plenty.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We have attempted to work around this by starting from a few dozen initial conditions, none of which usually converges to the &quot;right&quot; answer; I say this with near certainty since we have been testing the code piecewise and have confirmation (on randomized test data) that if the optimization starts near the &quot;correct&quot; values (in test), it converges to reasonable values, otherwise it gets nowhere close (although the resultant outcome has a far lower overall likelihood).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any tricks -- MATLAB, Stata, or otherwise -- to work around this problem? Is this an inherent issue with simulation versus closed-form analysis?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for your help.&lt;/p&gt;&#xA;" OwnerUserId="53" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-11-16T23:35:41.633" LastActivityDate="2010-11-16T23:35:41.633" Title="How can I work around &quot;lumpiness&quot; in simulated maximum likelihood estimation?" Tags="&lt;matlab&gt;&lt;stata&gt;&lt;maximum-likelihood&gt;&lt;optimization&gt;" AnswerCount="3" CommentCount="3" FavoriteCount="1" />
  <row Id="2905" PostTypeId="2" ParentId="2892" CreationDate="2010-09-20T18:49:08.317" Score="2" ViewCount="" Body="&lt;p&gt;&lt;em&gt;I tend to hear that usually 3 largest eigenvalues are the most important, while those close to zero are noise&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can test for that. See the paper linked in &lt;a href=&quot;http://stats.stackexchange.com/questions/2860/pca-on-out-of-sample-data/2877#2877&quot;&gt;this&lt;/a&gt; post for more detail. Again if your dealing with financial times series you might wanna correct for leptokurticity first (i.e. consider the series of garch-adjusted returns, not the raw returns).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;I've seen a few research papers investigating how naturally occuring eigenvalue distributions differ from those calculated from random correlation matrices (again, distinguising noise from signal).&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edward:&gt; Usually, one would do it the other way arround: look at the multivariate distribution of eigenvalues (of correlation matrices) coming from the application you want. Once you have identified a credible candidate for the distribution of eigenvalues, it should be fairly easy to generate from them. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The best procedure on how to identify the multivariate distribution of your eigenvalues depends on how many assets you want to consider simultaneously (i.e. what are the dimensions of your correlation matrix). There is a neat trick if $p\leq 10$ ($p$ being the number of assets). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit (comments by Shabbychef)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;four step procedure:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Suppose you have $j=1,...,J$ sub samples of multivariate data. You need an estimator of the variance-covariance matrix $\tilde{C}_j$ for each sub-sample $j$ ( you could use the classical estimator or a robust alternative such as the &lt;a href=&quot;http://agoras.ua.ac.be/abstract/Fasalg99.htm&quot; rel=&quot;nofollow&quot;&gt;fast MCD&lt;/a&gt;, which is well implemented in matlab, SAS, S,R,...). As usual, if your dealing with financial times series you would want to consider the series of garch-adjusted returns, not raw returns.&lt;/li&gt;&#xA;&lt;li&gt;For each sub sample $j$, compute $\tilde{\Lambda}_j=$  $\log(\tilde{\lambda}_1^j)$ ,..., $\log(\tilde{\lambda}_p^j)$ ,  the eigen values of $\tilde{C}_j$.&lt;/li&gt;&#xA;&lt;li&gt;Compute $CV(\tilde{\Lambda})$, the convex hull of the $J \times p$ matrix whose j-th entry is $\tilde{\Lambda}_j$ (again, this is well implemented in Matlab, R,...).&lt;/li&gt;&#xA;&lt;li&gt;Draw points at random from inside $CV(\tilde{\Lambda})$  (this done by giving weight $w_i$ to each of the edges of $CV(\tilde{\Lambda})$ where $w_i=\frac{\gamma_i}{\sum_{i=1}^{p}\gamma_i}$, where $\gamma_i$ is a draw from an unit exponential distribution (more details &lt;a href=&quot;http://geomblog.blogspot.com/2005/10/sampling-from-simplex.html3.&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;A limitation is that fast computation of the convex hull of a series of points becomes extremely slow when the number of dimensions is larger than 10. $J\geq2$&lt;/p&gt;&#xA;" OwnerUserId="603" LastEditorUserId="603" LastEditorDisplayName="" LastEditDate="2010-09-21T11:59:13.977" LastActivityDate="2010-09-21T11:59:13.977" CommunityOwnedDate="2010-09-21T00:39:28.817" CommentCount="5" />
  <row Id="2906" PostTypeId="1" AcceptedAnswerId="2908" CreationDate="2010-09-20T18:57:24.617" Score="4" ViewCount="124" Body="&lt;p&gt;Nassim Taleb, of &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/081297381X&quot; rel=&quot;nofollow&quot;&gt;Black Swan&lt;/a&gt; fame (or infamy), has elaborated on the concept and developed what he calls &lt;a href=&quot;http://www.edge.org/3rd_culture/taleb08/taleb08_index.html&quot; rel=&quot;nofollow&quot;&gt;&quot;a map of the limits of Statistics&quot;&lt;/a&gt;. His basic argument is that there is one kind of decision problem where the use of any statistical model is harmful. These would be any decision problems where the consequence of making the wrong decision could be inordinately high, and the underlying PDF is hard to know.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One example would be shorting a stock option. This kind of operation can lead to limitless (in theory, at least) loss; and the probability of such a loss is unknown. Many people in fact model the probability, but Taleb argues that the financial markets aren't old enough to allow one to be confident about any model. Just because every swan you have ever seen is white, that doesn't mean black swans are impossible or even unlikely.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So here's the question: is there such a thing as a consensus in the Statistics community about Mr. Taleb's arguments?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maybe this should be community wiki. I don't know.&lt;/p&gt;&#xA;" OwnerUserId="666" LastActivityDate="2010-09-20T19:13:08.843" CommunityOwnedDate="2010-09-20T20:44:51.923" Title="What is the community's take on the Fourth Quadrant?" Tags="&lt;distributions&gt;&lt;modeling&gt;&lt;random-variable&gt;" AnswerCount="1" FavoriteCount="4" />
  <row Id="2907" PostTypeId="2" ParentId="2686" CreationDate="2010-09-20T19:11:45.167" Score="1" ViewCount="" Body="&lt;p&gt;I would start with robust time series &lt;a href=&quot;http://cran.r-project.org/web/packages/robfilter/index.html&quot; rel=&quot;nofollow&quot;&gt;filters&lt;/a&gt; (i.e. time varying medians) because these are more simple and intuitive.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically, the robust time filter is to time series smoothers what the median is to the mean; a summary measures (in this case a time varying one) that is not sensitive to 'wired' observations so long as they do not represent the majority of the data. For a summary see &lt;a href=&quot;http://en.wikipedia.org/wiki/Robust_statistics#Example%3a_speed_of_light_data&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you need more sophisticated smoothers (i.e. non linear ones), you could do with robust Kalman &lt;a href=&quot;http://robkalman.r-forge.r-project.org/&quot; rel=&quot;nofollow&quot;&gt;filtering&lt;/a&gt; (although this requieres a slightly higher level of mathematical sophistication) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://cran.r-project.org/web/packages/robfilter/robfilter.pdf&quot; rel=&quot;nofollow&quot;&gt;This&lt;/a&gt; document contains the following example ( a code to run under &lt;a href=&quot;http://cran.r-project.org/mirrors.html&quot; rel=&quot;nofollow&quot;&gt;R&lt;/a&gt;, the open source stat software):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(robfilter)&#xA;data(Nile)&#xA;nile &amp;lt;- as.numeric(Nile)&#xA;obj &amp;lt;- wrm.filter(nile, width=11)&#xA;plot(obj)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/O3VJl.jpg&quot; alt=&quot;where the orginal time series is in black and the filtered version (filtered by repeated median) is overploted in red&quot;&gt;. The last documents contains a large number of references to papers and books. Other types of filters are implemented in the package, but the repeated median is a very simple one.&lt;/p&gt;&#xA;" OwnerUserId="603" LastEditorUserId="603" LastEditorDisplayName="" LastEditDate="2010-09-22T12:44:41.213" LastActivityDate="2010-09-22T12:44:41.213" CommentCount="2" />
  <row Id="2908" PostTypeId="2" ParentId="2906" CreationDate="2010-09-20T19:13:08.843" Score="8" ViewCount="" Body="&lt;p&gt;I was at a meeting of the ASA (American Statistical Association) a couple years ago where Taleb talked about his &quot;fourth quadrant&quot; and it seemed his remarks were well received.  Taleb was much more careful in his language when addressing an auditorium of statisticians than he has been in his popular writing.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some statisticians are offended by the provocative hyperbole in Taleb's books, but when he states his ideas professionally there's not too much to object to.  It's hard to argue that one can confidently estimate the probability of rare events with little or no data, or that one should make high-stakes decisions on such estimates if they can at all be avoided.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Here's a &lt;a href=&quot;http://www.johndcook.com/blog/2008/08/07/black-swan-talk/&quot; rel=&quot;nofollow&quot;&gt;blog post&lt;/a&gt; I wrote about Taleb's ASA talk shortly after the event.)&lt;/p&gt;&#xA;" OwnerUserId="319" LastActivityDate="2010-09-20T19:13:08.843" CommunityOwnedDate="2010-09-20T20:44:51.923" CommentCount="3" />
  <row Id="2909" PostTypeId="1" AcceptedAnswerId="4033" CreationDate="2010-09-20T19:59:31.487" Score="5" ViewCount="282" Body="&lt;p&gt;I am interested in the distribution of the maximum drawdown of a random walk: Let $X_0 = 0, X_{i+1} = X_i + Y_{i+1}$ where $Y_i \sim \mathcal{N}(\mu,1)$. The maximum drawdown after $n$ periods is $\max_{0 \le i \le j \le n} (X_i - X_j)$. A paper by &lt;a href=&quot;http://www.alumni.caltech.edu/~amir/drawdown-jrnl.pdf&quot; rel=&quot;nofollow&quot;&gt;Magdon-Ismail &lt;em&gt;et. al.&lt;/em&gt;&lt;/a&gt; gives the distribution for maximum drawdown of a Brownian motion with drift. The expression involves an infinite sum which includes some terms defined only implicitly. I am having problems writing an implementation which converges.  Is anyone aware of an alternative expression of the CDF or a reference implementation in code?&lt;/p&gt;&#xA;" OwnerUserId="795" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-10-27T20:46:36.273" LastActivityDate="2010-10-28T20:12:38.843" Title="Computing the cumulative distribution of max drawdown of random walk with drift" Tags="&lt;finance&gt;&lt;code&gt;&lt;cdf&gt;" AnswerCount="2" CommentCount="3" FavoriteCount="2" />
  <row Id="2910" PostTypeId="1" AcceptedAnswerId="3191" CreationDate="2010-09-20T20:39:08.183" Score="21" ViewCount="1191" Body="&lt;p&gt;We often hear of project management and design patterns in computer science, but less frequently in statistical analysis. However, it seems that a decisive step toward designing an effective and durable statistical project is to keep things organized. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I often advocate the use of R and a consistent organization of files in separate folders (raw data file, transformed data file, R scripts, figures, notes, etc.). The main reason for this approach is that it may be easier to run your analysis later (when you forgot how you happened to produce a given plot, for instance).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are the &lt;strong&gt;best practices for statistical project management&lt;/strong&gt;, or the recommendations you would like to give from your own experience? Of course, this applies to any statistical software. (&lt;em&gt;one answer per post, please&lt;/em&gt;)&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-10T09:17:51.210" LastActivityDate="2011-04-10T09:17:51.210" CommunityOwnedDate="2010-09-20T20:39:08.183" Title="How to efficiently manage a statistical analysis project?" Tags="&lt;best-practices&gt;&lt;project-management&gt;" AnswerCount="5" FavoriteCount="20" />
  <row Id="2911" PostTypeId="2" ParentId="2910" CreationDate="2010-09-20T20:42:21.877" Score="7" ViewCount="" Body="&lt;p&gt;This doesn't specifically provide an answer, but you may want to look at these related stackoverflow questions: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://stackoverflow.com/questions/1429907/workflow-for-statistical-analysis-and-report-writing&quot;&gt;&quot;Workflow for statistical analysis and report writing&quot;&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://stackoverflow.com/questions/2284446/organizing-r-source-code&quot;&gt;&quot;Organizing R Source Code&quot;&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://stackoverflow.com/questions/1266279/how-to-organize-large-r-programs&quot;&gt;&quot;How to organize large R programs?&quot;&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://stackoverflow.com/questions/2712421/r-and-version-control-for-the-solo-data-analyst&quot;&gt;&quot;R and version control for the solo data analyst&quot;&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://stackoverflow.com/questions/2295389/how-does-software-development-compare-with-statistical-programming-analysis&quot;&gt;&quot;How does software development compare with statistical programming/analysis ?&quot;&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://stackoverflow.com/questions/2286831/how-do-you-combine-revision-control-with-workflow-for-r&quot;&gt;&quot;How do you combine “Revision Control” with “WorkFlow” for R?&quot;&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You may also be interested in &lt;a href=&quot;http://www.johnmyleswhite.com/notebook/2010/09/19/why-use-projecttemplate-or-any-other-framework/&quot; rel=&quot;nofollow&quot;&gt;John Myles White's recent project&lt;/a&gt; to create a statistical project template. &lt;/p&gt;&#xA;" OwnerUserId="5" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-25T10:59:17.233" LastActivityDate="2010-09-25T10:59:17.233" CommunityOwnedDate="2010-09-20T20:42:21.877" CommentCount="5" />
  <row Id="2912" PostTypeId="2" ParentId="2892" CreationDate="2010-09-20T21:27:28.550" Score="1" ViewCount="" Body="&lt;p&gt;One way I have studied this problem in the past is to construct the 'eigenportfolios' of the correlation matrix. That is, take the eigenvector associated with the $k$th largest eigenvalue of the correlation matrix and scale it to a gross leverage of 1 (i.e. make the absolute sum of the vector equal to one). Then see if you can find any real physical or financial connection between the stocks which have large representation in the portfolio.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Usually the first eigenportfolio is almost equal weighted in every name, which is to say the 'market' portfolio consisting of all assets with equal dollar weights. The second eigenportfolio may have some semantical meaning, depending on which time period you look over: &lt;em&gt;e.g.&lt;/em&gt; mostly energy stocks, or bank stocks, etc. In my experience, you would be hard pressed to make any story out of the fifth eigenportfolio or beyond, and this depends in some part universe selection and the time period considered.  This is just fine because usually the fifth eigenvalue or so is not too far beyond the limits imposed by the Marchenko-Pastur distribution.&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-09-20T21:27:28.550" CommunityOwnedDate="2010-09-21T00:56:34.300" />
  <row Id="2913" PostTypeId="2" ParentId="2860" CreationDate="2010-09-20T21:29:31.903" Score="2" ViewCount="" Body="&lt;p&gt;What computational savings?  The PCA computation is based on the covariance (or correlation) matrix, whose size depends on the number of variables, not the number of data points.  The calculation of a covariance matrix is fast.  Even if you were doing PCA repeatedly (as part of a simulation, for instance), reducing from 1000 data points to 500 wouldn't even reduce the time by 50%.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-09-20T21:29:31.903" CommentCount="7" />
  <row Id="2914" PostTypeId="1" AcceptedAnswerId="2931" CreationDate="2010-09-20T21:35:58.207" Score="4" ViewCount="522" Body="&lt;p&gt;When you are the one doing the work, being aware of what you are doing you develop a sense of when you have over-fit the model.  For one thing, you can track the trend or deterioration in the Adjusted R Square of the model.  You can also track a similar deterioration in the p values of the regression coefficients of the main variables.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But, when you just read someone else study and you have no insight as to their own internal model development process how can you clearly detect if a model is over-fit or not.  &lt;/p&gt;&#xA;" OwnerUserId="1329" LastActivityDate="2011-04-26T01:38:14.393" Title="How to detect when a regression model is over-fit?" Tags="&lt;regression&gt;&lt;multivariate-analysis&gt;" AnswerCount="3" CommentCount="2" FavoriteCount="5" />
  <row Id="2915" PostTypeId="1" AcceptedAnswerId="2928" CreationDate="2010-09-20T22:16:05.190" Score="4" ViewCount="156" Body="&lt;p&gt;I am talking about the regression method that measures the impact of several layers of independent variables upon a dependent variable.&lt;/p&gt;&#xA;" OwnerUserId="1329" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-09-21T11:24:20.990" LastActivityDate="2010-09-21T11:24:20.990" Title="What is a good internet based source of information on Hierarchical Modeling?" Tags="&lt;modeling&gt;&lt;regression&gt;&lt;multilevel-analysis&gt;" AnswerCount="3" CommentCount="4" FavoriteCount="1" />
  <row Id="2916" PostTypeId="2" ParentId="2892" CreationDate="2010-09-20T23:05:48.593" Score="1" ViewCount="" Body="&lt;p&gt;Eigenvalues give magnitudes of principle components of data spread.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://yaroslavvb.com/upload/eigenvalues.png&quot;&gt;&#xA; First dataset was generated from Gaussian with covariance matrix $\left(\matrix{3&amp;amp;0\\0&amp;amp;1}\right)$ second dataset is the first dataset rotated by $\pi/4$&lt;/p&gt;&#xA;" OwnerUserId="511" LastActivityDate="2010-09-20T23:05:48.593" CommunityOwnedDate="2010-09-21T00:56:34.300" />
  <row Id="2917" PostTypeId="1" CreationDate="2010-09-20T23:14:27.380" Score="7" ViewCount="532" Body="&lt;h2&gt;Background&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I am conducting a meta-analysis that includes previously published data. Often, differences between treatments are reported with P-values, least significant differences (LSD), and other statistics but provide no direct estimate of the variance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the context of the model that I am using, an overestimate of variance is okay.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Problem&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Here is a list of transformations to $SE$ where $SE=\sqrt{MSE/n}$ &lt;a href=&quot;http://eskes.psychiatry.dal.ca/Files/2003_-_Saville_-_Basic_statistics_and_the_inconsistency_of_m.pdf&quot; rel=&quot;nofollow&quot;&gt;(Saville 2003)&lt;/a&gt; that I am considering, feedback appreciated; below, I assume that $\alpha=0.05$ so $1-^{\alpha}/_2=0.975$  and variables are normally distributed unless otherwise stated:&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Questions:&lt;/h2&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;given $P$, $n$, and treatment means $\bar X_1$ and $\bar X_2$ &#xA;$$SE=\frac{\bar X_1-\bar X_2}{t_{(1-\frac{P}{2},2n-2)}\sqrt{2/n}}$$&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;given LSD &lt;a href=&quot;http://apsjournals.apsnet.org/doi/abs/10.1094/PHYTO.2004.94.9.1013&quot; rel=&quot;nofollow&quot;&gt;(Rosenberg 2004)&lt;/a&gt;, $\alpha$, $n$, $b$ where $b$ is number of blocks, and $n=b$ by default for RCBD&#xA;$$SE = \frac{LSD}{t_{(0.975,n)}\sqrt{2bn}}$$&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;given MSD (minimum significant difference) &lt;a href=&quot;http://onlinelibrary.wiley.com/doi/10.1002/etc.5620190113/full&quot; rel=&quot;nofollow&quot;&gt;(Wang 2000)&lt;/a&gt;, $n$, $\alpha$, df = $2n-2$&#xA;$$SE = \frac{MSD}{t_{(0.975, 2n-2)}\sqrt{2}}$$&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;given a 95% Confidence Interval &lt;a href=&quot;http://eskes.psychiatry.dal.ca/Files/2003_-_Saville_-_Basic_statistics_and_the_inconsistency_of_m.pdf&quot; rel=&quot;nofollow&quot;&gt;(Saville 2003)&lt;/a&gt; (measured from mean to upper or lower confidence limit), $\alpha$, and $n$ $$SE = \frac{CI}{t_{(\alpha/2,n)}}$$&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;given Tukey's HSD, $n$, where $q$ is the 'studentized range statistic', $$SE = \frac{HSD}{q_{(0.975,n)}}$$&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;An R function to encapsulate these equations:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Example Data: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data &amp;lt;- data.frame(Y=rep(1,5), &#xA;                   stat=rep(1,5), &#xA;                   n=rep(4,5), &#xA;                   statname=c('SD', 'MSE', 'LSD', 'HSD', 'MSD') &#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Example Use:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;transformstats(data)    &#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The &lt;code&gt;transformstats&lt;/code&gt; function:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;transformstats &amp;lt;- function(data) {&#xA;  ## Transformation of stats to SE&#xA;  ## transform SD to SE&#xA;  if (&quot;SD&quot; %in% data$statname) {&#xA;    sdi &amp;lt;- which(data$statname == &quot;SD&quot;)&#xA;    data$stat[sdi] &amp;lt;- data$stat[sdi] / sqrt(data$n[sdi])&#xA;    data$statname[sdi] &amp;lt;- &quot;SE&quot;&#xA;      }&#xA;  ## transform MSE to SE&#xA;  if (&quot;MSE&quot; %in% data$statname) {&#xA;    msei &amp;lt;- which(data$statname == &quot;MSE&quot;)&#xA;    data$stat[msei] &amp;lt;- sqrt (data$stat[msei]/data$n[msei])&#xA;    data$statname[msei] &amp;lt;- &quot;SE&quot;&#xA;  }&#xA;  ## 95%CI measured from mean to upper or lower CI&#xA;  ## SE = CI/t&#xA;  if (&quot;95%CI&quot; %in% data$statname) {&#xA;    cii &amp;lt;- which(data$statname == '95%CI')&#xA;    data$stat[cii] &amp;lt;- data$stat[cii]/qt(0.975,data$n[cii])&#xA;    data$statname[cii] &amp;lt;- &quot;SE&quot;&#xA;  }&#xA;  ## Fisher's Least Significant Difference (LSD)&#xA;  ## conservatively assume no within block replication&#xA;  if (&quot;LSD&quot; %in% data$statname) {&#xA;    lsdi &amp;lt;- which(data$statname == &quot;LSD&quot;)&#xA;    data$stat[lsdi] &amp;lt;- data$stat[lsdi] / (qt(0.975,data$n[lsdi]) * sqrt( (2 * data$n[lsdi])))&#xA;    data$statname[lsdi] &amp;lt;- &quot;SE&quot;&#xA;  }&#xA;  ## Tukey's Honestly Significant Difference (HSD),&#xA;  ## conservatively assuming 3 groups being tested so df =2&#xA;  if (&quot;HSD&quot; %in% data$statname) {&#xA;    hsdi &amp;lt;- which(data$statname == &quot;HSD&quot; &amp;amp; data$n &amp;gt; 1)&#xA;    data$stat[hsdi] &amp;lt;- data$stat[hsdi] / (qtukey(0.975, data$n[lsdi], df = 2))&#xA;    data$statname[hsdi] &amp;lt;- &quot;SE&quot;&#xA;  }              &#xA;  ## MSD Minimum Squared Difference&#xA;  ## MSD = t_{\alpha/2, 2n-2}*SD*sqrt(2/n)&#xA;  ## SE  = MSD*n/(t*sqrt(2))&#xA;  if (&quot;MSD&quot; %in% data$statname) {&#xA;    msdi &amp;lt;- which(data$statname == &quot;MSD&quot;)&#xA;    data$stat[msdi] &amp;lt;- data$stat[msdi] * data$n[msdi] / (qt(0.975,2*data$n[lsdi]-2)*sqrt(2))&#xA;    data$statname[msdi] &amp;lt;- &quot;SE&quot;&#xA;  }&#xA;  if (FALSE %in% c('SE','none') %in% data$statname) {&#xA;    print(paste(trait, ': ERROR!!! data contains untransformed statistics'))&#xA;  }&#xA;  return(data)&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;References&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://eskes.psychiatry.dal.ca/Files/2003_-_Saville_-_Basic_statistics_and_the_inconsistency_of_m.pdf&quot; rel=&quot;nofollow&quot;&gt;Saville 2003Can J. Exptl Psych. (pdf)&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://apsjournals.apsnet.org/doi/abs/10.1094/PHYTO.2004.94.9.1013&quot; rel=&quot;nofollow&quot;&gt;Rosenberg et al 2004 (link)&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://onlinelibrary.wiley.com/doi/10.1002/etc.5620190113/full&quot; rel=&quot;nofollow&quot;&gt;Wang et al. 2000 Env. Tox. and Chem 19(1):113-117 (link)&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1381" LastEditorUserId="795" LastEditorDisplayName="" LastEditDate="2011-03-17T02:46:32.257" LastActivityDate="2011-03-17T02:47:49.370" CommunityOwnedDate="2010-09-22T17:02:36.783" Title="Are these formulas for transforming P, LSD, MSD, HSD, CI, to SE as an exact or inflated/conservative estimate of $\hat{\sigma}$ correct?" Tags="&lt;multiple-comparisons&gt;&lt;variance&gt;&lt;data-transformation&gt;&lt;meta-analysis&gt;" AnswerCount="3" CommentCount="2" FavoriteCount="6" />
  <row Id="2918" PostTypeId="1" CreationDate="2010-09-20T23:22:35.163" Score="3" ViewCount="373" Body="&lt;p&gt;Are Lorenz curves and QQ-plots the same? If not, where are the differences? I read about both of them and they appear to be two terms for the same type of plot / statistical technique to compare distributions. I was not able to find any confirmatory source for this. Perhaps you know?&lt;/p&gt;&#xA;" OwnerUserId="608" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-21T20:26:46.990" LastActivityDate="2011-04-21T20:26:46.990" Title="Is Lorenz curve the same as QQ-plot?" Tags="&lt;data-visualization&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="2919" PostTypeId="2" ParentId="2915" CreationDate="2010-09-20T23:37:22.590" Score="4" ViewCount="" Body="&lt;p&gt;I warmly recommend Doug Bate's &lt;a href=&quot;http://lme4.r-forge.r-project.org/book/&quot; rel=&quot;nofollow&quot;&gt;book&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="603" LastActivityDate="2010-09-20T23:37:22.590" CommentCount="2" />
  <row Id="2920" PostTypeId="2" ParentId="2917" CreationDate="2010-09-20T23:42:21.657" Score="6" ViewCount="" Body="&lt;p&gt;Your LSD equation looks fine.  If you want to get back to variance and you have a summary statistic that says something about variability or significance of an effect then you can almost always get back to variance—-you just need to know the formula.  For example, in your equation for LSD you want to solve for MSE, MSE = (LSD/t_)^2 / 2 * b&lt;/p&gt;&#xA;" OwnerUserId="601" LastActivityDate="2010-09-20T23:42:21.657" CommentCount="1" />
  <row Id="2921" PostTypeId="2" ParentId="2915" CreationDate="2010-09-20T23:52:04.230" Score="6" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://www.cmm.bristol.ac.uk/learning-training/index.shtml&quot; rel=&quot;nofollow&quot;&gt;The Centre for Multilevel Modelling&lt;/a&gt; has free online tutorials for multi-level modeling, and they have software tutorials for fitting models in both their MLwiN software and STATA.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You will probably want to check out all the questions with the &lt;a href=&quot;http://stats.stackexchange.com/questions/tagged/multilevel-analysis&quot;&gt;multilevel analysis&lt;/a&gt; tag here. You will find many other suggestions for books and resources.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also &lt;a href=&quot;http://www.cmm.bristol.ac.uk/team/HG_Personal/multbook1995.pdf&quot; rel=&quot;nofollow&quot;&gt;Harvey Goldstein&lt;/a&gt; has an online book, but I would suggest you check out the Centre for Multilevel Modelling first.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;good luck&lt;/p&gt;&#xA;" OwnerUserId="1036" LastEditorUserId="1036" LastEditorDisplayName="" LastEditDate="2010-09-21T02:52:02.937" LastActivityDate="2010-09-21T02:52:02.937" />
  <row Id="2922" PostTypeId="2" ParentId="2904" CreationDate="2010-09-21T00:04:45.017" Score="2" ViewCount="" Body="&lt;p&gt;Your likelihood function is non-concave (i.e. the Hessian matrix of your likelihood function is not SDN). From this it follows that&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;You will &lt;em&gt;only&lt;/em&gt; find a &lt;strong&gt;local&lt;/strong&gt; maximae to your likelihood function (no garantuee of global optimality)&lt;/li&gt;&#xA;&lt;li&gt;This maxima will &lt;em&gt;always&lt;/em&gt; depend on your choice of starting point.&lt;/li&gt;&#xA;&lt;li&gt;Your maximization procedure will &lt;em&gt;always&lt;/em&gt; be an iterative one.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Without directly solving the issues aboves, one way to handle them would be thru Monte carlo optimization (a review is &lt;a href=&quot;http://www.google.com/url?sa=t&amp;amp;source=web&amp;amp;cd=14&amp;amp;ved=0CCcQFjADOAo&amp;amp;url=https%3A%2F%2Foa.doria.fi%2Fbitstream%2Fhandle%2F10024%2F30051%2FTMP.objres.467.pdf%3Fsequence%3D1&amp;amp;rct=j&amp;amp;q=monte%20carlo%20optimization%20nonlinear%20models&amp;amp;ei=vgeYTIurAs6gOLK15NYP&amp;amp;usg=AFQjCNEwwWrMgja3vUNM43h4gT6oIKdoAQ&amp;amp;sig2=HPFupT9Xbnuf6RFr89CqWg&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;) which is basically a recasting of Rob's suggestion inside the frame of statistics.&lt;/p&gt;&#xA;" OwnerUserId="603" LastEditorUserId="603" LastEditorDisplayName="" LastEditDate="2010-09-22T22:33:24.393" LastActivityDate="2010-09-22T22:33:24.393" />
  <row Id="2924" PostTypeId="2" ParentId="2904" CreationDate="2010-09-21T00:42:04.910" Score="3" ViewCount="" Body="&lt;p&gt;It sounds like you need to use a more robust optimization algorithm that can handle local minima. Particle swarm methods work quite well in this case. Or you could try other evolutionary optimization methods or simulated annealing.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-09-21T00:42:04.910" CommentCount="0" />
  <row Id="2925" PostTypeId="1" AcceptedAnswerId="2963" CreationDate="2010-09-21T00:47:15.803" Score="2" ViewCount="231" Body="&lt;p&gt;Suppose that we want to generate a draw from the following distribution:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(X=0) = 0.5$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(X=1) = 0.5$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are two constraints though:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(a) The draw has to be on the basis of an external event.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(b) Related to (a), the draw must be verifiable by a third party. In other words, a third party should be able to verify that my draw was in fact $X = 0$ (say).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Qn 1: Can such a system be devised and if so how?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Qn 2: Can the system be extended to discrete variables with more than 2 possible outcomes (like the roll of a dice)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Qn 3: Similarly, can it be extended to continuous variables (e.g., the normal)?&lt;/p&gt;&#xA;" LastActivityDate="2010-09-23T02:54:38.517" Title="Is there a verifiable way to generate discrete random variables on the basis of an external event?" Tags="&lt;random-variable&gt;" AnswerCount="6" CommentCount="2" FavoriteCount="2" />
  <row Id="2926" PostTypeId="2" ParentId="305" CreationDate="2010-09-21T01:36:24.877" Score="1" ViewCount="" Body="&lt;p&gt;Two reasons I can think of:  &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Regular Student's T is pretty robust to heteroscedasticity if the sample sizes are equal.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If you believe strongly &lt;em&gt;a priori&lt;/em&gt; that the data is homoscedastic, then you lose nothing and might gain a small amount of power by using Studen'ts T instead of Welch's T.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;One reason that I would &lt;strong&gt;not&lt;/strong&gt; give is that Student's T is exact and Welch's T isn't.  IMHO the exactness of Student's T is academic because it's only exact for normally distributed data, and no real data is &lt;strong&gt;exactly&lt;/strong&gt; normally distributed.  I can't think of a single quantity that people actually measure and analyze statistically where the distribution could plausibly have a support of all real numbers.  For example, there are only so many atoms in the universe, and some quantities can't be negative.  Therefore, when you use any kind of T-test on real data, you're making an approximation anyhow.&lt;/p&gt;&#xA;" OwnerUserId="1347" LastActivityDate="2010-09-21T01:36:24.877" />
  <row Id="2927" PostTypeId="2" ParentId="2914" CreationDate="2010-09-21T02:14:36.227" Score="4" ViewCount="" Body="&lt;p&gt;When I'm fitting a model myself I generally use information criteria during the fitting process, such as &lt;a href=&quot;http://en.wikipedia.org/wiki/Akaike_information_criterion&quot; rel=&quot;nofollow&quot;&gt;AIC&lt;/a&gt; or &lt;a href=&quot;http://en.wikipedia.org/wiki/Bayesian_information_criterion&quot; rel=&quot;nofollow&quot;&gt;BIC&lt;/a&gt;, or alternatively &lt;a href=&quot;http://en.wikipedia.org/wiki/Likelihood_ratio_test&quot; rel=&quot;nofollow&quot;&gt;Likelihood-ratio tests&lt;/a&gt; for models fit based on maximum likelihood or &lt;a href=&quot;http://en.wikipedia.org/wiki/F-test#Regression_problems&quot; rel=&quot;nofollow&quot;&gt;F-test&lt;/a&gt; for models fit based on least squares.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All are conceptually similar in that they penalise additional parameters. They set a threshold of &quot;additional explanatory power&quot; for each new parameter added to a model. They are all a form of &lt;a href=&quot;http://en.wikipedia.org/wiki/Regularization_%28mathematics%29&quot; rel=&quot;nofollow&quot;&gt;regularisation&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For others' models I look at the methods section to see if such techniques are used and also use rules of thumb, such as the number of observations per parameter - if there are around 5 (or fewer) observations per parameter I start to wonder.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Always remember that a variable need need not be &quot;significant&quot; in a model to be important. I may be a &lt;a href=&quot;http://en.wikipedia.org/wiki/Confounding&quot; rel=&quot;nofollow&quot;&gt;confounder&lt;/a&gt; and should be included on that basis if your goal is to estimate the effect of other variables. &lt;/p&gt;&#xA;" OwnerUserId="521" LastActivityDate="2010-09-21T02:14:36.227" CommentCount="4" />
  <row Id="2928" PostTypeId="2" ParentId="2915" CreationDate="2010-09-21T02:25:20.493" Score="4" ViewCount="" Body="&lt;p&gt;UCLA has some good resources:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.ats.ucla.edu/stat/papers/mlmpapers.htm&quot; rel=&quot;nofollow&quot;&gt;Papers on multilevel modelling&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.ats.ucla.edu/stat/examples/&quot; rel=&quot;nofollow&quot;&gt;Textbook examples&lt;/a&gt; (see multilevel modelling)&lt;/li&gt;&#xA;&lt;li&gt;A free textbook on multilevel modelling by &lt;a href=&quot;http://www.ats.ucla.edu/stat/examples/msm_goldstein/&quot; rel=&quot;nofollow&quot;&gt;Harvey Goldstein&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;and more...&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-09-21T02:25:20.493" />
  <row Id="2929" PostTypeId="2" ParentId="2918" CreationDate="2010-09-21T02:46:15.140" Score="7" ViewCount="" Body="&lt;p&gt;The Lorenz curve is just a cumulative distribution function for a random variable bounded between 0 and 1, e.g., a proportion. In economics, the Lorenz curve asks, &quot;what fraction of income is earned by the lowest x% of earners?&quot; Typically, it is compared to the uniform distribution over [0,1], a distribution that would arise under perfect equality in income. The Gini coefficient is the area under the perfect equality curve less the area under the Lorenz curve, normalized by the area under the perfect equality curve; note that the area under the perfect equality curve is equal to 0.5.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, to be clear, while a QQ plot compares two distributions to one another, the Lorenz curve considers only one distribution that has a range over [0,1].&lt;/p&gt;&#xA;" OwnerUserId="401" LastActivityDate="2010-09-21T02:46:15.140" CommentCount="10" />
  <row Id="2930" PostTypeId="2" ParentId="2910" CreationDate="2010-09-21T03:00:47.613" Score="3" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0470144483&quot; rel=&quot;nofollow&quot;&gt;van Belle&lt;/a&gt; is &lt;em&gt;the&lt;/em&gt; source for the rules of successful statistical projects.&lt;/p&gt;&#xA;" OwnerUserId="666" LastActivityDate="2010-09-21T03:00:47.613" CommunityOwnedDate="2010-09-21T03:00:47.613" />
  <row Id="2931" PostTypeId="2" ParentId="2914" CreationDate="2010-09-21T03:32:23.540" Score="7" ViewCount="" Body="&lt;p&gt;Cross validation and regularization are fairly common techniques to prevent overfitting.  For a quick take, I'd recommend Andrew Moore's tutorial slides on the use of &lt;a href=&quot;http://www.autonlab.org/tutorials/overfit.html&quot; rel=&quot;nofollow&quot;&gt;cross-validation&lt;/a&gt; -- pay particular attention to the caveats.  For more detail, definitely read chapters 3 and 7 of &lt;a href=&quot;http://www-stat.stanford.edu/~tibs/ElemStatLearn/&quot; rel=&quot;nofollow&quot;&gt;EOSL&lt;/a&gt;, which cover the topic and associated matter in good depth.&lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-09-21T03:32:23.540" CommentCount="1" />
  <row Id="2932" PostTypeId="1" CreationDate="2010-09-21T03:38:50.050" Score="3" ViewCount="107" Body="&lt;p&gt;Does the use of metric spaces to describe the support of a random variable provide any greater illumination? I ask this after reading about how metrics spaces have been used to unify the mathematical measure theoretic nature of probability and the physical intuition that most associate with probability. You can read my inspiration here: &lt;a href=&quot;http://www.arsmathematica.net/archives/2009/02/14/complete-metric-spaces-and-the-interpretation-of-probability/&quot; rel=&quot;nofollow&quot;&gt;http://www.arsmathematica.net/archives/2009/02/14/complete-metric-spaces-and-the-interpretation-of-probability/&lt;/a&gt;&lt;/p&gt;&#xA;" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-24T14:00:05.347" LastActivityDate="2010-09-24T14:00:05.347" Title="Metric spaces and the support of a random variable" Tags="&lt;random-variable&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="2934" PostTypeId="1" CreationDate="2010-09-21T04:08:56.927" Score="2" ViewCount="293" Body="&lt;p&gt;I have distributional data which I represent as a density. The data represents frequencies of user activities on a computer screen (e.g. amount of clicks on the y or x-axis of that screen but also other activities that can be related to coordinates and can therefore be binned by those coordinates (e.g. 5 pixels bins)). I would like to compare two kinds of that behavior and find out how compatible their distributions are. Very general. No assumptions exist. I can't assume parametric conditions such as linearity or normality. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I read about Lorenz curves and the Gini coefficient to be very much like what I need to compare distributions but also know that those methods find application primarily for economic and sociological problems and are usually not applied for general distributions. Am I applying the wrong tool for the job? What is your opinion about this? What alternatives do you recommend in order to find out how similar two distributions are?&lt;/p&gt;&#xA;" OwnerUserId="608" LastActivityDate="2010-09-21T04:57:20.413" Title="Using Lorenz curve / Gini coefficient for (non-ecomoical) distribution data" Tags="&lt;distributions&gt;" AnswerCount="1" />
  <row Id="2935" PostTypeId="2" ParentId="1735" CreationDate="2010-09-21T04:29:35.863" Score="1" ViewCount="" Body="&lt;p&gt;Here is my suggestion.  Rerun your model(s) using one single regression.  And, the Summer/Winter variable would be simply a single dummy variable (1,0).  This way you would have a coefficient for Summer to differentiate it from Winter.  And, the regression coefficients for your three other variables would be consistent with one single weight rank.   &lt;/p&gt;&#xA;" OwnerUserId="1329" LastActivityDate="2010-09-21T04:29:35.863" CommentCount="3" />
  <row Id="2936" PostTypeId="2" ParentId="2925" CreationDate="2010-09-21T04:48:37.157" Score="3" ViewCount="" Body="&lt;p&gt;This reminds me of a question from Algorithms class a long time ago. Let the external event be a (preferrably continuous) random variable $Y$. To generate a value of $X$, take two independent observations of $Y$ and let $X$ be $1$ if the first observation of $Y$ is greater than the second, let it be $0$ if the second is greater than the first, and repeat the experiment if there is a tie.  Obviously, this works better if $Y$ is continuous. If only a coinflip is available as a generator, one can let the number of consecutive heads before seeing a tail be the random variable $Y$. (The homework question, I believe, was how to turn a biased coinflip into an unbiased coinflip. One part of the homework was proving that the process would terminate...)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This obviously can be extended to the discrete case as well, although one may run into more difficulties with ties. If you have $n$ possible outcomes for $X$, take some $k$ such that $n$ divides $k!$ and then partition the $k!$ possible orderings of $k$ observations from $Y$ into equivalence classes for $X$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;edit:&lt;/strong&gt; per @srikant's comment, an example of a possible generator of $Y$: (as anticipated by @andyW) Let $Z_i$ be the number of shares traded on a given highly-liquid ETF as reported by a given source over a fixed time period unambiguously indexed by $i$. Let $Y_i = \tan{(Z_i)},$ where the tangent function is computed by a fixed standard library (in a fixed revision of R, say, on a fixed platform.)  Such a generator of $Y$ is pseudorandom enough for me.  Other generators of $Z$ are also amenable to this process if they vary widely enough with respect to $2\pi$.&lt;/p&gt;&#xA;" OwnerUserId="795" LastEditorUserId="795" LastEditorDisplayName="" LastEditDate="2010-09-21T16:21:35.873" LastActivityDate="2010-09-21T16:21:35.873" CommentCount="2" />
  <row Id="2937" PostTypeId="2" ParentId="2934" CreationDate="2010-09-21T04:57:20.413" Score="2" ViewCount="" Body="&lt;p&gt;You can use a &lt;a href=&quot;http://en.wikipedia.org/wiki/Kolmogorov_Smirnov_Test#Two-sample_Kolmogorov.E2.80.93Smirnov_test&quot; rel=&quot;nofollow&quot;&gt;2-sample Kolmogorov-Smirnov test&lt;/a&gt; to compare the two distributions. Other tests for comparing 2-samples are the Anderson-Darling test (although the 2-sample form of this is less frequently used), and the Baumgartner-Weiss-Schindler test.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Before you jump into hypothesis testing, though, you may want to graphically inspect the two distributions, either by overlaying their empirical CDF, or better by an empirical &lt;a href=&quot;http://en.wikipedia.org/wiki/Q-Q_plot&quot; rel=&quot;nofollow&quot;&gt;Q-Q plot.&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-09-21T04:57:20.413" CommentCount="5" />
  <row Id="2938" PostTypeId="1" CreationDate="2010-09-21T05:15:16.743" Score="4" ViewCount="305" Body="&lt;p&gt;The qq-plot can be used to visualize how similar two distributions are (e.g. visualizing the similarity of a distribution to a normal distribution, but also to compare two artibrary data distributions). Are there any statistics that generate a more objective, numerical measure that represent their similarity (preferably in a normalized (0 &amp;lt;= x &amp;lt;= 1) form)? The Gini coefficient is for example used in economics when working with Lorenz curves; is there something for QQ-plots?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Best,&#xA;A.&lt;/p&gt;&#xA;" OwnerUserId="608" LastActivityDate="2010-09-21T15:15:12.333" Title="Quantifying QQ plot" Tags="&lt;distributions&gt;" AnswerCount="3" FavoriteCount="1" />
  <row Id="2939" PostTypeId="2" ParentId="2938" CreationDate="2010-09-21T05:35:11.600" Score="3" ViewCount="" Body="&lt;p&gt;As I say in response to your comment on your previous question, check out the Kolmogorov-Smirnov test. It uses the maximum absolute distance between two cumulative distribution functions (alternatively conceived as the maximum absolute distance of the curve in the QQ plot from the 45-degree line) as a statistic. The KS test can be found in R using the command &lt;code&gt;ks.test()&lt;/code&gt; in the 'stats' library. &lt;a href=&quot;http://sekhon.berkeley.edu/stats/html/ks.test.html&quot; rel=&quot;nofollow&quot;&gt;Here's more information about its R usage.&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="401" LastActivityDate="2010-09-21T05:35:11.600" CommentCount="4" />
  <row Id="2940" PostTypeId="2" ParentId="2925" CreationDate="2010-09-21T06:56:27.817" Score="0" ViewCount="" Body="&lt;p&gt;From the question, I gather you want a random number generated by the physical world. You could go with an mp3 mic in a subway/street station, sample 1 hour's worth of sound there (read noise). Do a random re-ordering of it. Convert it to frequency data and use events such as $freq_i&gt;quantile(freq,0.5)$-&gt;$X_i=1$. &lt;/p&gt;&#xA;" OwnerUserId="603" LastEditorUserId="603" LastEditorDisplayName="" LastEditDate="2010-09-21T14:53:31.260" LastActivityDate="2010-09-21T14:53:31.260" CommentCount="2" />
  <row Id="2941" PostTypeId="2" ParentId="2938" CreationDate="2010-09-21T07:46:52.397" Score="0" ViewCount="" Body="&lt;p&gt;I would say that the more or less canonical way to compare two distributions would be a chi-squared test. The statistic is not normalized, though, and it depends on how you choose the bins. The last point can of course be seen as a feature, not a bug: choosing bins appropriately allows you to look more closely for similarity in the tails than in the middle of the distributions, for instance.&lt;/p&gt;&#xA;" OwnerUserId="1352" LastActivityDate="2010-09-21T07:46:52.397" />
  <row Id="2942" PostTypeId="2" ParentId="2917" CreationDate="2010-09-21T08:55:35.920" Score="5" ViewCount="" Body="&lt;p&gt;I can only agree with John. Furthermore, perhaps &lt;a href=&quot;http://eskes.psychiatry.dal.ca/Files/2003_-_Saville_-_Basic_statistics_and_the_inconsistency_of_m.pdf&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; paper by David Saville helps you with some formula to recalculate variability measures from LSDs et al.:&lt;br&gt;&#xA;Saville D.J. (2003). &lt;a href=&quot;http://eskes.psychiatry.dal.ca/Files/2003_-_Saville_-_Basic_statistics_and_the_inconsistency_of_m.pdf&quot; rel=&quot;nofollow&quot;&gt;Basic statistics and the inconsistency of multiple comparison procedures.&lt;/a&gt; Canadian Journal of Experimental Psychology, 57, 167–175&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE:&lt;/strong&gt;&lt;br&gt;&#xA;If you are looking for more formulas to convert between various effect sizes, books on meta-analysis should provide a lot of these. However, I am not an expert in this area and can't recommend one.&lt;br&gt;&#xA;But, I remember that the book by Rosenthal and Rosnow once helped with some formula:&lt;br&gt;&#xA;&lt;strong&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0073531960&quot; rel=&quot;nofollow&quot;&gt;Essentials of Behavioral Research: Methods and Data Analysis&lt;/a&gt;&lt;/strong&gt;&lt;br&gt;&#xA;Furthermore, I have heard a lot of good things about the formulas in this book by Rosenthal, Rosnow &amp;amp; Rubin (although I have never used it):&lt;br&gt;&#xA;&lt;strong&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0521659809&quot; rel=&quot;nofollow&quot;&gt;Contrasts and Effect Sizes in Behavioral Research: A Correlational Approach&lt;/a&gt;&lt;/strong&gt; (You should definitely give it a try if a nearby library has it).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If this is not enough, perhaps ask another question on literature for converting effect sizes for meta-analyses. Perhaps someone more into meta-analysis has more grounded recommendations.&lt;/p&gt;&#xA;" OwnerUserId="442" LastEditorUserId="442" LastEditorDisplayName="" LastEditDate="2010-09-23T16:56:50.490" LastActivityDate="2010-09-23T16:56:50.490" CommentCount="1" />
  <row Id="2944" PostTypeId="2" ParentId="2860" CreationDate="2010-09-21T11:45:56.080" Score="3" ViewCount="" Body="&lt;p&gt;Following the comments exchange with Ebony (see Whuber's answer). I gather that in Ebony's application, $p$ is much larger than $n$ which is itself very large. In this case the complexity of computing the eigen decomposition is in the order of $O(n^3)$. Two solutions spring to mind:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;partial decomposition&lt;/strong&gt;: assuming $p$ is very large, it could be the case that the full eigen-decomposition is not needed. If only the $k$ largest eigen values (and corresponding vectors) are needed, presumably they could be obtained with complexity near $O(nk^2)$. Would such an algorithm be a solution to your problem ? &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Full decomposition&lt;/strong&gt;: in this case it may be better to draw $J$ random sub-samples of your observations of size $n_0$ suitably smaller than $n$ and compute $J$ pca decompositions. That would in turn give you $J$ values of each eigen values/vector which could be used to establish the sampling distribution of there population values (and there means would be a good estimator of the population eigen values/vectors). Given the $n^3$ complexity, this could be made to be much faster (by appropriately choosing $n_0$). A second benefit is that this procedure can be run in parallele across $m$ cores/computers yielding an overall complexity of $O(jm^{-1}n_0^3)$&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="603" LastEditorUserId="603" LastEditorDisplayName="" LastEditDate="2010-09-22T00:35:28.130" LastActivityDate="2010-09-22T00:35:28.130" CommentCount="3" />
  <row Id="2945" PostTypeId="2" ParentId="2828" CreationDate="2010-09-21T12:08:33.850" Score="1" ViewCount="" Body="&lt;p&gt;From Yaroslav's comments to Henrik's answer:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;but cross-validation seems to just postpone the task of assessing complexity. If you use data to pick your parameters and your model as in cross-validation, the relevant question becomes how estimate the amount of data needed for this &quot;meta&quot;-fitter to perform well&lt;/em&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wonder whether this is not in itself informative. You perform several $k$-fold CV with varying $k$ (say along a grid) and look which model performs better as $k$ increases. More specifically: I wonder whether any differentiation among the two model in there $CV(k)$ performance as a function of $k$ can be taken as evidence that this model (the one whose relative performance decreases less when $k$ increases) would be the less complex one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could even give a 'significance' flavor to this since the result of the procedure is directly in terms (units) of difference in out of sample forecasting error.&lt;/p&gt;&#xA;" OwnerUserId="603" LastEditorUserId="603" LastEditorDisplayName="" LastEditDate="2010-09-21T12:35:49.923" LastActivityDate="2010-09-21T12:35:49.923" CommentCount="9" />
  <row Id="2946" PostTypeId="2" ParentId="2932" CreationDate="2010-09-21T14:28:05.750" Score="2" ViewCount="" Body="&lt;p&gt;Interesting reference.  Its value for me lies in questioning the ability of measure theoretic probability to capture an &quot;intuition&quot; about probability (whatever that might mean) and going on to propose an intriguing distinction; namely, between a set of measure zero having a measure zero neighborhood and a set of measure zero all of whose proper neighborhoods have positive measure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is not apparent that separable metric spaces are the &quot;right&quot; way to capture this idea, though, as the comment by Matt Heath points out.  It sounds like we only need a predefined subcollection of measurable sets (not necessarily even satisfying the axioms of a topology).  Such a collection is conveniently obtained in a separable metric space but there are other ways to create such collections, too.  Thus is appears that the &lt;em&gt;idea&lt;/em&gt; presented here illuminates the connection between abstract measure theory and using random variables in models, but the use of metric spaces may be a bit of a red herring.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-09-21T14:28:05.750" />
  <row Id="2947" PostTypeId="2" ParentId="2938" CreationDate="2010-09-21T15:15:12.333" Score="0" ViewCount="" Body="&lt;p&gt;I &lt;a href=&quot;http://www.thinkingaboutthinking.org/wp-content/uploads/2010/05/Lawrence_BRM_in_press.pdf&quot; rel=&quot;nofollow&quot;&gt;recently&lt;/a&gt; used the correlation between the empirical CDF and the fitted CDF to quantify goodness-of-fit, and I wonder if this approach might also be useful in the current case, which as I understand it involves comparing two empirical data sets. Interpolation might be necessary if there are different numbers of observations between the sets.&lt;/p&gt;&#xA;" OwnerUserId="364" LastActivityDate="2010-09-21T15:15:12.333" CommentCount="5" />
  <row Id="2948" PostTypeId="1" AcceptedAnswerId="2951" CreationDate="2010-09-21T15:50:45.697" Score="9" ViewCount="841" Body="&lt;p&gt;Hello everyone,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm wondering if someone could suggest what are good starting points when it comes to performing community detection/graph partitioning/clustering on a graph that has &lt;strong&gt;weighted&lt;/strong&gt;, &lt;strong&gt;undirected&lt;/strong&gt; edges. The graph in question has approximately 3 million edges and each edge expresses the degree of similarity between the two vertices it connects. In particular, in this dataset edges are individuals and vertices are a measure of the similarity of their observed behavior.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the past I followed a suggestion I got here on stats.stackexchange.com and used igraph's implementation of Newman's modularity clustering and was satisfied with the results, but that was on a unweighted dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any specific algorithms I should be looking at?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks for any advice&#xA;~l&lt;/p&gt;&#xA;" OwnerUserId="1007" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-21T18:21:56.477" LastActivityDate="2011-02-03T19:17:14.433" Title="How to do community detection in a weighted social network/graph?" Tags="&lt;clustering&gt;&lt;networks&gt;&lt;data-visualization&gt;&lt;partitioning&gt;" AnswerCount="5" FavoriteCount="2" />
  <row Id="2950" PostTypeId="1" CreationDate="2010-09-21T17:19:53.943" Score="2" ViewCount="445" Body="&lt;p&gt;I am trying to determine significant differences between groups of data using a k-nearest neighbor randomization test in R. This test basically looks for similarity amongst homogenous groups and separates them out using random clustering. In the literature, this test is called a &quot;K-nearest neighbor (kNN) randomization test,&quot; however, I'm not certain if it called by other names elsewhere.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For my specific data, I have isotopic ratios given for various prey items. I have already grouped together ecologically similar prey item types, and now want to see if those groups differ from one another in their isotopic signature. This is where the kNN test would come in. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for all of your answers - I'm new to this site, so I'll address your inquiries as applicable in the comments section.&lt;/p&gt;&#xA;" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-01-18T00:37:31.553" LastActivityDate="2011-01-24T15:19:25.240" Title="kNN randomization test in R?" Tags="&lt;r&gt;&lt;k-nearest-neighbour&gt;" AnswerCount="4" CommentCount="5" FavoriteCount="1" />
  <row Id="2951" PostTypeId="2" ParentId="2948" CreationDate="2010-09-21T17:20:48.360" Score="6" ViewCount="" Body="&lt;p&gt;igraph  implementation of Newman's modularity clustering (fastgreedy function) can be used with weighted edges as well. Just add weight attribute to the edges and analyse as usual. In my experience, it run even faster with weights as there are less ties.&lt;/p&gt;&#xA;" OwnerUserId="1387" LastActivityDate="2010-09-21T17:20:48.360" CommentCount="1" />
  <row Id="2952" PostTypeId="2" ParentId="2948" CreationDate="2010-09-21T17:45:48.903" Score="2" ViewCount="" Body="&lt;p&gt;I know that &lt;a href=&quot;http://gephi.org/&quot; rel=&quot;nofollow&quot;&gt;Gephi&lt;/a&gt; can process undirected weighted graph, but I seem to remember it has to be stored in &lt;a href=&quot;http://gephi.org/users/supported-graph-formats/gdf-format/&quot; rel=&quot;nofollow&quot;&gt;GDF&lt;/a&gt;, which is pretty close to CSV, or Ucinet &lt;a href=&quot;http://gephi.org/users/supported-graph-formats/ucinet-dl-format/&quot; rel=&quot;nofollow&quot;&gt;DL&lt;/a&gt;. Be aware that it's still an alpha release.&#xA;Now, about clustering your graph, Gephi seems to lack clustering pipelines, except for the MCL algorithm that is now available in the latest version. There was a &lt;a href=&quot;http://code.google.com/p/google-summer-of-code-2009-gephi/&quot; rel=&quot;nofollow&quot;&gt;Google Code Project&lt;/a&gt; in 2009, &lt;a href=&quot;http://web.ecs.syr.edu/~pjmcswee/gephi.pdf&quot; rel=&quot;nofollow&quot;&gt;Gephi Network Statistics&lt;/a&gt; (featuring e.g. Newman’s modularity metric), but I don't know if something has been released in this direction. Anyway, it seems to allow some kind of modularity/clustering computations, but see also &lt;a href=&quot;http://www.rcasts.com/2010/04/social-network-analysis-using-r-and.html&quot; rel=&quot;nofollow&quot;&gt;Social Network Analysis using R and Gephi&lt;/a&gt; and &lt;a href=&quot;http://www.r-bloggers.com/data-preparation-for-social-network-analysis-using-r-and-gephi/&quot; rel=&quot;nofollow&quot;&gt;Data preparation for Social Network Analysis using R and Gephi&lt;/a&gt; (Many thanks to @Tal).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are used to Python, it is worth trying &lt;a href=&quot;http://networkx.lanl.gov/&quot; rel=&quot;nofollow&quot;&gt;NetworkX&lt;/a&gt; (Here is an example of a &lt;a href=&quot;http://networkx.lanl.gov/examples/drawing/weighted_graph.html&quot; rel=&quot;nofollow&quot;&gt;weighted graph&lt;/a&gt; with the corresponding code). Then you have many ways to carry out your analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should also look at &lt;a href=&quot;http://www.insna.org/software/index.html&quot; rel=&quot;nofollow&quot;&gt;INSNA - Social Network Analysis Software&lt;/a&gt; or Tim Evans's webpage about &lt;a href=&quot;http://155.198.210.128/~time/networks/&quot; rel=&quot;nofollow&quot;&gt;Complex Networks and Complexity&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-21T21:07:25.300" LastActivityDate="2010-09-21T21:07:25.300" CommentCount="3" />
  <row Id="2953" PostTypeId="2" ParentId="2950" CreationDate="2010-09-21T18:13:57.677" Score="3" ViewCount="" Body="&lt;p&gt;kmeans() (you can find it by typing &lt;code&gt;??&lt;/code&gt; followed by the name of what you want as in &lt;code&gt;??kmean&lt;/code&gt;). In general, a good trick is to first look it up on a R specific search &lt;a href=&quot;http://rseek.org/&quot; rel=&quot;nofollow&quot;&gt;engine&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="603" LastEditorUserId="603" LastEditorDisplayName="" LastEditDate="2010-09-21T18:44:13.107" LastActivityDate="2010-09-21T18:44:13.107" CommentCount="2" />
  <row Id="2954" PostTypeId="2" ParentId="2950" CreationDate="2010-09-21T21:34:04.403" Score="1" ViewCount="" Body="&lt;p&gt;You appear to have confused &quot;cluster analysis&quot; with &quot;classification&quot;. The former is where you don't know the groupings and wish to determine them from the training data to hand. Classification is where you know the groups and want to predict them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are a few packages in R that do this. For example, look at the results of this &lt;a href=&quot;http://ttp://search.r-project.org/cgi-bin/namazu.cgi?query=knn&amp;amp;max=100&amp;amp;result=normal&amp;amp;sort=score&amp;amp;idxname=functions&amp;amp;idxname=vignettes&amp;amp;idxname=views&quot; rel=&quot;nofollow&quot;&gt;R Site Search&lt;/a&gt; for suitable packages.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively, perhaps you are looking for a multivariate analysis of variance? In which case &lt;code&gt;lm()&lt;/code&gt; and &lt;code&gt;aov()&lt;/code&gt; would be worth looking at for a start. This presumes you want to &quot;model&quot; your data as a function of the group variable?&lt;/p&gt;&#xA;" OwnerUserId="1390" LastEditorUserId="1390" LastEditorDisplayName="" LastEditDate="2010-09-21T22:20:21.663" LastActivityDate="2010-09-21T22:20:21.663" CommentCount="3" />
  <row Id="2955" PostTypeId="2" ParentId="2950" CreationDate="2010-09-21T21:45:12.007" Score="2" ViewCount="" Body="&lt;p&gt;I am not sure to understand your question since you talk about k-means, which is basically an &lt;em&gt;unsupervised method&lt;/em&gt; (i.e. where classes are not known a priori), while at the same time you are saying that you already identified groups of individuals. So I would suggest to look at classification methods, or other &lt;em&gt;supervised methods&lt;/em&gt; where class membership is known and the objective is to find a weighted combination of your variables that minimize your classification error rate (this is just an example). For instance, &lt;a href=&quot;http://en.wikipedia.org/wiki/Linear_discriminant_analysis&quot; rel=&quot;nofollow&quot;&gt;LDA&lt;/a&gt; does a good job (see the CRAN task view on &lt;a href=&quot;http://cran.r-project.org/web/views/Multivariate.html&quot; rel=&quot;nofollow&quot;&gt;Multivariate Statistics&lt;/a&gt;), but look also at the machine learning community (widely represented on the stats.stackexchange) for other methods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now since you also talked of k-nearest neighbor, I wonder if you are not introducing a confusion between k-means and kNN. In this case, the corresponding R function is &lt;code&gt;knn()&lt;/code&gt; in the &lt;a href=&quot;http://cran.r-project.org/web/packages/class/index.html&quot; rel=&quot;nofollow&quot;&gt;class&lt;/a&gt; package (it includes cross-validation), or see the &lt;a href=&quot;http://cran.r-project.org/web/packages/kknn/index.html&quot; rel=&quot;nofollow&quot;&gt;kknn&lt;/a&gt; package.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-21T22:05:16.047" LastActivityDate="2010-09-21T22:05:16.047" />
  <row Id="2956" PostTypeId="1" CreationDate="2010-09-21T23:40:39.317" Score="15" ViewCount="299" Body="&lt;p&gt;Hi folks,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm the author of the &lt;a href=&quot;http://cran.r-project.org/package=ez&quot; rel=&quot;nofollow&quot;&gt;ez package&lt;/a&gt; for R, and I'm working on an update to include automatic computation of likelihood ratios (LRs) in the output of ANOVAs. The idea is to provide a LR for each effect that is analogous to the test of that effect that the ANOVA achieves. For example, the LR for main effect represents the comparison of a null model to a model that includes the main effect, the LR for an interaction represents the comparison of a model that includes both component main effects versus a model that includes both main effects &lt;em&gt;and&lt;/em&gt; their interaction, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, my understanding of LR computation comes from &lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pubmed/15732688&quot; rel=&quot;nofollow&quot;&gt;Glover &amp;amp; Dixon&lt;/a&gt; (&lt;a href=&quot;http://www.thinkingaboutthinking.org/wp-content/uploads/2010/09/Glover-Dixon-2004-Likelihood-ratios-a-simple-and-flexible-statistic-for-empirical-psychologists.pdf&quot; rel=&quot;nofollow&quot;&gt;PDF&lt;/a&gt;), which covers basic computations as well as corrections for complexity, and the appendix to &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0521009138&quot; rel=&quot;nofollow&quot;&gt;Bortolussi &amp;amp; Dixon&lt;/a&gt; (&lt;a href=&quot;http://www.thinkingaboutthinking.org/wp-content/uploads/2010/09/Dixon-Unknown-Evaluating-Evidence-Appendix-to-Psychonarratology.pdf&quot; rel=&quot;nofollow&quot;&gt;appendix PDF&lt;/a&gt;), which covers computations involving repeated-measures variables. To test my understanding, I developed &lt;a href=&quot;https://spreadsheets.google.com/ccc?key=0Ap2N_aeyRMGHdHJxUnVNeEl5VGtvY1RVLVc5UjU4Vmc&amp;amp;hl=en&quot; rel=&quot;nofollow&quot;&gt;this spreadsheet&lt;/a&gt;, which takes the dfs &amp;amp; SSs from an example ANOVA (generated from a 2*2*3*4 design using fake data) and steps through the computation of the LR for each effect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would really appreciate it if someone with a little more confidence with such computation could take a look and make sure I did everything correctly. For those that prefer abstract code, &lt;a href=&quot;http://gist.github.com/590789&quot; rel=&quot;nofollow&quot;&gt;here is the R code&lt;/a&gt; implementing the update to ezANOVA() (see esp. lines 15-95).&lt;/p&gt;&#xA;" OwnerUserId="364" LastEditorUserId="364" LastEditorDisplayName="" LastEditDate="2010-09-22T00:55:14.627" LastActivityDate="2010-10-04T13:04:25.893" CommunityOwnedDate="2010-09-21T23:40:39.317" Title="Have I computed these likelihood ratios correctly?" Tags="&lt;r&gt;&lt;anova&gt;&lt;code&gt;&lt;likelihood-ratio&gt;" AnswerCount="1" FavoriteCount="3" />
  <row Id="2957" PostTypeId="1" AcceptedAnswerId="7716" CreationDate="2010-09-22T01:15:49.873" Score="7" ViewCount="513" Body="&lt;p&gt;The Gauss-Markov theorem tells us that the OLS estimator is the best linear unbiased estimator for the linear regression model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But suppose I don't care about linearity and unbiasedness. Then is there some other (possible nonlinear/biased) estimator for the linear regression model which is the most efficient under the Gauss-Markov assumptions or some other general set of assumptions?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is of course one standard result: OLS itself is the best unbiased estimator if in addition to the Gauss-Markov assumptions we also assume that the errors are normally distributed. For some other particular distribution of errors I could compute the corresponding maximum-likelihood estimator.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I was wondering if there is some estimator which is better-than-OLS in some relatively general set of circumstances?&lt;/p&gt;&#xA;" OwnerUserId="1393" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-10-19T07:20:15.037" LastActivityDate="2011-03-01T01:07:08.683" Title="OLS is BLUE. But what if I don't care about unbiasedness and linearity?" Tags="&lt;regression&gt;&lt;unbiased-estimator&gt;" AnswerCount="3" FavoriteCount="4" />
  <row Id="2958" PostTypeId="1" CreationDate="2010-09-22T01:21:15.247" Score="2" ViewCount="227" Body="&lt;p&gt;I know the value for the 16% quartile, so I know the additive deviation for the given distribution. How do I find the deviation of the log of the given distribution on a multiplicative scale?&lt;/p&gt;&#xA;" LastActivityDate="2010-09-22T08:22:25.940" Title="How do you calculate the standard deviation on a multiplicative scale for a distribution that has been transformed logarithmically?" Tags="&lt;distributions&gt;&lt;standard-deviation&gt;&lt;logarithm&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="2959" PostTypeId="2" ParentId="2957" CreationDate="2010-09-22T01:47:20.280" Score="6" ViewCount="" Body="&lt;p&gt;I don't know if you are OK with the Bayes Estimate? If yes, then depending on the Loss function you can obtain different Bayes Estimates. A theorem by Blackwell states that Bayes Estimates are never unbiased. A decision theoretic argument states that every admissible rule ((i.e. or every other rule against which it is compared, there is a value of the parameter for which the the risk of the present rule is (strictly) less than that of rule against which it's being compared)) is a (generalized) Bayes rule.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;James-Stein Estimators are another class of estimators (which can be derived by Bayesian methods asymptotically) which are better than OLS in many cases. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;OLS can be inadmissible in many situations and James-Stein Estimator is an example. (also called Stein's paradox).  &lt;/p&gt;&#xA;" OwnerUserId="1307" LastEditorUserId="1307" LastEditorDisplayName="" LastEditDate="2011-02-28T16:24:13.567" LastActivityDate="2011-02-28T16:24:13.567" CommentCount="24" />
  <row Id="2960" PostTypeId="2" ParentId="2925" CreationDate="2010-09-22T03:42:41.153" Score="0" ViewCount="" Body="&lt;p&gt;an easy way to generate symmetric bernoulli trials is to flip a coin twice. if the first toss is H and the second is T, say X = 1. if it's the other way round, say X = 0. if the two tosses match [2H or 2T], discard the outcome and continue. no matter what the bias of the coin, X will be symmetric bernoulli.&lt;/p&gt;&#xA;" OwnerUserId="1112" LastActivityDate="2010-09-22T03:42:41.153" CommentCount="1" />
  <row Id="2961" PostTypeId="2" ParentId="2957" CreationDate="2010-09-22T04:00:30.520" Score="4" ViewCount="" Body="&lt;p&gt;There is a nice review paper by &lt;a href=&quot;http://webee.technion.ac.il/Sites/People/YoninaEldar/Download/67j-04490210.pdf&quot; rel=&quot;nofollow&quot;&gt;Kay and Eldar&lt;/a&gt; on biased estimation for the purpose of finding estimators with minimum mean square error.&lt;/p&gt;&#xA;" OwnerUserId="352" LastActivityDate="2010-09-22T04:00:30.520" CommentCount="1" />
  <row Id="2962" PostTypeId="1" AcceptedAnswerId="2969" CreationDate="2010-09-22T04:38:22.253" Score="5" ViewCount="494" Body="&lt;p&gt;Hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The statistics book I am reading recommends omega squared to measure the effects of my experiments. I have already proven using a split plot design (mix of within-subjects and between-subjects design) that my within-subjects factors are statistically significant with p&amp;lt;0.001 and F=17.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I'm looking to see how big is the difference... is there an implementation of omega squared somewhere for R (or python? I know... one can dream ;) Searching on the internet for R-related stuff is a pain the &lt;em&gt;*&lt;/em&gt;, I don't know how I manage to find stuff with C.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;thanks!&lt;/p&gt;&#xA;" OwnerUserId="1320" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-01-14T19:31:12.340" LastActivityDate="2011-01-14T19:31:12.340" Title="Omega squared for measure of effect in R?" Tags="&lt;r&gt;&lt;anova&gt;&lt;effect-size&gt;&lt;split-plot&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="2963" PostTypeId="2" ParentId="2925" CreationDate="2010-09-22T05:25:07.513" Score="4" ViewCount="" Body="&lt;p&gt;Just as another source of verifiable randomness: random.org generates random numbers from atmospheric noise.  They publish &lt;a href=&quot;http://www.random.org/files/&quot; rel=&quot;nofollow&quot;&gt;a daily file (most days) of random bits&lt;/a&gt;; the first digit of each day's file might prove suitably verifiable to your parties.&lt;/p&gt;&#xA;" OwnerUserId="71" LastActivityDate="2010-09-22T05:25:07.513" CommentCount="3" />
  <row Id="2965" PostTypeId="2" ParentId="2958" CreationDate="2010-09-22T08:22:25.940" Score="1" ViewCount="" Body="&lt;p&gt;I assume you are referring to something like the estimated coefficients in a logistic regression. These are the log-odds. The estimates usually have a standard error and symmetrical confidence interval. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example lets say an estimated log odds is 2 with an SE of 0.5 and 95% CI of 1.02 to 2.98. The odds ratio you calculate as exp(2) = 7.4.  To estimate a &quot;balanced&quot; 95% confidence interval for the odds ratio exponentiate both ends of the 95% CI on the linear scale, viz. exp(1.02) to exp(2.98) gives a 95% CI of 2.8 to 19.7. You could do the same with both ends of the SD or SE interval: exp(1.5) to exp(2.5) gives 4.5 to 12.2 which we might describe as a 68% confidence interval about the mean odds ratio of 7.4.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note this is not the same as the standard deviation of a &quot;transformed&quot; distribution, (&lt;a href=&quot;http://en.wikipedia.org/wiki/Log-normal_distribution&quot; rel=&quot;nofollow&quot;&gt;eg the log-normal&lt;/a&gt;), which is defined on the new scale and is symmetrical on that scale even if the distribution is skew.&lt;/p&gt;&#xA;" OwnerUserId="521" LastActivityDate="2010-09-22T08:22:25.940" />
  <row Id="2966" PostTypeId="1" AcceptedAnswerId="2967" CreationDate="2010-09-22T10:32:51.633" Score="4" ViewCount="105" Body="&lt;p&gt;&lt;strong&gt;Context&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;you have 200 observations of an individual's running time for the 100 metres measured once a day for 200 days.&lt;/li&gt;&#xA;&lt;li&gt;Assume the individual was not a runner before commencement of practice&lt;/li&gt;&#xA;&lt;li&gt;Based on the observed data and the 199 other observations, you want to estimate the latent time it would take the individual to run if they (a) applied maximal effort; and (b) had a reasonably good run for them (i.e., no major problems with the run; but still a typical run). Let's call this &lt;strong&gt;latent potential&lt;/strong&gt;.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Of course, the actual data would not measure latent potential directly. The data would be noisy:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Times would vary from run to run&lt;/li&gt;&#xA;&lt;li&gt;On some days the individual would be particularly slow because of one or more possible problems (e.g., tripping at the start, getting a cramp half way through, not putting in much effort). Such problems would result in  massive outliers&lt;/li&gt;&#xA;&lt;li&gt;On some days the individual would be slower than you'd expect, perhaps because of more minor issues.&lt;/li&gt;&#xA;&lt;li&gt;In general, with practice the runner would be expected to get faster in latent potential.&lt;/li&gt;&#xA;&lt;li&gt;In rare cases, it is possible for the runner to get slower in latent potential (e.g., injury)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The implications of this:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The occasional slow time might provide minimal information on what the individual is capable of. &lt;/li&gt;&#xA;&lt;li&gt;A fast time for the individual suggests that the individual is capable of such a fast time, but a small amount of this fast time might be good fortune on the day (e.g., the right wind, a little luck on the start).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The question:&lt;/strong&gt;&#xA;Thus, how could one estimate  latent potential at each of the 200 time points based on the available data and a few assumptions about the nature of running times?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Initial Thoughts:&lt;/strong&gt; I imagine there would be some form of Bayesian approach that combined the available information and assumptions to form an estimate, but I'm not sure where to look for such models. I'm also not quite clear how the effectiveness of such a model would be evaluated.&lt;/p&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-09-22T10:47:25.850" LastActivityDate="2010-09-27T09:18:34.153" Title="Estimating latent performance potential based on a sequence of observations" Tags="&lt;time-series&gt;&lt;bayesian&gt;&lt;latent-variable&gt;" AnswerCount="3" FavoriteCount="1" />
  <row Id="2967" PostTypeId="2" ParentId="2966" CreationDate="2010-09-22T11:01:39.503" Score="4" ViewCount="" Body="&lt;p&gt;You need to perform an isotonic (i.e. monotonic non decreasing) nonparametric regression (see page 6 of &lt;a href=&quot;http://cran.r-project.org/web/packages/monoProc/vignettes/monoproc.pdf&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; document for an example), then use $\hat{E}(y|x)+ \delta \hat{\sigma}(y|x)$ with $\delta&gt;0$ as the upper potential. There are many packages that will do that in R. I like this &lt;a href=&quot;http://www.stat.ucl.ac.be/ISdidactique/Rhelp/library/modreg/html/isoreg.html&quot; rel=&quot;nofollow&quot;&gt;one&lt;/a&gt; for its simplicity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Isotonic nonparametric is simply your regular scatterplot smoother with the added prior that more $x$ cannot decrease smoothed $y$ (i.e. drug dose vs effects).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the first comment below your design includes a $k$-vector dummy variable $z$ (controling for injury,running style) and a continous variable $x$ (days), assuming that $y$ (latent performance) is given by: $E(y|x,z)=m(x)+\alpha z+\delta$ where $m(x)$ is a monotone scatterplot smoother, $\delta&gt;0$ is known and $\alpha\in \mathbb{R}^k$. This types of model can be estimated by isotonic GAM (see &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.70.5142&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; paper implemented &lt;a href=&quot;http://cran.r-project.org/web/packages/bisoreg/index.html&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: i changed the link to the paper, (the old link was pointing to a derivative of the method by the same author).&lt;/p&gt;&#xA;" OwnerUserId="603" LastEditorUserId="603" LastEditorDisplayName="" LastEditDate="2010-09-27T09:18:34.153" LastActivityDate="2010-09-27T09:18:34.153" CommentCount="4" />
  <row Id="2968" PostTypeId="2" ParentId="2925" CreationDate="2010-09-22T11:53:01.753" Score="5" ViewCount="" Body="&lt;p&gt;Many countries have state lottery which is regularly audited, and whose results are announced online: e.g. &lt;a href=&quot;https://www.national-lottery.co.uk/player/p/results/lotto.ftl&quot; rel=&quot;nofollow&quot;&gt;UK national lottery&lt;/a&gt;. You just need to construct an appropriate function which maps this output space to your desired output. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A continuous distribution would be more tricky, but you could obtain a discrete approximation: in the UK case there are ${}^{49}C_6 \times 43$ = 601 304 088 equally likely outcomes, which, depending on context, could give sufficient granularity.&lt;/p&gt;&#xA;" OwnerUserId="495" LastActivityDate="2010-09-22T11:53:01.753" CommentCount="1" />
  <row Id="2969" PostTypeId="2" ParentId="2962" CreationDate="2010-09-22T12:32:55.730" Score="3" ViewCount="" Body="&lt;p&gt;I'd suggest that generalized eta square is considered (&lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pubmed/14664681&quot; rel=&quot;nofollow&quot;&gt;ref&lt;/a&gt;, &lt;a href=&quot;http://brm.psychonomic-journals.org/content/37/3/379.short&quot; rel=&quot;nofollow&quot;&gt;ref&lt;/a&gt;) a more appropriate measure of effect size. It is included in the ANOVA output in the &lt;a href=&quot;http://cran.r-project.org/package=ez&quot; rel=&quot;nofollow&quot;&gt;ez package&lt;/a&gt; for R.&lt;/p&gt;&#xA;" OwnerUserId="364" LastActivityDate="2010-09-22T12:32:55.730" />
  <row Id="2970" PostTypeId="2" ParentId="2966" CreationDate="2010-09-22T12:36:38.767" Score="2" ViewCount="" Body="&lt;p&gt;Just a guess.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First I would explore transformations of the data, such as converting time to speed or acceleration. Then I would consider the log of that, since it obviously won't be negative.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, since you are interested in the asymptote, I would try fitting (by least squares) a simple exponential to the transformed data, with time t being the x axis, and log-transformed speed (or acceleration) being the y axis. See how that works in predicting new measurements as time increases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A possible alternative to an an exponential function would be a Michaelis-Menten type of hyperbola.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Actually, I would strongly consider a mixed-effect population approach first (as with NONMEM), because each individual may not show enough information to evaluate different models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to go Bayesian, you could use WinBugs, and provide any prior distribution you want to the parameters of the exponential function. The book I found useful is Gilks, Richardson, Spiegelhalter, &quot;Markov Chain Monte Carlo in Practice&quot;, Chapman &amp;amp; Hall, 1996.&lt;/p&gt;&#xA;" OwnerUserId="1270" LastEditorUserId="1270" LastEditorDisplayName="" LastEditDate="2010-09-22T13:28:08.330" LastActivityDate="2010-09-22T13:28:08.330" CommentCount="2" />
  <row Id="2971" PostTypeId="1" AcceptedAnswerId="2985" CreationDate="2010-09-22T14:11:12.697" Score="9" ViewCount="332" Body="&lt;p&gt;If you fit a non linear function to a set of points (assuming there is only one ordinate for each abscissa) the result can either be: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;a very complex function with small residuals&lt;/li&gt;&#xA;&lt;li&gt;a very simple function with large residuals&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Cross validation is commonly used to find the &quot;best&quot; compromise between these two extremes. But what does &quot;best&quot; mean? Is it &quot;most likely&quot;? How would you even start to prove what the most likely solution is? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My inner voice is telling me that CV is finding some sort of minimum energy solution. This makes me think of entropy, which I vaguely know occurs in both stats and physics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems to me that the &quot;best&quot; fit is generated by minimising the sum of functions of complexity and error ie &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;minimising m where m = c(Complexity) + e(Error)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Does this make any sense? What would the functions c and e be?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please can you explain using non mathematical language, because I will not understand much maths.&lt;/p&gt;&#xA;" OwnerUserId="1134" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-23T10:54:12.720" LastActivityDate="2010-09-23T20:19:30.067" Title="What is the definition of &quot;best&quot; as used in the term &quot;best fit&quot; and cross validation?" Tags="&lt;model-selection&gt;&lt;cross-validation&gt;" AnswerCount="7" CommentCount="2" />
  <row Id="2972" PostTypeId="1" AcceptedAnswerId="2989" CreationDate="2010-09-22T14:25:01.887" Score="3" ViewCount="218" Body="&lt;p&gt;A little while back, J.M. &lt;a href=&quot;http://stats.stackexchange.com/questions/2746/how-to-efficiently-generate-positive-semi-definite-correlation-matrices/2786#2786&quot;&gt;suggested&lt;/a&gt; using the &lt;a href=&quot;http://www.jstor.org/stable/2156882&quot; rel=&quot;nofollow&quot;&gt;Stewart&lt;/a&gt; algorithm for generating $n$ by $n$ pseudo random orthogonal matrices in $O(n^2)$ time. He further noted that this methodology is implemented in Nick Higham's matrix computation toolbox (&lt;a href=&quot;http://www.maths.manchester.ac.uk/~higham/mctoolbox/&quot; rel=&quot;nofollow&quot;&gt;matlab&lt;/a&gt;). Now this package contains a bunch of .m files. Could anyone direct me to which one implements the pseudorandom orthogonal matrix generator? Alternatively, would anyone know of a R implementation of this algorithm?&lt;/p&gt;&#xA;" OwnerUserId="603" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-23T10:45:55.277" LastActivityDate="2010-09-23T10:45:55.277" Title="Pseudo-random orthogonal matrix generation" Tags="&lt;random-generation&gt;&lt;matrix&gt;" AnswerCount="1" />
  <row Id="2973" PostTypeId="2" ParentId="665" CreationDate="2010-09-22T14:31:43.833" Score="1" ViewCount="" Body="&lt;p&gt;Statistics is the pursuit of truth in the face of uncertainty. Probability is the tool that allows us to quantify uncertainty.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(I have provided another, longer, answer that assumed that what was being asked was something along the lines of &quot;how would you explain it to your grandmother?&quot;)&lt;/p&gt;&#xA;" OwnerUserId="666" LastActivityDate="2010-09-22T14:31:43.833" />
  <row Id="2974" PostTypeId="2" ParentId="7" CreationDate="2010-09-22T16:22:51.167" Score="1" ViewCount="" Body="&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.freebase.com/&quot; rel=&quot;nofollow&quot;&gt;http://www.freebase.com/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="957" LastActivityDate="2010-09-22T16:22:51.167" CommunityOwnedDate="2010-09-22T16:22:51.167" />
  <row Id="2975" PostTypeId="1" AcceptedAnswerId="2978" CreationDate="2010-09-22T16:35:45.250" Score="5" ViewCount="270" Body="&lt;p&gt;How do I detrend or normalize multiple series of data so that I can inter-compare between the series?&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Specifics below may not be appropriate for this forum. Please let me know and I can remove or re-phrase, but I think it might be helpful to fully understand the generic question above. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a data set that I would like help analyzing. I think this question belongs here and not in at &lt;a href=&quot;http://gis.stackexchange.com/&quot;&gt;http://gis.stackexchange.com/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Specifically, I have the following situation: Each series is collected from an airplane flying a path, and has a variable number of (value, lat,lon,time) tuples. I have several of these flightpaths, each at a different time, and flying different paths (sometimes crossing, sometimes not). Flights happened months apart, at different times of day, and due to natural phenomena, the data (thermal in this case) varies.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Part of the region flown over by multiple flights may or may not have an anomalous temperature signature. This is what I want to investigate. I am seeking an algorithm to detrend or normalize all of the flights so that I can increase my SNR and determine if there is a temperature anomaly over a sub-region.&lt;/p&gt;&#xA;" OwnerUserId="957" LastActivityDate="2010-09-26T07:45:00.947" Title="Normalizing or detrending groups of samples" Tags="&lt;time-series&gt;&lt;data-visualization&gt;&lt;variance&gt;&lt;normalization&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="1" />
  <row Id="2976" PostTypeId="1" AcceptedAnswerId="2998" CreationDate="2010-09-22T17:01:37.580" Score="6" ViewCount="308" Body="&lt;p&gt;Questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;I have a large correlation matrix. Instead of clustering individual correlations, I want to cluster variables based on their correlations to each other, ie if variable A and variable B have similar correlations to variables C to Z, then A and B should be part of the same cluster. A good real life example of this is different asset classes - intra asset-class correlations are higher than inter-asset class correlations.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I am also considering clustering variables in terms stregth relationship between them, eg when the correlation between variables A and B is close to 0, they act more or less independently. If suddenly some underlying conditions change and a strong correlation arises (positive or negative), we can think of these two variables as belonging to the same cluster. So instead of looking for positive correlation, one would look for relationship versus no relationship. I guess an analogy could be a cluster of positively and negatively charged particles. If the charge falls to 0, the particle drifts away from the cluster. However, both positive and negative charges attract particles to revelant clusters.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I apologise if some of this isn't very clear. Please let me know, I will clarify specific details.&lt;/p&gt;&#xA;" OwnerUserId="1250" LastActivityDate="2010-10-29T11:20:13.590" Title="Clustering variables based on correlations between them" Tags="&lt;correlation&gt;&lt;clustering&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="2977" PostTypeId="2" ParentId="2971" CreationDate="2010-09-22T17:04:06.030" Score="5" ViewCount="" Body="&lt;p&gt;I will offer a brief intuitive answer (at a fairly abstract level) till a better answer is offered by someone else: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, note that complex functions/models achieve better fit (i.e., have lower residuals) as they exploit some local features (think noise) of the dataset that are not present globally (think systematic patterns). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second, when performing cross validation we split the data into two sets: the training set and the validation set. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, when we perform cross validation, a complex model may not predict very well because by definition a complex model will exploit the local features of the training set. However, the local features of the training set could be very different compared the local features of the validation set resulting in poor predictive performance. Therefore, we have a tendency to select the model that captures the global features of the training and the validation datasets. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In summary, cross validation protects against overfitting by selecting the model that captures the global patterns of the dataset and by avoiding models that exploit some local feature of a dataset.&lt;/p&gt;&#xA;" LastActivityDate="2010-09-22T17:19:31.487" CommentCount="4" />
  <row Id="2978" PostTypeId="2" ParentId="2975" CreationDate="2010-09-22T17:21:32.487" Score="4" ViewCount="" Body="&lt;p&gt;Multi-level modelling where your data are grouped by flight as a random variable sounds like a good analysis method for this problem.  In R the code might be&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(lme4) #load the package)&#xA;lmer(temp ~ region + (1|flight))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is doable in a variety of statistics packages.  If region is simply in region or outside of region then a logistic form should be used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To directly address your question of normalizing you might like&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;temp - (mean_temp_for_flight - mean_temp)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This zeros the temperatures at the overall mean corrected for the individual flight mean.  So, if a flight had a mean temp of 20 over the region, and the mean temp of the region is 18, and your sample is 22, then the normalized value would be&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;22 - (20 - 18) = 20&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Essentially... by flight variability is eliminated.&lt;/p&gt;&#xA;" OwnerUserId="601" LastEditorUserId="601" LastEditorDisplayName="" LastEditDate="2010-09-26T07:45:00.947" LastActivityDate="2010-09-26T07:45:00.947" CommentCount="1" />
  <row Id="2979" PostTypeId="2" ParentId="2904" CreationDate="2010-09-22T17:41:07.970" Score="0" ViewCount="" Body="&lt;p&gt;You could avoid the problem altogether by simply estimating&lt;/p&gt;&#xA;&#xA;&lt;p&gt;W = alphaH * H + alphaM * M + alphaL * L + X * beta&lt;/p&gt;&#xA;&#xA;&lt;p&gt;using 2sls. The fact that H,M, and L are discrete doesn't violate any of the assumptions of 2sls. Of course, using maximum likelihood will produce more efficient estimates, but it relies on more assumptions. If nothing else, the 2sls estimates should provide good starting values for you maximization algorithm. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For maximizing the likelihood,you should try changing your simulation method to make the likelihood function smooth. I think a slight variant of the Geweke-Hajivassiliou-Keane (often just GHK) simulator would work. &lt;/p&gt;&#xA;" OwnerUserId="1229" LastActivityDate="2010-09-22T17:41:07.970" CommentCount="4" />
  <row Id="2980" PostTypeId="2" ParentId="2925" CreationDate="2010-09-22T18:26:26.657" Score="0" ViewCount="" Body="&lt;p&gt;I didn't quite understand what you meant by &quot;on the basis of an external event.&quot; But you can certainly flip a fair coin in a manner that a remote user can cryptographically verify.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider this algorithm:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Bob picks a uniformly random boolean value, TRUE or FALSE. He also chooses a large random number. He sends Alice the SHA-256 hash of the boolean value concatenated with the number. (E.g. he sends the hash of &quot;TRUE|12345678&quot;.) Since Alice doesn't know the random number and the hash is one-way, Alice doesn't know the boolean value.&lt;/li&gt;&#xA;&lt;li&gt;Alice flips a coin and sends Bob the value -- TRUE or FALSE.&lt;/li&gt;&#xA;&lt;li&gt;Bob reveals the random number, and thus his own boolean value. Alice verifies that the boolean value and the random number indeed hash to the value she received earlier.&lt;/li&gt;&#xA;&lt;li&gt;This final output of the coin flip is the exclusive-or of Alice's boolean value and Bob's boolean value.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;With this algorithm, no party can cheat without the cooperation of the counterparty. If either party plays fairly, the output will be a uniform random boolean value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(EDIT) I now understand the problem to mean you have no internal source of nondeterminism at all, so all randomness has to come from an external source and the algorithm has to be deterministic. In that case, we can still use cryptography to help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How about taking the SHA-256 of the PDF version of The New York Times every day, or the SHA-256 of the volume and closing price of all the stocks in the Dow Jones Industrial Average in alphabetical order by ticker symbol, or really the secure hash of anything that can be mutually observed and that you can't influence. If you want just one bit, take the first bit of the SHA-256.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want a normal distribution, you could take the whole thing in two parts (128 bits, then 128 bits) as two uniform deviates and use the Box-Muller transform to get two normal deviates.&lt;/p&gt;&#xA;" OwnerUserId="1122" LastEditorUserId="1122" LastEditorDisplayName="" LastEditDate="2010-09-23T02:54:38.517" LastActivityDate="2010-09-23T02:54:38.517" CommentCount="3" />
  <row Id="2981" PostTypeId="1" AcceptedAnswerId="2986" CreationDate="2010-09-22T18:49:28.433" Score="5" ViewCount="232" Body="&lt;p&gt;Hi everyone,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With the help of several people in this community I have been wetting my feet in clustering some social network data using &lt;a href=&quot;http://igraph.sourceforge.net/doc/R/fastgreedy.community.html&quot; rel=&quot;nofollow&quot;&gt;igraph's implementation of modularity-based clustering&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am having some trouble interpreting the output of this routine and how to use it to generate lists of members of each community detected. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This routine outputs a two-column matrix and a list of modularity values. From &lt;a href=&quot;http://igraph.sourceforge.net/doc/R/fastgreedy.community.html&quot; rel=&quot;nofollow&quot;&gt;the docs&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;merges: A matrix with two column, this&#xA;  represents a dendogram and contains&#xA;  all the merges the algorithm&#xA;  performed. Each line is one merge and&#xA;  it is given by the ids of the two&#xA;  communities merged. The community ids&#xA;  are integer numbers starting from zero&#xA;  and the communities between zero and&#xA;  the number of vertices (N) minus one&#xA;  belong to individual vertices. The&#xA;  first line of the matrix gives the&#xA;  first merge, this merge creates&#xA;  community N, the number of vertices,&#xA;  the second merge creates community&#xA;  N+1, etc. &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;modularity:  A numeric vector&#xA;  containing the modularity value of the&#xA;  community structure after performing&#xA;  every merge.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Working with this explanation and looking at the example at the bottom of the man page, I think the communities in this graph are&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1st community: 0 1 2 3 4&#xA;2nd community: 10 11 12 13 14&#xA;3rd community: 5 6 7 8 9&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Can someone who has used this method before confirm whether the right approach produces this result? I have basically (i) ignored the last two merges and (ii) gone over each row in the 'merges' matrix, combining each pair of vertices into a set while watching out for vertex values that are larger than the number of vertices (and therefore refer to another row in the 'merges' matrix). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks!&#xA;~l&lt;/p&gt;&#xA;" OwnerUserId="1007" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-24T13:59:30.403" LastActivityDate="2010-09-24T13:59:30.403" Title="Interpreting output of igraph's fastgreedy.community clustering method" Tags="&lt;clustering&gt;&lt;networks&gt;&lt;data-visualization&gt;&lt;partitioning&gt;&lt;igraph&gt;" AnswerCount="1" />
  <row Id="2982" PostTypeId="1" AcceptedAnswerId="3005" CreationDate="2010-09-22T20:53:20.303" Score="7" ViewCount="457" Body="&lt;p&gt;I am trying to summarize what I understood so far in penalized multivariate analysis with high-dimensional data sets, and I still struggle through getting a proper definition of &lt;em&gt;soft-thresholding&lt;/em&gt; vs. &lt;em&gt;Lasso&lt;/em&gt; (or $L_1$) penalization.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More precisely, I used sparse PLS regression to analyze 2-block data structure including genomic data (&lt;a href=&quot;http://en.wikipedia.org/wiki/Single-nucleotide_polymorphism&quot; rel=&quot;nofollow&quot;&gt;single nucleotide polymorphisms&lt;/a&gt;, where we consider the frequency of the minor allele in the range {0,1,2}, considered as a numerical variable) and continuous phenotypes (scores quantifying personality traits or cerebral asymmetry, also treated as continuous variables). The idea was to isolate the most influential predictors (here, the genetic variations on the DNA sequence) to explain inter-individual phenotypic variations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I initially used the &lt;a href=&quot;http://cran.r-project.org/web/packages/mixOmics/index.html&quot; rel=&quot;nofollow&quot;&gt;mixOmics&lt;/a&gt; R package (formerly &lt;code&gt;integrOmics&lt;/code&gt;) which features penalized &lt;a href=&quot;http://en.wikipedia.org/wiki/Partial_least_squares_regression&quot; rel=&quot;nofollow&quot;&gt;PLS&lt;/a&gt; regression and regularized &lt;a href=&quot;http://en.wikipedia.org/wiki/Canonical_correlation&quot; rel=&quot;nofollow&quot;&gt;CCA&lt;/a&gt;. Looking at the R code, we found that the &quot;sparsity&quot; in the predictors is simply induced by selecting the top $k$ variables with highest loadings (in absolute value) on the $i$th component, $i=1,\dots, k$ (the algorithm is iterative and compute variables loadings on $k$ components, deflating the predictors block at each iteration, see &lt;a href=&quot;http://hal.archives-ouvertes.fr/docs/00/30/02/04/PDF/sPLS.pdf&quot; rel=&quot;nofollow&quot;&gt;Sparse PLS: Variable Selection when Integrating Omics data&lt;/a&gt; for an overview).&#xA;On the contrary, the &lt;a href=&quot;http://cran.r-project.org/web/packages/spls/index.html&quot; rel=&quot;nofollow&quot;&gt;spls&lt;/a&gt; package co-authored by S. Keleş (see &lt;a href=&quot;http://www.stat.wisc.edu/~keles/Papers/SPLS_Nov07.pdf&quot; rel=&quot;nofollow&quot;&gt;Sparse Partial Least Squares Regression for Simultaneous Dimension Reduction and Variable Selection&lt;/a&gt;, for a more formal description of the approach undertaken by these authors) implements $L_1$-penalization for variable penalization.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is not obvious to me whether there is a strict &quot;bijection&quot;, so to say, between iterative feature selection based on soft-thresholding and $L_1$ regularization. So my question is: &lt;strong&gt;Is there any mathematical connection between the two?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;References&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Chun, H. and Kele ̧s, S. (2010), &lt;a href=&quot;http://www.stat.wisc.edu/~keles/Papers/C_SPLS.pdf&quot; rel=&quot;nofollow&quot;&gt;Sparse partial least squares for simultaneous dimension reduction and variable selection&lt;/a&gt;. &lt;em&gt;Journal of the Royal Statistical Society: Series B&lt;/em&gt;, &lt;em&gt;72&lt;/em&gt;, 3–25.&lt;/li&gt;&#xA;&lt;li&gt;Le Cao, K.-A., Rossouw, D., Robert-Granie, C., and Besse, P. (2008), &lt;a href=&quot;http://www.bepress.com/sagmb/vol7/iss1/art35/&quot; rel=&quot;nofollow&quot;&gt;A Sparse PLS for Variable Selection when Integrating Omics Data&lt;/a&gt;. &lt;em&gt;Statistical Applications in Genetics and Molecular Biology&lt;/em&gt;, &lt;em&gt;7&lt;/em&gt;, Article 35.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-12-21T15:30:40.507" LastActivityDate="2010-12-21T15:30:40.507" Title="Soft-thresholding vs. Lasso penalization" Tags="&lt;multivariate-analysis&gt;&lt;lasso&gt;&lt;feature-selection&gt;&lt;genetics&gt;" AnswerCount="2" FavoriteCount="3" />
  <row Id="2983" PostTypeId="2" ParentId="2971" CreationDate="2010-09-22T20:59:33.923" Score="0" ViewCount="" Body="&lt;p&gt;The error function is the error of your model (function) on the training data. The complexity is some norm (e.g., squared l2 norm) of the function you are trying to learn. Minimizing the complexity term essentially favors smooth functions, which do well not just on the training data but also on the test data. If you represent your function by a set of coefficients (say, if you are doing linear regression), penalizing the complexity by the squared norm would lead to small coefficient values in your function (penalizing other norms leads to different notions of complexity control). &lt;/p&gt;&#xA;" OwnerUserId="881" LastEditorUserId="881" LastEditorDisplayName="" LastEditDate="2010-09-22T21:05:22.323" LastActivityDate="2010-09-22T21:05:22.323" />
  <row Id="2984" PostTypeId="2" ParentId="2971" CreationDate="2010-09-22T21:47:06.203" Score="1" ViewCount="" Body="&lt;p&gt;From an optimization point of view, the problem (with $(p,q)\geq 1,\;\lambda&gt;0$), &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$(1)\;\underset{\beta|\lambda,x,y}{Arg\min.}||y-m(x,\beta)||_p+\lambda||\beta||_q$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;is equivalent to &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$(2)\;\underset{\beta|\lambda,x,y}{Arg\min.}||y-m(x,\beta)||_p$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$s.t.$ $||\beta||_q\leq\lambda$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which simply incorporates unto the objective function the prior information that $||\beta||_q\leq\lambda$.  If this prior turns out to be true, then it can be shown ($q=1,2$) that incorporating it unto the objective function minimizes the risk associated with $\hat{\beta}$ (i.e. very unformaly, improves the accuracy of $\hat{\beta}$)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\lambda$ is a so called meta-parameter (or latent parameter) that is not being optimized over (in which case the solution would trivially reduce to $\lambda=\infty$), but rather, reflects information not contained in the sample $(x,y)$ used to solve $(1)-(2)$ (for example other studies or expert's opinion). Cross validation is an attempt at constructing a data induced prior (i.e. slicing the dataset so that part of it is used to infer reasonable values of $\lambda$ and part of it used to estimate $\hat{\beta}|\lambda$).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As to your subquestion (why $e()=||y-m(x,\beta)||_p$) this is because for $p=1$ ($p=2$) this measure of distance between the model and the observations has (easely) derivable assymptotical properties (strong convergence to meaningfull population couterparts of $m()$).&lt;/p&gt;&#xA;" OwnerUserId="603" LastEditorUserId="603" LastEditorDisplayName="" LastEditDate="2010-09-23T00:56:12.477" LastActivityDate="2010-09-23T00:56:12.477" CommentCount="6" />
  <row Id="2985" PostTypeId="2" ParentId="2971" CreationDate="2010-09-22T22:11:42.100" Score="4" ViewCount="" Body="&lt;p&gt;I think this is an excellent question. I am going to paraphase it just to be sure I have got it right:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;It would seem that there are lots of&#xA;  ways to choose the complexity penalty&#xA;  function $c$ and error penalty&#xA;  function $e$. Which choice is `best'. &#xA;  What should best even &lt;em&gt;mean&lt;/em&gt;?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I think the answer (if there is one) will take you way beyond just cross-validation. I like how this question (and the topic in general) ties nicely to &lt;a href=&quot;http://en.wikipedia.org/wiki/Occam%27s_razor&quot; rel=&quot;nofollow&quot;&gt;Occam's Razor&lt;/a&gt; and the general concept of &lt;a href=&quot;http://en.wikipedia.org/wiki/Parsimony&quot; rel=&quot;nofollow&quot;&gt;parsimony&lt;/a&gt; that is fundamental to science. I am by no means an expert in this area but I find this question hugely interesting. The best text I know on these sorts of question is &lt;a href=&quot;http://www.hutter1.net/ai/uaibook.htm&quot; rel=&quot;nofollow&quot;&gt;Universal Artificial Intelligence&lt;/a&gt; by Marcus Hutter (don't ask me any questions about it though, I haven't read most of it). I went to a talk by Hutter and couple of years ago and was very impressed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You are right in thinking that there is a minimum entropy argument in there somewhere (used for the complexity penalty function $c$ in some manner). Hutter advocates the use of &lt;a href=&quot;http://en.wikipedia.org/wiki/Kolmogorov_complexity&quot; rel=&quot;nofollow&quot;&gt;Kolmogorov complexity&lt;/a&gt; instead of entropy. Also, Hutter's definition of `best' (as far as I remember) is (informally) the model that &lt;em&gt;best predicts the future&lt;/em&gt; (i.e. best predicts the data that will be observed in the future). I can't remember how he formalises this notion.&lt;/p&gt;&#xA;" OwnerUserId="352" LastEditorUserId="352" LastEditorDisplayName="" LastEditDate="2010-09-22T23:28:21.123" LastActivityDate="2010-09-22T23:28:21.123" CommentCount="4" />
  <row Id="2986" PostTypeId="2" ParentId="2981" CreationDate="2010-09-22T23:05:26.657" Score="2" ViewCount="" Body="&lt;p&gt;The function which is used for this purpose:&#xA;  community.to.membership(graph, merges, steps, membership=TRUE, csize=TRUE)&#xA;this can be used to extract membership based on the fastgreedy.community function results.&#xA;You have to provide number of steps - how many merges should be performed. The optimal number of steps(merges) is the one which produce the maximal modularity.&lt;/p&gt;&#xA;" OwnerUserId="1396" LastActivityDate="2010-09-22T23:05:26.657" CommentCount="2" />
  <row Id="2987" PostTypeId="2" ParentId="7" CreationDate="2010-09-22T23:57:02.360" Score="1" ViewCount="" Body="&lt;p&gt;This is probably the most complete list you'll find: &lt;a href=&quot;http://www.datawrangling.com/some-datasets-available-on-the-web&quot; rel=&quot;nofollow&quot;&gt;Some Datasets Available on the Web&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="635" LastActivityDate="2010-09-22T23:57:02.360" CommunityOwnedDate="2010-09-22T23:57:02.360" />
  <row Id="2988" PostTypeId="1" CreationDate="2010-09-23T00:12:51.567" Score="9" ViewCount="948" Body="&lt;p&gt;How does one calculate the sample size needed for a study in which a cohort of subjects will have a single continuous variable measured at the time of a surgery and then two years later they will be classified as functional outcome or impaired outcome.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We would like to see if that measurement could have predicted the bad outcome. At some point we may want to derive a cut point in the continuous variable above which we would try to intervene to diminish the probability of the impaired outcome. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas? Any R implementation.&lt;/p&gt;&#xA;" OwnerUserId="104" LastActivityDate="2010-12-03T11:40:53.517" Title="Sample size calculation for univariate logistic regression" Tags="&lt;logistic-regression&gt;&lt;sample-size&gt;" AnswerCount="4" CommentCount="3" FavoriteCount="3" />
  <row Id="2989" PostTypeId="2" ParentId="2972" CreationDate="2010-09-23T00:22:16.190" Score="7" ViewCount="" Body="&lt;p&gt;It's in the Test Matrix Toolbox, not the Matrix Computation Toolbox. The M-file &lt;code&gt;qmult.m&lt;/code&gt; (premultiplication by a Haar-distributed pseudorandom orthogonal matrix) can be found &lt;a href=&quot;http://www.netlib.org/toms/694&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; or &lt;a href=&quot;http://people.sc.fsu.edu/~jburkardt/m_src/test_matrix/qmult.m&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="830" LastActivityDate="2010-09-23T00:22:16.190" CommentCount="3" />
  <row Id="2990" PostTypeId="1" AcceptedAnswerId="3020" CreationDate="2010-09-23T00:45:19.997" Score="5" ViewCount="149" Body="&lt;p&gt;Our first polystats project research question is:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Are there reputation effects in SE communities and to what extent are reputation effects contingent on various variables such as size of the community, number of views per day, no of questions per day etc?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;See: &lt;a href=&quot;http://stats.stackexchange.com/q/2867/28&quot;&gt;Which one of the following questions should we address for our first Polystats Project?&lt;/a&gt; for votes on various proposed questions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am posting below some issues pertaining to data collection. As soon as reach consensus on these issues we can proceed to the exploratory data analysis phase of the project. Feel free to post any proposals that may occur to you and on which you think we need consensus. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Similar to the previous question, this poll will be open for 3 days till Sep 25th 00 hrs UTC.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please vote up or down vote as appropriate as this is a CW question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Previous Questions reg Polystats-Project-1:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Question 1: &lt;a href=&quot;http://stats.stackexchange.com/q/2512/28&quot;&gt;What should be our first Polystats Project?&lt;/a&gt;&lt;br&gt;&#xA;Question 2: &lt;a href=&quot;http://stats.stackexchange.com/q/2662/28&quot;&gt;What should be the data collection strategy for our first Polystats Project?&lt;/a&gt;&lt;br&gt;&#xA;Question 3: &lt;a href=&quot;http://stats.stackexchange.com/q/2867/28&quot;&gt;Which one of the following questions should we address for our first Polystats Project?&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Relevant Meta Thread:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://meta.stats.stackexchange.com/q/419/28Blockquote&quot;&gt;How about starting a Polystats Project?&lt;/a&gt;&lt;/p&gt;&#xA;" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-23T18:03:47.800" LastActivityDate="2010-09-24T17:10:13.087" CommunityOwnedDate="2010-09-23T00:45:19.997" Title="Poll on data collection issues for our first Polystats Project" Tags="&lt;polystats-project-1&gt;&lt;project-management&gt;" AnswerCount="9" FavoriteCount="1" />
  <row Id="2991" PostTypeId="2" ParentId="2990" CreationDate="2010-09-23T00:46:51.633" Score="0" ViewCount="" Body="&lt;p&gt;We should restrict data collection to Stackoverflow, Server Fault and Super User only.&lt;/p&gt;&#xA;" LastActivityDate="2010-09-23T00:46:51.633" CommunityOwnedDate="2010-09-23T00:46:51.633" />
  <row Id="2992" PostTypeId="2" ParentId="2990" CreationDate="2010-09-23T00:47:30.523" Score="1" ViewCount="" Body="&lt;p&gt;We should restrict data collection to SE 2.0 beta communities only.&lt;/p&gt;&#xA;" LastActivityDate="2010-09-23T00:47:30.523" CommunityOwnedDate="2010-09-23T00:47:30.523" />
  <row Id="2993" PostTypeId="2" ParentId="2990" CreationDate="2010-09-23T00:48:36.840" Score="0" ViewCount="" Body="&lt;p&gt;Data should be collected for all SE 2.0 beta communities, Stackoverflow, Server Fault and Super User.&lt;/p&gt;&#xA;" LastActivityDate="2010-09-23T00:48:36.840" CommunityOwnedDate="2010-09-23T00:48:36.840" />
  <row Id="2994" PostTypeId="2" ParentId="2990" CreationDate="2010-09-23T00:49:33.047" Score="3" ViewCount="" Body="&lt;p&gt;All of us should analyze the same dataset.&lt;/p&gt;&#xA;" LastActivityDate="2010-09-23T00:49:33.047" CommunityOwnedDate="2010-09-23T00:49:33.047" />
  <row Id="2995" PostTypeId="2" ParentId="2990" CreationDate="2010-09-23T00:50:57.600" Score="0" ViewCount="" Body="&lt;p&gt;We should collect a representative sample of the data as downloading the entire data (especially for Stackoverflow, SuperUser and Server Fault) is impractical. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you vote for this please suggest a sampling strategy as well. &lt;/p&gt;&#xA;" LastActivityDate="2010-09-23T00:50:57.600" CommunityOwnedDate="2010-09-23T00:50:57.600" />
  <row Id="2996" PostTypeId="2" ParentId="2990" CreationDate="2010-09-23T01:00:26.677" Score="0" ViewCount="" Body="&lt;p&gt;The following variables are available from the data dump:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;POSTS &#xA;Id PostTypeId AcceptedAnswerId CreationDate Score ViewCount Body &#xA;OwnerUserId LastEditorUserId LastEditDate LastActivityDate Title Tags &#xA;AnswerCount CommentCount FavoriteCount ClosedDate ParentId CommunityOwnedDate &#xA;&#xA;USERS &#xA;Id Reputation EmailHash CreationDate DisplayName LastAccessDate WebsiteUrl &#xA;Location Age AboutMe Views UpVotes DownVotes &#xA;&#xA;COMMENTS &#xA;Id PostId Score Text CreationDate UserId BADGES Id UserId Name Date &#xA;&#xA;POSTTAGS &#xA;PostId TagId TAGS Id TagName VOTES Id PostId VoteTypeId CreationDate &#xA;BountyAmount UserId VoteTypes Id Name&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;We do &lt;em&gt;not&lt;/em&gt; need to download comments and tags. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the users table, we need:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ID Reputation CreationDate Upvotes Downvotes Views&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;From the posts table we need:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ID PostTypeID OwnerUserID CreationDate Score ViewCount AnswerCount&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For each one of the sites, we need:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;SiteName NoOfUsers CreationDate VariousSummaryStats (No of views per day etc)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Edit this 'answer' to add data that you think we will require for our analysis. &lt;/p&gt;&#xA;" LastActivityDate="2010-09-23T01:00:26.677" CommunityOwnedDate="2010-09-23T01:00:26.677" />
  <row Id="2997" PostTypeId="2" ParentId="2990" CreationDate="2010-09-23T01:01:31.027" Score="5" ViewCount="" Body="&lt;p&gt;Data should be in CSV format.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you disagree suggest an alternative format.&lt;/p&gt;&#xA;" LastActivityDate="2010-09-23T01:01:31.027" CommunityOwnedDate="2010-09-23T01:01:31.027" CommentCount="1" />
  <row Id="2998" PostTypeId="2" ParentId="2976" CreationDate="2010-09-23T01:38:24.977" Score="6" ViewCount="" Body="&lt;p&gt;Here's a simple example in R using the &lt;code&gt;bfi&lt;/code&gt; dataset: bfi is a dataset of 25 personality test items organised around 5 factors.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(psych)&#xA;data(bfi)&#xA;x &amp;lt;- bfi &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;A hiearchical cluster analysis using the euclidan distance between variables based on the absolute correlation between variables can be obtained like so:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plot(hclust(dist(abs(cor(na.omit(x))))))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/aTgoU.png&quot; alt=&quot;alt text&quot;&gt;&#xA;The dendrogram shows how items generally cluster with other items according to theorised groupings (e.g., N (Neuroticism) items group together). It also shows how some items within clusters are more similar (e.g., C5 and C1 might be more similar than C5 with C3). It also suggests that the N cluster is less similar to other clusters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively you could do a standard factor analysis like so:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;factanal(na.omit(x), 5, rotation = &quot;Promax&quot;)&#xA;&#xA;&#xA;Uniquenesses:&#xA;   A1    A2    A3    A4    A5    C1    C2    C3    C4    C5    E1    E2    E3    E4    E5    N1 &#xA;0.848 0.630 0.642 0.829 0.442 0.566 0.635 0.572 0.504 0.603 0.541 0.457 0.541 0.420 0.549 0.272 &#xA;   N2    N3    N4    N5    O1    O2    O3    O4    O5 &#xA;0.321 0.526 0.514 0.675 0.625 0.804 0.544 0.630 0.814 &#xA;&#xA;Loadings:&#xA;   Factor1 Factor2 Factor3 Factor4 Factor5&#xA;A1  0.242  -0.154          -0.253  -0.164 &#xA;A2                          0.570         &#xA;A3         -0.100           0.522   0.114 &#xA;A4                  0.137   0.351  -0.158 &#xA;A5         -0.145           0.691         &#xA;C1                  0.630           0.184 &#xA;C2  0.131   0.120   0.603                 &#xA;C3  0.154           0.638                 &#xA;C4  0.167          -0.656                 &#xA;C5  0.149          -0.571           0.125 &#xA;E1          0.618   0.125  -0.210  -0.120 &#xA;E2          0.665          -0.204         &#xA;E3         -0.404           0.332   0.289 &#xA;E4         -0.506           0.555  -0.155 &#xA;E5  0.175  -0.525   0.234           0.228 &#xA;N1  0.879  -0.150                         &#xA;N2  0.875  -0.152                         &#xA;N3  0.658                                 &#xA;N4  0.406   0.342  -0.148           0.196 &#xA;N5  0.471   0.253           0.140  -0.101 &#xA;O1         -0.108                   0.595 &#xA;O2 -0.145   0.421   0.125   0.199         &#xA;O3         -0.204                   0.605 &#xA;O4          0.244                   0.548 &#xA;O5  0.139                   0.177  -0.441 &#xA;&#xA;               Factor1 Factor2 Factor3 Factor4 Factor5&#xA;SS loadings      2.610   2.138   2.075   1.899   1.570&#xA;Proportion Var   0.104   0.086   0.083   0.076   0.063&#xA;Cumulative Var   0.104   0.190   0.273   0.349   0.412&#xA;&#xA;Test of the hypothesis that 5 factors are sufficient.&#xA;The chi square statistic is 767.57 on 185 degrees of freedom.&#xA;The p-value is 5.93e-72 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-10-29T11:20:13.590" LastActivityDate="2010-10-29T11:20:13.590" CommentCount="7" />
  <row Id="2999" PostTypeId="2" ParentId="2988" CreationDate="2010-09-23T01:56:37.513" Score="7" ViewCount="" Body="&lt;p&gt;Sample size calculations for logistic regression are complex. I wont attempt to summarise it here. Reasonably accessible solutions to this problem are found in:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://onlinelibrary.wiley.com/doi/10.1002/sim.4780080704/abstract&quot; rel=&quot;nofollow&quot;&gt;Hsieh FY. Sample size tables for logistic regression. Statistics in Medicine. 1989 Jul;8(7):795-802.&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://onlinelibrary.wiley.com/doi/10.1002/%28SICI%291097-0258%2819980730%2917:14%3C1623%3a%3aAID-SIM871%3E3.0.CO;2-S/abstract&quot; rel=&quot;nofollow&quot;&gt;Hsieh FY, et al. A simple method of sample size calculation for linear and logistic regression. Statistics in Medicine. 1998 Jul 30;17(14):1623-34.&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An accessible discussion of the issues with example calculations can be found in the last chapter (Section 8.5 pp 339-347) of Hosmer &amp;amp; Lemeshow's &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0471356328&quot; rel=&quot;nofollow&quot;&gt;Applied Logistic Regression&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="521" LastActivityDate="2010-09-23T01:56:37.513" />
  <row Id="3000" PostTypeId="2" ParentId="2988" CreationDate="2010-09-23T02:13:36.460" Score="3" ViewCount="" Body="&lt;p&gt;a simple question about sample size is: how large a sample is needed to get a 95% confidence interval no longer than 2d for the [unknown] mean of the data distribution. another variant is: how large a sample is needed to have power 0.9 at $\theta = 1$ when testing H$_0: \theta = 0$. you don't seem to specify any criterion for choosing a sample size.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;actually, it sounds as tho your study will be conducted in a sequential fashion. in that case, it may pay to make that an explicit part of the experiment. sequential sampling can often be more efficient than a fixed sample-size experiment [fewer observations needed, on average].&lt;/p&gt;&#xA;&#xA;&lt;p&gt;farrel: i'm adding this in reply to your comment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;to get at a sample size, one usually specifies some sort of precision criterion for an estimate [such as length of a CI] OR power at a specified alternative of a test to be carried out on the data. you seem to have mentioned both of these criteria. there is nothing wrong with that, in principle: you just have to then do two sample size calculations - one to achieve the desired estimation precision - and another to get the desired power at the stated alternative. then the larger of the two sample sizes is what is required. [btw - other than saying 80% power - you don't seem to have mentioned what test you plan to perform - or the alternative at which you want the 80% power.]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;as for using sequential analysis: if subjects are enrolled in the study all at the same time, then a fixed sample size makes sense. but if the subjects are few and far between, it may take a year or two [or more] to get the required number enrolled. thus the trial could go on for three or four years [or more]. in that case, a sequential scheme offers the possibility of stopping sooner than that - if the effect[s] you are looking for become statistically significant earlier in the trial.&lt;/p&gt;&#xA;" OwnerUserId="1112" LastEditorUserId="1112" LastEditorDisplayName="" LastEditDate="2010-09-26T01:42:35.190" LastActivityDate="2010-09-26T01:42:35.190" CommentCount="1" />
  <row Id="3001" PostTypeId="1" AcceptedAnswerId="3002" CreationDate="2010-09-23T03:57:03.107" Score="6" ViewCount="157" Body="&lt;p&gt;So let's say you have a distribution where X is the 16% quantile. Then you take the log of all the values of the distribution. Would log(X) still be the 16% quantile in the log distribution?&lt;/p&gt;&#xA;" OwnerUserId="1395" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-11-02T13:51:35.310" LastActivityDate="2010-11-02T13:51:35.310" Title="Log graph question" Tags="&lt;standard-deviation&gt;&lt;logarithm&gt;&lt;quantiles&gt;&lt;homework&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="3002" PostTypeId="2" ParentId="3001" CreationDate="2010-09-23T04:30:57.217" Score="9" ViewCount="" Body="&lt;p&gt;Yes. Quantiles can be transformed under any monotonically increasing transformation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To see this, suppose $Y$ is the random variable and $q_{0.16}$ is the 16% quantile. Then &#xA;$$&#xA;\text{Pr}(Y\le q_{0.16}) = \text{Pr}(\log(Y)\le\log(q_{0.16})) = 0.16.&#xA;$$&#xA;Generally, if $f$ is monotonic and increasing then&#xA;$$&#xA;\text{Pr}(Y\le q_{\alpha}) = \text{Pr}(f(Y)\le f(q_{\alpha})) = \alpha.&#xA;$$&lt;/p&gt;&#xA;" OwnerUserId="159" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-09-24T03:06:00.800" LastActivityDate="2010-09-24T03:06:00.800" CommentCount="2" />
  <row Id="3003" PostTypeId="2" ParentId="7" CreationDate="2010-09-23T06:10:48.023" Score="1" ViewCount="" Body="&lt;p&gt;Peter Skomoroch maintains a list of datasets at &lt;a href=&quot;http://www.datawrangling.com/some-datasets-available-on-the-web&quot; rel=&quot;nofollow&quot;&gt;http://www.datawrangling.com/some-datasets-available-on-the-web&lt;/a&gt;. Many of the links provided as to places that list datasets.&lt;/p&gt;&#xA;" OwnerUserId="1392" LastActivityDate="2010-09-23T06:10:48.023" CommunityOwnedDate="2010-09-23T06:10:48.023" />
  <row Id="3004" PostTypeId="2" ParentId="2971" CreationDate="2010-09-23T07:39:23.560" Score="1" ViewCount="" Body="&lt;p&gt;Hello Bart, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A lot of people have excellent answers, here is my $0.02.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are two ways to look at &quot;best model&quot;, or &quot;model selection&quot;, speaking statistically:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1 An explanation that is as simple as&#xA;   possible, but no simpler (Attrib.&#xA;   Einstein) &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;- This is also called Occam's Razor, as explanation applies here.&#xA;- Have a concept of True model or a model which approximates the truth&#xA;- Explanation is like doing scientific research&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2 Prediction is the interest, similar to engineering development.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;- Prediction is the aim, and all that matters is that the model works&#xA;- Model choice should be based on quality of predictions&#xA;- Cf: Ein-Dor, P. &amp;amp; Feldmesser, J. (1987) Attributes of the performance of central processing units: a relative performance prediction model. Communications of the ACM 30, 308–317.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Widespread (mis)conception: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Model Choice is equivalent to choosing the best model&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For explanation we ought to be alert to be possibility of there being several&#xA;(roughly) equally good explanatory models. Simplicity helps both with communicating the concepts embodied in the model and in what psychologists call generalization, the ability to ‘work’ in scenarios very different from those in which the model was studied. So&#xA;there is a premium on few models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For prediction: (Dr Ripley's) good analogy is that of choosing between expert&#xA;opinions: if you have access to a large panel of experts, how would you&#xA;use their opinions?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cross Validation takes care of the prediction aspect. For details about CV please refer to this presentation by Dr. B. D. Ripley &lt;a href=&quot;http://www.stats.ox.ac.uk/~ripley/Nelder80.pdf&quot; rel=&quot;nofollow&quot; title=&quot;Brian Ripley's Model Selection Presentation&quot;&gt;Dr. Brian D. Ripley's presentation on model selection&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Citation: Please note that everything in this answer is from the presentation cited above. I am a big fan of this presentation and I like it. Other opinions may vary. The title of the presentation is: &quot;Selecting Amongst Large Classes of Models&quot; and was given at Symposium in Honour of John Nelder's 80th birthday, Imperial College, 29/30 March 2004, by Dr. Brian D. Ripley.&lt;/p&gt;&#xA;" OwnerUserId="1307" LastActivityDate="2010-09-23T07:39:23.560" />
  <row Id="3005" PostTypeId="2" ParentId="2982" CreationDate="2010-09-23T07:51:46.680" Score="1" ViewCount="" Body="&lt;p&gt;What i'll say holds for regression, but should be true for PLS also. So it's not a bijection because depeding on how much you enforce the constrained in the $l1$, you will have a variety of 'answers' while the second solution admits only $p$ possible answers (where $p$ is the number of variables) &amp;lt;-&gt; there are more solutions in the $l1$ formulation than in the 'truncation' formulation.&lt;/p&gt;&#xA;" OwnerUserId="603" LastEditorUserId="603" LastEditorDisplayName="" LastEditDate="2010-09-23T23:33:14.197" LastActivityDate="2010-09-23T23:33:14.197" CommentCount="7" />
  <row Id="3006" PostTypeId="1" AcceptedAnswerId="3010" CreationDate="2010-09-23T07:58:18.980" Score="7" ViewCount="198" Body="&lt;p&gt;An anonymous reader posted &lt;a href=&quot;http://jeromyanglim.blogspot.com/2009/11/tips-for-writing-up-research-in.html?showComment=1285227037363#c3803430820186070755&quot; rel=&quot;nofollow&quot;&gt;the following question on my blog&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Context:&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reader wanted to run a factor analysis on scales from a questionnaire - but the data was from paired husbands and wives. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question:&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Can factor analysis be run on dyadic data? If so, how?&lt;/li&gt;&#xA;&lt;li&gt;Would the independence assumption hold for factor analysis?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-09-23T09:54:06.647" LastActivityDate="2010-09-23T09:54:06.647" Title="Factor analysis of dyadic data" Tags="&lt;factor-analysis&gt;&lt;independence&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="3008" PostTypeId="2" ParentId="2988" CreationDate="2010-09-23T08:16:37.353" Score="6" ViewCount="" Body="&lt;p&gt;I usually find it easier and faster to run a simulation. Papers take a long time to read, to understand and finally come to the conclusion that they don't apply in the special case one is interested in.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, I would just pick a number of subjects, simulate the covariate you are interested in (distributed as you believe it will be), simulate good/bad outcomes based on the functional form you posit (threshold effects of the covariate? nonlinearity?) with the minimum (clinically) significant effect size you would like to detect, run the result through your analysis and see whether the effect is found at your alpha. Rerun this 10,000 times and look whether you found the effect in 80% of the simulations (or whatever other power you need). Adjust the number of subjects, repeat until you have a power you are happy with.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This has the advantage of being very general, so you are not confined to a specific functional form or a specific number or distribution of covariates. You can include dropouts, see chl's comment above, either at random or influenced by covariate or outcome. You basically code the analysis you are going to do on the final sample beforehand, which sometimes helps focus my thinking on the study design. And it is easily done in R (vectorize!).&lt;/p&gt;&#xA;" OwnerUserId="1352" LastEditorUserId="1352" LastEditorDisplayName="" LastEditDate="2010-10-02T20:05:22.870" LastActivityDate="2010-10-02T20:05:22.870" CommentCount="5" />
  <row Id="3009" PostTypeId="2" ParentId="3006" CreationDate="2010-09-23T09:07:55.583" Score="6" ViewCount="" Body="&lt;p&gt;Yes, s/he can run a factor analysis on dyadic data. I would start with Kenny et al.'s (2006) &quot;&lt;a href=&quot;http://davidakenny.net/kkc/kkc.htm&quot; rel=&quot;nofollow&quot;&gt;Dyadic Data Analysis&lt;/a&gt;&quot;. It is a great and extremly helpful book! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another option is &quot;Modeling Dyadic and Interdependent Data in the Developmental and Behavioral Sciences&quot; (Card et al. 2008). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(If your anonymous read is able to read German, s/he might be interested in this presentation &quot;&lt;a href=&quot;http://www.mzes.uni-mannheim.de/projekte/pairfam/spring07/Dyadische%20Datenanalyse%20mittels%20SEM.pdf&quot; rel=&quot;nofollow&quot;&gt;Dyadische Datenanalyse: Lineare &#xA;Strukturgleichungsmodelle&lt;/a&gt;&quot;).&lt;/p&gt;&#xA;" OwnerUserId="307" LastEditorUserId="307" LastEditorDisplayName="" LastEditDate="2010-09-23T09:17:29.330" LastActivityDate="2010-09-23T09:17:29.330" CommentCount="3" />
  <row Id="3010" PostTypeId="2" ParentId="3006" CreationDate="2010-09-23T09:34:29.073" Score="7" ViewCount="" Body="&lt;p&gt;Structural equation models are better suited for this kind of data, e.g. by introducing an extra factor for couple which allows to account for the dependence structure (paired responses). David A. Kenny reviewed the main points for &lt;a href=&quot;http://davidakenny.net/dyad.htm&quot; rel=&quot;nofollow&quot;&gt;analysis dyadic data&lt;/a&gt;; although it doesn't focus on questionnaire analysis, it may help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A couple of references :&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Olsen, JA and Kenny, DA (2006). &lt;a href=&quot;http://www.umass.edu/family/pdfs/OlsenKenny.pdf&quot; rel=&quot;nofollow&quot;&gt;Structural Equation Modeling With Interchangeable Dyads&lt;/a&gt;. &lt;em&gt;Psychological Methods&lt;/em&gt;, &lt;em&gt;11(2)&lt;/em&gt;, 127–141.&lt;/li&gt;&#xA;&lt;li&gt;McMahon,, JM, Pouget, ER, and Tortu, S (2006). &lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1550976/&quot; rel=&quot;nofollow&quot;&gt;A guide for multilevel modeling of dyadic data with binary outcomes using SAS PROC NLMIXED&lt;/a&gt;. &lt;em&gt;Comput Stat Data Anal.&lt;/em&gt;, &lt;em&gt;50(12)&lt;/em&gt;, 3663–3680. &lt;/li&gt;&#xA;&lt;li&gt;Thompson, L and Walker, AJ (1982). &lt;a href=&quot;https://mywebspace.wisc.edu/rlbrown3/web/dyad/351453.pdf&quot; rel=&quot;nofollow&quot;&gt;The Dyad as the Unit of Analysis: Conceptual and Methodological Issues&lt;/a&gt;. &lt;em&gt;Journal of Marriage and the Family&lt;/em&gt;, 889-900.&lt;/li&gt;&#xA;&lt;li&gt;Newsom, JT (2002). &lt;a href=&quot;https://mywebspace.wisc.edu/rlbrown3/web/dyad/Newsom2002.pdf&quot; rel=&quot;nofollow&quot;&gt;A multilevel structural equation model for dyadic data&lt;/a&gt;. &lt;em&gt;Structural Equation Modeling&lt;/em&gt;, &lt;em&gt;9(3)&lt;/em&gt;, 441-447.&lt;/li&gt;&#xA;&lt;li&gt;González, J, Tuerlinckx, F, and De Boeck, P (2009). &lt;a href=&quot;http://ppw.kuleuven.be/okp/_pdf/Gonzalez2009ASRIM.pdf&quot; rel=&quot;nofollow&quot;&gt;Analyzing structural relations in multivariate dyadic binary data&lt;/a&gt;. &lt;em&gt;Applied Multivariate Research&lt;/em&gt;, &lt;em&gt;13&lt;/em&gt;, 77-92.&lt;/li&gt;&#xA;&lt;li&gt;Gill, PS (2005). &lt;a href=&quot;http://www.stat.sfu.ca/~tim/papers/dyadic1.pdf&quot; rel=&quot;nofollow&quot;&gt;Bayesian Analysis of Dyadic Data&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;For more thorough description of the models for dyadic data (although not restrained to item analysis), I would suggest&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Kenny, DA, Kashy, DA, and Cook, WL (2006). &lt;em&gt;&lt;a href=&quot;http://davidakenny.net/kkc/kkc.htm&quot; rel=&quot;nofollow&quot;&gt;Dyadic Data Analysis&lt;/a&gt;&lt;/em&gt;. Guilford Press.&lt;/li&gt;&#xA;&lt;li&gt;Card, NA, Selig, JP, and Little, TD (2008). &lt;em&gt;&lt;a href=&quot;http://www.routledgeeducation.com/books/Modeling-Dyadic-and-Interdependent-Data-in-the-Developmental-and-Behavioral-Sciences-isbn9780805859737&quot; rel=&quot;nofollow&quot;&gt;Modeling Dyadic and Interdependent Data in the Developmental and Behavioral Sciences&lt;/a&gt;&lt;/em&gt;. Mahwah, NJ: Lawrence Erlbaum Associates.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-23T09:42:17.033" LastActivityDate="2010-09-23T09:42:17.033" CommentCount="2" />
  <row Id="3011" PostTypeId="2" ParentId="2971" CreationDate="2010-09-23T10:31:04.193" Score="3" ViewCount="" Body="&lt;p&gt;In a general machine-learning view the answer is fairly simple: we want to build model that will have the highest accuracy when predicting new data (unseen during training). Because we cannot directly test this (we don't have data from the future) we do Monte Carlo simulation of such a test -- and this is basically the idea underneath cross validation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There may be some issues about what is accuracy (for instance a business client can state that overshoot costs 5€ per unit and undershoot 0.01€ per unit, so it is better to build a less accurate but more undershooting model), but in general it is fairly intuitive per cent of true answers in classification and widely used explained variance in regression.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-09-23T10:31:04.193" />
  <row Id="3012" PostTypeId="5" CreationDate="2010-09-23T10:59:13.660" Score="0" ViewCount="0" Body="" OwnerUserId="-1" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-23T10:59:13.660" LastActivityDate="2010-09-23T10:59:13.660" />
  <row Id="3013" PostTypeId="4" CreationDate="2010-09-23T10:59:13.660" Score="0" ViewCount="0" Body="Model selection is a problem of judging which model from some set performs best. Popular methods include $R^2$, AIC and BIC criterions, test sets and cross-validation. To some extent, feature selection is a subproblem of model selection. " OwnerUserId="-1" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-23T10:59:13.660" LastActivityDate="2010-09-23T10:59:13.660" />
  <row Id="3014" PostTypeId="5" CreationDate="2010-09-23T11:06:19.013" Score="0" ViewCount="0" Body="" OwnerUserId="-1" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-23T11:06:19.013" LastActivityDate="2010-09-23T11:06:19.013" />
  <row Id="3015" PostTypeId="4" CreationDate="2010-09-23T11:06:19.013" Score="0" ViewCount="0" Body="Clustering is a problem of partitioning data into subsets of objects due to their mutual &quot;similarity&quot;. " OwnerUserId="-1" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-23T11:06:19.013" LastActivityDate="2010-09-23T11:06:19.013" />
  <row Id="3016" PostTypeId="5" CreationDate="2010-09-23T11:14:30.340" Score="0" ViewCount="0" Body="&lt;p&gt;Time series are data observed over time (either in continuous time or at discrete time periods).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Time series analysis includes trend identification, temporal pattern recognition, spectral analysis, and forecasting future values based on the past.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The salient characteristic of methods of time series analysis (as opposed to more general methods to analyze relationships among data) is accounting for the possibility of &lt;em&gt;serial correlation&lt;/em&gt; (also known as temporal correlation) among the data.  Positive serial correlation means successive observations in time tend to be close to one another, whereas negative serial correlation means successive observations tend to oscillate between extremes.  Time series analysis also differs from analyses of more general stochastic processes by focusing on the &lt;em&gt;inherent direction&lt;/em&gt; of time, creating a potential asymmetry between past and future.&lt;/p&gt;&#xA;" OwnerUserId="-1" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-11T19:39:55.603" LastActivityDate="2011-03-11T19:39:55.603" />
  <row Id="3017" PostTypeId="4" CreationDate="2010-09-23T11:14:30.340" Score="0" ViewCount="0" Body="Time series are data observed over time (either in continuous time or at discrete time periods)." OwnerUserId="-1" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-11T19:39:55.637" LastActivityDate="2011-03-11T19:39:55.637" />
  <row Id="3018" PostTypeId="5" CreationDate="2010-09-23T11:28:36.817" Score="0" ViewCount="0" Body="" OwnerUserId="-1" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-23T11:28:36.817" LastActivityDate="2010-09-23T11:28:36.817" />
  <row Id="3019" PostTypeId="4" CreationDate="2010-09-23T11:28:36.817" Score="0" ViewCount="0" Body="Hypothesis testing is a problem of assessing a chance that some regularity in the provided data supports given hypothesis rather than being an effect of random fluctuations or some other process described by alternative hypothesis." OwnerUserId="-1" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-23T11:28:36.817" LastActivityDate="2010-09-23T11:28:36.817" />
  <row Id="3020" PostTypeId="2" ParentId="2990" CreationDate="2010-09-23T13:57:26.397" Score="4" ViewCount="" Body="&lt;p&gt;As the many different responses in this thread already attest, it would be helpful to address some preliminary issues before asking this question.  The important ones include:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;The study question itself ought to be well defined.  Exactly what is a &quot;reputation effect&quot;?  (I have followed all the referenced threads, and have quickly reviewed them again, but can find no description or definition of this term.  Am I the only one who doesn't know precisely what it means?)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Guidance from previous work can be helpful.  E.g., does anyone know of any specific examples of &quot;reputation effects&quot;?  Is there some background literature describing a theory of such effects?  If so, what does it suggest about effect sizes, important covariates, explanatory factors, appropriate ways to measure them, and so on?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;What do we hope the scope of the study will be?  Do we aim do draw conclusions about all &quot;SE Communities&quot; or only about the ones whose data will be studied?  Is there any intent or hope of drawing more general conclusions about &quot;reputation&quot; itself?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Because this appears to be research &lt;em&gt;ab initio&lt;/em&gt;--that is, not building on or relying on previous studies--we will need a preliminary EDA phase.  It may be difficult to recommend a good sampling plan until some EDA is accomplished.  Indeed, a minimum of three phases of sampling seems to be required: some sampling--not necessarily even random--to conduct EDA, followed by more formal randomized sampling to obtain a small representative subset of data whose aim is to estimate effect sizes and variances well enough to determine how much more data will be needed to test the study questions to given degrees of confidence.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;...which implies we need to establish some targets for confidence in the study results.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Those targets may have to be adjusted to account for multiple comparisons.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;We cannot do any such adjusting until a set of specific hypotheses has been established, which is one of the aims of the EDA.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Can we be more specific about the actual cost (or its equivalent in volunteer effort) of sampling the data?  This would help us make tradeoffs between the desire for high confidence (achieved by larger sample sizes) and the pain of gathering the data needed to achieve such confidence.  If necessary, addressing this question can be delayed until after the EDA or preliminary sampling phase.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Once these questions have been answered it should be straightforward do specify a sampling protocol.  I don't imagine these questions will be difficult to answer, but the answers ought to be given and documented somewhere.  Such formalities seem especially important in light of the collaborative, distributed nature of the proposed research, so that a common understanding is established (and not just assumed).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Apropos this last point, it's starting to look like developing a study protocol and all its documentation solely as a set of threads on this site will quickly become unwieldy.  Who wants to wade through a long set of links to questions/comments/answers in order to look up basic information?  It would be &lt;em&gt;very&lt;/em&gt; useful to have a central location in which a small number of key documents are dynamically maintained.  These would eventually include a statement of the study goals (including background information), the study protocol, a data dictionary, and the various datasets themselves.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-09-23T13:57:26.397" CommunityOwnedDate="2010-09-23T13:57:26.397" CommentCount="4" />
  <row Id="3021" PostTypeId="2" ParentId="2990" CreationDate="2010-09-23T15:30:32.440" Score="5" ViewCount="" Body="&lt;p&gt;We need a few volunteers to get this project moving into high-gear. Building on @whuber's excellent &lt;a href=&quot;http://stats.stackexchange.com/questions/2990/poll-on-data-collection-issues-for-our-first-polystats-project/3020#3020&quot;&gt;answer&lt;/a&gt; to this question I have identified a few tasks that we need to do. I have divided up the tasks in such a way that they should not take up too much of any one person's time. Please edit this answer and write your name against a task if you wish to do that task. Please update this answer also with relevant pointers/answers to each task under its respective bullet point. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Setting up a wiki - Shane has volunteered to set this up. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The wiki link is at: &lt;a href=&quot;http://www.statalgo.com/polystatsproject/wiki/&quot; rel=&quot;nofollow&quot;&gt;http://www.statalgo.com/polystatsproject/wiki/&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;We need a definition for reputation as used by various fields and what it means in the context of Q&amp;amp;A communities. Some preliminary definitions from the top of my head:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Reputation = &quot;Expertise in a specific domain&quot;&lt;/li&gt;&#xA;&lt;li&gt;Reputation = &quot;Trustworthiness&quot;  &lt;/li&gt;&#xA;&lt;li&gt;Wiki has a section on '&lt;a href=&quot;http://en.wikipedia.org/wiki/Reputation#Online_Reputation&quot; rel=&quot;nofollow&quot;&gt;Online Reputation&lt;/a&gt;'  &lt;/li&gt;&#xA;&lt;li&gt;As per this sites &lt;a href=&quot;http://stats.stackexchange.com/faq&quot;&gt;FAQ&lt;/a&gt; &quot;Reputation is a rough measurement of how much the community trusts you.&quot;&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Can someone find out if there is a formal definition of reputation effect? (Srikant)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(a) Reputation Effect as Analogy to &lt;a href=&quot;http://en.wikipedia.org/wiki/Network_effect&quot; rel=&quot;nofollow&quot;&gt;Network effects&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The wiki states: &quot;In economics and business, a network effect (also called network  externality) is the effect that one user of a good or service has on the value of that product to other people. &lt;strong&gt;When network effect is present, the value of a product or service increases as more people use it.&lt;/strong&gt;[Emphasis added]&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, in our context, reputation effect would be: &quot;When reputation effects are present the reputation of a user increases faster the higher his/her reputation is.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(b) Another definition would be what?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Changing the unit of analysis to answers/questions as opposed to people) A reputation effect would be: &quot;When an individual has a higher reputation, their questions/answers receive more up-votes, ceteris paribus to answers by individuals with lower reputation&quot;.&#xA;Added by Andy W&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Can a few people dig up at a few references relevant to the project? &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Andy has set up a link at: &lt;a href=&quot;http://www.citeulike.org/groupfunc/14031/home&quot; rel=&quot;nofollow&quot;&gt;http://www.citeulike.org/groupfunc/14031/home&lt;/a&gt; which will have a list of references for our project.&lt;/li&gt;&#xA;&lt;li&gt;Another reference: &lt;a href=&quot;http://linkinghub.elsevier.com/retrieve/pii/S0167923605000849&quot; rel=&quot;nofollow&quot;&gt;A survey of trust and reputation systems for online service provision&lt;/a&gt;  &lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;What theoretical framework(s) would be appropriate for this project? &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Anything else we need to do please add as additional items.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" LastActivityDate="2010-09-24T17:10:13.087" CommunityOwnedDate="2010-09-23T15:30:32.440" CommentCount="5" />
  <row Id="3022" PostTypeId="2" ParentId="3001" CreationDate="2010-09-23T18:45:20.550" Score="1" ViewCount="" Body="&lt;p&gt;Yes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you say that &quot;X is the 16% quantile&quot;, what it means is that 16% of the sample have a lower value than X. The log of any number smaller than X is smaller than log(X) and the log of any number greater than X is greater than log(X), so the ordering is not changed.&lt;/p&gt;&#xA;" OwnerUserId="666" LastActivityDate="2010-09-23T18:45:20.550" />
  <row Id="3023" PostTypeId="2" ParentId="2686" CreationDate="2010-09-23T19:16:36.727" Score="1" ViewCount="" Body="&lt;p&gt;A simple solution that does not require the acquisition of specialized knowledge is to use &lt;a href=&quot;http://en.wikipedia.org/wiki/Control_chart&quot; rel=&quot;nofollow&quot;&gt;control charts&lt;/a&gt;. They're ridiculously easy to create and make it easy to tell special cause variation (such as when you are out of town) from common cause variation (such as when you have an actual low-productivity month), which seems to be the kind of information you want.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;They also preserve the data. Since you say you'll use the charts for many different purposes, I advise against performing any transformations in the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0945320531&quot; rel=&quot;nofollow&quot;&gt;Here is a gentle introduction&lt;/a&gt;. If you decide that you like control charts, you may want to &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0945320612&quot; rel=&quot;nofollow&quot;&gt;dive deeper&lt;/a&gt; into the subject. The benefits to your business will be huge. Control charts are reputed to have been a major contributor to &lt;a href=&quot;http://en.wikipedia.org/wiki/W._Edwards_Deming&quot; rel=&quot;nofollow&quot;&gt;the post-war Japanese economic boom&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is even an &lt;a href=&quot;http://cran.r-project.org/web/packages/qcc/&quot; rel=&quot;nofollow&quot;&gt;R package&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="666" LastActivityDate="2010-09-23T19:16:36.727" />
  <row Id="3024" PostTypeId="1" AcceptedAnswerId="3027" CreationDate="2010-09-23T19:38:40.190" Score="10" ViewCount="744" Body="&lt;p&gt;I understand that for certain datasets such as voting it performs better. Why is Poisson regression used over ordinary linear regression or logistic regression? What is the mathematical motivation for it? &lt;/p&gt;&#xA;" OwnerUserId="1392" LastEditorUserId="71" LastEditorDisplayName="" LastEditDate="2010-09-23T21:08:36.537" LastActivityDate="2010-09-23T23:28:22.660" Title="Why is Poisson regression used for count data?" Tags="&lt;regression&gt;&lt;poisson&gt;&lt;count-data&gt;" AnswerCount="5" FavoriteCount="2" />
  <row Id="3025" PostTypeId="2" ParentId="2971" CreationDate="2010-09-23T20:19:30.067" Score="2" ViewCount="" Body="&lt;p&gt;Great discussion here, but I think of cross-validation in a different way from the answers thus far (mbq and I are on the same page I think). So, I'll put in my two cents at the risk of muddying the waters...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cross-validation is a statistical technique for assessing the variability and bias, due to sampling error, in a model's ability to fit and predict data. Thus, &quot;best&quot; would be the model which provides the lowest generalization error, which would be in units of variability and bias. Techniques such as Bayesian and Bootstrap Model Averaging can be used to update a model in an algorithmic way based upon results from the cross validation effort.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.faqs.org/faqs/ai-faq/neural-nets/part3/section-12.html&quot; rel=&quot;nofollow&quot;&gt;This FAQ&lt;/a&gt; provides good information for more context of what informs my opinion.&lt;/p&gt;&#xA;" OwnerUserId="1080" LastActivityDate="2010-09-23T20:19:30.067" />
  <row Id="3026" PostTypeId="2" ParentId="3024" CreationDate="2010-09-23T20:28:48.930" Score="1" ViewCount="" Body="&lt;p&gt;My understanding is primarily because counts are always positive and discrete, the Poisson can summarize such data with one parameter. The main catch being that the variance equals the mean.&lt;/p&gt;&#xA;" LastActivityDate="2010-09-23T20:28:48.930" />
  <row Id="3027" PostTypeId="2" ParentId="3024" CreationDate="2010-09-23T20:42:45.613" Score="11" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Poisson_distribution&quot; rel=&quot;nofollow&quot;&gt;Poisson distributed&lt;/a&gt; data is intrinsically integer-valued, which makes sense for count data. Ordinary Least Squares (OLS, which you call &quot;linear regression&quot;) assumes that true values are &lt;a href=&quot;http://en.wikipedia.org/wiki/Normal_distribution&quot; rel=&quot;nofollow&quot;&gt;normally distributed&lt;/a&gt; around the expected value and can take any real value, positive or negative, integer or fractional, whatever. Finally, &lt;a href=&quot;http://en.wikipedia.org/wiki/Logistic_regression&quot; rel=&quot;nofollow&quot;&gt;logistic regression&lt;/a&gt; only works for data that is 0-1-valued (TRUE-FALSE-valued), like &quot;has a disease&quot; versus &quot;doesn't have the disease&quot;. Thus, the Poisson distribution makes the most sense for count data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That said, a normal distribution is often a rather good approximation to a Poisson one for data with a mean above 30 or so. And in a regression framework, where you have predictors influencing the count, an OLS with its normal distribution may be easier to fit and would actually be more general, since the Poisson distribution and regression assume that the mean and the variance are equal, while OLS can deal with unequal means and variances - for a count data model with different means and variances, one could use a &lt;a href=&quot;http://en.wikipedia.org/wiki/Negative_binomial_distribution&quot; rel=&quot;nofollow&quot;&gt;negative binomial distribution&lt;/a&gt;, for instance.&lt;/p&gt;&#xA;" OwnerUserId="1352" LastActivityDate="2010-09-23T20:42:45.613" />
  <row Id="3028" PostTypeId="2" ParentId="3024" CreationDate="2010-09-23T20:52:15.760" Score="8" ViewCount="" Body="&lt;p&gt;Essentially, it's because linear and logistic regression make the wrong kinds of assumptions about what count outcomes look like.  Imagine your model as a very stupid robot that will relentlessly follow your orders, no matter how nonsensical those orders are; it completely lacks the ability to evaluate what you tell it.  If you tell your robot that something like votes is distributed continuously from negative infinity to infinity, that's what it believes votes are like, and it might give you nonsensical predictions (Ross Perot will receive -10.469 votes in the upcoming election).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Conversely, the Poisson distribution is discrete and positive (or zero... zero counts as positive, yes?).  At a very minimum, this will force your robot to give you answers that could actually happen in real life.  They may or may not be &lt;em&gt;good&lt;/em&gt; answers, but they will at least be drawn from the possible set of &quot;number of votes cast&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, the Poisson has its own problems: it assumes that the mean of the vote count variable will also be the same as its variance.  I don't know if I've ever actually seen a non-contrived example where this was true.  Fortunately, bright people have come up with other distributions that are also positive and discrete, but that add parameters to allow the variance to, er, vary (e.g., negative binomial regression).&lt;/p&gt;&#xA;" OwnerUserId="71" LastActivityDate="2010-09-23T20:52:15.760" />
  <row Id="3029" PostTypeId="2" ParentId="3024" CreationDate="2010-09-23T23:10:50.283" Score="0" ViewCount="" Body="&lt;p&gt;Others have basically said the same thing I'm going to but I thought I'd add my take on it.  It depends on what you're doing exactly but a lot of times we like to conceptualize the problem/data at hand.  This is a slightly different approach compared to just building a model that predicts pretty well.  If we are trying to conceptualize what's going on it makes sense to model count data using a non-negative distribution that only puts mass at integer values.  We also have many results that essentially boil down to saying that under certain conditions count data really &lt;em&gt;is&lt;/em&gt; distributed as a poisson.  So if our goal is to conceptualize the problem it really makes sense to use a poisson as the response variable.  Others have pointed out other reasons why it's a good idea but if you're really trying to conceptualize the problem and really understand how data that you see could be generated then using a poisson regression makes a lot of sense in some situations.  &lt;/p&gt;&#xA;" OwnerUserId="1028" LastActivityDate="2010-09-23T23:10:50.283" />
  <row Id="3030" PostTypeId="2" ParentId="3024" CreationDate="2010-09-23T23:28:22.660" Score="2" ViewCount="" Body="&lt;p&gt;Mathematically if you start with the simple assumption that the probability of an event occurring in a defined interval $T = 1$ is $\lambda$ you can show the expected number of events in the interval $T = t$ is is $\lambda.t$, the variance is also $\lambda.t$ and the &lt;a href=&quot;http://www.umass.edu/wsp/statistics/lessons/poisson/derivation.html&quot; rel=&quot;nofollow&quot;&gt;probability distribution &lt;/a&gt; is&lt;br&gt;&#xA;$$p(N=n) =  \frac{(\lambda.t)^{n}e^{-\lambda.t}}{n!}$$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Via this and the &lt;a href=&quot;http://en.wikipedia.org/wiki/Poisson_distribution#Maximum_likelihood&quot; rel=&quot;nofollow&quot;&gt;maximum likelihood method&lt;/a&gt; &amp;amp; generalised linear models (or some other method) you arrive at &lt;a href=&quot;http://en.wikipedia.org/wiki/Poisson_regression&quot; rel=&quot;nofollow&quot;&gt;Poisson regression&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In simple terms Poisson Regression is the model that fits the assumptions of the underlying random process generating a small number of events at a rate (i.e. number per unit time) determined by other variables in the model.&lt;/p&gt;&#xA;" OwnerUserId="521" LastActivityDate="2010-09-23T23:28:22.660" />
  <row Id="3031" PostTypeId="1" CreationDate="2010-09-24T02:56:56.010" Score="2" ViewCount="114" Body="&lt;p&gt;I just loaded a csv file in R. When I ran the &lt;code&gt;summary&lt;/code&gt; command for one of the columns, I got the following: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; Error: unexpected symbol in &quot;summary k_low&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm pretty sure I know what the 'unexpected symbol' is: for observations in which we had no reliable data for the variable &lt;code&gt;k_low&lt;/code&gt;, we simply entered a period, or a &lt;code&gt;.&lt;/code&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a way to escape these observations in R? My boss uses STATA, which apparently escapes periods automatically, but I don't have STATA on my personal computer (and would prefer to use R in any case). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So basically: Is there a way to get R to bypass any 'symbols' and return a summary of only those observations for which numerical data was entered? &lt;/p&gt;&#xA;" OwnerUserId="1410" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-24T13:58:47.937" LastActivityDate="2010-09-24T13:58:47.937" Title="How to escape symbolic value in R" Tags="&lt;r&gt;" AnswerCount="4" />
  <row Id="3032" PostTypeId="2" ParentId="3031" CreationDate="2010-09-24T03:03:32.427" Score="5" ViewCount="" Body="&lt;p&gt;It looks like you just left out the parentheses. Try&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;summary(k_low)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-09-24T03:03:32.427" CommentCount="1" />
  <row Id="3034" PostTypeId="2" ParentId="3031" CreationDate="2010-09-24T03:13:31.783" Score="4" ViewCount="" Body="&lt;p&gt;Looks like Rob got it right but I'll illustrate how to fix the period problem.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; testdata &amp;lt;- c(1, 2, 3, &quot;.&quot;)&#xA;&amp;gt; testdata&#xA;[1] &quot;1&quot; &quot;2&quot; &quot;3&quot; &quot;.&quot;&#xA;&amp;gt; summary(testdata)&#xA;   Length     Class      Mode &#xA;        4 character character &#xA;&amp;gt; #That's not what we want....&#xA;&amp;gt; cleandata &amp;lt;- as.numeric(testdata)&#xA;Warning message:&#xA;NAs introduced by coercion &#xA;&amp;gt; cleandata&#xA;[1]  1  2  3 NA&#xA;&amp;gt; summary(cleandata)&#xA;   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    NA's &#xA;    1.0     1.5     2.0     2.0     2.5     3.0     1.0 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Notice that because the testdata vector had a period in it it converted everything to characters.  When we tell it to convert to numeric it doesn't know what to do with the periods so it automatically converts it to NA which is nice for what we want to do.  Then when you call summary on it it ignores the NAs for the summary but tells you how many NAs there are in the data.&lt;/p&gt;&#xA;" OwnerUserId="1028" LastActivityDate="2010-09-24T03:13:31.783" CommentCount="1" />
  <row Id="3035" PostTypeId="2" ParentId="423" CreationDate="2010-09-24T04:09:04.750" Score="19" ViewCount="" Body="&lt;p&gt;I just came across this and loved it:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/Vdqwt.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="253" LastActivityDate="2010-09-24T04:09:04.750" CommunityOwnedDate="2010-09-24T04:09:04.750" />
  <row Id="3036" PostTypeId="2" ParentId="423" CreationDate="2010-09-24T04:13:25.630" Score="14" ViewCount="" Body="&lt;p&gt;this too:&#xA;&lt;img src=&quot;http://i.imgur.com/6Z3QQ.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="253" LastActivityDate="2010-09-24T04:13:25.630" CommunityOwnedDate="2010-09-24T04:13:25.630" />
  <row Id="3037" PostTypeId="2" ParentId="3031" CreationDate="2010-09-24T04:16:19.980" Score="0" ViewCount="" Body="&lt;p&gt;Another point is that when you read in the data, it sound as if your &quot;.&quot; is really a missing value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So what you might wish to do when reading the data, is something like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;k_low &amp;lt;- read.date(...,     na.strings = &quot;.&quot;)&lt;/p&gt;&#xA;" OwnerUserId="253" LastActivityDate="2010-09-24T04:16:19.980" />
  <row Id="3038" PostTypeId="1" CreationDate="2010-09-24T05:24:00.240" Score="4" ViewCount="289" Body="&lt;p&gt;Imagine you have a study with two groups (e.g., males and females) looking at a numeric dependent variable (e.g., intelligence test scores) and you have the hypothesis that there are no group differences.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What is a good way to test whether there are no group differences?&lt;/li&gt;&#xA;&lt;li&gt;How would you determine the sample size needed to adequately test for no group differences?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Initial Thoughts:&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;It would not be enough to do a standard t-test because a failure to reject the null hypothesis does not mean that the parameter of interest is equal or close to zero; this is particularly the case with small samples.&lt;/li&gt;&#xA;&lt;li&gt;I could look at the 95% confidence interval and check that all values are within a sufficiently small range; perhaps plus or minus 0.3 standard deviations.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-09-24T10:43:06.417" LastActivityDate="2010-10-01T23:13:10.210" Title="Testing hypothesis of no group differences" Tags="&lt;hypothesis-testing&gt;&lt;t-test&gt;&lt;equivalence&gt;" AnswerCount="6" CommentCount="4" FavoriteCount="1" />
  <row Id="3039" PostTypeId="2" ParentId="3031" CreationDate="2010-09-24T05:37:32.793" Score="1" ViewCount="" Body="&lt;p&gt;When using &lt;a href=&quot;http://127.0.0.1:20298/library/utils/html/read.table.html&quot; rel=&quot;nofollow&quot;&gt;read.table, read.csv or read.delim&lt;/a&gt; use:  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;read.table(file,..., na.strings = &quot;.&quot;, ...)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;na.strings -    a character vector of strings which are to be interpreted as NA             values.&lt;br&gt;&#xA;                Blank fields are also considered to be missing values in logical, &#xA;                integer, numeric and complex fields.&lt;/p&gt;&#xA;" OwnerUserId="521" LastActivityDate="2010-09-24T05:37:32.793" />
  <row Id="3040" PostTypeId="2" ParentId="3038" CreationDate="2010-09-24T05:50:39.257" Score="3" ViewCount="" Body="&lt;p&gt;I think you are asking about &lt;a href=&quot;http://www.graphpad.com/library/BiostatsSpecial/article_182.htm&quot; rel=&quot;nofollow&quot;&gt;testing for equivalence&lt;/a&gt;. Essentially you need to decide how large a difference is acceptable for you to still conclude that the two groups are effectively equivalent. That decision defines the 95% (or other) confidence interval limits, and sample size calculations are made on this basis. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1584881607&quot; rel=&quot;nofollow&quot;&gt;whole book&lt;/a&gt; on the topic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A very common clinical &quot;equivalent&quot; of equivalence tests is a &lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2701110/pdf/nihms112245.pdf&quot; rel=&quot;nofollow&quot;&gt;non-inferiority test/ trial&lt;/a&gt;. In this case you &quot;prefer&quot; one group over the other (an established treatment) and design your test to show that the new treatment is not inferior to the established treatment at some level of statistical evidence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think I need to credit &lt;a href=&quot;http://stats.stackexchange.com/users/25/harvey-motulsky&quot;&gt;Harvey Motulsky&lt;/a&gt; for the &lt;a href=&quot;http://www.graphpad.com/welcome.htm&quot; rel=&quot;nofollow&quot;&gt;GraphPad.com&lt;/a&gt; site (under &lt;a href=&quot;http://www.graphpad.com/index.cfm?cmd=library.index&quot; rel=&quot;nofollow&quot;&gt;&quot;Library&quot;&lt;/a&gt;).&lt;/p&gt;&#xA;" OwnerUserId="521" LastEditorUserId="521" LastEditorDisplayName="" LastEditDate="2010-09-24T05:57:50.337" LastActivityDate="2010-09-24T05:57:50.337" />
  <row Id="3041" PostTypeId="1" AcceptedAnswerId="3042" CreationDate="2010-09-24T05:58:56.967" Score="2" ViewCount="114" Body="&lt;p&gt;I generated a file which contains a logical value either a &quot;TRUE&quot; or a &quot;FALSE&quot; on each line. Now I would like to read the logical data from the file into R. However the data that are read in are of mode &quot;character&quot; not logical values. I was wondering how to read the data as logical values from the file.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My R code is&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cat(FALSE,&quot;\n&quot;, file=&quot;1.txt&quot;, append=FALSE);&#xA;for (i in 2:5) cat(TRUE,&quot;\n&quot;,file=&quot;1.txt&quot;, append=TRUE);&#xA;a=scan(file=&quot;1.txt&quot;, what=&quot;logical&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The output is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; mode(a)&#xA;1] &quot;character&quot;&#xA;&amp;gt; mode(a[1])&#xA;[1] &quot;character&quot;&#xA;&amp;gt; a[1]&#xA;[1] &quot;FALSE&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want a[1] to be logical value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks and regards!&lt;/p&gt;&#xA;" OwnerUserId="1005" LastActivityDate="2010-09-24T06:45:41.737" Title="How to read logical data from a file in R" Tags="&lt;r&gt;" AnswerCount="2" />
  <row Id="3042" PostTypeId="2" ParentId="3041" CreationDate="2010-09-24T06:13:13.533" Score="3" ViewCount="" Body="&lt;p&gt;Instead of:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;a=scan(file=&quot;1.txt&quot;, what=&quot;logical&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;try:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;a = scan(file=&quot;1.txt&quot;, what=logical(0))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;See &lt;code&gt;?scan&lt;/code&gt; for documentation.&lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-09-24T06:13:13.533" CommentCount="2" />
  <row Id="3043" PostTypeId="2" ParentId="3041" CreationDate="2010-09-24T06:32:38.317" Score="2" ViewCount="" Body="&lt;pre&gt;&lt;code&gt;a&#xA;[1] &quot;FALSE&quot; &quot;TRUE&quot;  &quot;TRUE&quot;  &quot;TRUE&quot;  &quot;TRUE&quot; &#xA;a &amp;lt;- as.logical(a)&#xA;a&#xA;[1] FALSE  TRUE  TRUE  TRUE  TRUE&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="521" LastEditorUserId="521" LastEditorDisplayName="" LastEditDate="2010-09-24T06:45:41.737" LastActivityDate="2010-09-24T06:45:41.737" />
  <row Id="3044" PostTypeId="2" ParentId="3038" CreationDate="2010-09-24T08:25:03.020" Score="3" ViewCount="" Body="&lt;p&gt;Following Thylacoleo's answer, I did a little research.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;a href=&quot;http://cran.r-project.org/web/packages/equivalence/equivalence.pdf&quot; rel=&quot;nofollow&quot;&gt;equivalence&lt;/a&gt; package in R has the &lt;code&gt;tost()&lt;/code&gt; function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See Robinson and Frose (2004) &quot;&lt;a href=&quot;http://research.eeescience.utoledo.edu/lees/papers_PDF/Robinson_2004_EcolModell.pdf&quot; rel=&quot;nofollow&quot;&gt;Model validation using equivalence tests&lt;/a&gt;&quot; for more info.&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-09-24T08:25:03.020" CommentCount="1" />
  <row Id="3045" PostTypeId="2" ParentId="2746" CreationDate="2010-09-24T09:19:30.617" Score="0" ViewCount="" Body="&lt;p&gt;A cheap and cheerful approach I've used for testing is to generate m N(0,1) n-vectors V[k] and then use P = d*I + Sum{ V[k]*V[k]'} as an nxn psd matrix. With m &amp;lt; n this will be singular for d=0, and for small d will have high condition number.&lt;/p&gt;&#xA;" OwnerUserId="1412" LastActivityDate="2010-09-24T09:19:30.617" CommentCount="3" />
  <row Id="3046" PostTypeId="2" ParentId="3038" CreationDate="2010-09-24T09:43:08.590" Score="3" ViewCount="" Body="&lt;p&gt;In the medical sciences, it is preferable to use a confidence interval approach as opposed to two one-sided tests (tost). I also recommend graphing the point estimates, CIs, and a priori-determined equivalence margins to make things very clear.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your question would likely be addressed by such an approach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The CONSORT guidelines for non-inferiority/equivalence studies are quite useful in this regard.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See &lt;em&gt;Piaggio G, Elbourne DR, Altman DG, Pocock SJ, Evans SJ, and CONSORT Group. Reporting of noninferiority and equivalence randomized trials: an extension of the CONSORT statement. JAMA. 2006, Mar 8;295(10):1152-60.&lt;/em&gt; &lt;a href=&quot;http://jama.ama-assn.org/cgi/content/full/295/10/1152&quot; rel=&quot;nofollow&quot;&gt;(Link to full text.)&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="561" LastActivityDate="2010-09-24T09:43:08.590" />
  <row Id="3047" PostTypeId="2" ParentId="3038" CreationDate="2010-09-24T11:39:12.360" Score="2" ViewCount="" Body="&lt;p&gt;Besides the already mentioned possibility of some kind of &lt;strong&gt;equivalence test&lt;/strong&gt;, of which most of them, to the best of my knowledge, are mostly routed in the good old frequentist tradition, there is the possibility of conducting tests which really provide a quantification of evidence in favor of a null-hyptheses, namely &lt;strong&gt;bayesian tests&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An implementation of a bayesian t-test can be found here:&#xA;Wetzels, R., Raaijmakers, J. G. W., Jakab, E., &amp;amp; Wagenmakers, E.-J. (2009). &lt;a href=&quot;http://www.ejwagenmakers.com/2009/WetzelsEtAl2009Ttest.pdf&quot; rel=&quot;nofollow&quot;&gt;How to quantify support for and against the null hypothesis: A flexible WinBUGS implementation of a default Bayesian t-test.&lt;/a&gt; Psychonomic Bulletin &amp;amp; Review, 16, 752-760.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is also a tutorial on how to do all this in R:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.ruudwetzels.com/index.php?src=SDtest&quot; rel=&quot;nofollow&quot;&gt;http://www.ruudwetzels.com/index.php?src=SDtest&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;All props for this answer should go to my colleague David Kellen. I stole his answer from &lt;a href=&quot;http://stats.stackexchange.com/questions/305/when-conducting-a-t-test-why-would-one-prefer-to-assume-or-test-for-equal-varia/804#804&quot;&gt;this question&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="442" LastActivityDate="2010-09-24T11:39:12.360" CommentCount="1" />
  <row Id="3048" PostTypeId="1" AcceptedAnswerId="3066" CreationDate="2010-09-24T11:44:24.637" Score="16" ViewCount="400" Body="&lt;p&gt;Hi everyone,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a matrix where a(i,j) tells me how many times individual i viewed page j. There are 27K individuals and 95K pages. I would like to have a handful of &quot;dimensions&quot; or &quot;aspects&quot; in the space of pages which would correspond to sets of pages which are often viewed together. My ultimate goal is to then be able to compute how often individual i has viewed pages that fall in dimension 1, dimension 2, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have read the R documentation on &lt;a href=&quot;http://en.wikibooks.org/wiki/Data_Mining_Algorithms_In_R/Dimensionality_Reduction/Principal_Component_Analysis&quot; rel=&quot;nofollow&quot;&gt;principal component analysis&lt;/a&gt; and &lt;a href=&quot;http://en.wikibooks.org/wiki/Data_Mining_Algorithms_In_R/Dimensionality_Reduction/Singular_Value_Decomposition&quot; rel=&quot;nofollow&quot;&gt;single value decomposition&lt;/a&gt; and have executed these commands, but I am unsure how to proceed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I use dimensionality reduction to do this? Or is this really a clustering problem and I should instead look into clustering algorithms?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks for any insight&#xA;~l&lt;/p&gt;&#xA;" OwnerUserId="1007" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-24T13:19:19.210" LastActivityDate="2010-09-28T20:24:47.093" Title="How to do dimensionality reduction in R" Tags="&lt;r&gt;&lt;clustering&gt;&lt;dimensionality-reduction&gt;" AnswerCount="4" FavoriteCount="5" />
  <row Id="3049" PostTypeId="2" ParentId="3038" CreationDate="2010-09-24T11:50:32.963" Score="3" ViewCount="" Body="&lt;p&gt;There are a few papers I know of that could be helpful to you:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tryon, W. W. (2001). &lt;a href=&quot;http://dx.doi.org/10.1037//1082-989X.6.4.371&quot; rel=&quot;nofollow&quot;&gt;Evaluating statistical difference, equivalence, and indeterminacy using inferential confidence intervals: An integrated alternative method of conducting null hypothesis statistical tests.&lt;/a&gt; Psychological Methods, 6, 371-386.  (&lt;a href=&quot;http://www.unt.edu/rss/class/mike/5030/articles/tryonICI.pdf&quot; rel=&quot;nofollow&quot;&gt;FREE PDF&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And a correction:&lt;br&gt;&#xA;Tryon, W. W., &amp;amp; Lewis, C. (2008). &lt;a href=&quot;http://psycnet.apa.org/doi/10.1037/a0013158&quot; rel=&quot;nofollow&quot;&gt;An Inferential Confidence Interval Method of Establishing Statistical Equivalence That Corrects Tryon’s (2001) Reduction Factor.&lt;/a&gt; Psychological Methods, 13,  272-278. (&lt;a href=&quot;http://www.fordham.edu/academics/programs_at_fordham_/psychology/tryon/pdf/Tryon_%26_Lewis_2008_An_Inferential_Confidence_Interval_Method.pdf&quot; rel=&quot;nofollow&quot;&gt;FREE PDF&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furthermore:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Seaman, M. A. &amp;amp; Serlin, R. C. (1998). E&lt;a href=&quot;http://psycnet.apa.org/doi/10.1037/1082-989X.3.4.403&quot; rel=&quot;nofollow&quot;&gt;quivalence confidence intervals for two-group comparisons of means&lt;/a&gt;. Psychological Methods, Vol 3(4), 403-411.&lt;/p&gt;&#xA;" OwnerUserId="442" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-09-24T11:57:15.200" LastActivityDate="2010-09-24T11:57:15.200" />
  <row Id="3050" PostTypeId="2" ParentId="3048" CreationDate="2010-09-24T12:37:53.703" Score="3" ViewCount="" Body="&lt;p&gt;It is certainly a clustering problem. Check out Rs &lt;code&gt;cluster&lt;/code&gt; package to get an overview of algorithm options (&lt;code&gt;pam&lt;/code&gt; and &lt;code&gt;agnes&lt;/code&gt; are the best options to start; they represent two main streams in clustering -- &lt;a href=&quot;http://en.wikipedia.org/wiki/K-means_clustering&quot; rel=&quot;nofollow&quot;&gt;centroids&lt;/a&gt; and &lt;a href=&quot;http://en.wikipedia.org/wiki/Hierarchical_clustering&quot; rel=&quot;nofollow&quot;&gt;hierarchical&lt;/a&gt;).&lt;br&gt;&#xA;The main problem to use clustering on your data is to define a good similarity measure between pages; simple one is to use Manhattan distance; a bit more complex to count the number of common viewers and normalize it with, let's say, mean of number of viewers of the first and second page -- this should silence popularity effects.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: Ok, now I've saw the data size... it will probably make R explode, since it needs one triangle of $(\text{number of pages})\times(\text{number of pages})$ matrix to store distances. Check out this &lt;a href=&quot;http://www.stats.ox.ac.uk/~mercer/documents/Transfer.pdf&quot; rel=&quot;nofollow&quot;&gt;report&lt;/a&gt; for possible solutions.&lt;/p&gt;&#xA;" OwnerUserId="88" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-24T12:57:17.590" LastActivityDate="2010-09-24T12:57:17.590" />
  <row Id="3051" PostTypeId="1" AcceptedAnswerId="3054" CreationDate="2010-09-24T14:41:31.997" Score="3" ViewCount="652" Body="&lt;p&gt;I have a vector of values that I would like to report the average in windows along a smaller slide.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, for a vector of the following values:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;4, 5, 7, 3, 9, 8&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;A window size of 3 and a slide of 2 would do the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(4+5+7)/3 = 5.33&#xA;(7+3+9)/3 = 6.33&#xA;(9+8)/3 = 5.67&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And return a vector of these values:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;5.33, 6.33, 5.67&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is there a simple function that will do this for me? If it also returned the indices of the window starts that would be an added bonus. In this example that would be 1,3,5&lt;/p&gt;&#xA;" OwnerUserId="1024" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-24T17:17:56.337" LastActivityDate="2010-10-21T09:17:02.463" Title="Mean of a sliding window in R" Tags="&lt;r&gt;" AnswerCount="6" CommentCount="4" FavoriteCount="1" />
  <row Id="3052" PostTypeId="1" AcceptedAnswerId="3065" CreationDate="2010-09-24T15:09:52.740" Score="9" ViewCount="270" Body="&lt;p&gt;I'm examining some genomic coverage data which is basically a long list (a few million values) of integers, each saying how well (or &quot;deep&quot;)  this position in the genome is covered.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to look for &quot;valleys&quot; in this data, that is, regions which are significantly &quot;lower&quot; than their surrounding environment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that the size of the valleys I'm looking for may range from 50 bases to a few thousands.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What kind of paradigms would you recommend using to find those valleys?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some graphical examples for the data:&#xA;&lt;img src=&quot;http://i.imgur.com/5UEOg.png&quot; alt=&quot;alt text&quot;&gt;&#xA;&lt;img src=&quot;http://i.imgur.com/tlaV1.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE 2&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Defining what is a valley is of course one of the question I'm struggling with. These are obvious ones for me:&#xA;&lt;img src=&quot;http://i.imgur.com/MaR6O.jpg&quot; alt=&quot;alt text&quot;&gt;&#xA;&lt;img src=&quot;http://i.imgur.com/GP9UZ.jpg&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;but there some more complex situations. In general, there are 3 criteria I consider:&#xA;1. The (average? maximal?) coverage in the window with respect to the global average.&#xA;2. The (...) coverage in the window with respect to its immediate surrounding.&#xA;3. How large is the window: if I see very low coverage for a short span it is interesting, if I see very low coverage for a long span it's also interesting, if I see mildly low coverage for a short span it's &lt;strong&gt;not&lt;/strong&gt; really interesting, but if I see mildly low coverage for a long span - it is.. So it's a combination of the length of the sapn and it's coverage. The longer it is, the higher I let the coverage be and still consider it a valley.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Dave&lt;/p&gt;&#xA;" OwnerUserId="634" LastEditorUserId="634" LastEditorDisplayName="" LastEditDate="2010-09-24T18:02:47.390" LastActivityDate="2010-09-26T20:15:09.507" Title="How to look for valleys in a graph?" Tags="&lt;r&gt;&lt;distributions&gt;&lt;statistical-significance&gt;&lt;data-visualization&gt;" AnswerCount="4" CommentCount="8" FavoriteCount="3" />
  <row Id="3053" PostTypeId="2" ParentId="3051" CreationDate="2010-09-24T15:27:13.803" Score="2" ViewCount="" Body="&lt;p&gt;This simple line of code does the thing:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;((c(x,0,0) + c(0,x,0) + c(0,0,x))/3)[3:(length(x)-1)]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;if &lt;code&gt;x&lt;/code&gt; is the vector in question.&lt;/p&gt;&#xA;" OwnerUserId="1414" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-24T17:16:59.560" LastActivityDate="2010-09-24T17:16:59.560" CommentCount="2" />
  <row Id="3054" PostTypeId="2" ParentId="3051" CreationDate="2010-09-24T15:36:42.200" Score="8" ViewCount="" Body="&lt;p&gt;Function &lt;code&gt;rollapply&lt;/code&gt; in package zoo gets you close:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; require(zoo)&#xA;&amp;gt; TS &amp;lt;- zoo(c(4, 5, 7, 3, 9, 8))&#xA;&amp;gt; rollapply(TS, width = 3, by = 2, FUN = mean, align = &quot;left&quot;)&#xA;       1        3 &#xA;5.333333 6.333333&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It just won't compute the last value for you as it doesn't contain 3 observations. Maybe this will be sufficient for your real problem? Also, note that the returned object has the indices you want as the &lt;code&gt;names&lt;/code&gt; of the returned vector.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your example is making an assumption that there is an unobserved 0 in the last window. It might be more useful or realistic to pad with an &lt;code&gt;NA&lt;/code&gt; to represent the missing information and tell &lt;code&gt;mean&lt;/code&gt; to handle missing values. In this case we will have (8+9)/2 as our final windowed value.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; TS &amp;lt;- zoo(c(4, 5, 7, 3, 9, 8, NA))&#xA;&amp;gt; rollapply(TS, width = 3, by = 2, FUN = mean, na.rm = TRUE, align = &quot;left&quot;)&#xA;       1        3        5 &#xA;5.333333 6.333333 8.500000&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1390" LastEditorUserId="1390" LastEditorDisplayName="" LastEditDate="2010-09-25T09:03:09.933" LastActivityDate="2010-09-25T09:03:09.933" CommentCount="6" />
  <row Id="3055" PostTypeId="2" ParentId="3052" CreationDate="2010-09-24T15:43:45.877" Score="1" ViewCount="" Body="&lt;p&gt;There are many options for this, but one good one: you can use the &lt;code&gt;msExtrema&lt;/code&gt; function in the &lt;a href=&quot;http://cran.r-project.org/web/packages/msProcess/index.html&quot; rel=&quot;nofollow&quot;&gt;&lt;code&gt;msProcess&lt;/code&gt; package&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Edit:&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In financial performance analysis, this kind of analysis is often performed using a &quot;drawdown&quot; concept.  The &lt;code&gt;PerformanceAnalytics&lt;/code&gt; package has some &lt;a href=&quot;http://braverock.com/brian/R/PerformanceAnalytics/html/findDrawdowns.html&quot; rel=&quot;nofollow&quot;&gt;useful functions to find these valleys&lt;/a&gt;.  You could use the same algorithm here if you treat your observations as a time series.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are some examples of how you might be able to apply this to your data (where the &quot;dates&quot; are irrelevant but just used for ordering), but the first elements in the &lt;code&gt;zoo&lt;/code&gt; object would be your data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(PerformanceAnalytics)&#xA;x &amp;lt;- zoo(cumsum(rnorm(50)), as.Date(1:50))&#xA;findDrawdowns(x)&#xA;table.Drawdowns(x)&#xA;chart.Drawdown(x)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="5" LastEditorUserId="5" LastEditorDisplayName="" LastEditDate="2010-09-24T16:36:44.277" LastActivityDate="2010-09-24T16:36:44.277" CommentCount="6" />
  <row Id="3056" PostTypeId="2" ParentId="2909" CreationDate="2010-09-24T16:03:15.173" Score="1" ViewCount="" Body="&lt;p&gt;You might start by looking at the &lt;a href=&quot;http://help.rmetrics.org/fBasics/html/stats-maxdd.html&quot; rel=&quot;nofollow&quot;&gt;drawdown distribution functions in fBasics&lt;/a&gt;.  So you could easily simulate the brownian motion with drift and apply these functions as a start.&lt;/p&gt;&#xA;" OwnerUserId="5" LastEditorUserId="5" LastEditorDisplayName="" LastEditDate="2010-10-28T16:01:23.113" LastActivityDate="2010-10-28T16:01:23.113" CommentCount="3" />
  <row Id="3057" PostTypeId="2" ParentId="3052" CreationDate="2010-09-24T16:32:47.110" Score="3" ViewCount="" Body="&lt;p&gt;I'm completely ignorant of these data, but assuming the data are ordered (not in time, but by position?) it makes sense to make use of time series methods. There are lots of methods for identifying temporal clusters in data. Generally they are used to find high values but can be used for low values grouped together. I'm thinking here of scan statistics, cumulative sum statistics (and others) used to detect disease outbreaks in count data. Examples of these methods are in the surveillance package and the DCluster package.&lt;/p&gt;&#xA;" LastActivityDate="2010-09-24T16:32:47.110" CommentCount="1" />
  <row Id="3058" PostTypeId="2" ParentId="3051" CreationDate="2010-09-24T16:40:33.070" Score="0" ViewCount="" Body="&lt;p&gt;This will get you the window means and the index of the first value of the window:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#The data&#xA;x &amp;lt;- c(4, 5, 7, 3, 9, 8)&#xA;&#xA;#Set window size and slide&#xA;win.size &amp;lt;- 3&#xA;slide &amp;lt;- 2&#xA;&#xA;#Set up the table of results&#xA;results &amp;lt;- data.frame(index = numeric(), win.mean = numeric())&#xA;&#xA;#i indexes the first value of the window (the sill?)&#xA;i &amp;lt;- 1&#xA;#j indexes the row of the results to be added next&#xA;j &amp;lt;- 1&#xA;while(i &amp;lt; length(x)) {&#xA;    #This mean preserves the denominator of 3&#xA;    win.mean &amp;lt;- sum(x[i:(i+2)], na.rm = TRUE)/win.size&#xA;    #Insert the results&#xA;    results[j, ] &amp;lt;- c(i, win.mean)&#xA;    #Increment the indices for the next pass&#xA;    i &amp;lt;- i + slide&#xA;    j &amp;lt;- j + 1&#xA;    }&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Various caveats apply: haven't tested this against anything but your sample data; I believe that appending to data frames like this can get &lt;em&gt;really&lt;/em&gt; slow if you have lots of values (because it'll copy the data.frame each time); etc.  But it does produce what you asked for.&lt;/p&gt;&#xA;" OwnerUserId="71" LastActivityDate="2010-09-24T16:40:33.070" CommentCount="5" />
  <row Id="3059" PostTypeId="1" CreationDate="2010-09-24T17:04:59.187" Score="5" ViewCount="147" Body="&lt;p&gt;I'm investigating the effect of a continuous variable A on a measurement variable M stratified by another factor variable C in an observational dataset. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Due to heteroscedasticity I decided to use a bootstrapped regression analysis. However looking at the data, the background set of variables are not evenly distributed if I dichotomise A (present or not). I've just finished running another analysis where I do the same analysis after having matched the dataset for confounders (using CEM in R).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now the problem is which analysis to trust: the bootstrapped regression approach on the entire dataset or the bootstrapped version of the matched data? Under one of the factors in C the results diverge.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas how this can be analyzed?&lt;/p&gt;&#xA;" OwnerUserId="1291" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-09-27T00:25:43.633" LastActivityDate="2010-09-27T00:25:43.633" Title="Bootstrapped regression with total data or bootstrap with matched data?" Tags="&lt;bootstrap&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="3060" PostTypeId="2" ParentId="2686" CreationDate="2010-09-24T17:57:25.143" Score="1" ViewCount="" Body="&lt;p&gt;Buzzwords: interpolation, resampling, smoothing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your problem is similar to one encountered frequently in demography: people might have census counts broken down into age intervals, for example, and such intervals are not always of constant width.  You want to interpolate the distribution by age.  What this shares with your problem, aside from the variable width (= variable time intervals), is that the data tend to be non-negative.  In addition, many such datasets can have noise, but it has a particular form of negative correlation: a count that appears in one bin will not appear in neighboring bins, but might have been assigned to the wrong bin.  For example, older people may tend to round their ages to the nearest five years.  They are not overlooked but they might contribute to the wrong age group.  By and large, though, the data are complete and reliable.  In terms of this analogy we're talking about a full census; in your datasets you have actual electric bills, actual enrollments, and so on.  So it's just a question of apportioning the data reasonably to a set of intervals useful for further analysis (such as equally spaced times for time series analysis): that's where interpolation and resampling are involved.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are many interpolation techniques.  The commonest in demography were developed for simple calculation and are based on polynomial splines.  Many share a trick worth knowing, regardless of how you plan to process your data: don't attempt to interpolate the raw data; instead, &lt;em&gt;interpolate their cumulative sum.&lt;/em&gt;  The latter will be monotonically increasing due to the non-negativity of the original values, and therefore will tend to be relatively smooth.  This is why polynomial splines can work at all.  Another advantage of this approach is that although the fit may deviate from the data points (slightly, one hopes), overall it correctly reproduces the &lt;em&gt;totals&lt;/em&gt;, so that nothing is net lost or gained.  Of course, after fitting the cumulative values (as a function of time or age), you take first differences to estimate totals within any bin you like.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The simplest example of this approach is a linear spline: just connect successive points on the plot of cumulative $x$ vs. cumulative $t$ by line segments.  Estimate the counts in any time interval $[t_0, t_1]$ by reading off the values $x_0$ and $x_1$ of the splined curve at $t_0$ and $t_1$ respectively and using $x_1 - x_0$.  Better splines (cubic in some areas; quintic in many demographic apps) sometimes improve the estimates.  This is equivalent to your intuition of weighting the data and gives it a nice graphical interpretation.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-09-24T17:57:25.143" CommentCount="1" />
  <row Id="3061" PostTypeId="1" CreationDate="2010-09-24T18:40:39.993" Score="1" ViewCount="134" Body="&lt;p&gt;Does Lin's &lt;a href=&quot;http://en.wikipedia.org/wiki/Concordance_correlation_coefficient&quot; rel=&quot;nofollow&quot;&gt;concordance correlation coefficient&lt;/a&gt; assume that the 2 data sets have linear or monotonic tendencies? Or, can I measure the concordance between 2 data sets that have a sinusoidal tendency?&#xA;Thanks&#xA;NG&lt;/p&gt;&#xA;" OwnerUserId="1417" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-11-24T17:41:40.837" LastActivityDate="2010-11-24T17:41:40.837" Title="Does the concordance correlation coefficient make linearity or monotone assumptions?" Tags="&lt;correlation&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="3062" PostTypeId="1" CreationDate="2010-09-24T19:09:28.577" Score="1" ViewCount="78" Body="&lt;p&gt;Is there a statistical procedure that can measure the concordance between two sets of data that have, say, a sinusoidal tendency?&lt;/p&gt;&#xA;" OwnerUserId="1417" LastActivityDate="2010-09-24T19:21:21.743" Title="How do we find concordance between two sets of non-linear, non-monotonic data" Tags="&lt;correlation&gt;" AnswerCount="1" />
  <row Id="3063" PostTypeId="2" ParentId="3062" CreationDate="2010-09-24T19:21:21.743" Score="2" ViewCount="" Body="&lt;p&gt;short answer (the question is rather imprecise). Say $y_{1t}$ and $y_{2t}$ are your two sets, then $z_t=y_{1t}-y_{2t}$. Then it amounts to checking whether  $z_t$ is bounded (in your example it will be, if you sample $y_{1}$ and $y_{2}$ over a long enough amount to time).&lt;/p&gt;&#xA;" OwnerUserId="603" LastActivityDate="2010-09-24T19:21:21.743" />
  <row Id="3064" PostTypeId="1" AcceptedAnswerId="3349" CreationDate="2010-09-24T19:54:13.507" Score="9" ViewCount="216" Body="&lt;p&gt;I have the test results of a blood test administered to 2500 people four times at six-month intervals.  The results primarily consist of two measures of immune response - one in the presence of certain tuberculosis antigens, one in the absence.  Currently, each test evaluates to either positive or negative based on the difference between the antigen response and the nil response (with the idea being that if your immune system responds to TB antigens, you've likely been exposed to the bacterium itself at some point).  In essence, the test supposes that a non-exposed individual's distributions of nil and TB responses should be basically identical, whereas a person with TB exposure will have TB responses drawn from a different distribution (of higher values).  &lt;strong&gt;Caveat&lt;/strong&gt;: the responses are very, very non-normal, and values clump at both the natural floor and the instrument-truncated ceiling.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, it's seems pretty clear in this longitudinal setting that we're getting &quot;false positives&quot; (no actual gold standard for latent tuberculosis, I fear) that are caused by (typically small) fluctuations in the antigen and nil responses.  While this might be hard to avoid in some situations (you may only get one chance to test someone), there are many situations in which people are routinely tested for TB every year or so - in the US, this is common for healthcare workers, the military, homeless people staying at shelters, and so on.  It seems a shame to ignore prior test results because the extant criteria happen to be cross-sectional.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I &lt;em&gt;think&lt;/em&gt; that what I'd like to do is what I crudely conceive of as longitudinal mixture analysis.  Much like the cross-sectional criteria, I'd like to be able to estimate the probability that an individual's TB and nil responses are drawn from the same distribution - but have that estimate incorporate prior test results, as well as information from the sample as a whole (e.g., can I use the sample-wide distribution of within-individual variabilities to improve my estimates of a specific individual's distribution of nil or TB?).  The estimated probability would need to be able to change over time, of course, to account for the possibility of new infection.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've gotten myself totally twisted around trying to think about this in unusual ways, but I feel like this conceptualization is as good as any I'm going to come up with.  If something doesn't make sense, please feel free to ask for clarification.  If my understanding of the situation seems wrong, please feel free to tell me.  Thank you so much for your help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;In response to Srikant:&lt;/strong&gt;&#xA;It's a case of latent classification (TB-infected or not) using the two continuous (but non-normal and truncated) test results.  Right now, that classification is done using a cutoff (in its simplified form, TB - nil &gt; .35 -&gt; positive).  With test results presented as (nil, TB, result), the basic archetypes* are:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Probable Negative: (0.06, 0.15, -) (0.24, 0.23, -) (0.09, 0.11, -) (0.16, 0.15, -)&lt;br&gt;&#xA;Probable Positive: (0.05, 3.75, +) (0.05, 1.56, +) (0.06, 5.02, +) (0.08, 4.43, +)&lt;br&gt;&#xA;Wobbler:           (0.05, 0.29, -) (0.09, 0.68, +) (0.08, 0.31, -) (0.07, 0.28, -)  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The positive on the second test for the Wobbler is pretty clearly an aberration, but how would you model that?  While one line of my thinking is to estimate the &quot;true difference&quot; between TB and nil at each time point using a repeated-measures multilevel model, it occurred to me that what I really want to know is if the person's nil response and TB response are drawn from the same distribution, or if their immune system recognizes the TB antigens and activates, producing an increased response.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for what could cause a positive test other than infection: I'm not sure.  I suspect it's typically just within-person variation in results, but there's certainly a possibility of other factors.  We do have questionnaires from each time point, but I haven't looked into those too much yet.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;*Fabricated but illustrative data&lt;/p&gt;&#xA;" OwnerUserId="71" LastEditorUserId="71" LastEditorDisplayName="" LastEditDate="2010-09-29T20:51:01.840" LastActivityDate="2010-10-06T23:46:47.417" Title="Longitudinal comparison of two distributions" Tags="&lt;repeated-measures&gt;" AnswerCount="2" CommentCount="4" FavoriteCount="2" />
  <row Id="3065" PostTypeId="2" ParentId="3052" CreationDate="2010-09-24T20:12:41.937" Score="3" ViewCount="" Body="&lt;p&gt;You could use some sort of Monte Carlo approach, using for instance the moving average of your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Take a moving average of the data, using a window of a reasonable size (I guess it's up to you deciding how wide).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Throughs in your data will (of course) be characterized by a lower average, so now you need to find some &quot;threshold&quot; to define &quot;low&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To do that you randomly swap the values of your data (e.g. using &lt;code&gt;sample()&lt;/code&gt;) and recalculate the moving average for your swapped data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Repeat this last passage a reasonably high amount of times (&gt;5000) and store all the averages of these trials. So essentially you will have a matrix with 5000 lines, one per trial, each one containing the moving average for that trial.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At this point for each column you pick the 5% (or 1% or whatever you want) quantile, that is the value under which lies only 5% of the means of the randomized data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You now have a &quot;confidence limit&quot; (I'm not sure if that is the correct statistical term) to compare your original data with. If you find a part of your data that is lower than this limit then you can call that a through.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, bare in mind that not this nor any other mathematical method could ever give you any indication of biological significance, although I'm sure you're well aware of that. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT - an example&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(ares) # for the ma (moving average) function&#xA;&#xA;# Some data with peaks and throughs &#xA;values &amp;lt;- cos(0.12 * 1:100) + 0.3 * rnorm(100) &#xA;plot(values, t=&quot;l&quot;)&#xA;&#xA;# Calculate the moving average with a window of 10 points &#xA;mov.avg &amp;lt;- ma(values, 1, 10, FALSE)&#xA;&#xA;numSwaps &amp;lt;- 1000    &#xA;mov.avg.swp &amp;lt;- matrix(0, nrow=numSwaps, ncol=length(mov.avg))&#xA;&#xA;# The swapping may take a while, so we display a progress bar &#xA;prog &amp;lt;- txtProgressBar(0, numSwaps, style=3)&#xA;&#xA;for (i in 1:numSwaps)&#xA;{&#xA;# Swap the data&#xA;val.swp &amp;lt;- sample(values)&#xA;# Calculate the moving average&#xA;mov.avg.swp[i,] &amp;lt;- ma(val.swp, 1, 10, FALSE)&#xA;setTxtProgressBar(prog, i)&#xA;}&#xA;&#xA;# Now find the 1% and 5% quantiles for each column&#xA;limits.1 &amp;lt;- apply(mov.avg.swp, 2, quantile, 0.01, na.rm=T)&#xA;limits.5 &amp;lt;- apply(mov.avg.swp, 2, quantile, 0.05, na.rm=T)&#xA;&#xA;# Plot the limits&#xA;points(limits.5, t=&quot;l&quot;, col=&quot;orange&quot;, lwd=2)&#xA;points(limits.1, t=&quot;l&quot;, col=&quot;red&quot;, lwd=2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This will just allow you to graphically find the regions, but you can easily find them using something on the lines of &lt;code&gt;which(values&amp;gt;limits.5)&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="582" LastEditorUserId="582" LastEditorDisplayName="" LastEditDate="2010-09-24T21:37:23.963" LastActivityDate="2010-09-24T21:37:23.963" CommentCount="3" />
  <row Id="3066" PostTypeId="2" ParentId="3048" CreationDate="2010-09-25T00:23:28.127" Score="5" ViewCount="" Body="&lt;p&gt;Lets assume mat_pages[] contains pages in the columns (which you want to cluster) and individuals in the rows. You can cluster pages based on individual data in Rby using the following command: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  pc &amp;lt;- prcomp(x=mat_pages,center=TRUE,scale=TRUE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The loadings matrix is the matrix of eigenvectors of the SVD decomposition of&#xA;the data. They give the relative weight of each PAGE in the calculation of scores.&#xA;Loadings with larger absolute values have more influence in determining the score&#xA;of the corresponding principle component.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I should also point out the &lt;em&gt;short coming&lt;/em&gt; of using PCA to cluster pages. The reason for this is that loadings give larger weights to the PAGES with higher variation, regardless of whether this variation is actually because of the PAGE content or some other reason (may be technical or individual variation). The loadings do not necessarily reflect the true differences between groups, which (maybe) your main interest. BUT, this clustering truly reflects the differences in the group under the assumption that all the pages have same variance (I don't know if this is a valid assumption).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have a powerful computing facilities (which may be possible given your data size) - using hierarchical models may be a good idea. In R, it can be done using lme4 package.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;What do do after you have the scores?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a crude suggestion and the analysis depends greatly on how the data looks like. Also, I would guess this process would be highly infeasible to group the data of magnitude that you have. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pc.col &amp;lt;- paste(&quot;page&quot;, 1:27000, sep=&quot;.&quot;)&#xA;&#xA;pdf(&quot;principle-components.pdf&quot;)&#xA;plot(pc$x[,1:2]) ## Just look at the 1st two loadings (as you can see the groupings in a plane)&#xA;dev.off()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Hopefully, this can give you a picture of how the data is grouped into.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Warning:&lt;/strong&gt; this is not what I would recommend.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;My recommendation:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Problems like these arise frequently in genomics.In your case pages corresponds to genes and individuals corresponds to patients (basically individuals is has the same meaning as in genomics)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You want to cluster the pages based on data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can use a lot of clustering packages in R and have been pointed in other answers. A fundamental problem with packages is like hclust is how to determine the number of clusters. A few of my favorite ones are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;pvclust (Gives you clusters and also gives a p-value for each cluster. Using the p-value you can determine the statistically significant clusters. &lt;em&gt;Problem&lt;/em&gt;: requires a lot of computational power and I am not sure if it will work with data of your size)&lt;/li&gt;&#xA;&lt;li&gt;hopach (Gives you the estimated number of clusters, and the clusters)&lt;/li&gt;&#xA;&lt;li&gt;there are other packages available in Bioconductor, please check them out in the task view.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You can also use clustering algos like k-means etc. I am sure I saw a thread in this forum about clustering. The answers were very detailed. It was asked by Tal Galili if I remember correctly.&lt;/p&gt;&#xA;" OwnerUserId="1307" LastEditorUserId="1307" LastEditorDisplayName="" LastEditDate="2010-09-28T16:05:34.673" LastActivityDate="2010-09-28T16:05:34.673" CommentCount="11" />
  <row Id="3067" PostTypeId="2" ParentId="175" CreationDate="2010-09-25T00:38:39.800" Score="3" ViewCount="" Body="&lt;p&gt;There are two statistical distance measures that are specifically catered to detecting outliers and then considering whether such outliers should be removed from your linear regression.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first one is Cook's distance.  You can find a pretty good explanation of it at Wikipedia: &lt;a href=&quot;http://en.wikipedia.org/wiki/Cook%27s_distance&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Cook%27s_distance&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The higher the Cook's distance is the more influential (impact on regression coefficient) the observation is.  The typical cut-off point to consider removing the observation is a Cook's distance = 4/n (n is sample size). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second one is DFFITS which is also well covered by Wikipedia: &lt;a href=&quot;http://en.wikipedia.org/wiki/DFFITS&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/DFFITS&lt;/a&gt;.&#xA;The typical cut-off point to consider removing an observation is a DFFITS value of 2 times sqrt(k/n) where k is number of variables and n is the sample size. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Both measures usually give you similar results leading to similar observation selection.   &lt;/p&gt;&#xA;" OwnerUserId="1329" LastActivityDate="2010-09-25T00:38:39.800" />
  <row Id="3068" PostTypeId="2" ParentId="2966" CreationDate="2010-09-25T03:51:59.883" Score="0" ViewCount="" Body="&lt;p&gt;One could view the recorded times as biased estimates of the runner's latent ability. Many factors would cause the time to be worse than the latent best time, such as a bad start, headwind, a stumble, mis-judgement of pace, etc, while very few would cause the recorded times to be better than the latent best, such as a strong tailwind or running downhill. I am not terribly familiar with regression with biased errors, but apparently one can use the gamma family when performing GLM regression; one would use time as the dependent variable and observed times as the dependent variable.&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-09-25T03:51:59.883" />
  <row Id="3069" PostTypeId="1" AcceptedAnswerId="3070" CreationDate="2010-09-25T04:09:33.167" Score="5" ViewCount="515" Body="&lt;p&gt;I have been using Matlab for my work, but I have started learning Python lately. I employ statistical analysis, more precisely geo-statistics, in my work. I was wanting to ask, from your perspectives, which one among the two languages is good for general statistical/data analysis? What are the pros and cons, other than accessibility, for each?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Feel free to edit/correct the question to make it more clear. Thanks.&lt;/p&gt;&#xA;" OwnerUserId="1102" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-10-20T21:10:35.623" LastActivityDate="2010-10-20T21:10:35.623" Title="Among Matlab and Python, which one is good for statistical analysis?" Tags="&lt;matlab&gt;&lt;python&gt;" AnswerCount="3" CommentCount="4" />
  <row Id="3070" PostTypeId="2" ParentId="3069" CreationDate="2010-09-25T04:53:48.313" Score="15" ViewCount="" Body="&lt;p&gt;As a diehard Matlab user for the last 10+ years, I recommend you learn Python. Once you are sufficiently skilled in a language, when you work in a language you are learning, it will seem like you are not being productive enough, and you will fall back to using your default best language. At the very least, I would suggest you try to become equally proficient in a number of languages (I would suggest R as well). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I like about Matlab: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;I am proficient in it.&lt;/li&gt;&#xA;&lt;li&gt;It is the &lt;em&gt;lingua franca&lt;/em&gt; among numerical analysts.&lt;/li&gt;&#xA;&lt;li&gt;the profiling tool is very good. This is the only reason I use Matlab instead of octave.&lt;/li&gt;&#xA;&lt;li&gt;There is a freeware clone, octave, which has good compliance with the reference implementation.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;What I do not like about Matlab:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;There is not a good system to manage third party (free or otherwise) packages and scripts. Mathworks controls the 'central file exchange', and installation of add-on packages seems very clunky, nothing like the excellent system that R has. Furthermore, Mathworks has no incentive to improve this situation, because they make money on selling toolboxes, which compete with freeware packages;&lt;/li&gt;&#xA;&lt;li&gt;Licenses for parallel computation in Matlab are insanely expensive;&lt;/li&gt;&#xA;&lt;li&gt;Much of the m-code, including many of the toolbox functions, and some builtins, were designed to be obviously correct, at the expense of efficiency and/or usability. The most glaring example of this is Matlab's &lt;code&gt;median&lt;/code&gt; function, which &lt;em&gt;performs a sort of the data, then takes the middle value&lt;/em&gt;. This has been the wrong algorithm since the 70's.&lt;/li&gt;&#xA;&lt;li&gt;saving graphs to file is dodgy at best in Matlab.&lt;/li&gt;&#xA;&lt;li&gt;I have not found my user experience to have improved over the last 5 years (when I started using Matlab instead of octave), even though Mathworks continues to add bells and whistles. This indicates that I am not their target customer, rather they are looking to expand market share by making things worse for power users.&lt;/li&gt;&#xA;&lt;li&gt;There are now 2 ways to do object-oriented programming in Matlab, which is confusing at best. Legacy code using the old style will persist for some time.&lt;/li&gt;&#xA;&lt;li&gt;The Matlab UI is written in Java, which has unpleasant ideas about memory management.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-09-25T04:53:48.313" CommentCount="9" />
  <row Id="3071" PostTypeId="2" ParentId="3069" CreationDate="2010-09-25T05:25:21.730" Score="6" ViewCount="" Body="&lt;p&gt;Lets break it down into three areas (off the top of my head) where programming meets statistics: data crunching, numerical routines (optimization and such) and statistical libraries (modeling, etc).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the first, the biggest difference is that Python is a general purpose programming language.  Matlab is great as long as your world is roughly isomorphic to a fortran numeric array.  Once you start dealing with data munging and related issues, Python outshines Matlab.  For example, see Greg Wilson's book: &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0974514071&quot; rel=&quot;nofollow&quot;&gt;Data Crunching: Solve Everyday Problems Using Java, Python, and more&lt;/a&gt;.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the second, Matlab really does shine with numeric work.  A lot of the research community uses it and if you're looking for say, some algorithm related to a paper in compressed sensing, you're &lt;em&gt;far&lt;/em&gt; more likely to find an implementation in Matlab.  On the other hand, Matlab is kind of the PHP of scientific computing -- it strives to have a function for everything under the sun.  The resulting aesthetics and architecture are maddening if you're a programming language geek, but in utilitarian terms, it gets the job done.  A lot of this has become less relvant with the rise of Numpy/Scipy, you're just as likely to find optimization and machine learning libraries available for Python.  Interfacing with C is about as easy in either language.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the availability of statistical libraries for modeling and such, both are somewhat lacking when compared to something like R.  (Though I suspect both will meet the needs for 80% of people doing statistical work.)  For the Python side of things see this question: &lt;a href=&quot;http://stats.stackexchange.com/questions/1595/python-as-a-statistics-workbench&quot;&gt;Python as a statistics workbench&lt;/a&gt;.  For the Matlab side, I know there's a statistics toolbox, but I'll let someone more knowledgeable fill in the blanks (my experience with Matlab is limited to numerical work unrelated to statistics).&lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-09-25T05:25:21.730" CommentCount="1" />
  <row Id="3072" PostTypeId="2" ParentId="3051" CreationDate="2010-09-25T05:51:58.807" Score="1" ViewCount="" Body="&lt;p&gt;I can do this easily in Matlab and duck while you downvote me:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;%given vector x, windowsize, slide &#xA;idx1 = 1:slide:numel(x);&#xA;idx2 = min(numel(x) + 1,idx1 + windowsize);  %sic on +1 here and no -1;&#xA;cx = [0;cumsum(x(:))];  %pad out a zero, perform a cumulative sum;&#xA;rv = (cx(idx2) - cx(idx1)) / windowsize; %tada! the answer!&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;as a side effect, &lt;code&gt;idx1&lt;/code&gt; is the index of the element in the sum. I am sure this can be easily translated into R. The idiom &lt;code&gt;first:skip:last&lt;/code&gt; in Matlab gives the array first, first+skip, first+2skip, ..., first + n skip, where the last element in the array is no greater than &lt;code&gt;last&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;edit&lt;/strong&gt;: I had omitted the averaging part (divide by &lt;code&gt;windowsize&lt;/code&gt;).&lt;/p&gt;&#xA;" OwnerUserId="795" LastEditorUserId="795" LastEditorDisplayName="" LastEditDate="2010-09-26T16:00:42.393" LastActivityDate="2010-09-26T16:00:42.393" CommentCount="4" />
  <row Id="3073" PostTypeId="2" ParentId="3051" CreationDate="2010-09-25T08:31:02.847" Score="1" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://stats.stackexchange.com/questions/3051/mean-of-a-sliding-window-in-r/3072#3072&quot;&gt;shabbychef's answer&lt;/a&gt; in R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;slideMean&amp;lt;-function(x,windowsize=3,slide=2){&#xA; idx1&amp;lt;-seq(1,length(x),by=slide);&#xA; idx1+windowsize-&amp;gt;idx2;&#xA; idx2[idx2&amp;gt;(length(x)+1)]&amp;lt;-length(x)+1;&#xA; c(0,cumsum(x))-&amp;gt;cx;&#xA; return((cx[idx2]-cx[idx1])/windowsize);&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;EDIT: Indices you're looking for are just &lt;code&gt;idx1&lt;/code&gt;... this function can be easily modified to return them also, but it is almost equally fast to recreate them with another call to &lt;code&gt;seq(1,length(x),by=slide)&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="88" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-25T08:39:03.437" LastActivityDate="2010-09-25T08:39:03.437" CommentCount="1" />
  <row Id="3074" PostTypeId="2" ParentId="3069" CreationDate="2010-09-25T09:20:23.517" Score="4" ViewCount="" Body="&lt;p&gt;MATLAB is expensive, have almost empty statistical library and huge amount of drawbacks; yet  I think that it is way better than Python.&#xA;Plotting with matplot is complicated and the results are poor (you often need 20 lines of tense code to make something that R does better with one &lt;code&gt;plot&lt;/code&gt;), working interactively in Python hurts, NumPy syntax is very non-mathematical, so it is hard to sync the code with equations you have in mind, also its reference semantics are quite tricky. Finally the statistical/ML libraries are also small and what's worse incompatible with each other. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Paraphrasing Django's slogan it can do the job, but is not suitable for perfectionists with deadlines. &lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-09-25T09:20:23.517" CommentCount="1" />
  <row Id="3075" PostTypeId="2" ParentId="3064" CreationDate="2010-09-25T09:39:40.313" Score="2" ViewCount="" Body="&lt;p&gt;Tricky Matt, as many real-world stats problems are!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would start be defining your study aims/objectives.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Without knowing the true status of the subjects it will be hard to define the probability distributions for the TB+ and TB- test. Do you have questionairres regarding previous TB infection (or better, medical histories). Also I still test TB+ due to an immunisation in childhood - several decades ago - so previous immunisations need to be considered.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems to me your intrinsic question is: Does repeated TB testing affect test outcome? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It would be worth getting a copy of &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0198524846&quot; rel=&quot;nofollow&quot;&gt;Peter Diggle's Analysis of Longitudinal Data&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do some exploratory data analysis, particularly scatter plot matrices of the nil-test results at each time versus each other, and the TB test results at each time versus each other; and the TB vs nil scatter plots (at each time). Also take the differences (TB test - Nil test) and do the scatter plot matrices. Try transformations of the data and redo these - I imagine log(TB) - log(Nil) may help if the TB results are very large relative to Nil.  Look for linear relations in the correlations structure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another approach would be to take the defined test result (positive/ negative) and model this logitudibnally using a non-linear mixed effects model (logit link). Do some individuals flip between testing TB+ to TB- and is this related to their Nil test, TB test, TB - Nil or some transformation of test results?&lt;/p&gt;&#xA;" OwnerUserId="521" LastActivityDate="2010-09-25T09:39:40.313" CommentCount="4" />
  <row Id="3076" PostTypeId="2" ParentId="2910" CreationDate="2010-09-25T15:45:57.583" Score="6" ViewCount="" Body="&lt;p&gt;This overlaps with Shane's answer, but in my view there are two main piers:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Reproducibility&lt;/strong&gt;; not&#xA;only because you won't end with&#xA;results that are made &quot;somehow&quot; but&#xA;also be able to rerun the analysis&#xA;faster (on other data or with&#xA;slightly changed parameters) and have&#xA;more time to think about the results. For a huge data, you can first test your ideas on some small &quot;playset&quot; and then easily extend on the whole data.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Good documentation&lt;/strong&gt;; commented scripts under version&#xA;control, some research journal, even&#xA;ticket system for more complex&#xA;projects. Improves reproducibility, makes error tracking easier and writing final reports        trivial.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-09-25T15:45:57.583" CommunityOwnedDate="2010-09-25T15:45:57.583" CommentCount="1" />
  <row Id="3077" PostTypeId="2" ParentId="1735" CreationDate="2010-09-25T16:17:28.180" Score="1" ViewCount="" Body="&lt;p&gt;One answer is to do a seemingly unrelated regression. Suppose that you only have a single predictor plus an intercept. Create a data set (or data matrix) like&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;wo 1 wp 0 0&#xA;so 0 0  1 sp&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where 'wo' is the outcome in the winter season and 'wp' is the winter predictor/&quot;X&quot; value and 'so' is the summer outcome value and 'sp' is the summer predictor. The 1s represent the summer and winter intercept terms. Basically, you have two sets of variables: summer variables and winter variables. In the summer, all the winter variables are set to 0 and vice-versa in the summer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After you run a regression on the full set of summer and winter variables using the data template above, you get a full covariance matrix that can be used to compare the coefficients for the winter months to the summer months using standard regression procedures.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is one way of implementing @kwak's suggestion of having different slope coefficients for each season.&lt;/p&gt;&#xA;" OwnerUserId="401" LastActivityDate="2010-09-25T16:17:28.180" />
  <row Id="3078" PostTypeId="2" ParentId="1001" CreationDate="2010-09-25T16:27:42.863" Score="4" ViewCount="" Body="&lt;p&gt;For measuring the bin frequencies of two distributions, a pretty good test is the Chi Square test.  It is exactly what it is designed for.  And, it is even nonparametric.  The distribution don't even have to be normal or symmetric.  It is much better than the Kolmogorov-Smirnov test that is known to be weak in fitting the tails of the distribution where the fitting or diagnosing is often the most important. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Spearman's correlation won't be so precise in terms of capturing the similarities of your actual bin frequencies.  It will just tell you that your overall ranking of observations for the two distributions are similar.  Instead, when calculating the Chi Square test (long hand so to speak) you will be able to observe readily which bin frequencies differentials are the most responsible for driving down the overall p value of the Chi Square test. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another pretty good test is the Anderson-Darling test.  It is one of the best tests to diagnose the fit between two distributions.  However, in terms of giving information about the specific bin frequencies I suspect that the Chi Square test gives you more information.     &lt;/p&gt;&#xA;" OwnerUserId="1329" LastActivityDate="2010-09-25T16:27:42.863" CommentCount="2" />
  <row Id="3079" PostTypeId="2" ParentId="3059" CreationDate="2010-09-25T16:30:42.857" Score="5" ViewCount="" Body="&lt;p&gt;There is a wrinkle that you need to worry about. In the case of matching, you are throwing away observations (i.e., those that aren't matched and don't make it into your analysis) and some might be replicated. These decisions aren't random; they are a function of covariates. As a result, creating confidence intervals in this context are a bit complicated. To calculate the appropriate standard errors, see &lt;a href=&quot;http://ksghome.harvard.edu/~.aabadie.academic.ksg/sme.pdf&quot; rel=&quot;nofollow&quot;&gt;Large Sample Properties of (Matching Estimators for Average Treatment Effects&lt;/a&gt; by Abadie and Imbens. Additionally, Abadie also has a paper on &lt;a href=&quot;http://ideas.repec.org/p/nbr/nberte/0325.html&quot; rel=&quot;nofollow&quot;&gt;On the Failure of the Bootstrap for Matching Estimators&lt;/a&gt;. To implement Abadie-Imbens standard errors, see the Matching package in R by Jas Sekhon.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the question of which estimator to believe, it depends on how well you think that matching is correctly controlling for confounders, I'd be inclined to believe that approach. It seems that your first set of analyses doesn't control for those factors in any way? If you think that they are important, then you probably wouldn't be inclined to believe those results.&lt;/p&gt;&#xA;" OwnerUserId="401" LastActivityDate="2010-09-25T16:30:42.857" CommentCount="1" />
  <row Id="3080" PostTypeId="1" AcceptedAnswerId="3099" CreationDate="2010-09-26T04:25:32.470" Score="11" ViewCount="212" Body="&lt;p&gt;You so often come across in the press various studies that conclude directionally opposite results.  Those can be related to the testing of a new prescription drug or the merit of a specific nutrient or anything else for that matter. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When two such studies arrive at conflicting results how can you tell which one of the two is closest to the truth? &lt;/p&gt;&#xA;" OwnerUserId="1329" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-09-27T00:44:35.023" LastActivityDate="2010-09-27T03:26:45.223" Title="How to detect which one is the better study when they give you conflicting results?" Tags="&lt;hypothesis-testing&gt;&lt;clinical-trials&gt;" AnswerCount="3" CommentCount="4" FavoriteCount="3" />
  <row Id="3081" PostTypeId="2" ParentId="3080" CreationDate="2010-09-26T06:18:11.867" Score="8" ViewCount="" Body="&lt;p&gt;The &lt;a href=&quot;http://jeromyanglim.blogspot.com/2009/12/meta-analysis-tips-resources-and.html&quot; rel=&quot;nofollow&quot;&gt;meta analysis&lt;/a&gt; literature is relevant to your question. Using meta-analytic techniques you could generate an estimate of the effect of interest pooled across studies. Such techniques often weight studies in terms of their sample size.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Within the meta analysis context researchers talk about fixed effect and random effect models (see &lt;a href=&quot;http://onlinelibrary.wiley.com/doi/10.1111/1468-2389.00156/abstract&quot; rel=&quot;nofollow&quot;&gt;Hunter and Schmidt, 2002&lt;/a&gt;). A fixed effect model assumes that all studies are estimating the same population effect. A random-effects model assumes that studies differ in the population effect that is being estimated. A random-effects model is typically more appropriate. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As more studies accumulate looking at a particular relationship, more sophisticated approaches become possible. For example, you can code studies in terms of various properties, such as perceived quality, and then examine empirically whether the effect size varies with these study characteristics. Beyond quality there may be some theoretically relevant differences between the studies which would moderate the relationship (e.g., characteristic of the sample, dosage levels, etc.).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, I tend to trust studies with:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;bigger sample sizes&lt;/li&gt;&#xA;&lt;li&gt;greater methodological rigour&lt;/li&gt;&#xA;&lt;li&gt;a confirmatory orientation (e.g., not a study where they tested for correlations between 100 different nutrients and 50 health outcomes)&lt;/li&gt;&#xA;&lt;li&gt;absence of conflict of interest (e.g., not by a company with a commercial interest in showing a relationship; not by a researcher who has an incentive to find a significant result)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;But that said you need to keep random sampling and theoretically meaningful differences between studies as a plausible explanation of conflicting study findings.&lt;/p&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-09-26T11:09:08.063" LastActivityDate="2010-09-26T11:09:08.063" CommentCount="4" />
  <row Id="3082" PostTypeId="1" CreationDate="2010-09-26T07:37:14.583" Score="9" ViewCount="342" Body="&lt;p&gt;This is a follow-up to a Stackoverflow &lt;a href=&quot;http://stackoverflow.com/questions/3796786/random-number-generator-without-dupes-in-javascript/3796821#3796821&quot;&gt;question about shuffling an array randomly&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are established algorithms (such as the Knuth-Fisher-Yates Shuffle) that one should use to shuffle an array, rather than relying on &quot;naive&quot; ad-hoc implementations. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am now interested in proving (or disproving) that my naive algorithm is broken (as in: does not generate all possible permutations with equal probability).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the algorithm:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Loop a couple of times (length of array should do), and in every iteration, get two random array indexes and swap the two elements there.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Obviously, this needs more random numbers than KFY (twice as much), but aside from that does it work properly? And what would be the appropriate number of iterations (is &quot;length of array&quot; enough)?&lt;/p&gt;&#xA;" OwnerUserId="1421" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-10-21T11:17:09.983" LastActivityDate="2010-10-21T11:17:09.983" Title="What is wrong with this &quot;naive&quot; shuffling algorithm?" Tags="&lt;permutation&gt;&lt;randomness&gt;" AnswerCount="3" CommentCount="4" FavoriteCount="1" />
  <row Id="3083" PostTypeId="2" ParentId="3082" CreationDate="2010-09-26T09:41:18.333" Score="2" ViewCount="" Body="&lt;p&gt;Bear in mind I am not a statistician, but I'll put my 2cents.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I made a little test in R (careful, it's very slow for high &lt;code&gt;numTrials&lt;/code&gt;, the code can probably be optimized):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;numElements &amp;lt;- 1000&#xA;numTrials &amp;lt;- 5000&#xA;&#xA;swapVec &amp;lt;- function()&#xA;    {&#xA;    vec.swp &amp;lt;- vec&#xA;&#xA;    for (i in 1:numElements)&#xA;        {&#xA;        i &amp;lt;- sample(1:numElements)&#xA;        j &amp;lt;- sample(1:numElements)&#xA;&#xA;        tmp &amp;lt;- vec.swp[i]&#xA;        vec.swp[i] &amp;lt;- vec.swp[j]&#xA;        vec.swp[j] &amp;lt;- tmp&#xA;        }&#xA;&#xA;    return (vec.swp)&#xA;    }&#xA;&#xA;# Create a normally distributed array of numElements length&#xA;vec &amp;lt;- rnorm(numElements)&#xA;&#xA;# Do several &quot;swapping trials&quot; so we can make some stats on them&#xA;swaps &amp;lt;- vec&#xA;prog &amp;lt;- txtProgressBar(0, numTrials, style=3)&#xA;&#xA;for (t in 1:numTrials)&#xA;    {&#xA;    swaps &amp;lt;- rbind(swaps, swapVec())&#xA;    setTxtProgressBar(prog, t)&#xA;    }&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This will generate a matrix &lt;code&gt;swaps&lt;/code&gt; with &lt;code&gt;numTrials+1&lt;/code&gt; rows (one per trial + the original) and &lt;code&gt;numElements&lt;/code&gt; columns (one per each vector element). If the method is correct the distribution of each column (i.e. of the values for each element over the trials) should not be different from the distribution of the original data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because our original data was normally distributed we would expect all the columns not to deviate from that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we run&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;par(mfrow= c(2,2))&#xA;# Our original data&#xA;hist(swaps[1,], 100, col=&quot;black&quot;, freq=FALSE, main=&quot;Original&quot;)&#xA;# Three &quot;randomly&quot; chosen columns&#xA;hist(swaps[,1], 100, col=&quot;black&quot;, freq=FALSE, main=&quot;Trial # 1&quot;) &#xA;hist(swaps[,257], 100, col=&quot;black&quot;, freq=FALSE, main=&quot;Trial # 257&quot;)&#xA;hist(swaps[,844], 100, col=&quot;black&quot;, freq=FALSE, main=&quot;Trial # 844&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;We get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/QZJVZ.jpg&quot; alt=&quot;Histograms of random trials&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;which looks very promising. Now, if we want to statistically confirm the distributions do not deviate from the original I think we could use a Kolmogorov-Smirnov test (please can some statistician confirm this is right?) and do, for instance&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ks.test(swaps[1, ], swaps[, 234])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Which gives us p=0.9926&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we check all of the columns:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ks.results &amp;lt;- apply(swaps, 2, function(col){ks.test(swaps[1,], col)})&#xA;p.values &amp;lt;- unlist(lapply(ks.results, function(x){x$p.value})&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And we run&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;hist(p.values, 100, col=&quot;black&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;we get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/Bcno2.jpg&quot; alt=&quot;Histogram of Kolmogorov-Smirnov test p values&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, for the great majority of the elements of the array, your swap method has given a good result, as you can also see looking at the quartiles.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1&amp;gt; quantile(p.values)&#xA;       0%       25%       50%       75%      100% &#xA;0.6819832 0.9963731 0.9999188 0.9999996 1.0000000&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Note that, obviously, with a lower number of trials the situation is not as good:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;50 trials&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1&amp;gt; quantile(p.values)&#xA;          0%          25%          50%          75%         100% &#xA;0.0003399635 0.2920976389 0.5583204486 0.8103852744 0.9999165730&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;100 trials&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;          0%         25%         50%         75%        100% &#xA; 0.001434198 0.327553996 0.596603804 0.828037097 0.999999591 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;500 trials&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;         0%         25%         50%         75%        100% &#xA;0.007834701 0.504698404 0.764231550 0.934223503 0.999995887 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="582" LastEditorUserId="582" LastEditorDisplayName="" LastEditDate="2010-09-26T21:29:29.830" LastActivityDate="2010-09-26T21:29:29.830" />
  <row Id="3084" PostTypeId="2" ParentId="1875" CreationDate="2010-09-26T10:48:15.957" Score="3" ViewCount="" Body="&lt;p&gt;When the forecast says &quot;X percent chance of rain in (area)&quot;, it means that the numerical weather model has indicated rain in X percent of the area, for the time interval in question.  For example, it would normally be accurate to predict &quot;100 percent chance of rain in North America&quot;.  Bear in mind that the models are good at predicting dynamics and poor at predicting thermodynamics.&lt;/p&gt;&#xA;" OwnerUserId="1054" LastActivityDate="2010-09-26T10:48:15.957" CommentCount="1" />
  <row Id="3085" PostTypeId="2" ParentId="3082" CreationDate="2010-09-26T13:04:04.267" Score="4" ViewCount="" Body="&lt;p&gt;I think your simple algorithm will shuffle the cards correctly as the number shuffles tends to infinity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose you have three cards: {A,B,C}. Assume that your cards begin in the following order: A,B,C. Then after one shuffle you have following combinations:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;{A,B,C}, {A,B,C}, {A,B,C} #You get this if choose the same RN twice.&#xA;{A,C,B}, {A,C,B}&#xA;{C,B,A}, {C,B,A}&#xA;{B,A,C}, {B,A,C}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Hence, the probability of card A of being in position {1,2,3} is {5/9, 2/9, 2/9}.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we shuffle the cards a second time, then:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Pr(A in position 1 after 2 shuffles) = 5/9*Pr(A in position 1 after 1 shuffle) &#xA;                                     + 2/9*Pr(A in position 2 after 1 shuffle) &#xA;                                     + 2/9*Pr(A in position 3 after 1 shuffle) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This gives 0.407. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using the same idea, we can form a recurrence relationship, i.e:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Pr(A in position 1 after n shuffles) = 5/9*Pr(A in position 1 after (n-1) shuffles) &#xA;                                     + 2/9*Pr(A in position 2 after (n-1) shuffles) &#xA;                                     + 2/9*Pr(A in position 3 after (n-1) shuffles).&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Coding this up in R (see code below), gives probability of card A of being in position {1,2,3} as {0.33334, 0.33333, 0.33333} after ten shuffles.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;R code&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;## m is the probability matrix of card position&#xA;## Row is position&#xA;## Col is card A, B, C&#xA;m = matrix(0, nrow=3, ncol=3)&#xA;m[1,1] = 1; m[2,2] = 1; m[3,3] = 1&#xA;&#xA;## Transition matrix&#xA;m_trans = matrix(2/9, nrow=3, ncol=3)&#xA;m_trans[1,1] = 5/9; m_trans[2,2] = 5/9; m_trans[3,3] = 5/9&#xA;&#xA;for(i in 1:10){&#xA;  old_m = m&#xA;  m[1,1] = sum(m_trans[,1]*old_m[,1])&#xA;  m[2,1] = sum(m_trans[,2]*old_m[,1])&#xA;  m[3,1] = sum(m_trans[,3]*old_m[,1])&#xA;&#xA;  m[1,2] = sum(m_trans[,1]*old_m[,2])&#xA;  m[2,2] = sum(m_trans[,2]*old_m[,2])&#xA;  m[3,2] = sum(m_trans[,3]*old_m[,2])&#xA;&#xA;  m[1,3] = sum(m_trans[,1]*old_m[,3])&#xA;  m[2,3] = sum(m_trans[,2]*old_m[,3])&#xA;  m[3,3] = sum(m_trans[,3]*old_m[,3])&#xA;}  &#xA;m&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="8" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-09-26T17:28:00.303" LastActivityDate="2010-09-26T17:28:00.303" CommentCount="4" />
  <row Id="3086" PostTypeId="1" AcceptedAnswerId="3135" CreationDate="2010-09-26T13:40:48.663" Score="3" ViewCount="452" Body="&lt;p&gt;Currently I'm analysing around 300 items in the field of education.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm interested in the dimensionality of the dataset. So I compute a matrix of tetrachoric correlation. The goal is to do a factor analysis on this matrix.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By curiosity I compare to a matrix of Pearson correlation, and the results are different. When I compute differences between the matrices I have slight differences : no null mean with min and max ranging from $-0.3$ to $0.3$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any idea? Usually you read that, for dichotomous item, values are closed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Further, I have missing values because, due to very hard and very easy items, some modalities are never observed together. In this case do you have any good proposition (and reasons) to handle this problem? Suppress these items, Imputation, ... &lt;/p&gt;&#xA;" OwnerUserId="1154" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-09-27T10:27:38.730" LastActivityDate="2010-09-28T08:43:41.500" Title="Differences between tetrachoric and Pearson correlation" Tags="&lt;correlation&gt;&lt;factor-analysis&gt;&lt;psychometrics&gt;&lt;psychology&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="1" />
  <row Id="3087" PostTypeId="2" ParentId="3082" CreationDate="2010-09-26T16:23:19.830" Score="5" ViewCount="" Body="&lt;p&gt;It is broken, although if you perform enough shuffles it can be an excellent approximation (as the previous answers have indicated).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just to get a handle on what's going on, consider how often your algorithm will generate shuffles of a $k$ element array in which the first element is fixed, $k \ge 2$.  When permutations are generated with equal probability, this should happen $1/k$ of the time.  Let $p_n$ be the relative frequency of this occurrence after $n$ shuffles with your algorithm.  Let's be generous, too, and suppose you are actually selecting &lt;em&gt;distinct&lt;/em&gt; pairs of indexes uniformly at random for your shuffles, so that each pair is selected with probability $1/{k \choose 2}$ = $2/\left( k (k-1) \right)$.  (This means there are no &quot;trivial&quot; shuffles wasted.  On the other hand, it totally breaks your algorithm for a two-element array, because you alternate between fixing the two elements and swapping them, so if you stop after a predetermined number of steps, there is no randomness to the outcome whatsoever!)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This frequency satisfies a simple recurrence, because the first element is found in its original place after $n+1$ shuffles in two disjoint ways.  One is that it was fixed after $n$ shuffles and the next shuffle does not move the first element.  The other is that it was moved after $n$ shuffles but the $n+1^{st}$ shuffle moves it back.  The chance of &lt;em&gt;not&lt;/em&gt; moving the first element equals ${k-1 \choose 2}/{k \choose 2}$ = $(k-2)/k$, whereas the chance of moving the first element back equals $1/{k \choose 2}$ = $2/\left( k (k-1) \right)$.  Whence:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p_0 =1$$ because the first element starts out in its rightful place;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p_{n+1} = \frac{k-2}{k} p_n + \frac{2}{k(k-1)} \left( 1 - p_n \right).$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The solution is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p_n = 1/k + \left( \frac{k-3}{k-1} \right) ^n \frac{k-1}{k}.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Subtracting $1/k$, we see that the frequency is wrong by $\left( \frac{k-3}{k-1} \right) ^n \frac{k-1}{k}$.  For large $k$ and $n$, a good approximation is $\frac{k-1}{k} \exp(-\frac{2n}{k-1})$. This shows that the error &lt;em&gt;in this particular frequency&lt;/em&gt; will decrease exponentially with the number of swaps relative to the size of the array ($n/k$), indicating it will be difficult to detect with large arrays if you have made a relatively large number of swaps--but the error is always there.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is difficult to provide a comprehensive analysis of the errors in all frequencies.  It's likely they will behave like this one, though, which shows that &lt;em&gt;at a minimum&lt;/em&gt; you would need $n$ (the number of swaps) to be large enough to make the error acceptably small.  An approximate solution is &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$n \gt \frac{1}{2} \left(1 - (k-1) \log(\epsilon) \right)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\epsilon$ should be very small compared to $1/k$.  This implies $n$ should be several times $k$ for even crude approximations (&lt;em&gt;i.e.&lt;/em&gt;, where $\epsilon$ is on the order of $0.01$ times $1/k$ or so.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All this begs the question: why would you choose to use an algorithm that is not quite (but only approximately) correct, employs exactly the same techniques as another algorithm that is provably correct, and yet which requires more computation?&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Edit&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;Thilo's comment is apt (and I was hoping nobody would point this out, so I could be spared this extra work!).  Let me explain the logic.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;If you make sure to generate actual swaps each time, you're utterly screwed.  The problem I pointed out for the case $k=2$ extends to all arrays.  Only half of all the possible permutations can be obtained by applying an even number of swaps; the other half is obtained by applying an odd number of swaps.  Thus, in this situation, you can never generate anywhere near a uniform distribution of permutations (but there are so many possible ones that a simulation study for any sizable $k$ will be unable to detect the problem).  That's really bad.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Therefore it is wise to generate swaps at random by generating the two positions independently at random.  This means there is a $1/k$ chance each time of swapping an element with itself; that is, of doing nothing.  This process effectively slows down the algorithm a little bit: after $n$ steps, we expect only about $\frac{k-1}{k} N \lt N$ true swaps to have occurred.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Notice that the size of the error decreases monotonically with the number of distinct swaps.  Therefore, conducting fewer swaps on average also &lt;em&gt;increases&lt;/em&gt; the error, on average.  But this is a price you should be willing to pay in order to overcome the problem described in the first bullet.  Consequently, my error estimate is conservatively low, approximately by a factor of $(k-1)/k$.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I also wanted to point out an interesting apparent exception: a close look at the error formula suggests that there is &lt;em&gt;no&lt;/em&gt; error in the case $k=3$.  This is not a mistake: it is correct.  However, here I have examined only one statistic related to the uniform distribution of permutations.  The fact that the algorithm can reproduce this one statistic when $k=3$ (namely, getting the right frequency of permutations that fix any given position) does not guarantee the permutations have indeed been distributed uniformly.  Indeed, after $2n$ actual swaps, the only possible permutations that can be generated are $(123)$, $(321)$, and the identity.  Only the latter fixes any given position, so indeed exactly one-third of the permutations fix a position.  But half the permutations are missing!  In the other case, after $2n+1$ actual swaps, the only possible permutations are $(12)$, $(23)$, and $(13)$.  Again, exactly one of these will fix any given position, so again we obtain the correct frequency of permutations fixing that position, but again we obtain only half of the possible permutations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This little example helps reveal the main strands of the argument: by being &quot;generous&quot; we conservatively &lt;em&gt;underestimate&lt;/em&gt; the error rate for one particular statistic.  Because that error rate is nonzero for all $k \ge 4$, we see that the algorithm is broken.  Furthermore, by analyzing the decay in the error rate &lt;em&gt;for this statistic&lt;/em&gt; we establish a &lt;em&gt;lower bound&lt;/em&gt; on the number of iterations of the algorithm needed to have any hope at all of approximating a uniform distribution of permutations.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-09-27T15:21:15.717" LastActivityDate="2010-09-27T15:21:15.717" CommentCount="3" />
  <row Id="3088" PostTypeId="2" ParentId="3080" CreationDate="2010-09-26T16:26:44.410" Score="5" ViewCount="" Body="&lt;p&gt;I would hold off on considering meta-analysis until you've scrutinized sources if potential bias or variation in the target populations. If these are studies of treatment effects, was treatment randomly assigned? Were there deviations from the protocol? Was there noncompliance? Is there missing outcome data? Were the samples drawn from the same frame? Was there refusal to participate? Implementation errors?  Were standard errors computed correctly, accounting for clustering and robust to various parametric assumptions? Only after you have answered these questions do I think meta-analysis issues start to enter the picture.  It must be rare that for any two studies meta-analysis is appropriate, unless you are willing to make sone heroic assumptions.&lt;/p&gt;&#xA;" OwnerUserId="1424" LastActivityDate="2010-09-26T16:26:44.410" CommentCount="3" />
  <row Id="3089" PostTypeId="2" ParentId="3086" CreationDate="2010-09-26T16:30:04.203" Score="3" ViewCount="" Body="&lt;p&gt;Tetrachoric coefficient and Phi coefficient are indeed different. The tetrachoric coefficient is suitable for the following problem: Suppose there are two judges who judge cakes, say, on some continuous scale, then based on a fixed, perhaps unknown, cutoff, pronounce the cakes as &quot;bad&quot; or &quot;good&quot;. Suppose the latent continuous metric of the two judges has correlation coefficient $\rho$. Now generate 300 cakes, have both judges taste each of them, and generate a 2x2 contingency table of &quot;judge 1 bad/good&quot; vs &quot;judge 2 bad/good&quot;. Based on the data in this contingency table, the sample tetrachoric coefficient is an estimator (I believe it is the MLE, in fact), of the 'latent' correlation $\rho$. Note that the cutoffs employed by the two judges need not be known.  The Phi coefficient views the pronouncements &quot;bad&quot;, &quot;good&quot; themselves as the variable of interest, coded as 0/1, and is the sample Pearson coefficient of the 0/1 data.  These are not the same.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;edit&lt;/strong&gt; in response to @pbneau's comments: my suspicion was that the tetrachoic and phi coefficients would diverge in the limit cases: as $\rho \to 0$ and as the cutoffs for the latent rating move away from the mean ratings. I tested this with my own code (in Matlab) for tetrachoric and phi coefficient. I tested with zero mean, unit variance Gaussian latent ratings with population correlations of 0.01 and 0.25, and with cutoffs of 0,0 and 1.5,-0.5.  I ran 2048 experiments, each with 2048 'cakes'. The scatter fits for tetrachoric versus phi are shown here:&#xA;&lt;img src=&quot;http://i.imgur.com/w5vg8.jpg&quot; alt=&quot;bunch o scatter fits&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(looks like the image upload thing is not working; top row is $c1 = c2 = 0$, bottom is $c1 = 1.5, c2 = -0.5$, left column is $\rho = 0.01,$ right column is $\rho = 0.25$. The best fits along top row are $\rho^* = 1.5 \phi + 0$, $\rho^* = 1.4 \phi + 0.01$, along the bottom row, $\rho^* = 2.2 \phi$ and $\rho^* = 3.1 \phi - 0.04$. perhaps I can get this image hosted somewhere else that doesn't squash them so much...)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not sure you can read the text on these (the preview looks bad); the upshot is that when the cutoffs are at the population mean, and thus the contingency tables are 'balanced' row-wise and col-wise (top row of plots), you get good correlation between the two metrics, but the tetrachoric tends to be a bit larger than phi. When the cutoffs are somewhat imbalanced, you get slightly worse correlation between the metrics, and the phi appears to 'shink' towards zero.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus my initial intuition was only half correct: the worst case appears to be as the latent $\rho$ moves &lt;em&gt;away&lt;/em&gt; from zero, and the cutoffs move away from the latent means.  &lt;/p&gt;&#xA;" OwnerUserId="795" LastEditorUserId="795" LastEditorDisplayName="" LastEditDate="2010-09-28T00:12:55.770" LastActivityDate="2010-09-28T00:12:55.770" CommentCount="7" />
  <row Id="3090" PostTypeId="2" ParentId="725" CreationDate="2010-09-26T16:36:50.307" Score="3" ViewCount="" Body="&lt;p&gt;The best place to look for free/open source capabilities of this nature is GRASS GIS.  The image processing manual is at &lt;a href=&quot;http://grass.fbk.eu/gdp/imagery/grass4_image_processing.pdf&quot; rel=&quot;nofollow&quot;&gt;http://grass.fbk.eu/gdp/imagery/grass4_image_processing.pdf&lt;/a&gt; .  Because this is constantly undergoing development, it would be worthwhile posting an inquiry on one of the GRASS user lists (found through links on the home page at &lt;a href=&quot;http://grass.fbk.eu/&quot; rel=&quot;nofollow&quot;&gt;http://grass.fbk.eu/&lt;/a&gt; ).&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-09-26T16:36:50.307" CommentCount="2" />
  <row Id="3091" PostTypeId="1" AcceptedAnswerId="3095" CreationDate="2010-09-26T18:17:03.283" Score="4" ViewCount="299" Body="&lt;p&gt;I've seen a little bit here about the difference between statistical inference for random samples, and what happens when we actually have population data. Most arguments seem to suggest you &quot;never actually have the population&quot; and the population data you think you have represents some unobservable super population which is the data-generating process. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But say, we have a state, covered by counties, and we have data on some environmental aspect of each county, for example, the total area of forest. We want to see if some binary county-level outcome is related to forest - let's say presence or absence of a disease (thinking Lyme disease here). As far as I can tell, this qualifies as &quot;population&quot; level data. Would running a regular logistic regression, and the normal tests on parameters, yield correct estimates? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One thing that comes into this is whether the &quot;samples&quot; are independent. In this case, forest in county A is probably related to forest in the county B next door, meaning a reduction in the degrees of freedom and imprecision in standard error estimates (as I understand it). But even if we had totally independent data, would standard, sample-based statistical inference be appropriate here? &lt;/p&gt;&#xA;" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-26T19:17:10.030" LastActivityDate="2010-09-26T20:32:55.893" Title="Regression analysis and parameter estimates with populations" Tags="&lt;hypothesis-testing&gt;&lt;regression&gt;&lt;logistic-regression&gt;" AnswerCount="3" CommentCount="1" FavoriteCount="2" />
  <row Id="3092" PostTypeId="2" ParentId="3091" CreationDate="2010-09-26T18:47:29.700" Score="1" ViewCount="" Body="&lt;p&gt;I work in a related area using what is called the Population Approach, in pharmacometrics. Basically you have a sample of individuals from the general population, and a moderate-size sample of observations of each individual, plus demographic covariates for these individuals.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To model them, we want to make levels of model. We want to make an overall model that applies to the individuals that we hope will be representative of the entire population. The parameters of this overall model are called &quot;fixed effects&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Below that level, we have a model of each individual, because we expect individuals to differ from each other. So we have additional parameters that are different for each individual, and these are called &quot;random effects&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A key part to be estimated in the overall model is a measure of the variability of the random effects. I.e. we know the individuals will differ, but we want to model that variability.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A further level of modeling is the variability of individual observations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Applying this to your field is not something I fully understand, but I would recommend WinBugs as a modeling tool, and a book &quot;Markov Chain Monte Carlo in Practice&quot; by Gilks, Richardson, and Spiegelhalter. There are example problems in there that look like they might be applicable to your problem.&lt;/p&gt;&#xA;" OwnerUserId="1270" LastActivityDate="2010-09-26T18:47:29.700" />
  <row Id="3093" PostTypeId="2" ParentId="3091" CreationDate="2010-09-26T19:46:28.427" Score="2" ViewCount="" Body="&lt;p&gt;What you describe seems to refer to a particular case of multilevel modeling, where data are organized into a hierarchical structure; in your case, forests (1st level unit) nested in counties nested in states, but see &lt;a href=&quot;http://stats.stackexchange.com/q/1995/930&quot;&gt;Under what conditions should one use multilevel/hierarchical analysis?&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, the &quot;particular&quot; case comes from the fact that you want to account for the spatial proximity which might be a vector for the propagation of &lt;a href=&quot;http://en.wikipedia.org/wiki/Lyme_disease&quot; rel=&quot;nofollow&quot;&gt;Lyme disease&lt;/a&gt; (correct me if I am wrong), as is done in epidemiology where one is interested in studying the geography of infectious disease. In the usual case, we can use so-called &lt;em&gt;spatial models&lt;/em&gt; like the &lt;em&gt;multiple membership model&lt;/em&gt; or the &lt;em&gt;conditional autoregressive model&lt;/em&gt;, among others.&#xA;I enclose a couple of references about these approaches at the end, but I think you will find more references by looking at related studies in ecology or epidemiology. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, I think that you may pay a particular attention at the following paper of Langford et al. which features multilevel modeling with spatially correlated data:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Langford, IH, Leyland, AHL, Rasbash,&#xA;  and Goldstein, H (1999). &lt;a href=&quot;http://www.cmm.bristol.ac.uk/team/HG_Personal/multilevel-spatial-models.pdf&quot; rel=&quot;nofollow&quot;&gt;Multilevel&#xA;  modelling of the geographical&#xA;  distributions of diseases&lt;/a&gt;.&#xA;  &lt;em&gt;Journal of Royal Statistical Society C&lt;/em&gt;, &lt;em&gt;48&lt;/em&gt;, 253-268.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Harvey Goldstein is the author of an excellent book on multilevel modeling, &lt;a href=&quot;http://www.cmm.bristol.ac.uk/team/HG_Personal/multbook1995.pdf&quot; rel=&quot;nofollow&quot;&gt;Multilevel Statistical Models&lt;/a&gt; (the 2nd edition is available for free). Finally, the book of Andrew Gelman, &lt;a href=&quot;http://www.stat.columbia.edu/~gelman/arm/&quot; rel=&quot;nofollow&quot;&gt;Data Analysis Using Regression and Multilevel/Hierarchical Models&lt;/a&gt;, may provide additional clues about hierarchical/multilevel modeling.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;About software, I know there is the R &lt;a href=&quot;http://cran.r-project.org/web/packages/spdep/index.html&quot; rel=&quot;nofollow&quot;&gt;spdep&lt;/a&gt; package for modeling spatially correlated outcomes, but there are some examples of analysis of spatial hierachical data with WinBUGS on the &lt;a href=&quot;http://www.mrc-bsu.cam.ac.uk/bugs/weblinks/webresource.shtml&quot; rel=&quot;nofollow&quot;&gt;BUGS Project&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;References&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Browne, W.J., Goldstein, H. and Rasbash, J. (2001) &lt;a href=&quot;http://www.cmm.bristol.ac.uk/team/HG_Personal/multilevel-spatial-models.pdf&quot; rel=&quot;nofollow&quot;&gt;Multiple membership multiple classification (MMMC) models&lt;/a&gt;.  &lt;em&gt;Statistical Modelling&lt;/em&gt;, &lt;em&gt;1&lt;/em&gt;, 103-124.&lt;/li&gt;&#xA;&lt;li&gt;Lichstein, JW, Simons, TR, Shriner, SA, and Franzreb, KE (2002). &lt;a href=&quot;http://www4.ncsu.edu/~simons/Lichstein%20et%20al%20Ecol%20Monog.pdf&quot; rel=&quot;nofollow&quot;&gt;Spatial autocorrelation and autoregressive models in ecology&lt;/a&gt;. &lt;em&gt;Ecological Monographs&lt;/em&gt;, &lt;em&gt;72(3)&lt;/em&gt;, 445-463.&lt;/li&gt;&#xA;&lt;li&gt;Feldkircher, M (2007). &lt;a href=&quot;http://feldkircher.gzpace.net/Seminar%20Papers/spatial_car_feldkircher.pdf&quot; rel=&quot;nofollow&quot;&gt;A Spatial CAR Model applied to a Cross-Country Growth Regression&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;li&gt;Lawson, AB, Browne, WJ, and Vidal Rodeiro, CL (2003). &lt;em&gt;Disease mapping with WinBUGS and MLwiN&lt;/em&gt;. John Wiley &amp;amp; Sons.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-09-26T19:46:28.427" CommentCount="5" />
  <row Id="3094" PostTypeId="2" ParentId="3052" CreationDate="2010-09-26T20:15:09.507" Score="0" ViewCount="" Body="&lt;p&gt;Some of the &lt;a href=&quot;http://www.bioconductor.org&quot; rel=&quot;nofollow&quot;&gt;Bioconductor&lt;/a&gt;'s packages (e.g., &lt;a href=&quot;http://www.bioconductor.org/packages/2.3/bioc/html/ShortRead.html&quot; rel=&quot;nofollow&quot;&gt;ShortRead&lt;/a&gt;, &lt;a href=&quot;http://www.bioconductor.org/packages/2.2/bioc/html/Biostrings.html&quot; rel=&quot;nofollow&quot;&gt;Biostrings&lt;/a&gt;, &lt;a href=&quot;http://www.bioconductor.org/packages/2.3/bioc/html/BSgenome.html&quot; rel=&quot;nofollow&quot;&gt;BSgenome&lt;/a&gt;, &lt;a href=&quot;http://www.bioconductor.org/packages/2.3/bioc/html/IRanges.html&quot; rel=&quot;nofollow&quot;&gt;IRanges&lt;/a&gt;, &lt;a href=&quot;http://www.bioconductor.org/packages/2.6/bioc/html/genomeIntervals.html&quot; rel=&quot;nofollow&quot;&gt;genomeIntervals&lt;/a&gt;) offer facilities for dealing with genome positions or coverage vectors, e.g. for &lt;a href=&quot;http://en.wikipedia.org/wiki/Chip-Sequencing&quot; rel=&quot;nofollow&quot;&gt;ChIP-seq&lt;/a&gt; and identifying enriched regions. As for the other answers, I agree that any method relying on ordered observations with some threshold-based filter would allow to isolate low signal within a specific bandwith.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maybe you can also look at the methods used to identify so-called &quot;islands&quot;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Zang, C, Schones, DE, Zeng, C, Cui, K,&#xA;  Zhao, K, and Peng, W (2009). &lt;a href=&quot;http://bioinformatics.oxfordjournals.org/content/25/15/1952.full.pdf+html&quot; rel=&quot;nofollow&quot;&gt;A&#xA;  clustering approach for identification&#xA;  of enriched domains from histone&#xA;  modification ChIP-Seq data&lt;/a&gt;.&#xA;  Bioinformatics, &lt;em&gt;25(15)&lt;/em&gt;, 1952-1958.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-09-26T20:15:09.507" />
  <row Id="3095" PostTypeId="2" ParentId="3091" CreationDate="2010-09-26T20:32:55.893" Score="3" ViewCount="" Body="&lt;p&gt;I like what chl has written.  Despite that, I am moved to discuss whether this situation necessarily requires a complicated model.  But first, let's begin with responses to some comments.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(1) You don't lose any degrees of freedom due to spatial correlation of forest cover.  This is an explanatory variable, not the one you're trying to model.  You might lose &quot;degrees of freedom&quot; when the &lt;em&gt;residuals&lt;/em&gt; of the &lt;em&gt;dependent&lt;/em&gt; variable exhibit spatial autocorrelation.  Such correlation is not necessarily a given even when a map of the dependent variable itself suggests strong spatial correlation.  The reason is that the correlation in the map likely derives from the correlation in the forest cover (and other spatially distributed covariates).  Remember, in such models you don't ask whether the &quot;samples are independent&quot;--it's usually obvious they are not--but whether any deviations in them from their modeled values are independent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(2) Therefore, a conditional autoregressive model might not be necessary.  I think this modeling choice would be attractive only if you want to test a theory of &lt;em&gt;contagion&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now to answer the original question: yes, first run an ordinary logistic (or Poisson) model, because as a general principle it's good to try simple (but reasonable) models first.  &lt;em&gt;Provided its residuals do not exhibit strong spatial correlation,&lt;/em&gt; you are then ok with the results.  If there is evidence of correlation and evidence that it will appreciably affect your answers (coefficients, predictions, or whatever), consider using a generalized linear geostatistical model (GLM).  These are described in Diggle &amp;amp; Riberio, &lt;em&gt;Model-based Geostatistics&lt;/em&gt; (a relatively inexpensive and accessible text), which itself documents several R packages for making the estimates and some associated EDA tools (geoR and geoRglm).  The GLM approach lets you simultaneously fit your model and assess the degree of spatial autocorrelation.  The principal limitations I have found in these packages are (1) they don't handle anisotropy well--you can detect it but it's hard to incorporate it in a model--and (2) they don't have a provision for nested variograms, which somewhat limits your ability to model the spatial correlation.  Both of these are no problem for smallish datasets, because you need (typically) hundreds to thousands of observations, or more, to model correlation at this level of detail.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, a word about the &quot;population&quot; question.  I assume you are interested in more than a mere description of the data: you seek information about a possible association between disease and other observable factors.  Even when you have a comprehensive description of the data for a spatial region, it still does not act like a census, because the outcomes could have turned out otherwise.  Next year, with identical forest cover, there will be a slightly different pattern of disease.  In other regions of the country or the world and at other times, exactly the same combinations of explanatory variable values are likely to produce varying rates of disease.  Thus, you're modeling a &lt;em&gt;process&lt;/em&gt;, not a population.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-09-26T20:32:55.893" CommentCount="2" />
  <row Id="3096" PostTypeId="2" ParentId="2299" CreationDate="2010-09-26T22:37:18.040" Score="1" ViewCount="" Body="&lt;p&gt;Actually, Benford's Law is an incredibly powerful method.  This is because the Benford's frequency distribution of first digit is applicable to all sorts of data set that occur in the real or natural world.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You are right that you can use Benford's Law in only certain circumstances.  You say that the data has to have a uniform log distribution.  Technically, this is absolutely correct.  But, you could describe the requirement in a much simpler and lenient way.  All you need is that the data set range crosses at least one order of magnitude.  Let's say from 1 to 9 or 10 to 99 or 100 to 999.  If it crosses two orders of magnitude, you are in business.  And, Benford's Law should be pretty helpful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The beauty of Benford's Law is that it helps you narrow your investigation really quickly on the needle(s) within the hay stack of data.  You look for the anomalies whereby the frequency of first digit is much different than Benford frequencies.  Once you notice that there are two many 6s, you then use Benford's Law to focus on just the 6s; but, you take it now to the first two digits (60, 61, 62, 63, etc...).  Now, maybe you find out there are a lot more 63s then what Benford suggest (you would do that by calculating Benford's frequency: log(1+1/63) that gives you a value close to 0%).  So, you use Benford to the first three digits.  By the time you find out there are way too many 632s (or whatever by calculating Benford's frequency: log (1+1/632)) than expected you are probably on to something.  Not all anomalies are frauds.  But, most frauds are anomalies.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the data set that Marc Hauser manipulated are natural unconstrained data with a related range that was wide enough, then Benford's Law would be a pretty good diagnostic tool.  I am sure there are other good diagnostic tools also detecting unlikely patterns and by combining them with Benford's Law you could most probably have investigated the Marc Hauser affair effectively (taking into consideration the mentioned data requirement of Benford's Law).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I explain Benford's Law a bit more in this short presentation that you can see here: &#xA;&lt;a href=&quot;http://www.slideshare.net/gaetanlion/benfords-law-4669483&quot; rel=&quot;nofollow&quot;&gt;http://www.slideshare.net/gaetanlion/benfords-law-4669483&lt;/a&gt;          &lt;/p&gt;&#xA;" OwnerUserId="1329" LastEditorUserId="1329" LastEditorDisplayName="" LastEditDate="2010-09-26T22:53:23.677" LastActivityDate="2010-09-26T22:53:23.677" />
  <row Id="3098" PostTypeId="2" ParentId="2948" CreationDate="2010-09-27T01:33:45.997" Score="1" ViewCount="" Body="&lt;p&gt;I just came across the tnet package for R. The creator seems to be researching on community discovery in weighted and bipartite (two-mode) graphs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://opsahl.co.uk/tnet/content/view/15/27/&quot; rel=&quot;nofollow&quot;&gt;http://opsahl.co.uk/tnet/content/view/15/27/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have not yet use it.&lt;/p&gt;&#xA;" LastActivityDate="2010-09-27T01:33:45.997" />
  <row Id="3099" PostTypeId="2" ParentId="3080" CreationDate="2010-09-27T03:26:45.223" Score="3" ViewCount="" Body="&lt;p&gt;I think &lt;a href=&quot;http://stats.stackexchange.com/questions/3080/how-to-detect-which-one-is-the-better-study-when-they-give-you-conflicting-result/3088#3088&quot;&gt;Jeromy's answer&lt;/a&gt; is sufficient if you are examining two experimental studies or an actual meta-analysis. But often times we are faced with examining two non-experimental studies, and are tasked with assessing the validity of those two disparate findings.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As &lt;a href=&quot;http://stats.stackexchange.com/questions/3080/how-to-detect-which-one-is-the-better-study-when-they-give-you-conflicting-result/3088#3088&quot;&gt;Cyrus's grocery list of questions&lt;/a&gt; suggests, the topic itself is not amenable to short response, and whole books are in essence aimed to address such a question. For anyone interested in conducting research on non-experimental data, I would highly suggest you read &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://books.google.com/books?id=o7jaAAAAMAAJ&amp;amp;q&quot; rel=&quot;nofollow&quot;&gt;Experimental and quasi-experimental designs for generalized causal inference&lt;/a&gt; by William R. Shadish, Thomas D. Cook, Donald Thomas Campbell (Also I have heard that the older versions of this text are just as good).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Several items Jeromy referred to (bigger sample sizes, and greater methodological rigour), and everything that Cyrus mentions would be considered what Campbell and Cook refer to as &quot;Internal Validity&quot;. These include aspects of the research design and the statistical methods used to assess the relationship between X and Y. In particular as critics we are concerned about aspects of either that could bias the results, and diminish the reliability of the findings. As this is a forum devoted to statistical analysis, much of the answers are centered around statistical methods to ensure unbiased estimates of whatever relationship you are assessing. But their are other aspects of the research design unrelated to statistical analysis that diminish the validity of the findings no matter what rigourous lengths one goes to in their statistical analysis (such as Cyrus's mention of several aspects of experiment fidelity can be addressed but not solved with statistical methods, and if they occur will always diminish the validity of the studies results). There are many other aspects of internal validity that become crucial to assess in comparing results of non-experimental studies that are not mentioned here, and aspects of research designs that can distinguish reliability of findings. I don't think it is quite appropriate to go into too much detail here, but I would often take the results of a quasi-experimental study (such as an interrupted time series or a matched case-control) more seriously than I would a study that is not quasi experimental, regardless of the other aspects Jeromy or Cyrus mentioned (of course within some reason).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Campbell and Cook also refer to the &quot;external validity&quot; of studies. This aspect of research design is often much smaller in scope, and does not deserve as much attention as internal validity. External validity essentially deals with the generalizability of the findings, and I would say laymen can often assess external validity reasonably well as long as they are familiar with the subject. Long story short read Shadish's, Cook's and Campbell's book.&lt;/p&gt;&#xA;" OwnerUserId="1036" LastActivityDate="2010-09-27T03:26:45.223" />
  <row Id="3100" PostTypeId="1" CreationDate="2010-09-27T06:24:37.780" Score="3" ViewCount="143" Body="&lt;p&gt;I previously asked how to estimate the latent potential of a runner who &lt;a href=&quot;http://stats.stackexchange.com/questions/2966/estimating-latent-performance-potential-based-on-a-sequence-of-observations&quot;&gt;ran the 100 metres each day for 200 days&lt;/a&gt;. Latent skill was defined as &quot;the latent time it would take the individual to run if they (a) applied maximal effort; and (b) had a reasonably good run for them (i.e., no major problems with the run; but still a typical run).&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now assume that I estimated latent skill for the 100 metres for each of the 200 days, but that I also had data on the same 200 days but this time on running the 400 metres. Obviously I could repeat whatever process I adopted for the 100 metres to form an estimate of latent skill for the 400 metres at each of the 200 time points. In both cases I would expect the time to complete the runs to generally get faster with practice, but that raw data would vary from day to day.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to quantify the degree of consistency of the two curves. I don't really want to quantify the consistency of the observed data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If it makes a difference, the two methods I were considering using for estimating the effect of time, were nonlinear regression and &lt;a href=&quot;http://en.wikipedia.org/wiki/Isotonic_regression&quot; rel=&quot;nofollow&quot;&gt;isotonic regression&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My question:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Thus, what is a good way to quantify and calculate the consistency of the fitted curves for the 100 and 400 metres?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Initial thoughts:&lt;/strong&gt;&#xA;I had a few initial thoughts:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;estimate the fitted values for both curves and correlate the fitted values&lt;/li&gt;&#xA;&lt;li&gt;Use a parametric model like $\theta_1 \exp(-\theta_2t) + \theta_3 + \epsilon$ ($t$ is an index of day) and then quantify the degree to which constraining $\theta_2$ (the parameter that determines shape) to be equal across the 100 and 400 metres would lead to poorer fit.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-09-27T06:34:30.020" LastActivityDate="2010-09-27T11:42:29.113" Title="Quantifying the degree of consistency of two fitted curves" Tags="&lt;regression&gt;&lt;curve-fitting&gt;&lt;nonlinear-regression&gt;" AnswerCount="1" CommentCount="4" />
  <row Id="3101" PostTypeId="1" AcceptedAnswerId="3760" CreationDate="2010-09-27T07:25:32.120" Score="6" ViewCount="158" Body="&lt;p&gt;When trying to assess a validity of a claim relying on statistics, I was taught (in the school of epidemiology) that the scale to use is “&lt;a href=&quot;http://www.r-statistics.com/2010/02/for-fun-correlation-of-us-state-with-the-number-of-clicks-on-online-banners/&quot; rel=&quot;nofollow&quot;&gt;the pyramid of evidence&lt;/a&gt;“&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/EOI9q.gif&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, when conducting a discussion in the field of economics or political science, it is often very difficult (to impossible) to recreate a political situation so to allow experimenting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since this is the case, my questions are: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;can this be mitigated in any way?&lt;/li&gt;&#xA;&lt;li&gt;In what way can (or does) political/economic disciplines build their arguments (using statistics) in a valid way? &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The motivation&lt;/strong&gt; for my question started from reading a discussion about the privatization of the academic system in Israel.  e.g: Should the charging of students be the same for all disciplines or different. Should the running of the university be performed by the academic staff or by outside managers - and so on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One things that appears to happen in the discussion is that the people supporting the privatization (economists mostly) seems to be using statistics of all sorts to support their claims.  While the people from the other side of the argument are not as equipped with them (usually people from the humanities).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The economists seems to blame the humanities of not using numeric data.  While the humanities accuse the economists of being to speculative - and that the data they bring is open to too many interpretations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am trying to understand if these two groups of disciplines can create a more fruitful dialog, or is it just not possible due to the complexity of the subject matter, and the limitation of our control over it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The following discussion has a nice link in it, but it was stopped quite early:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://stats.stackexchange.com/questions/685/lies-damn-lies-and-statistics-closed&quot;&gt;http://stats.stackexchange.com/questions/685/lies-damn-lies-and-statistics-closed&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Also a bunch of good relevant discussion where introduced here:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://stats.stackexchange.com/search?q=causal-inference&quot;&gt;http://stats.stackexchange.com/search?q=causal-inference&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;p.s: Due to the subjectivity of the topic - I am marking this as community wiki.&lt;/p&gt;&#xA;" OwnerUserId="253" LastActivityDate="2010-10-19T14:34:44.807" CommunityOwnedDate="2010-09-27T07:25:32.120" Title="How much can the &quot;pyramid of evidence&quot; be applied to economics and political sciences?" Tags="&lt;causal-inference&gt;&lt;inference&gt;" AnswerCount="1" CommentCount="3" FavoriteCount="1" />
  <row Id="3102" PostTypeId="2" ParentId="3100" CreationDate="2010-09-27T08:59:59.863" Score="3" ViewCount="" Body="&lt;p&gt;My understanding is that your sample size is 200 observations for each running length (i.e. the sample size is such that doubling the number of observations has a non trivial effect on accuracy). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Denote $y_1$ the 100 meters series ($y_2$ the 400 meters series). I suggest pooling both series unto $y$ and adding a dummy $l_i=I(i\in y_1)$ (where $I()$ is the indicator function) and performing an isotonic regression of $\log(y)$ on $date$ and the dummies ($l$ as well as the $z$ controling for injuries, running style ect....). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because of the $\log()$ transformation on the left hand side, the value of the coefficient of $l$ can be understood as the average percentage increase in running time, al else equal, due to running a 4 time longer length. The doubling of sample size will improve accuracy since you have in effect only added a single parameter to the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, re-reading my previous answer, i noticed i had linked to the wrong article. By the same author, the correct reference is 'Bayesian Isotonic Regression and Trend Analysis',&#xA;B. Neelon, D. B. Dunson (the previous link was to the version of there approach adapted to the case of binary dependant variable). A non gated version of the article is &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.70.5142&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="603" LastEditorUserId="603" LastEditorDisplayName="" LastEditDate="2010-09-27T11:42:29.113" LastActivityDate="2010-09-27T11:42:29.113" CommentCount="2" />
  <row Id="3104" PostTypeId="1" AcceptedAnswerId="3108" CreationDate="2010-09-27T10:11:37.493" Score="6" ViewCount="280" Body="&lt;p&gt;I like introducing probability by discussing the &lt;a href=&quot;http://en.wikipedia.org/wiki/Boy_or_Girl_paradox&quot; rel=&quot;nofollow&quot;&gt;Boy or Girl&lt;/a&gt; or &lt;a href=&quot;http://en.wikipedia.org/wiki/Bertrand_paradox_%28probability%29&quot; rel=&quot;nofollow&quot;&gt;Bertrand&lt;/a&gt; paradox.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What other (short) problem/game provides a motivating introduction to probability? (&lt;em&gt;One answer per response, please&lt;/em&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S. This is about a gentle introduction to probability, but in my opinion it is relevant for statistics teaching as it allows to further discuss about discrete events, Bayes's theorem, probabilistic/measurable space, etc.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-20T18:22:17.697" LastActivityDate="2011-05-20T21:19:25.467" CommunityOwnedDate="2010-09-27T10:11:37.493" Title="What is your favorite problem for an introduction to probability?" Tags="&lt;teaching&gt;" AnswerCount="6" FavoriteCount="3" />
  <row Id="3105" PostTypeId="1" CreationDate="2010-09-27T10:19:33.873" Score="4" ViewCount="260" Body="&lt;p&gt;I wanted to find out what kind of different usages of stochastic processes theory in EE &amp;amp; CS are out there. For example, I find these kinds of usages interesting:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;using stochastic signal as carrier that is modulated by the information signal for communication&lt;/li&gt;&#xA;&lt;li&gt;using stochastic process analysis for improving photographed pictures after long exposition&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;It would be good if every example would be in separate answer, so it would be possible to up-vote those that are 'the best'. I would classify good example as one that:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;truly takes advantage of probability and statistics theory&lt;/li&gt;&#xA;&lt;li&gt;has greater 'usefulness' then other examples&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="1429" LastActivityDate="2010-09-28T01:25:32.580" CommunityOwnedDate="2010-09-27T12:01:56.200" Title="What are the examples for stochastic processes in Electrical Engineering and Computer Science?" Tags="&lt;stochastic-processes&gt;" AnswerCount="3" />
  <row Id="3106" PostTypeId="2" ParentId="3105" CreationDate="2010-09-27T10:36:49.463" Score="3" ViewCount="" Body="&lt;p&gt;What about monitoring network congestion. An underlying stochastic process is assumed to be driving the congestion, and the process has reached an equilibrium state and is stationary. That is, process characteristics such as the mean level does not depend on time. You can then use this the stochastic process to give you an idea of when you check the network for possible congestion.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some recent papers:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;N. Dueld, C. Lund, and M. Thorup. &lt;a href=&quot;http://www.usenix.org/event/imc05/tech/duffield.html&quot; rel=&quot;nofollow&quot;&gt;Optimal combination of sampled network measurements.&lt;/a&gt; Proceedings of IMC '05, Internet Measurement Conference, 91-104, 2005.&lt;/li&gt;&#xA;&lt;li&gt;B. M. Parker, S. G. Gilmour, and J. Schormans. &lt;a href=&quot;http://dx.doi.org/10.1049/iet-com.2008.0075&quot; rel=&quot;nofollow&quot;&gt;Measurement of packet loss probability by optimal design of packet probing experiments&lt;/a&gt;. IET Communications, 3:979-991, 2009.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="8" LastActivityDate="2010-09-27T10:36:49.463" CommunityOwnedDate="2010-09-27T12:01:56.200" />
  <row Id="3107" PostTypeId="2" ParentId="3104" CreationDate="2010-09-27T10:45:17.213" Score="6" ViewCount="" Body="&lt;p&gt;A standard example is the &lt;a href=&quot;http://en.wikipedia.org/wiki/Monty_Hall_problem&quot; rel=&quot;nofollow&quot;&gt;Monty-Hall&lt;/a&gt; game. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's how I approach this example:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Give the class sets of three cards and get them to play the game in pairs.&lt;/li&gt;&#xA;&lt;li&gt;Each pair plays the game following a particular strategy, i.e. always switching doors.&lt;/li&gt;&#xA;&lt;li&gt;Afterwards, I use the number of times the class won to calculate a Monte-Carlo estimate of winning. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="8" LastActivityDate="2010-09-27T10:45:17.213" CommunityOwnedDate="2010-09-27T10:45:17.213" />
  <row Id="3108" PostTypeId="2" ParentId="3104" CreationDate="2010-09-27T10:48:47.553" Score="7" ViewCount="" Body="&lt;p&gt;A good example of showing how people are non-random is to get the class to write down a number between 1 and 10. You then ask the 1's, 2's, .. to stand up.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What happens is that the majority of the class choose 7 and very few choose 1 and 10. This leads on to interesting questions, such as:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;How should you choose a random number.&lt;/li&gt;&#xA;&lt;li&gt;Designing an experiment?&lt;/li&gt;&#xA;&lt;li&gt;What do we mean by random?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="8" LastActivityDate="2010-09-27T10:48:47.553" CommunityOwnedDate="2010-09-27T10:48:47.553" CommentCount="4" />
  <row Id="3109" PostTypeId="1" AcceptedAnswerId="3326" CreationDate="2010-09-27T11:03:45.737" Score="5" ViewCount="82" Body="&lt;p&gt;For my thesis I am implementing a tracking algorithm that tracks clusters of datapoints.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I have a hard time finding research papers and/or an overview of commonly used algoritms in this subject.&lt;/p&gt;&#xA;" OwnerUserId="1411" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-09-27T11:12:02.683" LastActivityDate="2010-10-05T03:20:46.547" Title="Tracking Algorithms" Tags="&lt;machine-learning&gt;" AnswerCount="1" CommentCount="0" FavoriteCount="2" />
  <row Id="3110" PostTypeId="2" ParentId="3105" CreationDate="2010-09-27T11:48:54.790" Score="3" ViewCount="" Body="&lt;p&gt;Google's Pagerank &lt;a href=&quot;http://en.wikipedia.org/wiki/PageRank#Simplified_algorithm&quot; rel=&quot;nofollow&quot;&gt;algorithm&lt;/a&gt; is a stochastic approximation (through Markov's stationary state) of the proportion of visits received asymptotically by website $x_i$.&lt;/p&gt;&#xA;" OwnerUserId="603" LastActivityDate="2010-09-27T11:48:54.790" CommunityOwnedDate="2010-09-27T12:01:56.200" />
  <row Id="3111" PostTypeId="2" ParentId="3104" CreationDate="2010-09-27T12:00:06.260" Score="3" ViewCount="" Body="&lt;p&gt;The Drunkard's Walk by Leonard Mlodinow is full of such examples, including one on the meaning of a Positive HIV test that is 99.9% accurate. Using bayesian statistics, the actual odds of a positive test are less than 10% (a similar example is detailed in chapter two of Agresti's Introduction to Categorical Data Analysis book). Another example (i break the one example per answer but this is essentially the same problem from conditional probability)  is from the Simpson trial, where one of Simpson’s lawyers, Alan Dershowitz, noted that even though Simpson beat his wife, that hardly mattered, because in the United States, four million women are battered every year by their male partners, yet only one in 2,500 is ultimately murdered by her partner (1 in 1000), so, by the 'reasonable doubt' criterion, this is irrelevant. The jury found that argument persuasive, but it’s spurious. The relevant question was what percentage of all battered women who are murdered are killed by their abusers, which ain't 1 in 1000, but rather 9 in 10. &lt;/p&gt;&#xA;" OwnerUserId="603" LastEditorUserId="603" LastEditorDisplayName="" LastEditDate="2010-09-27T12:05:58.620" LastActivityDate="2010-09-27T12:05:58.620" CommunityOwnedDate="2010-09-27T12:00:06.260" CommentCount="2" />
  <row Id="3112" PostTypeId="1" AcceptedAnswerId="3127" CreationDate="2010-09-27T13:18:11.573" Score="3" ViewCount="466" Body="&lt;p&gt;I am using the epitools in R for calculating the confidence interval of relative risk.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://bm2.genes.nig.ac.jp/RGM2/R_current/library/epitools/man/riskratio.html&quot; rel=&quot;nofollow&quot;&gt;http://bm2.genes.nig.ac.jp/RGM2/R_current/library/epitools/man/riskratio.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are three methods inside for calculations: namely Wald, Small and Boot.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to find some article describing the three methods, but I can't find any, can anyone help? Thanks!&lt;/p&gt;&#xA;" OwnerUserId="588" LastActivityDate="2010-09-27T22:05:45.907" Title="Calculation of Relative Risk Confidence Interval" Tags="&lt;confidence-interval&gt;&lt;relative-risk&gt;" AnswerCount="2" />
  <row Id="3113" PostTypeId="1" CreationDate="2010-09-27T13:29:01.003" Score="5" ViewCount="361" Body="&lt;p&gt;I'm curious to know if anyone has a specific reference (text or journal article) to support the common practice in the medical literature of performing sample size calculation using methods that are parametric (i.e. assuming a normal distribution and a certain variance of measurements) when the analysis of the primary trial outcome will be done using non-parametric methods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An example: primary outcome is time to vomiting after giving a certain drug, which is known to have a mean value of 20 minutes (SD 6 minutes), but has a noticeably right-skewed distribution. The sample size calculation is done with the assumptions listed above, using the formula&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$n(\text{per-group})=f(\alpha,\beta) \times (2\sigma^2 /(\mu_1 - \mu_2)^2 )$,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $f(\alpha, \beta)$ changes based on the desired $\alpha$ and $\beta$ errors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, because of the skewness of the distribution, the analysis of the primary outcome will be based on ranks (non-parametric method such as the Mann Whitney U test).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this schema supportable by authors in the statistical literature, or should non-parametric sample size estimates be performed (and how would these be done)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My thoughts are that, for ease of calculation, it is acceptable to do the above practice. After all, sample size estimates are just that - estimates that make several assumptions already - all of which are likely slightly (or very!) imprecise. However, I am curious to know what others think, and specifically to know if there are any references to support this line of reasoning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks for any assistance.&lt;/p&gt;&#xA;" OwnerUserId="561" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-11-23T15:23:13.617" LastActivityDate="2010-11-23T19:45:50.180" Title="Parametric sample size calculation and non-parametric analysis" Tags="&lt;nonparametric&gt;&lt;sample-size&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="3114" PostTypeId="2" ParentId="3048" CreationDate="2010-09-27T14:10:46.990" Score="1" ViewCount="" Body="&lt;p&gt;Dimensionality reduction is basically applying clustering algorithm to the attributes (columns). Because of the fairly large dimensionality of your dataset, you might try to use SOM (self-organizing map/Kohonen net) to create a map for individuals or pages. You can then seen whether the are meaningful (interpretable) patterns. &lt;/p&gt;&#xA;" LastActivityDate="2010-09-27T14:10:46.990" />
  <row Id="3115" PostTypeId="1" CreationDate="2010-09-27T14:42:20.510" Score="5" ViewCount="419" Body="&lt;p&gt;How do you tell if the correlations at different lags obtained from the cross-correlation (ccf function) of two time series are significant. &lt;/p&gt;&#xA;" LastActivityDate="2011-05-31T20:50:26.023" Title="Cross-correlation significance in R" Tags="&lt;r&gt;&lt;statistical-significance&gt;&lt;cross-correlation&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="2" />
  <row Id="3116" PostTypeId="2" ParentId="3104" CreationDate="2010-09-27T15:13:38.850" Score="5" ViewCount="" Body="&lt;p&gt;I really like any problem that has some result that is counter-intuitive to what we'd like to think. The problems thus far are classics in the field of probability, so I'll add my favorite classic problem:&#xA;&lt;a href=&quot;http://en.wikipedia.org/wiki/Birthday_problem&quot; rel=&quot;nofollow&quot;&gt;The Birthday Problem&lt;/a&gt;. I always found it amazing that there was such a high probability of having two people with the same birthday with such a small sample. &lt;/p&gt;&#xA;" OwnerUserId="1118" LastActivityDate="2010-09-27T15:13:38.850" CommunityOwnedDate="2010-09-27T15:13:38.850" CommentCount="2" />
  <row Id="3117" PostTypeId="1" AcceptedAnswerId="3122" CreationDate="2010-09-27T16:31:58.150" Score="7" ViewCount="65" Body="&lt;p&gt;I've been studying hierachial reinforcement learning problems, and while a lot of papers propose algorithms for learning a policy, they all seem to assume they know in advance a graph structure describing the hierarchy of the actions in the domain. For example, &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.9.313&amp;amp;rep=rep1&amp;amp;type=ps&quot; rel=&quot;nofollow&quot;&gt;The MAXQ Method for Hierarchial Reinforcement Learning&lt;/a&gt; by Dietterich describes a graph of actions and sub-tasks for a simple Taxi domain, but not how this graph was discovered. How would you learn the hierarchy of this graph, and not just the policy?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words, using the paper's example, if a Taxi were driving around aimlessly, with little prior knowledge of the world, and only the primitive move-left/move-right/etc actions to take, how would it learn higher level actions like go-pick-up-passenger? If I'm understanding the paper correctly (and I may not be), it proposes how to update the policy for these high-level actions, but not how they're formed to begin with.&lt;/p&gt;&#xA;" OwnerUserId="741" LastEditorUserId="741" LastEditorDisplayName="" LastEditDate="2010-09-27T16:54:53.537" LastActivityDate="2010-09-27T18:14:16.977" Title="Learning the Structure of a Hierarchical Reinforcement Task" Tags="&lt;machine-learning&gt;" AnswerCount="1" />
  <row Id="3118" PostTypeId="2" ParentId="1557" CreationDate="2010-09-27T16:35:22.000" Score="2" ViewCount="" Body="&lt;p&gt;As mentioned by others, James-Stein is not often used directly, but is really the first paper on shrinkage, which in turn is used pretty much everywhere in single and multiple regression. The link between James-Stein and modern estimation is explained in detail &lt;a href=&quot;http://www-stat.stanford.edu/~candes/papers/NonlinearEstimation.pdf&quot; rel=&quot;nofollow&quot;&gt;in this paper&lt;/a&gt; by E.Candes. Going back to your question, I think James-Stein is an intellectual non-curiosity, in the sense that it was intellectual for sure, but had an incredibly disruptive effect on Statistics, and nobody could dismiss it as a curiosity afterwards. &lt;em&gt;Everyone&lt;/em&gt; thought that empirical means were an admissible estimator, and Stein proved them wrong with a counterexample. The rest is history.&lt;/p&gt;&#xA;" OwnerUserId="30" LastActivityDate="2010-09-27T16:35:22.000" />
  <row Id="3119" PostTypeId="1" AcceptedAnswerId="3124" CreationDate="2010-09-27T16:53:52.353" Score="3" ViewCount="55" Body="&lt;p&gt;I have 2 classifiers with different scales. How can I adjust one classifier to the scale of the other without loss of quality? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the scatter plot we have &lt;a href=&quot;http://i55.tinypic.com/2v0fm28.png&quot; rel=&quot;nofollow&quot;&gt;2 solutions plotted (x1, x2) against each other&lt;/a&gt;. I believe using linear combination as a link function is not appropriate in this particular case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Help is much appreciated.&lt;/p&gt;&#xA;" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-09-27T19:42:06.567" LastActivityDate="2010-09-27T19:46:54.843" Title="How can I adjust classifier to the scale of the other" Tags="&lt;classification&gt;&lt;data-transformation&gt;&lt;scales&gt;&lt;eda&gt;" AnswerCount="1" CommentCount="9" />
  <row Id="3120" PostTypeId="2" ParentId="1357" CreationDate="2010-09-27T17:39:46.393" Score="2" ViewCount="" Body="&lt;p&gt;More generally, mutual information is invariant under any smooth and uniquely invertible transformation of the variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See &quot;Estimating mutual information&quot; by A Kraskov, H Stögbauer, P Grassberger - Physical Review E, 2004 [&lt;a href=&quot;http://arxiv.org/pdf/cond-mat/0305641&quot; rel=&quot;nofollow&quot;&gt;http://arxiv.org/pdf/cond-mat/0305641&lt;/a&gt;]&lt;/p&gt;&#xA;" OwnerUserId="439" LastActivityDate="2010-09-27T17:39:46.393" />
  <row Id="3121" PostTypeId="1" CreationDate="2010-09-27T17:50:10.193" Score="2" ViewCount="250" Body="&lt;p&gt;I would like to know the formula or method of calculating this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, I want to know how many numbers I must pick in order to have 95% confidence that I will be within 2% of the average.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm also interested in doing a similiar analysis for the median as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Update: Numbers are not put back in the pool. I think this has to do with combinations rather than trials? For example, if I pick 1 number or 99 numbers there are 100 combinations, if I pick 100 numbers there is 1 combination etc. I think you could figure out all of the combinations when you pick 1 ball, 2 balls, 3 balls etc and see how many are within 2% of the median.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Im interested in seeing just how reliable the average price statistic is for the housing market. For example, in a single month 5% of the houses may sell, or maybe 20% will. Housing prices are all over the map but I wanted to do something simplistic like this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And now I have thought of something else...let's say I have 1-500 instead of 1-100. If I pick from 1-100 and pick 10 balls(10% of them), Im guessing that the answer would differ from taking 50 balls out of 1-500(10% of them as well)?&lt;/p&gt;&#xA;" LastEditorUserId="521" LastEditorDisplayName="" LastEditDate="2010-09-29T02:36:36.603" LastActivityDate="2010-09-29T02:36:36.603" Title="Numbers 1-100, calculating how many I must pick to get x% of reliability" Tags="&lt;sample-size&gt;&lt;experiment-design&gt;" AnswerCount="4" CommentCount="7" FavoriteCount="1" />
  <row Id="3122" PostTypeId="2" ParentId="3117" CreationDate="2010-09-27T18:05:25.237" Score="6" ViewCount="" Body="&lt;p&gt;According to this &lt;a href=&quot;http://rlai.cs.ualberta.ca/papers/barto03recent.pdf&quot; rel=&quot;nofollow&quot;&gt;paper&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In the current state-of-the-art, the&#xA;  designer of an RL system typically&#xA;  uses prior knowledge about the task to&#xA;  add a specific set of options to the&#xA;  set of primitive actions available to&#xA;  the agent.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Also see section 6.2 Learning Task Hierarchies in the same paper. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first idea that comes to my mind is that if you don't know task hierarchies, you should start with non-hierachial reinforcement learning and trying to discover the structure afterwards or while learning, i.e. you are trying to generalize your model. To me this task looks similar to Bayesian model merging technique for HMM (for example see this &lt;a href=&quot;http://www.icsi.berkeley.edu/NTL/papers/stolcke-thesis.pdf&quot; rel=&quot;nofollow&quot;&gt;thesis&lt;/a&gt;)&lt;/p&gt;&#xA;" OwnerUserId="1075" LastEditorUserId="1075" LastEditorDisplayName="" LastEditDate="2010-09-27T18:14:16.977" LastActivityDate="2010-09-27T18:14:16.977" />
  <row Id="3123" PostTypeId="2" ParentId="3121" CreationDate="2010-09-27T18:34:36.773" Score="2" ViewCount="" Body="&lt;p&gt;I guess that you are looking for &lt;a href=&quot;http://books.google.gr/books?id=WK7l-csM0-gC&amp;amp;lpg=PA176&amp;amp;ots=8wTHuo4PrW&amp;amp;dq=%22experimental%20design%20for%20estimating%20the%20proportion%22&amp;amp;pg=PA175#v=onepage&amp;amp;q&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;experimental design for estimating the mean&lt;/a&gt;. I am not aware of something similar for the median though.&lt;/p&gt;&#xA;" OwnerUserId="339" LastActivityDate="2010-09-27T18:34:36.773" CommentCount="1" />
  <row Id="3124" PostTypeId="2" ParentId="3119" CreationDate="2010-09-27T19:41:30.373" Score="3" ViewCount="" Body="&lt;p&gt;John Tukey described a simple method that works well in practice.  Pick three representative points within the scatterplot: one near each end and one in the middle.  Experiment with Box-Cox (power + log) transforms of either or both variables, &lt;em&gt;applying them only to the coordinates of the three representatives.&lt;/em&gt;  (Tukey did this by hand; a spreadsheet speeds up the calculation.)  You want to make the slope between the left point and the middle point closely equal the slope between the middle and the right point.  Applying the resulting transformation(s) will linearize the plot.  (It basically has to: if you can get these points to line up, everything will line up.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can hope for homoscedasticity, too.  Your plot seems to exhibit a little more scatter on the right hand side.  This suggests choosing Box-Cox parameters (powers) less than 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, I can eyeball three points on your plot at (0.2, 0.05), (0.5, 0.2), and (0.7, 0.4).  Using just powers (which are Box-Cox transformations up to an affine change) I get slopes of 0.70 and 0.71 when leaving the x-coordinate unchanged but taking the 0.3 power of the y-coordinate.  The 0.3 power isn't much fun--one would prefer the 1/2 or maybe the 1/3 power--so I also tried taking the square root (1/2 power) of x and the logarithm (0 &quot;power&quot;) of y.  The slopes are now 5.33 and 5.35: exactly equal within the imprecision of my coordinate estimates.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This suggests you should consider using $\left(\sqrt{x_1}, \log{x_2} \right)$ for your variables.  If this makes the scatter around the resulting linear scatterplot a little uneven (heteroscedastic), go back and try less severe powers, such as 1 and 1/3.  When you're done you can apply affine transformations (rescaling and recentering) to get the re-expressed variables in the range $[0, 1/2]$ if you really need that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Reference:&lt;/strong&gt;&#xA;Tukey, J, &lt;em&gt;EDA&lt;/em&gt;.  Addison-Wesley (1977).&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-09-27T19:46:54.843" LastActivityDate="2010-09-27T19:46:54.843" />
  <row Id="3125" PostTypeId="2" ParentId="3121" CreationDate="2010-09-27T20:03:37.737" Score="3" ViewCount="" Body="&lt;p&gt;I did a little simulation in R, which resulted in 98. If you go to 5% of the mean then you're down to 93 and if you go to 10% then it's 76.&#xA;Similar results are obtained for the median.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Please, read whuber's comments to my answer&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The code I used:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;numbers &amp;lt;- 1:100&#xA;numTrials &amp;lt;- 1000&#xA;&#xA;means &amp;lt;- matrix(nrow = numTrials, ncol = length(numbers))&#xA;&#xA;for (n in 1:length(numbers))&#xA;    {&#xA;    for (t in 1:numTrials)&#xA;        {&#xA;        vals &amp;lt;- sample(numbers, n)&#xA;        means[t, n] &amp;lt;- mean(vals) # or median if you want the median &#xA;        }&#xA;    }&#xA;&#xA;realAvg &amp;lt;- mean(numbers)&#xA;uplim &amp;lt;- realAvg + 0.05*realAvg&#xA;downlim &amp;lt;- realAvg - 0.05*realAvg&#xA;&#xA;res &amp;lt;- unlist(apply(means, 2, function(x){&#xA;       length(which(x&amp;gt;downlim &amp;amp; x&amp;lt;uplim))/length(x)}))&#xA;plot(1:length(numbers), res*100, t=&quot;l&quot;, xlab=&quot;Numbers picked&quot;, &#xA;       ylab=&quot;% trials with mean within limits&quot;)&#xA;abline(h=95, col=&quot;red&quot;)&#xA;&#xA;print(which(res&amp;gt;=0.95))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/f6WSF.jpg&quot; alt=&quot;Numbers picked vs % good trials&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: anyone would like to explain the &quot;sawtooth&quot; look of the graph?&lt;/p&gt;&#xA;" OwnerUserId="582" LastEditorUserId="582" LastEditorDisplayName="" LastEditDate="2010-09-28T21:11:22.577" LastActivityDate="2010-09-28T21:11:22.577" CommentCount="10" />
  <row Id="3126" PostTypeId="2" ParentId="3112" CreationDate="2010-09-27T20:12:24.250" Score="2" ViewCount="" Body="&lt;p&gt;I bookmarked this thread from r-help a while back:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.mail-archive.com/r-help@stat.math.ethz.ch/msg75740.html&quot; rel=&quot;nofollow&quot;&gt;Summary, was Re: Confidence interval for relative risk&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;and you might find the referenced PDF by Michael Dewey helpful:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.zen103156.zen.co.uk/rr.pdf&quot; rel=&quot;nofollow&quot;&gt;confidence intervals for risk ratios&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If you can though, get a copy of the following book.  I know it covers the unconditional likelihood and bootstrap methods for sure, and I suspect the small sample adjustment too (don't have a copy handy to check for the last):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://books.google.com/books?id=dm5FAAAAYAAJ&quot; rel=&quot;nofollow&quot;&gt;Biostatistical methods: the assessment of relative risks&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-09-27T20:12:24.250" CommentCount="1" />
  <row Id="3127" PostTypeId="2" ParentId="3112" CreationDate="2010-09-27T20:41:45.253" Score="5" ViewCount="" Body="&lt;p&gt;The three options that are proposed in &lt;code&gt;riskratio()&lt;/code&gt; refer to an asymptotic or large sample approach, an approximation for small sample, a resampling approach (asymptotic bootstrap, i.e. not based on percentile or bias-corrected). The former is described in Rothman's book (as referenced in the online help), chap. 14, pp. 241-244. The latter is relatively trivial so I will skip it. The small sample approach is just an adjustment on the calculation of the estimated relative risk.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we consider the following table of counts for subjects cross-classififed according to their exposure and disease status,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;          Exposed  Non-exposed  Total&#xA;Cases          a1           a0     m1&#xA;Non-case       b1           b0     m0&#xA;Total          n1           n0      N&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;the MLE of the risk ratio (RR), $\text{RR}=R_1/R_0$, is $\text{RR}=\frac{a_1/n_1}{a_0/n_0}$.&#xA;In the &lt;em&gt;large sample approach&lt;/em&gt;, a score statistic (for testing $R_1=R_0$, or equivalently, $\text{RR}=1$) is used, $\chi_S=\frac{a_1-\tilde a_1}{V^{1/2}}$, where the numerator reflects the difference between the oberved and expected counts for exposed cases and $V=(m_1n_1m_0n_0)/(n^2(n-1))$ is the variance of $a_1$. Now, that's all for computing the $p$-value because we know that $\chi_S$ follow a chi-square distribution. In fact, the three $p$-values (mid-$p$, Fisher exact test, and $\chi^2$-test) that are returned by &lt;code&gt;riskratio()&lt;/code&gt; are computed in the &lt;code&gt;tab2by2.test()&lt;/code&gt; function. For more information on mid-$p$, you can refer to&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Berry and Armitage (1995). &lt;a href=&quot;http://members.optusnet.com.au/gberrycons/GB%20Web_files/Pub%20124.pdf&quot; rel=&quot;nofollow&quot;&gt;Mid-P&#xA;  confidence intervals: a brief&#xA;  review&lt;/a&gt;. &lt;em&gt;The Statistician&lt;/em&gt;, 44(4),&#xA;  417-423.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Now, for computing the $100(1-\alpha)$ CIs, this asymptotic approach yields an approximate SD estimate for $\ln(\text{RR})$ of $(\frac{1}{a_1}-\frac{1}{n_1}+\frac{1}{a_0}-\frac{1}{n_0})^{1/2}$, and the Wald limits are found to be $\exp(\ln(\text{RR}))\pm Z_c \text{SD}(\ln(\text{RR}))$, where $Z_c$ is the corresponding quantile for the standard normal distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;em&gt;small sample approach&lt;/em&gt; makes use of an adjusted RR estimator: we just replace the denominator $a_0/n_0$ by $(a_0+1)/(n_0+1)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As to how to decide whether we should rely on the large or small sample approach, it is mainly by checking expected cell frequencies; for the $\chi_S$ to be valid, $\tilde a_1$, $m_1-\tilde a_1$, $n_1-\tilde a_1$ and $m_0-n_1+\tilde a_1$ should be $&gt; 5$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Working through the example of Rothman (p. 243), &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sel &amp;lt;- matrix(c(2,9,12,7), 2, 2)&#xA;riskratio(sel, rev=&quot;row&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which yields&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;$data&#xA;          Outcome&#xA;Predictor  Disease1 Disease2 Total&#xA;  Exposed2        9        7    16&#xA;  Exposed1        2       12    14&#xA;  Total          11       19    30&#xA;&#xA;$measure&#xA;          risk ratio with 95% C.I.&#xA;Predictor  estimate    lower    upper&#xA;  Exposed2 1.000000       NA       NA&#xA;  Exposed1 1.959184 1.080254 3.553240&#xA;&#xA;$p.value&#xA;          two-sided&#xA;Predictor  midp.exact fisher.exact chi.square&#xA;  Exposed2         NA           NA         NA&#xA;  Exposed1 0.02332167   0.02588706 0.01733469&#xA;&#xA;$correction&#xA;[1] FALSE&#xA;&#xA;attr(,&quot;method&quot;)&#xA;[1] &quot;Unconditional MLE &amp;amp; normal approximation (Wald) CI&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;By hand, we would get&#xA;$\text{RR} = (12/14)/(7/16)=1.96$, $\tilde a_1 = 19\times 14 / 30= 8.87$, $V = (8.87\times 11\times 16)/ \big(30\times (30-1)\big)= 1.79$, $\chi_S = (12-8.87)/\sqrt{1.79}= 2.34$, $\text{SD}(\ln(\text{RR})) = \left( 1/12-1/14+1/7-1/16 \right)^{1/2}=0.304$, $95\% \text{CIs} = \exp\big(\ln(1.96)\pm 1.645\times0.304\big)=[1.2;3.2]\quad \text{(rounded)}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The following papers also addresses the construction of the test statistic for the RR or the OR:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Miettinen and Nurminen (1985). &lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pubmed/4023479&quot; rel=&quot;nofollow&quot;&gt;Comparative analysis of two rates&lt;/a&gt;. *Statistics in Medicine, 4: 213-226.&lt;/li&gt;&#xA;&lt;li&gt;Becker (1989). &lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pubmed/2799127&quot; rel=&quot;nofollow&quot;&gt;A comparison of maximum likelihood and Jewell's estimators of the odds ratio and relative risk in single 2 × 2 tables&lt;/a&gt;. &lt;em&gt;Statistics in Medicine&lt;/em&gt;, 8(8): 987-996.&lt;/li&gt;&#xA;&lt;li&gt;Tian, Tang, Ng, and Chan (2008). &lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pubmed/18069723&quot; rel=&quot;nofollow&quot;&gt;Confidence intervals for the risk ratio under inverse sampling&lt;/a&gt;. &lt;em&gt;Statistics in Medicine&lt;/em&gt;, 27(17), 3301-3324.&lt;/li&gt;&#xA;&lt;li&gt;Walter and Cook (1991). &lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pubmed/1742440&quot; rel=&quot;nofollow&quot;&gt;A comparison of several point estimators of the odds ratio in a single 2 x 2 contingency table&lt;/a&gt;. &lt;em&gt;Biometrics&lt;/em&gt;, 47(3): 795-811.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Notes&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;As far as I know, there's no reference to relative risk in Selvin's book (also referenced in the online help).&lt;/li&gt;&#xA;&lt;li&gt;Alan Agresti has also some code for &lt;a href=&quot;http://www.stat.ufl.edu/~aa/cda/R/two-sample/R2/&quot; rel=&quot;nofollow&quot;&gt;relative risk&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="930" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-09-27T22:05:45.907" LastActivityDate="2010-09-27T22:05:45.907" />
  <row Id="3128" PostTypeId="2" ParentId="3115" CreationDate="2010-09-27T22:23:31.053" Score="6" ViewCount="" Body="&lt;p&gt;The variance of the cross-correlation coefficient under the null hypothesis of zero correlation is approx $1/n$ where $n$ is the length of the series. The coefficients are also asymptotically normal. So approximate critical values (at the 5% level) are $\pm 2/\sqrt{n}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These critical values are plotted automatically in R using &lt;code&gt;ccf(x,y)&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="159" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-09-27T23:35:18.967" LastActivityDate="2010-09-27T23:35:18.967" />
  <row Id="3129" PostTypeId="2" ParentId="3113" CreationDate="2010-09-27T23:47:03.250" Score="2" ViewCount="" Body="&lt;p&gt;It sounds dodgy to me. Nonparametric methods almost always involve more degrees of freedom than parametric methods and so need more data. In your particular example, the Mann-Whitney test has lower power than the t-test and so more data are required for the same specified power and size.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A simple way to do sample size calculation for any method (non-parametric or otherwise) is to use a bootstrap approach.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-09-27T23:47:03.250" CommentCount="2" />
  <row Id="3130" PostTypeId="1" AcceptedAnswerId="3131" CreationDate="2010-09-28T01:03:48.430" Score="8" ViewCount="80" Body="&lt;p&gt;Suppose we have a set A and a subset B.  If we know |A|, then we can calculate |B| by finding the probability p that an element chosen uniformly at random from A belongs to B.  Specifically |A|p=|B|.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose we generate n elements of A uniformly at random and use this data to estimate p (number of elements in B divided by n) and hence estimate |B|.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;How reliable is this estimate?  I.e. how can we compute the error?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;As a side question, is there a name for this technique?  (it seems to be a mathematical version of the &lt;a href=&quot;http://en.wikipedia.org/wiki/Mark_and_recapture&quot; rel=&quot;nofollow&quot;&gt;mark-and-recapture&lt;/a&gt; technique)&lt;/p&gt;&#xA;" OwnerUserId="386" LastActivityDate="2010-09-28T01:39:19.667" Title="Error in estimating the size of a set?" Tags="&lt;estimation&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="3131" PostTypeId="2" ParentId="3130" CreationDate="2010-09-28T01:21:27.053" Score="8" ViewCount="" Body="&lt;p&gt;You are estimating proportions. For concreteness, imagine that A is the population of voters and B is the set of voters who vote for a particular candidate. Thus, p would be the percentage of voters who would vote for that candidate. Let:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\pi$ be the true percentage of people who would vote for the candidate&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\pi = \frac{|B|}{|A|}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then each one of your samples is a bernoulli trial with probability $\pi$ or equivalently you can imagine that each one of your samples is a poll of potential voters asking them whether they would vote for the candidate. Thus, the MLE of $\pi$ is given by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$p = \frac{n_B}{n}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$n_B$ is the number of people who said they would vote for candidate or the number of elements which belong to the set B in your sample of size $n$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The standard error for your estimate is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\sqrt{\frac{\pi (1-\pi)}{n}}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The above can be approximated by using the MLE for $\pi$ i.e., by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\sqrt{\frac{p (1-p)}{n}}$&lt;/p&gt;&#xA;" LastActivityDate="2010-09-28T01:39:19.667" />
  <row Id="3132" PostTypeId="2" ParentId="3105" CreationDate="2010-09-28T01:25:32.580" Score="1" ViewCount="" Body="&lt;p&gt;As you have alluded to, communications engineering (and signal processing in general) is filled with stochastic processes. See, for example:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://books.google.com.au/books?id=wVNTPgAACAAJ&amp;amp;dq=communication+systems+carlson+5th&amp;amp;hl=en&amp;amp;ei=R0GhTIcakJi8A8udpdkD&amp;amp;sa=X&amp;amp;oi=book_result&amp;amp;ct=result&amp;amp;resnum=8&amp;amp;ved=0CFUQ6AEwBzgK&quot; rel=&quot;nofollow&quot;&gt;Chapter 9 and later of Communications Systems by Carlson and Crilly&lt;/a&gt;  &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0471149616&quot; rel=&quot;nofollow&quot;&gt;Chapter 13 of A Course in Digtial Signal Processing by Porat&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;This is only two textbook examples. You will find many (many) more. It's fair to say that your mobile phone, the internet and etcetera all rely extremely heavily on our understanding of stochastic processes.&lt;/p&gt;&#xA;" OwnerUserId="352" LastActivityDate="2010-09-28T01:25:32.580" CommunityOwnedDate="2010-09-28T01:25:32.580" />
  <row Id="3133" PostTypeId="2" ParentId="2513" CreationDate="2010-09-28T02:27:31.433" Score="0" ViewCount="" Body="&lt;p&gt;the short answer to your question is that with a sample size of n = 1280, a formal statistical test for equal variances is not really necessary. with that sample size, the variance 'estimates' you get are certainty-equivalents [almost certainly very close to the true population variances]. so you can just look at the two values and see if they look reasonably close or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;a comment about your linear model: i assume there is a random error term in it as well. i think there should also be an intercept term, unless you are sure the regression line goes thru the origin. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;also, do you treat the participant effect as a random effect? it seems you should, if each contributes more than one pair of ratings to the data. but that might cause problems with the usual methods for testing for effects other than a regression effect [for which the usual models do not have such terms]. this is because the usual models assume the different before-after [ratings] pairs are independent. but the random effect creates correlations between the ratings pairs for the same individual. such a model would have to be studied further to see if the hypothesis of no differential A'-rating effect still reduces to a test of equal variances.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;here is a bit more about the pitman test [altho what follows is probably unnecessary for you, in view of the 'short answer' above]:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;like tests for equality of variances for two [or more] independent samples, the pitman test is apparently sensitive to the non-normality of the responses. you don't mention the scale on which the variable &lt;em&gt;rating&lt;/em&gt; is measured, but especially if it is a likert-type scale, the assumption of normality for the responses may be questionable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://jeb.sagepub.com/content/15/3/237.abstract&quot; rel=&quot;nofollow&quot;&gt;wilcox&lt;/a&gt; did a study of the pitman and other tests of variance equality for paired data and wrote:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;But, it is concluded that, in terms of controlling both Type I and Type II errors, a satisfactory solution does not yet exist.&lt;/em&gt; [this was in 1990.]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;this doesn't necessarily mean that the pitman test would be invalid for your data, but it does suggest that some caution in using it might be appropriate. [if i remember correctly, the difficulty with homogeneity-of-variance tests arises when the data distribution [for &lt;em&gt;rating&lt;/em&gt; in your case] has a heavy tail [or tails]. then the actual level of the test can exceed the nominal level [of .05, say], resulting in too often falsely rejecting the hypothesis of equal variances when, in fact, they &lt;em&gt;are&lt;/em&gt; equal. [better power, but also worse false rejection rate.]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;it seems unlikely that responses on a likert scale would have heavy tails [or too big a 4$^{th}$ moment - as compared with its variance], but it might not hurt to see if the usual kertosis measure [4$^{th}$ central moment divided by variance$^2$ exceeds 3 [which is then an indication of heavy tails].&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://biomet.oxfordjournals.org/content/81/2/359.abstract&quot; rel=&quot;nofollow&quot;&gt;grambsch&lt;/a&gt; in 1994 compared the pitman test to alternative procedures - some of which are more complicated to carry out.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;some numerical results in her paper suggest that if the distributions of &lt;em&gt;rating&lt;/em&gt; are uniform or have the shape of a half-parabola over the range of responses [&lt;em&gt;i.e.&lt;/em&gt; have light tails], the pitman test is conservative: the actual type 1 error for a  nominal value of $\alpha = .05$ can be anywhere from .005 to about .05 - depending on the actual shape of the distributions and the sample size. [her results deal with samples sizes of 50 or less. i suppose with your sample size of 1280, her conclusions would have been somewhat different.] &lt;/p&gt;&#xA;&#xA;&lt;p&gt;if you are interested, she presents a modification of the pitman test that keeps the actual significance level close to the nominal value of .05, for both light- and heavy-tailed distributions.&lt;/p&gt;&#xA;" OwnerUserId="1112" LastActivityDate="2010-09-28T02:27:31.433" />
  <row Id="3134" PostTypeId="2" ParentId="3121" CreationDate="2010-09-28T07:19:03.973" Score="0" ViewCount="" Body="&lt;p&gt;A couple of points first:&lt;br&gt;&#xA;1) The answer will be very large, so we are safe to assume the &lt;a href=&quot;http://en.wikipedia.org/wiki/Central_limit_theorem&quot; rel=&quot;nofollow&quot;&gt;Central Limit Theorem&lt;/a&gt; is in operation, even if the distribution is all 1s and 100s (i.e. Bernoulli).&lt;br&gt;&#xA;2) We can use normal deviates (z-scores) rather than t-tests because the required N will be very large - this is a simplification.&lt;br&gt;&#xA;3. If we are sampling without replacement the problem statement seems to have only one interpretation - the numbers 1,...,100 - a uniform discrete distribution. whuber and nico have already provided answers, so I will deal with sampling with replacement and an undefined probability mass function (PMF).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Approach 1:&lt;br&gt;&#xA;Assume a &lt;a href=&quot;http://en.wikipedia.org/wiki/Uniform_distribution_%28discrete%29&quot; rel=&quot;nofollow&quot;&gt;discrete uniform distribution&lt;/a&gt; and sampling with replacement. The mean is 50.5, the SD is sqrt((n^2 - 1)/12) ~ 29&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We want a maximum error of 2% so delta = abs(est_mean - true_mean)/true_mean = 1.01&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assume a normal distribution, the standard error of the mean is:&#xA;\begin{align}&#xA;SEM &amp;amp;= SD/sqrt(n)\\&#xA;sqrt(n) &amp;amp;= SD/SEM\\&#xA;n &amp;amp;= (SD/SEM)^2\\&#xA;n &amp;amp;= (SD/(delta/1.96))^2\\&#xA;n &amp;amp;= (29/(1.01/1.96))^2\\&#xA;n &amp;amp;\approx 3167\\&#xA;\end{align}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Approach 2:&lt;br&gt;&#xA;Assume the worst case.&#xA;I'm pretty sure this is a 50:50 split in the &lt;a href=&quot;http://en.wikipedia.org/wiki/Probability_mass_function&quot; rel=&quot;nofollow&quot;&gt;pmf &lt;/a&gt; between 1 and 100. If so our mean is 50.5 and our &lt;a href=&quot;http://en.wikipedia.org/wiki/Standard_deviation#Discrete_random_variable&quot; rel=&quot;nofollow&quot;&gt;SD&lt;/a&gt; is sqrt(49.5^2)=49.5.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align}&#xA;SEM &amp;amp;= SD/sqrt(n)\\&#xA;sqrt(n) &amp;amp;= SD/SEM\\&#xA;n &amp;amp;= (SD/SEM)^2\\&#xA;n &amp;amp;= (SD/(delta/1.96))^2\\&#xA;n &amp;amp;= (49.5/(1.01/1.96))^2\\&#xA;n &amp;amp;\approx 9227\\&#xA;\end{align}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'll let someone else do the sample sizes for medians.&lt;/p&gt;&#xA;" OwnerUserId="521" LastEditorUserId="521" LastEditorDisplayName="" LastEditDate="2010-09-29T02:35:18.350" LastActivityDate="2010-09-29T02:35:18.350" CommentCount="3" />
  <row Id="3135" PostTypeId="2" ParentId="3086" CreationDate="2010-09-28T07:40:33.087" Score="4" ViewCount="" Body="&lt;p&gt;My best bet is that you are facing large imbalance between your categories responses, for some of your items.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you assume that your binary responses reflect individual locations on an underlying latent (i.e. continuous) trait, then correlating the two variables is ok, provided the cutoff is close to the mean of the bivariate density, as shown below (here the cutoff are set symmetrically at $(.5,.5)$, for a correlation of 0.5):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/ceUmY.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this case, the Person correlation will underestimate the true linear relationships between the two latent traits, especially in the mid-range of the correlation metric. On the other hand, when the cutoffs are clearly assymmetrical on both continuous variables, the tetrachoric correlation will generally overestimate the true relationship. The following picture illustrates the ideal case.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(polycor)&#xA;set.seed(101)&#xA;n &amp;lt;- 500&#xA;rho &amp;lt;- seq(0,1,length=500)&#xA;pc1 &amp;lt;- pc2 &amp;lt;- tc &amp;lt;- numeric(500)&#xA;&#xA;for (i in 1:500) {&#xA;  data &amp;lt;- rmvnorm(n, c(0, 0), matrix(c(1, rho[i], rho[i], 1), 2, 2))&#xA;  x &amp;lt;- data[,1]; y &amp;lt;- data[,2]&#xA;  xb &amp;lt;- ifelse(x&amp;gt;=mean(x), 1, 0); yb &amp;lt;- ifelse(y&amp;gt;=mean(y), 1, 0)&#xA;  pc1[i] &amp;lt;- cor(x, y)&#xA;  pc2[i] &amp;lt;- cor(xb, yb)&#xA;  tc[i] &amp;lt;- polychor(xb, yb)&#xA;}&#xA;&#xA;plot(pc1, pc2, cex=.6, col=&quot;red&quot;, xlab=&quot;True linear relationship&quot;,&#xA;     ylab=&quot;Observed correlation&quot;)&#xA;lines(lowess(pc1, pc2), col=&quot;red&quot;, lwd=2)&#xA;abline(0, 1, col=&quot;lightgray&quot;)&#xA;points(pc1, tc, cex=.6, col=&quot;blue&quot;)&#xA;lines(lowess(pc1, tc), col=&quot;blue&quot;, lwd=2)&#xA;legend(&quot;topleft&quot;, c(&quot;Pearson (0/1)&quot;,&quot;Tetrachoric&quot;), col=c(2,4), lty=1, bty=&quot;n&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/FZ14p.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, you can play with the choice of the cutoff, $\tau$, and see what happens when it is asymmetric and largely departs from the mean of the joint density of $x$ and $y$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To complement @shabbychef's response, the phi coefficient is generally used with &quot;truly&quot; categorical variables (no hypothesis about a continuous generating process are made) and reduces to Pearson correlation in this case ($\sqrt{\chi^2}/n$). The problem is then to factor out a correlation matrix constructed in such a way because communalities become meaningless.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To avoid this problem, we may rely on parametric item response modeling, e.g. mixed-effects logistic model (in this case, no need to worry about the cutoff, since it is estimated), or non-parametric model, like Mokken scaling; in this latest case, we only assume monotonicity on the latent trait, but no functional form relating one's location on the latent trait and the outcome (i.e. the probability of endorsing the item). However, in your case, it would be a pain and would not allow you to identify a structure in your correlation matrix. But it may be used afterwards.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, John Uebersax provides an in-depth discussion of tetrachoric correlation in relation of latent trait modeling, see &lt;a href=&quot;http://www.john-uebersax.com/stat/tetra.htm&quot; rel=&quot;nofollow&quot;&gt;Introduction to the Tetrachoric and Polychoric Correlation Coefficients&lt;/a&gt;. Also, Nunnally discussed a long ago of the advantages/disadvantages of relying on Pearson vs. Tetrachoric correlation coefficients in Factor Analysis, see e.g. pp. 570-573 (3rd ed.).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;References&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;O'Connor, B. &lt;a href=&quot;https://people.ok.ubc.ca/brioconn/nfactors/itemanalysis.html&quot; rel=&quot;nofollow&quot;&gt;Cautions Regarding Item-Level Factor Analyses&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;li&gt;Bernstein, I.H., Teng, G. (1989). Factoring items and factoring scales are different: Spurious evidence for multidimensionality due to item categorization. &lt;em&gt;Psychological Bulletin&lt;/em&gt;, &lt;em&gt;105&lt;/em&gt;, 467-477.&lt;/li&gt;&#xA;&lt;li&gt;Edwards, J.H. and Edwards, A.W.F. (1984). Approximating the tetrachoric correlation coefficient. &lt;em&gt;Biometrics&lt;/em&gt;, &lt;em&gt;40&lt;/em&gt;, 563.&lt;/li&gt;&#xA;&lt;li&gt;Castellan, N.J. (1966). On the estimation of the tetrachoric correlation coefficient. &lt;em&gt;Psychometrika&lt;/em&gt;, &lt;em&gt;31(1)&lt;/em&gt;, 67-73. &lt;/li&gt;&#xA;&lt;li&gt;Fitzgerald, P., Knuiman, M.W., Divitini, M.L., and Bartholomew, H.C. (1999). Effect of dichotomising a continuous variable on the assessment of familial aggregation: an empirical study using body mass index data from the Busselton Health Study. &lt;em&gt;J. Epidemiol. Biostat.&lt;/em&gt;, &lt;em&gt;4(4)&lt;/em&gt;, 321-327.&lt;/li&gt;&#xA;&lt;li&gt;Nunnally, J.C. and Bernstein, I.H. (1994). &lt;em&gt;Psychometric Theory&lt;/em&gt; (Third ed.). McGraw-Hill.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-28T08:43:41.500" LastActivityDate="2010-09-28T08:43:41.500" />
  <row Id="3136" PostTypeId="1" AcceptedAnswerId="3138" CreationDate="2010-09-28T08:29:32.933" Score="3" ViewCount="950" Body="&lt;p&gt;I have a data set with following structure:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;a word | number of occurrence of a word in a document | a document id &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can I perform a test for normal distribution in R? Probably it is an easy question but I am a R newbie.&lt;/p&gt;&#xA;" OwnerUserId="1389" LastActivityDate="2010-09-28T22:12:45.667" Title="How to perform a test using R to see if data follows normal distribution" Tags="&lt;r&gt;&lt;distributions&gt;&lt;normality&gt;" AnswerCount="4" CommentCount="5" FavoriteCount="1" />
  <row Id="3137" PostTypeId="2" ParentId="3136" CreationDate="2010-09-28T09:10:33.713" Score="5" ViewCount="" Body="&lt;p&gt;Assuming your dataset is called &lt;code&gt;words&lt;/code&gt; and has a &lt;code&gt;counts&lt;/code&gt; column, you can plot the histogram to have a visualization of the distribution:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;hist(words$counts, 100, col=&quot;black&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where 100 is the number of bins&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also do a normal Q-Q plot using&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;qqnorm(words$counts)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Finally, you can also use the Shapiro-Wilk test for normality&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;shapiro.test(word$counts)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Although, look at this discussion: &lt;a href=&quot;http://stats.stackexchange.com/questions/2492/normality-testing-essentially-useless&quot;&gt;Normality Testing: 'Essentially Useless?'&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="582" LastActivityDate="2010-09-28T09:10:33.713" />
  <row Id="3138" PostTypeId="2" ParentId="3136" CreationDate="2010-09-28T09:13:07.200" Score="5" ViewCount="" Body="&lt;p&gt;If I understand your question correctly, then to test if word occurrences in a set of documents follows a Normal distribution you can just use a shapiro-Wilk test and some qqplots. For example,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;## Generate two data sets&#xA;## First Normal, second from a t-distribution&#xA;words1 = rnorm(100); words2 = rt(100, df=3)&#xA;&#xA;## Have a look at the densities&#xA;plot(density(words1));plot(density(words2))&#xA;&#xA;## Perform the test&#xA;shapiro.test(words1); shapiro.test(words2)&#xA;&#xA;## Plot using a qqplot&#xA;qqnorm(words1);qqline(words1, col = 2)&#xA;qqnorm(words2);qqline(words2, col = 2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The qqplot commands give:&#xA;&lt;img src=&quot;http://i.imgur.com/hdgwQ.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can see that the second data set is clearly not Normal.&lt;/p&gt;&#xA;" OwnerUserId="8" LastActivityDate="2010-09-28T09:13:07.200" />
  <row Id="3140" PostTypeId="1" CreationDate="2010-09-28T13:28:00.863" Score="1" ViewCount="363" Body="&lt;p&gt;Hi all&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am currently working on matlab and I am new to it. I would like to know how do i average the results from the folds (or otherwise combined) to produce a single estimation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My dataset has 2 class labels +1 and -1. and k=5. I use the knnclassify() method in matlab in order to perform cross validation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Kindly help! &lt;/p&gt;&#xA;" LastEditorUserId="5" LastEditorDisplayName="" LastEditDate="2010-09-28T13:30:42.950" LastActivityDate="2011-01-04T19:11:00.380" Title="Knnclassify() matlab" Tags="&lt;cross-validation&gt;&lt;matlab&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="3141" PostTypeId="2" ParentId="3136" CreationDate="2010-09-28T14:30:29.593" Score="2" ViewCount="" Body="&lt;p&gt;In addition to the Shapiro-Wilk test of the stats package, the nortest package (available on CRAN) provides other normality tests.&lt;/p&gt;&#xA;" OwnerUserId="1355" LastActivityDate="2010-09-28T14:30:29.593" />
  <row Id="3142" PostTypeId="2" ParentId="3140" CreationDate="2010-09-28T15:19:52.463" Score="4" ViewCount="" Body="&lt;p&gt;The knnclassify() function uses the k-nearest neighbours classification algorithm to perform classification. The crossvalind() function is a separate one to divide a data set in to folds to perform cross-validation. Both are part of the Bioinformatics toolbox. I suggest that you read the documentation associated with them, so that you can be sure you are using them correctly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, however, with a single n-fold cross-validation (CV) run, you don't really need to average results from folds. You just collect the results from the individual folds.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, with 5-fold CV, your dataset is divided into 5 separate folds, used as follows:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Train with Folds 1..4, Test on Fold 5&lt;/li&gt;&#xA;&lt;li&gt;Train with Folds 1..3+5, Test on Fold 4&lt;/li&gt;&#xA;&lt;li&gt;Train with Folds 1+2+4+5 Test on Fold 3&#xA;and so on.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;So all of the data is used for testing exactly once. You can just combine the test results in a single list, and count how many it got right and wrong to compute the overall accuracy.&lt;/p&gt;&#xA;" OwnerUserId="1436" LastActivityDate="2010-09-28T15:19:52.463" />
  <row Id="3143" PostTypeId="1" AcceptedAnswerId="3144" CreationDate="2010-09-28T17:03:17.213" Score="3" ViewCount="260" Body="&lt;p&gt;Hi all,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm quite new to R and I have a following problem:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a simple 2-factor linear model:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# factor1 has 8 categorical values, factor2 has 6 categories&#xA;Rate ~ factor1 + factor2 &#xA;model1 &amp;lt;- lm(Rate~factor1+factor2, data=myData)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And want to put constraints SUM of factor1 coefficients = 0, the same for factor2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;None of the manuals gives any clue how to do this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found a link to similar problem here but it is different and I couldnt figure out how to modify it...&lt;/p&gt;&#xA;" OwnerUserId="1439" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-28T17:54:24.707" LastActivityDate="2010-09-28T17:54:24.707" Title="Linear Model with Constraints" Tags="&lt;r&gt;&lt;regression&gt;&lt;contrasts&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="3" />
  <row Id="3144" PostTypeId="2" ParentId="3143" CreationDate="2010-09-28T17:47:40.680" Score="4" ViewCount="" Body="&lt;p&gt;You can do this using contrasts:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;options(contrasts=c('contr.sum', 'contr.sum'))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;See &lt;code&gt;?contr.sum&lt;/code&gt; for more information.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;UPDATE: A little googling reveals a page which might be a little clearer:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://faculty.nps.edu/sebuttre/home/S/contrasts.html&quot; rel=&quot;nofollow&quot;&gt;Setting and Keeping Contrasts&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-09-28T17:47:40.680" CommentCount="3" />
  <row Id="3145" PostTypeId="2" ParentId="3143" CreationDate="2010-09-28T17:48:38.293" Score="5" ViewCount="" Body="&lt;p&gt;There is a trick to be had here. For simplicity, suppose you are trying to build a model of the form $$y = b_1 x_1 + b_2 x_2 + b_3 x_3,$$ subject to $b_1 + b_2 + b_3 = 0$. Simply re-express $b_3$ as $b_3 = - b_1 - b_2$, which is to say you are trying to build a model of the form&#xA;$$y = b_1 x_1 + b_2 x_2 - (b_1 + b_2) x_3 = b_1 (x_1 - x_3) + b_2 (x_2 - x_3).$$&#xA;So create new variables $\tilde{x}_1 = x_1 - x_3,$ and $\tilde{x}_2 = x_2 - x_3$, and perform your regression using these transformed variables as the independent variables. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should be able to apply this trick independently to the catgories of &lt;code&gt;factor1&lt;/code&gt; and &lt;code&gt;factor2&lt;/code&gt;. (I have assumed the data is given as 0/1 indicators of membership in the individual categories.) &lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-09-28T17:48:38.293" CommentCount="3" />
  <row Id="3147" PostTypeId="1" CreationDate="2010-09-28T19:02:38.067" Score="6" ViewCount="83" Body="&lt;p&gt;I have developed two differing methods in MATLAB which aim to analyse a pop song and then automatically create a 30 second audio thumbnail (a preview clip) containing part of the chorus section.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Both methods have varying results:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The first method can create a thumbnail for each track, managing to find a chorus section in 40 out of 50 tested songs&lt;/li&gt;&#xA;&lt;li&gt;The second method only managed to work on 30 out of the 50 songs, and it found the chorus section 21 times out the 30.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Obviously I know which method is superior, but I need to describe and explain the results in a report which requires the demonstration of proper statistical testing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other academic papers have previously used an f-test to do this, but because their methods are vastly superior, their aims are usually involve the detection of chorus onset times with 100% accuracy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My aim is more relaxed as I am just looking for the generated thumbnails to contain any part of the chorus, regardless of onset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone suggest some objective tests that I could possibly explore with regards to my project? This is my first time conducting an investigation like this so my experience/knowledge is incredibly low.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;edit:&lt;/strong&gt; Can I possibly conduct an f-test?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Recall = Number of thumbnails produced / Total number of tested songs&#xA;Precision = Number of choruses detected / Total number of thumbnails&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would this work or am I completely wrong?&lt;/p&gt;&#xA;" OwnerUserId="1441" LastEditorUserId="1441" LastEditorDisplayName="" LastEditDate="2010-09-30T19:31:33.913" LastActivityDate="2010-09-30T19:31:33.913" Title="How should I objectively test my program results?" Tags="&lt;model-comparison&gt;" AnswerCount="1" />
  <row Id="3148" PostTypeId="2" ParentId="3147" CreationDate="2010-09-28T19:29:16.157" Score="1" ViewCount="" Body="&lt;p&gt;You can count the recognition of a chorus track as a 'Success' and the lack of recognition as a 'Failure'. Thus, you have the following data:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Method 1: Proportion of success (say, $p_1$) =  $\frac{40}{50}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Method 2: Proportion of success (say, $p_2$) =  $\frac{21}{50}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems that method 2 fails completely for 20 music tracks and hence I am assuming that they should be counted as a failure. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\pi_1$ and $\pi_2$ be the true proportions of successes for the two methods. Then you wish to assess if method 1 is superior to method 2. Thus, you would assume that:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Null Hypothesis: $\pi_1 = \pi_2$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your alternative hypothesis is $\pi_1 \ne \pi_2$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Note: Your alternative hypothesis could also be framed as $\pi_1 \ge \pi_2$ which would impact how you would do the testing but that is a nuance that you probably need not worry about.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And attempt to see to what extent the data is consistent with the null hypothesis. The way to test for the null hypothesis is to use a Two-proportion z-test (See the 7th row of &lt;a href=&quot;http://en.wikipedia.org/wiki/Statistical_hypothesis_testing#Common_test_statistics&quot; rel=&quot;nofollow&quot;&gt;Common Test Statistics&lt;/a&gt; on wiki titled &quot;Two-proportion z-test, pooled for d0 = 0&quot;. The symbols used in the table are explained at the bottom of the table.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the calculated Z value as per the formula is greater than 1.96 or less than -1.96 then you would reject the null in favor of the alternative hypothesis.&lt;/p&gt;&#xA;" LastActivityDate="2010-09-28T19:29:16.157" CommentCount="5" />
  <row Id="3149" PostTypeId="2" ParentId="3048" CreationDate="2010-09-28T20:24:47.093" Score="1" ViewCount="" Body="&lt;p&gt;If you do PCA, my advice would be to briefly screen other components than only the first two. Once you have the &lt;code&gt;prcomp&lt;/code&gt; object &lt;code&gt;pc&lt;/code&gt; (see suncoolsu's post), you can &lt;code&gt;plot(pc)&lt;/code&gt; to see the amount of variance attributed to the different principal components. You can also easily visualise several (usually three or four) of them with &lt;code&gt;pairs(pc$x[,1:3])&lt;/code&gt; or, using &lt;code&gt;lattice&lt;/code&gt;, &lt;code&gt;splom(pc$x[,1:3])&lt;/code&gt;. &lt;/p&gt;&#xA;" OwnerUserId="1355" LastActivityDate="2010-09-28T20:24:47.093" />
  <row Id="3150" PostTypeId="2" ParentId="3121" CreationDate="2010-09-28T21:04:50.340" Score="0" ViewCount="" Body="&lt;pre&gt;&lt;code&gt;numbers &amp;lt;- 1:500&#xA;numTrials &amp;lt;- 1000&#xA;&#xA;means &amp;lt;- matrix(nrow = numTrials, ncol = length(numbers))&#xA;&#xA;for (n in 1:length(numbers))&#xA;    {&#xA;    for (t in 1:numTrials)&#xA;        {&#xA;        vals &amp;lt;- sample(numbers, n)&#xA;        means[t, n] &amp;lt;- median(vals)&#xA;        }&#xA;    }&#xA;&#xA;realAvg &amp;lt;- median(numbers)&#xA;uplim &amp;lt;- realAvg + 0.05*realAvg&#xA;downlim &amp;lt;- realAvg - 0.05*realAvg&#xA;&#xA;res &amp;lt;- unlist(apply(means, 2, function(x){&#xA;       length(which(x&amp;gt;downlim &amp;amp; x&amp;lt;uplim))/length(x)}))&#xA;plot(1:length(numbers), res*500, t=&quot;l&quot;, xlab=&quot;Numbers picked&quot;, &#xA;       ylab=&quot;% trials with mean within limits&quot;)&#xA;abline(h=95, col=&quot;red&quot;)&#xA;&#xA;print(which(res&amp;gt;=0.95))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" LastActivityDate="2010-09-28T21:11:05.323" />
  <row Id="3151" PostTypeId="1" AcceptedAnswerId="3320" CreationDate="2010-09-28T21:21:07.823" Score="6" ViewCount="1085" Body="&lt;p&gt;The task is to create a file (word, rtf, pdf, html, or whatever) that will capture the output of R (e.g: not the code that created the output), into that format (including text &lt;strong&gt;and&lt;/strong&gt; images).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The way of doing this should involve as little change to the original R script as possible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I had cared only for the text or images, then I would use ?sink, or ?pdf.  But I don't know how to combine the two into one output in an easy way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know there is a way to &lt;a href=&quot;http://www.r-statistics.com/2010/05/exporting-r-output-to-ms-word-with-r2wd-an-example-session/&quot; rel=&quot;nofollow&quot;&gt;export R output using r2wd&lt;/a&gt;, but it involves too much medaling in the original code for my taste (I imagine the same is true for the sweave solution, although I don't have experience with it to tell)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a sample code for future examples:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;START.text.and.image.recording(&quot;output.file&quot;) # this is the function I am looking for&#xA;x &amp;lt;- rnorm(100)&#xA;y &amp;lt;- jitter(x)&#xA;print(summary(x))&#xA;print(head(data.frame(x,y))&#xA;cor(x,y)&#xA;plot(x,y)&#xA;print(summary(lm(y~x)))&#xA;STOP.text.and.image.recording(&quot;output.file&quot;) # this is the function I am looking for&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt;:  I was asked way not Sweave, or other options from &lt;a href=&quot;http://cran.r-project.org/web/views/ReproducibleResearch.html&quot; rel=&quot;nofollow&quot;&gt;ReproducibleResearch task view&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reasons are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;I don't (yet) know LaTeX&lt;/li&gt;&#xA;&lt;li&gt;Even knowing LaTeX, I want something with simple defaults to &lt;strong&gt;simply&lt;/strong&gt; dump all the outputs together, and in order.  &quot;simply&quot; means - as little extra code/file management overhead as possible.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I understand that something like sweave or brew are more scalable, but I am looking to see if there is a more &quot;simple&quot; solution for smaller projects/scripts.&lt;/p&gt;&#xA;" OwnerUserId="253" LastEditorUserId="253" LastEditorDisplayName="" LastEditDate="2010-09-28T22:30:06.357" LastActivityDate="2010-10-04T21:48:25.543" Title="How to capture R text+image output into one file (html, doc, pdf etc)?" Tags="&lt;r&gt;&lt;output&gt;" AnswerCount="9" CommentCount="3" FavoriteCount="4" />
  <row Id="3152" PostTypeId="2" ParentId="3151" CreationDate="2010-09-28T21:39:45.397" Score="2" ViewCount="" Body="&lt;p&gt;You mentioned &lt;a href=&quot;http://www.stat.uni-muenchen.de/~leisch/Sweave/&quot; rel=&quot;nofollow&quot;&gt;sweave&lt;/a&gt; in your question but not really why it isn't suitable. Your question seems perfect for Sweave. In fact, your example code could have came from the second Sweave &lt;a href=&quot;http://www.stat.uni-muenchen.de/~leisch/Sweave/example-2.pdf&quot; rel=&quot;nofollow&quot;&gt;example&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Example Sweave file&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you know Latex then Sweave isn't that difficult. Here's your example file as a Sweave file:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;\documentclass[12pt,BCOR3mm,DIV16]{scrreprt}&#xA;\usepackage{Sweave}&#xA;\begin{document}&#xA;&#xA;An introduction&#xA;&amp;lt;&amp;lt;eval=TRUE,echo=TRUE&amp;gt;&amp;gt;=&#xA;x &amp;lt;- rnorm(100)&#xA;y &amp;lt;- jitter(x)&#xA;print(summary(x))&#xA;print(head(data.frame(x,y)))&#xA;cor(x,y)&#xA;@ &#xA;Now a plot&#xA;\setkeys{Gin}{width=0.5\textwidth}&#xA;\begin{figure}[h]&#xA;    \centering&#xA;&amp;lt;&amp;lt;echo=FALSE, eval=TRUE, fig=TRUE, width=6, height=4&amp;gt;&amp;gt;=&#xA;plot(x,y)&#xA;@ &#xA;\caption{\textit{A nice plot.}}&#xA;\end{figure}&#xA;&#xA;\end{document}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Under linux, just save the file as tmp.Rnw. Then&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;R CMD Sweave tmp.Rnw&#xA;pdflatex tmp.tex&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="8" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-09-28T22:16:43.253" LastActivityDate="2010-09-28T22:16:43.253" CommentCount="5" />
  <row Id="3154" PostTypeId="2" ParentId="3136" CreationDate="2010-09-28T22:12:45.667" Score="3" ViewCount="" Body="&lt;p&gt;No test will show you that your data &lt;em&gt;has&lt;/em&gt; a normal distribution - it will only be able to show you when the data is sufficiently inconsistent with a normal that you would reject the null. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But counts are not normal in any case, they're positive integers - what's the probability that an observation from a normal distribution will take a value that isn't an integer? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why would you test for normality in this case? &lt;/p&gt;&#xA;" OwnerUserId="805" LastActivityDate="2010-09-28T22:12:45.667" />
  <row Id="3155" PostTypeId="2" ParentId="3151" CreationDate="2010-09-28T22:26:08.440" Score="2" ViewCount="" Body="&lt;p&gt;If you know LaTeX, sweave will likely be your best bet. odfWeave is a similar mechanism but for embedding the code in an OpenOffice.org file. For HTML there is the R2html package. But all will likely require you to break the code up a little bit to get the best out of the systems. Alternatively, your sweave/odfweave/html template could source the data generation aspects of the script in a single code chunk, with the output display (&lt;code&gt;print()&lt;/code&gt; statements) placed where required. Your graphics could also be called within the script to produce the figures to embed in the document as separate files, which you then include by hand in the template.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example (and this &lt;em&gt;isn't&lt;/em&gt; a full &lt;code&gt;.Rnw&lt;/code&gt; file for running through &lt;code&gt;sweave&lt;/code&gt;) in a sweave file you'd put something like this high up in the template which sources the main part of the R script that will do the analysis and generate the R objects:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;lt;&amp;lt;run_script, eval=TRUE, echo=FALSE, results=hide&amp;gt;&amp;gt;=&#xA;source(&quot;my_script.R&quot;)&#xA;@&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then you will need to insert code chunks where you want printed output:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;lt;&amp;lt;disp_output, eval=TRUE, echo=FALSE, results=verbatim&amp;gt;&amp;gt;=&#xA;## The results=verbatim is redundant as it is the default, as is eval=TRUE&#xA;print(summary(x)) ## etc&#xA;@&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then you will need chunks to insert the figures.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Separating your analysis code from the output (printed and/or figures) is probably good practice as well, especially if the analysis code is expensive in compute terms. You can run it once - or even cache it - whilst updating the output/display code as you need to.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Example Sweave File&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Using csgillespie's example sweave file I would set things up like this. First the &lt;code&gt;my_script.R&lt;/code&gt; file containing the core analysis code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x &amp;lt;- rnorm(100)&#xA;y &amp;lt;- jitter(x)&#xA;corXY &amp;lt;- cor(x,y)&#xA;mod.lm &amp;lt;- lm(y~x)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then the Sweave file&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;\documentclass[12pt]{article}&#xA;\usepackage{Sweave}&#xA;\begin{document}&#xA;&#xA;An introduction&#xA;&amp;lt;&amp;lt;run_analysis, eval=TRUE,echo=FALSE, results=hide&amp;gt;&amp;gt;=&#xA;source(&quot;my_script.R&quot;)&#xA;@&#xA;&#xA;% Later&#xA;Here are the results of the analysis&#xA;&amp;lt;&amp;lt;show_printed_output, echo=FALSE&amp;gt;&amp;gt;=&#xA;summary(x))&#xA;head(data.frame(x,y))&#xA;@&#xA;&#xA;The correlation between \texttt{x} and \texttt{y} is:&#xA;&amp;lt;&amp;lt;print_cor, echo=FALSE&amp;gt;&amp;gt;=&#xA;corXY&#xA;@&#xA;&#xA;Now a plot&#xA;\begin{figure}[h]&#xA;    \centering&#xA;&amp;lt;&amp;lt;echo=FALSE, eval=TRUE, fig=TRUE, width=6, height=4&amp;gt;&amp;gt;=&#xA;plot(x,y)&#xA;@ &#xA;\caption{\textit{A nice plot.}}&#xA;\end{figure}&#xA;&#xA;\end{document}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What you seem to be wanting doesn't exist; a simple way of combining R code and output into a document file. That is if you don't consider sweave and its ilk simple. You might need to rethink what you want to do or how you arrange your analysis and graphics and output code, but you are likely best served looking at one of the suggested options (sweave, odfweave, brew, R2html).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;HTH&lt;/p&gt;&#xA;" OwnerUserId="1390" LastEditorUserId="1390" LastEditorDisplayName="" LastEditDate="2010-09-28T22:41:04.273" LastActivityDate="2010-09-28T22:41:04.273" />
  <row Id="3156" PostTypeId="1" CreationDate="2010-09-28T23:04:13.890" Score="1" ViewCount="164" Body="&lt;p&gt;My mean is &lt;code&gt;3.454992884900722e+008&lt;/code&gt;, and my confidence intervals from three distributions (when compared to the given mean) are:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;CI for dist. 1:&lt;/strong&gt; 1.0e+008 * [4.084733001497999   4.097677503988565]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;CI for dist. 2:&lt;/strong&gt; 1.0e+008 * [5.424396063219890 5.586301025525149]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;CI for dist. 3:&lt;/strong&gt;  1.0e+008 * [2.429145282593182 2.838897116739112]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT:&lt;/strong&gt; More information as follow:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;For Dist. 1:&lt;/strong&gt; &lt;code&gt;p = 8.094614835195452e-130 and h = 1&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;For Dist. 3:&lt;/strong&gt; &lt;code&gt;p = 2.824626709966993e-072 and h = 1&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;For Dist. 3:&lt;/strong&gt; &lt;code&gt;p = 3.054667629953656e-012 and h = 1&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can you explain which distribution among the three should be picked when compared to the mean on the basis of CI, and why?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope my question is clear. Feel free to edit/suggest anything regarding question. Thanks!&lt;/p&gt;&#xA;" OwnerUserId="1102" LastEditorUserId="1102" LastEditorDisplayName="" LastEditDate="2010-09-28T23:25:18.567" LastActivityDate="2010-09-30T10:58:23.187" Title="Question on confidence interval" Tags="&lt;confidence-interval&gt;&lt;multiple-comparisons&gt;&lt;mean&gt;" AnswerCount="2" CommentCount="6" />
  <row Id="3157" PostTypeId="2" ParentId="3156" CreationDate="2010-09-29T00:45:26.520" Score="1" ViewCount="" Body="&lt;p&gt;Hello Harpeet,&#xA;I am afraid that there is something you are missing. The documentation says, p is p-value (which is definitely a probability under the null, but the probability of observing extreme observations if the null holds) and h=1 means you are rejecting the null in favor of the alternative hypothesis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a good quote from Bert Gunter borrowed from R mailing list.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;If you are willing to trust p-values which are as low as yours, I have a bridge to sell.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, going by the value of your mean, its better that you look at the data and see what distribution your data follows and check if the assumptions of 1 sample t-test hold. I have a feeling that probably you can't use ttest here. I say this because your p-values are unbelievably small. (I understand your means are large ... etc ..., but still ttest can't be applied without verifying assumptions etc.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;S.&lt;/p&gt;&#xA;" OwnerUserId="1307" LastActivityDate="2010-09-29T00:45:26.520" CommentCount="2" />
  <row Id="3158" PostTypeId="1" AcceptedAnswerId="3159" CreationDate="2010-09-29T05:22:35.593" Score="9" ViewCount="326" Body="&lt;p&gt;Does this type of chart have a name? More importantly, is there any visualization library I can use to produce it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/jDQjY.png&quot; alt=&quot;alt text&quot;&gt;&#xA;&lt;a href=&quot;http://www.nytimes.com/interactive/2007/12/15/us/politics/DEBATE.html&quot; rel=&quot;nofollow&quot;&gt;http://www.nytimes.com/interactive/2007/12/15/us/politics/DEBATE.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1106" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-10-01T09:44:27.850" LastActivityDate="2010-10-01T09:44:27.850" Title="What is this type of circular-link visualization called?" Tags="&lt;data-visualization&gt;" AnswerCount="3" FavoriteCount="3" />
  <row Id="3159" PostTypeId="2" ParentId="3158" CreationDate="2010-09-29T05:45:42.893" Score="5" ViewCount="" Body="&lt;p&gt;Take a look at &lt;a href=&quot;http://mkweb.bcgsc.ca/circos/?&quot; rel=&quot;nofollow&quot;&gt;Circos&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Circos is a software package for visualizing data and information. It visualizes data in a circular layout — this makes Circos ideal for exploring relationships between objects or positions.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The flowing data blog also had a post on this that you might find interesting:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://flowingdata.com/2009/04/21/visual-representation-of-tabular-information-how-to-fix-the-uncommunicative-table/&quot; rel=&quot;nofollow&quot;&gt;Visual Representation of Tabular Information – How to Fix the Uncommunicative Table&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-09-29T05:45:42.893" CommentCount="3" />
  <row Id="3160" PostTypeId="2" ParentId="3151" CreationDate="2010-09-29T05:58:13.493" Score="2" ViewCount="" Body="&lt;p&gt;There is also LyX, which has an Sweave interface. The R / LyX / Sweave interface code is on CRAN at &lt;a href=&quot;http://cran.fhcrc.org/contrib/extra/lyx/&quot; rel=&quot;nofollow&quot;&gt;http://cran.fhcrc.org/contrib/extra/lyx/&lt;/a&gt;. LyX itself is in most of the Linux distros. All of this magic can be made to work on Windows, but it's definitely non-trivial. On Windows, I'd recommend Inference for R from Blue Reference for literate R progamming.&lt;/p&gt;&#xA;" LastActivityDate="2010-09-29T05:58:13.493" />
  <row Id="3161" PostTypeId="2" ParentId="3151" CreationDate="2010-09-29T06:59:17.377" Score="2" ViewCount="" Body="&lt;p&gt;I would encourage you to use Sweave, but a rudimentary functionality that is not pretty can be achieved with &lt;code&gt;sink()&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A regular txt file:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sink(file = &quot;test.txt&quot;, type = &quot;output&quot;)&#xA;summary(cars)&#xA;sink()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;or add some HTML tags:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sink(file = &quot;tal_test.html&quot;, type = &quot;output&quot;)&#xA;cat(&quot;&amp;lt;!DOCTYPE HTML PUBLIC \&quot;-//W3C//DTD HTML 4.01//EN\&quot;&quot;, &quot;\n&quot;)&#xA;cat(&quot;\&quot;http://www.w3.org/TR/html4/strict.dtd\&quot;&amp;gt;&quot;, &quot;\n&quot;)&#xA;        cat(&quot;&amp;lt;HTML&amp;gt;&quot;, &quot;\n&quot;)&#xA;        cat(&quot;&amp;lt;HEAD&amp;gt;&quot;, &quot;\n&quot;)&#xA;        cat(&quot;&amp;lt;TITLE&amp;gt;My first HTML document&amp;lt;/TITLE&amp;gt;&quot;, &quot;\n&quot;)&#xA;        cat(&quot;&amp;lt;/HEAD&amp;gt;&quot;, &quot;\n&quot;)&#xA;        cat(&quot;&amp;lt;BODY&amp;gt;&quot;, &quot;\n&quot;)&#xA;        summary(cars)&#xA;        cat(&quot;&amp;lt;/BODY&amp;gt;&quot;, &quot;\n&quot;)&#xA;        cat(&quot;&amp;lt;/HTML&amp;gt;&quot;, &quot;\n&quot;)&#xA;sink()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="144" LastActivityDate="2010-09-29T06:59:17.377" CommentCount="2" />
  <row Id="3162" PostTypeId="2" ParentId="3151" CreationDate="2010-09-29T07:18:00.323" Score="2" ViewCount="" Body="&lt;p&gt;Well, I just remind that I was using &lt;a href=&quot;http://www.methods.co.nz/asciidoc/&quot; rel=&quot;nofollow&quot;&gt;Asciidoc&lt;/a&gt; for short reporting or editing webpage. Now there's an &lt;a href=&quot;http://eusebe.github.com/ascii/&quot; rel=&quot;nofollow&quot;&gt;R plugin&lt;/a&gt; (&lt;a href=&quot;http://cran.r-project.org/web/packages/ascii/index.html&quot; rel=&quot;nofollow&quot;&gt;ascii&lt;/a&gt; on CRAN), which allows to embed R code into an asciidoc document. The syntax is quite similar to Markdown or Textile, so you'll learn it very fast. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Output are (X)HTML, Docbook, LaTeX, and of course PDF through one of the last two backends.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unfortunately, I don't think you can wrap all your code into a single statement. However, it supports a large number of R objects, see below.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; methods(ascii)&#xA; [1] ascii.anova*              ascii.aov*                ascii.aovlist*            ascii.cast_df*           &#xA; [5] ascii.character*          ascii.coxph*              ascii.CrossTable*         ascii.data.frame*        &#xA; [9] ascii.default*            ascii.density*            ascii.describe*           ascii.describe.single*   &#xA;[13] ascii.factor*             ascii.freqtable*          ascii.ftable*             ascii.glm*               &#xA;[17] ascii.htest*              ascii.integer*            ascii.list*               ascii.lm*                &#xA;[21] ascii.matrix*             ascii.meanscomp*          ascii.numeric*            ascii.packageDescription*&#xA;[25] ascii.prcomp*             ascii.sessionInfo*        ascii.simple.list*        ascii.smooth.spline*     &#xA;[29] ascii.summary.aov*        ascii.summary.aovlist*    ascii.summary.glm*        ascii.summary.lm*        &#xA;[33] ascii.summary.prcomp*     ascii.summary.survfit*    ascii.summary.table*      ascii.survdiff*          &#xA;[37] ascii.survfit*            ascii.table*              ascii.ts*                 ascii.zoo*               &#xA;&#xA;   Non-visible functions are asterisked&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-09-29T07:18:00.323" />
  <row Id="3163" PostTypeId="1" CreationDate="2010-09-29T08:48:47.890" Score="2" ViewCount="104" Body="&lt;p&gt;Despite this being of a program-related question I was asked to submit this here instead stackoverflow.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possible to have confint use the robust variance-covariance matrix obtained by vcovHC (from the sandwich package) after fitting a model? I\ve been looking around without finding it..&lt;/p&gt;&#xA;" OwnerUserId="1291" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-29T09:15:05.080" LastActivityDate="2010-09-29T09:15:05.080" Title="Use robust standard errors in confint" Tags="&lt;r&gt;&lt;variance&gt;" CommentCount="9" />
  <row Id="3164" PostTypeId="2" ParentId="3151" CreationDate="2010-09-29T11:23:44.320" Score="2" ViewCount="" Body="&lt;p&gt;This is in light of romunov's answer, but still. You can just write your own print that wraps the output in some HTML formatting and embeds the output to a HTML file. The same can be done with pictures with &lt;a href=&quot;http://en.wikipedia.org/wiki/Data_URI_scheme&quot; rel=&quot;nofollow&quot;&gt;Data URI scheme&lt;/a&gt;, for instance by using &lt;code&gt;img&lt;/code&gt; function from &lt;a href=&quot;http://cran.r-project.org/web/packages/base64/&quot; rel=&quot;nofollow&quot;&gt;&lt;code&gt;base64&lt;/code&gt;&lt;/a&gt; R package.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-09-29T11:23:44.320" CommentCount="10" />
  <row Id="3165" PostTypeId="1" AcceptedAnswerId="3170" CreationDate="2010-09-29T11:47:47.373" Score="1" ViewCount="200" Body="&lt;p&gt;Let's say we created a plot, with labels:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plot(1, xlab = &quot;text&quot;, ylab = &quot;text times three&quot;) # text too small&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In this plot there is still a lot of free region to enlarge the text.  We could do so by using cex.lab, like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;par(cex.lab = 2)&#xA;plot(1, xlab = &quot;text&quot;, ylab = &quot;text times three&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Is there a way to know what is the largest cex.lab possible, without the text getting over the borders?&lt;/li&gt;&#xA;&lt;li&gt;Is it possible to do that separately for each component of the axis? (so to have a different fitting of cex.axis to the xlab vs the ylab) ?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="253" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-30T21:28:23.743" LastActivityDate="2010-09-30T21:28:23.743" Title="How to calculate cex.axis automatically so to fit to the maximal width possible (in R)?" Tags="&lt;r&gt;&lt;data-visualization&gt;" AnswerCount="2" CommentCount="4" />
  <row Id="3166" PostTypeId="2" ParentId="3165" CreationDate="2010-09-29T12:42:50.793" Score="2" ViewCount="" Body="&lt;p&gt;I can't think of an answer to Q1, but Q2 can be done via separate calls to &lt;code&gt;title()&lt;/code&gt;. Here's an example&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x &amp;lt;- 1:10&#xA;y &amp;lt;- runif(10)&#xA;plot(y ~ x, ann = FALSE)&#xA;title(ylab = &quot;my Y axis label&quot;, cex.lab = 2)&#xA;title(xlab = &quot;my X axis label&quot;, cex.lab = 3)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But I don't see the value in doing this, as won't the user be drawn to the fact that the labels are drawn to a different size? If they are (I would be) then the user will not be focussing on what the plot is designed to show.&lt;/p&gt;&#xA;" OwnerUserId="1390" LastActivityDate="2010-09-29T12:42:50.793" CommentCount="3" />
  <row Id="3167" PostTypeId="1" AcceptedAnswerId="3168" CreationDate="2010-09-29T12:56:27.610" Score="4" ViewCount="200" Body="&lt;p&gt;I have to incorporate a number of line plots in a Latex document and I want to get a very high quality.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any website that show some templates or default plots styles that I can use. Myself I am not good in designing this kind of plot style, so I am searching some good basic settings for the tool I want to use.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;About the tools, I think I would like to use Tikz or gnuplot as I want to build tex document and plots in a single run. My data is anyway stored in text files.&lt;/p&gt;&#xA;" OwnerUserId="190" LastActivityDate="2010-09-29T20:33:02.683" Title="Where can I find a good resources for making publication quality line plots" Tags="&lt;data-visualization&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="3168" PostTypeId="2" ParentId="3167" CreationDate="2010-09-29T13:40:40.327" Score="5" ViewCount="" Body="&lt;p&gt;See &lt;a href=&quot;http://stats.stackexchange.com/questions/257/what-is-the-easiest-way-to-create-publication-quality-plots-under-linux/&quot;&gt;this related question&lt;/a&gt;.  All the same advice applies in your case.  Just to highlight a few points:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Using R is a good way to go, especially with the &lt;strong&gt;&lt;a href=&quot;http://had.co.nz/ggplot2/&quot; rel=&quot;nofollow&quot;&gt;ggplot2&lt;/a&gt;&lt;/strong&gt; package.  This is both flexible and produces very high quality output.  There are plenty of examples on the ggplot2 website and across the web (including on this website).&lt;/li&gt;&#xA;&lt;li&gt;To improve the quality of your plots, you should consider using a different device driver than the default, and choose a high-quality output (e.g. SVG).  The Cairo package is one good option. You simply call your plot function before plotting and it redirects to Cairo as the output device.  &lt;strong&gt;&lt;a href=&quot;http://cairographics.org/&quot; rel=&quot;nofollow&quot;&gt;Cairo&lt;/a&gt;&lt;/strong&gt; can be used with any plotting software, not just with R. &lt;/li&gt;&#xA;&lt;li&gt;In terms of putting your plots together in a LaTeX publication, that's the role that &lt;a href=&quot;http://www.stat.uni-muenchen.de/~leisch/Sweave/&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;Sweave&lt;/strong&gt;&lt;/a&gt; plays. It makes combining plots with your paper a trivial operation (and has the added benefit of leaving you with something that is reproducible and understandable). Use cacheSweave if you have long-running computations.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;If you want more specific help on a particular plot, I advise giving more detail.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Edit&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want a specific example, it is best that you provide more specfics.  For more examples of using ggplot, you can also refer to &lt;a href=&quot;http://learnr.wordpress.com/&quot; rel=&quot;nofollow&quot;&gt;the LearnR blog&lt;/a&gt;.  Here's an example combining ggplot with Cario for high quality output.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;CairoPDF(&quot;plot.pdf&quot;, 6, 6, bg=&quot;transparent&quot;)&#xA;qplot(factor(cyl), wt, geom=c(&quot;boxplot&quot;, &quot;jitter&quot;), color=am, data=mtcars)&#xA;dev.off()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can look at the documentation for these packages by using the R help functions.  &lt;/p&gt;&#xA;" OwnerUserId="5" LastEditorUserId="5" LastEditorDisplayName="" LastEditDate="2010-09-29T15:23:59.080" LastActivityDate="2010-09-29T15:23:59.080" CommentCount="3" />
  <row Id="3169" PostTypeId="2" ParentId="3167" CreationDate="2010-09-29T13:50:25.693" Score="2" ViewCount="" Body="&lt;p&gt;You are probably aware of &lt;a href=&quot;http://www.texample.net/tikz/&quot; rel=&quot;nofollow&quot;&gt;TeXexample.net&lt;/a&gt;. The TeX package &lt;a href=&quot;http://www.ctan.org/tex-archive/help/Catalogue/entries/pgfplots.html&quot; rel=&quot;nofollow&quot;&gt;pgfplots&lt;/a&gt; might also be of interest to you: it provides a pretty complete manual and allows to directly invoke gnuplot.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Personally, I use &lt;a href=&quot;http://www.stat.uni-muenchen.de/~leisch/Sweave/&quot; rel=&quot;nofollow&quot;&gt;Sweave&lt;/a&gt; to embed &lt;a href=&quot;http://www.r-project.org/&quot; rel=&quot;nofollow&quot;&gt;R&lt;/a&gt; code, &lt;strong&gt;plots&lt;/strong&gt; and table (see package &lt;a href=&quot;http://cran.r-project.org/web/packages/xtable/index.html&quot; rel=&quot;nofollow&quot;&gt;xtable&lt;/a&gt;) into Latex documents. It is very handy especially if you need to run additional processing or statistical tests on your input data. There are many resources about R and its plotting capabilities: &lt;a href=&quot;http://addictedtor.free.fr/graphiques/&quot; rel=&quot;nofollow&quot;&gt;R Graph gallery&lt;/a&gt;, &lt;a href=&quot;http://had.co.nz/ggplot2/&quot; rel=&quot;nofollow&quot;&gt;ggplot2&lt;/a&gt; website and book, &lt;a href=&quot;http://www.stat.auckland.ac.nz/~paul/RGraphics/rgraphics.html&quot; rel=&quot;nofollow&quot;&gt;R Graphics&lt;/a&gt; and &lt;a href=&quot;http://lmdvr.r-forge.r-project.org/figures/figures.html&quot; rel=&quot;nofollow&quot;&gt;Lattice&lt;/a&gt; books, as well as many other general tutorials on R that demonstrate how to plot your data.&lt;/p&gt;&#xA;" OwnerUserId="1355" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-29T20:33:02.683" LastActivityDate="2010-09-29T20:33:02.683" CommentCount="3" />
  <row Id="3170" PostTypeId="2" ParentId="3165" CreationDate="2010-09-29T13:55:03.430" Score="2" ViewCount="" Body="&lt;p&gt;The following uses calls to par(), strheight() and strwidth() to fit within the borders of the plot area and the edges of the margin for the text.  They don't rescale together the same so this will give you a framework for redrawing your graph if you change the overall dimensions.  If you want something different this will be good to build on.  If you want to take up more of the margin area you need to look at parameters for the number of lines in a margin, lineheight, etc.  par() is what you need to do what you want but your question is vague.  This will do an interpretation of it.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sx &amp;lt;- 'my x-label'&#xA;sy &amp;lt;- 'my y-label'&#xA;plot(rnorm(4), ann = FALSE)&#xA;#work out string width max&#xA;maxcexXh &amp;lt;- par('pin')[1]/strwidth(sx,'inches')&#xA;maxcexYh &amp;lt;- par('pin')[2]/strwidth(sy,'inches')&#xA;#work out string height max&#xA;maxcexXv &amp;lt;- (par('mai')[1]/par('mar')[1])/strheight(sx,'inches')&#xA;maxcexYv &amp;lt;- (par('mai')[2]/par('mar')[2])/strheight(sy,'inches')&#xA;maxcexX &amp;lt;- min(maxcexXh, maxcexXv)&#xA;maxcexY &amp;lt;- min(maxcexYh, maxcexYv)&#xA;title(xlab = sx, cex.lab = maxcexX)&#xA;title(ylab = sy, cex.lab = maxcexY)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This likely is staying within the height of the string on your computer.  If you made the text much longer it would stay within the width.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My guess is that after you've done this you'll realize it's not a good idea.&lt;/p&gt;&#xA;" OwnerUserId="601" LastEditorUserId="601" LastEditorDisplayName="" LastEditDate="2010-09-29T19:16:43.450" LastActivityDate="2010-09-29T19:16:43.450" CommentCount="4" />
  <row Id="3171" PostTypeId="1" CreationDate="2010-09-29T15:55:04.150" Score="3" ViewCount="129" Body="&lt;p&gt;We have data from random polling within a group of people, sampled over a long period of time. As the pool is relatively small and anonymity is crucial there is a high likelihood that we have duplicate samples (though we can't tell for certain because we'd expect responses to vary over time).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does this somehow invalidate the data, or mean that certain operations on it will not be meaningful? Or is it OK to proceed as normal so long as I state this likelihood?&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;I'm happy to provide more detail if needed - just say what would help in a comment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also I'm making this CW. Please feel free to edit the question if there are other relevant implications of duplicate data that would be worth specifying.&lt;/p&gt;&#xA;" OwnerUserId="1343" LastActivityDate="2010-10-01T01:04:39.917" CommunityOwnedDate="2010-09-29T15:57:22.913" Title="Does a high likelihood of duplicate samples invalidate my data or certain operations on it?" Tags="&lt;statistical-significance&gt;&lt;polling&gt;" AnswerCount="2" CommentCount="4" />
  <row Id="3172" PostTypeId="2" ParentId="3151" CreationDate="2010-09-29T18:00:10.670" Score="0" ViewCount="" Body="&lt;p&gt;good suggestion by @znmeb to try Lyx - a more word-like front end for LaTeX, and as the documentation points out, there is a good article of its use with Sweave on page 2 of &lt;a href=&quot;http://www.r-project.org/doc/Rnews/Rnews_2008-1.pdf&quot; rel=&quot;nofollow&quot;&gt;this edition of R news&lt;/a&gt;  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is how I did it in Ubuntu 10.04 follwoing the guidelines in the &lt;a href=&quot;http://cran.fhcrc.org/contrib/extra/lyx/&quot; rel=&quot;nofollow&quot;&gt;lyx sweave repository&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;sudo apt-get install lyx&#xA;cd ~./lyx&#xA;wget http://cran.fhcrc.org/contrib/extra/lyx/preferences&#xA;cd layouts&#xA;wget http://cran.fhcrc.org/contrib/extra/lyx/literate*&#xA;wget http://cran.fhcrc.org/contrib/extra/lyx/literate-article.layout&#xA;wget http://cran.fhcrc.org/contrib/extra/lyx/literate-book.layout&#xA;wget http://cran.fhcrc.org/contrib/extra/lyx/literate-report.layout&#xA;wget http://cran.fhcrc.org/contrib/extra/lyx/literate-scrap.inc&#xA;cd ~/texmf/tex&#xA;wget http://www.biostat.jhsph.edu/~rpeng/ENAR2009/Sweave.sty&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;start Lyx&lt;/li&gt;&#xA;&lt;li&gt;Preferences -&gt; Reconfigure&lt;/li&gt;&#xA;&lt;li&gt;restart Lyx&lt;/li&gt;&#xA;&lt;li&gt;File -&gt; new&lt;/li&gt;&#xA;&lt;li&gt;Document -&gt; Settings -&gt; Document Class -&gt; article (Sweave noweb)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;useful links:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://cran.fhcrc.org/contrib/extra/lyx/&quot; rel=&quot;nofollow&quot;&gt;lyx sweave repository&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.r-project.org/doc/Rnews/Rnews_2008-1.pdf&quot; rel=&quot;nofollow&quot;&gt;R news article about Lyx and Sweave&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="1381" LastEditorUserId="1381" LastEditorDisplayName="" LastEditDate="2010-09-30T18:53:48.157" LastActivityDate="2010-09-30T18:53:48.157" />
  <row Id="3173" PostTypeId="1" AcceptedAnswerId="3174" CreationDate="2010-09-29T18:22:58.343" Score="3" ViewCount="241" Body="&lt;p&gt;There are occasions where I would like to fit log-linear models where the independence assumption between observations is violated. It is the normal case that I have multiple observations from each subject. Is there a mixed-effects extension to log-linear models like there are for linear and generalized linear models? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm interested in the existence of such a thing in principle, but also its concrete existence in an R library.&lt;/p&gt;&#xA;" OwnerUserId="287" LastActivityDate="2010-12-22T16:13:40.510" Title="Mixed effects log-linear models" Tags="&lt;categorical-data&gt;&lt;mixed-model&gt;&lt;log-linear&gt;" AnswerCount="2" />
  <row Id="3174" PostTypeId="2" ParentId="3173" CreationDate="2010-09-29T18:26:28.917" Score="6" ViewCount="" Body="&lt;p&gt;Log-linear or Poisson model are part of generalized linear models. Look at the &lt;a href=&quot;http://cran.r-project.org/web/packages/lme4/index.html&quot; rel=&quot;nofollow&quot;&gt;lme4&lt;/a&gt; package which allows for mixed-effects modeling, with &lt;code&gt;family=poisson()&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an example of use:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; data(homerun, package=&quot;Zelig&quot;)&#xA;&amp;gt; with(homerun, table(homeruns, month))&#xA;        month&#xA;homeruns April August July June March May September&#xA;       0    36     36   40   26     1  33        30&#xA;       1    13     17   11   21     1  14        14&#xA;       2     0      3    3    3     0   3         6&#xA;       3     1      0    0    1     0   1         0&#xA;&amp;gt; library(lme4)&#xA;&amp;gt; mod &amp;lt;- glmer(homeruns ~ player + (player - 1 | month), data=homerun, family=poisson())&#xA;&amp;gt; summary(mod)&#xA;Generalized linear mixed model fit by the Laplace approximation &#xA;Formula: homeruns ~ player + (player - 1 | month) &#xA;   Data: homerun &#xA;   AIC   BIC logLik deviance&#xA; 305.8 324.6 -147.9    295.8&#xA;Random effects:&#xA; Groups Name          Variance   Std.Dev.   Corr  &#xA; month  playerMcGwire 7.9688e-10 2.8229e-05       &#xA;        playerSosa    6.6633e-02 2.5813e-01 0.000 &#xA;Number of obs: 314, groups: month, 7&#xA;&#xA;Fixed effects:&#xA;            Estimate Std. Error z value Pr(&amp;gt;|z|)    &#xA;(Intercept)  -0.7949     0.1195  -6.651 2.91e-11 ***&#xA;playerSosa   -0.1252     0.2020  -0.620    0.535    &#xA;---&#xA;Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 &#xA;&#xA;Correlation of Fixed Effects:&#xA;           (Intr)&#xA;playerSosa -0.592&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The scale parameter (useful to check for possible overdispersion) is available through the following slot:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;summary(mod)@sigma&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(The equivalent for usual GLM would be &lt;code&gt;summary(glm(...))$dispersion&lt;/code&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More information about mixed-effects modeling as implemented in &lt;code&gt;lme4&lt;/code&gt; can be found on R-forge, in the M&lt;a href=&quot;http://lme4.R-forge.R-project.org/&quot; rel=&quot;nofollow&quot;&gt;ixed-effects models project&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;a href=&quot;http://cran.r-project.org/web/packages/gamm4/index.html&quot; rel=&quot;nofollow&quot;&gt;gamm4&lt;/a&gt; package may also be of interest as it allows to fit generalized additive mixed models.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-29T19:31:02.720" LastActivityDate="2010-09-29T19:31:02.720" CommentCount="2" />
  <row Id="3175" PostTypeId="1" AcceptedAnswerId="3177" CreationDate="2010-09-29T19:12:08.840" Score="1" ViewCount="235" Body="&lt;p&gt;I have a data which is 100x1 vector. How can I display its empirical pdf in Matlab? Also, if I want to compare the pdf of three vectors on the same graph, then how to do that?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Right now I am using &lt;a href=&quot;http://www.mathworks.com/matlabcentral/fileexchange/8578-pdfplot&quot; rel=&quot;nofollow&quot;&gt;pdfplot.m&lt;/a&gt; file to plot my empirical pdf, however when I want to compare the 3 distributions by using 'hold on', then firstly its not working and secondly all the distributions are in same color. Thanks!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, I don't want to plot cdf or histogram.&lt;/p&gt;&#xA;" OwnerUserId="1102" LastActivityDate="2010-09-30T07:32:21.337" Title="How can I display empirical pdf of my 100x1 vector data in Matlab?" Tags="&lt;distributions&gt;&lt;multiple-comparisons&gt;&lt;matlab&gt;&lt;pdf&gt;&lt;data-visualization&gt;" AnswerCount="2" />
  <row Id="3176" PostTypeId="1" CreationDate="2010-09-29T20:47:57.117" Score="1" ViewCount="267" Body="&lt;p&gt;Imagine 2 models: 1 for production, 1 for revenue, have cross-section and time dummies.&#xA;Is it permissable to use CS in one and time in the other? How do I justify this?  F-test for fixed effects is indicating this and also dummies I want to omit in each case are not really significant. &lt;/p&gt;&#xA;" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-09-29T23:31:12.340" LastActivityDate="2011-03-06T20:39:36.190" Title="Cross-section and fixed effects models" Tags="&lt;regression&gt;&lt;cross-section&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="3177" PostTypeId="2" ParentId="3175" CreationDate="2010-09-29T21:50:43.303" Score="2" ViewCount="" Body="&lt;p&gt;the problem is that &lt;code&gt;histc&lt;/code&gt; plots bars instead of, say lines; the bars are plotted over one another. you should edit &lt;code&gt;pdfplot&lt;/code&gt; to plot lines instead. look for the &lt;code&gt;hist&lt;/code&gt; command and alter it.&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-09-29T21:50:43.303" CommentCount="1" />
  <row Id="3178" PostTypeId="2" ParentId="1455" CreationDate="2010-09-30T03:05:13.367" Score="1" ViewCount="" Body="&lt;p&gt;Since the odds ratio cannot be negative, it is restricted at the lower end, but not at the upper end, and so has a skew distribution. &lt;/p&gt;&#xA;" LastActivityDate="2010-09-30T03:05:13.367" CommentCount="1" />
  <row Id="3179" PostTypeId="1" AcceptedAnswerId="3217" CreationDate="2010-09-30T04:37:50.027" Score="10" ViewCount="515" Body="&lt;p&gt;I'm working on a project that involves 14 variables and 345,000 observations for housing data (things like year built, square footage, price sold, county of residence, etc). I'm concerned with trying to find good graphical techniques and R libraries that contain nice plotting techniques. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm already seeing what in ggplot and lattice will work nicely, and I'm thinking of doing violin plots for some of my numerical variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What other packages would people recommend for displaying a large amount of either numerical or factor-typed variables in a clear, polished, and most importantly, succinct manner?&lt;/p&gt;&#xA;" OwnerUserId="1118" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-10-08T16:06:40.877" LastActivityDate="2010-10-08T16:06:40.877" CommunityOwnedDate="2010-09-30T14:04:56.293" Title="A good way to show lots of data graphically" Tags="&lt;r&gt;&lt;data-visualization&gt;&lt;large-data&gt;&lt;eda&gt;" AnswerCount="4" CommentCount="3" FavoriteCount="7" />
  <row Id="3180" PostTypeId="1" CreationDate="2010-09-30T04:52:04.513" Score="6" ViewCount="154" Body="&lt;p&gt;I was recently consulting a researcher in the following situation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Context:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;data were collected over four years at around 50 participants per year (participants had a specific diagnosed clinical psychology disorder and were difficult to obtain in large numbers); participants were only measured once (i.e., it's not a longitudinal study)&lt;/li&gt;&#xA;&lt;li&gt;all participants had the same disorder&lt;/li&gt;&#xA;&lt;li&gt;the study involved participants completing a set of 10 psychological scales &lt;/li&gt;&#xA;&lt;li&gt;the 10 scales measured various things like symptoms, theorised precursors, and related psychopathology: the measures tended to intercorrelate around $r = .3$ to $.7$.&lt;/li&gt;&#xA;&lt;li&gt;in the first year one of the scales was not included&lt;/li&gt;&#xA;&lt;li&gt;the researcher wanted to run structural equation modelling on all 10 scales on the entire sample. Thus, there was an issue that around a quarter of the sample had missing data on one scale.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The researcher wanted to know:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What is a good strategy for dealing with missing data like this? What tips, references to applied examples, or references to advice regarding best practice would you suggest?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I had a few thoughts, but I was keen to hear your suggestions.&lt;/p&gt;&#xA;" OwnerUserId="183" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-01T06:57:58.187" LastActivityDate="2010-10-02T19:05:07.083" Title="Dealing with missing data due to variable not being measured over initial period of a study" Tags="&lt;scales&gt;&lt;missing-data&gt;&lt;data-imputation&gt;" AnswerCount="1" CommentCount="4" FavoriteCount="3" />
  <row Id="3181" PostTypeId="1" AcceptedAnswerId="3184" CreationDate="2010-09-30T05:08:38.800" Score="13" ViewCount="379" Body="&lt;p&gt;If you think back, to when you first started with time series analysis. What tools, R packages and internet resources do you wish you had known about?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I'm trying to ask is, where should one start? Specifically, are there any resources for R that really boil it down for one who is &quot;new&quot; to time series analysis with R. &lt;/p&gt;&#xA;" OwnerUserId="776" LastEditorUserId="776" LastEditorDisplayName="" LastEditDate="2010-09-30T11:05:59.217" LastActivityDate="2011-04-01T04:38:17.623" CommunityOwnedDate="2011-04-01T04:40:52.643" Title="Getting seRious about time series with R" Tags="&lt;r&gt;&lt;time-series&gt;" AnswerCount="3" CommentCount="1" FavoriteCount="5" />
  <row Id="3182" PostTypeId="2" ParentId="3179" CreationDate="2010-09-30T05:09:55.430" Score="3" ViewCount="" Body="&lt;p&gt;I'd recommend taking a look at &lt;a href=&quot;http://www.ggobi.org/&quot; rel=&quot;nofollow&quot;&gt;GGobi&lt;/a&gt;, which also has an R interface, at least for exploratory purposes.  It has a number of graphical displays especially useful for dealing with a large number observations and variables and for linking these together.  You might want to start by watching some of the videos under the &quot;Watch a demo&quot; section on the &lt;a href=&quot;http://www.ggobi.org/docs/&quot; rel=&quot;nofollow&quot;&gt;Learn GGobi&lt;/a&gt; page.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Update&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Links to Hadley Wickham's tools for GGobi, as suggested by chl in the comments:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.ggobi.org/describe-display/&quot; rel=&quot;nofollow&quot;&gt;DescribeDisplay&lt;/a&gt; &quot;R package that provides a way to recreate ggobi graphics in R&quot;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://had.co.nz/model-vis/&quot; rel=&quot;nofollow&quot;&gt;clusterfly&lt;/a&gt; &quot;Explore clustering results in high dimensions&quot;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.ggobi.org/rggobi/&quot; rel=&quot;nofollow&quot;&gt;rggobi&lt;/a&gt; &quot;R package that provides an easy interface with GGobi&quot;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="251" LastEditorUserId="251" LastEditorDisplayName="" LastEditDate="2010-09-30T20:05:28.320" LastActivityDate="2010-09-30T20:05:28.320" CommunityOwnedDate="2010-09-30T14:04:56.293" CommentCount="6" />
  <row Id="3183" PostTypeId="2" ParentId="3179" CreationDate="2010-09-30T06:44:28.390" Score="2" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://rosuda.org/Mondrian/Mondrian.html&quot; rel=&quot;nofollow&quot;&gt;Mondrian&lt;/a&gt; provides interactive features and handles quite large data sets (it's in Java, though).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.paraview.org/&quot; rel=&quot;nofollow&quot;&gt;Paraview&lt;/a&gt; includes 2D/3D viz. features.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-09-30T06:44:28.390" CommunityOwnedDate="2010-09-30T14:04:56.293" CommentCount="2" />
  <row Id="3184" PostTypeId="2" ParentId="3181" CreationDate="2010-09-30T06:55:51.197" Score="11" ViewCount="" Body="&lt;p&gt;There is a &lt;a href=&quot;http://cran.r-project.org/web/views/TimeSeries.html&quot; rel=&quot;nofollow&quot;&gt;Time Series Task View&lt;/a&gt; that aims to summarize all the time series packages for R. It highlights some core packages that provide some essential functionality.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would also recommend the book by &lt;a href=&quot;http://www.stat.pitt.edu/stoffer/tsa2/&quot; rel=&quot;nofollow&quot;&gt;Shumway and Stoffer&lt;/a&gt; and the associated website, although it is not so good for forecasting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My blog post on &lt;a href=&quot;http://robjhyndman.com/researchtips/econometrics-and-r/&quot; rel=&quot;nofollow&quot;&gt;&quot;Econometrics and R&quot;&lt;/a&gt; provides a few other references that are useful.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-09-30T06:55:51.197" CommunityOwnedDate="2011-04-01T04:40:52.643" />
  <row Id="3185" PostTypeId="2" ParentId="3181" CreationDate="2010-09-30T07:02:54.690" Score="5" ViewCount="" Body="&lt;p&gt;I've found the UseR! series book &lt;a href=&quot;http://www.springer.com/statistics/statistical+theory+and+methods/book/978-0-387-88697-8&quot; rel=&quot;nofollow&quot;&gt;Introductory Time Series with R&lt;/a&gt; by Cowpertwait an Metcalfe very useful in translating my time series statistics textbooks into R-speak.&lt;/p&gt;&#xA;" OwnerUserId="1390" LastActivityDate="2010-09-30T07:02:54.690" CommunityOwnedDate="2011-04-01T04:40:52.643" />
  <row Id="3186" PostTypeId="2" ParentId="3175" CreationDate="2010-09-30T07:24:51.460" Score="1" ViewCount="" Body="&lt;p&gt;I would suggest using the &lt;a href=&quot;http://www.mathworks.com/help/toolbox/stats/ksdensity.html&quot; rel=&quot;nofollow&quot;&gt;ksdensity&lt;/a&gt; function. In the following example I compare the pdf of the data in column 1 of matrix 1 with the pdf of the data in column 2 of matrix 2:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[f,x] = ksdensity(mat1(:,1));&#xA;plot(x,f,'--b');hold&#xA;[f,x] = ksdensity(mat2(:,2));&#xA;plot(x,f,'--m')&#xA;legend('Data 1 pdf', 'Data 2 pdf');&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="339" LastEditorUserId="339" LastEditorDisplayName="" LastEditDate="2010-09-30T07:32:21.337" LastActivityDate="2010-09-30T07:32:21.337" />
  <row Id="3187" PostTypeId="2" ParentId="3181" CreationDate="2010-09-30T07:36:56.130" Score="4" ViewCount="" Body="&lt;p&gt;For ecologists, &lt;a href=&quot;http://www.worldagroforestry.org/units/library/books/PDFs/Kindt%20b2005.pdf&quot; rel=&quot;nofollow&quot;&gt;Tree diversity analysis&lt;/a&gt; can be a first healthy step into the right direction. The book is free, it comes with an R package (&lt;a href=&quot;http://cran.r-project.org/web/packages/BiodiversityR/index.html&quot; rel=&quot;nofollow&quot;&gt;BiodiversityR&lt;/a&gt;) and gives you a taste of other eco-packages (like vegan).&lt;/p&gt;&#xA;" OwnerUserId="144" LastEditorUserId="1381" LastEditorDisplayName="" LastEditDate="2011-04-01T04:38:17.623" LastActivityDate="2011-04-01T04:38:17.623" CommunityOwnedDate="2011-04-01T04:40:52.643" CommentCount="1" />
  <row Id="3188" PostTypeId="2" ParentId="3179" CreationDate="2010-09-30T08:08:11.903" Score="3" ViewCount="" Body="&lt;p&gt;Hi there,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I feel you are actually asking two questions:  1) what types of visualizations to use and 2) what R package can produce them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the case of what type of graph to use, there are many, and it depends on your needs (e.g: types of variables - numeric, factor, geographic etc, and the type of connections you are interested to display):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;If you have many numeric variables, you might want to use a scatter plot matrix (have a look &lt;a href=&quot;http://www.statmethods.net/graphs/scatterplot.html&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;li&gt;If you have many factor variables, you might want to use a scatter plot matrix for factors (have a look &lt;a href=&quot;http://www.r-statistics.com/2010/04/correlation-scatter-plot-matrix-for-ordered-categorical-data/&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;li&gt;You could also go with doing some &lt;a href=&quot;http://en.wikipedia.org/wiki/Parallel_coordinates&quot; rel=&quot;nofollow&quot;&gt;Parallel coordinates&lt;/a&gt; there &lt;a href=&quot;http://stat.ethz.ch/R-manual/R-devel/library/MASS/html/parcoord.html&quot; rel=&quot;nofollow&quot;&gt;are&lt;/a&gt; &lt;a href=&quot;http://rss.acs.unt.edu/Rdoc/library/rflowcyt/html/parallel.coordinates.html&quot; rel=&quot;nofollow&quot;&gt;several&lt;/a&gt; &lt;a href=&quot;http://www.r-statistics.com/tag/parallel-coordinates/&quot; rel=&quot;nofollow&quot;&gt;ways&lt;/a&gt; to do it in R.&lt;/li&gt;&#xA;&lt;li&gt;For a wider range of graphical facilities in R, have a look at the &lt;a href=&quot;http://cran.r-project.org/web/views/Graphics.html&quot; rel=&quot;nofollow&quot;&gt;graphics task view&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Now regarding how to do it.  One problem with many data points is time till the plot is created.  ggplot2, iplots, ggobi are not very good for too many data points (at least from my experience).  In which case you might want to focus on R base graphics facilities, or sample your data and on that to use all the other tools.  Or you can hope that the people developing iplots extreme (or &lt;a href=&quot;http://www.rforge.net/Acinonyx/&quot; rel=&quot;nofollow&quot;&gt;Acinonyx&lt;/a&gt;) would get to an advance release stage.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good luck,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tal&lt;/p&gt;&#xA;" OwnerUserId="253" LastActivityDate="2010-09-30T08:08:11.903" CommunityOwnedDate="2010-09-30T14:04:56.293" CommentCount="4" />
  <row Id="3189" PostTypeId="5" CreationDate="2010-09-30T08:09:08.867" Score="0" ViewCount="0" Body="" OwnerUserId="-1" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-09-30T08:09:08.867" LastActivityDate="2010-09-30T08:09:08.867" />
  <row Id="3190" PostTypeId="4" CreationDate="2010-09-30T08:09:08.867" Score="0" ViewCount="0" Body="Forecasting involves estimating the value or distribution of a random variable which has not yet been observed. Usually, forecasting is applied to time series data where future values of a series are estimated based on past observations." OwnerUserId="-1" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-09-30T08:09:08.867" LastActivityDate="2010-09-30T08:09:08.867" />
  <row Id="3191" PostTypeId="2" ParentId="2910" CreationDate="2010-09-30T10:44:48.180" Score="17" ViewCount="" Body="&lt;p&gt;I am compiling a quick series of guidelines I found on &lt;a href=&quot;http://www.stackoverflow.com&quot; rel=&quot;nofollow&quot;&gt;SO&lt;/a&gt; (as suggested by @Shane), &lt;a href=&quot;http://biostar.stackexchange.com/&quot; rel=&quot;nofollow&quot;&gt;Biostar&lt;/a&gt; (hereafter, BS), and this SE. I tried my best to acknowledge ownership for each item, and to select first or highly upvoted answer. I also added things of my own, and flagged items that are specific to the [R] environment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Data management&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;create a project structure for keeping all things at the right place (data, code, figures, etc., &lt;a href=&quot;http://biostar.stackexchange.com/questions/822/how-do-you-manage-your-files-directories-for-your-projects/826#826&quot; rel=&quot;nofollow&quot;&gt;giovanni&lt;/a&gt; /BS)&lt;/li&gt;&#xA;&lt;li&gt;never modify raw data files (ideally, they should be read-only), copy/rename to new ones when making transformations, cleaning, etc. &lt;/li&gt;&#xA;&lt;li&gt;check data consistency (&lt;a href=&quot;http://stats.stackexchange.com/questions/2768/what-is-a-consistency-check/2785#2785&quot;&gt;whuber&lt;/a&gt; /SE)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Coding&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;organize source code in logical units or building blocks (&lt;a href=&quot;http://stackoverflow.com/questions/1429907/workflow-for-statistical-analysis-and-report-writing/1434424#1434424&quot;&gt;Josh Reich&lt;/a&gt;/&lt;a href=&quot;http://stackoverflow.com/questions/1429907/workflow-for-statistical-analysis-and-report-writing/1430569#1430569&quot;&gt;hadley&lt;/a&gt;/&lt;a href=&quot;http://stackoverflow.com/questions/1266279/how-to-organize-large-r-programs/1269808#1269808&quot;&gt;ars&lt;/a&gt; /SO; &lt;a href=&quot;http://biostar.stackexchange.com/questions/822/how-do-you-manage-your-files-directories-for-your-projects/826#826&quot; rel=&quot;nofollow&quot;&gt;giovanni&lt;/a&gt;/&lt;a href=&quot;http://biostar.stackexchange.com/questions/822/how-do-you-manage-your-files-directories-for-your-projects/829#829&quot; rel=&quot;nofollow&quot;&gt;Khader Shameer&lt;/a&gt; /BS)&lt;/li&gt;&#xA;&lt;li&gt;separate source code from editing stuff, especially for large project -- partly overlapping with previous item and reporting&lt;/li&gt;&#xA;&lt;li&gt;document everything, with e.g. [R]oxygen (&lt;a href=&quot;http://stackoverflow.com/questions/2284446/organizing-r-source-code/2284486#2284486&quot;&gt;Shane&lt;/a&gt; /SO) or consistent self-annotation in the source file -- a good discussion on Medstats, &lt;a href=&quot;http://groups.google.com/group/medstats/browse_thread/thread/601793e6ce36e789&quot; rel=&quot;nofollow&quot;&gt;Documenting analyses and data edits Options&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;[R] custom functions can be put in a dedicated file (that can be sourced when necessary), in a new environment (so as to avoid populating the top-level namespace, &lt;a href=&quot;http://stackoverflow.com/questions/1266279/how-to-organize-large-r-programs/1319786#1319786&quot;&gt;Brendan OConnor&lt;/a&gt; /SO), or a package (&lt;a href=&quot;http://stackoverflow.com/questions/1266279/how-to-organize-large-r-programs/1266400#1266400&quot;&gt;Dirk Eddelbuettel&lt;/a&gt;/&lt;a href=&quot;http://stackoverflow.com/questions/2284446/organizing-r-source-code/2284486#2284486&quot;&gt;Shane&lt;/a&gt; /SO)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Analysis&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;don't forget to set/record the seed you used when calling RNG or stochastic algorithms (e.g. k-means)&lt;/li&gt;&#xA;&lt;li&gt;for Monte Carlo studies, it may be interesting to store specs/parameters in a separate file (&lt;a href=&quot;http://neuralensemble.org/trac/sumatra&quot; rel=&quot;nofollow&quot;&gt;sumatra&lt;/a&gt; may be a good candidate, &lt;a href=&quot;http://biostar.stackexchange.com/questions/822/how-do-you-manage-your-files-directories-for-your-projects/826#826&quot; rel=&quot;nofollow&quot;&gt;giovanni&lt;/a&gt; /BS)&lt;/li&gt;&#xA;&lt;li&gt;don't limit yourself to one plot per variable, use multivariate (Trellis) displays and interactive visualization tools (e.g. GGobi)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Versioning&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;use some kind of CVS for easy tracking/export, e.g. Git (&lt;a href=&quot;http://stackoverflow.com/questions/2712421/r-and-version-control-for-the-solo-data-analyst/2715569#2715569&quot;&gt;Sharpie&lt;/a&gt;/&lt;a href=&quot;http://stackoverflow.com/questions/2545765/how-can-i-email-someone-a-git-repository/2545784#2545784&quot;&gt;VonC&lt;/a&gt;/&lt;a href=&quot;http://stackoverflow.com/questions/2286831/how-do-you-combine-revision-control-with-workflow-for-r/2290194#2290194&quot;&gt;JD Long&lt;/a&gt; /SO) -- this follows from nice questions asked by @Jeromy and @Tal&lt;/li&gt;&#xA;&lt;li&gt;backup everything, on a regular basis (&lt;a href=&quot;http://stackoverflow.com/questions/2712421/r-and-version-control-for-the-solo-data-analyst/2715569#2715569&quot;&gt;Sharpie&lt;/a&gt;/&lt;a href=&quot;http://stackoverflow.com/questions/2286831/how-do-you-combine-revision-control-with-workflow-for-r/2290194#2290194&quot;&gt;JD Long&lt;/a&gt; /SO)&lt;/li&gt;&#xA;&lt;li&gt;keep a log of your ideas, or rely on an issue tracker, like &lt;a href=&quot;http://ditz.rubyforge.org/ditz/&quot; rel=&quot;nofollow&quot;&gt;ditz&lt;/a&gt; (&lt;a href=&quot;http://biostar.stackexchange.com/questions/822/how-do-you-manage-your-files-directories-for-your-projects/826#826&quot; rel=&quot;nofollow&quot;&gt;giovanni&lt;/a&gt; /BS) -- partly redundant with the previous item since it is available in Git&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Editing/Reporting&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;[R] Sweave (&lt;a href=&quot;http://stackoverflow.com/questions/1429907/workflow-for-statistical-analysis-and-report-writing/1430013#1430013&quot;&gt;Matt Parker&lt;/a&gt; /SO)&lt;/li&gt;&#xA;&lt;li&gt;[R] brew (&lt;a href=&quot;http://stackoverflow.com/questions/1429907/workflow-for-statistical-analysis-and-report-writing/1436809#1436809&quot;&gt;Shane&lt;/a&gt; /SO)&lt;/li&gt;&#xA;&lt;li&gt;[R] [R2HTML]&lt;a href=&quot;http://cran.r-project.org/web/packages/R2HTML/index.html&quot; rel=&quot;nofollow&quot;&gt;26&lt;/a&gt; or &lt;a href=&quot;http://cran.r-project.org/web/packages/ascii/index.html&quot; rel=&quot;nofollow&quot;&gt;ascii&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;As a side note, Hadley Wickham offers a comprehensive overview of &lt;a href=&quot;http://github.com/hadley/devtools/wiki&quot; rel=&quot;nofollow&quot;&gt;R project management&lt;/a&gt;, including &lt;em&gt;reproducible exemplification&lt;/em&gt; and an &lt;em&gt;unified philosophy of data&lt;/em&gt;.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-07T11:40:47.677" LastActivityDate="2010-10-07T11:40:47.677" CommunityOwnedDate="2010-09-30T10:44:48.180" CommentCount="5" />
  <row Id="3192" PostTypeId="2" ParentId="3156" CreationDate="2010-09-30T10:58:23.187" Score="1" ViewCount="" Body="&lt;p&gt;Reporting a CI around a mean is not reporting the distribution of values, only an estimate of how well you captured that mean value.  It will always get smaller as n goes up.  It's NOT what you want because you want to see how well a point fits into a distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With your fairly large N's you might be able to do a nice plot of the distributions, all together on one plot overlayed in different colours and then place your point estimate.  You may be able be able to see from what group it seems most likely to come from.  Then estimate the z-score to the point value from each distribution.  Whichever is the lowest is the one I'd be most inclined to pick but this would depend a great deal on how these looked in terms of distribution.  For example, I'm guessing dist.3 is going to be the flattest and so it's going to have a very low z-score there.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Plot it out and see what it looks like.  (after dividing everything by 1e8 :) )&lt;/p&gt;&#xA;" OwnerUserId="601" LastActivityDate="2010-09-30T10:58:23.187" />
  <row Id="3193" PostTypeId="1" AcceptedAnswerId="3205" CreationDate="2010-09-30T11:16:25.453" Score="7" ViewCount="101" Body="&lt;p&gt;Hello data analyst community. I have the following problem:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given a set of n units and a timeline in days. A unit may be active at a certain day to a certain degree (in range from 0.0 to 1.0). A desirable outcome is that if a unit is active, it should be active for a series of consecutive days (or at maximum with one day break).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I have, of course is the opposite :). Now I want to measure or even better visualize the activity frequencies to &quot;prove&quot; an image-affine person that not all units behave as desired. The brute-force-approach is to draw a line for each unit (along the timeline), colored according to the degree of activity, but since n &gt; 30, the graph is big, colorful and you see nothing at all. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am afraid that I am searching in the wrong direction. Any ideas, suggestions ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT:&#xA;I think I was not able to explain my goal: I do not want to visualize the activity of a singular unit, but getting an idea of the activity frequency of all units involved. In the far end, I will have two groups of units and want to see graphically whether one group performed better than the other (better according to property described above). My apologies for not stating this earlier (thanks to the contributions up to this point, I was able to see what I actually want to know). &lt;/p&gt;&#xA;" OwnerUserId="264" LastEditorUserId="264" LastEditorDisplayName="" LastEditDate="2010-10-01T11:59:49.043" LastActivityDate="2010-10-01T11:59:49.043" Title="Visualizing activity frequency" Tags="&lt;time-series&gt;&lt;data-visualization&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="3194" PostTypeId="1" AcceptedAnswerId="3198" CreationDate="2010-09-30T13:04:31.537" Score="12" ViewCount="463" Body="&lt;p&gt;How can I test the fairness of a twenty sided die (d20)?  Obviously I would be comparing the distribution of values against a uniform distribution.  I vaguely remember using a Chi-square test in college.  How can I apply this to see if a die is fair?&lt;/p&gt;&#xA;" OwnerUserId="1456" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-03T17:48:29.270" LastActivityDate="2011-03-03T17:48:29.270" Title="How can I test the fairness of a d20?" Tags="&lt;test&gt;&lt;goodness-of-fit&gt;&lt;uniform&gt;&lt;dice&gt;" AnswerCount="6" FavoriteCount="3" />
  <row Id="3195" PostTypeId="2" ParentId="3171" CreationDate="2010-09-30T13:14:28.583" Score="1" ViewCount="" Body="&lt;p&gt;You would normally make the assumption of independence of observations in your modelling.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively if you expected corrlation between observations it would be good to model this and estimate that correlation. You can't do this as you don't know which observations are likely to be correlated.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you assume independence when some observations are in fact positively correlated you will understimate the between subject variance. This means you are more likely to find &quot;significant differences&quot; than statistical theory would suggest. You can think of it as appearing to have more samples than you in fact do have as some are almost repeats.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ps. Not having this as Community Wiki would encourage more answers!&lt;/p&gt;&#xA;" OwnerUserId="521" LastEditorUserId="521" LastEditorDisplayName="" LastEditDate="2010-10-01T00:19:40.320" LastActivityDate="2010-10-01T00:19:40.320" CommunityOwnedDate="2010-09-30T13:14:28.583" CommentCount="1" />
  <row Id="3196" PostTypeId="2" ParentId="3194" CreationDate="2010-09-30T13:34:56.983" Score="6" ViewCount="" Body="&lt;p&gt;Do you want to do it by hand, or in excel ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to do it in &lt;a href=&quot;http://www.r-project.org/&quot; rel=&quot;nofollow&quot;&gt;R&lt;/a&gt;, you can do it this way:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Step 1: roll your die (let's say) 100 times.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Step 2: count how many times you got each of your numbers&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Step 3: put them in R like this (write the number of times each die roll you got, instead of the numbers I wrote):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x &amp;lt;- as.table(c(1,2,3,4,5,6,7,80,9,10,11,12,13,14,15,16,17,18,19,20))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Step 4: simply run this command:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;chisq.test(x)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If the P value is low (e.g: bellow 0.05) - your die is not balanced.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This command simulates a balanced die (P= ~.5):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;chisq.test(table(sample(1:20, 100, T)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And this simulates an unbalanced die:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;chisq.test(table(c(rep(20,10),sample(1:20, 100, T))))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(It get's to be about P = ~.005)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now the real question is how many die's should be rolled to what level of power of detection.  If someone wants to go into solving that, he is welcomed...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Update:  There is also a nice article on this topic &lt;a href=&quot;http://www.textfiles.com/rpg/fairdice.txt&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="253" LastEditorUserId="253" LastEditorDisplayName="" LastEditDate="2010-09-30T14:28:19.940" LastActivityDate="2010-09-30T14:28:19.940" CommentCount="2" />
  <row Id="3197" PostTypeId="2" ParentId="3194" CreationDate="2010-09-30T13:40:00.953" Score="5" ViewCount="" Body="&lt;p&gt;If you are interested in just checking the number of times each number appears, then a Chi-squared test would be suitable. Suppose you roll a die N times. You would expect each value to come up N/20 times. All a chi-square test does is compare what you observed with what you get. If this difference is too large, then this would indicate a problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Other tests&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you were interested in other aspects of randonness, for example, if you dice gave the following output:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1, 2, 3, 4...., 20,1,2,..&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then although this output has the correct number of each individual value, it is clearly not random. In this case, take a look at this &lt;a href=&quot;http://stats.stackexchange.com/questions/30/testing-random-variate-generation-algorithms/&quot;&gt;question&lt;/a&gt;. This probably only makes sense for electronic dice.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Chi-squared test in R&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In R, this would be&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;##Roll 200 times&#xA;&amp;gt; rolls = sample(1:20, 200, replace=TRUE)&#xA;&amp;gt; chisq.test(table(rolls), p = rep(0.05, 20))&#xA;    Chi-squared test for given probabilities&#xA;data:  table(rolls) &#xA;X-squared = 16.2, df = 19, p-value = 0.6439&#xA;&#xA;## Too many 1's in the sample&#xA;&amp;gt; badrolls = cbind(rolls, rep(1, 10))   &#xA;&amp;gt; chisq.test(table(badrolls), p = rep(0.05, 20))&#xA;&#xA;    Chi-squared test for given probabilities&#xA;&#xA;data:  table(badrolls) &#xA;X-squared = 1848.1, df = 19, p-value &amp;lt; 2.2e-16&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="8" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-09-30T14:05:29.147" LastActivityDate="2010-09-30T14:05:29.147" />
  <row Id="3198" PostTypeId="2" ParentId="3194" CreationDate="2010-09-30T13:44:01.660" Score="8" ViewCount="" Body="&lt;p&gt;Here's an example with R code. The output is preceded by #'s.&#xA;A fair die:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;rolls &amp;lt;- sample(1:20, 200, replace = T)&#xA;table(rolls)&#xA;#rolls&#xA;# 1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 &#xA;# 7  8 11  9 12 14  9 14 11  7 11 10 13  8  8  5 13  9 10 11 &#xA; chisq.test(table(rolls), p = rep(0.05, 20))&#xA;&#xA;#         Chi-squared test for given probabilities&#xA;#&#xA;# data:  table(rolls) &#xA;# X-squared = 11.6, df = 19, p-value = 0.902&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;A biased die - numbers 1 to 10 each have a probability of 0.045; those 11-20 have a probability of 0.055 - 200 throws:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;rolls &amp;lt;- sample(1:20, 200, replace = T, prob=cbind(rep(0.045,10), rep(0.055,10)))&#xA;table(rolls)&#xA;#rolls&#xA;# 1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 &#xA;# 8  9  7 12  9  7 14  5 10 12 11 13 14 16  6 10 10  7  9 11 &#xA;chisq.test(table(rolls), p = rep(0.05, 20))&#xA;&#xA;#        Chi-squared test for given probabilities&#xA;#&#xA;# data:  table(rolls) &#xA;# X-squared = 16.2, df = 19, p-value = 0.6439&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;We have insufficient evidence of bias (p = 0.64).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A biased die, 1000 throws:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;rolls &amp;lt;- sample(1:20, 1000, replace = T, prob=cbind(rep(0.045,10), rep(0.055,10)))&#xA;table(rolls)&#xA;#rolls&#xA;# 1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 &#xA;# 42 47 34 42 47 45 48 43 42 45 52 50 57 57 60 68 49 67 42 63 &#xA;chisq.test(table(rolls), p = rep(0.05, 20))&#xA;&#xA;#        Chi-squared test for given probabilities&#xA;#&#xA;# data:  table(rolls) &#xA;# X-squared = 32.36, df = 19, p-value = 0.02846&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now p&amp;lt;0.05 and we are starting to see evidence of bias. You can use similar simulations to estimate the level of bias you can expect to detect and the number of throws needed to detect it with an given p-level.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Wow, 2 other answers even before I finished typing.&lt;/p&gt;&#xA;" OwnerUserId="521" LastEditorUserId="521" LastEditorDisplayName="" LastEditDate="2010-09-30T13:51:53.710" LastActivityDate="2010-09-30T13:51:53.710" CommentCount="3" />
  <row Id="3199" PostTypeId="1" AcceptedAnswerId="3210" CreationDate="2010-09-30T13:58:08.637" Score="4" ViewCount="230" Body="&lt;p&gt;I have a 2 dim matrix, and I want to know e.g. all the higher values are in the upper left corner. I can't just project it into R^3 and use a standard clustering algorithm because I don't want to consider the value as a dimension by itself. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there an algorithm I can use for this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To reformulate it, suppose it was like&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;| High values ... low values |&#xA;| ...&#xA;| Low values ...  ...        |&#xA;| ...&#xA;| High values .. low values  |&#xA;&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'd want to know that there's a &quot;cluster&quot; of high values in the upper left and lower left.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT 2:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The matrix represents an image. The values of each cell represent the concentration of a substance at that coordinate. I want to know how homogeneous the image is (i.e. how well &quot;mixed together&quot; the substance is). Additionally, I would like to know where the non-homogeneity (if any) is coming from.&lt;/p&gt;&#xA;" OwnerUserId="900" LastEditorUserId="900" LastEditorDisplayName="" LastEditDate="2010-10-06T18:13:12.177" LastActivityDate="2010-10-11T15:46:43.540" Title="Clustering of a matrix (homogeneity measurement)" Tags="&lt;clustering&gt;" AnswerCount="4" CommentCount="3" FavoriteCount="2" />
  <row Id="3200" PostTypeId="1" AcceptedAnswerId="3202" CreationDate="2010-09-30T14:07:56.490" Score="12" ViewCount="1099" Body="&lt;p&gt;Lets assume you are a social science researcher/econometrician trying to find relevant predictors of demand for a service. You have 2 outcome/depedent variables describing the demand (using the service yes/no, and the number of occasions). You have 10 predictor/ independent variables that could theoretically explain the demand (e g age, sex, income, price, race etc). Running two separate  multiple regressions will yield 20 coefficients estimations and their p-values. With enough independent variables in your regressions you would sooner or later find at least one variable with a statistically significant correlation between the dependent and independent variables. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My question: is it a good idea to correct the p-values for multiple tests if I want to include all independent variables in the regression? Any references to prior work are much appreciated.&lt;/strong&gt; &lt;/p&gt;&#xA;" OwnerUserId="1458" LastEditorUserId="1458" LastEditorDisplayName="" LastEditDate="2010-09-30T19:01:18.677" LastActivityDate="2010-10-07T17:42:22.733" Title="Is adjusting p-values in a multiple regression for multiple comparisons a good idea?" Tags="&lt;regression&gt;&lt;multiple-comparisons&gt;" AnswerCount="5" FavoriteCount="4" />
  <row Id="3201" PostTypeId="1" AcceptedAnswerId="3218" CreationDate="2010-09-30T14:14:06.623" Score="4" ViewCount="134" Body="&lt;p&gt;I have a database containing a large number of experts in a field. For each of those experts i have a variety of attributes/data points like:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;number of years of experience.&lt;/li&gt;&#xA;&lt;li&gt;licenses&lt;/li&gt;&#xA;&lt;li&gt;num of reviews&lt;/li&gt;&#xA;&lt;li&gt;textual content of those reviews&lt;/li&gt;&#xA;&lt;li&gt;The 5 star rating on each of those reviews, for a number of factors like speed, quality etc.&lt;/li&gt;&#xA;&lt;li&gt;awards, assosciations, conferences etc.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I want to provide a rating to these experts say out of 10 based on their importance. Some of the data points might be missing for some of the experts. Now my question is how do i come up with such an algorithm? Can anyone point me to some relevent literature? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also i am concerned that as with all rating/reviews the numbers might bunch up near some some values. For example most of them might end up getting an 8 or a 5. Is there a way to highlight litle differences into a larger difference in the score for only some of the attributes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some other discussions that i figured might be relevant:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://stats.stackexchange.com/questions/1848/bayesian-rating-system-with-multiple-categories-for-each-rating&quot;&gt;http://stats.stackexchange.com/questions/1848/bayesian-rating-system-with-multiple-categories-for-each-rating&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://stats.stackexchange.com/questions/2689/how-would-you-compute-imdb-movie-rating&quot;&gt;http://stats.stackexchange.com/questions/2689/how-would-you-compute-imdb-movie-rating&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://stats.stackexchange.com/questions/1/eliciting-priors-from-experts&quot;&gt;http://stats.stackexchange.com/questions/1/eliciting-priors-from-experts&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://stats.stackexchange.com/questions/2563/what-are-some-of-the-best-ranking-algorithms-with-inputs-as-up-and-down-votes&quot;&gt;http://stats.stackexchange.com/questions/2563/what-are-some-of-the-best-ranking-algorithms-with-inputs-as-up-and-down-votes&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="1459" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-30T21:26:54.663" LastActivityDate="2010-09-30T21:26:54.663" Title="How do I order or rank a set of experts?" Tags="&lt;rating&gt;" AnswerCount="3" CommentCount="1" FavoriteCount="3" />
  <row Id="3202" PostTypeId="2" ParentId="3200" CreationDate="2010-09-30T14:33:11.650" Score="9" ViewCount="" Body="&lt;p&gt;It seems your question more generally addresses the problem of identifying good predictors. In this case, you should consider using some kind of penalized regression (methods dealing with variable or &lt;a href=&quot;http://www.google.fr/url?sa=t&amp;amp;source=web&amp;amp;cd=1&amp;amp;ved=0CBgQFjAA&amp;amp;url=http%3A%2F%2Fen.wikipedia.org%2Fwiki%2FFeature_selection&amp;amp;ei=w5qkTLHOA8iK4gbr0MH9DA&amp;amp;usg=AFQjCNFGSoQruvgm6fTvAtx8NUU3MDBBcw&amp;amp;sig2=4wcurlN8irzgYgr3chlopA&quot; rel=&quot;nofollow&quot;&gt;feature selection&lt;/a&gt; are relevant too), with e.g. L1, L2 (or a combination thereof, the so-called &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.124.4696&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;elasticnet&lt;/a&gt;) penalties (look for related questions on this site, or the R &lt;a href=&quot;http://cran.r-project.org/web/packages/penalized/index.html&quot; rel=&quot;nofollow&quot;&gt;penalized&lt;/a&gt; and &lt;a href=&quot;http://cran.r-project.org/package=elasticnet&quot; rel=&quot;nofollow&quot;&gt;elasticnet&lt;/a&gt; package, among others). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, about correcting p-values for your regression coefficients (or equivalently your partial correlation coefficients) to protect against over-optimism (e.g. with Bonferroni or, better, step-down methods), it seems this would only be relevant if you are considering one model and seek those predictors that contribute a significant part of explained variance, that is if you don't perform model selection (with stepwise selection, or hierarchical testing). This article may be a good start: &lt;a href=&quot;http://mlrv.ua.edu/2006/Mundfrom-etal-MLRV-3.pdf&quot; rel=&quot;nofollow&quot;&gt;Bonferroni Adjustments in Tests for Regression Coefficients&lt;/a&gt;. Be aware that such correction won't protect you against multicolinearity issue, which affects the reported p-values. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given your data, I would recommend using some kind of iterative model selection techniques. In R for instance, the &lt;code&gt;stepAIC&lt;/code&gt; function allows to perform stepwise model selection by exact AIC. You can also estimate the relative importance of your predictors based on their contribution to $R^2$ using boostrap (see the &lt;a href=&quot;http://cran.r-project.org/web/packages/relaimpo/index.html&quot; rel=&quot;nofollow&quot;&gt;relaimpo&lt;/a&gt; package). I think that reporting effect size measure or % of explained variance are more informative than p-value, especially in a confirmatory model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It should be noted that stepwise approaches have also their drawbacks (e.g., Wald tests are not adapted to conditional hypothesis as induced by the stepwise procedure), or as indicated by Frank Harrell on &lt;a href=&quot;http://www.mail-archive.com/r-help@r-project.org/msg105680.html&quot; rel=&quot;nofollow&quot;&gt;R mailing&lt;/a&gt;, &quot;stepwise variable selection based on AIC has all the problems of stepwise variable selection based on P-values. AIC is just a restatement of the P-Value&quot; (but AIC remains useful if the set of predictors is already defined); a related question -- &lt;a href=&quot;http://stats.stackexchange.com/questions/1016/is-a-variable-significant-in-a-linear-regression-model&quot;&gt;Is a variable significant in a linear regression model?&lt;/a&gt; -- raised interesting comments (&lt;a href=&quot;http://stats.stackexchange.com/users/159/rob-hyndman&quot;&gt;@Rob&lt;/a&gt;, among others) about the use of AIC for variable selection. I append a couple of references at the end (including papers kindly provided by &lt;a href=&quot;http://stats.stackexchange.com/users/1352/stephan-kolassa&quot;&gt;@Stephan&lt;/a&gt;); there is also a lot of other references on &lt;a href=&quot;http://www.pmean.com/category/MultipleComparisons.html&quot; rel=&quot;nofollow&quot;&gt;P.Mean&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Frank Harrell authored a book on &lt;a href=&quot;http://biostat.mc.vanderbilt.edu/twiki/bin/view/Main/RmS&quot; rel=&quot;nofollow&quot;&gt;Regression Modeling Strategy&lt;/a&gt; which includes a lot of discussion and advices around this problem (§4.3, pp. 56-60). He also developed efficient R routines to deal with generalized linear models (See the &lt;a href=&quot;http://cran.r-project.org/web/packages/Design/index.html&quot; rel=&quot;nofollow&quot;&gt;Design&lt;/a&gt; or &lt;a href=&quot;http://cran.r-project.org/web/packages/rms/index.html&quot; rel=&quot;nofollow&quot;&gt;rms&lt;/a&gt; packages). So, I think you definitely have to take a look at it (his &lt;a href=&quot;http://biostat.mc.vanderbilt.edu/wiki/pub/Main/RmS/course2.pdf&quot; rel=&quot;nofollow&quot;&gt;handouts&lt;/a&gt; are available on his homepage).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;References&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Whittingham, MJ, Stephens, P, Bradbury, RB,  and Freckleton, RP (2006). &lt;a href=&quot;http://www.ncl.ac.uk/biology/assets/MWhitt_pdf/06j_animal.pdf&quot; rel=&quot;nofollow&quot;&gt;Why do we still use stepwise modelling in ecology and behaviour?&lt;/a&gt; &lt;em&gt;Journal of Animal Ecology&lt;/em&gt;, &lt;em&gt;75&lt;/em&gt;, 1182-1189.&lt;/li&gt;&#xA;&lt;li&gt;Austin, PC (2008). &lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pubmed/18539429?ordinalpos=1&quot; rel=&quot;nofollow&quot;&gt;Bootstrap model selection had similar performance for selecting authentic and noise variables compared to backward variable elimination: a simulation study&lt;/a&gt;. &lt;em&gt;Journal of Clinical Epidemiology&lt;/em&gt;, &lt;em&gt;61(10)&lt;/em&gt;, 1009-1017.&lt;/li&gt;&#xA;&lt;li&gt;Austin, PC and Tu, JV (2004). &lt;a href=&quot;http://uncwddas.googlecode.com/files/article2.pdf&quot; rel=&quot;nofollow&quot;&gt;Automated variable selection methods for logistic regression produced unstable models for predicting acute myocardial infarction mortality&lt;/a&gt;. &lt;em&gt;Journal of Clinical Epidemiology&lt;/em&gt;, &lt;em&gt;57&lt;/em&gt;, 1138–1146.&lt;/li&gt;&#xA;&lt;li&gt;Greenland, S (1994). &lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1566551/&quot; rel=&quot;nofollow&quot;&gt;Hierarchical regression for epidemiologic analyses of multiple exposures&lt;/a&gt;. &lt;em&gt;Environmental Health Perspectives&lt;/em&gt;, &lt;em&gt;102(Suppl 8)&lt;/em&gt;, 33–39.&lt;/li&gt;&#xA;&lt;li&gt;Greenland, S (2008). &lt;a href=&quot;http://ije.oxfordjournals.org/content/37/3/430.full&quot; rel=&quot;nofollow&quot;&gt;Multiple comparisons and association selection in general epidemiology&lt;/a&gt;. &lt;em&gt;International Journal of Epidemiology&lt;/em&gt;, &lt;em&gt;37(3)&lt;/em&gt;, 430-434.&lt;/li&gt;&#xA;&lt;li&gt;Beyene, J, Atenafu, EG, Hamid, JS, To, T, and Sung L (2009). &lt;a href=&quot;http://www.biomedcentral.com/1471-2288/9/64/&quot; rel=&quot;nofollow&quot;&gt;Determining relative importance of variables in developing and validating predictive models&lt;/a&gt;. &lt;em&gt;BMC Medical Research Methodology&lt;/em&gt;, &lt;em&gt;9&lt;/em&gt;, 64.&lt;/li&gt;&#xA;&lt;li&gt;Bursac, Z, Gauss, CH, Williams, DK, and Hosmer, DW (2008). &lt;a href=&quot;http://www.scfbm.org/content/3/1/17&quot; rel=&quot;nofollow&quot;&gt;Purposeful selection of variables in logistic regression&lt;/a&gt;. &lt;em&gt;Source Code for Biology and Medicine&lt;/em&gt;, &lt;em&gt;3&lt;/em&gt;, 17.&lt;/li&gt;&#xA;&lt;li&gt;Brombin, C, Finos, L, and Salmaso, L (2007). &lt;a href=&quot;http://www.mcp-conference.org/2007/presentations/Brombin_Chiara.pdf&quot; rel=&quot;nofollow&quot;&gt;Adjusting stepwise p-values in generalized linear models&lt;/a&gt;. &lt;em&gt;International Conference on Multiple Comparison Procedures&lt;/em&gt;. -- see &lt;code&gt;step.adj()&lt;/code&gt; in the R &lt;a href=&quot;http://cran.r-project.org/web/packages/someMTP/index.html&quot; rel=&quot;nofollow&quot;&gt;someMTP&lt;/a&gt; package.&lt;/li&gt;&#xA;&lt;li&gt;Wiegand, RE (2010). &lt;a href=&quot;http://onlinelibrary.wiley.com/doi/10.1002/sim.3943/abstract&quot; rel=&quot;nofollow&quot;&gt;Performance of using multiple stepwise algorithms for variable selection&lt;/a&gt;. &lt;em&gt;Statistics in Medicine&lt;/em&gt;, 29(15), 1647–1659.&lt;/li&gt;&#xA;&lt;li&gt;Moons KG, Donders AR, Steyerberg EW, and Harrell FE (2004). Penalized Maximum Likelihood Estimation to predict binary outcomes. &lt;em&gt;Journal of Clinical Epidemiology&lt;/em&gt;, &lt;em&gt;57(12)&lt;/em&gt;, 1262–1270.&lt;/li&gt;&#xA;&lt;li&gt;Tibshirani, R (1996). &lt;a href=&quot;http://www-stat.stanford.edu/~tibs/lasso/lasso.pdf&quot; rel=&quot;nofollow&quot;&gt;Regression shrinkage and selection via the lasso&lt;/a&gt;. &lt;em&gt;Journal of The Royal Statistical Society B&lt;/em&gt;, &lt;em&gt;58(1)&lt;/em&gt;, 267–288.&lt;/li&gt;&#xA;&lt;li&gt;Efron, B, Hastie, T, Johnstone, I, and Tibshirani, R (2004). &lt;a href=&quot;http://www-stat.stanford.edu/~hastie/Papers/LARS/LeastAngle_2002.pdf&quot; rel=&quot;nofollow&quot;&gt;Least Angle Regression&lt;/a&gt;. &lt;em&gt;Annals of Statistics&lt;/em&gt;, &lt;em&gt;32(2)&lt;/em&gt;, 407-499.&lt;/li&gt;&#xA;&lt;li&gt;Flom, PL and Cassell, DL (2007). &lt;a href=&quot;http://www.nesug.org/Proceedings/nesug07/sa/sa07.pdf&quot; rel=&quot;nofollow&quot;&gt;Stopping Stepwise: Why stepwise and similar selection methods are bad, and what you should use&lt;/a&gt;. &lt;em&gt;NESUG 2007 Proceedings&lt;/em&gt;.&lt;/li&gt;&#xA;&lt;li&gt;Shtatland, E.S., Cain, E., and Barton, M.B. (2001). &lt;a href=&quot;http://www2.sas.com/proceedings/sugi26/p222-26.pdf&quot; rel=&quot;nofollow&quot;&gt;The perils of stepwise logistic regression and how to escape them using information criteria and the Output Delivery System&lt;/a&gt;. &lt;em&gt;SUGI 26 Proceedings&lt;/em&gt; (pp. 222–226).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-07T17:42:22.733" LastActivityDate="2010-10-07T17:42:22.733" CommentCount="8" />
  <row Id="3203" PostTypeId="2" ParentId="3199" CreationDate="2010-09-30T14:35:49.713" Score="2" ViewCount="" Body="&lt;p&gt;The goal is just to find out a measure that will tell us how mixed up all the pixels are. Given 2 matrices of data with the exact same distribution of values, if the first one's values are ordered or clumped together in spatial groups and the 2nd one's values are well-dispersed (high points and not near other high points, low points not near other lows), what is the method of evaluating this dispersion/ clumpyness?&#xA;The matrices will have the exact same variance or standard deviation, so that is not a good method.&#xA;One idea is using the 2D Fourier Transform, because a more clumpy image intuitively has a lower frequency, but I'm not sure if that is actually a common or useful practice for this type of evaluation.&lt;/p&gt;&#xA;" LastActivityDate="2010-09-30T14:35:49.713" />
  <row Id="3204" PostTypeId="2" ParentId="3193" CreationDate="2010-09-30T15:03:59.807" Score="1" ViewCount="" Body="&lt;p&gt;How about creating small timelines for each unit, one on top of the other, sorted in order of most to least active? Think &lt;a href=&quot;http://en.wikipedia.org/wiki/Sparkline&quot; rel=&quot;nofollow&quot;&gt;sparklines&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could probably do something like highlight the inactive time as either a shaded portion of the chart, or a colored portion of the unit's timeline. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since each unit would have a small plot, you'd be able to see an individual's activity at a given time. And sorting them by activity would show how poorly some units are performing, as the plots get flatter (and/or more full of your inactivity indicator) as you go down the graphic. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't have any great ideas on what software to create this with. You might be able to do it with Lattice in R. &lt;/p&gt;&#xA;" OwnerUserId="298" LastActivityDate="2010-09-30T15:03:59.807" />
  <row Id="3205" PostTypeId="2" ParentId="3193" CreationDate="2010-09-30T15:36:57.600" Score="4" ViewCount="" Body="&lt;p&gt;You might be trying to incorporate too much information into the graphic.  The essence of the visualization seems to be the frequency with which units are active more than one day and, possibly, the times at which those units are active.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just to generate ideas--because there are many possible solutions--consider a display that provides a clear graphical distinction between the longer-term units and the shorter-term ones and allows assessments of the frequencies with which these occur.  One simple solution is a scatterplot where the contiguous activity of a unit between times $x$ and $x + y$ is indicated by a point at $(x,y)$.  Modify one salient characteristic of the point, such as its color, to emphasize the distinction between $y \ge 1$ and $y \lt 1$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a crude illustration: the first plots units on the vertical axis (200 of them), time on the horizontal (75 days; it needs a grid to show the units of time), and unit activities on a gray scale where darker corresponds to longer continuous activity.  The second shows similar data as a scatterplot.  The latter could be accompanied by a histogram of frequencies.  The former ought to have the units sorted vertically by their average length in service.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/IQuUW.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/37b7g.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-09-30T15:36:57.600" CommentCount="3" />
  <row Id="3206" PostTypeId="2" ParentId="3200" CreationDate="2010-09-30T15:53:06.523" Score="7" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;To a great degree you can do whatever you like&lt;/strong&gt; provided you hold out enough data at random to test whatever model you come up with based on the retained data.  A 50% split can be a good idea.  Yes, you lose some ability to detect relationships, but what you gain is enormous; namely, the &lt;em&gt;ability to replicate your work&lt;/em&gt; before it is published.  No matter how sophisticated the statistical techniques you bring to bear, you will be shocked at how many &quot;significant&quot; predictors wind up being entirely useless when applied to the confirmation data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bear in mind, too, that &quot;relevant&quot; for prediction means more than a low p-value.  That, after all, only means it's likely a relationship found in this particular dataset is not due to chance.  For prediction it's actually more important to find the variables that exert substantial influence on the predictand (without over-fitting the model); that is, to find the variables that are likely to be &quot;real&quot; &lt;em&gt;and,&lt;/em&gt; when varied throughout a reasonable range of values (not just the values that might occur in your sample!), cause the predictand to vary appreciably.  When you have hold-out data to confirm a model, you can be more comfortable provisionally retaining marginally &quot;significant&quot; variables that might not have low p-values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For these reasons (and building on chl's fine answer), although I have found stepwise models, AIC comparisons, and Bonferroni corrections quite useful (especially with hundreds or thousands of possible predictors in play), these should not be the sole determinants of which variables enter your model.  Do not lose sight of the guidance afforded by theory, either: variables having strong theoretical justification to be in a model usually should be kept in, even when they are not significant, provided they do not create ill-conditioned equations (&lt;em&gt;e.g.,&lt;/em&gt; collinearity).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;NB&lt;/strong&gt;: After you have settled on a model and confirmed its usefulness with the hold-out data, it's fine to recombine the retained data with the hold-out data for final estimation.  Thus, nothing is lost in terms of the precision with which you can estimate model coefficients.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-09-30T15:53:06.523" CommentCount="3" />
  <row Id="3207" PostTypeId="1" AcceptedAnswerId="3256" CreationDate="2010-09-30T16:02:39.110" Score="3" ViewCount="154" Body="&lt;p&gt;I previously asked this &lt;a href=&quot;http://stats.stackexchange.com/q/2917/1381&quot;&gt;question&lt;/a&gt; about the validity of my solutions for for $SE$ given $n$, $\bar{X_i}$ and summary statistics from post-hoc multiple comparisons such as Fisher's $LSD$ and Tukey's $HSD$, but I would like a general approach that can be applied to other post-hoc tests as well as other statistics. My hope is that this approach would be less prone to arithmetic error and easier to implement than my trying to find an analytical solution each time I encounter a new statistic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to get some estimate of $SE$ from previous studies that I will use in a meta-analysis. In addition, I would like to be able to do this for any statistic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a general simulation / optimization approach to this problem?&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;$\bar{X_1}$, $\bar{X_2}$, some statistic, and n are given&lt;/li&gt;&#xA;&lt;li&gt;I am looking for $SE\pm 10\%$&lt;/li&gt;&#xA;&lt;li&gt;As in the earlier post, over estimates of SE are o.k.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;My idea is that the solution would be something like the following examples in which LSD is given:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;se.est &amp;lt;- function(x1bar, x2bar, lsd_obs, n, se) {&#xA;       y &amp;lt;- c(rnorm(n, x1bar, se), rnorm(n, x2bar, se)).&#xA;       x &amp;lt;- c(rep(1,n), rep(2,n))&#xA;       mse &amp;lt;- sum(lm(y~factor(x))$residuals^2)/(2*n-2)&#xA;       lsd_est &amp;lt;- qt(0.975,n)*sqrt(2*mse/2)&#xA;       ans &amp;lt;- (lsd_obs - lsd_est)^2&#xA;       }&#xA;&#xA;SE &amp;lt;- optimize(par = c(x1bar, x2bar, lsd_obs,n), fn = se.est)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;or&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for (i in 1:10000){&#xA;     se[i] &amp;lt;- rgamma(1, 1, 0.01)&#xA;     y[i] &amp;lt;- c(rnorm(n, x1bar, se), rnorm(n, x2bar, se)).&#xA;     x[i] &amp;lt;- c(rep(1,n), rep(2,n))&#xA;     mse[i] &amp;lt;- sum(lm(y~factor(x))$residuals^2)/(2*n-2)&#xA;     lsd_est[i] &amp;lt;- qt(0.975,n)*sqrt(2*mse/2)&#xA;     }&#xA;&#xA; se &amp;lt;- se[which.min((lsd_est - lsd_obs)^2)]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Your suggestions for a quick and effective approach would be appreciated; also, is one of the above preferable or substantially more efficient? if you think that an analytical solution is preferable, please see my previous &lt;a href=&quot;http://stats.stackexchange.com/q/2917/1381&quot;&gt;post&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1381" LastEditorUserId="1381" LastEditorDisplayName="" LastEditDate="2010-10-01T13:52:49.637" LastActivityDate="2010-10-02T00:22:45.550" Title="Given sample size, group means, and misc. post-hoc range statistics, can you suggest a good way to estimate variance through simulation?" Tags="&lt;r&gt;&lt;standard-deviation&gt;&lt;variance&gt;&lt;meta-analysis&gt;" AnswerCount="1" FavoriteCount="2" />
  <row Id="3208" PostTypeId="2" ParentId="3200" CreationDate="2010-09-30T16:04:05.910" Score="1" ViewCount="" Body="&lt;p&gt;You can do a seemingly unrelated regression and use an F test. Put your data in a form like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Out1 1 P11 P12 0  0   0&#xA;Out2 0 0   0   1  P21 P22&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;so that the predictors for your first outcome have their values when that outcome is the y variable and 0 otherwise and vice-versa. So your y is a list of both outcomes. P11 and P12 are the two predictors for the first outcome and P21 and P22 are the two predictors for the second outcome. If sex, say, is a predictor for both outcomes, its use to predict outcome 1 should be in a separate variable/column when predicting outcome 2. This lets your regression have different slopes/impacts for sex for each outcome.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this framework, you can use standard F testing procedures.&lt;/p&gt;&#xA;" OwnerUserId="401" LastActivityDate="2010-09-30T16:04:05.910" />
  <row Id="3209" PostTypeId="2" ParentId="3201" CreationDate="2010-09-30T16:06:50.210" Score="1" ViewCount="" Body="&lt;p&gt;Do you think that you could quantify all those attributes? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If yes, I would suggest performing a principal component analysis. In the general case where all the correlations are positive (and if they aren't, you can easily get there using some transformation), the first principal component can be considered as a measure of the total importance of the expert, since it's a weighted average of all the attributes (and the weights would be the corresponding contributions of the variables - Under this perspective, the method itself will &lt;em&gt;reveal&lt;/em&gt; the importance of each attribute). The score that each expert achieves in the first principal component is what you need to rank them. &lt;/p&gt;&#xA;" OwnerUserId="339" LastActivityDate="2010-09-30T16:06:50.210" CommentCount="4" />
  <row Id="3210" PostTypeId="2" ParentId="3199" CreationDate="2010-09-30T16:10:52.277" Score="4" ViewCount="" Body="&lt;p&gt;This question is about spatial correlation.  Many methods exist to characterize and quantify this.  What they all have in common is comparing values at one location to those at nearby locations.  Usually, the reference distribution is some kind of spatial stochastic process where data are generated independently from point to point (&quot;complete spatial randomness&quot;).  Some methods only characterize average behavior while others provide more detailed exploratory tools to identify clusters of extreme values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For three different approaches, check out (1) the literature on geostatistics/kriging/variography; (2) other measures of spatial correlation such as Ripley's K and L functions or &lt;a href=&quot;http://edndoc.esri.com/arcobjects/9.2/net/shared/geoprocessing/spatial_statistics_tools/hot_spot_analysis_getis_ord_gi_star_spatial_statistics_.htm&quot; rel=&quot;nofollow&quot;&gt;the Getis-Ord $G_i$ statistics&lt;/a&gt; ; and (3) geographically weighted regression.  Accessible, non-technical, and sort of correct explanations of all these can be found on ESRI.com .  The Wikipedia articles are scanty and of variable quality, unfortunately.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first two approaches are well supported with R packages such as &lt;em&gt;spatstat&lt;/em&gt; and &lt;em&gt;geoRglm&lt;/em&gt;.  There is also free software for (2), of which some of the best known is &lt;a href=&quot;http://geodacenter.asu.edu/&quot; rel=&quot;nofollow&quot;&gt;Geoda&lt;/a&gt; and &lt;a href=&quot;http://www.icpsr.umich.edu/CrimeStat/about.html&quot; rel=&quot;nofollow&quot;&gt;CrimeStat&lt;/a&gt; .  I know of no free implementation of GWR (#3), but there are &lt;a href=&quot;http://ncg.nuim.ie/ncg/GWR/&quot; rel=&quot;nofollow&quot;&gt;good resources&lt;/a&gt; maintained by its inventors. &lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-09-30T16:10:52.277" CommentCount="2" />
  <row Id="3211" PostTypeId="2" ParentId="3201" CreationDate="2010-09-30T16:15:40.633" Score="2" ViewCount="" Body="&lt;p&gt;Ultimately this may not be solely a statistical exercise.  PCA is a very powerful quantitative method that will allow you to generate a score or weights on its first few principal components that you can use for ranking.  However, explaining what the principal components are is very challenging.  They are quantitative constructs.  They are not dialectic ones.  Thus, to explain what they truly mean is sometimes not possible.  This is especially true if you have an audience that is not quantitative.  They will have no idea what you are talking about.  And, will think of your PCA as some cryptic black box.    &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead, I would simply line up all the relevant variables and use a weighting system based on what one thinks the weighting should be.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think if you develop this for outsiders, customers, users, it would be great if you could embed the flexibility of deciding on the weighting to the users.&lt;br&gt;&#xA;Some users may value years of experience much more than certification and vice verse.  If you can leave that decision to them.  This way your algorithm is not a black box they don't understand and they are not comfortable with.  You keep it totally transparent and up to them based on their own relative valuation of what matters.&lt;/p&gt;&#xA;" OwnerUserId="1329" LastEditorUserId="1329" LastEditorDisplayName="" LastEditDate="2010-09-30T18:39:41.760" LastActivityDate="2010-09-30T18:39:41.760" CommentCount="8" />
  <row Id="3212" PostTypeId="1" AcceptedAnswerId="3213" CreationDate="2010-09-30T16:19:47.023" Score="7" ViewCount="238" Body="&lt;p&gt;I was wondering what are the differences between Mode, Class and Type of R objects?&#xA;Type of a R object can be obtained by typeof() function, mode by mode(), and class by class(). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also any other similar functions and concepts that I missed?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks and regards!&lt;/p&gt;&#xA;" OwnerUserId="1005" LastEditorUserId="1005" LastEditorDisplayName="" LastEditDate="2010-09-30T16:25:25.557" LastActivityDate="2010-09-30T16:32:06.770" Title="Mode, Class and Type of R objects" Tags="&lt;r&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="2" />
  <row Id="3213" PostTypeId="2" ParentId="3212" CreationDate="2010-09-30T16:32:06.770" Score="11" ViewCount="" Body="&lt;p&gt;The &lt;code&gt;class()&lt;/code&gt; is used to define/identify what &quot;type&quot; an object is from the point of view of object-oriented programming in R. So for&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; x &amp;lt;- 1:3&#xA;&amp;gt; class(x)&#xA;[1] &quot;integer&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;any generic function that has an &quot;integer&quot; method will be used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;typeof()&lt;/code&gt; gives the &quot;type&quot; of object from R's point of view, whilst &lt;code&gt;mode()&lt;/code&gt; gives the &quot;type&quot; of object from the point of view of Becker, Chambers &amp;amp; Wilks (1988). The latter may be more compatible with other S implementations according to the &lt;a href=&quot;http://stat.ethz.ch/R-manual/R-devel/doc/manual/R-lang.html#Objects&quot; rel=&quot;nofollow&quot;&gt;R Language Definition&lt;/a&gt; manual.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd probably err on the side of using &lt;code&gt;typeof()&lt;/code&gt; in most cases unless it was for passing R objects to compiled code, where &lt;code&gt;storage.mode()&lt;/code&gt; will be useful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is usefully discussed in the R Language Definition as linked to above.&lt;/p&gt;&#xA;" OwnerUserId="1390" LastActivityDate="2010-09-30T16:32:06.770" />
  <row Id="3214" PostTypeId="1" CreationDate="2010-09-30T18:19:59.433" Score="4" ViewCount="107" Body="&lt;p&gt;In my animal experiments, I do survival studies, which generate Kaplan-Meier survival curves for each group, which I then compare with an appropriate log rank test.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is: If I have run a survival experiment with identical variables, say, five times, and the final outcome (in very layman's terms) happens to be slightly different each time, is there a test that I can do estimate the variability (variance) of my experimental runs?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One major confounding factor is the fact that survival is a continuous variable over time. Therefore, it is difficult to reduce it to a single statistic (that can then be compared with a test). A lot of people use the median survival (expressed in units of time) as a single statistic surrogate for survival, but it can often be misleading - depending upon the slope of the survival curve, and may not represent the true nature of the survival outcome. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone here help? Please also let me know if further clarifications are needed.&lt;/p&gt;&#xA;" OwnerUserId="1464" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-30T18:40:21.410" LastActivityDate="2010-10-01T01:56:17.783" Title="Inter-experimental variation in survival experiment - how to estimate variability?" Tags="&lt;estimation&gt;&lt;survival&gt;&lt;variability&gt;" AnswerCount="1" CommentCount="3" FavoriteCount="1" />
  <row Id="3215" PostTypeId="1" AcceptedAnswerId="3216" CreationDate="2010-09-30T19:21:21.627" Score="6" ViewCount="188" Body="&lt;p&gt;Addition, substraction, multiplication and division of normal distributions are well defined, but what about trigonometric operations?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, let us suppose that I'm trying to find the angle of a triangular wedge (modelled as a right-angle triangle)  with the two catheti having dimensions d1 and d2, both described as normal distributions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Both intuition and simulation tell me that the resulting distribution is normal, with mean atan(mean(d1)/mean(d2)). But is there a way to compute the distribution of the resulting angle? References on where I'd find the answer?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(For a bit of context, I'm working on statistical tolerancing of mechanical parts. My first impulse would be to simply simulate the whole process, check if the end result is reasonably normal, and compute the standard deviation. But I'm wondering if there might be a neater analytical approach.)&lt;/p&gt;&#xA;" OwnerUserId="77" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-09-30T21:30:00.260" LastActivityDate="2010-10-01T06:10:53.957" Title="Trigonometric operations on standard deviations" Tags="&lt;distributions&gt;&lt;standard-deviation&gt;" AnswerCount="2" CommentCount="4" FavoriteCount="1" />
  <row Id="3216" PostTypeId="2" ParentId="3215" CreationDate="2010-09-30T20:19:47.273" Score="6" ViewCount="" Body="&lt;p&gt;In this interpretation, the triangle is a right triangle of side lengths $X$ and $Y$ distributed binormally with expectations $\mu_x$ and $\mu_y$, standard deviations $\sigma_x$ and $\sigma_y$, and correlation $\rho$.  We seek the distribution of $\arctan(Y/X)$.  To this end, standardize $X$ and $Y$ so that&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$X = \sigma_x \xi + \mu_x$$ and $$Y = \sigma_y \eta + \mu_y$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;with $\xi$ and $\eta$ standard normal variates with correlation $\rho$.  Let $\theta$ be an angle and for convenience write $q = \tan(\theta)$.  Then&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\mathbb{P}[\arctan(Y/X) \le \theta] = \mathbb{P}[Y \le q X]$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$=\mathbb{P}[\sigma_y \eta + \mu_y \le q \left( \sigma_x \xi + \mu_x \right)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$=\mathbb{P}[\sigma_y \eta - q \sigma_x \xi \le q \mu_x - \mu_y]$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The left hand side, being a linear combination of Normals, is normal, with mean $\mu_y \sigma_y - q \mu_x \sigma_x$ and variance $\sigma_y^2 + q^2 \sigma_x^2 - 2 q \rho \sigma_x \sigma_y$.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Differentiating the Normal cdf of these parameters with respect to $\theta$ yields the pdf of the angle.  The expression is fairly grisly, but a key part of it is the exponential&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\exp \left(-\frac{\left(\mu _y \left(\sigma _y+1\right)-\mu _x \left(\sigma _x+1\right) \tan (\theta )\right){}^2}{2 \left(-2 \rho  \sigma _x \sigma _y \tan (\theta )+\sigma _x^2+\sigma _y^2+\tan ^2(\theta )\right)}\right),$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;showing right away that the angle is &lt;em&gt;not&lt;/em&gt; normally distributed.  However, as your simulations show and intuition suggests, it should be approximately normal provided the variations of the side lengths are small compared to the lengths themselves.  In this case a &lt;a href=&quot;http://projecteuclid.org/DPubS?service=UI&amp;amp;version=1.0&amp;amp;verb=Display&amp;amp;handle=euclid.aoms/1177728652&quot; rel=&quot;nofollow&quot;&gt;Saddlepoint approximation&lt;/a&gt; ought to yield good results for specific values of $\mu_x$, $\mu_y$, $\sigma_x$, $\sigma_y$, and $\rho$, even though a closed-form general solution is not available.  The approximate standard deviation will drop right out upon finding the second derivative (with respect to $\theta$) of the logarithm of the pdf (as shown in equations (2.6) and (3.1) of the reference).  I recommend a computer algebra system (like MatLab or Mathematica) for carrying this out!&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-09-30T20:19:47.273" CommentCount="5" />
  <row Id="3217" PostTypeId="2" ParentId="3179" CreationDate="2010-09-30T20:40:28.717" Score="5" ViewCount="" Body="&lt;p&gt;The best &quot;graph&quot; is so obvious nobody has mentioned it yet: make maps.  Housing data depend fundamentally on spatial location (according to the old saw about real estate), so the very first thing to be done is to make a clear detailed map of each variable.  To do this well with a third of a million points really requires an industrial-strength GIS, which can make short work of the process.  After that it makes sense to go on and make probability plots and boxplots to explore univariate distributions, and to plot scatterplot matrices and wandering schematic boxplots, etc, to explore dependencies--but the maps will immediately suggest what to explore, how to model the data relationships, and how to break up the data geographically into meaningful subsets.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-09-30T20:40:28.717" CommunityOwnedDate="2010-09-30T20:40:28.717" CommentCount="3" />
  <row Id="3218" PostTypeId="2" ParentId="3201" CreationDate="2010-09-30T21:01:26.463" Score="4" ViewCount="" Body="&lt;p&gt;People have invented numerous systems for rating things (like experts) on multiple criteria: visit the Wikipedia page on &lt;a href=&quot;http://en.wikipedia.org/wiki/Multi-criteria_decision_analysis&quot; rel=&quot;nofollow&quot;&gt;Multi-criteria decision analysis&lt;/a&gt; for a list.  Not well represented there, though, is one of the most defensible methods out there: &lt;a href=&quot;http://books.google.com/books?hl=en&amp;amp;lr=&amp;amp;id=GPE6ZAqGrnoC&amp;amp;oi=fnd&amp;amp;pg=PR11&amp;amp;dq=multi+attribute+evaluation+theory+raiffa&amp;amp;ots=EjKJPzatEx&amp;amp;sig=fd9rfTTHcOVIdv9FbmrRcdlyDcA#v=onepage&amp;amp;q&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;Multi attribute valuation theory.&lt;/a&gt;    This includes a set of methods to evaluate trade-offs among sets of criteria in order to (a) determine an appropriate way to re-express values of the individual variables and (b) weight the re-expressed values to obtain a score for ranking.  The principles are simple and defensible, the mathematics is unimpeachable, and there's nothing fancy about the theory.  More people should know and practice these methods rather than inventing arbitrary scoring systems.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-09-30T21:01:26.463" CommentCount="1" />
  <row Id="3219" PostTypeId="2" ParentId="3158" CreationDate="2010-09-30T22:37:31.757" Score="5" ViewCount="" Body="&lt;p&gt;I found that the dependency graph in Flare is also similar to what I want:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://flare.prefuse.org/apps/dependency_graph&quot; rel=&quot;nofollow&quot;&gt;http://flare.prefuse.org/apps/dependency_graph&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1106" LastActivityDate="2010-09-30T22:37:31.757" />
  <row Id="3220" PostTypeId="2" ParentId="3215" CreationDate="2010-09-30T22:55:48.303" Score="6" ViewCount="" Body="&lt;p&gt;You are looking at &lt;em&gt;circular statistics&lt;/em&gt; and in particular a circular distribution called the &lt;em&gt;projected normal distribution&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For some reason this topic can be a little hard to google, but the two major texts on circular statistics are &lt;a href=&quot;http://books.google.com.au/books?id=wGPj3EoFdJwC&amp;amp;lpg=PA37&amp;amp;ots=Ph2sDAAER6&amp;amp;dq=Fisher%20statistical%20circular%20data%20book&amp;amp;pg=PP1#v=onepage&amp;amp;q=Fisher%20statistical%20circular%20data%20book&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;The Statistical Analysis of Circular Data&lt;/a&gt; by Fisher and &#xA;&lt;a href=&quot;http://onlinelibrary.wiley.com/book/10.1002/9780470316979&quot; rel=&quot;nofollow&quot;&gt;Directional Statistics&lt;/a&gt; by Mardia and Jupp.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a thorough analysis of the projected normal distribution see page 46 of Mardia and Jupp. There are closed form expressions (up to the error function integral) for the distribution, and as whuber has suggested, it looks similar to the normal when its `variance' (careful here, what does variance mean for a random variable on a circle?!) is small, i.e. when the distribution is quite concentrated at one point (or direction or angle).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.itee.uq.edu.au/~robertm/pictures/statsexchange/projnorm_mardia.pdf&quot; rel=&quot;nofollow&quot;&gt;I photocopied the relevant pages of Directional Statistics for you guys.&lt;/a&gt; The book can be a little hard to find.&lt;/p&gt;&#xA;" OwnerUserId="352" LastEditorUserId="352" LastEditorDisplayName="" LastEditDate="2010-10-01T06:10:53.957" LastActivityDate="2010-10-01T06:10:53.957" CommentCount="2" />
  <row Id="3221" PostTypeId="1" AcceptedAnswerId="3224" CreationDate="2010-10-01T00:16:07.093" Score="3" ViewCount="65" Body="&lt;p&gt;I was wondering how to create a vector with equivalent spacing between its consecutive elements in R? In Matlab, I can do [start:step:end].&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also if I want to plot a function with analytical form, do I have to evaluate the function on some sample points in its domain and plot these pair of points? Is there a R function that can take the function form and a set in its domain as arguments, and plot the graph of the function on the set?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks and regards!&lt;/p&gt;&#xA;" OwnerUserId="1005" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-10-01T09:36:53.557" LastActivityDate="2010-10-01T09:36:53.557" Title="How to create a vector with equivalent spacing between its consecutive elements in R?" Tags="&lt;r&gt;&lt;data-visualization&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="3222" PostTypeId="2" ParentId="3221" CreationDate="2010-10-01T00:20:28.803" Score="2" ViewCount="" Body="&lt;p&gt;See &lt;a href=&quot;http://stuff.mit.edu/afs/sipb/project/r-project/arch/i386_rhel3/lib/R/library/base/html/seq.html&quot; rel=&quot;nofollow&quot;&gt;seq&lt;/a&gt; for sequence generation:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;seq(from, to, by)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;or &lt;code&gt;?seq&lt;/code&gt; for help.&lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-10-01T00:20:28.803" />
  <row Id="3223" PostTypeId="2" ParentId="3158" CreationDate="2010-10-01T00:22:16.480" Score="3" ViewCount="" Body="&lt;p&gt;I would just add:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you point out, Flare has the dependency graph, which Aleks Jakulin &lt;a href=&quot;http://www.stat.columbia.edu/~cook/movabletype/archives/2009/06/visualizing_tab.html&quot; rel=&quot;nofollow&quot;&gt;argued was similar but better&lt;/a&gt;.  This was based original on the &lt;a href=&quot;http://www.win.tue.nl/~dholten/papers/bundles_infovis.pdf&quot; rel=&quot;nofollow&quot;&gt;&quot;Hierarchical Edge Bundles:&#xA;Visualization of Adjacency Relations in Hierarchical Data&quot;&lt;/a&gt; (Holden 2006).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I personally prefer to use &lt;a href=&quot;http://vis.stanford.edu/protovis/&quot; rel=&quot;nofollow&quot;&gt;Protovis&lt;/a&gt; to Flare directly, and you can look at &lt;a href=&quot;http://cs.stanford.edu/people/mbostock/iv/dependency-tree.html&quot; rel=&quot;nofollow&quot;&gt;Mike Bostock's example of the same graphic&lt;/a&gt;.  Here is also an example of &lt;a href=&quot;http://vis.stanford.edu/protovis/ex/arc.html&quot; rel=&quot;nofollow&quot;&gt;an Arc Diagram in Protovis&lt;/a&gt;, which is very similar but laid out linearly.&lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-10-01T00:22:16.480" />
  <row Id="3224" PostTypeId="2" ParentId="3221" CreationDate="2010-10-01T00:26:46.690" Score="3" ViewCount="" Body="&lt;ol&gt;&#xA;&lt;li&gt;Use &lt;code&gt;seq&lt;/code&gt; as suggested by ars&lt;/li&gt;&#xA;&lt;li&gt;For example,  &lt;code&gt;plot(sin, -pi, 2*pi)&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-10-01T00:26:46.690" CommentCount="4" />
  <row Id="3226" PostTypeId="1" AcceptedAnswerId="3227" CreationDate="2010-10-01T00:48:40.123" Score="4" ViewCount="660" Body="&lt;p&gt;I would like to plot two functions f1 and f2 on the same graph. With the following code, I found that the scale of the y axes for the two plots are different. Is there a way to make the scales same?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plot(f1, 0, 1)&#xA;par(new=TRUE)&#xA;plot(f2, 0, 1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1005" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-10-01T09:25:19.393" LastActivityDate="2010-10-01T09:25:19.393" Title="Multiple Plots in the Same Figure [R]" Tags="&lt;r&gt;&lt;data-visualization&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="3227" PostTypeId="2" ParentId="3226" CreationDate="2010-10-01T00:56:19.453" Score="3" ViewCount="" Body="&lt;p&gt;Use the &lt;code&gt;curve&lt;/code&gt; function:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plot(f1, 0, 1)&#xA;curve(f2, 0, 1, add=TRUE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="159" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-10-01T01:22:11.487" LastActivityDate="2010-10-01T01:22:11.487" />
  <row Id="3228" PostTypeId="2" ParentId="2910" CreationDate="2010-10-01T00:58:04.533" Score="0" ViewCount="" Body="&lt;p&gt;Just my 2 cents. I've found Notepad++ useful for this.  I can maintain separate scripts (program control, data formatting, etc.) and a .pad file for each project.  The .pad file call's all the scripts associated with that project.&lt;/p&gt;&#xA;" LastActivityDate="2010-10-01T00:58:04.533" CommunityOwnedDate="2010-10-01T00:58:04.533" CommentCount="1" />
  <row Id="3229" PostTypeId="2" ParentId="3171" CreationDate="2010-10-01T01:04:39.917" Score="0" ViewCount="" Body="&lt;p&gt;You should adjust your standard errors (and p-values, confidence intervals, etc) to account for the observations not being independent. You can do this under some reasonable assumptions even though you don't know which observations are of the same person. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, suppose you're estimating the mean of some variable, x. Let x_{it} be the observations in your sample with the understanding that x_{it} and x_{it+1} are not necessarily the same person. Let e_{it} = x_{it} - mean(x). A central limit theorem for dependent data (e.g. &lt;a href=&quot;http://en.wikipedia.org/wiki/Central_limit_theorem#Under_weak_dependence&quot; rel=&quot;nofollow&quot;&gt;this one&lt;/a&gt;) tells us that the sample mean is asymptotically normal with variance&lt;/p&gt;&#xA;&#xA;&lt;p&gt;V = lim E[1/NT (sum e_{it})^2 ] = lim 1/NT sum E[e_{it}e_{js}]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where the limit is as NT -&gt; infinity, the first sum is over i,t and the second is over i,t,j,s. Assume that e_{it} are uncorrelated for different individuals. Suppose for a fixed individual, that e_{it} is weakly stationary with v_s = E[e_{t} e_{t+s} ]. Also, let's suppose that probability of the same person being sampled at time period s conditional on being in at time t is p. If the pool and sample sizes is fixed, then p = (sample size)/(pool size). Given that, we know&lt;/p&gt;&#xA;&#xA;&lt;p&gt;E[e_{it}e_{js}] = v_0 if t=s else p v_{t-s}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now assuming that T -&gt; infinity, we get&lt;/p&gt;&#xA;&#xA;&lt;p&gt;V = v_0 + p 2 sum_{t=1}^infinity v_t&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is the asymptotic variance of the sample mean. To use this result, you need to be able to estimate V consistently. V is called the long run variance. If N is fixed, it can be estimated by kernel methods. If N -&gt; infinity seems like a more appropriate asymptotic approximation, then you need not resort to kernels. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;All this is just meant as example to get you started. You can modify many of the assumptions if they're not appropriate for your setting. For example, p could vary with t and s instead of being constant. If you're not just interested in sample averages, you can do similar calculations for any statistic you want. &lt;/p&gt;&#xA;" OwnerUserId="1229" LastActivityDate="2010-10-01T01:04:39.917" CommunityOwnedDate="2010-10-01T01:04:39.917" />
  <row Id="3230" PostTypeId="2" ParentId="3214" CreationDate="2010-10-01T01:07:46.603" Score="1" ViewCount="" Body="&lt;p&gt;I suggest you read Section 2.3 &amp;amp; 2.4 pp40-73 of Hosmer &amp;amp; Lemeshow's 1999 edition of &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0471154105&quot; rel=&quot;nofollow&quot;&gt;Applied Survival Analysis&lt;/a&gt;. This gives variances of various statistics of survivorship functions, such as 1) each time (allowing confidence interval estimation), 2) mean survival, etc. I'm not clear from your question what aspect experimental variability you want to summarise. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You will be aware there are several test statistics which can be used to compare KM survival functions, such as the log-rank test you mention. They differ on the weighting applied to variance estimates of deaths at different survival times. However, the pooled result, which forms the test statistic, is not usually regarded as a variance estimate, in the same way that a chi-squared statistic is not a variance estimator, but a test of whether the &lt;a href=&quot;http://en.wikipedia.org/wiki/Variance#Distribution_of_the_sample_variance&quot; rel=&quot;nofollow&quot;&gt;sample distribution of the variance&lt;/a&gt; follows what is expected from the null hypothesis.&lt;/p&gt;&#xA;" OwnerUserId="521" LastEditorUserId="521" LastEditorDisplayName="" LastEditDate="2010-10-01T01:56:17.783" LastActivityDate="2010-10-01T01:56:17.783" CommentCount="1" />
  <row Id="3231" PostTypeId="2" ParentId="1380" CreationDate="2010-10-01T01:34:16.313" Score="1" ViewCount="" Body="&lt;p&gt;Hello Shabbychef,&#xA;Your questions is a good one (given I understand correctly). I believe you have K, 2x2 tables which correspond K different methods (call Z) and your aim is to say .. method K_1, K_2 ... K_n (K_i belongs to {1,...,K}) have some association between prediction and truth and the remaining don't have a relation. If you think this is a correct interpretation, proceed ahead o/w ignore my answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is similar to partial tables when we control for Z (ie condition on a particular method) and study the XY (truth and prediction) relationship at fixed levels of Z. Basically, two way cross-sectional slices of the three way contingency table cross classify X and Y at several values of Z (1 .. K for you). These cross sections are called &lt;em&gt;Partial&lt;/em&gt;  &lt;em&gt;Tables&lt;/em&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Associations for partial tables are called conditional associations, because they refer to the effect of X on Y conditional of fixing Z at some level. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For reference see: Agresti's Categorical Data Analysis Book &lt;a href=&quot;http://books.google.com/books?id=hpEzw4T0sPUC&amp;amp;printsec=frontcover&amp;amp;dq=categorical+data+analysis&amp;amp;source=bl&amp;amp;ots=noIl9pcqSb&amp;amp;sig=HUEavYHiONx84lylUSCX8Utze2s&amp;amp;hl=en&amp;amp;ei=tzelTLi-CoOdlgeunoGvDA&amp;amp;sa=X&amp;amp;oi=book_result&amp;amp;ct=result&amp;amp;resnum=1&amp;amp;ved=0CCQQ6AEwAA#v=onepage&amp;amp;q&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;link&lt;/a&gt;. The example 2.3.2 (Death penalty example) is a special case and talks about 2x2x2 tables. Section 2.3.3 answers your question for the general scenario. Your question can be answered by testing if there is conditional independence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is accomplished using CMH (Cochran Mantel Haenszel) Test of Conditional independence. In R, this can be done using &lt;a href=&quot;http://sekhon.berkeley.edu/stats/html/mantelhaen.test.html&quot; rel=&quot;nofollow&quot;&gt;this function&lt;/a&gt;. CMH generalizations for IxJxK tables also exist. In your case under the null, the distribution of test statistic is hypergeometric and for greater details see Section 6.3 of the Agresti book.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hopefully this answers your questions. Otherwise, I am sorry for making you read this far.&lt;/p&gt;&#xA;" OwnerUserId="1307" LastActivityDate="2010-10-01T01:34:16.313" />
  <row Id="3232" PostTypeId="1" AcceptedAnswerId="3234" CreationDate="2010-10-01T03:25:35.153" Score="5" ViewCount="124" Body="&lt;p&gt;My office is going to implement a bundle of infection control measures in hospital and see if it can effectively reduce the infection rate of some pathogen. The unit of measurement will be &quot;case per thousand patient bed days&quot;. We have chosen 4 wards for implementing the control measures for 12 months, and do the measurement monthly, but by looking at the current infection data, even thses wards are considered as having the highest count of infections, they are still considered as relatively clean, as they have several months with zero measurement in rates.&#xA;I have created a regression model showing if they managed to reduce 50% of infection, and the betas in the model (with three variables inside), are not statistically significant. My colleagues are worried that with all these hard work, giving non-significant result will be very frustrating for the front line staff.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any alernative outcome measure, or even alternative statistical methods, for time series of rare events? Thanks!&lt;/p&gt;&#xA;" OwnerUserId="588" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-10-01T13:32:19.047" LastActivityDate="2010-10-01T13:32:19.047" Title="Dependent variable selection for loglinear segmented regression in time-series analysis of rare events" Tags="&lt;time-series&gt;&lt;epidemiology&gt;&lt;monitoring&gt;" AnswerCount="1" FavoriteCount="2" />
  <row Id="3233" PostTypeId="2" ParentId="3200" CreationDate="2010-10-01T04:36:06.710" Score="0" ViewCount="" Body="&lt;p&gt;this is really a comment - but the system isn't letting me post it as such.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;hmmm... interesting problem - bivariate regression with one variable [sort of] continuous and one dichotomous. usual MVN regression theory says doing separate regressions on each of two bivariate normal responses - and then piecing the results together - is the correct analysis - in the gauss-markov sense of minimizing the v-c matrix of the regression estimators among all unbiased linear estimators - and in the MVN case, among all unbiased estimators]. is 'separate regressions' still the best one can do when one regression is logistic? [unbiasedness seems a bit of a stretch in that case, except perhaps in an asymptotic sense.] this remark is a bit off-topic - but doesn't one always answer a question with a question?&lt;/p&gt;&#xA;" OwnerUserId="1112" LastActivityDate="2010-10-01T04:36:06.710" />
  <row Id="3234" PostTypeId="2" ParentId="3232" CreationDate="2010-10-01T08:16:11.410" Score="3" ViewCount="" Body="&lt;p&gt;I think you're right to conclude that there's little hope of finding a 'statistically significant' result from 4 wards over 12 months. Of course, that doesn't mean the control measures don't work &amp;mdash; just that your sample size is far too small (and the variability too large) to have much chance of finding evidence that it's worked. I'd guess that larger studies have been done and at least some evidence exists about what control measures work, and you should assume the same measures will work in your hospital too unless you have good reason for thinking it's different.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Within your one hospital, rather than looking a the p-value(s) as the 'bottom line' i think you'd be better looking at this as performance monitoring. I quick bit of googling found a 2004 &lt;a href=&quot;http://www.his.org.uk//resource_library.cfm?cit_id=243&amp;amp;FAArea1=customWidgets.content_view_1&amp;amp;usecache=false&quot; rel=&quot;nofollow&quot;&gt;Report from Key Indicators Joint Working Group&lt;/a&gt; of the &lt;a href=&quot;http://www.his.org.uk/&quot; rel=&quot;nofollow&quot;&gt;Hospital Infection Society&lt;/a&gt;, which may be one place to start.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-10-01T08:16:11.410" />
  <row Id="3235" PostTypeId="1" AcceptedAnswerId="3243" CreationDate="2010-10-01T11:46:09.530" Score="12" ViewCount="367" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;I would like to measure the time that it takes to repeat the running of a function. Are &lt;code&gt;replicate()&lt;/code&gt; and using for-loops equivalent?  For example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;system.time(replicate(1000, f()));&#xA;system.time(for(i in 1:1000){f()});&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Which is the prefered method.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;In the output of &lt;code&gt;system.time()&lt;/code&gt;, is &lt;code&gt;sys+user&lt;/code&gt; the actual CPU time for running the program? Is &lt;code&gt;elapsed&lt;/code&gt; a good measure of time performance of the program?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="1005" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-10-01T13:31:31.707" LastActivityDate="2010-10-02T16:06:05.113" Title="Timing functions in R" Tags="&lt;r&gt;" AnswerCount="5" CommentCount="1" FavoriteCount="4" />
  <row Id="3236" PostTypeId="2" ParentId="3235" CreationDate="2010-10-01T12:03:11.517" Score="8" ViewCount="" Body="&lt;p&gt;Regarding your two points:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;It's stylistic. I like &lt;code&gt;replicate()&lt;/code&gt; as it is functional.&lt;/li&gt;&#xA;&lt;li&gt;I tend to focus on &lt;code&gt;elapsed&lt;/code&gt;, i.e. the third number.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;What I often do is &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;N &amp;lt;- someNumber&#xA;mean(replicate( N, system.time( f(...) )[3], trimmed=0.05) )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;to get a trimmed mean of 90% of N repetitions of calling &lt;code&gt;f()&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Edited, with thanks to Hadley for catching a thinko.)&lt;/p&gt;&#xA;" OwnerUserId="334" LastEditorUserId="334" LastEditorDisplayName="" LastEditDate="2010-10-01T20:35:05.737" LastActivityDate="2010-10-01T20:35:05.737" CommentCount="7" />
  <row Id="3237" PostTypeId="2" ParentId="3235" CreationDate="2010-10-01T13:30:55.063" Score="5" ViewCount="" Body="&lt;p&gt;You can also time with timesteps returned by &lt;code&gt;Sys.time&lt;/code&gt;; this of course measures walltime, so real time computation time. Example code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Sys.time()-&amp;gt;start;&#xA;replicate(N,doMeasuredComputation());&#xA;print(Sys.time()-start);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-10-01T13:30:55.063" />
  <row Id="3238" PostTypeId="1" AcceptedAnswerId="3257" CreationDate="2010-10-01T14:58:01.400" Score="8" ViewCount="704" Body="&lt;p&gt;I have a set of time series data. Each series covers the same period, although the actual dates in each time series may not all 'line up' exactly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That is to say, if the Time series were to be read into a 2D matrix, it would look something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;date     T1   T2   T3 .... TN&#xA;1/1/01   100  59   42      N/A&#xA;2/1/01   120  29   N/A     42.5&#xA;3/1/01   110  N/A  12      36.82&#xA;4/1/01   N/A  59   40      61.82&#xA;5/1/01    05  99   42      23.68&#xA;...&#xA;31/12/01  100  59   42     N/A&#xA;&#xA;etc &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to write an R script that will segregate the time series {T1, T2, ... TN} into 'families' where a family is defined as a set of series which &quot;tend to move in sympathy&quot; with each other. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the 'clustering' part, I will need to select/define a kind of distance measure. I am not quite sure how to go about this, since I am dealing with time series, and a pair of series that may move in sympathy over one interval, may not do so in a subsequent interval.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am sure there are far more experienced/clever people than me on here, so I would be grateful for any suggestions, ideas on what algorithm/heuristic to use for the distance measure and how to use that in clustering the time series. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My guess is that there is NOT an established robust statistic method for doing this, so I would be very interested to see how people approach/solve this problem - thinking like a statistician.&lt;/p&gt;&#xA;" OwnerUserId="1216" LastEditorUserId="1216" LastEditorDisplayName="" LastEditDate="2010-10-01T15:11:05.647" LastActivityDate="2010-10-02T05:13:39.340" Title="Time series 'clustering' in R" Tags="&lt;r&gt;&lt;time-series&gt;&lt;clustering&gt;&lt;cointegration&gt;" AnswerCount="3" CommentCount="2" FavoriteCount="13" />
  <row Id="3239" PostTypeId="2" ParentId="3238" CreationDate="2010-10-01T15:05:48.723" Score="8" ViewCount="" Body="&lt;p&gt;Another way of saying &quot;tend to move in sympathy&quot; is &quot;cointegrated&quot;.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are two standard ways of calculating &lt;a href=&quot;http://en.wikipedia.org/wiki/Cointegration&quot; rel=&quot;nofollow&quot;&gt;cointegration&lt;/a&gt;: Engle-Granger method and the Johansen procedure.  These are covered in &lt;a href=&quot;http://www.springer.com/statistics/statistical+theory+and+methods/book/978-0-387-75966-1&quot; rel=&quot;nofollow&quot;&gt;&quot;Analysis of Integrated and Cointegrated Time Series with R&quot;&lt;/a&gt; (Pfaff 2008) and the related R &lt;a href=&quot;http://cran.r-project.org/web/packages/urca/index.html&quot; rel=&quot;nofollow&quot;&gt;urca package&lt;/a&gt;.  I highly recommend the book if you want to pursue these methods in R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also recommend that you look at &lt;a href=&quot;http://stackoverflow.com/questions/1714280/multivariate-time-series-modelling-in-r/1715488#1715488&quot;&gt;this question on multivariate time series&lt;/a&gt; and, in particular, at &lt;a href=&quot;http://faculty.chicagobooth.edu/ruey.tsay/teaching/mts/sp2009/&quot; rel=&quot;nofollow&quot;&gt;Ruey Tsay's course at U. Chicago&lt;/a&gt; which includes all the necessary R code.&lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-10-01T15:05:48.723" CommentCount="5" />
  <row Id="3240" PostTypeId="2" ParentId="3235" CreationDate="2010-10-01T15:48:30.923" Score="1" ViewCount="" Body="&lt;p&gt;They do different things.  Time what you wish done.  replicate() returns a vector of results of each execution of the function.  The for loop does not.  Therefore, they're not equivalent statements.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In addition, time a number of ways you want something done.  Then you can find the most efficient method.&lt;/p&gt;&#xA;" OwnerUserId="601" LastEditorUserId="601" LastEditorDisplayName="" LastEditDate="2010-10-02T16:06:05.113" LastActivityDate="2010-10-02T16:06:05.113" CommentCount="1" />
  <row Id="3241" PostTypeId="2" ParentId="3238" CreationDate="2010-10-01T16:34:37.630" Score="1" ViewCount="" Body="&lt;p&gt;Clustering time series is done fairly commonly by population dynamacists, particularily those that study insects to understand trends in outbreak and collapse. Look for work on Gypsy moth, Spruce budoworm, mountain pine beetle and larch budmoth. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the actual clustering you can choose whatever distance metric you like, each probably has it's own strengths and weeknesses relative to the kind of data being clustered, Kaufmann and Rousseeuw 1990. &lt;em&gt;Finding groups in data. An introduction to cluster analysis&lt;/em&gt; is a good place to start. Remember, the clustering method doesn't 'care' that you're using a time series, it only looks at the values measured at the same point of time.  If your two time series are not in enough synch over their lifespan they the won't (and perhaps shouldn't) cluster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where you will have problems is determining the number of clusters (families) to use after you've clusterd the time series. There are various ways of selecting a cut-off of informative clusters, but here the literature isn't that good. &lt;/p&gt;&#xA;" OwnerUserId="1475" LastActivityDate="2010-10-01T16:34:37.630" CommentCount="1" />
  <row Id="3242" PostTypeId="1" AcceptedAnswerId="3246" CreationDate="2010-10-01T17:04:11.523" Score="7" ViewCount="319" Body="&lt;p&gt;I have some data to which I am trying to fit a trendline.  I believe the data to follow a power law, and so have plotted the data on log-log axes looking for a straight line.  This has resulted in an (almost) straight line and so in Excel I have added a trendline for a power law.  Being a stats newb, my question is, what is now the best way for me to go from &quot;well the line &lt;em&gt;looks&lt;/em&gt; like it fits pretty well&quot; to &quot;numeric property &lt;code&gt;x&lt;/code&gt; &lt;em&gt;proves&lt;/em&gt; that this graph is fitted appropriately by a power law&quot;?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In Excel I can get an r-squared value, though given my limited knowledge of statistics, I don't even know whether this is actually appropriate under my specific circumstances.  I have included an image below showing the plot of the data I am working with in Excel.  I have a little bit of experience with R, so if my analysis is being limited by my tools, I am open to suggestions on how to go about improving it using R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/NhPf2.jpg&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="870" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-11-15T11:15:12.153" LastActivityDate="2010-11-15T11:15:12.153" Title="How to measure/argue the goodness of fit of a trendline to a power law?" Tags="&lt;goodness-of-fit&gt;" AnswerCount="3" CommentCount="0" FavoriteCount="3" />
  <row Id="3243" PostTypeId="2" ParentId="3235" CreationDate="2010-10-01T17:08:15.340" Score="4" ViewCount="" Body="&lt;p&gt;For effective timing of programs, especially when you are interested in comparing alternative solutions, you need a control!  A good way is to put the procedure you're timing into a function.  Call the function within a timing loop.  Write a stub procedure, essentially by stripping out all the code from your function and just returning from it (but leave all the arguments in).  Put the stub into your timing loop and re-time.  This measures all the overhead associated with the timing.  Subtract the stub time from the procedure time to get the net: this should be an accurate measure of the actual time needed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because most systems nowadays can be peremptorily interrupted, it is important to do several timing runs to check for variability.  Instead of doing one long run of $N$ seconds, do $m$ runs of about $N/m$ seconds each.  It helps to do this in a double loop all in one go.  Not only is that easier to handle, it introduces a little bit of negative correlation in each time series, which actually improves the estimates.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By using these basic principles of experimental design, you essentially control for any differences due to how you deploy the code (e.g., the difference between a for loop and replicate()).  That makes your problem go away.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-10-01T17:08:15.340" />
  <row Id="3244" PostTypeId="1" AcceptedAnswerId="3248" CreationDate="2010-10-01T17:21:51.887" Score="6" ViewCount="316" Body="&lt;p&gt;I know most of you probably feel that Google Docs is still a primitive tool.  It is no Matlab or R and not even Excel.  Yet, I am baffled at the power of this web based software that just uses the operating capability of a browser (and is compatible with many browsers that work very differently).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Mike Lawrence, active in this forum, has shared a spreadsheet with us using Google Docs doing some pretty fancy stuff with it.  I personally have replicated a pretty thorough hypothesis testing framework (including numerous parametric and nonparametric tests) originally done in Excel in Google Docs.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am interested if any of you have given Google Docs a try and have pushed it to its limits in interesting applications.  I am also interested to hear about the bugs or flaws you have encountered with Google Docs&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am designating this question &quot;for community wiki&quot; denoting that there are no best answers for this.  It is more of a survey than anything.  &lt;/p&gt;&#xA;" OwnerUserId="1329" LastEditorUserId="1329" LastEditorDisplayName="" LastEditDate="2010-10-01T19:19:20.980" LastActivityDate="2010-10-07T07:32:26.627" CommunityOwnedDate="2010-10-01T17:21:51.887" Title="Do some of you use Google Docs spreadsheet to conduct and share your statistical work with others?" Tags="&lt;software&gt;&lt;testing&gt;&lt;application&gt;&lt;hypothesis&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="3" />
  <row Id="3245" PostTypeId="2" ParentId="3242" CreationDate="2010-10-01T17:27:50.137" Score="0" ViewCount="" Body="&lt;p&gt;you might find some ideas here&#xA;&lt;a href=&quot;http://freakonometrics.blog.free.fr/index.php?post/2010/09/29/Detecting-distributions-with-infinite-mean&quot; rel=&quot;nofollow&quot;&gt;http://freakonometrics.blog.free.fr/index.php?post/2010/09/29/Detecting-distributions-with-infinite-mean&lt;/a&gt;&lt;/p&gt;&#xA;" LastActivityDate="2010-10-01T17:27:50.137" />
  <row Id="3246" PostTypeId="2" ParentId="3242" CreationDate="2010-10-01T18:22:17.717" Score="8" ViewCount="" Body="&lt;p&gt;See Aaron Clauset's page:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://tuvalu.santafe.edu/~aaronc/powerlaws/&quot; rel=&quot;nofollow&quot;&gt;Power-law Distributions in Empirical Data&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;which has links to code for fitting power laws (Matlab, R, Python, C++) as well as a paper by Clauset and Shalizi you should read first.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might want to read Clauset's and Shalizi's blogs posts on the paper first:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://cs.unm.edu/~aaron/blog/archives/2007/06/power_laws_and.htm&quot; rel=&quot;nofollow&quot;&gt;Power laws and all that jazz&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://cscs.umich.edu/~crshalizi/weblog/491.html&quot; rel=&quot;nofollow&quot;&gt;So You Think You Have a Power Law — Well Isn't That Special?&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="251" LastEditorUserId="251" LastEditorDisplayName="" LastEditDate="2010-10-01T18:42:15.803" LastActivityDate="2010-10-01T18:42:15.803" CommentCount="5" />
  <row Id="3247" PostTypeId="2" ParentId="3244" CreationDate="2010-10-01T18:31:56.090" Score="9" ViewCount="" Body="&lt;p&gt;As an enthusiast user of R, bash, Python, asciidoc, (La)TeX, open source sofwtare or any un*x tools, I cannot provide an objective answer. Moreover, as I often argue against the use of MS Excel or spreadsheet of any kind (well, you see your data, or part of it, but what else?), I would not contribute positively to the debate. I'm not the only one, e.g.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.burns-stat.com/pages/Tutor/spreadsheet_addiction.html&quot; rel=&quot;nofollow&quot;&gt;Spreadsheet Addiction&lt;/a&gt;, from P. Burns.    &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://tolstoy.newcastle.edu.au/R/help/04/12/9427.html&quot; rel=&quot;nofollow&quot;&gt;MS Excel’s precision and accuracy&lt;/a&gt;, a post on the 2004 R mailing-list     &lt;/li&gt;&#xA;&lt;li&gt;L. Knusel, &lt;a href=&quot;http://www.sciencedirect.com/science?_ob=ArticleURL&amp;amp;_udi=B6V8V-3SX6MW1-7&amp;amp;_user=10&amp;amp;_coverDate=01%2F02%2F1998&amp;amp;_rdoc=1&amp;amp;_fmt=high&amp;amp;_orig=search&amp;amp;_origin=search&amp;amp;_sort=d&amp;amp;_docanchor=&amp;amp;view=c&amp;amp;_searchStrId=1482343457&amp;amp;_rerunOrigin=google&amp;amp;_acct=C000050221&amp;amp;_version=1&amp;amp;_urlVersion=0&amp;amp;_userid=10&amp;amp;md5=37c2574f11ec79e1450143a179e0f294&amp;amp;searchtype=a&quot; rel=&quot;nofollow&quot;&gt;On the accuracy of statistical distributions in Microsoft Excel 97&lt;/a&gt;, Computational Statistics &amp;amp; Data Analysis, 26: 375–377, 1998. (&lt;a href=&quot;http://www.stat.uni-muenchen.de/~knuesel/elv/excelacc.pdf&quot; rel=&quot;nofollow&quot;&gt;pdf&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;li&gt;B.D. McCullough &amp;amp; B. Wilson, &lt;a href=&quot;http://www.sciencedirect.com/science?_ob=ArticleURL&amp;amp;_udi=B6V8V-45M6FB7-1&amp;amp;_user=10&amp;amp;_coverDate=10%2F28%2F2002&amp;amp;_rdoc=1&amp;amp;_fmt=high&amp;amp;_orig=search&amp;amp;_origin=search&amp;amp;_sort=d&amp;amp;_docanchor=&amp;amp;view=c&amp;amp;_searchStrId=1482342476&amp;amp;_rerunOrigin=google&amp;amp;_acct=C000050221&amp;amp;_version=1&amp;amp;_urlVersion=0&amp;amp;_userid=10&amp;amp;md5=da5e1130403a5de62ec14ffa6a0f778e&amp;amp;searchtype=a&quot; rel=&quot;nofollow&quot;&gt;On the accuracy of statistical procedures in Microsoft Excel&#xA;2000 and Excel XP&lt;/a&gt;, &lt;em&gt;Computational Statistics &amp;amp; Data Analysis&lt;/em&gt;, 40: 713–721, 2002. &lt;/li&gt;&#xA;&lt;li&gt;M. Altman, J. Gill &amp;amp; M.P. McDonald, &lt;em&gt;Numerical Issues in Statistical Computing for the Social Scientist&lt;/em&gt;, Wiley, 2004. [e.g., pp. 12–14]&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;A colleague of mine loose all his macros because of the lack of backward compatibility, etc. Another colleague tried to import genetics data (around 700 subjects genotyped on 800,000 markers, 120 Mo), just to &quot;look at them&quot;. Excel failed, Notepad gave up too... I am able to &quot;look at them&quot; with vi, and quickly reformat the data with some sed/awk or perl script. So I think there are different levels to consider when discussing about the usefulness of spreadsheets. Either you work on small data sets, and only want to apply elementary statistical stuff and maybe it's fine. Then, it's up to you to trust the results, or you can always ask for the source code, but maybe it would be simpler to do a quick test of all inline procedures with the &lt;a href=&quot;http://www.itl.nist.gov/div898/strd/&quot; rel=&quot;nofollow&quot;&gt;NIST benchmark&lt;/a&gt;. I don't think it corresponds to a good way of doing statistics simply because this is not a &lt;em&gt;true&lt;/em&gt; statistical software (IMHO), although as an update of the aforementioned list, newer versions of MS Excel seems to have demonstrated improvements in its accuracy for statistical analyses, see Keeling and Pavur, &lt;a href=&quot;http://www.sciencedirect.com/science?_ob=ArticleURL&amp;amp;_udi=B6V8V-4JHMGWJ-1&amp;amp;_user=10&amp;amp;_coverDate=05%2F01%2F2007&amp;amp;_rdoc=1&amp;amp;_fmt=high&amp;amp;_orig=search&amp;amp;_origin=search&amp;amp;_sort=d&amp;amp;_docanchor=&amp;amp;view=c&amp;amp;_searchStrId=1488136142&amp;amp;_rerunOrigin=google&amp;amp;_acct=C000050221&amp;amp;_version=1&amp;amp;_urlVersion=0&amp;amp;_userid=10&amp;amp;md5=2babab0b51c03746d5d7a74d31f1498c&amp;amp;searchtype=a&quot; rel=&quot;nofollow&quot;&gt;A comparative study of the reliability of nine statistical software packages&lt;/a&gt; (&lt;em&gt;CSDA&lt;/em&gt; 2007 51: 3811).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Still, about one paper out of 10 or 20 (in biomedicine, psychology, psychiatry) includes graphics made with Excel, sometimes without removing the gray background, the horizontal black line or the automatic legend (Andrew Gelman and Hadley Wickham are certainly as happy as me when seeing it). But more generally, it tend to be the most used &quot;software&quot; according to a &lt;a href=&quot;http://datafl.ws/120&quot; rel=&quot;nofollow&quot;&gt;recent poll&lt;/a&gt; on FlowingData, which remind me of an old talk of Brian Ripley (who co-authored the MASS R package, and write an excellent book on pattern recognition, among others):&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Let's not kid ourselves: the most&#xA;  widely used piece of software for&#xA;  statistics is Excel (B. Ripley via Jan&#xA;  De Leeuw), &lt;a href=&quot;http://bit.ly/dB5K6r&quot; rel=&quot;nofollow&quot;&gt;http://bit.ly/dB5K6r&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Now, if you feel it provides you with a quick and easier way to get your statistics done, why not? The problem is that there are still things that cannot be done (or at least, it's rather tricky) in such an environment. I think of bootstrap, permutation, multivariate exploratory data analysis, to name a few. Unless you are very proficient in VBA (which is neither a scripting nor a programming language), I am inclined to think that even minor operations on data are better handled under R (or Matlab, or Python, providing you get the right tool for dealing with e.g. so-called data.frame). Above all, I think Excel does not promote very good practices for the data analyst (but it also applies to any &quot;cliquodrome&quot;, see the discussion on Medstats about the need to maintain a record of data processing, &lt;a href=&quot;http://groups.google.com/group/medstats/browse_thread/thread/601793e6ce36e789&quot; rel=&quot;nofollow&quot;&gt;Documenting analyses and data edits&lt;/a&gt;), and I found this post on &lt;a href=&quot;http://www.practicalstats.com/xlsstats/excelstats.html&quot; rel=&quot;nofollow&quot;&gt;Practical Stats&lt;/a&gt; relatively illustrative of some of Excel pitfalls. Still, it applies to Excel, I don't know how it translates to GDocs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;About sharing your work, I tend to think that &lt;a href=&quot;http://github.com/&quot; rel=&quot;nofollow&quot;&gt;Github&lt;/a&gt; (or &lt;a href=&quot;http://gist.github.com/&quot; rel=&quot;nofollow&quot;&gt;Gist&lt;/a&gt; for source code) or &lt;a href=&quot;http://www.dropbox.com&quot; rel=&quot;nofollow&quot;&gt;Dropbox&lt;/a&gt; (although EULA might discourage some people) are very good options (revision history, grant management if needed, etc.). I cannot encourage the use of a software which basically store your data in a binary format. I know it can be imported in R, Matlab, Stata, SPSS, but to my opinion:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;data should definitively be in a text format, that can be read by another statistical software;&lt;/li&gt;&#xA;&lt;li&gt;analysis should be reproducible, meaning you should provide a complete script for your analysis and it should run (we approach the ideal case near here...) on another operating system at any time;&lt;/li&gt;&#xA;&lt;li&gt;your own statistical software should implement acknowledged algorithms and there should be an easy way to update it to reflect current best practices in statistical modeling;&lt;/li&gt;&#xA;&lt;li&gt;the sharing system you choose should include versioning and collaborative facilities.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;That's it.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-07T07:32:26.627" LastActivityDate="2010-10-07T07:32:26.627" CommunityOwnedDate="2010-10-01T18:31:56.090" CommentCount="6" />
  <row Id="3248" PostTypeId="2" ParentId="3244" CreationDate="2010-10-01T19:23:05.050" Score="7" ViewCount="" Body="&lt;p&gt;Hello Gaetan,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My main use for google spreadsheets have been with google forms, for collecting data, and then easily importing it into R.  Here is a post I wrote about it half a year ago:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.r-statistics.com/2010/03/google-spreadsheets-google-forms-r-easily-collecting-and-importing-data-for-analysis/&quot; rel=&quot;nofollow&quot;&gt;Google spreadsheets + google forms + R = Easily collecting and importing data for analysis&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, If you are into collaboration, my tool of choice is DropBox.  I wrote a post regarding it a few months ago:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.r-statistics.com/2010/05/syncing-files-across-computers-using-dropbox/&quot; rel=&quot;nofollow&quot;&gt;Syncing files across computers using DropBox&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have now been using it for about half a year on a project with 5 co-authors, and it has been invaluable (syncing data files from 3 contributers, everyone can see the latest version of the output I am producing, and everyone are looking at the same .docx file for the article).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Both posts offer video tutorials and verbal instructions.&lt;/p&gt;&#xA;" OwnerUserId="253" LastActivityDate="2010-10-01T19:23:05.050" CommunityOwnedDate="2010-10-01T19:23:05.050" CommentCount="2" />
  <row Id="3249" PostTypeId="1" CreationDate="2010-10-01T20:05:58.550" Score="7" ViewCount="291" Body="&lt;p&gt;Hi,&#xA;I want to forecast retail items (by week) using exponential smoothing.   I'm stuck right now in how to calculate, store, and apply the sesonality indexes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is that all examples I've found deal with a sort of simple seasonality.   In my case I have the following problems:&#xA;1.  Seasons don't occur on the same week every year:  they are movable.   Mardi-gras, lent, easter, and a few others.&#xA;2.  There are seasons that change depending on the year.  For example, there is a national holiday season.   Depending on whether the holiday is close to the weekend, customers will or will not leave town.  So it's kind of like having two seasons:  one where customers leave town, and one where they don't leave town.&#xA;3.  Sometimes two (or 3) seasons occur at the same time.   For example, we had &quot;Mardi-Gras&quot; season occurring at the same time as Valentine's season.&lt;br&gt;&#xA;4.  Sometimes seasons change in duration.   For example, &quot;Halloween season&quot; started earlier this year.   Christmas is also another example, where it seems like every year we start earlier to carry the products.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems to me that I need to find a way to set some sort of &quot;seasonal profiles&quot; which then, depending on the particular scenario are somehow added to obtain the correct seasonal index.  Does that make sense?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anybody know where I can find practical information on how to do this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks,&#xA;Edgard&lt;/p&gt;&#xA;" OwnerUserId="1479" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-10-03T03:39:27.527" LastActivityDate="2010-10-03T03:51:53.697" Title="Calculation of seasonality indexes for complex seasonality" Tags="&lt;time-series&gt;&lt;seasonality&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="3250" PostTypeId="2" ParentId="3235" CreationDate="2010-10-01T20:51:10.613" Score="1" ViewCount="" Body="&lt;p&gt;Regarding which timing metric to use, I can not add to the other responders.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regarding the function to use, I like using the ?benchmark from the &lt;a href=&quot;http://cran.r-project.org/web/packages/rbenchmark/index.html&quot; rel=&quot;nofollow&quot;&gt;rbenchmark package&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="253" LastActivityDate="2010-10-01T20:51:10.613" />
  <row Id="3251" PostTypeId="2" ParentId="3249" CreationDate="2010-10-01T20:54:45.623" Score="1" ViewCount="" Body="&lt;p&gt;A simple fix would be to include events dummies in your specification:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$(1) \hat{y_t}=\lambda_1 y_{t-1}+...+\lambda_k y_{t-k}+\phi_1 D_{t,1}+\phi_m D_{t,m}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $D_{t,m}$ is an indicator taking value $1$ if week $t$ has event $m$ (say Mardi gras) and 0 otherwise, for all $m$ events you deem important. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first part of the specification $\lambda_1 y_{t-1}+...+\lambda_k y_{t-k}$ is essentially a exponential smoothers but with weight varying as a function of lags (and estimated by OLS).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This pre-supposes that you have at least 20 observations for each event (i.e. 20 'mardi gras'). If this is not the case, you can try to bundle some events together (say mardi gras and labour day).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The R to fit (1) is rather straighforward, assuming dlsales is stationnary and D is your matrix of dummy variables :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;fit&amp;lt;-arima(dlsales,order=c(4,0,0),seasonal = list(order = c(1, 0, 0),period=52),xreg = D)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Starting from here, you can ask more specific questions about the part of my answer that are not familiar to you (i don't know what your level is in statistics).&lt;/p&gt;&#xA;" OwnerUserId="603" LastEditorUserId="603" LastEditorDisplayName="" LastEditDate="2010-10-02T07:42:40.397" LastActivityDate="2010-10-02T07:42:40.397" CommentCount="8" />
  <row Id="3252" PostTypeId="1" AcceptedAnswerId="3254" CreationDate="2010-10-01T21:52:02.977" Score="10" ViewCount="420" Body="&lt;p&gt;Exploratory data analysis (EDA) often leads to explore other &quot;tracks&quot; that do not necessarily belong to the initial set of hypotheses. I face such a situation in the case of studies with a limited sample size and a lot of data gathered through different questionnaires (socio-demographics data, neuropsychological or medical scales -- e.g., mental or physical functioning, depression/anxiety level, symptoms checklist). It happens that EDA helps to highlight some unexpected relationships (&quot;unexpected&quot; meaning that they were not included in the initial analysis plan) that translates into additional questions/hypothesis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As is the case for overfitting, data &lt;a href=&quot;http://en.wikipedia.org/wiki/Data_dredging&quot; rel=&quot;nofollow&quot;&gt;dredging&lt;/a&gt; or &lt;a href=&quot;http://en.wikipedia.org/wiki/Data_snooping&quot; rel=&quot;nofollow&quot;&gt;snooping&lt;/a&gt; does lead to results that do not generalize. However, when a lot of data is available, it is quite difficult (for the researcher or physician) to postulate a limited set of hypotheses.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to know if there are well-acknowledged methods, recommendations, or rules of thumb that may help to delineate EDA in the case of small-sample studies.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-10-25T06:50:55.937" LastActivityDate="2010-11-01T10:11:29.257" Title="How to cope with exploratory data analysis and data dredging in small-sample studies?" Tags="&lt;multiple-comparisons&gt;&lt;epidemiology&gt;&lt;small-sample&gt;&lt;eda&gt;" AnswerCount="2" CommentCount="4" FavoriteCount="6" />
  <row Id="3253" PostTypeId="2" ParentId="3038" CreationDate="2010-10-01T23:13:10.210" Score="1" ViewCount="" Body="&lt;p&gt;Bootstrap differences (e.g. the difference between the means) between the 2 sample groups and check for statistical significance. A more detailed description of this approach, albeit in a different context, can be found here &lt;a href=&quot;http://www.automated-trading-system.com/a-different-application-of-the-bootstrap/&quot; rel=&quot;nofollow&quot;&gt;http://www.automated-trading-system.com/a-different-application-of-the-bootstrap/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="226" LastActivityDate="2010-10-01T23:13:10.210" />
  <row Id="3254" PostTypeId="2" ParentId="3252" CreationDate="2010-10-01T23:19:38.370" Score="5" ViewCount="" Body="&lt;p&gt;I think the main thing is to be honest when reporting such results that they were unexpected findings from EDA and not part of the initial analysis plan based on an &lt;i&gt;a priori&lt;/i&gt; hypothesis. Some people like to label such results 'hypothesis generating': e.g. the &lt;a href=&quot;http://jco.ascopubs.org/content/23/30/7512.short&quot; rel=&quot;nofollow&quot;&gt;first hit&lt;/a&gt; from a &lt;a href=&quot;http://scholar.google.co.uk/scholar?hl=en&amp;amp;q=%22hypothesis+generating%22&quot; rel=&quot;nofollow&quot;&gt;search for this phrase on Google Scholar&lt;/a&gt; includes the following in the conclusion section of its abstract:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;i&gt;As this was an &quot;exploratory&quot; analysis, this effect should be considered as hypothesis generating and assessed prospectively in other trials...&lt;/i&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Though note that although this was a post-hoc subgroup analysis it was from a randomized control trial, not an observational study, in which the problem gets worse. Philip Cole poured scorn on the idea that observational ('epidemiologic') studies can generate hypotheses in a deliberately provocative but entertaining commentary:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P Cole. &lt;a href=&quot;http://www.jstor.org/stable/3702282&quot; rel=&quot;nofollow&quot;&gt;The hypothesis generating machine.&lt;/a&gt; &lt;i&gt;Epidemiology&lt;/i&gt; 1993; &lt;b&gt;4&lt;/b&gt;:271-273.&lt;/p&gt;&#xA;" OwnerUserId="449" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-10-02T07:49:14.927" LastActivityDate="2010-10-02T07:49:14.927" CommentCount="2" />
  <row Id="3255" PostTypeId="2" ParentId="3249" CreationDate="2010-10-01T23:54:36.450" Score="5" ViewCount="" Body="&lt;p&gt;For the kinds of seasonality you describe, the dummy variable approach is probably best. However, this is easier to handle in an ARIMA framework than an exponential smoothing framework. &#xA;\begin{aligned}&#xA;y_t &amp;amp;= a + b_1D_{t,1} + \cdots + b_mD_{t,m} + N_t\\&#xA;N_t &amp;amp;\sim \text{ARIMA}&#xA;\end{aligned}&#xA;where each $D_{t,k}$ variable corresponds to one of the holiday or festival events. This is how the &lt;code&gt;arima&lt;/code&gt; function in R will fit regression variables (as a regression with ARIMA errors, not as an ARIMAX model).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you really want to stick with the exponential smoothing framework, there is a discussion of how to include covariates in my &lt;a href=&quot;http://www.exponentialsmoothing.net&quot; rel=&quot;nofollow&quot;&gt;2008 book on exponential smoothing&lt;/a&gt;. You might also look at my recent paper on &lt;a href=&quot;http://robjhyndman.com/working-papers/complex-seasonality/&quot; rel=&quot;nofollow&quot;&gt;exponential smoothing with complex seasonality&lt;/a&gt; although the types of seasonal complications we discuss there are more difficult than the moving festival kind that you describe.&lt;/p&gt;&#xA;" OwnerUserId="159" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-10-03T03:51:53.697" LastActivityDate="2010-10-03T03:51:53.697" CommentCount="2" />
  <row Id="3256" PostTypeId="2" ParentId="3207" CreationDate="2010-10-02T00:22:45.550" Score="2" ViewCount="" Body="&lt;p&gt;With a small modification, the first version will be far more efficient. As it's written, se.est is a random function. Even at the same arguments, its value will change each time because of rnorm. This will mess up optimize. You should use the same random numbers each time se.est is called. Here's one way:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;e &amp;lt;- c(rnorm(2*n,0,1))&#xA;se.est &amp;lt;- function(se,x1bar, x2bar, lsd_obs, n, e) {&#xA;       y &amp;lt;- se*e + c(rep(x1bar,n),rep(x2bar,n)&#xA;       x &amp;lt;- c(rep(1,n), rep(2,n))&#xA;       mse &amp;lt;- sum(lm(y~factor(x))$residuals^2)/(2*n-2)&#xA;       lsd_est &amp;lt;- qt(0.975,n)*sqrt(2*mse/2)&#xA;       ans &amp;lt;- (lsd_obs - lsd_est)^2&#xA;       }&#xA;&#xA;SE &amp;lt;- optimize(f=se.est,interval=c(0.1,100),x1bar,x2bar,lsd_obs,n,e)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I didn't try to run this, so I may have made typos or syntax errors, but hopefully the idea is clear. If n is small, you should probably optimize multiple times with different e's, and use the something like the average or maybe the 90th percentile of the resulting SE's.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This may not be a concern, but if the original data was not normal, this method could give inaccurate results. Without assuming normality, you could write your problem as:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; max_y se(y) subject to mean1(y)=x1bar, mean(2)=x2bar, statistic(y)=s&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where you allow y to be a sample of size n with any distribution. Without any more restrictions, the solution will often be infinite (it depends on the statistic). &lt;/p&gt;&#xA;" OwnerUserId="1229" LastActivityDate="2010-10-02T00:22:45.550" />
  <row Id="3257" PostTypeId="2" ParentId="3238" CreationDate="2010-10-02T05:13:39.340" Score="4" ViewCount="" Body="&lt;p&gt;In data streaming and mining of time series databases, a common approach is to transform the series to a symbolic representation, then use a similarity metric, such as Euclidean distance, to cluster the series.  The most popular representations are SAX (Keogh &amp;amp; Lin) or the newer iSAX (Shieh &amp;amp; Keogh):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.cs.ucr.edu/~eamonn/SAX.htm&quot; rel=&quot;nofollow&quot;&gt;Symbolic Aggregate approXimation&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.cs.ucr.edu/~eamonn/iSAX/iSAX.html&quot; rel=&quot;nofollow&quot;&gt;iSAX: Indexing and Mining Terabyte Sized Time Series&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The pages above also contain references to distance metrics and clustering.  Keogh and crew are into reproducible research and pretty receptive to releasing their code.  So you could email them and ask.  I believe they tend to work in MATLAB/C++ though.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There was a recent effort to produce a Java and R implementation:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://code.google.com/p/jmotif/&quot; rel=&quot;nofollow&quot;&gt;jmotif&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I don't know how far along it is -- it's geared towards motif finding, but, depending on how far they've gotten, it should have the necessary bits you need to put something together for your needs (iSAX and distance metrics: since this part is common to clustering and motif finding).&lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-10-02T05:13:39.340" CommentCount="1" />
  <row Id="3258" PostTypeId="2" ParentId="3151" CreationDate="2010-10-02T08:22:18.313" Score="1" ViewCount="" Body="&lt;p&gt;Dear friends,  through the wonders of twitter, someone reached out and sent me a link to &lt;a href=&quot;http://ananelson.com/blog/2009/09/introducing-roux/&quot; rel=&quot;nofollow&quot;&gt;this page&lt;/a&gt;, regarding a package called &quot;roux&quot;.  It was created a year ago, and I have never heard about it (apparently neither have most of you).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This package seems to do exactly what I was looking for in my question, although the installation seems non trivial.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope to play with this solution and also to see if other R members might go into this project to better enhance R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for everyone's answers!&lt;/p&gt;&#xA;" OwnerUserId="253" LastActivityDate="2010-10-02T08:22:18.313" CommentCount="3" />
  <row Id="3259" PostTypeId="1" CreationDate="2010-10-02T09:29:19.373" Score="10" ViewCount="332" Body="&lt;p&gt;This question is about estimating cut-off scores on a multi-dimensional screening questionnaire to predict a binary endpoint, in the presence of correlated scales. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was asked about the interest of controlling for associated subscores when devising cut-off scores on each dimension of a measurement scale (personality traits) which might be used for alcoholism screening. That is, in this particular case, the person was not interested in adjusting on external covariates (predictors) -- which leads to (partial) area under covariate-adjusted ROC curve, e.g. (1-2) -- but essentially on other scores from the same questionnaire because they correlate one to each other (e.g. &quot;impulsivity&quot; with &quot;sensation seeking&quot;). It amounts to build an GLM which includes on the left-side the score of interest (for which we seek a cut-off) and another score computed from the same questionnaire, while on the right-hand side the outcome may be drinking status. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To clarify (per @robin request), suppose we have $j=4$ scores, say $x_j$ (e.g., anxiety, impulsivity, neuroticism, sensation seeking), and we want to find a cut-off value $t_j$ (i.e. &quot;positive case&quot; if $x_j&gt;t_j$, &quot;negative case&quot; otherwise) for each of them. We usually adjust for other risk factors like gender or age when devising such cut-off (using ROC curve analysis). Now, what about adjusting impulsivity (IMP) on gender, age, and sensation seeking (SS) since SS is known to correlate with IMP? In other words, we would have a cut-off value for IMP where effect of age, gender and anxiety level are removed. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Apart from saying that a cut-off must remain as simple as possible, my response was&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;About covariates, I would recommend&#xA;  estimating the AUCs with and without&#xA;  adjustment, just to see if the&#xA;  predictive performance increase. Here,&#xA;  your covariates are merely other&#xA;  subscores defined from the same&#xA;  measurement instrument and I never&#xA;  faced such a situation (usually, I&#xA;  adjust on known risk factors, like Age&#xA;  or Gender). [...] Also, since you are&#xA;  interested in prognostic issues (i.e.&#xA;  screening efficacy of the questionnaire), you&#xA;  may also be interested in estimating&#xA;  the positive predictive value (PPV,&#xA;  probability of patients with positive&#xA;  test results who are correctly&#xA;  classified) provided you are able to&#xA;  classify subjects as &quot;positive&quot; or&#xA;  &quot;negative&quot; depending on their&#xA;  subscores on your questionnaire. Note, however,&#xA;  that it is necessary to know the&#xA;  prevalence of this disorder to&#xA;  correctly interpret the PPV in turn...&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Do you have a more thorough understanding of this particular situation, with link to relevant papers when possible?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;References&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Janes, H and Pepe, MS (2008). &lt;a href=&quot;http://aje.oxfordjournals.org/content/168/1/89.full&quot; rel=&quot;nofollow&quot;&gt;Adjusting for Covariates in Studies of Diagnostic, Screening, or Prognostic Markers: An Old Concept in a New Setting&lt;/a&gt;. &lt;em&gt;American Journal of Epidemiology&lt;/em&gt;, 168(1): 89-97.&lt;/li&gt;&#xA;&lt;li&gt;Janes, H and Pepe, MS (2008). &lt;a href=&quot;http://www.bepress.com/uwbiostat/paper322/&quot; rel=&quot;nofollow&quot;&gt;Accommodating Covariates in ROC Analysis&lt;/a&gt;. &lt;em&gt;UW Biostatistics Working Paper Series&lt;/em&gt;, Paper 322.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-09T13:22:04.150" LastActivityDate="2010-11-09T13:22:04.150" Title="Adjusting for covariates in ROC curve analysis" Tags="&lt;epidemiology&gt;&lt;roc&gt;" CommentCount="2" FavoriteCount="3" />
  <row Id="3260" PostTypeId="1" CreationDate="2010-10-02T12:32:01.853" Score="3" ViewCount="107" Body="&lt;p&gt;Hi there&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a few (5-6) data sets, each is a function of time, with the time span the same between datasets.  These datasets are all statistics of various perspectives of something (the partitions of a graph), and I am trying to find points in time which show interesting changes across the different datasets.  Normally these interesting features are local minima, but this may not always be the case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So for example, dataset1. is the variance of some measure and at time 0.4 it has a local minima.  dataset2. is the number of maxima and it also has a local minima at 0.4.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is in reality sometimes a minima will show on all datasets and sometimes only on one or a few datasets.  So my goal is basically to combine the information from all datasets to increase my confidence that a local minima at a particular time, is an interesting and significant point.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A simple method would just be to count the number of datasets which have a local minima at that time and and use some threshold value, above which I define the minima to be significant.  But I am wondering if there are more intelligent ways, i.e. ways which can see which datasets are more robust and reliable etc.&lt;/p&gt;&#xA;" OwnerUserId="809" LastActivityDate="2010-11-01T03:15:58.853" Title="Combining many datasets to increase confidence" Tags="&lt;multiple-comparisons&gt;&lt;multivariate-analysis&gt;" AnswerCount="1" CommentCount="3" FavoriteCount="2" />
  <row Id="3261" PostTypeId="1" AcceptedAnswerId="3274" CreationDate="2010-10-02T17:05:12.987" Score="2" ViewCount="183" Body="&lt;p&gt;Suppose you had a method for estimating the population covariance of a vector-valued random variable given observations of that random variable, say $f(Z) \rightarrow C$, where the rows of $Z$ are observations of the random variable. Can one abuse this process to perform a least squares regression $y = x^T\beta + \epsilon$, for $n$-dimensional vector $x$? The idea is that one would have a vector of observations of $y$, call it $Y$, and a matrix of paired observations of $x$, call it $X$, then one would compute the covariance of $Z = [X\; Y]$ (concatenate the matrix next to the vector) via $f$, call it $C$, then let $\hat{\beta} = C_{1:n,1:n}^{-1} C_{1:n,n+1}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A few questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Will this work under optimistic conditions? (a simple simulation in Matlab shows that the precision is not great, but the results are within 4 sig figs, so I am guessing it will.)&lt;/li&gt;&#xA;&lt;li&gt;Is this a known trick? if so, does it have a name I can google search for, or is it so trivial that it doesn't require a name?&lt;/li&gt;&#xA;&lt;li&gt;Most importantly, if $f$ can deal with input where some values are missing (say, MCAR--missing completely at random), under what conditions will this technique behave reasonably for regression with missing values?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;edit&lt;/strong&gt; I &lt;em&gt;am&lt;/em&gt; assuming that $x$ is drawn from a zero mean process and that the regression has no intercept term.&lt;/p&gt;&#xA;" OwnerUserId="795" LastEditorUserId="795" LastEditorDisplayName="" LastEditDate="2010-10-03T04:21:50.877" LastActivityDate="2010-10-03T04:21:50.877" Title="Using Covariance Estimator to Perform Linear Regression?" Tags="&lt;regression&gt;&lt;algorithms&gt;&lt;missing-data&gt;&lt;variance-covariance&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="3262" PostTypeId="1" CreationDate="2010-10-02T17:06:33.500" Score="13" ViewCount="374" Body="&lt;p&gt;I am about to help teach statistics to medical students this semester.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've heard many horror stories about the fear of these students from learning statistics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone suggest what to do with this fear? (Either links to people who are discussing this, or offer suggestions from your own experience)&lt;/p&gt;&#xA;" OwnerUserId="253" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-10-02T17:39:01.153" LastActivityDate="2010-10-04T15:40:44.507" CommunityOwnedDate="2010-10-02T17:06:33.500" Title="How to teach students who fear statistics?" Tags="&lt;teaching&gt;" AnswerCount="7" FavoriteCount="8" />
  <row Id="3263" PostTypeId="2" ParentId="328" CreationDate="2010-10-02T17:17:44.863" Score="0" ViewCount="" Body="&lt;p&gt;Kennedy's &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1405182571&quot; rel=&quot;nofollow&quot;&gt;Guide to Econometrics&lt;/a&gt; is a good survey of techniques in econometrics--not detailed enough to get your hands dirty, but very good for discovering what techniques are being used.&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-10-02T17:17:44.863" />
  <row Id="3264" PostTypeId="2" ParentId="3262" CreationDate="2010-10-02T17:30:57.223" Score="7" ViewCount="" Body="&lt;p&gt;Not very much about how to deal with students' fear, but Andrew Gelman wrote an excellent book, &lt;a href=&quot;http://www.stat.columbia.edu/~gelman/bag-of-tricks/&quot; rel=&quot;nofollow&quot;&gt;Teaching Statistics, a bag of tricks&lt;/a&gt; (there's also some &lt;a href=&quot;http://www.stat.columbia.edu/~gelman/presentations/smithtalk.pdf&quot; rel=&quot;nofollow&quot;&gt;slides&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I like introducing a course by talking about randomness, elementary probability as found in games, causal association, permutation tests (because parametric tests provide good approximation to them :).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I just put an example that I like to show to students. This is from Phillip Good, in his book &lt;em&gt;&lt;a href=&quot;http://www.springer.com/statistics/statistical+theory+and+methods/book/978-0-387-20279-2&quot; rel=&quot;nofollow&quot;&gt;Permutation, Parametric, and Bootstrap Tests of Hypotheses&lt;/a&gt;&lt;/em&gt; (Springer, 2005 3rd ed.), where he introduces the general strategy of testing or decision making about statistical hypothesis and how to carry out a very simple and exact permutation test to solve the follwoing problem.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Shortly after I received my doctorate&#xA;  in statistics, I decided that if I&#xA;  really wanted to help bench scientists&#xA;  apply statistics I ought to become a&#xA;  scientist myself. So I went back to&#xA;  school to learn physiology and aging&#xA;  in cells raised in petri dishes.   &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;I soon learned there was a great deal&#xA;  more to an experiment than the random&#xA;  assignment of subjects to treatments.&#xA;  In general, 90% of experimental effort&#xA;  was spent mastering various arcane&#xA;  laboratory techniques, another 9% in&#xA;  developing new techniques to span the&#xA;  gap between what had been done and&#xA;  what I really wanted to do, and a mere&#xA;  1% on the experiment itself. But the&#xA;  moment of truth came finally–—it had&#xA;  to if I were to publish and not&#xA;  perish–—and I succeeded in cloning&#xA;  human diploid fibroblasts in eight&#xA;  culture dishes: Four of these dishes&#xA;  were filled with a conventional&#xA;  nutrient solution and four held an&#xA;  experimental &quot;life-extending&quot; solution&#xA;  to which vitamin E had been added.  &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;I waited three weeks with fingers&#xA;  crossed that there was no&#xA;  contamination of the cell cultures,&#xA;  but at the end of this test period&#xA;  three dishes of each type had&#xA;  survived. My technician and I&#xA;  transplanted the cells, let them grow&#xA;  for 24 hours in contact with a&#xA;  radioactive label, and then fixed and&#xA;  stained them before covering them with&#xA;  a photographic emulsion.  &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Ten days passed and we were ready to examine&#xA;  the autoradiographs. Two years had&#xA;  elapsed since I first envisioned this&#xA;  experiment and now the results were&#xA;  in: I had the six numbers I needed. &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;&quot;I've lost the labels,&quot; my technician&#xA;  said as she handed me the results.&#xA;  This was a dire situation. Without the&#xA;  labels, I had no way of knowing which&#xA;  cell cultures had been treated with&#xA;  vitamin E and which had not.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-02T18:33:09.533" LastActivityDate="2010-10-02T18:33:09.533" CommunityOwnedDate="2010-10-02T17:30:57.223" CommentCount="3" />
  <row Id="3265" PostTypeId="2" ParentId="3262" CreationDate="2010-10-02T18:10:23.917" Score="4" ViewCount="" Body="&lt;p&gt;This is a topic that would be of interest to members of the &lt;a href=&quot;http://www.lawrence.edu/fast/jordanj/isostat.html&quot; rel=&quot;nofollow&quot;&gt;Isolated Statisticians&lt;/a&gt; group in the ASA.  You are likely to get many useful responses from experienced teachers there, so I'll limit what I share here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's useful to understand where your students are coming from.  A low-stress pre-test can help you identify their strengths, weaknesses, and fears.  Sample tests for this purpose are provided in the instructor's manual to the Freedman, Pisani, Purves &lt;em&gt;Statistics&lt;/em&gt; text.  Get a copy of the manual through your institution.  (I think the publisher will send it free.)  (If you're really interested in this, I can post a version of these tests I have used for pre-assessment of undergraduates.)  Another good source of test material related to intro statistics is the &lt;a href=&quot;https://app.gen.umn.edu/artist/index.html&quot; rel=&quot;nofollow&quot;&gt;Artist&lt;/a&gt; Web site.  As a working statistician, of course you will want to engage in some quantitative measurement of the learning that occurs in your class ;-).  That site is a great resource for test questions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a large and growing literature about teaching intro stats.  A place to start is the online &lt;a href=&quot;http://www.amstat.org/publications/jse/&quot; rel=&quot;nofollow&quot;&gt;Journal of Statistical Education&lt;/a&gt;.  At a minimum you will find articles there about using case studies and datasets relevant to medical students; you might uncover some that specifically address teaching this population.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When asked to teach such courses, I have always found it helpful to reach out to the other faculty and, when possible, the students themselves to find out what they really need to know and what might motivate them.  Medical students are really busy and they didn't go to school to learn statistics, but they know they'll need to understand the papers they will be reading throughout their careers.  If you're not familiar with medical literature, a few hours with the best journals, like the Lancet and JAMA, will help you appreciate what they're working towards.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-10-02T18:10:23.917" CommunityOwnedDate="2010-10-02T18:10:23.917" />
  <row Id="3266" PostTypeId="2" ParentId="3180" CreationDate="2010-10-02T19:05:07.083" Score="3" ViewCount="" Body="&lt;p&gt;I like the partial identification approach to missing data of Manski. The basic idea is to ask: given all possible values the missing data could have, what is the set of values that the estimated parameters could take? This set might be very large, in which case you could consider restricting the distribution of the missing data. Manski has a bunch of papers and a book on this topic. &lt;a href=&quot;http://www.sciencedirect.com/science?_ob=ArticleURL&amp;amp;_udi=B6V07-4DW3JW4-2&amp;amp;_user=10&amp;amp;_coverDate=06%2F30%2F2005&amp;amp;_rdoc=5&amp;amp;_fmt=high&amp;amp;_orig=browse&amp;amp;_origin=browse&amp;amp;_zone=rslt_list_item&amp;amp;_srch=doc-info%28%23toc%235639%232005%23999609997%23593547%23FLP%23display%23Volume%29&amp;amp;_cdi=5639&amp;amp;_sort=d&amp;amp;_docanchor=&amp;amp;view=c&amp;amp;_ct=14&amp;amp;_acct=C000050221&amp;amp;_version=1&amp;amp;_urlVersion=0&amp;amp;_userid=10&amp;amp;md5=6a9ef3df98bd41c067b8cf99c81a5850&amp;amp;searchtype=a&quot; rel=&quot;nofollow&quot;&gt;This short paper&lt;/a&gt; is a good overview.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Inference in partially identified models can be complicated and is an active area of research. This &lt;a href=&quot;http://www.annualreviews.org/doi/abs/10.1146/annurev.economics.050708.143401?journalCode=economics&quot; rel=&quot;nofollow&quot;&gt;review&lt;/a&gt; (&lt;a href=&quot;http://www.google.com/url?sa=t&amp;amp;source=web&amp;amp;cd=5&amp;amp;sqi=2&amp;amp;ved=0CDEQFjAE&amp;amp;url=http%3A%2F%2Ffaculty.wcas.northwestern.edu%2F~ett094%2FPIE.pdf&amp;amp;rct=j&amp;amp;q=chernozhukov%20partial%20identification&amp;amp;ei=YH2nTJiWCMO88gbl2fSLDQ&amp;amp;usg=AFQjCNHw3d2pgEkRsO9ttBwEQfo7MHFdng&amp;amp;sig2=V9Adhrquz0p2ug47F8A-eQ&amp;amp;cad=rja&quot; rel=&quot;nofollow&quot;&gt;ungated pdf&lt;/a&gt;) is a good place to get started.&lt;/p&gt;&#xA;" OwnerUserId="1229" LastActivityDate="2010-10-02T19:05:07.083" CommentCount="4" />
  <row Id="3267" PostTypeId="2" ParentId="3262" CreationDate="2010-10-02T19:21:19.000" Score="7" ViewCount="" Body="&lt;p&gt;Try to personalize statistics. To show why understanding its concepts (even though they will forget the math, acknowledge it) is useful to &lt;em&gt;them&lt;/em&gt;. For instance, how to interpret breast cancer test results. To quote from &lt;a href=&quot;http://yudkowsky.net/rational/bayes&quot; rel=&quot;nofollow&quot;&gt;http://yudkowsky.net/rational/bayes&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Here's a story problem about a&#xA;  situation that doctors often&#xA;  encounter:&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;1% of women at age forty who&#xA;  participate in routine screening have&#xA;  breast cancer.  80% of women with&#xA;  breast cancer will get positive&#xA;  mammographies.  9.6% of women without&#xA;  breast cancer will also get positive&#xA;  mammographies.  A woman in this age&#xA;  group had a positive mammography in a&#xA;  routine screening.  What is the&#xA;  probability that she actually has&#xA;  breast cancer?&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;What do you think the answer is?  If&#xA;  you haven't encountered this kind of&#xA;  problem before, please take a moment&#xA;  to come up with your own answer before&#xA;  continuing.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Next, suppose I told you that most&#xA;  doctors get the same wrong answer on&#xA;  this problem - usually, only around&#xA;  15% of doctors get it right. &#xA;  (&quot;Really?  15%?  Is that a real&#xA;  number, or an urban legend based on an&#xA;  Internet poll?&quot;  It's a real number. &#xA;  See Casscells, Schoenberger, and&#xA;  Grayboys 1978; Eddy 1982; Gigerenzer&#xA;  and Hoffrage 1995; and many other&#xA;  studies.  It's a surprising result&#xA;  which is easy to replicate, so it's&#xA;  been extensively replicated.)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Since you're students will be medical doctors, make it clear: &lt;em&gt;if they don't understand statistics, they will give the wrong interpretation of the results to their patients&lt;/em&gt;. This is not an academical matter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also acknowledge that unless they go in research, they &lt;em&gt;will&lt;/em&gt; forget the details you will teach them. Don't even hope it's not the case. Aim for them to understand the fundamental concepts (type 1 and 2 errors, correlations and causations and so on) so when faced with a situation, they will remember &quot;hey, perhaps I shouldn't rush drawing a conclusion, but talk to someone who understand stats better&quot;. Preventing cognitive errors and teaching them to be inquisitive of the results provided by others (especially in an industry where large sums of money are at stake) will be signs you succeeded.&lt;/p&gt;&#xA;" OwnerUserId="840" LastActivityDate="2010-10-02T19:21:19.000" CommunityOwnedDate="2010-10-02T19:21:19.000" CommentCount="1" />
  <row Id="3268" PostTypeId="1" CreationDate="2010-10-02T19:59:17.413" Score="7" ViewCount="190" Body="&lt;p&gt;I have around ten groups (of companies). Each group is connected with each other group. The data I have is representing the strength of the connection. Imagine it's the number of times someone from group A sent an email to group B.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The strength of a connection can be 0. There are two connections between two groups, A-B and B-A.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;a) What would be a good way to visualize this? I could imagine, for example, that each group is a circle. Lines connect the circles and the thickness of the line represents the strength of the connection. Being able to indicate the size of the groups would be a plus but not required.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;b) Do you know of any software tools to visualize this? The tool shouldn't be too expensive or should be available as a trial version as for now it's just a proof of concept. It doesn't have to be web-based.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="1489" LastEditorUserId="1036" LastEditorDisplayName="" LastEditDate="2010-10-04T03:11:41.937" LastActivityDate="2011-05-03T08:26:00.337" Title="Visualization of connections between groups" Tags="&lt;data-visualization&gt;&lt;networks&gt;" AnswerCount="4" FavoriteCount="2" />
  <row Id="3269" PostTypeId="2" ParentId="3268" CreationDate="2010-10-02T20:17:39.443" Score="2" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://gephi.org/&quot; rel=&quot;nofollow&quot;&gt;Gephi&lt;/a&gt; is pretty good for visualization directed or undirected graphs/networks.  Another option might be &lt;a href=&quot;http://www.caida.org/tools/visualization/walrus/&quot; rel=&quot;nofollow&quot;&gt;Walrus&lt;/a&gt;.  &lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-10-02T20:17:39.443" />
  <row Id="3270" PostTypeId="1" AcceptedAnswerId="3278" CreationDate="2010-10-02T22:12:51.957" Score="6" ViewCount="476" Body="&lt;p&gt;I've been searching the internet far and wide... I have yet to find a really good overview of how to interpret 2D correspondence analysis plots. Could someone offer some advice on interpreting the distances between points? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Perhaps an example would help, here is a plot that's found on many of the websites I've seen that discuss correspondence analysis. The red triangles represent eye colour and the black dots represent hair colour.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/nVOUx.jpg&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Looking at the graph above, could you make a few statements about what you see in these data. Points of interest about the different dimensions and relationships between triangles and dots? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;An explanation of row-points verses column points, and the use of the word &quot;profile&quot; with particular focus on the example would be instrumental. &lt;/p&gt;&#xA;" OwnerUserId="776" LastActivityDate="2010-10-04T20:40:05.720" Title="Interpreting 2D Correspondence Analysis Plots" Tags="&lt;interpretation&gt;&lt;correspondence&gt;" AnswerCount="1" FavoriteCount="3" />
  <row Id="3271" PostTypeId="1" AcceptedAnswerId="3284" CreationDate="2010-10-02T22:16:46.023" Score="7" ViewCount="515" Body="&lt;p&gt;Hello All,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have seen a few queries on clustering in time series and specifically on clustering, but I don't think they answer my question. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Background:&lt;/strong&gt; I want to cluster genes in a time course experiment in yeast. There are four time points say: &lt;em&gt;t1&lt;/em&gt;  &lt;em&gt;t2&lt;/em&gt;  &lt;em&gt;t3&lt;/em&gt;  and  &lt;em&gt;t4&lt;/em&gt; and total number of genes &lt;em&gt;G&lt;/em&gt;. I have the data in form a matrix &lt;em&gt;M&lt;/em&gt; in which the columns represent the treatments (or time points)  &lt;em&gt;t1&lt;/em&gt;  &lt;em&gt;t2&lt;/em&gt;  &lt;em&gt;t3&lt;/em&gt;  and  &lt;em&gt;t4&lt;/em&gt;  and the rows represent the genes. Therefore, &lt;em&gt;M&lt;/em&gt; is a Gx4 matrix. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Problem:&lt;/strong&gt; I want to cluster the genes which behave the same across all time points &lt;em&gt;t1&lt;/em&gt;  &lt;em&gt;t2&lt;/em&gt;  &lt;em&gt;t3&lt;/em&gt;  and  &lt;em&gt;t4&lt;/em&gt;  as well as within a particular time point &lt;em&gt;ti&lt;/em&gt; , where i is in {1, 2, 3, 4} (In case we cannot do both the clusterings together, the clustering within a time point is more important than clustering across time points). In addition to this, I also want to draw a heatmap.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My Solution:&lt;/strong&gt; &#xA;I use the R code below to obtain a heatmap as well as the clusters using &lt;code&gt;hclust&lt;/code&gt; function in R (performs hierarchical clustering with euclidean distance)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    row.scaled.expr &amp;lt;- (expr.diff - rowMeans(expr.diff)) / rowSds(expr.diff)&#xA;&#xA;    breaks.expr &amp;lt;- c(quantile(row.scaled.expr[row.scaled.expr &amp;lt; 0],&#xA;                               seq(0,1,length=10)[-9]), 0,&#xA;                               quantile(row.scaled.expr[row.scaled.expr &amp;gt; 0],&#xA;                               seq(0,1,length=10))[-1] )&#xA;&#xA;&#xA;    blue.red.expr &amp;lt;- maPalette(low = &quot;blue&quot;, high = &quot;red&quot;, mid = &quot;white&quot;,&#xA;                     k=length(breaks.expr) - 1)&#xA;&#xA;    pdf(&quot;images/clust.pdf&quot;,&#xA;         height=30,width=20,pointsize=20)&#xA;    ht1 &amp;lt;- heatmap.2(row.scaled.expr, col = blue.red.expr, Colv = FALSE, key = FALSE, &#xA;      dendrogram = &quot;row&quot;, scale = &quot;none&quot;, trace = &quot;none&quot;,&#xA;      cex=1.5, cexRow=1, cexCol=2,&#xA;      density.info = &quot;none&quot;, breaks = breaks.expr, &#xA;      labCol = colnames(row.scaled.expr),&#xA;      labRow=&quot;&quot;,&#xA;      lmat=rbind( c(0, 3), c(2,1), c(0,4) ), lhei=c(0.25, 4, 0.25 ),&#xA;      main=expression(&quot;Heat Map&quot;),&#xA;      ylab=&quot;Genes in the Microarray&quot;,&#xA;      xlab=&quot;Treatments&quot;&#xA;      )&#xA;    dev.off()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I recently discovered &lt;code&gt;hopach&lt;/code&gt; package in &lt;em&gt;Bioconductor&lt;/em&gt; which can be used to estimate the number of clusters. Previously, I was randomly assigning the number of bins for the heatmap and cutting the tree at an appropriate height to get a pre-specified number of clusters. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Possible Problems in my solution:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;I may be not clustering the genes within a particular treatment and clustering genes only across treatments or vice versa.&lt;/li&gt;&#xA;&lt;li&gt;There may be better ways of obtaining a heatmap for the pattern I want to see (similar genes within a treatment and across treatments).&lt;/li&gt;&#xA;&lt;li&gt;There may be better visualization methods which I am not aware of.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;em&gt;csgillespie&lt;/em&gt; (moderator) has a more general document on his website in which he discusses all the aspects of time course analysis (including heatmaps and clustering). I would appreciate if you can point me to an articles which describe heatmaps and clustering in detail.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I have tried the &lt;code&gt;pvclust&lt;/code&gt; package, but it complains that &lt;em&gt;M&lt;/em&gt; is singular and then it crashes.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="1307" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-10-04T09:04:46.327" LastActivityDate="2010-10-12T23:32:12.033" Title="Clustering genes in a time course experiment" Tags="&lt;r&gt;&lt;machine-learning&gt;&lt;clustering&gt;&lt;microarray&gt;" AnswerCount="5" FavoriteCount="4" />
  <row Id="3272" PostTypeId="2" ParentId="3262" CreationDate="2010-10-02T22:51:17.683" Score="4" ViewCount="" Body="&lt;p&gt;I agree that making statistics personal/relevant is important, but that's not ultimately going to dispel the fear of the student. I think how the student feels about something often has more to do with the personality of the person teaching it, and how comfortable that person feels in the classroom, even when teaching uninterested or scared students. The first thing to do to do away with their fear, is to do away with your own...you shouldn't be afraid of teaching students who may be afraid, because ultimately counseling them over their fears isn't your responsibility. You are not a therapist. And yet, by being natural, fun, casual, corny, and likable, the student will be able to let go of their fear as they can begin to replace their feelings towards statistics with their feelings about you as a person and the environment of curiosity, fun, and learning you create. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;That's my belief and experience.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, here's what I recommend:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Reframe Statistics using mantras like &quot;Information is Beautiful&quot; and show them the blog of the same name. Mention things like how &quot;measuring something allows you to manage it&quot; and make wiser decisions. Yes, these are all ways of making it personal and relevant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Introduce them to sections of Freakonomics. It's a great book, and uses regular language to describe why statistical analysis is important and sexy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tell corny jokes constantly. This endears you to them. Be a goof. Do whatever is necessary for them to feel that they are cooler than you. Do whatever is necessary for them to feel that they are smarter than you (even though they trust you secretly have everything under control). There was an article in the NYTimes some years ago about the power of a teacher who is uncool. It allows students to relax. Wear blue converse all stars, do something weird an idiosyncratic so they know they have a chance, and that they have nothing to fear.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Give them things to play with. Get some colored markers (I've done this in university) and have them draw their graphs and notes in color. This makes them feel like they're in elementary school even if they're calculating standard deviation. Major help in overcoming fear.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Get some measurement gear, measure heart rate and have them running around. Demonstrate concepts by collecting data from students live in the classroom. Make them forget it's a statistics class, make them feel like it's a study they are involved in, or administering.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Demystify the math. An intro statistics course has no actual mathematical operation more difficult than an arithmetic class, it's just a sequence of many operations in row, and it's about learning to keep track of that. Tell them it's like a yoga practice in learning to be more organized.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Memorize everyone's name on the first and second day, absolutely. Calling them by their names, poking fun at them sometimes, letting them poke fun at you, are all ways to overcome fear. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;They ultimately want to know that you aren't going to hit them over the head with something that they can't handle (that's what fear is). Give them ample warning, and exaggerate how difficult things are as they come up. Start off the class by saying &quot;Boy, you guys are going to kill me, because today is going to be so hard your head might explode,&quot; and then when you teaching them variance, let's say, and they find it easy to calculate, then they'll get a greater confidence. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When something is actually difficult to calculate, give them a whole period to do it out, and maybe a second shot the next class depending on your time constraints. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;And again, it's ultimately about you. Do you know your statistics back and forth? Does it daunt you at all? Are you a fun teacher who makes students laugh and relax, or are you bumbling and not sure how you're steering the ship? Do you have the class time well managed, or are you constantly not sure how long something will take to teach? When you need to, can you be stern with them (they are medical students after all)? &lt;/p&gt;&#xA;" OwnerUserId="1490" LastActivityDate="2010-10-02T22:51:17.683" CommunityOwnedDate="2010-10-02T22:51:17.683" CommentCount="1" />
  <row Id="3273" PostTypeId="2" ParentId="3262" CreationDate="2010-10-02T23:58:10.383" Score="1" ViewCount="" Body="&lt;p&gt;&quot;Decision making in the face of uncertainty&quot; sounds a lot more interesting than &quot;statistics&quot; even though that's essentially what statistics is about.  Maybe you could lead with the decision-making aspect to build motivation for the course.&lt;/p&gt;&#xA;" OwnerUserId="319" LastActivityDate="2010-10-02T23:58:10.383" CommunityOwnedDate="2010-10-02T23:58:10.383" />
  <row Id="3274" PostTypeId="2" ParentId="3261" CreationDate="2010-10-03T04:14:41.910" Score="2" ViewCount="" Body="&lt;p&gt;your &quot;trick&quot; seems to be the solution to the [so-called] normal equations for multiple regression - which is the usual least-squares answer in multiple regression. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;as for missing data -  what $f$ do you have in mind that knows how to get $C$ in that case?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;there are methods like imputation for filling in missing values. perhaps &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0471183865&quot; rel=&quot;nofollow&quot;&gt;little and rubin&lt;/a&gt; can give further information on the issues involved.&lt;/p&gt;&#xA;" OwnerUserId="1112" LastActivityDate="2010-10-03T04:14:41.910" CommentCount="2" />
  <row Id="3275" PostTypeId="1" AcceptedAnswerId="3357" CreationDate="2010-10-03T04:42:40.847" Score="3" ViewCount="234" Body="&lt;p&gt;Let's say I'm playing 10 &quot;games&quot;.  For each game, I know the probability of winning, the probability of tying, and the probability of losing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From these values, I can calculate the probability of winning X games, the probability of losing X games, and the probability of tying X games (for X = 0 to 10).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm just trying to figure out the probability of each outcome after playing all 10 games.  For example, what is the probability of winning 7, losing 2, and tying 1?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas? Thanks!&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Here's some example data for 2 games:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Game 1:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;win: 23.3%&lt;/li&gt;&#xA;&lt;li&gt;tie: 1.1%&lt;/li&gt;&#xA;&lt;li&gt;lose: 75.6%&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Game 2:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;win: 29.5%&lt;/li&gt;&#xA;&lt;li&gt;tie: 3.2%&lt;/li&gt;&#xA;&lt;li&gt;lose: 67.3%&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Based on this, we can calculate the probability after playing 2 games of:&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;0 wins: 54.0%&lt;/li&gt;&#xA;&lt;li&gt;1 win: 39.1%&lt;/li&gt;&#xA;&lt;li&gt;2 wins: 6.9%&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;0 ties: 95.8%&lt;/li&gt;&#xA;&lt;li&gt;1 tie: 4.2%&lt;/li&gt;&#xA;&lt;li&gt;2 ties: 0.0%&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;0 losses: 8.0%&lt;/li&gt;&#xA;&lt;li&gt;1 loss: 41.1%&lt;/li&gt;&#xA;&lt;li&gt;2 losses: 50.9%&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Based on these numbers, what is the probability of each outcome?  The possible outcomes (W-L-T) would be:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;2-0-0&lt;/li&gt;&#xA;&lt;li&gt;1-1-0&lt;/li&gt;&#xA;&lt;li&gt;1-0-1&lt;/li&gt;&#xA;&lt;li&gt;0-1-1&lt;/li&gt;&#xA;&lt;li&gt;0-2-0&lt;/li&gt;&#xA;&lt;li&gt;0-0-2&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Obviously this is pretty trivial with 2 games, but is there a generic formula for finding the probability of &lt;em&gt;W&lt;/em&gt; wins, &lt;em&gt;T&lt;/em&gt; ties, and &lt;em&gt;L&lt;/em&gt; losses after X games?&lt;/p&gt;&#xA;" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-03T16:16:35.373" LastActivityDate="2010-10-06T02:23:39.773" Title="Probability of Game Outcomes (with ties!)" Tags="&lt;probability&gt;&lt;games&gt;&lt;multinomial&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="3276" PostTypeId="1" AcceptedAnswerId="3279" CreationDate="2010-10-03T05:01:18.027" Score="5" ViewCount="139" Body="&lt;p&gt;Suppose I have some i.i.d. data $x_1, \ldots, x_n \sim N(\mu, \sigma^2)$, where $\sigma^2$ is fixed and $\mu$ is unknown, and I want to estimate $\mu$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead of simply giving the MLE of $\mu = \bar{x}$, one could estimate&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(1) $\mu = \lambda \mu_0 + (1 - \lambda) \bar{x},$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;for some &quot;prior best guess&quot; $\mu_0$. This also has a nice Bayesian interpretation: we place a prior $\mu \sim N(\mu_0, \sigma^2_0)$ on $\mu$, and $\lambda$ is the weighted precision.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I seem to recall that this also has an explicit L2 regularization-interpretation (i.e., we choose some penalty and minimize the squared loss to get the above estimate), similar to things like lasso and ridge regression, but I can't remember how it goes. Can anyone explain what the L2 regularization-interpretation of (1) is?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[More general answers, where the data isn't necessarily normal distributed, are welcome as well.]&lt;/p&gt;&#xA;" OwnerUserId="1106" LastActivityDate="2010-10-03T16:53:49.277" Title="Regularization and Mean Estimation" Tags="&lt;bayesian&gt;&lt;estimation&gt;&lt;mean&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="2" />
  <row Id="3277" PostTypeId="2" ParentId="3268" CreationDate="2010-10-03T06:35:56.690" Score="1" ViewCount="" Body="&lt;p&gt;A quick couple of thoughts:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;I've used &lt;strong&gt;multidimensional scaling&lt;/strong&gt; to visualise connections between team members (i.e., a weighted network). Nodes with stronger connections then appear closer in the figure. Here's some &lt;a href=&quot;http://www.statmethods.net/advstats/mds.html&quot; rel=&quot;nofollow&quot;&gt;resources for implementing in R&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;li&gt;You could present a standard graph where &lt;strong&gt;line thickness&lt;/strong&gt; is based on strength of connection.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-10-03T06:35:56.690" />
  <row Id="3278" PostTypeId="2" ParentId="3270" CreationDate="2010-10-03T10:14:06.133" Score="7" ViewCount="" Body="&lt;p&gt;First, there are different ways to construct so-called &lt;a href=&quot;http://en.wikipedia.org/wiki/Biplot&quot; rel=&quot;nofollow&quot;&gt;biplots&lt;/a&gt; in the case of correspondence analysis. In all cases, the basic idea is to find a way to show the best 2D approximation of the &quot;distances&quot; between row cells and column cells. In other words, we seek a hierarchy (we also speak of &quot;ordination&quot;) of the relationships between rows and columns of a contingency table.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Very briefly, CA decomposes the chi-square statistic associated with the two-way table into orthogonal factors that maximize the separation between row and column scores (i.e. the frequencies computed from the table of profiles). Here, you see that there is some connection with PCA but the measure of variance (or the metric) retained in CA is the $\chi^2$, which only depends on column profiles (As it tends to give more importance to modalities that have large marginal values, we can also re-weight the initial data, but this is another story).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a more detailed answer.&#xA;The implementation that is proposed in the &lt;code&gt;corresp()&lt;/code&gt; function (in &lt;code&gt;MASS&lt;/code&gt;) follows from a view of CA as an SVD decomposition of dummy coded matrices representing the rows and columns (such that $R^tC=N$, with $N$ the total sample). This is in light with canonical correlation analysis. &#xA;In contrast, the French school of data analysis considers CA as a variant of the PCA, where you seek the directions that maximize the &quot;inertia&quot; in the data cloud. This is done by diagonalizing the inertia matrix computed from the centered and scaled (by marginals frequencies) two-way table, and expressing row and column profiles in this new coordinate system. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you consider a table with $i=1,\dots,I$ rows, and $j=1,\dots,J$ columns, each row is weighted by its corresponding marginal sum which yields a series of conditional frequencies associated to each row: $f_{j|i}=n_{ij}/n_{i\cdot}$. The marginal column is called the &lt;em&gt;mean profile&lt;/em&gt; (for rows). This gives us a vector of coordinates, also called a &lt;em&gt;profile&lt;/em&gt; (by row). For the column, we have $f_{i|j}=n_{ij}/n_{\cdot j}$. In both cases, we will consider the $I$ row profiles (associated to their weight $f_{i\cdot}$) as individuals in the column space, and the $J$ column profiles (associated to their weight $f_{\cdot j}$) as individuals in the row space. The metric used to compute the proximity between any two individuals is the $\chi^2$ distance. For instance, between two rows $i$ and $i'$, we have&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;d^2_{\chi^2}(i,i')=\sum_{j=1}^J\frac{n}{n_{\cdot j}}\left(\frac{n_{ij}}{n_{i\cdot}}-\frac{n_{i'j}}{n_{i'\cdot}} \right)^2&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may also see the link with the $\chi^2$ statistic by noting that it is simply the distance between observed and expected counts, where expected counts (under $H_0$, independence of the two variables) are computed as $n_{i\cdot}\times n_{\cdot j}/n$ for each cell $(i,j)$. If the two variables were to be independent, the row profiles would be all equal, and identical to the corresponding marginal profile. In other words, when there is independence, your contingency table is entirely determined by its margins.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you realize an PCA on the row profiles (viewed as individuals), replacing the euclidean distance by the $\chi^2$ distance, then you get your CA. The first principal axis is the line that is the closest to all points, and the corresponding eigenvalue is the inertia explained by this dimension. You can do the same with the column profiles. It can be shown that there is a symmetry between the two approaches, and more specifically that the principal components (PC) for the column profiles are associated to the same eigenvalues than the PCs for the row profiles. What is shown on a biplot is the coordinates of the individuals in this new coordinate system, although the individuals are represented in a separate factorial space. Provided each individual/modality is well represented in its factorial space (you can look at the $\cos^2$ of the modality with the 1st principal axis, which is a measure of the correlation/association), you can even interpret the proximity between elements $i$ and $j$ of your contingency table (as can be done by looking at the residuals of your $\chi^2$ test of independence, e.g. &lt;code&gt;chisq.test(tab)$expected-chisq.test(tab)$observed&lt;/code&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The total inertia of your CA (= the sum of eigenvalues) is the $\chi^2$ statistic divided by $n$ (which is Pearson's $\phi^2$).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Actually, there are several packages that may provide you with enhanced CAs compared to the function available in the &lt;code&gt;MASS&lt;/code&gt; package: &lt;a href=&quot;http://cran.r-project.org/web/packages/ade4/index.html&quot; rel=&quot;nofollow&quot;&gt;ade4&lt;/a&gt;, &lt;a href=&quot;http://cran.r-project.org/web/packages/FactoMineR/index.html&quot; rel=&quot;nofollow&quot;&gt;FactoMineR&lt;/a&gt;, &lt;a href=&quot;http://cran.r-project.org/web/packages/anacor/index.html&quot; rel=&quot;nofollow&quot;&gt;anacor&lt;/a&gt;, and &lt;a href=&quot;http://cran.r-project.org/web/packages/ca/index.html&quot; rel=&quot;nofollow&quot;&gt;ca&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The latest is the one that was used for your particular illustration, and a paper was published in the Journal of Statistical Software that explains most of its functionnalities: &lt;a href=&quot;http://www.jstatsoft.org/v20/i03&quot; rel=&quot;nofollow&quot;&gt;Correspondence Analysis in R, with Two- and Three-dimensional Graphics: The ca Package&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, your example on eye/hair colors can be reproduced in many ways:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data(HairEyeColor)&#xA;tab &amp;lt;- apply(HairEyeColor, c(1, 2), sum) # aggregate on gender&#xA;tab&#xA;&#xA;library(MASS)&#xA;plot(corresp(tab, nf=2))&#xA;corresp(tab, nf=2)&#xA;&#xA;library(ca)&#xA;plot(ca(tab))&#xA;summary(ca(tab, nd=2))&#xA;&#xA;library(FactoMineR)&#xA;CA(tab)&#xA;CA(tab, graph=FALSE)$eig  # == summary(ca(tab))$scree[,&quot;values&quot;]&#xA;CA(tab, graph=FALSE)$row$contrib&#xA;&#xA;library(ade4)&#xA;scatter(dudi.coa(tab, scannf=FALSE, nf=2))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In all cases, what we read in the resulting biplot is basically (I limit my interpretation to the 1st axis which explained most of the inertia):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;the first axis highlights the clear opposition between light and dark hair color, and between blue and brown eyes;&lt;/li&gt;&#xA;&lt;li&gt;people with blond hair tend to also have blue eyes, and people with black hair tend to have brown eyes.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;There is a lot of additional resources on data analysis on the &lt;a href=&quot;http://pbil.univ-lyon1.fr/R/enseignement.html&quot; rel=&quot;nofollow&quot;&gt;bioinformatics lab&lt;/a&gt; from Lyon, in France. This is mostly in French, but I think it would not be too much a problem for you. The following two handouts should be interesting as a first start:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://pbil.univ-lyon1.fr/R/pdf/tdr620.pdf&quot; rel=&quot;nofollow&quot;&gt;Initiation à l'analyse factorielle des correspondances&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://pbil.univ-lyon1.fr/R/pdf/tdr62.pdf&quot; rel=&quot;nofollow&quot;&gt;Pratique de l'analyse des correspondances&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Finally, when you consider a full disjonctive (dummy) coding of $k$ variables, you get the &lt;em&gt;multiple correspondence analysis&lt;/em&gt;.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-04T20:40:05.720" LastActivityDate="2010-10-04T20:40:05.720" CommentCount="9" />
  <row Id="3279" PostTypeId="2" ParentId="3276" CreationDate="2010-10-03T11:09:10.000" Score="3" ViewCount="" Body="&lt;p&gt;Sure, it would be equivalent to the following ridge-like optimization problem:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\underset{\mu\in\mathbb{R}|\mu_0,\lambda\geq0}{\min} ||x_i-\mu-\mu_0||_2+\lambda\mu^2$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For $\lambda=0$, $\mu+\mu_0$ goes to the OLS solution (i.e. $\bar{x}$), for $\lambda=\infty$, it shrinks to $\mu_0$.&lt;/p&gt;&#xA;" OwnerUserId="603" LastEditorUserId="603" LastEditorDisplayName="" LastEditDate="2010-10-03T12:15:18.037" LastActivityDate="2010-10-03T12:15:18.037" />
  <row Id="3280" PostTypeId="2" ParentId="3276" CreationDate="2010-10-03T11:12:14.313" Score="3" ViewCount="" Body="&lt;p&gt;Ridge regression (Hoerl and Kennard, 1988) was initially developed to overcome singularities when inverting $X^tX$ (by adding $\lambda$ to its diagonal elements). Thus, the &lt;em&gt;regularization&lt;/em&gt; in this case consists in working with a vc matrix $(X^tX-\lambda I)^{-1}$. This L2 penalization leads to &quot;better&quot; predictions than with usual OLS by optimizing the compromise between bias and variance (shrinkage), but it suffers from considering all coefficients in the model. The regression coefficients are found to be&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\hat\beta=\underset{\beta}{\operatorname{argmin}}\|Y-X\beta\|^2 + \lambda\|\beta\|^2&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;with $\vert\vert\beta\vert\vert^2 = \sum_{j=1}^p\beta_j^2$ (L2-norm).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From a bayesian perspective, you can consider that the $\beta$'s must be small and plug them into a prior distribution. The likelihood $\ell (y,X,\hat\beta,\sigma^2)$ can thus be weighted by the prior probability for $\hat\beta$ (assumed i.i.d. with zero mean and variance $\tau^2$), and the posterior is found to be&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;f(\beta|y,X,\sigma^2,\tau^2)=(y-\hat\beta^tX)^t(y-\hat\beta^tX)+\frac{\sigma^2}{\tau^2}\hat\beta^t\hat\beta&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\sigma^2$ is the variance of your $y$'s. It follows that this density is the opposite of the residual sum of squares that is to be minimized in the Ridge framework, after setting $\lambda=\sigma^2/\tau^2$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The bayesian estimator for $\hat\beta$ is thus the same as the OLS one when considering the Ridge loss function with a prior variance $\tau^2$. More details can be found in &lt;em&gt;The Elements of Statistical Learning&lt;/em&gt; from Hastie, Tibshirani, and Friedman (§3.4.3, p.60 in the 1st ed.). The &lt;a href=&quot;http://www-stat.stanford.edu/~tibs/ElemStatLearn/&quot; rel=&quot;nofollow&quot;&gt;second edition&lt;/a&gt; is also available for free.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-03T16:53:49.277" LastActivityDate="2010-10-03T16:53:49.277" />
  <row Id="3281" PostTypeId="1" AcceptedAnswerId="3282" CreationDate="2010-10-03T12:58:13.803" Score="5" ViewCount="181" Body="&lt;p&gt;I am looking for books / textbooks on statistics based on fractal mathematics. I know it is not a very well known area and it is rather difficult to find good literature. Any suggestions are welcome (books, textbooks, online materials).&lt;/p&gt;&#xA;" OwnerUserId="1250" LastEditorUserId="1499" LastEditorDisplayName="" LastEditDate="2010-10-07T22:52:35.817" LastActivityDate="2010-10-07T22:52:35.817" CommunityOwnedDate="2010-10-04T12:05:20.937" Title="Statistics based on fractal mathematics" Tags="&lt;books&gt;&lt;big-list&gt;&lt;fractal&gt;" AnswerCount="1" FavoriteCount="2" />
  <row Id="3282" PostTypeId="2" ParentId="3281" CreationDate="2010-10-03T13:15:14.493" Score="3" ViewCount="" Body="&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.nsf.gov/sbe/bcs/pac/nmbs/nmbs.jsp&quot; rel=&quot;nofollow&quot;&gt;Tutorials in Contemporary Nonlinear Methods for the Behavioral Sciences Web Book&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;It has several chapters discussing fractals.&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-10-03T13:15:14.493" CommentCount="2" />
  <row Id="3283" PostTypeId="2" ParentId="3275" CreationDate="2010-10-03T13:58:51.183" Score="5" ViewCount="" Body="&lt;p&gt;Let $p_W$ be the probability of a win, $p_T$ that of a tie, and $p_L$ of a loss.  Because the probabilities sum to 1, so does their tenth power.  Expanding this out expresses the tenth power as a linear combination of the monomials $p_W^W p_T^T p_L^L$ where each term picks out the probability of $W$ wins, $T$ ties, and $L$ losses.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In more detail,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$(p_W + p_T + p_L)^{10} = \sum_{W+T+L=10} {W+T+L \choose {W \quad T \quad L}}p_W^W p_T^T p_L^L$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where the &lt;em&gt;multinomial coefficient&lt;/em&gt; is computed as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$${W+T+L \choose {W \quad T \quad L}} = \frac{(W+T+L)!}{W! T! L!}\text{.}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, the probability of 7 wins, 2 ties, and 1 loss is given by&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\Pr[W=7, T=2, L=1] = {7+2+1 \choose {7 \quad 2 \quad 1}}p_W^7 p_T^2 p_L^1$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$=\frac{10!}{7! 2! 1!}p_W^7 p_T^2 p_L^1 = 360p_W^7 p_T^2 p_L^1\text{.}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is called the &lt;a href=&quot;http://en.wikipedia.org/wiki/Multinomial_distribution&quot; rel=&quot;nofollow&quot;&gt;Multinomial Distribution&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When the probabilities vary from one game to the other, there is nothing simpler than enumerating all the possible sequences that produce a given number of wins, ties, and losses.  The multinomial coefficients count the numbers of such sequences.  The value of 360 in the example indicates there are 360 distinct ways you can arrive at 7 wins, 2 ties, and a loss.  Each one of those ways contributes its own distinct sequence of probabilities as given by each of the ten games.  To obtain a complete table of the 66 possible answers requires computing all $(1+1+1)^{10} = 59049$ products of ten probabilities.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-10-03T14:08:45.343" LastActivityDate="2010-10-03T14:08:45.343" CommentCount="6" />
  <row Id="3284" PostTypeId="2" ParentId="3271" CreationDate="2010-10-03T16:49:21.400" Score="3" ViewCount="" Body="&lt;p&gt;It seems you just want to make a fair standard analysis, so I am not a best person to answer your question; yet I would suggest you to dive deeper into Bioconductor; it has a lot of useful stuff, nevertheless finding what you want is painful. For instance &lt;a href=&quot;http://www.bioconductor.org/packages/release/bioc/html/Mfuzz.html&quot; rel=&quot;nofollow&quot;&gt;Mfuzz&lt;/a&gt; package looks promising.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-10-03T16:49:21.400" CommentCount="1" />
  <row Id="3285" PostTypeId="2" ParentId="3271" CreationDate="2010-10-03T20:34:43.010" Score="4" ViewCount="" Body="&lt;p&gt;In complement to @mbq's response (&lt;code&gt;Mfuzz&lt;/code&gt; looks fine), I'll just put some references (PDFs) about clustering of time-course gene expression data:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Futschik, ME and Charlisle, B (2005). &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.94.8014&quot; rel=&quot;nofollow&quot;&gt;Noise robust clustering of gene expression time-course data&lt;/a&gt;. &lt;em&gt;Journal of Bioinformatics and Computational Biology&lt;/em&gt;, &lt;em&gt;3(4)&lt;/em&gt;, 965-988.&lt;/li&gt;&#xA;&lt;li&gt;Luan, Y and Li, H (2003). &lt;a href=&quot;http://bioinformatics.oxfordjournals.org/content/19/4/474.long&quot; rel=&quot;nofollow&quot;&gt;Clustering of time-course gene expression data using a mixed-effects model with B-splines&lt;/a&gt;. &lt;em&gt;Bioinformatics&lt;/em&gt;, &lt;em&gt;19(4)&lt;/em&gt;, 474-482.&lt;/li&gt;&#xA;&lt;li&gt;Tai YC and Speed, TP (2006). &lt;a href=&quot;http://www.stat.berkeley.edu/~yuchuan/preprints/667.pdf&quot; rel=&quot;nofollow&quot;&gt;A multivariate empirical Bayes statistic for replicated microarray time course data&lt;/a&gt;. &lt;em&gt;The Annals of Statistics&lt;/em&gt;, &lt;em&gt;34&lt;/em&gt;, 2387–2412.&lt;/li&gt;&#xA;&lt;li&gt;Schliep, A, Steinhoff, C, and Schönhuth, A (2004). &lt;a href=&quot;http://bioinformatics.oxfordjournals.org/content/20/suppl_1/i283.full.pdf+html&quot; rel=&quot;nofollow&quot;&gt;Robust inference of groups in gene expression time-courses using mixtures of HMMs&lt;/a&gt;. &lt;em&gt;Bioinformatics&lt;/em&gt;, &lt;em&gt;20(1)&lt;/em&gt;, i283-i228.&lt;/li&gt;&#xA;&lt;li&gt;Costa, IG, de Carvalho, F, and de Souto, MCP (2004). &lt;a href=&quot;http://www.scielo.br/pdf/gmb/v27n4/22434.pdf&quot; rel=&quot;nofollow&quot;&gt;Comparative analysis of clustering methods for gene expression time course data&lt;/a&gt;. &lt;em&gt;Genetics and Molecular Biology&lt;/em&gt;, &lt;em&gt;27(4)&lt;/em&gt;, 623-631.&lt;/li&gt;&#xA;&lt;li&gt;Inoue, LYT, Neira, M, Nelson, C, Gleave, M, and Etzioni, R (2006). &lt;a href=&quot;http://biostatistics.oxfordjournals.org/content/8/3/507.short&quot; rel=&quot;nofollow&quot;&gt;Cluster-based network model for time-course gene expression data&lt;/a&gt;. &lt;em&gt;Biostatistics&lt;/em&gt;, &lt;em&gt;8(3)&lt;/em&gt;, 507-525.&lt;/li&gt;&#xA;&lt;li&gt;Phang, TL, Neville, MC, Rudolph, M, and Hunter, L (2003). &lt;a href=&quot;http://helix-web.stanford.edu/psb03/phang.pdf&quot; rel=&quot;nofollow&quot;&gt;Trajectory Clustering: A Non-Parametric Method for Grouping Gene Expression Time Courses with Applications to Mammary Development&lt;/a&gt;. &lt;em&gt;Pacific Symposium on Biocomputing&lt;/em&gt;, &lt;em&gt;8&lt;/em&gt;, 351-362.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Did you try the &lt;code&gt;timecourse&lt;/code&gt; package (as suggested by @csgillespie in his &lt;a href=&quot;http://www.mas.ncl.ac.uk/~ncsg3/microarray/Spaper.pdf&quot; rel=&quot;nofollow&quot;&gt;handout&lt;/a&gt;)?&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-03T22:59:40.033" LastActivityDate="2010-10-03T22:59:40.033" CommentCount="3" />
  <row Id="3286" PostTypeId="1" AcceptedAnswerId="3288" CreationDate="2010-10-03T21:10:14.903" Score="4" ViewCount="186" Body="&lt;p&gt;I'm not a statistician, but I sometimes need I play around with data. I have two data sets, lists of values in the unit interval. I've plotted them as histograms, so I have an intuitive idea of how &quot;far apart&quot; they are. But I want to something a little more formal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My first thought was to just sum the the differences of the values in the bins, but this isn't that satisfactory. Then I thought of taking a three bin average and sum differences over these. (Apologies if I'm mangling statistics terminology)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I was thinking I'm probably reinventing the wheel, so I came here. Similar questions seem to point to &quot;Kolmogorov Smirnov tests&quot; or something like that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my question is this: is this the right method to calculate how far these data sets are apart? And is there an easy way to do this in R? Ideally just &lt;code&gt;KStest(data1,data2)&lt;/code&gt; or something.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit&lt;/strong&gt; To emphasise, I'm particularly interested in ways to measure how far the data are apart &lt;em&gt;directly&lt;/em&gt; rather than fitting a distribution to each and then measuring the distance between distributions. [Does that even make sense? I guess numerical calculations in R will be done by sampling from a distribution anyway...]&lt;/p&gt;&#xA;" OwnerUserId="1492" LastEditorUserId="1492" LastEditorDisplayName="" LastEditDate="2010-10-04T10:28:34.273" LastActivityDate="2010-10-04T16:33:09.597" Title="Distance between empirically generated distributions (in R)" Tags="&lt;r&gt;&lt;distributions&gt;" AnswerCount="3" CommentCount="2" FavoriteCount="2" />
  <row Id="3287" PostTypeId="1" AcceptedAnswerId="3417" CreationDate="2010-10-03T21:24:06.223" Score="2" ViewCount="218" Body="&lt;p&gt;I'd like to ensure that I understand the process correctly. This is a follow-up question to &lt;a href=&quot;http://stats.stackexchange.com/questions/3270/interpreting-2d-correspondence-analysis-plots&quot;&gt;http://stats.stackexchange.com/questions/3270/interpreting-2d-correspondence-analysis-plots&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(reshape) &#xA;library(ca)&#xA;&#xA;df &amp;lt;- read.csv(file=&quot;http://www.bertelsen.ca/R/smokers.csv&quot;)&#xA;colnames(df)[7] &amp;lt;- &quot;value&quot;  ## make reshape smart&#xA;df &amp;lt;- cast(df, SMOKER ~ GEO) ## reshape data&#xA;row.names(df) &amp;lt;- df$SMOKER ## rename rows&#xA;df &amp;lt;- df[2:ncol(df)] ## reset df&#xA;df &amp;lt;- df[-4,] ## Let's only look at people who have smoked&#xA;df &amp;lt;- df[c(&quot;AB&quot;,&quot;BC&quot;,&quot;ON&quot;,&quot;QC&quot;)] ## and only the biggest 4 provinces (KISS)&#xA;plot(ca(df))&#xA;&#xA;summary(ca(df))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Output&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Principal inertias (eigenvalues):&#xA;&#xA; dim    value      %   cum%   scree plot               &#xA; 1      0.002523  99.9  99.9  *************************&#xA; 2      3e-06000   0.1 100.0                           &#xA; 3      00000000   0.0 100.0                           &#xA;        -------- -----                                 &#xA; Total: 0.002526 100.0                                 &#xA;&#xA;&#xA;Rows:&#xA;    name   mass  qlt  inr    k=1  cor ctr    k=2 cor ctr  &#xA;1 | Crrn |  265 1000  191 |  -43 1000 191 |    1   0  43 |&#xA;2 | Dlys |  201 1000  351 |  -66 1000 351 |   -1   0  70 |&#xA;3 | Frmr |  470 1000  432 |   48 1000 432 |   -1   0  98 |&#xA;4 | Occs |   65 1000   26 |   31  964  25 |    6  36 789 |&#xA;&#xA;Columns:&#xA;    name   mass  qlt  inr    k=1  cor ctr    k=2 cor ctr  &#xA;1 |   AB |  116 1000  146 |  -56 1000 146 |   -1   0  34 |&#xA;2 |   BC |  142 1000  775 |  118 1000 776 |   -1   0  41 |&#xA;3 |   ON |  434 1000    7 |   -6  909   6 |    2  91 540 |&#xA;4 |   QC |  308 1000   72 |  -24  994  72 |   -2   6 385 |&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Looking at &lt;code&gt;summary(ca(df))&lt;/code&gt; I see that nearly 100% of the inertia is described by the row profile for both modalities (Type of smoker and Province, respectively).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/ygmU7.jpg&quot; alt=&quot;CA of Smoker Types in ON, QC, AB, and BC&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What (I think) should be immediate takeaways are: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;You are more likely to be a daily smoker if you live in AB, QC, or ON&lt;/li&gt;&#xA;&lt;li&gt;You are more likely to be a former smoker if you live in BC &lt;/li&gt;&#xA;&lt;li&gt;You are least likely to be a daily smoker if you live in BC (this fits with Canadian wide understanding of BC's &quot;active lifestyle&quot; culture)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;What could we say about occasional smokers? What would your analysis tell us through this correspondence plot and it's associated summary?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data Source: Statistics Canada, Canadian Community Health Survey (CCHS 3.1), 2005. The CANSIM table 105-0427 was an update of CANSIM table &lt;a href=&quot;http://cansim2.statcan.gc.ca/cgi-win/cnsmcgi.pgm?Lang=E&amp;amp;RootDir=CII/&amp;amp;Array_Pick=1&amp;amp;ArrayId=105-0227&amp;amp;C2DB=PRD&amp;amp;ResultTemplate=CII%2FCII___&quot; rel=&quot;nofollow&quot;&gt;105-0227&lt;/a&gt;. More current data are in CANSIM tables &lt;a href=&quot;http://cansim2.statcan.gc.ca/cgi-win/cnsmcgi.pgm?Lang=E&amp;amp;RootDir=CII/&amp;amp;Array_Pick=1&amp;amp;ArrayId=105-0501&amp;amp;C2DB=PRD&amp;amp;ResultTemplate=CII%2FCII___&quot; rel=&quot;nofollow&quot;&gt;105-0501&lt;/a&gt; and &lt;a href=&quot;http://cansim2.statcan.gc.ca/cgi-win/cnsmcgi.pgm?Lang=E&amp;amp;RootDir=CII/&amp;amp;Array_Pick=1&amp;amp;ArrayId=105-0502&amp;amp;C2DB=PRD&amp;amp;ResultTemplate=CII%2FCII___&quot; rel=&quot;nofollow&quot;&gt;105-0502&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="776" LastEditorUserId="776" LastEditorDisplayName="" LastEditDate="2010-10-05T01:21:39.017" LastActivityDate="2010-10-08T16:22:10.900" Title="Interpreting 2D Correspondence Analysis Plots Part II" Tags="&lt;r&gt;&lt;correspondence&gt;" AnswerCount="1" />
  <row Id="3288" PostTypeId="2" ParentId="3286" CreationDate="2010-10-03T21:59:50.240" Score="5" ViewCount="" Body="&lt;p&gt;You can do a Kolmogorov-Smirnov test using the &lt;code&gt;ks.test&lt;/code&gt; function. See &lt;code&gt;?ks.test&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, when you are looking for a function in R (and you don't know its name) try using &lt;code&gt;??&lt;/code&gt;. For instance, &lt;code&gt;??&quot;Kolmogorov Smirnov&quot;&lt;/code&gt;. If nothing comes up &lt;code&gt;RSiteSearch(&quot;whatever you're looking for&quot;)&lt;/code&gt; should help :) &lt;/p&gt;&#xA;" OwnerUserId="582" LastActivityDate="2010-10-03T21:59:50.240" CommentCount="2" />
  <row Id="3289" PostTypeId="1" CreationDate="2010-10-03T22:28:59.097" Score="3" ViewCount="104" Body="&lt;p&gt;a recent posting on &lt;a href=&quot;http://math.stackexchange.com/questions/5922/probability-of-card-decks-being-in-the-same-order-for-n-shufflers-over-x-amount-o&quot;&gt;math.stackexchange&lt;/a&gt; reminds me of a somewhat less ambitious$^1$ question i have been meaning to ask. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;suppose 3 people each have a deck of $M$ cards numbered from 1 to $M$, in random order. together, they turn over one card at a time. on any such trial, a match occurs if at least 2 cards show the same number. let $X$ be the number of matches that occur in the $M$ trials. what is the probability distribution of $X$? [for starters, what is ${\rm P}(X=0$)?]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;an historical note: the solution for 2 players is well-known; &lt;em&gt;cf&lt;/em&gt; $ $ &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0471257087&quot; rel=&quot;nofollow&quot;&gt;Feller, vol.1,3$^{rd}$ Ed., &lt;em&gt;p&lt;/em&gt; 107&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\quad {\rm P}(X=0) := p_{0}  = 1 - 1 + \frac{1}{2!} - \frac{1}{3!} + \cdots \pm \frac{1}{M!}$, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;which, for $M$ not too small, is about 1/e. [for $M$ = 10, the two numbers already agree, on rounding, to 5 decimal places.]   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;more generally,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\quad {\rm P}(X=k) := p_{k}  = \frac{1}{k!} \{ 1 - 1 + \frac{1}{2!} - \frac{1}{3!} + \cdots \pm \frac{1}{(N-k)!}\} \sim \frac{e^{-1}}{k!}.$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[so as $M \to\infty, X$ tends in law to a poisson variable with mean 1.]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;for $k=M-1$ this gives: $p_{M-1} = 0$ [which is obvious]. also, $p_M= \frac{1}{M!}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Feller obtains these formulae using the inclusion-exclusion principle.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;$^1$ in the sense that it involves only 3 people, rather than everyone in the world, and they shuffle their decks only once, rather than repeatedly since the beginning of time.&lt;/p&gt;&#xA;" OwnerUserId="1112" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-10-04T09:07:35.723" LastActivityDate="2010-10-04T09:07:35.723" ClosedDate="2010-10-14T08:57:22.040" Title="A matching problem for 3 decks of cards" Tags="&lt;probability&gt;&lt;games&gt;" CommentCount="2" FavoriteCount="1" />
  <row Id="3290" PostTypeId="2" ParentId="3268" CreationDate="2010-10-04T02:22:18.003" Score="4" ViewCount="" Body="&lt;p&gt;What you describe in your example is not only a network of relationships, but a network of &quot;flows&quot; between all groups. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Like you suggested in a) (and as Jeromy said as well) your graphic will likely be a visualization of one group (or node) linked to other groups. Most of my knowledge of this subject is visualizing flows between geographic spaces, but many of the same issues still apply.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think this paper does a good job summarizing visualization techniques in regards to mapping flows.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://dx.doi.org/10.1016/j.compenvurbsys.2009.01.007&quot; rel=&quot;nofollow&quot;&gt;From spatial interaction data to spatial interaction information? Geovisualisation and spatial structures of migration from the 2001 UK census&lt;/a&gt;&#xA;by: Alasdair Rae&#xA;Computers, Environment and Urban Systems, Vol. 33, No. 3. (May 2009), pp. 161-178. (PDF &lt;a href=&quot;http://mediamapping.wikischolars.columbia.edu/file/view/Rae+-+2009+-+From+spatial+interaction+data+to+spatial+interacti.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Typically visualizing flows in geographic space has three main problems. One is that it is difficult to distinguish between in-flows and out-flows. The second is that long lines tend to dominate the graphic. Three is that over-lapping or too many flows tend to make the graphic look very noisy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second problem may be solved by however you organize the nodes on your graphic (like Jeromy suggested cluster nodes together with strong relationships). It may also be easier to use small multiple graphs to distinguish between in-flows, out-flows, and reciprocal flows (i.e. map your nodes to a specific space and then have seperate graphics displaying in-flows, out-flows). I have not seen any examples of flows in networks like you describe, so I do not know if the self-organizing graphics have the problem of over-lapping lines.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have experience programming in Python you may want to check out the &lt;a href=&quot;http://networkx.lanl.gov/&quot; rel=&quot;nofollow&quot;&gt;NetworkX&lt;/a&gt; package. (The Gephi package Ars linked to looks pretty awesome as well). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is similar in nature to questions brought up on the GIS stackexchange forum, and &lt;a href=&quot;http://gis.stackexchange.com/questions/778/representation-of-network-flows&quot;&gt;here&lt;/a&gt; is a question with answers you may be interested in.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good Luck&lt;/p&gt;&#xA;" OwnerUserId="1036" LastEditorUserId="1036" LastEditorDisplayName="" LastEditDate="2010-10-04T03:08:41.037" LastActivityDate="2010-10-04T03:08:41.037" />
  <row Id="3291" PostTypeId="1" CreationDate="2010-10-04T04:31:35.903" Score="7" ViewCount="249" Body="&lt;p&gt;Suppose you had $200 US to build a (very) small library of statistics books. What would your choices be? You may assume free shipping from Amazon, and any freely available texts from the internet are fair game, but assume a 5 cent charge per page to print.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(I was inspired by a mailing from Dover books, but most of their offerings seem a bit out of date)&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-10-06T12:22:32.077" CommunityOwnedDate="2010-10-04T04:31:35.903" Title="Building a Statistics Library, with Knapsack Constraint" Tags="&lt;books&gt;&lt;big-list&gt;" AnswerCount="6" CommentCount="0" FavoriteCount="1" />
  <row Id="3292" PostTypeId="2" ParentId="3291" CreationDate="2010-10-04T04:48:07.010" Score="0" ViewCount="" Body="&lt;p&gt;A tad pricey, but &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0673990850&quot; rel=&quot;nofollow&quot;&gt;Bruning and Kintz's &lt;em&gt;Computational Handbook of Statistics&lt;/em&gt;&lt;/a&gt; ($95.80) would certainly fit in your knapsack.&lt;/p&gt;&#xA;" OwnerUserId="830" LastEditorUserId="795" LastEditorDisplayName="" LastEditDate="2010-10-04T20:29:00.157" LastActivityDate="2010-10-04T20:29:00.157" CommunityOwnedDate="2010-10-04T04:48:07.010" />
  <row Id="3293" PostTypeId="2" ParentId="3291" CreationDate="2010-10-04T07:15:14.340" Score="3" ViewCount="" Body="&lt;ol&gt;&#xA;&lt;li&gt;Harrell, FE. &lt;em&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1441929185&quot; rel=&quot;nofollow&quot;&gt;Regression Modeling Strategies&lt;/a&gt;&lt;/em&gt; (Springer, 2010, 2nd ed.)&lt;/li&gt;&#xA;&lt;li&gt;Izenman, AJ. &lt;em&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0387781889&quot; rel=&quot;nofollow&quot;&gt;Modern Multivariate Statistical Techniques: Regression, Classification, and Manifold Learning&lt;/a&gt;&lt;/em&gt; (Springer, 2008)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;You should have money left to print part of The &lt;em&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/3540404643&quot; rel=&quot;nofollow&quot;&gt;Handbook of Computational Statistics&lt;/a&gt;&lt;/em&gt; (Gentle et al., Springer 2004) and &lt;em&gt;&lt;a href=&quot;http://www-stat.stanford.edu/~tibs/ElemStatLearn/&quot; rel=&quot;nofollow&quot;&gt;The Elements of Statistical Learning&lt;/a&gt;&lt;/em&gt; (Hastie et al., Springer 2009 2nd ed.) that are circulating on the web. As the latter mostly covers the same topics than Izenman's book (as pointed by &lt;a href=&quot;http://stats.stackexchange.com/users/603/kwak&quot;&gt;@kwak&lt;/a&gt;), either may be replaced by one of the &lt;a href=&quot;http://www.elsevier.com/wps/find/simple_search.cws_home?boost=true&amp;amp;needs_keyword=true&amp;amp;adv=false&amp;amp;action=simple_search&amp;amp;default=default&amp;amp;keywords=handbook+of+statistics&amp;amp;submitTopNav=Search&quot; rel=&quot;nofollow&quot;&gt;Handbook of Statistics&lt;/a&gt; published by North-Holland, depending on your field of interests.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-04T15:49:02.887" LastActivityDate="2010-10-04T15:49:02.887" CommunityOwnedDate="2010-10-04T07:15:14.340" CommentCount="4" />
  <row Id="3294" PostTypeId="1" CreationDate="2010-10-04T08:33:05.990" Score="7" ViewCount="395" Body="&lt;p&gt;I am looking for resources (tutorials, textbooks, webcast, etc) to learn about Markov Chain and HMMs. My background is as a biologist, and I'm currently involved in a bioinformatics-related project. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, what are the necessary mathematical background I need to have a sufficient understanding of Markov models &amp;amp; HMMs?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've been looking around using Google but so far I have yet to find a good introductory tutorial -- but I'm sure somebody here knows better :).&lt;/p&gt;&#xA;" OwnerUserId="1495" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-10-04T21:17:12.667" LastActivityDate="2010-10-07T17:59:07.497" CommunityOwnedDate="2010-10-04T09:13:27.440" Title="Resources for Learning Markov Chain and Hidden Markov Models" Tags="&lt;markov-process&gt;&lt;big-list&gt;&lt;tutorial&gt;&lt;hmm&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="6" />
  <row Id="3295" PostTypeId="2" ParentId="3271" CreationDate="2010-10-04T09:04:29.530" Score="2" ViewCount="" Body="&lt;p&gt;Just to add to the other answers (which look like they should solve your problem), did you try using standard clustering algorithms for you data when constructing your dendrogram? For example, &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;heatmap.2(dataset, &amp;lt;standard args&amp;gt;,&#xA;          hclustfun = function(c){hclust(c, method= 'average')}&#xA;          )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Instead of using the average distance for clustering, you can also use &quot;ward&quot;, &quot;single&quot;, &quot;median&quot;, ... See &lt;code&gt;?hclust&lt;/code&gt; for a full list.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To extract clusters, use the &lt;code&gt;hclust&lt;/code&gt; command directly and then use the &lt;code&gt;cutree&lt;/code&gt; command. For example,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;hc = hclust(dataset)&#xA;cutree(hc)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;More details can be found at my &lt;a href=&quot;http://www.mas.ncl.ac.uk/~ncsg3/microarray/&quot; rel=&quot;nofollow&quot;&gt;webpage&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="8" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-10-04T21:13:21.687" LastActivityDate="2010-10-04T21:13:21.687" CommentCount="2" />
  <row Id="3296" PostTypeId="1" AcceptedAnswerId="3308" CreationDate="2010-10-04T09:36:25.393" Score="6" ViewCount="239" Body="&lt;p&gt;I have a pretty large data set (~300 cases with ~40 continuous attributes, binary labeled) which I used to create several alternative predictive models. To do this, the set was divided to training and validation subsets (~60:40% respectively). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have noticed that there are several samples (both in the training and the validation subsets) that are being misclassified by all or most of the alternative models that I test. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suspect that there is something special about these &quot;trouble making&quot; samples. What are the general guidelines for discovering the possible reasons behind the misbehavior of the models on specific samples?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update 1&lt;/strong&gt; I'm using logistic regression for this task. The parameter selection is done by exhaustively searching combinations of up to 4 predictors with 10-fold cross valiation. It is worth mentioning that the P values that is calculated by the model for the misclassified samples are usually very different from the default classification threshold of 0.5. In other words, not only is the model wrong about those cases, it is also very confident about itself &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update 2 -- what I have already done&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I agree that insights from the study domain are crucial, but to date we have failed to discover anything significant. Also, I tried to remove the &quot;bad&quot; samples from the training set, and keeping the validation set and the parameter selection algorithm untouched. This led to better performance on the training set (naturally), but also improved significantly the performance on the validation set. Is this an indication that the &quot;bad&quot; samples were actually &quot;bad&quot;?&lt;/p&gt;&#xA;" OwnerUserId="1496" LastEditorUserId="1496" LastEditorDisplayName="" LastEditDate="2010-10-10T09:43:24.690" LastActivityDate="2010-10-12T15:02:24.580" Title="Dealing with &quot;trouble maker&quot; samples" Tags="&lt;hypothesis-testing&gt;&lt;modeling&gt;&lt;large-data&gt;&lt;outliers&gt;&lt;best-practices&gt;" AnswerCount="3" CommentCount="4" FavoriteCount="2" />
  <row Id="3297" PostTypeId="2" ParentId="3286" CreationDate="2010-10-04T09:49:33.380" Score="2" ViewCount="" Body="&lt;p&gt;A standard way to compare distributions is to use the &lt;a href=&quot;http://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence&quot; rel=&quot;nofollow&quot;&gt;Kullback-Leibler divergence&lt;/a&gt;. As usual, there's an R package that does this for you! From the &lt;code&gt;?KLdiv&lt;/code&gt; help page in the &lt;a href=&quot;http://cran.r-project.org/web/packages/flexmix/&quot; rel=&quot;nofollow&quot;&gt;&lt;code&gt;flexmix&lt;/code&gt;&lt;/a&gt; package, we get the following bit of code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;## Gaussian and Student t are much closer to each other than&#xA;## to the uniform:&#xA;&amp;gt; library(flexmix)&#xA;&amp;gt; x = seq(-3, 3, length=200)&#xA;&amp;gt; y = cbind(u=dunif(x), n=dnorm(x), t=dt(x, df=10))&#xA;&amp;gt; matplot(x, y, type=&quot;l&quot;)&#xA;&#xA;&amp;gt; round(KLdiv(y),3)&#xA;      u     n     t&#xA;u 0.000 1.082 1.108&#xA;n 4.661 0.000 0.004&#xA;t 4.686 0.005 0.000&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Notice that the comparison isn't symmetric: so uniform vs Normal is different from Normal vs Uniform.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You didn't explain why you wanted to compare distributions. Giving a use-case may get you more specific answers.&lt;/p&gt;&#xA;" OwnerUserId="8" LastActivityDate="2010-10-04T09:49:33.380" />
  <row Id="3298" PostTypeId="2" ParentId="3294" CreationDate="2010-10-04T10:16:23.660" Score="4" ViewCount="" Body="&lt;p&gt;Here are some tutorials (available as PDFs):&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Dugad and Desai, &lt;a href=&quot;http://vision.ai.uiuc.edu/dugad/hmm_tut.html&quot; rel=&quot;nofollow&quot;&gt;A tutorial on hidden markov models&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Valeria De Fonzo1, Filippo Aluffi-Pentini2 and Valerio Parisi (2007). &lt;a href=&quot;http://www.bentham.org/cbio/Samples/cbio2-1/0005CBIO.pdf&quot; rel=&quot;nofollow&quot;&gt;Hidden Markov Models in Bioinformatics&lt;/a&gt;. &lt;em&gt;Current Bioinformatics&lt;/em&gt;, &lt;em&gt;2&lt;/em&gt;, 49-61.&lt;/li&gt;&#xA;&lt;li&gt;Smith, K. &lt;a href=&quot;http://www.google.fr/url?sa=t&amp;amp;source=web&amp;amp;cd=13&amp;amp;ved=0CCcQFjACOAo&amp;amp;url=http%3A%2F%2Fciteseerx.ist.psu.edu%2Fviewdoc%2Fdownload%3Fdoi%3D10.1.1.88.351%26rep%3Drep1%26type%3Dpdf&amp;amp;rct=j&amp;amp;q=bioinformatics%20introduction%20to%20markov%20chain%20HMM&amp;amp;ei=CKmpTJL0ENmJ4gaYraihDQ&amp;amp;usg=AFQjCNFy-AqKqpkKuQGOQpO42invnaLi4A&amp;amp;sig2=MPBuL2kvm0TVAN8JrtuvEA&quot; rel=&quot;nofollow&quot;&gt;Hidden Markov Models in Bioinformatics with Application to Gene Finding in Human DNA&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Also take a look at &lt;a href=&quot;http://www.bioconductor.org&quot; rel=&quot;nofollow&quot;&gt;Bioconductor&lt;/a&gt; tutorials. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I assume you want free resources; otherwise, &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/3540241663&quot; rel=&quot;nofollow&quot;&gt;Bioinformatics&lt;/a&gt; from Polanski and Kimmel (Springer, 2007) provides a nice overview (§2.8-2.9) and applications (Part II).&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-10-04T10:16:23.660" CommunityOwnedDate="2010-10-04T10:16:23.660" />
  <row Id="3299" PostTypeId="2" ParentId="2956" CreationDate="2010-10-04T12:45:23.117" Score="3" ViewCount="" Body="&lt;p&gt;Although the reasoning about calculating the LR from the SS values is quite fair, a least-squares method is equivalent but not the same as a likelihood estimate. (The difference can be illustrated eg in the calculation of the se, which is divided by (n-1) in a least-squares approach and divided by n in a maximum-likelihood. The maximum likelihood estimate is thus consistent, but slightly biased). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This has some implications: you can calculate the LR as the likelihood is proportional to $\frac{1}{s}$, but that doesn't give you the likelihood of your anova model itself. It just tells you something about the ratio. As the AIC is classically defined in terms of the likelihood, I'm not sure if you can use the AIC as you intend.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've looked at the spreadsheet, but the values for the &quot;uncorrected LR within&quot;(I'm also not completely following what exactly you're trying to calculate there) seem highly unlikely to me. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On a side note, the power of LR testing is that you can just contrast the models you want, you don't have to do that for all of them (which lowers the multitesting error). If you do this for every term, your LR is completely equivalent to an F test, and in the case of least squares, as far as I know even numerically about the same.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your mile may vary, but I've never been confident mixing concepts of two different frameworks (i.e. least squares versus maximum likelihood). Personally, I'd report the F statistics and implement the LR in a function that allows to compare models (eg. the anova function for lme models which does exactly that).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My 2 cents.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS : I did look at your code, but couldn't really figure out all the variables. If you would annotate your code using comments, that would make life a bit easier again. The EXCEL sheet is also not the easiest to figure out. I'll check again later to see if I can make something from it.&lt;/p&gt;&#xA;" OwnerUserId="1124" LastEditorUserId="1124" LastEditorDisplayName="" LastEditDate="2010-10-04T13:04:25.893" LastActivityDate="2010-10-04T13:04:25.893" CommunityOwnedDate="2010-10-04T12:45:23.117" />
  <row Id="3300" PostTypeId="2" ParentId="3262" CreationDate="2010-10-04T13:02:30.140" Score="1" ViewCount="" Body="&lt;p&gt;I teach undergraduate biology students, and The Fear is rife among them.  I generally start by telling them three things:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Statistics is not maths, it's logic.  And if you're doing a science degree at a respected university, you eveidently don't have any problems with using logic to solve problems.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) If you can add, subtract, multiply, divide and tell whether one number is bigger than another, you can do all the maths necessary for an undergrad stats course.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) People learn differently, so if you don't understand one lecturer/textbook/explanation, ask or find another one.  (I try to give 2-3 types of explanation for a given idea where I can and tell them to remember the one that makes sense to them).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, I err on the side of visual explanations as opposed to purely verbal or mathematical ones, as this seems work for the majority of students.&lt;/p&gt;&#xA;" OwnerUserId="266" LastActivityDate="2010-10-04T13:02:30.140" CommunityOwnedDate="2010-10-04T13:02:30.140" />
  <row Id="3301" PostTypeId="2" ParentId="3291" CreationDate="2010-10-04T14:08:55.490" Score="0" ViewCount="" Body="&lt;p&gt;What topics are you interested in?  I learned from &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/007310874X&quot; rel=&quot;nofollow&quot;&gt;KNNL&lt;/a&gt; ($157.50), but oh gosh I couldn't imagine carrying this thing around -- you'd be asking for a reading list on scoliosis correction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;General Statistics&quot; is certainly an area of interest, but I'm curious if you're more interested in depth, breadth, or some mix of both.&lt;/p&gt;&#xA;" OwnerUserId="1499" LastEditorUserId="795" LastEditorDisplayName="" LastEditDate="2010-10-04T20:26:32.840" LastActivityDate="2010-10-04T20:26:32.840" CommunityOwnedDate="2010-10-04T14:08:55.490" />
  <row Id="3302" PostTypeId="2" ParentId="3291" CreationDate="2010-10-04T15:04:32.177" Score="2" ViewCount="" Body="&lt;p&gt;As a social scientist I would have to vouch for the &lt;a href=&quot;http://www.sagepub.com/booksSeries.nav?forceIgnore=True&amp;amp;sortBy=bookPublishDate&amp;amp;prodTypes=books&amp;amp;sizeBooks=&amp;amp;reverseFlag=true&amp;amp;series=Series486&amp;amp;&amp;amp;start=1&amp;amp;_requestid=381729&quot; rel=&quot;nofollow&quot;&gt;Sage Green Books&lt;/a&gt;. If you are a bargain shopper you would be able to get between 10 to 20 books for 200 dollars (assuming no shipping). For those not familiar these are all introductions to various methodologies aimed at people with no more than an introduction to statistical inference in most circumstances.&lt;/p&gt;&#xA;" OwnerUserId="1036" LastActivityDate="2010-10-04T15:04:32.177" CommunityOwnedDate="2010-10-04T15:04:32.177" />
  <row Id="3303" PostTypeId="2" ParentId="3262" CreationDate="2010-10-04T15:40:44.507" Score="3" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://www.edwardtufte.com/tufte/mosteller_p1&quot; rel=&quot;nofollow&quot;&gt;Frederick Mosteller said:&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;When I think of teaching a class, I think of five main components, not all ordinarily used in one lecture. They are&lt;/p&gt;&#xA;  &#xA;  &lt;ol&gt;&#xA;  &lt;li&gt;Large-scale application&lt;/li&gt;&#xA;  &lt;li&gt;Physical demonstration&lt;/li&gt;&#xA;  &lt;li&gt;Small-scale application (specific)&lt;/li&gt;&#xA;  &lt;li&gt;Statistical or probabilistic principle&lt;/li&gt;&#xA;  &lt;li&gt;Proof or plausibility argument&lt;/li&gt;&#xA;  &lt;/ol&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Tufte also mentioned (I don't have the source here but I think it was from Mosteller as well) the &lt;strong&gt;PGP&lt;/strong&gt; framework:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Particular&lt;/li&gt;&#xA;&lt;li&gt;General&lt;/li&gt;&#xA;&lt;li&gt;Particular&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The idea is that you should start with an example (it helps if the example is relevant to the students), then develop the general solution, then close with another example.&lt;/p&gt;&#xA;" OwnerUserId="666" LastActivityDate="2010-10-04T15:40:44.507" CommunityOwnedDate="2010-10-04T15:40:44.507" CommentCount="1" />
  <row Id="3306" PostTypeId="2" ParentId="3286" CreationDate="2010-10-04T16:33:09.597" Score="2" ViewCount="" Body="&lt;p&gt;First thing : define &quot;distance&quot;. Sounds like a stupid question, but what do you mean as distance? Is the data paired? Then -and only then- it makes sense to look at the sum of (squared) differences to decide about the distance between two datasets. If not, you have to resort to other means.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Next question is : is the data distributed in the same manner? If so, you can see the difference between the means as the &quot;location shift&quot; of your data (or the distance between both datasets).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But if neither of both is true, how do you define the distance between datasets then? Do you take shape of the distribution into account for example? You really have to think about those issues before trying to calculate a distance. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This said : One (naive) possibility is to use the mean of the differences between all possible x-y combinations. Formalized this is :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Dist=\sqrt{\frac{1}{n_1 n_2}\sum_{i=1}^{n_1} \sum_{j=1}^{n_2}(X_i - Y_j)^2}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In R :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x &amp;lt;- rnorm(10)&#xA;y &amp;lt;- rnorm(10,2)&#xA;sqrt(mean(outer(x,y,&quot;-&quot;)^2))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you allow for negative distances, you can drop the sqrt and the ^2 :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mean(outer(x,y,&quot;-&quot;))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;A simulation shows easily that this will give indeed the difference between the means in the example, as both distributions are equal in this case. But be warned that negative distances are not allowed in many applications. In the first scenario, the number will always a bit larger than the difference between the mean. In any case, if you're interested in the difference between the &lt;strong&gt;center&lt;/strong&gt; of your datasets, define the center and calculate the difference between those centers. That might very well be what you're after.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Contrary to the other suggestions, this approach does not make any assumptions about the distribution of your data. Which makes it applicable in all situations, but also difficult to interprete.&lt;/p&gt;&#xA;" OwnerUserId="1124" LastActivityDate="2010-10-04T16:33:09.597" />
  <row Id="3307" PostTypeId="1" CreationDate="2010-10-04T16:44:53.327" Score="6" ViewCount="234" Body="&lt;p&gt;I am looking for a fractal-based statistical measure which could be used as alternative to correlation between two variables (I know that hurst exponent can be used for auto-correlation).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is anyone aware of such measures?&lt;/p&gt;&#xA;" OwnerUserId="1250" LastEditorUserId="1499" LastEditorDisplayName="" LastEditDate="2010-10-07T22:52:00.950" LastActivityDate="2010-10-17T15:59:56.937" Title="Fractal alternative to correlation" Tags="&lt;correlation&gt;&lt;fractal&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="3" />
  <row Id="3308" PostTypeId="2" ParentId="3296" CreationDate="2010-10-04T17:39:16.307" Score="7" ViewCount="" Body="&lt;p&gt;I think this will require domain expertise. If I were you, I would spend time examining these samples and their provenance, to figure out what (if anything) is wrong with them. If the samples were collected by a colleague working in some application domain, they may be able to help you with this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sometimes, samples can indeed be 'bad'. For example, they could be mislabelled, collected under different circumstances from the rest, collected using off-calibration equipment, or there may be many other reasons why they're outliers. However, you shouldn't just say &quot;these are probably bad&quot; and delete them; much better to identify what's wrong with them so that you can verify that they're bad and justify their deletion.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One reason for caution is that they might not actually be bad, just drawn from part of your sample space that's not well represented in the data. In that case, you shouldn't toss them out, you should (if possible) collect more like them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another reason is that the samples at the extremities of a concept are the ones that may be hardest to classify correctly, but if they are not actually bad and you remove them, you just end up with new samples at the extremities. To take an artificial example, suppose you are classifying samples as Hot/NotHot, and everything above 50 degrees should be hot. Samples at 49.9 degrees and 50.1 degrees are quite similar even though they're different sides of your decision boundary, so they're just hard to classify and they're not outliers that should be tossed. Also, if you remove them, you may find that two new samples (49.8 and 50.2 degrees) that were previously being classified correctly are now getting misclassified.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One final point: when you say that samples in the training set are generally being misclassified, do you mean under a cross-validation scheme or literally that when you test on the training data that they are misclassified? If the latter, it could be that the classification methods you are using are not able to capture the data variance sufficiently well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps a little ... &lt;/p&gt;&#xA;" OwnerUserId="1436" LastActivityDate="2010-10-04T17:39:16.307" CommentCount="1" />
  <row Id="3309" PostTypeId="1" AcceptedAnswerId="3310" CreationDate="2010-10-04T17:43:27.613" Score="3" ViewCount="95" Body="&lt;p&gt;For example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;k &amp;lt;- 100&#xA;R &amp;lt;- 10000&#xA;max.g &amp;lt;- numeric(R)&#xA;&#xA;for(i in 1:R) max.g [i] &amp;lt;- max(rnorm(k))&#xA;&#xA;hist(max.g)  # We can see it's right tailed...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I remember once encountering that there is a name for this type of distributions, but the name alludes me.&lt;/p&gt;&#xA;" OwnerUserId="253" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-10-04T20:03:08.383" LastActivityDate="2010-10-04T20:03:08.383" Title="Is there an analytical expression for the distribution of the max of a normal k sample?" Tags="&lt;distributions&gt;&lt;maximum&gt;&lt;normal-distribution&gt;" AnswerCount="2" />
  <row Id="3310" PostTypeId="2" ParentId="3309" CreationDate="2010-10-04T17:47:51.937" Score="5" ViewCount="" Body="&lt;p&gt;Properly normalized, it's closely approximated by a &lt;a href=&quot;http://en.wikipedia.org/wiki/Gumbel_distribution&quot; rel=&quot;nofollow&quot;&gt;Gumbel distribution&lt;/a&gt; as shown by &lt;a href=&quot;http://en.wikipedia.org/wiki/Extreme_value_theory&quot; rel=&quot;nofollow&quot;&gt;Extreme value theory&lt;/a&gt;.  Alternative names are provided in the links.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-10-04T17:47:51.937" CommentCount="1" />
  <row Id="3311" PostTypeId="2" ParentId="3252" CreationDate="2010-10-04T17:56:18.403" Score="8" ViewCount="" Body="&lt;p&gt;I just drop some references about &lt;strong&gt;data dredging&lt;/strong&gt; and &lt;strong&gt;clinical studies&lt;/strong&gt; for the interested reader. This is intended to extend &lt;a href=&quot;http://stats.stackexchange.com/users/449/onestop&quot;&gt;@onestop&lt;/a&gt;'s fine answer. I tried to avoid articles focusing only on multiple comparisons or design issues, although studies with multiple endpoints continue to present challenging and controversial discussions (long after Rothman's claims about &lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pubmed/2081237?dopt=AbstractPlus&amp;amp;holding=f1000,f1000m,isrctn&quot; rel=&quot;nofollow&quot;&gt;useless adjustments&lt;/a&gt;, &lt;em&gt;Epidemiology&lt;/em&gt; 1990, 1: 43-46; or see Feise's review in &lt;em&gt;&lt;a href=&quot;http://www.biomedcentral.com/1471-2288/2/8/&quot; rel=&quot;nofollow&quot;&gt;BMC Medical Research Methodology&lt;/a&gt;&lt;/em&gt; 2002, 2:8).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My understanding is that, although I talked about &lt;em&gt;exploratory data analysis&lt;/em&gt;, my question more generally addresses the use of data mining, with its potential pitfalls, in parallel to hypothesis-driven testing.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Koh, HC and Tan, G (2005). &lt;a href=&quot;http://www.himss.org/content/files/jhim/19-2/datamining.pdf&quot; rel=&quot;nofollow&quot;&gt;Data Mining Applications in Healthcare&lt;/a&gt;. &lt;em&gt;Journal of Healthcare Information Management&lt;/em&gt;, 19(2), 64-72.&lt;/li&gt;&#xA;&lt;li&gt;Ioannidis, JPA (2005). &lt;a href=&quot;http://www.plosmedicine.org/article/info%3adoi/10.1371/journal.pmed.0020124&quot; rel=&quot;nofollow&quot;&gt;Why most published research findings are false&lt;/a&gt;. &lt;em&gt;PLoS Medicine&lt;/em&gt;, 2(8), e124.&lt;/li&gt;&#xA;&lt;li&gt;Anderson, DR, Link, WA, Johnson, DH, and Burnham, KP (2001). &lt;a href=&quot;http://digitalcommons.unl.edu/cgi/viewcontent.cgi?article=1227&amp;amp;context=usgsnpwrc&quot; rel=&quot;nofollow&quot;&gt;Suggestions for Presenting the Results of Data Analysis&lt;/a&gt;. &lt;em&gt;The Journal of Wildlife Management&lt;/em&gt;, 65(3), 373-378. -- this echoes @onestop's comment about the fact that we have to acknowledge the data-driven exploration/modeling beyond the initial set of hypotheses&lt;/li&gt;&#xA;&lt;li&gt;Michels, KB and Rosner, BA (1996). &lt;a href=&quot;http://www.cceb.med.upenn.edu/pages/courses/EPI520/2006/Michels_Rosner_1996.pdf&quot; rel=&quot;nofollow&quot;&gt;Data trawling: to fish or not to fish&lt;/a&gt;. &lt;em&gt;Lancet&lt;/em&gt;, 348, 1152-1153.&lt;/li&gt;&#xA;&lt;li&gt;Lord, SJ, Gebski, VJ, and Keech, AC (2004). &lt;a href=&quot;http://www.mja.com.au/public/issues/181_08_181004/lor10602_fm.html&quot; rel=&quot;nofollow&quot;&gt;Multiple analyses in clinical trials: sound science or data dredging?&lt;/a&gt;. &lt;em&gt;The Medical Journal of Australia&lt;/em&gt;, 181(8), 452-454.&lt;/li&gt;&#xA;&lt;li&gt;Smith, GD and Ebrahim, S (2002). &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=28DD37D54B42A126402B4A7290738FA8?doi=10.1.1.84.6814&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;Data dredging, bias, or confounding&lt;/a&gt;. &lt;em&gt;BMJ&lt;/em&gt;, 325, 1437-1438.&lt;/li&gt;&#xA;&lt;li&gt;Afshartous, D and Wolf, M (2007). &lt;a href=&quot;http://www.iew.uzh.ch/institute/people/wolf/publications/jrss-a_2007.pdf&quot; rel=&quot;nofollow&quot;&gt;Avoiding ‘data snooping’ in multilevel and mixed effects models&lt;/a&gt;. &lt;em&gt;Journal of the Royal Statistical Society A&lt;/em&gt;, 170(4), 1035–1059&lt;/li&gt;&#xA;&lt;li&gt;Anderson, DR, Burnham, KP, Gould, WR, and Cherry, S (2001). &lt;a href=&quot;http://warnercnr.colostate.edu/~anderson/PDF_files/Concerns.pdf&quot; rel=&quot;nofollow&quot;&gt;Concerns about finding effects that are actually spurious&lt;/a&gt;. &lt;em&gt;Widlife Society Bulletin&lt;/em&gt;, 29(1), 311-316.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-01T10:11:29.257" LastActivityDate="2010-11-01T10:11:29.257" CommunityOwnedDate="2010-10-19T20:50:18.963" CommentCount="2" />
  <row Id="3312" PostTypeId="2" ParentId="3291" CreationDate="2010-10-04T18:24:36.603" Score="6" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1441923225&quot; rel=&quot;nofollow&quot;&gt;All of Statistics: A Concise Course in Statistical Inference&lt;/a&gt; - US$ 79.11&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0521743850&quot; rel=&quot;nofollow&quot;&gt;Statistical Models: Theory and Practice&lt;/a&gt; - US$ 40.00&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/052168689X&quot; rel=&quot;nofollow&quot;&gt;Data Analysis Using Regression and Multilevel/Hierarchical Models&lt;/a&gt; -US$ 47.99&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Grand total of US$ 167.11&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As &lt;em&gt;chl&lt;/em&gt; suggested, you still have money left to print nearly all of &lt;a href=&quot;http://www-stat.stanford.edu/~tibs/ElemStatLearn/&quot; rel=&quot;nofollow&quot;&gt;Hastie&lt;/a&gt; (~746 pages, or $37).&lt;/p&gt;&#xA;" OwnerUserId="666" LastEditorUserId="666" LastEditorDisplayName="" LastEditDate="2010-10-05T01:56:29.007" LastActivityDate="2010-10-05T01:56:29.007" CommunityOwnedDate="2010-10-04T18:24:36.603" />
  <row Id="3313" PostTypeId="1" AcceptedAnswerId="3319" CreationDate="2010-10-04T19:47:23.197" Score="4" ViewCount="164" Body="&lt;p&gt;I have a S-Plus &lt;a href=&quot;https://home.zhaw.ch/~brw/index.html?tools&quot; rel=&quot;nofollow&quot;&gt;library&lt;/a&gt; which I'd like to convert to R. I am a programmer, but I don't know anything about S-Plus or R. From my research it seems that they are highly compatibile. Is that true? The code I want to convert only uses core S-Plus libraries.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have attached a picture of the library as seen in the S-Plus 8.0 Object Explorer. Besides function source files, there are a few entries which I'm not sure how to transpose to R. For example the last 5 (oneDay, ...), which seem to be some sort of global variables, and have specific values assigned to them. What would be the equivalent of them in R?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/102FK.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="749" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-10-04T21:15:49.663" LastActivityDate="2010-12-15T08:30:42.657" Title="How hard is to convert a library from S-PLUS 8.0 to R?" Tags="&lt;r&gt;&lt;splus&gt;" AnswerCount="3" />
  <row Id="3314" PostTypeId="2" ParentId="3309" CreationDate="2010-10-04T19:52:45.160" Score="5" ViewCount="" Body="&lt;p&gt;Hi. You will find exact expressions for the &lt;em&gt;full pdf&lt;/em&gt; of the $n^{th}$ order statistics (as a function of $n$, the sample size) in the following paper: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Percentage Points and Modes of Order Statistics from the Normal Distribution&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Shanti S. Gupta&#xA;Source: Ann. Math. Statist. Volume 32, Number 3 (1961), 888-893.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also includes exact expression for the medians and means of $n^{th}$ order statistics as a function of $n$ (i could type a few here but the paper is un-gated). Some of these expressions are surprisingly simple.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;H/T to John D Cook for the pointer.&lt;/p&gt;&#xA;" OwnerUserId="603" LastActivityDate="2010-10-04T19:52:45.160" CommentCount="2" />
  <row Id="3315" PostTypeId="2" ParentId="3313" CreationDate="2010-10-04T20:09:29.433" Score="2" ViewCount="" Body="&lt;p&gt;Your S+ functions should work in R, except if they have dependencies on some Libraries that are not currently supported on R (most of the time, it is the reverse, though). About your &quot;global&quot; variables, the easiest way is to use single assignment in your script file (although you can explicitely can write to the global environment from within a function, see &lt;code&gt;help(&quot;&amp;lt;&amp;lt;-&quot;)&lt;/code&gt;, but this is not recommended), unless you plan to make an R package. In this case, you should look at the R documentation, &lt;a href=&quot;http://cran.r-project.org/doc/manuals/R-exts.html&quot; rel=&quot;nofollow&quot;&gt;Writing R Extensions&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maybe you can also take a look at &lt;a href=&quot;http://www.omegahat.org/RinS/Docs/overview.pdf&quot; rel=&quot;nofollow&quot;&gt;Using R and S-Plus together: The Best of Both Worlds&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-10-04T20:09:29.433" />
  <row Id="3316" PostTypeId="1" CreationDate="2010-10-04T20:21:16.290" Score="7" ViewCount="346" Body="&lt;p&gt;Supposing one has a time series from which one can take various measurements such as period, maximum, minimum, average etc. and then use these to create a model sine wave with the same attributes, are there any statistical approaches one can use that could quantify how closely the actual data fit the assumed model? The number of data points in the series would range between 10 and 50 points. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A very simplistic first thought of mine was to ascribe a value to the directional movement of the sine wave, i.e. +1 +1 +1 +1 -1 -1 -1 -1 -1 -1 -1 -1 +1 +1 +1 +1, do the same to the actual data, and then somehow quantify the degree of similarity of directional movement. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: Having given more thought to what I really want to do with my data, and in light of responses to my original question, what I need is a decision making algorithm to choose between competing assumptions: namely that my data is basically linear (or trending) with noise that could possibly have cyclic elements; my data is basically cyclic with no directional trend to speak of; the data is essentially just noise; or it is transitioning between any of these states.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My thoughts now are to maybe combine some form of Bayesian analysis and Euclidean/LMS metric. The steps in this approach would be&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Create the assumed sine wave from data measurements&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Fit a LMS straight line to the data&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Derive an Euclidean or LMS metric for departures from the original data for each of the above&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Create a Bayesian prior for each based on this metric i.e. 60 % of the combined departures attach to one, 40 % to the other, hence favour the 40 %&lt;/p&gt;&#xA;&#xA;&lt;p&gt;slide a window one data point along the data and repeat the above to obtain new % metrics for this slightly changed data set - this is the new evidence - do the Bayesian analysis to create a posterior and change the probabilities that favour each assumption&lt;/p&gt;&#xA;&#xA;&lt;p&gt;repeat along the whole data set (3000+ data points) with this sliding window (window length 10-50 data points). The hope/intent is to identify the predominant/favoured assumption at any point in the data set and how this changes with time &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any comments on this potential methodology would be welcome, particularly on how I could actually implement the Bayesian analysis part.&lt;/p&gt;&#xA;" OwnerUserId="226" LastEditorUserId="226" LastEditorDisplayName="" LastEditDate="2010-10-07T01:12:03.433" LastActivityDate="2010-10-07T13:15:02.057" Title="Statistical similarity of time series" Tags="&lt;time-series&gt;&lt;classification&gt;" AnswerCount="3" FavoriteCount="2" />
  <row Id="3317" PostTypeId="2" ParentId="3200" CreationDate="2010-10-04T20:40:30.743" Score="6" ViewCount="" Body="&lt;p&gt;I think this is a very good question; it gets to the heart of the contentious multiple testing &quot;problem&quot; that plagues fields ranging from epidemiology to econometrics. After all, how &lt;em&gt;can&lt;/em&gt; we know if the significance we find is spurious or not? How true is our multivariable model?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In terms of technical approaches to offset the likelihood of publishing noise variables, I would heartily agree with 'whuber' that using part of your sample as training data and the rest as test data is a good idea. This is an approach that gets discussed in the technical literature, so if you take the time you can probably find out some good guidelines for when and how to use it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But to strike more directly at the &lt;em&gt;philosophy&lt;/em&gt; of multiple testing, I suggest you read the articles I reference below, some of which support the position that adjustment for multiple testing is often harmful (costs power), unnecessary, and may even be a logical fallacy. I for one do not automatically accept the claim that our ability to investigate one potential predictor is inexorably reduced by the investigation of another. The &lt;em&gt;family-wise&lt;/em&gt; Type 1 error rate may increase as we include more predictors in a given model, but so long as we do not go beyond the limits of our sample size, the probability of Type 1 error for each &lt;em&gt;individual&lt;/em&gt; predictor is constant; and controlling for family-wise error does not illuminate which specific variable is noise and which is not. Of course, there are cogent counter-arguments as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, as long as you limit your list of potential variables to those which are plausible (ie, would have known pathways to the outcome) then the risk of spuriousness is already handled fairly well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I would add that a &lt;strong&gt;predictive&lt;/strong&gt; model is not as concerned with the &quot;truth-value&quot; of its predictors as a &lt;strong&gt;causal&lt;/strong&gt; model; there may be a great deal of confounding in the model, but so long as we explain a large degree of the variance then we aren't too concerned. This makes the job easier, at least in one sense. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cheers,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Brenden, Biostatistical Consultant&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: you may want to do a zero-inflated Poisson regression for the data you describe, instead of two separate regressions.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Perneger T.V. &lt;a href=&quot;http://www.bmj.com/content/316/7139/1236.extract&quot; rel=&quot;nofollow&quot;&gt;What's wrong with Bonferroni adjustments&lt;/a&gt;. BMJ 1998; 316 : 1236&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="1501" LastActivityDate="2010-10-04T20:40:30.743" CommentCount="4" />
  <row Id="3318" PostTypeId="2" ParentId="175" CreationDate="2010-10-04T21:29:50.943" Score="1" ViewCount="" Body="&lt;p&gt;Sharpie, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Taking your question literally, I would argue that there are no statistical tests or rules of thumb can be used as a basis for &lt;em&gt;excluding&lt;/em&gt; outliers in linear regression analysis (as opposed to determining whether or not a given observation is an outlier). This must come from subject-area knowledge. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think the best way to start is to ask whether the outliers even make sense, especially given the other variables you've collected. For example, is it really reasonable that you have a 600 pound woman in your study, which recruited from various sports injury clinics? Or, isn't it strange that a person is listing 55 years or professional experience when they're only 60 years old? And so forth. Hopefully, you then have a reasonable basis for either throwing them out or getting the data compilers to double-check the records for you. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would also suggest robust regression methods and the transparent reporting of dropped observations, as suggested by Rob and Chris respectively.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps,&#xA;Brenden&lt;/p&gt;&#xA;" OwnerUserId="1501" LastActivityDate="2010-10-04T21:29:50.943" />
  <row Id="3319" PostTypeId="2" ParentId="3313" CreationDate="2010-10-04T21:31:19.277" Score="3" ViewCount="" Body="&lt;p&gt;I would say that your functions will &lt;em&gt;mostly&lt;/em&gt; work in R, but that is far from being universally true.  I went through the same conversion before myself, and it was actually quite painful.  Especially given all the functions that are actually writing in a foreign language (like C).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an example: it looks like you have some S-Plus &lt;code&gt;timeSeries&lt;/code&gt; objects in there.  It can be painful to convert functions that use or rely on the timeSeries object in S-Plus, and you will have a slightly rude awakening because the R time series libraries simply aren't as standardized or as user friendly.  This is not to criticize the R libraries, but simply to point out that S-Plus was created by one company and there is only one &lt;code&gt;timeSeries&lt;/code&gt; object and everything uses it consistently.  &lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-10-04T21:31:19.277" CommentCount="1" />
  <row Id="3320" PostTypeId="2" ParentId="3151" CreationDate="2010-10-04T21:48:25.543" Score="2" ViewCount="" Body="&lt;p&gt;I wrote a script called Roux about a year ago which does this. I wanted to be able to create HTML transcripts from running an R script, including any images, without having to change the script.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You call Roux from the command line, like this:&#xA;roux example.R&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and roux will:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;run the script in R (requiring the Roux package first automatically)&lt;/li&gt;&#xA;&lt;li&gt;syntax highlight the .Rout output using Pygments&lt;/li&gt;&#xA;&lt;li&gt;insert images in the correct location&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;the Roux R package is a very small R package which modifies plot() and some other functions to automatically write to a random filename rather than the default interactive graphics device.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have used this a lot, and it works really well for me, although I'm sure if more people use it with new packages then minor issues will arise, most likely that you'll have a different function which generates a graph and Roux won't know that it should open a PNG device for you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since speaking with Tal about this I have updated and improved the code, and it's now up here:&#xA;&lt;a href=&quot;http://bitbucket.org/ananelson/roux/src&quot; rel=&quot;nofollow&quot;&gt;http://bitbucket.org/ananelson/roux/src&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;so if you run into any issues, please report them to the issue tracker there on Bitbucket.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have added support for LaTeX transcripts so you can easily create PDFs which have the transcript of your R script including images. (You can see an example if you look in the example-output directory, find the &quot;raw&quot; link to download it.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You do need to have Python and the Pygments python library intalled. If you have an older version of Python and run into any issues, please let me know.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wrote about Roux on my blog but didn't publicize it that much because my efforts have been focused on a new project called Dexy which is intended as a replacement for Sweave. If you want more flexibility and control or are interested in literate documentation then you might want to check out Dexy too.&lt;/p&gt;&#xA;" LastActivityDate="2010-10-04T21:48:25.543" />
  <row Id="3321" PostTypeId="1" CreationDate="2010-10-04T21:57:45.903" Score="5" ViewCount="387" Body="&lt;p&gt;I am looking to rank data that, in some cases, the larger value has the rank of 1.  I am relatively new to R, but I don't see how I can adjust this setting in the rank function.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x &amp;lt;- c(23,45,12,67,34,89)&#xA;rank(x)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;generates:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[1] 2 4 1 5 3 6&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;when I want it to be:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[1] 5 3 6 2 4 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I assume this is very basic, but any  help you can provide will be very much appreciated.&lt;/p&gt;&#xA;" OwnerUserId="569" LastActivityDate="2010-10-04T22:43:34.287" Title="Rank in R - descending order" Tags="&lt;r&gt;" AnswerCount="1" />
  <row Id="3322" PostTypeId="2" ParentId="3321" CreationDate="2010-10-04T22:43:23.163" Score="11" ViewCount="" Body="&lt;p&gt;You could negate &lt;code&gt;x&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; rank(-x)&#xA;[1] 5 3 6 2 4 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1390" LastActivityDate="2010-10-04T22:43:23.163" CommentCount="1" />
  <row Id="3324" PostTypeId="1" CreationDate="2010-10-04T22:44:57.123" Score="9" ViewCount="906" Body="&lt;p&gt;I have inherited some data analysis code that, not being an econometrician, I am struggling to understand. One model runs an instrumental variables regression with the following STATA command&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ivreg my_dv var1 var2 var3 (L.my_dv = D2.my_dv D3.my_dv D4.my_dv)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This dataset is a panel with multiple sequential observations for this set of variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why is this code using the lagged values of the DV as instruments? As I understand it (from digging into an old textbook), IV estimation is used when there is a problem because of a regressor being correlated with the error term. However, nothing is mentioned of choosing lags of the DV as instruments. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A comment on this line of the code mentions &quot;causality&quot;. Any help in figuring out what was the goal here would be most welcome.&lt;/p&gt;&#xA;" OwnerUserId="1007" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-10-05T17:02:50.580" LastActivityDate="2010-10-07T08:45:27.100" Title="Why use a lagged DV as an instrumental variable?" Tags="&lt;regression&gt;&lt;stata&gt;&lt;instrumental-variables&gt;" AnswerCount="3" CommentCount="2" FavoriteCount="1" />
  <row Id="3325" PostTypeId="2" ParentId="3316" CreationDate="2010-10-05T00:45:41.003" Score="7" ViewCount="" Body="&lt;p&gt;The Euclidean distance is a common metric in machine learning.  The following slides provide a good overview of this area along with references:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.cs.ucr.edu/~eamonn/sdm04slides_lite.ppt&quot; rel=&quot;nofollow&quot;&gt;Making Time-series Classification More Accurate Using Learned Constraints&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://timemachine.iic.harvard.edu/site_media/i/umaatutorial.pdf&quot; rel=&quot;nofollow&quot;&gt;Introduction to Machine Learning Research on Time Series&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Also see the references on Keogh's benchmarks page for time series classification:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.cs.ucr.edu/~eamonn/time_series_data/&quot; rel=&quot;nofollow&quot;&gt;UCR Time Series Classification/Clustering Page&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-10-05T00:45:41.003" />
  <row Id="3326" PostTypeId="2" ParentId="3109" CreationDate="2010-10-05T03:20:46.547" Score="2" ViewCount="" Body="&lt;p&gt;I believe that your use of the word &quot;tracking&quot; may be leading your web queries off topic.  Are you interested in time-series?  Could you provide a simple example of &quot;tracks clusters of datapoints&quot;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using the word &quot;tracks&quot; do you mean over time, through space, or perhaps in reference to a chemical or physical process?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Further, with &quot;clusters of datapoints&quot;, are you talking about objects (image recognition), are you talking about survey data &amp;amp; searching for trends, or something more akin to tracking the correlated movements of stock prices?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Answer:&lt;/strong&gt; You might consider googling &quot;Spatial Temporal Cluster Algorithms&quot; or perhaps &quot;Agent based modeling&quot; or &quot;Spatio-temporal point processes&quot;.  But again, the suggestions can be refined with a bit of insight on your desired application or context.&lt;/p&gt;&#xA;" OwnerUserId="1499" LastActivityDate="2010-10-05T03:20:46.547" CommentCount="1" />
  <row Id="3327" PostTypeId="2" ParentId="3316" CreationDate="2010-10-05T03:34:51.777" Score="5" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;If you have a specific model you wish to compare against:&lt;/strong&gt;  I would recommend Least-squares as a metric to minimize and score possible parameter values against a specific dataset.  All you basically have to do is plug in your parameter estimates, use those to generate predicted values, and compute the average squared deviation from the true values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;However, You might consider turning your question around slightly:&lt;/strong&gt; &quot;Which model would best fit my data?&quot;  In which case I would suggest making an assumption of a normally distributed error term ~ something one could argue is akin to the least squares assumption.  Then, &lt;em&gt;depending on your choice of model,&lt;/em&gt; you could make an assumption about how you think the other model parameters are distributed (assigning a Bayesian prior) and the use something like the MCMC package from R to sample from the distribution of the parameters.  Then you could look at posterior means &amp;amp; variances to get an idea of which model has the best fit.&lt;/p&gt;&#xA;" OwnerUserId="1499" LastActivityDate="2010-10-05T03:34:51.777" CommentCount="3" />
  <row Id="3328" PostTypeId="1" AcceptedAnswerId="3334" CreationDate="2010-10-05T04:00:11.767" Score="5" ViewCount="183" Body="&lt;p&gt;&lt;strong&gt;Question:&lt;/strong&gt; With a 10 dimensional  MCMC chain, let's say I'm prepared to hand you a matrix of the draws: 100,000 iterations (rows) by 10 parameters (columns), how best can I identify the posterior modes?  I'm especially concerned with multiple modes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Background:&lt;/strong&gt; I consider myself a computationally savvy statistician, but when a colleague asked me this question, I was ashamed that I couldn't come up with a reasonable answer.  The primary concern is that multiple modes may appear, but only if at least eight or so of the ten dimensions are considered.  My first thought would be to use a kernel density estimate, but a search through R revealed nothing promising for problems of greater than three dimensions.  The colleague has proposed an ad-hoc binning strategy in ten dimensions and searching for a maximum, but my concern is that bandwidth might either lead to significant sparsity problems or to a lack of resolution to discern multiple modes.  That said, I'd happily accept suggestions for automated bandwidth suggestions, links to a 10 kernel density estimator, or anything else which you know about.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Concerns:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;We believe that the distribution may be quite skewed; hence, we wish to identify the posterior mode(s) and not the posterior means.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;We are concerned that there may be several posterior modes.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If possible, we'd prefer an R based suggestion.  But any algorithm will do as long as it isn't incredibly difficult to implement.  I guess I'd prefer not to implement a N-d kernel density estimator with automated bandwidth selection from scratch.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="1499" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-01-17T23:59:07.313" LastActivityDate="2011-04-18T19:47:30.297" Title="Given a 10D MCMC chain, how can I determine its posterior mode(s) in R?" Tags="&lt;r&gt;&lt;bayesian&gt;&lt;mcmc&gt;&lt;k-nearest-neighbour&gt;&lt;mode&gt;" AnswerCount="4" />
  <row Id="3329" PostTypeId="2" ParentId="2635" CreationDate="2010-10-05T04:15:12.117" Score="1" ViewCount="" Body="&lt;p&gt;A Gamma distribution is not a conjugate prior for a Gamma distribution.  There is a conjugate prior for the Gamma distribution developed by Miller (1980) whose details you can find on &lt;a href=&quot;http://en.wikipedia.org/wiki/Conjugate_prior&quot; rel=&quot;nofollow&quot;&gt;Wikipedia&lt;/a&gt; and also in the pdf linked in footnote 6.  Checkout section 3.2 on page 25 of &lt;a href=&quot;http://www.stat.columbia.edu/~cook/movabletype/mlm/CONJINTRnew%2BTEX.pdf&quot; rel=&quot;nofollow&quot;&gt;this pdf&lt;/a&gt;, there is a prior with four parameters: p, q, r, &amp;amp; s&lt;/p&gt;&#xA;" OwnerUserId="1499" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-11-04T04:53:50.283" LastActivityDate="2010-11-04T04:53:50.283" CommentCount="0" />
  <row Id="3330" PostTypeId="2" ParentId="3324" CreationDate="2010-10-05T05:28:39.143" Score="4" ViewCount="" Body="&lt;p&gt;I don't know Stata, so I can't comment on the specific model.  But the use of lagged variables is a fairly common approach when dealing with simultaneity bias in general and creating instrumental variables in particular.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say you have a feedback between two variables in your model: the independent variable (such as price) and the dependent variable (such as quantity).  Then both are endogeneous (their causes arise from within the model) and perturbations to the error term will affect &lt;em&gt;both&lt;/em&gt; variables.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To solve this, you want to make the independent variable (price) exogeneous so that perturbations in the error affect only the dependent variable (quantity).  This is accomplished by creating new exogeneous variables by regressing the other exogeneous variables in your model on price.  These new exogeneous variables are your instrumental variables (IVs).  The IVs are derived from exogeneous terms and thus not correlated with the error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But to do this, you need to figure out which variables are exogeneous so they can be used to derive the IVs.  We can note that lagged variables &quot;occurred&quot; in the past and thus can't be correlated with the error in the present.  Lagged variables are thus exogeneous and become convenient candidates for deriving IVs.  (However, note that the preceding argument fails when the errors are autocorrelated.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A good introduction and reference to this is &lt;a href=&quot;http://books.google.com/books?id=64vt5TDBNLwC&quot; rel=&quot;nofollow&quot;&gt;&lt;em&gt;Introductory econometrics: a modern approach&lt;/em&gt;&lt;/a&gt; by Wooldridge.&lt;/p&gt;&#xA;" OwnerUserId="251" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-10-05T05:39:47.353" LastActivityDate="2010-10-05T05:39:47.353" />
  <row Id="3331" PostTypeId="1" AcceptedAnswerId="3363" CreationDate="2010-10-05T07:45:19.550" Score="7" ViewCount="473" Body="&lt;p&gt;I have sales data for a series of outlets, and want to categorise them based on the shape of their curves over time.  The data looks roughly like this (but obviously isn't random, and has some missing data):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;n.quarters &amp;lt;- 100&#xA;n.stores &amp;lt;- 20&#xA;if (exists(&quot;test.data&quot;)){&#xA;  rm(test.data)&#xA;}&#xA;for (i in 1:n.stores){&#xA;  interval &amp;lt;- runif(1, 1, 200)&#xA;  new.df &amp;lt;- data.frame(              &#xA;    var0 = interval + c(0, cumsum(runif(49, -5, 5))),&#xA;    date = seq.Date(as.Date(&quot;1990-03-30&quot;), by=&quot;3 month&quot;, length.out=n.quarters),&#xA;    store = rep(paste(&quot;Store&quot;, i, sep=&quot;&quot;), n.quarters))&#xA;  if (exists(&quot;test.data&quot;)){&#xA;    test.data &amp;lt;- rbind(test.data, new.df)    &#xA;  } else {&#xA;    test.data &amp;lt;- new.df&#xA;  }&#xA;}&#xA;test.data$store &amp;lt;- factor(test.data$store)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I would like to know how I can cluster based on the &lt;em&gt;shape&lt;/em&gt; of the curves in R.  I had considered the following approach:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Create a new column by linearly transforming each store's var0 to a value between 0.0 and 1.0 for the entire time series.&lt;/li&gt;&#xA;&lt;li&gt;Cluster these transformed curves using the &lt;a href=&quot;http://cran.r-project.org/web/packages/kml/index.html&quot; rel=&quot;nofollow&quot;&gt;&lt;code&gt;kml&lt;/code&gt; package&lt;/a&gt; in R.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I have two questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Is this a reasonable exploratory approach?&lt;/li&gt;&#xA;&lt;li&gt;How can I transform my data into the longitudinal data format that &lt;code&gt;kml&lt;/code&gt; will understand?  Any R snippets would be much appreciated!&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="179" LastEditorUserId="179" LastEditorDisplayName="" LastEditDate="2010-10-05T23:05:43.947" LastActivityDate="2010-10-07T17:45:59.027" Title="Is it possible to do time-series clustering based on curve shape?" Tags="&lt;r&gt;&lt;time-series&gt;&lt;clustering&gt;" AnswerCount="2" CommentCount="5" FavoriteCount="5" />
  <row Id="3332" PostTypeId="2" ParentId="3328" CreationDate="2010-10-05T07:54:19.357" Score="1" ViewCount="" Body="&lt;p&gt;Have you considered 'PRIM / bump hunting' ? (see e.g. Section 9.3. of 'The Elements of Statistical Learning' by Tibshirani et al. or ask your favourite search engine). Not sure whether that's implemented in R though.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[ As far as I understood are you trying to find the mode of the probability density from which your 100'000 rows are drawn. So your problem would be partially solved by finding an appropriate &lt;code&gt;density estimation&lt;/code&gt; method ].&lt;/p&gt;&#xA;" OwnerUserId="961" LastActivityDate="2010-10-05T07:54:19.357" CommentCount="1" />
  <row Id="3333" PostTypeId="1" CreationDate="2010-10-05T07:54:48.897" Score="2" ViewCount="266" Body="&lt;p&gt;I asked a question earlier on &lt;a href=&quot;http://stats.stackexchange.com/questions/2483/how-to-model-and-test-a-decision-support-system-e-g-a-terrorist-warning-system&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Essentially, I am trying to evaluate a warning system that consists of a light bulb (of a specific color) being switched on, to indicate a predicted warning threat level. Srikant came up with a simple solution which involved calculating the posterior probabilities of the warning system, using bayes theorem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am now contemplating implementing this in R. However, I am not quite sure how to proceed, since I am an R newbie.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;warning light, event level occured?&#xA;red, yes&#xA;none, yes&#xA;green, no&#xA;&#xA;etc ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I would like to write a simple R script that will help me to calculate:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;a). the posterior probablity of an event occuring, for a given state of the light bulb&#xA;b). a C.I to attach to the posterior probability obtained in (a) above&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since I am new to R, I would be grateful for the steps (and commands) required to do the above. My data will be in a simple csv file in the format described, so I can simply scan() it into R. How to proceed from there, is what I need help with..&lt;/p&gt;&#xA;" OwnerUserId="1216" LastActivityDate="2010-10-05T19:41:51.290" Title="Calculating posterior probabilities in R?" Tags="&lt;r&gt;&lt;bayesian&gt;" AnswerCount="2" />
  <row Id="3334" PostTypeId="2" ParentId="3328" CreationDate="2010-10-05T07:58:19.880" Score="6" ViewCount="" Body="&lt;p&gt;Have you considered using a nearest neighbour approach ? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;e.g. building a list of the &lt;code&gt;k&lt;/code&gt; nearest neighbours for each of the 100'000 points and then consider the data point with the smallest distance of the &lt;code&gt;kth&lt;/code&gt; neighbour a mode. In other words: find the point with the 'smallest bubble' containing &lt;code&gt;k&lt;/code&gt; other points around this point.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not sure how robust this is and the choice for &lt;code&gt;k&lt;/code&gt; is obviously influencing the results.&lt;/p&gt;&#xA;" OwnerUserId="961" LastActivityDate="2010-10-05T07:58:19.880" CommentCount="2" />
  <row Id="3335" PostTypeId="2" ParentId="3328" CreationDate="2010-10-05T09:39:19.830" Score="2" ViewCount="" Body="&lt;p&gt;This is only a partial answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I recently used &lt;a href=&quot;http://www.umiacs.umd.edu/~morariu/figtree/&quot; rel=&quot;nofollow&quot;&gt;figtree&lt;/a&gt; for multidimensional kernel density estimates. It's a C package and I got it to work fairly easily. However, I only used it to estimate the density at particular points, not calculate summary statistics.&lt;/p&gt;&#xA;" OwnerUserId="8" LastActivityDate="2010-10-05T09:39:19.830" />
  <row Id="3336" PostTypeId="2" ParentId="3324" CreationDate="2010-10-05T10:31:48.563" Score="7" ViewCount="" Body="&lt;p&gt;Edit: Given the clarification on the stata code provided by Andy W below, i changed my answer to better adress the question. You will find the old version of my answer below the current one. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems your code is a clumsy attempt at DIY'ing the Arellano-Bond estimator (assuming ivreg estimates with 2SOLS). You can find more details on the use and logic of the A/B estimator &lt;a href=&quot;http://www.cemmap.ac.uk/wps/cwp0209.pdf&quot; rel=&quot;nofollow&quot;&gt;in this nice review paper&lt;/a&gt; as well as in &lt;a href=&quot;http://www.nyu.edu/its/pubs/connect/fall03/pdfs/yaffee_primer.pdf&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; broader introduction. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a nutshell and within 3 lines: although the A/B estimator is indeed an (generalized) IV estimator, it is not used to address any issue of causality. The IV's in this context are used to provide efficient estimation of the AR coefficient in the context of panel data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would recommend against re-inventing the wheel here, and instead using ready made toolbox to perform such estimations. For stata, you can use the &lt;a href=&quot;http://ideas.repec.org/c/boc/bocode/s435901.html&quot; rel=&quot;nofollow&quot;&gt;XTABOND2&lt;/a&gt; (or XTABOND if you are running STAT11) package.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;old response:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A simple example will help you here. Suppose you have two variable $x_t$ and $y_t$ sampled over time such that the correlation between $x_t$ and $y_t$ is very high. You would like to make a claim about $x_t$ causing $y_t$ but unfortunately there is a very good competing and credible theory under which $y_t$ causes $x_t$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To disentangle the two competing models, you regress $y_t$ on $x_{t-1}$ (instead of $x_t$). Often, you will lose in precision (i.e. the correlation between variables sampled at different times is usually lower than the correlation between variable sampled simultaneously). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The way the two competing models - $y_t\leftarrow x_{t-1}$ and $x_{t-1} \leftarrow y_{t}$ - are now disentangled is that, presumably, there is not a good theory under which an $x$ from one period ago can be caused by a current $y$ ('the past cannot be caused by the future'), excluding the second sense of causality.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that the use of this trick is only valid if both variables (the $y_t$ and $x_{t-1}$ are stationary $I(0)$).&lt;/p&gt;&#xA;" OwnerUserId="603" LastEditorUserId="603" LastEditorDisplayName="" LastEditDate="2010-10-07T08:45:27.100" LastActivityDate="2010-10-07T08:45:27.100" CommentCount="3" />
  <row Id="3337" PostTypeId="1" CreationDate="2010-10-05T10:47:46.470" Score="4" ViewCount="236" Body="&lt;p&gt;Hi there! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;After reading &lt;a href=&quot;http://robjhyndman.com/researchtips/crossvalidation&quot; rel=&quot;nofollow&quot;&gt;one of the &quot;Research tips&quot;&lt;/a&gt; of R.J. Hyndman about cross-validation and time series, I came back to an old question of mine that I'll try to formulate here. The idea is that in classification or regression problems, the ordering of the data is not important, and hence &lt;em&gt;k&lt;/em&gt;-fold cross-validation can be used. On the other hand, in time series, the ordering of the data is obviously of a great importance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, when using a machine learning model to forecast time series, a common strategy is to reshape the series {y_1, ..., y_T} into a set of &quot;input-output vectors&quot; which, for a time t, have the form (y_{t-n+1}, ..., y_{t-1}, y_{t}; y_{t+1}), sorry for the latex-sy notation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, once this reshaping has been done, can we consider that the resulting set of &quot;input-output vectors&quot; do not need to be ordered? If we use, for example, a feed-forward neural network with n inputs to &quot;learn&quot; these data, we would arrive at the same results no matter the order in which we show the vectors to the model. And therefore, could we use k-fold cross-validation the standard way, without the need to re-fit the model each time?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sorry if this seems as a silly question or if it is already answered in the site (I haven't been able to find it).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks! &lt;/p&gt;&#xA;" OwnerUserId="1504" LastActivityDate="2010-10-05T21:16:56.510" Title="Ordering of time series for machine learning" Tags="&lt;time-series&gt;&lt;machine-learning&gt;&lt;cross-validation&gt;" AnswerCount="1" FavoriteCount="4" />
  <row Id="3338" PostTypeId="1" CreationDate="2010-10-05T11:28:09.010" Score="4" ViewCount="72" Body="&lt;p&gt;I'm using a genetic algorithm to generate a string that produces certain results I map into a line/bar graph. I'm trying to rate how closely the results produced by the genetic string compares to a graph I have produce manually by hand.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently I am comparing the graphs be finding the area of difference between the two and getting a percentage representing where it sits between 0 difference and a worst case difference being where they don't overlap at all.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Though I don't feel this rating really represents how closely the shape matches the other. As this is what concerns me the most, as two different results could have the same difference in terms of area but one may be a closer shape then the other, and I would like the one with the closer shape to have a higher rating!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is, is there a standard way of rating graphs where i can get a percentage of how closely one matches the other and can you provide the way of going about this?&lt;/p&gt;&#xA;" OwnerUserId="1505" LastActivityDate="2010-10-05T11:51:35.757" Title="Rating how closely one graph models another" Tags="&lt;data-visualization&gt;&lt;model-comparison&gt;&lt;genetic-algorithms&gt;" AnswerCount="1" />
  <row Id="3339" PostTypeId="2" ParentId="3338" CreationDate="2010-10-05T11:51:35.757" Score="2" ViewCount="" Body="&lt;p&gt;You question is very closely related to this &lt;a href=&quot;http://stats.stackexchange.com/questions/3286/distance-between-empirically-generated-distributions-in-r&quot;&gt;question&lt;/a&gt; about differences between distributions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Have a look at the three answers. That should give you some R code to get started. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;BTW, I don't think your question is duplicate. It's a just a question that shares answers.&lt;/p&gt;&#xA;" OwnerUserId="8" LastActivityDate="2010-10-05T11:51:35.757" CommentCount="3" />
  <row Id="3340" PostTypeId="2" ParentId="2348" CreationDate="2010-10-05T12:11:53.590" Score="1" ViewCount="" Body="&lt;p&gt;I use a slice sampler - originally proposed by Neal(2003), which I tune through heuristic optimization.&lt;/p&gt;&#xA;" OwnerUserId="1499" LastActivityDate="2010-10-05T12:11:53.590" />
  <row Id="3341" PostTypeId="2" ParentId="725" CreationDate="2010-10-05T12:18:20.193" Score="3" ViewCount="" Body="&lt;p&gt;Not an R package, but D. A. Landgrebe from Purdue (author of &lt;em&gt;Signal theory methods in multispectral remote sensing&lt;/em&gt;) has sponsored the &lt;a href=&quot;https://engineering.purdue.edu/~biehl/MultiSpec/&quot; rel=&quot;nofollow&quot;&gt;MultiSpec&lt;/a&gt; freeware.  Its a rather clunky GUI but gets the job done for most of the common hyperspectral algorithms.&lt;/p&gt;&#xA;" OwnerUserId="179" LastActivityDate="2010-10-05T12:18:20.193" CommentCount="1" />
  <row Id="3342" PostTypeId="1" AcceptedAnswerId="3343" CreationDate="2010-10-05T12:32:27.393" Score="3" ViewCount="322" Body="&lt;p&gt;I have some data that I've built histograms out of in R. Now I want to play around with the data, but first I want to summarise it in the same way the histogram does. That is, I want to take my data vector, and count how many points fall in each bin interval in the same way that R's &lt;code&gt;hist&lt;/code&gt; function does.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was about to do this from scratch, but then I thought: R already knows how to do this, I just need to find out how to take the first half of the &lt;code&gt;hist&lt;/code&gt; function and just run that. So how do I do this?&lt;/p&gt;&#xA;" OwnerUserId="1492" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-10-06T14:31:44.757" LastActivityDate="2010-10-06T14:31:44.757" Title="Recreating R's hist function's bin counting" Tags="&lt;r&gt;&lt;histogram&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="3343" PostTypeId="2" ParentId="3342" CreationDate="2010-10-05T12:38:30.410" Score="6" ViewCount="" Body="&lt;p&gt;Perhaps I've misunderstood what you want, but &lt;code&gt;hist()&lt;/code&gt; returns all the details required to produce the histogram that is plotted. But you don't need to plot it and you can capture the returned object for subsequent use. So if the histogram contains the relevant summary you are after, this should be all you need. Here's an example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; h &amp;lt;- hist(islands, plot = FALSE)&#xA;&amp;gt; str(h)&#xA;List of 7&#xA; $ breaks     : num [1:10] 0 2000 4000 6000 8000 10000 12000 14000 16000 18000&#xA; $ counts     : int [1:9] 41 2 1 1 1 1 0 0 1&#xA; $ intensities: num [1:9] 4.27e-04 2.08e-05 1.04e-05 1.04e-05 1.04e-05 ...&#xA; $ density    : num [1:9] 4.27e-04 2.08e-05 1.04e-05 1.04e-05 1.04e-05 ...&#xA; $ mids       : num [1:9] 1000 3000 5000 7000 9000 11000 13000 15000 17000&#xA; $ xname      : chr &quot;islands&quot;&#xA; $ equidist   : logi TRUE&#xA; - attr(*, &quot;class&quot;)= chr &quot;histogram&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Note the use of &lt;code&gt;plot = FALSE&lt;/code&gt; to avoid the superfluous plot side-effect.&lt;/p&gt;&#xA;" OwnerUserId="1390" LastActivityDate="2010-10-05T12:38:30.410" CommentCount="1" />
  <row Id="3345" PostTypeId="2" ParentId="3333" CreationDate="2010-10-05T13:21:50.150" Score="1" ViewCount="" Body="&lt;p&gt;By the tone of your question, it may be handy to read a little bit about generating draws from a posterior. Two introductory springer books (full of R codes) comes to mind (in order of preference): &lt;/p&gt;&#xA;&#xA;&lt;p&gt;a) Bayesian computation with R.&#xA;Springer, 2007. J. Albert.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;b) Introducing Monte Carlo Methods with R.&#xA;Springer 2010. C. P. Robert, G. Casella&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You will find more detailed explanations and ways to circumvent important captcha's in these that you could possibly get from a 15 lines answer on here.&lt;/p&gt;&#xA;" OwnerUserId="603" LastActivityDate="2010-10-05T13:21:50.150" CommentCount="1" />
  <row Id="3346" PostTypeId="2" ParentId="125" CreationDate="2010-10-05T13:54:13.333" Score="1" ViewCount="" Body="&lt;p&gt;I simply must to include &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0412055511&quot; rel=&quot;nofollow&quot;&gt;MCMC in Practice&lt;/a&gt;.  It provides an excellent introduction to MCMC, perhaps not as general as other books mentioned, but excellent for gaining insight and intuition.  I would recommend reading it after (or in parallel with) &lt;a href=&quot;http://bayes.bgsu.edu/bcwr/&quot; rel=&quot;nofollow&quot;&gt;Bayesian Computation with R&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1499" LastActivityDate="2010-10-05T13:54:13.333" CommunityOwnedDate="2010-10-05T13:54:13.333" />
  <row Id="3347" PostTypeId="2" ParentId="1904" CreationDate="2010-10-05T15:08:21.867" Score="0" ViewCount="" Body="&lt;p&gt;Though it hasn't been around for decades, I'm currently looking forward to &lt;a href=&quot;http://madison.byu.edu/mcmski/&quot; rel=&quot;nofollow&quot;&gt;MCMSki&lt;/a&gt;.  One could also mention the &lt;a href=&quot;http://www.uv.es/valenciameeting&quot; rel=&quot;nofollow&quot;&gt;Valencia Meetings&lt;/a&gt;, but those only happen every four years (and you've already missed the 2010 meeting).&lt;/p&gt;&#xA;" OwnerUserId="1499" LastActivityDate="2010-10-05T15:08:21.867" CommunityOwnedDate="2010-10-05T15:08:21.867" CommentCount="3" />
  <row Id="3348" PostTypeId="2" ParentId="3199" CreationDate="2010-10-05T17:06:35.777" Score="3" ViewCount="" Body="&lt;p&gt;You could also consider &lt;a href=&quot;http://en.wikipedia.org/wiki/Moran%27s_I&quot; rel=&quot;nofollow&quot;&gt;Moran's I&lt;/a&gt; which is available in the R package &quot;ape&quot;.  And then simply use a weighting based on distance:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;nRows &amp;lt;- 30;&#xA;nCols &amp;lt;- 15;&#xA;&#xA;nPixels &amp;lt;- nRows * nCols;&#xA;&#xA;# Create a Random Image&#xA;image = matrix(0, nrow=nRows, ncol=nCols);&#xA;&#xA;for ( I in 1:nRows )&#xA;  {&#xA;    for ( J in 1:nCols )&#xA;      {&#xA;        image[I,J] &amp;lt;- sample.int(256,1) - 1;&#xA;      } &#xA;  }&#xA;&#xA;# 1D to 2D Index Function&#xA;reverseIndex &amp;lt;- function ( vectorIdx, nRows, nCols )&#xA;  {&#xA;    # If you're using row major for some odd reason, you'll&#xA;    # need to flip these.&#xA;&#xA;    J &amp;lt;- floor((vectorIdx-1) / nCols);&#xA;    I &amp;lt;- (vectorIdx-1) - nCols*J;&#xA;&#xA;    return (c(I+1,J+1));&#xA;  } &#xA;&#xA;# Distance Function&#xA;distFunc &amp;lt;- function(I,J)&#xA;  {&#xA;    idx1 &amp;lt;- reverseIndex(I, nRows, nCols);&#xA;    idx2 &amp;lt;- reverseIndex(J, nRows, nCols);&#xA;&#xA;    return (sqrt((idx1 - idx2) %*% (idx1 - idx2)));&#xA;  }&#xA;&#xA;# Create Distance Matrix&#xA;dist = matrix(0, nrow=nPixels, ncol=nPixels);&#xA;&#xA;for ( J in 1:nPixels )&#xA;  {&#xA;    dist[,J] &amp;lt;- sapply(1:nPixels, distFunc, J);&#xA;  }&#xA;&#xA;# Invert Distance for Moran's I&#xA;invDist = 1.0 / dist;&#xA;diag(invDist) = 0.0;&#xA;&#xA;# Load Library&#xA;library(ape);&#xA;&#xA;# Compute Moran's I:&#xA;Moran.I(as.vector(image), dist);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;R guru's feel free to weigh in on my code - I'm sure there are better / faster implementations.  I know that for loops are insanely slow in R.  Note that this will simply provide a measure &amp;amp; test of association, it will not identify &lt;em&gt;where&lt;/em&gt; that association is in your matrix.&lt;/p&gt;&#xA;" OwnerUserId="1499" LastActivityDate="2010-10-05T17:06:35.777" CommentCount="2" />
  <row Id="3349" PostTypeId="2" ParentId="3064" CreationDate="2010-10-05T17:08:52.073" Score="2" ViewCount="" Body="&lt;p&gt;This is not a complete answer but I hope it gives you some ideas as to how to model the situation in a coherent manner. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Assumptions&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;The values at the lower end of the scale follow a normal distribution truncated from below.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The values at the upper end of the scale follow a normal distribution truncated from above.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Note: I know that you said that the data is not normal but I am assuming that you are referring to the distribution of all the values whereas the above assumptions pertain to the values at the lower and the upper end of the scale.)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;A person's underlying state (whether they have TB or not) follows a first-order markov chain. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Model&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;$D_i(t)$ be 1 if at time $t$ the $i^\mbox{th}$ person has TB and 0 otherwise,&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;$RTB_i(t)$ be the test response to the TB test at time $t$ of the the $i^\mbox{th}$ person, &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;$RN_i(t)$ be the test response to the NILL test at time $t$ of the the $i^\mbox{th}$ person, &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;$f(RN_i(t) | D_i(t)=0) \sim N(\mu_l,\sigma_l^2) I(RN_i(t) &gt; R_l)$&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;$f(RN_i(t) | D_i(t)=1) \sim N(\mu_l,\sigma_l^2) I(RN_i(t) &gt; R_l)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Points 4 and 5 capture the idea that a person's response to the NILL test is not dependent on disease status.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;$f(RTB_i(t) | D_i(t)=0) \sim N(\mu_l,\sigma_l^2) I(RTB_i(t) &gt; R_l)$&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;$f(RTB_i(t) | D_i(t)=1) \sim N(\mu_u,\sigma_u^2) I(RTB_i(t) &amp;lt; R_u)$&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;$\mu_u &gt; \mu_l$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Points 6, 7 and 8 capture the idea that a person's response to the TB test &lt;em&gt;is dependent&lt;/em&gt; on disease status.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;$p(t)$ be the probability that a person's catches TB during the 6 months preceding time $t$ given that they were disease free during the previous test period. Thus, the state transition matrix would like the one below:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\begin{bmatrix}&#xA;1-p(t) &amp;amp;  p(t) \\ &#xA;0 &amp;amp;  1&#xA;\end{bmatrix}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Prob(D_i(t)=1 | D_i(t-1) = 0) = p(t)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Prob(D_i(t)=0 | D_i(t-1) = 0) = 1-p(t)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Prob(D_i(t)=1 | D_i(t-1) = 1) = 1$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Prob(D_i(t)=0 | D_i(t-1) = 1) = 0$&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Your test criteria states that:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\hat{D}_i(t) = &#xA;\begin{cases} &#xA; 1, &amp;amp; RTB_i(t) - RN_i(t) \ge 0.35 \\&#xA; 0, &amp;amp; otherwise &#xA;\end{cases}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, as you see from the structure of the model you can actually parameterize the cut-offs and change the whole problem to that of what should be your cut-offs to accurately diagnose patients. Thus, the wobbler problem seems to be more an issue with your choice of cut-offs rather than anything else.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to choose the 'right' cut-offs, you can take historical data about patients definitively identified as having TB and estimate the resulting parameters of the above setup. You could use some criteria such as number of patients correctly classified as having TB or not as a metric to identify the 'best' model. For simplicity, you could assume that $p(t)$ to be a time invariant parameter which seems reasonable in the absence of epidemics etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope that is useful.&lt;/p&gt;&#xA;" LastActivityDate="2010-10-06T23:46:47.417" CommentCount="1" />
  <row Id="3350" PostTypeId="2" ParentId="2611" CreationDate="2010-10-05T17:34:51.493" Score="2" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Disclaimer:&lt;/strong&gt; This idea might be foolish &amp;amp;  I'm not going to pretend to understand the theoretical implications of what I'm proposing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;&lt;strong&gt;Suggestion&lt;/strong&gt;&quot;  &lt;strong&gt;:&lt;/strong&gt;  Why don't you simply impute 100 (I know you normally do 5) datasets, run the lme4 or nmle, get the confidence intervals (you have 100 of them) and then:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using a small interval width (say range / 1000 or something), test over the range of possible values of each parameter and include only those small intervals which appear in at least 95 of the 100 CIs.  You would then have a Monte Carlo &quot;average&quot; of your confidence intervals.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm sure there are issues (or perhaps theoretical problems) with this approach.  For instance, you &lt;em&gt;could&lt;/em&gt; end up with a set of &lt;em&gt;disjoint&lt;/em&gt; intervals.  This may or may not be a bad thing depending on your field. Note that this is only possible if you have at least two &lt;em&gt;completely non-overlapping&lt;/em&gt; confidence intervals which are separated by a region with less than 95% coverage.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might also consider something closer to the Bayesian treatment of missing data to get a posterior credible region which &lt;em&gt;would certainly be better formed &amp;amp; more theoretically support&lt;/em&gt; than my ad-hoc suggestion.&lt;/p&gt;&#xA;" OwnerUserId="1499" LastActivityDate="2010-10-05T17:34:51.493" />
  <row Id="3351" PostTypeId="2" ParentId="3333" CreationDate="2010-10-05T19:41:51.290" Score="0" ViewCount="" Body="&lt;p&gt;This should actually be a fairly simple operation to do in R. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;bulbs &amp;lt;- read.csv(&quot;bulbs.csv&quot;)&#xA;bulbs &amp;lt;- table(bulbs)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now just sum along the rows or columns to get the marginal values.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sum(bulbs[,1]) # counts of no event occurring&#xA;sum(bulbs[1,]) # counts of green light&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now just take these frequencies and plug them to get any posterior you want.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;bulbs[1,1]/sum(bulbs[,1]) # probability of green light given no event occurred&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1392" LastActivityDate="2010-10-05T19:41:51.290" CommentCount="0" />
  <row Id="3352" PostTypeId="1" AcceptedAnswerId="3353" CreationDate="2010-10-05T20:18:07.143" Score="3" ViewCount="167" Body="&lt;p&gt;If $X$ is a categorical variable, and I am interested in the posterior distributions of $\beta_1$, where $\beta_1$ is a vector of coefficients, one for each level of X, are these equivalent models?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Model 1:&#xA;$$Y \sim ( \beta_0 + \beta_1X_1,\sigma^2)$$&#xA;$$\beta_1 \sim N(0,\tau)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Model 2:&#xA;$$Y \sim(\beta_1X_1,\sigma^2)$$&#xA;$$\beta_1 \sim N(\beta_0,\tau)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In both models, $\beta_0$ has an informed prior, $\tau$ has a weak prior:&#xA;$$\beta_0\sim \text{Gamma}(6,3)$$&#xA;$$\tau \sim \text{Gamma}(0.1,0.1)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If these are not equivalent, why not?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I ask because although I thought these would be the same, but when implemented in JAGS, the MCMC chain for $\beta_0$ from the first model has much higher autocorrelation than the second.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have put my R code here &lt;a href=&quot;https://netfiles.uiuc.edu/dlebauer/www/SO.test.tgz&quot; rel=&quot;nofollow&quot;&gt;SO.test.tgz&lt;/a&gt;&#xA;It can be run from the command line, with rjags installed, thus:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;wget https://netfiles.uiuc.edu/dlebauer/www/SO.test.tgz&#xA;tar -zxf SO.test.tgz&#xA;R --vanilla &amp;lt; SO.test.R&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1381" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-06T11:18:19.023" LastActivityDate="2010-10-13T11:46:32.707" Title="Are these equivalent representations of the same hierarchical Bayesian model?" Tags="&lt;modeling&gt;&lt;bayesian&gt;&lt;multilevel-analysis&gt;" AnswerCount="3" FavoriteCount="1" />
  <row Id="3353" PostTypeId="2" ParentId="3352" CreationDate="2010-10-05T20:37:30.390" Score="4" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Updated Response:&lt;/strong&gt;  You still don't have a full specification for model #2.  However, I can sort of guess what you mean -- correct me if I'm wrong. The trouble is that the statements $Y = \beta_1 X$ &amp;amp; $Y = \beta_0 + \beta_1 X$ are not probabilistic.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;[&lt;/strong&gt; Aside: In a mathematical sense, you're defining a set of linear equations.  If you have only one datapoint then you can solve $Y = \beta_1 X$ for the only value of $\beta_1$ which satisfies this equation.  If you have more than one value of $Y$ and $X_1$ then this represents an overconstrained system -- without a solution. &lt;strong&gt;]&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My hunch is that you mean the following: $$Y \sim N( \beta_1 X_1, \sigma^2)$$ for some value of $\sigma^2$ which is assumed to be known -- (or perhaps you also want to estimate it).  This places your random effects model within the context of a standard linear mixed effects model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we then assume that $Y$ is distributed as I have stated, then:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Answer:&lt;/strong&gt; No, they are not equal.  In the first model, $\beta_0$ is essentially an intercept (Imagine multiplying $\beta_0$ by $X_0$ where $X_0$ is always equal to 1).  In the second model, $\beta_0$ represents a random offset from $\beta_1$.  Note that this model (#2) wouldn't make sense to someone who doesn't do Bayes...  It would be identical to running a linear regression where you have two predictors that are perfectly multi-collinear, but since you've made distributional assumptions within a Bayesian model, you &lt;em&gt;can&lt;/em&gt; estimate it.  That said, I'm not sure you &lt;em&gt;should&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note:&lt;/strong&gt; You can run the first model without the added distributional assumption (which I included above more for model #2). However, I've never seen this sort of specification.  I think it would be identical to stating that $(Y - \beta_1 X_1) \sim Gamma(6,3)$.  In other words, an error term which is distributed as a gamma random variable.  My hunch is that you instead want a random intercept model (with normally distributed errors).  If that be the case, use model #1, don't use model #2.  Yes $\beta_0$ and $\beta_1$ will exhibit autocorrelation &lt;strong&gt;until you standardize the levels of your categorical variable -- make sure that the summation across all individuals is equal to zero&lt;/strong&gt; (which you can do by subtracting the mean of $X_1$ from every single observation of $X_1$).&lt;/p&gt;&#xA;" OwnerUserId="1499" LastEditorUserId="1499" LastEditorDisplayName="" LastEditDate="2010-10-06T03:49:19.403" LastActivityDate="2010-10-06T03:49:19.403" CommentCount="1" />
  <row Id="3354" PostTypeId="2" ParentId="3337" CreationDate="2010-10-05T21:16:56.510" Score="1" ViewCount="" Body="&lt;p&gt;Interesting question!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The approach you describe is certainly very widely used by people using standard ML methods that require fixed-length feature vectors of attributes, to analyse time series data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the post that you link to, Hyndman points out that there are correlations between the reshaped data vectors (samples). This could be problematic, as k-CV (or other evaluation methods that divide data at random into training and testing sets) assumes that all samples are independent. However, I don't think this concern is relevant for the case of a standard ML methods, that treat attributes separately. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For explanation, let me simplify your notation by assuming n=3, so the the first few data vectors (labelled alphabetically) will be:&#xA;A: (y_1, y_2, y_3: y_4)&#xA;B: (y_2, y_3, y_4: y_5)&#xA;B: (y_3, y_4, y_5: y_6)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Clearly, A and B have terms such as y_2 in common. But, for A, this is the value of its second attribute whereas for B this is the value of its first attribute.&lt;/p&gt;&#xA;" OwnerUserId="1436" LastActivityDate="2010-10-05T21:16:56.510" />
  <row Id="3355" PostTypeId="2" ParentId="3352" CreationDate="2010-10-05T21:56:23.063" Score="3" ViewCount="" Body="&lt;p&gt;Hello David,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The only similarity in the two models is the general type of models they belong to, otherwise they are not similar in general as pointed out by M. Tibbits.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Both these models belong the class of hierarchical models with varying slope (cf &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/052168689X&quot; rel=&quot;nofollow&quot;&gt;Gelman and Hill 2006&lt;/a&gt; for detailed treatment)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The answer for &quot;why not&quot; are many and one of them is pointed out by M. Tibbits. A few more are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;In the model 2 the slope on an average is 6/3, where as it is 0 for model 1.(This description can be made much better if we had data but it is &quot;almost&quot; accurate given the limited description)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;You would expect the data in model 1 to be distributed randomly along a approximately horizontal line where as in model 2 you would expect data to be randomly distributed along a line of slope approx. beta_0 and &lt;em&gt;zero&lt;/em&gt; intercept.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;These answers can be best varified if you similated the data based on your hierarchical setting and look at the results of the analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;S.  &lt;/p&gt;&#xA;" OwnerUserId="1307" LastActivityDate="2010-10-05T21:56:23.063" CommentCount="1" />
  <row Id="3356" PostTypeId="2" ParentId="3324" CreationDate="2010-10-06T02:22:05.297" Score="5" ViewCount="" Body="&lt;p&gt;For those not familiar with the following code snippet from Stata the OP provided&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;ivreg my_dv var1 var2 var3 (L.my_dv = D2.my_dv D3.my_dv D4.my_dv)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;this equation can be read as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Y_t = \alpha + \beta_1 (Var1) + \beta_2 (Var1) + \beta_3 (Var1) + \beta_4 (\tilde{Y}_{t-1})$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\tilde{Y}_{t-1}$ is estimated by &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\tilde{Y}_{t-1} = \alpha + Z_1(\Delta^{2}Y_t) + Z_2(\Delta^{3}Y_t) + Z_3(\Delta^{4}Y_t)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(i.e. the first stage of the IV equation is within the parenthesis in the Stata code)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The deltas represent second, third, and fourth order differences, and they are used as excluded instruments to estimate the lag of the dependent variable. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In Stata code, the &lt;code&gt;L.&lt;/code&gt; indicates lagging that variable by $t-1$, and &lt;code&gt;D.&lt;/code&gt; signifies first order differences of that variable, and hence &lt;code&gt;D2.&lt;/code&gt; signifies second order differencing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Intially I could not think of any logical reasoning why someone would do this. But Kwak pointed out (referencing this &lt;a href=&quot;http://www.cemmap.ac.uk/wps/cwp0209.pdf&quot; rel=&quot;nofollow&quot;&gt;paper&lt;/a&gt;) that the Arellano-Bond method uses the differences as instruments to estimate the auto-regressive component of the model. (Also intially I had assumed that the differences would only have an effect if the series is non-stationary, which Bond states in that linked paper the differences will only be weak instruments in the case the series is a random walk, on pg. 21)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As suggestions on further reading material as introductions to instrumental variables,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another poster in this &lt;a href=&quot;http://stats.stackexchange.com/questions/563/what-is-an-instrumental-variable/709#709&quot;&gt;response&lt;/a&gt; (Charlie) linked to some &lt;a href=&quot;http://cgibbons.berkeley.edu/Courses/ECON141_S10/IV.pdf&quot; rel=&quot;nofollow&quot;&gt;slides&lt;/a&gt; he prepared that I like and would suggest are worth looking into for an intro to instrumental variables. I would also suggest this &lt;a href=&quot;http://csda.albany.edu/colloquium/IVEstimation050509_CSDA.ppt&quot; rel=&quot;nofollow&quot;&gt;powerpoint&lt;/a&gt; a professor of mine prepared for a workshop as an introduction as well. As a last suggestion for anyone instrested in learning more about instrumental variables you should look up the work of Joshua Angrist.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is my initial answer&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;While I agree with everything that Kwak and ars have stated, I still can not think of any reason why someone would use the differences of the &lt;em&gt;dependent&lt;/em&gt; variable as instruments to estimate the lag of the dependent variable (if people do not know Stata code, the &lt;code&gt;L.&lt;/code&gt; indicates lagging that variable by $t-1$, and &lt;code&gt;D.&lt;/code&gt; signifies first order differences of that variable, and hence &lt;code&gt;D2.&lt;/code&gt; signifies second order differencing).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In all applications I have seen, people use the lag of &lt;em&gt;independent&lt;/em&gt; variables as instruments to estimate the lag of the dependent variable (for reasons ars talks about). But this is based on the assumption that the lagged independent variables are exogenous to the error term in the time period they are being applied. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I do not know of any reasoning in which the differences of the dependent variable would be considered exogenous. As far as I'm aware it is not accepted practice to difference only one side of the equation, and would produce rather illogical results (&lt;a href=&quot;http://www.griffith.edu.au/__data/assets/pdf_file/0017/88100/Greenberg-2001.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; is a paper that critiques someone about the reverse situation in which they included a variables level as a predictor of a differenced series.) If you rearrange the terms in the IV equation it actually looks similar to an augmented Dickey Fuller test.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While the simplest answer would be to ask the person who wrote the code, can anybody give an example in which this procedure would be acceptable, or any situation in which this procedure would return some meaningful results? As is I can not think of any logical reasoning why the differences would have an effect on the levels except in the case the series is non-stationary.&lt;/p&gt;&#xA;" OwnerUserId="1036" LastEditorUserId="1036" LastEditorDisplayName="" LastEditDate="2010-10-07T03:14:40.720" LastActivityDate="2010-10-07T03:14:40.720" CommentCount="4" />
  <row Id="3357" PostTypeId="2" ParentId="3275" CreationDate="2010-10-06T02:23:39.773" Score="0" ViewCount="" Body="&lt;p&gt;I've figured out how to do this in O(n^3) time (where n is the number of games):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Use a 4-dimensional array: &#xA;   probabilityOfOutcome[gamesLookedAt][wins][losses][ties]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Initially, we have &quot;looked at&quot; zero games.  After looking at zero games, there is a 100% probability of being 0-0-0, so set:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;probabilityOfOutcome[0][0][0][0] = 1;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Look at&quot; one additional game at a time, basing each outcome probability on the last step:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;for every outcome in probabilityOfOutcome[g-1]:  &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;consider the probability of winning&#xA;game g&lt;/li&gt;&#xA;&lt;li&gt;consider the probability of losing&#xA;game g&lt;/li&gt;&#xA;&lt;li&gt;consider the probability of tying&#xA;game g&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;For g games, there will be (g^2 + 3g + 2)/2 possible outcomes, and you will have to iterate over g games... therefore O(g^3) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can go into more detail with the for-loop if people still need clarification (it is actually a triple nested for-loop in practice).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;:-)&lt;/p&gt;&#xA;" LastActivityDate="2010-10-06T02:23:39.773" CommentCount="1" />
  <row Id="3358" PostTypeId="2" ParentId="2611" CreationDate="2010-10-06T09:35:15.300" Score="4" ViewCount="" Body="&lt;p&gt;Repeated comment from above:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;i'm not sure that a proper analytical solution to this problem even exists. I've looked at some additional literature, but this problem is elegantly overlooked everywhere. I've also noticed that Yucel &amp;amp; Demirtas (in the article i mentioned, page 798) write:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;These multiply imputed datasets were&#xA;  used to estimate the model […] using&#xA;  the R package &lt;code&gt;lme4&lt;/code&gt; leading to 10&#xA;  sets of (beta, se(beta)), (sigma_b,&#xA;  se(sigma_b)) which were then combined&#xA;  using the MI combining rules defined&#xA;  by Rubin.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It seems they used some kind of shortcut to estimate the SE of the variance component (which is, of course, inappropriate, since the CI is asymmetrical) and then applied the classic formula.&lt;/p&gt;&#xA;" OwnerUserId="1266" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-06T14:31:47.217" LastActivityDate="2010-10-06T14:31:47.217" CommentCount="1" />
  <row Id="3359" PostTypeId="1" AcceptedAnswerId="3360" CreationDate="2010-10-06T11:52:36.327" Score="4" ViewCount="179" Body="&lt;p&gt;I am looking at trigger efficiencies, meaning that I have some device that fires on $k$ out of $n$ events. In the end I am interested in some estimate of the efficiency $\epsilon$ which is the probability to fire on a randomly given event. Using a Bayesian approach with a uniform prior over $[0,1]$ I can model the probability distribution for $\epsilon$ as a Beta distribution $\beta(\epsilon; k+1, n-k+1)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now comes the question: I calculate the efficiency using &quot;bootstrapping&quot; which means that the final trigger efficiency is the product of two trigger efficiencies, both of which can be modelled as Beta distributions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I calculate this product of the two Beta PDFs for large values of $k_{1,2}$ and $n_{1,2}$ efficiently? Is there a closed form of the product (AFAIK not)? At the moment I am doing this numerically, but this is rather slow.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://stats.stackexchange.com/questions/2066/how-can-i-numerically-approximate-values-for-a-beta-distribution-with-large-alp&quot;&gt;This question&lt;/a&gt; has the answer how to evaluate integrals of the Beta distribution for large argument values, but this does not help here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope my question is clear and not completely stupid...&lt;/p&gt;&#xA;" OwnerUserId="1512" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-10-06T14:23:03.013" LastActivityDate="2010-10-06T14:23:03.013" Title="Product of beta distributions" Tags="&lt;distributions&gt;&lt;beta&gt;" AnswerCount="1" />
  <row Id="3360" PostTypeId="2" ParentId="3359" CreationDate="2010-10-06T12:19:12.380" Score="2" ViewCount="" Body="&lt;p&gt;According to the abstract of &lt;a href=&quot;http://scitation.aip.org/getabs/servlet/GetabsServlet?prog=normal&amp;amp;id=SMJMAP000018000004000721000001&amp;amp;idtype=cvips&amp;amp;gifs=yes&amp;amp;ref=no&quot; rel=&quot;nofollow&quot;&gt;this paper&lt;/a&gt;, &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The density function of products of&#xA;  random beta variables is a Meijer&#xA;  $G$-function which is expressible in&#xA;  closed form when the parameters are&#xA;  integers.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;However, I imagine the closed form requires a great deal of combinatorial calculation and hence would not be practically useful.  The slow numerical algorithm you mentioned is probably faster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This paper may be more useful since it does not require integer parameters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.springerlink.com/content/j3w2789gn8374234/&quot; rel=&quot;nofollow&quot;&gt;The distribution of product of independent beta random variables with application to multivariate analysis&lt;/a&gt;  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I haven't read the paper, but the abstract sounds promising.&lt;/p&gt;&#xA;" OwnerUserId="319" LastActivityDate="2010-10-06T12:19:12.380" CommentCount="2" />
  <row Id="3361" PostTypeId="2" ParentId="3291" CreationDate="2010-10-06T12:22:32.077" Score="4" ViewCount="" Body="&lt;p&gt;You might want to spend $1.20 printing out Matthias Vallentin's &lt;a href=&quot;http://www.cs.berkeley.edu/~mavam/dl/probstat.pdf&quot; rel=&quot;nofollow&quot;&gt;probability and statistics cheat sheet&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="319" LastActivityDate="2010-10-06T12:22:32.077" CommunityOwnedDate="2010-10-06T12:22:32.077" CommentCount="2" />
  <row Id="3362" PostTypeId="1" AcceptedAnswerId="3375" CreationDate="2010-10-06T13:27:08.537" Score="5" ViewCount="178" Body="&lt;p&gt;Hi!  I'd like to assess the impact of an upcoming policy implementation, as measured by changes in questionnaire response to a Likert-scale question.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand I could use a difference-in-difference approach.  However, in my situation there is no single obvious comparison, non-treated population.  I think I'd like to use the &quot;Synthetic Control Method for Comparative Case Studies&quot; as described by Abadie et al and implemented as Synth in R.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Alberto Abadie, Alexis Diamond, Jens Hainmueller. Journal of the American Statistical Association. June 1, 2010, 105(490): 493-505. doi:10.1198/jasa.2009.ap08746.  &lt;a href=&quot;http://courses.gov.harvard.edu/gov3009/spring07/hainmueller.pdf&quot; rel=&quot;nofollow&quot;&gt;full text&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;As summarized in the R help for synth: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;synth estimates the effect of an&#xA;  intervention of interest by comparing&#xA;  the evolution of an aggregate outcome&#xA;  for a unit affected by the&#xA;  intervention to the evolution of the&#xA;  same aggregate outcome for a synthetic&#xA;  control group.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;synth constructs this synthetic&#xA;  control group by searching for a&#xA;  weighted combination of control units&#xA;  chosen to approximate the unit&#xA;  affected by the intervention in terms&#xA;  of the outcome predictors. The&#xA;  evolution of the outcome for the&#xA;  resulting synthetic control group is&#xA;  an estimate of the counterfactual of&#xA;  what would have been observed for the&#xA;  affected unit in the absence of the&#xA;  intervention. [..] the synth function&#xA;  routinely searches for the set of&#xA;  weights that generate the best fitting&#xA;  convex combination of the control&#xA;  units. In other words, the predictor&#xA;  weight matrix V is chosen among all&#xA;  positive definite diagonal matrices&#xA;  such that MSPE is minimized for the&#xA;  pre-intervention period.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;See also the useful summary by Srikant Vadali in answers below.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this method appropriate for survey/sampled data?  Is there anything I need to do differently, or just use my Likert-response mean as the dependent variable?  Any suggestions about how I'd power such a beast?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you!&lt;/p&gt;&#xA;" OwnerUserId="644" LastEditorUserId="644" LastEditorDisplayName="" LastEditDate="2010-10-07T12:59:02.717" LastActivityDate="2010-10-07T12:59:02.717" Title="Can I use Synthetic Control Method for Comparative Case Studies with survey data?" Tags="&lt;econometrics&gt;&lt;survey&gt;&lt;causal-inference&gt;&lt;panel-data&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="3363" PostTypeId="2" ParentId="3331" CreationDate="2010-10-06T14:00:52.570" Score="5" ViewCount="" Body="&lt;p&gt;Several directions for analyzing longitudinal data were discussed in the link provided by @Jeromy, so I would suggest you to read them carefully, especially those on functional data analysis. Try googling for &quot;Functional Clustering of Longitudinal Data&quot;, or the PACE Matlab toolbox which is specifically concerned with model-based clustering of irregularly sampled trajectories (Peng and Müller, &lt;a href=&quot;http://anson.ucdavis.edu/~mueller/distance_AOAS.pdf&quot; rel=&quot;nofollow&quot;&gt;Distance-based clustering of sparsely observed stochastic processes, with applications to online auctions&lt;/a&gt;, &lt;em&gt;Annals of Applied Statistics&lt;/em&gt; 2008 2: 1056). I can imagine that there may be a good statistical framework for financial time series, but I don't know about that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;code&gt;kml&lt;/code&gt; package basically relies on k-means, working (by default) on euclidean distances between the $t$ measurements observed on $n$ individuals. What is called a &lt;em&gt;trajectory&lt;/em&gt; is just the series of observed values for individual $i$, $y_i=(y_{i1},y_{i2},\dots,y_{it})$, and $d(y_i,y_j)=\sqrt{t^{-1}\sum_{k=1}^t(y_{ik}-y_{jk})^2}$. Missing data are handled through a slight modification of the preceding distance measure (Gower adjustment) associated to a nearest neighbor-like imputation scheme (for computing Calinski criterion). As I don't represent myself what you real data would look like, I cannot say if it will work. At least, it work with longitudinal growth curves, &quot;polynomial&quot; shape, but I doubt it will allow you to detect very specific patterns (like local minima/maxima at specific time-points with time-points differing between clusters, by a translation for example). If you are interested in clustering possibly misaligned curves, then you definitively have to look at other solutions; &lt;a href=&quot;http://homes.stat.unipd.it/mgri/SIS2010/Program/contributedpaper/658-960-1-RV.pdf&quot; rel=&quot;nofollow&quot;&gt;Functional clustering and alignment&lt;/a&gt;, from Sangalli et al., and references therein may provide a good starting point.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Below, I show you some code that may help to experiment with it (my seed is generally set at 101, if you want to reproduce the results). Basically, for using &lt;code&gt;kml&lt;/code&gt; you just have to construct a &lt;code&gt;clusterizLongData&lt;/code&gt; object (an &lt;code&gt;id&lt;/code&gt; number for the first column, and the $t$ measurements in the next columns). &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(lattice)&#xA;xyplot(var0 ~ date, data=test.data, groups=store, type=c(&quot;l&quot;,&quot;g&quot;))&#xA;&#xA;tw &amp;lt;- reshape(test.data, timevar=&quot;date&quot;, idvar=&quot;store&quot;, direction=&quot;wide&quot;)&#xA;parallel(tw[,-1], horizontal.axis=F, &#xA;         scales=list(x=list(rot=45, &#xA;                            at=seq(1,ncol(tw)-1,by=2), &#xA;                            labels=substr(names(tw[,-1])[seq(1,ncol(tw)-1,by=2)],6,100), &#xA;                            cex=.5)))&#xA;&#xA;library(kml)&#xA;names(tw) &amp;lt;- c(&quot;id&quot;, paste(&quot;t&quot;, 1:(ncol(tw)-1)))&#xA;tw.cld &amp;lt;- as.cld(tw)&#xA;cld.res &amp;lt;- kml(tw.cld,nbRedrawing=5)&#xA;plot(tw.cld)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The next two figures are the raw simulated data and the five-cluster solution (according to Calinski criterion, also used in the &lt;a href=&quot;http://cran.r-project.org/web/packages/fpc/index.html&quot; rel=&quot;nofollow&quot;&gt;fpc&lt;/a&gt; package). I don't show the &lt;a href=&quot;http://i.imgur.com/vySFu.png&quot; rel=&quot;nofollow&quot;&gt;scaled version&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/x5MR1.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/4W6Lo.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-06T15:18:24.760" LastActivityDate="2010-10-06T15:18:24.760" CommentCount="3" />
  <row Id="3364" PostTypeId="1" AcceptedAnswerId="3365" CreationDate="2010-10-06T15:15:25.997" Score="2" ViewCount="158" Body="&lt;p&gt;Hi, I have conducted an IRT analysis using package &lt;a href=&quot;http://cran.r-project.org/web/packages/ltm/index.html&quot; rel=&quot;nofollow&quot;&gt;ltm&lt;/a&gt; in R 2.10. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;fit1 &amp;lt;- grm(data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When producing plots of item response characteristic curves (ICCs), &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plot(fit1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;the text &quot;Item response characteristic curves - Item:NAME&quot; is printed as the main title of each plot. NAME is substituted with the actual name of the items in the dataset. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A common practice is to present all the individual ICC plots on a single page, and then all that is needed in the title of each plot is the NAME (or at the most &quot;Item: NAME&quot;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have not been able to achieve this, if I create a vector with the actual item names (e.g. v1, v2, v3...Vn) they are all used as the title in each of the plots. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I would like to achieve is to have the title of each plot simply read &quot;Item: NAME&quot; where NAME is replaced/pasted with the names of the actual variables in my data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The variable names themselves are available in the coefficients &lt;code&gt;fit1&lt;/code&gt; object, and can be retrieved with &lt;code&gt;names(fit1$coefficients)&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyone have a suggestion for how to achieve this? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Reproducible sample (thanks @Chi)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dat &amp;lt;- Science[c(1,3,4,7)]&#xA;fit1 &amp;lt;- grm(dat)&#xA;par(mfrow = c(2,2))&#xA;plot(fit1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This plots each of the four ICCs on the same device, but all titles contain the abovementioned text that I wish to remove. It also contains the text &quot;Item: NAME&quot; (i.e. comfort, environment, work, benefit) which I would like to retain.  &lt;/p&gt;&#xA;" OwnerUserId="913" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-10-18T07:50:52.467" LastActivityDate="2010-10-18T07:50:52.467" Title="R package ltm: How to manipulate title on item response category characteristic curve plot" Tags="&lt;r&gt;&lt;data-visualization&gt;&lt;psychometrics&gt;" AnswerCount="1" />
  <row Id="3365" PostTypeId="2" ParentId="3364" CreationDate="2010-10-06T15:31:00.033" Score="2" ViewCount="" Body="&lt;p&gt;Try using the argument &lt;code&gt;main=&lt;/code&gt; when calling &lt;code&gt;plot()&lt;/code&gt;, e.g. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dat &amp;lt;- Science[c(1,3,4,7)]&#xA;fit1 &amp;lt;- grm(dat)&#xA;plot(fit1, items=1, main=paste(&quot;Item&quot;, names(dat)[1], sep=&quot;: &quot;))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;See &lt;code&gt;help(plot.grm)&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, you can embed all ICC curves in the same figure by using &lt;code&gt;par()&lt;/code&gt;, e.g.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;opar &amp;lt;- par(mfrow=c(2,2))&#xA;for (i in 1:4)&#xA;  plot(fit1, items=i, main=paste(&quot;Item&quot;, names(dat)[i], sep=&quot;: &quot;))&#xA;par(opar)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-10-06T15:31:00.033" CommentCount="4" />
  <row Id="3367" PostTypeId="2" ParentId="152" CreationDate="2010-10-06T16:57:44.897" Score="3" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://www.math.u-psud.fr/select/people/celeux/Welcome.html&quot; rel=&quot;nofollow&quot;&gt;Gilles Celeux&lt;/a&gt; also worked on the problem of label switching, e.g.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;G. Celeux, Bayesian inference for&#xA;  Mixture: the label switching problem.&#xA;  &lt;em&gt;Proceedings Compstat 98&lt;/em&gt;, pp. 227-232, Physica-Verlag (1998).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;As a complement to @darrenjw's fine answer, here are two online papers that reviewed alternative strategies: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Jasra et al., &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.105.6380&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;Markov Chain Monte Carlo Methods and the Label Switching Problem in Bayesian Mixture Modelling&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Sperrin et al., &lt;a href=&quot;http://eprints.lancs.ac.uk/26391/1/labels-sc-techrep-apr09.pdf&quot; rel=&quot;nofollow&quot;&gt;Probabilistic relabelling strategies for the label switching problem in Bayesian mixture models&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-10-06T16:57:44.897" />
  <row Id="3368" PostTypeId="1" CreationDate="2010-10-06T19:52:48.773" Score="3" ViewCount="89" Body="&lt;p&gt;Suppose I have three time series with the same length and the same temporal interval. If I upsample each of the three time series through interpolation, that will cause aliasing. However, if I'm interested in the information embedded in the average of the three time series, is such an upsampling justified? In other words, does the above upsamping and then averaging among three time series increase the Nyquist frequency by three times? Thanks!&lt;/p&gt;&#xA;" OwnerUserId="1513" LastActivityDate="2010-10-06T21:44:56.557" Title="Does averaging across multiple time series render higher Nyquist frequency?" Tags="&lt;time-series&gt;" AnswerCount="1" CommentCount="5" />
  <row Id="3369" PostTypeId="1" CreationDate="2010-10-06T20:05:29.200" Score="0" ViewCount="359" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;Possible Duplicate:&lt;/strong&gt;&lt;br&gt;&#xA;  &lt;a href=&quot;http://stats.stackexchange.com/questions/1576/what-are-the-differences-between-factor-analysis-and-principal-component-analysis&quot;&gt;What are the differences between Factor Analysis and Principal Component Analysis&lt;/a&gt;  &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&#xA;&#xA;&lt;p&gt;I am trying to understand the difference between PCA and FA. Through google research, I have come to understand that PCA accounts for all variance, while FA accounts for only common variance and ignores unique variance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I am having a difficult time wrapping my head around how exactly this occurs. I know PCA rotates the axis used to describe the data in order to eliminate all covariance. Does this step still occur in FA? If not, what differentiates FA from PCA? Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="1514" LastActivityDate="2010-10-06T21:51:18.863" ClosedDate="2010-10-07T01:30:50.850" Title="Difference between FA and PCA" Tags="&lt;pca&gt;&lt;factor-analysis&gt;" AnswerCount="0" CommentCount="0" FavoriteCount="0" />
  <row Id="3370" PostTypeId="2" ParentId="1611" CreationDate="2010-10-06T20:10:59.000" Score="2" ViewCount="" Body="&lt;p&gt;I think bayesian statistics come into play in two different contexts. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the one hand, some researcher/statistician are definitely convinced of the &quot;bayesian spirit&quot; and, acknowledging the limit of the classical frequentist hypothesis framework, have decided to concentrate on bayesian thinking. Studies in experimental psychology highlighting small effect sizes or borderline statistical significance are now increasingly relying on the bayesian framework. In this respect, I like to cite some of the extensive work of Bruno Lecoutre (1-4) who contributed to developing the use of fiducial risk and bayesian (M)ANOVA. I think the fact we can readily interpret a confidence interval in terms of probabilities applied on the parameter of interest (i.e. depending on the prior distribution) is a radical turn in statistical thinking. I can also imagine that everybody is actually aware of the ever growing work of Andrew Gelman in this domain, as pointed by @Skrikant, or of the incentive given by the &lt;a href=&quot;http://www.bayesian.org/&quot; rel=&quot;nofollow&quot;&gt;International Society for Bayesian Analysis&lt;/a&gt; to use bayesian models. Frank Harrell also provides interesting outlines of &lt;a href=&quot;http://biostat.mc.vanderbilt.edu/wiki/pub/Main/ClinStat/bayes.pdf&quot; rel=&quot;nofollow&quot;&gt;Bayesian Methods for Clinicians&lt;/a&gt;, as applied to &lt;a href=&quot;http://biostat.mc.vanderbilt.edu/wiki/pub/Main/ClinStat/essence.rct.html&quot; rel=&quot;nofollow&quot;&gt;RCTs&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, the bayesian approach has proved successful in diagnostic medicine (5), and is often used as an ultimate alternative where traditional statistics would fail, if applicable at all. I am thinking of a psychometrical paper (6) where authors were interested in assessing the agreement between radiologists about the severity of hip fractures from a very limited data set (12 doctors x 15 radiography) and use an Item Response model for polytomous items.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, a recent 45-pages paper published in &lt;em&gt;Statistics in Medicine&lt;/em&gt; provides an interesting overview of the &quot;penetrance&quot; of bayesian modeling in biostatistics:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Ashby, D (2006). &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.132.5449&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;Bayesian statistics&#xA;  in medicine: a 25 year review&lt;/a&gt;.&#xA;  &lt;em&gt;Statistics in Medicine&lt;/em&gt;, 25(21), 3589-631.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;References&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Rouanet H., Lecoutre B. (1983). Specific inference in ANOVA: From significance tests to Bayesian procedures. &lt;em&gt;British Journal of Mathematical and Statistical Psychology&lt;/em&gt;, &lt;em&gt;36&lt;/em&gt;, 252-268.&lt;/li&gt;&#xA;&lt;li&gt;Lecoutre B., Lecoutre M.-P., Poitevineau J. (2001). Uses, abuses and misuses of significance tests in the scientific community: Won't the Bayesian choice be unavoidable? &lt;em&gt;International Statistical Review&lt;/em&gt;, &lt;em&gt;69&lt;/em&gt;, 399-418.&lt;/li&gt;&#xA;&lt;li&gt;Lecoutre B. (2006). Isn't everyone a Bayesian?. &lt;em&gt;Indian Bayesian Society News Letter&lt;/em&gt;, &lt;em&gt;III&lt;/em&gt;, 3-9.&lt;/li&gt;&#xA;&lt;li&gt;Lecoutre B. (2006). And if you were a Bayesian without knowing it? In A. Mohammad-Djafari (Ed.): &lt;em&gt;26th Workshop on Bayesian Inference and Maximum Entropy Methods in Science and Engineering&lt;/em&gt;. Melville : AIP Conference Proceedings Vol. 872, 15-22.&lt;/li&gt;&#xA;&lt;li&gt;Broemeling, L.D. (2007). &lt;em&gt;&lt;a href=&quot;http://www.crcnetbase.com/isbn/9781584887683&quot; rel=&quot;nofollow&quot;&gt;Bayesian Biostatistics and Diagnostic Medicine&lt;/a&gt;&lt;/em&gt;. Chapman and Hall/CRC.&lt;/li&gt;&#xA;&lt;li&gt;Baldwin, P., Bernstein, J., and Wainer, H. (2009). Hip psychometrics. &lt;em&gt;Statistics in Medicine&lt;/em&gt;, 28(17), 2277-92.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-08T07:17:49.240" LastActivityDate="2010-10-08T07:17:49.240" CommunityOwnedDate="2010-10-06T20:10:59.000" />
  <row Id="3371" PostTypeId="2" ParentId="1576" CreationDate="2010-10-06T20:33:36.393" Score="4" ViewCount="" Body="&lt;p&gt;You are right about your first point, although in FA you generally work with both (uniqueness and communality).&#xA;The choice between PCA and FA is a long-standing debate among psychometricians. I don't quite follow your points, though. Rotation of principal axes can be applied whatever the method is used to constructed latent factors. In fact, most of the times this is the VARIMAX rotation (orthogonal rotation, considering uncorrelated factors) that is used, for practical reasons (easiest interpretation, easiest scoring rules or interpretation of factor scores, etc.), although oblique rotation (e.g. PROMAX) might probably better reflect the reality (latent constructs are often correlated one each other), at least in the tradition of FA where you assume that a latent construct is really at the heart of the observed inter-correlations between your variables. The point is that PCA followed by VARIMAX rotation somewhat distorts the interpretation of the linear combinations of the original variables in the &quot;data analysis&quot; tradition (see the work of Michel Tenenhaus). From a psychometrical perspectice, FA models are to be preferred since they explicitely account for measurement errors, while PCA doesn't care about that. Briefly stated, using PCA you are expressing each component (factor) as a linear combination of the variables, whereas in FA these are the variables that are expressed as linear combinations of the factors (including communalities and uniqueness components, as you said).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I recommend you to read first the following discussions about this topic:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://stats.stackexchange.com/questions/1576/what-are-the-differences-between-factor-analysis-and-principal-component-analysis&quot;&gt;What are the differences between Factor Analysis and Principal Component Analysis&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://stats.stackexchange.com/questions/2742/on-the-use-of-oblique-rotation-after-pca/2747#2747&quot;&gt;On the use of oblique rotation after PCA&lt;/a&gt; -- see reference therein&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-06T20:49:44.070" LastActivityDate="2010-10-06T20:49:44.070" CommentCount="1" />
  <row Id="3372" PostTypeId="1" AcceptedAnswerId="3376" CreationDate="2010-10-06T21:16:06.060" Score="7" ViewCount="291" Body="&lt;p&gt;I must clarify immediately that I am a practicing software developer, not a statistician, and that my college stats class was a &lt;em&gt;very&lt;/em&gt; long time ago…&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That said, I would like to know if there is a method for accumulating a set of descriptive statistics that could then be used to produce a boxplot, that does not entail storing a bunch of individual samples?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I am trying to do is produce a graphical summary of queue service times within a complex multi-queue process. I have in the past used a package called tnftools that allowed large samples to be accumulated and then post-processed into a nice graph of response times and outliers… But tnftools are not available for my current platform.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ideally I would like to be able to accumulate a set of descriptive statistics &quot;on the fly&quot; as the process runs, and then extract the data for analysis on demand. But I cannot simply have the process accumulate samples as the memory / IO involved in doing so would have an unacceptable impact on the performance of the system.&lt;/p&gt;&#xA;" OwnerUserId="1515" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-10-07T08:54:50.567" LastActivityDate="2010-10-07T08:54:50.567" Title="Is it possible to accumulate a set of statistics that describes a large number of samples such that I can then produce a boxplot?" Tags="&lt;algorithms&gt;&lt;median&gt;&lt;quantiles&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="5" />
  <row Id="3373" PostTypeId="2" ParentId="3368" CreationDate="2010-10-06T21:29:23.210" Score="2" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Simple Answer:&lt;/strong&gt; No.  (At least I don't think so -- It's been 10 years since my EE classes).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Suggested Reading:&lt;/strong&gt; You might check out the Wikipedia page on &lt;a href=&quot;http://en.wikipedia.org/wiki/Oversampling&quot; rel=&quot;nofollow&quot;&gt;Oversampling&lt;/a&gt; or also this page on &lt;a href=&quot;http://zone.ni.com/devzone/cda/tut/p/id/3947&quot; rel=&quot;nofollow&quot;&gt;Reducing the effects of noise&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, multiple samples with independent white noise will, when averaged, raise the signal to noise ratio.  But they will not increase the Nyquist frequency.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Short Example:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Construct three cell towers equidistant from a point source.  Make a cell call in the center and record it independently on each of the three towers at a common sampling rate, say the 450 MHz GSM band.  Then average each sample across the three towers and viola, triple the S/N (assuming that local noise characteristics of each tower are independent and all received the same signal at exactly the same strength).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, even though you upped the S/N by a factor of three, you didn't change the &lt;a href=&quot;http://en.wikipedia.org/wiki/Nyquist_frequency&quot; rel=&quot;nofollow&quot;&gt;Nyquist frequency&lt;/a&gt;.  Reference included for those interested.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note:&lt;/strong&gt; In my example, I'm assuming that the samples collected at the three towers are &quot;in phase&quot; -- equidistant -- if so, then what I've stated above is correct.  If the samples you're collecting are out of phase, then this might in fact raise the Nyquist frequency.  That said, you might want to ask this question somewhere else -- Unfortunately the corresponding &lt;a href=&quot;http://area51.stackexchange.com/proposals/1691/signal-image-video-processing&quot;&gt;Signal, Image, &amp;amp; Video Processing&lt;/a&gt; and &lt;a href=&quot;http://area51.stackexchange.com/proposals/18854/electronics-electrical-engineering&quot;&gt;Electronics &amp;amp; Electrical Engineering&lt;/a&gt; boards have yet to really take off.  You might post your question there, but I'm not sure if it will ever be answered.&lt;/p&gt;&#xA;" OwnerUserId="1499" LastEditorUserId="1499" LastEditorDisplayName="" LastEditDate="2010-10-06T21:44:56.557" LastActivityDate="2010-10-06T21:44:56.557" CommentCount="2" />
  <row Id="3374" PostTypeId="2" ParentId="1576" CreationDate="2010-10-06T21:51:18.863" Score="4" ViewCount="" Body="&lt;p&gt;From my response here:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://stats.stackexchange.com/questions/612/is-psychprincipal-function-still-pca-when-using-rotation&quot;&gt;http://stats.stackexchange.com/questions/612/is-psychprincipal-function-still-pca-when-using-rotation&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Principal Components Analysis (PCA) and Common Factor Analysis (CFA) are distinct methods. Often, they produce similar results and PCA is used as the default extraction method in the SPSS Factor Analysis routines. This undoubtedly results in a lot of confusion about the distinction between the two.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The bottom line is, these are two different models, conceptually. In PCA, the components are actual orthogonal linear combinations that maximize the total variance. In FA, the factors are linear combinations that maximize the shared portion of the variance--underlying &quot;latent constructs&quot;. That's why FA is often called &quot;common factor analysis&quot;. FA uses a variety of optimization routines and the result, unlike PCA, depends on the optimization routine used and starting points for those routines. Simply there is not a single unique solution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In R, the factanal() function provides CFA with a maximum likelihood extraction. So, you shouldn't expect it to reproduce an SPSS result which is based on a PCA extraction. It's simply not the same model or logic. I'm not sure if you would get the same result if you used SPSS's Maximum Likelihood extraction either as they may not use the same algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For better or for worse in R, you can, however, reproduce the mixed up &quot;factor analysis&quot; that SPSS provides as its default. Here's the process in R. With this code, I'm able to reproduce the SPSS Principal Component &quot;Factor Analysis&quot; result using this dataset. (With the exception of the sign, which is indeterminant). That result could also then be rotated using any of Rs available rotation methods.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Load the base dataset attitude to work with.&#xA;data(attitude)&#xA;# Compute eigenvalues and eigen vectors of the correlation matrix.&#xA;pfa.eigen&amp;lt;-eigen(cor(attitude))&#xA;# Print and note that eigen values are those produced by SPSS.&#xA;# Also note that SPSS will extract 2 components as eigen values &amp;gt; 1 = 2&#xA;pfa.eigen$values&#xA;# set a value for the number of factors (for clarity)&#xA;factors&amp;lt;-2&#xA;# Extract and transform two components.&#xA;pfa.eigen$vectors [ , 1:factors ]  %*% &#xA;+ diag ( sqrt (pfa.eigen$values [ 1:factors ] ),factors,factors )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="485" LastActivityDate="2010-10-06T21:51:18.863" CommentCount="2" />
  <row Id="3375" PostTypeId="2" ParentId="3362" CreationDate="2010-10-07T00:02:32.620" Score="2" ViewCount="" Body="&lt;p&gt;[Caveat: I have not read the paper so the below may be nonsense for all I know ...]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Based on the summary of the R package I would venture to guess that you could use the proposed methodology for the survey data provided the following conditions are met:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;You have survey data from control groups during pre-intervention periods. These control groups need not be identical to the treatment groups.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The data you have is time series data.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Provided points 1 and 2 are met my best intuition as to how the method works is as follows:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;First, construct a 'hypothetical' (synthetic in their words) control group that behaves as identical as possible as the treatment group. The hypothetical group is constructed by taking a convex combination of the control group data you have. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an example, suppose that you want to measure student performance on math. Your control groups could be different sections whereas the treatment group is one specific section. You construct the hypothetical control group such that the weighted (with the weights summing to 1 and hence convex) average of the scores of the control group sections is as close as possible to the scores of the treatment group before the intervention (i.e., use MSPE which is Mean Squared Prediction Error).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Second, extrapolate the hypothetical group's scores into the post-intervention period using the parameter estimates from step 1.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Since, the hypothetical group has been constructed to be identical to the treatment group pre-intervention, the post-intervention scores of the hypothetical group provides an appropriate counter-factual evidence to the treatment group's post-intervention scores to assess the effectiveness of the intervention.&lt;/p&gt;&#xA;" LastActivityDate="2010-10-07T00:02:32.620" CommentCount="4" />
  <row Id="3376" PostTypeId="2" ParentId="3372" CreationDate="2010-10-07T00:33:55.410" Score="11" ViewCount="" Body="&lt;p&gt;For 'on the fly' boxplot, you will need 'on the fly' min/max (trivial) as well as 'on the fly' quartiles (0.25,0.5=median and 0.75).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A lot of work has been going on recently in the problem of online (or 'on the fly')  algorithm for median computation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A recent developements is &lt;strong&gt;binmedian&lt;/strong&gt;. As a side-kick, it also enjoy better worst case complexity than &lt;strong&gt;quickselect&lt;/strong&gt; (which is neither online nor single pass). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can find the associated paper as well as C and FORTRAN code online &lt;a href=&quot;http://www-stat.stanford.edu/~ryantibs/median/&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;. You may have to check the licencing details with the authors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You will also need a single pass algorithm for the quartiles, for which you can use the approach above and the following recursive characterization of the quartiles in terms of medians:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Q_{0.75}(x) \approx Q_{0.5}(x_i:x_i &gt; Q_{0.5}(x))$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Q_{0.25}(x) \approx Q_{0.5}(x_i:x_i &amp;lt; Q_{0.5}(x))$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;i.e. the 25 (75) percent quartile is very close to the median of those observations that are smaller (larger) than the median.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Addendum:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There exist a host of older multi-pass methods for computing quantiles. A popular approach is to maintain/update a deterministically sized reservoir of observations randomly selected from the stream and recursively compute quantiles (see &lt;a href=&quot;http://www.cs.ucsb.edu/~suri/psdir/ency.pdf&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; review) on this reservoir. This (and related) approach are superseded by the one proposed above.&lt;/p&gt;&#xA;" OwnerUserId="603" LastEditorUserId="603" LastEditorDisplayName="" LastEditDate="2010-10-07T08:41:44.510" LastActivityDate="2010-10-07T08:41:44.510" CommentCount="2" />
  <row Id="3377" PostTypeId="1" CreationDate="2010-10-07T03:26:23.003" Score="9" ViewCount="564" Body="&lt;p&gt;Hi everybody.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a little problem that is making me freaking out.&#xA;I have to write procedure for an online acquisition process of a multivariate time series.&#xA;At every time interval (for example 1 second), I get a new sample, which is basically a floating point vector of size N.&#xA;The operation I need to do is a little bit tricky:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;For each new sample, I compute the percentuals for that sample (by normalizing the vector so that the elements will sum to 1).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I calculate the average percentuals vector in the same way, but using the past values.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;For each past value, I compute the absolute deviation of the percentuals vector related to that sample with the global average percentuals vector computed at step 2. This way, the absolute deviation is always a number between 0 (when the vector is equal to the average vector) and 2 (when it is totaly different).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Using the average of the deviations for all the previous samples, I compute the mean absolute deviation, which is again a number between 0 and 2.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I use the mean absolute deviation to detect if a new sample is compatible with the other samples (by comparing its absolute deviation with the mean absolute deviation of the whole set computed at step 4).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Since every time a new sample is collected the global average changes (and so the mean absolute deviation changes as well), is there a way to compute this value without scanning the whole data set multiple times? (one time for computing the global average percentuals, and one time for collecting the absolute deviations).&#xA;Ok, I know it's absolutely easy to calculate the global averages without scanning the whole set, since I just have to use a temporary vector for storing the sum of each dimension, but what about the mean absolute deviation? Its calculation includes the &lt;code&gt;abs()&lt;/code&gt; operator, so I need to access to all the past data!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for your help.&lt;/p&gt;&#xA;" OwnerUserId="667" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-10-08T16:04:53.503" LastActivityDate="2010-11-09T05:36:27.843" Title="Online algorithm for mean absolute deviation and large data set" Tags="&lt;algorithms&gt;&lt;quantiles&gt;&lt;online&gt;&lt;large-data&gt;" AnswerCount="4" FavoriteCount="1" />
  <row Id="3378" PostTypeId="2" ParentId="3377" CreationDate="2010-10-07T05:17:09.267" Score="4" ViewCount="" Body="&lt;p&gt;If you can accept some inaccuracy, this problem can be solved easily by &lt;em&gt;binning&lt;/em&gt; counts. That is, pick some largeish number $M$ (say $M = 1000$), then initialize some integer bins $B_{i,j}$ for $i = 1\ldots M$ and $j = 1\ldots N$, where $N$ is the vector size, as zero. Then when you see the $k$th observation of a percentual vector, increment $B_{i,j}$ if the $j$th element of this vector is between $(i-1)/M$ and $i/M$, looping over the $N$ elements of the vector. (I am assuming your input vectors are non-negative, so that when you compute your 'percentuals', the vectors are in the range $[0,1]$. )&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At any point in time, you can estimate the mean vector from the bins, and the mean absolute deviation. After observing $K$ such vectors, the $j$th element of the mean is estimated by &#xA;$$\bar{X}_j = \frac{1}{K} \sum_i \frac{i - 1/2}{M} B_{i,j},$$ and the $j$th element of the mean absolute deviation is estimated by $$\frac{1}{K} \sum_i | \bar{X_j} - \frac{i - 1/2}{M} | B_{i,j}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;edit&lt;/strong&gt;: this is a specific case of a more general approach where you are building an empirical density estimate. This could be done with polynomials, splines, etc, but the binning approach is the easiest to describe and implement. &lt;/p&gt;&#xA;" OwnerUserId="795" LastEditorUserId="795" LastEditorDisplayName="" LastEditDate="2010-10-07T16:44:47.327" LastActivityDate="2010-10-07T16:44:47.327" CommentCount="3" />
  <row Id="3380" PostTypeId="1" CreationDate="2010-10-07T10:50:24.480" Score="2" ViewCount="50" Body="&lt;p&gt;I work with amino acid sequences and I want to use a self-made model to tell me something about it, lets call it $f(\text{seq})$. Now i want to know the contribution of every position in the sequence onto the model, i.e. my question is what is the importance/effect of amino acid $A$ occuring at position $I$ in the sequence with respect to the model?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I visualize something like that?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to use my model also on several sequences of differing lengths. Somehow this throws a monkeywrench into my plans of using a neural net...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is simple yet I did not find anything about it. Pointers would be appreciated. Or any comment you might have. This whole idea of mine is pretty unfinished and I don't really know yet what I want. So feel free to criticize, I will update the question accordingly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ah and if this is the wrong place to put this here please tell me also (:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;cheers and thanks&lt;/p&gt;&#xA;&#xA;&lt;p&gt;note: this is a repost from th.stackexchange.&lt;/p&gt;&#xA;" OwnerUserId="1516" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-07T11:10:56.880" LastActivityDate="2010-10-07T11:27:06.060" Title="Analysis of variables of varying numbers" Tags="&lt;machine-learning&gt;&lt;feature-selection&gt;&lt;bioinformatics&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="3381" PostTypeId="1" CreationDate="2010-10-07T10:59:51.483" Score="2" ViewCount="247" Body="&lt;p&gt;Say I make histograms H1, H2, H4 ...of the same set of data&#xA;with bins 1, 2, 4 ... wide.&#xA;Then the bins containing a given x have counts and averages&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;n1 av1 in H1,&#xA;n2 av2 in H2,&#xA;n4 av4 in H4 ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How should one weight these to estimate data(x) ?&lt;br&gt;&#xA;One possibility would be &amp;Sigma; wj*avj / &amp;Sigma; wj&#xA;with wj = nj / j, for the first two non-zero nj.&lt;br&gt;&#xA;The general goals are&#xA;to fill holes where bins are empty,&#xA;and to smooth bins with many data points.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(H2 etc. can either be built together with H1,&#xA;or on-the-fly from H1 as needed: a space-time tradeoff.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Added Monday 11 Oct, trying to clarify:&#xA;there are two related subquestions:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;local vs global:&lt;br&gt;&#xA;if almost all the terms in &amp;Sigma; K( xj near x ) are 0, there must be a better way. (For 1d it hardly matters, but in 2d or 3d, N^2 N^3 get big quickly.)&lt;br&gt;&#xA;One way of finding xj near x (to sum K at those only) is to grow shells in a regular grid -- poor for clumpy or hole-y data.&#xA;Another is to find Nnear nearest neighbours in a Kd tree -- but what should Nnear be ?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;adaptive histograms ?&lt;br&gt;&#xA;A method that adapts to clumpy data, as TINs (triangulations) adapt to terrains, would be nice.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="557" LastEditorUserId="557" LastEditorDisplayName="" LastEditDate="2010-10-11T17:24:25.637" LastActivityDate="2010-10-11T17:24:25.637" Title="How to smooth histograms with bins 1, 2, 4, ... wide ?" Tags="&lt;smoothing&gt;&lt;histogram&gt;" AnswerCount="1" CommentCount="4" />
  <row Id="3382" PostTypeId="2" ParentId="3380" CreationDate="2010-10-07T11:27:06.060" Score="1" ViewCount="" Body="&lt;p&gt;This is a tricky topic in general; because of the size variability you mentioned it rather boils down to motif finding. Yet I can propose four ideas:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Supposing that you can align sequences (ok, you always can, but let's say it yields non nonsense consensus), you can reduce the problem to the conserved region and make attributes out of aligned residues. Finding important residues will then be a simple feature selection.&lt;/li&gt;&#xA;&lt;li&gt;Look for motifs using &lt;a href=&quot;http://meme.sdsc.edu/meme4_4_0/intro.html&quot; rel=&quot;nofollow&quot;&gt;MEME&lt;/a&gt; or something more fancy to make PWM and just hope it will work. Sequence logo will show you the residue importance based on the predictive power it applies. &lt;/li&gt;&#xA;&lt;li&gt;Make attributes out of N-gram spectra, i.e. counts of all possible N-residue words for few values of N (till your computer won't explode). Or use SVM with string kernel, it usually boils to the same thing.&lt;/li&gt;&#xA;&lt;li&gt;&amp;lt;&amp;lt; Can't talk about it yet, research in progress ;-) &amp;gt;&gt; &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-10-07T11:27:06.060" CommentCount="2" />
  <row Id="3383" PostTypeId="1" AcceptedAnswerId="3639" CreationDate="2010-10-07T12:30:23.340" Score="3" ViewCount="487" Body="&lt;p&gt;This sounds like an easy setup, and I'm absolutely sure it's not too complicated to do, but somehow I can't figure out how to approach this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The setting is as follows : There are 6 treatments with 16 subjects in each treatment. Weight is measured for every subject before and after treatment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The hypothesis to be tested : is there a difference between treatments in effect (the difference in weight before and after treatment).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Very simple repeated measures design, but the thing is that the subjects are not identified. The only thing you know is that it's the same subjects, but you can't use a repeated measures due to the lack of some id variable. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Up til now, I constructed the mean difference as the difference between the means at t=0 and t=1 for every tank. Calculating the SE for each of these differences allows me to do a series of T-tests and put a Dunn-Sidak correction on it afterwards. But somehow this is cumbersome and feels not right.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively, I can substract the mean for each  treatment at t=0, and do a classical one-way ANOVA. But I am not sure how I should correct the variances in order to count for the fact that I don't substract a fixed number, but a sample mean.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the appropriate model to use here, and if possible, where do I find that function in R? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;it is NOT aov(weight~treatment*time), as this will make a comparison between eg t=0/treatment=1 and t=1/treatment=3. That's not what I want.&lt;/p&gt;&#xA;" OwnerUserId="1124" LastActivityDate="2010-10-15T16:34:25.457" Title="Compare treatments on mean difference between two times" Tags="&lt;anova&gt;&lt;mean&gt;&lt;repeated-measures&gt;" AnswerCount="2" CommentCount="13" FavoriteCount="1" />
  <row Id="3384" PostTypeId="2" ParentId="3316" CreationDate="2010-10-07T13:15:02.057" Score="2" ViewCount="" Body="&lt;p&gt;Your &quot;simplistic first thought&quot; of qualitatively representing just the directional movement is similar in spirit to Keogh's SAX algorithm for comparing time series. I'd recommend you take a look at it: &lt;a href=&quot;http://www.cs.ucr.edu/~eamonn/SAX.htm&quot; rel=&quot;nofollow&quot;&gt;Eamonn Keogh &amp;amp; Jessica Lin: SAX&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From your edit, it sounds like you're now thinking about tackling the problem differently, but you might find that SAX provides a piece of the puzzle.&lt;/p&gt;&#xA;" OwnerUserId="1436" LastActivityDate="2010-10-07T13:15:02.057" />
  <row Id="3385" PostTypeId="2" ParentId="3383" CreationDate="2010-10-07T13:21:46.283" Score="1" ViewCount="" Body="&lt;p&gt;First since you do not know which subjects are which, I think it makes more sense to treat each group of observations (before treatment and after treatment) as seperate observations (i.e. as if you randomly assigned treatment to 96 out of 192 subjects). I know this seems obvious but for heuristic purposes I think it helps clarify the question at hand. While this isn't optimal (regression to the mean) I would say you are still better off than many observational studies (assuming no self-selection into treatment). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My initial thought was if treatments are randomly assigned you might as well treat all the pre-treatments as one big control group. You could then use an OLS framework with dummy variables to estimate the treatment effects (with pre-treatments as the reference category). If you know what pre-treatments go to each treatment you could run an anova to see if any non-neglible differences exist between the mean weight in pre-treatment groups. If there are differences in the pre-treatment groups that are not ignorable you should be able to use a multi level framework. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since fish are in the same tank a multi-level framework may be appropriate anyway.&lt;/p&gt;&#xA;" OwnerUserId="1036" LastActivityDate="2010-10-07T13:21:46.283" CommentCount="2" />
  <row Id="3386" PostTypeId="1" CreationDate="2010-10-07T14:48:10.447" Score="2" ViewCount="1648" Body="&lt;p&gt;I have to do a binary logistic regression with a lot of independent variables. Most of them are binary. Few ones are categorical with more than two possible values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which is the best way to deal with such variables?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, for a variable with three possible values, I suppose that two dummy variables have to be created. Then, in a step-wise regression procedure, it is better to test in the same time both the two dummy variables, or to test them separately?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since I will use SPSS which I do not remember very well, which way does SPSS operate?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Moreover, for a categorical ordinal variable, it is a good thing to use dummy variables which recreate the ordinal scale (for example, using three dummy variables for a 4-state ordinal variable, put 0-0-0 for level 1, 1-0-0 for level 2, 1-1-0 for level 3 and 1-1-1 for level 4, instead of 0-0-0, 1-0-0, 0-1-0 and 0-0-1 for the 4 levels)? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for your answers.&lt;/p&gt;&#xA;" OwnerUserId="1219" LastEditorUserId="1219" LastEditorDisplayName="" LastEditDate="2010-10-08T13:07:05.160" LastActivityDate="2010-10-08T13:07:05.160" Title="How to deal with not-binary categorical variables in logistic regression (SPSS)" Tags="&lt;logistic-regression&gt;&lt;categorical-data&gt;&lt;spss&gt;&lt;ordered-variables&gt;" AnswerCount="3" CommentCount="1" />
  <row Id="3387" PostTypeId="2" ParentId="3386" CreationDate="2010-10-07T15:18:08.673" Score="5" ViewCount="" Body="&lt;p&gt;The UCLA website has a bunch of great tutorials for every procedure broken down by the software type that you're familiar with.  Check out &lt;a href=&quot;http://www.ats.ucla.edu/stat/spss/output/logistic.htm&quot; rel=&quot;nofollow&quot;&gt;Annotated SPSS Output: Logistic Regression&lt;/a&gt; -- the SES variable they mention is categorical (and not binary).  SPSS will automatically create the indicator variables for you.  There's also a page dedicated to &lt;a href=&quot;http://www.ats.ucla.edu/stat/spss/webbooks/reg/chapter3/spssreg3.htm&quot; rel=&quot;nofollow&quot;&gt;Categorical Predictors in Regression with SPSS&lt;/a&gt; which has specific information on how to change the default codings and a page specific to &lt;a href=&quot;http://www.ats.ucla.edu/stat/spss/topics/logistic_regression.htm&quot; rel=&quot;nofollow&quot;&gt;Logistic Regression&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1499" LastActivityDate="2010-10-07T15:18:08.673" />
  <row Id="3388" PostTypeId="2" ParentId="3386" CreationDate="2010-10-07T15:56:05.430" Score="1" ViewCount="" Body="&lt;p&gt;Logistic regression is a pretty flexible method.  It can readily use as independent variables categorical variables.  Most software that use Logistic regression should let you use categorical variables.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an example, let's say one of your categorical variable is temperature defined into three categories: cold/mild/hot.  As you suggest you could interpret that as three separate dummy variables each with a value of 1 or 0.  But, the software should let you use a single categorical variable instead with text value cold/mild/hot.  And, the logit regression would derive coefficient (or constant) for each of the three temperature conditions.  If one is not significant, the software or the user could readily take it out (after observing t stat and p value).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The main benefit of grouping categorical variable categories into a single categorical variable is model efficiency.  A single column in your model can handle as many categories as needed for a single categorical variable.  If instead, you use a dummy variable for each categories of a categorical variable your model can quickly grow to have numerous columns that are superfluous given the mentioned alternative. &lt;/p&gt;&#xA;" OwnerUserId="1329" LastActivityDate="2010-10-07T15:56:05.430" CommentCount="8" />
  <row Id="3389" PostTypeId="2" ParentId="3377" CreationDate="2010-10-07T16:27:19.653" Score="3" ViewCount="" Body="&lt;p&gt;There is also a parametric approach. Ignoring the vector nature of your data, and looking only at the marginals, it suffices to solve the problem: find an online algorithm to compute the mean absolute deviation of scalar $X$. If (and this is the big 'if' here) you thought that $X$ followed some probability distribution with unknown parameters, you could estimate the parameters using some online algorithm, then compute the mean absolute deviation based on that parametrized distribution. For example, if you thought that $X$ was (approximately) normally distributed, you could estimate its standard deviation, as $s$, and the mean absolute deviation would be estimated by $s \sqrt{2 / \pi}$ (see &lt;a href=&quot;http://en.wikipedia.org/wiki/Half-normal_distribution&quot; rel=&quot;nofollow&quot;&gt;Half Normal Distribution&lt;/a&gt;).&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-10-07T16:27:19.653" CommentCount="2" />
  <row Id="3390" PostTypeId="1" AcceptedAnswerId="4057" CreationDate="2010-10-07T17:00:40.833" Score="5" ViewCount="288" Body="&lt;p&gt;The &lt;a href=&quot;http://www.riskglossary.com/link/cornish_fisher.htm&quot; rel=&quot;nofollow&quot;&gt;Cornish-Fisher Expansion&lt;/a&gt; provides a way to estimate the quantiles of a distribution based on moments. (In this sense, I see it as a complement to the &lt;a href=&quot;http://en.wikipedia.org/wiki/Edgeworth_expansion#Edgeworth_series&quot; rel=&quot;nofollow&quot;&gt;Edgeworth Expansion&lt;/a&gt;, which gives an estimate of the cumulative distribution based on moments.) I would like to know in which situations would one prefer the Cornish-Fisher expansion for empirical work over the sample quantile, or vice-versa. A few guesses:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Computationally, sample moments can be computed online, whereas online estimation of sample quantiles is difficult. In this case, the C-F 'wins'.&lt;/li&gt;&#xA;&lt;li&gt;If one had the ability to forecast moments, the C-F would allow one to leverage these forecasts for quantile estimation.&lt;/li&gt;&#xA;&lt;li&gt;The C-F Expansion can possibly give estimates of quantiles outside the range of observed values, whereas the sample quantile probably should not.&lt;/li&gt;&#xA;&lt;li&gt;I am not aware of how to compute a confidence interval around the quantile estimates given by C-F. In this case, sample quantile 'wins'.&lt;/li&gt;&#xA;&lt;li&gt;It seems like the C-F Expansion requires one to estimate multiple higher moments of a distribution. The errors in these estimates probably compound in such a way that the C-F Expansion has a higher standard error than the sample quantile.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Any others? Does anybody have experience using both of these methods?&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-10-28T22:34:50.633" Title="Why Use the Cornish-Fisher Expansion Instead of Sample Quantile?" Tags="&lt;distributions&gt;&lt;finance&gt;&lt;quantiles&gt;&lt;empirical&gt;" AnswerCount="1" />
  <row Id="3391" PostTypeId="2" ParentId="3331" CreationDate="2010-10-07T17:42:05.903" Score="4" ViewCount="" Body="&lt;p&gt;You could look at the work of &lt;a href=&quot;http://www.cs.ucr.edu/~eamonn/&quot; rel=&quot;nofollow&quot;&gt;Eamonn Keogh&lt;/a&gt; (UC Riverside) on time series clustering. His website has a lot of resources. I think he provides Matlab code samples, so you'd have to translate this to R.&lt;/p&gt;&#xA;" OwnerUserId="1436" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-07T17:45:59.027" LastActivityDate="2010-10-07T17:45:59.027" CommentCount="1" />
  <row Id="3392" PostTypeId="1" AcceptedAnswerId="3398" CreationDate="2010-10-07T17:44:32.840" Score="15" ViewCount="388" Body="&lt;p&gt;It seems that lots of people (including me) like to do exploratory data analysis in Excel. Some limitations, such as the number of rows allowed in a spreadsheet, are a pain but in most cases don't make it impossible to use Excel to play around with data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.pages.drexel.edu/~bdm25/excel2007.pdf&quot; rel=&quot;nofollow&quot;&gt;A paper by McCullough and Heiser&lt;/a&gt;, however, practically screams that you will get your results all wrong -- and probably burn in hell as well -- if you try to use Excel.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this paper correct or is it biased? The authors do sound like they hate Microsoft.&lt;/p&gt;&#xA;" OwnerUserId="666" LastActivityDate="2011-05-08T03:01:16.803" Title="Excel as a statistics workbench" Tags="&lt;software&gt;&lt;computational-statistics&gt;&lt;excel&gt;" AnswerCount="7" FavoriteCount="2" />
  <row Id="3393" PostTypeId="2" ParentId="3294" CreationDate="2010-10-07T17:59:07.497" Score="3" ViewCount="" Body="&lt;p&gt;There is also a really good book by Oliver Cappe et. al: &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0387402640&quot; rel=&quot;nofollow&quot;&gt;Inference in Hidden Markov Models&lt;/a&gt;.  However, it is fairly theoretical and very light on the applications.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is another book with examples in R, but I couldn't stand it - &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1584885734&quot; rel=&quot;nofollow&quot;&gt;Hidden Markov Models for Time Series&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.s. The speech recognition community also has a ton of literature on this subject.&lt;/p&gt;&#xA;" OwnerUserId="1499" LastActivityDate="2010-10-07T17:59:07.497" CommunityOwnedDate="2010-10-07T17:59:07.497" />
  <row Id="3394" PostTypeId="2" ParentId="3392" CreationDate="2010-10-07T18:15:35.337" Score="6" ViewCount="" Body="&lt;p&gt;Incidently, a question around the use of Google spreadsheets raised contrasting (hence, interesting) opinions about that, &lt;a href=&quot;http://stats.stackexchange.com/questions/3244/do-some-of-you-use-google-docs-spreadsheet-to-conduct-and-share-your-statistical/3247#3247&quot;&gt;Do some of you use Google Docs spreadsheet to conduct and share your statistical work with others?&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have in mind an older paper which didn't seem so pessimist, but it is only marginally cited in the paper you mentioned: Keeling and Pavur, &lt;a href=&quot;http://www.sciencedirect.com/science?_ob=ArticleURL&amp;amp;_udi=B6V8V-4JHMGWJ-1&amp;amp;_user=10&amp;amp;_coverDate=05%2F01%2F2007&amp;amp;_rdoc=1&amp;amp;_fmt=high&amp;amp;_orig=search&amp;amp;_origin=search&amp;amp;_sort=d&amp;amp;_docanchor=&amp;amp;view=c&amp;amp;_searchStrId=1488136142&amp;amp;_rerunOrigin=google&amp;amp;_acct=C000050221&amp;amp;_version=1&amp;amp;_urlVersion=0&amp;amp;_userid=10&amp;amp;md5=2babab0b51c03746d5d7a74d31f1498c&amp;amp;searchtype=a&quot; rel=&quot;nofollow&quot;&gt;A comparative study of the reliability of nine statistical software packages&lt;/a&gt; (CSDA 2007 51: 3811). But now, I found yours on my hard drive. There was also a special issue in 2008, see &lt;a href=&quot;http://www.pages.drexel.edu/~bdm25/excel-intro.pdf&quot; rel=&quot;nofollow&quot;&gt;Special section on Microsoft Excel 2007&lt;/a&gt;, and more recently in the Journal of Statistical Software: &lt;a href=&quot;http://www.jstatsoft.org/v34/i04/paper&quot; rel=&quot;nofollow&quot;&gt;On the Numerical Accuracy of Spreadsheets&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think it is a long-standing debate, and you will find varying papers/opinions about Excel reliability for statistical computing. I think there are different levels of discussion (what kind of analysis do you plan to do, do you rely on the internal solver, are there non-linear terms that enter a given model, etc.), and sources of numerical inaccuracy might arise as the result of &lt;em&gt;proper computing errors&lt;/em&gt; or &lt;em&gt;design choices&lt;/em&gt; issues; this is well summarized in &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;M. Altman, J. Gill &amp;amp; M.P. McDonald,&#xA;  &lt;em&gt;Numerical Issues in Statistical&#xA;  Computing for the Social Scientist&lt;/em&gt;,&#xA;  Wiley, 2004.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Now, for exploratory data analysis, there are various alternatives that provide enhanced visualization capabilities, multivariate and dynamic graphics, e.g. &lt;a href=&quot;http://www.ggobi.org/&quot; rel=&quot;nofollow&quot;&gt;GGobi&lt;/a&gt; -- but see related threads on this wiki.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But, clearly the first point you made addresses another issue (IMO), namely that of using a spreadsheet to deal with large data set: it is simply not possible to import a large csv file into Excel (I'm thinking of genomic data, but it applies to other kind of high-dimensional data). It has not been built for that purpose.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-07T18:37:23.927" LastActivityDate="2010-10-07T18:37:23.927" CommentCount="1" />
  <row Id="3395" PostTypeId="1" AcceptedAnswerId="3396" CreationDate="2010-10-07T19:21:25.180" Score="3" ViewCount="121" Body="&lt;p&gt;I am currently working on a model which takes two parameters and produces a measurement statistic. Think of it as Z = f(X,Y). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Z is a matrix of my statistics and I am creating a surface plot of it in matlab. Basically, I am looking for a mathematical/analytical way of determining if the surface is smooth, or if it is jagged. Do large values tend to be clustered together or are they dispersed throughout the matrix? - that is my question. Basically, how mixed up are the values of my matrix?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to run the model over different parameter sets and I want to be able to analytically determine which one of my surfaces is the smoothest, has the greatest clustering of large values, and ideally, has no negative values. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help will be greatly appreciated and please let me know if you need any further information. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cheers&lt;/p&gt;&#xA;" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-07T19:23:34.423" LastActivityDate="2010-10-08T02:43:53.970" Title="Smoothness of a surface" Tags="&lt;clustering&gt;&lt;smoothing&gt;&lt;matlab&gt;&lt;spatial&gt;&lt;autocorrelation&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="2" />
  <row Id="3396" PostTypeId="2" ParentId="3395" CreationDate="2010-10-07T19:52:10.740" Score="3" ViewCount="" Body="&lt;p&gt;One model for this situation is to view $Z$ as a realization of a stationary 2D stochastic process.  The limiting behavior at zero (distance) of its empirical &lt;a href=&quot;http://en.wikipedia.org/wiki/Variogram&quot; rel=&quot;nofollow&quot;&gt;variogram&lt;/a&gt; or correlogram provides information about its smoothness: if the limiting correlation is less than one, the process is not even (mean square) continuous.  Otherwise (&lt;strong&gt;Theorem&lt;/strong&gt;) &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;A stationary stochastic process with&#xA;  correlation function $\rho(u)$ is $k$&#xA;  times mean-square differentiable if&#xA;  and only if $\rho(u)$ is $2k$ times&#xA;  differentiable at $u=0$ &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;(Diggle &amp;amp; Ribeiro, &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0387329072&quot; rel=&quot;nofollow&quot;&gt;Model-based Geostatistics&lt;/a&gt;).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Procedures variofit, likfit, and eyefit in the &lt;a href=&quot;http://cran.r-project.org/web/packages/geoR/index.html&quot; rel=&quot;nofollow&quot;&gt;geoR&lt;/a&gt; package for R provide ways to estimate and visualize the variogram.  None of these procedures require that you have all positive values, but they tend to work best when the values are not terrifically skewed.  You must also remove any secular trend initially present in the surface; robust regression of $Z$ against $X$ and $Y$ is one way to do that and other ways (that simultaneously estimate the trend and the variogram of the residuals) are available in geoR.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-10-08T02:43:53.970" LastActivityDate="2010-10-08T02:43:53.970" CommentCount="6" />
  <row Id="3397" PostTypeId="2" ParentId="3392" CreationDate="2010-10-07T19:57:40.057" Score="7" ViewCount="" Body="&lt;p&gt;Well, the question whether the paper is correct or biased should be easy: you could just replicate some of their analyses and see whether you get the same answers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;McCullough has been taking different versions of MS Excel apart for some years now, and apparently MS haven't seen fit to fix errors he pointed out years ago in previous versions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't see a problem with playing around with data in Excel. But to be honest, I would not do my &quot;serious&quot; analyses in Excel. My main problem would not be inaccuracies (which I guess will only very rarely be a problem) but the impossibility of tracking and replicating my analyses a year later when a reviewer or my boss asks why I didn't do X - you can save your work and your blind alleys in commented R code, but not in a meaningful way in Excel.&lt;/p&gt;&#xA;" OwnerUserId="1352" LastActivityDate="2010-10-07T19:57:40.057" CommentCount="3" />
  <row Id="3398" PostTypeId="2" ParentId="3392" CreationDate="2010-10-07T20:15:27.567" Score="16" ViewCount="" Body="&lt;p&gt;Use the right tool for the right job and exploit the strengths of the tools you are familiar with.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In Excel's case there are some salient issues:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Please don't use a spreadsheet to manage data, even if your data will fit into one. You're just asking for trouble, terrible trouble.  There is virtually no protection against typographical errors, wholesale mixing up of data, truncating data values, etc., etc.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Many of the statistical functions indeed are broken.  The t distribution is one of them.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The default graphics are awful.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;It is missing some fundamental statistical graphics, especially boxplots.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The random number generator is a joke (but despite that is still effective for educational purposes).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Avoid the high-level functions and most of the add-ins; they're c**p.  But this is just a general principle of safe computing: if you're not sure what a function is doing, don't use it.  Stick to the low-level ones (which include arithmetic functions, ranking, exp, ln, trig functions, and--within limits--the normal distribution functions).  &lt;em&gt;Never&lt;/em&gt; use an add-in that produces a graphic: it's going to be terrible.  (NB: it's dead easy to create your own probability plots from scratch.  They'll be correct and highly customizable.)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In its favor, though, are the following:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Its basic numerical calculations are as accurate as double precision floats can be.  They include some useful ones, such as log gamma.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;It's quite easy to wrap a control around input boxes in a spreadsheet, making it possible to create dynamic simulations easily.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If you need to share a calculation with non-statistical people, most will have some comfort with a spreadsheet and none at all with statistical software, no matter how cheap it may be.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;It's easy to write effective numerical macros, including porting old Fortran code, which is quite close to VBA.  Moreover, the execution of VBA is reasonably fast.  (For example, I have code that accurately computes non-central t distributions from scratch and three different implementations of Fast Fourier Transforms.)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;It supports some effective simulation and Monte-Carlo add-ons like Crystal Ball and @Risk.  (They use their own RNGs, by the way--I checked.)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The immediacy of interacting directly with (a small set of) data is unparalleled: it's better than any stats package, Mathematica, etc.  When used as a giant calculator with loads of storage, a spreadsheet really comes into its own.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;em&gt;Good&lt;/em&gt; EDA, using robust and resistant methods, is not easy, but after you have done it once, you can set it up again quickly.  With Excel you can effectively reproduce &lt;em&gt;all&lt;/em&gt; the calculations (although only some of the plots) in Tukey's EDA book, including median polish of n-way tables (although it's a bit cumbersome).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In direct answer to the original question, there is a bias in that paper: it focuses on the material that Excel is weakest at and that a competent statistician is least likely to use.  That's not a criticism of the paper, though, because warnings like this need to be broadcast.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-10-07T20:15:27.567" CommentCount="4" />
  <row Id="3399" PostTypeId="2" ParentId="3392" CreationDate="2010-10-07T21:36:51.820" Score="4" ViewCount="" Body="&lt;p&gt;The papers and other participants point out to technical weaknesses.  Whuber does a good job of outlining at least some of its strengths.  I personally do extensive statistical work in Excel (hypothesis testing, linear and multivariate regressions) and love it.  I use Excel 2003 with a capacity of 256 columns and 65,000 rows which can handle just about 100% of the data sets I use.  I understand Excel 2007 has extended that capacity by a huge amount (rows in the millions).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As Whuber mentions, Excel also serves as a starting platform for a multitude of pretty outstanding add-in software that are all pretty powerful and easy to use.  I am thinking of Crystal Ball and @Risk for Monte Carlo Simulation; XLStat for all around powerful stats and data analysis; What's Best for optimization.  And, the list goes on.  It's like Excel is the equivalent of an IPod or IPad with a zillion of pretty incredible Apps.  Granted the Excel Apps are not cheap.  But, for what they are capable of doing they are typically pretty great bargains.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as model documentation is concerned, it is so easy to insert a text box where you can literally write a book about your methodology, your sources, etc...  You can also insert comments in any cell.  So, if anything Excel is really good for facilitating embedded documentation.          &lt;/p&gt;&#xA;" OwnerUserId="1329" LastActivityDate="2010-10-07T21:36:51.820" CommentCount="2" />
  <row Id="3400" PostTypeId="1" AcceptedAnswerId="3411" CreationDate="2010-10-07T23:27:47.727" Score="8" ViewCount="272" Body="&lt;p&gt;&lt;strong&gt;Question:&lt;/strong&gt; From the standpoint of statistician (or a practitioner), can one infer causality using &lt;a href=&quot;http://en.wikipedia.org/wiki/Propensity_score&quot; rel=&quot;nofollow&quot;&gt;propensity scores&lt;/a&gt; with an observational study (&lt;strong&gt;not an experiment&lt;/strong&gt;)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please, do not want to start a flame war or a fanatical debate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Background:&lt;/strong&gt; Within our stat PhD program, we've only touched on causal inference through working groups and a few topic sessions.  However, there are some very prominent researchers in other departments (e.g. HDFS, Sociology) who are actively using them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've already witnessed some pretty heated debate on this issue. It is not my intention to start one here.  That said, what references have you encountered?  What viewpoints do you have?  For example, one argument I've heard against propensity scores as a causal inference technique is that one can never infer causality due omitted variable bias -- if you leave out something important, you break the causal chain. Is this an unresolvable problem?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Disclaimer:&lt;/strong&gt; This question may not have a correct answer -- completely cool with clicking cw, but I'm personally very interested in the responses and would be happy with a few good references which include real-world examples.&lt;/p&gt;&#xA;" OwnerUserId="1499" LastActivityDate="2010-10-14T09:10:27.027" CommunityOwnedDate="2010-10-07T23:27:47.727" Title="From a statistical perspective, can one infer causality using propensity scores with an observational study?" Tags="&lt;causal-inference&gt;&lt;propensity-scores&gt;" AnswerCount="4" />
  <row Id="3402" PostTypeId="1" AcceptedAnswerId="3403" CreationDate="2010-10-08T00:51:55.783" Score="7" ViewCount="203" Body="&lt;p&gt;I'm a newbie at stats, so if I make any mistaken assumptions here please tell me.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There's a population &lt;code&gt;N&lt;/code&gt; of people. (For example &lt;code&gt;N&lt;/code&gt; can be 1,000,000.) Some of the people are redheads. I take a sample &lt;code&gt;n&lt;/code&gt; of people (say 10,) and find that &lt;code&gt;j&lt;/code&gt; of them are redheads.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What can I say about the general proportion of redheads in the population? I mean, my best approximation is probably &lt;code&gt;j/n&lt;/code&gt;, but what would be the standard deviation of that approximation?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By the way, what is the accepted term for this?&lt;/p&gt;&#xA;" OwnerUserId="1523" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-10-08T02:39:46.587" LastActivityDate="2010-10-08T23:30:44.833" Title="What's the accuracy of data obtained through a random sample?" Tags="&lt;standard-deviation&gt;&lt;sample-size&gt;&lt;binomial&gt;&lt;standard-error&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="1" />
  <row Id="3403" PostTypeId="2" ParentId="3402" CreationDate="2010-10-08T01:01:57.537" Score="6" ViewCount="" Body="&lt;p&gt;You can think of this as a binomial trial -- your trials are sampling &quot;redhead&quot; or &quot;not readhead&quot;.  In which case, you can build a confidence interval for your sample proportion ($j/n$) as documented on Wikipedia:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval&quot; rel=&quot;nofollow&quot;&gt;Binomial proportion confidence interval&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;A 95% confidence interval basically says that, using the same sampling algorithm, if you repeated this 100 times, the true proportion would lie in the stated interval 95 times.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Update&lt;/em&gt; By the way, I think the term you're looking for might be &lt;a href=&quot;http://en.wikipedia.org/wiki/Standard_error_%28statistics%29&quot; rel=&quot;nofollow&quot;&gt;standard error&lt;/a&gt; which is the standard deviation of the sampled proportions.  In this case, it's $\sqrt{{p (1-p)} \over {n}}$ where $p$ is your estimated proportion.  Note that as $n$ increases, the standard error decreases.&lt;/p&gt;&#xA;" OwnerUserId="251" LastEditorUserId="251" LastEditorDisplayName="" LastEditDate="2010-10-08T01:12:34.190" LastActivityDate="2010-10-08T01:12:34.190" CommentCount="6" />
  <row Id="3404" PostTypeId="2" ParentId="3400" CreationDate="2010-10-08T01:39:26.280" Score="4" ViewCount="" Body="&lt;p&gt;Only a prospective randomized trial can determine causality. In observational studies, there will always be the chance of an unmeasured or unknown covariate which makes ascribing causality impossible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, observational trials can provide evidence of a strong association between x and y, and are therefore useful for hypothesis generation. These hypotheses then need to be confirmed with a randomized trial.&lt;/p&gt;&#xA;" OwnerUserId="561" LastActivityDate="2010-10-08T01:39:26.280" CommunityOwnedDate="2010-10-08T01:39:26.280" CommentCount="1" />
  <row Id="3405" PostTypeId="2" ParentId="3392" CreationDate="2010-10-08T02:35:37.343" Score="9" ViewCount="" Body="&lt;p&gt;An interesting paper about using Excel in a Bioinformatics setting is:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Mistaken Identifiers: Gene name errors&#xA;  can be introduced inadvertently when&#xA;  using Excel in bioinformatics, BMC&#xA;  Bioinformatics, 2004 (&lt;a href=&quot;http://www.biomedcentral.com/1471-2105/5/80&quot; rel=&quot;nofollow&quot;&gt;link&lt;/a&gt;).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This short paper describes the problem of automatic type conversions in Excel (in particular &lt;a href=&quot;http://www.biomedcentral.com/1471-2105/5/80/figure/F1&quot; rel=&quot;nofollow&quot;&gt;date&lt;/a&gt; and floating point conversions). For example, the gene name Sept2 is converted to 2-Sept. You can actually find this error in &lt;a href=&quot;http://www.biomedcentral.com/1471-2105/5/80/figure/F2&quot; rel=&quot;nofollow&quot;&gt;online databases&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using Excel to manage medium to large amounts of data is dangerous. Mistakes can easily creep in without the user noticing.&lt;/p&gt;&#xA;" OwnerUserId="8" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-10-08T13:01:56.017" LastActivityDate="2010-10-08T13:01:56.017" CommentCount="6" />
  <row Id="3406" PostTypeId="2" ParentId="3386" CreationDate="2010-10-08T05:44:57.060" Score="2" ViewCount="" Body="&lt;p&gt;I wrote a post a while back on &lt;a href=&quot;http://jeromyanglim.blogspot.com/2009/12/multinomial-logistic-regression-spss.html&quot; rel=&quot;nofollow&quot;&gt;multinomial logistic regression resources in SPSS&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-10-08T05:44:57.060" />
  <row Id="3407" PostTypeId="1" AcceptedAnswerId="3410" CreationDate="2010-10-08T07:59:32.177" Score="8" ViewCount="331" Body="&lt;p&gt;I am building an android application that records accelerometer data during sleep, so as to analyze sleep trends and optionally wake the user near a desired time during light sleep.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have already built the component that collects and stores data, as well as the alarm. &lt;b&gt;I still need to tackle the beast of displaying and saving sleep data in a really meaningful and clear way, one that preferably also lends itself to analysis.&lt;/b&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A couple of pictures say two thousand words: (I can only post one link due to low rep)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's the unfiltered data, the sum of movement, collected at 30 second intervals&#xA;&lt;img src=&quot;http://electricsleep.googlecode.com/files/es-ss2.png&quot;/&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And the same data, smoothed by my own manifestation of moving average smoothing&#xA;&lt;img src=&quot;http://electricsleep.googlecode.com/files/es-ss1.png&quot;/&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;edit) both charts reflect calibration- there is a minimum 'noise' filter and maximum cutoff filter, as well as a alarm trigger level (the white line)&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unfortunately, neither of these are optimal solutions- the first is a little hard to understand for the average user, and the second, which is easier to understand, hides a lot of what is really going on. In particular the averaging removes the detail of spikes in movement- and I think those can be meaningful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So why are these charts so important? These time-series are displayed throughout the night as feedback to the user, and will be stored for reviewing/analysis later. The smoothing will ideally lower memory cost (both RAM and storage), and make rendering faster on these resource-starved phones/devices.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Clearly there is a better way to smooth the data- I have some vague ideas, such as using linear regression to figure out 'sharp' changes in movement and modifying my moving average smoothing according. I really need some more guidance and input before I dive headfirst into something that could be solved more optimally.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="1520" LastEditorUserId="1520" LastEditorDisplayName="" LastEditDate="2010-10-08T17:35:14.813" LastActivityDate="2010-10-08T17:35:14.813" Title="Smoothing time series data" Tags="&lt;time-series&gt;&lt;smoothing&gt;&lt;signal-processing&gt;&lt;java&gt;" AnswerCount="3" FavoriteCount="3" />
  <row Id="3408" PostTypeId="2" ParentId="3407" CreationDate="2010-10-08T09:35:30.260" Score="7" ViewCount="" Body="&lt;p&gt;There are many nonparametric smoothing algorithms including splines and loess. But they will smooth out the sudden changes too. So will low-pass filters. I think you might need a wavelet-based smoother which allows the sudden jumps but still smooths the noise.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Check out &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0521640687&quot; rel=&quot;nofollow&quot;&gt;Percival and Walden (2000)&lt;/a&gt; and the associated &lt;a href=&quot;http://cran.r-project.org/web/packages/wmtsa/index.html&quot; rel=&quot;nofollow&quot;&gt;R package&lt;/a&gt;. Although you want a java solution, the algorithms in the R package are open-source and you might be able to translate them.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-10-08T09:35:30.260" />
  <row Id="3409" PostTypeId="2" ParentId="3407" CreationDate="2010-10-08T09:54:24.820" Score="1" ViewCount="" Body="&lt;p&gt;This is somewhat tangential to what you're asking, but it may be worth taking a look at the Kalman filter.&lt;/p&gt;&#xA;" OwnerUserId="439" LastActivityDate="2010-10-08T09:54:24.820" />
  <row Id="3410" PostTypeId="2" ParentId="3407" CreationDate="2010-10-08T09:57:22.490" Score="8" ViewCount="" Body="&lt;p&gt;First up, the requirements for compression and analysis/presentation are not necessarily the same -- indeed, for analysis you might want to keep &lt;em&gt;all&lt;/em&gt; the raw data and have the ability to slice and dice it in various ways. And what works best for you will depend very much on what you want to get out of it. But there are a number of standard tricks that you could try:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Use differences rather than raw data&lt;/li&gt;&#xA;&lt;li&gt;Use thresholding to remove low-level noise. (Combine with differencing to ignore small changes.)&lt;/li&gt;&#xA;&lt;li&gt;Use &lt;em&gt;variance&lt;/em&gt; over some time window rather than average, to capture activity level rather than movement&lt;/li&gt;&#xA;&lt;li&gt;Change the time base from fixed intervals to variable length runs and accumulate into a single data point sequences of changes for which some criterion holds (eg, differences in same direction, up to some threshold)&lt;/li&gt;&#xA;&lt;li&gt;Transform data from real values to &lt;a href=&quot;http://en.wikipedia.org/wiki/Level_of_measurement&quot; rel=&quot;nofollow&quot;&gt;ordinal&lt;/a&gt; (eg low, medium, high); you could also do this on time bins rather than individual samples -- eg, activity level for each 5 minute stretch&lt;/li&gt;&#xA;&lt;li&gt;Use an appropriate &lt;a href=&quot;http://www.songho.ca/dsp/convolution/convolution.html&quot; rel=&quot;nofollow&quot;&gt;convolution kernel&lt;/a&gt;* to smooth more subtly than your moving average or pick out features of interest such as sharp changes.&lt;/li&gt;&#xA;&lt;li&gt;Use an &lt;a href=&quot;http://en.wikipedia.org/wiki/Fast_Fourier_transform&quot; rel=&quot;nofollow&quot;&gt;FFT&lt;/a&gt; library to calculate a &lt;a href=&quot;http://en.wikipedia.org/wiki/Frequency_spectrum#Spectrum_analysis&quot; rel=&quot;nofollow&quot;&gt;power spectrum&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The last may be a bit expensive for your purposes, but would probably give you some very useful presentation options, in terms of &quot;sleep rhythms&quot; and such. (I know next to nothing about Android, but it's &lt;em&gt;conceivable&lt;/em&gt; that some/many/all handsets might have built in DSP hardware that you can take advantage of.)&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;* Given how central convolution is to digital signal processing, it's surprisingly difficult to find an accessible intro online. Or at least in 3 minutes of googling. Suggestions welcome!&lt;/p&gt;&#xA;" OwnerUserId="174" LastEditorUserId="174" LastEditorDisplayName="" LastEditDate="2010-10-08T10:11:31.953" LastActivityDate="2010-10-08T10:11:31.953" />
  <row Id="3411" PostTypeId="2" ParentId="3400" CreationDate="2010-10-08T11:30:29.323" Score="6" ViewCount="" Body="&lt;p&gt;At the beginning of an article aiming at promoting the use of PSs in epidemiology, Oakes and Church (1) cited Hernán and Robins's claims about confounding effect in epidemiology (2):&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Can you guarantee that the results&#xA;  from your observational study are&#xA;  unaffected by unmeasured confounding?&#xA;  The only answer an epidemiologist can&#xA;  provide is ‘no’.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This is not just to say that we cannot ensure that results from observational studies are unbiased or useless (because, as @propofol said, their results can be useful for designing RCTs), but also that PSs do certainly not offer a complete solution to this problem, or at least do not necessarily yield better results than other matching or multivariate methods (see e.g. (10)).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Propensity scores (PS) are, by construction, &lt;em&gt;probabilistic&lt;/em&gt; not &lt;em&gt;causal&lt;/em&gt; indicators. The choice of the covariates that enter the propensity score function is a key element for ensuring its reliability, and their weakness, as has been said, mainly stands from not controlling for unobserved confounders (which is quite likely in retrospective or &lt;a href=&quot;http://en.wikipedia.org/wiki/Case-control_study&quot; rel=&quot;nofollow&quot;&gt;case-control&lt;/a&gt; studies). Others factors have to be considered: (a) model misspecification will impact direct effect estimates (not really more than in the OLS case, though), (b) there may be missing data at the level of the covariates, (c) PSs do not overcome synergistic effects which are know to affect causal interpretation (8,9).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for references, I found Roger Newson's slides -- &lt;a href=&quot;http://www.imperial.ac.uk/nhli/r.newson/miscdocs/causconf1.pdf&quot; rel=&quot;nofollow&quot;&gt;Causality, confounders, and propensity scores&lt;/a&gt; -- relatively well-balanced about the pros and cons of using propensity scores, with illustrations from real studies.&#xA;There were also several good papers discussing the use of propensity scores in observational studies or environmental epidemiology two years ago in &lt;em&gt;Statistics in Medicine&lt;/em&gt;, and I enclose a couple of them at the end (3-6). But I like Pearl's review (7) because it offers a larger perspective on causality issues (PSs are discussed p. 117 and 130). Obviously, you will find many more illustrations by looking at applied research. I would like to add two recent articles from William R Shadish that came across Andrew Gelman's website (11,12). The use of propensity scores is discussed, but the two papers more largely focus on causal inference in observational studies (and how it compare to randomized settings).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;References&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Oakes, J.M. and Church, T.R. (2007). &lt;a href=&quot;http://aje.oxfordjournals.org/content/165/10/1119.full&quot; rel=&quot;nofollow&quot;&gt;Invited Commentary: Advancing Propensity Score Methods in Epidemiology&lt;/a&gt;. &lt;em&gt;American Journal of Epidemiology&lt;/em&gt;, 165(10), 1119-1121.&lt;/li&gt;&#xA;&lt;li&gt;Hernan M.A. and Robins J.M. (2006). &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.158.2969&quot; rel=&quot;nofollow&quot;&gt;Instruments for causal inference: an epidemiologist's dream?&lt;/a&gt; &lt;em&gt;Epidemiology&lt;/em&gt;, 17, 360-72.&lt;/li&gt;&#xA;&lt;li&gt;Rubin, D. (2007). &lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pubmed/17072897&quot; rel=&quot;nofollow&quot;&gt;The design versus the analysis of observational studies for causal effects: Parallels with the design of randomized trials&lt;/a&gt;. &lt;em&gt;Statistics in Medicine&lt;/em&gt;, 26, 20–36.&lt;/li&gt;&#xA;&lt;li&gt;Shrier, I. (2008). &lt;a href=&quot;http://onlinelibrary.wiley.com/doi/10.1002/sim.3172/abstract&quot; rel=&quot;nofollow&quot;&gt;Letter to the editor&lt;/a&gt;. &lt;em&gt;Statistics in Medicine&lt;/em&gt;, 27, 2740–2741.&lt;/li&gt;&#xA;&lt;li&gt;Pearl, J. (2009). &lt;a href=&quot;ftp://ftp.cs.ucla.edu/pub/stat_ser/r345-sim.pdf&quot; rel=&quot;nofollow&quot;&gt;Remarks on the method of propensity score&lt;/a&gt;. &lt;em&gt;Statistics in Medicine&lt;/em&gt;, 28, 1415–1424.&lt;/li&gt;&#xA;&lt;li&gt;Stuart, E.A. (2008). &lt;a href=&quot;http://www.epi.msu.edu/janthony/requests/propensity/Stuart_Commentary_3.pdf&quot; rel=&quot;nofollow&quot;&gt;Developing practical recommendations for the use of propensity scores: Discussion of ‘A critical appraisal of propensity score matching in the medical literature between 1996 and 2003’ by Peter Austin&lt;/a&gt;. &lt;em&gt;Statistics in Medicine&lt;/em&gt;, 27, 2062–2065. &lt;/li&gt;&#xA;&lt;li&gt;Pearl, J. (2009). &lt;a href=&quot;http://www.i-journals.org/ss/viewarticle.php?id=57&quot; rel=&quot;nofollow&quot;&gt;Causal inference in statistics: An overview&lt;/a&gt;. &lt;em&gt;Statistics Surveys&lt;/em&gt;, 3, 96-146.&lt;/li&gt;&#xA;&lt;li&gt;Oakes, J.M. and Johnson, P.J. (2006). &lt;a href=&quot;http://www.google.fr/url?sa=t&amp;amp;source=web&amp;amp;cd=18&amp;amp;ved=0CFMQFjAHOAo&amp;amp;url=http%3A%2F%2Fwww.tc.umn.edu%2F~oakes007%2FFiles%2FBook%2FOakes%2520%26%2520Johnson%2520p-score%2520chapter.pdf&amp;amp;ei=iu6uTLSeC8jN4Abz-PH3Bg&amp;amp;usg=AFQjCNEse-1Vh8p9OFKkgLIk1Wq9Mn9b4A&amp;amp;sig2=nhrdB0PA0m3vNlIfKuR6pg&quot; rel=&quot;nofollow&quot;&gt;Propensity score matching for social epidemiology&lt;/a&gt;. In &lt;em&gt;Methods in Social Epidemiology&lt;/em&gt;, J.M. Oakes and S. Kaufman (Eds.), pp. 364-386. Jossez-Bass.&lt;/li&gt;&#xA;&lt;li&gt;Höfler, M (2005). &lt;a href=&quot;http://www.biomedcentral.com/1471-2288/5/28&quot; rel=&quot;nofollow&quot;&gt;Causal inference based on counterfactuals&lt;/a&gt;. &lt;em&gt;BMC Medical Research Methodology&lt;/em&gt;, 5, 28.&lt;/li&gt;&#xA;&lt;li&gt;Winkelmayer, W.C. and Kurth, T. (2004). &lt;a href=&quot;http://ndt.oxfordjournals.org/content/19/7/1671.full&quot; rel=&quot;nofollow&quot;&gt;Propensity scores: help or hype?&lt;/a&gt; &lt;em&gt;Nephrology Dialysis Transplantation&lt;/em&gt;, 19(7), 1671-1673.&lt;/li&gt;&#xA;&lt;li&gt;Shadish, W.R., Clark, M.H., and Steiner, P.M. (2008). &lt;a href=&quot;http://www.stat.columbia.edu/~cook/movabletype/mlm/shadish1.pdf&quot; rel=&quot;nofollow&quot;&gt;Can Nonrandomized Experiments Yield Accurate Answers? A Randomized Experiment Comparing Random and Nonrandom Assignments&lt;/a&gt;. &lt;em&gt;JASA&lt;/em&gt;, 103(484), 1334-1356.&lt;/li&gt;&#xA;&lt;li&gt;Cook, T.D., Shadish, W.R., and Wong, V.C. (2008). &lt;a href=&quot;http://www.stat.columbia.edu/~cook/movabletype/mlm/shadish2.pdf&quot; rel=&quot;nofollow&quot;&gt;Three Conditions under Which Experiments and Observational Studies Produce Comparable Causal Estimates: New Findings from Within-Study Comparisons&lt;/a&gt;. &lt;em&gt;Journal of Policy Analysis and Management&lt;/em&gt;, 27(4), 724–750.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-14T09:10:27.027" LastActivityDate="2010-10-14T09:10:27.027" CommunityOwnedDate="2010-10-08T11:30:29.323" CommentCount="1" />
  <row Id="3412" PostTypeId="1" CreationDate="2010-10-08T12:34:11.673" Score="11" ViewCount="353" Body="&lt;p&gt;Hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have an experiment that I'll try to abstract here.  Imagine I toss three white stones in front of you and ask you to make a judgment about their position.  I record a variety of properties of the stones and your response.   I do this over a number of subjects.  I generate two models.  One is that the nearest stone to you predicts your response, and the other is that the geometric center of the stones predicts your response.  So, using lmer in R I could write.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mNear   &amp;lt;- lmer(resp ~ nearest + (1|subject), REML = FALSE)&#xA;mCenter &amp;lt;- lmer(resp ~ center  + (1|subject), REML = FALSE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE AND CHANGE - more direct version that incorporates several helpful comments&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I could try&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;anova(mNear, mCenter)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Which is incorrect, of course, because they're not nested and I can't really compare them that way.  I was expecting anova.mer to throw an error but it didn't.  But the possible nesting that I could try here isn't natural and still leaves me with somewhat less analytical statements.  When models are nested naturally (e.g. quadratic on linear) the test is only one way.  But in this case what would it mean to have asymmetric findings?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, I could make a model three:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mBoth &amp;lt;- lmer(resp ~ center + nearest + (1|subject), REML = FALSE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then I can anova.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;anova(mCenter, mBoth)&#xA;anova(mNearest, mBoth)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is fair to do and now I find that the center adds to the nearest effect (the second command) but BIC actually goes up when nearest is added to center (correction for the lower parsimony).  This confirms what was suspected.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But is finding this sufficient?  And is this fair when center and nearest are so highly correlated?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a better way to analytically compare the models when it's not about adding and subtracting explanatory variables (degrees of freedom)?&lt;/p&gt;&#xA;" OwnerUserId="601" LastEditorUserId="601" LastEditorDisplayName="" LastEditDate="2011-03-13T16:27:57.167" LastActivityDate="2011-03-13T16:27:57.167" Title="Comparing mixed effect models with the same number of degrees of freedom" Tags="&lt;r&gt;&lt;mixed-model&gt;&lt;model-selection&gt;" AnswerCount="4" CommentCount="2" FavoriteCount="2" />
  <row Id="3413" PostTypeId="1" AcceptedAnswerId="3414" CreationDate="2010-10-08T13:24:33.177" Score="3" ViewCount="307" Body="&lt;p&gt;I am looking for the Hurst exponent calculation methodology. Please suggest online materials / methodology papers.&lt;/p&gt;&#xA;" OwnerUserId="1250" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-10-20T21:10:57.607" LastActivityDate="2010-10-20T21:10:57.607" Title="Hurst exponent calculation methodology" Tags="&lt;big-list&gt;&lt;fractal&gt;" AnswerCount="2" CommentCount="0" />
  <row Id="3414" PostTypeId="2" ParentId="3413" CreationDate="2010-10-08T13:33:08.560" Score="5" ViewCount="" Body="&lt;p&gt;The calculation is covered on &lt;a href=&quot;http://en.wikipedia.org/wiki/Hurst_exponent&quot; rel=&quot;nofollow&quot;&gt;the related wikipedia page&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;R has several implementations for this:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The &lt;a href=&quot;http://cran.r-project.org/web/packages/fArma&quot; rel=&quot;nofollow&quot;&gt;fArma package&lt;/a&gt; provides 10 different functions to estimate the Hurst exponent (see &lt;a href=&quot;http://help.rmetrics.org/fArma/html/LrdModelling.html&quot; rel=&quot;nofollow&quot;&gt;LrdModelling&lt;/a&gt;).&lt;/li&gt;&#xA;&lt;li&gt;The Rwave package has the &lt;a href=&quot;http://www.stat.ucl.ac.be/ISdidactique/Rhelp/library/Rwave/html/hurst.est.html&quot; rel=&quot;nofollow&quot;&gt;hurst.est() function&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;li&gt;The fractal package has the &lt;a href=&quot;http://bm2.genes.nig.ac.jp/RGM2/R_current/library/fractal/man/hurstACVF.html&quot; rel=&quot;nofollow&quot;&gt;hurstACVF() function&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;li&gt;The &lt;a href=&quot;http://cran.r-project.org/web/packages/dvfBm/&quot; rel=&quot;nofollow&quot;&gt;dvfBm package&lt;/a&gt; is intended entirely for this purpose: &quot;Hurst exponent estimation of a fractional Brownian motion by using discrete variations methods in presence of outliers and/or an additive noise&quot;.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The methods covered by fArma are taken from the &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.55.8251&quot; rel=&quot;nofollow&quot;&gt;&quot;Estimators for Long-Range Dependence: An Empirical Study&quot;&lt;/a&gt; (Taqqu, Teverovsky, Willinger 1995).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Edit:&lt;/em&gt; Just to add from the M. Tibbits comment below, you can find the &lt;a href=&quot;http://www.mathworks.com/matlabcentral/fileexchange/9842&quot; rel=&quot;nofollow&quot;&gt;Hurst exponent&lt;/a&gt; code for Matlab offered under a BSD license.  The description:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;This is an implementation of the Hurst&#xA;  exponent calculation that is smaller,&#xA;  simpler, and quicker than most others.&#xA;  It does a dispersional analysis on the&#xA;  data and then uses Matlab's polyfit to&#xA;  estimate the Hurst exponent. It comes&#xA;  with a test driver that you can&#xA;  delete.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="5" LastEditorUserId="5" LastEditorDisplayName="" LastEditDate="2010-10-08T15:05:16.480" LastActivityDate="2010-10-08T15:05:16.480" CommentCount="4" />
  <row Id="3415" PostTypeId="2" ParentId="3412" CreationDate="2010-10-08T13:41:22.563" Score="5" ViewCount="" Body="&lt;p&gt;Still, you can compute confidence intervals for your fixed effects, and report AIC or BIC (see e.g. &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.133.2052&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;Cnann et al.&lt;/a&gt;, Stat Med 1997 16: 2349). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, you may be interested in taking a look at &lt;a href=&quot;http://star.psy.ohio-state.edu/coglab/People/roger/pdf/mathpsy04.pdf&quot; rel=&quot;nofollow&quot;&gt;Assessing model mimicry using the parametric bootstrap&lt;/a&gt;, from Wagenmakers et al. which seems to more closely resemble your initial question about assessing the quality of two competing models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Otherwise, the two papers about measures of explained variance in LMM that come to my mind are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Lloyd J. Edwards, Keith E. Muller, Russell D. Wolfinger, Bahjat F. Qaqish and Oliver Schabenberger (2008). &lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2587505/&quot; rel=&quot;nofollow&quot;&gt;An R2 statistic for fixed effects in the linear mixed model&lt;/a&gt;, &lt;em&gt;Statistics in Medicine&lt;/em&gt;, 27(29), 6137–6157.&lt;/li&gt;&#xA;&lt;li&gt;Ronghui Xu (2003). Measuring explained variation in linear mixed effects models, &lt;em&gt;Statistics in Medicine&lt;/em&gt;, 22(22), 3527–3541.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;But maybe there are better options.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-08T13:49:37.653" LastActivityDate="2010-10-08T13:49:37.653" />
  <row Id="3416" PostTypeId="2" ParentId="3412" CreationDate="2010-10-08T13:53:52.570" Score="2" ViewCount="" Body="&lt;p&gt;I do not know R well enough to parse your code but here is one idea: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Estimate a model where you have both center and near as covariates (call this mBoth). Then mCenter and mNear are nested in mBoth and you could use mBoth as a benchmark to compare the relative performance of mCenter and mNear.&lt;/p&gt;&#xA;" LastActivityDate="2010-10-08T13:53:52.570" CommentCount="3" />
  <row Id="3417" PostTypeId="2" ParentId="3287" CreationDate="2010-10-08T16:16:58.473" Score="2" ViewCount="" Body="&lt;p&gt;I'm an ecologist, so I apologise in advance is this sounds a bit strange :-)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I like to think of these plots in terms of weighted averages. The region points are at the weighted averages of the smoking status classes and vice versa.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem with the above figure is the axis scaling and the fact that you can't display all the relationships (chi-square distance between regions and chi-square distance between smoking status) on the one figure. By the looks of it, the figure is using a what is known as symmetric scaling which has been shown to be a good compromise preserving as much of the information in the sets of scores as possible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not familiar with the &lt;code&gt;ca&lt;/code&gt; package but I am with the vegan package and it's &lt;code&gt;cca&lt;/code&gt; function:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(vegan)&#xA;df &amp;lt;- data.frame(df)&#xA;ord &amp;lt;- cca(df)&#xA;plot(ord, scaling = 3)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The last plot is a bit easier to read than the one you show but AFAICT they are the same (or at least similarly scaled).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I would say that occasional smokers are lower in number than expected in QC, BC and AB, and most associated with ON, but that in all regions, occasional smokers are low in number - they differ markedly from the expected number.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, there is a single dominant &quot;gradient&quot; or axis of variation in these data and as the second axis represents so little variation, I would likely not interpret this component at all.&lt;/p&gt;&#xA;" OwnerUserId="1390" LastEditorUserId="1390" LastEditorDisplayName="" LastEditDate="2010-10-08T16:22:10.900" LastActivityDate="2010-10-08T16:22:10.900" CommentCount="1" />
  <row Id="3419" PostTypeId="1" AcceptedAnswerId="3673" CreationDate="2010-10-08T17:52:20.097" Score="5" ViewCount="194" Body="&lt;p&gt;There are umpteen million research papers regarding relationships between various patient attributes (e.g. how does gene x affect condition y?). What I am interested in though is a distance metric between patients in toto. Sort of like if I were constructing a dating site, I'd want to know how similar two people are. (Except in this case &quot;similarity&quot; means health similarity rather than personality similarity or whatever dating sites look at.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could anyone point me to research regarding this problem? So far the only paper I've found that really attempts to tackle this is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Melton, G. B., S. Parsons, F. P. Morrison, A. S. Rothschild, M. Markatou, and G. Hripcsak. “Inter-patient distance metrics using SNOMED CT defining relationships.” Journal of Biomedical Informatics 39, no. 6 (2006): 697-705.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To clarify my question (because it's slightly different than many on this site): I am not asking &quot;I have some data set, how can I analyze it?&quot; I am asking &quot;if I gave a doctor a data set, how would &lt;em&gt;they&lt;/em&gt; analyze it?&quot; I don't particularly care if there is some relation between attribute X and Y. What I care about is if a &lt;em&gt;doctor thinks&lt;/em&gt; there is a relation between X and Y. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I.e. my question is: I give a doctor two patient charts. How can I predict what they think the similarity is? Do they look at certain attributes? Is it even possible to make a statement like &quot;they are .9 similar?&quot; Is a better statement &quot;they are .9 similar on dimension X and .8 similar on dimension Y?&quot; How are chronic conditions different than temporary ones? etc. etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is maybe on the fringes of what this site is intended for, but I'm hoping someone has dealt with this and can point me in a good direction, even though it's not a question about a given statistical technique per se.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT 2:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you all for your suggestions. However, I was really looking for people who have done this work before - I don't have access to a lot of data, so I was hoping to find someone who did have these data and utilize their conclusions.&lt;/p&gt;&#xA;" OwnerUserId="900" LastEditorUserId="900" LastEditorDisplayName="" LastEditDate="2010-10-15T16:18:12.350" LastActivityDate="2010-10-17T19:06:12.167" Title="Patient distance metrics" Tags="&lt;clustering&gt;&lt;biostatistics&gt;" AnswerCount="5" />
  <row Id="3420" PostTypeId="2" ParentId="3296" CreationDate="2010-10-08T19:38:39.063" Score="1" ViewCount="" Body="&lt;p&gt;Addressing the issue mentioned under Update 2.  You are dealing with outliers.  Those outliers have a significant impact on your Logistic Regression coefficients.  By removing them, you found that your models performed better on the validation set.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does it mean that the outliers are &quot;bad&quot;?  No.  It means that they are influential.  There are several measures of statistical distances to confirm how far away and influential such outliers are.  Those include Cook's D and DFFITS. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having identified the trouble makers, you are struggling with whether to keep them in or not.  Ultimately, this may be a qualitative judgment rather than a statistical question.  Here are a couple of investigative questions that may be helpful in making this qualitative decision: &#xA;1) First, are the outliers truly bad due to poor measurements?&#xA;2) Is it more important for your models to be correct in the tails where outliers reside or be more accurate in the vast majority of the cases?  &lt;/p&gt;&#xA;" OwnerUserId="1329" LastActivityDate="2010-10-08T19:38:39.063" />
  <row Id="3421" PostTypeId="2" ParentId="3419" CreationDate="2010-10-08T20:05:14.780" Score="2" ViewCount="" Body="&lt;p&gt;The whole field of &lt;a href=&quot;http://www.amazon.com/s/ref=nb_sb_noss?url=search-alias%3Dstripbooks&amp;amp;field-keywords=Cluster+analysis&amp;amp;x=0&amp;amp;y=0&quot; rel=&quot;nofollow&quot;&gt;Cluster Analysis&lt;/a&gt; is relevant to your concept of multi-variable statistical distance.  The linked book on the subject is very short and pretty good.   &lt;/p&gt;&#xA;" OwnerUserId="1329" LastActivityDate="2010-10-08T20:05:14.780" />
  <row Id="3422" PostTypeId="2" ParentId="3412" CreationDate="2010-10-08T20:58:17.027" Score="3" ViewCount="" Body="&lt;p&gt;Following ronaf's suggestion leads to a more recent paper by Vuong for a Likelihood Ratio Test on nonnested models.  It's based on the KLIC (Kullback-Leibler Information Criterion) which is similar to the AIC in that it minimizes the KL distance.  But it sets up a probabilistic specification for the hypothesis so the use of the LRT leads to a more principled comparison.  A more accessible version of the Cox and Vuong tests is presented by Clarke et al; in particular see Figure 3 which presents the algorithm for computing the Vuong LRT test.  &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://ideas.repec.org/a/ecm/emetrp/v57y1989i2p307-33.html&quot; rel=&quot;nofollow&quot;&gt;Likelihood Ratio Tests for Model Selection and Non-nested Hypotheses&lt;/a&gt; (Vuong, 1999)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.rochester.edu/college/psc/clarke/450315.pdf&quot; rel=&quot;nofollow&quot;&gt;Testing Nonnested Models of International Relations: Reevaluating Realism&lt;/a&gt; (Clarke et al, 2000)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;It seems there are R implementations of the Vuong test in other models, but not lmer.  Still, the outline mentioned above should be sufficient to implement one.  I don't think you can obtain the likelihood evaluated at each data point from lmer as required for the computation.  In a note on sig-ME, Douglas Bates has &lt;a href=&quot;https://stat.ethz.ch/pipermail/r-sig-mixed-models/2007q3/000246.html&quot; rel=&quot;nofollow&quot;&gt;some pointers&lt;/a&gt; that might be helpful (in particular, the &lt;a href=&quot;http://cran.r-project.org/web/packages/lme4/vignettes/Theory.pdf&quot; rel=&quot;nofollow&quot;&gt;vignette&lt;/a&gt; he mentions).  &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Older&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another option is to consider the fitted values from the models in a test of prediction accuracy.  The Williams-Kloot statistic may be appropriate here.  The basic approach is to regress the actual values against a linear combination of the fitted values from the two models and test the slope:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.jstor.org/pss/2334426&quot; rel=&quot;nofollow&quot;&gt;A Test for Discriminating Between Models&lt;/a&gt; (Atikinson, 1969)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.springerlink.com/content/n607872562066081/&quot; rel=&quot;nofollow&quot;&gt;Growth and the Welfare State in the EU: A Causality Analysis&lt;/a&gt; (Herce et al, 2001)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The first paper describes the test (and others), while the second has an application of it in an econometric panel model.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;When using &lt;code&gt;lmer&lt;/code&gt; and comparing AICs, the function's default is to use the REML method (Restricted Maximum Likelihood).  This is fine for obtaining less biased estimates, but when comparing models, you should re-fit with &lt;code&gt;REML=FALSE&lt;/code&gt; which uses the Maximum Likelihood method for fitting.  The &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1441903178&quot; rel=&quot;nofollow&quot;&gt;Pinheiro/Bates book&lt;/a&gt; mentions some condition under which it's OK to compare AIC/Likelihood with either REML or ML, and these may very well apply in your case.  However, the general recommendation is to simply re-fit.  For example, see Douglas Bates' post here:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://finzi.psych.upenn.edu/R/Rhelp02a/archive/117488.html&quot; rel=&quot;nofollow&quot;&gt;How can I extract the AIC score from a mixed model object produced using lmer?&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="251" LastEditorUserId="251" LastEditorDisplayName="" LastEditDate="2010-10-10T00:10:47.563" LastActivityDate="2010-10-10T00:10:47.563" CommentCount="3" />
  <row Id="3423" PostTypeId="1" CreationDate="2010-10-08T21:01:20.033" Score="6" ViewCount="113" Body="&lt;p&gt;I'm working on a web app, and I'm creating some data viz tools for it. For one particular series, I've got an extremely wide variance in data values (0 to millions). We're using a column chart to view the data now, which of course results in some columns that are a pixel high or smaller. We already have some ways to slice the data that helps a bit, but I was wondering if there were different kinds of visualizations out there in common use that deal with this type of situation better. And if so, if there were JS libraries that help implement them.&lt;/p&gt;&#xA;" OwnerUserId="1531" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-10-09T15:40:21.837" LastActivityDate="2010-10-09T15:40:21.837" Title="Recommendations for visualization type when data has an extremely wide variance" Tags="&lt;data-visualization&gt;" AnswerCount="1" CommentCount="0" FavoriteCount="1" />
  <row Id="3424" PostTypeId="2" ParentId="3423" CreationDate="2010-10-08T21:20:22.510" Score="7" ViewCount="" Body="&lt;p&gt;A standard approach to dealing with data that has a wide variance is to use a &lt;a href=&quot;http://en.wikipedia.org/wiki/Logarithmic_scale&quot; rel=&quot;nofollow&quot;&gt;log scale&lt;/a&gt; (or some other kind of scaling approach) regardless of the visualization itself.  This could be applied in any graphical package (including a JS library like &lt;a href=&quot;http://vis.stanford.edu/protovis/&quot; rel=&quot;nofollow&quot;&gt;Protovis&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another strategy is to use bands, and fold the data over several times (as &lt;a href=&quot;http://vis.stanford.edu/protovis/ex/horizon.html&quot; rel=&quot;nofollow&quot;&gt;in this example&lt;/a&gt;), although personally I find this approach to be harder to read.  This ends up &lt;a href=&quot;http://vis.berkeley.edu/papers/horizon/&quot; rel=&quot;nofollow&quot;&gt;looking like&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/j03UP.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="5" LastEditorUserId="5" LastEditorDisplayName="" LastEditDate="2010-10-08T21:36:33.643" LastActivityDate="2010-10-08T21:36:33.643" CommentCount="5" />
  <row Id="3425" PostTypeId="1" AcceptedAnswerId="3433" CreationDate="2010-10-08T21:31:00.867" Score="9" ViewCount="400" Body="&lt;p&gt;I am not sure how this should be termed, so please correct me if you know a better term.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've got two lists.  One of 55 items (e.g: a vector of strings), the other of 92.  The item names are similar but not identical.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wish to find the best candidate*&lt;em&gt;s&lt;/em&gt;* in the 92 list to the items in the 55 list (I will then go through it and pick the correct fitting).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can it be done?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ideas I had where to:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;See all the ones that match (using something list ?match)&lt;/li&gt;&#xA;&lt;li&gt;Try a distance matrix between the strings vectors, but I am not sure how to best define it (number of identical letters, what about order of strings?)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;So what package/functions/field-of-research deals with such a task, and how?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Update: Here is an example of the vectors I wish to match&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;vec55 &amp;lt;- c(&quot;Aeropyrum pernix&quot;, &quot;Archaeoglobus fulgidus&quot;, &quot;Candidatus_Korarchaeum_cryptofilum&quot;, &#xA;&quot;Candidatus_Methanoregula_boonei_6A8&quot;, &quot;Cenarchaeum_symbiosum&quot;, &#xA;&quot;Desulfurococcus_kamchatkensis&quot;, &quot;Ferroplasma acidarmanus&quot;, &quot;Haloarcula_marismortui_ATCC_43049&quot;, &#xA;&quot;Halobacterium sp.&quot;, &quot;Halobacterium_salinarum_R1&quot;, &quot;Haloferax volcanii&quot;, &#xA;&quot;Haloquadratum_walsbyi&quot;, &quot;Hyperthermus_butylicus&quot;, &quot;Ignicoccus_hospitalis_KIN4&quot;, &#xA;&quot;Metallosphaera_sedula_DSM_5348&quot;, &quot;Methanobacterium thermautotrophicus&quot;, &#xA;&quot;Methanobrevibacter_smithii_ATCC_35061&quot;, &quot;Methanococcoides_burtonii_DSM_6242&quot;&#xA;)&#xA;vec91 &amp;lt;- c(&quot;Acidilobus saccharovorans 345-15&quot;, &quot;Aciduliprofundum boonei T469&quot;, &#xA;&quot;Aeropyrum pernix K1&quot;, &quot;Archaeoglobus fulgidus DSM 4304&quot;, &quot;Archaeoglobus profundus DSM 5631&quot;, &#xA;&quot;Caldivirga maquilingensis IC-167&quot;, &quot;Candidatus Korarchaeum cryptofilum OPF8&quot;, &#xA;&quot;Candidatus Methanoregula boonei 6A8&quot;, &quot;Cenarchaeum symbiosum A&quot;, &#xA;&quot;Desulfurococcus kamchatkensis 1221n&quot;, &quot;Ferroglobus placidus DSM 10642&quot;, &#xA;&quot;Halalkalicoccus jeotgali B3&quot;, &quot;Haloarcula marismortui ATCC 43049&quot;, &#xA;&quot;Halobacterium salinarum R1&quot;, &quot;Halobacterium sp. NRC-1&quot;, &quot;Haloferax volcanii DS2&quot;, &#xA;&quot;Halomicrobium mukohataei DSM 12286&quot;, &quot;Haloquadratum walsbyi DSM 16790&quot;, &#xA;&quot;Halorhabdus utahensis DSM 12940&quot;, &quot;Halorubrum lacusprofundi ATCC 49239&quot;, &#xA;&quot;Haloterrigena turkmenica DSM 5511&quot;, &quot;Hyperthermus butylicus DSM 5456&quot;, &#xA;&quot;Ignicoccus hospitalis KIN4/I&quot;, &quot;Ignisphaera aggregans DSM 17230&quot;, &#xA;&quot;Metallosphaera sedula DSM 5348&quot;, &quot;Methanobrevibacter ruminantium M1&quot;, &#xA;&quot;Methanobrevibacter smithii ATCC 35061&quot;, &quot;Methanocaldococcus fervens AG86&quot;, &#xA;&quot;Methanocaldococcus infernus ME&quot;, &quot;Methanocaldococcus jannaschii DSM 2661&quot;, &#xA;&quot;Methanocaldococcus sp. FS406-22&quot;, &quot;Methanocaldococcus vulcanius M7&quot;, &#xA;&quot;Methanocella paludicola SANAE&quot;, &quot;Methanococcoides burtonii DSM 6242&quot;, &#xA;&quot;Methanococcus aeolicus Nankai-3&quot;, &quot;Methanococcus maripaludis C5&quot;, &#xA;&quot;Methanococcus maripaludis C6&quot;, &quot;Methanococcus maripaludis C7&quot;, &#xA;&quot;Methanococcus maripaludis S2&quot;, &quot;Methanococcus vannielii SB&quot;, &#xA;&quot;Methanococcus voltae A3&quot;, &quot;Methanocorpusculum labreanum Z&quot;, &#xA;&quot;Methanoculleus marisnigri JR1&quot;, &quot;Methanohalobium evestigatum Z-7303&quot;, &#xA;&quot;Methanohalophilus mahii DSM 5219&quot;, &quot;Methanoplanus petrolearius DSM 11571&quot;, &#xA;&quot;Methanopyrus kandleri AV19&quot;, &quot;Methanosaeta thermophila PT&quot;, &#xA;&quot;Methanosarcina acetivorans C2A&quot;, &quot;Methanosarcina barkeri str. Fusaro&quot;, &#xA;&quot;Methanosarcina mazei Go1&quot;, &quot;Methanosphaera stadtmanae DSM 3091&quot;, &#xA;&quot;Methanosphaerula palustris E1-9c&quot;, &quot;Methanospirillum hungatei JF-1&quot;, &#xA;&quot;Methanothermobacter marburgensis str. Marburg&quot;, &quot;Methanothermobacter thermautotrophicus str. Delta H&quot;, &#xA;&quot;Nanoarchaeum equitans Kin4-M&quot;, &quot;Natrialba magadii ATCC 43099&quot;, &#xA;&quot;Natronomonas pharaonis DSM 2160&quot;, &quot;Nitrosopumilus maritimus SCM1&quot;, &#xA;&quot;Picrophilus torridus DSM 9790&quot;, &quot;Pyrobaculum aerophilum str. IM2&quot;, &#xA;&quot;Pyrobaculum arsenaticum DSM 13514&quot;, &quot;Pyrobaculum calidifontis JCM 11548&quot;, &#xA;&quot;Pyrobaculum islandicum DSM 4184&quot;, &quot;Pyrococcus abyssi GE5&quot;, &quot;Pyrococcus furiosus DSM 3638&quot;, &#xA;&quot;Pyrococcus horikoshii OT3&quot;, &quot;Staphylothermus hellenicus DSM 12710&quot;, &#xA;&quot;Staphylothermus marinus F1&quot;, &quot;Sulfolobus acidocaldarius DSM 639&quot;, &#xA;&quot;Sulfolobus islandicus L.D.8.5&quot;, &quot;Sulfolobus islandicus L.S.2.15&quot;, &#xA;&quot;Sulfolobus islandicus M.14.25&quot;, &quot;Sulfolobus islandicus M.16.27&quot;, &#xA;&quot;Sulfolobus islandicus M.16.4&quot;, &quot;Sulfolobus islandicus Y.G.57.14&quot;, &#xA;&quot;Sulfolobus islandicus Y.N.15.51&quot;, &quot;Sulfolobus solfataricus P2&quot;, &#xA;&quot;Sulfolobus tokodaii str. 7&quot;, &quot;Thermococcus gammatolerans EJ3&quot;, &#xA;&quot;Thermococcus kodakarensis KOD1&quot;, &quot;Thermococcus onnurineus NA1&quot;, &#xA;&quot;Thermococcus sibiricus MM 739&quot;, &quot;Thermofilum pendens Hrk 5&quot;, &#xA;&quot;Thermoplasma acidophilum DSM 1728&quot;, &quot;Thermoplasma volcanium GSS1&quot;, &#xA;&quot;Thermoproteus neutrophilus V24Sta&quot;, &quot;Thermosphaera aggregans DSM 11486&quot;, &#xA;&quot;Vulcanisaeta distributa DSM 14429&quot;, &quot;uncultured methanogenic archaeon RC-I&quot;&#xA;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(Thanks)&lt;/p&gt;&#xA;" OwnerUserId="253" LastEditorUserId="253" LastEditorDisplayName="" LastEditDate="2010-10-09T13:16:28.960" LastActivityDate="2010-10-09T20:14:40.313" Title="How to quasi match two vectors of strings (in R) ?" Tags="&lt;r&gt;&lt;text-mining&gt;" AnswerCount="4" CommentCount="3" FavoriteCount="4" />
  <row Id="3426" PostTypeId="2" ParentId="2948" CreationDate="2010-10-08T21:42:44.397" Score="1" ViewCount="" Body="&lt;p&gt;I've an java implementation for non-overlapping, weighted/unweighted network that could probably handle 3 million nodes (I've tested it for a million node dataset). However, it works like k-means, and needs the number of partitions to be detected as an input (k in kmeans). You can find more info &lt;a href=&quot;http://www.google.ca/url?sa=t&amp;amp;source=web&amp;amp;cd=2&amp;amp;sqi=2&amp;amp;ved=0CBgQhgIwAQ&amp;amp;url=http%3A%2F%2Frepository.library.ualberta.ca%2Fdspace%2Fbitstream%2F10048%2F1529%2F1%2Fthesis.pdf&amp;amp;ei=T4-vTM2INpKCsQPGxfH9Aw&amp;amp;usg=AFQjCNE97bt6zDadI3BU0jfgbJz8C2i9ng&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;. &#xA;Email me ( rabbanyk- at - ualberta - dot - ca ) if it could help. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cheers,&lt;/p&gt;&#xA;" LastActivityDate="2010-10-08T21:42:44.397" />
  <row Id="3427" PostTypeId="2" ParentId="3425" CreationDate="2010-10-08T21:45:29.480" Score="9" ViewCount="" Body="&lt;p&gt;There are many ways to measure distances between two strings. Two important (standard) approaches widely implemented in R are the Levenshtein and the Hamming distance. The former is avalaible in package 'MiscPsycho' and the latter in 'e1071'. Using these, i would simply compute a 92 by 55 matrix of pairwise distances, then proceed from there (i.e. the best candidate match for string &quot;1&quot; in list 1 is the string &quot;x&quot; from list 2 with smallest distance to string &quot;1&quot;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively, there is a function compare() in package RecordLinkage that seems to be designed to do what you want and uses the so called Jaro-Winkler &lt;a href=&quot;http://en.wikipedia.org/wiki/Jaro%E2%80%93Winkler_distance&quot; rel=&quot;nofollow&quot;&gt;distance&lt;/a&gt; which seems more appropriate for the task at hand, but i've had no experience with it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: i'm editing my answer to include Brandon's comment as well as Tal's code, to find a match to &quot;Aeropyrum pernix&quot;, the first entry of &lt;strong&gt;vec55&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;agrep(vec55[1],vec91,ignore.case=T,value=T,max.distance = 0.1, useBytes = FALSE)&#xA;[1] &quot;Aeropyrum pernix K1&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="603" LastEditorUserId="603" LastEditorDisplayName="" LastEditDate="2010-10-09T20:14:40.313" LastActivityDate="2010-10-09T20:14:40.313" CommentCount="7" />
  <row Id="3428" PostTypeId="2" ParentId="3412" CreationDate="2010-10-08T22:27:14.820" Score="2" ViewCount="" Body="&lt;p&gt;there is a paper by &lt;a href=&quot;http://projecteuclid.org/DPubS/Repository/1.0/Disseminate?view=body&amp;amp;id=pdf_1&amp;amp;handle=euclid.bsmsp/1200512162&quot; rel=&quot;nofollow&quot;&gt;d.r.cox&lt;/a&gt; that discusses testing separate [unnested] models. it considers a few examples, which do not rise to the complexity of mixed models. [as my facility with R code is limited, i'm not quite sure what your models are.]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;altho cox's paper may not solve your problem directly, it may be helpful in two possible ways.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;you can search google scholar for citations to his paper, to see if subsequent such results come closer to what you want.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;if you are of an analytical bent, you could try applying cox's method to your problem.&#xA;[perhaps not for the faint-hearted.]&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;btw - cox does mention in passing the idea srikant broached of combining the two models into a larger one. he doesn't pursue how one would then decide which model is better, but he remarks that even if neither model is very good, the combined model might give an adequate fit to the data. [it's not clear in your situation that a combined model would make sense.]&lt;/p&gt;&#xA;" OwnerUserId="1112" LastEditorUserId="1112" LastEditorDisplayName="" LastEditDate="2010-10-08T22:34:54.307" LastActivityDate="2010-10-08T22:34:54.307" CommentCount="1" />
  <row Id="3429" PostTypeId="2" ParentId="346" CreationDate="2010-10-08T22:49:46.743" Score="2" ViewCount="" Body="&lt;p&gt;I re-direct you to my answer to a similar &lt;a href=&quot;http://stats.stackexchange.com/questions/3372/is-it-possible-to-accumulate-a-set-of-statistics-that-describes-a-large-number-of/3376#3376&quot;&gt;question&lt;/a&gt;. In a nutshell, it's a read once, 'on the fly' algorithm with $O(n)$ worst case complexity to compute the (exact) median.&lt;/p&gt;&#xA;" OwnerUserId="603" LastActivityDate="2010-10-08T22:49:46.743" />
  <row Id="3430" PostTypeId="2" ParentId="3402" CreationDate="2010-10-08T23:11:33.593" Score="0" ViewCount="" Body="&lt;p&gt;if your sample size $n$ is not such a tiny fraction of the population size $N$ as in your example, and if you sample without replacement [Sw/oR], a better expression for the [estimated] SE is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\hat{SE} = \sqrt{\frac{N - n}{N}\frac{\hat p \hat q}{n}},$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\hat p$ is the estimated proportion $j/n$ and $\hat q = 1- \hat p$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[the term $\frac{N-n}{N}$ is called the FPC [finite population correction].&lt;/p&gt;&#xA;&#xA;&lt;p&gt;altho whuber's remark is technically correct, it seems to suggest that &lt;em&gt;nothing&lt;/em&gt; can be done to get, say, a confidence interval for the true proportion $p$. if $n$ is large enough to make a normal approximation reasonable [$np &gt; 10$, say], it is unlikely one would get $j=0$. also, if the sample size is large enough for a normal approximation using the true $SE$ to be reasonable, using $\hat{SE}$ instead also gives a reasonable approximation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[if your $n$ is really small and you use Sw/oR, you may have to use the exact hypergeometric distribution for $j$ instead of a normal approximation. if you do SwR, the size of $N$ is irrelevant and you can use exact binomial methods to get a CI for $p$.] &lt;/p&gt;&#xA;&#xA;&lt;p&gt;in any case, since $p(1-p) \le 1/4$, one could always be conservative and use $\frac{1}{2\sqrt{n}}$ in place of  $\sqrt{\frac{\hat p \hat q}{n}}$ in the above. if you do that, it takes a sample of $n = 1,111$ to get an estimated ME [margin of error = 2$\hat {SE}$] of $\pm$.03 [regardless of how big $N$ is!].&lt;/p&gt;&#xA;" OwnerUserId="1112" LastEditorUserId="1112" LastEditorDisplayName="" LastEditDate="2010-10-08T23:30:44.833" LastActivityDate="2010-10-08T23:30:44.833" />
  <row Id="3431" PostTypeId="2" ParentId="3413" CreationDate="2010-10-08T23:53:06.367" Score="2" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://www.gnu.org/software/octave/&quot; rel=&quot;nofollow&quot;&gt;Octave&lt;/a&gt; has a built-in Hurst Exponent function.&lt;/p&gt;&#xA;" OwnerUserId="226" LastActivityDate="2010-10-08T23:53:06.367" />
  <row Id="3432" PostTypeId="2" ParentId="3425" CreationDate="2010-10-09T00:12:17.553" Score="6" ViewCount="" Body="&lt;p&gt;To supplement Kwak's useful answer, allow me to add some simple principles and ideas.  A good way to determine the metric is by considering how the strings might vary from their target.  &quot;Edit distance&quot; is useful when the variation is a combination of typographic errors like transposing neighbors or mis-typing a single key.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another useful approach (with a slightly different philosophy) is to map every string into one representative of a class of related strings.  The &quot;&lt;a href=&quot;http://en.wikipedia.org/wiki/Soundex&quot; rel=&quot;nofollow&quot;&gt;Soundex&lt;/a&gt;&quot; method does this: the Soundex code for a word is a sequence of four characters encoding the principal consonant and groups of similar-sounding internal consequence.  It is used when words are phonetic misspellings or variants of one another. In the example application you would fetch all target words whose Soundex code equals the Soundex code for each probe word.  (There could be zero or multiple targets fetched this way.)&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-10-09T00:12:17.553" CommentCount="2" />
  <row Id="3433" PostTypeId="2" ParentId="3425" CreationDate="2010-10-09T02:41:55.300" Score="3" ViewCount="" Body="&lt;p&gt;I've had similar problems. (seen here: &lt;a href=&quot;http://stackoverflow.com/questions/2231993/merging-two-data-frames-using-fuzzy-approximate-string-matching-in-r&quot;&gt;http://stackoverflow.com/questions/2231993/merging-two-data-frames-using-fuzzy-approximate-string-matching-in-r&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most of the recommendations that I received fell around: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;pmatch()&lt;/code&gt;, and &lt;code&gt;agrep()&lt;/code&gt;, &lt;code&gt;grep()&lt;/code&gt;, &lt;code&gt;grepl()&lt;/code&gt; are three functions that if you take the time to look through will provide you with some insight into approximate string matching either by approximate string or approximate regex.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Without seeing the strings, it's hard to provide you with hard example of how to match them. If you could provide us with some example data I'm sure we could come to a solution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another option that I found works well is to flatten the strings, &lt;code&gt;tolower()&lt;/code&gt;, looking at the first letter of each word within the string and then comparing. Sometimes that works without a hitch. Then there are more complicated things like the distances mentioned in other answers. Sometimes these work, sometimes they're horrible - it really depends on the strings. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can we see them?&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Update&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;It looks like agrep() will do the trick for most of these. Note that agrep() is just R's implementation of Levenshtein distance. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;agrep(vec55[1],vec91,value=T)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Some don't compute although, I'm not even sure if Ferroplasm acidaramus is the same as Ferroglobus placidus DSM 10642, for example: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;agrep(vec55[7],vec91,value=T) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I think you may be a bit SOL for some of these and perhaps creating an index from scratch is the best bet. ie,. Create a table with id numbers for vec55, and then manually create a reference to the id's in vec55 in vec91. Painful, I know, but a lot of it can be done with agrep(). &lt;/p&gt;&#xA;" OwnerUserId="776" LastEditorUserId="776" LastEditorDisplayName="" LastEditDate="2010-10-09T20:00:20.550" LastActivityDate="2010-10-09T20:00:20.550" CommentCount="3" />
  <row Id="3434" PostTypeId="2" ParentId="3377" CreationDate="2010-10-09T03:27:11.417" Score="4" ViewCount="" Body="&lt;p&gt;I've used the following approach in the past to calculate absolution deviation moderately efficiently (note, this a programmers approach, not a statisticians, so indubitably there may be clever tricks like &lt;a href=&quot;http://stats.stackexchange.com/questions/3377/online-algorithm-for-mean-absolute-deviation-and-large-data-set/3378#3378&quot;&gt;shabbychef's&lt;/a&gt; that might be more efficient). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;WARNING:&lt;/strong&gt; This is not an online algorithm.  It requires &lt;code&gt;O(n)&lt;/code&gt; memory.  Furthermore, it has a worst case performance of &lt;code&gt;O(n)&lt;/code&gt;, for datasets like &lt;code&gt;[1, -2, 4, -8, 16, -32, ...]&lt;/code&gt; (i.e. the same as the full recalculation). [1] &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, because it still performs well in many use cases it might be worth posting here.  For example, in order to calculate the absolute deviance of 10000 &lt;em&gt;random&lt;/em&gt; numbers between -100 and 100 as each item arrives, my algorithm takes less than one second, while the full recalculation takes over 17 seconds (on my machine, will vary per machine and according to input data).  You need to maintain the entire vector in memory however, which may be a constraint for some uses.  The outline of the algorithm is as follows:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Instead of having a single vector to store past measurements, use three sorted priority queues (something like a min/max heap).  These three lists partition the input into three: items greater than the mean, items less than the mean and items equal to the mean.&lt;/li&gt;&#xA;&lt;li&gt;(Almost) every time you add an item the mean changes, so we need to repartition.  The crucial thing is the sorted nature of the partitions which means that instead of scanning every item in the list to repartion, we only need to read those items we are moving.  While in the worst case this will still require &lt;code&gt;O(n)&lt;/code&gt; move operations, for many use-cases this is not so.&lt;/li&gt;&#xA;&lt;li&gt;Using some clever bookkeeping, we can make sure that the deviance is correctly calculated at all times, when repartitioning and when adding new items.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Some sample code, in python, is below.  Note that it only allows items to be added to the list, not removed.  This could easily be added, but at the time I wrote this I had no need for it.  Rather than implement the priority queues myself, I have used the &lt;a href=&quot;http://stutzbachenterprises.com/blist/sortedlist.html&quot; rel=&quot;nofollow&quot;&gt;sortedlist&lt;/a&gt; from Daniel Stutzbach's excellent &lt;a href=&quot;http://stutzbachenterprises.com/blist/&quot; rel=&quot;nofollow&quot;&gt;blist package&lt;/a&gt;, which use &lt;a href=&quot;http://en.wikipedia.org/wiki/B%2B_tree&quot; rel=&quot;nofollow&quot;&gt;B+Tree&lt;/a&gt;s internally.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider this code licensed under the &lt;a href=&quot;http://www.opensource.org/licenses/mit-license.html&quot; rel=&quot;nofollow&quot;&gt;MIT license&lt;/a&gt;.  It has not been significantly optimised or polished, but has worked for me in the past.  New versions will be available &lt;a href=&quot;http://github.com/fmark/phes-code/blob/master/deviance_list.py&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.  Let me know if you have any questions, or find any bugs.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from blist import sortedlist&#xA;import operator&#xA;&#xA;class deviance_list:&#xA;    def __init__(self):&#xA;        self.mean =  0.0&#xA;        self._old_mean = 0.0&#xA;        self._sum =  0L&#xA;        self._n =  0  #n items&#xA;        # items greater than the mean&#xA;        self._toplist =  sortedlist()&#xA;        # items less than the mean&#xA;        self._bottomlist = sortedlist(key = operator.neg)&#xA;        # Since all items in the &quot;eq list&quot; have the same value (self.mean) we don't need&#xA;        # to maintain an eq list, only a count&#xA;        self._eqlistlen = 0&#xA;&#xA;        self._top_deviance =  0&#xA;        self._bottom_deviance =  0&#xA;&#xA;    @property&#xA;    def absolute_deviance(self):&#xA;        return self._top_deviance + self._bottom_deviance&#xA;&#xA;    def append(self,  n):&#xA;        # Update summary stats&#xA;        self._sum += n&#xA;        self._n +=  1&#xA;        self._old_mean =  self.mean&#xA;        self.mean =  self._sum /  float(self._n)&#xA;&#xA;        # Move existing things around&#xA;        going_up = self.mean &amp;gt; self._old_mean&#xA;        self._rebalance(going_up)&#xA;&#xA;        # Add new item to appropriate list&#xA;        if n &amp;gt;  self.mean:&#xA;            self._toplist.add(n)&#xA;            self._top_deviance +=  n -  self.mean&#xA;        elif n == self.mean: &#xA;            self._eqlistlen += 1&#xA;        else:&#xA;            self._bottomlist.add(n)&#xA;            self._bottom_deviance += self.mean -  n&#xA;&#xA;&#xA;    def _move_eqs(self,  going_up):&#xA;        if going_up:&#xA;            self._bottomlist.update([self._old_mean] *  self._eqlistlen)&#xA;            self._bottom_deviance += (self.mean - self._old_mean) * self._eqlistlen&#xA;            self._eqlistlen = 0&#xA;        else:&#xA;            self._toplist.update([self._old_mean] *  self._eqlistlen)&#xA;            self._top_deviance += (self._old_mean - self.mean) * self._eqlistlen&#xA;            self._eqlistlen = 0&#xA;&#xA;&#xA;    def _rebalance(self, going_up):&#xA;        move_count,  eq_move_count = 0, 0&#xA;        if going_up:&#xA;            # increase the bottom deviance of the items already in the bottomlist&#xA;            if self.mean !=  self._old_mean:&#xA;                self._bottom_deviance += len(self._bottomlist) *  (self.mean -  self._old_mean)&#xA;                self._move_eqs(going_up)&#xA;&#xA;&#xA;            # transfer items from top to bottom (or eq) list, and change the deviances&#xA;            for n in iter(self._toplist):&#xA;                if n &amp;lt; self.mean:&#xA;                    self._top_deviance -= n -  self._old_mean&#xA;                    self._bottom_deviance += (self.mean -  n)&#xA;                    # we increment movecount and move them after the list&#xA;                    # has finished iterating so we don't modify the list during iteration&#xA;                    move_count +=  1&#xA;                elif n == self.mean:&#xA;                    self._top_deviance -= n -  self._old_mean&#xA;                    self._eqlistlen += 1&#xA;                    eq_move_count +=  1&#xA;                else:&#xA;                    break&#xA;            for _ in xrange(0,  move_count):&#xA;                self._bottomlist.add(self._toplist.pop(0))&#xA;            for _ in xrange(0,  eq_move_count):&#xA;                self._toplist.pop(0)&#xA;&#xA;            # decrease the top deviance of the items remain in the toplist&#xA;            self._top_deviance -= len(self._toplist) *  (self.mean -  self._old_mean)&#xA;        else:&#xA;            if self.mean !=  self._old_mean:&#xA;                self._top_deviance += len(self._toplist) *  (self._old_mean -  self.mean)&#xA;                self._move_eqs(going_up)&#xA;            for n in iter(self._bottomlist): &#xA;                if n &amp;gt; self.mean:&#xA;                    self._bottom_deviance -= self._old_mean -  n&#xA;                    self._top_deviance += n -  self.mean&#xA;                    move_count += 1&#xA;                elif n == self.mean:&#xA;                    self._bottom_deviance -= self._old_mean -  n&#xA;                    self._eqlistlen += 1&#xA;                    eq_move_count +=  1&#xA;                else:&#xA;                    break&#xA;            for _ in xrange(0,  move_count):&#xA;                    self._toplist.add(self._bottomlist.pop(0))&#xA;            for _ in xrange(0,  eq_move_count):&#xA;                self._bottomlist.pop(0)&#xA;&#xA;            # decrease the bottom deviance of the items remain in the bottomlist&#xA;            self._bottom_deviance -= len(self._bottomlist) *  (self._old_mean -  self.mean)&#xA;&#xA;&#xA;if __name__ ==  &quot;__main__&quot;:&#xA;    import random&#xA;    dv =  deviance_list()&#xA;    # Test against some random data,  and calculate result manually (nb. slowly) to ensure correctness&#xA;    rands = [random.randint(-100,  100) for _ in range(0,  1000)]&#xA;    ns = []&#xA;    for n in rands: &#xA;        dv.append(n)&#xA;        ns.append(n)&#xA;        print(&quot;added:%4d,  mean:%3.2f,  oldmean:%3.2f,  mean ad:%3.2f&quot; %&#xA;              (n, dv.mean,  dv._old_mean,  dv.absolute_deviance / dv.mean))&#xA;        assert sum(ns) == dv._sum,  &quot;Sums not equal!&quot;&#xA;        assert len(ns) == dv._n,  &quot;Counts not equal!&quot;&#xA;        m = sum(ns) / float(len(ns))&#xA;        assert m == dv.mean,  &quot;Means not equal!&quot;&#xA;        real_abs_dev = sum([abs(m - x) for x in ns])&#xA;        # Due to floating point imprecision, we check if the difference between the&#xA;        # two ways of calculating the asb. dev. is small rather than checking equality&#xA;        assert abs(real_abs_dev - dv.absolute_deviance) &amp;lt; 0.01, (&#xA;            &quot;Absolute deviances not equal. Real:%.2f,  calc:%.2f&quot; %  (real_abs_dev,  dv.absolute_deviance))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;[1] If symptoms persist, see your doctor.&lt;/p&gt;&#xA;" OwnerUserId="179" LastEditorUserId="179" LastEditorDisplayName="" LastEditDate="2010-11-09T05:33:11.947" LastActivityDate="2010-11-09T05:33:11.947" CommentCount="2" />
  <row Id="3435" PostTypeId="2" ParentId="3425" CreationDate="2010-10-09T03:32:47.390" Score="2" ViewCount="" Body="&lt;p&gt;I would also suggest you check out &lt;a href=&quot;http://en.wikipedia.org/wiki/N-gram&quot; rel=&quot;nofollow&quot;&gt;N-grams&lt;/a&gt;  and the &lt;a href=&quot;http://en.wikipedia.org/wiki/Damerau%E2%80%93Levenshtein_distance&quot; rel=&quot;nofollow&quot;&gt;Damerau–Levenshtein&lt;/a&gt; distance besides the other suggestions of Kwak.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.15.178&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;paper&lt;/a&gt; compares the accuracy of a few different edit distances mentioned here (and is highly cited according to google scholar).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you can see there are many different ways to approach this, and you can even combine different metrics (the paper I linked to talks about this alittle bit). I think the Levenshtein and related based metrics make the most intuitive sense, especially if errors occur because of human typing. N-grams are also simple and make sense for data that is not names or words per say. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;While soundex is an option, the little bit of work I have seen (which is admittedly a very small amount) soundex does not perform as well as Levenshstein or other edit distances for matching names. And the Soundex is limited to phonetic phrases likely inputted by human typers, where as Levenshtein and N-grams have a potentially broader scope (especially N-gram, but I would expect the Levenshtein distance to perform better for non-words as well).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can't help as far as packages, but the concept of N-grams is pretty simple (I did make an SPSS macro to do N-grams recently, but for such a small project I would just go with the already made packages in R the other posters have suggested). &lt;a href=&quot;http://hetland.org/coding/python/levenshtein.py&quot; rel=&quot;nofollow&quot;&gt;Here&lt;/a&gt; is an example of calculating the Levenshtein distance in python.&lt;/p&gt;&#xA;" OwnerUserId="1036" LastActivityDate="2010-10-09T03:32:47.390" CommentCount="1" />
  <row Id="3436" PostTypeId="2" ParentId="3381" CreationDate="2010-10-09T09:16:11.677" Score="2" ViewCount="" Body="&lt;p&gt;I would see each histogram as a different model (parametrized by the width). Fitting a smoothing spline or some other kind of smoother for each of the models is simple.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can then do model selection (such as cross-validation) to choose the histogram width that gives the best results, or do model stacking to fit least-squares weights on the models. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, why not directly smooth the data instead of clustering it into histogram bars first? There are finite-window width kernels that don't use the entire dataset for prediction at a given point. Practicality and speed depends on what you are really trying to obtain, but I am sure there exist simpler solutions.&lt;/p&gt;&#xA;" OwnerUserId="1526" LastActivityDate="2010-10-09T09:16:11.677" CommentCount="1" />
  <row Id="3437" PostTypeId="2" ParentId="3419" CreationDate="2010-10-09T11:59:06.653" Score="2" ViewCount="" Body="&lt;p&gt;The simple idea is to make PCA and base distance of few first components (yet I don't like this technique because of assumptions it makes).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The complex idea is to use machine learning; the resulting distances will expose the classifier structure, so will be about as good as the classification accuracy. The simplest approach here is just random forest object distance (&lt;a href=&quot;http://www.stat.berkeley.edu/~breiman/RandomForests/cc_home.htm#cluster&quot; rel=&quot;nofollow&quot;&gt;Breiman's example&lt;/a&gt;), but you can also use the kernel justified by SVM, see for instance &lt;a href=&quot;http://www.biomedcentral.com/1471-2105/8/S7/S18&quot; rel=&quot;nofollow&quot;&gt;Winters-Hilt &amp;amp; Merat 2007&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-10-09T11:59:06.653" />
  <row Id="3438" PostTypeId="1" AcceptedAnswerId="3440" CreationDate="2010-10-09T13:34:40.713" Score="4" ViewCount="551" Body="&lt;p&gt;See this Wikipedia page:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval#Agresti-Coull_Interval&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval#Agresti-Coull_Interval&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To get the Agresti-Coull Interval, one needs to calculate a percentile of the normal distribution, called $z$. How do I calculate tha percentile? Is there a ready-made function that does this in Wolfram Mathematica and/or Python/NumPy/SciPy?&lt;/p&gt;&#xA;" OwnerUserId="1523" LastActivityDate="2010-10-09T16:09:00.780" Title="Calculating percentile of normal distribution" Tags="&lt;python&gt;&lt;normal-distribution&gt;" AnswerCount="4" FavoriteCount="1" />
  <row Id="3439" PostTypeId="2" ParentId="3438" CreationDate="2010-10-09T13:40:55.500" Score="1" ViewCount="" Body="&lt;p&gt;Well, you didn't ask about R, but in R you do it using ?qnorm&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(It's actually the quantile, not the percentile, or so I believe)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; qnorm(.5)&#xA;[1] 0&#xA;&amp;gt; qnorm(.95)&#xA;[1] 1.644854&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="253" LastActivityDate="2010-10-09T13:40:55.500" CommentCount="3" />
  <row Id="3440" PostTypeId="2" ParentId="3438" CreationDate="2010-10-09T14:08:55.643" Score="2" ViewCount="" Body="&lt;p&gt;For &lt;em&gt;Mathematica&lt;/em&gt;, &lt;code&gt;$VersionNumber &amp;gt; 5&lt;/code&gt; you can use&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Quantile[NormalDistribution[mu,sigma], 100 q]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;for the q-th percentile.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Otherwise, you have to load the appropriate Statistics package first.&lt;/p&gt;&#xA;" OwnerUserId="830" LastActivityDate="2010-10-09T14:08:55.643" CommentCount="2" />
  <row Id="3441" PostTypeId="2" ParentId="3438" CreationDate="2010-10-09T14:20:58.783" Score="1" ViewCount="" Body="&lt;p&gt;In Python, you can use the &lt;a href=&quot;http://www.scipy.org/SciPyPackages/Stats&quot; rel=&quot;nofollow&quot;&gt;stats&lt;/a&gt; module from the &lt;a href=&quot;http://www.scipy.org/&quot; rel=&quot;nofollow&quot;&gt;scipy&lt;/a&gt; package (look for &lt;code&gt;cdf()&lt;/code&gt;, as in the following &lt;a href=&quot;http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.norm.html&quot; rel=&quot;nofollow&quot;&gt;example&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(It seems the &lt;a href=&quot;http://bonsai.hgc.jp/~mdehoon/software/python/special.html&quot; rel=&quot;nofollow&quot;&gt;transcendantal&lt;/a&gt; package also includes usual cumulative distributions).&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-10-09T14:20:58.783" />
  <row Id="3442" PostTypeId="2" ParentId="97" CreationDate="2010-10-09T15:03:51.913" Score="3" ViewCount="" Body="&lt;p&gt;For basic summaries, I agree that reporting frequency tables and some indication about central tendency is fine. For inference, a recent article published in PARE discussed t- vs. MWW-test, &lt;a href=&quot;http://pareonline.net/pdf/v15n11.pdf&quot; rel=&quot;nofollow&quot;&gt;Five-Point Likert Items: t test versus Mann-Whitney-Wilcoxon&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For more elaborated treatment, I would recommend reading Agresti's review on ordered categorical variables:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Liu, Y and Agresti, A (2005). &lt;a href=&quot;http://petra.euitio.uniovi.es/~i1770184/Archivos/t141/Test_agresti.pdf&quot; rel=&quot;nofollow&quot;&gt;The&#xA;  analysis of ordered categorical data:&#xA;  An overview and a survey of recent&#xA;  developments&lt;/a&gt;. &lt;em&gt;Sociedad de&#xA;  Estadística e Investigación Operativa&#xA;  Test&lt;/em&gt;, 14(1), 1-73.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It largely extends beyond usual statistics, like threshold-based model (e.g. proportional odds-ratio), and is worth reading in place of Agresti's &lt;a href=&quot;http://www.stat.ufl.edu/~aa/cda/cda.html&quot; rel=&quot;nofollow&quot;&gt;CDA&lt;/a&gt; book.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Below I show a picture of three different ways of treating a Likert item; from top to bottom, the &quot;frequency&quot; (nominal) view, the &quot;numerical&quot; view, and the &quot;probabilistic&quot; view (a &lt;a href=&quot;http://en.wikipedia.org/wiki/Polytomous_Rasch_model&quot; rel=&quot;nofollow&quot;&gt;Partial Credit Model&lt;/a&gt;):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/ppdQg.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data comes from the &lt;code&gt;Science&lt;/code&gt; data in the &lt;code&gt;ltm&lt;/code&gt; package, where the item concerned &lt;em&gt;technology&lt;/em&gt; (&quot;New technology does not depend on basic scientific research&quot;, with response &quot;strongly disagree&quot; to &quot;strongly agree&quot;, on a four-point scale)&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-09T20:28:32.153" LastActivityDate="2010-10-09T20:28:32.153" />
  <row Id="3443" PostTypeId="2" ParentId="3400" CreationDate="2010-10-09T16:02:58.157" Score="3" ViewCount="" Body="&lt;p&gt;Propensity scores are typically used in the matching literature. Propensity scores use pre-treatment covariates to estimate the probability of receiving treatment. Essentially, a regression (either just regular OLS or logit, probit, etc) is used to calculate the propensity score with treatment as your outcome and pre-treatment variables are your covariates. Once a good estimate of the propensity score is obtained, subjects with similar propensity scores, but different treatments received, are matched to one another. The treatment effect is the difference in means between these two groups.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rosenbaum and Rubin (1983) show that matching treated and control subjects using just the propensity score is sufficient to remove all bias in the estimate of the treatment effect stemming from the observed pre-treatment covariates used to construct the score. Note that this proof requires the use of the true propensity score, rather than an estimate. The advantage of this approach is it turns a problem of matching in multiple dimensions (one for each pre-treatment covariate) into a univariate matching case---a great simplification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rosenbaum, Paul R. and Donald B. Rubin. 1983. &quot;&lt;a href=&quot;http://faculty.smu.edu/millimet/classes/eco7377/papers/rosenbaum%20rubin%2083a.pdf&quot; rel=&quot;nofollow&quot;&gt;The Central Role of the Propensity Score in Observational Studies for Causal Effects&lt;/a&gt;.&quot; Biometrika. 70(1): 41--55.&lt;/p&gt;&#xA;" OwnerUserId="401" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-09T18:23:45.387" LastActivityDate="2010-10-09T18:23:45.387" CommunityOwnedDate="2010-10-09T16:02:58.157" CommentCount="1" />
  <row Id="3444" PostTypeId="2" ParentId="3438" CreationDate="2010-10-09T16:09:00.780" Score="2" ViewCount="" Body="&lt;p&gt;John Cook's page, &lt;a href=&quot;http://www.johndcook.com/distributions_scipy.html&quot; rel=&quot;nofollow&quot;&gt;Distributions in Scipy&lt;/a&gt;, is a good reference for this type of stuff:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [15]: import scipy.stats&#xA;&#xA;In [16]: scipy.stats.norm.ppf(0.975)&#xA;Out[16]: 1.959963984540054&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-10-09T16:09:00.780" CommentCount="1" />
  <row Id="3445" PostTypeId="1" CreationDate="2010-10-09T16:28:24.350" Score="1" ViewCount="276" Body="&lt;p&gt;Dear all, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have the following data, which is the output from ms hudson software. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;segsites: 6 &#xA;positions: 0.1256 0.3122 0.3218 0.4970 0.5951 0.7943 &#xA;001010 &#xA;110101 &#xA;010100 &#xA;001010 &#xA;010100 &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to make R function to calculate the R-Square across pairs separated by &amp;lt;10% (the difference between positions of SNPs must be &amp;lt; 0.10) of the simulated genomic region. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have search this from many days but still did not find how to do in R. &#xA;Could anyone help me in this regards? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance! &#xA;Regards, &#xA;Jacki&lt;/p&gt;&#xA;" OwnerUserId="1534" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-11T15:10:12.740" LastActivityDate="2010-10-11T15:10:12.740" Title="How to Calculate the Pairwise LD for the given data?" Tags="&lt;r&gt;&lt;correlation&gt;&lt;genetics&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="3446" PostTypeId="1" AcceptedAnswerId="3454" CreationDate="2010-10-09T17:47:05.413" Score="6" ViewCount="271" Body="&lt;p&gt;I've been looking at some of the packages from the High perf task &lt;a href=&quot;http://cran.r-project.org/web/views/HighPerformanceComputing.html&quot; rel=&quot;nofollow&quot;&gt;view&lt;/a&gt; dealing with GPU computations, and given that most GPU seem to be an order of magnitude stronger at performing single precision arithmetics than DP &lt;a href=&quot;http://en.wikipedia.org/wiki/Comparison_of_Nvidia_graphics_processing_units&quot; rel=&quot;nofollow&quot;&gt;ones&lt;/a&gt;, i was wondering:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Why none of the packages gives more control to the user on the type of precision required ? I can see many applications in statistics where SP arithmetics (i.e. number coded with 7 digit accuracy)  are good enough for practical use (if i am overestimating the gains involved, let me know). &lt;/li&gt;&#xA;&lt;li&gt;Is python more flexible on this ? If so why ? I don't see why the absence of a 'single' type in R would make including such an option (together with a warning) in say GPUtools or magma impossible (though i'll be happy to be shown wrong).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;PS: i'm specifically thinking of applications were the numbers are already dimension-wise scaled and centered (so that Chebychev's &lt;a href=&quot;http://en.wikipedia.org/wiki/Chebyshev%27s_inequality&quot; rel=&quot;nofollow&quot;&gt;inequality&lt;/a&gt; is binding) dimension-wise. &lt;/p&gt;&#xA;" OwnerUserId="603" LastEditorUserId="603" LastEditorDisplayName="" LastEditDate="2010-10-09T18:20:29.930" LastActivityDate="2010-10-10T05:22:38.977" Title="Is single precision floating point so bad?" Tags="&lt;r&gt;&lt;python&gt;&lt;gpu&gt;" AnswerCount="4" CommentCount="2" />
  <row Id="3447" PostTypeId="2" ParentId="3445" CreationDate="2010-10-09T18:14:22.367" Score="2" ViewCount="" Body="&lt;p&gt;I know the &lt;code&gt;LDheatmap&lt;/code&gt; function/package can calculate the pairwise LDs, see &lt;code&gt;ldhm$LDmatrix&lt;/code&gt; in the example below.  I'm not familiar with the software you mention or how to get data into the required format for &lt;code&gt;LDheatmap&lt;/code&gt;.   &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; library(LDheatmap)&#xA;&amp;gt; data(CEUData)&#xA;&amp;gt; ldhm &amp;lt;- LDheatmap(CEUSNP, genetic.distances=CEUDist, LDmeasure=&quot;r&quot;)&#xA;&amp;gt; head(ldhm$LDmatrix, 1)&#xA;          rs4615512   rs2283089 rs1894731 rs2283092 rs2283093 rs6978939 rs6979287&#xA;rs4615512        NA 0.003329066 0.1657227 0.1657227 0.1269654 0.1657227 0.1209495&#xA;          rs6979572 rs2283094 rs2283095 rs6467111 rs1111183 rs2237789 rs2299531 rs2299532&#xA;rs4615512 0.1197102 0.8395061 0.1197102  0.099289 0.3470553 0.1197102 0.3746656 0.3470553&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-10-09T18:14:22.367" CommentCount="1" />
  <row Id="3448" PostTypeId="2" ParentId="3445" CreationDate="2010-10-09T18:14:36.530" Score="1" ViewCount="" Body="&lt;p&gt;There are various R/Bioconductor packages that allow you to compute pairwise correlation for SNPs in linkage disequilibrium, see the CRAN Task View &lt;a href=&quot;http://cran.r-project.org/web/views/Genetics.html&quot; rel=&quot;nofollow&quot;&gt;Statistical Genetics&lt;/a&gt;. As I worked directly with whole genome scan, I've been mainly using &lt;code&gt;snpMatrix&lt;/code&gt;, but &lt;a href=&quot;http://cran.r-project.org/web/packages/LDheatmap/index.html&quot; rel=&quot;nofollow&quot;&gt;LDheatmap&lt;/a&gt; or &lt;a href=&quot;http://cran.r-project.org/web/packages/mapLD/index.html&quot; rel=&quot;nofollow&quot;&gt;mapLD&lt;/a&gt; are fine. However, usually they expect genotype data (AA, AB, or BB), so I guess you will have to first convert your binary-encoded haplotype... About the filter on location, I also guess you just have to consider the pairwise $R^2$ or $D'$ for proximal SNPs (usually, we draw a so-called heatmap of pairwise LD, which is roughly speaking the lower-diag elements of the correlation matrix, so you just have to consider the very first off diagonal elements).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now that I've read some papers, I'm not sure you will achieve your goals with the aforementioned method. To my knowledge, few packages allow to cope with multiallelic loci or haplotype blocks, one example being the &lt;a href=&quot;http://cran.r-project.org/web/packages/gap/index.html&quot; rel=&quot;nofollow&quot;&gt;gap&lt;/a&gt; package from JH Zhao (see also a review in the &lt;a href=&quot;http://www.jstatsoft.org/v23/i08/paper&quot; rel=&quot;nofollow&quot;&gt;Journal of Statistical Software&lt;/a&gt;). The &lt;code&gt;LDkl()&lt;/code&gt; function for example computes D' and $\rho$ from a vector of haplotype frequencies, which can easily be plotted using &lt;code&gt;image()&lt;/code&gt; or &lt;code&gt;levelplot()&lt;/code&gt; from the &lt;code&gt;lattice&lt;/code&gt; package.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-10T14:05:27.840" LastActivityDate="2010-10-10T14:05:27.840" CommentCount="6" />
  <row Id="3449" PostTypeId="2" ParentId="134" CreationDate="2010-10-09T19:02:09.717" Score="5" ViewCount="" Body="&lt;p&gt;It's bad form to sort an array to compute a median. Medians (and other quantiles) are typically computed using the &lt;a href=&quot;http://www.ics.uci.edu/~eppstein/161/960125.html&quot; rel=&quot;nofollow&quot;&gt;quickselect&lt;/a&gt; algorithm, with $O(n)$ complexity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may also want to look at my answer to a recent related question &lt;a href=&quot;http://stats.stackexchange.com/questions/3372/is-it-possible-to-accumulate-a-set-of-statistics-that-describes-a-large-number-of/3376#3376&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="603" LastActivityDate="2010-10-09T19:02:09.717" />
  <row Id="3450" PostTypeId="2" ParentId="3400" CreationDate="2010-10-09T19:17:12.947" Score="3" ViewCount="" Body="&lt;p&gt;The question seems to involve two things that really ought to be considered separately. First is whether one can infer causality from an observational study, and on that you might contrast the views of, say, Pearl (2009), who argues yes so long as you can model the process properly, versus the view @propofol, who will find many allies in experimental disciplines and who may share some of the thoughts expressed in (a rather obscure but nonetheless good) essay by Gerber et al (2004).  Second, assuming that you do think that causality can be inferred from observational data, you might wonder whether propensity score methods are useful in doing so.  Propensity score methods include various conditioning strategies as well as inverse propensity weighting.  A nice review is given by Lunceford and Davidian (2004).  They have good properties but certain assumptions are required (most specifically, &quot;conditional independence&quot;) for them to be consistent. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A little wrinkle though: propensity score matching and weighting are also used in the analysis of randomized experiments when, for example, there is an interest in computing &quot;indirect effects&quot; and also when there are problems of potentially non-random attrition or drop out (in which case what you have resembles an observational study).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;References&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Gerber A, et al. 2004. &quot;The illusion of learning from observational research.&quot; In Shapiro I, et al, &lt;em&gt;Problems and Methods in the Study of Politics&lt;/em&gt;, Cambridge University Press.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lunceford JK, Davidian M. 2004. &quot;Stratification and weighting via the propensity score in estimation of causal treatment effects: a comparative study.&quot; &lt;em&gt;Statistics in Medicine&lt;/em&gt; 23(19):2937–2960.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Pearl J. 2009. &lt;em&gt;Causality (2nd Ed.)&lt;/em&gt;, Cambridge University Press.&lt;/p&gt;&#xA;" OwnerUserId="96" LastActivityDate="2010-10-09T19:17:12.947" CommunityOwnedDate="2010-10-09T19:17:12.947" CommentCount="1" />
  <row Id="3451" PostTypeId="2" ParentId="2849" CreationDate="2010-10-09T19:40:28.550" Score="2" ViewCount="" Body="&lt;p&gt;I guess the current (econometrics) industry standard for this setting is fixed effects regression.  Take a look at the section on panel data in &lt;a href=&quot;http://www-personal.umich.edu/~nicholsa/ciwod.pdf&quot; rel=&quot;nofollow&quot;&gt;this paper&lt;/a&gt; by Austin Nichols for a concise discussion.  For these kinds of analyses you want larger N, typically, though.  (By the way, for those with a background in statistics rather than econometrics, the usage of &quot;fixed effects&quot; is different for econometricians.  For them, it means a regression that uses differencing or dummy variables to account for unmeasured (linearly additive) confounders in a repeat observation setting. This is different than what statisticians usually mean, which is usually in contrasting &quot;fixed&quot; and &quot;random&quot; effects.)&lt;/p&gt;&#xA;" OwnerUserId="96" LastActivityDate="2010-10-09T19:40:28.550" />
  <row Id="3452" PostTypeId="2" ParentId="3446" CreationDate="2010-10-09T20:12:24.353" Score="3" ViewCount="" Body="&lt;ol&gt;&#xA;&lt;li&gt;Because before GPUs there was no practical sense of using single reals; you never have too much accuracy and memory is usually not a problem. And supporting only doubles made R design simpler. (Although R supports reading/writing single reals.)&lt;/li&gt;&#xA;&lt;li&gt;Yes, because Python is aimed to be more compatible with compiled languages. Yet you are right that it is possible for R libraries' wrappers to do in-fly conversion (this of course takes time but this is a minor problem); you can try e-mailing GPU packages' maintainers requesting such changes.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-10-09T20:12:24.353" />
  <row Id="3453" PostTypeId="2" ParentId="3446" CreationDate="2010-10-09T20:40:43.653" Score="2" ViewCount="" Body="&lt;p&gt;I presume that by GPU programming, you mean programming nvidia cards? In which case the underlying code calls from R and python are to C/&lt;a href=&quot;http://en.wikipedia.org/wiki/CUDA&quot; rel=&quot;nofollow&quot;&gt;CUDA&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;The simple reason that only single precision is offered is because that is what most GPU cards support.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the new nvidia &lt;a href=&quot;http://www.nvidia.com/object/fermi_architecture.html&quot; rel=&quot;nofollow&quot;&gt;Fermi&lt;/a&gt; architecture does support double precision. If you bought a nvidia graphics card this year, then it's probably a Fermi. Even here things aren't simple: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;You get a slight performance hit if you compile with double precision (a factor of two if I remember correctly).&lt;/li&gt;&#xA;&lt;li&gt;On the cheaper cards Fermi cards, nvidia intentionally disabled double precision. However, it is possible to get round this and run double precision programs. I managed to do this on my GeForce GTX 465 under linux.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;To answer the question in your title, &quot;Is single precision OK?&quot;, it depends on your application (sorry crap answer!). I suppose everyone now uses double precision because it no longer gives a performance hit. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I dabbled with GPUs, programming suddenly became far more complicated. You have to worry about things like:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;warpsize and arranging your memory properly.&lt;/li&gt;&#xA;&lt;li&gt;#threads per kernel.&lt;/li&gt;&#xA;&lt;li&gt;debugging is horrible - there's no print statement in the GPU kernel statements&lt;/li&gt;&#xA;&lt;li&gt;lack of random number generators&lt;/li&gt;&#xA;&lt;li&gt;Single precision. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="8" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-10-09T20:47:49.703" LastActivityDate="2010-10-09T20:47:49.703" CommentCount="9" />
  <row Id="3454" PostTypeId="2" ParentId="3446" CreationDate="2010-10-09T22:56:45.643" Score="5" ViewCount="" Body="&lt;p&gt;From the &lt;a href=&quot;http://cran.r-project.org/web/packages/gputools/gputools.pdf&quot; rel=&quot;nofollow&quot;&gt;GPUtools help file&lt;/a&gt;, it seems that &lt;code&gt;useSingle=TRUE&lt;/code&gt; is the default for the functions.&lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-10-09T22:56:45.643" CommentCount="2" />
  <row Id="3455" PostTypeId="2" ParentId="3446" CreationDate="2010-10-10T05:22:38.977" Score="1" ViewCount="" Body="&lt;p&gt;The vast majority of GPUs in circulation only support single precision floating point.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as the title question, you need to look at the data you'll be handling to determine if single precision is enough for you.  Often, you'll find that singles are perfectly acceptable for &gt;90% of the data you handle, but will fail spectacularly for that last 10%; unless you have an easy way of determining whether your particular data set will fail or not, you're stuck using double precision for everything.&lt;/p&gt;&#xA;" OwnerUserId="1539" LastActivityDate="2010-10-10T05:22:38.977" CommentCount="4" />
  <row Id="3456" PostTypeId="2" ParentId="3419" CreationDate="2010-10-10T06:12:37.237" Score="2" ViewCount="" Body="&lt;p&gt;There is a subfield called Distance Metric Learning. One such method is Information Theoretic Metric Learning (ITML).&lt;/p&gt;&#xA;" OwnerUserId="1540" LastActivityDate="2010-10-10T06:12:37.237" />
  <row Id="3457" PostTypeId="2" ParentId="138" CreationDate="2010-10-10T06:27:50.547" Score="1" ViewCount="" Body="&lt;p&gt;One more:  R bloggers has many posts with tutorials materials:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.r-bloggers.com/?s=tutorial&quot; rel=&quot;nofollow&quot;&gt;http://www.r-bloggers.com/?s=tutorial&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="253" LastActivityDate="2010-10-10T06:27:50.547" CommunityOwnedDate="2010-10-10T06:27:50.547" />
  <row Id="3458" PostTypeId="1" AcceptedAnswerId="3459" CreationDate="2010-10-10T09:27:49.817" Score="7" ViewCount="387" Body="&lt;p&gt;I am looking for an alternative to Classification Trees which might yield better predictive power.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data I am dealing with has factors for both the explanatory and the explained variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I remember coming across random forests and neural networks in this context, although never tried them before, are there another good candidate for such a modeling task (in R, obviously)?&lt;/p&gt;&#xA;" OwnerUserId="253" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-10-10T13:24:22.520" LastActivityDate="2010-10-12T08:56:16.850" Title="Alternatives to classification trees, with better predictive (e.g: CV) performance?" Tags="&lt;r&gt;&lt;machine-learning&gt;&lt;classification&gt;&lt;cart&gt;" AnswerCount="4" CommentCount="1" FavoriteCount="5" />
  <row Id="3459" PostTypeId="2" ParentId="3458" CreationDate="2010-10-10T09:50:16.577" Score="7" ViewCount="" Body="&lt;p&gt;I think it would be worth giving a try to Random Forests (&lt;a href=&quot;http://cran.r-project.org/web/packages/randomForest/index.html&quot; rel=&quot;nofollow&quot;&gt;randomForest&lt;/a&gt;); some references were provided in response to related questions: &lt;a href=&quot;http://stats.stackexchange.com/questions/2306/feature-selection-for-final-model-when-performing-cross-validation-in-machine-l/2307#2307&quot;&gt;Feature selection for “final” model when performing cross-validation in machine learning&lt;/a&gt;; &lt;a href=&quot;http://stats.stackexchange.com/questions/2410/can-cart-models-be-made-robust&quot;&gt;Can CART models be made robust?&lt;/a&gt;. Boosting/bagging render them more stable than a single CART which is known to be very sensitive to small perturbations. Some authors argued that it performed as well as penalized SVM or &lt;a href=&quot;http://en.wikipedia.org/wiki/Gradient_boosting&quot; rel=&quot;nofollow&quot;&gt;Gradient Boosting Machines&lt;/a&gt; (see, e.g. Cutler et al., 2009). I think they certainly outperform NNs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Boulesteix and Strobl provides a nice overview of several classifiers in &lt;a href=&quot;http://www.biomedcentral.com/1471-2288/9/85/&quot; rel=&quot;nofollow&quot;&gt;Optimal classifier selection and negative bias in error rate estimation: an empirical study on high-dimensional prediction&lt;/a&gt; (BMC MRM 2009 9: 85). I've heard of another good study at the &lt;a href=&quot;http://www.iqb.hu-berlin.de/veranst/EAM-SMABS&quot; rel=&quot;nofollow&quot;&gt;IV EAM meeting&lt;/a&gt;, which should be under review in &lt;em&gt;Statistics in Medicine&lt;/em&gt;,&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;João Maroco&lt;/strong&gt;, Dina Silva, Manuela Guerreiro, Alexandre de Mendonça. Do&#xA;  Random Forests Outperform Neural&#xA;  Networks, Support Vector Machines and&#xA;  Discriminant Analysis classifiers? A&#xA;  case study in the evolution to&#xA;  dementia in elderly patients with&#xA;  cognitive complaints&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I also like the &lt;a href=&quot;http://caret.r-forge.r-project.org/Classification_and_Regression_Training.html&quot; rel=&quot;nofollow&quot;&gt;caret&lt;/a&gt; package: it is well documented and allows to compare predictive accuracy of different classifiers on the same data set. It takes care of managing training /test samples, computing accuracy, etc in few user-friendly functions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;a href=&quot;http://cran.r-project.org/web/packages/glmnet/index.html&quot; rel=&quot;nofollow&quot;&gt;glmnet&lt;/a&gt; package, from Friedman and coll., implements penalized GLM (see the review in the &lt;a href=&quot;http://www.jstatsoft.org/v33/i01/&quot; rel=&quot;nofollow&quot;&gt;Journal of Statistical Software&lt;/a&gt;), so you remain in a well-known modeling framework.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Otherwise, you can also look for &lt;em&gt;association rules&lt;/em&gt; based classifiers (see the CRAN Task View on &lt;a href=&quot;http://cran.r-project.org/web/views/MachineLearning.html&quot; rel=&quot;nofollow&quot;&gt;Machine Learning&lt;/a&gt; or the &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.144.5575&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;Top 10 algorithms in data mining&lt;/a&gt; for a gentle introduction to some of them).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd like to mention another interesting approach that I plan to re-implement in R (actually, it's Matlab code) which is &lt;a href=&quot;http://www.utdallas.edu/~herve/abdi-wafo2010-mudica-inpress.pdf&quot; rel=&quot;nofollow&quot;&gt;Discriminant Correspondence Analysis&lt;/a&gt; from Hervé Abdi. Although initially developed to cope with small-sample studies with a lot of explanatory variables (eventually grouped into coherent blocks), it seems to efficiently combine classical DA with data reduction techniques.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;References&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Cutler, A., Cutler, D.R., and Stevens, J.R. (2009). Tree-Based Methods, in &lt;em&gt;High-Dimensional Data Analysis in Cancer Research&lt;/em&gt;, Li, X. and Xu, R. (eds.), pp. 83-101, Springer.&lt;/li&gt;&#xA;&lt;li&gt;Saeys, Y., Inza, I., and Larrañaga, P. (2007). &lt;a href=&quot;http://bioinformatics.oxfordjournals.org/content/23/19/2507.abstract&quot; rel=&quot;nofollow&quot;&gt;A review of feature selection techniques in bioinformatics&lt;/a&gt;. Bioinformatics, 23(19): 2507-2517.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-10T13:25:47.393" LastActivityDate="2010-10-10T13:25:47.393" CommentCount="1" />
  <row Id="3460" PostTypeId="1" AcceptedAnswerId="3464" CreationDate="2010-10-10T09:55:00.890" Score="14" ViewCount="438" Body="&lt;p&gt;For some of us, refereeing papers is part of the job. When refereeing statistical methodology papers, I think advice from other subject areas is fairly useful, i.e. &lt;a href=&quot;http://cstheory.stackexchange.com/questions/1893/how-do-i-referee-a-paper&quot;&gt;computer science&lt;/a&gt; and &lt;a href=&quot;http://mathoverflow.net/questions/36596/refereeing-a-paper&quot; rel=&quot;nofollow&quot;&gt;Maths&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This question concerns reviewing more applied statistical papers. By this I mean, the paper is submitted to a non-statistical/mathematical journal and statistics is just mentioned in the &quot;methods&quot; section.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some particular questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;How much effort should we put in to understand the application area?&lt;/li&gt;&#xA;&lt;li&gt;How much time should I spend on a report?&lt;/li&gt;&#xA;&lt;li&gt;How picky are you when looking at figures/tables.&lt;/li&gt;&#xA;&lt;li&gt;How do you cope with the data not being available.&lt;/li&gt;&#xA;&lt;li&gt;Do you try and rerun the analysis used.&lt;/li&gt;&#xA;&lt;li&gt;What's the maximum number of papers your would review in a year?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Have a missed any questions? Feel free to edit or add a comment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I coming to this question as a statistician reviewing a biology paper, but I'm interested in the statistical review of any non-mathematical discipline.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;I'm not sure if this should be a CW. On one hand it's a bit open, but on the other I can see myself accepting an answer. Also, answers will probably be fairly long.&lt;/p&gt;&#xA;" OwnerUserId="8" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-10-11T08:27:32.407" LastActivityDate="2010-10-12T09:49:10.387" Title="Reviewing statistics in papers" Tags="&lt;journals&gt;&lt;referee&gt;" AnswerCount="3" FavoriteCount="3" />
  <row Id="3461" PostTypeId="2" ParentId="3458" CreationDate="2010-10-10T10:19:27.863" Score="6" ViewCount="" Body="&lt;p&gt;For multi-class classification, support vector machines are also a good choice. I typically use the the R kernlab package for this. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;See the following JSS paper for a good discussion: &lt;a href=&quot;http://www.jstatsoft.org/v15/i09/&quot; rel=&quot;nofollow&quot;&gt;http://www.jstatsoft.org/v15/i09/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-10-10T10:19:27.863" CommentCount="5" />
  <row Id="3462" PostTypeId="2" ParentId="3296" CreationDate="2010-10-10T11:10:14.780" Score="2" ViewCount="" Body="&lt;p&gt;I think you are suffering from the presence of outliers in your design matrix. &#xA;The remedy is to detect them using a multivariate robust estimator of location/scale (just as you can use the median to detect outliers in an univariate setting but you can't use the mean because the mean itself is sensitive to the presence of outliers). High quality estimators are already present in the R-base tool (through MASS).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I advise you to read the following (non technical) summary introduction to multivariate robust method:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P. J. Rousseeuw and K. van Driessen (1999) A fast algorithm for&#xA;     the minimum covariance determinant estimator. Technometrics&#xA;     &lt;em&gt;41&lt;/em&gt;, 212-223.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are many good implementation in R, one i recommend particularly is covMcd() in package &lt;strong&gt;robustbase&lt;/strong&gt; (better than the MASS implementation because it includes the small sample correction factor).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A typical use would be:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x&amp;lt;-mydata #your 300 by 40 matrix of **design variables**&#xA;out&amp;lt;-covMcd(x)&#xA;ind.out&amp;lt;-which(out$mcd.wt==0)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, ind.out contains the indexes of the observations flagged as outliers. You should exclude them from your sample and re-run your classification procedure on the 'decontaminated' sample. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think it will stabilize your results, solve your problem. Let us know :)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT:&lt;/strong&gt; As  pointed out by Chl (in the comments, below). It could be advisable, in your case, to supplement the hard rejection rule used in the code above by a graphical method (an implementation of which can be found in the R package &lt;strong&gt;mvoutlier&lt;/strong&gt;). This is wholly consistent with the approach proposed in my answer, in fact it is well explained (and illustrated) in the paper i cite above. Therefore, i will just point out two arguments in its favor that may be particularly relevant to your case (assuming that you indeed have an outlier problem and that these can be found by the &lt;strong&gt;mcd&lt;/strong&gt;):&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Provides a visually strong illustration of the problem with outliers as each observations is associated with a measure of its influence on the resulting estimates (observations with outsized influence then stand out).&lt;/li&gt;&#xA;&lt;li&gt;The approach i proposed applies a strong rejection rule: in a nutshell, any observation whose influence over the final estimates is larger than some threshold is considered an outlier. The graphical approach might help you save some observation, by trying to recover those observations whose influence over the estimator is beyond the threshold but only by a small amount. It is important in the context of your model because 300 observations in a 40 dimensional space is rather sparse already.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="603" LastEditorUserId="603" LastEditorDisplayName="" LastEditDate="2010-10-12T15:02:24.580" LastActivityDate="2010-10-12T15:02:24.580" CommentCount="3" />
  <row Id="3463" PostTypeId="1" CreationDate="2010-10-10T11:11:52.523" Score="5" ViewCount="261" Body="&lt;p&gt;I have two time series S, and T. they have the same frequency and the same length.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to calculate (using R), the correlation between this pair (i.e. S and T), and also be able to calculate the significance of the correlation), so I can determine whether the correlation is due to chance or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to do this in R, and am looking for pointers/skeletal framework to get me started.&lt;/p&gt;&#xA;" OwnerUserId="1216" LastActivityDate="2010-10-13T06:37:12.110" Title="Computing correlation (and the significance of said correlation) between a pair of time series" Tags="&lt;r&gt;&lt;time-series&gt;&lt;correlation&gt;" AnswerCount="2" CommentCount="3" FavoriteCount="5" />
  <row Id="3464" PostTypeId="2" ParentId="3460" CreationDate="2010-10-10T11:27:35.330" Score="12" ViewCount="" Body="&lt;p&gt;I am not sure about which area of science you are referring to (I'm sure the answer would be really different if dealing with biology vs physics for instance...)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyway, as a biologist, I will answer from a &quot;biological&quot; point of view:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;How much effort should we put in to understand the application area?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I tend at least to read the previous papers from the same authors and look for a few review on the subject if I am not too familiar with it. This is especially true when dealing with new techniques I don't know, because I need to understand if they did all the proper controls etc. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;How much time should I spend on a report?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;As much as needed (OK, dumb answer, I know! :P)&#xA;In general I would not like someone reviewing my paper to do an approximative job just because he/she has other things to do, so I try not to do it myself.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;How picky are you when looking at figures/tables.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Quite picky. Figures are the first thing you look at when browsing through a paper. They need to be consistent (e.g. right titles on the axes, correct legend etc.). On occasion I have suggested to use a different kind of plot to show data when I thought the one used was not the best. This happens a lot in biology, a field that is dominated by the &quot;barplot +/- SEM&quot; type of graph.&#xA;I'm also quite picky on the &quot;materials and methods&quot; section: a perfect statistical analysis on a inherently wrong biological model is completely useless.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;How do you cope with the data not being available.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;You just do and trust the Authors, I guess. In many cases in biology there's not much you can do, especially when dealing with things like imaging or animal behaviour and similar. Unless you want people to publish tons of images, videos etc (that you most likely would not go through anyways), but that may be very unpractical. If you think the data are really necessary ask for the authors to provide them as supplementary data/figures.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Do you try and rerun the analysis used.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Only if I have serious doubts on the conclusions drawn by the authors.&#xA;In biology there's often a difference between what is (or not) &quot;statistically significant&quot; and what is &quot;biologically significant&quot;. I prefer a thinner statistical analysis with good biological reasoning then the other way around. But again, in the very unlikely event that I were to review a bio-statistics paper (ahah, that would be some fun!!!) I would probably pay much more attention to the stats than to the biology in there. &lt;/p&gt;&#xA;" OwnerUserId="582" LastEditorUserId="582" LastEditorDisplayName="" LastEditDate="2010-10-10T11:33:27.747" LastActivityDate="2010-10-10T11:33:27.747" CommentCount="3" />
  <row Id="3465" PostTypeId="2" ParentId="3458" CreationDate="2010-10-10T12:08:40.070" Score="1" ViewCount="" Body="&lt;p&gt;As already mentioned Random Forests are a natural &quot;upgrade&quot; and, these days, SVM are generally  the recommended technique to use. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to add that more often than not switching to SVM yields very disappointing results. Thing is, whilst techniques like random trees are almost trivial to use, SVM are a bit trickier. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found this paper invaluable back when I used SVM for the first time (A Practical Guide to Support Vector Classication) &lt;a href=&quot;http://www.csie.ntu.edu.tw/~cjlin/papers/guide/guide.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.csie.ntu.edu.tw/~cjlin/papers/guide/guide.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In R you can use the the e1071 package for SVM, it links against the de facto standard (in free software at least!) libSVM library. &lt;/p&gt;&#xA;" OwnerUserId="300" LastEditorUserId="300" LastEditorDisplayName="" LastEditDate="2010-10-10T17:31:45.723" LastActivityDate="2010-10-10T17:31:45.723" CommentCount="1" />
  <row Id="3466" PostTypeId="1" AcceptedAnswerId="3467" CreationDate="2010-10-10T13:04:18.347" Score="10" ViewCount="752" Body="&lt;p&gt;Imagine the following common design:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;100 participants are randomly allocated to either a treatment or a control group&lt;/li&gt;&#xA;&lt;li&gt;the dependent variable is numeric and measured pre- and post- treatment&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Three obvious options for analysing such data are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Test the group by time interaction effect in mixed ANOVA&lt;/li&gt;&#xA;&lt;li&gt;Do an ANCOVA with condition as the IV and the pre- measure as the covariate and post measure as the DV&lt;/li&gt;&#xA;&lt;li&gt;Do a t-test with condition as the IV and pre-post change scores as the DV&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question:&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What is the best way to analyse such data? &lt;/li&gt;&#xA;&lt;li&gt;Are there reasons to prefer one approach over another?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-12-25T06:55:23.157" Title="Best practice when analysing pre-post treatment-control designs" Tags="&lt;anova&gt;&lt;ancova&gt;&lt;clinical-trials&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="2" />
  <row Id="3467" PostTypeId="2" ParentId="3466" CreationDate="2010-10-10T13:59:47.777" Score="9" ViewCount="" Body="&lt;p&gt;There is a huge literature around this topic (change/gain scores), and I think the best references come from the biomedical domain, e.g. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Senn, S (2007). &lt;em&gt;Statistical issues in&#xA;  drug development&lt;/em&gt;. Wiley (chap. 7 pp.&#xA;  96-112)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In biomedical research, interesting work has also been done in the study of &lt;a href=&quot;http://en.wikipedia.org/wiki/Crossover_study&quot; rel=&quot;nofollow&quot;&gt;cross-over trials&lt;/a&gt; (esp. in relation to &lt;em&gt;carry-over&lt;/em&gt; effects, although I don't know how applicable it is to your study).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://pareonline.net/pdf/v14n6.pdf&quot; rel=&quot;nofollow&quot;&gt;From Gain Score t to ANCOVA F (and vice versa)&lt;/a&gt;, from Knapp &amp;amp; Schaffer, provides an interesting review of ANCOVA vs. t approach (the so-called Lord's Paradox). The simple analysis of change scores is not the recommended way for pre/post design according to Senn  in his article &lt;a href=&quot;http://onlinelibrary.wiley.com/doi/10.1002/sim.2682/abstract&quot; rel=&quot;nofollow&quot;&gt;Change from baseline and analysis of covariance revisited&lt;/a&gt; (Stat. Med. 2006 25(24)). Moreover, using a mixed-effects model (e.g. to account for the correlation between the two time points) is not better because you really need to use the &quot;pre&quot; measurement as a covariate to increase precision (through adjustment). Very briefly:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The use of change scores (post $-$ pre, or outcome $-$ baseline) does not solve the problem of imbalance; the correlation between pre and post measurement is &amp;lt; 1, and the correlation between pre and (post $-$ pre) is generally negative -- it follows that if the treatment (your group allocation) as measured by raw scores happens to be an unfair disadvantage compared to control, it will have an unfair advantage with change scores.&lt;/li&gt;&#xA;&lt;li&gt;The variance of the estimator used in ANCOVA is generally lower than that for raw or change scores (unless correlation between pre and post equals 1).&lt;/li&gt;&#xA;&lt;li&gt;If the pre/post relationships differ between the two groups (slope), it is not as much of a problem than for any other methods (the change scores approach also assumes that the relationship is identical between the two groups -- the parallel slope hypothesis).&lt;/li&gt;&#xA;&lt;li&gt;Under the null hypothesis of equality of treatment (on the outcome), no interaction treatment x baseline is expected; it is dangerous to fit such a model, but in this case one must use centered baselines (otherwise, the treatment effect is estimated at the covariate origin).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I also like &lt;a href=&quot;http://public.kenan-flagler.unc.edu/faculty/edwardsj/Edwards01b.pdf&quot; rel=&quot;nofollow&quot;&gt;Ten Difference Score Myths&lt;/a&gt; from Edwards, although it focuses on difference scores in a different context; but here is an &lt;a href=&quot;http://www.ori.org/~keiths/bibliography/statistics-prepost.html&quot; rel=&quot;nofollow&quot;&gt;annotated bibliography&lt;/a&gt; on the analysis of pre-post change (unfortunately, it doesn't cover very recent work). Van Breukelen also compared ANOVA vs. ANCOVA in randomized and non-randomized setting, and his conclusions support the idea that ANCOVA is to be preferred, at least in randomized studies (which prevent from regression to the mean effect).&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-10T19:11:05.503" LastActivityDate="2010-10-10T19:11:05.503" />
  <row Id="3471" PostTypeId="1" AcceptedAnswerId="3493" CreationDate="2010-10-10T20:35:54.560" Score="1" ViewCount="99" Body="&lt;p&gt;Is it possible to load an S-PLUS Linux workspace in Windows?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I try it I get this error: &quot;Problem in exists(name, where = db):&#xA;This directory has both Unix style __&lt;em&gt;nonfile and Windows style&#xA;_&lt;/em&gt;_nonfi&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The __nonfi file is created when I first try to load that Linux&#xA;workspace in Windows.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any way to convert it to a Windows workspace?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm using S-PLUS 8.0.&lt;/p&gt;&#xA;" OwnerUserId="749" LastActivityDate="2010-10-11T20:27:59.543" Title="Load Linux workspace in S-PLUS for Windows" Tags="&lt;splus&gt;" AnswerCount="2" />
  <row Id="3472" PostTypeId="2" ParentId="3307" CreationDate="2010-10-10T21:16:38.417" Score="4" ViewCount="" Body="&lt;p&gt;I doubt you're going to find a single answer to this, given the space of fractal dimensions.  Most papers (in physics, geology) looking at correlation simply stick to a Pearson correlation with fractal math reserved for identifying dimension/self-similarity, etc.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But you might be interested  in the following papers which use a &quot;Correlation Fractal Dimension&quot; as a similarity metric.  The second paper mentions a fractal clustering algorithm which employs this metric.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.33.4521&quot; rel=&quot;nofollow&quot;&gt;Estimating the Selectivity of Spatial Queries Using the `Correlation' Fractal Dimension&lt;/a&gt; (Belussi, Faloutsos, 1995)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://homepages.dcc.ufmg.br/~nivio/cursos/pa03/seminarios/seminario3/seminario3.ps&quot; rel=&quot;nofollow&quot;&gt;Characterizing Datasets Using Fractal Methods&lt;/a&gt; (Abrahao, Barbosa, 2003)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-10-10T21:16:38.417" />
  <row Id="3473" PostTypeId="1" CreationDate="2010-10-10T22:59:00.803" Score="0" ViewCount="115" Body="&lt;p&gt;Good day dear folks!&#xA;I'm apoloziging if my query may seem too much demanding, but i'm confronted to an issue, i've runned a statistical analysis involoving many tools (factor analysis, alpha of cronbach, spearman correlation) i was very happy with it, but i've been told that my approach is too much odd!!!! insignificant and irrealistic!! i will be very grateful if any one can read it and tell me what he honestly think about my work??i would like to have a neutral opinion about my analysis, i've worked so hard and i'm pretty sure that they are wrong but need the opinion of an outsider mastering statistic and data analysis to understand what is wrong with me.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks a lot in advance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Amaelle&lt;/p&gt;&#xA;" OwnerUserId="1544" LastActivityDate="2010-10-10T22:59:00.803" ClosedDate="2010-10-10T23:02:53.557" Title="data analysis consultancy" Tags="&lt;factor-analysis&gt;" CommentCount="1" />
  <row Id="3474" PostTypeId="1" AcceptedAnswerId="3481" CreationDate="2010-10-11T03:48:14.103" Score="6" ViewCount="281" Body="&lt;p&gt;As the title says, I'm looking for the marginal densities of $f (x,y) = c \sqrt{1 - x^2 - y^2}, x^2 + y^2 \leq 1$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far I have found $c$ to be $\frac{3}{2 \pi}$. I figured that out through converting $f(x,y)$ into polar coordinates and integrating over $drd\theta$, which is why I'm stuck on the marginal densities portion. I know that $f_x(x) = \int_{-\infty}^\infty f(x,y)dy$, but I'm not sure how to solve that without getting a big messy integral, and I know the answer isn't supposed to be a big messy integral. Is it possible to instead find $F(x,y)$, and then take $\frac{dF}{dx}$ to find $f_x(x)$? That seems like the intuitive way to do it but I can't seem to find anything in my textbook that states those relationships, so I didn't want to make the wrong assumptions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, I apologize if this is inappropriate to this forum. I wasn't sure if I should ask this here or at the math stackexchange since I feel like my problem might have more to do with integration than statistics, but I might also be misunderstanding the relationship between marginal density and joint density.&lt;/p&gt;&#xA;" OwnerUserId="1545" LastEditorUserId="603" LastEditorDisplayName="" LastEditDate="2010-10-11T15:18:05.577" LastActivityDate="2010-10-11T15:18:05.577" Title="Finding marginal densities of $f (x,y) = c \sqrt{1 - x^2 - y^2}, x^2 + y^2 \leq 1$" Tags="&lt;multivariable&gt;&lt;homework&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="1" />
  <row Id="3475" PostTypeId="2" ParentId="2467" CreationDate="2010-10-11T06:30:02.287" Score="4" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Caution:&lt;/strong&gt; I'm assuming that when you said &quot;classification&quot;, you are rather referring to cluster analysis (as understood in French), that is an unsupervised method for allocating individuals in homogeneous groups without any prior information/label. It's not obvious to me how class membership might come into play in your question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'll take a different perspective from the other answers and suggest you to try to do a data reduction (through PCA) of your $p$ variables followed by a mix of Ward's hierarchical and k-means clustering (this is called mixed clustering in the French literature, the basic idea is that HC is combined to a weighted k-means to consolidate the partition) on the first two or three factorial axes. This was proposed by Ludovic Lebart et coll. and is actually implemented in the &lt;a href=&quot;http://cran.r-project.org/web/packages/FactoClass/index.html&quot; rel=&quot;nofollow&quot;&gt;FactoClass&lt;/a&gt; package. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The advantages are as follows: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;If any part of your survey is not clearly unidimensional, you will be able to gauge item contribution to the second axis, and this may help to flag those items for further inspection;&lt;/li&gt;&#xA;&lt;li&gt;Clustering is done on the PCA scores (or you can work with a multiple correspondence analysis, though in the case of binary items it amounts to yield the same results than a scaled PCA), and thanks to the mixed clustering the resulting partition is more stable and allow to spot potential extreme respondents; you can also introduce supplementary variable (like gender, SES or age), which is useful to inspect between-group homogeneity.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In this case, no rotation is supposed to be applied to the principal axes. Considering a subspace with $qFactoClass package relies on &lt;a href=&quot;http://cran.r-project.org/web/packages/ade4/index.html&quot; rel=&quot;nofollow&quot;&gt;ade4&lt;/a&gt; for the factorial methods, and allows to visualize clusters in the factorial space, as shown below:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/JFtTa.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, the problem with so-called &lt;em&gt;tandem approach&lt;/em&gt; is that there is no guarantee that the low-dimensional representation that is produced by PCA or MCA will be an optimal representation for identifying cluster structures. This is nicely discussed in Hwang et al. (2006), but I'm not aware of any implementation of the algorithm they proposed. Basically, the idea is to combine MCA and k-means in a single step, which amounts to minimize two criteria simultaneously (the standard homeogeneity criterion and the residual SS).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;References&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Lebart, L, Morineau, A, and Piron, M (2000). &lt;em&gt;Statistique exploratoire multidimensionnelle&lt;/em&gt; (3rd ed.). Dunod.&lt;/li&gt;&#xA;&lt;li&gt;Hwang, H, Dillon, WR, and Takane, Y (2006). &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.133.4665&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;An extension of multiple correspondence analysis for identifying heterogeneous subgroups of respondents&lt;/a&gt;. &lt;em&gt;Psychometrika&lt;/em&gt;, &lt;em&gt;71&lt;/em&gt;, 161-171.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-13T19:20:16.370" LastActivityDate="2010-10-13T19:20:16.370" />
  <row Id="3476" PostTypeId="1" AcceptedAnswerId="3477" CreationDate="2010-10-11T06:39:50.770" Score="2" ViewCount="511" Body="&lt;p&gt;Python &lt;a href=&quot;http://matplotlib.sourceforge.net/&quot; rel=&quot;nofollow&quot;&gt;matplotlib&lt;/a&gt; has a &lt;a href=&quot;http://matplotlib.sourceforge.net/api/pyplot_api.html#matplotlib.pyplot.boxplot&quot; rel=&quot;nofollow&quot;&gt;boxplot command&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Normally, all the parts of the graph are numerically ticked. How can I change the ticks to names instead of positions?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For illustration, I mean the Mon Tue Wed labels like in this boxplot:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/ZFENf.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="190" LastActivityDate="2010-10-12T18:59:33.793" Title="How to name the ticks in a python matplotlib boxplot" Tags="&lt;python&gt;&lt;matplotlib&gt;" AnswerCount="2" />
  <row Id="3477" PostTypeId="2" ParentId="3476" CreationDate="2010-10-11T07:12:12.017" Score="6" ViewCount="" Body="&lt;p&gt;Use the second argument of &lt;code&gt;xticks&lt;/code&gt; to set the labels:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import pylab&#xA;&#xA;data = [[np.random.rand(100)] for i in range(3)]&#xA;pylab.boxplot(data)&#xA;pylab.xticks([1, 2, 3], ['mon', 'tue', 'wed'])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-10-11T07:12:12.017" />
  <row Id="3478" PostTypeId="2" ParentId="3460" CreationDate="2010-10-11T08:09:36.607" Score="5" ViewCount="" Body="&lt;p&gt;My POV would be reviewing a paper in psychology or forecasting on its statistical merits. I'll mostly second Nico's very good remarks.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;How much effort should we put in to&#xA;  understand the application area?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Quite a lot, actually. I wouldn't trust myself to comment on more than the most basic statistical problems without having understood the area. Fortunately, this is often not very hard in many branches of psychology.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;How much time should I spend on a&#xA;  report?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I'll go out on a limb and state a specific time: I'll spend anything between two and eight hours on a review, sometimes more. If I find that I'm spending more than a day on a paper, it probably means that I'm really not qualified to understand it, so I'll recommend the journal find someone else (and try to suggest some people).&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;How picky are you when looking at&#xA;  figures/tables.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Very&lt;/em&gt; picky indeed. The figures are going to be what people remember of a paper and what ends up in lecture presentations without much context, so these really need to be done well.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;How do you cope with the data not&#xA;  being available.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In psychology, the data are usually not shared - measuring 50 people by MRI is very expensive, and the authors will want to use these data for further papers, so I kind of understand their reluctance to just give out the data. So anyone who does share their data gets a big bonus in my book, but not sharing is understandable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In forecasting, many datasets are publicly available. In this case I usually recommend that the authors share their code (and do so myself).&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Do you try and rerun the analysis&#xA;  used.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Without the data, there is only so much one can learn from this. I'll play around with simulated data if something is very surprising about the paper's results; otherwise one can often tell appropriate from inappropriate methods without the data (once one understands the area, see above).&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;What's the maximum number of papers&#xA;  your would review in a year?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;There is really little to add to whuber's point above - assuming that every paper with on average &lt;em&gt;n&lt;/em&gt; coauthors I (co-)submit gets 3 reviews, one should really aim at reviewing at least 3/(&lt;em&gt;n+1&lt;/em&gt;) papers for each own submission (counting submissions rather than own papers which may be rejected and resubmitted). And of course, the number of submissions as well as the number of coauthors varies strongly with the discipline.&lt;/p&gt;&#xA;" OwnerUserId="1352" LastEditorUserId="1352" LastEditorDisplayName="" LastEditDate="2010-10-12T09:49:10.387" LastActivityDate="2010-10-12T09:49:10.387" CommentCount="3" />
  <row Id="3479" PostTypeId="1" AcceptedAnswerId="3482" CreationDate="2010-10-11T08:37:31.890" Score="6" ViewCount="214" Body="&lt;p&gt;What is the rationale, if any, to use Discriminant Analysis (DA) on the results of a clustering algorithm like k-means, as I see it from time to time in the literature (essentially on clinical subtyping of mental disorders)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is generally not recommended to test for group differences on the variables that were used during cluster construction since they support the maximisation (resp. minimisation) of between-class (resp. within-class) inertia. So, I am not sure to fully appreciate the added value of predictive DA, unless we seek to embed individuals in a factorial space of lower dimension and get an idea of the &quot;generalizability&quot; of such a partition. But even in this case, cluster analysis remains fundamentally an exploratory tool, so using class membership computed this way to further derive a scoring rule seems strange at first sight.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any recommendations, ideas or pointers to relevant papers?&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-10-11T15:10:57.870" Title="Cluster Analysis followed by Discriminant Analysis" Tags="&lt;clustering&gt;&lt;discriminant-analysis&gt;" AnswerCount="1" />
  <row Id="3480" PostTypeId="2" ParentId="3471" CreationDate="2010-10-11T13:11:51.507" Score="1" ViewCount="" Body="&lt;p&gt;I don't know for certain, but that won't stop me from wildly speculating:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The __nonfi file lists what's in the workspace.  You can open it with a text editor and look at the contents.  It might be possible to either manipulate the unix version (e.g. using dos2unix) or else copy the contents over into your new file.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That said, I doubt that this will work since some of the S-Plus files are in binary format, and I have run into trouble in the past when I have tried to manually change the __nonfi file.  This question might be better served by Tibco technical support.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please update us here if you get an answer.&lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-10-11T13:11:51.507" CommunityOwnedDate="2010-10-11T20:31:15.973" />
  <row Id="3481" PostTypeId="2" ParentId="3474" CreationDate="2010-10-11T14:55:07.220" Score="5" ViewCount="" Body="&lt;p&gt;Geometry helps here.  The graph of $f$ is a spherical dome of unit radius.  (It follows immediately that its volume is half that of a unit sphere, $(4 \pi /3)/2$, whence $c=3/(2 \pi)$.)  The marginal densities are given by areas of vertical cross-sections through this sphere.  Obviously each cross-section is a semicircle: to obtain the marginal density, find its radius as a function of the remaining variable and use the formula for the area of a circle.  Normalizing the resulting univariate function to have unit area turns it into a density.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-10-11T14:55:07.220" CommentCount="9" />
  <row Id="3482" PostTypeId="2" ParentId="3479" CreationDate="2010-10-11T15:10:57.870" Score="3" ViewCount="" Body="&lt;p&gt;I don't know of any papers on this.  I've used this approach, for descriptive purposes.  DFA provides a nice way to summarize group differences and dimensionality with respect to the original variables.  One might more easily just profile the groups on the original variables, however, this loses the inherently multivariate nature of the clustering problem.  DFA allows you to describe the groups while keeping the multivariate character of the problem intact.  So, it can assist with the interpretation of the clusters, where that is a goal.  This is particularly ideal when there is a close relationship between your clustering method and your classification method--e.g., DFA and Ward's method. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You are right about the testing problem.  I published a paper using the Cluster Analysis with DFA follow-up to describe the clustering solution.  I presented the DFA results with no test statistics.  A reviewer took issue with that.  I conceded and put the test statistics and p values in there, with the disclaimer that these p-values should not be interpreted in the traditional manner.  &lt;/p&gt;&#xA;" OwnerUserId="485" LastActivityDate="2010-10-11T15:10:57.870" />
  <row Id="3483" PostTypeId="2" ParentId="3460" CreationDate="2010-10-11T15:35:26.567" Score="7" ViewCount="" Body="&lt;p&gt;This addresses the new question #6: &quot;What's the maximum number of papers you would review in a year?&quot;  I'm responding as a member of several editorial boards.  The perennial problem is finding enough reviewers.  Depending on the journal, every submitted paper needs one to three peer reviewers, usually three.  If the journal has an $x$% acceptance rate, then the mean number of reviews &lt;em&gt;per accepted paper&lt;/em&gt; obviously is around $3/(x/100)$.  &lt;em&gt;E.g.&lt;/em&gt;, if the acceptance rate is 33%, the editors need to obtain nine reviews for every paper published.  If you, as an author, take this seriously, then you should attempt to provide nine reviews (or whatever the number turns out to be for your target journals) for every paper you publish!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was moved to write this due to the strong parallel with voting on this site: in order for you to garner a reputation of $r$, other people have to upvote some combination of $r/10$ of your answers and $r/5$ of your questions.  Thus, if you're pulling your weight, a check of your profile should show &lt;em&gt;at least&lt;/em&gt; $r/10$ upvotes.  That is the case for many but certainly not all of the most active members of this site.  Something to think about...  Remember to vote!&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-10-11T15:56:43.347" LastActivityDate="2010-10-11T15:56:43.347" CommentCount="3" />
  <row Id="3484" PostTypeId="1" AcceptedAnswerId="3487" CreationDate="2010-10-11T15:42:06.377" Score="5" ViewCount="387" Body="&lt;p&gt;I have been looking at analyst job postings and one of the most common requirement is experience of SAS. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Unless your organisation currently uses SAS, how can you train as a SAS user?&lt;/li&gt;&#xA;&lt;li&gt;What programming language would be equivalent to SAS that employers might be happy to accept?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="1077" LastActivityDate="2011-05-20T06:18:32.570" Title="Obtaining SAS experience" Tags="&lt;sas&gt;&lt;job&gt;" AnswerCount="4" FavoriteCount="1" />
  <row Id="3485" PostTypeId="2" ParentId="3199" CreationDate="2010-10-11T15:46:43.540" Score="3" ViewCount="" Body="&lt;p&gt;Good question.  A trivial way to find &quot;cluster of high values in the upper left&quot;&#xA;(as opposed to correlations)&#xA;is to split the image into tiles and look at tile means. For example,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;means of 100 x 100 tiles:&#xA;[[ 82  78  80  94  99 100]&#xA; [ 80  53  66  62  80 100]&#xA; [ 82  61  65  64  72  98]&#xA; [ 87  83  99  81  80 100]&#xA; [100 100 100 100 100 100]]&#xA;&#xA;means of 50 x 50 tiles:&#xA;[[100  85  84 100  70  96 100 100 100 100 100]&#xA; [ 83  59  57  71  67  88  89  86  98 100 100]&#xA; [ 87  58  54  49  71  74  71  61  61 100 100]&#xA; [100  76  58  52  59  61  55  59  65  95 100]&#xA; [100  62  59  60  57  63  60  60  59  97 100]&#xA; [100  68  65  59  59  82  76  61  61  70  95]&#xA; [ 83  64  76  66  96 100  96  61  80  67 100]&#xA; [100 100  97  92 100 100  84  82  83  88 100]&#xA; [100 100 100 100 100 100 100 100 100 100 100]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(a plot with average height / colour in each tile would be 10x better).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(If you're looking for features in images, what's a &quot;feature&quot; ?&#xA;E.g. a red stop sign, as in&#xA;&lt;a href=&quot;http://www.cse.yorku.ca/~sizints/histogram_search/histogram_search_main.html&quot; rel=&quot;nofollow&quot;&gt;Histograms for feature representation&lt;/a&gt;&#xA;)&lt;/p&gt;&#xA;" OwnerUserId="557" LastActivityDate="2010-10-11T15:46:43.540" />
  <row Id="3487" PostTypeId="2" ParentId="3484" CreationDate="2010-10-11T17:08:44.130" Score="4" ViewCount="" Body="&lt;p&gt;I would recommend going through a self-study course such as the &lt;a href=&quot;http://www.ats.ucla.edu/stat/sas/&quot; rel=&quot;nofollow&quot;&gt;UCLA website&lt;/a&gt; and specifically the &lt;a href=&quot;http://www.ats.ucla.edu/stat/sas/sk/default.htm&quot; rel=&quot;nofollow&quot;&gt;SAS Starter Kit&lt;/a&gt;.  If you learn better within an interactive environment, I would suggest checking out online course offerings such as the &lt;a href=&quot;http://www.worldcampus.psu.edu/AppliedStatisticsCertificate_CourseList.shtml&quot; rel=&quot;nofollow&quot;&gt;World Campus SAS courses&lt;/a&gt; offered at Penn State University (Stat 480, 481, &amp;amp; 482).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update:&lt;/strong&gt; Sorry should've read more carefully, I agree with @Christoper.Aden that there aren't really any equivalent languages with SAS.  You can learn R to perform statistical calculations, but if you need to use SAS, then learning R will only be a small step in the right direction (general programming knowledge - the two languages are incredibly different in practice).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would recommend getting an academic discount version of SAS if you enroll in a program like I mentioned above - Penn State currently sells a 1yr licensed copy of SAS for $30 (only to students).&lt;/p&gt;&#xA;" OwnerUserId="1499" LastEditorUserId="1499" LastEditorDisplayName="" LastEditDate="2010-10-11T17:37:01.560" LastActivityDate="2010-10-11T17:37:01.560" CommentCount="2" />
  <row Id="3488" PostTypeId="2" ParentId="3463" CreationDate="2010-10-11T17:17:26.393" Score="1" ViewCount="" Body="&lt;p&gt;You can use the ccf function to get the cross-correlation, but this will only give you a plot.  If the estimated cross correlations fall outside the dash red line, then you can conclude that there is a statistically significant cross-correlation.  But I do not know of a package with a formally encapsulated test.  Example from ccf doc:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(graphics)&#xA;&#xA;## Example from Venables &amp;amp; Ripley (Provided in  CCF help file)&#xA;ccf(mdeaths, fdeaths, ylab = &quot;cross-correlation&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Note, that the question of significance test is also discussed &lt;a href=&quot;http://stats.stackexchange.com/questions/3115/cross-correlation-significance-in-r&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1499" LastActivityDate="2010-10-11T17:17:26.393" CommentCount="1" />
  <row Id="3489" PostTypeId="1" AcceptedAnswerId="3508" CreationDate="2010-10-11T17:21:37.217" Score="5" ViewCount="598" Body="&lt;p&gt;I have a biometric system that outputs a distribution of scores that resembles a Gaussian distribution (similar to the example graph in the following link: &lt;a href=&quot;http://support.bioid.com/sdk/docs/About_EER.htm&quot; rel=&quot;nofollow&quot;&gt;LINK&lt;/a&gt;). My point of confusion is how I calculate the False Acceptance Rate. How does threshold factor into the whole problem?&lt;/p&gt;&#xA;" OwnerUserId="1224" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-10-20T21:12:08.117" LastActivityDate="2010-10-20T21:12:08.117" Title="Calculating False Acceptance Rate for a Gaussian Distribution of scores" Tags="&lt;bioinformatics&gt;" AnswerCount="3" FavoriteCount="2" />
  <row Id="3490" PostTypeId="2" ParentId="3484" CreationDate="2010-10-11T17:27:37.583" Score="4" ViewCount="" Body="&lt;p&gt;As far as SAS goes, getting &lt;a href=&quot;http://support.sas.com/certify/&quot; rel=&quot;nofollow&quot;&gt;certified is resume gold&lt;/a&gt;. The SAS Institute offers &lt;a href=&quot;http://support.sas.com/certify/creds/prep.html&quot; rel=&quot;nofollow&quot;&gt;classes and exams&lt;/a&gt; to receive the certification. There are also books you can use if you are self-motivated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Getting SAS is quite difficult if your company does not have it. I'm on a college campus, and they offer academic discounts on student liscenses and the campus labs have it installed on some machines. If you want something a little similar, but cheaper, give JMP a try. It's probably the closest thing to the SAS feel.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For similar languages, it would probably depend on your field. The social sciences would probably be more receptive to seeing SPSS on your resume than would the economics-related work. &lt;/p&gt;&#xA;" OwnerUserId="1118" LastActivityDate="2010-10-11T17:27:37.583" CommentCount="4" />
  <row Id="3491" PostTypeId="2" ParentId="3489" CreationDate="2010-10-11T17:33:16.783" Score="5" ViewCount="" Body="&lt;p&gt;I'm not certain.  I'm curious as to the other responses you get.  However, I think you'll need to clarify a bit:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does your Gaussian distribution represent the scores for a population of individuals which should be rejected by your biometric system?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If so, then I think you simply need to compute a cumulative probability - i.e. the percentage of individuals which should be rejected but who, by random chance, fall above your threshold and are &quot;falsely accepted&quot; by your biometric device.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, it could be as simply as computing the number of people who randomly fall above your threshold divided by the total number of &quot;should be rejected&quot; people.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But again, I'm not certain of my response and I think you need to clarify what your assumptions are, what your threshold is, and how you wish to classify individuals as &quot;falsely rejected&quot;.&lt;/p&gt;&#xA;" OwnerUserId="1499" LastActivityDate="2010-10-11T17:33:16.783" CommentCount="7" />
  <row Id="3492" PostTypeId="2" ParentId="3484" CreationDate="2010-10-11T18:07:43.963" Score="1" ViewCount="" Body="&lt;p&gt;The programming language most similar to SAS is... SAS.  Which you can interpret using &lt;a href=&quot;http://en.wikipedia.org/wiki/World_Programming_System&quot; rel=&quot;nofollow&quot;&gt;WPS, which will run SAS code and evidently costs substantially less than a SAS license&lt;/a&gt; and has &lt;a href=&quot;http://www.teamwpc.co.uk/tryorbuy&quot; rel=&quot;nofollow&quot;&gt;a 30 day free trial&lt;/a&gt;.  I haven't used it myself, but it should get you started programming in the SAS language.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As M. Tibbits suggests, I don't think that experience with R would be helpful in most corporate settings.  I also don't think that SPSS experience will be all that helpful, either, and my sense is that it has a less than stellar reputation outside of the social sciences.&lt;/p&gt;&#xA;" OwnerUserId="71" LastActivityDate="2010-10-11T18:07:43.963" CommentCount="2" />
  <row Id="3493" PostTypeId="2" ParentId="3471" CreationDate="2010-10-11T20:27:59.543" Score="3" ViewCount="" Body="&lt;p&gt;Tibco support gave me a solution:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Create a new Windows workspace&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Attach the Linux workspace&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;attach(&quot;C:\\Linux\\Workspace\\Path&quot;)&lt;/code&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Copy the contents of the Linux workspace to the Windows workspace&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;objs &amp;lt;- objects(2)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;for (i in objs) assign(i, value=get(i, where=2), where=1)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;objs &amp;lt;- objects(2, meta=1)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;for (i in objs) assign(i, value=get(i, where=2, meta=1), where=1, meta=1)&lt;/code&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Almost everything was copied. There were problems with functions named like &lt;code&gt;lBounds&amp;lt;-&lt;/code&gt;, which were renamed to &lt;code&gt;lBounds_-&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="749" LastActivityDate="2010-10-11T20:27:59.543" />
  <row Id="3494" PostTypeId="2" ParentId="3271" CreationDate="2010-10-11T21:48:29.173" Score="0" ViewCount="" Body="&lt;p&gt;One crucial question about Mfuzz that remains unanswered is how many clusters to use. I am still confused about it. I am analyzing 20k genes at 5 different timepoints. My best guess is to cluster them by number of GO groupings, but I'm not so sure about that. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Btw, csgillespie, your paper has been a great help!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regards,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Shankar&lt;/p&gt;&#xA;" LastActivityDate="2010-10-11T21:48:29.173" CommentCount="1" />
  <row Id="3495" PostTypeId="2" ParentId="3377" CreationDate="2010-10-11T23:15:33.223" Score="0" ViewCount="" Body="&lt;p&gt;The following provides an inaccurate approximation, although the inaccuracy will depend on the distribution of the input data.  It is an online algorithm, but only approximates the absolute deviance.  It is based on a &lt;a href=&quot;http://www.johndcook.com/blog/2008/09/26/comparing-three-methods-of-computing-standard-deviation/&quot; rel=&quot;nofollow&quot;&gt;well known algorithm&lt;/a&gt; for calculating variance online, described by &lt;a href=&quot;http://www.jstor.org/stable/1266577&quot; rel=&quot;nofollow&quot;&gt;Welford&lt;/a&gt; in the 1960s.  His algorithm, translated into R, looks like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;M2 &amp;lt;- 0&#xA;mean &amp;lt;- 0&#xA;n &amp;lt;- 0&#xA;&#xA;var.online &amp;lt;- function(x){&#xA;    n &amp;lt;&amp;lt;- n + 1&#xA;    diff &amp;lt;- x - mean&#xA;    mean &amp;lt;&amp;lt;- mean + diff / n&#xA;    M2 &amp;lt;&amp;lt;- M2 + diff * (x - mean)&#xA;    variance &amp;lt;- M2 / (n - 1)&#xA;    return(variance)&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It performs very similarly to R's builtin variance function:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(2099)&#xA;n.testitems &amp;lt;- 1000&#xA;n.tests &amp;lt;- 100&#xA;differences &amp;lt;- rep(NA, n.tests)&#xA;for (i in 1:n.tests){&#xA;        # Reset counters&#xA;        M2 &amp;lt;- 0&#xA;        mean &amp;lt;- 0&#xA;        n &amp;lt;- 0&#xA;&#xA;        xs &amp;lt;- rnorm(n.testitems)&#xA;        for (j in 1:n.testitems){&#xA;                v &amp;lt;- var.online(xs[j])&#xA;        }&#xA;&#xA;        differences[i] &amp;lt;- abs(v - var(xs))&#xA;&#xA;}&#xA;summary(differences)&#xA;     Min.   1st Qu.    Median      Mean   3rd Qu.      Max. &#xA;0.000e+00 2.220e-16 4.996e-16 6.595e-16 9.992e-16 1.887e-15 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Modifying the algorithm to calculate absolute deviation simply involves an additional &lt;code&gt;sqrt&lt;/code&gt; call.  However, the &lt;code&gt;sqrt&lt;/code&gt; introduces inaccuracies that are reflected in the result:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;absolute.deviance.online &amp;lt;- function(x){&#xA;    n &amp;lt;&amp;lt;- n + 1&#xA;    diff &amp;lt;- x - mean&#xA;    mean &amp;lt;&amp;lt;- mean + diff / n&#xA;    a.dev &amp;lt;&amp;lt;- a.dev + sqrt(diff * (x - mean))&#xA;    return(a.dev)&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The errors, calculated as above, are much greater than for the variance calculation:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    Min.  1st Qu.   Median     Mean  3rd Qu.     Max. &#xA;0.005126 0.364600 0.808000 0.958800 1.360000 3.312000 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, depending on your use case, this magnitude of error might be acceptable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/bLZr0.png&quot; alt=&quot;historgram of differences&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="179" LastEditorUserId="179" LastEditorDisplayName="" LastEditDate="2010-11-09T05:36:27.843" LastActivityDate="2010-11-09T05:36:27.843" CommentCount="9" />
  <row Id="3496" PostTypeId="1" CreationDate="2010-10-11T23:45:58.103" Score="9" ViewCount="236" Body="&lt;p&gt;(I'm a newbie at stats. I'm a mathematician and a programmer and I'm trying to build something like a naive Bayesian spam filter.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've noticed in many places that people tend to break down the denominator in the equation from Bayes' Theorem. So instead of this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\frac{P(A|B)\cdot P(B)}{P(A)}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We are presented with this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\frac{P(A|B)\cdot P(B)}{P(A|B)\cdot P(B)+P(A|\neg B)\cdot P(\neg B)}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can see that this convention is used in &lt;a href=&quot;http://en.wikipedia.org/wiki/Bayesian_spam_filtering#Computing_the_probability_that_a_message_containing_a_given_word_is_spam&quot; rel=&quot;nofollow&quot;&gt;this Wikipedia article&lt;/a&gt; and in this &lt;a href=&quot;http://mail.python.org/pipermail/python-dev/2002-August/028216.html&quot; rel=&quot;nofollow&quot;&gt;insightful post&lt;/a&gt; by Tim Peters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am baffled by this. Why is the denominator broken down like this? How does that help things at all? What's so complicated about calculating $P(A)$, which in the case of spam filters would be &lt;code&gt;The probability that the word &quot;cheese&quot; appears in an email, regardless of whether it's spam or not&lt;/code&gt;?&lt;/p&gt;&#xA;" OwnerUserId="1523" LastActivityDate="2010-10-12T08:30:55.770" Title="Why break down the denominator in Bayes' Theorem?" Tags="&lt;bayesian&gt;" AnswerCount="3" CommentCount="1" FavoriteCount="4" />
  <row Id="3497" PostTypeId="1" CreationDate="2010-10-11T23:47:31.373" Score="8" ViewCount="176" Body="&lt;p&gt;Hello All,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a fairly larege file 100M rows and 30 columns or so on which I would like to run multiple regressions. I have specialized code to run the regressions on the entire file, but what I would like to do is draw random samples from the file and run them in R.&#xA;The strategy is:&#xA;              randomly sample N rows from the file without replacement&#xA;              run a regression and save the coefficients of interest&#xA;              repeat this process M times with different samples &#xA;              for each coefficient calculate the means and standard errors  of &#xA;                the coefficeints over M runs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to interpret the mean computed over M runs as an estimate of the values of the coefficients computed on the whole data set, and the stadard errors of the means as estimates of the standard errors of the coefficients computed on the entire data set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Experiments show this to be a promising strategy, but I am not sure about the underlying theory. Are my estimators consistent efficient and unbiased? If they are consistent how quickly should they converge? What tradeoffs of M and N are best?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would very much appreciate it if someone could point me to the papers, books etc. with the relevanth theory.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Best regards and many thanks,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Joe Rickert&lt;/p&gt;&#xA;" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-10-12T08:23:45.100" LastActivityDate="2010-12-19T21:49:54.970" Title="Doing regressions on samples from a very large file: are the means and SEs of the sample coefficients consistent estimators?" Tags="&lt;r&gt;&lt;regression&gt;&lt;large-data&gt;&lt;bootstrap&gt;" AnswerCount="3" CommentCount="1" FavoriteCount="2" />
  <row Id="3500" PostTypeId="2" ParentId="3496" CreationDate="2010-10-12T01:53:18.930" Score="5" ViewCount="" Body="&lt;p&gt;One reason for using the total probability rule is that we often deal with the component probabilities in that expression and it's straightforward to find the marginal probability by simply plugging in the values.  For an illustration of this, see the following example on Wikipedia:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Bayes%27_theorem#Example_1%3a_Drug_testing&quot; rel=&quot;nofollow&quot;&gt;Bayes' Theorem &gt; Example 1: Drug Testing&lt;/a&gt; &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Another reason is recognizing equivalent forms of Bayes' Rule by manipulating that expression.  For example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(B|A) = \frac{P(A|B) P(B)}{P(A|B)P(B) + P(A|\lnot B)P(\lnot B)}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Divide through the RHS by the numerator:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(B|A) = \frac{1} {1 + \frac{P(A|\lnot B)}{P(A|B)} \frac{P(\lnot B)}{P(B)}}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which is a nice equivalent form for Bayes' Rule, made even handier by subtracting this from the original expression to obtain:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\frac{P(\lnot B|A)}{P(B|A)} = \frac{P(A|\lnot B)} {P(A|B)} \frac {P(\lnot B)} {P(B)}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is Bayes' Rule stated in terms of Odds, i.e. &lt;em&gt;posterior&lt;/em&gt; odds against B = Bayes factor against B times the &lt;em&gt;prior&lt;/em&gt; odds against B.  (Or you could invert it to get an expression in terms of odds for B.)  The Bayes factor is the ratio of the likelihoods of your models.  Given that we're uncertain about the underlying data generating mechanism, we observe data and update our beliefs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not sure if you find this useful, but hopefully it's not baffling; you should obviously work with the expression that works best for your scenario.  Maybe someone else can pipe in with even better reasons.&lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-10-12T01:53:18.930" CommentCount="1" />
  <row Id="3501" PostTypeId="2" ParentId="3496" CreationDate="2010-10-12T02:16:56.347" Score="8" ViewCount="" Body="&lt;p&gt;The short answer to your question is, &quot;most of the time we don't know what P(cheese) is, and it is often (relatively) difficult to calculate.&quot; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The longer answer why Bayes' Rule/Theorem is normally stated in the way that you wrote is because in Bayesian problems we have - sitting in our lap - a prior distribution (the P(B) above) and likelihood (the P(A|B), P(A|notB) above) and it is a relatively simple matter of multiplication to compute the posterior (the P(B|A)).  Going to the trouble to reexpress P(A) in its summarized form is effort that could be spent elsewhere.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It might not seem so complicated in the context of an email because, as you rightly noted, it's just P(cheese), right?  The trouble is that with more involved on-the-battlefield Bayesian problems the denominator is an unsightly integral, which may or may not have a closed-form solution.  In fact, sometimes we need sophisticated Monte Carlo methods just to &lt;em&gt;approximate&lt;/em&gt; the integral and churning the numbers can be a real pain in the rear. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But more to the point, we usually don't even care what P(cheese) is.  Bear in mind, we are trying to hone our belief regarding whether or not an email is &lt;strong&gt;spam&lt;/strong&gt;, and couldn't care less about the marginal distribution of the data (the P(A), above).  It is just a normalization constant, anyway, which doesn't depend on the parameter;  the act of summation washes out whatever info we had about the parameter.  The constant is a nuisance to calculate and is ultimately irrelevant when it comes to zeroing in on our beliefs about whether or not the email's spam.  Sometimes we are obliged to calculate it, in which case the quickest way to do so is with the info we already have: the prior and likelihood.&lt;/p&gt;&#xA;" OwnerUserId="1108" LastEditorUserId="1108" LastEditorDisplayName="" LastEditDate="2010-10-12T03:49:44.033" LastActivityDate="2010-10-12T03:49:44.033" />
  <row Id="3502" PostTypeId="2" ParentId="3489" CreationDate="2010-10-12T03:00:29.283" Score="5" ViewCount="" Body="&lt;p&gt;it sounds as tho the following simplified situation may capture the essence of your problem:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;there are two populations of individuals: A = acceptable individuals and U = unacceptables. associated&#xA;with each individual is a 'score' $X$. suppose in each of the two populations, the scores have &#xA;gaussian distributions, where in A, the [true] mean is $\mu_A$ and in U, it is $\mu_U$.&#xA;we can also suppose [altho need not] that the distributions have the same SD = $\sigma$. &#xA;all three [or four] parameters are presumably known.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;suppose $\mu_A &gt; \mu_U$, so it makes sense to accept an individual if their 'score' $X$ is above some threshold $c$, say.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;there are two ways this rule can go wrong:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;an $X$ from U can exceed $c$, leading to a false acceptance.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;an $X$ from A can be below $c$, leading to a false rejection.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;the probabilities&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$err_{falseacc} = P(N(\mu_U, \sigma^2) &gt; c)$$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;and &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$err_{falserej} = P(N(\mu_A, \sigma^2) &amp;lt; c)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;are the two error rates associated with the rule. you are focusing on $err_{falseacc}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;it is not difficult to see that as the threshold $c$ is changed, one error-rate will decrease and the&#xA;other will increase. so $c$ has to be chosen to give values of both error-rates that one can live with.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;once you choose $c$, as others have already remarked, the error rates can be calculated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;in the language of statistics, you are testing two hypotheses about the $\mu$ of the population that the individual with observed score $X$ came from. one hypothesis is H$_A: \mu = \mu_A$ and the other is&#xA;H$_U: \mu = \mu_U$. the 'test' to decide between these two hypotheses is the above rule and the error&#xA;rates given above are [somewhat unhelpfully] called the type I and type II errors, or [equally unhelpfully, &#xA;IMHO] the sensitivity and the specificity or [likewise] the producer's risk and the consumer's risk. which is which depends on which of the two hypotheses is designated&#xA;as the 'null hypothesis', a distinction that may not be entirely helpful in this context.&lt;/p&gt;&#xA;" OwnerUserId="1112" LastActivityDate="2010-10-12T03:00:29.283" />
  <row Id="3503" PostTypeId="2" ParentId="3497" CreationDate="2010-10-12T04:24:05.447" Score="2" ViewCount="" Body="&lt;p&gt;The greater the sample N, the smaller the standard error (higher t stat, and smaller the respective p values) associated with all your regression coefficients.  The greater M, the more datapoints you will have and the smaller will be your standard error of the mean of the coefficients over M runs.  Such means should have a standard error that is normally distributed per the Central Limit Theorem.  In terms of convergence of such means, I am not sure there are any statistical principles that dictate this.  I suspect if your random sampling is well done (no structural bias, etc...) the convergence should occur fairly rapidly.  That is something you just may have to observe empirically.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Otherwise, your method seems good, I don't see any problem with it.   &lt;/p&gt;&#xA;" OwnerUserId="1329" LastEditorUserId="1329" LastEditorDisplayName="" LastEditDate="2010-10-12T16:10:17.650" LastActivityDate="2010-10-12T16:10:17.650" CommentCount="3" />
  <row Id="3504" PostTypeId="1" AcceptedAnswerId="3506" CreationDate="2010-10-12T04:26:28.497" Score="5" ViewCount="847" Body="&lt;p&gt;I measured response variable $Y$ at three levels of factor $A$ and four levels of factor $B$, $n=6$ reps/treatment. Results include&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;$A$ has strong effects on $Y$.&lt;/li&gt;&#xA;&lt;li&gt;$B$ has no effect on $Y$&lt;/li&gt;&#xA;&lt;li&gt;There is no $A*B$ interaction&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I would like to report all three results (the second two are actually more interesting than the first since all were expected). So far I have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;The effect of A on Y was significant (ANOVA, $P&amp;lt;0.001$) and $Y$ was different at each level of $A$ (Tukey HSD, $P&amp;lt;0.001$). There was no effect of $B$ on $Y$, and there was no interaction between $A$ and $B.$ This implies that $B$ had no effect on $Y$ &lt;strong&gt;and that the effect of $B$ on $Y$ was similar for all levels of $A$.&lt;/strong&gt;&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Text with bold emphasis is in question) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This approach minimizes the double negative as in &quot;the effect of $B$ on $Y$ was not different at different levels of $A$&quot;. Still, neither sounds 'right' to me, thus my post.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my field, as in many fields of science, non-significant responses are rarely mentioned in the text, and even more rarely are they interpreted, but in this case, previous studies find that $B$ almost always affects $Y$ and the lack of an interaction is notable.&lt;/p&gt;&#xA;" OwnerUserId="1381" LastEditorUserId="1381" LastEditorDisplayName="" LastEditDate="2010-10-12T08:34:15.140" LastActivityDate="2010-10-12T11:11:42.463" Title="If an ANOVA indicates no main effect and no interaction, should the lack of interaction be stated?" Tags="&lt;anova&gt;&lt;multiple-comparisons&gt;&lt;interpretation&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="3505" PostTypeId="2" ParentId="3504" CreationDate="2010-10-12T07:34:16.077" Score="4" ViewCount="" Body="&lt;p&gt;Hello David,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please let me know if you have replicates in your experiment. As you mention using Tukey HSD, I am guessing you don't have any replicates. If you experiment analyzes test of additivity in a two-way factorial Analysis of Variance (ANOVA) with one observation per cell, then please read ahead or otherwise ignore my solution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming I am thinking in the right direction, it may be possible that &lt;strong&gt;B affects A, but it doesn't affect Y in form of a multiplicative term like A*B&lt;/strong&gt;. In general, B can &lt;em&gt;affect&lt;/em&gt;  &lt;em&gt;Y&lt;/em&gt; in even if the interaction &lt;em&gt;A&lt;/em&gt; * &lt;em&gt;B&lt;/em&gt; is not significant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reason for this is the assumption of special form of relation between A and B (ie A*B) that affects Y in Tukey HSD Test. Please see the wikipedia article about &lt;a href=&quot;http://en.wikipedia.org/wiki/Tukey%27s_test_of_additivity&quot; rel=&quot;nofollow&quot;&gt;Tukey HSD&lt;/a&gt; for further details.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am sorry if this was not the test that you were conducting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But even if the interaction is not statistically significant, it doesn't imply that B doesn't affect Y. Even if this is a randomized experiment and you are allowed to relate significance and causation, but &lt;strong&gt;not able to reject the null hypothesis $H_o$ (ie p-value is $\geq \alpha$ ) doesn't mean that you accept $H_o$&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please accept my apologies, if I completely misunderstood you.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update after David's details&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A few common themes that should be addressed first:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;If you expect interaction in your main effects here &lt;em&gt;A and B&lt;/em&gt;, then testing for the significance of main effects doesn't make sense. (cf. an elegant paper by Dr. Venables &lt;a href=&quot;http://www.google.com/search?hl=en&amp;amp;&amp;amp;sa=X&amp;amp;ei=BSK0TJfsJ9DWngekmaDGDA&amp;amp;sqi=2&amp;amp;ved=0CCEQBSgA&amp;amp;q=venable+S+plus+conference+washington+exegeses&amp;amp;spell=1&amp;amp;emsg=NCSR&amp;amp;noj=1&amp;amp;ei=BSK0TJfsJ9DWngekmaDGDA&quot; rel=&quot;nofollow&quot;&gt;first link&lt;/a&gt; )&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I think doing sub-sampling doesn't help you (when I say &lt;em&gt;help&lt;/em&gt;, I mean in the context of the number degrees of the freedom of F distribution which is used to test is &lt;em&gt;treatment effect&lt;/em&gt; = 0). By sub-sampling, I mean replication at Block by Treatment level. In RCB, the degrees of freedom wasted in replication can only be used to test if the block effect is present or not. Using Blocks in expt. design is a &quot;trick&quot; to control for the extra variation which is not the main interest. In your case variance because of &lt;em&gt;trays&lt;/em&gt; (block) is not of importance, so I don't know if it is useful to waste extra resources (ie money) in testing if the variance due to trays is significant or not. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I would be afraid to come to the conclusion that there was no block effect. The block effect in your case is random, and the test uses some &quot;strong assumptions&quot; to determine the sampling distribution (which may or may not be true depending on the strong assumptions). If your experiment started with RCB, please retain the structure of the data. &lt;em&gt;Pooling of samples from block may confound your conclusions&lt;/em&gt;. You are again committing the error of accepting $H_o$ if you say there is no &lt;em&gt;block effect&lt;/em&gt; (I am assuming you conclude no block effect by testing hypotheses. right?).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The point 4. further stresses my point 3. of choosing &lt;em&gt;not to subsample&lt;/em&gt;, as essentially subsampling lets you test something (trays effect) which is not of primary interest to you. Even if this was of interest, you invest your money in replication to test something whose sampling distribution is not appropriately determined.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;In the comment you say 12 treatments, but you only mention &lt;em&gt;A&lt;/em&gt; and &lt;em&gt;B&lt;/em&gt; as your treatment variables. Did you mean 12 levels for treatments &lt;em&gt;A&lt;/em&gt; and &lt;em&gt;B&lt;/em&gt; respectively? If yes, then you have only 2 treatments with 12 levels. If no, then you need to change the model.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;For further details about subsampling in RCB, refer to chapter 3 of George Casella's &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0387759646&quot; rel=&quot;nofollow&quot;&gt;design book&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sincere apologies again, if I misunderstood you.&lt;/p&gt;&#xA;" OwnerUserId="1307" LastEditorUserId="1307" LastEditorDisplayName="" LastEditDate="2010-10-12T11:11:42.463" LastActivityDate="2010-10-12T11:11:42.463" CommentCount="7" />
  <row Id="3506" PostTypeId="2" ParentId="3504" CreationDate="2010-10-12T07:39:17.673" Score="5" ViewCount="" Body="&lt;p&gt;Well, it depends if the interaction was your main hypothesis or not. If this the case, then you are encouraged to report the negative result, otherwise you can simply refit your model (without the B and A:B terms) to get a better estimate of A.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, the part of your conclusion that you emphasized doesn't sound correct to me. You can only prove that an observed difference of means is different from 0 (or any other fixed value, according to your alternative hypothesis), you cannot &quot;accept&quot; the null. If your test is non-significant, it simply means that you cannot reject $H_0$. Non-significant results can also reflect lack of power (Type II error).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, rather than simply reporting crude p-values, it would be better (and it actually follows the &lt;a href=&quot;http://apastyle.apa.org/&quot; rel=&quot;nofollow&quot;&gt;APA&lt;/a&gt; recommendations) to also report some kind of effect size or difference of means, together with your inferential results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an example for reporting results from a factorial ANOVA (it has to be rework to fit your specific experimental design since your factors have a lot of levels):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A two-way analysis of variance yielded a main effect for A factor, $F(\nu_1,\nu_2) = 0.00$, $p &amp;lt; .05$, such that the average &quot;value&quot; was significantly higher in the $a_1$ condition (Mean=0.00, SD=0.00) compared to $a_2$ (Mean=0.00, SD=0.00) and $a_3$ (Mean=0.00, SD=0.00, Tukey HSD, all p&amp;lt;0.05). The main effect of B was non-significant ($F(\nu_1,\nu_2) = 0.00$, $p = 0.00$), and no interaction effect was found significant ($F(\nu_1,\nu_2) = 0.00$, $p = 0.00$) indicating that the effect of A on Y was not significantly different across the B levels.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-10-12T07:39:17.673" CommentCount="2" />
  <row Id="3507" PostTypeId="2" ParentId="3497" CreationDate="2010-10-12T07:49:44.903" Score="5" ViewCount="" Body="&lt;p&gt;Hello Joseph,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you can assume that your rows of your data matrix are &lt;strong&gt;exchangeable&lt;/strong&gt; then your modelling strategy should work well. Your method should be fine under the conditions stated by &lt;em&gt;Gaetan Lion&lt;/em&gt; before. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The reason why your method will work&lt;/strong&gt; (&lt;em&gt;given the exchangeability assumption holds&lt;/em&gt;) is that it be taken as a special case of parametric bootstrap in which you take re-sample N rows of big sample, fit a model and store the coefficients and repeat this M times (in traditional bootstrap terminology your &lt;em&gt;M&lt;/em&gt; is equivalent to &lt;em&gt;B&lt;/em&gt;) and take average of the &lt;em&gt;M&lt;/em&gt; coefficient estimates. You can also look at it from a permutation testing view point as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But all these results are true if the (&lt;em&gt;hard to verify&lt;/em&gt;) exchangeability assumption holds. If exchangeability assumption doesn't hold, the answer in that case becomes a bit complicated. Probably you need to take care of the subgroups in your data which are exchangeable and perform your process conditioned on these subgroups. Basically, hierarchical modeling.&lt;/p&gt;&#xA;" OwnerUserId="1307" LastEditorUserId="1307" LastEditorDisplayName="" LastEditDate="2010-10-12T07:57:05.120" LastActivityDate="2010-10-12T07:57:05.120" CommentCount="4" />
  <row Id="3508" PostTypeId="2" ParentId="3489" CreationDate="2010-10-12T08:19:36.063" Score="10" ViewCount="" Body="&lt;p&gt;Just to add to other responses, here is a brief recap' on terminology.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For any biometric or classification system, the main performance indicator is the &lt;a href=&quot;http://en.wikipedia.org/wiki/Receiver_operating_characteristic&quot; rel=&quot;nofollow&quot;&gt;receiver operating characteristic&lt;/a&gt; (ROC) curve, which is a plot of true acceptance rate (TAR=1-FRR, the false rejection rate) against false acceptance rate (FAR), which is computed as the &lt;em&gt;number of false instances classified as positive among all intruder and impostor cases&lt;/em&gt;. The closer the curve is to the top left corner, the better it is (this corresponds to maximizing the so-called area under the curve or AUC). Generally, such curves are generated offline from a database of previous records. In the biometric literature, FAR is sometimes defined such that the &quot;impostor&quot; makes zero effort to obtain a match. Here, I'm roughly quoting &lt;em&gt;&lt;a href=&quot;http://books.google.fr/books?id=fefutm-Dhy0C&amp;amp;printsec=frontcover&amp;amp;dq=biometrics+boulgouris&amp;amp;source=bl&amp;amp;ots=etdSIp3ahh&amp;amp;sig=0D8m0AG45RQSIMmGLU7nFbGo4LE&amp;amp;hl=fr&amp;amp;ei=lmW0TOfwMNTa4AbG_8mgDQ&amp;amp;sa=X&amp;amp;oi=book_result&amp;amp;ct=result&amp;amp;resnum=1&amp;amp;ved=0CBgQ6AEwAA&quot; rel=&quot;nofollow&quot;&gt;Biometrics&lt;/a&gt;&lt;/em&gt;, from Boulgouris et al. (chap. 26). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, you may choose your cutoff by using standard ROC tools (search for &quot;ROC analysis&quot; on &lt;a href=&quot;http://www.rseek.org/&quot; rel=&quot;nofollow&quot;&gt;Rseek&lt;/a&gt;) to find the best compromise between FAR and TAR (this is not necessarily that cutoff that maximizes the AUC, it depends on your objectives).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, as has been highlighted in other responses, this compromise between FAR and TAR led to similar interpretation in psychophysics, classification, or biomedical science. It's just a matter of terminology, and we often speak of Hit rate &lt;em&gt;vs.&lt;/em&gt; False Alarm rate; sensibility &lt;em&gt;vs.&lt;/em&gt; specificty.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are some pictures to complement other responses, which I hope will help you to draw the parallel with decision theory and statistical testing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let an individual be facing a two-alternative choice experiment. Depending on the location of his internal criterion, his response may lead to Hit or False Alarm (response &gt; criterion), or alternatively Correct Rejection or Miss (response &amp;lt; criterion). The corresponding probabilistic response curve resemble your situation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/2yunb.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most classical textbooks on Statistics provide a Table similar to the one below, where we describe the probabilities of incorrectly rejecting a null hypothesis ($\alpha$) vs. falsely “accepting” the null ($\beta$) where in fact the alternative is true.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/Cwhns.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This leads to quite the same picture as with the psychophysical threshold model:&#xA;&lt;img src=&quot;http://i.imgur.com/KRJ3B.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-16T09:07:13.943" LastActivityDate="2010-10-16T09:07:13.943" CommentCount="5" />
  <row Id="3509" PostTypeId="2" ParentId="3496" CreationDate="2010-10-12T08:30:55.770" Score="3" ViewCount="" Body="&lt;p&gt;Previous replies are detailed enough, but an intuitive way of looking why $P (A) $ (ie dinominator in the Bayes theorem) is broken into two cases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is hard to comment about what is the $P(A)$ without any knowledge whether the email is &lt;em&gt;ham or spam&lt;/em&gt;. You are correct that &quot;cheese&quot; appears in spam as well as in ham, but if you look at the probability of appearance of &quot;cheese&quot; given that the email is &lt;em&gt;ham&lt;/em&gt; ($P(A | B)$, $B$ is representative for &lt;em&gt;ham&lt;/em&gt;), you can definitely say a lot of it. At least in my case, I don't receive a lot of spams which contain cheese, therefore in my case $P(A | B)$ will be high (say 90%). Similarly, $P(A | \neg B)$ will be low in my case, as not a lot of spams contain the word cheese. Basically, we try to look at the occurrence of event of interest (here &lt;em&gt;A&lt;/em&gt;) partitioned into two &lt;strong&gt;disjoint&lt;/strong&gt; events, $B$ and $\neg B$. If we partition &lt;em&gt;A&lt;/em&gt; into two separate events, we can say better about the conditional probabilities $P(A | B)$ and $P(A | \neg B)$. In order to get the total probability we also need to weight for the conditional probabilities for the occurance of events on which we condition ie $P(B)$ and $P(\neg B)$. Therefore the final expression  $$P(A) = P(A|B)\cdot P(B)+P(A|\neg B)\cdot P(\neg B)$$&lt;/p&gt;&#xA;" OwnerUserId="1307" LastActivityDate="2010-10-12T08:30:55.770" />
  <row Id="3510" PostTypeId="2" ParentId="3458" CreationDate="2010-10-12T08:56:16.850" Score="4" ViewCount="" Body="&lt;p&gt;It's important to bear in mind that there's no one algorithm that's always better than others. As stated by Wolpert and Macready, &quot;any two algorithms are equivalent when their performance is averaged across all possible problems.&quot; (See &lt;a href=&quot;http://en.wikipedia.org/wiki/No_free_lunch_in_search_and_optimization&quot; rel=&quot;nofollow&quot;&gt;Wikipedia&lt;/a&gt; for details.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a given application, the &quot;best&quot; one is generally one that is most closely aligned to your application in terms of the assumptions it makes, the kinds of data it can handle, the hypotheses it can represent, and so on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So it's a good idea to characterise your data according to criteria such as:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Do I have a very large data set or a modest one?&lt;/li&gt;&#xA;&lt;li&gt;Is the dimensionality high?&lt;/li&gt;&#xA;&lt;li&gt;Are variables numerical (continuous/discrete) or symbolic, or a mix, and/or can they be transformed if necessary?&lt;/li&gt;&#xA;&lt;li&gt;Are variables likely to be largely independent or quite dependent?&lt;/li&gt;&#xA;&lt;li&gt;Are there likely to be redundant, noisy, or irrelevant variables?&lt;/li&gt;&#xA;&lt;li&gt;Do I want to be able to inspect the model generated and try to make sense of it?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;By answering these, you can eliminate some algorithms and identify others as potentially relevant, and then maybe end up with a small set of candidate methods that you have intelligently chosen as likely to be useful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sorry not to give you a simple answer, but I hope this helps nonetheless!&lt;/p&gt;&#xA;" OwnerUserId="1436" LastActivityDate="2010-10-12T08:56:16.850" CommentCount="1" />
  <row Id="3511" PostTypeId="1" AcceptedAnswerId="3513" CreationDate="2010-10-12T09:01:26.290" Score="6" ViewCount="557" Body="&lt;p&gt;I have programmed a logistic regression using the &lt;a href=&quot;http://en.wikipedia.org/wiki/Iteratively_reweighted_least_squares&quot; rel=&quot;nofollow&quot;&gt;IRLS algorithm&lt;/a&gt;. I would like to apply a &lt;a href=&quot;http://en.wikipedia.org/wiki/Least_squares#LASSO_method&quot; rel=&quot;nofollow&quot;&gt;LASSO penalization&lt;/a&gt; in order to automatically select the right features. At each iteration, the following is solved:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\mathbf{\left(X^TWX\right) \delta\hat\beta=X^T\left(y-p\right)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let $\lambda$ be a non-negative real number. I am not penalizing the intercept as suggested in &lt;a href=&quot;http://www-stat.stanford.edu/~tibs/ElemStatLearn/&quot; rel=&quot;nofollow&quot;&gt;&lt;em&gt;The Elements of. Statistical Learning&lt;/em&gt;&lt;/a&gt;. Ditto for the already zero coefficients. Otherwise, I subtract a term from the right-hand side:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\mathbf{X^T\left(y-p\right)-\lambda\times \mathrm{sign}\left(\hat\beta\right)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I am unsure about the modification of the IRLS algorithm. Is it the right way to do?&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Edit: Although I was not confident about it, here is one of the solutions I finally came up with. What is interesting is this solution corresponds to what I now understand about LASSO. There are indeed &lt;strong&gt;two&lt;/strong&gt; steps at each iteration instead of merely one:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;the first step is the same as before : we make an iteration of the algorithm (as if $\lambda=0$ in the formula for the gradient above),&lt;/li&gt;&#xA;&lt;li&gt;the second step is the new one: we apply a soft-thresholding to each component (except for the component $\beta_0$, which corresponds to the intercept) of the vector $\beta$ obtained at the first step. This is called &lt;a href=&quot;http://www.ricam.oeaw.ac.at/conferences/aip2009/minisymposia/slides/aip2009_m19_de_mol_christine.pdf&quot; rel=&quot;nofollow&quot;&gt;Iterative Soft-Thresholding Algorithm&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;$$\forall i \geq 1, \beta_{i}\leftarrow\mathrm{sign}\left(\beta_{i}\right)\times\max\left(0,\,\left|\beta_{i}\right|-\lambda\right)$$&lt;/p&gt;&#xA;" OwnerUserId="1351" LastEditorUserId="1351" LastEditorDisplayName="" LastEditDate="2011-03-19T16:31:40.760" LastActivityDate="2011-03-19T16:31:40.760" Title="How to apply LASSO to IRLS (logistic regression)?" Tags="&lt;logistic-regression&gt;&lt;algorithms&gt;&lt;lasso&gt;&lt;feature-selection&gt;&lt;glm&gt;" AnswerCount="3" CommentCount="1" FavoriteCount="1" />
  <row Id="3512" PostTypeId="2" ParentId="3511" CreationDate="2010-10-12T09:53:32.290" Score="5" ViewCount="" Body="&lt;p&gt;The LASSO loss function has a discontinuity at zero along each axis, so IRLS is going to have problems with it.  I have found a sequential minimal optimisation (SMO) type approach very effective, see e.g.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://bioinformatics.oxfordjournals.org/content/19/17/2246&quot; rel=&quot;nofollow&quot;&gt;http://bioinformatics.oxfordjournals.org/content/19/17/2246&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;a version with MATLAB software is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://bioinformatics.oxfordjournals.org/content/22/19/2348&quot; rel=&quot;nofollow&quot;&gt;http://bioinformatics.oxfordjournals.org/content/22/19/2348&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;the software is available here:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://theoval.cmp.uea.ac.uk/~gcc/cbl/blogreg/&quot; rel=&quot;nofollow&quot;&gt;http://theoval.cmp.uea.ac.uk/~gcc/cbl/blogreg/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Basic idea is to optimise the coefficients one at at time, and test to see if you cross the discontinuity one coefficient at a time, which is easy as you are perfoming a scalar optimisation.  It may sound slow, but it is actually pretty efficient (although I expect better algorithms have been developed since - probably by Keerthi or Chih-Jen Lin who are both leading experts in that sort of thing).&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2010-10-12T09:53:32.290" CommentCount="1" />
  <row Id="3513" PostTypeId="2" ParentId="3511" CreationDate="2010-10-12T10:47:21.650" Score="8" ViewCount="" Body="&lt;p&gt;This  problem is typically solved by  fit by coordinate descent (&lt;a href=&quot;http://www.jstatsoft.org/v33/i01/paper&quot; rel=&quot;nofollow&quot;&gt;see here&lt;/a&gt;). This method is both safer more efficient numerically, algorithmically easier to implement and applicable to a more general array of models (also including Cox regression). An R implementation is available in the &lt;strong&gt;R&lt;/strong&gt; package &lt;strong&gt;&lt;a href=&quot;http://cran.r-project.org/web/packages/glmnet/index.html&quot; rel=&quot;nofollow&quot;&gt;glmnet&lt;/a&gt;&lt;/strong&gt;. The codes are open source (partly in and in C, partly in R), so you can use them as blueprints.&lt;/p&gt;&#xA;" OwnerUserId="603" LastActivityDate="2010-10-12T10:47:21.650" CommentCount="2" />
  <row Id="3514" PostTypeId="1" CreationDate="2010-10-12T12:34:08.707" Score="3" ViewCount="284" Body="&lt;p&gt;While most of the tests for normality help for continous variables, is there a way to test the normality assumptions for binary variables. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;From what i've read on wiki, the K-S test can be applied for continous variables. How do we perform normality tests for binary ( or even categorical variables)?&lt;/p&gt;&#xA;" LastActivityDate="2010-10-12T12:38:31.813" Title="What is the normality test for Binary Data?" Tags="&lt;normality&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="3515" PostTypeId="2" ParentId="3514" CreationDate="2010-10-12T12:38:31.813" Score="7" ViewCount="" Body="&lt;p&gt;There is no such thing as normality of categorical variable. Normal distribution is a continuous distribution so in assumption don't cover categorical output.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-10-12T12:38:31.813" CommentCount="3" />
  <row Id="3516" PostTypeId="1" CreationDate="2010-10-12T13:17:55.790" Score="4" ViewCount="397" Body="&lt;p&gt;I have a binary variable (which takes values 0,1). I have about 100k records of it. How do I determine if it follows the binomial distribution?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(I'm bascially trying to test for normality. And, if the data is not normal, I might have to apply a transformation to get the variable into a binomial distribution.)&lt;/p&gt;&#xA;" LastEditorUserId="1499" LastEditorDisplayName="" LastEditDate="2010-10-12T15:35:00.833" LastActivityDate="2010-10-18T09:29:56.657" Title="Binomial test for a binary variable" Tags="&lt;binomial&gt;&lt;assumptions&gt;" AnswerCount="5" />
  <row Id="3517" PostTypeId="2" ParentId="3516" CreationDate="2010-10-12T13:31:01.233" Score="14" ViewCount="" Body="&lt;p&gt;You cannot determine this through a statistical test, for a trivial reason and a profound reason.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The trivial reason is that your data consist of $k$ ones and $n-k$ zeros with $n$ about 100k.  These data conform extremely closely to a Bernoulli($k/n$) distribution.  No testing is necessary.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The profound reason is that you are implicitly assuming the data are independently random--but they might not be.  If, for instance, they are collected by sampling a process over time, then you might be seeing long strings of $0$ followed by long strings of $1$.  Modeling these as draws from a Bernoulli distribution would likely be a poor choice.  Another possibility is that the values are independent but the probability of a $1$ is varying over time.  (This would be an &quot;overdispersed&quot; Binomial model.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;No&lt;/em&gt; transformation of $ 0, 1 $ will produce a normal distribution!  Perhaps what you are hoping is that some statistic, such as the sample mean, is normally distributed.  The Central Limit Theorem guarantees that, &lt;em&gt;provided&lt;/em&gt; the values are independent and that the probabilities are not tending over time to either $0$ or $1$.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-10-12T13:31:01.233" CommentCount="2" />
  <row Id="3518" PostTypeId="2" ParentId="3516" CreationDate="2010-10-12T13:40:22.543" Score="5" ViewCount="" Body="&lt;p&gt;I completely agree with @whuber -- just wanted to add:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you were to &lt;em&gt;try&lt;/em&gt; to transform the data.  How would you go about doing so?  You would map 0 to some number say, -5 and 1 to some other number?, say 5?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So now instead of having:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;0 0 0 1 0 1 1 0 1 0 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You have:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;-5 -5 -5 5 -5 5 5 -5 5 -5 5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This cannot possible be normally distributed because &lt;strong&gt;you still only have two values!&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each of these entries &lt;em&gt;could however&lt;/em&gt; be Binomial(1,p) just as @whuber described [the same as Bernoulli(p) ], but &lt;strong&gt;not&lt;/strong&gt; Binomial(N,p) because N is never greater than 1 if you only have binary data.&lt;/p&gt;&#xA;" OwnerUserId="1499" LastActivityDate="2010-10-12T13:40:22.543" />
  <row Id="3519" PostTypeId="1" AcceptedAnswerId="3527" CreationDate="2010-10-12T13:58:35.760" Score="7" ViewCount="1077" Body="&lt;p&gt;I started to do Monte Carlo in R as a hobby, but eventually a financial analyst advised to migrate to Matlab.&#xA;I'm an experienced software developer.&#xA;but a Monte Carlo beginner.&#xA;I want to construct static models with sensitivity analysis, later dynamic models.&#xA;Need good libraries/ algorithms that guide me.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To me seems that R has excellent libraries, and I suspect mathlab is preferred by inexperienced programmers because of the easy pascal-like language. The R language is based on scheme and this is hard for beginners, but not for me. If Matlab/ Octave has not advantages on the numerical/ library side I would stick with R.&lt;/p&gt;&#xA;" OwnerUserId="778" LastActivityDate="2011-01-20T21:11:26.137" Title="Is Matlab/octave or R better suited for monte carlo simulation?" Tags="&lt;r&gt;&lt;matlab&gt;&lt;monte-carlo&gt;" AnswerCount="5" CommentCount="4" FavoriteCount="0" />
  <row Id="3520" PostTypeId="1" AcceptedAnswerId="3523" CreationDate="2010-10-12T14:59:04.240" Score="9" ViewCount="235" Body="&lt;p&gt;I see the concept of 'exchangeability' being used in different contexts (e.g., bayesian models) but I have never understood the term very well. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;What does this concept mean?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Under what circumstances is this concept invoked and why?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="1558" LastActivityDate="2010-10-12T15:42:19.113" Title="Can someone explain the concept of 'exchangeability'?" Tags="&lt;bayesian&gt;&lt;basic-concepts&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="3521" PostTypeId="2" ParentId="3519" CreationDate="2010-10-12T15:01:30.577" Score="12" ViewCount="" Body="&lt;p&gt;I use both.  I often prototype functions &amp;amp; algorithms in Matlab because, as stated, it is easier to express an algorithm in something which is close to a pure mathematical language.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;R does have excellent libraries.  I'm still learning it, but I'm starting to leave Matlab in the dust because once you know R, it's also fairly easy to prototype functions there.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I find that if you want algorithms to function efficiently within a production environment, it is best to move to a compiled language like C++.  I have experience wrapping C++ into both Matlab and R (and excel for that matter), but I've had a better experience with R.  &lt;em&gt;Disclaimer:&lt;/em&gt; Being a grad student, I haven't used a recent version of Matlab for my dlls, I've been working almost exclusively in Matlab 7.1 (which is like 4 years old).  Perhaps the newer versions work better, but I can think of two situations off the top of my head where a C++ dll in the back of Matlab caused Windows XP to blue screen because I walked inappropriately outside an array bounds -- a very hard problem to debug if your computer reboots every time you make that mistake...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lastly, the R community appears to be growing much faster and with much more momentum than the Matlab community ever had.  Further, as it's free you also don't have deal with the Godforsaken flexlm license manager.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: Almost all of my development is in MCMC algorithms right now.  I do about 90% in production in C++ with the visualization in R using ggplot2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update for Parallel Comments:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A fair amount of my development time right now is spent on parallelizing MCMC routines (it's my PhD thesis).  I have used Matlab's parallel toolbox and Star P's solution (which I guess is now owned by &lt;a href=&quot;http://www.microsoft.com/pathways/star-p/&quot; rel=&quot;nofollow&quot;&gt;Microsoft??&lt;/a&gt; -- jeez another one is gobbled up...)  I found the parallel toolbox to be a configuration &lt;strong&gt;nightmare&lt;/strong&gt; -- when I used it, it required root access to every single client node.  I think they've fixed that little &quot;bug&quot; now, but still a mess.  I found *'p solution to be elegant, but often difficult to profile.  I have not used &lt;a href=&quot;http://www.accelereyes.com/&quot; rel=&quot;nofollow&quot;&gt;Jacket&lt;/a&gt;, but I've heard good things.  I also have not used the more recent versions of the parallel toolbox which also support GPU computation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have virtually no experience with the R parallel packages.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's been my experience that parallelizing code must occur at the C++ level where you have a finer granularity of control for task decomposition and memory/resource allocation.  I find that if you attempt to parallelize programs at a high level, you often only receive a minimal speedup &lt;em&gt;unless your code is trivially decomposable&lt;/em&gt; (also called dummy-parallelism).  That said, you can even get reasonable speedups using a &lt;strong&gt;single-line&lt;/strong&gt; at the C++ level using OpenMP:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#pragma omp parallel for&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;More complicated schemes have a learning curve, but I really like where gpgpu things are going.  As of JSM this year, the few people I talked to about GPU development in R quote it as being only &quot;toes in the deep end&quot; so to speak.  But as stated, I have minimal experience -- to change in the near future.&lt;/p&gt;&#xA;" OwnerUserId="1499" LastEditorUserId="1499" LastEditorDisplayName="" LastEditDate="2010-10-12T17:30:48.863" LastActivityDate="2010-10-12T17:30:48.863" CommentCount="5" />
  <row Id="3522" PostTypeId="2" ParentId="3516" CreationDate="2010-10-12T15:39:46.737" Score="0" ViewCount="" Body="&lt;p&gt;Hey, thank you folks for clearing this up. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This was an effort as a prelude to cluster analysis. I also understand that normality of variables is more a nice to have feature for cluster analysis and that the distance measures would be valid even otherwise. Your views?&lt;/p&gt;&#xA;" LastActivityDate="2010-10-12T15:39:46.737" CommentCount="3" />
  <row Id="3523" PostTypeId="2" ParentId="3520" CreationDate="2010-10-12T15:42:19.113" Score="13" ViewCount="" Body="&lt;p&gt;Exchangeability is meant to capture symmetry in a problem, symmetry in a sense that does not require independence. Formally, a sequence is exchangeable if its joint probability distribution is a symmetric function of its $n$ arguments.  Intuitively it means we can swap around, or reorder, variables in the sequence without changing their joint distribution.  For example, every IID (independent, identically distributed) sequence is exchangeable - but not the other way around.  Every exchangeable sequence is identically distributed, though. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Imagine a table with a bunch of urns on top, each containing different proportions of red and green balls.  We choose an urn at random (according to some prior distribution), and then take a sample (without replacement) from the selected urn.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that the reds and greens that we observe are NOT independent.  And it is maybe not a surprise to learn that the sequence of reds and greens we observe is an exchangeable sequence.  What &lt;strong&gt;is&lt;/strong&gt; maybe surprising is that EVERY exchangeable sequence can be imagined this way, for a suitable choice of urns and prior distribution. (see Diaconis/Freedman (1980) &quot;Finite Exchangeable Sequences&quot;, Ann. Prob.).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The concept is invoked in all sorts of places, and it is especially useful in Bayesian contexts because in those settings we have a prior distribution (our knowledge of the distribution of urns on the table) and we have a likelihood running around (a model which loosely represents the sampling procedure from a given, fixed, urn).  We observe the sequence of reds and greens (the data) and use that information to update our beliefs about the particular urn in our hand (i.e., our posterior), or more generally, the urns on the table.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Exchangeable random variables are especially wonderful because if we have infinitely many of them then we have tomes of mathematical machinery at our fingertips not the least of which being de Finetti's Theorem; see Wikipedia for an introduction.&lt;/p&gt;&#xA;" OwnerUserId="1108" LastActivityDate="2010-10-12T15:42:19.113" CommentCount="4" />
  <row Id="3524" PostTypeId="2" ParentId="3519" CreationDate="2010-10-12T16:10:12.330" Score="1" ViewCount="" Body="&lt;p&gt;If your simulations will involve relatively sophisticated techniques, then R is the way to go, because it is likely that routines you'll need will be available in R, but not necessarily in matlab.&lt;/p&gt;&#xA;" OwnerUserId="247" LastActivityDate="2010-10-12T16:10:12.330" />
  <row Id="3525" PostTypeId="2" ParentId="3519" CreationDate="2010-10-12T16:44:23.817" Score="6" ViewCount="" Body="&lt;p&gt;Although I almost exclusively use &lt;code&gt;R&lt;/code&gt;, I really admire the profiler in &lt;code&gt;Matlab&lt;/code&gt;.&lt;br&gt;&#xA;When your program is kind of slow you normally want to know where the bottleneck is. Matlab's profiler is a great tool for achieving this as it tells you how much time is spend on each line of the code.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At least to me, using &lt;code&gt;Rprof&lt;/code&gt; is incomparably worse. I can't figure out which call is the bottleneck. Using &lt;code&gt;Rprof&lt;/code&gt; you don't get the information on how much time is spend on each line, but how much time is spend on each primitive function (or so). However, a lot of the same primitive functions are called by a lot of different functions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Although I recommend &lt;code&gt;R&lt;/code&gt; (because it is just great: free, extremely powerful, ...) if you know you have to profile your code a lot, Matlab is way better. And to be fair, there are multicore and parallel computing toolboxes in Matlab (though, extremely pricey). &lt;/p&gt;&#xA;" OwnerUserId="442" LastActivityDate="2010-10-12T16:44:23.817" CommentCount="1" />
  <row Id="3526" PostTypeId="1" CreationDate="2010-10-12T16:53:32.110" Score="2" ViewCount="364" Body="&lt;p&gt;The marascuillo procedure as described &lt;a href=&quot;http://www.itl.nist.gov/div898/handbook/prc/section4/prc474.htm&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; seems to be a test that addresses the issue of multiple comparisons for proportions when you want to test which specific proportions are different from each other after rejecting the null in an overall chi-square test.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I am not very familiar with this test. So, my questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;What nuances (if any) should I worry about when using this test?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I know of at least two other approaches (see below) to address the same issue. Which test is the 'better' approach?:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Performing &quot;partitioned chi square&quot; refereed in this &lt;a href=&quot;http://stats.stackexchange.com/questions/1133/multiple-chi-squared-tests/1210#1210&quot;&gt;answer&lt;/a&gt; by @Brett Magill &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Using a &lt;a href=&quot;http://en.wikipedia.org/wiki/Holm%E2%80%93Bonferroni_method&quot; rel=&quot;nofollow&quot;&gt;Holm–Bonferroni method&lt;/a&gt; to adjust p-values. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" LastActivityDate="2010-10-13T03:14:21.180" Title="Has anyone used the Marascuillo procedure for comparing multiple proportions?" Tags="&lt;multiple-comparisons&gt;&lt;chi-squared&gt;" AnswerCount="2" CommentCount="3" FavoriteCount="1" />
  <row Id="3527" PostTypeId="2" ParentId="3519" CreationDate="2010-10-12T17:22:31.393" Score="10" ViewCount="" Body="&lt;p&gt;To be honest, I think any question you ask around here about R vs ... will be bias towards R. Remember that R is by far the most used &lt;a href=&quot;http://stats.stackexchange.com/tags&quot;&gt;tag&lt;/a&gt;!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What I do&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My current working practice is to use R to prototype and use C when I need an extra boost of speed. It used to be that I would have to switch to C very quickly (again for my particular applications), but the R &lt;a href=&quot;http://www.rforge.net/doc/packages/multicore/multicore.html&quot; rel=&quot;nofollow&quot;&gt;multicore&lt;/a&gt; libraries have helped delay that switch. Essentially, you make a &lt;code&gt;for&lt;/code&gt; loop run in parallel with a trivial change.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I should mention that my applications are &lt;em&gt;very&lt;/em&gt; computationally intensive.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Recommendation&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To be perfectly honest, it really depends on exactly what you want to do. So I'm basing my answer on this statement in your question.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;I want to construct static models&#xA;  with sensitivity analysis, later&#xA;  dynamic models. Need good libraries/&#xA;  algorithms that guide me&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I'd imagine that this problem would be ideally suited to prototyping in R and using C when needed (or some other compiled language). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On saying that, typically Monte-Carlo/sensitivity analysis doesn't involve particularly advanced statistical routines - of course it may needed other advanced functionality. So I think (without more information) that you &lt;em&gt;could&lt;/em&gt; carry out your analysis in any language, but being completely biased, I would recommend R! &lt;/p&gt;&#xA;" OwnerUserId="8" LastActivityDate="2010-10-12T17:22:31.393" CommentCount="1" />
  <row Id="3528" PostTypeId="2" ParentId="1874" CreationDate="2010-10-12T18:05:37.703" Score="2" ViewCount="" Body="&lt;p&gt;I tend to use Gaussian process models for this and similar surface estimation (Possible relevant examples &lt;a href=&quot;http://www.ece.uvic.ca/~btill/papers/learning/Evans_etal_1993.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;http://www.springerlink.com/content/mv8888524v86043g/&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;).  But perhaps your question would be best asked over on &lt;a href=&quot;http://stackoverflow.com/questions/343108/3d-surface-reconstruction-algorithm&quot;&gt;Stack Overflow?&lt;/a&gt;  Could you provide more details on your input data (contours from a surface model of MRI data?) as well as your desired outputs: scale parameter which minimizes L2 distance between two sets of contours? Volume? Distance?  Which distance?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The answer will probably also be specific to the programming language you're working with because, for example, I believe OpenGL has a built in function for determining the minimum distance from a point to the surface (&lt;a href=&quot;http://www.nvidia.com/content/GTC/posters/2010/A15-GPU-Algorithms-for-NURBS-Minimum-Distance-and-Clearance-Computations.pdf&quot; rel=&quot;nofollow&quot;&gt;possible example?&lt;/a&gt;).&lt;/p&gt;&#xA;" OwnerUserId="1499" LastEditorUserId="1499" LastEditorDisplayName="" LastEditDate="2010-10-12T18:11:52.460" LastActivityDate="2010-10-12T18:11:52.460" />
  <row Id="3529" PostTypeId="2" ParentId="3476" CreationDate="2010-10-12T18:59:33.793" Score="5" ViewCount="" Body="&lt;p&gt;ars has the right, and succinct answer. I'll add that when learning how to use matplotlib, I found the &lt;a href=&quot;http://matplotlib.sourceforge.net/gallery.html#&quot; rel=&quot;nofollow&quot;&gt;thumbnail gallery&lt;/a&gt; to be really useful for finding relevant code and examples.&#xA;For your case, I submitted &lt;a href=&quot;http://matplotlib.sourceforge.net/examples/pylab_examples/boxplot_demo2.html&quot; rel=&quot;nofollow&quot;&gt;this boxplot example&lt;/a&gt; that shows you other functionality that could be useful (like rotating the tick mark text, adding upper Y-axis tick marks and labels, adding color to the boxes, etc.) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/I2acK.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="1080" LastActivityDate="2010-10-12T18:59:33.793" CommentCount="1" />
  <row Id="3530" PostTypeId="2" ParentId="298" CreationDate="2010-10-12T18:59:34.423" Score="7" ViewCount="" Body="&lt;p&gt;I always hesitate to jump into a thread with as many excellent responses as this, but it strikes me that few of the answers provide any reason to prefer the logarithm to some other transformation that &quot;squashes&quot; the data, such as a root or reciprocal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Before getting to that, let's &lt;strong&gt;recapitulate&lt;/strong&gt; the wisdom in the existing answers in a more general way.  &lt;em&gt;Some&lt;/em&gt; non-linear re-expression of the dependent variable is indicated when any of the following apply:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;The residuals have a skewed distribution.  The purpose of a transformation is to obtain residuals that are approximately symmetrically distributed (about zero, of course).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The spread of the residuals changes systematically with the values the dependent variable (&quot;heteroscedasticity&quot;).  The purpose of the transformation is to remove that systematic change in spread, achieving approximate &quot;homoscedasticity.&quot;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;To linearize a relationship.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;When scientific theory indicates.  For example, chemistry often suggests expressing concentrations as logarithms (giving activities or even the well-known pH).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;When a more nebulous statistical theory suggests the residuals reflect &quot;random errors&quot; that do not accumulate additively.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;To simplify a model.  For example, sometimes a logarithm can simplify the number and complexity of &quot;interaction&quot; terms.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;(These indications can conflict with one another; in such cases, judgment is needed.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, &lt;strong&gt;when is a &lt;em&gt;logarithm&lt;/em&gt; specifically indicated&lt;/strong&gt; instead of some other transformation?&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;The residuals have a &quot;strongly&quot; positively skewed distribution.  In his book on EDA, John Tukey provides quantitative ways to estimate the transformation (within the family of Box-Cox, or power, transformations) based on rank statistics of the residuals.  It really comes down to the fact that if taking the log symmetrizes the residuals, it was probably the right form of re-expression; otherwise, some other re-expression is needed.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;When the SD of the residuals is directly proportional to the fitted values (and not to some power of the fitted values).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;When the relationship is close to exponential.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;When residuals are believed to reflect multiplicatively accumulating errors.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;You really want a model in which marginal changes in the explanatory variables are interpreted in terms of multiplicative (percentage) changes in the dependent variable.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Finally, some &lt;strong&gt;&lt;em&gt;non&lt;/em&gt; - reasons to use a re-expression&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Making outliers not look like outliers.  An outlier is a datum that does not fit some parsimonious, relatively simple description of the data.  Changing one's description in order to make outliers look better is usually an incorrect reversal of priorities: first obtain a scientifically valid, statistically good description of the data and then explore any outliers.  Don't let the occasional outlier determine how to describe the rest of the data!&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Because the software automatically did it.  (Enough said!)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Because all the data are positive.  (Positivity often implies positive skewness, but it does not have to.  Furthermore, other transformations can work better.  For example, a root often works best with counted data.)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;To make &quot;bad&quot; data (perhaps of low quality) appear well behaved.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;To be able to plot the data.  (If a transformation is needed to be able to plot the data, it's probably needed for one or more good reasons already mentioned.  If the only reason for the transformation truly is for plotting, go ahead and do it--but &lt;em&gt;only&lt;/em&gt; to plot the data.  Leave the data untransformed for analysis.)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-10-12T21:33:05.457" LastActivityDate="2010-10-12T21:33:05.457" CommentCount="1" />
  <row Id="3531" PostTypeId="1" CreationDate="2010-10-12T19:54:05.787" Score="6" ViewCount="183" Body="&lt;p&gt;I would like to perform reversible jump on a network model, but before arriving there, I'm wondering if there are any R packages which support reversible jump for a user specified generalized linear model or spatial-GLM?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Something as simple as an RJMCMC procedure (in R) for the selection of predictors in a logistic regression would be a nice place for me to start?  Does such a function exist?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Through googling, I've only found &lt;a href=&quot;http://cran.r-project.org/web/packages/RJaCGH/index.html&quot; rel=&quot;nofollow&quot;&gt;RJaCGH&lt;/a&gt; which appears to be a bit more complicated (and application specific) than I was hoping for.&lt;/p&gt;&#xA;" OwnerUserId="1499" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-10-12T20:07:43.090" LastActivityDate="2011-02-09T04:39:53.350" Title="Are there any R functions which support Reversible Jump MCMC for a GLM or SGLM?" Tags="&lt;r&gt;&lt;bayesian&gt;&lt;mcmc&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="3532" PostTypeId="1" AcceptedAnswerId="3535" CreationDate="2010-10-12T20:14:49.203" Score="11" ViewCount="152" Body="&lt;p&gt;When programming in R, I've used the &lt;a href=&quot;http://www.rforge.net/doc/packages/multicore/multicore.html&quot; rel=&quot;nofollow&quot;&gt;multicore&lt;/a&gt; package a few times. However, I've never seen a statement about how it handles it's random numbers. When I use openMP with C, I'm careful to use a proper parallel RNG, but with R I've assume that something sensible happens. Can anyone confirm that something sensible does happen?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Example&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the documentation, we have &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x &amp;lt;- foreach(icount(1000), .combine = &quot;+&quot;) %do% rnorm(4)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How are the &lt;code&gt;rnorm&lt;/code&gt;`s generated?&lt;/p&gt;&#xA;" OwnerUserId="8" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-10-14T08:11:42.007" LastActivityDate="2010-10-14T08:11:42.007" Title="Random numbers and the multicore package" Tags="&lt;r&gt;&lt;random-generation&gt;&lt;parallel-computing&gt;&lt;multicore&gt;" AnswerCount="2" FavoriteCount="2" />
  <row Id="3533" PostTypeId="2" ParentId="298" CreationDate="2010-10-12T20:26:52.947" Score="2" ViewCount="" Body="&lt;p&gt;For more on whuber's excellent point about reasons to prefer the logarithm to some other transformations such as a root or reciprocal, but focussing on the unique &lt;em&gt;interpretability&lt;/em&gt; of the regression coefficients resulting from log-transformation compared to other transformations, see:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Oliver N. Keene. The log transformation is special. &lt;em&gt;Statistics in Medicine&lt;/em&gt; 1995; 14(8):811-819. DOI:&lt;a href=&quot;http://dx.doi.org/10.1002/sim.4780140810&quot; rel=&quot;nofollow&quot;&gt;10.1002/sim.4780140810&lt;/a&gt;. (PDF of dubious legality available at &lt;a href=&quot;http://rds.epi-ucsf.org/ticr/syllabus/courses/25/2009/04/21/Lecture/readings/log.pdf&quot; rel=&quot;nofollow&quot;&gt;http://rds.epi-ucsf.org/ticr/syllabus/courses/25/2009/04/21/Lecture/readings/log.pdf&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you log the &lt;em&gt;independent&lt;/em&gt; variable &lt;em&gt;x&lt;/em&gt;  to base &lt;em&gt;b&lt;/em&gt;, you can interpret the regression coefficient (and CI) as the change in the dependent variable &lt;em&gt;y&lt;/em&gt;  per &lt;em&gt;b&lt;/em&gt;-fold increase in &lt;em&gt;x&lt;/em&gt;. (Logs to base 2 are therefore often useful as they correspond to the change in &lt;em&gt;y&lt;/em&gt; per doubling in &lt;em&gt;x&lt;/em&gt;, or logs to base 10 if &lt;em&gt;x&lt;/em&gt; varies over many orders of magnitude, which is rarer). Other transformations, such as square root, have no such simple interpretation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you log the &lt;em&gt;dependent&lt;/em&gt; variable &lt;em&gt;y&lt;/em&gt; (not the original question but one which several of the previous answers have addressed), then I find Stephen Cole's idea of 'sympercents' attractive for presenting the results (i even used them in a paper once), though they don't seem to have caught on all that widely:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tim J Cole. Sympercents: symmetric percentage differences on the 100 log(e) scale simplify the presentation of log transformed data. &lt;em&gt;Statistics in Medicine&lt;/em&gt; 2000; 19(22):3109-3125. DOI:&lt;a href=&quot;http://www3.interscience.wiley.com/journal/75500451/abstract&quot; rel=&quot;nofollow&quot;&gt;10.1002/1097-0258(20001130)19:22&amp;lt;3109::AID-SIM558&gt;3.0.CO;2-F&lt;/a&gt; [I'm so glad &lt;em&gt;Stat Med&lt;/em&gt; stopped using &lt;a href=&quot;http://en.wikipedia.org/wiki/SICI&quot; rel=&quot;nofollow&quot;&gt;SICIs&lt;/a&gt; as DOIs...]&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-10-12T20:26:52.947" />
  <row Id="3534" PostTypeId="2" ParentId="3532" CreationDate="2010-10-12T20:52:14.757" Score="4" ViewCount="" Body="&lt;p&gt;You might want to look at page 5 of this &lt;a href=&quot;http://cran.r-project.org/web/packages/multicore/multicore.pdf&quot; rel=&quot;nofollow&quot;&gt;document&lt;/a&gt; and of this &lt;a href=&quot;http://cran.r-project.org/web/packages/doMC/vignettes/gettingstartedMC.pdf&quot; rel=&quot;nofollow&quot;&gt;document&lt;/a&gt;. By default, under R, each core sets is own seed (i seem to recall using high precision time). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;NB: if you use foreach() from Revolution-computing under windows then i suspect something sensible will &lt;strong&gt;not&lt;/strong&gt; happen. Windows is not POSIX compliant, and this should pose problems when each core needs a different high prec. starting time to set it's seed (unfortunately i don't have windows handy so i can't check this empirically).&lt;/p&gt;&#xA;" OwnerUserId="603" LastEditorUserId="603" LastEditorDisplayName="" LastEditDate="2010-10-13T06:40:54.883" LastActivityDate="2010-10-13T06:40:54.883" CommentCount="1" />
  <row Id="3535" PostTypeId="2" ParentId="3532" CreationDate="2010-10-12T20:54:46.470" Score="5" ViewCount="" Body="&lt;p&gt;I'm not sure how the &lt;code&gt;foreach&lt;/code&gt; works (from the doMC package, I guess), but in multicore if you did something like &lt;code&gt;mclapply&lt;/code&gt; the &lt;code&gt;mc.set.seed&lt;/code&gt; parameter defaults to &lt;code&gt;TRUE&lt;/code&gt; which gives each process a different seed (e.g. &lt;code&gt;mclapply(1:1000, rnorm)&lt;/code&gt;).  I assume your code is translated into something similar, i.e. it boils down to calls to &lt;code&gt;parallel&lt;/code&gt; which has the same convention.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But also see page 16 of the &lt;a href=&quot;http://www.stat.umn.edu/~charlie/parallel/&quot; rel=&quot;nofollow&quot;&gt;slides&lt;/a&gt; by Charlie Geyer, which recommends the &lt;a href=&quot;http://cran.r-project.org/web/packages/rlecuyer/index.html&quot; rel=&quot;nofollow&quot;&gt;rlecuyer&lt;/a&gt; package for parallel independent streams with theoretical guarantees.  Geyer's page also has sample code in R for the different setups.&lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-10-12T20:54:46.470" CommentCount="1" />
  <row Id="3536" PostTypeId="2" ParentId="3526" CreationDate="2010-10-12T20:56:39.273" Score="1" ViewCount="" Body="&lt;p&gt;Just a partial answer because I've never heard of this method. From what I read in the link you provided, it seems to be a single-step procedure (much like Bonferroni, except we rework the test statistics instead of the p-value) which is likely to be too conservative.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In R, there is a function &lt;code&gt;pairwise.prop.test()&lt;/code&gt; which allows any correction for multiple comparisons (single-step or step-down FWER methods or FDR-based), but it is quit what you already suggested (although Bonferroni is by far too conservative, but still very used in practice).&#xA;A resampling approach, using permutation, might be interesting too. The &lt;code&gt;coin&lt;/code&gt; R package provides a well-established testing framework in this respect, see §5 of &lt;a href=&quot;http://cran.r-project.org/web/packages/coin/vignettes/coin_implementation.pdf&quot; rel=&quot;nofollow&quot;&gt;Implementing a Class of Permutation Tests: The coin Package&lt;/a&gt;, but I never had to deal with permutation tests on categorical data in a post-hoc way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;About the analysis of subdivided contingency tables, I generally consider specific associations as a guide to develop additional hypotheses (as for any unplanned comparisons), but this is another question. I generally just use visualization tools, like mosaicplot from &lt;a href=&quot;http://www.math.yorku.ca/SCS/friendly.html#grcat&quot; rel=&quot;nofollow&quot;&gt;Michael Friendly&lt;/a&gt;, Pearson's residuals, and if I seek to explain specific patterns of association I use log-linear models.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-12T21:23:24.407" LastActivityDate="2010-10-12T21:23:24.407" CommentCount="1" />
  <row Id="3537" PostTypeId="1" CreationDate="2010-10-12T22:33:09.020" Score="1" ViewCount="301" Body="&lt;p&gt;I have 3 factors&#xA;factor1 (treatment) with 2 levels (control, stress)&#xA;factor2 (Variates) with 12 Levels (Var1, Var2,....Var12)&#xA;factor3 (Time)  with 12 levels (Week1, Week2,..., Week12)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The treatment has 3 replicates for control and 6 replicates for stress.&#xA;Would it be unbalanced Design?&#xA;What design would you suggest in this case?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;looking forward for your positive response!&#xA;Thanks in advance!&#xA;Regrads,&#xA;Jacki&lt;/p&gt;&#xA;" OwnerUserId="1534" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-01-14T19:30:39.253" LastActivityDate="2011-01-14T19:30:39.253" Title="Unbalanced repeated measure design for the given data?" Tags="&lt;anova&gt;&lt;experiment-design&gt;&lt;split-plot&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="0" />
  <row Id="3538" PostTypeId="2" ParentId="3537" CreationDate="2010-10-12T22:43:53.553" Score="2" ViewCount="" Body="&lt;p&gt;Hello Jacki,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Judging by your description- Yes, your design is unbalanced. But you haven't specified what happens with the factor2? or when you say variates - you mean to say that you include co-variates in the model as in &lt;strong&gt;ANCOVA&lt;/strong&gt;. As you say, repeated measures, I am assuming each of the 3 replicates in control and 6 replicates in stress have 12 measurements across the time points. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is very hard to suggest a design unless you tell us:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;What do you want to infer from the experiement, i-e- parameter(s) of interest.&lt;/li&gt;&#xA;&lt;li&gt;Any major reason for 3 and 6 replicates in the different treatment levels of factor1.&lt;/li&gt;&#xA;&lt;li&gt;Any other experiment specific detail (may or may not be technical). &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Jackie, given the limited details, a wild guess of your design (pictorially) will be: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;&lt;strong&gt;&lt;/em&gt;__&lt;em&gt;_&lt;/em&gt;__&lt;em&gt;_&lt;/em&gt;_&lt;/strong&gt;&lt;em&gt;Control&lt;/em&gt;&lt;strong&gt;&lt;em&gt;_&lt;/em&gt;__&lt;em&gt;_&lt;/em&gt;__&lt;em&gt;_&lt;/em&gt;__&lt;em&gt;_&lt;/em&gt;__&lt;em&gt;_&lt;/em&gt;___&lt;/strong&gt;&lt;em&gt;Stress&lt;/em&gt;&lt;strong&gt;&lt;em&gt;_&lt;/em&gt;__&lt;em&gt;_&lt;/em&gt;__&lt;em&gt;_&lt;/em&gt;____&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;&lt;strong&gt;&lt;/em&gt;__&lt;em&gt;_&lt;/em&gt;___&lt;/strong&gt;&lt;em&gt;V1 V2 ... V12&lt;/em&gt;&lt;strong&gt;&lt;em&gt;_&lt;/em&gt;__&lt;em&gt;&lt;/strong&gt;|&lt;/em&gt;&lt;strong&gt;&lt;em&gt;_&lt;/em&gt;____&lt;/strong&gt;&lt;em&gt;V1 V2 ... V12&lt;/em&gt;&lt;strong&gt;&lt;em&gt;_&lt;/em&gt;__&lt;em&gt;_&lt;/em&gt;__&lt;em&gt;_&lt;/em&gt;_&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;&lt;strong&gt;&lt;/em&gt;___&lt;/strong&gt;_W1 | . | . |&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;&lt;strong&gt;&lt;/em&gt;___&lt;/strong&gt;_W2 | . | . | &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;&lt;strong&gt;&lt;/em&gt;____&lt;/strong&gt;.....  | . | . |&lt;/p&gt;&#xA;&#xA;&lt;p&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;&lt;strong&gt;&lt;/em&gt;___&lt;/strong&gt;_W12| . | . |&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this table Vs represent varieties and Ws represent weeks. I wanted to ask if all the varieties were present in control and stress, respectively. Control groups include plants not treated with NO3 and stress group were treated with NO3. &quot;.&quot; represent the reading of chlorophyll.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the above design is true, then your design is a simple Split Plot Design with Control as the whole plot treatment and you can treat v1, v2 .. v12 as blocks or &quot;biological replicates&quot; (the choice depends on your specific field &lt;em&gt;and&lt;/em&gt; if you want to answer questions related to varieties). And time would be the repeated measure. The correlation between the time points can be modeled using various models, one of the popular choices is AR(1) (SAS also gives you a bunch of options in &lt;code&gt;PROC GLIMIX&lt;/code&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to answer questions related to how treatment (control and stress) interact with time, you can look at the interaction term in the ANOVA analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please let me know if this answers your question? or there are more details need to be added. I would be particularly interested to know the use of Variety variable, specifically, is it just a block or do you actually want to answer questions related to variety of plants.  &lt;/p&gt;&#xA;" OwnerUserId="1307" LastEditorUserId="1307" LastEditorDisplayName="" LastEditDate="2010-10-13T17:37:12.843" LastActivityDate="2010-10-13T17:37:12.843" CommentCount="5" />
  <row Id="3539" PostTypeId="1" AcceptedAnswerId="3546" CreationDate="2010-10-12T22:48:56.690" Score="8" ViewCount="1233" Body="&lt;p&gt;Which inter-rater reliability methods are most appropriate for ordinal or interval data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I believe that &quot;Joint probability of agreement&quot; or &quot;Kappa&quot; are designed for nominal data. Whilst &quot;Pearson&quot; and &quot;Spearman&quot; can be used, they are mainly used for two raters (although they can be used for more than two raters).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What other measures are suitable for ordinal or interval data, i.e. more than two raters?&lt;/p&gt;&#xA;" OwnerUserId="1564" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-21T06:05:00.953" LastActivityDate="2010-10-21T16:27:24.690" Title="Inter-rater reliability for ordinal or interval data" Tags="&lt;psychometrics&gt;&lt;reliability&gt;&lt;inter-rater&gt;" AnswerCount="2" CommentCount="4" FavoriteCount="3" />
  <row Id="3540" PostTypeId="2" ParentId="3539" CreationDate="2010-10-12T23:11:53.597" Score="4" ViewCount="" Body="&lt;p&gt;The &lt;a href=&quot;http://en.wikipedia.org/wiki/Intra-class_correlation_coefficient&quot; rel=&quot;nofollow&quot;&gt;Intraclass correlation&lt;/a&gt; may be used for ordinal data.  But there are some caveats, primarily that the raters cannot be distinguished.  For more on this and how to choose among different versions of the ICC, see:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://tiny.cc/qc9gf&quot; rel=&quot;nofollow&quot;&gt;Intraclass correlations: uses in assessing rater reliability&lt;/a&gt; (Shrout, Fleiss, 1979)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-10-12T23:11:53.597" />
  <row Id="3541" PostTypeId="2" ParentId="3271" CreationDate="2010-10-12T23:32:12.033" Score="0" ViewCount="" Body="&lt;p&gt;Suncoolsu- Thank you!!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One package I have tried out before is STEM. One thing that struck me with that program is that out of ~24k genes it only clustered ~500 of them. The k-means clustering that it uses is clearly not ideal for noisy microarray data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Btw, does anyone know good methods of classifying cause and effect relationships in longitudinal microarray data? &lt;/p&gt;&#xA;" LastActivityDate="2010-10-12T23:32:12.033" CommentCount="1" />
  <row Id="3542" PostTypeId="1" AcceptedAnswerId="3543" CreationDate="2010-10-13T01:57:30.580" Score="25" ViewCount="580" Body="&lt;p&gt;I've seen various theoretical treatments of graphics, such as the &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0387987746&quot; rel=&quot;nofollow&quot;&gt;Grammar of Graphics&lt;/a&gt;. But I have seen nothing equivalent with regards to tables. Over the while I have developed an informal model of good practice in table design. &#xA;However, I'd like to be able to provide a good reference to students.&#xA;The &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1557987912&quot; rel=&quot;nofollow&quot;&gt;APA Style Manual&lt;/a&gt; has a few tips on table design, but it is only a starting point.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Question:&#xA;What is a good resource that provides theoretical and practical advice on the presentation of numeric results in tables?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE:&lt;/strong&gt; It would be particularly useful to have a good free online resource.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: I'm not sure if this should be community wiki. I feel as if there might be a correct answer.&lt;/p&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-11-11T00:44:27.600" LastActivityDate="2011-03-19T12:09:23.263" Title="What is a good resource on table design?" Tags="&lt;tables&gt;" AnswerCount="6" CommentCount="9" FavoriteCount="7" />
  <row Id="3543" PostTypeId="2" ParentId="3542" CreationDate="2010-10-13T02:26:22.177" Score="18" ViewCount="" Body="&lt;p&gt;Ed Tufte has a few pages on this in his classic &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0961392142&quot; rel=&quot;nofollow&quot;&gt;&quot;The Visual Display of Quantitative Information&quot;&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a much more detailed treatment, there is Jane Miller's &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0226526313&quot; rel=&quot;nofollow&quot;&gt;Chicago Guide to Writing about Numbers&lt;/a&gt;. I've never seen anything else like it. It has a whole chapter on &quot;Creating Effective Tables&quot;.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-10-13T02:26:22.177" />
  <row Id="3544" PostTypeId="2" ParentId="3526" CreationDate="2010-10-13T03:14:21.180" Score="0" ViewCount="" Body="&lt;p&gt;I would like to see the Marasculiio procedure used more often. Quite frequently I see people calculating the chi-square on a subset of the main table ie two categories at the time but without actually doing the partitioning correctly. The reason why they do it this way as far as iI understood is that they can't bear grouping the categories as that will make the interpretation really hard. At the end of the day it depends of the audience as well coz if they don't know it they might just recommend the usual Bonferroni approach&lt;/p&gt;&#xA;" OwnerUserId="694" LastActivityDate="2010-10-13T03:14:21.180" CommentCount="1" />
  <row Id="3545" PostTypeId="2" ParentId="3542" CreationDate="2010-10-13T03:34:07.880" Score="11" ViewCount="" Body="&lt;p&gt;Stephen Few's book &lt;a href=&quot;http://www.powells.com/biblio/62-9780970601995-0&quot; rel=&quot;nofollow&quot;&gt;&lt;em&gt;Show Me the Numbers: Designing Tables and Graphs to Enlighten&lt;/em&gt;&lt;/a&gt; has a couple of chapters devoted to tabular display of information.  It's good and recommended, but it's not quite Grammar of Graphics if that's what you're after.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Update&lt;/em&gt; This sounds interesting, but I haven't read it: &lt;a href=&quot;http://www.amazon.com/Handbook-tabular-presentation-design-statistical/dp/B0007DKT4C/&quot; rel=&quot;nofollow&quot;&gt;&lt;em&gt;Handbook of tabular presentation: How to design and edit statistical tables, a style manual and case book&lt;/em&gt;&lt;/a&gt;.  (Curious to hear any comments from someone in the know ..)&lt;/p&gt;&#xA;" OwnerUserId="251" LastEditorUserId="251" LastEditorDisplayName="" LastEditDate="2010-10-13T03:39:35.697" LastActivityDate="2010-10-13T03:39:35.697" />
  <row Id="3546" PostTypeId="2" ParentId="3539" CreationDate="2010-10-13T06:12:24.867" Score="7" ViewCount="" Body="&lt;p&gt;The Kappa ($\kappa$) statistic is a quality index that compares observed agreement between 2 raters on a nominal or ordinal scale with agreement expected by chance alone (as if raters were tossing up). Extensions for the case of multiple raters exist (2, pp. 284–291). In the case of &lt;em&gt;ordinal data&lt;/em&gt;, you can use the &lt;a href=&quot;http://en.wikipedia.org/wiki/Cohen%27s_kappa#Weighted_Kappa&quot; rel=&quot;nofollow&quot;&gt;weighted&lt;/a&gt; $\kappa$, which basically reads as usual $\kappa$ with off-diagonal elements contributing to the measure of agreement. Fleiss (3) provided guidelines to interpret $\kappa$ values but these are merely rules of thumbs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The $\kappa$ statistic is asymptotically equivalent to the ICC estimated from a two-way random effects ANOVA, but significance tests and SE coming from the usual ANOVA framework are not valid anymore with binary data. It is better to use bootstrap to get confidence interval (CI). Fleiss (8) discussed the connection between weighted kappa and the intraclass correlation (ICC).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It should be noted that some psychometricians don't very much like $\kappa$ because it is affected by the prevalence of the object of measurement much like predictive values are affected by the prevalence of the disease under consideration, and this can lead to paradoxical results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Inter-rater reliability for $k$ raters can be estimated with Kendall’s coefficient of concordance, $W$. When the number of items or units that are rated $n &gt; 7$, $k(n − 1)W \sim \chi^2(n − 1)$. (2, pp. 269–270). This asymptotic approximation is valid for moderate value of $n$ and $k$ (6), but with less than 20 items $F$ or permutation tests are more suitable (7). There is a close relationship between Spearman’s $\rho$ and Kendall’s $W$ statistic: $W$ can be directly calculated from the mean of the pairwise Spearman correlations (for untied observations only).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Polychoric_correlation&quot; rel=&quot;nofollow&quot;&gt;Polychoric&lt;/a&gt; (ordinal data) correlation may also be used as a measure of inter-rater agreement. Indeed, they allow to&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;estimate what would be the correlation if ratings were made on a continuous scale,&lt;/li&gt;&#xA;&lt;li&gt;test marginal homogeneity between raters.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In fact, it can be shown that it is a special case of latent trait modeling, which allows to relax distributional assumptions (4).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;About &lt;em&gt;continuous&lt;/em&gt; (or so assumed) measurements, the ICC which quantifies the proportion of variance attributable to the between-subject variation is fine. Again, bootstraped CIs are recommended. As @ars said, there are basically two versions -- agreement and consistency -- that are applicable in the case of agreement studies (5), and that mainly differ on the way sum of squares are computed; the “consistency” ICC is generally estimated without considering the Item×Rater interaction. The ANOVA framework is useful with specific block design where one wants to minimize the number of ratings (&lt;a href=&quot;http://en.wikipedia.org/wiki/Block_design&quot; rel=&quot;nofollow&quot;&gt;BIBD&lt;/a&gt;) -- in fact, this was one of the original motivation of Fleiss's work. It is also the best way to go for &lt;em&gt;multiple raters&lt;/em&gt;. The natural extension of this approach is called the &lt;a href=&quot;http://en.wikipedia.org/wiki/Generalizability_theory&quot; rel=&quot;nofollow&quot;&gt;Generalizability Theory&lt;/a&gt;. A brief overview is given in &lt;a href=&quot;http://www.google.fr/url?sa=t&amp;amp;source=web&amp;amp;cd=17&amp;amp;ved=0CEAQFjAGOAo&amp;amp;url=http%3A%2F%2Flegacy.samsi.info%2F200910%2Fpsycho%2Fpresentations%2FJohnson_raters.pdf&amp;amp;ei=KXu1TKy3IMv44Aaju7SgDQ&amp;amp;usg=AFQjCNGzJnLQqrpaS8x0MiV2YO79w0SYyg&amp;amp;sig2=YgINb6uPkwAwz8eHgLWxng&quot; rel=&quot;nofollow&quot;&gt;Rater Models: An Introduction&lt;/a&gt;, otherwise the standard reference is Brennan's book, reviewed in &lt;a href=&quot;http://www.springerlink.com/content/vu6216157m666366/&quot; rel=&quot;nofollow&quot;&gt;Psychometrika 2006 71(3)&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for general references, I recommend chapter 3 of &lt;em&gt;Statistics in Psychiatry&lt;/em&gt;, from Graham Dunn (Hodder Arnold, 2000). For a more complete treatment of reliability studies, the best reference to date is &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Dunn, G (2004). &lt;em&gt;Design and Analysis of&#xA;  Reliability Studies&lt;/em&gt;. Arnold. See the&#xA;  review in the &lt;a href=&quot;http://ije.oxfordjournals.org/content/34/2/499.1.full&quot; rel=&quot;nofollow&quot;&gt;International Journal&#xA;  of Epidemiology&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;A good online introduction is available on John Uebersax's website, &lt;a href=&quot;http://www.john-uebersax.com/stat/icc.htm&quot; rel=&quot;nofollow&quot;&gt;Intraclass Correlation and Related Methods&lt;/a&gt;; it includes a discussion of the pros and cons of the ICC approach, especially with respect to ordinal scales.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Relevant R packages for two-way assessment (ordinal or continuous measurements) are found in the &lt;a href=&quot;http://cran.r-project.org/web/views/Psychometrics.html&quot; rel=&quot;nofollow&quot;&gt;Psychometrics&lt;/a&gt; Task View; I generally use either the &lt;a href=&quot;http://cran.r-project.org/web/packages/psy/index.html&quot; rel=&quot;nofollow&quot;&gt;psy&lt;/a&gt;, &lt;a href=&quot;http://cran.r-project.org/web/packages/psych/index.html&quot; rel=&quot;nofollow&quot;&gt;psych&lt;/a&gt;, or &lt;a href=&quot;http://cran.r-project.org/web/packages/irr/index.html&quot; rel=&quot;nofollow&quot;&gt;irr&lt;/a&gt; packages. There's also the &lt;a href=&quot;http://cran.r-project.org/web/packages/concord/index.html&quot; rel=&quot;nofollow&quot;&gt;concord&lt;/a&gt; package but I never used it. For dealing with more than two raters, the &lt;a href=&quot;http://cran.r-project.org/web/packages/lme4/index.html&quot; rel=&quot;nofollow&quot;&gt;lme4&lt;/a&gt; package is the way to go for it allows to easily incorporate random effects, but most of the reliability designs can be analysed using the &lt;code&gt;aov()&lt;/code&gt; because we only need to estimate variance components.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;References&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;J Cohen. Weighted kappa: Nominal scale agreement with provision for scales disagreement of partial credit. &lt;em&gt;Psychological Bulletin&lt;/em&gt;, &lt;em&gt;70&lt;/em&gt;, 213–220, 1968.&lt;/li&gt;&#xA;&lt;li&gt;S Siegel and Jr N John Castellan. &lt;em&gt;Nonparametric Statistics for the Behavioral&#xA;Sciences&lt;/em&gt;. McGraw-Hill, Second edition, 1988.&lt;/li&gt;&#xA;&lt;li&gt;J L Fleiss. &lt;em&gt;Statistical Methods for Rates and Proportions&lt;/em&gt;. New York: Wiley, Second&#xA;edition, 1981.&lt;/li&gt;&#xA;&lt;li&gt;J S Uebersax. &lt;em&gt;The tetrachoric and polychoric correlation coefficients&lt;/em&gt;. Statistical Methods for Rater Agreement web site, 2006. Available at: &lt;a href=&quot;http://john-uebersax.com/stat/tetra.htm&quot; rel=&quot;nofollow&quot;&gt;http://john-uebersax.com/stat/tetra.htm&lt;/a&gt;. Accessed February 24, 2010.&lt;/li&gt;&#xA;&lt;li&gt;P E Shrout and J L Fleiss. &lt;a href=&quot;http://www3.uta.edu/faculty/ricard/COED/Shrout%20&amp;amp;%20Fleiss%20%281979%29%20Intraclass%20correlations.pdf&quot; rel=&quot;nofollow&quot;&gt;Intraclass correlation: Uses in assessing rater reliability&lt;/a&gt;. &lt;em&gt;Psychological Bulletin&lt;/em&gt;, &lt;em&gt;86&lt;/em&gt;, 420–428, 1979.&lt;/li&gt;&#xA;&lt;li&gt;M G Kendall and B Babington Smith. &lt;a href=&quot;http://projecteuclid.org/DPubS?service=UI&amp;amp;version=1.0&amp;amp;verb=Display&amp;amp;handle=euclid.aoms/1177732186&quot; rel=&quot;nofollow&quot;&gt;The problem of m rankings&lt;/a&gt;. &lt;em&gt;Annals of Mathematical Statistics&lt;/em&gt;, &lt;em&gt;10&lt;/em&gt;, 275–287, 1939.&lt;/li&gt;&#xA;&lt;li&gt;P Legendre. &lt;a href=&quot;http://www.bio.umontreal.ca/legendre/reprints/Coefficient_of_concordance.pdf&quot; rel=&quot;nofollow&quot;&gt;Coefficient of concordance&lt;/a&gt;. In N J Salkind, editor, &lt;em&gt;Encyclopedia of Research Design&lt;/em&gt;. SAGE Publications, 2010.&lt;/li&gt;&#xA;&lt;li&gt;J L Fleiss. &lt;a href=&quot;http://www.wpic.pitt.edu/research/biometrics/Publications/Biometrics%20Archives%20PDF/456-Fleiss&amp;amp;Cohen19730001.pdf&quot; rel=&quot;nofollow&quot;&gt;The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability&lt;/a&gt;. &lt;em&gt;Educational and Psychological Measurement&lt;/em&gt;, &lt;em&gt;33&lt;/em&gt;, 613-619, 1973.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-21T16:27:24.690" LastActivityDate="2010-10-21T16:27:24.690" CommentCount="3" />
  <row Id="3547" PostTypeId="2" ParentId="3463" CreationDate="2010-10-13T06:37:12.110" Score="2" ViewCount="" Body="&lt;p&gt;How do you define correlation for non stationary time series? Do you plan to take the correlation of the diff or these time series?&#xA;If not, I suggest you look for cointegration rather than correlation (cf Granger etc...)&lt;/p&gt;&#xA;" OwnerUserId="1709" LastActivityDate="2010-10-13T06:37:12.110" />
  <row Id="3548" PostTypeId="2" ParentId="555" CreationDate="2010-10-13T08:53:32.477" Score="5" ViewCount="" Body="&lt;p&gt;ANOVA can be used with categorical explanatory variables (factors) that take more than 2 values (levels), and gives a basic test that the mean response is the same for every value. This avoids the regression problem on carrying multiple pairwise t-tests between those levels:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Multiple t-tests on a fixed 5% significance level, would make roughly 5% of them give wrong results. &lt;/li&gt;&#xA;&lt;li&gt;These tests are not independed from each other. Comparing A's levels with B's is connected with comparing A's to C's, as A's data are used in both tests.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;It is better to use &lt;a href=&quot;http://en.wikipedia.org/wiki/Contrast_%28statistics%29&quot; rel=&quot;nofollow&quot;&gt;contrasts&lt;/a&gt; for different combinations on the factor levels you want to test.&lt;/p&gt;&#xA;" OwnerUserId="1077" LastEditorUserId="1077" LastEditorDisplayName="" LastEditDate="2010-10-29T13:26:46.320" LastActivityDate="2010-10-29T13:26:46.320" />
  <row Id="3549" PostTypeId="1" AcceptedAnswerId="3555" CreationDate="2010-10-13T09:40:17.420" Score="16" ViewCount="484" Body="&lt;p&gt;In a multiple linear regression, why is it possible to have a highly significant F statistic (p&amp;lt;.001) but have very high p-values on all the regressor's t tests?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my model, there are 10 regressors. One has a p-value of 0.1 and the rest are above 0.9&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;For dealing with this problem see the &lt;a href=&quot;http://stats.stackexchange.com/questions/3561/dealing-with-correlated-regressors&quot;&gt;follow-up question&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1077" LastEditorUserId="1077" LastEditorDisplayName="" LastEditDate="2010-10-15T12:27:40.230" LastActivityDate="2010-10-15T12:27:40.230" Title="F and t statistics in a regression" Tags="&lt;hypothesis-testing&gt;&lt;regression&gt;&lt;t-test&gt;&lt;multicollinearity&gt;" AnswerCount="3" CommentCount="0" FavoriteCount="3" />
  <row Id="3550" PostTypeId="2" ParentId="1610" CreationDate="2010-10-13T10:15:00.153" Score="2" ViewCount="" Body="&lt;p&gt;(a bit joke answer I invented just a minute ago)&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;A first class person thinks he is always right.&lt;/li&gt;&#xA;&lt;li&gt;A second class person thinks he is always wrong.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The first class person can only make a type I error (because sometimes he will be wrong).&lt;/li&gt;&#xA;&lt;li&gt;The second class person can only make a type II error (because sometimes he will be right).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="1219" LastActivityDate="2010-10-13T10:15:00.153" />
  <row Id="3551" PostTypeId="2" ParentId="3516" CreationDate="2010-10-13T11:07:31.753" Score="0" ViewCount="" Body="&lt;p&gt;ALL binary variables have the binomial distribution, provided that the probability of success (probability to observe 1) does not change and that all their instances are independent. A rule of thumb says that binomial distribution can be fairly approximated by normal distribution when n*p&gt;30, with n=number of instances, p=probability of success.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, I argue that your question is about testing for independence and constant success rate.&#xA;For the former, you can use Bradley run test &lt;a href=&quot;http://www.itl.nist.gov/div898/handbook/eda/section3/eda35d.htm&quot; rel=&quot;nofollow&quot;&gt;http://www.itl.nist.gov/div898/handbook/eda/section3/eda35d.htm&lt;/a&gt; (I suppose it is also known by another name). For the latter, I have only a rough answer: you can split your sample in k subgroups and then build a test using the k proportions of success in the subgroups.  &lt;/p&gt;&#xA;" OwnerUserId="1219" LastActivityDate="2010-10-13T11:07:31.753" />
  <row Id="3552" PostTypeId="1" CreationDate="2010-10-13T11:31:21.963" Score="3" ViewCount="169" Body="&lt;p&gt;Hello,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've got a model that I've developed in R, but also need to express in SAS. It's a double GLM, that is, I fit both the mean and (log-)variance as linear combinations of the predictors:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$E(Y) = X_1'b_1$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\log V(Y) = X_2'b_2$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where Y has a normal distribution, $X_1$ and $X_2$ are the vectors of independent variables, and $b_1$ and $b_2$ are the coefficients to be estimated. $X_1$ and $X_2$ can be the same, but need not be.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can fit this in R using gls() and the varComb and varIdent functions. I've also written a custom function that maximises the likelihood using optim/nlminb, and verified that it returns the same output as gls.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would now like to translate this into SAS. I know that I can use PROC MIXED:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;proc mixed;&#xA;  class x2;&#xA;  model y = x1;&#xA;  repeated /group = x2;&#xA;run;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, this only gives me what I want if I have 1 variable in the /GROUP option. If I enter 2 or more variables, MIXED can only handle this by treating each individual combination of levels as a distinct group (that is, it takes the cartesian product). For example, if I have 2 variables in $X_2$, with 3 and 4 levels respectively, MIXED will fit 12 parameters for the variance. What I want is for the log-variance to be additive in the variables specified, ie 6 parameters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a way of doing this in MIXED or any other proc? I could probably code something in NLP, but I'd really prefer not to.&lt;/p&gt;&#xA;" OwnerUserId="1569" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-10-13T12:34:00.090" LastActivityDate="2010-10-14T03:36:05.547" Title="Replicating R model in SAS" Tags="&lt;r&gt;&lt;sas&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="3553" PostTypeId="2" ParentId="3549" CreationDate="2010-10-13T11:45:32.477" Score="15" ViewCount="" Body="&lt;p&gt;This happens when the predictors are highly correlated. Imagine a situation where there are only two predictors with very high correlation. Individually, they both also correlate closely with the response variable. Consequently, the F-test has a low p-value (it is saying that the predictors together are highly significant in explaining the variation in the response variable). But the t-test for each predictor has a high p-value because after allowing for the effect of the other predictor there is not much left to explain.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-10-13T11:45:32.477" CommentCount="0" />
  <row Id="3554" PostTypeId="2" ParentId="3352" CreationDate="2010-10-13T11:46:32.707" Score="0" ViewCount="" Body="&lt;p&gt;If $x_1=0$, then $Y$~$(\beta_0,\sigma^2)$ in model 1 and $Y$~$(0,\sigma^2)$ in model 2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If $x_1=1$, then $Y$~$(\beta_0+\beta_1,\sigma^2)$ in model 1 and $Y$~$(\beta_1,\sigma^2)$ in model 2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Look for example at the first line: is $\beta_0$ a random variable or is a zero constant?&lt;/p&gt;&#xA;" OwnerUserId="1219" LastActivityDate="2010-10-13T11:46:32.707" />
  <row Id="3555" PostTypeId="2" ParentId="3549" CreationDate="2010-10-13T12:29:11.170" Score="16" ViewCount="" Body="&lt;p&gt;As Rob mentions, this occurs when you have highly correlated variables. The standard example I use is predicting weight from shoe size. You can predict weight equally well with the right or left shoe size. But together it doesn't work out.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Brief simulation example&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;RSS = 3:10 #Right shoe size&#xA;LSS = rnorm(RSS, RSS, 0.1) #Left shoe size - similar to RSS&#xA;cor(LSS, RSS) #correlation ~ 0.99&#xA;&#xA;weights = 120 + rnorm(RSS, 10*RSS, 10)&#xA;&#xA;##Fit a joint model&#xA;m = lm(weights ~ LSS + RSS)&#xA;&#xA;##F-value is very small, but neither LSS or RSS are significant&#xA;summary(m)&#xA;&#xA;##Fitting RSS or LSS separately gives a significant result. &#xA;summary(lm(weights ~ LSS))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="8" LastActivityDate="2010-10-13T12:29:11.170" CommentCount="2" />
  <row Id="3556" PostTypeId="1" CreationDate="2010-10-13T14:22:54.863" Score="5" ViewCount="277" Body="&lt;p&gt;Can anyone report on their experience with an adaptive kernel density estimator?&lt;br&gt;&#xA;(There are many synonyms:&#xA;adaptive | variable | variable-width, KDE | histogram | interpolator ...)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Variable_kernel_density_estimation&quot; rel=&quot;nofollow&quot;&gt;Variable kernel density estimation&lt;/a&gt;&#xA;says &quot;we vary the width of the kernel in different regions of the sample space.&#xA;There are two methods ...&quot;&#xA;actually, more:&#xA;neighbors within some radius,&#xA;KNN nearest neighbors (K usually fixed),&#xA;Kd trees, multigrid...&lt;br&gt;&#xA;Of course no single method can do everything,&#xA;but adaptive methods look attractive.&lt;br&gt;&#xA;See for example the nice picture of an adaptive 2d mesh in&#xA;&lt;a href=&quot;http://en.wikipedia.org/wiki/Finite_element_method&quot; rel=&quot;nofollow&quot;&gt;Finite element method&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd like to hear what worked / what didn't work for real data,&#xA;especially &gt;= 100k scattered data points in 2d or 3d.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Added 2 Nov: here's a plot of a &quot;clumpy&quot; density (piecewise x^2 * y^2),&#xA;a nearest-neighbor estimate, and Gaussian KDE with Scott's factor.&#xA;While one (1) example doesn't prove anything,&#xA;it does show that NN can fit sharp hills reasonably well&#xA;(and, using KD trees, is fast in 2d, 3d ...)&#xA;&lt;img src=&quot;http://i.imgur.com/ulkUd.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="557" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-01-17T23:59:29.293" LastActivityDate="2011-01-17T23:59:29.293" Title="Adaptive kernel density estimators?" Tags="&lt;kde&gt;&lt;k-nearest-neighbour&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="2" />
  <row Id="3557" PostTypeId="2" ParentId="3549" CreationDate="2010-10-13T14:38:26.963" Score="2" ViewCount="" Body="&lt;p&gt;A keyword to search for would be &quot;collinearity&quot; or &quot;multicollinearity&quot;. This can be detected using diagnostics like &lt;a href=&quot;http://en.wikipedia.org/wiki/Variance_inflation_factor&quot; rel=&quot;nofollow&quot;&gt;Variance Inflation Factors&lt;/a&gt; (VIFs) or methods as described inthe textbook &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0471691178&quot; rel=&quot;nofollow&quot;&gt;&quot;Regression Diagnostics: Identifying Influential Data and Sources of Collinearity&quot;&lt;/a&gt; by Belsley, Kuh and Welsch. VIFs are much easier to understand, but they can't deal with collinearity involving the intercept (i.e., predictors that are almost constant by themselves or in a linear combination) - conversely, the BKW diagnostics are far less intuitive but &lt;em&gt;can&lt;/em&gt; deal with collinearity involving the intercept.&lt;/p&gt;&#xA;" OwnerUserId="1352" LastActivityDate="2010-10-13T14:38:26.963" />
  <row Id="3558" PostTypeId="2" ParentId="1337" CreationDate="2010-10-13T15:07:55.360" Score="1" ViewCount="" Body="&lt;p&gt;This one's from the xkcd forums:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Three statisticians are out hunting.&#xA;  Bird flies up out of the bush, and the&#xA;  first statistician aims and fires.&#xA;  Unfortunately for them, he missed, the&#xA;  bullet going about a foot below the&#xA;  bird. The second one fires, but the&#xA;  bullet goes about a foot above the&#xA;  bird.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;The third statistician puts down his&#xA;  gun and says, &quot;All right! We got&#xA;  him!...&quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="144" LastActivityDate="2010-10-13T15:07:55.360" CommunityOwnedDate="2010-10-13T15:07:55.360" CommentCount="3" />
  <row Id="3559" PostTypeId="1" AcceptedAnswerId="3562" CreationDate="2010-10-13T16:12:57.630" Score="7" ViewCount="2037" Body="&lt;p&gt;I have a &lt;code&gt;SPSS&lt;/code&gt; Output for a logistic regression. This output reports two measure for the model fit, &lt;code&gt;Cox &amp;amp; Snell&lt;/code&gt; and &lt;code&gt;Nagelkerke&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So as a rule of thumb, which of these R² measures would you report as the model fit?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or, which of these fit indices is the one that is usually reported in journals?&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Some Background: The regression tries to predict the presence or absence of some bird (capercaillie) from some environmental variables (e.g., steepness, vegetation cover, ...). Unfortunately, the bird did not appear very often (35 hits to 468 misses) so the regression performs rather poorly. Cox &amp;amp; Snell is .09, Nagelkerke, .23.&lt;br&gt;&#xA;The subject is environmental sciences or ecology.&lt;/p&gt;&#xA;" OwnerUserId="442" LastActivityDate="2010-10-14T18:14:21.470" Title="Logistic Regression: Which pseudo R-squared measure is the one to report (Cox &amp; Snell or Nagelkerke)?" Tags="&lt;logistic-regression&gt;&lt;goodness-of-fit&gt;" AnswerCount="5" CommentCount="1" FavoriteCount="1" />
  <row Id="3560" PostTypeId="2" ParentId="3559" CreationDate="2010-10-13T16:36:53.413" Score="2" ViewCount="" Body="&lt;p&gt;I would prefer the Nagelkerke as this model fit attains 1 when the model fits perfectly giving the reader a sense of how far your model is from perfect fit. The Cox &amp;amp; Shell does not attain 1 for perfect model fit and hence interpreting a value of 0.09 is a bit harder. See this url for further info on &lt;a href=&quot;http://www.ats.ucla.edu/stat/mult_pkg/faq/general/Psuedo_RSquareds.htm&quot; rel=&quot;nofollow&quot;&gt;Pseudo RSquared&lt;/a&gt; for an explanation of various types of fits. &lt;/p&gt;&#xA;" LastActivityDate="2010-10-13T16:36:53.413" CommentCount="6" />
  <row Id="3561" PostTypeId="1" AcceptedAnswerId="3569" CreationDate="2010-10-13T17:34:00.057" Score="10" ViewCount="498" Body="&lt;p&gt;In a multiple linear regression with  highly correlated regressors, what is the best strategy to use? Is it a legitimate approach to add the product of all the correlated regressors?&lt;/p&gt;&#xA;" OwnerUserId="1077" LastEditorUserId="1352" LastEditorDisplayName="" LastEditDate="2010-10-13T19:36:47.703" LastActivityDate="2010-10-14T07:24:54.397" Title="Dealing with correlated regressors" Tags="&lt;regression&gt;&lt;multicollinearity&gt;" AnswerCount="6" CommentCount="2" FavoriteCount="2" />
  <row Id="3562" PostTypeId="2" ParentId="3559" CreationDate="2010-10-13T17:46:39.857" Score="13" ViewCount="" Body="&lt;p&gt;Normally I wouldn't report $R^2$ at all.  Hosmer and Lemeshow, in their textbook &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0471356328&quot; rel=&quot;nofollow&quot;&gt;&lt;em&gt;Applied Logistic Regression&lt;/em&gt;&lt;/a&gt; (2nd Ed.), explain why:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In general, [$R^2$ measures] are based on various comparisons of the predicted values from the fitted model to those from [the base model], the no data or intercept only model and, as a result, do not assess goodness-of-fit.  We think that a true measure of fit is one based strictly on a comparison of observed to predicted values from the fitted model.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;[At p. 164.]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Concerning various ML versions of $R^2$, the &quot;pseudo $R^2$&quot; stat, they mention that it is not &quot;recommended for routine use, as it is not as intuitively easy to explain,&quot; but they feel obliged to describe it because various software packages report it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;They conclude this discussion by writing,&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;...low $R^2$ values in logistic regression are the norm and this presents a problem when reporting their values to an audience accustomed to seeing linear regression values. ... Thus [arguing by reference to running examples in the text] we do not recommend routine publishing of $R^2$ values with results from fitted logistic models.  However, they may be helpful in the model building state as a statistic to evaluate competing models.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;[At p. 167.]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My experience with some large logistic models (100k to 300k records, 100 - 300 explanatory variables) has been exactly as H &amp;amp; L describe.  I could achieve relatively high $R^2$ with my data, up to about 0.40.  These corresponded to classification error rates between 3% and 15% (false negatives and false positives, balanced, as confirmed using 50% hold-out datasets).  As H &amp;amp; L hinted, I had to spend a lot of time disabusing the client (a sophisticated consultant himself, who was familiar with $R^2$) concerning $R^2$ and getting him to focus on what mattered in the analysis (the classification error rates).  I can warmly recommend describing the results of your analysis without reference to $R^2$, which is more likely to mislead than not.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-10-13T17:46:39.857" CommentCount="3" />
  <row Id="3563" PostTypeId="2" ParentId="3561" CreationDate="2010-10-13T17:48:25.090" Score="1" ViewCount="" Body="&lt;p&gt;I'm no expert on this, but my first thought would be to run a principal component analysis on the predictor variables, then use the resulting principal components to predict your dependent variable.&lt;/p&gt;&#xA;" OwnerUserId="364" LastEditorUserId="364" LastEditorDisplayName="" LastEditDate="2010-10-13T18:00:55.567" LastActivityDate="2010-10-13T18:00:55.567" CommentCount="4" />
  <row Id="3564" PostTypeId="1" AcceptedAnswerId="3581" CreationDate="2010-10-13T17:48:45.233" Score="3" ViewCount="238" Body="&lt;p&gt;I would like to apply KDE to inventory replenishment, but I am not sure how to use the analysis to predict future sales based on past sales. Given a set of data and having applied KDE to it (probably using a Gaussian distribution), how do I make a prediction about the future? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for any help! Please let me know if I can clarify - I'm only starting to pick up the language for talking about KDE ... I'm glad to do reading on my own - pointers to any resources would be welcome.&lt;/p&gt;&#xA;" OwnerUserId="1574" LastEditorUserId="1574" LastEditorDisplayName="" LastEditDate="2010-10-13T22:24:24.733" LastActivityDate="2010-10-13T22:49:26.513" Title="How to use Kernel Density Estimation for Prediction?" Tags="&lt;time-series&gt;&lt;forecasting&gt;&lt;smoothing&gt;&lt;kde&gt;&lt;kernel&gt;" AnswerCount="2" CommentCount="3" FavoriteCount="1" />
  <row Id="3565" PostTypeId="2" ParentId="3561" CreationDate="2010-10-13T17:48:53.860" Score="1" ViewCount="" Body="&lt;p&gt;Hello,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of the ways to &lt;em&gt;reduce the effects of correlation&lt;/em&gt; is to &lt;strong&gt;standardize the regressors&lt;/strong&gt;. In standardizing, all the regressors are subtracted by their respective means and divided by their respective standard deviations. Specifically, if $X$ is the regression matrix:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$x_{ij}^{standardized}=\frac {x_{ij}-\overline{x_{.j}}} {s_{j}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is not a remedy, but definitely a step in the right direction.&lt;/p&gt;&#xA;" OwnerUserId="1307" LastEditorUserId="1307" LastEditorDisplayName="" LastEditDate="2010-10-13T18:09:04.173" LastActivityDate="2010-10-13T18:09:04.173" CommentCount="9" />
  <row Id="3566" PostTypeId="2" ParentId="3559" CreationDate="2010-10-13T18:02:06.783" Score="5" ViewCount="" Body="&lt;p&gt;Both indices are measures of strength of association (i.e. whether any predictor is associated with the outcome, as for an LR test), and can be used to quantify predictive ability or model performance. A single predictor may have a significant effect on the outcome but it might not necessarily be so useful for &lt;em&gt;predicting individual response&lt;/em&gt;, hence the need to assess model performance as a whole (wrt. the null model). The Nagelkerke $R^2$ is useful because it has a maximum value of 1.0, as Srikant said. This is just a normalized version of the $R^2$ computed from the likelihood ratio, $R^2_{\text{LR}}=1-\exp(-\text{LR}/n)$, which has connection with the Wald statistic for overall association, as originally proposed by Cox and Snell.&#xA;Other indices of predictive ability are Brier score, the C index (concordance probability or ROC area), or Somers' D, the latter two providing a better measure of predictive discrimination.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The only assumptions made in logistic regression are that of &lt;em&gt;linearity&lt;/em&gt; and &lt;em&gt;additivity&lt;/em&gt; (+ independence). Although many global goodness-of-fit tests (like the Hosmer &amp;amp; Lemeshow $\chi^2$ test, but see my &lt;a href=&quot;http://stats.stackexchange.com/questions/3559/logistic-regression-which-pseudo-r-squared-measure-is-the-one-to-report-cox-s/3574#3574&quot;&gt;comment&lt;/a&gt; to @onestop) have been proposed, they generally lack power. For assessing model fit, it is better to rely on visual criteria (stratified estimates, nonparametric smoothing) that help to spot local or global departure between predicted and observed outcomes (e.g. non-linearity or interaction), and this is largely detailed in Harrell's &lt;a href=&quot;http://biostat.mc.vanderbilt.edu/twiki/bin/view/Main/RmS&quot; rel=&quot;nofollow&quot;&gt;RMS handout&lt;/a&gt;. On a related subject (calibration tests), Steyerberg (&lt;a href=&quot;http://www.clinicalpredictionmodels.org/&quot; rel=&quot;nofollow&quot;&gt;&lt;em&gt;Clinical Prediction Models&lt;/em&gt;&lt;/a&gt;, 2009) points to the same approach for assessing the agreement between observed outcomes and predicted probabilities:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Calibration is related to&#xA;  goodness-of-fit, which relates to the&#xA;  ability of a model to fit a given set&#xA;  of data. Typically, there is no single&#xA;  goodness-of-fit test that has good&#xA;  power against all kinds of lack of fit&#xA;  of a prediction model. Examples of&#xA;  lack of fit are missed&#xA;  non-linearities, interactions, or an&#xA;  inappropriate link function between&#xA;  the linear predictor and the outcome.&#xA;  Goodness-of-fit can be tested with a&#xA;  $\chi^2$ statistic. (p. 274)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;He also suggests to rely on the absolute difference between smoothed observed outcomes and predicted probabilities either visually, or with the so-called Harrell's E statistic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More details can be found in Harrell's book, &lt;em&gt;Regression Modeling Strategies&lt;/em&gt; (pp. 203-205, 230-244, 247-249). For a more recent discussion, see also&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Steyerberg, EW, Vickers, AJ, Cook, NR, Gerds, T, Gonen, M, Obuchowski, N, Pencina, MJ, and Kattan, MW (2010). &lt;a href=&quot;http://www.epibiostat.ucsf.edu/courses/RoadmapK12/SIGS/Steyerberg_Epidemiol_2010.pdf&quot; rel=&quot;nofollow&quot;&gt;Assessing the Performance of Prediction Models, A Framework for Traditional and Novel Measures&lt;/a&gt;. &lt;em&gt;Epidemiology&lt;/em&gt;, &lt;em&gt;21(1)&lt;/em&gt;, 128-138.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-14T18:14:21.470" LastActivityDate="2010-10-14T18:14:21.470" CommentCount="3" />
  <row Id="3567" PostTypeId="2" ParentId="3561" CreationDate="2010-10-13T18:12:33.317" Score="8" ViewCount="" Body="&lt;p&gt;You can use principal components or ridge regression to deal with this problem.  On the other hand, if you have two variables that are correlated highly enough to cause problems with parameter estimation, then you could almost certainly drop either one of the two without losing much in terms of prediction--because the two variables carry the same information.  Of course, that only works when the problem is due to &lt;em&gt;two&lt;/em&gt; highly correlated independents.  When the problem involves more than two variables that are together nearly collinear (any two of which may have only moderate correlations), you'll probably need one of the other methods.&lt;/p&gt;&#xA;" OwnerUserId="485" LastEditorUserId="485" LastEditorDisplayName="" LastEditDate="2010-10-13T19:44:22.127" LastActivityDate="2010-10-13T19:44:22.127" CommentCount="1" />
  <row Id="3568" PostTypeId="2" ParentId="1610" CreationDate="2010-10-13T18:43:21.297" Score="3" ViewCount="" Body="&lt;p&gt;I'll try not to be redundant with other responses (although it seems a little bit what J. M. already suggested), but I generally like showing the following two pictures:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/9TEBZ.jpg&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/YBAL1.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-10-13T18:43:21.297" />
  <row Id="3569" PostTypeId="2" ParentId="3561" CreationDate="2010-10-13T19:25:05.967" Score="7" ViewCount="" Body="&lt;p&gt;Principal components make a lot of sense... mathematically. However, I'd be wary of simply using some mathematical trick in this case and hoping that I don't need to &lt;em&gt;think&lt;/em&gt; about my problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd recommend thinking a little about what kind of predictors I have, what the independent variable is, &lt;em&gt;why&lt;/em&gt; my predictors are correlated, whether some of my predictors are actually measuring the same underlying reality (if so, whether I can just work with a single measurement and which of my predictors would be best for this), what I am doing the analysis for - if I'm not interested in inference, only in prediction, then I could actually leave things just as they are, as long as future predictor values are similar to past ones.&lt;/p&gt;&#xA;" OwnerUserId="1352" LastActivityDate="2010-10-13T19:25:05.967" CommentCount="2" />
  <row Id="3570" PostTypeId="2" ParentId="3559" CreationDate="2010-10-13T19:33:11.767" Score="5" ViewCount="" Body="&lt;p&gt;I found Tue Tjur's short &lt;a href=&quot;http://pubs.amstat.org/doi/abs/10.1198/tast.2009.08210&quot; rel=&quot;nofollow&quot;&gt;paper&lt;/a&gt; on various proposals for a coefficient of determination in logistic models quite enlightening. He does a good job highlighting pros and cons - and of course offers a new definition. Very much recommended (though I have no favorite myself).&lt;/p&gt;&#xA;" OwnerUserId="1352" LastActivityDate="2010-10-13T19:33:11.767" CommentCount="1" />
  <row Id="3571" PostTypeId="2" ParentId="3561" CreationDate="2010-10-13T19:43:22.190" Score="1" ViewCount="" Body="&lt;p&gt;Here is another thought that is inspired by Stephan's &lt;a href=&quot;http://stats.stackexchange.com/questions/3561/dealing-with-correlated-regressors/3569#3569&quot;&gt;answer&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If some of your correlated regressors are meaningfully related (e.g., they are different measures of intelligence i.e., verbal, math etc) then you can create a single variable that measures the same variable using one of the following techniques:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Sum the regressors (appropriate if the regressors are components of a whole, e.g., verbal IQ + math IQ = Overall IQ)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Average of the regressors (appropriate if the regressors are measuring the same underlying construct e.g., size of left shoe, size of right shoe to measure length of feet)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Factor analysis (to account for errors in measurements and to extract a latent factor)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You can then drop all the correlated regressors and replace them with the one variable that emerges from the above analysis. &lt;/p&gt;&#xA;" LastActivityDate="2010-10-13T19:43:22.190" CommentCount="1" />
  <row Id="3572" PostTypeId="2" ParentId="3561" CreationDate="2010-10-13T19:48:14.617" Score="1" ViewCount="" Body="&lt;p&gt;I was about to say much the same thing as Stephan Kolassa above (so have upvoted his answer). I'd only add that sometimes multicollinearity can be due to using &lt;a href=&quot;http://en.wikipedia.org/wiki/Intensive_and_extensive_properties&quot; rel=&quot;nofollow&quot;&gt;extensive variables&lt;/a&gt; which are all highly correlated with some measure of size, and things can be improved by using intensive variables, i.e. dividing everything through by some measure of size. E.g. if your units are countries, you might divide by population, area, or GNP, depending on context. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Oh - and to answer the second part of the original question: I can't think of &lt;em&gt;any&lt;/em&gt; situation when adding the product of all the correlated regressors would be a good idea. How would it help? What would it mean? &lt;/p&gt;&#xA;" OwnerUserId="449" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-10-14T07:24:54.397" LastActivityDate="2010-10-14T07:24:54.397" CommentCount="1" />
  <row Id="3573" PostTypeId="2" ParentId="3564" CreationDate="2010-10-13T19:54:25.117" Score="3" ViewCount="" Body="&lt;p&gt;I would have thought that KDE bear little if any relationship to predicting future sales based on past sales. Sounds more like &lt;a href=&quot;http://en.wikipedia.org/wiki/Time_series&quot; rel=&quot;nofollow&quot;&gt;time series analysis&lt;/a&gt; to me, though that's really not my area.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-10-13T19:54:25.117" CommentCount="3" />
  <row Id="3574" PostTypeId="2" ParentId="3559" CreationDate="2010-10-13T20:08:11.973" Score="3" ViewCount="" Body="&lt;p&gt;I was also going to say 'neither of them', so i've upvoted whuber's answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As well as criticising R^2, Hosmer &amp;amp; Lemeshow did propose an alternative measure of goodness-of-fit for logistic regression that is sometimes useful. This is based on dividing the data into (say) 10 groups of equal size (or as near as possible) by ordering on the predicted probability (or equivalently, the linear predictor) then comparing the observed to expected number of positive responses in each group and performing a chi-squared test. This 'Hosmer-Lemeshow goodness-of-fit test' is implemented in most statistical software packages.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-10-13T20:08:11.973" CommentCount="3" />
  <row Id="3575" PostTypeId="1" AcceptedAnswerId="3576" CreationDate="2010-10-13T20:45:25.290" Score="3" ViewCount="127" Body="&lt;p&gt;I've got a data table like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ID  Low Color    Med Color     High Color&#xA; 1        234          123            324&#xA; 2          4          432           3423&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The rows are widgets, the columns are color levels. Would you call this table &quot;widgets by color level&quot; or &quot;color levels by widget&quot;?&lt;/p&gt;&#xA;" OwnerUserId="1531" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-10-14T17:30:04.140" LastActivityDate="2010-10-14T17:30:04.140" Title="Is it &quot;rows by columns&quot; or &quot;columns by rows&quot;?" Tags="&lt;terminology&gt;" AnswerCount="4" CommentCount="0" />
  <row Id="3576" PostTypeId="2" ParentId="3575" CreationDate="2010-10-13T20:50:40.067" Score="5" ViewCount="" Body="&lt;p&gt;&quot;That depends.&quot;  Rows are usually considered observations, and columns are variables.  So I would say &lt;strong&gt;widgets by color level&lt;/strong&gt; in your context.  But it really depends on which are your &lt;a href=&quot;http://en.wikipedia.org/wiki/Dependent_and_independent_variables&quot; rel=&quot;nofollow&quot;&gt;dependent and independent variables&lt;/a&gt; (or how you're interpreting the data).&lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-10-13T20:50:40.067" CommentCount="2" />
  <row Id="3577" PostTypeId="2" ParentId="3575" CreationDate="2010-10-13T20:51:02.110" Score="2" ViewCount="" Body="&lt;p&gt;Of course you can view this table either way by transposing it.  Conventionally, in a database rows represent objects and columns contain their attributes, whence this presentation would typically be viewed as a list of widgets, not a list of color levels.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-10-13T20:51:02.110" />
  <row Id="3578" PostTypeId="2" ParentId="3575" CreationDate="2010-10-13T20:54:42.053" Score="1" ViewCount="" Body="&lt;p&gt;Typically, we talk about a r(ow) X (c)olumn matrix, from Linear Algebra.  So, a matrix with 2 rows and 3 columns is a 2 X 3 matrix.  By that logic, I'd call your data frame a &quot;Widgets by Color&quot; table.&lt;/p&gt;&#xA;" OwnerUserId="485" LastActivityDate="2010-10-13T20:54:42.053" />
  <row Id="3579" PostTypeId="2" ParentId="3575" CreationDate="2010-10-13T21:46:38.637" Score="2" ViewCount="" Body="&lt;p&gt;for a table like that, I say it the same way I'd say &quot;n by k&quot; for a matrix with n rows and k columns (i.e. rows first).&lt;/p&gt;&#xA;" OwnerUserId="805" LastActivityDate="2010-10-13T21:46:38.637" />
  <row Id="3581" PostTypeId="2" ParentId="3564" CreationDate="2010-10-13T22:49:26.513" Score="4" ViewCount="" Body="&lt;p&gt;You can use conditional kernel density estimation to obtain the density of sales at time $t+h$ conditional on the values of sales at times $t, t-1, t-2, \dots$ This gives you a density forecast rather than a point forecast. The problem is that the conditioning is difficult in a density setting when the number of conditioning variables is more than 2. See &lt;a href=&quot;http://www.jstor.org/stable/1390887&quot; rel=&quot;nofollow&quot;&gt;this paper&lt;/a&gt; for a discussion of the basic idea.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An alternative procedure that imposes more assumptions (but allows more conditioning variables) is to fit an additive autoregression such as described in &lt;a href=&quot;http://www.jstor.org/stable/2290787&quot; rel=&quot;nofollow&quot;&gt;Chen and Tsay (1993)&lt;/a&gt; and then use kde on the residuals to obtain the forecast densities.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I suspect that both of these are more complicated than what you really want. I suggest you read a textbook on demand forecasting such as &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0534262686&quot; rel=&quot;nofollow&quot;&gt;Levenbach and Cleary (2006)&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-10-13T22:49:26.513" CommentCount="1" />
  <row Id="3582" PostTypeId="2" ParentId="3556" CreationDate="2010-10-14T01:16:51.910" Score="0" ViewCount="" Body="&lt;p&gt;Loess/lowess is basically a variable KDE method, with the width of the kernel being set by the nearest-neighbour approach. I've found that it works pretty well, certainly much better than any fixed-width model when the density of data points varies markedly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One thing to be aware of with KDE and multi-dimensional data is the curse of dimensionality. Other things being equal, there are far fewer points within a set radius when p ~ 10, than when p ~ 2. This may not be a problem for you if you only have 3d data, but it's something to keep in mind.&lt;/p&gt;&#xA;" OwnerUserId="1569" LastActivityDate="2010-10-14T01:16:51.910" CommentCount="10" />
  <row Id="3583" PostTypeId="2" ParentId="3552" CreationDate="2010-10-14T03:22:37.420" Score="2" ViewCount="" Body="&lt;p&gt;Rather than the code you present, I assume you're doing something like &lt;code&gt;class x21 x22&lt;/code&gt; followed by the &lt;code&gt;repeated&lt;/code&gt; clause &lt;code&gt;group=x21*x22&lt;/code&gt; to end up with 12 parameters.  This is the only option I'm aware of within SAS, i.e. I don't think you can get a straightforward stratification of the variation across the combined levels of two variables.  But, unless I'm missing something, you should be able re-code your variable into a combined &lt;code&gt;x3&lt;/code&gt; to achieve this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, just to check -- I assumed that your &lt;code&gt;x2&lt;/code&gt; variable is prepared such that it holds the appropriate exponential values for your log-linear variance model.  Otherwise, you may want to check out the &lt;code&gt;local&lt;/code&gt; option to the repeated statement.  For example, the following model in R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;wgt &amp;lt;- varExp(form =~ x)&#xA;fit &amp;lt;- gls(y ~ x, weights=wgt, data=dat)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;corresponds to something like the following in SAS:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;proc mixed data=dat;&#xA;  model y = x / solution;&#xA;  repeated / local=exp(x);&#xA;run;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The covariance parameter estimates will differ by a factor of 2 between R and SAS.  This is because the R version models $var(\epsilon_{ij}) = \sigma^2 e^{ 2bx_{i} }$ (see &lt;code&gt;?varExp&lt;/code&gt;).  I'm guessing you're processing your input and not using varExp because this isn't easy to accomplish in R if you have factor variables, but I thought I'd mention it in case it's helpful.&lt;/p&gt;&#xA;" OwnerUserId="251" LastEditorUserId="251" LastEditorDisplayName="" LastEditDate="2010-10-14T03:36:05.547" LastActivityDate="2010-10-14T03:36:05.547" CommentCount="0" />
  <row Id="3584" PostTypeId="1" AcceptedAnswerId="3585" CreationDate="2010-10-14T03:36:34.627" Score="8" ViewCount="278" Body="&lt;p&gt;I have a dataset asking people whether they have been to a certain places (e.g. A, B, C, D), and they can make more than one choice, then a specimen is taken from their nose to see if they are infected with some disease.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to find out the relative risk of getting infected for one going to a certain place, I can only think of logistic regression right now, is there any other suggestions?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="588" LastEditorUserId="1352" LastEditorDisplayName="" LastEditDate="2010-10-14T20:37:16.073" LastActivityDate="2010-10-14T20:37:16.073" Title="How to deal with survey question with multiple response?" Tags="&lt;logistic-regression&gt;" AnswerCount="1" FavoriteCount="2" />
  <row Id="3585" PostTypeId="2" ParentId="3584" CreationDate="2010-10-14T04:40:09.423" Score="2" ViewCount="" Body="&lt;p&gt;You can still use logistic regression because your outcome is dichotomous, infected vs not-infected. I would just simply take a dummy variable approach and use no travel as the reference category (i.e. for each of your places you have a variable coded as 1 if they visited that place and coded as 0 if they did not visit that place). As such if you transform your beta coefficients to odds (i.e. exponentiate the log odds) the interpretation of the dummy variable for location A would be the odds ratio of visiting location A over not visiting location A controlling for other places one visited. Also note in this approach multi-collinearity is a concern (e.g. if many of the people who travel to A also travel to B it may bias each of their coefficients).&lt;/p&gt;&#xA;" OwnerUserId="1036" LastActivityDate="2010-10-14T04:40:09.423" CommentCount="3" />
  <row Id="3586" PostTypeId="1" AcceptedAnswerId="3593" CreationDate="2010-10-14T05:14:57.183" Score="2" ViewCount="196" Body="&lt;p&gt;I am trying to calculate the relative risk for being tested as positive for people aged &gt;25 or &amp;lt;=25, and here is the result.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;$data&#xA;&#xA;        Negative Positive Total&#xA;  &amp;gt;25        115       11   126&#xA;  &amp;lt;=25       117        3   120&#xA;  Total      232       14   246&#xA;&#xA;$measure&#xA;      risk ratio with 95% C.I.&#xA;        estimate      lower    upper&#xA;  &amp;gt;25  1.0000000         NA       NA&#xA;  &amp;lt;=25 0.2863636 0.08189025 1.001391&#xA;&#xA;$p.value&#xA;      two-sided&#xA;       midp.exact fisher.exact chi.square&#xA;  &amp;gt;25          NA           NA         NA&#xA;  &amp;lt;=25 0.03815237   0.05147711 0.03500415&#xA;&#xA;$correction&#xA;[1] FALSE&#xA;&#xA;attr(,&quot;method&quot;)&#xA;[1] &quot;Unconditional MLE &amp;amp; normal approximation (Wald) CI&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;One thing that puzzled me a lot is that why I get a statistically significant result from chi.square (I don't use fisher's exact test because NONE OF THE expected value is small than 5), but the risk ratio includes 1, how can I explain? Thanks.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;CORRECTION&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Sorry for the mistake, I rechecked the data and NONE of the expected cell is smaller than 5&lt;/p&gt;&#xA;" OwnerUserId="588" LastEditorUserId="588" LastEditorDisplayName="" LastEditDate="2010-10-14T07:09:09.693" LastActivityDate="2010-10-15T06:16:16.390" Title="Inconsistency between Chi-sq and CI Estimation using Wald test" Tags="&lt;chi-squared&gt;&lt;relative-risk&gt;" AnswerCount="3" CommentCount="2" />
  <row Id="3587" PostTypeId="2" ParentId="3586" CreationDate="2010-10-14T06:34:56.423" Score="0" ViewCount="" Body="&lt;p&gt;Hello lokheart,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not sure what is the $H_o$ in the case of the hypothesis test. Usually its $H_o: OR=1$. If this is true then, your situation can arise because of the following reasons: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;em&gt;asymptotics&lt;/em&gt; that you comparing might be very different in the two cases. The test statistic's sampling distribution for relative risk and OR scenario can reach their asymptotic distribution differently, hence the discrepancy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But if $H_o: RR=1$ then&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you mention yourself, one of the cell count is &amp;lt; 5. This condition should hold for all the cells for the Chi Sq test as well (&lt;a href=&quot;http://en.wikipedia.org/wiki/Pearson%27s_chi-square_test&quot; rel=&quot;nofollow&quot;&gt;AFAIK&lt;/a&gt;). Due to this the &lt;em&gt;asymptotic&lt;/em&gt; Chi Sq distribution may not be completely accurate. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The p-value may be calculated based on the &lt;strong&gt;NORMAL APPROXIMATION&lt;/strong&gt; where as the CI may be calculated based on the &lt;strong&gt;CHI SQ APPROXIMATION&lt;/strong&gt;. But this is total guess work. Which package are you using?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P-value and CI should agree as they are calculated based on the same $H_o$ (as you expect).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The main reason for the discrepancy &lt;strong&gt;may be&lt;/strong&gt; the &lt;strong&gt;Asymptotic Nature of the Wald test&lt;/strong&gt;. No one can tell when the asymptotics is true, all you can do is hope for the best and assume your sample to be &quot;good enough&quot;, so that if you have &quot;large enough&quot; sample asymptotics are true. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When such a situation arises, it is better to trust Non Parametric test like the Fisher's Exact Test because the parametric tests are based on asymptotic assumptions. FET being non-parametric (you are basically counting all possible tables that can occur as extreme cases conditioned on the present table), is less restrictive, hence more trustworthy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your point is valid that FET is not trust worthy as one cell count &amp;lt; 5, but in that case the Chi Sq distribution is not trustworthy as well. &lt;/p&gt;&#xA;" OwnerUserId="1307" LastEditorUserId="1307" LastEditorDisplayName="" LastEditDate="2010-10-14T06:53:23.313" LastActivityDate="2010-10-14T06:53:23.313" CommentCount="1" />
  <row Id="3588" PostTypeId="2" ParentId="3556" CreationDate="2010-10-14T06:52:05.923" Score="2" ViewCount="" Body="&lt;p&gt;The article * D. G. Terrell; D. W. Scott (1992). &quot;Variable kernel density estimation&quot;. Annals of Statistics 20: 1236–1265.* cited at the end of the Wikipedia article you yourself cite clearly states that unless the observations space is very sparse the variable kernel method is &lt;strong&gt;not&lt;/strong&gt; recommended on the basis of global root mean squared error (both local and global) for Gaussian distributed random variables: (through theoretical arguments) they cite the figures of $n\leq 450$ ($n$ is the sample size) and (through bootstrapping results) $p\geq 4$ ($p$ is the number of dimension) as the settings in which variable kernel method become competitive with fixed width ones (judging from your question you are not in these settings). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The intuition behind these results is that if you are not in very sparse settings, then, the local density simply does not vary enough for the gain in bias to outdo the loss in efficiency (and hence the AMISE of variable width kernel increases relative to the AMISE of fixed width). Also, given the large sample size you have (and the small dimensions) the fixed width kernel will be very local already, diminishing any potential gains in terms of bias. &lt;/p&gt;&#xA;" OwnerUserId="603" LastActivityDate="2010-10-14T06:52:05.923" CommentCount="4" />
  <row Id="3589" PostTypeId="1" AcceptedAnswerId="3605" CreationDate="2010-10-14T06:56:42.463" Score="4" ViewCount="447" Body="&lt;p&gt;In a problem I am working on, I have two random variables, X and Y. I need to figure out how closely correlated the two of them are, but they are of different dimensions. The rank of the row space of X is 4350, and the rank of the row space of Y is substantially larger, in the tens of thousands. Both X and Y have the same number of columns.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need a measure of correlation between the two variables, and Pearson's r requires X and Y to have equal dimension (at least R requires the two r.v.'s to be). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do I have any hope of doing a correlation between these two, or should I find some way of pruning off observations from Y?&lt;/p&gt;&#xA;" OwnerUserId="1118" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-15T14:41:18.400" LastActivityDate="2010-12-30T10:58:21.897" Title="Correlation between two variables of unequal size" Tags="&lt;time-series&gt;&lt;correlation&gt;&lt;finance&gt;" AnswerCount="6" CommentCount="6" FavoriteCount="2" />
  <row Id="3590" PostTypeId="2" ParentId="2948" CreationDate="2010-10-14T07:15:50.350" Score="2" ViewCount="" Body="&lt;p&gt;Hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Gephi implements the Louvain Modularity method: &lt;a href=&quot;http://wiki.gephi.org/index.php/Modularity&quot; rel=&quot;nofollow&quot;&gt;http://wiki.gephi.org/index.php/Modularity&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;cheers&lt;/p&gt;&#xA;" LastActivityDate="2010-10-14T07:15:50.350" CommentCount="1" />
  <row Id="3591" PostTypeId="2" ParentId="3586" CreationDate="2010-10-14T07:44:10.447" Score="1" ViewCount="" Body="&lt;p&gt;I think suncoolsu is on the right track, though RR = 1 if and only if OR = 1, so those null hypotheses are in fact one and the same.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Pearson's chi-square test can be derived as a score test. In finite samples, score tests are, in general, better behaved than Wald tests, but less good than likelihood-ratio tests.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want your CIs and p-values to look more consistent, you may be best switching to a different (more accurate) way of calculating the CI. I see the R package you're using offers a small-sample adjustment or a bootstrap method as alternatives to the standard Wald method.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-10-14T07:44:10.447" CommentCount="4" />
  <row Id="3592" PostTypeId="1" AcceptedAnswerId="3604" CreationDate="2010-10-14T07:45:21.910" Score="5" ViewCount="141" Body="&lt;p&gt;Hello all,&#xA;I'm wondering about this question:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;short version:&#xA;how to adequately compare the effect of a reclassification of the same subjects on survival&lt;/p&gt;&#xA;&#xA;&lt;p&gt;long version:&#xA;I have one cancer cohort that was sorted into TNM classes long time ago. The definition of the classes has now been updated and I like to know if, say, class 1 of the old classification has a better/worse survival than class 1 of the updated classification. My problem is that the cohort stays the same, only the sorting into classes changes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Mini example:&#xA;patients 1,2 are in class I_old and patients 3,4 in class II_old, patient 1 stays in class I_new but patient 2 goes to class II_new, patient 3 stays in II_new, patient 4 changes to I_new. If I want to compare the effect of the reclassification in class I, I would have to compare the survival of patients 1 and 2 versus the the survival of patients 1 and 4. So patient 1 is in both of the groups.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I doubt that the standard procedures like logrank test are adequate here. But what can be done? I have a very vague idea about proportional hazards regression with a time varying covariate (namely indicator for &quot;before&quot; and &quot;after&quot; update) or a frailty model but cannot tell at all if I'm going in the right direction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What do you think?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;psj&lt;/p&gt;&#xA;" OwnerUserId="1573" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-11-11T23:24:58.520" LastActivityDate="2010-11-11T23:24:58.520" Title="Survival analysis, one cohort, two classifications" Tags="&lt;hypothesis-testing&gt;&lt;survival&gt;&lt;non-independent&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="1" />
  <row Id="3593" PostTypeId="2" ParentId="3586" CreationDate="2010-10-14T07:45:42.830" Score="4" ViewCount="" Body="&lt;p&gt;The Wald test for contingency tables is known to be misleading or conservative and the general advice is to prefer the Likelihood Ratio Test:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; library(vcd)&#xA;&amp;gt; mat &amp;lt;- matrix(c(115, 117, 11, 3), 2, 2)&#xA;&amp;gt; assocstats(mat)$chisq_tests&#xA;                      X^2 df   P(&amp;gt; X^2)&#xA;Likelihood Ratio 4.730771  1 0.02962760&#xA;Pearson          4.444973  1 0.03500415&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Wald type tests are of the form $\frac{\hat{\theta} - \theta_0} {SE(\theta)}$ with the assumption that the difference is normal, which is often untenable with smaller samples or low counts in frequency tables.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;An intuitive reason for discounting the Wald test is that you're using the available data to estimate both $\hat\theta$ and the standard error to conduct the test.  Contrast this with the LRT which uses only one estimate from the data.  In a similar vein, I have &lt;a href=&quot;http://stats.stackexchange.com/questions/359/the-trinity-of-tests-in-maximum-likelihood-what-to-do-when-faced-with-contradict&quot;&gt;another answer&lt;/a&gt; that tries to explain why the Wald CI can be kooky at times and why the LRT should generally be preferred.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might also be interested in a paper that explored some of the ways in which the Wald test can be misleading in these types of analyses:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.jstor.org/pss/2286473&quot; rel=&quot;nofollow&quot;&gt;Wald's Test as Applied to Hypotheses in Logit Analysis&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="251" LastEditorUserId="251" LastEditorDisplayName="" LastEditDate="2010-10-15T06:16:16.390" LastActivityDate="2010-10-15T06:16:16.390" CommentCount="1" />
  <row Id="3594" PostTypeId="2" ParentId="3589" CreationDate="2010-10-14T08:59:31.170" Score="6" ViewCount="" Body="&lt;p&gt;So the problem is one of missing data (not all Y have a corresponding X, where correspondence is operationalized via time points). I don't think there is much to do here than just to throw away the Y you don't have an X for and calculate the correlation on the full pairs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may want to read up on financial time series, though I don't have a good reference handy at this point (ideas, anyone?). Stock prices often exhibit time-varying volatilities, which can be modeled, e.g., by &lt;a href=&quot;http://en.wikipedia.org/wiki/GARCH&quot; rel=&quot;nofollow&quot;&gt;GARCH&lt;/a&gt;. It is conceivable that your two time series X and Y exhibit positive correlations during periods of low volatility (when the economy grows, all stock prices tend to increase), but negative correlations when overall volatility is high (on 9/11, airlines tanked while money fled to safer investments). So just calculating an overall correlation may be too dependent on your observation time frame.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;UPDATE: I think you may want to look at &lt;a href=&quot;http://en.wikipedia.org/wiki/Vector_autoregression&quot; rel=&quot;nofollow&quot;&gt;VAR (vector autoregressive)&lt;/a&gt; models.&lt;/p&gt;&#xA;" OwnerUserId="1352" LastEditorUserId="1352" LastEditorDisplayName="" LastEditDate="2010-10-14T13:21:45.070" LastActivityDate="2010-10-14T13:21:45.070" CommentCount="1" />
  <row Id="3595" PostTypeId="1" CreationDate="2010-10-14T10:28:15.937" Score="13" ViewCount="767" Body="&lt;p&gt;Hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently I have to analyze approximately 20M records and build prediction models. So far I have tried out Statistica, SPSS, RapidMiner and R. Among these Statistica seems to be most suitable to deal with data mining and RapidMiner user interface is also very handy but it seems that Statistica, RapidMiner and SPSS are suitable just for smaller datasets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could anyone recommend a good tool for large datasets?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks you!&lt;/p&gt;&#xA;" OwnerUserId="315" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-10-14T17:54:48.267" LastActivityDate="2010-10-18T03:12:18.987" CommunityOwnedDate="2010-10-14T13:05:32.563" Title="Statistics and data mining software tools for dealing with large datasets" Tags="&lt;large-data&gt;&lt;software&gt;&lt;data-mining&gt;" AnswerCount="6" CommentCount="3" FavoriteCount="6" />
  <row Id="3596" PostTypeId="2" ParentId="2061" CreationDate="2010-10-14T10:31:18.257" Score="0" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://datajournalism.stanford.edu/&quot; rel=&quot;nofollow&quot;&gt;http://datajournalism.stanford.edu/&lt;/a&gt; : Video on visualisation&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-10-14T10:31:18.257" CommunityOwnedDate="2010-10-14T10:31:18.257" CommentCount="1" />
  <row Id="3597" PostTypeId="2" ParentId="3595" CreationDate="2010-10-14T11:15:57.903" Score="13" ViewCount="" Body="&lt;p&gt;I'll second @suncoolsu comment: The dimensionality of your data set is not the only criterion that should orient you toward a specific software. For instance, if you're just planning to do unsupervised clustering or use PCA, there are several dedicated tools that cope with large data sets, as commonly encountered in genomic studies.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, R (64 bits) handles large data pretty well, and you still have the option to use disk storage instead of RAM access, but see CRAN Task View &lt;a href=&quot;http://cran.r-project.org/web/views/HighPerformanceComputing.html&quot; rel=&quot;nofollow&quot;&gt;High-Performance and Parallel Computing with R&lt;/a&gt;. Standard GLM will easily accomodate 20,000 obs. (but see also &lt;a href=&quot;http://cran.r-project.org/web/packages/speedglm/index.html&quot; rel=&quot;nofollow&quot;&gt;speedglm&lt;/a&gt;) within reasonable time, as shown below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; require(MASS)&#xA;&amp;gt; n &amp;lt;- 20000&#xA;&amp;gt; X &amp;lt;- mvrnorm(n, mu=c(0,0), Sigma=matrix(c(1,.8,.8,1), 2, 2))&#xA;&amp;gt; df &amp;lt;- cbind.data.frame(X, grp=gl(4, n/4), y=sample(c(0,1), n, rep=TRUE))&#xA;&amp;gt; system.time(glm(y ~ ., data=df))&#xA;   user  system  elapsed&#xA;  0.361   0.018    0.379&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To give a more concrete illustration, I used R to process and analyse large genetic data (800 individuals x 800k &lt;a href=&quot;http://www.google.fr/url?sa=t&amp;amp;source=web&amp;amp;cd=2&amp;amp;ved=0CCIQFjAB&amp;amp;url=http%3A%2F%2Fen.wikipedia.org%2Fwiki%2FSingle-nucleotide_polymorphism&amp;amp;rct=j&amp;amp;q=single%20nucleotide%20polymorphisms&amp;amp;ei=hDu4TKyRLJDp4gai8MGNDg&amp;amp;usg=AFQjCNGPTlkC2hY6vz7EXSbJe4_YYrumgw&amp;amp;sig2=sweuWH-AMVYdmg_NT2uX8Q&amp;amp;cad=rja&quot; rel=&quot;nofollow&quot;&gt;SNPs&lt;/a&gt;, where the main statistical model was a stratified GLM with several covariates (2 min); that was made possible thanks to efficient R and C codes available in the &lt;a href=&quot;http://www.google.fr/url?sa=t&amp;amp;source=web&amp;amp;cd=1&amp;amp;ved=0CBUQFjAA&amp;amp;url=http%3A%2F%2Fwww.bioconductor.org%2Fpackages%2F2.3%2Fbioc%2Fhtml%2FsnpMatrix.html&amp;amp;rct=j&amp;amp;q=snpMatrix&amp;amp;ei=Qjy4TNDJOIXG4AbD1JHbDQ&amp;amp;usg=AFQjCNFJM_G_MFWKEDrIILpdWh8AVATn4g&amp;amp;sig2=wMAoR3l7SB41EG-Lvx9MOQ&amp;amp;cad=rja&quot; rel=&quot;nofollow&quot;&gt;snpMatrix&lt;/a&gt; package (in comparison, the same kind of model took about 8 min using a dedicated C++ software (&lt;a href=&quot;http://www.google.fr/url?sa=t&amp;amp;source=web&amp;amp;cd=1&amp;amp;ved=0CBUQFjAA&amp;amp;url=http%3A%2F%2Fpngu.mgh.harvard.edu%2F~purcell%2Fplink%2F&amp;amp;rct=j&amp;amp;q=plink&amp;amp;ei=kDy4TMmgAZTe4AbPwcm9Dg&amp;amp;usg=AFQjCNHI9oFtR2YOxmlkKaGIMkTnNxTbSA&amp;amp;sig2=yWlRajpLg9eCIIOKoqy5hA&amp;amp;cad=rja&quot; rel=&quot;nofollow&quot;&gt;plink&lt;/a&gt;). I also worked on a clinical study (12k patients x 50 variables of interest) and R fits my needs too. Finally, as far as I know, the &lt;a href=&quot;http://cran.r-project.org/web/packages/lme4/index.html&quot; rel=&quot;nofollow&quot;&gt;lme4&lt;/a&gt; package is the only software that allow to fit mixed-effects model with unbalanced and large data sets (as is the case in large-scale educational assessment).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Stata/SE is another software that can handle &lt;a href=&quot;http://www.stata.com/order/new/corp/difference.html&quot; rel=&quot;nofollow&quot;&gt;large data set&lt;/a&gt;. SAS and SPSS are file based software, so they will handle large volumes of data. A comparative review of software for datamining is available in &lt;a href=&quot;http://www.information-management.com/specialreports/20060321/1049954-1.html&quot; rel=&quot;nofollow&quot;&gt;Data Mining Tools: Which One is Best for CRM&lt;/a&gt;. For visualization, there are also plenty of options; maybe a good start is &lt;a href=&quot;http://www.springer.com/statistics/statistical+theory+and+methods/book/978-0-387-32906-2&quot; rel=&quot;nofollow&quot;&gt;Graphics of large datasets: visualizing a million&lt;/a&gt; (&lt;a href=&quot;http://www.jstatsoft.org/v17/b01/paper&quot; rel=&quot;nofollow&quot;&gt;reviewed&lt;/a&gt; in the JSS by P Murrell), and all related threads on this site.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-15T11:39:14.457" LastActivityDate="2010-10-15T11:39:14.457" CommunityOwnedDate="2010-10-14T13:05:32.563" CommentCount="8" />
  <row Id="3598" PostTypeId="2" ParentId="3589" CreationDate="2010-10-14T11:45:25.467" Score="1" ViewCount="" Body="&lt;p&gt;Given the extra information in your comments I'd recommend looking at two correlations.  The first would be the common time periods that the companies were both around.  So, if one was around 2 years earlier you'd just drop that data and look at the rest.  The second would be the relative time periods.  In the second one you're not correlating actual time but time measured since the company went public.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The former would be strongly influenced by general economic forces shared within the same time period.  The latter would be influenced by properties shared by companies as they change after the IPO.&lt;/p&gt;&#xA;" OwnerUserId="601" LastEditorUserId="601" LastEditorDisplayName="" LastEditDate="2010-10-14T19:09:49.620" LastActivityDate="2010-10-14T19:09:49.620" />
  <row Id="3599" PostTypeId="1" AcceptedAnswerId="3607" CreationDate="2010-10-14T12:36:05.257" Score="5" ViewCount="83" Body="&lt;p&gt;Could anyone give me hints as to a model framework that can be used in the following setting:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The outcome A is dichotomous. I want to investigate the effect of a continuous variable B  and a continous variable C on A in a longitudinal setting. So far so good. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is that C depends on B, in that values of B above a threshold (Unfortunately we do not know the threshold and the threshold itself might not be constant over time) will change C. The change in C will furthermore reduce B in the next measurement.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I decipher the isolated effect og B and C on A? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would a mixed-logistic-regression model be OK?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ie;t=time&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A~B*t+C*t+B*C*t&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any other ideas?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;//M&lt;/p&gt;&#xA;" OwnerUserId="1291" LastEditorUserId="1291" LastEditorDisplayName="" LastEditDate="2010-10-14T19:09:39.820" LastActivityDate="2010-10-14T19:09:39.820" Title="Model suggestion" Tags="&lt;modeling&gt;&lt;causal-inference&gt;" AnswerCount="3" CommentCount="2" />
  <row Id="3600" PostTypeId="2" ParentId="3589" CreationDate="2010-10-14T13:31:19.233" Score="1" ViewCount="" Body="&lt;p&gt;@Jeromy Anglim specified this correctly.  Having the extra information when only one of the time series existed would provide no value here.  And in principle, the data should be sampled at the same time for it to be meaningful using conventional correlation measures.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a more general problem, I would add that there are techniques to deal with irregularly spaced time series data.  You can search for &quot;irregularly spaced time series correlation&quot;.  Some of the recent work has been done on &lt;a href=&quot;http://www.ssc.upenn.edu/~fdiebold/papers/paper29/temp.pdf&quot; rel=&quot;nofollow&quot;&gt;&quot;Realized Volatility and Correlation&quot;&lt;/a&gt; (Andersen, Bollerslev, Diebold, and Labys 1999) using high-frequency data.   &lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-10-14T13:31:19.233" />
  <row Id="3601" PostTypeId="2" ParentId="3595" CreationDate="2010-10-14T13:44:36.413" Score="4" ViewCount="" Body="&lt;p&gt;It is hard to give a good answer without knowing what kind of models you have in mind.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For linear regression, I have successfully used the &lt;i&gt;biglm&lt;/i&gt; package in R.&lt;/p&gt;&#xA;" OwnerUserId="439" LastActivityDate="2010-10-14T13:44:36.413" CommunityOwnedDate="2010-10-14T13:44:36.413" />
  <row Id="3602" PostTypeId="2" ParentId="3589" CreationDate="2010-10-14T13:49:41.147" Score="0" ViewCount="" Body="&lt;p&gt;Another way to solve such a problem is to impute the missing data for the shorter series using a time series model which may or may not make sense in a particular context. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your context, imputing the stock prices into the past would mean that you are asking the following counter-factual question: What would be the stock price for company X had it gone public n years in the past instead of when it actually went public? Such a data imputation could potentially be done by taking into account stock prices of related companies, general market trends etc. But, such an analysis may not make sense or may not be needed given the goals of your project.&lt;/p&gt;&#xA;" LastActivityDate="2010-10-14T13:49:41.147" />
  <row Id="3603" PostTypeId="2" ParentId="3599" CreationDate="2010-10-14T13:57:04.500" Score="1" ViewCount="" Body="&lt;p&gt;Your problem is one of multi-collinear regressors (since B and C are correlated). I would suggest that you look at the answers to the question: &lt;a href=&quot;http://stats.stackexchange.com/q/3561/28&quot;&gt;Dealing with correlated regressors&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The following paper may also be relevant in your context: &lt;a href=&quot;http://dx.doi.org/10.1016/j.csda.2005.03.011&quot; rel=&quot;nofollow&quot;&gt;Using principal components for estimating logistic regression with high-dimensional multicollinear data&lt;/a&gt; &lt;/p&gt;&#xA;" LastActivityDate="2010-10-14T13:57:04.500" CommentCount="3" />
  <row Id="3604" PostTypeId="2" ParentId="3592" CreationDate="2010-10-14T13:58:37.590" Score="4" ViewCount="" Body="&lt;p&gt;I'll concentrate on your example question: does class 1 of the old classification scheme have a better or worse survival than class 1 of the updated classification scheme?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can form four mutually exclusive groups of patients:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(a) Patients who weren't in class 1 under either scheme. Clearly, they don't help us answer the question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(b) Patients who were in class 1 under both schemes. Clearly, they don't help us answer the question either.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(c) Patients who were in class 1 under the old scheme, but aren't in class 1 under the new scheme.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(d) Patients who weren't in class 1 under the old scheme, but are in class 1 under the new scheme.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Compare survival in groups (c) and (d). If survival if better in (c), then class 1 of the old scheme has better survival than class 1 of the new scheme. If survival if better in (d), then class 1 of the new scheme has better survival than class 1 of the old scheme.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-10-14T13:58:37.590" CommentCount="1" />
  <row Id="3605" PostTypeId="2" ParentId="3589" CreationDate="2010-10-14T14:07:20.877" Score="4" ViewCount="" Body="&lt;p&gt;No amount of imputation, time series analysis, GARCH models, interpolation, extrapolation, or other fancy algorithms will do anything to create information where it does not exist (although they can create that illusion ;-).  The history of Y's price before X went public is useless for assessing their subsequent correlation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sometimes (often preparatory to an IPO) analysts use internal accounting information (or records of private stock transactions) to retrospectively reconstruct hypothetical prices for X's stock before it went public.  Conceivably such information could be used to enhance estimates of correlation, but given the extremely tentative nature of such backcasts, I doubt the effort would be of any help except initially when there are only a few days or weeks of prices for X available.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-10-14T14:07:20.877" CommentCount="5" />
  <row Id="3606" PostTypeId="2" ParentId="3599" CreationDate="2010-10-14T14:21:48.683" Score="2" ViewCount="" Body="&lt;p&gt;You have clearly stated a part of your model:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;C depends on B, in that values of B above a threshold will change C. The change in C will furthermore reduce B in the next measurement.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;By &quot;next measurement&quot; I understand you mean next &lt;em&gt;in time.&lt;/em&gt;  Let's index time as $t = 0, 1, 2, \ldots$.  Then the dependence of C on B sounds like a contemporaneous one.  If we adopt a simple linear model (which can readily be expanded to incorporate covariates and variable but predetermined thresholds) and let $u$ be a constant threshold,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$C(t) = \beta_1 I_{B(t) \gt u} + \epsilon$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;with random deviations $\epsilon$ (which I won't bother to index; you know the drill).  Here, $I$ is the indicator function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$B(t+1) = B(t) - \beta_2 (C(t) - C(t-1)) + \delta$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and again $\delta$ represents random (independent) deviations.  I'm stuck here because you haven't specified more precisely just &lt;em&gt;how&lt;/em&gt; B changes in response to a change in C; I have merely provided one possible interpretation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$A(t) = \beta_3 B(t) + \beta_4 C(t) + \beta_5 + \gamma$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;with independent random deviations $\gamma$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The presence of that indicator function in the first formula is problematic: it makes this a nonlinear problem.  However, this seems to be an essential feature of the situation; I would be loth to ignore it in the name of simplicity or ease of calculation (although both are important considerations).  The lags $B(t+1) - B(t)$ and $C(t) - C(t-1)$ also point towards autoregressive models, another complication.  Because of these issues, the most tractable approach might be with Bayesian techniques: parameterize the distributions of $\epsilon$, $\delta$, and $\gamma$, provide priors for those parameters and the $\beta$s, and let the machinery (&lt;em&gt;e.g.&lt;/em&gt;, WinBUGS or RBUGS) roll.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-10-14T14:21:48.683" />
  <row Id="3607" PostTypeId="2" ParentId="3599" CreationDate="2010-10-14T14:23:57.400" Score="1" ViewCount="" Body="&lt;p&gt;Not easy at all. This starts to sound like the sort of thing that &lt;a href=&quot;http://en.wikipedia.org/wiki/James_Robins&quot; rel=&quot;nofollow&quot;&gt;Jamie Robins&lt;/a&gt; and colleagues have done a lot of work on. To quote the start of the abstract of one of their papers:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;&lt;em&gt;In observational studies with exposures or treatments that vary over time, standard approaches for adjustment of confounding are biased when there exist time-dependent confounders that are also affected by previous treatment.&lt;/em&gt;&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Robins JM, Hernán MA, Brumback B.  Marginal Structural Models and Causal Inference in Epidemiology.  &lt;em&gt;Epidemiology&lt;/em&gt; 2000;11:550-560. &lt;a href=&quot;http://www.jstor.org/stable/3703997&quot; rel=&quot;nofollow&quot;&gt;http://www.jstor.org/stable/3703997&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In their example, low CD4 count means you're more likely to get anti-HIV drugs, which (hopefully) increase your CD4 count. Sounds very much like your setting. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There's been a lot of work in this area recently (i.e. more recently than that paper), and it's not easy to get to grips with from the journal papers. &lt;a href=&quot;http://www.hsph.harvard.edu/faculty/miguel-hernan/causal-inference-book/&quot; rel=&quot;nofollow&quot;&gt;Hernán &amp;amp; Robins are writing a book&lt;/a&gt;, which should help a lot -- it's not finished yet, but there's a draft of the first 10 chapters available.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-10-14T14:23:57.400" CommentCount="1" />
  <row Id="3608" PostTypeId="2" ParentId="2271" CreationDate="2010-10-14T14:55:28.180" Score="8" ViewCount="" Body="&lt;p&gt;I recently came upon &lt;a href=&quot;http://www.cbs.dtu.dk/~eklund/beeswarm/&quot; rel=&quot;nofollow&quot;&gt;the beeswarm package&lt;/a&gt;, that bears some similarity.  &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The bee swarm plot is a&#xA;  one-dimensional scatter plot like&#xA;  &quot;stripchart&quot;, but with closely-packed,&#xA;  non-overlapping points.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Here's an example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  library(beeswarm)&#xA;  beeswarm(time_survival ~ event_survival, data = breast,&#xA;    method = 'smile',&#xA;    pch = 16, pwcol = as.numeric(ER),&#xA;    xlab = '', ylab = 'Follow-up time (months)',&#xA;    labels = c('Censored', 'Metastasis'))&#xA;  legend('topright', legend = levels(breast$ER),&#xA;    title = 'ER', pch = 16, col = 1:2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://www.cbs.dtu.dk/~eklund/beeswarm/beeswarm_example_03.png&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-10-14T14:55:28.180" CommentCount="2" />
  <row Id="3609" PostTypeId="1" CreationDate="2010-10-14T17:57:07.050" Score="2" ViewCount="218" Body="&lt;p&gt;It seems to me that rather than using non robust&#xA;estimation methods with  robust standard errors it would be better to use&#xA;robust estimation from the outset. I wonder what other people think.&lt;/p&gt;&#xA;" OwnerUserId="1585" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-14T17:59:07.143" LastActivityDate="2010-10-15T04:27:41.783" Title="Robust standard errors for panel data vs robust estimation for panel data" Tags="&lt;robust&gt;&lt;panel-data&gt;" AnswerCount="2" FavoriteCount="2" />
  <row Id="3610" PostTypeId="2" ParentId="3595" CreationDate="2010-10-14T18:00:23.600" Score="3" ViewCount="" Body="&lt;p&gt;We trained 3.5M observations and 44 features using 64-bit R on an EC2 instance with 32GB ram and 4 cores. We used random forests and it worked well. Note that we had to preprocess/manipulate the data before training. &lt;/p&gt;&#xA;" OwnerUserId="616" LastActivityDate="2010-10-14T18:00:23.600" CommunityOwnedDate="2010-10-14T18:00:23.600" />
  <row Id="3611" PostTypeId="1" AcceptedAnswerId="3612" CreationDate="2010-10-14T18:40:10.550" Score="4" ViewCount="303" Body="&lt;p&gt;Is X (hazard) variable in Cox proportional hazard regression analysis always has to be time? If not, could you provide an example, please?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can age of cancer patient be a hazard variable? How it can be interpreted? Risk of getting cancer at certain age? Would Cox regression be a legitimate analysis to study association between gene expression and age?&lt;/p&gt;&#xA;" OwnerUserId="1586" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-14T19:09:51.113" LastActivityDate="2010-10-15T12:01:27.760" Title="Cox regression and time scale" Tags="&lt;regression&gt;&lt;survival&gt;&lt;hazard&gt;" AnswerCount="4" />
  <row Id="3612" PostTypeId="2" ParentId="3611" CreationDate="2010-10-14T19:05:25.133" Score="4" ViewCount="" Body="&lt;p&gt;Usually, age at baseline is used as a covariate (because it is often associated to disease/death), but it can be used as your time scale as well (I think it is used in some longitudinal studies, because you need to have enough people at risk along the time scale, but I can't remember actually -- just found these slides about &lt;a href=&quot;http://staff.pubhealth.ku.dk/~bxc/SPE.2002/Slides/survival.pdf&quot; rel=&quot;nofollow&quot;&gt;Analysing cohort studies assuming a continuous time scale&lt;/a&gt; which talk about cohort studies). In the interpretation, you should replace event time by age, and you might include age at diagnosis as a covariate. This would make sense when you study age-specific mortality of a particular disease (as illustrated in these &lt;a href=&quot;http://www.dina.dk/~torbenm/DINA/survival/L4.pdf&quot; rel=&quot;nofollow&quot;&gt;slides&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maybe this article is interesting since it contrasts the two approaches, time-on-study vs. chronological age: &lt;a href=&quot;http://stat.fsu.edu/techreports/M996.pdf&quot; rel=&quot;nofollow&quot;&gt;Time Scales in Cox Model: Effect of Variability Among Entry Ages on Coefficient Estimates&lt;/a&gt;. Here is another paper:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cheung, YB, Gao, F, and Khoo, KS (2003). &lt;a href=&quot;http://www.jclinepi.com/article/S0895-4356%2802%2900536-X/abstract&quot; rel=&quot;nofollow&quot;&gt;Age at diagnosis and the choice of survival analysis methods in cancer epidemiology&lt;/a&gt;. &lt;em&gt;Journal of Clinical Epidemiology&lt;/em&gt;, 56(1), 38-43.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But there are certainly better papers.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-10-14T19:05:25.133" CommentCount="3" />
  <row Id="3613" PostTypeId="2" ParentId="3611" CreationDate="2010-10-14T19:10:51.673" Score="5" ViewCount="" Body="&lt;p&gt;No, it doesn't always have to be time.  Many censored responses can be modeled with survival analysis techniques.  In his book &lt;em&gt;Nondetects and Data Analysis&lt;/em&gt;, Dennis Helsel advocates using the negative of a concentration in place of time (in order to cope with nondetects, which when negated become right-censored values).  A &lt;a href=&quot;http://www.practicalstats.com/nada/nada/downloads_files/NADAforR_Examples.pdf&quot; rel=&quot;nofollow&quot;&gt;synopsis&lt;/a&gt; is available on the Web (pdf format) and an R package, &lt;a href=&quot;http://water.usgs.gov/software/NADA_for_R/&quot; rel=&quot;nofollow&quot;&gt;NADA&lt;/a&gt;, implements this.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-10-14T19:10:51.673" CommentCount="6" />
  <row Id="3614" PostTypeId="1" CreationDate="2010-10-14T19:23:39.887" Score="7" ViewCount="606" Body="&lt;p&gt;I want to calculate the probability distribution for the total of a combination of dice. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I remember that the probability of is the number of combinations that total that number over the total number of combinations (assuming the dice have a uniform distribution). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are the formulas for&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The number of combinations total&lt;/li&gt;&#xA;&lt;li&gt;The number of combinations that total a certain number&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="1456" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-10-14T22:40:19.860" LastActivityDate="2010-10-19T23:38:01.513" Title="How to easily determine the results distribution for multiple dice?" Tags="&lt;probability&gt;&lt;dice&gt;" AnswerCount="6" />
  <row Id="3615" PostTypeId="2" ParentId="3614" CreationDate="2010-10-14T19:58:44.760" Score="4" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Approximate Solution&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I explained the exact solution earlier (see below). I will now offer an approximate solution which may suit your needs better. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$X_i$ be the outcome of a roll of a $s$ faced dice where $i=1, ... n$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$S$ be the total of all $n$ dice.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\bar{X}$ be the sample average.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By definition, we have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\bar{X} = \frac{\sum_iX_i}{n}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\bar{X} = \frac{S}{n}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The idea now is to visualize the process of observing ${X_i}$ as the outcome of throwing the same dice $n$ times instead of as outcome of throwing $n$ dice. Thus, we can invoke the central limit theorem (ignoring technicalities associated with going from discrete distribution to continuous), we have as $n \rightarrow \infty$:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\bar{X} \sim N(\mu, \sigma^2/n)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\mu = (s+1)/2$ is the mean of the roll of a single dice and&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\sigma^2 = (s^2-1)/12$ is the associated variance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The above is obviously an approximation as the underlying distribution $X_i$ has discrete support. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$S = n \bar{X}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, we have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$S \sim N(n \mu, n \sigma^2)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Exact Solution&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;a href=&quot;http://en.wikipedia.org/wiki/Dice#Probability&quot; rel=&quot;nofollow&quot;&gt;wiki&lt;/a&gt; has a brief explanation as how to calculate the required probabilities. I will elaborate a bit more as to why the wiki's explanation makes sense. To the extent possible I have used similar notation as the wiki.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose that you have $n$ dice each with $s$ faces and you want to compute the probability that a single roll of all $n$ dice the total adds up to $k$. The approach is as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Define:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$F_{s,n}(k)$: Probability that you get a total of $k$ on a single roll of $n$ dices with $s$ faces.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By definition, we have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$F_{s,1}(k) = \frac{1}{s}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The above states that if you just have one dice with $s$ faces the probability of obtaining a total $k$ between 1 and s is the familiar $\frac{1}{s}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider the situation when you roll two dice: You can obtain a sum of $k$ as follows: The first roll is between 1 to $k-1$ and the corresponding roll for the second one is between $k-1$ to $1$. Thus, we have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$F_{s,2}(k) = \sum_{i=1}^{i=k-1}{F_{s,1}(i) F_{s,1}(k-i)}$  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now consider a roll of three dice: You can get a sum of $k$ if you roll a 1 to $k-2$ on the first dice and the &lt;em&gt;sum&lt;/em&gt; on the remaining two dice is between $k-1$ to $2$. Thus, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$F_{s,3}(k) = \sum_{i=1}^{i=k-2}{F_{s,1}(i) F_{s,2}(k-i)}$  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Continuing the above logic, we get the recursion equation:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$F_{s,n}(k) = \sum_{i=1}^{i=k-n+1}{F_{s,1}(i) F_{s,n-1}(k-i)}$  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;See the wiki link for more details.&lt;/p&gt;&#xA;" LastActivityDate="2010-10-14T21:33:02.707" CommentCount="2" />
  <row Id="3616" PostTypeId="1" AcceptedAnswerId="3617" CreationDate="2010-10-14T20:17:42.653" Score="2" ViewCount="594" Body="&lt;p&gt;I have a simple time series with 5-10 data points per data set at regular intervals.  I am wondering what is the best way to determine whether two data sets are different.  Should i try t-tests on each data point, or look at the area under the curves or is there some kind of multivariate model that would work better?&lt;/p&gt;&#xA;" OwnerUserId="1327" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-10-15T02:50:22.197" LastActivityDate="2010-11-05T14:54:01.720" Title="What is the best statistical test for a time series?" Tags="&lt;time-series&gt;&lt;statistical-significance&gt;" AnswerCount="3" CommentCount="5" FavoriteCount="1" />
  <row Id="3617" PostTypeId="2" ParentId="3616" CreationDate="2010-10-14T21:14:33.157" Score="4" ViewCount="" Body="&lt;p&gt;You will need to specify precisely what you mean by &quot;different&quot;. You will also need to specify what assumptions you are willing to make about the serial correlation structure within each time series. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;With t-tests, you are comparing the &lt;em&gt;mean&lt;/em&gt; of each group and you are assuming that the groups consist of independent observations with equal variances (the latter is sometimes relaxed). When testing time series, the assumption of independence is usually not reasonable, but then you need to replace it with a specified correlation structure -- e.g., you might assume that the time series follow AR(1) processes with equal autocorrelation. Consequently, even comparing the means of two or more time series is considerably more difficult than with independent data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would carefully specify what assumptions I was willing to make about each time series, and what I was wishing to compare, and then use a parametric bootstrap (based on the assumed model) to carry out the test.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-10-14T21:14:33.157" />
  <row Id="3618" PostTypeId="2" ParentId="3614" CreationDate="2010-10-14T22:29:58.243" Score="3" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Exact solutions&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The number of combinations in $n$ throws is of course $6^n$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These calculations are most readily done using the probability generating function for one die,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p(x) = x + x^2 + x^3 + x^4 + x^5 + x^6 = x \frac{1-x^6}{1-x}.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Actually this is $6$ times the pgf--I'll take care of the factor of $6$ at the end.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The pgf for $n$ rolls is $p(x)^n$.  We can calculate this fairly directly--it's not a closed form but it's a useful one--using the Binomial Theorem:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p(x)^n = x^n (1 - x^6)^n (1 - x)^{-n}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$= x^n (\sum_{k=0}^{n} {n \choose k} (-1)^k x^{6k} )( \sum_{j=0}^{\infty} {-n \choose j} (-1)^j x^j).$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The number of ways to obtain a sum equal to $m$ on the dice is the coefficient of $x^m$ in this product, which we can isolate as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\sum_{6k + j = m - n} {n \choose k}{-n \choose j}(-1)^{k+j}.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The sum is over all nonnegative $k$ and $j$ for which $6k + j = m - n$; it therefore is finite and has only about $(m-n)/6$ terms.  For example, the number of ways to total $m = 14$ in $n = 3$ throws is a sum of just two terms, because $11 = 14-3$ can be written only as $6 \cdot 0 + 11$ and $6 \cdot 1 + 5$:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$-{3 \choose 0} {-3 \choose 11} + {3 \choose 1}{-3 \choose 5}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$= 1 \frac{(-3)(-4)\cdots(-13)}{11!} + 3 \frac{(-3)(-4)\cdots(-7)}{5!}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$= \frac{1}{2} 12 \cdot 13 - \frac{3}{2} 6 \cdot 7 = 15.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(You can also be clever and note that the answer will be the same for $m = 7$ by the symmetry 1 &amp;lt;--&gt; 6, 2 &amp;lt;--&gt; 5, and 3 &amp;lt;--&gt; 4 and there's only one way to expand $7 - 3$ as $6 k + j$; namely, with $k = 0$ and $j = 4$, giving&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ {3 \choose 0}{-3 \choose 4} = 15 \text{.)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The probability therefore equals $15/6^3$ = $5/36$, about 14%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By the time this gets painful, the Central Limit Theorem provides good approximations (at least to the central terms where $m$ is between $\frac{7 n}{2} - 3 \sqrt{n}$ and $\frac{7 n}{2} + 3 \sqrt{n}$: on a relative basis, the approximations it affords for the tail values get worse and worse as $n$ grows large).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I see that this formula is given in the Wikipedia article Srikant references but no justification is supplied nor are examples given.  If perchance this approach looks too abstract, fire up your favorite computer algebra system and ask it to expand the $n^{\text{th}}$ power of $x + x^2 + \cdots + x^6$: you can read the whole set of values right off.  &lt;em&gt;E.g.&lt;/em&gt;, a Mathematica one-liner is&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;With[{n=3}, CoefficientList[Expand[(x + x^2 + x^3 + x^4 + x^5 + x^6)^n], x]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-10-17T21:25:11.627" LastActivityDate="2010-10-17T21:25:11.627" CommentCount="4" />
  <row Id="3619" PostTypeId="2" ParentId="3595" CreationDate="2010-10-15T01:03:14.220" Score="5" ViewCount="" Body="&lt;p&gt;There is &lt;a href=&quot;http://www.stat.purdue.edu/~sguha/rhipe/&quot; rel=&quot;nofollow&quot;&gt;RHIPE&lt;/a&gt; package (R-Hadoop integration). It is can make it very easy (with exceptions) to analyze large amounts of data in R.&lt;/p&gt;&#xA;" OwnerUserId="1307" LastActivityDate="2010-10-15T01:03:14.220" CommunityOwnedDate="2010-10-15T01:03:14.220" CommentCount="2" />
  <row Id="3620" PostTypeId="2" ParentId="3609" CreationDate="2010-10-15T01:22:06.443" Score="0" ViewCount="" Body="&lt;p&gt;Hello Dave,&#xA;I don't know if I understand you correctly, but still I will give it a shot.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think robust estimation from the outset would be better in most of the cases. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Reason:&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you estimation is not robust, outliers might severely affect your estimate. Still, in general, you will be far the true value. This may be also looked as the confidence interval not containing the true value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In picture form: (This can happen, outliers cause the estimate to be very far from truth). If the estimate was found by robust methods, it would have been closer to the True Value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;----------||------------------------[------------------Not Robust Est---------------------]-&lt;/p&gt;&#xA;&#xA;&lt;p&gt;........True Value..................&amp;lt;--------CI found by Robust Method--------------&gt;&lt;/p&gt;&#xA;" OwnerUserId="1307" LastActivityDate="2010-10-15T01:22:06.443" />
  <row Id="3621" PostTypeId="2" ParentId="3611" CreationDate="2010-10-15T02:06:12.473" Score="2" ViewCount="" Body="&lt;p&gt;On the age-scale vs. time-scale issue, chl has some good references and captures the essentials -- in particular, the requirement that the at-risk set contain sufficient subjects from all ages as would arise in a longitudinal study.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would only note that there is no general consensus around this yet, but there is some literature to suggest that age should be preferred as the time scale in certain cases.  In particular, if you have a situation where time doesn't accumulate in the same way for all subjects, for example due to exposure to some toxic material, then age may be more appropriate.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, you can handle that specific example on a time-scale Cox PH model by using age as a time varying covariate -- rather than a fixed covariate at start time.  You need to think about the mechanism behind your object of study to figure out which time scale is more appropriate.  Sometimes it's worth fitting both models to existing data to see if discrepancies arise and how they might be explained before designing your new study.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, the obvious difference in analyzing the two is that on an age-scale, the interpretation of survival is with respect to an absolute scale (age), whereas on a time-scale, it's relative to the start/entry date of the study.&lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-10-15T02:06:12.473" />
  <row Id="3622" PostTypeId="2" ParentId="3609" CreationDate="2010-10-15T02:40:18.180" Score="6" ViewCount="" Body="&lt;p&gt;They're robust with respect to different things.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you use robust regression to obtain an estimate of fixed effect in panel data, then you're computing an estimate that's resistant to outliers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you use robust standard errors for your OLS estimator, it's because you suspect that the assumption behind your error model is violated.  For example, in panel data, your errors may be autocorrelated and not iid, and your robust standard error offers protection against such phenomena.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But, robust standard errors don't guard against outliers, and robust regression doesn't necessarily account for autocorrelation.  Though I think there are techniques to provide robust standard errors for robust estimators, you would only use these when both conditions are true: your data have outliers &lt;em&gt;and&lt;/em&gt; your error model assumption is violated.  &lt;/p&gt;&#xA;" OwnerUserId="251" LastEditorUserId="251" LastEditorDisplayName="" LastEditDate="2010-10-15T04:27:41.783" LastActivityDate="2010-10-15T04:27:41.783" />
  <row Id="3623" PostTypeId="1" AcceptedAnswerId="3626" CreationDate="2010-10-15T04:44:29.807" Score="2" ViewCount="144" Body="&lt;p&gt;I have a logistic regression (in SAS, for reference) with continuous and categorical predictors (with reference coding), and an interaction term between one of each type (assume for now that the categorical variable in question has three response levels, reference coded to $c_1$ and $c_2$):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$logit(p) = a + (continuous terms) + (categorical terms) + b_1 (c_1 x) + b_2 (c_2 x)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $b_1$ and $b_2$ are the estimated coefficients from my code. From this I can obviously get an expression for the probability $p$ of my outcome. I want to estimate the average p by plugging in the means of the continuous terms and the proportions of the categorical terms. But what do I do with the interaction term(s)? Do I set $(c_i x) = mean(x)$? Or do I set it to $proportion(c_i)mean(x)$?&lt;/p&gt;&#xA;" OwnerUserId="1144" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-10-15T08:27:19.977" LastActivityDate="2010-10-15T08:27:19.977" Title="Plugging in mean values/proportions to a logistic regression with continuous-discrete interaction" Tags="&lt;logistic-regression&gt;&lt;categorical-data&gt;&lt;fitting&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="3625" PostTypeId="2" ParentId="3614" CreationDate="2010-10-15T06:19:31.640" Score="3" ViewCount="" Body="&lt;p&gt;There's a very neat way of computing the combinations or probabilities in a spreadsheet (such as excel) that computes the convolutions directly. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'll do it in terms of probabilities and illustrate it for six sided dice but you can do it for dice with any number of sides (including adding different ones).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(btw it's also easy in something like R or matlab that will do convolutions)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Start with a clean sheet, in a few columns, and move down a bunch of rows from the top (more than 6).&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;put the value 1 in a cell. That's the probabilities associated with 0 dice. put a 0 to its left; that's the value column - continue down from there with 1,2,3 down as far as you need.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;move one column to the right and down a row from the '1'. enter the formula &quot;=average(&quot; then left-arrow up-arrow (to highlight the cell with 1 in it), hit &quot;:&quot; (to start entering a range) and then up-arrow 5 times, followed by &quot;)&quot; and press Enter - so you end up with a formula like =average(c4:c9)  (where here c9 is the cell with the 1 in it). Copy the formula and paste it to the 5 cells below it. They should each contain 0.16667 (ish).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Don't type anything into the empty cells these formulas refer to!&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;move down 1 and to the right 1 from the top of that column of values and paste another 11 values. These will be the probabilities for two dice. It doesn't matter if you paste a few too many, you'll just get zeroes.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;repeat step 3 for the next column for three dice, and again for four, five, etc dice.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;If you want combination counts instead of probabilities, use &quot;sum&quot; in place of &quot;average&quot;&lt;/p&gt;&#xA;" OwnerUserId="805" LastActivityDate="2010-10-15T06:19:31.640" />
  <row Id="3626" PostTypeId="2" ParentId="3623" CreationDate="2010-10-15T06:39:37.757" Score="3" ViewCount="" Body="&lt;p&gt;Regardless of the interaction term, this procedure isn't going to estimate the average p, because logit is a nonlinear function so the mean of the logit isn't the same as the logit of the mean. If you want to calculate the expected proportion of positive outcomes in a sample, the easiest way is to calculated the predicted p for each individual and average them.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-10-15T06:39:37.757" CommentCount="2" />
  <row Id="3627" PostTypeId="2" ParentId="3623" CreationDate="2010-10-15T07:35:14.613" Score="4" ViewCount="" Body="&lt;p&gt;The predicted grand average (see onestop's answer) may not be all that informative - after all, you are fitting a model to understand systematic deviations from it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can predict your p for any setting of your predictors. Given that you are interested in the effects of the predictors, it would make sense to look at what happens when your continuous x takes values at the first, second and third quartile of the values in your sample, while the categorical predictor varies over the categories you have.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then again, you sound like you have more than one continuous and more than one categorical variable, in which case an enumeration as I proposed in the previous paragraph becomes unintelligible. Better look at and report p for &quot;interesting&quot; or &quot;relevant&quot; predictor settings.&lt;/p&gt;&#xA;" OwnerUserId="1352" LastActivityDate="2010-10-15T07:35:14.613" />
  <row Id="3628" PostTypeId="1" AcceptedAnswerId="3629" CreationDate="2010-10-15T10:48:41.760" Score="2" ViewCount="482" Body="&lt;p&gt;From the output of a logistic regression in JMP, I read about two binary variables:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Var1 estimate -0.1007384&#xA;Var2 estimate  0.21528927&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and then&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Odds ratio for Var1 lev1/lev2 1.2232078 reciprocal 0.8175225&#xA;Odds ratio for Var2 lev1/lev2 0.6501329 reciprocal 1.5381471&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now I obtain &lt;code&gt;1.2232078&lt;/code&gt; as &lt;code&gt;exp(2*0.1007384)&lt;/code&gt;, and similarly for the other odds ratio.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, my question is: why I have to multiply by two? &#xA;Is the relation between a coefficient c and its odds ratio r equal to r=exp(c)? Isn't it? &lt;/p&gt;&#xA;" OwnerUserId="1219" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-15T14:42:37.287" LastActivityDate="2010-10-15T14:42:37.287" Title="Relation between logistic regression coefficient and odds ratio in JMP" Tags="&lt;logistic-regression&gt;&lt;odds-ratio&gt;&lt;jmp&gt;" AnswerCount="1" CommentCount="0" />
  <row Id="3629" PostTypeId="2" ParentId="3628" CreationDate="2010-10-15T10:54:40.603" Score="5" ViewCount="" Body="&lt;p&gt;Ok, I drop a quick response. Your idea is correct in that the regression coefficient is the log of the OR. More precisely, if $b$ is your regression coefficient, $\exp(b)$ is the odds ratio corresponding to &lt;em&gt;a one unit change&lt;/em&gt; in your variable. So, to get back to the adjusted odds, you need to know what are the internal coding convention for your factor levels. Usually, for a binary variable it is 0/1 or 1/2. But if it happens that your levels are represented as -1/+1 (which I suspect here), then you have to multiply the regression coefficient by 2 when exponentiating. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The same would apply if you were working with a continuous variable, like age, and want to express the odds for 5 years ($\exp(5b)$) instead of 1 year ($\exp(b)$).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update:&lt;/strong&gt; I just found this about &lt;a href=&quot;http://www.jmp.com/support/faq/jmp2079.shtml&quot; rel=&quot;nofollow&quot;&gt;JMP coding for nominal variables&lt;/a&gt; (version &amp;lt; 7).&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-15T11:08:31.683" LastActivityDate="2010-10-15T11:08:31.683" CommentCount="1" />
  <row Id="3630" PostTypeId="1" CreationDate="2010-10-15T11:00:45.113" Score="3" ViewCount="102" Body="&lt;p&gt;Hi All,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is anybody aware of advanced variance reduction techniques (i.e. importance sampling or stratified sampling) for the simulation of &lt;a href=&quot;http://en.wikipedia.org/wiki/CIR_process&quot; rel=&quot;nofollow&quot;&gt;Cox-Ingersoll-Ross process&lt;/a&gt;? Ideally I would like to see a concrete implementation if available...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regards&lt;/p&gt;&#xA;" OwnerUserId="1443" LastEditorUserId="1443" LastEditorDisplayName="" LastEditDate="2010-10-18T15:03:36.683" LastActivityDate="2010-10-18T15:03:36.683" Title="CIR Process-Variance reduction" Tags="&lt;r&gt;&lt;mcmc&gt;&lt;stochastic-processes&gt;" FavoriteCount="1" />
  <row Id="3631" PostTypeId="2" ParentId="3611" CreationDate="2010-10-15T12:01:27.760" Score="3" ViewCount="" Body="&lt;p&gt;Per the OP's request, heres another application I have seen survival analysis used in a spatial context (although obviously different than measuring environmental substances &lt;a href=&quot;http://stats.stackexchange.com/questions/3611/cox-regression-and-time-scale/3613#3613&quot;&gt;mentioned&lt;/a&gt; by whuber) is modeling the distance between events in space. Heres one example in &lt;a href=&quot;http://goo.gl/mtk8&quot; rel=&quot;nofollow&quot;&gt;criminology&lt;/a&gt; and here is one in &lt;a href=&quot;http://dx.doi.org/10.1016/S0277-9536%2899%2900349-4&quot; rel=&quot;nofollow&quot;&gt;epidemiology&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reasoning behind using survival analysis to measure the distance between events is not per say an issue of censoring (although censoring can definately occur in a spatial context), it is more so because of the similar distributions between time to event characteristics and distance between events characteristics (i.e. they both have similar types of error structures (frequently distance decay) that violate OLS and so the non-parametric solutions are ideal for both).&lt;/p&gt;&#xA;" OwnerUserId="1036" LastActivityDate="2010-10-15T12:01:27.760" CommentCount="1" />
  <row Id="3632" PostTypeId="1" CreationDate="2010-10-15T14:14:33.220" Score="2" ViewCount="151" Body="&lt;p&gt;Suppose we need to take an action on a population with income (x) more than $5,000. Income is not observed directly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should we use logistic regression to estimate x, or should we use logistic regression to estimate the probability of x&gt;5000 directly? (What are the drawbacks/advantage of the methods?)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: Yes - by logistic I meant logistic regression. The other variables I have are Financial history, Demograpgic and Credit Bureau variables. For example, the balance, utilization, gender, # cars, house owned or not, external card balance, external # bad debts, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="994" LastEditorUserId="994" LastEditorDisplayName="" LastEditDate="2010-10-16T04:50:04.427" LastActivityDate="2010-10-17T04:02:18.357" Title="Should we regress x or use logistic regression on x&gt;5000" Tags="&lt;regression&gt;&lt;logistic-regression&gt;" AnswerCount="2" CommentCount="3" />
  <row Id="3633" PostTypeId="2" ParentId="3616" CreationDate="2010-10-15T15:08:03.417" Score="1" ViewCount="" Body="&lt;p&gt;Maybe repeated measures anova is what you want. It allows you to compare the subjects (inter subject factors) while taking the correlated structure of the &quot;time series&quot; per subject (intra subject factor). It is an easy but dated method and can be found in the context of &quot;general linear models&quot;, it needs some additional features (e.g. sphericity). Another way could be mixed linear models which allow for more general correlations structures (even AR(1) like Rob suggested) and unbalanced data.&lt;/p&gt;&#xA;" OwnerUserId="1573" LastActivityDate="2010-10-15T15:08:03.417" />
  <row Id="3634" PostTypeId="1" CreationDate="2010-10-15T15:42:56.067" Score="6" ViewCount="458" Body="&lt;p&gt;Hi!&#xA;I have to analyze a factorial design with five factors (one of them nested in another one) and numeric responses. I would like to perform a nonparametric ANOVA, but of course I can't use neither Kruskall Wallis nor Friedman test (I have replicated measures). Is there a command or a code in R that could help me?&#xA;Thank you!&#xA;Stefania&lt;/p&gt;&#xA;" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-10-15T16:41:33.940" LastActivityDate="2011-03-31T16:14:14.503" Title="Multi-way nonparametric anova" Tags="&lt;r&gt;&lt;anova&gt;&lt;nonparametric&gt;" AnswerCount="5" CommentCount="1" FavoriteCount="3" />
  <row Id="3635" PostTypeId="2" ParentId="3419" CreationDate="2010-10-15T15:50:12.507" Score="2" ViewCount="" Body="&lt;p&gt;I might be misunderstanding your goals here, but to me it woulds like a &lt;a href=&quot;http://en.wikipedia.org/wiki/Multidimensional_scaling&quot; rel=&quot;nofollow&quot;&gt;multi-dimensional scaling&lt;/a&gt; (MDS) problem.  I've never used MDS myself, but my sense is that it should allow you to derive a global measure of similarity as well as dimensional measures of similarity. My memory is that it is able to handle both continuous items (e.g. pulse rate) and nominal items (e.g. Gender) which seems like it would be an important consideration for what you are trying to do.&lt;/p&gt;&#xA;" OwnerUserId="196" LastActivityDate="2010-10-15T15:50:12.507" />
  <row Id="3636" PostTypeId="1" CreationDate="2010-10-15T15:53:01.893" Score="0" ViewCount="131" Body="&lt;p&gt;I have a sample set of values that were taken over a period of time. However, the delta time between each sample is different.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do I need to account for the different time deltas in the std-dev?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is std-dev even appropriate for this kind of data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;hr&gt;&#xA;More info...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data are temperature samples.&lt;br&gt;&#xA;The time range is from 1 hour to multiple days.&lt;/p&gt;&#xA;" OwnerUserId="1595" LastEditorUserId="1595" LastEditorDisplayName="" LastEditDate="2010-10-15T16:14:01.973" LastActivityDate="2010-10-18T12:24:04.177" Title="How to calculate the standard deviation on a sample set with irregular time periods" Tags="&lt;standard-deviation&gt;" AnswerCount="2" CommentCount="4" FavoriteCount="0" />
  <row Id="3637" PostTypeId="2" ParentId="3636" CreationDate="2010-10-15T16:02:52.387" Score="1" ViewCount="" Body="&lt;p&gt;Yes, you do need to account for the irregularity of the time series because volatility scales with time.  Depending upon the distribution and independence assumptions, &lt;a href=&quot;http://ideas.repec.org/p/fmg/fmgdps/dp439.html&quot; rel=&quot;nofollow&quot;&gt;sometimes a &quot;square root of time&quot; rule can be appropriate&lt;/a&gt;.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this data sampled irregularly intraday or across a longer time period?  What kind of data is it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For dealing with high-frequency financial data, you can apply a &lt;a href=&quot;http://www.ssc.upenn.edu/~fdiebold/papers/paper29/temp.pdf&quot; rel=&quot;nofollow&quot;&gt;realized volatility&lt;/a&gt; measure, which is available in R in the &lt;a href=&quot;http://cran.r-project.org/web/packages/realized/index.html&quot; rel=&quot;nofollow&quot;&gt;realized package&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-10-15T16:02:52.387" />
  <row Id="3638" PostTypeId="2" ParentId="3634" CreationDate="2010-10-15T16:27:19.637" Score="4" ViewCount="" Body="&lt;p&gt;Tukey's Median Polish is implemented in R as medpolish {stats}.  See Chapter 6 in &lt;a href=&quot;http://www.stats.ox.ac.uk/pub/MASS4/VR4stat.pdf&quot; rel=&quot;nofollow&quot;&gt;Venables and Ripley&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-31T16:14:14.503" LastActivityDate="2011-03-31T16:14:14.503" CommentCount="2" />
  <row Id="3639" PostTypeId="2" ParentId="3383" CreationDate="2010-10-15T16:34:25.457" Score="2" ViewCount="" Body="&lt;p&gt;The one-way ANOVA approach you mention sounds fine to me.  Sure the individual change scores aren't going to be the &quot;true change&quot; by any means, but they are better than nothing.  If anything the resulting variance in the model should be over estimated as a consequence of this procedure.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In R the easiest way to do ANOVA in simple designs (IMO) is to use ezANOVA (package ez).  E.g. ezANOVA(data,dv=.(WeightChange),sid=.(PseudoSubjID),between=.(Time))&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can't quite say anything about instantiation, but another approach might be to find the optimal set of paired scores such that the difference between scores is minimized and then treat that pairing as if it is the true pairing.  I want to say that approach should be conservative, minimizing the differences between t0 and t1, but your mileage may vary.&lt;/p&gt;&#xA;" OwnerUserId="196" LastActivityDate="2010-10-15T16:34:25.457" />
  <row Id="3640" PostTypeId="1" AcceptedAnswerId="3644" CreationDate="2010-10-15T16:45:49.320" Score="4" ViewCount="267" Body="&lt;p&gt;&lt;strong&gt;Problem:&lt;/strong&gt;&#xA;I am parameterizing distributions for use as a priors and data in a Bayesian meta-analysis. The data are provided in the literature as summary statistics, almost exclusively assumed to be normally distributed (although none of the variables can be &amp;lt; 0, some are ratios, some are mass, and etc.). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have come across two cases for which I have no solution. Sometimes the parameter of interest is the inverse of the data or the ratio of two variables.    &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Examples:&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;the ratio of two normally distributed variables:&#xA;&lt;ul&gt;&#xA;&lt;li&gt;data: mean and sd for percent nitrogen and percent carbon &lt;/li&gt;&#xA;&lt;li&gt;parameter: ratio of carbon to nitrogen.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;the inverse of a normally distributed variable:&#xA;&lt;ul&gt;&#xA;&lt;li&gt;data: mass/area&lt;/li&gt;&#xA;&lt;li&gt;parameter: area/mass&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;My current approach is to use simulation:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;e.g. for a set of percent carbon and nitrogen data with means: xbar.n,c, variance: se.n,c, and sample size: n.n, n.c:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(1)&#xA;per.c &amp;lt;- rnorm(100000, xbar.c, se.c*n.c) # percent C&#xA;per.n &amp;lt;- rnorm(100000, xbar.n, se.n*n.n) # percent N&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to parameterize ratio.cn = perc.c/perc.n&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# parameter of interest&#xA;ratio.cn &amp;lt;- perc.c / perc.n&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then choose the best fit distributions with range $0 \rightarrow \infty$ for my prior &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(MASS)&#xA;dist.fig &amp;lt;- list()&#xA;for(dist.i in c('gamma', 'lognormal', 'weibull')) {&#xA;    dist.fit[[dist.i]] &amp;lt;- fitdist(ratio.cn, dist.i)&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question:&lt;/strong&gt;&#xA;Is this a valid approach? Are there other / better approaches?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update:&lt;/strong&gt; the Cauchy distribution, which is defined as the ratio of two normals with $\mu=0$, has limited utility since I would like to estimate variance. Perhaps I could calculate the variance of a simulation of n draws from a Cauchy?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I did find the following closed-form approximations but I haven't tested to see if they give the same results... &lt;a href=&quot;https://netfiles.uiuc.edu/dlebauer/www/hayya1975nrt.pdf&quot; rel=&quot;nofollow&quot;&gt;Hayya et al, 1975&lt;/a&gt;&#xA;$$\hat{\mu}_{y:x} = \mu_y/mu_x + \sigma^2_x * \mu_y  / \mu_x^3 + cov(x,y) * \sigma^2_x * \sigma^2_y / \mu_x^2$$&#xA;$$\hat{\sigma}^2_{y:x} = \sigma^2_x\times\mu_y / mu_x^4 + \sigma^2_y / mu_x^2 - 2 * cov(x,y) * \sigma^2_x * \sigma^2_y / mu_x^3$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hayya, J. and Armstrong, D. and Gressis, N., 1975. A note on the ratio of two normally distributed variables. Management Science 21: 1338--1341&lt;/p&gt;&#xA;" OwnerUserId="1381" LastEditorUserId="1381" LastEditorDisplayName="" LastEditDate="2010-10-15T19:28:25.393" LastActivityDate="2010-10-15T19:45:31.940" Title="How to parameterize the ratio of two normally distributed variables, or the inverse of one?" Tags="&lt;distributions&gt;&lt;bayesian&gt;&lt;variance&gt;&lt;random-variable&gt;&lt;meta-analysis&gt;" AnswerCount="2" CommentCount="3" FavoriteCount="2" />
  <row Id="3641" PostTypeId="1" CreationDate="2010-10-15T16:50:38.817" Score="3" ViewCount="118" Body="&lt;p&gt;I would like to project the data in this graph for at least 4 or 5 periods. Unfortunately, that won't be possible with a moving average. A regression will result in negative values after the 3rd period. What are my forecasting options?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically, what i'm trying to do, is predict where the boomer hump is gonna be based soley on the population and age per year data&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt; AT work we have lots of image sites blocked, &lt;a href=&quot;http://97.107.136.148/age_dist.jpg&quot; rel=&quot;nofollow&quot;&gt;here's the image&lt;/a&gt; in case you can't see it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT 2&lt;/strong&gt; Updated Image&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/Ad80M.jpg&quot; alt=&quot;State Wide Age Distribution&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="59" LastEditorUserId="59" LastEditorDisplayName="" LastEditDate="2010-10-15T17:25:32.207" LastActivityDate="2010-10-15T19:43:31.437" Title="Forecasting Age distribution" Tags="&lt;time-series&gt;&lt;forecasting&gt;" AnswerCount="1" CommentCount="9" />
  <row Id="3642" PostTypeId="2" ParentId="3634" CreationDate="2010-10-15T16:52:13.963" Score="1" ViewCount="" Body="&lt;p&gt;You might check out the ezBoot() function in the ez package for bootstrapping confidence intervals on your effects of interest.&lt;/p&gt;&#xA;" OwnerUserId="364" LastActivityDate="2010-10-15T16:52:13.963" />
  <row Id="3643" PostTypeId="2" ParentId="3640" CreationDate="2010-10-15T17:05:22.430" Score="0" ViewCount="" Body="&lt;p&gt;Could you not assume that $y^{-1} \sim N(.,.)$ for the inverse of a normal random variable and do the necessary bayesian computation after identifying the appropriate parameters for the normal distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My suggestion below to use the Cauchy does not work as pointed out in the comments by ars and John.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strike&gt;The ratio of two normally random variables follows the &lt;a href=&quot;http://en.wikipedia.org/wiki/Cauchy_distribution&quot; rel=&quot;nofollow&quot;&gt;Cauchy&lt;/a&gt; distribution. You may want to use this idea to identify the parameters of the cauchy that most closely fits the data you have.&lt;/strike&gt;&lt;/p&gt;&#xA;" LastActivityDate="2010-10-15T18:42:03.223" CommentCount="16" />
  <row Id="3644" PostTypeId="2" ParentId="3640" CreationDate="2010-10-15T17:17:18.427" Score="3" ViewCount="" Body="&lt;p&gt;You might want to look at some of the references under the Wikipedia article on &lt;a href=&quot;http://en.wikipedia.org/wiki/Ratio_distribution&quot; rel=&quot;nofollow&quot;&gt;Ratio Distribution&lt;/a&gt;.  It's possible you'll find better approximations or distributions to use.  Otherwise, your approach seems sound.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Update&lt;/em&gt; I think a better reference might be:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.jstor.org/pss/2283145&quot; rel=&quot;nofollow&quot;&gt;Ratios of Normal Variables and Ratios of Sums of Uniform Variables&lt;/a&gt; (Marsaglia, 1965)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;See formulas 2-4 on page 195.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Update 2&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On your updated question regarding variance from a Cauchy -- as John Cook pointed out in the comments, the variance doesn't exist.  So, taking a sample variance simply won't work as an &quot;estimator&quot;.  In fact, you'll find that your sample variance does not converge at all and fluctuates wildly as you keep taking samples.  &lt;/p&gt;&#xA;" OwnerUserId="251" LastEditorUserId="251" LastEditorDisplayName="" LastEditDate="2010-10-15T19:45:31.940" LastActivityDate="2010-10-15T19:45:31.940" CommentCount="4" />
  <row Id="3645" PostTypeId="2" ParentId="3634" CreationDate="2010-10-15T17:43:27.083" Score="1" ViewCount="" Body="&lt;p&gt;The &lt;a href=&quot;http://cc.oulu.fi/~jarioksa/softhelp/vegan.html&quot; rel=&quot;nofollow&quot;&gt;vegan&lt;/a&gt; package implements permutation testing for distance based ANOVA, which should work with multi-way, repeated measures data.  &lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-10-15T17:43:27.083" CommentCount="2" />
  <row Id="3646" PostTypeId="1" AcceptedAnswerId="3648" CreationDate="2010-10-15T19:27:02.353" Score="2" ViewCount="220" Body="&lt;p&gt;I have a univariate data set that's approximately normally distributed. I am happy to assume that the population is normally distributed, and I'd like to estimate the mean and variance of the population.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My textbook suggests (as I understand it) that since my sample size is large (1000's of data points), it is reasonable to take the sample mean and sample variance as my estimates for the population sample/variance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I'm also vaguely aware that regression can be used to fit a model to data. So in the case of my problem, which is a more reasonable approach (and why?):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Just use the sample mean/variance values as estimates for the population mean/variance&lt;/li&gt;&#xA;&lt;li&gt;Fit the data to a normal model and use the calculated mean/variance.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="1598" LastActivityDate="2010-10-15T20:20:29.600" Title="Simple Estimates vs Model for calculating mean and variance of population" Tags="&lt;normal-distribution&gt;" AnswerCount="2" CommentCount="9" FavoriteCount="1" />
  <row Id="3647" PostTypeId="2" ParentId="3641" CreationDate="2010-10-15T19:43:31.437" Score="5" ViewCount="" Body="&lt;p&gt;For demographic forecasting of any quality whatsoever you need to account for birth and death rates and, if possible, migration, breaking them down by gender (at a minimum) and, if possible, by race.  These rates have all changed substantially during your time period and are likely to continue changing in the future.  This information is available on the US Census Bureau's site.  Many states have state-specific mortality tables available.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have performed such projections at state and even census tract levels with the same aim in mind (to support market research analysis for assisted living facilities) and found that a careful projection will differ substantially from purely statistical (demographically ignorant) procedures like moving averages or regression.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-10-15T19:43:31.437" CommentCount="3" />
  <row Id="3648" PostTypeId="2" ParentId="3646" CreationDate="2010-10-15T19:48:34.417" Score="6" ViewCount="" Body="&lt;p&gt;If I understand your question, and you mean using a least squares model of the form $Y=\beta + \epsilon$ where $\epsilon\sim N(0,\sigma^2)$ these two approaches are equivalent. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A simple R example will demonstrate this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#generate pseudo-data&#xA;set.seed(0)&#xA;n &amp;lt;- 1000&#xA;x &amp;lt;- rnorm(n)&#xA;&#xA;# approach 1: calculation    &#xA;sum(x)/n #mean&#xA;mse &amp;lt;- sum((x-mean(x))^2)/n #mse&#xA;se &amp;lt;- sqrt(mse/n) #std error&#xA;&#xA;# approach 2: model&#xA;model &amp;lt;- lm(x~1)&#xA;model$coefficients[1] #mean&#xA;sqrt(sum(model$residuals^2)/n)/sqrt(n) #standard error&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1381" LastEditorUserId="1381" LastEditorDisplayName="" LastEditDate="2010-10-15T20:20:29.600" LastActivityDate="2010-10-15T20:20:29.600" CommentCount="4" />
  <row Id="3649" PostTypeId="2" ParentId="3646" CreationDate="2010-10-15T19:58:01.210" Score="3" ViewCount="" Body="&lt;p&gt;I was about to make the same point as David, except illustrating using Stata rather than R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;&#xA;. summarize length&#xA;&#xA;    Variable |       Obs        Mean    Std. Dev.       Min        Max&#xA;-------------+--------------------------------------------------------&#xA;      length |        74    &lt;b&gt;187.9324&lt;/b&gt;    &lt;b&gt;22.26634&lt;/b&gt;        142        233&#xA;&#xA;. regress length&#xA;&#xA;      Source |       SS       df       MS              Number of obs =      74&#xA;-------------+------------------------------           F(  0,    73) =    0.00&#xA;       Model |           0     0           .           Prob &gt; F      =       .&#xA;    Residual |  36192.6622    73  495.789893           R-squared     =  0.0000&#xA;-------------+------------------------------           Adj R-squared =  0.0000&#xA;       Total |  36192.6622    73  495.789893           Root MSE      =  &lt;b&gt;22.266&lt;/b&gt;&#xA;&#xA;------------------------------------------------------------------------------&#xA;      length |      Coef.   Std. Err.      t    P&gt;|t|     [95% Conf. Interval]&#xA;-------------+----------------------------------------------------------------&#xA;       _cons |   &lt;b&gt;187.9324&lt;/b&gt;   2.588409    72.61   0.000     182.7737    193.0911&#xA;------------------------------------------------------------------------------&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I've added the bolding to highlight that the mean and standard deviation are the same as the estimate of the constant and root mean square error from a linear regression with no covariates.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-10-15T19:58:01.210" />
  <row Id="3650" PostTypeId="1" AcceptedAnswerId="3658" CreationDate="2010-10-15T20:17:13.240" Score="4" ViewCount="249" Body="&lt;p&gt;I have a two part question;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First Part:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have an urn with 20 balls, 2 of those balls are purple, and I pull out 6 balls at random. I witness 100 realizations of this process. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given the observed frequency at which I drew purple balls, how do I determine if I am really pulling balls out at random? Also given that there are 2 purple balls, I have a hunch that if purple balls are pulled out disproprionately, I expect that both purple balls would be pulled out (i.e. I'm more interested in seeing if 2 purple balls are pulled out disproportionately than I am if 1 purple ball is pulled out more frequently than expected).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second Part:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have an urn with a variable number of balls, a variable number of purple balls within that urn, and a variable number of draws. I witness 100 realizations of this process, and I observe in each realization how many balls there were, how many of those balls were purple, and how many balls I drew from the urn.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Same questions; Given the observed frequency at which I drew purple balls, how do I determine if I am really pulling balls out at random? Again I'm more interested to see if higher frequencies of purple balls are disproportionately drawn than I am to see if 1 purple ball being drawn happens more than expected by chance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(I'm open to suggestions for title of question and tags)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Srikant suggested I may need to make distributional assumptions about my variables, which I am willing to do.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lets say the number of balls in the urn is uniform between 20 and 30, the number of purple balls is uniform between 0 and 4, and the number of draws is uniform between 6 and 12.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See my &lt;a href=&quot;http://stats.stackexchange.com/questions/3650/expected-distribution-of-random-draws/3873#3873&quot;&gt;answer&lt;/a&gt; that describes my motivation for asking this question.&lt;/p&gt;&#xA;" OwnerUserId="1036" LastEditorUserId="1036" LastEditorDisplayName="" LastEditDate="2010-10-22T04:58:04.037" LastActivityDate="2010-10-22T16:38:05.127" Title="Expected distribution of random draws" Tags="&lt;distributions&gt;&lt;probability&gt;" AnswerCount="3" CommentCount="1" />
  <row Id="3651" PostTypeId="2" ParentId="3650" CreationDate="2010-10-15T20:23:22.837" Score="2" ViewCount="" Body="&lt;p&gt;First Part: The draws from the urn follow a &lt;a href=&quot;http://en.wikipedia.org/wiki/Hypergeometric_distribution&quot; rel=&quot;nofollow&quot;&gt;hypergeometric distribution&lt;/a&gt; assuming random draws. Any deviation from the theoretical probabilities vis-a-vis the observed frequencies can be evaluated using chi-square tests.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second Part: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$n \sim U(20,30)$ be the total number of balls in the urn&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$p \sim U(0,4)$ be the number of purple balls.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$d \sim U(6,12)$ be the number of draws.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$n_{pi}$ be the number of purple balls we the $i^\mbox{th}$ draw.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(n_{pi}|n,p,d) = \frac{\binom{p}{n_{pi}} \binom{n-p}{d-n_{pi}}}{\binom{n}{d}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can integrate the above probabilities using the priors for $n$, $p$ and $d$ which will give you the expected frequencies to observe purple balls provided you draw them at random. You can then compare the expected frequencies with the observed frequencies to assess if the process is truly random.&lt;/p&gt;&#xA;" LastActivityDate="2010-10-16T00:13:23.233" CommentCount="8" />
  <row Id="3652" PostTypeId="1" CreationDate="2010-10-15T20:24:37.687" Score="4" ViewCount="346" Body="&lt;p&gt;Suppose one has two independent samples from the same population, and different methods were used on the two samples to derive point estimate and confidence intervals. In trivial cases a sensible person would just pool the two samples and use one method to do the analysis, but let's suppose for the moment that different method has to be used due to limitation of one of the sample such as missing data. These two separate analyses would generate independent, equally valid estimates for the population attribute of interest. Intuitively I think there should be a way to properly combine these two estimates, both in terms of point estimate and confidence interval, resulting in a better estimation procedure. My question is what should be the best way to do it? I can imagine a weighted mean of some sort according to the information/sample size in each sample, but what about the confidence intervals? &lt;/p&gt;&#xA;" OwnerUserId="1600" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-10-15T23:18:35.950" LastActivityDate="2010-10-15T23:18:35.950" Title="Combining two confidence intervals/point estimates" Tags="&lt;confidence-interval&gt;&lt;meta-analysis&gt;" AnswerCount="3" />
  <row Id="3653" PostTypeId="1" AcceptedAnswerId="3657" CreationDate="2010-10-15T20:30:05.690" Score="5" ViewCount="302" Body="&lt;p&gt;I have a multivariate regression, which includes interactions. For example, to get the estimate of the treatment effect for the poorest quintile I need to add the coefficients from the treatment regressor to the coefficient from the interaction variable (which interacts treatment and quintile 1). When adding two coefficients from a regression, how does one obtain standard errors? Is it possible to add the standard errors from the two coefficients? What about the t-stats? Is it possible to add these as well? I'm guessing not but I can't find any guidance on this. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks so much in advance for your help!&lt;/p&gt;&#xA;" OwnerUserId="834" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-15T20:31:52.317" LastActivityDate="2010-10-17T04:24:19.630" Title="Adding coefficients to obtain interaction effects - what to do with SEs?" Tags="&lt;regression&gt;&lt;standard-deviation&gt;&lt;standard-error&gt;" AnswerCount="2" />
  <row Id="3654" PostTypeId="2" ParentId="3652" CreationDate="2010-10-15T20:32:35.200" Score="1" ViewCount="" Body="&lt;p&gt;This is not unlike a stratified sample.  So, pooling the samples for a point estimate and standard error seems like a reasonable approach.  The two samples would be weighted by sample proportion.  &lt;/p&gt;&#xA;" OwnerUserId="485" LastActivityDate="2010-10-15T20:32:35.200" />
  <row Id="3655" PostTypeId="2" ParentId="3652" CreationDate="2010-10-15T20:33:13.343" Score="4" ViewCount="" Body="&lt;p&gt;You could do a pooled estimate as follows. You can then use the pooled estimates to generate a combined confidence interval. Specifically, let:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\bar{x_1} \sim N(\mu,\frac{\sigma^2}{n_1})$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\bar{x_2} \sim N(\mu,\frac{\sigma^2}{n_2})$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using the confidence intervals for the two cases, you can re-construct the standard errors for the estimates and replace the above with:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\bar{x_1} \sim N(\mu,SE_1)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\bar{x_2} \sim N(\mu,SE_2)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A pooled estimate would be:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\bar{x} = \frac{n_1 \bar{x_1} + n_2 \bar{x_2}}{n_1 + n_2}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\bar{x} \sim N(\mu,\frac{{n_1}^2  SE_1 + {n_2}^2 SE_2}{(n_1+n_2)^2})$&lt;/p&gt;&#xA;" LastActivityDate="2010-10-15T20:33:13.343" CommentCount="3" />
  <row Id="3657" PostTypeId="2" ParentId="3653" CreationDate="2010-10-15T20:44:47.467" Score="6" ViewCount="" Body="&lt;p&gt;I think this the expression for $SE_{b_{new}}$:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\sqrt{SE_1^2 + SE_2^2+2Cov(b_1,b_2)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can work with this new standard error to find your new test statistic for testing $H_o: \beta=0$&lt;/p&gt;&#xA;" OwnerUserId="1307" LastEditorUserId="1307" LastEditorDisplayName="" LastEditDate="2010-10-15T22:09:51.290" LastActivityDate="2010-10-15T22:09:51.290" CommentCount="4" />
  <row Id="3658" PostTypeId="2" ParentId="3650" CreationDate="2010-10-15T22:00:06.187" Score="2" ViewCount="" Body="&lt;p&gt;The expected frequency of observing $k$ purple balls in $d$ draws (without replacement) from an urn of $p$ purple balls and $n-p$ other balls is obtained by counting and equals&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{{p \choose k} {n-p \choose d-k} }{{n \choose d}}.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Test a sample (of say $100$) such experiments with a chi-squared statistic using these probabilities as the reference.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the second case, integrate over the prior distributions.  There is no nice formula for that, but the integration (actually a sum for these discrete variables) can be carried out exactly if you wish.  In the example given in the edited section -- independent uniform distributions of $n$ from $20$ to $30$ (thus having a one in 11 chance of being any value between $20$ and $30$ inclusive), of $p$ from $0$ to $4$, and of $d$ from $6$ to $12$ -- the result is a probability distribution on the possible numbers of purples ($0, 1, 2, 3, 4$) with values&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$0: 69728476151/142333251060 = 0.489896$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$1: 8092734193/24540215700 = 0.329774$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$2: 36854/258825 = 0.14239$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$3: 169436/4917675 = 0.0344545$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$4: 17141/4917675 = 0.00348559$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Use a chi-squared test for this situation, too.  As usual when conducting chi-squared tests, you will want to lump the last two or three categories into one because their expectations are less than $5$ (for $100$ repetitions).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is no problem with zero values.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit&lt;/strong&gt; (in response to a followup question)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The integrations are performed as multiple sums.  In this case, there is some prior distribution for $n$, a prior distribution for $p$, and a prior distribution for $d$.  For each possible ordered triple of outcomes $(n,p,d)$ together they give a probability $\Pr(n,p,d)$.  (With uniform distributions as above this probability is a constant equal to $1/((30-20+1)(4-0+1)(12-6+1))$.)  One forms the sum over all possible values of $(n,p,d)$ (a triple sum in this case) of&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\Pr(n,p,d) \frac{{p \choose k} {n-p \choose d-k} }{{n \choose d}}.$$&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-10-18T18:21:25.497" LastActivityDate="2010-10-18T18:21:25.497" CommentCount="2" />
  <row Id="3659" PostTypeId="2" ParentId="3652" CreationDate="2010-10-15T23:18:18.057" Score="3" ViewCount="" Body="&lt;p&gt;Sounds a lot like &lt;a href=&quot;http://en.wikipedia.org/wiki/Meta-analysis&quot; rel=&quot;nofollow&quot;&gt;meta-analysis&lt;/a&gt; to me. Your assumption that the samples are from the same population means you can use fixed-effect meta-analysis (rather than random-effects meta-analysis). The generic inverse-variance method takes a set of independent estimates and their variances as input, so doesn't require the full data and works even if different estimators have been used for different samples. The combined estimate is then a weighted average of the separate estimates, weighting each estimate by the inverse of its variance. The variance of the combined estimate is the inverse of the sum of the weights (the inverses of the variances).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You want to work on a scale where the sampling distribution of the estimate is approximately normal, or at least a scale on which the confidence intervals are approximately symmetric, so a log transformed scale is usual for ratio estimates (risk ratios, odds ratios, rate ratios...). In other cases a &lt;a href=&quot;http://en.wikipedia.org/wiki/Variance-stabilizing_transformation&quot; rel=&quot;nofollow&quot;&gt;variance-stabilising transformation&lt;/a&gt; would be useful, e.g. a square-root transformation for Poisson data, an arcsin-square-root transformation for binomial data, etc.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-10-15T23:18:18.057" />
  <row Id="3660" PostTypeId="2" ParentId="3595" CreationDate="2010-10-16T01:09:31.757" Score="4" ViewCount="" Body="&lt;p&gt;Most of the algorithms on &lt;a href=&quot;http://mahout.apache.org/&quot; rel=&quot;nofollow&quot;&gt;Apache Mahout&lt;/a&gt; scale way beyond 20M records, even with high-dimensional data. If you only need to build a prediction model, there are specific tools like Vowpal Wabbit (http://hunch.net/~vw/) that can easily scale to billions of records on a single machine.&lt;/p&gt;&#xA;" OwnerUserId="635" LastActivityDate="2010-10-16T01:09:31.757" CommunityOwnedDate="2010-10-16T01:09:31.757" CommentCount="1" />
  <row Id="3661" PostTypeId="1" CreationDate="2010-10-16T01:44:55.913" Score="5" ViewCount="247" Body="&lt;p&gt;Hi all&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have 3 observers that each take 2 measurements (length and weight) on 100 individuals; these procedures are repeated once (i.e., the same measurements are taken on the same 100 individuals by the same 3 observers), so that the data set is duplicated (i.e., early reading and late reading). &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;What is the best way to figure out how each individual observer's measurements vary between the late and early trial measurements?&lt;/li&gt;&#xA;&lt;li&gt;How can I best compare how close or different the measurements of length (or weight) differ among the 3 observers? &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="1603" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-10-16T08:43:26.653" LastActivityDate="2010-10-18T07:17:51.227" Title="Repeatability and measurement error from and between observers" Tags="&lt;anova&gt;&lt;error&gt;&lt;measurement&gt;&lt;reliability&gt;&lt;inter-rater&gt;" AnswerCount="2" FavoriteCount="2" />
  <row Id="3662" PostTypeId="2" ParentId="3634" CreationDate="2010-10-16T07:49:40.183" Score="0" ViewCount="" Body="&lt;p&gt;@chl The reason why I need a non-parametric approach is that my responses don't satisfy the hypothesis of the model ANOVA (non gaussian residuals, heteroskedasticity, strange dispersion of residuals versus fits) and no transformation is useful (i've  also tried tried with a Box-Cox transformation with bad results).&#xA;Thank you everybody for the answers!!!&#xA;Stefania&lt;/p&gt;&#xA;" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-16T09:05:00.363" LastActivityDate="2010-10-16T09:05:00.363" CommentCount="1" />
  <row Id="3663" PostTypeId="2" ParentId="3661" CreationDate="2010-10-16T08:43:54.767" Score="8" ViewCount="" Body="&lt;p&gt;What you describe is a reliability study where each subject is going to be assessed by the same three raters on two occasions. Analysis can be done separately on the two outcomes (length and weight, though I assume they will be highly correlated and you're not interested in how this correlation is reflected in raters' assessments). Estimating measurement reliability can be done in two ways: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The original approach (as described in Fleiss, 1987) relies on the analysis of variance components through an ANOVA table, where we assume no subject by rater interaction (the corresponding SS is constrained to 0) -- of course, you won't look at $p$-values, but at the MSs corresponding to relevant effects;&lt;/li&gt;&#xA;&lt;li&gt;A mixed-effects model allows to derive variance estimates, considering time as a fixed effect and subject and/or rater as random-effect(s) (the latter distinction depends on whether you consider that your three observers were taken or sampled from a pool of potential raters or not -- if the rater effect is small, the two analyses will yield quite the same estimate for outcome reliability).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In both cases, you will be able to derive a single intraclass correlation coefficient, which is a measure of reliability of the assessments (under the Generalizability Theory, we would call them generalizability coefficients), which would answer your second question. The first question deals with a potential effect of time (considered as a fixed effect), which I discussed here, &lt;a href=&quot;http://stats.stackexchange.com/questions/1015/reliability-in-elicitation-exercise&quot;&gt;Reliability in Elicitation Exercise&lt;/a&gt;. More details can be found in Dunn (1989) or Brennan (2001).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have an &lt;a href=&quot;http://gist.github.com/631183&quot; rel=&quot;nofollow&quot;&gt;R example script&lt;/a&gt; on Github which illustrates both approaches. I think it would not be too difficult to incorporate rater effects in the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;References&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Fleiss, J.L. (1987). &lt;em&gt;The design and analysis of clinical experiments&lt;/em&gt;. New York: Wiley.&lt;/li&gt;&#xA;&lt;li&gt;Dunn, G. (1989). &lt;em&gt;Design and analysis of reliability studies&lt;/em&gt;. Oxford&lt;/li&gt;&#xA;&lt;li&gt;Brennan, R.L. (2001). &lt;em&gt;Generalizability Theory&lt;/em&gt;. Springer&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-18T07:17:51.227" LastActivityDate="2010-10-18T07:17:51.227" CommentCount="0" />
  <row Id="3664" PostTypeId="2" ParentId="3661" CreationDate="2010-10-16T08:50:42.457" Score="2" ViewCount="" Body="&lt;p&gt;You need to repeat the same process separately for length and weight, as these are completely separate outcomes with different units and methods of measurement.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd start, as so often, by plotting some exploratory graphs. In this case a set of &lt;a href=&quot;http://en.wikipedia.org/wiki/Bland-Altman_plot&quot; rel=&quot;nofollow&quot;&gt;Bland–Altman&lt;/a&gt; (diffference vs. average) plots, one for each observer. If the plots for each observer look similar, I'd do a combined plot too. I'd look for any patterns in these plots, e.g. does the variability in the difference stay reasonably constant with the mean? (if not, i might consider some variance-stabilizing transformation). For each observer I'd then calculate the mean difference between early and late readings, to quantify whether there's a systematic difference, and the standard deviation of the difference as a way of quantifying how much each observer's measurements vary between late and early readings. I might then conduct a formal statistical test for the equality of the variances of the differences, such as the &lt;a href=&quot;http://en.wikipedia.org/wiki/Brown%E2%80%93Forsythe_test&quot; rel=&quot;nofollow&quot;&gt;Brown–Forsythe test&lt;/a&gt;. If there's no strong evidence that the variances differ substantially between observers, I'd move on to ANOVA as I see has just been described by chl.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-10-16T08:50:42.457" CommentCount="3" />
  <row Id="3665" PostTypeId="2" ParentId="3632" CreationDate="2010-10-16T14:58:21.727" Score="3" ViewCount="" Body="&lt;p&gt;If your only interest is whether their income is over $\$$5,000 or not, and it doesn't make a difference how far from that threshold their income actually is, then I would recommend using a classification technique (not necessarily logistic regression, try a range of methods and use whatever gives best out-of-sample performance) rather than regression.  Vladimir Vapnik (inventor of the support vector machine) says you should always aim to solve the problem at hand directly, rather than use a more general method and post-process the result.  That is a reasonable argument; if you are not interested how far above $5000 someones income is, why expend resources modelling the regression function a long way above that threshold?  So if you  have a classification problem, use a classifier, rather than threshold a regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note however it is likely that false-positive and false-negative costs may be different though and factor that into your classifier design.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;HTH&lt;/p&gt;&#xA;" OwnerUserId="887" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-10-16T15:08:36.650" LastActivityDate="2010-10-16T15:08:36.650" CommentCount="1" />
  <row Id="3666" PostTypeId="2" ParentId="2730" CreationDate="2010-10-16T15:47:52.163" Score="6" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0256117365&quot; rel=&quot;nofollow&quot;&gt;Applied Linear Statistical Models&lt;/a&gt; by Neter, Kutner, Wasserman, and Nachtscheim, has a very exhaustive (and exhausting!) treatment of ANOVA and ANCOVA.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It also covers power analysis, linear regression, multilinear regression, and introduces some MANOVA. It's a very long text, but does a very thorough job. I've linked you to the fourth edition. I doubt there's a huge difference over the fifth edition, and it's substantially cheaper.&lt;/p&gt;&#xA;" OwnerUserId="1118" LastActivityDate="2010-10-16T15:47:52.163" CommentCount="2" />
  <row Id="3667" PostTypeId="2" ParentId="1015" CreationDate="2010-10-16T18:41:49.013" Score="3" ViewCount="" Body="&lt;p&gt;Maybe I misunderstood the question, but what you are describing sounds like a test-retest reliability study on your Q scores. You have a series of experts each going to assess a number of items or questions, at two occasions (presumably fixed in time). So, basically you can assess the temporal stability of the judgments by computing an &lt;em&gt;intraclass correlation coefficient&lt;/em&gt; (ICC), which will give you an idea of the variance attributable to subjects in the variability of observed scores (or, in other words of the closeness of the observations on the same subject relative to the closeness of observations on different subjects).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The ICC may easily be obtained from a mixed-effect model describing the measurement $y_{ij}$ of subject $i$ on occasion $j$ as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;y_{ij}=\mu+u_i+\varepsilon_{ij},\quad \varepsilon\sim\mathcal{N}(0,\sigma^2)&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $u_i$ is the difference between the overall mean and subject $i$'s mean measurement, and $\varepsilon_{ij}$ is the measurement error for subject $i$ on occasion $j$. Here, this is a random-effect model. Unlike a standard ANOVA with subjects as factor, we consider the $u_i$ as random (i.i.d.) effects, $u_i\sim\mathcal{N}(0,\tau^2)$, independent of the error terms. Each measurement differ from the overall mean $\mu$ by the sum of the two error terms, among which the $u_i$ is shared between occasion on the same subjects. The total variance is then $\tau^2+\sigma^2$ and the proportion of the total variance that is accounted for by the subjects is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\rho=\frac{\tau^2}{\tau^2+\sigma^2}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;which is the ICC, or the reliability index from a psychometrical point of view.&#xA;Note that this reliability is sample-dependent (as it depends on the between-subject variance). Instead of the mixed-effects model, we could derive the same results from a two-way ANOVA (subjects + time, as factors) and the corresponding Mean Squares. You will find additional references in those related questions: &lt;a href=&quot;http://stats.stackexchange.com/questions/3661/repeatability-and-measurement-error-from-and-between-observers/3663#3663&quot;&gt;Repeatability and measurement error from and between observers&lt;/a&gt;, and &lt;a href=&quot;http://stats.stackexchange.com/questions/3539/inter-rater-reliability-for-ordinal-or-interval-data/3546&quot;&gt;Inter-rater reliability for ordinal or interval data&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In R, you can use the &lt;code&gt;icc()&lt;/code&gt; function from the &lt;a href=&quot;http://cran.r-project.org/web/packages/psy/index.html&quot; rel=&quot;nofollow&quot;&gt;psy&lt;/a&gt; package; the random intercept model described above corresponds to the &quot;agreement&quot; ICC, while incorporating the time effect as a fixed factor would yield the &quot;consistency&quot; ICC. You can also use the &lt;code&gt;lmer()&lt;/code&gt; function from the &lt;a href=&quot;http://cran.r-project.org/web/packages/lme4/index.html&quot; rel=&quot;nofollow&quot;&gt;lme4&lt;/a&gt; package, or the &lt;code&gt;lme()&lt;/code&gt; function from the &lt;a href=&quot;http://cran.r-project.org/web/packages/nlme/index.html&quot; rel=&quot;nofollow&quot;&gt;nlme&lt;/a&gt; package. The latter has the advantage that you can easily obtain 95% CIs for the variance components (using the &lt;code&gt;intervals()&lt;/code&gt; function). Dave Garson provided a nice overview (with SPSS illustrations) in &lt;a href=&quot;http://faculty.chass.ncsu.edu/garson/PA765/reliab.htm&quot; rel=&quot;nofollow&quot;&gt;Reliability Analysis&lt;/a&gt;, and &lt;a href=&quot;http://www.indiana.edu/~statmath/stat/all/hlm/hlm.pdf&quot; rel=&quot;nofollow&quot;&gt;Estimating Multilevel Models using SPSS, Stata, SAS, and R&lt;/a&gt; constitutes a useful tutorial, with applications in educational assessment. But the definitive reference is Shrout and Fleiss (1979), &lt;a href=&quot;http://www3.uta.edu/faculty/ricard/COED/Shrout%20&amp;amp;%20Fleiss%20%281979%29%20Intraclass%20correlations.pdf&quot; rel=&quot;nofollow&quot;&gt;Intraclass Correlations: Uses in Assessing Rater Reliability&lt;/a&gt;, &lt;em&gt;Psychological Bulletin&lt;/em&gt;, 86(2), 420-428.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have also added an &lt;a href=&quot;http://gist.github.com/631183&quot; rel=&quot;nofollow&quot;&gt;example R script&lt;/a&gt; on Githhub, that includes the ANOVA and mixed-effect approaches.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, should you add a constant value to all of the values taken at the second occasion, the Pearson correlation would remain identical (because it is based on deviations of the 1st and 2nd measurements from their &lt;em&gt;respective means&lt;/em&gt;), whereas the reliability as computed through the random intercept model (or the agreement ICC) would decrease.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;BTW, Cronbach's alpha is not very helpful in this case because it is merely a measure of the internal consistency (yet, another form of &quot;reliability&quot;) of an unidimensional scale; it would have no meaning should it be computed on items underlying different constructs. Even if your questions survey a single domain, it's hard to imagine mixing the two series of measurements, and Cronbach's alpha should be computed on each set separately. Its associated 95% confidence interval (computed by bootstrap) should give an indication about the stability of the internal structure between the two test occasions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an example of applied work with ICC, I would suggest&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Johnson, SR, Tomlinson, GA, Hawker,&#xA;  GA, Granton, JT, Grosbein, HA, and&#xA;  Feldman, BM (2010). &lt;a href=&quot;http://www.sciencedirect.com/science?_ob=ArticleURL&amp;amp;_udi=B6T84-4XR5W4K-1&amp;amp;_user=10&amp;amp;_coverDate=04%2F30%2F2010&amp;amp;_rdoc=1&amp;amp;_fmt=high&amp;amp;_orig=search&amp;amp;_origin=search&amp;amp;_sort=d&amp;amp;_docanchor=&amp;amp;view=c&amp;amp;_searchStrId=1505690480&amp;amp;_rerunOrigin=google&amp;amp;_acct=C000050221&amp;amp;_version=1&amp;amp;_urlVersion=0&amp;amp;_userid=10&amp;amp;md5=f532362cc8e30e572daf72564197eab1&amp;amp;searchtype=a&quot; rel=&quot;nofollow&quot;&gt;A valid and&#xA;  reliable belief elicitation method for&#xA;  Bayesian priors&lt;/a&gt;. &lt;em&gt;Journal of&#xA;  Clinical Epidemiology&lt;/em&gt;, 63(4),&#xA;  370-383.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-01T16:32:27.973" LastActivityDate="2010-11-01T16:32:27.973" />
  <row Id="3668" PostTypeId="2" ParentId="3632" CreationDate="2010-10-17T04:02:18.357" Score="2" ViewCount="" Body="&lt;p&gt;What are you trying to predict? Is your outcome just an indicator of whether income is above 5000 dollars or not? If so, that is the best that you can predict; that is, you can't predict anyone's income, only whether he has high (above 5000) income or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If this is your outcome and what you'd like to predict, the question is why would we favor using logit versus a linear probability model (standard OLS) for a binary outcome. Logit forces your prediction to be between 0 and 1, while LPM can predict any value for the probability of having high income (which is also the expected value of an indicator for having high income). This distinction is especially important if you have multiple predictors in your model or are trying to extrapolate to values of your predictors outside those of your sample. The downside to logit is the coefficients are difficult to interpret directly; instead, you typically report and consider marginal effects. Marginal effects depend upon the levels of your covariates, which can be confusing. Logit is also less robust to deviations from the normality and identical distribution assumptions about the error terms as I recall.&lt;/p&gt;&#xA;" OwnerUserId="401" LastActivityDate="2010-10-17T04:02:18.357" />
  <row Id="3669" PostTypeId="2" ParentId="3653" CreationDate="2010-10-17T04:09:30.837" Score="1" ViewCount="" Body="&lt;p&gt;To be more general, if you create a (row) vector for the estimate that you care about $R$ such that your estimator is equal to $R\beta$, then the variance of that estimator is $R\hat{V}R^\prime$, where $\hat{V}$ is the estimated variance-covariance matrix of your regression. Your estimate is distributed Normal or t, depending upon the assumption that you are making (Law of Large Numbers v. assuming normal errors in your regression model). Alternatively, you can test multiple estimates if you let $R$ be a matrix. This is known as a Wald test. The distribution in this case is a $\chi^2_r$, where $r$ is the number of rows in your matrix (assuming that the rows are linearly independent).&lt;/p&gt;&#xA;" OwnerUserId="401" LastEditorUserId="401" LastEditorDisplayName="" LastEditDate="2010-10-17T04:24:19.630" LastActivityDate="2010-10-17T04:24:19.630" CommentCount="1" />
  <row Id="3670" PostTypeId="2" ParentId="3542" CreationDate="2010-10-17T04:16:04.893" Score="7" ViewCount="" Body="&lt;p&gt;You might check out the documentation for the LaTeX package &lt;a href=&quot;http://www.ctan.org/tex-archive/macros/latex/contrib/booktabs/booktabs.pdf&quot; rel=&quot;nofollow&quot;&gt;booktabs&lt;/a&gt;; it gives general guidance and implements its design suggestions in LaTeX tables.&lt;/p&gt;&#xA;" OwnerUserId="401" LastActivityDate="2010-10-17T04:16:04.893" />
  <row Id="3671" PostTypeId="2" ParentId="3542" CreationDate="2010-10-17T07:36:38.073" Score="3" ViewCount="" Body="&lt;p&gt;I hope this answer is not too off topic, but a couple of days ago I have seen this link on visualizing tables at StackExchange:&#xA;&lt;a href=&quot;http://flowingdata.com/2009/04/21/visual-representation-of-tabular-information-how-to-fix-the-uncommunicative-table/&quot; rel=&quot;nofollow&quot;&gt;Visual Representation of Tabular Information – How to Fix the Uncommunicative Table&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1607" LastActivityDate="2010-10-17T07:36:38.073" CommentCount="2" />
  <row Id="3672" PostTypeId="1" AcceptedAnswerId="3674" CreationDate="2010-10-17T11:24:28.837" Score="5" ViewCount="137" Body="&lt;p&gt;Could anyone suggest some statistical measures to describe the distribution of a dendrogram? If I have two dendrograms, how could can I quantify their structural differences?&lt;/p&gt;&#xA;" OwnerUserId="1250" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-10-18T08:07:08.010" LastActivityDate="2010-10-18T08:07:08.010" Title="A measure to describe the distribution of a dendrogram" Tags="&lt;distributions&gt;&lt;time-series&gt;&lt;clustering&gt;&lt;dendrogram&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="1" />
  <row Id="3673" PostTypeId="2" ParentId="3419" CreationDate="2010-10-17T12:06:38.520" Score="3" ViewCount="" Body="&lt;p&gt;You asked a difficult question, but I'm a little bit surprised that the various clues that were suggested to you received so little attention. I upvoted all of them because I think they basically are useful responses, though in their actual form they call for further bibliographic work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Disclaimer:&lt;/strong&gt; I never had to deal with such a problem, but I regularly have to expose statistical results that may differ from physician's &lt;em&gt;a priori&lt;/em&gt; beliefs and I learn a lot from unraveling their lines of reasoning. Also, I have some background in teaching human decision/knowledge from the perspective of Artificial Intelligence and Cognitive Science, and I think what you asked is not so far from how experts actually decide that two objects are similar or not, based on their attributes and a common understanding of their relationships.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From your question, I noticed two interesting assertions. The first one related to how an expert assess the similarity or difference between two set of measurements:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;I don't particularly care if there is&#xA;  some relation between attribute X and&#xA;  Y. What I care about is if a doctor&#xA;  thinks there is a relation between X&#xA;  and Y.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The second one, &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;How can I predict what they think the&#xA;  similarity is? Do they look at certain&#xA;  attributes?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;looks like it is somewhat subsumed by the former, but it seems more closely related to what are the most salient attributes that allow to draw a clear separation between the objects of interest.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To the first question, I would answer: Well, if there is no characteristic or objective relationship between any two subjects, what would be the rationale for making up an hypothetical one? Rather, I think the question should be: If I only have limited resources (knowledge, time, data) to take a decision, how do I optimize my choice? To the second question, my answer is: Although it seems to partly contradicts your former assertion (if there is no relationship at all, it implies that the available attributes are not discriminative or useless), I think that most of the time this is a combination of attributes that makes sense, and not only how a given individual scores on a single attribute.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let me dwell on these two points.&#xA;Human beings have a limited or &lt;a href=&quot;http://en.wikipedia.org/wiki/Bounded_rationality&quot; rel=&quot;nofollow&quot;&gt;bounded rationality&lt;/a&gt;, and can take a decision (often the right one) without examining all possible solutions. There is also a close connection with &lt;a href=&quot;http://en.wikipedia.org/wiki/Abduction_%28logic%29&quot; rel=&quot;nofollow&quot;&gt;abductive reasoning&lt;/a&gt;.&#xA;It is well known that there is some variability between individual judgments, and even between judgments from the same expert at two occasions. This is what we are interested in in reliability studies. But you want to know how these experts elaborate their judgments. There is a huge amount of papers about that in cognitive psychology, especially on the fact that &lt;em&gt;relative judgments&lt;/em&gt; are easier and more reliable than &lt;em&gt;absolute&lt;/em&gt; ones. Doctors' decisions are interesting in this respect because they are able to take a &quot;good&quot; decision with a limited amount of information, but at the same time they benefit from an ever growing internal knowledge base from which they can draw expected relationships (extrapolation). In other words, they have a built-in inference (assumed to be hypothetico-deductive) machinery and accumulate positive evidence or counterfactuals from there experience or practice. Reproducing this inferential ability and the use of declarative knowledge was the aim of several expert or &lt;a href=&quot;http://en.wikipedia.org/wiki/Production_system&quot; rel=&quot;nofollow&quot;&gt;production rule&lt;/a&gt; systems in the 70s, the most famous one being &lt;a href=&quot;http://en.wikipedia.org/wiki/Mycin&quot; rel=&quot;nofollow&quot;&gt;MYCIN&lt;/a&gt;, and more generally of Artifical Intelligence earlier in 1946 (Can we reproduce on an artificial system the intelligent behavior observed in man?). Automatic treatment of speech, problem solving, visual shape recognition are still active projects nowadays and they all have to do with identifying salient features and their relationships to make an appropriate decision (i.e., how far should two patterns be to be judged as the emanation of two distinct generating processes?).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In sum, our doctors are able to draw an optimal inference from a limited amount of data, compensating from noise that arises simply as a byproduct of individual variability (at the level of the patients). Thus, there is a clear connection with statistics and probability theory, and the question is what conscious or subconscious methodology help doctors forming their judgments. &lt;a href=&quot;http://en.wikipedia.org/wiki/Semantic_network&quot; rel=&quot;nofollow&quot;&gt;Semantic networks&lt;/a&gt; (SN), &lt;a href=&quot;http://en.wikipedia.org/wiki/Bayesian_network&quot; rel=&quot;nofollow&quot;&gt;belief networks&lt;/a&gt;, and &lt;a href=&quot;http://en.wikipedia.org/wiki/Decision_tree&quot; rel=&quot;nofollow&quot;&gt;decision trees&lt;/a&gt; are all relevant to the question you asked. The paper you cited is about using an &lt;a href=&quot;http://en.wikipedia.org/wiki/Ontology_%28information_science%29&quot; rel=&quot;nofollow&quot;&gt;ontology&lt;/a&gt; as a basis of formal judgments, but it is no more than an extension of SNs, and many projects were initiated in this direction (I can think of the &lt;a href=&quot;http://www.geneontology.org/&quot; rel=&quot;nofollow&quot;&gt;Gene Ontology&lt;/a&gt; for genomic studies, but many others exist in different domains). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, look at the following hierarchical classification of diagnostic categories (it is roughly taken from Dunn 1989, p. 25):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/y3t62.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And now take a look at the &lt;a href=&quot;http://ICD9Data.com/&quot; rel=&quot;nofollow&quot;&gt;ICD classification&lt;/a&gt;; I think it is not too far from this schematic classification. Mental disorders are organized into distinct categories, some of them being closer one to each other. What render them similar is the closeness of their expression (phenotype) in any patient, and the fact that they share some similarities in their somatic/psychological etiology. Assessing whether two doctors would make the same diagnostic is a typical example of an &lt;em&gt;inter-rater agreement&lt;/em&gt; study, where two psychiatrists are asked to place each of several patients in mutually exclusive categories. The hierarchical structure should be reflected in the disagreement between each doctor, that is they may not agree on the finer distinction between diagnostic classes (the leafs) but if they were to disagree between insomnia and schizophrenia, well it would be little bit disconcerting... How these two doctors decide on which class a given patient belongs to is no more than a clustering problem: How likely are two individuals, given a set of observed values on different attributes, to be similar enough so that I decide they share the same class membership? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, some attributes are more influential than others, and this is exactly what is reflected in the weight attributed to a given attribute in Latent Class Analysis (which can be thought of as a &lt;a href=&quot;http://www.statisticalinnovations.com/articles/kmeans2a.htm&quot; rel=&quot;nofollow&quot;&gt;probabilistic extension&lt;/a&gt; of clustering methods like k-means), or the &lt;a href=&quot;http://www.stat.berkeley.edu/~breiman/RandomForests/cc_home.htm&quot; rel=&quot;nofollow&quot;&gt;variable importance&lt;/a&gt; in Random Forests. We need to put things into boxes, because at first sight it's simpler. The problem is that often things overlap to some extent, so we need to consider different levels of categorization.&#xA;In fact, &lt;em&gt;cluster analysis&lt;/em&gt; is at the heart of the actual DSM categories, and many papers actually turn around assigning one patient to a specific syndromic category, based on the profile of his response to a battery of neuropsychological assessments. This merely looks like a &lt;em&gt;subtyping&lt;/em&gt; approach; each time, we seek to refine a preliminary well-established diagnostic category, by adding exception rules or an additional relevant symptom or impairment. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A related topic is &lt;em&gt;decision trees&lt;/em&gt; which are by far the most well understood statistical techniques by physicians. Most of the time, they described a nested series of boolean assertions (Do you have a sore throat? If yes, do you have a temperature? etc.; but look at an example of public &lt;a href=&quot;http://www.crengland.com/content/managementservices/flu/flu-decision-tree.pdf&quot; rel=&quot;nofollow&quot;&gt;influenza diagnostic tree&lt;/a&gt;) according to which we can form a decision regarding patients proximity (i.e. how similar patients are wrt. attributes considered for building the tree -- the closer they are the more likely they are to end up in the same leaf). &lt;a href=&quot;http://en.wikipedia.org/wiki/Association_rule_learning&quot; rel=&quot;nofollow&quot;&gt;Association rules&lt;/a&gt; and the  &lt;a href=&quot;http://en.wikipedia.org/wiki/C4.5_algorithm&quot; rel=&quot;nofollow&quot;&gt;C4.5 algorithm&lt;/a&gt; rely quite on the same idea. On a related topic, there's the &lt;a href=&quot;http://www.ijmijournal.com/article/S1386-5056%2807%2900122-0/abstract&quot; rel=&quot;nofollow&quot;&gt;patient rule-induction method&lt;/a&gt; (PRIM). Now clearly, we must make a distinction between all those methods, that make an efficient use of a large body of data and incorporate bagging or boosting to compensate for model fragility or overfitting issues, and doctors who cannot process huge amount of data in an automatic and algorithmic manner. But, for small to moderate amount of descriptors, I think they perform quite good after all.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The yes-or-no approach is not the panacea, though. In behavioral genetics and psychiatry, it is commonly argued that the classification approach is probably not the best way to go, and that common diseases (learning disorders, depression, personality disorders, etc.) reflect a continuum rather than classes of opposite valence. Nobody's perfect!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In conclusion, I think doctors actually hold a kind of internalized inference engine that allows them to assign patients into distinctive classes that are characterized by a weighted combination of available evidences; in other words, they are able to organize their knowledge in an efficient manner, and these internal representations and the relationships they share may be augmented throughout experience. &lt;a href=&quot;http://en.wikipedia.org/wiki/Case-based_reasoning&quot; rel=&quot;nofollow&quot;&gt;Case-based reasoning&lt;/a&gt; probably comes into play at some point too. All of this may be subjected to (a) revision with newly available data (we are not simply acting as definitive binary classifiers, and are able to incorporate new data in our decision making), and (b) subjective biases arising from past experience or wrong self-made association rules. However, they are prone to errors, as every decision systems...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All statistical techniques reflecting these steps -- decisions trees, bagging/boosting, cluster analysis, latent cluster analysis -- seems relevant to your questions, although they may be hard to instantiate in a single decision rule. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are a couple of references that might be helpful, as a first start to how doctors make their decisions:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.informatica2009.sld.cu/Members/fatihozbek/a-clinical-decision-support-system-for-clinicians-for-determining-appropriate-radiolog131c-imaging-examination/at_download/trabajo&quot; rel=&quot;nofollow&quot;&gt;A clinical decision support system for clinicians for determining appropriate radiologic imaging examination&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Grzymala-Busse, JW. &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.86.2566&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;Selected Algorithms of Machine Learning from Examples&lt;/a&gt;. &lt;em&gt;Fundamenta Informaticae&lt;/em&gt; 18 (1993), 193–207&lt;/li&gt;&#xA;&lt;li&gt;Santiago Medina, L, Kuntz, KM, and Pomeroy, S. &lt;a href=&quot;http://ec.digaden.edu.mx/moodle/moodledata/99/01medint/08_9neuder/30cef/113d255.pdf&quot; rel=&quot;nofollow&quot;&gt;Children With Headache Suspected of Having a Brain Tumor: A Cost-Effectiveness Analysis of Diagnostic Strategies&lt;/a&gt;. &lt;em&gt;Pediatrics&lt;/em&gt; 108 (2001), 255-263&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.psychiatrictimes.com/display/article/10168/57556&quot; rel=&quot;nofollow&quot;&gt;Building Better Algorithms for the Diagnosis of Nontraumatic Headache&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Jenkins, J, Shields, M, Patterson, C, and Kee, F. &lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2083892/pdf/672.pdf&quot; rel=&quot;nofollow&quot;&gt;Decision making in asthma exacerbation: a clinical judgement analysis&lt;/a&gt;. Arch Dis Child 92 (2007), 672–677&lt;/li&gt;&#xA;&lt;li&gt;Croskerry, P. &lt;a href=&quot;http://www.crnns.ca/documents/AEM%20Ms%20-%20Achieving%20Quality%20in%20Decision%20MakingEducation%20copyPC.pdf&quot; rel=&quot;nofollow&quot;&gt;Achieving quality in clinical decision making: cognitive strategies and detection of bias&lt;/a&gt;. Acad Emerg Med 9(11) (2002), 1184-204.&lt;/li&gt;&#xA;&lt;li&gt;Cahan, A, Gilon, D, Manor, O, and Paltiel. &lt;a href=&quot;http://qjmed.oxfordjournals.org/content/96/10/763.full.pdf+html&quot; rel=&quot;nofollow&quot;&gt;Probabilistic reasoning and clinical decision-making: do doctors overestimate diagnostic probabilities?&lt;/a&gt; QJM 96(10) (2003), 763-769&lt;/li&gt;&#xA;&lt;li&gt;Wegwarth, O, Gaissmaier, W, and Gigerenzer, G. &lt;a href=&quot;http://library.mpib-berlin.mpg.de/ft/ow/OW_Smart_2009.pdf&quot; rel=&quot;nofollow&quot;&gt;Smart strategies for doctors and doctors-in-training: heuristics in medicine&lt;/a&gt;. Medical Education 43 (2009), 721–728&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-17T19:06:12.167" LastActivityDate="2010-10-17T19:06:12.167" CommentCount="2" />
  <row Id="3674" PostTypeId="2" ParentId="3672" CreationDate="2010-10-17T14:37:56.007" Score="4" ViewCount="" Body="&lt;p&gt;See this SO question: &lt;a href=&quot;http://stackoverflow.com/questions/2218395/how-do-you-compare-the-similarity-between-two-dendrograms-in-r&quot;&gt;http://stackoverflow.com/questions/2218395/how-do-you-compare-the-similarity-between-two-dendrograms-in-r&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-10-17T14:37:56.007" CommentCount="1" />
  <row Id="3675" PostTypeId="2" ParentId="3307" CreationDate="2010-10-17T15:45:28.087" Score="2" ViewCount="" Body="&lt;p&gt;I agree with @ars that you are unlikely to get one answer for this (you may also have more success on &lt;a href=&quot;http://mathoverflow.net&quot; rel=&quot;nofollow&quot;&gt;http://mathoverflow.net&lt;/a&gt;, since our community tends to be more applied, while this technique would have very little real-world usage).  The Abrahao/Barbosa paper is a good reference.  Just to provide some additional sources:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This paper looks at the correlation between fractal dimensions, which seems like a reasonable approach to the problem.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://arxiv.org/abs/0908.3950&quot; rel=&quot;nofollow&quot;&gt;&quot;The correlation of fractal structures in the photospheric and the coronal magnetic field&quot;&lt;/a&gt; (Dimitropoulou, Georgoulis, Isliker, Vlahos, Anastasiadis, Strintzi, Moussas 2009)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;This paper uses the multi-fractal spectra to estimate correlation:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://arxiv.org/abs/0901.4844&quot; rel=&quot;nofollow&quot;&gt;&quot;Continuous wavelet transform based time-scale and multi-fractal analysis of the nonlinear oscillations in a hollow cathode glow discharge plasma&quot;&lt;/a&gt; (Nurujjaman, Narayanan, Iyengar 2009)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Regarding the &quot;Correlation Fractal Dimension&quot;, this paper provides a fast algorithm:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://arxiv.org/abs/0905.4138&quot; rel=&quot;nofollow&quot;&gt;&quot;Faster estimation of the correlation fractal dimension using box-counting&quot;&lt;/a&gt; (Attikos, Doumpos 2009)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="5" LastEditorUserId="5" LastEditorDisplayName="" LastEditDate="2010-10-17T15:59:56.937" LastActivityDate="2010-10-17T15:59:56.937" />
  <row Id="3676" PostTypeId="1" CreationDate="2010-10-17T16:13:28.657" Score="4" ViewCount="224" Body="&lt;p&gt;&lt;strong&gt;Background:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm a junior researcher at an institute dealing with regional issues, particularly involving drug policy. Almost two years ago, one of our senior researchers began collecting arrest data about a nearby large city. He had been transcribing newspaper police blotter by hand until I joined a year and change ago and convinced him to switch to automated data collection.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We're now ready to start analyzing this dataset. It contains about 20k arrest records with virtually no missing or invalid values (I'd estimate that there are less than 100 such records, and some of those are just parser errors). These records include full name and home address of the arrested person, arresting officers, exact charges, etc. (we've even found SSNs in the data). At this point, we're just exploring the data, but we expect to find differences in number and severity of charges according to race, arrest location, and home location.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The dataset has two variables that, to my understanding, are categorical multiple-response questions. &quot;Arresting Officer&quot; is two columns, one of which always has a value and one of which may have an a value (for the second officer present at the arrest). &quot;Charges&quot; is five columns, each containing a single charge and in no particular order. To complicate the structure further, a person arrested on more than five charges will be issued multiple records.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We messed around with the above dataset for a few hours so far and have gotten as far as being able to obtain useful counts and percentages in SPSS by using the Multiple Response Set feature and plugging that into Custom Tables. This is a fine start, but we'd like to move on to more detailed analysis at some point. Unfortunately, neither of us are aware of the recommended (or anti-recommended) methods for analyzing a categorical multiple-response question. This also applies to our desire to eventually group specific charges (e.g., &quot;possession of controlled substances &amp;lt; 4 oz.&quot;) into broader categories such as 'violent crimes' or 'drug crimes'.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that I'm not looking for just a snap answer here, as we don't have a deadline. I would be glad to do readings, so feel free to point me in the direction of tutorials, textbooks, and so on. I'm also not particularly attached to SPSS - it's just what my coworker is used to. If there are clear disadvantages to using it for this type of problem, I don't mind learning something new.&lt;/p&gt;&#xA;" OwnerUserId="1609" LastActivityDate="2010-10-17T20:10:30.190" Title="&quot;Multiple response&quot; analysis of arrest records" Tags="&lt;dataset&gt;&lt;spss&gt;" AnswerCount="3" FavoriteCount="2" />
  <row Id="3677" PostTypeId="2" ParentId="633" CreationDate="2010-10-17T18:21:12.987" Score="2" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://biostat.jhsph.edu/~iruczins/&quot; rel=&quot;nofollow&quot;&gt;Ingo Ruczinski&lt;/a&gt; has contributed to promote the use of &lt;a href=&quot;http://kooperberg.fhcrc.org/logic/documents/intro.html&quot; rel=&quot;nofollow&quot;&gt;Logic regression&lt;/a&gt; for data set consisting of binary variables, with an emphasis on higher-order interaction terms. The main advantage compared to usual or penalized GLMs is that it is more parcimonious in terms of degrees of freedom. The outcome may be continuous or categorical, and continuous covariates can be added to the model (or the outcome can first be residualized on them if these are the binary predictors that are of interest). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The original paper&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Ruczinski I, Kooperberg C, LeBlanc ML&#xA;  (2003). &lt;a href=&quot;http://kooperberg.fhcrc.org/papers/2003logic.pdf&quot; rel=&quot;nofollow&quot;&gt;Logic Regression&lt;/a&gt;.&#xA;  &lt;em&gt;Journal of Computational and Graphical Statistics&lt;/em&gt;, 12(3), 475-511.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;includes several applications in biomedical studies, and a comparison of LR with &lt;a href=&quot;http://en.wikipedia.org/wiki/Classification_and_regression_tree&quot; rel=&quot;nofollow&quot;&gt;CART&lt;/a&gt; and &lt;a href=&quot;http://en.wikipedia.org/wiki/Multivariate_adaptive_regression_splines&quot; rel=&quot;nofollow&quot;&gt;MARS&lt;/a&gt;. Although it has mainly been applied in large-scale genetic studies (e.g. genome-wide association studies), it should work with any binary variables whose combinations of interest can be expressed with a set of logical operators.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;a href=&quot;http://cran.r-project.org/web/packages/LogicReg/index.html&quot; rel=&quot;nofollow&quot;&gt;LogicReg&lt;/a&gt; R package implements this technique; see also the related packages on CRAN and Bioconductor, esp. &lt;a href=&quot;http://cran.r-project.org/web/packages/LogicForest/index.html&quot; rel=&quot;nofollow&quot;&gt;LogicForest&lt;/a&gt; which shares some ideas with Random Forests.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-02T18:47:07.033" LastActivityDate="2010-11-02T18:47:07.033" />
  <row Id="3678" PostTypeId="2" ParentId="3676" CreationDate="2010-10-17T18:37:01.210" Score="3" ViewCount="" Body="&lt;p&gt;I can't particularly comment on how to handle multiple response categories, but you need to further refine your question for people on this forum to be able to give useful advice.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You mention various interests, such as some sort of drug policy intervention, and differential charges according to race, arrest location, and home location. For the differential charges their is a huge body of criminological literature assessing various aspects of this. Are you interested in discretionary behavior of particular officers (or racially prejudiced treatment)? Are you interested in aspects of disproportionate minority contact with the criminal justice system? There is such a wide variety of potential questions I can not give any advice. Of what nature is the drug policy intervention? Are you interested in criminal histories and the effect of some policy?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The nature of your data is pretty typical. Some recent arrest data I worked with had an average of around 3 charges per arrest (I remember 1 case having 20 charges in a single arrest). You will typically have some charges that tend to come together (and some times functionally redundant charges). Often times drug possession charges are not alone because the offender did something else to attract an officers attention (most often another crime), and upon arrest they were searched and drugs were found. You will undoubtedly have a core of prolific offenders in your data, and for any analysis you will want to know their histories, and likely take them into account for your analysis (do you have unique identifiers for individuals or do you have to match based on names, DOBs, and/or SSN's?) You may also have co-offending behavior that may be of interest.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most projects I have been involved in (including my own work) have dealt with the multiple charges in two ways. One is only include the &quot;top&quot; charge according to some ranking criteria, the other is to only analyze a particular subset of charges. This is hardly universal advice though, and without knowing the question you are addressing it is probably not advisable to do either of these at the onset. If you clump any charges together (e.g. treat possession of weed the same as possession of cocaine) I would suggest you do it on theoretical grounds as opposed to using some sort of statistical methods (although again depending on the question some type of stat clustering method may be useful).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The more specific questions you have the better this community will be able to give advice. The nature of your data may seem complicated but many people on this forum will have had experience with similar data structures (at least in various aspects). &lt;/p&gt;&#xA;" OwnerUserId="1036" LastEditorUserId="1036" LastEditorDisplayName="" LastEditDate="2010-10-17T19:27:17.977" LastActivityDate="2010-10-17T19:27:17.977" />
  <row Id="3679" PostTypeId="2" ParentId="3676" CreationDate="2010-10-17T18:41:41.327" Score="1" ViewCount="" Body="&lt;p&gt;It is not clear what you questions you are trying to answer but here are are several ways to deal with the multiple-response data:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Arresting Officer&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Convert the two columns into a single count variable (1 or 2) which indicates the no of arresting officers. You will lose the arresting officer's identities but perhaps that is not of interest to you per se?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Charges&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Again convert to a count variable or perhaps a weighted count with the weights being the severity of the crime. The count variable represents the number of charges against the defendant.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The above strategies to deal with multiple-response variables has drawbacks as you lose information (e.g., identity of officers, details of specific crimes charged etc) but perhaps that is acceptable to you given the research questions you want to pursue.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A better answer can potentially be given but that will require some knowledge of what you are trying to accomplish with the data.&lt;/p&gt;&#xA;" LastActivityDate="2010-10-17T18:41:41.327" />
  <row Id="3680" PostTypeId="2" ParentId="1621" CreationDate="2010-10-17T20:07:00.620" Score="2" ViewCount="" Body="&lt;p&gt;There was a read paper last week at the Royal Statistical Society on MCMC techniques over Riemann  manifolds, primarily using the Fisher information metric:&#xA;&lt;a href=&quot;http://www.rss.org.uk/main.asp?page=1836#Oct_13_2010_Meeting&quot; rel=&quot;nofollow&quot;&gt;http://www.rss.org.uk/main.asp?page=1836#Oct_13_2010_Meeting&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The results seem promising, though as the authors point out, in many models of interest (such as mixture models) the Fisher information has no analytic form.&lt;/p&gt;&#xA;" OwnerUserId="495" LastEditorUserId="495" LastEditorDisplayName="" LastEditDate="2010-10-17T20:12:38.687" LastActivityDate="2010-10-17T20:12:38.687" CommentCount="1" />
  <row Id="3681" PostTypeId="2" ParentId="3676" CreationDate="2010-10-17T20:10:30.190" Score="1" ViewCount="" Body="&lt;p&gt;I've examined associations between multiple response catgeogrical variables in the past basically following the log-linear approach for marginal data outlined in the following:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.amstat.org/sections/srms/proceedings/y2003/Files/JSM2003-000394.pdf&quot; rel=&quot;nofollow&quot;&gt;Strategies for Modeling Two Categorical Variables with Multiple Category Choices&lt;/a&gt; (Bilder, Loughlin, 2003)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Your case may be more complicated since you're looking at more than just Officer by Charges.  But the Bilder paper and references within may be a good start for exploring your modeling choices.  The nice thing is that I was able to fit this in R without much trouble.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem you're likely to run into is ending up with a sparse contingency table which can lead to convergence problems when fitting your log-linear model.  In this respect, I think Andy's and Srikant's advice will serve you well -- you'll have to make some assumptions or simplifications suitable for your domain.  Figure out what question you're interested in and see if you can reduce the dimensions in some way.  &lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-10-17T20:10:30.190" />
  <row Id="3682" PostTypeId="1" AcceptedAnswerId="3683" CreationDate="2010-10-17T20:29:40.313" Score="6" ViewCount="111" Body="&lt;p&gt;What is the meaning of $\|a\|_p=\left(\sum _{i=1}^n \left|a_i(t)\right|{}^p\right){}^{\frac{1}{p}}$?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This formula is called out on the fifth page of &lt;em&gt;An Improved Data Stream Summary: The Count-Min Sketch and its Applications&lt;/em&gt; (which can be found &lt;a href=&quot;http://www.madalgo.au.dk/img/SumSchoo2007_Lecture%20slides/Bibliography/p14_Cormode_JAl_05.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;). I'm implementing the Count-Min Sketch and can understand the basic concepts just fine, but some of the finer points are explained in terms of this equation and some other terminology that I'm unacquainted with.&lt;/p&gt;&#xA;" OwnerUserId="1515" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-10-18T00:17:22.833" LastActivityDate="2010-10-18T00:17:22.833" Title="What is the meaning of $\|a\|_p=\left(\sum _{i=1}^n \left|a_i(t)\right|{}^p\right){}^{\frac{1}{p}}$?" Tags="&lt;summary-statistics&gt;" AnswerCount="2" />
  <row Id="3683" PostTypeId="2" ParentId="3682" CreationDate="2010-10-17T20:36:56.287" Score="7" ViewCount="" Body="&lt;p&gt;It's the $L^p$ norm.  See for example the Wikipedia articles:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Lp_space&quot; rel=&quot;nofollow&quot;&gt;$L^p$ space&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Minkowski_distance&quot; rel=&quot;nofollow&quot;&gt;Minkowski distance&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If you use $p = 2$, you'll find it resolves to the more familiar Euclidean norm -- i.e. the most familiar measure used as length of the vector $a$.  Other values of p give others ways of measuring length as outlined in the article -- see the sections on Euclidean norm, Taxicab norm, etc.&lt;/p&gt;&#xA;" OwnerUserId="251" LastEditorUserId="251" LastEditorDisplayName="" LastEditDate="2010-10-17T20:42:02.297" LastActivityDate="2010-10-17T20:42:02.297" CommentCount="2" />
  <row Id="3684" PostTypeId="2" ParentId="3614" CreationDate="2010-10-17T20:44:19.483" Score="2" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Characteristic_function_%28probability_theory%29&quot; rel=&quot;nofollow&quot;&gt;Characteristic functions&lt;/a&gt; can make computations involving the sums and differences of &lt;a href=&quot;http://en.wikipedia.org/wiki/Random_variable&quot; rel=&quot;nofollow&quot;&gt;random variables&lt;/a&gt; really easy. &lt;a href=&quot;http://www.wolfram.com/&quot; rel=&quot;nofollow&quot;&gt;Mathematica&lt;/a&gt; has lots of functions to work with statistical distributions, including a builtin to transform a distribution into its characteristic function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd like to illustrate this with two concrete examples: (1) Suppose you wanted to determine the results of rolling a collection of dice with &lt;i&gt;differing&lt;/i&gt; numbers of sides, e.g., roll two six-sided dice plus one eight-sided die (i.e., &lt;b&gt;2d6+d8&lt;/b&gt;)? Or (2) suppose you wanted to find the difference of two dice rolls (e.g., &lt;b&gt;d6-d6&lt;/b&gt;)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An easy way to do this would be to use the characteristic functions of the underlying discrete uniform distributions. If a random variable $X$ has a &lt;a href=&quot;http://en.wikipedia.org/wiki/Probability_mass_function&quot; rel=&quot;nofollow&quot;&gt;probability mass function&lt;/a&gt; $f$, then its characteristic function $\varphi_X(t)$ is just the discrete &lt;a href=&quot;http://en.wikipedia.org/wiki/Fourier_transform&quot; rel=&quot;nofollow&quot;&gt;Fourier Transform&lt;/a&gt; of $f$, i.e., $\varphi_X(t) = \mathcal{F}\{f\}(t) = E[e^{i t X}]$. A theorem tells us:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;i&gt;If the independent random variables $X$ and $Y$ have corresponding probability mass functions $f$ and $g$, then the pmf $h$ of the sum $X + Y$ of these RVs is the &lt;a href=&quot;http://en.wikipedia.org/wiki/Convolution&quot; rel=&quot;nofollow&quot;&gt;convolution&lt;/a&gt; of their pmfs $h(n) = (f \ast g)(n) = \sum_{m=-\infty}^\infty f(m) g(n-m)$.&lt;/i&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can use the convolution property of Fourier Transforms to restate this more simply in terms of characteristic functions:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;i&gt;The characteristic function $\varphi_{X+Y}(t)$ of the sum of independent random variables $X$ and $Y$ equals the product of their characteristic functions $\varphi_{X}(t) \varphi_{Y}(t)$.&lt;/i&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This Mathematica function will make the characteristic function for an s-sided die:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;MakeCf[s_] := &#xA; Module[{Cf}, &#xA;  Cf := CharacteristicFunction[DiscreteUniformDistribution[{1, s}], &#xA;    t];&#xA;  Cf]&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The pmf of a distribution can be recovered from its characteristic function, because Fourier Transforms are invertible. Here is the Mathematica code to do it:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;RecoverPmf[Cf_] := &#xA;  Module[{F}, &#xA;    F[y_] := SeriesCoefficient[Cf /. t -&gt; -I*Log[x], {x, 0, y}];&#xA;    F]&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Continuing our example, let F be the pmf that results from 2d6+d8.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;&#xA;F := RecoverPmf[MakeCf[6]^2 MakeCf[8]]&#xA;&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are $6^2 \cdot 8 = 288$ outcomes. The domain of support of F is $S=\{3,\ldots,20\}$. Three is the min because you're rolling three dice. And twenty is the max because $20 = 2 \cdot 6 + 8$. If you want to see the image of F, compute&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;In:= F /@ Range[3, 20]&#xA;&#xA;Out= {1/288, 1/96, 1/48, 5/144, 5/96, 7/96, 13/144, 5/48, 1/9, 1/9, \&#xA;5/48, 13/144, 7/96, 5/96, 5/144, 1/48, 1/96, 1/288}&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you want to know the number of outcomes that sum to 10, compute&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;In:= 6^2 8 F[10]&#xA;&#xA;Out= 30&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;i&gt;If the independent random variables $X$ and $Y$ have corresponding probability mass functions $f$ and $g$, then the pmf $h$ of the difference $X - Y$ of these RVs is the &lt;a href=&quot;http://en.wikipedia.org/wiki/Cross-correlation&quot; rel=&quot;nofollow&quot;&gt;cross-correlation&lt;/a&gt; of their pmfs $h(n) = (f \star g)(n) = \sum_{m=-\infty}^\infty f(m) g(n+m)$.&lt;/i&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can use the cross-correlation property of Fourier Transforms to restate this more simply in terms of characteristic functions:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;i&gt;The characteristic function $\varphi_{X-Y}(t)$ of the difference of two independent random variables ${X,Y}$ equals the product of the characteristic function $\varphi_{X}(t)$ and $\varphi_{Y}(-t)$ (N.B. the negative sign in front of the variable t in the second characteristic function).&lt;/i&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, using Mathematica to find the pmf G of d6-d6:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;&#xA;G := RecoverPmf[MakeCf[6] (MakeCf[6] /. t -&gt; -t)]&#xA;&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are $6^2 = 36$ outcomes. The domain of support of G is $S=\{-5,\ldots,5\}$. -5 is the min because $-5=1-6$. And 5 is the max because $6-1=5$. If you want to see the image of G, compute&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;In:= G /@ Range[-5, 5]&#xA;&#xA;Out= {1/36, 1/18, 1/12, 1/9, 5/36, 1/6, 5/36, 1/9, 1/12, 1/18, 1/36}&#xA;&lt;/pre&gt;&#xA;" LastActivityDate="2010-10-17T20:44:19.483" CommentCount="3" />
  <row Id="3685" PostTypeId="1" AcceptedAnswerId="3692" CreationDate="2010-10-17T21:57:55.460" Score="7" ViewCount="651" Body="&lt;p&gt;Hierarchical clustering can be represented by a dendrogram. Cutting a dendrogram at a certain level gives a set of clusters. Cutting at another level gives another set of clusters. How would you pick where to cut the dendrogram? Is there something we could consider an optimal point? If I look at a dendrogram across time as it changes, should I cut at the same point?&lt;/p&gt;&#xA;" OwnerUserId="1250" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-17T22:01:28.690" LastActivityDate="2010-10-18T21:01:49.780" Title="Where to cut a dendrogram?" Tags="&lt;clustering&gt;&lt;dendrogram&gt;" AnswerCount="4" CommentCount="1" FavoriteCount="2" />
  <row Id="3686" PostTypeId="2" ParentId="3685" CreationDate="2010-10-17T22:30:44.930" Score="1" ViewCount="" Body="&lt;p&gt;Perhaps one of the simplest methods would be a graphical representation in which the x-axis is the number of groups and the y-axis any evaluation metric as the distance or the similarity. In that plot you usually can observe two differentiated regions, being the x-axis value at the 'knee' of the line the 'optimal' number of cluster. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are also some statistics that could hepl on this task: Hubert' gamma, pseudo-t², pseudo-F or cubic clustering criteria (CCC) among others. &lt;/p&gt;&#xA;" OwnerUserId="221" LastEditorUserId="221" LastEditorDisplayName="" LastEditDate="2010-10-18T06:13:33.683" LastActivityDate="2010-10-18T06:13:33.683" CommentCount="0" />
  <row Id="3687" PostTypeId="2" ParentId="3682" CreationDate="2010-10-18T00:16:28.227" Score="4" ViewCount="" Body="&lt;p&gt;This paper does not appear to use $L^p$ norms in any essential way--every one of the results references the $L^1$ norm explicitly.  The problem itself determines which norm to use.  In this case interest focuses on the cardinality of multisets.  A multiset is represented as a vector of counts of its elements, whence its cardinality happens to be the same as its $L^1$ norm.  Often results proven for one norm may hold without any change needed in the proof for a wide range of $p$ (typically $1 \le p \le \infty$).  The opportunity for greater generality at no cost will lead many papers like this to talk about $L^p$ norms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$L^p$ norms come into their own in discussions of duality in Hilbert and Banach space theory.  Advanced, but introductory (it's not a contradiction!) &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0070542341&quot; rel=&quot;nofollow&quot;&gt;books on analysis&lt;/a&gt; usually cover this material thoroughly.  For an introduction to some of the relationships among these norms, read about the &lt;a href=&quot;http://en.wikipedia.org/wiki/H%C3%B6lder%27s_inequality&quot; rel=&quot;nofollow&quot;&gt;Holder Inequality&lt;/a&gt; and the &lt;a href=&quot;http://en.wikipedia.org/wiki/Minkowski_inequality&quot; rel=&quot;nofollow&quot;&gt;Minkowski Inequality&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-10-18T00:16:28.227" CommentCount="3" />
  <row Id="3688" PostTypeId="2" ParentId="3595" CreationDate="2010-10-18T03:05:26.233" Score="1" ViewCount="" Body="&lt;p&gt;SAS Enterprise Miner version 6.2 would have no problem handling 20 million observations, and a variety of models which can be adapted to your situation. The issue with SAS is usually the cost however. Here's a summary of what SAS EM can do:&#xA;&lt;a href=&quot;http://support.sas.com/documentation/cdl/en/whatsnew/62580/HTML/default/viewer.htm#emwhatsnew62.htm&quot; rel=&quot;nofollow&quot;&gt;SAS EM 6.2: What's New &lt;/a&gt;&lt;/p&gt;&#xA;" LastActivityDate="2010-10-18T03:12:18.987" CommunityOwnedDate="2010-10-18T03:05:26.233" />
  <row Id="3689" PostTypeId="1" AcceptedAnswerId="3690" CreationDate="2010-10-18T03:08:15.450" Score="5" ViewCount="129" Body="&lt;p&gt;I'm interested in exploring statistical models (or modifications thereof) designed to handle a specific type of problem. Due to my ignorance of statistical terminology, I can only describe this type of problem by (contrived) examples:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose we're interested in estimating the likelihood that a given cell phone customer will drop his/her service (churn) over the course of the next month (October). We are given cell phone user data from the last three months (July, August, September).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In estimating churn probability for October, we'll want to make use of a model that emphasizes inputs that are consistently predictive of churn each month, suppressing the importance of those whose coefficients vary heavily on a month-by-month basis. For example, &quot;market&quot; may be consistently predictive of churn, but an externality specific to one of the training datasets (e.g., marketing of new phone by a competitor in a specific market in August) may affect a coefficient in a way that won't necessarily apply to October. The market coefficient should be corrected/compressed/averaged when predicting October's churn likelihood.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's a less contrived example. In estimating the likelihood that a voter will vote in the 2012 election, we may use voter-level data from the 2000, 2004, and 2008 elections to train a logistic regression on turnout. If &quot;party&quot; is an input, 2000 and 2004 coefficients may differ significantly from 2008 party coefficient due to exogenous circumstances (a unique political environment). In estimating 2012 vote likelihood for a given voter, we'll want to trim/compress/average the party coefficient to &quot;smooth out&quot; election-year-specific differences that can't be easily quantified.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone point me in the right direction? Any guidance on how to better ask this question? What terminology should I be using? Thanks in advance for your help.&lt;/p&gt;&#xA;" OwnerUserId="1611" LastEditorUserId="1352" LastEditorDisplayName="" LastEditDate="2010-10-21T13:00:00.973" LastActivityDate="2010-10-21T13:00:00.973" Title="Coefficient / model averaging to control for exogent circumstances in predicting future events" Tags="&lt;time-series&gt;&lt;logistic-regression&gt;&lt;classification&gt;&lt;forecasting&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="3690" PostTypeId="2" ParentId="3689" CreationDate="2010-10-18T03:53:26.107" Score="4" ViewCount="" Body="&lt;p&gt;I am not sure you need any special tricks as long as the relevant factors are captured by the model. To keep things simple I will discuss the issue in the context of linear regression. The same intuition carries over to the time series setting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose, that you want to predict the monthly sales of cell phones for brand X and to illustrate the ideas suppose that you use linear regression to do so. The issue you have is that there are some factors that impact sales every month (e.g., weather of the month to take a silly example) and some factors that are idiosyncratic to a month (e.g., launch of a new cell phone). You wish to account for both factors consistently.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Denote the factors that impact sales every month by $C_m$ (where 'C' stands for common and 'm' for month) and idiosyncratic factors by $I_m$ (where I stands for idiosyncratic and m for month). Then, your model would be:  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$S_m = \beta_c C_M + \beta_i I_m + \epsilon$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the months where there is no idiosyncratic factors $I_m$ would be 0 and hence the impact on sales would be captured by $\beta_c$ and in the months when you &lt;em&gt;do&lt;/em&gt; have idiosyncratic factors the impact on sales would be $\beta_c + \beta_i$ with $\beta_i$ presumably being negative if the idiosyncratic factor dampens sales for brand X. Thus, the coefficients 'automatically' adjust themselves provided all the necessary factors are present in the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, it seems to me you do not have to worry about the issue you raise if you do account for all factors whether they are idiosyncratic to a time period or common across time periods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit in response to comment&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your example of concert and increased cell phone usage by young consumers is an example of an interaction effect. To put it in different words your example is saying that the effect of the concert on cell phone use is higher for younger consumers rather than older consumers. Basically, your model is saying the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Minutes = beta1 age + beta2 * concert + beta3 * age * concert + error&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, the beta3 parameter will capture the differential impact how younger/older consumers respond to the concert.&lt;/p&gt;&#xA;" LastActivityDate="2010-10-18T14:21:03.917" CommentCount="3" />
  <row Id="3691" PostTypeId="2" ParentId="3685" CreationDate="2010-10-18T04:49:46.067" Score="1" ViewCount="" Body="&lt;p&gt;A joke: &quot;there are 10 kinds of computer programmers: those that understand binary, and those that don't&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There isn't really an answer. It's somewhere between 1 and N.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, you can think about it from a profit perspective.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, in marketing one uses segmentation, which is much like clustering. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A message (an advertisement or letter, say) that is tailored for each individual will have the highest response rate. A generic message tailored to the average will have the lowest response rate. Having say three messages tailored to three segments will be somewhere in between. This is the revenue side.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A message that is tailored to each individual will have the highest cost. A generic message tailored to the average will have the lowest cost. Three messages tailored to three segments will be somewhere in between. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say paying a writer to write a custom message costs 1000, two cost 2000 and so on. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say by using one message, your revenue will be 5000. If you segmented your customers into 2 segments, and wrote tailored messages to each segment, your response rate will be higher. Say revenues are now 7500. With three segments, a slightly higher response rate, and your revenues are 9000. One more segment, and you're at 9500. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To maximize profit, keep segmenting until the marginal revenue from segmenting equals the marginal cost of segmenting. In this example, you would use three segments to maximize profit. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Segments  Revenue  Cost  Profit&#xA;1         5000     1000  4000&#xA;2         7500     2000  5500&#xA;3         9000     3000  6000&#xA;4         9500     4000  5500&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="74" LastActivityDate="2010-10-18T04:49:46.067" />
  <row Id="3692" PostTypeId="2" ParentId="3685" CreationDate="2010-10-18T05:56:44.473" Score="3" ViewCount="" Body="&lt;p&gt;There is no definitive answer since cluster analysis is essentially an exploratory approach; the interpretation of the resulting hierarchical structure is context-dependent and often several solutions are equally good from a theoretical point of view.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Several clues were given in a related question, &lt;a href=&quot;http://stats.stackexchange.com/questions/2597/what-stop-criteria-for-agglomerative-hierarchical-clustering-are-used-in-practice&quot;&gt;What stop-criteria for agglomerative hierarchical clustering are used in practice?&lt;/a&gt; I generally use visual criteria, e.g. silhouette plots, and some kind of numerical criteria, like Dunn’s validity index, Hubert's gamma, G2/G3 coefficient, or the corrected Rand index. Basically, we want to know how well the original distance matrix is approximated in the cluster space, so a measure of the &lt;a href=&quot;http://en.wikipedia.org/wiki/Cophenetic_correlation&quot; rel=&quot;nofollow&quot;&gt;cophenetic correlation&lt;/a&gt; is also useful. I also use k-means, with several starting values, and the &lt;a href=&quot;http://gremlin1.gdcb.iastate.edu/MIP/gene/MicroarrayData/gapstatistics.pdf&quot; rel=&quot;nofollow&quot;&gt;gap statistic&lt;/a&gt; to determine the number of clusters that minimize the within-SS. The concordance with Ward hierarchical clustering gives an idea of the stability of the cluster solution (You can use &lt;code&gt;matchClasses()&lt;/code&gt; in the &lt;a href=&quot;http://cran.r-project.org/web/packages/e1071/index.html&quot; rel=&quot;nofollow&quot;&gt;e1071&lt;/a&gt; package for that).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You will find useful resources in the CRAN Task View &lt;a href=&quot;http://cran.r-project.org/web/views/Cluster.html&quot; rel=&quot;nofollow&quot;&gt;Cluster&lt;/a&gt;, including &lt;a href=&quot;http://cran.r-project.org/web/packages/pvclust/index.html&quot; rel=&quot;nofollow&quot;&gt;pvclust&lt;/a&gt;, &lt;a href=&quot;http://cran.r-project.org/web/packages/fpc/index.html&quot; rel=&quot;nofollow&quot;&gt;fpc&lt;/a&gt;, &lt;a href=&quot;http://cran.r-project.org/web/packages/clValid/index.html&quot; rel=&quot;nofollow&quot;&gt;clv&lt;/a&gt;, among others. Also worth to give a try is the &lt;a href=&quot;http://cran.r-project.org/web/packages/clValid/index.html&quot; rel=&quot;nofollow&quot;&gt;clValid&lt;/a&gt; package (&lt;a href=&quot;http://www.jstatsoft.org/v25/i04/paper&quot; rel=&quot;nofollow&quot;&gt;described&lt;/a&gt; in the &lt;em&gt;Journal of Statistical Software&lt;/em&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, if your clusters change over time, this is a bit more tricky; why choosing the first cluster-solution rather than another? Do you expect that some individuals move from one cluster to another as a result of an underlying process evolving with time?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are some measure that try to match clusters that have a maximum absolute or relative overlap, as was suggested to you in your preceding question. Look at &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.164.6189&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;Comparing Clusterings - An Overview&lt;/a&gt; from Wagner and Wagner.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-18T21:01:49.780" LastActivityDate="2010-10-18T21:01:49.780" CommentCount="1" />
  <row Id="3693" PostTypeId="2" ParentId="3685" CreationDate="2010-10-18T07:16:33.477" Score="0" ViewCount="" Body="&lt;p&gt;I am in agree with chl. Cluster analyses are exploratory approaches and the interpretation of results, for this particular case the optimal number of clusters, depends on your context. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, in my work is common to used cluster analyses to classify individuals based on several characteristics and sometimes the number of clusters is preset. In this case, our objective is to find the set of classificatory variables that best distinguishes the individuals belonging to different clusters.&lt;/p&gt;&#xA;" OwnerUserId="221" LastActivityDate="2010-10-18T07:16:33.477" />
  <row Id="3694" PostTypeId="2" ParentId="3614" CreationDate="2010-10-18T07:19:52.593" Score="3" ViewCount="" Body="&lt;p&gt;Here's another way to calculate the probability distribution of the sum of two dice by hand using convolutions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To keep the example really simple, we're going to calculate the probability distribution of the sum of a three-sided die (d3) whose random variable we will call X and a two-sided die (d2) whose random variable we'll call Y.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You're going to make a table. Across the top row, write the probability distribution of X (outcomes of rolling a fair d3). Down the left column, write the probability distribution of Y (outcomes of rolling a fair d2).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You're going to construct the &lt;a href=&quot;http://en.wikipedia.org/wiki/Outer_product&quot; rel=&quot;nofollow&quot;&gt;outer product&lt;/a&gt; of the top row of probabilities with the left column of probabilities. For example, the lower-right cell will be the product of Pr[X=3]=1/3 times Pr[Y=2]=1/2 as shown in the accompanying figure. In our simplistic example, all the cells equal 1/6.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Next, you're going to sum along the oblique lines of the outer-product matrix as shown in the accompanying diagram. Each oblique line passes through one-or-more cells which I've colored the same: The top line passes through one blue cell, the next line passes through two red cells, and so on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/WQWgE.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each of the sums along the obliques represents a probability in the resulting distribution. For example, the sum of the red cells equals the probability of the two dice summing to 3. These probabilities are shown down the right side of the accompanying diagram.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This technique can be used with any two discrete distributions with finite support. And you can apply it iteratively. For example, if you want to know the distribution of three six-sided dice (3d6), you can first calculate 2d6=d6+d6; then 3d6=d6+2d6.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a free (but closed license) programming language called &lt;a href=&quot;http://www.jsoftware.com/&quot; rel=&quot;nofollow&quot;&gt;J&lt;/a&gt;. Its an array-based language with its roots in APL. It has builtin operators to perform outer products and sums along the obliques in matrices, making the technique I illustrated quite simple to implement.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the following J code, I define two verbs. First the verb &lt;code&gt;d&lt;/code&gt; constructs an array representing the pmf of an s-sided die. For example, &lt;code&gt;d 6&lt;/code&gt; is the pmf of a 6-sided die. Second, the verb &lt;code&gt;conv&lt;/code&gt; finds the outer product of two arrays and sums along the oblique lines. So &lt;code&gt;conv~ d 6&lt;/code&gt; prints out the pmf of 2d6:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;d=:$%&#xA;conv=:+//.@(*/)&#xA;|:(2+i.11),:conv~d 6&#xA; 2 0.0277778&#xA; 3 0.0555556&#xA; 4 0.0833333&#xA; 5  0.111111&#xA; 6  0.138889&#xA; 7  0.166667&#xA; 8  0.138889&#xA; 9  0.111111&#xA;10 0.0833333&#xA;11 0.0555556&#xA;12 0.0277778&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As you can see, J is cryptic, but terse.&lt;/p&gt;&#xA;" LastActivityDate="2010-10-18T07:19:52.593" CommentCount="1" />
  <row Id="3695" PostTypeId="1" AcceptedAnswerId="3697" CreationDate="2010-10-18T07:51:12.160" Score="6" ViewCount="426" Body="&lt;p&gt;When plotting a boxplot with python matplotblib, the lines halfway the plot is the median of the distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a possibility to instead have the line at the average. Or to plot it next to it in a different style.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, because it is common for the line to be the median, will it really confuse my readers if I make it the average (off course I will add a note what the middle line is)?&lt;/p&gt;&#xA;" OwnerUserId="190" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-10-18T12:02:05.757" LastActivityDate="2010-10-18T12:02:05.757" Title="Show average instead of median in boxplot" Tags="&lt;data-visualization&gt;&lt;python&gt;&lt;matplotlib&gt;&lt;boxplot&gt;" AnswerCount="2" />
  <row Id="3696" PostTypeId="2" ParentId="3695" CreationDate="2010-10-18T08:05:49.140" Score="12" ViewCount="" Body="&lt;p&gt;To answer your second question: Yes, I think it will be confusing to put the line at the mean instead of the median. The precise rules controlling the length of the 'whiskers' (if any) and treatment of outliers vary, but everyone keeps to Tukey's use of the box as displaying the median and lower and upper quartiles. For highly skew distributions, the mean could be &lt;em&gt;outside&lt;/em&gt; the box, which would look very odd. Common usage is that the median goes with the interquartile range, while the mean goes with standard deviation (or standard error of the mean if you're interested in inference rather than data description). If you want to show the mean visually, i'd use a different symbol to display it to avoid confusion. &lt;/p&gt;&#xA;" OwnerUserId="449" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-10-18T08:28:14.397" LastActivityDate="2010-10-18T08:28:14.397" CommentCount="1" />
  <row Id="3697" PostTypeId="2" ParentId="3695" CreationDate="2010-10-18T08:40:10.110" Score="14" ViewCount="" Body="&lt;p&gt;This code makes the boxplots then places a circle marking the mean for each box.  You can use a different symbol by specifying the &lt;a href=&quot;http://matplotlib.sourceforge.net/api/pyplot_api.html?highlight=scatter#matplotlib.pyplot.scatter&quot; rel=&quot;nofollow&quot;&gt;marker&lt;/a&gt; argument in the call to &lt;code&gt;scatter&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import pylab&#xA;&#xA;# 3 boxes&#xA;data = [[np.random.rand(100)] for i in range(3)]&#xA;pylab.boxplot(data)&#xA;&#xA;# mark the mean    &#xA;means = [np.mean(x) for x in data]&#xA;pylab.scatter([1, 2, 3], means)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/nTpJH.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-10-18T08:40:10.110" CommentCount="2" />
  <row Id="3698" PostTypeId="1" AcceptedAnswerId="3770" CreationDate="2010-10-18T09:08:51.310" Score="2" ViewCount="253" Body="&lt;p&gt;I am trying to understand a published analysis. This is the data of interest:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;pre&gt;&#xA;      D1&gt;0    D1&amp;lt;0&#xA;D2&gt;0   7        2     9&#xA;D2&amp;lt;0   9        15   24&#xA;total  16       17   33&#xA;&lt;/pre&gt;The author notes that 17/33 is 51.5% and states:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&quot;we expect about 50% of the D1's to be negative, and that is what we actually observe here (z=.08, p=n.s)&quot;.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Now I assume that the z comparison is  17/33 vs 16.5/33 (chance?) and that the statistic employed is the z-ratio for the significance of the difference between two proportions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However my own calculation for this comes out as z=.123. Can anyone help with the discrepancy?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;UPDATE: The author has responded and admits that the published z is an error. He gives the corrected answer as z=0.17.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am still not sure how he worked it out though:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;We are not considering the difference between two proportions, since they&#xA;   are not independent 16/33 = 17/33 -1. We are interested whether the&#xA;   proportion of  D1&amp;lt;0 significantly deviated from the expected value of 0.5&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The nearest I can get is R prop.test function. Is this directly comparable?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;prop.test(c(17, 16.5), c( 33, 33))&#xA;&#xA;2-sample test for equality of proportions with continuity correction&#xA;data:  c(17, 16.5) out of c(33, 33) &#xA;X-squared = 0, df = 1, p-value = 1&#xA;alternative hypothesis: two.sided &#xA;95 percent confidence interval:&#xA; -0.2411995  0.2715025 &#xA;sample estimates:&#xA;   prop 1    prop 2 &#xA;0.5151515 0.5000000 &#xA;&lt;/pre&gt;&#xA;" OwnerUserId="1614" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-10-28T10:40:02.167" LastActivityDate="2010-10-28T10:40:02.167" Title="Understanding published data: z-ratio for proportions" Tags="&lt;contingency-tables&gt;" AnswerCount="2" CommentCount="4" />
  <row Id="3699" PostTypeId="2" ParentId="3516" CreationDate="2010-10-18T09:29:56.657" Score="0" ViewCount="" Body="&lt;p&gt;Whuber, ch1 &amp;amp; esp. glassy, thank you all for the responses. Helps me get a better perspective. Appreciate it.&lt;/p&gt;&#xA;" LastActivityDate="2010-10-18T09:29:56.657" CommentCount="1" />
  <row Id="3700" PostTypeId="2" ParentId="3636" CreationDate="2010-10-18T12:24:04.177" Score="0" ViewCount="" Body="&lt;p&gt;Based on Shane's answer I entered some data into Excel.  What I came up with is that I need to multiple the square of the different from the mean by the number of seconds until the next sample.  This assumes the reading was steady just until the next reading.&lt;/p&gt;&#xA;" OwnerUserId="1595" LastActivityDate="2010-10-18T12:24:04.177" />
  <row Id="3701" PostTypeId="2" ParentId="3698" CreationDate="2010-10-18T12:27:47.530" Score="1" ViewCount="" Body="&lt;p&gt;I'm not sure the discrepancy is worth worrying about. The exact p-value is clearly 1 for a 2-sided test of p=0.5 given 17 positive responses out of 33, as there's no integer closer to 33/2 than 17. With small or moderate N as here there's no good reason for not doing the exact test (even without a PC, as the cdf of the binomial is commonly included in statistical tables). With a modern PC and decent software it's no problem even for large N, but for large N the resulting p-value will be very similar to that from a z-test anyway.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-10-18T12:27:47.530" CommentCount="1" />
  <row Id="3702" PostTypeId="1" CreationDate="2010-10-18T13:48:33.180" Score="1" ViewCount="243" Body="&lt;p&gt;I am newbee and i am trying with functional data analysis. I have a 8x11 matrix data, how can i input into R as an object in this form:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;$hgtm&#xA;&#xA;     boy01 boy02 boy03 boy04 boy05&#xA;1     81.3  76.2  76.8  74.1  74.2 &#xA;1.25  84.2  80.4  79.8  78.4  76.3 &#xA;1.5   86.4  83.2  82.6  82.6  78.3  &#xA;1.75  88.9  85.4  84.7  85.4  80.3 &#xA;2     91.4  87.6  86.7  88.1  82.2  &#xA;3    101.1  97.0  94.2  98.6  89.4 &#xA;4    109.5 104.6 100.4 104.4  96.9 &#xA;5    115.8 112.3 107.1 111.0 104.1 &#xA;&#xA;$hgtf&#xA;&#xA;     girl01 girl02 girl03 girl04 girl05 &#xA;1      76.2   74.6   78.2   77.7     76   &#xA;1.25   80.4   78.0   81.8   80.5     80   &#xA;1.5    83.3   82.0   85.4   83.3     83  &#xA;1.75   85.7   86.9   87.9   87.0     86   &#xA;2      87.7   90.0   89.6   90.3     89   &#xA;3      96.0   94.9   97.1   98.6     96  &#xA;4     103.8  102.1  109.2  106.3    103   &#xA;5     110.7  109.2  116.2  113.9    110  &#xA;&#xA; $age&#xA;&#xA; [1]  1.00  1.25  1.50  1.75  2.00  3.00  4.00  5.00&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;very much appreciated,&lt;/p&gt;&#xA;" OwnerUserId="1615" LastEditorUserId="485" LastEditorDisplayName="" LastEditDate="2010-10-18T19:24:35.417" LastActivityDate="2010-10-18T19:24:35.417" Title="Data manipulation in R for functional data analysis" Tags="&lt;r&gt;&lt;longitudinal&gt;" AnswerCount="2" CommentCount="7" />
  <row Id="3703" PostTypeId="2" ParentId="3702" CreationDate="2010-10-18T13:56:05.490" Score="2" ViewCount="" Body="&lt;p&gt;It looks like you want to use a named &lt;strong&gt;list&lt;/strong&gt;, since each object is of different dimensions above.  Here is an example with some dummy data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;hgtm &amp;lt;- matrix(1:100, ncol=5)&#xA;hgtf &amp;lt;- matrix(1:100, ncol=5)&#xA;age &amp;lt;- 1:10&#xA;namvan &amp;lt;- list(hgtm=hgtm, hgtf=hgtf, age=age)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now you can reference each object by name:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; str(namvan)&#xA;List of 3&#xA; $ hgtm: int [1:20, 1:5] 1 2 3 4 5 6 7 8 9 10 ...&#xA; $ hgtf: int [1:20, 1:5] 1 2 3 4 5 6 7 8 9 10 ...&#xA; $ age : int [1:10] 1 2 3 4 5 6 7 8 9 10&#xA;&amp;gt; namvan$age&#xA; [1]  1  2  3  4  5  6  7  8  9 10&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-10-18T13:56:05.490" CommentCount="1" />
  <row Id="3704" PostTypeId="1" AcceptedAnswerId="3712" CreationDate="2010-10-18T14:21:06.687" Score="2" ViewCount="227" Body="&lt;p&gt;I posted this question earlier and am rewriting it in hopes of getting some guidance. I am using a weighted regression (after propensity score matching) to obtain estimates of the effects of a treatment (treat) on an outcome for different income quintiles. I include interactions of the quintiles and the treatment in the equation, as shown below:  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;regress outcome treat quintile2 quin3 quin4 quin5 treatXquin2 treatXquin3 treatXquin4   treatXquin5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To obtain the effect of treatment on quintile2 I add the coefficient on treat with the coefficient on treatXquin2. However, what do I do with the standard errors? Can I simply add the standard errors on the two coefficients together? Can I add the t-stats together? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any advice would be much appreciated. I'm not a stats expert but have a basic understanding of econometrics. &lt;/p&gt;&#xA;" OwnerUserId="834" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-10-18T17:48:59.000" LastActivityDate="2010-10-18T22:08:54.650" Title="Adding coefficients to obtain interaction effects - can I add standard errors?" Tags="&lt;regression&gt;&lt;standard-error&gt;&lt;multivariate-analysis&gt;&lt;stata&gt;" AnswerCount="2" CommentCount="4" />
  <row Id="3705" PostTypeId="2" ParentId="3702" CreationDate="2010-10-18T14:59:44.807" Score="2" ViewCount="" Body="&lt;p&gt;As to how you would get the data into R and into one of these formats, we'd need to know more about what your input file looks like and the format that it is in.  However, here are some general tips on formatting the type of data that you have for analysis in any system.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Singer (Applied Longitudinal Data Analysis) suggests two generally useful layouts for the statistical analysis of longitudinal data: the person-level (mutlivariate) structure or the person-period (univariate) structure.  The latter is generally preferrable for a number of reasons.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The person-level data structure (or the multivariate format) contains one row of data for each observational unit (such as persons) and a variable for each measurement period.  Age would not be included in the data set and would be implicit in the levels of your time factor (e.g., in a repeated measures ANOVA).  This structure can lead naturally to summaries that aren't very meaningful, is less efficient than it could be, and cannot account for your unequally spaced observations (differing age intervals between observations) or time-varying covariates. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;That data setup might look something like this...&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Mutlivariate&#xA;    ID  Gender  height1 height2 height3 height4 height5&#xA;    1   Boy     76.2   74.6   78.2   77.7     76&#xA;    2   Boy     80.4   78.0   81.8   80.5     80&#xA;    3   Boy     83.3   82.0   85.4   83.3     83&#xA;    4   Girl    96.0   94.9   97.1   98.6     96&#xA;    5   Girl    87.7   90.0   89.6   90.3     89&#xA;    6   Girl    85.7   86.9   87.9   87.0     86&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;A preferable layout is often the person-period layout (or the univariate format) where each individual has a record for each time for which they were observed.  The person-period dataset has a number of advantages.  First, it leads to more natural summaries of the data, e.g. getting an average by group, by time or by group and time is now straight forward.  Second, the dataset will accommodate entry of unequal intervals in the time dimension, such as you have here.  In addition, if you have them, you could add columns for any other demographic covariates and these could differ over time.  Also, data in this format is prepared for modern analytical techniques such as multilevel modeling.  Finally, the univariate data structure is consistent with good practice in database design and normalization, increasing efficiency and making it appropriate for the typical query structure.    &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The univariate layout would look something like this...&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Univariate&#xA;    ID  Age Gender  Height&#xA;    1   1   Boy     76.2&#xA;    1   1.5 Boy     74.6&#xA;    1   3   Boy     78.2&#xA;    1   5   Boy     77.7&#xA;    2   1   Girl    80.4&#xA;    2   1.5 Girl    81.8&#xA;    2   3   Girl    80.5&#xA;    2   5   Girl    80&#xA;    3   1   Boy     115.8&#xA;    3   1.5 Boy     112.3&#xA;    3   3   Boy     111.0&#xA;    3   5   Boy     104.1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="485" LastEditorUserId="485" LastEditorDisplayName="" LastEditDate="2010-10-18T18:24:25.100" LastActivityDate="2010-10-18T18:24:25.100" CommentCount="3" />
  <row Id="3706" PostTypeId="2" ParentId="3704" CreationDate="2010-10-18T15:09:48.413" Score="-1" ViewCount="" Body="&lt;p&gt;I am quite sure it's fine to just add the standard errors on the two coefficients together. I can then obtain the t-stats by dividing the sum of the coefficients by the sum of the standard errors. If anyone disagrees please let me know. &lt;/p&gt;&#xA;" OwnerUserId="834" LastActivityDate="2010-10-18T15:09:48.413" CommentCount="2" />
  <row Id="3707" PostTypeId="1" AcceptedAnswerId="3714" CreationDate="2010-10-18T15:10:23.190" Score="3" ViewCount="298" Body="&lt;p&gt;I plotted below the standarized results of: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;RAND()&lt;/li&gt;&#xA;&lt;li&gt;RAND() * RAND() &lt;/li&gt;&#xA;&lt;li&gt;...&lt;/li&gt;&#xA;&lt;li&gt;RAND() * RAND()  * RAND() * RAND()  * RAND() * RAND()&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;It seems that the results are getting to zero, is that because you're multiplying a bunch of numbers close to zero together, or is there another explanation?&#xA;&lt;img src=&quot;http://i.imgur.com/5asaM.jpg&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is no real motivation behind this. Working on a spreadsheet earlier today, I was in a situation where i was multiplying a constant * Rand() * Rand() .. each rand() corresponded to a different variable (column), so I wanted to find out what goes on if you multiply a bunch of random numbers together.&lt;/p&gt;&#xA;" OwnerUserId="59" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-10-18T17:43:41.720" LastActivityDate="2010-10-19T10:10:46.127" Title="Difference between Excel's RAND(), RAND()*RAND(), etc ..." Tags="&lt;probability&gt;&lt;uniform&gt;&lt;monte-carlo&gt;&lt;excel&gt;&lt;central-limit-theorem&gt;" AnswerCount="6" CommentCount="1" />
  <row Id="3708" PostTypeId="1" AcceptedAnswerId="3720" CreationDate="2010-10-18T15:12:10.027" Score="6" ViewCount="346" Body="&lt;p&gt;Hi All,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a data set of 100 geographic regions for which the unemployment rate has been observed during the last 9 years. Now, I want to simulate/forecast from this data the next year unemployment rate for all the 100 regions. &#xA;What would be a suitable approach? I was thinking to use an auto-regressive mixed effect model, but I fear that I do not have enough data...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regards &lt;/p&gt;&#xA;" OwnerUserId="1443" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-10-18T15:36:51.290" LastActivityDate="2010-10-19T15:23:18.643" Title="Forecasting unemployment rate" Tags="&lt;time-series&gt;&lt;mcmc&gt;&lt;mixed-model&gt;&lt;forecasting&gt;" AnswerCount="2" CommentCount="4" FavoriteCount="1" />
  <row Id="3709" PostTypeId="2" ParentId="3707" CreationDate="2010-10-18T15:18:01.300" Score="2" ViewCount="" Body="&lt;p&gt;Since MS-Excel's Rand() gives a uniformly distributed number between 0 and 1, how did you get ranges from almost -2 to almost 4?&lt;/p&gt;&#xA;" OwnerUserId="247" LastActivityDate="2010-10-18T15:18:01.300" CommentCount="6" />
  <row Id="3710" PostTypeId="2" ParentId="3707" CreationDate="2010-10-18T15:18:11.330" Score="4" ViewCount="" Body="&lt;p&gt;&quot;In Excel, the Rand function returns a random number that is greater than or equal to 0 and less than 1. The Rand function returns a new random number each time your spreadsheet recalculates.&quot; -http://www.techonthenet.com/excel/formulas/rand.php&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because RAND() is always less than one and greater than zero, multiplying it by itself will make it smaller. As you do that over and over, you will get closer to zero. If you want something that gives you a random number between 0 and a, you can do a*RAND() instead.&lt;/p&gt;&#xA;" OwnerUserId="1118" LastActivityDate="2010-10-18T15:18:11.330" />
  <row Id="3711" PostTypeId="2" ParentId="3707" CreationDate="2010-10-18T15:29:33.363" Score="3" ViewCount="" Body="&lt;p&gt;I am not sure why your graph has values from -2 to 4 but for what it is worth here is the answer to the text of your question:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose that $U \sim U[0,1]$. Then the cdf of $U$ is given by $F(u) = u$ for $u \in (0,1)$ and 1 otherwise.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you multiply different iid realizations of the random draws you are essentially computing the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Y = U^n$ where $n$ is the number of times you are multiplying the random draws.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, the corresponding cdf is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$F(y) = P(Y \le y)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;i.e., &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$F(y) = P(U^n \le y)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;i.e.,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$F(y) = P(U \le y^{1/n})$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;i.e., &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$F(y) = y^{1/n}$ for $y \in (0,1)$ and 1 otherwise.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The above cdf of $Y$ converges to a &lt;a href=&quot;http://en.wikipedia.org/wiki/Dirac_delta_function&quot; rel=&quot;nofollow&quot;&gt;dirac-delta&lt;/a&gt; function on $Y=0$ as $n \rightarrow \infty$. Thus, $E(y) \rightarrow 0$ as $n \rightarrow \infty$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The above convergence is also related to &lt;a href=&quot;http://en.wikipedia.org/wiki/Stochastic_dominance#First-order_stochastic_dominance&quot; rel=&quot;nofollow&quot;&gt;first-order stochastic dominance&lt;/a&gt; in the following sense: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose that $n_1 &gt; n_2$. Then, it is the case that:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$F(y|n_1) \ge F(y|n_2)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Intuitively, the above result states that: In visual terms as $n$ increases the cdf of $Y$ shifts to the right. This happens because the pdf associated with $Y$ starts concentrating at the lower end of the interval $[0,1]$ and asymptotically all the pdf concentrates at 0 which explains the observed behavior.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;General Case&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;@whuber's comment to this answer gives the solution when $Y$ is the product of $n$ independent, &lt;em&gt;different&lt;/em&gt; random variables drawn from [0,1].&lt;/p&gt;&#xA;" LastActivityDate="2010-10-18T19:01:39.267" CommentCount="6" />
  <row Id="3712" PostTypeId="2" ParentId="3704" CreationDate="2010-10-18T15:47:22.743" Score="4" ViewCount="" Body="&lt;p&gt;Your impact of treatment depends updon quintile status. After running the 'regress' command, type, 'lincom treat + treatXquin2' to get the impact of treatment for those in quintile 2. This gives a confidence interval and standard error for this estimate. Note that, as is typical in Stata, using the names of the variables in a post-estimation step refers to the coefficients on those variables, not the variables themselves.&lt;/p&gt;&#xA;" OwnerUserId="401" LastEditorUserId="401" LastEditorDisplayName="" LastEditDate="2010-10-18T22:08:54.650" LastActivityDate="2010-10-18T22:08:54.650" CommentCount="5" />
  <row Id="3713" PostTypeId="1" AcceptedAnswerId="3716" CreationDate="2010-10-18T15:58:40.990" Score="14" ViewCount="585" Body="&lt;p&gt;When using cluster analysis on a data set to group similar cases, one needs to choose among a large number of clustering methods and measures of distance.  Sometimes, one choice might influence the other, but there are many possible combinations of methods.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone have any recommendations on how to choose among the various clustering algorithms and distance measures?  How is this related to the nature of the variables (e.g., categorical or numerical) and the clustering problem?  Is there are optimal technique?&lt;/p&gt;&#xA;" OwnerUserId="485" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-11-07T11:07:48.293" LastActivityDate="2010-11-09T16:38:14.217" Title="Choosing clustering method" Tags="&lt;clustering&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="13" />
  <row Id="3714" PostTypeId="2" ParentId="3707" CreationDate="2010-10-18T16:00:46.523" Score="5" ViewCount="" Body="&lt;p&gt;Standardization is good, but it's not the right standardization for this situation.  It helps to see that multiplying values of RAND() is the same as adding their logarithms (followed by a subsequent exponentiation).  Because the different calls to RAND() are supposed to be independent, those logarithms are still independently distributed.  As a simple calculation shows, their common distribution actually has a mean and variance.  (In fact, its negative is an exponential distribution.)  The Central Limit Theorem applies.  It says that the &lt;em&gt;logs&lt;/em&gt;, suitably standardized, converge to a normal distribution.  We conclude that these products--standardized to have a constant &lt;em&gt;geometric&lt;/em&gt; mean and constant &lt;em&gt;geometric&lt;/em&gt; variance--are converging to the exponential of a normally distributed variable: that is, a &lt;a href=&quot;http://en.wikipedia.org/wiki/Log-normal_distribution&quot; rel=&quot;nofollow&quot;&gt;lognormal distribution&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-10-18T16:06:05.273" LastActivityDate="2010-10-18T16:06:05.273" />
  <row Id="3715" PostTypeId="1" CreationDate="2010-10-18T17:19:09.747" Score="4" ViewCount="243" Body="&lt;p&gt;I'd like to use Bayes' Theorem on data obtained through a small random sample, and I want to use Agresti-Coull (or any other alternative technique) to know how big the uncertainty is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is Bayes' Theorem:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(A|B) = \frac{P(B|A)\cdot P(A)}{P(B)}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, all the data I have on this system is obtained from small random samples, so there's a large uncertainty involved with all three variables, $P(B|A)$, $P(A)$ and $P(B)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've been using Agresti-Coull to obtain both the value and the uncertainty for each of these three variables. (I represent the &lt;code&gt;number+-uncertainty&lt;/code&gt; as a &lt;a href=&quot;http://packages.python.org/uncertainties/#an-easy-to-use-calculator&quot; rel=&quot;nofollow&quot;&gt;&lt;code&gt;ufloat&lt;/code&gt;&lt;/a&gt; object using the &lt;a href=&quot;http://packages.python.org/uncertainties/&quot; rel=&quot;nofollow&quot;&gt;&lt;code&gt;uncertainties&lt;/code&gt;&lt;/a&gt; package.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But using Agresti-Coull three times separately for these three variables is a problem; They are dependent on each other. So I've been getting impossible results. For example, if you let $P(B)$'s uncertainty pull it downward, and the respective uncertainties of $P(B|A)$ and $P(A)$ pull them upwards, you get a total probability bigger than one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a way to do Agresti-Coull-style approximation on the whole Bayes expression instead of doing it on the three pieces separately?&lt;/p&gt;&#xA;" OwnerUserId="1523" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-10-18T17:57:01.283" LastActivityDate="2010-11-02T21:34:53.780" Title="Bayes' Theorem and Agresti-Coull: Will it blend?" Tags="&lt;bayesian&gt;&lt;python&gt;&lt;approximation&gt;&lt;uncertainty&gt;" AnswerCount="3" CommentCount="7" FavoriteCount="1" />
  <row Id="3716" PostTypeId="2" ParentId="3713" CreationDate="2010-10-18T17:24:05.970" Score="12" ViewCount="" Body="&lt;p&gt;There is no definitive answer to your question, as even within the same method the choice of the distance to represent individuals (dis)similarity may yield different result, e.g. when using euclidean vs. squared euclidean in hierarchical clustering. As an other example, for binary data, you can choose the Jaccard index as a measure of similarity and proceed with classical hierarchical clustering; but there are alternative approaches, like the Mona (&lt;a href=&quot;http://stat.ethz.ch/R-manual/R-devel/library/cluster/html/mona.html&quot; rel=&quot;nofollow&quot;&gt;Monothetic Analysis&lt;/a&gt;) algorithm which only considers one variable at a time, while other hierarchical approaches (e.g. classical HC, Agnes, Diana) use all variables at each step. The k-means approach has been extended in various way, including partitioning around medoids (PAM) or representative objects rather than centroids (Kaufman and Rousseuw, 1990), or fuzzy clustering (Chung and Lee, 1992). For instance, the main difference between the k-means and PAM is that PAM minimizes a sum of dissimilarities rather than a sum of squared euclidean distances; fuzzy clustering allows to consider &quot;partial membership&quot; (we associate to each observation a weight reflecting class membership). And for methods relying on a probabilistic framework, or so-called model-based clustering (or &lt;a href=&quot;http://spitswww.uvt.nl/~vermunt/ermss2004f.pdf&quot; rel=&quot;nofollow&quot;&gt;latent profile analysis&lt;/a&gt; for the psychometricians), there is a great package: &lt;a href=&quot;http://www.stat.washington.edu/mclust/&quot; rel=&quot;nofollow&quot;&gt;Mclust&lt;/a&gt;. So definitively, you need to consider how to define the resemblance of individuals as well as the method for linking individuals together (recursive or iterative clustering, strict or fuzzy class membership, unsupervised or semi-supervised approach, etc.).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Usually, to assess cluster stability, it is interesting to compare several algorithm which basically &quot;share&quot; some similarity (e.g. k-means and hierarchical clustering, because euclidean distance work for both). For assessing the concordance between two cluster solutions, some pointers were suggested in response to this question, &lt;a href=&quot;http://stats.stackexchange.com/questions/3685/where-to-cut-a-dendrogram&quot;&gt;Where to cut a dendrogram?&lt;/a&gt; (see also the cross-references for other link on this website). If you are using R, you will see that several packages are already available in Task View on Cluster Analysis, and several packages include vignettes that explain specific methods or provide case studies. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www-users.cs.umn.edu/~kumar/dmbook/ch8.pdf&quot; rel=&quot;nofollow&quot;&gt;Cluster Analysis: Basic Concepts and Algorithms&lt;/a&gt; provides a good overview of several techniques used in Cluster Analysis.&#xA;As for a good recent book with R illustrations, I would recommend chapter 12 of Izenman, &lt;em&gt;Modern Multivariate Statistical Techniques&lt;/em&gt; (Springer, 2008). A couple of other standard references is given below:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Cormack, R., 1971. A review of classification. &lt;em&gt;Journal of the Royal Statistical Society, A&lt;/em&gt; 134, 321–367.&lt;/li&gt;&#xA;&lt;li&gt;Everitt, B., 1974. &lt;em&gt;Cluster analysis&lt;/em&gt;. London: Heinemann Educ. Books.&lt;/li&gt;&#xA;&lt;li&gt;Gordon, A., 1987. A review of hierarchical classification. &lt;em&gt;Journal of the Royal Statistical Society, A&lt;/em&gt; 150, 119–137.&lt;/li&gt;&#xA;&lt;li&gt;Gordon, A., 1999. &lt;em&gt;Classification&lt;/em&gt;, 2nd Edition. Chapman and Hall.&lt;/li&gt;&#xA;&lt;li&gt;Kaufman, L., Rousseuw, P., 1990. &lt;em&gt;Finding Groups in Data: An Introduction to Cluster Analysis&lt;/em&gt;. New York, Wiley.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-09T16:38:14.217" LastActivityDate="2010-11-09T16:38:14.217" />
  <row Id="3717" PostTypeId="2" ParentId="3708" CreationDate="2010-10-18T18:36:53.263" Score="3" ViewCount="" Body="&lt;p&gt;Given the nature of your data I would suggest you investigate the use of &lt;a href=&quot;http://en.wikipedia.org/wiki/Exponential_smoothing&quot; rel=&quot;nofollow&quot;&gt;exponential smoothing&lt;/a&gt; as well as fitting ARIMA type models, especially due to the temporal constraints within your data. Although I wouldn't doubt spatial dependencies exist, I would be abit skeptical about their usefulness in forecasting (in what I would imagine are fairly large areas), especially since any spatial dependency will likely be already captured (at least to a certain extent) in previous observations in the series.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where the spatial dependencies may be helpful is if you have small area estimation problems, and you can use the spatial dependency in your data to help smooth out your estimations in those noisy geographic regions. This may not be a problem though since you have aggregated data for a full year.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You shouldn't take my word for it though, and should investigate economics literature on the subject and assess various forecasting methods yourself. Its quite possible other variables are useful predictors of future unemployment in similar panel settings.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First I'd like to clarify that I did not mean that the OP should simply prefer some type of exponential smoothing over other techniques. I think the OP should assess performance of various forecasting methods using a hold out sample of 1 or 2 time periods. I do not know the literature for forecasting unemployment, but I have not seen any method so obviously superior that others should be dismissed outright in any context. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Kwak mentions a key point I did not consider initially (and Stephan's comment makes the same point very succinctly as well). The panel nature of the data allows one to estimate an auto-regressive compenent in the model much easier than in a single time series. So I would follow his suggestion and consider the A/B estimator a good bet to provide the best forecast accuracy. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm still sticking with my initial suggestion though that I am skeptical of the usefulness of the spatial dependence, and one should assess a models predictive accuracy with and without the spatial component. In terms of prediction it is not simply whether some sort of spatial auto-correlation exists, it is whether that spatial auto-correlation is useful in predicting future values independent of past observations in the series.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For simplifying my reasoning, lets denote&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$R_{t}$ corresponds to a geographic region $R$ at time $t$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$R_{t-1}$ corresponds to a geographic region $R$ at the previous time period&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$W_{t-1}$ corresponds to however one wants to define the spatial relationship for for the neighbors of $R_{t}$ at the previous time period&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this case $R$ is some attribute and $W$ is that same attribute in the neighbors of $R$ (i.e. an endogenous spatial lag.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In pretty much all cases of lattice areal data, we have a relationship between $R$ and $W$. Two general explanations for this relationship are&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) The General Social Process Theory&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is when there are processes that affect $R$ and $W$ simultaneously that result in similar values with some sort of spatial organization. The support of the data does not distinguish between the forces that shape attributes in a broader scope than the areal units encompass. (I imagine there is a better name for this, so if someone could help me out.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) The Spatial Externalities Theory&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is when some attribute of $W$ directly affects an attribute of $R$. Srikant's example of job diffusion is an example of this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the context of forecasting, the general social process model may not be all that helpful in forecasting. In this case, $R_{t-1}$ and $W_{t-1}$ are reflective of the same external shocks, and so $W_{t-1}$ is less likely to have exogenous power to predict $R_{t}$ independent of $R_{t-1}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;IMO the spatial externalities case I would expect $W_{t-1}$ to have a greater potential to forecast $R_{t}$ independent of $R_{t-1}$ in the short run because $R_{t-1}$ and $W_{t-1}$ can be reflective of different external shocks to the system. This is my opinion though and you typically can't distinguish between the general social process model and the spatial externalities model through empirical means in a cross sectional design (they are probably both occurring to a certain extent in many contexts). Hence I would attempt to validate its usefulness before simply incorporating it into the forecast. Better knowledge of the literature and social processes would definately be helpful here to guide your model building. In criminology only in a very limited set of circumstances does the externalities model make sense (but I imagine it is more likely in economics data). Models of spatial hedonic housing prices often show very strong spatial effects, and in that context I would expect the spatial component to have a strong ability to forecast housing prices. (I like Luc Anselin's explanation of these two different processes better than mine in &lt;a href=&quot;http://dx.doi.org/10.1007/978-3-7908-2070-6_2&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; paper, PDF &lt;a href=&quot;https://www.ace.uiuc.edu/reap/SPARC/2007-01_AnselinLozano.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Often how we define $W$ is a further problem in this setting. Most conceptions of $W$ are very simplistic and probably aren't entirely reflective of real geographic processes. Here kwaks suggestion of adding a random component to the $W$ effect for each $R$ makes alot of sense. An example would be we would expect New York City to influence its neighbors, but we wouldn't expect NYC's neighbors to have all that much influence on NYC. This still doesn't solve how to either decide what is a neighbor or how to best represent the effects of neighbors. What kwak suggests is essential a local version of Geary's C (spatial differences), local Moran's I (spatial averages) is a common approach as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm still alittle surprised at the negative responses to my suggestion to use simpler smoothing methods (even if they are meant for univariate time series). Am I naive to think exponential smoothing or some other type of moving window technique won't perform at least comparably well enough to more complicated procedures to assess it? I would be more worried if the series were such that we would expect seasonal components, but that is not the case here.&lt;/p&gt;&#xA;" OwnerUserId="1036" LastEditorUserId="1036" LastEditorDisplayName="" LastEditDate="2010-10-19T15:23:18.643" LastActivityDate="2010-10-19T15:23:18.643" CommentCount="6" />
  <row Id="3718" PostTypeId="2" ParentId="3707" CreationDate="2010-10-18T19:09:55.590" Score="0" ViewCount="" Body="&lt;p&gt;There is no mysterious reason. If you multiply a bunch of numbers between 0 an 1, the result will forcibly be close to 0. The average result for RAND()*RAND()*RAND()*RAND()*RAND()*RAND() should be something close to (0.5^6), that is, 0.015625.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Be careful using Excel's RAND() function, though. It's not the best random number generator in the world.&lt;/p&gt;&#xA;" OwnerUserId="666" LastEditorUserId="666" LastEditorDisplayName="" LastEditDate="2010-10-18T19:18:41.650" LastActivityDate="2010-10-18T19:18:41.650" CommentCount="1" />
  <row Id="3719" PostTypeId="1" AcceptedAnswerId="3721" CreationDate="2010-10-18T19:10:41.067" Score="2" ViewCount="186" Body="&lt;p&gt;How is it possible to calculate Pearson/Spearman correlation of more than two judges in MATLAB? &#xA;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="1564" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-10-18T19:13:44.903" LastActivityDate="2011-05-08T17:46:41.447" Title="Is it possible to calculate Pearson/Spearman correlation of more than two judges in MATLAB?" Tags="&lt;correlation&gt;&lt;matlab&gt;" AnswerCount="2" />
  <row Id="3720" PostTypeId="2" ParentId="3708" CreationDate="2010-10-18T19:19:01.650" Score="5" ViewCount="" Body="&lt;p&gt;The Arellano-Bond estimator has been designed for precisely this type of problems.&#xA;You will find a short non-technical paper with a examples &lt;a href=&quot;http://www.cemmap.ac.uk/wps/cwp0209.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;. In a nutshell, it combines the information embedded in the large number of cross-section to make up for the small number of points in each series. This estimator is widely used and implemented: it is avalaible in the default gretl &lt;a href=&quot;http://gretl.sourceforge.net/&quot; rel=&quot;nofollow&quot;&gt;package&lt;/a&gt;, but also in stata via the &lt;a href=&quot;http://ideas.repec.org/c/boc/bocode/s435901.html&quot; rel=&quot;nofollow&quot;&gt;XTABOND2&lt;/a&gt; package and in R too, via the plm &lt;a href=&quot;http://cran.r-project.org/web/packages/plm/vignettes/plm.pdf&quot; rel=&quot;nofollow&quot;&gt;package&lt;/a&gt; (you should easily find a large number of paper using it).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT:&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given that spatial correlation may indeed be informative (see Andy's post), i would advice to add a variable:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$s_{it} = u_{it} - \bar{u}_{-it}$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $u_{it}$ is (eventually the $\log()$ of) the unemployment rate of region $i$ at time $t$ and $\bar{u}_{-it}$ its average value among $k$ geographical neighbors of region $i$ (excluding region $i$). I would advise trying different values of $k$ until small changes in $k$ do not affect the estimation end-result/conclusions. Then, for efficient and consistent estimation of $\beta_s$ (the coefficient associated with the variable $s$) i would use OLS for the main effect and allow for a random component to the error terms to account for inter-regional heterogenity in $\beta_s$; thereby leveraging the fact that the R package plm allows to combine gmm (i.e. Arellano-Bond) and random effect coefficients. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Concerning Andy W's remark: you could read &lt;a href=&quot;http://www.econbrowser.com/archives/2008/04/regional_propag.html&quot; rel=&quot;nofollow&quot;&gt;these&lt;/a&gt; &lt;a href=&quot;http://www.econbrowser.com/archives/2008/03/recessions_at_t.html&quot; rel=&quot;nofollow&quot;&gt;two&lt;/a&gt; documents for a non technical summary. The full paper version is &lt;a href=&quot;http://ideas.repec.org/p/fip/fedlwp/2009-013.html&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;. Note the reliance on &lt;strong&gt;both&lt;/strong&gt; a large number of cross section and time dimension.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: Thanks @Srikant. I think i get it now :)&lt;/p&gt;&#xA;" OwnerUserId="603" LastEditorUserId="603" LastEditorDisplayName="" LastEditDate="2010-10-19T15:03:26.300" LastActivityDate="2010-10-19T15:03:26.300" CommentCount="26" />
  <row Id="3721" PostTypeId="2" ParentId="3719" CreationDate="2010-10-18T19:37:48.260" Score="3" ViewCount="" Body="&lt;p&gt;Store the evaluation of the judges in a matrix X where each column represents one judge and the rows are the corresponding evaluations. Then &lt;code&gt;corr(x)&lt;/code&gt; gives the required pairwise correlation matrix. From Matlab's help:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;RHO = corr(X) returns a p-by-p matrix containing the pairwise linear correlation coefficient between each pair of columns in the n-by-p matrix X.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Is this what you want?&lt;/p&gt;&#xA;" LastActivityDate="2010-10-18T19:37:48.260" CommentCount="2" />
  <row Id="3722" PostTypeId="2" ParentId="3715" CreationDate="2010-10-18T19:53:58.210" Score="0" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://projecteuclid.org/DPubS?service=UI&amp;amp;version=1.0&amp;amp;verb=Display&amp;amp;handle=euclid.aos/1015362189&quot; rel=&quot;nofollow&quot;&gt;Brown, Cai, and DasGupta, AS, 2002&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://projecteuclid.org/DPubS?service=UI&amp;amp;version=1.0&amp;amp;verb=Display&amp;amp;handle=euclid.ss/1009213286&quot; rel=&quot;nofollow&quot;&gt;Brown, Cai, and DasGupta, Stat Sci, 2001&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't know if I understand you correctly, but in my knowledge the above two papers are the most cited ones recently when it comes to binomial proportions' CI and estimation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sorry if this is not what you wanted.&lt;/p&gt;&#xA;" OwnerUserId="1307" LastActivityDate="2010-10-18T19:53:58.210" />
  <row Id="3723" PostTypeId="2" ParentId="3715" CreationDate="2010-10-18T19:57:08.563" Score="3" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Error_propagation&quot; rel=&quot;nofollow&quot;&gt;Error propagation&lt;/a&gt; won't work in the way handled by the uncertainties package.  As you note, they're dependent, so you have to take the covariances into account.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can obtain the variance of your distribution P(B|A) using the &lt;a href=&quot;http://en.wikipedia.org/wiki/Delta_method&quot; rel=&quot;nofollow&quot;&gt;Delta Method&lt;/a&gt; and use that to obtain a confidence interval.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With Bayesian inference, you might find it simpler to use a &lt;a href=&quot;http://en.wikipedia.org/wiki/Credible_interval&quot; rel=&quot;nofollow&quot;&gt;credible interval&lt;/a&gt;.  The following slides do a good job of explaining how to obtain this:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://home.uchicago.edu/~grynav/bayes/ABSLec5.ppt&quot; rel=&quot;nofollow&quot;&gt;Bayesian analysis of one, two, and n-parameter models&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://bayes.bgsu.edu/nsf_web/tutorial/a_brief_tutorial.htm&quot; rel=&quot;nofollow&quot;&gt;A Brief Tutorial on Bayesian Thinking&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-10-18T19:57:08.563" CommentCount="2" />
  <row Id="3724" PostTypeId="1" AcceptedAnswerId="3725" CreationDate="2010-10-18T20:54:08.583" Score="5" ViewCount="310" Body="&lt;p&gt;According to Microsoft Excel Help:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;VAR uses the following formula:&lt;br&gt;&#xA;  &lt;img src=&quot;http://i.imgur.com/ywIpP.gif&quot; alt=&quot;Formula&quot;&gt;&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;where x is the sample mean&#xA;  AVERAGE(number1,number2,…) and n is&#xA;  the sample size.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Shouldn't it be n, rather than n - 1, in the denominator?&lt;/p&gt;&#xA;" OwnerUserId="1618" LastActivityDate="2010-10-23T06:00:07.847" Title="Microsoft Excel formula for variance" Tags="&lt;variance&gt;&lt;excel&gt;&lt;unbiased-estimator&gt;" AnswerCount="2" CommentCount="5" />
  <row Id="3725" PostTypeId="2" ParentId="3724" CreationDate="2010-10-18T21:01:57.917" Score="6" ViewCount="" Body="&lt;p&gt;Use VARP for the variance you want (&quot;population variance&quot;).  VAR is the unbiased estimator for a normally distributed population.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-10-18T21:01:57.917" CommentCount="6" />
  <row Id="3726" PostTypeId="2" ParentId="1224" CreationDate="2010-10-18T21:44:50.017" Score="3" ViewCount="" Body="&lt;p&gt;Are you calculating your chi-square statistic by squaring the difference between the logHRs and dividing by the variance of this diff? If so, that sounds absolutely fine to me. With only two centres, perhaps I wouldn't usually think of this a test for (between-centre) heterogeneity - i might call it a test for interaction with centre, or a test for between-centre difference in the effect (i.e. in the hazard ratio in this case). That's just terminology though. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One thing that probably &lt;em&gt;wouldn't&lt;/em&gt; be appropriate would be to &lt;em&gt;estimate&lt;/em&gt; the between-centre variance, as with only two centre you really enough information for a meaningful estimate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another possible mistake to avoid: if the result of the test is 'not statistically significant' do &lt;em&gt;not&lt;/em&gt; drop the centre effects, i.e. &lt;em&gt;do&lt;/em&gt; still stratify by centre. Even if the HRs are the same in the two centres there could still be confounding by centre, i.e. the pooled HR could still be different from both. And even if that's not a 'significant' difference, avoiding any possibility of confounding by centre characteristics is &lt;em&gt;always&lt;/em&gt; worth the loss of 1 degree of freedom. (In any case, it's usually a mistake to decide the method of analysis based on the results of statistical tests, an approach sometimes called 'data snooping' - see &lt;a href=&quot;http://stats.stackexchange.com/q/499/449&quot;&gt;this earlier topic&lt;/a&gt;). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Note that I'm not suggesting the questioner was going to do either of those things, i'm just taking the opportunity to be particularly pedagogical.)&lt;/p&gt;&#xA;" OwnerUserId="449" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-10-19T06:44:37.850" LastActivityDate="2010-10-19T06:44:37.850" />
  <row Id="3727" PostTypeId="1" AcceptedAnswerId="3789" CreationDate="2010-10-18T22:19:27.650" Score="9" ViewCount="112" Body="&lt;p&gt;What's the best way to approximate $Pr[n \leq X \leq m]$ for two given integers $m,n$ when you know the mean $\mu$, variance $\sigma^2$, skewness $\gamma_1$ and excess kurtosis $\gamma_2$ of a discrete distribution $X$, and it is clear from the (non-zero) measures of shape $\gamma_1$ and $\gamma_2$ that a normal approximation is not appropriate?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ordinarily, I would use a normal approximation with integer correction...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Pr[(n - \text{½})\leq X \leq (m + \text{½})]&#xA; = Pr[\frac{(n - \text{½})-\mu}{\sigma}\leq Z \leq \frac{(m + \text{½})-\mu}{\sigma}]&#xA; = \Phi(\frac{(m + \text{½})-\mu}{\sigma}) - \Phi(\frac{(n - \text{½})-\mu}{\sigma})$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;...if the skewness and excess kurtosis were (closer to) 0, but that's not the case here. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have to perform multiple approximations for different discrete distributions with different values of $\gamma_1$ and $\gamma_2$. So I'm interested in finding out if there is an established a procedure that uses $\gamma_1$ and $\gamma_2$ to select a better approximation than the normal approximation.&lt;/p&gt;&#xA;" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-10-19T17:31:18.077" LastActivityDate="2010-10-20T08:32:13.313" Title="Approximating $Pr[n \leq X \leq m]$ for a discrete distribution" Tags="&lt;distributions&gt;&lt;probability&gt;&lt;normality&gt;&lt;approximation&gt;&lt;moments&gt;" AnswerCount="3" FavoriteCount="1" />
  <row Id="3728" PostTypeId="2" ParentId="499" CreationDate="2010-10-18T22:26:57.827" Score="3" ViewCount="" Body="&lt;p&gt;Variable selection techniques, in general (whether stepwise, backward, forward, all subsets, AIC, etc.), capitalize on chance or random patterns in the sample data that do not exist in the population.  The technical term for this is over-fitting and it is especially problematic with small datasets, though it is not exclusive to them.  By using a procedure that selects variables based on best fit, all of the random variation that looks like fit &lt;strong&gt;&lt;em&gt;in this particular sample&lt;/em&gt;&lt;/strong&gt; contributes to estimates and standard errors.  This is a problem for &lt;strong&gt;&lt;em&gt;both&lt;/em&gt;&lt;/strong&gt; prediction and interpretation of the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Specifically, r-squared is too high and parameter estimates are biased (they are too far from 0), standard errors for parameters are too small (and thus p-values and intervals around parameters are too small/narrow).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The best line of defense against these problems is to build models thoughtfully and include the predictors that make sense based on theory, logic, and previous knowledge.  If a variable selection procedure is necessary, you should select a method that penalizes the parameter estimates (shrinkage methods) by adjusting the parameters and standard errors to account for over-fitting.  Some common shrinkage methods are Ridge Regression, Least Angle Regression, or the lasso.  In addition, cross-validation using a training dataset and a test dataset or model-averaging can be useful to test or reduce the effects of over-fitting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Harrell is a great source for a detailed discussion of these problems.  &lt;em&gt;Harrell (2001). &quot;Regression Modeling Strategies.&quot;&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="485" LastActivityDate="2010-10-18T22:26:57.827" />
  <row Id="3729" PostTypeId="2" ParentId="3727" CreationDate="2010-10-19T00:21:55.433" Score="1" ViewCount="" Body="&lt;p&gt;You could try to use &lt;a href=&quot;http://en.wikipedia.org/wiki/Skew_normal_distribution&quot; rel=&quot;nofollow&quot;&gt;skew normal distribution&lt;/a&gt; and see if excess kurtosis for your particular data sets is sufficiently close to the excess kurtosis of the distribution for given skewness. If it is, you can use the skew normal distribution cdf to estimate the probability. If not, you would have to come up with a transformation to the normal/skew pdf similar to the one used for the skew normal distribution, which would give you control over both skewness and excess kurtosis.&lt;/p&gt;&#xA;" OwnerUserId="1448" LastActivityDate="2010-10-19T00:21:55.433" />
  <row Id="3730" PostTypeId="1" CreationDate="2010-10-19T01:14:16.967" Score="9" ViewCount="2324" Body="&lt;p&gt;I get this question frequently enough in my statistics consulting work, that I thought I'd post it here. I have an answer, which is posted below, but I was keen to hear what others have to say.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question:&lt;/strong&gt; If you have two variables that are not normally distributed, should you use Spearman's&#xA;rho for the correlation?&lt;/p&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-10-19T08:58:55.663" LastActivityDate="2010-10-21T05:14:09.703" Title="Pearson's or Spearman's correlation with non-normal data" Tags="&lt;correlation&gt;&lt;normality&gt;" AnswerCount="4" FavoriteCount="4" />
  <row Id="3731" PostTypeId="2" ParentId="3730" CreationDate="2010-10-19T01:27:47.607" Score="1" ViewCount="" Body="&lt;p&gt;&lt;em&gt;Updated&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The question asks us to choose between Pearson's and Spearman's method when &lt;em&gt;normality&lt;/em&gt; is questioned.  Restricted to this concern, I think the following paper should inform anyone's decision:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.jstor.org/pss/2346598&quot; rel=&quot;nofollow&quot;&gt;On the Effects of Non-Normality on the Distribution of the Sample Product-Moment&#xA;Correlation Coefficient&lt;/a&gt; (Kowalski, 1975)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;It's quite nice and provides a survey of the considerable literature, spanning decades, on this topic -- starting from Pearson's &quot;mutilated and distorted surfaces&quot; and robustness of distribution of $r$.  At least part of the contradictory nature of the &quot;facts&quot; is that much of this work was done before the advent of computing power -- which complicated things because the type of non-normality had to be considered and was hard to examine without simulations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Kowalski's analysis concludes that the distribution of $r$ is &lt;em&gt;not&lt;/em&gt; robust in the presence of non-normality and recommends alternative procedures.  The entire paper is quite informative and recommended reading, but skip to the very short conclusion at the end of the paper for a summary.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If asked to choose between one of Spearman and Pearson when normality is violated, the distribution free alternative is worth advocating, i.e. Spearman's method.  &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Previously&lt;/em&gt; ..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Spearman's correlation is a rank based correlation measure; it's non-parametric and does not rest upon an assumption of normality.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The sampling distribution for Pearson's correlation does assume normality; in particular this means that although you can compute it, conclusions based on significance testing may not be sound.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As Rob points out in the comments, with large sample this is not an issue.  With small samples though, where normality is violated, Spearman's correlation should be preferred.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Update&lt;/em&gt; Mulling over the comments and the answers, it seems to me that this boils down to the usual non-parametric vs. parametric tests debate.  Much of the literature, e.g. in biostatistics, doesn't deal with large samples.  I'm generally not cavalier with relying on asymptotics.  Perhaps it's justified in this case, but that's not readily apparent to me.&lt;/p&gt;&#xA;" OwnerUserId="251" LastEditorUserId="251" LastEditorDisplayName="" LastEditDate="2010-10-21T02:05:03.130" LastActivityDate="2010-10-21T02:05:03.130" CommentCount="10" />
  <row Id="3732" PostTypeId="2" ParentId="499" CreationDate="2010-10-19T01:50:06.413" Score="4" ViewCount="" Body="&lt;p&gt;Richard Berk has a recent article where he demonstrates through simulation the problems of such data snooping and statistical inference. As Rob &lt;a href=&quot;http://stats.stackexchange.com/questions/499/when-can-you-use-data-based-criteria-to-specify-a-regression-model/531#531&quot;&gt;suggested&lt;/a&gt; it is more problematic than simply correcting for multiple hypothesis tests.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://dx.doi.org/10.1007/s10940-009-9077-7&quot; rel=&quot;nofollow&quot;&gt;Statistical Inference After Model Selection&lt;/a&gt;&#xA;by: Richard Berk, Lawrence Brown, Linda Zhao&#xA;Journal of Quantitative Criminology, Vol. 26, No. 2. (1 June 2010), pp. 217-236. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;PDF version &lt;a href=&quot;http://www-stat.wharton.upenn.edu/~lzhao/papers/MyPublication/StatInfAfterMS_JQC_2010.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1036" LastActivityDate="2010-10-19T01:50:06.413" CommentCount="3" />
  <row Id="3733" PostTypeId="2" ParentId="3730" CreationDate="2010-10-19T01:53:19.697" Score="9" ViewCount="" Body="&lt;p&gt;Pearson's correlation is a measure of the linear relationship between two continuous random variables. It does not assume normality although it does assume finite variances and finite covariance. When the variables are bivariate normal, Pearson's correlation provides a complete description of the association.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Spearman's correlation applies to ranks and so provides a measure of a monotonic relationship between two continuous random variables. It is also useful with ordinal data and is robust to outliers (unlike Pearson's correlation).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The distribution of either correlation coefficient will depend on the underlying distribution, although both are asymptotically normal because of the central limit theorem.&lt;/p&gt;&#xA;" OwnerUserId="159" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-10-19T07:48:02.297" LastActivityDate="2010-10-19T07:48:02.297" CommentCount="2" />
  <row Id="3734" PostTypeId="1" CreationDate="2010-10-19T02:14:32.050" Score="11" ViewCount="498" Body="&lt;p&gt;In several different contexts we invoke the &lt;a href=&quot;http://en.wikipedia.org/wiki/Central_limit_theorem&quot; rel=&quot;nofollow&quot;&gt;central limit theorem&lt;/a&gt; to justify whatever statistical method we want to adopt (e.g., approximate the binomial distribution by a normal distribution). I understand the technical details as to why the theorem is true but it just now occurred to me that I do not really understand the intuition behind the central limit theorem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, what is the intuition behind the central limit theorem? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Layman explanations would be ideal. If some technical detail is needed please assume that I understand the concepts of a pdf, cdf, random variable etc but have no knowledge of convergence concepts, characteristic functions or anything to do with measure theory.&lt;/p&gt;&#xA;" LastActivityDate="2010-10-23T21:20:45.120" Title="What intuitive explanation is there for the central limit theorem?" Tags="&lt;intuition&gt;&lt;central-limit-theorem&gt;" AnswerCount="4" CommentCount="3" FavoriteCount="4" />
  <row Id="3735" PostTypeId="2" ParentId="3194" CreationDate="2010-10-19T02:53:05.170" Score="5" ViewCount="" Body="&lt;p&gt;A chi-squared test tests all possible deviations from strict uniformity. This is reasonable with a d4 or a d6, but with a d20, you're probably more interested in checking that the probability that you roll under (or possibly exceed) each outcome is close to what it should be.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I am getting at is that there are some kinds of deviations from fairness that will heavily impact whatever you're using a d20 for and other kinds of deviations that hardly matter at all, and the chi-squared test will divide power between more interesting and less interesting alternatives. The consequence is that to have enough power to pick up even fairly moderate deviations from fairness, you need a &lt;em&gt;huge&lt;/em&gt; number of rolls - far more than you would ever want to sit and generate. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Hint: come up with a few sets of non-uniform probabilities for your d20 that will most heavily impact the outcome that you're using the d20 for and use simulation and chi-squared tests to find out what power you have against them for various numbers of rolls, so you get some idea of the number of rolls you will need.) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are a variety of ways of checking for &quot;interesting&quot; deviations (ones that will be more likely to substantively affect typical uses of a d20)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My recommendation is to do an ECDF test (Kolmogorov-Smirnov/Anderson-Darling-type test - but you'll probably want to adjust for the conservativeness that results from the distribution being discrete - at least by lifting the nominal alpha level, but even better by just simulating the distribution to see how the distribution of the test statistic goes for a d20).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These can still pick up any kind of deviation, but they put relatively more weight on the more important kinds of deviation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An even more powerful approach is to specifically construct a test statistic that is specifically sensitive to the most important alternatives to you, but it involves a bit more work.&lt;/p&gt;&#xA;" OwnerUserId="805" LastEditorUserId="805" LastEditorDisplayName="" LastEditDate="2010-10-19T12:39:50.413" LastActivityDate="2010-10-19T12:39:50.413" />
  <row Id="3736" PostTypeId="2" ParentId="3734" CreationDate="2010-10-19T03:35:38.610" Score="7" ViewCount="" Body="&lt;p&gt;Intuition is a tricky thing.  It's even trickier with theory in our hands tied behind our back.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The CLT is all about sums of tiny, independent disturbances.  &quot;Sums&quot; in the sense of the sample mean, &quot;tiny&quot; in the sense of finite variance (of the population), and &quot;disturbances&quot; in the sense of plus/minus around a central (population) value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For me, the device that appeals most directly to &lt;em&gt;intuition&lt;/em&gt; is the quincunx, or 'Galton box', see Wikipedia (for 'bean machine'?)  The idea is to roll a tiny little ball down the face of a board adorned by a lattice of equally spaced pins.  On its way down the ball diverts right and left (...randomly, independently) and collects at the bottom.  Over time, we see a nice bell shaped mound form right before our eyes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The CLT says the same thing. It is a mathematical description of this phenomenon (more precisely, the quincunx is physical evidence for the normal approximation to the binomial distribution).  Loosely speaking, the CLT says that as long as our population is not overly misbehaved (that is, if the tails of the PDF are sufficiently thin), then the sample mean (properly scaled) behaves just like that little ball bouncing down the face of the quincunx:  sometimes it falls off to the left, sometimes it falls off to the right, but most of the time it lands right around the middle, in a nice bell shape.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The majesty of the CLT (to me) is that the &lt;em&gt;shape&lt;/em&gt; of the underlying population is irrelevant.  Shape only plays a role insofar as it delegates the length of time we need to wait (in the sense of sample size).&lt;/p&gt;&#xA;" OwnerUserId="1108" LastEditorUserId="1108" LastEditorDisplayName="" LastEditDate="2010-10-19T03:55:41.573" LastActivityDate="2010-10-19T03:55:41.573" />
  <row Id="3737" PostTypeId="2" ParentId="3734" CreationDate="2010-10-19T04:18:09.647" Score="0" ViewCount="" Body="&lt;p&gt;I gave up on trying to come up with an intuitive version and came up with some simulations.  I have one that does a quincunx and some others that do things like show how even a skewed raw reaction time distribution will become normal if you collect enough RT's per subject.  I think they help but they're new in my class this year and I haven't graded the first test yet.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One thing that I thought was good was being able to show the law of large numbers as well.  I could show how variable things are with small sample sizes and then show how they stabilize with large ones.  I do a bunch of other large number demos as well.  I can show the interaction in the quincunx between the numbers of random processes and the numbers of samples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(turns out not being able to use a chalk or white board in my class may have been a blessing)&lt;/p&gt;&#xA;" OwnerUserId="601" LastActivityDate="2010-10-19T04:18:09.647" />
  <row Id="3738" PostTypeId="2" ParentId="3730" CreationDate="2010-10-19T06:08:06.590" Score="1" ViewCount="" Body="&lt;p&gt;From an applied perspective, I am more concerned with choosing an approach that summarises the relationship between two variables in a way that aligns with my research question. I think that determining a method for getting accurate standard errors and p-values is a question that should come second. Even if you chose not to rely on asymptotics, there's always the option to bootstrap or change distributional assumptions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a general rule, I prefer Pearson's correlation because &#xA;(a) it generally aligns more with my theoretical interests; (b) it enables more direct comparability of findings across studies, because most studies in my area report Pearson's correlation; and (c) in many settings there is minimal difference between Pearson and Spearman correlation coefficients. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, there are situations where I think Pearson's correlation on raw variables is misleading. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Outliers:&lt;/strong&gt; Outliers can have great influence on Pearson's correlations. Many outliers in applied settings reflect measurement failures or other factors that the model is not intended to generalise to. One option is to remove such outliers. Univariate outliers do not exist with Spearman's rho because everything is converted to ranks. Thus, Spearman is more robust. &lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Highly skewed variables:&lt;/strong&gt; When correlating skewed variables, particularly highly skewed variables, a log or some other transformation often makes the underlying relationship between the two variables clearer (e.g., brain size by body weight of animals). In such settings it may be that the raw metric is not the most meaningful metric anyway. Spearman's rho has a similar effect to transformation by converting both variables to ranks. From this perspective, Spearman's rho can be seen as a quick-and-dirty approach (or more positively, it is less subjective) whereby you don't have to think about optimal transformations.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In both cases above, I would advise researchers to either consider adjustment strategies (e.g., transformations, outlier removal/adjustment) before applying Pearson's correlation or use Spearman's rho.&lt;/p&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-10-19T07:36:34.143" LastActivityDate="2010-10-19T07:36:34.143" />
  <row Id="3739" PostTypeId="1" CreationDate="2010-10-19T06:14:17.770" Score="2" ViewCount="166" Body="&lt;p&gt;I have sampled a number of $x$ values from a normal distribution with mean 0 and sd 0.2. I then transformed these $x$ values to $y$ values using the formula $y = e^x/(e^x + 1)$. I know that the $y$ values will have a mean of 0.5 and all lie between 0 and 1. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Do these y values fit some common distribution? &lt;/li&gt;&#xA;&lt;li&gt;Is there a way to figure this out in R? &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;What I am looking for is if there is a simple distribution to describe the y values that would allow me to describe it in terms of a few parameters and possibly sample directly from it in R.&lt;/p&gt;&#xA;" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-10-19T07:17:41.907" LastActivityDate="2010-10-19T07:48:17.050" Title="Does data transformed in a certain way from a normal distribution fit some other common distribution?" Tags="&lt;distributions&gt;&lt;data-transformation&gt;" AnswerCount="3" CommentCount="2" />
  <row Id="3740" PostTypeId="2" ParentId="3739" CreationDate="2010-10-19T06:40:30.707" Score="1" ViewCount="" Body="&lt;p&gt;The trivial way is just to realize this transformation and see what happens:&#xA;&lt;img src=&quot;http://i.imgur.com/5vCml.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This looks pretty normal, indeed qqnorm confirms it:&#xA;&lt;img src=&quot;http://i.imgur.com/sEQfX.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-10-19T06:40:30.707" CommentCount="5" />
  <row Id="3742" PostTypeId="1" CreationDate="2010-10-19T06:44:15.630" Score="0" ViewCount="48" Body="&lt;p&gt;I might be posting in the wrong place. Where can I find information on bin packing &quot;magazines in a box&quot;? I assume this would be 3 dimensional bin packing correct?&lt;/p&gt;&#xA;" OwnerUserId="1622" LastActivityDate="2010-10-19T06:44:15.630" ClosedDate="2010-10-19T06:46:38.127" Title="Advice on bin packing" Tags="&lt;algorithms&gt;" CommentCount="1" />
  <row Id="3743" PostTypeId="2" ParentId="3739" CreationDate="2010-10-19T06:50:40.420" Score="3" ViewCount="" Body="&lt;p&gt;You have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$X \sim N(\mu,\sigma^2)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By definition:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Y = \frac{e^X}{e^X+1}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, the cdf of $Y$ is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(Y \le y) = P(\frac{e^X}{e^X+1} \le y)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Simplifying the RHS, we get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(Y \le y) = P(X \le -log(\frac{1-y}{y}) )$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(Y \le y) = \Phi(-log(\frac{1-y}{y}),\mu,\sigma^2)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Differentiating the above wrt $y$, we get the pdf $f(y)$ as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$f(y) = (\frac{1}{y(1-y)}) \phi(-log(\frac{1-y}{y}),\mu,\sigma^2)$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strike&gt;I do not think the above pdf has a standard name.&lt;/strike&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;onestop identifies the correct name for the pdf in his &lt;a href=&quot;http://stats.stackexchange.com/questions/3739/does-data-transformed-in-a-certain-way-from-a-normal-distribution-fit-some-other/3745#3745&quot;&gt;answer&lt;/a&gt;: &lt;a href=&quot;http://en.wikipedia.org/wiki/Logit-normal_distribution&quot; rel=&quot;nofollow&quot;&gt;Logit-normal distribution&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Reg how to sample it in R you can use the &lt;a href=&quot;http://en.wikipedia.org/wiki/Inverse_transform_sampling&quot; rel=&quot;nofollow&quot;&gt;inverse-transform sampling&lt;/a&gt;. The idea is as follows:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Generate a uniform random variate: $U \sim U[0,1]$&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Set $U = \Phi(-log(\frac{1-y}{y}),\mu,\sigma^2)$ and invert for $y$.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;However, the use of the inverse method sampling is not necessary as a you could just sample values of $Y$ by sampling from $X$ first and then doing the logistic transformation (as given in your question).&lt;/p&gt;&#xA;" LastActivityDate="2010-10-19T07:48:17.050" />
  <row Id="3744" PostTypeId="2" ParentId="3730" CreationDate="2010-10-19T07:07:16.707" Score="5" ViewCount="" Body="&lt;p&gt;Don't forget &lt;strong&gt;Kendall's tau&lt;/strong&gt;! Roger Newson has argued for the superiority of Kendall's &lt;em&gt;τ&lt;sub&gt;a&lt;/sub&gt;&lt;/em&gt; over Spearman's correlation &lt;em&gt;r&lt;sub&gt;S&lt;/sub&gt;&lt;/em&gt; as a rank-based measure of correlation in a paper whose full text is now freely available online:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Newson R. &lt;a href=&quot;http://www.stata-journal.com/sjpdf.html?articlenum=st0007&quot; rel=&quot;nofollow&quot;&gt;Parameters behind &quot;nonparametric&quot; statistics: Kendall's tau,Somers' D and median differences&lt;/a&gt;. &lt;em&gt;Stata Journal&lt;/em&gt; 2002; 2(1):45-64.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;He references (on p47) Kendall &amp;amp; Gibbons (1990) as arguing that &quot;...confidence intervals for Spearman’s &lt;em&gt;r&lt;sub&gt;S&lt;/sub&gt;&lt;/em&gt; are less reliable and less interpretable than confidence intervals for Kendall’s &lt;em&gt;τ&lt;/em&gt;-parameters, but the sample Spearman’s &lt;em&gt;r&lt;sub&gt;S&lt;/sub&gt;&lt;/em&gt; is much more easily calculated without a computer&quot; (which is no longer of much importance of course). Unfortunately I don't have easy access to a copy of their book:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Kendall, M. G. and J. D. Gibbons. 1990. &lt;em&gt;Rank Correlation Methods&lt;/em&gt;. 5th ed. London:&#xA;Griffin.&lt;/p&gt;&#xA;" OwnerUserId="449" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-10-21T05:14:09.703" LastActivityDate="2010-10-21T05:14:09.703" CommentCount="1" />
  <row Id="3745" PostTypeId="2" ParentId="3739" CreationDate="2010-10-19T07:14:21.320" Score="2" ViewCount="" Body="&lt;p&gt;This is called a &lt;a href=&quot;http://en.wikipedia.org/wiki/Logit-normal_distribution&quot; rel=&quot;nofollow&quot;&gt;logit-normal distribution&lt;/a&gt; (by analogy with the much more common log-normal distribution). Knowing that doesn't simplify sampling from it, however, or change the parameters used to describe it, which are still the mean and SD (or variance) of the parent normal distribution.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-10-19T07:14:21.320" CommentCount="2" />
  <row Id="3746" PostTypeId="1" CreationDate="2010-10-19T08:10:54.190" Score="8" ViewCount="241" Body="&lt;p&gt;I need to write a program to find the average GPS point from a population of points.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;In practice the following happens:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Each month a person records a GPS point of the same static asset.&lt;/li&gt;&#xA;&lt;li&gt;Because of the nature of GPS, these points differ slightly each month.&lt;/li&gt;&#xA;&lt;li&gt;Sometimes the person makes a mistake a records the wrong assest at a completely different location.&lt;/li&gt;&#xA;&lt;li&gt;Each GPS point has a certainty weight (&lt;a href=&quot;http://en.wikipedia.org/wiki/Dilution_of_precision_%28GPS%29&quot; rel=&quot;nofollow&quot;&gt;HDOP&lt;/a&gt;) that indicates how accurate the current GPS data is.  GPS points with better HDOP values are preferred over lower ones..&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How do I determine the following:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Deal with data with 2 values vs. a single value like age. (Find the average age in a population of people)&lt;/li&gt;&#xA;&lt;li&gt;Determine the outliers. In the example below these would be [-28.252, 25.018] and [-28.632, 25.219]&lt;/li&gt;&#xA;&lt;li&gt;After excluding the outliers, find the average GPS point in this it might be [-28.389, 25.245].&lt;/li&gt;&#xA;&lt;li&gt;It would be a bonus if can work the &quot;weight&quot; provided by HDOP value for each point.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/ecYHJ.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="1623" LastEditorUserId="1352" LastEditorDisplayName="" LastEditDate="2010-10-19T09:28:06.077" LastActivityDate="2010-10-19T20:11:33.707" Title="Finding the average GPS point" Tags="&lt;outliers&gt;&lt;spatial&gt;" AnswerCount="3" CommentCount="1" />
  <row Id="3747" PostTypeId="2" ParentId="3746" CreationDate="2010-10-19T08:30:31.317" Score="3" ViewCount="" Body="&lt;p&gt;Rob Hyndman recently posed a &lt;a href=&quot;http://stats.stackexchange.com/questions/213/what-is-the-best-way-to-identify-outliers-in-multivariate-data&quot;&gt;question about detecting outliers in multivariate data&lt;/a&gt;. The answers may provide a couple of possible approaches (and otherwise, you may want to put the question of finding 2-d outliers in a separate question).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And you can average your remaining GPS data component by component - add all the first components up and divide by the number of points, that will give you the first component of the average. Same with the second components.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This averaging can be weighted by HDOP. Sum up the products of the first component, multiplied with the corresponding HDOP score, and divide the sum by the sum of the HDOP scores. Same with the second components.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'll take the liberty of removing the &quot;normal-distribution&quot; tag...&lt;/p&gt;&#xA;" OwnerUserId="1352" LastActivityDate="2010-10-19T08:30:31.317" CommentCount="1" />
  <row Id="3748" PostTypeId="2" ParentId="2214" CreationDate="2010-10-19T09:24:21.443" Score="2" ViewCount="" Body="&lt;p&gt;The Levy distribution has 4 parameter. Each of them has a quantile-based sample equivalent:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;$\mu$, the location parameter, can be estimated by the median. This is a high efficiency alternative (ARE$\approx 0.85$).&lt;/li&gt;&#xA;&lt;li&gt;$\gamma$, the scale parameter, can be estimated by the median absolute deviation (or more efficiently yet by the Qn estimator (1) with ARE similar to that of the median)&lt;/li&gt;&#xA;&lt;li&gt;$\beta$, the skew parameter, can be estimated by the $S_k$ estimator, with &#xA;$S_k=(Q_x(\frac{3}{4})-2Q_x(\frac{1}{2})+Q_x(\frac{1}{4}))(Q_x(\frac{3}{4})-Q_x(\frac{1}{4}))^{-1}$ where $Q_x(\tau)$ is the $\tau$^th quantile of $x$.&lt;/li&gt;&#xA;&lt;li&gt;$\alpha$, the tail parameter, can be estimated by Moors's quantile based kurtosis estimator (2).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;List of references:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;P.J. Rousseeuw, C. Croux (1993) Alternatives to the Median&#xA; Absolute Deviation, JASA, &lt;em&gt;88&lt;/em&gt;, 1273-1283.&lt;/li&gt;&#xA;&lt;li&gt;J. J. A. Moors, (1988) A Quantile Alternative for Kurtosis &#xA;Journal of the Royal Statistical Society. Series D (The Statistician)&#xA;Vol. 37, No. 1, pp. 25-32&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="603" LastActivityDate="2010-10-19T09:24:21.443" />
  <row Id="3749" PostTypeId="1" AcceptedAnswerId="3981" CreationDate="2010-10-19T10:04:10.973" Score="10" ViewCount="462" Body="&lt;p&gt;Hello&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am discovering the marvellous world of such called &quot;Hidden Markov Models&quot;, also called &quot;regime switching models&quot;. &#xA;I would like to adapt a HMM in R to detect trends and turning points. I would like to build the model as generic as possible so that I can test it on many prices.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Can anyone recommend a paper?&lt;/strong&gt; I have seen (and read) (more than) a few but I am looking for a simple model that is easy to implement.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, &lt;strong&gt;what R packages are recommended?&lt;/strong&gt; I can see there is a lot of them doing HMM.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have bought  the book  &quot;Hidden Markov models for time series: an introduction using R&quot;, let see what's in it ;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Fred&lt;/p&gt;&#xA;" OwnerUserId="1709" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-10-26T02:19:11.633" LastActivityDate="2010-10-26T06:19:55.590" Title="Usage of HMM in quantitative finance. Examples of HMM that works to detect trend / turning points?" Tags="&lt;r&gt;&lt;time-series&gt;&lt;finance&gt;&lt;hmm&gt;" AnswerCount="1" FavoriteCount="6" />
  <row Id="3750" PostTypeId="2" ParentId="3707" CreationDate="2010-10-19T10:10:46.127" Score="1" ViewCount="" Body="&lt;p&gt;Still using excel??? Try R...&#xA;;)&lt;/p&gt;&#xA;" OwnerUserId="1709" LastActivityDate="2010-10-19T10:10:46.127" CommentCount="1" />
  <row Id="3751" PostTypeId="2" ParentId="3734" CreationDate="2010-10-19T10:27:08.170" Score="1" ViewCount="" Body="&lt;p&gt;The nicest animation I know:&#xA;&lt;a href=&quot;http://www.ms.uky.edu/~mai/java/stat/GaltonMachine.html&quot; rel=&quot;nofollow&quot;&gt;http://www.ms.uky.edu/~mai/java/stat/GaltonMachine.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The simplest words I have read: &lt;a href=&quot;http://elonen.iki.fi/articles/centrallimit/index.en.html&quot; rel=&quot;nofollow&quot;&gt;http://elonen.iki.fi/articles/centrallimit/index.en.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;If you sum the results of these ten&#xA;  throws, what you get is likely to be&#xA;  closer to 30-40 than the maximum, 60&#xA;  (all sixes) or on the other hand, the&#xA;  minumum, 10 (all ones).&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;The reason for this is that you can&#xA;  get the middle values in many more&#xA;  different ways than the extremes.&#xA;  Example: when throwing two dice: 1+6 =&#xA;  2+5 = 3+4 = 7, but only 1+1 = 2 and&#xA;  only 6+6 = 12.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;That is: even though you get any of&#xA;  the six numbers equally likely when&#xA;  throwing one die, the extremes are&#xA;  less probable than middle values in&#xA;  sums of several dice.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="1219" LastActivityDate="2010-10-19T10:27:08.170" />
  <row Id="3752" PostTypeId="1" CreationDate="2010-10-19T11:35:15.373" Score="10" ViewCount="153" Body="&lt;p&gt;This question is prompted by &lt;a href=&quot;http://stats.stackexchange.com/q/3556/159&quot;&gt;discussion elsewhere&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Variable kernels are often used in local regression. For example, loess is widely used and works well as a regression smoother, and is based on a kernel of variable width that adapts to data sparsity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, variable kernels are usually thought to lead to poor estimators in kernel density estimation (see &lt;a href=&quot;http://www.jstor.org/pss/2242011&quot; rel=&quot;nofollow&quot;&gt;Terrell and Scott, 1992&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there an intuitive reason why they would work well for regression but not for density estimation?&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-10-20T15:51:18.260" Title="If variable kernel widths are often good for kernel regression, why are they generally not good for kernel density estimation?" Tags="&lt;nonparametric&gt;&lt;smoothing&gt;&lt;kde&gt;&lt;kernel&gt;&lt;loess&gt;" AnswerCount="1" CommentCount="6" FavoriteCount="1" />
  <row Id="3753" PostTypeId="2" ParentId="3746" CreationDate="2010-10-19T11:54:28.387" Score="5" ViewCount="" Body="&lt;p&gt;One of the problems with multivariate data is deciding on, and then interpreting, a suitable metric for calculating distances, hence clever but somewhat hard-to-explain concepts such as Mahalanobis distance. But in this case surely the choice is obvious - &lt;a href=&quot;http://en.wikipedia.org/wiki/Euclidean_distance&quot; rel=&quot;nofollow&quot;&gt;Euclidean distance&lt;/a&gt;. I'd suggest a simple heuristic algorithm something like:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Calculate the (unweighted) centroid of the data points, i.e. the (unweighted) means of the 2 coordinates&lt;/li&gt;&#xA;&lt;li&gt;Calculate the Euclidean distance of all the readings from the centroid&lt;/li&gt;&#xA;&lt;li&gt;Exclude any readings that are further than a certain distance (to be determined based on your experience and knowledge of the technology, or failing that a bit of &lt;s&gt;trial and error&lt;/s&gt; cross-validation - 100m, 1km, 10km??)&lt;/li&gt;&#xA;&lt;li&gt;Calculate the weighted average of both coords of the remaining points, weighting by the inverse of the HDOP score (or some monotonic function of it - i had a quick look at the wikipedia page linked in the question and think maybe you don't need such a function but i'd need to study it further to be sure)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;There are clearly several ways to make this more sophisticated, such as down-weighting outliers or using &lt;a href=&quot;http://en.wikipedia.org/wiki/M-estimator&quot; rel=&quot;nofollow&quot;&gt;M-estimators&lt;/a&gt; rather than simply excluding them, but I'm not sure whether such sophistication is really necessary here.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-10-19T11:54:28.387" />
  <row Id="3754" PostTypeId="1" CreationDate="2010-10-19T12:00:38.310" Score="5" ViewCount="436" Body="&lt;p&gt;Hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;trying to read a large dataset in R the console displayed the follwing errors:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data&amp;lt;-read.csv(&quot;UserDailyStats.csv&quot;, sep=&quot;,&quot;, header=T, na.strings=&quot;-&quot;, stringsAsFactors=FALSE)&#xA;&amp;gt; data = data[complete.cases(data),]&#xA;&amp;gt; dataset&amp;lt;-data.frame(user_id=as.character(data[,1]),event_date= as.character(data[,2]),day_of_week=as.factor(data[,3]),distinct_events_a_count=as.numeric(as.character(data[,4])),total_events_a_count=as.numeric(as.character(data[,5])),events_a_duration=as.numeric(as.character(data[,6])),distinct_events_b_count=as.numeric(as.character(data[,7])),total_events_b=as.numeric(as.character(data[,8])),events_b_duration= as.numeric(as.character(data[,9])))&#xA;Error: cannot allocate vector of size 94.3 Mb&#xA;In addition: Warning messages:&#xA;1: In data.frame(user_msisdn = as.character(data[, 1]), calls_date = as.character(data[,  :&#xA;  NAs introduced by coercion&#xA;2: In data.frame(user_msisdn = as.character(data[, 1]), calls_date = as.character(data[,  :&#xA;  NAs introduced by coercion&#xA;3: In class(value) &amp;lt;- &quot;data.frame&quot; :&#xA;  Reached total allocation of 3583Mb: see help(memory.size)&#xA;4: In class(value) &amp;lt;- &quot;data.frame&quot; :&#xA;  Reached total allocation of 3583Mb: see help(memory.size)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Does anyone know how to read large datasets? The size of UserDailyStats.csv is approximately 2GB.&lt;/p&gt;&#xA;" OwnerUserId="315" LastActivityDate="2010-10-19T16:35:03.540" Title="How to read large dataset in R" Tags="&lt;r&gt;&lt;large-data&gt;" AnswerCount="5" CommentCount="1" FavoriteCount="1" />
  <row Id="3755" PostTypeId="2" ParentId="3754" CreationDate="2010-10-19T13:02:15.267" Score="1" ViewCount="" Body="&lt;p&gt;What platform are you running R on? How much physical and virtual memory does the machine have?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, you might find the following relevant: &lt;a href=&quot;http://stat.ethz.ch/R-manual/R-devel/library/base/html/Memory-limits.html&quot; rel=&quot;nofollow&quot;&gt;http://stat.ethz.ch/R-manual/R-devel/library/base/html/Memory-limits.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="439" LastActivityDate="2010-10-19T13:02:15.267" CommentCount="1" />
  <row Id="3756" PostTypeId="2" ParentId="3754" CreationDate="2010-10-19T13:09:51.147" Score="5" ViewCount="" Body="&lt;p&gt;Two basic things:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;That complaint addresses &lt;em&gt;all of the memory&lt;/em&gt; in the R session, not just the one object that you're loading.  And unless you're using something like &lt;code&gt;ff&lt;/code&gt;, everything in your session is in memory.&lt;/li&gt;&#xA;&lt;li&gt;One Windows, you need to specify how much memory can be used by R.  Have a look at &lt;code&gt;help(memory.limit)&lt;/code&gt;.  Even though you're using 64-bit, it won't default to use all your available memory.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Otherwise, you can consider using &lt;code&gt;bigmemory&lt;/code&gt; to handle larger datasets if it's still a problem.  Some relevant sources:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The &lt;a href=&quot;http://cran.r-project.org/web/views/HighPerformanceComputing.html&quot; rel=&quot;nofollow&quot;&gt;High Performance Computing view&lt;/a&gt; on CRAN.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://dirk.eddelbuettel.com/papers/useR2010hpcTutorialHandout.pdf&quot; rel=&quot;nofollow&quot;&gt;Dirk Eddelbuettel's tutorial&lt;/a&gt; on the subject.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.bytemining.com/2010/08/taking-r-to-the-limit-part-ii-large-datasets-in-r/&quot; rel=&quot;nofollow&quot;&gt;Ryan Rosario's presentation&lt;/a&gt; on the subject.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;As a final suggestion, you can try calling &lt;code&gt;gc()&lt;/code&gt; to free up memory before running your command, although in principle R will do this automatically as it needs to.&lt;/p&gt;&#xA;" OwnerUserId="5" LastEditorUserId="5" LastEditorDisplayName="" LastEditDate="2010-10-19T13:16:54.847" LastActivityDate="2010-10-19T13:16:54.847" />
  <row Id="3757" PostTypeId="1" CreationDate="2010-10-19T14:00:19.707" Score="3" ViewCount="473" Body="&lt;p&gt;In my data, the RT (gaze) of individuals (ID) is examined as a function of a visual conditions, the factor size (small, medium, large). &#xA;Base model:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;print(Base &amp;lt;- lmer(RT ~ Size + (1|ID), data=rt), cor=F)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Random effect:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;print(NoCor &amp;lt;- lmer(RT ~ Size + (0+Size|ID) , data=rt))&#xA;print(WithCor &amp;lt;- lmer(RT ~ Size + (1+Size|ID), data=rt))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Addition of ID slopes improves the Base model. My question is, how can a significant random effect (Size/ID) be interpreted when there is no relationship between the random and fixed effect, i.e., when the correlation between the random factor and the fixed facor does not improve the model [the anova(NoCr, WithCor) does not show a significant improvement]?&lt;/p&gt;&#xA;" OwnerUserId="1626" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-10-20T21:08:01.323" LastActivityDate="2010-10-20T21:08:01.323" Title="Random effect slopes in linear mixed models" Tags="&lt;r&gt;&lt;mixed-model&gt;&lt;random-effects-model&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="2" />
  <row Id="3758" PostTypeId="1" CreationDate="2010-10-19T14:27:51.110" Score="5" ViewCount="312" Body="&lt;p&gt;I'm a new R user and had just tried running friedman on non-normal and heteroscedastic data on seagrass.  I am testing whether biomass is significantly different between sites across years.  R (&lt;code&gt;friedman&lt;/code&gt; function from &lt;code&gt;agricolae&lt;/code&gt; package) returned result like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Friedman's Test&#xA;===============&#xA;Adjusted for ties&#xA;Value: 0.01333333&#xA;Pvalue chisq : 0.9080726&#xA;F value : 0.01316678&#xA;Pvalue F: 0.9086942 0.9087002&#xA;&#xA;Alpha     : 0.05&#xA;t-Student : 1.990847&#xA;LSD       : 17.34995&#xA;&#xA;Means with the same letter are not significantly different.&#xA;GroupTreatment and Sum of the ranks&#xA;a   1   119 &#xA;a   2   118 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I know this means no significant difference given the p-value chisq.  But what does p-value F mean?  &lt;/p&gt;&#xA;" OwnerUserId="1627" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-10-20T09:04:39.833" LastActivityDate="2010-10-20T09:04:39.833" Title="What is a meaning of &quot;p-value F&quot; from Friedman test?" Tags="&lt;r&gt;&lt;nonparametric&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="3759" PostTypeId="1" AcceptedAnswerId="3761" CreationDate="2010-10-19T14:29:50.193" Score="3" ViewCount="144" Body="&lt;p&gt;I'm looking for an implementation of FNN (or better yet, a SOFNN  as described by &lt;a href=&quot;http://www.neural-forecasting-competition.com/.../43-NN3_kuremoto-resubmission%20only%20documentation.pdf&quot; rel=&quot;nofollow&quot;&gt;Forecasting Time Series by SOFNN with Reinforcement Learning&lt;/a&gt;).  Any language, though preference is Java, C#, C++ in that order.  &lt;/p&gt;&#xA;" OwnerUserId="1127" LastEditorUserId="5" LastEditorDisplayName="" LastEditDate="2010-10-19T14:47:40.057" LastActivityDate="2011-05-27T05:38:19.287" Title="Looking for impl of a Fuzzy Neural Network (FNN or SOFNN)" Tags="&lt;machine-learning&gt;&lt;neural-networks&gt;" AnswerCount="1" CommentCount="3" FavoriteCount="1" />
  <row Id="3760" PostTypeId="2" ParentId="3101" CreationDate="2010-10-19T14:34:44.807" Score="3" ViewCount="" Body="&lt;p&gt;Way back in 1965, Sir Austin Bradford Hill wrote &lt;a href=&quot;http://www.edwardtufte.com/tufte/hill&quot; rel=&quot;nofollow&quot;&gt;a great essay&lt;/a&gt; about something very akin to the Pyramid of Evidence, where he discussed how the piling up of evidence can increase our confidence in hypotheses of causality in Medicine.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most of the factors he discusses can be applied to Economics and political sciences.&lt;/p&gt;&#xA;" OwnerUserId="666" LastActivityDate="2010-10-19T14:34:44.807" CommunityOwnedDate="2010-10-19T14:34:44.807" CommentCount="1" />
  <row Id="3761" PostTypeId="2" ParentId="3759" CreationDate="2010-10-19T14:58:49.700" Score="2" ViewCount="" Body="&lt;p&gt;I'm not an expert on this subject, but I believe that FNN are sometimes referred to as a &lt;a href=&quot;http://en.wikipedia.org/wiki/Neuro-fuzzy&quot; rel=&quot;nofollow&quot;&gt;neuro-fuzzy system&lt;/a&gt; (also &lt;a href=&quot;http://www.scholarpedia.org/article/Fuzzy_neural_network&quot; rel=&quot;nofollow&quot;&gt;referenced here&lt;/a&gt;).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are several implementations that I can find for that subject, including in &lt;a href=&quot;http://fuzzy.cs.uni-magdeburg.de/nefcon/&quot; rel=&quot;nofollow&quot;&gt;C++&lt;/a&gt; and &lt;a href=&quot;http://fuzzy.cs.uni-magdeburg.de/nefclass/nefclass-j/&quot; rel=&quot;nofollow&quot;&gt;Java&lt;/a&gt;, but I can't confirm whether they would be relevant to your problem.&lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-10-19T14:58:49.700" CommentCount="1" />
  <row Id="3762" PostTypeId="1" CreationDate="2010-10-19T15:39:13.843" Score="3" ViewCount="193" Body="&lt;p&gt;I have 6 sets of interval data each of which between 0 and 1. Each set, calculated by a computer program, is related to the degree of similarity between some sounds (pairwise).&#xA;What do you think in the best inter-rater reliability measure I can use to see how close the 6 judges are?&#xA;If I want to explain the data in each set, it can be: 0.98, 0.01, 0.5, ... which shows 'sound1' and 'sound2' are very similar (0.98), 'sound1' and 'sound3' are much different (0.01) and so on. &#xA;Thank you so much.&lt;/p&gt;&#xA;" OwnerUserId="1564" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-19T17:11:20.393" LastActivityDate="2010-10-20T09:27:39.170" ClosedDate="2010-10-20T17:16:08.053" Title="The best measure of reliability for interval data between 0 and 1" Tags="&lt;psychometrics&gt;&lt;reliability&gt;&lt;inter-rater&gt;" AnswerCount="2" CommentCount="9" />
  <row Id="3763" PostTypeId="2" ParentId="3754" CreationDate="2010-10-19T15:48:37.687" Score="1" ViewCount="" Body="&lt;p&gt;Given that you shamelessly cross-post, allow me to equally shamelessly refer you to &lt;a href=&quot;http://stackoverflow.com/questions/3968073/how-to-read-large-dataset-in-r&quot;&gt;my answer to your identical question on StackOverlow&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="334" LastActivityDate="2010-10-19T15:48:37.687" CommentCount="1" />
  <row Id="3764" PostTypeId="2" ParentId="3758" CreationDate="2010-10-19T16:00:57.583" Score="3" ViewCount="" Body="&lt;p&gt;I generally used &lt;code&gt;friedman.test()&lt;/code&gt; which doesn't return any F statistic. If you consider that you have $b$ blocks, for which you assigned ranks to observations belonging to each of them, and that you sum these ranks for each of your $a$ groups (let denote them sum $R_i$), then the Friedman statistic is defined as &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;F_r=\frac{12}{ba(a+1)}\sum_{i=1}^aR_i^2-3b(a+1)&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and follows a $\chi^2(a-1)$, for $a$ and $b$ sufficiently large. Quoting Zar (&lt;em&gt;Biostatistical Analysis&lt;/em&gt;, 4th ed., pp. 263-264), this approximation is conservative (hence, test has low power) and we can use an F-test, with&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;F_{\text{obs}}=\frac{(b-1)F_r}{b(a-1)-F_r}&#xA;$$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;which is to be compared to an F distribution with $a-1$ and $(a-1)(b-1$) degrees of freedom.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-19T16:07:40.880" LastActivityDate="2010-10-19T16:07:40.880" />
  <row Id="3765" PostTypeId="2" ParentId="3754" CreationDate="2010-10-19T16:00:58.520" Score="0" ViewCount="" Body="&lt;p&gt;I,m totally agree with Dirk answer. One suggestion. I have found very useful the use of programming languages such as AWK or others when assessing large databases. So, I was able to filter the data I wanted to include in my analysis, reducing the final size of dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Moreover, in your code you are duplicating the same data set twice (data and dataset). If you want to define your variables as factor, numeric, etc, you could use of colClasses option in the read.table function.&lt;/p&gt;&#xA;" OwnerUserId="221" LastActivityDate="2010-10-19T16:00:58.520" />
  <row Id="3766" PostTypeId="2" ParentId="3762" CreationDate="2010-10-19T16:14:39.170" Score="4" ViewCount="" Body="&lt;p&gt;Referring to your comments to @Henrik, I'm inclined to think that you rather have continuous measurements on a set of objects (here, your similarity measure) for 6 raters. You can compute an &lt;a href=&quot;http://en.wikipedia.org/wiki/Intraclass_correlation&quot; rel=&quot;nofollow&quot;&gt;intraclass correlation&lt;/a&gt; coefficient, as described here &lt;a href=&quot;http://stats.stackexchange.com/questions/1015/reliability-in-elicitation-exercise/3667#3667&quot;&gt;Reliability in Elicitation Exercise&lt;/a&gt;. It will provide you with a measure of agreement (or concordance) between all 6 judges wrt. assessments they made, or more precisely the part of variance that is explained by between-rater variance. There's a working R script in appendix.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that this assumes that your measures are considered as real valued measurement (I refer to @onestop's comment), not really proportions of similarity or whatever between your paired sounds. I don't know of a specific version of the ICC for % or values bounded on an interval, only for binary or ranked data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Following your comments about parameters of interest and language issue:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;There are many other online ressources on the ICC; I think &lt;a href=&quot;http://www.uvm.edu/~dhowell/StatPages/More_Stuff/icc/icc.html&quot; rel=&quot;nofollow&quot;&gt;David Howell&lt;/a&gt; provides a gentle and well illustrated introduction to it. Its discussion generalize to k-sample (judges/raters) without any difficulty I think, or see this chapter from Sea and Fortna on &lt;a href=&quot;http://www.mebsbooks.com/cover_note/books/7/6/6/9781402004667/9781402004667-c6.pdf&quot; rel=&quot;nofollow&quot;&gt;Psychometric Methods&lt;/a&gt;. What you have to think to is mainly whether you want to consider your raters as an unique set of observers, not necessarily representative of all the raters that would have assess your object of measurement (this is called a fixed effect), or as a random sample of raters sampled from a larger (hypothetical) population of potential raters: in the former case, this corresponds to a one-way anova or a consistency ICC, in the latter case we talk about an agreement ICC.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;A colleague of mine successfully used &lt;a href=&quot;http://www.mathworks.co.uk/matlabcentral/fileexchange/21501-intraclass-correlation-coefficients&quot; rel=&quot;nofollow&quot;&gt;Kevin Brownhill's script&lt;/a&gt; (from Matlab Central file exchange). The ICC you are interested in is then &lt;code&gt;cse=3&lt;/code&gt; (if you consider that your raters are not representative of a more general population of raters). &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-20T09:27:39.170" LastActivityDate="2010-10-20T09:27:39.170" CommentCount="9" />
  <row Id="3767" PostTypeId="2" ParentId="3762" CreationDate="2010-10-19T16:33:04.320" Score="0" ViewCount="" Body="&lt;p&gt;If you want to compare just two measures, simply take the &lt;a href=&quot;http://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient&quot; rel=&quot;nofollow&quot;&gt;correlation coefficient (Pearson's &lt;em&gt;r&lt;/em&gt;)&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="442" LastActivityDate="2010-10-19T16:33:04.320" CommentCount="4" />
  <row Id="3768" PostTypeId="2" ParentId="3754" CreationDate="2010-10-19T16:35:03.540" Score="1" ViewCount="" Body="&lt;p&gt;Since you're on 64-bit Windows, make sure that you have installed and are running the 64-bit version of R for Windows.  Then, follow the instructions on Gary King's page:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://gking.harvard.edu/zelig/docs/How_do_I2.html&quot; rel=&quot;nofollow&quot;&gt;How do I increase the memory for R?&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-10-19T16:35:03.540" />
  <row Id="3769" PostTypeId="2" ParentId="3758" CreationDate="2010-10-19T16:43:16.423" Score="3" ViewCount="" Body="&lt;p&gt;It seems the output is from the &lt;code&gt;agricolae&lt;/code&gt; package using the method &lt;code&gt;friedman&lt;/code&gt;.  The relevant lines for computing the two statistics in that function are:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;T1.aj &amp;lt;- (m[2] - 1) * (t(s) %*% s - m[1] * C1)/(A1 - C1)&#xA;T2.aj &amp;lt;- (m[1] - 1) * T1.aj/(m[1] * (m[2] - 1) - T1.aj)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Comparing this with the formula in chl's answer, you'll notice that &lt;code&gt;T2.adj&lt;/code&gt; (&quot;F value&quot;) corresponds to $F_{obs}$ and &lt;code&gt;T1.adj&lt;/code&gt; (&quot;Value&quot;) to $F_r$.&lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-10-19T16:43:16.423" CommentCount="7" />
  <row Id="3770" PostTypeId="2" ParentId="3698" CreationDate="2010-10-19T17:10:57.197" Score="2" ViewCount="" Body="&lt;p&gt;The author is using a single proportion test. Try the following in R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(p-0.5)/sqrt(0.5*0.5/33)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where p = 17/33.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See the wiki for the test statistic where the above test is called the &lt;a href=&quot;http://en.wikipedia.org/wiki/Statistical_hypothesis_testing#Common_test_statistics&quot; rel=&quot;nofollow&quot;&gt;One-proportion z-test&lt;/a&gt;. &lt;/p&gt;&#xA;" LastActivityDate="2010-10-19T17:10:57.197" CommentCount="1" />
  <row Id="3771" PostTypeId="2" ParentId="3727" CreationDate="2010-10-19T17:29:33.383" Score="2" ViewCount="" Body="&lt;p&gt;Fitting a distribution to data using the first four moments is exactly what Karl Pearson devised the &lt;a href=&quot;http://en.wikipedia.org/wiki/Pearson_distribution&quot; rel=&quot;nofollow&quot;&gt;Pearson family of continuous probability distributions&lt;/a&gt; for (maximum likelihood is much more popular these days of course). Should be straightforward to fit the relevant member of that family then use the same type of continuity correction as you give above for the normal distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I assume you must have a truly enormous sample size? Otherwise sample estimates of skewness and especially kurtosis are often hopelessly imprecise, as well as being highly sensitive to outliers.In any case, I highly recommend you have a look at &lt;a href=&quot;http://www.research.ibm.com/people/h/hosking/lmoments.html&quot; rel=&quot;nofollow&quot;&gt;L-moments&lt;/a&gt; as an alternative that have several advantages over ordinary moments that can be advantageous for fitting distributions to data.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-10-19T17:29:33.383" />
  <row Id="3772" PostTypeId="1" AcceptedAnswerId="3775" CreationDate="2010-10-19T17:43:46.987" Score="6" ViewCount="369" Body="&lt;p&gt;I am evaluating the effectiveness of 5 different methods to predict a particular binary outcome (call them 'Success' and 'Failure'). The data look like so:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Method    Sample_Size    Success    Percent_Success&#xA;1         28             4          0.14  &#xA;2         19             4          0.21  &#xA;3         24             7          0.29  &#xA;4         21             13         0.61  &#xA;5         22             9          0.40 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I would like to perform a test among these 5 methods to assess the relative superiority of the methods. In other words, I want to order the methods in order of performance as method 1 &gt; method 2 &gt; ... method 5. To avoid the issue of multiple comparisons, I plan to do a permutation test along the following lines:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Step 1: Pool all the data so that the overall sample size is 114 with overall 37 successes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Step 2: Randomly split the data into 5 groups with the corresponding sample sizes of 28, 19, 24, 21 and 22.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Step 3: Increment a counter if the observed order of Percent_Success from step 2 is consistent with the ordering of my data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Step 4: Repeat steps 2 and 3 many times (say 10000).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Desired p-value = Final Counter Value / 10000.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Is the above procedure ok?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Is there anything in R that would enable me to perform the above test?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Any suggestions for improvement or alternative methods would be helpful.  &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="1558" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-10-19T18:58:16.153" LastActivityDate="2010-11-17T23:55:00.300" Title="Can I use permutation tests for to avoid the multiple comparison problem in the context of proportions?" Tags="&lt;r&gt;&lt;multiple-comparisons&gt;&lt;permutation&gt;" AnswerCount="3" />
  <row Id="3773" PostTypeId="2" ParentId="3772" CreationDate="2010-10-19T18:17:52.227" Score="1" ViewCount="" Body="&lt;p&gt;Your suggested Monte-Carlo permutation test procedure will produce a p-value for a test of the null hypothesis that the probability of success is the same for all methods. But there's little reason for doing a &lt;em&gt;Monte Carlo&lt;/em&gt; permutation test here when the corresponding &lt;em&gt;exact&lt;/em&gt; permutation test is perfectly feasible. That's Fisher's exact test (well, some people reserve that name for 2x2 tables, in which case it's a conditional exact test). I've just typed your data into Stata and -tabi ..., exact- gave p=.0067 (for comparison, Pearson's chi-squared test gives p=.0059). I'm sure there's an equivalent function in R which the R gurus will soon add.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you really want to look at ranking you may be best using a Bayesian approach, as it can give a simple interpretation as the probability that each method is truly the best, second best, third best, ... . That comes at the price of requiring you to put priors on your probabilities, of course. The maximum likelihood estimate of the ranks is simply the observed ordering, but it's difficult to quantify the uncertainty in the ranking in a frequentist framework in a way that can be easily interpreted, as far as i'm aware.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I realise I haven't mentioned multiple comparisons, but I just don't see how that comes into this.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-10-19T18:17:52.227" CommentCount="1" />
  <row Id="3774" PostTypeId="2" ParentId="7" CreationDate="2010-10-19T18:41:58.457" Score="2" ViewCount="" Body="&lt;p&gt;Adding a couple to the list:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Lots of in-depth financial data on publicly-traded companies, going back many decades: &lt;a href=&quot;http://www.mergent.com/servius&quot; rel=&quot;nofollow&quot;&gt;http://www.mergent.com/servius&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Rich information on 16+ million businesses in the US: &lt;a href=&quot;http://compass.webservius.com&quot; rel=&quot;nofollow&quot;&gt;http://compass.webservius.com&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Both available via a REST API and have free trial plans.&lt;/p&gt;&#xA;" OwnerUserId="1629" LastActivityDate="2010-10-19T18:41:58.457" CommunityOwnedDate="2010-10-19T18:41:58.457" />
  <row Id="3775" PostTypeId="2" ParentId="3772" CreationDate="2010-10-19T18:58:56.603" Score="5" ViewCount="" Body="&lt;p&gt;The proposed procedure does not answer your question.  It only estimates the frequency, under the null hypothesis, with which your observed order would occur.  But under that null, to a good approximation, all orders are equally likely, whence your calculation will produce a value close to 1/5! = about 0.83%.  That tells us nothing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One more obvious observation: the order, based on your data, is 4 &gt; 5 &gt; 3 &gt; 2 &gt; 1.  Your estimates of their relative superiorities are 0.61 - 0.40 = 21%, 0.40 - 0.21 = 11%, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, suppose your question concerns the extent to which any of the ${5 \choose 2} = 10$ &lt;em&gt;differences in proportions&lt;/em&gt; could be due to chance under the null hypothesis of no difference.  You can indeed evaluate these ten questions with a permutation test.  However, in each iteration you need to track ten indicators of relative difference in proportion, not one global indicator of the total order.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For your data, a simulation with 100,000 iterations gives the results&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{array}{ccccc}&#xA;  &amp;amp; 5 &amp;amp; 4 &amp;amp; 3 &amp;amp; 2 \cr&#xA; 1 &amp;amp; 0.02439 &amp;amp; 0.0003 &amp;amp; 0.13233 &amp;amp; 0.29961 \cr&#xA; 2 &amp;amp; 0.09763 &amp;amp; 0.00374 &amp;amp; 0.29222 &amp;amp;  \cr&#xA; 3 &amp;amp; 0.20253 &amp;amp; 0.00884 &amp;amp;  &amp;amp; \cr&#xA; 4 &amp;amp; 0.08702 &amp;amp;  &amp;amp;  &amp;amp; &#xA;\end{array}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The differences in proportions between method 4 and methods 1, 2, and 3 are unlikely to be due to chance (with estimated probabilities 0.03%, 0.37%, 0.88%, respectively) but the other differences might be.  There is some evidence (p = 2.44%) of a difference between methods 1 and 5.  Thus it appears you can have confidence that the differences in proportions involved in the relationships 4 &gt; 3, 4 &gt; 2, and 4 &gt; 1 are all positive, and most likely so is the difference in 5 &gt; 1.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-10-19T19:19:20.613" LastActivityDate="2010-10-19T19:19:20.613" CommentCount="3" />
  <row Id="3776" PostTypeId="2" ParentId="3746" CreationDate="2010-10-19T20:11:33.707" Score="2" ViewCount="" Body="&lt;p&gt;Call the HDOP the independent variable. Use this for weighting later on. So you have sets of co-ordinates - call this (x1,y1); (x2,y2), etc...&#xA;First ignore outliers. Calculate the weighted averages of the x co-ordinates as [(x1*h1)+(x2*h2) +....+ (xn*hn)] / [sum(h1,h2,...,hn)] where h1,h2,... is the HDOP value. Do the same for the y co-ordinates. This will give a fairly accurate average value for each co-ordinate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Dealing with outliers can be a bit tricky. How do you know if they are outliers or not? Strictly you need to determine a statistical fit to the observations and within a confidence interval determine if they are genuine or not. Looking at the question the Poison Distribution does come to mind. But this is probably a lot of work and I'm sure you don't want to go into this. Maybe use an approximation? Say you assume that the average co-ordinate value is a good mean to use. Then determine a value for the standard deviation. I think the standard dev or the poison distribution is 1/(mean). Then approximate using the normal distribution and a 95% confidence interval. Say if an observation is outside the interval (mean-*1.645*std dev ; mean + 1.645*std dev) then it is an outlier? Give this a go. Maybe go do a bit of reading on the poison distribution and incorporate the HDOP value into this? &lt;/p&gt;&#xA;" LastActivityDate="2010-10-19T20:11:33.707" />
  <row Id="3778" PostTypeId="2" ParentId="3614" CreationDate="2010-10-19T23:38:01.513" Score="4" ViewCount="" Body="&lt;p&gt;Yet another way to quickly compute the probability distribution of a dice roll would be to use a specialized calculator designed just for that purpose.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.diku.dk/hjemmesider/ansatte/torbenm/&quot; rel=&quot;nofollow&quot;&gt;Torben Mogensen&lt;/a&gt;, a CS professor at &lt;a href=&quot;http://www.diku.dk/&quot; rel=&quot;nofollow&quot;&gt;DIKU&lt;/a&gt; has an excellent dice roller called &lt;a href=&quot;http://topps.diku.dk/torbenm/troll.msp&quot; rel=&quot;nofollow&quot;&gt;Troll&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Troll dice roller and probability calculator prints out the probability distribution (pmf, histogram, and optionally cdf or ccdf), mean, spread, and mean deviation for a variety of complicated dice roll mechanisms. Here are a few examples that show off Troll's dice roll language:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Roll 3 6-sided dice and sum them: &lt;code&gt;sum 3d6&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Roll 4 6-sided dice, keep the highest 3 and sum them: &lt;code&gt;sum largest 3 4d6&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Roll an &quot;exploding&quot; 6-sided die (i.e., any time a &quot;6&quot; comes up, add 6 to your total and roll again): &lt;code&gt;sum (accumulate y:=d6 while y=6)&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Troll's &lt;a href=&quot;http://en.wikipedia.org/wiki/Standard_ML&quot; rel=&quot;nofollow&quot;&gt;SML&lt;/a&gt; &lt;a href=&quot;http://www.diku.dk/hjemmesider/ansatte/torbenm/Troll/Troll.zip&quot; rel=&quot;nofollow&quot;&gt;source code&lt;/a&gt; is available, if you want to see how its implemented.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Professor Morgensen also has a 29-page paper, &quot;&lt;a href=&quot;http://www.diku.dk/hjemmesider/ansatte/torbenm/Troll/RPGdice.pdf&quot; rel=&quot;nofollow&quot;&gt;Dice Rolling Mechanisms in RPGs&lt;/a&gt;,&quot; in which he discusses many of the dice rolling mechanisms implemented by Troll and some of the mathematics behind them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A similar piece of free, open-source software is &lt;a href=&quot;http://www.semistable.com/dicelab/&quot; rel=&quot;nofollow&quot;&gt;Dicelab&lt;/a&gt;, which works on both Linux and Windows.&lt;/p&gt;&#xA;" LastActivityDate="2010-10-19T23:38:01.513" />
  <row Id="3779" PostTypeId="1" AcceptedAnswerId="6135" CreationDate="2010-10-20T00:28:37.190" Score="16" ViewCount="552" Body="&lt;p&gt;Suppose you had a bag with $n$ tiles, each with a letter on it. There are $n_A$ tiles with letter 'A', $n_B$ with 'B', and so on, and $n_*$ 'wildcard' tiles (we have $n = n_A + n_B + \ldots + n_Z + n_*$). Suppose you had a dictionary with a finite number of words. You pick $k$ tiles from the bag without replacement. How would you compute (or estimate) the probability that you can form zero words from the dictionary given the $k$ tiles selected?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For those not familiar with Scrabble (TM), the wildcard character can be used to match any letter. Thus the word [&lt;strong&gt;BOOT&lt;/strong&gt;] could be 'spelled' with the tiles 'B', '*', 'O', 'T'. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To give some idea of the scale of the problem, $k$ is smallish, like 7, $n$ is around 100, and the dictionary contains about 100,000 words of size $k$ or smaller.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;edit:&lt;/strong&gt; By 'form a word', I mean a word of length no greater than $k$. Thus, if the word [&lt;strong&gt;A&lt;/strong&gt;] is in the dictionary, then by drawing even a single 'A' from the bag, one has 'formed a word'. The problem of wildcards is radically simplified if one can assume there are words of length 1 in the dictionary. For if there are, any draw of a wildcard automatically can match a length 1 word, and thus one can concentrate on the case where there are no wildcards. Thus the more slippery form of the problem has no 1-letter words in the dictionary.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, I should explicitly state that the order in which the letters are drawn from the bag is immaterial. One does not have to draw the letters in the 'correct' order of the word.&lt;/p&gt;&#xA;" OwnerUserId="795" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-01-11T08:38:13.353" LastActivityDate="2011-01-11T08:38:13.353" Title="Probability of not drawing a word from a bag of letters in Scrabble" Tags="&lt;sampling&gt;&lt;games&gt;&lt;probability&gt;" AnswerCount="4" CommentCount="5" FavoriteCount="3" />
  <row Id="3780" PostTypeId="2" ParentId="3779" CreationDate="2010-10-20T00:54:20.850" Score="2" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Monte Carlo Approach&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The quick and dirty approach is to do a monte carlo study. Draw $k$ tiles $m$ times and for each draw of $k$ tiles see if you can form a word. Denote the number of times you could form a word by $m_w$. The desired probability would be:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$1 - \frac{m_w}{m}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Direct Approach&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let the number of words in the dictionary be given by $S$. Let $t_s$ be the number of ways in which we can form the $s^\mbox{th}$ word. Let the number of letters needed by the $s^\mbox{th}$ word be denoted by ${m_a, m_b, ..., m_z}$ (i.e., the $s^\mbox{th}$ word needs $m_a$ number of 'a' letters etc). Denote the number of words we can form with all tiles by $N$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$N = \binom{n}{k}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$t_s = \binom{n_a}{m_a} \binom{n_b}{m_b} ... \binom{n_z}{m_z}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Including the impact of wildcard tiles is a bit trickier. I will defer that issue for now.)  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, the desired probability is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$1 - \frac{\sum_s{t_s}}{N}$$&lt;/p&gt;&#xA;" LastActivityDate="2010-10-20T16:26:06.390" CommentCount="8" />
  <row Id="3781" PostTypeId="1" CreationDate="2010-10-20T01:04:56.197" Score="3" ViewCount="93" Body="&lt;p&gt;I've got a dataset where someone counted birds in the breeding season over 10 years. For each year (x site), we want to see how reduced sampling might affect our ability to detect a trend. So to that end, I have simulated various datasets from the original where we cut down sampling to once every 45 days (and 60, 90,120). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I fit the same negative binomial model to the original and each simulated dataset (5oo datasets for each of the different sampling intervals).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So now I have 500 regressions (and coefficients) for each dataset x location. Is there some way to throw a confidence band around these? Is it something very trivial, like computing CI around a mean?&lt;/p&gt;&#xA;" OwnerUserId="1451" LastActivityDate="2010-10-20T12:58:47.653" Title="Is there a way to compute confidence intervals for regression estimates of simulated data?" Tags="&lt;confidence-interval&gt;&lt;negative-binomial&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="1" />
  <row Id="3782" PostTypeId="1" CreationDate="2010-10-20T01:11:04.410" Score="7" ViewCount="205" Body="&lt;p&gt;I designed a field experiment with 4 independent factors but data is not normal and heteroscedastic.  Friedman test (agricolae package) from R only fits for rbd.  Can anybody suggest how to analyze my data please?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks.&lt;/p&gt;&#xA;" OwnerUserId="1627" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-10-20T07:08:22.333" LastActivityDate="2010-10-20T08:53:24.307" Title="How to do factorial analysis for a non-normal and heteroscedastic data?" Tags="&lt;r&gt;&lt;nonparametric&gt;&lt;experiment-design&gt;" AnswerCount="3" FavoriteCount="1" />
  <row Id="3783" PostTypeId="2" ParentId="3782" CreationDate="2010-10-20T01:52:30.530" Score="1" ViewCount="" Body="&lt;p&gt;Package &lt;a href=&quot;http://cc.oulu.fi/~jarioksa/softhelp/vegan.html&quot; rel=&quot;nofollow&quot;&gt;vegan&lt;/a&gt; implements some permutation testing procedures using a distance based approach.  For factor analysis, you should take a look at section 5 of the &lt;a href=&quot;http://cc.oulu.fi/~jarioksa/opetus/metodi/vegantutor.pdf&quot; rel=&quot;nofollow&quot;&gt;documentation&lt;/a&gt;.  There's also more information in the paper:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pubmed/19673867&quot; rel=&quot;nofollow&quot;&gt;On distance-based permutation tests for between-group comparisons&lt;/a&gt; (Reiss et al, 2010)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You might also be interested in skimming this table:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://bama.ua.edu/~jleeper/627/choosestat.html&quot; rel=&quot;nofollow&quot;&gt;Choosing the Correct Statistical Test&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="251" LastEditorUserId="251" LastEditorDisplayName="" LastEditDate="2010-10-20T02:22:15.280" LastActivityDate="2010-10-20T02:22:15.280" CommentCount="1" />
  <row Id="3784" PostTypeId="2" ParentId="3757" CreationDate="2010-10-20T02:40:28.530" Score="5" ViewCount="" Body="&lt;p&gt;First, you should compare models from lmer after fitting with ML (maximum likelihood) since the default is REML.  So something like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;fit.nc &amp;lt;- update(NoCor, REML=FALSE)&#xA;fit.wc &amp;lt;- udpate(WithCor, REML=FALSE)&#xA;anova(fit.nc, fit.wc)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It would help to see the output of the random effects variation from your fits.  For example, to answer: is there a strong correlation between the intercept and slope and what are the variation sizes?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you find that the random slope only model (NoCor) provides the best fit, then this means that the Size variable has a different effect between groups (depending on the variation).  But the no intercept implies that the mean response at some zero level (baseline for your factor Size) is the same across all groups.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A random slope only model is not as common unless informed by theory -- usually we assume baseline variation between groups (random intercept) and then let effects (slope) vary as well.  If you don't think there's a good reason to accept the slope-only model, then you may want to keep the random intercept &amp;amp; slope model since it may conform better to theory.&lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-10-20T02:40:28.530" />
  <row Id="3785" PostTypeId="2" ParentId="3757" CreationDate="2010-10-20T06:31:31.430" Score="3" ViewCount="" Body="&lt;p&gt;In this case it would not be expected to find substantive differences between the 'NoCor' model and the 'WithCor' model that you have specified. This is because 'Size' is a factor, not a numeric covariate, and what changes between the two models is that instead of the random effects being referenced to the intercept (base level) the random effects are set as stand alone. You can see when you contrast them with ANOVA that there is no difference in AIC, BIC and logLik and no differences in the degrees of freedom (both models have the same number of parameters). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've found that it might be better to create dummy variables if you want to estimate a variance component for each level of a factor, without including correlations. Something like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;rt$Size1&amp;lt;- ifelse(rt$Size == &quot;small&quot;, 1, 0)&#xA;rt$Size2&amp;lt;- ifelse(rt$Size == &quot;med&quot;, 1, 0)&#xA;rt$Size3&amp;lt;- ifelse(rt$Size == &quot;large&quot;, 1, 0)&#xA;NoCor2 &amp;lt;- lmer(RT ~ Size + (0+Size1|ID) + (0+Size2|ID) + (0+Size3|ID), data=rt)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You might also want to try the slightly simpler model:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;NoCorHom &amp;lt;- lmer(RT ~ Size + (1|ID) + (1|ID:Size), data=rt)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can see from the model summary that this fits a single variance for the size factor, equivalent to assuming sphericity and homogeneity (just like a regular repeated measures ANOVA).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If Size was numeric then you would be looking at something like the below to compare the correlation parameter:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;NoCor3 &amp;lt;- lmer(RT ~ Size + (1|ID) + (0+Size|ID), data=rt)&#xA;#vs&#xA;Cor &amp;lt;- lmer(RT ~ Size + (1+Size|ID), data=rt)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="966" LastActivityDate="2010-10-20T06:31:31.430" />
  <row Id="3786" PostTypeId="2" ParentId="3782" CreationDate="2010-10-20T07:22:44.740" Score="1" ViewCount="" Body="&lt;p&gt;The &lt;strong&gt;Skillings-Mack test&lt;/strong&gt;  is a general Friedman-type test that can be used in almost any block design with an arbitrary missing-data structure. It's part of the &lt;code&gt;asbio&lt;/code&gt; package for R, and there's a user-written package &lt;code&gt;skilmack&lt;/code&gt; for Stata.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Skillings, J. H., and G. A. Mack. 1981. &lt;a href=&quot;http://www.jstor.org/stable/1268034&quot; rel=&quot;nofollow&quot;&gt;On the use of a Friedman-type statistic in balanced and unbalanced block designs.&lt;/a&gt; &lt;em&gt;Technometrics&lt;/em&gt; 23: 171-177. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Aho, K. &lt;a href=&quot;http://finzi.psych.upenn.edu/R/library/asbio/&quot; rel=&quot;nofollow&quot;&gt;&lt;code&gt;asbio&lt;/code&gt;: A collection of statistical tools for biologists.&lt;/a&gt; Version 0.3-24. 2010-9-18. &lt;em&gt;Comprehensive R Archive Network (CRAN)&lt;/em&gt; 2010-09-19.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Chatfield, M. and Mander, A. &lt;a href=&quot;http://www.stata-journal.com/article.html?article=st0167&quot; rel=&quot;nofollow&quot;&gt;The Skillings–Mack test (Friedman test when there are missing data).&lt;/a&gt; &lt;em&gt;Stata Journal&lt;/em&gt; 9(2):299-305. &lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-10-20T07:22:44.740" />
  <row Id="3787" PostTypeId="1" AcceptedAnswerId="3790" CreationDate="2010-10-20T08:22:36.250" Score="9" ViewCount="476" Body="&lt;p&gt;Hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a unimodal distribution which is moderately skewed, we have the following empirical relationship between mean, mode and median:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Mean-Mode) ~ 3(Mean-Median)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could someone please explain how the relationship was derived? I am  really curious to know. Did Karl Pearson plot thousands of these relationships before forming this conclusion or is there a logical line of reasoning behind this relationship?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you,&lt;/p&gt;&#xA;" OwnerUserId="1636" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-10-20T08:33:13.067" LastActivityDate="2011-02-09T00:02:49.230" Title="Empirical Relationship Between Mean, Median and Mode" Tags="&lt;distributions&gt;&lt;summary-statistics&gt;&lt;proof&gt;&lt;empirical&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="3788" PostTypeId="1" CreationDate="2010-10-20T08:24:46.747" Score="4" ViewCount="217" Body="&lt;p&gt;&lt;strong&gt;1-&lt;/strong&gt; How can I check if a set of data can be assumed as IID data?&#xA;I'm not so familiar with statistics, but I guess I should look at the first lag of autocorrelation for independent distribution. Have no idea about identical distribution condition!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;2-&lt;/strong&gt; It seems that I was not clear enough! &#xA;I'm trying to detect outliers in a series of records (turbulent flow velocity in a river). I transform data into wavelet space and then I shrink the wavelets over a certain threshold.  Since standard deviation is the worst option as an scale estimator, I looked for a new estimator.&#xA;Rousseeuw and Croux developed new robust estimators for measuring dispersion in iid random variables, Sn and Qn.&#xA;I don't know offhand if the high breakdown properties they enjoy carry over to the time-series case or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the answer given by kwak, I can infer that wavelets do NOT follow independent distribution property. Since after shrinkage, location of non-zero elements indicates the spike location in the original time series. Am I true? (shuffling the indices results in losing location of spikes) If so, other scale estimators like median absolute deviation (MAD) are not valid in case of time series as we calculate the median.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How about identical distribution assumption requirements?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;3-&lt;/strong&gt; OK, let me ask my question in simple manner:&#xA;I want to use robust scale estimators Sn and Qn for shrinking a series of wavelets. the wavelets are obtained from decomposing observations of a turbulent flow field velocity vectors collected at 1 Hz sampling rate.&#xA;if the data can be assumed as iid e.g. Qn has breakpoint of 50% and efficiency of 82% (Gaussian distribution).&#xA;My question is &lt;strong&gt;the high breakdown properties they enjoy carry over to the time-series case or not.&lt;/strong&gt;&#xA;Or how can i approve that the wavelets follow iid characteristics.&lt;/p&gt;&#xA;" OwnerUserId="1637" LastEditorUserId="1637" LastEditorDisplayName="" LastEditDate="2010-10-21T09:21:55.823" LastActivityDate="2010-10-21T09:21:55.823" Title="How can the IID assumption be checked in a given dataset?" Tags="&lt;distributions&gt;&lt;time-series&gt;&lt;autocorrelation&gt;" AnswerCount="3" CommentCount="2" />
  <row Id="3789" PostTypeId="2" ParentId="3727" CreationDate="2010-10-20T08:32:13.313" Score="2" ViewCount="" Body="&lt;p&gt;This is an interesting question, which doesn't really have a good solution. There a few different ways of tackling this problem. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Assume an underlying distribution and match moments - as suggested in the answers by @ivant and @onestop. One downside is that the multivariate generalisation may be unclear.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Saddlepoint approximations. In this paper:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Gillespie, C.S. and Renshaw, E. &lt;a href=&quot;http://dx.doi.org/10.1016/j.mbs.2006.08.026&quot; rel=&quot;nofollow&quot;&gt;An improved saddlepoint approximation.&lt;/a&gt; &lt;em&gt;Mathematical Biosciences&lt;/em&gt;, 2007.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We look at recovering a pdf/pmf when given only the first few moments. We found that this approach works when the skewness isn't too large.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Laguerre expansions:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Mustapha, H. and Dimitrakopoulosa, R. &lt;a href=&quot;http://dx.doi.org/10.1016/j.camwa.2010.08.008&quot; rel=&quot;nofollow&quot;&gt;Generalized Laguerre expansions of multivariate probability densities with moments&lt;/a&gt;. &lt;em&gt;Computers &amp;amp; Mathematics with Applications&lt;/em&gt;, 2010. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The results in this paper seem more promising, but I haven't coded them up.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="8" LastActivityDate="2010-10-20T08:32:13.313" />
  <row Id="3790" PostTypeId="2" ParentId="3787" CreationDate="2010-10-20T08:34:16.963" Score="6" ViewCount="" Body="&lt;p&gt;It's not a full explanation (i may come back to it depending on the comments), but here's an intuitive derivation:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Denoting $\mu$ the mean ($\neq$ average), $m$ the median, $\sigma$ one standard deviation, $M$ the mode, $sgn()$ the sign function and $X$ the (random) dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's well known that $|\mu-m|\leq\sigma$. Certainly, Chebyshev's inequality must imply that $|\mu-M|&amp;lt;3\sigma$. Finally, it's easy to show that $sgn(\mu-M)=sgn(\mu-m)$ (with the convention that if $sgn(\mu-m)==0$, then both sides equal 1).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All these hold for any unimodal distribution (regardless of skew).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: this is a frequent textbook exercise: &#xA;$\begin{eqnarray}&#xA;|\mu-m|&amp;amp;=&amp;amp;|E(X-m)| \&#xA;&amp;amp;\leq&amp;amp; E|X-m| \&#xA;&amp;amp;\leq&amp;amp; E|X-\mu|  \&#xA;&amp;amp;=&amp;amp; E\sqrt((X-\mu)^2) \&#xA;&amp;amp;\leq&amp;amp; \sqrt(E(X-\mu)^2) \&#xA;&amp;amp;=&amp;amp;\sigma&#xA;\end{eqnarray}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first equality derives from the definition of the mean, the third comes about because the mean is the unique minimiser (among all $c$'s) of $E|X-c|$ and the fourth from Jensen's inequality (i.e. the definition of a convex function). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Chebyshef's inequality states that: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(|X-\mu|&gt;\alpha)\leq \frac{\sigma^2}{\alpha^2}$&lt;/p&gt;&#xA;" OwnerUserId="603" LastEditorUserId="603" LastEditorDisplayName="" LastEditDate="2010-10-20T15:55:12.290" LastActivityDate="2010-10-20T15:55:12.290" CommentCount="6" />
  <row Id="3791" PostTypeId="2" ParentId="3782" CreationDate="2010-10-20T08:53:24.307" Score="0" ViewCount="" Body="&lt;p&gt;Hello Kathy,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you suggest you &quot;designed&quot; an experiment, it would be better if can you give a description of your design and data set. Even if the data is heteroscedastic and non-normal, probably some variable transformations might help and you may be able to take advantage of the design. The t-test is fairly robust to the normality assumptions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;S.&lt;/p&gt;&#xA;" OwnerUserId="1307" LastActivityDate="2010-10-20T08:53:24.307" CommentCount="2" />
  <row Id="3792" PostTypeId="2" ParentId="3788" CreationDate="2010-10-20T08:57:39.543" Score="2" ViewCount="" Body="&lt;p&gt;You don't frame the two problems the right way. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given a random dataset, ie a collection of observations $x_{ij}$ lying in general &lt;a href=&quot;http://en.wikipedia.org/wiki/General_position&quot; rel=&quot;nofollow&quot;&gt;position&lt;/a&gt; you can always make the $n$ $x_{i}\in\mathbb{R}^p$ independent from one another by randomly shuffling the $n$ indexes. The real question is whether you will lose information doing this. In some context you will (times series, panel data, cluster analysis, functional analysis,...) in others you won't. That's for the first I in IID.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The 'ID' is also defined with respect to what you mean by distribution. Any mixture of distribution is also a distribution. Most often, 'ID' is a portmanteau term for 'unimodal'.&lt;/p&gt;&#xA;" OwnerUserId="603" LastEditorUserId="603" LastEditorDisplayName="" LastEditDate="2010-10-20T11:01:37.220" LastActivityDate="2010-10-20T11:01:37.220" CommentCount="4" />
  <row Id="3793" PostTypeId="1" AcceptedAnswerId="3807" CreationDate="2010-10-20T09:55:52.417" Score="3" ViewCount="72" Body="&lt;p&gt;Is there a term that describes what I'm trying to do below?&#xA;Also, how would you do this using something like JMP or Excel? (or do I need to code this in something like perl?)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given this sort of data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ID| opened     | closed     | quantity&#xA;--------------------------------------&#xA;1 | 2010-01-01 | 2010-01-03 | 1&#xA;2 | 2010-01-02 |            | 2&#xA;3 | 2010-01-02 | 2010-10-05 | 3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'd like to get this data and then graph with x being a time line and Y being total quantity open:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;on date    | total quantity open&#xA;---------------------------------&#xA;2010-01-01 | 1&#xA;2010-01-02 | 6&#xA;2010-01-03 | 5&#xA;2010-01-04 | 5&#xA;2010-01-05 | 2 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1641" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-10-20T15:32:06.020" LastActivityDate="2010-10-20T15:32:06.020" Title="Is there a term for generating time-line based data from individual points? Also how would I do this?" Tags="&lt;terminology&gt;&lt;excel&gt;" AnswerCount="2" CommentCount="4" />
  <row Id="3794" PostTypeId="2" ParentId="3793" CreationDate="2010-10-20T10:16:28.447" Score="0" ViewCount="" Body="&lt;p&gt;One simple algorithm which could be implemented easily would be:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Step 1:&#xA;Add variables to your data frame; one for each time you want to graph. In pseudo code&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;if time_i &amp;gt;= opened AND  time_i &amp;lt; closed then quantity else 0&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Step 2:&#xA; sum the rows for each generated time variable to get the quantity at each time&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Step 3:&#xA;It would then be straight forward to graph the resulting data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All these steps would be straightforward in Excel, R, or any number of other programs.&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-10-20T10:16:28.447" CommentCount="6" />
  <row Id="3795" PostTypeId="1" CreationDate="2010-10-20T10:33:59.067" Score="3" ViewCount="251" Body="&lt;p&gt;I am looking for a software tool (preferably open source) to draw structural equation/mixture models efficiently and prettily. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;After looking into xfig and graphviz I now stick to the general vector graphics package &lt;a href=&quot;http://inkscape.org&quot; rel=&quot;nofollow&quot;&gt;inkscape&lt;/a&gt; because it seems most flexible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to poll the stat.stackexchange community: How do you draw your structural equation/mixture models? What software do you use?&lt;/p&gt;&#xA;" OwnerUserId="767" LastActivityDate="2010-10-20T14:55:41.667" Title="How do you draw structural equation/MPLUS models?" Tags="&lt;modeling&gt;&lt;data-visualization&gt;&lt;sem&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="3796" PostTypeId="2" ParentId="3795" CreationDate="2010-10-20T10:58:25.770" Score="2" ViewCount="" Body="&lt;p&gt;I use the &lt;a href=&quot;http://cran.r-project.org/web/packages/psych/index.html&quot; rel=&quot;nofollow&quot;&gt;psych&lt;/a&gt; R package for CFA and John Fox's &lt;a href=&quot;http://cran.r-project.org/web/packages/sem/index.html&quot; rel=&quot;nofollow&quot;&gt;sem&lt;/a&gt; package with simple SEM. Note that the graphical backend is graphviz. I don't remember if the &lt;a href=&quot;http://lavaan.ugent.be/&quot; rel=&quot;nofollow&quot;&gt;lavaan&lt;/a&gt; package provides similar or better facilities.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Otherwise, the &lt;a href=&quot;http://www.vcu.edu/mx/&quot; rel=&quot;nofollow&quot;&gt;Mx software&lt;/a&gt; for genetic modeling features a graphical interface in its Windows flavour, and you can export the model with path coefficients.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-20T11:13:51.817" LastActivityDate="2010-10-20T11:13:51.817" />
  <row Id="3798" PostTypeId="2" ParentId="570" CreationDate="2010-10-20T11:31:09.770" Score="7" ViewCount="" Body="&lt;p&gt;I worked with Laura Trinchera who contributed a nice R package for PLS-path modeling, &lt;a href=&quot;http://cran.r-project.org/web/packages/plspm/&quot; rel=&quot;nofollow&quot;&gt;plspm&lt;/a&gt;. It includes several graphical output for various kind of 2- and k-block data structures.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I just discovered the &lt;a href=&quot;http://www.bethanykok.com/plotSEMM.html&quot; rel=&quot;nofollow&quot;&gt;plotSEMM&lt;/a&gt; R package. It's more related to your second point, though, and is restricted to graphing bivariate relationships. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for recent references on diagnostic plot for SEMs, here are two papers that may be interesting (for the second one, I just browsed the abstract recently but cannot find an ungated version):&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Sanchez BN, Houseman EA, and Ryan LM. &lt;a href=&quot;http://deepblue.lib.umich.edu/bitstream/2027.42/65983/1/j.1541-0420.2008.01022.x.pdf&quot; rel=&quot;nofollow&quot;&gt;Residual-Based Diagnostics for Structural Equation Models&lt;/a&gt;. &lt;em&gt;Biometrics&lt;/em&gt; (2009) 65, 104–115&lt;/li&gt;&#xA;&lt;li&gt;Yuan KH and Hayashi K. &lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pubmed/20853955&quot; rel=&quot;nofollow&quot;&gt;Fitting data to model: Structural equation modeling diagnosis using two scatter plots&lt;/a&gt;, &lt;em&gt;Psychological Methods&lt;/em&gt; (2010)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-10-20T11:31:09.770" CommentCount="6" />
  <row Id="3799" PostTypeId="1" AcceptedAnswerId="3813" CreationDate="2010-10-20T11:59:04.173" Score="8" ViewCount="138" Body="&lt;p&gt;I've got following problem:&lt;br&gt;&#xA;- We have set of N people&lt;br&gt;&#xA;- We have set of K images&lt;br&gt;&#xA;- Each person rates some number of images. A person might like or not like an image (these are the only two possibilites)  .&#xA;- The problem is how to calculate likelihood that some person likes a particular image.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'll give example presenting my intuition.&lt;br&gt;&#xA;N = 4&lt;br&gt;&#xA;K = 5&lt;br&gt;&#xA;+ means that person likes image&lt;br&gt;&#xA;- means that person doesn't like image&lt;br&gt;&#xA;0 means that person hasn't been asked about the image, and that value should be predicted  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x 1 2 3 4 5    &#xA;1 + - 0 0 +   &#xA;2 + - + 0 +  &#xA;3 - - + + 0  &#xA;4 - 0 - - -&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Person 1 will probably like image 3 because, person 2 has similar preferences and person 2 likes image 3.&lt;br&gt;&#xA;Person 4 will probably not like image 2 because no one  else likes it and in addition person 4 does not like most images.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any well known method, which can be used to calculate such likelihood?  &lt;/p&gt;&#xA;" OwnerUserId="1643" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-10-21T09:06:04.907" LastActivityDate="2010-10-21T09:06:04.907" Title="Probability that someone will like image" Tags="&lt;missing-data&gt;&lt;rating&gt;" AnswerCount="2" CommentCount="3" FavoriteCount="1" />
  <row Id="3800" PostTypeId="2" ParentId="3799" CreationDate="2010-10-20T12:29:20.677" Score="5" ViewCount="" Body="&lt;p&gt;This looks like a good problem for machine learning, so I'll concentrate on this group of methods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First and the most obvious idea is the kNN algorithm. There you first calculate the similarity among viewers and then predict the missing votes with the average vote on this picture cast by similar users. For details see &lt;a href=&quot;http://en.wikipedia.org/wiki/KNN&quot; rel=&quot;nofollow&quot;&gt;Wikipedia&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another idea is to grow unsupervised random forest on this data (either way, with attributes in images or people, whatever is better) and impute missing data based on the forest structure; the whole method is implemented and described in R &lt;code&gt;randomForest&lt;/code&gt; package, look for &lt;code&gt;rfImpute&lt;/code&gt; function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, you can restructure the problem to a plain classification task, say make an object of each zero in matrix and try to think of some reasonable descriptors (like average viewer vote, average image vote, vote of a most, second most, ... similar viewer, same with image, possibly some external data (average hue of image, age of voter, etc). And then try various classifiers on this data (SVM, RF, NB, ...). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are also some more complex possibilities; for an overview you can look for Netflix prize challenge (which was a similar problem) solutions.&lt;/p&gt;&#xA;" OwnerUserId="88" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-10-20T12:49:42.360" LastActivityDate="2010-10-20T12:49:42.360" CommentCount="1" />
  <row Id="3801" PostTypeId="2" ParentId="3781" CreationDate="2010-10-20T12:58:47.653" Score="2" ViewCount="" Body="&lt;p&gt;Yes, you have repetitions of each scenario, so you can make a histogram of each coefficient values. Now treat it as a real distribution and just find a band that encloses this 99, 95 or whatever per cent of its area -- this will be the nonparametric approximation of CI.&lt;br&gt;&#xA;The simpler way is to assume normality and just count standard deviation over repetitions; this is useful when the number of repetitions is low, but will obviously give bad results when the over-repetition distribution is far from normal.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-10-20T12:58:47.653" CommentCount="2" />
  <row Id="3802" PostTypeId="2" ParentId="3795" CreationDate="2010-10-20T13:33:51.450" Score="5" ViewCount="" Body="&lt;p&gt;I use &lt;a href=&quot;http://openmx.psyc.virginia.edu/&quot; rel=&quot;nofollow&quot;&gt;OpenMx&lt;/a&gt; for SEM modeling where I simply use the &lt;a href=&quot;http://openmx.psyc.virginia.edu/docs/OpenMx/latest/_static/Rdoc/omxGraphviz.html&quot; rel=&quot;nofollow&quot;&gt;omxGraphViz&lt;/a&gt; function to return a dotfile.  I haven't found it too inflexible -- the default output looks pretty good and though I've rarely needed to modify the dotfile, it's not hard to do.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Update&lt;/em&gt; By the way, Graphviz can output SVG files, which can be imported into Inkscape, giving you the best of both worlds.  :)&lt;/p&gt;&#xA;" OwnerUserId="251" LastEditorUserId="251" LastEditorDisplayName="" LastEditDate="2010-10-20T14:55:41.667" LastActivityDate="2010-10-20T14:55:41.667" CommentCount="1" />
  <row Id="3803" PostTypeId="2" ParentId="3779" CreationDate="2010-10-20T14:29:00.610" Score="6" ViewCount="" Body="&lt;p&gt;Srikant is right: a Monte Carlo study is the way to go.  There are two reasons.  First, the answer depends &lt;em&gt;strongly&lt;/em&gt; on the structure of the dictionary.  Two extremes are (1) the dictionary contains every possible single-letter word.  In this case, the chance of not making a word in a draw of $1$ or more letters is zero.  (2) The dictionary contains &lt;em&gt;only&lt;/em&gt; words formed out of a single letter (&lt;em&gt;e.g.&lt;/em&gt;, &quot;a&quot;, &quot;aa&quot;, &quot;aaa&quot;, &lt;em&gt;etc&lt;/em&gt;.).  The chance of not making a word in a draw of $k$ letters is easily determined and obviously is nonzero.  Any definite closed-form answer would have to incorporate the entire dictionary structure and would be a truly awful and long formula.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second reason is that MC indeed is feasible: you just have to do it right.  The preceding paragraph provides a clue: don't just generate words at random and look them up; instead, &lt;em&gt;analyze the dictionary first&lt;/em&gt; and exploit its structure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One way represents the words in the dictionary as a tree.  The tree is rooted at the empty symbol and branches on each letter all the way down; its leaves are (of course) the words themselves.  However, &lt;em&gt;we can also insert all nontrivial permutations of every word into the tree,&lt;/em&gt; too (up to $k!-1$ of them for each word).  This can be done efficiently because one does not have to store all those permutations; only the edges in the tree need to be added.  The leaves remain the same.  In fact, this can be simplified further by insisting that the tree be followed &lt;em&gt;in alphabetical order&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words, to determine whether a multiset of $k$ characters is in the dictionary, &lt;em&gt;first arrange the elements into sorted order,&lt;/em&gt; then look for this sorted &quot;word&quot; in a tree constructed from the sorted representatives of the words in the original dictionary.  This will actually be smaller than the original tree because it merges all sets of words that are sort-equivalent, such as {stop, post, pots, opts, spot}.  In fact, in an English dictionary this class of words would never be reached anyway because &quot;so&quot; would be found first.  Let's see this in action.  The sorted multiset is &quot;opst&quot;; the &quot;o&quot; would branch to all words containing only the letters {o, p, ..., z}, the &quot;p&quot; would branch to all words containing only {o, p, ..., z} and at most one &quot;o&quot;, and finally the &quot;s&quot; would branch to the leaf &quot;so&quot;!  (I have assumed that none of the plausible candidates &quot;o&quot;, &quot;op&quot;, &quot;po&quot;, &quot;ops&quot;, or &quot;pos&quot; are in the dictionary.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A modification is needed to handle wildcards: I'll let the programmer types among you think about that.  It won't increase the dictionary size (it should decrease it, in fact); it will slightly slow down the tree traversal, but without changing it in any fundamental way.  In any dictionary that contains a single-letter word, like English (&quot;a&quot;, &quot;i&quot;), there is no complication: the presence of a wildcard means you can form a word!  (This hints that the original question might not be as interesting as it sounds.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The upshot is that a single dictionary lookup requires (a) sorting a $k$-letter multiset and (b) traversing no more than $k$ edges of a tree.  The running time is $O(k \log(k))$.  If you cleverly generate random multisets in sorted order (I can think of several efficient ways to do this), the running time reduces to $O(k)$.  Multiply this by the number of iterations to get the total running time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I bet you could conduct this study with a real Scrabble set and a million iterations in a matter of seconds.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-10-20T14:29:00.610" CommentCount="8" />
  <row Id="3804" PostTypeId="1" AcceptedAnswerId="3832" CreationDate="2010-10-20T14:43:05.913" Score="2" ViewCount="220" Body="&lt;p&gt;Suppose I'm modeling a set of processes using a beta-binomial prior. I can build parameterized beta-binomial models that average over large groups of the processes to give reasonable, although coarse, priors. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$p_i \sim \beta B(n, \alpha_i, \beta_i)$ (roughly)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know how to update those priors using observed partial data via Bayes' rule. However, for a subset of the priors, I actually have a little more historical data that I'd like to incorporate into the prior, call it $h_j$, where $j \in h$ is a subset of the $i$s. So the result would be an updated distribution, call it $p'_i$. That additional data is a scalar. For example, if I've got a beta-binomial with $n=9$, $\alpha=2$ and $\beta=3$ (see the examples for the &lt;code&gt;dbetabin.ab&lt;/code&gt; function in the VGAM R package), it has a mode of 3, but I might have additional prior information that suggests the mode should be closer to 6. I happen to know that this additional information is only modestly predictive ($r$ of .4, say). But it's still better than nothing, and for this particular process, it's known to be a better predictor than the expected value of my existing beta-binomial prior ($r$ of around .3).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, what I'm looking for, is a way to update the beta-binomial, using this scalar, so that the result is also a beta-binomial, which I can then update like any of my other process models as data comes in. (That is, I need a closed-form expression.) $(\alpha'_i, \beta'_i) = f(\alpha_i, \beta_i, h_i, \theta)$, where $\theta$ has something to do with the relative estimated predictiveness of the original beta-binomial and the scalar $h$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What's a reasonable approach here? Is there a way to adjust the $\alpha$ and $\beta$ parameters so that the central tendency is pulled an appropriate amount towards my modestly-predictive scalar? I'm happy to use cross-validation or something to identify a weighting parameter, if that's the right way to go about this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="6" LastEditorUserId="6" LastEditorDisplayName="" LastEditDate="2010-10-20T18:16:12.263" LastActivityDate="2010-10-21T05:51:11.413" Title="Updating a beta-binomial" Tags="&lt;bayesian&gt;&lt;beta-binomial&gt;&lt;prior&gt;&lt;updating&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="1" />
  <row Id="3805" PostTypeId="1" AcceptedAnswerId="3806" CreationDate="2010-10-20T14:47:38.207" Score="3" ViewCount="99" Body="&lt;p&gt;I have a very simple model.  This model uses data that are not given as continuous distributions, but are described by percentiles.  What is the best way to sample these percentile bins, when the bins are of unequal size?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, for example, to select the body weight for a given individual, I pick a random number between 0-100, then match this value to the nearest percentile.  I don't interpolate or extrapolate, I just match the value I draw to the nearest bin.  (Extrapolating isn't a good idea given the data.)  Let's say, for body weight, the percentiles I have are 25, 50 and 75.  But this gives bin sizes of 37.5 (0-37.5), 25 (37.5-62.5), and 37.5 (62.5-100).  So because of the unequal bin sizes, I'm going to be sampling both the 25% and 75% bins much more than I'll be sampling the median, 50%, bin.  This is the opposite of what I'd like to happen.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I could weight the bins, but that seems arbitrary.  Or, instead of drawing my random number from a uniform distribution 0-100, I could draw it from a normal distribution centered at the median, but that also seems arbitrary.  Or, alternatively, I'd love to be convinced that I don't actually have a problem here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas on how I could better set this up?  Thanks!&lt;/p&gt;&#xA;" OwnerUserId="1645" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-10-20T15:06:32.823" LastActivityDate="2010-10-20T15:06:32.823" Title="Sampling with unequal bins?" Tags="&lt;distributions&gt;&lt;modeling&gt;&lt;sampling&gt;&lt;discrete-data&gt;&lt;percentile&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="3806" PostTypeId="2" ParentId="3805" CreationDate="2010-10-20T15:05:41.183" Score="2" ViewCount="" Body="&lt;p&gt;In the spirit of simplicity, while aiming to attain &lt;em&gt;some&lt;/em&gt; realism (which is not possible without both interpolation and extrapolation), consider fitting a distribution to the percentiles and sampling from it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For body weights we can expect a power between 1/2 and 1/3 to be normally distributed.  By trial and error you can find a power that symmetrizes the percentiles (specifically, look for a $p$ for which $q_{50}^p - q_{25}^p$ is approximately equal to $q_{75}^p - q_{50}^p$).  The transformed percentiles easily determine a unique Normal distribution: $q_{50}^p$ is its mean and the scaled  IQR $(q_{75}^p - q_{25}^p) / 1.349$ is its standard deviation.  To obtain a random body weight, draw from this normal and invert the transformation (that is, apply the $1/p$ power).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This example is offered as an illustration only, not as a general recipe.  For other attributes (like age or, in another context, recurrence times of major floods) it's wise to let empirical knowledge and theory suggest an appropriate distribution to fit to the percentiles: a Normal distribution is not always appropriate, even after transforming the percentiles for symmetry.  It is also wise in many situations to allow for outliers.  &lt;em&gt;E.g.&lt;/em&gt;, you could &quot;contaminate&quot; your normal distribution by occasionally drawing from a distribution with a higher mean and SD, simulating the occasional grossly obese person.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-10-20T15:05:41.183" CommentCount="1" />
  <row Id="3807" PostTypeId="2" ParentId="3793" CreationDate="2010-10-20T15:30:33.467" Score="2" ViewCount="" Body="&lt;p&gt;Use SUMIF() to compute the total open to date and the total closed to date.  The difference at any time is the total currently open.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's suppose the data you show are in the range A1:D4 in Excel.  Reserve four columns for your output: the two shown plus two for intermediate calculations.  Let's suppose they are columns E:H.  The formulas are:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Column E has the dates in ascending order exactly as shown in your output.&#xA;Column F is computed by propagating this formula from F2 down as far as needed:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;=SUMIF(B$2:B$4,&quot;&amp;lt;=&quot; &amp;amp; $E2,$D$2:$D$4)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Extend the row index &quot;4&quot; as far down as needed to cover your data.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Column G is computed by propagating the formula from F2 over to G2 and then down.  For example, the formula in G3 will be&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;=SUMIF(C$2:C$4,&quot;&amp;lt;=&quot; &amp;amp; $E3,$D$2:$D$4)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Column H is the difference of columns G and F: it contains the results you need.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My spreadsheet looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ID  Opened  Closed  Quantity    Date    Open    Closed  Net&#xA;1   1/1/2010    1/3/2010    1   1/1/2010    1   0   1&#xA;2   1/2/2010                2   1/2/2010    6   0   6&#xA;3   1/2/2010    1/5/2010    3   1/3/2010    6   1   5&#xA;                                1/4/2010    6   1   5&#xA;                                1/5/2010    6   4   2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The translation to R is straightforward for those who prefer that environment.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-10-20T15:30:33.467" CommentCount="1" />
  <row Id="3808" PostTypeId="2" ParentId="3752" CreationDate="2010-10-20T15:51:18.260" Score="1" ViewCount="" Body="&lt;p&gt;There seem to be two different questions here, which I'll try to split:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) how is KS, kernel smoothing, different from KDE, kernel density estimation ?&#xA;Well, say I have an estimator / smoother / interpolator&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;est( xi, fi -&amp;gt; gridj, estj )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and also happen to know the &quot;real&quot; densityf() at the xi.  Then running&#xA;&lt;code&gt;est( x, densityf )&lt;/code&gt;&#xA;must give an estimate of densityf(): a KDE.&#xA;It may well be that KSs and KDEs are evaluated differently &amp;mdash;&#xA;different smoothness criteria, different norms &amp;mdash;&#xA;but I don't see a fundamental difference.  What am I missing ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) How does dimension affect estimation or smoothing, &lt;em&gt;intuitivly&lt;/em&gt; ?&#xA;Here's a toy example, just to help intuition.&#xA;Consider a box of N=10000 points in a uniform grid,&#xA;and a window, a line or square or cube, of W=64 points within it:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;                1d          2d          3d          4d&#xA;---------------------------------------------------------------&#xA;data            10000       100x100     22x22x22    10x10x10x10&#xA;side            10000       100         22          10&#xA;window          64          8x8         4x4x4       2.8^4&#xA;side ratio      .64 %       8 %         19 %        28 %&#xA;dist to win     5000        47          13          7&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here &quot;side ratio&quot; is window side / box side,&#xA;and &quot;dist to win&quot; is a rough estimate of the mean distance&#xA;of a random point in the box to a randomly-placed window.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does this make any sense at all ?&#xA;(A picture or applet would really help: anyone ?)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The idea is that a fixed-size window within a fixed-size box&#xA;has very different nearness to the rest of the box, in 1d 2d 3d 4d.&#xA;This is for a uniform grid;&#xA;maybe the strong dependence on dimension carries over to&#xA;other distributions, maybe not.&#xA;Anyway, it looks like a strong general effect, an aspect of the curse of dimensionality.&lt;/p&gt;&#xA;" OwnerUserId="557" LastActivityDate="2010-10-20T15:51:18.260" />
  <row Id="3809" PostTypeId="2" ParentId="3804" CreationDate="2010-10-20T16:09:43.653" Score="1" ViewCount="" Body="&lt;p&gt;Assume that prior2 is a beta random variable and set $\alpha$ and $\beta$ as needed subject to the constraint that $\frac{\alpha-1}{\alpha + \beta -2} = 6$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In response to your comment:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Getting to prior2:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Fix either $\alpha$ or $\beta$ at the same value as prior1 and tweak the other to match the desired mode. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If the above does not work then you can use whatever constraints you want to impose (e.g., same variance) and use some sort of routine (e.g., optimization) to get to your desired mode (e.g., Min abs($\frac{\alpha-1}{\alpha + \beta -2} - 6)$ subject to constraints) or simply play around till your prior2 parameters are consistent with your constraints.  &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Accommodating the fact that you do not fully believe in prior2:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A principled way to approach the issue of 20% trust in prior2 is to assume mixture priors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, your prior is: $f(\alpha_1,\beta_1|-) 0.8 + f(\alpha_2,\beta_2|-) 0.2$. You could multiply your likelihood with the above mixture priors to get a beta-binomial model.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" LastActivityDate="2010-10-20T18:39:15.817" CommentCount="4" />
  <row Id="3810" PostTypeId="1" CreationDate="2010-10-20T16:38:35.493" Score="3" ViewCount="433" Body="&lt;p&gt;I have data from a survey comprised of several measures that used different Likert-type scaling (4-, 5-, and 6-point scales).  I would like to run a principal components analysis using the data from these measures.  It seems to me that I need to transform this data in some way so that the power of all items is equivalent prior to analysis.  However, I am uncertain how to proceed.  &lt;/p&gt;&#xA;" OwnerUserId="1647" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-03-30T18:38:33.207" LastActivityDate="2011-03-30T18:38:33.207" Title="Data transformation for Principal Components Analysis from different Likert scales" Tags="&lt;pca&gt;&lt;data-transformation&gt;&lt;categorical-data&gt;&lt;psychometrics&gt;&lt;likert&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="1" />
  <row Id="3812" PostTypeId="2" ParentId="3810" CreationDate="2010-10-20T17:20:23.490" Score="5" ViewCount="" Body="&lt;p&gt;As suggested by @whuber, you can &quot;abstract&quot; the scale effect by working with a standardized version of your data. If you're willing to accept that an interval scale is the support of each of your item (i.e. the distance between every two response categories would have the same meaning for every respondents), then linear correlations are fine. But you can also compute &lt;a href=&quot;http://en.wikipedia.org/wiki/Polychoric_correlation&quot; rel=&quot;nofollow&quot;&gt;polychoric correlation&lt;/a&gt; to better account for the discretization of a latent variable (see the R package &lt;a href=&quot;http://cran.r-project.org/web/packages/polycor/index.html&quot; rel=&quot;nofollow&quot;&gt;polycor&lt;/a&gt;). Of note, it's a largely more computer-intensive job, but it works quite well in R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another possibility is to combine optimal scaling within your PCA, as implemented in the &lt;a href=&quot;http://cran.r-project.org/web/packages/homals/index.html&quot; rel=&quot;nofollow&quot;&gt;homals&lt;/a&gt; package. The idea is to find a suitable non-linear transformation of each scale, and this is very nicely described by Jan de Leeuw in the accompagnying vignette or the JSS article, &lt;a href=&quot;http://www.jstatsoft.org/v31/i04/paper&quot; rel=&quot;nofollow&quot;&gt;Gifi Methods for Optimal Scaling in R: The Package homals&lt;/a&gt;. There are several examples included.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a more thorough understanding of this approach with any factorial method, see the work of &lt;a href=&quot;http://takane.brinkster.net/Yoshio/&quot; rel=&quot;nofollow&quot;&gt;Yoshio Takane&lt;/a&gt; in the 80s.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Similar points were raised by @Jeromy and @mbq on related questions, &lt;a href=&quot;http://stats.stackexchange.com/questions/539/does-it-ever-make-sense-to-treat-categorical-data-as-continuous&quot;&gt;Does it ever make sense to treat categorical data as continuous?&lt;/a&gt;, &lt;a href=&quot;http://stats.stackexchange.com/questions/548/how-can-i-use-optimal-scaling-to-scale-an-ordinal-categorical-variable&quot;&gt;How can I use optimal scaling to scale an ordinal categorical variable?&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-20T18:18:35.790" LastActivityDate="2010-10-20T18:18:35.790" CommentCount="2" />
  <row Id="3813" PostTypeId="2" ParentId="3799" CreationDate="2010-10-20T19:01:44.367" Score="5" ViewCount="" Body="&lt;p&gt;I believe this is a standard problem of &lt;a href=&quot;http://en.wikipedia.org/wiki/Collaborative_filtering&quot; rel=&quot;nofollow&quot;&gt;Collaborative Filtering&lt;/a&gt;. A google search gives thousands of results.&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-10-20T19:01:44.367" CommentCount="4" />
  <row Id="3814" PostTypeId="1" CreationDate="2010-10-20T19:09:31.910" Score="34" ViewCount="1042" Body="&lt;p&gt;I recently asked a question regarding general principles around &lt;a href=&quot;http://stats.stackexchange.com/questions/3460/reviewing-statistics-in-papers&quot;&gt;reviewing statistics in papers&lt;/a&gt;. What I would now like to ask, is what particularly irritates you when reviewing a paper, i.e. what's the best way to really annoy a statistical referee!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One example per answer please.&lt;/p&gt;&#xA;" OwnerUserId="8" LastActivityDate="2011-05-21T16:04:11.060" CommunityOwnedDate="2010-10-20T19:12:23.073" Title="How to annoy a statistical referee" Tags="&lt;big-list&gt;&lt;referee&gt;" AnswerCount="9" CommentCount="2" FavoriteCount="15" />
  <row Id="3815" PostTypeId="2" ParentId="3814" CreationDate="2010-10-20T19:13:42.330" Score="19" ViewCount="" Body="&lt;p&gt;Irene Stratton and colleague published a short paper about a closely related question:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Stratton IM, Neil A. &lt;a href=&quot;http://www.blackwell-synergy.com/doi/abs/10.1111/j.1464-5491.2004.01443.x&quot; rel=&quot;nofollow&quot;&gt;How to ensure your paper is rejected by the statistical reviewer&lt;/a&gt;. &lt;em&gt;Diabetic Medicine&lt;/em&gt; 2005; 22(4):371-373. &lt;/p&gt;&#xA;" OwnerUserId="449" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-10-20T19:27:15.717" LastActivityDate="2010-10-20T19:27:15.717" CommunityOwnedDate="2010-10-20T19:13:42.330" CommentCount="0" />
  <row Id="3816" PostTypeId="2" ParentId="3814" CreationDate="2010-10-20T19:28:41.487" Score="25" ViewCount="" Body="&lt;p&gt;What particularly irritates &lt;em&gt;me personally&lt;/em&gt; is people who clearly used user-written packages for statistical software but don't cite them properly, or at all, thereby failing to give any credit to the authors. Doing so is particularly important when the authors are in academia and their jobs depend on publishing papers &lt;em&gt;that get cited&lt;/em&gt;. (Perhaps I should add that, in my field, many of the culprits are not statisticians.)&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-10-20T19:28:41.487" CommunityOwnedDate="2010-10-20T19:28:41.487" CommentCount="3" />
  <row Id="3817" PostTypeId="2" ParentId="3814" CreationDate="2010-10-20T19:29:29.613" Score="21" ViewCount="" Body="&lt;p&gt;Goodness me, so many things come to mind...&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Stepwise regression&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://biostat.mc.vanderbilt.edu/twiki/bin/view/Main/CatContinuous&quot; rel=&quot;nofollow&quot;&gt;Splitting continuous data into&#xA;groups&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Giving p-values but no measure of&#xA;effect size&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Describing data using the mean and&#xA;the standard deviation without&#xA;indicating whether the data were&#xA;more-or-less symmetric and unimodal&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Figures without clear captions (are&#xA;those error bars standard errors of&#xA;the mean, or standard deviations&#xA;within groups, or what?)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="1352" LastActivityDate="2010-10-20T19:29:29.613" CommunityOwnedDate="2010-10-20T19:29:29.613" CommentCount="6" />
  <row Id="3818" PostTypeId="1" CreationDate="2010-10-20T19:37:42.267" Score="1" ViewCount="295" Body="&lt;p&gt;I have three variables,&lt;br&gt;&#xA;1. Irrational Beliefs (Categorical)&lt;br&gt;&#xA;2. Anxiety State/ Trait (Categorical)&lt;br&gt;&#xA;3. Personality Traits (Categorical)&lt;br&gt;&#xA;Which statistical analyses can be used?&lt;/p&gt;&#xA;" OwnerUserId="1649" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-10-20T20:18:42.270" LastActivityDate="2010-10-21T07:23:20.837" ClosedDate="2010-10-22T16:04:22.067" Title="Analysis with three categorical variables" Tags="&lt;categorical-data&gt;" AnswerCount="2" CommentCount="5" FavoriteCount="0" />
  <row Id="3819" PostTypeId="2" ParentId="3814" CreationDate="2010-10-20T19:41:32.543" Score="13" ViewCount="" Body="&lt;p&gt;The code used to generate the simulated results is not provided. After asking for the code, it demands additional work to get it to run on a referee generated dataset.&lt;/p&gt;&#xA;" OwnerUserId="603" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-10-21T15:43:30.690" LastActivityDate="2010-10-21T15:43:30.690" CommunityOwnedDate="2010-10-20T19:41:32.543" />
  <row Id="3820" PostTypeId="1" AcceptedAnswerId="3825" CreationDate="2010-10-20T19:43:42.297" Score="4" ViewCount="212" Body="&lt;p&gt;I've been asked a question - if I calculate some sd, can I change one value and still keep the same sd.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The answer is simply yes.  For example: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sd(c(2,3,4))&#xA;sd(c(3,4,5))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But what I wondered then is, assuming you change k values, under what rules do you change them so to always keep the same sd (is it true to ask what are the degrees of freedom here?!)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not sure how this applies to anything practical, but I imagine there was a good deal of theoretical work on such questions - but I don't know where to even look for them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="253" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-10-20T20:56:22.620" LastActivityDate="2010-10-20T20:56:22.620" Title="When does the sd stay the same, even after values in the sample were changed?" Tags="&lt;standard-deviation&gt;" AnswerCount="2" CommentCount="10" />
  <row Id="3821" PostTypeId="2" ParentId="3820" CreationDate="2010-10-20T19:48:29.650" Score="2" ViewCount="" Body="&lt;p&gt;Suppose that you have a random variable $X$ and you wish to find the set of transformations $Y=f(X)$ such that the standard deviation of $Y$ is the identical to the standard deviation of $X$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider first the set of linear transformations:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Y = a X + b$ where $a, b$ are constants.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is clear that:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Var(Y) = a^2 Var(X)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, the only set of linear transformations that preserve standard deviations are linear translations but not scaling by any factor other than -1 (see comment by mbq to this answer). I suspect that non-linear transformations do not preserve standard deviations.&lt;/p&gt;&#xA;" LastActivityDate="2010-10-20T20:15:24.157" CommentCount="10" />
  <row Id="3822" PostTypeId="1" CreationDate="2010-10-20T19:57:52.263" Score="2" ViewCount="103" Body="&lt;p&gt;I have 6 similarity matrices reflecting the proximity of 100 pairs of sounds. Similarity indices are in the range [0,1], with higher values indicating sounds perceived as more alike. What is the most appropriate metric I can use to estimate the inter-rater reliability between them?&#xA;Thanks a lot.&lt;/p&gt;&#xA;" OwnerUserId="1564" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-20T20:09:39.270" LastActivityDate="2010-10-22T07:59:16.753" Title="Inter-rater reliability between similarity matrices" Tags="&lt;reliability&gt;&lt;inter-rater&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="1" />
  <row Id="3823" PostTypeId="2" ParentId="3814" CreationDate="2010-10-20T20:23:38.740" Score="15" ViewCount="" Body="&lt;p&gt;&lt;em&gt;Plagiarism&lt;/em&gt; (theoretical or methodological). My first review was indeed for a paper figuring many unreferenced copy/paste from a well-established methodological paper published 10 years ago.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just found a couple of interesting papers on this topic: &lt;a href=&quot;http://wrt-howard.syr.edu/Bibs/PlagScience.htm&quot; rel=&quot;nofollow&quot;&gt;Authorship and plagiarism in science&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the same vein, I find &lt;a href=&quot;http://en.wikipedia.org/wiki/Fabrication_%28science%29&quot; rel=&quot;nofollow&quot;&gt;&lt;em&gt;falsification&lt;/em&gt;&lt;/a&gt; (of data or results) the worst of all.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-21T06:27:00.043" LastActivityDate="2010-10-21T06:27:00.043" CommunityOwnedDate="2010-10-20T20:23:38.740" CommentCount="4" />
  <row Id="3824" PostTypeId="2" ParentId="3814" CreationDate="2010-10-20T20:29:24.090" Score="9" ViewCount="" Body="&lt;p&gt;When we ask the authors for &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;minor comment about an idea we have (in this sense, this not considered as a reason for rejecting the paper but just to be sure the authors are able to discuss another POV), or &lt;/li&gt;&#xA;&lt;li&gt;unclear or contradicting results,&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;and that authors don't really answer in case (1) or that the incriminated results in (2) disappear from the MS.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-10-20T20:29:24.090" CommunityOwnedDate="2010-10-20T20:29:24.090" />
  <row Id="3825" PostTypeId="2" ParentId="3820" CreationDate="2010-10-20T20:47:18.890" Score="4" ViewCount="" Body="&lt;p&gt;The question is about the data, not random variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let $X = (x_1, x_2, \ldots, x_n)$ be the data and $Y = (y_1, y_2, \ldots, y_n)$ be additive changes to the data so that the new values are $(x_1+y_1, \ldots, x_n+y_n)$.  From&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Var(X) = Var(X+Y) = Var(X) + 2 Cov(X,Y) + Var(Y)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;we deduce that&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Var(Y) + 2 Cov(X,Y) = 0$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;is necessary for the variance to be unchanged.  Add in $n-k$ additional constraints to zero out all but $k$ of the $y_i$ (there are ${n \choose k}$ ways to do this) and note that all $n-k+1$ constraints almost everywhere have linearly independent derivatives.  By the Implicit Function Theorem, this defines a manifold of $n - (n-k+1)$ = $k-1$ dimensions (plus perhaps a few singular points): those are your degrees of freedom.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, with $X = (2, 3, 4)$ we compute&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$3 Var(y) = y_1^2 + y_2^2 + y_3^2 - (y_1+y_2+y_3)^2/3$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$3 Cov(x,y) = (2 y_1 + 3 y_2 + 4 y_3) - 3(y_1 + y_2 + y_3)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we set (arbitrarily) $y_2 = y_3 = 0$ the solutions are $y_1 = 0$ (giving the original data) and $y_2 = 3$ (the posted solution).  If instead we require $y_2 = 0$ the only solution is $y_2 = 0$: you can't keep the SD constant by changing $y_2$.  Similarly we can set $y_3 = -3$ while fixing the other two values.  That exhausts the possibilities for $k=1$. If we set only $y_3 = 0$ (&lt;em&gt;i.e.&lt;/em&gt;, $k$ = 2) then we get a set of solutions&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$y_1 = y_2 = 0$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$y_1 = 4, y_2 = 2$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$y_2^2 - y_1 y_2 + y_1^2 - 3y_1 == 0$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;which consists of an ellipse and two isolated points in the $(y_1, y_2)$ plane.  Similar sets of solutions arise in the choices $y_2 = 0$ and $y_1 = 0$.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-10-20T20:53:20.620" LastActivityDate="2010-10-20T20:53:20.620" CommentCount="2" />
  <row Id="3826" PostTypeId="1" AcceptedAnswerId="3877" CreationDate="2010-10-20T20:51:59.437" Score="2" ViewCount="1030" Body="&lt;p&gt;How does one calculate Cohen's d and confidence intervals after logit in Stata?&lt;/p&gt;&#xA;" LastActivityDate="2010-10-22T07:24:42.113" Title="How does one calculate Cohen's d and confidence intervals after logit in Stata?" Tags="&lt;confidence-interval&gt;&lt;stata&gt;&lt;effect-size&gt;&lt;cohens-d&gt;" AnswerCount="2" />
  <row Id="3827" PostTypeId="2" ParentId="3826" CreationDate="2010-10-20T21:02:48.473" Score="6" ViewCount="" Body="&lt;p&gt;Cohen’s d is not directly available in Stata, and you have to resort on external macros, e.g. &lt;a href=&quot;http://econpapers.repec.org/software/bocbocode/s456738.htm&quot; rel=&quot;nofollow&quot;&gt;sizefx&lt;/a&gt; (&lt;code&gt;ssc install sizefx&lt;/code&gt;). It works fine if you have to series of values, but I found it less handy when you work with a full data set because there's no possibility to pass options to this command (e.g. &lt;code&gt;by()&lt;/code&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyway, you can still use the original formula (with pooled SDs), &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\delta_c=\frac{\bar x_1-\bar x_2}{s_p}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $s_p=\sqrt{\frac{(n_1-1)s_1^2+(n_2-1)s_2^2)}{(n_1+n_2-2)}}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an example by hand:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;. webuse lbw&#xA;. logit low age smoke&#xA;. graph box age, by(low)&#xA;. tabstat age, by(low) statistics(mean sd N)&#xA;&#xA;Summary for variables: age&#xA;     by categories of: low (birthweight&amp;lt;2500g)&#xA;&#xA;     low |      mean        sd         N&#xA;---------+------------------------------&#xA;       0 |  23.66154  5.584522       130&#xA;       1 |  22.30508  4.511496        59&#xA;---------+------------------------------&#xA;   Total |   23.2381  5.298678       189&#xA;----------------------------------------&#xA;&#xA;. display &quot;Cohen's d: = &quot; (23.66154-22.30508) / sqrt((129*(5.584522)^2+58*(4.511496)^2)/187)&#xA;&#xA;Cohen's d: = .25714326&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is in agreement with what R would give:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(MBESS)&#xA;res &amp;lt;- smd(Mean.1=23.66154, Mean.2=22.30508, &#xA;           s.1=5.584522, s.2=4.511496, n.1=130, n.2=59)&#xA;ci.smd(smd=res, n.1=130, n.2=59, conf.level=0.95)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;that is an effect size of 0.257 with 95% CI [-0.052;0.566].&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In contrast, &lt;code&gt;sizefx&lt;/code&gt; gives results that differ a little (I have use &lt;code&gt;separate age, by(low)&lt;/code&gt; and collapse the results in a new data window, here two columns labeled &lt;code&gt;age0&lt;/code&gt; and &lt;code&gt;age1&lt;/code&gt;), the ES version calculated above corresponding to what is referred to as Hedge's g below (unless I miss something in the &lt;a href=&quot;http://fmwww.bc.edu/repec/bocode/s/sizefx.ado&quot; rel=&quot;nofollow&quot;&gt;code&lt;/a&gt; I read):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;. sizefx age0 age1&#xA;&#xA;Cohen's d and Hedges' g for: age0 vs. age1&#xA;Cohen's d statistic (pooled variance) = .26721576&#xA;Hedges' g statistic = .26494154&#xA;&#xA;Effect size correlation (r) for: age0 vs. age1&#xA;ES correlation r = .13243109&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-21T09:57:00.433" LastActivityDate="2010-10-21T09:57:00.433" CommentCount="5" />
  <row Id="3828" PostTypeId="2" ParentId="3818" CreationDate="2010-10-20T21:46:53.727" Score="2" ViewCount="" Body="&lt;p&gt;If you're after a general way to model multiway &lt;a href=&quot;http://en.wikipedia.org/wiki/Contingency_table&quot; rel=&quot;nofollow&quot;&gt;contingency tables&lt;/a&gt;, one powerful approach is to use &lt;a href=&quot;http://en.wikipedia.org/wiki/Poisson_regression&quot; rel=&quot;nofollow&quot;&gt;Poisson regression&lt;/a&gt;, often called a &lt;em&gt;log-linear model&lt;/em&gt; in this context. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A classic paper on this is &lt;a href=&quot;http://www.jstor.org/stable/2347125&quot; rel=&quot;nofollow&quot;&gt;Nelder (1974)&lt;/a&gt;, but there are now more suitable resources for pedagogical purposes freely available online thanks to the generosity of some lecturers or their universities, such as Princeton's Germán Rodríguez: &lt;a href=&quot;http://data.princeton.edu/wws509/notes/c5.pdf&quot; rel=&quot;nofollow&quot;&gt;pdf version&lt;/a&gt;, &lt;a href=&quot;http://data.princeton.edu/wws509/notes/c5s1.html&quot; rel=&quot;nofollow&quot;&gt;html version&lt;/a&gt;. There must be less mathematical introductory material out there though?&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-10-20T21:46:53.727" CommentCount="1" />
  <row Id="3829" PostTypeId="2" ParentId="3814" CreationDate="2010-10-20T22:55:12.767" Score="9" ViewCount="" Body="&lt;p&gt;When they don't sufficiently explain their analysis and/or include simple errors that make it difficult to work out what actually was done. This often includes throwing around a lot of jargon, by way of explanation, which is more ambiguous than the author seems to realize and also may be misused.&lt;/p&gt;&#xA;" LastActivityDate="2010-10-20T22:55:12.767" CommunityOwnedDate="2010-10-20T22:55:12.767" CommentCount="1" />
  <row Id="3830" PostTypeId="2" ParentId="3788" CreationDate="2010-10-21T00:46:34.603" Score="0" ViewCount="" Body="&lt;p&gt;If your looking to check a couple assumptions about your residuals being iid, there are a couple graphical methods I would suggest you try:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;QQ Plots: Good ways to detect whether the residuals fit a normal distribution. This is a useful method for figuring out if the residuals are identically distributed.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Plotting residuals against index value or time: If your observations are taken from earliest to latest, and there appears to be a correlation between what value of the index you are on and the value of the residual, you may question the first I in IID. If you fit a linear model that is supposed to be time-independent, knowing what value of your index you have/day of the study should not give you any additional information you cannot get simply by knowing the mean of the residuals.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="1118" LastActivityDate="2010-10-21T00:46:34.603" CommentCount="2" />
  <row Id="3831" PostTypeId="2" ParentId="3788" CreationDate="2010-10-21T00:47:55.787" Score="1" ViewCount="" Body="&lt;p&gt;Whether a set of observations are iid or not is a decision that is typically taken after a consideration of the underlying data generating process. In your case, the underlying data generating process seems to be the measurements of the speed of a river. I would not consider these observations to be independent. If a particular measurement is on the high end of the scale the next measurement is also likely to be on the high end of the scale. In other words, if I know one measurement I can infer something about the likely values for my next measurement. However, the values are likely to be identically distributed as the errors in your measurement probably come from the methodology used to measure velocity. I would imagine that you would use the same methodology to collect multiple measurements. You could of course use formal statistical tests to see if your observations are iid but for that you have to specify a data generating process, estimate the model under iid assumptions and examine for residuals for deviation from iid.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, do note that I know nothing about engineering wavelets, wavelet space so I may be way-off in my above assumptions/answer.&lt;/p&gt;&#xA;" LastActivityDate="2010-10-21T00:53:22.890" CommentCount="3" />
  <row Id="3832" PostTypeId="2" ParentId="3804" CreationDate="2010-10-21T04:34:05.837" Score="1" ViewCount="" Body="&lt;p&gt;Lets see if I understand Harlan's (and Srikant's) formulation correctly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\pi_1 \sim beta(\alpha_1,\beta_1)$$&#xA;$$\pi_2 \sim beta(\alpha_2,\beta_2)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say, $\pi_1$ corresponds to the set of data for which you have less information &lt;em&gt;apriori&lt;/em&gt; and $\pi_2$ is for the more precise data set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using Srikant's formulation:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\pi(p) \propto \pi_1(p) \alpha + \pi_2(p) (1-\alpha)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, the complete hierarchical formulation will be:&#xA;$$\alpha \sim beta(\alpha_0,\beta_0)$$&#xA;$$p|\alpha \sim \pi(p)$$&#xA;$$y_i | p \sim B(n_i,p) $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I assume here that $y_i|p$ are iid. I don't know if this is a valid assumption in your case. Am I correct?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can choose $\alpha_0$ and $\beta_0$ in such a way that mean of this beta distribution is 0.8 (or 0.2) acc. to your formulation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now the MCMC sampling can be done, by using OpenBUGS or JAGS (untested). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I will add more to this (and recheck formulation) as soon as I get more time. But please point out if you see a fallacy in my argument.&lt;/p&gt;&#xA;" OwnerUserId="1307" LastEditorUserId="1307" LastEditorDisplayName="" LastEditDate="2010-10-21T05:51:11.413" LastActivityDate="2010-10-21T05:51:11.413" />
  <row Id="3833" PostTypeId="2" ParentId="3818" CreationDate="2010-10-21T06:43:36.237" Score="4" ViewCount="" Body="&lt;p&gt;@onestop is literally correct. If you have three unordered categorical variables, techniques like loglinear modelling are appropriate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;@Neelam However, &lt;strong&gt;I doubt that your data is unordered categorical&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From my experience with measures of psychological scales measuring irrational beliefs, anxiety states and traits, and personality traits, the scales are numeric.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, a typical personality scale might have 10 items with each item measured on a 5-point scale. Thus, if you were to sum the scale you get scores ranging from a minimum possible score of 10 to a maximum possible score of 50. I.e., there are 41 possible values. Thus, in some sense the variables are categorical, but they are also ordinal, and they are also typically treated as numeric variables. See my comments to this earlier question regarding &lt;a href=&quot;http://stats.stackexchange.com/questions/206/discrete-and-continuous/210#210&quot;&gt;discrete and continuous variables&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, in the empirical research that I read with similar scales, researchers typically treat such scales as numeric variables. Thus, methods such as correlation, regression, and PCA are standard.&lt;/p&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-10-21T07:23:20.837" LastActivityDate="2010-10-21T07:23:20.837" CommentCount="3" />
  <row Id="3834" PostTypeId="2" ParentId="3822" CreationDate="2010-10-21T07:22:40.340" Score="2" ViewCount="" Body="&lt;p&gt;My first idea would be to try some kind of cluster analysis (e.g. &lt;a href=&quot;http://en.wikipedia.org/wiki/Hierarchical_clustering&quot; rel=&quot;nofollow&quot;&gt;hierarchical clustering&lt;/a&gt;) on each similarity matrix, and compare the classification trees across raters. We can derive a similarity index from all dendrograms, as discussed here, &lt;a href=&quot;http://stats.stackexchange.com/questions/3672/a-measure-to-describe-the-distribution-of-a-dendrogram&quot;&gt;A measure to describe the distribution of a dendrogram&lt;/a&gt;, or in this review, &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.164.6189&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;Comparing Clusterings - An Overview&lt;/a&gt; from Wagner and Wagner.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You benefit from working with already existing distance matrices, thus such methods will really reflect the nature of your data, and you can still derive a single numerical value to quantify the closeness of method-specific assessments. The following article may be interesting, if you need to refer to existing work:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Hamer, RM and Cunningham, JW. &lt;a href=&quot;http://apm.sagepub.com/content/5/1/63.abstract&quot; rel=&quot;nofollow&quot;&gt;Cluster&#xA;  Analyzing Profile Data Confounded with&#xA;  Interrater Differences: A Comparison&#xA;  of Profile Association Measures&lt;/a&gt;.&#xA;  &lt;em&gt;Applied Psychological Measurement&lt;/em&gt;&#xA;  (1981) 5(1): 63-72.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Another approach would be to apply some kind of &lt;a href=&quot;http://en.wikipedia.org/wiki/Principal_component_analysis&quot; rel=&quot;nofollow&quot;&gt;Principal Component Analysis&lt;/a&gt; on each similarity matrix, and keep only the first principal component (the linear combination of all 100 items that account for the maximum of variance). More precisely, as you work with (dis)similarity indices or a particular distance/proximity metric, it is sometimes referred to as Principal Coordinate Analysis or &lt;a href=&quot;http://en.wikipedia.org/wiki/Multidimensional_scaling&quot; rel=&quot;nofollow&quot;&gt;Multidimensional Scaling&lt;/a&gt; (MDS), although PCA and MDS would yield similar results when dissimilarities are defined as euclidean distances. There is a working example in Izenman's book (&lt;a href=&quot;http://books.google.fr/books?id=1CuznRORa3EC&amp;amp;printsec=frontcover&amp;amp;dq=modern+multivariate+statistical+techniques&amp;amp;source=bl&amp;amp;ots=Uq98CVdyIf&amp;amp;sig=LE3N0YX4M1dP93cvZC4cotrnsRE&amp;amp;hl=fr&amp;amp;ei=Bum_TPbhGsuO4gb7_ZiCDQ&amp;amp;sa=X&amp;amp;oi=book_result&amp;amp;ct=result&amp;amp;resnum=4&amp;amp;ved=0CCsQ6AEwAw&quot; rel=&quot;nofollow&quot;&gt;&lt;em&gt;Modern Multivariate Statistical Techniques&lt;/em&gt;&lt;/a&gt;, chapter 13, &quot;perceptions of color in human vision&quot;, pp. 468-470) and a discussion on so-called &lt;em&gt;all-pairs design&lt;/em&gt; pp. 471-472. You can then compare the 6 linear combinations (i.e., the weights associated to each sound by rater-specific MDS) to assess their consistency across raters. There, an ICC (as described in my &lt;a href=&quot;http://ttp://stats.stackexchange.com/questions/3762/the-best-measure-of-reliability-for-interval-data-between-0-and-1-closed&quot; rel=&quot;nofollow&quot;&gt;previous answer&lt;/a&gt;) could make sense, but I don't know of any application of it in this particular case.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-22T07:59:16.753" LastActivityDate="2010-10-22T07:59:16.753" />
  <row Id="3835" PostTypeId="1" AcceptedAnswerId="3846" CreationDate="2010-10-21T07:58:40.967" Score="3" ViewCount="52" Body="&lt;p&gt;Hi,This example was taken from Mathematical Statistics : A Unified Introduction (ISBN 9780387227696), page 58, under the section 'The Principle of Least Squares'. I think my problem has more to do with algebra but since this is a statistics book, I thought I should post here...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyway, I need to estimate k base on this equation:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/JQM6B.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The goal is to reach here:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/nqyzM.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I managed to follow the example until I got stuck here:&#xA; &lt;img src=&quot;http://i.imgur.com/1Xs42.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The explanation to reach to the goal from above is described as follows:&#xA; &lt;img src=&quot;http://i.imgur.com/j2UyX.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know how to get k but I totally don't understand how the 'middle term' was eliminated. Please help. I will provide more details if needed. &lt;/p&gt;&#xA;" OwnerUserId="1636" LastActivityDate="2010-10-22T01:18:58.907" Title="Estimating k in d=kv" Tags="&lt;least-squares&gt;" AnswerCount="1" />
  <row Id="3836" PostTypeId="1" AcceptedAnswerId="3839" CreationDate="2010-10-21T08:42:45.930" Score="5" ViewCount="150" Body="&lt;p&gt;I would like to create a vector of functions using a two agruments function 'func', for instance this one:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;func = function(number, coefficient) {  &#xA;     return(coefficient*number)  &#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;here is how I create the vector of functions:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;vector_of_functions = NULL  &#xA;for (i in 1:4) {  &#xA;     vector_of_functions = c(vector_of_functions, &#xA;                             function(number) func(number=number, coefficient=i))  &#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My issue is that all functions that compose my vector are the same, even if they have been created using different &lt;code&gt;i&lt;/code&gt; through the loop. It seams that they are evaluated using the last value of &lt;code&gt;i&lt;/code&gt; (which is a global variable here).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anybody have an idea?&lt;/p&gt;&#xA;" OwnerUserId="1709" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-10-21T08:52:59.013" LastActivityDate="2010-10-21T09:22:36.303" Title="How to create a vector of functions?" Tags="&lt;r&gt;" AnswerCount="1" FavoriteCount="2" />
  <row Id="3837" PostTypeId="2" ParentId="3051" CreationDate="2010-10-21T09:17:02.463" Score="3" ViewCount="" Body="&lt;pre&gt;&lt;code&gt;library(zoo)&#xA;x=c(4, 5, 7, 3, 9, 8)&#xA;rollmean(x,3)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;or&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(TTR)&#xA;x=c(4, 5, 7, 3, 9, 8)&#xA;SMA(x,3)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1709" LastActivityDate="2010-10-21T09:17:02.463" />
  <row Id="3838" PostTypeId="1" CreationDate="2010-10-21T09:18:01.947" Score="0" ViewCount="74" Body="&lt;p&gt;Perhaps this question is a bit borderline for this site (although I'm hoping it's a reasonable question for someone specialising in bioinformatics).&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;I'm after a straightforward explanation of the difference between 18S rRNA and 18S rDNA.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The 18S rRNA is frequently used in phylogeny (see &lt;a href=&quot;http://en.wikipedia.org/wiki/18S_ribosomal_RNA&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/18S_ribosomal_RNA&lt;/a&gt;).  For example, GenBank ref. AY742884 (which I accessed from &lt;a href=&quot;http://www.ncbi.nlm.nih.gov/nuccore/55274595&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;) includes&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;LOCUS       AY742884                1846 bp    DNA     linear   INV 17-OCT-2005&#xA;DEFINITION  Physopelta gutta 18S ribosomal RNA gene, partial sequence.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which makes me confused, since it uses both terms DNA and RNA.&lt;/p&gt;&#xA;" OwnerUserId="386" LastActivityDate="2010-10-21T09:18:01.947" ClosedDate="2010-10-21T09:25:16.910" Title="What's the difference between 18S rRNA and 18S rDNA?" Tags="&lt;bioinformatics&gt;" CommentCount="3" />
  <row Id="3839" PostTypeId="2" ParentId="3836" CreationDate="2010-10-21T09:22:36.303" Score="5" ViewCount="" Body="&lt;p&gt;Use a &lt;a href=&quot;http://en.wikipedia.org/wiki/Factory_function&quot; rel=&quot;nofollow&quot;&gt;factory function&lt;/a&gt; with a &lt;a href=&quot;http://en.wikipedia.org/wiki/Closure_%28computer_science%29&quot; rel=&quot;nofollow&quot;&gt;closure&lt;/a&gt; over its argument (which will hold the value of the looping variable):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; # the factory function&#xA;&amp;gt; makefunc &amp;lt;- function(x) { x; function() x }&#xA;&amp;gt; funclist &amp;lt;- list()&#xA;&amp;gt; for (i in 1:3) funclist[[i]] &amp;lt;- makefunc(i)&#xA;&amp;gt; funclist[[1]]()&#xA;[1] 1&#xA;&amp;gt; funclist[[2]]()&#xA;[1] 2&#xA;&amp;gt; &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-10-21T09:22:36.303" CommentCount="1" />
  <row Id="3840" PostTypeId="1" CreationDate="2010-10-21T10:49:45.380" Score="4" ViewCount="163" Body="&lt;p&gt;I have two time series, series1 and series2. My aim is to find &lt;strong&gt;how much Series2 is different from Series1, automatically/quantitatively&lt;/strong&gt;.&lt;br&gt;&#xA;Image can be seen in its original size by &lt;a href=&quot;http://img713.imageshack.us/img713/3401/image001eg.gif&quot; rel=&quot;nofollow&quot;&gt;clicking here&lt;/a&gt;.&#xA;&lt;img src=&quot;http://i.imgur.com/0N5fN.gif&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Series1 is the expected result.&#xA;Series2 is the test/incoming series.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am providing a histogram plot, where Series2 is represented in dark brown colour. You can also note in the x-axis between 221 and 353 there is a significant variation. ie Series2 is less than Series1. I am coding using C++.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think, crosscorrelation will help, but produces a value based on similarity rather than dissimilarity. I see people talk about Kolmogorov-Smirnov Test. Is this the test which i should be performing?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;UPDATE 1: I am trying to perform a template matching. I have divided my template image in to 8x8 blocks as well as my incoming test image. I am trying to compare one block in template image with the same block(based on the spatial pixel positions) in the test image. I calculate the intensity sum within each block. I obtain series1 for the Template image and have Series2 for the test image.&lt;/p&gt;&#xA;" OwnerUserId="1655" LastEditorUserId="1352" LastEditorDisplayName="" LastEditDate="2010-10-21T11:20:27.300" LastActivityDate="2010-10-21T14:06:00.943" Title="Automatic test measuring dissimilarity between two time series" Tags="&lt;distributions&gt;&lt;time-series&gt;&lt;computational-statistics&gt;&lt;cross-correlation&gt;&lt;kolmogorov-smirnov&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="1" />
  <row Id="3841" PostTypeId="1" AcceptedAnswerId="3866" CreationDate="2010-10-21T11:07:03.723" Score="10" ViewCount="86" Body="&lt;p&gt;Hello-&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have two years of data which looks basically like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Date   &lt;strong&gt;&lt;em&gt;_&lt;/em&gt;__&lt;em&gt;&lt;/strong&gt;    Violence Y/N? _&lt;/em&gt;  Number of patients&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1/1/2008    &lt;strong&gt;&lt;em&gt;_&lt;/em&gt;___&lt;em&gt;&lt;/strong&gt;    0  &lt;strong&gt;&lt;/em&gt;__&lt;em&gt;_&lt;/em&gt;__&lt;em&gt;_&lt;/em&gt;____&lt;/strong&gt; 11&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2/1/2008 &lt;strong&gt;&lt;em&gt;_&lt;/em&gt;__&lt;em&gt;_&lt;/em&gt;&lt;/strong&gt;       0  &lt;strong&gt;&lt;em&gt;_&lt;/em&gt;__&lt;em&gt;_&lt;/em&gt;__&lt;em&gt;_&lt;/em&gt;__&lt;/strong&gt; 11&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3/1/2008 &lt;strong&gt;&lt;em&gt;_&lt;/em&gt;____&lt;/strong&gt;&lt;em&gt;1  &lt;strong&gt;&lt;/em&gt;__&lt;em&gt;_&lt;/em&gt;__&lt;em&gt;_&lt;/em&gt;____&lt;/strong&gt; 12&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4/1/2008 &lt;strong&gt;&lt;em&gt;_&lt;/em&gt;____&lt;/strong&gt;&lt;em&gt;0  &lt;strong&gt;&lt;/em&gt;__&lt;em&gt;_&lt;/em&gt;__&lt;em&gt;_&lt;/em&gt;____&lt;/strong&gt; 12&lt;/p&gt;&#xA;&#xA;&lt;p&gt;...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;31/12/2009_&lt;strong&gt;&lt;em&gt;_&lt;/em&gt;__&lt;/strong&gt;      0_&lt;strong&gt;&lt;em&gt;_&lt;/em&gt;__&lt;em&gt;_&lt;/em&gt;__&lt;em&gt;_&lt;/em&gt;__&lt;/strong&gt;                 14&lt;/p&gt;&#xA;&#xA;&lt;p&gt;i.e. two years of observations, one per day, of a psychiatric ward, which indicate whether there was a violence incident on that day (1 is yes, 0 no) as well as the number of patients on the ward. The hypothesis that we wish to test is that more patients on the ward is associated with an increased probability of violence on the ward.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We realise, of course, that we will have to adjust for the fact that when there are more patients on the ward, violence is more likely because there are just more of them- we are interested in whether each individual’s probability of violence goes up when there are more patients on the ward.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've seen several papers which just use logistic regression, but I think that is wrong because there is an autoregressive structure (although, looking at the autocorrelation function, it doesn’t get above .1 at any lag, although this is above the “significant” blue dashed line that R draws for me).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just to make things more complicated, I can if I wish to break down the results into individual patients, so the data would look just as it does above, except I would have the data for each patient, 1/1/2008, 2/1/2008 etc. and an ID code going down the side so the data would show the whole history of incidents for each patient separately (although not all patients are present for all days, not sure whether that matters).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to use lme4 in R to model the autoregressive structure within each patient, but some Googling comes up with the quotation “lme4 is not set up to deal with autoregressive structures”. Even if it were, I’m not sure I grasp how to write the code anyway.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just in case anyone notices, I asked a question like this a while ago, they are different datasets with different problems, although actually solving this problem will help with that one (someone suggested I use mixed methods previously, but this autoregression thing has made me unsure how to do this).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I’m a bit stuck and lost to be honest. Any help gratefully received!&lt;/p&gt;&#xA;" OwnerUserId="199" LastActivityDate="2010-10-21T19:22:27.857" Title="Two years of data describing occurence of violence- testing association with number of patients on ward" Tags="&lt;r&gt;&lt;mixed-model&gt;&lt;autocorrelation&gt;&lt;longitudinal&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="1" />
  <row Id="3842" PostTypeId="1" AcceptedAnswerId="3843" CreationDate="2010-10-21T11:13:44.080" Score="4" ViewCount="246" Body="&lt;p&gt;I want to create a bardiagram for these data in R (read from a CVS file):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Experiment_Name  MetricA   MetricB&#xA;Just_X             2        10&#xA;Just_X_and_Y       3        20&lt;/p&gt;&#xA;&#xA;&lt;p&gt;to have the following diagram:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/kGwFA.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am beginner and I do not know even how to start.&lt;/p&gt;&#xA;" OwnerUserId="1389" LastEditorUserId="1352" LastEditorDisplayName="" LastEditDate="2010-10-21T11:18:38.177" LastActivityDate="2010-10-22T10:34:59.290" Title="How to create a barplot diagram where bars are side-by-side in R" Tags="&lt;r&gt;&lt;data-visualization&gt;&lt;barplot&gt;" AnswerCount="3" FavoriteCount="3" />
  <row Id="3843" PostTypeId="2" ParentId="3842" CreationDate="2010-10-21T11:44:58.290" Score="6" ViewCount="" Body="&lt;p&gt;I shall assume that you are able to import your data in R with &lt;code&gt;read.table()&lt;/code&gt; or the short-hand &lt;code&gt;read.csv()&lt;/code&gt; functions. Then you can apply any summary functions you want, for instance &lt;code&gt;table&lt;/code&gt; or &lt;code&gt;mean&lt;/code&gt;, as below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x &amp;lt;- replicate(4, rnorm(100))&#xA;apply(x, 2, mean)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;or&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x &amp;lt;- replicate(2, sample(letters[1:2], 100, rep=T))&#xA;apply(x, 2, table)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The idea is to end up with a matrix or table for the summary values you want to display.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the graphical output, look at the &lt;code&gt;barplot()&lt;/code&gt; function with the option &lt;code&gt;beside=TRUE&lt;/code&gt;, e.g.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;barplot(matrix(c(5,3,8,9),nr=2), beside=T, &#xA;        col=c(&quot;aquamarine3&quot;,&quot;coral&quot;), &#xA;        names.arg=LETTERS[1:2])&#xA;legend(&quot;topleft&quot;, c(&quot;A&quot;,&quot;B&quot;), pch=15, &#xA;       col=c(&quot;aquamarine3&quot;,&quot;coral&quot;), &#xA;       bty=&quot;n&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The &lt;code&gt;space&lt;/code&gt; argument can be used to add an extra space between juxtaposed bars.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/xUv00.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-21T11:56:50.343" LastActivityDate="2010-10-21T11:56:50.343" CommunityOwnedDate="2010-10-21T20:53:54.653" />
  <row Id="3844" PostTypeId="2" ParentId="1053" CreationDate="2010-10-21T12:17:31.507" Score="1" ViewCount="" Body="&lt;p&gt;I found &quot;Analysis of survival data&quot; by Cox and Oakes (Chapman and Hall Monographs on Statistics and Applied Probability - vol. 21) to be very readable and informative.  No material on survival analysis in R though.&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2010-10-21T12:17:31.507" CommunityOwnedDate="2010-10-21T12:17:31.507" />
  <row Id="3845" PostTypeId="1" AcceptedAnswerId="5257" CreationDate="2010-10-21T12:32:34.307" Score="2" ViewCount="160" Body="&lt;p&gt;Hi all,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please see &lt;a href=&quot;http://stats.stackexchange.com/questions/3708/forecasting-unemployment-rate&quot;&gt;question&lt;/a&gt; for the background.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Following the advice of @kwak and @Andy W, I have decided to use the package &lt;strong&gt;plm&lt;/strong&gt; in &lt;code&gt;R&lt;/code&gt; to fit my model. Here an excerpt of the data &lt;code&gt;df&lt;/code&gt; (the numbers are made up, not real data!):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;    reg year          ur         mur cl&#xA;1    1 2001 0.000698717 0.012483361  1&#xA;2    2 2001 0.008283762 0.011899896  1&#xA;3    3 2001 0.001863817 0.012393738  1&#xA;4    4 2001 0.005344206 0.012126016  1&#xA;5    5 2001 0.007475083 0.011962102  1&#xA;6    6 2001 0.002785111 0.012322869  1&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where &lt;strong&gt;reg&lt;/strong&gt; is the region indicator, &lt;strong&gt;year&lt;/strong&gt; is the year of measurement, &lt;strong&gt;ur&lt;/strong&gt; is the unemployment rate, &lt;strong&gt;mur&lt;/strong&gt; is the average unemployment rate in the neighboring regions (given by &lt;strong&gt;cl&lt;/strong&gt;) excluding the current region (see @kwak suggestion). Below is the code that I used in &lt;code&gt;R&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;fm &amp;lt;- pgmm(log(ur)~lag(log(ur),1)+log(mur)|lag(log(ur),2:40),data=df)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have several question regarding this model: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;I guess I should choose &lt;code&gt;effect=&quot;individual&quot;&lt;/code&gt; to avoid having time dependent intercepts (?). But doing so the code is crashing (Error in names(coefficients) &amp;lt;- c(namesX, namest))!&lt;/li&gt;&#xA;&lt;li&gt;Which &lt;code&gt;model&lt;/code&gt; should I choose, &lt;code&gt;onestep&lt;/code&gt; or &lt;code&gt;twosteps&lt;/code&gt;?&lt;/li&gt;&#xA;&lt;li&gt;How do I decide on the numbers of instruments (40 is just a guess)?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Assuming that I have fitted this model successfully, how do I simulate from it?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regards  &lt;/p&gt;&#xA;" OwnerUserId="1443" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-08T14:48:53.647" LastActivityDate="2010-12-08T14:48:53.647" Title="Forecasting unemployment rate with plm" Tags="&lt;r&gt;&lt;panel-data&gt;&lt;gmm&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="1" />
  <row Id="3846" PostTypeId="2" ParentId="3835" CreationDate="2010-10-21T12:33:03.850" Score="1" ViewCount="" Body="&lt;p&gt;There is a constraint imposed a few lines above the highlighted text which states:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\sum_i{v_i (d_i-k v_i)} = 0$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does the that help?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In response to your second comment:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider: $\sum_i{(d_i - l v_i)^2}$. This can be re-written as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\sum_i{((d_i - k v_i) + (k v_i - l v_i))^2}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Expanding the square, we have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\sum_i{(d_i - k v_i)^2 + \sum_i(k v_i - l v_i)^2 + \sum_i{2 (d_i - k v_i) (k v_i - l v_i)}}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Simplifying the above, we have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\sum_i{(d_i - k v_i)^2 + (k-l)^2 \sum_i{v_i^2} + 2 (k-l) \sum_i{v_i (d_i - k v_i)}}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, if we choose $k$ such that $\sum_i{v_i (d_i-k v_i)} = 0$ then it follows that:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\sum_i{(d_i - l v_i)^2} = \sum_i{(d_i - k v_i)^2} + (k-l)^2 \sum_i{v_i^2}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But, then it follows that:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\sum_i{(d_i - l v_i)^2} &gt; \sum_i{(d_i - k v_i)^2}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;as long as $k \ne l$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, what we have shown is the following: If $k$ satisfies the constraint we imposed then it must be the case that the corresponding SSE is less than the SSE for any other $l$ that we can choose. Thus, the value of $k$ that satisfies the constraint &lt;em&gt;is&lt;/em&gt; our least squares estimate.&lt;/p&gt;&#xA;" LastActivityDate="2010-10-22T01:18:58.907" CommentCount="3" />
  <row Id="3847" PostTypeId="2" ParentId="3842" CreationDate="2010-10-21T12:58:47.983" Score="5" ViewCount="" Body="&lt;p&gt;Here  ggplot version:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(ggplot2)&#xA;df = melt(data.frame(A=c(2, 10), B=c(3, 20), &#xA;          experiment=c(&quot;X&quot;, &quot;X &amp;amp; Y&quot;)),&#xA;          variable_name=&quot;metric&quot;)&#xA;&#xA;ggplot(df, aes(experiment, value, fill=metric)) + &#xA;       geom_bar(position=&quot;dodge&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/F0F7m.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="1443" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-10-22T10:16:06.170" LastActivityDate="2010-10-22T10:16:06.170" CommentCount="3" />
  <row Id="3848" PostTypeId="2" ParentId="3840" CreationDate="2010-10-21T13:21:24.800" Score="1" ViewCount="" Body="&lt;p&gt;There are many different distance measures.  For starters, there's always the correlation.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can look at the &lt;a href=&quot;http://en.wikipedia.org/wiki/Mean_squared_error&quot; rel=&quot;nofollow&quot;&gt;mean square error&lt;/a&gt;.  In R, you can see the algorithm for time series in Rob Hyndman's &lt;a href=&quot;http://cran.us.r-project.org/web/packages/ftsa/&quot; rel=&quot;nofollow&quot;&gt;ftsa&lt;/a&gt; package (see the &lt;code&gt;error&lt;/code&gt; function).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.115.6594&quot; rel=&quot;nofollow&quot;&gt;Liao (2005)&lt;/a&gt; for a nice short survey of time series similarity measures, including euclidean distance, root mean square distance, mikowski distance, pearson's correlation, dynamic time warping distance, kullback-liebler disance, symmetric Chernoff information divergence, and cross-correlation.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;T. Warren Liao, &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.115.6594&quot; rel=&quot;nofollow&quot;&gt;&quot;Clustering of time series data—a survey&quot;&lt;/a&gt; Pattern Recognition, 2005&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="5" LastEditorUserId="5" LastEditorDisplayName="" LastEditDate="2010-10-21T14:06:00.943" LastActivityDate="2010-10-21T14:06:00.943" CommentCount="1" />
  <row Id="3849" PostTypeId="1" AcceptedAnswerId="5429" CreationDate="2010-10-21T13:38:32.160" Score="2" ViewCount="221" Body="&lt;p&gt;Are there any tools that can format and write to external file output of principal component analysis in Stata? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm thinking about something that will work in similar vein to [excellent] family of &lt;a href=&quot;http://repec.org/bocode/e/estout/index.html&quot; rel=&quot;nofollow&quot;&gt;estout&lt;/a&gt; commands.&lt;/p&gt;&#xA;" OwnerUserId="22" LastActivityDate="2010-12-13T12:09:19.943" Title="How can I format and export PCA output in Stata?" Tags="&lt;pca&gt;&lt;stata&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="3850" PostTypeId="2" ParentId="1053" CreationDate="2010-10-21T14:42:28.390" Score="3" ViewCount="" Body="&lt;p&gt;David Collett. &lt;em&gt;Modelling Survival Data in Medical Research&lt;/em&gt;, Second Edition. Chapman &amp;amp; Hall/CRC. 2003. &lt;a href=&quot;http://en.wikipedia.org/wiki/Special%3aBookSources/9781584883258&quot; rel=&quot;nofollow&quot;&gt;ISBN 978-1584883258&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Software section focuses on SAS not R though.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-10-21T14:42:28.390" CommunityOwnedDate="2010-10-21T14:42:28.390" />
  <row Id="3853" PostTypeId="1" AcceptedAnswerId="3854" CreationDate="2010-10-21T15:01:50.123" Score="2" ViewCount="206" Body="&lt;p&gt;I have the following &lt;em&gt;R&lt;/em&gt; code for creating &lt;em&gt;barplot&lt;/em&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# ... here I read cvs file&#xA;&#xA;mx &amp;lt;- rbind(results$&quot;AVG.P10.&quot;)   &#xA;colnames(mx) &amp;lt;- results$&quot;RUN&quot;    &#xA;rownames(mx) &amp;lt;- &quot;AVG P&quot;    &#xA;postscript(file=&quot;avg_p_result.ps&quot;)    &#xA;barplot(mx, beside=T,  col=c(&quot;grey&quot;), names.arg= results$&quot;RUN&quot;, &#xA;        cex.axis = 1.5, cex.lab=1.5)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have tried &lt;code&gt;cex.lab=1.5&lt;/code&gt; but it does not work at all. &lt;/p&gt;&#xA;" OwnerUserId="1389" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-21T16:20:10.117" LastActivityDate="2010-10-21T16:21:07.947" Title="How to increase size of label fonts in barplot" Tags="&lt;r&gt;&lt;boxplot&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="3854" PostTypeId="2" ParentId="3853" CreationDate="2010-10-21T15:15:49.950" Score="4" ViewCount="" Body="&lt;p&gt;According to &lt;code&gt;?barplot&lt;/code&gt;, you need to use &lt;code&gt;cex.names=1.5&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;barplot(mx, beside=TRUE, col=c(&quot;grey&quot;), names.arg=results$&quot;RUN&quot;, &#xA;        cex.axis=1.5, cex.names=1.5)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1657" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-21T16:21:07.947" LastActivityDate="2010-10-21T16:21:07.947" />
  <row Id="3855" PostTypeId="1" AcceptedAnswerId="3861" CreationDate="2010-10-21T15:48:52.837" Score="4" ViewCount="116" Body="&lt;p&gt;Imagine I have a test for a disease that returns either a positive or a negative result.  I administer the test to one group twice in a single session; in another group, I administer it twice with two weeks separating the tests.  The hypothesis is that the tests taken at the same time will be more consistent with each other than the tests taken two weeks apart.  How can I test this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bonus question: What if the two groups consisted of the same people?&lt;/p&gt;&#xA;" OwnerUserId="71" LastActivityDate="2010-10-21T17:14:20.710" Title="Comparing test-retest reliabilities" Tags="&lt;reliability&gt;" AnswerCount="2" />
  <row Id="3856" PostTypeId="1" AcceptedAnswerId="3865" CreationDate="2010-10-21T16:20:09.727" Score="4" ViewCount="175" Body="&lt;p&gt;Are there standard ways of analyzing and generating &quot;clumpy&quot; distributions?&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;analyze: how clumpy is a given point cloud (in 1d, 2d, nd),&#xA;what are its clumpy coefficients?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;generate or synthesize a pseudo-random cloud with coefficients C&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;(These are the basics for any family of distributions, e.g. normal.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are many kinds of clumpiness in nature&#xA;(traffic jams, clumpiness of climate changes),&#xA;so it's a wide term with I imagine various attempts at description&#xA;and various links to &quot;classical&quot; statistics.&lt;br&gt;&#xA;I'm looking for an overview; pictures would be nice.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Added Friday 22 Oct: I had hoped to find methods for &lt;em&gt;both&lt;/em&gt; analysis and synthesis,&lt;br&gt;&#xA;abstract &amp;lt;-&gt; real both ways; surely a wide-ranging abstraction must do both.&#xA;Still looking ...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Experts please add tags).&lt;/p&gt;&#xA;" OwnerUserId="557" LastEditorUserId="557" LastEditorDisplayName="" LastEditDate="2010-10-22T13:32:28.723" LastActivityDate="2010-11-28T08:03:09.037" Title="Analyze and generate &quot;clumpy&quot; distributions?" Tags="&lt;clustering&gt;&lt;spatial&gt;&lt;autocorrelation&gt;" AnswerCount="4" CommentCount="1" />
  <row Id="3857" PostTypeId="1" AcceptedAnswerId="3863" CreationDate="2010-10-21T16:26:22.880" Score="17" ViewCount="295" Body="&lt;p&gt;&lt;a href=&quot;http://www.hsl.gov.uk/&quot; rel=&quot;nofollow&quot;&gt;My workplace&lt;/a&gt; has employees from a very wide range of disciplines, so we generate data in lots of different forms.  Consequently, each team has developed its own system for storing data.  Some use Access or SQL databases; some teams (to my horror) are reliant almost entirely on Excel spreadsheets.  Often, data formats change from project to project.  In some cases calling it a 'system' is too kind.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problems this entails are that I have to write new code to clean the data for every project, which is expensive; people manually editing spreadsheets make reproducibility and auditing of data near impossible; and even worse,  there is a chance that data gets lost or made incorrect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've been given the opportunity to discuss these problems with a board member of the company and I need to work out what to tell him.  I think I've already persuaded him that we have a problem and that getting this right will enable better science and saving money.  The question is: what should we be aiming for, and how do we get there?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More specifically:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How should we store data, in way that lets us track it from creation to publishing in a paper?  (Databases stored on a central server?)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do you go about standardising database formats?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any good resources for educating people on how to care for data?  (As a general rule, occupational hygienists and explosives engineers aren't data nerds; so non-technical content preferred.)&lt;/p&gt;&#xA;" OwnerUserId="478" LastActivityDate="2010-10-25T15:54:27.227" CommunityOwnedDate="2010-10-22T14:03:02.960" Title="How do I get people to take better care of data?" Tags="&lt;dataset&gt;" AnswerCount="7" CommentCount="1" FavoriteCount="6" />
  <row Id="3858" PostTypeId="2" ParentId="3855" CreationDate="2010-10-21T16:28:13.923" Score="1" ViewCount="" Body="&lt;p&gt;Perhaps, computing the tetrachoric correlation would be useful. See this url: &lt;a href=&quot;http://www.john-uebersax.com/stat/tetra.htm&quot; rel=&quot;nofollow&quot;&gt;Introduction to the Tetrachoric and Polychoric Correlation Coefficients&lt;/a&gt;&lt;/p&gt;&#xA;" LastActivityDate="2010-10-21T16:28:13.923" />
  <row Id="3859" PostTypeId="2" ParentId="3856" CreationDate="2010-10-21T16:36:55.843" Score="3" ViewCount="" Body="&lt;p&gt;I think suitable 'clumpy coefficients' are measures of spatial autocorrelation such as &lt;a href=&quot;http://en.wikipedia.org/wiki/Moran%27s_I&quot; rel=&quot;nofollow&quot;&gt;Moran's I&lt;/a&gt; and &lt;a href=&quot;http://en.wikipedia.org/wiki/Geary%27s_C&quot; rel=&quot;nofollow&quot;&gt;Geary's C&lt;/a&gt;. Spatial statistics is not my area and I don't know about simulation though.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-10-21T16:36:55.843" />
  <row Id="3860" PostTypeId="2" ParentId="3856" CreationDate="2010-10-21T16:36:59.993" Score="3" ViewCount="" Body="&lt;p&gt;You could calculate an &lt;a href=&quot;http://www.passagesoftware.net/webhelp/Dispersion_Indices.htm&quot; rel=&quot;nofollow&quot;&gt;index of dispersion&lt;/a&gt; measure over your space to gauge clumpiness.  One starting point for more information would be the ecology packages and literature to see how they simulate such things.&lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-10-21T16:36:59.993" />
  <row Id="3861" PostTypeId="2" ParentId="3855" CreationDate="2010-10-21T17:14:20.710" Score="4" ViewCount="" Body="&lt;p&gt;Both situations are specific cases of test-retest, except that the recall period is null in the first case you described. I would also expect a larger agreement in the former case, but that may be confounded with a learning or memory effect. A chance-corrected measure of agreement, like &lt;a href=&quot;http://en.wikipedia.org/wiki/Cohen%27s_kappa&quot; rel=&quot;nofollow&quot;&gt;Cohen's kappa&lt;/a&gt;, can be used with binary variables, and bootstraped confidence intervals might be compared in the two situations (this is better than using $\kappa$ sampling variance directly). This should give an indication of the reliability of your measures, or in this case diagnostic agreement, at the two occasions. A &lt;a href=&quot;http://j.mp/cI0TZr&quot; rel=&quot;nofollow&quot;&gt;McNemar test&lt;/a&gt; which tests for marginal homogeneity in matched pairs can also be used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An approach based on the &lt;a href=&quot;http://www.john-uebersax.com/stat/icc.htm&quot; rel=&quot;nofollow&quot;&gt;intraclass correlation&lt;/a&gt; is still valid and, provided your prevalence is not extreme, should be closed to&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;a simple Pearson correlation (which, for binary data, is also called a &lt;a href=&quot;http://en.wikipedia.org/wiki/Phi_coefficient&quot; rel=&quot;nofollow&quot;&gt;phi coefficient&lt;/a&gt;) or the tetrachoric version suggested by @Skrikant,&lt;/li&gt;&#xA;&lt;li&gt;the aforementioned kappa (for a large sample, and assuming that the marginal distributions for case at the two occasions are the same, $\kappa\approx\text{ICC}$ from a one-way ANOVA).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;About your bonus question, you generally need 3 time points to separate the lack of (temporal) stability -- which can occur if the latent class or trait your are measuring evolve over time -- from the lack of reliability (see for an illustration the model proposed by &lt;a href=&quot;http://www.jstor.org/pss/2093858&quot; rel=&quot;nofollow&quot;&gt;Wiley and Wiley&lt;/a&gt;, 1970, &lt;em&gt;American Sociological Review&lt;/em&gt; 35).&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-10-21T17:14:20.710" />
  <row Id="3862" PostTypeId="2" ParentId="3857" CreationDate="2010-10-21T17:34:29.003" Score="5" ViewCount="" Body="&lt;p&gt;One free online resource is the set of &lt;a href=&quot;http://www.ssc.rdg.ac.uk/publications/guides.html&quot; rel=&quot;nofollow&quot;&gt;Statistical Good Practice Guidelines&lt;/a&gt; from the &lt;a href=&quot;http://www.ssc.rdg.ac.uk/&quot; rel=&quot;nofollow&quot;&gt;Statistical Services Centre at the University of Reading&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In particular:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.ssc.rdg.ac.uk/publications/guides/topdmg.html&quot; rel=&quot;nofollow&quot;&gt;Data Management Guidelines for Experimental Projects&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.ssc.rdg.ac.uk/publications/guides/topdbp.html&quot; rel=&quot;nofollow&quot;&gt;The Role of a Database Package in Managing Research Data&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="449" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-10-21T17:39:34.460" LastActivityDate="2010-10-21T17:39:34.460" CommunityOwnedDate="2010-10-22T14:03:02.960" CommentCount="1" />
  <row Id="3863" PostTypeId="2" ParentId="3857" CreationDate="2010-10-21T17:45:14.913" Score="6" ViewCount="" Body="&lt;p&gt;It's worth considering ideas from the software world.  In particular you might think of setting up: a &lt;a href=&quot;http://en.wikipedia.org/wiki/Revision_control&quot; rel=&quot;nofollow&quot;&gt;version control repository&lt;/a&gt; and a central database server.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Version control probably helps you out with otherwise free floating files, such as Excel and text files, etc.  But this could also include files associated with data, such as R, SAS, etc.  The idea is that there's a system which tracks changes to your files allowing you to know what happened when and rollback to a point in the past if needed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where you already have SQL databases, the best thing you can do is set up a central server and hire a capable &lt;a href=&quot;http://en.wikipedia.org/wiki/Database_administrator&quot; rel=&quot;nofollow&quot;&gt;DBA&lt;/a&gt;.    The DBA is the person tasked with ensuring and mantaining the integrity of the data.  Part of the job description involves things like backups and tuning.  But another part is more relevant here -- controlling how data enters the system, ensuring that constraints are met, access policies are in place to prevent harm to the data, setting up views to expose custom or simplified data formats, etc.  In short, implementing a methodology around the data process.  Even if you don't hire an actual DBA (the good ones are very hard to recruit), having a central server still enables you to start thinking about instituting some kind of methodology around data.  &lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-10-21T17:45:14.913" CommunityOwnedDate="2010-10-22T14:03:02.960" CommentCount="2" />
  <row Id="3864" PostTypeId="2" ParentId="3857" CreationDate="2010-10-21T17:55:24.157" Score="4" ViewCount="" Body="&lt;p&gt;I think first of all you have to ask yourself: why do people use Excel to do tasks Excel was not made for?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) They already know how to use it&#xA;2) It works. Maybe in a clumsy way but &lt;strong&gt;it works&lt;/strong&gt; and that's what they want&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I copy a series of numbers in, press a button and I have a plot. As easy as that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, make them understand what advantages they can have by using centralized datasets, proper databases (note that Access is NOT one of those) and so on. But remember the two points above: you need to set up a system that works and it's easy to use.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've seen too many times badly made systems that made me want to go back not to Excel but to pen and paper!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just as an example, we have an horrible ordering system where I work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We used to have to fill in an order form which was an Excel spreadsheet where you would input the name of the product, the quantity, the cost etc. It would add everything up, add TVA etc etc, you printed it, gave it to the secretary who would make the order and that was it. Unefficient, but it worked.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now we have an online ordering system, with a centralized DB and everything. It's an horror.&#xA;It should not take me 10 minutes to fill in a damn form because of the unituitive keyboard shortcuts and the various oddities of the software. And note that I am quite informatics-savvy, so imagine what happens to people who don't like computers...&lt;/p&gt;&#xA;" OwnerUserId="582" LastActivityDate="2010-10-21T17:55:24.157" CommunityOwnedDate="2010-10-22T14:03:02.960" CommentCount="1" />
  <row Id="3865" PostTypeId="2" ParentId="3856" CreationDate="2010-10-21T18:58:11.697" Score="2" ViewCount="" Body="&lt;p&gt;If assessing spatial auto-correlation is what your interested in, here is a paper that simulates data and evaluates different auto-regressive models in R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://dx.doi.org/10.1111/j.1466-8238.2007.00334.x&quot; rel=&quot;nofollow&quot;&gt;Spatial autocorrelation and the selection of simultaneous autoregressive models&lt;/a&gt;&#xA;by: W. D. Kissling, G. Carl&#xA;Global Ecology and Biogeography, Vol. 17, No. 1. (January 2008), pp. 59-71. (PDF available &lt;a href=&quot;http://www.iaps2010.ufz.de/data/kissling_carl_geb3346435.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unfortunately they do not have the code in R they used to generate the simulated data, but they do have the code available of how they fit each of the models in the &lt;a href=&quot;http://onlinelibrary.wiley.com/doi/10.1111/j.1466-8238.2007.00334.x/suppinfo&quot; rel=&quot;nofollow&quot;&gt;supplementary material&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It would definately help though if you could be a little more clear about the nature of your data. Many of the techniques intended for spatial analysis will probably not be implemented in higher dimensional data, and I am sure there are other techniques that are more suitable. Some type of K-nearest neighbors technique might be useful, and make sure to change your search term from clumpy to cluster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some other references you may find helpful. I would imagine the best resources for simulating data in such a manner would be with packages in the R program.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Websites I suggest you check out the &lt;a href=&quot;http://www.spatstat.org/&quot; rel=&quot;nofollow&quot;&gt;Spatstat&lt;/a&gt; R package page, and the R Cran Task View for &lt;a href=&quot;http://cran.r-project.org/web/views/Spatial.html&quot; rel=&quot;nofollow&quot;&gt;spatial data&lt;/a&gt;. I would also suggest you check out the &lt;a href=&quot;http://geodacenter.asu.edu/&quot; rel=&quot;nofollow&quot;&gt;GeoDa&lt;/a&gt; center page, and you never know the &lt;a href=&quot;http://groups.google.com/group/openspace-list&quot; rel=&quot;nofollow&quot;&gt;OpenSpace&lt;/a&gt; Google group may have some helpful info. I also came across this &lt;a href=&quot;http://www.mail-archive.com/r-sig-geo@stat.math.ethz.ch/info.html&quot; rel=&quot;nofollow&quot;&gt;R mailing list concerning geo data&lt;/a&gt;, but I have not combed the archive very much at this point (but I'm sure there is useful data in there).&lt;/p&gt;&#xA;" OwnerUserId="1036" LastEditorUserId="1036" LastEditorDisplayName="" LastEditDate="2010-10-22T19:08:26.677" LastActivityDate="2010-10-22T19:08:26.677" CommentCount="4" />
  <row Id="3866" PostTypeId="2" ParentId="3841" CreationDate="2010-10-21T19:22:27.857" Score="2" ViewCount="" Body="&lt;p&gt;Here is an idea that connects your binary dependent variable to a continuous, unobserved variable; a connection that may let you leverage the power of time series models for continuous variables. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Define:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$V_{w,t} = 1$ if violent incident happened in ward $w$ during time period $t$ and 0 otherwise&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P_{w,t}$ : Propensity for violence in ward $w$ at time $t$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P_{w,t}$ is assumed to be a continuous variable that in some sense represents 'pent-up' feelings of the inmates which boil over at some time and results in violence. Following this reasoning, we have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$V_{w,t} = \begin{cases} 1 &amp;amp; \mbox{if } P_{w,t} \ge \tau \\&#xA;                         0 &amp;amp; \mbox{otherwise}&#xA;           \end{cases}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\tau$ is an unobserved threshold which triggers a violent act.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can then use a time series model for $P_{w,t}$ and estimate the relevant parameters. For example, you could model $P_{w,t}$ as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P_{w,t} = \alpha_0 + \alpha_1 P_{w,t-1} + ... + \alpha_p P_{w,t-p}+ \beta n_{w,t} + \epsilon_t$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$n_{w,t}$ is the number of patients in ward $w$ at time $t$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could see if $\beta$ is significantly different from 0 to test your hypothesis that &quot;more patients lead to an increase in probability of violence&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The challenge of the above model specification is that you do not really observe $P_{w,t}$ and thus the above is not your usual time series model. I do not know anything about R so perhaps someone else will chip in if there is a package that would let you estimate models like the above.&lt;/p&gt;&#xA;" LastActivityDate="2010-10-21T19:22:27.857" />
  <row Id="3867" PostTypeId="2" ParentId="1053" CreationDate="2010-10-21T20:35:15.033" Score="2" ViewCount="" Body="&lt;p&gt;Take a look at the course page for &lt;a href=&quot;http://socserv.socsci.mcmaster.ca/jfox/Courses/soc761/index.html&quot; rel=&quot;nofollow&quot;&gt;Sociology 761: Statistical Applications in Social Research&lt;/a&gt;. Professor &lt;a href=&quot;http://socserv.socsci.mcmaster.ca/jfox/&quot; rel=&quot;nofollow&quot;&gt;John Fox&lt;/a&gt; at McMaster University has &lt;a href=&quot;http://socserv.socsci.mcmaster.ca/jfox/Courses/soc761/survival-analysis.pdf&quot; rel=&quot;nofollow&quot;&gt;course notes on survival analysis&lt;/a&gt; as well as an example &lt;a href=&quot;http://socserv.socsci.mcmaster.ca/jfox/Courses/soc761/survival-analysis.R&quot; rel=&quot;nofollow&quot;&gt;R script&lt;/a&gt; and &lt;a href=&quot;http://socserv.socsci.mcmaster.ca/jfox/Courses/soc761/Canada-mortality.txt&quot; rel=&quot;nofollow&quot;&gt;several&lt;/a&gt; &lt;a href=&quot;http://socserv.socsci.mcmaster.ca/jfox/Courses/soc761/Henning.txt&quot; rel=&quot;nofollow&quot;&gt;data&lt;/a&gt; &lt;a href=&quot;http://socserv.socsci.mcmaster.ca/jfox/Courses/soc761/Rossi.txt&quot; rel=&quot;nofollow&quot;&gt;files&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For another perspective, see  &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1566986737&quot; rel=&quot;nofollow&quot;&gt;Models for Quantifying Risk, 3/e&lt;/a&gt;, the standard textbook for &lt;a href=&quot;http://www.soa.org/education/exam-req/edu-exam-m-detail.aspx&quot; rel=&quot;nofollow&quot;&gt;actuarial exam 3/MLC&lt;/a&gt;. The bulk of the book, chapters 3-10, covers survival-contingent payment models.&lt;/p&gt;&#xA;" LastActivityDate="2010-10-21T20:35:15.033" CommunityOwnedDate="2010-10-21T20:35:15.033" />
  <row Id="3868" PostTypeId="1" CreationDate="2010-10-21T23:47:03.803" Score="3" ViewCount="174" Body="&lt;p&gt;The nonlinear  model I am fitting  gamma distribution with inverse or log is not converging. There is one observation having zero value in the response variable. Does this zero affects to model the data? Any answer warmly appreciated! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;thank you&lt;br&gt;&#xA;  Maereg  &lt;/p&gt;&#xA;" OwnerUserId="1662" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-10-24T12:17:52.580" LastActivityDate="2010-10-24T12:17:52.580" Title="Undefined link function in gamma distribution" Tags="&lt;glm&gt;&lt;fitting&gt;&lt;link-function&gt;" CommentCount="2" />
  <row Id="3869" PostTypeId="1" AcceptedAnswerId="3871" CreationDate="2010-10-22T01:02:20.173" Score="1" ViewCount="143" Body="&lt;p&gt;I know that if events $A$ and $B$ are independent, then $P(A\cap B) = P(A).P(B)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lets say I roll a dice, and define the following events:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$x$ = Number is a multiple of $3 = \lbrace 3, 6\rbrace$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$y$ = Number is more than $3 = \lbrace 4, 5, 6\rbrace$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$z$ = Number is even  =  $\lbrace 2, 4, 6 \rbrace$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(x) * P(y)$: 0.166666666667&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(x \cap y)$   : 0.166666666667&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(x) * P(z)$ : 0.166666666667&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(x \cap z)$    : 0.166666666667&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(y) * P(z)$ : 0.25&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(y \cap z)$    : 0.333333333333&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is, why are $xy$ and $xz$ independent events but $yz$ is not? I had expected all combinations of events to be independent as neither events affect any other.&lt;/p&gt;&#xA;" OwnerUserId="1636" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-10-22T01:19:20.960" LastActivityDate="2010-10-22T01:33:49.437" Title="What are independent events?" Tags="&lt;probability&gt;&lt;independence&gt;" AnswerCount="2" />
  <row Id="3870" PostTypeId="2" ParentId="3869" CreationDate="2010-10-22T01:16:57.240" Score="2" ViewCount="" Body="&lt;p&gt;I find it easiest to think of independence in terms of conditional probabilities: $X$ and $Y$ are independent if $P(Y|X)=P(Y)$ and $P(X|Y)=P(X)$. That is knowing one does not change the probability of the other. In this example,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{aligned}&#xA;P(Y|X) &amp;amp;= P(Y \cap X) / P(X) = 1/2 = P(Y)\\&#xA;P(Y|Z) &amp;amp;= P(Y \cap Z) / P(Z) = 2/3 \ne P(Y)\\&#xA;P(Z|X) &amp;amp;= P(Z \cap X) / P(X) = 1/2 = P(Z)&#xA;\end{aligned}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So knowing $X$ does not affect the probability of either $Y$ or $Z$. But knowing $Z$ does affect the probability of $Y$.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-10-22T01:16:57.240" />
  <row Id="3871" PostTypeId="2" ParentId="3869" CreationDate="2010-10-22T01:33:49.437" Score="3" ViewCount="" Body="&lt;p&gt;To add to the Rob's &lt;a href=&quot;http://stats.stackexchange.com/questions/3869/what-are-independent-events/3870#3870&quot;&gt;answer&lt;/a&gt; the joint probability of two events in the discrete case can be decomposed as follows (See the wiki's definition for the &lt;a href=&quot;http://en.wikipedia.org/wiki/Joint_probability_distribution#Discrete_case&quot; rel=&quot;nofollow&quot;&gt;discrete case&lt;/a&gt;) :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(A \cap B) = P(A|B) P(B)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, if the fact that we observe the event $B$ is &lt;strong&gt;not&lt;/strong&gt; informative about the probability of our observing the event $A$ then it must be the case that:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(A|B) = P(A)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, if the two variables are independent (i.e., the occurrence of one event does not tell us anything about the other event) then it must be the case that:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(A \cap B) = P(A) P(B)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The crucial condition for independence of two events is: $P(A|B) = P(A)$. This condition is essentially stating that the &quot;Probability of our observing event A over the unrestricted sample space = Probability of our observing the event A if the sample space is restricted to the set of possible outcomes compatible with the event B&quot;. Thus, the knowledge that the event B occurred has no information about the probability of observing the event A.&lt;/p&gt;&#xA;" LastActivityDate="2010-10-22T01:33:49.437" CommentCount="6" />
  <row Id="3872" PostTypeId="1" CreationDate="2010-10-22T04:37:04.527" Score="0" ViewCount="343" Body="&lt;p&gt;What statistical test can I use to compare two ratios from two independent samples. The ratios are after to before results. I need to compare the after/before ratios for two independent models and show whether they are have significant difference or not. Please help!&lt;/p&gt;&#xA;" LastActivityDate="2010-10-22T05:16:26.863" Title="Statistical test to compare two ratios from two independent models" Tags="&lt;statistical-significance&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="3873" PostTypeId="2" ParentId="3650" CreationDate="2010-10-22T04:55:00.487" Score="1" ViewCount="" Body="&lt;p&gt;So here is my motivation for the questions, although I know this is not necessary I like it when people follow up on their questions so I will do the same. I would like to thank both Srikant and whuber for their helpful answers. (I ask no-one upvote this as it is not an answer to the question and both whuber's and Srikant's deserve to be above this, and you should upvote their excellent answers.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The other day for a class field trip I sat in on the proceedings of an appeals court. Several of the criminal appeals brought before that day concerned issues surrounding &lt;a href=&quot;http://en.wikipedia.org/wiki/Batson_v._Kentucky&quot; rel=&quot;nofollow&quot;&gt;Batson challenges&lt;/a&gt;. A Batson challenge concerns the use of racial discrimination when an attorney uses what are called &lt;a href=&quot;http://en.wikipedia.org/wiki/Peremptory_challenge&quot; rel=&quot;nofollow&quot;&gt;peremptory challenges&lt;/a&gt; during the &lt;a href=&quot;http://en.wikipedia.org/wiki/Voir_dire&quot; rel=&quot;nofollow&quot;&gt;voir dire&lt;/a&gt; process of jury selection. (I'm in the US so this is entirely in the context of USA criminal law).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Two separate questions arose in the deliberations that were statistical in nature. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first question was the chance that two out of two asian jurors (the purple balls) seated currently in the venire panel (the urn consisting of the total number of balls) would be selected by chance (the total number of peremptory challenges used equals the number of balls drawn). The attorney in this case stated the probability that both Asian jurors would be selected was $1/28$. I don't have the materials the attorneys presented to the court of appeals, so I do not know how the attorney calculated this probability. But this is essentially my question #1, so given the formula for the &lt;a href=&quot;http://en.wikipedia.org/wiki/Hypergeometric_distribution&quot; rel=&quot;nofollow&quot;&gt;Hypergeometric distribution&lt;/a&gt; I calculated the expected probability given,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$n = 20$ The number of jurors seated in the venire panel&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$p = 2$ The number of Asian jurors seated in the venire panel, both of whom were picked&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$d = 6$ The number of peremptory challenges of use to an attorney&lt;/p&gt;&#xA;&#xA;&lt;p&gt;which then leads to an expected value of&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{\binom{p}{p} \binom{n-p}{d-p}}{\binom{n}{d}}=\frac{\binom{2}{2} \binom{20-2}{6-2}}{\binom{20}{6}}=\frac{3}{38}.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;*note these are my best guesses of the values based on what I know of the case, if I had the court record I could know for sure. Values &lt;a href=&quot;http://www.wolframalpha.com/input/?i=binomial%5b18,4%5d/binomial%5b20,6%5d&quot; rel=&quot;nofollow&quot;&gt;calculated&lt;/a&gt; using Wolfram Alpha.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Although the court is unlikely to establish a bright line rule which states the probability threshold that establishes a &lt;em&gt;prima facie&lt;/em&gt; case of racial discrimination, at least one judge thought the use of such statistics is applicable to establishing the validity of a Batson challenge.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this case there wasn't much doubt in the eyes of the court that Asian jurors had a stereotype among attorneys that they were pro-prosecution. But in a subsequent case a defense attorney used a Batson challenge to claim a prosecutor was being racially biased by eliminating 4 out of 6 Black women. The judges in this appeal were somewhat skeptical that Black women were a group that had a cognizable stereotype attached to them, but this again is a question amenable to statistical knowledge. Hence my question #2, given 100 observations could I determine if black women were eliminated using peremptory challenges in a non-random manner. In reality the attorneys are not eliminating prospective jurors based on only the race and sex of the juror, but that would not preclude someone from determining if the pattern of peremptory challenges at least appears or does not appear random (although non-randomness does not necessarily indicate racial discrimination).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Again I'd like to say thank you both to whuber and Srikant for their answers. &lt;/p&gt;&#xA;" OwnerUserId="1036" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-10-22T16:38:05.127" LastActivityDate="2010-10-22T16:38:05.127" CommunityOwnedDate="2010-10-22T12:52:59.893" CommentCount="1" />
  <row Id="3874" PostTypeId="1" AcceptedAnswerId="3876" CreationDate="2010-10-22T05:03:26.650" Score="5" ViewCount="1048" Body="&lt;p&gt;I have data from patients treated with 2 different kinds of treatments during surgery.&#xA;I need to analyze its effect on heart rate. &#xA;The heart rate measurement is taken every 15 minutes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given that the surgery length can be different for each patient, each patient can have between 7 and 10 heart rate measurements. &#xA;So an unbalanced design should be used. &#xA;I'm doing my analysis using R. And have been using the ez package to do repeated measure mixed effect ANOVA. But I do not know how to analyse unbalanced data. Can anyone help?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suggestions on how to analyze the data are also welcomed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Update:&lt;br&gt;&#xA;As suggested, I fitted the data using the &lt;code&gt;lmer&lt;/code&gt; function and found that the best model is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;heart.rate~ time + treatment + (1|id) + (0+time|id) + (0+treatment|time)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;with the following result:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Random effects:&#xA; Groups   Name        Variance   Std.Dev. Corr   &#xA; id       time        0.00037139 0.019271        &#xA; id       (Intercept) 9.77814104 3.127002        &#xA; time     treat0      0.09981062 0.315928        &#xA;          treat1      1.82667634 1.351546 -0.504 &#xA; Residual             2.70163305 1.643665        &#xA;Number of obs: 378, groups: subj, 60; time, 9&#xA;&#xA;Fixed effects:&#xA;             Estimate Std. Error t value&#xA;(Intercept) 72.786396   0.649285  112.10&#xA;time         0.040714   0.005378    7.57&#xA;treat1       2.209312   1.040471    2.12&#xA;&#xA;Correlation of Fixed Effects:&#xA;       (Intr) time  &#xA;time   -0.302       &#xA;treat1 -0.575 -0.121&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now I'm lost at interpreting the result. &#xA;Am I right in concluding that the two treatments differed in affecting heart rate? What does the correlation of -504 between treat0 and treat1 means?&lt;/p&gt;&#xA;" OwnerUserId="1663" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-08T01:04:33.640" LastActivityDate="2011-04-08T01:04:33.640" Title="Unbalanced mixed effect ANOVA for repeated measures" Tags="&lt;r&gt;&lt;mixed-model&gt;&lt;repeated-measures&gt;&lt;lmer&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="8" />
  <row Id="3875" PostTypeId="2" ParentId="3872" CreationDate="2010-10-22T05:16:26.863" Score="1" ViewCount="" Body="&lt;p&gt;Any test for independence of a 2x2 contingency table will do! A chi-square or t-test are the textbook simple solutions. The &quot;best&quot; test in this situation is called Barnard's test for superiority -- the StatXact software will happily calculate this for you.&lt;/p&gt;&#xA;" OwnerUserId="1122" LastActivityDate="2010-10-22T05:16:26.863" CommentCount="1" />
  <row Id="3876" PostTypeId="2" ParentId="3874" CreationDate="2010-10-22T05:36:47.643" Score="7" ViewCount="" Body="&lt;p&gt;The lme/lmer functions from the nlme/lme4 packages are able to deal with unbalanced designs. You should make sure that time is a numeric variable. You would also probably want to test for different types of curves as well. The code will look something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(lme4)&#xA;#plot data with a plot per person including a regression line for each&#xA;xyplot(heart.rate ~ time|id, groups=treatment, type= c(&quot;p&quot;, &quot;r&quot;), data=heart)&#xA;&#xA;#Mixed effects modelling&#xA;#variation in intercept by participant&#xA;lmera.1 &amp;lt;- lmer(heart.rate ~ treatment * time + (1|id), data=heart)&#xA;#variation in intercept and slope without correlation between the two&#xA;lmera.2 &amp;lt;- lmer(heart.rate ~ treatment * time + (1|id) + (0+time|id), data=heart)&#xA;#As lmera.1 but with correlation between slope and intercept&#xA;lmera.3 &amp;lt;- lmer(heart.rate ~ treatment * time + (1+time|id), data=heart)&#xA;&#xA;#Determine which random effects structure fits the data best&#xA;anova(lmera.1, lmera.2, lmera.3)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To get quadratic models use the formula &quot;heart.rate ~ treatment * time * I(time^2) + (random effects)&quot;.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Update:&lt;/em&gt;&lt;br&gt;&#xA;In this case where treatment is a between-subjects factor, I would stick with the model specifications above. I don't think the term (0+treatment|time) is one that you want included in the model, to me it doesn't make sense in this instance to treat time as a random-effects grouping variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But to answer your question of &quot;what does the correlation &lt;em&gt;-0.504&lt;/em&gt; mean between &lt;em&gt;treat0&lt;/em&gt; and &lt;em&gt;treat1&lt;/em&gt;&quot; this is the correlation coefficient between the two treatments where each time grouping is one pair of values. This makes more sense if id is the grouping factor and treatment is a within-subjects variable. Then you have an estimate of the correlation between the intercepts of the two conditions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Before making any conclusions about the model, refit it with lmera.2 and include REML=F. Then load the &quot;languageR&quot; package and run:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plmera.2&amp;lt;-pvals.fnc(lmera.2)&#xA;plmera.2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then you can get p-values, but by the looks of it, there is probably a significant effect of time and a significant effect of treatment.&lt;/p&gt;&#xA;" OwnerUserId="966" LastEditorUserId="966" LastEditorDisplayName="" LastEditDate="2010-10-28T15:44:12.870" LastActivityDate="2010-10-28T15:44:12.870" CommentCount="7" />
  <row Id="3877" PostTypeId="2" ParentId="3826" CreationDate="2010-10-22T07:24:42.113" Score="2" ViewCount="" Body="&lt;p&gt;Given the current issues with the user-written &lt;code&gt;sizefx&lt;/code&gt; command for Stata that chl and I have uncovered, here's an alternative way of doing things in Stata using the user-written &lt;code&gt;&lt;a href=&quot;http://ideas.repec.org/c/boc/bocode/s456798.html&quot; rel=&quot;nofollow&quot;&gt;metan&lt;/a&gt;&lt;/code&gt; command. This is designed for meta-analysis of study-level summary data so you need to enter the means, sds and ns (which i've taken from chl's answer) in one row, then run a 'meta-analysis' on this single study (so the pooled SMD is the same as the SMD for the single study, and heterogeneity measures are undefined):. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&#xA;input mean0 sd0 n0 mean1 sd1 n1&#xA;&#xA;         mean0        sd0         n0      mean1        sd1         n1&#xA;  1. 23.66154  5.584522  130  22.30508  4.511496        59&#xA;  2. end&#xA;&#xA;. list&#xA;&#xA;     +------------------------------------------------------+&#xA;     |    mean0        sd0    n0      mean1        sd1   n1 |&#xA;     |------------------------------------------------------|&#xA;  1. | 23.66154   5.584522   130   22.30508   4.511496   59 |&#xA;     +------------------------------------------------------+&#xA;&#xA;. metan n1 mean1 sd1 n0 mean0 sd0, cohen&#xA;&#xA;           Study     |     SMD   [95% Conf. Interval]     % Weight&#xA;---------------------+---------------------------------------------------&#xA;1                    | -0.257      -0.566     0.052        100.00&#xA;---------------------+---------------------------------------------------&#xA;I-V pooled SMD       | -0.257      -0.566     0.052        100.00&#xA;---------------------+---------------------------------------------------&#xA;&#xA;  Heterogeneity chi-squared =   0.00 (d.f. = 0) p =    .&#xA;  I-squared (variation in SMD attributable to heterogeneity) =     .%&#xA;&#xA;  Test of SMD=0 : z=   1.63 p = 0.103&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The resulting point estimate and CI for Cohen's d agree with those that chl calculated by hand and using &lt;code&gt;ci.smd()&lt;/code&gt; from R's MBESS library.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm less clear if this is entirely answering the original question, which asked to calculate Cohen's d &lt;em&gt;after &lt;code&gt;logit&lt;/code&gt;&lt;/em&gt;. But Cohen's d is one way of calculating the standardized mean difference, which I thought applied only to a continuous outcome measured in two groups. Perhaps the questioner could clarify.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-10-22T07:24:42.113" CommentCount="7" />
  <row Id="3878" PostTypeId="2" ParentId="2125" CreationDate="2010-10-22T09:17:26.323" Score="2" ViewCount="" Body="&lt;p&gt;Correlation analysis only quantifies the relation between two variables ignoring which is dependent variable and which is independent. But before appliyng regression you have to calrify that impact of which variable you want to check on the other variable.&lt;/p&gt;&#xA;" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-10-22T09:51:55.363" LastActivityDate="2010-10-22T09:51:55.363" />
  <row Id="3879" PostTypeId="1" AcceptedAnswerId="3880" CreationDate="2010-10-22T10:05:15.167" Score="3" ViewCount="362" Body="&lt;p&gt;This question is related to &lt;a href=&quot;http://stats.stackexchange.com/questions/3842/how-to-create-a-barplot-diagram-where-bars-are-side-by-side-in-r&quot;&gt;my previous question&lt;/a&gt;. I would like to put values over bars in barplot. I am beginner in plotting in R.&lt;/p&gt;&#xA;" OwnerUserId="1389" LastActivityDate="2010-10-22T12:13:40.343" Title="How to put values over bars in barplot in R" Tags="&lt;r&gt;&lt;data-visualization&gt;" AnswerCount="3" />
  <row Id="3880" PostTypeId="2" ParentId="3879" CreationDate="2010-10-22T10:10:01.447" Score="5" ViewCount="" Body="&lt;p&gt;To add text to a plot, just use the &lt;code&gt;text&lt;/code&gt; command. From @chl's &lt;a href=&quot;http://stats.stackexchange.com/questions/3842/how-to-create-a-barplot-diagram-where-bars-are-side-by-side-in-r/3843#3843&quot;&gt;answer&lt;/a&gt; to your previous question:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;##Create data&#xA;x = replicate(2, sample(letters[1:2], 100, rep=T))&#xA;apply(x, 2, table)&#xA;&#xA;##Create plot&#xA;barplot(matrix(c(5, 3, 8, 9), nr=2), beside=TRUE, &#xA;        col=c(&quot;aquamarine3&quot;, &quot;coral&quot;), &#xA;        names.arg=LETTERS[1:2], &#xA;        ylim=range(0,12)))&#xA;&#xA;##Add text at coordinates: x=2, y=6&#xA;##Use trial and error to place them &#xA;text(2, 6, &quot;A label&quot;)&#xA;text(5, 10, &quot;Another label&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/Oz3iD.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="8" LastActivityDate="2010-10-22T10:10:01.447" CommentCount="1" />
  <row Id="3881" PostTypeId="2" ParentId="3842" CreationDate="2010-10-22T10:34:59.290" Score="1" ViewCount="" Body="&lt;p&gt;?barplot&lt;/p&gt;&#xA;&#xA;&lt;p&gt;reading the help file is sometimes quicker than posting on a forum...&lt;/p&gt;&#xA;" OwnerUserId="1709" LastActivityDate="2010-10-22T10:34:59.290" CommentCount="1" />
  <row Id="3882" PostTypeId="2" ParentId="3879" CreationDate="2010-10-22T11:20:54.903" Score="0" ViewCount="" Body="&lt;p&gt;If you're learing to plot in R you might look at the R graph gallery:&#xA;&lt;a href=&quot;http://addictedtor.free.fr/graphiques/&quot; rel=&quot;nofollow&quot;&gt;http://addictedtor.free.fr/graphiques/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All the graphs there are posted with the code used to build them. Its a good resource.&lt;/p&gt;&#xA;" OwnerUserId="1475" LastActivityDate="2010-10-22T11:20:54.903" />
  <row Id="3883" PostTypeId="2" ParentId="3879" CreationDate="2010-10-22T12:13:40.343" Score="3" ViewCount="" Body="&lt;p&gt;Another example of the use of &lt;code&gt;text&lt;/code&gt; command&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;u &amp;lt;- c(3.2,6.6,11.7,16.3,16.6,15.4,14.6,12.7,11.4,10.2,9.8,9.1,9.1,9.0,8.8,8.4,7.7)&#xA;p &amp;lt;-c(3737,3761,3784,3802,3825,3839,3850,3862,3878,3890,3901,3909,3918,3926,3935,3948)&#xA;     -c(385,394,401,409,422,430,434,437,437,435,436,437,439,442,447,452)&#xA;e &amp;lt;- c(2504,2375,2206,2071,2054,2099,2127,2170,2222,2296,2335,2367,2372,2365,2365,2401)&#xA;&#xA;par(mar=c(2,3,1,2),las=1,mgp=c(2.2,1,0))&#xA;x &amp;lt;- barplot(u,names.arg=1990:2006,col=&quot;palegreen4&quot;,space=.3,ylim=c(0,20),&#xA;              ylab=&quot;%&quot;,xaxs=&quot;r&quot;,xlim=c(.8,21.6))&#xA;text(x,u+.4,format(u,T))&#xA;lines(x[-17],100*e/p-55,lwd=2)&#xA;points(x[-17],100*e/p-55,lwd=2,cex=1.5,pch=1)&#xA;axis(4,seq(0,20,5),seq(55,75,5))&#xA;text(c(x[1]+1,x[5],x[16]+1),c(100*e/p-55)[c(1,5,16)]+c(0,-1,0),&#xA;      format(c(100*e/p)[c(1,5,16)],T,di=3))&#xA;box()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="339" LastActivityDate="2010-10-22T12:13:40.343" CommentCount="1" />
  <row Id="3884" PostTypeId="2" ParentId="3857" CreationDate="2010-10-22T12:28:30.873" Score="2" ViewCount="" Body="&lt;p&gt;I underline all answers given already, but let's call a cat a cat: in many workspaces it is hardly impossible to convince management that investment in &quot;exotic&quot; softwaretools (exotic to them, that is) is necessary, let alone hiring somebody that could set it up and maintain it. I have told quite some clients that they would benefit greatly from hiring a statistician with a thorough background on software and databases, but &quot;no can do&quot; is the general response.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So as long as that ain't going to happen, there are some simple things you can do with Excel that will make life easier. And the first of this is without doubt version control. More info on version control with Excel can be found &lt;a href=&quot;http://stackoverflow.com/questions/131605/best-way-to-do-version-control-for-ms-excel&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Some things about using excel&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;People using EXCEL very often like the formula features of EXCEL. Yet, this is the most important source of errors within EXCEL sheets, and of problems when trying to read in EXCEL files as far as my experience goes. I refuse to work with sheets containing formulas.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also force everybody I work with to deliver the EXCEL sheets in a plain format, meaning that:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The first row contains the names of the different variables&lt;/li&gt;&#xA;&lt;li&gt;The spreadsheet starts in cell A1&lt;/li&gt;&#xA;&lt;li&gt;All data is put in columns, without interruptions and without formatting.&lt;/li&gt;&#xA;&lt;li&gt;If possible, the data is saved in .csv format as well. It's not difficult to write a VBA script that will extract the data, reformat it and put it in a .csv file. This also allows for better version control, as you can make a .csv dump of the data every day.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If there is a general structure the data always has, then it might be good to develop a template with underlying VB macros to add data and generate the dataset for analysis. This in general will avoid that every employee comes up with his own &quot;genius&quot; system of data storage, and it allows you to write your code in function of this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This said, if you can convince everybody to use SQL (and a front end for entering data), you can link R directly to that one. This will greatly increase performance.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Data structure and management&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;As a general rule, the data stored in databases (or EXCEL sheets if they insist) should be the absolute minimum, meaning that any variable that can be calculated from some other variables should not be contained in the database. Mind you, sometimes it can be beneficial to store those derived or transformed variables as well, if the calculations are tedious and take a long time. But these should be stored in a seperate database, if necessary linked to the original one. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thought should be given as well to what is considered as one case (and hence one row). As as an example, people tend to produce time series by making a new variable for each time point. While this makes sense in an EXCEL, reading in these data demands quite some flipping around of the data matrix. Same for comparing groups: There should be one group indicator and one response variable, not a response variable for each group. This way data structures can be standardized as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A last thing I run into frequently, is the use of different metrics. Lengths are given in meters or centimeters, temperatures in Celcius, Kelvin or Farenheit, ... One should indicate in any front end or any template what the unit is in which the variable is measured. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;And even after all these things, you still want to have a data control step before you actually start with the analysis. Again, this can be any script that runs daily (e.g. overnight) on new entries, and that flags problems immediately (out of range, wrong type, missing fields, ...) so they can be corrected as fast as possible. If you have to return to an entry that was made 2 months ago to find out what is wrong and why, you better get some good &quot;Sherlock-skills&quot; to correct it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;my 2 cents&lt;/p&gt;&#xA;" OwnerUserId="1124" LastActivityDate="2010-10-22T12:28:30.873" CommunityOwnedDate="2010-10-22T14:03:02.960" CommentCount="5" />
  <row Id="3885" PostTypeId="1" AcceptedAnswerId="3886" CreationDate="2010-10-22T13:05:30.993" Score="3" ViewCount="83" Body="&lt;p&gt;Three unbiased estimators of standard deviation have been presented in the literature for m samples of size n. The first one is based on the sample ranges and can be obtained by&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\hat \sigma_1 = \frac{\bar R}{d_2}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second one is based on sample standard deviations and can be calculated by&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\hat \sigma_2 = \frac{\bar S}{c_4}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The third estimator is due to sample variance and can be estimated as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\hat \sigma_3 = \frac{\sqrt{\bar {S^2}}}{k_4}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The denominators are constant values dependent on the sample size n and make the estimators unbiased. Does anybody know of a method to get their value in R (instead of using tables in statistical quality control books)?&lt;/p&gt;&#xA;" OwnerUserId="339" LastActivityDate="2010-10-22T13:22:57.057" Title="Computing unbiased estimators of $\sigma$ for m samples of size n" Tags="&lt;r&gt;&lt;standard-deviation&gt;&lt;quality-control&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="3886" PostTypeId="2" ParentId="3885" CreationDate="2010-10-22T13:22:57.057" Score="2" ViewCount="" Body="&lt;p&gt;Ok, I finally found them &lt;a href=&quot;http://finzi.psych.upenn.edu/R/library/IQCC/html/00Index.html&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="339" LastActivityDate="2010-10-22T13:22:57.057" CommentCount="2" />
  <row Id="3887" PostTypeId="2" ParentId="3857" CreationDate="2010-10-22T13:35:54.403" Score="1" ViewCount="" Body="&lt;p&gt;VisTrails: A Python-Based Scientific Workflow and Provenance System&#xA;This talk given at PyCon 2010 has some good ideas. Worth listening to even if you are not interested in using VisTrails or python. In the end I think if you would be able to require that there be a clear document way to reproduce the data. And require some validation that they can.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Quoting&#xA;&quot;In this talk, we will give an overview of VisTrails (http://www.vistrails.org), a python-based open-source scientific workflow that transparently captures provenance (i.e., lineage) of both data products and the processes used to derive these products. We will show how VisTrails can be used to streamline data exploration and visualization. Using real examples, we will demonstrate key features of the system, including the ability to visually create information processing pipelines that combine multiple tools and Iibraries such as VTK, pylab, and matplotlib. We will also show how VisTrails leverages provenance information not only to support result reproducibility, but also to simplify the creation and refinement of pipelines.&quot;&lt;/p&gt;&#xA;" OwnerUserId="1189" LastActivityDate="2010-10-22T13:35:54.403" CommunityOwnedDate="2010-10-22T14:03:02.960" CommentCount="1" />
  <row Id="3888" PostTypeId="1" AcceptedAnswerId="3889" CreationDate="2010-10-22T13:58:09.980" Score="1" ViewCount="188" Body="&lt;p&gt;I have a series of boxplots that i am generating with ggplot2.  I want to control the order in which they are displayed.  Is there a way to control this order?  I have two genotypes and i want them displayed as WT then KO rather than the reverse (which is what i am getting as a default).  My code right now is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;p &amp;lt;- qplot(genotype, activity.ratio, data = df) &#xA;p + geom_boxplot() + geom_jitter()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Also if this is better as a SO question than this forum then please let me know (and if it is the correct forum can someone create a ggplot tag).&lt;/p&gt;&#xA;" OwnerUserId="1327" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-11-30T16:42:29.257" LastActivityDate="2010-11-30T16:42:29.257" Title="Reorder categorical data in ggplot2" Tags="&lt;r&gt;&lt;boxplot&gt;&lt;ggplot2&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="1" />
  <row Id="3889" PostTypeId="2" ParentId="3888" CreationDate="2010-10-22T14:06:48.103" Score="4" ViewCount="" Body="&lt;p&gt;Would that help?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x &amp;lt;- gl(2, 20, 40, labels=c(&quot;KO&quot;,&quot;WT&quot;))&#xA;y &amp;lt;- rnorm(40)&#xA;qplot(x,y)&#xA;qplot(relevel(x, &quot;WT&quot;),y)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-10-22T14:06:48.103" />
  <row Id="3890" PostTypeId="1" AcceptedAnswerId="3891" CreationDate="2010-10-22T14:08:01.243" Score="1" ViewCount="614" Body="&lt;p&gt;I would like to use the fuction &lt;strong&gt;weighted.median()&lt;/strong&gt; in package &lt;strong&gt;&lt;a href=&quot;http://www.stat.ucl.ac.be/ISdidactique/Rhelp/library/R.basic/html/weighted.median.html&quot; rel=&quot;nofollow&quot;&gt;R.basic&lt;/a&gt;&lt;/strong&gt; (and some other ones too, although this one is most important).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;install.packages(&quot;R.basic&quot;) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;gives me:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;package ‘R.basic’ is not available&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(R.basic) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;says &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Error in library(&quot;R.basic&quot;) : there is no package called 'R.basic'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;There is a downloadable version &lt;a href=&quot;http://www1.maths.lth.se/help/R/R.classes/&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;. But, typing:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;install.packages(&quot;~/Documents/R.classes_0.62.tar.gz&quot;, CRAN=NULL)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Gives &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Error in do_install(pkg) : &#xA;  this seems to be a bundle -- and they are defunct&#xA;Warning message:&#xA;In install.packages(&quot;~/Documents/R.classes_0.62.tar.gz&quot;,  :&#xA;  installation of package '~/Documents/R.classes_0.62.tar.gz' had non-zero exit status&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How to get thru ? &lt;/p&gt;&#xA;" OwnerUserId="603" LastEditorUserId="603" LastEditorDisplayName="" LastEditDate="2010-10-22T21:42:16.497" LastActivityDate="2010-10-22T21:42:16.497" Title="package R.basic" Tags="&lt;r&gt;" AnswerCount="1" />
  <row Id="3891" PostTypeId="2" ParentId="3890" CreationDate="2010-10-22T14:12:55.267" Score="4" ViewCount="" Body="&lt;p&gt;That's because the package doesn't exist on CRAN (see &lt;a href=&quot;http://cran.r-project.org/web/packages/&quot; rel=&quot;nofollow&quot;&gt;the package list&lt;/a&gt;).  It may have failed a build in a recent version of R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can install it directly from the author's site like so:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;install.packages(c(&quot;R.basic&quot;), contriburl=&quot;http://www.braju.com/R/repos/&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;See &lt;a href=&quot;http://www.braju.com/R/&quot; rel=&quot;nofollow&quot;&gt;Henrik Bengtsson's home page&lt;/a&gt; for more detail.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Edit&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just to add a little further: it looks like this package fails to build on later versions of R.  You should probably get the source and build it yourself, or else contact the package author.&lt;/p&gt;&#xA;" OwnerUserId="5" LastEditorUserId="5" LastEditorDisplayName="" LastEditDate="2010-10-22T14:21:15.993" LastActivityDate="2010-10-22T14:21:15.993" CommentCount="4" />
  <row Id="3892" PostTypeId="1" AcceptedAnswerId="3895" CreationDate="2010-10-22T15:08:17.383" Score="8" ViewCount="198" Body="&lt;p&gt;Why does OLS estimation involve taking vertical deviations of the points to the line rather than horizontal distances?&lt;/p&gt;&#xA;" OwnerUserId="333" LastActivityDate="2011-01-19T04:56:27.243" Title="Why vertical distances?" Tags="&lt;least-squares&gt;" AnswerCount="3" />
  <row Id="3893" PostTypeId="1" AcceptedAnswerId="4095" CreationDate="2010-10-22T15:12:03.423" Score="14" ViewCount="546" Body="&lt;p&gt;In all contexts I am familiar with cross-validation it is solely used with the goal of increasing predictive accuracy. Can the logic of cross validation be extended in estimating the unbiased relationships between variables? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;While &lt;a href=&quot;http://dx.doi.org/10.1007/s10940-009-9077-7&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; paper by Richard Berk demonstrates the use of a hold out sample for parameter selection in the &quot;final&quot; regression model (and demonstrates why step-wise parameter selection is not a good idea), I still don't see how that exactly ensures unbiased estimates of the effect X has on Y any more so than choosing a model based on logic and prior knowledge of the subject.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I ask that people cite examples in which one used a hold-out sample to aid in causal inference, or general essays that may help my understanding. I also don't doubt my conception of cross validation is naive, and so if it is say so. It seems offhand the use of a hold out sample would be amenable to causal inference, but I do not know of any work that does this or how they would do this. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Citation for the Berk Paper:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://dx.doi.org/10.1007/s10940-009-9077-7&quot; rel=&quot;nofollow&quot;&gt;Statistical Inference After Model Selection&lt;/a&gt;&#xA;by: Richard Berk, Lawrence Brown, Linda Zhao&#xA;Journal of Quantitative Criminology, Vol. 26, No. 2. (1 June 2010), pp. 217-236. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;PDF version &lt;a href=&quot;http://www-stat.wharton.upenn.edu/~lzhao/papers/MyPublication/StatInfAfterMS_JQC_2010.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://stats.stackexchange.com/q/3252/1036&quot;&gt;This&lt;/a&gt; question on exploratory data analysis in small sample studies by chl prompted this question. &lt;/p&gt;&#xA;" OwnerUserId="1036" LastEditorUserId="1036" LastEditorDisplayName="" LastEditDate="2010-10-22T19:40:14.977" LastActivityDate="2010-10-31T22:40:29.637" Title="Can cross validation be used for causal inference?" Tags="&lt;cross-validation&gt;&lt;causal-inference&gt;" AnswerCount="6" FavoriteCount="4" />
  <row Id="3894" PostTypeId="1" CreationDate="2010-10-22T15:15:30.043" Score="3" ViewCount="48" Body="&lt;p&gt;I was wondering whether it's me mis-remembering something or is there a dimensionless measure of dispersion calculated as: range/(mean or median). Where range is a difference between max and min values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S. Could this be called &quot;relative range&quot;?&lt;/p&gt;&#xA;" OwnerUserId="219" LastEditorUserId="219" LastEditorDisplayName="" LastEditDate="2010-10-22T15:24:29.867" LastActivityDate="2010-10-22T15:33:55.257" Title="Is there a parameter that is calculated as a range to mean ratio?" Tags="&lt;distributions&gt;" AnswerCount="2" CommentCount="3" />
  <row Id="3895" PostTypeId="2" ParentId="3892" CreationDate="2010-10-22T15:16:37.303" Score="9" ViewCount="" Body="&lt;p&gt;OLS (&lt;a href=&quot;http://en.wikipedia.org/wiki/Ordinary_least_squares&quot; rel=&quot;nofollow&quot;&gt;ordinary least squares&lt;/a&gt;) assumes that the values represented by the horizontal distances are either predetermined by the experimenter or measured with high accuracy (relative to the vertical distances).  When there is a question of uncertainty in the horizontal distances, you shouldn't be using OLS, but instead should look into &lt;a href=&quot;http://en.wikipedia.org/wiki/Errors-in-variables_models&quot; rel=&quot;nofollow&quot;&gt;errors-in-variables models&lt;/a&gt; or, possibly, &lt;a href=&quot;http://en.wikipedia.org/wiki/Principal_component_analysis&quot; rel=&quot;nofollow&quot;&gt;principal components analysis&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-01-19T04:56:27.243" LastActivityDate="2011-01-19T04:56:27.243" CommentCount="2" />
  <row Id="3896" PostTypeId="2" ParentId="3894" CreationDate="2010-10-22T15:26:45.180" Score="2" ViewCount="" Body="&lt;p&gt;The &lt;a href=&quot;http://en.wikipedia.org/wiki/Coefficient_of_variation&quot; rel=&quot;nofollow&quot;&gt;coefficient of variation&lt;/a&gt; is defined as (standard deviation) / mean. Is that what you're thinking of??&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-10-22T15:26:45.180" CommentCount="1" />
  <row Id="3897" PostTypeId="2" ParentId="3894" CreationDate="2010-10-22T15:33:55.257" Score="2" ViewCount="" Body="&lt;p&gt;The measure can be seen in a few publications. I can remember a paper by Lord (1947) using the range as an approximation to the standard deviation for the t-test. So the range to mean ratio would be an approximation to the coefficient of variation which onestop has mentioned already.&lt;/p&gt;&#xA;" OwnerUserId="1573" LastActivityDate="2010-10-22T15:33:55.257" CommentCount="1" />
  <row Id="3898" PostTypeId="1" CreationDate="2010-10-22T15:41:29.937" Score="6" ViewCount="177" Body="&lt;p&gt;I am working with a given factor model of the form &#xA;$$y=B x +\epsilon$$&#xA;where $x$ is a random vector in $R^M$, $B$ is an $N\times M$ matrix of factor loadings, $y$ is a random vector in $R^N$ (with $N \gg M$) and $\epsilon$ is a vector of mean-zero innovations, independent of $x$. For simplicity, the innovations are normally distributed and pairwise independent. My question is relatively simple to state: How can I test whether a factor is significant, in the same sense in which a predictor is significant in linear regression?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Clarification&lt;/strong&gt;: I should perhaps have emphasized that I am working with a &lt;strong&gt;given&lt;/strong&gt; model, and I have to assess the predictive value of each factor of the model. In other terms, is there a simple way to assess whether dropping a factor from a model would result in a loss of predictive power, &lt;em&gt;without&lt;/em&gt; comparing two different models with and without the factor? In the latter case, there is a great deal of literature on model selection, based on ML, GMM etc. (The rotational invariance of factor models doesn't play an essential role, btw).&lt;/p&gt;&#xA;" OwnerUserId="30" LastEditorUserId="30" LastEditorDisplayName="" LastEditDate="2010-10-23T22:35:32.570" LastActivityDate="2011-04-09T18:09:15.317" Title="Factor Significance for Factor Model" Tags="&lt;hypothesis-testing&gt;&lt;factor-analysis&gt;" AnswerCount="3" FavoriteCount="1" />
  <row Id="3899" PostTypeId="2" ParentId="1053" CreationDate="2010-10-22T16:15:02.407" Score="1" ViewCount="" Body="&lt;p&gt;I learned from &lt;a href=&quot;http://amzn.com/0471754994&quot; rel=&quot;nofollow&quot;&gt;Hosmer &amp;amp; Lemeshow &amp;amp; May&lt;/a&gt;, which covers the basics. It also helped that I found a really cheap copy...&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-10-22T16:15:02.407" CommunityOwnedDate="2010-10-22T16:15:02.407" />
  <row Id="3900" PostTypeId="2" ParentId="3898" CreationDate="2010-10-22T16:21:05.233" Score="1" ViewCount="" Body="&lt;p&gt;The short answer is: There is something you can do but I am not sure how meaningful it will be. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Long answer: I will give the long answer for a simple model where we have only one unknown latent factor. The idea carries over to the more general case albeit with more complications. It follows from your factor model that:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$E(y) = B E(x)$ and&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Var(y) = B^T  B Var(x) + \sigma^2$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Note: Keep in mind that in this simplified model: $x$ is a scalar we are dealing with only one factor).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since the data is normally distributed the above two equations determine your likelihood function. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, do note that you have identification issues here as you have to estimate $B$ and $x$ simultaneously. The traditional way to solve the identification issue is to assume that:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$B(1) = 1$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(i.e., set the factor loading of the first element to 1. Otherwise we can scale $B$ by $\alpha$ and scale $E(x)$ by $\frac{1}{\alpha}$ and obtain an identical E(y)).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It follows that:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;E(x) = E(y(1)) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words, the above identification constraint on B has effectively constrained the mean of our factor to be the sample mean of our first dependent variable (i.e., y(1)).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For similar reasons, we assume that:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Var(x) = 1$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Otherwise you could just scale $B$ by $\sqrt{\alpha}$ and scale $Var(x)$ by $\frac{1}{\alpha}$ and your likelihood function will not change).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since we impose an identification constraint on the distribution of x (which is in some sense arbitrary) I am not sure how meaningful it is to perform statistical testing for a factor. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could compute factor scores and perform standard statistical testing using the mean of $E(x) = E(y(1))$ and $Var(x) = 1$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, the answer to your question depends on:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Is the conclusion from the above statistical test invariant to your choice of identification constraints?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I do not know the answer to the above question.&lt;/p&gt;&#xA;" LastActivityDate="2010-10-22T16:21:05.233" />
  <row Id="3901" PostTypeId="2" ParentId="2504" CreationDate="2010-10-22T19:15:54.743" Score="3" ViewCount="" Body="&lt;p&gt;The counterexample is not relevant to the question asked. You want to test the &lt;em&gt;null&lt;/em&gt; hypothesis that a sample of i.i.d. random variables is drawn from a distribution having finite variance, &lt;em&gt;at a given significance level&lt;/em&gt;. I recommend a good reference text like &quot;Statistical Inference&quot; by Casella to understand the use and the limit of hypothesis testing. Regarding h.t. on finite variance, I don't have a reference handy, but the following paper addresses a similar, but stronger, version of the problem, i.e., if the distribution tails follow a power law. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://arxiv.org/PS_cache/arxiv/pdf/0706/0706.1062v2.pdf&quot; rel=&quot;nofollow&quot;&gt;POWER-LAW DISTRIBUTIONS IN EMPIRICAL DATA&lt;/a&gt; &lt;em&gt;SIAM Review&lt;/em&gt; &lt;strong&gt;51&lt;/strong&gt; (2009): 661--703. &lt;/p&gt;&#xA;" OwnerUserId="30" LastActivityDate="2010-10-22T19:15:54.743" />
  <row Id="3902" PostTypeId="1" AcceptedAnswerId="3906" CreationDate="2010-10-22T19:26:11.547" Score="1" ViewCount="243" Body="&lt;p&gt;Does 'corr' function in Matlab consider correction for tied ranks while calculating Spearman correlation coefficient?&#xA;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="1564" LastEditorUserId="1564" LastEditorDisplayName="" LastEditDate="2010-10-22T19:36:01.037" LastActivityDate="2010-10-23T00:16:03.480" Title="Does corr function in Matlab consider correction for tied ranks?" Tags="&lt;correlation&gt;&lt;matlab&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="3903" PostTypeId="2" ParentId="3892" CreationDate="2010-10-22T20:36:49.587" Score="1" ViewCount="" Body="&lt;p&gt;Interesting Question. My answer would be that when we are fitting an OLS model we are implicitly and primarily trying to predict/explain the dependent variable at hand - the &quot;Y&quot; in the &quot;Y vs X.&quot; As such, our main concern would be to minimize the distance from our fitted line to the actual observations with respect to the outcome, which means minimizing the vertical distance. This of course defines the residuals. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, least squares formulas are easier to derive than most other competing methods, which is perhaps why it came around first. :P&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As 'whuber' alludes to above, there are other approaches that treat X and Y with equal emphasis when fitting a best-fit line. One such approach that I'm aware of is &quot;principal lines&quot; or &quot;principal curves&quot; regression, which minimizes the orthogonal distances between the points and the line (instead of a vertical error lines you have ones at 90 degrees to the fitted line). I post one reference below for your reading. It's lengthy but very accessible and enlightening. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps,&#xA;Brenden&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Trevor Hastie. &lt;a href=&quot;http://www.slac.stanford.edu/cgi-wrap/getdoc/slac-r-276.pdf&quot; rel=&quot;nofollow&quot;&gt;Principal Curves and Surfaces&lt;/a&gt;, PhD thesis, Stanford University; 1984&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="1501" LastActivityDate="2010-10-22T20:36:49.587" />
  <row Id="3904" PostTypeId="2" ParentId="3734" CreationDate="2010-10-22T21:11:42.460" Score="7" ViewCount="" Body="&lt;p&gt;I apologize in advance for the length of this post: it is with some trepidation that I let it out in public at all, because it takes some time and attention to read through and undoubtedly has typographic errors and expository lapses.  But here it this for those who are interested in the fascinating topic, offered in the hope that it will encourage you to identify one or more of the many parts of the CLT for further elaboration in responses of your own.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Most attempts at &quot;explaining&quot; the CLT are illustrations or just restatements that assert it's true.  A really penetrating, correct explanation would have to explain an awful lot of things.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Before looking at this further, let's be clear about what the CLT says.  As you all know, there are versions that vary in their generality.  The common context is a sequence of random variables.  For intuitive explanations that hold up rigorously I find it helpful to think of a probability space as a box with distinguishable objects.  It doesn't matter what those objects are but I'll call them &quot;tickets.&quot;  We make one &quot;observation&quot; of a box by thoroughly mixing up the tickets and drawing one out; that ticket constitutes the observation.  After recording it for later analysis we return the ticket to the box so that its contents remain unchanged. A &quot;random variable&quot; is just a number written on each ticket.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;in 1735, Abraham de Moivre considered the case of a single box where the numbers on the tickets were only zeros and ones (&quot;Bernoulli trials&quot;), with some of each number present.  He imagined making $n$ physically &lt;em&gt;independent&lt;/em&gt; observations, yielding a sequence of values $x_1, x_2, \ldots, x_n$, all of which are zero or one.  The &lt;em&gt;sum&lt;/em&gt; of those values, $y_n = x_1 + x_2 + \ldots + x_n$ is random because the terms in the sum are.  Therefore, if we could repeat this procedure many times, various sums (ranging from $0$ through $n$) would appear with various frequencies.  Now one would expect--and it's true--that for very large values of $n$, all the relative frequencies would be quite small.  If we were to be so bold (or foolish) as to attempt to &quot;take a limit&quot; or &quot;let $n$ go to $\infty$&quot;, we would conclude that all frequencies reduce to $0$.  But &lt;em&gt;if we simply draw a histogram&lt;/em&gt; of the frequencies, without paying any attention to how its axes are labeled, we see that the histograms for large $n$ all look the same: in some sense, they &lt;em&gt;approach a limit.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The insight here is to draw the histogram first and label its axes later.  With large $n$ the histogram covers a large range of values centered around $n/2$ (on the horizontal axis) and a vanishingly small range of values (on the vertical axis).  Fitting this curve into the plotting region has required both a &lt;em&gt;shifting&lt;/em&gt; and &lt;em&gt;rescaling&lt;/em&gt; of the histogram.  The mathematical description of this is that for each $n$ we can choose some central value $m_n$ (not necessarily unique!) to position the histogram and some scale value $s_n$ (not necessarily unique!) to make it fit within the axes.  This is done mathematically by changing $y_n$ to $z_n = (y_n - m_n) / s_n$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Remember that a histogram represents frequencies by &lt;em&gt;areas&lt;/em&gt; between it and the horizontal axis.  The eventual stability of these histograms for large values of $n$ should therefore be stated in terms of area.  So, pick any interval of values you like, say from $a$ to $b \gt a$ and, as $n$ increases, track the area of the part of the histogram of $z_n$ that horizontally spans the interval $(a, b]$.  The CLT asserts several things:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;em&gt;No matter what $a$ and $b$ are,&lt;/em&gt; if we choose the sequences $m_n$ and $s_n$ appropriately (in a way that does not depend on $a$ or $b$ at all), this area indeed approaches a limit as $n$ gets large.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The sequences $m_n$ and $s_n$ can be chosen in a way that depends only on $n$, the average of values in the box, and a measure of spread of those values--but on nothing else--so that regardless of what is in the box, the limit is always the same.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Specifically, that limiting area is the area under the curve $y = \exp(-z^2/2) / \sqrt{2 \pi}$ between $a$ and $b$: this is the formula of that universal limiting histogram.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first generalization of the CLT adds,&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;When the box can contain numbers in addition to zeros and ones, &lt;em&gt;exactly the same conclusions hold&lt;/em&gt; (provided that the proportions of extremely large or small numbers in the box are not &quot;too great,&quot; a criterion that has a precise quantitative statement).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The next generalization, and perhaps the most amazing one, replaces this single box of tickets with an ordered indefinitely long array of boxes with tickets.  Each box can have different numbers on its tickets in different proportions.  The observation $x_1$ is made by drawing a ticket from the first box, $x_2$ comes from the second box, and so on.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;em&gt;Exactly the same conclusions hold&lt;/em&gt; provided the contents of the boxes are &quot;not too different&quot; (there are several precise, but different, quantitative characterizations of what &quot;not too different&quot; has to mean; they allow an astonishing amount of latitude).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;These five assertions, at a minimum, need explaining.  There's more.  Several intriguing aspects of the setup are implicit in all the statements.  For example, &lt;em&gt;what is special about the sum&lt;/em&gt;?  Why don't we have central limit theorems for other mathematical combinations of numbers such as their product or their maximum?  (It turns out we do, but they are not quite so general nor do they always have such a clean, simple conclusion unless they can be reduced to the CLT.)  The sequences of $m_n$ and $s_n$ are not unique but they're &lt;em&gt;almost&lt;/em&gt; unique in the sense that eventually they have to approximate the expectation of the sum of $n$ tickets and the &lt;em&gt;standard deviation&lt;/em&gt; of the sum, respectively (which, in the first two statements of the CLT, equals $\sqrt{n}$ times the standard deviation of the box).  The standard deviation is one measure of the spread of values, but it is by no means the only one nor is it the most &quot;natural,&quot; either historically or for many applications.  (Many people would choose something like a median absolute deviation from the median, for instance.)  Why does the SD appear in such an essential way?  Or take the formula for the limiting histogram: who would have expected it to take such a form?  It says the &lt;em&gt;logarithm&lt;/em&gt; of the probability density is a &lt;em&gt;quadratic&lt;/em&gt; function.  Why?  Is there some intuitive or clear, compelling explanation for this?&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;I confess I'm unable to reach the ultimate goal of supplying answers that are simple enough to meet Srikant's challenging criteria for intuitiveness and simplicity, but I have sketched this background in the hope that others might be inspired to fill in some of the many gaps.  I think a good demonstration will ultimately have to rely on an elementary analysis of how values between $\alpha_n = a s_n + m_n$ and $\beta_n = b s_n + m_n$ can arise in forming the sum $x_1 + x_2 + \ldots + x_n$.  Going back to the single-box version of the CLT, the case of a symmetric distribution is simpler to handle: its median equals its mean, so there's a 50% chance that $x_i$ will be less than the box's mean and a 50% chance that $x_i$ will be greater than its mean.  Moreover, when $n$ is sufficiently large, the positive deviations from the mean ought to compensate for the negative deviations in the mean.  (This requires some careful justification, not just hand waving.)  Thus we ought primarily to be concerned about &lt;em&gt;counting&lt;/em&gt; the numbers of positive and negative deviations and only have a secondary concern about their &lt;em&gt;sizes.&lt;/em&gt;  (Of all the things I have written here, this might be the most useful at providing some intuition about why the CLT works.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This shows, to some degree anyway, why the first generalization of the CLT does not really uncover anything that wasn't in de Moivre's original Bernoulli trial version. At this point it looks like there nothing for it but to do a little math: we need to count the number of distinct ways in which the number of positive deviations from the mean can differ from the number of negative deviations by any predetermined value $k$, where evidently $k$ is one of $-n, -n+2, \ldots, n-2, n$.  But because vanishingly small errors will disappear in the limit, we don't have to count precisely; we only need to approximate the counts.  To this end it suffices to know that&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\text{The number of ways to obtain } k \text{ positive and } n-k \text{ negative values out of } n$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\text{equals } \frac{n-k}{k}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\text{times the number of ways to get } k-1 \text{ positive and } n-k+1 \text { negative values.}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(That's a perfectly elementary result so I won't bother to write down the justification.)  Now we approximate wholesale.  The maximum frequency occurs when $k$ is as close to $n/2$ as possible (also elementary).  Let's write $m = n/2$.  Then, &lt;em&gt;relative to the maximum frequency,&lt;/em&gt; the frequency of $m+j$ positive deviations ($j \ge 0$) is estimated by the product&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{m-1}{m+1} \frac{m-2}{m+2} \cdots \frac{m-j}{m+j}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$=\frac{1 - 1/m}{1 + 1/m} \frac{1-2/m}{1+2/m} \cdots \frac{1-j/m}{1+j/m}.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;135 years before de Moivre was writing, John Napier invented logarithms to simplify multiplication, so let's take advantage of this.  Using the approximation&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\log(\frac{1-x}{1+x}) \sim -2x,$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;we find that the log of the relative frequency is approximately&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$-2/m - 4/m - \cdots - (2j)/m \sim -\frac{j^2}{m}.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because the cumulative error is proportional to $j^4/m^3$, this ought to work well provided $j^4$ is small relative to $m^3$.  That covers a greater range of values of $j$ than is needed.  (It suffices for the approximation to work for $j$ arbitrarily larger than $\sqrt{m}$.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Obviously much more analysis of this sort should be presented to justify the other assertions in the CLT, but I'm running out of time, space, and energy and I've probably lost 90% of the people who started reading this anyway.  This simple approximation, though, suggests how de Moivre might originally have suspected that there is a universal limiting distribution, that its logarithm is a quadratic function, and that the proper scale factor $s_n$ must be proportional to $\sqrt{n}$ (because $j^2/m = 2 j^2 / n = 2 (j/\sqrt{n})^2$).  It is difficult to imagine how this important quantitative relationship could be explained without invoking some kind of mathematical information and reasoning; anything less would leave the precise shape of the limiting curve a complete mystery.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-10-23T21:20:45.120" LastActivityDate="2010-10-23T21:20:45.120" CommentCount="5" />
  <row Id="3906" PostTypeId="2" ParentId="3902" CreationDate="2010-10-23T00:16:03.480" Score="3" ViewCount="" Body="&lt;p&gt;looking in the &lt;code&gt;corr&lt;/code&gt; function, I see that it calls &lt;code&gt;tiedrank&lt;/code&gt;: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;$ grep -B 1 tiedrank corr.m&#xA;      case 'k' % Kendall's tau&#xA;          [xrank, xadj] = tiedrank(xi,1);&#xA;          [yrank, yadj] = tiedrank(yj,1);&#xA;--&#xA;       case 's' % Spearman's rank correlation&#xA;          [xrank, xadj] = tiedrank(xi,0);&#xA;          [yrank, yadj] = tiedrank(yj,0);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Issuing a &lt;code&gt;help tiedrank&lt;/code&gt; in Matlab should help you further along.&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-10-23T00:16:03.480" />
  <row Id="3907" PostTypeId="1" AcceptedAnswerId="4073" CreationDate="2010-10-23T01:00:11.397" Score="5" ViewCount="299" Body="&lt;p&gt;We have prospective data from an observational registry and wish to consider the affects of a gene on time to cardiovascular events. The data includes standard data like age, gender, ... and also the date(age) of a cardio event.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general terms what are the conditions/assumptions/requirements of a starting time for survival analysis? Entry into the registry is at random times for each individual (as he or she is recruited into the registry).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is obviously no data at which the registry participants' genes changed, and there no intervention. Is it valid to use the date of enrollment in the registry as the start time for the survival analysis?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We (group working on this) are asking as some work has been done and the the enrollment date was used as the start time for the survival interval and there is some disagreement now as to whether this is a valid approach.&lt;/p&gt;&#xA;" OwnerUserId="1189" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-10-24T16:31:20.160" LastActivityDate="2010-10-29T19:58:17.210" Title="Start time requirements or assumptions for survival analysis" Tags="&lt;regression&gt;&lt;survival&gt;" AnswerCount="4" CommentCount="7" />
  <row Id="3909" PostTypeId="2" ParentId="3893" CreationDate="2010-10-23T05:14:26.467" Score="7" ViewCount="" Body="&lt;p&gt;This is a really interesting question and I don't offer any specific citations.  However, in general, I'd say, NO, in and of itself, cross-validation does not offer any insight into causality.  In absence of a designed experiment, the issue of causality is always uncertain.  As you suggest, cross-validation can and will improve predictive accuracy.  This, alone, says nothing about causality.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Absent of a designed experiment, causal inference would require a model that includes all of the relevant predictors--something that we can rarely guarantee in an observational study.  Moreover, a simple lag variable, for example (or anything highly correlated with whatever outcome we were trying to predict) would produce a good model and one which could be validated in multiple samples.  That does not mean, however, that we can infer causation.  Cross-validation assures repeatability in predictions and nothing more.  Causality is a matter of design and logic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT:&#xA;Here's an example to illustrate.  I could build a model with good predictive accuracy that predicts the population of a city based on the amount of money the city spends on trash removal.  I could use cross-validation to test the accuracy of that model as well as other methods to improve the accuracy of prediction and get more stable parameters.  Now, while this model works great for prediction, the causal logic is wrong--the causal direction is reversed.  No matter what the folks in the Public Works Department might argue, increasing their budget for trash removal would not be a good strategy to increase the city's population (the causal interpretation).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The issues of accuracy and repeatability of a model are separate from our ability to make  causal inferences about the relationships we observe.  Cross-validation helps us with the former and not with the latter.  Now, IF we are estimating a &quot;correct&quot; model in terms of specifying a casual relationship (for example, trying to determine what our trash removal budget should be based on our expected population next year), cross-validation can help us to have greater confidence in our estimate of that effect.  However, cross-validation does nothing to help us choose the &quot;correct&quot; model with regard to causal relationships.  Again, here we need to rely on the design of the study, our subject matter expertise, theory, and logic.&lt;/p&gt;&#xA;" OwnerUserId="485" LastEditorUserId="485" LastEditorDisplayName="" LastEditDate="2010-10-24T18:29:37.743" LastActivityDate="2010-10-24T18:29:37.743" CommentCount="5" />
  <row Id="3910" PostTypeId="2" ParentId="3724" CreationDate="2010-10-23T06:00:07.847" Score="1" ViewCount="" Body="&lt;p&gt;Use VAR (with n-1 denominator) when you wish to estimate the variance of the underlying population from the sample, or VARP (with n denominator) when the sample &lt;em&gt;is&lt;/em&gt; the population. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I find the name &quot;population variance&quot; quite ambiguous...&lt;/p&gt;&#xA;" OwnerUserId="1679" LastActivityDate="2010-10-23T06:00:07.847" />
  <row Id="3911" PostTypeId="1" CreationDate="2010-10-23T07:48:41.417" Score="9" ViewCount="472" Body="&lt;p&gt;If I understand correctly a confidence interval of a parameter is an interval constructed by a &lt;em&gt;method&lt;/em&gt; which yields intervals containing the true value for a specified proportion of samples. So the 'confidence' is about the method rather than the interval I compute from a particular sample. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a user of statistics I have always felt cheated by this since the space of all samples is hypothetical. All I have is one sample and I want to know what that sample tells me about a parameter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this judgement wrong? Are there ways of looking at confidence intervals, at least in some circumstances, which would be meaningful to users of statistics?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[This question arises from second thoughts after dissing confidence intervals in a math.se answer &lt;a href=&quot;http://math.stackexchange.com/questions/7564/calculating-a-sample-size-based-on-a-confidence-level/7572#7572&quot;&gt;http://math.stackexchange.com/questions/7564/calculating-a-sample-size-based-on-a-confidence-level/7572#7572&lt;/a&gt; ]&lt;/p&gt;&#xA;" OwnerUserId="1393" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-10-23T08:47:26.290" LastActivityDate="2010-11-03T00:38:16.730" Title="When are confidence intervals useful?" Tags="&lt;confidence-interval&gt;&lt;interpretation&gt;" AnswerCount="5" FavoriteCount="4" />
  <row Id="3912" PostTypeId="2" ParentId="3911" CreationDate="2010-10-23T08:09:48.700" Score="6" ViewCount="" Body="&lt;p&gt;I like to think of CIs as some way to escape the Hypothesis Testing (HT) framework, at least the binary decision framework following &lt;a href=&quot;http://j.mp/awJEkH&quot; rel=&quot;nofollow&quot;&gt;Neyman&lt;/a&gt;'s approach, and keep in line with theory of measurement in some way. More precisely, I view them as more close to the reliability of an estimation (a difference of means, for instance), and conversely HT are more close to hypothetico-deductive reasoning, with its pitfalls (we cannot accept the null, the alternative is often stochastic, etc.). Still, with both interval estimation and HT we have to rely on distribution assumptions most of the time (e.g. a sampling distribution under $H_0$), which allows to make inference from our sample to the general population or a representative one (at least in the frequentist approach).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In many context, CIs are complementary to usual HT, and I view them as in the following picture (it is under $H_0$):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/9VSib.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;that is, under the HT framework (left), you look at how far your statistic is from the null, while with CIs (right) you are looking at the null effect &quot;from your statistic&quot;, in a certain sense. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, note that for certain kind of statistic, like odds-ratio, HT are often meaningless and it is better to look at its associated CI which is assymmetrical and provide more relevant information as to the direction and precision of the association, if any.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-23T08:17:26.293" LastActivityDate="2010-10-23T08:17:26.293" CommentCount="9" />
  <row Id="3913" PostTypeId="2" ParentId="3892" CreationDate="2010-10-23T08:22:28.610" Score="1" ViewCount="" Body="&lt;p&gt;It possibly also relates to designed experiments - if x is a controlled quantity that is part of the experimental design, it is treated as deterministic;  whilst y is the outcome,  and is a random quantity. x might be a continuous quantity (eg concentration of some drug) but could be a 0/1 split (leading to a 2 sample t-test assuming y is Gaussian).  If x is a continuous quantity there may be some measurement error, but typically if this is much smaller than the variability of y then this is ignored.  &lt;/p&gt;&#xA;" OwnerUserId="1680" LastActivityDate="2010-10-23T08:22:28.610" />
  <row Id="3914" PostTypeId="2" ParentId="3911" CreationDate="2010-10-23T08:45:56.277" Score="4" ViewCount="" Body="&lt;p&gt;An alternative approach relevant to your 2nd Q, &quot;Are there ways of looking at confidence intervals, at least in some circumstances, which would be meaningful to users of statistics?&quot;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should take a look at &lt;a href=&quot;http://en.wikipedia.org/wiki/Bayesian_statistics&quot; rel=&quot;nofollow&quot;&gt;Bayesian inference&lt;/a&gt; and the resulting &lt;a href=&quot;http://en.wikipedia.org/wiki/Credible_interval&quot; rel=&quot;nofollow&quot;&gt;&lt;em&gt;credible&lt;/em&gt; intervals&lt;/a&gt;. A 95% credible interval &lt;em&gt;can&lt;/em&gt; be interpreted as an interval which you believe has 95% probability of including the true parameter value. The price you pay is that you need to put a &lt;a href=&quot;http://en.wikipedia.org/wiki/Prior_probability&quot; rel=&quot;nofollow&quot;&gt;prior probability&lt;/a&gt; distribution on the values you believe the true parameter is likely to take &lt;em&gt;before&lt;/em&gt; collecting the data. And &lt;em&gt;your&lt;/em&gt; prior may differ from someone else's prior, so your resulting credible intervals may also differ even when you use the same data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is only my quick and crude attempt to summarise! A good recent textbook with a practical focus is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Andrew Gelman, John B. Carlin, Hal S. Stern and Donald B. Rubin. &quot;Bayesian Data Analysis&quot; (2nd edition). Chapman &amp;amp; Hall/CRC, 2003. ISBN &lt;a href=&quot;http://en.wikipedia.org/wiki/Special%3aBookSources/9781584883883&quot; rel=&quot;nofollow&quot;&gt;978-1584883883&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-10-23T08:45:56.277" CommentCount="4" />
  <row Id="3915" PostTypeId="1" CreationDate="2010-10-23T11:19:34.040" Score="4" ViewCount="259" Body="&lt;p&gt;I am trying to assess a 20-item multliple choice test.  I want to perform an item analysis such as can be found in &lt;a href=&quot;http://www.utexas.edu/academic/ctl/assessment/iar/students/report/itemanalysis-example.php&quot; rel=&quot;nofollow&quot;&gt;this example&lt;/a&gt;.  So for each question I want the P-value and the correlation with the total, and the distribution of the options selected.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't know anything about the various statistical software packages out there, but I'd like to use R as I'm comfortable with programming and R is open source.  The pseudo-workflow I envision is:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;prepare data in excel and export to CSV &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;load data in R&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;load a package that does what I need&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;execute that package's commands &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;export and report.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I am confident with 1 and 2 but having trouble with 3, probably because I don't have the statistical vocabulary to compare the packages I browsed on CRAN.  &lt;code&gt;ltm&lt;/code&gt; looks like it could be the right package but I can't tell.  Whatever package is used, what would the commands be?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Side question: in the linked example, what do you suppose MC and MI stand for?&lt;/p&gt;&#xA;" OwnerUserId="1681" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-25T13:55:00.963" LastActivityDate="2010-10-25T13:55:00.963" Title="Item Analysis for an R newbie" Tags="&lt;r&gt;&lt;correlation&gt;&lt;psychometrics&gt;&lt;scales&gt;" AnswerCount="2" />
  <row Id="3916" PostTypeId="2" ParentId="3915" CreationDate="2010-10-23T11:59:38.010" Score="5" ViewCount="" Body="&lt;p&gt;I can suggest you at least two packages that allow to perform these tasks: &lt;a href=&quot;http://cran.r-project.org/web/packages/psych/index.html&quot; rel=&quot;nofollow&quot;&gt;psych&lt;/a&gt; (&lt;code&gt;score.items&lt;/code&gt;) and &lt;a href=&quot;http://cran.r-project.org/web/packages/ltm/index.html&quot; rel=&quot;nofollow&quot;&gt;ltm&lt;/a&gt; (&lt;code&gt;descript&lt;/code&gt;). The &lt;a href=&quot;http://cran.r-project.org/web/packages/CTT/index.html&quot; rel=&quot;nofollow&quot;&gt;CTT&lt;/a&gt; package seems also to process MCQ but I have no experience with it. More information can be found on W Revelle's website, &lt;a href=&quot;http://www.personality-project.org/&quot; rel=&quot;nofollow&quot;&gt;The Personality Project&lt;/a&gt;, esp. the page dedicated to &lt;a href=&quot;http://www.personality-project.org/r/r.guide.html&quot; rel=&quot;nofollow&quot;&gt;psychometrics with R&lt;/a&gt; which provides step-by-step instructions for importing, analyzing and report data. Also, the CRAN Task View on &lt;a href=&quot;http://cran.r-project.org/web/views/Psychometrics.html&quot; rel=&quot;nofollow&quot;&gt;Psychometrics&lt;/a&gt; includes many additional resources.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As described in your link, MC stands for &quot;Mean total raw score of the persons who answered the item with the correct response&quot;, and MI for &quot;Mean total score of the persons who did not answer the item with the correct response.&quot;. Point-biserial correlation (R(IT)) is also available in the &lt;code&gt;ltm&lt;/code&gt; package (&lt;code&gt;biserial.cor&lt;/code&gt;). This is basically an indicator of the discrimination power of the item (since it is the correlation of item and total score), and is related to the discrimination parameter of a 2-PL IRT model or factor loading in Factor Analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you really want to reproduce the table you show, I guess you will have to wrap some of this code with custom code, at least to output the same kind of table. I've made a &lt;a href=&quot;http://gist.github.com/642219&quot; rel=&quot;nofollow&quot;&gt;quick and dirty example&lt;/a&gt; which reproduce your table:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dat &amp;lt;- replicate(10, sample(LETTERS[1:4], 100, rep=TRUE))&#xA;dat[3,2] &amp;lt;- dat[67,5] &amp;lt;- NA&#xA;itan(dat)&#xA;&#xA;         P      R    MC    MI NC OMIT  A  B  C  D&#xA; [1,] 0.23 -0.222 2.870 2.169 23    0 23 22 32 23&#xA; [2,] 0.32 -0.378 3.062 1.985 32    1 32 20 14 33&#xA; [3,] 0.18 -0.197 2.889 2.207 18    0 18 33 22 27&#xA; [4,] 0.33 -0.467 3.212 1.896 33    0 33 18 29 20&#xA; [5,] 0.27 -0.355 3.111 2.056 27    1 27 23 23 26&#xA; [6,] 0.17 -0.269 3.118 2.169 17    0 17 25 25 33&#xA; [7,] 0.21 -0.260 3.000 2.152 21    0 21 24 25 30&#xA; [8,] 0.24 -0.337 3.125 2.079 24    0 24 32 22 22&#xA; [9,] 0.13 -0.218 3.077 2.218 13    0 13 29 33 25&#xA;[10,] 0.25 -0.379 3.200 2.040 25    0 25 25 31 19&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As these are random responses, biserial correlation and item difficulty are not very meaningful (except to check that data are truly random :). Also, it is worth checking for possible errors, since I drafted the R function in 10'...&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-23T19:56:24.563" LastActivityDate="2010-10-23T19:56:24.563" CommentCount="4" />
  <row Id="3917" PostTypeId="2" ParentId="3893" CreationDate="2010-10-23T12:53:21.263" Score="6" ViewCount="" Body="&lt;p&gt;This is a good question, but the answer is definitely no: cross-validation will not improve causal inference. If you have a mapping between symptoms and diseases, cross-validation will help to insure that your model matches their joint distribution better than if you had simply fit your model to the entire raw data set, but it can't ever tell you anything about the directionality of causation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cross-validation is very important and worth studying, but it does nothing more than prevent you from overfitting to noise in your data set. If you'd like to understand it more, I'd suggest Chapter 7 of ESL: &lt;a href=&quot;http://www-stat.stanford.edu/~hastie/Papers/ESLII.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www-stat.stanford.edu/~hastie/Papers/ESLII.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="303" LastActivityDate="2010-10-23T12:53:21.263" CommentCount="4" />
  <row Id="3918" PostTypeId="2" ParentId="3911" CreationDate="2010-10-23T13:06:45.637" Score="3" ViewCount="" Body="&lt;p&gt;To address your question directly: Suppose that you are contemplating the use of a machine to fill a cereal box with a certain amount of cereal. Obviously, you do not want to overfill/underfill the box. You want to assess the reliability of the machine. You perform a series of tests like so: (a) Use the machine to fill the box and (b) Measure the amount of cereal that is filled in the box. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using the data collected you construct a confidence interval for the amount of cereal that the machine is likely to fill in the box. This confidence interval tells us that the interval we obtained has a 95% probability that it will contain the the true amount of cereal the machine will put in the box. As you say, the interpretation of the confidence interval relies on hypothetical, unseen samples generated by the method under consideration. But, this is precisely what we want in our context. In the above context, we &lt;em&gt;will&lt;/em&gt; use the machine repeatedly to fill the box and thus we care about hypothetical, unseen realizations of the amount of cereal the machine fills in the box.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To abstract away from the above context: a confidence interval gives us a &lt;em&gt;guarantee&lt;/em&gt; that if we were to use the method under investigation (in the above example method = machine) &lt;em&gt;repeatedly&lt;/em&gt; there is a 95% probability that the confidence interval will have the true parameter.&lt;/p&gt;&#xA;" LastActivityDate="2010-10-23T13:06:45.637" CommentCount="4" />
  <row Id="3919" PostTypeId="2" ParentId="3898" CreationDate="2010-10-23T13:33:45.217" Score="1" ViewCount="" Body="&lt;p&gt;Just speaking on a practical level, in my discipline (psychology) I have never seen this done for pure factor analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That being said, the significance (fit really) of a statistical model is normally tested by the use of Structural Equation Modelling, where you attempt to reproduce the observed matrix of data from the structure you have proposed through the use of factor analysis. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The SEM, lavaan or OpenMx Packages for R will all do this. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Technically, the Chi square test will tell you if a factor model fits perfectly, but this statistic is almost always significant with any appreciable (200+) sample size. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The psych package for R also gives you the Bayesian Information Criterion as a measure of fit after you specify a factor model, but I am unsure as to how useful this is. &lt;/p&gt;&#xA;" OwnerUserId="656" LastActivityDate="2010-10-23T13:33:45.217" />
  <row Id="3920" PostTypeId="2" ParentId="3915" CreationDate="2010-10-23T13:45:54.690" Score="4" ViewCount="" Body="&lt;p&gt;The psych package function alpha produces what you are looking for.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To export this, save the object and use the xtable function to produce LaTeX markup, which can then be formatted by any LaTeX editor. &lt;/p&gt;&#xA;" OwnerUserId="656" LastActivityDate="2010-10-23T13:45:54.690" CommentCount="1" />
  <row Id="3921" PostTypeId="1" CreationDate="2010-10-23T15:04:28.670" Score="5" ViewCount="361" Body="&lt;p&gt;What are good ways to visualize set of Likert responses?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, a set of items inquiring about the importance of X to one's decisions about A, B, C, D, E, F &amp;amp; G? Is there something better than stacked bar charts?&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What should be done with responses of N/A? How might they be represented?&lt;/li&gt;&#xA;&lt;li&gt;Should the bar charts report percentages, or number of responses? (i.e. should the bars total the same length?)&lt;/li&gt;&#xA;&lt;li&gt;If percentages, should the denominator include invalid and/or N/A responses?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I have my own views, but I am looking for other people's ideas.&lt;/p&gt;&#xA;" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-03-30T18:40:07.603" LastActivityDate="2011-03-30T18:40:07.603" Title="Visualizing Likert Item Response Data" Tags="&lt;data-visualization&gt;&lt;scales&gt;&lt;likert&gt;" AnswerCount="3" FavoriteCount="5" />
  <row Id="3922" PostTypeId="2" ParentId="3921" CreationDate="2010-10-23T15:11:30.267" Score="8" ViewCount="" Body="&lt;p&gt;Stacked barcharts are generally well understood by non-statisticians, provided they are gently introduced. It is useful to scale them on a common metric (e.g. 0-100%), with a gradual color for each category if these are ordinal item (e.g. Likert). I prefer &lt;a href=&quot;http://polisci.msu.edu/jacoby/research/dotplots/tpm/Jacoby,%20Dotplots,%20TPM%20Draft.pdf&quot; rel=&quot;nofollow&quot;&gt;dotchart&lt;/a&gt; (Cleveland dot plot), when there are not too many items and no more than 3-5 responses categories. But it is really a matter of visual clarity. I generally provide % as it is a standardized measure, and only report both % and counts with non-stacked barchart. Here is an example of what I mean:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data(Environment, package=&quot;ltm&quot;)&#xA;Environment[sample(1:nrow(Environment), 10),1] &amp;lt;- NA&#xA;na.count &amp;lt;- apply(Environment, 2, function(x) sum(is.na(x)))&#xA;tab &amp;lt;- apply(Environment, 2, table)/&#xA;       apply(apply(Environment, 2, table), 2, sum)*100&#xA;dotchart(tab, xlim=c(0,100), xlab=&quot;Frequency (%)&quot;, &#xA;         sub=paste(&quot;N&quot;, nrow(Environment), sep=&quot;=&quot;))&#xA;text(100, c(2,7,12,17,22,27), rev(na.count), cex=.8)&#xA;mtext(&quot;# NA&quot;, side=3, line=0, at=100, cex=.8)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/rYZ6W.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Better rendering could be achieved with &lt;code&gt;lattice&lt;/code&gt; or &lt;code&gt;ggplot2&lt;/code&gt;. All items have the same response categories in this particular example, but in more general case we might expect different ones, so that showing all of them would not seem redundant as is the case here. It would be possible, however, to give the same color to each response category so as to facilitate reading. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I would say stacked barcharts are better when all items have the same response category, as they help to appreciate the frequency of one response modality across items:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/KZN1z.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can also think of some kind of heatmap, which is useful if there are many items with similar response category.&#xA;&lt;img src=&quot;http://i.imgur.com/EXsrf.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Missing responses (esp. when non negligible or localized on specific item/question) should be reported, ideally for each item. Generally, % of responses for each category are computed without NA. This is what is usually done in survey or psychometrics (we speak of &quot;expressed or observed responses&quot;). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;P.S.&lt;/strong&gt;&#xA;I can think of more fancy things like the picture shown below (the first one was made by hand, the second is from &lt;code&gt;ggplot2&lt;/code&gt;, &lt;code&gt;ggfluctuation(as.table(tab))&lt;/code&gt;), but I don't think it convey as accurate information as dotplot or barchart since surface variations are difficult to appreciate.&#xA;&lt;img src=&quot;http://i.imgur.com/qTq4c.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/kv5LN.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-24T08:44:48.867" LastActivityDate="2010-10-24T08:44:48.867" CommentCount="1" />
  <row Id="3923" PostTypeId="2" ParentId="1576" CreationDate="2010-10-23T15:11:36.053" Score="3" ViewCount="" Body="&lt;p&gt;For me (and i hope this is useful) factor analysis is much more useful than PCA. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Recently, i had the pleasure of analysing a scale through factor analysis. This scale (although its widely used in industry) was developed by using PCA, and to my knowledge had never been factor analysed. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When i performed the factor analysis (principal axis) i discovered that the communalities for three of the items were less than 30%, which means that over 70% of the items variance was not being analysed. PCA just transforms the data into a new combination and doesnt care about communalities. My conclusion was that the scale was not a very good one from a psychometric point of view, and i've confirmed this with a different sample. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Essentially, if you want to predict using the factors, use PCA, while if you want to understand the latent factors, use Factor Analysis.&lt;/p&gt;&#xA;" OwnerUserId="656" LastActivityDate="2010-10-23T15:11:36.053" />
  <row Id="3924" PostTypeId="1" CreationDate="2010-10-23T16:31:41.513" Score="0" ViewCount="146" Body="&lt;p&gt;Hi. Sorry if this is a very basic question, my statistics knowledge is very low!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've got a dataset which is basically a set of greyscale values in a greyscale texture (ie. 0-255). What I want to do is take the mean value, and find out what the values would be at a specified standard-deviation to the left and right of the mean. Is this possible? I presume I'd have to presume a normal distribution?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope that makes sense!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thankyou for any help,&lt;br&gt;&#xA;Dan.&lt;/p&gt;&#xA;" OwnerUserId="1683" LastEditorUserId="1683" LastEditorDisplayName="" LastEditDate="2010-10-23T16:46:21.077" LastActivityDate="2011-05-22T06:49:01.170" Title="Value based on a supplied standard-deviation" Tags="&lt;standard-deviation&gt;&lt;basic-concepts&gt;" AnswerCount="1" CommentCount="8" />
  <row Id="3925" PostTypeId="2" ParentId="3921" CreationDate="2010-10-23T18:34:23.693" Score="5" ViewCount="" Body="&lt;p&gt;I think chl's answer is great.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One thing I might add, is for the case you'd want to compare the correlation between the items.  For that you can use something like a &lt;a href=&quot;http://www.r-statistics.com/2010/04/correlation-scatter-plot-matrix-for-ordered-categorical-data/&quot; rel=&quot;nofollow&quot;&gt;Correlation scatter-plot matrix for ordered-categorical data&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/hCzCn.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(That code still needs some tweaking - but it gives the general idea...)&lt;/p&gt;&#xA;" OwnerUserId="253" LastActivityDate="2010-10-23T18:34:23.693" CommentCount="2" />
  <row Id="3926" PostTypeId="1" CreationDate="2010-10-23T19:21:33.553" Score="2" ViewCount="111" Body="&lt;p&gt;I'm looking at two plant categories (all species fall into one type or the other) at several different types of locations. Within each type of location, I have several samples, so I have an average of each type of plant as it is found in each type of location.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Within each type of location, I want to compare the average number of one type to the average number of the other type. The number of one type of plant is not necessarily independent of the other (there is only a certain amount of resources after all).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I feel as though there must be a simple test to look at this, but I can't remember what it is. For the difference in average number of ONE type across the different locations, I used an ANOVA; if there were only two locations I could have used a t-test. If I had one population type and one type of plant, and looked at it before and after some manipulation, then I would use a paired/dependent t-test... &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestions would be phenomenally helpful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" LastActivityDate="2010-12-23T06:44:25.977" Title="How to look at two different (but related) averages within a population" Tags="&lt;categorical-data&gt;&lt;mean&gt;" AnswerCount="3" FavoriteCount="1" />
  <row Id="3927" PostTypeId="2" ParentId="3926" CreationDate="2010-10-23T20:50:56.860" Score="0" ViewCount="" Body="&lt;p&gt;Assuming that you are looking at proportions you need to do a &lt;a href=&quot;http://en.wikipedia.org/wiki/Statistical_hypothesis_testing#Common_test_statistics&quot; rel=&quot;nofollow&quot;&gt;One-proportion z-test&lt;/a&gt;. The link takes you to the wiki which has common test statisitics used in various situations including the one-proportion z-test.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I have understood your situation correctly you have the following set-up. Let:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$n$ be the total number of trees in a location,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$n_a$ be the number of trees in that location of type 'a'.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, your hypothesis potentially is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$H_0: \pi_a = \pi_0$,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$H_A: \pi_a \ne \pi_0$,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\pi_a$ is the true proportion of trees of type $a$,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You are approximating $\pi_a$ with $p_a$ where:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$p_a = \frac{n_a}{n}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\pi_0$ is your hypothesis about the true proportion for that type.&lt;/p&gt;&#xA;" LastActivityDate="2010-10-23T20:50:56.860" CommentCount="2" />
  <row Id="3928" PostTypeId="1" AcceptedAnswerId="3951" CreationDate="2010-10-23T21:24:24.377" Score="6" ViewCount="143" Body="&lt;p&gt;I have developmental data collected across several grades (1-6), where each child in each grade is measured many times. I would like to be able to assess whether there are any linear or non-linear trends in the response variable across grade. Does it make sense to run a first lmer treating grade as continuous, obtain the residuals, then run a second lmer treating grade as a factor? That is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;fit1 = lmer(&#xA;   formula = response ~ (1|child)+grade_as_numeric&#xA;   , data = my_data&#xA;   , family = gaussian&#xA;)&#xA;my_data$resid = residuals(fit1)&#xA;fit2 = lmer(&#xA;   formula = resid ~ (1|child)+grade_as_factor&#xA;   , data = my_data&#xA;   , family = gaussian&#xA;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As I understand it, fit1 will tell me if there are any linear trends in the data, while fit2 will tell me if there are any non-linear trends in the data in addition to the linear trends obtained in fit1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If this is sensible, how might I apply it to a second binomial response variable given that the residuals from a binomial model are not 0/1?&lt;/p&gt;&#xA;" OwnerUserId="364" LastActivityDate="2010-10-24T17:16:01.820" Title="Assessing linearity in a mixed effects model" Tags="&lt;mixed-model&gt;&lt;binomial&gt;" AnswerCount="1" FavoriteCount="2" />
  <row Id="3929" PostTypeId="2" ParentId="3926" CreationDate="2010-10-23T21:55:19.983" Score="0" ViewCount="" Body="&lt;p&gt;For a particular plot we estimate &quot;% cover&quot; per plant species (summed across all species within a type to give the % cover for the plant type); in a forest canopy (as one example of the locations we used) this means that the total will definitely be greater than 100%, so a statistic using proportions wont work...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I say 'plants', I mean all the plants there (trees, herbs, shrubs, etc). The two categories were &quot;native&quot; and &quot;introduced/invasive&quot; so all plants will fall into one or the other.&lt;/p&gt;&#xA;" LastActivityDate="2010-10-23T21:55:19.983" CommentCount="2" />
  <row Id="3930" PostTypeId="1" CreationDate="2010-10-23T22:02:20.557" Score="5" ViewCount="335" Body="&lt;p&gt;Most standard distributions in R have a family of commands - pdf/pmf, cdf/cmf, quantile, random deviates (for example- dnorm, pnorm, qnorm, rnorm).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know it's easy enough to make use of some standard commands to reproduce these functions for the discrete uniform distributions, but is there already a preferred built-in family of functions for modeling discrete uniform distributions in R that I'm unaware of?&lt;/p&gt;&#xA;" LastActivityDate="2010-10-25T08:34:59.653" Title="Are there default functions for discrete uniform distributions in R?" Tags="&lt;r&gt;&lt;distributions&gt;&lt;uniform&gt;" AnswerCount="2" />
  <row Id="3931" PostTypeId="1" AcceptedAnswerId="3934" CreationDate="2010-10-23T22:04:57.160" Score="15" ViewCount="359" Body="&lt;p&gt;So I've been asked today in class why you divide the sum of square error with $(n-1)$ instead of with $n$, when calculating the sd.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I said I am not going to answer it in class (since I didn't wanna go into unbiased estimators), but later I wondered - &lt;strong&gt;is there&lt;/strong&gt; an intuitive explanation for this?!&lt;/p&gt;&#xA;" OwnerUserId="253" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-10-25T11:25:35.593" LastActivityDate="2010-10-25T11:25:35.593" Title="Intuitive explanation for dividing in (n-1) when calculating sd?" Tags="&lt;teaching&gt;&lt;standard-error&gt;" AnswerCount="5" CommentCount="1" FavoriteCount="1" />
  <row Id="3932" PostTypeId="2" ParentId="3931" CreationDate="2010-10-23T22:21:58.407" Score="7" ViewCount="" Body="&lt;p&gt;A common one is that the definition of variance (of a distribution) is the second moment recentered around a &lt;em&gt;known, definite&lt;/em&gt; mean, whereas the estimator uses an &lt;em&gt;estimated&lt;/em&gt; mean.  This loss of a degree of freedom (given the mean, you can reconstitute the dataset with knowledge of just $n-1$ of the data values) requires the use of $n-1$ rather than $n$ to &quot;adjust&quot; the result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Such an explanation is consistent with the estimated variances in ANOVA and variance components analysis.  It's really just a special case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The need to make &lt;em&gt;some&lt;/em&gt; adjustment that inflates the variance can, I think, be made intuitively clear with a valid argument that isn't just &lt;em&gt;ex post facto&lt;/em&gt; hand-waving.  (I recollect that Student may have made such an argument in his 1908 paper on the t-test.)  Why the adjustment to the variance should be &lt;em&gt;exactly&lt;/em&gt; a factor of $n/(n-1)$ is harder to justify, especially when you consider that the adjusted SD is &lt;em&gt;not&lt;/em&gt; an unbiased estimator.  (It is merely the square root of an unbiased estimator of the variance.  Being unbiased usually does not survive a nonlinear transformation.)  So, in fact, the correct adjustment to the SD to remove its bias is &lt;em&gt;not&lt;/em&gt; a factor of $\sqrt{n/(n-1)}$ at all!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some introductory textbooks don't even bother introducing the adjusted sd: they teach one formula (divide by $n$).  I first reacted negatively to that when teaching from such a book but grew to appreciate the wisdom: to focus on the concepts and applications, the authors strip out all inessential mathematical niceties.  It turns out that nothing is hurt and nobody is misled.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-10-23T22:21:58.407" CommentCount="3" />
  <row Id="3934" PostTypeId="2" ParentId="3931" CreationDate="2010-10-24T03:46:26.027" Score="13" ViewCount="" Body="&lt;p&gt;The standard deviation calculated with a divisor of n-1 is a standard deviation calculated from the sample as an estimate of the standard deviation of the population from which the sample was drawn. Because the observed values fall, on average, closer to the sample mean than to the population mean, the standard deviation which is calculated using deviations from the sample mean underestimates the desired standard deviation of the population. Using n-1 instead of n as the divisor corrects for that by making the result a little bit bigger. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that the correction has a larger proportional effect when n is small than when it is large, which is what we want because when n is larger the sample mean is likely to be a good estimator of the population mean.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When the sample is the whole population we use the standard deviation with n as the divisor because the sample mean &lt;em&gt;is&lt;/em&gt; population mean.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(I note parenthetically that nothing that starts with &quot;second moment recentered around a known, definite mean&quot; is going to fulfil the questioner's request for an intuitive explanation.)&lt;/p&gt;&#xA;" OwnerUserId="1679" LastActivityDate="2010-10-24T03:46:26.027" CommentCount="2" />
  <row Id="3935" PostTypeId="2" ParentId="3911" CreationDate="2010-10-24T04:23:55.957" Score="4" ViewCount="" Body="&lt;p&gt;&quot;You are correct in saying that the 95% confidence intervals are things that result from using a &lt;em&gt;method&lt;/em&gt; that works in 95% of cases, rather than any individual interval having a 95% likelihood of containing the expected value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The logical basis and interpretation of confidence limits are, even now, a matter of controversy.&quot; {David Colquhoun, 1971, Lectures on Biostatistics}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That quotation is taken from a statistics textbook published in 1971, but I would contend that it is still true in 2010. The controversy is probably most extreme in the case of confidence intervals for binomial proportions. There are many competing methods for calculating those confidence intervals, but they are all inaccurate in one or more senses and even the worst performing method has proponents among textbook authors. Even so called ‘exact’ intervals fail to yield the properties expected of confidence intervals.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a paper written for surgeons (widely known for their interest in statistics!), John Ludbrook and I argued for the routine use of confidence intervals calculated using a uniform Bayesian prior because such intervals have frequentist properties as good as any other method (on average exactly 95% coverage over all true proportions) but, importantly, much better coverage over all observed proportions (exactly 95% coverage). The paper, because of its target audience, is not terribly detailed and so it may not convince all statistician, but I am working on a follow-up paper with the full set of results and justifications.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a case where the Bayesian approach has frequentist properties as good as the frequentist approach, something that happens fairly often. The assumption of a uniform prior is not problematical because a uniform distribution of population proportions is built into every calculation of frequentist coverage that I've come across.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You ask: &quot;Are there ways of looking at confidence intervals, at least in some circumstances, which would be meaningful to users of statistics?&quot; My answer, then, is that for binomial confidence intervals one can get intervals that contain the population proportion exactly 95% of the time for all observed proportions. That is a yes. However, the conventional use of confidence intervals expects coverage for all population proportions and for that the answer is &quot;No!&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The length of the answers to your question, and the various responses to them suggests that confidence intervals are widely misunderstood. If we change our objective from coverage for all true parameter values to coverage of the true parameter value for all sample values, it might get easier because the intervals will then be shaped to be directly relevant to the observed values rather than for the performance of the &lt;em&gt;method&lt;/em&gt; per se.&lt;/p&gt;&#xA;" OwnerUserId="1679" LastActivityDate="2010-10-24T04:23:55.957" />
  <row Id="3936" PostTypeId="2" ParentId="3924" CreationDate="2010-10-24T04:47:40.530" Score="0" ViewCount="" Body="&lt;p&gt;I suspect that you want to re-scale the values so that they have a specified standard deviation. If so, then it is easy. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's assume that you have a mean of x and a standard deviation of y, and the desired standard deviation is z. Subtract x from every observation, then divide them by y so that you have a mean of zero and a standard deviation of one. Now multiply by z and add x to each value. You will end up with the original mean, x, and a standard deviation of z.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This will work whatever the distribution of the data as it is simply linearly re-scaling the x-axis of the distribution.&lt;/p&gt;&#xA;" OwnerUserId="1679" LastActivityDate="2010-10-24T04:47:40.530" />
  <row Id="3937" PostTypeId="1" CreationDate="2010-10-24T05:19:13.343" Score="0" ViewCount="70" Body="&lt;p&gt;Whenever I start a new research project, I have to dig through innumerable survey questions to find rating scales that other projects found sensitive enough to get statistically valid results. The only standard psychological measures are very expensive and usually pertain to mental illness. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I generally end up copying existing surveys, although this is generally considered fair use I don't care to dance around copyright.  Is there a StackOverflow/PatternLibrary/Best Practices compendium of this sort of information?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maybe StatExchange can be used for this...&lt;/p&gt;&#xA;" OwnerUserId="1689" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-10-24T08:48:03.697" LastActivityDate="2010-10-24T08:48:03.697" ClosedDate="2010-10-24T10:25:16.450" Title="Pattern library for psychological/political/other questions?" Tags="&lt;survey&gt;" CommentCount="5" />
  <row Id="3938" PostTypeId="2" ParentId="3926" CreationDate="2010-10-24T06:20:31.360" Score="1" ViewCount="" Body="&lt;p&gt;A location (Between) x Plant Type (Within) mixed ANOVA may do what you want.  The disadvantage of this approach is that there is no clear post-hoc to perform at each location, so you'll have to do a bonferroni correction or take some other approach (e.g. &lt;a href=&quot;http://stats.stackexchange.com/questions/575/post-hocs-for-within-subjects-tests&quot;&gt;question 1&lt;/a&gt; or &lt;a href=&quot;http://stats.stackexchange.com/questions/1266/a-non-parametric-repeated-measures-multi-way-anova-in-r&quot;&gt;question 2&lt;/a&gt;).  Since your data is sourced from percentages and is bounded at the upper and lower extremities you may want to transform the raw data first (see this &lt;a href=&quot;http://www.sportsci.org/resource/stats/counts.html&quot; rel=&quot;nofollow&quot;&gt;issue/approach&lt;/a&gt;).&lt;/p&gt;&#xA;" OwnerUserId="196" LastActivityDate="2010-10-24T06:20:31.360" />
  <row Id="3939" PostTypeId="2" ParentId="3930" CreationDate="2010-10-24T07:12:40.887" Score="2" ViewCount="" Body="&lt;p&gt;The &lt;a href=&quot;http://finzi.psych.upenn.edu/views/Distributions.html&quot; rel=&quot;nofollow&quot;&gt;CRAN Task View: Probability Distributions&lt;/a&gt; page says:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The discrete uniform distribution can be easily obtained with the basic functions. &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I guess something on the lines of this should do:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;a &amp;lt;- round(runif(1000, min=0, max=100))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As csgillespie pointed out, this is not correct...&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;a &amp;lt;- ceiling(runif(1000, min=0, max=100))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;will work though (note that the example will generate values between &lt;strong&gt;1&lt;/strong&gt; and 100, not 0 and 100)&lt;/p&gt;&#xA;" OwnerUserId="582" LastEditorUserId="582" LastEditorDisplayName="" LastEditDate="2010-10-25T08:34:59.653" LastActivityDate="2010-10-25T08:34:59.653" CommentCount="2" />
  <row Id="3940" PostTypeId="2" ParentId="3930" CreationDate="2010-10-24T07:36:16.800" Score="6" ViewCount="" Body="&lt;p&gt;As nico wrote, they're not implemented in R. Assuming we work in 1..k, those functions should look like:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For random generation:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;rdu&amp;lt;-function(n,k) sample(1:k,n,replace=T)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;PDF:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ddu&amp;lt;-function(x,k) ifelse(x&amp;gt;=1 &amp;amp; x&amp;lt;=k &amp;amp; round(x)==x,1/k,0) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;CDF:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pdu&amp;lt;-function(x,k) ifelse(x&amp;lt;1,0,ifelse(x&amp;lt;=k,floor(x)/k,1))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-10-24T07:36:16.800" CommentCount="3" />
  <row Id="3941" PostTypeId="2" ParentId="3931" CreationDate="2010-10-24T10:28:15.323" Score="1" ViewCount="" Body="&lt;p&gt;This is a total intuition, but the simplest answer is that is a correction made to make standard deviation of one-element sample undefined rather than 0.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-10-24T10:28:15.323" CommentCount="1" />
  <row Id="3942" PostTypeId="2" ParentId="3931" CreationDate="2010-10-24T11:01:31.550" Score="3" ViewCount="" Body="&lt;p&gt;You can gain a deeper understanding of the &lt;em&gt;n&lt;/em&gt;&amp;minus;1 term through geometry alone, not just why it's not &lt;em&gt;n&lt;/em&gt; but why it takes exactly this form, but you may first need to build up your intuition cope with &lt;em&gt;n&lt;/em&gt;-dimensional geometry. From there, however, it's a small step to a deeper understanding of degrees of freedom in linear models (i.e. model df &amp;amp; residual df). I think there's little doubt that &lt;a href=&quot;http://en.wikipedia.org/wiki/Ronald_Fisher&quot; rel=&quot;nofollow&quot;&gt;Fisher&lt;/a&gt; thought this way. Here's a book that builds it up gradually:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Saville DJ, Wood GR. &lt;em&gt;Statistical methods: the geometric approach&lt;/em&gt;. 3rd edition. New York: Springer-Verlag; 1991. 560 pages. &lt;a href=&quot;http://en.wikipedia.org/wiki/Special%3aBookSources/9780387975177&quot; rel=&quot;nofollow&quot;&gt;9780387975177&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Yes, 560 pages. I did say gradually.)&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-10-24T11:01:31.550" CommentCount="2" />
  <row Id="3943" PostTypeId="1" AcceptedAnswerId="3946" CreationDate="2010-10-24T13:15:49.687" Score="5" ViewCount="742" Body="&lt;p&gt;In which cases should one prefer the one over the other?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found someone who claims an advantage for Kendall, &lt;a href=&quot;http://www.rsscse.org.uk/ts/bts/noether/text.html&quot; rel=&quot;nofollow&quot;&gt;for pedagogical reasons&lt;/a&gt;, are there other reasons?&lt;/p&gt;&#xA;" OwnerUserId="253" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-10-24T15:46:10.027" LastActivityDate="2010-10-31T10:29:50.693" Title="Kendall Tau or Spearman's rho?" Tags="&lt;correlation&gt;&lt;nonparametric&gt;" AnswerCount="3" CommentCount="1" FavoriteCount="2" />
  <row Id="3944" PostTypeId="1" AcceptedAnswerId="3948" CreationDate="2010-10-24T13:58:01.527" Score="3" ViewCount="283" Body="&lt;p&gt;Reading about methods and results of statistical analysis, especially in epidemiology, I very often &lt;a href=&quot;http://www.annals.org/content/136/2/122.abstract&quot; rel=&quot;nofollow&quot;&gt;hear&lt;/a&gt; about &lt;strong&gt;&lt;em&gt;adjustment&lt;/em&gt;&lt;/strong&gt; or &lt;strong&gt;&lt;em&gt;controlling&lt;/em&gt;&lt;/strong&gt; of the models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How would you explain, to a non-statistician, the purpose of that? How do you interpret your results after controlling for certain variable?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Small walk-through in Stata or R, or a pointer to one online, would a true gem.&lt;/p&gt;&#xA;" OwnerUserId="22" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-10-24T15:26:26.470" LastActivityDate="2010-10-24T17:43:38.477" Title="Explain model adjustment, in plain English" Tags="&lt;regression&gt;&lt;modeling&gt;&lt;epidemiology&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="3945" PostTypeId="2" ParentId="3943" CreationDate="2010-10-24T14:24:27.680" Score="4" ViewCount="" Body="&lt;p&gt;I refer the honorable gentleman to &lt;a href=&quot;http://stats.stackexchange.com/questions/3730/pearsons-or-spearmans-correlation-with-non-normal-data/3744#3744&quot;&gt;my previous answer&lt;/a&gt;: &quot;...confidence intervals for Spearman’s r&lt;sub&gt;S&lt;/sub&gt; are less reliable and less interpretable than confidence intervals for Kendall’s τ-parameters&quot;, according to Kendall &amp;amp; Gibbons (1990).&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-10-24T14:24:27.680" CommentCount="2" />
  <row Id="3946" PostTypeId="2" ParentId="3943" CreationDate="2010-10-24T14:26:35.747" Score="6" ViewCount="" Body="&lt;p&gt;I found that Spearman correlation is mostly used in place of usual linear correlation when working with integer valued scores on a measurement scale, when it has a moderate number of possible scores or when we don't want to make rely on assumptions about the bivariate relationships. As compared to Pearson coefficient, the interpretation of Kendall's tau seems to me less direct than that of Spearman's rho, in the sense that it quantifies the difference between the % of concordant and discordant pairs among all possible pairwise events. In my understanding, Kendall's tau more closely resembles &lt;a href=&quot;http://en.wikipedia.org/wiki/Gamma_test_%28statistics%29&quot; rel=&quot;nofollow&quot;&gt;Goodman-Kruskal Gamma&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I just browsed an article from Larry Winner in the J. Statistics Educ. (2006) which discusses the use of both measures, &lt;a href=&quot;http://www.amstat.org/publications/jse/v14n3/datasets.winner.html&quot; rel=&quot;nofollow&quot;&gt;NASCAR Winston Cup Race Results for 1975-2003&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also found &lt;a href=&quot;http://stats.stackexchange.com/questions/3730/pearsons-or-spearmans-correlation-with-non-normal-data/3744#3744&quot;&gt;@onestop&lt;/a&gt; answer about &lt;a href=&quot;http://stats.stackexchange.com/questions/3730/pearsons-or-spearmans-correlation-with-non-normal-data&quot;&gt;Pearson's or Spearman's correlation with non-normal data&lt;/a&gt; interesting in this respect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of note, Kendall's tau (the &lt;em&gt;a&lt;/em&gt; version) has connection to Somers' D (and Harrell's C) used for predictive modelling (see e.g., &lt;a href=&quot;http://www.imperial.ac.uk/nhli/r.newson/miscdocs/intsomd1.pdf&quot; rel=&quot;nofollow&quot;&gt;Interpretation of Somers’ D under four simple models&lt;/a&gt; by RB Newson and reference &lt;a href=&quot;http://www.jstatsoft.org/v15/i01/paper&quot; rel=&quot;nofollow&quot;&gt;6&lt;/a&gt; therein, and articles by Newson published in the Stata Journal 2006). An overview of rank-sum tests is provided in &lt;a href=&quot;http://www.jstatsoft.org/v15/i01/paper&quot; rel=&quot;nofollow&quot;&gt;Efficient Calculation of Jackknife Confidence Intervals for Rank Statistics&lt;/a&gt;, that was published in the JSS (2006).&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-31T10:29:50.693" LastActivityDate="2010-10-31T10:29:50.693" CommentCount="1" />
  <row Id="3947" PostTypeId="1" AcceptedAnswerId="3954" CreationDate="2010-10-24T15:11:52.427" Score="13" ViewCount="365" Body="&lt;p&gt;I understand the basics of what a Support Vector Machines' aim is in terms of classifying an input set into several different classes, but what I don't understand is some of the nitty-gritty details. For starters, I'm a bit confused by the use of Slack Variables. What is their purpose?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm doing a classification problem where I've captured pressure readings from sensors I've placed on the insole of a shoe. A subject will sit, stand, and walk for a couple of minutes while pressure data is recorded. I want to train a classifier to be able to determine whether a person is sitting, standing or walking and be able to do that for any future test data. What classifier type do I need to try? What is the best way for me to train a classifier from the data I've captured? I have 1000 entries for sitting, standing and walking (3x1000=3000 total), and they all have the following feature vector form. (pressurefromsensor1, pressurefromsensor2, pressurefromsensor3, pressurefromsensor4)&lt;/p&gt;&#xA;" OwnerUserId="1224" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-10-26T19:45:29.510" LastActivityDate="2010-10-26T19:45:29.510" Title="Help me understand Support Vector Machines" Tags="&lt;machine-learning&gt;&lt;classification&gt;&lt;svm&gt;" AnswerCount="1" FavoriteCount="7" />
  <row Id="3948" PostTypeId="2" ParentId="3944" CreationDate="2010-10-24T15:20:40.330" Score="9" ViewCount="" Body="&lt;p&gt;Easiest to explain by way of an example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Imagine study finds that people who watched the World Cup final were more likely to suffer a heart attack during the match or in the subsequent 24 hours than those who didn't watch it. Should the government ban football from TV? But men are more likely to watch football than women, and men are also more likely to have a heart attack than women. So the &lt;em&gt;association&lt;/em&gt; between football-watching and heart attacks might be explained by a &lt;em&gt;third factor&lt;/em&gt; such as sex that affects both. (Sociologists would distinguish here between &lt;em&gt;gender&lt;/em&gt;, a cultural construct that is associated with football-watching, and &lt;em&gt;sex&lt;/em&gt;, a biological category that is associated with heart-attack incidence, but the two are cleary very strongly correlated so i'm going to ignore that distinction for simplicity.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Statisticians, and especially epidemiologists, call such a third factor a &lt;em&gt;confounder&lt;/em&gt;, and the phenomenon &lt;em&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Confounding&quot; rel=&quot;nofollow&quot;&gt;confounding&lt;/a&gt;&lt;/em&gt;. The most obvious way to remove the problem is to look at the association between football-watching and heart-attack incidence in men and women separately, or in the jargon, to &lt;em&gt;stratify&lt;/em&gt; by sex. If we find that the association (if there still is one) is similar in both sexes, we may then choose to combine the two estimates of the association across the two sexes. The resulting estimate of the association between football-watching and heart-attack incidence is then said to be &lt;em&gt;adjusted&lt;/em&gt; or &lt;em&gt;controlled&lt;/em&gt; for sex.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We would probably also wish to control for other factors in the same way. Age is another obvious one (in fact epidemiologists either stratify or adjust/control almost every association by age and sex). Socio-economic class is probably another. Others can get trickier, e.g. should we adjust for beer consumption while watching the match? Maybe yes, if we're interested in the effect of the stress of watching the match alone; but maybe no, if we're considering banning broadcasting of World Cup football and that would also reduce beer consumption. Whether given variable is a confounder or not depends on precisely what question we wish to address, and this can require very careful thought and get quite tricky and even contentious.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Clearly then, we may wish to adjust/control for several factors, some of which may be measured in several categories (e.g. social class) while others may be continuous (e.g. age). We could deal with the continuous ones by splitting into (age-)groups, thereby turning them into categorical ones. So say we have 2 sexes, 5 social class groups and 7 age groups. We can now look at the association between football-watching and heart-attack incidence in 2×5×7 = 70 strata. But if our study is fairly small, so some of those strata contain very few people, we're going to run into problems with this approach. And in practice we may wish to adjust for a dozen or more variables. An alternative way of adjusting/controlling for variables that is particularly useful when there are many of them is provided by &lt;em&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Regression_model&quot; rel=&quot;nofollow&quot;&gt;regression analysis&lt;/a&gt;&lt;/em&gt; with multiple dependent variables, sometimes known as &lt;em&gt;multivariable&lt;/em&gt; regression analysis. (There are different types of regression models depending on the type of outcome variable: least squares regression, logistic regression, proportional hazards (Cox) regression...). In observational studies, as opposed to experiments, we nearly always want to adjust for many potential confounders, so in practice adjustment/control for confounders is often done by regression analysis, though there are other alternatives too though, such as standardization, weighting, propensity score matching...&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-10-24T15:20:40.330" CommentCount="1" />
  <row Id="3949" PostTypeId="2" ParentId="3943" CreationDate="2010-10-24T15:57:33.700" Score="4" ViewCount="" Body="&lt;p&gt;Again somewhat philosophical answer; the basic difference is that Spearman's Rho is an attempt to extend R^2 (=&quot;variance explained&quot;) idea over nonlinear interactions, while Kendall's Tau is rather intended to be a test statistic for nonlinear correlation test.&#xA;So, Tau should be used for testing nonlinear correlations, Rho as R extension (or for people familiar with R^2 -- explaining Tau to unsuspecting audience in limited time is painful).  &lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-10-24T15:57:33.700" />
  <row Id="3950" PostTypeId="1" AcceptedAnswerId="3956" CreationDate="2010-10-24T16:34:57.410" Score="4" ViewCount="70" Body="&lt;p&gt;I am currently working with feature-vectors that are made up of continuous attributes, so I can use the euclidean distance for things like KNN-classification and clustering. Now I want to add a nominal attribute that has a special distance function defined. What options do I have of combining these distance functions, so I still obtain one distance for two vectors?&lt;/p&gt;&#xA;" OwnerUserId="977" LastActivityDate="2010-10-24T20:03:13.467" Title="What options are there to combine different distance functions?" Tags="&lt;distance-functions&gt;" AnswerCount="1" FavoriteCount="2" />
  <row Id="3951" PostTypeId="2" ParentId="3928" CreationDate="2010-10-24T17:16:01.820" Score="4" ViewCount="" Body="&lt;p&gt;Not really an answer, but I was interested in trying it out...&#xA;I assume that the pattern is not easily recognisable just by plotting? So I tried to make up some data that might behave this way:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(69)&#xA;id&amp;lt;- rep(1:20, each=6)&#xA;x&amp;lt;-rep(1:6, 20)&#xA;y&amp;lt;-jitter(x+id/5, factor=5) + jitter(sin(x), factor=5)&#xA;df1&amp;lt;-data.frame(id, x, y)&#xA;&#xA;plot(y~x)&#xA;xyplot(y~x|id, data=df1, type=&quot;l&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For me, if I had this data without knowing how it was made, I think I would have trouble picking out the overlaid signal and perhaps assume it was linear. Resid vs fitted of lmer1 (below) doesn't show much, but the resid vs x is more suggestive.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;lmer1&amp;lt;-lmer(y~x+(1|id), data=df1, REML=F)&#xA;xyplot(resid(lmer1)~fitted(lmer1), type=c(&quot;p&quot;, &quot;smooth&quot;))&#xA;xyplot(resid(lmer1)~x, type=c(&quot;p&quot;, &quot;smooth&quot;))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Using your suggestion of conducting an ANOVA on the residuals gives a significant effect, indicating perhaps some kind of systematic difference: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;lmer2&amp;lt;-lmer(resid(lmer1)~factor(x)+(1|id), data=df1, REML=F) #sd attributed to id is 0&#xA;anova(lmer2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So perhaps this method may be useful to determine whether you need to include higher order terms by using, maybe, increasing order polynomials:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;lmer3.1 &amp;lt;- lmer(y~poly(x,2)+(1|id), data=df1, REML=F)&#xA;lmer3.2 &amp;lt;- lmer(y~poly(x,3)+(1|id), data=df1, REML=F)&#xA;lmer3.3 &amp;lt;- lmer(y~poly(x,4)+(1|id), data=df1, REML=F)&#xA;anova(lmer1, lmer3.1, lmer3.2, lmer3.3)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In this method the cubic function 'wins' and might be a useful approximation of what's going on and comes quite close to the generated model:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;lmer4&amp;lt;-lmer(y~x+sin(x)+(1|id), data=df1)&#xA;anova(lmer3.2, lmer4)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I don't really know if this helps or not, but hopefully this simulated data mirrors your problem and somebody can use this to give a more exact answer to your question. &#xA;I'm not sure about the binomial part of your question.&lt;/p&gt;&#xA;" OwnerUserId="966" LastActivityDate="2010-10-24T17:16:01.820" CommentCount="2" />
  <row Id="3952" PostTypeId="1" CreationDate="2010-10-24T17:23:21.573" Score="0" ViewCount="97" Body="&lt;p&gt;I'd like to get beyond my elementary understanding of statistics (StdDev) and get some intermediate knowledge and skills. There are tons of books and on-line resources, but for gaining new skills outside my day-to-day work, I find that the discipline of a classroom works best.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In NYC there are plenty of schools and professional education institutions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any recommendations?&lt;/p&gt;&#xA;" OwnerUserId="276" LastEditorUserId="5" LastEditorDisplayName="" LastEditDate="2010-10-24T20:01:48.527" LastActivityDate="2010-10-24T20:01:48.527" CommunityOwnedDate="2010-10-24T20:00:55.253" ClosedDate="2010-10-25T10:01:36.880" Title="What classes (NYC area) would you recommend for statistical analysis?" Tags="&lt;basic-concepts&gt;" CommentCount="7" />
  <row Id="3953" PostTypeId="2" ParentId="3944" CreationDate="2010-10-24T17:43:38.477" Score="5" ViewCount="" Body="&lt;p&gt;Onestop explained it pretty well, I'll just give a simple R example with made up data. Say x is weight and y is height, and we want to find out if there's a difference between males and females:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(69)&#xA;x &amp;lt;- rep(1:10,2)&#xA;y &amp;lt;- c(jitter(1:10, factor=4), (jitter(1:10, factor=4)+2))&#xA;sex &amp;lt;- rep(c(&quot;f&quot;, &quot;m&quot;), each=10)&#xA;df1 &amp;lt;- data.frame(x,y,sex)&#xA;with(df1, plot(y~x, col=c(1,2)[sex]))&#xA;lm1 &amp;lt;- lm(y~sex, data=df1)&#xA;lm2 &amp;lt;- lm(y~sex+x, data=df1)&#xA;anova(lm1); anova(lm2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can see that without controlling for weight (in anova(lm1)) there's very little difference between the sexes, but when weight is included as a covariate (controlled for in lm2) then the difference becomes more apparent.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#In case you want to add the fitted lines to the plot&#xA;coefs2 &amp;lt;- coef(lm2)&#xA;abline(coefs2[1], coefs2[3], col=1)&#xA;abline(coefs2[1]+coefs2[2], coefs2[3], col=2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="966" LastActivityDate="2010-10-24T17:43:38.477" CommentCount="1" />
  <row Id="3954" PostTypeId="2" ParentId="3947" CreationDate="2010-10-24T17:44:34.170" Score="21" ViewCount="" Body="&lt;p&gt;I think you are trying to start from a bad end. What one should know about SVM to use it is just that this algorithm is finding a hyperplane in hyperspace of attributes that separates two classes best, where best means with biggest margin between classes (the knowledge how it is done is your enemy here, because it blurs the overall picture), as illustrated by a famous picture like this:&#xA;&lt;img src=&quot;http://i.imgur.com/v5DDG.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, there are some problems left.&lt;br&gt;&#xA;First of all, what to with those nasty outliers laying shamelessly in a center of cloud of points of a different class?&lt;br&gt;&#xA;&lt;img src=&quot;http://i.imgur.com/WWNkm.png&quot; alt=&quot;alt text&quot;&gt;&lt;br&gt;&#xA;To this end we allow the optimizer to leave certain samples mislabelled, yet punish each of such examples. To avoid multiobjective opimization, penalties for mislabelled cases are merged with margin size with an use of additional parameter C which controls the balance among those aims.&lt;br&gt;&#xA;Next, sometimes the problem is just not linear and no good hyperplane can be found. Here, we introduce kernel trick -- we just project the original, nonlinear space to a higher dimensional one with some nonlinear transformation, of course defined by a bunch of additional parameters, hoping that in the resulting space the problem will be suitable for  a plain SVM:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/A0wtW.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Yet again, with some math and we can see that this whole transformation procedure can be elegantly hidden by modifying objective function by replacing dot product of objects with so-called kernel function.&lt;br&gt;&#xA;Finally, this all works for 2 classes, and you have 3; what to do with it? Here we create 3 2-class classifiers (sitting -- no sitting, standing -- no standing, walking -- no walking) and in classification combine those with voting. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ok, so problems seems solved, but we have to select kernel (here we consult with our intuition and pick RBF) and fit at least few parameters (C+kernel). And we must have overfit-safe objective function for it, for instance error approximation from cross-validation. So we leave computer working on that, go for a coffee, come back and see that there are some optimal parameters. Great! Now we just start nested cross-validation to have error approximation and voila. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This brief workflow is of course too simplified to be fully correct, but shows reasons why I think you should first try with &lt;a href=&quot;http://en.wikipedia.org/wiki/Random_forest&quot; rel=&quot;nofollow&quot;&gt;random forest&lt;/a&gt;, which is almost parameter-independent, natively multiclass, provides unbiased error estimate and perform almost as good as well fitted SVMs.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-10-24T17:44:34.170" CommentCount="1" />
  <row Id="3955" PostTypeId="1" CreationDate="2010-10-24T19:55:39.883" Score="14" ViewCount="333" Body="&lt;h3&gt;Context:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;I have a group of websites where I record the number of visits on a daily basis:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;W0 = { 30, 34, 28, 30, 16, 13, 8, 4, 0, 5, 2, 2, 1, 2, .. } &#xA;W1 = { 1, 3, 21, 12, 10, 20, 15, 43, 22, 25, .. }&#xA;W2 = { 0, 0, 4, 2, 2, 5, 3, 30, 50, 30, 30, 25, 40, .. } &#xA;...&#xA;Wn &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h3&gt;General Question:&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;How do I determine which sites are the most active?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;By this I mean receiving more visits or having a sudden increase in visits during the last few days. For illustration purposes, in the small example above W0 would be initially popular but is starting to show abandoning, W1 is showing a steady popularity (with some isolated peak), and W3 an important raise after a quiet start). &lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Initial thoughts:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;I found this &lt;a href=&quot;http://stackoverflow.com/questions/1635703/understanding-algorithms-for-measuring-trends&quot;&gt;thread on SO&lt;/a&gt; where a simple formula is described:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;&#xA;// pageviews for most recent day &lt;br/&gt;&#xA;y2 = pageviews[-1] &lt;br/&gt;&#xA;// pageviews for previous day &lt;br/&gt;&#xA;y1 = pageviews[-2] &lt;br/&gt;&#xA;// Simple baseline trend algorithm &lt;br/&gt;&#xA;slope = y2 - y1 &lt;br/&gt;&#xA;trend = slope  * log(1.0 +int(total_pageviews)) &lt;br/&gt;&#xA;error = 1.0/sqrt(int(total_pageviews)) &lt;br/&gt;&#xA;return trend, error &lt;br/&gt;&#xA;&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This looks good and easy enough, but I'm having a problem with it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The calculation is based on slopes. This is fine and is one of the features I'm interested in, but IMHO it has problems for non-monotonic series. Imagine that during some days we have a constant number of visits (so the slope = 0), then the above trend would be zero. &lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Questions:&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;How do I handle both cases (monotonic increase/decrease) and large number of hits? &lt;/li&gt;&#xA;&lt;li&gt;Should I use separate formulas?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="1694" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-02-25T03:01:17.720" LastActivityDate="2011-02-26T14:46:45.880" Title="Determining whether a website is active using daily visits" Tags="&lt;time-series&gt;&lt;forecasting&gt;" AnswerCount="3" CommentCount="5" FavoriteCount="1" />
  <row Id="3956" PostTypeId="2" ParentId="3950" CreationDate="2010-10-24T20:03:13.467" Score="5" ViewCount="" Body="&lt;p&gt;I can think of three:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Combine them in a linear manner ($d=d_1+\alpha d_2$) and find best $\alpha$ by some optimization, let's say minimizing CV error for kNN or minimizing silhouette for clustering.&lt;/li&gt;&#xA;&lt;li&gt;Train separate classifiers / cluster the data few times based on both distances and then blend the results. This may not work too well because you have only 2 base methods.&lt;/li&gt;&#xA;&lt;li&gt;For classification only, you may use &quot;klNN&quot; -- get $k$ neighbors based on first metric and $l$ based on the second.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-10-24T20:03:13.467" />
  <row Id="3958" PostTypeId="1" CreationDate="2010-10-24T20:23:32.503" Score="1" ViewCount="186" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;Possible Duplicate:&lt;/strong&gt;&lt;br&gt;&#xA;  &lt;a href=&quot;http://stats.stackexchange.com/questions/3/what-are-some-valuable-statistical-analysis-open-source-projects&quot;&gt;What are some valuable Statistical Analysis open source projects?&lt;/a&gt;  &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&#xA;&#xA;&lt;p&gt;I understand the basics of statistical analysis, but I am not good at math.  What is the best website or software package (preferably FOSS) that can analysize the data, find correlations, levels of confidence, etc. automagically/with a great UI?&lt;/p&gt;&#xA;" OwnerUserId="1689" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-10-24T23:15:48.183" LastActivityDate="2010-10-24T23:15:48.183" ClosedDate="2010-10-24T21:24:43.057" Title="What is your favorite, easy to use statistical analysis website or software package?" Tags="&lt;software&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="1" />
  <row Id="3959" PostTypeId="2" ParentId="3958" CreationDate="2010-10-24T20:48:31.153" Score="3" ViewCount="" Body="&lt;p&gt;Your objectives seem rather vague, but I think the open-source &lt;a href=&quot;http://www.R-project.org/&quot; rel=&quot;nofollow&quot;&gt;R statistical package&lt;/a&gt; should fit your needs, and beyond. Although primarily a command line driven software, you will find several useful GUIs, e.g. &lt;a href=&quot;http://socserv.mcmaster.ca/jfox/Misc/Rcmdr/&quot; rel=&quot;nofollow&quot;&gt;Rcommander&lt;/a&gt; or &lt;a href=&quot;http://www.deducer.org/&quot; rel=&quot;nofollow&quot;&gt;deducer&lt;/a&gt; to help you start with.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;a href=&quot;http://cran.r-project.org/&quot; rel=&quot;nofollow&quot;&gt;CRAN&lt;/a&gt; website contains everything you need to start with R, including a lot of &lt;a href=&quot;http://cran.r-project.org/manuals.html&quot; rel=&quot;nofollow&quot;&gt;official&lt;/a&gt; and &lt;a href=&quot;http://cran.r-project.org/other-docs.html&quot; rel=&quot;nofollow&quot;&gt;contributed&lt;/a&gt; documentation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;R is made of several additional packages (a kind of extensions to the core statistical functions), and you will find interesting pointers on these related questions: &lt;a href=&quot;http://stats.stackexchange.com/questions/73/what-r-packages-do-you-find-most-useful-in-your-daily-work&quot;&gt;What R packages do you find most useful in your daily work?&lt;/a&gt;, &lt;a href=&quot;http://stats.stackexchange.com/questions/1676/i-just-installed-the-latest-version-of-r-what-packages-should-i-obtain&quot;&gt;I just installed the latest version of R. What packages should I obtain?&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-10-24T20:48:31.153" CommunityOwnedDate="2010-10-24T21:18:47.680" />
  <row Id="3960" PostTypeId="2" ParentId="3958" CreationDate="2010-10-24T21:15:53.470" Score="0" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://rapidminer.com/&quot; rel=&quot;nofollow&quot;&gt;RapidMiner&lt;/a&gt; is a nice GUI based, workflow data mining tool. It's open source and runs on mac, linux, windows. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think R and RapidMiner will end up as the predominant tools, with R being for people that like command-line style linux-like work, and RapidMiner for Windows and Mac types that prefer GUI based work. &lt;/p&gt;&#xA;" OwnerUserId="74" LastActivityDate="2010-10-24T21:15:53.470" />
  <row Id="3961" PostTypeId="1" AcceptedAnswerId="3971" CreationDate="2010-10-24T23:27:39.733" Score="3" ViewCount="214" Body="&lt;p&gt;Suppose you have an $n$-vector $X$. For a fixed real number, $r$ between $-1$ and $1$, can one generate a random permutation of the integers $1,2,\ldots,n$, call it $i_1,i_2,\ldots,i_n$ such that the vector $X$ and the vector $\tilde{X}$ defined by $\tilde{X_j} = X_{i_j}$ have expected sample correlation of $r$?  I am looking for a process that generates such permutations. Without loss of generality, I believe one may assume $X$ has zero sample mean, and unit sample standard deviation. &lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-10-25T20:04:31.760" Title="Random Permutation of a Vector with a Fixed Expected Sample Correlation to the Original?" Tags="&lt;correlation&gt;&lt;sampling&gt;&lt;permutation&gt;&lt;monte-carlo&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="1" />
  <row Id="3963" PostTypeId="1" AcceptedAnswerId="3969" CreationDate="2010-10-25T03:31:56.330" Score="3" ViewCount="98" Body="&lt;p&gt;Below is part of the proof of large deviations result. K is cumulant generating function. Can anyone explain how the last step follows? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://yaroslavvb.com/upload/large-dev.png&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is page 157 of McCullagh's &quot;Tensor Methods in Statistics&quot;&lt;/p&gt;&#xA;" OwnerUserId="511" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-10-25T06:47:04.867" LastActivityDate="2010-10-25T14:33:53.893" Title="Large deviations proof question" Tags="&lt;probability&gt;&lt;proof&gt;" AnswerCount="1" />
  <row Id="3965" PostTypeId="1" AcceptedAnswerId="4001" CreationDate="2010-10-25T08:52:48.763" Score="3" ViewCount="296" Body="&lt;p&gt;Hello there, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;i am working with a set of distributions. I have so far analyzed several probability distributions with respect to each other, that means i have made a t-test, to see how probable it is that two events happen at the same time according to those distributions.&#xA;That means i have computed:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;let x1 and x2 be probability distributions    &#xA;prob(x1=Z, x2=Z), z be any value&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;thats what the t-test gave me. though i used something similar so the distributions can have different variances. I have now computed the overlap of both distributions with respect to each other.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;what i want to do now is look at the whole thing backwards. I want to look at a certain range and be able to give a probability that there is 1 or two of those occurrences.&#xA;perhaps another way to look at it is using dice. I have two dice with differing probability distributions (probably gaussian for me, but lets assume one throws 6 more often, the other one 3 more often).  The first way to look at it would be to tell whether both dice throw the same number. What i want to do is to tell if both dice throw a number in a certain range, for example 5 and 6. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;let x1 and x2 be probability distributions    &#xA;prob(x1=Z, x2=Z), z be a value a certain range &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I think it would help if you could tell me what the name of the thing is that i am trying to do, so i can read up. But i would also be happy to have a solution that works out of the box.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;any pointers how i can compute this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;cheers and thanks for advice&lt;/p&gt;&#xA;&#xA;&lt;p&gt;edit:&#xA;provided an example.&lt;/p&gt;&#xA;" OwnerUserId="1516" LastEditorUserId="1516" LastEditorDisplayName="" LastEditDate="2010-10-26T11:16:36.720" LastActivityDate="2010-10-26T17:17:14.607" Title="Overlapping probability distributions" Tags="&lt;distributions&gt;&lt;probability&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="3966" PostTypeId="2" ParentId="3931" CreationDate="2010-10-25T09:51:10.803" Score="2" ViewCount="" Body="&lt;p&gt;Sample variance can be thought of to be the exact mean of the pairwise &quot;energy&quot; $(x_i-x_j)^2/2$ between all sample points. The definition of sample variance then becomes&#xA;$$ s^2 = \frac{2}{n(n-1)}\sum_{i&amp;lt; j}\frac{(x_i-x_j)^2}{2} = \frac{1}{n-1}\sum_{i=1}^n(x_i-\bar{x})^2 .$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This also agrees with defining variance of a random variable as the expectation of the pairwise energy, i.e. let $X$ and $Y$ be independent random variables with the same distribution, then&#xA;$$ V(X) = E\left(\frac{(X-Y)^2}{2}\right) = E((X-E(X))^2) . $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To go from the random variable defintion of variance to the defintion of sample variance is a matter of estimating a expectation by a mean which is can be justified by the philosophical principle of typicality: The sample is a typical representation the distribution. (Note, this is related to, but not the same as estimation by moments.)&lt;/p&gt;&#xA;" LastActivityDate="2010-10-25T09:51:10.803" />
  <row Id="3967" PostTypeId="1" CreationDate="2010-10-25T10:14:17.670" Score="7" ViewCount="299" Body="&lt;p&gt;I'm a pharmacologist and, in my experience, almost all papers in basic biomedical research use Student's t-test (either to support inference or to conform to expectations...). A couple of years ago it came to my attention that Student's t-test is not the most efficient test that might be used: sequential tests offer much more power for any sample size, or a far smaller sample size on average for equivalent power.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sequential procedures of varying complexity are used in clinical research but I've &lt;em&gt;never&lt;/em&gt; seen one used in a basic biomedical research publication. I note that they are also absent from the introductory level statistics textbooks that are all that most basic scientists are likely to see.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is three-fold:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Given the very substantial efficiency advantage of sequential tests, why are they not more widely used? &lt;/li&gt;&#xA;&lt;li&gt;Is there a drawback associated with the used of sequential methods that would mean that their use by non-statisticians is to be discouraged? &lt;/li&gt;&#xA;&lt;li&gt;Are statistics students taught about sequential testing procedures?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="1679" LastEditorUserId="1679" LastEditorDisplayName="" LastEditDate="2010-10-26T00:58:33.383" LastActivityDate="2010-10-29T22:21:19.530" Title="Sequential hypothesis testing in basic science" Tags="&lt;hypothesis-testing&gt;&lt;teaching&gt;&lt;statistical-significance&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="2" />
  <row Id="3969" PostTypeId="2" ParentId="3963" CreationDate="2010-10-25T14:33:53.893" Score="8" ViewCount="" Body="&lt;p&gt;Think back to the proof of Chebychev's inequality, and you'll be home free.  The RHS of the next-to-the-last equality is $P[\exp (\xi (X - x)) \geq 1]$.  Now think of when $X$ has a PDF $f$ (but this is not required);  the probability is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\int_{\exp (\xi (u - x))\geq 1} 1 \cdot f(u) \mathrm{d}u&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, over the region of integration the quantity $\exp (\xi (u - x)) \geq 1$, so by monotonicity we get that the above integral is at most&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\int_{\exp (\xi (u - x))\geq 1} \exp (\xi (u - x)) \cdot f(u) \mathrm{d}u&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and since the integrand is nonnegative this last integral is at most&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\int \exp (\xi (u - x)) \cdot f(u) \mathrm{d}u&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;which is exactly $\exp[ -\xi x]$ multiplied by the moment generating function of $X$, denoted $M(\xi)$.  Finally, recognize $M = \exp (K)$ and we're finished.  By the way, the infimum holds because the inequality is true for every $\xi &gt;0$.&lt;/p&gt;&#xA;" OwnerUserId="1108" LastActivityDate="2010-10-25T14:33:53.893" />
  <row Id="3970" PostTypeId="2" ParentId="3893" CreationDate="2010-10-25T14:39:37.197" Score="1" ViewCount="" Body="&lt;p&gt;Hello,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I guess this is an intuitive way to think about the relation between CV and causal inference: (please correct if I am wrong)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I always think about CV as a way to evaluate the performance of a model in predictions. However, in causal inference we are more concerned with something equivalent to Occam's Razor (parsimony), hence CV won't help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="1307" LastActivityDate="2010-10-25T14:39:37.197" CommentCount="3" />
  <row Id="3971" PostTypeId="2" ParentId="3961" CreationDate="2010-10-25T14:46:32.913" Score="5" ViewCount="" Body="&lt;p&gt;The answers are no, not for all $r$ in general; yes, for a restricted range of $r$ that is readily computed; but there remain a wide set of choices to be made.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I will use a standard notation where the action of a permutation $\sigma$ is written $ X^\sigma_i = X_{\sigma (i)}$ and the set of all permutations of the $n$ coordinates is $S_n$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you note in the question, upon standardizing $X$ it suffices to investigate $\mathbb{E}[{X^\sigma}'X]$.  Because $X'X = 1$, a correlation of $r = 1$ is certainly attainable by means of the identity permutation $\epsilon$ (where $\epsilon(i) = i$ for all $i$).  However, for any given $X$ there is a minimum attainable correlation: it is realized by associating the $k^\text{th}$ smallest component of $X^\sigma$ with the $k^\text{th}$ largest component of $X$.  For example, with $X = (-2,1,1)/\sqrt{6}$ the smallest possible correlation of $-1/2$ is achieved by $X^\sigma = (1,1,-2)/\sqrt{6}$.  Let's call this minimum correlation $r_{min}(X)$ and let $\sigma_{min}(X)$ be any permutation achieving this minimum value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Every&lt;/em&gt; possible expected correlation of value between $r_{min}(X)$ and $1$ is attainable by means of a distribution supported on just $\sigma_{min}$ and $\epsilon$.  Specifically, set&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p = \frac{r - r_{min}}{1 - r_{min}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and generate the permutation $\sigma_{min}$ with probability $1 - p$ and the permutation $\epsilon$ with probability $p$.  (If $r_{min} = 1$ this formula is undefined but there's nothing to do anyway.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suspect you would like a more &quot;interesting&quot; distribution of permutations than this.  To create this you will need to add more conditions.  Here's one way to frame your problem: to every permutation $\sigma$ corresponds the number $f(\sigma) = {X^\sigma}'X$. An arbitrary probability distribution over the permutations assigns a non-negative value $p(\sigma)$ to each permutation according to the axioms of probability.  The expectation of $f$, which is the expected correlation between $X$ and $X^\sigma$, of course equals&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\sum_{\sigma \in S_n}{p(\sigma)f(\sigma)}.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given a desired expected correlation $r$, you therefore have freedom to choose the $n!$ values $p(\sigma)$ subject to the conditions&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\sum_{\sigma \in S_n}{p(\sigma)} = 1,$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\sum_{\sigma \in S_n}{p(\sigma)f(\sigma)} = r,$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p(\sigma) \ge 0 \text{ for all } \sigma \in S_n.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have merely demonstrated that this linear program is feasible if and only if $r_{min} \le r \le 1$.  You are free to choose among the solutions (a convex set of distributions) in any way you like.  For instance, you might prefer to use as uniform a choice of permutations as possible, in which case you might seek to minimize the variance of the $p(\sigma)$ (thought of just as a set of numbers) subject to the preceding conditions.  That's a &lt;em&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Quadratic_programming&quot; rel=&quot;nofollow&quot;&gt;quadratic program&lt;/a&gt;&lt;/em&gt;, for which there are many good solution methods and much available software.  Solving this (exactly) will become problematic once $n$ exceeds about $8$ or so, because it involves $n!$ variables and you'll just overwhelm the software.  In such cases you might want to restrict the distributions further, such as requiring them to be only cyclic and anti-cyclic permutations of the sorted coordinates (just $2n$ variables).  Another possibility is to choose a bunch of permutations randomly--making sure to include the order-reversing permutation among them so the minimum correlation can be included--and then finding an approximately uniform distribution among them.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-10-25T14:52:57.817" LastActivityDate="2010-10-25T14:52:57.817" CommentCount="6" />
  <row Id="3972" PostTypeId="2" ParentId="3857" CreationDate="2010-10-25T14:59:22.130" Score="1" ViewCount="" Body="&lt;p&gt;I just came across this webpage hosted by ICPSR on &lt;a href=&quot;http://www.icpsr.umich.edu/icpsrweb/ICPSR/dmp/index.jsp&quot; rel=&quot;nofollow&quot;&gt;data management plans&lt;/a&gt;. Although I think the goals of ICPSR will be somewhat different than your business (e.g. they are heavily interested in making the data readily able to be disseminated without violating confidentiality), I imagine they have useful information to businesses. Particularly advice on creating metadata seems to me to be universal. &lt;/p&gt;&#xA;" OwnerUserId="1036" LastActivityDate="2010-10-25T14:59:22.130" CommunityOwnedDate="2010-10-25T14:59:22.130" />
  <row Id="3973" PostTypeId="2" ParentId="3857" CreationDate="2010-10-25T15:54:27.227" Score="1" ViewCount="" Body="&lt;p&gt;In the case of a much smaller scales, I experienced using dropbox fora sharing/syncing a copy of the data files (and scripts and results) with other researchers/collaborators (I wrote about it &lt;a href=&quot;http://www.r-statistics.com/2010/05/syncing-files-across-computers-using-dropbox/&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The other tool I have used is google docs for collecting and sharing data (about which I &lt;a href=&quot;http://www.r-statistics.com/2010/03/google-spreadsheets-google-forms-r-easily-collecting-and-importing-data-for-analysis/&quot; rel=&quot;nofollow&quot;&gt;wrote here&lt;/a&gt;)&lt;/p&gt;&#xA;" OwnerUserId="253" LastActivityDate="2010-10-25T15:54:27.227" CommunityOwnedDate="2010-10-25T15:54:27.227" />
  <row Id="3974" PostTypeId="1" CreationDate="2010-10-25T16:39:19.197" Score="5" ViewCount="127" Body="&lt;p&gt;I have (trivariate: multivariate with three variables) data that appears to be good empirical and reasonable theoretical fit for a (univariate) convolution of an exponential and a normal distribution (some times called exp-norm or exGauss distributions).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My data are samples from the joint distribution:  J(R,G,B)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It appears that the marginals of R,G,B follow:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$R=R_N+R_E$ with $R_N \sim N(\mu_R,\sigma^2_R)$ and $R_E \sim Exp(\lambda_R)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(and likewise for G,B).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to effectively summarize the marginal, conditional, and joint distributions.  The main purpose of the summary is to compare these distributions (marginal, conditional, and joint) with other distributions generated in a like manner.  My difficulty is that (1) I don't know a form for the joint (or the conditional) distribution and (2) following from that, I have no parameters to estimate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, I need either a distribution free (non-parametric) approach to working with this data or I need to figure out a multivariate distribution that matches the data and the form of the marginals.  Or should I think about other options?&lt;/p&gt;&#xA;" OwnerUserId="1704" LastEditorUserId="1704" LastEditorDisplayName="" LastEditDate="2010-10-26T19:01:47.800" LastActivityDate="2010-10-26T19:01:47.800" Title="From Marginal Exp-Norm Distributions to What Conditionals and Joint?" Tags="&lt;distributions&gt;&lt;multivariable&gt;" AnswerCount="1" CommentCount="4" />
  <row Id="3975" PostTypeId="2" ParentId="3921" CreationDate="2010-10-25T17:40:25.237" Score="9" ViewCount="" Body="&lt;p&gt;I like the centered count view. This particular version removes the neutral answers (effectively treating neutral and n/a as the same) to show only the amount of agree/disagree opinions. The 0 point is where red and blue meet. The count axis is clipped out.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/3lV2o.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For comparison, here are the same five responses as stacked percentages, showing both neutral (gray) and no answer (white).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/A6Mmx.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="1191" LastActivityDate="2010-10-25T17:40:25.237" CommentCount="5" />
  <row Id="3976" PostTypeId="2" ParentId="3974" CreationDate="2010-10-25T17:50:35.663" Score="2" ViewCount="" Body="&lt;p&gt;Would &lt;a href=&quot;http://en.wikipedia.org/wiki/Copula_%28statistics%29&quot; rel=&quot;nofollow&quot;&gt;copulas&lt;/a&gt; be any use here? I don't know enough about them, or your problem, to be sure.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-10-25T17:50:35.663" CommentCount="6" />
  <row Id="3977" PostTypeId="1" AcceptedAnswerId="3978" CreationDate="2010-10-25T20:17:25.713" Score="0" ViewCount="131" Body="&lt;p&gt;M or MM? I prefer &quot;M&quot; -- it's shorter, but someone here was pushing for &quot;MM&quot;.&lt;/p&gt;&#xA;" OwnerUserId="1531" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-06-03T18:12:42.273" LastActivityDate="2011-06-03T18:12:42.273" ClosedDate="2010-10-25T23:52:05.623" Title="How do you abbreviate &quot;Millions&quot;?" Tags="&lt;terminology&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="3978" PostTypeId="2" ParentId="3977" CreationDate="2010-10-25T21:13:34.243" Score="0" ViewCount="" Body="&lt;p&gt;I would use M, as it is the SI prefix for million (mega-). &lt;/p&gt;&#xA;" OwnerUserId="582" LastActivityDate="2010-10-25T21:13:34.243" />
  <row Id="3979" PostTypeId="2" ParentId="3977" CreationDate="2010-10-25T21:39:42.333" Score="0" ViewCount="" Body="&lt;p&gt;I would use &quot;million&quot;, or 10^6, because M stands for molar (concentration in moles per litre) in my discipline.&lt;/p&gt;&#xA;" OwnerUserId="1679" LastActivityDate="2010-10-25T21:39:42.333" CommentCount="1" />
  <row Id="3980" PostTypeId="1" AcceptedAnswerId="4021" CreationDate="2010-10-26T00:24:40.997" Score="4" ViewCount="88" Body="&lt;p&gt;I have some data corresponding to changes in a binomial variable - i.e, in month 1 there were n1 trials and k1 successes, and in month 2 there were n2 trials and k2 successes.  Say I have M of these cases, and in between month 1 and month 2 there were performed a number of different operations (so for case 1 we might have tried treatments a and b, and for case 2 b,c,and d), each of which could have increased or decreased the success rate.  I would like to examine the effects of these treatments by regressing on the categorical covariates corresponding to the presence or absence of a,b,c,d,etc - what is the best way to go about this?  I suppose I am looking for something analogous to a binomial ancova, but using change in the dependent variable.&lt;/p&gt;&#xA;" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-10-26T06:21:20.107" LastActivityDate="2010-10-27T14:04:15.370" Title="What distribution to use to model changes in ratios?" Tags="&lt;time-series&gt;&lt;regression&gt;&lt;binomial&gt;&lt;ancova&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="3981" PostTypeId="2" ParentId="3749" CreationDate="2010-10-26T05:40:40.813" Score="4" ViewCount="" Body="&lt;p&gt;Hello Matt, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think a few methods that can be used, but not designed specifically for you, are as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Modeling approaches:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Topic Models (used to find patters in a set of documents and/or information retrieval)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;a.  Simplest one is LDA&lt;/p&gt;&#xA;&#xA;&lt;p&gt;b.  Dynamic topic models (IMHO, most suited for your case, without much domain knowledge)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;c. Correlated topic models (IMHO, if 2. is not good, it makes sense to try this)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These approaches are not used in finance (I am not aware, as I don't work in specifically in finance), but they have very general applicability. They use the latent variable formulation, which is very similar to that of HMM. They have shown to be state-of-the-art in topic modeling. You can watch a nice presentation by David Blei (great presenter, apart from his awesome!! research) &lt;a href=&quot;http://video.google.com/videoplay?docid=3077213787166426672#&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;. The specific references, the slides for the presentation, and more complicated models can be accessed from his &lt;a href=&quot;http://www.cs.princeton.edu/~blei/topicmodeling.html&quot; rel=&quot;nofollow&quot;&gt;website&lt;/a&gt;. He is doing some great work which is very general, so it might not be surprising if he has already done something in finance. Another great reference in the same field is his advisor, &lt;a href=&quot;http://www.cs.berkeley.edu/~jordan/publications.html&quot; rel=&quot;nofollow&quot;&gt;Michael Jordan's&lt;/a&gt;, website. Its hard to find specific references there as he publishes so much!&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Time series and sequential data models (HMM specifically)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Apart from Jordan and Blei, the other prolific research is Zoubin Ghahramani (and his coauthor Beal). You can find &lt;a href=&quot;http://learning.eng.cam.ac.uk/zoubin/timeseries.html&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; the specific HMM models that you require. A few impressive ones are: The infinite hidden markov models, Time sensitive Dirichlet Process Mixture Models. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Software&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a R package called &lt;a href=&quot;http://cran.r-project.org/web/packages/lda/&quot; rel=&quot;nofollow&quot;&gt;lda&lt;/a&gt; and topicmodels for most of the &quot;good&quot; models. Blei and Ghahramani maintain C, Matlab codes on their website as well.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Good Luck!&lt;/p&gt;&#xA;" OwnerUserId="1307" LastActivityDate="2010-10-26T06:19:55.590" CommentCount="4" />
  <row Id="3982" PostTypeId="1" CreationDate="2010-10-26T05:56:12.743" Score="3" ViewCount="345" Body="&lt;pre&gt;&lt;code&gt;var w = 810,&#xA;    h = 400,&#xA;    mapMargin = 30;&#xA;&#xA;geo = pv.Geo.scale().range(w, h);&#xA;&#xA;var vis = new pv.Panel()&#xA;    .width(w)&#xA;    .height(h)&#xA;    .top(50)&#xA;    .bottom(30)&#xA;    .def(&quot;i&quot;, -1);&#xA;&#xA;&#xA;var dot = vis.add(pv.Dot)&#xA;    .data(geoPopList)&#xA;    .left(function(d) {return geo(d.center).x})&#xA;    .top(function(d) {return geo(d.center).y})&#xA;    .radius(function(d) {return areaToRadius(d.speakers)})&#xA;    .fillStyle(function(d){return col(d.speakers)})&#xA;    .strokeStyle(&quot;#ffffff&quot;);&#xA;&#xA;vis.render();&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm using above code to make Dot Chart in Protovis. How at add zoom and pan to this? I saw an example at &lt;a href=&quot;http://vis.stanford.edu/protovis/ex/transform.html&quot; rel=&quot;nofollow&quot;&gt;http://vis.stanford.edu/protovis/ex/transform.html&lt;/a&gt;. But problem with my code is how to transform on geo scale.&lt;/p&gt;&#xA;" OwnerUserId="1706" LastEditorUserId="5" LastEditorDisplayName="" LastEditDate="2010-10-26T13:52:10.313" LastActivityDate="2010-10-26T13:52:10.313" Title="Adding Zoom &amp; Pan for Protovis Dot Chart with GeoScale" Tags="&lt;data-visualization&gt;&lt;protovis&gt;&lt;javascript&gt;" CommentCount="7" />
  <row Id="3983" PostTypeId="2" ParentId="2691" CreationDate="2010-10-26T06:25:04.147" Score="1" ViewCount="" Body="&lt;p&gt;Here is the link to &quot;Analysing ecological data&quot; by Alain F. Zuur, Elena N. Ieno, Graham M. Smith, where the example with the overhead-projector and the hand is given:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://books.google.de/books?id=mmPvf-l7xFEC&amp;amp;lpg=PA15&amp;amp;ots=b_5iizOr3p&amp;amp;dq=Zuur%20et%20al%20in%20Analyzing%20ecological%20data&amp;amp;hl=en&amp;amp;pg=PA194#v=onepage&amp;amp;q&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;http://books.google.de/books?id=mmPvf-l7xFEC&amp;amp;lpg=PA15&amp;amp;ots=b_5iizOr3p&amp;amp;dq=Zuur%20et%20al%20in%20Analyzing%20ecological%20data&amp;amp;hl=en&amp;amp;pg=PA194#v=onepage&amp;amp;q&amp;amp;f=false&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="230" LastActivityDate="2010-10-26T06:25:04.147" />
  <row Id="3984" PostTypeId="1" CreationDate="2010-10-26T07:28:19.323" Score="4" ViewCount="610" Body="&lt;p&gt;I am running a logistic model. The actual model dataset has more than 100 variables but I am choosing a test data set in which there are around 25 variables. Before that I also made a dataset which had 8-9 variables. I am being told that AIC and SC values can be used to compare the model. I observed that the model had higher SC values even when the variable had low p values ( ex. 0053) . To my intuition a model which has variables having good significance level should result in low SC and AIC values. But that isnt happening.&#xA;Can someone please clarify this. In short I want to ask the following questions:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) does the number of variable have anything to do with SC AIC ?&#xA;2) should i concentrate on p values or low SC AIC values ?&#xA;3) what are the typical ways of reducing SC AIC values ?&lt;/p&gt;&#xA;" OwnerUserId="1763" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-26T08:16:20.113" LastActivityDate="2010-10-27T20:31:24.907" Title="AIC and SC value" Tags="&lt;logistic-regression&gt;&lt;aic&gt;" AnswerCount="2" />
  <row Id="3986" PostTypeId="2" ParentId="3984" CreationDate="2010-10-26T08:15:47.120" Score="7" ViewCount="" Body="&lt;p&gt;It is quite difficult to answer your question in a precise manner, but it seems to me you are comparing two criteria (information criteria and p-value) that don't give the same information. For all information criteria (AIC, or Schwarz criterion), the smaller they are the better the fit of your model is (from a statistical perspective) as they reflect a trade-off between the lack of fit and the number of parameters in the model; for example, the Akaike criterion reads $-2\log(\ell)+2k$, where $k$ is the number of parameters. However, unlike AIC, SC is consistent: the probability of choosing incorrectly a bigger model converges to 0 as the sample size increases. They are used for comparing models, but you can well observe a model with significant predictors that provide poor fit (large residual deviance). If you can achieve a different model with a lower AIC, this is suggestive of a poor model. And, if your sample size is large, $p$-values can still be low which doesn't give much information about model fit. At least, look if the AIC shows a significant decrease when comparing the model with an intercept only and the model with covariates. However, if your interest lies in finding the best subset of predictors, you definitively have to look at methods for variable selection.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would suggest to look at &lt;em&gt;penalized regression&lt;/em&gt;, which allows to perform variable selection to avoid overfitting issues. This is discussed in Frank Harrell's Regression Modeling Strategies (p. 207 ff.), or Moons et al., &lt;a href=&quot;http://www.jclinepi.com/article/S0895-4356%2804%2900171-4/abstract&quot; rel=&quot;nofollow&quot;&gt;Penalized maximum likelihood estimation to directly adjust diagnostic and prognostic prediction models for overoptimism: a clinical example&lt;/a&gt;, J Clin Epid (2004) 57(12). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;See also the &lt;a href=&quot;http://cran.r-project.org/web/packages/Design/index.html&quot; rel=&quot;nofollow&quot;&gt;Design&lt;/a&gt; (&lt;code&gt;lrm&lt;/code&gt;) and &lt;a href=&quot;http://cran.r-project.org/web/packages/stepPlr/index.html&quot; rel=&quot;nofollow&quot;&gt;stepPlr&lt;/a&gt; (&lt;code&gt;step.plr&lt;/code&gt;) R packages, or the &lt;a href=&quot;http://cran.r-project.org/web/packages/penalized/index.html&quot; rel=&quot;nofollow&quot;&gt;penalized&lt;/a&gt; package. You may browse related questions on &lt;em&gt;variable selection&lt;/em&gt; on this SE.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-27T09:44:59.743" LastActivityDate="2010-10-27T09:44:59.743" CommentCount="4" />
  <row Id="3987" PostTypeId="2" ParentId="3967" CreationDate="2010-10-26T09:13:52.530" Score="3" ViewCount="" Body="&lt;p&gt;I don't know much of sequential tests and their application outside of interim analysis (Jennison and Turnbull, 2000) and computerized adaptive testing (van der Linden and Glas, 2010). One exception is in some fMRI studies that are associated to large costs and difficulty to enroll subjects. Basically, in this case sequential testing primarily aims at stopping the experiment earlier. So, I am not surprised that these very tailored approaches are not taught in usual statistical classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sequential tests are not without their pitfalls, though (type I and II error have to be specified in advance, choice of the stopping rule and multiple look at results should be justified, p-values are not uniformly distributed under the null as in a fixed sample design, etc.). In most design, we work with a pre-specified experimental setting or a preliminary power study was carried out, to optimize some kind of cost-effectiveness criterion, in which case standard testing procedures apply.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found, however, the following paper from Maik Dierkes about fixed &lt;em&gt;vs.&lt;/em&gt; open sample design very interesting: &lt;a href=&quot;http://www.wiwi.uni-muenster.de/fcm/downloads/forschen/2008_A_claim_for_sequential_designs_of_experiments.pdf&quot; rel=&quot;nofollow&quot;&gt;A claim for sequential designs of experiments&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-29T22:21:19.530" LastActivityDate="2010-10-29T22:21:19.530" CommentCount="6" />
  <row Id="3989" PostTypeId="1" AcceptedAnswerId="3992" CreationDate="2010-10-26T09:34:08.073" Score="4" ViewCount="180" Body="&lt;p&gt;Let Qn = Cqn.{|Xi-Xj|;i &amp;lt; j}_(k) so for a very short sample like {1,3,6,2,7,5} it can be calculated from finding the kth static order of pairwise differences:  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;7 6 5 3 2 1&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;1&lt;/strong&gt; 6 5 4 2 1&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;2&lt;/strong&gt; 5 4 3 1&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;3&lt;/strong&gt; 4 3 2&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;5&lt;/strong&gt; 2 1&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;6&lt;/strong&gt; 1&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;7&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;h=[n/2]+1=4&lt;/p&gt;&#xA;&#xA;&lt;p&gt;k=h(h-1)/2=8&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus Qn=Cqn. 2&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Obviously for large samples saying consist of 80,000 records we need very large memory.&#xA;Is there anyway to calculate Qn in 1D space instead of 2D?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A link to the answer &lt;a href=&quot;ftp://ftp.win.ua.ac.be/pub/preprints/92/Timeff92.pdf&quot; rel=&quot;nofollow&quot;&gt;ftp://ftp.win.ua.ac.be/pub/preprints/92/Timeff92.pdf&lt;/a&gt;&#xA;although I cannot fully understand it.&lt;/p&gt;&#xA;" OwnerUserId="1637" LastEditorUserId="1637" LastEditorDisplayName="" LastEditDate="2010-10-26T10:07:35.697" LastActivityDate="2010-11-09T13:06:41.067" Title="How to calculate Rousseeuw’s and Croux’ (1993) Qn scale estimator for large samples?" Tags="&lt;data-transformation&gt;&lt;scales&gt;&lt;robust&gt;&lt;optimal-scaling&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="1" />
  <row Id="3992" PostTypeId="2" ParentId="3989" CreationDate="2010-10-26T10:20:01.583" Score="5" ViewCount="" Body="&lt;p&gt;No, $O(n\log(n))$ is the lower theoretical bound (see (1)) for selecting the $k^{th}$ element among all $\frac{n(n-1)}{2}$ possible $|x_i - x_j|: 1 \leq i \lt j \leq n$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can get $O(1)$ space, but only by naively checking all combinations of $x_i-x_j$ in time $O(n^2)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The good news is that you can use the $\tau$ estimator of scale (see (2) and (3) for an improved version and some timing comparisons), implemented in the function &#xA;&lt;strong&gt;scaleTau2()&lt;/strong&gt; in the &lt;strong&gt;R&lt;/strong&gt; package &lt;a href=&quot;http://cran.r-project.org/web/packages/robustbase/index.html3.&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;robustbase&lt;/strong&gt;&lt;/a&gt;. The univariate $\tau$ estimator is an iterative &lt;strong&gt;MM&lt;/strong&gt; estimator of scale. It starts at the &lt;strong&gt;mad&lt;/strong&gt; value and typically reaches the optimum in two iterations. It has 95 percent Gaussian efficiency, 50 percent breakdown point, and complexity of $O(n)$ time and $O(1)$ space (plus it can easily be made 'online', shaving off half the computational costs in repeated use -- although you will have to dig into the &lt;strong&gt;R&lt;/strong&gt; code to implement this option, it is rather straightforward to do).&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The complexity of selection and ranking in X + Y and matrices with sorted columns&#xA;G. N. Frederickson and D. B. Johnson, Journal of Computer and System Sciences&#xA;Volume 24, Issue 2, April 1982, Pages 197-208.&lt;/li&gt;&#xA;&lt;li&gt;Yohai, V. and Zamar, R. (1988). High breakdown point estimates of regression by means of the minimization of an efficient scale. Journal of the American Statistical Association 83 406–413.&lt;/li&gt;&#xA;&lt;li&gt;Maronna, R. and Zamar, R. (2002). Robust estimates of location and dispersion for high-&#xA;dimensional data sets. Technometrics 44 307–317&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit&lt;/strong&gt; To use this &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Fire up R (its free and can be downloaded from &lt;a href=&quot;http://www.freestatistics.org/cran/2.&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;install the package by tipping &lt;/p&gt;&#xA;&#xA;&lt;p&gt;install.packages(&quot;robustbase&quot;)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;load the package by tipping &lt;/p&gt;&#xA;&#xA;&lt;p&gt;library(&quot;robustbase&quot;)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;load your data file and run the function:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;mydatavector&amp;lt;-read.table(&quot;adress to my file in text format&quot;,header=T)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;scaleTau2(mydatavector)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="603" LastEditorUserId="603" LastEditorDisplayName="" LastEditDate="2010-11-09T13:06:41.067" LastActivityDate="2010-11-09T13:06:41.067" CommentCount="7" />
  <row Id="3993" PostTypeId="2" ParentId="3893" CreationDate="2010-10-26T10:20:31.147" Score="5" ViewCount="" Body="&lt;p&gt;It seems to me that your question more generally addresses different flavour of validation for a predictive model: Cross-validation has somewhat more to do with &lt;em&gt;internal validity&lt;/em&gt;, or at least the initial modelling stage, whereas drawing causal links on a wider population is more related to &lt;em&gt;external validity&lt;/em&gt;. By that (and as an update following @Brett's nice remark), I mean that we usually build a model on a working sample, assuming an hypothetical conceptual model (i.e. we specify the relationships between predictors and the outcome(s) of interest), and we try to obtain reliable estimates with a minimal classification error rate or a minimal prediction error. Hopefully, the better the model performs, the better it will allow us to predict outcome(s) on unseen data; still, CV doesn't tell anything about the &quot;validity&quot; or adequacy of the hypothesized causal links. We could certainly achieve decent results with a model where some moderation and/or mediation effects are neglected or simply not known in advance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My point is that whatever the method you use to validate your model (and holdout method is certainly not the best one, but still it is widely used in epidemiological study to alleviate the problems arising from stepwise model building), you work with the same sample (which we assume is representative of a larger population). On the contrary, generalizing the results and the causal links inferred this way to new samples or a plausibly related population is usually done by &lt;em&gt;replication studies&lt;/em&gt;. This ensures that we can safely test the predictive ability of our model in a &quot;superpopulation&quot; which features a larger range of individual variations and may exhibit other potential factors of interest.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your model might provide valid predictions for your working sample, and it includes all potential confounders you may have think of; however, it is possible that it will not perform as well with new data, just because other factors appear in the intervening causal path that were not identified when building the initial model. This may happen if some of the predictors and the causal links inferred therefrom depend on the particular trial centre where patients were recruited, for example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In genetic epidemiology, many &lt;a href=&quot;http://en.wikipedia.org/wiki/Genome-wide_association_study&quot; rel=&quot;nofollow&quot;&gt;genome-wide association studies&lt;/a&gt; fail to replicate just because we are trying to model complex diseases with an oversimplified view on causal relationships between DNA markers and the observed phenotype, while it is very likely that gene-gene (epistasis), gene-diseases (pleiotropy), gene-environment, and population substructure all come into play, but see for example &lt;a href=&quot;http://www.nature.com/nrg/journal/v10/n5/abs/nrg2544.html&quot; rel=&quot;nofollow&quot;&gt;Validating, augmenting and refining genome-wide association signals&lt;/a&gt; (Ioannidis et al., Nature Reviews Genetics, 2009 10). So, we can build-up a performant model to account for the observed cross-variations between a set of genetic markers (with very low and sparse effect size) and a multivariate pattern of observed phenotypes (e.g., volume of white/gray matter or localized activities in the brain as observed through fMRI, responses to neuropsychological assessment or personality inventory), still it won't perform as expected on an independent sample.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for a general reference on this topic, can recommend chapter 17 and Part III of &lt;a href=&quot;http://www.clinicalpredictionmodels.org/&quot; rel=&quot;nofollow&quot;&gt;Clinical Prediction Models&lt;/a&gt;, from EW Steyerberg (Springer, 2009). I also like the following article from Ioannidis:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Ioannidis, JPA, &lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1182327/&quot; rel=&quot;nofollow&quot;&gt;Why Most Published&#xA;  Research Findings Are False?&lt;/a&gt; PLoS&#xA;  Med. 2005 2(8): e124&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-27T20:34:55.987" LastActivityDate="2010-10-27T20:34:55.987" CommentCount="8" />
  <row Id="3997" PostTypeId="1" CreationDate="2010-10-26T15:10:55.440" Score="3" ViewCount="420" Body="&lt;p&gt;I have a dataset with a very simple logfile-like structure, I want to subset the data according to date ranges but can only do on one parameter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;my data looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;            date_time loc_id node  energy   kgco2 &#xA;1 2009-02-27 00:11:08     87  103 0.00000 0.00000 &#xA;2 2009-02-27 01:05:05     87  103 7.00000 3.75900 &#xA;3 2009-02-27 02:05:05     87  103 6.40039 3.43701 &#xA;4 2009-02-27 03:05:05     87  103 4.79883 2.57697 &#xA;5 2009-02-27 04:05:05     87  103 4.10156 2.20254 &#xA;6 2009-02-27 05:05:05     87  103 2.59961 1.39599&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;the file includes data for a a whole year, I want to create summary plots for every month and perhaps week&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am processing the date_time as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; dt &amp;lt;-as.POSIXlt(ae$date_time)&#xA;&amp;gt; ae$dt &amp;lt;- dt&#xA;&amp;gt; names(ae$dt)&#xA;[1] &quot;sec&quot;   &quot;min&quot;   &quot;hour&quot;  &quot;mday&quot;  &quot;mon&quot;   &quot;year&quot;  &quot;wday&quot;  &quot;yday&quot;  &quot;isdst&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;now I'm trying to subset the data as:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; x &amp;lt;- ae$energy[ae$dt$year==&quot;110&quot; &amp;amp; ae$dt$mon==&quot;10&quot;]&#xA;&amp;gt; x&#xA;numeric(0)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&quot;110&quot; is because the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; range(ae$dt$year)&#xA;[1] 109 110&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have also tried the following with no luck:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; d &amp;lt;- subset(ae, (dt$year==&quot;110&quot; &amp;amp; dt$mon==&quot;10&quot;), select=energy)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;these however do work:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; d &amp;lt;- subset(ae, dt$year==&quot;110&quot;, select=energy)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and so does this&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; d &amp;lt;- subset(ae, dt$mon==&quot;10&quot;, select=energy)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;any ideas on how can I subset by selecting both year and month?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks,&lt;/p&gt;&#xA;" OwnerUserId="1716" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-10-26T15:22:53.140" LastActivityDate="2010-10-27T14:34:10.650" Title="Subsetting a data-frame in R based on dates" Tags="&lt;r&gt;&lt;time-series&gt;&lt;dataframe&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="3998" PostTypeId="2" ParentId="3997" CreationDate="2010-10-26T15:35:24.063" Score="3" ViewCount="" Body="&lt;p&gt;A few points:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;I'm not sure why that's happening.  Clearly the POSIXlt slots are wrong.  I typically use POSIXct unless I absolutely need to adjust the slots.&lt;/li&gt;&#xA;&lt;li&gt;One option is to use the dates directly rather than messing with the slots, and say &lt;code&gt;&amp;lt;=&lt;/code&gt; and &lt;code&gt;&amp;gt;=&lt;/code&gt; to subset.  Something like &lt;code&gt;ae[ae$date &amp;gt;= as.POSIXlt(&quot;2009-10-01&quot;) &amp;amp; ae$date &amp;lt; as.POSIXlt(&quot;2009-11-01&quot;),]&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;You should consider using a time series for this, since that's the exact purpose of that data structure (and they provide many useful functions for dealing with data over time).  One of the most common is &lt;code&gt;zoo&lt;/code&gt;.  &lt;code&gt;xts&lt;/code&gt; also includes a number of functions that can help with this kind of thing.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-10-26T15:35:24.063" CommentCount="1" />
  <row Id="3999" PostTypeId="1" AcceptedAnswerId="4003" CreationDate="2010-10-26T15:39:43.867" Score="2" ViewCount="385" Body="&lt;p&gt;Hello, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have 5 categories, each category is divided into the subcategories low, medium and high. An object can belong to one or more of these categories with a number between 1 and 100 in each subcategory but the sum for each category can no exceed 100. Is there are a way to summarise this into one single number?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any hints or directions are very welcomed.&lt;/p&gt;&#xA;" OwnerUserId="791" LastActivityDate="2010-10-27T00:29:02.957" Title="How to create an index" Tags="&lt;summary-statistics&gt;" AnswerCount="1" CommentCount="4" FavoriteCount="1" />
  <row Id="4001" PostTypeId="2" ParentId="3965" CreationDate="2010-10-26T17:17:14.607" Score="2" ViewCount="" Body="&lt;p&gt;I can't make sense of any of the statements in the question, but you're asking for terminology and that request at least I can understand and appreciate, so here's some to get you started.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I will italicize technical terms and key concepts to draw them to your attention.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The upward face of a die after it is thrown or the side of a coin that shows after it is flipped are &lt;em&gt;outcomes.&lt;/em&gt;  The assignment of numbers to outcomes, such as labeling the faces of a die with 1, 2, ..., 6 or the faces of a coin 0 and 1, is called a &lt;em&gt;random variable.&lt;/em&gt;  A range (or, generally, a set) of values for which we would like to know a probability is called an &lt;em&gt;event.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With two dice you have two random quantities and are using two random variables to analyze them.  This is a &lt;em&gt;multivariate&lt;/em&gt; setting.  Now the outcomes consist of ordered pairs: one outcome for the first variable, one outcome for the second.  These outcomes have &lt;em&gt;joint probabilities.&lt;/em&gt;  The most fundamental issue concerns whether the outcomes of one are somehow connected with the outcomes of the other.  When they are not, the two variables are &lt;em&gt;independent&lt;/em&gt;.  Throws of two dice are usually physically unrelated, so we expect them to be independent.  If you are using dice as a model for some other phenomenon, though, watch out!  You need to determine whether it is appropriate to treat your variables as independent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With two independent variables $X$ and $Y$, the &lt;em&gt;probabilities multiply.&lt;/em&gt;  Specifically, let $E$ be an event for $X$ (that is, a set of numbers that $X$ might attain) and let $F$ be an event for $Y$.  Then the &lt;em&gt;joint probability&lt;/em&gt; of $E$ and $F$, meaning the probability that $X$ has a value in $E$ (written $\Pr_X[E]$) &lt;em&gt;and&lt;/em&gt; $Y$ has a value in $F$, equals the product of the probabilities, $\Pr_X[E] \quad \Pr_Y[F]$.  When the variables are not independent, the joint probability of $E$ and $F$ may differ (considerably) from this product.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It sounds to me like you are asking about either the &lt;strong&gt;theoretical computation of joint probabilities&lt;/strong&gt; or about how to &lt;strong&gt;estimate joint probabilities from observations&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An excellent place to get up to speed quickly on all this stuff, as well as sort out what t-tests really do, what probability distributions are, and what &quot;gaussian&quot; really means, is to read through Gonick &amp;amp; Smith's &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0062731025&quot; rel=&quot;nofollow&quot;&gt;&lt;em&gt;A Cartoon Guide to Statistics&lt;/em&gt;&lt;/a&gt; (seriously!).&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-10-26T17:17:14.607" />
  <row Id="4002" PostTypeId="2" ParentId="3907" CreationDate="2010-10-26T18:18:51.977" Score="1" ViewCount="" Body="&lt;p&gt;Since the individuals were born with the 'genetic change' I would use their birth as the starting time instead of the time at which they entered the registry. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The following is my reasoning: First, ignore the effect of other variables on survival such as gender, income levels, exercise levels etc. For the sake of illustration we will assume that these variables have no differential impact on the time at which they have a cardio event. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second, I am assuming that you wish to investigate the following question: Does the difference in gene types (broadly speaking) result in a differential impact on the time it takes for a person to have a cardio event? I suppose that there is an underlying theory which states that the answer to the above question is 'yes'. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, consider two individuals both of whom had the gene change. One, enters the registry just before a cardio event and the other has a cardio event several years after entering the registry. However, for the sake of dicussion assume that the ages of both people are the same. Thus, the time to cardio event is technically the same (their age). However, if you use the starting time as the time at which they enter the registry you would draw different conclusions about how long it takes for the cardio event to happen which would bias your conclusions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am assuming that my interpretation of your goals and the situation is correct. Please correct me if I am wrong about some aspect. &lt;/p&gt;&#xA;" LastActivityDate="2010-10-26T18:29:36.550" CommentCount="3" />
  <row Id="4003" PostTypeId="2" ParentId="3999" CreationDate="2010-10-26T20:57:01.053" Score="2" ViewCount="" Body="&lt;p&gt;Your solution should depend on how you plan to use the information.  If, for instance, you intend to use these data as potential explanatory variables in a model, then you are better off without an index, because that is likely to cause a loss of explanatory power.  Just use the original variables.  If, on the other hand, you would like to make a map to portray degrees of urbanization in a simple manner, then an index makes sense.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What remains in the second case is to make &quot;urbanization&quot; operational.  Suppose, for the sake of exploring this issue, that instead of using the word &quot;urbanization&quot; you used some term that was completely unintelligible to me.  How would I go about finding out what it meant?  &lt;em&gt;There is nothing in the data that will reveal the answer.&lt;/em&gt;  What you need is either a quantitative definition of urbanization in terms of these 15 variables or else you need a 16th variable that correctly captures the degree of urbanization in some &quot;test&quot; or &quot;calibration&quot; cases.  Then you could statistically explore the correlations among the 15 original variables and the degree of urbanization with the aim of finding a combination of those 15 that is reliably associated with urbanization.  This can be done using &lt;a href=&quot;http://en.wikipedia.org/wiki/Canonical_correlation&quot; rel=&quot;nofollow&quot;&gt;canonical correlation analysis&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An alternative that (I suspect) is frequently used is just to dodge these fundamental considerations and make up an answer.  Some people invent &quot;weights&quot; for each of the categories, form the weighted sum of the values, and declare that value to be whatever they would like it to be, whether it's &quot;urbanization&quot; or &quot;environmental impact&quot; or whatever.  The problems of that approach in an objective or scientific context ought to be obvious, but it is an easy solution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An intermediate approach remains agnostic about what &quot;urbanization&quot; might mean and merely seeks a parsimonious description of the variables you have.  In a special situation they might all be a mixture of two extremes, allowing the vector of 15 values to be described by a single parameter (the mixture proportions).  You could take that parameter to be an &quot;index&quot; of the data and then go on to explore the extent to which it seems to agree with your idea of urbanization.  This approach is carried out using &lt;a href=&quot;http://en.wikipedia.org/wiki/Principal_component_analysis&quot; rel=&quot;nofollow&quot;&gt;Principal Components Analysis&lt;/a&gt; (PCA) or &lt;a href=&quot;http://en.wikipedia.org/wiki/Factor_analysis&quot; rel=&quot;nofollow&quot;&gt;Factor Analysis&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All these methods (canonical correlation, PCA, FA) are typically available in full-featured statistical software.  PCA has been discussed at length on &lt;a href=&quot;http://stats.stackexchange.com/q/2691/919&quot;&gt;this site&lt;/a&gt;: explore the &quot;PCA&quot; tag for more information.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-10-26T20:57:01.053" CommentCount="1" />
  <row Id="4004" PostTypeId="1" CreationDate="2010-10-26T21:40:51.110" Score="1" ViewCount="158" Body="&lt;p&gt;Does anyone know of a step-by-step guide for the practical implementation of Gifi Methods for Optimal Scaling in R: The Package homals? Although I have an OK theoretical understanding (thanks chl for directing me to articles), I'm a tech novice and some of the language/technical information is a bit too complicated for me. On that note, can I do the same thing within SPSS?? &lt;/p&gt;&#xA;" OwnerUserId="1647" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-10-27T06:50:09.687" LastActivityDate="2010-10-27T07:48:02.677" Title="How to understand optimal Scaling in R: The Package homals for novices" Tags="&lt;r&gt;&lt;pca&gt;&lt;spss&gt;" AnswerCount="2" />
  <row Id="4005" PostTypeId="1" CreationDate="2010-10-26T22:58:33.633" Score="2" ViewCount="112" Body="&lt;p&gt;I am trying work out the variance of the population estimate var (R) where R = X/Y (X = sum(x) and Y = sum(y)).  For each member of the population I know y and I have stratified the population and taken a random sample of each strata.  For each member of the sample I need to estimate x.  x is not known with certainty and the estimate of x has its own standard error (which I can estimate from a pre sample calibration exercise).  However, which I know enough to know that this error should be taken into account my knowledge of sample statistics is limited to the situation where x is a quantity that can be exactly determined.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If anyone can help with an answer or point me to anappropriate reference source it would be appreciated.&lt;/p&gt;&#xA;" LastActivityDate="2010-10-27T01:36:31.307" Title="Stratified Random Sampling of Ratio (X/Y) where X itself is an estimate" Tags="&lt;standard-error&gt;&lt;sampling&gt;" AnswerCount="1" />
  <row Id="4006" PostTypeId="2" ParentId="4004" CreationDate="2010-10-26T23:17:16.707" Score="2" ViewCount="" Body="&lt;p&gt;Check out homals &lt;a href=&quot;http://cran.r-project.org/web/packages/homals/vignettes/homals.pdf&quot; rel=&quot;nofollow&quot;&gt;vignette&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-10-26T23:17:16.707" CommentCount="2" />
  <row Id="4007" PostTypeId="2" ParentId="4004" CreationDate="2010-10-27T00:46:03.703" Score="3" ViewCount="" Body="&lt;p&gt;With regards to SPSS, I believe you need the &lt;a href=&quot;http://www.spss.com/software/statistics/categories/&quot; rel=&quot;nofollow&quot;&gt;Categories add-on module&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;ftp://ftp.spss.com/pub/web/wp/SWPOPT.pdf&quot; rel=&quot;nofollow&quot;&gt;SPSS White Paper&lt;/a&gt; has further information including lots of references.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.ccsr.ac.uk/methods/festival2004/programme/Thu/am/JCR/documents/Bell_000.ppt&quot; rel=&quot;nofollow&quot;&gt;PPT presentation&lt;/a&gt; by Richard Bell&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-10-27T07:48:02.677" LastActivityDate="2010-10-27T07:48:02.677" />
  <row Id="4008" PostTypeId="2" ParentId="4005" CreationDate="2010-10-27T01:36:31.307" Score="2" ViewCount="" Body="&lt;p&gt;I am going to translate what you said into math and answer your question. If my translation into math is not correct please clarify via the comments and I will edit my response as appropriate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$R = \frac{\sum_i{X_i}}{\sum_i{Y_i}}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$i= 1, 2, ..n$ index members of the population,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Y_i$ is a constant&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$X_i$ is a random variable with an estimated $Var(X_i)$ (this is what you call as the standard error).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Var(R) = \frac{ n Var(X_i) }{(\sum_i{Y_i})^2}$&lt;/p&gt;&#xA;" LastActivityDate="2010-10-27T01:36:31.307" />
  <row Id="4009" PostTypeId="1" CreationDate="2010-10-27T02:29:31.147" Score="2" ViewCount="868" Body="&lt;h3&gt;Initial Question:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;I am trying to calculate inter-rater reliability. Previous researchers in this area have used intraclass correlation. SPSS has options for two-way random, mixed random and one-way random models. SPSS help says choose the right one based on whether the 'people effects are random' and 'item effects are random', can anybody please explain to me what these terms mean?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also I wanted to know how to structure my data file - I think I need to put each raters' rating of each item in a separate column (rather than each rater on a separate row), is this right?&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;UPDATED question:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Thanks to the help I have received, I have visited the websites, which were useful. I have decided that I need to use the mixed model, and I have transposed my files so that each rater is a column. &lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Additivity:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;The websites also mentioned some assumptions, one of which is &lt;em&gt;additivity&lt;/em&gt;, tested with Tukey's test of non-additivity. Apparently, this tests the hypothesis that there is no multiplicative interaction between the cases and the items. I think maybe I have too many variables in my calculation, because the examples others use have a lot less, e.g. each rater making only one rating, as described below, my raters have made many more ratings than that. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What does the additivity assumption mean?&lt;/li&gt;&#xA;&lt;li&gt;What should be done when the test of non-additivity is statistically significant?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;Inter-rater reliability with multiple objects and properties&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;I will explain in more detail what I have done. I have five raters and they have rated facial and vocal expressions of participants watching different emotional films. Raters make 18 ratings per film, and there are 16 films, so there is a total of 288 variables per rater, per participant rated. Each rater has rated the same 4 participants each, and I have placed these participants into different files. I thought about making seperate files for each of the 16 films, but that would be so many files.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;How should I calculate ICC and organise my files in this specific situation?&lt;/li&gt;&#xA;&lt;li&gt;Should I calculate a mean of individual ICCs or perhaps select a random sample?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="2025" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-13T03:45:55.433" LastActivityDate="2011-04-13T03:45:55.433" Title="Inter-rater reliability using Intra-class correlation with ratings for multiple objects on multiple properties" Tags="&lt;spss&gt;&lt;reliability&gt;&lt;intraclass-correlation&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="1" />
  <row Id="4010" PostTypeId="1" CreationDate="2010-10-27T04:05:47.197" Score="0" ViewCount="77" Body="&lt;p&gt;I am now having a sequence of numbers, e.g. $x_1, x_2,\dots , x_N$, ($0\leq x_i\leq 1$), which may represent the correlation between a couple of measurements. I want to set a threshold for this sequence, that for $x_i$ which are greater than the threshold may have a significant difference with $x_j$ which are smaller than the threshold.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or in another form, assuming these values are obtained from a distribution which is unknown, I want to determine a threshold, that $\Pr(x_i &gt; \text{threshold}) = 0.1$, any non-parametric method can determine this threshold? &lt;/p&gt;&#xA;" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-27T09:21:47.860" LastActivityDate="2010-10-27T09:21:47.860" Title="Set a threshold for a sequence" Tags="&lt;correlation&gt;&lt;nonparametric&gt;" CommentCount="2" />
  <row Id="4011" PostTypeId="2" ParentId="3907" CreationDate="2010-10-27T04:32:45.037" Score="0" ViewCount="" Body="&lt;p&gt;@Skirant you lost me on your last comment, but I agree with Whuber that by using birth at start start you are distorting your sample as it doesn't take into account people with that gene change that actually already had the event or died from it. On top of it once people enter the regstry there is a chance that they change their behaviour to compensate for the higher risk. I suggest you use entry into the registry as start time and age (at entry) as a covariate or age at event as a time dependent covariate&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let me know how you go&lt;/p&gt;&#xA;" OwnerUserId="694" LastActivityDate="2010-10-27T04:32:45.037" CommentCount="4" />
  <row Id="4012" PostTypeId="2" ParentId="3907" CreationDate="2010-10-27T06:40:00.510" Score="2" ViewCount="" Body="&lt;p&gt;You need to be careful to distinguish two different &quot;start times&quot; in studies such as this:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The origin for the time variable, i.e. the point which you're calling &lt;em&gt;t&lt;/em&gt;=0 for each participant&lt;/li&gt;&#xA;&lt;li&gt;The time at which an individual  enters the study, i.e. the time from which you would record an event if one happened&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In the simple cases first taught in survival analysis, these times are assumed to be the same. For long-term cohort studies, it's usually much better to allow them to differ. The most suitable time origin for cohort studies of chronic diseases (such as cardiovascular disease here) is usually date of birth, as Srikant suggests above. That's because for chronic diseases the baseline hazard varies strongly with age. But (unless it's a birth cohort) individuals don't enter the study when they are born. If they have an event before they enter the study you wouldn't record this, and that could cause bias if you don't handle it properly by distinguishing entry time from time origin. This is sometimes called &lt;em&gt;delayed-entry&lt;/em&gt;.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-10-27T06:40:00.510" />
  <row Id="4013" PostTypeId="1" AcceptedAnswerId="4015" CreationDate="2010-10-27T07:47:44.047" Score="7" ViewCount="279" Body="&lt;p&gt;I've read through &lt;a href=&quot;http://stats.stackexchange.com/questions/665/whats-the-difference-between-probability-and-statistics&quot;&gt;this thread&lt;/a&gt;, and it looks to me like it can be said that:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;statistics = induction ?&lt;/li&gt;&#xA;&lt;li&gt;probability = deduction ?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;But I am wondering if there might be some more details on the comparison that I am missing.  For example, is statistics equal to induction, or is it just a particular case of it?  It does seem that probability is a sub case of deduction (since it is a sub case of mathematical thinking).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know this is a picky question, but in a sense this is why I am asking it - because I want to be sure how these terms can be compared accurately.&lt;/p&gt;&#xA;" OwnerUserId="253" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-10-27T22:21:11.183" LastActivityDate="2010-10-30T14:16:19.130" Title="Can you say that statistics and probability is like induction and deduction?" Tags="&lt;probability&gt;&lt;theory&gt;" AnswerCount="3" CommentCount="1" />
  <row Id="4014" PostTypeId="2" ParentId="4013" CreationDate="2010-10-27T08:18:14.943" Score="3" ViewCount="" Body="&lt;p&gt;Yes! Maybe statistics isn't strictly &lt;em&gt;equal&lt;/em&gt; to induction, but statistics is the solution to the &lt;a href=&quot;http://en.wikipedia.org/wiki/Problem_of_induction&quot; rel=&quot;nofollow&quot;&gt;problem of induction&lt;/a&gt; in my opinion.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-10-27T08:18:14.943" />
  <row Id="4015" PostTypeId="2" ParentId="4013" CreationDate="2010-10-27T08:28:20.517" Score="8" ViewCount="" Body="&lt;p&gt;I think it is the best to quickly recap the meaning of inductive and deductive reasoning before answering your question.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Deductive Reasoning: &quot;Deductive arguments are attempts to show that a conclusion necessarily follows from a set of premises. A deductive argument is valid if the conclusion does follow necessarily from the premises, i.e., if &lt;strong&gt;the conclusion must be true provided that the premises are true&lt;/strong&gt;. A deductive argument is sound if it is valid and its premises are true. Deductive arguments are valid or invalid, sound or unsound, but are never false or true.&quot; (&lt;a href=&quot;http://en.wikipedia.org/wiki/Deductive_reasoning&quot; rel=&quot;nofollow&quot;&gt;quoted from wikipedia&lt;/a&gt;, emphasis added). &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&quot;Inductive reasoning, also known as induction or inductive logic, or &lt;strong&gt;educated guess&lt;/strong&gt; in colloquial English, is a kind of reasoning that allows for the possibility that the conclusion is false even where all of the premises are true. &lt;strong&gt;The premises of an inductive logical argument indicate some degree of support (inductive probability) for the conclusion but do not entail it; that is, they do not ensure its truth.&lt;/strong&gt;&quot; (&lt;a href=&quot;http://en.wikipedia.org/wiki/Inductive_reasoning&quot; rel=&quot;nofollow&quot;&gt;from wikipedia&lt;/a&gt;, emphasis added)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;To stress the main difference: &lt;strong&gt;Whereas deductive reasoning transfers the truth from premises to conclusions, inductive reasoning does not.&lt;/strong&gt; That is, whereas for deductive reasoning you never broaden your knowledge (i.e., everything is in the premises, but sometimes hidden and needs to be demonstrated via proofs), inductive reasoning allows you to broaden your knowledge (i.e., you may gain new insights that are not already contained in the premises, however, for the cost of not knowing their truth).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How does this relate to probability and statistics?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my eyes, probability is necessarily deductive. It is a branch of math. So based on some axioms or ideas (supposedly true ones) it deduces theories.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, statistics is not necessarily inductive. Only if you try to use it for generating knowledge about unobserved entities (i.e., pursuing inferential statistics, see also onestop's answer). However, if you use statistics to describe the sample (i.e., decriptive statistics) or if you sampled the whole population, it is still deductive as you do not get any more knowledge or information as that is already present in the sample.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, if you think about statistics as being the heroic endeavor of scientists trying to use mathematical methods to find regularities that govern the interplay of the empirical entities in the world, which is in fact never successful (i.e., we will never really know if any of our theories is true), then, yeah, this is induction.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update:&lt;/strong&gt; After reading Conjugate Prior's answer (and after some overnight thinking) I would like to add something. I think the question on whether or not (inferential) statistical reasoning is deductive or inductive depends on what exactly it is that you are interested in, i.e., what kind of conclusion you are striving for.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are interested in probabilistic conclusions, then statistical reasoning is deductive. This means, if you want to know if e.g., in 95 out of 100 cases the population value is within a certain interval (i.e., confidence interval) , then you can get a truth value (true or not true) for this statement. You can say (if the assumptions are true)  that it is the case that in 95 out of 100 cases the population value is within the interval. However, in no empirical case you will know if the population value is in your obtained CI. Either it is or not, but there is no way to be sure. The same reasoning applies for probabilities in classical p-value and Bayesian statistics. You can be sure about probabilities.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, if you are interested in conclusions about empirical entities (e.g., where is the population value) you can only argue inductive. You can use all available statistical methods to accumulate evidence that support certain propositions about empirical entities or the causal mechanisms with which they interact. But you will never be certain on any of these propositions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To recap: The point I want to make that it is important at what you are looking. Probabilites you can deduce, but for every definite proposition about things you can only find evidence in favor for. Not more. See also onestop's link to the induction problem.&lt;/p&gt;&#xA;" OwnerUserId="442" LastEditorUserId="442" LastEditorDisplayName="" LastEditDate="2010-10-30T14:16:19.130" LastActivityDate="2010-10-30T14:16:19.130" CommentCount="2" />
  <row Id="4016" PostTypeId="2" ParentId="4009" CreationDate="2010-10-27T09:13:38.807" Score="7" ViewCount="" Body="&lt;p&gt;These are distinct ways of accounting for raters or items variance in overall variance, following &lt;a href=&quot;http://www3.uta.edu/faculty/ricard/COED/Shrout%20&amp;amp;%20Fleiss%20%281979%29%20Intraclass%20correlations.pdf&quot; rel=&quot;nofollow&quot;&gt;Shrout and Fleiss (1979)&lt;/a&gt; (cases 1 to 3 in Table 1):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;em&gt;One-way random effects model&lt;/em&gt;: raters are considered as sampled from a larger pool of potential raters, hence they are treated as random effects; the ICC is then interpreted as the % of total variance accounted for by subjects/items variance. This is called the consistency ICC.&lt;/li&gt;&#xA;&lt;li&gt;&lt;em&gt;Two-way random effects model&lt;/em&gt;: both factors -- raters and items/subjects -- are viewed as random effects, and we have two variance components (or mean squares) in addition to the residual variance; we further assume that raters assess all items/subjects; the ICC gives in this case the % of variance attributable to raters + items/subjects.&lt;/li&gt;&#xA;&lt;li&gt;&lt;em&gt;Two-way mixed model&lt;/em&gt;: contrary to the one-way approach, here raters are considered as fixed effects (no generalization beyond the sample at hand) but items/subjects are treated as random effects; the unit of analysis may be the individual or the average ratings.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I would say raters have to be entered as columns, although I'm not a specialist of SPSS.&#xA;Dave Garson's &lt;a href=&quot;http://faculty.chass.ncsu.edu/garson/PA765/reliab.htm&quot; rel=&quot;nofollow&quot;&gt;dedicated website&lt;/a&gt; is worth looking at for those working with SPSS. There is also a complete on-line tutorial on &lt;a href=&quot;http://www.nyu.edu/its/statistics/Docs/intracls.html&quot; rel=&quot;nofollow&quot;&gt;reliability analysis&lt;/a&gt; (Robert A. Yaffee). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For theoretical consideration about the mixed-effect approach, please consider reading my answer to this related question: &lt;a href=&quot;http://stats.stackexchange.com/questions/1015/reliability-in-elicitation-exercise/3667#3667&quot;&gt;Reliability in Elicitation Exercise&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-27T09:24:38.043" LastActivityDate="2010-10-27T09:24:38.043" />
  <row Id="4017" PostTypeId="1" CreationDate="2010-10-27T09:59:40.793" Score="6" ViewCount="462" Body="&lt;p&gt;I would like to learn to use &lt;strong&gt;Rcpp&lt;/strong&gt;. I went through the docs on the package's CRAN website, but i feel working on a practical example (second practical, considering convolve3) would be more helpfull . &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I propose the following code, from the the &lt;strong&gt;robustbase&lt;/strong&gt; package because it's neither too long nor too short, uses a combination of R types and R functions and has one of those small arithmetic iterations that are too slow in R. How would you go about Rcpp-ing it ?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;scaleTau2&amp;lt;-function (x, c1 = 4.5, c2 = 3, consistency = TRUE, mu.too = FALSE){&#xA;n &amp;lt;- length(x)&#xA;medx &amp;lt;- median(x)&#xA;x. &amp;lt;- abs(x - medx)&#xA;sigma0 &amp;lt;- median(x.)&#xA;mu &amp;lt;- if (c1 &amp;gt; 0) {&#xA;    x. &amp;lt;- x./(sigma0 * c1)&#xA;    w &amp;lt;- 1 - x. * x.&#xA;    w &amp;lt;- ((abs(w) + w)/2)^2&#xA;    sum(x * w)/sum(w)&#xA;}&#xA;else medx&#xA;x &amp;lt;- (x - mu)/sigma0&#xA;rho &amp;lt;- x^2&#xA;rho[rho &amp;gt; c2^2] &amp;lt;- c2^2&#xA;if (!identical(consistency, FALSE)) {&#xA;    Erho &amp;lt;- function(b) 2*((1-b^2)*pnorm(b)-b*dnorm(b)+b^2)-1&#xA;    Es2 &amp;lt;- function(c2) Erho(c2*qnorm(3/4))&#xA;    nEs2 &amp;lt;-ifelse(consistency == &quot;finiteSample&quot;,n-2,n)*Es2(c2)&#xA;}&#xA;else nEs2 &amp;lt;- n&#xA;c(if (mu.too) mu, sigma0 * sqrt(sum(rho)/nEs2))&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Please explain as much as you can. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt; It's really the idea of a step by step explanation of how you would go about converting a well written (and documented) R code (so at least the foundations are okay) unto an efficient implementation. The choice of the code is arguable a bit random but i think it reflects the arch-typical script on our blueprints (calls R functions that one doesn't want to translate, uses arithmetic loops....).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT2&lt;/strong&gt; from the comments i realize this may actually be a big work to do in C++ (i didn't realize it when posting the code). In regard of this, using individual pieces as pedagogical devices is ok. I'll eventually parse the pieces together by editing the question.&lt;/p&gt;&#xA;" OwnerUserId="603" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-27T19:04:31.570" LastActivityDate="2010-10-27T19:04:31.570" Title="Translate R to C++ (eventually with Rcpp)" Tags="&lt;r&gt;&lt;c++&gt;" AnswerCount="1" CommentCount="5" FavoriteCount="3" />
  <row Id="4018" PostTypeId="2" ParentId="4017" CreationDate="2010-10-27T12:17:28.647" Score="6" ViewCount="" Body="&lt;p&gt;Interesting question, but quite possibly too challenging to be discussed briefly:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;You would need a C++-side implementations of &lt;code&gt;median()&lt;/code&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The cited code from package &lt;a href=&quot;http://cran.r-project.org/package=robustbase&quot; rel=&quot;nofollow&quot;&gt;robustbase&lt;/a&gt; is highly 'R-optimised' which may not be the best starting point.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://cran.r-project.org/package=Rcpp&quot; rel=&quot;nofollow&quot;&gt;Rcpp&lt;/a&gt; is not an 'R compiler' that you toss any such function at to 'make it faster'. It is more about connecting existing C++ code, or writing new C++ code.  &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Of course the above can be translated (Turing-equivalence and all that) but that may not be the best way to learn about using &lt;a href=&quot;http://cran.r-project.org/package=Rcpp&quot; rel=&quot;nofollow&quot;&gt;Rcpp&lt;/a&gt;. I think we have simpler examples on the mailing list.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Lastly, isn't this a programming question for SO? ;-) &lt;/p&gt;&#xA;" OwnerUserId="334" LastActivityDate="2010-10-27T12:17:28.647" CommentCount="9" />
  <row Id="4019" PostTypeId="1" AcceptedAnswerId="4020" CreationDate="2010-10-27T12:56:45.837" Score="3" ViewCount="270" Body="&lt;p&gt;Hello everyone.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let us consider a comparison of two machine learning algorithms (A and B) on some dataset. Results (root mean squared error) of both algorithms depend on randomly generated initial approximation (parameters).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;When I use the same parameters for both algorithms, &quot;usually&quot; A slightly outperforms B. How many different experiments (&lt;em&gt;with different parameters /&lt;/em&gt; updated &lt;em&gt;/&lt;/em&gt;) have I to perform to make &quot;sure&quot; that A is better than B?&lt;/li&gt;&#xA;&lt;li&gt;How to measure significance of my results? (To what extent I am &quot;sure&quot;?)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Relevant links are welcome!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS. I've seen papers in which authors use t-test and p-value; but i'm not sure if it is ok to use them in a such situation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE.&lt;/strong&gt;&#xA;The problem is that A (almost) always outperforms B if initial params and learning/validation/testing sets are the same; but it doesn't neccessarily hold if they differ.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I see the following approaches here:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;split data into disjoint sets D_1, D_2, ...; generate parameters params_1; compare A(params_1, D_2, ...,) and B(params_1, D_2, ...,) on D_1;&#xA; generate  params_2; compare A(params_2, D_1, D_3,...) and B(params_2, D_1, D_3,...) on D_2 and so on.&#xA; Remember how often A outperforms B.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;split data into disjoint sets D_1, D_2, ...; generate parameters params_1a and params_1b; compare A(params_1a, D_2, ...,) and B(params_1b, D_2, ...,) on D_1; ....&#xA; Remember how often A outperforms B.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;first, do cross-validation for A. Then, independently, for B. Compare results.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Which approach is better? How to find significance of the result in this best case?&lt;/p&gt;&#xA;" OwnerUserId="1725" LastEditorUserId="1725" LastEditorDisplayName="" LastEditDate="2010-10-27T20:00:28.423" LastActivityDate="2010-10-27T20:00:28.423" Title="Measuring statistical significance of machine learning algorithms comparison" Tags="&lt;machine-learning&gt;&lt;statistical-significance&gt;" AnswerCount="1" CommentCount="6" FavoriteCount="2" />
  <row Id="4020" PostTypeId="2" ParentId="4019" CreationDate="2010-10-27T13:17:40.497" Score="3" ViewCount="" Body="&lt;ol&gt;&#xA;&lt;li&gt;You have two biases to remove here -- the selection of the initial parameters set and the selection of train/test data.&#xA;So, I don't think it is good to compare algorithms based on the same initial parameters set; I would just run the evaluation over few different initial sets for each of the algorithms to get more general approximation.&#xA;The next step is something that you are probably doing already, so using some kind of cross-validation.&lt;/li&gt;&#xA;&lt;li&gt;t-test is a way to go (I assume that you are getting this RMS as a mean from cross validation [and evaluation over few different starting parameters set, supposing you decided to use my first suggestion], so you can also calculate the standard deviation); more fancy method is to use Mann-Whitney-Wilcoxon test.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Cross-validation_%28statistics%29&quot; rel=&quot;nofollow&quot;&gt;Wikipedia article about cross validation&lt;/a&gt; is quite nice and have some references worth reading.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE AFTER UPDATE&lt;/strong&gt;: I still think that making paired test (Dikran's way) looks suspicious. &lt;/p&gt;&#xA;" OwnerUserId="88" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-10-27T16:48:32.687" LastActivityDate="2010-10-27T16:48:32.687" CommentCount="7" />
  <row Id="4021" PostTypeId="2" ParentId="3980" CreationDate="2010-10-27T14:04:15.370" Score="0" ViewCount="" Body="&lt;p&gt;You are still looking for binomial regression. You have to rearrange the data so that each observation is one set of binomial trials (separately for pre- and post), adding a &quot;time&quot; and &quot;subject&quot; variable. The effect of &quot;time&quot; on the probability of success is like the pre-post difference (on the logit scale), its interactions with the type of treatment are the differential effects of each treatment. The model will need a subject effect as well to capture the baseline of each subject. This could be a fixed or random effect. You should also watch out for overdispersion, and use a quasi-binomial model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you don't understand the above, you might need to consult a statistician in real life - there is no simple solution to your problem, and it is easy to mess up.&lt;/p&gt;&#xA;" OwnerUserId="279" LastActivityDate="2010-10-27T14:04:15.370" />
  <row Id="4022" PostTypeId="2" ParentId="3997" CreationDate="2010-10-27T14:34:10.650" Score="2" ViewCount="" Body="&lt;p&gt;ok timeseries seem to have done the trick:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;aets &amp;lt;- as.xts(read.zoo(&quot;n8_energy_actual2009_2010.csv&quot;, header=TRUE, sep=&quot;,&quot;, FUN=as.POSIXct))&#xA;eats.2010 &amp;lt;- aets[&quot;2010-01::2010-10&quot;]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1716" LastActivityDate="2010-10-27T14:34:10.650" />
  <row Id="4023" PostTypeId="1" CreationDate="2010-10-27T14:39:11.707" Score="8" ViewCount="499" Body="&lt;p&gt;I am comparing two groups of mutants each of which can have only one out of 21 different phenotypes. I would like to see whether distribution of these outcomes is similar between two groups. I found an &lt;a href=&quot;http://www.fon.hum.uva.nl/Service/Statistics/EqualDistribX2.html&quot; rel=&quot;nofollow&quot;&gt;online test&lt;/a&gt; &#xA; that calculates the &quot;Chi-square test for equality of distributions&quot; and gives me some plausible results. However, I have quite a few zeroes in this Table, so can I use chi-square in this case at all? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the table with two groups and counts of particular phenotypes:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;2 1&#xA;2 3&#xA;1 6&#xA;1 4&#xA;13 77&#xA;7 27&#xA;0 1&#xA;0 4&#xA;0 2&#xA;2 7&#xA;2 3&#xA;1 5&#xA;1 9&#xA;2 6&#xA;0 3&#xA;3 0&#xA;1 3&#xA;0 3&#xA;1 0&#xA;1 2&#xA;0 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1727" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-10-27T18:22:28.543" LastActivityDate="2010-10-28T13:00:07.440" Title="Chi-square test for equality of distributions: how many zeroes does it tolerate?" Tags="&lt;distributions&gt;&lt;chi-squared&gt;&lt;contingency-tables&gt;" AnswerCount="2" CommentCount="3" FavoriteCount="2" />
  <row Id="4024" PostTypeId="2" ParentId="4023" CreationDate="2010-10-27T15:11:25.390" Score="5" ViewCount="" Body="&lt;p&gt;The usual guidelines are that the expected counts should be greater than 5, but it can be somewhat relaxed as discussed in the following article:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Campbell , I, &lt;a href=&quot;http://onlinelibrary.wiley.com/doi/10.1002/sim.2832/abstract&quot; rel=&quot;nofollow&quot;&gt;Chi-squared and&#xA;  Fisher–Irwin tests of two-by-two&#xA;  tables with small sample&#xA;  recommendations&lt;/a&gt;, Statistics in&#xA;  Medicine (2007) 26(19): 3661–3675.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;See also Ian Campbell's &lt;a href=&quot;http://www.iancampbell.co.uk/twobytwo/twobytwo.htm&quot; rel=&quot;nofollow&quot;&gt;homepage&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that in R, there's always the possibility to compute $p$-value by a Monte Carlo approach (&lt;code&gt;chisq.test(..., sim=TRUE)&lt;/code&gt;), instead of relying on the asymptotic distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In you case, it appears that about 80% of the expected counts are below 5, and 40% are below 1. Would it make sense to aggregate some of the observed phenotypes?&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-27T15:28:30.650" LastActivityDate="2010-10-27T15:28:30.650" CommentCount="2" />
  <row Id="4025" PostTypeId="1" CreationDate="2010-10-27T15:45:20.203" Score="2" ViewCount="317" Body="&lt;p&gt;I am using the &lt;code&gt;naivebayes&lt;/code&gt; function of the &lt;a href=&quot;http://cran.r-project.org/web/packages/e1071/index.html&quot; rel=&quot;nofollow&quot;&gt;e1071&lt;/a&gt; library. Some example commands are:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = naiveBayes(Species ~ ., data = iris)&#xA;pred = predict(model, iris[,])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My question is: how can I obtain the maximum likelihood estimate for the conditional probability distribution of this model?&lt;/p&gt;&#xA;" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-10-27T22:08:34.000" LastActivityDate="2010-11-26T22:44:24.210" Title="MLE for Naive Bayes in R" Tags="&lt;r&gt;&lt;machine-learning&gt;&lt;e1071&gt;&lt;naive-bayes&gt;" AnswerCount="1" CommentCount="4" />
  <row Id="4026" PostTypeId="2" ParentId="4025" CreationDate="2010-10-27T16:26:56.013" Score="1" ViewCount="" Body="&lt;p&gt;This seems a bit ambiguous... What's wrong with &lt;code&gt;model$table&lt;/code&gt;?&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-10-27T16:26:56.013" CommentCount="2" />
  <row Id="4027" PostTypeId="1" AcceptedAnswerId="4032" CreationDate="2010-10-27T17:11:14.887" Score="1" ViewCount="365" Body="&lt;p&gt;Does someone know an easy way to get STATA to display more than just 3 digits of the p-value (significance value) when running regressions, specifically a Tobit regression? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Normally STATA reports, e.g., that the p-value is .001 or .065, but I would like to see more digits, e.g. .0011123 or .065320.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To be clear, I don't want to (necessarily) alter the way the data is produced in the regression table. I only want to be able to get STATA to display more digits for those p-values I am interested in.&lt;/p&gt;&#xA;" OwnerUserId="1729" LastActivityDate="2010-10-27T21:08:32.963" Title="How can I get more digits for the p-value in STATA?" Tags="&lt;regression&gt;&lt;stata&gt;&lt;p-value&gt;" AnswerCount="2" CommentCount="8" />
  <row Id="4028" PostTypeId="2" ParentId="4027" CreationDate="2010-10-27T17:41:00.247" Score="2" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://www.stata.com/help.cgi?whatsnew&quot; rel=&quot;nofollow&quot;&gt;Stata 11.1 introduced&lt;/a&gt; a &lt;code&gt;&lt;a href=&quot;http://www.stata.com/help.cgi?set+pformat&quot; rel=&quot;nofollow&quot;&gt;set pformat&lt;/a&gt;&lt;/code&gt; command that specifies the output format of p-values in coefficient tables. (I don't know about &lt;a href=&quot;http://www.stata.com/support/faqs/res/statalist.html#spell&quot; rel=&quot;nofollow&quot;&gt;STATA&lt;/a&gt; I'm afraid as I think that was discontinued some time in the 1980s).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By the way, you'd probably be better off asking such completely Stata-specific questions on &lt;a href=&quot;http://www.stata.com/statalist/&quot; rel=&quot;nofollow&quot;&gt;Statalist&lt;/a&gt; rather than here.&lt;/p&gt;&#xA;" OwnerUserId="449" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-10-27T17:51:44.913" LastActivityDate="2010-10-27T17:51:44.913" CommentCount="1" />
  <row Id="4029" PostTypeId="2" ParentId="4023" CreationDate="2010-10-27T18:28:19.503" Score="5" ViewCount="" Body="&lt;p&gt;Perfectly feasible these days to do &lt;a href=&quot;http://en.wikipedia.org/wiki/Fisher%27s_exact_test&quot; rel=&quot;nofollow&quot;&gt;Fisher's 'exact' test&lt;/a&gt; on such a table. I just got p =  0.087 using Stata (&lt;code&gt;&lt;a href=&quot;http://www.stata.com/help.cgi?tabi&quot; rel=&quot;nofollow&quot;&gt;tabi&lt;/a&gt; 2 1 \ 2 3 \ .... , exact&lt;/code&gt;. Execution took 0.19 seconds).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT after chl's comment below (tried adding as a comment but can't format):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It works in R 2.12.0 for me, though i had to increase the 'workspace' option over its default value of 200000: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; fisher.test(x)&#xA;Error in fisher.test(x) : FEXACT error 7.&#xA;LDSTP is too small for this problem.&#xA;Try increasing the size of the workspace.&#xA;&amp;gt; system.time(result&amp;lt;-fisher.test(x, workspace = 400000))&#xA;   user  system elapsed &#xA;   0.11    0.00    0.11 &#xA;&amp;gt; result$p.value&#xA;[1] 0.0866764&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(The execution time is slightly quicker than in Stata, but that's of dubious relevance given the time taken to work out the meaning of the error message, which uses 'workspace' to mean something different from R's usual meaning despite the fact that fisher.test is part of R's core 'stats' package.)&lt;/p&gt;&#xA;" OwnerUserId="449" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-10-28T13:00:07.440" LastActivityDate="2010-10-28T13:00:07.440" CommentCount="5" />
  <row Id="4030" PostTypeId="1" CreationDate="2010-10-27T18:35:57.930" Score="4" ViewCount="313" Body="&lt;p&gt;I want to extend the standard &lt;a href=&quot;http://en.wikipedia.org/wiki/Error_correction_model&quot; rel=&quot;nofollow&quot;&gt;VECM&lt;/a&gt; form:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;Δx[t] = δ0 + ... + Π x[t-1] + Φ1 Δx[t-1] + Φ1 Δx[t-2] + ... + ε&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;to include exogenous variables (i.e. variables that are participant in describing the relationships amongst &lt;code&gt;Δx[t]&lt;/code&gt;, but are not in the dependent set &lt;code&gt;Δx[t]&lt;/code&gt;).  Let's call these new variables &lt;code&gt;z&lt;/code&gt;.  Would like to determine a MLE of the coefficients in the revised VECM:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;Δx[t] = δ0 + ... + Π x[t-1] + Φ1 Δx[t-1] + ... + θ1 z[t-1] + ... +ε&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am familiar with the johansen method in determining the MLE coefficients for the 1st form. I'm not sure how can modify the method to determine the second form.   Could use an optimiser to find the coefficients for VECM with exogenous vars, but thought there is probably a more direct approach.&lt;/p&gt;&#xA;" OwnerUserId="1127" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-27T18:54:53.467" LastActivityDate="2011-04-26T18:27:01.343" Title="Finding coefficients for VECM + exogenous variables" Tags="&lt;time-series&gt;&lt;econometrics&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="1" />
  <row Id="4031" PostTypeId="2" ParentId="3984" CreationDate="2010-10-27T20:31:24.907" Score="4" ViewCount="" Body="&lt;p&gt;Grouping SC and AIC together &lt;strong&gt;IS WRONG&lt;/strong&gt;. They are very different things, even though people heavily misuse them. AIC is meaningful when you are predicting things, using SC in this scenario can lead (not all the times) to wrong results. Similarly, if you are interested in doing model selection with the principle of parsimony (Occam's Razor) SC is better. I don't want to go into the theoretical details, but in a nutshell: SC -- good for parsimonious models when you want something equivalent to simplest possible model to explain your data, AIC -- When you want to predict. AIC doesn't assume that your true model lies in the model space where as SC does.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Secondly, using p-values and information criteria together can be also misleading as explained by &lt;strong&gt;chl&lt;/strong&gt;.   &lt;/p&gt;&#xA;" OwnerUserId="1307" LastActivityDate="2010-10-27T20:31:24.907" />
  <row Id="4032" PostTypeId="2" ParentId="4027" CreationDate="2010-10-27T21:08:32.963" Score="3" ViewCount="" Body="&lt;p&gt;Follow up the tobit command with&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;est tab, p(%12.10g)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(for example).  This ought to work even in pretty old versions of Stata.  A little less easy is to write your own output procedure.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-10-27T21:08:32.963" CommentCount="2" />
  <row Id="4033" PostTypeId="2" ParentId="2909" CreationDate="2010-10-27T22:39:50.370" Score="2" ViewCount="" Body="&lt;p&gt;This is an alternating sum.  Each successive pair almost cancels; such pair-sums eventually decrease monotonically.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One approach, then, is to compute the sum in pairs where $n$ = {1,2}, {3,4}, {5,6}, etc. (Doing so eliminates a lot of floating point error, too.) Some more tricks can help:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(1) To solve $\tan(t) =  t / \alpha$ for a positive constant $\alpha$, a good starting value to search--and an excellent approximation for the $n^\text{th}$ largest root-- is $t = (n + 1/2)\pi - \frac{\alpha}{(n + 1/2)\pi}$.  I suspect Newton-Raphson should work really well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(2) After a small number of initial terms, the sums of pairs start decreasing in size very, very consistently.  The logarithms of the absolute values of exponentially-spaced pairs quickly decrease almost linearly.  This means you can interpolate among a very small number of calculated pair-sums to estimate all the pair-sums you did not compute.  For example, by computing the values for only pairs (2,3), (4,5), (8,9), (16,17), ..., (16384, 16385) and constructing the interpolating polynomial for these (thought of as the values of a function at 1, 2, ..., 14) and using the arguments $h = \mu = \sigma = 1$, I was able to achieve six-figure precision for the worst-case errors.  (Even nicer, the errors oscillate in sign, suggesting the precision in the summed interpolated values might be quite a bit better than six figures.)  You could probably estimate the limiting sum to good precision by extrapolating linearly off the end of these values (which translates to a power law) and integrating the extrapolating function out to infinity.  To complete this example calculation you also need the first term.  That gives six-figure precision by means of only 29 computed terms in the summation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(3) Note that the function really depends on $h/\sigma$ and $\mu/\sigma$, not on all three of these variables independently.  The dependence on $T$ is weak (as it should be); you might be content to fix its value throughout all your calculations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(4) On top of all this, consider using some &lt;a href=&quot;http://en.wikipedia.org/wiki/Series_acceleration&quot; rel=&quot;nofollow&quot;&gt;series-acceleration methods&lt;/a&gt;, like &lt;a href=&quot;http://en.wikipedia.org/wiki/Aitken%27s_delta-squared_process&quot; rel=&quot;nofollow&quot;&gt;Aitken's method&lt;/a&gt;.  A good accounting of this appears in &lt;a href=&quot;http://www.nr.com/&quot; rel=&quot;nofollow&quot;&gt;Numerical Recipes&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Added&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(5) You can estimate the tail of the sum with an integral.  Upon writing $\theta_n = (n + 1/2)\pi - 1/t_n$, the equation $\tan(\theta_n) =  \theta_n / \alpha$ (with $\alpha = \mu h / \sigma^2$) can be solved for $t_n$, which is small, and then for $\theta_n$ by substituting back.  Expanding the tangent in a Taylor series in $t_n$ gives the approximate solution&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\theta_n = z - \frac{\alpha}{ z} - \frac{\alpha^2 - \alpha^3/3 }{z^3} + O\left((\frac{\alpha}{n})^5\right)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $z = (n + 1/2)\pi$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Provided $n$ is sufficiently large, the exponential factors of the form $1 - \exp(\frac{-\sigma^2 \theta_n^2 T}{2 h^2}) \exp(\frac{-\mu^2 T}{2 \sigma^2})$ become extremely close to 1 so you can neglect them.  Typically these terms can be neglected even for small $n$ because $\theta_n^2$ is $\Theta\left(n^2\right)$, making the first exponential go to zero extremely quickly.  (This happens once $n$ substantially exceeds $\alpha / T^{1/2}$.  Do your calculations for large $T$ if you can!)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using this expression for $\theta_n$ to sum the terms for $n$ and $n+1$ lets us approximate them (once all the smoke clears) as &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{2}{\pi  n^2}-\frac{4}{\pi  n^3}+\frac{13 \pi ^2+6 (4-3 \alpha ) \alpha }{2 \pi ^3 n^4}+O\left(\frac{1}{n^5}\right) \text{.}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Replacing the sum starting at $n = 2N$ by an integral over $N$ starting at $N - 1/4$ approximates the tail.  (The integral has to be multiplied by a common factor of $\exp(-\alpha)$.)  The error in the integral is $O(1/n^4)$.  Thus, to achieve three significant figures you will typically need to compute around eight or so of the terms in the sum and then add this tail approximation.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-10-28T20:12:38.843" LastActivityDate="2010-10-28T20:12:38.843" CommentCount="1" />
  <row Id="4034" PostTypeId="1" AcceptedAnswerId="4038" CreationDate="2010-10-27T23:56:53.503" Score="8" ViewCount="275" Body="&lt;p&gt;This may be a naive question, but here goes. If I have a set of empirical data and fit a kernel density to it, and then obtain a new single value which possibly comes from the same process which generated the original data set, can I assign a probability that this new value belongs to the set/process by simply reading the value off the y axis where the new value on the x axis intersects the kernel density line and dividing by the area under the density line?&lt;/p&gt;&#xA;" OwnerUserId="226" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-10-28T08:22:03.600" LastActivityDate="2010-10-28T08:22:03.600" Title="Interpretation/use of kernel density" Tags="&lt;kde&gt;" AnswerCount="1" FavoriteCount="3" />
  <row Id="4036" PostTypeId="5" CreationDate="2010-10-28T07:26:22.207" Score="0" ViewCount="0" Body="&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.stata.com&quot; rel=&quot;nofollow&quot;&gt;stata.com&lt;/a&gt; (official website)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Stata&quot; rel=&quot;nofollow&quot;&gt;Wikipedia article&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Some other sources of Stata help:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.stata.com/help.cgi?search&quot; rel=&quot;nofollow&quot;&gt;Stata's built-in search facilities&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.stata.com/support/statalist/faq&quot; rel=&quot;nofollow&quot;&gt;Statalist FAQ&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.ats.ucla.edu/stat/stata/&quot; rel=&quot;nofollow&quot;&gt;Resources from UCLA&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;'Stata' is an invented word, not an acronym, and therefore should not have all its letters capitalized (i.e. 'STATA' is considered incorrect). See the &lt;a href=&quot;http://www.stata.com/support/faqs/res/statalist.html#spell&quot; rel=&quot;nofollow&quot;&gt;last item of the Statalist FAQ&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="-1" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-10-28T07:46:12.183" LastActivityDate="2010-10-28T07:46:12.183" />
  <row Id="4037" PostTypeId="4" CreationDate="2010-10-28T07:26:22.207" Score="0" ViewCount="0" Body="Stata is a proprietary  cross-platform general-purpose statistical software package." OwnerUserId="-1" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-10-28T07:46:12.213" LastActivityDate="2010-10-28T07:46:12.213" />
  <row Id="4038" PostTypeId="2" ParentId="4034" CreationDate="2010-10-28T07:55:48.317" Score="6" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;No&lt;/strong&gt;, I'm afraid not. The kernel density estimand is the &lt;a href=&quot;http://en.wikipedia.org/wiki/Probability_density_function&quot; rel=&quot;nofollow&quot;&gt;probability density function&lt;/a&gt;. The &lt;em&gt;y&lt;/em&gt;-value is an estimate of the probability density at that value of &lt;em&gt;x&lt;/em&gt;, so the area under the curve between &lt;em&gt;x&lt;/em&gt;&lt;sub&gt;1&lt;/sub&gt; and  &lt;em&gt;x&lt;/em&gt;&lt;sub&gt;2&lt;/sub&gt; estimates the probability of the random variable &lt;em&gt;X&lt;/em&gt;&amp;nbsp;  falling between &lt;em&gt;x&lt;/em&gt;&lt;sub&gt;1&lt;/sub&gt; and  &lt;em&gt;x&lt;/em&gt;&lt;sub&gt;2&lt;/sub&gt;, &lt;em&gt;assuming&lt;/em&gt; that &lt;em&gt;X&lt;/em&gt; was generated by the same process that generated the data which you fed into the kernel density estimate. The kernel density estimate doesn't say anything about the probability a new value was generated by the same process.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-10-28T07:55:48.317" />
  <row Id="4039" PostTypeId="2" ParentId="3715" CreationDate="2010-10-28T08:15:50.047" Score="4" ViewCount="" Body="&lt;p&gt;When applying the formula for P(B|A) for &lt;a href=&quot;http://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval#Agresti-Coull_Interval&quot; rel=&quot;nofollow&quot;&gt;Agresti-Coull&lt;/a&gt;, it seems important to me to use, for the denominator (ñ), a number with uncertainty.  The formula ñ=P(A)*N+4 (where N is the size of your sample) gives you this number, after you calculate P(A) with an uncertainty.  With the &lt;a href=&quot;http://packages.python.org/uncertainties/&quot; rel=&quot;nofollow&quot;&gt;uncertainties package&lt;/a&gt;, this would be:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Calculation of P(A):&#xA;P_A = ufloat((…, …))&#xA;# Calculation of P(B|A):&#xA;P_B_A = …/(P_A*N+4)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Thus, P(B|A) is automatically correlated to P(A).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furthermore, you must make sure that you feed &lt;code&gt;ufloat()&lt;/code&gt; with &lt;em&gt;standard deviations&lt;/em&gt;.  This mean using a particular z_{1-alpha/2} value, in the Agresti-Coull formula.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="1735" LastEditorUserId="1735" LastEditorDisplayName="" LastEditDate="2010-11-02T21:34:53.780" LastActivityDate="2010-11-02T21:34:53.780" CommentCount="1" />
  <row Id="4040" PostTypeId="1" AcceptedAnswerId="4041" CreationDate="2010-10-28T08:24:14.630" Score="7" ViewCount="300" Body="&lt;p&gt;In R, I have a data frame comprising a class label &lt;em&gt;C&lt;/em&gt; (a factor) and two measurements, &lt;em&gt;M1&lt;/em&gt; and &lt;em&gt;M2&lt;/em&gt;. How do I compute the correlation between &lt;em&gt;M1&lt;/em&gt; and &lt;em&gt;M2&lt;/em&gt; within each class?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ideally, I'd get back a data frame with one row for each class and two columns: the class label &lt;em&gt;C&lt;/em&gt; and the correlation.&lt;/p&gt;&#xA;" OwnerUserId="439" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-10-28T09:13:26.317" LastActivityDate="2010-10-29T15:23:57.553" Title="R: compute correlation by group" Tags="&lt;r&gt;&lt;correlation&gt;" AnswerCount="3" FavoriteCount="4" />
  <row Id="4041" PostTypeId="2" ParentId="4040" CreationDate="2010-10-28T08:40:21.723" Score="10" ViewCount="" Body="&lt;p&gt;The package plyr is the way to go.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a simple solution:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;xx &amp;lt;- data.frame(group = rep(1:4, 100), a = rnorm(400) , b = rnorm(400) )&#xA;head(xx)&#xA;&#xA;require(plyr)&#xA;func &amp;lt;- function(xx)&#xA;{&#xA;return(data.frame(COR = cor(xx$a, xx$b)))&#xA;}&#xA;&#xA;ddply(xx, .(group), func)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The output will be:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  group         COR&#xA;1     1  0.05152923&#xA;2     2 -0.15066838&#xA;3     3 -0.04717481&#xA;4     4  0.07899114&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="253" LastActivityDate="2010-10-28T08:40:21.723" CommentCount="5" />
  <row Id="4042" PostTypeId="2" ParentId="4040" CreationDate="2010-10-28T10:46:38.670" Score="6" ViewCount="" Body="&lt;p&gt;If you are inclined to use functions in the base package, you can use the &lt;code&gt;by&lt;/code&gt; function, then reassemble the data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;xx &amp;lt;- data.frame(group = rep(1:4, 100), a = rnorm(400) , b = rnorm(400) )&#xA;head(xx)&#xA;&#xA;# This returns a &quot;by&quot; object&#xA;result &amp;lt;- by(xx[,2:3], xx$group, function(x) {cor(x$a, x$b)})&#xA;&#xA;# You get pretty close to what you want if you coerce it into a data frame via a matrix&#xA;result.dataframe &amp;lt;- as.data.frame(as.matrix(result))&#xA;&#xA;# Add the group column from the row names&#xA;result.dataframe$C &amp;lt;- rownames(result)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1195" LastActivityDate="2010-10-28T10:46:38.670" CommentCount="1" />
  <row Id="4043" PostTypeId="1" CreationDate="2010-10-28T11:29:57.120" Score="6" ViewCount="167" Body="&lt;p&gt;I have a few time series that were (for technical reasons) acquired with slightly different time intervals, ranging between 19 and 21 seconds.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, I would like to average the values of these different time series over time, so I thought that I could do some sort of interpolation of the values at some regular interval (e.g. each 20 seconds).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could someone point out a good way to do this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you&lt;/p&gt;&#xA;&#xA;&lt;p&gt;nico&lt;/p&gt;&#xA;" OwnerUserId="582" LastActivityDate="2010-10-28T13:33:07.927" Title="Averaging time series with different sampling interval" Tags="&lt;time-series&gt;&lt;interpolation&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="4044" PostTypeId="1" AcceptedAnswerId="4047" CreationDate="2010-10-28T13:06:25.417" Score="11" ViewCount="353" Body="&lt;p&gt;I'm looking for some good terminology to describe what I'm trying to do, to make it easier to look for resources.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, say I have two clusters of points A and B, each associated to two values, X and Y, and I want to measure the &quot;distance&quot; between A and B - i.e. how likely is it that they were sampled from the same distribution (I can assume that the distributions are normal). For example, if X and Y are correlated in A but not in B, the distributions are different.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Intuitively, I would get the covariance matrix of A, and then look at how likely each point in B is to fit in there, and vice-versa (probably using someting like Mahalanobis distance).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But that is a bit &quot;ad-hoc&quot;, and there is probably a more rigorous way of describing this (of course, in practice I have more than two datasets with more than two variables - I'm trying to identify which of my datasets are outliers).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="1737" LastActivityDate="2010-11-06T15:10:28.317" Title="Measuring the &quot;distance&quot; between two multivariate distributions" Tags="&lt;multivariate-analysis&gt;&lt;terminology&gt;&lt;distance-functions&gt;" AnswerCount="3" CommentCount="3" FavoriteCount="2" />
  <row Id="4045" PostTypeId="2" ParentId="4043" CreationDate="2010-10-28T13:33:07.927" Score="6" ViewCount="" Body="&lt;p&gt;The &lt;a href=&quot;http://cran.r-project.org/package=zoo&quot; rel=&quot;nofollow&quot;&gt;zoo&lt;/a&gt; package is very good at that (as is &lt;a href=&quot;http://cran.r-project.org/package=xts&quot; rel=&quot;nofollow&quot;&gt;xts&lt;/a&gt; which extends it).  The &lt;a href=&quot;http://cran.r-project.org/package=zoo&quot; rel=&quot;nofollow&quot;&gt;zoo&lt;/a&gt; vignettes have e.g. this example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;zr3 &amp;lt;- zooreg(rnorm(9), start=as.yearmon(2000), frequency=12)&#xA;zr3&#xA;aggregate(zr3, as.yearqtr, mean)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;A (regular) series is created with monthly frequency, and the averaged by quarter. It works the very same way for &lt;code&gt;POSIXct&lt;/code&gt; objects at much higher granularity; see the vignette.  I suspect that the R-SIG-Finance list archives have plenty of related examples too.&lt;/p&gt;&#xA;" OwnerUserId="334" LastActivityDate="2010-10-28T13:33:07.927" CommentCount="3" />
  <row Id="4046" PostTypeId="2" ParentId="4044" CreationDate="2010-10-28T13:47:54.260" Score="7" ViewCount="" Body="&lt;p&gt;Hmm, the &lt;a href=&quot;http://en.wikipedia.org/wiki/Bhattacharyya_distance&quot; rel=&quot;nofollow&quot;&gt;Bhattacharyya distance&lt;/a&gt; seems to be what I'm looking for, though the &lt;a href=&quot;http://en.wikipedia.org/wiki/Hellinger_distance&quot; rel=&quot;nofollow&quot;&gt;Hellinger distance&lt;/a&gt; works too.&lt;/p&gt;&#xA;" OwnerUserId="1737" LastActivityDate="2010-10-28T13:47:54.260" CommentCount="2" />
  <row Id="4047" PostTypeId="2" ParentId="4044" CreationDate="2010-10-28T14:19:15.170" Score="7" ViewCount="" Body="&lt;p&gt;There is also the &lt;a href=&quot;http://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence&quot; rel=&quot;nofollow&quot;&gt;Kullback-Leibler&lt;/a&gt; divergence, which is related to the Hellinger Distance you mention above.&lt;/p&gt;&#xA;" OwnerUserId="1390" LastEditorUserId="1390" LastEditorDisplayName="" LastEditDate="2010-10-28T14:27:09.980" LastActivityDate="2010-10-28T14:27:09.980" CommentCount="2" />
  <row Id="4048" PostTypeId="1" CreationDate="2010-10-28T14:47:24.517" Score="0" ViewCount="181" Body="&lt;p&gt;Hello,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have been having a bit of trouble with this issue with a while and have decided to ask for help!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am looking for a solution on how to find the nearest country based on a latitude and longitude.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now i know this is called reverse geocoding.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The issue i have is that this needs to be stored on a handheld device with limited memory and no internet connection.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So!  I am looking for a way of passing in a latitude and longitude to the smallest data set possible and receiving the country that this latitude and longitude resides in.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would i need a data set containing the latitude and longitude values of the countries borders possible?  if so does anyone know where i may find this dataset?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any advice or pointers is highly appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tom&lt;/p&gt;&#xA;" LastActivityDate="2010-10-28T14:47:24.517" ClosedDate="2010-10-28T16:59:41.537" Title="Finding the nearest country based on latitude and longitude" Tags="&lt;mathematics&gt;&lt;information-geometry&gt;" CommentCount="2" />
  <row Id="4049" PostTypeId="2" ParentId="3955" CreationDate="2010-10-28T15:38:11.823" Score="4" ViewCount="" Body="&lt;p&gt;It sounds like you are looking for an &quot;online changepoint detection method.&quot;  (That's a useful phrase for Googling.)  Some useful recent (and accessible) papers are &lt;a href=&quot;http://arxiv.org/PS_cache/arxiv/pdf/0710/0710.3742v1.pdf&quot; rel=&quot;nofollow&quot;&gt;Adams &amp;amp; MacKay&lt;/a&gt; (a Bayesian approach) and &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.23.6570&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;Keogh et al.&lt;/a&gt;  You might be able to press the &lt;strong&gt;&lt;a href=&quot;http://surveillance.r-forge.r-project.org/&quot; rel=&quot;nofollow&quot;&gt;surveillance&lt;/a&gt;&lt;/strong&gt; package for R into service.  Isolated large numbers of hits can be found using &lt;a href=&quot;http://en.wikipedia.org/wiki/Statistical_process_control&quot; rel=&quot;nofollow&quot;&gt;statistical process control methods&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-10-28T15:38:11.823" />
  <row Id="4050" PostTypeId="2" ParentId="4013" CreationDate="2010-10-28T17:45:54.457" Score="2" ViewCount="" Body="&lt;p&gt;Statistics is the deductive approach to induction.  Consider the two main approaches to statistical inference: Frequentist and Bayesian.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assume you are a Frequentist (in the style of Fisher, rather than Neyman for convenience).  You wonder whether a parameter of substantive interest takes a particular value, so you construct a model, choose a statistic relating to the parameter, and perform a test.  The p-value generated by your test indicates the probability of seeing a statistic as or more extreme than the statistic computed from the sample you have, assuming that your model is correct.  You get a small enough p-value so you reject the hypothesis that the parameter does take that value.  Your reasoning is deductive: Assuming the model is correct, either the parameter really does take the value of substantive interest but yours is an unlikely sample to see, or it does not take in fact that value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Turning from hypothesis test to confidence intervals: you have a 95% confidence interval for your parameter which does not contain the value substantive interest.  Your reasoning is again deductive: assuming the model is correct, either this is one of those rare intervals that will appear 1 in 20 times when the parameter really does have the value of substantive interest (because your sample is an unlikely one), or the parameter does not in fact have that value.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now assume you are a Bayesian (in the style of Laplace rather than Gelman).  Your model assumptions and calculations give you a (posterior) probability distribution over the parameter value.  Most of the mass of this distribution is far from the value of substantive interest, so you conclude that the parameter probably does not have this value.  Your reasoning is again deductive: assuming your model to be correct and if the prior distribution represented your beliefs about the parameter, then your beliefs about it in the light of the data are described by your posterior distribution which puts very little probability on that value.  Since this distribution offers little support for the value of substantive interest, you might conclude that the parameter does not in fact have the value.   (Or you might be content to state the probability it does).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In all three cases you get a logical disjunction to base your action on which is derived deductively/mathematically from assumptions.  These assumptions are usually about a model of how the data is generated, but may also be prior beliefs about other quantities.&lt;/p&gt;&#xA;" OwnerUserId="1739" LastActivityDate="2010-10-28T17:45:54.457" CommentCount="3" />
  <row Id="4051" PostTypeId="2" ParentId="1164" CreationDate="2010-10-28T18:14:53.187" Score="5" ViewCount="" Body="&lt;p&gt;So 'classical models' (whatever they are - I assume you mean something like simple models taught in textbooks and estimated by ML) fail on some, perhaps many, real world data sets.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If a model fails then there are two basic approaches to fixing it:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Make fewer assumptions (less model)&lt;/li&gt;&#xA;&lt;li&gt;Make more assumptions (more model)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Robust statistics, quasi-likelihood, and GEE approaches take the first approach by changing the estimation strategy to one where the model does not hold for all data points (robust) or need not characterize all aspects of the data (QL and GEE).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The alternative is to try to build a model that explicitly models the source of contaminating data points, or the aspects of the original model that seems to be false, while keeping the estimation method the same as before.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some intuitively prefer the former (it's particularly popular in economics), and some intuitively prefer the latter (it's particular popular among Bayesians, who tend to be happier with more complex models, particularly once they realize they're going to have use simulation tools for inference anyway). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Fat tailed distributional assumptions, e.g. using the negative binomial rather than poisson or t rather than normal, belong to the second strategy.  Most things labelled 'robust statistics' belong to the first strategy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a practical matter, deriving estimators for the first strategy for realistically complex problems seems to be quite hard.  Not that that's a reason for not doing so, but it is perhaps an explanation for why it isn't done very often.&lt;/p&gt;&#xA;" OwnerUserId="1739" LastActivityDate="2010-10-28T18:14:53.187" CommentCount="1" />
  <row Id="4052" PostTypeId="1" AcceptedAnswerId="4056" CreationDate="2010-10-28T18:33:14.513" Score="7" ViewCount="394" Body="&lt;p&gt;Is the power of a logistic regression and a t-test equivalent?  If so, they should be &quot;data density equivalent&quot; by which I mean that the same number of underlying observations yields the same power given a fixed alpha of .05.  Consider two cases:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;[The parametric t-test]: 30 draws from a binomial observation are made and the resulting values are averaged.  This is done 30 times for group A (which has a binomial Pr of .70 of occurring) and 30 times for group B (which has a binomial Pr of .75 of occurring).  This yields 30 means per group that represent a summary of 1,800 draws from a binomial distribution.  A 58df t-test is performed to compare the means.&lt;/li&gt;&#xA;&lt;li&gt;[The logistic regression]: A logistic regression is performed with a dummy coded slope representing group membership and each of the 1,800 draws.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;My question has two parts:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Given a set alpha of .05, will the power of these methodologies be the same or different?  Why? How can I prove it?&lt;/li&gt;&#xA;&lt;li&gt;Is the answer to question 1 is sensitive to the sample sizes going into the t-test, sample size of each group in the t-test, underlying binomial probabilities, or some other factor? If so, how can I know (without simulation) that the power is indeed different and what sort of changes will produce what sort of changes in power?  Alternatively, provide worked out R code that solves the issue using simulation.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="196" LastEditorUserId="196" LastEditorDisplayName="" LastEditDate="2010-11-01T20:17:20.833" LastActivityDate="2010-11-02T19:19:49.510" Title="How does the power of a logistic regression and a t-test compare?" Tags="&lt;logistic-regression&gt;&lt;t-test&gt;&lt;power&gt;&lt;power-analysis&gt;" AnswerCount="2" FavoriteCount="2" />
  <row Id="4053" PostTypeId="1" CreationDate="2010-10-28T19:54:31.930" Score="12" ViewCount="199" Body="&lt;p&gt;Currently I am analyzing social networks (not virtual) and I am observing the connections between people. If a person would choose another person to connect with randomly, the number of connections within a group of people would be distributed normally - at least so claims the book I am currently reading.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The question is the following: how can we know the distribution is Gaussian (normal)? There are also other distributions such as Poisson, Rice, Riley etc. The problem with Gaussian distribution in theory is that the values go from -infinity to +infinity (although the probabilities go toward zero) and the number of connections cannot be negative.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone know which distribution can be expected in case each person independently (randomly) picks-up another person to connect with?&lt;/p&gt;&#xA;" OwnerUserId="315" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-10-28T23:56:19.090" LastActivityDate="2010-10-28T23:56:19.090" Title="How to know which probability distribution to expect?" Tags="&lt;distributions&gt;&lt;networks&gt;&lt;central-limit-theorem&gt;" AnswerCount="2" CommentCount="3" FavoriteCount="1" />
  <row Id="4054" PostTypeId="2" ParentId="4053" CreationDate="2010-10-28T20:18:31.910" Score="1" ViewCount="" Body="&lt;p&gt;The answer is dependent on the assumptions that you are willing to make. A social network constantly evolves over time and hence is not a static entity. Therefore, you need to make some assumptions about how the network evolves over time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The trivial answer under the stated conditions is: If the network size is $n$ then as  asymptotically (in the sense of 'as time goes to infinity')&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Prob(\mbox{No of connections for any individual} = n-1) =1$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If a person selects another person at random to connect to then eventually everyone will be connected. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, real life networks do not behave this way. People differ in several aspects.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;At any time a person has a fixed network size and the probability of another connection being made is a function of his/her network size (as people introduce other people etc).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;A person has his/her own intrinsic tendency to form a connection (as some are introvert/exterovert etc).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;These probabilities change over time, context etc. I am not sure there is a straightforward answer unless we make some assumptions about the structure of the network (e.g., density of the network, how people behave etc).&lt;/p&gt;&#xA;" LastActivityDate="2010-10-28T20:37:39.610" CommentCount="3" />
  <row Id="4055" PostTypeId="2" ParentId="4053" CreationDate="2010-10-28T20:23:08.333" Score="4" ViewCount="" Body="&lt;p&gt;When there are $n$ people and the number of connections made by person $i, 1 \le i \le n,$ is $X_i$, then the total number of connections is $S_n = \sum_{i=1}^n{X_i} / 2$.  Now if we take the $X_i$ to be random variables, assume they are independent and their variances are not &quot;too unequal&quot; as more and more people are added to the mix, then the &lt;a href=&quot;http://en.wikipedia.org/wiki/Central_limit_theorem&quot; rel=&quot;nofollow&quot;&gt;Lindeberg-Levy Central Limit Theorem&lt;/a&gt; applies.  It asserts that the &lt;em&gt;cumulative distribution function&lt;/em&gt; of the &lt;em&gt;standardized&lt;/em&gt; sum converges to the cdf of the normal distribution.  That means roughly that a histogram of the sum will look more and more like a Gaussian (a &quot;bell curve&quot;) as $n$ grows large.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's review what this does &lt;em&gt;not&lt;/em&gt; say:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;It does not assert that the distribution of $S_n$ is ever &lt;em&gt;exactly&lt;/em&gt; normal.  It can't be, for the reasons you point out.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;It does not imply the expected number of connections converges.  In fact, it must diverge (go to infinity).  The standardization is a recentering and rescaling of the distribution; the amount of rescaling is growing without limit.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;It says nothing when the $X_i$ are not independent or when their variances change too much as $n$ grows.  (However, there are generalizations of the CLT for &quot;slightly&quot; dependent series of variables.)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-10-28T20:23:08.333" CommentCount="3" />
  <row Id="4056" PostTypeId="2" ParentId="4052" CreationDate="2010-10-28T20:59:00.237" Score="7" ViewCount="" Body="&lt;p&gt;If I have computed correctly, logistic regression asymptotically has the same power as the t-test.  To see this, write down its log likelihood and compute the expectation of its Hessian at its global maximum (its negative estimates the variance-covariance matrix of the ML solution).  Don't bother with the usual logistic parameterization: it's simpler just to parameterize it with the two probabilities in question.  The details will depend on exactly how you test the significance of a logistic regression coefficient (there are several methods).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That these tests have similar powers should not be too surprising, because the chi-square theory for ML estimates is based on a normal approximation to the log likelihood, and the t-test is based on a normal approximation to the distributions of proportions.  The crux of the matter is that both methods make the same estimates of the two proportions and both estimates have the same standard errors.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;An actual analysis might be more convincing.  Let's adopt some general terminology for the values in a given group (A or B):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;$p$ is the probability of a 1.&lt;/li&gt;&#xA;&lt;li&gt;$n$ is the size of each set of draws.&lt;/li&gt;&#xA;&lt;li&gt;$m$ is the number of sets of draws.&lt;/li&gt;&#xA;&lt;li&gt;$N = m n$ is the amount of data.&lt;/li&gt;&#xA;&lt;li&gt;$k_{ij}$ (equal to $0$ or $1$) is the value of the $j^\text{th}$ result in the $i^\text{th}$ set of draws.&lt;/li&gt;&#xA;&lt;li&gt;$k_i$ is the total number of ones in the  $i^\text{th}$ set of draws.&lt;/li&gt;&#xA;&lt;li&gt;$k$ is the total number of ones.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Logistic regression&lt;/strong&gt; is essentially the ML estimator of $p$.  Its logarithm is given by&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\log(\mathbb{L}) = k \log(p) + (N-k) \log(1-p).$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Its derivatives with respect to the parameter $p$ are&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{\partial \log(\mathbb{L})}{ \partial p} = \frac{k}{p} - \frac{N-k}{1-p} \text{ and}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$-\frac{\partial^2 \log(\mathbb{L})}{\partial p^2} = \frac{k}{p^2} + \frac{N-k}{(1-p)^2}.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Setting the first to zero yields the ML estimate ${\hat{p} = k/N}$ and plugging that into the reciprocal of the second expression yields the variance $\hat{p}(1 - \hat{p})/N$, which is the square of the standard error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;strong&gt;t statistic&lt;/strong&gt; will be obtained from estimators based on the data grouped by sets of draws; namely, as the difference of the means (one from group A and the other from group B) divided by the standard error of that difference, which is obtained from the standard deviations of the means.  Let's look at the mean and standard deviation for a given group, then.  The mean equals $k/N$, which is identical to the ML estimator $\hat{p}$.  The standard deviation in question is the &lt;em&gt;standard deviation of the draw means;&lt;/em&gt; that is, it is the standard deviation of the set of $k_i/n$.  Here is the crux of the matter, so let's explore some possibilities.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Suppose the data aren't grouped into draws at all: that is, $n = 1$ and $m = N$.  The $k_{i}$ are the draw means.  Their &lt;em&gt;sample&lt;/em&gt; variance equals $N/(N-1)$ times $\hat{p}(1 - \hat{p})$.  From this it follows that the standard error is identical to the ML standard error apart from a factor of $\sqrt{N/(N-1)}$, which is essentially $1$ when $N = 1800$.  Therefore--apart from this tiny difference--any tests based on logistic regression will be the same as a t-test and we will achieve essentially the same power.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;When the data are grouped, the (true) variance of the $k_i/n$ equals $p(1-p)/n$ because the statistics $k_i$ represent the sum of $n$ Bernoulli($p$) variables, each with variance $p(1-p)$.  Therefore the &lt;em&gt;expected&lt;/em&gt; standard error of the mean of $m$ of these values is the square root of $p(1-p)/n/m = p(1-p)/N$, just as before.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Number 2 indicates the power of the test should not vary appreciably with how the draws are apportioned (that is, with how $m$ and $n$ are varied subject to $m n = N$), apart perhaps from a fairly small effect from the adjustment in the sample variance (unless you were so foolish as to use extremely few sets of draws within each group).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Limited simulations&lt;/strong&gt; to compare $p = 0.70$ to $p = 0.74$ (with 10,000 iterations apiece) involving  $m = 900, n = 1$ (essentially logistic regression); $m = n = 30$; and $m = 2, n = 450$ (maximizing the sample variance adjustment) bear this out: the power (at $\alpha = 0.05$, one-sided) in the first two cases is 0.59 whereas in the third, where the adjustment factor makes a material change (there are now just two degrees of freedom instead of 1798 or 58), it drops to 0.36.  Another test comparing $p = 0.50$ to $p = 0.52$ gives powers of 0.22, 0.21, and 0.15, respectively: again, we observe only a slight drop from no grouping into draws (=logistic regression) to grouping into 30 groups and a substantial drop down to just two groups.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The morals&lt;/strong&gt; of this analysis are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;You don't lose much when you partition your $N$ data values into a large number $m$ of relatively small groups of &quot;draws&quot;.&lt;/li&gt;&#xA;&lt;li&gt;You can lose appreciable power using small numbers of groups ($m$ is small, $n$--the amount of data per group--is large).&lt;/li&gt;&#xA;&lt;li&gt;You're best off not grouping your $N$ data values into &quot;draws&quot; at all.  Just analyze them as-is (using any reasonable test, including logistic regression and t-testing).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-10-29T16:45:45.883" LastActivityDate="2010-10-29T16:45:45.883" CommentCount="3" />
  <row Id="4057" PostTypeId="2" ParentId="3390" CreationDate="2010-10-28T22:34:50.633" Score="4" ViewCount="" Body="&lt;p&gt;I have never seen C-F used for empirical estimates.  Why bother?  You have outlined a good set of reasons why not.  (I don't think C-F &quot;wins&quot; even in case 1 due to the instability of estimates of higher-order cumulants and their lack of resistance.)  It is intended for theoretical approximations.  Johnson &amp;amp; Kotz, in their &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0471584940&quot; rel=&quot;nofollow&quot;&gt;encyclopedic work on distributions&lt;/a&gt;, routinely use C-F expansions to develop approximations to distribution functions.  Such approximations were useful to supplement tables (or even create them) before powerful statistical software was widespread.  They can still be useful on platforms where appropriate code is not available such as quick-and-dirty spreadsheet calculations.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-10-28T22:34:50.633" CommentCount="1" />
  <row Id="4058" PostTypeId="1" CreationDate="2010-10-29T01:31:29.823" Score="6" ViewCount="676" Body="&lt;p&gt;One of the assumption of logistic regression is the linearity in the logit. So once I got my model up and running I test for nonlinearity using Box-Tidwell test. One of my continuous predictors (X) has tested positive for nonlinearity. What am I suppose to do next?&#xA;As this is a violation of the assumptions shall I get rid of the predictor (X) or include the nonlinear transformation (X*X). Or transform the variable into a categorical?&#xA;If you have a reference could you please point me to that too?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks&lt;/p&gt;&#xA;" OwnerUserId="694" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-10-29T07:06:48.617" LastActivityDate="2010-10-29T09:02:47.340" Title="Testing nonlinearity in logistic regression" Tags="&lt;logistic-regression&gt;&lt;assumptions&gt;" AnswerCount="3" FavoriteCount="2" />
  <row Id="4059" PostTypeId="1" CreationDate="2010-10-29T04:44:40.503" Score="0" ViewCount="235" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;Possible Duplicate:&lt;/strong&gt;&lt;br&gt;&#xA;  &lt;a href=&quot;http://stats.stackexchange.com/questions/4025/mle-for-naive-bayes-in-r&quot;&gt;MLE for Naive Bayes in R&lt;/a&gt;  &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&#xA;&#xA;&lt;p&gt;Hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need the formula representation of maximum likelihood estimate for an example conditional probability distribution for Naive Bayes algorithm. I couldn't find anything from my Google searches. I hope somebody can answer my question or point out a good resource.&lt;/p&gt;&#xA;" LastActivityDate="2010-10-29T04:44:40.503" ClosedDate="2010-10-29T10:23:31.367" Title="Maximum Likelihood formula for Naive Bayes" Tags="&lt;computational-statistics&gt;" CommentCount="5" />
  <row Id="4060" PostTypeId="2" ParentId="4058" CreationDate="2010-10-29T07:50:26.763" Score="3" ViewCount="" Body="&lt;p&gt;I would suggest to use restricted cubic splines (&lt;code&gt;rcs&lt;/code&gt; in R, see the &lt;a href=&quot;http://cran.r-project.org/web/packages/Hmisc/index.html&quot; rel=&quot;nofollow&quot;&gt;Hmisc&lt;/a&gt; and &lt;a href=&quot;http://cran.r-project.org/web/packages/Design/index.html&quot; rel=&quot;nofollow&quot;&gt;Design&lt;/a&gt; packages for examples of use), instead of adding power of $X$ in your model. This approach is the one that is recommended by Frank Harrell, for instance, and you will find a nice illustration in his handouts (§2.5 and chap. 9) on &lt;em&gt;Regression Modeling Strategies&lt;/em&gt; (see the &lt;a href=&quot;http://biostat.mc.vanderbilt.edu/twiki/bin/view/Main/RmS&quot; rel=&quot;nofollow&quot;&gt;companion website&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can compare the results with your Box-Tidwell test by using the &lt;code&gt;boxTidwell()&lt;/code&gt; in the &lt;a href=&quot;http://cran.r-project.org/web/packages/car/index.html&quot; rel=&quot;nofollow&quot;&gt;car&lt;/a&gt; package.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Transforming continuous predictors into categorical ones is generally not a good idea, see e.g. &lt;a href=&quot;http://biostat.mc.vanderbilt.edu/wiki/Main/CatContinuous&quot; rel=&quot;nofollow&quot;&gt;Problems Caused by Categorizing Continuous Variables&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-29T08:09:39.157" LastActivityDate="2010-10-29T08:09:39.157" />
  <row Id="4061" PostTypeId="2" ParentId="4058" CreationDate="2010-10-29T08:37:57.233" Score="3" ViewCount="" Body="&lt;p&gt;It &lt;em&gt;may&lt;/em&gt; be appropriate to include a nonlinear transformation of &lt;em&gt;x&lt;/em&gt;, but probably not simply &lt;em&gt;x&lt;/em&gt; × &lt;em&gt;x&lt;/em&gt;, i.e x&lt;sup&gt;2&lt;/sup&gt;. I believe you may find this a useful reference in determining which transformation to use:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;G. E. P. Box and Paul W. Tidwell (1962). Transformation of the Independent Variables. &lt;em&gt;Technometrics&lt;/em&gt; Volume 4 Number 4, pages 531-550. &lt;a href=&quot;http://www.jstor.org/stable/1266288&quot; rel=&quot;nofollow&quot;&gt;http://www.jstor.org/stable/1266288&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some consider the Box-Tidwell family of transformations to be &lt;em&gt;more&lt;/em&gt; general than is often appropriate for interpretability and parsimony. Patrick Royston and Doug Altman introduced the term &lt;strong&gt;fractional polynomials&lt;/strong&gt; for  Box-Tidwell transformations with simple rational powers in an influential 1994 paper:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P. Royston and D. G. Altman (1994). Regression using fractional polynomials of continuous covariates: parsimonious parametric modeling. &lt;em&gt;Applied Statistics&lt;/em&gt; Volume 43: pages 429–467. &lt;a href=&quot;http://www.jstor.org/stable/2986270&quot; rel=&quot;nofollow&quot;&gt;http://www.jstor.org/stable/2986270&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Patrick Royston in particular has continued to work and publish both papers and software on this, culminating in a book with Willi Sauerbrei:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P. Royston and W. Sauerbrei (2008). &lt;em&gt;Multivariable Model-building: A Pragmatic Approach to Regression Analysis Based on Fractional Polynomials for Modelling Continuous Variables&lt;/em&gt;. Chichester, UK: Wiley. ISBN 978-0-470-02842-1&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-10-29T08:37:57.233" />
  <row Id="4062" PostTypeId="1" AcceptedAnswerId="4064" CreationDate="2010-10-29T08:56:14.133" Score="8" ViewCount="682" Body="&lt;p&gt;I'm referring to something like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/bQ8b1.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;suggested dataset for showing a solutions: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data(mtcars)&#xA;plot(hclust(dist(mtcars)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="253" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-10-29T12:24:53.710" LastActivityDate="2010-11-05T03:17:06.993" Title="How to plot a fan (Polar) Dendrogram in R?" Tags="&lt;r&gt;&lt;data-visualization&gt;&lt;dendrogram&gt;" AnswerCount="2" CommentCount="4" FavoriteCount="5" />
  <row Id="4063" PostTypeId="2" ParentId="4058" CreationDate="2010-10-29T09:02:47.340" Score="2" ViewCount="" Body="&lt;p&gt;Don't forget to check for interactions between X and other independent variables.  Leaving interactions unmodeled can make X look like it has a non-linear effect when it simply has a non-additive one.&lt;/p&gt;&#xA;" OwnerUserId="1739" LastActivityDate="2010-10-29T09:02:47.340" CommentCount="1" />
  <row Id="4064" PostTypeId="2" ParentId="4062" CreationDate="2010-10-29T09:43:52.803" Score="8" ViewCount="" Body="&lt;p&gt;In phylogenetics, this is a fan phylogram, so you can convert this to &lt;code&gt;phylo&lt;/code&gt; and use &lt;code&gt;ape&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(ape)&#xA;library(cluster) &#xA;data(mtcars)&#xA;plot(as.phylo(hclust(dist(mtcars))),type=&quot;fan&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Result:&lt;br&gt;&#xA;&lt;img src=&quot;http://i.imgur.com/iCxeV.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-10-29T09:43:52.803" CommentCount="4" />
  <row Id="4065" PostTypeId="1" AcceptedAnswerId="4178" CreationDate="2010-10-29T12:39:52.583" Score="6" ViewCount="132" Body="&lt;p&gt;Hello fellow number crunchers&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to generate n random scores (together with a class label) as if they had been produced by a binary classification model. In detail, the following properties are required:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;every score is between 0 and 1&lt;/li&gt;&#xA;&lt;li&gt;every score is associated with a binary label with values &quot;0&quot; or &quot;1&quot; (latter is positive class)&lt;/li&gt;&#xA;&lt;li&gt;the overall precision of the scores should be e.g. 0.1 (&amp;lt;- parameter of the generator)&lt;/li&gt;&#xA;&lt;li&gt;the ratio of scores with label &quot;1&quot; should be higher than overall precision in the top-section and lower in the bottom section (&amp;lt;- the &quot;model-quality&quot; should also be a parameter of the generator)&lt;/li&gt;&#xA;&lt;li&gt;the scores should be in such a way, that a resulting roc curve is smooth (and not e.g. that a bunch of scores with label &quot;1&quot; are at the top and the rest of the scores with label &quot;1&quot; is at the bottom of the list).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Does anyone have an idea how to approach this? Maybe via generation of a roc-curve and then generating the points from that cure? Thanks in advance!&lt;/p&gt;&#xA;" OwnerUserId="264" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-10-29T16:09:07.507" LastActivityDate="2010-11-03T16:47:43.637" Title="Random generation of scores similar to those of a classification model" Tags="&lt;machine-learning&gt;&lt;classification&gt;&lt;roc&gt;&lt;random-generation&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="2" />
  <row Id="4067" PostTypeId="2" ParentId="4040" CreationDate="2010-10-29T15:23:57.553" Score="6" ViewCount="" Body="&lt;p&gt;Another example using base packages and Tal's example data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;DataCov &amp;lt;- do.call( rbind, lapply( split(xx, xx$group),&#xA;             function(x) data.frame(group=x$group[1], mCov=cov(x$a, x$b)) ) )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1657" LastActivityDate="2010-10-29T15:23:57.553" CommentCount="2" />
  <row Id="4068" PostTypeId="1" AcceptedAnswerId="4069" CreationDate="2010-10-29T17:45:09.530" Score="6" ViewCount="328" Body="&lt;p&gt;I was horrified to find recently that Matlab returns $0$ for the &lt;em&gt;sample&lt;/em&gt; variance of a scalar input:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt; var(randn(1),0)   %the '0' here tells var to give sample variance&#xA;ans =&#xA;     0&#xA;&amp;gt;&amp;gt; var(randn(1),1)   %the '1' here tells var to give population variance&#xA;ans =&#xA;     0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Somehow, the sample variance is not dividing by $0 = n-1$ in this case. R returns a NaN for a scalar:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; var(rnorm(1,1))&#xA;[1] NA&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What do you think is a sensible way to define the &lt;strike&gt;population&lt;/strike&gt; sample variance for a scalar? What consequences might there be for returning a zero instead of a NaN?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;edit&lt;/strong&gt;: from the help for Matlab's &lt;code&gt;var&lt;/code&gt;: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;VAR normalizes Y by N-1 if N&amp;gt;1, where N is the sample size.  This is&#xA;an unbiased estimator of the variance of the population from which X is&#xA;drawn, as long as X consists of independent, identically distributed&#xA;samples. For N=1, Y is normalized by N. &#xA;&#xA;Y = VAR(X,1) normalizes by N and produces the second moment of the&#xA;sample about its mean.  VAR(X,0) is the same as VAR(X).&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;a cryptic comment in the m code for `var states:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;if w == 0 &amp;amp;&amp;amp; n &amp;gt; 1&#xA;    % The unbiased estimator: divide by (n-1).  Can't do this&#xA;    % when n == 0 or 1.&#xA;    denom = n - 1;&#xA;else&#xA;    % The biased estimator: divide by n.&#xA;    denom = n; % n==0 =&amp;gt; return NaNs, n==1 =&amp;gt; return zeros&#xA;end&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;i.e.&lt;/em&gt; they explicitly choose not to return a &lt;code&gt;NaN&lt;/code&gt; even when the user requests a sample variance on a scalar. My question is why they should choose to do this, not how.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;edit&lt;/strong&gt;: I see that I had erroneously asked about how one should define the population variance of a scalar (see strike through line above). This probably caused a lot of confusion.&lt;/p&gt;&#xA;" OwnerUserId="795" LastEditorUserId="795" LastEditorDisplayName="" LastEditDate="2010-10-29T19:43:36.580" LastActivityDate="2010-10-30T22:29:28.380" Title="How should one define the sample variance for scalar input?" Tags="&lt;r&gt;&lt;variance&gt;&lt;matlab&gt;&lt;code&gt;" AnswerCount="3" CommentCount="8" />
  <row Id="4069" PostTypeId="2" ParentId="4068" CreationDate="2010-10-29T18:14:18.963" Score="4" ViewCount="" Body="&lt;p&gt;Scalars can't 'have' a population variance although they can be single samples from population that has a (population) variance.  If you want to estimate that then you need at least: more than one data point in the sample, another sample from the same distribution, or some prior information about the population variance by way of a model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;btw R has returned missing (NA) not NaN&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;is.nan(var(rnorm(1,1)))&#xA;[1] FALSE&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1739" LastActivityDate="2010-10-29T18:14:18.963" CommentCount="4" />
  <row Id="4070" PostTypeId="2" ParentId="4068" CreationDate="2010-10-29T18:15:06.807" Score="3" ViewCount="" Body="&lt;p&gt;Hello Shabbychef,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am sure people in this forum will have better answers, here is what I think:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think R's answer is logical. The random variable has a population variance, but it turns out that with 1 sample you don't have enough degrees of freedom to estimate sample variance i-e- you are trying to extract information that is &lt;em&gt;NOT&lt;/em&gt; there.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regarding Matlab's answer, I don't know how to justify 0, except that it is from the numerator.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consequences can be bizarre. But I can think of anything else related to the estimation.&lt;/p&gt;&#xA;" OwnerUserId="1307" LastEditorUserId="1307" LastEditorDisplayName="" LastEditDate="2010-10-30T22:29:28.380" LastActivityDate="2010-10-30T22:29:28.380" CommentCount="6" />
  <row Id="4071" PostTypeId="2" ParentId="4068" CreationDate="2010-10-29T18:34:27.813" Score="1" ViewCount="" Body="&lt;p&gt;I think Matlab is using the following logic for a scalar (analogous to how we define population variance) to avoid having to deal with NA and NAN.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Var(x) = \frac{(x - \bar{x})^2}{1} = 0$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The above follows as for a scalar: $\bar{x} = x$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Their definition is probably a programming convention that may perhaps make some aspect of coding easier. &lt;/p&gt;&#xA;" LastActivityDate="2010-10-29T18:34:27.813" CommentCount="4" />
  <row Id="4072" PostTypeId="1" AcceptedAnswerId="4074" CreationDate="2010-10-29T19:24:27.813" Score="3" ViewCount="267" Body="&lt;p&gt;I'd like to make an assertion about whether individuals in my dataset exceed legal standards for commitment, which is largely determined by estimated risk of recidivism.  I have estimated a logit model predicting recidivism.  Here is sample code written trying to determine what percentage of individuals had predicted recidivism of at least 75% with 75% confidence:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;.logit sexrecS [INDEPENDENT VARIABLES]&#xA;.predict sr1&#xA;.predict stdsr1, stdp&#xA;.gen byte sr1C1R1=0 &#xA;.replace sr1C1R1=1 if 1/(1 + exp(ln((1-sr1)/sr1) - 0.67448975*stdsr1)) &amp;gt; .75&#xA;.sum sr1C1R1 if sr1!=. &amp;amp; sexrecS!=.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I later replace 0.67448975 with 1.281551566 to reflect a higher legal standard of proof, but the code generates more, not fewer, hits, so I know I've done something wrong.&lt;/p&gt;&#xA;" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-10-29T19:53:44.330" LastActivityDate="2010-10-29T20:44:13.563" Title="How does one calculate confidence intervals on predictions generated by logit in Stata?" Tags="&lt;confidence-interval&gt;&lt;stata&gt;&lt;logit&gt;" AnswerCount="1" CommentCount="6" />
  <row Id="4073" PostTypeId="2" ParentId="3907" CreationDate="2010-10-29T19:38:10.120" Score="4" ViewCount="" Body="&lt;p&gt;The starting time of the study is immaterial: it's just an origin for the clock.  What you want to consider are the &lt;em&gt;states&lt;/em&gt; in which the subjects can be found and the ages at which they transition from one to another.  In this situation a minimum set of states would be&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;[&lt;strong&gt;Born&lt;/strong&gt;]: &quot;Born with gene.&quot;  This always happens at age 0, of course.&lt;/li&gt;&#xA;&lt;li&gt;[&lt;strong&gt;Enrolled&lt;/strong&gt;]: &quot;Enrolled in study.&quot;&lt;/li&gt;&#xA;&lt;li&gt;[&lt;strong&gt;Endpoint&lt;/strong&gt;]: &quot;Cardiovascular event identified.&quot;&lt;/li&gt;&#xA;&lt;li&gt;[&lt;strong&gt;Death&lt;/strong&gt;]: &quot;Death.&quot;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;(This framework will allow multiple &quot;endpoint&quot; states to be modeled.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;em&gt;multistate&lt;/em&gt; analysis supposes there is a transition probability from some of these states to others.  The relevant ones would be&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;[&lt;strong&gt;Born&lt;/strong&gt;] --&gt; [&lt;strong&gt;Death&lt;/strong&gt;].  These account for people who never enrolled.&lt;/li&gt;&#xA;&lt;li&gt;[&lt;strong&gt;Born&lt;/strong&gt;] --&gt; [&lt;strong&gt;Endpoint&lt;/strong&gt;].  Are you considering these people?  Are they even allowed into the study?&lt;/li&gt;&#xA;&lt;li&gt;[&lt;strong&gt;Born&lt;/strong&gt;] --&gt; [&lt;strong&gt;Enrolled&lt;/strong&gt;].  These are all the people selected for the study (who haven't died and don't already exhibit the cardiovascular disease).&lt;/li&gt;&#xA;&lt;li&gt;[&lt;strong&gt;Enrolled&lt;/strong&gt;] --&gt; [&lt;strong&gt;Endpoint&lt;/strong&gt;].  These are people in the study diagnosed with a cardiovascular disease.&lt;/li&gt;&#xA;&lt;li&gt;[&lt;strong&gt;Enrolled&lt;/strong&gt;] --&gt; [&lt;strong&gt;Death&lt;/strong&gt;].  These people died in the study without a diagnosis of cardiovascular disease.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The &lt;a href=&quot;http://en.wikipedia.org/wiki/Nelson%E2%80%93Aalen_estimator&quot; rel=&quot;nofollow&quot;&gt;Nelson-Aalen estimator&lt;/a&gt; can be generalized to estimate the rates of these transitions.  It's a simple estimator, summing the ratios of events occurring to the numbers of people at risk for them to occur.  The conclusion of the recent TAS article &lt;a href=&quot;http://pubs.amstat.org/doi/abs/10.1198/tast.2010.08259&quot; rel=&quot;nofollow&quot;&gt;Two Pitfalls in Survival Analyses of Time-Dependent Exposure&lt;/a&gt; is that if you get your multistate model wrong, you will miscount the number of people at risk in various states and that will bias the results.  Its message is clear: get the multistate model right.  If the study truly is prospective--that is, if you identify people with the gene at birth and follow them--then there is no question about the right model.  Similarly, if enrollment in the study is independent of the presence of the gene, there will be no bias.  Otherwise, this framework calls out for incorporating the study &lt;em&gt;selection probabilities&lt;/em&gt; into the model and shows how to account for deaths and prior disease before enrollment was possible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This paper also illustrates a nice tool for analyzing these subtleties: the &lt;a href=&quot;http://www.jlund.dk/files/master_thesis/lexissjs.ps.gz&quot; rel=&quot;nofollow&quot;&gt;Lexis Diagram&lt;/a&gt;.  (Look at the figures in the end of this rather technical paper.)  I believe these diagrams can be produced with the &lt;strong&gt;&lt;a href=&quot;http://cran.r-project.org/web/packages/Epi/index.html&quot; rel=&quot;nofollow&quot;&gt;epi&lt;/a&gt;&lt;/strong&gt; package in R.  You might find them helpful for having discussions with your colleagues about the appropriate model to adopt.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ASA members and people with university library privileges probably already have online access to this article: it's worth reading.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-10-29T19:58:17.210" CommentCount="4" />
  <row Id="4074" PostTypeId="2" ParentId="4072" CreationDate="2010-10-29T19:58:28.380" Score="3" ViewCount="" Body="&lt;p&gt;Don't try to do too much at once.  Note, too:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;'Predict' without options computes the &lt;em&gt;probabilities.&lt;/em&gt;  I don't think you want those for this calculation.  Use 'predict, xb' to obtain the linear fits.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Subtract a multiple of the standard error of prediction (obtained via 'predict, stdp') from the linear prediction.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;em&gt;Now&lt;/em&gt; transform the limit with a logistic transformation to convert it to a probability.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Compare that to .75 (or whatever).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;At each step of the way summarize (and even graph) the results to make sure they are behaving as you expect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's actually simpler, if not quite as clear, to replace the last two steps by a direct comparison of the limit to the logit of 0.75, defined as ln(.75) - ln(.25) = 1.099.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-10-29T20:44:13.563" LastActivityDate="2010-10-29T20:44:13.563" CommentCount="7" />
  <row Id="4075" PostTypeId="1" AcceptedAnswerId="4079" CreationDate="2010-10-30T07:55:10.057" Score="5" ViewCount="329" Body="&lt;p&gt;Hi... I have two populations, One with N (no of observations)=38704 and other with N=1313662. These data sets have some 25 variables, all continuous. I took mean of each in each data set and computed the test statistic using the formula &#xA;(t=mean difference/std error). The problem is of the degree of freedom. By formula of df=N1+N2-2 we'll have more freedom than the table can handle. Any suggestions on this? How to check the t statistic here. I know that the t-test is used for handling samples but what if we apply this on large samples.&lt;/p&gt;&#xA;" OwnerUserId="1763" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-10-30T16:29:44.363" LastActivityDate="2010-10-30T19:37:59.957" Title="How to perform t-test with huge samples?" Tags="&lt;t-test&gt;" AnswerCount="3" FavoriteCount="3" />
  <row Id="4076" PostTypeId="2" ParentId="4075" CreationDate="2010-10-30T08:40:46.110" Score="7" ViewCount="" Body="&lt;p&gt;The $t$ distribution tend to the $z$ (gaussian) distribution when $n$ is large (in fact, when $n&gt;30$, they are almost identical, see the picture provided by @onestop). In your case, I would say that $n$ is VERY large, so that you can just use a $z$-test. As a consequence of the sample size, any VERY small differences will be declared significant. So, it is worth asking yourself if these tests (with the full data set) are really interesting. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just to be sure, as your data set includes 25 variables, you are making 25 tests? If this is the case, you probably need to correct for multiple comparisons so as not to inflate the type I error rate (see related thread on this site).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;BTW, the R software would gives you the p-values you are looking for, no need to rely on Tables:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; x1 &amp;lt;- rnorm(n=38704)&#xA;&amp;gt; x2 &amp;lt;- rnorm(n=1313662, mean=.1)&#xA;&amp;gt; t.test(x1, x2, var.equal=TRUE)&#xA;&#xA;    Two Sample t-test&#xA;&#xA;data:  x1 and x2 &#xA;t = -17.9156, df = 1352364, p-value &amp;lt; 2.2e-16&#xA;alternative hypothesis: true difference in means is not equal to 0 &#xA;95 percent confidence interval:&#xA; -0.1024183 -0.0822190 &#xA;sample estimates:&#xA;  mean of x   mean of y &#xA;0.007137404 0.099456039 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-30T13:09:02.090" LastActivityDate="2010-10-30T13:09:02.090" />
  <row Id="4077" PostTypeId="2" ParentId="4075" CreationDate="2010-10-30T08:41:36.613" Score="9" ViewCount="" Body="&lt;p&gt;Student's &lt;em&gt;t&lt;/em&gt;-distribution becomes closer and closer the the standard normal distribution as the degrees of freedom get larger. With  1313662 + 38704 – 2 = 1352364 degrees of freedom, the &lt;em&gt;t&lt;/em&gt;-distribution will be indistinguishable from the standard normal distribution, as can be seen in the picture below (unless perhaps you're in the very extreme tails and you're interested in distinguishing absolutely tiny &lt;em&gt;p&lt;/em&gt;-values from even tinier ones). So you can use the table for the standard normal distribution instead of the table for the &lt;em&gt;t&lt;/em&gt;-distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/r7sSi.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="449" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-30T13:09:47.210" LastActivityDate="2010-10-30T13:09:47.210" CommentCount="7" />
  <row Id="4078" PostTypeId="2" ParentId="1562" CreationDate="2010-10-30T14:56:29.207" Score="6" ViewCount="" Body="&lt;p&gt;There's nothing wrong with the existing answers, but I suspect that you're looking for a &lt;em&gt;causal&lt;/em&gt; sense of dependence rather than an associational one, that is: whether A causes B rather than whether B is more predictable when you know A.  The chi^2 test is working with the second sense.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even in the simplest case of the first kind of dependence you would ideally experimentally manipulate B and observe the effect on A and vice versa.  Judea Pearl points out that this is the difference between the ordinary sense of conditional probability &lt;/p&gt;&#xA;&#xA;&lt;p&gt;P(&quot;I observe that A has value a&quot; | &quot;I observe that B has value b&quot;) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;and a quite different thing that we might slightly misleadingly write as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P(&quot;I observe that A has value a&quot; | &quot;I &lt;em&gt;fix&lt;/em&gt; B to have value b&quot;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These need not, of course, be the same number.&lt;/p&gt;&#xA;" OwnerUserId="1739" LastActivityDate="2010-10-30T14:56:29.207" CommentCount="1" />
  <row Id="4079" PostTypeId="2" ParentId="4075" CreationDate="2010-10-30T19:37:59.957" Score="8" ViewCount="" Body="&lt;p&gt;chl already mentioned the trap of multiple comparisons when conducting simultaneously 25 tests with the same data set.  An easy way to handle that is to adjust the p value threshold by dividing them by the number of tests (in this case 25).  The more precise formula is: Adjusted p value = 1 - (1 - p value)^(1/n).  However, the two different formulas derive almost the same adjusted p value. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is another major issue with your hypothesis testing exercise.  You will most certainly run into a Type I error (false positive) whereby you will uncover some really trivial differences that are extremely significant at the 99.9999% level.  This is because when you deal with a sample of such a large size (n = 1,313,662), you will get a standard error that is very close to 0.  That's because the square root of 1,313,662 = 1,146.  So, you will divide the standard deviation by 1,146.  In short, you will capture minute differences that may be completely immaterial.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would suggest you move away from this hypothesis testing framework and instead conduct an Effect Size type analysis.  Within this framework the measure of statistical distance is the standard deviation.  Unlike the standard error, the standard deviation is not artificially shrunk by the size of the sample.  And, this approach will give you a better sense of the material differences between your data sets.  Effect Size is also much more focused on confidence interval around the mean average difference which is much more informative than the hypothesis testing focus on statistical significance that often is not significant at all.  Hope that helps.           &lt;/p&gt;&#xA;" OwnerUserId="1329" LastActivityDate="2010-10-30T19:37:59.957" CommentCount="4" />
  <row Id="4080" PostTypeId="1" AcceptedAnswerId="4090" CreationDate="2010-10-30T21:23:43.047" Score="4" ViewCount="118" Body="&lt;p&gt;I asked this question at Mathoverflow but they recommend me to ask here. I need to find factored joint distribution of &lt;a href=&quot;http://www.cs.huji.ac.il/~nir/Abstracts/FrGG1.html&quot; rel=&quot;nofollow&quot;&gt;Tree Augmented Naive Bayes&lt;/a&gt; algorithm. I read the paper but I couldn't figure out the answer. Any help or pointers appreciated.&lt;/p&gt;&#xA;" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-11-01T08:50:52.740" LastActivityDate="2010-11-01T08:50:52.740" Title="Factored Joint Distribution of Tree Augmented Naive Bayes Algorithm" Tags="&lt;machine-learning&gt;&lt;naive-bayes&gt;" AnswerCount="1" FavoriteCount="2" />
  <row Id="4081" PostTypeId="1" AcceptedAnswerId="4082" CreationDate="2010-10-30T21:31:17.513" Score="6" ViewCount="226" Body="&lt;p&gt;I had been using the term &quot;Heywood Case&quot; somewhat informally to refer to situations where an online, 'finite response' iteratively updated estimate of the variance became negative due to numerical precision issues. (I am using a variant of Welford's method to add data and to remove older data.)  I was under the impression that it applied to any situation where a variance estimate became negative, either due to numerical error or modeling error, but a colleague was confused by my usage of the term. A google search doesn't turn up much, other than that it is used in Factor Analysis, and seems to refer to the consequences of a negative variance estimate. What is the precise definition? And who was the original Heywood?&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-10-30T22:22:12.180" Title="What is the precise definition of a &quot;Heywood Case&quot;?" Tags="&lt;variance&gt;&lt;factor-analysis&gt;&lt;definition&gt;&lt;online&gt;" AnswerCount="1" />
  <row Id="4082" PostTypeId="2" ParentId="4081" CreationDate="2010-10-30T22:22:12.180" Score="5" ViewCount="" Body="&lt;p&gt;Googling &quot;&lt;a href=&quot;http://www.google.com/search?q=heywood+negative+variance&quot; rel=&quot;nofollow&quot;&gt;Heywood negative variance&lt;/a&gt;&quot; quickly answers these questions.  Looking at a recent (2008) &lt;a href=&quot;http://web.missouri.edu/~kolenikovs/papers/heywood-8.pdf&quot; rel=&quot;nofollow&quot;&gt;paper by Kolenikov &amp;amp; Bollen&lt;/a&gt;, for example, indicates that:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&quot; “Heywood cases” [are] negative estimates of variances or correlation estimates greater than one in absolute value...&quot;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&quot;The original paper (Heywood 1931) considers specific parameterizations of factor analytic models, in which some parameters necessary to describe the correlation matrices were greater than 1.&quot;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Reference&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Heywood, H. B. (1931), ‘On finite sequences of real numbers’, Proceedings of the Royal Society of London.  Series A, Containing Papers of a Mathematical and Physical Character 134(824), 486–501.&quot;&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-10-30T22:22:12.180" CommentCount="4" />
  <row Id="4083" PostTypeId="2" ParentId="3893" CreationDate="2010-10-31T05:14:12.747" Score="0" ViewCount="" Body="&lt;p&gt;I thank everyone for their answers, but the question has grown to something I did not intend it to, being mainly an essay on the general notion of causal inference with no right answer. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I initially intended the question to probe the audience for examples of the use of cross validation for causal inference. I had assumed such methods existed, as the notion of using a test sample and hold out sample to assess repeatability of effect estimates seemed logical to me. Like John noted, what I was suggesting isn't dissimilar to bootstrapping, and I would say it resembles other methods we use to validate results such as subset specificity tests or non-equivalent dependent variables (bootstrapping relaxes parametric assumptions of models, and the subset tests in a more general manner are used as a check that results are logical in varied situations). None of these methods meets any of the other answers standards of proof for causal inference, but I believe they are still useful for causal inference.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;chl's comment is correct in that my assertion for using cross validation is a check on internal validity to aid in causal inference. But I ask we throw away the distinction between internal and external validity for now, as it does nothing to further the debate. chl's example of genome wide studies in epidemiology I would consider a prime example of poor internal validity, making strong inferences inherently dubious. I think the genome association studies are actually an example of what I asked for. Do you think the inferences between genes and disease are improved through the use of cross-validation (as oppossed to just throwing all markers into one model and adjusting p-values accordingly?)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Below I have pasted a copy of a table in the Berk article I cited in my question. While these tables were shown to demonstrate the false logic of using step-wise selection criteria and causal inference on the same model, lets pretend no model selection criteria were used, and the parameters in both the training and hold out sample were determined &lt;em&gt;A priori&lt;/em&gt;. This does not strike me as an unrealistic result. Although I could not say which estimate is correct and which is false, doesn't the inconsistency in the Assault Conviction and the Gun conviction estimates between the two models cast doubt that either has a true causal effect on sentence length? Is knowing that variation not useful? If we lose nothing by having a hold out sample to test our model why can't we use cross-validation to improve causal inference (or I am missing what we are losing by using a hold out sample?)&#xA;&lt;img src=&quot;http://i.imgur.com/0UUyE.jpg&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="1036" LastEditorUserId="1036" LastEditorDisplayName="" LastEditDate="2010-10-31T05:22:56.657" LastActivityDate="2010-10-31T05:22:56.657" CommunityOwnedDate="2010-10-31T05:14:12.747" CommentCount="2" />
  <row Id="4084" PostTypeId="1" AcceptedAnswerId="4085" CreationDate="2010-10-31T14:09:43.783" Score="9" ViewCount="222" Body="&lt;p&gt;I've got a large set of data (20,000 data points), from which I want to take repeated samples of 10 data points. However, once I've picked those 10 data points, I want them to not be picked again.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've tried using the &lt;code&gt;sample&lt;/code&gt; function, but it doesn't seem to have an option to sample without replacement over multiple calls of the function. Is there a simple way to do this?&lt;/p&gt;&#xA;" OwnerUserId="261" LastActivityDate="2010-11-01T09:53:00.917" Title="How to take many samples of 10 from a large list, without replacement overall" Tags="&lt;r&gt;&lt;sample&gt;" AnswerCount="3" FavoriteCount="4" />
  <row Id="4085" PostTypeId="2" ParentId="4084" CreationDate="2010-10-31T14:55:22.850" Score="7" ViewCount="" Body="&lt;p&gt;You could call sample once on the entire data set to permute it.  Then when you want to get a sample you could grab the first 10.  If you want another sample grab the next 10.  So on and so forth.&lt;/p&gt;&#xA;" OwnerUserId="1028" LastActivityDate="2010-10-31T14:55:22.850" CommentCount="0" />
  <row Id="4086" PostTypeId="1" AcceptedAnswerId="4096" CreationDate="2010-10-31T15:21:13.410" Score="5" ViewCount="385" Body="&lt;p&gt;I am not that familiar with the analysis of time series data.  However, I have what I think is a simple prediction task to address.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have about five years of data from a common generating process.  Each year represents a monotonically increasing function with a non-linear component.  I have counts for each week over a 40 week cycle for each year.  The process starts, the function begins at zero, increases rather quickly over the first half of the function, slowing over the second half before leveling during the last five weeks.  The process is consistent across years with small differences in rate of change and volume across the segments from year to year.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;y1={0, Nt1, Nt2, ... Nt39, Nt40)&#xA;...&#xA;...&#xA;y5={0, Nt1, Nt2, ... Nt39, Nt40)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where Ntx equal the count at time x.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The goal is to take the N at tx (or better t0 to tx, or the slope to that point) and predict the N at t40.  For example, if Nt10 is 5000 what is the expected value of Nt40 for that year.  So, the question is, how would you model such data?  It's easy enough to summarize and visualize.  But I'd like a model to facilitate predictions and incorporate a measure of error.&lt;/p&gt;&#xA;" OwnerUserId="485" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-11-01T01:20:52.790" LastActivityDate="2011-03-21T23:19:35.030" Title="Analyzing Time Series" Tags="&lt;time-series&gt;&lt;forecasting&gt;" AnswerCount="3" CommentCount="2" />
  <row Id="4087" PostTypeId="2" ParentId="4084" CreationDate="2010-10-31T16:32:55.587" Score="2" ViewCount="" Body="&lt;p&gt;This should work:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x &amp;lt;- rnorm(20000)&#xA;x.copy &amp;lt;- x&#xA;samples &amp;lt;- list()&#xA;i &amp;lt;- 1&#xA;while (length(x) &amp;gt;= 10){&#xA;    tmp &amp;lt;- sample(x, 10)&#xA;    samples[[i]] &amp;lt;- tmp&#xA;    i &amp;lt;- i+1&#xA;    x &amp;lt;- x[-match(tmp, x)]&#xA;}&#xA;&#xA;table(unlist(samples) %in% x.copy)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, I don't think that's the most elegant solution...&lt;/p&gt;&#xA;" OwnerUserId="307" LastActivityDate="2010-10-31T16:32:55.587" />
  <row Id="4088" PostTypeId="2" ParentId="4084" CreationDate="2010-10-31T18:48:59.330" Score="8" ViewCount="" Body="&lt;p&gt;Dason's thought, implemented in R: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sample &amp;lt;- split(sample(datapoints), rep(1:(length(datapoints)/10+1), each=10))&#xA;sample[[13]] # the thirteenth sample&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1739" LastEditorUserId="1739" LastEditorDisplayName="" LastEditDate="2010-11-01T09:53:00.917" LastActivityDate="2010-11-01T09:53:00.917" CommentCount="4" />
  <row Id="4089" PostTypeId="1" CreationDate="2010-10-31T19:17:24.210" Score="12" ViewCount="553" Body="&lt;p&gt;I'm sure I've come across a function like this in an R package before, but after extensive Googling I can't seem to find it anywhere. The function I'm thinking of produced a graphical summary for a variable given to it, producing output with some graphs (a histogram and perhaps a box and whisker plot) and some text giving details like mean, SD etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm pretty sure this function wasn't included in base R, but I can't seem to find the package I used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone know of a function like this, and if so, what package it is in?&lt;/p&gt;&#xA;" OwnerUserId="261" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-24T11:32:48.747" LastActivityDate="2011-05-06T22:08:51.820" Title="Graphical data overview (summary) function in R" Tags="&lt;r&gt;&lt;data-visualization&gt;&lt;summary-statistics&gt;" AnswerCount="6" FavoriteCount="8" />
  <row Id="4090" PostTypeId="2" ParentId="4080" CreationDate="2010-10-31T19:26:30.673" Score="1" ViewCount="" Body="&lt;p&gt;Where $A_1$, $A_2$, ..., $A_n$ are the attribute variables and $C$ is the class variable, it's:  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(C) \prod_{i=1}^{n} P(A_i | A_{\pi(i)}, C)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $\pi(i)$ is the index of the parent of $A_i$ in the tree or $0$ if $i$ is the root and $P(A_i | A_0, C)$ is defined to be $P(A_i | C)$.&lt;/p&gt;&#xA;" OwnerUserId="1756" LastActivityDate="2010-10-31T19:26:30.673" CommentCount="1" />
  <row Id="4091" PostTypeId="2" ParentId="4086" CreationDate="2010-10-31T21:10:25.720" Score="3" ViewCount="" Body="&lt;p&gt;What your asking is essentially what Box Jenkins ARIMA modeling does (your yearly cycles would be referred to as seasonal components). Besides looking up materials on your own, I would suggest &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://books.google.com/books?id=D6-CAAAAIAAJ&amp;amp;q=Applied+Time+Series+Analysis+for+the+Social+Sciences&amp;amp;dq=Applied+Time+Series+Analysis+for+the+Social+Sciences&amp;amp;hl=en&amp;amp;ei=QaLNTKD-HcH88Aabt-Al&amp;amp;sa=X&amp;amp;oi=book_result&amp;amp;ct=result&amp;amp;resnum=1&amp;amp;ved=0CC8Q6AEwAA&quot; rel=&quot;nofollow&quot;&gt;Applied Time Series Analysis for the Social Sciences&lt;/a&gt; 1980&#xA;by R McCleary ; R A Hay ; E E Meidinger ; D McDowall&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Although I can think of reasonable reasons for why you want to forecast further into the future (and hence assess the error when doing so) it is often very difficult in practice. If you have very strong seasonal components it will be more feasible. Otherwise your estimates will likely reach an equilibrium in relatively few future time periods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you plan on using R to fit your models you should probably check out Rob Hyndman's &lt;a href=&quot;http://robjhyndman.com/&quot; rel=&quot;nofollow&quot;&gt;website&lt;/a&gt; (Hopefully he will give you better advice than me!)&lt;/p&gt;&#xA;" OwnerUserId="1036" LastEditorUserId="1036" LastEditorDisplayName="" LastEditDate="2010-10-31T21:17:30.207" LastActivityDate="2010-10-31T21:17:30.207" />
  <row Id="4092" PostTypeId="2" ParentId="4089" CreationDate="2010-10-31T21:13:06.657" Score="13" ViewCount="" Body="&lt;p&gt;Frank Harrell's &lt;a href=&quot;http://cran.r-project.org/web/packages/Hmisc/index.html&quot; rel=&quot;nofollow&quot;&gt;Hmisc&lt;/a&gt; package has some basic graphics with options for annotation. But I think that was you're looking for is in the &lt;a href=&quot;http://cran.r-project.org/web/packages/Design/index.html&quot; rel=&quot;nofollow&quot;&gt;Design&lt;/a&gt; package: check the &lt;code&gt;summary.formula()&lt;/code&gt; and related &lt;code&gt;plot&lt;/code&gt; wrap functions. I also like the &lt;code&gt;describe()&lt;/code&gt; function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For additional information, have a look at the &lt;a href=&quot;http://lib.stat.cmu.edu/S/Harrell/Hmisc.html&quot; rel=&quot;nofollow&quot;&gt;The Hmisc Library&lt;/a&gt; or &lt;a href=&quot;http://biostat.mc.vanderbilt.edu/twiki/pub/Main/RS/sintro.pdf&quot; rel=&quot;nofollow&quot;&gt;An Introduction to S-Plus and the Hmisc and Design Libraries&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are some pictures taken from the on-line help (&lt;code&gt;bpplt&lt;/code&gt;, &lt;code&gt;describe&lt;/code&gt;, and &lt;code&gt;plot(summary(...))&lt;/code&gt;):&#xA;&lt;img src=&quot;http://i.imgur.com/KddtF.png&quot; alt=&quot;alt text&quot;&gt;&#xA;&lt;img src=&quot;http://i.imgur.com/veycp.png&quot; alt=&quot;alt text&quot;&gt;&#xA;&lt;img src=&quot;http://i.imgur.com/lgUrm.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many other examples can be browsed on-line on the &lt;a href=&quot;http://rgm2.lab.nig.ac.jp/RGM2&quot; rel=&quot;nofollow&quot;&gt;R Graphical Manual&lt;/a&gt;, see &lt;a href=&quot;http://rgm2.lab.nig.ac.jp/RGM2/packages.php?query=package%3aHmisc&quot; rel=&quot;nofollow&quot;&gt;Hmisc&lt;/a&gt; and &lt;a href=&quot;http://rgm2.lab.nig.ac.jp/RGM2/packages.php?query=package%3aDesign&quot; rel=&quot;nofollow&quot;&gt;Design&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-01T09:33:06.527" LastActivityDate="2010-11-01T09:33:06.527" />
  <row Id="4093" PostTypeId="1" CreationDate="2010-10-31T21:18:20.970" Score="6" ViewCount="682" Body="&lt;p&gt;Just wondering if anyone can help me interpret PCA scores. So my data comes from a questionnaire on atititudes toward bears. According to the loadings, I have interpreted one of my principal components as &quot;fear of bears&quot;. Then would the scores of that principal component be related to how each respondent measures up to that principal component (whether he/she scores positively/negatively on it)? &#xA;Any help would be much appreciated! Thank you.&#xA;A&lt;/p&gt;&#xA;" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-05T14:37:15.737" LastActivityDate="2010-11-05T14:37:15.737" Title="Interpreting PCA scores" Tags="&lt;pca&gt;&lt;scales&gt;" AnswerCount="3" FavoriteCount="8" />
  <row Id="4094" PostTypeId="2" ParentId="4093" CreationDate="2010-10-31T21:34:24.127" Score="4" ViewCount="" Body="&lt;p&gt;Basically, the factor scores are computed as the raw responses weighted by the factor loadings. So, you need to look at the factor loadings of your first dimension to see how each variable relate to the principal component. Observing high positive (resp. negative) loadings associated to specific variables means that these variables contribute positively (resp. negatively) to this component; hence, people scoring high on these variables will tend to have higher (resp. lower) factor scores on this particular dimension.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Drawing the correlation circle is useful to have a general idea of the variables that contribute &quot;positively&quot; vs. &quot;negatively&quot; (if any) to the first principal axis, but if you are using R you may have a look at the &lt;a href=&quot;http://cran.r-project.org/web/packages/FactoMineR/index.html&quot; rel=&quot;nofollow&quot;&gt;FactoMineR&lt;/a&gt; package and the &lt;code&gt;dimdesc()&lt;/code&gt; function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an example with the &lt;code&gt;USArrests&lt;/code&gt; data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; data(USArrests)&#xA;&amp;gt; library(FactoMineR)&#xA;&amp;gt; res &amp;lt;- PCA(USArrests)&#xA;&amp;gt; dimdesc(res, axes=1)  # show correlation of variables with 1st axis&#xA;$Dim.1&#xA;$Dim.1$quanti&#xA;         correlation  p.value&#xA;Assault        0.918 5.76e-21&#xA;Rape           0.856 2.40e-15&#xA;Murder         0.844 1.39e-14&#xA;UrbanPop       0.438 1.46e-03&#xA;&amp;gt; res$var$coord  # show loadings associated to each axis&#xA;         Dim.1  Dim.2  Dim.3   Dim.4&#xA;Murder   0.844 -0.416  0.204  0.2704&#xA;Assault  0.918 -0.187  0.160 -0.3096&#xA;UrbanPop 0.438  0.868  0.226  0.0558&#xA;Rape     0.856  0.166 -0.488  0.0371&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As can be seen from the latest result, the first dimension mainly reflects violent acts (of any kind). If we look at the individual map, it is clear that states located on the right are those where such acts are most frequent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/uOgTV.png&quot; alt=&quot;alt text&quot;&gt;&#xA;&lt;img src=&quot;http://i.imgur.com/fnp7W.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may also be interested in this related question: &lt;a href=&quot;http://stats.stackexchange.com/questions/222/what-are-principal-component-scores&quot;&gt;What are principal component scores?&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-10-31T22:01:49.937" LastActivityDate="2010-10-31T22:01:49.937" />
  <row Id="4095" PostTypeId="2" ParentId="3893" CreationDate="2010-10-31T22:40:29.637" Score="2" ViewCount="" Body="&lt;p&gt;I think it's useful to review what we know about cross-validation.  Statistical results around CV fall into two classes: efficiency and consistency.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Efficiency is what we're usually concerned with when building predictive models.  The idea is that we use CV to determine a model with asymtptotic guarantees concerning the loss function.  The most famous result here is due to &lt;a href=&quot;http://www.jstor.org/pss/2984877&quot; rel=&quot;nofollow&quot;&gt;Stone 1977&lt;/a&gt; and shows that LOO CV is asymptotically equivalent to AIC.  But, Brett provides a good example where you can find a predictive model which doesn't inform you on the causal mechanism.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consistency is what we're concerned with if our goal is to find the &quot;true&quot; model.  The idea is that we use CV to determine a model with asymptotic guarantees that, given that our model space includes the true model, we'll discover it with a large enough sample.  The most famous result here is due to &lt;a href=&quot;http://www.jstor.org/pss/2290328&quot; rel=&quot;nofollow&quot;&gt;Shao 1993&lt;/a&gt; concerning linear models, but as he states in his abstract, his &quot;shocking discovery&quot; is opposite of the result for LOO.  For linear models, you can achieve consistency using LKO CV as long as $k/n \rightarrow 1$ as $n \rightarrow \infty$.  Beyond linear mdoels, it's harder to derive statistical results.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But suppose you can meet the consistency criteria and your CV procedure leads to the true model: $Y = \beta X + e$.  What have we learned about the causal mechanism?  We simply know that there's a well defined correlation between $Y$ and $X$, which doesn't say much about causal claims.  From a traditional perspective, you need to bring in experimental design with the mechanism of control/manipulation to make causal claims.  From the perspective of Judea Pearl's framework, you can bake causal assumptions into a structural model and use the probability based calculus of counterfactuals to derive some claims, but you'll need to satisfy &lt;a href=&quot;http://bayes.cs.ucla.edu/BOOK-2K/jw.html&quot; rel=&quot;nofollow&quot;&gt;certain properties&lt;/a&gt;.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Perhaps you could say that CV can help with causal inference by identifying the true model (provided you can satisfy consistency criteria!).  But it only gets you so far; CV by itself isn't doing any of the work in either framework of causal inference.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're interested further in what we can say with cross-validation, I would recommend Shao 1997 over the widely cited 1993 paper:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www3.stat.sinica.edu.tw/statistica/j7n2/j7n21/j7n21.htm&quot; rel=&quot;nofollow&quot;&gt;An Asymptotic Theory for Linear Model Selection&lt;/a&gt; (Shao, 1997)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You can skim through the major results, but it's interesting to read the discussion that follows.  I thought the comments by Rao &amp;amp; Tibshirani, and by Stone, were particularly insightful.  But note that while they discuss consistency, no claims are ever made regarding causality.&lt;/p&gt;&#xA;" OwnerUserId="251" LastActivityDate="2010-10-31T22:40:29.637" CommentCount="2" />
  <row Id="4096" PostTypeId="2" ParentId="4086" CreationDate="2010-11-01T01:18:37.127" Score="4" ViewCount="" Body="&lt;p&gt;Probably the simplest approach is, as Andy W suggested, to use a seasonal univariate time series model. If you use R, try either &lt;code&gt;auto.arima()&lt;/code&gt; or &lt;code&gt;ets()&lt;/code&gt; from the &lt;a href=&quot;http://cran.r-project.org/web/packages/forecast/&quot; rel=&quot;nofollow&quot;&gt;forecast package&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Either should work ok, but a general time series method does not use all the information provided. In particular, it seems that you know the shape of the curve in each year, so it might be better to use that information by modelling each year's data accordingly. What follows is a suggestion that tries to incorporate this information.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It sounds like some kind of sigmoidal curve will do the trick. e.g., a shifted logistic:&#xA;\begin{equation}&#xA;f_{t,j} = \frac{r_te^{a_t(j-b_t)}}{1+e^{a_t(j-b_t)}}&#xA;\end{equation}&#xA;for year $t$ and week $j$ where $a_t$, $b_t$ and $r_t$ are parameters to be estimated. $r_t$ is the asymptotic maximum, $a_t$ controls the rate of increase and $b_t$ is the mid-point when $f_{t,j}=r_t/2$. (Another parameter will be needed to allow the asymmetry you describe whereby the rate of increase up to time $b_t$ is faster than that after $b_t$. The simplest way to do this is to allow $a_t$ to take different values before and after time $b_t$.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The parameters can be estimated using least squares for each year. The parameters each form time series: ${a_1,\dots,a_n}$, ${b_1,\dots,b_n}$ and ${r_1,\dots,r_n}$.  These can be forecast using standard time series methods, although with $n=5$ you probably can't do much apart from using the mean of each series for producing forecasts. Then, for year 6, an estimate of the value at week $j$ is simply $\hat{f}(6,j)$ where the forecasts of $a_6$, $b_6$ and $r_6$ are used. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once data start to be observed for year 6 you will want to update this estimate. As each new observation is obtained, estimate the sigmoidal curve to the data from year 6 (you will need at least three observations to start with as there are three parameters). Then take a weighted average of the forecasts obtained using the data up to year 5 and the forecast obtained using only the data from year 6, where the weights are equal to $(40-t)/36$ and $(t-4)/36$ respectively. That is very ad hoc, and I'm sure it can be made more objective by placing it in the context of a larger stochastic model. Nevertheless, it will probably work ok for your purposes.&lt;/p&gt;&#xA;" OwnerUserId="159" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-11-01T03:18:34.553" LastActivityDate="2010-11-01T03:18:34.553" />
  <row Id="4097" PostTypeId="2" ParentId="3260" CreationDate="2010-11-01T03:15:58.853" Score="2" ViewCount="" Body="&lt;p&gt;I will outline an approach that requires no &quot;training&quot; at all; it is up to you to determine its utility in this case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A simple (and nonparametric) hypothetical model is that all datasets are independent, that none has a trend, and that their variations from one time period to the next are mutually independent.  This implies the probability with which two pre-specified datasets simultaneously have local minima would equal the product of the probabilities with which each has local minima, with obvious (but more complex) generalizations to three or more pre-specified datasets (which I illustrate below).  In particular, you can estimate the &lt;em&gt;probabilities&lt;/em&gt; of local minima by means of their observed &lt;em&gt;frequencies&lt;/em&gt; in each dataset.  From these you can compute the probabilities of simultaneous local minima among 2, 3, or, generally, $k$ or more of the datasets.  When the probability for $k$ or more is so low that it is unlikely to occur during the time span you have observed, you can take the simultaneous occurrence of $k$ or more local minima to be &quot;significant&quot; relative to this null hypothesis of independence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, suppose you have five datasets, each observed 100 times, with local minima appearing 8, 9, 10, 11, and 12 times in them.  All five would simultaneously exhibit a local minimum (8/100) * (9/100) * (10/100) * (11/100) * (12/100) = 0.00095% of the time, so even within 100 observations the expected number of simultaneous minima (of 100 * 0.00095% = 0.00095) is so ridiculously low that five simultaneous minima surely would be significant evidence of an &quot;interesting&quot; point.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Local minima among the first four datasets (unaccompanied by a local minimum in the fifth) would have an expected frequency of 100 * (8/100) * (9/100) * (10/100) * (11/100) * ((100-12)/100) = .00697.  Similarly we could compute the expected frequency of local minima among the other combinations of four of the datasets.  The total frequency of exactly four simultaneous minima is 0.04375. Added to the frequency of five simultaneous minima this gives 0.0447 as the expected number of times you would observe four &lt;em&gt;or&lt;/em&gt; five simultaneous local minima in 100 observations: still pretty rare and therefore significant if it turns up.  A similar computation for the ten combinations of three simultaneous local minima shows that you would expect &lt;em&gt;at least&lt;/em&gt; three local minima 0.8452 times out of 100.  So, observing one or two such events would not be unusual and you might not consider them significant.  Obviously the expected number of two-way minima would be substantial (you should expect to see around 40 of them out of 100) and you would be unlikely to consider any of those significant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The example illustrates how you could go about computing thresholds for significance in terms of the number of simultaneous local minima for any number of datasets that are observed for any number of time periods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can give a more precise accounting of the situation by means of the Poisson distribution.  Take the occurrence of four or more simultaneous minima in the example.  Under the null hypothesis (of independent datasets), this is rare enough that the actual count should have a Poisson distribution with expectation 0.8452.  This implies there is a 94.59% chance of observing two or fewer such events.  Thus, if you see three or more three- or four- or five-way minima you could take this to be significant evidence of lack of independence (with about 95% confidence).  However, in this case you could not point to a specific time that is significant; you could only say that there are more threefold minima than there should be.  Any one of them would be a reasonable candidate for an &quot;interesting&quot; time, but further investigation should ensue before you stipulate that any &lt;em&gt;particular&lt;/em&gt; one of these times really demonstrates a departure from independence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This model might or might not be appropriate for your data.  You can check that by examining the data.  If your data have trends or exhibit serial correlation you would need a more complex version of this model.  Nevertheless, the same kind of analysis can help you decide what constitutes an &quot;interesting&quot; or &quot;significant&quot; syzygy of local minima.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-11-01T03:15:58.853" />
  <row Id="4098" PostTypeId="2" ParentId="573" CreationDate="2010-11-01T03:29:42.770" Score="5" ViewCount="" Body="&lt;p&gt;The audio application is a one-dimensional simplification of the two-dimensional image classification problem.  A phoneme (for example) is the audio analog of an image feature such as an edge or a circle.  In either case such features have an essential locality: they are characterized by values within a relatively small neighborhood of an image location or moment of speech.  Convolutions are a controlled, regular form of weighted averaging of values within local neighborhoods.  From this originates the hope that a &lt;em&gt;convolutional&lt;/em&gt; form of a DBN can be successful at identifying and discriminating features that are meaningful.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-11-01T03:29:42.770" CommentCount="1" />
  <row Id="4099" PostTypeId="1" CreationDate="2010-11-01T10:19:44.367" Score="7" ViewCount="590" Body="&lt;p&gt;I am currently assessing multicollinearity in my datasets. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What threshold values of VIF and condition index below/above suggest a problem?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;VIF:&lt;/strong&gt;&#xA;I have heard that VIF $\geq 10$ is a problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After removing two problem variables, VIF is $\leq 3.96$ for each variable.&#xA;Do the variables need more treatment or does this VIF seem fine? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Condition Index:&lt;/strong&gt;&#xA;I have heard that a Condition Index (CI) of 30 or more is a problem. &#xA;My highest CI is 16.66. Is this a problem? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Other Issues:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Are there any other dos/donts that need to be considered? &lt;/li&gt;&#xA;&lt;li&gt;Are there any other things that I need to keep in mind?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="1763" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-11-01T12:31:29.040" LastActivityDate="2011-03-14T10:57:21.963" Title="VIF, Condition Index and eigen values" Tags="&lt;multicollinearity&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="1" />
  <row Id="4100" PostTypeId="2" ParentId="2537" CreationDate="2010-11-01T10:49:03.003" Score="2" ViewCount="" Body="&lt;p&gt;Following up on chl's IRT suggestion and taking a different view of the analysis (and as an answer to the original question 2).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would see if there was dominance structure in the items, e.g. an item ordering where people that like 2 tend to like 1 but not 3 and people who like 3 tend to like 1 and 2, etc.  If so, there's a scale of some kind underneath your items and you might do better to build a measurement model of the underlying score and use those measurements as your dependent variable.  Parametric IRT models, e.g. in the R package ltm, would give you an expected score and per person standard error, if you wanted to use that as a weight in the final regression.  The mokken package can be used to see if there might be a scale in there in the first place.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The regression model decisions are then a separate but now slightly easier issue, for which existing comments provide a good overview.  Personally I'd go for a mixed effects model using lmer, but that's just my preference for more rather than less model.&lt;/p&gt;&#xA;" OwnerUserId="1739" LastActivityDate="2010-11-01T10:49:03.003" CommentCount="2" />
  <row Id="4101" PostTypeId="1" AcceptedAnswerId="4103" CreationDate="2010-11-01T11:37:13.793" Score="7" ViewCount="279" Body="&lt;p&gt;I would like to do an intervention analysis to quantify the results of a policy decision on the sales of alcohol over time.  I am fairly new to time series analysis, however, so I have some beginners questions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An examination of the literature reveals that other researchers have used ARIMA to model the time-series sales of alcohol, with a dummy variables as regressor to model the effect of the intervention.  While this seems like a reasonable approach, my data set is slightly richer than those I have encoutnered in the literature.  Firstly, my data set is disaggregated by beverage type (i.e. beer, wine, spirits), and then further disaggregated by geographical zone.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While I could create separate ARIMA analyses for each disagregated group and then compare the results, I suspect there is a better approach here.  Could anyone more familiar with multi-dimensional time-series data provide some poitners or suggestions?&lt;/p&gt;&#xA;" OwnerUserId="179" LastActivityDate="2011-04-01T01:07:15.667" Title="Intervention analysis with multi-dimensional time-series" Tags="&lt;time-series&gt;&lt;multivariate-analysis&gt;&lt;arima&gt;&lt;intervention-analysis&gt;" AnswerCount="3" FavoriteCount="2" />
  <row Id="4102" PostTypeId="2" ParentId="1432" CreationDate="2010-11-01T12:19:08.010" Score="0" ViewCount="" Body="&lt;p&gt;As I read in the internet, the working residuals is the &lt;strong&gt;residuals in the final iteration of any iteratively weighted least squares method&lt;/strong&gt;..Some body like chl or onestop can explain it better but I reckon it should be the residuals when we think its the last iteration of our running of model. That can give rise to discussion that model running is an iterative exercise.&lt;/p&gt;&#xA;" OwnerUserId="1763" LastActivityDate="2010-11-01T12:19:08.010" />
  <row Id="4103" PostTypeId="2" ParentId="4101" CreationDate="2010-11-01T12:21:36.523" Score="4" ViewCount="" Body="&lt;p&gt;The ARIMA model with a dummy variable for an intervention is a special case of a linear model with ARIMA errors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can do the same here but with a richer linear model including factors for the beverage type and geographical zones.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In R, the model can be estimated using arima() with the regression variables included via the xreg argument. Unfortunately, you will have to code the factors using dummy variables, but otherwise it is relatively straightforward.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-11-01T12:21:36.523" />
  <row Id="4104" PostTypeId="1" AcceptedAnswerId="4109" CreationDate="2010-11-01T12:24:49.967" Score="3" ViewCount="181" Body="&lt;p&gt;I would like to use data mining to try to find a good workout schemes. The input dataset will contain the parameters of a set of workouts with dates and different performance and medical measures. The problem is that the influence of each individual workout will be different depending on the time that has passed after this workout. For instance on the next day after the workout the performance will degrade, but eventually will improve to the higher level than before the workout. So each performance measure will be the cumulative result of different workouts.&#xA;I know that I can run a series of workouts of the same type during a period of time and then use difference in the performance as a predictor. But I was wondering if there are any algorithm that allow to take into account the time component and run analysis that will take into account individual workouts.&lt;/p&gt;&#xA;" OwnerUserId="255" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-11-01T18:29:13.710" LastActivityDate="2010-11-01T18:38:55.810" Title="Data mining algorithm suggestion" Tags="&lt;algorithms&gt;&lt;data-mining&gt;&lt;eda&gt;" AnswerCount="1" CommentCount="6" FavoriteCount="1" />
  <row Id="4105" PostTypeId="2" ParentId="1432" CreationDate="2010-11-01T14:03:54.977" Score="1" ViewCount="" Body="&lt;p&gt;I find the &lt;code&gt;binnedplot&lt;/code&gt; function in the R package &lt;a href=&quot;http://cran.r-project.org/web/packages/arm/index.html&quot; rel=&quot;nofollow&quot; title=&quot;the arm package on CRAN&quot;&gt;arm&lt;/a&gt; gives a very helpful plot of residuals.  It's described nicely on p.97-101 of &lt;a href=&quot;http://www.stat.columbia.edu/~gelman/arm/&quot; rel=&quot;nofollow&quot; title=&quot;Gelman and Hill (2007) Data Analysis Using Regression and Multilevel/Hierarchical Models. Cambridge University Press&quot;&gt;Gelman and Hill 2007&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1739" LastActivityDate="2010-11-01T14:03:54.977" />
  <row Id="4106" PostTypeId="2" ParentId="4101" CreationDate="2010-11-01T16:40:54.657" Score="4" ViewCount="" Body="&lt;p&gt;If you wanted to model the sales of drinks types as a vector [sales of wine at t, sales of beer at t, sales of spirits at t], you might want to look at Vector Autoregression (VAR) models.  You probably want the VARX variety that have a vector of exogenous variables like region and the policy intervention dummy, alongside the wine, beer and spirits sequences.  They are fairly straightforward to fit and you'd get impulse response functions to express the impact of exogenous shocks, which might also be of interest.  There's comprehensive discussion in Lütkepohl's book on multivariate time series.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, I'm certainly no economist but it seems to me that you might also think about ratios of these drinks types as well as levels.  People probably operate under a booze budget constraint - I know I do - which would couple the levels and (anti-)correlate the errors.&lt;/p&gt;&#xA;" OwnerUserId="1739" LastActivityDate="2010-11-01T16:40:54.657" />
  <row Id="4108" PostTypeId="2" ParentId="2077" CreationDate="2010-11-01T17:39:33.897" Score="0" ViewCount="" Body="&lt;p&gt;Could you fit a loess/spline through the data and use the residuals? Would the residuals be stationary?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Seems fraught with issues to consider, and perhaps there would not be as clear an indication of an overly-flexible curve as there is for over-differencing.&lt;/p&gt;&#xA;" OwnerUserId="1764" LastActivityDate="2010-11-01T17:39:33.897" />
  <row Id="4109" PostTypeId="2" ParentId="4104" CreationDate="2010-11-01T18:28:40.480" Score="4" ViewCount="" Body="&lt;p&gt;Your first task is to find a reasonable model relating an outcome $Y$ to the sequence of workouts that preceded it.  One might start by supposing that the outcome depends quite generally on a linear combination of time-weighted workout efforts $X$, but such a model would be unidentifiable (from having more parameters than data points).  One popular simplification is to suppose that the &quot;influence&quot; of a workout at time $t$ on the outcome at time $s$ is &lt;/p&gt;&#xA;&#xA;&lt;p&gt;a. proportional to the intensity of the workout,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;b. decays exponentially; that is, is reduced by a factor $\exp(\theta(t-s)))$ for some unknown decay rate $\theta$, and&lt;/p&gt;&#xA;&#xA;&lt;p&gt;c. independently adds to the influences of all other workouts preceding time $t$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course we must be prepared to allow some deviation between the actual outcome and that predicted by the model; it is natural to model that deviation as a set of independent random variables of zero mean.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This leads to a formal model which can serve as a useful point of departure for EDA.  To write it down, let the times be $t_1 \lt t_2 \lt \ldots \lt t_n$ with corresponding workout intensities $x_1, x_2, \ldots, x_n$ and let the outcomes be measured at times $s_1 \lt s_2 \lt \ldots \lt s_m$ with values $y_1, \ldots, y_m$, respectively.  The model is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$y_j =\alpha + \beta \exp(-\theta(s_j - t_{k})) \left( x_{k} + \exp(-\theta \Delta_{k,k-1}) x_{k-1} + \cdots + \exp(-\theta \Delta_{k, 1})x_1 \right) + \epsilon_j$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\alpha$ and $\beta$ are coefficients in a linear relation, $k$ is the index of the most recent workout preceding time $s_j$, $\Delta_{i,j} = t_i - t_j$ is the time elapsed between the $i^\text{th}$ and $j^\text{th}$ workouts, and the $\epsilon_j$ are independent random variables with zero expectations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This can get messy when workouts and endpoint measurements are unevenly spaced.  If to a good approximation the spacing between a workout and the next measurement is constant (say, a time difference of $s$) and--as an expository simplification--if each workout is followed by a measurement (so that $m = n$), then this model suggests some useful EDA procedures.  As an abbreviation, let's write (somewhat loosely)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$f_k(x,t,\theta) =  \left( x_{k} + \exp(-\theta \Delta_{k,k-1}) x_{k-1} + \cdots + \exp(-\theta \Delta_{k, 1})x_1 \right)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;for the weighted sum of the workouts up to and including the $k^\text{th}$ one, whence&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$y_k = \alpha + \gamma f_k(x,t,\theta) + \epsilon_k$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\gamma = \beta \exp(-\theta s)$.  Note that this formulation accommodates any irregular time sequence of workouts, so it's not grossly oversimplified.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you want to know is whether this makes sense: do the data at all behave like this?  We're really asking about the possibility of a &lt;em&gt;linear&lt;/em&gt; relationship between the $x$'s and the $y$'s.  We need that to hold for at least one decay constant $\theta$ with a reasonable value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One way to check is to note there is a relatively simple relationship between successive terms $f_{k+1}$ and $f_k$; you let $f_k$ decay for additional time $t_{k+1} - t_k$ and add $x_{k+1}$ to it:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$f_{k+1}(x,t,\theta) = x_{k+1} + \exp(-\theta \Delta_{k+1,k}) f_k(x,t,\theta).$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(This formula, by the way, provides an efficient way to compute all the $f_k$ by starting at $f_1 = x_1$ and continuing recursively for $k = 2, 3, \ldots, n$--a simple spreadsheet formula.  It is a generalization of the weighted running averages used extensively in financial analysis.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Equivalently, we can isolate $x_k$ by subtracting the right hand term.  This suggests exploring the relationship between the &lt;em&gt;adjusted values&lt;/em&gt; $z_k = y_{k+1} - \exp(-\theta \Delta_{k+1,k})y_k$ and the workouts $x_k$, because&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$z_k = (1 - \exp(-\theta \Delta_{k+1,k}))\alpha + \gamma x_k + (\epsilon_{k+1} - \exp(-\theta \Delta_{k+1,k}))\epsilon_k).$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;If the workouts are approximately regularly spaced,&lt;/em&gt; so that $\Delta_{k+1,k}$ is roughly constant, then for any fixed value of $\theta$ this expression is in the form &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$z = \text{ constant + constant *} x + \text{ error}.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The error terms will be positively correlated (in pairs) but still unbiased.  It is now clear how to check linearity: Pick a trial value for $\theta$, compute the $z$'s (which depend on it), make a scatterplot of the $z$'s versus the $x$'s, and look for linearity. Vary $\theta$ interactively to search for linear-looking scatterplots.  If you can produce one, you already have a reasonable estimate of $\theta$.  You can then estimate the other parameters ($\alpha$ and $\beta$) if you like.  If you cannot produce a linear scatterplot, use standard EDA techniques to re-express the variables (the $x$'s and $y$'s) until you can.  Look for a value of $\theta$ that minimizes the typical sizes of the residuals: that is a rough estimate of the decay rate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't expect this method to be highly stable: there is likely a wide range of values of $\theta$ that will induce linearity and relatively small residuals in the scatterplot.  But that is something for you to find out.  If you discover that only a narrow range of values accomplishes this, then you can have confidence that the decay effect is there and can be estimated.  Use maximum likelihood; it will be convenient to suppose the $\epsilon$'s are normally distributed.  (The profile likelihood, with $\theta$ fixed, is an ordinary least squares problem, so it will be easy to fit this model.  Alternatively you could try fitting the relationship between $z$ and $x$ directly using generalized least squares, but I think that would be trickier to implement.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This all might sound complicated but it's actually quite simple.  You could set up a spreadsheet in which $\theta$ is the value in a cell, add a $\theta$-varying control to a scatterplot of $x$ and $z$ (computed in the spreadsheet from a column of $t$ values and a column of $y$ values), and simply adjust the control to straighten out the plot.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It will be harder to explore a dataset in which there are fewer (or more) measurements $y_j$ than there are workouts $x_k$ or where the temporal spacing between workouts and measurements varies a lot.  You might have to settle for maximum likelihood solutions alone, without the benefit of supporting graphics to verify the reasonableness and the adequacy of this model &lt;em&gt;a priori&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even if my assumptions do not agree with your situation in all details, I hope that this discussion at least suggests effective approaches for furthering your investigation.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-01T18:38:55.810" LastActivityDate="2010-11-01T18:38:55.810" CommentCount="2" />
  <row Id="4110" PostTypeId="2" ParentId="25" CreationDate="2010-11-01T19:45:21.320" Score="2" ViewCount="" Body="&lt;p&gt;I'm new here, and perhaps &quot;financial time series&quot; has a specific definition... But given that I don't know it, my question for you would be what you mean: quarterly/monthly economic data, daily market prices, hourly or higher-frequency data, etc? And by &quot;modeling&quot;, do you mean working with textbook ARIMA/ARCH solutions, or things a bit more exotic (such as dynamic linear systems), or exotic/custom experimentation?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;R is flexible and free, though less GUI-fied than most. It also has packages covering everything from daily stock prices to dynamic linear systems and optimization packages. (In fact, the hard part will be deciding which time series and which financial packages to use.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;GRETL is free and has a reasonable GUI, though it's econometric, not really daily market oriented. I've heard of Oxmetrics, which appears to have a very complete every-possible-variant-of-ARCH package available for it. If you're talking monthly/quarterly economic data, you could also use X12-ARIMA, which is a benchmark of sorts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've used all kinds of GUIs for programming/processing data, but for some reason RapidMiner's never really clicked with me. Something strange about its workflow that I've just never gotten.&lt;/p&gt;&#xA;" OwnerUserId="1764" LastActivityDate="2010-11-01T19:45:21.320" CommunityOwnedDate="2010-11-01T19:45:21.320" />
  <row Id="4111" PostTypeId="1" AcceptedAnswerId="4116" CreationDate="2010-11-01T20:00:57.660" Score="11" ViewCount="320" Body="&lt;p&gt;A student asked me today, &quot;How do they know how many people attended a large group event, for example, the Stewart/Colbert 'Rally to Restore Sanity' in Washington D.C.?&quot;  News outlets report estimates in the tens of thousands, but what methods are used to get those estimates, and how reliable are they?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One article apparently based their estimate on parking permits...  but what other techniques do we have?  Please note I am not talking about capture/recapture experiments or anything of the like.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't have any idea.  I would guess in advance that there aren't specific methods for something like this, and whatever's there is very ad hoc (such as how many parking permits were sold). Is this true? For purposes of national security - of course - it would be possible to have an analyst sit down with satellite photographs and physically bean-count the number of people there. I doubt this method is used very often. &lt;/p&gt;&#xA;" OwnerUserId="1108" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-11-02T13:57:31.997" LastActivityDate="2011-04-19T05:09:08.220" Title="How to estimate how many people attended an event (say, a political rally)?" Tags="&lt;estimation&gt;&lt;sampling&gt;" AnswerCount="7" CommentCount="5" FavoriteCount="2" />
  <row Id="4112" PostTypeId="1" CreationDate="2010-11-01T20:16:11.330" Score="4" ViewCount="148" Body="&lt;p&gt;In &lt;a href=&quot;http://stats.stackexchange.com/questions/4052/how-does-the-power-of-a-logistic-regression-and-a-t-test-compare&quot;&gt;another question&lt;/a&gt; I asked how the power of a logistic regression and a t-test compare.  The accepted answer affirmed that their power should be roughly equivalent.  However, an assumption in that question was that the draws were entirely independent.  What if this were not the case?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Again there are two approaches being compared here:&#xA;   1. [The parametric t-test]: 30 draws from a binomial observation are made and the resulting values are averaged. This is done 30 times for group A (which has a binomial Pr of .70 of occurring + some random perturbation in Pr in each set of 30 observations) and 30 times for group B (which has a binomial Pr of .75 of occurring, also with Pr perturbation within a set of 30 observations). This yields 30 means per group that represent a summary of 1,800 draws from a binomial distribution. A 58df t-test is performed to compare the means.&#xA;   2. [The mixed effect logistic regression]: A mixed effect logistic regression is performed with a dummy coded slope representing group membership and each of the 1,800 draws as well as a random intercept for each of the groups.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question has two parts:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Given a set alpha of .05, will the power of these methodologies be the same or different? Why? How can I prove it?&lt;/li&gt;&#xA;&lt;li&gt;Is the answer to question 1 is sensitive to the sample sizes going into the t-test, sample size of each group in the t-test, underlying binomial probabilities, the degree of perturbation due to sub-group identity, or some other factor? If so, how can I know (without simulation) that the power is indeed different and what sort of changes will produce what sort of changes in power? Alternatively, provide worked out R code that solves the issue using simulation.  Specifically, I am interested in a way to quantify the extent to which power is potentially lose in the parametric t-test relative to the mixed effects logistic regression.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="196" LastActivityDate="2010-11-01T20:16:11.330" Title="How does the power of a mixed effects logistic regression and a t-test compare?" Tags="&lt;logistic-regression&gt;&lt;mixed-model&gt;&lt;t-test&gt;&lt;power&gt;&lt;power-analysis&gt;" />
  <row Id="4113" PostTypeId="2" ParentId="964" CreationDate="2010-11-01T20:22:48.497" Score="1" ViewCount="" Body="&lt;p&gt;I've found that the books I've read tend to mention the &quot;why&quot; behind diff and log. And it's easy to see for yourself. Try this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data (AirPassengers)&#xA;plot (AirPassengers)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Notice the seasonal pattern, but also notice the upward trend. So try&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plot (diff (AirPassengers))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;See how the upward trend is gone? By looking at the change each month instead of the actual data, you're seeing the patterns more clearly. You've stabilized the time series in some sense.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But also note that the pattern gets larger towards the right side of the graph. That's because the pattern is not additive (add amount Y), but rather multiplicative (add a percentage, or multiply). Logs turn multiplication into addition, so:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plot (diff (log (AirPassengers)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and you have a stable pattern that you can better perceive what's actually happening over time to the pattern, independently of the trend. A time series with this kind of stability is called &quot;stationary&quot;. (Of course, &quot;stationary&quot; has a very technical meaning beyond looking &quot;stable&quot;, but let's not go there for now.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Obviously, the first step to analysis is to understand and preprocess your data and a graph of the raw data is an essential step in the process. Graphing the diff(log(foo)) is just confirming that you understand the data and the results look appropriate, and it gives you a bit more insight into the seasonal patterns of the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are also tests (ADF, etc) and graphs (ACF) that would be used to confirm that differencing of a series is called for, but it can't hurt to look at things as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the transformation reveals the de-trended data (if the transform is indeed called for) for you to look at, and it is the data that you will actually feed into follow-on analysis. (Though some software you use will do the diff and the log under-the-hood for you and you'll only ever see the output which is reversed back to the original data scale)&lt;/p&gt;&#xA;" OwnerUserId="1764" LastActivityDate="2010-11-01T20:22:48.497" />
  <row Id="4114" PostTypeId="1" CreationDate="2010-11-01T20:31:17.030" Score="8" ViewCount="195" Body="&lt;p&gt;I've been reading some about Generalized Least Squares (GLS) and trying to tie it back to my basic econometric background. I recall in grad school using Seemingly Unrelated Regression (SUR) which seems somewhat similar to GLS. One paper I stumbled on even referred to SUR as &quot;special case&quot; of GLS. But I still can't wrap my brain around the similarities and differences. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;so the question:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are the similarities and differences between GLS and SUR? What are the hallmarks of a problem which should use one method over the other?&lt;/p&gt;&#xA;" OwnerUserId="29" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-03T09:55:08.037" LastActivityDate="2010-11-03T09:55:08.037" Title="Difference between GLS and SUR" Tags="&lt;regression&gt;&lt;gls&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="4115" PostTypeId="2" ParentId="4114" CreationDate="2010-11-01T20:39:05.573" Score="5" ViewCount="" Body="&lt;p&gt;In a narrow sense, GLS (and in particular Feasible GLS or FGLS) is an estimation method applied to SUR models. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;SUR implies a &lt;em&gt;system&lt;/em&gt; of m equations that are assumed to have correlated errors, and (F)GLS helps to recover from this -- see &lt;a href=&quot;http://en.wikipedia.org/wiki/Seemingly_unrelated_regressions&quot; rel=&quot;nofollow&quot;&gt;Wikipedia on Seemingly Unrelated Regressions&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;GLS, on the other hand, is a method of incorporating information from the covariance structure of your model.  See &lt;a href=&quot;http://en.wikipedia.org/wiki/Feasible_generalized_least_squares&quot; rel=&quot;nofollow&quot;&gt;Wikipedia on GLS&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To recap, you can use the latter (GLS) to estimate the former (SUR).&lt;/p&gt;&#xA;" OwnerUserId="334" LastActivityDate="2010-11-02T01:18:26.927" CommentCount="2" />
  <row Id="4116" PostTypeId="2" ParentId="4111" CreationDate="2010-11-01T20:52:32.800" Score="7" ViewCount="" Body="&lt;p&gt;You could estimate the people per square meter and multiply this by the size of the area.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an article on this topic: &lt;a href=&quot;http://www.lifeslittlemysteries.com/how-is-crowd-size-estimated--1074/&quot; rel=&quot;nofollow&quot;&gt;http://www.lifeslittlemysteries.com/how-is-crowd-size-estimated--1074/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1765" LastEditorUserId="1765" LastEditorDisplayName="" LastEditDate="2010-11-01T20:57:55.307" LastActivityDate="2010-11-01T20:57:55.307" CommentCount="1" />
  <row Id="4117" PostTypeId="2" ParentId="4111" CreationDate="2010-11-01T21:05:58.687" Score="1" ViewCount="" Body="&lt;p&gt;A police officer told me once that they had rules of thumb to guesstimate attendance at demonstrations (don't ask me for specifics), probably based on what Tim said. &lt;/p&gt;&#xA;" OwnerUserId="1766" LastActivityDate="2010-11-01T21:05:58.687" CommentCount="2" />
  <row Id="4118" PostTypeId="2" ParentId="4111" CreationDate="2010-11-01T22:21:56.600" Score="3" ViewCount="" Body="&lt;p&gt;Tim's linked article is great, though I think the company that counts people in grids is making it out to be easier than it really is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the local (DC) papers, I've seen quotes about Metro rider usage (except there were two other major events downtown the same day), attempts to count people at security checkpoints, grid square counting from aerial photos, quoting the numbers put on Park Service event applications, etc, none of which impress me in a large town with many things going on concurrently.&lt;/p&gt;&#xA;" OwnerUserId="1764" LastActivityDate="2010-11-01T22:21:56.600" CommentCount="1" />
  <row Id="4119" PostTypeId="2" ParentId="4111" CreationDate="2010-11-01T22:22:07.313" Score="0" ViewCount="" Body="&lt;p&gt;Here's an idea (but I am not sure this could work in practice): place a free wifi access point, and count the number of connections ( of iPhones, blackbery...). &lt;/p&gt;&#xA;" LastActivityDate="2010-11-01T22:22:07.313" CommentCount="1" />
  <row Id="4120" PostTypeId="1" AcceptedAnswerId="4130" CreationDate="2010-11-01T22:35:51.763" Score="4" ViewCount="213" Body="&lt;p&gt;I'm working with a dataset for which I only have means, standard deviations, and sample sizes for different levels of a continuous predictor.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;E.G.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;Y  X SD_Y N_Y&#xA;5  1   3   4&#xA;10 2   6   2&#xA;15 3   2   8&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I would like to determine the regression line that fits this data.  I'm wracking my brains to remember how data points should be weighted in a linear regression (I'm also interested in using a generalized linear model as well)- by sample size, variance, SD?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any pointers?&lt;/p&gt;&#xA;" OwnerUserId="101" LastActivityDate="2010-11-02T17:24:09.077" Title="Using weighted regression to obtain fit lines for which I only have summary data" Tags="&lt;regression&gt;" AnswerCount="4" CommentCount="2" FavoriteCount="1" />
  <row Id="4121" PostTypeId="1" AcceptedAnswerId="4122" CreationDate="2010-11-01T23:13:00.463" Score="0" ViewCount="280" Body="&lt;p&gt;If X and Y are standardized variables and are perfectly positively correlated with respect to each other, how can i prove that $E[(X-Y)^2] = 0$?&lt;/p&gt;&#xA;" OwnerUserId="1395" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-11-02T13:51:18.837" LastActivityDate="2010-11-02T13:51:18.837" Title="Statistics Proof that $E[(X-Y)^2] = 0$" Tags="&lt;theory&gt;&lt;homework&gt;" AnswerCount="1" />
  <row Id="4122" PostTypeId="2" ParentId="4121" CreationDate="2010-11-01T23:27:27.143" Score="10" ViewCount="" Body="&lt;p&gt;$E[(X-Y)^2) = E(X^2) + E(Y^2) - 2E(XY)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Use the fact that $X, Y$ are standardized and perfectly correlated to make appropriate substitutions above to get the desired result. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: I am not providing the complete solution. Hopefully, the above hints will set you on the right path.&lt;/p&gt;&#xA;" LastActivityDate="2010-11-01T23:27:27.143" CommentCount="5" />
  <row Id="4123" PostTypeId="2" ParentId="4120" CreationDate="2010-11-02T00:42:39.077" Score="0" ViewCount="" Body="&lt;p&gt;I think I would calculate normalized variables (z=(x-mean(x))/(sd(x)), and run the regression. Or you could work out a way to generate samples in a bootstrap. I'm not shure if this would be the textbook solution, but intuitively it should work. &lt;/p&gt;&#xA;" OwnerUserId="1766" LastActivityDate="2010-11-02T00:42:39.077" />
  <row Id="4124" PostTypeId="2" ParentId="4120" CreationDate="2010-11-02T00:50:27.137" Score="3" ViewCount="" Body="&lt;p&gt;Let the disaggregrate model be:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Y_{ia} = X_a \beta + \epsilon_i$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\epsilon_i \sim N(0,\sigma^2)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your aggregate model is given by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Y_a = \frac{\sum_i(Y_{ia})}{n_a}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$n_a$ is the number of observations you have corresponding to the $a$ index.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, it follows that:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Y_a = X_a \beta + \epsilon_a$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\epsilon_a \sim N(0, \frac{\sigma^2}{n_a} )$ and&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$a=1, 2, ... A$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, the OLS estimate would be given by minimizing:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\sum_a(Y_a - X_a \beta)^2$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which yields the usual solution. So, I do not think there is any difference as far as the estimate for the slope parameters are concerned.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit 1&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a small simulation in R which illustrates the above idea (apologies for the flaky code as I am using questions such as the above to learn R).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(1);&#xA;n &amp;lt;- c(4,2,8);&#xA;x &amp;lt;- c(1,2,3);&#xA;data &amp;lt;- matrix(0,14,2)&#xA;mean_data &amp;lt;- matrix(0,3,2)&#xA;index &amp;lt;- 1;&#xA;for (i in 1 : 3)&#xA;{&#xA;    for(obs in 1:n[i])&#xA;    {&#xA;        data[index,1] &amp;lt;- x[i];&#xA;        data[index,2] &amp;lt;- x[i]*8 + 1.5*rnorm(1);&#xA;&#xA;        mean_data[i,1] = x[i];&#xA;        mean_data[i,2] = mean_data[i,2] + data[index,2];&#xA;&#xA;        index = index + 1;  &#xA;    }&#xA;    mean_data[i,2] = mean_data[i,2] / n[i];&#xA;}&#xA;beta &amp;lt;- lm(mean_data[,2] ~ mean_data[,1]);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The above code yields the output when you type &lt;code&gt;beta&lt;/code&gt;: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Call:&#xA;lm(formula = mean_data[, 2] ~ mean_data[, 1])&#xA;&#xA;Coefficients:&#xA;   (Intercept)  mean_data[, 1]  &#xA;      -0.03455         7.99326  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit 2&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, OLS is not efficient as error variances are not equal. Thus, using MLE ideas, we need to minimize:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\sum_a{n_a (Y_a - X_a \beta)^2}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words, we want to minimize:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\sum_a{(\sqrt{n_a} Y_a - \sqrt{n_a} X_a \beta)^2}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, the MLE can be written as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let $W$ be a diagonal matrix with the $\sqrt{n_a}$ along the diagonal. Thus, the MLE estimate can be written as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$(X' X)^{-1} X' Y$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Y = W [Y_1,Y_2,...Y_A]'$ and&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$X = W [X_1,X_2,...X_A]'$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another way to think about this is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider the variance of $Y$. The transformation given above for $Y$ ensures that the variance of the individual values of $Y$ are identical thus satisfying the conditions of the &lt;a href=&quot;http://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem&quot; rel=&quot;nofollow&quot;&gt;Gauss-Markov theorem&lt;/a&gt; that OLS is BLUE.&lt;/p&gt;&#xA;" LastActivityDate="2010-11-02T01:56:12.977" />
  <row Id="4125" PostTypeId="1" CreationDate="2010-11-02T01:22:05.690" Score="2" ViewCount="271" Body="&lt;p&gt;How do you show that the point of averages (x,y) lies on the estimated regression line?&lt;/p&gt;&#xA;" OwnerUserId="1395" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-11-02T13:50:27.137" LastActivityDate="2010-11-05T13:59:57.563" Title="Regression Proof that the point of averages (x,y) lies on the estimated regression line" Tags="&lt;distributions&gt;&lt;regression&gt;&lt;proof&gt;&lt;homework&gt;" AnswerCount="1" CommentCount="7" />
  <row Id="4126" PostTypeId="2" ParentId="964" CreationDate="2010-11-02T01:47:17.577" Score="0" ViewCount="" Body="&lt;p&gt;Time serie analysis is easier on stationary data. (more tools)&#xA;When a time serie is non stationary, you can try to find another time series, linked to the initial one, and which is stationary.&#xA;In many cases taking the difference will be enough. (Time series integrated with order 1) Sometimes you'll have to take the logarithm before (for instance for financial time series) The reason is that investment returns are in percentage!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just don't forget before doing that, there are also some tools for non stationary data. Sometimes taking the diff or the log return may lead to spurious regressions!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Fred&lt;/p&gt;&#xA;" OwnerUserId="1709" LastActivityDate="2010-11-02T01:47:17.577" />
  <row Id="4127" PostTypeId="1" CreationDate="2010-11-02T02:58:49.797" Score="0" ViewCount="155" Body="&lt;p&gt;Hi&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Has anybody already participated or plan to participate to any of these competitions?&#xA;Any feedbacks or opinions are welcome.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.kdd.org/kdd2010/kddcup.shtml&quot; rel=&quot;nofollow&quot;&gt;http://www.kdd.org/kdd2010/kddcup.shtml&lt;/a&gt;  &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.kdd.org/kdd2011/kddcup.shtml&quot; rel=&quot;nofollow&quot;&gt;http://www.kdd.org/kdd2011/kddcup.shtml&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.kdnuggets.com/datasets/competitions.html&quot; rel=&quot;nofollow&quot;&gt;http://www.kdnuggets.com/datasets/competitions.html&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Fred&lt;/p&gt;&#xA;" OwnerUserId="1709" LastActivityDate="2010-11-02T11:38:21.107" ClosedDate="2010-11-02T13:49:58.510" Title="Data mining competition" Tags="&lt;r&gt;&lt;machine-learning&gt;" CommentCount="4" />
  <row Id="4130" PostTypeId="2" ParentId="4120" CreationDate="2010-11-02T06:01:22.763" Score="3" ViewCount="" Body="&lt;p&gt;Weight each mean by the number of points that went into computing it. You can later use the estimated standard deviations to test the hypothesis of homoscedasticity that this approach assumes. If the Ns are as small as in your example then this test probably wouldn't have much power unless the SDs vary greatly. &lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-11-02T06:01:22.763" CommentCount="1" />
  <row Id="4131" PostTypeId="1" CreationDate="2010-11-02T06:26:26.320" Score="5" ViewCount="207" Body="&lt;p&gt;&lt;strong&gt;CONTEXT:&lt;/strong&gt;&#xA;I am modelling the relation between time (1 to 30) and a DV for a set of 60 participants. Each participant has their own time series.&#xA;For each participant I am examining the fit of 5 different theoretically plausible functions  within a nonlinear regression framework.&#xA;One function has one parameter; three functions have three parameters; and one function has five parameters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to use a decision rule to determine which function provides the most &quot;theoretically meaningful&quot; fit.&#xA;However, I don't want to reward over-fitting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Over-fitting seems to come in two varieties. One form is the standard sense whereby an additional parameter enables slightly more of the random variance to be explained. A second sense is where there is an outlier or some other slight systematic effect, which is of minimal theoretical interest. Functions with more parameters sometimes seem capable of capturing these anomalies and get rewarded.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I initially used AIC. And I have also experimented with increasing the penalty for parameters.&#xA;In addition to using $2k$: [$\mathit{AIC}=2k + n[\ln(2\pi \mathit{RSS}/n) + 1]$];&#xA;I've also tried $6k$ (what I call AICPenalised).&#xA;I have inspected scatter plots with fit lines imposed and corresponding recommendations based on AIC and AICPenalised. Both AIC and AICPenalised  provide reasonable recommendations. About 80% of the time they agree. However, where they disagree, AICPenalised seems to make  recommendations that are more theoretically meaningful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;QUESTION:&lt;/strong&gt;&#xA;Given a set of nonlinear regression function fits:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What is a good criterion for deciding on a best fitting function in nonlinear regression?&lt;/li&gt;&#xA;&lt;li&gt;What is a principled way of adjusting the penalty for number of parameters?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-11-02T08:55:32.367" LastActivityDate="2010-11-02T09:41:07.780" Title="Comparing model fits across a set of nonlinear regression models" Tags="&lt;aic&gt;&lt;nonlinear-regression&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="2" />
  <row Id="4133" PostTypeId="2" ParentId="4131" CreationDate="2010-11-02T09:41:07.780" Score="1" ViewCount="" Body="&lt;p&gt;For each participant, compute the cross-validated (leave one out) prediction error per functional form and assign the participant the form with the smallest one.  That should do something to keep the overfitting under control.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That approach ignores higher level problem structure: the population has groups that are assumed to share a functional form, so data from one participant with the a particular form is potentially useful for estimating the parameters of another with the same form.  But it's a start, if not a finish, for the analysis.&lt;/p&gt;&#xA;" OwnerUserId="1739" LastActivityDate="2010-11-02T09:41:07.780" />
  <row Id="4134" PostTypeId="2" ParentId="25" CreationDate="2010-11-02T11:17:57.750" Score="1" ViewCount="" Body="&lt;p&gt;While not exactly cheap, MATLAB is widely used in the financial industry for time series modelling: &lt;a href=&quot;http://www.mathworks.com&quot; rel=&quot;nofollow&quot;&gt;http://www.mathworks.com&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="439" LastEditorUserId="439" LastEditorDisplayName="" LastEditDate="2010-11-02T11:48:26.543" LastActivityDate="2010-11-02T11:48:26.543" CommunityOwnedDate="2010-11-02T11:17:57.750" CommentCount="1" />
  <row Id="4135" PostTypeId="2" ParentId="25" CreationDate="2010-11-02T12:04:30.563" Score="1" ViewCount="" Body="&lt;p&gt;I really like to work with R, because in the end you will find almost anything, and you have a very good support with the mailing lists. The downside of R is that helpful bits which fit your specific problems might be spread over a large range of packages, and you might not always be able to find them. Another point may be a lock-in, with that I mean that after a time learning R, you will probably be unmotivated to relearn another software, but this will happen in any system.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With regard to Matlab being expensive - if on a budget, Octave will work just as well, at least it did for the things I needed to do with it, which were rather basic.&lt;/p&gt;&#xA;" OwnerUserId="1766" LastEditorUserId="1766" LastEditorDisplayName="" LastEditDate="2010-11-02T12:15:04.513" LastActivityDate="2010-11-02T12:15:04.513" CommunityOwnedDate="2010-11-02T12:04:30.563" />
  <row Id="4136" PostTypeId="2" ParentId="4111" CreationDate="2010-11-02T12:19:33.383" Score="0" ViewCount="" Body="&lt;p&gt;As an alternative to WiFi mentioned by &lt;a href=&quot;http://stats.stackexchange.com/users/1767/uri&quot;&gt;Uri&lt;/a&gt;, you could place Bluetooth scanner(s) in 'strategic' locations of your venue. I've attended a presentation during &lt;a href=&quot;http://www.geo.uzh.ch/~plaube/mpa10/index.html&quot; rel=&quot;nofollow&quot;&gt;MPA workshop&lt;/a&gt; about &lt;a href=&quot;http://sunsite.informatik.rwth-aachen.de/Publications/CEUR-WS/Vol-652/MPA10-05.pdf&quot; rel=&quot;nofollow&quot;&gt;such development&lt;/a&gt; in Netherlands.&lt;/p&gt;&#xA;" OwnerUserId="22" LastActivityDate="2010-11-02T12:19:33.383" />
  <row Id="4137" PostTypeId="2" ParentId="2715" CreationDate="2010-11-02T13:02:40.370" Score="1" ViewCount="" Body="&lt;p&gt;In a &lt;strong&gt;forecasting problem&lt;/strong&gt; (i.e. when you need to forecast $Y_{t+h}$ given $(Y_t,X_t)$ $t&gt;T$, with the use of a learning set $(Y_1,X_1),\dots, (Y_T,X_T)$ ), the rule of the thumb (to be done before any complex modelling) are&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;strong&gt;Climatology&lt;/strong&gt; ($Y_{t+h}$ forecast by the mean observed value over the learning set, possibly by removing obvious periodic patterns) &lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Persistance&lt;/strong&gt; ($Y_{t+h}$ forecast by the last observed value: $Y_t$. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;What I often do now as a last simple benchmark/rule of the thumb is using randomForest($Y_{t+h}$~$Y_t+X_t$,data=leanringSet) in R software. It gives you (with 2 lines of code in R) a first idea of what can be achieved without any modelling.&lt;/p&gt;&#xA;" OwnerUserId="223" LastActivityDate="2010-11-02T13:02:40.370" CommunityOwnedDate="2010-11-02T13:02:40.370" />
  <row Id="4138" PostTypeId="1" CreationDate="2010-11-02T13:57:43.190" Score="4" ViewCount="260" Body="&lt;p&gt;Given a $n$-dimensional multivariate normal distribution $X=(x_i) \sim \mathcal{N}(\mu, \Sigma)$ with mean $\mu$ and covariance matrix $\Sigma$, what is the probability that $\forall j\in {1,\ldots,n}:x_1 \geq x_j$?&lt;/p&gt;&#xA;" OwnerUserId="767" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-10T07:40:41.250" LastActivityDate="2010-11-10T07:40:41.250" Title="What is the probability that random variable $x_1$ is maximum of random vector $X=(x_i)$ from a multivariate normal distribution?" Tags="&lt;probability&gt;&lt;multivariate-analysis&gt;&lt;normal-distribution&gt;" AnswerCount="3" CommentCount="7" />
  <row Id="4139" PostTypeId="2" ParentId="4138" CreationDate="2010-11-02T15:22:26.983" Score="2" ViewCount="" Body="&lt;p&gt;&lt;em&gt;Answer updated thanks to remarks from Whuber and Srikant&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;Proposition&lt;/strong&gt;&#xA;  Let C=[C_1;C_2] be a 2*n matrix, $X^0=(X^0_i)\sim \mathcal{N} (0,\Sigma)$ $\mathbb{R}^n$ valued. Let $\Sigma^Y=^tC\Sigma C=(\sigma^Y_{ij})$. Then, for $u_1,u_2\in\mathbb{R}$&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;$P(^tC_1X^0\geq u_1\text{ and } ^tC_2X^0\geq u_2)=\mathbb{E}\left [ \bar{\Phi}\left (\frac{u_2-\frac{\sigma^Y_{21}}{\sigma^Y_{11}}^tC_1X^0}{\sqrt{\sigma^Y_{22}-\frac{\sigma^Y_{21}\sigma^Y_{12}}{\sigma^Y_{11}}}} \right )1_{ ^tC_1X^0\geq u_1 } \right ]$ &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;where $\bar{\Phi}=P(\mathcal{N}(0,1)&gt;z)$&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Answer to the question when the dimension is 3&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assume $i=1$, $\Sigma=(\sigma_{ij})$. &#xA;The probability $P(X_1&gt;X_2 \text{ and }X_1&gt;X_3)$ is obtained using the preceding proposition with $X^0=X-\mu$, $C_1=(1,-1,0)$, $C_2=(1,0,-1)$, $u_1=\mu_2-\mu1$ and $u_2=\mu_3-\mu1$. This gives&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\sigma^Y_{11}=\sigma_{11}+\sigma_{22}-2\sigma_{12}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\sigma^Y_{22}=\sigma_{11}+\sigma_{33}-2\sigma_{13}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\sigma^Y_{12}=\sigma_{11}+2\sigma_{23}-\sigma_{31}-\sigma_{21}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Proof of the proposition&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assume $c\in\mathbb{R}^n$ and $\Sigma$ has full rank. It is easy to show that for any $u\in\mathbb{R}$&#xA;$$P(^tcX^0&gt;u)=\bar{\Phi} \left (\frac{u}{\|\Sigma^{1/2}c\|_2} \right )$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let us denote  $Y_1=^tC_1X^0,Y_2=^tC_2X^0$. From the correlation theorem, since $Y=(Y_1,Y_2)$ is centered gaussian in $\mathbb{R}^2$  with covariance $\Sigma^Y$&#xA;then $Y_2|Y_1$ is gaussian with mean $\frac{\sigma^Y_{21}}{\sigma^Y_{11}}Y_1$ and variance  $\sigma^Y_{22}-\frac{\sigma^Y_{21}}{\sigma^Y_{11}}\sigma^Y_{12}$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This, with&#xA;$P(Y_1&gt;u_1 \text{ and }  Y_2&gt;u_2)=\mathbb{E}\left [\mathbb{E}[1_{Y_2\geq u_2 }|Y_1]  1_{Y_1\geq u_1 }\right ] $&#xA;gives the desired result. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How to extend the proposition&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we want to be able to solve the initial problem with dimension larger than $3$, &#xA;we need to compute&#xA;$P(\forall j \; ^tc_jX^0\geq u_j) $&#xA;(for well chosen $u_j$). Set $Y=(Y_1,\dots,Y_n)$ with $Y_j=^tc_jX$ centered $\mathbb{R}$-valued gaussians.&lt;br&gt;&#xA;You can use the correlation theorem iteratively to derive the distribution of $Y_1|Y_{2:n}$, $Y_2|Y_{3:n}$..... This may give something like a  recurcive formulation of the solution to the proposition when C is $p*n$ (recurcive on $p$). &lt;/p&gt;&#xA;" OwnerUserId="223" LastEditorUserId="223" LastEditorDisplayName="" LastEditDate="2010-11-04T12:13:00.860" LastActivityDate="2010-11-04T12:13:00.860" CommunityOwnedDate="2010-11-03T19:01:28.107" CommentCount="6" />
  <row Id="4140" PostTypeId="1" CreationDate="2010-11-02T15:27:13.997" Score="3" ViewCount="131" Body="&lt;p&gt;Hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I feel I'm pretty new to this, since some time passed since my last statistics assignment, so please bear with me.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am analyzing results of a biological experiment.  Basically, I'm looking at a some graph over a genome, where each position in the genome has a value, and I'm looking for local minima (peaks).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, I have to set some threshold since relatively high local minima also occur by chance. I can simulate the experiment computationally and get new data, but this is quite resource demanding (I can't run 1000 simulations, perhaps 100 or even only 20).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I'm currently doing is run some simulations; for each simulation: find all local minima, build a CDF for the local minima values. I then average all the simulation CDFs over all simulations so I have one 'average' CDF (CDF_simulations) which is supposed to show how would local minima distribute if everything in my genome is random.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I do the same for the real data: find minima and build CDFs for their values, so I now have two CDFs - one for the real data and one for the average of the simulations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I now search for the max x such that CDF_simulations(x) / CDF_realdata(x) is &amp;lt; 10%. I report all minima in the real data with value &amp;lt; x as &quot;true&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think this method should get me to a FP rate of 10%.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Does this make sense?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;What is this method called and where can I find more about it?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;How should I scan the CDFs to find the right x? Sometimes for low x's CDF_simulations(x) &gt; CDF_realdata(x).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Where does the number of simulations come into play? Does it make sense to simply build an averaged CDF as I did?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I think this is quite common, and the name FDR also pops to mind, but when reading about FDR I couldn't exactly figure how to apply it to my situation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any comments and references (hopefully user-friendly ones) will be appreciated!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&#xA;Dave&lt;/p&gt;&#xA;" OwnerUserId="634" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-11-02T18:25:52.760" LastActivityDate="2010-11-02T18:25:52.760" Title="How can I control the false positives rate?" Tags="&lt;multiple-comparisons&gt;&lt;statistical-significance&gt;&lt;cdf&gt;" CommentCount="3" />
  <row Id="4141" PostTypeId="2" ParentId="4125" CreationDate="2010-11-02T15:33:09.130" Score="2" ViewCount="" Body="&lt;p&gt;To get you started: $\bar y = 1/n \sum y_i = 1/n \sum (\hat y_i + \hat \epsilon_i)$ then plug in, how the $\hat y_i$ are estimated by the $x_i$ and you're almost done.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: since no one replied, here the rest for sake of completeness:&#xA;the $\hat y_i$ are estimated by $\hat y_i=\hat \beta_0 + \hat \beta_1 x_{i1} + \ldots + \hat \beta_n x_{in} + \hat \epsilon_i$, so you  get $\bar y = 1/n \sum \hat \beta_0 + \hat \beta_1 x_{i1} + \ldots + \hat \beta_n x_{in}$ (the  $\hat \epsilon_i$ sum to zero) and finally:&lt;br&gt;&#xA;$\bar y = \hat \beta_0 + \hat \beta_1 \bar x_{1} + \ldots + \hat \beta_n \bar x_{n}$. And that's it: The regression line goes through the point $(\bar x, \bar y)$&lt;/p&gt;&#xA;" OwnerUserId="1573" LastEditorUserId="1573" LastEditorDisplayName="" LastEditDate="2010-11-05T13:59:57.563" LastActivityDate="2010-11-05T13:59:57.563" />
  <row Id="4142" PostTypeId="1" CreationDate="2010-11-02T16:49:01.617" Score="2" ViewCount="217" Body="&lt;p&gt;I have two statistical tests which are inverse of each other, meaning that the null hypothesis are reversed. I want to use both the tests to take a decision. For this purpose, I am planning to do the following:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;If both the tests point to same result (by, say, rejecting the null hypothesis in test A and not rejecting the null hypothesis in test B, or vice versa), then I go ahead and take that decision&lt;/li&gt;&#xA;&lt;li&gt;On the other hand, if the results of the two tests are conflicting, I measure the difference of the p-values from alpha (say, 0.05) for each test, and go with the one having the largest deviation (I have some tie-breaking rules, but let's leave that here)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;It sounds reasonable to me, but is there some statistical ground in interpreting p-values like this? I certainly haven't seen similar application before (in my limited exposure).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: Let me clarify the question some more with the actual context and tests. I am testing for unit roots in time series, in order to determine if the series needs to be differenced in order to render it mean-stationary. The particular tests are KPSS test, with null being no unit roots, and ADF test with null being an unit root exists. Although the null hypotheses are related (and inverse of each other), but the test regression and the statistics are quite different in my opinion.&lt;/p&gt;&#xA;" OwnerUserId="528" LastEditorUserId="528" LastEditorDisplayName="" LastEditDate="2010-11-02T18:18:32.637" LastActivityDate="2010-11-03T00:10:51.297" Title="Using the distance of p-value from alpha" Tags="&lt;time-series&gt;&lt;hypothesis-testing&gt;&lt;statistical-significance&gt;" AnswerCount="3" CommentCount="2" FavoriteCount="1" />
  <row Id="4143" PostTypeId="2" ParentId="4142" CreationDate="2010-11-02T17:12:03.430" Score="2" ViewCount="" Body="&lt;p&gt;Hello Samik,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a question based on what you asked and assuming I understood you correctly.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Why do you need two null hypotheses to decide about one decision? &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Are you doing this just because we null hypotheses can only be rejected and never accepted? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now the answer to your question about using p-value as a measure of &quot;truth of the null&quot; (similar to measuring the distance of p-value from $\alpha$) is not justified from a statistical perspective. This can lead to possible wrong decisions. &lt;a href=&quot;http://www.stat.duke.edu/~berger/papers/99-13.html&quot; rel=&quot;nofollow&quot;&gt;Here&lt;/a&gt; is a great reference explaining the dangers of using p-values in the sense you are trying to use.&lt;/p&gt;&#xA;" OwnerUserId="1307" LastActivityDate="2010-11-02T17:12:03.430" CommentCount="5" />
  <row Id="4144" PostTypeId="2" ParentId="4120" CreationDate="2010-11-02T17:15:39.277" Score="5" ViewCount="" Body="&lt;p&gt;This is Analysis of Variance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After all, consider one of the $y$'s, with standard deviation $s$, and let its predicted value (which depends on the corresponding $x$) be $f$.  The original aim is to vary $f$ (within constraints depending on the model; often $f$ is required to be a linear function of $x$) to minimize the sum of squared residuals.  &lt;em&gt;Suppose&lt;/em&gt; we had the original dataset available.  Let the values summarized by a particular $y$ be $y_1, y_2, \ldots, y_k$, so that $y$ is their mean and $s$ is their standard deviation.  Their contribution to the sum of squares of residuals equals&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\sum_{i=1}^k{\left( y_i - f \right)^2} = k \left(y - f \right)^2 + k' s^2 \text{.}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(I have written $k'$ because its value depends on how you compute your standard deviations: it is $k$ for one convention and $k-1$ for another.)  Because the last term does not depend on $f$, &lt;em&gt;it does not affect the minimization&lt;/em&gt;: we can neglect it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The other term on the right hand side shows that you want to perform a &lt;strong&gt;weighted least squares&lt;/strong&gt; calculation with weights equal to the counts $k$ (the &quot;N_Y&quot; column of the data).  Equivalently, you can create a &lt;strong&gt;synthetic dataset&lt;/strong&gt; by making $N_Y$ copies of each datum $(X,Y)$ and performing &lt;strong&gt;ordinary least squares regression&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that this analysis presumes nothing about the form of the prediction function: it can include any explanatory variables you like and have any form, even nonlinear ones.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note also that the weighting &lt;em&gt;does not depend on the standard deviations.&lt;/em&gt;  This is because implicitly we have assumed that the variance of the y's is constant, so that all of the differences among the observed standard deviations are attributed to random fluctuations.  This hypothesis can be tested in the usual ways (&lt;em&gt;e.g.&lt;/em&gt;, with F-tests).  For the example data it holds up: those standard deviations do not vary significantly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit&lt;/strong&gt;&#xA;I see, in retrospect, that this answer merely reiterates @onestop's pithy response.  I'm leaving it up because it demonstrates why @onestop is correct.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-11-02T17:24:09.077" LastActivityDate="2010-11-02T17:24:09.077" CommentCount="3" />
  <row Id="4145" PostTypeId="2" ParentId="2142" CreationDate="2010-11-02T17:21:52.077" Score="1" ViewCount="" Body="&lt;p&gt;The question is about marginal effects (of X on Y), I think, not so much about interpreting individual coefficients.  As folk have usefully noted, these are only sometimes identifiable with an effect size, e.g. when there are linear and additive relationships.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If that's the focus then the (conceptually, if not practically) simplest way to think about the problem would seem to be this: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To get the marginal effect of X on Y in a linear normal regression model with no interactions, you &lt;em&gt;can&lt;/em&gt; just look at the coefficient on X.  But that's not quite enough since it is estimated not known.  In any case, what one really wants for marginal effects is some kind of plot or summary that provides a prediction about Y for a range of values of X, and a measure of uncertainty.  Typically one might want the predicted mean Y and a confidence interval, but one might also want predictions for the complete conditional distribution of Y for an X.  That distribution is wider than the fitted model's sigma estimate because it takes into account uncertainty about the model coefficients.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are various closed form solutions for simple models like this one.  For current purposes we can ignore them and think instead more generally about how to get that marginal effects graph by simulation, in a way that deals with arbitrarily complex models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assume you want the effects of varying X on the mean of Y, and you're happy to fix all the other variables at some meaningful values.  For each new value of X, take a size B sample from the distribution of model coefficients.  An easy way to do so in R is to assume that it is Normal with mean &lt;code&gt;coef(model)&lt;/code&gt; and covariance matrix &lt;code&gt;vcov(model)&lt;/code&gt;.  Compute a new expected Y for each set of coefficients and summarize the lot with an interval.  Then move on to the next value of X.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems to me that this method should be unaffected by any fancy transformations applied to any of the variables, provided you also apply them (or their inverses) in each sampling step.  So, if the fitted model has log(X) as a predictor then log your new X before multiplying it by the sampled coefficient.  If the fitted model has sqrt(Y) as a dependent variable then square each predicted mean in the sample before summarizing them as an interval.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In short, more programming but less probability calculation, and clinically comprehensible marginal effects as a result.  This 'method' is sometimes referred to CLARIFY in the political science literature, but is quite general.&lt;/p&gt;&#xA;" OwnerUserId="1739" LastActivityDate="2010-11-02T17:21:52.077" />
  <row Id="4146" PostTypeId="1" AcceptedAnswerId="4149" CreationDate="2010-11-02T17:33:15.593" Score="6" ViewCount="162" Body="&lt;p&gt;When developing a general purpose time-series software, is it a good idea to make it scale invariant? How would one do that?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I took a time series of around 40 points, and then multiplied by factors ranging from 10E-9 to 10E3 and then ran through the ARIMA functions of Forecast Pro and Minitab. In Forecast Pro, all resulted in the same answer (automatic modeling), whereas in Minitab, they were not. Not sure what Forecast Pro does, but they might just scale up or down all the numbers to a certain scale (let's say 100s) before running the model. Is this good idea in general?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a link to the data I used for testing:&#xA;&lt;a href=&quot;https://spreadsheets.google.com/ccc?key=0AlEuN_O-nek1dGpiOWpOdEMtUUU5aDFKZk1oUS02YXc&amp;amp;hl=en&amp;amp;authkey=CNT-nKEB&quot; rel=&quot;nofollow&quot;&gt;Google spreadsheet&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="528" LastActivityDate="2011-03-27T13:32:27.927" Title="Scale-invariant analysis of time series" Tags="&lt;time-series&gt;&lt;scale-invariance&gt;" AnswerCount="1" />
  <row Id="4147" PostTypeId="2" ParentId="4052" CreationDate="2010-11-02T19:19:49.510" Score="4" ViewCount="" Body="&lt;p&gt;Here is code in R that illustrates the simulation of whuber's &lt;a href=&quot;http://stats.stackexchange.com/questions/4052/how-does-the-power-of-a-logistic-regression-and-a-t-test-compare/4056#4056&quot;&gt;answer&lt;/a&gt;. Feedback on improving my R code is more than welcome.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;N &amp;lt;- 900            # Total number data points&#xA;m &amp;lt;- 30;            # Size of draw per set&#xA;n &amp;lt;- 30;            # No of sets&#xA;&#xA;p_null &amp;lt;- 0.70;     # Null hypothesis&#xA;p_alternate &amp;lt;- 0.74 # Alternate hypothesis&#xA;tot_iter &amp;lt;- 10000;&#xA;&#xA;set.seed(1);        # Initialize random seed&#xA;null_rejected &amp;lt;- 0; # Set counter to 0&#xA;for (iter in 1:tot_iter)&#xA;{&#xA;    draws1 &amp;lt;- matrix(0,m,n);&#xA;    draws2 &amp;lt;- matrix(0,m,n);&#xA;    means1 &amp;lt;- matrix(0,m);&#xA;    means2 &amp;lt;- matrix(0,m);&#xA;&#xA;    for (obs in 1:m)&#xA;    {&#xA;        draws1[obs,] &amp;lt;- rbinom(n,1,p_null);&#xA;        draws2[obs,] &amp;lt;- rbinom(n,1,p_alternate);&#xA;&#xA;        means1[obs,] &amp;lt;- mean(draws1[obs,]);&#xA;        means2[obs,] &amp;lt;- mean(draws2[obs,]);&#xA;    }&#xA;    if (t.test(means1,means2,alternative=&quot;l&quot;)$p.value &amp;lt;= 0.05)&#xA;    {&#xA;        null_rejected &amp;lt;- null_rejected + 1; &#xA;    }&#xA;}&#xA;power &amp;lt;- null_rejected / tot_iter&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" LastActivityDate="2010-11-02T19:19:49.510" CommentCount="7" />
  <row Id="4148" PostTypeId="2" ParentId="4142" CreationDate="2010-11-02T19:45:15.580" Score="4" ViewCount="" Body="&lt;p&gt;I think you are confusing the concepts of stationarity vs unit roots. Unit root implies non-stationarity but the converse need not hold. Thus, KPSS and ADF tests are not both a test for unit roots. KPSS tests for stationarity whereas ADF tests for &lt;em&gt;one&lt;/em&gt; particular form of non-stationarity (i.e., existence of unit roots). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would suggest going through these &lt;a href=&quot;http://www.econ.ku.dk/metrics/Econometrics2_07_I/LectureNotes/unitroottest.pdf&quot; rel=&quot;nofollow&quot;&gt;notes&lt;/a&gt; and in particular see the figure 1  excerpted from the notes below for example of non-stationary series that are not because of unit roots:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/T9H98.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" LastActivityDate="2010-11-02T19:45:15.580" CommentCount="1" />
  <row Id="4149" PostTypeId="2" ParentId="4146" CreationDate="2010-11-02T22:39:26.827" Score="5" ViewCount="" Body="&lt;p&gt;If the software computes the sum of squared errors in optimization (and most will), then you can run into trouble with very large numbers or very small numbers because of how floating point numbers are stored. The same applies to any statistical modelling, not just time series analysis. One way to avoid the problem is to scale the data before running the model, and then re-scale the results. For most time series models, including all linear models, that will work. Some nonlinear models won't scale however.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I'm analysing data I will often scale the data myself, not just to prevent possible optimization problems but also to make graphs and tables easier to read.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-11-02T22:39:26.827" CommentCount="6" />
  <row Id="4150" PostTypeId="1" CreationDate="2010-11-02T23:59:15.377" Score="2" ViewCount="183" Body="&lt;p&gt;I have about 1000 data points from some thick tailed distribution that I would like to fit a parametrized distribution to.  From my data, I've made some adjustments and constructed an empirical distribution (so I have percentiles).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the best way to fit a mixture of parametrized distribution functions (pareto, lognormal, gamma, etc) to this empirical distribution?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far I have been using Excel to maximize the grouped MLE function; using solver to maximize the parameters subject to sum(weights = 1).  I have R as well but am new to it.  It is pretty obvious that excel is getting stuck in local maxima.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How would you maximize a MLE function for a mixture distribution?&lt;/p&gt;&#xA;" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-27T16:02:32.933" LastActivityDate="2011-03-27T20:36:29.907" Title="Optimization of MLE for mixture problems" Tags="&lt;maximum-likelihood&gt;&lt;curve-fitting&gt;&lt;em-algorithm&gt;&lt;mixture&gt;" AnswerCount="3" CommentCount="2" />
  <row Id="4151" PostTypeId="2" ParentId="4142" CreationDate="2010-11-03T00:10:51.297" Score="0" ViewCount="" Body="&lt;p&gt;I don't know anything about unit roots, but your problem may be analogous to an &lt;em&gt;equivalence test&lt;/em&gt; where, say, a new drug is tested for inferiority, equivalence or superiority compared to another drug. That is a reasonably common issue in clinical testing and so there is quite a literature on it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might like to start here: &lt;a href=&quot;http://www.graphpad.com/library/BiostatsSpecial/article_182.htm&quot; rel=&quot;nofollow&quot;&gt;http://www.graphpad.com/library/BiostatsSpecial/article_182.htm&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1679" LastActivityDate="2010-11-03T00:10:51.297" />
  <row Id="4152" PostTypeId="2" ParentId="4150" CreationDate="2010-11-03T00:20:02.080" Score="2" ViewCount="" Body="&lt;p&gt;Hi Robert,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to fit univariate distribution on your data, please try using &lt;code&gt;fitdistr&lt;/code&gt; function in the MASS package in &lt;code&gt;R&lt;/code&gt;. &lt;a href=&quot;http://rss.acs.unt.edu/Rdoc/library/MASS/html/fitdistr.html&quot; rel=&quot;nofollow&quot;&gt;Here&lt;/a&gt; is the information on how to use it. I am assuming that you have the full data set in addition to the quantiles.&lt;/p&gt;&#xA;" OwnerUserId="1307" LastActivityDate="2010-11-03T00:20:02.080" CommentCount="2" />
  <row Id="4153" PostTypeId="2" ParentId="3911" CreationDate="2010-11-03T00:38:16.730" Score="2" ViewCount="" Body="&lt;p&gt;I think the premise of this question is flawed because it denies the distinction between the &lt;em&gt;uncertain&lt;/em&gt; and the &lt;em&gt;known&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Describing a coin flip provides a good analogy.  Before the coin is flipped, the outcome is uncertain; afterwards, it is no longer &quot;hypothetical.&quot;  Confusing this &lt;em&gt;fait accompli&lt;/em&gt; with the actual situation we wish to understand (the behavior of the coin, or decisions that are to be made as a result of its outcome) essentially denies a role for probability in understanding the world.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This contrast is thrown in sharp relief within an experimental or regulatory arena.  In such cases the scientist or the regulator know they will be faced with situations whose outcomes, at any time beforehand, are unknown, yet they must make important determinations such as how to design the experiment or establish the criteria to use in determining compliance with regulations (for drug testing, workplace safety, environmental standards, and so on).  These people and the institutions for which they work need &lt;em&gt;methods&lt;/em&gt; and knowledge of the &lt;em&gt;probabilistic characteristics of those methods&lt;/em&gt; in order to develop optimal and defensible strategies, such as good experimental designs and fair decision procedures that err as little as possible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Confidence intervals, despite their classically poor justification, fit into this decision-theoretic framework.  When a method of constructing a random interval has a combination of good properties, such as assuring a minimal expected coverage of the interval and minimizing the expected length of the interval--both of them &lt;em&gt;a priori&lt;/em&gt; properties, not &lt;em&gt;a posteriori&lt;/em&gt; ones--then over a long career of using that method we can minimize the costs associated with the actions that are indicated by that method.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-11-03T00:38:16.730" />
  <row Id="4154" PostTypeId="2" ParentId="4150" CreationDate="2010-11-03T00:39:58.123" Score="3" ViewCount="" Body="&lt;p&gt;You should google &lt;a href=&quot;http://en.wikipedia.org/wiki/Expectation-maximization_algorithm&quot; rel=&quot;nofollow&quot;&gt;EM algorithm&lt;/a&gt;. The wiki has a description of the algorithm with an example of the application of this algorithm to &lt;a href=&quot;http://en.wikipedia.org/wiki/Expectation-maximization_algorithm#Example%3a_Gaussian_mixture&quot; rel=&quot;nofollow&quot;&gt;gaussian mixtures&lt;/a&gt;. Perhaps, someone else can point out an R package for you.&lt;/p&gt;&#xA;" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-11-03T00:45:52.243" LastActivityDate="2010-11-03T00:45:52.243" CommentCount="3" />
  <row Id="4155" PostTypeId="1" AcceptedAnswerId="4156" CreationDate="2010-11-03T02:22:36.290" Score="3" ViewCount="146" Body="&lt;p&gt;I've been learning X12-ARIMA by looking at data from a friend's service company, and wondering how to model the capacity of the company. That is, if the company is limited by a particular resource to only be able to handle 1,000 customers a week, how do I keep my ARIMA model from happily predicting 1,200 customers next summer?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(This isn't an issue with time series like GDP or stock prices, which don't have a hard cap.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It doesn't seem that you can do anything in the optimization phase (which is simply choosing parameters), nor with exogenous variables (which drive the process, not react to it). Maybe changing the ARIMA model to a State Space representation would help? (Any recommendations on an R package to do this? I've looked at several and DLM's many matrices confuse me at this point.)&lt;/p&gt;&#xA;" OwnerUserId="1764" LastActivityDate="2011-03-04T20:38:15.360" Title="Saturation in ARIMA (et al) models?" Tags="&lt;time-series&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="4156" PostTypeId="2" ParentId="4155" CreationDate="2010-11-03T03:01:37.603" Score="3" ViewCount="" Body="&lt;p&gt;If Y is customer demand, than you are observing X=min(Y,1000) due to resource constraints. The actual Y could be larger, but you never observe it. So if you fit a time series model to X, you can set the forecasts to min(F,1000) where F is the forecast from the time series model. I don't think there is a need to do anything more fancy than that.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-11-03T03:01:37.603" CommentCount="4" />
  <row Id="4157" PostTypeId="1" AcceptedAnswerId="4163" CreationDate="2010-11-03T03:18:12.013" Score="2" ViewCount="160" Body="&lt;p&gt;I've really only seen EM used for mixtures where one can point out multiple modes visually - e.g, the classic mixture of gaussians example.  I would like to use EM for a mixture of an empirically defined, sharply peaked distribution and something that is more uniform - does anyone have an idea as to how much confidence I should put in the resulting estimates, or prior experience with similar applications?&lt;/p&gt;&#xA;" OwnerUserId="1777" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-27T16:05:08.313" LastActivityDate="2011-03-27T16:05:08.313" Title="Using the EM Algorithm for unimodal distributions?" Tags="&lt;modeling&gt;&lt;mixture&gt;&lt;em-algorithm&gt;" AnswerCount="3" CommentCount="1" />
  <row Id="4158" PostTypeId="2" ParentId="3814" CreationDate="2010-11-03T04:01:10.777" Score="7" ViewCount="" Body="&lt;p&gt;Confusing p-values and effect size (i.e. stating my effect is large because I have a really tiny p-value).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Slightly different than Stephan's &lt;a href=&quot;http://stats.stackexchange.com/questions/3814/how-to-annoy-a-statistical-referee/3817#3817&quot;&gt;answer&lt;/a&gt; of excluding effect sizes but giving p-values. I agree you should give both (and hopefully understand the difference!)&lt;/p&gt;&#xA;" OwnerUserId="1036" LastActivityDate="2010-11-03T04:01:10.777" CommunityOwnedDate="2010-11-03T04:01:10.777" />
  <row Id="4163" PostTypeId="2" ParentId="4157" CreationDate="2010-11-03T07:46:46.563" Score="3" ViewCount="" Body="&lt;p&gt;There are two questions here: 1) how much confidence should you put in your model with peaked and flat components. 2) how much confidence should you put in the EM algorithm as a way to fit this model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Question 1 has the same answers as any other model, e.g. a regression model with particular covariates or a factor analysis model with a certain number of factors.  The only specific consideration I can think of is that you may be introducing a the alternative flat data generating source to make a more robust estimate of peaked data source.  This is a standard noisy measurement model.  For comparison, you might also work with a fatter tailed peaked source, e.g. T-distribution vs. Normal + Uniform.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for Question 2, EM is just a maximum likelihood method.  This means, first that there may be better parameter values available, because it may have found a local minimum, and second that you can get degenerate solutions because there is no prior / regularization in the setup.  Both are standard ML problems, not really anything to do with the EM algorithm, although the latter is probably made slightly worse by having missing data in the mix.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a more elaborate discussion, see McLachlan and Krishnan's 'EM Algorithm and Extensions'.&lt;/p&gt;&#xA;" OwnerUserId="1739" LastActivityDate="2010-11-03T07:46:46.563" />
  <row Id="4164" PostTypeId="2" ParentId="4138" CreationDate="2010-11-03T08:29:08.687" Score="3" ViewCount="" Body="&lt;p&gt;I interpreted the question to ask the distribution of the maximal element of a multivariate normal. In this case, the CDF can be computed from the CDF of a multivariate normal. This usually doesn't have a nice solution (even in terms of the univariate normal CDF), however can be evaluated numerically. In R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(mvtnorm)&#xA;# given xl, mu and sigma&#xA;pmvnorm(upper=rep(xl,length(mu)), mean=mu, sigma=sigma)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However on re-reading the question, it seems to be asking the probability that a particular element of the vector is maximal. In this case, I'd agree with G. Jay Kerns.&lt;/p&gt;&#xA;" OwnerUserId="495" LastEditorUserId="495" LastEditorDisplayName="" LastEditDate="2010-11-04T12:13:54.883" LastActivityDate="2010-11-04T12:13:54.883" CommentCount="1" />
  <row Id="4165" PostTypeId="1" CreationDate="2010-11-03T08:57:40.697" Score="4" ViewCount="444" Body="&lt;p&gt;I wish to decide if I should take a course called &quot;INTRODUCTION TO STOCHASTIC PROCESSES&quot; which will be held next semester in my University.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I asked the lecturer how studying such a course would help me as a statistician, he said that since he comes from probability, he knows very little of statistics and doesn't know how to answer my question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can make an un-educated guess that stochastic processes are important in statistics.  But I am also curious to know how.&#xA;That is, &lt;strong&gt;in what fields/methods, will basic understanding in &quot;stochastic processes&quot; will help me do better statistics?&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="253" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-11-03T10:10:14.340" LastActivityDate="2010-11-05T14:04:18.887" CommunityOwnedDate="2010-11-03T15:51:11.590" Title="How will studying &quot;stochastic processes&quot; help me as a statistician?" Tags="&lt;probability&gt;&lt;stochastic-processes&gt;" AnswerCount="6" CommentCount="1" FavoriteCount="2" />
  <row Id="4166" PostTypeId="2" ParentId="4165" CreationDate="2010-11-03T10:35:34.100" Score="2" ViewCount="" Body="&lt;p&gt;A deep understanding of &lt;strong&gt;survival analysis&lt;/strong&gt; requires knowledge of counting processes, martingales, Cox processes... See e.g. Odd O. Aalen, Ørnulf Borgan, Håkon K. Gjessing. &lt;em&gt;Survival and event history analysis: a process point of view&lt;/em&gt;. Springer, 2008. &lt;a href=&quot;http://en.wikipedia.org/wiki/Special%3aBookSources/9780387202877&quot; rel=&quot;nofollow&quot;&gt;ISBN 9780387202877&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having said that, many applied statisticians (including me) &lt;em&gt;use&lt;/em&gt; survival analysis without any understanding of stochastic processes. I'm not likely to make any advances to the theory though.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-11-03T10:35:34.100" CommunityOwnedDate="2010-11-03T15:51:11.590" />
  <row Id="4167" PostTypeId="2" ParentId="4165" CreationDate="2010-11-03T10:36:25.520" Score="10" ViewCount="" Body="&lt;p&gt;Stochastic processes underlie many ideas in statistics such as time series, markov chains, markov processes, bayesian estimation algorithms (e.g., Metropolis-Hastings) etc. Thus, a study of stochastic processes will be useful in two ways:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Enable you to develop models for situations of interest to you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An exposure to such a course, may enable you to identify a standard stochastic process that works given your problem context. You can then modify the model as needed to accommodate the idiosyncrasies of your specific context.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Enable you to better understand the nuances of the statistical methodology that uses stochastic processes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are several key ideas in stochastic processes such as convergence, stationarity that play an important role when we want to analyze a stochastic process. It is my belief that a course in stochastic process will let you appreciate better the need for caring about these issues and why they are important.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Can you be a statistician without taking a course in stochastic processes? Sure. You can always  use the software that is available to perform whatever statistical analysis you want. However, a basic understanding of stochastic processes is very helpful in order to make a correct choice of methodology, in order to understand what is really happening in the black box etc. Obviously, you will not be able to contribute to the theory of stochastic processes with a basic course but in my opinion it will make you a better statistician. My general rule of thumb for coursework: The more advanced course you take the better off you will be in the long-run.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By way of analogy: You can perform a t-test without knowing any probability theory or  statistics testing methodology. But, a knowledge of probability theory and statistical testing methodology is extremely useful in understanding the output correctly and in choosing the correct statistical test.&lt;/p&gt;&#xA;" LastActivityDate="2010-11-03T10:42:13.277" CommunityOwnedDate="2010-11-03T15:51:11.590" />
  <row Id="4168" PostTypeId="2" ParentId="4165" CreationDate="2010-11-03T10:42:41.717" Score="3" ViewCount="" Body="&lt;p&gt;The short answer probably is that all observable processes, which we may want to analyze with statistical tools, are stochastic processes, that is, they contain some element of randomness. The course will probably teach you the mathematics behind these stochastic processes, e. g. distribution functions, which will allow you to grasp the function of your statistical tools. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think you can compare it with an automobile: As you can drive your car without understanding the engineering behind it, and without theoretical knowledge about the dynamics of your car on the road, you can apply statistical tools to your data without understanding how these tools work, as long as you understand the output. This will probably be good enough if you want to do basic statistics with well behaved data. But if you really want to get the most out of your car, to see where it's limits are, you need knowledge about the engineering, the dynamics of your car on roads and in curves and so on. And if you want to get the most out of your data with the help of your statistical tools, you need to understand how data generation can be modeled, how tests are devised and what the assumptions behind your tests are to be able to see where those assumptions are not valid.&lt;/p&gt;&#xA;" OwnerUserId="1766" LastActivityDate="2010-11-03T10:42:41.717" CommunityOwnedDate="2010-11-03T15:51:11.590" CommentCount="1" />
  <row Id="4169" PostTypeId="2" ParentId="4157" CreationDate="2010-11-03T11:38:25.917" Score="0" ViewCount="" Body="&lt;p&gt;I have a &lt;a href=&quot;http://www.thinkingaboutthinking.org/wp-content/uploads/2010/05/Lawrence_BRM_in_press.pdf&quot; rel=&quot;nofollow&quot;&gt;paper&lt;/a&gt; in press that explores application of EM to estimation of a Von Mises &amp;amp; uniform mixture in the circular domain. (The Von Mises is the circular analogue of a gaussian.)&lt;/p&gt;&#xA;" OwnerUserId="364" LastActivityDate="2010-11-03T11:38:25.917" CommentCount="3" />
  <row Id="4170" PostTypeId="2" ParentId="192" CreationDate="2010-11-03T12:52:22.440" Score="2" ViewCount="" Body="&lt;p&gt;Arguably, the question is not very precise. Rather than enumerating all measures of association for $2\times 2$ tables, I shall concentrate on the way such measures may be constructed and how to select the one that is most appropriate with respect to hypothesis or constraints relevant to a cross-classification. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The very first questions to ask are: what does the table reflect (concordance, agreement, association between two attributes, etc.), do you seek an overall measure of association or do you think one of the two variables plays a specific role (which would justify the search for an &quot;oriented&quot; association), do you consider either or both of the margins fixed (row and/or columns totals)? All of this impact on the method to choose and the way to interpret the results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The $2\times 2$ case&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Two-by-two tables are often treated separately from $I\times J$ tables because we often consider that variables play a symmetric role in this particular case. Obviously, this is not always the case: cross-classification of exposure and disease, as commonly found in epidemiological studies, is an example where both variables play a distinct role, which may lend to more than a simple interpretation in terms of association. Another one is $2\times 2$ tables constructed for studying the screening properties of a given diagnostic instrument. Although the odds-ratio (compared to, e.g. the relative risk) keeps its nice properties, we may be interested in predictive/negative positive values or specificity/sensibility, which means working with other quantities of interest. Hence, the need to specifiy whether the problem at hand implies two variables that are purely acting in a symmetrical way, or not, because it influences the way we interpret the results or derive a useful measure of association, agreement, or discrimination.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the sake of clarity, I will consider that data (counts) are arranged in the following way:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/41Wjh.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically, measures of association for $2\times 2$ tables can be grouped in two classes: those relying on (a) (a function of) the cross-product ratio and those based on (b) the product-moment (Pearson) correlation, or a function thereof. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The cross-product ratio, mostly known as the odds-ratio, is simply $\alpha=p_{11}p_{22}/p_{12}p_{21}$. It is invariant under rows &lt;em&gt;and&lt;/em&gt; columns interchange, and transformations of margins that preserves $\sum_{i,j}p_{ij}=1$. In epidemiology, we usually think of it as a measure of association where rows (or columns) are fixed: $p_{11}/p_{12}$ is then the odds of being in the first column (e.g., diseased) conditional on being in the first row (e.g., exposed), and likewise $p_{21}/p_{22}$ is the odds for the second row, or in other words &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\alpha=\frac{p_{11}/p_{12}}{p_{21}/p_{22}}.&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Yule's $Q=(\alpha-1)/(\alpha+1)$ fall into the former case, (a). Yule also proposed a measure of &quot;colligation&quot;, $Y$, as $(\sqrt{\alpha}-1)/(\sqrt{\alpha}+1)$. Yule's $Q$ can be interpreted as the difference between conditional probabilities of like and unlike &quot;orders&quot; for two individuals chosen at random; it is identical to Goodman and Kruskal's $\gamma$ measure of association for $I\times J$ tables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For (b), we can derive a correlation coefficient for a $2\times 2$ table by thinking of the table as a combination of each of two variables scores (taking value 0 and 1, for the first and second row/column, resp.). Then, the coefficient $\rho$ is defined as the covariance divided by the square root of the product of the variances:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\rho=\frac{p_{22}-p_{2\cdot}p_{\cdot 2}}{\sqrt{p_{1\cdot}p_{2\cdot}p_{\cdot 1}p_{\cdot 2}}},&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;which is equivalent to putting $p_{11}p_{22}-p_{21}p_{12}$ in the numerator. Plugging in the observed counts, Pearson's $r$ is the MLE of $\rho$ under a multinomial sampling model.&#xA;It is invariant under rows &lt;em&gt;and&lt;/em&gt; columns interchange, and positive linear transformation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It can be shown (Yule, 1912) that $\rho$ is identical to Yule's $Y$ if we standardize our table such that row and column margins sum to 1/2, i.e. $p_{11}^*=p_{22}^*=0.5\left(\sqrt{\alpha}/(\sqrt{\alpha}+1)\right)$ and $p_{12}^*=p_{21}^*=0.5\left(1/(\sqrt{\alpha}+1)\right)$. By doing this, we remove the information coming from the margins, such that $Y=2(p_{11}^*-p_{12}^*)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Correlation-based measures are connected to the usual Pearson's chi-square statistic, since&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\Phi^2=\sum_{i=1}^2\sum_{j=1}^2\frac{(p_{ij}-p_{i\cdot}p_{\cdot j})^2}{p_{i\cdot}p_{\cdot j}},&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;that is,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\Phi^2=\frac{(p_{11}p_{22}-p_{21}p_{12})^2}{p_{1\cdot}p_{2\cdot}p_{\cdot 1}p_{\cdot 2}}=\rho^2.&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a $2\times 2$ table, we thus have $r^2=\chi^2/N$. Pearson also proposed to use $\sqrt{\rho^2/(1+\rho^2)}$ as a measure of association, and he coined it the &lt;em&gt;coefficient of mean square contingency&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As to how to choose the correct measure (a &lt;em&gt;vs.&lt;/em&gt; b), it clearly depends on whether we want to be sensitive to marginal totals (in this case, $\rho$ cannot take its full range of possible values in $[-1;1]$), and whether we consider  that we observe a full association even if one of the four cells is zero (in this case, $\rho$ cannot take the value $+1$ or $-1$ if only one of the cells is zero, which is not the case of Yule's $Q$).&#xA;Of note, correlation-based measures are better if they are used in a correlation matrix (e.g., for factor analysis), because we cannot guarantee that a matrix composed of Yule's $Q$ coefficient will be positive definite. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The $I\times J$ case&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Like for the $2\times 2$ case, we can derive measures of association based on different quantities. Measures based on chi-square include &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Pearson's $P$ coefficient based on $\Phi^2$ (see above), $\sqrt{\Phi^2/(\Phi^2+1)}$ (to overcome the fact that $\Phi^2$ no longer lies in $[0;1]$ when $I$ or $J&gt;2$);&lt;/li&gt;&#xA;&lt;li&gt;Tschuprow's $T=\left(\Phi^2/\sqrt{(I-1)(J-1)}\right)^{1/2}$, which behaves better than $P$ in square tables (in that it can reach a maximum value of 1, for full or complete association);&lt;/li&gt;&#xA;&lt;li&gt;Cramer's $V$ is another derivation, and $V=\left(\Phi^2/\text{min}(I-1,J-1)\right)^{1/2}$ (we have $V\geq T$ for all $I,J&gt;2$).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;These measures are all measures of association where none of the variables plays a specific role. In case a $\chi^2$ test is significant, it is more interesting to look at how the expected counts depart from the observed counts (i.e. look at the Pearson residuals) in all $(i,j)$ cells, or to use something like a &lt;a href=&quot;http://www.statmethods.net/advgraphs/mosaic.html&quot; rel=&quot;nofollow&quot;&gt;mosaic plot&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Goodman and Kruskall (1954) also proposed a predictive measure of association between rows and columns, or more specifically a measure of proportional reduction in error in predicting one column category when the row category is known as opposed to the case when the latter one is unknown. This is called $\lambda_{C|R}$ and its MLE is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\hat\lambda_{C|R}=\frac{\sum_{i=1}^Ix_{im}-x_{\cdot m}}{N-x_{\cdot m}}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $x_{im}$ and $x_{\cdot m}$ stand for the maximum for the $i$th row and the column totals. This measure is interesting because it has a nicer interpretation than $\chi^2$-based measure, but it also has some drawbacks: when there is statistical independence, $\lambda_{C|R}$ is not necessarily zero, for instance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A measure of the proportion of explained variance (derived from Gini's total variation) may be derived from the total sum of squares (SS) in an $I\times J$ table&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\text{TSS}=\frac{N}{2}-\frac{1}{2N}\sum_{i=1}^Ix_{i\cdot}^2,&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;which can be partitioned as a within- and between-group SS. Of interest here is the variance explained by considering the different categories (BSS) divided by the total variance, TSS. Like in the ANOVA framework, we have BSS=TSS-WSS, where&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\text{WSS}=\frac{N}{2}-\frac{1}{2}\sum_{j=1}^J\frac{1}{x_{\cdot j}}\sum_{i=1}^Ix_{ij}^2,&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;so that we can derive BSS/TSS as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\hat\tau_{R|C}=\frac{\sum_j\frac{1}{x_{\cdot j}}\sum_i x_{ij}^2-\frac{1}{N}\sum_ix_{i\cdot}^2}{N-\frac{1}{N}\sum_ix_{i\cdot}^2}.&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This measure can be interpreted as &quot;the relative decrease in the proportion of incorrect predictions when we go from predicting the row category based only on the row marginal probabilities to predicting the row category based on the conditional proportions $p_{ij}/p_{\cdot j}$&quot; (Bishop et al., 2007, p. 391).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, measures based on the cross-product ratios are also available, as well as measures of agreement for ordinal variables, but I realize now that I need to stop (and thank the reader who reached the end of this overview).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A thorough overview of measures of association may be found in Bishop et al. (2007), from which I grabbed most of the above discussion, and of course Agresti (2002), about which Laura Thompson made a complete R adaptation in his textbook &lt;a href=&quot;https://home.comcast.net/~lthompson221/Splusdiscrete2.pdf&quot; rel=&quot;nofollow&quot;&gt;R (and S-PLUS) Manual to Accompany Agresti's Categorical Data Analysis&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;References&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Agresti, A. (2002). &lt;em&gt;Categorical Data Analysis&lt;/em&gt;. Wiley. &lt;a href=&quot;http://www.stat.ufl.edu/~aa/cda/cda.html&quot; rel=&quot;nofollow&quot;&gt;Companion website&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Bishop, Y.M., Fienberg, S.E., and Holland, P.W. (2007). &lt;em&gt;Discrete Multivariate Analysis&lt;/em&gt;. Springer.&lt;/li&gt;&#xA;&lt;li&gt;Goodman, L.A. and Kruskall, W.H. (1954). Measures of association for cross-classification. &lt;em&gt;JASA&lt;/em&gt;, &lt;em&gt;49&lt;/em&gt;, 732-764.&lt;/li&gt;&#xA;&lt;li&gt;Yule, G.U. (1912). On the methods of measuring association between two attributes. &lt;em&gt;Journal of the Royal Society&lt;/em&gt;, &lt;em&gt;75&lt;/em&gt;, 579-642.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-03T15:47:37.890" LastActivityDate="2010-11-03T15:47:37.890" />
  <row Id="4171" PostTypeId="2" ParentId="4165" CreationDate="2010-11-03T13:00:45.750" Score="1" ViewCount="" Body="&lt;p&gt;Just for the sake of completeness, an IID sequence of random variables is also a stochastic process (a very simple one).&lt;/p&gt;&#xA;" OwnerUserId="247" LastActivityDate="2010-11-03T13:00:45.750" CommunityOwnedDate="2010-11-03T15:51:11.590" />
  <row Id="4172" PostTypeId="1" CreationDate="2010-11-03T13:17:39.483" Score="3" ViewCount="180" Body="&lt;p&gt;Hello, I am new to the area of statistics and I am hoping you can suggest methods I may use. Sorry if this is long but I might as well be as clear as possible on my first post :)&lt;br&gt;&#xA;&lt;em&gt;What I am worried most is that I may miss out on assumptions and draw conclusions based on statistical tests that, in fact, cannot be applied to my situation.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;In a nutshell:&lt;/strong&gt;&lt;br&gt;&#xA;We are replacing a measurement tool + methodology with another tool and a similar methodology and I would like to prove that the new tool &amp;amp; methodology provide the same &quot;results&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The data reported :&lt;/strong&gt; &#xA;Each tool reports 1) the GPS position, 2) a category of measurement (type 1, type 2, type 3) (the categories are the same for both measurement tools and relate to what is being measured, they should report the same thing), and 3) a quantized value of a continuous value. The measurement tools probably quantize the value with different algorithms but, according to spec, they should provide the same value.&lt;br&gt;&#xA;Given what we're measuring the measurements are definately not stationnary and, since we're measuring a physical quantity, I assume the time series are autocorrelated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How the setups differ:&lt;/strong&gt;&lt;br&gt;&#xA;&lt;em&gt;Setup 1 (historical setup) :&lt;/em&gt; uses tool &quot;A&quot;, takes a measurement 3 times a minute and reports the GPS position, the category of the measurement and the discrete value&lt;br&gt;&#xA;&lt;em&gt;Setup 2 (new setup) :&lt;/em&gt; uses tool &quot;B&quot;, takes a measurement up to every second (but not necessarily based on distance criteria between measurements) and report the GPS position, the category and discrete value too&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Our experiment:&lt;/strong&gt;&lt;br&gt;&#xA;We put both tools in a car and traveled enough to gather over 100.000 data points for setup 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What I would like to prove:&lt;/strong&gt;&lt;ul&gt;&lt;li&gt;the categories reported by setup 1 and 2 do not significantly differ&lt;/li&gt;&lt;li&gt;the discrete value measurements do not significantly differ either&lt;/li&gt;&lt;li&gt;if the new setup and a bias or skew compared to the other one&lt;/li&gt;&lt;/ul&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What I have done so far:&lt;/strong&gt;&lt;br&gt;&#xA;I have matched each data point of setup 1 to a single data point in setup 2 (the one that is &quot;closest geographically&quot; in a 4 minute-time window).  &lt;em&gt;Is this even statistically sound ?&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Regarding the discrete value reported,&lt;ul&gt;&lt;li&gt;I drew a scatter plot of the discrete values for matched data points with bubble sizes corresponding to the count for each (x,y) : the data clusters along a 45° angle line as expected but I can see there is some bias.&lt;br&gt;&#xA;There is also some spread a round that line&lt;/li&gt;&#xA;&lt;li&gt;I drew a Bland-Altman/Tukey diagram of the same data and I now see that the average difference depends on the average mean. That's interesting to know&lt;/li&gt;&#xA;&lt;li&gt;I computed the pearson correlation for matches that are in the same category : I get 0.87 which seems to be high enough to look good.&lt;br&gt;&#xA;&lt;em&gt;Can Pearson be applied given I have no idea if the distribution is normal and since the measurements are definalty not independent inside the time series ? Would the U test be better ?&lt;/em&gt;&lt;/li&gt;&#xA;&lt;li&gt;I tried to compute a t test but I'm getting t values in the &quot;80&quot; range because SQRT(N) is huge&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to use all the data collected in setup 2 rather than only the data that was matched 1 to 1. There is about 4 times more data reported by setup 2 than setup 1.&lt;br&gt;&#xA;I've been looking into non-parametric tests and I believe that is what applies to my case as well as the whole notion of inter-rater agreement. So it seems like my next steps will be to use R to compute Cohen's Kappa and KrippenDorff's alpha.&lt;br&gt;&#xA;&lt;em&gt;Would computing these and finding high correlations be enough to make my point ?&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Regarding categories reported, again the data reported in the time series are correlated because if category 1 is reported then the chance of the next category being reported being 1 is higher than if category 2 had been reported.&lt;br&gt;&#xA;&lt;em&gt;Given that there are three categories, what kind of tests could I apply ?&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;thanks for your suggestions&lt;/p&gt;&#xA;" OwnerUserId="1784" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-03T15:25:35.183" LastActivityDate="2010-12-23T16:59:38.090" Title="How to &quot;prove&quot; that new measurement tool &amp; process gives same result as old ?" Tags="&lt;time-series&gt;&lt;correlation&gt;&lt;reliability&gt;&lt;inter-rater&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="1" />
  <row Id="4173" PostTypeId="2" ParentId="4157" CreationDate="2010-11-03T14:28:21.180" Score="2" ViewCount="" Body="&lt;p&gt;I have used various algorithms, including Bayesian approaches (and, I am sorry to confess, even Excel many years ago), to fit mixtures.  When there is not a clear visual indication of the two (or more components) in the histogram, you can expect the likelihood function to be extremely flat--almost parabolic--near its peak.  This is because the visual impression translates mathematically into an ability to trade off some proportion of one mixture with an equivalent proportion of the other (adjusting the parameters of the components to keep a good fit) while making only a minor change to the likelihood.  In many cases it's difficult to pin down the maximum likelihood.  (This is evidenced by regime-switching in the Markov chains, for instance: a chain will pursue an area where one component predominates and after longish periods switch to an area where another component predominates, never really settling down to a single optimum.)  In any event &lt;em&gt;you also want to assess uncertainty.&lt;/em&gt;  This is reflected by how much change is needed in the mixture parameters to reduce the likelihood by some threshold amount.  The near-parabolic flatness near the optimum delineates a long &quot;ridge&quot; of near-optimum values, resulting in a long elliptical confidence region for the mixture.  Usually the major axis of that ellipse corresponds to the mixture proportions.  Thus, you might conclude that your data are $p$ percent of component A and $1-p$ percent of component B, but $p$ might be anywhere from 0 to 70%.  (Yes, there are boundary value problems with mixtures, too.)  It can take an extraordinary amount of data to reduce these wide confidence intervals if you can even reliably find them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These problems are exacerbated when only the tails of the data provide most of the information needed to disentangle the distributions.  This would often be the case for unimodal data.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-11-03T14:28:21.180" />
  <row Id="4174" PostTypeId="1" AcceptedAnswerId="4180" CreationDate="2010-11-03T14:52:52.013" Score="4" ViewCount="279" Body="&lt;p&gt;At work we have a hardware device that is failing for some yet to be determined reason. I have been tasked to see if I can make this device &lt;em&gt;not&lt;/em&gt; fail by making changes to its software driver. I have constructed a software test bench which iterates over the driver functions which I feel are most likely to cause the device to fail. So far I have forced 7 such failures and the iterations that the device failed on are as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;100&#xA;22&#xA;36&#xA;44&#xA;89&#xA;24&#xA;74&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Mean = 55.57&#xA;Stdev = 31.81&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Next, I made some software changes to the device driver and was able to run the device for 223 iterations &lt;em&gt;without&lt;/em&gt; failure before I manually stopped the test.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to be able to go back to my boss and say &quot;The fact that we were able to run the device for 223 iterations without failure means that my software change has a X% probability of fixing the problem.&quot; I would also be satisfied with the converse probability that the device will still fail with this fix.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we were to assume that the iteration the device fails on is normally distributed, we can say that going 223 iterations without failure is 5.26 standard deviations from the mean which roughly has a 1-in-14 million chance of happening. However, because we only have a sample size of 7 (not including the 223), I'm fairly certain it would be unwise to assume normality.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is where I think the Student's t-test comes into play. Using the t-test with 6 degrees of freedom, I've calculated that the actual population mean has a 99% probability of being less than 94.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;So now my question to you guys is whether or not I am allowed to say with 99% certainty that hitting 223 iterations without failure is a 4.05 sigma event, i.e. $\frac{(223 - 94)}{31.81} = 4.05$ ?&lt;/strong&gt; Am I allowed to use the 31.81 sample standard deviation in that calculation or is there some other test I should do to get a 99% confidence on what the maximum standard deviation is and then use that in my calculation for how many sigmas 223 really is away from the mean at the 99% confidence level?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The answers I received here are beyond any expectation I had.  I truly appreciate the time and thought many of you have put into your answers.  There is much for me to think about.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In response to whuber's concern that the data does not seem to follow an exponential distribution, I believe I have the answer for as to why.  Some of these trials were run with what I thought would be a software fix but ultimately ended in failure.  I would not be surprised if those trials were the 74 89 100 grouping that we see.  Although I wasn't able to fix the problem it certainly seems like I was able to skew the data.  I will check my notes to see if this is the case and my apologies for not remembering to include that piece of information earlier. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lets assume the above is true and we were to remove 74 89 100 from the data set.  If I were to re-run the device with the original driver and get additional failure data points with values 15 20 23, how would you then compute the exponentially distributed parametric prediction limit at the 95% confidence level? Would you feel that this prediction limit is still a better statistic than assuming independent Bernoulli trials to find the probability of no failure at 223 iterations?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Looking more closely at the wikipedia page on Prediction Limits I calculated the parametric prediction limits at the 99% confidence level assuming unknown population mean and unknown stdev on Excel as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\bar{X_n} = 55.57$&lt;br&gt;&#xA;$S_n = 31.81$&lt;br&gt;&#xA;$T_a = T.INV\Bigl(\frac{1+.99}{2},6\Bigr)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Lower Limit = 55.57 - 3.707*31.81*\sqrt{1+\frac{1}{7}} = -70.51$&lt;br&gt;&#xA;$Upper Limit = 55.57 + 3.707*31.81*\sqrt{1+\frac{1}{7}} = 181.65$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since my trial of 223 is outside the 99% confidence interval of [-70.51 , 181.65] can I assume with 99% probability that this is fixed assuming that the underlying distribution is the T-Distribution?  I wanted to make sure my understanding was correct even though the underlying distribution is most likely exponential, not normal.  I have no clue in the slightest how to adjust the equation for an underlying exponential distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE 2&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I'm really intrigued with this 'R' software, I've never seen it before.  Back when I took my stats class (several years ago) we used SAS.  Anyway, with the cursory knowledge I gathered from Owe Jessen's example and a bit of help from Google, I think I came up with the following R code to produce the &lt;strong&gt;Prediction Limits&lt;/strong&gt; with the hypothetical dataset assuming an &lt;em&gt;Exponential Distribution&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let me know if I got this right:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;fails &amp;lt;- c(22, 24, 36, 44, 15, 20, 23)&#xA;fails_xfm &amp;lt;- fails^(1/3)&#xA;Y_bar &amp;lt;- mean(fails_xfm)&#xA;Sy &amp;lt;- sd(fails_xfm)&#xA;df &amp;lt;- length(fails_xfm) - 1&#xA;no_fail &amp;lt;- 223&#xA;&#xA;percentile &amp;lt;- c(.9000, .9500, .9750, .9900, .9950, .9990, .9995, .9999)&#xA;quantile &amp;lt;- qt(percentile, df)&#xA;&#xA;UPL &amp;lt;- (Y_bar + quantile*Sy*sqrt(1+(1/length(fails_xfm))))^3&#xA;plot(percentile,UPL)&#xA;abline(h=no_fail,col=&quot;red&quot;) &#xA;text(percentile, UPL, percentile, cex=0.9, pos=2, col=&quot;red&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://img411.imageshack.us/img411/5246/grafr.png&quot; alt=&quot;Prediction Limits&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="1786" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-11-05T09:11:27.013" LastActivityDate="2010-11-05T09:11:27.013" Title="Estimating the probability that a software change fixed a problem" Tags="&lt;distributions&gt;&lt;hypothesis-testing&gt;&lt;t-test&gt;&lt;prediction-limit&gt;&lt;failure&gt;" AnswerCount="4" CommentCount="6" FavoriteCount="3" />
  <row Id="4175" PostTypeId="1" AcceptedAnswerId="4324" CreationDate="2010-11-03T15:07:30.070" Score="2" ViewCount="419" Body="&lt;p&gt;I am trying to understand how I can use resampling techniques to compliment my pre-planned analyses. This is not homework. I have a 5 sided die. 30 subjects call a number (1-5) and then roll the die. If it matches it's a hit, if not it's a miss. Each subject does this 25 times. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;N is the the number of trials (=25) and p is the probability of getting it correct (=.2) then the population value (mu) of the mean number correct is n*p=5. The population standard deviation, sigma, is square-root(n*p*[1-p]), which is 2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The experimental hypothesis (H1) is that subjects in this study will score above chance (above mu). The null hypothesis (H0) assumes a binomial distribution for each subject (they will score at mu).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[&lt;em&gt;Please don't get too worried about why I am doing this. If it helps you to understand the problem then you can think of it as an ESP test (and therefore I am testing the ability of subjects to score above mu). Also if it helps, imagine that the task is a virtual reality die throwing task, where the virtual 5-sided die performs according to chance. There can be no bias from an imperfect die because the die is virtual.&lt;/em&gt;]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Okay. So before I conducted the &quot;experiment&quot; I had planned to compare the 30 subjects score with a one-sample t-test (comparing it to the null that mu=5). Then I discovered that the one-sample z-test was a more powerful test given what we know about the null hypothesis. Okay.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a simulation of my data in R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;binom.samp1 &amp;lt;- as.data.frame(matrix(rbinom(30*1, size=25, prob=0.2), ncol=1))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now R has a binom.test function, which gives the p-value regarding the number of successes over the number of trials. For my collected data (not the simulated data given):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;binom.test(174, 750, 1/5, alternative=&quot;g&quot;)&#xA;number of successes = 174, number of trials = 750, p-value = 0.01722&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now the one-sample t-test that I had originally planned to use (mainly because I'd never heard of the alternatives - should've paid more attention in higher statistics):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;t.test(binom.samp1-5, alternative=&quot;g&quot;)&#xA;t = 1.7647, df = 29, p-value = 0.04407&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and for completedness sake: the one-sample z-test (&lt;a href=&quot;http://rgm2.lab.nig.ac.jp/RGM2/R_man-2.9.0/library/BSDA/man/z.test.html&quot; rel=&quot;nofollow&quot;&gt;BSDA package&lt;/a&gt;):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;z.test(binom.samp1, mu=5, sigma.x=2, alternative=&quot;g&quot;)&#xA;z = 2.1909, p-value = 0.01423&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So. My first question is, &lt;strong&gt;am I right in concluding that the &lt;a href=&quot;http://sekhon.berkeley.edu/stats/html/binom.test.html&quot; rel=&quot;nofollow&quot;&gt;binom.test&lt;/a&gt; is the correct test given the data and hypothesis?&lt;/strong&gt; In other words, does &lt;em&gt;t&lt;/em&gt; approximate to &lt;em&gt;z&lt;/em&gt; which approximates to the exact binom.test (&lt;a href=&quot;http://en.wikipedia.org/wiki/Bernoulli_trial&quot; rel=&quot;nofollow&quot;&gt;Bernoulli trial&lt;/a&gt;)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now my second question relates to the resampling methods. I have several books by Philip Good and I've read plenty on permutation and bootstrapping. I was just going to use the one-sample permutation test given in the &lt;a href=&quot;http://pbil.univ-lyon1.fr/library/DAAG/html/onet.permutation.html&quot; rel=&quot;nofollow&quot;&gt;DAAG&lt;/a&gt; package:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;onet.permutation(binom.samp1-5)&#xA;0.114&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And the perm.test function in the &lt;a href=&quot;http://www.stat.ucl.ac.be/ISdidactique/Rhelp/library/exactRankTests/html/perm.test.html&quot; rel=&quot;nofollow&quot;&gt;exactRankTests&lt;/a&gt; package gives this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;perm.test(binom.samp1, mu=5, alternative=&quot;g&quot;, exact=TRUE)&#xA;T = 42, p-value = 0.05113&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;I have the feeling that what I want to do is conduct a one-sample permutation binom.test.&lt;/strong&gt; The only way I can see it working is if I take a subset of the 30 subjects and calculate the binom.test and then repeat it for a large number of N. &lt;strong&gt;Does this sound like a reasonable idea?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, I did repeat this experiment with the same equipment (the 5 sided die) but a larger sample size (50 people) and I got exactly what I expected. My understanding is that the two studies are like a &lt;a href=&quot;http://en.wikipedia.org/wiki/Bean_machine&quot; rel=&quot;nofollow&quot;&gt;Galton box&lt;/a&gt; that hasn't filled up yet. The 30 n experiment has a bit of a skew, but had it been run for longer it would have filled up to the binomial. &lt;strong&gt;Is this all gibberish?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;binom.test(231, 1250, 1/5, alternative=&quot;g&quot;)&#xA;number of successes = 231, number of trials = 1250, p-value = 0.917&#xA;&#xA;&amp;gt;t.test(binom.samp2-5)&#xA;t = -1.2249, df = 49, p-value = 0.2265&#xA;&#xA;&amp;gt;z.test(binom.samp2, mu=5, sigma.x=2)&#xA;z = -1.3435, p-value = 0.1791&#xA;&#xA;&amp;gt;onet.permutation(binom.samp2-5)&#xA;0.237&#xA;&#xA;&amp;gt;perm.test(binom.samp2, mu=5, alternative=&quot;g&quot;, exact=TRUE)&#xA;T = 35, p-value = 0.8991&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1614" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-10T07:39:23.930" LastActivityDate="2010-11-10T07:39:23.930" CommunityOwnedDate="2010-11-04T10:02:45.460" Title="Resampling, binomial, z- and t-test: help with real data" Tags="&lt;r&gt;&lt;hypothesis-testing&gt;" AnswerCount="2" CommentCount="10" />
  <row Id="4176" PostTypeId="2" ParentId="4174" CreationDate="2010-11-03T15:16:19.827" Score="3" ViewCount="" Body="&lt;p&gt;There are a few ways of doing this problem. The way I would tackle this problem is as follows.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data you have comes from a &lt;a href=&quot;http://en.wikipedia.org/wiki/Geometric_distribution&quot; rel=&quot;nofollow&quot;&gt;geometric&lt;/a&gt; distribution. That is, the number of &lt;a href=&quot;http://en.wikipedia.org/wiki/Bernoulli_distribution&quot; rel=&quot;nofollow&quot;&gt;Bernoulli trials&lt;/a&gt; before a failure. The geometric distribution has one parameter &lt;em&gt;p&lt;/em&gt;, which is the probability of failure at each point. For your data set, we estimate &lt;em&gt;p&lt;/em&gt; as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{equation}&#xA;\hat p^{-1} = \frac{100 + 22 + 36 + 44 + 89 + 24 + 74}{7} = 55.57&#xA;\end{equation}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So $\hat p = 1/55.57 = 0.018$. From the CDF, the probability of having a run of 223 iterations and observing a failure is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{equation}&#xA;1-(1-\hat p)^{223} = 0.983&#xA;\end{equation}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the probability of running 223 iterations and &lt;em&gt;not&lt;/em&gt; having a failure is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{equation}&#xA;1- 0.983 = 0.017&#xA;\end{equation}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So it seems likely (but not overwhelming so) that you have fixed the problem. If you have a run of about 300 iterations than the probability goes down to 0.004&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Some notes&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;A bernoulli trial is just tossing a coin, i.e. there are only two outcomes.&lt;/li&gt;&#xA;&lt;li&gt;The geometric distribution is usually phrased in terms of success (rather than failure). For you a success is when the machine breaks!&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="8" LastActivityDate="2010-11-03T15:16:19.827" CommentCount="9" />
  <row Id="4177" PostTypeId="2" ParentId="4165" CreationDate="2010-11-03T15:23:45.933" Score="3" ViewCount="" Body="&lt;p&gt;You need to be careful how you ask this question. Since you could substitute almost anything in place of &lt;em&gt;stochastic processes&lt;/em&gt; and it would still be potentially useful. For example, a course in biology could help with biological statistical consultancy since you know more biology!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I presume that you have a choice of modules that you can take, and you need to pick $n$ of them. The real question is what modules should I pick (that question probably isn't appropriate for this site!)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To answer your question, you are still very early in your career and at this moment in time you should try to get a wide selection of courses under your belt. Furthermore, if you are planning a career in academia then some more mathematical courses, like &lt;em&gt;stochastic processes&lt;/em&gt; would be useful.&lt;/p&gt;&#xA;" OwnerUserId="8" LastActivityDate="2010-11-03T15:23:45.933" CommunityOwnedDate="2010-11-03T15:51:11.590" CommentCount="2" />
  <row Id="4178" PostTypeId="2" ParentId="4065" CreationDate="2010-11-03T15:47:20.410" Score="2" ViewCount="" Body="&lt;p&gt;Some time has passed and I think I might have a solution at hand. I will describe my approach briefly to give you the general idea. The code should be enough to figure out the details. I like to attach code here, but it is a lot and stackexchange makes it not easy to do so. I am of course happy to answer any comments, also I appreciate any criticism.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the code: &lt;a href=&quot;http://www.steffen-springer.net/dev/R/binaryModelScores.R&quot; rel=&quot;nofollow&quot;&gt;binaryModelScores.R&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The strategy:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Approximate a smooth ROC-Curve by using the &lt;a href=&quot;http://en.wikipedia.org/wiki/Logistic_function&quot; rel=&quot;nofollow&quot;&gt;Logistic function&lt;/a&gt; in the interval [0,6]&lt;/li&gt;&#xA;&lt;li&gt;By adding a parameter k one can influence the shape of the curve to fit the desired model quality, measured by AUC (Area Under Curve). The resulting function is $f_k(x)=\frac{1}{(1+exp(-k*x))}$. If k-&gt; 0, AUC approaches 0.5 (no optimization), if k -&gt; Inf, AUC approaches 1 (optimal model). As a handy approach, k should be in the interval [0.0001,100]. By some basic calculus, one can create a function to map k to AUC and vice versa. &lt;/li&gt;&#xA;&lt;li&gt;Now, given you have a roc-curve which matches the desired AUC, determine a score by sample from [0,1] uniformly. This represents the fpr (&lt;a href=&quot;http://en.wikipedia.org/wiki/Sensitivity_and_specificity&quot; rel=&quot;nofollow&quot;&gt;False-Positive-Rate&lt;/a&gt;) on the ROC-curve. For simplicity, the score is calculated then as 1-fpr.&lt;/li&gt;&#xA;&lt;li&gt;The label is now determined by sampling from a Bernoulli Distribution with p calculated using the slope of the ROC-Curve at this fpr and the desired overall precision of the scores. In detail: weight(label=&quot;1&quot;):= slope(fpr) mutiplicated by overallPrecision, weight(label=&quot;0&quot;):= 1 multiplicated by (1-overallPrecision). Normalize the weights so that they sum up to 1 to determine p and 1-p.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Here is an example ROC-Curve for AUC = 0.6 and overall precision = 0.1 (also in the code below)&#xA;&lt;img src=&quot;http://i.imgur.com/ObKTU.png&quot; alt=&quot;alt text&quot;&gt;&#xA;&lt;img src=&quot;http://i.imgur.com/xRUf2.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Notes: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;the resulting AUC is not exactly the same as the input AUC, in fact, there is a small error (around 0.02). This error originates from the way the label of a score is determined. An improvement could be to add a parameter to control the size of the error.&lt;/li&gt;&#xA;&lt;li&gt;the score is set as 1-fpr. This is arbitrary since the ROC-Curve does not care how the scores look like as long as they can be sorted. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="264" LastEditorUserId="264" LastEditorDisplayName="" LastEditDate="2010-11-03T16:47:43.637" LastActivityDate="2010-11-03T16:47:43.637" CommentCount="2" />
  <row Id="4179" PostTypeId="2" ParentId="4174" CreationDate="2010-11-03T15:49:57.703" Score="2" ViewCount="" Body="&lt;p&gt;I think you could torture your data a bit with bootstrapping. Following cgillspies calculations with the geometric distribution, I played around a bit and came up with the following R-code - any corrections greatly appreciated:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;fails &amp;lt;- c(100, 22, 36, 44, 89, 24, 74) # Observed data&#xA;N &amp;lt;- 100000 # Number of replications&#xA;Ncol &amp;lt;- length(fails) # Number of columns in the data-matrix&#xA;boot.m &amp;lt;- matrix(sample(fails,N*Ncol,replac=T),ncol=Ncol) # The bootstrap data matrix&#xA;# it draws a vector of Ncol results from the original data, and replicates this N-times&#xA;p.hat &amp;lt;- function(x){p.hat = 1/(sum(x)/length(x))} # Function to calculate the &#xA;# probability of failure&#xA;p.vec &amp;lt;- apply(boot.m,1,p.hat) # calculates the probabilities for each of the   &#xA;# replications&#xA;quant.p &amp;lt;- quantile(p.vec,probs=0.01) # calculates the 1%-quantile of the probs.&#xA;hist(p.vec) # draws a histogram of the probabilities&#xA;abline(v=quant.p,col=&quot;red&quot;) # adds a line where quant.p is&#xA;no.fail &amp;lt;- 223 # Repetitions without a fail after the repair&#xA;(prob.fail &amp;lt;- 1 - pgeom(no.fail,prob=quant.p)) # Prob of no fail after 223 reps with &#xA;# failure prob qant.p&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The idea was to get a worst-case value for the probability, and then use it to calculate the probability of observing no fail after 223 iterations, given the prior failure probability. The worst case of course being a low failure probability to begin with, which would raise the likelihood of observing no failure after 223 iterations without fixing the problem. &#xA;The result was 6.37% - as I understand it, you would have had a 6%-probability of not observing a failure after 223 trials if the problem still exists.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, you could generate samples of trials and calculate the probability from  that:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;boot.fails &amp;lt;- rbinom(N,size=no.fail, prob=quant.p) # repeats draws with succes-rate &#xA;# quant.prob N times.&#xA;mean(boot.fails==0) # Ratio of no successes &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;with the result of 6.51%.&lt;/p&gt;&#xA;" OwnerUserId="1766" LastEditorUserId="1766" LastEditorDisplayName="" LastEditDate="2010-11-03T19:18:57.650" LastActivityDate="2010-11-03T19:18:57.650" CommunityOwnedDate="2010-11-03T15:49:57.703" CommentCount="5" />
  <row Id="4180" PostTypeId="2" ParentId="4174" CreationDate="2010-11-03T16:37:07.917" Score="6" ViewCount="" Body="&lt;p&gt;This question asks for a &lt;strong&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Prediction_interval&quot; rel=&quot;nofollow&quot;&gt;prediction limit&lt;/a&gt;.&lt;/strong&gt;  This tests whether a future statistic is &quot;consistent&quot; with previous data.  (In this case, the future statistic is the post-fix value of 223.)  It accounts for a chance mechanism or uncertainty in three ways:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;The data themselves can vary by chance.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Because of this, any estimates made from the data are uncertain.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The future statistic can also vary by chance.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Estimating a probability distribution from the data handles (1).  But if you simply compare the future value to predictions from that distribution you are ignoring (2) and (3).  This will exaggerate the significance of any difference that you note.  This is why it can be important to use a prediction limit method rather than some &lt;em&gt;ad hoc&lt;/em&gt; method.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Failure times are often taken to be &lt;a href=&quot;http://en.wikipedia.org/wiki/Exponential_distribution&quot; rel=&quot;nofollow&quot;&gt;exponentially distributed&lt;/a&gt; (which is essentially a continuous version of a geometric distribution).  The exponential is a special case of the &lt;a href=&quot;http://en.wikipedia.org/wiki/Gamma_distribution&quot; rel=&quot;nofollow&quot;&gt;Gamma distribution&lt;/a&gt; with &quot;shape parameter&quot; 1.  &lt;em&gt;Approximate&lt;/em&gt; prediction limit methods for gamma distributions have been worked out, as published by Krishnamoorthy, Mathew, and Mukherjee in a &lt;a href=&quot;http://www.ucs.louisiana.edu/~kxk4695/GammaR2.pdf&quot; rel=&quot;nofollow&quot;&gt;2008 Technometrics article&lt;/a&gt;.  The calculations are relatively simple.  I won't discuss them here because there are more important issues to attend to first.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Before applying any parametric procedure you should check that the data at least approximately conform to the procedure's assumptions.  In this case we can check whether the data look exponential (or geometric) by making an &lt;a href=&quot;http://www.itl.nist.gov/div898/handbook/eda/section3/probplot.htm&quot; rel=&quot;nofollow&quot;&gt;exponential probability plot&lt;/a&gt;.  This procedure matches the sorted data values $k_1, k_2, \ldots, k_7$ = $22, 24, 36, 44, 74, 89, 100$ to percentage points of (any) exponential distribution, which can be computed as the negative logarithms of $1 - (1 - 1/2)/7, 1 - (2 - 1/2)/7, \ldots, 1 - (7 - 1/2)/7$.  When I do that the plot looks decidedly curved, suggesting that these data are &lt;em&gt;not&lt;/em&gt; drawn from an exponential (or geometric) distribution.  With either of those distributions you should see a cluster of shorter failure times and a straggling tail of longer failure times.  Here, the initial clustering is apparent at $22, 24, 26, 44$, but after a relatively long gap from $44$ to $74$ there is another cluster at $74, 89, 100$.  &lt;em&gt;This should cause us to mistrust the results of our parametric models.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One approach in this situation is to use a &lt;a href=&quot;http://en.wikipedia.org/wiki/Censoring_%28statistics%29&quot; rel=&quot;nofollow&quot;&gt;nonparametric prediction limit&lt;/a&gt;.  That's a dead simple procedure in this case: if the post-fix value is the largest of all the values, that should be evidence that the fix actually lengthened the failure times.  If all &lt;em&gt;eight&lt;/em&gt; values (the seven pre-fix data and the one post-fix value) come from the same distribution and are independent, there is only a $1/8$ chance that the eighth value will be the largest.  Therefore, we can say with $1 - 1/8 = 87.5$% confidence that the fix has improved the failure times.  This procedure also correctly handles the &lt;em&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Censoring_%28statistics%29&quot; rel=&quot;nofollow&quot;&gt;censoring&lt;/a&gt;&lt;/em&gt; in the last value, which really records a failure time of some unknown value greater than 233.  (If a parametric prediction limit happens to exceed 233--and I suspect [based on experience and on the result of @Owe Jessen's bootstrap] it would be close if we were to calculate it with 95% confidence--we would determine that the &lt;em&gt;number&lt;/em&gt; 233 is not inconsistent with the other data, but that would leave unanswered the question concerning the &lt;em&gt;true time to failure,&lt;/em&gt; for which 233 is only an underestimate.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Based on @csgillespie's calculations, which--as I argued above--likely overestimate the confidence as $98.3$%, we nevertheless have found a window in which the actual confidence is likely to lie: it's at least $87.5$% and somewhat less than $98.3$% (assuming we have any faith in the geometric distribution model).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;I will conclude by sharing my greatest concern&lt;/strong&gt;: the question as stated could easily be misinterpreted as an appeal to use statistics to make an impression or sanctify a conclusion, rather than provide genuinely useful information about uncertainty.  If there are additional reasons to suppose that the fix has worked, then the best course is to invoke them and don't bother with statistics.  Make the case on its technical merits.  If, on the other hand, there is little assurance that the fix was effective--we just don't know for sure--and the objective here is to decide whether the data warrant proceeding &lt;em&gt;as if it did work,&lt;/em&gt; then a prudent decision maker will likely prefer the conservative confidence level afforded by the non-parametric procedure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For (hypothetical) data {22, 24, 36, 44, 15, 20, 23} the exponential probability plot is not terrifically non-linear:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/qJA2L.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(If this looks non-linear to you, generate probability plots for a few hundred realizations of seven draws from an Exponential[25] distribution to see how much they will wiggle by chance alone.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore with this modified dataset you can feel more comfortable using the equations in &lt;a href=&quot;http://www.ucs.louisiana.edu/~kxk4695/GammaR2.pdf&quot; rel=&quot;nofollow&quot;&gt;Krishnamoorthy &lt;em&gt;et al.&lt;/em&gt;&lt;/a&gt; (&lt;em&gt;op. cit.&lt;/em&gt;) to compute a prediction limit.  However, the harmonic mean of 25.08 and relatively small SD (around 10) indicate the prediction limit for any typical confidence level (&lt;em&gt;e.g.&lt;/em&gt;, 95% or 99%) will be much less than 223.  The principle in play here is that one uses statistics for insight and to make difficult decisions.  Statistical procedures are of little (additional) help when the results are obvious.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-11-04T04:13:33.710" LastActivityDate="2010-11-04T04:13:33.710" CommentCount="4" />
  <row Id="4181" PostTypeId="2" ParentId="4138" CreationDate="2010-11-03T16:51:13.893" Score="5" ViewCount="" Body="&lt;p&gt;The question reads to me like the OP was asking when $U = (X,Y,Z)^{\mathrm{T}}$ are jointly normal then what is the probability $P(X \geq  Y \mbox{ and } X \geq Z)$?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For that question we could look at the joint distribution of $AU$ where $A$ looks like &#xA;$$&#xA;A=\left[&#xA;\begin{array}{ccc}&#xA;1 &amp;amp; -1 &amp;amp; 0 \newline&#xA;1 &amp;amp; 0 &amp;amp; -1&#xA;\end{array}\right]&#xA;$$&#xA;Of course, $AU$ is also jointly normal with mean $A\mu$ and variance-covariance $A\Sigma A^{\mathrm{T}}$, and the desired probability is $P(AU &gt; \mathbf{0}_{n-1})$.  We could get this in R with something like&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(1)&#xA;&#xA;Mu &amp;lt;- c(1,2,3)&#xA;&#xA;library(MCMCpack)&#xA;S &amp;lt;- rwish(3, diag(3))  # get var-cov matrix&#xA;&#xA;A &amp;lt;- matrix(c(1,-1,0, 1,0,-1), nrow = 2, byrow = TRUE)&#xA;&#xA;newMu &amp;lt;- as.vector(A %*% Mu)&#xA;newS &amp;lt;- A %*% S %*% t(A)&#xA;&#xA;library(mvtnorm)&#xA;pmvnorm(lower=c(0,0), mean = newMu, sigma = newS)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which is about 0.1446487 on my system.  If a person knew something about the matrix $\Sigma$ then (s)he might even be able to write something down that looks like a formula (I haven't tried, though).&lt;/p&gt;&#xA;" OwnerUserId="1108" LastEditorUserId="1108" LastEditorDisplayName="" LastEditDate="2010-11-03T19:22:08.373" LastActivityDate="2010-11-03T19:22:08.373" CommentCount="2" />
  <row Id="4182" PostTypeId="2" ParentId="4174" CreationDate="2010-11-03T18:48:02.257" Score="1" ViewCount="" Body="&lt;p&gt;I faced this problem myself and decided to try Fisher's exact test. This has the advantage that the arithmetic boils down to something you can do with JavaScript. I put this on a web page - &lt;a href=&quot;http://www.mcdowella.demon.co.uk/FlakyPrograms.html&quot; rel=&quot;nofollow&quot;&gt;http://www.mcdowella.demon.co.uk/FlakyPrograms.html&lt;/a&gt; - this should work either from there or if you download it to your computer (which you are welcome to do). I think you have a total of 382 successes and 7 failures in the old version, and 223 successes and 0 failures in the new one, and that you could get this at random with probability about 4% even if the new version was no better.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suggest that you run it a bit more. You can play about with the web page to see how the probability changes if you survive longer - I would go for something over 1000 - in fact I'd try hard to turn it into something I could run automatically and then let it overnight to really blitz the problem.&lt;/p&gt;&#xA;" OwnerUserId="1789" LastActivityDate="2010-11-03T18:48:02.257" CommentCount="4" />
  <row Id="4183" PostTypeId="2" ParentId="539" CreationDate="2010-11-03T20:14:59.663" Score="7" ViewCount="" Body="&lt;p&gt;I will assume that a &quot;categorical&quot; variable actually stands for an ordinal variable, otherwise it doesn't make much sense to treat it as a continuous one, unless it's a binary variable (coded 0/1) as pointed by @Rob. Then, I would say that the problem is not that much the way we treat the variable, although many models for categorical data analysis have been developed so far--see e.g., &lt;a href=&quot;http://petra.euitio.uniovi.es/~i1770184/Archivos/t141/Test_agresti.pdf&quot; rel=&quot;nofollow&quot;&gt;The analysis of ordered categorical data: An overview and a survey of recent developments&lt;/a&gt; from Liu and Agresti--, than the underlying measurement scale we assume. My response will focus on this second point, although I will first briefly discuss the assignment of numerical scores to variable categories or levels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By using a simple numerical recoding of an ordinal variable, you are assuming that the variable has interval properties (in the sense of the classification given by Stevens, 1946). From a measurement theory perspective (in psychology), this may often be a too strong assumption, but for basic study (i.e. where a single item is used to express one's opinion about a daily activity with clear wording) any monotone scores should give comparable results. Cochran (1954) already pointed that&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;any set of scores gives a &lt;em&gt;valid&lt;/em&gt;&#xA;  test, provided that they are&#xA;  constructed without consulting the&#xA;  results of the experiment. If the set&#xA;  of scores is poor, in that it badly&#xA;  distorts a numerical scale that really&#xA;  does underlie the ordered&#xA;  classification, the test will not be&#xA;  sensitive. The scores should therefore&#xA;  embody the best insight available&#xA;  about the way in which the&#xA;  classification was constructed and&#xA;  used.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;(Many thanks to @whuber for reminding me about this throughout one of his comments, which lead me to re-read Agresti's book from which this citation comes from)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Actually, several tests treat implicitly such variables as interval scales; for example, the $M^2$ statistic for testing a linear trend (as an alternative to simple independence) is based on a correlational approach ($M^2=(n-1)r^2$, Agresti, 2002, p. 87). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Well, you can also decide to recode your variable on an irregular range, or aggregate some of its levels, but in this case strong imbalance between recoded categories may distort statistical tests, e.g. the aforementioned trend test. &#xA;A nice alternative for assigning distance between categories was already proposed by @Jeromy, namely optimal scaling. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, let's discuss the second point I made, that of the underlying measurement model. I'm always hesitating about adding the &quot;psychometrics&quot; tag when I see this kind of question, because the construction and analysis of measurement scales come under Psychometric Theory (Nunnally and Bernstein, 1994, for a neat overview). I will not dwell on all the models that are actually headed under the &lt;a href=&quot;http://en.wikipedia.org/wiki/Item_response_theory&quot; rel=&quot;nofollow&quot;&gt;Item Response Theory&lt;/a&gt;, and I kindly refer the interested reader to I. Partchev's tutorial, &lt;a href=&quot;http://www.metheval.uni-jena.de/irt/VisualIRT.pdf&quot; rel=&quot;nofollow&quot;&gt;A visual guide to item response theory&lt;/a&gt;, for a gentle introduction to IRT, and to references (5-8) listed at the end for possible IRT taxonomies. Very briefly, the idea is that rather than assigning arbitrary distances between variable categories, you assume a latent scale and estimate their location on that continuum, together with individuals ability or liability. A simple example is worth several mathematical notation, so let's consider the following item (coming from the &lt;a href=&quot;http://groups.eortc.be/qol/questionnaires_qlqc30.htm&quot; rel=&quot;nofollow&quot;&gt;EORTC QLQ-C30&lt;/a&gt; health-related quality of life questionnaire):&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Did you worry?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;which is coded on a four-point scale, ranging from &quot;Not at all&quot; to &quot;Very much&quot;. Raw scores are computed by assigning a score of 1 to 4. Scores on items belonging to the same scale can then be added together to yield a so-called scale score, which denotes one's &lt;em&gt;rank&lt;/em&gt; on the underlying construct (here, a mental health component). Such summated scale scores are very practical because of scoring easiness (for the practician or nurse), but they are nothing more than a discrete (ordered) scale. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can also consider that the probability of endorsing a given response category obeys some kind of a logistic model, as described in I. Partchev's tutorial, referred above. Basically, the idea is that of a kind of threshold model (which lead to equivalent formulation in terms of the proportional or cumulative odds models) and we model the odds of being in one response category rather the preceding one or the odds of scoring above a certain category, conditional on subjects' location on the latent trait. In addition, we may impose that response categories are equally spaced on the latent scale (this is the Rating Scale model)--which is the way we do by assigning regularly spaced numerical scores-- or not (this is the Partial Credit model). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Clearly, we are not adding very much to Classical Test Theory, where ordinal variable are treated as numerical ones. However, we introduce a probabilistic model, where we assume a continuous scale (with interval properties) and where specific errors of measurement can be accounted for, and we can plug these factorial scores in any regression model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;References&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;S S Stevens. On the theory of scales of measurement. &lt;em&gt;Science&lt;/em&gt;, &lt;strong&gt;103&lt;/strong&gt;: 677-680, 1946.&lt;/li&gt;&#xA;&lt;li&gt;W G Cochran. Some methods of strengthening the common $\chi^2$ tests. &lt;em&gt;Biometrics&lt;/em&gt;, &lt;strong&gt;10&lt;/strong&gt;: 417-451, 1954.&lt;/li&gt;&#xA;&lt;li&gt;J Nunnally and I Bernstein. &lt;em&gt;Psychometric Theory&lt;/em&gt;. McGraw-Hill, 1994&lt;/li&gt;&#xA;&lt;li&gt;Alan Agresti. &lt;em&gt;Categorical Data Analysis&lt;/em&gt;. Wiley, 1990.&lt;/li&gt;&#xA;&lt;li&gt;C R Rao and S Sinharay, editors. &lt;em&gt;Handbook of Statistics, Vol. 26: Psychometrics&lt;/em&gt;. Elsevier Science B.V., The Netherlands, 2007.&lt;/li&gt;&#xA;&lt;li&gt;A Boomsma, M A J van Duijn, and T A B Snijders. &lt;em&gt;Essays on Item Response Theory&lt;/em&gt;. Springer, 2001.&lt;/li&gt;&#xA;&lt;li&gt;D Thissen and L Steinberg. A taxonomy of item response models. &lt;em&gt;Psychometrika&lt;/em&gt;, &lt;strong&gt;51(4)&lt;/strong&gt;: 567–577, 1986.&lt;/li&gt;&#xA;&lt;li&gt;P Mair and R Hatzinger. &lt;a href=&quot;http://www.jstatsoft.org/v20/i09&quot; rel=&quot;nofollow&quot;&gt;Extended Rasch Modeling: The eRm Package for the Application of IRT Models in R&lt;/a&gt;. &lt;em&gt;Journal of Statistical Software&lt;/em&gt;, &lt;strong&gt;20(9)&lt;/strong&gt;, 2007.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-04T07:29:48.097" LastActivityDate="2010-11-04T07:29:48.097" />
  <row Id="4184" PostTypeId="1" AcceptedAnswerId="4869" CreationDate="2010-11-04T00:01:32.990" Score="1" ViewCount="81" Body="&lt;p&gt;[Cross post from &lt;a href=&quot;http://math.stackexchange.com/q/8830/2949&quot;&gt;here&lt;/a&gt;, figured this community may be more relevant]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am working in the field of machine learning, and I have come across a few papers that show relationships between Gröbner bases and discrete probability. So I come here for help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can you please explain how can Gröbner bases used to describe discrete probability?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have looked at Gröbner bases and I understand the general concepts (and used Maple to calculate a few examples). So it is the link that is missing for me.&lt;/p&gt;&#xA;" OwnerUserId="1793" LastActivityDate="2010-11-24T08:52:57.620" Title="How can Gröbner bases used to describe discrete probability?" Tags="&lt;probability&gt;" AnswerCount="1" />
  <row Id="4185" PostTypeId="1" AcceptedAnswerId="4186" CreationDate="2010-11-04T00:34:49.023" Score="1" ViewCount="444" Body="&lt;p&gt;Hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://uk.answers.yahoo.com/question/index?qid=20080702080708AAsRZpt&quot; rel=&quot;nofollow&quot;&gt;Here&lt;/a&gt; is a link that describes the formula to find the mode of grouped data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.tutors4you.com/modegraphically.htm&quot; rel=&quot;nofollow&quot;&gt;Here&lt;/a&gt; is a link that gives a graphical method to finding the mode of grouped data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Question: Can someone please explain how the formula corresponds with the graphical method? The formula is an interpolation but I am not able to see how it expresses the idea of the graphical method. If it helps, I understand the formula for calculating the mean of grouped data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you.&lt;/p&gt;&#xA;" OwnerUserId="1636" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-11-04T15:30:42.843" LastActivityDate="2010-11-04T15:30:42.843" Title="The formula for finding the mode of grouped data" Tags="&lt;summary-statistics&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="4186" PostTypeId="2" ParentId="4185" CreationDate="2010-11-04T00:44:18.200" Score="2" ViewCount="" Body="&lt;p&gt;I have not bothered to check the math at the link you gave but from the figure the mode is the intersection of the two diagonal lines. The end points of both the lines is known so all you need to do is to find out the intersection of those two lines to get the mode.&lt;/p&gt;&#xA;" LastActivityDate="2010-11-04T00:44:18.200" CommentCount="3" />
  <row Id="4187" PostTypeId="1" CreationDate="2010-11-04T03:08:14.567" Score="5" ViewCount="785" Body="&lt;p&gt;Hello&#xA;I have two problems that sound like natural candidates for multilevel/mixed models, which I have never used.  The simpler, and one that I hope to try as an introduction, is as follows:&#xA;The data looks like many rows of the form &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;x y innergroup outergroup&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where x is a numeric covariate upon which I want to regress y (another numeric variable), each y belongs to an innergroup, and each innergroup is nested in an outergroup (i.e, all the y in a given innergroup belong to the same outergroup).  Unfortunately, innergroup has a lot of levels (many thousands), and each level has relatively few observations of y, so I thought this sort of model might be appropriate.  My questions are&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;How do I write this sort of multilevel formula?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Once &lt;strong&gt;lmer&lt;/strong&gt; fits the model, how does one go about predicting from it?  I have fit some simpler toy examples, but have not found a predict() function.  Most people seem more interested in inference than prediction with this sort of technique.&#xA;I have several million rows, so the computations might be an issue, but I can always cut it down as appropriate.  &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I won't need to do the second for some time, but I might as well begin thinking about it and playing around with it.  I have similar data as before, but without x, and y is now a binomial variable of the form $(n,n-k)$.  y also exhibits a lot of overdispersion, even within innergroups.  Most of the $n$ are no more than 2 or 3 (or less), so to derive estimates of the success rates of each $y_i$ I have been using the beta-binomial shrinkage estimator $(\alpha+k_i)/(\alpha+\beta+n_i)$, where $\alpha$ and $\beta$ are estimated by MLE for each innergroup separately.  This is has been somewhat adequate, but data sparsity still plagues me, so I would like to use all the data available.  From one perspective, this problem is easier since there is no covariate, but from the other the binomial nature makes it more difficult.  Does anyone have any high (or low!) level guidance?&lt;/p&gt;&#xA;" OwnerUserId="1777" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-04T11:47:23.530" LastActivityDate="2010-11-12T20:09:03.653" Title="Using lmer for prediction" Tags="&lt;r&gt;&lt;mixed-model&gt;&lt;maximum-likelihood&gt;&lt;glm&gt;" AnswerCount="4" CommentCount="2" FavoriteCount="1" />
  <row Id="4191" PostTypeId="1" AcceptedAnswerId="4194" CreationDate="2010-11-04T04:35:38.390" Score="4" ViewCount="377" Body="&lt;p&gt;So MCMC algorithms like metropolis hastings and gibbs sampling are ways of sampling from the joint posterior distributions.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think I understand and can implement metropolis-hasting pretty easily-- you simply choose starting points somehow, and 'walk the parameter space' randomly, guided by the posterior density and proposal density.  Gibbs sampling seems very similar but more efficient since it updates only one parameter at a time, while holding the others constant, effectively walking the space in an orthogonal fashion. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to do this, you need the full conditional of each parameter in analytical form*.  But where do these full conditionals come from? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;P(x1 | x2,...,xn) = P(x1,...,xn) / P(x2, ..., xn), and to get the demoninator you need to marginalize the joint over x1. That seems like a whole lot of work to do analytically if there are many parameter, and might not be tractable unless the joint dist. isn't very 'nice'.  I realize that if you use conjugacy throughout the model, the full conditionals may be easy, but there's got to be a better way for more general situations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All examples of gibbs sampling I've seen online use toy examples (like sampling from a multivariate normal, where the conditionals are just normals themselves), and seem to dodge this issue.  &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Or do you need the full conditionals in analytical form at all? How do programs like winbugs do it?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="1795" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-11-04T10:32:49.003" LastActivityDate="2010-11-05T02:03:58.703" Title="Gibbs sampling-- where do the full conditionals come from?" Tags="&lt;bayesian&gt;&lt;mcmc&gt;&lt;gibbs&gt;" AnswerCount="2" FavoriteCount="2" />
  <row Id="4193" PostTypeId="1" CreationDate="2010-11-04T05:49:02.013" Score="7" ViewCount="173" Body="&lt;p&gt;Most asymptotic results in statistics prove that as $n \rightarrow \infty$ an estimator (such as the MLE) converges to a normal distribution based on a second-order taylor expansion of the likelihood function. I believe there's a similar result in Bayesian literature, the &quot;Bayesian Central Limit Theorem&quot;, which shows that the posterior converges asymptotically to a normal as $n \rightarrow \infty$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is - does the distribution converge to something &quot;before&quot; it becomes normal, based on the third term in the Taylor series? Or is this not possible to do in general?&lt;/p&gt;&#xA;" OwnerUserId="1760" LastEditorUserId="223" LastEditorDisplayName="" LastEditDate="2010-12-17T07:49:54.047" LastActivityDate="2010-12-17T07:49:54.047" Title="Do third order asymptotics exist?" Tags="&lt;mathematical-statistics&gt;&lt;asymptotics&gt;" AnswerCount="4" CommentCount="1" FavoriteCount="1" />
  <row Id="4194" PostTypeId="2" ParentId="4191" CreationDate="2010-11-04T05:57:44.330" Score="4" ViewCount="" Body="&lt;p&gt;Yes, you are right, the conditional distribution needs to be found analytically, but I think there are lots of examples where the full conditional distribution is easy to find, and has a far simpler form than the joint distribution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The intuition for this is as follows, in most &quot;realistic&quot; joint distributions $P(X_1,\dots,X_n)$, &lt;em&gt;most&lt;/em&gt; of the $X_i$'s are generally conditionally independent of most of the other random variables. That is to say, some of the variables have &lt;em&gt;local&lt;/em&gt; interactions, say $X_i$ depends on $X_{i-1}$ and $X_{i+1}$, but doesn't interact with everything, hence the conditional distributions should simplify significantly as $Pr(X_i|X_1, \dots, X_i) = Pr(X_i|X_{i-1}, X_{i+1})$&lt;/p&gt;&#xA;" OwnerUserId="1760" LastActivityDate="2010-11-04T05:57:44.330" CommentCount="3" />
  <row Id="4196" PostTypeId="1" CreationDate="2010-11-04T06:34:24.897" Score="1" ViewCount="130" Body="&lt;p&gt;Hello,&#xA;I have a time series X(t). Each X(t) has three possible outcomes : A,B,C. I am interested in the ratio of A,B,C to the total. Assuming N is the total number of data points I have gathered for X(t),&lt;br&gt;&#xA;&lt;em&gt;How can I compute the confidence levels for A/N, B/N and C/N when the X(t) are &quot;intuitively&quot; not independent ?&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For exemple : X(t) is an indication of whether a car is : moving (speed&gt;0), stopped (speed=0) or its motor is off. The data I gathered is a time slice pertaining to a car. To me, those categories are not independent because when the car is moving at X(t) is is very likely to be still moving at X(t). [Am I correct?]&lt;/p&gt;&#xA;" OwnerUserId="1784" LastEditorUserId="1784" LastEditorDisplayName="" LastEditDate="2010-11-04T21:15:43.877" LastActivityDate="2010-11-04T21:15:43.877" Title="Confidence interval for ratio in timeseries" Tags="&lt;time-series&gt;&lt;confidence-interval&gt;&lt;non-independent&gt;" CommentCount="4" />
  <row Id="4197" PostTypeId="2" ParentId="4193" CreationDate="2010-11-04T06:37:44.763" Score="3" ViewCount="" Body="&lt;p&gt;Here is an attempt to answer your insightful question. I have seen the inclusion of the 3rd term of the Taylor series to increase the speed of convergence of the series to the true distribution. However, I haven't seen (in my limited experience) the usage of third and higher moments. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As pointed out by John D. Cook in his blogs (&lt;a href=&quot;http://www.johndcook.com/blog/2010/09/20/skewness-andkurtosis/&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;http://www.johndcook.com/blog/2008/09/30/quantifying-the-error-in-the-central-limit-theorem/&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;), there hasn't been much work done in this direction, apart from the &lt;a href=&quot;http://en.wikipedia.org/wiki/Berry%E2%80%93Ess%C3%A9en_theorem&quot; rel=&quot;nofollow&quot;&gt;Berry-Esseen theorem&lt;/a&gt;. My guess would be (from the observation in the blog about the approximation error being bounded by $n^{1/2}$), as the asymptotic normality of mle is guaranteed at a rate of converge of $n^{1/2}$ ($n$, being sample size), considering higher moments won't improve on the normality result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, I guess, the answer to your question should be &lt;em&gt;no&lt;/em&gt;. The asymptotic distribution converges to a normal dist.(by CLT, under regularity conditions of Lindberg's CLT). However, using higher order terms may increase the rate of convergence to the asymptotic distribution. &lt;/p&gt;&#xA;" OwnerUserId="1307" LastActivityDate="2010-11-04T06:37:44.763" />
  <row Id="4199" PostTypeId="2" ParentId="4193" CreationDate="2010-11-04T08:09:42.603" Score="2" ViewCount="" Body="&lt;p&gt;Definitely not my area, but I'm pretty sure third- and higher-order asymptotics exist. Is this any help?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Robert L. Strawderman. &lt;a href=&quot;http://www.jstor.org/stable/2669788&quot; rel=&quot;nofollow&quot;&gt;Higher-Order Asymptotic Approximation: Laplace, Saddlepoint, and Related Methods&lt;/a&gt; &lt;em&gt;Journal of the American Statistical Association&lt;/em&gt; Vol. 95, No. 452 (Dec., 2000), pp. 1358-1364 &lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-11-04T08:09:42.603" />
  <row Id="4200" PostTypeId="1" CreationDate="2010-11-04T09:02:43.647" Score="1" ViewCount="349" Body="&lt;p&gt;I have a large number (hundreds to thousands) of noisy time series that represent contemporaneous observations from different subjects.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hypothesise that there exist lead-lag relationships between observations for different subjects (or groups of subjects.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to explore the potential use of such lead-lag relationships for the purposes of predicting the individual series.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What methods might I consider for this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;edit: To be clear, I am not looking at pairwise relationships. What I am looking for is a method that would look at the mountain of data at hand and attempt to discover (potentially non-linear) lead-lag relationships between &lt;em&gt;arbitrary&lt;/em&gt; groups of series and the individual series to be predicted.&lt;/p&gt;&#xA;" OwnerUserId="439" LastEditorUserId="439" LastEditorDisplayName="" LastEditDate="2010-11-08T07:26:18.083" LastActivityDate="2011-02-23T20:19:48.650" Title="Using lead-lag relationships for time series prediction" Tags="&lt;time-series&gt;" AnswerCount="3" FavoriteCount="1" />
  <row Id="4201" PostTypeId="2" ParentId="2635" CreationDate="2010-11-04T10:12:31.840" Score="1" ViewCount="" Body="&lt;p&gt;I believe M. Tibbit's answer refers to the general case of a gamma with unknown shape and scale. If the shape α is known and the sampling distribution for x is gamma(α, β) and the prior distribution on β is gamma(α0, β0), the posterior distribution for β is gamma(α0 + n, β0 + Σxi). See this &lt;a href=&quot;http://www.johndcook.com/conjugate_prior_diagram.html&quot; rel=&quot;nofollow&quot;&gt;diagram&lt;/a&gt; and the references at the bottom.&lt;/p&gt;&#xA;" OwnerUserId="319" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-11-04T10:14:48.267" LastActivityDate="2010-11-04T10:14:48.267" />
  <row Id="4202" PostTypeId="2" ParentId="4187" CreationDate="2010-11-04T11:28:13.980" Score="6" ViewCount="" Body="&lt;p&gt;Expressing factors relationships using R formulas follows from Wilkinson's notation, where '*' denotes crossing and '/' nesting, but there are some particularities in the way formula for mixed-effects models, or more generally random effects, are handled. For example, two crossed random effects might be represented as &lt;code&gt;(1|x1)+(1|x2)&lt;/code&gt;. I have interpreted your description as a case of nesting, much like classes are nested in schools (nested in states, etc.), so a basic formula with &lt;code&gt;lmer&lt;/code&gt; would look like (unless otherwise stated, a &lt;code&gt;gaussian&lt;/code&gt; family is used by default):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;y ~ x + (1|A:B) + (1|A)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where A and B correspond to your inner and outer factors, respectively. A is nested in B, and both are treated as random factors. In the older &lt;a href=&quot;http://cran.r-project.org/web/packages/nlme/index.html&quot; rel=&quot;nofollow&quot;&gt;nlme&lt;/a&gt; package, this would correspond to something like &lt;code&gt;lme(y ~ x, random=~ 1 | A/B)&lt;/code&gt;. If A was to be considered as a fixed factor, the formula should read &lt;code&gt;y ~ x + A + (1|A:B)&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But it is worth checking more precisely D. Bates' specifications for the &lt;a href=&quot;http://cran.r-project.org/web/packages/lme4/index.html&quot; rel=&quot;nofollow&quot;&gt;lme4&lt;/a&gt; package, e.g. in his forthcoming textbook, &lt;a href=&quot;http://lme4.r-forge.r-project.org/&quot; rel=&quot;nofollow&quot;&gt;lme4: Mixed-effects Modeling with R&lt;/a&gt;, or the many handouts available on the same webpage. In particular, there is an example for such nesting relations in &lt;a href=&quot;http://www.stat.wisc.edu/~bates/PotsdamGLMM/LMMD.pdf?bcsi_scan_CBA24F92DB3F63E2=0&amp;amp;bcsi_scan_filename=LMMD.pdf&quot; rel=&quot;nofollow&quot;&gt;Fitting Linear Mixed-Effects Models, the lme4 Package in R&lt;/a&gt;. John Maindonald's tutorial also provides a nice overview: &lt;a href=&quot;http://www.maths.anu.edu.au/~johnm/r-book/xtras/mlm-ohp.pdf&quot; rel=&quot;nofollow&quot;&gt;The Anatomy of a Mixed Model Analysis, with R’s lme4 Package&lt;/a&gt;. Finally, section 3 of the R vignette on &lt;a href=&quot;http://cran.r-project.org/web/packages/lme4/vignettes/Implementation.pdf&quot; rel=&quot;nofollow&quot;&gt;lme4 imlementation&lt;/a&gt; includes an example of the analysis of a nested structure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is no &lt;code&gt;predict()&lt;/code&gt; function in &lt;a href=&quot;http://cran.r-project.org/web/packages/lme4/index.html&quot; rel=&quot;nofollow&quot;&gt;lme4&lt;/a&gt;, and you have to compute yourself predicted individual values using the estimated fixed (see &lt;code&gt;?fixeff&lt;/code&gt;) and random (see &lt;code&gt;?raneff&lt;/code&gt;) effects, but see also this thread on the &lt;a href=&quot;http://r.789695.n4.nabble.com/no-predict-function-in-lme4-td1679131.html&quot; rel=&quot;nofollow&quot;&gt;lack of predict function in lme4&lt;/a&gt;. Still, you can generate a sample from the posterior distribution using the &lt;code&gt;mcmcsamp()&lt;/code&gt; function. Sometimes, it might clash, though. See the &lt;a href=&quot;https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models&quot; rel=&quot;nofollow&quot;&gt;sig-me&lt;/a&gt; mailing list for more updated information.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-04T22:51:14.920" LastActivityDate="2010-11-04T22:51:14.920" />
  <row Id="4203" PostTypeId="1" AcceptedAnswerId="4206" CreationDate="2010-11-04T11:33:05.190" Score="1" ViewCount="120" Body="&lt;p&gt;Suppose I have a set of $N$ experimental points of the form &lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{equation}&#xA;\{x_i, y_i, d_i\},&#xA;\end{equation}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $i=1,...,N,$ and $d_i$ are errorbars for $y_i$. To fit the data, I minimize the reduced chi-square&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{equation}&#xA;\chi^2(p) = \sum_{i=1}^N \frac{[y_i - f(x_i,p)]^2}{d_i^2},&#xA;\end{equation}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $f(x,p)$ is a (generally non-linear) function parametrized by some parameter $p$ (there might be more than one parameter, but it doesn't really matter). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is: given the optimal parameter $p_0$, i.e. $\chi^2(p)$ is minimal at $p=p_0$, and assuming the $y_i$'s are independent and are Normally distributed, what can be said about the distribution of $f(x, p0)$?&lt;/p&gt;&#xA;" OwnerUserId="1197" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-04T13:28:31.173" LastActivityDate="2010-11-04T13:28:31.173" Title="What is the distribution of a chi-square minimizing function?" Tags="&lt;distributions&gt;&lt;chi-squared&gt;&lt;fitting&gt;" AnswerCount="1" FavoriteCount="0" />
  <row Id="4204" PostTypeId="2" ParentId="4187" CreationDate="2010-11-04T12:36:04.560" Score="5" ViewCount="" Body="&lt;p&gt;The &lt;a href=&quot;http://cran.r-project.org/package=ez&quot; rel=&quot;nofollow&quot;&gt;ez&lt;/a&gt; package contains the ezPredict() function, which obtains predictions from lmer models where prediction is based on the fixed effects only. It's really just a wrapper around the approach detailed in the &lt;a href=&quot;http://glmm.wikidot.com/faq&quot; rel=&quot;nofollow&quot;&gt;glmm wiki&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="364" LastActivityDate="2010-11-04T12:36:04.560" />
  <row Id="4205" PostTypeId="2" ParentId="4200" CreationDate="2010-11-04T13:00:44.517" Score="4" ViewCount="" Body="&lt;p&gt;You can choose from about 40 years of research and countless books, dissertations, monographs etc.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given that your question is not all that focussed yet, maybe an introductory time-series book could help.  In a nutshell, the autocorrelation function gives clues to lead/lag relationships that may be present in a single time-series, or between two series.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rob has done a lot of research into sensibly automating the process of identifying how many / which leads/lags to use, so please look at his &lt;a href=&quot;http://cran.r-project.org/package=forecast&quot; rel=&quot;nofollow&quot;&gt;forecast&lt;/a&gt; package for R and &lt;a href=&quot;http://robjhyndman.com/&quot; rel=&quot;nofollow&quot;&gt;other research&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="334" LastActivityDate="2010-11-04T13:00:44.517" />
  <row Id="4206" PostTypeId="2" ParentId="4203" CreationDate="2010-11-04T13:13:23.710" Score="1" ViewCount="" Body="&lt;p&gt;Another term for your fitting procedure would be weighted &lt;a href=&quot;http://en.wikipedia.org/wiki/Non-linear_least_squares&quot; rel=&quot;nofollow&quot;&gt;non-linear least squares&lt;/a&gt;. The weights are a very minor complication. Fitting non-linear least squares is more tricky than &lt;a href=&quot;http://en.wikipedia.org/wiki/Ordinary_least_squares&quot; rel=&quot;nofollow&quot;&gt;ordinary least squares&lt;/a&gt;, but once the fitting is done the asymptotic ($N \to \infty$) distribution of the estimates is given by the same &lt;a href=&quot;http://en.wikipedia.org/wiki/Linear_least_squares#Large_sample_properties&quot; rel=&quot;nofollow&quot;&gt;large-sample theory&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-11-04T13:13:23.710" CommentCount="5" />
  <row Id="4209" PostTypeId="2" ParentId="4193" CreationDate="2010-11-04T14:46:20.700" Score="2" ViewCount="" Body="&lt;p&gt;It is not possible for a sequence to &quot;converge&quot; to one thing and then to another.  The higher-order terms in an asymptotic expansion will go to zero.  What they tell you is &lt;em&gt;how close to zero&lt;/em&gt; they are for any given value of $n$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the Central Limit Theorem (as an example) the appropriate expansion is that of the logarithm of the characteristic function: the cumulant generating function (cgf).  Standardization of the distributions fixes the zeroth, first, and second terms of the cgf.  The remaining terms, whose coefficients are the &lt;em&gt;cumulants&lt;/em&gt;, depend on $n$ in an orderly way.  The standardization that occurs in the CLT (dividing the sum of $n$ random variables by something proportional to $n^{1/2}$--without which convergence will not occur) causes the $m^\text{th}$ cumulant--which after all depends on $m^\text{th}$ moments--to be divided by $(n^{1/2})^m = n^{m/2}$, but at the same time because we are summing $n$ terms, the net result is that the $m^\text{th}$ order term is proportional to $n/n^{m/2} = n^{-(m-2)/2}$.  Thus the third cumulant of the standardized sum is proportional to $1/n^{1/2}$, the fourth cumulant is proportional to $1/n$, and so on.  These are the higher-order terms.  (For details, see &lt;a href=&quot;http://www.cs.toronto.edu/~yuvalf/CLT.pdf&quot; rel=&quot;nofollow&quot;&gt;this paper of Yuval Filmus&lt;/a&gt; for example.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, a high negative power of $n$ is much smaller than a low negative power. &#xA;We can always be assured of this by taking a sufficiently large value of $n$.  Thus, for really large $n$ we can neglect all negative powers of $n$: they converge to zero.  Along the way to convergence, &lt;em&gt;departures&lt;/em&gt; from the ultimate limit are measured with increasing accuracy by the additional terms: the $1/n^{1/2}$ term is an initial &quot;correction,&quot; or departure from the limiting value; the next $1/n$ term is a smaller, more quickly-vanishing correction added to that, and so on.  In brief, the additional terms give you a picture of how quickly the sequence converges to its limit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These additional terms can help us make &lt;em&gt;corrections&lt;/em&gt; for finite (usually small) values of $n$.  They show up all the time in this regard, such as &lt;a href=&quot;http://projecteuclid.org/DPubS/Repository/1.0/Disseminate?view=body&amp;amp;id=pdf_1&amp;amp;handle=euclid.aos/1176324314&quot; rel=&quot;nofollow&quot;&gt;Chen's modification of the t-test&lt;/a&gt;, which exploits the third-order ($1/n^{1/2}$) term.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-11-04T14:46:20.700" CommentCount="2" />
  <row Id="4210" PostTypeId="2" ParentId="4193" CreationDate="2010-11-04T15:24:41.853" Score="4" ViewCount="" Body="&lt;p&gt;You are searching for the Edgeworth series aren't you?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Edgeworth_series#Edgeworth_series&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Edgeworth_series#Edgeworth_series&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(note that Edgeworth died in 1926, should be in most famous statisticians?  )&lt;/p&gt;&#xA;" OwnerUserId="223" LastEditorUserId="223" LastEditorDisplayName="" LastEditDate="2010-12-17T07:49:30.333" LastActivityDate="2010-12-17T07:49:30.333" />
  <row Id="4211" PostTypeId="1" AcceptedAnswerId="4213" CreationDate="2010-11-04T16:11:48.220" Score="7" ViewCount="227" Body="&lt;p&gt;I have a data like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; table(A,B,C)&#xA;, , C = FALSE&#xA;&#xA;       B&#xA;A       FALSE TRUE&#xA;  FALSE   177   42&#xA;  TRUE      6    8&#xA;&#xA;, , C = TRUE&#xA;&#xA;       B&#xA;A       FALSE TRUE&#xA;  FALSE     5   31&#xA;  TRUE      4   10&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can I plot this on a single graph, possibly without imposing any hierarchy?&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-11-04T17:34:35.440" Title="How to visualize 3D contingency matrix?" Tags="&lt;data-visualization&gt;&lt;contingency-tables&gt;" AnswerCount="1" FavoriteCount="2" />
  <row Id="4212" PostTypeId="2" ParentId="2576" CreationDate="2010-11-04T16:30:27.683" Score="1" ViewCount="" Body="&lt;p&gt;You can rank ordinal distributions by means of an intuitive dominance criterion: the answers to one question are better than the answers to another when it is more likely than not that a randomly chosen answer to the first will be better than a randomly chosen answer to the second.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In more detail: put all the answers to question $X$ into one hat and all the answers to question $Y$ into another hat.  Draw one answer from each hat at random.  We will compare these answers, which we can do because they are on an ordinal scale.  Let's also agree to resolve any ties by flipping a fair coin.  Let $p(X,Y)$ be the probability that the answer to $X$ is better than the answer to $Y$. Rank $X$ ahead of $Y$ when $p$ exceeds $1/2$ and rank $X$ behind $Y$ when $p$ is less than $1/2$.  If $p$ equals $1/2$, declare a tie between $X$ and $Y$.  (By virtue of our tie-resolution procedure, $p(X,Y) + p(Y,X) = 1$, implying the ranking does not depend on the sequence in which we draw the two answers.)&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;The calculation is a simple exercise for &quot;just&quot; a programmer (and a fun one if you are interested in efficient calculation, although that's unlikely to matter here).  To make this proposal clear, though, I will illustrate it.  Suppose all answers are on an integral scale from one to four, with four best.  Write the answer distributions in the form $(k_1, k_2, k_3, k_4)$ where $k_3$ counts the number of &quot;3&quot;'s among the answers to a question, for example.  For this example suppose $X$ has distribution $(4, 2, 0, 4)$ and $Y$ has distribution $(1, 6, 1, 2)$ (ten answers each).  (Stop for a moment to consider which of these distributions ought to be considered &quot;best&quot; and note that they have identical means of 2.4 and identical medians of 2, suggesting this is a difficult comparison to make.)  Then:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;There is a 4/10 chance of drawing a &quot;4&quot; for $X$.  In this case,&#xA;&lt;ul&gt;&#xA;&lt;li&gt;There is a 2/10 chance of drawing a &quot;4&quot; for $Y$ for  a tie;&lt;/li&gt;&#xA;&lt;li&gt;There is an 8/10 chance of drawing less than &quot;4&quot; for $Y$, a win for $X$.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;This contributes $(4/10)[(2/10)0.5 + 8/10] = 0.36$ to $p(X,Y)$.  Continuing similarly,&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Drawing a &quot;3&quot; for $X$ is impossible; it contributes $0$ to $p(X,Y)$.&lt;/li&gt;&#xA;&lt;li&gt;Drawing a &quot;2&quot; for $X$ contributes $(2/10)[(6/10)0.5 + 1/10] = .08$.&lt;/li&gt;&#xA;&lt;li&gt;Drawing a &quot;1&quot; for $X$ contributes $(4/10)[(1/10)0.5] = 0.02$.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Whence $p(X,Y) = 0.36 + 0.00 + 0.08 + 0.02 = 0.46$.  Because this value is less than $1/2$, we conclude $X$ should be ranked &lt;em&gt;lower&lt;/em&gt; than $Y$.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;This idea is related to that of &lt;a href=&quot;http://www.jstor.org/pss/2290088&quot; rel=&quot;nofollow&quot;&gt;Pitman Closeness&lt;/a&gt; and to certain non-parametric &lt;a href=&quot;http://books.google.com/books?id=u97pzxRjaCQC&amp;amp;pg=PA384&amp;amp;lpg=PA384&amp;amp;dq=slippage+test+distribution+statistics&amp;amp;source=bl&amp;amp;ots=ixuCqitLQ5&amp;amp;sig=xs-rPHYaZhmplWDewk-cgdsnm7E&amp;amp;hl=en&amp;amp;ei=u9zSTMWwD4OBlAevs9DnDg&amp;amp;sa=X&amp;amp;oi=book_result&amp;amp;ct=result&amp;amp;resnum=10&amp;amp;ved=0CDwQ6AEwCQ#v=onepage&amp;amp;q=slippage%20test%20distribution%20statistics&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;slippage tests&lt;/a&gt; (which decide whether one distribution has &quot;slipped&quot;--changed values--with respect to other distributions based on random samples of them), such as the &lt;a href=&quot;http://en.wikipedia.org/wiki/Mann%E2%80%93Whitney_U&quot; rel=&quot;nofollow&quot;&gt;Mann-Whitney (aka Wilcoxon)&lt;/a&gt; test.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-11-04T20:10:51.363" LastActivityDate="2010-11-04T20:10:51.363" CommentCount="4" />
  <row Id="4213" PostTypeId="2" ParentId="4211" CreationDate="2010-11-04T16:33:27.503" Score="8" ViewCount="" Body="&lt;p&gt;I would try some kind of 3D heatmap, &lt;a href=&quot;http://www.datavis.ca/papers/drew/&quot; rel=&quot;nofollow&quot;&gt;mosaic plot&lt;/a&gt; or a &lt;a href=&quot;http://www.improving-visualisation.org/visuals/tag=%3asieve+plot/mode=1/sort=alpha&quot; rel=&quot;nofollow&quot;&gt;sieve plot&lt;/a&gt; (available in the &lt;a href=&quot;http://cran.r-project.org/web/packages/vcd/index.html&quot; rel=&quot;nofollow&quot;&gt;vcd&lt;/a&gt; package). Isn't the base &lt;code&gt;mosaicplot()&lt;/code&gt; function working with three-way table? (at least &lt;code&gt;mosaic3d()&lt;/code&gt; in the &lt;a href=&quot;http://cran.r-project.org/web/packages/vcdExtra/index.html&quot; rel=&quot;nofollow&quot;&gt;vcdExtra&lt;/a&gt; package should work, see e.g. &lt;a href=&quot;http://datavis.ca/R/&quot; rel=&quot;nofollow&quot;&gt;http://datavis.ca/R/&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's an example (including a conditional plot):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;A &amp;lt;- sample(c(T,F), 100, replace=T)&#xA;B &amp;lt;- sample(c(T,F), 100, replace=T)&#xA;C &amp;lt;- sample(c(T,F), 100, replace=T)&#xA;tab &amp;lt;- table(A,B,C)&#xA;library(vcd)&#xA;sieve(tab, shade=TRUE)&#xA;cotabplot(tab)&#xA;library(vcdExtra)&#xA;mosaic3d(tab, type=&quot;expected&quot;, box=TRUE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/MCbni.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/IwMp8.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/TtUpt.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Actually, the rendering of &lt;code&gt;mosaic3d()&lt;/code&gt; rely on the &lt;a href=&quot;http://cran.r-project.org/web/packages/rgl/index.html&quot; rel=&quot;nofollow&quot;&gt;rgl&lt;/a&gt; package, so it is hard to give a pretty picture of the result.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-04T17:34:35.440" LastActivityDate="2010-11-04T17:34:35.440" CommentCount="2" />
  <row Id="4214" PostTypeId="1" CreationDate="2010-11-04T16:49:31.543" Score="4" ViewCount="131" Body="&lt;p&gt;On both a practical and philosophical level, how should you choose the scope when performing multiple comparisons?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When a study performs 10 tests to check the hypothesis that 10 explanatory variable are predictive for &quot;something&quot; (on the same dataset), the test should obviously be corrected.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What if there where ten studies, each testing for one different explanatory variable  - when doing a meta analysis, should their P values be corrected?  (will knowing if these studies where done on the same dataset, or on different datasets make a difference?)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But then, what if we add to the mix another 100 researchers, all of them where just not very good at their jobs (all where testing &quot;junk&quot; variables) - automatically that will ruin our chances at finding anything after correction.  But is that a reflection of something already happening in real life science?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, let's assume the same researchers is doing a hundred studies, on different fields, asking one question in each of them.  Should he have corrected his P values from these 100 studies?  What if the questions are different but on the same study/dataset?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are criterions would you offer for choosing the scoping of performing multiple comparisons correction?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;p.s: I understand my question relates to &lt;a href=&quot;http://stats.stackexchange.com/questions/1458/why-is-multiple-comparison-a-problem&quot;&gt;this one&lt;/a&gt;, but since there are new people on the site, and since there is somewhat of a difference, I allowed myself to ask the above question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;p.p.s: I don't think this question has a &quot;right answer&quot;, thus I choose to have it a community wiki, but for some reason I can't find how to do it in the screen today...&lt;/p&gt;&#xA;" OwnerUserId="253" LastActivityDate="2010-11-04T19:10:54.393" Title="Choosing the scope when performing multiple comparisons?" Tags="&lt;multiple-comparisons&gt;&lt;meta-analysis&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="4215" PostTypeId="2" ParentId="2169" CreationDate="2010-11-04T16:51:47.097" Score="1" ViewCount="" Body="&lt;p&gt;As you know, from&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Var[q] = Var[\sum_i w_i x_{(i)}] = \sum_i\sum_j w_i w_j Cov[x_{(i)}, x_{(j)}]$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;it follows you need only compute the variances and covariances of the order statistics.  To do this, diagonalize the covariance matrix!  Although this cannot be done in general, M. A. Stephens has &lt;a href=&quot;http://biomet.oxfordjournals.org/content/62/1/23.full.pdf&quot; rel=&quot;nofollow&quot;&gt;obtained (heuristically) an asymptotic diagonalization&lt;/a&gt;.  (The eigenvectors are Hermite polynomials.)  In the spirit of PCA, limiting your calculations to the largest few eigenvalues can greatly reduce the computational effort and might produce a reasonable approximation, depending on the structure of the $w_i$.  In fact, if you adjust that weight vector to be a linear combination of a small number of the eigenvectors, that will assure a simple calculation of $Var[q]$ and perhaps not cost you much in terms of the accuracy of $q$ itself.  At worst, a preliminary eigendecomposition of $\vec{w}$ will then require only $O(N)$ calculations of the variance rather than $O(N^2)$.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-11-04T16:51:47.097" CommentCount="1" />
  <row Id="4216" PostTypeId="1" CreationDate="2010-11-04T17:10:25.227" Score="0" ViewCount="81" Body="&lt;p&gt;Hey guys, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;how do i go from an integral with boundaries [a,b] to a very small interval with lim=&gt; zero?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;what i am doing: I want to convert an equation &#xA;&lt;code&gt;Integrate [f(x), a, b ]&lt;/code&gt; &#xA;into &#xA;&lt;code&gt;Integrate [f(x), lim(a=&amp;gt;b)&lt;/code&gt;&#xA;so that the boundaries disappear and i get a thin slice. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope the question is clear enough. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;thanks for the advice&lt;/p&gt;&#xA;" OwnerUserId="1516" LastActivityDate="2010-11-04T17:10:25.227" ClosedDate="2010-11-04T19:29:04.967" Title="Integral boundaries to zero in mathematica" Tags="&lt;mathematical-statistics&gt;" CommentCount="3" />
  <row Id="4217" PostTypeId="2" ParentId="4214" CreationDate="2010-11-04T18:14:15.557" Score="1" ViewCount="" Body="&lt;p&gt;Think of the following two experiments:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Experiment A; Throw a fair coin 10 times to assess Prob(Heads).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Experiment B: Throw a fair dice 5 times to assess Prob(Face showing 1).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To take the coin toss &lt;a href=&quot;http://en.wikipedia.org/wiki/Multiple_comparisons#Example_.E2.80.94_Flipping_coins&quot; rel=&quot;nofollow&quot;&gt;example&lt;/a&gt; from the wiki: We may wish to declare a coin as biased if we observe more than 9 heads out of 10 tosses. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, if I were to repeat experiment A 100 times then there is 34% chance (see the wiki for the calculations) that we would identify a coin as biased when it is not thus increasing out type I error probability from 0.05 to 0.34. Therefore, we need to control for multiple comparisons in this context.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, note that our trials in experiment A have no influence on our results as far as experiment B is concerned as that is a completely different data generating process. The above suggests that we have to control for multiple comparisons for the two experiments separately instead of collectively.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words, controlling for multiple comparisons should be done whenever the comparisons involve the same data generating process.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Strictly speaking the above example of coin vs dice is not a good example as that would be analogous to experiments that investigate two very different questions (e.g., estimate whether smoking causes cancer and estimate if jumping red lights leads to an accident). In these contexts, controlling for multiple comparisons collectively for the two experiments is meaningless.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On further thinking, it is not clear to me if the data generating process really has a special role to play as far as multiple comparisons are concerned. Even if the data generating process were to be different (perhaps because of different covariates) you will still run the risk of increasing type I error because of multiple comparisons. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, it seems to me that what matters is whether the multiple comparisons involve making judgement about the &lt;em&gt;same&lt;/em&gt; null hypothesis. As long as the multiple comparisons involve the same &lt;em&gt;null&lt;/em&gt; hypothesis we have to correct for multiple comparisons to keep Type I error at desired levels (e.g., 0.05).&lt;/p&gt;&#xA;" LastActivityDate="2010-11-04T19:10:54.393" CommentCount="2" />
  <row Id="4218" PostTypeId="1" CreationDate="2010-11-04T22:03:16.153" Score="0" ViewCount="66" Body="&lt;p&gt;For example some common ones are engagement, churn, ARPU. The problem with the metrics I mentioned (besides needen to know more) is that I do not understand clearly how are they measure and what they mean in real life.&lt;/p&gt;&#xA;" OwnerUserId="1808" LastActivityDate="2010-11-04T22:03:16.153" Title="What are the most relevant metrics for social games, and how are they calculated?" Tags="&lt;feature-selection&gt;&lt;measurement&gt;" CommentCount="4" />
  <row Id="4219" PostTypeId="1" AcceptedAnswerId="4227" CreationDate="2010-11-05T01:24:23.780" Score="4" ViewCount="266" Body="&lt;p&gt;Hi all,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm using McNemar's test. Basically this question is about best practices when reporting results using McNemar's test.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to report the effect size. What is a sensible effect size for McNemar's test? I've seen the odds ratio b/c and the proportions b/(b+c) and c/(b+c) both used in papers. If I say what b and c are then all possible effect sizes can be computed. However I haven't seen this, is it bad form? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="1540" LastActivityDate="2010-11-05T10:40:01.677" Title="Effect size of McNemar's Test" Tags="&lt;hypothesis-testing&gt;&lt;nonparametric&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="4220" PostTypeId="1" AcceptedAnswerId="4223" CreationDate="2010-11-05T01:25:39.520" Score="12" ViewCount="504" Body="&lt;p&gt;On the wiki page about naive bayes classifiers &lt;a href=&quot;http://en.wikipedia.org/wiki/Naive_Bayes_classifier&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; there is this line &quot;P(height|male) = 1.5789 (A probability distribution over 1 is OK. It is the area under the bell curve that is equal to 1.)&quot; How can a value &gt; 1 be OK? I thought all probability values were expressed in the range 0 &amp;lt;= p &amp;lt;= 1. Furthermore, given that it is possible to have such a value, how is that value obtained in the example shown on the page? &lt;/p&gt;&#xA;" OwnerUserId="226" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-01-11T21:31:52.993" LastActivityDate="2011-01-11T22:56:03.347" Title="Probability distribution value exceeding 1 is OK?" Tags="&lt;distributions&gt;&lt;probability&gt;&lt;normal-distribution&gt;" AnswerCount="3" FavoriteCount="2" />
  <row Id="4221" PostTypeId="2" ParentId="4220" CreationDate="2010-11-05T01:38:47.637" Score="10" ViewCount="" Body="&lt;p&gt;This is a common mistake from not understanding the difference between probability mass functions, where the variable is discrete, and probability density functions, where the variable is continuous.  See &lt;a href=&quot;http://www.itl.nist.gov/div898/handbook/eda/section3/eda361.htm&quot; rel=&quot;nofollow&quot;&gt;What is a probability distribution&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;continuous probability functions are&#xA;  defined for an infinite number of&#xA;  points over a continuous interval, the&#xA;  probability at a single point is&#xA;  always zero. Probabilities are&#xA;  measured over intervals, not single&#xA;  points. That is, the area under the&#xA;  curve between two distinct points&#xA;  defines the probability for that&#xA;  interval. This means that the height&#xA;  of the probability function can in&#xA;  fact be greater than one. The property&#xA;  that the integral must equal one is&#xA;  equivalent to the property for&#xA;  discrete distributions that the sum of&#xA;  all the probabilities must equal one.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="493" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-11-05T15:20:19.447" LastActivityDate="2010-11-05T15:20:19.447" CommentCount="1" />
  <row Id="4222" PostTypeId="2" ParentId="4191" CreationDate="2010-11-05T01:56:14.007" Score="4" ViewCount="" Body="&lt;p&gt;I think you've missed the main advantage of algorithms like of Metropolis-Hastings. For Gibbs sampling, you will need to sample from the full conditionals.  You are right, that is rarely easy to do. The main advantage of Metropolis-Hastings algorithms is that you can still sample one parameter at a time, but you only need to know the full conditionals up to proportionality.  This is because the denominators cancel in the acceptance criteria function  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The unnormalized full conditionals are often available.  For instance, in your example&#xA;$P(x_1 | x_2,...,x_n) \propto P(x_1,...,x_n)$, which you have.  You don't need to do any integrals analytically. In most applications, a lot more will likely cancel too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Programs like WinBugs/Jags typically take Metropolis-Hastings or slice sampling steps that only require the conditionals up to proportionality.  These are easily available from the DAG.  Given conjugacy, they also sometimes take straight Gibbs steps or fancy block stops.&lt;/p&gt;&#xA;" OwnerUserId="493" LastEditorUserId="493" LastEditorDisplayName="" LastEditDate="2010-11-05T02:03:58.703" LastActivityDate="2010-11-05T02:03:58.703" CommentCount="2" />
  <row Id="4223" PostTypeId="2" ParentId="4220" CreationDate="2010-11-05T02:32:49.170" Score="12" ViewCount="" Body="&lt;p&gt;That Wiki page is abusing language by referring to this number as a probability.  You are correct that it is not.  It is actually a &lt;em&gt;probability per foot&lt;/em&gt;.  Specifically, the value of 1.5789 (for a height of 6 feet) implies that the probability of a height between, say, 5.99 and 6.01 feet is close to 1.5789 [1/foot] * (6.01 - 5.99) [feet] = 0.0316 [unitless].  &lt;em&gt;This&lt;/em&gt; value &lt;em&gt;must&lt;/em&gt; not exceed 1, as you know.  (The small range of heights (0.02 in this example) is a crucial part of the probability apparatus.  It is the &quot;differential&quot; of height, which I will abbreviate $d(\text{height})$.)  Probabilities per unit of something are called &lt;em&gt;densities&lt;/em&gt; by analogy to other densities, like mass per unit volume.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Bona fide&lt;/em&gt; probability &lt;em&gt;densities&lt;/em&gt; can have arbitrarily large values, even infinite ones.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;The value of 1.5789 /foot is obtained in that example by estimating that the heights of males have a normal distribution with mean 5.855 feet and variance 3.50e-2 square feet.  (This can be found in a previous table.) The square root of that variance is the standard deviation, 0.18717 feet.  We re-express 6 feet as the number of SDs from the mean:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$z = (6 - 5.855) / 0.18717 = 0.7747$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The division by the standard deviation produces a relation&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$dz = d(\text{height})/0.18717$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Normal probability density, by definition, equals&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{1}{\sqrt{2 \pi}}\exp(-z^2/2)dz = 0.29544\ d(\text{height}) / 0.18717 = 1.5789\  d(\text{height}).$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Actually, I cheated: I simply asked Excel to compute NORMDIST(6, 5.855, 0.18717, FALSE).  But then I really did check it against the formula, just to be sure.)  When we strip the &lt;em&gt;essential&lt;/em&gt; differential $d(\text{height})$ from the formula only the number $1.5789$ remains, like the Cheshire Cat's smile.  We, the readers, need to understand that the number has to be multiplied by a small difference in heights in order to produce a probability.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-11-05T16:10:29.070" LastActivityDate="2010-11-05T16:10:29.070" CommentCount="4" />
  <row Id="4224" PostTypeId="2" ParentId="4062" CreationDate="2010-11-05T03:17:06.993" Score="4" ViewCount="" Body="&lt;p&gt;Did you see this post? &lt;a href=&quot;http://groups.google.com/group/ggplot2/browse_thread/thread/8e1efd0e7793c1bb&quot; rel=&quot;nofollow&quot;&gt;http://groups.google.com/group/ggplot2/browse_thread/thread/8e1efd0e7793c1bb&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Take the example, add coord_polar() and reverse the axes and you get pretty close:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(cluster) &#xA;data(mtcars)&#xA;x &amp;lt;- as.phylo(hclust(dist(mtcars)))&#xA;&#xA;p &amp;lt;- ggplot(data=x)&#xA;p &amp;lt;- p + geom_segment(aes(y=x,x=y,yend=xend,xend=yend), colour=&quot;blue&quot;,alpha=1) &#xA;p &amp;lt;- p + geom_text(data=label.phylo(x), aes(x=y, y=x, label=label),family=3, size=3) + xlim(0, xlim) + coord_polar()&#xA;&#xA;theme &amp;lt;- theme_update(  axis.text.x = theme_blank(),&#xA;                        axis.ticks = theme_blank(),&#xA;                        axis.title.x = theme_blank(),&#xA;                        axis.title.y = theme_blank(),&#xA;                        legend.position = &quot;none&quot;&#xA;                     )&#xA;p &amp;lt;- p + theme_set(theme)&#xA;print(p)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1809" LastActivityDate="2010-11-05T03:17:06.993" CommentCount="2" />
  <row Id="4225" PostTypeId="1" AcceptedAnswerId="4249" CreationDate="2010-11-05T05:22:01.777" Score="3" ViewCount="199" Body="&lt;p&gt;I was thinking about CI and subjective Bayesian and I have following two questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;If a subjective (&lt;em&gt;not objective&lt;/em&gt;) Bayesian would care if her predictions don't do well in the real world.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;A classical statistician would not care if her confidence statement is (obviously) wrong for a given data set (as in Welch's Paradox, where conditioning is on ancillary statistics leads to the resolution of pathological behavior).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I think my answer for 1. is YES and 2. is NO. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I don't know if I am thinking along the right lines. Can I have some more insights?&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Welch's example&lt;/strong&gt;: This example works for any $n$, but we will take $n=2$ for simplicity. $X_1, X_2 \sim U(\theta - 1/2, \theta +1/2)$ (iid), $\theta \in R$. This implies $X_1 - \theta \sim U(-1/2, 1/2)$ (iid). $(X_1 + X_2) /2 - \theta$ (note that this is &lt;em&gt;NOT&lt;/em&gt; a statistic) has a distribution independent of $\theta$. We can choose c &gt; 0 s.t. $Prob_{\theta} [-c \le (X_1 + X_2) /2 - \theta \le c] = 1- \alpha (~0.99)$, implying $((X_1 + X_2) /2 - c, (X_1 + X_2) /2 + c)$ is the 99% CI of $\theta$. The interpretation of this CI is: if we sample repeatedly, we will get different $(X_1 + X_2) /2$ and (at least) 99% times it will contain true \theta. &lt;strong&gt;But for a particular set of $X_1, X_2$, we can't say if the CI contains $\theta$&lt;/strong&gt;. Now, consider the following data: $X_1 =0$ and $X_2=1$, as $|X_1 - X_2|=1$, we know FOR SURE (Prob =1) that the interval $(X_1, X_2)$ contains theta (one possible criticism, $P{|X_1 - X_2|=1} = 0$, but we can handle it mathematically and I won't discuss it). (Better details are in Pratt, 1961; Lehman, Chap 10, 2nd Edition, Prob 27, 28; Kiefer, 1977; Berger and Wolpert, 1988)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;S.&lt;/p&gt;&#xA;" OwnerUserId="1307" LastEditorUserId="1307" LastEditorDisplayName="" LastEditDate="2010-11-05T18:03:22.370" LastActivityDate="2010-11-05T18:03:22.370" Title="Subjective Bayesian's care for real world validation and classical statistician's worry about CI related paradoxes for a given data set?" Tags="&lt;bayesian&gt;&lt;confidence-interval&gt;" AnswerCount="2" CommentCount="3" />
  <row Id="4226" PostTypeId="1" AcceptedAnswerId="4235" CreationDate="2010-11-05T06:35:47.567" Score="7" ViewCount="234" Body="&lt;p&gt;Quantum Mechanics has generalized probability theory to negative/imaginary numbers, mostly to explain interference patterns, wave/particle duality and generally weird things like that. It can be seen more abstractly, however, as a noncommutative generalisation of Bayesian probability (quote from Terrence Tao). I'm curious about these things, though by no means an expert. Does this have any applications outside Quantum Mechanics? Just curious.&lt;/p&gt;&#xA;" OwnerUserId="1760" LastEditorUserId="1760" LastEditorDisplayName="" LastEditDate="2010-11-05T06:45:53.887" LastActivityDate="2010-11-07T17:05:11.557" Title="Do negative probabilities/probability amplitudes have applications outside quantum mechanics?" Tags="&lt;probability-theory&gt;" AnswerCount="3" FavoriteCount="1" />
  <row Id="4227" PostTypeId="2" ParentId="4219" CreationDate="2010-11-05T07:17:15.547" Score="4" ViewCount="" Body="&lt;p&gt;In general, I think best practice for presenting measures of effect size depends on the question of interest and the usual practice in your field. There's little point reporting an effect measure that readers will be unfamiliar with. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having said that, in this particular case if you want a &lt;em&gt;single&lt;/em&gt; effect measure I think the (conditional) odds ratio &lt;em&gt;b&lt;/em&gt; / &lt;em&gt;c&lt;/em&gt; is the obvious choice. Giving both proportions &lt;em&gt;b&lt;/em&gt; / (&lt;em&gt;b&lt;/em&gt; + &lt;em&gt;c&lt;/em&gt;) and &lt;em&gt;c&lt;/em&gt; / (&lt;em&gt;b&lt;/em&gt; + &lt;em&gt;c&lt;/em&gt;) gives more information and clearly the odds ratio can be derived from these two. I wouldn't call two proportions an 'effect size' though.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd strongly encourage you to report &lt;em&gt;all four&lt;/em&gt; cells of the 2×2 table concordancy table, however. That way nothing is hidden and anyone can calculate whatever statistics they wish. It's only four numbers, after all.&lt;/p&gt;&#xA;" OwnerUserId="449" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-11-05T10:40:01.677" LastActivityDate="2010-11-05T10:40:01.677" />
  <row Id="4228" PostTypeId="2" ParentId="4093" CreationDate="2010-11-05T07:42:26.210" Score="0" ViewCount="" Body="&lt;p&gt;PCA results (the different dimensions or commponents) generally can't be translated into a real concept I think is wrong to assume that one of the components is &quot;fear of bears&quot; what lead you to think that was what the component meant? &#xA;Principal components procedure transforms your data matrix to a new data matrix with the same or less amount of dimensions, and the resulting dimensions range from the one that better explains the variance to the one that explains it the less. This components are calculated based on a combination of the original variables with the calculated eigenvectors. &#xA;Overal PCA procedure does convert the original variables to orthogonal ones (linearly independent).&#xA;Hope this helps you clarify a little about pca procedure&lt;/p&gt;&#xA;" OwnerUserId="1808" LastActivityDate="2010-11-05T07:42:26.210" CommentCount="3" />
  <row Id="4230" PostTypeId="2" ParentId="4089" CreationDate="2010-11-05T07:58:21.050" Score="0" ViewCount="" Body="&lt;p&gt;Maybe you are looking for the library ggplot2 that lets you plot things in a pretty way.&#xA;Or you can check this website that seems to have lots of R graphic utilities&#xA;&lt;a href=&quot;http://addictedtor.free.fr/graphiques/&quot; rel=&quot;nofollow&quot;&gt;http://addictedtor.free.fr/graphiques/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1808" LastActivityDate="2010-11-05T07:58:21.050" />
  <row Id="4231" PostTypeId="1" CreationDate="2010-11-05T08:52:03.723" Score="3" ViewCount="152" Body="&lt;p&gt;Hi..I have heard the terms training and validating a model. I know that we select variables which are most statistically significant and we look for other things like multicollinearity. My question is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;what does traning a model involves more than this ? &lt;/p&gt;&#xA;" OwnerUserId="1763" LastActivityDate="2010-11-05T12:50:14.080" Title="Training a model" Tags="&lt;logistic-regression&gt;" AnswerCount="1" CommentCount="3" FavoriteCount="2" />
  <row Id="4232" PostTypeId="2" ParentId="4226" CreationDate="2010-11-05T10:04:37.290" Score="4" ViewCount="" Body="&lt;p&gt;Well, kind of. I'm by no means an expert either, but I read &lt;a href=&quot;http://www.espenhaug.com/NegativeProbabilitiesHaug.pdf&quot; rel=&quot;nofollow&quot;&gt;THIS&lt;/a&gt; article by Esben Haug and found it quite interesting.&lt;/p&gt;&#xA;" OwnerUserId="1819" LastActivityDate="2010-11-05T10:04:37.290" />
  <row Id="4233" PostTypeId="1" CreationDate="2010-11-05T10:46:22.953" Score="2" ViewCount="328" Body="&lt;p&gt;hi..I know about $r^2$ tells you about the amount of variation that can be explained by the predictor variables. I have run a model in which the rsquare has value 0.3010 but has false positive rate of around 15.60%. So this model which is logit in nature predicts 84% cases right. I want to know two things:&lt;br&gt;&#xA;1) Is this false positive rate significant, i.e. is this prediction good enough?&lt;br&gt;&#xA;2) Does it matter if I $r^2$ is as low as 0.3 if my model is good in prediction as we know that $r^2$ is a value used for explaning and not predicting?&lt;/p&gt;&#xA;" OwnerUserId="1763" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-11-05T13:08:36.773" LastActivityDate="2010-11-06T16:17:06.830" Title="Significance of $r^2$ value" Tags="&lt;hypothesis-testing&gt;&lt;regression&gt;&lt;logistic-regression&gt;" AnswerCount="1" CommentCount="5" />
  <row Id="4234" PostTypeId="2" ParentId="4093" CreationDate="2010-11-05T11:19:36.060" Score="2" ViewCount="" Body="&lt;p&gt;For me, PCA scores are just re-arrangements of the data in a form that allows me to explain the data set with less variables. The scores represent how much each item relates to the component. You can name them as per factor analysis, but its important to remember that they are not latent variables, as PCA analyses all variance in the data set, not just the elements held in common (as factor analysis does).&lt;/p&gt;&#xA;" OwnerUserId="656" LastActivityDate="2010-11-05T11:19:36.060" CommentCount="4" />
  <row Id="4235" PostTypeId="2" ParentId="4226" CreationDate="2010-11-05T12:23:25.587" Score="5" ViewCount="" Body="&lt;p&gt;Yes.  I like the article Soren shared very much, and together with the references in that article I would recommend Muckenheim, W. &lt;em&gt;et al.&lt;/em&gt; (1986). &lt;a href=&quot;http://www.sciencedirect.com/science?_ob=ArticleURL&amp;amp;_udi=B6TVP-46SX0JS-88&amp;amp;_user=10&amp;amp;_coverDate=02%2F28%2F1986&amp;amp;_rdoc=1&amp;amp;_fmt=high&amp;amp;_orig=search&amp;amp;_origin=search&amp;amp;_sort=d&amp;amp;_docanchor=&amp;amp;view=c&amp;amp;_searchStrId=1530555242&amp;amp;_rerunOrigin=google&amp;amp;_acct=C000050221&amp;amp;_version=1&amp;amp;_urlVersion=0&amp;amp;_userid=10&amp;amp;md5=1fc99b45c91f0d7e00c238a3649d7532&amp;amp;searchtype=a&quot; rel=&quot;nofollow&quot;&gt;A Review of Extended Probabilities&lt;/a&gt;. Phys. Rep. 133 (6) 337-401.  It's a physics paper for sure, but the applications there are not all related to quantum physics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My personal favorite application relates to &lt;a href=&quot;http://en.wikipedia.org/wiki/De_Finetti%27s_theorem&quot; rel=&quot;nofollow&quot;&gt;de Finetti's Theorem&lt;/a&gt; (also Bayesian in flavor):  if we don't mind negative probabilities then it turns out that &lt;strong&gt;all&lt;/strong&gt; exchangeable sequences (even finite, perhaps negatively correlated ones) are a (signed) mixture of IID sequences.  Of course, this itself has applications in quantum mechanics, in particular, that Fermi-Dirac statistics yield the same type of (signed) mixture representation that Bose-Einstein statistics do. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My second personal favorite application (outside of physics proper) relates to &lt;a href=&quot;http://en.wikipedia.org/wiki/Infinite_divisibility_%28probability%29&quot; rel=&quot;nofollow&quot;&gt;infinite divisible&lt;/a&gt; (ID) distributions, which classically includes normal, gamma, poisson, ... the list continues.  It isn't too hard to show that ID distributions must have unbounded support, which immediately kills distributions like the binomial or uniform (discrete+continuous) distributions.  But if we permit negative probabilities then these problems disappear and the binomial, uniform (discrete+continuous), and a whole bunch of other distributions then become infinitely divisible - in this &lt;em&gt;extended&lt;/em&gt; sense, please bear in mind.  ID distributions relate to statistics in that they are limiting distributions in generalized central limit theorems.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By the way, the first application is whispered folklore among probabilists and the infinite divisibility stuff is proved &lt;a href=&quot;http://maurice.bgsu.edu/record=b2719985~S0&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;, an informal electronic copy being &lt;a href=&quot;http://people.ysu.edu/~gkerns/pdf/Jdiss.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Presumably there is a bunch of material on &lt;a href=&quot;http://www.arxiv.org/&quot; rel=&quot;nofollow&quot;&gt;arXiv&lt;/a&gt;, too, though I haven't checked there in quite some time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a final remark, whuber is absolutely right that it isn't really legal to call anything a probability that doesn't lie in $[0,1]$, at the very least, not for the time being. Given that &quot;negative probabilities&quot; have been around for so long I don't see this changing in the near future, not without some kind of colossal breakthrough.&lt;/p&gt;&#xA;" OwnerUserId="1108" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-11-07T17:05:11.557" LastActivityDate="2010-11-07T17:05:11.557" CommentCount="2" />
  <row Id="4236" PostTypeId="2" ParentId="1980" CreationDate="2010-11-05T12:26:45.920" Score="1" ViewCount="" Body="&lt;p&gt;Looking for examples and practices is a good way to learn, but I just wanted to mention that reproducibility has not only technical/script rerun side, but also code style and structuring aspect, minimization of side effects in core functions etc. &#xA;I personally found that Chambers book Software for Data Analysis allows to understand more deeply techniques that help to avoid reliability and reproducibility issues on R code level.&lt;/p&gt;&#xA;" OwnerUserId="1820" LastActivityDate="2010-11-05T12:26:45.920" />
  <row Id="4237" PostTypeId="2" ParentId="4231" CreationDate="2010-11-05T12:39:03.030" Score="5" ViewCount="" Body="&lt;p&gt;Although the &lt;a href=&quot;http://en.wikipedia.org/wiki/Curse_of_dimensionality&quot; rel=&quot;nofollow&quot;&gt;curse of dimensionality&lt;/a&gt; and multicollinearity are distinct issues, cross-validation is used for building a predictive model: we usually estimate parameters of our model on training samples, and assess its generalizability on test samples. This yields a measure of model performance, which can be a % of prediction accuracy if we work with a classification model, or an RMSEA if it is a regression model. The idea is that the better the model performs, the better it will allow us to predict outcome(s) on unseen data.&#xA;To overcome the problem of overfitting when building a predictive model, we may also introduce some kind of variable or &lt;a href=&quot;http://en.wikipedia.org/wiki/Feature_selection&quot; rel=&quot;nofollow&quot;&gt;feature selection&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cross-validation may be done in various way (split or holdout method, k-fold, leave-one-out, etc.) but the general idea to keep in mind is that the final model is assessed on individuals who do not participate to its construction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may find additional information by looking at the &quot;&lt;a href=&quot;http://stats.stackexchange.com/questions/tagged/cross%2Dvalidation&quot;&gt;cross-validation&lt;/a&gt;&quot; or &quot;&lt;a href=&quot;http://stats.stackexchange.com/questions/tagged/feature%2Dselection&quot;&gt;feature selection&lt;/a&gt;&quot; tags.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-05T12:50:14.080" LastActivityDate="2010-11-05T12:50:14.080" />
  <row Id="4238" PostTypeId="1" CreationDate="2010-11-05T13:32:21.913" Score="4" ViewCount="83" Body="&lt;p&gt;We are often interested in estimating the limiting distribution of a parameter in situations where the data exhibit dependence within clusters. For example, a study of the effects of a household-level treatment on household-level outcomes must contend with the possibility that households within villages will have correlated outcomes.   Thus, when computing standard errors for the treatment effects, we typically take this clustering into account by using a &quot;cluster robust&quot; covariance estimator or perhaps a &quot;random effects&quot; model. (See &lt;a href=&quot;http://bit.ly/bAah5L&quot; rel=&quot;nofollow&quot;&gt;http://bit.ly/bAah5L&lt;/a&gt; for an example.)  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The properties of these covariance estimators are typically studied by assuming the source of the clustering are common &quot;shocks&quot; that occur within a group.  That is for groups indexed by $g$ and units within groups indexed by $i$, we typically write down a model of the form, $y_{ig} = \alpha_g + \epsilon_{ig}$, where the $\alpha_g$'s denote the group level shocks.  We then typically assume that the $\alpha_g$ are independent across groups with group level variance $\sigma^2_{\alpha}$ and the $\epsilon_{ig}$ are independent draws with variance $\sigma^2_{\epsilon}$, in which case the intra-cluster correlation is the familiar $\frac{\sigma^2_{\alpha}}{\sigma^2_{\alpha}+\sigma^2_{\epsilon}}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's what I am wondering: does this linear &quot;group shocks&quot; model implicitly rule out important forms of within-cluster correlation? (I am limiting the question to correlation for now, and not other types of non-correlation dependence.) That is, are there ways that within-cluster correlation can arise that cannot, through algebraic manipulation, be expressed in terms of a group-level shock?  Or is the group level shock characterization general with respect to within group correlation?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am inclined to try working out some of the examples, but I thought I'd put this up to see if someone else has already done the heavy (or maybe it's not so heavy) lifting.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If this is not a well-posed question (i.e. unanswerable), I'd also welcome comments about why.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Note I am not putting &quot;clustering&quot; on the tag list here because it's not about cluster methods as the term is usually applied.)&lt;/p&gt;&#xA;" OwnerUserId="96" LastEditorUserId="96" LastEditorDisplayName="" LastEditDate="2010-11-05T13:44:57.407" LastActivityDate="2010-11-05T13:44:57.407" Title="Sources of within-cluster correlation other than &quot;random shocks&quot;" Tags="&lt;correlation&gt;&lt;random-effects-model&gt;&lt;basic-concepts&gt;" AnswerCount="1" FavoriteCount="2" />
  <row Id="4239" PostTypeId="1" CreationDate="2010-11-05T13:34:50.963" Score="7" ViewCount="475" Body="&lt;p&gt;I am designing a questionnaire for my dissertation.  I am in the process of validating the questionnaire I have applied a Cronbach's alpha test to the initial sample group.  The responses to the questionnaire are on a Likert scale; can anyone suggest any further tests to apply to help test its validity. I am not an expert on statistics so any help would be appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have been doing some research and it appears I can do a Rasch analysis has anyone got any free software sites to apply this test and advice?&lt;/p&gt;&#xA;" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-11-07T11:09:10.120" LastActivityDate="2010-11-30T12:37:24.433" Title="Validating questionnaires" Tags="&lt;psychometrics&gt;&lt;scales&gt;&lt;survey&gt;" AnswerCount="2" FavoriteCount="4" />
  <row Id="4240" PostTypeId="2" ParentId="4238" CreationDate="2010-11-05T13:42:40.777" Score="1" ViewCount="" Body="&lt;p&gt;If you're interested in financial applications, I guess you can find some elements here, in the slides (starting from 50)&#xA;&lt;a href=&quot;http://freakonometrics.blog.free.fr/index.php?post/2008/10/11/noise-traders&quot; rel=&quot;nofollow&quot;&gt;http://freakonometrics.blog.free.fr/index.php?post/2008/10/11/noise-traders&lt;/a&gt;&lt;/p&gt;&#xA;" LastActivityDate="2010-11-05T13:42:40.777" />
  <row Id="4241" PostTypeId="2" ParentId="4165" CreationDate="2010-11-05T14:04:18.887" Score="0" ViewCount="" Body="&lt;p&gt;Other areas of application for stochastic processes: (1) Asymptotic theory: This builds on PeterR's comment about an IID sequence.  Law of large numbers and central limit theorem results require an understanding of stochastic processes. This is so fundamental in so many areas of application that I am inclined to say that &lt;em&gt;anyone&lt;/em&gt; with a graduate degree in stats or a field that uses sampling or frequentist inference ought to have key stochastic processes results under their belt.  (2) Structural equation modeling for causal inference a la Judea Pearl: Analyzing directed acyclic graphs (DAGs) of causal processes requires some handle of stochastic process theory. &lt;/p&gt;&#xA;" OwnerUserId="96" LastActivityDate="2010-11-05T14:04:18.887" CommunityOwnedDate="2010-11-05T14:04:18.887" />
  <row Id="4242" PostTypeId="2" ParentId="4239" CreationDate="2010-11-05T14:14:26.580" Score="10" ViewCount="" Body="&lt;p&gt;I will assume that your questionnaire is to be considered as one unidimensional scale (otherwise, Cronbach's alpha doesn't make very much sense). It is worth running an exploratory factor analysis to check for that. It will also allow you to see how items relate to the scale (i.e., through their loadings).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basic steps for validating your items and your scale should include:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;a complete report on the items' basic statistics (range, quartiles, central tendency, ceiling and floor effects if any);&lt;/li&gt;&#xA;&lt;li&gt;checking the internal consistency as you've done with your alpha (best, give 95% confidence intervals, because it is sample-dependent);&lt;/li&gt;&#xA;&lt;li&gt;describe you summary measure (e.g., total or mean score, aka scale score) with usual statistics (histogram + density, quantiles etc.);&lt;/li&gt;&#xA;&lt;li&gt;check your summary responses against specific covariates which are supposed to be related to the construct your are assessing -- this is referred to as known-group validity;&lt;/li&gt;&#xA;&lt;li&gt;if possible, check your summary responses against known instruments that purport to measure the same construct (&lt;a href=&quot;http://en.wikipedia.org/wiki/Concurrent_validity&quot; rel=&quot;nofollow&quot;&gt;concurrent&lt;/a&gt; or convergent validity).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If your scale is not unidimensional, these steps have to be done for each subscale, and you could also factor out the correlation matrix of your factors to assess the second-order factor structure (or use structural equation modeling, or confirmatory factor analysis, or whatever you want). You can also assess convergent and discriminant validity by using Multi-trait scaling or Multi-trait multi-method modeling (based on interitem correlations within and between scales), or, again, SEMs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, I would say that Item Response Theory would not help that much unless you are interested in shortening your questionnaire, filtering out some items that show &lt;a href=&quot;http://en.wikipedia.org/wiki/Differential_item_functioning&quot; rel=&quot;nofollow&quot;&gt;differential item functioning&lt;/a&gt;, or use your test in some kind of a &lt;a href=&quot;http://en.wikipedia.org/wiki/Computerized_adaptive_testing&quot; rel=&quot;nofollow&quot;&gt;computer adaptive test&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In any case, the &lt;a href=&quot;http://en.wikipedia.org/wiki/Rasch_model&quot; rel=&quot;nofollow&quot;&gt;Rasch model&lt;/a&gt; is for binary items. For polytomous ordered items, the most commonly used models are :&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;the graded response model&lt;/li&gt;&#xA;&lt;li&gt;the partial credit model&lt;/li&gt;&#xA;&lt;li&gt;the rating scale model.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Only the latter two are from the Rasch family, and they basically use an adjacent odds formulation, with the idea that subject has to &quot;pass&quot; several thresholds to endorse a given response category. The difference between these two models is that the PCM does not impose that thresholds are equally spaced on the theta (&lt;em&gt;ability&lt;/em&gt;, or subject location on the latent trait) scale. The graded response model relies on a cumulative odds formulation. Be aware that these models all suppose that the scale is unidimensional; i.e., there's only one latent trait. There are additional assumptions like, e.g., local independence (i.e., the correlations between responses are explained by variation on the ability scale). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyway, you will find a very complete documentation and useful clues to apply psychometric methods in R in volume 20 of the Journal of Statistical Software: &lt;a href=&quot;http://www.jstatsoft.org/v20&quot; rel=&quot;nofollow&quot;&gt;Special Volume: Psychometrics in R&lt;/a&gt;. Basically, the most interesting R packages that I use in my daily work are: &lt;a href=&quot;http://cran.r-project.org/web/packages/ltm/index.html&quot; rel=&quot;nofollow&quot;&gt;ltm&lt;/a&gt;, &lt;a href=&quot;http://cran.r-project.org/web/packages/eRm/index.html&quot; rel=&quot;nofollow&quot;&gt;eRm&lt;/a&gt;, &lt;a href=&quot;http://cran.r-project.org/web/packages/psych/index.html&quot; rel=&quot;nofollow&quot;&gt;psych&lt;/a&gt;, &lt;a href=&quot;http://cran.r-project.org/web/packages/psy/index.html&quot; rel=&quot;nofollow&quot;&gt;psy&lt;/a&gt;. Others are referenced on the CRAN task view &lt;a href=&quot;http://cran.r-project.org/web/views/Psychometrics.html&quot; rel=&quot;nofollow&quot;&gt;Psychometrics&lt;/a&gt;. Other resources of interest are: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.psych.upenn.edu/~baron/rpsych/rpsych.html&quot; rel=&quot;nofollow&quot;&gt;Notes on the use of R for psychology experiments and questionnaires&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://personality-project.org/r/&quot; rel=&quot;nofollow&quot;&gt;Using R for psychological research&lt;/a&gt; (W. Revelle is actually writing a book on &lt;a href=&quot;http://personality-project.org/r/book/&quot; rel=&quot;nofollow&quot;&gt;psychometrics in R&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;li&gt;the &lt;a href=&quot;http://r-forge.r-project.org/projects/psychor/&quot; rel=&quot;nofollow&quot;&gt;PsychoR&lt;/a&gt; project (it does not focus on IRT and scale development, though).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;A good review on the use of FA vs. IRT in scale development can be found in Scale construction and evaluation in practice: &lt;a href=&quot;http://www.ppsw.rug.nl/~boomsma/tenholt_vanduijn_boomsma_2010.pdf&quot; rel=&quot;nofollow&quot;&gt;A review of factor analysis versus item response theory applications&lt;/a&gt;, by ten Holt et al (Psychological Test and Assessment Modeling (2010) 52(3): 272-297).&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-30T12:37:24.433" LastActivityDate="2010-11-30T12:37:24.433" />
  <row Id="4243" PostTypeId="2" ParentId="4239" CreationDate="2010-11-05T14:51:50.307" Score="5" ViewCount="" Body="&lt;p&gt;While supporting everything said above, i would suggest that you do the following (in similiar enough order)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Firstly, you should be using R, if not you should start. The following advice is predicated on the use of R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'll assume that you have, at this point, calculated the descriptive statistics et al. If not, the psych package has a describe() function which should give you the stats you need. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Install the psych package from CRAN. &#xA;Load the psych package. &#xA;Use the fa.parallel routine on your data. This should give you a number of factors to retain. &#xA;Then, use the VSS(routine). This calculates the MAP criterion which gives you a different (normally) number of factors to retain. &#xA;Use a form of factor analysis (not principal components) and an oblique rotation for each number of factors. If your factors do not appear to be correlated after an oblique rotation, switch to orhogonal rotation. This is as an orthogonal structure can be determined from an oblique rotation, but not vice versa. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Extract all the factor solutions between the MAP criterion and the parallel analysis criterion. Determine which of these has the best fit indices and makes the most sense. This is the one you should retain. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On IRT, having used both ltm and eRm, I would suggest starting with eRm. It has better graphics functions for your models, and support for polytomous models is greater. That being said, it only fits Rasch models, and often data from psychological questionnaires do not meet the requirements for them. &#xA;Good luck! Psychometrics is a lot of fun, as you will no doubt discover.  &lt;/p&gt;&#xA;" OwnerUserId="656" LastActivityDate="2010-11-05T14:51:50.307" CommentCount="1" />
  <row Id="4244" PostTypeId="2" ParentId="3616" CreationDate="2010-11-05T14:54:01.720" Score="1" ViewCount="" Body="&lt;p&gt;If you want to assume simple linear trend, you can take the difference of each data set at the various time points and test that the slope of the line is zero.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-Ralph Winters&lt;/p&gt;&#xA;" LastActivityDate="2010-11-05T14:54:01.720" />
  <row Id="4245" PostTypeId="1" AcceptedAnswerId="4246" CreationDate="2010-11-05T14:54:16.593" Score="3" ViewCount="559" Body="&lt;p&gt;My problem with understanding this expression might come from the fact that English is not my first language, but I don't understand why it's used in this way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The marginal mean is typically the mean of a group or subgroup's measures of a variable in an experiment, but why not just use the word mean? What's the marginal here for?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See the &lt;a href=&quot;http://en.wiktionary.org/wiki/marginal&quot; rel=&quot;nofollow&quot;&gt;definition of marginal from wiktionary&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="1320" LastActivityDate="2011-01-19T18:34:36.517" Title="What is the meaning of 'Marginal mean'?" Tags="&lt;terminology&gt;&lt;marginal&gt;" AnswerCount="3" />
  <row Id="4246" PostTypeId="2" ParentId="4245" CreationDate="2010-11-05T15:01:44.253" Score="5" ViewCount="" Body="&lt;p&gt;Perhaps, the term originates from how the data is represented in a contingency table. See this &lt;a href=&quot;http://en.wikipedia.org/wiki/Contingency_table#Example&quot; rel=&quot;nofollow&quot;&gt;example&lt;/a&gt; from the wiki.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the above example, we would speak of marginal totals for gender and handedness when referring to the last column and the bottom row respectively. If you see the wiktionary the first definition of marginal is:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;ol&gt;&#xA;  &lt;li&gt;of, relating to, or located at a margin or an edge&lt;/li&gt;&#xA;  &lt;/ol&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Since the totals (and means if means are reported) are at the edge of the table they are referred to as marginal totals (and marginal means if the edges have means).&lt;/p&gt;&#xA;" LastActivityDate="2010-11-05T15:01:44.253" CommentCount="3" />
  <row Id="4247" PostTypeId="1" CreationDate="2010-11-05T15:11:02.717" Score="2" ViewCount="274" Body="&lt;p&gt;I am CS major, please be patient if my question is not well-stated. This is cross-posted from &lt;a href=&quot;http://math.stackexchange.com/questions/9057/hypothesis-testing-on-mixture-models&quot;&gt;math&lt;/a&gt; as suggested by J.M.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The dataset is quantitative mass spectrometry (MS) data. By labeling proteins of two different samples A and B, we get the relative abundance of 100 to thousands of proteins in A/B. Alongside with this ratio, we can estimate its variance based on the signal intensities. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Wanted: A list of proteins significantly different from the set of all protein ratios.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most proteins remain unchanged between A and B. The population of log-ratios distributes around 1. The histogram shows a bell shape with fat tails. Two-term Gaussian mixture model has been found to provide a good fit to experimental noise. I suppose it would work good for this data - think of experimental and biological noise.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;How to test for significantly different ratios on such a two-term Gaussian mixture model?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Thanks for your responses!&lt;/p&gt;&#xA;" OwnerUserId="1829" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-27T16:03:22.683" LastActivityDate="2011-03-27T16:03:22.683" Title="How to do hypothesis testing on Gaussian mixture model?" Tags="&lt;hypothesis-testing&gt;&lt;mixture&gt;" CommentCount="6" FavoriteCount="1" />
  <row Id="4248" PostTypeId="2" ParentId="4226" CreationDate="2010-11-05T15:30:54.033" Score="4" ViewCount="" Body="&lt;p&gt;QM does not use negative or imaginary probabilities: if it did, they would no longer be probabilities!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What can be (and usually is) a complex value is the quantum mechanical &lt;em&gt;wave function&lt;/em&gt; $\psi$.  From it the &lt;em&gt;probability amplitude&lt;/em&gt; (which is a &lt;em&gt;bona fide&lt;/em&gt; probability density) can be constructed; it is variously written $&amp;lt;\psi|\psi&gt;$ or $\|\psi\|^2$.  When $\psi$ has (complex) scalar values, $\|\psi\|^2 = \psi^* \psi$.  In every case these values are nonnegative real numbers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For details, see the section on &quot;Postulates of Quantum Mechanics&quot; in the &lt;a href=&quot;http://en.wikipedia.org/wiki/Mathematical_formulations_of_quantum_mechanics&quot; rel=&quot;nofollow&quot;&gt;Wikipedia article&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-11-05T15:30:54.033" />
  <row Id="4249" PostTypeId="2" ParentId="4225" CreationDate="2010-11-05T16:02:06.010" Score="3" ViewCount="" Body="&lt;p&gt;For many reasons you're right about 1.  I certainly wouldn't heed the advice of someone who did not care about whether it is any good!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Number 2, as you have expressed it, does not characterize good practice.  If there are possible datasets where a CI (or any decision procedure, for that matter) is clearly wrong, then that procedure is inadmissible by definition (because you can replace it by one that is at least as good and sometimes better, no matter what).  Yes, inadmissible procedures are sometimes used in practice, but it can be argued that such procedures actually are admissible when we include the cost of performing the procedure itself within the loss function.  (In other words, a quick and dirty method that works ok can be considered superior to one that requires extensive time and effort to calculate and works only a little better.)  But in this hypothetical case, the &quot;obviously&quot; clause indicates it takes no effort to recognize some wrong intervals and replace them with better ones.  Therefore, although I do think your reasoning is good, we should conclude that a thoughtful concerned &quot;classical&quot; statistician would indeed care: the answer should be YES to both questions.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-11-05T16:02:06.010" CommentCount="4" />
  <row Id="4250" PostTypeId="2" ParentId="4245" CreationDate="2010-11-05T16:42:02.713" Score="1" ViewCount="" Body="&lt;p&gt;I'd assume it means the sample analogue of the marginal expectation $\operatorname{E}(X)$, as opposed to the sample analogue of a conditional expectation $\operatorname{E}(X \mid Y)$, where $Y$ could be anything.&lt;/p&gt;&#xA;" OwnerUserId="449" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-11-05T17:00:14.750" LastActivityDate="2010-11-05T17:00:14.750" CommentCount="4" />
  <row Id="4251" PostTypeId="2" ParentId="4225" CreationDate="2010-11-05T17:33:07.933" Score="3" ViewCount="" Body="&lt;p&gt;For the second question, I believe the answer is &quot;Yes&quot;. I will quote Andrew Gelman here, &quot;..in general there is no coverage guarantee because frequency properties depend on nuisance&#xA;parameters which can only be ignored in some special cases of pivotal test statistics&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can take a look at the following paper for some really nice discussion of relative merits and demerits of these procedures. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.stat.columbia.edu/~gelman/research/published/badbayesresponsemain.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.stat.columbia.edu/~gelman/research/published/badbayesresponsemain.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;V.S.&lt;/p&gt;&#xA;" OwnerUserId="1831" LastActivityDate="2010-11-05T17:33:07.933" CommentCount="1" />
  <row Id="4252" PostTypeId="1" AcceptedAnswerId="4254" CreationDate="2010-11-05T17:35:33.833" Score="2" ViewCount="335" Body="&lt;p&gt;How do you exactly calculate this? Let's say truncated by 10%?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can imagine how to do it if you have 10 entries or so, but how can I do it for a lot of entries?&lt;/p&gt;&#xA;" OwnerUserId="1833" LastEditorUserId="1833" LastEditorDisplayName="" LastEditDate="2010-11-06T19:11:54.750" LastActivityDate="2010-11-06T23:16:32.310" Title="How to calculate truncated mean?" Tags="&lt;mean&gt;&lt;truncation&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="1" />
  <row Id="4253" PostTypeId="2" ParentId="4089" CreationDate="2010-11-05T17:44:49.703" Score="0" ViewCount="" Body="&lt;p&gt;Its probably not exactly what you are looking for, but the pairs.panels() function in the psych package for R may prove useful. It gives you  correlation values in the upper diagonal, loess lines and points in the lower diagonal, and shows a histogram of each variable's scores in the diagonal line of the matrix. I personally think its one of the best graphical summaries of data around. &lt;/p&gt;&#xA;" OwnerUserId="656" LastActivityDate="2010-11-05T17:44:49.703" />
  <row Id="4254" PostTypeId="2" ParentId="4252" CreationDate="2010-11-05T17:52:59.327" Score="5" ViewCount="" Body="&lt;p&gt;Trimmed mean involves trimming $P$ percent observations from both ends.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;E.g.: If you are asked to compute a 10% trimmed mean, $P = 10$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given a bunch of observations, $X_i$:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;First find $n$ = number of observations.&lt;/li&gt;&#xA;&lt;li&gt;Reorder them as &quot;order statistics&quot; $X_i$ from the smallest to the largest.  &lt;/li&gt;&#xA;&lt;li&gt;Find lower case $p = P/100$ = proportion trimmed.&lt;/li&gt;&#xA;&lt;li&gt;Compute $n p$.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;If $n p$ is an integer use $k = n p$  and trim $k$ observations at both ends.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$R$ = remaining observations = $n - 2k$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Trimmed mean = $(1/R) \left( X_{k+1} + X_{k+2} + \ldots + X_{n-k} \right).$  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Example&lt;/strong&gt;: Find 10% trimmed mean of &lt;/p&gt;&#xA;&#xA;&lt;p&gt;2, 4, 6, 7, 11, 21, 81, 90, 105, 121&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here, $n = 10, p = 0.10, k = n p = 1$ which is an integer so trim exactly one observation&#xA;at each end, since $k = 1$. Thus trim off 2 and 121.  We are left with $R = n - 2k = 10 - 2 = 8$ observations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;10% trimmed mean= (1/8) * (4 + 6 + 7 + 11 + 21 + 81 + 90 + 105) = 40.625&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If $ n p$ has a fractional part present, trimmed mean is a bit more&#xA;complicated. In the above example, if we wanted 15% trimmed mean,&#xA;$P = 15, p = 0.15, n = 10, k = n p = 1.5$. This has integer part 1 and fractional&#xA;part 0.5 is present.  $R = n - 2k = 10 - 2 * 1.5 = 10 - 3 = 7$.&#xA;Thus $R = 7$ observations are retained. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Addendum upon @whuber's comment: To remain unbiased (after removing 2 and 121), it seems we must remove half of the 4 and half of the 105 for a trimmed mean of $(4/2 + 6 + 7 + 11 + 21 + 81 + 90 + 105/2)/7 = 34.64$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Source:&lt;/strong&gt; &lt;a href=&quot;http://www.fordham.edu/economics/vinod/st1trim.doc&quot; rel=&quot;nofollow&quot;&gt;Class notes on P percent trimmed mean&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="69" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-11-05T21:32:22.447" LastActivityDate="2010-11-05T21:32:22.447" CommentCount="8" />
  <row Id="4255" PostTypeId="2" ParentId="4089" CreationDate="2010-11-05T18:42:31.133" Score="1" ViewCount="" Body="&lt;p&gt;To explore dataset I really like &lt;code&gt;rattle&lt;/code&gt;. Install the package and just call &lt;code&gt;rattle()&lt;/code&gt;. The interface is quite self explainatory.&lt;/p&gt;&#xA;" OwnerUserId="582" LastActivityDate="2010-11-05T18:42:31.133" CommentCount="3" />
  <row Id="4256" PostTypeId="2" ParentId="2397" CreationDate="2010-11-05T19:28:01.477" Score="1" ViewCount="" Body="&lt;p&gt;Is it not the case that $|S_{-i}|=|S_{-j}|$ for all $i,j$ where $S_{-i}$ is the Multinomial covariance matrix with the $i$-th row and column removed?  Since this is the case, I don't understand what you mean by &quot;freedom of choice&quot; as any &quot;choice&quot; is equivalent.&lt;/p&gt;&#xA;" OwnerUserId="1835" LastEditorUserId="1835" LastEditorDisplayName="" LastEditDate="2010-11-05T19:33:51.173" LastActivityDate="2010-11-05T19:33:51.173" CommentCount="6" />
  <row Id="4257" PostTypeId="2" ParentId="4233" CreationDate="2010-11-05T20:41:18.610" Score="4" ViewCount="" Body="&lt;p&gt;First, no model is perfect unless it is over-fit.  So, your false positive rate is not unusual.  Is a false positive rate of 16% good or bad?  If it is lower than the natural proportion in the data it is OK.  If it is not it is really bad.  The key is by how much your model reduces the error rate.  That's measured in several ways as mentioned in the second paragraph.    &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second, R Square is not a very well accepted measure of a model's robustness and explanatory power for Logit regression.  I would not rely solely on it.  There are other R Square measures earmarked for Logit regression.  They include Pseudo R Square, Aldrich and Nelson R Square, and McFadden R Square.  They all measure the reduction in error as captured by the -2LL calculation between the Baseline model that simply assumes that the probability for all observations is the average probability or the average proportion (if all values are 1 or 0).  Another answer also mentions the Cox &amp;amp; Snell R Square and the Nagelkerke R Square.  I suspect those are other variants on the same theme focusing on the model's error reduction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would re-frame the investigation on the robustness of your model by focusing on the mentioned different R Square measures.  I would also calculate the Chi Square p value for the whole model.  The difference in the -2LL between the baseline model and the model is a Chi Square value.  The number of variables gives you the DF.  The resulting Chi Square p value gives you a measure of statistical significance for the whole model (probability that results were due to randomness). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maybe even more importantly, I would focus on the statistical significance of the independent variables you are using.  The Wald statistics (really another Chi Square test) fits that purpose well.  In this case, a variable regressed coefficient divided by the coefficient standard error gives you the Wald value.  DF is always 1.  And, the resulting p value conveys the statistical significance of this variable.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;With the above, you can then remove or add variables and attempt to improve your model.   &lt;/p&gt;&#xA;" OwnerUserId="1329" LastEditorUserId="1329" LastEditorDisplayName="" LastEditDate="2010-11-06T16:17:06.830" LastActivityDate="2010-11-06T16:17:06.830" CommentCount="3" />
  <row Id="4258" PostTypeId="1" CreationDate="2010-11-05T21:17:42.250" Score="7" ViewCount="379" Body="&lt;p&gt;I would like to automate the choice of burn-in for an MCMC chain, e.g. by removing the first n rows based on a convergence diagnostic. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To what extent can this step be safely automated? Even if I still double check the autocorrelation, mcmc trace, and pdfs, it would be nice to have the choice of burn-in length automated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is general, but it would be great if you could provide specifics for dealing with an R mcmc.object; I am using the rjags and coda packages in R.&lt;/p&gt;&#xA;" OwnerUserId="1381" LastEditorUserId="1381" LastEditorDisplayName="" LastEditDate="2010-11-09T16:23:28.803" LastActivityDate="2010-11-10T00:17:12.523" Title="Can I semi-automate MCMC convergence diagnostics to set the burn-in length?" Tags="&lt;r&gt;&lt;bayesian&gt;&lt;mcmc&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="3" />
  <row Id="4259" PostTypeId="1" CreationDate="2010-11-05T21:17:57.507" Score="6" ViewCount="434" Body="&lt;p&gt;I work in the field of data mining and have had very little formal schooling in statistics. Lately I have been reading a lot of work  that focuses on Bayesian paradigms for learning and mining, which I find very interesting. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is (in several parts), given a problem is there a general framework by which it is possible to construct a statistical model? What are the first things you do when given a dataset of which you'd like to model the underlying process? Are there good books/tutorials out there that explain this process or is it a matter of experience? Is inference in the forefront of your mind when constructing your model or do you first aim to describe the data before you worry about how to use it to compute?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any insight would be greatly appreciated!&#xA;Thanks.&lt;/p&gt;&#xA;" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-11-06T16:43:06.420" LastActivityDate="2010-12-03T16:23:28.423" CommunityOwnedDate="2010-11-06T16:39:22.227" Title="Tips and tricks to get started with statistical modeling?" Tags="&lt;bayesian&gt;&lt;modeling&gt;&lt;books&gt;&lt;eda&gt;" AnswerCount="4" CommentCount="3" FavoriteCount="7" />
  <row Id="4260" PostTypeId="2" ParentId="2343" CreationDate="2010-11-05T21:33:34.133" Score="1" ViewCount="" Body="&lt;p&gt;Let $\tilde{\omega}$ be the equivalence class of a given tree $\omega$ (i.e. $\omega_1 \sim \omega_2$ iif $\omega_1 \in \tilde{\omega}_2$). In your question, you define a probability (say $\tilde{P}$) on the equivalence class and you want a formula to compute rapidly $\tilde{P}(\tilde{\omega}=\tilde{w})$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Obviously the trivial formula you though of when you said &quot;compute the number of trees that are isomorphic to t in some sense&quot; is&#xA;$$\tilde{P}(\tilde{\omega})=\sum_{\omega\in\tilde{\omega}}P(\omega)$$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $P$ is the probability measure without considering the equivalent class (two isomorphic trees can be different). $P$ is easy to compute from the paper but if the trees are big the preceding formula is not really friendly. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a type of equivalence for which you can have a nice formula (such as the one in the paper): it is when your equivalence relation can be restated recursively (according to the tree). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let $\mathcal{S}_k$ be the group of the bijections of $(1,\dots,k)$ on itself.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;Definition 1.&lt;/strong&gt; $w_1$ and $w_2$ are &lt;strong&gt;$\mathcal{S}_k$-equivalent&lt;/strong&gt; if any node has not more than $k$ sons and if&lt;/p&gt;&#xA;  &#xA;  &lt;ol&gt;&#xA;  &lt;li&gt;they have the same root (say $s\in V$ )&lt;/li&gt;&#xA;  &lt;li&gt;&lt;p&gt;the sons of s in $w_1$ and $w_2$ are equal say to $V_1V_2\dots,V_k$ and there exists $\sigma\in \mathcal{S}_k$ such that  &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;a. $V_{\sigma(1)}V_{\sigma(2)}\dots V_{\sigma(k)}=V_1V_2\dots V_k$&#xD;&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;b. $\forall i=1\dots,k$ the sub-tree of $w_1$ with root $V_{\sigma(i)}$ is equivalent with the sub-tree of $w_2$ with root $V_{i}$. &lt;/p&gt;&lt;/li&gt;&#xA;  &lt;/ol&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;To this definition, you can associate another one&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;Definition 2.&lt;/strong&gt; Let us define the weigth $q(s-&amp;gt;V_1V_2\dots V_k )$ of an occurence of a rule $(s-&amp;gt;V_1V_2\dots V_k)$ in a tree $w$ by the number of elements $\sigma$ in $\mathcal{S}$   such that &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;a. $V_{\sigma(1)}V_{\sigma(2)}\dots V_{\sigma(k)}=V_1V_2\dots V_k$  &#xA;    b. for any $i$ such that $\sigma(i)\neq i$, the subtree generated by $V_{\sigma(i)}$ (in $w$) is not equal to the subtree generated by $V_i$. &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;with this second definition, and in the case when the equivalent class is defined through  $\mathcal{S}_k$  -equivalence, the equation 2 p. 1994 in Chi (2004) is still true if you replace $p(s-&amp;gt;n)$ by $q(s-&amp;gt;n)p(s-&amp;gt;n)$ exept for the transitions giving leaves or when . &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your example     &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;* A: 0 -&amp;gt; 1 1 (1.) has weight 2 (identity+permutation of (1,2)) except if the sons are equal&#xA;&#xA;* B: 1 -&amp;gt; 0 0 (0.5) has weight 2 (identity+permutation of (1,2)) except if the sons are equal&#xA;&#xA;* C: 1 -&amp;gt; 0 1 (0.5) has weigh 1 (only identiy)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Hence your tree has probability 0.5 since each transition is applyed once and only the first one does not give leaves. In the two other tree we are in the case of equal sons (this gives weight 1) and the probabilities are 0.25. These definitions are a bit complicated for a tree with 3 cases... I guess it can lead to fast calculation and can be interesting in the case of a very large tree ? &lt;/p&gt;&#xA;" OwnerUserId="223" LastEditorUserId="223" LastEditorDisplayName="" LastEditDate="2010-11-22T16:36:15.373" LastActivityDate="2010-11-22T16:36:15.373" />
  <row Id="4261" PostTypeId="1" AcceptedAnswerId="4263" CreationDate="2010-11-05T22:12:23.317" Score="1" ViewCount="219" Body="&lt;p&gt;This relates to a previous question of mine which didn't gain many responses, perhaps because it wasn't very clear nd well written. I hope this time I will be more accurate and get your much appreciated assistance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am analyzing results of a biological experiment. The results given as a single value ( non-negative integer) per genomic position. I am interested in valleys, or local minimam over this series of values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wish to control the false positives rate and get the significance for each local minima. I can shuffle the raw data which was used to produce the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So what I do is to shuffle the raw data, create the new series of values,search for all local minima and keep their values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, I have something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data_set  local_minima_values&#xA;=============================&#xA;true_data 4 9 1 27 12 0 0 2 5 32 0 1 5 70 2&#xA;sim_1 14 25 94 59 32&#xA;sim_2 52 0 14 74 82 12 54&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Note the number of local minima naturally varies between simulations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, my idea was to calculate an ECDF for each simulation and then combine those ECDFs into a single &quot;average ECDF&quot; which represents the null hypothesis. Then, I can assign a p-value for each local minima from the true data, and see how significant ('surprising') it is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My questions are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Does this make sense?&lt;/li&gt;&#xA;&lt;li&gt;How do I create an average ECDF? I can't just merge the values from all simulation together and get and ECDF for this merged set, since the number of minima found in each simulation differs, and I think all simulations should have the same contribution to the average ECDF, or am I wrong?&lt;/li&gt;&#xA;&lt;li&gt;How should I take the number of simulations (shuffles) into account?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Thanks,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Dave&lt;/p&gt;&#xA;&#xA;&lt;p&gt;p.s. I'm working with R.&lt;/p&gt;&#xA;" OwnerUserId="634" LastActivityDate="2010-11-07T12:25:30.237" Title="Should I use an average ECDF?" Tags="&lt;r&gt;&lt;statistical-significance&gt;&lt;sampling&gt;&lt;permutation&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="1" />
  <row Id="4262" PostTypeId="1" CreationDate="2010-11-05T22:50:38.007" Score="3" ViewCount="168" Body="&lt;p&gt;I have a process which writes statistics from a server system to a file each second in this format:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;label1 label2 label3&#xA;344 666 787&#xA;344 849 344&#xA;939 994 344&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;There are a number of different values which I need graphs for, and each value is added to the bottom of the file each second. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am looking for a nice way to show these numbers graphically, preferably a program which auto-updates each second and shows the different graphs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone recommend such a program for mac? Preferably a free one :)&lt;/p&gt;&#xA;" OwnerUserId="1845" LastActivityDate="2010-11-06T00:36:39.673" Title="Graphing real-time data from a text file" Tags="&lt;data-visualization&gt;&lt;real-time&gt;" AnswerCount="2" FavoriteCount="3" />
  <row Id="4263" PostTypeId="2" ParentId="4261" CreationDate="2010-11-05T23:20:38.963" Score="1" ViewCount="" Body="&lt;p&gt;To average the ECDFs, I'd do something like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;impute_resolution = 1e3&#xA;values_to_impute = seq(&#xA;    min(my_data$true_data)&#xA;    , max(my_data$true_data)&#xA;    , length.out = impute_resoluton&#xA;)&#xA;&#xA;ecdfs = matrix(NA,nrow=length(values_to_impute))&#xA;&#xA;for(i in 1:(ncol(my_data)-1)){ #assumes column 1 is true_data&#xA;    this_ecdf = ecdf(my_data[,i+1])&#xA;    ecdfs[i,] = this_ecdf(values_to_impute)&#xA;}&#xA;&#xA;mean_ecdf = colMeans(ecdfs)&#xA;plot(&#xA;    x = values_to_impute&#xA;    , y = mean_ecdf&#xA;    , type = 'l'&#xA;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="364" LastActivityDate="2010-11-05T23:20:38.963" />
  <row Id="4264" PostTypeId="2" ParentId="4262" CreationDate="2010-11-05T23:43:15.160" Score="6" ViewCount="" Body="&lt;p&gt;I had really good luck with the KDE program &lt;a href=&quot;http://kst.kde.org&quot; rel=&quot;nofollow&quot;&gt;kst&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Kst is the fastest real-time large-dataset viewing and plotting tool available&#xA;  and has basic data analysis functionality.&#xA;  Kst contains many powerful built-in features and is expandable with plugins&#xA;  and extensions. Extensive help is available, both from within Kst and on the&#xA;  web.&#xA;  Kst is licenced under the GPL and is freely available for anyone.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It can handle much higher update frequency than once a second. It also happens to be file-based, but I don't know how much work it is to get this going on OS X.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On Ubuntu and Debian, &lt;code&gt;sudo apt-get install kst&lt;/code&gt; is all it takes.&lt;/p&gt;&#xA;" OwnerUserId="334" LastActivityDate="2010-11-05T23:43:15.160" />
  <row Id="4265" PostTypeId="2" ParentId="4262" CreationDate="2010-11-06T00:36:39.673" Score="2" ViewCount="" Body="&lt;p&gt;RRDTool looks like it might be exactly what you are looking for. I've never tried to run it on a Mac but it looks like someone has some info on that here: &lt;a href=&quot;http://rrdtool.darwinports.com/&quot; rel=&quot;nofollow&quot;&gt;http://rrdtool.darwinports.com/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good luck!&lt;/p&gt;&#xA;" OwnerUserId="118" LastActivityDate="2010-11-06T00:36:39.673" CommentCount="1" />
  <row Id="4266" PostTypeId="2" ParentId="2397" CreationDate="2010-11-06T00:52:44.303" Score="1" ViewCount="" Body="&lt;p&gt;There no inherent problem with the singular covariance here.  Your asymptotic distribution is the singular normal.  See &lt;a href=&quot;http://fedc.wiwi.hu-berlin.de/xplore/tutorials/mvahtmlnode34.html&quot; rel=&quot;nofollow&quot;&gt;http://fedc.wiwi.hu-berlin.de/xplore/tutorials/mvahtmlnode34.html&lt;/a&gt; which gives the density of the singular normal.&lt;/p&gt;&#xA;" OwnerUserId="1860" LastActivityDate="2010-11-06T00:52:44.303" CommentCount="2" />
  <row Id="4267" PostTypeId="1" AcceptedAnswerId="4268" CreationDate="2010-11-06T03:32:17.627" Score="9" ViewCount="468" Body="&lt;p&gt;I recently read Skillicorn's book on matrix decompositions, and was a bit disappointed, as it was targeted to an undergraduate audience. I would like to compile (for myself and others) a short bibliography of essential papers (surveys, but also breakthrough papers) on matrix decompositions. What I have in mind primarily is something on SVD/PCA (and robust/sparse variants), and NNMF, since those are by far the most used. Do you all have any recommendation/suggestion? I am holding off mine not to bias the answers. I would ask to limit each answer to 2-3 papers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S.: I refer to these two decompositions as the most used &lt;em&gt;in data analysis&lt;/em&gt;. Of course QR, Cholesky, LU and polar are very important in numerical analysis. That is not the focus of my question though.&lt;/p&gt;&#xA;" OwnerUserId="30" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-11-06T18:59:40.517" LastActivityDate="2010-12-16T08:38:52.443" CommunityOwnedDate="2010-11-06T13:01:36.173" Title="Essential papers on matrix decompositions" Tags="&lt;matrix-decomposition&gt;&lt;svd&gt;&lt;numerics&gt;" AnswerCount="5" CommentCount="1" FavoriteCount="8" />
  <row Id="4268" PostTypeId="2" ParentId="4267" CreationDate="2010-11-06T07:17:02.640" Score="10" ViewCount="" Body="&lt;p&gt;How do you know that SVD and NMF are by far the most used &lt;a href=&quot;http://en.wikipedia.org/wiki/Matrix_decomposition&quot; rel=&quot;nofollow&quot;&gt;matrix decompositions&lt;/a&gt; rather than LU, Cholesky and QR? My personal favourite 'breakthrough' would have to be the guaranteed rank-revealing QR algorithm,&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Chan, Tony F. &quot;Rank revealing QR factorizations&quot;. &lt;em&gt;Linear Algebra and its Applications&lt;/em&gt; Volumes 88-89, April 1987, Pages 67-82. DOI:&lt;a href=&quot;http://dx.doi.org/10.1016/0024-3795%2887%2990103-0&quot; rel=&quot;nofollow&quot;&gt;10.1016/0024-3795(87)90103-0&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;... a development of the earlier idea of QR with column-pivoting:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Businger, Peter; Golub, Gene H. (1965). Linear least squares solutions by Householder transformations. &lt;em&gt;Numerische Mathematik&lt;/em&gt; Volume 7, Number 3, 269-276, DOI:&lt;a href=&quot;http://dx.doi.org/10.1007/BF01436084&quot; rel=&quot;nofollow&quot;&gt;10.1007/BF01436084&lt;/a&gt;  &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;A (&lt;em&gt;the&lt;/em&gt;?) classic textbook is:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Golub, Gene H.; Van Loan, Charles F. (1996). &lt;em&gt;Matrix Computations&lt;/em&gt; (3rd ed.), Johns Hopkins, ISBN &lt;a href=&quot;http://en.wikipedia.org/wiki/Special%3aBookSources/978-0-8018-5414-9&quot; rel=&quot;nofollow&quot;&gt;978-0-8018-5414-9&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;(i know you didn't ask for textbooks but i can't resist)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt;&#xA;A bit more googling finds a paper whose abstract suggests we could be slightly at cross porpoises. My above text was coming from a 'numerical linear algebra' (NLA) perspective; possibly you're concerned more with an 'applied statistics / psychometrics' (AS/P) perspective? Could you perhaps clarify?&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Lawrence Hubert, Jacqueline Meulman and Willem Heiser. &lt;a href=&quot;http://www.jstor.org/stable/2653377&quot; rel=&quot;nofollow&quot;&gt;Two Purposes for Matrix Factorization: A Historical Appraisal.&lt;/a&gt; &lt;em&gt;SIAM Review&lt;/em&gt; Vol. 42, No. 1 (Mar., 2000), pp. 68-82&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="449" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-11-06T08:39:48.787" LastActivityDate="2010-11-06T08:39:48.787" CommunityOwnedDate="2010-11-06T13:01:36.173" CommentCount="3" />
  <row Id="4269" PostTypeId="2" ParentId="4175" CreationDate="2010-11-06T10:28:50.867" Score="1" ViewCount="" Body="&lt;p&gt;From Sidney Siegel:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;With a large enough sample the binomial distribution tends toward the normal distribution. A rule of thumb is that NPQ must be equal to at least 9.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I believe in this case it is 750 independent observations * 1/5 *4/5 =120. Thus the parametric one-sample t-test is appropriate and the most powerful test. And so I'll just stick with the one-sample permutation t-test (DAAG package) as the resampling method to compare with the table-lookup test.&lt;/p&gt;&#xA;" OwnerUserId="1614" LastActivityDate="2010-11-06T10:28:50.867" CommunityOwnedDate="2010-11-06T10:28:50.867" CommentCount="2" />
  <row Id="4270" PostTypeId="2" ParentId="4044" CreationDate="2010-11-06T11:07:48.890" Score="3" ViewCount="" Body="&lt;p&gt;The most complete survey is provided in &lt;em&gt;&lt;a href=&quot;http://books.google.com/books?id=ziDGGIkhqlMC&amp;amp;pg=PA493&amp;amp;dq=Leandro+Pardo+Complutense+University,+Chapman+Hall+2006&amp;amp;hl=en&amp;amp;ei=Zm_VTOqGI8H6lwepsfn9CA&amp;amp;sa=X&amp;amp;oi=book_result&amp;amp;ct=result&amp;amp;resnum=1&amp;amp;ved=0CC4Q6AEwAA#v=onepage&amp;amp;q&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;Statistical Inference Based on Divergence Measures&lt;/a&gt;&lt;/em&gt; by Leandro Pardo, Complutense University, Chapman Hall 2006.&lt;/p&gt;&#xA;" OwnerUserId="1873" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-11-06T15:10:28.317" LastActivityDate="2010-11-06T15:10:28.317" CommentCount="1" />
  <row Id="4271" PostTypeId="2" ParentId="1645" CreationDate="2010-11-06T14:06:09.300" Score="1" ViewCount="" Body="&lt;p&gt;In fact the Kiefer Salmon test and the Jarque Bera test are critically different as shown  in several places but most recently &lt;a href=&quot;http://www.econ.sinica.edu.tw/upload/file/1118.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; -&lt;em&gt;Moment Tests for Standardized Error Distributions: A Simple Robust Approach&lt;/em&gt; by Yi-Ting Chen. The Kiefer Salmon test by construction is robust in the face of ARCH type error structures unlike the standard Jarque Bera test. The paper by Yi-Ting Chen develops and discusses what I think are likely to be the best tests around at the moment.&lt;/p&gt;&#xA;" OwnerUserId="1873" LastActivityDate="2010-11-06T14:06:09.300" CommentCount="1" />
  <row Id="4272" PostTypeId="1" AcceptedAnswerId="4274" CreationDate="2010-11-06T17:53:05.250" Score="11" ViewCount="408" Body="&lt;p&gt;In what circumstances should one consider using regularization methods (ridge, lasso or least angles regression) instead of OLS?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In case this helps steer the discussion, my main interest is improving predictive accuracy.&lt;/p&gt;&#xA;" OwnerUserId="439" LastActivityDate="2010-11-06T20:29:14.500" Title="When to use regularization methods for regression?" Tags="&lt;regression&gt;&lt;least-squares&gt;&lt;lasso&gt;&lt;ridge-regression&gt;" AnswerCount="2" FavoriteCount="4" />
  <row Id="4273" PostTypeId="2" ParentId="4272" CreationDate="2010-11-06T18:22:43.530" Score="6" ViewCount="" Body="&lt;p&gt;A theoretical justification for the use of ridge regression is that its solution is the posterior mean given a normal prior on the coefficients.  That is, if you care about squared error and you believe in a normal prior, the ridge estimates are optimal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Similarly, the lasso estimate is the posterior mode under a double-exponential prior on your coefficients. This is optimal under a zero-one loss function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In practice, these techniques typically improve predictive accuracy in situations where you have many correlated variables and not a lot of data.  While the OLS estimator is best linear unbiased, it has high variance in these situations.  If you look at the bias-variance trade off, prediction accuracy improves because the small increase in bias is more than offset by the large reduction in variance.&lt;/p&gt;&#xA;" OwnerUserId="1815" LastActivityDate="2010-11-06T18:22:43.530" />
  <row Id="4274" PostTypeId="2" ParentId="4272" CreationDate="2010-11-06T19:09:19.687" Score="9" ViewCount="" Body="&lt;p&gt;Short answer: Whenever you are facing one of these situations: large number of variables or low ratio of no. observations to no. variables (including the $n\ll p$ case), high collinearity, seeking for a sparse solution (i.e., embed feature selection when estimating model parameters), or accounting for variables grouping in high-dimensional data set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ridge regression generally yields better predictions than OLS solution, through a better compromise between bias and variance. Its main drawback is that all predictors are kept in the model, so it is not very interesting if you seek a parcimonious model or want to apply some kind of feature selection. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To achieve sparsity, the lasso is more appropriate but it will not necessarily yield good results in presence of high collinearity (it has been observed that if predictors are highly correlated, the prediction performance of the lasso is dominated by ridge regression). The second problem with L1 penalty is that the lasso solution is not uniquely determined when the number of variables is greater than the number of subjects (this is not the case of ridge regression). The last drawback of lasso is that it tends to select only one variable among a group of predictors with high pairwise correlations. In this case, there are alternative solutions like the &lt;a href=&quot;http://www-stat.stanford.edu/~tibs/ftp/sparse-grlasso.pdf&quot; rel=&quot;nofollow&quot;&gt;group&lt;/a&gt; (i.e., achieve shrinkage on block of covariates, that is some blocks of regression coefficients are exactly zero) or &lt;a href=&quot;http://www.stanford.edu/group/SOL/papers/fused-lasso-JRSSB.pdf&quot; rel=&quot;nofollow&quot;&gt;fused&lt;/a&gt; lasso. The &lt;a href=&quot;http://www-stat.stanford.edu/~tibs/glasso/index.html&quot; rel=&quot;nofollow&quot;&gt;Graphical Lasso&lt;/a&gt; also offers promising features for GGMs (see the R &lt;a href=&quot;http://cran.r-project.org/web/packages/glasso/index.html&quot; rel=&quot;nofollow&quot;&gt;glasso&lt;/a&gt; package).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But, definitely, the &lt;em&gt;elasticnet&lt;/em&gt; criteria, which is a combination of L1 and L2 penalties achieve both shrinkage and automatic variable selection, and it allows to keep $m&gt;p$ variables in the case where $n\ll p$. Following Zou and Hastie (2005), it is defined as the argument that minimizes (over $\beta$)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;L(\lambda_1,\lambda_2,\mathbf{\beta}) = \|Y-X\beta\|^2 + \lambda_2\|\beta\|^2 + \lambda_1\|\beta\|_1&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\|\beta\|^2=\sum_{j=1}^p\beta_j^2$ and $\|\beta\|^1=\sum_{j=1}^p|\beta_j |$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The lasso can be computed with an algorithm based on coordinate descent as described in the recent paper by Friedman and coll., &lt;a href=&quot;http://www.jstatsoft.org/v33/i01/paper&quot; rel=&quot;nofollow&quot;&gt;Regularization Paths for Generalized Linear Models via Coordinate Descent&lt;/a&gt; (JSS, 2010) or the LARS algorithm. In R, the &lt;a href=&quot;http://cran.r-project.org/web/packages/penalized/index.html&quot; rel=&quot;nofollow&quot;&gt;penalized&lt;/a&gt;, &lt;a href=&quot;http://cran.r-project.org/web/packages/lars/index.html&quot; rel=&quot;nofollow&quot;&gt;lars&lt;/a&gt; or &lt;a href=&quot;http://cran.r-project.org/web/packages/biglars/index.html&quot; rel=&quot;nofollow&quot;&gt;biglars&lt;/a&gt;, and &lt;a href=&quot;http://cran.r-project.org/web/packages/glmnet/index.html&quot; rel=&quot;nofollow&quot;&gt;glmnet&lt;/a&gt; packages are useful packages; in Python, there's the &lt;a href=&quot;http://scikit-learn.sourceforge.net/&quot; rel=&quot;nofollow&quot;&gt;scikit.learn&lt;/a&gt; toolkit, with extensive &lt;a href=&quot;http://scikit-learn.sourceforge.net/modules/glm.html#lars-algorithm&quot; rel=&quot;nofollow&quot;&gt;documentation&lt;/a&gt; on the algorithms used to apply all three kind of regularization schemes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for general references, the &lt;a href=&quot;http://www-stat.stanford.edu/~tibs/lasso.html&quot; rel=&quot;nofollow&quot;&gt;Lasso page&lt;/a&gt; contains most of what is needed to get started with lasso regression and technical details about L1-penalty, and this related question features essential references, &lt;a href=&quot;http://stats.stackexchange.com/questions/866/when-should-i-use-lasso-vs-ridge&quot;&gt;When should I use lasso vs ridge?&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-06T20:29:14.500" LastActivityDate="2010-11-06T20:29:14.500" CommentCount="2" />
  <row Id="4275" PostTypeId="2" ParentId="4267" CreationDate="2010-11-06T19:47:38.620" Score="2" ViewCount="" Body="&lt;p&gt;For NNMF, Lee and Seung describe an iterative algorithm which is very simple to implement. Actually they give two similar algorithms, one for minimizing Frobenius norm of residual, the other for minimizing Kullback-Leibler Divergence of the approximation and original matrix.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Daniel Lee, H. Sebastian Seung, &lt;a href=&quot;http://hebb.mit.edu/people/seung/papers/nmfconverge.pdf&quot; rel=&quot;nofollow&quot;&gt;Algorithms for Non-negative Matrix Factorization,&lt;/a&gt; Advances in Neural Information Processing Systems 13: Proceedings of the 2000 Conference. MIT Press. pp. 556–562.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-11-06T19:47:38.620" CommunityOwnedDate="2010-11-06T19:47:38.620" />
  <row Id="4276" PostTypeId="1" CreationDate="2010-11-06T21:12:01.923" Score="6" ViewCount="357" Body="&lt;p&gt;I have a sensor which is capturing accelerometer data as a person walks. What I'm interested in extracting is each signal fragment when a step is taken. The Z-axis is what is used since only one axis is required to detect changes in steps. The images below illustrates a sample Z-axis gait signal (for 400 iterations).&lt;img src=&quot;http://i.imgur.com/nZyPt.jpg&quot; alt=&quot;Z-axis from accelerometer during normal gait cycle&quot;&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The image below illustrates the first-half of the above signal (for 200 iterations). &lt;img src=&quot;http://i.imgur.com/aCP0X.jpg&quot; alt=&quot;Z-axis from accelerometer during normal gait cycle first half&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The subject is initially standing still and then begins walking at ~X=30. Notice how there is an apparent pattern as the user walks. What I'm interested in using Autocorrelation to smooth the Z-axis signal using Matlab to smooth the signal (based on the image below). Unfortunately, I don't have a strong signal processing background, and I have a decent grasp of Matlab. How can I go about achieving smoothing the gait signal so I can extract steps? The literature that I'm using suggests that steps may be extracted by looking at the peaks of the smoothed signal. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/TaxQC.png&quot; alt=&quot;Goal&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other sources have suggested the use of Hidden Markov Models to extract each of the gait cycles, but I thought about a simpler signal processing approach before I consider using something advanced. However, what would be the best strategy if I wanted to pursue this course of action? &lt;/p&gt;&#xA;" OwnerUserId="1224" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-26T00:45:12.177" LastActivityDate="2011-03-26T00:45:12.177" Title="Using autocorrelation to find commonly occurring signal fragments" Tags="&lt;matlab&gt;&lt;autocorrelation&gt;&lt;signal-processing&gt;&lt;markov-process&gt;" AnswerCount="1" CommentCount="4" FavoriteCount="1" />
  <row Id="4277" PostTypeId="2" ParentId="4252" CreationDate="2010-11-06T23:16:32.310" Score="0" ViewCount="" Body="&lt;p&gt;In addition to the above answer, if there are many entries (say n), then first sorting them takes time O(n log n). However, there is a linear-time solution.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Compute the P-quantile L and (1-P)-quantile U. There is a simple (quicksort-like) algorithm for this that runs in expected linear time. There is also a more complicated algorithm that runs in worst case linear time. Both can be found, for example, in:&#xA;Cormen, Leiserson, Rivest, Stein: Introduction to Algortithms. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Scan through all values and add those between L and U. This obviously takes linear time.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If there are ties and the computed quantiles exist several times among the values, we might have added too many or too few values and may need to correct for this appropriately. Since we know how many numbers we added in step 2, and also how many times we have seen L and U, this can be done in constant time.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Divide the total sum by the number of summands.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Note that the above recipe is only worthwhile if n is really large and sorting all of them would be a performance hit, perhaps a few millions.&lt;/p&gt;&#xA;" LastActivityDate="2010-11-06T23:16:32.310" />
  <row Id="4279" PostTypeId="1" CreationDate="2010-11-07T05:23:12.080" Score="4" ViewCount="341" Body="&lt;p&gt;I would like build a web application using R. I am using Windows Vista and have an Apache server. I have tried &lt;a href=&quot;http://rpad.googlecode.com/svn-history/r76/Rpad_homepage/index.html&quot; rel=&quot;nofollow&quot;&gt;Rpad&lt;/a&gt;, but I was not able to correctly configure it. How do I set up Rpad as I am not that well off with PHP and Apache server? Or are there other ways to use R on Apache server?&lt;/p&gt;&#xA;" OwnerUserId="1886" LastEditorUserId="509" LastEditorDisplayName="" LastEditDate="2011-02-03T21:30:15.487" LastActivityDate="2011-02-03T21:30:15.487" Title="How can I integrate R with PHP?" Tags="&lt;r&gt;" AnswerCount="2" FavoriteCount="3" />
  <row Id="4280" PostTypeId="2" ParentId="4279" CreationDate="2010-11-07T05:42:05.297" Score="8" ViewCount="" Body="&lt;p&gt;Here is the easiest way to do it that I found:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This implementation of PHP and R consists of only two files. One written in PHP, and the other an R script. The PHP returns a form which uses the GET method to send a variable N to the server. When the form is submitted, the PHP will then execute an R script from the shell using a combination of the PHP command exec() and the Rscript shell command. This command will pass the variable N to the R script. The R script will then execute and save a histogram plot of N normally distributed values to the filesystem. Finally, when the R script is complete, the PHP will return the HTML tag containing the saved images path. First, &lt;strong&gt;the PHP file&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;lt; ?php&#xA;// poorman.php     &#xA;echo &quot;&amp;lt; form action='poorman.php' method='get'&amp;gt;&quot;;    &#xA;echo &quot;Number values to generate: &amp;lt; input type='text' name='N' /&amp;gt;&quot;;    &#xA;echo &quot;&amp;lt; input type='submit' /&amp;gt;&quot;;    &#xA;echo &quot;&amp;lt; /form&amp;gt;&quot;;     &#xA;if( isset($_GET['N']))    &#xA;{&#xA;  $N = $_GET['N'];&#xA;&#xA;  // execute R script from shell    &#xA;  // this will save a plot at temp.png to the filesystem&#xA;&#xA;  exec(&quot;Rscript my_rscript.R $N&quot;);&#xA;&#xA;  // return image tag&#xA;&#xA;  $nocache = rand();&#xA;&#xA;  echo(&quot;&amp;lt; img src='temp.png?$nocache' /&amp;gt; &quot;);    &#xA;}    &#xA;?&amp;gt;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and &lt;strong&gt;the R script&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# my_rscript.R     &#xA;args &amp;lt;- commandArgs(TRUE)     &#xA;N &amp;lt;- args[1]    &#xA;x &amp;lt;- rnorm(N,0,1)     &#xA;png(filename=&quot;temp.png&quot;, width=500, height=500)&#xA;hist(x, col=&quot;lightblue&quot;)    &#xA;dev.off()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here are some more you are welcome to try:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://danpolant.com/r-integration-with-php/&quot; rel=&quot;nofollow&quot;&gt;http://danpolant.com/r-integration-with-php/&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://steve-chen.net/document/r/r_php&quot; rel=&quot;nofollow&quot;&gt;http://steve-chen.net/document/r/r_php&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="1808" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-07T08:50:19.673" LastActivityDate="2010-11-07T08:50:19.673" CommentCount="1" />
  <row Id="4281" PostTypeId="2" ParentId="3713" CreationDate="2010-11-07T07:12:45.540" Score="2" ViewCount="" Body="&lt;p&gt;You can't know in advance which clustering algorithm would be better, but there are some clues, for example if you want to cluster images there are certain algorithms you should try first like Fuzzy Art, or if you want to group faces you should start with (GGCI) global geometric clustering for image.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyway this does not guarantee the best result, so what I would do is use a program that allows you to methodically run different cluster algorithms, such as weka, RapidMiner or even R (which is non visual), There I will set the program to launch all the different clustering algorithms I can, with all the possible different distances, and if they need parameters, experiment each one with a variety of different parameter values (besides if I do not know the amount of clusters, run each one with a variety of numbers of it). Once you settled the experiment, leave it running, but remember to store somewhere the results of each clustering run. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then compare the results in order to obtain the best resulting clustering. This is tricky because there are several metrics you can compare and not all are provided by every algorithm. For example fuzzy clustering algorithms have different metrics than non-fuzzy, but they can still be compared by considering the fuzzy resulting groups as non-fuzzy,&#xA;I will stick for the comparison to the classic metrics such as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;•   SSE: sum of the square error from the items of each cluster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;•   Inter cluster distance: sum of the square distance between each cluster centroid.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;•   Intra cluster distance for each cluster: sum of the square distance from the items of each cluster to its centroid.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;•   Maximum Radius: largest distance from an instance to its cluster centroid.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;•   Average Radius: sum of the largest distance from an instance to its cluster centroid divided by the number of clusters.&lt;/p&gt;&#xA;" OwnerUserId="1808" LastActivityDate="2010-11-07T07:12:45.540" />
  <row Id="4282" PostTypeId="2" ParentId="2717" CreationDate="2010-11-07T07:18:14.823" Score="0" ViewCount="" Body="&lt;p&gt;Before you try running the clustering on the matrix you can try doing one of the factor analysis techniques, and keep just the most important variables to compute the distance matrix.&#xA;Another thing you can do is to try use fuzzy-methods which tend to work better (at least in my experience) in this kind of cases, try first Cmeans, Fuzzy K-medoids,  and Specially GKCmeans.&lt;/p&gt;&#xA;" OwnerUserId="1808" LastActivityDate="2010-11-07T07:18:14.823" />
  <row Id="4283" PostTypeId="2" ParentId="4279" CreationDate="2010-11-07T08:21:28.343" Score="7" ViewCount="" Body="&lt;p&gt;If you ever think to switch to Linux, the best way would be to use &lt;a href=&quot;http://rapache.net/&quot; rel=&quot;nofollow&quot;&gt;RApache&lt;/a&gt;, which is an Apache module that embeds an R interpreter (&lt;code&gt;mod_R&lt;/code&gt;) in the webserver&lt;/p&gt;&#xA;" OwnerUserId="582" LastActivityDate="2010-11-07T08:21:28.343" />
  <row Id="4284" PostTypeId="1" AcceptedAnswerId="4287" CreationDate="2010-11-07T10:57:29.053" Score="3" ViewCount="262" Body="&lt;p&gt;I am looking for an intuitive explanation of the bias-variance tradeoff, both in general and specifically in the context of linear regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="439" LastEditorUserId="439" LastEditorDisplayName="" LastEditDate="2010-11-07T14:07:09.783" LastActivityDate="2010-11-07T14:07:09.783" Title="Intuitive explanation of the bias-variance tradeoff?" Tags="&lt;regression&gt;&lt;variance&gt;&lt;statistical-bias&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="4285" PostTypeId="2" ParentId="4261" CreationDate="2010-11-07T12:25:30.237" Score="1" ViewCount="" Body="&lt;p&gt;David, before discussing the implementation, I'd discuss question #1. I think the approach &quot;makes sense&quot;, if you mean by this that the approach is intuitive. However, from the perspective of bootstrapping, the approach probably won't work. One well-known failure of bootstrap is in fact in the case of maxima and minima. The intuition behind this is that the estimators for these statistics have very large variance: a nonparametric approach based on the empirical distribution doesn't help estimate these statistics. Most approaches are parametric. If interested I can suggest references on the latter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A reference on the failure of boostrap for extremal statistics is Efron and Tibshirani's very own &quot;An Introduction to the Bootstrap&quot;. Check page 81.&lt;/p&gt;&#xA;" OwnerUserId="30" LastActivityDate="2010-11-07T12:25:30.237" />
  <row Id="4286" PostTypeId="1" AcceptedAnswerId="4289" CreationDate="2010-11-07T12:54:08.700" Score="2" ViewCount="135" Body="&lt;p&gt;I have the following setting. I have &lt;em&gt;n&lt;/em&gt; Hermitian Positive Semidefinite (HPSD) matrices, and a metric induced by a matrix norm. I am primarily interested in the Frobenius norm and the operator norm. I want to extract the principal &quot;principal component&quot; for this set of observations, i.e., a 1-dimensional subspace in HPSD such that that the sum of squares of the minimum distances between the matrices with this subspace is minimized. I am not interested in second, third etc components, since they are not even well defined in this space, since I have not defined a scalar product for simplicity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the case of the Frobenius norm, the problem can be reduced to traditional PCA, by using as input vectors the stacked versions of the input matrices. But in the case of the operator norms, I can't find a strategy to attack the problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Has anyone seen this specific problem before? Recommendations and references are highly appreciated.&lt;/li&gt;&#xA;&lt;li&gt;Has anyone dealt with computation of PCA in the case of non-euclidean distances? &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="30" LastActivityDate="2010-11-07T14:19:31.577" Title="Principal Component Analysis among matrices" Tags="&lt;pca&gt;&lt;dimensionality-reduction&gt;" AnswerCount="1" FavoriteCount="2" />
  <row Id="4287" PostTypeId="2" ParentId="4284" CreationDate="2010-11-07T13:21:12.523" Score="6" ViewCount="" Body="&lt;p&gt;Imagine some 2D data--let's say height versus weight for students at a high school--plotted on a pair of axes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now suppose you fit a straight line through it. This line, which of course represents a set of predicted values, has zero statistical variance. But the bias is (probably) high--i.e., it doesn't fit the data very well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Next, suppose you model the data with a high-degree polynomial spline. You're not satisfied with the fit, so you increase the polynomial degree until the fit improves (and it will, to arbitrary precision, in fact). Now you have a situation with bias that tends to zero, but the variance is very high.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that the bias-variance trade-off doesn't describe a proportional relationship--i.e., if you plot bias versus variance you won't necessarily see a straight line through the origin with slope -1. In the polynomial spline example above, reducing the degree almost certainly increases the variance much less than it decreases the bias.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The bias-variance tradeoff is also embedded in the sum-of-squares error function. Below, I have rewritten (but not altered) the usual form of this equation to emphasize this: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/A1JDU.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the right-hand side, there are three terms: the first of these is just the irreducible error (the variance in the data itself); this is beyond our control so ignore it. The &lt;em&gt;second&lt;/em&gt; term is the &lt;strong&gt;square of the bias&lt;/strong&gt;; and the &lt;em&gt;third&lt;/em&gt; is the &lt;strong&gt;variance&lt;/strong&gt;. It's easy to see that as one goes up the other goes down--they can't both vary together in the same direction. Put another way, you can think of least-squares regression as (implicitly) finding the optimal combination of bias and variance from among candidate models.&lt;/p&gt;&#xA;" OwnerUserId="438" LastActivityDate="2010-11-07T13:21:12.523" CommentCount="3" />
  <row Id="4288" PostTypeId="1" AcceptedAnswerId="4546" CreationDate="2010-11-07T13:51:20.053" Score="2" ViewCount="100" Body="&lt;p&gt;When trying to estimate the number of sampling units with an attribute, is there a good algebraic way to aggregate over propensity scores for that attribute which each have their own error? For example, when the propensity scores may be calculated with varying amounts of information from each sampling unit and each has its own standard error on that propensity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or they could be expected values of beta distributions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the latter case I know I can simulate beta-bernoulli outcomes from each sampling unit and add up the results many times; but is there a consistent estimator of the result of this difficult to scale process? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In short, how do people aggregate propensity scores of varying reliabilities?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suppose I worded it poorly; the data I have is all either binary or categorical and each observation is accompanied by the probability it was observed correctly. So suppose I have 5 persons; 3 of which had a value of 1 for an attribute, 2 of which had a value of 0 for that attribute, each of which had a probability .8,.81,.82.,.83,.84 respectively of being observed correctly. What is the expected value of p(having that attribute)?&lt;/p&gt;&#xA;" OwnerUserId="1893" LastEditorUserId="1893" LastEditorDisplayName="" LastEditDate="2010-11-16T17:26:48.193" LastActivityDate="2011-02-15T06:22:56.137" Title="Aggregation of propensity scores with varying reliability" Tags="&lt;distributions&gt;&lt;probability&gt;&lt;beta-binomial&gt;&lt;propensity-scores&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="4289" PostTypeId="2" ParentId="4286" CreationDate="2010-11-07T14:19:31.577" Score="5" ViewCount="" Body="&lt;p&gt;I don't know if this is exactly what you are looking for (esp. I don't know how large is $n$ and what you intend to do with these results), however I have successfully used &lt;a href=&quot;http://pbil.univ-lyon1.fr/R/articles/arti113.pdf&quot; rel=&quot;nofollow&quot;&gt;coinertia analysis&lt;/a&gt; when I was working with two data sets (same observations in rows), and for more than two data sets there are K-table methods, as implemented in the &lt;a href=&quot;http://cran.r-project.org/web/packages/ade4/index.html&quot; rel=&quot;nofollow&quot;&gt;ade4&lt;/a&gt; R package. &lt;a href=&quot;http://pbil.univ-lyon1.fr/R/pdf/course7.pdf&quot; rel=&quot;nofollow&quot;&gt;An introduction to K-table analyses&lt;/a&gt; outlines the main principles. When the objective is to link two or more Tables, &lt;a href=&quot;http://en.wikipedia.org/wiki/Generalized_canonical_correlation&quot; rel=&quot;nofollow&quot;&gt;Generalized Canonical Correlation Analysis&lt;/a&gt; is also an option.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems to me that you can choose non-euclidean metric, provided it has some meaning for the data at hand and the interpretation of the factorial space. You can see an example with the use of &lt;code&gt;kdist()&lt;/code&gt; in &lt;a href=&quot;http://cran.r-project.org/web/packages/ade4/index.html&quot; rel=&quot;nofollow&quot;&gt;ade4&lt;/a&gt; for applying an PCA on different distance matrices. Jollife's book on &lt;em&gt;Principal component analysis&lt;/em&gt; should provide additional hints about this (but I didn't check). There's also all the work made in the spirit of Gifi on non-linear methods (in R, a lot of packages have been developed by Jan de Leeuw, see the &lt;a href=&quot;https://r-forge.r-project.org/projects/psychor/&quot; rel=&quot;nofollow&quot;&gt;PsychoR&lt;/a&gt; project).&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-11-07T14:19:31.577" CommentCount="1" />
  <row Id="4290" PostTypeId="2" ParentId="2419" CreationDate="2010-11-07T14:57:07.923" Score="3" ViewCount="" Body="&lt;p&gt;You can use R decision tree library using Rpy(http://rpy.sourceforge.net/). Also check the article &quot;building decision trees using python&quot;(http://onlamp.com/pub/a/python/2...).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;there is also&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://opencv.willowgarage.com/documentation/index.html&quot; rel=&quot;nofollow&quot;&gt;http://opencv.willowgarage.com/documentation/index.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://research.engineering.wustl.edu/~amohan/&quot; rel=&quot;nofollow&quot;&gt;http://research.engineering.wustl.edu/~amohan/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1808" LastActivityDate="2010-11-07T14:57:07.923" />
  <row Id="4291" PostTypeId="2" ParentId="2419" CreationDate="2010-11-07T15:04:46.763" Score="1" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://jboost.sourceforge.net/&quot; rel=&quot;nofollow&quot;&gt;JBoost&lt;/a&gt; is an awesome library. It is definitely not written in Python, however It is somewhat language agnostic, because it can be executed from the command line and such so it can be &quot;driven&quot; from Python. I've used it in the past and liked it a lot, particularly the visualization stuff.&lt;/p&gt;&#xA;" OwnerUserId="1540" LastActivityDate="2010-11-07T15:04:46.763" />
  <row Id="4292" PostTypeId="1" AcceptedAnswerId="4293" CreationDate="2010-11-07T17:28:27.877" Score="4" ViewCount="178" Body="&lt;p&gt;Here is a look at my data. We asked the same respondents (n=~400) to provide us with their current and future consumption as a proportion of total expenditure. Plotted here are the mean proportions of total expenditure for each category for &quot;Now&quot; and &quot;Later&quot;, respectively current and future.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I'm looking for is a statistical method that I can use to test if the other categories are increasing while the largest category is decreasing. The graph shows this, to some degree, but I would like to make sure that it is statistically valid.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/o3g4r.jpg&quot; alt=&quot;Example&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="776" LastEditorUserId="776" LastEditorDisplayName="" LastEditDate="2010-11-07T18:05:08.847" LastActivityDate="2010-11-07T18:44:42.407" Title="How to test if change is significant across multiple categories?" Tags="&lt;hypothesis-testing&gt;&lt;statistical-significance&gt;" AnswerCount="2" CommentCount="6" FavoriteCount="1" />
  <row Id="4293" PostTypeId="2" ParentId="4292" CreationDate="2010-11-07T17:50:01.673" Score="6" ViewCount="" Body="&lt;p&gt;There are subtle issues involving the difference between designed comparisons and &lt;em&gt;post-hoc&lt;/em&gt; comparisons, of which this likely is an example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;If,&lt;/em&gt; before collecting the data, you anticipated this kind of pattern, you could employ a simple nonparametric test.  The null hypothesis would be that all changes are due to chance with the alternative being that a specified category was increasing and the other eight categories were decreasing.  Under the null, positive changes have a 50% chance of occurring, implying the chance of the alternative is $(0.50)^8(1 - 0.50)^1$ = $0.002$: highly significant evidence for the alternative.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The analysis for a &lt;em&gt;post-hoc&lt;/em&gt; observation is difficult because we can't even get started with describing the situation.  Exactly what kind of pattern would you happen to notice and considered worthy of testing?  So many are possible, with no accurate description available, that all we can say (from experience) is that (a) &lt;em&gt;it is highly likely that any interested investigator would notice&lt;/em&gt; &lt;strong&gt;some&lt;/strong&gt; &lt;em&gt;pattern in the data&lt;/em&gt; and (b) a &lt;em&gt;post-hoc&lt;/em&gt; hypothesis test could be constructed to &quot;demonstrate&quot; the &quot;high significance&quot; of that pattern, exactly as I did above.  For these reasons, applying hypothesis tests after the fact to support claims of &quot;statistical validity&quot; for exploratory results is frowned upon.  (Among statisticians, who should know better, it is called &quot;&lt;a href=&quot;http://en.wikipedia.org/wiki/Data-snooping_bias&quot; rel=&quot;nofollow&quot;&gt;data snooping&lt;/a&gt;&quot; or worse.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One way out is to conduct your analysis with &lt;em&gt;c.&lt;/em&gt; half the data, randomly selected.  Look for any patterns you like.  Construct an appropriate suite of hypothesis tests for those patterns and then &lt;em&gt;apply them to the held-out data only.&lt;/em&gt;  This is in the spirit of the scientific requirement for &lt;strong&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Replication_%28scientific_method%29&quot; rel=&quot;nofollow&quot;&gt;replication&lt;/a&gt;&lt;/strong&gt;.  If you don't do this, then you would be obliged to repeat your experiment to confirm whatever you're seeing in the data you currently have.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-11-07T17:50:01.673" CommentCount="4" />
  <row Id="4294" PostTypeId="1" AcceptedAnswerId="4299" CreationDate="2010-11-07T18:32:20.763" Score="2" ViewCount="255" Body="&lt;p&gt;I'm taking a graduate course in regression analysis and I'm suck on a particular homework question that &lt;em&gt;should&lt;/em&gt; be very simple to me!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have the following model:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;E(y) = B0 + B1x1 + B2x2 + B3x3 + B4x1x3 + B5x2x3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;x3&lt;/code&gt; is coded as &lt;code&gt;1&lt;/code&gt; if &quot;smoker&quot; and &lt;code&gt;0&lt;/code&gt; if &quot;non-smoker&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore the regression equations are:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x3 = 1: E(y) = (B3 + B0) + (B1 + B4)x1 + (B2 + B5)x2&#xA;x3 = 0: E(y*) = B0 + B1x1 + B2x2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now I know how to test for parallelism if &lt;code&gt;x2&lt;/code&gt; is absent in the models:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;H0: B4 = 0&#xA;H1: B4 != 0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But I'm lost as to what to do with the inclusion of the &lt;code&gt;x2&lt;/code&gt; variable. Parallelism is obviously testing for slope, but I'm not sure where to find the &quot;slope&quot; coefficient. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was thinking about using an &lt;code&gt;F-Test&lt;/code&gt; but then I realized I don't actually want to test the &lt;em&gt;whole&lt;/em&gt; model, just the parallelism.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could someone please point me into the right direction? Even hints would be sufficient. &lt;/p&gt;&#xA;" OwnerUserId="1894" LastEditorUserId="1894" LastEditorDisplayName="" LastEditDate="2010-11-07T20:21:28.377" LastActivityDate="2010-11-07T21:35:04.993" Title="How to test for parallelism for two linear models?" Tags="&lt;hypothesis-testing&gt;&lt;regression&gt;&lt;homework&gt;" AnswerCount="1" />
  <row Id="4295" PostTypeId="2" ParentId="4292" CreationDate="2010-11-07T18:44:42.407" Score="1" ViewCount="" Body="&lt;p&gt;Given the additional information you've subsequently posted I'm not sure any statistical test is going to be that informative.  If you had a strong prediction of a pattern such as this or similar, this is such a low probability event that you're pretty much set just getting these data.  With an N of 400 almost any tests will most definitely be significant.  Some good descriptive stats like confidence intervals would be very useful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would suggest that caution be made in your description of the downward trend being remotely meaningful.  It's such a tiny amount that, yeah, if your N is big enough it will be significant.  But is that tiny drop in percentage meaningful?  I think the more meaningful statement is that it's not an increase like the others and that it is staying roughly flat.  Don't try to change the story of really small effects with statistical tests.&lt;/p&gt;&#xA;" OwnerUserId="601" LastActivityDate="2010-11-07T18:44:42.407" CommentCount="7" />
  <row Id="4296" PostTypeId="1" CreationDate="2010-11-07T20:08:22.533" Score="4" ViewCount="841" Body="&lt;p&gt;Hi,does anybody have a good example for Time Series Forecasting/smoothening using Kalman Filter in R?&lt;/p&gt;&#xA;" OwnerUserId="1896" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-11-07T20:45:37.680" LastActivityDate="2010-11-08T09:23:42.357" Title="R code for time series forecasting using Kalman filter" Tags="&lt;r&gt;&lt;time-series&gt;&lt;kalman-filter&gt;" AnswerCount="3" FavoriteCount="3" />
  <row Id="4297" PostTypeId="2" ParentId="4296" CreationDate="2010-11-07T20:22:24.477" Score="12" ViewCount="" Body="&lt;p&gt;Have you looked at &lt;a href=&quot;http://cran.r-project.org/web/views/TimeSeries.html&quot; rel=&quot;nofollow&quot;&gt;Time Series&lt;/a&gt; Task View at CRAN?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It lists several entries for packages covering Kalman filtering:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://cran.r-project.org/package=dlm&quot; rel=&quot;nofollow&quot;&gt;dlm&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://cran.r-project.org/package=FKF&quot; rel=&quot;nofollow&quot;&gt;FKF&lt;/a&gt; &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://cran.r-project.org/package=KFAS&quot; rel=&quot;nofollow&quot;&gt;KFAS&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;There are also other packages&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://cran.r-project.org/package=robKalman&quot; rel=&quot;nofollow&quot;&gt;robKalman&lt;/a&gt; &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://cran.r-project.org/package=SSPIR&quot; rel=&quot;nofollow&quot;&gt;SSPIR&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;and possibly more as this is a pretty common techique for time series estimation.&lt;/p&gt;&#xA;" OwnerUserId="334" LastEditorUserId="334" LastEditorDisplayName="" LastEditDate="2010-11-07T20:36:34.367" LastActivityDate="2010-11-07T20:36:34.367" />
  <row Id="4298" PostTypeId="1" AcceptedAnswerId="4314" CreationDate="2010-11-07T21:23:52.657" Score="3" ViewCount="216" Body="&lt;p&gt;This question is a follow up to my earlier question &lt;a href=&quot;http://stats.stackexchange.com/questions/4220/probability-distribution-over-1-is-ok&quot;&gt;here&lt;/a&gt; &#xA;and is also related, in intent, to &lt;a href=&quot;http://stats.stackexchange.com/questions/3316/statistical-similarity-of-time-series&quot;&gt;this question&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On &lt;a href=&quot;http://en.wikipedia.org/wiki/Naive_Bayes_classifier&quot; rel=&quot;nofollow&quot;&gt;this wiki page&lt;/a&gt; &lt;em&gt;probability density values&lt;/em&gt; from an assumed normal distribution for the training set are used to calculate a Bayesian posterior rather than actual probability values. However, if a training set is not normally distributed would it be equally as valid to use a &lt;em&gt;density value&lt;/em&gt; taken from the &lt;em&gt;kernel density estimate&lt;/em&gt; of the training set to calculate a Bayesian posterior? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In its intended application this &lt;em&gt;kernel density estimate&lt;/em&gt; would be taken from a theoretically ideal empirical data set generated by MC techniques. &lt;/p&gt;&#xA;" OwnerUserId="226" LastActivityDate="2010-11-08T14:55:21.340" Title="Use of kernel density estimate in Naive Bayes Classifier?" Tags="&lt;bayesian&gt;&lt;kde&gt;" AnswerCount="1" />
  <row Id="4299" PostTypeId="2" ParentId="4294" CreationDate="2010-11-07T21:35:04.993" Score="3" ViewCount="" Body="&lt;p&gt;I am bit unsure what exactly you mean by 'parallelism' but perhaps you mean that you want to test if the interaction terms are significant or not in which case you would do a joint test that B4=0 and B5=0. &lt;/p&gt;&#xA;" LastActivityDate="2010-11-07T21:35:04.993" CommentCount="2" />
  <row Id="4300" PostTypeId="2" ParentId="4296" CreationDate="2010-11-07T22:25:28.340" Score="0" ViewCount="" Body="&lt;p&gt;For good examples look at the &lt;a href=&quot;http://cran.r-project.org/web/packages/dlm/vignettes/dlm.pdf&quot; rel=&quot;nofollow&quot;&gt;dlm vignette&lt;/a&gt; I would avoid all the other packages if you don't have a clear idea of what you want to do and how.&lt;/p&gt;&#xA;" OwnerUserId="300" LastActivityDate="2010-11-07T22:25:28.340" />
  <row Id="4301" PostTypeId="2" ParentId="726" CreationDate="2010-11-07T22:41:34.370" Score="3" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Everybody believes in the exponential law of errors [&lt;em&gt;i.e.&lt;/em&gt;, the Normal distribution]: the experimenters, because they think it can be proved by mathematics; and the mathematicians, because they believe it has been established by observation.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Whittaker, E. T. and Robinson, G. &quot;Normal Frequency Distribution.&quot; Ch. 8 in The Calculus of Observations: &lt;a href=&quot;http://books.google.com/books?id=PlEjcAAACAAJ&amp;amp;dq=The+Calculus+of+Observations%3a+An+Introduction+to+Numerical+Analysis&amp;amp;hl=en&amp;amp;ei=5CrXTLPsMoGBlAfR2fH8CA&amp;amp;sa=X&amp;amp;oi=book_result&amp;amp;ct=result&amp;amp;resnum=1&amp;amp;ved=0CDEQ6AEwAA&quot; rel=&quot;nofollow&quot;&gt;A Treatise on Numerical Mathematics&lt;/a&gt;, 4th ed. New York: Dover, pp. 164-208, 1967. p. 179.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Quoted at &lt;a href=&quot;http://mathworld.wolfram.com/NormalDistribution.html&quot; rel=&quot;nofollow&quot;&gt;Mathworld.com&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-01-19T04:42:18.593" LastActivityDate="2011-01-19T04:42:18.593" CommunityOwnedDate="2010-11-07T22:41:34.370" />
  <row Id="4302" PostTypeId="2" ParentId="726" CreationDate="2010-11-07T23:16:13.807" Score="5" ViewCount="" Body="&lt;p&gt;Statistics - A subject which most statisticians find difficult but in which nearly all physicians are expert. &lt;/p&gt;&#xA;" LastActivityDate="2010-11-07T23:16:13.807" CommunityOwnedDate="2010-11-07T23:16:13.807" CommentCount="2" />
  <row Id="4303" PostTypeId="1" CreationDate="2010-11-08T09:03:46.017" Score="2" ViewCount="712" Body="&lt;p&gt;I wish to analyze the following :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Predictor Variable (IV): Satisfaction of sexual needs as important (4 items scale and respond based on 4 point likert scale. Sum up to get the item score.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Response Variable (DV): Condom usagae (2 options : never or sometime)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Questions:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Should I use binary logistic or multinomial logistic? (some people tell me to use multinomial logistic but a book said to only use it when the DV has more than two levels, and  my DV only has 2 levels - never or sometime).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;How can I use SPSS to analyse this? I need step by step help. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The SPSS dialog box for logistic regression has three boxes:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Dependent :  I put in Condom Usage &lt;/li&gt;&#xA;&lt;li&gt;Factor(s): (I am not sure should whether I put Satisfaction of sexual needs as important here?)&lt;/li&gt;&#xA;&lt;li&gt;Covariate : (I am not sure whether I should put Satisfaction of sexual needs as important here?)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I am very sorry. Maybe my question sounds silly but i really need help, as i am beginner. I couldn't get any tutor within my town. &lt;/p&gt;&#xA;" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-11-08T12:14:02.673" LastActivityDate="2010-11-08T12:14:02.673" Title="Binary or Multinomial Logistic Regression?" Tags="&lt;logistic-regression&gt;&lt;spss&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="4304" PostTypeId="2" ParentId="4296" CreationDate="2010-11-08T09:13:11.370" Score="5" ViewCount="" Body="&lt;p&gt;In addition to the packages mentioned in other answers, you may want to look at &#xA;package &lt;strong&gt;&lt;a href=&quot;http://cran.r-project.org/package=forecast&quot; rel=&quot;nofollow&quot;&gt;forecast&lt;/a&gt;&lt;/strong&gt; which deals with a particular class of models cast in state-space form and package &lt;strong&gt;&lt;a href=&quot;http://cran.r-project.org/package=MARSS&quot; rel=&quot;nofollow&quot;&gt;MARSS&lt;/a&gt;&lt;/strong&gt; with examples and applications in biology (see in particular the well-writen manual, Chap. 5).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For general applications, I aggree, though, with the previous answers, with &#xA;&lt;strong&gt;&lt;a href=&quot;http://cran.r-project.org/package=dlm&quot; rel=&quot;nofollow&quot;&gt;dlm&lt;/a&gt;&lt;/strong&gt; being in my view a versatile and powerful package (well described in the book &lt;em&gt;Dynamic Linear Models in R&lt;/em&gt;, by Petris et al.), &lt;strong&gt;&lt;a href=&quot;http://cran.r-project.org/package=KFAS&quot; rel=&quot;nofollow&quot;&gt;KFAS&lt;/a&gt;&lt;/strong&gt; offering routines which implement most of the algorithms described in the excellent &lt;a href=&quot;http://books.google.com/books?id=XRCu5iSz_HwC&amp;amp;dq=Time+Series+Analysis+by+State+Space+Methods+Durbin&amp;amp;source=bl&amp;amp;ots=hPqyNTb2x1&amp;amp;sig=VM5XYGxIp7xxYvsLiAb_reJLtjM&amp;amp;hl=es&amp;amp;ei=lL3XTLvTMc6D5Abm1ejyBw&amp;amp;sa=X&amp;amp;oi=book_result&amp;amp;ct=result&amp;amp;resnum=1&amp;amp;ved=0CBwQ6AEwAA&quot; rel=&quot;nofollow&quot;&gt;Time Series Analysis by State Space Methods&lt;/a&gt; and &lt;strong&gt;&lt;a href=&quot;http://cran.r-project.org/package=FKF&quot; rel=&quot;nofollow&quot;&gt;FKF&lt;/a&gt;&lt;/strong&gt; with limited facilities and no examples, but being the fastest.&lt;/p&gt;&#xA;" OwnerUserId="892" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-11-08T09:23:42.357" LastActivityDate="2010-11-08T09:23:42.357" CommentCount="1" />
  <row Id="4305" PostTypeId="1" AcceptedAnswerId="4309" CreationDate="2010-11-08T09:40:03.757" Score="4" ViewCount="284" Body="&lt;p&gt;Is there a recommendation on the number of times that an experiment should be replicated? As many of you know, is not always possible to make many replicas. What would be the recommended minimum? Is there some references to support it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my particular case (animal reproduction), for reasons of seasonality, I can only replicate experiments 3 times and I have sometimes been criticized for the low number of replicates performed. Could be considered appropriate to assess the effect that a parameter measured 3 times in the same individuals have on the performance of these individuals?&lt;/p&gt;&#xA;" OwnerUserId="221" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-11-08T12:56:21.207" LastActivityDate="2010-11-08T12:56:21.207" Title="At least, how many times an experiment should be replicated?" Tags="&lt;estimation&gt;&lt;sample-size&gt;&lt;experiment-design&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="4306" PostTypeId="2" ParentId="2419" CreationDate="2010-11-08T10:30:23.147" Score="1" ViewCount="" Body="&lt;p&gt;I have experienced the similar situation with you, I find Orange is hard to tune (maybe it is my problem). In the end, I used Peter Norivig's code for his famous book, in there he provided a well written code framework for tree, all you need is to add boosting in it. This way, you can code anything you like. &lt;/p&gt;&#xA;" OwnerUserId="806" LastActivityDate="2010-11-08T10:30:23.147" />
  <row Id="4308" PostTypeId="1" CreationDate="2010-11-08T12:01:22.877" Score="4" ViewCount="168" Body="&lt;p&gt;I was just wondering-- is it possible/practical to apply bayes' theorem without an analytical expression for the prior, only samples? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, say you have sufficient draws from a posterior distribution from a previous experiment via MCMC methods, and you'd like to use that posterior as the prior for a new one.  You have an analytical expression for the likelihood as before, but now only samples from the (new) prior.  How would you proceed?&lt;/p&gt;&#xA;" OwnerUserId="1795" LastActivityDate="2010-11-09T09:34:18.210" Title="Is it possible to apply Bayes Theorem with only samples from the prior?" Tags="&lt;bayesian&gt;&lt;mcmc&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="4309" PostTypeId="2" ParentId="4305" CreationDate="2010-11-08T12:07:50.973" Score="5" ViewCount="" Body="&lt;p&gt;There is no such thing as a minimum (or maximum) sample size rule. It depends on the size of the effect you are trying to measure. Your description of the experiment is slightly unclear, but consider this example, if you measured blood pressure in three different people, what could you conclude about blood pressure in the population? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Likewise, if you are conducting a clinical trial and it's clear (using statistical arguments) that one of the treatments is harmful, should you continue?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another comment. In experiments concerning animals/people I would consider it unethical to conduct an experiment that has &lt;strong&gt;no&lt;/strong&gt; chance of success due to low sample sizes. If in doubt, find a local friendly statistician. Most institutions have them somewhere. &lt;/p&gt;&#xA;" OwnerUserId="8" LastActivityDate="2010-11-08T12:07:50.973" CommentCount="5" />
  <row Id="4310" PostTypeId="2" ParentId="4303" CreationDate="2010-11-08T12:09:59.670" Score="5" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Binary or Multinomial:&lt;/strong&gt;&#xA;Perhaps the following rules will simplify the choice:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;If you have only two levels to your dependent variable then you use binary logistic regression. &lt;/li&gt;&#xA;&lt;li&gt;If you have three or more unordered levels to your dependent variable, then you'd look at multinomial logistic regression. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;A few points:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Satisfaction with sexual needs ranges from 4 to 16  (i.e., 13 distinct values). Such a variable is typically treated as a metric  predictor (i.e., in the covariate box in SPSS).&lt;/li&gt;&#xA;&lt;li&gt;Possibly your dependent variable is causing some confusion because as you phrase it, it is not a standard dichotomy. It sounds like a frequency item that could range from never, to occasionally, to sometimes, to often, to always, etc. However, I'm guessing that either you have explicitly collapsed categories or you have required the respondent to implicitly collapse the categories down to a binary choice. As a side note, if you did have an ordered set of frequency categories, then you might want to use a model that incorporated that order.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;SPSS:&lt;/strong&gt;&#xA;I posted some &lt;a href=&quot;http://jeromyanglim.blogspot.com/2009/09/logistic-regression-resources-in-spss.html&quot; rel=&quot;nofollow&quot;&gt;links to tutorials in SPSS and R&lt;/a&gt; for conducting binary logistic regression.&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-11-08T12:09:59.670" />
  <row Id="4311" PostTypeId="2" ParentId="4308" CreationDate="2010-11-08T12:14:57.690" Score="6" ViewCount="" Body="&lt;p&gt;The short answer is yes. Have a look at &lt;a href=&quot;http://en.wikipedia.org/wiki/Particle_filter&quot; rel=&quot;nofollow&quot;&gt;sequential MCMC/ particle filters&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Essentially, your prior consists of a bunch of particles ($M$). So to sample from your prior, just select a particle with probability $1/M$. Since each particle has equal probability of being chosen, this term disappears in the M-H ratio.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A big problem with particle filters is particle degeneracy. This happens because you are trying to represent a continuous distribution with discrete particles - there's no such thing as a free lunch!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Clarification for Srikant Vadali&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The question as I read it is: I have output, i.e. posterior from a MCMC scheme. I want to use this posterior as a prior for a new data set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This (probably) means that you have a discrete representation of a continuous distribution, i.e. a particle representation. So rather than doing a random walk on a continous distribution (say), you need to pick values from your prior, i.e. you pick a particle. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://rsif.royalsocietypublishing.org/content/6/31/187.abstract&quot; rel=&quot;nofollow&quot;&gt;Toni et al.&lt;/a&gt;, use this idea with ABC.&lt;/p&gt;&#xA;" OwnerUserId="8" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-11-09T09:34:18.210" LastActivityDate="2010-11-09T09:34:18.210" CommentCount="6" />
  <row Id="4312" PostTypeId="1" AcceptedAnswerId="4321" CreationDate="2010-11-08T13:03:59.743" Score="0" ViewCount="384" Body="&lt;p&gt;This is related to another &lt;a href=&quot;http://stats.stackexchange.com/questions/4175/resampling-binomial-z-and-t-test-help-with-real-data&quot;&gt;question&lt;/a&gt; I asked recently. To recap:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[&lt;em&gt;I had 30 people call a number and then roll a 5 sided die. If the call matches the subsequent face then the trial is a hit, else it is a miss. Each subject completes 25 trials (rolls) and thus, each participant has a score out of 25. Since the die is a virtual one, it cannot be biased. Before the experiment was conducted I was going to compare the subjects score with a one-sample t-test (compared to mu of 5). However I was pointed towards the more powerful z-test, which is appropriate because we know the population parameters for the null hypothesis: that everyone should score at chance. Since NPQ means the binomial approximates to the normal or Gaussian distribution, we can use a parametric test. So I could just forget about it all and go back to the t-test I planned to use, but it seems to me that although the z-test is not often used in real research it is appropriate here. That was the conclusion from my previous question. Now I am trying to understand how to use resampling methods (either permutation or bootstrap) to compliment my parametric analysis.&lt;/em&gt;]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Okay. I am trying to program a one-sample permutation &lt;a href=&quot;http://statistic-on-air.blogspot.com/2009/07/one-sample-z-test.html&quot; rel=&quot;nofollow&quot;&gt;z-test&lt;/a&gt;, using the &lt;a href=&quot;http://pbil.univ-lyon1.fr/library/DAAG/DESCRIPTION&quot; rel=&quot;nofollow&quot;&gt;DAAG&lt;/a&gt; package &lt;a href=&quot;http://pbil.univ-lyon1.fr/library/DAAG/html/onet.permutation.html&quot; rel=&quot;nofollow&quot;&gt;onet.permutation&lt;/a&gt; as inspiration. This is as far as I've got:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;perm.z.test = function(x, mu, var, n, prob, nsim){&#xA;  nx &amp;lt;- length(x)&#xA;  mx &amp;lt;- mean(x)&#xA;  z &amp;lt;- array(, nsim)&#xA;  for (i in 1:nsim) {&#xA;    mn &amp;lt;- rbinom(nx*1, size=n, prob=prob)&#xA;    zeta = (mean(mn) - mu) / (sqrt(var/nx))&#xA;    z[i] &amp;lt;- zeta&#xA;  }&#xA;  pval &amp;lt;- (sum(z &amp;gt;= abs(mx)) + sum(z &amp;lt;= -abs(mx)))/nsim&#xA;  print(signif(pval, 3))&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Where: &lt;code&gt;x&lt;/code&gt; is the variable to test, &lt;code&gt;n&lt;/code&gt; is the the number of trials (=25) and &lt;code&gt;prob&lt;/code&gt; is the probability of getting it correct (=.2). The population value (&lt;code&gt;mu&lt;/code&gt;) of the mean number correct is n*p. The population standard deviation, &lt;code&gt;var&lt;/code&gt;, is square-root(n*p*[1-p]).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I guess this compares x to an array composed of randomly generated binomial sample. If I centre x at 0 (variable-mu) I get a p-value. &lt;strong&gt;Can somebody confirm that it is doing what I think it is doing?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My testing gives this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; binom.samp1 &amp;lt;- as.data.frame(matrix(rbinom(30*1, size=25, prob=0.2), &#xA;                                     ncol=1))&#xA;&amp;gt; z.test(binom.samp1$V1, mu=5, sigma.x=2)&#xA;data:  binom.samp1$V1 &#xA;z = 0.7303, p-value = 0.4652&#xA;&#xA;&amp;gt; perm.z.test(binom.samp1$V1-5, 5, 2, 25, .2, 2000)&#xA;[1] 0.892&#xA;&#xA;&amp;gt; binom.samp1 &amp;lt;- as.data.frame(matrix(rbinom(1000*1, size=25, prob=0.2),&#xA;                                     ncol=1))&#xA;&amp;gt; perm.z.test(binom.samp1$V1-5, 5, 2, 25, .2, 2000)&#xA;[1] 0.937&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Does this look right?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;UPDATE:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since this obviously doesn't do what I want, I do have another angle. This &lt;a href=&quot;http://www.stat.umn.edu/geyer/old/5601/examp/perm.html#one&quot; rel=&quot;nofollow&quot;&gt;website&lt;/a&gt; offers this advice:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;There is no reason whatsoever why a&#xA;  permutation test has to use any&#xA;  particular test statistic. Any test&#xA;  statistic will do! ... For one-sample&#xA;  or paired two-sample tests, in&#xA;  particular, for Wilcoxon signed rank&#xA;  tests, the permutations are really&#xA;  subsets. The permutation distribution&#xA;  choses an arbitrary subset to mark +&#xA;  and the complementary subset is marked&#xA;  -. Either subset can be empty.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;What about an arbitrary subset with a one-sample z-test?&lt;/p&gt;&#xA;" OwnerUserId="1614" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-11-09T01:25:20.130" LastActivityDate="2010-11-09T01:25:20.130" Title="Help with a one-sample permutation z-test" Tags="&lt;r&gt;&lt;permutation&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="4313" PostTypeId="2" ParentId="726" CreationDate="2010-11-08T14:12:20.500" Score="1" ViewCount="" Body="&lt;p&gt;&quot;After 17 years of interacting with physicians, I have come to realize that many of them are adherents of a religion they call Statistics... Like any good religion, it involves vague mysteries capable of contradictory and irrational interpretation. It has a priesthood and a class of mendicant friars. And it provides Salvation: Proper invocation of the religious dogmas of Statistics will result in publication in prestigious journals.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/David_Salsburg&quot; rel=&quot;nofollow&quot;&gt;David S. Salsburg&lt;/a&gt; (author of &lt;em&gt;The Lady Tasting Tea&lt;/em&gt;), quoted at &quot;&lt;a href=&quot;http://www.pithypedia.com/?author=David+S.+Salsburg&quot; rel=&quot;nofollow&quot;&gt;Pithypedia&lt;/a&gt;&quot;.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-11-08T14:12:20.500" CommunityOwnedDate="2010-11-08T14:12:20.500" />
  <row Id="4314" PostTypeId="2" ParentId="4298" CreationDate="2010-11-08T14:55:21.340" Score="2" ViewCount="" Body="&lt;p&gt;I have read both the first linked earlier question, especially the answer of whuber and the comments on this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The answer is yes, you can do that, i.e. using the density from a kde of a numeric variable as conditional probability ($P(X=x|C=c)$ in the bayes theorem. &#xA;$P(C=c|X=x)=P(C=c)*P(X=x|C=c)/P(X=x)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By assuming that d(height) is equal across all classes, d(height) is normalized out when the theorem is applied, i.e. when $P(X=x|C=c)$ is divided by $P(X=x)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This paper could be interesting for you: &lt;a href=&quot;http://www.cs.iastate.edu/~honavar/bayes-continuous.pdf&quot; rel=&quot;nofollow&quot;&gt;estimating continuous distributions in bayesian classifiers&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="264" LastActivityDate="2010-11-08T14:55:21.340" />
  <row Id="4315" PostTypeId="2" ParentId="726" CreationDate="2010-11-08T15:05:19.573" Score="1" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;It would be illogical to assume that&#xA;  all conditions remain stable&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;~ Spock, &quot;The Enterprise Incident&quot;,stardata 5027.3&lt;/p&gt;&#xA;" OwnerUserId="264" LastActivityDate="2010-11-08T15:05:19.573" CommunityOwnedDate="2010-11-08T15:05:19.573" />
  <row Id="4316" PostTypeId="1" CreationDate="2010-11-08T15:58:52.233" Score="3" ViewCount="572" Body="&lt;p&gt;I would like to the opinion of people here on the follwoing question;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can one think of efficacy versus effectiveness in a way that can help remove confusion? It appears that even well respected researchers seem to use the terms interchangeably. The context of this question is within a health framework i.e. looking at one or more therapies in the treatment of a condition.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If possible could any people also shed some light on what type of study designs would be most appropriate in determining both types of results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any authoritative journal publications, books, or web dictionaries that may help me?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you all for your help in advance.&lt;/p&gt;&#xA;" OwnerUserId="431" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-10T07:38:29.250" LastActivityDate="2010-11-10T20:24:26.790" Title="What is the difference between effectiveness and efficacy in determining the benefit of therapy 'A' on condition 'B'?" Tags="&lt;epidemiology&gt;&lt;causal-inference&gt;&lt;clinical-trials&gt;&lt;definition&gt;&lt;instrumental-variables&gt;" AnswerCount="3" />
  <row Id="4317" PostTypeId="2" ParentId="4258" CreationDate="2010-11-08T16:03:39.950" Score="1" ViewCount="" Body="&lt;p&gt;Here is one approach at the automation. Feedback much appreciated. This is an attempt to replace initial visual inspection with computation, followed by subsequent visual inspection, in keeping with standard practice. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This solution actually incorporates two potential solutions, first, calculate burn-in to remove the length of chain before some threshold is reached, and then using the autocorrelation matrix to calculate the thinning interval.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;calculate a vector of the maximum median Gelman-Rubin convergence diagnostic shrink factor (grsf) for all variables in the&lt;/li&gt;&#xA;&lt;li&gt;find the minimum number of samples at which the grsf across all variables goes below some threshold, e.g. 1.1 in the example, perhaps lower in practice&lt;/li&gt;&#xA;&lt;li&gt;sub sample the chains from this point to the end of the chain&lt;/li&gt;&#xA;&lt;li&gt;thin the chain using the autocorrelation of the most autocorrelated chain&lt;/li&gt;&#xA;&lt;li&gt;visually confirm convergence with trace, autocorrelation, and density plots &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;The mcmc object can be downloaded here: &lt;a href=&quot;https://netfiles.uiuc.edu/dlebauer/www/jags.out.Rdata&quot; rel=&quot;nofollow&quot;&gt;jags.out.Rdata&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# jags.out is the mcmc.object with m variables&#xA;library(coda)    &#xA;load('jags.out.Rdata')&#xA;# 1. calculate max.gd.vec, &#xA;# max.gd.vec is a vector of the maximum shrink factor&#xA;max.gd.vec     &amp;lt;- apply(gelman.plot(jags.out)$shrink[, ,'median'], 1, max)&#xA;# 2. will use window() to subsample the jags.out mcmc.object&#xA;# 3. start window at min(where max.gd.vec &amp;lt; 1.1, 100) &#xA;window.start   &amp;lt;- max(100, min(as.numeric(names(which(max.gd.vec - 1.1 &amp;lt; 0)))))&#xA;jags.out.trunc &amp;lt;- window(jags.out, start = window.start)&#xA;# 4. calculate thinning interval&#xA;# thin.int is the chain thin interval&#xA;# step is very slow &#xA;# 4.1 find n most autocorrelated variables&#xA;n = min(3, ncol(acm))&#xA;acm             &amp;lt;- autocorr.diag(jags.out.trunc)&#xA;acm.subset      &amp;lt;- colnames(acm)[rank(-colSums(acm))][1:n]&#xA;jags.out.subset &amp;lt;- jags.out.trunc[,acm.subset]&#xA;# 4.2 calculate the thinning interval&#xA;# ac.int is the time step interval for autocorrelation matrix&#xA;ac.int          &amp;lt;- 500 #set high to reduce computation time&#xA;thin.int        &amp;lt;- max(apply(acm2 &amp;lt; 0, 2, function(x) match(T,x)) * ac.int, 50)&#xA;# 4.3 thin the chain &#xA;jags.out.thin   &amp;lt;- window(jags.out.trunc, thin = thin.int)&#xA;# 5. plots for visual diagnostics&#xA;plot(jags.out.thin)&#xA;autocorr.plot(jags.win.out.thin)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;--update--&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As implemented in R the computation of the autocorrelation matrix is slower than would be desirable (&gt;15 min in some cases), to a lesser extent, so is computation of the GR shrink factor. There is a question about how to speed up step 4 on stackoverflow &lt;a href=&quot;http://stackoverflow.com/q/4110937/199217&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;--update part 2--&lt;/p&gt;&#xA;&#xA;&lt;p&gt;additional answers:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;It is not possible to diagnose convergence, only to diagnose lack of convergence &lt;a href=&quot;https://netfiles.uiuc.edu/dlebauer/www/jags.out.Rdata&quot; rel=&quot;nofollow&quot;&gt;(Brooks, Giudici, and Philippe, 2003)&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The function autorun.jags from the package &lt;a href=&quot;http://cran.r-project.org/web/packages/runjags/&quot; rel=&quot;nofollow&quot;&gt;runjags&lt;/a&gt; automates calculation of run length and convergence diagnostics. It does not start monitoring the chain until the Gelman rubin diagnostic is below 1.05; it calculates the chain length using the Raftery and Lewis diagnostic.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Gelman et al's (Gelman 2004 Bayesian Data Analysis, p. 295, &lt;a href=&quot;http://pubs.amstat.org/doi/abs/10.1198/1061860031347&quot; rel=&quot;nofollow&quot;&gt;Gelman and Shirley, 2010&lt;/a&gt;) state that they use a conservative approach of discarding the 1st half of the chain. Although a relatively simple solution, in practice this is sufficient to solve the issue for my particular set of models and data.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#code for answer 3&#xA;chain.length &amp;lt;- summary(jags.out)$end&#xA;jags.out.trunc &amp;lt;- window(jags.out, start = chain.length / 2)&#xA;# thin based on autocorrelation if &amp;lt; 50, otherwise ignore&#xA;acm &amp;lt;- autocorr.diag(jags.out.trunc, lags = c(1, 5, 10, 15, 25))&#xA;# require visual inspection, check acceptance rate&#xA;if (acm == 50) stop('check acceptance rate, inspect diagnostic figures') &#xA;thin.int &amp;lt;- min(apply(acm2 &amp;lt; 0, 2, function(x) match(TRUE, x)), 50)&#xA;jags.out.thin &amp;lt;- window(jags.out.trunc, thin = thin.int)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1381" LastEditorUserId="1381" LastEditorDisplayName="" LastEditDate="2010-11-10T00:17:12.523" LastActivityDate="2010-11-10T00:17:12.523" CommunityOwnedDate="2010-11-08T23:28:28.553" CommentCount="1" />
  <row Id="4318" PostTypeId="2" ParentId="726" CreationDate="2010-11-08T16:06:40.543" Score="6" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;On two occasions I have been asked [by&#xA;  members of Parliament], ‘Pray, Mr.&#xA;  Babbage, if you put into the machine&#xA;  wrong figures, will the right answers&#xA;  come out?’ I am not able rightly to&#xA;  apprehend the kind of confusion of&#xA;  ideas that could provoke such a&#xA;  question.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Charles Babbage&lt;/p&gt;&#xA;" OwnerUserId="1614" LastActivityDate="2010-11-08T16:06:40.543" CommunityOwnedDate="2010-11-08T16:06:40.543" CommentCount="2" />
  <row Id="4319" PostTypeId="2" ParentId="4312" CreationDate="2010-11-08T16:13:13.950" Score="2" ViewCount="" Body="&lt;p&gt;Caveat: I'm not sure I fully understand your question. With this in mind, your solution does, IMHO, not provide a one-sample permutation z-Test, as it does not use the original data while performing some re-labeling of the experimental units consistent with the Null hypothesis in the given experimental design. Actually I do not see how any re-labeling can be performed at all in your situation that appears to be one-sample-from-one-population with a test for the distribution's location parameter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The onet.permutation() function you cite is, IMHO, misleadingly named as it refers to the test of two dependent samples. For each unit it randomizes which of its two values belongs to sample 1, and which one to sample 2. This is equivalent to randomizing the sign of the unit-wise difference between the two samples, as done by the lines&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mn &amp;lt;- sample(c(-1, 1), n, replace = TRUE)&#xA;xbardash &amp;lt;- mean(mn * abs(x))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(x is the difference between two dependent samples) Your function does something else: it simulates new data and creates a simulated distribution that does not stem from the empirical data.&lt;/p&gt;&#xA;" OwnerUserId="1909" LastActivityDate="2010-11-08T16:13:13.950" CommentCount="6" />
  <row Id="4320" PostTypeId="1" AcceptedAnswerId="4323" CreationDate="2010-11-08T16:18:53.253" Score="1" ViewCount="264" Body="&lt;p&gt;(1) I am looking for a package for computing the power of a matrix. If you have some good recommendation please let me know.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(2) I searched on the internet and followed what some said to install a package called &quot;Malmig&quot; in R but after selecting the mirror site, it failed:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In install.packages(&quot;Malmig&quot;) : package ‘Malmig’ is not available&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Some idea why?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="1005" LastActivityDate="2010-11-09T05:55:40.453" Title="Compute the power of a matrix in R" Tags="&lt;r&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="4321" PostTypeId="2" ParentId="4312" CreationDate="2010-11-08T16:27:25.407" Score="2" ViewCount="" Body="&lt;p&gt;I don't think it is going a one sample Z; to me it looks like it is a test against a certain set of priors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm confused, why are you doing a one sample Z using binomial data as your source data?  You could simply create a distribution of N successes and see what quantile your actual data was in.  However the above method doesn't look like a permutation test per-se to me; as your code doesn't actually permute the observed values between 1 and 0. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;That being said, let me comment on your code - to me it looks as though z is defined as&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;zeta = (mean(mn) - mu) / (sqrt(var/nx))&#xA;z[i] &amp;lt;- zeta&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Thus, each score in Z is like a Z score of the randomly created binomial vector using the priors you've selected as the null hypothesis.  But then you compare that Z to abs(mx); where mx is defined as the mean of your observed binomial vector.  At the very least this looks like a problem to me. Z scores should be either compared to some other Z score or means should be compared to means.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As I alluded to above, it is odd that you'd put all of this under a structure of a Z-test.  The Z score is nominally a linear transform of the differences between means, as such the result of a test like this should be the same whether you use a Z score or simply look at the differences between means.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Moreover, what you are doing seems a like an attempt to test the observed value against some priors rather than an actual one sample permutation test.  What you want to test against is something like permbinom (code provided below) where it could be the case for each observed value that it either was a success or it was not a success.  This is in-line with Tukey's classic example with the lady who claimed she could tell whether tea or milk was added first.  Critically different from your test, the assumption of this permutation test is that the null hypothesis is fixed at p = .5.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;permbinom &amp;lt;- function(x)&#xA;{&#xA;  newx &amp;lt;- x&#xA;    nx &amp;lt;- length(x) &#xA;    change &amp;lt;- rbinom(n=nx,size=1,prob=.5) &#xA;    #This code is readable but inefficient&#xA;    #Swap the values between 1 and 0 if change == 1&#xA;    for (i in 1:nx)&#xA;    {&#xA;        if ((change[i] == 1) &amp;amp; (x[i] == 1)) {newx[i] &amp;lt;- 0} &#xA;        if ((change[i] == 1) &amp;amp; (x[i] == 0)) {newx[i] &amp;lt;- 1} &#xA;    }&#xA;  return(newx)&#xA;}&#xA;permtest &amp;lt;- function(x,nsim=2000)&#xA;{&#xA;   permref &amp;lt;- rep(NA,nsim)&#xA;   obsn &amp;lt;- sum(x)&#xA;   for (i in 1:nsim)&#xA;     {&#xA;      permref[i] &amp;lt;- sum(permbinom(x))&#xA;     }&#xA;     pval &amp;lt;- min(c(&#xA;              (sum(permref &amp;gt; obsn)*2/nsim),&#xA;              (sum(permref &amp;lt; obsn)*2/nsim)&#xA;                ))&#xA;  return(pval)&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm not 100% confident regarding how I'm calculating the p-value here; so if someone would kindly correct me if I'm doing it wrong I'll incorporate that as an edit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For reference, here is a faster permutation function for one-sample tests of binomial data.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;permbinomf &amp;lt;- function(x)&#xA;{&#xA;  newx &amp;lt;- x&#xA;    nx &amp;lt;- length(x) &#xA;    change &amp;lt;- rbinom(n=nx,size=1,prob=.5) &#xA;    #This code is readable but inefficient&#xA;    #Swap the values between 1 and 0 if change == 1&#xA;newx &amp;lt;- x + change&#xA;newx &amp;lt;- ifelse(newx==2,0,newx)&#xA;  return(newx)&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt;  The question is also put forth, &quot;What about an arbitrary subset with a one-sample z-test?&quot;.  That would also work, assuming you had a large enough sample to subset.  However, it would not be a permutation test, it would be more akin to a bootstrap.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit 2:&lt;/strong&gt; Perhaps the most important answer to your question, is this: You are doing something acceptable (if you fix the Z vs mean computational error noted above), but you aren't doing what you think you are doing.  You are comparing your results to results where the null hypothesis is true.  This is essentially a Monte-Carlo simulation and if you correct the math (and I suggest you also simplify it) it is an acceptable technique for testing your hypothesis.  Also note, my answer above is for a two-tailed test.  As noted in the other question, you are ignoring the nesting of binomial observations under participants but independence isn't an assumption in a permutation or monte-carlo test so you should be more or less fine.  Though, as also noted there you ignore the possibility that some people are doing better than chance and others are performing at chance.&lt;/p&gt;&#xA;" OwnerUserId="196" LastEditorUserId="196" LastEditorDisplayName="" LastEditDate="2010-11-08T17:18:47.030" LastActivityDate="2010-11-08T17:18:47.030" CommentCount="8" />
  <row Id="4322" PostTypeId="2" ParentId="4316" CreationDate="2010-11-08T16:46:18.607" Score="4" ViewCount="" Body="&lt;p&gt;I'm not a specialist of this domain in epidemiological studies, but it seems to me that &lt;em&gt;efficacy&lt;/em&gt; has to do with the observed effect in a controlled setting, like a randomized clinical trial, whereas &lt;em&gt;effectiveness&lt;/em&gt; has more to do with a larger range of outcomes or environmental factors (potentially unobserved or non manipulated in the RCT), hence it has a wider scope. At least, I've often heard of &lt;a href=&quot;http://en.wikipedia.org/wiki/Cost-effectiveness_analysis&quot; rel=&quot;nofollow&quot;&gt;cost-effectiveness&lt;/a&gt; studies in pharmacoeconomics, and treatment efficacy (e.g., when comparing two treatment arms).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Quoting this article &lt;a href=&quot;http://www.australianprescriber.com/magazine/23/6/114/5/&quot; rel=&quot;nofollow&quot;&gt;Efficacy, effectiveness, efficiency&lt;/a&gt;,&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;efficacy is &quot;the extent to which a drug has the ability to bring about its intended effect under ideal circumstances, such as in a randomised clinical trial&quot;&lt;/li&gt;&#xA;&lt;li&gt;effectiveness is &quot;the extent to which a drug achieves its intended effect in the usual clinical setting&quot;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;As for other references, I would suggest starting with &lt;a href=&quot;http://schizophreniabulletin.oxfordjournals.org/content/26/3/533.full.pdf&quot; rel=&quot;nofollow&quot;&gt;Pitfalls of Multisite Randomized Clinical Trials of Efficacy and Effectiveness&lt;/a&gt; from Helena C Kraemer (&lt;em&gt;Schizophrenia Bulletin&lt;/em&gt; 26(3), 2000), and references therein. For example, it is read that &quot;efficacy and effectiveness are opposite extremes on a complex multidimensional continuum of decision making in research design&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Coming back from the &lt;a href=&quot;http://www.ispor.org/&quot; rel=&quot;nofollow&quot;&gt;ISPOR&lt;/a&gt; 13th European conference, I've heard that the European Federation of Pharmaceutical Industries and Associations (&lt;a href=&quot;http://www.efpia.org&quot; rel=&quot;nofollow&quot;&gt;EFPIA&lt;/a&gt;) considers there's now agreement on the following definitions:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;em&gt;relative efficacy&lt;/em&gt; can be defined as the extent to which an intervention does more good than harm, under ideal circumstances, compared to one or more alternative interventions;&lt;/li&gt;&#xA;&lt;li&gt;&lt;em&gt;relative effectiveness&lt;/em&gt; can be defined as the extent to which an intervention does more good than harm compared to one or more alternatives for achieving the desired results when provided under the usual circumstances of health care practice.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-09T22:31:20.787" LastActivityDate="2010-11-09T22:31:20.787" />
  <row Id="4323" PostTypeId="2" ParentId="4320" CreationDate="2010-11-08T17:03:27.557" Score="8" ViewCount="" Body="&lt;p&gt;Package expm provides the matrix %^% number operator notation for its function matpow():&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; library(expm)    &#xA;&amp;gt; mat &amp;lt;- matrix(1:9, nrow=3)&#xA;&amp;gt; mat %^% 2&#xA;     [,1] [,2] [,3]&#xA;[1,]   30   66  102&#xA;[2,]   36   81  126&#xA;[3,]   42   96  150&#xA;&#xA;# check&#xA;&amp;gt; mat %*% mat&#xA;     [,1] [,2] [,3]&#xA;[1,]   30   66  102&#xA;[2,]   36   81  126&#xA;[3,]   42   96  150&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;There's also sqrtm() for taking roots and expm() for matrix exponential.&lt;/p&gt;&#xA;" OwnerUserId="1909" LastActivityDate="2010-11-08T17:03:27.557" />
  <row Id="4324" PostTypeId="2" ParentId="4175" CreationDate="2010-11-08T17:07:23.140" Score="1" ViewCount="" Body="&lt;p&gt;Answer #1:  binom.test is in some ways a &quot;more correct&quot; test because it doesn't assume normality; yes - you'll get more power out of the normality assumption, and it might be reasonable - but to any extent you violate the assumptions of the test you may increase your type-I error rate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Explanation #1:  Though with a high number of trials results from a binomial data source approaches normality, it isn't perfectly normal.  To convince yourself about this you can use a Shapiro-Wilk test for normality, e.g. shapiro.test(rbinom(30,25,.2)) [where 30 is your number of participants, 25 is your number of trials, and .2 is the underlying probability of success].  You'll note with random data sometimes normality is significantly violated and sometimes it isn't.  Your own data will tell the story you need to know.  But, in general, because it is possible to violate normality under these circumstances, I prefer to avoid making the assumption.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Answer #2:  See my answer &lt;a href=&quot;http://stats.stackexchange.com/questions/4312/help-with-a-one-sample-permutation-z-test-r/4321#4321&quot;&gt;elsewhere&lt;/a&gt;.  What you are proposing sounds like a bootstrap of permutation test results.  Don't do that; it is odd and you won't be able to publish it.  The binom.test is sufficient for your data and hypothesis.  I'd suggest that you don't confuse matters by doing a permutation test or parametric test where the binomial distribution is clearly the best fit for the process generating your data.  Also, it is confusing that in one case you'd be willing to make assumptions (e.g. normality) but elsewhere and to use a permutation test.  The strength of permutation tests is that they don't tend to make as many assumptions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Answer #3:  It isn't gibberish.  You might want to consider breaking your questions down in the future.  It is a bit much for a single question here.  In short, standard statistical approaches can lead to a failure to replicate in the way you describe because either 1) the results from experiment 1 were due to a Type I error or 2) the results from experiment 2 were due to a Type II error.  Does N = 50 provide enough power that you can be confident in the results?&lt;/p&gt;&#xA;" OwnerUserId="196" LastActivityDate="2010-11-08T17:07:23.140" CommunityOwnedDate="2010-11-08T17:07:23.140" CommentCount="1" />
  <row Id="4325" PostTypeId="1" CreationDate="2010-11-08T17:46:28.193" Score="3" ViewCount="87" Body="&lt;p&gt;There are some cases where I would like to correct for multiple comparisons where the null hypotheses vary.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if I am performing a chi-squared test or a fisher's exact test on multiple contingency tables of different sizes, the null hypotheses vary from table to table. Can I still apply the Benjamini-Hochberg procedure for multiple comparisons using the p-values from these tests? If not, what should be done in this situation?&lt;/p&gt;&#xA;" LastActivityDate="2010-11-08T18:11:16.983" Title="Do the null hypotheses used for benjamini-hochberg have to be identical?" Tags="&lt;multiple-comparisons&gt;&lt;statistical-significance&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="4326" PostTypeId="2" ParentId="4325" CreationDate="2010-11-08T18:11:16.983" Score="5" ViewCount="" Body="&lt;p&gt;No, there is no such restriction. The only restriction is whether the tests themselves are independent or not, and there is even some wiggle room there: if the tests are positively correlated, you can pretend they are independent. See e.g &lt;a href=&quot;http://en.wikipedia.org/wiki/False_discovery_rate&quot; rel=&quot;nofollow&quot;&gt;False Discovery Rate&lt;/a&gt; or the references from &lt;a href=&quot;http://genomics.princeton.edu/storeylab/qvalue/&quot; rel=&quot;nofollow&quot;&gt;Qvalue&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-11-08T18:11:16.983" />
  <row Id="4327" PostTypeId="2" ParentId="4200" CreationDate="2010-11-08T18:17:46.173" Score="0" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Granger_causality&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Granger_causality&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Barrett, Barnett &amp;amp; Seth have a paper which extends the idea of Granger causality to the multivariate case.&lt;/p&gt;&#xA;" OwnerUserId="1349" LastActivityDate="2010-11-08T18:17:46.173" />
  <row Id="4328" PostTypeId="1" AcceptedAnswerId="4330" CreationDate="2010-11-08T18:58:46.730" Score="4" ViewCount="529" Body="&lt;p&gt;Does anyone know of some well written code for reversible jump MCMC? Preferably a simple demo application to compliment papers on the subject, that would be useful in understanding the process.&lt;/p&gt;&#xA;" OwnerUserId="1913" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-11-09T08:48:39.140" LastActivityDate="2010-11-09T09:27:58.247" Title="Reversible Jump MCMC code (Matlab or R)" Tags="&lt;r&gt;&lt;mcmc&gt;&lt;matlab&gt;" AnswerCount="2" FavoriteCount="2" />
  <row Id="4329" PostTypeId="2" ParentId="4316" CreationDate="2010-11-08T19:10:33.560" Score="3" ViewCount="" Body="&lt;p&gt;A standard, but not fully online, dictionary is the Dictionary of Epidemiology sponsored by the &lt;a href=&quot;http://ieaweb.org&quot; rel=&quot;nofollow&quot;&gt;International Epidemiological Association&lt;/a&gt;. The latest edition is the fifth, but the fourth edition appears to be latest &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0195141695&quot; rel=&quot;nofollow&quot;&gt;partly available online via Amazon&lt;/a&gt;. You want to look at p57-8. It says this distinction between effectiveness, efficacy and efficiency is due to &lt;a href=&quot;http://www.cochrane.org/about-us/history/archie-cochrane&quot; rel=&quot;nofollow&quot;&gt;Archie Cochrane&lt;/a&gt; in his 1972 book &lt;em&gt;Effectiveness and efficiency: random reflections on health services&lt;/em&gt;, &lt;a href=&quot;http://books.google.co.uk/books?id=jRsEIDJwSy8C&quot; rel=&quot;nofollow&quot;&gt;much of which is available on Google Books&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I won't quote too much from the above to avoid breaching copyright. I'll only note that although the Dictionary says, &quot;ideally the determination of efficacy is &lt;em&gt;based on&lt;/em&gt; the results of a randomized controlled trial&quot; (my italics), you need &lt;a href=&quot;http://dx.doi.org/10.1002/sim.4780100110&quot; rel=&quot;nofollow&quot;&gt;instrumental variables estimation methods to determine efficacy&lt;/a&gt; if there is substantial non-compliance.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-11-08T19:10:33.560" CommentCount="1" />
  <row Id="4330" PostTypeId="2" ParentId="4328" CreationDate="2010-11-08T19:30:49.540" Score="4" ViewCount="" Body="&lt;p&gt;RJMCMC was introduced by &lt;a href=&quot;http://www.stats.bris.ac.uk/~peter/Welcome.html&quot; rel=&quot;nofollow&quot;&gt;Peter Green&lt;/a&gt; in a &lt;a href=&quot;http://dx.doi.org/10.1093/biomet/82.4.711&quot; rel=&quot;nofollow&quot;&gt;1995 paper&lt;/a&gt; that is a citation classic. He wrote a Fortran program called &lt;a href=&quot;http://www.stats.bris.ac.uk/~peter/AutoRJ/&quot; rel=&quot;nofollow&quot;&gt;AutoRJ&lt;/a&gt; for automatic RJMCMC; his page on this links to David Hastie's C program &lt;a href=&quot;http://www.davidhastie.me.uk/automix/&quot; rel=&quot;nofollow&quot;&gt;AutoMix&lt;/a&gt;. There's a list of freely available software for various RJMCMC algorithms in Table 1 of a &lt;a href=&quot;http://web.maths.unsw.edu.au/~scott/papers/paper_tenyears.pdf&quot; rel=&quot;nofollow&quot;&gt;2005 paper by Scott Sisson&lt;/a&gt;. A Google search also finds &lt;a href=&quot;http://www.dcs.gla.ac.uk/inference/SERRS/Code.html&quot; rel=&quot;nofollow&quot;&gt;some pseudocode from a group at the University of Glasgow&lt;/a&gt; that may be useful in understanding the principles if you want to program it yourself.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-11-08T19:30:49.540" />
  <row Id="4331" PostTypeId="1" CreationDate="2010-11-08T20:11:55.963" Score="1" ViewCount="368" Body="&lt;p&gt;I've got a data-set which I assume is uniformly distributed. Say I've got &lt;code&gt;N=20000&lt;/code&gt; samples and a suspected &lt;code&gt;p=0.25&lt;/code&gt;. This means that I would expect each option to show up roughly &lt;code&gt;5000&lt;/code&gt; times.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I calculate the following interval &lt;code&gt;[5000 - x, 5000 + x]&lt;/code&gt; such that I can say with a certain confidence that the data-set is probably &lt;em&gt;NOT&lt;/em&gt; uniformly distributed since the number of times an option shows up falls outside of the interval?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt;&#xA;ABCDBCDADBCDA, BDCAADBCDADBA, ADCDBDACDBDAD, CDBDACDBDACDA, That's some sample data. A sample is one cookie string! Now I want for each position in that cookie string determine if a character there is too rare or too common at that position. So I count, for all samples, the number of A's on positon 0, the number of B's, C's and D's. Suppose I get a count of 5 A's on position 0 and I would expect a count of roughly 50 A's then the character A is too rare at position 0. That's what I want to do for each character position. &lt;/p&gt;&#xA;" LastActivityDate="2010-11-09T02:56:51.557" Title="Uniform Distribution Test" Tags="&lt;distributions&gt;&lt;test&gt;&lt;uniform&gt;" AnswerCount="1" CommentCount="6" />
  <row Id="4332" PostTypeId="2" ParentId="4331" CreationDate="2010-11-09T02:56:51.557" Score="3" ViewCount="" Body="&lt;p&gt;You might try assuming--as your null hypothesis--that the distribution is discrete uniform independent of string position.  Then tabulate the frequencies of each letter by position in a 4 x 13 contingency table.  You can then test for non-independence with a simple &lt;strong&gt;chi-square test&lt;/strong&gt;; with n=20,000 &lt;strong&gt;observations&lt;/strong&gt; in your &lt;strong&gt;one sample&lt;/strong&gt;, you shouldn't have any sparse table problems.  You can also eyeball this with a stacked bar chart, one 4-color ABCD bar for each string position.  This is useful if you reject the null with the chi-square test.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just to be sure, you might also want to check your data overall to see if it actually fits a discrete uniform distribution using a chi-square goodness of fit test.  After all, the distribution of characters could be independent of position without being uniformly distributed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to estimate confidence intervals, treat the ABCD distribution as a multinomial distribution.  You can estimate standard errors from the variance-covariance matrix, which has diagonal (variance) entries &lt;code&gt;np[i](1-p[i])&lt;/code&gt; and off-diagonal (covariance) entries &lt;code&gt;-np[i]p[j]&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A good reference for all this is Agresti's &lt;em&gt;Categorical Data Analysis&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the distribution of letters seems independent of position, you can do further exploration using the &lt;strong&gt;runs test&lt;/strong&gt;.  Treat the letters as ordinal data, &lt;code&gt;A &amp;lt; B &amp;lt; C &amp;lt; D&lt;/code&gt;; the runs test will check for runs that are too long (too few) or too short (too many).  This type of test, and many others, are described in Knuth's  &lt;em&gt;Seminumerical Algorithms&lt;/em&gt;, where he discusses tests for random number generators.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Looks like you have a lot of tabulating to do--enjoy!&lt;/p&gt;&#xA;" OwnerUserId="1920" LastActivityDate="2010-11-09T02:56:51.557" />
  <row Id="4333" PostTypeId="2" ParentId="4320" CreationDate="2010-11-09T05:55:40.453" Score="1" ViewCount="" Body="&lt;p&gt;There is the following code you can write:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;library(Biodem)&#xA;  png(filename=&quot;images/mtx.exp_%03d.png&quot; ,width=480, height=480)&lt;/p&gt;&#xA;  &#xA;  &lt;h3&gt;Name: mtx.exp&lt;/h3&gt;&#xA;  &#xA;  &lt;h3&gt;Title: Calculates the n-th power of a matrix&lt;/h3&gt;&#xA;  &#xA;  &lt;h3&gt;Aliases: mtx.exp&lt;/h3&gt;&#xA;  &#xA;  &lt;h3&gt;Keywords: array manip methods&lt;/h3&gt;&#xA;  &#xA;  &lt;h3&gt;** Examples&lt;/h3&gt;&#xA;  &#xA;  &lt;p&gt;test&amp;lt;-matrix(c(1:16), 4,4)&#xA;  pow.test&amp;lt;-mtx.exp(test,10)&#xA;  pow.test&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;for more details:http://rgm2.lab.nig.ac.jp/RGM2/R_man-2.9.0/library/Biodem/man/mtx.exp.html&lt;/p&gt;&#xA;" OwnerUserId="1808" LastActivityDate="2010-11-09T05:55:40.453" />
  <row Id="4334" PostTypeId="1" AcceptedAnswerId="4456" CreationDate="2010-11-09T07:30:15.237" Score="3" ViewCount="342" Body="&lt;p&gt;Hi, &#xA;I would like to implement the model proposed in &lt;a href=&quot;http://ideas.repec.org/p/arx/papers/0808.1710.html&quot; rel=&quot;nofollow&quot;&gt;Dynamic modeling of mean-reverting spreads&lt;/a&gt; (Kostas Triantafyllopoulos, Giovanni Montana).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;They propose to model a time serie Y_t with the following equations:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(1) Y_t = A_t + B_t * Y_(t-1) + e_t  &#xA;(2) A_t = Phi1 * A_(t-1) + nu1_t  &#xA;(3) B_t = Phi2 * B_(t-1) + nu2_t&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;That can be expressed in a state space form:  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(1') Y_t = F_t * theta_t + e_t  &#xA;(2') theta_t = Phi * theta_(t-1) + nu_t&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;with   &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;F_t = (1, Y_t-1)  &#xA;Phi = diag(Phi1, Phi2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I would like to use R to perform a bayesian update of this model. I have studied the package 'dlm' but in the book &lt;a href=&quot;http://www.springer.com/statistics/statistical+theory+and+methods/book/978-0-387-77237-0&quot; rel=&quot;nofollow&quot;&gt;Dynamic linear models with R&lt;/a&gt;, written by Giovanni Petris (author of the 'dlm' package), it is written (page 113) &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&quot;The matrix F_t of a DLM cannot depend on past values of the observations&quot;.   &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;However, it seams that it is the case in the model proposed above. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone understand this last sentence and eventually help me perform this implementation in R?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Fred&lt;/p&gt;&#xA;" OwnerUserId="1709" LastEditorUserId="1709" LastEditorDisplayName="" LastEditDate="2010-11-10T01:59:02.990" LastActivityDate="2010-12-24T13:12:37.723" Title="State space form of time varying AR(1)" Tags="&lt;r&gt;&lt;bayesian&gt;&lt;dynamic-regression&gt;" AnswerCount="3" CommentCount="4" FavoriteCount="3" />
  <row Id="4335" PostTypeId="1" CreationDate="2010-11-09T08:55:58.867" Score="14" ViewCount="265" Body="&lt;p&gt;I just discovered the &lt;code&gt;comment&lt;/code&gt; function in R.  Example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x &amp;lt;- matrix(1:12, 3,4)&#xA;comment(x) &amp;lt;- c(&quot;This is my very important data from experiment #0234&quot;,&#xA;                &quot;Jun 5, 1998&quot;)&#xA;x&#xA;comment(x)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is the first time I came by this function and was wondering what are common/useful uses of it.&#xA;Since it is quite difficult to search &quot;R comment&quot; in google and find relevant results, I was hoping someone here might share with his experience.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(p.s: why is it I can't find the &quot;community wiki&quot; button?!)&lt;/p&gt;&#xA;" OwnerUserId="253" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-11-09T09:34:59.927" LastActivityDate="2010-11-09T11:54:24.230" Title="What is a good use of the 'comment' function in R?" Tags="&lt;r&gt;" AnswerCount="3" CommentCount="3" FavoriteCount="7" />
  <row Id="4336" PostTypeId="2" ParentId="4335" CreationDate="2010-11-09T09:09:25.487" Score="10" ViewCount="" Body="&lt;p&gt;One thing I often find myself doing in my R scripts for a particular data analysis task is to include comments in the script about the units of variables in my data frames. I work with environmental data and chemists and ecologists seem to enjoy using a wide range of different units for the same things (mg L$^{-1}$ vs mu eq L$^{-1}$, etc). My colleagues usually store this information in the row immediately below the column names in Excel sheets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd see &lt;code&gt;comment()&lt;/code&gt; as a nice way of attaching this information to a data frame for future reference.&lt;/p&gt;&#xA;" OwnerUserId="1390" LastActivityDate="2010-11-09T09:09:25.487" />
  <row Id="4337" PostTypeId="1" AcceptedAnswerId="4340" CreationDate="2010-11-09T09:23:00.533" Score="8" ViewCount="306" Body="&lt;p&gt;I need to do an experiment. First let me describe present situation. The company that I work for is a cinema. It has a gaming section where people who are waiting for movies can pass time by playing games. People can pay only by using prepaid membership card. Unfortunately this gaming section is not generating enough sales. We are trying to find the cause(s).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My hypothesis is if we accept cash as payment, sales will increase.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My plan is to have experimental group and control group. The experimental group will accept cash payment, the control group doesn't. The sales of both groups are tallied before and after the experiment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The difficult thing about this is that I can't find a way to isolate the 'cash payment' factor from other factors:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;When the movie playing in the cinema is good, more people will come and sales will also increase&lt;/li&gt;&#xA;&lt;li&gt;Each cinema only has one gaming section, I can't split it into two sections (one accepts cash, the other doesn't)&lt;/li&gt;&#xA;&lt;li&gt;If several sites accept cash and several others don't, I don't think I can compare the results directly because the visitors are different, the number of gaming units are different&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I'm looking for suggestions to isolate this 'cash payment' variable, or maybe another approach altogether.&lt;/p&gt;&#xA;" OwnerUserId="1922" LastActivityDate="2010-11-11T13:12:53.023" Title="What to do with confounding variables?" Tags="&lt;experiment-design&gt;" AnswerCount="3" CommentCount="2" FavoriteCount="1" />
  <row Id="4338" PostTypeId="2" ParentId="4328" CreationDate="2010-11-09T09:27:58.247" Score="2" ViewCount="" Body="&lt;p&gt;The book &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1439811873&quot; rel=&quot;nofollow&quot;&gt;Bayesian Analysis for Population Ecology&lt;/a&gt; by King et al. describes RJMCMC in the context of population ecology. I found there description very clear and they provide R code in the appendix. However, I did find the R code very FORTRAN/C in style.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The book also has an associated &lt;a href=&quot;http://lemur.mcs.st-and.ac.uk/Book-website/&quot; rel=&quot;nofollow&quot;&gt;webpage&lt;/a&gt;, but some of the code found in the book isn't on the website.&lt;/p&gt;&#xA;" OwnerUserId="8" LastActivityDate="2010-11-09T09:27:58.247" />
  <row Id="4339" PostTypeId="2" ParentId="4335" CreationDate="2010-11-09T09:33:50.550" Score="4" ViewCount="" Body="&lt;p&gt;Similar facilities exist in other packages, such as the &lt;a href=&quot;http://www.stata.com/help.cgi?notes&quot; rel=&quot;nofollow&quot;&gt;-notes- command in Stata&lt;/a&gt;. We use this to document &lt;em&gt;full&lt;/em&gt; details of a variable, e.g. details of assay for a biochemical measurement, or exact wording of the question asked for questionnaire data. This is often too much info for the variable name or label, one or both of which are displayed in the output of every analysis involving the variable and are therefore best kept reasonably short.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-11-09T09:33:50.550" />
  <row Id="4340" PostTypeId="2" ParentId="4337" CreationDate="2010-11-09T10:58:27.400" Score="5" ViewCount="" Body="&lt;p&gt;Here are some suggestions relating your to bullet points above:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What about using the daily takings as an explanatory variable?&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What you need to do is form an equation where you predict gaming sales given a number of other factors. There factors will include things you are interested in such as whether they used a prepaid card. However, you need to also include factors that you aren't interested in but have to adjust for, such as daily takings. Obviously, if the film is a blockbuster then gaming sales will increase.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;Suppose you have N cinemas. Select N/2 cinemas and put them in group A and rest go in Group B. Now let Group A be the control group and B the experimental group. If possible, alternate this set-up, i.e. make Group A the experimental setup for a few weeks.&lt;/li&gt;&#xA;&lt;li&gt;If you can mix over groups (point above) then this isn't problem. Even if you can't you can include a variable representing the number of gaming units.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The statistical techniques you will probably need is &lt;a href=&quot;http://en.wikipedia.org/wiki/Linear_regression&quot; rel=&quot;nofollow&quot;&gt;multiple linear regression&lt;/a&gt; (MLR). Essentially, you build an equation of the form:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Gaming sales = a0 + a1*Prepaid + a2*Takens + a3*&amp;lt;other things&amp;gt;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;em&gt;a0&lt;/em&gt;, &lt;em&gt;a1&lt;/em&gt;, &lt;em&gt;a2&lt;/em&gt; are just numbers&lt;/li&gt;&#xA;&lt;li&gt;&lt;em&gt;Prepaid&lt;/em&gt; is either 0 or 1&lt;/li&gt;&#xA;&lt;li&gt;&lt;em&gt;Takens&lt;/em&gt; are the daily takens.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;MLR will allow you calculate the values of &lt;em&gt;a0-a2&lt;/em&gt;. So if &lt;em&gt;a1&lt;/em&gt; is large this indicates that Prepaid is important. &lt;/p&gt;&#xA;" OwnerUserId="8" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-11-09T11:51:48.430" LastActivityDate="2010-11-09T11:51:48.430" CommentCount="3" />
  <row Id="4341" PostTypeId="1" CreationDate="2010-11-09T11:36:58.940" Score="3" ViewCount="566" Body="&lt;p&gt;I do have a big list of numeric values (including duplicates) and I do want to group them into  ranges in order to see if how do they distribute.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say there are 1000 values ranging from 0 to 2.000.000 and I do want to group them. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I achieve this, preferably in Excel or SQL.&lt;/p&gt;&#xA;" OwnerUserId="1901" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-11-09T13:00:22.120" LastActivityDate="2010-11-16T14:27:29.113" Title="How do I group a list of numeric values into ranges?" Tags="&lt;excel&gt;&lt;sql&gt;" AnswerCount="5" FavoriteCount="1" />
  <row Id="4342" PostTypeId="2" ParentId="4335" CreationDate="2010-11-09T11:54:24.230" Score="8" ViewCount="" Body="&lt;p&gt;To second @ucfagls, Frank Harrell has developed efficient ways to handle annotated data.frame in R in his &lt;a href=&quot;http://cran.r-project.org/web/packages/Hmisc/index.html&quot; rel=&quot;nofollow&quot;&gt;Hmisc&lt;/a&gt; package. For example, the &lt;code&gt;label()&lt;/code&gt; and &lt;code&gt;units()&lt;/code&gt; functions allow to add dedicated attributes to R objects. I find them very handy when producing summary of data.frame (e.g., with &lt;code&gt;describe()&lt;/code&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another useful way of using such an extra attribute is to apply a timestamp on a data set. I also add an attribute for things like random seed, fold number (when I use k-kold or LOO cross-validation).&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-11-09T11:54:24.230" />
  <row Id="4343" PostTypeId="2" ParentId="4341" CreationDate="2010-11-09T12:25:21.007" Score="4" ViewCount="" Body="&lt;p&gt;I'll assume that you've already determined the number of categories you'll use. Let's say you want to use 20 categories. Then they will be:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Category 1: [0 - 100,000)&lt;/li&gt;&#xA;&lt;li&gt;Category 2: [100,000 - 200,000)&lt;/li&gt;&#xA;&lt;li&gt;Category 3: [200,000 - 300,000)&lt;/li&gt;&#xA;&lt;li&gt;...&lt;/li&gt;&#xA;&lt;li&gt;Category 19: [1,800,000 - 1,900,000)&lt;/li&gt;&#xA;&lt;li&gt;Category 20: [1,900,000 - 2,000,000]&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Note that the label of each category can be defined as &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;FLOOR (x / category_size) + 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is trivial to define as a computed column in SQL or as a formula in Excel.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that the last category is infinitesimally larger than the others, since it is closed on both sides. If you happen to get a value of exactly 2,000,000 you might erroneously classify it as falling into category 21, so you have to treat this exception with an ugly &quot;IF&quot; (in Excel) or &quot;CASE&quot; (in SQL).&lt;/p&gt;&#xA;" OwnerUserId="666" LastEditorUserId="666" LastEditorDisplayName="" LastEditDate="2010-11-09T12:56:17.023" LastActivityDate="2010-11-09T12:56:17.023" CommentCount="3" />
  <row Id="4345" PostTypeId="2" ParentId="4341" CreationDate="2010-11-09T13:00:47.157" Score="3" ViewCount="" Body="&lt;p&gt;Why group them? Instead, how about estimate the probability density function (PDF) of the distributions from which the data arise? Here's an R-based example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(123)&#xA;dat &amp;lt;- c(sample(2000000, 500), rnorm(100, 1000000, 1000), &#xA;         rnorm(150, 1500000, 100),rnorm(150, 500000, 10),&#xA;         rnorm(180, 10000, 10), rnorm(10, 1000, 5), 1:10)&#xA;dens &amp;lt;- density(dat)&#xA;plot(dens)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If the data are strictly bounded (0, 2,000,000) then the kernel density estimate is perhaps not best suited. You could fudge things by asking it to only evaluate the density between the bounds:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dens2 &amp;lt;- density(dat, from = 0, to = 2000000)&#xA;plot(dens2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Alternatively there is the histogram - a coarse version of the kernel density. What you specifically talk about is binning your data. There are lots of rules/approaches to selecting equal-width bins (i.e. the number of bins) from the data. In R the default is Sturges rule, but it also includes the Freedman-Diaconis rule and Scott's rule. There are others as well - see the Wikipedia page on &lt;a href=&quot;http://en.wikipedia.org/wiki/Histogram&quot; rel=&quot;nofollow&quot;&gt;histograms&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;hist(dat)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you are not interested in the kernel density plot or the histogram per se, rather just the binned data, then you can compute the number of bins using the &lt;code&gt;nclass.X&lt;/code&gt; family of functions where &lt;code&gt;X&lt;/code&gt; is one of &lt;code&gt;Sturges&lt;/code&gt;, &lt;code&gt;scott&lt;/code&gt; or &lt;code&gt;FD&lt;/code&gt;. And then use &lt;code&gt;cut()&lt;/code&gt; to bin your data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cut.dat &amp;lt;- cut(dat, breaks = nclass.FD(dat), include.lowest = TRUE)&#xA;table(cut.dat)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which gives:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; cut.dat&#xA;  [-2e+03,2.21e+05] (2.21e+05,4.43e+05] (4.43e+05,6.65e+05] (6.65e+05,8.88e+05] &#xA;                247                  60                 215                  61 &#xA;(8.88e+05,1.11e+06] (1.11e+06,1.33e+06] (1.33e+06,1.56e+06] (1.56e+06,1.78e+06] &#xA;                153                  51                 205                  50 &#xA;   (1.78e+06,2e+06] &#xA;                 58&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;in R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, binning is fraught with problems, most notably; How do you know that your choice of bins hasn't influenced the resulting impression you get of the way the data are distributed?&lt;/p&gt;&#xA;" OwnerUserId="1390" LastEditorUserId="1390" LastEditorDisplayName="" LastEditDate="2010-11-09T13:57:35.120" LastActivityDate="2010-11-09T13:57:35.120" CommentCount="5" />
  <row Id="4346" PostTypeId="1" CreationDate="2010-11-09T13:07:49.020" Score="0" ViewCount="110" Body="&lt;p&gt;Random forest generates so many trees the RF takes the input as a dataset and then it classifys the trees instead of trees we have to give voting or mechanism(which we introduce the mechanism ) we have to determine which tree has the best performance according to the measures(F Measure,Accuracy, ROC CUrve&lt;/p&gt;&#xA;&#xA;&lt;p&gt;we have a Random Forest java code by using the same copy of RF code we have to select the Random Forest with best split by modifiying with this method we have to determine the the best tree by the classify instance we use classification by the best tree . As same we have to select the Random Forest weighted Majority with the same code &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then How can we do this in java code i dont  know how to add the functionality and where??  &lt;/p&gt;&#xA;" LastActivityDate="2010-11-09T13:07:49.020" ClosedDate="2010-12-07T12:53:21.903" Title="metric based random forest" Tags="&lt;machine-learning&gt;" CommentCount="2" />
  <row Id="4347" PostTypeId="1" AcceptedAnswerId="4423" CreationDate="2010-11-09T14:58:56.320" Score="2" ViewCount="101" Body="&lt;p&gt;Hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;from what you have read or heard about, which is a good book on fuzzy logic/sets/systems? I'm interested in basic of fuzzy systems, fuzzification/defuzzification etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks,&#xA;Lucian&lt;/p&gt;&#xA;" OwnerUserId="976" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-11-09T15:21:45.263" LastActivityDate="2010-11-11T04:45:20.507" Title="Good fuzzy textbook" Tags="&lt;books&gt;&lt;big-list&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="4348" PostTypeId="2" ParentId="4341" CreationDate="2010-11-09T15:08:16.407" Score="2" ViewCount="" Body="&lt;p&gt;You have requested an Excel or SQL solution.  The easiest way in Excel is to use its &quot;Analysis&quot; add-in to create a histogram.  It will automatically create the bins (ranges of values) but, optionally, accepts a list of bin cutpoints as input and uses them.  The output includes a parallel list of bin counts.  This is especially handy for irregular-width bins.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a one-off calculation: if the data change or the cutpoints change, you have to go through the entire dialog again.  A more flexible option is to use COUNTIF to count all values less than or equal to any given bin cutpoint.  The first differences of such an array give the bin counts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a working example.  The data are in a column named &quot;Simulation_Z&quot; (which in this particular case is defined to be an entire column, such as &lt;code&gt;$C:$C&lt;/code&gt;).  The formulae shown below are copied from columns L2:N10 of a sheet in the same workbook.  They were created by copying the first one downward (but notice the special formula for the first count in N3).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Cut Count up                            Count&#xA;-3.0    =COUNTIF(Simulation_Z, &quot;&amp;lt;=&quot; &amp;amp; L3)   =M3&#xA;-2.0    =COUNTIF(Simulation_Z, &quot;&amp;lt;=&quot; &amp;amp; L4)   =M4-M3&#xA;-1.0    =COUNTIF(Simulation_Z, &quot;&amp;lt;=&quot; &amp;amp; L5)   =M5-M4&#xA; 0.0    =COUNTIF(Simulation_Z, &quot;&amp;lt;=&quot; &amp;amp; L6)   =M6-M5&#xA; 1.0    =COUNTIF(Simulation_Z, &quot;&amp;lt;=&quot; &amp;amp; L7)   =M7-M6&#xA; 2.0    =COUNTIF(Simulation_Z, &quot;&amp;lt;=&quot; &amp;amp; L8)   =M8-M7&#xA; 3.0    =COUNTIF(Simulation_Z, &quot;&amp;lt;=&quot; &amp;amp; L9)   =M9-M8&#xA;=MAX(Simulation_Z)  =COUNTIF(Simulation_Z, &quot;&amp;lt;=&quot; &amp;amp; L10)  =M10-M9&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Column L (&quot;Cut&quot;) stipulates the &lt;em&gt;upper&lt;/em&gt; limits of each bin.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This procedure simultaneously defines the bins and computes their counts, which are then available for further testing (&lt;em&gt;e.g.&lt;/em&gt;, $\chi\text{-squared}$) or plotting.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-11-13T01:32:33.880" LastActivityDate="2010-11-13T01:32:33.880" />
  <row Id="4350" PostTypeId="1" CreationDate="2010-11-09T15:34:36.077" Score="3" ViewCount="147" Body="&lt;p&gt;Hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am trying to use R to conduct community division within my weighted network (based from an association matrix). I tried with igraph but I encountered some problems. I usually use the program Socprog (Whitehead 2009) for my analysis, but as I would like to conduct a community division with Newman modularity (2006) on 1000 bootstraps of my data in order to calculate the comembership matrice resulting from all the community division of the bootstraps. So I am looking for a script that would conduct a community division (Newman 2006) from an association matrice and that would give a list of individuals with their cluster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone have a R script that works for this analysis? Any advice to implement this analysis would be much apreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Best regards&lt;/p&gt;&#xA;" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-11-09T16:24:51.990" LastActivityDate="2011-03-06T13:39:34.930" Title="How to conduct community division of a social network with R?" Tags="&lt;r&gt;&lt;networks&gt;" AnswerCount="1" FavoriteCount="3" />
  <row Id="4353" PostTypeId="1" CreationDate="2010-11-09T17:43:49.160" Score="1" ViewCount="281" Body="&lt;p&gt;when fitting GAMM with R, I would like to know why when the smooth function is linear, the confidence interval is zero around the middle (the dotted lines are crossing each other at the middle)&lt;/p&gt;&#xA;" LastEditorUserId="5" LastEditorDisplayName="" LastEditDate="2010-11-09T17:54:31.160" LastActivityDate="2010-11-09T19:04:51.730" Title="Fitting GAMM model in R" Tags="&lt;r&gt;&lt;mixed-model&gt;&lt;fitting&gt;" AnswerCount="1" />
  <row Id="4354" PostTypeId="1" AcceptedAnswerId="4355" CreationDate="2010-11-09T18:27:48.047" Score="13" ViewCount="352" Body="&lt;p&gt;Hello All,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was wondering if there are any distributions besides the normal where the mean and variance are independent of each other (or in other words, where the variance is not a function of the mean).&lt;/p&gt;&#xA;" OwnerUserId="1934" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-11-09T19:23:38.407" LastActivityDate="2010-11-12T17:08:50.283" Title="Distributions other than the normal where mean and variance are independent" Tags="&lt;distributions&gt;" AnswerCount="2" CommentCount="5" FavoriteCount="3" />
  <row Id="4355" PostTypeId="2" ParentId="4354" CreationDate="2010-11-09T18:52:16.460" Score="8" ViewCount="" Body="&lt;p&gt;Note: Please read answer by @G. Jay Kerns, and see &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1584881704&quot; rel=&quot;nofollow&quot;&gt;Carlin and Lewis 1996&lt;/a&gt; or your favorite probability reference for background on the calculation of mean and variance as the expectated value and second moment of a random variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A quick scan of Appendix A in Carlin and Lewis (1996) provides the following distributions which are similar in this regard to the normal, in that the same distribution parameters are not used in the calculations of the mean and variance. As pointed out by @robin, when calculating parameter estimates from a sample, the sample mean is required to calculate sigma.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Multivariate Normal&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$E(X) = \mu$$&#xD;&#xA;$$Var(X) = \Sigma$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;t&lt;/strong&gt; and &lt;strong&gt;multivariate t:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$E(X) = \mu$$&#xD;&#xA;$$Var(X) = \nu\sigma^2/(\nu - 2)$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Double exponential:&lt;/strong&gt;&#xA;$$E(X) = \mu$$&#xD;&#xA;$$Var(X) = 2\sigma^2$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Cauchy:&lt;/strong&gt;&#xA;With some qualification it could be argued that the mean and variance of the Cauchy are not dependent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$E(X)$ and $Var(X)$ do not exist&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Reference&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1584881704&quot; rel=&quot;nofollow&quot;&gt;Carlin, Bradley P., and Thomas A. Louis. 1996. Bayes and Empirical bayes Methods for Data Analysis, 2nd ed. Chapman and Hall/CRC, New York&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1381" LastEditorUserId="1381" LastEditorDisplayName="" LastEditDate="2010-11-12T17:08:50.283" LastActivityDate="2010-11-12T17:08:50.283" CommentCount="6" />
  <row Id="4356" PostTypeId="1" AcceptedAnswerId="5802" CreationDate="2010-11-09T18:55:17.830" Score="0" ViewCount="177" Body="&lt;p&gt;I know that R's &lt;code&gt;rpart&lt;/code&gt; function keeps the data it would need to implement multivariate split, but I don't know if it's actually performing multivariate splits.  I've tried researching it online looking at the &lt;code&gt;rpart&lt;/code&gt; docs, but I don't see any information that it can do it or is doing it.  Anyone know for sure?&lt;/p&gt;&#xA;" OwnerUserId="1929" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-11-09T19:22:47.847" LastActivityDate="2010-12-30T06:31:03.853" Title="Does rpart use multivariate splits by default?" Tags="&lt;r&gt;&lt;multivariate-analysis&gt;&lt;decision-trees&gt;" AnswerCount="4" CommentCount="2" />
  <row Id="4357" PostTypeId="2" ParentId="7" CreationDate="2010-11-09T19:00:39.677" Score="0" ViewCount="" Body="&lt;p&gt;Usage Over Time &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A very large Excel spreadsheet available for download containing data points for all online activities, with user demographics, over time. Please read Tip Sheet (below) before downloading or using this spreadsheet.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://pewinternet.org/Trend-Data/Usage-Over-Time.aspx&quot; rel=&quot;nofollow&quot;&gt;http://pewinternet.org/Trend-Data/Usage-Over-Time.aspx&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="253" LastActivityDate="2010-11-09T19:00:39.677" CommunityOwnedDate="2010-11-09T19:00:39.677" />
  <row Id="4358" PostTypeId="2" ParentId="4353" CreationDate="2010-11-09T19:04:51.730" Score="2" ViewCount="" Body="&lt;p&gt;It is due to the default for argument &lt;code&gt;'seWithMean'&lt;/code&gt; in &lt;code&gt;plot.gam()&lt;/code&gt;, which is &lt;code&gt;FALSE&lt;/code&gt;. This plots confidence intervals purely for the centred smooth function only, and there is no uncertainty at 0. If we add in the uncertainty in the mean then you get the more familiar confidence interval.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's an example, but using &lt;code&gt;gam()&lt;/code&gt; rather than &lt;code&gt;gamm()&lt;/code&gt; as it shows the same issue:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;## dummy data:&#xA;set.seed(123)&#xA;dat &amp;lt;- data.frame(x = 1:10, y = 1:10 + rnorm(10))&#xA;plot(y ~ x, data = dat)&#xA;&#xA;## load mgcv and fit an AM to the dummy data&#xA;require(mgcv)&#xA;mod &amp;lt;- gam(y ~ s(x), data = dat, method = &quot;ML&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The default plot shows the credible intervals as you describe:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plot(mod)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;whilst more natural credible intervals are given by&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plot(mod, seWithMean = TRUE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1390" LastActivityDate="2010-11-09T19:04:51.730" CommentCount="1" />
  <row Id="4359" PostTypeId="2" ParentId="4354" CreationDate="2010-11-09T19:09:47.093" Score="12" ViewCount="" Body="&lt;p&gt;In fact, the answer is &quot;no&quot;.  Independence of the sample mean and variance characterizes the normal distribution.  This was shown by Eugene Lukacs in &lt;a href=&quot;http://www.jstor.org/stable/2236166&quot; rel=&quot;nofollow&quot;&gt;&quot;A Characterization of the Normal Distribution&quot;, The Annals of Mathematical Statistics, Vol. 13, No. 1 (Mar., 1942), pp. 91-93.&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I didn't know this, but Feller, &quot;Introduction to Probability Theory and Its Applications, Volume II&quot; (1966, pg 86) says that R.C. Geary proved this, too.&lt;/p&gt;&#xA;" OwnerUserId="1108" LastEditorUserId="1108" LastEditorDisplayName="" LastEditDate="2010-11-09T19:32:28.120" LastActivityDate="2010-11-09T19:32:28.120" CommentCount="7" />
  <row Id="4360" PostTypeId="1" AcceptedAnswerId="4362" CreationDate="2010-11-09T19:23:19.233" Score="8" ViewCount="552" Body="&lt;p&gt;Let's say we are repeatedly tossing a fair coin, and we know number of heads and tails should be roughly equal. When we see a result like 10 heads and 10 tails for a total of 20 tosses, we believe the results and are inclined to believe the coin is fair.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Well when you see a result like 10000 heads and 10000 tails for a total of 20000 tosses, I actually would question the validity of the result (did the experimenter fake the data), as I know this is more unlikely than, say a result of 10093 heads and 9907 tails.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the statistical argument behind my intuition?&lt;/p&gt;&#xA;" OwnerUserId="578" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-11-23T17:05:19.157" LastActivityDate="2010-12-12T17:50:29.487" Title="Statistical argument for why 10,000 heads from 20,000 tosses suggests invalid data" Tags="&lt;confidence-interval&gt;&lt;binomial&gt;" AnswerCount="6" FavoriteCount="4" />
  <row Id="4361" PostTypeId="2" ParentId="4356" CreationDate="2010-11-09T19:31:10.937" Score="1" ViewCount="" Body="&lt;p&gt;As fas as I know, it doesn't;  but have not used it for a while. If I understand you well, you might want to look at package &lt;a href=&quot;http://cran.at.r-project.org/package=mvpart&quot; rel=&quot;nofollow&quot;&gt;mvpart&lt;/a&gt; instead.&lt;/p&gt;&#xA;" OwnerUserId="892" LastActivityDate="2010-11-09T19:31:10.937" />
  <row Id="4362" PostTypeId="2" ParentId="4360" CreationDate="2010-11-09T19:37:13.950" Score="18" ViewCount="" Body="&lt;p&gt;Assuming a fair coin the outcome of 10000 heads and 10000 tails is actually more likely than an outcome of 10093 heads and 9907 tails. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, when you say that a real experimenter is unlikely to obtain an equal number of heads and tails, you are implicitly invoking Bayes theorem. Your prior belief about a real experiment is that Prob(No of heads = 10000 in 20000 tosses | Given that experimenter is not faking) is close to 0. Thus, when you see an actual outcome that the 'No of heads = 10000' your posterior about Prob(Experimenter is not faking | observed outcome of 10000 heads) is also close to 0. Thus, you conclude that the experimenter is faking the data.&lt;/p&gt;&#xA;" LastActivityDate="2010-11-09T19:37:13.950" CommentCount="14" />
  <row Id="4363" PostTypeId="2" ParentId="4356" CreationDate="2010-11-09T20:03:11.627" Score="1" ViewCount="" Body="&lt;p&gt;Your terminology is confusing. Due you mean splits using more than one variable, or a tree that allows for a multivariate (as opposed to a univariate) response? I presume the latter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;F. Tusell has pointed you to the mvpart package, which adds a multivariate criterion for node impurity that is evaluated for all possible splits at each stage of tree building.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An alternative is the &lt;a href=&quot;http://cran.r-project.org/web/packages/party/index.html&quot; rel=&quot;nofollow&quot;&gt;party&lt;/a&gt; package, whose function &lt;code&gt;ctree()&lt;/code&gt; can handle multivariate responses.&lt;/p&gt;&#xA;" OwnerUserId="1390" LastActivityDate="2010-11-09T20:03:11.627" CommentCount="3" />
  <row Id="4364" PostTypeId="1" AcceptedAnswerId="4373" CreationDate="2010-11-09T20:19:21.667" Score="10" ViewCount="620" Body="&lt;p&gt;A standardized Gaussian distribution on $\mathbb{R}$ can be defined by giving explicitly its density: &#xA;$$ \frac{1}{\sqrt{2\pi}}e^{-x^2/2}$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;or its characteristic function. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As recalled in &lt;a href=&quot;http://stats.stackexchange.com/questions/4354/distributions-other-than-the-normal-where-mean-and-variance-are-independent&quot;&gt;this&lt;/a&gt; question it is also the only distribution for which the sample mean and variance are independent. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are other surprising alternative characterization of Gaussian measures that you know ? I will accept the most surprising answer :)  &lt;/p&gt;&#xA;" OwnerUserId="223" LastEditorUserId="223" LastEditorDisplayName="" LastEditDate="2011-02-15T15:06:31.437" LastActivityDate="2011-02-15T18:07:59.233" CommunityOwnedDate="2010-11-09T22:31:31.643" Title="What is the most surprising characterization of the Gaussian (normal) distribution?" Tags="&lt;probability&gt;&lt;normal-distribution&gt;&lt;mathematical-statistics&gt;" AnswerCount="9" CommentCount="0" FavoriteCount="3" />
  <row Id="4365" PostTypeId="2" ParentId="4364" CreationDate="2010-11-09T20:30:11.373" Score="5" ViewCount="" Body="&lt;p&gt;Gaussian distributions are the only &lt;a href=&quot;http://en.wikipedia.org/wiki/Stable_distribution&quot; rel=&quot;nofollow&quot;&gt;sum-stable&lt;/a&gt; distributions with finite variance. &lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-11-09T20:30:11.373" CommunityOwnedDate="2010-11-09T22:31:31.643" CommentCount="4" />
  <row Id="4366" PostTypeId="2" ParentId="4364" CreationDate="2010-11-09T20:36:44.480" Score="7" ViewCount="" Body="&lt;p&gt;The continuous distribution with fixed variance which maximizes &lt;a href=&quot;http://en.wikipedia.org/wiki/Differential_entropy&quot; rel=&quot;nofollow&quot;&gt;differential entropy&lt;/a&gt; is the Gaussian distribution.&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-11-09T20:36:44.480" CommunityOwnedDate="2010-11-09T22:31:31.643" />
  <row Id="4367" PostTypeId="1" AcceptedAnswerId="4369" CreationDate="2010-11-09T20:59:38.060" Score="3" ViewCount="244" Body="&lt;p&gt;I believe $p[x]$ is a probability distribution, where&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{equation}&#xA;p[x] = \frac{1}{\pi (1+x^2)} &#xA;\end{equation}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;since it's positive everywhere and integrates to 1 on $-\infty, \infty$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The mean is 0 by symmetry, even though integrating $xp[x]$ on &#xA;$-\infty, \infty$ does not converge. This is &quot;suspicious&quot; since &#xA;$p[x]$ is supposed to be a probability distribution, but reasonable &#xA;because $xp[x]$ is $O(1/x)$ which is known to diverge. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The bigger problem is in computing the standard deviation. Since $x^2 p[x]$ &#xA;also diverges, since $x^2 p[x]$ is $O(1)$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If this isn't a probability distribution, why not? If it is, is its &#xA;standard deviation infinite? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The cumulative distribution function is $\arctan[x]/\pi$ if that helps. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Someone mentioned this might be a gamma distribution, but that isn't &#xA;clear to me. &lt;/p&gt;&#xA;" OwnerUserId="1566" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-11-10T09:57:21.213" LastActivityDate="2010-11-10T09:57:21.213" Title="Can a probability distribution have infinite standard deviation?" Tags="&lt;distributions&gt;&lt;standard-deviation&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="4368" PostTypeId="1" AcceptedAnswerId="4370" CreationDate="2010-11-09T21:05:20.573" Score="1" ViewCount="206" Body="&lt;p&gt;Why in the white test, we estimate auxiliary regression model of the squared residuals (in the original model) and not just plain residuals?&lt;/p&gt;&#xA;" OwnerUserId="333" LastActivityDate="2010-11-17T04:00:28.513" Title="Auxiliary Model in the White Test" Tags="&lt;heteroscedasticity&gt;" AnswerCount="2" />
  <row Id="4369" PostTypeId="2" ParentId="4367" CreationDate="2010-11-09T21:07:46.350" Score="9" ViewCount="" Body="&lt;p&gt;To answer your question title: Yes, a probability distribution can have infinite standard deviation (see below).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your example is a special case of the &lt;a href=&quot;http://en.wikipedia.org/wiki/Cauchy_distribution&quot; rel=&quot;nofollow&quot;&gt;Cauchy distribution&lt;/a&gt; whose mean or variance does not exist. Set the location parameter to 0 and the scale to 1 for the Cauchy to get to your pdf. &lt;/p&gt;&#xA;" LastActivityDate="2010-11-09T21:07:46.350" CommentCount="1" />
  <row Id="4370" PostTypeId="2" ParentId="4368" CreationDate="2010-11-09T21:41:21.690" Score="1" ViewCount="" Body="&lt;p&gt;Because the White's test examines for heteroscedasticity by assuming that:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\sigma^2 = \beta_0 + x_1 \beta_1 + x_2 \beta_2 + ...$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(include squares and cross-product of covariates on the right hand side)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words, White's test investigates if the error variance is heteroscedastic by regressing an estimate of $\sigma^2$ against the available regressors. An estimate of $\sigma^2$ is given by the squared residuals as the mean of the residuals is zero.&lt;/p&gt;&#xA;" LastActivityDate="2010-11-09T21:46:44.320" CommentCount="2" />
  <row Id="4371" PostTypeId="1" AcceptedAnswerId="4381" CreationDate="2010-11-09T22:45:45.627" Score="6" ViewCount="118" Body="&lt;p&gt;In &quot;The Elements of Statistical Learning&quot; (2nd ed), p63, the authors give the following two formulations of the ridge regression problem:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ \hat{\beta}^{ridge} = \underset{\beta}{\operatorname{argmin}} \left\{ \sum_{i=1}^N(y_i-\beta_0-\sum_{j=1}^p x_{ij} \beta_j)^2 + \lambda \sum_{j=1}^p \beta_j^2 \right\} $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ \hat{\beta}^{ridge} = \underset{\beta}{\operatorname{argmin}} \sum_{i=1}^N(y_i-\beta_0-\sum_{j=1}^p x_{ij} \beta_j)^2 \text{, subject to } \sum_{j=1}^p \beta_j^2 \leq t.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is claimed that the two are equivalent, and that there is a one-to-one correspondence between the parameters $\lambda$ and $t$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It would appear that the first formulation is a Lagrangian relaxation of the second. However, I never had an intuitive understanding of how or why Lagrangian relaxations work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a simple way to demonstrate that the two formulations are indeed equivalent? If I have to choose, I'd prefer intuition over rigour.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="439" LastActivityDate="2010-11-10T08:58:59.530" Title="Lagrangian relaxation in the context of ridge regression" Tags="&lt;ridge-regression&gt;" AnswerCount="2" FavoriteCount="3" />
  <row Id="4372" PostTypeId="2" ParentId="4334" CreationDate="2010-11-09T22:54:54.083" Score="4" ViewCount="" Body="&lt;p&gt;This is a very unusual state space model because the dynamics are included in both the observation equation (1') and the state equation (2'). Usually, the dynamics are only in the state equation and the observation equation is a linear function of the state vector. I don't think any of the state space implementations in R will allow dynamics in the observation equation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is possible to re-write the model so that the dynamics are all in the state equation, but then it becomes non-linear.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suspect you will have to write your own code, or ask the authors if they can give you theirs.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-11-09T22:54:54.083" CommentCount="2" />
  <row Id="4373" PostTypeId="2" ParentId="4364" CreationDate="2010-11-09T23:09:28.487" Score="10" ViewCount="" Body="&lt;p&gt;My personal most surprising is the one about the sample mean and variance, but here is another (maybe) surprising characterization: if $X$ and $Y$ are IID with finite variance with $X+Y$ and $X-Y$ independent, then $X$ and $Y$ are normal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Intuitively, we can usually identify when variables are not independent with a scatterplot.  So imagine a scatterplot of $(X,Y)$ pairs that looks independent.   Now rotate by 45 degrees and look again: if it &lt;strong&gt;still&lt;/strong&gt; looks independent, then the $X$ and $Y$ coordinates individually must be normal (this is all speaking loosely, of course).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To see why the intuitive bit works, take a look at&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\left[&#xA;\begin{array}{cc}&#xA;\cos45^{\circ} &amp;amp; -\sin45^{\circ} \newline &#xA;\sin45^{\circ} &amp;amp; \cos45^{\circ} &#xA;\end{array}&#xA;\right]&#xA;\left[&#xA;\begin{array}{c}&#xA;x \newline&#xA;y&#xA;\end{array}&#xA;\right]= \frac{1}{\sqrt{2}}&#xA;\left[&#xA;\begin{array}{c}&#xA;x-y \newline&#xA;x+y&#xA;\end{array}&#xA;\right]&#xA;$$&lt;/p&gt;&#xA;" OwnerUserId="1108" LastActivityDate="2010-11-09T23:09:28.487" CommunityOwnedDate="2010-11-09T23:09:28.487" CommentCount="2" />
  <row Id="4375" PostTypeId="2" ParentId="4360" CreationDate="2010-11-09T23:44:11.400" Score="12" ViewCount="" Body="&lt;p&gt;I like Srikant's explanation, and I think the Bayesian idea is probably the best way to approach a problem like this.  But here is another way to see it without Bayes: (in R)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dbinom(10, size = 20, prob = 0.5)/dbinom(10000, 20000, 0.5)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which is about 31.2 on my system.  In other words, it is over 30 times more likely to see 10 out of 20 than it is to see 10,000 out of 20,000, even with a fair coin in both cases.  This ratio increases without bound as the sample size increases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a sort of likelihood ratio approach, but again, in my gut this feels like a Bayesian judgement call more than anything else. &lt;/p&gt;&#xA;" OwnerUserId="1108" LastActivityDate="2010-11-09T23:44:11.400" CommentCount="2" />
  <row Id="4377" PostTypeId="1" AcceptedAnswerId="4378" CreationDate="2010-11-10T02:00:03.560" Score="1" ViewCount="118" Body="&lt;p&gt;I want to model the probability of a binary variable x given some predictor, d.  It needs two parameters:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;One parameter that sets the &quot;break point&quot;, at which p(x=1 | d) = 0.5.&lt;/li&gt;&#xA;&lt;li&gt;One parameter that sets the &quot;softness&quot;, i.e. how abruptly the probability changes around the break point.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;(Sorry I'm sure there's standard terminology for this, but I'm a bit of a novice.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A logistic regression model feels like the right approach, but the domain of d is [0, inf] (d is a distance metric).  Is there a standard way to transform d to [-inf, inf] so a logistic regression can be used (e.g. f(x) = x-1/x)?   Or is there another common model for this kind of scenario?&lt;/p&gt;&#xA;" OwnerUserId="1938" LastEditorUserId="1938" LastEditorDisplayName="" LastEditDate="2010-11-10T02:29:39.833" LastActivityDate="2010-11-10T02:36:38.237" Title="Logistic regression with non-negative parameter" Tags="&lt;logistic-regression&gt;&lt;classification&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="4378" PostTypeId="2" ParentId="4377" CreationDate="2010-11-10T02:36:38.237" Score="5" ViewCount="" Body="&lt;p&gt;Logistic regression satisfies your requirements -- two parameters controlling the mid-point and the rate of change. But there is no restriction on the domain of predictors. You can just fit a logistic regression using d as it is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, if d is highly skewed (as it might be with that domain), then taking a log of d may be useful to prevent the fit being dominated by a few observations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If d includes exact zeros, then you obviously can't take logs. In that case, if you still think you need a transformation, then I'd use an &lt;a href=&quot;http://robjhyndman.com/researchtips/transformations/&quot; rel=&quot;nofollow&quot;&gt;inverse hyperbolic sine transformation&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-11-10T02:36:38.237" CommentCount="2" />
  <row Id="4379" PostTypeId="2" ParentId="4371" CreationDate="2010-11-10T03:54:06.940" Score="1" ViewCount="" Body="&lt;p&gt;The correspondence can most easily be shown using the &lt;a href=&quot;http://books.google.com/books?id=NiQw5ZEw2IIC&amp;amp;lpg=PA177&amp;amp;dq=envelope%20theorem%20shadow%20price&amp;amp;pg=PA177#v=onepage&amp;amp;q=envelope%20theorem%20shadow%20price&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;Envelope Theorem&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, the standard Lagrangian will have an additional $\lambda \cdot t$ term. This will not affect the maximization problem if we are just treating $\lambda$ as given, so Hastie et al drop it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, if you differentiate the full Lagrangian with respect to $t$, the Envelope Theorem says you can ignore the indirect effects of $t$ through $\beta$, because you're at a maximum. What you'll be left with is the Lagrange multipler from $\lambda \cdot t$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But what does this mean intuitively?  Since the constraint binds at the maximum, the derivative of the Lagrangian, evaluated at the maximum, is the same as the deriviate the original objective.  Therefore the Lagrange multiplier gives the shadow price -- the value in terms of the objective -- of relaxing the constraint by increasing $t$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I assume this is the correspondence Hastie et al. are referring to.&lt;/p&gt;&#xA;" OwnerUserId="493" LastActivityDate="2010-11-10T03:54:06.940" />
  <row Id="4381" PostTypeId="2" ParentId="4371" CreationDate="2010-11-10T08:58:59.530" Score="3" ViewCount="" Body="&lt;p&gt;If you merely want an intuitive explanation, go at 1.03.26 of this &lt;a href=&quot;http://www.youtube.com/watch?v=VxQ8VHm1Ci4&quot; rel=&quot;nofollow&quot;&gt;video&lt;/a&gt; (to the end), there is an intuitive explanation of how constraints relate to objective function.&lt;/p&gt;&#xA;" OwnerUserId="603" LastActivityDate="2010-11-10T08:58:59.530" CommentCount="2" />
  <row Id="4382" PostTypeId="2" ParentId="4364" CreationDate="2010-11-10T11:57:37.740" Score="0" ViewCount="" Body="&lt;p&gt;The expectation plus minus the standard deviation are the saddle points of the function. &lt;/p&gt;&#xA;" OwnerUserId="253" LastActivityDate="2010-11-10T11:57:37.740" CommunityOwnedDate="2010-11-10T11:57:37.740" CommentCount="1" />
  <row Id="4383" PostTypeId="1" AcceptedAnswerId="4389" CreationDate="2010-11-10T12:32:49.143" Score="6" ViewCount="255" Body="&lt;p&gt;[edits made in response to feedback- thanks :-) ]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Doh! More edits! Sorry!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hello-&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am doing some rather rough and ready data collection with a survey sent out to healthcare staff using a published scale about morale and other such issues.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The only thing is that the scale is rather long with all the other things in the survey and I would like to reduce its size by cutting each subscale in half and only using half the items. My intuition is that this is fine, since the subscales are inter-correlated, and while it's not ideal for publication-standard research, it's okay just for a bit of intra-organisational fact finding.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wondered if anyone had any thoughts on the validity of doing this, pitfalls, or anything else. References particularly are gratefully received because my colleagues will need some convincing!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks,&#xA;Chris B&lt;/p&gt;&#xA;&#xA;&lt;p&gt;edits-&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Yes it is a validated scale with known psychometric properties.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's unidimensional and it has subscales, if that's the right way to put it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'll be working at the subscale and total, not the item, level.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;30 items, probably about 40-60 individuals.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cheers!&lt;/p&gt;&#xA;" OwnerUserId="199" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-03-30T18:43:46.680" LastActivityDate="2011-03-30T18:43:46.680" Title="Can one validly reduce the numbers of items in a published Likert-scale?" Tags="&lt;psychometrics&gt;&lt;scales&gt;&lt;reliability&gt;&lt;likert&gt;" AnswerCount="4" CommentCount="3" FavoriteCount="1" />
  <row Id="4384" PostTypeId="2" ParentId="4341" CreationDate="2010-11-10T14:07:28.267" Score="1" ViewCount="" Body="&lt;p&gt;In Excel, a simple way to group numeric data into bins is via the Pivot Table. Pull the numeric variable into the &quot;row labels&quot;. Now right-click on any of the values in this right column and choose &quot;Group&quot;. You can set the min and max of the overall range and the bin size (equal bins widths for all data). &lt;/p&gt;&#xA;" OwnerUserId="1945" LastActivityDate="2010-11-10T14:07:28.267" CommentCount="2" />
  <row Id="4385" PostTypeId="2" ParentId="4316" CreationDate="2010-11-10T14:16:39.320" Score="1" ViewCount="" Body="&lt;p&gt;One place where I have seen this come up is in discussions of using &quot;intention to treat&quot; analysis versus an analysis that tries to get at the &quot;efficacy&quot; of a treatment in experiments with imperfect compliance. See the Wikipedia article on &quot;intention to treat&quot; (&lt;a href=&quot;http://en.wikipedia.org/wiki/Intention_to_treat_analysis&quot; rel=&quot;nofollow&quot;&gt;link&lt;/a&gt;), which includes some references.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a run-of-the-mill randomized control trial with noncompliance, the intention to treat estimate examines only the difference between those &lt;em&gt;assigned&lt;/em&gt; to treatment and control.  However, noncompliance means that some people who were assigned to treatment may not have actually taken it, and some assigned to be in the control group may have actually received the treatment.  If so, the intention to treat estimate may understate the average treatment effect that would obtain were all members of the population under study to actually take up the treatment.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When this kind of noncompliance is present, the analyst has a decision to make.  She could decide to simply do the intention to treat analysis, justifying it by saying that in the real world, we cannot control compliance, and so the intention to treat analysis is more &quot;realistic&quot; as an estimate of what would happen were this treatment approved for use clinically.  I have seen this referred to as an analysis of a treatment's &quot;effectiveness.&quot; Or, she may use some kind of adjustment method to try to get at how people who actually took up the treatment differed from those who didn't.  She could justify this by saying that what we are really interested in knowing is the biological (in the case of a medical trial) &quot;efficacy&quot; of the treatment, and to do so, we need to make the comparison between those who actually took the treatment and those who didn't.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The issue for an analysis of biological efficacy is, what &quot;kind of adjustment method&quot; is valid?  The current state of the art, as I understand, is to view an experiment with noncompliance as an instrumental variables problem, a la Angrist, Imbens and Rubin (1996) (&lt;a href=&quot;http://www.jstor.org/stable/2291629&quot; rel=&quot;nofollow&quot;&gt;gated link&lt;/a&gt;), or, more generally to view the problem in terms of &quot;principle stratification&quot;, a la Frangakis and Rubin (2002) (&lt;a href=&quot;http://www.jstor.org/stable/3068286&quot; rel=&quot;nofollow&quot;&gt;gated link&lt;/a&gt;).  As such, the randomization serves as an instrument that nonparametrically identifies &quot;efficacy&quot; effects for at least certain subpopulations---namely, those who would comply with their treatment or control assignment.  Beyond this, one could impose a more stringent model in order to identify efficacy effects, but then one may wonder, why did you bother to do a randomized experiment in the first place?&lt;/p&gt;&#xA;" OwnerUserId="96" LastEditorUserId="96" LastEditorDisplayName="" LastEditDate="2010-11-10T20:24:26.790" LastActivityDate="2010-11-10T20:24:26.790" />
  <row Id="4386" PostTypeId="2" ParentId="2419" CreationDate="2010-11-10T14:36:25.517" Score="1" ViewCount="" Body="&lt;p&gt;I have the same issue right now: I code in Python daily, use R once in a while, and need a good boosted regression tree algorithm. While there are lots of great Python packages for advanced analytics, my searching has not found a good offering for this particular algorithm. So, the route I think I'll be taking in coming weeks is to use the &lt;a href=&quot;http://cran.r-project.org/web/packages/gbm/&quot; rel=&quot;nofollow&quot;&gt;GBM package in R&lt;/a&gt;. There is a good paper showing practical issues with using it that can be found &lt;a href=&quot;http://www-stat.stanford.edu/~hastie/Papers/Ecology/Elith%20Leathwick%20Hastie_2008submitted.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;. Importantly, the GBM package was basically used &quot;off the shelf&quot; to w&lt;a href=&quot;http://www.cybaea.net/Blogs/Data/How-to-win-the-KDD-Cup-Challenge-with-R-and-gbm.html&quot; rel=&quot;nofollow&quot;&gt;in the 2009 KDD Cup&lt;/a&gt;. So, I'll probably do all of my pre and post modeling in Python and use RPy to go back and forth with R/GBM. &lt;/p&gt;&#xA;" OwnerUserId="1080" LastActivityDate="2010-11-10T14:36:25.517" />
  <row Id="4387" PostTypeId="2" ParentId="4259" CreationDate="2010-11-10T14:38:26.370" Score="4" ViewCount="" Body="&lt;p&gt;In Statistics, like in Data Mining, you start with data and a goal. In statistics there is a lot of focus on inference, that is, answering population-level questions using a sample. In data mining the focus is usually prediction: you create a model from your sample (training data) in order to predict test data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The process in statistics is then: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Explore the data using summaries and graphs - depending on how data-driven the statistician, some will be more open-minded, looking at the data from all angles, while others (especially social scientists) will look at the data through the lens of the question of interest (e.g., plot especially the variables of interest and not others)&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Choose an appropriate statistical model family (e.g., linear regression for a continuous Y, logistic regression for a binary Y, or Poisson for count data), and perform model selection&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Estimate the final model&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Test model assumptions to make sure they are reasonably met (different from testing for predictive accuracy in data mining)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Use the model for inference -- this is the main step that differs from data mining. The word &quot;p-value&quot; arrives here...&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Take a look at any basic stats textbook and you'll find a chapter on Exploratory Data Analysis followed by some distributions (that will help choose reasonable approximating models), then inference (confidence intervals and hypothesis tests) and regression models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I described to you the classic statistical process. However, I have many issues with it. The focus on inference has completely dominated the fields, while prediction (which is extremely important and useful) has been nearly neglected. Moreover, if you look at how social scientists use statistics for inference, you'll find that they use it quite differently! You can check out more about this &lt;a href=&quot;http://www.rhsmith.umd.edu/faculty/gshmueli/web/html/explain-predict.html&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1945" LastActivityDate="2010-11-10T14:38:26.370" CommunityOwnedDate="2010-11-10T14:38:26.370" />
  <row Id="4388" PostTypeId="2" ParentId="4337" CreationDate="2010-11-10T14:57:03.523" Score="2" ViewCount="" Body="&lt;p&gt;How about comparing the before and after you introduce the cash option across the two groups? Say you assign half the cinemas to the cash option (treatment) and half continue with no-cash (control). Now, you can compare how sales changed in the treatment group following the introduction of the cash option, and also compare how sales changes in the control group. If indeed the cash option is effective, then the change in the treatment group will be bigger than the change in the control group.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I recall reading an interesting statistical analysis done by Prof Ayala Cohen at the Technion's statistical lab for assessing the effect of removing advertising boards from a major highway in Israel on accidents in a similar fashion: to control for other factors that changed during this period, they compared the reduction in accidents before/after to a parallel highway where advertising boards remained there throughout the period.&lt;/p&gt;&#xA;" OwnerUserId="1945" LastActivityDate="2010-11-10T14:57:03.523" CommentCount="4" />
  <row Id="4389" PostTypeId="2" ParentId="4383" CreationDate="2010-11-10T15:32:50.900" Score="5" ViewCount="" Body="&lt;p&gt;Although there is still some information lacking (No. individuals and items per subscale), here are some general hints about scale reduction. Also, since you are working at the questionnaire level, I don't see why its length matters so much (after all, you will just give summary statistics, like total or mean scores).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I shall assume that (a) you have a set of K items measuring some construct related to morale, (b) your &quot;unidimensional&quot; scale is a second-order factor that might be subdivided into different facets, (c) you would like to reduce your scale to k &amp;lt; K items so as to summarize with sufficient accuracy subjects' totalled scale scores while preserving the content validity of the scale.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;About content/construct validity&lt;/em&gt; of this validated scale: The number of items has certainly been choosen so as to best reflect the construct of interest. By shortening the questionnaire, you are actually reducing construct coverage. It would be good to check that the factor structure remains the same when considering only half of the items (which could also impact the way you select them, after all). This can be done using traditional FA techniques. You hold the responsability of interpreting the scale in a spirit similar to that of the authors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;About scores reliability&lt;/em&gt;: Although it is a sample-dependent measure, scores reliability decreases when decreasing the number of items (cf. &lt;a href=&quot;http://en.wikipedia.org/wiki/Spearman-Brown_prediction_formula&quot; rel=&quot;nofollow&quot;&gt;Spearman-Brown formula&lt;/a&gt;); another way to see that is that the standard error of measurement (SEM) will increase, but see &lt;a href=&quot;http://www.ncme.org/pubs/items/16.pdf&quot; rel=&quot;nofollow&quot;&gt;An NCME Instructional Module on Standard Error of Measurement&lt;/a&gt;, by Leo M Harvill.&#xA;Needless to say, it applies to every indicator that depends on the number of items (e.g., Cronbach's alpha which can be used to estimate one form of reliability, namely the internal consistency). Hopefully, this will not impact any between-group comparisons based on raw scores.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, my recommendations (the easiest way) would be:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Select your items so as to maximise construct coverage; check the dimensionality with FA and coverage with univariate responses distributions;&lt;/li&gt;&#xA;&lt;li&gt;Compare average interitem correlations to previously reported ones; &lt;/li&gt;&#xA;&lt;li&gt;Compute internal consistency for the full scale and your composites; check that they are in agreement with published statistics on the original scale (no need to test anything, these are sample-dependent measures);&lt;/li&gt;&#xA;&lt;li&gt;Test the linear (or polychoric, or rank) correlations between original and reduced (sub)scores, to ensure that they are comparable (i.e., that individuals locations on the latent trait do no vary to a great extent, as objectivated through the raw scores);&lt;/li&gt;&#xA;&lt;li&gt;If you have an external subject-specific variable (e.g., gender, age, or best a measure related to morale), compare &lt;a href=&quot;http://luna.cas.usf.edu/~mbrannic/files/tnm/val1.htm&quot; rel=&quot;nofollow&quot;&gt;known-group validity&lt;/a&gt; between the two forms.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The hard way would be to rely on &lt;a href=&quot;http://en.wikipedia.org/wiki/Item_response_theory&quot; rel=&quot;nofollow&quot;&gt;Item Response Theory&lt;/a&gt; to select those items that carry the maximum of information on the latent trait -- scale reduction is actually one of its best application. Models for polytomous items were partly described in this thread, &lt;a href=&quot;http://stats.stackexchange.com/questions/4239/validating-questionnaires&quot;&gt;Validating questionnaires&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update after your 2nd update&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Forget about any IRT models for polytomous items with so few subjects.&lt;/li&gt;&#xA;&lt;li&gt;Factor Analysis will also suffer from such a low sample size; you will get unreliable factor loadings estimates.&lt;/li&gt;&#xA;&lt;li&gt;30 items divided by 2 = 15 items (it's easy to get an idea of the increase in the corresponding SEM for the total score), but it will definitively get worse if you consider subscales (this was actually my 2nd question--No. items per subscale, if any)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="930" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-11-11T05:39:58.823" LastActivityDate="2010-11-11T05:39:58.823" />
  <row Id="4390" PostTypeId="2" ParentId="4383" CreationDate="2010-11-10T15:35:46.517" Score="6" ViewCount="" Body="&lt;p&gt;I guess there's no clear-cut &quot;yes/no&quot; answer to your question. If you arbitrarily drop items from sub-scales to create a short form of the original questionnaire, you lose the long form's psychometric validation. Things that can change are the factorial structure of the questionnaire, reliability of sub-scales, item-total correlations, etc. (you'll note I'm used to classical test theory thinking, not IRT). Plus, you can't use any standardization of the original questionnaire. That's why short forms of established questionnaires have to undergo a separate validation phase.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Depending on your requirements, all ist not lost however. You may not need standardization because you may only want to compare results within your sample without making &quot;absolute&quot; judgements with respect to a reference population. IMHO, it would be a plus if you had the chance to validate the short form with the original form at least for a sub-sample of your group. This may allow you to see if results are similar.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general though, results to a questionnaire can be surprisingly sensitive to its item composition. People do not robotically fill out questionnaires but make all sorts of tacit assumptions and cognitive inferences: &quot;what is this really about?&quot;, &quot;what am I expected to report here?&quot;, &quot;what do they actually want to know?&quot;. This can be heavily influenced by the given context of items, cf. Schwarz, N. 1996. Cognition and Communication: Judgmental Biases, Research Methods, and the Logic of Conversation. Mahwah, NJ: Lawrence Erlbaum.&lt;/p&gt;&#xA;" OwnerUserId="1909" LastEditorUserId="1909" LastEditorDisplayName="" LastEditDate="2010-11-10T16:26:03.803" LastActivityDate="2010-11-10T16:26:03.803" />
  <row Id="4391" PostTypeId="2" ParentId="4383" CreationDate="2010-11-10T15:36:31.583" Score="1" ViewCount="" Body="&lt;p&gt;One reference that you may find useful for this: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://dx.doi.org/10.1111/j.1744-6570.2002.tb00108.x&quot; rel=&quot;nofollow&quot;&gt;STANTON, J. M., SINAR, E. F., BALZER, W. K. and SMITH, P. C. (2002), ISSUES AND STRATEGIES FOR REDUCING THE LENGTH OF SELF-REPORT SCALES. Personnel Psychology, 55: 167–194.&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1871" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-11-10T22:49:43.327" LastActivityDate="2010-11-10T22:49:43.327" />
  <row Id="4392" PostTypeId="1" AcceptedAnswerId="4395" CreationDate="2010-11-10T17:13:54.410" Score="1" ViewCount="165" Body="&lt;p&gt;Let $Z=(X+Y)/2$, where $X$ and $Y$ are independent normally-distributed random variables with known variances $\sigma^2_X$ and $\sigma^2_Y$ and unknown (and possibly different) means. Given a sample $x_1$ from $X$ and $y_1$ from $Y$, what is the maximum likelihood estimator of the mean of $Z$?&lt;/p&gt;&#xA;" LastActivityDate="2010-11-11T01:30:38.347" Title="What is the maximum likelihood estimator of the mean of two normally-distributed variables?" Tags="&lt;estimation&gt;&lt;maximum-likelihood&gt;&lt;normal-distribution&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="4393" PostTypeId="2" ParentId="4347" CreationDate="2010-11-10T17:18:08.687" Score="1" ViewCount="" Body="&lt;p&gt;I have no experience with fuzzy things (well, apart from &lt;a href=&quot;http://en.wikipedia.org/wiki/Fuzzy_Felt&quot; rel=&quot;nofollow&quot;&gt;Fuzzy Felt&lt;/a&gt;) but this book looks interesting:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Buckley, James J. &lt;em&gt;Fuzzy probability and statistics&lt;/em&gt;. Springer, 2006. ISBN &lt;a href=&quot;http://en.wikipedia.org/w/index.php?title=Special%3ABookSources&amp;amp;isbn=9783540308416&quot; rel=&quot;nofollow&quot;&gt;9783540308416&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-11-10T17:18:08.687" />
  <row Id="4394" PostTypeId="1" CreationDate="2010-11-10T17:37:00.063" Score="2" ViewCount="351" Body="&lt;p&gt;It occurred to me that, while I've pieced together some ideas over the years about the differences between statistics and biostatistics, I've never heard a formal explanation.  What is the distinction between these two disciplines (currently)?  And why did this distinction begin in the first place?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: I've not been specific enough in my original question.  I understand that biostatistics is the application and development of statistics in the biomedical field.  But what are some specific examples of the distinctions?  For example, what distinguishes graduate education in the two fields?  What is the purpose of having distinct academic departments for the two disciplines (a distinction I see in no other field)?&lt;/p&gt;&#xA;" OwnerUserId="71" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-10T18:51:24.437" LastActivityDate="2010-11-10T20:05:47.163" Title="What is the difference between statistics and biostatistics?" Tags="&lt;terminology&gt;&lt;biostatistics&gt;" AnswerCount="3" CommentCount="2" />
  <row Id="4395" PostTypeId="2" ParentId="4392" CreationDate="2010-11-10T17:58:49.740" Score="2" ViewCount="" Body="&lt;p&gt;If $Z = \frac{X+Y}{2}$ then it must be that:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Z \sim N(\frac{\mu_X + \mu_Y}{2} , \frac{\sigma_X^2 + \sigma_Y^2}{4})$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, the mle of the mean of $Z$ given that we observe $z=(x_1 + y_1)/2$ is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\frac{x_1 + y_1}{2}$&lt;/p&gt;&#xA;" LastActivityDate="2010-11-10T18:16:36.003" CommentCount="1" />
  <row Id="4396" PostTypeId="1" AcceptedAnswerId="4400" CreationDate="2010-11-10T18:13:38.233" Score="0" ViewCount="211" Body="&lt;p&gt;I have a data file in which each participant is tested many times in several blocks. The data is formatted so that each trial contains a participant id, a block number and a score. What I want to do is group the data by participant and block, then rank the scores for each group, so I can use the top 50%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;p_id, block, score&#xA;------------------&#xA;1     1      35&#xA;1     1      84&#xA;1     1      12&#xA;1     1      76&#xA;1     2      51&#xA;1     2      67&#xA;1     2      54&#xA;1     2      18&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;With this data I would have 2 groups. Of the first (p_id=1,block=1) the second and fourth scores would be used (score=84,score=76). Of the second group, the second and third would be used (score=67,score=54).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How would I go about doing this in SPSS?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(I'm using v16, but have the option of upgrading if necessary.)&lt;/p&gt;&#xA;" OwnerUserId="1950" LastEditorUserId="1950" LastEditorDisplayName="" LastEditDate="2010-11-10T18:54:34.407" LastActivityDate="2010-12-26T13:37:32.393" Title="In SPSS, is there any way to define a use groups of data based on combinations of variables?" Tags="&lt;spss&gt;" AnswerCount="3" CommentCount="2" />
  <row Id="4397" PostTypeId="2" ParentId="4394" CreationDate="2010-11-10T18:26:30.110" Score="4" ViewCount="" Body="&lt;p&gt;When I look at the Wikipedia entry for &lt;a href=&quot;http://en.wikipedia.org/wiki/Biostatistics&quot; rel=&quot;nofollow&quot;&gt;biostatistics&lt;/a&gt;, the relation to &lt;em&gt;biometrics&lt;/em&gt; doesn't seem so obvious to me since, historically, biometrics was more concerned with characterizing individuals by some phenotypes of interest, with large applications in population genetics (as exemplified by the work of Fisher), whereas part of this discipline now focus on biometric systems (whose objectives are the &quot;recognition or identification of individuals based on some physical or behavioral characteristics that are intrinsically unique for each individual&quot;, according to Boulgouris et al., &lt;em&gt;Biometrics&lt;/em&gt;, 2010). Anyway, there still are reviews like &lt;a href=&quot;http://biomet.oxfordjournals.org/&quot; rel=&quot;nofollow&quot;&gt;Biometrika&lt;/a&gt; and &lt;a href=&quot;http://www.biometrics.tibs.org/&quot; rel=&quot;nofollow&quot;&gt;Biometrics&lt;/a&gt;; although I read the latter on an irregular basis, most articles focus on &quot;biostatistical&quot; theoretical or applied work. The same applies for &lt;a href=&quot;http://biostatistics.oxfordjournals.org/&quot; rel=&quot;nofollow&quot;&gt;Biostatistics&lt;/a&gt;. By &quot;biostatistical&quot; applications, I mean that it has to do with applications or models related to the biomedical domain, in a wide sense (biology, health science, genetics, etc.). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;According to the &lt;em&gt;Encyclopedia of Biostatistics&lt;/em&gt; (2005, 2nd ed.), &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;(...) As is clear from the above examples,&#xA;  biostatistics is problem oriented. It&#xA;  is specifically directed to questions&#xA;  that arise in biomedical science. The&#xA;  methods of biostatistics are the&#xA;  methods of statistics -- concepts&#xA;  directed at variation in observations&#xA;  and methods for extracting information&#xA;  from observations in the face of&#xA;  variation from various sources, but&#xA;  notably from variation in the&#xA;  responses of living organisms and&#xA;  particularly human beings under study.&#xA;  Biostatistical activity spans a broad&#xA;  range of scientific inquiry, from the&#xA;  basic structure and functions of human&#xA;  beings, through the interactions of&#xA;  human beings with their environment,&#xA;  including problems of environmental&#xA;  toxicities and sanitation, health&#xA;  enhancement and education, disease&#xA;  prevention and therapy, the&#xA;  organization of health care systems&#xA;  and health care financing.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In sum, I think that Biostatistics is part of a super-family--Statistics--, and share most of its methods, but has a more focused area of interest (hence, an historical background, specific designs, and a general theoretical framework) and dedicated modeling strategies.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-10T18:48:10.007" LastActivityDate="2010-11-10T18:48:10.007" />
  <row Id="4398" PostTypeId="2" ParentId="4394" CreationDate="2010-11-10T18:33:59.103" Score="4" ViewCount="" Body="&lt;p&gt;To quote the &quot;Encyclopedic dictionary of mathematics&quot; by Kiyosi Itô (ed.):&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;In many applied fields there exist systems of statistical methods which have been developed specifically for the respective fields, and although all of them are based essentially on the same general principles of statistical inference, each has its own special techniques and procedures. Specific names have been invented, such as biometrics, econometrics, psychometrics, technometrics, sociometrics, etc.&lt;/blockquote&gt;&#xA;" OwnerUserId="439" LastActivityDate="2010-11-10T18:33:59.103" />
  <row Id="4399" PostTypeId="1" AcceptedAnswerId="4415" CreationDate="2010-11-10T19:06:41.087" Score="1" ViewCount="103" Body="&lt;p&gt;I'm looking at attempting to capture the regularity of a time series of events, one measurement per day, with a year's worth of data, and there can be at most one event a day. Say for example the day you do laundry. What I want to capture is a measurement of regularity. Capturing irregularity is straight forward: goodness of fit of the times between consecutive events with a poisson distribution. But this doesn't distinguish between poor fitting series very well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I want to measure the other end. How good am I to sticking to a weekly or bi-weekly schedule. My instinct is I want to fit an autoregressive model and take the variance of epsilon. Does this sound right? And how do I normalize for frequency? That is I don't want the epsilon to predict whether I do laundry 50 times a year or 100 times. Just the regularity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or is there a conventional way of doing this that I'm overlooking? In this case I'm biased towards fast conventional techniques rather than creative techniques.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&#xA;Steven&lt;/p&gt;&#xA;" OwnerUserId="1951" LastActivityDate="2010-11-11T06:02:57.460" Title="How do I capture regularity of a time series in a normalized way?" Tags="&lt;time-series&gt;&lt;pattern-recognition&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="1" />
  <row Id="4400" PostTypeId="2" ParentId="4396" CreationDate="2010-11-10T19:20:20.963" Score="3" ViewCount="" Body="&lt;p&gt;Ignore my initial post and take Jon's advice, heres some sample code from the RANKS command where X1 is the variable to rank and GROUPVAR is the variable identifying groups:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;RANK VARIABLES=X1 (A) BY GROUPVAR&#xA;  /RANK&#xA;  /PRINT=YES&#xA;  /TIES=MEAN.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can either go through the GUI to see all of its options or look up the ranks command in the help file. I can't see any reason to prefer my prior code over this simpler command, and it has options to specify fractional ranks or ntiles the ranks fall within (which would be the number you need to specify your select statement to only take the top half of the scores.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally I would note there is a &lt;a href=&quot;http://groups.google.com/group/comp.soft-sys.stat.spss/topics&quot; rel=&quot;nofollow&quot;&gt;Google group forum&lt;/a&gt; that answers SPSS questions and the UGA &lt;a href=&quot;http://spssx-discussion.1045642.n5.nabble.com/&quot; rel=&quot;nofollow&quot;&gt;SPSS listserv&lt;/a&gt; are both good places to ask SPSS questions.&lt;/p&gt;&#xA;" OwnerUserId="1036" LastEditorUserId="1036" LastEditorDisplayName="" LastEditDate="2010-12-22T20:28:59.727" LastActivityDate="2010-12-22T20:28:59.727" />
  <row Id="4401" PostTypeId="1" CreationDate="2010-11-10T19:25:32.353" Score="1" ViewCount="117" Body="&lt;p&gt;In our usage of R for a non-trivial data analysis and estimation project, we've been repeatedly burnt by how tolerant R is toward misspelled or missing columns in a data frame. Typical example is calculating the weighted mean of a variable MYVAR in a data frame using another variable WEIGHT for weights:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;m &amp;lt;- weighted.mean(tbl$MYVAR, w = tbl$WEIGHT, na.rm = TRUE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Suppose I make a typo in the WEIGHT name in the operation above. What will happen in that R will expand my misspelled column into NULL and will use it for performing the weighted mean resulting in a non-weighted one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, the question: is there any way to make R treat attempts to &quot;read&quot; a non-existent variables in a data frame as an error?&lt;/p&gt;&#xA;" OwnerUserId="1330" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-10T19:35:48.507" LastActivityDate="2010-11-10T19:50:12.800" Title="Make R report error on using non-existent column name in a data frame" Tags="&lt;r&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="4402" PostTypeId="2" ParentId="4401" CreationDate="2010-11-10T19:35:20.923" Score="1" ViewCount="" Body="&lt;p&gt;Maybe, you can enclose your code into try-catch blocks, see &lt;code&gt;?try&lt;/code&gt; and the associated examples. It is easy to test for the class of the results (&quot;try-error&quot;) in turn, e.g.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; res &amp;lt;- try(log(&quot;A&quot;), silent=TRUE)&#xA;&amp;gt; class(res)&#xA;[1] &quot;try-error&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can also test directly for the correct spelling, by first listing the variables of interest--in your case, &lt;code&gt;MYVAR&lt;/code&gt; and &lt;code&gt;WEIGHT&lt;/code&gt;-- and test that they are part of the data.frame &lt;code&gt;df&lt;/code&gt;, e.g.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df &amp;lt;- data.frame(x=rnorm(100), g1=gl(2, 50), g2=gl(5,20))&#xA;sel.vars &amp;lt;- c(&quot;x&quot;,&quot;g2&quot;)&#xA;ifelse(all(sel.vars %in% colnames(df)), &amp;lt;compute things here&amp;gt;, &quot;fail&quot;) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-10T19:43:46.667" LastActivityDate="2010-11-10T19:43:46.667" CommentCount="1" />
  <row Id="4403" PostTypeId="1" CreationDate="2010-11-10T19:36:52.677" Score="6" ViewCount="156" Body="&lt;p&gt;We need to estimate the probability of a person getting a question right for a given content area given his history of getting questions in the same content area right in the past.  We would also presumably have records on how others have done on this question and in this content area.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a good way or ways of doing this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(This is sort of an education theory question, but I couldn't find a better place to post this question.)&lt;/p&gt;&#xA;" OwnerUserId="1618" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-22T22:28:42.057" LastActivityDate="2010-11-22T22:28:42.057" Title="Estimating the probability of a person getting a question right" Tags="&lt;probability&gt;&lt;psychometrics&gt;&lt;testing&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="4404" PostTypeId="2" ParentId="4401" CreationDate="2010-11-10T19:50:12.800" Score="2" ViewCount="" Body="&lt;p&gt;Hmm... when I tried out your example with some fake data, &lt;code&gt;weighted.mean()&lt;/code&gt; actually failed:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#Some fake data&#xA;dat &amp;lt;- data.frame(x = rnorm(100), weight = rnorm(100))&#xA;&#xA;#The right weight var&#xA;weighted.mean(x = dat$x, w = dat$weight)&#xA;[1] 0.6161606&#xA;&#xA;#Misspelled weight var&#xA;weighted.mean(x = dat$x, w = dat$wieght)&#xA;Error in weighted.mean.default(x = dat$x, w = dat$wieght) : &#xA;  'x' and 'w' must have the same length&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But anyway, another way to cope with this problem is to access your variables via indexing - it returns an error if you try to pick non-existant columns:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dat$wieght&#xA;NULL&#xA;&#xA;dat[ , &quot;wieght&quot;]&#xA;Error in `[.data.frame`(dat, , &quot;wieght&quot;) : undefined columns selected&#xA;&#xA;weighted.mean(x = dat[ , &quot;x&quot;], w = dat[ , &quot;wieght&quot;], na.rm = TRUE)&#xA;Error in `[.data.frame`(dat, , &quot;wieght&quot;) : undefined columns selected&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="71" LastActivityDate="2010-11-10T19:50:12.800" CommentCount="3" />
  <row Id="4405" PostTypeId="2" ParentId="4403" CreationDate="2010-11-10T19:58:04.940" Score="7" ViewCount="" Body="&lt;p&gt;If I understand your question correctly, you have a set of items (pass-fail) and you want to assess the probability of endorsing the $k$th item given its preceding responses? If that's the case, what is usually done in psychometrics for educational assessment is to rely on &lt;a href=&quot;http://en.wikipedia.org/wiki/Item_response_theory&quot; rel=&quot;nofollow&quot;&gt;Item Response Model&lt;/a&gt;, like the &lt;a href=&quot;http://en.wikipedia.org/wiki/Rasch_model&quot; rel=&quot;nofollow&quot;&gt;Rasch Model&lt;/a&gt;. In short, you model the probability of endorsing an item as a function of item difficulty and person ability (the more proficient an individual is, the more likely his response to an easy item will be correct). This assumes that the content you are assessing is unidimensional, and that the items can be ordered by difficulty on that scale. A &lt;a href=&quot;http://en.wikipedia.org/wiki/Guttman_scale&quot; rel=&quot;nofollow&quot;&gt;Guttman model&lt;/a&gt; is rarely applicable, so we may allow for some &quot;imperfect&quot; response patterns (e.g., 111011101110000, the 4th and 8th items were failed although the examinee reached the 11th item before giving up), but the sum score is a sufficient statistic for the Rasch Model. Under this approach, you need to have responses from other individuals on the same set of items. To get an idea, look at the &lt;code&gt;LSAT&lt;/code&gt; data set and the way it is analysed in the &lt;a href=&quot;http://cran.r-project.org/web/packages/ltm/index.html&quot; rel=&quot;nofollow&quot;&gt;ltm&lt;/a&gt; R package.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have described a psychometrical model, not a purely probabilistic framework for estimating the probability of failing after the $k$th item, with all other items right (this would follow a geometric law).&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-10T20:05:36.893" LastActivityDate="2010-11-10T20:05:36.893" CommentCount="1" />
  <row Id="4406" PostTypeId="2" ParentId="4394" CreationDate="2010-11-10T20:05:47.163" Score="1" ViewCount="" Body="&lt;p&gt;Biostatistics, biometrics and biometry are synonyms. Medical statistics (sometimes called 'clinical biostatistics' for no clear reason) is a subset of these.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-11-10T20:05:47.163" CommentCount="3" />
  <row Id="4407" PostTypeId="1" CreationDate="2010-11-10T20:10:22.157" Score="1" ViewCount="165" Body="&lt;p&gt;I have a dendrogram called &lt;code&gt;x&lt;/code&gt; and I want to reorder it. First, I make sure that &lt;code&gt;x&lt;/code&gt; is a dendrogram, and it is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; class(x) &#xA;[1] &quot;dendrogram&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then I check the order of things in my dendrogram, and it's exactly as it is drawn, i.e. everything makes sense:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; order.dendrogram(x) &#xA;[1] 6 7 8 5 9 10 11 1 3 2 4 13 16 12 14 15 17&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now I decide on a new order:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; y &amp;lt;- c(6,7,8,5,9,1,3,2,4,13,16,17,15,14,12,10,11)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I now try to reorder my dendrogram and store the reordered one as &lt;code&gt;rx&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; rx &amp;lt;- reorder(x, y, agglo.FUN=mean)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I check the order of &lt;code&gt;rx&lt;/code&gt;, and it is reordered from &lt;code&gt;x&lt;/code&gt;, but it is not reordered in quite the way I want (almost, but not quite):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; order.dendrogram(rx) &#xA;[1] 6 7 8 9 5 4 2 1 3 16 13 17 15 14 12 10 11&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I played around a bit with the &lt;code&gt;agglo.FUN=mean&lt;/code&gt; argument, but never could get the dendrogram to be ordered just like I wanted. The ordering I'm asking for is completely compatible with the dendrogram - I'm just rotating some branches. Does anyone know what I'm doing wrong?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Eric&lt;/p&gt;&#xA;" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-10T20:39:12.800" LastActivityDate="2010-11-10T20:39:12.800" Title="Reordering a dendrogram in R" Tags="&lt;r&gt;" CommentCount="2" FavoriteCount="1" />
  <row Id="4408" PostTypeId="1" CreationDate="2010-11-10T20:13:27.987" Score="2" ViewCount="148" Body="&lt;p&gt;Hey every one,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could anybody provides me a latest material related to Cross validation especially R package?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regards,&#xA;Jacki&lt;/p&gt;&#xA;" OwnerUserId="1534" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-10T20:37:14.640" LastActivityDate="2010-11-10T20:51:05.220" Title="Latest article or new development in cross validation?" Tags="&lt;r&gt;&lt;machine-learning&gt;&lt;cross-validation&gt;" AnswerCount="2" FavoriteCount="2" />
  <row Id="4409" PostTypeId="1" CreationDate="2010-11-10T20:16:58.023" Score="0" ViewCount="46" Body="&lt;p&gt;Hello all,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not sure if this is the right forum to ask this question, so please tell me if I'm in the wrong place.  I need to find out how to calculate a &quot;blended&quot; interest rate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-- sorry wrong forum -- thanks for showing me the door&lt;/p&gt;&#xA;" LastActivityDate="2010-11-10T21:12:25.287" ClosedDate="2010-11-10T22:33:01.443" Title="Blended Interest Rates" Tags="&lt;finance&gt;" CommentCount="2" />
  <row Id="4410" PostTypeId="2" ParentId="4408" CreationDate="2010-11-10T20:22:27.040" Score="4" ViewCount="" Body="&lt;p&gt;Your question is not really precise, but I think the &lt;a href=&quot;http://caret.r-forge.r-project.org/&quot; rel=&quot;nofollow&quot;&gt;caret&lt;/a&gt; package and its associated vignettes may be a good start. Quoting the website, it is&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;a set of functions that attempt to&#xA;  streamline the process for creating&#xA;  predictive models.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In fact, it depends on a lot of other R packages dedicated to ML (see the list of suggested packages on &lt;a href=&quot;http://cran.r-project.org/web/packages/caret/index.html&quot; rel=&quot;nofollow&quot;&gt;CRAN&lt;/a&gt;), but it definitively simplifies the management of cross-validation scheme (k-fold, leave-one-out).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www-stat.stanford.edu/~tibs/ElemStatLearn/&quot; rel=&quot;nofollow&quot;&gt;The Elements of Statistical Learning&lt;/a&gt; (Hastie et al.) is available on-line in its second edition, and all illustrations are done in R. Cross-validation is described at length in Chapter 7.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-11-10T20:22:27.040" />
  <row Id="4411" PostTypeId="2" ParentId="4408" CreationDate="2010-11-10T20:51:05.220" Score="4" ViewCount="" Body="&lt;p&gt;The following is a recent survey: &quot;A survey of cross-validation procedures for model selection&quot; by Sylvain Arlot and Alain Celisse (Statistics Surveys, Volume 4 (2010), 40-79.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The full paper can be downloaded by following the PDF link on &lt;a href=&quot;http://projecteuclid.org/DPubS?service=UI&amp;amp;version=1.0&amp;amp;verb=Display&amp;amp;handle=euclid.ssu/1268143839&quot; rel=&quot;nofollow&quot;&gt;this page&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;&lt;b&gt;Abstract&lt;/b&gt;&lt;br&gt;&lt;br&gt;&#xA;Used to estimate the risk of an estimator or to perform model selection, cross-validation is a widespread strategy because of its simplicity and its (apparent) universality. Many results exist on model selection performances of cross-validation procedures. This survey intends to relate these results to the most recent advances of model selection theory, with a particular emphasis on distinguishing empirical statements from rigorous theoretical results. As a conclusion, guidelines are provided for choosing the best cross-validation procedure according to the particular features of the problem in hand.&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="439" LastActivityDate="2010-11-10T20:51:05.220" CommentCount="1" />
  <row Id="4412" PostTypeId="2" ParentId="1780" CreationDate="2010-11-10T20:53:29.547" Score="0" ViewCount="" Body="&lt;p&gt;I would recommend using Association Rule Learning for this. It allows you to find words that often co-occur. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have a lot of data, it will be much faster than calculating a correlation matrix. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;See my video series on text mining &lt;a href=&quot;http://vancouverdata.blogspot.com/2010/11/text-analytics-with-rapidminer-part-3.html&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;. Includes a tutorial on Association Rules for text. &lt;/p&gt;&#xA;" OwnerUserId="74" LastActivityDate="2010-11-10T20:53:29.547" />
  <row Id="4413" PostTypeId="2" ParentId="4259" CreationDate="2010-11-10T20:58:12.680" Score="1" ViewCount="" Body="&lt;p&gt;As far as books go, &quot;The Elements of Statistical Learning&quot; by Hastie, Tibshirani and Friedman is very good.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The full book is available on the &lt;a href=&quot;http://www-stat.stanford.edu/~tibs/ElemStatLearn/&quot; rel=&quot;nofollow&quot;&gt;authors' web site&lt;/a&gt;; you may want to take a look to see if it is at all suitable for your needs.&lt;/p&gt;&#xA;" OwnerUserId="439" LastActivityDate="2010-11-10T20:58:12.680" CommunityOwnedDate="2010-11-10T20:58:12.680" />
  <row Id="4414" PostTypeId="2" ParentId="4259" CreationDate="2010-11-10T21:11:31.560" Score="1" ViewCount="" Body="&lt;p&gt;As for (on-line) references, I would recommend looking at Andrew Moore's tutorial slides on &lt;a href=&quot;http://www.autonlab.org/tutorials/&quot; rel=&quot;nofollow&quot;&gt;Statistical Data Mining&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are many textbooks on data mining and machine learning; maybe a good starting point is &lt;a href=&quot;http://mitpress.mit.edu/catalog/item/default.asp?ttype=2&amp;amp;tid=3520&quot; rel=&quot;nofollow&quot;&gt;Principles of Data Mining&lt;/a&gt;, by Hand et al., and &lt;a href=&quot;http://mitpress.mit.edu/catalog/item/default.asp?tid=10341&amp;amp;ttype=2&quot; rel=&quot;nofollow&quot;&gt;Introduction to Machine Learning&lt;/a&gt;, by Alpaydin.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-11-10T21:11:31.560" CommunityOwnedDate="2010-11-10T21:11:31.560" />
  <row Id="4415" PostTypeId="2" ParentId="4399" CreationDate="2010-11-10T22:42:02.583" Score="3" ViewCount="" Body="&lt;p&gt;Let $X_t$ be the time between events. Then for very regular events, $X_t$ will be approximately constant. e.g., if you do your laundry every Monday, then $X_t=7$ for all $t$. So you could just use the variance of $X_t$, where the small variance corresponds to highly regular and large variance corresponds to low regularity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update:&lt;/strong&gt;&lt;br&gt;&#xA;If outliers are a problem, or you do not want occasional &quot;misses&quot; to affect the result, use the interquartile range (IQR) instead of the variance. If that removes too many observations, try the difference of the 90th and 10th percentiles.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To normalize, divide by the median (safer than assuming you know the frequency). Thus, one possible measure which should work for you is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;(q_{0.9} - q_{0.1})/q_{0.5}&#xD;&#xA;$$&#xD;&#xA;&#xD;&#xA;where $q_{\alpha}$ denotes the $\alpha$-quantile.&lt;/p&gt;&#xA;" OwnerUserId="159" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-11-11T06:02:57.460" LastActivityDate="2010-11-11T06:02:57.460" CommentCount="6" />
  <row Id="4417" PostTypeId="1" CreationDate="2010-11-11T00:33:28.430" Score="2" ViewCount="89" Body="&lt;p&gt;In Bayesian statistics, it is often mentioned that the posterior distribution is intractable and thus approximate inference must be applied. What are the factors that cause this intractability? &lt;/p&gt;&#xA;" OwnerUserId="1913" LastActivityDate="2010-11-11T06:53:21.377" Title="Intractable posterior distributions" Tags="&lt;bayesian&gt;&lt;approximation&gt;&lt;inference&gt;" AnswerCount="1" />
  <row Id="4418" PostTypeId="1" CreationDate="2010-11-11T01:35:14.477" Score="0" ViewCount="90" Body="&lt;p&gt;Let $Z=(X+Y)/2$, where $X$ and $Y$ are independent normally-distributed random variables with known variances $\sigma^2_X$ and $\sigma^2_Y$ and unknown (and possibly different) means. Given a sample $x_1$ from $X$ and $y_1$ from $Y$, what is the minimum mean squared error estimator of the mean of $Z$? Is there a biased estimator with an MSE improved over the maximum likelihood estimator $\frac{x_1+y_1}{2}$? Can we generalize to $n$ mutually-independent random variables?&lt;/p&gt;&#xA;" LastActivityDate="2010-11-11T06:18:01.563" Title="What is the minimum mean squared error estimator of the mean of two normally-distributed variables?" Tags="&lt;estimation&gt;&lt;mean&gt;&lt;normal-distribution&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="4419" PostTypeId="2" ParentId="898" CreationDate="2010-11-11T02:16:48.133" Score="0" ViewCount="" Body="&lt;p&gt;I'm not sure what sort of data you need to process, but statistical algorithms are used quite frequently in robotics, media art, digital signal processing etc. It may be worthwhile to borrow from these domains.&lt;/p&gt;&#xA;" OwnerUserId="162" LastActivityDate="2010-11-11T02:16:48.133" />
  <row Id="4420" PostTypeId="2" ParentId="4418" CreationDate="2010-11-11T03:30:24.730" Score="1" ViewCount="" Body="&lt;p&gt;Minimize SSE = &amp;Sigma;&lt;sub&gt;i&lt;/sub&gt; (z&lt;sub&gt;i&lt;/sub&gt;-a)&lt;sup&gt;2&lt;/sup&gt; with respect to a and you get the average of the two sample means, so it looks like your MLE is also your minimum MSE estimator, whether you have one (x,y) pair or several.  Not surprising, since the expression for the SSE and the log likelihood function are almost identical.&lt;/p&gt;&#xA;" OwnerUserId="1920" LastActivityDate="2010-11-11T03:30:24.730" />
  <row Id="4421" PostTypeId="2" ParentId="898" CreationDate="2010-11-11T04:28:54.030" Score="0" ViewCount="" Body="&lt;p&gt;you have postrank&#xA;&lt;a href=&quot;http://data.postrank.com/content&quot; rel=&quot;nofollow&quot;&gt;http://data.postrank.com/content&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do you sort what content is current and meaningful? We do it for you. PostRank gathers original content, as well as information at the feed and story level. This includes title, author, language, tags, and related links. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We analyze that data to deliver timely, socially relevant information from millions of online sources. Then we deliver it how you want it. Real-time, custom, filtered access to every piece of content we index, along with its rich metadata&lt;/p&gt;&#xA;" OwnerUserId="1808" LastActivityDate="2010-11-11T04:28:54.030" />
  <row Id="4422" PostTypeId="1" AcceptedAnswerId="4436" CreationDate="2010-11-11T04:33:28.400" Score="4" ViewCount="328" Body="&lt;p&gt;FULL DISCLOSURE: This is homework.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have been provided with a small data set (n=21) the data are messy, looking at it in a scatterplot matrix provides me with little to no insight. I've been provided with 8 variables that are metrics created from a longditudinal study (BI, CONS, CL, CR, ..., VOBI). The other measurements are of mutual fund sales, returns, asset levels, market share, share of sales, and proportion of sales to assets&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/hNTBd.jpg&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Correlations, are everywhere. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;               BI       CONS           CL          CR         QT        COM        CONV        VOBI          s            r           a          ms         ss       share      share2&#xA;BI      1.0000000  0.7620445  0.639830594  0.70384322  0.7741463  0.8451500  0.84704440  0.85003686  0.2106773 -0.238431047  0.36184548  0.40007830  0.4076563  0.31643802 -0.28283564&#xA;CONS    0.7620445  1.0000000  0.933595967  0.96979599  0.9892533  0.9069803  0.96781703  0.93416972  0.2316209 -0.074351798  0.31952292  0.40259511  0.4442877  0.24783884 -0.14788906&#xA;CL      0.6398306  0.9335960  1.000000000  0.88297431  0.8993748  0.8133169  0.89922684  0.81132166  0.1200420 -0.001107093  0.22132116  0.26729067  0.3033221  0.07650924 -0.25595278&#xA;CR      0.7038432  0.9697960  0.882974312  1.00000000  0.9788150  0.8965754  0.92335363  0.90848199  0.2934774 -0.119340914  0.35973640  0.46409570  0.5012178  0.32832247 -0.09005985&#xA;QT      0.7741463  0.9892533  0.899374782  0.97881497  1.0000000  0.9216887  0.95458369  0.94848419  0.2826278 -0.108430256  0.35520090  0.43290221  0.4823314  0.31761015 -0.12903075&#xA;COM     0.8451500  0.9069803  0.813316918  0.89657544  0.9216887  1.0000000  0.90302002  0.89682825  0.4305866 -0.255581594  0.50724121  0.55718441  0.5773171  0.40378679 -0.12085524&#xA;CONV    0.8470444  0.9678170  0.899226843  0.92335363  0.9545837  0.9030200  1.00000000  0.96097892  0.1993837 -0.065237725  0.32010735  0.41843335  0.4531298  0.28873934 -0.19668858&#xA;VOBI    0.8500369  0.9341697  0.811321664  0.90848199  0.9484842  0.8968283  0.96097892  1.00000000  0.2424889 -0.087126942  0.30390489  0.40390750  0.4845432  0.36588655 -0.07137107&#xA;s       0.2106773  0.2316209  0.120041993  0.29347742  0.2826278  0.4305866  0.19938371  0.24248894  1.0000000 -0.173034217  0.91766914  0.84673519  0.8596887  0.61299987  0.32072790&#xA;r      -0.2384310 -0.0743518 -0.001107093 -0.11934091 -0.1084303 -0.2555816 -0.06523773 -0.08712694 -0.1730342  1.000000000 -0.22512978 -0.18337773 -0.1030943 -0.17650579  0.51768144&#xA;a       0.3618455  0.3195229  0.221321163  0.35973640  0.3552009  0.5072412  0.32010735  0.30390489  0.9176691 -0.225129778  1.00000000  0.92445370  0.8656139  0.63049461  0.03876774&#xA;ms      0.4000783  0.4025951  0.267290668  0.46409570  0.4329022  0.5571844  0.41843335  0.40390750  0.8467352 -0.183377734  0.92445370  1.00000000  0.9572730  0.77582501  0.08435813&#xA;ss      0.4076563  0.4442877  0.303322147  0.50121775  0.4823314  0.5773171  0.45312978  0.48454322  0.8596887 -0.103094325  0.86561394  0.95727301  1.0000000  0.83931302  0.24371447&#xA;share   0.3164380  0.2478388  0.076509240  0.32832247  0.3176102  0.4037868  0.28873934  0.36588655  0.6129999 -0.176505786  0.63049461  0.77582501  0.8393130  1.00000000  0.20313930&#xA;share2 -0.2828356 -0.1478891 -0.255952782 -0.09005985 -0.1290307 -0.1208552 -0.19668858 -0.07137107  0.3207279  0.517681444  0.03876774  0.08435813  0.2437145  0.20313930  1.00000000 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, I've tried running a number of &quot;tests&quot;, for example:  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;summary.lm(share2 ~ BI + ...) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, none of them provide any reasonable result (mostly negative adjusted R^2). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm wondering, if you had data where it looked like there was no relationships (linear at least). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What would your next steps be? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S: I did try a number of model formulas that contained interaction effects and received much better results (R^2 Ra^2 &gt; 80% and significant f-tests) but not all the interaction effects where significant.  &lt;/p&gt;&#xA;" OwnerUserId="776" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-11-11T06:06:59.780" LastActivityDate="2010-11-11T18:07:36.833" Title="Small sample linear regression: Where to start" Tags="&lt;regression&gt;&lt;homework&gt;&lt;methodology&gt;" AnswerCount="4" CommentCount="2" FavoriteCount="1" />
  <row Id="4423" PostTypeId="2" ParentId="4347" CreationDate="2010-11-11T04:45:20.507" Score="3" ViewCount="" Body="&lt;p&gt;here is the best book I would recomend for the subject:&#xA;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0131011715&quot; rel=&quot;nofollow&quot;&gt;http://www.amazon.com/Fuzzy-Sets-Logic-Theory-Applications/dp/0131011715&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an easy to read book:&#xA;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0671875353&quot; rel=&quot;nofollow&quot;&gt;http://www.amazon.com/Fuzzy-Logic-Revolutionary-Computer-Technology/dp/0671875353&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Besides here are a list of links that might help:&#xA;&lt;a href=&quot;http://www.seattlerobotics.org/encoder/mar98/fuz/flindex.html&quot; rel=&quot;nofollow&quot;&gt;http://www.seattlerobotics.org/encoder/mar98/fuz/flindex.html&lt;/a&gt;&#xA;http://www.fuzzy-logic.com/&#xA;&lt;a href=&quot;http://videolectures.net/acai05_berthold_fl/&quot; rel=&quot;nofollow&quot;&gt;http://videolectures.net/acai05_berthold_fl/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1808" LastActivityDate="2010-11-11T04:45:20.507" />
  <row Id="4424" PostTypeId="2" ParentId="898" CreationDate="2010-11-11T05:30:32.227" Score="0" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Bayesian_network&quot; rel=&quot;nofollow&quot;&gt;Bayesian networks&lt;/a&gt; are perfect for online estimation, and offer a great diversity of models.&lt;/p&gt;&#xA;" OwnerUserId="1709" LastActivityDate="2010-11-11T05:30:32.227" />
  <row Id="4425" PostTypeId="2" ParentId="4422" CreationDate="2010-11-11T05:35:52.333" Score="2" ViewCount="" Body="&lt;p&gt;If you're frustrated with too many correlations, and since you already have your covariance matrix (well almost) you could do a principal components analysis. You'll end up with fewer dimensions, which is probably fine considering your data set size, and what you end up with won't be intercorrelated anymore.&lt;/p&gt;&#xA;" OwnerUserId="1951" LastActivityDate="2010-11-11T05:35:52.333" CommentCount="6" />
  <row Id="4426" PostTypeId="2" ParentId="4383" CreationDate="2010-11-11T05:57:00.530" Score="3" ViewCount="" Body="&lt;p&gt;I'd add one point.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Be aware of the distinction between group (e.g., comparing group means over time) and individual level measurement (e.g., correlating scores on the scale with other scales at the individual-level).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Reliability applies differently to the two levels.&#xA;Perhaps the following simplification helps:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Reliability of group-level measurement is heavily influenced by the number of participants you have and the degree to which there is true variability at the group-level.&lt;/li&gt;&#xA;&lt;li&gt;Reliability of individual-level measurement is heavily influenced by the number of items you have and the degree to which individuals truly vary.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-11-11T05:57:00.530" CommentCount="1" />
  <row Id="4427" PostTypeId="2" ParentId="1980" CreationDate="2010-11-11T06:22:29.180" Score="3" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://www.econ.uiuc.edu/~roger/research/repro/&quot; rel=&quot;nofollow&quot;&gt;Koenker and Zeileis&lt;/a&gt; provide a webpage with a relatively complete example.&#xA;They share:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Rnw (Sweave code)&lt;/li&gt;&#xA;&lt;li&gt;R analysis code&lt;/li&gt;&#xA;&lt;li&gt;Final PDF&lt;/li&gt;&#xA;&lt;li&gt;Discussion of version control issues&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-11-11T06:22:29.180" />
  <row Id="4428" PostTypeId="2" ParentId="4417" CreationDate="2010-11-11T06:53:21.377" Score="2" ViewCount="" Body="&lt;p&gt;The issue is mainly that Bayesian analysis involves &lt;em&gt;integrals&lt;/em&gt;, often multidimensional ones in realistic problems, and it's these integrals that are typically intractable analytically (except in a few special cases requiring the use of conjugate priors). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;By contrast, much of non-Bayesian statistics is based on &lt;a href=&quot;http://en.wikipedia.org/wiki/Maximum_likelihood&quot; rel=&quot;nofollow&quot;&gt;maximum likelihood&lt;/a&gt; -- finding the maximum of a (usually multidimensional) function, which involves knowledge of its &lt;a href=&quot;http://en.wikipedia.org/wiki/Derivative&quot; rel=&quot;nofollow&quot;&gt;derivatives&lt;/a&gt;, i.e. differentiation. Even so numerical methods are used in many more complex problems, but it's possible to get further more often without them, and the numerical methods can be simpler (even if less simple ones may perform better in practice).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I'd say it comes down to the fact that differentiation is more tractable than integration. &lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-11-11T06:53:21.377" />
  <row Id="4429" PostTypeId="1" CreationDate="2010-11-11T06:59:46.883" Score="12" ViewCount="297" Body="&lt;p&gt;It is helpful to study the data analysis code of experts.&#xA;I've recently been perusing &lt;a href=&quot;https://github.com/&quot; rel=&quot;nofollow&quot;&gt;github&lt;/a&gt; and there are a number of people sharing data analysis code there. This includes a few R Packages (which of course are available directly from CRAN), but also several examples of reproducible research, particularly using R (&lt;a href=&quot;https://github.com/languages/R&quot; rel=&quot;nofollow&quot;&gt;see this R list on github&lt;/a&gt;). &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Who are good people to follow on github to learn about best practice in data analysis?&lt;/li&gt;&#xA;&lt;li&gt;Optionally, what kind of code do they share and why is this useful?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-11-12T01:15:34.080" LastActivityDate="2010-11-12T10:35:00.423" CommunityOwnedDate="2010-11-11T15:07:15.367" Title="Who to follow on github to learn about best practice in data analysis?" Tags="&lt;best-practices&gt;" AnswerCount="3" FavoriteCount="11" />
  <row Id="4430" PostTypeId="2" ParentId="4429" CreationDate="2010-11-11T07:35:57.207" Score="8" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;https://github.com/hadley&quot; rel=&quot;nofollow&quot;&gt;Hadley Wickham&lt;/a&gt;. He has several exploratory data analysis projects on Github that you can look at (e.g., &quot;data-baby-names&quot;), and given the awesomeness of ggplot2/plyr/reshape, I have a default (but admittedly blind) trust in his best practices, particularly with respect to his own packages.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Plus, you get an early heads up on other projects he's working on!&lt;/p&gt;&#xA;" OwnerUserId="1106" LastActivityDate="2010-11-11T07:35:57.207" CommunityOwnedDate="2010-11-11T15:07:15.367" CommentCount="3" />
  <row Id="4431" PostTypeId="1" AcceptedAnswerId="4486" CreationDate="2010-11-11T07:50:43.590" Score="1" ViewCount="166" Body="&lt;p&gt;Way back when, I used to work in finance, and I remember helping a coworker use some kind of block bootstrap. (I believe the application was: we had weekly data on some financial indicator X, along with weekly data on some stock, and we wanted to measure how well X could be used to predict the stock's movements. And I believe we needed to bring in the bootstrap, because we only had a couple months of weekly data, so we didn't really have many datapoints. I might be misremembering all this, though.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In any case, I totally forget now how the block bootstrap worked, and I want to remember/review/learn more, so can anyone suggest a good tutorial on it? I tried googling, but all I found were some random research papers. I also tried looking in my copy of Efron &amp;amp; Tibshirani's &quot;An Introduction to the Bootstrap Book&quot;, but didn't find anything (unless it's under a name other than &quot;block bootstrap&quot;).&lt;/p&gt;&#xA;" OwnerUserId="1106" LastActivityDate="2010-11-12T19:28:49.717" Title="Resources to learn about block bootstrap in time series analysis" Tags="&lt;time-series&gt;&lt;regression&gt;&lt;bootstrap&gt;" AnswerCount="3" FavoriteCount="1" />
  <row Id="4432" PostTypeId="2" ParentId="4429" CreationDate="2010-11-11T07:59:55.433" Score="3" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://www.diegovalle.net/&quot; rel=&quot;nofollow&quot;&gt;Diego Valle Jones&lt;/a&gt;. His &lt;a href=&quot;https://github.com/diegovalle&quot; rel=&quot;nofollow&quot;&gt;Github&lt;/a&gt;, especially &lt;a href=&quot;https://github.com/diegovalle/Homicide-MX-Drug-War&quot; rel=&quot;nofollow&quot;&gt;analysis of homicides in Mexico&lt;/a&gt; is really interesting.&lt;/p&gt;&#xA;" OwnerUserId="22" LastActivityDate="2010-11-11T07:59:55.433" CommunityOwnedDate="2010-11-11T15:07:15.367" />
  <row Id="4433" PostTypeId="2" ParentId="4431" CreationDate="2010-11-11T08:21:47.770" Score="3" ViewCount="" Body="&lt;p&gt;Try the &lt;a href=&quot;http://fedc.wiwi.hu-berlin.de/xplore/ebooks/html/csa/node132.html&quot; rel=&quot;nofollow&quot;&gt;Handbook of Computational Statistics, Part III, section 2.4&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-11-11T08:21:47.770" />
  <row Id="4434" PostTypeId="2" ParentId="4431" CreationDate="2010-11-11T08:24:39.063" Score="1" ViewCount="" Body="&lt;p&gt;The textbook by &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0387293175&quot; rel=&quot;nofollow&quot;&gt;Shumway and Stoffer&lt;/a&gt; has a short section on bootstrapping time series (state-space) models. Also you may look to:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Pfeffermann, D. and Tiller, R. (2005) Bootstrap approximation to prediction MSE for State-Space models with estimated parameters, &lt;em&gt;Journal of Time Series Analisys&lt;/em&gt;, 26, 893-916, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;and references therein.&lt;/p&gt;&#xA;" OwnerUserId="892" LastActivityDate="2010-11-11T08:24:39.063" />
  <row Id="4435" PostTypeId="2" ParentId="4429" CreationDate="2010-11-11T08:57:25.920" Score="4" ViewCount="" Body="&lt;p&gt;I also follow &lt;a href=&quot;http://stats.stackexchange.com/users/303/john-myles-white&quot;&gt;John Myles White&lt;/a&gt;'s GitHub &lt;a href=&quot;https://github.com/johnmyleswhite&quot; rel=&quot;nofollow&quot;&gt;repository&lt;/a&gt;. There are several data-oriented projects, but also interesting stuff for R developers:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/johnmyleswhite/ProjectTemplate&quot; rel=&quot;nofollow&quot;&gt;ProjectTemplate&lt;/a&gt;, a template system for building R project;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/johnmyleswhite/log4r&quot; rel=&quot;nofollow&quot;&gt;log4r&lt;/a&gt;, a logging system.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-12T10:35:00.423" LastActivityDate="2010-11-12T10:35:00.423" CommunityOwnedDate="2010-11-11T15:07:15.367" />
  <row Id="4436" PostTypeId="2" ParentId="4422" CreationDate="2010-11-11T09:20:18.010" Score="5" ViewCount="" Body="&lt;p&gt;I'd probably take a look at a ridge regression or, better, the lasso. These techniques are often used when there is multicolinearity. There are several options for doing this in R: See the Regularized and Shrinkage Methods section of the &lt;a href=&quot;http://cran.r-project.org/web/views/MachineLearning.html&quot; rel=&quot;nofollow&quot;&gt;Machine Learning &amp;amp; Statistical Learning&lt;/a&gt; Task View on CRAN.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You don't have enough data to start thinking about some of the techniques listed in other sections of that Task View.&lt;/p&gt;&#xA;" OwnerUserId="1390" LastActivityDate="2010-11-11T09:20:18.010" CommentCount="2" />
  <row Id="4437" PostTypeId="1" AcceptedAnswerId="4438" CreationDate="2010-11-11T09:26:39.780" Score="-1" ViewCount="242" Body="&lt;p&gt;so we have a0:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;a0[1,1:2]&#xA;l11      l12&#xA;-0.921 0.389593&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then; &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; is.numeric(a0[1,1:2])&#xA;[1] FALSE&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Ok, the text file containing them is a bit of a mess. Then:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; as.numeric(a0[1,1:2])&#xA;[1] 131   3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I &lt;em&gt;know&lt;/em&gt; there was a trick to solve that. I just can't remember what it was...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: sample file: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; -0.921      0.389593    0.99998742210431    -0.00501553917135373    0.999984216926007   -0.00561835375178836   1   2.36    10  2   0.05    1   1&#xA; 0.923842    0.382812    0.999998286086073   -0.00185143860707771    0.999999995689246   9.28520798418186e-05   0   2.03    10  2   0.05    2   1&#xA;     -0.82904    0.559261    0.905909593351804   -0.423471142668741      0.899659482046882   -0.436592277031025     1   1.51    10  2   0.05    3   1&#xA; -0.796621   0.604487    0.852012998984401   -0.523520629547306      0.882445298428051   -0.470415024507325     0   2.53    10  2   0.05    4   1&#xA; 0.836046    0.54875     0.9211906384523     0.389111561930306   0.926385125966013   0.376577479928014  1   2.66    10  2   0.05    5   1&#xA; -0.873104   -0.487579   0.942147732871282   0.335197925777448   0.957896161639222   0.287114861191206  0   2.5 10  2   0.05    6   1&#xA; -0.728606   0.684948    0.838364634070403   -0.545109842453794      0.770224459340949   -0.637772908042465     1   3.97    10  2   0.05    7   1&#xA; 0.759379    -0.650765   0.842087096982091   -0.539341562552223      0.765143566165036   -0.643859707666391     0   1.13    10  2   0.05    8   1&#xA; -0.497911   -0.867269   0.480229829449393   0.877142697003747   0.225403470821452   0.974265505569012  1   0.62    10  2   0.05    9   1&#xA; 0.465581    0.885042    0.173457702239973   0.98484131997679    0.159038890626454   0.987272318698497  0   0.79    10  2   0.05    10  1&#xA; -0.772559   -0.634978   0.866527018491628   0.499130169619119   0.859460127819      0.511202786269155  1   19.53   10  2   0.05    11  1&#xA; 0.81446     0.580278    0.943553667636478   0.331219679804432   0.895967382559323   0.444119859260758  0   1.73    10  2   0.05    12  1&#xA; -0.792377   0.610095    0.865518659616982   -0.500876681284748      0.877697131009408   -0.479215761654241     1   1.24    10  2   0.05    13  1&#xA; 0.844213    -0.536081   0.899692596172177   -0.436524034152723      0.906010964105467   -0.423254217841573     0   2.02    10  2   0.05    14  1&#xA; 0.421542    0.906835    0.103422987236111   0.99463746446188    0.102255903720432   0.994758126458044  1   0.47    10  2   0.05    15  1&#xA; 0.409305    0.912408    0.0729480114866381      0.997335744681873   0.0838704194992194      0.996476669437386  0   0.45    10  2   0.05    16  1&#xA; -0.664573   -0.747275   0.58764731149828    0.809117196263214   0.603464234608522   0.797390066120936  1   0.85    10  2   0.05    17  1&#xA; 0.599777    0.800191    0.524086235602366   0.851665202795172   0.567148160033905   0.823615786984536  0   1.32    10  2   0.05    18  1&#xA; -0.397025   -0.917846   0.030139624142486   0.999545698333273   0.0311215453692834      0.999515607388813  1   0.46    10  2   0.05    19  1&#xA; -0.393222   -0.919468   0.0278258557873373      0.999612785907474   0.0265821201726647      0.999646633009448  0   0.42    10  2   0.05    20  1&#xA; 0.772559    0.634978    0.924006045897534   0.38237785911949    0.908075241643739   0.418807062397072  1   8.12    20  2   0.05    1   2&#xA; -0.807518   -0.589907   0.947419893700908   0.319993039017663   0.96499482496042    0.262268922672146  0   5.8 20  2   0.05    2   2&#xA; V1      V2      V3      V4      V5      V6     NA  NA  NA  NA  NA  NA  NA&#xA; -0.526762   0.850092    -0.311928648717808      0.95010552998553    -0.316412744506619      0.94862161851488   0   3.25    20  2   0.05    4   2&#xA; -0.5161     -0.856543   0.309033921455991   0.951051016186583   0.248027721702134   0.96875293510123   1   1.82    20  2   0.05    5   2&#xA; 0.491867    0.870721    0.268019541588267   0.96341347578639    0.167808915882198   0.98581954116889   0   2.69    20  2   0.05    6   2&#xA; 0.58991     0.807579    0.51533762936385    0.856987238972464   0.421665311535527   0.906751545379244  1   1.91    20  2   0.05    7   2&#xA; -0.549902   -0.835295   0.442022781946357   0.897003823983155   0.390584116545383   0.920567242466547  0   2.06    20  2   0.05    8   2&#xA; 0.800218    -0.599709   0.852476180705526   -0.522766067500292      0.833708092721497   -0.552205411174757     1   47.26   20  2   0.05    9   2&#xA; 0.837387    -0.546652   0.910736035285173   -0.41298895146607   0.863018073213623   -0.505173044912974     0   5.36    20  2   0.05    10  2&#xA; 0.46345     0.886149    0.179828914086857   0.98369790162343    0.176294786076906   0.984337415931193  1   1.84    20  2   0.05    11  2&#xA; V1      V2      V3      V4      V5      V6     NA  NA  NA  NA  NA  NA  NA&#xA;     0.759379    0.650765    0.817799312011448   0.575503505874294   0.824911507994246   0.565261889727814  1   7.7 20  2   0.05    13  2&#xA; 0.793956    0.608013    0.822124323319108   0.56930799836916    0.812590005961236   0.582835724893317  0   3.8 20  2   0.05    14  2&#xA; 0.706746    0.707516    0.70298536741427    0.711204311855196   0.695840302062382   0.718196542755348  1   5.1 20  2   0.05    15  2&#xA; 0.663932    0.747827    0.717210127099939   0.696856967809958   0.711000892677057   0.70319110532801   0   4.17    20  2   0.05    16  2&#xA; 0.813568    -0.581494   0.904931790793546   -0.425556640191628      0.912655929947496   -0.408728704071137     1   4.24    20  2   0.05    17  2&#xA; -0.836046   0.54875     0.924852673074549   -0.380325561995065      0.92915112499605    -0.369700131077304     0   3.78    20  2   0.05    18  2&#xA; V1      V2      V3      V4      V5      V6     NA  NA  NA  NA  NA  NA  NA&#xA; 0.586824    -0.809738   -0.610122851736008      0.792306825535108   -0.588615757355873      0.80841294533943   0   3.07    20  2   0.05    20  2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="603" LastEditorUserId="603" LastEditorDisplayName="" LastEditDate="2010-11-11T09:45:06.437" LastActivityDate="2010-11-11T10:19:23.073" Title="R and as.numeric()" Tags="&lt;r&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="4438" PostTypeId="2" ParentId="4437" CreationDate="2010-11-11T09:32:00.040" Score="3" ViewCount="" Body="&lt;p&gt;This is because you have read the numbers as factors; if you use &lt;code&gt;read.table&lt;/code&gt;, try &lt;code&gt;header=T&lt;/code&gt; or restructure the data before read. Some sample of the file should be helpful to resolve it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Workaround would be to first convert factors to strings using &lt;code&gt;as.character&lt;/code&gt; and then back to numbers with &lt;code&gt;as.numeric&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: Code working for provided example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;readLines('&amp;lt;filename&amp;gt;')-&amp;gt;z&#xA;read.table(textConnection(z[-grep('V',z)]))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="88" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-11-11T10:11:26.683" LastActivityDate="2010-11-11T10:11:26.683" CommentCount="6" />
  <row Id="4439" PostTypeId="2" ParentId="4437" CreationDate="2010-11-11T10:11:18.593" Score="3" ViewCount="" Body="&lt;p&gt;Would that help?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; a &amp;lt;- as.data.frame(matrix(scan(&quot;1.txt&quot;, what=&quot;character&quot;, &#xA;                                 na.strings=c(&quot;NA&quot;,paste(&quot;V&quot;,1:6,sep=&quot;&quot;))), &#xA;                     nc=13, byrow=T))&#xA;&amp;gt; class(a[,1])&#xA;[1] &quot;factor&quot;&#xA;&amp;gt; for (i in 1:ncol(a)) a[,i] &amp;lt;- as.numeric(as.character(a[,i]))&#xA;&amp;gt; class(a[,1])&#xA;[1] &quot;numeric&quot;&#xA;&amp;gt; summary(a)  # should work here&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The way you import data doesn't matter so much; I think the critical part if to convert value as character then as numeric (this allows to convert levels of a factor to their numerical counterparts).&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-11T10:19:23.073" LastActivityDate="2010-11-11T10:19:23.073" CommentCount="2" />
  <row Id="4440" PostTypeId="2" ParentId="4422" CreationDate="2010-11-11T10:40:28.693" Score="4" ViewCount="" Body="&lt;p&gt;I find @ucfagls's idea most appropriate here, since you have very few observations and a lot of variables. Ridge regression should do its job for prediction purpose.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another way to analyse the data would be to rely on &lt;a href=&quot;http://en.wikipedia.org/wiki/Partial_least_squares_regression&quot; rel=&quot;nofollow&quot;&gt;PLS regression&lt;/a&gt; (in this case, PLS1), which bears some idea with regression on PCA scores but seems more interesting in your case. As multicollinearity might be an issue there, you can look at &lt;em&gt;sparse&lt;/em&gt; solution (see e.g., the &lt;a href=&quot;http://cran.r-project.org/web/packages/spls/index.html&quot; rel=&quot;nofollow&quot;&gt;spls&lt;/a&gt; or the &lt;a href=&quot;http://cran.r-project.org/web/packages/mixOmics/index.html&quot; rel=&quot;nofollow&quot;&gt;mixOmics&lt;/a&gt; R packages).&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-11-11T10:40:28.693" />
  <row Id="4441" PostTypeId="1" AcceptedAnswerId="4444" CreationDate="2010-11-11T10:41:58.623" Score="4" ViewCount="177" Body="&lt;p&gt;Hi, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am collecting longitudinal data using for 4 time waves.  Although the survey is administrated to the same population, different individuals may decide to complete it at each time point. As a result there are a number of individuals that only completed it once, others that completed it twice, some that completed it three times and others who participated in all four waves. For example at the moment there are about 2000 participants in time 1 and 1900 for time 2 but only 1200 that participated in both time 1 and time 2 (at the moment i am still collecting data for time 3 so i don't know yet what the final matched sample will be). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data are from different organizations so I would like to model this with using mixed effects with the lmer in R. e.g. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;&#xA;lmer(outcome~&quot;some repeated variables&quot;+&quot;organization level variables&quot;+timewave+(timewave|subject)+(1|organizations))&#xA;&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My questions are &#xA;&#xA;&lt;li&gt; Do i need to remove individuals who completed it only once or twice to use a random slope for time? &#xA;&lt;li&gt; Is it meaningful to also try to fit a quadratic effect for time given that there are only 4 waves? (and would i need to remove subjects that have not participated in all four?)&#xA;&lt;/li&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;George&lt;/p&gt;&#xA;" OwnerUserId="1871" LastActivityDate="2010-11-11T13:17:09.457" Title="How many data points do we need for mixed effects longitudinal data?" Tags="&lt;r&gt;&lt;mixed-model&gt;&lt;panel-data&gt;&lt;longitudinal&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="3" />
  <row Id="4442" PostTypeId="2" ParentId="4403" CreationDate="2010-11-11T12:58:08.970" Score="-1" ViewCount="" Body="&lt;p&gt;Sounds like a classic data mining task:&#xA;Y_i = whether person i gets the question right, &#xA;X_i (vector) = set of past performance of person i.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using a set of past data on n people (i=1,...,n), you can fit a predictive model of the sort Y = function of X.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A variety of models can be used to predict the performance of a new person on this question. If you are short of data, then logistic regression and discriminant analysis would be good choices. If you have data on lots of people, then you could try more data-driven methods such as classification trees, k-nearest neighbors, or neural nets.&lt;/p&gt;&#xA;" OwnerUserId="1945" LastActivityDate="2010-11-11T12:58:08.970" CommentCount="1" />
  <row Id="4443" PostTypeId="2" ParentId="4337" CreationDate="2010-11-11T13:12:53.023" Score="0" ViewCount="" Body="&lt;p&gt;Hi Endy and all,&#xA;Aside from my practical statistical suggestion, I wanted to raise a slightly different issue: I realize that the cinema's goal is to maximize revenues, and of course the analysis (and strategy) can be geared towards that goal. However, I would like to suggest a broader, holistic view that companies as well as analysts should consider: the overall benefit. In this case, we can consider the value of the gaming addition to cinema goers. Are they happier or more satisfied with the overall experience? (this can be evaluated, e.g., via a quick questionnaire). Or, if the gaming is educational, for instance, then perhaps there is added benefit to those playing? I recall that in several cinemas in the U.S. there are word games on the screen before a movie starts. These can be perceived as fun and educational and could therefore be value added. In fact, if movie goers perceive the gaming service as value added, then they will likely choose this cinema over others and perhaps even visit it more frequently.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I am trying to say is that it is useful to define &quot;success&quot; in a broad manner and to think big. In the end, success will depend also on the wellness of the &quot;customers&quot; and the impact of &quot;treatments&quot; on society, culture, the environment, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sorry if this is too philosophical, but I have had so many MBA students maximizing short-term financial gains and too few thinking of issues that are not monetary. Yet, data mining and statistics can be used for broader causes.&lt;/p&gt;&#xA;" OwnerUserId="1945" LastActivityDate="2010-11-11T13:12:53.023" CommentCount="2" />
  <row Id="4444" PostTypeId="2" ParentId="4441" CreationDate="2010-11-11T13:17:09.457" Score="6" ViewCount="" Body="&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;No&lt;/strong&gt;, you don't need to remove individuals with data for only only one (or only a limited number) of timepoints. You're right to think that individuals with only one timepoint contribute nothing to estimation of the slope but they contribute to estimation of the intercept and you want to estimate both jointly. The maths and the algorithm deal with this  so you don't need to worry about it, and you're more likely to make errors than the programmers of lmer if you try to second-guess things by dropping observations you don't think will contribute.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Yes&lt;/strong&gt;, you could fit a quadratic effect for time with 4 timepoints. In fact, if you designed an experiment to look for a quadratic effect, you might well &lt;em&gt;choose&lt;/em&gt; to have 4 timepoints. (3 would  in principle maximise your power, but 4 allows 1 d.f. to test for fit of the quadratic curve). Clearly you need at least some people to participate in at least 3 waves. But as above, don't remove subjects that didn't.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-11-11T13:17:09.457" />
  <row Id="4445" PostTypeId="1" AcceptedAnswerId="4449" CreationDate="2010-11-11T14:38:22.373" Score="10" ViewCount="293" Body="&lt;p&gt;A database of (population, area, shape) can be used to map population density by assigning a constant value of population/area to each shape (which is a polygon such as a Census block, tract, county, state, whatever).  Populations are usually not uniformly distributed within their polygons, however.  &lt;a href=&quot;http://en.wikipedia.org/wiki/Dasymetric_map&quot; rel=&quot;nofollow&quot;&gt;Dasymetric mapping&lt;/a&gt; is the process of refining these density estimates by means of auxiliary data.  It is an important problem in the social sciences as &lt;a href=&quot;http://www.springerlink.com/content/cw2x27r34417877h/&quot; rel=&quot;nofollow&quot;&gt;this recent review&lt;/a&gt; indicates. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose, then, that we have available an auxiliary map of land cover (or any other discrete factor).  In the simplest case we can use obviously uninhabitable areas like waterbodies to delineate where the population isn't and, accordingly, assign all the population to the remaining areas.  More generally, each Census unit $j$ is carved into $k$ portions having surface areas $x_{ji}$, $i = 1, 2, \ldots, k$.  Our dataset is thereby augmented to a list of tuples&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$(y_{j}, x_{j1}, x_{j2}, \ldots, x_{jk})$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $y_{j}$ is the population (assumed measured without error) in unit $j$ and--although this is not strictly the case--we may assume every $x_{ji}$ is also exactly measured.  In these terms, the objective is to partition each $y_{j}$ into a sum&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ y_j = z_{j1} + z_{j2} + \cdots + z_{jk} $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where each $z_{ji} \ge 0$ and $z_{ji}$ estimates the population within unit $j$ residing in land cover class $i$.  The estimates need to be unbiased.  This partition refines the population density map by assigning the density $z_{ji}/x_{ji}$ to the intersection of the $j^{\text{th}}$ Census polygon and the $i^{\text{th}}$ land cover class.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This problem differs from standard regression settings in salient ways:  &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The partitioning of each $y_{j}$ must be exact.  &lt;/li&gt;&#xA;&lt;li&gt;The components of every partition must be non-negative.  &lt;/li&gt;&#xA;&lt;li&gt;There is (by assumption) no error in any of the data: all population counts $y_{j}$ and all areas $x_{ji}$ are correct.  &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;There are many approaches to a solution, such as the &quot;&lt;a href=&quot;http://astro.temple.edu/~jmennis/pubs/mennis_cagis06.pdf&quot; rel=&quot;nofollow&quot;&gt;intelligent dasymetric mapping&lt;/a&gt;&quot; method, but all those I have read about have &lt;em&gt;ad hoc&lt;/em&gt; elements and an obvious potential for bias.  I am seeking answers that suggest creative, computationally tractable statistical methods.  The immediate application concerns a collection of &lt;em&gt;c.&lt;/em&gt; $10^{5}$ - $10^{6}$ Census units averaging 40 people apiece (although a sizable fraction have 0 people) and about a dozen land cover classes.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="123" LastEditorDisplayName="" LastEditDate="2010-11-12T03:03:10.257" LastActivityDate="2011-01-03T18:23:17.097" Title="Model for population density estimation" Tags="&lt;modeling&gt;&lt;unbiased-estimator&gt;&lt;spatial&gt;" AnswerCount="3" CommentCount="2" FavoriteCount="3" />
  <row Id="4446" PostTypeId="1" AcceptedAnswerId="4448" CreationDate="2010-11-11T14:59:10.677" Score="2" ViewCount="121" Body="&lt;p&gt;The standard sum of squares as I know it is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&amp;Sigma; (X-m)&lt;sup&gt;2&lt;/sup&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where m is the mean.&#xA;I ran into a different one which can be written two ways:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&amp;Sigma;(X&lt;sup&gt;2&lt;/sup&gt;) - (&amp;Sigma;X)&lt;sup&gt;2&lt;/sup&gt;/n = &amp;Sigma;(X&lt;sup&gt;2&lt;/sup&gt;) - m &amp;Sigma;X&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I believe the latter is called the &quot;correction term for the mean&quot; (e.g. &lt;a href=&quot;http://www.itl.nist.gov/div898/handbook/prc/section4/prc431.htm&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;).  My algebra seems to be inadequate to show these are equivalent, so I was looking for a derivation.  Thanks.&lt;/p&gt;&#xA;" OwnerUserId="1959" LastActivityDate="2010-11-12T12:37:14.680" Title="Sum of squares two ways, how are they connected?" Tags="&lt;basic-concepts&gt;" AnswerCount="2" />
  <row Id="4448" PostTypeId="2" ParentId="4446" CreationDate="2010-11-11T15:15:36.757" Score="4" ViewCount="" Body="&lt;p&gt;Expanding the square we get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\sum_i(X_i-m)^2 = \sum_i(X_i^2 + m^2 - 2 X_i m)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\sum_i(X_i-m)^2 = \sum_i{X_i^2} + \sum_i{m^2} - 2 \sum_i{X_i m}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since $m$ is a constant, we have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\sum_i(X_i-m)^2 = \sum_i{X_i^2} + n m^2 - 2 m \sum_i{X_i}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\sum_i{X_i} = n m$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\sum_i(X_i-m)^2 = \sum_i{X_i^2} + n m^2 - 2 n m^2$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which on simplifying gets us:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\sum_i(X_i-m)^2 = \sum_i{X_i^2} - n m^2$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, we get can rewrite the rhs of the above in two ways:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\sum_i{X_i^2} - m (n m)  = \sum_i{X_i^2} - m \sum_i{X_i}$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(as $n m = \sum_i{X_i}$)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\sum_i{X_i^2} - n (m)^2  = \sum_i{X_i^2} - \frac{(\sum_i{X_i})^2}{n}$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(as $m = \frac{\sum_i{X_i}}{n}$)&lt;/p&gt;&#xA;" LastActivityDate="2010-11-11T15:15:36.757" CommentCount="1" />
  <row Id="4449" PostTypeId="2" ParentId="4445" CreationDate="2010-11-11T16:13:57.317" Score="3" ViewCount="" Body="&lt;p&gt;You might want to check &lt;a href=&quot;http://www.informatik.uni-trier.de/~ley/db/indices/a-tree/l/Langford%3aMitchel.html&quot; rel=&quot;nofollow&quot;&gt;work&lt;/a&gt; of Mitchel Langford on dasymetric mapping. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;He build rasters representing population distribution of Wales and some of his methodological approaches might be useful here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Update: You might also have a look at work of &lt;a href=&quot;http://www.temple.edu/gus/mennis/index.htm&quot; rel=&quot;nofollow&quot;&gt;Jeremy Mennis&lt;/a&gt; (especially &lt;a href=&quot;http://www.blackwell-compass.com/subject/geography/article_view?article_id=geco_articles_bpl220&quot; rel=&quot;nofollow&quot;&gt;these&lt;/a&gt; &lt;a href=&quot;http://onlinelibrary.wiley.com/doi/10.1111/0033-0124.10042/abstract&quot; rel=&quot;nofollow&quot;&gt;two&lt;/a&gt; articles).&lt;/p&gt;&#xA;" OwnerUserId="22" LastEditorUserId="22" LastEditorDisplayName="" LastEditDate="2010-12-13T18:08:03.437" LastActivityDate="2010-12-13T18:08:03.437" CommentCount="1" />
  <row Id="4450" PostTypeId="2" ParentId="4445" CreationDate="2010-11-11T16:42:34.450" Score="2" ViewCount="" Body="&lt;p&gt;Interesting question. Here is a tentative stab at approaching this from a statistical angle. Suppose that we come up with a way to assign a population count to each area $x_{ji}$. Denote this relationship as below:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$z_{ji} = f(x_{ji},\beta)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Clearly, whatever functional form we impose on $f(.)$ will be at best an approximation to the real relationship and thus the need to incorporate error into the above equation. Thus, the above becomes:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$z_{ji} = f(x_{ji},\beta) + \epsilon_{ji}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\epsilon_{ji} \sim N(0,\sigma^2)$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The distributional error assumption on the error term is for illustrative purposes. If necessary we can change it as appropriate. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, we need an exact decomposition of $y_{ji}$. Thus, we need to impose a constraint on the error terms and the function $f(.)$ as below:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\sum_i{\epsilon_{ji}} = 0$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\sum_i{f(x_{ji},\beta)} = y_j$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Denote the stacked vector of ${z_{ji}}$ by $z_j$ and the stacked deterministic terms of ${f(x_{ji},\beta)}$ by $f_j$. Thus, we have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$z_j \sim N(f_j,\sigma^2 I) I({f_j}' e = y_j) I((z_j-f_j)' e = 0)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$e$ is a vector of ones of appropriate dimension.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first indicator constraint captures the idea that the sum of the deterministic terms should sum to $y_j$ and the second one captures the idea that the error residuals should sum to 0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Model selection is trickier as we are decomposing the observed $y_j$ exactly. Perhaps, a way to approach model selection is to choose the model that yields the lowest error variance i.e., the one that yields the lowest estimate of $\sigma^2$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit 1&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thinking some more the above formulation can be simplified as it has more constraints than needed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$z_{ji} = f(x_{ji},\beta) + \epsilon_{ji}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\epsilon_{ji} \sim N(0,\sigma^2)$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Denote the stacked vector of ${z_{ji}}$ by $z_j$ and the stacked deterministic terms of ${f(x_{ji},\beta)}$ by $f_j$. Thus, we have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$z_j \sim N(f_j,\sigma^2 I) I({z_j}' e = y_j)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$e$ is a vector of ones of appropriate dimension.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The constraint on $z_j$ ensures an exact decomposition.&lt;/p&gt;&#xA;" LastActivityDate="2010-11-11T17:01:08.360" CommentCount="1" />
  <row Id="4451" PostTypeId="1" CreationDate="2010-11-11T17:50:04.680" Score="5" ViewCount="789" Body="&lt;p&gt;I am looking for social network datasets (twitter, friendfeed, facebook, lastfm, etc.) for classification tasks, preferibly in arff format. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My searches at UCI and Google wasn't successful so far.. any suggestions?&lt;/p&gt;&#xA;" LastActivityDate="2011-02-17T10:55:37.620" CommunityOwnedDate="2010-11-14T14:49:35.420" Title="Social network datasets" Tags="&lt;classification&gt;&lt;dataset&gt;" AnswerCount="9" CommentCount="1" FavoriteCount="10" />
  <row Id="4452" PostTypeId="2" ParentId="4422" CreationDate="2010-11-11T18:07:36.833" Score="3" ViewCount="" Body="&lt;p&gt;It seems to me that the only thing worth doing here is testing a very focussed hypothesis, if you have one. But it seems like you don't. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;With so few cases and so many variables, anything else would (in my opinion) be a fishing expedition. That could be a bit useful, perhaps, to generate an hypothesis to test with new data. But any results from a multivariate unfocussed analysis of these data is likely to be a false positive coincidental finding that probably won't hold up with new data. &lt;/p&gt;&#xA;" OwnerUserId="25" LastActivityDate="2010-11-11T18:07:36.833" CommentCount="1" />
  <row Id="4453" PostTypeId="1" AcceptedAnswerId="4526" CreationDate="2010-11-11T19:09:57.167" Score="3" ViewCount="251" Body="&lt;p&gt;Suppose I have a set $\mathcal{S}$ of $N$ distinct items. Now consider the set $\mathcal{P}$ of all possible pairs that I can draw from $S$. Naturally, $|\mathcal{P}| = \binom{N}{2}$. Now when I draw $k$ items (pairs) from $\mathcal{P}$ with a uniform distribution, what is the expected number of distinct items from $S$ in those $k$ pairs?&lt;/p&gt;&#xA;" OwnerUserId="977" LastActivityDate="2010-11-30T18:21:49.360" Title="How to calculate the expected number of distinct items when drawing pairs?" Tags="&lt;binomial&gt;&lt;expected-value&gt;" AnswerCount="1" />
  <row Id="4454" PostTypeId="1" CreationDate="2010-11-11T19:24:19.787" Score="2" ViewCount="223" Body="&lt;p&gt;Are there any decent tools for writing/designing questionnaires before providing to programmers? Currently Microsoft Word is being used and tracking changes and keeping it standardized has become a headache.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt;: I think I'm being a little misunderstood here. Here's a scenario: A client speaks to a statistician/expert in survey design so they can design a survey to receive feedback from their customers. Once that survey is designed (not in the fashion sense) they might give it to an artist and programmer to actually create the survey (for the web or whatever). I am wondering what some good ways to communicate the logical design of the survey to the programmer might be. &lt;/p&gt;&#xA;" OwnerUserId="305" LastEditorUserId="305" LastEditorDisplayName="" LastEditDate="2010-11-18T14:34:15.647" LastActivityDate="2010-11-18T14:34:15.647" Title="What tool do you use to communicate your survey questionnaires?" Tags="&lt;software&gt;&lt;survey&gt;" AnswerCount="5" CommentCount="5" />
  <row Id="4455" PostTypeId="2" ParentId="4454" CreationDate="2010-11-11T20:18:09.713" Score="1" ViewCount="" Body="&lt;p&gt;A combination of Perl + CGI is generally interesting for small surveys/questionnaires (because I hate PHP + MySQL). A gentle introduction can be found in &lt;a href=&quot;http://www.web-research-design.net/&quot; rel=&quot;nofollow&quot;&gt;How to Conduct Behavioral Research over the Internet: A Beginner's Guide to Html and Cgi/Perl&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, I think that Ruby and Rails should provide very handy tools for that particular purpose. I can think of &lt;a href=&quot;https://github.com/breakpointer/surveyor&quot; rel=&quot;nofollow&quot;&gt;surveyor&lt;/a&gt;, for example. I'm quite sure there are similar tools in Python.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for an all-in-one system (no need to program anything, multiple and linked form available, automatic mailing, etc.), there's &lt;a href=&quot;http://www.limesurvey.org/&quot; rel=&quot;nofollow&quot;&gt;Lime Survey&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For off-line questionnaires, I would prefer $\LaTeX$ or Docbook.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-11T20:38:22.387" LastActivityDate="2010-11-11T20:38:22.387" CommentCount="3" />
  <row Id="4456" PostTypeId="2" ParentId="4334" CreationDate="2010-11-11T20:45:50.343" Score="4" ViewCount="" Body="&lt;p&gt;You can build this model with AD Model Builder's random effects package.&#xA;This is free software available at &lt;a href=&quot;http://admb-project.org&quot; rel=&quot;nofollow&quot;&gt;http://admb-project.org&lt;/a&gt;.  What you will&#xA;get is full information maximum likelihood solutions with the ability to try&#xA;MCMC methods afterwards if you wish. The idea is to regard this as a random&#xA;effects problem and integrate over the random effects via the Laplace approximation.  The trick is parameterize if properly so that the Hessian&#xA;with respect to the random effects is sparse.  I built the model and a simulator. It seems to work well.  To give you an idea I have included the&#xA;ADMB source for the model.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;DATA_SECTION&#xA;  init_int nobs&#xA;  init_vector Y(1,nobs)&#xA;  vector resids(2,nobs)&#xA;PARAMETER_SECTION&#xA;  init_number Phi1&#xA;  init_number Phi2&#xA;  init_bounded_number log_sigma(-5.0,5.0,2);&#xA;  init_bounded_number log_p1(-5.0,5.0,2);&#xA;  init_bounded_number log_p2(-5.0,5.0,2);&#xA;  objective_function_value f&#xA;  init_number A2&#xA;  init_number B2&#xA;  random_effects_vector A(3,nobs)&#xA;  random_effects_vector B(3,nobs)&#xA;  vector nu_1(3,nobs);&#xA;  vector nu_2(3,nobs);&#xA;  vector pred_Y(2,nobs);&#xA;  sdreport_number sigma&#xA;  sdreport_number p1&#xA;  sdreport_number p2&#xA;PROCEDURE_SECTION&#xA;&#xA;  f0(log_sigma,log_p1,log_p2,A2,B2);&#xA;&#xA;  f2(3,log_sigma,log_p1,log_p2,A(3),B(3),A2,B2,Phi1,Phi2);&#xA;&#xA;  for (int i=4;i&amp;lt;=nobs;i++)&#xA;  {&#xA;    f2(i,log_sigma,log_p1,log_p2,A(i),B(i),A(i-1),B(i-1),Phi1,Phi2);&#xA;  }&#xA;&#xA;  if (sd_phase())&#xA;  {&#xA;    sigma=exp(log_sigma);&#xA;    p1=exp(log_p1);&#xA;    p2=exp(log_p2);&#xA;  }&#xA;&#xA;&#xA;SEPARABLE_FUNCTION void f0( const prevariable&amp;amp; log_sigma, const prevariable&amp;amp; log_p1, const prevariable&amp;amp; log_p2, const prevariable&amp;amp; A2, const prevariable&amp;amp; B2)&#xA;&#xA;  dvariable sigma=exp(log_sigma);&#xA;  dvariable p1=exp(log_p1);&#xA;  dvariable p2=exp(log_p2);&#xA;  f+=square(A2)+square(B2);&#xA;  resids(2)=value(r);&#xA;  f+=log_sigma+0.5*square(r/sigma);&#xA;&#xA;&#xA;SEPARABLE_FUNCTION void f2(int i, const prevariable&amp;amp; log_sigma, const prevariable&amp;amp; log_p1, const prevariable&amp;amp; log_p2, const prevariable&amp;amp; Ai, const prevariable&amp;amp; Bi, const prevariable&amp;amp; Ai1, const prevariable&amp;amp; Bi1, const prevariable&amp;amp; Phi1,const prevariable&amp;amp; Phi2)&#xA;&#xA;  dvariable sigma=exp(log_sigma);&#xA;  dvariable p1=exp(log_p1);&#xA;  dvariable p2=exp(log_p2);&#xA;&#xA;  dvariable r=Y(i)-Ai-Bi*Y(i-1);&#xA;  resids(i)=value(r);&#xA;  f+=log_sigma+0.5*square(r/sigma);&#xA;  dvariable nu_1i=(Ai-Phi1*Ai1);&#xA;  dvariable nu_2i=(Bi-Phi2*Bi1);&#xA;  f+=log_p1+0.5*square(nu_1i/p1);&#xA;  f+=log_p2+0.5*square(nu_2i/p2);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;There is a &quot;gotcha&quot; with this approach.&#xA;Note that one can set A(i) such that&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Y(i)-A(i)-B(i)*Y(i-1)=0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then if you let sigma-&gt;0 the log-likelihood -&gt; infinity.&#xA;So the &quot;real&quot; answer is a local maximum. You can stabilize the&#xA;estimation by putting a lower bound on sigma or keeping sigma fixed&#xA;at a reasonable value for a while. &lt;/p&gt;&#xA;" OwnerUserId="1585" LastEditorUserId="1585" LastEditorDisplayName="" LastEditDate="2010-11-12T18:10:48.460" LastActivityDate="2010-11-12T18:10:48.460" CommentCount="4" />
  <row Id="4457" PostTypeId="2" ParentId="4451" CreationDate="2010-11-11T20:53:17.367" Score="2" ViewCount="" Body="&lt;p&gt;A large index of facebook pages was created and is available as a torrent (It is ~2.8Gb) &lt;a href=&quot;http://btjunkie.org/torrent/Facebook-directory-personal-details-for-100-million-users/3979e54c73099d291605e7579b90838c2cd86a8e9575&quot; rel=&quot;nofollow&quot;&gt;http://btjunkie.org/torrent/Facebook-directory-personal-details-for-100-million-users/3979e54c73099d291605e7579b90838c2cd86a8e9575&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Twitter datasets are tagged on Infochimps: &lt;a href=&quot;http://infochimps.com/tags/twitter&quot; rel=&quot;nofollow&quot;&gt;http://infochimps.com/tags/twitter&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A lastfm dataset is available at &lt;a href=&quot;http://mtg.upf.edu/node/1671&quot; rel=&quot;nofollow&quot;&gt;http://mtg.upf.edu/node/1671&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1874" LastEditorUserId="1874" LastEditorDisplayName="" LastEditDate="2010-11-13T16:30:53.093" LastActivityDate="2010-11-13T16:30:53.093" CommunityOwnedDate="2010-11-14T14:49:35.420" />
  <row Id="4458" PostTypeId="2" ParentId="4454" CreationDate="2010-11-11T20:54:25.463" Score="1" ViewCount="" Body="&lt;p&gt;docs.google.com&lt;/p&gt;&#xA;" OwnerUserId="666" LastActivityDate="2010-11-11T20:54:25.463" CommentCount="0" />
  <row Id="4459" PostTypeId="2" ParentId="1980" CreationDate="2010-11-11T21:30:55.937" Score="4" ViewCount="" Body="&lt;p&gt;I have a few such examples &lt;a href=&quot;http://jakebowers.org/papers.html&quot; rel=&quot;nofollow&quot;&gt;on my research papers page&lt;/a&gt;. (I am not allowed to post more than one hyperlink as a new member. So I'll just describe the papers on that site.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(1) &quot;Making Effects Manifest in Randomized Experiments&quot;  uses R's vignette system. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(2) &quot;Attributing Effects to a Cluster Randomized Get-Out-The-Vote Campaign&quot; was a more complex paper involving some time consuming simulations. We used a Makefile based system and posted it to the Dataverse &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(3) &quot;EDA for HLM&quot; was my earliest attempt. Here I just put the data and associated Sweave files in a tarball.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One problem we discovered when creating our JASA archive was that versions and defaults of CRAN packages changed. So, in that archive, we also include the versions of the packages that we used. The vignette based system will probably break as folks change their packages (not sure how to include extra packages within the package that is the Compendium). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, I wonder about what to do when R itself changes. Are there ways to produce, say, a virtual machine that reproduces the entire computational environment used for a paper such that the virtual machine is not enormous?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyway, I hope that these examples help. At least they show some of my own experiments in this area.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Here are some plain text hyperlinks.)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;  [2]: http://jakebowers.org/manifesteffects-compendium-howto.txt&#xA;  [3]: http://hdl.handle.net/1902.1/12174&#xA;  [4]: http://hdl.handle.net/1902.1/13376&#xA;&lt;/pre&gt;&#xA;" OwnerUserId="909" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-11T21:37:18.953" LastActivityDate="2010-11-11T21:37:18.953" CommentCount="1" />
  <row Id="4460" PostTypeId="1" CreationDate="2010-11-11T21:52:47.977" Score="0" ViewCount="48" Body="&lt;p&gt;Been setting up R on Windows Vista (64bit but using all 32 bit binaries) R seems to be fine but I don't seem to be able to get Rpy going.. now Rpy tells me that it cannot find the R version,, I've posted to R they suggest the problem is Rpy, but no answer there, so wondering if anyone else has this working?&lt;/p&gt;&#xA;" OwnerUserId="1963" LastActivityDate="2010-11-11T21:52:47.977" ClosedDate="2010-11-11T23:11:30.033" Title="Set up Rpy on Windows Vista" Tags="&lt;r&gt;&lt;software&gt;" CommentCount="1" />
  <row Id="4461" PostTypeId="2" ParentId="4454" CreationDate="2010-11-11T22:02:31.417" Score="1" ViewCount="" Body="&lt;p&gt;For mocking up what you want the survey to look like for a programmer, I would definitely just code it up in HTML.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;HTML is easy in general; for things like this, it will be dead simple.  I don't think you'd need to fuss with any CSS to make something useful for your programmer.  Conversely...&lt;/li&gt;&#xA;&lt;li&gt;You or your programmer can use CSS to drastically change the appearance of the survey without needing to edit the individual elements.&lt;/li&gt;&#xA;&lt;li&gt;Version control becomes very easy - just use something like &lt;a href=&quot;http://git-scm.com/&quot; rel=&quot;nofollow&quot;&gt;git&lt;/a&gt;, &lt;a href=&quot;http://subversion.tigris.org/&quot; rel=&quot;nofollow&quot;&gt;SVN&lt;/a&gt;, or &lt;a href=&quot;http://www.fossil-scm.org/&quot; rel=&quot;nofollow&quot;&gt;Fossil&lt;/a&gt; that will let you easily distribute the most recent version of the survey and will let you track exactly which changes were made, when, and by whom.&lt;/li&gt;&#xA;&lt;li&gt;It's free!&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="71" LastActivityDate="2010-11-11T22:02:31.417" CommentCount="1" />
  <row Id="4462" PostTypeId="1" CreationDate="2010-11-11T22:21:09.420" Score="2" ViewCount="383" Body="&lt;p&gt;I do a bunch of Real Estate Reporting and the Median price is often reported. Particularly by the NAR (National Association Of Realtors) ,,, How can this be,, As best I can figure they only get the Medians from each area,, so how should they calculate a national median, a median of medians a a simple average of medians or weighted average of medians or....??? How valid can it be,, I know that the NAR is not getting the total transaction table so can a reasonable representation of Median (nationally be done? Particularly since regional density and prices and market variances are so large now?&lt;/p&gt;&#xA;" OwnerUserId="1963" LastActivityDate="2010-11-15T17:50:20.730" Title="Median of Medians calculation" Tags="&lt;median&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="1" />
  <row Id="4463" PostTypeId="2" ParentId="4451" CreationDate="2010-11-11T22:33:57.397" Score="3" ViewCount="" Body="&lt;p&gt;Check out the Stanford large network dataset collection: &lt;a href=&quot;http://snap.stanford.edu/data/&quot; rel=&quot;nofollow&quot;&gt;SNAP&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1913" LastActivityDate="2010-11-11T22:33:57.397" CommunityOwnedDate="2010-11-14T14:49:35.420" />
  <row Id="4464" PostTypeId="2" ParentId="4451" CreationDate="2010-11-11T23:06:48.887" Score="3" ViewCount="" Body="&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://blog.infochimps.com/2008/12/29/massive-scrape-of-twitters-friend-graph/&quot; rel=&quot;nofollow&quot;&gt;A huge twitter dataset that includes followers, not just tweets&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://infochimps.com/collections/twitter-census&quot; rel=&quot;nofollow&quot;&gt;large collection of twitter datasets here&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="1808" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-11-14T11:22:27.570" LastActivityDate="2010-11-14T11:22:27.570" CommunityOwnedDate="2010-11-14T14:49:35.420" />
  <row Id="4465" PostTypeId="1" AcceptedAnswerId="4479" CreationDate="2010-11-11T23:10:10.583" Score="7" ViewCount="187" Body="&lt;p&gt;I am a student, working with a team on a large-scale ecological experiment. We want to analyze survival data which has been derived from an experimental design with some pseudo-replication. This pseudo-replication was not discovered, unfortunately, until the middle of the experiment, at which point the design could not be altered.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The experimental design involves comparing the survival data of several treatment groups, each comprised of 10 replicates (aquaria) with 10 individuals in each tank. We have measured mortality as a response variable, in response to different environmental stressors. The trouble is that we cannot say that each of the deaths that occurred, have occurred independently since each tank has many individuals. We would like to acknowledge this problem and address it in our analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All of the survival analysis tools we are aware of, assume independence between replicates. We are considering using Kaplan-Meier curves, Cox proportional hazard, or even a glm with Gamma error distributions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas about how we can properly address this problem in order to detect a difference in survivorship between the treatment groups?&lt;/p&gt;&#xA;" OwnerUserId="1862" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-11-11T23:48:57.587" LastActivityDate="2010-11-12T13:22:26.720" Title="Any ideas about how to analyze survival data with pseudo-replication (dependent data)?" Tags="&lt;modeling&gt;&lt;survival&gt;&lt;experiment-design&gt;&lt;independence&gt;&lt;frailty&gt;" AnswerCount="1" CommentCount="5" FavoriteCount="2" />
  <row Id="4466" PostTypeId="1" CreationDate="2010-11-12T01:05:10.823" Score="17" ViewCount="303" Body="&lt;p&gt;&lt;strong&gt;Context:&lt;/strong&gt;&#xA;In response to an earlier question about reproducible research &lt;a href=&quot;http://stats.stackexchange.com/questions/1980/complete-substantive-examples-of-reproducible-research-using-r/4459#4459&quot;&gt;Jake wrote&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;One problem we discovered when&#xA;  creating our JASA archive was that&#xA;  versions and defaults of CRAN packages&#xA;  changed. So, in that archive, we also&#xA;  include the versions of the packages&#xA;  that we used. The vignette based&#xA;  system will probably break as folks&#xA;  change their packages (not sure how to&#xA;  include extra packages within the&#xA;  package that is the Compendium).&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Finally, I wonder about what to do&#xA;  when R itself changes. Are there ways&#xA;  to produce, say, a virtual machine&#xA;  that reproduces the entire&#xA;  computational environment used for a&#xA;  paper such that the virtual machine is&#xA;  not enormous?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What are good strategies for ensuring that reproducible data analysis is reproducible in the future (say, five, ten, or twenty years after publication)?&lt;/li&gt;&#xA;&lt;li&gt;Specifically, what are good strategies for maximising ongoing reproducibility when using Sweave and R?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;This seems to be related to the issue of ensuring that a reproducible data analysis project will run on someone else's machine with slightly different defaults, packages, etc.&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2011-01-27T14:57:16.553" Title="How to increase longer term reproducibility of research (particularly using R and Sweave)" Tags="&lt;r&gt;&lt;reproducible-research&gt;&lt;sweave&gt;" AnswerCount="7" FavoriteCount="5" />
  <row Id="4467" PostTypeId="2" ParentId="4466" CreationDate="2010-11-12T01:14:44.813" Score="9" ViewCount="" Body="&lt;p&gt;One strategy involves using the &lt;code&gt;cacher&lt;/code&gt; package.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Peng RD, Eckel SP (2009). &quot;Distributed reproducible research using cached computations,&quot; IEEE Computing in Science and Engineering, 11 (1), 28–34. (&lt;a href=&quot;http://www.scribd.com/full/28735488?access_key=key-1nhq0segw2ptjaxzhf6t&quot; rel=&quot;nofollow&quot;&gt;PDF online&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;li&gt;also see more articles on&#xA;&lt;a href=&quot;http://www.biostat.jhsph.edu/~rpeng/cv.html&quot; rel=&quot;nofollow&quot;&gt;Roger Peng's website&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Further discussion and examples can be found in the book:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.springer.com/statistics/life+sciences,+medicine+%26+health/book/978-0-387-78166-2&quot; rel=&quot;nofollow&quot;&gt;Statistical Methods for Environmental Epidemiology with R&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;However, I don't have first hand experience of its effectiveness in ensuring ongoing reproducibility.&lt;/p&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-11-12T03:31:43.277" LastActivityDate="2010-11-12T03:31:43.277" CommentCount="1" />
  <row Id="4468" PostTypeId="2" ParentId="4466" CreationDate="2010-11-12T02:07:06.107" Score="17" ViewCount="" Body="&lt;p&gt;At some level, this becomes impossible.  Consider the case of the famous Pentium floating point bug: you not only need to conserve your models, your data, your parameters, your packages, all external packages, the host system or language (say, R) as well as the OS ... plus potentially the hardware it all ran on.  Now consider that some results may be simulation based and required a particular cluster of machines...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That's just a bit much for being practical.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With that said, I think more pragmatic solutions of versioning your code (and maybe also your data) in revisions control, storing versions of all relevant software and making it possible to reproduce the results by running a single top-level script may be a &quot;good enough&quot; compromise.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your mileage may vary. This also differs across disciplines or industry.  But remember the old saw about the impossibility of foolproof systems: you merely create smarter fools.&lt;/p&gt;&#xA;" OwnerUserId="334" LastEditorUserId="334" LastEditorDisplayName="" LastEditDate="2010-11-12T03:11:01.320" LastActivityDate="2010-11-12T03:11:01.320" CommentCount="1" />
  <row Id="4469" PostTypeId="1" AcceptedAnswerId="4472" CreationDate="2010-11-12T03:06:08.610" Score="1" ViewCount="226" Body="&lt;p&gt;Is there a way to type in a regression formula in SPSS in the same manner as R, with weights?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, in R, I would write something like: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;lm(y ~ B1 + B2 + B1*B2, data=df, weights=x) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How would I go about recreating this in SPSS/PASW? &lt;/p&gt;&#xA;" OwnerUserId="776" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-11-12T05:42:30.897" LastActivityDate="2010-11-13T03:18:29.467" Title="Using R's formula style lm() with SPSS" Tags="&lt;r&gt;&lt;spss&gt;" AnswerCount="3" />
  <row Id="4470" PostTypeId="2" ParentId="4469" CreationDate="2010-11-12T03:48:13.660" Score="6" ViewCount="" Body="&lt;p&gt;In SPSS you can have the GUI write the syntax for you via the paste button. If you go through the regression command in the window there is an option to include weights.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is some sample code it just had the program output for me:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  REGRESSION&#xA;  /MISSING LISTWISE&#xA;  /REGWGT=VAR3&#xA;  /STATISTICS COEFF OUTS R ANOVA&#xA;  /CRITERIA=PIN(.05) POUT(.10)&#xA;  /NOORIGIN &#xA;  /DEPENDENT VAR1&#xA;  /METHOD=ENTER VAR2.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As with any program, I would suggest you check the documentation on how SPSS implements weights in OLS (I personally have no idea).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A comment by Wolfgang below points out that R and SPSS implement weights in the same manner (although I would still suggest checking out the documentation of how they implement weights.)&lt;/p&gt;&#xA;" OwnerUserId="1036" LastEditorUserId="1036" LastEditorDisplayName="" LastEditDate="2010-11-12T14:25:58.977" LastActivityDate="2010-11-12T14:25:58.977" CommentCount="4" />
  <row Id="4471" PostTypeId="1" CreationDate="2010-11-12T04:41:13.047" Score="2" ViewCount="366" Body="&lt;p&gt;For chi squared distribution, how would I find quantiles for the following three cases:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A) $P(X^2&amp;gt; X^2_{\alpha})=0.01$ when $v = 21$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;B) $P(X^2 &amp;lt; X^2_{\alpha})=0.95$ when $v =6$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;C) $P(X^2_{\alpha} &amp;lt; X^2 &amp;lt;23.209) = 0.015$ when $v = 10$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here $X^2_{\alpha}$ is the $\alpha$-quantile of the $\chi^2_{v}$ distribution.  &lt;/p&gt;&#xA;" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-11-12T12:12:46.030" LastActivityDate="2010-11-12T13:12:37.960" Title="Calculating quantiles for chi squared distribution" Tags="&lt;chi-squared&gt;&lt;homework&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="4472" PostTypeId="2" ParentId="4469" CreationDate="2010-11-12T04:42:32.533" Score="6" ViewCount="" Body="&lt;p&gt;In recent versions of SPSS you can run R code directly in SPSS on your SPSS datasets.&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-11-12T04:42:32.533" CommentCount="2" />
  <row Id="4473" PostTypeId="1" AcceptedAnswerId="4474" CreationDate="2010-11-12T06:50:57.487" Score="2" ViewCount="239" Body="&lt;p&gt;I've created a model (cue ominous music) in R based on previous months data using lm(). Now, I would like to see how well it predicts the current months data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, my model predicts sales figures. I have figures for January, February and March. My model is based on figures from January and February and I would like to estimate how good of a predictor the model is of March's sales. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a method for doing this without having to manually apply all the coeficients and weights? &lt;/p&gt;&#xA;" OwnerUserId="776" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-04-13T10:34:22.903" LastActivityDate="2011-04-13T15:01:29.147" Title="Validating a linear model with R, lm()" Tags="&lt;r&gt;&lt;regression&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="4474" PostTypeId="2" ParentId="4473" CreationDate="2010-11-12T07:08:19.540" Score="7" ViewCount="" Body="&lt;p&gt;?predict   (which will implement ?predict.lm)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Make sure you put into &quot;newdata&quot; a data.frame with the exact same variable names.&lt;/p&gt;&#xA;" OwnerUserId="253" LastActivityDate="2010-11-12T07:08:19.540" CommentCount="2" />
  <row Id="4476" PostTypeId="2" ParentId="4446" CreationDate="2010-11-12T12:37:14.680" Score="0" ViewCount="" Body="&lt;p&gt;Although the formula are equal, the practical difference is ease-of-calculation if you're doing it by hand. If all I had was a piece of paper and a pencil, I'd prefer the second formula--- $\sum X^2$ and $\sum X$ together take less time and are less error prone to calculate than $\sum (X - m)^2$.&lt;/p&gt;&#xA;" OwnerUserId="1916" LastActivityDate="2010-11-12T12:37:14.680" CommentCount="1" />
  <row Id="4477" PostTypeId="2" ParentId="4471" CreationDate="2010-11-12T12:47:56.360" Score="5" ViewCount="" Body="&lt;p&gt;By hand, you need to refer to a tabulated distribution of the $\chi^2$, which should be found easily on the web (e.g., &lt;a href=&quot;http://www.unc.edu/~farkouh/usefull/chi.html&quot; rel=&quot;nofollow&quot;&gt;this one&lt;/a&gt;). Let x denotes the quantile of interest, and v the degrees of freedom of the chi-square distribution.&#xA;You just have to know that the total area under the curve (i.e., the density) equals 1 (this will help you to work through the third case), and that such Tables generally gives P(X &amp;lt; x) = p, for a certain v. Knowing x (resp. p), you can find the approximated value of p (resp. x). In the aforementioned Table, the first cell reads: P(X &amp;lt; 1.32) = 0.25, for a 1-df chi-square.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have R, the &lt;code&gt;qchisq()&lt;/code&gt; function gives you the requested quantiles (look at the on-line help to be sure of what is returned, esp. the &lt;code&gt;lower.tail&lt;/code&gt; argument). For the preceding example, we would use &lt;code&gt;qchisq(0.25, 1, lower.tail=FALSE)&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is always a good idea to draw the corresponding density curve, as illustrated below. Note that p3 is also 1-P(X &amp;lt; q2).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/0wB5n.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-12T13:12:37.960" LastActivityDate="2010-11-12T13:12:37.960" />
  <row Id="4478" PostTypeId="2" ParentId="4466" CreationDate="2010-11-12T12:54:03.257" Score="11" ViewCount="" Body="&lt;p&gt;The first step in reproducibility is making sure the data are in a format that is easy for future researchers to read. Flat files are the clear choice here (Fairbairn in press).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To make the code useful over the long term, perhaps the best thing to do is write clear documentation that explains both what the code does and also how it works, so that if your tool chain disappears, your analysis can be reimplemented in some future system. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Fairbairn (in press) &lt;a href=&quot;http://onlinelibrary.wiley.com/doi/10.1111/j.1558-5646.2010.01182.x/abstract&quot; rel=&quot;nofollow&quot;&gt;The advent of mandatory data archiving&lt;/a&gt;. Evolution DOI: 10.1111/j.1558-5646.2010.01182.x&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="1916" LastActivityDate="2010-11-12T12:54:03.257" CommentCount="1" />
  <row Id="4479" PostTypeId="2" ParentId="4465" CreationDate="2010-11-12T13:22:26.720" Score="2" ViewCount="" Body="&lt;p&gt;Here's some thoughts on what I'd do using relatively simple methods (i.e. avoiding frailty models, which I admit I've never used and don't really understand, so someone else may like to provide an answer involving them). I'm assuming you don't have other forms of censoring apart from the end of the experiment and that there are no time-dependent exposures (i.e. the treatment is either constant or applied only at the start before any deaths have taken place)&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Do some descriptive statistics and Kaplan-Meier plots ignoring the issue of dependence and therefore without reporting or displaying standard errors, confidence intervals or p-values.&lt;/li&gt;&#xA;&lt;li&gt;Ignore the time component and just count the number of deaths out of the total starting number in each aquarium. Fit a generalized linear model with a binomial distribution to these counts, using either a logistic link function (to give odds ratios) or a log link function (giving easier-to-interpret risk ratios at the price of potential problems with fitting the model). I think this is along the same lines as the analysis you said you're considering in the first comment to the question. As your mortalities are reasonably low, the loss in power over a full  survival analysis with frailty modelling will probably be modest. This overcomes the dependence issue as you are using each aquarium as the unit of analysis instead of each individual. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-11-12T13:22:26.720" CommentCount="1" />
  <row Id="4480" PostTypeId="1" AcceptedAnswerId="4481" CreationDate="2010-11-12T16:26:13.343" Score="1" ViewCount="129" Body="&lt;p&gt;Or more specifically, why would the index of a dataframe not be in ascending numerical order?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I display the dataframe, the first row has column names, the data begins on second row, but the leftmost column (which doesn't have a header), is not in order, instead it shows as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2&lt;br&gt;&#xA;1&lt;br&gt;&#xA;4&lt;br&gt;&#xA;6&lt;br&gt;&#xA;5&lt;br&gt;&#xA;3&lt;br&gt;&lt;/p&gt;&#xA;" OwnerUserId="1965" LastActivityDate="2010-11-12T16:35:15.587" Title="What order are the rows in a dataframe in R displayed in?" Tags="&lt;r&gt;&lt;dataframe&gt;" AnswerCount="2" />
  <row Id="4481" PostTypeId="2" ParentId="4480" CreationDate="2010-11-12T16:32:03.157" Score="5" ViewCount="" Body="&lt;p&gt;Data frames have &lt;code&gt;rownames&lt;/code&gt; attribute which is shown as the captionless leftmost column; you've propably altered the order of rows in data frame and it just shows the original indices. Try  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; rownames(df)&amp;lt;-NULL  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;to recreate it to 1:N.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-11-12T16:32:03.157" />
  <row Id="4482" PostTypeId="2" ParentId="4480" CreationDate="2010-11-12T16:35:15.587" Score="5" ViewCount="" Body="&lt;p&gt;If you reorder a data frame the row.names with be reordered to match. Note that if you haven't supplied rownames, R makes them up for you as characters &lt;code&gt;as.character(1:nrow(obj))&lt;/code&gt;, where &lt;code&gt;obj&lt;/code&gt; is your data frame. Hence these are row &lt;em&gt;names&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; df &amp;lt;- data.frame(A = 5:1, B = LETTERS[1:5])&#xA;&amp;gt; df&#xA;  A B&#xA;1 5 A&#xA;2 4 B&#xA;3 3 C&#xA;4 2 D&#xA;5 1 E&#xA;&amp;gt; df[order(df$A),]&#xA;  A B&#xA;5 1 E&#xA;4 2 D&#xA;3 3 C&#xA;2 4 B&#xA;1 5 A&#xA;&amp;gt; df[sample(5),]&#xA;  A B&#xA;3 3 C&#xA;2 4 B&#xA;4 2 D&#xA;1 5 A&#xA;5 1 E&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But I could just as well have applied some rownames:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; rownames(df) &amp;lt;- letters[1:5]&#xA;&amp;gt; df&#xA;  A B&#xA;a 5 A&#xA;b 4 B&#xA;c 3 C&#xA;d 2 D&#xA;e 1 E&#xA;&amp;gt; df[sample(5),]&#xA;  A B&#xA;d 2 D&#xA;c 3 C&#xA;a 5 A&#xA;b 4 B&#xA;e 1 E&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You would need to show us how your data were read in, example data, and what code you are using if the above doesn't answer your Q.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the rownames are not important, ignore them or do:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;rownames(df) &amp;lt;- NULL&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1390" LastActivityDate="2010-11-12T16:35:15.587" />
  <row Id="4483" PostTypeId="2" ParentId="4187" CreationDate="2010-11-12T16:50:03.513" Score="4" ViewCount="" Body="&lt;p&gt;I would use the &quot;logit.mixed&quot; function in &lt;a href=&quot;http://cran.at.r-project.org/web/packages/Zelig/index.html&quot; rel=&quot;nofollow&quot;&gt;Zelig&lt;/a&gt;, which is a wrapper for lime4 and makes it very convenient to do prediction and simulation. &lt;/p&gt;&#xA;" OwnerUserId="1966" LastActivityDate="2010-11-12T16:50:03.513" />
  <row Id="4484" PostTypeId="2" ParentId="4466" CreationDate="2010-11-12T17:35:23.330" Score="6" ViewCount="" Body="&lt;p&gt;If you are interested in the virtual machine route, I think it would be doable via a small linux distribution with the specific version of R and packages installed.  Data is included, along with scripts, and package the whole thing in a &lt;a href=&quot;http://www.virtualbox.org/&quot; rel=&quot;nofollow&quot;&gt;virtual box&lt;/a&gt; file.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This does not get around hardware problems mentioned earlier such as the Intel CPU bug.&lt;/p&gt;&#xA;" OwnerUserId="1965" LastActivityDate="2010-11-12T17:35:23.330" />
  <row Id="4485" PostTypeId="1" AcceptedAnswerId="4488" CreationDate="2010-11-12T18:35:44.143" Score="3" ViewCount="201" Body="&lt;p&gt;As described in a few of my previous questions (&lt;a href=&quot;http://stats.stackexchange.com/q/3207/1381&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;http://stats.stackexchange.com/q/2917/1381&quot;&gt;there&lt;/a&gt;), I am interested in deriving summary statistics from statistics reported in the literature.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would very much appreciate any advice into the validity or errors found in the following calculations. &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;To solve for $MSE$ given $F$, $df_{\text{group}}$, and $SS$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is required when a partial anova table is provided.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given:&#xA;\begin{equation}\label{eq:f}&#xA;F = MS_g/MS_e&#xA;\end{equation}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $g$ indicates the group, or treatment. &#xA;Rearranging this equation gives:&#xA;$$MS_e=MS_g/F$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$MS_x = SS_x/df_x$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Substitute $SS_g/df_g$ for $MS_g$ in the first equation&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$F=\frac{SS_g/df_g}{MS_e}$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then solve for $MS_e$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{equation}\label{eq:mse}&#xA;MS_e = \frac{SS_g}{df_g\times F}&#xA;\end{equation}&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Example&lt;/strong&gt; from table 3 in &lt;a href=&quot;http://instaar.metapress.com/index/L5L27V837P506226.pdf&quot; rel=&quot;nofollow&quot;&gt;Starr 2008&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The results are from one (two?) factor ANOVA with repeated measures, with treatment and week as the factors and no replication. &quot;Effects of treatment on individual species were analyzed using Repeated Measures ANOVA in the statistical package Superanova (Abacus Concepts, Berkeley).&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/1Khhq.png&quot; alt=&quot;Starr 2008 Table 3&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We will calculate MSE from the $SS_{\text{treatment}}$ df_{\text{treatment}}, and $F$-value given in the table; these are $109.58$, $2$, and $0.570$, respectively; $df_{\text{weeks}}$ is given as $10$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the 1997 \textit{Eriphorium vaginatum}, the mean $A_{max}$ in table 4 is $13.49$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Calculate $MS_e$: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$MS_e = \frac{109.58}{0.57 \times 2} = 96.12$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;If this is the correct way, then shouldn't $MS_e$ be the same calculated based on factor a or factor b?  &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;This is only a first draft, but my first attempt at presenting this detailed of a mathematical derivation. Feedback on the writing and presentation would be much appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="1381" LastEditorUserId="1381" LastEditorDisplayName="" LastEditDate="2010-11-14T04:50:42.380" LastActivityDate="2010-11-14T04:50:42.380" Title="Is this the correct way to calculate $MSE$ from $SS_{\text{treatment(s)}}$, $df_{\text{treatment(s)}}$, $F$?" Tags="&lt;anova&gt;&lt;meta-analysis&gt;&lt;error&gt;&lt;summary-statistics&gt;&lt;degrees-of-freedom&gt;" AnswerCount="1" />
  <row Id="4486" PostTypeId="2" ParentId="4431" CreationDate="2010-11-12T19:28:49.717" Score="2" ViewCount="" Body="&lt;p&gt;I have relied on &lt;a href=&quot;http://books.google.com/books?id=e4f8sqm439UC&amp;amp;printsec=frontcover&amp;amp;dq=resampling+methods+for+dependent+data&amp;amp;source=bl&amp;amp;ots=hwangs1qCA&amp;amp;sig=WzmEiF0W2sRXlSm5MTvK6ESkt7k&amp;amp;hl=en&amp;amp;ei=_pPdTJnWIcGYnwfaqpCzDw&amp;amp;sa=X&amp;amp;oi=book_result&amp;amp;ct=result&amp;amp;resnum=1&amp;amp;ved=0CBIQ6AEwAA#v=onepage&amp;amp;q&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;Resampling Methods for Dependent Data&lt;/a&gt; by S.N. Lahiri and found it quite helpful. Once you determine some flavors you want to look at more closely (e.g. Circular Block Bootstrap, Stationary Block Bootstrap) it will be easier to find on-line resources discussing actual use cases and implementation details.&lt;/p&gt;&#xA;" OwnerUserId="1080" LastActivityDate="2010-11-12T19:28:49.717" />
  <row Id="4487" PostTypeId="2" ParentId="4187" CreationDate="2010-11-12T20:09:03.653" Score="1" ViewCount="" Body="&lt;p&gt;Stephen Raudenbush has a book chapter in the Handbook of Multilevel Analysis on &quot;&lt;a href=&quot;http://www.springerlink.com/content/978-0-387-73183-4#section=142726&amp;amp;page=1&amp;amp;locus=0&quot; rel=&quot;nofollow&quot;&gt;Many Small Groups&lt;/a&gt;&quot;. If you are only interested in the effects of x on y and have no interest in higher level effects, his suggestion is simply to estimate a fixed effects model (i.e. a dummy variable for all possible higher level groupings).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't know how applicable that is towards prediction, but I would imagine some of what he writes is applicable to what you are trying to accomplish.&lt;/p&gt;&#xA;" OwnerUserId="1036" LastActivityDate="2010-11-12T20:09:03.653" />
  <row Id="4488" PostTypeId="2" ParentId="4485" CreationDate="2010-11-12T21:06:36.833" Score="3" ViewCount="" Body="&lt;p&gt;Your derivation is perfectly fine for regular (non-repeated meaures) ANOVA. For repeated measures ANOVA the F-statistics does not always equal $MS_g/MS_e$. Assuming weeks is the repeated measure, this formula is correct for the weeks and weeks*treatmeant terms (note that they will give the same MSE), but not for the treatment term. There is another term that is not shown in this table: subjects within treatment, and the MSE you found actually corresponds to that term.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are interested in complicated ANOVA tables, I would recommend a rather old book, that has a lot of these formulas worked out:&lt;br&gt;&#xA;O. J. Dunn and V.A. Clark: Applied Statistics: Analysis of Variance and Regression, Wiley&lt;/p&gt;&#xA;" OwnerUserId="279" LastActivityDate="2010-11-12T21:06:36.833" CommentCount="1" />
  <row Id="4489" PostTypeId="1" AcceptedAnswerId="4490" CreationDate="2010-11-13T01:46:15.850" Score="5" ViewCount="317" Body="&lt;p&gt;Consider the following graph:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x &amp;lt;- 1:100&#xA;y1 &amp;lt;- rnorm(100)&#xA;y2 &amp;lt;- rnorm(100)+100&#xA;&#xA;par(mar=c(5,5,5,5))&#xA;&#xA;plot(x,y1,pch=0,type=&quot;b&quot;,col=&quot;red&quot;,yaxt=&quot;n&quot;,ylim=c(-8,2),ylab=&quot;&quot;)&#xA;axis(side=2, at=c(-2,0,2))&#xA;mtext(&quot;red line&quot;, side = 2, line=2.5, at=0)&#xA;&#xA;par(new=T)&#xA;plot(x,y2,pch=1,type=&quot;b&quot;,col=&quot;blue&quot;,yaxt=&quot;n&quot;,ylim=c(98,108), ylab=&quot;&quot;)&#xA;axis(side=4, at=c(98,100,102), labels=c(&quot;98%&quot;,&quot;100%&quot;,&quot;102%&quot;))&#xA;mtext(&quot;blue line&quot;, side=4, line=2.5, at=100)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can I remove the automatically generated borders and preserve just the axis lines, to achieve a Tufte's style? &lt;/p&gt;&#xA;" OwnerUserId="990" LastActivityDate="2010-11-13T03:19:10.860" Title="Removing borders in R plots for achieving Tufte's axis" Tags="&lt;r&gt;&lt;data-visualization&gt;" AnswerCount="2" />
  <row Id="4490" PostTypeId="2" ParentId="4489" CreationDate="2010-11-13T02:04:40.637" Score="6" ViewCount="" Body="&lt;p&gt;Add &lt;code&gt;bty=&quot;n&quot;&lt;/code&gt; in both plot commands.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For time series, add &lt;code&gt;frame.plot=FALSE&lt;/code&gt; for the same effect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For fancier Tufte axes, see &lt;a href=&quot;http://addictedtor.free.fr/graphiques/RGraphGallery.php?graph=81&quot; rel=&quot;nofollow&quot;&gt;http://addictedtor.free.fr/graphiques/RGraphGallery.php?graph=81&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="159" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-11-13T03:17:31.450" LastActivityDate="2010-11-13T03:17:31.450" CommentCount="2" />
  <row Id="4491" PostTypeId="1" AcceptedAnswerId="4494" CreationDate="2010-11-13T03:08:04.377" Score="1" ViewCount="107" Body="&lt;p&gt;I'm using the following code to insert vertical lines in a timeseries plot:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;abline(v=as.Date(&quot;2007-09-19&quot;),col=&quot;red&quot;,lty=2)&#xA;abline(v=as.Date(&quot;2008-03-10&quot;),col=&quot;red&quot;,lty=2)&#xA;abline(v=as.Date(&quot;2008-04-28&quot;),col=&quot;red&quot;,lty=2)&#xA;abline(v=as.Date(&quot;2009-08-07&quot;),col=&quot;red&quot;,lty=2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But I would rather prefer a small triangle pointing up in the bottom axis, along with a label. Hints?&lt;/p&gt;&#xA;" OwnerUserId="990" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-13T09:21:23.217" LastActivityDate="2010-11-13T13:32:38.697" Title="Inserting small arrows (triangles) in bottom axis pointing up with R" Tags="&lt;r&gt;&lt;data-visualization&gt;" AnswerCount="3" />
  <row Id="4492" PostTypeId="2" ParentId="4469" CreationDate="2010-11-13T03:18:29.467" Score="3" ViewCount="" Body="&lt;p&gt;1) You can always find out exactly how an algorithm in SPSS is defined mathematically from the Algorithms link on the Help menu.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) SPSS also has a Weighted Least Square procedure that allows you to model the error variance and correct for heteroscedasticity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) In order to run R within SPSS (version 16 or later), you just need to install the R plugin or Essentials (depending on version) from SPSS Developer Central, www.spss.com/devcentral.  The integration is free.  You can also get SPSS syntax and dialog boxes for a number of useful R packages from the same site.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;HTH&lt;/p&gt;&#xA;" OwnerUserId="1971" LastActivityDate="2010-11-13T03:18:29.467" />
  <row Id="4493" PostTypeId="2" ParentId="4489" CreationDate="2010-11-13T03:19:10.860" Score="3" ViewCount="" Body="&lt;p&gt;If you use&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;par(bty = 'n') &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Before calling plot that will fix it for zoo.  It might also fix it for a variety of situations where it isn't passable to the plotting command.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Check out bty option in the par() help for other kinds of frames for the plot)&lt;/p&gt;&#xA;" OwnerUserId="601" LastActivityDate="2010-11-13T03:19:10.860" />
  <row Id="4494" PostTypeId="2" ParentId="4491" CreationDate="2010-11-13T03:28:41.470" Score="3" ViewCount="" Body="&lt;p&gt;There's a variety of ways.  You could use a pch = '^' and put that near the bottom or look up an arrow symbol character.  Also you could just do what you're doing with the arrows() command and draw short arrows instead of abline().  With arrows you would need to specify both ends instead of just the horizontal position.. e.g.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;arrows(as.Date(&quot;2009-08-07&quot;),0, as.Date(&quot;2009-08-07&quot;), 0.5, col=&quot;red&quot;,lty=2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="601" LastEditorUserId="601" LastEditorDisplayName="" LastEditDate="2010-11-13T13:32:38.697" LastActivityDate="2010-11-13T13:32:38.697" CommentCount="2" />
  <row Id="4496" PostTypeId="1" AcceptedAnswerId="4509" CreationDate="2010-11-13T04:08:55.587" Score="2" ViewCount="180" Body="&lt;p&gt;Consider the following code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(zoo)&#xA;data &amp;lt;- read.csv(file=&quot;summary.csv&quot;,sep=&quot;,&quot;,head=TRUE)&#xA;data = zoo(data$compressed, as.Date(data$date))&#xA;data &amp;lt;- aggregate(data, identity, tail, 1)&#xA;days = seq(start(data), end(data), &quot;day&quot;)&#xA;data2 = na.locf(merge(data, zoo(,days)))&#xA;&#xA;par(bty = 'n')&#xA;plot(data2,xlab='',ylab='entropy (bytes)')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can one:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Adjust both the horizontal and vertical limits o the drawn axis to match the start and end of data (as an example, while the y value may vary between 20 and 1525, the axis shows 0 and 1500).&lt;/li&gt;&#xA;&lt;li&gt;Increase the horizontal resolution (at least add some minor ticks) when the timeseries spans for a couple of years. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Using:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plot(data2,xlab='',ylab='entropy (bytes)', xaxs = 'i', yaxs = 'i')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/dLc7l.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which is not exactly what I had in mind. The vertical scale doesn't start with the exat minimum (around 25), nor does it end with the exact maximum. The horizontal scale starts and ends in the middle. Hints?&lt;/p&gt;&#xA;" OwnerUserId="990" LastEditorUserId="990" LastEditorDisplayName="" LastEditDate="2010-11-13T21:10:22.787" LastActivityDate="2010-11-13T21:10:22.787" Title="Adjusting axis properties in a timeseries graph" Tags="&lt;r&gt;&lt;data-visualization&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="4497" PostTypeId="2" ParentId="3542" CreationDate="2010-11-13T05:24:16.083" Score="4" ViewCount="" Body="&lt;p&gt;If you are interested in table design, I would definitely recommend two papers on the subject by Andrew Gelman:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A necessary preface to the paper on table design is Gelman et al, 2002 &lt;a href=&quot;http://pubs.amstat.org/doi/abs/10.1198/000313002317572790&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;Let's practice what we preach: Turning Tables Into Graphs&lt;/strong&gt;&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But &lt;a href=&quot;http://www.stat.columbia.edu/~gelman/research/published/tables4.pdf&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;Why Tables are Really Much Better than Graphs&lt;/strong&gt;&lt;/a&gt; (unpublished) gives more direct input on table design, including the following gems: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;good tables have lots of numbers&lt;/li&gt;&#xA;&lt;li&gt;don't obsess about clarity&lt;/li&gt;&#xA;&lt;li&gt;use exact numbers, minimum of four significant digits&lt;/li&gt;&#xA;&lt;li&gt;use default table design provided by your favorite software&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Andrew Gelman, Cristian Pasarica, Rahul Dodhia. The American Statistician. May 1, 2002, 56(2): 121-130. doi:10.1198/000313002317572790.&lt;/p&gt;&#xA;" OwnerUserId="1381" LastEditorUserId="1381" LastEditorDisplayName="" LastEditDate="2010-11-15T16:40:18.797" LastActivityDate="2010-11-15T16:40:18.797" CommentCount="2" />
  <row Id="4498" PostTypeId="1" CreationDate="2010-11-13T05:43:15.703" Score="6" ViewCount="199" Body="&lt;p&gt;They all seem to represent random variables by the nodes and (in)dependence via the (possibly directed) edges.  I'm esp interested in a bayesian's point-of-view.&lt;/p&gt;&#xA;" OwnerUserId="1795" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-13T20:33:21.760" LastActivityDate="2010-11-13T20:33:21.760" Title="What's the relation between hierarchical models, neural networks, graphical models, bayesian networks?" Tags="&lt;causal-inference&gt;&lt;neural-networks&gt;&lt;multilevel-analysis&gt;&lt;graphical-model&gt;" AnswerCount="2" />
  <row Id="4499" PostTypeId="2" ParentId="4491" CreationDate="2010-11-13T06:03:11.923" Score="0" ViewCount="" Body="&lt;p&gt;You could manually construct the triangles using the 'polygon' command.  There might be a nice way to automate the construction of these but depending on how many you want to make it probably wouldn't be too bad to figure out the coordinates by hand.&lt;/p&gt;&#xA;" OwnerUserId="1028" LastActivityDate="2010-11-13T06:03:11.923" />
  <row Id="4500" PostTypeId="1" AcceptedAnswerId="4506" CreationDate="2010-11-13T06:26:43.493" Score="5" ViewCount="215" Body="&lt;p&gt;&lt;strong&gt;Context:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've recently adopted version control as part of my data analysis work (finally I may hear you saying: see my &lt;a href=&quot;http://stackoverflow.com/questions/2712421/r-and-version-control-for-the-solo-data-analyst&quot;&gt;earlier question on SO&lt;/a&gt;). This prompted me to think more about repositories and the directory structure I use for my projects. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My typical research work involves one or more studies (i.e., data that I have collected) which gets written up as one or more publications (journal articles, book chapters, presentations, reports, etc.).  Analyses and reports are typically produced using a combination of R, LaTeX, Sweave, textual data files and so on. I really like the idea of being able to upload a single self-contained repository that can be used to analyse data and reproduce a publication.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In particular, I've been thinking about publications, studies, data, and common code, and how these entities map on to repositories. For example, is it better to have a separate repository for each publication or is it better to have each publication as an individual folder within the larger repository. I'm evolving a few thoughts on this, but I was keen to hear other options.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What strategies do people use to map studies, publications, and analyses onto repositories?&lt;/li&gt;&#xA;&lt;li&gt;When should related entities (e.g., publications, studies, etc.) be split into multiple repositories?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-11-14T23:06:00.820" CommunityOwnedDate="2010-11-13T14:00:03.970" Title="Repositories and data analysis projects" Tags="&lt;reproducible-research&gt;&lt;project-management&gt;" AnswerCount="2" FavoriteCount="2" />
  <row Id="4501" PostTypeId="1" AcceptedAnswerId="4503" CreationDate="2010-11-13T07:02:26.113" Score="3" ViewCount="197" Body="&lt;p&gt;I've been thinking recently about conventions for figure width in data analysis reports that lead to PDFs (e.g., in A4 or Letter size). My usual context is R, Sweave, and LaTeX.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The default figure width in Sweave is 80% of the text-width (i.e., the width of a paragraph of text).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;\setkeys{Gin}{width=0.80\textwidth}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Questions:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Is it reasonable for the width of figures to be wider than the text-width (i.e., to spill into the margins)?&lt;/li&gt;&#xA;&lt;li&gt;Is the 80% rule of text-width a good one or would, for example 100%, or some other value be better?&lt;/li&gt;&#xA;&lt;li&gt;Should the figure widths be consistent throughout a document (or perhaps with two sizes for small and large figures) or should the width be adapted completely to the content?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Any references or thoughts on best practice in this regards would be most welcome.&lt;/p&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-11-14T12:00:25.573" LastActivityDate="2010-11-14T12:00:25.573" Title="Typographic conventions for width of figures in LaTeX data analysis reports" Tags="&lt;sweave&gt;&lt;figures&gt;" AnswerCount="1" CommentCount="6" FavoriteCount="5" />
  <row Id="4502" PostTypeId="2" ParentId="4498" CreationDate="2010-11-13T08:37:26.060" Score="8" ViewCount="" Body="&lt;p&gt;A Bayesian network is a type of graphical model. The other &quot;big&quot; type of graphical model is a Markov Random Field (MRF).  Graphical models are used for inference, estimation and in general, to model the world.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The term hierarchical model is used to mean many things in different areas.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While neural networks come with &quot;graphs&quot; they generally don't encode dependence information, and the nodes don't represent random variables. NNs are different because they are discriminative. Popular neural networks are used for classification and regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Kevin Murphy has an excellent introduction to these topics &lt;a href=&quot;http://www.cs.ubc.ca/~murphyk/Bayes/bnintro.html&quot; rel=&quot;nofollow&quot;&gt;available here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1540" LastActivityDate="2010-11-13T08:37:26.060" CommentCount="2" />
  <row Id="4503" PostTypeId="2" ParentId="4501" CreationDate="2010-11-13T09:06:44.280" Score="6" ViewCount="" Body="&lt;p&gt;I'll second @onestop comment about the fact that this question seems marginally related to statistical analysis or reporting. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;That being said, I can't refrain from thinking of Ed. Tufte's work on the display of quantitative information, especially the design of his books which mixes different graphics layouts: some figures or tables are put in the margin, other in the body with caption in the margin, and large figures may extend beyond the body (full page width). The &lt;a href=&quot;http://code.google.com/p/tufte-latex/&quot; rel=&quot;nofollow&quot;&gt;tufte-latex&lt;/a&gt; project offers $\LaTeX$ classes for articles/handouts and books in the spirit of Ed. Tufte's design. Some examples are included on the project page; I particularly like the &lt;a href=&quot;http://tufte-latex.googlecode.com/files/sample-handout-3.5.0.pdf&quot; rel=&quot;nofollow&quot;&gt;example handout&lt;/a&gt;.&#xA;On a related point, I also like the &lt;a href=&quot;http://cc.oulu.fi/~jarioksa/opetus/metodi/vegantutor.pdf&quot; rel=&quot;nofollow&quot;&gt;tutorial&lt;/a&gt; from the &lt;a href=&quot;http://cran.r-project.org/web/packages/vegan/index.html&quot; rel=&quot;nofollow&quot;&gt;vegan&lt;/a&gt; R package.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My personal approach is to use 80% or 100% of text width (and keep it consistent across all the document), but I often play with the &lt;code&gt;width&lt;/code&gt;, &lt;code&gt;height&lt;/code&gt;, and &lt;code&gt;cex&lt;/code&gt; arguments of &lt;code&gt;pdf()&lt;/code&gt; when exporting figure so as to get the most clean and readable figure. It also happens to me to rely on a different layout--figure 60% and caption 40%, side by side, 100% of text width--for small illustrations or graphics.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-13T10:18:47.823" LastActivityDate="2010-11-13T10:18:47.823" CommunityOwnedDate="2010-11-13T10:25:47.020" CommentCount="1" />
  <row Id="4504" PostTypeId="2" ParentId="4491" CreationDate="2010-11-13T09:19:10.647" Score="3" ViewCount="" Body="&lt;p&gt;I prefer to use dedicated symbol for that purpose. For example, use &lt;code&gt;points()&lt;/code&gt; with &lt;code&gt;pch=17&lt;/code&gt; (filled triangle) like in the example below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dd &amp;lt;- dotplot(m~i|p,data=res.EQ.long,subset=i %in% eq5d.items.names[1:5],&#xA;              ylab=&quot;Score&quot;,ylim=c(0.5,3.5),  &#xA;              scales=list(x=list(rot=45,at=1:5,&#xA;                          labels=eq5d.items.names[1:5],cex=c(.8,.8))),&#xA;              main=&quot;EQ-5D&quot;,&#xA;              panel=function(x,y,subscripts,...){&#xA;              panel.dotplot(x,y,...)&#xA;              panel.lines(x=c(1,5),y=3,lwd=1.5, col=&quot;gray50&quot;)&#xA;              panel.segments(x0=c(1,3,5),y0=rep(2.95,3),&#xA;                             x1=c(1,3,5),y1=rep(3.05,5),&#xA;                             lwd=1.5, col=&quot;gray50&quot;)&#xA;              panel.text(x=c(1,3,5),y=3,c(0,50,100),cex=.6,pos=3)&#xA;              panel.text(x=3,y=3.2,&quot;Health State&quot;,cex=.6,pos=3)&#xA;              panel.points(x=(4*ref[packet.number()])/100+1,y=2.9,&#xA;                           pch=17)&#xA;              panel.points(x=1:5,y=res.EQ2[1:5,packet.number()],&#xA;                           col=&quot;red&quot;,pch=19,cex=.6)&#xA;              },&#xA;              key=list(text=list(c(&quot;Mean&quot;,&quot;Median&quot;)),x=0.7,y=0.9,&#xA;                       lines=list(col=c(&quot;blue&quot;,&quot;red&quot;),type=&quot;p&quot;,&#xA;                       pch=19,cex=.8)))&#xA;print(dd)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/ikc4z.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The complete list of symbols can be obtained with &lt;code&gt;show.pch()&lt;/code&gt; from the &lt;a href=&quot;http://cran.r-project.org/web/packages/Hmisc/index.html&quot; rel=&quot;nofollow&quot;&gt;Hmisc&lt;/a&gt; package.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-11-13T09:19:10.647" />
  <row Id="4505" PostTypeId="2" ParentId="4498" CreationDate="2010-11-13T10:05:45.920" Score="2" ViewCount="" Body="&lt;p&gt;As &lt;a href=&quot;http://stats.stackexchange.com/questions/4498/whats-the-relation-between-hierarchical-models-neural-networks-graphical-model/4502#4502&quot;&gt;@carlosdc&lt;/a&gt; said, a bayesian network is a type of Graphical Model (i.e., a directed acyclic graph (DAG) whose structure defines a set of conditional independence properties). &lt;a href=&quot;http://en.wikipedia.org/wiki/Hierarchical_Bayes_model&quot; rel=&quot;nofollow&quot;&gt;Hierarchical Bayes Models&lt;/a&gt; can also be represented as DAGs; &lt;a href=&quot;http://www.labmedinfo.org/download/lmi339.pdf&quot; rel=&quot;nofollow&quot;&gt;Hierarchical Naive Bayes Classifiers for uncertain data&lt;/a&gt;, by Bellazzi et al., provides a good introduction to classification with such models. About hierarchical models, I think many articles can be retrieved by googling with appropriate keywords; for example, I found this one:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;C. H. Jackson, N. G. Best and S.&#xA;  Richardson. &lt;a href=&quot;http://biostatistics.oxfordjournals.org/content/10/2/335.full&quot; rel=&quot;nofollow&quot;&gt;Bayesian graphical models&#xA;  for regression on multiple data sets&#xA;  with different variables&lt;/a&gt;.&#xA;  &lt;em&gt;Biostatistics&lt;/em&gt; (2008) 10(2): 335-351.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Michael I. Jordan has a nice tutorial on &lt;a href=&quot;http://www.cs.berkeley.edu/~jordan/papers/statsci.ps&quot; rel=&quot;nofollow&quot;&gt;Graphical Models&lt;/a&gt;, with various applications based on the factorial &lt;a href=&quot;http://en.wikipedia.org/wiki/Hidden_Markov_model&quot; rel=&quot;nofollow&quot;&gt;Hidden Markov model&lt;/a&gt; in bioinformatics or natural language processing. His book, &lt;a href=&quot;http://mitpress.mit.edu/catalog/item/default.asp?tid=8141&amp;amp;ttype=2&quot; rel=&quot;nofollow&quot;&gt;Learning in Graphical Models&lt;/a&gt; (MIT Press, 1998), is also worth reading (there's an application of GMs to structural modeling with &lt;a href=&quot;http://www.mrc-bsu.cam.ac.uk/bugs/&quot; rel=&quot;nofollow&quot;&gt;BUGS&lt;/a&gt; code, pp. 575-598)&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-11-13T10:05:45.920" />
  <row Id="4506" PostTypeId="2" ParentId="4500" CreationDate="2010-11-13T11:37:51.957" Score="4" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Regarding your first question:&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;What strategies do people use to map studies, publications, and analyses onto repositories?&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;One year ago or so, I decided to have one repository for each publication, presentation or semester/class. My typical directory looks like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;.git&lt;br&gt;&#xA;.gitignore&lt;br&gt;&#xA;README.org&lt;br&gt;&#xA;ana&lt;br&gt;&#xA;dat&lt;br&gt;&#xA;doc&lt;br&gt;&#xA;org&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The underlying idea is (hopefully) obvious: Each publication, presentation, class is an &quot;autarchic entity&quot; which I could easily share with others.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By the way, you are not the first to ask this question: &lt;a href=&quot;http://stackoverflow.com/questions/2475520/one-repository-multiple-projects-without-getting-mixed-up&quot;&gt;One repository/multiple projects without getting mixed up?&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I also started using git for managing projects which might result in some publications (the directory structure follows roughly John Myles White's &lt;a href=&quot;https://github.com/johnmyleswhite/ProjectTemplate&quot; rel=&quot;nofollow&quot;&gt;ProjectTemplate&lt;/a&gt;, without using it, though). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;.git&lt;br&gt;&#xA;.gitignore&lt;br&gt;&#xA;README.org&lt;br&gt;&#xA;ana&lt;br&gt;&#xA;data&lt;br&gt;&#xA;docs&lt;br&gt;&#xA;graphs&lt;br&gt;&#xA;lib&lt;br&gt;&#xA;org&lt;br&gt;&#xA;reports&lt;br&gt;&#xA;tests&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Regarding your second question:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;When should related entities (e.g., publications, studies, etc.) be split into multiple repositories?&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I cannot think of any reason to split a publication, conference etc. related repository into multiple repositories. But I would be interested in other opinions...&lt;/p&gt;&#xA;" OwnerUserId="307" LastActivityDate="2010-11-13T11:37:51.957" CommunityOwnedDate="2010-11-13T14:00:03.970" CommentCount="7" />
  <row Id="4507" PostTypeId="1" AcceptedAnswerId="4514" CreationDate="2010-11-13T13:41:20.420" Score="4" ViewCount="110" Body="&lt;p&gt;Let $G = (V,E)$  be a directed acyclic graph.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let $i \rightarrow j$  be an edge such that the parents of $j$  are exactly:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;the parents of $i$,&lt;/li&gt;&#xA;&lt;li&gt;and $i$.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Let $L\left(G \right)$  be the set defined by {a vertex and its parents, for all vertices in $G$}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let $G&amp;#39; = (V,F)$  where $F$  is $E$  with $j \rightarrow i$ instead of $i \rightarrow j$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given a set of random variables $(X_{v})_{v \in V}$  for which the joint density can be factorized over $L\left(G \right)$, I would like to show that the joint density could also be factorized over $L\left(G&amp;#39; \right)$. &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Let $P$ be the parents of $i$ in the graph $G$, thus the parents of $j$ in the graph $G&amp;#39;$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I would like to show that there exists $g_{i}$ and $g_{j}$ such that: $g_{i}(x_{i},x_{j},x_{P}) \times g_{j}(x_{j},x_{P}) = f_{i}(x_{i},x_{P}) \times f_{j}(x_{i},x_{j},x_{P})$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am unsure whether I could just consider that there was a swap of $i$ and $j$, and I could take $g_{i}=f_{j}$ and $g_{j}=f_{i}$. Or maybe, there is something to do with conditional probabilities and Bayes formula.&lt;/p&gt;&#xA;" OwnerUserId="1351" LastEditorUserId="1351" LastEditorDisplayName="" LastEditDate="2010-11-13T21:21:50.660" LastActivityDate="2010-11-13T22:15:43.690" Title="Factorization of a joint density over a directed graph" Tags="&lt;distributions&gt;&lt;data-visualization&gt;&lt;homework&gt;&lt;graphical-model&gt;" AnswerCount="1" />
  <row Id="4508" PostTypeId="2" ParentId="4496" CreationDate="2010-11-13T16:04:13.600" Score="2" ViewCount="" Body="&lt;p&gt;For the first question try:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plot(data2,xlab='',ylab='entropy (bytes)', xaxs = 'i', yaxs = 'i')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1050" LastEditorUserId="1050" LastEditorDisplayName="" LastEditDate="2010-11-13T16:54:51.440" LastActivityDate="2010-11-13T16:54:51.440" />
  <row Id="4509" PostTypeId="2" ParentId="4496" CreationDate="2010-11-13T17:28:45.037" Score="2" ViewCount="" Body="&lt;p&gt;For the second Q, you can use &lt;code&gt;axis.Date()&lt;/code&gt; with argument &lt;code&gt;labels = FALSE&lt;/code&gt; to add minor tick marks at locations defined by argument &lt;code&gt;at&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1390" LastActivityDate="2010-11-13T17:28:45.037" CommentCount="1" />
  <row Id="4510" PostTypeId="1" AcceptedAnswerId="4511" CreationDate="2010-11-13T17:47:12.277" Score="1" ViewCount="266" Body="&lt;p&gt;I have fit a linear model using the &lt;code&gt;lm&lt;/code&gt; function in R...&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model &amp;lt;- lm(trans.baseline.CD4 ~ hiv$Julian.Date)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;... and I would like to assess the quality of the model's fit. Is there a function in R that will do this? Alternatively, I found a formula for goodness-of-fit involving the sum of squared residuals given the null and alternative hypotheses, but I don't know how to get these values either. Any pointers?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="1973" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-13T18:17:03.820" LastActivityDate="2010-11-13T18:17:03.820" Title="How to compute goodness of fit for a linear model in R" Tags="&lt;r&gt;&lt;regression&gt;&lt;goodness-of-fit&gt;" AnswerCount="2" />
  <row Id="4511" PostTypeId="2" ParentId="4510" CreationDate="2010-11-13T17:56:39.150" Score="4" ViewCount="" Body="&lt;p&gt;It all starts with &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; summary(model)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;after your fit.  There are numerous commands to assess the fit, test commands, compare alternative models, ... in base R as well as in add-on packages on CRAN. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But you may want to do some reading, for example with Dalgaard's book or another introduction to statistics with R.&lt;/p&gt;&#xA;" OwnerUserId="334" LastActivityDate="2010-11-13T17:56:39.150" CommentCount="3" />
  <row Id="4512" PostTypeId="2" ParentId="4510" CreationDate="2010-11-13T17:59:01.260" Score="1" ViewCount="" Body="&lt;p&gt;You should probably take a look at this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://stackoverflow.com/questions/1181025/goodness-of-fit-functions-in-r&quot;&gt;http://stackoverflow.com/questions/1181025/goodness-of-fit-functions-in-r&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1540" LastActivityDate="2010-11-13T17:59:01.260" CommentCount="0" />
  <row Id="4513" PostTypeId="1" CreationDate="2010-11-13T21:43:52.613" Score="3" ViewCount="108" Body="&lt;p&gt;Can you give me examples of machine learning algorithms which learn from the statistical properties of the dataset not the individual observations itself i.e. employ the statistical query model?&lt;/p&gt;&#xA;" OwnerUserId="1961" LastActivityDate="2010-11-13T21:43:52.613" Title="Statistical query model algorithms?" Tags="&lt;machine-learning&gt;&lt;algorithms&gt;" CommentCount="2" FavoriteCount="2" />
  <row Id="4514" PostTypeId="2" ParentId="4507" CreationDate="2010-11-13T22:15:43.690" Score="4" ViewCount="" Body="&lt;p&gt;What you are looking for is termed a &lt;strong&gt;covered edge reversal&lt;/strong&gt;, which preserves Markov equivalence of directed acyclic graphs. A proof of equivalence is given in Lemma 1 of &lt;em&gt;&lt;a href=&quot;http://research.microsoft.com/pubs/65664/uai95.pdf&quot; rel=&quot;nofollow&quot;&gt;A Transformational Characterization of Equivalent Bayesian Network Structures&lt;/a&gt;&lt;/em&gt;, by Max Chickering. The relationship between Markov properties and factorisation is well established in the theory of graphical models, for example &lt;a href=&quot;http://books.google.co.uk/books?id=G_4E_w_wJzcC&amp;amp;lpg=PA74&amp;amp;vq=DF&amp;amp;pg=PA74#v=onepage&amp;amp;q=Theorem%205.14&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;Theorem 5.14&lt;/a&gt; of &lt;em&gt;Probabilistic Networks and Expert Systems&lt;/em&gt;.&lt;/p&gt;&#xA;" OwnerUserId="495" LastActivityDate="2010-11-13T22:15:43.690" CommentCount="2" />
  <row Id="4515" PostTypeId="1" CreationDate="2010-11-13T22:59:12.890" Score="2" ViewCount="161" Body="&lt;p&gt;Lets say we have $T$ tags and $N$ articles and lets say that for each tag $t_{i}$ we know that it has tagged $n_{i}$ articles. Meaning that, frequency($t_{i}$)=$n_{i}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given the above information how could I compute the probability P($t_{i}$ to tag a new article) or more generally how could I go about scoring the &quot;importance&quot; of each tag in Tag set ?&lt;/p&gt;&#xA;" OwnerUserId="1975" LastEditorUserId="1643" LastEditorDisplayName="" LastEditDate="2011-04-28T13:06:39.350" LastActivityDate="2011-04-28T13:06:39.350" Title="How to measure/weight the importance of tags?" Tags="&lt;probability&gt;&lt;data-mining&gt;&lt;predictive-models&gt;&lt;scores&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="2" />
  <row Id="4516" PostTypeId="2" ParentId="4396" CreationDate="2010-11-13T23:21:12.727" Score="2" ViewCount="" Body="&lt;p&gt;Actually, if you want ranks, try the RANK command.  Or use the menus: Transform&gt;Rank Cases.  It can generate ranks within groups.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you prefer the AGGREGATE approach, note that it does NOT require sorting first.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;HTH,&#xA;Jon Peck&lt;/p&gt;&#xA;" OwnerUserId="1971" LastActivityDate="2010-11-13T23:21:12.727" CommentCount="1" />
  <row Id="4517" PostTypeId="1" AcceptedAnswerId="4518" CreationDate="2010-11-14T02:50:03.993" Score="6" ViewCount="1619" Body="&lt;p&gt;Is it possible to have a (multiple) regression equation with two or more dependent variables?  Sure, you could run two separate regression equations, one for each DV, but that doesn't seem like it would capture any relationship between the two DVs?&lt;/p&gt;&#xA;" OwnerUserId="1977" LastActivityDate="2011-02-09T12:13:03.597" Title="Regression with multiple dependent variables?" Tags="&lt;regression&gt;" AnswerCount="5" />
  <row Id="4518" PostTypeId="2" ParentId="4517" CreationDate="2010-11-14T03:32:24.503" Score="11" ViewCount="" Body="&lt;p&gt;Yes, it is possible.  What you're interested is is called &quot;Multivariate Multiple Regression&quot; or just &quot;Multivariate Regression&quot;.  I don't know what software you are using, but you can do this in R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's a link that provides examples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.psych.yorku.ca/lab/psy6140/lectures/MultivariateRegression2x2.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.psych.yorku.ca/lab/psy6140/lectures/MultivariateRegression2x2.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="485" LastActivityDate="2010-11-14T03:32:24.503" CommentCount="2" />
  <row Id="4519" PostTypeId="1" AcceptedAnswerId="5931" CreationDate="2010-11-14T05:28:34.397" Score="1" ViewCount="109" Body="&lt;p&gt;I plotted a set of about 200,000 points and got a triangular shaped region.  The shape is roughly like the triangle made by the points $(1,0)$, $(0,1)$ and $(0,0)$.  My points have the property that if $(a,b)$ is a member of the set, then $(b,a)$ is also a member.  I wanted to do a regression on the variable associated with the second coordinate y on the variable associated with the first coordinate x. I did a linear regression but I'm bothered by the fact that the region is shaped nothing like a line.  Is there a better way to do this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;b&gt;Edit&lt;/b&gt;: The points represent a type of game.  For each player i and j, i can attack j or j can attack i.  The points are produced by looking at (number of times where i attacks j,  of times j attacks i) for all pairs i and j.  This is the reason that we get (a,b) being a member of my set if (b,a) is a member.  I felt that the triangle region suggests that players who don't attack enough have a greater chance of being attacked.  Just to emphasize, my points fill the whole region bounded by a triangle similar to (1,0), (0,1) and (0,0), not necessarily uniformly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;b&gt;Second Edit&lt;/b&gt;:The region is shaped like the triangle I mentioned but bigger.  Lets say for arguments sake, it roughly looks like the triangle (1000,0), (0,1000), (0,0).  There are a few scattered points outside the triangle and the triangle itself is not completely filled out.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My goal is to characterize or relate the number of attacks that i makes on j to the number of attacks that j makes on i.&lt;/p&gt;&#xA;" OwnerUserId="847" LastEditorUserId="847" LastEditorDisplayName="" LastEditDate="2010-11-14T20:50:03.263" LastActivityDate="2011-01-03T19:53:37.627" Title="Regression on a triangular shaped region of points representing a symmetric relation" Tags="&lt;regression&gt;" AnswerCount="1" CommentCount="4" />
  <row Id="4520" PostTypeId="2" ParentId="4515" CreationDate="2010-11-14T09:40:39.110" Score="1" ViewCount="" Body="&lt;p&gt;First for calculating the importance of each tag in a new article I would use the bag of words format to store the corpus information plus the new article added to the last row. I would calculate the tf-idf of the sparse matrix containing the rows with articles and the columns with words. Then I would just Keep the columnns with the tags. Then I would get the tf-idf values of the new article tags, add them all in order to get the total, an to calculate the probability of each one I would divide the tf-idf of the tag from the last article cell divided the calculated total and multiply it by 100. This will provide me the percentage that each tag represents from the article.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For calculating the importance of a tag in a tag-set I would add all the tag columns. From those totals I would add them all and get the grand total. Then I would take each tag divide it for the grand total and multiply it by 100. This will provide me with the percentage or total importance of each tag.  &lt;/p&gt;&#xA;" OwnerUserId="1808" LastActivityDate="2010-11-14T09:40:39.110" CommentCount="1" />
  <row Id="4521" PostTypeId="2" ParentId="4517" CreationDate="2010-11-14T09:43:19.713" Score="-1" ViewCount="" Body="&lt;p&gt;I would do this by first transforming the regression variables to PCA calculated variables, and then I would to the regression with the PCA calculated variables. Of course I would store the eigenvectors to be able to calculate the corresponding pca values when I have a new instance I wanna classify.&lt;/p&gt;&#xA;" OwnerUserId="1808" LastActivityDate="2010-11-14T09:43:19.713" CommentCount="1" />
  <row Id="4522" PostTypeId="2" ParentId="4517" CreationDate="2010-11-14T10:44:06.097" Score="0" ViewCount="" Body="&lt;p&gt;It's called structural equation model or simultaneous equation model. &lt;/p&gt;&#xA;" OwnerUserId="1966" LastActivityDate="2010-11-14T10:44:06.097" CommentCount="1" />
  <row Id="4523" PostTypeId="1" AcceptedAnswerId="4524" CreationDate="2010-11-14T16:21:47.483" Score="0" ViewCount="102" Body="&lt;p&gt;The following code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(zoo)&#xA;data &amp;lt;- read.csv(file=&quot;summary.csv&quot;,sep=&quot;,&quot;,head=TRUE)&#xA;cum  = zoo(data$dcomp, as.Date(data$date))&#xA;data = zoo(data$compressed, as.Date(data$date))&#xA;data &amp;lt;- aggregate(data, identity, tail, 1)&#xA;cum  &amp;lt;- aggregate(cum, identity, sum, 1)&#xA;days = seq(start(data), end(data), &quot;day&quot;)&#xA;data2 = na.locf(merge(data, zoo(,days)))&#xA;&#xA;pdf(file='timeseries.pdf',width=9,height=5)&#xA;par(bty = 'n')&#xA;plot(data2,xlab='',ylab='entropy (bytes)',axes=FALSE)&#xA;axis(side = 2, at=c(991, 20000, 40000, 53048))&#xA;axis(side = 2, at=c(10000, 30000),labels=FALSE)&#xA;lines(cum,type=&quot;h&quot;,lwd=0.3,col=rgb(0.64,0.08,0.00))&#xA;axis.Date(side = 1, days, at=c(&quot;2007-07-25&quot;, &quot;2008-01-01&quot;, &quot;2009-01-01&quot;, &quot;2010-01-01&quot;, &quot;2010-06-21&quot;))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Yields the following Graph:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/k3Yei.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The questions (all related to the bottom axis) are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;How can I make it display '2007/07' and '2010/06' in the beginning and end labels?&lt;/li&gt;&#xA;&lt;li&gt;How can I make it to automatically add minor ticks to the months (without labels)? &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="990" LastActivityDate="2010-11-14T17:28:03.050" Title="Tweaking axis properties in a timeseries graph" Tags="&lt;r&gt;&lt;data-visualization&gt;" AnswerCount="1" />
  <row Id="4524" PostTypeId="2" ParentId="4523" CreationDate="2010-11-14T17:28:03.050" Score="2" ViewCount="" Body="&lt;p&gt;Two quick suggestions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;code&gt;axis.Date()&lt;/code&gt; has a format field, set it to `&quot;%Y/%m&quot; to get '2007/07' (but then for all labels)&lt;/li&gt;&#xA;&lt;li&gt;See &lt;code&gt;help(axTicks)&lt;/code&gt; and &lt;code&gt;help(rug)&lt;/code&gt; which should help you.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Also try the &lt;a href=&quot;http://packages.r-project.org/package=xts&quot; rel=&quot;nofollow&quot;&gt;xts&lt;/a&gt; package which extends the &lt;a href=&quot;http://packages.r-project.org/package=zoo&quot; rel=&quot;nofollow&quot;&gt;zoo&lt;/a&gt; package. You may find reading the source for the plotting functions in both packages helpful.&lt;/p&gt;&#xA;" OwnerUserId="334" LastActivityDate="2010-11-14T17:28:03.050" CommentCount="1" />
  <row Id="4525" PostTypeId="2" ParentId="4500" CreationDate="2010-11-14T17:50:08.343" Score="2" ViewCount="" Body="&lt;p&gt;I keep a separate repository for each project, with a project being centered around a particular data set or question being addressed. The repo contains the data, code, and Sweave documents/plots that explain and express the results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I maintain a separate repo for each discrete publication or presentation because&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;A single project may result in multiple publications or presentations. Once a publication is out or you've given the presentation, you are essentially &quot;done&quot; with the contents of that repo so they don't need to be dragged around with the project.&lt;/li&gt;&#xA;&lt;li&gt;A &lt;em&gt;output&lt;/em&gt; (publication/presentation/chapter) may contain data from more than one project.&lt;/li&gt;&#xA;&lt;li&gt;Not all of the results from a project will end up in a particular piece of &lt;em&gt;output&lt;/em&gt;. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Code that is reusable across projects gets its own repo, as well. If I come up with a new &amp;amp; discrete question using data that is already in one repo, I'll copy that data to a new repo.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to be really strict about it, many version control systems offer the idea of &quot;subprojects&quot;, but I've found that to be overkill.&lt;/p&gt;&#xA;" OwnerUserId="1916" LastActivityDate="2010-11-14T17:50:08.343" CommunityOwnedDate="2010-11-14T17:50:08.343" CommentCount="1" />
  <row Id="4526" PostTypeId="2" ParentId="4453" CreationDate="2010-11-14T19:02:45.520" Score="5" ViewCount="" Body="&lt;p&gt;To avoid the ambiguity of a terminology that refers to everything as an &quot;item,&quot; let's suppose the set $S$ consists of $n$ distinct &quot;letters.&quot;  And, because this is a pretty generalization of the Birthday Paradox, let's generalize it a little further and set $P$ to be the set of all $r$-letter words having all distinct letters.  The question as stated corresponds to $r=2$.  We ask more generally&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;What is the expected number of distinct letters found in the $r k$-letter word obtained by concatenating $k$ random $r$-letter words of distinct letters?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;We can compute this in three stages:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;For any $m$, $r \le m \le n$, find the proportion of the $k$-fold $r$-letter words having $m$  &lt;em&gt;or fewer&lt;/em&gt; distinct letters.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;From this obtain the probability $p_{n,k,r}(m)$ of exactly $m$ distinct letters.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Compute the expected number of letters by summing $m p_{n,k,r}(m)$ over $m$ (from $r$ to $n$).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;We find the proportions in #1 with a polynomial expansion.  Let $f_r(x_1, x_2, \ldots, x_n)$ be the average of all words of length $r$ in which the letters are distinct.  For example, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$f_2(x_1, x_2, x_3) = \left( x_1 x_2 + x_2 x_1 + x_2 x_3 + x_3 x_2 + x_1 x_3 + x_3 x_1 \right)/6 \text{.}$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The $k^\text{th}$ power of this (computed as a polynomial) displays the probabilities with which which words can be formed by a $k$-fold concatenation.  For example, inspection of the coefficients of&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$f_2(x_1, x_2, x_3)^2 = \left( x_1^2 x_2^2 + x_2^2 x_3^2 + x_1^2 x_3^2 + 2 x_1^2 x_2 x_3 + 2 x_1 x_2^2 x_3 + 2 x_1 x_2 x_3^2 \right) / 9 $$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;shows that the chance of forming a word with two $x_1$'s and two $x_2$'s (which can occur in four ways out of the 36 possible ways: $x_1x_2x_1x_2$, $x_1x_2x_2x_1$, $x_2x_1x_1x_2$, and $x_2x_1x_2x_1$) is 1/9.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When we set some of the $x_i$ equal to zero, they drop out of $f_r^k$ altogether, and when we set the remaining $x_i$ equal to one, what remains is the sum of their coefficients.  Thus we could hope to find the probability of having $m$ distinct letters by setting exactly $m$ of the $x_i$ to one (and the remainder to zero) and summing the value of $f_r^k$ over all such possibilities.  (This is easily done because the symmetry of $f_r$ implies all these values are equal, so we need only compute $f_r^k(1,\ldots,1,0,\ldots,0)$ and multiply that by ${n}\choose{m}$.)  Unfortunately, this also over-counts words having fewer than $m$ distinct letters, so we have to apply the &lt;a href=&quot;http://en.wikipedia.org/wiki/Inclusion%E2%80%93exclusion_principle&quot; rel=&quot;nofollow&quot;&gt;Principle of Inclusion-Exclusion&lt;/a&gt; to extract the desired value.  The punchline is that $f_r$ is easy to evaluate; when we set $n-m$ of its arguments to $0$, there remain ${n}\choose{m}$ terms, all of which will evaluate to $1$, whence&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$f_r^k(1,\ldots,1,0,\ldots,0) = {{m}\choose{k}} / {{n}\choose{k}}.$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Putting this together gives&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p_{n,k,r}(m) = {{n}\choose{m}} \sum_{j=0}^{m-r}{(-1)^j {{m}\choose{m-j}} {{m-j}\choose{r}}^k } / {{n}\choose{r}}^k \text{.}$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The expectation equals&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\mathbb{E}[\text{# distinct letters}] = \sum_{m=r}^n{p_{n,k,r}(m) m}.$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This looks like an expression that is unlikely to simplify.  Its calculation requires $O(n^2)$ effort.  When $k$ is small, this can be reduced to $O(n r k)$ by dynamic programming using a recursive relationship between the probability distribution for $k$ and the probability distribution for $k+1$.  The recursion is obtained by considering how $m+r$ distinct letters can arise in a $k+1$-fold concatenation: either $r$ distinct letters are adjoined to a word of $m$ distinct letters, or $r-1$ distinct letters are adjoined to a word of $m+1$ distinct letters, ..., or no distinct letters are introduced to a word of $m+r$ distinct letters.  Rather than write down this recursion generally, I'll display the case $r=2$:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\eqalign{&#xD;&#xA;p_{n,k+1,2}(m+2) &amp;amp;= \Bigl[ {{n-m}\choose{2}} p_{n,k,2}(m) \cr&#xD;&#xA;&amp;amp;+ (m+1)(n-m+1)p_{n,k,2}(m+1) \cr&#xD;&#xA;&amp;amp;+ {{m+2}\choose{2}} p_{n,k,2}(m+2) \Bigr] / {{n}\choose{2}}&#xD;&#xA;}\text{.}$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The dynamic program computes the entire distribution for the case $k=1$ and then uses this recursion to compute it for $k=2, \ldots$ up to the desired value.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-11-30T18:21:49.360" LastActivityDate="2010-11-30T18:21:49.360" CommentCount="2" />
  <row Id="4527" PostTypeId="1" AcceptedAnswerId="4588" CreationDate="2010-11-14T19:55:14.707" Score="0" ViewCount="187" Body="&lt;p&gt;I am exploring the use of the &lt;code&gt;survreg&lt;/code&gt; function in R to analyze my current experiment. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone know what the &quot;Value&quot; column in the output of the function stands for? &lt;/p&gt;&#xA;" OwnerUserId="1862" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-14T19:59:53.407" LastActivityDate="2010-11-16T11:37:33.347" Title="What is the &quot;Value&quot; output of Survreg in R?" Tags="&lt;r&gt;&lt;survival&gt;&lt;interpretation&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="4528" PostTypeId="1" AcceptedAnswerId="4534" CreationDate="2010-11-14T20:39:07.227" Score="2" ViewCount="285" Body="&lt;p&gt;I have been trying to discern what exactly the &quot;coef&quot; and &quot;(exp)coef&quot; output of coxph signify. It seems that the &quot;(exp)coef&quot; are comparisons of the first variable in the model according to the group assigned in the command. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How does the coxph function arrive at the values for &quot;coef&quot; and &quot;(exp)coef&quot;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Additionally, how does coxph determine these values when there is censoring involved?&lt;/p&gt;&#xA;" OwnerUserId="1862" LastActivityDate="2010-11-15T22:25:52.023" Title="What is the difference between the &quot;coef&quot; and &quot;(exp)coef&quot; output of coxph in R?" Tags="&lt;r&gt;&lt;survival&gt;&lt;interpretation&gt;" AnswerCount="2" />
  <row Id="4529" PostTypeId="2" ParentId="4528" CreationDate="2010-11-14T21:10:23.387" Score="3" ViewCount="" Body="&lt;p&gt;To quote the documentation for the print method for a coxph object, obtained in R by typing &lt;code&gt;?survival::print.coxph&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;code&gt;coefficients&lt;/code&gt;    the coefficients of the linear predictor, which multiply the columns of the model matrix.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;That's all the documentation the author of the package provides. The package contains no user guide or package vignette. R is not designed to be user-friendly, and the documentation assumes you already have understand the statistical methods involved.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd assume that the the &lt;code&gt;coef&lt;/code&gt; column gives the above &lt;code&gt;coefficients&lt;/code&gt;, and the &lt;code&gt;exp(coef)&lt;/code&gt; column is the exponential of these. As Cox regression involves a log link function, the coefficients are the &lt;em&gt;log&lt;/em&gt; hazard ratios. Exponentiating them therefore gives you back hazard ratios.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-11-14T21:10:23.387" CommentCount="1" />
  <row Id="4530" PostTypeId="1" CreationDate="2010-11-14T21:14:33.743" Score="1" ViewCount="386" Body="&lt;p&gt;Yesterday, I ran a repeated-measures ANCOVA.  The purpose was to determine the usability of two computer systems, call them 1 and 2.  Each subject completed three (conceptually different) tasks, call them A B and C, on each system.  (The covariate was whether the subject had prior experience with the current system, system 1).  The dependent variable was time taken to complete each task.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A GLM in SPSS prints out both the standard and multivariate results.  The standard, within-subjects results indicated a main effect of system and an interaction of system*task.  Now why would I need to use the multivariate results?  They were significant as well, and I've heard multivariate tests have more power.  But this seems to provide too much wiggle room for researchers-- if the within subjects test isn't significant, they can just look to see if the multivariate results are significant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, since there was an interaction system*task, I ran simple main effects on each task.  Turned out, A and B were significant in one direction, C was significant in the other.  This seems to suggest to me that doing a MANOVA is a bad idea here.  But how would I even know that if I was only looking at the multivariate results?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In short, I guess I just don't understand when to use MANOVA vs ANOVA if I have more than one DV.&lt;/p&gt;&#xA;" OwnerUserId="1977" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-15T00:39:23.103" LastActivityDate="2010-11-15T00:39:23.103" Title="When to use multivariate tests?" Tags="&lt;anova&gt;&lt;spss&gt;&lt;multivariate-analysis&gt;&lt;manova&gt;" AnswerCount="2" FavoriteCount="2" />
  <row Id="4531" PostTypeId="2" ParentId="4530" CreationDate="2010-11-14T22:29:08.370" Score="4" ViewCount="" Body="&lt;p&gt;This is not a complete answer as ANOVA with repeated-measures is a complex topic. You can look at your design from a multivariate point of view if you regard your data not as representing realisations of one DV in different conditions, but of (ultimately) different DVs which are to be analysed simultaneously. (Note that the multiple DVs will not correspond to the 2*3 cells themselves but will represent all linearly independent differences between the levels of a factor for the test of its main effect. For the &quot;system&quot; factor, there is just 1 difference variable, hence the multivariate test for this factor should be equivalent to the univariate approach.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One consideration when choosing between univariate und multivariate tests are the test's assumptions. The univariate approach assumes (among others) sphericity of the error variance-covariance matrix for each of the three tests (factors A, B, interaction A*B). This may be doubtful in many cases - which is why the epsilon-correction schemes were developed. But these are not universally regarded as the best solution. (Note that the sphericity assumption here automatically holds for the &quot;system&quot; factor as it only has 2 levels.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suggest chapters 11-14 in Maxwell &amp;amp; Delaney (2004). Designing Experiments and Analyzing Data. NJ: Lawrence Erlbaum Associates. It is a long read but worth it - the chapters provide an in-depth explanation of the univariate and multivariate approach to one-way and multi-way repeated measures ANOVA.&lt;/p&gt;&#xA;" OwnerUserId="1909" LastEditorUserId="1909" LastEditorDisplayName="" LastEditDate="2010-11-14T22:54:11.630" LastActivityDate="2010-11-14T22:54:11.630" />
  <row Id="4534" PostTypeId="2" ParentId="4528" CreationDate="2010-11-15T00:02:10.313" Score="3" ViewCount="" Body="&lt;p&gt;If you have a single explanatory variable, say treatment group, a Cox's regression model is fitted with &lt;code&gt;coxph()&lt;/code&gt;; the coefficient (&lt;code&gt;coef&lt;/code&gt;) reads as a regression coefficient (in the context of the Cox model, described hereafter) and its exponential gives you the hazard in the treatment group (compared to the control or placebo group). For example, if $\hat\beta=-1.80$, then the hazard is $\exp(-1.80)=0.165$, that is 16.5%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you may know, the hazard function is modeled as &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;h(t)=h_0(t)\exp(\beta&amp;#39;x)&#xD;&#xA;$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $h_0(t)$ is the baseline hazard. The hazards depend multiplicatively on the covariates, and $\exp(\beta_1)$ is the ratio of the hazards between two individuals whose values of $x_1$ differ by one unit when all other covariates are held constant. The ratio of the hazards of any two individuals $i$ and $j$ is $\exp\big(\beta&amp;#39;(x_i-x_j)\big)$, and is called the hazard ratio (or incidence rate ratio). This ratio is assumed to be constant over time, hence the name of &lt;em&gt;proportional hazard&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To echo your preceding question about &lt;code&gt;survreg&lt;/code&gt;, here the form of $h_0(t)$ is left unspecified; more precisely, this is a semi-parametric model in that only the effects of covariates are parametrized, and not the hazard function. In other words, we don't make any distribution assumption about survival times.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The regression parameters are estimated by maximizing the partial log-likelihood defined by&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;\ell=\sum_f\log\left(\frac{\exp(\beta&amp;#39;x_f)}{\sum_{r(f)}\exp(\beta&amp;#39;x_r)}\right)&#xD;&#xA;$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where the first summation is over all deaths or failures $f$, and the second summation is over all subjects $r(f)$ still alive (but at risk) at the time of failure -- this is known as the &lt;em&gt;risk set&lt;/em&gt;. In other words, $\ell$ can be interpreted as the log profile likelihood for $\beta$ after eliminating $h_0(t)$ (or in other words, the LL where the $h_0(t)$ have been replaced by functions of $\beta$ that maximize the likelihood with respect to $h_0(t)$ for a fixed vector $\beta$). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;About censoring, it is not clear whether you refer to left censoring (as might be the case if we consider an origin for the time scale that is earlier than the time when observation began, also called &lt;em&gt;delayed entry&lt;/em&gt;), or right-censoring. In any case, more details about the computation of the regression coefficients and how the &lt;a href=&quot;http://cran.r-project.org/web/packages/survival/index.html&quot; rel=&quot;nofollow&quot;&gt;survival&lt;/a&gt; package handles censoring can be found in Therneau and Grambsch, &lt;a href=&quot;http://www.springer.com/statistics/life+sciences,+medicine+%26+health/book/978-0-387-98784-2&quot; rel=&quot;nofollow&quot;&gt;Modeling Survival Data&lt;/a&gt; (Springer, 2000). &lt;a href=&quot;http://mayoresearch.mayo.edu/mayo/research/biostat/therneau-book.cfm&quot; rel=&quot;nofollow&quot;&gt;Terry Therneau&lt;/a&gt; is the author of the former S package. An &lt;a href=&quot;http://www.mayo.edu/hsr/people/therneau/survival.ps&quot; rel=&quot;nofollow&quot;&gt;online tutorial&lt;/a&gt; is available.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.ddiez.com/teac/surv/R_survival.pdf&quot; rel=&quot;nofollow&quot;&gt;Survival Analysis in R&lt;/a&gt;, by David Diez, provides a good introduction to Survival Analysis in R. A brief overview of $\chi^2$ tests for regression parameters is given p. 10. Hopefully, this should help clarifying the on-line help quoted by &lt;a href=&quot;http://stats.stackexchange.com/questions/4528/what-is-the-difference-between-the-coef-and-expcoef-output-of-coxph-in-r/4529#4529&quot;&gt;@onestop&lt;/a&gt;, &quot;coefficients the coefficients of the linear predictor, which multiply the columns of the model matrix.&quot; For an applied textbook, I recommend &lt;a href=&quot;http://www.springer.com/statistics/life+sciences,+medicine+%26+health/book/978-0-387-98862-7&quot; rel=&quot;nofollow&quot;&gt;Analyzing Medical Data Using S-PLUS&lt;/a&gt;, by Everitt and Rabe-Hesketh (Springer, 2001, chap. 16 and 17), from which most of the above comes from.&#xA;Another useful reference is John Fox's appendix on &lt;a href=&quot;http://cran.r-project.org/doc/contrib/Fox-Companion/appendix-cox-regression.pdf&quot; rel=&quot;nofollow&quot;&gt;Cox Proportional-Hazards Regression for Survival Data&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-15T22:25:52.023" LastActivityDate="2010-11-15T22:25:52.023" CommentCount="1" />
  <row Id="4535" PostTypeId="2" ParentId="4530" CreationDate="2010-11-15T00:28:09.360" Score="3" ViewCount="" Body="&lt;p&gt;Your question brings up a substantial problem with SPSS... too much automatic output.  You are absolutely correct in your implication that the automatic generation of multiple tests is problematic.  Some people take the fact that their computer program does this as evidence that it is good procedure.  It isn't.  Trust your statistical instincts on that point.&lt;/p&gt;&#xA;" OwnerUserId="601" LastActivityDate="2010-11-15T00:28:09.360" />
  <row Id="4536" PostTypeId="2" ParentId="4517" CreationDate="2010-11-15T00:34:37.603" Score="4" ViewCount="" Body="&lt;p&gt;@Brett's response is fine. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are interested in describing your two-block structure, you could also use &lt;a href=&quot;http://en.wikipedia.org/wiki/Partial_least_squares_regression&quot; rel=&quot;nofollow&quot;&gt;PLS regression&lt;/a&gt;. Basically, it is a regression framework which relies on the idea of building successive (orthogonal) linear combinations of the variables belonging to each block such that their covariance is maximal. Here we consider that one block $X$ contains explanatory variables, and the other block $Y$ responses variables, as shown below:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/And8n.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We seek &quot;latent variables&quot; who account for a maximum of information (in a linear fashion) included in the $X$ block while allowing to predict the $Y$ block with minimal error. The $u_j$ and $v_j$ are the loadings (i.e., linear combinations) associated to each dimension. The optimization criteria reads&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;\max_{\mid u_h\mid =1,\mid v_h\mid =1}\text{cov}(X_{h-1}u_h,Yv_h)\quad \big(\equiv \max\text{cov}(\xi_h,\omega_h)\big)&#xD;&#xA;$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $X_{h-1}$ stands for the deflated (i.e., residualized) $X$ block, after the $h^\text{th}$ regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The correlation between factorial scores on the first dimension ($\xi_1$ and $\omega_1$) reflects the magnitude of the $X$-$Y$ link.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-11-15T00:34:37.603" />
  <row Id="4537" PostTypeId="2" ParentId="3242" CreationDate="2010-11-15T06:44:49.957" Score="1" ViewCount="" Body="&lt;p&gt;If you're interested in bivariate power-law functions (as opposed to univariate power-law distributions), then&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Warton et al. &quot;&lt;a href=&quot;http://www.maths.unsw.edu.au/statistics/files/preprint-2005-02.pdf&quot; rel=&quot;nofollow&quot;&gt;Bivariate line-fitting methods for allometry&lt;/a&gt;.&quot; Biol. Rev. 81, 259-201 (2006)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;is an excellent reference. In this case, regression is the right thing to do, although there can be some corrections (OLS vs. RMA, etc.) depending on what you want the results of the regression to mean.&lt;/p&gt;&#xA;" LastActivityDate="2010-11-15T06:44:49.957" />
  <row Id="4538" PostTypeId="2" ParentId="4364" CreationDate="2010-11-15T07:06:37.643" Score="2" ViewCount="" Body="&lt;p&gt;Let $\eta$ and $\xi$ be two independent random variables with a common symmetric distribution such that &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ P\left ( \left |\frac{\xi+\eta}{\sqrt{2}}\right | \geq t \right )\leq P(|\xi|\geq t).$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then these random variables are gaussian. (Obviously, if the $\xi$ and $\eta$ are centered gaussian, it is true.) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is the &lt;a href=&quot;http://www.math.ethz.ch/EMIS/journals/EJP-ECP/EcpVol1/paper2.pdf&quot; rel=&quot;nofollow&quot;&gt;Bobkov-Houdre Theorem&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="223" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-11-15T15:07:17.930" LastActivityDate="2010-11-15T15:07:17.930" CommunityOwnedDate="2010-11-15T07:06:37.643" />
  <row Id="4539" PostTypeId="1" AcceptedAnswerId="4541" CreationDate="2010-11-15T10:23:55.267" Score="2" ViewCount="924" Body="&lt;p&gt;I have kinda puzzled when I heard from the other about the calculation of incidence rate&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From Kenneth Rothman's Modern Epidemiology, the incidence rate is calculated as &lt;code&gt;number of cases in a fix period of time&lt;/code&gt; divided by &lt;code&gt;person-time at risk at that period of time&lt;/code&gt;, which mean if the patient has the disease in the middle of the month, only the first half of patient days will be included in the denominator, but not for the second half. Am I correct?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Problem arises when I saw someone use the following formula for calculation:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;number of cases in a fix period of time&lt;/code&gt;/&lt;code&gt;total person-time at the period of time&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If this is used, then the incidence rate will be underestimated, because the denominator now is bigger than it should be. But this formula seems to be used often, I wonder what this formula does. Thanks!&lt;/p&gt;&#xA;" OwnerUserId="588" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-11-15T11:57:43.427" LastActivityDate="2011-04-11T19:22:04.317" Title="Calculation of incidence rate for epidemiological study in hospital" Tags="&lt;epidemiology&gt;&lt;definition&gt;" AnswerCount="2" />
  <row Id="4540" PostTypeId="2" ParentId="4539" CreationDate="2010-11-15T11:16:20.133" Score="1" ViewCount="" Body="&lt;p&gt;I think Rothman's definition should be used. In the second definition all patients seem to be given the same duration (is this correct?) so that the incidence will not be the incidence rate but will be proportional to the cumulative incidence (cases / total in given time frame)&lt;/p&gt;&#xA;" OwnerUserId="1573" LastActivityDate="2010-11-15T11:16:20.133" />
  <row Id="4541" PostTypeId="2" ParentId="4539" CreationDate="2010-11-15T11:33:03.200" Score="3" ViewCount="" Body="&lt;p&gt;It is commonly admitted that the denominator for IRs is the &quot;population at risk&quot; (i.e., all individuals in which the studied event(s) may occur). Although your first formula is generally used, I found in &lt;em&gt;The new public health&lt;/em&gt;, by Tulchinsky and Varavikova (Elsevier, 2009, 2nd. ed., &lt;a href=&quot;http://books.google.fr/books?id=T2yQwTxRRfcC&amp;amp;pg=PA84&amp;amp;lpg=PA84&amp;amp;dq=incidence+rate+denominator&amp;amp;source=bl&amp;amp;ots=A3HGM26jbQ&amp;amp;sig=4-yAB3WHPLcMQHmiLRcseKASEsU&amp;amp;hl=fr&amp;amp;ei=FRjhTNegA4SGhQf-kYCLDQ&amp;amp;sa=X&amp;amp;oi=book_result&amp;amp;ct=result&amp;amp;resnum=2&amp;amp;ved=0CCYQ6AEwATgo#v=onepage&amp;amp;q=incidence%20rate%20denominator&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;p. 84&lt;/a&gt;) that a distinction is made between &lt;em&gt;ordinary incidence rate&lt;/em&gt;, where the average size of the population in the fixed period of time is used in the denominator, and &lt;em&gt;person-time incidence rate&lt;/em&gt;, with PT at risk in the denominator.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Obviously, when individuals not at risk of the disease are included in the denominator, the resultant measure of disease frequency will underestimate the true incidence of disease in the population under investigation, but see &lt;a href=&quot;http://www.healthknowledge.org.uk/public-health-textbook/research-methods/1a-epidemiology/numerators-denominators-populations&quot; rel=&quot;nofollow&quot;&gt;Numerators, denominators and populations at risk&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-11-15T11:33:03.200" CommentCount="1" />
  <row Id="4542" PostTypeId="1" CreationDate="2010-11-15T16:23:13.670" Score="1" ViewCount="78" Body="&lt;p&gt;After running a study based on interactive genetic algorithms, I have a univariate data file containing multiple participants each doing multiple generations (blocks) of multiple trials.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there an established approach used to analyse this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've looked at using a grand mean on each of the component variables. I've also looked at the modal values of these variables after filtering out all but the final generation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Neither approach seems to properly capture the richness of the data, so I'd like to know how other people would handle this.&lt;/p&gt;&#xA;" OwnerUserId="1950" LastActivityDate="2010-11-16T17:23:45.307" Title="Are there any recommended approaches for analysing data from genetic algorithms?" Tags="&lt;genetic-algorithms&gt;" AnswerCount="1" />
  <row Id="4543" PostTypeId="1" AcceptedAnswerId="4554" CreationDate="2010-11-15T16:35:22.907" Score="3" ViewCount="364" Body="&lt;p&gt;I am fitting a linear model in R using &lt;code&gt;glmnet&lt;/code&gt;. The original (non-regularized) model was fitted using &lt;code&gt;lm&lt;/code&gt; and did not have a constant term (i.e. it was in the form &lt;code&gt;lm(y~0+x1+x2,data)&lt;/code&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;glmnet&lt;/code&gt; takes a matrix of predictors and a vector of responses. I've been reading &lt;code&gt;glmnet&lt;/code&gt; documentation, and can find no mention of the constant term.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, is there a way to ask &lt;code&gt;glmnet&lt;/code&gt; to force the linear fit through the origin?&lt;/p&gt;&#xA;" OwnerUserId="439" LastEditorUserId="439" LastEditorDisplayName="" LastEditDate="2010-11-15T20:54:46.520" LastActivityDate="2010-11-15T20:54:46.520" Title="In R, does &quot;glmnet&quot; fit an intercept?" Tags="&lt;r&gt;&lt;regression&gt;&lt;lasso&gt;" AnswerCount="1" FavoriteCount="2" />
  <row Id="4544" PostTypeId="1" AcceptedAnswerId="4549" CreationDate="2010-11-15T17:01:36.680" Score="5" ViewCount="573" Body="&lt;p&gt;Please provide R code which allows one to conduct a between-subjects ANOVA with -3, -1, 1, 3 contrasts.  I understand there is a debate regarding the appropriate Sum of Squares (SS) type for such an analysis.  However, as the default type of SS used in SAS and SPSS (Type III) is considered the standard in my area.  Thus I would like the results of this analysis to match perfectly what is generated by those statistics programs.  To be accepted an answer must directly call aov(), but other answers may be voted up (espeically if they are easy to understand/use).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sample.data &amp;lt;- data.frame(IV=rep(1:4,each=20),DV=rep(c(-3,-3,1,3),each=20)+rnorm(80))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt; Please note, the contrast I am requesting is not a simple linear or polynomial contrast but is a contrast derived by a theoretical prediction, i.e. the type of contrasts discussed by Rosenthal and Rosnow.&lt;/p&gt;&#xA;" OwnerUserId="196" LastEditorUserId="196" LastEditorDisplayName="" LastEditDate="2010-11-15T21:49:39.773" LastActivityDate="2010-11-15T21:49:39.773" Title="How does one do a Type-III SS ANOVA in R with contrast codes?" Tags="&lt;r&gt;&lt;anova&gt;&lt;contrasts&gt;" AnswerCount="3" CommentCount="1" FavoriteCount="1" />
  <row Id="4545" PostTypeId="2" ParentId="4544" CreationDate="2010-11-15T17:30:52.187" Score="4" ViewCount="" Body="&lt;p&gt;You may want to have a look at this blog post:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://myowelt.blogspot.com/2008/05/obtaining-same-anova-results-in-r-as-in.html&quot; rel=&quot;nofollow&quot;&gt;Obtaining the same ANOVA results in R as in SPSS - the difficulties with Type II and Type III sums of squares&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(&lt;em&gt;Spoiler:&lt;/em&gt; add &lt;code&gt;options(contrasts=c(&quot;contr.sum&quot;, &quot;contr.poly&quot;))&lt;/code&gt; at the beginning of your script)&lt;/p&gt;&#xA;" OwnerUserId="582" LastActivityDate="2010-11-15T17:30:52.187" />
  <row Id="4546" PostTypeId="2" ParentId="4288" CreationDate="2010-11-15T17:36:43.753" Score="0" ViewCount="" Body="&lt;p&gt;I'm not sure I understand your question, but you have no other answers, so I thought I'd give it a shot and revive this topic:  If I understand you correctly for a number of different sampling units (say people) you have estimates that they have given attribute, e.g. there is a probability that they have eaten turkey today - Pr(Turkey).  You also have an estimate of the error in your record of Pr(Turkey).  Thus, you expect that Joe has a Pr(Turkey) of 50% +/- 10%.  Given information of this sort, I think you are interested in the expected number of your sampling units that will have eaten Turkey.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I believe you can simply sum the propensity scores and you'll get the expected number of sampling units with the attribute so long as the error of the estimate is symmetrical.  In short, I'm proposing that the errors of the estimate become unimportant when you aggregate.  The only time the variance is going to matter is if you want the expected variance of your aggregate score.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt; Given the edit to the question I think the approach outlined above may still be salvageable, especially for binary observations.  Taking the supposition that we have 5 persons; 3 of which had a value of 1 for an attribute, 2 of which had a value of 0 for that attribute, each of which had a probability .8,.81,.82.,.83,.84 respectively of being observed correctly. We want to find the expected value of p(having that attribute).  We can invert the probability of having that attribute for those who were judged not to have the attribute such that we can phrase each of the 5 observations in terms of their probability of having the attribute, .8, .81, .82, (1 - .83) .17, and (1 - .84) .16, respectively.  Thus the average Pr of having the attribute should be (.80+.81+.82+.17+.16)/5 = .552.  Two problems exist with this approach.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;It assumes that probabilities are equally probable (if you will).  That is, that that a certainty of .95 and .75 means that the average probability of having the attribute is .85.  It may be the case that averaging on a linear scale isn't the right sort of thing to do.  Perhaps these probabilities should be converted to logits then back, in which case the average probability of .95 (logit: 2.94) and .75 (logit: 1.10) is .88 (logit: 2.02).&lt;/li&gt;&#xA;&lt;li&gt;It has little to say about the case of non-binary classifications.  Should one judge that if the categorization is .90 likely to be correct and there are two other options that there is a .05 chance it is one and .05 chance it is the other?  Should base rates for the other options be considered?  Etc.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="196" LastEditorUserId="196" LastEditorDisplayName="" LastEditDate="2010-11-17T05:28:47.827" LastActivityDate="2010-11-17T05:28:47.827" CommentCount="2" />
  <row Id="4547" PostTypeId="2" ParentId="4462" CreationDate="2010-11-15T17:50:20.730" Score="1" ViewCount="" Body="&lt;p&gt;The median of medians is not the same as the median of the raw scores.  A simple case of this is that when you have an odd number of sales, the median is the middle value; when you have an even number of sales, the median is commonly taken as the average between those two values. A more &quot;real world&quot; challenge to this is that states will sell differing numbers of houses and thus the median of their medians is a poor guess as to the median of all home sales.  Though it also will to be precise, a good first pass estimation would be to find the median of values where each state's median is reflected a number of times proportional to the number of sales in that state.  Thus, I am essentially suggesting a weighted median.&lt;/p&gt;&#xA;" OwnerUserId="196" LastActivityDate="2010-11-15T17:50:20.730" CommentCount="4" />
  <row Id="4548" PostTypeId="2" ParentId="4451" CreationDate="2010-11-15T18:07:57.280" Score="0" ViewCount="" Body="&lt;p&gt;A good resource for finding datasets is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://datasets.reddit.com&quot; rel=&quot;nofollow&quot;&gt;/r/datasets&lt;/a&gt; on Reddit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A quick glance at that page reveals &lt;a href=&quot;http://networkdata.ics.uci.edu/resources.php&quot; rel=&quot;nofollow&quot;&gt;this source&lt;/a&gt;, which might contain something useful for you.&lt;/p&gt;&#xA;" OwnerUserId="115" LastEditorUserId="115" LastEditorDisplayName="" LastEditDate="2010-12-10T05:46:27.450" LastActivityDate="2010-12-10T05:46:27.450" CommunityOwnedDate="2010-11-15T18:07:57.280" />
  <row Id="4549" PostTypeId="2" ParentId="4544" CreationDate="2010-11-15T18:26:07.910" Score="5" ViewCount="" Body="&lt;p&gt;Type III sum of squares for ANOVA are readily available through the &lt;code&gt;Anova()&lt;/code&gt; function from the &lt;a href=&quot;http://cran.r-project.org/web/packages/car/index.html&quot; rel=&quot;nofollow&quot;&gt;car&lt;/a&gt; package.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Contrast coding can be done in several ways, using &lt;code&gt;C()&lt;/code&gt;, the &lt;code&gt;contr.*&lt;/code&gt; family (as indicated by @nico), or directly the &lt;code&gt;contrasts()&lt;/code&gt; function/argument. This is detailed in §6.2 (pp. 144-151) of &lt;em&gt;&lt;a href=&quot;http://www.stats.ox.ac.uk/pub/MASS4/&quot; rel=&quot;nofollow&quot;&gt;Modern Applied Statistics with S&lt;/a&gt;&lt;/em&gt; (Springer, 2002, 4th ed.). Note that &lt;code&gt;aov()&lt;/code&gt; is just a wrapper function for the &lt;code&gt;lm()&lt;/code&gt; function. It is interesting when one wants to control the error term of the model (like in a within-subject design), but otherwise they both yield the same results (and whatever the way you fit your model, you still can output ANOVA or LM-like summaries with &lt;code&gt;summary.aov&lt;/code&gt; or &lt;code&gt;summary.lm&lt;/code&gt;). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't have SPSS to compare the two outputs, but something like&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; library(car)&#xA;&amp;gt; sample.data &amp;lt;- data.frame(IV=factor(rep(1:4,each=20)),&#xA;                            DV=rep(c(-3,-3,1,3),each=20)+rnorm(80))&#xA;&amp;gt; Anova(lm1 &amp;lt;- lm(DV ~ IV, data=sample.data, &#xA;                  contrasts=list(IV=contr.poly)), type=&quot;III&quot;)&#xA;Anova Table (Type III tests)&#xA;&#xA;Response: DV&#xA;            Sum Sq Df F value    Pr(&amp;gt;F)    &#xA;(Intercept)  18.08  1  21.815  1.27e-05 ***&#xA;IV          567.05  3 228.046 &amp;lt; 2.2e-16 ***&#xA;Residuals    62.99 76                      &#xA;---&#xA;Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;is worth to try in first instance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;About factor coding in R vs. SAS: R considers the baseline or reference level as the first level in lexicographic order, whereas SAS considers the last one. So, to get comparable results, either you have to use &lt;code&gt;contr.SAS()&lt;/code&gt; or to &lt;code&gt;relevel()&lt;/code&gt; your R factor.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-11-15T18:26:07.910" CommentCount="5" />
  <row Id="4550" PostTypeId="2" ParentId="4544" CreationDate="2010-11-15T18:31:59.447" Score="0" ViewCount="" Body="&lt;p&gt;Try the Anova command in the car library.  Use the type=&quot;III&quot; argument, as it defaults to type II. For example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(car)&#xA;mod &amp;lt;- lm(conformity ~ fcategory*partner.status, data=Moore, contrasts=list(fcategory=contr.sum, partner.status=contr.sum))&#xA;Anova(mod, type=&quot;III&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="101" LastActivityDate="2010-11-15T18:31:59.447" CommentCount="1" />
  <row Id="4551" PostTypeId="1" CreationDate="2010-11-15T18:46:37.113" Score="33" ViewCount="1629" Body="&lt;p&gt;I'm a grad student in psychology, and as I pursue more and more independent studies in statistics, I am increasingly amazed by the inadequacy of my formal training. Both personal and second hand experience suggests that the paucity of statistical rigor in undergraduate and graduate training is rather ubiquitous within psychology. As such, I thought it would be useful for independent learners like myself to create a list of &quot;Statistical Sins&quot;, tabulating statistical practices taught to grad students as standard practice that are in fact either superseded by superior (more powerful, or flexible, or robust, etc) modern methods or shown to be frankly invalid. Anticipating that other fields might also experience a similar state of affairs, I propose a community wiki where we can collect a list of Statistical Sins across disciplines. Please submit one &quot;sin&quot; per answer.&lt;/p&gt;&#xA;" OwnerUserId="364" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-11-15T20:29:26.957" LastActivityDate="2011-01-06T15:28:05.167" CommunityOwnedDate="2010-11-15T20:28:53.713" Title="What are common statistical sins?" Tags="&lt;best-practices&gt;" AnswerCount="34" CommentCount="6" FavoriteCount="23" />
  <row Id="4552" PostTypeId="2" ParentId="4551" CreationDate="2010-11-15T18:56:19.547" Score="6" ViewCount="" Body="&lt;p&gt;Correlation implies causation, which is not as bad as accepting the Null Hypothesis.&lt;/p&gt;&#xA;" OwnerUserId="1307" LastActivityDate="2010-11-15T18:56:19.547" CommunityOwnedDate="2010-11-15T20:28:53.713" CommentCount="4" />
  <row Id="4553" PostTypeId="2" ParentId="4551" CreationDate="2010-11-15T18:57:56.170" Score="13" ViewCount="" Body="&lt;p&gt;Dichotomization of a continuous predictor variable to either &quot;simplify&quot; analysis or to solve for the &quot;problem&quot; of non-linearity in the effect of the continuous predictor.&lt;/p&gt;&#xA;" OwnerUserId="364" LastEditorUserId="364" LastEditorDisplayName="" LastEditDate="2010-11-15T21:25:05.107" LastActivityDate="2010-11-15T21:25:05.107" CommunityOwnedDate="2010-11-15T20:28:53.713" CommentCount="3" />
  <row Id="4554" PostTypeId="2" ParentId="4543" CreationDate="2010-11-15T19:02:36.613" Score="4" ViewCount="" Body="&lt;p&gt;Yes, an intercept is included in a &lt;a href=&quot;http://cran.r-project.org/web/packages/glmnet/index.html&quot; rel=&quot;nofollow&quot;&gt;glmnet&lt;/a&gt; model, but it is not regularized (cf. &lt;a href=&quot;http://www-stat.stanford.edu/~jhf/ftp/glmnet.pdf&quot; rel=&quot;nofollow&quot;&gt;Regularization Paths for Generalized Linear Models via Coordinate Descent&lt;/a&gt;, p. 13). More details about the implementation could certainly be obtained by carefully looking at the code (for a gaussian family, it is the &lt;code&gt;elnet()&lt;/code&gt; function that is called by &lt;code&gt;glmnet()&lt;/code&gt;), but it is in Fortran.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could try the &lt;a href=&quot;http://cran.r-project.org/web/packages/penalized/index.html&quot; rel=&quot;nofollow&quot;&gt;penalized&lt;/a&gt; package, which allows to remove the intercept by passing &lt;code&gt;unpenalized = ~0&lt;/code&gt; to &lt;code&gt;penalized()&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; x &amp;lt;- matrix(rnorm(100*20),100,20)&#xA;&amp;gt; y &amp;lt;- rnorm(100)&#xA;&amp;gt; fit1 &amp;lt;- penalized(y, penalized=x, unpenalized=~0, &#xA;                    standardize=TRUE) &#xA;&amp;gt; fit2 &amp;lt;- lm(y ~ 0+x)&#xA;&amp;gt; plot((coef(fit1) + coef(fit2))/2, coef(fit2)-coef(fit1))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To get Lasso regularization, you might try something like&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; fit1b &amp;lt;- penalized(y, penalized=x, unpenalized=~0, &#xA;                     standardize=TRUE, lambda1=1, steps=20)&#xA;&amp;gt; show(fit1b)&#xA;&amp;gt; plotpath(fit1b)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As can be seen in the next figure, there is little differences between the regression parameters computed with both methods (left), and you can plot the Lasso path solution very easily (right).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/waDPa.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-11-15T19:02:36.613" CommentCount="1" />
  <row Id="4555" PostTypeId="2" ParentId="4551" CreationDate="2010-11-15T19:12:03.287" Score="11" ViewCount="" Body="&lt;p&gt;Analysis of rate data (accuracy, etc) using ANOVA, thereby assuming that rate data has Gaussian distributed error when it's actually binomially distributed.&#xA;&lt;a href=&quot;http://dx.doi.org/10.1016/j.jml.2007.11.004&quot; rel=&quot;nofollow&quot;&gt;Dixon (2008)&lt;/a&gt; provides a discussion of the consequences of this sin and exploration of more appropriate analysis approaches.&lt;/p&gt;&#xA;" OwnerUserId="364" LastEditorUserId="364" LastEditorDisplayName="" LastEditDate="2010-11-16T15:47:31.527" LastActivityDate="2010-11-16T15:47:31.527" CommunityOwnedDate="2010-11-15T20:28:53.713" CommentCount="4" />
  <row Id="4556" PostTypeId="1" CreationDate="2010-11-15T19:12:46.883" Score="1" ViewCount="788" Body="&lt;p&gt;Having the Transfer Function of a discrete system as such: &#xA;$$H(z) = \frac{0.8}{z(z-0.8)}$$&#xD;&#xA;I am asked to find the Steady State Gain of the system. &#xA;I have the solution and it simply states:&lt;br&gt;&#xA;Steady State Gain:&lt;br&gt;&#xA;$$H(1) = \frac{0.8}{1-0.8} = 4$$  &#xA;I have no idea why this is the case, and the book, the slides, the solutions, my (limited) notes and the internet are seemingly empty on the subject. Is the Steady State Gain of a system always the outcome of the Transfer Function applied to 1? That just sounds ridiculous, especially since I'm not finding any references to it online.&lt;br&gt;&#xA;I was chased out of mathoverflow with this question, those guys really hate homework... Then again, who doesn't. &lt;/p&gt;&#xA;" OwnerUserId="1994" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-11-15T22:57:21.427" LastActivityDate="2011-01-30T18:04:18.460" Title="How to compute the steady state gain from the transfer function of a discrete time system?" Tags="&lt;time-series&gt;&lt;homework&gt;" AnswerCount="2" />
  <row Id="4557" PostTypeId="2" ParentId="4551" CreationDate="2010-11-15T19:16:20.850" Score="2" ViewCount="" Body="&lt;p&gt;Application of least-squares minimization when maximum-likelihood procedures exist.&lt;/p&gt;&#xA;" OwnerUserId="364" LastActivityDate="2010-11-15T19:16:20.850" CommunityOwnedDate="2010-11-15T20:28:53.713" CommentCount="5" />
  <row Id="4558" PostTypeId="2" ParentId="4551" CreationDate="2010-11-15T19:26:09.143" Score="8" ViewCount="" Body="&lt;p&gt;A current popular one is plotting 95% confidence intervals that only relate to the variance of an effect around the raw performance values in repeated measures designs.  For example, a plot of reaction times in a repeated measures design with confidence intervals where the error term is derived from the MSE of a repeated measures ANOVA.  These confidence intervals don't represent anything sensible.  They certainly don't represent anything about the absolute reaction time.  You could use the error term to generate confidence intervals around the effect but that is rarely done.&lt;/p&gt;&#xA;" OwnerUserId="601" LastActivityDate="2010-11-15T19:26:09.143" CommunityOwnedDate="2010-11-15T20:28:53.713" CommentCount="3" />
  <row Id="4559" PostTypeId="2" ParentId="4517" CreationDate="2010-11-15T19:59:37.720" Score="0" ViewCount="" Body="&lt;p&gt;Did you already come across the term &quot;canonical correlation&quot;? There you have sets of variables on the independent as well as on the dependent side. But maybe there are more modern concepts available, the descriptions I have are all of the eighties/nineties...&lt;/p&gt;&#xA;" OwnerUserId="1818" LastActivityDate="2010-11-15T19:59:37.720" CommentCount="2" />
  <row Id="4560" PostTypeId="2" ParentId="4551" CreationDate="2010-11-15T20:01:39.920" Score="4" ViewCount="" Body="&lt;p&gt;My intro psychometrics course in undergrad spent at least two weeks teaching how to perform a stepwise regression. Is there any situation where stepwise regression is a good idea?&lt;/p&gt;&#xA;" OwnerUserId="1118" LastActivityDate="2010-11-15T20:01:39.920" CommunityOwnedDate="2010-11-15T20:28:53.713" CommentCount="4" />
  <row Id="4561" PostTypeId="1" CreationDate="2010-11-15T20:14:11.647" Score="1" ViewCount="227" Body="&lt;p&gt;For a university paper, I have to compare whether two program versions are statistically significantly different. The comparison in terms of runtime performance is straightforward -- I determine whether the confidence intervals for the two sets of measurements overlap. However, in the case of memory footprint, I have no sets of measurements, because the memory footprint does not vary from measurement to measurement (e.g., version 1 has a constant footprint of 920 kb and version 2 consumes 1040 kb). Is there any way to compare both values in a statistically sound manner?&lt;/p&gt;&#xA;" OwnerUserId="1992" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-11-15T22:58:10.100" LastActivityDate="2010-11-16T00:13:45.617" Title="Comparison of two values in a statistically sound manner" Tags="&lt;hypothesis-testing&gt;&lt;statistical-significance&gt;&lt;homework&gt;&lt;inference&gt;" AnswerCount="2" />
  <row Id="4562" PostTypeId="2" ParentId="4561" CreationDate="2010-11-15T20:32:16.927" Score="4" ViewCount="" Body="&lt;p&gt;It is quite simple; memory footprint is constant, this means it has no deviation and thus any difference is significant. &lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-11-15T20:32:16.927" CommentCount="2" />
  <row Id="4563" PostTypeId="2" ParentId="4551" CreationDate="2010-11-15T21:39:05.163" Score="6" ViewCount="" Body="&lt;p&gt;Failing to test the assumption that error is normally distributed and has constant variance between treatments.  These assumptions aren't always tested, thus least-squares model fitting is probably often used when it is actually inappropriate.&lt;/p&gt;&#xA;" OwnerUserId="101" LastEditorUserId="319" LastEditorDisplayName="" LastEditDate="2010-11-17T16:29:42.977" LastActivityDate="2010-11-17T16:29:42.977" CommunityOwnedDate="2010-11-15T21:39:05.163" CommentCount="4" />
  <row Id="4564" PostTypeId="1" AcceptedAnswerId="4763" CreationDate="2010-11-15T21:47:29.903" Score="5" ViewCount="106" Body="&lt;p&gt;It is known that &lt;a href=&quot;http://en.wikipedia.org/wiki/Belief_propagation&quot; rel=&quot;nofollow&quot;&gt;belief propagation&lt;/a&gt; gives exact result on trees, are there interesting examples when &lt;a href=&quot;http://www.merl.com/publications/TR2001-22/&quot; rel=&quot;nofollow&quot;&gt;Generalized Belief Propagation&lt;/a&gt; is exact? (&lt;b&gt;edit&lt;/b&gt; junction tree is not interesting because it is exactly solvable without GBP)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the surface, Belief Propagation passes messages between cliques and separators, whereas GBP allows more general region hierarchy. That helps with convergence rate, but I wonder if this also extends the class of exactly solvable inference problems.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;b&gt;Edit&lt;/b&gt;: as Thomas Minka points out, junction tree algorithm can be viewed as a version of generalized belief propagation. But it can also be viewed as a version of (cluster)belief propagation. What I'm wondering specifically is if GBP can give exact solution for any problem for which BP can't. The motivation is that with exact solution GBP gives result in a finite number of steps and you can view the result as a kind of algebraic factorization of the problem, in the spirit of &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.125.8954&quot; rel=&quot;nofollow&quot;&gt;Generalized Distributive Law&lt;/a&gt; paper&lt;/p&gt;&#xA;" OwnerUserId="511" LastEditorUserId="511" LastEditorDisplayName="" LastEditDate="2010-11-20T22:30:54.403" LastActivityDate="2010-11-20T23:51:23.590" Title="When is Generalized Belief Propagation exact?" Tags="&lt;graphical-model&gt;" AnswerCount="1" />
  <row Id="4565" PostTypeId="1" CreationDate="2010-11-15T22:35:23.360" Score="0" ViewCount="62" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;Possible Duplicate:&lt;/strong&gt;&lt;br&gt;&#xA;  &lt;a href=&quot;http://stats.stackexchange.com/questions/3194/how-can-i-test-the-fairness-of-a-d20&quot;&gt;How can I test the fairness of a d20?&lt;/a&gt;  &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&#xA;&#xA;&lt;p&gt;I know that if I throw a fair six-sided die many times, each of the six sides should come up roughly an equal number of times.  I also know, in the short run, they all won't come up &lt;em&gt;exactly&lt;/em&gt; the same number of times, in fact one side might occur much more than another.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I determine with a reasonable amount certainty that a die is fair?  Suppose, I want to be at least 90% certain it's fair.  How many trials do I have to run to establish fairness and what variability in the number of times that each side shows up is still consistent with it being fair (with 90% confidence)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do you generalize this if there are &lt;em&gt;n&lt;/em&gt; possible outcomes and each of them should occur with the same probability as the others?&lt;/p&gt;&#xA;" OwnerUserId="1618" LastActivityDate="2010-11-15T22:35:23.360" ClosedDate="2010-11-15T23:42:17.520" Title="Determining whether a die is flawed" Tags="&lt;uniform&gt;&lt;probability&gt;" CommentCount="0" />
  <row Id="4566" PostTypeId="2" ParentId="4561" CreationDate="2010-11-16T00:13:45.617" Score="3" ViewCount="" Body="&lt;p&gt;You're incorrect about the confidence intervals.  Assuming they are 95% confidence intervals then they can overlap quite a bit and be statistically significantly different.  What you need is for the confidence interval of the difference between the two to not overlap 0.  The standard error of the difference is sqrt(2 *((var1+var2)/(2(n-1)))/n) and the df for the confidence interval is 2(n-1) (assuming you ran the test equal numbers of times for both programs).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively you could average the w (one side of the confidence interval) from each condition and then multiply that by sqrt(2).  That will give you a minimum difference for statistical significance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the program can run quickly just run it 1e5 times or more (lots and lots) and call the resulting means and standard deviations approximate population values.  Your standard error will be so small that a confidence interval will be almost 0 and any difference would be significant.  You could even report overlayed histogram plots and CDFs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(while I would endorse this last method you might need to do the CIs anyway)&lt;/p&gt;&#xA;" OwnerUserId="601" LastActivityDate="2010-11-16T00:13:45.617" />
  <row Id="4567" PostTypeId="2" ParentId="4551" CreationDate="2010-11-16T00:15:32.840" Score="25" ViewCount="" Body="&lt;p&gt;Most interpretations of p-values are sinful! The conventional usage of p-values is badly flawed; a fact that, in my opinion, calls into question the standard approaches to the teaching of hypothesis tests and tests of significance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Haller and Krause have found that statistical instructors are almost as likely as students to misinterpret p-values. (Take the test in their paper and see how you do.) Steve Goodman makes a good case for discarding the conventional (mis-)use of the p -value in favor of likelihoods. The Hubbard paper is also worth a look.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Haller and Krauss. &lt;a href=&quot;http://www.dgps.de/fachgruppen/methoden/mpr-online/issue16/art1/article.html&quot; rel=&quot;nofollow&quot;&gt;Misinterpretations of significance: A problem students share with their teachers&lt;/a&gt;. Methods of Psychological Research (2002) vol. 7 (1) pp. 1-20 (&lt;a href=&quot;http://www.dgps.de/fachgruppen/methoden/mpr-online/issue16/art1/haller.pdf&quot; rel=&quot;nofollow&quot;&gt;PDF&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hubbard and Bayarri. &lt;a href=&quot;http://pubs.amstat.org/doi/abs/10.1198/0003130031856&quot; rel=&quot;nofollow&quot;&gt;Confusion over Measures of Evidence (p's) versus Errors (α's) in Classical Statistical Testing&lt;/a&gt;. The American Statistician (2003) vol. 57 (3)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Goodman. Toward evidence-based medical statistics. 1: The P value fallacy. Ann Intern Med (1999) vol. 130 (12) pp. 995-1004 (&lt;a href=&quot;http://psg-mac43.ucsf.edu/ticr/syllabus/courses/4/2003/11/13/Lecture/readings/Steven%20Goodman.pdf&quot; rel=&quot;nofollow&quot;&gt;PDF&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also see: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Wagenmakers, E-J. A practical solution to the pervasive problems of p values. Psychonomic Bulletin &amp;amp; Review, 14(5), 779-804.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;for some clear cut cases where even the nominally &quot;correct&quot; interpretation of a p-value has been made incorrect due to the choices made by the experimenter.&lt;/p&gt;&#xA;" OwnerUserId="1679" LastEditorUserId="196" LastEditorDisplayName="" LastEditDate="2010-12-12T21:23:35.587" LastActivityDate="2010-12-12T21:23:35.587" CommunityOwnedDate="2010-11-16T00:15:32.840" CommentCount="8" />
  <row Id="4568" PostTypeId="1" CreationDate="2010-11-16T01:24:41.520" Score="1" ViewCount="230" Body="&lt;p&gt;Thanks for all the answer for the question &lt;a href=&quot;http://stats.stackexchange.com/questions/4539/calculation-of-incidence-rate-for-epidemiological-study-in-hospital&quot;&gt;Calculation of incidence rate for epidemiological study in hospital&lt;/a&gt;. And here come's the second part of the question:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What about the prevalence rate then? I have read &lt;code&gt;The new public health&lt;/code&gt; as suggested by Chi, the book says that prevalence is usually not available when using ordinary incidence rate, but I saw another formula here:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;total case count in that period of time&lt;/code&gt;/&lt;code&gt;total patient bed days during that period of time&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It puzzled me again, what is it? I have never heard of prevalence calculated using denominator as &lt;code&gt;patient-bed days&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="588" LastActivityDate="2010-11-16T01:24:41.520" Title="Calculation of incidence rate for epidemiological study in hospital (Part II) (Prevalence Rate this time)" Tags="&lt;epidemiology&gt;&lt;definition&gt;" />
  <row Id="4569" PostTypeId="1" AcceptedAnswerId="4577" CreationDate="2010-11-16T01:27:37.710" Score="3" ViewCount="402" Body="&lt;p&gt;I would like to gather input from people in the field about the Yates continuity correction for 2 x 2 contingency tables. &lt;a href=&quot;http://en.wikipedia.org/wiki/Yates%27_correction_for_continuity&quot; rel=&quot;nofollow&quot;&gt;The Wikipedia article&lt;/a&gt; mentions it may adjust too far, and is thus only used in a limited sense. The &lt;a href=&quot;http://stats.stackexchange.com/questions/939/yates-correction-for-continuity-only-for-2x2&quot;&gt;related post here&lt;/a&gt; doesn't offer much further insight.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So to the people who use these tests regularly, what are your thoughts? Is it better to use the correction or not?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And a real world example which would yield different results at the 95% confidence level. Note this was a homework problem, but our class does not deal with the Yates continuity correction at all, so sleep easy knowing you aren't doing my homework for me.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;samp &amp;lt;- matrix(c(13, 12, 15, 3), byrow = TRUE, ncol = 2)&#xA;colnames(samp) &amp;lt;- c(&quot;No&quot;, &quot;Yes&quot;)&#xA;rownames(samp) &amp;lt;- c(&quot;Female&quot;, &quot;Male&quot;)&#xA;&#xA;chisq.test(samp, correct = TRUE)&#xA;chisq.test(samp, correct = FALSE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="696" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-11-16T12:02:58.650" LastActivityDate="2010-11-16T12:02:58.650" Title="Yates continuity correction for 2 x 2 contingency tables" Tags="&lt;categorical-data&gt;&lt;chi-squared&gt;&lt;basic-concepts&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="4570" PostTypeId="2" ParentId="4551" CreationDate="2010-11-16T02:30:58.087" Score="1" ViewCount="" Body="&lt;p&gt;I would say, doing tests and regressions on a small set of data.&lt;br&gt;&#xA;&lt;strong&gt;&lt;em&gt;Edit&lt;/strong&gt;: Without looking at the confidence intervals, or when the confidence intervals/error bars are not easy to calculate.&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="1709" LastEditorUserId="1709" LastEditorDisplayName="" LastEditDate="2010-11-19T09:35:48.883" LastActivityDate="2010-11-19T09:35:48.883" CommunityOwnedDate="2010-11-16T02:30:58.087" CommentCount="6" />
  <row Id="4571" PostTypeId="2" ParentId="4569" CreationDate="2010-11-16T02:48:09.623" Score="3" ViewCount="" Body="&lt;p&gt;If you have counts low enough that the Yates Correction is a worry (as in your example), you probably should be using Fisher's exact test.  Otherwise, I recommend that after you use the chi-square test on a 2x2 table, you confirm your test with a log odds-ratio z-test.&lt;/p&gt;&#xA;" OwnerUserId="1920" LastActivityDate="2010-11-16T02:48:09.623" CommentCount="2" />
  <row Id="4572" PostTypeId="2" ParentId="4551" CreationDate="2010-11-16T02:49:20.940" Score="5" ViewCount="" Body="&lt;p&gt;The one that I see quite often and always grinds my gears is the assumption that a significant main effect in one group and a non-significant main effect in another group implies an effect x group interaction.&lt;/p&gt;&#xA;" OwnerUserId="196" LastEditorUserId="196" LastEditorDisplayName="" LastEditDate="2010-11-17T15:09:45.533" LastActivityDate="2010-11-17T15:09:45.533" CommunityOwnedDate="2010-11-16T02:49:20.940" />
  <row Id="4573" PostTypeId="2" ParentId="4451" CreationDate="2010-11-16T07:16:21.443" Score="2" ViewCount="" Body="&lt;p&gt;visit the Max Planck institute. They have also collected several datasets for OSNs.&lt;/p&gt;&#xA;" LastActivityDate="2010-11-16T07:16:21.443" CommunityOwnedDate="2010-11-16T07:16:21.443" CommentCount="1" />
  <row Id="4574" PostTypeId="2" ParentId="4551" CreationDate="2010-11-16T07:52:02.757" Score="2" ViewCount="" Body="&lt;p&gt;(With a bit of luck this will be controversial.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using a Neyman-Pearson approach to statistical analysis of scientific experiments. Or, worse, using an ill-defined hybrid of Neyman-Pearson and Fisher.&lt;/p&gt;&#xA;" OwnerUserId="1679" LastActivityDate="2010-11-16T07:52:02.757" CommunityOwnedDate="2010-11-16T07:52:02.757" CommentCount="2" />
  <row Id="4575" PostTypeId="1" AcceptedAnswerId="4576" CreationDate="2010-11-16T08:10:48.690" Score="7" ViewCount="278" Body="&lt;p&gt;Are there good tutorials on object-oriented programming in R?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It would be great if it included the following:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;how to define a class;&lt;/li&gt;&#xA;&lt;li&gt;differences between S3 and S4 classes;&lt;/li&gt;&#xA;&lt;li&gt;operator overloading (I'd like to be able to write &lt;code&gt;a+b&lt;/code&gt; where &lt;code&gt;a&lt;/code&gt; and &lt;code&gt;b&lt;/code&gt; are instances of the class I have in mind).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="439" LastActivityDate="2010-11-16T08:30:14.563" Title="Tutorials on object-oriented programming in R" Tags="&lt;r&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="4" />
  <row Id="4576" PostTypeId="2" ParentId="4575" CreationDate="2010-11-16T08:20:20.087" Score="7" ViewCount="" Body="&lt;p&gt;Hadley Wicham's wiki on &lt;a href=&quot;https://github.com/hadley/devtools/wiki/&quot; rel=&quot;nofollow&quot;&gt;devtools&lt;/a&gt; is great resource for the necessary information in a concise form. However, if you want an exhaustive resource, the R language manual's &lt;a href=&quot;http://cran.r-project.org/doc/manuals/R-lang.html#Object_002doriented-programming&quot; rel=&quot;nofollow&quot;&gt;OOP&lt;/a&gt; section may be helpful. I am sure more experienced members will have better suggestions.&lt;/p&gt;&#xA;" OwnerUserId="1307" LastActivityDate="2010-11-16T08:20:20.087" CommentCount="1" />
  <row Id="4577" PostTypeId="2" ParentId="4569" CreationDate="2010-11-16T08:22:44.013" Score="3" ViewCount="" Body="&lt;p&gt;Yates' correction results in tests that are more conservative as with Fisher's &quot;exact&quot; tests. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an online tutorial on the use of &lt;a href=&quot;http://faculty.london.edu/cstefanescu/Yates.pdf&quot; rel=&quot;nofollow&quot;&gt;Yates’s continuity correction&lt;/a&gt;, by Stefanescu et al, which clearly points to various flaws of systematic correction for continuity (pp. 4-6). Quoting Agresti (&lt;a href=&quot;http://www.stat.ufl.edu/~aa/cda/cda.html&quot; rel=&quot;nofollow&quot;&gt;CDA&lt;/a&gt; 2002), &quot;Yates (1934) mentioned that Fisher suggested the hypergeometric to him for an exact test&quot;, which led to the continuity-corrected version of the $\chi^2$. Agresti also indicated that Fisher's test is a good alternative now that computers can do it even for large samples (p. 103). Now, the point is that choosing a test really depends on the question that is asked and the assumptions that are made by each of them (e.g., in the case of the Fisher's test we assume that margins are fixed).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case, Fisher test and corrected $\chi^2$ agree and yield $p$-value above 5%. In the case of the ordinary $\chi^2$, if $p$-values are computed using a Monte Carlo approach (see &lt;code&gt;simulate.p.value&lt;/code&gt;), then it fails to reach significance too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other useful references dealing with small sample size issues and the overuse of Fisher's test, include:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;I. Campbell, &lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pubmed/17315184&quot; rel=&quot;nofollow&quot;&gt;Chi-squared and Fisher–Irwin tests of two-by-two tables with small sample recommendations&lt;/a&gt;, Statistics in Medicine 26(19): 3661–3675, 2007.&lt;/li&gt;&#xA;&lt;li&gt;Mark G. Haviland, &lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pubmed/2362976&quot; rel=&quot;nofollow&quot;&gt;Yates's correction for continuity and the analysis of 2 × 2 contingency tables&lt;/a&gt;, Statistics in Medicine 9(4): 363–367, 1990.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-11-16T08:22:44.013" CommentCount="1" />
  <row Id="4578" PostTypeId="2" ParentId="4515" CreationDate="2010-11-16T08:25:35.340" Score="1" ViewCount="" Body="&lt;p&gt;I am not absolutely sure what you want to achieve but I will give it a try.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Measuring Importance&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The concept of importance without any target is rather ambiguous. Suppose you have the tags &quot;football&quot;,&quot;crops&quot; and &quot;funny&quot;. Now want to create a classification algorithm to determine whether a certain article is in the category &quot;agriculture&quot; or &quot;sport&quot;. In this case the tags &quot;football&quot; and &quot;crops&quot; are more important than &quot;funny&quot;. On the other hand, if you want to classify documents whether they are funny or not, &quot;football&quot; and &quot;crops&quot; maybe less helpful. I think you get the idea.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sooo ... in relation to a certain label, you can measure the importance of tags by calculating the &lt;a href=&quot;http://en.wikipedia.org/wiki/Information_gain_ratio&quot; rel=&quot;nofollow&quot;&gt;information gain ratio&lt;/a&gt; or related measures.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Calculating a probability&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Without such a target, you can &quot;only&quot; calculate the probability that a certain tags occurs. In detail: You can calculate this probability either a priori (|Tag=ti|/|tags|) or a posteriori, i.e. the probability that a given article is tagged with a certain tag ti (prob(Tag=ti|Article=art)). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The latter can be achieved by constructing a bayes model (e.g. naive bayes) for each tag. In this case a binary label is constructed with value &quot;1&quot; if the tag is present, else &quot;0&quot;.&lt;/p&gt;&#xA;" OwnerUserId="264" LastActivityDate="2010-11-16T08:25:35.340" />
  <row Id="4579" PostTypeId="1" CreationDate="2010-11-16T08:27:45.800" Score="1" ViewCount="122" Body="&lt;p&gt;My question is: plus or minus?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is the Lagrange function:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;the objective PLUS lambda times constraints, or&lt;/li&gt;&#xA;&lt;li&gt;the objective function MINUS lambda times constraints? &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Example:&#xA;want to maximize A=xy&#xA;subject to g(x,y)=2x+y-400=0&lt;/p&gt;&#xA;&#xA;&lt;p&gt;is F(x,y,lambda):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;xy + lambda (2x+y-400), or&lt;/li&gt;&#xA;&lt;li&gt;xy - lambda (2x+y-400)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I found both notations. Does that mean one can use them interchangeably (i.e. they are the same)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for help&lt;/p&gt;&#xA;" OwnerUserId="2005" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-11-16T09:40:08.800" LastActivityDate="2010-11-16T09:40:08.800" Title="Lagrange function" Tags="&lt;optimization&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="4580" PostTypeId="1" CreationDate="2010-11-16T08:28:42.500" Score="8" ViewCount="448" Body="&lt;p&gt;Who of you in this forum uses &quot;&gt;R with the &lt;a href=&quot;http://cran.r-project.org/web/packages/multicore/index.html&quot; rel=&quot;nofollow&quot;&gt;multicore&lt;/a&gt;, &lt;a href=&quot;http://cran.r-project.org/web/packages/snow/index.html&quot; rel=&quot;nofollow&quot;&gt;snow&lt;/a&gt; packages, or &lt;a href=&quot;http://www.nvidia.com/object/cuda_home_new.html&quot; rel=&quot;nofollow&quot;&gt;CUDA&lt;/a&gt;, so for advanced calculations that need more power than a workstation CPU? On which hardware do you compute these scripts? At home/work or do you have data center access somewhere?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The background of these questions is the following: I am currently writing my M.Sc. thesis about R and High Performance Computing and need a strong knowledge about who actually uses R. I read that R had 1 million users in 2008, but that's more or less the only user statistics I could find on this topic - so I hope for your answers!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sincerely Heinrich&lt;/p&gt;&#xA;" OwnerUserId="2006" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-16T21:05:51.823" LastActivityDate="2010-11-17T01:34:10.317" CommunityOwnedDate="2010-11-16T10:24:08.903" Title="Who uses R with multicore, SNOW or CUDA package for resource intense computing?" Tags="&lt;r&gt;&lt;mathematical-statistics&gt;&lt;computing&gt;&lt;parallel-computing&gt;" AnswerCount="4" CommentCount="2" FavoriteCount="3" />
  <row Id="4581" PostTypeId="2" ParentId="4575" CreationDate="2010-11-16T08:30:14.563" Score="5" ViewCount="" Body="&lt;p&gt;In addition to &lt;a href=&quot;http://stats.stackexchange.com/questions/4575/tutorials-on-object-oriented-programming-in-r/4576#4576&quot;&gt;@suncoolsu&lt;/a&gt; excellent response, there is &lt;a href=&quot;http://cran.r-project.org/doc/contrib/Genolini-S4tutorialV0-5en.pdf&quot; rel=&quot;nofollow&quot;&gt;A (Not So) Short Introduction to S4&lt;/a&gt;, by Christophe Genolini. It is available on CRAN website.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-11-16T08:30:14.563" />
  <row Id="4582" PostTypeId="2" ParentId="4579" CreationDate="2010-11-16T09:27:03.307" Score="1" ViewCount="" Body="&lt;p&gt;It is exactly the same!&lt;br&gt;&#xA;You want the constraint to be respected, and you don't care about the sign of g(x,y)&lt;/p&gt;&#xA;" OwnerUserId="1709" LastActivityDate="2010-11-16T09:27:03.307" CommentCount="3" />
  <row Id="4583" PostTypeId="1" CreationDate="2010-11-16T09:39:35.840" Score="2" ViewCount="115" Body="&lt;p&gt;Do you know any papers utilizing reinforcement learning (RL) techniques to build recommender systems?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I only know these:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.citeulike.org/user/imrchen/article/3831221&quot; rel=&quot;nofollow&quot;&gt;An MDP-Based Recommender System&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.citeulike.org/user/mboehmer/article/4090243&quot; rel=&quot;nofollow&quot;&gt;Usage-based web recommendations: a reinforcement learning approach&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;On the one side it is intuitively correct to apply RL in this area, but on the other side it is hard to test RL models build in the university ivory tower. Hence it would not surprise me if there aren't any more papers.&lt;/p&gt;&#xA;" OwnerUserId="264" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-11-16T10:42:03.560" LastActivityDate="2010-11-16T10:42:03.560" CommunityOwnedDate="2010-11-16T10:41:51.920" Title="Reinforcement Learning in recommender systems" Tags="&lt;machine-learning&gt;&lt;reinforcement-learning&gt;&lt;recommender-system&gt;" CommentCount="0" />
  <row Id="4584" PostTypeId="2" ParentId="4580" CreationDate="2010-11-16T09:49:15.153" Score="3" ViewCount="" Body="&lt;p&gt;Since you ask, I am using the &lt;a href=&quot;http://cran.r-project.org/web/packages/foreach/index.html&quot; rel=&quot;nofollow&quot;&gt;foreach&lt;/a&gt; package with the &lt;a href=&quot;http://cran.r-project.org/web/packages/multicore/index.html&quot; rel=&quot;nofollow&quot;&gt;multicore&lt;/a&gt; backend. I use it to split an &lt;a href=&quot;http://en.wikipedia.org/wiki/Embarrassingly_parallel&quot; rel=&quot;nofollow&quot;&gt;embarrassingly parallel&lt;/a&gt; workload across multiple cores on a single Nehalem box with lots of RAM. This works pretty well for the task at hand.&lt;/p&gt;&#xA;" OwnerUserId="439" LastActivityDate="2010-11-16T09:49:15.153" CommunityOwnedDate="2010-11-16T10:24:08.903" CommentCount="3" />
  <row Id="4585" PostTypeId="1" AcceptedAnswerId="4586" CreationDate="2010-11-16T09:52:30.103" Score="2" ViewCount="255" Body="&lt;p&gt;I was reading about &lt;a href=&quot;http://en.wikipedia.org/wiki/Exponential_smoothing&quot; rel=&quot;nofollow&quot;&gt;exponential smoothing&lt;/a&gt; on Wikipedia.  I saw a sentence saying that I could find the optimal parameter $\alpha$ by using the method of least squares.  How do I go about doing that?  Are there other ways I can find the right $\alpha$?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;b&gt;Edit:&lt;/b&gt; Also, if possible, I'd like a way to handle missing values.&lt;/p&gt;&#xA;" OwnerUserId="847" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-11-16T10:28:32.583" LastActivityDate="2010-11-16T10:28:32.583" Title="Optimal parameter $\alpha$ for exponential smoothing using least squares" Tags="&lt;time-series&gt;&lt;estimation&gt;&lt;optimization&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="4586" PostTypeId="2" ParentId="4585" CreationDate="2010-11-16T10:04:36.643" Score="3" ViewCount="" Body="&lt;p&gt;Minimize the sum of squared one-step forecast errors. If $\hat{Y}_t$ is the prediction of $Y_t$ given $Y_1,\dots,Y_{t-1}$, then $e_t=Y_t-\hat{Y}_t$ is the one-step forecast error. So minimize $e_2^2+\cdots+e_n^2$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also use maximum likelihood estimation as discussed in &lt;a href=&quot;http://www.exponentialsmoothing.net&quot; rel=&quot;nofollow&quot;&gt;my Springer book&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're just using simple exponential smoothing, and are happy to assume normal errors with constant variance, then an ARIMA(0,1,1) model is equivalent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you use a state space representation (such as in the innovations state space form, or by writing the ARIMA model in state space form), then handling missing values is easy. For example, the R function &lt;code&gt;arima()&lt;/code&gt; will handle missing values without complaint.&lt;/p&gt;&#xA;" OwnerUserId="159" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-11-16T10:18:31.483" LastActivityDate="2010-11-16T10:18:31.483" CommentCount="7" />
  <row Id="4587" PostTypeId="2" ParentId="4580" CreationDate="2010-11-16T10:23:55.127" Score="2" ViewCount="" Body="&lt;p&gt;I work in academy and I'm using multicore for some heavy benchmarks of machine learning algorithms, mostly on our Opteron based Sun Constellation and some smaller clusters; those are also rather embarrassingly parallel problems so multicore's main role is to spread computation over node without multiplication of memory usage.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-11-16T10:23:55.127" CommunityOwnedDate="2010-11-16T10:24:08.903" CommentCount="4" />
  <row Id="4588" PostTypeId="2" ParentId="4527" CreationDate="2010-11-16T11:37:33.347" Score="1" ViewCount="" Body="&lt;p&gt;The value column simply gives the values for the coefficient estimates.&lt;/p&gt;&#xA;" OwnerUserId="1806" LastActivityDate="2010-11-16T11:37:33.347" CommentCount="2" />
  <row Id="4589" PostTypeId="2" ParentId="4551" CreationDate="2010-11-16T12:19:21.080" Score="6" ViewCount="" Body="&lt;p&gt;Especially in epidemiology and public health - using arithmetic instead of logarithmic scale when reporting graphs of relative measures of association (hazard ratio, odds ratio or risk ratio).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More information &lt;a href=&quot;http://www.thelancet.com/journals/lancet/article/PIIS0140-6736%2810%2960541-7/fulltext&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="22" LastActivityDate="2010-11-16T12:19:21.080" CommunityOwnedDate="2010-11-16T12:19:21.080" CommentCount="1" />
  <row Id="4590" PostTypeId="2" ParentId="4551" CreationDate="2010-11-16T13:30:48.163" Score="11" ViewCount="" Body="&lt;p&gt;Testing the hypotheses $H_0:  \mu=0$ versus $H_1: \mu\neq 0$&#xD;&#xA; (for example in a gaussian setting)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;to justify that $\mu=0$ in a model (i.e mix &quot;$H_0$ is not rejected&quot; and &quot;$H_0$ is true&quot;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A very good example of that type of (very bad) reasonning is when you test wether the variances of two gaussians are equal (or not) before testing if their mean are equal or not with the assumption of equal variance. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another example occurs when you test normality (versus non normality) to justify normality. every statistician has done that in is life ? it is baaad :) (and should push people to check robustness to non gaussianity) &lt;/p&gt;&#xA;" OwnerUserId="223" LastEditorUserId="223" LastEditorDisplayName="" LastEditDate="2010-12-01T16:16:01.913" LastActivityDate="2010-12-01T16:16:01.913" CommunityOwnedDate="2010-11-16T13:30:48.163" CommentCount="2" />
  <row Id="4591" PostTypeId="2" ParentId="4341" CreationDate="2010-11-16T14:27:29.113" Score="0" ViewCount="" Body="&lt;p&gt;Another option using Excel that gives you a fair amount of flexibility over the number &amp;amp; size of bins is the &lt;code&gt;=frequency(data_array, bins_array)&lt;/code&gt; function. It is an array function that expects two arguments. The first argument is your data, the second is your bins that you define.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's assume your data is in cells A1 - A1000 and create bins in cells B1 - B20. You would want to highlight cells C1 - C21 and then type something like &lt;code&gt;=FREQUENCY(A1:A100, B1:B21)&lt;/code&gt;. Unlike normal functions, array functions must be entered with the key combination &lt;code&gt;SHIFT + CTRL + ENTER&lt;/code&gt;. You should see the counts fill down for all bins, if not - you most likely only hit enter and the first cell is calculated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are plenty of good tutorials online explaining this in more detail, here's a decent &lt;a href=&quot;http://www.exceluser.com/solutions/q0001/frequency.htm&quot; rel=&quot;nofollow&quot;&gt;one&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="696" LastActivityDate="2010-11-16T14:27:29.113" />
  <row Id="4592" PostTypeId="2" ParentId="4551" CreationDate="2010-11-16T16:24:50.713" Score="8" ViewCount="" Body="&lt;p&gt;Not really answering the question, but there's an entire book on this subject:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Phillip I. Good, James William Hardin (2003). &lt;em&gt;Common errors in statistics (and how to avoid them).&lt;/em&gt; Wiley. ISBN &lt;a href=&quot;http://en.wikipedia.org/w/index.php?title=Special%3ABookSources&amp;amp;isbn=9780471460688&quot; rel=&quot;nofollow&quot;&gt;9780471460688&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-11-16T16:24:50.713" CommunityOwnedDate="2010-11-16T16:24:50.713" CommentCount="1" />
  <row Id="4593" PostTypeId="2" ParentId="4542" CreationDate="2010-11-16T17:23:45.307" Score="1" ViewCount="" Body="&lt;p&gt;Hi. The following approaches might be of interest. In (1) &lt;a href=&quot;http://www.chemoton.org/ref31.html&quot; rel=&quot;nofollow&quot;&gt;http://www.chemoton.org/ref31.html&lt;/a&gt; a GA was used on image analysis but data was first treated. In order to treat the data, the GA was hybridized with K-means clustering, a well-know data classification procedure. Among classification the GA was also able to perform data feature extraction. Other approaches are also available from (2) ref 39 in chemoton org (3) ref 51 in chemoton org and (4) ref 42 in chemoton org among many other works. Hope it helps. Best, v. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(1) Vitorino Ramos, Fernando Muge, Map Segmentation by Colour Cube Genetic K-Mean Clustering, Proc. of  ECDL´2000 - 4th European Conference on Research and Advanced Technology for Digital Libraries, J. Borbinha and T. Baker (Eds.), ISBN 3-540-41023-6, Lecture Notes in Computer Science, Vol. 1923, pp. 319-323, Springer-Verlag -Heidelberg, Lisbon, Portugal, 18-20 Sep. 2000.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(2) Vitorino Ramos, Fernando Muge, Pedro Pina, Self-Organized Data and Image Retrieval as a Consequence of Inter-Dynamic Synergistic Relationships in Artificial Ant Colonies, in Javier Ruiz-del-Solar, Ajith Abraham and Mario Köppen (Eds.), Frontiers in Artificial Intelligence and Applications, Soft Computing Systems - Design, Management and Applications, 2nd Int. Conf. on  Hybrid Intelligent Systems, IOS Press, Vol. 87, ISBN 1 5860 32976, pp. 500-509, Santiago, Chile, Dec. 2002.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(3) Vitorino Ramos, Ajith Abraham, Evolving a Stigmergic Self-Organized Data-Mining, in ISDA-04, 4th Int. Conf. on Intelligent Systems, Design and Applications, Budapest, Hungary, ISBN 963-7154-30-2, pp. 725-730, August 26-28, 2004.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(4) Vitorino Ramos, Juan J. Merelo, Self-Organized Stigmergic Document Maps: Environment as a Mechanism for Context Learning, in AEB´2002 – 1st Spanish Conference on Evolutionary and Bio-Inspired Algorithms, E. Alba, F. Herrera, J.J. Merelo et al. (Eds.), pp. 284-293, Centro Univ. de Mérida, Mérida, Spain, 6-8 Feb. 2002. &lt;/p&gt;&#xA;" LastActivityDate="2010-11-16T17:23:45.307" />
  <row Id="4595" PostTypeId="1" CreationDate="2010-11-16T17:25:53.533" Score="3" ViewCount="234" Body="&lt;p&gt;Genetic algorithms are avoided in econometry literature as often as possible, but still sometimes they are inevitable. The question is: &lt;strong&gt;Which well known models are the most difficult to estimate using conventional algorithms?&lt;/strong&gt; (By Conventional algorithms I mean Gauss-Newton method, Levenberg–Marquardt algorithm, and so on)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Motivation:&lt;/strong&gt; I want to test some heuristic methods, and I need some benchmark to be sure that this particular model is really hard to estimate.&lt;/p&gt;&#xA;" OwnerUserId="1643" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-11-16T22:59:21.257" LastActivityDate="2011-02-15T00:22:52.000" Title="Nonlinear models which are hard to estimate" Tags="&lt;estimation&gt;&lt;genetic-algorithms&gt;&lt;nonlinear-regression&gt;&lt;optimization&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="1" />
  <row Id="4596" PostTypeId="2" ParentId="4580" CreationDate="2010-11-16T19:10:24.867" Score="4" ViewCount="" Body="&lt;p&gt;I am a biologist who models the effects of inter-annual climatic variation on population dynamics of several migratory species. My datasets are very large (spatially intensive data) so I run my R code using &lt;code&gt;multicore&lt;/code&gt; on Amazon EC2 servers. If my task is particularly resource intensive, I will choose a High Memory Quadruple Extra Large instance which comes with 26 CPU units, 8 cores, and 68G of RAM. In this case I usually run 4-6 scripts simultaneously, each of which is working through a fairly large data set. For smaller tasks, I choose servers with 4-6 cores and about 20 gigs of RAM.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I launch these instances (usually spot instances because they are cheaper but can terminate anytime the current rate exceeds what I have chosen to pay), run the script for several hours, and then terminate the instance once my script has finished. As for the machine image (Amazon Machine Image), I took someone elses Ubuntu install, updated R, installed my packages, and saved that as my private AMI on my S3 storage space.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My personal machine is a dualcore macbook pro and it has a hard time forking multicore calls. Feel free to email if you have other questions.&lt;/p&gt;&#xA;" OwnerUserId="1451" LastActivityDate="2010-11-16T19:10:24.867" CommunityOwnedDate="2010-11-16T19:10:24.867" CommentCount="2" />
  <row Id="4597" PostTypeId="1" CreationDate="2010-11-16T20:41:34.210" Score="3" ViewCount="183" Body="&lt;p&gt;I'm from the land of Stack Overflow, and I think this is the best place to ask this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm working on a piece of software which needs to implement the wiggliness of a set of data. Here's a sample of the input I would receive, merged with the lightness plot of each vertical pixel strip (I can't post images?):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://i.imgur.com/coUW6.jpg&quot; rel=&quot;nofollow&quot;&gt;http://i.imgur.com/coUW6.jpg&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is easy to see that the left margin is &lt;em&gt;really&lt;/em&gt; wiggly (i.e. has a ton of minima/maxima), and I want to generate a set of critical points of the image. I've applied a Gaussian smoothing function to the data ~ 10 times, but it seems to be pretty wiggly to begin with.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've resorted to computing the number of minima/maxima of the function within an interval (using the derivative and its proximity to zero) and dividing that by the length of the interval, but that isn't easy on the computer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How would I go about finding the critical points of this set of data (I use Python, and I couldn't find many low-pass filters for the data)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="2002" LastActivityDate="2010-11-22T22:26:51.780" Title="Determine &quot;wiggliness&quot; of set of data" Tags="&lt;data-visualization&gt;&lt;smoothing&gt;" AnswerCount="3" CommentCount="3" />
  <row Id="4598" PostTypeId="2" ParentId="4597" CreationDate="2010-11-16T21:05:45.083" Score="1" ViewCount="" Body="&lt;p&gt;What about Moving Averages?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: For calculating moving standard deviations, this is a quick and dirty way to do it in R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;n.x &amp;lt;- 1000&#xA;x &amp;lt;- cumsum(rnorm(n.x))&#xA;plot(x,type=&quot;l&quot;)&#xA;win &amp;lt;- 20&#xA;roll.sd &amp;lt;- as.vector(rep(NA,n.x))&#xA;for(i in 1:(n.x-win)){roll.sd[i] &amp;lt;- sd(x[i:(i+win)])}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I think quantmod has a build-in function for it. You could build, in a similar way, a moving average for first differences of the time series.&lt;/p&gt;&#xA;" OwnerUserId="1766" LastEditorUserId="1766" LastEditorDisplayName="" LastEditDate="2010-11-17T12:15:25.700" LastActivityDate="2010-11-17T12:15:25.700" CommentCount="6" />
  <row Id="4599" PostTypeId="2" ParentId="3194" CreationDate="2010-11-16T21:24:33.410" Score="3" ViewCount="" Body="&lt;p&gt;Nobody has suggested a Bayesian approach yet? I know the question has been answered already, but what the heck.  Below is for only a 3-sided die, but I'm guessing it's obvious how to fix it for $n=37$ sides.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, in line with what @Glen_b said, a bayesian is not actually interested whether or not the die is &lt;em&gt;exactly&lt;/em&gt; fair - it isn't. What (s)he cares about is whether it's close &lt;em&gt;enough&lt;/em&gt;, whatever &quot;enough&quot; means in the context, say, within 5% of fair for each side.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If $p_1$, $p_2$, and $p_3$ represent the probabilites of rolling 1, 2, and 3, respectively, then we represent our prior knowledge about $p=(p_1,p_2,p_3)$ with a prior distribution, and to make the math easy we could choose a &lt;a href=&quot;http://en.wikipedia.org/wiki/Dirichlet_distribution&quot; rel=&quot;nofollow&quot;&gt;Dirichlet distribution&lt;/a&gt;.  Note that $p_1+p_2+p_3=1$.  For a non-informative prior we might pick prior parameters, say, $\alpha_0=(1,1,1)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If $X=(X_1,X_2,X_3)$ represents the observed counts of 1,2,3 then of course $X$ has a &lt;a href=&quot;http://en.wikipedia.org/wiki/Multinomial_distribution&quot; rel=&quot;nofollow&quot;&gt;multinomial distribution&lt;/a&gt; with parameters $p=(p_1,p_2,p_3)$, and the theory says that the posterior is also a Dirichlet distribution with parameters $\alpha=(x_1+1,x_2+1,x_3+1)$.  (Dirichlet is called a &lt;a href=&quot;http://en.wikipedia.org/wiki/Conjugate_prior&quot; rel=&quot;nofollow&quot;&gt;conjugate prior&lt;/a&gt;, here.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We observe data, find the posterior with Bayes' rule, then ALL inference is based on the posterior.  Want an estimate for $p$?  Find the mean of the posterior.  Want confidence intervals (no, rather &lt;a href=&quot;http://en.wikipedia.org/wiki/Credible_interval&quot; rel=&quot;nofollow&quot;&gt;credible intervals&lt;/a&gt;)?  Calculate some areas under the posterior.  For complicated problems in the real world we usually simulate from the posterior and get simulated estimates for all of the above. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyway, here's how (with R):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, get some data.  We roll the die 500 times.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(1)&#xA;y &amp;lt;- rmultinom(1, size = 500, prob = c(1,1,1))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(we're starting with a fair die; in practice these data would be observed.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Next, we simulate 5000 observations of $p$ from the posterior and take a look at the results.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(MCMCpack)&#xA;A &amp;lt;- MCmultinomdirichlet(y, alpha0 = c(1,1,1), mc = 5000)&#xA;plot(A)&#xA;summary(A)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Finally, let's estimate our posterior probability (after observing the data) that the die is within 0.05 of fair in each coordinate.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;B &amp;lt;- as.matrix(A)&#xA;f &amp;lt;- function(x) all((x &amp;gt; 0.28)*(x &amp;lt; 0.38))&#xA;mean(apply(B, MARGIN = 1, FUN = f))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The result is about 0.9486 on my machine.  (Not a surprise, really.  We started with a fair die after all.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Quick remark: it probably isn't reasonable for us to have used a non-informative prior in this example.  Since there's even a question presumably the die appears approximately balanced in the first place, so it may be better to pick a prior that is concentrated closer to 1/3 in all coordinates.  Above this would simply have made our estimated posterior probability of &quot;close to fair&quot; even higher.&lt;/p&gt;&#xA;" OwnerUserId="1108" LastActivityDate="2010-11-16T21:24:33.410" />
  <row Id="4600" PostTypeId="1" CreationDate="2010-11-16T22:11:41.093" Score="1" ViewCount="208" Body="&lt;p&gt;I have run a factorial type test in a processing plant and have run a forward and backward step regression in R. I'm not even an amateur statistician so I want to know if I can look at the regression results and the anova created from the regression and know what percent of the measured variation of the dependent variable was caused by the purposeful manipulation of the independent variables.&lt;/p&gt;&#xA;" LastActivityDate="2011-05-07T02:43:55.463" Title="How to explain the variation of a dependent variable by the variation of independent variables" Tags="&lt;regression&gt;&lt;anova&gt;&lt;interpretation&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="4601" PostTypeId="1" CreationDate="2010-11-16T22:20:03.617" Score="0" ViewCount="176" Body="&lt;p&gt;Hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm don't have a lot of training in statistics so what I ask might be trivial or might be as well a complicated problem. I'm mostly looking for ideas and links so that I can read and learn about my problem. Here is what I want to accomplish:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to compare two measurement devices each of which provide a different set of outcome measures. I want to see which device is more sensitive. So what I do is to conduct an experiment measuring two randomly select groups from two populations. What I want to know is which device show the most difference between the two groups. This is of course not trivial since 1) devices provide different outcomes and 2) each devices has multiple outcome. Since I'm comparing two devices, I guess what I'm doing is basically comparing two multivariate effect sizes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was thinking that one can measure the Mahalanobis distance between the two groups using each device (lets call them D1 and D2). So basically all I want to do is to compare D1 and D2. The problem is that the two device provide different outcomes so D1 and D2 are not directly comparable (are they?). What I need is probably a form of normalized distance as well its confidence intervals so that I can compare the distances.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Am I on the right track? How do you normalize Mahalanobis distance and how do you calculate its confidence interval? Are there other methods to compare two devices as outlined above? And finally, what if the design of experiment force two correlated groups, like measuring before/after intervention? Now what?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks,&#xA;AlefSin &lt;/p&gt;&#xA;" OwnerUserId="2020" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-11-16T22:37:22.383" LastActivityDate="2010-11-16T22:37:22.383" Title="How to compare two multivariate effect sizes?" Tags="&lt;multivariate-analysis&gt;&lt;effect-size&gt;" />
  <row Id="4602" PostTypeId="2" ParentId="4595" CreationDate="2010-11-16T22:29:48.383" Score="2" ViewCount="" Body="&lt;p&gt;The &lt;strong&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Travelling_salesman_problem&quot; rel=&quot;nofollow&quot;&gt;travelling salesman problem&lt;/a&gt;&lt;/strong&gt; is surely an archetypal hard optimization problem. To quote Wikipedia, &quot;it is used as a benchmark for many optimization methods&quot;.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-11-16T22:29:48.383" CommentCount="3" />
  <row Id="4603" PostTypeId="1" AcceptedAnswerId="4614" CreationDate="2010-11-16T22:31:23.040" Score="2" ViewCount="198" Body="&lt;p&gt;Are there any open-source Java implementations of lasso or least angles regression?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Pure Java code would be best, but clean implementations in other languages would also be of interest. I am already aware of the existence of a variety of R packages than can do lasso/LAR fits.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="439" LastEditorUserId="439" LastEditorDisplayName="" LastEditDate="2010-11-16T22:53:34.213" LastActivityDate="2010-11-18T17:39:55.700" Title="Java implementations of the lasso" Tags="&lt;regression&gt;&lt;lasso&gt;&lt;java&gt;" AnswerCount="2" />
  <row Id="4604" PostTypeId="1" AcceptedAnswerId="4613" CreationDate="2010-11-16T22:57:41.137" Score="3" ViewCount="163" Body="&lt;p&gt;I'm doing some some analysis of an arbitrary string of text, modelling it as a Markov chain where the state is simply the value of the previous character. Call the current character &lt;code&gt;c&lt;/code&gt; and the previous character &lt;code&gt;p&lt;/code&gt;; then it is trivial to calculate &lt;code&gt;P(c | p)&lt;/code&gt; for the given sample text. However, if there are not many cases of a certain (p, c) pair, or in general not many samples used. I know qualitatively that the fewer the samples, the higher the uncertainty/error in this conditional probability is. However, how can I quantify this?&lt;/p&gt;&#xA;" OwnerUserId="1810" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-11-17T07:59:02.207" LastActivityDate="2011-03-22T11:32:23.003" Title="Uncertainty of conditional probability evaluated from sample" Tags="&lt;markov-process&gt;&lt;conditional-probability&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="4605" PostTypeId="2" ParentId="4580" CreationDate="2010-11-17T01:34:10.317" Score="1" ViewCount="" Body="&lt;p&gt;I use snow and snowfall for course parallelization on HPC clusters and CUDA for fine data parallel processing.  I'm in Epidemiology doing disease transmission modeling.  So I use both.&lt;/p&gt;&#xA;" OwnerUserId="1364" LastActivityDate="2010-11-17T01:34:10.317" CommunityOwnedDate="2010-11-17T01:34:10.317" CommentCount="3" />
  <row Id="4606" PostTypeId="1" CreationDate="2010-11-17T02:04:27.900" Score="2" ViewCount="102" Body="&lt;p&gt;I am thinking of using &lt;a href=&quot;http://www.mathworks.com/matlabcentral/fileexchange/14034-kernel-density-estimator&quot; rel=&quot;nofollow&quot;&gt;this code&lt;/a&gt; in a Monte Carlo routine to generate Kernel Density Estimates for subsequent use in a Naive Bayes Classifier &lt;a href=&quot;http://stats.stackexchange.com/questions/4298/use-of-kernel-density-estimate-in-naive-bayes-classifier&quot;&gt;(see this earlier post)&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The author of the code states on the above linked page that it will &quot;recognise that the data you have provided is perfectly discrete and since discrete data does not need smoothing, the selected bandwidth should be zero&quot; in the case of ties. However, I do not want this but can envisage that, given the parameters I intend to use, multiple repetitions in a large Monte Carlo routine will very likely produce ties in the generated data. To prevent this I am thinking of limiting the number of iterations in the MC routine and employing checks to ensure non-discrete generated data for use in the Kernel Density Estimate code, and then repeating. Perhaps this is more clearly explained thus: instead of a 100,000 iteration MC, which is highly likely to produce unwanted ties, and then one KDE, do a 100 iteration MC and one KDE and repeat this 100 and 1 routine 1000 times. This will result in 1000 separate but similar continuous KDEs which can then be averaged to produce a single &quot;unified&quot; continuous KDE. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this a valid approach? Are averaged KDEs a statistically sound methodology? The discussion section &lt;a href=&quot;http://en.wikipedia.org/wiki/Naive_Bayes_classifier&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; would seem to imply that even if there are deficiencies in this approach, they wouldn't necessarily be too troublesome.&lt;/p&gt;&#xA;" OwnerUserId="226" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-11-17T08:00:05.233" LastActivityDate="2010-11-18T16:35:12.947" Title="Averaged continuous Kernel Density Estimates in lieu of a discrete Kernel Density Estimate in Monte Carlo Proceedure" Tags="&lt;kde&gt;&lt;discrete-data&gt;&lt;continuous-data&gt;&lt;monte-carlo&gt;" AnswerCount="0" CommentCount="1" />
  <row Id="4607" PostTypeId="2" ParentId="2171" CreationDate="2010-11-17T02:44:28.000" Score="1" ViewCount="" Body="&lt;p&gt;Another good alternative is the protovis library &lt;a href=&quot;http://vis.stanford.edu/protovis/%22Protovis%22&quot; rel=&quot;nofollow&quot;&gt;http://vis.stanford.edu/protovis/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Its a very well crafted JavaScript library that can create some beautiful visualizations if you have the time and ability to write the modest amount of JavaScript code needed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also highly recommend Tableau &lt;a href=&quot;http://www.tableausoftware.com&quot; rel=&quot;nofollow&quot;&gt;http://www.tableausoftware.com&lt;/a&gt;. Its great for rapidly exploring data sets and creating many different visualizations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Both products have roots at the Stanford Visualization Group.&lt;/p&gt;&#xA;" OwnerUserId="1246" LastActivityDate="2010-11-17T02:44:28.000" CommunityOwnedDate="2010-11-17T02:44:28.000" />
  <row Id="4608" PostTypeId="1" AcceptedAnswerId="4621" CreationDate="2010-11-17T03:15:58.023" Score="4" ViewCount="229" Body="&lt;p&gt;I'm trying to implement basic gradient descent and I'm testing it with a hinge loss function ie $\max(0,1-y\ \mathbf{w.x})$. However, I'm confused about the gradient of the hinge loss. I'm under the impression that it is &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;\frac{\partial }{\partial w} =&#xD;&#xA;\begin{cases}      &#xD;&#xA;-y\ \mathbf{x} &amp;amp;\text{if } y\ \mathbf{x\cdot w} &amp;lt; 1 \\&#xD;&#xA;0&amp;amp;\text{if } y\ \mathbf{x\cdot w} \geq 1      &#xD;&#xA;\end{cases}&#xD;&#xA;$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But doesn't this return a matrix the same size as $\mathbf{x}$? I thought we were looking to return a vector of length $\mathbf{w}$? Clearly, I've got something confused somewhere. Can someone point in the right direction here?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've included some basic code in case my description of the task was not clear &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#Run standard gradient descent&#xA;gradient_descent&amp;lt;-function(fw, dfw, n, lr=0.01)&#xA;{&#xA;    #Date to be used&#xA;    x&amp;lt;-t(matrix(c(1,3,6,1,4,2,1,5,4,1,6,1), nrow=3))&#xA;    y&amp;lt;-c(1,1,-1,-1)&#xA;    w&amp;lt;-matrix(0, nrow=ncol(x))&#xA;&#xA;    print(sprintf(&quot;loss: %f,x.w: %s&quot;,sum(fw(w,x,y)),paste(x%*%w, collapse=',')))&#xA;    #update the weights 'n' times&#xA;    for (i in 1:n)&#xA;    {&#xA;      w&amp;lt;-w-lr*dfw(w,x,y)&#xA;      print(sprintf(&quot;loss: %f,x.w: %s&quot;,sum(fw(w,x,y)),paste(x%*%w,collapse=',')))&#xA;    }&#xA;}&#xA;#Hinge loss&#xA;hinge&amp;lt;-function(w,x,y) max(1-y%*%x%*%w, 0)&#xA;d_hinge&amp;lt;-function(w,x,y){ dw&amp;lt;-t(-y%*%x); dw[y%*%x%*%w&amp;gt;=1]&amp;lt;-0; dw}&#xA;gradient_descent(hinge, d_hinge, 100, lr=0.01)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Update:&#xA;While the answer below helped my understanding of the problem, the output of this algorithm is still incorrect for the given data. The loss function reduces by 0.25 each time but converge too fast and the resulting weights do not result in a good classification. Currently the output looks like&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#y=1,1,-1,-1&#xA;&quot;loss: 1.000000, x.w: 0,0,0,0&quot;&#xA;&quot;loss: 0.750000, x.w: 0.06,-0.1,-0.08,-0.21&quot;&#xA;&quot;loss: 0.500000, x.w: 0.12,-0.2,-0.16,-0.42&quot;&#xA;&quot;loss: 0.250000, x.w: 0.18,-0.3,-0.24,-0.63&quot;&#xA;&quot;loss: 0.000000, x.w: 0.24,-0.4,-0.32,-0.84&quot;&#xA;&quot;loss: 0.000000, x.w: 0.24,-0.4,-0.32,-0.84&quot;&#xA;&quot;loss: 0.000000, x.w: 0.24,-0.4,-0.32,-0.84&quot;&#xA;...  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2023" LastEditorUserId="2023" LastEditorDisplayName="" LastEditDate="2010-11-17T22:04:14.297" LastActivityDate="2010-11-17T22:04:14.297" Title="Gradient of Hinge loss" Tags="&lt;loss-functions&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="4609" PostTypeId="1" AcceptedAnswerId="9919" CreationDate="2010-11-17T03:19:10.763" Score="1" ViewCount="442" Body="&lt;p&gt;Hi,&#xA;from what I can tell, PASW v.18 (the new version of SPSS) only gives you the p value of nonparametric tests. I am calculating Kruskal Wallis and Mann Whitney tests, and need to report the test statistic, not just p. Can someone please help?&#xA;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="2025" LastActivityDate="2011-04-24T04:00:58.850" Title="How do I get non-parametric test values (not just p) in PASW 18?" Tags="&lt;nonparametric&gt;&lt;spss&gt;" AnswerCount="2" />
  <row Id="4610" PostTypeId="1" CreationDate="2010-11-17T03:33:13.613" Score="3" ViewCount="274" Body="&lt;p&gt;Hi all,&lt;br&gt;&#xA;    Statistics were never my strong point and it's my first question, so please be gentle :) I'm doing some research using Computational Fluid Dynamics, CFD, to model the flow of an oil aerosol through a fibrous filter. The aerosol has a droplet size distribution that is log-normal. The existing code only allows the specification of a single size, I have to add the log-normal support.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I need to be able to do is if someone describes an aerosol flow as having, say, 10,000 droplets, an average droplet size of 425nm and a standard dev of 15, be able to, within my code, calculate that there are:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;aaa particles of size 100nm&lt;br&gt;&#xA;bbb particles of size 200nm&lt;br&gt;&#xA;ccc particles of size 300nm&lt;br&gt;&#xA;ddd particles of size 400nm&lt;br&gt;&#xA;...&lt;br&gt;&#xA;...&lt;br&gt;&#xA;...&lt;br&gt;&#xA;iii particles of size 900nm&lt;br&gt;&#xA;jjj particles of size 1000nm  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;when we are looking at particles in the range 100nm - 1000nm with 100nm step sizes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In turn I need to push this information, the particle counts, along with a lot of other stuff into the CFD solver and see what happens to the droplets. The big questions is, how do I calculate the number of particles for each step size. As I said, my stat's is quite limited, so I don't even know if saying &lt;em&gt;the average is 425nm...&lt;/em&gt; is the right way to frame the input data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My Masters supervisor has told me one thing, using the pdf, but my reading of the definition of log-normal, yes on Wikipedia, leads me to think he is wrong. Any thoughts on how I can work this out are greatly appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Andrew      &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Updated:&lt;br&gt;&#xA;Thanks for the answers, but this is where my minimal stats shines through. OK so we rephrase that we have a GM of, say 328nm, and a GSD of 14.8nm. I then look at the definition of the cumulative dist func. and see:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;cdf = 1/2 + 1/2 * erf[(ln(x) - mu) / sqrt(2 * sigma^2)]   -  From wikipedia&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Am I correct in that mu = GM = 328nm and sigma = GSD = 14.8nm? Then in turn, what do I use for my x value? If I need to get values at 100nm, 200nm, 300nm,....., 900nm, 1000nm, do I use these values, lets just drop the &quot;nm&quot; part for now, as X, one at a time, or do I use a ln/log of them or some other magic number based upon the value in question? I don't need to actually code the erf() as c++ has a function call for it, it's just how I calculate the actual value that I pass into the c++ function that is causing me angst.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Am I correct in that when I can calculate the input to the erf function that I calculate the cfd values which will be in the range 0 -&gt; 1 which I then multiply by my sample size to get the number of particles below the point in question. I then just subtract successive values to get the particles in a range - correct/sort of correct/wrong???  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once again, thanks for the help,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Andrew&lt;/p&gt;&#xA;" LastActivityDate="2010-11-18T01:22:50.987" Title="Log normal distributions - particle sizes in an aerosol" Tags="&lt;distributions&gt;&lt;lognormal&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="4611" PostTypeId="2" ParentId="4368" CreationDate="2010-11-17T04:00:28.513" Score="1" ViewCount="" Body="&lt;p&gt;Note that&#xA;$\text{Var}(\epsilon | x_i) = E[\epsilon^2 | x_i] - \left(E[\epsilon | x_i]\right)^2.$&#xD;&#xA;Recall a typical Gauss-Markov assumption that $E[\epsilon | x_i]=0$. Hence,&#xA;$\text{Var}(\epsilon | x_i) = E[\epsilon^2 | x_i].$&#xD;&#xA;Since we don't observe $\epsilon^2$, we have to use some estimate. The best estimate that we have is $\hat{e}^2$, where $\hat{e}$ is our residual. The variance is the expectation of the squared error and our best estimate of this is the squared residuals; hence, they are the outcome of our auxiliary regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, we suppose that $\sigma^2_i$ varies across units $i$. One way to check this is to determine whether the squared residuals are correlated with other stuff that varies across units, namely, the variables in the regression that we care about, plus their squares, and the interactions, as suggested by White. If the squared residuals can't be predicted by stuff that we observe that varies across units, then we cannot reject the null hypothesis that the expected squared residuals and don't vary across units, which suggests that the variance doesn't vary across units and the errors are homoskedastic.&lt;/p&gt;&#xA;" OwnerUserId="401" LastActivityDate="2010-11-17T04:00:28.513" />
  <row Id="4612" PostTypeId="1" AcceptedAnswerId="4615" CreationDate="2010-11-17T07:28:49.070" Score="1" ViewCount="362" Body="&lt;p&gt;Which good econometrics textbooks would you recommend?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;edit: There are quite a few books out there, with varying levels of mathematical sophistication. It would be good to get some idea of how technical the book you're recommending is.&lt;/p&gt;&#xA;" OwnerUserId="439" LastEditorUserId="439" LastEditorDisplayName="" LastEditDate="2010-11-17T14:04:47.070" LastActivityDate="2010-11-18T13:14:58.757" CommunityOwnedDate="2010-11-17T11:14:47.670" Title="Good econometrics textbooks?" Tags="&lt;books&gt;&lt;econometrics&gt;&lt;big-list&gt;" AnswerCount="9" CommentCount="0" FavoriteCount="2" />
  <row Id="4613" PostTypeId="2" ParentId="4604" CreationDate="2010-11-17T07:30:42.387" Score="3" ViewCount="" Body="&lt;p&gt;You can  use the &lt;a href=&quot;http://en.wikipedia.org/wiki/Hoeffding%27s_inequality&quot; rel=&quot;nofollow&quot;&gt;Hoeffding inequality&lt;/a&gt;: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ P(|\hat{p}_n-p|\geq t)\leq 2e^{-2nt^2}$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;($\hat{p}_n$ is your estimated probability).&#xA;For small $n$, Markov ineqality may be more efficient:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(|\hat{p}_n-p|\geq t)\leq \frac{p(1-p)}{nt^2} $$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, with the second inequality, if $n=50$ the probability that the difference between your estimated probability and the truth is greater than $0.25$ is lower than $\frac{1}{4*50*0.25^2}=0.08$ ($p(1-p)&amp;lt;1/4$ but it can be much smaller...). Is 8% chance to have difference larger than $0.25$ small enough for you ? &lt;/p&gt;&#xA;" OwnerUserId="223" LastEditorUserId="223" LastEditorDisplayName="" LastEditDate="2010-11-23T14:37:19.163" LastActivityDate="2010-11-23T14:37:19.163" CommentCount="4" />
  <row Id="4614" PostTypeId="2" ParentId="4603" CreationDate="2010-11-17T07:34:31.840" Score="3" ViewCount="" Body="&lt;p&gt;About clean implementation in Python, there is the &lt;a href=&quot;http://scikit-learn.sourceforge.net/&quot; rel=&quot;nofollow&quot;&gt;scikit.learn&lt;/a&gt; toolkit. The &lt;a href=&quot;http://scikit-learn.sourceforge.net/modules/glm.html&quot; rel=&quot;nofollow&quot;&gt;L1/L2 regularization scheme&lt;/a&gt; (incl. &lt;em&gt;elasticnet&lt;/em&gt;) works great with GLM (LARS and coordinate descent algorithms available). Don't know about Java implementation.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-11-17T07:34:31.840" CommentCount="1" />
  <row Id="4615" PostTypeId="2" ParentId="4612" CreationDate="2010-11-17T07:40:10.787" Score="3" ViewCount="" Body="&lt;p&gt;Definitively &lt;a href=&quot;http://pages.stern.nyu.edu/~wgreene/Text/econometricanalysis.htm&quot; rel=&quot;nofollow&quot;&gt;Econometric Analysis&lt;/a&gt;, by Greene. I'm not an econometrician, but I found this book very useful and well written.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-11-17T07:40:10.787" CommunityOwnedDate="2010-11-17T09:14:40.483" CommentCount="1" />
  <row Id="4616" PostTypeId="2" ParentId="4610" CreationDate="2010-11-17T07:44:20.333" Score="3" ViewCount="" Body="&lt;p&gt;Saying 'the average is 425nm' is probably &lt;em&gt;not&lt;/em&gt; the best way to frame the input. You're better parameterising a &lt;a href=&quot;http://en.wikipedia.org/wiki/Log-normal_distribution&quot; rel=&quot;nofollow&quot;&gt;log-normal distribution&lt;/a&gt; by its &lt;a href=&quot;http://en.wikipedia.org/wiki/Geometric_mean&quot; rel=&quot;nofollow&quot;&gt;geometric mean&lt;/a&gt; and &lt;a href=&quot;http://en.wikipedia.org/wiki/Geometric_standard_deviation&quot; rel=&quot;nofollow&quot;&gt;geometric standard deviation&lt;/a&gt;. Having done so, calculating the probability of an observation between two given values is just a matter of taking the difference between the two corresponding values of the cumulative distribution function.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-11-17T07:44:20.333" />
  <row Id="4617" PostTypeId="2" ParentId="4612" CreationDate="2010-11-17T07:56:14.970" Score="3" ViewCount="" Body="&lt;p&gt;Depends on what level you're after. At a postgraduate level, the one i've most often seen referenced and recommended, and have therefore looked at most myself, is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Wooldridge, Jeffrey M. &lt;em&gt;Econometric Analysis of Cross Section and Panel Data&lt;/em&gt;. MIT Press, 2001. ISBN &lt;a href=&quot;http://en.wikipedia.org/w/index.php?title=Special%3ABookSources&amp;amp;isbn=9780262232197&quot; rel=&quot;nofollow&quot;&gt;9780262232197&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most of what little I know about econometrics I learnt from this book. 776 pages without a single graph.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-11-17T07:56:14.970" CommunityOwnedDate="2010-11-17T14:15:50.470" CommentCount="1" />
  <row Id="4618" PostTypeId="2" ParentId="4612" CreationDate="2010-11-17T08:02:48.417" Score="1" ViewCount="" Body="&lt;p&gt;It depends on what you really want, (GMM, time series, panel...) but i can recommand those two book :&#xA; Fumio Hayashi's &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0691010188&quot; rel=&quot;nofollow&quot;&gt;Econometric&lt;/a&gt;&#xA;and Davidson and McKinnon Econometric Theory and methods&#xA;and For a course in econometric time series, Hamilton's Time Serie Analysis is great&lt;/p&gt;&#xA;" OwnerUserId="2028" LastActivityDate="2010-11-17T08:02:48.417" CommunityOwnedDate="2010-11-17T14:15:50.470" />
  <row Id="4619" PostTypeId="1" CreationDate="2010-11-17T08:08:19.927" Score="1" ViewCount="96" Body="&lt;p&gt;I have two surveys of two separate populations (I don't know that they are necessarily distinct, but they are from two different databases) that ask a similar set of questions.  Some questions are basic demographics (e.g. age, income), while other questions are a bit more detailed or about their opinions (e.g. brand preferences, spending habits).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I prove statistically that the two populations are &quot;the same,&quot; or at least comparable?  I know that I can do a t-test for individual questions, but is there a way to establish similarity on more than one dimension?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The goal is to combine the surveys from these two populations into one series of survey data.  For example, we may run survey A every six months, but we run survey B every month, except when we run survey A.  I would then like to combine the results from survey A and survey B to have a monthly series of survey data.&lt;/p&gt;&#xA;" OwnerUserId="1195" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-17T14:37:06.837" LastActivityDate="2010-11-18T13:06:58.253" Title="Establishing that the population sampled of two separate surveys is the same" Tags="&lt;psychometrics&gt;&lt;survey&gt;" AnswerCount="2" />
  <row Id="4620" PostTypeId="1" CreationDate="2010-11-17T08:36:48.267" Score="1" ViewCount="367" Body="&lt;p&gt;Say, for example, that I want to determine the market share or relative popularity of coffee houses within a certain population through a survey.  What is the best way to write a question that accurately measures this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some issues that I am thinking of:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't want to ask a single choice question (&quot;Which coffee house do you go to?&quot;), because the coffee houses are not mutually exclusive.  I may enjoy more than one equally often.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I ask a multiple choice question, then I can't really get a real &quot;market share,&quot; since the sum of the proportions of people who go to each coffee house sum to over 100%.  I can only say that &quot;x% of people in this population go to this coffee house.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possible to ask a series of questions (&quot;Which coffee house do you prefer the most?&quot; &quot;Which coffee house do you prefer the second most?&quot;, or &quot;Rank the following coffee houses&quot;)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can I ask a multiple choice question then rebase the proportion to the total number of responses?  For example, if I have 100 respondents, but they selected 200 coffee houses (because each respondent said they went to two coffee houses, maybe), can I calculate a frequency table based on 200, the number of selections, instead of the number of respondents?&lt;/p&gt;&#xA;" OwnerUserId="1195" LastActivityDate="2010-11-17T18:47:25.040" Title="Determining market share from multiple choice questions on a survey" Tags="&lt;survey&gt;" AnswerCount="4" />
  <row Id="4621" PostTypeId="2" ParentId="4608" CreationDate="2010-11-17T09:25:37.863" Score="4" ViewCount="" Body="&lt;p&gt;To get the gradient we differentiate the loss with respect to $i$th component of $w$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rewrite hinge loss in terms of $w$ as $f(g(w))$ where $f(z)=\max(0,1-y\ z)$ and $g(w)=\mathbf{x}\cdot \mathbf{w}$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using chain rule we get&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{\partial}{\partial w_i} f(g(w))=\frac{\partial f}{\partial z} \frac{\partial g}{\partial w_i} $$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First derivative term is evaluated at $g(w)=x\cdot w$ becoming $-y$ when $\mathbf{x}\cdot w&amp;lt;1$, and 0 when $x\cdot w&amp;gt;0$. Second derivative term becomes $x_i$. So in the end you get&#xA;$$&#xD;&#xA;\frac{\partial f(g(w))}{\partial w_i} =&#xD;&#xA;\begin{cases}      &#xD;&#xA;-y\ x_i &amp;amp;\text{if } y\ \mathbf{x}\cdot \mathbf{w} &amp;lt; 1 \\&#xD;&#xA;0&amp;amp;\text{if } y\ \mathbf{x}\cdot \mathbf{w} &amp;gt; 1      &#xD;&#xA;\end{cases}&#xD;&#xA;$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since $i$ ranges over the components of $x$, you can view the above as a vector quantity, and write $\frac{\partial}{\partial w}$ as shorthand for $(\frac{\partial}{\partial w_1},\frac{\partial}{\partial w_2},\ldots)$&lt;/p&gt;&#xA;" OwnerUserId="511" LastActivityDate="2010-11-17T09:25:37.863" CommentCount="2" />
  <row Id="4622" PostTypeId="2" ParentId="4612" CreationDate="2010-11-17T09:41:47.610" Score="1" ViewCount="" Body="&lt;p&gt;I would definitely recommend M. Verbeek's &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0471899828&quot; rel=&quot;nofollow&quot;&gt;A Guide to Modern Econometrics&lt;/a&gt;.&#xA;Woolwridge is too worldly (and this long-windedness loses the reader's focus too early in the chapters). Greene (i'm referring to the 5th edition) often gets lost in minutiae: i.e. strives to catalog formulae that are orthogonal to the main subject of the chapter (good for references, but again, not ideal for learning).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;i've not read the Hayashi (thought i suspect it's a bit outdated now). Hamilton, is really focused on...TSA so it's a bit off the mark for general econometrics. &lt;/p&gt;&#xA;" OwnerUserId="603" LastActivityDate="2010-11-17T09:41:47.610" CommunityOwnedDate="2010-11-17T14:15:50.470" CommentCount="1" />
  <row Id="4623" PostTypeId="1" CreationDate="2010-11-17T10:16:58.917" Score="4" ViewCount="171" Body="&lt;p&gt;I have a 3-dimensional sample $(X_k,Y_k,Z_k), k=1, \ldots, N$ which I suspect to be uniform on some parallelepiped in $R^3$ (i.e. a set of the form [a;b]X[c;d]X[e;f], where numbers a,b,c,d,e,f are unknown).&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;How should I estimate numbers a, b, c, d, e, f? Obviously I can try MLE, but then my estimates are biased. Does unbiased estimates exist?&lt;/li&gt;&#xA;&lt;li&gt;How can I check that my sample is indeed uniform?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="2029" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-11-17T12:24:02.010" LastActivityDate="2010-11-17T13:14:30.247" Title="How to check that a sample suits multi-dimensional uniform distribution?" Tags="&lt;distributions&gt;&lt;hypothesis-testing&gt;&lt;estimation&gt;" AnswerCount="1" />
  <row Id="4624" PostTypeId="1" AcceptedAnswerId="4626" CreationDate="2010-11-17T10:33:27.813" Score="4" ViewCount="122" Body="&lt;p&gt;I am fitting an L1-regularized linear regression to a very large dataset (with n&gt;&gt;p.) The variables are known in advance, but the observations arrive in small chunks. I would like to maintain the lasso fit after each chunk.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can obviously re-fit the entire model after seeing each new set of observations. This, however, would be pretty inefficient given that there is a lot of data. The amount of &lt;i&gt;new&lt;/i&gt; data that arrives at each step is very small, and the fit is unlikely to change much between steps.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there anything I can do to reduce the overall computational burden?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was looking at the LARS algorithm of Efron et al., but would be happy to consider any other fitting method if it can be made to &quot;warm-start&quot; in the way described above.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Notes:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;I am mainly looking for an algorithm, but pointers to existing software packages that can do this may also prove insightful.&lt;/li&gt;&#xA;&lt;li&gt;In addition to the current lasso trajectories, the algorithm is of course welcome to keep other state.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Bradley Efron, Trevor Hastie, Iain&#xA;  Johnstone and Robert Tibshirani,&#xA;  &lt;a href=&quot;http://projecteuclid.org/euclid.aos/1083178935&quot; rel=&quot;nofollow&quot;&gt;Least Angle Regression&lt;/a&gt;, &lt;em&gt;Annals&#xA;  of Statistics&lt;/em&gt; (with discussion)&#xA;  (2004) 32(2), 407--499.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="439" LastEditorUserId="439" LastEditorDisplayName="" LastEditDate="2010-11-17T15:53:54.630" LastActivityDate="2010-11-17T15:53:54.630" Title="Updating the lasso fit with new observations" Tags="&lt;regression&gt;&lt;lasso&gt;" AnswerCount="1" />
  <row Id="4625" PostTypeId="2" ParentId="4609" CreationDate="2010-11-17T10:47:04.823" Score="4" ViewCount="" Body="&lt;p&gt;PASW 18 does give you the test statistic in addition to the p value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if you have selected Mann Whitney test, the output from SPSS will include a Test Statistics box that shows the mann Whitney U statistic &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The same thing applies for the Kruskal Wallis test, although note that SPSS labels the statistic Chi-Square, rather than H.&lt;/p&gt;&#xA;" OwnerUserId="2030" LastActivityDate="2010-11-17T10:47:04.823" CommentCount="1" />
  <row Id="4626" PostTypeId="2" ParentId="4624" CreationDate="2010-11-17T10:59:33.970" Score="2" ViewCount="" Body="&lt;p&gt;The lasso is fitted through LARS (an iterative process, that starts at some initial estimate $\beta^0$). By default $\beta^0=0_p$ but you can change this in most implementation (and replace it by the optimal $\beta^*_{old}$ you already have). The closest $\beta^*_{old}$ is to $\beta_{new}^*$, the smaller the number of LARS iteration you will have to step to get to $\beta_{new}^*$.&lt;/p&gt;&#xA;" OwnerUserId="603" LastActivityDate="2010-11-17T10:59:33.970" CommentCount="4" />
  <row Id="4628" PostTypeId="2" ParentId="4612" CreationDate="2010-11-17T11:51:36.763" Score="2" ViewCount="" Body="&lt;p&gt;I really like Kennedy's A Guide to Econometrics, which is unusual in its setup, since every topic is discussed on three different levels, first in a non-technical way, then going into details of application and finally going into theoretical details, although the theoretical parts are a bit superficial.&lt;/p&gt;&#xA;" OwnerUserId="1766" LastActivityDate="2010-11-17T11:51:36.763" CommunityOwnedDate="2010-11-17T11:51:36.763" />
  <row Id="4629" PostTypeId="2" ParentId="4551" CreationDate="2010-11-17T12:03:29.180" Score="1" ViewCount="" Body="&lt;p&gt;In psychology, the cardinal sin (for me) is the use of principal components analysis to examine the hypothesised latent structure underlying a psychometric test. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Not testing for normality before using tests which assume this. &lt;/p&gt;&#xA;" OwnerUserId="656" LastActivityDate="2010-11-17T12:03:29.180" CommunityOwnedDate="2010-11-17T12:03:29.180" CommentCount="1" />
  <row Id="4630" PostTypeId="2" ParentId="4623" CreationDate="2010-11-17T13:14:30.247" Score="2" ViewCount="" Body="&lt;ol&gt;&#xA;&lt;li&gt;For the 1D continuous uniform distribution U(a,b) the uniformly minimum variance unbiased (UMVU) estimates of a and b can be obtained in closed form by a &lt;a href=&quot;http://en.wikipedia.org/wiki/Maximum_spacing_estimation#Example_2&quot; rel=&quot;nofollow&quot;&gt;straightforward example of maximum spacing estimation.&lt;/a&gt; Can't see any reason that applying this separately for each dimension wouldn't give you UMVU estimates of all parameters of your multivariate uniform distribution&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-11-17T13:14:30.247" CommentCount="5" />
  <row Id="4631" PostTypeId="2" ParentId="4620" CreationDate="2010-11-17T13:42:36.827" Score="1" ViewCount="" Body="&lt;p&gt;Frequency show the values that variables take in a sample.&#xA;In other words, shows the amount of individuals said that prefer coffeehouse A. This amount would be the frequency of coffeehouse A.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;axis:&#xA;X:coffeehouses&#xA;Y:amount of individuals&lt;/p&gt;&#xA;" OwnerUserId="1746" LastActivityDate="2010-11-17T13:42:36.827" CommentCount="2" />
  <row Id="4632" PostTypeId="2" ParentId="4612" CreationDate="2010-11-17T13:49:06.863" Score="1" ViewCount="" Body="&lt;p&gt;One at a somewhat lower level of mathematical sophistication than Wooldridge (less dense, more pictures), but a bit more up to date on some of the fast-moving areas:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Murray, Michael P. &lt;em&gt;Econometrics: A Modern Introduction.&lt;/em&gt; Addison Wesley, 2006. 976 pp. ISBN &lt;a href=&quot;http://en.wikipedia.org/w/index.php?title=Special%3ABookSources&amp;amp;isbn=9780321113610&quot; rel=&quot;nofollow&quot;&gt;9780321113610&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Seems that it's not available for preview on the web and the publisher is out of stock, but you can view &lt;a href=&quot;http://wps.aw.com/aw_murray_economtrcs_1/37/9551/2445250.cw/index.html&quot; rel=&quot;nofollow&quot;&gt;pdfs of 11 web extensions&lt;/a&gt; to get an idea of its style.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-11-17T13:49:06.863" CommunityOwnedDate="2010-11-17T13:49:06.863" CommentCount="1" />
  <row Id="4633" PostTypeId="2" ParentId="4610" CreationDate="2010-11-17T14:02:40.130" Score="5" ViewCount="" Body="&lt;p&gt;As @onestop writes, the GM and GSD are natural parameters for a lognormal distribution.  However, they can be estimated from the arithmetic mean ($\mu$) and (usual) SD ($\sigma$) just by solving the &lt;a href=&quot;http://en.wikipedia.org/wiki/Log-normal_distribution&quot; rel=&quot;nofollow&quot;&gt;formulas&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\mu = \exp(\nu+ \tau^2/2) \text{ and}$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\sigma^2 = \left( \exp(\tau^2)-1\right) \exp(2 \nu+ \tau^2) = \left( \exp(\tau^2)-1\right)  \mu^2$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;for $\nu$ (the logarithm of the GM) and $\tau$ (the logarithm of the GSD).  Evidently&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\tau^2 = \log{\frac{\sigma^2 + \mu^2}{\mu^2}} \text{ and}$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\nu = \frac{1}{2} \log{\frac{\mu^4}{\sigma^2 + \mu^2}}.$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The distribution of the &lt;em&gt;logarithms&lt;/em&gt; of the particle sizes is Normal with mean $\nu$ and variance $\tau^2$, reducing your problem to the elementary one of computing (or looking up) values of the cumulative normal distribution.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-11-17T14:02:40.130" />
  <row Id="4634" PostTypeId="2" ParentId="4612" CreationDate="2010-11-17T14:15:22.657" Score="2" ViewCount="" Body="&lt;p&gt;&quot;&lt;a href=&quot;http://www.springer.com/economics/econometrics/book/978-0-387-77316-2&quot; rel=&quot;nofollow&quot;&gt;Applied Econometrics with R&lt;/a&gt;&quot; (Kleiber, Zeileis 2008) is a good introduction using R, and is accompanied by the &lt;a href=&quot;http://cran.r-project.org/web/packages/AER/index.html&quot; rel=&quot;nofollow&quot;&gt;AER package&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-11-17T14:15:22.657" CommunityOwnedDate="2010-11-17T14:15:22.657" CommentCount="2" />
  <row Id="4635" PostTypeId="2" ParentId="4612" CreationDate="2010-11-17T14:17:55.447" Score="4" ViewCount="" Body="&lt;p&gt;&quot;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0691120358&quot; rel=&quot;nofollow&quot;&gt;Mostly Harmless Econometrics: An Empiricist's Companion&lt;/a&gt;&quot; (Angrist, Pischke 2008) is a less technical and entertaining summary of the field.  I wouldn't describe it as a beginner book, but it's well worth reading once you understand the basics.&lt;/p&gt;&#xA;" OwnerUserId="5" LastEditorUserId="5" LastEditorDisplayName="" LastEditDate="2010-11-17T16:30:37.287" LastActivityDate="2010-11-17T16:30:37.287" CommunityOwnedDate="2010-11-17T14:17:55.447" CommentCount="3" />
  <row Id="4636" PostTypeId="2" ParentId="4620" CreationDate="2010-11-17T14:21:11.723" Score="1" ViewCount="" Body="&lt;p&gt;You can ask consumers a question along the following lines:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Out of every 100 visits to a coffee house how many times do you visit each one of the following coffee house? Please ensure that the total adds up to 100.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;A. Option 1&lt;br&gt;&#xA;  B. Option 2.. etc&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;You can then normalize to get percentage of times each consumer goes to each one of the coffee houses. A simplistic analysis would then assume that each consumer visits a coffee house the same number of times during a year and simply compute the average of the percentages across all consumers to get an estimate of market shares. A more sophisticated analysis would compute the weighted averages with the weights being the the number of times that a consumer goes to coffee shops. You can get the weights as well via a survey question by asking them how many times they visit coffee shops in a year etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are other ways to estimate market shares but the above seems to be a simple yet reasonable approach.&lt;/p&gt;&#xA;" LastActivityDate="2010-11-17T14:21:11.723" CommentCount="3" />
  <row Id="4637" PostTypeId="2" ParentId="4620" CreationDate="2010-11-17T14:21:31.443" Score="2" ViewCount="" Body="&lt;p&gt;I'd suggest several Qs along these lines:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Which is the one coffee house you go to most often?&lt;/li&gt;&#xA;&lt;li&gt;What other coffee houses do you visit more than once a month (say)? [Probe to negative, i.e. keep asking &quot;.. and which others do you visit more than once a month?&quot; &quot;.. and which others?&quot; until interviewee answers &quot;none&quot;/&quot;that's it&quot; / &quot;no others&quot; ]&lt;/li&gt;&#xA;&lt;li&gt;What other coffee houses can you recall having ever visited? [Probe to negative]&lt;/li&gt;&#xA;&lt;li&gt;[For each coffee house mentioned in (1) or (2)]: &quot;How many times have you visited Cafe Y in the last month?&quot;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Tabulating responses to (1) gives &quot;X% of people said they go to Cafe Y most often&quot; for each Y, with sum &amp;lt;=100% (can be &amp;lt;100% as some people never visit coffee houses). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Combining responses to (1) &amp;amp; (2) gives &quot;X% of people go to Cafe Y more than once a month&quot; for each Y, with sum &gt; 100% but still with denominator N for each coffee house, where N is number of respondents. Similarly combining (1), (2) &amp;amp; (3) gives &quot;X% report ever having visited Cafe Y&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Taking the mean of (4) for each coffee house gives you &quot;Cafe Y was visited an average of m times in the preceeding month&quot;.  Denominator is N again (remember to include those respondents who didn't mention Cafe Y as zeroes).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In principle you could refine things further, but asking more Qs that this may increase your respondent quit rate.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-11-17T14:21:31.443" />
  <row Id="4638" PostTypeId="2" ParentId="4619" CreationDate="2010-11-17T14:36:48.460" Score="3" ViewCount="" Body="&lt;p&gt;I think that for subject-specific characteristics, like demographic data, you can proceed the usual way (t-test, etc.). This will help showing that your samples don't differ according to these variables. About self-reported attitude data, if you have very few items, skip to step 2, otherwise step 1 might be appropriate. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;1. Assessing measurement equivalence&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rather than saying that the two populations (or actually, samples) are &quot;the same&quot;, I would say you have to show that your two questionnaires are assessing the same construct(s). This is what is done in cross-cultural surveys or international clinical trials where health-related quality of life is used as a secondary endpoint, for example. In each case, we have a set of items that purports to assess different dimensions, and we want to demonstrate whether we are measuring individuals in the same way irrespective of their country. When dealing with uni- or multidimensional scales, it is know as &lt;em&gt;measurement invariance&lt;/em&gt; in psychometrics, that is you want to show that the factorial structure is comparable between the two groups. But, the same remark would apply as well if we were considering longitudinal data (I interpret your question as involving different samples at each time point). A &lt;em&gt;multi-group confirmatory factor analysis&lt;/em&gt; is appropriate in this case. Standard references include:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Meredith, W (1993). Measurement invariance, factor analysis, and factorial invariance. &lt;em&gt;Psychometrika&lt;/em&gt;, &lt;em&gt;58&lt;/em&gt;, 525-543.&lt;/li&gt;&#xA;&lt;li&gt;Vandenberg, RJ and Lance, CE (2000). A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. &lt;em&gt;Organizational Research Methods&lt;/em&gt;, &lt;em&gt;3&lt;/em&gt;, 4-70.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In R, the &lt;a href=&quot;http://cran.r-project.org/web/packages/lavaan/index.html&quot; rel=&quot;nofollow&quot;&gt;lavaan&lt;/a&gt; package provides facilities for that kind of analysis, but see the documentation: &lt;a href=&quot;http://users.ugent.be/~yrosseel/lavaan/lavaan_usersguide_0.3-1.pdf&quot; rel=&quot;nofollow&quot;&gt;lavaan: an R package for structural equation modeling and more&lt;/a&gt; (§6.2). Otherwise, you have to resort on &lt;a href=&quot;http://www.statmodel.com/&quot; rel=&quot;nofollow&quot;&gt;Mplus&lt;/a&gt; or a good software for SEMs. &lt;a href=&quot;http://www.unc.edu/~rcm/psy236/measinv.pdf&quot; rel=&quot;nofollow&quot;&gt;Studying Measurement Invariance Using Confirmatory Factor Analysis&lt;/a&gt; provides illustration with &lt;a href=&quot;http://www.ssicentral.com/lisrel/&quot; rel=&quot;nofollow&quot;&gt;LISREL&lt;/a&gt; syntax.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may want to consider data from 6 to 12 months (to collect 1 or 2 waves for survey A). After that, I think you can just pool your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;2. Assessing group comparability&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, if you cannot define a clear construct common to those two questionnaires, or if you have so few items that it would make no sense to consider a scale, then you can rely on basic group statistics for each item (using e.g., t-test, trend test for ordinal data, tests for nominal data, etc.). In this case, you are essentially studying between-group differences. This basically tells you whether (aggregated) scores differ, but not whether items are perceived as having the same meaning (or underlying the same construct) across the two groups.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-17T18:22:13.840" LastActivityDate="2010-11-17T18:22:13.840" />
  <row Id="4639" PostTypeId="1" CreationDate="2010-11-17T15:59:25.153" Score="1" ViewCount="401" Body="&lt;p&gt;In R, the &lt;code&gt;drop1&lt;/code&gt;command outputs something neat.&lt;br&gt;&#xA;These two commands should get you some output:&lt;br&gt;&#xA;&lt;code&gt;example(step)#-&amp;gt; swiss&lt;/code&gt;&lt;br&gt;&#xA;&lt;code&gt;drop1(lm1, test=&quot;F&quot;)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Mine looks like this:   &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; drop1(lm1, test=&quot;F&quot;)&#xA;Single term deletions&#xA;&#xA;Model:&#xA;Fertility ~ Agriculture + Examination + Education + Catholic + &#xA;    Infant.Mortality&#xA;                 Df Sum of Sq    RSS    AIC F value     Pr(F)    &#xA;&amp;lt;none&amp;gt;                        2105.0 190.69                      &#xA;Agriculture       1    307.72 2412.8 195.10  5.9934  0.018727 *  &#xA;Examination       1     53.03 2158.1 189.86  1.0328  0.315462    &#xA;Education         1   1162.56 3267.6 209.36 22.6432 2.431e-05 ***&#xA;Catholic          1    447.71 2552.8 197.75  8.7200  0.005190 ** &#xA;Infant.Mortality  1    408.75 2513.8 197.03  7.9612  0.007336 ** &#xA;---&#xA;Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What does all of this mean? I'm assuming that the &quot;stars&quot; help in deciding which input variables are to be kept. &#xA;Looking at the output above, I want to throw away the &quot;Examination&quot; variable and focus on the &quot;Education&quot; variable, is interpretation this correct?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, the AIC value, lower is better, yes?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ed. Please note the Community Wiki answer below and add to it if you see fit, to clarify this output. &lt;/p&gt;&#xA;" OwnerUserId="1994" LastEditorUserId="1994" LastEditorDisplayName="" LastEditDate="2010-11-18T10:53:37.813" LastActivityDate="2010-11-18T10:53:37.813" Title="Interpreting the drop1 output in R" Tags="&lt;r&gt;&lt;regression&gt;&lt;homework&gt;&lt;stepwise-regression&gt;" AnswerCount="2" CommentCount="5" FavoriteCount="2" />
  <row Id="4640" PostTypeId="1" CreationDate="2010-11-17T16:24:18.900" Score="3" ViewCount="170" Body="&lt;p&gt;In R, the &lt;code&gt;step&lt;/code&gt; command is supposedly intended to help you select the input variables to your model, right?&lt;br&gt;&#xA;If you want to follow along, these commands get you the same output I'm looking at: &#xA;&lt;code&gt;example(step)#-&amp;gt; swiss&lt;/code&gt;&lt;br&gt;&#xA;&lt;code&gt;step(lm1)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's not very long this example code, so I'll just paste it here as well for faster reference:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; step(lm1)&#xA;Start:  AIC=190.69&#xA;Fertility ~ Agriculture + Examination + Education + Catholic + &#xA;    Infant.Mortality&#xA;&#xA;                   Df Sum of Sq    RSS    AIC&#xA;- Examination       1     53.03 2158.1 189.86&#xA;&amp;lt;none&amp;gt;                          2105.0 190.69&#xA;- Agriculture       1    307.72 2412.8 195.10&#xA;- Infant.Mortality  1    408.75 2513.8 197.03&#xA;- Catholic          1    447.71 2552.8 197.75&#xA;- Education         1   1162.56 3267.6 209.36&#xA;&#xA;Step:  AIC=189.86&#xA;Fertility ~ Agriculture + Education + Catholic + Infant.Mortality&#xA;&#xA;                   Df Sum of Sq    RSS    AIC&#xA;&amp;lt;none&amp;gt;                          2158.1 189.86&#xA;- Agriculture       1    264.18 2422.2 193.29&#xA;- Infant.Mortality  1    409.81 2567.9 196.03&#xA;- Catholic          1    956.57 3114.6 205.10&#xA;- Education         1   2249.97 4408.0 221.43&#xA;&#xA;Call:&#xA;lm(formula = Fertility ~ Agriculture + Education + Catholic +     Infant.Mortality, data = swiss)&#xA;&#xA;Coefficients:&#xA;     (Intercept)       Agriculture         Education  &#xA;         62.1013           -0.1546           -0.9803  &#xA;        Catholic  Infant.Mortality  &#xA;          0.1247            1.0784  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, when I look at this, I guess the last Step table is the model which we should use? The last few lines include the &quot;Call&quot; function, which describes the actual model and what input variables it includes, and the &quot;Coefficients&quot; are the actual parameter estimates for these values, right? So this is the model I want, right?&#xA;I'm trying to extrapolate this to my project, where there are more variables. &lt;/p&gt;&#xA;" OwnerUserId="1994" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-11-17T16:28:07.110" LastActivityDate="2010-11-18T06:46:07.077" Title="Interpreting the step output in R" Tags="&lt;r&gt;&lt;homework&gt;&lt;stepwise-regression&gt;" AnswerCount="2" FavoriteCount="2" />
  <row Id="4641" PostTypeId="2" ParentId="4639" CreationDate="2010-11-17T16:28:20.420" Score="4" ViewCount="" Body="&lt;p&gt;&lt;code&gt;drop1&lt;/code&gt; gives you a comparison of models based on the AIC criterion, and when using the option &lt;code&gt;test=&quot;F&quot;&lt;/code&gt; you add a &quot;type II ANOVA&quot; to it, &lt;strong&gt;as explained in the help files&lt;/strong&gt;. As long as you only have continuous variables, this table is exactly equivalent to  &lt;code&gt;summary(lm1)&lt;/code&gt;, as the F-values are just those T-values squared. P-values are exactly the same.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So what to do with it? Interprete it in exactly that way: it expresses in a way if the model without that term is &quot;significantly&quot; different from the model with that term. Mind the &quot;&quot; around significantly, as the significance here cannot be interpreted as most people think. (multi-testing problem and all...)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And regarding the AIC : the lower the better seems more like it. AIC is a value that goes for the &lt;em&gt;model&lt;/em&gt;, not for the variable. So the best model from that output would be the one without the variable examination.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Mind you, the calculation of both AIC and the F statistic are different from the R functions &lt;code&gt;AIC(lm1)&lt;/code&gt; resp. &lt;code&gt;anova(lm1)&lt;/code&gt;. For &lt;code&gt;AIC()&lt;/code&gt;, that information is given on the help pages of &lt;code&gt;extractAIC()&lt;/code&gt;. For the &lt;code&gt;anova()&lt;/code&gt; function, it's rather obvious that type I and type II SS are not the same.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm trying not to be rude, but if you don't understand what is explained in the help files there, you shouldn't be using the function in the first place. Stepwise regression is incredibly tricky, jeopardizing your p-values in a most profound manner. So again, do &lt;strong&gt;not&lt;/strong&gt; base yourself on the p-values. Your model should reflect your hypothesis and not the other way around. &lt;/p&gt;&#xA;" OwnerUserId="1124" LastEditorUserId="1124" LastEditorDisplayName="" LastEditDate="2010-11-17T17:07:15.003" LastActivityDate="2010-11-17T17:07:15.003" CommentCount="5" />
  <row Id="4642" PostTypeId="1" AcceptedAnswerId="4654" CreationDate="2010-11-17T17:17:21.297" Score="6" ViewCount="159" Body="&lt;p&gt;I am familiar with meta analysis and meta regression techniques (using the R package &lt;code&gt;metafor&lt;/code&gt; from Viechtbauer), but I recently stumbled on a problem I can't easily solve. Say we have a disease that can go from mother to the unborn child, and it has been studied already a number of times. Mother and child were tested for the virus right after birth. As an unborn child can impossibly get the virus other than from the mother, one would expect crosstabulations like :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;           | neg kid | pos kid&#xA;mother neg |    A    |   C=0&#xA;-----------|---------|--------&#xA;mother pos |    B    |   D&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Obviously using odds ratios (OR) gives errors as one would be dividing by 0. Same for relative risks :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\frac{A/(A+B)}{0/(0+D)}$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now the researchers want to test the (senseless) hypothesis whether infection of the child is related to the infection of the mother (which seems very, very obvious). I'm trying to reformulate the hypothesis and come up with something that makes sense, but I can't really find something. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To complicate things, some kids with negative moms actually are positive, probably due to infection in the first week. So I only have a number of studies where C = 0. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anybody an idea on how to statistically summarize the data of different studies following such a pattern. Links to scientific papers are also more than welcome. &lt;/p&gt;&#xA;" OwnerUserId="1124" LastEditorUserId="1124" LastEditorDisplayName="" LastEditDate="2010-11-17T23:12:37.027" LastActivityDate="2010-11-23T21:43:38.687" Title="Meta analysis on studies with 0-frequency cells" Tags="&lt;meta-analysis&gt;&lt;odds-ratio&gt;&lt;relative-risk&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="1" />
  <row Id="4643" PostTypeId="1" AcceptedAnswerId="4646" CreationDate="2010-11-17T18:06:56.853" Score="4" ViewCount="116" Body="&lt;p&gt;I originally asked this on a machine learning site, but one of the responses made me think that maybe this site is more suitable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose you have two weighted coins, and every day you flip each one a number of times and record the total number of heads. So on the tenth day you might have flipped coin A 106 times, coin B 381 times, and recorded 137 heads. Supposing your goal is to figure out the weights of each coin, is it reasonable to just regress the number of heads on the number of flips for each coin? E.g, something along the lines of:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;num_heads ~ num_flips_A + num_flips_B + intercept&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However,it doesn't seem to make sense to have an intercept term in this scenario (it is negative for my data, which is confusing), so I tried subtracting out -1 from the formula, and it seemed to yield reasonable results. My first question is whether this approach is a good one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, suppose that you suspect the existence of a third coin, C, that someone else is flipping unbeknownst to you, and the heads for that coin are getting mixed up in your count. The number of flips for this coin are not recorded, but you do not particularly care about its weight - it's more of a confounding factor. Then would it be reasonable to fit a similar regression, but constrain the intercept to be positive?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for any help&lt;/p&gt;&#xA;" OwnerUserId="1777" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-11-18T14:25:57.537" LastActivityDate="2010-11-18T14:25:57.537" Title="Toy regression question with latent variables" Tags="&lt;regression&gt;&lt;latent-variable&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="4644" PostTypeId="2" ParentId="4639" CreationDate="2010-11-17T18:07:33.793" Score="0" ViewCount="" Body="&lt;p&gt;For reference, these are the values that are included in the table:&lt;br&gt;&#xA;&lt;code&gt;Df&lt;/code&gt; refers to &lt;a href=&quot;http://en.wikipedia.org/wiki/Degrees_of_freedom_%28statistics%29&quot; rel=&quot;nofollow&quot;&gt;Degrees of freedom&lt;/a&gt;, &quot;the number of degrees of freedom is the number of values in the final calculation of a statistic that are free to vary.&quot;  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;code&gt;Sum of Sq&lt;/code&gt; column refers to the &lt;a href=&quot;http://en.wikipedia.org/wiki/Sum_of_squares&quot; rel=&quot;nofollow&quot;&gt;sum of squares&lt;/a&gt; (or more precisely &lt;a href=&quot;http://en.wikipedia.org/wiki/Squared_deviations&quot; rel=&quot;nofollow&quot;&gt;sum of squared deviations&lt;/a&gt;). In short this is a measure of the amount that each individual value deviates from the overall mean of those values.&lt;br&gt;&#xA;&lt;code&gt;RSS&lt;/code&gt; is the &lt;a href=&quot;http://en.wikipedia.org/wiki/Residual_sum_of_squares&quot; rel=&quot;nofollow&quot;&gt;Residual Sum of Squares&lt;/a&gt;. These are a measure of how much the predicted value of the dependent (or output) variable varies from the true value for each data point in the set (or more colloquially: each &quot;line&quot; in the data table). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;AIC&lt;/code&gt; is the &lt;a href=&quot;http://en.wikipedia.org/wiki/Akaike_information_criterion&quot; rel=&quot;nofollow&quot;&gt;Akaike information criterion&lt;/a&gt; which is generally regarded as &quot;too complex to explain&quot; but is, in short, a measure of the goodness of fit of an estimated statistical model. If you require further details, you will have to turn to dead trees with words on them. Or Wikipedia and the resources there.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;code&gt;F value&lt;/code&gt; is used to perform what's called an &lt;a href=&quot;http://en.wikipedia.org/wiki/F-test&quot; rel=&quot;nofollow&quot;&gt;F-test&lt;/a&gt; and from it is derived the &lt;code&gt;Pr(F)&lt;/code&gt; value, which describes how likely (or Probable = Pr) that F value is. A Pr(F) value close to zero (indicated by &lt;code&gt;***&lt;/code&gt;) is indicative of an input variable that is in some way important to include in a good model, that is, a model that does not include it is &quot;significantly&quot; different than the one that does.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All of these values are, in the context of the &lt;code&gt;drop1&lt;/code&gt; command, calculated to compare the overall model (including all the input variables) with the model resulting from removing that one specific variable per each line in the output table. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, if this can be improved upon, please feel free to add to it or clarify any issues. My goal is only to clarify and provide a better &quot;reverse lookup&quot; referene from the output of an R command to the actual meaning of it. &lt;/p&gt;&#xA;" OwnerUserId="1994" LastEditorUserId="1994" LastEditorDisplayName="" LastEditDate="2010-11-17T22:01:08.247" LastActivityDate="2010-11-17T22:01:08.247" CommunityOwnedDate="2010-11-18T09:52:47.537" CommentCount="7" />
  <row Id="4645" PostTypeId="2" ParentId="4620" CreationDate="2010-11-17T18:47:25.040" Score="1" ViewCount="" Body="&lt;p&gt;I think there is a big difference (both practical and how you approach the problem) in popularity vis a vis market share. Since market share is more analytically challenging, I'll focus on that. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my opinion, the best solution to this problem is going to involve a set of stated preference experiments, more formally known as &lt;a href=&quot;http://en.wikipedia.org/wiki/Discrete_choice&quot; rel=&quot;nofollow&quot;&gt;discrete choice modeling&lt;/a&gt;. DCM has applications in a variety of contexts all involving a consumer and some choice they need to make. That choice may be what coffee shop to go to, what computer to buy, or how to get to work in the morning. If you think about the important the important attributes that may impact consumer choice when picking a coffee shop to visit, you will be able to design experiments to capture this information. This could include price, offerings, proximity to work, brand loyalty, etc. You would want to design a set of experiemtns that tests each of these attributes independently of one another and then use one of the DCM techniques to develop coefficient estimates for those attributes. Turning that into a market share simulator is a relatively easy task, though model validation is always an issue. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my work, the &quot;seminal&quot; reference text is by &lt;a href=&quot;http://mitpress.mit.edu/catalog/item/default.asp?ttype=2&amp;amp;tid=8271&quot; rel=&quot;nofollow&quot;&gt;Ben Akiva and Lerman&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ken Train's book is also a great &lt;a href=&quot;http://www.listinet.com/bibliografia-comuna/Cdu339-267B.pdf&quot; rel=&quot;nofollow&quot;&gt;resource&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, &lt;a href=&quot;http://www.sawtoothsoftware.com/&quot; rel=&quot;nofollow&quot;&gt;Sawtooth Software&lt;/a&gt; is a fairly large player in this market and they have a variety of tools that can assist in the design and analysis of DCM. I don't use their tools that often however.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another method that you may be able to adapt is the &lt;a href=&quot;http://en.wikipedia.org/wiki/Van_Westendorp%27s_Price_Sensitivity_Meter&quot; rel=&quot;nofollow&quot;&gt;Van Westendorp method&lt;/a&gt;. We recently used a set of questions to validate our DCM for a high end electronics study we performed. Van Westendorp was quick, easy, and provided instant feedback for our client. It certainly has it's drawbacks, but that's for another discussion.&lt;/p&gt;&#xA;" OwnerUserId="696" LastActivityDate="2010-11-17T18:47:25.040" CommentCount="1" />
  <row Id="4646" PostTypeId="2" ParentId="4643" CreationDate="2010-11-17T20:56:40.703" Score="4" ViewCount="" Body="&lt;p&gt;Let's take the second question first: in most cases the presence of C makes the parameters unidentifiable and there's no way to estimate anything.  Constraining the intercept to be positive won't help at all.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first problem can be solved by Maximum Likelihood.  After all, writing $p$ for the probability that a flip of A is heads, $q$ for the probability that B is heads, and $k_i$, $n_i$, and $m_i$ for the numbers of heads, number of flips of A, and number of flips of B in the the $i^\text{th}$ toss, respectively, we know&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\Pr[k_i] = \sum_{j=0}^{k_i}{{{n_i}\choose{j}}{{m_i}\choose{k_i-j}}p^j(1-p)^{n-j}q^{k_i-j}(1-q)^{m_i-k_i-j}}$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(taking ${r}\choose{s}$ to be zero whenever $r \lt s$ or $s \lt 0$).  The product of these expressions (over $i$) is the likelihood to be maximized by varying $p$ and $q$ subject to $0 \le p, q \le 1$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maximizing this likelihood for large amounts of data does not look appetizing.  If we assume both $p$ and $q$ are sufficiently far from $0$ and $1$ and that the numbers of coin flips are usually large enough that all of $n_i p$, $n_i(1-p)$, $m_i q$, and $m_i (1-q)$ exceed $5$ or so, we can apply the Normal approximation to the Binomial.  This tells us that, to a good approximation, the distribution of $k_i$ is Normal with mean $p n_i + q m_i$ and variance $p(1-p) n_i + q(1-q) m_i$.  Thus, &lt;em&gt;weighted&lt;/em&gt; least squares regression (WLS) of $k$ on $(n,m)$&#xD;&#xA; &lt;em&gt;without any intercept term&lt;/em&gt; is justified.  The weights are inversely proportional to these variances.  However, the variances depend on the solution.  Several approaches immediately come to mind:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Use ML based on the Normal approximation.  This likelihood is not as messy as the (more accurate) Binomial likelihood.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Guess initial values for $(p, q)$, use these to compute the weights, and perform WLS to update the estimates of $(p, q)$.  Replace the initial values with these estimates and iterate until convergence occurs (one hopes).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Obtain Bayesian estimates for the parameters. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;h2&gt;Edit&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Assuming, as suggested in a comment, that C is flipped approximately the same number of times (say $l$) in each trial, let's analyze this situation using the Normal approximation.  Letting the expectation of C be $r$, the expectation of $k_i$ becomes $p n_i + q m_i + r l$ with variance $p(1-p)n_i + q(1-q)m_i + r(1-r)l$.  We can consider playing the iterated WLS game by estimating $p$, $q$, and $\rho = r l$ with WLS (constraining the intercept to be nonnegative as originally suggested) and updating the previous values of the parameters with the new estimates $\hat{p}$, $\hat{q}$, and $\hat{\rho} / l$.  Thus, &lt;em&gt;when $l$ is known,&lt;/em&gt; it appears that $p$, $q$, and $r$ are indeed identifiable and that--assuming the iterative method works--there is an effective algorithm for finding them.  Note, too, that the Normal approximation should work well provided the expectation of the $k_i$ is typically between 5 and $n_i + m_i + l - 5$; that is, some of $p$, $q$, and $r$ can be quite small or even zero and this still should work provided that neither heads nor tails is a rare occurrence among the &lt;em&gt;totals&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When $l$ is not known or varies we can still estimate $r l$ provided we have a reasonable estimate for the contribution of C to the variance, $r(1-r)l$.  &quot;Reasonable&quot; means that when we vary $l$ and $r$ within meaningful ranges, the changes in the &lt;em&gt;total&lt;/em&gt; variances $p(1-p)n_i + q(1-q)m_i + r(1-r)l$ do not alter the parameter estimates appreciably.  This would be the case, for instance, when the contribution of C to the total number of heads is &lt;em&gt;consistently&lt;/em&gt; relatively small.  (We don't really care about $r l$--it's a nuisance parameter--but if it is not accurately estimated then there must be some bias in the estimation of $p$ and $q$.)&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-11-18T14:22:23.357" LastActivityDate="2010-11-18T14:22:23.357" CommentCount="1" />
  <row Id="4647" PostTypeId="2" ParentId="4642" CreationDate="2010-11-17T21:12:39.227" Score="4" ViewCount="" Body="&lt;p&gt;Usually 0's imply that you have to use exact methods instead of relying on asymptotical methods such as meta-analysis with odds ratios. If you are willing to assume that the study effect is fixed, an exact Maentel-Hanszel test is the way to go. For an exact random effects analysis, you have to use a binomial regression model with a random study effect. I have done both in a recent applied paper, but the methods section there would not be more helpful to you, as it essentially conveys this information.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This paper is not applied, but this is where I got the idea from when confronted with the same issue:&lt;br&gt;&#xA;[1] Hans C. van Houwelingen, Lidia R. Arends, and Theo Stijnen. Advanced methods in meta-analysis: multivariate approach and meta-regression. &lt;em&gt;Statistics in Medicine&lt;/em&gt;, 2002; 21:589–624&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the paper where I used this approach (it is not apparent in the abstract, but is mentioned in the methods section):&lt;br&gt;&#xA;[2]  Trivedi H, Nadella R, Szabo A. Hydration with sodium bicarbonate for the prevention of contrast-induced nephropathy: a meta-analysis of randomized controlled trials. &lt;em&gt;Clin Nephrol.&lt;/em&gt; 2010 Oct;74(4):288-96.&lt;/p&gt;&#xA;" OwnerUserId="279" LastEditorUserId="279" LastEditorDisplayName="" LastEditDate="2010-11-23T21:43:38.687" LastActivityDate="2010-11-23T21:43:38.687" CommentCount="2" />
  <row Id="4648" PostTypeId="2" ParentId="4640" CreationDate="2010-11-17T21:16:43.250" Score="0" ViewCount="" Body="&lt;p&gt;The part of the printout at the end is the model you are left with. You can also get it if you capture the value of the &lt;code&gt;step&lt;/code&gt; function:  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;final.mod &amp;lt;- step(lm1)&#xA;final.mod&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="279" LastActivityDate="2010-11-17T21:16:43.250" />
  <row Id="4649" PostTypeId="1" CreationDate="2010-11-17T21:17:44.603" Score="1" ViewCount="292" Body="&lt;p&gt;If the outcome of a market could be expressed as a probability it might be: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Outcome - Description - Probability as a %&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Up a lot 20% (a move of say more than 10%) &lt;/li&gt;&#xA;&lt;li&gt;Down a lot 20% &lt;/li&gt;&#xA;&lt;li&gt;Up a bit 20% (a move of between 0 and 10%) &lt;/li&gt;&#xA;&lt;li&gt;Down a bit 20% &lt;/li&gt;&#xA;&lt;li&gt;Sideways 20% &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;So the probability of any single outcome is 1/5 or 20%. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please could some one educate me on the math of adding another and subsequent markets?&lt;/p&gt;&#xA;" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-11-17T21:53:57.087" LastActivityDate="2010-11-17T22:22:02.307" Title="How to combine probabilities?" Tags="&lt;probability&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="4650" PostTypeId="2" ParentId="4649" CreationDate="2010-11-17T21:36:42.957" Score="1" ViewCount="" Body="&lt;p&gt;OK, let me say something about the multinomial distribution since I brought it up.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose I have 2 dice both with 5 faces each. Assuming that the dice are fair, the probability of one particular face turning up with one die is indeed $1/5$ or 20%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, let's ask what happens when we throw two dice and ask ourselves what the probability is of having them both showing the same number. Two ones could come up with a probability of $1/25$, two twos could turn up with the same probability, etc... the total probabiliy would thus be $5 \times 1/25$ or $1/5$. A quicker way to see this, whatever comes out for the first die is setting the bar for the second die. So the second die has $1/5$ of coming out right.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's ask the same question for three dice. The probability will be $1/25$ by an analoguous reasoning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The multinomial distribution allows us to compute much more. It can compute the probability of a particular number combination coming out if we don't care about the order in which the numbers come out. This probability for $n$ fair dice with $p$ sides each is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\mathcal{P}(n_1,\ldots,n_p)=\frac{n!}{n_1!n_2!\ldots n_p!} \frac{1}{p^n}$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Take for instance the probability for throwing (1,1,2) with 3 5-sided dice, that is two ones and one two:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\mathcal{P}(2,1,0,0,0)=\frac{3!}{2!1!0!0!0!} \frac{1}{5^3} = \frac{3}{125}$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;What do you mean by market here? You mean something like the Dow Jones Industrial average?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If that's what you mean, then the way a probability would be expressed would be something like this. Denoting the Dow Jones Industrial Average in function of time as $D(t)$, we can ask what is the probability to have a rise of 10% over a certain time interval $\Delta t$. Say this probability is 5%. This would be expressed as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ \mathbb{P}[D(t+\Delta t)-D(t) &amp;gt; 0.1 D(t)] = 0.05$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, this is just notation. No actual information has been put in. If you're asking how can we model the Dow Jones Industrial Average, I'm afraid this is not something that can be given a quick answer to. I'd suggest you start reading up on the subject because it involves a lot of math. Maybe an easy start:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Stock_market&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Stock_market&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Expect the learning curve to be steep though.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The important thing to understand in your case though is that just because you can express something with % it doesn't mean it is a probability. It could correspond to a relative change, for instance, a relative change in the value of a stock. That is not a probability. But there is a probability associated to the occurance of that change.&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="2036" LastEditorUserId="2036" LastEditorDisplayName="" LastEditDate="2010-11-17T22:22:02.307" LastActivityDate="2010-11-17T22:22:02.307" CommentCount="8" />
  <row Id="4651" PostTypeId="1" CreationDate="2010-11-17T21:39:01.213" Score="1" ViewCount="155" Body="&lt;p&gt;I am trying to calculate the intra-observer variability on a 17 items test, in a group of n observer, with SPSS....&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think I must use ICC, but I don't know how to present the data for it&lt;/p&gt;&#xA;&#xA;&lt;p&gt;each item in row, and each n observer in colunms, twice, and then compare one each time? is there another way for all?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;thanks&lt;/p&gt;&#xA;&#xA;&lt;p&gt;David&lt;/p&gt;&#xA;" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-11-17T21:57:55.963" LastActivityDate="2010-11-17T21:57:55.963" Title="How to present data for spss variability intra-observer?" Tags="&lt;spss&gt;&lt;psychometrics&gt;&lt;intraclass-correlation&gt;" CommentCount="1" FavoriteCount="1" />
  <row Id="4652" PostTypeId="1" CreationDate="2010-11-17T23:10:31.310" Score="4" ViewCount="141" Body="&lt;p&gt;I'm trying to run a basic gradient descent algorithm with a absolute loss function. I can get it to converge to a good solution by it requires a much lower step size and more iterations than had I used square loss. Is this normal? Should I expect absolute loss to take a longer time to come to a good solution or potentially oscillate around a solution more than say squared loss?  &lt;/p&gt;&#xA;" OwnerUserId="2023" LastEditorUserId="2023" LastEditorDisplayName="" LastEditDate="2010-11-19T05:03:38.713" LastActivityDate="2010-11-19T05:14:16.130" Title="Gradient descent oscillating a lot. Have I chosen my step direction incorrectly?" Tags="&lt;optimization&gt;&lt;loss-functions&gt;" AnswerCount="3" FavoriteCount="1" />
  <row Id="4653" PostTypeId="2" ParentId="3772" CreationDate="2010-11-17T23:55:00.300" Score="0" ViewCount="" Body="&lt;p&gt;@whuber Do you have R code perhaps to share on how you did this?&lt;/p&gt;&#xA;" OwnerUserId="2040" LastActivityDate="2010-11-17T23:55:00.300" CommentCount="2" />
  <row Id="4654" PostTypeId="2" ParentId="4642" CreationDate="2010-11-18T00:25:36.700" Score="3" ViewCount="" Body="&lt;p&gt;Seems to me this is one of the rare situations where it might well be better to meta-analyse risk &lt;em&gt;differences&lt;/em&gt; rather than risk ratios or odds ratios.  The risk difference $P(Kid_+ | Mum_+) - P(Kid_+|Mum_-)$ is estimated in each study by $D/(B+D) - C/(A+C)$. That should be finite in all studies even when $C=0$, so there should be no problem meta-analysing it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I agree it seems pretty pointless to consider testing the hypothesis that this risk difference is zero. But it's meaningful to estimate how large it is, i.e. &lt;em&gt;how much&lt;/em&gt; more likely a kid is to have the virus when their mum has it than when their mums doesn't.&lt;/p&gt;&#xA;" OwnerUserId="449" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-11-18T00:36:46.683" LastActivityDate="2010-11-18T00:36:46.683" CommentCount="1" />
  <row Id="4655" PostTypeId="1" AcceptedAnswerId="4657" CreationDate="2010-11-18T00:37:37.360" Score="4" ViewCount="270" Body="&lt;p&gt;&lt;strong&gt;Background&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;notation: RV= random variable, $\mu=$ mean $m=$ median&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Jensen's Inequality considers the relationship between the mean of a function of an RV and the function of the mean of an RV.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If $f(x)$ strictly convex:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\mu (f(x)) &amp;gt; f(\mu (x))\mathrm{\hspace{20mm}(1)}$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Conversely if -f(x) is strictly convex: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\mu (f(x)) &amp;lt; f(\mu (x))$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An analogous property of the median has been presented (&lt;a href=&quot;http://dx.doi.org/10.1016/j.spl.2004.11.010&quot; rel=&quot;nofollow&quot;&gt;Merkle et al 2005&lt;/a&gt;, &lt;a href=&quot;http://milanmerkle.com/documents/radovi/SPL-71.pdf&quot; rel=&quot;nofollow&quot;&gt;pdf&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;motivation&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a nonlinear &lt;a href=&quot;http://www.esajournals.org/doi/full/10.1890/0012-9615%282001%29071%5B0557%3AAMFSVD%5D2.0.CO%3B2&quot; rel=&quot;nofollow&quot;&gt;function&lt;/a&gt; of positive random variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In practice, I find that the function of the medians provides a much better estimate of the median of the function than does the estimate of the mean of the function from the function of the means. I am interested in learning the conditions for which this is true.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;question&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Under what conditions will the function of a median be closer to the median of a function than the mean of a function is to a function of the mean?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Specifically for what types of $f(x)$ and $x$ is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\mu (f(x)) - f(\mu (x)) &amp;gt; m (f(x)) - f(m (x))$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;simulation results&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I used an empirical approach (the one I know) to investigate this question for a function of a single variable: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Interestingly, for $x&amp;gt;0$, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$m(x^2)\simeq m(x)^2$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(1)&#xA;x&amp;lt;-cbind(rlnorm(100, 1), rbeta(100, 1, 5), rgamma(100,0.5,0.5))&#xA;quad &amp;lt;- function(x)x^2&#xA;&#xA;median.x &amp;lt;- apply(x,2,quantile,0.5)&#xA;mean.x &amp;lt;- apply(x,2,mean)&#xA;&#xA;colMeans(quad(x))&#xA;quad(mean.x)&#xA;&#xA;apply(quad(x), 2, quantile, 0.5)&#xA;quad(median.x)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For a slightly more complicated function, my proposal (equation 1) is true&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;miscfn &amp;lt;- function(x) 1 + x + x^log(x^2) - exp(-2(x)*5^x &#xA;&#xA;colMeans(miscfn(x))&#xA;miscfn(mean.x)&#xA;&#xA;apply(miscfn(x), 2, quantile, 0.5)&#xA;miscfn(median.x)&#xA;&#xA;abs(apply(miscfn(x),2,mean)-miscfn(mean.x)) &amp;gt; abs(apply(miscfn(x), 2, quantile, 0.5) - miscfn(median.x))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, before I begin to use this observation in my work, I would like to know more about its conditions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;References&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://dx.doi.org/10.1016/j.spl.2004.11.010&quot; rel=&quot;nofollow&quot;&gt;Merkle et al 2005 Jensen's inequality for medians. Statistics &amp;amp; Probability Letters, Volume 71, Issue 3, 1 March 2005, Pages 277-281&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1381" LastEditorUserId="1381" LastEditorDisplayName="" LastEditDate="2010-11-18T15:59:28.850" LastActivityDate="2010-12-01T05:56:23.050" Title="Is there a relationship between the median of a function of random variables and the function of the median of random variables?" Tags="&lt;probability&gt;&lt;mean&gt;&lt;random-variable&gt;&lt;theory&gt;&lt;median&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="4656" PostTypeId="2" ParentId="4652" CreationDate="2010-11-18T00:49:28.300" Score="4" ViewCount="" Body="&lt;p&gt;When you say 'a absolute loss function', do you mean you're using &lt;a href=&quot;http://en.wikipedia.org/wiki/Least_absolute_deviations&quot; rel=&quot;nofollow&quot;&gt;least absolute deviations&lt;/a&gt; (LAD) instead of the more usual ordinary least squares (OLS)? As that wikipedia article says, although LAD is more robust to outliers than OLS it can be unstable and even have multiple solutions, so it doesn't seem that surprising if it's harder to find the minimum of the objective function even when there's only one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're trying this because you're after some sort of &lt;a href=&quot;http://en.wikipedia.org/wiki/Robust_regression&quot; rel=&quot;nofollow&quot;&gt;robust regression&lt;/a&gt;, I think there are several more attractive alternatives than LAD.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-11-18T00:49:28.300" />
  <row Id="4657" PostTypeId="2" ParentId="4655" CreationDate="2010-11-18T01:09:48.170" Score="3" ViewCount="" Body="&lt;p&gt;Let the cdf of $x$ be denoted by $F_X(x)$. Thus, the median of $X$ denoted by $m_x$ satisfies:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$F_X(m_x)=0.5$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider $Y = X^2$. Thus, the cdf of $Y$ is given by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(Y \le y) = P(X^2 \le y)$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words, the cdf of $Y$ is given by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$F_Y(y) = F_X(\sqrt{y}) - F_X(-\sqrt{y})$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The median for $Y$ denoted by $m_Y$ satisfies:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$F_Y(m_y)=0.5$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words, it should satisfy:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$F_X(\sqrt{m_y}) - F_X(-\sqrt{m_y}) = 0.5$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If $m_y = (m_x)^2$ then it must be that:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$F_X(m_x) - F_X(-m_x) = 0.5$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The above with the first equation suggests that the relationship $m(x^2) = m(x)^2$ will only hold if $F_X(-m_x) = 0$. Thus, the relationship holds only if the support of $X$ is positive. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The examples you examined in your code have a positive support and hence you find that $m(x^2) = m(x)^2$. If you try a uniform distribution (e.g., U(-1,1)) you will find that $m(x^2) \ne m(x)^2$&#xD;&lt;/p&gt;&#xA;" LastActivityDate="2010-11-18T01:09:48.170" CommentCount="1" />
  <row Id="4658" PostTypeId="1" AcceptedAnswerId="4664" CreationDate="2010-11-18T01:52:10.523" Score="5" ViewCount="247" Body="&lt;p&gt;Hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have two set of data that are roughly centered around zero but I suspect that they have different tails. &#xA;I know a few tests to compare the distribution to a normal distribution, but I would like to compare directly the two distributions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a &lt;strong&gt;simple test to compare the fatness of tail of 2 distributions&lt;/strong&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;br&gt;&#xA;fRed&lt;/p&gt;&#xA;" OwnerUserId="1709" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-11-18T10:34:38.613" LastActivityDate="2010-11-18T19:16:24.867" Title="Comparison of the tails of two sample distributions" Tags="&lt;distributions&gt;&lt;test&gt;&lt;kurtosis&gt;" AnswerCount="3" CommentCount="4" />
  <row Id="4659" PostTypeId="1" AcceptedAnswerId="4684" CreationDate="2010-11-18T02:51:42.343" Score="7" ViewCount="426" Body="&lt;p&gt;I'm more of a programmer than a statistician, so I hope this question isn't too naive.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It happens in sampling program executions at random times. If I take N=10 random-time samples of the program's state, I could see function Foo being executed on, for example, I=3 of those samples. I'm interested in what that tells me about the actual fraction of time F that Foo is in execution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand that I is binomially distributed with mean F*N. I also know that, given I and N, F follows a beta distribution. In fact I've verified by program the relationship between those two distributions, which is&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cdfBeta(I, N-I+1, F) + cdfBinomial(N, F, I-1) = 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The problem is I don't have an intuitive feel for the relationship. I can't &quot;picture&quot; why it works.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: All the answers were challenging, especially @whuber's, which I still need to grok, but bringing in order statistics was very helpful. Nevertheless I've realized I should have asked a more basic question: Given I and N, what is the distribution for F? Everyone has pointed out that it's Beta, which I knew. I finally figured out from Wikipedia (&lt;a href=&quot;http://en.wikipedia.org/wiki/Conjugate_prior&quot; rel=&quot;nofollow&quot;&gt;Conjugate prior&lt;/a&gt;) that it appears to be &lt;code&gt;Beta(I+1, N-I+1)&lt;/code&gt;. After exploring it with a program, it appears to be the right answer. So, I would like to know if I'm wrong. And, I'm still confused about the relationship between the two cdfs shown above, why they sum to 1, and if they even have anything to do with what I really wanted to know.&lt;/p&gt;&#xA;" OwnerUserId="1270" LastEditorUserId="1270" LastEditorDisplayName="" LastEditDate="2010-11-20T14:59:45.877" LastActivityDate="2010-11-20T14:59:45.877" Title="Relationship between Binomial and Beta distributions" Tags="&lt;binomial&gt;&lt;beta-binomial&gt;&lt;beta&gt;" AnswerCount="4" CommentCount="4" />
  <row Id="4660" PostTypeId="2" ParentId="4658" CreationDate="2010-11-18T03:06:39.537" Score="1" ViewCount="" Body="&lt;p&gt;The Chi Square test (Goodness-of-Fit test) will be very good at comparing the tails of two distributions since it is structured to compare two distributions by buckets of values (graphically represented by a histogram).  And, the tails will consist in the far most buckets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even though this test focuses on the whole distribution, not just the tail you can readily observe how much of the Chi Square value or divergence is derived by the difference in the tails's fatness.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Watch that the derived histogram may actually give you visually much more information regarding the respective fatness of the tails than any test related statistical significance.  It is one thing to state that tails fatness are statistically different.  It is another to visually observe it.  They say a picture is worth a thousand words.  Sometimes it is also worth a thousand numbers (it makes sense given that graphs encapsulate all the numbers).      &lt;/p&gt;&#xA;" OwnerUserId="1329" LastEditorUserId="1329" LastEditorDisplayName="" LastEditDate="2010-11-18T19:16:24.867" LastActivityDate="2010-11-18T19:16:24.867" CommentCount="7" />
  <row Id="4661" PostTypeId="2" ParentId="4658" CreationDate="2010-11-18T03:47:12.950" Score="1" ViewCount="" Body="&lt;p&gt;How about fitting the &lt;a href=&quot;http://tolstoy.newcastle.edu.au/~rking/gld/&quot; rel=&quot;nofollow&quot;&gt;generalized lambda distribution&lt;/a&gt; and bootstrapping confidence intervals on the 3rd and 4th parameters?&lt;/p&gt;&#xA;" OwnerUserId="364" LastActivityDate="2010-11-18T03:47:12.950" CommentCount="1" />
  <row Id="4662" PostTypeId="2" ParentId="4640" CreationDate="2010-11-18T06:46:07.077" Score="2" ViewCount="" Body="&lt;p&gt;The last step table is indeed the end result of the &quot;stepwise regression&quot;.  The caveat here is that usually you don't want to use this approach when there is a principled way to approach your model specification.  The call is the lm call which would produce the equation used in the final step.  Coefficients are the actual parameter estimates.  It is notable that because you did note define a scope or direction parameter the type of stepwise approach used was a backwards stepwise in which variable terms were evaluated for dropping at each step, the variable which when dropped improved the model the least was removed and the entire process repeated until it became the case that no single variable could be dropped which would improve over the model entering that &quot;step&quot; of the procedure.  I.E. At the final step Fertility ~ Agriculture + Education + Catholic + Infant.Mortality produced an AIC of 189.86, and dropping any one of those variables did not result in a lower AIC (indicative of a better model fit).&lt;/p&gt;&#xA;" OwnerUserId="196" LastActivityDate="2010-11-18T06:46:07.077" />
  <row Id="4663" PostTypeId="1" AcceptedAnswerId="4790" CreationDate="2010-11-18T07:28:22.207" Score="10" ViewCount="554" Body="&lt;p&gt;Least-angle regression and the lasso tend to produce very similar regularization paths (identical except when a coefficient crosses zero.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;They both can be efficiently fit by virtually identical algorithms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there ever any practical reason to prefer one method over the other?&lt;/p&gt;&#xA;" OwnerUserId="439" LastActivityDate="2011-01-16T17:42:01.520" Title="Least-angle regression vs. lasso" Tags="&lt;regression&gt;&lt;lasso&gt;" AnswerCount="4" FavoriteCount="3" />
  <row Id="4664" PostTypeId="2" ParentId="4658" CreationDate="2010-11-18T08:46:46.627" Score="3" ViewCount="" Body="&lt;p&gt;This question seems to belong to the same family as &lt;a href=&quot;http://stats.stackexchange.com/q/1853/449&quot;&gt;this earlier one about testing whether two samples have the same skew&lt;/a&gt;, so &lt;a href=&quot;http://stats.stackexchange.com/questions/1853/testing-two-independent-samples-for-null-of-same-skew/1953#1953&quot;&gt;you may like to read my answer to that&lt;/a&gt;. I believe that &lt;strong&gt;L-moments&lt;/strong&gt; would be useful here too for the same reasons (specifically &lt;em&gt;L-&lt;strike&gt;skewness&lt;/strike&gt;kurtosis&lt;/em&gt; in this case).&lt;/p&gt;&#xA;" OwnerUserId="449" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-11-18T12:45:27.967" LastActivityDate="2010-11-18T12:45:27.967" CommentCount="2" />
  <row Id="4665" PostTypeId="2" ParentId="4652" CreationDate="2010-11-18T09:21:12.907" Score="-1" ViewCount="" Body="&lt;p&gt;I think you should definitely try other methods there is for example Lasso:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In some contexts a regularized version of the least squares solution may be preferable. The LASSO (least absolute shrinkage and selection operator) algorithm, for example, finds a least-squares solution with the constraint that | β | 1, the L1-norm of the parameter vector, is no greater than a given value. Equivalently, it may solve an unconstrained minimization of the least-squares penalty with α | β | 1 added, where α is a constant (this is the Lagrangian form of the constrained problem.) This problem may be solved using quadratic programming or more general convex optimization methods, as well as by specific algorithms such as the least angle regression algorithm. The L1-regularized formulation is useful in some contexts due to its tendency to prefer solutions with fewer nonzero parameter values, effectively reducing the number of variables upon which the given solution is dependent. For this reason, the LASSO and its variants are fundamental to the field of compressed sensing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;NEvertheless though the idea of least absolute deviations regression is just as straightforward as that of least squares regression, the least absolute deviations line is not as simple to compute efficiently. Unlike least squares regression, least absolute deviations regression does not have an analytical solving method. Therefore, an iterative approach is required. The following is an enumeration of some least absolute deviations solving methods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Simplex-based methods (such as the Barrodale-Roberts algorithm)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Iteratively re-weighted least squares&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Wesolowsky’s direct descent method&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Li-Arce’s maximum likelihood approach&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Check all combinations of point-to-point lines for minimum sum of errors&#xA;Simplex-based methods are the “preferred” way to solve the least absolute deviations problem. A Simplex method is a method for solving a problem in linear programming. The most popular algorithm is the Barrodale-Roberts modified Simplex algorithm. The algorithms for IRLS, Wesolowsky's Method, and Li's Method can be found in Appendix A of this document,[7] among other methods. Checking all combinations of lines traversing any two (x,y) data points is another method of finding the least absolute deviations line. Since it is known that at least one least absolute deviations line traverses at least two data points, this method will find a line by comparing the SAE of each line, and choosing the line with the smallest SAE. In addition, if multiple lines have the same, smallest SAE, then the lines outline the region of multiple solutions. Though simple, this final method is inefficient for large sets of data.&lt;/p&gt;&#xA;" OwnerUserId="1808" LastActivityDate="2010-11-18T09:21:12.907" CommentCount="2" />
  <row Id="4666" PostTypeId="2" ParentId="4663" CreationDate="2010-11-18T09:46:57.010" Score="1" ViewCount="" Body="&lt;p&gt;In some contexts a regularized version of the least squares solution may be preferable. The LASSO (least absolute shrinkage and selection operator) algorithm, for example, finds a least-squares solution with the constraint that | β | 1, the L1-norm of the parameter vector, is no greater than a given value. Equivalently, it may solve an unconstrained minimization of the least-squares penalty with α | β | 1 added, where α is a constant (this is the Lagrangian form of the constrained problem.) This problem may be solved using quadratic programming or more general convex optimization methods, as well as by specific algorithms such as the least angle regression algorithm. The L1-regularized formulation is useful in some contexts due to its tendency to prefer solutions with fewer nonzero parameter values, effectively reducing the number of variables upon which the given solution is dependent.[11] For this reason, the LASSO and its variants are fundamental to the field of compressed sensing.&lt;/p&gt;&#xA;" OwnerUserId="1808" LastActivityDate="2010-11-18T09:46:57.010" CommentCount="4" />
  <row Id="4667" PostTypeId="1" CreationDate="2010-11-18T09:59:09.933" Score="4" ViewCount="98" Body="&lt;p&gt;Through a project I am now working on (which I won't link to so to not have this an ad question), I came to realize how difficult it is to find R resources, not in English.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus my question is - what resources do you know of, recommend, for learning R in non-English languages?  (tutorials, blogs, wiki's, forums, and so on)&lt;/p&gt;&#xA;" OwnerUserId="253" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-11-18T10:47:39.200" LastActivityDate="2010-11-18T12:42:26.417" CommunityOwnedDate="2010-11-18T10:31:45.933" Title="R resources in non-English languages" Tags="&lt;r&gt;&lt;big-list&gt;&lt;tutorial&gt;&lt;blog&gt;&lt;resources&gt;" AnswerCount="5" CommentCount="4" />
  <row Id="4668" PostTypeId="1" CreationDate="2010-11-18T10:03:40.477" Score="1" ViewCount="90" Body="&lt;p&gt;my question is “what is mutual information ?(i understood that it will get  some idea about linear as well as nonlinear dependencies) but, how can i interpret it in the plot (ie.,) how can i differentiate the nonlinear from the linear dependencies?&lt;/p&gt;&#xA;" LastActivityDate="2010-11-18T10:03:40.477" ClosedDate="2010-11-23T23:30:20.173" Title="non-linear time series" Tags="&lt;correlation&gt;" CommentCount="7" FavoriteCount="1" />
  <row Id="4669" PostTypeId="2" ParentId="4667" CreationDate="2010-11-18T10:10:55.083" Score="3" ViewCount="" Body="&lt;p&gt;All RSS feeds I follow are in English actually, so I'll just point to tutorials available in French, or made by French researchers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Apart from the &lt;a href=&quot;http://cran.r-project.org/other-docs.html&quot; rel=&quot;nofollow&quot;&gt;Contributed Documentation&lt;/a&gt; on CRAN, I often browse the R website hosted at the &lt;a href=&quot;http://pbil.univ-lyon1.fr/R/enseignement.html&quot; rel=&quot;nofollow&quot;&gt;bioinformatics lab&lt;/a&gt; in Lyon (France); it is mostly in French, but it also includes english material. I also like &lt;a href=&quot;http://www.math.univ-toulouse.fr/~besse/enseignement.html&quot; rel=&quot;nofollow&quot;&gt;Philippe Besse&lt;/a&gt; resources (SAS + R).&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-11-18T10:10:55.083" CommunityOwnedDate="2010-11-18T10:10:55.083" />
  <row Id="4670" PostTypeId="2" ParentId="4667" CreationDate="2010-11-18T10:36:54.880" Score="4" ViewCount="" Body="&lt;p&gt;There doesn't appear to be much in Russian, but here is a couple of links:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://herba.msu.ru/shipunov/software/r/r-ru.htm&quot; rel=&quot;nofollow&quot;&gt;http://herba.msu.ru/shipunov/software/r/r-ru.htm&lt;/a&gt; contains pointers to a number of Russian-language R resources;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://voliadis.ru/taxonomy/term/18&quot; rel=&quot;nofollow&quot;&gt;http://voliadis.ru/taxonomy/term/18&lt;/a&gt; is a blog with some R content.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="439" LastActivityDate="2010-11-18T10:36:54.880" CommunityOwnedDate="2010-11-18T10:36:54.880" CommentCount="1" />
  <row Id="4671" PostTypeId="1" AcceptedAnswerId="4673" CreationDate="2010-11-18T10:46:34.230" Score="2" ViewCount="195" Body="&lt;p&gt;I am using an automatic model selection procedure, &quot;step&quot;. The model of depart (the biggest possible) is a polynom, say of the 4th degree.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Depart&amp;lt;-lm(y~x+I(x^2)+I(x^3)+I(x^4))&#xA;&#xA;Final&amp;lt;-step(Depart)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I need to transform the Final model to a corresponding function. How can i do this?&lt;/p&gt;&#xA;" OwnerUserId="2043" LastActivityDate="2010-11-18T12:00:38.503" Title="R: How to create a function from a model?" Tags="&lt;r&gt;&lt;stepwise-regression&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="4672" PostTypeId="2" ParentId="4667" CreationDate="2010-11-18T10:57:43.987" Score="5" ViewCount="" Body="&lt;p&gt;In german: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.wiwi.uni-bielefeld.de/%7Ewolf/skripten/Rkurs.pdf&quot; rel=&quot;nofollow&quot;&gt;A short introduction to R&lt;/a&gt; very short, covers only the basics of R programming&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://de.wikibooks.org/wiki/GNU_R&quot; rel=&quot;nofollow&quot;&gt;http://de.wikibooks.org/wiki/GNU_R&lt;/a&gt; teaches the basics of R programmming in detail and also contains some examples of producing graphics and statistics.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://cran.r-project.org/doc/contrib/Sawitzki-Einfuehrung.pdf&quot; rel=&quot;nofollow&quot;&gt;cran.r-project.org/doc/contrib/Sawitzki-Einfuehrung.pdf&lt;/a&gt; a lengthy introduction into statistics with R with a smaller focus on programming.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="264" LastActivityDate="2010-11-18T10:57:43.987" CommunityOwnedDate="2010-11-18T10:57:43.987" />
  <row Id="4673" PostTypeId="2" ParentId="4671" CreationDate="2010-11-18T12:00:38.503" Score="7" ViewCount="" Body="&lt;p&gt;Do you mean something like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;f&amp;lt;-function(newdata)predict(Final,data.frame(x=newdata))&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;?&lt;/p&gt;&#xA;" OwnerUserId="439" LastActivityDate="2010-11-18T12:00:38.503" CommentCount="4" />
  <row Id="4674" PostTypeId="2" ParentId="4667" CreationDate="2010-11-18T12:14:32.640" Score="3" ViewCount="" Body="&lt;p&gt;Here is a german blog with some posts on R:&#xA;&lt;a href=&quot;http://blog.berndweiss.net/tag/r/&quot; rel=&quot;nofollow&quot;&gt;http://blog.berndweiss.net/tag/r/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Recently started, with no posts on R yet, but focused on open data, is this blog:&#xA;&lt;a href=&quot;http://blog.zeit.de/open-data&quot; rel=&quot;nofollow&quot;&gt;http://blog.zeit.de/open-data&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="573" LastActivityDate="2010-11-18T12:14:32.640" CommunityOwnedDate="2010-11-18T12:14:32.640" />
  <row Id="4675" PostTypeId="2" ParentId="4667" CreationDate="2010-11-18T12:42:26.417" Score="4" ViewCount="" Body="&lt;p&gt;Some german blog entries:&#xA;&lt;a href=&quot;http://www.schockwellenreiter.de/blog/tag/r/&quot; rel=&quot;nofollow&quot;&gt;http://www.schockwellenreiter.de/blog/tag/r/&lt;/a&gt;&#xA;and&#xA;&lt;a href=&quot;http://markheckmann.wordpress.com/category/r-r-code/&quot; rel=&quot;nofollow&quot;&gt;http://markheckmann.wordpress.com/category/r-r-code/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1050" LastActivityDate="2010-11-18T12:42:26.417" CommunityOwnedDate="2010-11-18T12:42:26.417" />
  <row Id="4676" PostTypeId="2" ParentId="4619" CreationDate="2010-11-18T13:06:58.253" Score="2" ViewCount="" Body="&lt;p&gt;To add to chl's answer, another step you can take to ensure your data is representative of the population as a whole is to compare both samples to a third party data set. In the United States, there is the &lt;a href=&quot;http://factfinder.census.gov/servlet/DatasetMainPageServlet?_program=ACS&amp;amp;_submenuId=datasets_2&amp;amp;_lang=en&quot; rel=&quot;nofollow&quot;&gt;American Community Survey&lt;/a&gt; which I often use to compare the data I work with to the population for a given region. The other thing to consider is whether or not your data should match up with ACS data (or whatever you are comparing to). For example, the data I work with often relates to estimating travel demand and so I am interested only in the &quot;traveling&quot; population for a region. The ACS data samples from a broader range than the &quot;traveling&quot; population, so some differences may be expected.&lt;/p&gt;&#xA;" OwnerUserId="696" LastActivityDate="2010-11-18T13:06:58.253" CommentCount="1" />
  <row Id="4677" PostTypeId="2" ParentId="4612" CreationDate="2010-11-18T13:14:58.757" Score="1" ViewCount="" Body="&lt;p&gt;I also like the &quot;Mostly Harmless Econometrics: An Empiricist's Companion&quot;. &lt;/p&gt;&#xA;" OwnerUserId="1966" LastActivityDate="2010-11-18T13:14:58.757" CommunityOwnedDate="2010-11-18T13:14:58.757" />
  <row Id="4678" PostTypeId="2" ParentId="4454" CreationDate="2010-11-18T13:20:41.933" Score="3" ViewCount="" Body="&lt;p&gt;As you and Matt Parker both noted, there can be a big difference in the preparation of a survey script that is digested by your client and how you prepare the script for your programmers. In a professional setting, &quot;client&quot; friendly scripts generally win focus and the programmers are left to put the pieces together as they go along. However, there are some things you can do to make your programmer happy. The scripts we generally develop are Word based so my tips are related to Word, but should be fairly transferable to other platforms. A few tips or suggestions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Include &quot;preamble&quot; for your script that outlines what is conveyed in the document. This may include question text, question answers, branching logic, programmer notes, etc. How you differentiate that in the document is a style preference, but for example all of our notes to the programmer are in &amp;lt;&gt; tags and reviewers are instructed to ignore them for their purposes.&lt;/li&gt;&#xA;&lt;li&gt;To the extent you can, try and list any intermediate variables or calculations that are needed for future questions. &lt;/li&gt;&#xA;&lt;li&gt;Refer to questions by some identifiable name and not something generic such as Q3, Q7, Q19. Not only does this prevent you from having to change a bunch of references downstream when you decide to move or add a question, but is just generally more meaningful to say something like &quot;Branch over this question if purpose is vacation&quot; then &quot;Branch over this question if Q4.A is XX&quot;. &lt;/li&gt;&#xA;&lt;li&gt;Try and include the programmer in the questionnaire design phase. If this is a complicated survey, having the programmer on board as a sounding board for different techniques may open your eyes to something you would have otherwise overlooked.&lt;/li&gt;&#xA;&lt;li&gt;Finally, tell the programmer up front they have the liberty to change flow / etc as it makes sense in a programming context. As long as they are able to develop the final product as the client envisions it, any intermediate changes they make to what they program will come out in the wash.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="696" LastActivityDate="2010-11-18T13:20:41.933" />
  <row Id="4679" PostTypeId="2" ParentId="4659" CreationDate="2010-11-18T14:03:15.770" Score="3" ViewCount="" Body="&lt;p&gt;Look at the pdf of Binomial as a function of $x$:  $$f(x) = {n\choose{x}}p^{x}(1-p)^{n-x}$$ and the pdf of Beta as a function of $p$: $$g(p)=\frac{\Gamma(a+b)}{\Gamma(a)\Gamma(b)}p^{a-1}(1-p)^{b-1}$$  &#xA;You probably can see that with an appropriate (integer) choice for $a$ and $b$ these are the same. As far as I can tell, that's all there is to this relationship: the way $p$ enters into the binomial pdf just happens to be called a Beta distribution. &lt;/p&gt;&#xA;" OwnerUserId="279" LastActivityDate="2010-11-18T14:03:15.770" CommentCount="5" />
  <row Id="4680" PostTypeId="2" ParentId="4454" CreationDate="2010-11-18T14:18:18.587" Score="1" ViewCount="" Body="&lt;p&gt;I wrote a post some time ago about how to use Google spreadsheets + google forms + R for easily collecting and sharing data.  It might prove useful to you or others:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.r-statistics.com/2010/03/google-spreadsheets-google-forms-r-easily-collecting-and-importing-data-for-analysis/&quot; rel=&quot;nofollow&quot;&gt;http://www.r-statistics.com/2010/03/google-spreadsheets-google-forms-r-easily-collecting-and-importing-data-for-analysis/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="253" LastActivityDate="2010-11-18T14:18:18.587" />
  <row Id="4681" PostTypeId="2" ParentId="2988" CreationDate="2010-11-18T14:47:29.653" Score="3" ViewCount="" Body="&lt;p&gt;Following on from the post by Stephan Kolassa (I can't add this as a comment), I have some alternative code for a simulation.  This uses the same basic structure, but is exploded a bit more, so perhaps it is a little easier to read.  It also is based on the code by &lt;a href=&quot;http://sas-and-r.blogspot.com/2009/06/example-72-simulate-data-from-logistic.html&quot; rel=&quot;nofollow&quot;&gt;Kleinman and Horton&lt;/a&gt; to simulate the logistic regression.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;nn is the number in the sample.  The covariate should be continuously normally distributed, and standardized to mean 0 and sd 1.  We use rnorm(nn) to generate this.  We select an odds ratio and store it in odds.ratio.  We also pick a number for the intercept.  Choice of this number governs what proportion of the sample experience the &quot;event&quot; (e.g. 0.1, 0.4, 0.5).  You have to play around with this number until you get the right proportion.  The following code gives you a proportion of 0.1 with a sample size of 950 and an OR of 1.5:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;nn &amp;lt;- 950&#xA;runs &amp;lt;- 10000&#xA;intercept &amp;lt;- log(9)&#xA;odds.ratio &amp;lt;- 1.5&#xA;beta &amp;lt;- log(odds.ratio)&#xA;proportion  &amp;lt;-  replicate(&#xA;              n = runs,&#xA;              expr = {&#xA;                  xtest &amp;lt;- rnorm(nn)&#xA;                  linpred &amp;lt;- intercept + (xtest * beta)&#xA;                  prob &amp;lt;- exp(linpred)/(1 + exp(linpred))&#xA;                  runis &amp;lt;- runif(length(xtest),0,1)&#xA;                  ytest &amp;lt;- ifelse(runis &amp;lt; prob,1,0)&#xA;                  prop &amp;lt;- length(which(ytest &amp;lt;= 0.5))/length(ytest)&#xA;                  }&#xA;            )&#xA;summary(proportion)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;summary(proportion) confirms that the proportion is ~ 0.1&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then using the same variables, the power is calculated over 10000 runs:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;result &amp;lt;-  replicate(&#xA;              n = runs,&#xA;              expr = {&#xA;                  xtest &amp;lt;- rnorm(nn)&#xA;                  linpred &amp;lt;- intercept + (xtest * beta)&#xA;                  prob &amp;lt;- exp(linpred)/(1 + exp(linpred))&#xA;                  runis &amp;lt;- runif(length(xtest),0,1)&#xA;                  ytest &amp;lt;- ifelse(runis &amp;lt; prob,1,0)&#xA;                  summary(model &amp;lt;- glm(ytest ~ xtest,  family = &quot;binomial&quot;))$coefficients[2,4] &amp;lt; .05&#xA;                  }&#xA;            )&#xA;print(sum(result)/runs)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I think that this code is correct - I checked it against the examples given in Hsieh, 1998 (table 2), and it seems to agree with the three examples given there.  I also tested it against the example on p 342 - 343 of Hosmer and Lemeshow, where it found a power of 0.75 (compared to 0.8 in Hosmer and Lemeshow).  So it may be that in some circumstances this approach underestimates power.  However, when I've run the same example in this &lt;a href=&quot;http://biostat.hitchcock.org/MeasurementError/Analytics/PowerCalculationsforLogisticRegression.asp&quot; rel=&quot;nofollow&quot;&gt;on-line calculator&lt;/a&gt;, I've found that it agrees with me and not the result in Hosmer and Lemeshow.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If anyone can tell us why this is the case, I'd be interested to know.&lt;/p&gt;&#xA;" OwnerUserId="1991" LastEditorUserId="1991" LastEditorDisplayName="" LastEditDate="2010-12-03T11:40:53.517" LastActivityDate="2010-12-03T11:40:53.517" CommentCount="3" />
  <row Id="4684" PostTypeId="2" ParentId="4659" CreationDate="2010-11-18T15:51:22.313" Score="5" ViewCount="" Body="&lt;p&gt;Consider the order statistics $x_{[0]} \le x_{[1]} \le \cdots \le x_{[n]}$ of $n+1$ independent draws from a uniform distribution.  Because &lt;a href=&quot;http://en.wikipedia.org/wiki/Order_statistic#The_order_statistics_of_the_uniform_distribution&quot; rel=&quot;nofollow&quot;&gt;order statistics have Beta distributions&lt;/a&gt;, the chance that $x_{[k]}$ does not exceed $p$ is given by the Beta integral&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\Pr[x_{[k]} \le p] = \frac{1}{B(k+1, n-k+1)} \int_0^p{x^k(1-x)^{n-k}dx}.$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Why is this?  Here is a non-rigorous but memorable demonstration.  The chance that $x_{[k]}$ lies between $p$ and $p + dp$ is the chance that out of $n+1$ uniform values, $k$ of them lie between $0$ and $p$, at least one of them lies between $p$ and $p + dp$, and the remainder lie between $p + dp$ and $1$.  To first order in the infinitesimal $dp$ we only need to consider the case where exactly one value (namely, $x_{[k]}$ itself) lies between $p$ and $p + dp$ and therefore $n - k$ values exceed $p + dp$.  Because all values are independent and uniform, this probability is proportional to $p^k (dp) (1 - p - dp)^{n-k}$.  To first order in $dp$ this equals $p^k(1-p)^{n-k}dp$, precisely the integrand of the Beta distribution.  The term $\frac{1}{B(k+1, n-k+1)}$ can be computed directly from this argument as the multinomial coefficient ${n+1}\choose{k,1, n-k}$ or derived indirectly as the normalizing constant of the integral.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By definition, the event $x_{[k]} \le p$ is that the $k+1^\text{st}$ value does not exceed $p$.  Equivalently, &lt;em&gt;at least&lt;/em&gt; $k+1$ of the values do not exceed $p$: this simple (and I hope obvious) assertion provides the intuition you seek. The probability of the equivalent statement is given by the Binomial distribution,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\Pr[\text{at least }k+1\text{ of the }x_i \le p] = \sum_{j=k+1}^{n+1}{{n+1}\choose{j}} p^j (1-p)^{n+1-j}.$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;In summary&lt;/strong&gt;, the Beta integral breaks the calculation of an event into a series of calculations: finding at least $k+1$ values in the range $[0, p]$, whose probability we normally would compute with a Binomial cdf, is broken down into mutually exclusive cases where &lt;em&gt;exactly&lt;/em&gt; $k$ values are in the range $[0, x]$ and 1 value is in the range $[x, x+dx]$ for all possible $x$, $0 \le x \lt p$, and $dx$ is an infinitesimal length.  Summing over all such &quot;windows&quot; $[x, x+dx]$--that is, integrating--must give the same probability as the Binomial cdf.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/EcpL3.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-11-18T23:03:39.207" LastActivityDate="2010-11-18T23:03:39.207" CommentCount="10" />
  <row Id="4685" PostTypeId="1" AcceptedAnswerId="4728" CreationDate="2010-11-18T16:19:43.160" Score="0" ViewCount="204" Body="&lt;p&gt;What is the difference between soft and hard expectation maximization?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: ok, i've found out this paper: &lt;a href=&quot;http://ttic.uchicago.edu/~dmcallester/ttic101-07/lectures/em/em.pdf&quot; rel=&quot;nofollow&quot;&gt;http://ttic.uchicago.edu/~dmcallester/ttic101-07/lectures/em/em.pdf&lt;/a&gt;&#xA;that explain quite well the situations&lt;/p&gt;&#xA;" OwnerUserId="2046" LastEditorUserId="2046" LastEditorDisplayName="" LastEditDate="2010-11-19T11:20:52.973" LastActivityDate="2010-11-19T14:31:20.210" Title="Soft and Hard EM (Expectation Maximization)" Tags="&lt;fitting&gt;&lt;em-algorithm&gt;&lt;unsupervised-learning&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="4686" PostTypeId="1" CreationDate="2010-11-18T16:48:34.630" Score="5" ViewCount="151" Body="&lt;p&gt;Suppose that I have a population, each represented by a bit $b_i$ for $i \in \{1,\ldots, n\}$. I would like to compute an estimate $\hat{B}$ of the parameter $B = \sum_{i=1}^nb_i$ so that with high probability, the error $|\hat{B}-B| \leq k$ for some fixed $k$. However, I have to pay a cost $c_i$ to sample bit $b_i$, and this cost may be different for each $i$. I want to find the minimum-cost sample that satisfies my accuracy constraint. Clearly, uniform sampling is not necessarily optimal. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Has this been studied? Is there a known optimal solution specifying the probability $p_i$ that I should sample each bit $b_i$ to compute $\hat{B}$?  &lt;/p&gt;&#xA;" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-11-18T23:06:13.260" LastActivityDate="2010-11-22T15:43:17.257" Title="Sampling with non-uniform costs" Tags="&lt;sample-size&gt;&lt;sampling&gt;" AnswerCount="3" CommentCount="4" />
  <row Id="4687" PostTypeId="1" AcceptedAnswerId="5693" CreationDate="2010-11-18T17:32:43.533" Score="3" ViewCount="261" Body="&lt;p&gt;I am trying to get a deeper understanding of the various types of Bayesian networks. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most of the literature/lectures I've come across use discrete random variables exclusively and only mention continuous random variables in passing.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems if you want to mix discrete and continuous variables in a hybrid network, then you have a few different scenarios to handle:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;Child Is       Parent(s) Are&#xA;----------------------------------------&#xA;Discrete       All Discrete&#xA;Discrete       All Continuous&#xA;Discrete       Hybrid (Discrete and Continuous)&#xA;Continuous     All Discrete&#xA;Continuous     All Continuous&#xA;Continuous     Hybrid (Discrete and Continuous)&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I understand how to specify conditional probabilities (i.e. $p(N=n_i|P)$)  for the cases where the parents (P) are all discrete.  The conditional probabilities are essentially a lookup table that maps a condition to either a probability mass function (when child is discrete) or probability density function (when child is continuous).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The other cases are numbing my brain.  How does one go about specifying conditional probabilities of a given node when any of its parents are continuous?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit&lt;/strong&gt; Terminology has been (hopefully) corrected based on feedback.&lt;/p&gt;&#xA;" OwnerUserId="1474" LastEditorUserId="1474" LastEditorDisplayName="" LastEditDate="2010-12-22T17:15:48.067" LastActivityDate="2010-12-22T18:30:25.820" Title="Specifying conditional probabilities in hybrid Bayesian networks" Tags="&lt;bayesian&gt;&lt;random-variable&gt;&lt;graphical-model&gt;&lt;conditional-probability&gt;&lt;prior&gt;" AnswerCount="2" CommentCount="3" />
  <row Id="4688" PostTypeId="2" ParentId="4603" CreationDate="2010-11-18T17:39:55.700" Score="1" ViewCount="" Body="&lt;p&gt;I've just come across &lt;a href=&quot;https://mlpy.fbk.eu/&quot; rel=&quot;nofollow&quot;&gt;mlpy&lt;/a&gt;, which also has an implementation of the lasso (in Python.)&lt;/p&gt;&#xA;" OwnerUserId="439" LastActivityDate="2010-11-18T17:39:55.700" CommentCount="2" />
  <row Id="4689" PostTypeId="1" AcceptedAnswerId="4690" CreationDate="2010-11-18T18:16:48.990" Score="2" ViewCount="150" Body="&lt;p&gt;What are the differences between generative and discriminant models (in the context of Bayesian learning and inference)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and what it is concerned with prediction, decision theory or unsupervised learning?&lt;/p&gt;&#xA;" OwnerUserId="2046" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-11-19T10:05:43.087" LastActivityDate="2010-11-19T10:05:43.087" Title="Generative vs discriminant models" Tags="&lt;bayesian&gt;&lt;predictive-models&gt;&lt;unsupervised-learning&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="4690" PostTypeId="2" ParentId="4689" CreationDate="2010-11-18T19:01:08.403" Score="7" ViewCount="" Body="&lt;p&gt;Both are used in supervised learning where you want to learn a rule that maps input x to output y, given a number of training examples of the form {(x_i,y_i)}. A generative model (e.g., naive Bayes) explicitly models the joint probability distribution p(x,y) and then uses the Bayes rule to compute p(y|x). On the other hand, a discriminative model (e.g., logistic regression) directly models p(y|x).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some people argue that the discriminative model is better in the sense that it directly models the quantity you care about (y), so you don't have to spend your modeling efforts on the input x (you need to compute p(x|y) as well in a generative model). However, the generative model has its own advantages such as the capability of dealing with missing data, etc. For some comparison, you can take a look at this paper: &lt;a href=&quot;http://ai.stanford.edu/~ang/papers/nips01-discriminativegenerative.pdf&quot; rel=&quot;nofollow&quot;&gt;On Discriminative vs. Generative classifiers: A comparison of logistic regression and naive Bayes&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There can be cases when one model is better than the other (e.g., discriminative models usually tend to do better if you have lots of data; generative models may be better if you have some extra unlabeled data). In fact, there exists hybird models too that try to bring in the best of both worlds. See this paper for an example: &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.89.8245&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;Principled hybrids of generative and discriminative models&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="881" LastActivityDate="2010-11-18T19:01:08.403" CommentCount="1" />
  <row Id="4691" PostTypeId="1" CreationDate="2010-11-18T20:40:16.053" Score="2" ViewCount="365" Body="&lt;p&gt;My teacher is asking whether it is possible to look at the crohnbachs's alpha when looking at the internal reliability of an &lt;strong&gt;ordinal&lt;/strong&gt; scale. She thinks, because you use the mean of it, it is not possible, but I've seen it in previous research as well. I think it is possible, but I cannot  give a technical explanation for this. Who can help me?&lt;/p&gt;&#xA;" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-22T22:28:11.373" LastActivityDate="2010-11-22T22:28:11.373" Title="Internal reliability for an ordinal scale" Tags="&lt;psychometrics&gt;&lt;homework&gt;&lt;reliability&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="4692" PostTypeId="2" ParentId="2728" CreationDate="2010-11-18T20:58:25.817" Score="0" ViewCount="" Body="&lt;p&gt;I came across a package recently that is supposed to help with this and they reference using it with pam. I haven't tried it yet but it looks promising. Check out the vignette. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://cran.r-project.org/web/packages/clues/index.html&quot; rel=&quot;nofollow&quot;&gt;http://cran.r-project.org/web/packages/clues/index.html&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="2050" LastActivityDate="2010-11-18T20:58:25.817" />
  <row Id="4693" PostTypeId="2" ParentId="4686" CreationDate="2010-11-18T21:14:38.350" Score="0" ViewCount="" Body="&lt;p&gt;If the costs $c_i$ are known &lt;em&gt;a priori&lt;/em&gt;, it seems like a greedy sampling would give you some guarantees. That is, sample the $n-2k$ bits in order of increasing cost. This gives a $k$-error guarantee on $B$ with probability $1$ in the obvious way. I am curious if this strategy is the limit strategy of some sane sequence of strategies that provide a guarantee with probability $1-\epsilon$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the algorithm is to be deterministic, and the $c_i$ are set by an adversary, I do not think you can do better than this.&lt;/p&gt;&#xA;" OwnerUserId="795" LastEditorUserId="795" LastEditorDisplayName="" LastEditDate="2010-11-20T05:09:34.580" LastActivityDate="2010-11-20T05:09:34.580" CommentCount="11" />
  <row Id="4694" PostTypeId="1" AcceptedAnswerId="4705" CreationDate="2010-11-18T21:49:01.410" Score="2" ViewCount="144" Body="&lt;p&gt;&lt;a href=&quot;http://www.ambion.com/techlib/tn/95/954.html&quot; rel=&quot;nofollow&quot;&gt;Here&lt;/a&gt; is an example of hierarchical clustering of genes in the microarray data using the &lt;strong&gt;weighted pair gene method&lt;/strong&gt; in &lt;code&gt;Spotfire&lt;/code&gt;. I am not sure how to do this in &lt;code&gt;R&lt;/code&gt;. In the &lt;code&gt;hclust&lt;/code&gt; function, I see &lt;code&gt;ward&quot;, &quot;single&quot;, &quot;complete&quot;, &quot;average&quot;, &quot;mcquitty&quot;, &quot;median&quot; or &quot;centroid&quot;&lt;/code&gt; as the methods. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, lets say I have performed hierarchical clustering and found groups of genes using &lt;code&gt;cuttree&lt;/code&gt; method. I wanted to plot the expression of genes in a group across columns (which may represent treatment, time, etc.). And I want to do this for all the groups separately. In a way similar to the &lt;a href=&quot;http://www.bioconductor.org/packages/release/bioc/html/Mfuzz.html&quot; rel=&quot;nofollow&quot;&gt;Mfuzz&lt;/a&gt; package's way of showing clusters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can any one please help me?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;TIA for any pointers.&lt;/p&gt;&#xA;" OwnerUserId="1307" LastActivityDate="2010-11-19T07:43:58.327" Title="How to do weighted pair hierarchical clustering in R?" Tags="&lt;r&gt;&lt;clustering&gt;&lt;microarray&gt;" AnswerCount="1" FavoriteCount="2" />
  <row Id="4695" PostTypeId="1" CreationDate="2010-11-18T22:11:04.460" Score="4" ViewCount="128" Body="&lt;p&gt;Suppose I have time series observations from distributions drawn from some population. That is, I observe $X_{t,i}$ for $t=1,2,...,T,$ and $i=1,2,...,n$, where I believe that $X_{t,i}$ have pdf $f(\theta_i)$. (I have some idea about the distribution of the $\theta_i$, but that may not be important here.) I have some sample statistic which is a good estimator of $\theta_i$ given some observations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, there is the suspicion that, in fact, the $\theta_i$ are not stationary, rather the observations come from $f(\theta_{t,i})$, where the $\theta_{t,i}$ are changing slowly over time. How can I test this, either by a formal hypothesis test or an 'eyeball' test?  The amount of data available in the time domain is not so great (&lt;em&gt;i.e.&lt;/em&gt; $T$ is not so large), thus partitioning the time domain and computing the sample estimate on each partition would only be advisable for a small number (say 5) of partitions (because otherwise the standard error of the estimate is too great).  However, the number of series, $n$, is largeish, say 10,000.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I realize there are a number of gaps in this question, &lt;em&gt;e.g.&lt;/em&gt; how the $\theta_{t,i}$ might be varying with time, the standard error of the parameter estimator, etc. However, any hints would be appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To be concrete, one could think of the $X_{t,i}$ as being normally distributed with mean $\theta$ and standard deviation $1$, and the sample statistic is the sample mean.&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-11-21T04:25:37.277" Title="How to test for parameter stationarity?" Tags="&lt;time-series&gt;&lt;estimation&gt;&lt;stationarity&gt;" AnswerCount="2" />
  <row Id="4696" PostTypeId="2" ParentId="4691" CreationDate="2010-11-18T22:15:45.467" Score="3" ViewCount="" Body="&lt;p&gt;From a practical perspective, I don't see any obvious reason to not use Cronbach's alpha with ordinal items (e.g., Likert-type items), as is commonly done in most of the studies. It is a lower bound for reliability, and is essentially used as an indicator of internal consistency of a test or questionnaire. The usual assumptions pertaining to a correct interpretation of its value are as follows: (i) no residual correlations, (ii) items have identical loadings, and (iii) the scale is unidimensional. In fact, the sole case where alpha will be essentially the same as reliability is the case of uniformly high factor loadings, no error covariances, and unidimensional instrument (1).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, we can speak of an &lt;em&gt;ordinal reliability alpha&lt;/em&gt;. For instance, Zumbo et coll. (2) use a polychoric correlation matrix input to calculate alpha parallel to Cronbach. Their simulation studies lead them to conclude that ordinal reliability alpha provides &quot;consistently suitable estimates of the theoretical reliability, regardless of the magnitude of the theoretical reliability, the number of scale points, and the skewness of the scale point distributions. In contrast, coefficient alpha is in general a negatively biased&#xA;estimate of reliability&quot; for ordinal data (p. 21). Ordinal reliability alpha will normally be higher than the corresponding Cronbach’s alpha.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Otherwise, the usual Cronbach's $\alpha$ is influenced by the number of items in the test and interitem correlations (for a fixed sample size $N=300$, even with modest--albeit perfect--correlation between items, e.g. $\rho = 0.35$, Cronbach’s $\alpha$ would still be at 0.943 with 30 items, and 0.910 with 20 items). There're subtle issues with Cronbach's $\alpha$ and departure from the unidimensionality assumption (systematic errors can greatly inflate the estimate of alpha, especially with large sample sizes) or the presence of inconsistent responses (random responses may inflate Cronbach’s alpha when their mean differ from that of the true responses). If the variables being tested are all dichotomous, Cronbach’s alpha is the same as Kuder-Richardson coefficient (3).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of note, there are alternative ways to estimate the reliability of test scores, see e.g., Zinbarg et al. (4).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A good review is &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Bruce Thompson. &lt;em&gt;Score Reliability.&#xA;  Contemporary Thinking on Reliability&#xA;  issues&lt;/em&gt;. Sage Publications, 2003.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;References&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;T Raykov. &lt;a href=&quot;http://www.eric.ed.gov:80/ERICWebPortal/search/detailmini.jsp?_nfpb=true&amp;amp;_&amp;amp;ERICExtSearch_SearchValue_0=EJ568168&amp;amp;ERICExtSearch_SearchType_0=no&amp;amp;accno=EJ568168&quot; rel=&quot;nofollow&quot;&gt;Scale reliability, Cronbach’s coefficient alpha, and violations of essential tau-equivalence for fixed congeneric components&lt;/a&gt;. &lt;em&gt;Multivariate Behavioral Research&lt;/em&gt;, &lt;strong&gt;32&lt;/strong&gt;: 329-254, 1997.&lt;/li&gt;&#xA;&lt;li&gt;B D Zumbo, A M Gadermann, and C Zeisser. &lt;a href=&quot;http://educ.ubc.ca/faculty/zumbo/papers/ordinal_alpha_reprint.pdf&quot; rel=&quot;nofollow&quot;&gt;Ordinal versions of coefficients alpha and theta for likert rating scales&lt;/a&gt;. &lt;em&gt;Journal of Modern Applied Statistical Methods&lt;/em&gt;, &lt;strong&gt;6&lt;/strong&gt;: 21-29, 2007.&lt;/li&gt;&#xA;&lt;li&gt;G F Kuder and M W Richardson. &lt;a href=&quot;http://www.springerlink.com/content/8jw2386883q22513/&quot; rel=&quot;nofollow&quot;&gt;The theory of the estimation of test reliability&lt;/a&gt;. &lt;em&gt;Psychometrika&lt;/em&gt;, &lt;strong&gt;2&lt;/strong&gt;: 151-160, 1937.&lt;/li&gt;&#xA;&lt;li&gt;R E Zinbarg, W Revelle, I Yovel, and W Li. &lt;a href=&quot;http://personality-project.org/revelle/publications/zinbarg.revelle.pmet.05.pdf&quot; rel=&quot;nofollow&quot;&gt;Cronbach’s $\alpha$, Revelle’s $\beta$, and McDonald’s $\omega_h$: Their relations with each other and two alternative conceptualizations of reliability&lt;/a&gt;. &lt;em&gt;Psychometrika&lt;/em&gt;, &lt;strong&gt;70(1)&lt;/strong&gt;: 123-133, 2005.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-18T22:28:43.780" LastActivityDate="2010-11-18T22:28:43.780" />
  <row Id="4697" PostTypeId="2" ParentId="4686" CreationDate="2010-11-18T22:38:15.733" Score="4" ViewCount="" Body="&lt;p&gt;Methods to find a solution are well known, but this is a messy problem.  A tiny example reveals much, so consider the case $n = 2$.  Let the cost of sampling bit 1 be $c_1 = 1$ and the cost of sampling bit 2 be $c_2 = c$.  Without any loss of generality assume this is the expensive bit: $c \ge 1$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Either we sample both bits at a cost of $1 + c$ because we have to in order to keep the error low, or else we will sample bit 2 with probability $\pi$ and bit 1 with probability $1 - \pi$.  Let's assume the value of $k$ is large enough that we won't be compelled to sample both bits.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An unbiased estimator is $\hat{B} = b_1 / (1 - \pi)$ if we sample bit 1 and $\hat{B} = b_2 / \pi$ if we sample bit 2.  (This is the &lt;a href=&quot;http://www.amstat.org/sections/srms/Proceedings/papers/1988_082.pdf&quot; rel=&quot;nofollow&quot;&gt;Horvitz-Thompson estimator&lt;/a&gt;.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The error rate depends on the state of the population&lt;/strong&gt;.  I interpret the problem to require that the &lt;em&gt;expected&lt;/em&gt; error size be assured of not exceeding the limit $k$ *no matter what the state of the population may be.*  We cannot remove the word &quot;expected&quot; here, because (except for nearly exhaustive samples), the maximum error size can be arbitrarily close to 1 for large populations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are $2^2 = 4$ possible states, which can be fully enumerated in this small problem:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\eqalign{&#xD;&#xA;\text{Prob.} &amp;amp;b_1 &amp;amp;b_2 &amp;amp;B &amp;amp;\text{Observation} &amp;amp;\hat{B} &amp;amp;\text{Error} \cr&#xD;&#xA;1 - \pi      &amp;amp;0   &amp;amp;0   &amp;amp;0 &amp;amp;0                  &amp;amp;0       &amp;amp;0\cr&#xD;&#xA;\pi          &amp;amp;0   &amp;amp;0   &amp;amp;0 &amp;amp;0                  &amp;amp;0       &amp;amp;0\cr&#xD;&#xA;1 - \pi      &amp;amp;0   &amp;amp;1   &amp;amp;1 &amp;amp;0                  &amp;amp;0       &amp;amp;-1\cr&#xD;&#xA;\pi          &amp;amp;0   &amp;amp;1   &amp;amp;1 &amp;amp;1                  &amp;amp;1/\pi   &amp;amp;1/\pi - 1\cr&#xD;&#xA;1 - \pi      &amp;amp;1   &amp;amp;0   &amp;amp;1 &amp;amp;1                  &amp;amp;1/(1-\pi) &amp;amp;1/(1-\pi) - 1\cr&#xD;&#xA;\pi          &amp;amp;1   &amp;amp;0   &amp;amp;1 &amp;amp;0                  &amp;amp;0       &amp;amp;-1\cr&#xD;&#xA;1 - \pi      &amp;amp;1   &amp;amp;1   &amp;amp;2 &amp;amp;1                  &amp;amp;1/(1-\pi) &amp;amp;1/(1-\pi) - 2\cr&#xD;&#xA;\pi          &amp;amp;1   &amp;amp;1   &amp;amp;2 &amp;amp;1                  &amp;amp;1/\pi   &amp;amp;1/\pi - 2&#xD;&#xA;}$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Taking expectations for each possible state $(b_1, b_2)$ condenses this into the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\eqalign{&#xD;&#xA; b_1 &amp;amp;b_2 &amp;amp;\text{Error distribution} &amp;amp;\mathbb{E}[|\text{Error}|]\cr&#xD;&#xA; 0   &amp;amp;0   &amp;amp;(0, 0)                    &amp;amp;0\cr&#xD;&#xA; 0   &amp;amp;1   &amp;amp;(-1, 1/\pi-1)             &amp;amp;2(1 - \pi)\cr&#xD;&#xA; 1   &amp;amp;0   &amp;amp;(1/(1-\pi)-1, -1)         &amp;amp;2\pi \cr&#xD;&#xA; 1   &amp;amp;1   &amp;amp;(1/(1-\pi) - 2, 1/\pi - 2) &amp;amp;2 - 4\pi&#xD;&#xA;}$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In computing the expected absolute error I have assumed $\pi \le 1/2$: we will favor sampling the cheaper bit whenever possible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose, for example, $k = 3/2$.  That is, we aim to find a sampling scheme that keeps the absolute error to $3/2$ or less with &quot;high probability&quot; while minimizing the &lt;em&gt;expected&lt;/em&gt; cost.  (I realize this choice of $k$ is artificial because we might attempt to improve the estimator--at risk of biasing it slightly--by constraining its estimates to 0, 1, or 2; but the purpose here is to look ahead to a situation with large $n$, where such improvements will be unlikely.  The &lt;em&gt;mathematical patterns&lt;/em&gt; are important in this example, not its (lack of) realism.)  Evidently we would like to minimize the chance of paying for the expensive bit; that is, to make $\pi$ as small as possible.  The final column in the previous table constrains $\pi$; it implies that &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$2(1-\pi) \le k,\quad 2\pi \le k,\quad 2 - 4\pi \le k.$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For $k \ge 1$ all constraints can be satisfied provided&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\max(1-k/2, 1/2 - k/4) \le \pi \le k/2.$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because the expected cost is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\mathbb{E}[\text{Cost}] = 1 + (c-1)\pi,$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;the unique cost-minimizing solution for $k=3/2$ is $\pi = 1/4$: &lt;em&gt;regardless&lt;/em&gt; of the differences in expenses, we should sample the cheap bit with probability $3/4$ and the expensive bit with probability $1/4$, for an expected cost of $1 + (c-1)/4$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This example reveals many things, including&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;There can be solutions cheaper than simple random sampling&lt;/strong&gt; (which in this case would select each bit with probability $1/2$ for an expected cost of $1 + (c-1)/2$).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Finding a solution involves an optimization with an exponential number of (increasingly complicated) constraints in $n$.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The selection probabilities will depend on the value of $k$.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;We cannot guarantee a fixed cost&lt;/strong&gt;; all we can hope for--because randomization is essential--is an optimal &lt;em&gt;expected&lt;/em&gt; cost.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;As always, the optimal sample size will depend on $k$ (the limit on the amount of error).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;As a practical matter, I think most people would have more information than contained in this abstract problem.  Even if they didn't, if $n$ were large and a substantial sample size were contemplated, it would make sense to devote part of the sampling budget to the purpose of modeling a relationship between the costs and the values (the $c_i$ and the $b_i$).  With such a model in hand one could greatly simplify the analysis and identify an optimal or near-optimal program to spend the remaining sampling budget (or even, in some cases, to establish that the targeted error rate is unlikely to be achieved).  For this reason, and because the exponential growth in the constraints is troublesome, I am reluctant to pursue a more detailed analysis of this problem.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-11-18T22:44:36.383" LastActivityDate="2010-11-18T22:44:36.383" CommentCount="2" />
  <row Id="4698" PostTypeId="1" AcceptedAnswerId="5024" CreationDate="2010-11-18T22:54:30.957" Score="2" ViewCount="115" Body="&lt;p&gt;I have two models $M_1$ and $M_2$ that I am using to try and compare to observed data $D$. $M_1$ is an $n_1$-dimensional model, and $M_2$ is an $n_2$-dimensional problem. The Bayes factor $K$ to compare the models can be calculated using:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$K = P(D|M_1)/P(D|M_2) $&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;assuming no prior preference for either model. The numerator and denominator can be written as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(D|M_i) = \int P(D|\mathbf{w},M_i) P(\textbf{w}|M_i) d\mathbf{w}$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\mathbf{w}$ is the parameter vector, so the integral is over parameter space.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now say that due to e.g. computational constraints, one can only compute $M_1$ and $M_2$ for a finite number of random samples of the parameter vector $\mathbf{w}$, where the number of samples is given by $s_1$ and $s_2$. Would it be acceptable to then say that the integral above becomes a summation over the random samples, and assuming the random samples are uniformly distributed through parameter space, $P(\textbf{w}|M_i)$ becomes $1/s_i$, so that:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(D|M_i) = \sum_{j=1}^{s_i} P(D|\mathbf{w_j},M_i) / s_i$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and so what is being compared in the Bayes factor $K$ is the ratio of the average probability over all the samples for each model? &lt;/p&gt;&#xA;" OwnerUserId="2052" LastActivityDate="2010-11-30T09:33:50.900" Title="Bayesian model comparison for randomly sampled sets of models" Tags="&lt;probability&gt;&lt;bayesian&gt;&lt;modeling&gt;" AnswerCount="2" CommentCount="0" />
  <row Id="4699" PostTypeId="2" ParentId="4695" CreationDate="2010-11-18T23:18:18.273" Score="2" ViewCount="" Body="&lt;p&gt;This problem is encountered in quality control/&lt;a href=&quot;http://en.wikipedia.org/wiki/Statistical_process_control&quot; rel=&quot;nofollow&quot;&gt;statistical process control&lt;/a&gt; settings.  There's a large literature, as you have hinted, because different parameters as estimated in various ways from different forms of sampling different distributions can be expected to vary in different ways.  The purpose is to detect that variation on-line as soon as possible after it occurs without triggering too many false detections along the way.  Consider using a control chart (&lt;a href=&quot;http://p://en.wikipedia.org/wiki/Control_chart&quot;&gt;1&lt;/a&gt;, &lt;a href=&quot;http://www.itl.nist.gov/div898/handbook/pmc/section3/pmc31.htm&quot; rel=&quot;nofollow&quot;&gt;2&lt;/a&gt;).  In your concrete situation a good choice is a combined Shewhart-CUSUM control chart.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-11-18T23:18:18.273" CommentCount="1" />
  <row Id="4700" PostTypeId="1" AcceptedAnswerId="4702" CreationDate="2010-11-19T00:03:28.163" Score="12" ViewCount="875" Body="&lt;p&gt;In simple terms, how would you explain (perhaps with simple examples) the difference between fixed effect, random effect and mixed effect models? &lt;/p&gt;&#xA;" OwnerUserId="1991" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-19T07:58:26.983" LastActivityDate="2010-11-21T18:00:46.553" Title="What is the difference between fixed effect, random effect and mixed effect models?" Tags="&lt;mixed-model&gt;&lt;random-effects-model&gt;&lt;definition&gt;&lt;fixed-effects-model&gt;" AnswerCount="5" FavoriteCount="2" />
  <row Id="4701" PostTypeId="2" ParentId="4659" CreationDate="2010-11-19T01:22:33.283" Score="1" ViewCount="" Body="&lt;p&gt;As you noted, the Beta distribution describes the distribution of the trial probability parameter $F$, while the binomial distribution describes the distribution of the outcome parameter $I$. Rewriting your question, what you asked about was why &#xA;$$P(F \le \frac {i+1} n)+P(I \le fn-1)=1$$&#xD;&#xA;$$P(Fn \le i+1)+P(I+1 \le fn)=1$$&#xD;&#xA;$$P(Fn \le i+1)=P(fn&amp;lt;I+1)$$&#xD;&#xA;That is, the likelihood that the observation plus one is greater than the expectation of the observation is the same as the likelihood that the observation plus one is greater than the expectation of the observation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I admit that this may not help intuit the original formulation of the problem, but maybe it helps to at least see how the two distributions use the same underlying model of repeated Bernoulli trials to describe the behavior of different parameters. &lt;/p&gt;&#xA;" OwnerUserId="2456" LastEditorUserId="2456" LastEditorDisplayName="" LastEditDate="2010-11-19T03:52:36.887" LastActivityDate="2010-11-19T03:52:36.887" CommentCount="3" />
  <row Id="4702" PostTypeId="2" ParentId="4700" CreationDate="2010-11-19T01:40:26.773" Score="5" ViewCount="" Body="&lt;p&gt;Blogger Andrew Gelman says that the terms 'fixed effect' and 'random effect' have variable meanings depending on who uses them (&lt;a href=&quot;http://www.stat.columbia.edu/~cook/movabletype/archives/2005/01/why_i_dont_use.html&quot; rel=&quot;nofollow&quot;&gt;link&lt;/a&gt;). Perhaps you can pick out which one of the 5 definitions applies to your case. In general it may be better to either look for equations which describe the probability model the authors are using (when reading) or write out the full probability model you want to use (when writing).&lt;/p&gt;&#xA;" OwnerUserId="1146" LastActivityDate="2010-11-19T01:40:26.773" CommentCount="2" />
  <row Id="4703" PostTypeId="2" ParentId="4700" CreationDate="2010-11-19T04:11:03.943" Score="3" ViewCount="" Body="&lt;p&gt;Fixed effect: Something the experimenter directly manipulates and is often repeatable, e.g., drug administration - one group gets drug, one group gets placebo.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Random effect: Source of random variation / experimental units e.g., individuals drawn (at random) from a population for a clinical trial.&#xA;Random effects estimates the variability&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Mixed effect: Includes both, the fixed effect in these cases are estimating the population level coefficients, while the random effects can account for individual differences in response to an effect, e.g., each person receives both the drug and placebo on different occasions, the fixed effect estimates the effect of drug, the random effects terms would allow for each person to respond to the drug differently.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;General categories of mixed effects - repeated measures, longitudinal, hierarchical, split-plot.&lt;/p&gt;&#xA;" OwnerUserId="966" LastActivityDate="2010-11-19T04:11:03.943" CommentCount="1" />
  <row Id="4704" PostTypeId="2" ParentId="4652" CreationDate="2010-11-19T05:14:16.130" Score="1" ViewCount="" Body="&lt;p&gt;This is possibly a consequence of a known deficiency of steepest descent algorithms in general. Using a &lt;a href=&quot;http://en.wikipedia.org/wiki/Conjugate_gradient_method&quot; rel=&quot;nofollow&quot;&gt;conjugate gradient algorithm&lt;/a&gt; may improve convergence.&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-11-19T05:14:16.130" />
  <row Id="4705" PostTypeId="2" ParentId="4694" CreationDate="2010-11-19T07:43:58.327" Score="2" ViewCount="" Body="&lt;p&gt;About your first question, it seems that the &lt;code&gt;mcquitty&lt;/code&gt; option corresponds to WPGMA clustering, while &lt;code&gt;average&lt;/code&gt; is for UPGMA. It is just by looking at the &lt;a href=&quot;http://svn.r-project.org/R/trunk/src/library/stats/R/hclust.R&quot; rel=&quot;nofollow&quot;&gt;source code&lt;/a&gt;, so it is worth to double check it. But it also referred as is in the &lt;code&gt;upgma()&lt;/code&gt; function from the &lt;a href=&quot;http://cran.r-project.org/web/packages/phangorn/&quot; rel=&quot;nofollow&quot;&gt;phangorn&lt;/a&gt; package.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;About your second question, I think you just have to subset your genes by the group labels found after &lt;code&gt;cutree&lt;/code&gt;, and then plot expression profiles as usual.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-11-19T07:43:58.327" CommentCount="1" />
  <row Id="4706" PostTypeId="2" ParentId="4700" CreationDate="2010-11-19T07:50:50.317" Score="2" ViewCount="" Body="&lt;p&gt;Not really a formal definition, but I like the following slides: &lt;a href=&quot;http://www.ling.upenn.edu/~johnson4/johnson_panel.pdf&quot; rel=&quot;nofollow&quot;&gt;Mixed models and why sociolinguists should use them&lt;/a&gt;, from Daniel Ezra Johnson. A brief recap' is offered on slide 4. Although it mostly focused on psycholinguistic studies, it is very useful as a first step.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-11-19T07:50:50.317" CommentCount="1" />
  <row Id="4707" PostTypeId="1" CreationDate="2010-11-19T08:52:54.817" Score="1" ViewCount="182" Body="&lt;p&gt;As we know, we can get the same eigenvector if we apply PCA to the same data. But, is it possible that we get the same eigenvectors after we apply PCA to two totally different data sets (still same dimension)?&lt;/p&gt;&#xA;" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-11-26T17:43:38.080" LastActivityDate="2010-11-26T17:43:38.080" Title="Can two different data sets get the same eigenvector in PCA?" Tags="&lt;pca&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="4708" PostTypeId="1" CreationDate="2010-11-19T08:58:02.683" Score="3" ViewCount="208" Body="&lt;p&gt;I am looking for a method to use to test for equality of two cumulative density functions.&lt;/p&gt;&#xA;" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-11-19T09:10:19.747" LastActivityDate="2010-11-19T13:29:47.970" Title="Comparing two cumulative density functions" Tags="&lt;distributions&gt;&lt;hypothesis-testing&gt;" AnswerCount="5" FavoriteCount="1" />
  <row Id="4709" PostTypeId="2" ParentId="4708" CreationDate="2010-11-19T09:01:45.890" Score="2" ViewCount="" Body="&lt;p&gt;Plot their inverses against one another, i.e. make a quantile-quantile plot:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Q-Q_plot&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Q-Q_plot&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2036" LastActivityDate="2010-11-19T09:01:45.890" />
  <row Id="4710" PostTypeId="2" ParentId="4708" CreationDate="2010-11-19T09:05:05.633" Score="2" ViewCount="" Body="&lt;p&gt;Take a look at the &lt;a href=&quot;http://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test&quot; rel=&quot;nofollow&quot;&gt;Kolmogorov–Smirnov test&lt;/a&gt; (&lt;a href=&quot;http://sekhon.berkeley.edu/stats/html/ks.test.html&quot; rel=&quot;nofollow&quot;&gt;ks.test&lt;/a&gt; in R.)&lt;/p&gt;&#xA;" OwnerUserId="439" LastActivityDate="2010-11-19T09:05:05.633" />
  <row Id="4711" PostTypeId="2" ParentId="4707" CreationDate="2010-11-19T09:08:43.633" Score="4" ViewCount="" Body="&lt;p&gt;Any pure rotation of the datset would also give the same set of principal components, in which case the answer would be trivially &quot;yes&quot;, depending on the definition of &quot;totally different&quot; (arguably if two datsets have the same principal components then they can't be &quot;totally&quot; different).&lt;/p&gt;&#xA;" OwnerUserId="887" LastEditorUserId="887" LastEditorDisplayName="" LastEditDate="2010-11-19T09:54:37.667" LastActivityDate="2010-11-19T09:54:37.667" CommentCount="4" />
  <row Id="4712" PostTypeId="2" ParentId="4551" CreationDate="2010-11-19T09:49:52.633" Score="9" ViewCount="" Body="&lt;p&gt;Something I see a surprising amount in conference papers and even journals is making multiple comparisons (e.g. of bivariate correlations) and then reporting all the p&amp;lt;.05s as &quot;significant&quot; (ignoring the rightness or wrongness of that for the moment).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know what you mean about psychology graduates, as well- I've finished a PhD in psychology and I'm still only just learning really. It's quite bad, I think psychology needs to take quantitative data analysis more seriously if we're going to use it (which, clearly, we should)&lt;/p&gt;&#xA;" OwnerUserId="199" LastActivityDate="2010-11-19T09:49:52.633" CommunityOwnedDate="2010-11-19T09:49:52.633" CommentCount="1" />
  <row Id="4713" PostTypeId="1" AcceptedAnswerId="4720" CreationDate="2010-11-19T10:01:37.860" Score="12" ViewCount="607" Body="&lt;p&gt;I have used the following r code to estimate the confidence intervals of a binomial proportion because I understand that that substitutes for a &quot;power calculation&quot; when designing receiver operating characteristic curve designs looking at detection of diseases in a population.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;n is 150, and the disease, we believe, is 25% prevalent in the population. I have calculated the values for 75% sensitivity and 90% specificity (because that's what people seem to do).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    binom.test(c(29,9), p=0.75, alternative=c(&quot;t&quot;), conf.level=0.95)&#xA;&#xA;    binom.test(c(100, 12), p=0.90, alternative=c(&quot;t&quot;), conf.level=0.95)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have also visited this site:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://statpages.org/confint.html&quot; rel=&quot;nofollow&quot;&gt;http://statpages.org/confint.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which is a java page which calculates binomial confidence intervals, and it gives the same answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyway, after that lengthy set-up, I want to ask why the confidence intervals are not symmetric, e.g. sensitivity is &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   95 percent confidence interval:&#xA;   0.5975876 0.8855583 &#xA;&#xA;   sample estimate probability: 0.7631579 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Sorry if this is a stupid question, but everywhere I look seems to suggest that they will be symmetric, and a colleague of mine seems to think they will be too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks!&lt;/p&gt;&#xA;" OwnerUserId="199" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-11-21T00:31:02.090" LastActivityDate="2010-11-21T00:31:02.090" Title="Binomial confidence interval estimation - why is it not symmetric?" Tags="&lt;confidence-interval&gt;&lt;binomial&gt;" AnswerCount="5" FavoriteCount="1" />
  <row Id="4714" PostTypeId="2" ParentId="4713" CreationDate="2010-11-19T10:12:28.887" Score="3" ViewCount="" Body="&lt;p&gt;Binomial distribution is just &lt;strong&gt;not&lt;/strong&gt; symmetric, yet this fact emerges especially for $p$ near $0$ or $1$ and for small $n$; most people use it for $p\approx 0.5$ and so the confusion.&lt;/p&gt;&#xA;" OwnerUserId="88" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-19T11:01:47.507" LastActivityDate="2010-11-19T11:01:47.507" />
  <row Id="4715" PostTypeId="2" ParentId="4700" CreationDate="2010-11-19T10:29:35.280" Score="1" ViewCount="" Body="&lt;p&gt;I also find that sometimes is difficult to determine when an effect must be considered as fixed or as random effect. Althought there are some recommendations about this fact, not always is easy to take the right decision.&lt;/p&gt;&#xA;" OwnerUserId="221" LastActivityDate="2010-11-19T10:29:35.280" />
  <row Id="4716" PostTypeId="2" ParentId="4713" CreationDate="2010-11-19T11:13:46.303" Score="8" ViewCount="" Body="&lt;p&gt;To see why it should not be symmetric, think of the situation where $p=0.9$ and you get 9 successes in 10 trials. Then $\hat{p}=0.9$ and the 95% CI for $p$ is [0.554, 0.997]. The upper limit cannot be greater than 1 obviously, so most of the uncertainty must fall to the left of $\hat{p}$.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-11-19T11:13:46.303" />
  <row Id="4717" PostTypeId="1" CreationDate="2010-11-19T11:32:02.027" Score="7" ViewCount="533" Body="&lt;p&gt;In the literature on hierarchical/multilevel models I have often read about &quot;nested models&quot; and &quot;non-nested models&quot;, but what does this mean? Could anyone maybe give me some examples or tell me about the mathematical implications of this phrasing? Thanks in advance!&lt;/p&gt;&#xA;" OwnerUserId="1082" LastEditorUserId="1082" LastEditorDisplayName="" LastEditDate="2010-11-19T13:03:09.767" LastActivityDate="2010-11-26T15:10:17.447" Title="What is the difference between a &quot;nested&quot; and a &quot;non-nested&quot; model?" Tags="&lt;multilevel-analysis&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="4719" PostTypeId="2" ParentId="4717" CreationDate="2010-11-19T12:01:19.740" Score="5" ViewCount="" Body="&lt;p&gt;Nested versus non-nested can mean a whole lot of things. You have nested designs versus crossed designs (see eg &lt;a href=&quot;http://www.psychstat.missouristate.edu/introbook/sbk21m.htm&quot; rel=&quot;nofollow&quot;&gt;this explanation&lt;/a&gt;). You have nested models in model comparison. Nested means here that all terms of a smaller model occur in a larger model. This is a necessary condition for using most model comparison tests like likelihood ratio tests.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the context of multilevel models I think it's better to speak of nested and non-nested factors. The difference is in how the different factors are related to one another. In a nested design, the levels of one factor only make sense within the levels of another factor. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say you want to measure the oxygen production of leaves. You sample a number of tree species, and on every tree you sample some leaves on the bottom, in the middle and on top of the tree. This is a nested design. The difference for leaves in a different position only makes sense within one tree species. So comparing bottom leaves, middle leaves and top leaves over all trees is senseless. Or said differently: leaf position should not be modelled as a main effect. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Non-nested factors is a combination of two factors that are not related. Say you study patients, and are interested in the difference of age and gender. So you have a factor ageclass and a factor gender that are not related. You should model both age and gender as a main effect, and you can take a look at the interaction if necessary. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The difference is not always that clear. If in my first example the tree species are closely related in form and physiology, you could consider leaf position also as a valid main effect. In many cases, the choice for a nested design versus a non-nested design is more a decision of the researcher than a true fact. &lt;/p&gt;&#xA;" OwnerUserId="1124" LastActivityDate="2010-11-19T12:01:19.740" />
  <row Id="4720" PostTypeId="2" ParentId="4713" CreationDate="2010-11-19T12:44:10.127" Score="8" ViewCount="" Body="&lt;p&gt;They're believed to be symmetric because quite often a normal approximation is used. This one works well enough in case p lies around 0.5. &lt;code&gt;binom.test&lt;/code&gt; on the other hand reports &quot;exact&quot; Clopper-Pearson intervals, which are based on the F distribution (see &lt;a href=&quot;http://people.ee.duke.edu/~kst/ECE257/confbin.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; for the exact formulas of both approaches).  If we would implement the Clopper-Pearson interval in R it would be something like (see &lt;em&gt;&lt;strong&gt;note&lt;/strong&gt;&lt;/em&gt;):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Clopper.Pearson &amp;lt;- function(x,n,conf.level){&#xA;    alpha &amp;lt;- (1 - conf.level) / 2&#xA;    QF.l &amp;lt;- qf(1 - alpha, 2*n - 2*x + 2, 2*x)&#xA;    QF.u &amp;lt;- qf(1 - alpha, 2*x + 2, 2*n - 2*x)&#xA;&#xA;    ll &amp;lt;- if (x == 0)&#xA;          0&#xA;      else x / ( x + (n-x+1)*QF.l )&#xA;&#xA;    uu &amp;lt;- if (x == 0)&#xA;          0&#xA;      else (x+1)*QF.u / ( n - x + (x+1)*QF.u )&#xA;    c(ll,uu)&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You see both in the link and in the implementation that the formula for the upper and the lower limit are completely different. The only case of a symmetric confidence interval is when p=0.5. Using the formulas from the link and taking into account that in this case $n = 2\times x$ it's easy to derive yourself how it comes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I personally understood it better looking at the confidence intervals based on a logistic approach. Binomial data is generally modelled using a logit link function, defined as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$logit(x) = \log ( \frac{x}{1-x} )$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This link function &quot;maps&quot; the error term in a logistic regression to a normal distribution. As a consequence, confidence intervals in the logistic framework are symmetric around the logit values, much like in the classic linear regression framework. The logit transformation is used exactly to allow for using the whole normality-based theory around the linear regression. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;After doing the inverse transformation&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$logit^{-1}(x) = \frac{e^x}{1+e^{x}}$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You get an asymmetric interval again. Now these confidence intervals are actually biased. Their coverage is not what you would expect, especially at the boundaries of the binomial distribution. Yet, as an illustration they show you why it is logic that a binomial distribution has asymmetric confidence intervals. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;An example in R:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;logit &amp;lt;- function(x) log(x/(1-x))&#xA;inv.logit &amp;lt;- function(x) exp(x)/(1+exp(x))&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x &amp;lt;- c(0.2, 0.5 , 0.8)&#xA;lx &amp;lt;- logit(x)&#xA;upper &amp;lt;- lx + 2&#xA;lower &amp;lt;- lx - 2&#xA;&#xA;logxtab &amp;lt;- cbind(lx,upper,lower)&#xA;logxtab # the confidence intervals are symmetric by construction&#xA;xtab &amp;lt;- inv.logit(logxtab)&#xA;xtab # back transformation gives asymmetric confidence intervals&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;&lt;strong&gt;note&lt;/strong&gt; : In fact, R uses the beta distribution, but this is completely equivalent and computationally a bit more efficient. The implementation in R is thus different from what I show here, but it gives exactly the same result.&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="1124" LastEditorUserId="1124" LastEditorDisplayName="" LastEditDate="2010-11-20T00:30:40.107" LastActivityDate="2010-11-20T00:30:40.107" CommentCount="3" />
  <row Id="4721" PostTypeId="2" ParentId="4708" CreationDate="2010-11-19T12:53:56.033" Score="1" ViewCount="" Body="&lt;p&gt;The QQ-plot and the Kolmogorov-Smirnov test are two widely used options. A QQ-plot requires some level of expertise, as the decision is based on your own judgement. See also the answers to &lt;a href=&quot;http://stats.stackexchange.com/questions/2492/normality-testing-essentially-useless/2498#2498&quot;&gt;this question&lt;/a&gt; for more discussion about both tests. I use there the Shapiro-Wilks test for normality, which can be seen as a parametric counterpart of the KS test in case the comparison is made with a normal distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For reference, I'd like to point out the book &lt;a href=&quot;http://www.springer.com/statistics/book/978-0-387-92709-1&quot; rel=&quot;nofollow&quot;&gt;Comparing Distributions from prof. dr. Olivier Thas.&lt;/a&gt; This gives a thorough overview of parametric, semi-parametric and non-parametric approaches to the topic.&lt;/p&gt;&#xA;" OwnerUserId="1124" LastActivityDate="2010-11-19T12:53:56.033" />
  <row Id="4722" PostTypeId="2" ParentId="4708" CreationDate="2010-11-19T13:16:41.517" Score="0" ViewCount="" Body="&lt;p&gt;Lately I've been playing with comparing distributions by computing the difference between their empirical CDFs and then bootstrapping intervals on this difference. Differences between the distributions in location, scale, and each tail all have different and rather noticeable effects on the DECDF function.&lt;/p&gt;&#xA;" OwnerUserId="364" LastActivityDate="2010-11-19T13:16:41.517" CommentCount="1" />
  <row Id="4723" PostTypeId="2" ParentId="4708" CreationDate="2010-11-19T13:29:47.970" Score="0" ViewCount="" Body="&lt;p&gt;Might be worth looking at some variant of the &lt;a href=&quot;http://en.wikipedia.org/wiki/Anderson%E2%80%93Darling_test&quot; rel=&quot;nofollow&quot;&gt;Anderson-Darling&lt;/a&gt; or &lt;a href=&quot;http://en.wikipedia.org/wiki/Cram%C3%A9r%E2%80%93von-Mises_criterion&quot; rel=&quot;nofollow&quot;&gt;Cramer-von Mises&lt;/a&gt; statistics.  The latter is essentially a weighted least-squares distance between two CDFs.&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2010-11-19T13:29:47.970" />
  <row Id="4724" PostTypeId="2" ParentId="4717" CreationDate="2010-11-19T13:33:31.053" Score="1" ViewCount="" Body="&lt;p&gt;Nested vs non-nested models come up in conjoint analysis and &lt;a href=&quot;http://en.wikipedia.org/wiki/Independence_of_irrelevant_alternatives&quot; rel=&quot;nofollow&quot;&gt;IIA&lt;/a&gt;. Consider the &quot;red bus blue bus problem&quot;. You have a population where 50% of people take a car to work and the other 50% take the red bus. What happens if you add a blue bus which has the same specifications as the red bus to the equation? A  &lt;a href=&quot;http://en.wikipedia.org/wiki/Multinomial_logit&quot; rel=&quot;nofollow&quot;&gt;multinomial logit&lt;/a&gt; model will predict 33% share for all three modes. We intuitively know this is not correct as the red bus and blue bus are more similar to one another than to the car and will thus take more share from one another before taking share from the car. That is where a nesting structure comes in, which is typically specified as a lambda coefficient on the similar alternatives.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ben Akiva has put together a nice set of slides outlining the theory on this &lt;a href=&quot;http://www.google.com/url?sa=t&amp;amp;source=web&amp;amp;cd=13&amp;amp;ved=0CCMQFjACOAo&amp;amp;url=http%3A%2F%2Focw.mit.edu%2Fcourses%2Fcivil-and-environmental-engineering%2F1-201j-transportation-systems-analysis-demand-and-economics-fall-2008%2Flecture-notes%2FMIT1_201JF08_lec04.pdf&amp;amp;rct=j&amp;amp;q=red%20blue%20blue%20bus%20problem%20nested%20logit&amp;amp;ei=rnvmTOimD8X6lwf0z6DiCw&amp;amp;usg=AFQjCNG2vENYDIIw0Q6WtIF3eU65uB1mXg&amp;amp;sig2=QNCugsgvgq4wezmhR1VYeA&amp;amp;cad=rja&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;. He begins talking about nested logit around slide 23.&lt;/p&gt;&#xA;" OwnerUserId="696" LastEditorUserId="696" LastEditorDisplayName="" LastEditDate="2010-11-26T15:10:17.447" LastActivityDate="2010-11-26T15:10:17.447" />
  <row Id="4725" PostTypeId="2" ParentId="4713" CreationDate="2010-11-19T13:37:37.487" Score="6" ViewCount="" Body="&lt;p&gt;@Joris mentioned the symmetric or &quot;asymptotic&quot; interval, that is most likely the one you are expecting.  @Joris also mentioned the &quot;exact&quot; Clopper-Pearson intervals and gave you a reference which looks very nice.  There is another confidence interval for proportions which you will likely encounter (note it is also not symmetric), the &quot;Wilson&quot; interval which is a type of asymptotic interval based on inverting the score test.  The endpoints of the interval solve  (in $p$) the equation&#xA;$$&#xD;&#xA;(\hat{p} - p)/\sqrt{p(1-p)}=\pm z_{\alpha/2}&#xD;&#xA;$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyway, you can get all three in R with the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(Hmisc)&#xA;binconf(29, 38, method = &quot;asymptotic&quot;)&#xA;binconf(29, 38, method = &quot;exact&quot;)&#xA;binconf(29, 38, method = &quot;wilson&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Note that method &quot;wilson&quot; is the same confidence interval used by prop.test without Yates' continuity correction:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;prop.test(29, 38, correct = FALSE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;See &lt;a href=&quot;https://home.comcast.net/~lthompson221/&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; for Laura Thompson's free SPLUS + R manual which accompanies Agresti's Categorical Data Analysis in which these issues are discussed in great detail.&lt;/p&gt;&#xA;" OwnerUserId="1108" LastActivityDate="2010-11-19T13:37:37.487" CommentCount="2" />
  <row Id="4726" PostTypeId="2" ParentId="4707" CreationDate="2010-11-19T14:12:05.453" Score="1" ViewCount="" Body="&lt;p&gt;An eigenvector is just a statistic. You can get copies of almost any useful statistic from a wide variety of different data, as long as they are similar in the relevant manner.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The eigenvector represents a particular linear relationship between the variables. As long as you use variables with the same units and what appears to be the same linear relationship, after normalization, you will get the same eigenvector.&lt;/p&gt;&#xA;" OwnerUserId="2456" LastActivityDate="2010-11-19T14:12:05.453" />
  <row Id="4727" PostTypeId="1" CreationDate="2010-11-19T14:30:36.020" Score="2" ViewCount="157" Body="&lt;p&gt;I am interested in comparing a non-linear model with up to 12 parameters to many datasets. However, each instance of the model takes a significant amount of time to compute (~1 hour), so I am pre-computing instances of the model for various parameter values and then comparing these to all the different datasets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are various ways to sample parameter space - so far, I've come across regular grids (impossible here), sparse grids + interpolation, Monte-Carlo random sampling, and there are probably others. Which approach would be optimal for a fixed amount of computing resources, and therefore a fixed number of model instances?&lt;/p&gt;&#xA;" OwnerUserId="2052" LastActivityDate="2011-05-18T18:47:45.903" Title="Non-linear model fitting in many dimensions" Tags="&lt;modeling&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="4728" PostTypeId="2" ParentId="4685" CreationDate="2010-11-19T14:31:20.210" Score="1" ViewCount="" Body="&lt;p&gt;Based on the paper you linked to, I would argue that the term EM usually refers to the &quot;soft&quot; version. The key distinction seems to be that instead of taking an expectation in the E-step, the &quot;hard&quot; version finds a mode.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A good explanation of the distinction is available in chapters 20-22 of David Mackay's book (which is &lt;a href=&quot;http://www.inference.phy.cam.ac.uk/mackay/itila/&quot; rel=&quot;nofollow&quot;&gt;available online&lt;/a&gt;).&lt;/p&gt;&#xA;" OwnerUserId="495" LastActivityDate="2010-11-19T14:31:20.210" />
  <row Id="4729" PostTypeId="1" CreationDate="2010-11-19T14:48:02.893" Score="4" ViewCount="160" Body="&lt;p&gt;I am about to do a laboratory experiment in the scientific field of soil ecology and hydrology. Beforehand I want to make sure not to make any crucial mistakes, and therefore I would appreciate any hints and comments from your side. The main issue is how to deal with high natural variability (~25%) and a rather small sample size (total max=20).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Short Description of the Experiment:&lt;/strong&gt;  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We will put soil cores in cylinders and keep them under different moisture scenarios. Three different kind of Carbon forms will be measured for about one month.&lt;br&gt;&#xA;We want to know if the variables change within groups and between groups.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The experimental design that was proposed is as follows:&lt;br&gt;&#xA;There will be 2 different soil types, 2 different treatments + one control, and each treatment should be replicated three times. The total number of cylinders is thus 18 cylinders. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The variability in field measurements can be as high as 25% within one group. Due to pracitcal reasons there can not be more than 20 cylincers in total.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My Questions:  &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Would it make more sense to have only one treatment and one control, but each one is replicated five times?   &lt;/li&gt;&#xA;&lt;li&gt;Will I be able to draw any reliable conclusions from this kind of experiment under these conditions?  &lt;/li&gt;&#xA;&lt;li&gt;How should I set the parameters to make some power calculations using e.g. &lt;a href=&quot;http://www.psycho.uni-duesseldorf.de/abteilungen/aap/gpower3/&quot; rel=&quot;nofollow&quot;&gt;G*Power 3&lt;/a&gt;? Which test should I choose? What should I set the effect size to and what should the numbers for df be?  &lt;/li&gt;&#xA;&lt;li&gt;How should I analyze the data after the experiment is done? Should I use ANOVA? Can I use a mixed effect model?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="2063" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-11-23T23:32:26.473" LastActivityDate="2010-11-23T23:32:26.473" Title="How to setup a laboratory experiment in Ecological Research under high natural variability" Tags="&lt;experiment-design&gt;&lt;best-practices&gt;&lt;basic-concepts&gt;&lt;power-analysis&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="4730" PostTypeId="2" ParentId="4686" CreationDate="2010-11-19T15:56:00.497" Score="4" ViewCount="" Body="&lt;p&gt;Despite promising not to, I have thought about this problem further.  This approach differs enough from the previous one I outlined that it seems worthwhile posting it as a separate reply.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Both @Aniko and @shabbychef are right: you need to &quot;almost exhaust the population&quot; with &quot;greedy sampling.&quot;  But there's a twist--on occasion you can get away with a small sample.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's first change the notation (only slightly) to provide a clear interpretation of the constraint in the question.  Assume that a (small) threshold error probability $p$ and a maximum error size $\epsilon$ (in place of $k$, which will have uses elsewhere) have been specified, so that we require&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\Pr[|\hat{B}-B| \leq \epsilon] \ge 1 - p$$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;regardless of the (unknown) values of the $b_i$ (the &quot;state&quot; of the population).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let $c_i$ be the cost of sampling element $i$ in the population.  Suppose $A$ is a subset of the population that is sampled with probability $\pi_A$, at a cost of $c(A) = \sum_{i \in A}{c_i}$.  Let $m$ be the number of elements of $A$ (written $m = |A|$) and let $k$ be the number of them that are 1's.  This information tells us that $B$ surely lies between $k$ and $k + n - m$ no matter what the state of the population may be.  Provided only that &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$k + n - m - \epsilon \le \hat{B} \le k + \epsilon,$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;we are assured that the error associated with the sample $A$ cannot exceed $\epsilon$. This is &lt;em&gt;not&lt;/em&gt; possible whenever $m \lt n - 2 \epsilon$.  Let's say that such a subset is &quot;small&quot; (with respect to $\epsilon$ and $n$) and otherwise is &quot;large.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is perhaps the only subtlety: when a sample is small we still have a chance of not making an error, provided we use a randomized estimator.  An example of the best ones I can find is &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\hat{B} = k + (2j-1)\epsilon \text{ with probability } \frac{1}{l},\ j=1,2,\ldots, l$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $l = \lceil{\frac{n-m}{2 \epsilon}\rceil}$.  No matter what the values of the unsampled data are, this procedure has at least a chance of $2/l$ of being within $\epsilon$ of the correct total $B$.  Using such an estimator, the probability of an unacceptable error is bounded by the expected chance that the randomized estimate will have too great an error:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\Pr[|\hat{B}-B| \gt \epsilon] \le \sum_{A}{\pi_A(1 - \frac{1}{\lceil{\frac{n-|A|}{2 \epsilon}\rceil}})}.$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(The coefficient when $|A| = n$ appears to be undefined but actually is zero; the sum really needs to extend only over the small subsets where randomization is actually needed.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We have obtained a &lt;em&gt;linear program&lt;/em&gt; for the sample probabilities $\pi_A$; to wit,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Minimize&lt;/strong&gt; the expected cost $\sum_{A}{\pi_A c(A)}$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;subject to&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;$\sum_{A}{\pi_A(1 - \frac{1}{\lceil{(n-|A|)/(2 \epsilon)\rceil}})} \le p,$&#xD;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;$\sum_{A}{\pi_A} = 1,$&#xD;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;$\pi_A \ge 0$ for all subsets $A$.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;This is a simple linear program (but with $2^n$ variables), easy to set up and easy to solve provided the population has about 16 or fewer bits.  When some of the costs are the same, the number of variables can be substantially reduced.  With larger populations, approximate methods would be needed to obtain a solution.   Generally, the solution cannot include any small samples with appreciable probability: most of the probability must be concentrated on large samples.  Among those, it will select the cheapest (which can be found with the greedy algorithm).  These heuristics allow for simple, rapid approximations to good solutions.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;The solutions can be interesting.  Here are some examples.  As an abbreviation, let $f(c, \epsilon, p)$ indicate a solution for cost vector $c = (c_1, c_2, \ldots, c_n)$ and problem constraints $\epsilon$ and $p$.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;$f((1,1), 1/2, 1/20)$ samples each element with probability 9/20 and &lt;em&gt;obtains no data&lt;/em&gt; with probability 1/10.  The expected cost is 0.9.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why is this?  If we sample element 1 we observe $b_1$ and estimate $\hat{B}$ = $b_1 + 1/2$.  This is certainly within $1/2$ of the correct value.  When we take no sample we estimate that $B$ equals $1/2$ with 50% probability and otherwise estimate that $B$ equals $3/2$.  No matter what the population is, this guessing will return the correct answer (within an error of $1/2$) 50% of the time.  Thus we make an error greater than $1/2$ only 50% of $1/10$ of the time, which meets the targeted error rate of $1/20$.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;$f((1,1,5,5), 1/2, 1/20)$ elects to sample both cheap bits no matter what.  In addition, there is a 45% chance it will also include bit 3 (but not bit 4) and a 45% chance it will also include bit 4 (but not bit 3).  The reasoning is similar to the previous situation.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;$f((1,1,5,5), 1/4, 1/20)$ samples &lt;em&gt;the entire population&lt;/em&gt; with a $33/35$ probability and otherwise obtains no data with $2/35$ probability.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;$f((1,2,3,4), 1/2, 1/20)$ samples bits 1, 2, and 3 with $14/15$ probability and otherwise obtains no data.  The expected cost equals $84/15$ = 5.6.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-11-22T15:43:17.257" LastActivityDate="2010-11-22T15:43:17.257" />
  <row Id="4731" PostTypeId="2" ParentId="4727" CreationDate="2010-11-19T17:03:45.997" Score="0" ViewCount="" Body="&lt;p&gt;My understanding is that the optimal methodology is going to depend on the surface texture of parametrization errors.  That is, how often do parameters reflect or manifest as interactions?  If each parameter is an individual value that has its own distinct minima independent of all other parameters, model fitting should proceed easily with sparse grids and interpolation.  If instead there are lots of local minima you might find that spare grids and interpolation will seldom fall on the same values.  Given the time required to fit your model I doubt Monte-Carlo random sampling will be the optimal approach.  Another approach which you haven't considered yet are genetic algorithms, but again convergence to a single answer may be difficult.&lt;/p&gt;&#xA;" OwnerUserId="196" LastActivityDate="2010-11-19T17:03:45.997" />
  <row Id="4733" PostTypeId="1" CreationDate="2010-11-19T17:53:49.017" Score="2" ViewCount="135" Body="&lt;p&gt;Can you suggest a tutorial or book chapter on basics of data analysis with exponentially distributed data / exponential noise,&#xA;at undergraduate level?&lt;br&gt;&#xA;By &quot;basics&quot; I mean:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;is a given set of data exponentially distributed?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;should one generally use median instead of mean for exponential data?&#xA;Should one simply trim the top 10% (for some value of 10)?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;least-squares curve fitting, where I &#xA;believe errors to be exponential?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;(Ok, I'll be made to write out &quot;it's not that simple&quot; 100 times.)&lt;/p&gt;&#xA;" OwnerUserId="557" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-11-19T18:07:59.837" LastActivityDate="2010-12-19T19:44:26.317" Title="Basics of data analysis with exponential data/noise?" Tags="&lt;robust&gt;&lt;tutorial&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="4734" PostTypeId="1" AcceptedAnswerId="4760" CreationDate="2010-11-19T18:55:37.910" Score="2" ViewCount="420" Body="&lt;p&gt;I know what you're thinking, this is a duplicate of &quot;&lt;a href=&quot;http://stats.stackexchange.com/questions/1576/what-are-the-differences-between-factor-analysis-and-principal-component-analysis&quot;&gt;What are the differences between Factor Analysis and Principal Component Analysis&lt;/a&gt;&quot;, but it isn't really.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That other question deals with Confirmatory Factor Analysis. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Either way, I would like to know what the difference is :)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="74" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-11-19T21:53:28.093" LastActivityDate="2010-12-13T04:50:07.943" Title="What is the difference between Exploratory Factor Analysis and Principal Components Analysis (PCA)?" Tags="&lt;pca&gt;&lt;factor-analysis&gt;" AnswerCount="2" CommentCount="5" FavoriteCount="1" />
  <row Id="4735" PostTypeId="1" CreationDate="2010-11-19T19:01:58.203" Score="3" ViewCount="270" Body="&lt;p&gt;These terms seem to get throw around together a lot, but I would like to know what you think the differences are, if any. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, how do PCA, and NMF fit in?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="74" LastEditorUserId="74" LastEditorDisplayName="" LastEditDate="2010-12-23T22:34:23.727" LastActivityDate="2010-12-23T22:34:23.727" Title="What are the differences among Latent Semantic Analysis (LSA), Latent Semantic Indexing (LSI), and Singular Value Decomposition (SVD)?" Tags="&lt;pca&gt;&lt;text-mining&gt;&lt;svd&gt;&lt;nmf&gt;" AnswerCount="2" />
  <row Id="4736" PostTypeId="2" ParentId="4733" CreationDate="2010-11-19T19:28:52.357" Score="1" ViewCount="" Body="&lt;p&gt;One of the basic techniques for working with skewed positive data is to analyse it on the log-scale. However its appropriateness depends on what you are really trying to achieve.&lt;/p&gt;&#xA;" OwnerUserId="279" LastActivityDate="2010-11-19T19:28:52.357" />
  <row Id="4737" PostTypeId="1" AcceptedAnswerId="4740" CreationDate="2010-11-19T19:33:02.400" Score="3" ViewCount="364" Body="&lt;p&gt;I have a class with a set of descriptive statistic functions (mean, median, kurtosis, etc...).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I need to include weight (array) into my equations. My first thought was to just create a weighted version of the functions where the additional weight array is passed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I was wondering if there is a way to alter the data (array) so that the same functions can be used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: This is my first Statistics project so I am learning as I go. Sorry if this is a stupid question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a way to weight the data before calculating the descriptive statistics - so that the existing functions can be used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If this is possible then I could add an additional function to weight the data before it is passed into the descriptive functions.&lt;/p&gt;&#xA;" OwnerUserId="2060" LastActivityDate="2010-11-19T21:49:43.513" Title="How to add weight to data in descriptive statistics?" Tags="&lt;algorithms&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="4738" PostTypeId="2" ParentId="4735" CreationDate="2010-11-19T20:00:45.343" Score="1" ViewCount="" Body="&lt;p&gt;LSA and LSI are mostly used synonymously, with the information retrieval community usually referring to it as LSI. LSA/LSI uses SVD to decompose the term-document matrix A into a term-concept matrix U, a singular value matrix S, and a concept-document matrix V in the form: A = USV'. The wikipedia page has a detailed description of &lt;a href=&quot;http://en.wikipedia.org/wiki/Latent_semantic_indexing&quot; rel=&quot;nofollow&quot;&gt;latent semantic indexing&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="881" LastActivityDate="2010-11-19T20:00:45.343" />
  <row Id="4739" PostTypeId="2" ParentId="4659" CreationDate="2010-11-19T20:26:32.220" Score="0" ViewCount="" Body="&lt;p&gt;In Bayesian land, the Beta distribution is the conjugate prior for the p parameter of the Binomial distribution.&lt;/p&gt;&#xA;" OwnerUserId="1860" LastActivityDate="2010-11-19T20:26:32.220" />
  <row Id="4740" PostTypeId="2" ParentId="4737" CreationDate="2010-11-19T20:34:51.880" Score="3" ViewCount="" Body="&lt;p&gt;It isn't entirely clear from your question what sort of 'weight' you are talking about.  But I imagine it is a simple matter of wanting to count certain observations more than than others... &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you wanted to, and your weights were integer values (or you can find the lowest common denominator to multiply by that will give you integer values), you could simply expand your data out to match the weights.  That is, a data point with a weight of 2 could be represented twice in your data array.  This is fine for descriptives such as mean, median, kurtosis, and skew.  However, it may be be problematic for calculations such as a sample estimate of the standard deviation where N matters and the difference between a raw N-1 and a N-1 where N is representative of the restructured array might be meaningful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The only shortcut I can think of that might apply is to multiply your array by the weights and then analyze those results.  For the mean, to renormalize the value, you will need to divide the result by (sum)Weights/N_original.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, for the median, kurtosis, and skew you won't be able to (readily) use this technique and I think you will have to revert to altering your function to produce new data arrays.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good luck.&lt;/p&gt;&#xA;" OwnerUserId="196" LastActivityDate="2010-11-19T20:34:51.880" CommentCount="2" />
  <row Id="4741" PostTypeId="1" AcceptedAnswerId="4743" CreationDate="2010-11-19T20:35:06.833" Score="1" ViewCount="133" Body="&lt;p&gt;I do a counting experiment where I count observations as a function of two float parameters $x_1$ and $x_2$. This leads to a two-dimensional histogram where each bin corresponds to the number of observations with $x_1$ and $x_2$ in some range.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I now see a lot of bins with zero counts in them (even though more counts are in principle possible) and wonder how to best describe the uncertainty in these bins.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Knowing that I had $N$ observations which can fall into $k$ bins, I could assign an &quot;expected rate&quot; $n=\frac{N}{k}$ per bin and calculate the uncertainty with Poisson statistic. I however also know that my underlying process does not distribute events flat in $x_1$ and $x_2$ (but with a distribution I would like to extract from the measurement), how can I consistently assign an uncertainty in zero-count bins?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I do not want to rebin my data or drop these zero bins to not loose information.&lt;/p&gt;&#xA;" OwnerUserId="56" LastActivityDate="2010-11-20T04:59:03.040" Title="Uncertainty on zero counts for binned result" Tags="&lt;confidence-interval&gt;&lt;poisson&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="4742" PostTypeId="2" ParentId="4737" CreationDate="2010-11-19T21:49:43.513" Score="5" ViewCount="" Body="&lt;p&gt;Weights can arise in data analysis through various mechanisms, each of which requires its own formulas:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;A dataset with many duplicate results can be summarized by listing each unique result together with its frequency of occurrence.  This is the definition @drknexus assumes in order to provide a definite answer (after recognizing that other definitions are possible).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;When datasets represent averages or other statistics, their values have known (or at least pre-estimated) levels of uncertainty.  The weights can represent those levels.  (Typically the appropriate weight to use is the inverse of the variance.)  These are incorporated in methods like &lt;a href=&quot;http://www.itl.nist.gov/div898/handbook/pmd/section1/pmd143.htm&quot; rel=&quot;nofollow&quot;&gt;weighted least squares regression&lt;/a&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Many datasets obtained through observational studies in the social and biological sciences arise from complex sampling schemes in which units/subjects have differing chances of being selected.  The appropriate weights to use in estimates are usually the inverses of the selection probabilities, as in the &lt;a href=&quot;http://www.amstat.org/sections/srms/proceedings/y2001/Proceed/00278.pdf&quot; rel=&quot;nofollow&quot;&gt;Hansen-Hurwitz Estimator&lt;/a&gt; and the &lt;a href=&quot;http://en.wikipedia.org/wiki/Horvitz%E2%80%93Thompson_estimator&quot; rel=&quot;nofollow&quot;&gt;Horvitz-Thompson Estimator&lt;/a&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Various &lt;a href=&quot;http://en.wikipedia.org/wiki/Robust_statistics&quot; rel=&quot;nofollow&quot;&gt;robust methods&lt;/a&gt;, such as &lt;a href=&quot;http://en.wikipedia.org/wiki/Iteratively_reweighted_least_squares&quot; rel=&quot;nofollow&quot;&gt;IRLS regression&lt;/a&gt;, iteratively reweight data in order to de-emphasize atypical values.  These weights can enter into formulas in ways that differ from (1) - (3) above.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Thus, you need first to decide what your weights mean and what the purpose of computing the weighted statistics might be.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-11-19T21:49:43.513" />
  <row Id="4743" PostTypeId="2" ParentId="4741" CreationDate="2010-11-19T21:59:27.513" Score="2" ViewCount="" Body="&lt;p&gt;Nothing wrong with what you've done so far. The null model includes only a constant, i.e. a flat event rate. Fitting more complex &lt;a href=&quot;http://en.wikipedia.org/wiki/Poisson_regression&quot; rel=&quot;nofollow&quot;&gt;Poisson regression&lt;/a&gt; models will allow the expected value to vary. It's hard to tell what forms the more complex models should take as you've told us nothing about the source of the data, how much data you have, or the question(s) you wish to answer. Binning the data may help suggest what models are appropriate, but you may be better keeping continuous covariates continuous and fitting some smoother form (splines, fractional polynomials, local regression...).&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-11-19T21:59:27.513" CommentCount="1" />
  <row Id="4744" PostTypeId="2" ParentId="4713" CreationDate="2010-11-19T22:03:50.017" Score="4" ViewCount="" Body="&lt;p&gt;There &lt;em&gt;are&lt;/em&gt; symmetric confidence intervals for the Binomial distribution: asymmetry is not forced on us, despite all the reasons already mentioned.  The symmetric intervals are usually considered inferior in that&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Although they are &lt;em&gt;numerically&lt;/em&gt; symmetric, they are not symmetric &lt;em&gt;in probability&lt;/em&gt;: that is, their one-tailed coverages differ from each other.  This--a necessary consequence of the possible asymmetry of the Binomial distribution--is the crux of the matter.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Often one endpoint has to be unrealistic (less than 0 or greater than 1), as @Rob Hyndman points out.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Having said that, I suspect that numerically symmetric CIs might have some good properties, such as tending to be shorter than the probabilistically symmetric ones in some circumstances.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-11-19T22:03:50.017" />
  <row Id="4746" PostTypeId="2" ParentId="4729" CreationDate="2010-11-19T22:16:04.327" Score="2" ViewCount="" Body="&lt;p&gt;Whether you have a reasonable chance of obtaining (i.e. power to obtain) reliable conclusions depends on how big the effects are you wish to be able to detect. With such small numbers they'll have to be very large. Clearly having fewer treatments and more replications per treatment will give you at least a bit more power, or equivalently the aiblity to detect somewhat smaller effects with the same power.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To put some rough numbers on that, let's ignore the soil types for simplicity (including them will make things more gloomy) and do some standard power calculations two-sample for 2-sample t-tests. If you compare one treatment vs control with 10 in each group (i.e. 20 in total) you'll have 80% power to detect a difference between treatment and control of 1.25 standard deviations (SDs). With two treatments + control, 6 in each group (18 in total), you have 80% power to detect a difference of 1.4 SDs between both treatments &lt;em&gt;combined&lt;/em&gt; and control, or 1.6 SDs between &lt;em&gt;either&lt;/em&gt; treatment by itself and control (or between the two treatments). It may well be sensible use a log-transform (or perhaps some other transform) your data prior to analysis, in which case the SDs are the SDs of the transformed variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the &lt;em&gt;social&lt;/em&gt; sciences, &lt;a href=&quot;http://en.wikipedia.org/wiki/Effect_size#.22Small.22.2C_.22medium.22.2C_.22large.22&quot; rel=&quot;nofollow&quot;&gt;an effect of around 0.8 SDs or over would often be considered &quot;large&quot;&lt;/a&gt;, and designing a study to detect to have decent power only to detect a bigger effect than this might be politely described as &quot;optimistic&quot;. But remember that the SD here is the SD of the &lt;em&gt;residual, unexplained&lt;/em&gt; variation. You can reduce this by either (1) making your experimental units more uniform or (2) explaining more of the variation by other means. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;The lower the uncontrolled variability the higher the power you'll have to detect effects due to the factors open to experimental manipulation. You say &quot;variability in &lt;em&gt;field&lt;/em&gt; measurements can be as high as 25% within one group&quot;. But this is a &lt;em&gt;laboratory&lt;/em&gt; experiment; is there a reason the variability need be this high in the lab? Can you homogenise your soil before you start the experiment? I guess this may destroy the soil structure though?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Can you take &lt;em&gt;baseline&lt;/em&gt; measurements before the treatments are applied? Using these to explain some of the inate variability between units by either analysing change since baseline or (better) adding them to the model as covariates (.e. ANCOVA) may help a &lt;em&gt;lot&lt;/em&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Sorry I haven't mentioned G*Power 3 but i've never heard of it and from a quick look the link you gave it looks considerably more sophisticated, and therefore complicated, than is necessary here.&lt;/p&gt;&#xA;" OwnerUserId="449" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-11-19T23:09:37.230" LastActivityDate="2010-11-19T23:09:37.230" CommentCount="3" />
  <row Id="4747" PostTypeId="1" AcceptedAnswerId="4748" CreationDate="2010-11-20T00:40:53.173" Score="1" ViewCount="1051" Body="&lt;p&gt;I was wondering what is your recommended way to compute the inverse of a matrix?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The ways I found seem not satisfactory. For example, &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&gt; c=rbind(c(1, -1/4), c(-1/4, 1))&lt;br&gt;&#xA;  &gt; c&lt;br&gt;&#xA;       [,1]  [,2]&lt;br&gt;&#xA;  [1,]  1.00 -0.25&lt;br&gt;&#xA;  [2,] -0.25  1.00&lt;br&gt;&#xA;  &gt; inv(c)&lt;br&gt;&#xA;  Error: could not find function &quot;inv&quot;&lt;br&gt;&#xA;  &gt; solve(c)&lt;br&gt;&#xA;           [,1]      [,2]&lt;br&gt;&#xA;  [1,] 1.0666667 0.2666667&lt;br&gt;&#xA;  [2,] 0.2666667 1.0666667&lt;br&gt;&#xA;  &gt; solve(c)*c&lt;br&gt;&#xA;             [,1]        [,2]&lt;br&gt;&#xA;  [1,]  1.06666667 -0.06666667&lt;br&gt;&#xA;  [2,] -0.06666667  1.06666667&lt;br&gt;&#xA;  &gt; qr.solve(c)*c&lt;br&gt;&#xA;             [,1]        [,2]&lt;br&gt;&#xA;  [1,]  1.06666667 -0.06666667&lt;br&gt;&#xA;  [2,] -0.06666667  1.06666667  &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="1005" LastActivityDate="2010-11-20T02:17:10.947" Title="Inverse of matrix in R" Tags="&lt;r&gt;&lt;matrix-inverse&gt;" AnswerCount="2" />
  <row Id="4748" PostTypeId="2" ParentId="4747" CreationDate="2010-11-20T01:13:35.630" Score="7" ViewCount="" Body="&lt;p&gt;solve(c) does give the correct inverse. The issue with your code is that you are using the wrong operator for matrix multiplication. You should use solve(c) %*% c to invoke matrix multiplication in R. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;R performs element by element multiplication when you invoke solve(c) * c.&lt;/p&gt;&#xA;" LastActivityDate="2010-11-20T02:17:10.947" />
  <row Id="4749" PostTypeId="2" ParentId="4747" CreationDate="2010-11-20T01:44:55.550" Score="2" ViewCount="" Body="&lt;p&gt;You can use the function &lt;strong&gt;&lt;em&gt;ginv()&lt;/em&gt;&lt;/strong&gt; (generalized inverse) in the &lt;strong&gt;MASS&lt;/strong&gt; package&lt;/p&gt;&#xA;" OwnerUserId="438" LastActivityDate="2010-11-20T01:44:55.550" />
  <row Id="4750" PostTypeId="2" ParentId="4655" CreationDate="2010-11-20T03:29:37.537" Score="3" ViewCount="" Body="&lt;p&gt;It seems to me that if $f$ is strictly monotonic, $m \circ f=f \circ m$, and the question reduces to $\mu\circ f&amp;gt;f\circ\mu$, which is covered by Jensen's inequality. So strict convexity and strict monotonicity together would be a sufficient condition.&lt;/p&gt;&#xA;" OwnerUserId="2456" LastEditorUserId="2456" LastEditorDisplayName="" LastEditDate="2010-12-01T05:56:23.050" LastActivityDate="2010-12-01T05:56:23.050" CommentCount="2" />
  <row Id="4751" PostTypeId="2" ParentId="4741" CreationDate="2010-11-20T04:59:03.040" Score="1" ViewCount="" Body="&lt;p&gt;Try running a nonparametric smoother (also called a kernel density estimate) over your data to estimate the expected value  (and therefore proportion &#xA;&#xA;&lt;p&gt;If you do have covariate data, see how the nonparametric smooth compares to the regression model that onestop recommends.  A parametric model is usually a lot less wiggly than a simple smoothed estimate.&lt;/p&gt;&#xA;" OwnerUserId="1920" LastActivityDate="2010-11-20T04:59:03.040" />
  <row Id="4752" PostTypeId="2" ParentId="4735" CreationDate="2010-11-20T07:05:49.040" Score="0" ViewCount="" Body="&lt;p&gt;Notably while LSA and LSI use SVD to do their magic, there is a computationally and conceptually simpler method called HAL (Hyperspace Analogue to Language) that sifts through text keeping track of preceding and subsequent contexts.  Vectors are extracted from these (often weighted) co-occurrence matrices and specific words are selected to index the semantic space.  In many ways I'm given to understand it performs as well as LSA without requiring the mathematically/conceptually complex step of SVD.  See Lund &amp;amp; Burgess, 1996 for details.&lt;/p&gt;&#xA;" OwnerUserId="196" LastActivityDate="2010-11-20T07:05:49.040" CommentCount="2" />
  <row Id="4753" PostTypeId="1" CreationDate="2010-11-20T07:22:34.117" Score="4" ViewCount="218" Body="&lt;p&gt;I am interested in finding some practical (and reasonably well accepted) techniques for finding the underlying factors of a sparse matrix.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Specifically, I have a very large sparse matrix whose cells appear to be populated from an approximately geometric distribution.  In its natural form the matrix is square.  The values in the cells represent item x item co-occurrences under case 1 over the diagonal and under case 2 under the diagonal.  If necessary I can subset the matrix to particularly interesting items in order to make it rectangular.  I believe that there are meaningful factors underlying this structure.  However my understanding is that because the matrix is sparse factor analysis is not an appropriate approach.  What approach can I take that will make it most likely that I can find interpretable patterns in the data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I saw that there was &lt;a href=&quot;http://stats.stackexchange.com/questions/4267/essential-papers-on-matrix-decompositions&quot;&gt;another question&lt;/a&gt; asking for references on sparse variants of PCA, but I think I'm looking for something more akin to an obliquely rotated factor solution.  I'm willing to dig into suggested readings somewhat, but my prior experience with factor analysis (and related techniques) is limited, and I prefer a relatively straightforward answer (one with R code is even better).&lt;/p&gt;&#xA;" OwnerUserId="196" LastActivityDate="2011-05-26T15:05:23.787" Title="How can one extract meaningful factors from a sparse matrix?" Tags="&lt;r&gt;&lt;pca&gt;&lt;factor-analysis&gt;&lt;matrix-decomposition&gt;" AnswerCount="4" />
  <row Id="4754" PostTypeId="2" ParentId="4597" CreationDate="2010-11-20T07:53:17.650" Score="1" ViewCount="" Body="&lt;p&gt;A moving standard deviation sounds like a reasonable thing to use... here is a toy example in poorly written untested poorly optimized pseudo-C, things may go out of bounds or not work as I expect, but you should get the general idea:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;const int NPixelColumns; //The number of pixels columns&#xA;const int WindowSize; //The size of the moving window for the standard deviation&#xA;double BrightnessVals[NPixelColumns]; //Someplace to store your data initially&#xA;int startIndex; //Where the moving window starts&#xA;int lcv; //Generic loop control variable&#xA;&#xA;for (startIndex = 0; startIndex++; startIndex &amp;lt; (NPixelColumns-WindowSize))&#xA;{&#xA;   int endIndex = startIndex + (WindowSize-1);&#xA;   double sum; //the sum of values in the windows&#xA;   double xbar; //the mean in the window&#xA;   double deltasq[WindowSize]; //the squared differences between the mean and the value&#xA;   double SS=0; //the sum of deltasq&#xA;   for (lcv = startIndex; lcv++; lcv &amp;lt;= endIndex)&#xA;   {&#xA;       sum += BrightnessVals[lcv];&#xA;   }&#xA;   xbar = sum/WindowSize;&#xA;   for (lcv = 0; lcv++; lcv &amp;lt; WindowSize)&#xA;   {&#xA;       deltasq[lcv] = pow(BrightnessVals[startIndex+lcv]-xbar,2);&#xA;       SS += deltasq[lcv];&#xA;   }&#xA;   printf(&quot;At step %i the moving SD is: %f&quot;, startIndex, SS/sqrt(WindowSize-1));&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In R this kind of thing is a snap:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sdwindow &amp;lt;- function(start,end,data)&#xA;{&#xA;    return(sd(data[start:end]))&#xA;}&#xA;nsamp &amp;lt;- 1000 #The number of samples to look over&#xA;windowsize &amp;lt;- 10 #The size of the window to get the SD of&#xA;x &amp;lt;- rnorm(nsamp) #Sample data&#xA;start &amp;lt;- 1:(nsamp-windowsize) #starting points for the window&#xA;end &amp;lt;- (windowsize+1):nsamp #ending points for the window&#xA;doit &amp;lt;- Vectorize(sdwindow, vectorize.args = c(&quot;start&quot;,&quot;end&quot;)) #save me the trouble of figuring out mapply for the nth time.&#xA;doit(start,end,x) #generate the result&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="196" LastEditorUserId="196" LastEditorDisplayName="" LastEditDate="2010-11-20T08:07:46.623" LastActivityDate="2010-11-20T08:07:46.623" CommentCount="1" />
  <row Id="4756" PostTypeId="1" CreationDate="2010-11-20T12:05:26.533" Score="3" ViewCount="270" Body="&lt;p&gt;I have a random sample of Bernoulli random variables $X_1 ... X_N$, where $X_i$ are i.i.d.r.v. and $P(X_i = 1) = p$, and $p$ is an unknown parameter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Obviously, one can find an estimate for $p$: $\hat{p}:=(X_1+\dots+X_N)/N$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is how can I build a confidence interval for $p$?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you!&lt;/p&gt;&#xA;" OwnerUserId="2029" LastActivityDate="2011-01-12T02:58:18.993" Title="Confidence interval for Bernoulli sampling" Tags="&lt;distributions&gt;&lt;hypothesis-testing&gt;&lt;confidence-interval&gt;" AnswerCount="2" />
  <row Id="4757" PostTypeId="2" ParentId="4756" CreationDate="2010-11-20T12:21:25.060" Score="2" ViewCount="" Body="&lt;p&gt;The wikipedia has details on how to calculate &lt;a href=&quot;http://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval&quot; rel=&quot;nofollow&quot;&gt;confidence intervals for bernoulli sampling&lt;/a&gt;.&lt;/p&gt;&#xA;" LastActivityDate="2010-11-20T15:01:45.250" CommentCount="2" />
  <row Id="4759" PostTypeId="1" AcceptedAnswerId="4782" CreationDate="2010-11-20T14:57:19.843" Score="4" ViewCount="467" Body="&lt;p&gt;Hi guys, just for curiosity...&#xA;What language is used most here?&#xA;R? MATLAB? Python? Java?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What for prototype or for production?&#xA;For example I think MATLAB is mostly used for prototyping, python for both prot. and production...&lt;/p&gt;&#xA;" OwnerUserId="2046" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-11-21T20:26:44.323" LastActivityDate="2010-11-22T15:36:31.757" CommunityOwnedDate="2010-11-20T19:17:18.483" Title="What programming language for statistical inference?" Tags="&lt;r&gt;&lt;matlab&gt;&lt;python&gt;&lt;java&gt;" AnswerCount="5" CommentCount="2" FavoriteCount="1" />
  <row Id="4760" PostTypeId="2" ParentId="4734" CreationDate="2010-11-20T16:31:27.420" Score="3" ViewCount="" Body="&lt;p&gt;Essentially, principal components analysis breaks down the data into chunks which represent the variance of your matrix.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Factor analysis does the same, BUT it only examines the variance which is common to multiple items. Basically, EFA is a tool for determining latent structure, while PCA is a tool for reducing the number of items. Both are useful, but FA tends to be more useful (for me, at least). &lt;/p&gt;&#xA;" OwnerUserId="656" LastActivityDate="2010-11-20T16:31:27.420" />
  <row Id="4761" PostTypeId="2" ParentId="4759" CreationDate="2010-11-20T19:19:08.980" Score="5" ViewCount="" Body="&lt;p&gt;It should be clear by &lt;a href=&quot;http://stats.stackexchange.com/tags&quot;&gt;looking at the most popular tags&lt;/a&gt; that R is the most popular language on this site.  Whether that makes it the most popular language for statistical analysis can't be inferred directly, but one might suppose as much.&lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-11-20T19:19:08.980" CommunityOwnedDate="2010-11-20T19:19:08.980" />
  <row Id="4762" PostTypeId="1" AcceptedAnswerId="7383" CreationDate="2010-11-20T20:22:06.530" Score="1" ViewCount="411" Body="&lt;p&gt;Some googling revealed that doing the F-test for Lack-of-Fit in SPSS is not so trivial. It seems one has to “trick” SPSS to do that. See for example &lt;a href=&quot;http://www.math.umt.edu/olear/STAT458/Lab%205.pdf&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt;. Can anybody suggest a better source of information on how this can be done? I have SPSS 16. Of course I know it can be easily done using R but I am interested in the SPSS way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks &lt;/p&gt;&#xA;" OwnerUserId="339" LastActivityDate="2011-02-18T20:40:13.133" Title="F-test for Lack-of-Fit in SPSS" Tags="&lt;regression&gt;&lt;anova&gt;&lt;spss&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="4763" PostTypeId="2" ParentId="4564" CreationDate="2010-11-20T21:51:48.870" Score="2" ViewCount="" Body="&lt;p&gt;GBP includes the junction tree algorithm as a special case, and since junction tree is exact, GBP will be exact whenever the region graph corresponds to a junction tree.  This is the only general case where GBP is exact, as shown by Theorem 14 of &lt;a href=&quot;http://www.eecs.berkeley.edu/~ananth/2002+/Payam/submittedkikuchi.pdf&quot; rel=&quot;nofollow&quot;&gt;Pakzad and Anantharam (Neural Computation, 2005)&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2074" LastEditorUserId="2074" LastEditorDisplayName="" LastEditDate="2010-11-20T23:51:23.590" LastActivityDate="2010-11-20T23:51:23.590" CommentCount="0" />
  <row Id="4764" PostTypeId="2" ParentId="2772" CreationDate="2010-11-20T23:45:03.373" Score="0" ViewCount="" Body="&lt;p&gt;How &lt;em&gt;could&lt;/em&gt; they be different if they are numerically equivalent?&lt;/p&gt;&#xA;" OwnerUserId="1146" LastActivityDate="2010-11-20T23:45:03.373" />
  <row Id="4765" PostTypeId="2" ParentId="4759" CreationDate="2010-11-21T02:00:15.490" Score="5" ViewCount="" Body="&lt;p&gt;Well, you can PAY for MATLAB, and then either (1) program the stuff you really need from the ground up or (2) PAY MORE for MATLAB toolboxes. And discover that doing useful statistics in MATLAB was an afterthought handled in the increasingly less useful Statistics Toolbox. Or...you can download R for FREE and search for (and find!) the packages you need, which you can also download for FREE.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lots of small scale production stuff can be done in R.  If you're doing something really big (think US census), you probably need to go learn SAS--and get your employer to pay for it.&lt;/p&gt;&#xA;" OwnerUserId="1920" LastActivityDate="2010-11-21T02:00:15.490" CommunityOwnedDate="2010-11-21T02:00:15.490" />
  <row Id="4766" PostTypeId="1" CreationDate="2010-11-21T04:17:20.387" Score="3" ViewCount="325" Body="&lt;p&gt;Randomized SVD decomposes a matrix by extracting the first k singular values/vectors using k+p random projections. This works surprisingly well for large matrices.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question concerns the singular values that are output from the algorithm. Why aren't the values equal to the first k-singular values if you do the full SVD?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Below I have a simple implementation in R. Any suggestions on improving the performance would be appreciated.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; rsvd = function(A, k=10, p=5){&#xA;       n = nrow(A)&#xA;       y = A %*% matrix(rnorm(n * (k+p)), nrow=n)&#xA;       q = qr.Q(qr(y))&#xA;       b = t(q) %*% A&#xA;       svd = svd(b)&#xA;       list(u=q %*% svd$u, d=svd$d, v=svd$v)&#xA;    }&#xA;&#xA;    &amp;gt; set.seed(10)&#xA;&#xA;    &amp;gt; A &amp;lt;- matrix(rnorm(500*500),500,500)&#xA;&#xA;    &amp;gt; svd(A)$d[1:15]&#xA;     [1] 44.94307 44.48235 43.78984 43.44626 43.27146 43.15066 42.79720 42.54440 42.27439 42.21873 41.79763 41.51349 41.48338 41.35024 41.18068&#xA;&#xA;    &amp;gt; rsvd.o(A,10,5)$d&#xA;     [1] 34.83741 33.83411 33.09522 32.65761 32.34326 31.80868 31.38253 30.96395 30.79063 30.34387 30.04538 29.56061 29.24128 29.12612 27.61804&#xA;&#xA;    B &amp;lt;- matrix(rnorm(500*50),500,500)  # rank 50&#xA;&#xA;&amp;gt; rsvd(B,10,5)$d&#xA; [1] 86.48035 83.02114 81.03988 80.04358 77.24979 76.10945 74.47357 74.08382&#xA; [9] 72.85898 72.06897 69.59526 67.70750 66.53867 62.96446 61.50838&#xA;&#xA;&amp;gt; svd(B)$d[1:15]&#xA; [1] 92.44779 91.47689 88.71948 88.08170 87.24533 85.13312 84.14741 83.71757&#xA; [9] 82.80832 81.43005 80.73903 79.92959 78.87421 78.33509 77.38431&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As Joris pointed out, I have this posted on stackoverflow as well. you can find the revelant conversation here&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://stackoverflow.com/questions/4224031/randomized-svd-singular-values&quot;&gt;http://stackoverflow.com/questions/4224031/randomized-svd-singular-values&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also see the relevant paper by Martinsson et al: &lt;a href=&quot;http://dx.doi.org/10.1016/j.acha.2010.02.003&quot; rel=&quot;nofollow&quot;&gt;A randomized algorithm for the decomposition of matrices&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2078" LastActivityDate="2010-11-28T16:01:55.277" Title="Randomized SVD and singular values" Tags="&lt;matrix-decomposition&gt;&lt;svd&gt;" AnswerCount="2" CommentCount="7" FavoriteCount="1" />
  <row Id="4767" PostTypeId="2" ParentId="4695" CreationDate="2010-11-21T04:25:37.277" Score="2" ViewCount="" Body="&lt;p&gt;This is a pretty general problem in time series analysis. I'd probably start by looking at some descriptive statistics like the cross-correlation to see if the samples are roughly independent over time. You could also test whether the correlation between successive samples is significant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or you could go the model-fitting route in which case one simple thing to do is to fit an auto-regressive model with some order k and then do model comparison versus the static model. If you assume that $\theta$ just follows a Gaussian random walk, then model you're describing is exactly a Kalman filter. So that might be another thing to look at.&lt;/p&gt;&#xA;" OwnerUserId="2077" LastActivityDate="2010-11-21T04:25:37.277" />
  <row Id="4768" PostTypeId="1" AcceptedAnswerId="4772" CreationDate="2010-11-21T05:20:48.913" Score="7" ViewCount="478" Body="&lt;p&gt;I was asked this question during an interview for a trading position with a proprietary trading firm. I would very much like to know the answer to this question and the intuition behind it. Thank you!  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Amoeba Question: &#xA;A population of amoebas starts with 1. After 1 period that amoeba can divide into 1, 2, 3, or 0 (it can die) with equal probability. What is the probability that the entire population dies out eventually?&lt;/strong&gt; &lt;/p&gt;&#xA;" OwnerUserId="2079" LastEditorUserId="2079" LastEditorDisplayName="" LastEditDate="2010-11-21T06:10:39.223" LastActivityDate="2010-11-21T18:41:24.833" Title="Amoeba Interview Question" Tags="&lt;probability&gt;" AnswerCount="3" CommentCount="4" FavoriteCount="3" />
  <row Id="4769" PostTypeId="2" ParentId="4768" CreationDate="2010-11-21T05:45:09.687" Score="2" ViewCount="" Body="&lt;p&gt;This sounds related to the &lt;a href=&quot;http://en.wikipedia.org/wiki/Galton%E2%80%93Watson_process&quot; rel=&quot;nofollow&quot;&gt;Galton Watson&lt;/a&gt; process, originally formulated to study the survival of surnames. The probability depends on the expected number of sub-amoebas after a single division. In this case that expected number is $3/2,$ which is greater than the critical value of $1$, and thus the probability of extinction is less than $1$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;By considering the expected number of amoeba after $k$ divisions, one can easily show that if the expected number after one division is less than $1$, the probability of extinction is $1$. The other half of the problem, I am not so sure about.&lt;/p&gt;&#xA;" OwnerUserId="795" LastEditorUserId="795" LastEditorDisplayName="" LastEditDate="2010-11-21T18:05:47.807" LastActivityDate="2010-11-21T18:05:47.807" />
  <row Id="4770" PostTypeId="2" ParentId="4267" CreationDate="2010-11-21T09:44:03.770" Score="0" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Witten, Tibshirani - Penalized matrix decomposition&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.biostat.washington.edu/~dwitten/Papers/pmd.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.biostat.washington.edu/~dwitten/Papers/pmd.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://cran.r-project.org/web/packages/PMA/index.html&quot; rel=&quot;nofollow&quot;&gt;http://cran.r-project.org/web/packages/PMA/index.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Martinsson, Rokhlin, Szlam, Tygert - Randomized SVD&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://cims.nyu.edu/~tygert/software.html&quot; rel=&quot;nofollow&quot;&gt;http://cims.nyu.edu/~tygert/software.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://cims.nyu.edu/~tygert/blanczos.pdf&quot; rel=&quot;nofollow&quot;&gt;http://cims.nyu.edu/~tygert/blanczos.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2078" LastEditorUserId="2078" LastEditorDisplayName="" LastEditDate="2010-11-22T19:18:52.707" LastActivityDate="2010-11-22T19:18:52.707" CommunityOwnedDate="2010-11-21T09:44:03.770" CommentCount="2" />
  <row Id="4771" PostTypeId="2" ParentId="2806" CreationDate="2010-11-21T09:56:02.610" Score="0" ViewCount="" Body="&lt;p&gt;you could trying using a couple of options.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1- &lt;strong&gt;Penalized Matrix Decomposition&lt;/strong&gt;. You apply some penalty constraints on the u's and v's to get some sparsity. Quick algorithm that has been used on genomics data&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See Whitten Tibshirani. They also have an R-pkg. &quot; A penalized matrix decomposition, with applications to sparse principal components and canonical correlation analysis.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2- &lt;strong&gt;Randomized SVD&lt;/strong&gt;. Since SVD is a master algorithm, find a very quick approximation might be desirable, especially for exploratory analysis. Using randomized SVD, you can do PCA on huge datasets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See Martinsson, Rokhlin, and Tygert &quot;A randomized algorithm for the decomposition of matrices&quot;. Tygert has code for a very fast implementation of PCA.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Below is a simple implementation of randomized SVD in R.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ransvd = function(A, k=10, p=5) {&#xA;  n = nrow(A)&#xA;  y = A %*% matrix(rnorm(n * (k+p)), nrow=n)&#xA;  q = qr.Q(qr(y))&#xA;  b = t(q) %*% A&#xA;  svd = svd(b)&#xA;  list(u=q %*% svd$u, d=svd$d, v=svd$v)&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2078" LastActivityDate="2010-11-21T09:56:02.610" />
  <row Id="4772" PostTypeId="2" ParentId="4768" CreationDate="2010-11-21T11:47:10.303" Score="12" ViewCount="" Body="&lt;p&gt;Cute problem. This is the kind of stuff that probabilists do in their heads for fun.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The technique is to assume that there is such a probability of extinction, call it $P$.  Then, looking at a one-deep decision tree for the possible outcomes we see--using the Law of Total Probability--that&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P=\frac{1}{4} + \frac{1}{4}P + \frac{1}{4}P^2 + \frac{1}{4}P^3$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;assuming that, in the cases of 2 or 3 &quot;offspring&quot; their extinction probabilities are IID.  This equation has two feasible roots, $1$ and $\sqrt{2}-1$.  Someone smarter than me might be able to explain why the $1$ isn't plausible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Jobs must be getting tight -- what kind of interviewer expects you to solve cubic equations in your head?&lt;/p&gt;&#xA;" OwnerUserId="1920" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-11-21T18:41:24.833" LastActivityDate="2010-11-21T18:41:24.833" CommentCount="5" />
  <row Id="4773" PostTypeId="2" ParentId="4768" CreationDate="2010-11-21T11:48:51.067" Score="7" ViewCount="" Body="&lt;p&gt;Some back of the envelope calculation (litterally - I had an envelope lying around on my desk) gives me a probability of 42/111 (38%) of never reaching a population of 3.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I ran a quick Python simulation, seeing how many populations had died off by 20 generations (at which point they usually either died out or are in the thousands), and got 4164 dead out of 10000 runs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the answer is 42%.&lt;/p&gt;&#xA;" OwnerUserId="1737" LastActivityDate="2010-11-21T11:48:51.067" CommentCount="1" />
  <row Id="4774" PostTypeId="2" ParentId="4172" CreationDate="2010-11-21T15:17:02.290" Score="1" ViewCount="" Body="&lt;p&gt;First of all, a question if interest: If you want the measurements not to differ significantly, why change the tool at all? Simply to get more frequent measurements, or for economical reasons?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now for the reply.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I do not entirely understand how you gathered the data. You say both instruments were colocated, and that instrument B gathered data more frequently. For comparison purposes, you would want those measurements of B that were made at the same place as A. As you cannot match them exactly, you need to interpolate. For this, I would use timestamps, but assume you don't have them, as you went with the GPS coordinates (although you did say &quot;4-minute interval&quot;). I'll assume it's reasonable to assume the GPS coordinates are accurate for both instruments, and that the effect measured doesn't noticeably vary over very small distances, such as the location the instruments are in on the vehicle. For interpolation, you need a model of the variability of the measured effect between datapoints. You say you used the nearest neighbour. That's perfectly acceptable, but I would probably go with a linear interpolant myself. Using all datapoints would require a very good model, which you may not yet have.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for the comparisons:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;You found that the instruments do differ, depending on the true value. This means at least one of the instruments is biased, but you will not be able to tell from the measurements alone which one. You may be able to fit a generalized linear model &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The correlation you measured is actually not all that good for what is supposed to be 1. The Pearson correlation does not require normality, but it does assume the variables are stationary and independent - which yours aren't.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Large N isn't a problem, it's a good thing. Your large t-value also tells you that if your variable is normally distributed, the instruments are not equivalent.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;In testing the reported categories, start with a simple crosstabulation to see if any particular mismatch is more likely than others. That the time series is autocorrelated doesn't matter when looking just at the differences. To test for independence of the categories, you first need to determine whether categories depend on the discretized variable, and if not, whether they nevertheless are ordered or merely unordered labels. The choice of test comes after you characterize how you think the data &lt;em&gt;should&lt;/em&gt; behave.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="2456" LastActivityDate="2010-11-21T15:17:02.290" CommentCount="1" />
  <row Id="4775" PostTypeId="1" AcceptedAnswerId="4776" CreationDate="2010-11-21T16:24:16.763" Score="8" ViewCount="261" Body="&lt;p&gt;I was wondering if it is possible to do symbolic computation in R? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was hoping to get the inverse of a symbolic covariance matrix of 3D Gaussian distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also can I do symbolic integration and differentiation in R?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks and regards!&lt;/p&gt;&#xA;" OwnerUserId="1005" LastActivityDate="2010-11-21T23:40:42.427" Title="Symbolic computation in R?" Tags="&lt;r&gt;" AnswerCount="3" FavoriteCount="1" />
  <row Id="4776" PostTypeId="2" ParentId="4775" CreationDate="2010-11-21T16:35:20.947" Score="8" ViewCount="" Body="&lt;p&gt;Yes.  There is the &lt;a href=&quot;http://cran.r-project.org/web/packages/Ryacas/index.html&quot; rel=&quot;nofollow&quot;&gt;Ryacas package&lt;/a&gt; which is hosted on Google Code &lt;a href=&quot;http://code.google.com/p/ryacas/&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.  Ryacas has recently been expanded/converted to the rMathpiper package which is hosted &lt;a href=&quot;http://code.google.com/p/rmathpiper/&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.  I have used Ryacas and it is straightforward, but you will need to install &lt;a href=&quot;http://yacas.sourceforge.net/&quot; rel=&quot;nofollow&quot;&gt;Yacas&lt;/a&gt; in order for it to work (Yacas does all the heavy lifting; Ryacas is just an R interface to Yacas).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is also the rSymPy project hosted on Google Code &lt;a href=&quot;http://rsympy.googlecode.com/&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.  I haven't tried this one.  The idea is similar, though, link to the sympy CAS which does the symbolic work.&lt;/p&gt;&#xA;" OwnerUserId="1108" LastActivityDate="2010-11-21T16:35:20.947" />
  <row Id="4777" PostTypeId="2" ParentId="4775" CreationDate="2010-11-21T17:29:00.630" Score="6" ViewCount="" Body="&lt;p&gt;Some things are also in base R --- see &lt;code&gt;help(deriv)&lt;/code&gt; or &lt;code&gt;help(D)&lt;/code&gt;.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A simple example from that help page:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;R&amp;gt; trig.exp &amp;lt;- expression(sin(cos(x + y^2)))&#xA;R&amp;gt; ( D.sc &amp;lt;- D(trig.exp, &quot;x&quot;) )&#xA;-(cos(cos(x + y^2)) * sin(x + y^2))&#xA;R&amp;gt;      all.equal(D(trig.exp[[1]], &quot;x&quot;), D.sc)&#xA;[1] TRUE&#xA;R&amp;gt; &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="334" LastActivityDate="2010-11-21T17:29:00.630" />
  <row Id="4778" PostTypeId="2" ParentId="4775" CreationDate="2010-11-21T17:51:35.997" Score="2" ViewCount="" Body="&lt;p&gt;It makes more sense to use a &quot;real&quot; CAS like &lt;a href=&quot;http://en.wikipedia.org/wiki/Maxima_%28software%29&quot; rel=&quot;nofollow&quot;&gt;Maxima&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="1966" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-11-21T23:40:42.427" LastActivityDate="2010-11-21T23:40:42.427" CommentCount="1" />
  <row Id="4779" PostTypeId="2" ParentId="4700" CreationDate="2010-11-21T18:00:46.553" Score="1" ViewCount="" Body="&lt;p&gt;The distinction is only meaningful in the context of non-Bayesian statistics. In Bayesian statistics, all model parameters are &quot;random&quot;. &lt;/p&gt;&#xA;" OwnerUserId="1966" LastActivityDate="2010-11-21T18:00:46.553" />
  <row Id="4780" PostTypeId="2" ParentId="4766" CreationDate="2010-11-21T18:37:03.600" Score="3" ViewCount="" Body="&lt;p&gt;I do not think the singular values should match those of the full matrix. You are computing an approximation of the input matrix by projection onto $k+p$ random vectors. For a rank $k+p$ matrix to approximate a rank $n \gg k+p$ matrix, the trace should probably be the same, but then if the first $k$ singular values are to overlap, you have to push a lot of 'variance' to the last $p$ singular values of the approximation (probably so many that they are no longer the least significant singular values).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another way of looking at this is one is approximating $A = U\Sigma V&amp;#39;$ by another decomposition, $T \Gamma W&amp;#39;$. We should not expect a fast randomized algorithm to magically work such that $T$ is the first $k$ columns of $U$, $\Gamma$ is a submatrix of $\Sigma$, etc.  &lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-11-21T18:37:03.600" CommentCount="1" />
  <row Id="4781" PostTypeId="2" ParentId="4267" CreationDate="2010-11-21T19:28:40.950" Score="1" ViewCount="" Body="&lt;p&gt;Maybe, you can find interesting &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://ttic.uchicago.edu/~nati/Publications/thesis.pdf&quot; rel=&quot;nofollow&quot;&gt;[Learning with Matrix Factorizations]&lt;/a&gt; PhD thesis by Nathan Srebro,&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.sze.hu/~gtakacs/download/kdd_netflix_workshop_2008.pdf&quot; rel=&quot;nofollow&quot;&gt;[Investigation of Various Matrix Factorization Methods for Large Recommender Systems]&lt;/a&gt;, Gábor Takács et.al. and almost the same technique described &lt;a href=&quot;http://sifter.org/~simon/journal/20061211.html&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;  &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The last two links show how &lt;strong&gt;sparse&lt;/strong&gt; matrix factorizations are used in Collaborative Filtering. However, I believe that SGD-like factorization algorithms can be useful somewhere else (at least they are extremely easy to code)&lt;/p&gt;&#xA;" OwnerUserId="1725" LastEditorUserId="1725" LastEditorDisplayName="" LastEditDate="2010-11-23T09:02:09.487" LastActivityDate="2010-11-23T09:02:09.487" CommunityOwnedDate="2010-11-21T19:28:40.950" />
  <row Id="4782" PostTypeId="2" ParentId="4759" CreationDate="2010-11-21T21:02:25.647" Score="5" ViewCount="" Body="&lt;p&gt;I couldnt agree more with a vote for R. R is the &quot;Lingua Franca&quot; of the statistics world. It is the definition of cutting edge, while most packages for MATLAB and SAS take several months. The language is very simple to understand as opposed to SAS. It also gives you the power to connect with C/C++/Python and databases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider Revolution Analytics version of R for a bit more performance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.revolutionanalytics.com/products/revolution-r.php&quot; rel=&quot;nofollow&quot;&gt;http://www.revolutionanalytics.com/products/revolution-r.php&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2078" LastActivityDate="2010-11-21T21:02:25.647" CommunityOwnedDate="2010-11-21T21:02:25.647" CommentCount="3" />
  <row Id="4783" PostTypeId="1" CreationDate="2010-11-21T21:21:46.837" Score="7" ViewCount="315" Body="&lt;p&gt;I have some cumulative frequency data. A line $y=ax+b$ looks like it fits the data extremely well, but there is cyclic/periodic wiggle in the line. I would like to estimate when the cumulative frequency will reach a certain value $c$. When I plot the residuals vs. fitted values, I get a beautiful sinusoidal behavior. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, to add another complication, note that in the residuals plots&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/3GIg5.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;there are two cycles that have lower values than the others, which represents a weekend effect that also must be taken into account.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, where do I go from here? How can I combine some cosine, sine, or cyclic term into a regression model to approx. estimate when the cumulative frequency will equal $c$?&lt;/p&gt;&#xA;" OwnerUserId="2083" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-11-21T23:08:42.967" LastActivityDate="2010-11-22T17:43:59.917" Title="How to add periodic component to linear regression model?" Tags="&lt;time-series&gt;&lt;regression&gt;" AnswerCount="4" FavoriteCount="1" />
  <row Id="4784" PostTypeId="2" ParentId="4783" CreationDate="2010-11-21T21:55:09.817" Score="7" ViewCount="" Body="&lt;p&gt;You could try the wonderful &lt;code&gt;stl()&lt;/code&gt; method -- it decomposes (using iterated &lt;code&gt;loess()&lt;/code&gt; fitting) into trend and seasonal and remainder. This may just pick up your oscillations here. &lt;/p&gt;&#xA;" OwnerUserId="334" LastActivityDate="2010-11-21T21:55:09.817" />
  <row Id="4785" PostTypeId="2" ParentId="4783" CreationDate="2010-11-21T22:57:47.480" Score="6" ViewCount="" Body="&lt;p&gt;If you know the frequency of the oscillation, you can include two additional predictors, sin(2π w t) and cos(2π w t) -- set w to get the desired wavelength -- and this will model the oscillation.  You need both terms to fit the amplitude and the phase angle.  If there is more than one frequency, you will need a sine and cosine term for each frequency.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you don't know what the frequencies are, the standard way to isolate multiple frequencies is to detrend the data (get the residuals from the linear fit, as you have done) and run a discrete Fourier transform against the residuals.  A quick and dirty way to do this is in MS-Excel, which has a Fourier Analysis tool in the Data Analysis Add-In.  Run the analysis against the residuals, take the absolute value of the transforms, and bar graph the result.  The peaks will be your major frequency components that you want to model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you add these cyclic predictors, pay close attention to their p-values in your regression, and don't overfit.  Use only those frequencies that are statistically significant.  Unfortunately, this may make fitting the low frequencies a little difficult.&lt;/p&gt;&#xA;" OwnerUserId="1920" LastActivityDate="2010-11-21T22:57:47.480" CommentCount="1" />
  <row Id="4786" PostTypeId="1" CreationDate="2010-11-21T23:09:28.083" Score="5" ViewCount="216" Body="&lt;p&gt;I am looking at using Interior Point method for optimizing a convex function. The convex function is basically the log-likelihood of a binary logistic regression model. Can I use this technique?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In generally, is there anything that prevents applying a constrained optimization technique to an unconstrained problem? From what I think, an unconstrained problem is just a constrained problem without the constraints and thus should be solvable using these techniques.&lt;/p&gt;&#xA;" OwnerUserId="2071" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-11-21T23:10:19.877" LastActivityDate="2010-11-22T12:52:59.547" Title="Can constrained optimization techniques be applied to unconstrained problems?" Tags="&lt;optimization&gt;" AnswerCount="3" />
  <row Id="4787" PostTypeId="2" ParentId="4786" CreationDate="2010-11-22T00:09:23.823" Score="2" ViewCount="" Body="&lt;p&gt;As far as I know, there is no reason to stop you from applying constrained optimization to an unconstrainted problem. However, this may not be a great idea in terms of computational complexity and convergence. For example, fitting a logistic regression model can done efficiently with the Newton-Raphson approach (or the Fisher scoring variant). I am not sure if there is much to gain with the interior point approach in this particular case.&lt;/p&gt;&#xA;" OwnerUserId="530" LastActivityDate="2010-11-22T00:09:23.823" CommentCount="4" />
  <row Id="4788" PostTypeId="2" ParentId="4786" CreationDate="2010-11-22T05:03:20.090" Score="1" ViewCount="" Body="&lt;p&gt;The general sense in optimization is that if you have a convex function and no constraints, you want to use the &quot;powerful stuff&quot;, gradient descent, Newton, etc. Without constraints interior point methods are not very good (competitive). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In particular for the problem you're studying (binary logistic regression) you should consider trying simple stochastic gradient descent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Nothing really stops you from applying constrained optimization techniques to unconstrained problems. The same way nothing stops from pushing (instead of riding) your car to work. But you should definitely try interior point methods w/o constraints and convince yourself about it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally you mention that you want to try linear programming-based methods presumably without constraints, what you plan to do in this case I don't quite understand.&lt;/p&gt;&#xA;" OwnerUserId="1540" LastEditorUserId="1540" LastEditorDisplayName="" LastEditDate="2010-11-22T09:03:55.123" LastActivityDate="2010-11-22T09:03:55.123" CommentCount="2" />
  <row Id="4789" PostTypeId="2" ParentId="4759" CreationDate="2010-11-22T05:44:40.397" Score="2" ViewCount="" Body="&lt;p&gt;R and SAS have each there pros and cons. I think more statisticians need to embrace the fact that lots of great statistical software is available, rather than endlessly bicker about which is superior.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;R is free. SAS is very expensive. R gives you the ability to do just about anything. SAS may or may not. R and has amazing graphical abilities. Seeing SAS graphics makes it feel like 1985 all over again. SAS has great customer support. R support = hours of searching mailing list archives. Also with a name like &quot;R&quot;, search engine results are often poor. R is extremely slow and does not deal well with large data sets. SAS does fine with large data sets. SAS tends to be more robust. In my experience, when it comes to mixed effects modeling or anything involving design of experiments (such as analyzing crossover designs), SAS is superior.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For large scale, brute force simulations, I use Fortran. I used to use C, but have found  Fortran is much easier to use. I've never used MATLAB. If I need statistical power of R but the speed of Fortran, I will write the time-intensive operations (i.e. loops) in Fortran and call the subroutine from R.&lt;/p&gt;&#xA;" LastActivityDate="2010-11-22T05:44:40.397" CommunityOwnedDate="2010-11-22T05:44:40.397" CommentCount="3" />
  <row Id="4790" PostTypeId="2" ParentId="4663" CreationDate="2010-11-22T10:56:39.330" Score="2" ViewCount="" Body="&lt;p&gt;The &quot;no free lunch&quot; theorems suggest that there are no a-priori distinctions between statistical inference algorithms, i.e. whether LARS or LASSO works best depends on the nature of the particular dataset.  In practice then, it is best to try both and use some reliable estimator of generalisation performance to decide which to use in operation (or use an ensemble).  As the differences between LARS and LASSO are rather slight, the differences in performance are likely to be rather slight as well, but in general there is only one way to find out for sure!&lt;/p&gt;&#xA;" OwnerUserId="887" LastEditorUserId="887" LastEditorDisplayName="" LastEditDate="2010-11-22T11:25:15.910" LastActivityDate="2010-11-22T11:25:15.910" CommentCount="1" />
  <row Id="4793" PostTypeId="2" ParentId="4786" CreationDate="2010-11-22T12:52:59.547" Score="2" ViewCount="" Body="&lt;p&gt;As far as I'm concerned, constrained optimization is a less-than-optimal way of avoiding strong fluctuations in your parameters for the independents due to a bad model-specification. Pretty often a constraint is &quot;needed&quot; when the variance-covariance matrix is ill-structured, when there is a lot of (unaccounted) correlation between independents, when you have aliasing or near-aliasing in datasets, when you gave the model too many degrees of freedom, and so on. Basically, every condition that inflates the variance on the parameter estimates will cause an unconstrained method to behave poorly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can look at constrained optimization, but I reckon you should first take a look at your model if you believe constrained optimization is necessary. This for two reasons :&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;There's no way you can still rely on the inference, even on the estimated variances for your parameters&lt;/li&gt;&#xA;&lt;li&gt;You have no control over the amount of bias you introduce. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;So depending on the goal of the analysis, constrained optimization can be a sub-optimal solution (purely estimating the parameters) or inappropriate (when inference is needed).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On a side note, penalized methods (in this case penalized likelihoods) are specifically designed for these cases, and introduce the bias in a controlled manner where it is accounted for (mostly). Using these, there is no need to go into constrained methods, as the classic optimization algorithms will do a pretty good job. And with the correct penalization, inference is still valid in many cases. So I'd rather go for such a method instead of putting arbitrary constraints that are not backed up with an inferential framework. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My 2 cents, YMMV.&lt;/p&gt;&#xA;" OwnerUserId="1124" LastActivityDate="2010-11-22T12:52:59.547" CommentCount="5" />
  <row Id="4794" PostTypeId="2" ParentId="4783" CreationDate="2010-11-22T15:13:43.460" Score="3" ViewCount="" Body="&lt;p&gt;Let's begin by observing that ordinary least squares fitting for these data is likely inappropriate.  If the individual data being accumulated are assumed, as usual, to have random error components, then the error in the cumulative data (&lt;em&gt;not&lt;/em&gt; the &lt;a href=&quot;http://en.wikipedia.org/wiki/Cumulative_frequency&quot; rel=&quot;nofollow&quot;&gt;cumulative frequencies&lt;/a&gt;--that's something different than what you have) is the cumulative sum of all the error terms.  This makes the cumulative data heteroscedastic (they become more and more variable over time) and strongly positively correlated.  Because these data are so regularly behaved, and there's so much of them, there's little problem with the &lt;em&gt;fit&lt;/em&gt; you will get, but your estimates of errors, your predictions (which is what the question is all about), and especially your standard errors of prediction can be way off.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A standard procedure for analyzing such data starts with the original values.  Take the day-to-day differences to remove the higher-frequency sinusoidal component.  Take the weekly differences of those to remove a possible week-to-week cycle.  Analyze what's left.  &lt;a href=&quot;http://en.wikipedia.org/wiki/Autoregressive_integrated_moving_average&quot; rel=&quot;nofollow&quot;&gt;ARIMA&lt;/a&gt; modeling is a powerful flexible approach, but start simply: graph those differenced data to see what's going on, then move on from there.  Note, too, that with less than two weeks of data your estimates of the weekly cycle will be poor and this uncertainty will dominate the uncertainty in the predictions.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-11-22T15:13:43.460" />
  <row Id="4795" PostTypeId="2" ParentId="4759" CreationDate="2010-11-22T15:36:31.757" Score="4" ViewCount="" Body="&lt;p&gt;&quot;Popularity&quot; depends on the community and the definition of &quot;statistics&quot;.  World-wide, taking a broad view of &quot;statistical inference&quot; as including any methods of drawing conclusions or taking actions based on quantitative data, there is little question that &lt;strong&gt;&lt;a href=&quot;http://www.pcworld.com/businesscenter/article/166123/forrester_microsoft_office_in_no_danger_from_competitors.html&quot; rel=&quot;nofollow&quot;&gt;Excel&lt;/a&gt;&lt;/strong&gt; &lt;a href=&quot;http://blogs.technet.com/b/office2010/archive/2010/06/15/office-2010-availability.aspx&quot; rel=&quot;nofollow&quot;&gt;beats&lt;/a&gt; all other applications, including &lt;a href=&quot;http://www.kdnuggets.com/2010/06/software-popularity-of-data-analysis-software.html&quot; rel=&quot;nofollow&quot;&gt;R, SAS, Stata, SPSS, and S-Plus&lt;/a&gt;.  (The links point to different kinds of statistics, but they are highly suggestive, to say the least.)  Python and MATLAB aren't even blips in the statistics.  I am not saying that this is a good thing or that we should like it: that's just how it is and that's how it's going to stay for a very long time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We shouldn't draw any inferences from what may appear to be popular &quot;here&quot; in this forum.  Commercial software vendors support their own forums, so naturally a place like SE will favor people using less actively supported software, especially free, open-source, and academic solutions.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-11-22T15:36:31.757" CommunityOwnedDate="2010-11-22T15:36:31.757" />
  <row Id="4796" PostTypeId="1" AcceptedAnswerId="4797" CreationDate="2010-11-22T15:42:04.420" Score="1" ViewCount="175" Body="&lt;p&gt;To me, the two are similar in the sense that &lt;a href=&quot;http://.wikipedia.org/wiki/Slice_sampling&quot; rel=&quot;nofollow&quot;&gt;slice sampling&lt;/a&gt; is just &lt;a href=&quot;http://en.wikipedia.org/wiki/Gibbs_sampling&quot; rel=&quot;nofollow&quot;&gt;Gibbs sampling&lt;/a&gt; for the uniform distribution over the area under the plot of the density function. Is that right?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was wondering if someone can compare between &lt;a href=&quot;http://www.stat.purdue.edu/~jianzhan/notes/Gibbs.pdf&quot; rel=&quot;nofollow&quot;&gt;slice sampling and Gibbs sampling&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, in terms of rate of convergence, which one is better? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you can think about other aspects, please feel free to reply.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks and regards!&lt;/p&gt;&#xA;" OwnerUserId="1005" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-11-30T16:40:59.760" LastActivityDate="2010-11-30T16:40:59.760" Title="Comparison of Slice sampling and Gibbs sampling" Tags="&lt;bayesian&gt;&lt;mcmc&gt;&lt;gibbs&gt;&lt;simulation&gt;&lt;slice-sampler&gt;" AnswerCount="1" />
  <row Id="4797" PostTypeId="2" ParentId="4796" CreationDate="2010-11-22T15:54:00.333" Score="1" ViewCount="" Body="&lt;p&gt;I am not sure if the question is well posed. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you &lt;em&gt;can&lt;/em&gt; use both the gibbs sampler and slice sampling to sample from a posterior I would use the gibbs sampler as the slice sampler seems unnecessary to me. Use of a slice sampler introduces additional variables which at the very least increases run time for the sampler. So, I am not sure why one would use the slice sampler if we can use the gibbs sampler. If you cannot use the gibbs but you can use the slice then your question seems irrelevant. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, I am not sure why one would consider handicapping the sampler by using the slice when a gibbs sampler can be used. &lt;/p&gt;&#xA;" LastActivityDate="2010-11-22T15:54:00.333" CommentCount="2" />
  <row Id="4798" PostTypeId="2" ParentId="4783" CreationDate="2010-11-22T17:43:59.917" Score="2" ViewCount="" Body="&lt;p&gt;Clearly the dominant oscillation has period one day. Looks like there are also lower-frequency components relating to the day of the week, so add a component with frequency one week (i.e. one-seventh of a day) and its first few harmonics. That gives a model of the form:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\mbox{E}(y) = c + a_0 \cos(2\pi t) + b_0 \sin(2\pi t) + a_1 \cos(2 \pi t/7) + b_1 \sin(2 \pi t/7) + a_2 \cos(4 \pi t/7) + b_2 \sin(4 \pi t/7) + \ldots $$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&amp;ndash; assuming $t$ is measured in days. Here $y$ is the &lt;em&gt;raw&lt;/em&gt; data, not its cumulative sum.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-11-22T17:43:59.917" />
  <row Id="4799" PostTypeId="1" CreationDate="2010-11-22T18:41:41.273" Score="4" ViewCount="271" Body="&lt;p&gt;Where can I find a good proof that CRF based models and logistic regression based models are convex? Is there a general trick to test/prove that a model or objective function is convex?&lt;/p&gt;&#xA;" OwnerUserId="2071" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-11-23T15:24:42.790" LastActivityDate="2010-11-23T19:16:41.050" Title="Proof that CRF models and logistic models are convex functions" Tags="&lt;logistic-regression&gt;&lt;optimization&gt;" AnswerCount="2" />
  <row Id="4800" PostTypeId="2" ParentId="4799" CreationDate="2010-11-22T19:30:36.430" Score="4" ViewCount="" Body="&lt;p&gt;One trick is to rewrite objective functions in terms of functions which are known to be convex.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Objective function of ML trained log-linear model is a sum of negative log-likelihoods, so it's sufficient to show that negative log-likelihood for each datapoint is convex.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Considering datapoint fixed, we can write its negative log-likelihood term as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$-\langle \theta,\phi(y)\rangle+\log \sum_y \exp(\langle \theta,\phi(y)\rangle)$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First term is linear so it's sufficient to show that second term, known as the log-normalizer, is convex.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Write it as $f(\mathbf{g}(\mathbf{\theta}))$ where $f(\mathbf{y})=\log \sum_y \exp y$ and $g_y(\theta)=\langle \mathbf{\theta},\phi(y)\rangle$. Here $g$ is a linear function, and $f$ is a known convex function called log-sum-exp. See page 72 of Boyd's Convex Optimization &lt;a href=&quot;http://www.stanford.edu/~boyd/cvxbook/&quot; rel=&quot;nofollow&quot;&gt;book&lt;/a&gt;. Composition of a convex function and a linear function is convex, see section 3.2.2&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another approach is to use the fact that log-normalizer is the cumulant generating function. For instance see example 3.41 in Boyd's book, or Proposition 3.1 in Wainwright's &quot;Graphical models, exponential families, and variational inference&quot; &lt;a href=&quot;http://www.eecs.berkeley.edu/~wainwrig/Papers/WaiJor08_FTML.pdf&quot; rel=&quot;nofollow&quot;&gt;manuscript&lt;/a&gt;. This means that second derivative is the covariance matrix of sufficient statistic $\phi$ which by definition is positive semi-definite, which means that Hessian of the log-normalizer is positive semi-definite. Positive semi-definite Hessian guarantees the function is convex, see section 3.1.4 of Boyd's book.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Technically, the log-normalizer is not the traditional cumulant generating function. CGF is $g(\phi)=\log(Z(\theta+\phi))-\log(Z(\theta))$. However, derivative of log-normalizer evaluated at $\theta$ is the same as the derivative of the CGF evaluated at $\mathbf{0}$, so it produces cumulants just like CGF.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I couldn't find full proof of equivalence, usually people omit it because it's just several steps of uninspiring algebra. A very terse derivation for continuous output space is on page 5 of Xinhua Zhang's &quot;Graphical Models&quot; &lt;a href=&quot;http://dspace.anu.edu.au/bitstream/1885/49340/1/01front.pdf&quot; rel=&quot;nofollow&quot;&gt;thesis&lt;/a&gt;. I believe a saw full derivation in Lawrence D. Brown's &quot;Fundamentals of statistical exponential families&quot;&lt;/p&gt;&#xA;" OwnerUserId="511" LastEditorUserId="511" LastEditorDisplayName="" LastEditDate="2010-11-23T19:16:41.050" LastActivityDate="2010-11-23T19:16:41.050" />
  <row Id="4801" PostTypeId="1" CreationDate="2010-11-22T19:30:49.830" Score="0" ViewCount="37" Body="&lt;p&gt;The product of two unbiased estimates, x and y, from the same data is a biased estimator. However, I don't want to split the data into two sections. I would like to take the average estimate for x and y using different sections of the data. This to me seems intuitively similar to cross-validation and bootstrapping. I'd like to find some theoretical analysis on this process but I've not really been able to find any information. It seems like a pretty general idea that someone must have done before.&lt;/p&gt;&#xA;" LastActivityDate="2010-11-22T19:30:49.830" Title="Reusing samples for product estimators" Tags="&lt;estimation&gt;&lt;sample-size&gt;&lt;sampling&gt;" CommentCount="2" />
  <row Id="4802" PostTypeId="1" CreationDate="2010-11-22T21:25:42.370" Score="2" ViewCount="101" Body="&lt;p&gt;I am trying to determine if I can model my system as a M/M/1 queue and if so do the numbers I get from it help me at all.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can model my system like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/dLpOm.jpg&quot; alt=&quot;block diagram&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;System Description&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;A.&lt;/strong&gt; I want to spec out the max number of clients I can have connect to my system. Each client can generate at max 40 messages/second that are placed in the buffer. If link B goes down it will buffer all messages. For the sake of my math I am going to assume that it is actually generating a fixed 40 messages/second.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;B.&lt;/strong&gt; I am specifying that this link should be able to go down for 3 days, but normal operation spec this as a network connection that all clients will have to communicate on. This network connection may also have traffic unrelated to my system on it. The link is fast enough that it isn't a bottle neck at all. I have found that it can easily pass 500,000 messages in less then 10 seconds.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;C.&lt;/strong&gt; Allows for the clients to dump any data as fast as link B will allow. This queue can be as big as needed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;D.&lt;/strong&gt; This isn't really a link per say, block E pulls directly from the queue.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;E, F, and G.&lt;/strong&gt; This is where the fun comes. Initially every packet I received I placed a single SQL query to the database. I found that each query took about 5-15ms with a few taking 1ms and a few taking 600+ms. I don't have a model for this delay and suspect I will need one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I realized that this delay was going to drastically hurt my systems performance so I started looking into how to improve this performance. I found that if I grouped multiple messaged from C together and did a larger SQL query I was able to improve performance. When I grouped 10 messages per query I was observing 10-20ms, again with a few very short and a few very long. So 10-20ms/10messages gives me an effective 1-2ms per message instead of the 5-15ms I had before.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My Objectives&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Determine how to model E, F, and G. I figured the easiest and most accurate method would be by experimentation, but I am not sure what data I should be looking to collect.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Determine how many messages I should group together in order to minimize latency while maximizing the number of clients in stage A that I can have. I figured I could have a variable number of grouped messages based off of load, but I don't know how to come up with what this should look like.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Finally be able to list a max number of clients I can have in stage A and say with some amount of certainty that their messages can be stored to the database in x time once it gets to stage C.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Any help would be great. Thanks!&lt;/p&gt;&#xA;" OwnerUserId="2095" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-04-13T10:33:37.850" LastActivityDate="2011-05-13T11:45:54.837" Title="How to predict and optimize a queue?" Tags="&lt;modeling&gt;&lt;experiment-design&gt;&lt;queueing&gt;" AnswerCount="1" />
  <row Id="4803" PostTypeId="1" AcceptedAnswerId="4841" CreationDate="2010-11-22T21:32:18.330" Score="1" ViewCount="132" Body="&lt;p&gt;I'm analyzing a noisy time series where  where the inter-event interval is known to follow a two-gamma mixture distribution. If there was a simple model that would generate that kind of thing, it would be pretty simple to implement into BUGS. But otherwise, I can't think of anything that wouldn't be prohibitively kludgey.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone think of a model that induces mixture-gamma waiting times?&lt;/p&gt;&#xA;" OwnerUserId="996" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-27T16:02:58.457" LastActivityDate="2011-03-27T16:02:58.457" Title="Modeling a gamma-mixture waiting model in BUGS" Tags="&lt;bayesian&gt;&lt;mcmc&gt;&lt;bugs&gt;&lt;mixture&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="4804" PostTypeId="2" ParentId="4799" CreationDate="2010-11-22T21:38:15.220" Score="1" ViewCount="" Body="&lt;p&gt;First, convexity is not only a feature of a function, but rather, a function &lt;em&gt;and&lt;/em&gt; the domain over which it is defined.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To address your question more directly, another trick (rather another formulation) is to compute the Hessian matrix of your likelihood function. A per &lt;a href=&quot;http://en.wikipedia.org/wiki/Convex_function&quot; rel=&quot;nofollow&quot;&gt;wiki&lt;/a&gt; &lt;em&gt;a continuous, twice differentiable function of several variables is convex on a convex set if and only if its Hessian matrix is positive semidefinite on the interior of the convex set&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since the Hessian is real symmetric, it is sufficient to have diagonal &lt;a href=&quot;http://en.wikipedia.org/wiki/Diagonally_dominant_matrix&quot; rel=&quot;nofollow&quot;&gt;dominance&lt;/a&gt;, for it to be PSD (this is obvious to show for the logistic model).&lt;/p&gt;&#xA;" OwnerUserId="603" LastEditorUserId="603" LastEditorDisplayName="" LastEditDate="2010-11-22T21:50:21.293" LastActivityDate="2010-11-22T21:50:21.293" />
  <row Id="4805" PostTypeId="1" AcceptedAnswerId="4808" CreationDate="2010-11-22T21:43:45.823" Score="1" ViewCount="233" Body="&lt;p&gt;SAS EM has a node that generates the transformation for a dataset that makes it best fit a normal distribution. I can't imagine the people at SAS are the only ones to figure out how to do this. Is there any documentation out there on methodologies that do the same? Or an open source implementation?&lt;/p&gt;&#xA;" OwnerUserId="1951" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-11-22T21:58:02.020" LastActivityDate="2011-02-21T20:55:54.027" Title="Methodology for data transform that best fits normal" Tags="&lt;data-transformation&gt;&lt;normal-distribution&gt;" AnswerCount="2" CommentCount="5" />
  <row Id="4806" PostTypeId="2" ParentId="4597" CreationDate="2010-11-22T22:26:51.780" Score="0" ViewCount="" Body="&lt;p&gt;You might try to find the correlation of the series of differences with the moving average of the series of differences using some scale for the moving average (for example, use a 10 point moving average as the scale). This way you can get an idea about how &quot;wiggly&quot; the series is at different scales. This tells you whether the series has a tendency to move back and forth or keep moving in the same direction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the correlation is reasonably negative that implies that it is wiggly, it goes one way then comes back near where it was before. If the correlation is zero then it's not wiggly, if it moves in one direction, it has no tendency to move back where it came from. If it's positive then you might call it &quot;anti-wiggly&quot;, it has a tendency to keep moving in the same direction it has been moving in (it has lots of trends). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Repeat this at several different scales, say 2, 4, 8, 16, 24 ... data points, and then you can look at the graph.&lt;/p&gt;&#xA;" OwnerUserId="1146" LastActivityDate="2010-11-22T22:26:51.780" />
  <row Id="4807" PostTypeId="1" CreationDate="2010-11-22T23:14:02.377" Score="5" ViewCount="94" Body="&lt;p&gt;I study labor and delivery as an epidemiologist. It is well established that a large fetus has a higher risk of causing maternal birth trauma.  But a large baby is also likely to be delivered by cesarean section due to difficult labor.  Those delivered by cesarean section will not have any risk for maternal birth trauma and, therefore, excluded from an analysis using trauma as the outcome.  Thus, informative censoring is a real issue to estimate an unbiased relative risk, particularly when the cesarean rate is high.  Are there any methods to adjust for such a bias? &lt;/p&gt;&#xA;" OwnerUserId="2100" LastActivityDate="2010-11-23T15:36:12.530" Title="Informative censoring caused by cesarean section" Tags="&lt;censoring&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="1" />
  <row Id="4808" PostTypeId="2" ParentId="4805" CreationDate="2010-11-22T23:51:15.713" Score="1" ViewCount="" Body="&lt;p&gt;Maybe the Box–Cox transformation or other sort of &lt;a href=&quot;http://en.wikipedia.org/wiki/Power_transform&quot; rel=&quot;nofollow&quot;&gt;power transform&lt;/a&gt;?&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-11-22T23:51:15.713" CommentCount="1" />
  <row Id="4809" PostTypeId="1" CreationDate="2010-11-22T23:57:08.500" Score="0" ViewCount="116" Body="&lt;p&gt;I'm working on a meta-analysis and have generated a quirky question for which I'm at a bit of a loss.  The MA is for a large set of factorial experiments.  Calculating the Log Response Ratio (LRR) and variance in said ratio for the experimental data is a cinch, and we're comparing the effects of one type of treatment to the other (and any interactions).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, my group is curious at examining the effect of HALF the level of one of the treatments (it's a continuous treatment, and we've been looking at the highest versus the lowest level of the treatment).  We have fit nonlinear curves for each experiment that describe how the treatment effects the response over a wide range of treatment levels.  We've got the coefficient error and residual error for each of these curve fits.  And how we want to calculate log(full treatment) - log(half treatment) from the fitted curve for the this half-treatment log response ratio.  Easy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But...how would we then calculate the variance in the half-treatment log response ratio?  Would we use the SE estimates for the curve coefficients?  The residual error?  What would the sample size be for the variance calculation?  Or is this unimportant?  Thoughts? &lt;/p&gt;&#xA;" OwnerUserId="101" LastActivityDate="2010-11-22T23:57:08.500" Title="Determining variance of meta-analysis log-response ratio generated from fitted curve" Tags="&lt;variance&gt;&lt;meta-analysis&gt;" CommentCount="1" />
  <row Id="4810" PostTypeId="1" AcceptedAnswerId="4885" CreationDate="2010-11-23T01:18:41.480" Score="6" ViewCount="643" Body="&lt;p&gt;This may be too much of a general question but I hope I can find help here. I am starting a RA job in my university and my topic will be related to Internet Traffic Analysis. I am fairly new to the world of analysis but I guess in the world of research this is what I have to do a lot of.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've been through a few papers and in a lot of them I find they use Probability Density (PDF), CDF, CCDF etc to explain the results they've obtained. For example, PDF of user session duration, CDF of bytes transferred each day etc. I have taken probability and statistics class so I understand what they are but I am still confused to the cases in which such representation would be chosen.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, if there is anyone out there who does such graphs and analysis (in any other topic in general or this) could you tell me simply in what situation would I use one or the other of these representations&lt;/p&gt;&#xA;" OwnerUserId="2101" LastActivityDate="2010-11-25T14:30:56.387" Title="How to use CDF and PDF statistics for analysis" Tags="&lt;mathematical-statistics&gt;" AnswerCount="3" />
  <row Id="4811" PostTypeId="2" ParentId="2691" CreationDate="2010-11-23T04:32:49.263" Score="0" ViewCount="" Body="&lt;p&gt;The following youtube video might be helpful: &quot;A layman's introduction to principal component analysis&quot;&#xA;&lt;a href=&quot;http://www.youtube.com/watch?v=BfTMmoDFXyE&quot; rel=&quot;nofollow&quot;&gt;http://www.youtube.com/watch?v=BfTMmoDFXyE&lt;/a&gt;&lt;/p&gt;&#xA;" LastActivityDate="2010-11-23T04:32:49.263" CommentCount="2" />
  <row Id="4812" PostTypeId="1" CreationDate="2010-11-23T06:53:05.397" Score="3" ViewCount="282" Body="&lt;p&gt;I'm currently working as a teaching assistant at my University, in an introductory statistics course (for medical students).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Offline, there are many books available with information to aid the teacher.  However what I am interested to know is if you might direct me to any (good) &lt;strong&gt;resources that provide exercises (with solutions) in statistics, which are available online?&lt;/strong&gt; (e.g: teachers notes).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The subject material can range between descriptive statistics, probability, and parametric/non-parametric statistical inference.&lt;/p&gt;&#xA;" OwnerUserId="253" LastActivityDate="2010-11-23T07:50:07.560" CommunityOwnedDate="2010-11-23T10:22:21.343" Title="Resources for learning statistics - exercises (with solutions), available online?" Tags="&lt;teaching&gt;&lt;learning&gt;&lt;resources&gt;" AnswerCount="1" CommentCount="3" FavoriteCount="2" />
  <row Id="4813" PostTypeId="2" ParentId="4812" CreationDate="2010-11-23T07:50:07.560" Score="2" ViewCount="" Body="&lt;p&gt;This is a totally amazing online resource for multi-level modelling, not sure if that's any good to you:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.cmm.bristol.ac.uk/&quot; rel=&quot;nofollow&quot;&gt;http://www.cmm.bristol.ac.uk/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even includes a test at the start to give you an idea of where to start.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And should this be community wiki?&lt;/p&gt;&#xA;" OwnerUserId="199" LastActivityDate="2010-11-23T07:50:07.560" CommunityOwnedDate="2010-11-23T07:50:07.560" CommentCount="2" />
  <row Id="4814" PostTypeId="1" AcceptedAnswerId="4815" CreationDate="2010-11-23T09:38:30.733" Score="4" ViewCount="303" Body="&lt;p&gt;I have two LME models with the same interaction, one containing both main effects and one containing only one main effect, say :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ H\_CE = Season + Crownlevel + Season:Crownlevel , random = 1|CollectorID $$&#xD;&#xA;and&#xA;$$ H\_CE = Season + Season:Crownlevel , random = 1|CollectorID $$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are 4 levels of each, and every combination of Season, Crownlevel and CollectorID&#xA;The AIC, BIC and log likelihood of both models are completely equal. Given the formula for AIC being&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ \mathit{AIC} = 2k - 2\ln(L)\ $$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;one would expect this to be different, even if the likelihoods are exactly the same. In the end, they have a different number of parameters. Or so I thought...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Trying this toy example in R :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(nlme)&#xA;&#xA;Season &amp;lt;- rep(as.factor(rep(letters[1:4],each=4)),4)&#xA;Crownlevel &amp;lt;-rep(as.factor(rep(letters[11:14],4)),4)&#xA;CollectorID &amp;lt;-rep(letters[20:23],each=16)&#xA;X &amp;lt;-  model.matrix(~Season+Crownlevel+Season:Crownlevel)&#xA;B &amp;lt;- c(1,1,-2,2,0.3,0.4,0.4,2,3,1,-2,-3,-4,2,1,2)&#xA;H_CE &amp;lt;- X %*% B + rnorm(16*4)&#xA;KBM &amp;lt;- data.frame(Season,Crownlevel,H_CE,CollectorID)&#xA;&#xA;model1 &amp;lt;- lme(H_CE~Season+Crownlevel+Season:Crownlevel,data=KBM,&#xA;       method=&quot;ML&quot;,random=~1|CollectorID)&#xA;model1e &amp;lt;- lme(H_CE~Season+Season:Crownlevel,data=KBM,&#xA;       method=&quot;ML&quot;,random=~1|CollectorID)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I get : &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;anova(model1,model1e)&#xA;        Model df      AIC      BIC    logLik&#xA;model1      1 18 174.1834 213.0433 -69.09168&#xA;model1e     2 18 174.1834 213.0433 -69.09168&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What am I missing here? Why are the numbers completely equal? It has to do something with the model specification, but I can't really see what. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The model specification in itself is faulty, I know that. But I can't explain what makes it return a different set of parameters, but exactly the same residuals, likelihood and degrees of freedom :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; all.equal(residuals(model1),residuals(model1e))&#xA;[1] TRUE&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;EDIT :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As fabians rightfully pointed out, both models are perfectly equivalent. Yet, I fail to see why in the AIC calculation the same value for the number of parameters &lt;code&gt;k&lt;/code&gt; is used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT2 : &#xA;the &lt;code&gt;k&lt;/code&gt; in AIC uses the df, which explains everything. Thx for all the answers.&lt;/p&gt;&#xA;" OwnerUserId="1124" LastEditorUserId="1124" LastEditorDisplayName="" LastEditDate="2010-11-23T22:23:46.577" LastActivityDate="2010-11-23T22:23:46.577" Title="Why do I get equal AIC, BIC and log likelihood for different models in LME framework?" Tags="&lt;r&gt;&lt;mixed-model&gt;&lt;maximum-likelihood&gt;&lt;aic&gt;&lt;bic&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="4815" PostTypeId="2" ParentId="4814" CreationDate="2010-11-23T10:00:56.627" Score="5" ViewCount="" Body="&lt;p&gt;The models are exactly equivalent. In both models you effectively specify one parameter for each combination of levels of Season and Crownlevel - the only difference is the parameterization: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the first model, you fit main effects for Season and Crownlevel and an interaction effect to capture the combination-specific deviations from the main effects. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the second model, you specify only the main effect of season, and the interaction effect then captures the deviations for each crownlevel within a season. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;H_CE~Season:Crownlevel&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;would also yield an equivalent model, with one parameter for each combination of season and crownlevel (minus one that is non-identifiable because of the intercept, i.e. constitutes the reference category). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;BTW: I don't think your model specification is faulty, which specification is better depends on the inference you want to do with your model. &lt;/p&gt;&#xA;" OwnerUserId="1979" LastActivityDate="2010-11-23T10:00:56.627" CommentCount="5" />
  <row Id="4816" PostTypeId="1" AcceptedAnswerId="4821" CreationDate="2010-11-23T10:12:46.490" Score="2" ViewCount="133" Body="&lt;p&gt;Let $X \sim \chi^2_k$ be a chi-squared distributed random variable with $k$ degrees of freedom. What are the sharpest known bounds for the following probabilities&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;\mathbb{P}[X &amp;gt; t] \leq 1 - \delta_1(t, k)&#xD;&#xA;$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;\mathbb{P}[X &amp;lt; z] \leq 1 - \delta_2(z, k)&#xD;&#xA;$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\delta_1$ and $\delta_2$ are some functions. Pointers to relevant papers would be appreciated. &lt;/p&gt;&#xA;" OwnerUserId="168" LastActivityDate="2010-11-23T17:48:00.947" Title="What are the sharpest known tail bounds for $\chi_k^2$ distributed variables?" Tags="&lt;probability&gt;&lt;chi-squared&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="4817" PostTypeId="1" AcceptedAnswerId="4956" CreationDate="2010-11-23T10:48:00.267" Score="3" ViewCount="209" Body="&lt;p&gt;I wasn't able to find one on Google, and was wondering if one exists...&lt;/p&gt;&#xA;" OwnerUserId="253" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-11-23T16:27:15.573" LastActivityDate="2010-11-27T07:53:59.150" Title="Is there an unpaired version of the sign test?" Tags="&lt;hypothesis-testing&gt;&lt;nonparametric&gt;&lt;median&gt;&lt;sign-test&gt;" AnswerCount="4" />
  <row Id="4818" PostTypeId="1" AcceptedAnswerId="4822" CreationDate="2010-11-23T10:51:09.000" Score="5" ViewCount="266" Body="&lt;p&gt;I just noticed how the non exact McNemar's test uses the chi square asymptotic distribution.  But since the exact test (for the two case table) relies on the binomial distribution, how come it is not common to suggest the normal approximation to the binomial distribution?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="253" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-11-23T17:01:45.777" LastActivityDate="2010-11-23T21:09:56.640" Title="Why does the McNemar's test use chi-square and not the normal distribution?" Tags="&lt;distributions&gt;&lt;binomial&gt;&lt;chi-squared&gt;&lt;normal-distribution&gt;" AnswerCount="2" />
  <row Id="4819" PostTypeId="2" ParentId="4817" CreationDate="2010-11-23T10:56:34.037" Score="2" ViewCount="" Body="&lt;p&gt;The extension goes thorugh introducing rank to somewhat regulate the order of data and the result are Wilcoxon tests (Mann-Whitney in particular).&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-11-23T10:56:34.037" CommentCount="8" />
  <row Id="4820" PostTypeId="2" ParentId="4817" CreationDate="2010-11-23T11:51:57.410" Score="2" ViewCount="" Body="&lt;p&gt;I'm not sure if such a test can exist conceptually. The sign test uses the pairing of the data to decide whether one value is bigger than the &lt;strong&gt;corresponding&lt;/strong&gt; other value. But in an unpaired situation there is nothing like a corresponding other value (every value in the other group could be a potential counterpart for comparison). Correct me please, if I'm not getting the point...&lt;/p&gt;&#xA;" OwnerUserId="1573" LastActivityDate="2010-11-23T11:51:57.410" CommentCount="2" />
  <row Id="4821" PostTypeId="2" ParentId="4816" CreationDate="2010-11-23T12:02:06.880" Score="1" ViewCount="" Body="&lt;p&gt;The Sharpest bound I know is that of &lt;a href=&quot;http://projecteuclid.org/DPubS?service=UI&amp;amp;version=1.0&amp;amp;verb=Display&amp;amp;handle=euclid.aos/1015957395&quot; rel=&quot;nofollow&quot;&gt;Massart and Laurent&lt;/a&gt; Lemma 1 p1325. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A corollary of their bound is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(X-k\geq 2\sqrt{kx}+2x)\leq \exp (-x) $$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(k-X\geq 2\sqrt{kx})\leq \exp (-x) $$&#xD;&lt;/p&gt;&#xA;" OwnerUserId="223" LastEditorUserId="223" LastEditorDisplayName="" LastEditDate="2010-11-23T17:48:00.947" LastActivityDate="2010-11-23T17:48:00.947" CommentCount="2" />
  <row Id="4822" PostTypeId="2" ParentId="4818" CreationDate="2010-11-23T12:33:32.047" Score="5" ViewCount="" Body="&lt;p&gt;A close-to-intuitive answer:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Take a closer look at the formula for the McNemar test, given the table&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;      pos | neg&#xA;----|-----|-----&#xA;pos |  a  |  b&#xA;----|-----|-----&#xA;neg |  c  |  d&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The McNemar statistic &lt;code&gt;M&lt;/code&gt; is calculated as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ M = {(b-c)^2 \over b+c} $$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The definition of a $\chi^2$ distribution with k degrees of freedom is that it consists of the sum of &lt;strong&gt;squares&lt;/strong&gt; of k independent standard normal variables. if the 4 numbers are large enough, &lt;code&gt;b&lt;/code&gt; and &lt;code&gt;c&lt;/code&gt;, and thus &lt;code&gt;b-c&lt;/code&gt; and &lt;code&gt;b+c&lt;/code&gt; can be approximated by a normal distribution. Given the formula for M, it's easily seen that with large enough values &lt;code&gt;M&lt;/code&gt; will indeed follow approximately a $\chi^2$ distribution with 1 degree of freedom.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;EDIT :&#xA;As onstop rightfully indicated, the normal approximation is in fact completely equivalent. That's rather trivial given the argument using the approximation of &lt;code&gt;b-c&lt;/code&gt; by the normal distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The exact binomial version is also equivalent to the sign test, in the sense that in this version the binomial distribution is used to compare &lt;code&gt;b&lt;/code&gt; to $Binom(b+c,0.5)$. Or we can say that under the null hypothesis the distribution of b can be approximated by $N(0.5\times(b+c),0.5^2\times(b+c)$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or, equivalently:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{b-(\frac{b+c}{2})}{\frac{\sqrt{b+c}}{2}}\sim N(0,1)$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;which simplifies to&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ \frac{b-c}{\sqrt{b+c}}\sim N(0,1)$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;or, when taken the square on both sides, to $M \sim \chi^2_1$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hence, the normal approximation &lt;strong&gt;is&lt;/strong&gt; used. It is the same as the $\chi^2$ approximation.&lt;/p&gt;&#xA;" OwnerUserId="1124" LastEditorUserId="1124" LastEditorDisplayName="" LastEditDate="2010-11-23T21:09:56.640" LastActivityDate="2010-11-23T21:09:56.640" CommentCount="9" />
  <row Id="4823" PostTypeId="1" AcceptedAnswerId="4824" CreationDate="2010-11-23T12:44:40.340" Score="1" ViewCount="254" Body="&lt;p&gt;I've got a dataframe like the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   case simulation temp plank oxygen&#xA;1     1          1    8     7     11&#xA;2     2          1   16    10     15&#xA;...&#xA;17   17          2   26    12     17&#xA;18   18          2   15     8     12&#xA;19   19          2   28    11     21&#xA;20   20          2   24     6     14&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'd like to get summaries split by the levels of the simulation variable. For example, I'd like the mean of &lt;code&gt;temp&lt;/code&gt; for simulation==1 and simulation==2, and the same for standard deviation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At the moment I'm using the following code, which is pretty horrible:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df &amp;lt;- read.csv(&quot;data.csv&quot;)&#xA;attach(df)&#xA;&#xA;# Create subset variables&#xA;temp1 = subset(temp, simulation==1)&#xA;temp2 = subset(temp, simulation==2)&#xA;&#xA;plank1 = subset(plank, simulation==1)&#xA;plank2 = subset(plank, simulation==2)&#xA;&#xA;oxygen1 = subset(oxygen, simulation==1)&#xA;oxygen2 = subset(oxygen, simulation==2)&#xA;&#xA;print(sd(temp1))&#xA;print(sd(temp2))&#xA;print(sd(plank1))&#xA;print(sd(plank2))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm sure there must be an automatic way to do this in R, but I can't find it. I've tried using &lt;code&gt;summary(df ~ simulation)&lt;/code&gt;, but that doesn't produce anything useful.&lt;/p&gt;&#xA;" OwnerUserId="261" LastActivityDate="2010-11-23T14:31:45.363" Title="Automatically produce summary by factor variable in R" Tags="&lt;r&gt;" AnswerCount="2" />
  <row Id="4824" PostTypeId="2" ParentId="4823" CreationDate="2010-11-23T12:49:02.623" Score="5" ViewCount="" Body="&lt;p&gt;Check out the &lt;code&gt;by()&lt;/code&gt; or &lt;code&gt;tapply()&lt;/code&gt; functions. Basically,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;tapply(y, g, mean)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;will give you the mean of &lt;code&gt;y&lt;/code&gt; by levels of &lt;code&gt;g&lt;/code&gt;. If you want to get a data.frame from the resulting aggregated measures, use &lt;code&gt;aggregate()&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A more elaborated solution is available through the &lt;code&gt;summary.formula()&lt;/code&gt; function in the &lt;a href=&quot;http://cran.r-project.org/web/packages/Hmisc/index.html&quot; rel=&quot;nofollow&quot;&gt;Hmisc&lt;/a&gt; package.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-11-23T12:49:02.623" />
  <row Id="4825" PostTypeId="2" ParentId="3113" CreationDate="2010-11-23T14:04:56.840" Score="1" ViewCount="" Body="&lt;p&gt;Some people seem to use a concept of Pitman Asymptotic Relative Efficiency (ARE) to inflate the sample size obtained by using a sample size formula for a parametric test. Ironically, in order to compute it, one has to assume a distribution again... see e.g. &lt;a href=&quot;http://www.childrens-mercy.org/stats/size/mann.asp&quot; rel=&quot;nofollow&quot;&gt;Sample size for the Mann-Whitney U test&lt;/a&gt; There are some links in the end of the article that provide pointers for further reading.&lt;/p&gt;&#xA;" OwnerUserId="1573" LastEditorUserId="1573" LastEditorDisplayName="" LastEditDate="2010-11-23T19:45:50.180" LastActivityDate="2010-11-23T19:45:50.180" />
  <row Id="4826" PostTypeId="2" ParentId="4823" CreationDate="2010-11-23T14:31:45.363" Score="2" ViewCount="" Body="&lt;p&gt;package &lt;code&gt;doBy&lt;/code&gt; has a &lt;code&gt;summaryBy&lt;/code&gt; function that has a formula based syntax like the one you tried. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, i think that question would have been better asked on stackexchange.&lt;/p&gt;&#xA;" OwnerUserId="1979" LastActivityDate="2010-11-23T14:31:45.363" CommentCount="2" />
  <row Id="4827" PostTypeId="2" ParentId="4817" CreationDate="2010-11-23T14:35:44.533" Score="5" ViewCount="" Body="&lt;p&gt;Good (2005) defines the one-sample sign-test for the location parameter $\theta$ for a continuous symmetric variable $X$ as follows:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Take the difference $D_i$ of each observation to the location parameter $\theta_0$ under the null hypothesis.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Define an indicator variable $Z_i$ as $0$ when $D_i &amp;lt; 0$, and as $1$ when $D_i &amp;gt; 0$. Since $X$ is continuous, $P(D_i = 0) = 0$.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Calculate test statistic $T=\sum_i Z_i$.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The distribution of $T$ is is found by generating all $2^N$ possible outcomes of the $Z_i$ indicator variables (2 possibilities for each observation with equal probability $\frac{1}{2}$ under H0). This leads to the binomial distribution as in the sign test for 2 dependent samples.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The justification for step 4 is: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Suppose we had lost track of the signs&#xA;  of the deviations [...]. We could&#xA;  attach new signs at random [...]. If&#xA;  we are correct in our hypothesis that&#xA;  the variable has a symmetric&#xA;  distribution about $\theta_0$, the&#xA;  resulting values should have precisely&#xA;  the same distribution as the original&#xA;  observations. That is, the absolute&#xA;  values of the deviations are&#xA;  sufficient for regenerating the&#xA;  sample. (p34f)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I agree that this reasoning seems somewhat different from a 2-sample permutation test where you re-assign experimental conditions to observations with the justification of exchangeability under H0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good, P. 2005. Permutation, Parametric, and Bootstrap Tests of Hypotheses. New York: Springer.&lt;/p&gt;&#xA;" OwnerUserId="1909" LastEditorUserId="1909" LastEditorDisplayName="" LastEditDate="2010-11-23T15:31:39.493" LastActivityDate="2010-11-23T15:31:39.493" CommentCount="4" />
  <row Id="4828" PostTypeId="2" ParentId="4807" CreationDate="2010-11-23T15:05:47.490" Score="1" ViewCount="" Body="&lt;p&gt;When I initially wrote the comments below I had assumed that the Heckman estimator could be used for dichotomous outcomes, but the second paper I cite says there is no direct analog. Hopefully someone can point to different and more applicable resources. I still leave my initial comment up as I still feel those papers are helpful. I'm not sure how acceptable it would be viewed to use OLS (as oppossed to logistic regression) simply so you can incorporate the Heckman correction estimate.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;The work of James Heckman would be applicable to your problem, especially if you have an instrument with which you can estimate the probability of being chosen for a C-section independent of trauma risk.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://dx.doi.org/10.2307/1912352&quot; rel=&quot;nofollow&quot;&gt;Sample Selection Bias as a Specification Error&lt;/a&gt;&#xA;by: James J. Heckman&#xA;Econometrica, Vol. 47, No. 1. (1979), pp. 153-161. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://vanpelt.sonoma.edu/users/c/cuellar/econ411/Heckman.pdf&quot; rel=&quot;nofollow&quot;&gt;PDF version&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also as an intro into the logic of the Heckman selection estimator intended for a largely non-technical audience, I enjoy this paper &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://dx.doi.org/10.1007/s10940-007-9024-4&quot; rel=&quot;nofollow&quot;&gt;Is the Magic Still There? The Use of the Heckman Two-Step Correction for Selection Bias in Criminology&lt;/a&gt;&#xA;by: Shawn Bushway, Brian Johnson, Lee Slocum&#xA;Journal of Quantitative Criminology, Vol. 23, No. 2. (1 June 2007), pp. 151-178. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.crim.umd.edu/Faculty/userfiles/40/HeckmanCorrection.pdf&quot; rel=&quot;nofollow&quot;&gt;PDF version&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1036" LastEditorUserId="1036" LastEditorDisplayName="" LastEditDate="2010-11-23T15:36:12.530" LastActivityDate="2010-11-23T15:36:12.530" />
  <row Id="4829" PostTypeId="2" ParentId="4753" CreationDate="2010-11-23T17:00:09.463" Score="0" ViewCount="" Body="&lt;p&gt;I might suggest non-negative matrix factorization. The iterative algorithm of Lee and Seung is easy to implement and should be amenable to sparse matrices (although it involves Hadamard products, which some sparse matrix packages may not support.).&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-11-23T17:00:09.463" />
  <row Id="4830" PostTypeId="1" AcceptedAnswerId="4873" CreationDate="2010-11-23T17:35:51.753" Score="5" ViewCount="490" Body="&lt;p&gt;Full Disclosure: This is homework. I've included a link to the dataset ( &lt;a href=&quot;http://www.bertelsen.ca/R/logistic-regression.sav&quot; rel=&quot;nofollow&quot;&gt;http://www.bertelsen.ca/R/logistic-regression.sav&lt;/a&gt; )&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My goal is to maximize the prediction of loan defaulters in this data set.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Every model that I have come up with so far, predicts &gt;90% of non-defaulters, but &amp;lt;40% of defaulters making the classification efficiency overall ~80%. So, I wonder if there are interaction effects between the variables? Within a logistic regression, other than testing each possible combination is there a way to identify potential interaction effects? Or alternatively a way to boost the efficiency of classification of defaulters. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm stuck, any recommendations would be helpful in your choice of words, R-code or SPSS syntax. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My primary variables are outlined in the following histogram and scatterplot (with the exception of the dichotomous variable)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A description of the primary variables: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;age: Age in years&#xA;employ: Years with current employer&#xA;address: Years at current address&#xA;income: Household income in thousands&#xA;debtinc: Debt to income ratio (x100)&#xA;creddebt: Credit card debt in thousands&#xA;othdebt: Other debt in thousands&#xA;default: Previously defaulted (dichotomous, yes/no, 0/1)&#xA;ed: Level of education (No HS, HS, Some College, College, Post-grad)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Additional variables are just transformations of the above. I also tried converting a few of the continuous variables into categorical variables and implementing them in the model, no luck there. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you'd like to pop it into R, quickly, here it is: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;## R Code&#xA;df &amp;lt;- read.spss(file=&quot;http://www.bertelsen.ca/R/logistic-regression.sav&quot;, use.value.labels=T, to.data.frame=T)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/aVqtZ.jpg&quot; alt=&quot;alt text&quot;&gt;&#xA;&lt;img src=&quot;http://i.imgur.com/VQJDg.jpg&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="776" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-11-24T11:46:49.317" LastActivityDate="2010-12-07T16:30:00.257" Title="Better Classification of default in logistic regression" Tags="&lt;r&gt;&lt;logistic-regression&gt;&lt;spss&gt;&lt;homework&gt;" AnswerCount="6" CommentCount="4" />
  <row Id="4831" PostTypeId="1" AcceptedAnswerId="4833" CreationDate="2010-11-23T17:41:19.050" Score="5" ViewCount="568" Body="&lt;p&gt;When transforming variables, do you have to use all of the same transformation? For example, can I pick and choose differently transformed variables, as in: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let, x1,x2,x3 be age,length of employment, length of residence, and income&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Y = B1*sqrt(x1) + B2*-1/(x2) + B3*log(x3)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Or must you be consistent with your transforms and use all of the same? as in: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Y = B1*log(x1) + B2*log(x2) + B3*log(x3) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My understanding is that the goal of transformation is to address the problem of normality. Looking at histograms of each variable we can see that they present very different distributions, which would lead me to believe that the transformations required are different on a variable by variable basis. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;## R Code&#xA;df &amp;lt;- read.spss(file=&quot;http://www.bertelsen.ca/R/logistic-regression.sav&quot;, use.value.labels=T, to.data.frame=T)&#xA;hist(df[1:7]) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/5qPcd.jpg&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lastly, how valid is it to transform variables using log(xn + 1) where xn has 0 values? Does this transform need to be consistent across all variables or is it used adhoc even for those variables which do not include 0's?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;## R Code &#xA;plot(df[1:7])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/ShjOG.jpg&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="776" LastEditorUserId="776" LastEditorDisplayName="" LastEditDate="2010-11-23T18:03:41.510" LastActivityDate="2010-11-23T18:03:41.510" Title="Logistic Regression: Transforming Variables" Tags="&lt;logistic-regression&gt;&lt;data-transformation&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="4832" PostTypeId="1" AcceptedAnswerId="4835" CreationDate="2010-11-23T17:44:28.190" Score="3" ViewCount="507" Body="&lt;p&gt;In SPSS output there is a pretty little classification table available when you perform a logistic regression, is the same possible with R? If so, how?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/yFS6P.jpg&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="776" LastEditorUserId="776" LastEditorDisplayName="" LastEditDate="2010-11-23T17:52:19.663" LastActivityDate="2010-11-23T21:33:23.423" Title="Logistic Regression: Classification Tables a la SPSS in R" Tags="&lt;r&gt;&lt;logistic-regression&gt;&lt;spss&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="1" />
  <row Id="4833" PostTypeId="2" ParentId="4831" CreationDate="2010-11-23T17:55:52.673" Score="8" ViewCount="" Body="&lt;p&gt;One transforms the &lt;em&gt;dependent&lt;/em&gt; variable to achieve approximate &lt;em&gt;symmetry&lt;/em&gt; and &lt;em&gt;homoscedasticity&lt;/em&gt; of the &lt;em&gt;residuals&lt;/em&gt;.  Transformations of the &lt;em&gt;independent&lt;/em&gt; variables have a different purpose: after all, in this regression all the independent values are taken as fixed, not random, so &quot;normality&quot; is inapplicable.  The main objective in these transformations is to achieve &lt;em&gt;linear&lt;/em&gt; relationships with the dependent variable (or, really, with its logit).  (This objective over-rides auxiliary ones such as reducing excess &lt;a href=&quot;http://www.jerrydallal.com/LHSP/diagnose.htm&quot; rel=&quot;nofollow&quot;&gt;leverage&lt;/a&gt; or achieving a simple interpretation of the coefficients.)  These relationships are a property of the data and the phenomena that produced them, so you need the flexibility to choose appropriate re-expressions of each of the variables separately from the others.  Specifically, not only is it not a problem to use a log, a root, and a reciprocal, it's rather common.  The principle is that there is (usually) nothing special about how the data are originally expressed, so you should let the data suggest re-expressions that lead to effective, accurate, useful, and (if possible) theoretically justified models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The histograms--which reflect the univariate distributions--often hint at an initial transformation, but are not dispositive.  Accompany them with scatterplot matrices so you can examine the relationships among all the variables.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Transformations like $\log(x + c)$ where $c$ is a positive constant &quot;start value&quot; can work--and can be indicated even when no value of $x$ is zero--but sometimes they destroy linear relationships.  When this occurs, a good solution is to create &lt;em&gt;two&lt;/em&gt; variables.  One of them equals $\log(x)$ when $x$ is nonzero and otherwise is anything; it's convenient to let it default to zero.  The other, let's call it $z_x$, is an indicator of whether $x$ is zero: it equals 1 when $x = 0$ and is 0 otherwise.  These terms contribute a sum&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\beta \log(x) + \beta_0 z_x$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;to the estimate.  When $x \gt 0$, $z_x = 0$ so the second term drops out leaving just $\beta \log(x)$.  When $x = 0$, &quot;$\log(x)$&quot; has been set to zero while $z_x = 1$, leaving just the value $\beta_0$.  Thus, $\beta_0$ estimates the effect when $x = 0$ and otherwise $\beta$ is the coefficient of $\log(x)$.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-11-23T18:03:33.650" LastActivityDate="2010-11-23T18:03:33.650" CommentCount="1" />
  <row Id="4834" PostTypeId="1" AcceptedAnswerId="4836" CreationDate="2010-11-23T17:59:32.200" Score="3" ViewCount="211" Body="&lt;p&gt;One of the assumptions for using the Wilcoxon sign-rank test is that the underlying distribution is continuous (&lt;a href=&quot;http://en.wikipedia.org/wiki/Wilcoxon_signed-rank_test#Assumptions&quot; rel=&quot;nofollow&quot;&gt;see here&lt;/a&gt;.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, there are cases (for example, when analyzing Likert scale data) where this assumption might not necessarily hold.  In such cases, what test can you use? And how would you do it with R?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(My only bet here is to use a randomization test on the median - which I imagine can be easily done using the boot package.)&lt;/p&gt;&#xA;" OwnerUserId="253" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-11-23T18:18:57.223" LastActivityDate="2010-11-23T18:18:57.223" Title="Alternative to the Wilcoxon test when the distribution isn't continuous?" Tags="&lt;r&gt;&lt;nonparametric&gt;&lt;sign-test&gt;&lt;wilcoxon&gt;" AnswerCount="1" CommentCount="4" FavoriteCount="1" />
  <row Id="4835" PostTypeId="2" ParentId="4832" CreationDate="2010-11-23T18:00:23.230" Score="4" ViewCount="" Body="&lt;p&gt;I'm not aware of a specific command, but this might be a start:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# generate some data&#xA;&amp;gt; N &amp;lt;- 100&#xA;&amp;gt; X &amp;lt;- rnorm(N, 175, 7)&#xA;&amp;gt; Y &amp;lt;- 0.4*X + 10 + rnorm(N, 0, 3)&#xA;&#xA;# dichotomize Y&#xA;&amp;gt; Yfac &amp;lt;- cut(Y, breaks=c(-Inf, median(Y), Inf), labels=c(&quot;lo&quot;, &quot;hi&quot;))&#xA;&#xA;# logistic regression&#xA;&amp;gt; glmFit &amp;lt;- glm(Yfac ~ X, family=binomial(link=&quot;logit&quot;))&#xA;&#xA;# predicted probabilities&#xA;&amp;gt; Yhat &amp;lt;- fitted(glmFit)&#xA;&#xA;# choose a threshold for dichotomizing according to predicted probability&#xA;&amp;gt; thresh  &amp;lt;- 0.5&#xA;&amp;gt; YhatFac &amp;lt;- cut(Yhat, breaks=c(-Inf, thresh, Inf), labels=c(&quot;lo&quot;, &quot;hi&quot;))&#xA;&#xA;# contingency table and marginal sums&#xA;&amp;gt; cTab &amp;lt;- table(Yfac, YhatFac)&#xA;&amp;gt; addmargins(cTab)&#xA;        YhatFac&#xA;Yfac   lo  hi Sum&#xA;  lo   36  14  50&#xA;  hi   12  38  50&#xA;  Sum  48  52 100&#xA;&#xA;# percentage correct for training data&#xA;&amp;gt; sum(diag(cTab)) / sum(cTab)&#xA;[1] 0.74&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1909" LastActivityDate="2010-11-23T18:00:23.230" CommentCount="3" />
  <row Id="4836" PostTypeId="2" ParentId="4834" CreationDate="2010-11-23T18:17:49.580" Score="5" ViewCount="" Body="&lt;p&gt;I have found that the Wilcoxon statistic is still fine for this purpose and that small simulations do a good job of estimating the size and the power of the test.  I suspect this is more powerful than just comparing the two medians.  The main concern is lack of power due to extensive numbers of ties, but that concern attaches to &lt;em&gt;any&lt;/em&gt; solution you can conceive of: there's no way around it (except to design instruments that offer a wider range of responses!).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To perform the simulation, concatenate the two data arrays (of lengths $n$ and $m$) into a single array (of length $n+m$).  In each iteration randomly permute the elements of the array and break the result into the first $n$ and last $m$ elements.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-11-23T18:17:49.580" CommentCount="8" />
  <row Id="4837" PostTypeId="2" ParentId="4364" CreationDate="2010-11-23T18:37:54.900" Score="4" ViewCount="" Body="&lt;p&gt;There's an entire book written about this: &quot;Characterizations of the normal probability law&quot;, A. M. Mathai &amp;amp; G. Perderzoli.  A brief review in &lt;a href=&quot;http://www.jstor.org/pss/2286313&quot; rel=&quot;nofollow&quot;&gt;JASA (Dec. 1978)&lt;/a&gt; mentions the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Let $X_1, \ldots, X_n$ be independent random variables.  Then $\sum_{i=1}^n{a_i x_i}$ and $\sum_{i=1}^n{b_i x_i}$ are independent, where $a_i b_i \ne 0$, if and only if $X_i$ [are] normally distributed.&quot;&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-11-23T18:37:54.900" CommunityOwnedDate="2010-11-23T18:37:54.900" CommentCount="3" />
  <row Id="4838" PostTypeId="2" ParentId="47" CreationDate="2010-11-23T18:59:58.170" Score="1" ViewCount="" Body="&lt;p&gt;This is problem with clustering, you can't tell what is considered a cluster. I would see what is the reason behind clustering users, specify my threshold value, and use hierarchical clustering.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my experience, one has to set either the number of cluster needed, or the threshold value (the distance value that binds two data point together). &lt;/p&gt;&#xA;" OwnerUserId="2108" LastActivityDate="2010-11-23T18:59:58.170" />
  <row Id="4839" PostTypeId="1" CreationDate="2010-11-23T19:10:52.133" Score="4" ViewCount="253" Body="&lt;p&gt;I have a survey where I have asked people which type of computer games they enjoy and whether they consider themselves a hardcore gamer. I allowed people to select multiple genres, but now I am unsure what to do with my data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I initially thought factor analysis, with the idea being if there were genres that belonged to a particular type, they would separate out and I would be able to see a pattern. However, since I have the data on whether they consider themselves hardcore or not, it seems like I should use it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I did use it, would it make sense to test each genre individually, e.g. RTS-hardcore vs. RTS-non-hardcore, or should I look into combinations of genres?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sorry, it has been a while since I've done any stats, so my knowledge is a bit rusty. Any help would be greatly appreciated!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[EDIT] @Srikant: Yes, I was planning to make the answers binary. It didn't make sense to me to score different answers with values depending on say the number of genres somebody chose, because they might not necessarily play each game evenly, and I would have no way to determine what ratio of gameplay each genre had, so binary seemed the most fair to me.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;@mbq &amp;amp; @chl: The aim is just to see if certain genres tend to be more &quot;hardcore&quot; than other genres. I was expecting to find RTS, MOBA and FPS to cluster towards hardcore seeing as they tend to have a higher learning curve than say music/rhythm games.&lt;/p&gt;&#xA;" OwnerUserId="2127" LastEditorUserId="2127" LastEditorDisplayName="" LastEditDate="2010-11-25T12:46:22.433" LastActivityDate="2010-11-25T12:46:22.433" Title="Should I use factor analysis on my data?" Tags="&lt;factor-analysis&gt;&lt;survey&gt;" AnswerCount="3" CommentCount="3" />
  <row Id="4840" PostTypeId="1" CreationDate="2010-11-23T19:11:35.047" Score="1" ViewCount="130" Body="&lt;p&gt;Hi, my friend is a chemist and his problem is to predict the level of ozone concentration in a single site. We have the data for the last 12 years.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We want to predict the concentration for the coming years (as much as possible).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know this data set is small, so my question is is this possible? And how much of data is required if not?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the tool/method to use?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a plot of my data:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/KP3IQ.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;any suggestions/ help?&lt;/p&gt;&#xA;" OwnerUserId="2108" LastEditorUserId="2108" LastEditorDisplayName="" LastEditDate="2010-11-27T15:10:15.177" LastActivityDate="2011-05-19T21:20:30.913" Title="How to predict Ozone concentration in few years time?" Tags="&lt;machine-learning&gt;&lt;forecasting&gt;&lt;predictive-models&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="1" />
  <row Id="4841" PostTypeId="2" ParentId="4803" CreationDate="2010-11-23T20:00:49.273" Score="3" ViewCount="" Body="&lt;p&gt;A Gamma distribution with an integer shape parameter is an &lt;a href=&quot;http://en.wikipedia.org/wiki/Erlang_distribution&quot; rel=&quot;nofollow&quot;&gt;Erlang distribution&lt;/a&gt; which in turn is a generalization of the Exponential distribution often used to model waiting times. Thus, perhaps one approach is to use an Erlang distribution instead of a Gamma to model your waiting times. One way to justify the Erlang distribution is to assume that inter-event times are triggered by a certain number of underlying exponential inter-event times. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, think of water droplets falling into a balloon at some unknown rate. As water droplets keep falling the balloon will pop at some point. The waiting time for a balloon to pop will be erlang distributed as long as the inter-arrival times of the water droplets are exponentially distributed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A mixture erlang distribution can have the following interpretation: The inter-event times are now dependent on two separate exponentially distributed processes with perhaps different parameters. &lt;/p&gt;&#xA;" LastActivityDate="2010-11-23T20:00:49.273" CommentCount="2" />
  <row Id="4842" PostTypeId="2" ParentId="4818" CreationDate="2010-11-23T20:23:40.877" Score="3" ViewCount="" Body="&lt;p&gt;Won't the two approaches come to the same thing? The relevant chi-square distribution has one degree of freedom so is simply the distribution of the square of a random variable with a standard normal distribution. I'd have to go through the algebra to check, which I haven't got time to do right now, but I'd be surprised if you don't end up with exactly the same answer both ways.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-11-23T20:23:40.877" CommentCount="3" />
  <row Id="4843" PostTypeId="1" AcceptedAnswerId="4848" CreationDate="2010-11-23T20:47:07.533" Score="3" ViewCount="111" Body="&lt;p&gt;The &quot;standard&quot; way to compute Kappa for a predictive classification model (Witten and Frank page 163) is to construct the random confusion matrix in such a way that the number of predictions for each class is the same as the model predicted. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a visual, see (right side is the random):&#xA;&lt;img src=&quot;http://i.imgur.com/oetej.jpg&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone know why this is the case, instead of truly creating a random confusion matrix where the prior probabilities drive the number of predictions for each class. That seems the more accurate comparison against &quot;a null model&quot;. For example, in this case, the number of actual and predicted classes would coincide (in the image uploaded, this would mean that the columns of the random confusion matrix would be 100, 60 and 40 respectively).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;br&gt;&#xA;BMiner &lt;/p&gt;&#xA;" OwnerUserId="2040" LastEditorUserId="2040" LastEditorDisplayName="" LastEditDate="2010-11-23T21:30:51.223" LastActivityDate="2010-11-23T22:09:40.567" Title="Kappa for Predictive Model" Tags="&lt;predictive-models&gt;" AnswerCount="2" />
  <row Id="4844" PostTypeId="1" CreationDate="2010-11-23T20:52:02.350" Score="3" ViewCount="240" Body="&lt;p&gt;I have a timeseries of data which was gathered by driving a car around randomly.&#xA;One data point was gathered every minute and each data point is either &quot;Yes&quot; or &quot;No&quot;. Yes = temperature is above a threshold, No = it is below the threshold.   Assume the car never stops.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm interested in the proportion of Yeses and related confidence intervals.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm inclined to use binomial confidence intervals except that it seems to me that the data points are not independent.&lt;br&gt;&#xA;Would the fact that the data was gathered by driving around randomly buy me independence? How could I derive meaningful confidence intervals?&lt;/p&gt;&#xA;" OwnerUserId="1784" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-11-23T20:58:18.350" LastActivityDate="2011-01-03T20:11:46.050" Title="Confidence interval based on time series" Tags="&lt;time-series&gt;&lt;confidence-interval&gt;&lt;independence&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="4845" PostTypeId="2" ParentId="4830" CreationDate="2010-11-23T20:53:52.237" Score="4" ViewCount="" Body="&lt;p&gt;I'm not a logistic regression expert, but isn't it just a problem of unbalanced data? Probably you have much more non-defaulters than defaulters what may shift the prediction to deal better with larger class. Try to kick out some non-defaulters and see what happens.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-11-23T20:53:52.237" CommentCount="7" />
  <row Id="4846" PostTypeId="2" ParentId="4832" CreationDate="2010-11-23T21:33:23.423" Score="2" ViewCount="" Body="&lt;p&gt;Thomas D. Fletcher has a function called &lt;code&gt;ClassLog()&lt;/code&gt; (for &quot;classification analysis for a logistic regression model&quot;) in his &lt;a href=&quot;http://cran.r-project.org/web/packages/QuantPsyc/&quot; rel=&quot;nofollow&quot;&gt;QuantPsyc&lt;/a&gt; package. However, I like @caracal's response because it is self-made and easily customizable.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-11-23T21:33:23.423" />
  <row Id="4847" PostTypeId="2" ParentId="4843" CreationDate="2010-11-23T21:34:43.747" Score="2" ViewCount="" Body="&lt;p&gt;This is just an outer product of actual and predicted class frequencies times all objects count. In R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;A&amp;lt;-c(100,60,40)&#xA;P&amp;lt;-c(120,60,20)&#xA;numCases&amp;lt;-sum(A) #=sum(P) also&#xA;(A/numCases)%o%(P/numCases)*numCases&#xA;A%o%P/numCases #same simplified&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-11-23T21:34:43.747" />
  <row Id="4848" PostTypeId="2" ParentId="4843" CreationDate="2010-11-23T21:53:33.310" Score="5" ViewCount="" Body="&lt;p&gt;It might be useful to consider Cohen's $\kappa$ in the context of inter-rater-agreement. Suppose you have two raters individually assigning the same set of objects to the same categories. You can then ask for overall agreement by dividing the sum of the diagonal of the confusion matrix by the total sum. But this does not take into account that the two raters will also, to some extent, agree by chance. $\kappa$ is supposed to be a chance-corrected measure conditional on the baseline frequencies with which the raters use the categories (marginal sums).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The expected frequency of each cell under the assumption of independence given the marginal sums is then calculated just like in the $\chi^2$ test - this is equivalent to Witten &amp;amp; Frank's description (see mbq's answer). For chance-agreement, we only need the diagonal cells. In R&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# generate the given data&#xA;&amp;gt; lvls &amp;lt;- factor(1:3, labels=letters[1:3])&#xA;&amp;gt; rtr1 &amp;lt;- rep(lvls, c(100, 60, 40))&#xA;&amp;gt; rtr2 &amp;lt;- rep(rep(lvls, nlevels(lvls)), c(88,10,2, 14,40,6, 18,10,12))&#xA;&amp;gt; cTab &amp;lt;- table(rtr1, rtr2)&#xA;&amp;gt; addmargins(cTab)&#xA;     rtr2&#xA;rtr1    a   b   c Sum&#xA;  a    88  10   2 100&#xA;  b    14  40   6  60&#xA;  c    18  10  12  40&#xA;  Sum 120  60  20 200&#xA;&#xA;&amp;gt; library(irr)       # for kappa2()&#xA;&amp;gt; kappa2(cbind(rtr1, rtr2))&#xA; Cohen's Kappa for 2 Raters (Weights: unweighted)&#xA; Subjects = 200 &#xA;   Raters = 2 &#xA;    Kappa = 0.492 &#xA;        z = 9.46 &#xA;  p-value = 0 &#xA;&#xA;# observed frequency of agreement (diagonal cells)&#xA;&amp;gt; fObs &amp;lt;- sum(diag(cTab)) / sum(cTab)&#xA;&#xA;# frequency of agreement expected by chance (like chi^2)&#xA;&amp;gt; fExp &amp;lt;- sum(rowSums(cTab) * colSums(cTab)) / sum(cTab)^2&#xA;&amp;gt; (fObs-fExp) / (1-fExp)    # Cohen's kappa&#xA;[1] 0.4915254&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Note that $\kappa$ is not universally accepted at doing a good job, see, e.g., &lt;a href=&quot;http://www.john-uebersax.com/stat/kappa.htm&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;, or &lt;a href=&quot;http://www.agreestat.com/research_papers.html&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;, or the literature cited in the Wikipedia article.&lt;/p&gt;&#xA;" OwnerUserId="1909" LastEditorUserId="1909" LastEditorDisplayName="" LastEditDate="2010-11-23T22:09:40.567" LastActivityDate="2010-11-23T22:09:40.567" CommentCount="1" />
  <row Id="4849" PostTypeId="2" ParentId="4839" CreationDate="2010-11-23T22:12:47.053" Score="3" ViewCount="" Body="&lt;p&gt;Correspondence analysis might be a good fit. Creating a graph that shows the relationship between Hard-Core gaming (or not) and the different types of games that they are playing. The result would be the ability to say with cautious confidence that Hard Core Gamers play a certain group of game types (or not). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;With respect to factor analysis, I would recommend reviewing some of the answers to a question I asked previously, just so you're clear on what your goal is: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://stats.stackexchange.com/questions/1576/what-are-the-differences-between-factor-analysis-and-principal-component-analysis&quot;&gt;What are the differences between Factor Analysis and Principal Component Analysis&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Interpreting and creating correspondence analysis plots is discussed here (with examples on how to create the plots as well, pay strict attention to the differences in taking inference from horizontal and vertical axis distances) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://stats.stackexchange.com/questions/3270/interpreting-2d-correspondence-analysis-plots&quot;&gt;Interpreting 2D Correspondence Analysis Plots&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="776" LastActivityDate="2010-11-23T22:12:47.053" />
  <row Id="4850" PostTypeId="2" ParentId="1525" CreationDate="2010-11-23T22:38:38.997" Score="6" ViewCount="" Body="&lt;p&gt;I have hesitated to wade into this discussion, but because it seems to have gotten sidetracked over a trivial issue concerning how to express numbers, maybe it's worthwhile refocusing it.  A point of departure for your consideration is this:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;A probability is a hypothetical property.  Proportions summarize observations.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;A &lt;a href=&quot;http://en.wikipedia.org/wiki/Probability_interpretations#Frequentism&quot; rel=&quot;nofollow&quot;&gt;frequentist&lt;/a&gt; might rely on laws of large numbers to justify statements like &quot;the long-run proportion of an event [is] its probability.&quot;  This supplies meaning to statements like &quot;a probability is an expected proportion,&quot; which otherwise might appear merely tautological.  Other interpretations of probability also lead to connections between probabilities and proportions but they are less direct than this one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In our models we usually take probabilities to be &lt;em&gt;definite&lt;/em&gt; but &lt;em&gt;unknown.&lt;/em&gt;  I am reluctant to apply the term &quot;uncertain&quot; in this context. Before we conduct a sequence of observations, the [eventual] proportion, like any future event, is indeed &quot;uncertain&quot;.  &lt;em&gt;After&lt;/em&gt; we make those observations, the proportion is both &lt;em&gt;definite&lt;/em&gt; and &lt;em&gt;known.&lt;/em&gt;  (Perhaps this is what is meant by &quot;guaranteed&quot; in the OP.)  Much of our knowledge about the [hypothetical] probability is mediated through these uncertain observations and the idea that &lt;em&gt;they might have turned out otherwise.&lt;/em&gt;  In &lt;em&gt;this&lt;/em&gt; sense--that uncertainty about the observations is transmitted back to uncertain knowledge of the underlying probability--it seems justifiable to refer to the probability as &quot;uncertain.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In any event it is apparent that probabilities and proportions function differently in statistics, despite their similarities and intimate relationships.  It would be a mistake to take them to be the same thing.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-11-23T22:38:38.997" CommentCount="8" />
  <row Id="4851" PostTypeId="1" AcceptedAnswerId="4853" CreationDate="2010-11-23T23:02:12.753" Score="3" ViewCount="91" Body="&lt;p&gt;I'm trying to precompute the distributions of several random variables. In particular, these random variables are the results of functions evaluated at locations in a genome, so there will be on the order of 10^8 or 10^9 values for each. The functions are pretty smooth, so I don't think I'll lose much accuracy by only evaluating at every 2nd/10th/100th? base or so, but regardless there will be a large number of samples. My plan is to precompute quantile tables (maybe percentiles) for each function and reference these in the execution of my main program to avoid having to compute these distribution statistics in every run.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I don't really see how I can easily do this: storing, sorting, and reducing an array of 10^9 floats isn't really feasible, but I can't think of another way that doesn't lose information about the distribution. Is there a way of measuring the quantiles of a sample distribution that doesn't require storing the whole thing in memory?&lt;/p&gt;&#xA;" OwnerUserId="2111" LastActivityDate="2010-11-23T23:29:44.537" Title="Efficient Empirical CDF Computation / Storage" Tags="&lt;distributions&gt;&lt;python&gt;&lt;bioinformatics&gt;&lt;cdf&gt;&lt;biostatistics&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="4852" PostTypeId="2" ParentId="4851" CreationDate="2010-11-23T23:28:58.350" Score="1" ViewCount="" Body="&lt;p&gt;Here's a possible computing solution to your problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the pdf is sparse (by sparse I mean that long stretches are zero) as your graph indicates, then you could perhaps exploit that structure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, you could have one vector which gives the pdf for non-zero regions and another vector indicating where the non-zero regions begin and end.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could extend this idea if you have a large number of constant areas in your pdf. &lt;/p&gt;&#xA;" OwnerUserId="8" LastActivityDate="2010-11-23T23:28:58.350" />
  <row Id="4853" PostTypeId="2" ParentId="4851" CreationDate="2010-11-23T23:29:44.537" Score="5" ViewCount="" Body="&lt;p&gt;Why not create bins &lt;em&gt;a priori&lt;/em&gt; to cover the expected range of a variable, scan through the data once to record all the bin counts, and then estimate the percentiles via interpolation of the EDF?  If the bins are evenly distributed, locating the bin for a value is a $O(1)$ operation; in the worst case it's a binary search costing $O(\log(k))$ operations for $k$ bins.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-11-23T23:29:44.537" CommentCount="1" />
  <row Id="4854" PostTypeId="1" AcceptedAnswerId="4861" CreationDate="2010-11-23T23:50:10.213" Score="6" ViewCount="599" Body="&lt;p&gt;In Logistic Regression, is there a need to be as concerned about multicollinearity as you would be in straight up OLS regression? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, with a logistic regression, where multicollinearity exists, would you need to be cautious (as you would in OLS regression) with taking inference from the Beta coefficients? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For OLS regression one &quot;fix&quot; to high multicollinearity is ridge regression, is there something like that for logistic regression? Also, dropping variables, or combining variables. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What approaches are reasonable for reducing the effects of multicollinearity in a logistic regression? Are they essentially the same as OLS? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Note: this is not for the purpose of a designed experiment)&lt;/p&gt;&#xA;" OwnerUserId="776" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-11-24T15:01:27.013" LastActivityDate="2010-11-24T15:01:27.013" Title="Logistic Regression - Multicollinearity Concerns/Pitfalls" Tags="&lt;regression&gt;&lt;logistic-regression&gt;&lt;multicollinearity&gt;" AnswerCount="1" FavoriteCount="2" />
  <row Id="4855" PostTypeId="2" ParentId="4839" CreationDate="2010-11-24T01:22:36.983" Score="5" ViewCount="" Body="&lt;p&gt;The first step is to define your research question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A few possible research questions given your data include:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;How can genres of video games be grouped into a smaller set?&lt;/li&gt;&#xA;&lt;li&gt;How are genres of video games or groups of genres related to self-identifying as a hard-core gamer?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Then, you could present a table of frequencies and percentages of genre by hard-core gamer status.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could also divide the analysis into two steps:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;grouping types of video games; this could be done conceptually (e.g., based on prior knowledge of ways of grouping gaming genres) or using a data driven approach such as PCA, factor analysis (perhaps on &lt;a href=&quot;http://www.john-uebersax.com/stat/tetra.htm&quot; rel=&quot;nofollow&quot;&gt;tetrachoric correlations&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;li&gt;examine differences in endorsement of video gram types across hard-core gamer category.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;As mentioned by Brandon correspondence analysis would also be another nice option.&lt;/p&gt;&#xA;" OwnerUserId="183" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-24T07:56:08.127" LastActivityDate="2010-11-24T07:56:08.127" />
  <row Id="4856" PostTypeId="1" AcceptedAnswerId="4857" CreationDate="2010-11-24T01:36:32.617" Score="3" ViewCount="183" Body="&lt;p&gt;How can I rewrite an AR(p) model in state-space form?&#xA;Max(p)=5 and I want to use Kalman Predictor.&lt;/p&gt;&#xA;" OwnerUserId="1637" LastActivityDate="2010-11-24T12:49:59.810" Title="Rewriting AR model in State-Space form" Tags="&lt;time-series&gt;&lt;kalman-filter&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="4" />
  <row Id="4857" PostTypeId="2" ParentId="4856" CreationDate="2010-11-24T02:11:51.600" Score="3" ViewCount="" Body="&lt;p&gt;I suggest you buy the excellent book by G. Petris, S. Petrone and P. Campagnoli &lt;a href=&quot;http://www.springer.com/statistics/statistical+theory+and+methods/book/978-0-387-77237-0&quot; rel=&quot;nofollow&quot;&gt;Dynamic Linear Models with R&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You will learn that any ARMA model &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Y_t = \sum_{j=1}^{r}\phi_jY_{t-j} + \sum_{j=1}^{r-1}\psi_{j}\epsilon_t$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;can be expressed in the following form:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$  &#xD;&#xA;\begin{matrix}  &#xD;&#xA;Y_t &amp;amp; = &amp;amp; F\theta_t\\     &#xD;&#xA;\theta_{t+1} &amp;amp; = &amp;amp; G\theta_{t}+R\epsilon_t  &#xD;&#xA;\end{matrix}  $&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;with&#xA;$&#xD;&#xA;F=\begin{bmatrix}&#xD;&#xA;1 &amp;amp; 0 &amp;amp; ... &amp;amp; 0&#xD;&#xA;\end{bmatrix}  &#xD;&#xA;$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$&#xD;&#xA;G=\begin{bmatrix}&#xD;&#xA;\phi_1 &amp;amp; 1 &amp;amp; 0 &amp;amp; ... &amp;amp; 0\\ &#xD;&#xA;\phi_2 &amp;amp; 0 &amp;amp; 1 &amp;amp; ... &amp;amp; 0\\ &#xD;&#xA;... &amp;amp; ... &amp;amp; ... &amp;amp; ... &amp;amp; ...\\ &#xD;&#xA;\phi_{r-1} &amp;amp; 0 &amp;amp; ... &amp;amp; 0 &amp;amp; 1\\ &#xD;&#xA;\phi_r &amp;amp; 0 &amp;amp; ... &amp;amp; 0 &amp;amp; 0&#xD;&#xA;\end{bmatrix}  &#xD;&#xA;$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$&#xD;&#xA;R={\begin{bmatrix}&#xD;&#xA;1 &amp;amp; \psi_1 &amp;amp; ... &amp;amp; \psi_{r-2} &amp;amp; \psi_{r-1}&#xD;&#xA;\end{bmatrix}}&amp;#39;&#xD;&#xA;$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your specific case, just set $r=5$ and $\psi_{j=1..5}=0$.&lt;br&gt;&#xA;You can use the package &lt;a href=&quot;http://www.google.com.sg/url?sa=t&amp;amp;source=web&amp;amp;cd=2&amp;amp;sqi=2&amp;amp;ved=0CB0QFjAB&amp;amp;url=http%3A%2F%2Fcran.us.r-project.org%2Fweb%2Fpackages%2Fdlm%2Findex.html&amp;amp;ei=5XzsTIL_GcbirAeT9-TpAw&amp;amp;usg=AFQjCNFU5Fn1Pjk6h3yIrtuUC2ttt4RIVA&quot; rel=&quot;nofollow&quot;&gt;dlm&lt;/a&gt; to use the Kalman filter on this model.&lt;br&gt;&#xA;fRed&lt;/p&gt;&#xA;" OwnerUserId="1709" LastEditorUserId="1709" LastEditorDisplayName="" LastEditDate="2010-11-24T10:57:39.583" LastActivityDate="2010-11-24T10:57:39.583" CommentCount="5" />
  <row Id="4858" PostTypeId="1" CreationDate="2010-11-24T04:12:41.550" Score="3" ViewCount="685" Body="&lt;p&gt;I have been reading a good book called &lt;a href=&quot;http://gseacademic.harvard.edu/alda/&quot; rel=&quot;nofollow&quot;&gt;Applied Longitudinal Data Analysis: Modeling Change and Event Occurrence&lt;/a&gt; by Judith Singer and John Willet. The book shows that by modeling in 2 levels, we can model the individual change in level 1 and in level 2 model for systematic interindividual differences in change.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;a href=&quot;http://www.ats.ucla.edu/stat/r/examples/alda/&quot; rel=&quot;nofollow&quot;&gt;R codes&lt;/a&gt; for the examples only shows how to do the lme to obtain the fixed and random effects. However, the text  suggested that we should test for the variance components whether these random effects are significant or not. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, one of the code does only the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(nlme)&#xA;&#xA;model.a &amp;lt;- lme(alcuse~ 1, alcohol1, random= ~1 |id)&#xA;summary(model.a)&#xA;&#xA;Linear mixed-effects model fit by REML&#xA; Data: alcohol1 &#xA;       AIC      BIC    logLik&#xA;  679.0049 689.5087 -336.5025&#xA;&#xA;Random effects:&#xA; Formula: ~1 | id&#xA;        (Intercept)  Residual&#xA;StdDev:   0.7570578 0.7494974&#xA;&#xA;Fixed effects: alcuse ~ 1 &#xA;                Value  Std.Error  DF  t-value p-value&#xA;(Intercept) 0.9219549 0.09629638 164 9.574139       0&#xA;&#xA;Standardized Within-Group Residuals:&#xA;       Min         Q1        Med         Q3        Max &#xA;-1.8892070 -0.3079143 -0.3029178  0.6110925  2.8562135 &#xA;&#xA;Number of Observations: 246&#xA;Number of Groups: 82&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But the text narration would have the following: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;fixed effect: 0.922*** (s = 0.096) -&gt; available in the output &lt;/li&gt;&#xA;&lt;li&gt;within person variance: 0.562*** (s = 0.062) -&gt; can be obtained from the output (random effect residual std. dev squared)&lt;/li&gt;&#xA;&lt;li&gt;between person variance: 0.564*** (s = 0.119)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;My work involves a lot of analysis for longitudinal data so I really need to get a hold of this idea. Your help is very much appreciated.&lt;/p&gt;&#xA;" OwnerUserId="1663" LastActivityDate="2010-11-24T20:48:13.937" Title="Longitudinal Data analysis by multilevel modeling in R" Tags="&lt;r&gt;&lt;longitudinal&gt;" AnswerCount="4" FavoriteCount="2" />
  <row Id="4859" PostTypeId="2" ParentId="4830" CreationDate="2010-11-24T04:31:36.440" Score="2" ViewCount="" Body="&lt;p&gt;You might just try including all of the interaction effects. You can then use L1/L2-regularized logistic regression to minimize over-fitting and take advantage of any helpful features. I really like Hastie/Tibshirani's glmnet package (http://cran.r-project.org/web/packages/glmnet/index.html).&lt;/p&gt;&#xA;" OwnerUserId="2077" LastActivityDate="2010-11-24T04:31:36.440" CommentCount="2" />
  <row Id="4860" PostTypeId="2" ParentId="3061" CreationDate="2010-11-24T05:05:17.127" Score="1" ViewCount="" Body="&lt;p&gt;This statistic measures a &lt;a href=&quot;http://www.jstor.org/pss/2532051&quot; rel=&quot;nofollow&quot;&gt;kind of correlation&lt;/a&gt; between two sets of data.  Its calculation requires no assumptions about what their scatterplot looks like (if that's what you mean by &quot;tendency&quot;).&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-11-24T05:05:17.127" />
  <row Id="4861" PostTypeId="2" ParentId="4854" CreationDate="2010-11-24T05:13:29.350" Score="8" ViewCount="" Body="&lt;p&gt;All of the same principles concerning multicollinearity apply to logistic regression as they do to OLS. The same diagnostics assessing multicollinearity can be used (e.g. VIF, condition number, auxiliary regressions.), and the same dimension reduction techniques can be used (such as combining variables via principal components analysis).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This &lt;a href=&quot;http://stats.stackexchange.com/questions/4272/when-to-use-regularization-methods-for-regression/4274#4274&quot;&gt;answer&lt;/a&gt; by chl will lead you to some resources and R packages for fitting penalized logistic models (as well as a good discussion on these types of penalized regression procedures). But some of your comments about &quot;solutions&quot; to multicollinearity are a bit disconcerting to me. If you only care about estimating relationships for variables that are not collinear these &quot;solutions&quot; may be fine, but if your interested in estimating coefficients of variables that are collinear these techniques do not solve your problem. Although the problem of multicollinearity is technical in that your matrix of predictor variables can not be inverted, it has a logical analog in that your predictors are not independent, and their effects cannot be uniquely identified.&lt;/p&gt;&#xA;" OwnerUserId="1036" LastEditorUserId="1036" LastEditorDisplayName="" LastEditDate="2010-11-24T13:14:11.800" LastActivityDate="2010-11-24T13:14:11.800" CommentCount="3" />
  <row Id="4862" PostTypeId="2" ParentId="4858" CreationDate="2010-11-24T05:32:41.863" Score="2" ViewCount="" Body="&lt;p&gt;One of the best resources on multilevel analysis in R is John Fox's web appendix to the text &quot;An R and S-PLUS Companion to Applied Regression&quot;.  It provides a great overview of the method and means to calculate some of the more familiar measures from the R NLME output.  The appendix is available on CRAN.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the link:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://cran.r-project.org/doc/contrib/Fox-Companion/appendix-mixed-models.pdf&quot; rel=&quot;nofollow&quot;&gt;http://cran.r-project.org/doc/contrib/Fox-Companion/appendix-mixed-models.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="485" LastActivityDate="2010-11-24T05:32:41.863" />
  <row Id="4863" PostTypeId="2" ParentId="4810" CreationDate="2010-11-24T05:35:57.333" Score="2" ViewCount="" Body="&lt;p&gt;I guess it depends on what statistics or findings you are going to find out, research, study, or report. I'm assuming you will prob be using these graphs to represent findings for your university topic, right? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Like for example, if you want to present your finding about say, 'How long users stays on a a certain website', it may be good to show it in CDF as it shows the accumulated time he spent on that website, through the pages etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, if you want to simply show the probability of users clicking on an advert link (e.g. Google adwords link) then you may want to present it in PDF form as it will probably be a normal distribution bell curve and you can show the probability of that heppening.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps,&#xA;Jeff&lt;/p&gt;&#xA;" LastActivityDate="2010-11-24T05:35:57.333" />
  <row Id="4864" PostTypeId="2" ParentId="4830" CreationDate="2010-11-24T05:41:16.503" Score="1" ViewCount="" Body="&lt;p&gt;I know your question is about logistic regression and as it is a homework assignment so your approach may be constrained.  However, if your interest is in interactions and accuracy of classification, it might be interesting to use something like CART to model this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's some R code to do produce the basic tree.  I've set rpart loose on the enire data frame here.  Perhaps not the best approach without some prior knowledge and a cross validation method:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(foreign)&#xA;df &amp;lt;- read.spss(file=&quot;http://www.bertelsen.ca/R/logistic-regression.sav&quot;, use.value.labels=T, to.data.frame=T) &#xA;library(rpart) &#xA;fit&amp;lt;-rpart(default~.,method=&quot;anova&quot;,data=df)&#xA; pfit&amp;lt;- prune(fit, cp=   fit$cptable[which.min(fit$cptable[,&quot;xerror&quot;]),&quot;CP&quot;])&#xA;&#xA;# plot the pruned tree &#xA; plot(pfit, uniform=TRUE, &#xA;   main=&quot;Pruned Classification Tree for Loan Default&quot;)&#xA;text(pfit, use.n=TRUE, all=TRUE, cex=.8)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm not sure right off how to produce the classifcation table.  It shouldn't be too hard from the predicted values from the model object and the original values.  Anyone have any tips here?&lt;/p&gt;&#xA;" OwnerUserId="485" LastEditorUserId="485" LastEditorDisplayName="" LastEditDate="2010-11-24T06:51:58.127" LastActivityDate="2010-11-24T06:51:58.127" CommentCount="3" />
  <row Id="4866" PostTypeId="2" ParentId="4858" CreationDate="2010-11-24T07:54:23.777" Score="3" ViewCount="" Body="&lt;p&gt;The &lt;code&gt;intervals()&lt;/code&gt; function should provide you with $100(1-\alpha)$ confidence intervals for the random effects in your model, see &lt;code&gt;help(intervals.lme)&lt;/code&gt; for more information.&#xA;You can also test if any of the variance components can be droped from the model by using &lt;code&gt;anova()&lt;/code&gt; (which amounts to do an LRT between two nested models).&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-11-24T07:54:23.777" CommentCount="0" />
  <row Id="4867" PostTypeId="2" ParentId="4839" CreationDate="2010-11-24T08:13:19.087" Score="2" ViewCount="" Body="&lt;p&gt;As an alternative to CA suggested by @Brandon, you could also try &lt;a href=&quot;http://en.wikipedia.org/wiki/Multiple_correspondence_analysis&quot; rel=&quot;nofollow&quot;&gt;Multiple Correspondence Analysis&lt;/a&gt; which has the advantage of considering all types of games at the same time (unlike CA), which are probably scored as binary variable (in this case, the MCA solution will be close to the PCA one--factor scores and eigenvalues are linearly related). Basically this will give you an idea of how games group together, if any. At the same time, you can use your &quot;hardcore gamer&quot; status (yes/no) as an illustrative variable (i.e., this variable will not participate to the construction of the factorial axes), which will help you identifying how it related to these clusters of variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;a href=&quot;http://cran.r-project.org/web/packages/FactoMineR/index.html&quot; rel=&quot;nofollow&quot;&gt;FactoMineR&lt;/a&gt; R package offers all of what is needed for such kind of analysis, see &lt;code&gt;MCA()&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you didn't say anything about your sample size, it's hard to suggest confirmatory or model-based approaches, like logistic regression or latent class regression. But you can look at &lt;a href=&quot;http://en.wikipedia.org/wiki/Random_forest&quot; rel=&quot;nofollow&quot;&gt;Random Forests&lt;/a&gt; and try to identify the &quot;best&quot; variables that allow to predict your outcome with a minimal classification error rate (see the &lt;a href=&quot;http://cran.r-project.org/web/packages/randomForest/index.html&quot; rel=&quot;nofollow&quot;&gt;randomForest&lt;/a&gt; package).&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-11-24T08:13:19.087" />
  <row Id="4868" PostTypeId="1" AcceptedAnswerId="4871" CreationDate="2010-11-24T08:47:26.607" Score="4" ViewCount="432" Body="&lt;p&gt;Hello,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possible to do k-fold cross-validation to &lt;strong&gt;test&lt;/strong&gt; all data, rather than using kfcv to find the optimal hypothesis as is typically done.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say I want to use a svms on a dataset of size 1000. Could I use 900 events to train the svm for the testing of the other 100 events. Then use a separate 900 events to train the svm for a separate 100 events, repeating this process 10 times until all of the data has been tested.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would events tested using separately trained svms be comparable? i.e. through this technique could I then use my entire dataset instead of setting aside a certain fraction for training, or is this a statistically unwise thing to do?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for any help,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Colorado&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS) Any references to why this would['nt] work is greatly appreciated, I'm discussing this technique in an academic setting.&lt;/p&gt;&#xA;" OwnerUserId="2113" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-11-24T12:47:15.397" LastActivityDate="2010-11-26T05:57:08.417" Title="Using k-fold cross-validation to test all data" Tags="&lt;machine-learning&gt;&lt;cross-validation&gt;" AnswerCount="3" FavoriteCount="2" />
  <row Id="4869" PostTypeId="2" ParentId="4184" CreationDate="2010-11-24T08:52:57.620" Score="1" ViewCount="" Body="&lt;p&gt;The paper &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.ricam.oeaw.ac.at/Groebner-Bases-Bibliography/gbbib_files/publication_582.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.ricam.oeaw.ac.at/Groebner-Bases-Bibliography/gbbib_files/publication_582.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;by Pistone et al seems to be the source paper for this.&lt;/p&gt;&#xA;" OwnerUserId="2114" LastActivityDate="2010-11-24T08:52:57.620" />
  <row Id="4870" PostTypeId="2" ParentId="4858" CreationDate="2010-11-24T09:28:58.507" Score="7" ViewCount="" Body="&lt;p&gt;First point: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You need to be careful if you want to test whether the variance of a random effect is 0:&#xA;The standard $\chi^2$-asymptotics for the LR that are used in &lt;code&gt;anova()&lt;/code&gt; do not apply for LRTs or restricted likelihood ratio tests on variance components since the null hypothesis is on the edge of the parameter space. Using this wrong reference distribution will yield ridiculously conservative tests. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sources: &lt;a href=&quot;http://www.jstor.org/stable/2289471&quot; rel=&quot;nofollow&quot;&gt;Self &amp;amp; Liang (1987)&lt;/a&gt;, &lt;a href=&quot;http://onlinelibrary.wiley.com/doi/10.1111/j.1467-9868.2004.00438.x/full&quot; rel=&quot;nofollow&quot;&gt;Crainiceanu &amp;amp; Ruppert (2003)&lt;/a&gt;, &lt;a href=&quot;http://www.stat.uni-muenchen.de/~greven/files/greven_iwsm07.pdf&quot; rel=&quot;nofollow&quot;&gt;Greven, Crainiceanu, Küchenhoff (2008)&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For exact LR tests on variances in linear mixed models with uncorrelated random effects you can use my package &lt;a href=&quot;http://cran.r-project.org/web/packages/RLRsim/index.html&quot; rel=&quot;nofollow&quot;&gt;RLRsim&lt;/a&gt;. &#xA;For generalized linear mixed models or models with correlated random effects, I would strongly recommend a parametric bootstrap to approximate the correct p-value if the LRT or RLRT (the latter has more power, see sources above) are somewhere in the vicinity of the critical value for the (wrong) standard reference distribution. Code for &lt;code&gt;lme4&lt;/code&gt; at the end of the post. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second point:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It may be dangerous to use standard confidence intervals for variance parameter estimates, since their distribution is usually very skewed and so using a symmetric interval is a little simplistic- Douglas Bates has some material on this &lt;a href=&quot;http://lme4.r-forge.r-project.org/book/&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Parametric bootstrap code for &lt;code&gt;lme4&lt;/code&gt;-models:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#m0 is the lmer model under the null hypothesis (i.e. the smaller model)&#xA;#mA is the lmer model under the alternative&#xA;bootstrapAnova &amp;lt;- function(mA, m0, B=1000){&#xA;     oneBootstrap &amp;lt;- function(m0, mA){&#xA;         d &amp;lt;- drop(simulate(m0))&#xA;         m2 &amp;lt;-refit(mA, newresp=d)&#xA;         m1 &amp;lt;-refit(m0, newresp=d)&#xA;         return(anova(m2,m1)$Chisq[2])&#xA;     }  &#xA;     nulldist &amp;lt;-  if(!require(multicore)){&#xA;         replicate(B, oneBootstrap(m0, mA))&#xA;     } else {&#xA;         unlist(mclapply(1:B, function(x) oneBootstrap(m0, mA)))&#xA;     }   &#xA;     ret &amp;lt;- anova(mA, m0)&#xA;     ret$&quot;Pr(&amp;gt;Chisq)&quot;[2] &amp;lt;- mean(ret$Chisq[2] &amp;lt; nulldist)&#xA;     names(ret)[7] &amp;lt;- &quot;Pr_boot(&amp;gt;Chisq)&quot;&#xA;     attr(ret, &quot;heading&quot;) &amp;lt;- c(attr(ret, &quot;heading&quot;)[1], &#xA;          paste(&quot;Parametric bootstrap with&quot;, B,&quot;samples.&quot;),&#xA;          attr(ret, &quot;heading&quot;)[-1])&#xA;     attr(ret, &quot;nulldist&quot;) &amp;lt;- nulldist&#xA;     return(ret)&#xA;}&#xA;#use like this (and increase B if you want reviewers to believe you):&#xA;(bRLRT &amp;lt;- bootstrapAnova(mA=&amp;lt;BIG MODEL&amp;gt;, m0=&amp;lt;SMALLER MODEL&amp;gt;))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1979" LastActivityDate="2010-11-24T09:28:58.507" CommentCount="1" />
  <row Id="4871" PostTypeId="2" ParentId="4868" CreationDate="2010-11-24T10:43:38.363" Score="5" ViewCount="" Body="&lt;p&gt;As far as I understand your question, it can be formulated this way:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Instead of calculating a quality measure for each of the k validation-folds and then calculate the average, may I aggregate all folds an then calculate my quality measure, hence getting only one instead of k values?&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This question requires two perspectives:&lt;br&gt;&#xA;From the perspective of crossvalidation itself it is ok, because training- and testsamples still have an empty intersection etc. Since you just aggregate multiple samples drawn without replacement, the test-distribution is not spoiled. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the perspective of the model, it depends whether the model produces comparable scores. SVM will work in my opinion, but imagine a model which min-max-normalizes the scores across the test-set (iiek), so that the calculation of a representative decision threshold across all test-samples will be quite hard.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, a lot of techniques which require the estimation of a parameter, which itself depends on the quality of a model, utilize this approach. A concrete example is the calculating of an operator to calibrate the scores of a classification model, e.g. Platt Scaling. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furthermore (also this not an completely satisfying argument) the open source software Rapidminer has an operator for this approach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: &#xA;I want to point out, that although this approach is useful to get reliable quality measures for only small datasets, it may be hard to perform statistical tests to compare the significance of two models, since cv cannot be repeated endlessly (example: how to estimate whether the assumption of the so-often-misused t-test is satisfied if you only have 6 data points?).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PPS: Also interested, I was not able to find a paper focusing on the examination of this approach. The papers I have seen so far using this technique did not bother to reference it.&lt;/p&gt;&#xA;" OwnerUserId="264" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-11-24T12:48:29.310" LastActivityDate="2010-11-24T12:48:29.310" CommentCount="5" />
  <row Id="4872" PostTypeId="1" CreationDate="2010-11-24T10:56:16.390" Score="3" ViewCount="151" Body="&lt;p&gt;what is the minimum entry degree to be employed as a statistician&lt;/p&gt;&#xA;" OwnerUserId="2120" LastActivityDate="2010-11-24T10:56:16.390" ClosedDate="2010-11-24T11:42:41.677" Title="degree to become a statistician" Tags="&lt;careers&gt;" CommentCount="4" />
  <row Id="4873" PostTypeId="2" ParentId="4830" CreationDate="2010-11-24T11:43:04.703" Score="5" ViewCount="" Body="&lt;p&gt;In unbalanced datasets such as this, you can usually improve classification performance by moving away from using a fitted probability of .5 as your cutpoint for classifying cases into defaulters and non-defaulters. For example, I get correct classification rates of .88 and .58 with a cutpoint of .4 for a glm with all 2nd-order interactions. (Which probably leads to overfitting and seems to have some rank issues, but that's another story.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;m &amp;lt;- glm(default ~ (age + employ + address + income + debtinc + &#xA;                    creddebt + othdebt + ed)^2,&#xA;   family=binomial(), data=df)&#xA;p &amp;lt;- predict(m, newdata=df, type=&quot;response&quot;)&#xA;&#xA;getMisclass &amp;lt;- function(cutoff, p, labels){&#xA;   pred &amp;lt;- factor(1*(p &amp;gt; cutoff), labels=c(&quot;No Default&quot;, &quot;Default&quot;)) &#xA;   t &amp;lt;- table(pred, labels)&#xA;   cat(&quot;cutoff &quot;, cutoff, &quot;:\n&quot;)&#xA;   print(t)&#xA;   cat(&quot;correct    :&quot;, round(sum(t[c(1,4)])/sum(t), 2),&quot;\n&quot;)&#xA;   cat(&quot;correct No :&quot;, round(t[1]/sum(t[,1]), 2),&quot;\n&quot;)&#xA;   cat(&quot;correct Yes:&quot;, round(t[4]/sum(t[,2]), 2),&quot;\n\n&quot;)&#xA;   invisible(t)&#xA;}&#xA;cutoffs &amp;lt;- seq(.1,.9,by=.1)&#xA;sapply(cutoffs, getMisclass, p=p, labels=df$default)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;partial output:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cutoff  0.3 :&#xA;            labels&#xA;pred           No  Yes&#xA;  No Default 3004  352&#xA;  Default     740  903&#xA;correct    : 0.78 &#xA;correct No : 0.8 &#xA;correct Yes: 0.72 &#xA;&#xA;cutoff  0.4 :&#xA;            labels&#xA;pred           No  Yes&#xA;  No Default 3278  532&#xA;  Default     466  723&#xA;correct    : 0.8 &#xA;correct No : 0.88 &#xA;correct Yes: 0.58 &#xA;&#xA;cutoff  0.5 :&#xA;        labels&#xA;pred           No  Yes&#xA;  No Default 3493  685&#xA;  Default     251  570&#xA;correct    : 0.81 &#xA;correct No : 0.93 &#xA;correct Yes: 0.45 &#xA;&#xA;cutoff  0.6 :&#xA;            labels&#xA;pred           No  Yes&#xA;  No Default 3606  824&#xA;  Default     138  431&#xA;correct    : 0.81 &#xA;correct No : 0.96 &#xA;correct Yes: 0.34 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1979" LastActivityDate="2010-11-24T11:43:04.703" CommentCount="2" />
  <row Id="4874" PostTypeId="2" ParentId="4868" CreationDate="2010-11-24T12:43:13.730" Score="1" ViewCount="" Body="&lt;p&gt;Yes it is; and while this is a very reliable way of reporting error, I would say it is even encouraged.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-11-24T12:43:13.730" />
  <row Id="4875" PostTypeId="2" ParentId="4856" CreationDate="2010-11-24T12:49:59.810" Score="2" ViewCount="" Body="&lt;p&gt;Another good book that covers this is &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0198523548&quot; rel=&quot;nofollow&quot;&gt;Time Series Analysis by State Space Methods&lt;/a&gt; by Durbin and Koopman (pp 46-49.)&lt;/p&gt;&#xA;" OwnerUserId="439" LastActivityDate="2010-11-24T12:49:59.810" />
  <row Id="4876" PostTypeId="1" CreationDate="2010-11-24T13:20:08.147" Score="0" ViewCount="43" Body="&lt;p&gt;I am trying to track the performance of a set of homogeneous subsystems vs the system as a whole. The performance metric is measured by a mean value based on a set of tasks performed by a the subsystem, for example, the mean wait time before scheduling task.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each subsystem may be responsible for varying numbers of tasks. I can calculate the mean wait time for each subsystem and for the system as a whole.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is how to measure and visualize how each subsystem is performing compared to the whole system average (or maybe there's a better aggregate measure for the whole system?)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Obviously being below the mean shows the subsystem is performing relatively better and above the mean vice versa, and the distance from the whole system mean represents that, but I struggle to get real meaning or provide a good visualization from that. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a better way? Or if I'm on the right track, how do I assign meaning / visualize the metric?&lt;/p&gt;&#xA;" OwnerUserId="2122" LastActivityDate="2010-11-24T16:15:31.593" Title="Comparing and visualizing rates of subsystems" Tags="&lt;data-visualization&gt;&lt;mean&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="4877" PostTypeId="1" AcceptedAnswerId="4879" CreationDate="2010-11-24T14:15:28.017" Score="2" ViewCount="168" Body="&lt;p&gt;I have imported a datafile in R. It has different columns. There is one column that has the name of Operating System belonging to that row information. I wanted to get the &lt;strong&gt;percentage&lt;/strong&gt; share of each unique OS (windows, linux, ios etc) from this column and plot it. I am very new to R and would like to know if there is any inbuilt way to do this in R. &lt;/p&gt;&#xA;" OwnerUserId="2101" LastEditorUserId="2101" LastEditorDisplayName="" LastEditDate="2010-11-24T14:56:18.557" LastActivityDate="2010-11-24T20:49:04.323" Title="Creating plots for String type columns in R" Tags="&lt;r&gt;&lt;data-visualization&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="4878" PostTypeId="1" CreationDate="2010-11-24T14:21:34.157" Score="2" ViewCount="151" Body="&lt;p&gt;We're using Excel Solver to model a DEA problem.  We've worked with a few resources and feel confident that the model is returning correct results but I need some help interpreting them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When solver completes its analysis, it provides an answer report that gives the slack found in some of the constraints -- which are binding and which are not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's a sample result for a DMU that was found to be inefficient when Solving to Minimize the Inputs:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;              WeightedSum             Outputs = DMU&#xA;           of Reference Set          Inputs = DMU * E   Slack&#xA;Output1        2,892,230     &amp;gt;=        2,892,230        binding&#xA;Output2        66,229,077    &amp;gt;=        66,229,077       binding&#xA;Output3        9,724,273     &amp;gt;=        5,372,010        4,352,263&#xA;Output4        3,123,300     &amp;gt;=        2,511,864        611,436&#xA;Output5        21,609        &amp;gt;=        16,039           5,570&#xA;Output6        120,527       &amp;gt;=        33,234           87,292&#xA;Output7        8,548         &amp;gt;=        2,769            5,779&#xA;Output8        17,314        &amp;gt;=        12,605           4,709&#xA;Output9        748           &amp;gt;=        616              132&#xA; Input1        19,775        &amp;lt;=        23,314           3,539&#xA; Input2        9,132         &amp;lt;=        9,132            binding&#xA; Input3        20,028        &amp;lt;=        143,604          123,576&#xA; Input4        74,293        &amp;lt;=        74,293           binding&#xA; Input5        15,558        &amp;lt;=        15,558           binding&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is it correct to assert that those inputs found to be binding could not be further optimized but those that contained slack are where the inefficiencies are found?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can the same be said for Outputs? or does that work the opposite?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or is slack just a bi-product of the analysis and not relevant?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for your help!&lt;/p&gt;&#xA;" OwnerUserId="1256" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-11-24T15:16:06.487" LastActivityDate="2010-11-24T15:44:21.667" Title="Slack values in Data Envelopment Analysis" Tags="&lt;econometrics&gt;&lt;excel&gt;" AnswerCount="2" />
  <row Id="4879" PostTypeId="2" ParentId="4877" CreationDate="2010-11-24T14:29:50.333" Score="2" ViewCount="" Body="&lt;p&gt;It seems the &lt;code&gt;barplot()&lt;/code&gt; will be your friend in that case, e.g.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x &amp;lt;- sample(c(&quot;Win&quot;,&quot;Linux&quot;,&quot;Mac&quot;), 100, replace=TRUE)&#xA;barplot(table(x))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This will work for variables of type &lt;code&gt;character&lt;/code&gt; or &lt;code&gt;factor&lt;/code&gt;. Another option is to use Cleveland's &lt;a href=&quot;http://www.b-eye-network.com/view/2468&quot; rel=&quot;nofollow&quot;&gt;dotplot&lt;/a&gt;, see &lt;code&gt;dotchart()&lt;/code&gt; (or &lt;code&gt;dotplot()&lt;/code&gt;in the &lt;a href=&quot;http://cran.r-project.org/web/packages/lattice/index.html&quot; rel=&quot;nofollow&quot;&gt;lattice&lt;/a&gt; package).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could replace &lt;code&gt;table(x)&lt;/code&gt; by &lt;code&gt;table(x)/sum(table(x))*100&lt;/code&gt; to express data as % rather than counts. I know there are more elegant solutions in additional packages, but I can't remember their names actually. The &lt;code&gt;table()&lt;/code&gt; function will also work for two-way classification, and marginal totals can easily be computed in a similar way; e.g. &lt;code&gt;apply(table(x, y), 1, sum)&lt;/code&gt; gives rows marginal frequencies.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-24T15:04:50.940" LastActivityDate="2010-11-24T15:04:50.940" CommentCount="4" />
  <row Id="4880" PostTypeId="2" ParentId="4877" CreationDate="2010-11-24T15:06:04.903" Score="4" ViewCount="" Body="&lt;p&gt;Simulating some data for a good start&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x&amp;lt;-sample(c(&quot;Win&quot;,&quot;Linux&quot;,&quot;Mac&quot;),721,replace=TRUE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The good practice is to hold such categorical variables as factors; one can convert character vector to factor with &lt;code&gt;factor&lt;/code&gt; function:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x&amp;lt;-factor(x)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then, &lt;code&gt;plot&lt;/code&gt; called on it will make a barplot with counts; to convert it into per-cents, you can use &lt;code&gt;table&lt;/code&gt; that makes counts of each level:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;t&amp;lt;-table(x); barplot(t/sum(t)*100,ylab=&quot;Per cent&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="88" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-24T20:49:04.323" LastActivityDate="2010-11-24T20:49:04.323" />
  <row Id="4881" PostTypeId="2" ParentId="4878" CreationDate="2010-11-24T15:30:23.087" Score="3" ViewCount="" Body="&lt;p&gt;As I understand data envelopment analysis, you optimize efficiency of the decision making unit (DMU) subject to a set of constraints on input availability and output requirements. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since you have identified the maximum efficiency the non-binding input constraints indicate that the amount of slack in inputs is unnecessary expenditure and can be avoided without sacrificing efficiency.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the slack with respect to output constraints needs to be dealt with more carefully. If you do not care about producing more than the minimum required then the slack on the output constraints is irrelevant. However, if you do care about producing more than the minimum required then perhaps you should either penalize excess output suitably or transform the greater than equal constraints into an equality constraint.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do note that imposing penalties or converting your output constraints into equality constraints may decrease efficiency but that is a trade-off you need to take keeping the context in mind.&lt;/p&gt;&#xA;" LastActivityDate="2010-11-24T15:30:23.087" />
  <row Id="4882" PostTypeId="2" ParentId="4878" CreationDate="2010-11-24T15:44:21.667" Score="2" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Is it correct to assert that those inputs found to be binding could not be further optimized but those that contained slack are where the inefficiencies are found?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;First of all, this is a multivariate problem, so it is really hard to conclude anything about individual inputs/outputs. However I would rather say the opposite - the inputs where the constraints are binding could be further optimized except you are hitting a boundary condition that stops you from doing so, while inputs where there is some slack left found an optimum. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a simple example, suppose you want to minimize $(x-3)^2+y$ with $x \geq 0$, $y\geq 0$ constraints. You will find the optimum at $x=3$, $y=0$ with $x$ having a slack of 3, and $y$ being a binding constraint. That's because you could get smaller values of your output if $y$ were allowed to be negative, but you prohibited that. On the other hand, $x$ would not benefit from allowing negative values. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, one cannot make far-reaching conclusions, because minimizing $(x-3)^2+y^2$ would give the same results even though having negative $y$'s would not help. Similarly, it is possible to create a function where the global minimum would occur for a negative $x$ even though there is a local minimum at $x=3$. Additionally, with constraints on the output, even inputs with slack might not be local optima because one of the output constraints could have kicked in. You can check the latter by looking at derivatives (I think Solver might show them) - a zero derivative implies a local optimum (or inflection point, but that's less likely).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sorry for the rambling answer, but the main point is that if your problem is sufficiently complex, then any simple interpretation is likely be wrong in some cases.&lt;/p&gt;&#xA;" OwnerUserId="279" LastActivityDate="2010-11-24T15:44:21.667" />
  <row Id="4883" PostTypeId="2" ParentId="4876" CreationDate="2010-11-24T16:15:31.593" Score="2" ViewCount="" Body="&lt;p&gt;It sounds like you should lookup the topic of &lt;a href=&quot;http://en.wikipedia.org/wiki/Statistical_process_control&quot; rel=&quot;nofollow&quot;&gt;statistical process control&lt;/a&gt;. Its origins lie in manufacturing where deviations from expected process output are used to raise flags to ensure that quality of manufactured items is as per pre-established norms. In particular, see the wikipedia description for &lt;a href=&quot;http://en.wikipedia.org/wiki/Control_chart&quot; rel=&quot;nofollow&quot;&gt;control charts&lt;/a&gt; which should get you started on the visualization aspect. &lt;/p&gt;&#xA;" LastActivityDate="2010-11-24T16:15:31.593" />
  <row Id="4884" PostTypeId="1" AcceptedAnswerId="4890" CreationDate="2010-11-24T16:23:43.947" Score="2" ViewCount="169" Body="&lt;p&gt;Hello fellow number crunchers&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope this a valid question for this forum. I am a lonesome quarter-statistician and have trouble finding someone to ask.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Introduction:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/A/B_testing&quot; rel=&quot;nofollow&quot;&gt;Wikipedia Explanation of AB-Tests&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.abtests.com/&quot; rel=&quot;nofollow&quot;&gt;A ton of examples for typical use of AB-Tests&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The AB-Test has become really popular since it is so easy to implement and execute. Additionally the web is floated with blogs explaining how to determine the significance of the results. All in all, it seems that there is less discussion about the control or exclusion of possible &quot;influential&quot; variables (on the other hand, controlling such variables is quite hard on the web).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most AB-Tests comparing the outcomes of both groups by simple counting how many clicks or conversions every group has generated. Than a binomial distribution is assumed for each group and hence statistical tests are performed to see which group got the greater p. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;So the question is:&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Is it &quot;better&quot; to compare the outcome of both groups without any aggregation or it is &quot;better&quot; to aggregate e.g. on daily basis ?&lt;/em&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Example:&lt;/strong&gt;&#xA;The AB-Test is to check whether a landing page creates more newsletter subscriptions (&amp;lt;- conversions in this case). The AB-Test is deployed/online the whole test-time. In group A the landing page's main color is blue, in group B the main color is red. Assume 2000 visitors per day, i.e. each group gets roughly 1000 visitors. In this case &quot;without aggregation&quot; means, that I get 1000 datapoints per group per day meanwhile &quot;aggregation on daily basis&quot; means, that I get one (!) datapoint per group per day.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Discussion&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The latter (aggregation on daily baiss) would allow the pairing of values, which in turn can capture daily effects like peaks in user behavior and preferences, but it extends the duration of AB-Tests, because it takes longer to collect enough datapoints. On the other hand, the former (no aggregation) seems to the strategy of the majority, because ... I dont know, maybe because a) with enough traffic you can make nearly anything significant within one day b) it is the easiest thing to do.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One (possibly influencing) example to stimulate your thoughts: &#xA;Assume that on one day during the test the &quot;National Blue Day&quot; is celebrated, so the color blue together with positive emotions is visible in all the media. This day group A has created a ton of conversions more than group B. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This difference clearly affects the test if aggregated on daily basis (increase of variance) or (if not aggregated at all) it either vanishes in the sea of data (in the case that multiple days are collected without aggregation) or it leads to the wrong results (if the day is the first and only day of the test).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another example: Assume that the landing page belongs to a vegetable company. One day the &quot;National Vegetable Day&quot; is celebrated and now everyone wants to subscribe to the newsletter, no matter what the color is. This short-time effect is captured by aggregation on daily basis and a paired test, but it increases the variance in the case of no aggregation (because no paired test can performed here (is this even correct ?))&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All in all: Am I on the right track or do I miss something completely ?&lt;/p&gt;&#xA;" OwnerUserId="264" LastEditorUserId="1036" LastEditorDisplayName="" LastEditDate="2011-05-09T17:54:04.670" LastActivityDate="2011-05-09T17:54:04.670" Title="Aggregation-Level in AB-Tests" Tags="&lt;hypothesis-testing&gt;&lt;aggregation&gt;&lt;ab-test&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="1" />
  <row Id="4885" PostTypeId="2" ParentId="4810" CreationDate="2010-11-24T16:29:10.620" Score="7" ViewCount="" Body="&lt;p&gt;It's partly a matter of taste and convention, but theory, attention to your objectives, and a smidgen of cognitive neuroscience [see the references] can provide some guidance. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because a pdf and a cdf convey the same information, the distinction between them arises from &lt;em&gt;how&lt;/em&gt; they do it: a pdf represents probability with &lt;em&gt;areas&lt;/em&gt; while a cdf represents probability with (vertical) &lt;em&gt;distances&lt;/em&gt;.  Studies show that people compare distances faster and more accurately than they compare areas and that they systematically mis-estimate areas.  Thus, if your purpose is to provide a graphical tool for reading off probabilities, your should favor using a cdf.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Pdfs and cdfs also represent probability &lt;em&gt;density&lt;/em&gt;: the former does so by means of &lt;em&gt;height&lt;/em&gt; while the latter represents density by &lt;em&gt;slope&lt;/em&gt;.  Now the tables are turned, because people are poor estimators of slope (which is the tangent of an angle; we tend to see the angle itself).  Densities are good at conveying information about modes, heaviness of tails, and gaps.  Favor using pdfs in such situations and anywhere else where local details of the probability distribution need to be emphasized.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sometimes a pdf or cdf provides useful theoretical information.  Its value (or rather the inverse thereof) is involved in formulas for standard errors for quantiles, extremes, and rank statistics.  Display a pdf rather than a cdf in such situations.  When studying multivariate correlations in a nonparametric setting, such as with &lt;a href=&quot;http://en.wikipedia.org/wiki/Copula_%28statistics%29&quot; rel=&quot;nofollow&quot;&gt;copulas&lt;/a&gt;, the cdf turns out to be more useful (perhaps because it is the function that transforms a continuous probability law into a uniform one).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A pdf or cdf can be intimately associated with a particular statistical test.  The &lt;a href=&quot;http://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test&quot; rel=&quot;nofollow&quot;&gt;Kolmogorov-Smirnov test&lt;/a&gt; (and the KS statistic) has a simple graphical representation in terms of a vertical buffer around the cdf; it has no simple graphical representation in terms of the pdf (that I know of).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The ccdf (complementary cdf) is used in special applications that focus on survivorship and rare events.  It use tends to be used by convention.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;References&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;W.S. Cleveland (1994). The Elements of Graphing Data. Summit, NJ, USA: Hobart Press. ISBN 0-9634884-1-4&lt;/p&gt;&#xA;&#xA;&lt;p&gt;B.D. Dent (1999). Cartography: Thematic Map Design 5th Ed. Boston, MA, USA: WCB McGraw-Hill.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A.M. MacEachren (2004).  How Maps Work. New York, NY, USA: The Guilford Press.  ISBN 1-57230-040-X&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-11-25T14:30:56.387" LastActivityDate="2010-11-25T14:30:56.387" CommentCount="1" />
  <row Id="4886" PostTypeId="1" AcceptedAnswerId="4888" CreationDate="2010-11-24T17:33:01.623" Score="3" ViewCount="168" Body="&lt;p&gt;I have a data set that I need to analyze in R. A simplified version of it would be like this&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;SessionNo.  Objects       OtherColumns&#xA;    A          2               .&#xA;    A          3&#xA;    B          4&#xA;    C          1&#xA;    D          2&#xA;    D          1&#xA;    D          2&#xA;    D          3&#xA;    E          5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;here each sessionno. represents one session of a broswer but due to the relation with other columns in the data it is aggregated like shown. So, Session 1 is now fragmented into two rows etc. What I need to find is the avg. number of objects downloaded per session (or any other statistics for no. of rows in each session). So, how do I count the no of objects for each session in R. Here, 5 objects in session A, 4 in session B, 8 in session D etc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I guess one way would be to sum the whole Objects column and count the no. of unique session numbers in SessionNo. But I guess it would be more of a general solution if I could group the unique session number with the total number of objects aggregated in it? Any suggestions on how to accomplish that in R?&lt;/p&gt;&#xA;" OwnerUserId="2101" LastActivityDate="2010-11-25T13:46:52.060" Title="Getting aggregated share in R" Tags="&lt;r&gt;&lt;aggregation&gt;" AnswerCount="5" />
  <row Id="4887" PostTypeId="2" ParentId="4886" CreationDate="2010-11-24T17:45:13.697" Score="2" ViewCount="" Body="&lt;p&gt;If I am reading your question correctly, something like the following should do it:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;aggregate(x$Objects,by=list(x$SessionNo.),sum)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where &lt;code&gt;x&lt;/code&gt; is the data frame containing your data. This will give you, for each unique session number, the sum of the object counts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can of course substitute other functions (including your own) on place of the &lt;code&gt;sum&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="439" LastEditorUserId="439" LastEditorDisplayName="" LastEditDate="2010-11-25T07:42:10.487" LastActivityDate="2010-11-25T07:42:10.487" />
  <row Id="4888" PostTypeId="2" ParentId="4886" CreationDate="2010-11-24T17:48:57.147" Score="3" ViewCount="" Body="&lt;p&gt;Perhaps this might help:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;tapply(df$Objects, df$SessionNo., sum)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1050" LastEditorUserId="1050" LastEditorDisplayName="" LastEditDate="2010-11-25T10:31:03.683" LastActivityDate="2010-11-25T10:31:03.683" CommentCount="8" />
  <row Id="4889" PostTypeId="2" ParentId="4886" CreationDate="2010-11-24T17:53:47.613" Score="1" ViewCount="" Body="&lt;p&gt;I personally like using the plyr and or reshape packages for tasks like this. If you're just starting with R, I would highly recommend getting to know them well. They've solved nearly all of my data manipulation tasks.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ddply(df, .(sessionNo.) function(x) data.frame(&#xA;obj.count=sum(Objects)&#xA;))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;OR, cast&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;colnames(df[1:2]) &amp;lt;- c(&quot;variable&quot;,&quot;value&quot;)&#xA;cast(df[1:2], variable ~ value, sum)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="776" LastEditorUserId="776" LastEditorDisplayName="" LastEditDate="2010-11-25T02:08:57.187" LastActivityDate="2010-11-25T02:08:57.187" CommentCount="1" />
  <row Id="4890" PostTypeId="2" ParentId="4884" CreationDate="2010-11-24T18:09:45.540" Score="1" ViewCount="" Body="&lt;p&gt;The right level of aggregation depends on the time period over which you wish to generalize. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, you want to deploy A during nights across several sites but are unsure about its effectiveness relative to the existing option B. Thus, you may deploy A over a small number of sites and see its effects relative to the alternative B. In such a scenario, you need to aggregate the effects of A across all the nights that it was deployed to assess the relative impact of A vs B. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To use your example from the last para: If the interest lies in evaluating the impact of A across all days (possibly because A will be deployed on all days) then the 'right' thing to do is to aggregate across all days so that the test of A's effectiveness is not biased. &lt;/p&gt;&#xA;" LastActivityDate="2010-11-24T18:09:45.540" CommentCount="3" />
  <row Id="4891" PostTypeId="2" ParentId="4810" CreationDate="2010-11-24T18:41:20.210" Score="5" ViewCount="" Body="&lt;p&gt;I agree with whuber's answer, but have one additional minor point:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The CDF has a simple non-parametric estimator that needs no choices to be made: the &lt;a href=&quot;http://en.wikipedia.org/wiki/Empirical_distribution_function&quot; rel=&quot;nofollow&quot;&gt;empirical distribution function&lt;/a&gt;. It's not &lt;em&gt;quite&lt;/em&gt; so simple to estimate a PDF. If you use a histogram you need to choose the bin width and the starting point for the first bin. If you use &lt;a href=&quot;http://en.wikipedia.org/wiki/Kernel_density_estimation&quot; rel=&quot;nofollow&quot;&gt;kernel density estimation&lt;/a&gt; you need to choose the kernel shape and bandwidth. A suspicious or cynical reader may wonder if you really chose these entirely &lt;em&gt;a priori&lt;/em&gt; or if you tried a few different values and chose the ones that gave the result you most liked.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is only a minor point though. The ones whuber made are more important, so i'd probably only use this to choose when I was still undecided after considering those.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-11-24T18:41:20.210" CommentCount="1" />
  <row Id="4892" PostTypeId="1" AcceptedAnswerId="5052" CreationDate="2010-11-24T18:47:26.677" Score="3" ViewCount="220" Body="&lt;p&gt;What is the &lt;a href=&quot;http://en.wikipedia.org/wiki/VC_dimension&quot; rel=&quot;nofollow&quot;&gt;VC dimension&lt;/a&gt; of &lt;a href=&quot;http://en.wikipedia.org/wiki/Support_vector_machine&quot; rel=&quot;nofollow&quot;&gt;SVM&lt;/a&gt; with the polynomial kernel $k(x,x&amp;#39;)=(1+&amp;lt;x,x&amp;#39;&amp;gt;_{\mathbb{R^{2}}})^{2}$ for binary classification in $\mathbb{R^{2}}$?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It would be equal or more than v iff there exists a set of v points such that, any labeling (-1 or +1) of the points given, there exists a correct separating border.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It would be strictly less than v iff for all sets of v points, there exists a labeling of the points such that there is no correct separating border.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this case, the separating border is a conic section or a line, so any idea based on these curves rather than SVM is welcome.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strike&gt;For instance, let $x=(x_{1},x_{2}) \in \mathbb{R^{2}}$. It is mapped to $\mathbb{R^{6}}$ using a certain $\phi$ and an hyperplane in this new space is a conic section in the former space. So my guess would be the VC dimension of a linear classifier in $\mathbb{R^{6}}$, which is 7.&lt;/strike&gt;&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Actually, &lt;strong&gt;the answer is 6&lt;/strong&gt;: the equation of an hyperplane in $\mathbb{R^{d}}$ is $$&amp;lt;w,x&amp;gt;_{\mathbb{R^{d}}}+b=0$$ where $w \in \mathbb{R^{d}}$ and $b \in \mathbb{R}$. Here, we have $b=1$ and $w \in \mathbb{R^{5}}$, not $\mathbb{R^{6}}$.&lt;/p&gt;&#xA;" OwnerUserId="1351" LastEditorUserId="1351" LastEditorDisplayName="" LastEditDate="2010-12-01T13:12:35.813" LastActivityDate="2010-12-01T13:12:35.813" Title="VC dimension of SVM with polynomial kernel in $\mathbb{R^{2}}$" Tags="&lt;classification&gt;&lt;homework&gt;&lt;kernel&gt;&lt;svm&gt;&lt;binary-data&gt;" AnswerCount="2" />
  <row Id="4893" PostTypeId="2" ParentId="4892" CreationDate="2010-11-24T20:21:22.247" Score="3" ViewCount="" Body="&lt;p&gt;If I'm not mistaken:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$[1 + &amp;lt;x,y&amp;gt;]^2$ = $&amp;lt;[1, \sqrt{2}y_1, y_1^2, \sqrt{2}y_2, y_2^2, \sqrt{2}y_1y_2], [1, \sqrt{2}x_1, x_1^2, \sqrt{2}x_2, x_2^2, \sqrt{2}x_1x_2]^T&amp;gt;$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the separating rule is linear in $R^6$. Consequently, VC-dim is 7.&lt;/p&gt;&#xA;" OwnerUserId="1725" LastEditorUserId="1725" LastEditorDisplayName="" LastEditDate="2010-11-24T20:35:21.060" LastActivityDate="2010-11-24T20:35:21.060" CommentCount="2" />
  <row Id="4894" PostTypeId="2" ParentId="4858" CreationDate="2010-11-24T20:48:13.937" Score="5" ViewCount="" Body="&lt;p&gt;(I would post this as a comment to the previous answer by 'chi', but don't see that that is possible here.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Be very careful with &lt;code&gt;intervals()&lt;/code&gt; and &lt;code&gt;anova()&lt;/code&gt; from the &lt;code&gt;nlme&lt;/code&gt; package, these have exactly the flaws that @fabians points out above -- they rely on the standard chi-squared distribution applied to a quadratic approximation of the shape of the likelihood profile for the variances. The newer &lt;code&gt;lme4a&lt;/code&gt; package (on r-forge) allows the creation of likelihood profiles for the variance parameters, which takes care of the quadratic approximation part (although not the distributional part).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, it is fairly hotly debated whether dropping non-significant variance components is a good idea or not (don't have an immediate reference, but this has been discussed on r-sig-mixed-models).&lt;/p&gt;&#xA;" OwnerUserId="2126" LastActivityDate="2010-11-24T20:48:13.937" CommentCount="1" />
  <row Id="4895" PostTypeId="2" ParentId="118" CreationDate="2010-11-24T20:49:39.210" Score="4" ViewCount="" Body="&lt;p&gt;The answer that best satisfied me is that it falls out naturally from the generalization of a sample to n-dimensional euclidean space. It's certainly debatable whether that's something that should be done, but in any case:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assume your $n$ measurements $X_i$ are each an axis in $\mathbb R^n$. Then your data $x_i$ define a point $\bf x$ in that space. Now you might notice that the data are all very similar to each other, so you can represent them with a single location parameter $\mu$ that is constrained to lie on the line defined by $X_i=\mu$. Projecting your datapoint onto this line gets you $\hat\mu=\bar x$, and the distance from the projected point $\hat\mu\bf 1$ to the actual datapoint is $\sqrt{\frac{n-1} n}\hat\sigma=\|\bf x-\hat\mu\bf 1\|$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This approach also gets you a geometric interpretation for correlation, $\hat\rho=\cos \angle(\vec{\bf\tilde x},\vec{\bf\tilde y})$.&lt;/p&gt;&#xA;" OwnerUserId="2456" LastActivityDate="2010-11-24T20:49:39.210" CommentCount="4" />
  <row Id="4896" PostTypeId="2" ParentId="4886" CreationDate="2010-11-25T00:00:21.353" Score="2" ViewCount="" Body="&lt;p&gt;The function that is specifically designed for this task is ave(). By default it returned the mean within a group and returns a vector of the same length as the two input arguments. It is designed to fill in columns of either means or deviations from means. If this is in a dataframe with name &quot;tst&quot;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; tst$tmn &amp;lt;- with(tst, ave(Objects, SessionNo.))&#xA;&amp;gt; tst$devmn &amp;lt;- tst$Objects- with(tst, ave(Objects, SessionNo.))&#xA;&amp;gt; tst&#xA;  SessionNo. Objects tmn devmn&#xA;1          A       2 2.5  -0.5&#xA;2          A       3 2.5   0.5&#xA;3          B       4 4.0   0.0&#xA;4          C       1 1.0   0.0&#xA;5          D       2 2.0   0.0&#xA;6          D       1 2.0  -1.0&#xA;7          D       2 2.0   0.0&#xA;8          D       3 2.0   1.0&#xA;9          E       5 5.0   0.0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2129" LastActivityDate="2010-11-25T00:00:21.353" CommentCount="1" />
  <row Id="4897" PostTypeId="2" ParentId="4663" CreationDate="2010-11-25T00:29:06.963" Score="2" ViewCount="" Body="&lt;p&gt;LASSO is not an algorithm per se, but an operator. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are many different ways to derive efficient algorithms for $\ell_1$ regularized problems. For instance, one can use quadratic programming to them tackle directly. I guess this is what you refer to as LASSO. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another one is LARS, very popular because of its simplicity, connection with forward procedures (yet not too greedy), very constructive proof and easy generalization. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even compared with state of the art quadratic programming solvers, LARS can be much more efficient.&lt;/p&gt;&#xA;" OwnerUserId="358" LastActivityDate="2010-11-25T00:29:06.963" />
  <row Id="4898" PostTypeId="1" AcceptedAnswerId="4917" CreationDate="2010-11-25T01:33:26.563" Score="6" ViewCount="454" Body="&lt;p&gt;If you choose to analyse a pre-post treatment-control design with a continuous dependent variable using a mixed ANOVA, there are various ways of quantifying the effect of being in the treatment group.&#xA;The interaction effect is one main option.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, I particularly like Cohen's d type measures (i.e., ${\frac{\mu_1 - \mu_2}{\sigma}}$). I don't like variance explained measures because results vary based on irrelevant factors such as relative sample sizes of groups.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, I was thinking I could quantify the effect as follows&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;$\Delta\mu_c = \mu_{c2} - \mu_{c1}$&#xD;&lt;/li&gt;&#xA;&lt;li&gt;$\Delta\mu_t = \mu_{t2} - \mu_{t1}$&#xD;&lt;/li&gt;&#xA;&lt;li&gt;Thus, the effect size could be defined as $\frac{\Delta\mu_t - \Delta\mu_c}{\sigma}$&#xD;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;where $c$ refers to control, $t$ to treatment, and 1 and 2 to pre and post respectively.&#xA;$\sigma$ could be the pooled standard deviation at time 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Questions:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Is it appropriate to label this effect size measure &lt;code&gt;d&lt;/code&gt;?&lt;/li&gt;&#xA;&lt;li&gt;Does this approach seem reasonable?&lt;/li&gt;&#xA;&lt;li&gt;What is standard practice for effect size measures for such designs?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastActivityDate="2011-04-19T02:15:25.713" Title="Effect size for interaction effect in pre-post treatment-control design" Tags="&lt;anova&gt;&lt;mixed-model&gt;&lt;effect-size&gt;&lt;cohens-d&gt;" AnswerCount="3" />
  <row Id="4899" PostTypeId="2" ParentId="118" CreationDate="2010-11-25T03:01:40.620" Score="1" ViewCount="" Body="&lt;p&gt;Estimating the standard deviation of a distribution requires to choose a distance.&lt;br&gt;&#xA;Any of the following distance can be used:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$d_{n}((X)_{i=1..I},\mu)=(\sum \left | X-\mu \right |^n)^{1/n}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We usually use the natural euclidean distance (n=2), which is the one everybody uses in daily life.&#xA;The distance that you propose is the one with $n=1$.&lt;br&gt;&#xA;Both are good candidates but they are different.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One could decide to use $n=3$ as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not sure that you will like my answer, my point contrary to others is not to demonstrate that $n=2$ is better. I think that if you want to estimate the standard deviation of a distribution, you can absolutely use a different distance.&lt;/p&gt;&#xA;" OwnerUserId="1709" LastEditorUserId="1709" LastEditorDisplayName="" LastEditDate="2011-01-05T03:41:08.693" LastActivityDate="2011-01-05T03:41:08.693" CommentCount="2" />
  <row Id="4900" PostTypeId="2" ParentId="4898" CreationDate="2010-11-25T03:49:17.473" Score="3" ViewCount="" Body="&lt;p&gt;I believe that generalized eta-square (&lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pubmed/14664681&quot; rel=&quot;nofollow&quot;&gt;Olejnik &amp;amp; Algena, 2003&lt;/a&gt;; &lt;a href=&quot;http://brm.psychonomic-journals.org/content/37/3/379&quot; rel=&quot;nofollow&quot;&gt;Bakeman, 2005&lt;/a&gt;) provides a reasonable solution to the quantification of effect size that generalizes across between-Ss and within-Ss designs. If I read those references correctly, generalized eta-square should also generalize across sample sizes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Generalized eta-square is automatically computed by the ezANOVA() function in the ez package for R.&lt;/p&gt;&#xA;" OwnerUserId="364" LastActivityDate="2010-11-25T03:49:17.473" CommentCount="1" />
  <row Id="4901" PostTypeId="1" AcceptedAnswerId="4940" CreationDate="2010-11-25T05:32:53.903" Score="4" ViewCount="418" Body="&lt;p&gt;Other than literally testing each possible combination of variable(s) in a model (x1:x2 or x1*x2 ... xn-1 * xn). How do you identify if an interaction SHOULD or COULD exist between your independent (hopefully) variables? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are best practices in attempting to identify interactions?&#xA;Is there a graphical technique that you could or do use?&lt;/p&gt;&#xA;" OwnerUserId="776" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-26T15:00:30.057" LastActivityDate="2010-11-27T03:38:45.647" Title="Identifying Interaction Effects" Tags="&lt;regression&gt;&lt;modeling&gt;&lt;interaction&gt;" AnswerCount="5" CommentCount="3" FavoriteCount="1" />
  <row Id="4902" PostTypeId="2" ParentId="4901" CreationDate="2010-11-25T07:07:59.057" Score="1" ViewCount="" Body="&lt;p&gt;How large is $n$ ? how many observations do you have ? this is crutial ... &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Sobol indices&lt;/strong&gt; will tell you the &lt;strong&gt;proportion of variance explained by interaction&lt;/strong&gt; if you have a lot of observations and a few $n$, otherwise you will have to do modelling (linear to start with). You have a nice R package for that called sensitivity. Anyway the idea is quite often that of decomposing the variance (also called generalized ANOVA). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to know if this proportion of variance is significant you will have to do modelling (roughly, you need to know the number of degree of freedom of your model to compare it to the variance). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are your variables discrete or continuous ? bounded or not really (i.e you don't know the maximum) ? &lt;/p&gt;&#xA;" OwnerUserId="223" LastActivityDate="2010-11-25T07:07:59.057" CommentCount="1" />
  <row Id="4903" PostTypeId="2" ParentId="4901" CreationDate="2010-11-25T07:50:59.160" Score="6" ViewCount="" Body="&lt;p&gt;My best practice would be to &lt;strong&gt;think&lt;/strong&gt; about the problem to hand &lt;em&gt;before&lt;/em&gt; fitting the model. What is a plausible model given the phenomenon you are studying? Fitting all possible combinations of variables and interactions sounds like data dredging to me.&lt;/p&gt;&#xA;" OwnerUserId="1390" LastActivityDate="2010-11-25T07:50:59.160" CommentCount="10" />
  <row Id="4904" PostTypeId="1" CreationDate="2010-11-25T11:16:41.540" Score="5" ViewCount="184" Body="&lt;p&gt;I'm searching for a statistical method to determine if a player is cheating in an online game. The game is a Quake3 like game (ego-shooter).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given a number of positive points and a number of negative points per player (score) and given n players (n&amp;lt;=64). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The score comes together like this (positive/negative seen from anti-cheat-perspective):&lt;br /&gt;&#xA;positive = number of time the player died himself&lt;br /&gt;&#xA;negative = number of opposite team's players killed&lt;br /&gt;&#xA;&lt;br /&gt;&lt;br /&gt;&#xA;Additional available values: &lt;br /&gt;&#xA;t = Time in which this score has been accomplished&lt;br /&gt;&#xA;c = number of current number of players&lt;br /&gt;&#xA;m = number of times a player killed someone from the same team (teamkiller, very negative if higher than 1 or 2 over a 15 minutes period)&lt;br /&gt;&#xA;&lt;br /&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each player can cheat independent of each other. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I used the standard deviation (on the value v = (negative+1)/(postive+1) per player, assigning the exceeding of 3 times standard deviation with absolute certainty (100%) of cheating - although my experience has been that exceeding stddev alone would be sufficient in most cases), which works great on a single player but fails miserably as soon as several players are cheating at the same time.  Also the practise of defining 3 times stddev as 100% allows for &gt; 100% probabilities. Unfortunately, the percentage of players cheating in an unfiltered game is roughly 70 to 80 percent or even higher, if accounting for more subtle cheats.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One more thing to consider is, if there is a group of people cheating, and one that isn't, the distribution is no longer a normal distribution, but a camel-bump like distribution.&#xA;Is there any useful algorithm or formula for this problem ?&lt;/p&gt;&#xA;" OwnerUserId="2132" LastEditorUserId="2132" LastEditorDisplayName="" LastEditDate="2010-11-25T11:21:15.593" LastActivityDate="2010-12-03T04:58:22.370" Title="Method to reliably determine abnormal statistical values" Tags="&lt;outliers&gt;" AnswerCount="2" CommentCount="3" FavoriteCount="1" />
  <row Id="4905" PostTypeId="2" ParentId="4904" CreationDate="2010-11-25T12:01:24.780" Score="8" ViewCount="" Body="&lt;p&gt;Your use of the stddev indicates you look at every variable seperately. If you look at them together, you might have more chance. An outlier in one dimension can be coincidence, an outlier in more dimensions is more surely an anomaly. I don't know much about games, but I reckon that you could use find extra variables like traveled distance in the game and so on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can use outlier theory for detecting anomalies. A very naive way of looking for outliers is using the &lt;a href=&quot;http://en.wikipedia.org/wiki/Mahalanobis_distance&quot; rel=&quot;nofollow&quot;&gt;mahalanobis distance&lt;/a&gt;. This is a measure that takes into account the spread of your data, and calculates the relative distance from the center. It will be less sensitive to outliers in one statistic, but can be seen as a way of finding gamers where the combination of the statistics is odd.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A similar approach is building a model and taking a look at the error terms. This does essentially the same: it looks for gamers that don't fit the general pattern. It's a technique that's also used in financial services to find fraud cases. The model can go from a basic linear model to more complex models. If you apply your algorithm to the error terms of the model fitted without player i, you essentially calculate something similar to &lt;a href=&quot;http://en.wikipedia.org/wiki/Cook%27s_distance&quot; rel=&quot;nofollow&quot;&gt;Cook's distance&lt;/a&gt; of a certain player. In combination with the &lt;a href=&quot;http://en.wikipedia.org/wiki/DFFITS&quot; rel=&quot;nofollow&quot;&gt;DFFITS&lt;/a&gt; measure and the leverage, it is often used for detection of outliers and/or influential points in regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could as well use supervised classification: You train an algorithm with genuine gamers and known cheaters. There's a multitude of techniques available there, starting from neural networks and classification trees to support vector machines and random forests.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Genetic algorithms are used more and more as well, as they can progress in knowledge when time goes by. If you check the assumed cheaters, you could -much like a spam filter- correct the wrongly classified gamers. The algorithm will continuously learn to predict better when a gamer is a cheater. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As mbq mentioned, without example data it's impossible to give you an algorithm; I don't even know on which measurements one can work. But this should give you some ideas about the multitude of available methods, from very naive to pretty complex. Much can be learned from fraud detection if you'd like to google around a bit further.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A starter could be &lt;a href=&quot;http://home.cc.gatech.edu/ccl/uploads/45/multiple-algorithms-for-fraud.pdf&quot; rel=&quot;nofollow&quot;&gt;this article&lt;/a&gt; of Wheeler and Aitken. Another interesting overview of possible techniques is found in &lt;a href=&quot;http://linktoprofessionals.com/files/data/Analysis,%20reports/Survey%20of%20Fraud%20Detection%20Techniques.pdf&quot; rel=&quot;nofollow&quot;&gt;this article&lt;/a&gt; by Kou et al (&lt;a href=&quot;http://ieeexplore.ieee.org/Xplore/defdeny.jsp?url=http%3A%2F%2Fieeexplore.ieee.org%2Fstamp%2Fstamp.jsp%3Ftp%3D%26arnumber%3D1297040%26userType%3Dinst&amp;amp;denyReason=-133&amp;amp;arnumber=1297040&amp;amp;productsMatched=null&amp;amp;userType=inst&quot; rel=&quot;nofollow&quot;&gt;alternative link to publication&lt;/a&gt;)&lt;/p&gt;&#xA;" OwnerUserId="1124" LastEditorUserId="1124" LastEditorDisplayName="" LastEditDate="2010-11-25T12:46:04.647" LastActivityDate="2010-11-25T12:46:04.647" />
  <row Id="4906" PostTypeId="2" ParentId="4886" CreationDate="2010-11-25T13:46:52.060" Score="1" ViewCount="" Body="&lt;p&gt;I often use &lt;code&gt;by&lt;/code&gt;, which is a wrapper around &lt;code&gt;tapply&lt;/code&gt;. Result is a list with a weee different print method compared to &lt;code&gt;tapply&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df &amp;lt;- data.frame(SessionNo. = sample(x = c(&quot;A&quot;, &quot;B&quot;, &quot;C&quot;, &quot;D&quot;, &quot;E&quot;),&#xA;                size = 20, replace = TRUE,&#xA;                prob = c(0.05, 0.1, 0.4, 0.4, 0.05)),&#xA;        Objects = sample(x = 1:5, size = 20, replace = TRUE))&#xA;&#xA;df.out &amp;lt;- by(data = df$Objects, INDICES = df$SessionNo.,&#xA;        FUN = mean, simplify = FALSE)&#xA;head(df.out)&#xA;$A&#xA;[1] 3&#xA;&#xA;$B&#xA;[1] 1&#xA;&#xA;$C&#xA;[1] 3.222222&#xA;&#xA;$D&#xA;[1] 3.833333&#xA;&#xA;$E&#xA;[1] 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="144" LastActivityDate="2010-11-25T13:46:52.060" CommentCount="2" />
  <row Id="4907" PostTypeId="2" ParentId="4901" CreationDate="2010-11-25T13:51:42.120" Score="5" ViewCount="" Body="&lt;p&gt;Fitting a tree model (i.e. using R), will help you identify complex interactions between the explanatory variables. Read the example on page 30 &lt;a href=&quot;http://www.google.gr/url?sa=t&amp;amp;source=web&amp;amp;cd=6&amp;amp;ved=0CEEQFjAF&amp;amp;url=http%3A%2F%2Fwww3.imperial.ac.uk%2Fpls%2Fportallive%2Fdocs%2F1%2F1171924.PDF&amp;amp;rct=j&amp;amp;q=fit%20a%20tree%20model%20to%20see%20whether%20complex%20interactions%20between%20the%20explanatory%20variables%20are%20indicated&amp;amp;ei=kmfuTK7tK9eAhAfa7tzvCw&amp;amp;usg=AFQjCNGLtFOjUeikDVtZiDUBhOV5X1Zklg&amp;amp;sig2=L_z9Y9tydlZx55kUtXrfPQ&amp;amp;cad=rja&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="339" LastActivityDate="2010-11-25T13:51:42.120" CommentCount="3" />
  <row Id="4908" PostTypeId="1" CreationDate="2010-11-25T13:55:10.873" Score="5" ViewCount="216" Body="&lt;p&gt;Hi all!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lets imagine that we have some experiments. Each experiment may result in one of the outcomes: A, B, C. So we have probabilities distribution for each experiment $P_A, P_B, P_C$ which is context-dependent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;E.g.:  &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;$Context_1 \Rightarrow \{P_A^1, P_B^1, P_C^1\},$ experimental outcome is A&lt;/li&gt;&#xA;&lt;li&gt;$Context_2 \Rightarrow \{P_A^2, P_B^2, P_C^2\},$ experimental outcome is B&lt;/li&gt;&#xA;&lt;li&gt;$Context_3 \Rightarrow \{P_A^3, P_B^3, P_C^3\},$ experimental outcome is A&lt;/li&gt;&#xA;&lt;li&gt;$Context_4 \Rightarrow \{P_A^4, P_B^4, P_C^4\},$ experimental outcome is C&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;This probabilities are calculated by some function $F:Context\rightarrow \{P_A, P_B, P_C\}$  &#xA;I want to estimate an absolute trust rate of this function. In other words, I want to be able to say &quot;we can trust this function on 86%&quot; like we do when we deal with Pearson's chi-square test.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestions?&lt;/p&gt;&#xA;" OwnerUserId="1314" LastEditorUserId="1314" LastEditorDisplayName="" LastEditDate="2010-11-26T08:27:35.030" LastActivityDate="2010-11-28T23:09:43.057" Title="Chi-square analog for context-dependent distributions" Tags="&lt;distributions&gt;&lt;probability&gt;&lt;chi-squared&gt;&lt;predictor&gt;" AnswerCount="2" CommentCount="5" />
  <row Id="4909" PostTypeId="1" CreationDate="2010-11-25T14:13:31.967" Score="1" ViewCount="132" Body="&lt;p&gt;The empirical cumulative distribution function of a random variable, given observations $x_\left( k \right) &amp;gt; x_\left( k-1 \right)$, $k \in \mathbb N$, $k \le n$, is defined as $F_{emp}(x_\left( k \right) &amp;gt; X \ge x_\left( k-1 \right)) = \frac k {n+1}$ and $F_{emp}(X \ge x_\left(n\right))=1$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why? As long as we're interpolating, wouldn't it make sense to use some interpolation method with less error? A simple nearest neighbour or piecewise average interpolant would be an improvement, and a cubic interpolant would get us a differentiable empirical density function, too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The above definition won't even give you the piecewise infimum of the cdf, because the variable is random. It certainly approaches the true function as $n\to\infty$, but then so would any other interpolant. Surely at least linear interpolants were considered.&lt;/p&gt;&#xA;" OwnerUserId="2456" LastEditorUserId="2456" LastEditorDisplayName="" LastEditDate="2010-11-25T14:28:40.100" LastActivityDate="2011-05-13T23:07:29.717" Title="Interpolating the empirical cumulative function" Tags="&lt;estimation&gt;&lt;random-variable&gt;&lt;interpolation&gt;&lt;empirical&gt;" AnswerCount="2" />
  <row Id="4910" PostTypeId="2" ParentId="4909" CreationDate="2010-11-25T14:34:31.573" Score="2" ViewCount="" Body="&lt;p&gt;The EDF is the CDF of the population constituted by the data themselves.  This is &lt;em&gt;exactly&lt;/em&gt; what you need to describe and analyze any resampling process from the dataset, including nonparametric bootstrapping, jackknifing, cross-validation, etc.  Not only that, it's perfectly general: any kind of interpolation would be invalid for discrete distributions.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-11-25T14:34:31.573" CommentCount="2" />
  <row Id="4911" PostTypeId="2" ParentId="4360" CreationDate="2010-11-25T15:18:15.367" Score="2" ViewCount="" Body="&lt;p&gt;For the Bayesian explanation, you need a prior probability distribution on the reported results by a lying coin flipper, as well as a prior probability of lying. When you see a value that is much more likely under the lying distribution than the random flips one, that makes your posterior probability of lying much higher.&lt;/p&gt;&#xA;" OwnerUserId="2134" LastActivityDate="2010-11-25T15:18:15.367" />
  <row Id="4912" PostTypeId="1" AcceptedAnswerId="4948" CreationDate="2010-11-25T16:32:33.287" Score="2" ViewCount="395" Body="&lt;p&gt;Suppose my sample comprises 350 instances. Each instance has two independent binary (&lt;code&gt;pos&lt;/code&gt; and &lt;code&gt;neg&lt;/code&gt;) evaluations, &lt;strong&gt;A&lt;/strong&gt; and &lt;strong&gt;B&lt;/strong&gt;. If my alternative hypothesis is that A is more often positive than B, how would I express whether the difference is statistically significant?&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;To be more precise on the data: This question came to mind when looking at patient blood serum data. These are 350 blood samples positive against &lt;strong&gt;X&lt;/strong&gt;. Each sample may or may not also be positive against A or B, whereas A and B are both more often positive if X is positive, but biologically are not dependent on one another.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my initial thought was to look whether A is more often positive than B. In the example it was, A is positive in ~80% and B in ~20%. So I thought about assessing this difference for significance and this is where I got stuck. :)&lt;/p&gt;&#xA;" OwnerUserId="1785" LastEditorUserId="1785" LastEditorDisplayName="" LastEditDate="2010-11-25T20:28:48.560" LastActivityDate="2010-11-28T19:19:27.373" Title="Calculating significance of difference between two binary values on one dataset" Tags="&lt;hypothesis-testing&gt;&lt;statistical-significance&gt;&lt;sample&gt;&lt;binary-data&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="4913" PostTypeId="2" ParentId="4912" CreationDate="2010-11-25T17:16:36.760" Score="3" ViewCount="" Body="&lt;p&gt;Your addendum suggests that A and B are dependent samples since they come from the same &quot;instance&quot;, i.e., patient. In that case, I propose the McNemar-Test which tests for a (two-sided) hypothesis of unequal proportions:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; N    &amp;lt;- 350&#xA;&amp;gt; A    &amp;lt;- factor(rbinom(N, size=1, p=0.6), labels=c(&quot;pos&quot;, &quot;neg&quot;))&#xA;&amp;gt; B    &amp;lt;- factor(rbinom(N, size=1, p=0.4), labels=c(&quot;pos&quot;, &quot;neg&quot;))&#xA;&amp;gt; cTab &amp;lt;- table(A, B)&#xA;&amp;gt; mcnemar.test(cTab, correct=FALSE)   # no continuity correction&#xA;        McNemar's Chi-squared test&#xA;data:  cTab &#xA;McNemar's chi-squared = 29.1236, df = 1, p-value = 6.79e-08&#xA;&#xA;# alternative - Monte-Carlo permutation test&#xA;&amp;gt; library(coin)          # for symmetry_test()&#xA;&amp;gt; symmetry_test(cTab, teststat=&quot;quad&quot;, distribution=approximate(B=9999))&#xA;        Approximative General Independence Test&#xA;data:  response by groups (A, B) &#xA;         stratified by block &#xA;chi-squared = 29.1236, p-value &amp;lt; 2.2e-16&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1909" LastEditorUserId="1909" LastEditorDisplayName="" LastEditDate="2010-11-25T20:10:00.573" LastActivityDate="2010-11-25T20:10:00.573" CommentCount="4" />
  <row Id="4914" PostTypeId="1" CreationDate="2010-11-25T17:26:03.753" Score="3" ViewCount="203" Body="&lt;p&gt;In a data set with thousands of data points, I am testing different short-term and longer term data outputs based on 5 rolling data points all the way to 100 rolling data points (which each value being a separate column in excel: 5, 6,..., 100). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The test I developed (with rudimentary knowledge of the whole thing) is to check which are the better &quot;fits&quot; of these rolling data outputs (kinda like moving averages, but not quite), in other words, if column 5, 17 or 98. In theory, the better fits should produce lower standard deviations, in the conventional calculation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;An analogy for better understanding is something like this: if I do a 5 point moving average on a five period sin wave, the output should be zero or flat, the same for a 6 point moving average on a six period sin wave, and so on all the way to 100. This is the same as saying that the standard deviation in each case should be zero. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the test really is choosing the lowest standard deviation in each column from 5 all the way to 100, which should then provide the best &quot;fit&quot; (to go back to the analogy, fit with the &quot;sin wave&quot;). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, as the statistical inclined will quickly note (am afraid am not one of them for sure), this method presents two major difficulties:  &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The longer data sets, ie closer to 100 rolling data points, have larger absolute deviations, thus producing higher standard deviations. So a lower standard deviation in the shorter data series may not be indicative of better &quot;fitting&quot;, just lower absolute deviations;   &lt;/li&gt;&#xA;&lt;li&gt;Each column has a different number of rolling data points, so once again the numbers and statistical fit may not be comparable between the columns. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;So my question is therefore: is there a statistical measure better than the standard deviation I used above, such that it is not sensitive to absolute deviations and number of data points? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="2137" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-11-25T22:44:35.630" LastActivityDate="2010-11-27T18:30:58.310" Title="Developing a statistical test to ascertain better &quot;fit&quot;" Tags="&lt;time-series&gt;&lt;standard-deviation&gt;&lt;goodness-of-fit&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="1" />
  <row Id="4915" PostTypeId="2" ParentId="4914" CreationDate="2010-11-25T19:49:13.047" Score="2" ViewCount="" Body="&lt;p&gt;Based on the information given, I think that you could consider AIC, a measure of likelihood that is penalized by degrees of freedom.&lt;/p&gt;&#xA;" OwnerUserId="1381" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-25T21:41:36.517" LastActivityDate="2010-11-25T21:41:36.517" CommentCount="3" />
  <row Id="4916" PostTypeId="2" ParentId="4360" CreationDate="2010-11-25T20:33:30.693" Score="4" ViewCount="" Body="&lt;p&gt;I think your intuition is flawed. It seems you are implicitly comparing a single, &quot;very special&quot; result (exactly 10000 heads) with a set of many results (all &quot;non-special&quot; numbers of heads close to 10000). However, the definition of &quot;special&quot; is an arbitrary choice based on our psychology. How about binary 10000000000000 (decimal 8192) or Hex ABC (decimal 2748) - would that be suspiciously special as well? As Joris Meys commented, the Bayes argument would essentially be the same for any single number of heads, implying that each outcome would be suspicious.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To expand the argument a bit: you want to test a hypothesis (&quot;the experimenter is faking&quot;), and then you choose a test statistic (number of heads). Now, is this test statistic suited to tell you something about your hypothesis? To me, it seems the chosen test statistic is not informative (not a function of a parameter specified as a fixed value in the hypothesis). This goes back to the question what you mean by &quot;cheating&quot;. If that means the experimenter controls the coin at will, then this is not reflected in the test statistic. I think you need to be more precise to find a quantifiable indicator, and thus make the question amenable to a statistical test.&lt;/p&gt;&#xA;" OwnerUserId="1909" LastEditorUserId="1909" LastEditorDisplayName="" LastEditDate="2010-11-25T23:48:14.830" LastActivityDate="2010-11-25T23:48:14.830" CommentCount="2" />
  <row Id="4917" PostTypeId="2" ParentId="4898" CreationDate="2010-11-25T23:37:14.657" Score="5" ViewCount="" Body="&lt;p&gt;Yes, what you are suggesting is exactly what has been suggested in the literature. See, for example: Morris, S. B. (2008). Estimating effect sizes from pretest-posttest-control group designs. Organizational Research Methods, 11(2), 364-386 (&lt;a href=&quot;http://orm.sagepub.com/cgi/content/abstract/11/2/364&quot; rel=&quot;nofollow&quot;&gt;link&lt;/a&gt;, but unfortunately, no free access). The article also describes different methods for estimating this effect size measure. You can use the letter &quot;d&quot; to denote the effect size, but you should definitely provide an explanation of what you calculated (otherwise, readers will probably assume that you calculated the standardized mean difference only for the post-test scores).&lt;/p&gt;&#xA;" OwnerUserId="1934" LastActivityDate="2010-11-25T23:37:14.657" CommentCount="1" />
  <row Id="4918" PostTypeId="1" CreationDate="2010-11-26T00:46:30.300" Score="-3" ViewCount="141" Body="&lt;p&gt;The input is a bunch of numbers. How can I calculate a threshold that there are 10% of the totally amount of numbers are above this threshold. I think this is equally as to calculate the distribution function, but what I need is the threshold number, not the &lt;/p&gt;&#xA;" OwnerUserId="2141" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-27T11:49:30.547" LastActivityDate="2010-11-27T11:49:30.547" ClosedDate="2010-11-30T16:38:57.343" Title="How to get the required data?" Tags="&lt;distributions&gt;" AnswerCount="2" CommentCount="4" FavoriteCount="0" />
  <row Id="4919" PostTypeId="2" ParentId="4918" CreationDate="2010-11-26T00:51:35.250" Score="0" ViewCount="" Body="&lt;p&gt;My understanding of the question is that given a series of numbers, which is &lt;em&gt;the number&lt;/em&gt; that has 10% of all the given numbers larger than &lt;em&gt;this number&lt;/em&gt; and 90% smaller? If this understanding is correct, a software percentile function &lt;a href=&quot;http://en.wikipedia.org/wiki/Percentile&quot; rel=&quot;nofollow&quot;&gt;(see here)&lt;/a&gt; or a decile function &lt;a href=&quot;http://en.wikipedia.org/wiki/Decile&quot; rel=&quot;nofollow&quot;&gt;(see here)&lt;/a&gt; might be what is needed.&lt;/p&gt;&#xA;" OwnerUserId="226" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-27T11:47:55.777" LastActivityDate="2010-11-27T11:47:55.777" CommentCount="3" />
  <row Id="4920" PostTypeId="1" AcceptedAnswerId="4926" CreationDate="2010-11-26T00:58:39.783" Score="7" ViewCount="391" Body="&lt;p&gt;Using Pearson's Correlation Coefficient, I have several variables that are highly correlated ($\rho = 0.978$ and $\rho = 0.989$ for 2 pairs of variables that are in my model).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;em&gt;reason&lt;/em&gt; some of the variables are highly correlated is because one variable is used in the &lt;em&gt;calculation&lt;/em&gt; for another variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$B = V / 3000$ and&#xA;$E = V * D$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$B$ and $E$ have $\rho = 0.989$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possible for me to just &quot;throw away&quot; one of the variables?&lt;/p&gt;&#xA;" OwnerUserId="1894" LastEditorUserId="1894" LastEditorDisplayName="" LastEditDate="2010-11-26T01:03:50.200" LastActivityDate="2010-11-26T19:16:07.317" Title="Can I simply remove one of two predictor variables that are highly linearly correlated?" Tags="&lt;regression&gt;&lt;correlation&gt;&lt;modeling&gt;" AnswerCount="4" FavoriteCount="1" />
  <row Id="4921" PostTypeId="2" ParentId="4920" CreationDate="2010-11-26T01:17:49.297" Score="0" ViewCount="" Body="&lt;p&gt;If D is not a constant, then B and E are effectively two different variables because of the variations in D. The high correlation indicates that D is practically constant throughout the training data. If that is the case, then you can discard either B or E.&lt;/p&gt;&#xA;" OwnerUserId="2071" LastActivityDate="2010-11-26T01:17:49.297" CommentCount="2" />
  <row Id="4923" PostTypeId="2" ParentId="4920" CreationDate="2010-11-26T03:36:26.110" Score="3" ViewCount="" Body="&lt;p&gt;B is a linear transform of V. E represents an interaction between V and D.  Have you considered specifying a model that is Y = Intercept + V + D + V:D?  As @euphoria83 suggests, it seems likely that there is little variation in D, so it may not solve your problem; however it should at least make the independent contributions of V and D clear.  Be sure to center both V and D beforehand.&lt;/p&gt;&#xA;" OwnerUserId="196" LastEditorUserId="196" LastEditorDisplayName="" LastEditDate="2010-11-26T19:16:07.317" LastActivityDate="2010-11-26T19:16:07.317" CommentCount="1" />
  <row Id="4924" PostTypeId="2" ParentId="4360" CreationDate="2010-11-26T04:13:38.217" Score="4" ViewCount="" Body="&lt;p&gt;The conclusion you draw will be VERY dependent on the prior you choose for the probability of cheating and the prior probability that, given the flipper is lying, x heads are reported.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Putting the most mass on P(10000 heads reported|lying) is a little counter intuitive in my opinion. Unless the reporter is naive, I can't imagine anyone reporting that kind of falsified data (largely for the reasons you mentioned in the original post; it's too suspicious to most people.) If the coin really is unfair and the flipper were to report falsified data, then I think a more reasonable (and very approximate) prior on the reported results might be a discrete uniform prior P(X heads reported|lying) = 1/201 for the integers {9900, ..., 10100} and P(x heads reported|lying) = 0 for all other x. Suppose that you think the prior probability of lying is 0.5. Then some posterior probabilities are:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P(lying|9900 heads reported) = P(lying|10100 heads reported) = 0.70;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P(lying|9950 heads reported) = P(lying|10050 heads reported) = 0.54;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P(lying|10000 heads reported) = 0.47.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most reasonable numbers of reported heads from a fair coin will result in suspicion. Just to show how sensitive the posterior probabilities are to your priors, if the prior probability of cheating is lowered to 0.10, then the posterior probabilities become:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P(lying|9900 heads reported) = P(lying|10100 heads reported) = 0.21;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P(lying|9950 heads reported) = P(lying|10050 heads reported) = 0.11;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P(lying|10000 heads reported) = 0.09.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I think the original (and highly rated answer) could be expanded a little bit; in no way should you conclude that the data is falsified without thoroughly considering prior information. Also, just thinking about this intuitively, it seems that the posterior probabilities of lying are likely to be influenced more by the prior probability of lying rather than by the prior distribution of heads reported given that the flipper is lying (except for priors that put all their mass on a small number of heads reported given the flipper is lying, such as in my example.)&lt;/p&gt;&#xA;" OwnerUserId="2144" LastEditorUserId="2144" LastEditorDisplayName="" LastEditDate="2010-11-26T05:06:35.820" LastActivityDate="2010-11-26T05:06:35.820" CommentCount="2" />
  <row Id="4925" PostTypeId="2" ParentId="4918" CreationDate="2010-11-26T04:34:08.287" Score="5" ViewCount="" Body="&lt;p&gt;I'm not sure if this is really what you are asking for, so please clarify or state otherwise if this isn't what you had in mind. In the mean time, here in an approach to calculate a 10% threshold and then also select the elements of that list which are above that threshold. This is using &lt;a href=&quot;http://www.r-project.org/&quot; rel=&quot;nofollow&quot;&gt;R&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#Generate list of &quot;random numbers&quot; with normal distribution, mean = 0, sd = 3&#xA;x &amp;lt;- rnorm(n = 1000, mean = 0, sd = 3) &#xA;&#xA;#Returns the 90% quantile of the bunch of random numbers above&#xA;quantile(x, .9)&#xA;&#xA;#Subset x and only return those which are above the 90% threshold.&#xA;x[x &amp;gt; quantile(x, .9)]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="696" LastEditorUserId="696" LastEditorDisplayName="" LastEditDate="2010-11-26T16:35:47.747" LastActivityDate="2010-11-26T16:35:47.747" />
  <row Id="4926" PostTypeId="2" ParentId="4920" CreationDate="2010-11-26T05:43:31.740" Score="7" ViewCount="" Body="&lt;p&gt;Both B and E are derived from V.  B and E are clearly not truly &quot;independent&quot; variables from each other.  The underlying variable that really matters here is V.  You should probably disgard both B and E in this case and keep V only.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a more general situation, when you have two independent variables that are very highly correlated, you definitely should remove one of them because you run into the multicollinearity conundrum and your regression model's regression coefficients related to the two highly correlated variables will be unreliable.  Also, in plain English if two variables are so highly correlated they will obviously impart nearly exactly the same information to your regression model.  But, by including both you are actually weakening the model.  You are not adding incremental information.  Instead, you are infusing your model with noise.  Not a good thing. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One way you could keep highly correlated variables within your model is to use instead of regression a Principal Component Analysis (PCA) model.  PCA models are made to get rid off multicollinearity.  The trade off is that you end up with two or three principal components within your model that are often just mathematical constructs and are pretty much incomprehensible in logical terms.  PCA is therefore frequently abandoned as a method whenever you have to present your results to an outside audience such as management, regulators, etc...  PCA models create cryptic black boxes that are very challenging to explain.       &lt;/p&gt;&#xA;" OwnerUserId="1329" LastActivityDate="2010-11-26T05:43:31.740" CommentCount="5" />
  <row Id="4927" PostTypeId="2" ParentId="4868" CreationDate="2010-11-26T05:57:08.417" Score="0" ViewCount="" Body="&lt;p&gt;I'm not 100% clear on the question, but I have a few points to add:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm assuming that the error you are trying to estimate is the prediction error. If so, I agree that 10 fold cross validation would be good (and likely unbiased) approximation of the true prediction error &lt;em&gt;IF&lt;/em&gt; your training sets are sufficiently large. Large in this case means that the training sets provide enough information to build a &quot;good&quot; SVM (one that, in a sense, captures most of the underlying relationships between between the predictors and response.) Training sets of size 900 are more than likely large enough. In fact, unless the SVM you are fitting is extremely complex, I would recommend using a 5-fold cross validation in order to get a more precise estimate of prediction error (and yes, you can average the error estimates of the 5 folds to obtain an final estimate.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With regards to the question:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Would events tested using separately trained svms be comparable? i.e. through this technique could I then use my entire dataset instead of setting aside a certain fraction for training, or is this a statistically unwise thing to do?&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't understand this question, but since the phrase &quot;entire dataset&quot; is in a post about CV, I just want to warn you that estimating prediction error from models fit to all available data is generally a bad idea. For cross validation to make sense, each training set/test set pair should have no points in common. Otherwise, the true error will likely be underestimated.&lt;/p&gt;&#xA;" OwnerUserId="2144" LastActivityDate="2010-11-26T05:57:08.417" />
  <row Id="4928" PostTypeId="2" ParentId="4908" CreationDate="2010-11-26T06:24:56.867" Score="1" ViewCount="" Body="&lt;p&gt;Sounds like you'll need a HMM to do that.&#xA;Have you read&lt;br&gt;&#xA;Lawrence R. Rabiner (February 1989). &lt;a href=&quot;http://www.ece.ucsb.edu/Faculty/Rabiner/ece259/Reprints/tutorial%20on%20hmm%20and%20applications.pdf&quot; rel=&quot;nofollow&quot;&gt;&quot;A tutorial on Hidden Markov Models and selected applications in speech recognition&quot;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are a few examples of models parameters discovery on page 259.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The question is how do you plan to update the probabilites? Maybe you can use a kind of success ratio? (if the player has not managed to win much recently using one of the 3 strategies, he may use it less and try more the two others...)&lt;/p&gt;&#xA;" OwnerUserId="1709" LastEditorUserId="1709" LastEditorDisplayName="" LastEditDate="2010-11-26T06:47:18.580" LastActivityDate="2010-11-26T06:47:18.580" CommentCount="2" />
  <row Id="4929" PostTypeId="2" ParentId="4920" CreationDate="2010-11-26T08:13:43.497" Score="4" ViewCount="" Body="&lt;p&gt;Here is an answer from the point of view of a machine learner, although I am afraid that I'll be beaten by real statisticians for it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Is it possible for me to just &quot;throw away&quot; one of the variables?&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Well, the question is what type of model you want to use for prediction. It depends e.g. on ...&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;can the model with correlated predictors ? E.g. although NaiveBayes theoretically has problems with correlated variables, experiments have shown that it still can perform well.&lt;/li&gt;&#xA;&lt;li&gt;how does the model process the predictor variables ? E.g. the difference between B and V will be normalized out in a probability density estimation, maybe the same for E and V depending on the variance of D (as euphoria already said)&lt;/li&gt;&#xA;&lt;li&gt;which usage combination of B and E (one, none, both) delivers the best result, estimated by a mindful crossvalidation + a test on a holdout set ?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Sometimes we machine learners even perform genetic optimization to find the best arithmetic combination of a set of predictors. &lt;/p&gt;&#xA;" OwnerUserId="264" LastEditorUserId="264" LastEditorDisplayName="" LastEditDate="2010-11-26T08:21:31.757" LastActivityDate="2010-11-26T08:21:31.757" CommentCount="1" />
  <row Id="4930" PostTypeId="1" AcceptedAnswerId="4931" CreationDate="2010-11-26T08:54:39.863" Score="5" ViewCount="286" Body="&lt;p&gt;&lt;strong&gt;Disclaimer: I posted this question on Stackoverflow, but I thought maybe this is better suited for this platform.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do you test your own k-means implementation for multidimensional data sets?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was thinking of running an already existing implementation (i.e., Matlab) on the data and compare the results with my algorithm. But this would require both algorithms to work more than roughly the same, and the mapping between the two results probably is no piece of cake.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you have a better idea?&lt;/p&gt;&#xA;" OwnerUserId="2147" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-26T09:43:22.517" LastActivityDate="2010-11-26T19:55:50.543" Title="How do you test an implementation of k-means?" Tags="&lt;clustering&gt;&lt;algorithms&gt;" AnswerCount="3" FavoriteCount="1" />
  <row Id="4931" PostTypeId="2" ParentId="4930" CreationDate="2010-11-26T09:15:39.287" Score="6" ViewCount="" Body="&lt;p&gt;The k-means includes a stochastic component, so it is very unlikely you will get the same result unless you have exactly the same implementation and use the same starting configuration. However, you could see if your results are in agreement with well-known implementations (don't know about Matlab, but implementation of k-means algorithm in R is well explained, see &lt;a href=&quot;http://jmlr.csail.mit.edu/proceedings/papers/v9/telgarsky10a/telgarsky10a.pdf&quot; rel=&quot;nofollow&quot;&gt;Hartigan &amp;amp; Wong, 1979&lt;/a&gt;). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for comparing two series of results, there still is an issue with label switching if it is to be run multiple times. Again, in the &lt;a href=&quot;http://cran.r-project.org/web/packages/e1071/index.html&quot; rel=&quot;nofollow&quot;&gt;e1071&lt;/a&gt; R package, there is a very handy function (;&lt;code&gt;matchClasses()&lt;/code&gt;) that might be used to find the 'best' mapping between two categories in a two-way classification table. Basically, the idea is to rearrange the rows so as to maximise their agreement with columns, or use a greedy approach and permute rows and columns until the sum of on the diagonal (raw agreement) is maximal. Coefficient of agreement like the &lt;a href=&quot;http://en.wikipedia.org/wiki/Cohen%27s_kappa&quot; rel=&quot;nofollow&quot;&gt;Kappa&lt;/a&gt; statistic are also provided.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, about how to benchmark your implementation, there are a lot of freely available data, or you can simulate a dedicated data set (e.g., through a finite mixture model, see the &lt;a href=&quot;http://cran.r-project.org/web/packages/MixSim/index.html&quot; rel=&quot;nofollow&quot;&gt;MixSim&lt;/a&gt; package).&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-11-26T09:15:39.287" CommentCount="2" />
  <row Id="4932" PostTypeId="1" AcceptedAnswerId="4933" CreationDate="2010-11-26T10:34:05.437" Score="1" ViewCount="49" Body="&lt;p&gt;I have a data frame like this&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;User     OS&#xA;A        Windows&#xA;A        Linux&#xA;B        MacOS&#xA;C        Linux&#xA;C        FreeBSD&#xA;C        Windows&#xA;D        Windows  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What I want to do is plot two types of statistics.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The share of users with different number of OSs. So, it would be the fraction of the users with 1 OS, with OS etc.&lt;/li&gt;&#xA;&lt;li&gt;The share of users with only one OS and more than one OS.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;For this I tried using &lt;code&gt;tapply(OS, User, unique)&lt;/code&gt;. But don't know how to go about plotting the results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was wondering if this is the right way and what more would I need to do to to get the plots I wanted.&lt;/p&gt;&#xA;" OwnerUserId="2101" LastActivityDate="2010-11-26T10:51:53.453" Title="R getting share of users with multiple of an element" Tags="&lt;r&gt;&lt;distributions&gt;&lt;aggregation&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="4933" PostTypeId="2" ParentId="4932" CreationDate="2010-11-26T10:45:18.043" Score="4" ViewCount="" Body="&lt;p&gt;You could try something like&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; df &amp;lt;- data.frame(User=sample(LETTERS[1:10], 100, rep=T), &#xA;                   OS=sample(c(&quot;Win&quot;,&quot;Lin&quot;,&quot;Mac&quot;), 100, rep=T))&#xA;&amp;gt; (res &amp;lt;- with(df, tapply(OS, User, function(x) length(unique(x)))))&#xA;A B C D E F G H I J &#xA;2 3 3 3 3 3 3 3 3 3 &#xA;&amp;gt; barplot(table(res)) # for counts&#xA;&amp;gt; barplot(table(ifelse(res==1, &quot;1&quot;, &quot;2+&quot;)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Replace &lt;code&gt;table()&lt;/code&gt; by &lt;code&gt;prop.table()&lt;/code&gt; if you want proportions instead of counts, as suggested by &lt;a href=&quot;http://stats.stackexchange.com/users/696/chase&quot;&gt;@Chase&lt;/a&gt; in a comment to your preceding question.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-26T10:51:53.453" LastActivityDate="2010-11-26T10:51:53.453" />
  <row Id="4934" PostTypeId="1" AcceptedAnswerId="4935" CreationDate="2010-11-26T11:00:49.113" Score="1" ViewCount="190" Body="&lt;p&gt;Are there any good papers or books dealing with the use of coordinate descent for L1 (lasso) and/or elastic net regularization for linear regression problems?&lt;/p&gt;&#xA;" OwnerUserId="439" LastEditorUserId="439" LastEditorDisplayName="" LastEditDate="2010-11-26T11:13:50.427" LastActivityDate="2010-11-26T15:07:41.943" Title="Coordinate descent for the lasso or elastic net" Tags="&lt;regression&gt;&lt;lasso&gt;&lt;elastic-net&gt;&lt;regularization&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="4935" PostTypeId="2" ParentId="4934" CreationDate="2010-11-26T11:21:34.733" Score="2" ViewCount="" Body="&lt;p&gt;I earlier suggested the recent paper by Friedman and coll., &lt;a href=&quot;http://www.jstatsoft.org/v33/i01/paper&quot; rel=&quot;nofollow&quot;&gt;Regularization Paths for Generalized Linear Models via Coordinate Descent&lt;/a&gt;, published in the Journal of Statistical Software (2010). Here are some other references that might be useful:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www-stat.stanford.edu/~tibs/stat315a/Supplements/fuse.pdf&quot; rel=&quot;nofollow&quot;&gt;Pathwise coordinate optimization&lt;/a&gt;, by Friedman and coll.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www-stat.stanford.edu/~hastie/TALKS/glmnet.pdf&quot; rel=&quot;nofollow&quot;&gt;Fast Regularization Paths via Coordinate Descent&lt;/a&gt;, by Hastie (UseR! 2009)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://projecteuclid.org/euclid.aoas/1206367819&quot; rel=&quot;nofollow&quot;&gt;Coordinate descent algorithms for lasso penalized regression&lt;/a&gt;, by Wu and Lange (Ann. Appl. Stat. 2(1): 224-244, 2008; also on available on &lt;a href=&quot;http://arxiv.org/&quot; rel=&quot;nofollow&quot;&gt;arXiv.org&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.eecs.umich.edu/~aey/sparse/sparse9.pdf&quot; rel=&quot;nofollow&quot;&gt;Coordinate Descent for Sparse Solutions of Underdetermined Linear Systems of Equations&lt;/a&gt;, by Yagle (a bit too complex for me)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-11-26T11:21:34.733" CommentCount="2" />
  <row Id="4936" PostTypeId="1" AcceptedAnswerId="4938" CreationDate="2010-11-26T11:28:53.567" Score="5" ViewCount="301" Body="&lt;p&gt;What are the pros and cons of using LARS [1] versus using coordinate descent for fitting L1-regularized linear regression?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am mainly interested in performance aspects (my problems tend to have &lt;code&gt;N&lt;/code&gt; in the hundreds of thousands and &lt;code&gt;p&lt;/code&gt; &amp;lt; 20.) However, any other insights would also be appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;edit: Since I've posted the question, chl has kindly pointed out a paper [2] by Friedman et al where coordinate descent is shown to be considerably faster than other methods. If that's the case, should I as a practitioner simply forget about LARS in favour of coordinate descent?&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;[1] Efron, Bradley; Hastie, Trevor; Johnstone, Iain and Tibshirani, Robert (2004). &quot;Least Angle Regression&quot;. Annals of Statistics 32 (2): pp. 407–499.&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;[2] Jerome H. Friedman, Trevor Hastie, Rob Tibshirani, &quot;Regularization Paths for Generalized Linear Models via Coordinate Descent&quot;, Journal of Statistical Software, Vol. 33, Issue 1, Feb 2010.&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="439" LastEditorUserId="439" LastEditorDisplayName="" LastEditDate="2010-11-26T11:49:28.603" LastActivityDate="2010-11-27T09:46:13.680" Title="LARS vs coordinate descent for the lasso" Tags="&lt;regression&gt;&lt;lasso&gt;&lt;regularization&gt;" AnswerCount="1" />
  <row Id="4937" PostTypeId="2" ParentId="4930" CreationDate="2010-11-26T11:53:26.917" Score="0" ViewCount="" Body="&lt;p&gt;Since k-means contains decisions that are randomly chosen (the initialization part only), I think the best way to try your algorithm is to select the initial points and let them fixed in your algorithm first and then choose another source code of the algorithm and fix the points in the same way. Then you can compare for real the results.&lt;/p&gt;&#xA;" OwnerUserId="1808" LastActivityDate="2010-11-26T11:53:26.917" />
  <row Id="4938" PostTypeId="2" ParentId="4936" CreationDate="2010-11-26T13:15:57.283" Score="4" ViewCount="" Body="&lt;p&gt;In scikit-learn the implementation of &lt;a href=&quot;http://scikit-learn.sourceforge.net/modules/linear_model.html#lasso&quot; rel=&quot;nofollow&quot;&gt;Lasso with coordinate descent&lt;/a&gt; tends to be faster than our implementation of LARS although for small p (such as in your case) they are roughly equivalent (LARS might even be a bit faster with the latest optimizations available in the master repo). Furthermore coordinate descent allows for efficient implementation of elastic net regularized problems. This is not the case for LARS (that solves only Lasso, aka L1 penalized problems).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Elastic Net penalization tends to yield a better generalization than Lasso (closer to the solution of ridge regression) while keeping the nice sparsity inducing features of Lasso (supervised feature selection).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For large N (and large p, sparse or not) you might also give a &lt;a href=&quot;http://scikit-learn.sourceforge.net/modules/sgd.html&quot; rel=&quot;nofollow&quot;&gt;stochastic gradient descent&lt;/a&gt; (with L1 or elastic net penalty) a try (also implemented in scikit-learn).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit&lt;/strong&gt;: here are some &lt;a href=&quot;http://picasaweb.google.com/olivier.grisel/MachineLearning#5543914944364375506&quot; rel=&quot;nofollow&quot;&gt;benchmarks comparing LassoLARS and the coordinate descent implementation&lt;/a&gt; in scikit-learn&lt;/p&gt;&#xA;" OwnerUserId="2150" LastEditorUserId="2150" LastEditorDisplayName="" LastEditDate="2010-11-27T09:46:13.680" LastActivityDate="2010-11-27T09:46:13.680" CommentCount="8" />
  <row Id="4939" PostTypeId="1" CreationDate="2010-11-26T14:09:15.643" Score="8" ViewCount="235" Body="&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/Gr3f3.png&quot; alt=&quot;IEEE CSDA Practice Exam - Engineering Statistics - Control Chart&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am given 15 points. The control limits are at +/- 3 $\sigma$. Points 1, 4, 5, 6, 7, 8, 9, 10, 11, 13, and 15 fall within the control limits. Points 2, 3, 12, and 14 are outside of the control limits, with 2 being below the lower control limit and 3, 12, and 14 being above the upper control limit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I know if points 2, 3, 12, and 14 are out of control caused by chance causes or caused by assignable causes?&lt;/p&gt;&#xA;" OwnerUserId="110" LastEditorUserId="110" LastEditorDisplayName="" LastEditDate="2010-11-26T16:26:40.237" LastActivityDate="2010-11-30T11:58:59.917" Title="Given a control chart that shows the mean and upper/lower control limits, how do I tell if the cause of out of control points is assignable or not?" Tags="&lt;control-chart&gt;&lt;engineering-statistics&gt;" AnswerCount="4" CommentCount="3" FavoriteCount="1" />
  <row Id="4940" PostTypeId="2" ParentId="4901" CreationDate="2010-11-26T14:58:38.717" Score="7" ViewCount="" Body="&lt;p&gt;Cox and Wermuth (1996) or Cox (1984) discussed some methods for detecting interactions. The problem is usually how general the interaction terms should be. Basically, we &#xA;(a) fit (and test) all second-order interaction terms, one at a time, and (b) plot their corresponding p-values (i.e., the No. terms as a function of $1-p$). The idea is then to look if a certain number of interaction terms should be retained: Under the assumption that all interaction terms are null the distribution of the p-values should be uniform (or equivalently, the points on the scatterplot should be roughly distributed along a line passing through the origin).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, as &lt;a href=&quot;http://stats.stackexchange.com/questions/4901/identifying-interaction-effects/4903#4903&quot;&gt;@Gavin&lt;/a&gt; said, fitting many (if not all) interactions might lead to overfitting, but it is also useless in a certain sense (some high-order interaction terms often have no sense at all). However, this has to do with interpretation, not detection of interactions, and a good review was already provided by Cox in &lt;a href=&quot;http://arxiv.org/PS_cache/arxiv/pdf/0712/0712.1106v1.pdf&quot; rel=&quot;nofollow&quot;&gt;Interpretation of interaction: A review&lt;/a&gt; (&lt;em&gt;The Annals of Applied Statistics&lt;/em&gt; 2007, 1(2), 371–385)--it includes references cited above. Other lines of research worth to look at are study of &lt;a href=&quot;http://en.wikipedia.org/wiki/Epistasis&quot; rel=&quot;nofollow&quot;&gt;epistatic effects&lt;/a&gt; in genetic studies, in particular methods based on graphical models (e.g., &lt;a href=&quot;http://biostatistics.oxfordjournals.org/content/early/2009/07/22/biostatistics.kxp025.abstract&quot; rel=&quot;nofollow&quot;&gt;An efficient method for identifying statistical interactors in gene association networks&lt;/a&gt;).&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-11-26T14:58:38.717" />
  <row Id="4941" PostTypeId="2" ParentId="4934" CreationDate="2010-11-26T15:07:41.943" Score="0" ViewCount="" Body="&lt;p&gt;I've just come across this &lt;a href=&quot;http://videolectures.net/kdd08_hastie_rpcd/&quot; rel=&quot;nofollow&quot;&gt;lecture&lt;/a&gt; by Hastie and thought that others might find it interesting.&lt;/p&gt;&#xA;" OwnerUserId="439" LastActivityDate="2010-11-26T15:07:41.943" />
  <row Id="4942" PostTypeId="1" AcceptedAnswerId="4951" CreationDate="2010-11-26T15:22:47.893" Score="1" ViewCount="207" Body="&lt;p&gt;The fisher linear classifier for two classes is a classifier with this discriminant function:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$h(x) = V^{T}X + v_0$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$V = \left[ \frac{1}{2}\Sigma_1 + \frac{1}{2}\Sigma_2\right]^{-1}(M_2-M_1)$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and $M_1$, $M_2$ are means and $\Sigma_1$,$\Sigma_2$ are covariances of the classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$V$ can be calculated easily but the fisher criterion cannot give us the optimum $v_0$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, how we can find the optimum $v_0$ analytically if we are using fisher classifier?&lt;/p&gt;&#xA;" OwnerUserId="2148" LastActivityDate="2010-11-26T20:33:56.237" Title="Threshold for Fisher linear classifier" Tags="&lt;classification&gt;&lt;optimization&gt;" AnswerCount="1" CommentCount="8" />
  <row Id="4943" PostTypeId="2" ParentId="4939" CreationDate="2010-11-26T15:45:12.343" Score="5" ViewCount="" Body="&lt;p&gt;My understanding of control charts is a little bit different... After the first signal at observation 2, wouldn't the process would be stopped and checked for problems, and then restarted?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In any case, you could use a p-value argument. The probability of observing 4 or more observations (out of 15) beyond their control limits is VERY tiny if the process is actually in control. Let's say the probability of an observation going outside of the control limits while the process is actually in control is about 0.01 (this exact probability depends on the in control distribution of the data), so if the process is in control, we expect a false alarm (ie out of control signal caused by random chance) every 100 observations or so. The probability of observing 4 or more out of control signals (out of 15) while the process is in control is about 0.000012, so it's very unlikely that the signals are due to random chance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While an actual diagnosis would require you to look at the chart and possibly actually investigate the physical process, because the out of control points are both below and above the control limits, I'm betting there was a scale shift (i.e. increase in variance.)  &lt;/p&gt;&#xA;" OwnerUserId="2144" LastEditorUserId="2144" LastEditorDisplayName="" LastEditDate="2010-11-26T19:39:31.937" LastActivityDate="2010-11-26T19:39:31.937" CommentCount="4" />
  <row Id="4944" PostTypeId="2" ParentId="4939" CreationDate="2010-11-26T16:24:12.580" Score="5" ViewCount="" Body="&lt;p&gt;Yes, you should find and assignable cause for every point that's outside the limits. But things are a little more complicated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First you have to determine if the process is in control, since a control chart is meaningless when the process is out of control. Nearly 1/4 of your observations falling outside the limits is a strong sign that the process may be out of control. Looking at the chart would be useful to determine whether the process is under control or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Besides falling outside the control limits, there are other potential reasons for needing to look for assignable causes for certain observations. For example, if you have several observations in a row falling on the same side of the mean -- especially if they're near the control limit -- they may need to assigned a special cause.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I might be able to be more specific if you'd post the chart itself.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to learn more about control charts, &lt;a href=&quot;http://www.spcpress.com/reading_room.php&quot; rel=&quot;nofollow&quot;&gt;SPC Press&lt;/a&gt; has a number of useful free resources. You might also want to look at &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0945320531&quot; rel=&quot;nofollow&quot;&gt;this book&lt;/a&gt;: it's short, concise and very informative.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;(Edit:)&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I assumed we were talking about real-world data, not an exam question. In this case, the correct answer really is the first one: the points outside the control limits are (probably) caused by assignable causes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The exam is a little sloppy in its terminology, though: you can't actually tell with 100% certainty that the points outside the control limits are not caused by chance. You can only say that there is a 99.7% probability that a particular point outside the limits is not caused by chance.&lt;/p&gt;&#xA;" OwnerUserId="666" LastEditorUserId="666" LastEditorDisplayName="" LastEditDate="2010-11-30T11:58:59.917" LastActivityDate="2010-11-30T11:58:59.917" CommentCount="1" />
  <row Id="4945" PostTypeId="1" CreationDate="2010-11-26T16:34:00.810" Score="0" ViewCount="392" Body="&lt;p&gt;Hello - I &lt;a href=&quot;http://stats.stackexchange.com/q/4914/919&quot;&gt;posted a question earlier&lt;/a&gt; but failed miserably in trying to explain what I am looking for (thanks to those who tried to help me anyway). Will try again, starting with a clean sheet.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Standard deviations are sensitive to scale. Since I am trying to perform a statistical test where the best result is predicted by the lowest standard deviation amongst different data sets, is there a way to &quot;normalize&quot; it for scale, or use a different standard-deviation-type test altogether?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unfortunately dividing the resulting standard deviation by the mean in my case does not work, as the mean is almost always close to zero.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="2137" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-11-26T20:58:19.493" LastActivityDate="2010-12-29T17:44:29.093" Title="&quot;Normalized&quot; standard deviation" Tags="&lt;standard-deviation&gt;" AnswerCount="2" CommentCount="3" FavoriteCount="1" />
  <row Id="4946" PostTypeId="2" ParentId="4930" CreationDate="2010-11-26T16:46:09.033" Score="3" ViewCount="" Body="&lt;p&gt;The mapping between two sets of results is easy to compute, because the information you obtain in a test can be represented as a set of three-tuples: the first component is a (multidimensional) point, the second is an (arbitrary) cluster label supplied by your algorithm, and the third is an (arbitrary) cluster label supplied by a reference algorithm.  Construct the $k$ by $k$ classification table for the label pairs: if the results agree, it will be a multiple of a permutation matrix.  That is, each row and each column must have exactly one nonzero cell.  That's a simple check to program.  It's also straightforward to track small deviations from this ideal back to individual data points so you can see precisely how the two answers differ if they differ at all.  I wouldn't bother to compute statistical measures of agreement: either there is perfect agreement (up to permutation) or there is not, and in the latter case you need to track down all points of disagreement to understand how they occur.  The results either agree or they do not; any amount of disagreement, even at just one point, needs checking.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might want to use several kinds of datasets for testing: (1) published datasets with published k-means results; (2) synthetic datasets with obvious strong clusters; (3) synthetic datasets with no obvious clustering.  (1) is a good discipline to use whenever you write &lt;em&gt;any&lt;/em&gt; math or stats program.  (2) is easy to do in many ways, such as by generating some random points to serve as centers of clusters and then generating point clouds by randomly displacing the cluster centers relatively small amounts.  (3) provides some random checks that potentially uncover unexpected behaviors; again, that's a good general testing discipline.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In addition, consider creating datasets that stress the algorithm by lying just on the boundaries between extreme solutions.  This will require creativity and a deep understanding of your algorithm (which presumably you have!).  One example I would want to check in any event would be sets of vectors of the form $i \mathbb{v}$ where $\mathbb{v}$ is a vector with no zero components and $i$ takes on sequential integral values $0, 1, 2, \ldots, n-1$.  I would also want to check the algorithm on sets of vectors that form equilateral polygons.  In either situation, cases where $n$ is &lt;em&gt;not&lt;/em&gt; a multiple of $k$ are particularly interesting, including where $n$ is &lt;em&gt;less&lt;/em&gt; than $k$.  What is common to these situations is that (a) they use all the dimensions of the problem, yet (b) the correct solutions are geometrically obvious, and (c) there are multiple correct solutions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Form random equilateral polygons in $d \ge 2$ dimensions by starting with two nonzero vectors $\mathbb{u}$ and $\mathbb{v}$ chosen at random.  (A good way is to let their $2d$ components be independent standard normal variates.)  Rescale them to have unit length; let's call these $\mathbb{x}$ and $\mathbb{z}$.  Remove the $\mathbb{x}$ component from $\mathbb{z}$ by means of the formula&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\mathbb{w} = \mathbb{z} - ( \mathbb{z} \cdot \mathbb{x} ) \mathbb{x}.$$  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Obtain $\mathbb{y}$ by rescaling $\mathbb{w}$ to have unit length.  If you like, uniformly rescale both $\mathbb{x}$ and $\mathbb{y}$ randomly.  The vectors $\mathbb{x}$ and $\mathbb{y}$ form an orthogonal basis for a random 2D subspace in $d$ dimensions.  An equilateral polygon of $n$ vertices is obtained as the set of $\cos(2 \pi k / n) \mathbb{x} + \sin(2 \pi k / n) \mathbb{y}$ as the integer $k$ ranges from $0$ through $n-1$.)&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-26T19:55:50.543" LastActivityDate="2010-11-26T19:55:50.543" CommentCount="1" />
  <row Id="4947" PostTypeId="2" ParentId="4914" CreationDate="2010-11-26T16:47:25.097" Score="3" ViewCount="" Body="&lt;p&gt;Smoothing, rolling averages, running means... are all nice ways (perhaps) to display data. But using the results of smoothed data as an input to any statistical analysis is likely to give misleading results, especially when done by novices. William Briggs emphasizes this point in his excellent blog in &lt;a href=&quot;http://wmbriggs.com/blog/?p=195&quot; rel=&quot;nofollow&quot;&gt;this article&lt;/a&gt; and &lt;a href=&quot;http://wmbriggs.com/blog/?p=735&quot; rel=&quot;nofollow&quot;&gt;this one&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="25" LastEditorUserId="25" LastEditorDisplayName="" LastEditDate="2010-11-26T16:53:01.127" LastActivityDate="2010-11-26T16:53:01.127" CommentCount="1" />
  <row Id="4948" PostTypeId="2" ParentId="4912" CreationDate="2010-11-26T17:23:03.043" Score="3" ViewCount="" Body="&lt;p&gt;You are conducting a &lt;strong&gt;one-sided&lt;/strong&gt; test of a &lt;strong&gt;difference of proportions&lt;/strong&gt;.  Because all four outcomes--A, not A, B, not B--occur often (70 times or more in this case), the Normal approximation to the Binomial distribution will be just fine.  Let $a$ be the number of occurrences of A, $b$ the number of occurrences of B, and $n$ the total sample (about 350).  Under the null hypothesis $a = b$ the variance of a single observation is estimated with the combined data, $s^2 = (a+b)/(2n) \cdot (1 - (a+b)/(2n))$. The variances of A and B are estimated as $s^2/n$.  The test statistic therefore is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$z = \frac{a/n - b/n}{\sqrt{s^2/n + s^2/n}}.$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The p-value equals $1 - \Phi^{-1}(z)$ where $\Phi^{-1}$ is the percentage point function for the standard normal distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, with $n = 350$, $a = 280$, and $b = 70$, we estimate $a$ as 0.8, $b$ as 0.2, and $s^2$ as 0.25, giving $z = 15.87$.  Obviously that's not due to chance.  The result will be equally strong and obvious for any values of $a$, $b$, and $n$ anywhere close to these.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-11-28T19:19:27.373" LastActivityDate="2010-11-28T19:19:27.373" CommentCount="3" />
  <row Id="4949" PostTypeId="1" CreationDate="2010-11-26T19:36:32.340" Score="3" ViewCount="296" Body="&lt;p&gt;If two classes $w_1$ and $w_2$ have normal distribution with known parameters ($M_1$, $M_2$ as their means and $\Sigma_1$,$\Sigma_2$ are their covariances) how we can calculate error of the Bayes classifier for them theorically?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also suppose the variables are in N-dimensional space.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note:&lt;/strong&gt; A copy of this question is also available at &lt;a href=&quot;http://math.stackexchange.com/q/11891/4051&quot;&gt;http://math.stackexchange.com/q/11891/4051&lt;/a&gt; that is still unanswered. If any of these question get answered, the other one will be deleted.&lt;/p&gt;&#xA;" OwnerUserId="2148" LastEditorUserId="1739" LastEditorDisplayName="" LastEditDate="2010-12-26T12:51:32.047" LastActivityDate="2010-12-31T09:41:57.923" Title="Calculating the error of Bayes classifier analytically" Tags="&lt;probability&gt;&lt;homework&gt;&lt;normality&gt;&lt;naive-bayes&gt;" AnswerCount="3" CommentCount="4" FavoriteCount="1" />
  <row Id="4950" PostTypeId="2" ParentId="4939" CreationDate="2010-11-26T20:16:29.420" Score="2" ViewCount="" Body="&lt;p&gt;I found something interesting tucked away in a study document from the IEEE geared toward this exam:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;ul&gt;&#xA;  &lt;li&gt;Data points falling within the UCL and LCL range are considered to be in control and caused by chance causes.&lt;/li&gt;&#xA;  &lt;li&gt;Outliers falling above the UCL or below the LCL are considered to be out of control and caused by assignable causes.&lt;/li&gt;&#xA;  &lt;li&gt;If a number of points fall systematically above or below the mean (but are within the UCL and LCL) this may indicate a nonrandom out-of-control state.&lt;/li&gt;&#xA;  &lt;li&gt;The goal of a control chart is to detect out-of-control states quickly.&lt;/li&gt;&#xA;  &lt;li&gt;The chart, alone, will not indicate the root causes of the event, but it will provide investigative leads.&lt;/li&gt;&#xA;  &lt;/ul&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Apparently, if you cross either the UCL or LCL, there has to be an assignable cause.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This makes sense, given the &lt;a href=&quot;https://secure.wikimedia.org/wikipedia/en/wiki/Assignable_cause#Special-cause_variation&quot; rel=&quot;nofollow&quot;&gt;Wikipedia definition of characteristics of assignable (special) cause&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;ul&gt;&#xA;  &lt;li&gt;New, unanticipated, emergent or previously neglected phenomena within the system;&lt;/li&gt;&#xA;  &lt;li&gt;Variation inherently unpredictable, even probabilistically;&lt;/li&gt;&#xA;  &lt;li&gt;Variation outside the historical experience base; and&lt;/li&gt;&#xA;  &lt;li&gt;Evidence of some inherent change in the system or our knowledge of it.&lt;/li&gt;&#xA;  &lt;/ul&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="110" LastActivityDate="2010-11-26T20:16:29.420" CommentCount="2" />
  <row Id="4951" PostTypeId="2" ParentId="4942" CreationDate="2010-11-26T20:33:56.237" Score="3" ViewCount="" Body="&lt;p&gt;When $X$ is normally distributed with &lt;em&gt;known&lt;/em&gt; mean $M_1$ and covariance $\Sigma_1$ or with mean $M_2$ and covariance $\Sigma_2$, as indicated in comments to the question, then $V^{\ &amp;#39;}X$ is normally distributed either with mean $\mu_1 = V^{\ &amp;#39;} M_1$ and covariance $\sigma_1^2 = V^{\ &amp;#39;} \Sigma_1 V$ or with mean $\mu_2 = V^{\ &amp;#39;} M_2$ and covariance $\sigma_2^2 = V^{\ &amp;#39;} \Sigma_2 V$; $\mu_2 \gt \mu_1$.  We might then care to optimize the chance of correct classification.  This can be done &lt;em&gt;provided we stipulate a prior distribution for the two classes.&lt;/em&gt;  Letting $\pi_1$ be the chance of class 1 and $\pi_2$ the chance of class 2 and $\phi$ the standard normal pdf, then the posterior probabilities of the classes are equal (and therefore $x$ is at the threshold) when&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$f(x) = \pi_1 \phi(\frac{x - \mu_1}{\sigma_1}) - \pi_2 \phi(\frac{x - \mu_2}{\sigma_2}) = 0.$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There will be at most one zero of $f$ between $x = \mu_1$ and $x = \mu_2$.  (When the zeros lie outside this interval we might question the utility of this classifier.)  Assuming one exists and choosing $v_0$ to be the &lt;em&gt;negative&lt;/em&gt; of this zero gives a linear classifier $X \to V^{\ &amp;#39;}X + v_0$ that, when negative, indicates class 1 is more likely than class 2 and, when positive, indicates class 2 is more likely than class 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A simple case arises when the two classes are taken to be equally likely, $\pi_1 = \pi_2 = 1/2,$ for then it is clear from the symmetry and unimodality of $\phi$ that $v_0 = -(\mu_1 + \mu_2)/2$.  Note, though, that in general it is &lt;em&gt;not&lt;/em&gt; the case that the zero equals $\pi_1 \mu_1 + \pi_2 \mu_2$ (although that might be a good starting guess in a systematic search for the zero).&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-11-26T20:33:56.237" CommentCount="4" />
  <row Id="4952" PostTypeId="2" ParentId="4939" CreationDate="2010-11-26T20:46:32.820" Score="4" ViewCount="" Body="&lt;p&gt;(Sorry for posting a new answer, I can't reply to comments directly yet)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't really agree with the statement:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Apparently, if you cross either the UCL or LCL, there has to be an assignable cause&quot; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To keep things simple, if your in control distribution is N(0,1), then you will still obtain false alarms once every 370 observations, on average, using a UCL of 3 and LCL of -3. When the chart signals, the process needs to be investigated. Only then can a reason for the signal be assigned (ie process change or random error.) Setting the UCL and LCL requires the user to balance the desired false alarm/missed detection rate (analogous to the Type I/Type II error trade off in hypothesis testing.) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also wait until a few signals to actually stop and investigate the process, but in that case, you may detect the shift too late if it really occurred at the first signal. Again, you can't have something for nothing and the user must use their judgment to decide on how to set up the control chart and monitor the process.&lt;/p&gt;&#xA;" OwnerUserId="2144" LastActivityDate="2010-11-26T20:46:32.820" CommentCount="1" />
  <row Id="4953" PostTypeId="2" ParentId="4901" CreationDate="2010-11-27T03:38:45.647" Score="4" ViewCount="" Body="&lt;p&gt;I'll preface this response as I entirely agree with Gavin, and if your interested in fitting any type of model it should be reflective of the phenomenon under study. What the problem is with the logic of identifying any and all effects (and what Gavin refers to when he says data dredging) is that you could fit an infinite number of interactions, or quadratic terms for variables, or transformations to your data, and you would inevitably find &quot;significant&quot; effects for some variation of your data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As chl states, these higher order interaction effects don't really have any interpretation, and frequently even the lower order interactions don't make any sense. If your interested in developing a causal model you should only include terms you believe could be pertinent to your dependent variable &lt;em&gt;A priori&lt;/em&gt; to fitting your model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you believe they can increase predictive power of your model, you should look up resources on model selection techniques to prevent over-fitting your model.&lt;/p&gt;&#xA;" OwnerUserId="1036" LastActivityDate="2010-11-27T03:38:45.647" />
  <row Id="4954" PostTypeId="1" CreationDate="2010-11-27T06:11:33.187" Score="0" ViewCount="152" Body="&lt;p&gt;I am trying to estimate parameters of a two dimensional Normal distribution using Gibbs sampling. While it was very easy transform the posterior equation for mean vector to a single dimension normal function for sampling, I am not able to same for sigma(covariance).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do I need to use the Wishart distribution as prior and then convert the posterior into a single dimensional gamma function ? &lt;/p&gt;&#xA;" OwnerUserId="2157" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-11-27T11:42:17.817" LastActivityDate="2010-11-27T12:27:28.083" Title="Posterior expression for Gibbs sampling" Tags="&lt;mcmc&gt;&lt;gibbs&gt;" AnswerCount="1" />
  <row Id="4956" PostTypeId="2" ParentId="4817" CreationDate="2010-11-27T07:53:59.150" Score="1" ViewCount="" Body="&lt;p&gt;O.k,&#xA;I found that there is an unpaired solution to a sign test (A test of medians).  It is called &quot;Median test&quot; And you can read about it &lt;a href=&quot;http://en.wikipedia.org/wiki/Median_test&quot; rel=&quot;nofollow&quot;&gt;in Wikipedia&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="253" LastActivityDate="2010-11-27T07:53:59.150" />
  <row Id="4957" PostTypeId="2" ParentId="4949" CreationDate="2010-11-27T12:13:06.333" Score="0" ViewCount="" Body="&lt;p&gt;Here you might find several clues for your question, maybe is not there the full response but certainly very valuable parts of it.&#xA;&lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2766788/&quot; rel=&quot;nofollow&quot;&gt;http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2766788/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1808" LastActivityDate="2010-11-27T12:13:06.333" />
  <row Id="4958" PostTypeId="2" ParentId="4954" CreationDate="2010-11-27T12:27:28.083" Score="3" ViewCount="" Body="&lt;p&gt;The Wishart distribution is the conjugate prior for the likelihood that comes from assuming a normally distributed error term for the linear regression model. You have to assume that either the covariance matrix is an inverted-wishart or the precision matrix (i.e., the inverse of the covariance matrix) is a wishart distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as sampling is concerned it is possible to sample directly from the wishart and even from the multivariate normal. Thus, I do not think it is needed to sample from univariate normals or univariate gammas. Look around in the software you are using to see if it has samplers for the multivariate normal and the wishart.&lt;/p&gt;&#xA;" LastActivityDate="2010-11-27T12:27:28.083" CommentCount="3" />
  <row Id="4959" PostTypeId="1" AcceptedAnswerId="4960" CreationDate="2010-11-27T14:54:04.023" Score="5" ViewCount="354" Body="&lt;p&gt;If $x_i$ is exponentially distributed $(i=1,...,n)$ with parameter $\lambda$ and $x_i$'s are mutually independent, what is the expectation of&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$E\left\{ \left(\sum_{i=1}^n {x_i} \right)^2 \right\}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;in terms of $n$ and $\lambda$ and possibly other constants?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note:&lt;/strong&gt; This question has gotten a mathematical answer on &lt;a href=&quot;http://math.stackexchange.com/q/12068/4051&quot;&gt;http://math.stackexchange.com/q/12068/4051&lt;/a&gt;. The readers would take a look at it too.&lt;/p&gt;&#xA;" OwnerUserId="2148" LastEditorUserId="2148" LastEditorDisplayName="" LastEditDate="2011-03-05T18:02:11.450" LastActivityDate="2011-03-05T18:02:11.450" Title="Expectation of $\left(\sum_{i=1}^n {x_i} \right)^2$" Tags="&lt;expected-value&gt;&lt;exponential&gt;&lt;gamma-distribution&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="1" />
  <row Id="4960" PostTypeId="2" ParentId="4959" CreationDate="2010-11-27T15:46:40.613" Score="20" ViewCount="" Body="&lt;p&gt;If $x_i \sim Exp(\lambda)$, then (under independence), $y = \sum x_i \sim Gamma(n, 1/\lambda)$, so $y$ is gamma distributed (see &lt;a href=&quot;http://en.wikipedia.org/wiki/Exponential_distribution&quot; rel=&quot;nofollow&quot;&gt;wikipedia&lt;/a&gt;). So, we just need $E[y^2]$. Since $Var[y] = E[y^2] - E[y]^2$, we know that $E[y^2] = Var[y] + E[y]^2$. Therefore, $E[y^2] = n/\lambda^2 + n^2/\lambda^2 = n(1+n)/\lambda^2$ (see &lt;a href=&quot;http://en.wikipedia.org/wiki/Gamma_distribution&quot; rel=&quot;nofollow&quot;&gt;wikipedia&lt;/a&gt; for the expectation and variance of the gamma distribution).&lt;/p&gt;&#xA;" OwnerUserId="1934" LastActivityDate="2010-11-27T15:46:40.613" CommentCount="5" />
  <row Id="4961" PostTypeId="1" CreationDate="2010-11-27T16:24:53.693" Score="4" ViewCount="279" Body="&lt;p&gt;Unlike other articles, I found the &lt;a href=&quot;http://en.wikipedia.org/wiki/Regularization_%28mathematics%29&quot; rel=&quot;nofollow&quot;&gt;wikipedia&lt;/a&gt; entry for this subject unreadable for a non-math person (like me).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understood the basic idea, that you favor models with fewer rules. What I don't get is how do you get from a set of rules to a 'regularization score' which you can use to sort the models from least to most overfit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can you describe a simple regularization method?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm interested in the context of analyzing statistical trading systems. It would be great if you could describe if/how I can apply regularization to analyze the following two predictive models:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Model 1 - price going up when:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;exp_moving_avg(price, period=50) &gt; exp_moving_avg(price, period=200)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Model 2 - price going up when:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;price[n] &amp;lt; price[n-1] 10 times in a row&lt;/li&gt;&#xA;&lt;li&gt;exp_moving_avg(price, period=200) going up&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;But I'm more interested in getting a feeling for how you do regularization. So if you know better models for explaining it please do.&lt;/p&gt;&#xA;" OwnerUserId="749" LastActivityDate="2010-11-29T10:46:54.530" Title="What is regularization in plain english?" Tags="&lt;regularization&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="2" />
  <row Id="4962" PostTypeId="2" ParentId="423" CreationDate="2010-11-27T18:14:10.090" Score="4" ViewCount="" Body="&lt;p&gt;Almost a Chi square...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/OaiBC.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="253" LastActivityDate="2010-11-27T18:14:10.090" CommunityOwnedDate="2010-11-27T18:14:10.090" />
  <row Id="4964" PostTypeId="1" AcceptedAnswerId="4965" CreationDate="2010-11-27T19:42:14.197" Score="8" ViewCount="511" Body="&lt;p&gt;All of my variables are continuous. There are no levels. Is it possible to even &lt;em&gt;have&lt;/em&gt; interaction between the variables?&lt;/p&gt;&#xA;" OwnerUserId="1894" LastEditorUserId="1894" LastEditorDisplayName="" LastEditDate="2010-11-27T19:51:05.120" LastActivityDate="2010-11-28T11:40:27.143" Title="Is interaction possible between two continuous variables?" Tags="&lt;regression&gt;&lt;modeling&gt;&lt;interaction&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="4965" PostTypeId="2" ParentId="4964" CreationDate="2010-11-27T20:47:34.893" Score="12" ViewCount="" Body="&lt;p&gt;Yes, why not? The same consideration as for categorical variables would apply in this case: The effect of $X_1$ on the outcome $Y$ is not the same depending on the value of $X_2$. To help visualize it, you can think of the values taken by $X_1$ when $X_2$ takes high or low values. Contrary to categorical variables, here interaction is just represented by the product of $X_1$ and $X_2$. Of note, it's better to center your two variables first (so that the coefficient for say $X_1$ reads as the effect of $X_1$ when $X_2$ is at its sample mean).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As kindly suggested by @whuber, an easy way to see how $X_1$ varies with $Y$ as a function of $X_2$ when an interaction term is included, is to write down the model $\mathbb{E}(Y|X)=\beta_0+\beta_1X_1+\beta_2X_2+\beta_3X_1X_2$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, it can be seen that the effect of a one-unit increase in $X_1$ when $X_2$ is held constant may be expressed as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;\begin{eqnarray*}&#xD;&#xA;\mathbb{E}(Y|X_1+1,X_2)-\mathbb{E}(Y|X_1,X_2)&amp;amp;=&amp;amp;\beta_0+\beta_1(X_1+1)+\beta_2X_2+\beta_3(X_1+1)X_2\\&#xD;&#xA;&amp;amp;&amp;amp;-\big(\beta_0+\beta_1X_1+\beta_2X_2+\beta_3X_1X_2\big)\\&#xD;&#xA;&amp;amp;=&amp;amp; \beta_1+\beta_3X_2&#xD;&#xA;\end{eqnarray*}&#xD;&#xA;$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Likewise, the effect when $X_2$ is increased by one unit while holding $X_1$ constant is $\beta_2+\beta_3X_1$. This demonstrates why it is difficult to interpret the effects of $X_1$ ($\beta_1$) and $X_2$ ($\beta_2$) in isolation. This will even be more complicated if both predictors are highly correlated. It is also important to keep in mind the linearity assumption that is being made in such a linear model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can have a look at &lt;a href=&quot;http://books.google.fr/books?id=LcWLUyXcmnkC&amp;amp;pg=PA130&amp;amp;lpg=PA130&amp;amp;dq=regression+interaction+between+two+continuous+variable&amp;amp;source=bl&amp;amp;ots=flkc0hSY1d&amp;amp;sig=QyoUq4jd0AJPZC7Px0JpAUZh3jg&amp;amp;hl=fr&amp;amp;ei=b27xTOTQEcGChQeAzJGmCw&amp;amp;sa=X&amp;amp;oi=book_result&amp;amp;ct=result&amp;amp;resnum=2&amp;amp;ved=0CB8Q6AEwATgK#v=onepage&amp;amp;q=regression%20interaction%20between%20two%20continuous%20variable&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;Multiple regression: testing and interpreting interactions&lt;/a&gt;, by Leona S. Aiken, Stephen G. West, and Raymond R. Reno (Sage Publications, 1996), for an overview of the different kind of interaction effects in multiple regression. (This is probably not the best book, but it's available through Google)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a toy example in R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(mvtnorm)&#xA;set.seed(101)&#xA;n &amp;lt;- 300                      # sample size&#xA;S &amp;lt;- matrix(c(1,.2,.8,0,.2,1,.6,0,.8,.6,1,-.2,0,0,-.2,1), &#xA;            nr=4, byrow=TRUE) # cor matrix&#xA;X &amp;lt;- as.data.frame(rmvnorm(n, mean=rep(0, 4), sigma=S))&#xA;colnames(X) &amp;lt;- c(&quot;x1&quot;,&quot;x2&quot;,&quot;y&quot;,&quot;x1x2&quot;)&#xA;summary(lm(y~x1+x2+x1x2, data=X))&#xA;pairs(X)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where the output actually reads:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Coefficients:&#xA;            Estimate Std. Error t value Pr(&amp;gt;|t|)    &#xA;(Intercept) -0.01050    0.01860  -0.565    0.573    &#xA;x1           0.71498    0.01999  35.758   &amp;lt;2e-16 ***&#xA;x2           0.43706    0.01969  22.201   &amp;lt;2e-16 ***&#xA;x1x2        -0.17626    0.01801  -9.789   &amp;lt;2e-16 ***&#xA;---&#xA;Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 &#xA;&#xA;Residual standard error: 0.3206 on 296 degrees of freedom&#xA;Multiple R-squared: 0.8828, Adjusted R-squared: 0.8816 &#xA;F-statistic: 743.2 on 3 and 296 DF,  p-value: &amp;lt; 2.2e-16 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And here is how the simulated data looks like:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/kmJdm.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To illustrate @whuber's second comment, you can always look at the variations of $Y$ as a function of $X_2$ at different values of $X_1$ (e.g., terciles or deciles); trellis displays are useful in this case. With the data above, we would proceed as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(Hmisc)&#xA;X$x1b &amp;lt;- cut2(X$x1, g=5) # consider 5 quantiles (60 obs. per group)&#xA;coplot(y~x2|x1b, data=X, panel = panel.smooth)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/uDcnV.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-28T11:40:27.143" LastActivityDate="2010-11-28T11:40:27.143" CommentCount="6" />
  <row Id="4966" PostTypeId="2" ParentId="4945" CreationDate="2010-11-27T20:58:14.037" Score="2" ViewCount="" Body="&lt;p&gt;If all your measurements are using the same units, then you've already addressed the scale problem; what's bugging you is degrees of freedom and precision of your estimates of standard deviation.  If you recast your problem as comparing &lt;em&gt;variances&lt;/em&gt;, then there are plenty of standard tests available.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For two independent samples, you can use the F test; its null distribution follows the (surprise) F distribution which is indexed by degrees of freedom, so it implicitly adjusts for what you're calling a scale problem.  If you're comparing more than two samples, either Bartlett's or Levene's test might be suitable.  Of course, these have the same problem as one-way ANOVA, they don't tell you which variances differ significantly.  However, if, say, Bartlett's test did identify inhomogeneous variances, you could do follow-up pairwise comparisons with the F test and make a Bonferroni adjustment to maintain your experimentwise Type I error (alpha).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can get details for all of this stuff in the &lt;a href=&quot;http://www.itl.nist.gov/div898/handbook/&quot; rel=&quot;nofollow&quot;&gt;NIST/SEMATECH e-Handbook of Statistical Methods&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1920" LastActivityDate="2010-11-27T20:58:14.037" CommentCount="1" />
  <row Id="4967" PostTypeId="2" ParentId="134" CreationDate="2010-11-27T21:45:52.897" Score="3" ViewCount="" Body="&lt;p&gt;If you maintain a length-k window of data as a sorted doubly linked list then, by means of a binary search (to insert each new element as it gets shifted into the window) and a circular array of pointers (to immediately locate elements that need to be deleted), each shift of the window requires O(log(k)) effort for inserting one element, only O(1) effort for deleting the element shifted out of the window, and only O(1) effort to find the median (because every time one element is inserted or deleted into the list you can update a pointer to the median in O(1) time).  The total effort for processing an array of length N therefore is O((n-k)log(k)) &amp;lt;= O(n log(k)).  This is better than any of the other methods proposed so far and it is not an approximation, it is exact.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-11-27T21:45:52.897" CommentCount="3" />
  <row Id="4968" PostTypeId="2" ParentId="423" CreationDate="2010-11-27T22:27:41.770" Score="13" ViewCount="" Body="&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/u5HhK.png&quot; alt=&quot;http://imgs.xkcd.com/comics/extrapolating.png&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;By the third trimester, there will be hundreds of babies inside you.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Also from &lt;a href=&quot;http://xkcd.com/605/&quot; rel=&quot;nofollow&quot;&gt;XKCD&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2166" LastEditorUserId="442" LastEditorDisplayName="" LastEditDate="2010-11-29T14:53:40.757" LastActivityDate="2010-11-29T14:53:40.757" CommunityOwnedDate="2010-11-27T22:27:41.770" />
  <row Id="4970" PostTypeId="1" CreationDate="2010-11-28T04:38:02.957" Score="2" ViewCount="187" Body="&lt;p&gt;The matrix could be as large as $2500\times 2500$, what is the best algorithm to do that, is there some algorithm that is easy to write a program, is there any convenient packages for that?&lt;/p&gt;&#xA;" OwnerUserId="2141" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-28T11:55:30.167" LastActivityDate="2010-11-28T16:10:35.137" Title="How to diagonalize a large sparse symmetric matrix, to get the eigen values and eigenvectors?" Tags="&lt;algorithms&gt;&lt;matrix-decomposition&gt;" AnswerCount="3" />
  <row Id="4971" PostTypeId="2" ParentId="4970" CreationDate="2010-11-28T05:09:44.260" Score="4" ViewCount="" Body="&lt;p&gt;I don't know much about eigenvalues or what they are applicable to, but R seems to have a built in function for this purpose named &lt;code&gt;eigen()&lt;/code&gt;. Calculating the eigenvalues &amp;amp; eigenvectors for a 2500 * 2500 matrix took ~ 1 minute on my machine.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; sampData &amp;lt;- runif(6250000, 0, 2)&#xA;&amp;gt; x &amp;lt;- matrix(sampData, ncol = 2500, byrow = TRUE)&#xA;&amp;gt; system.time(eigen(x))&#xA;   user  system elapsed &#xA;  79.74    2.90   65.69 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This question has also come up on &lt;a href=&quot;http://stackoverflow.com/questions/713878/how-expensive-is-it-to-compute-the-eigenvalues-of-a-matrix&quot;&gt;Stack Overflow&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="696" LastActivityDate="2010-11-28T05:09:44.260" />
  <row Id="4972" PostTypeId="1" AcceptedAnswerId="4973" CreationDate="2010-11-28T06:29:15.097" Score="5" ViewCount="358" Body="&lt;p&gt;I am a novice in statistics so please correct me if I am doing something fundamentally wrong. After wrestling for a long time with R in trying to fit my data to a good distribution, I figured out that it fits the Cauchy distribution with the following parameters: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   location      scale  &#xA;  37.029894   18.678936 &#xA; ( 3.405665) ( 2.779136)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The data was from a survey where 100 people were asked how many friends they talked to over a period of 20 days and I am trying to see if it fits a known distribution. I generated the QQ-plot with the reference line and it looks like the image given below. From what I have been reading on the web, if the points fall close to the reference line then it is a good evidence that the data comes from this distribution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, is this a good evidence to say that the distribution is Cauchy or do I need to run any more tests? If so, can someone tell me the physical interpretation of this result? I mean, I read that if the data falls into a Cauchy distribution, then it will not have a mean and standard deviation but can someone help me understand this in plain English? If it does not have a mean then from what I understand, I cannot sample from this distribution. What is one supposed to infer about the population based on this result? Or should I be looking at other models?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE: What am I trying to achieve?&lt;/strong&gt;&#xA;I am trying to evaluate how much time it takes for some arbitrary piece of information to propagate for a population of size X. As this depends on the communication patterns of people, what I was trying to do was to build a model that could use the information from the 100 people I surveyed to give me patterns for the X number where X could be 500 or 1000. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;QQ-Plot&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/HvmjM.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Density Distribution of my data&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/Ob7hS.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cauchy Distribution&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/qGHmd.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;QQ-Plot when trying to fit a Normal distribution to my data&#xA;&lt;img src=&quot;http://i.imgur.com/hUubx.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From all the suggestions, I think I now understand why this cannot be a Cauchy distribution. Thanks to everyone. @HairyBeast suggested that I look at a negative binomial distribution so I plotted the following as well:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;QQ-Plot when Negative Binomial Distribution was used&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/DyHHV.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Negative Binomial Distribution&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/DPNLZ.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="2164" LastEditorUserId="2164" LastEditorDisplayName="" LastEditDate="2010-11-29T06:57:47.993" LastActivityDate="2010-11-29T06:57:47.993" Title="Fitting a model to my data" Tags="&lt;r&gt;&lt;distributions&gt;&lt;dataset&gt;" AnswerCount="2" CommentCount="10" FavoriteCount="4" />
  <row Id="4973" PostTypeId="2" ParentId="4972" CreationDate="2010-11-28T07:13:23.237" Score="10" ViewCount="" Body="&lt;p&gt;First off, your response variable is discrete. The Cauchy distribution is continuous. Second, your response variable is non-negative. The Cauchy distribution with the parameters you specified puts about 1/5 of its mass on negative values. Whatever you have been reading about the QQ norm plot is false. Points falling close to the line is evidence of normality, not evidence in favor of being Cauchy distributed (EDIT: Disregard these last 2 sentences; a QQ Cauchy plot - not a QQ norm plot - was used, which is fine.) The Poisson distribution, used for modeling count data, is inappropriate since the variance is much larger than the mean. The Binomial distribution is also inappropriate since theoretically, your response variable has no upper bound. I'd look into the negative binomial distribution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a final note, your data does not necessarily have to come from a well known, &quot;named&quot; distribution. It may have come from a mixture of distributions, or may have a &quot;true&quot; distribution whose mass function is not a nice transformation of x to P(X=x). Don't try too hard to &quot;force&quot; a distribution to the data.&lt;/p&gt;&#xA;" OwnerUserId="2144" LastEditorUserId="2144" LastEditorDisplayName="" LastEditDate="2010-11-28T22:37:06.110" LastActivityDate="2010-11-28T22:37:06.110" CommentCount="3" />
  <row Id="4974" PostTypeId="2" ParentId="3856" CreationDate="2010-11-28T08:03:09.037" Score="1" ViewCount="" Body="&lt;p&gt;Typical measures of autocorrelation, such as Moran's I, are global estimates of clumpiness and could be masked by a trend or by &quot;averaging&quot; of clumpiness. There are two ways you could handle this:&#xA;1) Use a local measure of autocorrelation - but the drawback is you don't get a single number for clumpiness.  An example of this would be Local Moran's I*&#xA;Here is a document (from a google search) that at least introduces the terms and gives some derivations&#xA;&lt;a href=&quot;http://onlinelibrary.wiley.com/doi/10.1111/0022-4146.00224/abstract&quot; rel=&quot;nofollow&quot;&gt;http://onlinelibrary.wiley.com/doi/10.1111/0022-4146.00224/abstract&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Use a statistic specifically geared towards point distributions and their clumpieness at various spatial scales, such as Ripley's K&#xA;&lt;a href=&quot;http://scholar.google.com/scholar?q=Ripley%27s+K&amp;amp;hl=en&amp;amp;as_sdt=0&amp;amp;as_vis=1&amp;amp;oi=scholart&quot; rel=&quot;nofollow&quot;&gt;http://scholar.google.com/scholar?q=Ripley%27s+K&amp;amp;hl=en&amp;amp;as_sdt=0&amp;amp;as_vis=1&amp;amp;oi=scholart&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="787" LastActivityDate="2010-11-28T08:03:09.037" CommentCount="1" />
  <row Id="4975" PostTypeId="2" ParentId="4972" CreationDate="2010-11-28T08:50:51.503" Score="6" ViewCount="" Body="&lt;p&gt;Agree with HairyBeast (+1) that Cauchy is not appropriate here (it's symmetric for one thing) and that negative binomial might well be better.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Disagree about QQ-plot though. You can do a QQ-plot for any distribution, not just normal. What you say about interpretation of a QQ-plot is correct, but note that 2 of your points lie very far indeed from the straight line.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the Cauchy's lack of moments: this doesn't affect sampling. Once you know the parameters of the distribution sampling from it is easy (as the quantile function has a closed form) and the lack of moments is irrelevant. But the fact that the Cauchy distribution doesn't even have a mean does indicate that it's inappropriate here, as clearly it &lt;em&gt;is&lt;/em&gt; meaningful to ask what's the expected number of friends with whom a person has a conversation in a 20-day period.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-11-28T08:50:51.503" CommentCount="3" />
  <row Id="4976" PostTypeId="2" ParentId="4970" CreationDate="2010-11-28T12:05:28.813" Score="4" ViewCount="" Body="&lt;p&gt;Take a look at &lt;a href=&quot;http://www.grycap.upv.es/slepc/documentation/reports/str6.pdf&quot; rel=&quot;nofollow&quot;&gt;A Survey of Software for Sparse Eigenvalue Problems&lt;/a&gt; by Hernández et al.&lt;/p&gt;&#xA;" OwnerUserId="439" LastEditorUserId="439" LastEditorDisplayName="" LastEditDate="2010-11-28T15:40:35.627" LastActivityDate="2010-11-28T15:40:35.627" />
  <row Id="4977" PostTypeId="2" ParentId="4961" CreationDate="2010-11-28T12:51:05.570" Score="1" ViewCount="" Body="&lt;p&gt;Put in simple terms, regularization is about benefiting the solutions you'd expect to get. As you mention, for example you can benefit &quot;simple&quot; solutions, for some definition of simplicity. If your problem has rules, one definition can be fewer rules. But this is problem-dependent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You're asking the right question, however. For example in Support Vector Machines this &quot;simplicity&quot; comes from breaking ties in the direction of &quot;maximum margin&quot;. This margin is something that can be clearly defined in terms of the problem. There is a very good geometric derivation in the &lt;a href=&quot;http://en.wikipedia.org/wiki/Support_vector_machine&quot; rel=&quot;nofollow&quot;&gt;SVM article in Wikipedia&lt;/a&gt;. It turns out that the regularization term is, arguably at least, the &quot;secret sauce&quot; of SVMs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do you do regularization? In general that comes with the method you use, if you use SVMs you're doing L2 regularization, if your using &lt;a href=&quot;http://en.wikipedia.org/wiki/Lasso_%28statistics%29#LASSO_method&quot; rel=&quot;nofollow&quot;&gt;LASSO you're doing L1 regularization&lt;/a&gt; (see what hairybeast is saying). However, if you're developing your own method, you need to know how to tell desirable solutions from non-desirable ones, and have a function that quantifies this. In the end you'll have a cost term and a regularization term, and you want to optimize the sum of both.&lt;/p&gt;&#xA;" OwnerUserId="1540" LastActivityDate="2010-11-28T12:51:05.570" />
  <row Id="4978" PostTypeId="1" AcceptedAnswerId="4984" CreationDate="2010-11-28T13:12:18.827" Score="10" ViewCount="304" Body="&lt;p&gt;I'm in no way a statistician (I've had a course in mathematical statistics but nothing more than that), and recently, while studying information theory and statistical mechanics, I met this thing called &quot;uncertainty measure&quot;/&quot;entropy&quot;. I read Khinchin derivation of it as a measure of uncertainty and it made sense to me. Another thing that made sense was Jaynes description of MaxEnt to get a statistic when you know the arithmetic mean of one or more function/s on the sample (assuming you accept $-\sum p_i\ln p_i$ as a measure of uncertainty of course).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I searched on the net to find the relationship with other methods of statistical inference, and God was I confused. For example &lt;a href=&quot;http://www.mdpi.org/entropy/papers/e3020058.pdf&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; paper suggest, assuming that i got it right, that you just get a ML estimator under a suitable reformulation of the problem; MacKey, in his book, says that MaxEnt can give you weird things, and you should't use it even for a starting estimate in a Bayesian inference; etc.. I'm having trouble finding good comparisons.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is, could you provide an explanation and/or good refences of weak and strong points of MaxEnt as a statistical inference method with quantitative comparisons to other methods (when applied to toy models for example)? &lt;/p&gt;&#xA;" OwnerUserId="2171" LastActivityDate="2011-02-17T16:03:19.943" Title="Comparison between MaxEnt, ML, Bayes and other kind of statistical inference methods" Tags="&lt;entropy&gt;&lt;inference&gt;" AnswerCount="3" FavoriteCount="7" />
  <row Id="4979" PostTypeId="2" ParentId="4978" CreationDate="2010-11-28T15:27:54.147" Score="10" ViewCount="" Body="&lt;p&gt;For an entertaining critique of maximum entropy methods, I'd recommend reading some old newsgroup posts on sci.stat.math and sci.stat.consult, particularly the ones by Radford Neal:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://groups.google.com/group/sci.stat.consult/browse_thread/thread/e3db790d034d5d59/59efc35f440767ee?hl=en&amp;amp;ie=UTF-8&quot; rel=&quot;nofollow&quot;&gt;How informative is the Maximum Entropy method? (1994)&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://groups.google.com/group/sci.stat.math/browse_thread/thread/4f4bc100752d4855/9636b6fe1ba893f1?hl=en&amp;amp;ie=UTF-8&quot; rel=&quot;nofollow&quot;&gt;Maximum Entropy Imputation  (2002)&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://groups.google.com/group/sci.stat.consult/browse_thread/thread/5d57a0a01704971b/2cf57ceb8ec46e0f?hl=en&amp;amp;ie=UTF-8&quot; rel=&quot;nofollow&quot;&gt;Explanation of Maximum Entropy  (2004)&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I'm not aware of any comparisons between maxent and other methods: part of the problem seems to be that maxent is not really a framework, but an ambiguous directive (&quot;when faced with an unknown, simply maximise the entropy&quot;), which is interpreted in different ways by different people.&lt;/p&gt;&#xA;" OwnerUserId="495" LastEditorUserId="495" LastEditorDisplayName="" LastEditDate="2010-11-28T22:05:12.887" LastActivityDate="2010-11-28T22:05:12.887" CommentCount="2" />
  <row Id="4980" PostTypeId="1" CreationDate="2010-11-28T15:34:13.590" Score="2" ViewCount="141" Body="&lt;p&gt;What open-source implementations -- in any language -- exist out there that can compute lasso regularisation paths for linear regression by coordinate descent?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far I am aware of:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://cran.r-project.org/web/packages/glmnet/index.html&quot; rel=&quot;nofollow&quot;&gt;glmnet&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://scikit-learn.sourceforge.net/&quot; rel=&quot;nofollow&quot;&gt;scikits.learn&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Anything else out there?&lt;/p&gt;&#xA;" OwnerUserId="439" LastActivityDate="2010-11-29T10:18:48.507" Title="Lasso fitting by coordinate descent: open-source implementations?" Tags="&lt;regression&gt;&lt;lasso&gt;&lt;regularization&gt;" AnswerCount="1" />
  <row Id="4981" PostTypeId="2" ParentId="4766" CreationDate="2010-11-28T15:54:33.683" Score="1" ViewCount="" Body="&lt;p&gt;We have implemented this (along with a power iteration refinement) in the &lt;a href=&quot;http://scikit-learn.sourceforge.net&quot; rel=&quot;nofollow&quot;&gt;scikit-learn&lt;/a&gt; python package.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Our &lt;a href=&quot;https://github.com/scikit-learn/scikit-learn/blob/master/scikits/learn/utils/extmath.py#L97&quot; rel=&quot;nofollow&quot;&gt;implementation&lt;/a&gt; is able to find the exact same singular values and vectors if k + p &gt; rank(M) as demonstrated in the &lt;a href=&quot;https://github.com/scikit-learn/scikit-learn/blob/master/scikits/learn/utils/tests/test_svd.py#L14&quot; rel=&quot;nofollow&quot;&gt;tests&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you cut (k + p) before reaching near zero singular values (i.e. in the k + p &amp;lt; rank(M) case) then the singular vectors are indeed different from the ones you get with the un-truncated version but they might still be very useful in practice for features extraction in machine learning: for instance 'truncated' eigenfaces at 150 work as good for face recognition task with SVM as the top 150 first singular vectors of the full decomposition even though the rank of my faces dataset seems to be much higher.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This randomized / truncated SVD method looks really interesting in practice: it can really cut down the computation time as shown in this &lt;a href=&quot;https://github.com/scikit-learn/scikit-learn/blob/fc5da45c9e94bc18a55cd592a70bf1c21da75852/benchmarks/plot_bench_svd.py&quot; rel=&quot;nofollow&quot;&gt;benchmark&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/uBBar.png&quot; alt=&quot;comparing randomized andv deterministi SVD implementations&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="2150" LastEditorUserId="2150" LastEditorDisplayName="" LastEditDate="2010-11-28T16:01:55.277" LastActivityDate="2010-11-28T16:01:55.277" />
  <row Id="4982" PostTypeId="2" ParentId="4970" CreationDate="2010-11-28T16:10:35.137" Score="1" ViewCount="" Body="&lt;p&gt;2500x2500 is not such a large problem. Even without leveraging the sparsity the SVD implementation of scipy.linalg is able to decompose it in less than a minute. See &lt;a href=&quot;http://stats.stackexchange.com/questions/4766/randomized-svd-and-singular-values/4981#4981&quot;&gt;my answer&lt;/a&gt; to a related questions for more details.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For larger problems you will need to use the sparsity explicitly. The &lt;a href=&quot;http://nlp.fi.muni.cz/projekty/gensim/&quot; rel=&quot;nofollow&quot;&gt;gensim&lt;/a&gt; project my help you for middle size problems that fit on a single computer but not in RAM and the &lt;a href=&quot;http://mahout.apache.org&quot; rel=&quot;nofollow&quot;&gt;mahout&lt;/a&gt; implementation is able to deal with sparse matrices that don't even fit on a single hard-drive.&lt;/p&gt;&#xA;" OwnerUserId="2150" LastActivityDate="2010-11-28T16:10:35.137" />
  <row Id="4983" PostTypeId="1" AcceptedAnswerId="5053" CreationDate="2010-11-28T16:13:12.200" Score="1" ViewCount="295" Body="&lt;p&gt;I am using &lt;a href=&quot;http://svmlight.joachims.org/&quot; rel=&quot;nofollow&quot;&gt;SVM-light&lt;/a&gt; with Matlab, for linear SVM. I would like to understand the output model, but I cannot find any documentation or help about it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the output:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;        sv_num: 639&#xA;   upper_bound: 547&#xA;             b: 1.4023&#xA;      totwords: 576&#xA;        totdoc: 2000&#xA;     loo_error: -1&#xA;    loo_recall: -1&#xA; loo_precision: -1&#xA;      xa_error: 14.9500&#xA;     xa_recall: 82.6000&#xA;  xa_precision: 86.8559&#xA;       maxdiff: 9.9611e-004&#xA;    r_delta_sq: 575.0052&#xA;   r_delta_avg: 23.9792&#xA;  model_length: 0.4847&#xA;          loss: 357.5542&#xA;         vcdim: 136.1071&#xA;         alpha: [2000x1 double]&#xA;   lin_weights: []&#xA;         index: [2000x1 double]&#xA;        supvec: [639x576 double]&#xA;   kernel_parm: [1x1 struct]&#xA;example_length: 23.9792&#xA;             a: [2000x1 double]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So far, I understand there are &lt;code&gt;sv_num=639&lt;/code&gt; support vectors. However, there are 638 positive index in the &lt;code&gt;index&lt;/code&gt; vector ; ditto for the positive values of the &lt;code&gt;alpha&lt;/code&gt; vector. The first value of the &lt;code&gt;alpha&lt;/code&gt; vector is zero, and the non-zero values are then between 2 and 639. Same remark about the &lt;code&gt;supvec&lt;/code&gt; vector. So it seems there are indeed 638 support vectors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Actually, I would like to get the hyperplane &lt;code&gt;W&lt;/code&gt; and the parameter &lt;code&gt;b&lt;/code&gt; myself to see if I could understand the output model correctly. Since the kernel is linear, I thought of a sum over some coefficients times some support vectors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For information, &lt;code&gt;a(i)&lt;/code&gt; &quot;should&quot; store the values &lt;code&gt;alpha_i&lt;/code&gt; times &lt;code&gt;y_i&lt;/code&gt; for training samples &lt;code&gt;x_i&lt;/code&gt;. And there are 638 non-zero values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Has anyone used svmlight before? Can you understand the output model? Or is there a documentation anywhere?&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;I manage to get one thing right:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;MyCoeff = model.a ./ y;&#xA;MyIndex = 1+max(model.index,0); % between 1 and 639&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The support vector corresponding to any non-zero &lt;code&gt;MyCoeff(i)&lt;/code&gt; is:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;x(i,:)&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;model.supvec(MyIndex(i),:)&lt;/code&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;And &lt;code&gt;MyIndex&lt;/code&gt; is 1 iff &lt;code&gt;MyCoeff&lt;/code&gt; is zero. And &lt;code&gt;supvec(1,:)&lt;/code&gt; is the zero vector.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Here are four pieces of code I have made:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;MyCoeff = model.a ./ y;&#xA;W1 = zeros(1, h*w);&#xA;for i=1:length(MyCoeff)&#xA;    W1 = W1 + MyCoeff(i)*x(i,:);&#xA;end;&#xA;&#xA;W2 = zeros(1, h*w);&#xA;for i=1:model.sv_num&#xA;    W2 = W2 + model.alpha(i)*model.supvec(i,:);&#xA;end;&#xA;&#xA;eq((MyCoeff~=0),(model.index&amp;gt;0))&#xA;MesIndices = 1+max(model.index,0); % entre 1 et 639&#xA;W3 = zeros(1, h*w);&#xA;for i=1:length(MyCoeff)&#xA;    W3 = W3 + MyCoeff(i)*model.supvec(MesIndices(i),:);&#xA;end;&#xA;&#xA;W4 = zeros(1, h*w);&#xA;for i=1:length(MyCoeff)&#xA;    W4 = W4 +  model.alpha(MesIndices(i))*x(i,:);&#xA;end;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here are the outputs:&#xA;&lt;img src=&quot;http://i.imgur.com/A51VP.png&quot; alt=&quot;http://imgur.com/FErWW.png&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have understood the &lt;code&gt;index&lt;/code&gt; link between the support vectors &lt;code&gt;supvec&lt;/code&gt; and the matrix &lt;code&gt;x&lt;/code&gt;. But I have not found the link between &lt;code&gt;alpha&lt;/code&gt; and &lt;code&gt;a&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;I got it: apart from the difference of indices, &lt;code&gt;model.a&lt;/code&gt; and &lt;code&gt;model.alpha&lt;/code&gt; are the same. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for i=1:length(MesIndices)&#xA;    W1 = W1 + model.a(i)*x(i,:);%*y(i);&#xA;    W2 = W2 + model.alpha(MesIndices(i))*model.supvec(MesIndices(i),:);%*y(i);&#xA;    W3 = W3 + model.a(i)*model.supvec(MesIndices(i),:);%*y(i);&#xA;    W4 = W4 +  model.alpha(MesIndices(i))*x(i,:);%*y(i);&#xA;end;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The question now is whether I should multiply by &lt;code&gt;y(i)&lt;/code&gt;. I believe I should NOT since &lt;code&gt;model.a&lt;/code&gt; is already &lt;code&gt;y&lt;/code&gt; times the weight of the corresponding support vector, and $y\in\{-1,1\}$. So the good result should be the ORANGE face, which you can get by using one of the four lines above. Too bad I could not find any documentation. :')&lt;/p&gt;&#xA;" OwnerUserId="1351" LastEditorUserId="1351" LastEditorDisplayName="" LastEditDate="2010-11-28T18:56:12.620" LastActivityDate="2010-12-01T13:23:03.733" Title="Output of linear SVM model in Matlab using SVM-light" Tags="&lt;matlab&gt;&lt;svm&gt;&lt;output&gt;" AnswerCount="1" />
  <row Id="4984" PostTypeId="2" ParentId="4978" CreationDate="2010-11-28T22:05:16.413" Score="9" ViewCount="" Body="&lt;p&gt;MaxEnt and Bayesian inference methods correspond to different ways of incorporating information into your modeling procedure. Both can be put on axiomatic ground (John Skilling's &lt;a href=&quot;http://yaroslavvb.com/papers/skilling-axioms.pdf&quot; rel=&quot;nofollow&quot;&gt;&quot;Axioms of Maximum Entropy&quot;&lt;/a&gt; and Cox's &lt;a href=&quot;http://yaroslavvb.com/papers/cox-algebra.pdf&quot; rel=&quot;nofollow&quot;&gt;&quot;Algebra of Probable Inference&quot;&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bayesian approach is straightforward to apply if your prior knowledge comes in a form of a measurable real-valued function over your hypothesis space, so called &quot;prior&quot;. MaxEnt is straightforward when the information comes as a set of hard constraints on your hypothesis space. In real life, knowledge comes neither in &quot;prior&quot; form nor in &quot;constraint&quot; form, so success of your method depends on your ability to represent your knowledge in the corresponding form. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On a toy problem, Bayesian model averaging will give you &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.27.153&quot; rel=&quot;nofollow&quot;&gt;lowest&lt;/a&gt; average log-loss (averaged over many model draws) when the prior matches the true distribution of hypotheses. MaxEnt approach will give you &lt;a href=&quot;http://www.cwi.nl/~pdg/ftp/AOS231.pdf&quot; rel=&quot;nofollow&quot;&gt;lowest&lt;/a&gt; worst-case log-loss when its constraints are satisfied (worst taken over all possible priors)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;E.T.Jaynes, considered a father of &quot;MaxEnt&quot; methods also relied on Bayesian methods. On &lt;a href=&quot;http://omega.albany.edu:8008/ETJ-PS/cc14g.ps&quot; rel=&quot;nofollow&quot;&gt;page 1412&lt;/a&gt; of his &lt;a href=&quot;http://omega.albany.edu:8008/JaynesBook.html&quot; rel=&quot;nofollow&quot;&gt;book&lt;/a&gt;, he gives an example where Bayesian approach resulted in a good solution, followed by an example where MaxEnt approach is more natural.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maximum likelihood essentially takes the model to lie inside some pre-determined model space and trying to fit it &quot;as hard as possible&quot; in a sense that it'll have the highest sensitivity to data out of all model-picking methods restricted to such model space. Whereas MaxEnt and Bayesian are frameworks, ML is a concrete model fitting method, and for some particular design choices, ML can end up the method coming out of Bayesian or MaxEnt approach. For instance, MaxEnt with equality constraints is equivalent to Maximum Likelihood fitting of a certain exponential family. Similarly, an approximation to Bayesian Inference can lead to regularized Maximum Likelihood solution. If you choose your prior to make your conclusions maximally sensitive to data, result of Bayesian inference will correspond to Maximum Likelihood fitting. For instance, when inferring $p$ over Bernoulli trials, such prior &lt;a href=&quot;http://www.amstat.org/publications/jse/v12n2/zhu.pdf&quot; rel=&quot;nofollow&quot;&gt;would be&lt;/a&gt; the limiting distribution Beta(0,0)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Real-life Machine Learning successes are often a mix of various philosophies. For instance, &quot;Random Fields&quot; were &lt;a href=&quot;http://www.stat.ucla.edu/~yuille/courses/Stat231-Fall08/Pietra.pdf&quot; rel=&quot;nofollow&quot;&gt;derived&lt;/a&gt; from MaxEnt principles. Most popular implementation of the idea, regularized CRF, involves adding a &quot;prior&quot; on the parameters. As a result, the method is not really MaxEnt nor Bayesian, but  influenced by both schools of thought.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've collected some links on philosophical foundations of Bayesian and MaxEnt approaches &lt;a href=&quot;http://yaroslavvb.blogspot.com/2010/09/maxent-or-bayesian.html&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;http://yaroslavvb.blogspot.com/2005/03/joy-of-scanning.html&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note on terminology: sometimes people call their method Bayesian &lt;a href=&quot;http://www.google.com/search?q=%22not+really+bayesian%22&quot; rel=&quot;nofollow&quot;&gt;simply if&lt;/a&gt; it uses Bayes rule at some point. Likewise, &quot;MaxEnt&quot; is sometimes used for some method that favors high entropy solutions. This is not the same as &quot;MaxEnt inference&quot; or &quot;Bayesian inference&quot; as described above&lt;/p&gt;&#xA;" OwnerUserId="511" LastEditorUserId="511" LastEditorDisplayName="" LastEditDate="2010-11-29T02:32:55.783" LastActivityDate="2010-11-29T02:32:55.783" CommentCount="1" />
  <row Id="4985" PostTypeId="2" ParentId="4908" CreationDate="2010-11-28T23:09:43.057" Score="0" ViewCount="" Body="&lt;p&gt;I may be misunderstanding your question (and falling into the same misunderstanding as fRed despite your explanation), in which case I apologize.  It seems to me like you are saying you already know the various P_A, P_B, and P_C values for each context.  I assume P_A + P_B + P_C = 1? Given those priors and the actual outcome you want to characterize the accuracy of the already established P values?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One approach might be simulation.  Given the various P_A, P_B, and P_C values for each context you could generate guesses from the model at the probabilities indicated by each P value and then compare that to the obtained data.  Averaging over many simulations you should get a sense of the average hit-rate of the model.&lt;/p&gt;&#xA;" OwnerUserId="196" LastActivityDate="2010-11-28T23:09:43.057" />
  <row Id="4986" PostTypeId="2" ParentId="4980" CreationDate="2010-11-29T00:23:57.847" Score="2" ViewCount="" Body="&lt;p&gt;I have a MATLAB and C/C++ implementation &lt;a href=&quot;http://www.emakalic.org/blog/?page_id=7&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let me know if you find it useful.&lt;/p&gt;&#xA;" OwnerUserId="530" LastActivityDate="2010-11-29T00:23:57.847" CommentCount="1" />
  <row Id="4987" PostTypeId="1" AcceptedAnswerId="4990" CreationDate="2010-11-29T01:02:22.203" Score="2" ViewCount="420" Body="&lt;p&gt;I must have generated at least 5 Q-Q plots until now when trying to fit my data into a known distribution but I just noticed something that I could not understand. In the figure shown below, from what I've read from the wiki, X-axis is supposed to read &quot;Negative Binomial Theoretical Quantiles&quot; and Y-axis is supposed to read &quot;Data quantiles&quot;. Agreed that this makes perfect sense. But when I looked at the figure, the X and Y axis go beyond 100 but how can there be quantiles beyond 100? What do they mean if they exist? Or is this graph produced by the qqplot of R totally different? Can someone help me understand this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The way I was generating this data was using the following script:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(MASS)&#xA;&#xA;# Define the data&#xA;data &amp;lt;- c(67, 81, 93, 65, 18, 44, 31, 103, 64, 19, 27, 57, 63, 25, 22, 150,&#xA;          31, 58, 93, 6, 86, 43, 17, 9, 78, 23, 75, 28, 37, 23, 108, 14, 137,&#xA;          69, 58, 81, 62, 25, 54, 57, 65, 72, 17, 22, 170, 95, 38, 33, 34, 68, &#xA;          38, 117, 28, 17, 19, 25, 24, 15, 103, 31, 33, 77, 38, 8, 48, 32, 48, &#xA;          26, 63, 16, 70, 87, 31, 36, 31, 38, 91, 117, 16, 40, 7, 26, 15, 89, &#xA;          67, 7, 39, 33, 58)&#xA;&#xA;# Fit the data to a model&#xA;params = fitdistr(data, &quot;Negative Binomial&quot;)&#xA;&#xA;#using the answer from params create a set of theoretical values&#xA;plot(qnbinom(ppoints(data), size=2.3539444, mu=50.7752809), sort(data))&#xA;abline(0,1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/QeB2Y.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="2164" LastEditorUserId="2164" LastEditorDisplayName="" LastEditDate="2010-11-29T01:14:51.370" LastActivityDate="2010-12-17T06:44:15.940" Title="Understanding a Quantile-Quantile Plot" Tags="&lt;r&gt;&lt;distributions&gt;&lt;basic-concepts&gt;&lt;quantiles&gt;" AnswerCount="2" CommentCount="4" />
  <row Id="4988" PostTypeId="1" CreationDate="2010-11-29T02:08:24.257" Score="4" ViewCount="151" Body="&lt;p&gt;Suppose we have a distribution for  $\mathbf{x}\in \{-1,1\}^n$ and are interested in representing $p(\mathbf{x})$ as a linear exponential family with sufficient statistics of the form $1,x_1,x_2,\ldots,x_1 x_2,\ldots,x_1 x_2 \cdots x_n$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose we know that &lt;strike&gt;entropy&lt;/strike&gt; some measure of complexity of target distribution is bounded. We can view the parameter vector as a probability distribution $D$. Can we say anything about how the mass in $D$ is distributed among terms of various orders for some interesting complexity measure? Are there complexity measures that force mass onto low-order terms?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Restricting attention to graphical models means discarding 3rd and higher order terms. I'm interested in some explanation why this works so well&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This was motivated by a similar &lt;a href=&quot;http://cstheory.stackexchange.com/q/3332/434&quot;&gt;question&lt;/a&gt; on CSTheory&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;b&gt;Edit 11/29&lt;/b&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Below is the probability simplex of of distributions over $\{-1,1\}$ valued $x_1,x_2$, in &quot;probability&quot; coordinates and in &quot;log-linear&quot; coordinates. In second picture, x,y axes correspond to coefficients in front of $x_1$ and $x_2$, z axis is the coefficient of $x_1 x_2$ term. You can see first and second order dimensions are symmetric, so bounding entropy (or any other symmetric function of distribution) is not enough to force mass onto low-order coefficients....&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://yaroslavvb.com/upload/save/stats.SE.simplex.png&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Mathematica &lt;a href=&quot;http://yaroslavvb.com/upload/save/stats.SE.simplex.nb&quot; rel=&quot;nofollow&quot;&gt;source&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="511" LastEditorUserId="511" LastEditorDisplayName="" LastEditDate="2010-11-29T23:00:28.080" LastActivityDate="2010-11-29T23:00:28.080" Title="Asymmetry between high order and low order interaction terms" Tags="&lt;model-selection&gt;&lt;entropy&gt;&lt;graphical-model&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="4989" PostTypeId="1" AcceptedAnswerId="5248" CreationDate="2010-11-29T02:23:14.403" Score="3" ViewCount="231" Body="&lt;p&gt;My data consists of individual level observations nested within countries over time. I would like to use multilevel models along with some sort of selection model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have three related questions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Are there any issues or concerns with using a Heckman selection model twice? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a model with two selection stages. While I can't conceive of any reason why this shouldn't work, I also haven't seen any examples of a heckman selection model being used twice. Assume I also have the necessary exclusion restrictions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) My primary tool is R. There exists a package, sampleSelection, but my understanding is that algorithm it uses may not be appropriate for panel data / multilevel models. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In stata, my understanding is the aforementioned is possible, but I'm not sure a package in R exists for doing heckman selection with multilevel models. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) Are there better ways for accomplishing what I am trying to do? Perhaps using instruments alone would be better? If so, are there any R packages you could point me to? If there are no R packages, I am still interested in this answer since I would be happy to code something if necessary. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you. &lt;/p&gt;&#xA;" OwnerUserId="2050" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-11-29T12:42:38.483" LastActivityDate="2010-12-08T10:22:15.833" Title="Multi-stage selection model with panel data in R" Tags="&lt;r&gt;&lt;panel-data&gt;&lt;multilevel-analysis&gt;" AnswerCount="1" />
  <row Id="4990" PostTypeId="2" ParentId="4987" CreationDate="2010-11-29T06:04:07.037" Score="4" ViewCount="" Body="&lt;p&gt;Hello Legend:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think R is doing perfectly what you want it to do.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You are plotting:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;x = qnbinom(ppoints(data),&#xA;  size=2.3539444, mu=50.7752809)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;which is:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;[1]   3   5   7   9  10  11  12  13 &#xA;  14  15  16  17  18  19  20 [16]  21 &#xA;  21  22  23  24  25  25  26  27  28  28&#xA;  29  30  31  31 [31]  32  33  34  35 &#xA;  35  36  37  38  39  39  40  41  42  43&#xA;  44 [46]  45  45  46  47  48  49  50 &#xA;  51  52  53  54  55  56  57  59 [61] &#xA;  60  61  62  63  65  66  68  69  71  72&#xA;  74  76  77  79  81 [76]  84  86  89 &#xA;  91  94  97 101 105 110 116 123 132 146&#xA;  175&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;with respect to &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;y = sort(data)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;which is:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;[1]   6   7   7   8   9  14  15  15 &#xA;  16  16  17  17  17  18  19 [16]  19 &#xA;  22  22  23  23  24  25  25  25  26  26&#xA;  27  28  28  31 [31]  31  31  31  31 &#xA;  32  33  33  33  34  36  37  38  38  38&#xA;  38 [46]  39  40  43  44  48  48  54 &#xA;  57  57  58  58  58  62  63  63 [61] &#xA;  64  65  65  67  67  68  69  70  72  75&#xA;  77  78  81  81  86 [76]  87  89  91 &#xA;  93  93  95 103 103 108 117 117 137 150&#xA;  170&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Therefore, you have 100+ values on both the axis. If you want to plot quantiles, you need to tell R to do so by doing this:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;plot(pnbinom(sort(data), size=2.3539444, mu=50.7752809), ppoints(data))&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="1307" LastEditorUserId="1307" LastEditorDisplayName="" LastEditDate="2010-11-29T06:15:40.893" LastActivityDate="2010-11-29T06:15:40.893" CommentCount="2" />
  <row Id="4991" PostTypeId="1" CreationDate="2010-11-29T07:43:32.537" Score="5" ViewCount="280" Body="&lt;p&gt;Hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to implement (in R) the following very simple Dynamic Linear Model for which I have 2 &lt;strong&gt;unknown time varying parameters&lt;/strong&gt; (the variance of the observation error $\epsilon^1_t$ and the variance of the state error $\epsilon^2_t$).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$  &#xD;&#xA;\begin{matrix}  &#xD;&#xA;Y_t &amp;amp; = &amp;amp; \theta_t + \epsilon^1_t\\     &#xD;&#xA;\theta_{t+1} &amp;amp; = &amp;amp; \theta_{t}+\epsilon^2_t  &#xD;&#xA;\end{matrix}  $&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to estimate these parameters for each time, &lt;strong&gt;without any look ahead bias&lt;/strong&gt;.&#xA;From what I understand, I can use either a MCMC (on a rolling window to avoid the look ahead bias), or a particule filter (or Sequencial Monte Carlo - SMC).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Which method would you use&lt;/strong&gt;, and&lt;br&gt;&#xA;&lt;strong&gt;What are the pros and cons of these two methods?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bonus question: In these methods, how do you select the speed of change of the parameters? I guess we have to input an information here, because there is a bargain between using a lot of data to estimate the parameters and using less data to react more quickly to a change in the parameter?&lt;/p&gt;&#xA;" OwnerUserId="1709" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-14T21:37:07.510" LastActivityDate="2010-12-14T21:58:57.510" Title="Estimating parameters of a dynamic linear model" Tags="&lt;r&gt;&lt;mcmc&gt;&lt;dlm&gt;&lt;particle-filter&gt;" AnswerCount="2" CommentCount="5" FavoriteCount="3" />
  <row Id="4993" PostTypeId="2" ParentId="4961" CreationDate="2010-11-29T10:32:29.997" Score="1" ViewCount="" Body="&lt;p&gt;Suppose you perform learning via empirical risk minimization. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;More precisely:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;you have got your non-negative  loss function $L(actualvalue, predictedvalue)$ which characterize how bad your predictions are&lt;/li&gt;&#xA;&lt;li&gt;you want to fit your model in a such way that its predictions minimize mean of loss function, calculated only on training data (the only data you have)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Then the aim of learning process is to find $Model = argmin \sum L(actual, predicted(Model))$ (this method is called empirical risk minimization)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But if you haven't got enough data and there is a huge amount of variables in your model, it is very probable to find such a model that not only explain patterns but also explains random noise in your data. This effect is called overfitting and it leads to degradation of generalization ability of your model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to avoid overfitting a regularization term is introduced into the target function:&#xA;$Model = argmin \sum L(actual, predicted(Model)) + \lambda R(Model)$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Usually, this term $R(Model)$ imposes a special penalty on complex models. For instance, on models with large coefficients (L2 regularization, R=sum of squares of coefficients) or with a lot if non-zero coefficients (L1 regularization, R=sum of absolute values of coefficients). If we are training decision tree, R can be its depth.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another point of view is that R introduces our prior knowledge about a form of the best model (&quot;it doesn't have too large coefficients&quot;, &quot;it is almost orthogonal to $\bar a$&quot;)&lt;/p&gt;&#xA;" OwnerUserId="1725" LastEditorUserId="1725" LastEditorDisplayName="" LastEditDate="2010-11-29T10:46:54.530" LastActivityDate="2010-11-29T10:46:54.530" />
  <row Id="4994" PostTypeId="1" CreationDate="2010-11-29T13:08:53.283" Score="0" ViewCount="102" Body="&lt;p&gt;Hi guys,&lt;br&gt;&#xA;I've got the point that bayesian inference is hard to compute and so some approximations are needed and I've got the normal EM (but I think it is not truly bayesian approach cause parameters are not treated as random variables), but I'm in trouble understanding variational Bayes EM.&lt;br&gt;&#xA;I've found some materials but some physical terms like free energy and entropy makes me confused.&lt;br&gt;&#xA;Anyone can help me?&lt;/p&gt;&#xA;" OwnerUserId="2046" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-11-29T15:12:04.560" LastActivityDate="2010-11-29T15:12:04.560" Title="Understanding variational Bayes EM" Tags="&lt;bayesian&gt;&lt;em-algorithm&gt;" CommentCount="1" FavoriteCount="1" />
  <row Id="4995" PostTypeId="2" ParentId="4945" CreationDate="2010-11-29T15:19:15.583" Score="0" ViewCount="" Body="&lt;p&gt;Harvey:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You're absolutely right that the F and Bartlett's tests won't work to compare raw data with smoothed data!  Once the data has been smoothed, there's all manner of autocorrelation in there, and the testing becomes much more complicated.   Better to compare separate--and hopefully independent--sequences.&lt;/p&gt;&#xA;" OwnerUserId="1920" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-11-29T16:53:58.290" LastActivityDate="2010-11-29T16:53:58.290" CommentCount="2" />
  <row Id="4996" PostTypeId="2" ParentId="4988" CreationDate="2010-11-29T15:51:33.780" Score="3" ViewCount="" Body="&lt;p&gt;Under these assumptions any permutation of the monomials leads to exactly the same results, implying there is no inherent distinction between the low order terms and other terms.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;If this doesn't seem convincing, let's look at a simple example.  Pick any one of the monomials and set $f(\mathbf{x}, \theta)$ proportional to $\exp(\theta t(\mathbf{x}))$ where $t$ evaluates to $1$ for the selected monomial and otherwise equals $0$.  In other words, letting $C(\theta) = \left(\exp(\theta) + 2^n - 1\right)^{-1}$ be the normalizing coefficient,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\eqalign{&#xD;&#xA;f &amp;amp;= C(\theta) e^{\theta} &amp;amp;\text{ for this monomial}\cr&#xD;&#xA;  &amp;amp;= C(\theta) &amp;amp;\text{ for all other monomials.}&#xD;&#xA;}$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The entropy of this distribution equals&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\eqalign{&#xD;&#xA;H(\theta) &amp;amp;= -\mathbb{E}\left[\log(f)\right] \cr&#xD;&#xA;          &amp;amp;= \frac{\log(2^n-1) + (1+e^x)\log(1+e^x) - x e^x}{1 + e^x}&#xD;&#xA;}$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(writing $x = \theta - \log(2^n-1)$ to better isolate the dependence on $n$).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As $\theta$ ranges from $0$ to $\infty$, the entropy descends from $n/\log(2)$, which is the maximum attainable for any distribution, to $0$ (asymptotically), making the transition near $\theta = n \log(2)$.  At this value of $\theta$ half the mass is concentrated on the chosen monomial and the entropy is approximately one-third of its maximum possible value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can play a more complicated game by focusing various amounts of the probability on any subset of the monomials we choose and putting vanishing amounts of probability on the remaining monomials.  With such an approach we can meet any feasible lower bound on the entropy with a distribution that is as heavily concentrated on the chosen subset as possible.  There is nothing to distinguish one subset from another.  Thus, nothing informative can be said in general concerning the concentration of the distribution on the low order terms.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-11-29T16:19:52.817" LastActivityDate="2010-11-29T16:19:52.817" CommentCount="1" />
  <row Id="4997" PostTypeId="1" AcceptedAnswerId="5003" CreationDate="2010-11-29T16:08:13.153" Score="5" ViewCount="455" Body="&lt;p&gt;I'm using AIC (Akaike's Information Criterion) to compare non-linear models in R. Is it valid to compare the AICs of different types of model? Specifically, I'm comparing a model fitted by glm versus a model with a random effect term fitted by glmer (lme4).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If not, is there a way such a comparison can be done? Or is the idea completely invalid?&lt;/p&gt;&#xA;" OwnerUserId="2182" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-11-29T17:49:53.220" LastActivityDate="2010-11-29T21:48:11.017" Title="Can AIC compare across different types of model?" Tags="&lt;model-selection&gt;&lt;aic&gt;" AnswerCount="2" FavoriteCount="3" />
  <row Id="4998" PostTypeId="1" CreationDate="2010-11-29T17:25:57.397" Score="0" ViewCount="126" Body="&lt;p&gt;In brief, my question is: what is the uncertainty when I divide X data into Y groups? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In detail:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have high-resolution count data which I've binned into hourly counts. The result is anything from 0 to 200 counts per hour.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each count includes measurements of a number of physical properties, which I have divided into four categories. I now want to compare these four categories based on a crucial variable W whose value ranges from -20 to 20. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The unfortunate thing is that my counts are normally distributed about W=0. So by the time  I get to W=-20 or W=20, the counts are ~1, while at W=15 the counts are 12. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to know the minimum number of counts that I can work with, and (probably fallaciously) figured that since the minimum # of counts for 4 categories is 4, I could triple that and aim for 12. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I calculate, say, the number of counts for a 95% confidence interval?&lt;/p&gt;&#xA;" OwnerUserId="2183" LastActivityDate="2010-11-29T17:25:57.397" ClosedDate="2011-04-11T01:47:31.197" Title="What is the uncertainty when dividing 25 samples into 4 groups?" Tags="&lt;data-transformation&gt;&lt;categorical-data&gt;&lt;count-data&gt;&lt;p-value&gt;&lt;uncertainty&gt;" CommentCount="2" />
  <row Id="4999" PostTypeId="1" AcceptedAnswerId="7276" CreationDate="2010-11-29T17:26:43.070" Score="3" ViewCount="178" Body="&lt;p&gt;I recently came across the following paper:  &quot;&lt;a href=&quot;http://www.cs.huji.ac.il/~shais/papers/ShalevTewari09.pdf&quot; rel=&quot;nofollow&quot;&gt;Stochastic Methods for $\ell_1$ Regularized Loss Minimization&lt;/a&gt;&quot; by Shai Shalev-Shwartz and Ambuj Tewari, ICML 2009.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the paper, the authors propose a modification of the coordinate descent algorithm for the LASSO in which the coordinates (the $\beta$s) are updated in a random order. This modification appears to have better run-time performance than deterministic coordinate descent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can you offer any intuition for why such a modification would make the algorithm faster in practice?&lt;/p&gt;&#xA;" OwnerUserId="439" LastEditorUserId="439" LastEditorDisplayName="" LastEditDate="2010-11-29T23:35:42.850" LastActivityDate="2011-03-18T02:33:38.683" Title="Stochastic coordinate descent for $\ell_1$ regularization" Tags="&lt;regression&gt;&lt;lasso&gt;&lt;regularization&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="5000" PostTypeId="2" ParentId="4698" CreationDate="2010-11-29T17:30:31.260" Score="0" ViewCount="" Body="&lt;p&gt;Model evidence $P(D|M_i)$ can be viewed as an expectation of $P(D|w, M_i)$ with respect to distribution $P(w|M_i)$. You can then use Monte-Carlo methods to estimate it with required precision.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other suitable options include using Laplace Approximation and then finding closed-form solution for evidence (as they do in RVM).&lt;/p&gt;&#xA;" OwnerUserId="2184" LastActivityDate="2010-11-29T17:30:31.260" />
  <row Id="5001" PostTypeId="2" ParentId="212" CreationDate="2010-11-29T17:49:27.980" Score="2" ViewCount="" Body="&lt;p&gt;The Sclite tool from &lt;a href=&quot;http://www.itl.nist.gov/iad/mig//tools/&quot; rel=&quot;nofollow&quot;&gt;NIST&lt;/a&gt; offers a statistical test to compare two ASR systems on the same test set (http://www.itl.nist.gov/iad/mig//tools/).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the test you described several of the test offered would be suitable (including the sign test) but not all are equally powerful.&lt;/p&gt;&#xA;" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-11-29T18:25:11.713" LastActivityDate="2010-11-29T18:25:11.713" CommentCount="1" />
  <row Id="5003" PostTypeId="2" ParentId="4997" CreationDate="2010-11-29T18:01:09.517" Score="2" ViewCount="" Body="&lt;p&gt;It depends. AIC is a function of the log likelihood. If both types of model compute the log likelihood the same way (i.e. include the same constant) then yes you can, if the models are &lt;strong&gt;nested&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm reasonably certain that &lt;code&gt;glm()&lt;/code&gt; and &lt;code&gt;lmer()&lt;/code&gt; don't use comparable log likelihoods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The point about nested models is also up for discussion. Some say AIC is only valid for nested models as that is how the theory is presented/worked through. Others use it for all sorts of comparisons.&lt;/p&gt;&#xA;" OwnerUserId="1390" LastEditorUserId="1390" LastEditorDisplayName="" LastEditDate="2010-11-29T20:43:10.753" LastActivityDate="2010-11-29T20:43:10.753" CommentCount="10" />
  <row Id="5004" PostTypeId="1" AcceptedAnswerId="5040" CreationDate="2010-11-29T19:08:24.390" Score="2" ViewCount="151" Body="&lt;p&gt;What is the Unscented Kalman Filter and when is it used in preference to other types of filters?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;edit: I find the Wikipedia explanation a bit too technical to be readily understood.&lt;/p&gt;&#xA;" OwnerUserId="439" LastEditorUserId="439" LastEditorDisplayName="" LastEditDate="2010-11-29T20:22:17.710" LastActivityDate="2010-12-01T01:59:00.507" Title="What is the Unscented Kalman Filter?" Tags="&lt;kalman-filter&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="1" />
  <row Id="5005" PostTypeId="1" CreationDate="2010-11-29T20:35:43.210" Score="2" ViewCount="129" Body="&lt;p&gt;The following piece of Perl code randomly maps a set of ranges onto a circumference of a circle. In the example, the circumference is of length 1000 and legal ranges are e.g. (0,8)=0,1,2,...,8 and (995,2)=995,996,...,999,0,1,2 (i.e. zero-based coordinates; both start and end are inclusive).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I take some arbitrary position on the circumference (e.g. 36) and count how many ranges cover it in each simulation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;finally, I calculate the mean and variance of this statistic.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;use strict;&#xA;use warnings;&#xA;&#xA;use Statistics::Descriptive;&#xA;&#xA;my $n_simulations   = 1000;&#xA;my $circumference   = 1000;&#xA;my @lengths_distrib = (100) x 100;    # distibution of range lengths&#xA;my $some_pos        = 36;             # arbitrary position&#xA;&#xA;my $stat = Statistics::Descriptive::Full-&amp;gt;new();&#xA;&#xA;foreach my $sim ( 1 .. $n_simulations ) {&#xA;&#xA; # randomly map ranges onto circumference&#xA;&#xA; my @random_ranges =&#xA;  map { my $start = int( rand($circumference) ); [ $start, ( $start + $_ -1 ) % $circumference ] }&#xA;  @lengths_distrib;&#xA;&#xA; # count how many range contain $some_pos&#xA; my $num_covering_ranges = scalar(&#xA;  grep { ( $_-&amp;gt;[0] &amp;lt;= $some_pos and $_-&amp;gt;[1] &amp;gt;= $some_pos ) or ( $_-&amp;gt;[1] &amp;lt; $_-&amp;gt;[0] and $_-&amp;gt;[1] &amp;gt; $some_pos ) }&#xA;   @random_ranges&#xA; );&#xA; $stat-&amp;gt;add_data($num_covering_ranges);&#xA;}&#xA;print $stat-&amp;gt;mean, ' ', $stat-&amp;gt;variance, &quot;\n&quot;;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To the best of my knowledge, this kind of random variable should follow Poisson distribution (law of rare events and so on). Hence, the mean and variance should be equal. However, the variance seems to systematically be a bit lower than the mean.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What am I missing?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE&lt;/strong&gt;&#xA;Following whuber response, a couple of notes and additions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;em&gt;the Poisson distribution is a good approximation of the binomial distribution if n is at least 20 and p is smaller than or equal to 0.05, and an excellent approximation if n ≥ 100 and np ≤ 10.&lt;/em&gt; source: &lt;a href=&quot;http://en.wikipedia.org/wiki/Poisson_distribution&quot; rel=&quot;nofollow&quot;&gt;wikipedia&lt;/a&gt;. The example I gave here is a toy one. I usually use much larger circumferences (length ~ 3M), many more ranges (around 25k) an relatively small ranges (size ~ 2.5k), so I think in general Poisson distribution should be appropriate. I will give it a look again on some real data.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;In my real data, I have e.g. some 25k ranges of different sizes. Currently, I'm simulating random mapping of these clones on the circumference, count the coverage of some point of interest in each simulation (this could be an arbitrary fixed point since all point are equal), then get the mean of this coverage over some 100 or 1000 simulations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, I use a CDF of a Poisson distribution with this mean to get a P-value for the actual observation (number of ranges covering a specific point of interest). I take lower tail since I'm interested in under-covered positions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How would you suggest going around this? Should I use a Binomial distribution? How can I evaluate its parameters? Note this is not exactly a Binomial RV, since while I do have n independent trials, the chances for each trial to succeed is not equal (it depends on the range size). Perhaps I should simply take the average success probability (average length) and not even have to use simulations, right? In other words, I guess what I'm asking is how does the sum of independent Bernoulli RV with &lt;strong&gt;different p's&lt;/strong&gt; look like?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I should also note that while this example only deals with the number of ranges covering a single position, I'm also interested in the number of ranges covering some range. For now I use the simulations scheme exactly the same way described before.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="634" LastEditorUserId="634" LastEditorDisplayName="" LastEditDate="2010-11-30T07:01:38.970" LastActivityDate="2010-11-30T07:01:38.970" Title="Unexpected under-dispersion in Perl simulations of Poisson RV" Tags="&lt;distributions&gt;&lt;binomial&gt;&lt;random-variable&gt;&lt;poisson&gt;&lt;simulation&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="5006" PostTypeId="2" ParentId="423" CreationDate="2010-11-29T20:45:50.423" Score="6" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://phd.kt.pri.ee/2009/08/11/captain-statistics-episode-1/&quot; rel=&quot;nofollow&quot; title=&quot;Captain Statistics&quot;&gt;Here&lt;/a&gt;'s a somewhat more technical one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/FycZF.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-11-29T23:10:51.197" LastActivityDate="2010-11-29T23:10:51.197" CommunityOwnedDate="2010-11-29T20:45:50.423" />
  <row Id="5007" PostTypeId="1" AcceptedAnswerId="5009" CreationDate="2010-11-29T20:54:10.447" Score="3" ViewCount="1110" Body="&lt;p&gt;I have a plot I'm making in ggplot2 to summarize data that is from a 2 x 4 x 3 celled dataset.  I have been able to make panels for the 2 leveled variable using facet_grid(. ~ Age) and to set the x and y axes using aes(x=4leveledVariable, y=DV).  I used aes(group=3leveledvariable, lty=3leveledvariable) to produce the plot so far.  This gives me a visualization that is paneled by the 2 level variable, with the X axis representing the 4 leveled variable and different lines ploted within the panels for the 3 leveled variable.  But the key for the 3 leveled variable is titled with the 3 leveled variable's name and I want it to be a title that has a character space in it.  How can I rename the title of the legend?  I've done quite a few google searches and looked through the ggplot2 manual but I haven't managed to solve this problem yet, so I'm turning to the CrossValidated community.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Things I've tried that don't seem to work (where abp is my ggplot2 object):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; abp &amp;lt;- abp + opts(legend.title=&quot;Town Name&quot;)&#xA; abp &amp;lt;- abp + scale_fill_continuous(&quot;Town Name&quot;)&#xA; abp &amp;lt;- abp + opts(group=&quot;Town Name&quot;)&#xA; abp &amp;lt;- abp + opts(legend.title=&quot;Town Name&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Example data:&#xA;ex.data &amp;lt;- data.frame(DV=rnorm(2*4*3),V2=rep(1:2,each=4*3),V4=rep(1:4,each=3),V3=1:3)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="196" LastEditorUserId="196" LastEditorDisplayName="" LastEditDate="2010-11-30T01:24:54.040" LastActivityDate="2011-03-06T23:17:04.847" Title="How can I change the title of a legend in ggplot2?" Tags="&lt;r&gt;&lt;data-visualization&gt;&lt;ggplot2&gt;" AnswerCount="2" CommentCount="6" />
  <row Id="5009" PostTypeId="2" ParentId="5007" CreationDate="2010-11-29T21:08:00.030" Score="5" ViewCount="" Body="&lt;p&gt;You can change the title of the legend by modifying the scale for that legend. Here's an example using the CO2 dataset&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(ggplot2)&#xA;&#xA;p &amp;lt;- qplot(conc, uptake, data = CO2, colour = Type) + scale_colour_discrete(name = &quot;Fancy Title&quot;)&#xA;p &amp;lt;- p + facet_grid(. ~ Treatment)&#xA;p &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;EDIT:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using the example data from above, here is a working solution. I think this mimics the plot that @drknexus is trying to create. As a side note, if anyone can explain why we have to treat V3 as a factor for it to be mapped to the legend, I'd appreciate it.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;p &amp;lt;- qplot(V4, DV, data = ex.data, geom = &quot;line&quot;, group = V3, lty = factor(V3)) &#xA;p &amp;lt;- p + scale_linetype_discrete(name = &quot;Fancy Title&quot;) + facet_grid(. ~ V2)&#xA;p &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/ZD8V7.jpg&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="696" LastEditorUserId="696" LastEditorDisplayName="" LastEditDate="2010-11-29T22:54:18.910" LastActivityDate="2010-11-29T22:54:18.910" CommentCount="6" />
  <row Id="5010" PostTypeId="2" ParentId="4997" CreationDate="2010-11-29T21:48:11.017" Score="1" ViewCount="" Body="&lt;p&gt;This is a great question that I've been curious about for a while.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For models in the same family (ie. auto-regressive models of order k or polynomials) AIC/BIC makes a lot of sense. In other cases it's less clear. Computing the log-likelihood exactly (with the constant terms) should work, but using more complicated model comparison such as Bayes Factors is probably better (http://www.jstor.org/stable/2291091).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the models have the same loss/error-function one alternative is to just compare the cross-validated log-likelihoods. That's usually what I try to do when I'm not sure AIC/BIC makes sense in a certain situation.&lt;/p&gt;&#xA;" OwnerUserId="2077" LastActivityDate="2010-11-29T21:48:11.017" />
  <row Id="5011" PostTypeId="1" AcceptedAnswerId="5137" CreationDate="2010-11-29T22:35:10.797" Score="5" ViewCount="134" Body="&lt;p&gt;Kernel density estimator is given by &#xA;$$\hat{f}(x,h)=\frac{1}{nh}\sum_{i=1}^{n}K(\frac{x-X_{i}}{h})$$ &#xA;where $X_1,...X_n$  i.i.d with some unknown density $f$, $h$ - bandwith,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$K$ - kernel function    (&#xA;$\int_{-\infty}^{\infty}K(x)dx=1$,&#xA;$\int_{-\infty}^{\infty}K(x)xdx=0$,&#xA;$\int_{-\infty}^{\infty}K(x)x^2dx&amp;lt;\infty$).&#xA;The bias  can be computed using Taylor expansion:&#xA;$$\int_{-\infty}^{\infty}\frac{1}{h}K(\frac{x-y}{h})f(y)dy-f(x)=\int_{-\infty}^{\infty}K(y)\left(f(x-hy)-f(x)\right)dy$$ $$=\int_{-\infty}^{\infty}K(y)\left(f&amp;#39;(x)hy+\frac{1}{2}f&amp;#39;&amp;#39;(x)(hy)^{2}+o(h^{2})\right)dy=\frac{1}{2}f&amp;#39;&amp;#39;(x)h^{2}+o(h^{2})$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How to deal with periodic   kernel and $f$ ($\int_{0}^{1}K(x)dx=1$,$\int_{0}^{1}K(x)xdx=0$,$\int_{0}^{1}K(x)x^2dx&amp;lt;\infty$)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I use taylor expansion?($\int_{0}^{1}\frac{1}{h}K(\frac{y-x}{h})f(y)dy=\int_{-\frac{x}{h}}^{1-\frac{x}{h}}K(y)f(x-yh)dy\neq\int_{0}^{1}K(y)f(x-yh)dy$ -I can't use kernel properties)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could you recommend a good book about  kernel smoothing for circular data?&lt;/p&gt;&#xA;" OwnerUserId="2189" LastActivityDate="2010-12-04T12:13:36.183" Title="Bias for kernel density estimator (periodic case)" Tags="&lt;kernel&gt;" AnswerCount="1" FavoriteCount="2" />
  <row Id="5012" PostTypeId="2" ParentId="5005" CreationDate="2010-11-29T23:23:02.660" Score="3" ViewCount="" Body="&lt;p&gt;If I understood correctly, a &quot;range&quot; of length $k$ (such as $k=100$) within a circumference of length $n$ (such as $n = 1000$) has a chance of $(2k+1)/n$ of covering a given point on the circumference, and all the chances in a simulation are independent.  Therefore in a simulation of $N$ trials (such as $N=1000$) the count of ranges that cover a given point should follow a $\text{Binomial}(\frac{2k+1}{n}, N)$ distribution.  This distribution has mean $(2k+1)\frac{N}{n}$ and variance $(2k+1)(1 - \frac{2k+1}{n})\frac{N}{n}$.  The variance of your &lt;em&gt;data&lt;/em&gt; (the simulation trial outcomes) will deviate from this latter value due to chance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These considerations identify two reasons for your observations:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;The variance &lt;em&gt;ought&lt;/em&gt; to be a fraction of the mean; specifically, it should only be about $1 - \frac{2k+1}{n}$ of the mean.  In the case $k=100$ and $n=1000$, the variance should only be around 79.9% of the mean.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;When you conduct a small number of trials (and $N=1000$ is quite small for simulations of this type), expect noticeable chance deviations between the results and the expectations.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-11-29T23:23:02.660" CommentCount="3" />
  <row Id="5013" PostTypeId="1" AcceptedAnswerId="5030" CreationDate="2010-11-29T23:53:09.483" Score="4" ViewCount="331" Body="&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/bBDqg.png&quot; alt=&quot;N=2762&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;N=2762&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've been exploring a data set that seems to give rise to this kind of plot rather frequently. Would you say this is one population with a different than normal population? Or are two populations confounding the normal distribution?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It used matplotlib and scipy.stats:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;      (x,y), (slope, inter, cor) = stats.probplot(data, dist='norm')&#xA;&#xA;      osmf = x.take([0, -1])  # endpoints&#xA;      osrf = slope * osmf + inter&#xA;      self.ax.plot(x, y, ',', osmf, osrf, '-', **self.kwargs)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2191" LastActivityDate="2010-12-03T03:44:23.953" Title="Quantile-Quantile Plot with Unknown Distribution?" Tags="&lt;distributions&gt;" AnswerCount="3" CommentCount="3" />
  <row Id="5014" PostTypeId="2" ParentId="5013" CreationDate="2010-11-30T02:33:35.903" Score="-1" ViewCount="" Body="&lt;p&gt;You may want to take a look at the   &lt;a href=&quot;http://en.wikipedia.org/wiki/Anderson%E2%80%93Darling_test#Test_for_Normality&quot; rel=&quot;nofollow&quot;&gt;Anderson-Darling&lt;/a&gt; test for normality which empirically tests whether or not your data comes from a given distribution. @chl recommends looking at the &lt;code&gt;scipy&lt;/code&gt; toolkit, specifically &lt;code&gt;anderson()&lt;/code&gt; in &lt;code&gt;morestats.py&lt;/code&gt; for an implementation.&lt;/p&gt;&#xA;" OwnerUserId="696" LastEditorUserId="696" LastEditorDisplayName="" LastEditDate="2010-12-03T03:44:23.953" LastActivityDate="2010-12-03T03:44:23.953" CommentCount="10" />
  <row Id="5015" PostTypeId="1" AcceptedAnswerId="5021" CreationDate="2010-11-30T02:36:57.163" Score="3" ViewCount="174" Body="&lt;p&gt;I am interested in learning (and implementing) an alternative to polynomial interpolation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I am having trouble finding a good description of how these methods work, how they relate, and how they compare.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would appreciate your input on the pros/cons/conditions under which these methods or alternatives would be useful, but some good references to texts, slides, or podcasts would be sufficient.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="1381" LastEditorUserId="1381" LastEditorDisplayName="" LastEditDate="2010-11-30T02:42:38.533" LastActivityDate="2010-11-30T07:30:57.407" Title="What are the advantages / disadvantages of using splines, smoothed splines, and gaussian process emulators?" Tags="&lt;interpolation&gt;&lt;splines&gt;" AnswerCount="1" CommentCount="3" FavoriteCount="0" />
  <row Id="5016" PostTypeId="1" CreationDate="2010-11-30T04:34:16.530" Score="2" ViewCount="136" Body="&lt;p&gt;I'm trying to compute some p-values for samples from a distribution of sums of ~1000 random variables. The exact distribution of these random variables isn't known, but I have empirical estimates that I think are pretty accurate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far I've been using the central limit theorem to produce a normal approximation for this sum, which does OK since n is relatively large, but not great. Computing the exact distribution of the sum via convolutions is too slow, but I only really care about having high accuracy in the tails; everywhere else a rough approximation is fine.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any methods that will allow me to improve my estimates in the tails (or just one tail) without having to compute the entire convolutions? I'm not sure if this would be a variant on the CLT implementation or something completely different. My gut feeling is that there isn't any way to do this, so I'd be open to any kind of solution at all!&lt;/p&gt;&#xA;" OwnerUserId="2111" LastActivityDate="2010-11-30T05:21:01.157" Title="Central Limit Theorem Tails" Tags="&lt;normal-distribution&gt;&lt;approximation&gt;&lt;central-limit-theorem&gt;" AnswerCount="1" CommentCount="4" />
  <row Id="5017" PostTypeId="2" ParentId="5016" CreationDate="2010-11-30T05:21:01.157" Score="2" ViewCount="" Body="&lt;p&gt;$n=1000$ gets you actually extremely close to normal if you do not go to the extreme tails ($n=10$ is often close enough to normal in the central region), but if you need to go there, estimating a PDF might well be impossible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To get an event very far out in the tails, extreme fluctuations in your sumands need to happen. Since you say your distributions were &quot;empirical estimates&quot; you should ask yourself if you have enough input data to understand the tails of your summand distributions. In many applications empirical distributions are being truncated in the tails, since they are not well understood and might be the only contributors in some parameter regions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;No matter if I truncate my distributions or not, I often use &lt;a href=&quot;http://de.wikipedia.org/wiki/Monte-Carlo-Simulation&quot; rel=&quot;nofollow&quot;&gt;Monte Carlo simulations&lt;/a&gt; to compute convolutions of (not parametrized) empirical distributions since computing power is cheap.&lt;/p&gt;&#xA;" OwnerUserId="56" LastActivityDate="2010-11-30T05:21:01.157" CommentCount="2" />
  <row Id="5018" PostTypeId="2" ParentId="4551" CreationDate="2010-11-30T06:18:04.063" Score="3" ViewCount="" Body="&lt;p&gt;My old stats prof had a &quot;rule of thumb&quot; for dealing with outliers: If you see an outlier on your scatterplot, cover it up with your thumb :)&lt;/p&gt;&#xA;" OwnerUserId="74" LastActivityDate="2010-11-30T06:18:04.063" CommunityOwnedDate="2010-11-30T06:18:04.063" />
  <row Id="5019" PostTypeId="2" ParentId="4551" CreationDate="2010-11-30T06:19:58.040" Score="11" ViewCount="" Body="&lt;p&gt;Reporting p-values when you did data-mining (hypothesis discovery) instead of statistics (hypothesis testing)&lt;/p&gt;&#xA;" OwnerUserId="74" LastActivityDate="2010-11-30T06:19:58.040" CommunityOwnedDate="2010-11-30T06:19:58.040" />
  <row Id="5020" PostTypeId="2" ParentId="4551" CreationDate="2010-11-30T06:54:37.230" Score="5" ViewCount="" Body="&lt;p&gt;A few mistakes that bother me:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Assuming unbiased estimators are always better than biased estimators.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Assuming that a high R^2 implies a good model, low R^2 implies a bad model.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Incorrectly interpreting/applying correlation.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Reporting point estimates without standard error.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Using methods which assume some sort of Multivariate Normality (such as Linear Discriminant Analysis) when more robust, better performing, non/semiparametric methods are available. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Using p-value as a measure of strength between a predictor and the response, rather than as a measure of how much evidence there is of &lt;em&gt;some&lt;/em&gt; relationship. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="2144" LastActivityDate="2010-11-30T06:54:37.230" CommunityOwnedDate="2010-11-30T06:54:37.230" CommentCount="1" />
  <row Id="5021" PostTypeId="2" ParentId="5015" CreationDate="2010-11-30T07:30:57.407" Score="1" ViewCount="" Body="&lt;p&gt;There is some material on splines and smoothing splines in &lt;a href=&quot;http://www-stat.stanford.edu/~tibs/ElemStatLearn/&quot; rel=&quot;nofollow&quot;&gt;The Elements of &#xA;Statistical Learning&lt;/a&gt; by Hastie et al.&lt;/p&gt;&#xA;" OwnerUserId="439" LastActivityDate="2010-11-30T07:30:57.407" />
  <row Id="5022" PostTypeId="2" ParentId="4551" CreationDate="2010-11-30T08:02:28.100" Score="7" ViewCount="" Body="&lt;p&gt;Being exploratory but pretending to be confirmatory. This &lt;em&gt;can&lt;/em&gt; happen when one is modifying the analysis strategy (i.e. model fitting, variable selection and so on) data driven or result driven but not stating this openly and then only reporting the &quot;best&quot; (i.e. with smallest p-values) results as if it had been the only analysis. This also pertains to the point if multiple testing that Chris Beeley made and results in a high false positive rate in scientific reports.&lt;/p&gt;&#xA;" OwnerUserId="1573" LastEditorUserId="1573" LastEditorDisplayName="" LastEditDate="2010-12-01T15:50:26.957" LastActivityDate="2010-12-01T15:50:26.957" CommunityOwnedDate="2010-11-30T08:02:28.100" />
  <row Id="5023" PostTypeId="1" CreationDate="2010-11-30T08:37:51.447" Score="1" ViewCount="100" Body="&lt;p&gt;Hi all,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any comprehensive reference on (or introduction to) how people have tried to model non-independent random variables? I already know about mixing processes, which express in various ways according to various coefficients how &quot;future&quot; events depend on &quot;past&quot; events, but that's about it...&lt;/p&gt;&#xA;" OwnerUserId="2197" LastActivityDate="2010-11-30T16:00:56.360" Title="Modelling dependence between random variables" Tags="&lt;random-variable&gt;&lt;non-independent&gt;" AnswerCount="1" CommentCount="5" FavoriteCount="1" />
  <row Id="5024" PostTypeId="2" ParentId="4698" CreationDate="2010-11-30T09:33:50.900" Score="1" ViewCount="" Body="&lt;p&gt;Yes, you can do that. However, your I'd like to play with your formulas a little bit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the model is determined by the parameters, than $P(D|M_i)=\int P(D|w,M_i)P(M_i|w)*P(w)dw$ should be more appropriate. Since I guess the model is determined by the parameters in a deterministic (instead of stochastic) way, the formula can be abbreviated to $P(D|M_i)=\int P(D|w,M_i)*P(w)dw$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given this and only a finite uniform sample over the parameter space, your approximation is indeed correct.&lt;/p&gt;&#xA;" OwnerUserId="264" LastActivityDate="2010-11-30T09:33:50.900" />
  <row Id="5025" PostTypeId="1" AcceptedAnswerId="5032" CreationDate="2010-11-30T10:07:53.553" Score="8" ViewCount="260" Body="&lt;p&gt;Suppose we have a simple linear regression model $Z = aX + bY$ and would like to test the null hypothesis $H_0: a=b=\frac{1}{2}$ against the general alternative. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think one can use the estimate of $\hat{a}$ and $SE(\hat{a})$ and further apply a $Z$-test to get the confidence interval around $\frac{1}{2}$. Is this ok?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The other question is strongly related to this one. Suppose that we have a sample $\{(x_1,y_1,z_1),\ldots ,(x_n,y_n,z_n) \}$ and we compute $\chi^2$ statistics &lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{equation}&#xA;\sum_{i=1}^n \frac{(z_i-\frac{x_i+y_i}{2})^2}{\frac{x_i+y_i}{2}}.&#xA;\end{equation}&#xA;Can these statistics be used to test the same null hypothesis?&lt;/p&gt;&#xA;" OwnerUserId="1215" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-11-30T16:45:51.890" LastActivityDate="2010-11-30T19:25:30.990" Title="How to test if the slopes in the linear model are equal to a fixed value?" Tags="&lt;hypothesis-testing&gt;&lt;regression&gt;" AnswerCount="2" FavoriteCount="2" />
  <row Id="5026" PostTypeId="1" CreationDate="2010-11-30T11:26:15.473" Score="9" ViewCount="860" Body="&lt;p&gt;What is the difference between data mining, statistics, machine learning and AI?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would it be accurate to say that they are 4 fields attempting to solve very similar problems but with different approaches? What exactly do they have in common and where do they differ? If there is some kind of hierarchy between them, what would it be?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Similar questions have been asked previously but I still don't get it:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://stats.stackexchange.com/questions/1521/data-mining-and-statistical-analysis&quot;&gt;Data Mining and Statistical Analysis&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://stats.stackexchange.com/questions/6/the-two-cultures-statistics-vs-machine-learning&quot;&gt;The Two Cultures: statistics vs. machine learning?&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="2199" LastActivityDate="2011-01-09T19:59:14.237" CommunityOwnedDate="2010-11-30T13:19:58.653" Title="What is the difference between data mining, statistics, machine learning and AI?" Tags="&lt;machine-learning&gt;&lt;data-mining&gt;" AnswerCount="4" CommentCount="7" FavoriteCount="8" />
  <row Id="5028" PostTypeId="2" ParentId="5026" CreationDate="2010-11-30T12:05:06.173" Score="6" ViewCount="" Body="&lt;p&gt;We can say that they are all related, but they are all differences things.&#xA;Although you can have things in common among them such in statistics and data mining that you use clustering methods.&lt;br&gt;&#xA;Let me try to briefly define each:  &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Statistics is a very old discipline mainly based in classical mathematical methods, which can be used for the same purpose that data mining sometimes is which is classifying and grouping things.  &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Data mining consists in building models in order to detect the patterns that allow us to classify or predict situations given an amount of facts or factors. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Artificial intelligence (check Marin Minsky*) is the discipline that tries to emulate how the brain works with programming methods, for example building a program that plays chess.  &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Machine learning is the task of building knowledge and storing it in some for in the computer, that form can be of mathematical models, algorithms, etc... Anything that can help detect patterns.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="1808" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-11-30T13:22:31.740" LastActivityDate="2010-11-30T13:22:31.740" CommunityOwnedDate="2010-11-30T13:19:58.653" CommentCount="2" />
  <row Id="5030" PostTypeId="2" ParentId="5013" CreationDate="2010-11-30T15:35:14.227" Score="6" ViewCount="" Body="&lt;p&gt;There are a variety of different possibilities. For example, a chi-square distribution with degrees of freedom in the range of 30-40 would give rise to such a qq-plot. In R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x &amp;lt;- rchisq(10000, df=35)&#xA;qqnorm(x)&#xA;qqline(x)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;looks like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/U60rs.jpg&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A mixture of two normals with different means doesn't apply though.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x &amp;lt;- c(rnorm(10000/2, mean=0), rnorm(10000/2, mean=2))&#xA;qqnorm(x)&#xA;qqline(x)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;looks like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/EHoPs.jpg&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note how the points cross the line, which is a different pattern than the one you observe.&lt;/p&gt;&#xA;" OwnerUserId="1934" LastActivityDate="2010-11-30T15:35:14.227" />
  <row Id="5031" PostTypeId="2" ParentId="5023" CreationDate="2010-11-30T16:00:56.360" Score="1" ViewCount="" Body="&lt;p&gt;OK, I think in what exists now, what comes closest to what you are looking for is the coalescent theory. &lt;a href=&quot;http://en.wikipedia.org/wiki/Coalescence_%28genetics%29&quot; rel=&quot;nofollow&quot;&gt;Quoting from wikipedia&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In genetics, coalescent theory is a retrospective model of population genetics. It employs a sample of individuals from a population to trace all alleles of a gene shared by all members of the population to a single ancestral copy, known as the most recent common ancestor (MRCA; sometimes also termed the coancestor to emphasize the coalescent relationship[1]). The inheritance relationships between alleles are typically represented as a gene genealogy, similar in form to a phylogenetic tree. This gene genealogy is also known as the coalescent; understanding the statistical properties of the coalescent under different assumptions forms the basis of coalescent theory. The coalescent runs models of genetic drift backward in time to investigate the genealogy of antecedents.[2] In the most simple case, coalescent theory assumes no recombination, no natural selection, and no gene flow or population structure. Advances in coalescent theory, however, allow extension to the basic coalescent, and can include recombination, selection, and virtually any arbitrarily complex evolutionary or demographic model in population genetic analysis. The mathematical theory of the coalescent was originally developed in the early 1980s by John Kingman[3].&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The citations list of the wikipedia article mentions a primer on coalescence, looks like a good place to start.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.oeb.harvard.edu/faculty/wakeley/John/Reprints/WakeleyChapterLongDraft.pdf&quot; rel=&quot;nofollow&quot;&gt;This paper gives a review of coalescence and natural selection.&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://onlinelibrary.wiley.com/doi/10.1111/j.1461-0248.2004.00572.x/pdf&quot; rel=&quot;nofollow&quot;&gt;This paper gives a relatively lowbrow example of how coalescence theory is used in the case of neutral selection.&lt;/a&gt; This can help you get the feeling for the ideas. It also contains references to seminal papers in the field.&lt;/p&gt;&#xA;" OwnerUserId="2036" LastActivityDate="2010-11-30T16:00:56.360" />
  <row Id="5032" PostTypeId="2" ParentId="5025" CreationDate="2010-11-30T16:25:00.833" Score="7" ViewCount="" Body="&lt;p&gt;In linear regression the assumption is that $X$ and $Y$ are not random variables.  Therefore, the model&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Z = a X + b Y + \epsilon$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;is algebraically the same as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Z - \frac{1}{2} X - \frac{1}{2} Y = (a  - \frac{1}{2})X + (b  - \frac{1}{2})Y + \epsilon = \alpha X + \beta Y + \epsilon.$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here, $\alpha = a - \frac{1}{2}$ and $\beta  =b - \frac{1}{2}$.  The error term $\epsilon$ is unaffected.  Fit this model, estimating the coefficients as $\hat{\alpha}$ and $\hat{\beta}$, respectively, and test the hypothesis $\alpha = \beta = 0$ in the usual way.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;The statistic written at the end of the question is not a chi-squared statistic, despite its formal similarity to one.  A chi-squared statistic involves &lt;em&gt;counts&lt;/em&gt;, not data values, and must have expected values in its denominator, not covariates.  It's possible for one or more of the denominators $\frac{x_i+y_i}{2}$ to be zero (or close to it), showing that something is seriously wrong with this formulation.  If even that isn't convincing, consider that the units of measurement of $Z$, $X$, and $Y$ could be anything (such as drams, parsecs, and pecks), so that a linear combination like $z_i - (x_i+y_i)/2$ is (in general) meaningless.   It doesn't test anything.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-11-30T19:25:30.990" LastActivityDate="2010-11-30T19:25:30.990" CommentCount="2" />
  <row Id="5033" PostTypeId="2" ParentId="5013" CreationDate="2010-11-30T17:08:52.140" Score="5" ViewCount="" Body="&lt;p&gt;Your dataset clearly is not normal.  (With this much data, any goodness of fit test will tell you that.)  But you can read much more than that from the normal probability plot:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;The generally smooth curvature does not hint at a mixture structure.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The upper tail is too stretched out (values too high compared to the reference distribution).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The lower tail is too compressed (values also too high).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;This suggests that a mild &lt;a href=&quot;http://en.wikipedia.org/wiki/Power_transform&quot; rel=&quot;nofollow&quot;&gt;Box-Cox transformation&lt;/a&gt; will produce nearly-normal, or at least symmetric, data.   To find it, consider some key values on this plot: the median, found above the x-value of 0, is about 0.90; +2 standard deviations is about 0.99; and -2 standard deviations is about 0.825.  The nonlinearity is apparent from the simple calculations 0.99 - 0.90 = 0.09 whereas 0.90 - 0.825 = 0.075: the rise from the median to the upper tail is greater than the rise from the lower tail to the median.  We can equalize the slopes by trying out some simple re-expressions &lt;em&gt;of these three values only.&lt;/em&gt;  For example, taking the reciprocals of the three key data values (Box-Cox power of -1) gives&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1/0.825 = 1.21&#xA;1/0.90  = 1.11; 1.21 - 1.11 = 0.10 (new slope is 0.050 per SD)&#xA;1/0.99  = 1.01; 1.11 - 1.01 = 0.10 (0.050 per SD)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Because the slopes &lt;em&gt;of the re-expressed values&lt;/em&gt; are now equal, we know the plot of &lt;em&gt;reciprocals&lt;/em&gt; of the data will be approximately linear between -2 and +2 SDs.  As a check, let's pick more points further out into the tails and see what the reciprocal does to them.  I estimate that the value in the plot at -3 SD from the mean is around 0.79 and the value +3 SD from the mean is 1.05.  The two slopes in question equal 0.053 and 0.052 per SD: close enough to each other &lt;em&gt;and&lt;/em&gt; to the slopes found between -2 and +2 SD.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My estimates--based on the plot as shown on a monitor--are crude, so you will want to repeat these (simple, quick) calculations with the actual data.  Nevertheless, there is considerable evidence that your data when suitably re-expressed with a simple transformation will be close to normally distributed.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-11-30T17:08:52.140" />
  <row Id="5034" PostTypeId="2" ParentId="5025" CreationDate="2010-11-30T17:14:08.923" Score="7" ViewCount="" Body="&lt;p&gt;You can test this hypothesis with a full versus reduced model test. Here is how you do this. First, fit the model $Z = aX + bY$ and get the residuals from that model. Square the residuals and sum them up. This is the sum of square error for the full model. Let's call this $SSE_f$. Next, calculate $Z - \hat{Z}$ where $\hat{Z} = 1/2*X + 1/2*Y$. These are your residuals under the null hypothesis. Square them and sum them up. This is the sum of square error for the reduced model. Let's call this $SSE_r$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now compute:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;F = $((SSE_r - SSE_f)/2) / (SSE_f / (n-2))$,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $n$ is the sample size. Under $H_0$, this F-statistic follows an F-distribution with $2$ and $n-2$ degrees of freedom.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an example using R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x &amp;lt;- rnorm(n)&#xA;y &amp;lt;- rnorm(n)&#xA;z &amp;lt;- 1/2*x + 1/2*y + rnorm(n) ### note I am simulating under H0 here&#xA;&#xA;res &amp;lt;- lm(z ~ x + y - 1)&#xA;summary(res)&#xA;SSE.f &amp;lt;- sum(resid(res)^2)&#xA;&#xA;zhat  &amp;lt;- 1/2*x + 1/2*y&#xA;SSE.r &amp;lt;- sum((z-zhat)^2)&#xA;&#xA;F &amp;lt;- ((SSE.r - SSE.f) / 2) / (SSE.f / (n-2))&#xA;pf(F, 2, n-2, lower.tail=FALSE) ### this is the p-value&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Reject the null if the p-value is below .05 (if your $\alpha$ is indeed .05).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I assume you really meant for your model not to contain an intercept. In other words, I assume you are really working with the model $Z = aX + bY$ and not $Z = c + aX + bY$.&lt;/p&gt;&#xA;" OwnerUserId="1934" LastActivityDate="2010-11-30T17:14:08.923" CommentCount="1" />
  <row Id="5035" PostTypeId="1" AcceptedAnswerId="5036" CreationDate="2010-11-30T19:19:28.500" Score="1" ViewCount="81" Body="&lt;p&gt;I have a study which gives the hazard ratios for certain interventions like:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Intervention 1, HR: .9&lt;/li&gt;&#xA;&lt;li&gt;Intervention 2, HR: .8&lt;/li&gt;&#xA;&lt;li&gt;...&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Is there any way in which I could predict the relative risk for e.g. someone who has intervention 1 and 2? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think if I assume that the interventions are independent, then the HR of doing both is just the product of the individuals' HRs, right?&lt;/p&gt;&#xA;" OwnerUserId="900" LastActivityDate="2010-11-30T20:20:48.150" Title="Can covariates in a Cox Proportional Hazard model be combined in any meaningful way?" Tags="&lt;survival&gt;&lt;hazard&gt;" AnswerCount="1" />
  <row Id="5036" PostTypeId="2" ParentId="5035" CreationDate="2010-11-30T20:20:48.150" Score="5" ViewCount="" Body="&lt;p&gt;The HR of doing both is the product of the separate HRs if you assume that there's no &lt;em&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Interaction_%28statistics%29&quot; rel=&quot;nofollow&quot;&gt;interaction&lt;/a&gt;&lt;/em&gt; between the two effects on the multiplicative scale. &lt;a href=&quot;http://en.wikipedia.org/wiki/Independence_%28probability_theory%29&quot; rel=&quot;nofollow&quot;&gt;Independence&lt;/a&gt; is a separate issue from interaction.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-11-30T20:20:48.150" />
  <row Id="5037" PostTypeId="2" ParentId="4551" CreationDate="2010-11-30T20:22:45.437" Score="1" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://sqab.psychology.org/movies/Grace01High.wmv&quot; rel=&quot;nofollow&quot;&gt;http://sqab.psychology.org/movies/Grace01High.wmv&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This video was highly informative for me in the use and interpretation of statistical tests.&lt;/p&gt;&#xA;" OwnerUserId="2202" LastActivityDate="2010-11-30T20:22:45.437" CommunityOwnedDate="2010-11-30T20:22:45.437" />
  <row Id="5038" PostTypeId="1" AcceptedAnswerId="5039" CreationDate="2010-11-30T21:01:07.213" Score="6" ViewCount="386" Body="&lt;p&gt;Hi,&lt;br&gt;&#xA;I'm experimenting with R and found that an anova() needs an object of type lm. But why should I continue with an anova after this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; x &amp;lt;- data.frame(rand=rnorm(100), factor=sample(c(&quot;A&quot;,&quot;B&quot;,&quot;C&quot;),100,replace=TRUE))&#xA;&amp;gt; head(x)&#xA;        rand factor&#xA;1  0.9640502      B&#xA;2 -0.5038238      C&#xA;3 -1.5699734      A&#xA;4 -0.8422324      B&#xA;5  0.2489113      B&#xA;6 -1.4685439      A&#xA;&#xA;&amp;gt; model &amp;lt;- lm(x$rand ~ x$factor))&#xA;&amp;gt; summary(model)&#xA;&#xA;Call:&#xA;lm(formula = x$rand ~ x$factor)&#xA;&#xA;Residuals:&#xA;     Min       1Q   Median       3Q      Max &#xA;-2.74118 -0.89259  0.02904  0.59726  3.19762 &#xA;&#xA;Coefficients:&#xA;            Estimate Std. Error t value Pr(&amp;gt;|t|)&#xA;(Intercept)  -0.1878     0.1845  -1.018    0.311&#xA;x$factorB    -0.1284     0.2689  -0.477    0.634&#xA;x$factorC     0.4246     0.2689   1.579    0.118&#xA;&#xA;Residual standard error: 1.107 on 97 degrees of freedom&#xA;Multiple R-squared: 0.04345, Adjusted R-squared: 0.02372 &#xA;F-statistic: 2.203 on 2 and 97 DF,  p-value: 0.1160&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This tells me everything I need, or does it not? I'm curious why you want to continue with an anova(model)&lt;/p&gt;&#xA;" OwnerUserId="2091" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-01T00:12:14.820" LastActivityDate="2010-12-01T00:12:14.820" Title="What are the ANOVA's benefits over a normal linear model?" Tags="&lt;r&gt;&lt;anova&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="4" />
  <row Id="5039" PostTypeId="2" ParentId="5038" CreationDate="2010-11-30T21:46:50.907" Score="12" ViewCount="" Body="&lt;p&gt;Let's look at what you get when you actually use the anova() function (the numbers are different than in your example, since I don't know what seed you used for generating the random numbers, but the point remains the same):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; anova(model)&#xA;&#xA;Analysis of Variance Table&#xA;&#xA;Response: x$rand&#xA;          Df  Sum Sq Mean Sq F value Pr(&amp;gt;F)&#xA;x$factor   2   4.142  2.0708  1.8948 0.1559&#xA;Residuals 97 106.009  1.0929        &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The F-test for the factor is testing simultaneously $H_0: \beta_1 = \beta_2 = 0$, i.e., the hypothesis that the factor in general is not significant. A common strategy is to first test this omnibus hypothesis before digging down which of the levels of the factor are different from each other.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, you can use the anova() function for full versus reduced model tests. For example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; x &amp;lt;- data.frame(rand=rnorm(100), factor=sample(c(&quot;A&quot;,&quot;B&quot;,&quot;C&quot;),100,replace=TRUE), &#xA;                  y1=rnorm(100), y2=rnorm(100))&#xA;&amp;gt; model1 &amp;lt;- lm(x$rand ~ x$factor + x$y1 + x$y2)&#xA;&amp;gt; model2 &amp;lt;- lm(x$rand ~ x$factor)&#xA;&amp;gt; anova(model2, model1)&#xA;&#xA;Analysis of Variance Table&#xA;&#xA;Model 1: x$rand ~ x$factor&#xA;Model 2: x$rand ~ x$factor + x$y1 + x$y2&#xA;Res.Df    RSS Df Sum of Sq      F Pr(&amp;gt;F)&#xA;1     97 105.06                           &#xA;2     95 104.92  2   0.13651 0.0618 0.9401&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which is a comparison of the full model with the factor and two covariates (y1 and y2) and the reduced model, where we assume that the slopes of the two covariates are both simultaneously equal to zero.&lt;/p&gt;&#xA;" OwnerUserId="1934" LastActivityDate="2010-11-30T21:46:50.907" CommentCount="5" />
  <row Id="5040" PostTypeId="2" ParentId="5004" CreationDate="2010-12-01T01:59:00.507" Score="2" ViewCount="" Body="&lt;p&gt;Google is your friend&#xA;I have found&lt;br&gt;&#xA;&lt;a href=&quot;http://www.acsu.buffalo.edu/~terejanu/files/tutorialUKF.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.acsu.buffalo.edu/~terejanu/files/tutorialUKF.pdf&lt;/a&gt;&lt;br&gt;&#xA;and&lt;br&gt;&#xA;&lt;a href=&quot;http://signal.hut.fi/kurssit/s884221/ukf.pdf&quot; rel=&quot;nofollow&quot;&gt;http://signal.hut.fi/kurssit/s884221/ukf.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;clear enough. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Modify your question if you have something more precise to ask. What don't you understand in the UKF?&lt;/p&gt;&#xA;" OwnerUserId="1709" LastActivityDate="2010-12-01T01:59:00.507" />
  <row Id="5041" PostTypeId="1" AcceptedAnswerId="5066" CreationDate="2010-12-01T05:22:59.070" Score="4" ViewCount="133" Body="&lt;p&gt;For classification, what theoretical results are between cross-validation estimate of accuracy and generalisation accuracy?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I particularly asking about results in a PAC-like framework where no assumptions are made that your function class contains the &quot;true&quot; function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would love to know if there are theorems of the form: If your leave-one-out cross validation error-rate is $\theta$ on $N$ examples, then your generalisation error rate is lower than $\theta+\varepsilon$ with probability $f(\theta, \varepsilon, N)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If so, what are the general proof techniques to obtain them? What is the theoretical framework?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If a fully general theorem is impossible, what extra conditions, if any, allow you to arrive at this type of conclusion?&lt;/p&gt;&#xA;" OwnerUserId="2067" LastActivityDate="2010-12-02T02:45:59.310" Title="Theoretical results for cross-validation estimation of classification accuracy?" Tags="&lt;classification&gt;&lt;cross-validation&gt;" AnswerCount="1" FavoriteCount="2" />
  <row Id="5042" PostTypeId="1" AcceptedAnswerId="5044" CreationDate="2010-12-01T06:49:03.427" Score="3" ViewCount="476" Body="&lt;p&gt;So, I have 16 trials in which I am trying to authenticate a person from a biometric trait using Hamming Distance. My threshold is set to 3.5. My data is below and only trial 1 is a True Positive:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Trial   Hamming Distance&#xA;1   0.34&#xA;2   0.37&#xA;3   0.34&#xA;4   0.29&#xA;5   0.55&#xA;6   0.47&#xA;7   0.47&#xA;8   0.32&#xA;9   0.39&#xA;10  0.45&#xA;11  0.42&#xA;12  0.37&#xA;13  0.66&#xA;14  0.39&#xA;15  0.44&#xA;16  0.39&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My point of confusion is that I am really unsure about how to make an ROC curve (FPR vs. TPR OR FAR vs. FRR) from this data. It doesn't really matter which one, but I'm just really confused about how to go about calculating it. Any help would be appreciated.&lt;/p&gt;&#xA;" OwnerUserId="1224" LastActivityDate="2010-12-01T09:21:31.927" Title="Calculate ROC curve for data" Tags="&lt;mathematical-statistics&gt;&lt;roc&gt;" AnswerCount="2" />
  <row Id="5043" PostTypeId="1" CreationDate="2010-12-01T06:57:43.170" Score="4" ViewCount="75" Body="&lt;p&gt;I'm currently looking at the unconstrained primal form of the one-vs-all classifier&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\sum\limits_{i=1}^{N_I} \sum\limits_{k=1,\atop k \neq y_i}^{N_K} L(1+ \mathbf{w_k}\cdot\mathbf{x_i}-\mathbf{w_{y_i}}\cdot\mathbf{x_i})$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where &lt;br/&gt;&#xA;$N_I$ is the number of instances, &lt;br/&gt;&#xA;$N_K$ is the number of classes, &lt;br/&gt;&#xA;$N_F$ is the number of features, &lt;br/&gt;&#xA;$X$ is a $N_K \times N_F$ data matrix&lt;br/&gt;&#xA;$y$ is a vector of class labels  &lt;br/&gt;&#xA;$W$ is an $N_K \times N_I$ matrix where each corresponds to the weights for the hyperplane splitting one class from the rest &lt;br/&gt;&#xA;$L$ is some arbitrary loss function&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My understanding is that the functional above tries to find a hyperplane for each class that maximizes the distance between the samples within the associated class to all other samples. If the hyperplanes are corerctly positioned then $\mathbf{w_k}\cdot\mathbf{x_i}$ should always be negative, $\mathbf{w_{y_i}}\cdot\mathbf{x_i}$ should always be positive and our loss function should come back fairly low. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm trying to implement this using $L$=hinge loss which I believe in this case will end up being ie $max(0,1+\mathbf{w_k}\cdot\mathbf{x_i}-\mathbf{w_{y_i}}\cdot\mathbf{x_i}$). However, in such a situation couldn't we end up with a situation where the hyperplanes classify all samples as belonging to every class. For example, if we are looking at the hyperplane seperating class 1 from all other classes, provided that $1+\mathbf{w_k}\cdot\mathbf{x_i}&amp;lt;\mathbf{w_{y_i}}\cdot\mathbf{x_i}$ then the incurred loss will be 0 despite $\mathbf{x_i}$ being classified as the wrong class. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where have I gone wrong? Or does it not matter if $\mathbf{w_k}\cdot\mathbf{x_i}$ is negative or positive provided that $\mathbf{w_{y_i}}\cdot\mathbf{x_i}$ ends up with a higher score? I have a feeling that perhaps my use of the hinge function as I've described here is incorrect but my use of Google today has only led to more confusion.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On a related note, how come there is a 1 in the functional above anyway? I would have though that it would make little difference. &lt;/p&gt;&#xA;" OwnerUserId="2023" LastEditorUserId="2023" LastEditorDisplayName="" LastEditDate="2010-12-01T13:47:27.890" LastActivityDate="2010-12-01T13:47:27.890" Title="Hinge Loss with one-vs-all classifier" Tags="&lt;classification&gt;&lt;loss-functions&gt;" />
  <row Id="5044" PostTypeId="2" ParentId="5042" CreationDate="2010-12-01T07:32:11.380" Score="3" ViewCount="" Body="&lt;p&gt;I suggest &lt;a href=&quot;http://home.comcast.net/~tom.fawcett/public_html/papers/ROC101.pdf&quot; rel=&quot;nofollow&quot;&gt;ROC Graphs: Notes and Practical Considerations for Reasearchers&lt;/a&gt; by Tom Fawcett, really an excellent read. As far as I understand your question, you will find everything you need in this paper.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: Inspired by Adam I also want to recommend my favorite R-package for this task: &lt;a href=&quot;http://rocr.bioinf.mpi-sb.mpg.de/&quot; rel=&quot;nofollow&quot;&gt;ROCR&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="264" LastEditorUserId="264" LastEditorDisplayName="" LastEditDate="2010-12-01T09:21:31.927" LastActivityDate="2010-12-01T09:21:31.927" CommentCount="2" />
  <row Id="5045" PostTypeId="1" CreationDate="2010-12-01T07:50:58.993" Score="3" ViewCount="94" Body="&lt;p&gt;Consider a series like CPI (inflation), which I know is composed of a series of component prices (e.g. meat prices, grain prices, non-food prices, etc.), which in turn are also composed of a series of component prices (e.g. average meat prices are a combination of pork, beef, and chicken prices).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I wanted to use a regression to find the components of CPI and their weightings, then is it better to use the final components (regress CPI against pork, beef, and chicken prices), or is it better to create fitted version of the middle components, then regress against CPI (regress average meat prices against pork, beef, and chicken prices, then fit a meat price series, then regress CPI against the fitted meat price series)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, I should note that some series which were significant when fitting the interim components in the layered method - the latter method - lose their significance when the regression is flattened.  So is is possible that the latter, layered method will return a better result because it includes the effect of more components?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have tried this with actual data and the latter method gives better results.&lt;/p&gt;&#xA;" OwnerUserId="1195" LastEditorUserId="1195" LastEditorDisplayName="" LastEditDate="2010-12-01T09:39:25.810" LastActivityDate="2011-01-10T16:46:43.897" Title="In a linear regression whose components can also be broken down, is it better to do multi-layered regression, or flatten to final components?" Tags="&lt;time-series&gt;&lt;regression&gt;" AnswerCount="1" FavoriteCount="2" />
  <row Id="5046" PostTypeId="2" ParentId="4556" CreationDate="2010-12-01T08:04:02.930" Score="0" ViewCount="" Body="&lt;p&gt;find steady state gain of transfer function H(s), we let s=0. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;since z=exp(sT), to find steady state gain of H(z), let z=exp(0)=1.&lt;/p&gt;&#xA;" LastActivityDate="2010-12-01T08:04:02.930" />
  <row Id="5047" PostTypeId="2" ParentId="5042" CreationDate="2010-12-01T08:46:39.030" Score="2" ViewCount="" Body="&lt;p&gt;Why do you want to make an ROC curve? Do you want to graph the curve for your dependent variables, or are you looking to use it as a test statistic in order to gauge the accuracy of your probability predictions (in which case you're looking for the AUC [area under the curve]).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're familiar with R, the verification package in R has two functions that you will find useful: roc.plot(), which will allow you to plot your ROC curve, and roc.area() which will allow you to calculate the AUC.&lt;/p&gt;&#xA;" OwnerUserId="2166" LastActivityDate="2010-12-01T08:46:39.030" CommentCount="1" />
  <row Id="5048" PostTypeId="1" CreationDate="2010-12-01T10:05:09.643" Score="1" ViewCount="94" Body="&lt;p&gt;I have a time serie that I want to analyse through a wavelet decomposition.&lt;br&gt;&#xA;I am using the R package &lt;strong&gt;WaveThres&lt;/strong&gt;.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am interested in the wavelet autocorrelation, but I struggle to understand what does it mean precisely.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have from the book &lt;a href=&quot;http://www.springer.com/statistics/statistical+theory+and+methods/book/978-0-387-75960-9&quot; rel=&quot;nofollow&quot;&gt;Wavelet Methods in Statistics with R&lt;/a&gt; the following formula  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\Psi_j(\tau)=\sum_{k}\phi_{j,k}(0)\phi_{j,k}(\tau)$  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\tau\in\mathbb{Z}$ being the lag of the autocorrelation&lt;br&gt;&#xA;and&lt;br&gt;&#xA;$\left \{\phi_{j,k}(t)=\phi_{j,k-t}  \right \}_{j,k}$ a set of non decimated wavelets&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would really appreciate to understand the meaning of this formula, and (why/if) it is different from performing a multi resolution analysis (MRA) and computing the Pearson autocorrelation coefficient on a detail.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;fRed&lt;/p&gt;&#xA;" OwnerUserId="1709" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-12-01T10:18:28.720" LastActivityDate="2010-12-02T22:25:41.843" Title="Wavelet auto correlation" Tags="&lt;correlation&gt;&lt;autocorrelation&gt;&lt;wavelet&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="5049" PostTypeId="1" CreationDate="2010-12-01T11:28:41.573" Score="6" ViewCount="109" Body="&lt;p&gt;Let $A$ be a finite set and suppose we want to compute the size of some subset $X$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Motivation&lt;/em&gt;: If we can generate elements $x$ of $A$ uniformly at random, then we can estimate the size of $A$ by random sampling.  That is, we take $n$ random samples from $A$, if $m$ of them belong to $X$, then $|X|/|A| \approx m/n$.  Unfortunately, for what I do, usually $|A|$ is massive and $|X|$ (while massive) is quite small with respect to $|A|$.  So if I attempt to perform the above estimation, I'm likely to get $m=0$, which, while not useless, is not really all that satisfying.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, I have an idea that I'm hoping will speed up this process.  Instead of throwing darts at a massive dart board, why don't I throw balls?  That is, instead of sampling elements $x \in A$, we sample subsets of $A$.  Surely I should be able to infer something about the density of $X$ in $A$ from this experiment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose $A$ is equipped with a metric $d(x,y)$ (I have in mind the Hamming distance).  For any $y \in A$ let $Y(y)=\{x \in A:d(x,y) \leq k\}$ be the closed ball of radius $k$ in $A$ centered at $t$.  Since we can sample elements $x \in A$ uniformly at random, we can sample $k$-balls $Y_k(t)$ uniformly at random.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose (a) every $x \in A$ belong to exactly the same number of $k$-balls and (b) every $k$-ball has the same size $r$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now suppose I generate $k$-balls $Y_1,Y_2,\ldots,Y_n$ uniformly at random and suppose $m=\sum_{i=1}^n |Y_i \cap X|$.  It seems we can estimate $|A|$ in a similar way, that is $|X|/|A| \approx \frac{m}{rn}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my questions are:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Am I correct in that we can approximate $|X|$ this way?  If so, I doubt I'm the first to think of this, so is there a name for this method?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I did actually test this on some sets, and it seems to match what I claim.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Are there any drawbacks to this approach?  (e.g. is it less accurate?  do I need more samples?)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="386" LastEditorUserId="386" LastEditorDisplayName="" LastEditDate="2010-12-01T12:25:50.300" LastActivityDate="2010-12-01T16:29:30.777" Title="Can we estimate the size of a subset X of a set A, by randomly sampling subsets of A?" Tags="&lt;estimation&gt;" AnswerCount="3" CommentCount="2" />
  <row Id="5050" PostTypeId="2" ParentId="5049" CreationDate="2010-12-01T12:23:22.067" Score="2" ViewCount="" Body="&lt;p&gt;OK, try reading the wikipedia page for &lt;a href=&quot;http://en.wikipedia.org/wiki/Monte_Carlo_integration&quot; rel=&quot;nofollow&quot;&gt;Monte Carlo integration&lt;/a&gt;. You'll see they mention a stratified version. Stratification is the technical term in statistics for what you attempt: subdividing in subsets (subsamples). I guess the references can help you further.&lt;/p&gt;&#xA;" OwnerUserId="2036" LastActivityDate="2010-12-01T12:23:22.067" />
  <row Id="5051" PostTypeId="2" ParentId="5049" CreationDate="2010-12-01T12:34:07.033" Score="2" ViewCount="" Body="&lt;p&gt;I assume your measure is finite. WLOG it can be a probability.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first procedure you mention is the good old &lt;strong&gt;empirical probability estimate&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\hat{P}(Y\in X)= | \{ x_i \in X\} | /n $&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(there montecarlo estimate of an inetgral is a good interpretation also). In high dimension it does not work since $\{x_i\in X\}$ is likely to be empty for typical A. As you have noticed, you need regularization. How sophisticate regularisation you need is related to the dimension of your space. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;An idea is to enlarge $X$ or even give a weight to $x_i$ that is not in $X$ according to its distance to $X$, this is what I ould call &lt;strong&gt;kernel probability estimate&lt;/strong&gt; (by analogy with &lt;a href=&quot;http://en.wikipedia.org/wiki/Kernel_density_estimation&quot; rel=&quot;nofollow&quot;&gt;kernel density estimate&lt;/a&gt;): &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\hat{P}(Y\in X)= 1/(c(k) n)\sum_{i} K(d(x_i,X)/k)   $&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $K$ is a kernel that integrates to $1$ (in your case it can be $K(x)=1\{x\leq 1\}$ but gaussian kernel has good properties) and $c(k)$ a well chosen normalization constant (i.e such that $\hat{P}(Y\in A)=1$).&lt;/p&gt;&#xA;" OwnerUserId="223" LastEditorUserId="223" LastEditorDisplayName="" LastEditDate="2010-12-01T16:08:12.340" LastActivityDate="2010-12-01T16:08:12.340" />
  <row Id="5052" PostTypeId="2" ParentId="4892" CreationDate="2010-12-01T13:10:35.207" Score="0" ViewCount="" Body="&lt;p&gt;The answer is 6. See the edits above for details.&lt;/p&gt;&#xA;" OwnerUserId="1351" LastActivityDate="2010-12-01T13:10:35.207" />
  <row Id="5053" PostTypeId="2" ParentId="4983" CreationDate="2010-12-01T13:23:03.733" Score="0" ViewCount="" Body="&lt;pre&gt;&lt;code&gt;sv_num: one added to the number of support vectors&#xA;     b: offset&#xA; alpha: 0, then &quot;a&quot; corresponding to the support vectors, then only zeros&#xA; index: link index between &quot;a&quot; and alpha, &quot;x&quot; and supvec &#xA;supvec: 0, then &quot;x&quot; corresponding to the support vectors&#xA;     a: weights_i (between 0 and C) times y_i&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1351" LastActivityDate="2010-12-01T13:23:03.733" />
  <row Id="5054" PostTypeId="1" CreationDate="2010-12-01T16:14:58.283" Score="3" ViewCount="157" Body="&lt;p&gt;Say we are studying Twitter hashtags over time.  We monitor how popular they are day to day.  Some hashtags may be volatile (i.e. &quot;lunch&quot;, &quot;Celtics&quot;, &quot;Friday&quot;).  Their popularity rises and falls frequently. Some hashtags may be in the process of becoming unpopular (i.e. &quot;Gulf oil spill&quot;, &quot;Transformers 2&quot;, &quot;Christine O'Donnell&quot;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a mathematical model that can distinguish between a hashtag that has temporarily fallen in popularity but is likely to go up in popularity later and a hashtag that is sinking and likely to stay sunk in popularity?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;thanks&lt;/p&gt;&#xA;" OwnerUserId="2217" LastActivityDate="2011-05-12T20:00:40.270" Title="Is there a mathematical model that distinguishes between volatility and trend?" Tags="&lt;volatility-forecasting&gt;" AnswerCount="3" CommentCount="3" FavoriteCount="1" />
  <row Id="5055" PostTypeId="2" ParentId="5049" CreationDate="2010-12-01T16:29:30.777" Score="3" ViewCount="" Body="&lt;p&gt;For any subset $Y$ of $A$, let $\pi(Y)$ be the probability you will select it in your sampling.  You have described a random variable&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$f(Y) = |Y \cap X|.$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;em&gt;total&lt;/em&gt; of $f$ in the population of subsets of $A$ is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\tau(X) = \sum_{Y \subset A}|Y \cap X| = 2^{|A|-1}|X|.$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From a sample (with replacement) of subsets of $A$, say $Y_1, Y_2, \ldots, Y_m$, the &lt;a href=&quot;http://www.math.umt.edu/patterson/549/Unequalp.pdf&quot; rel=&quot;nofollow&quot;&gt;Hansen-Hurwitz Estimator&lt;/a&gt; obtains an unbiased estimate of this total as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\hat{f}_\pi = \sum_{i=1}^{m} \frac{|Y_i \cap X|}{\pi(Y_i)} .$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Dividing this by $2^{|A|-1}|A|$ therefore estimates $|X|/|A|$.  The variance of $\hat{f}_\pi$ is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\text{Var}(\hat{f}_\pi) = \frac{1}{m} \sum_{Y \subset A} \pi(Y) \left( \frac{|Y \cap X|}{\pi(Y)} - 2^{|A|-1}|X| \right)^2\text{.}$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Dividing this by $2^{2(|A|-1)}|A|^2$ yields the sampling variance of $|X|/|A|$.  Given $A$, $X$, and a proposed sampling procedure (which specifies $\pi(Y)$ for all $Y \subset A$), choose a value of $m$ (the sample size) for which the estimation variance becomes acceptably small.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-12-01T16:29:30.777" CommentCount="1" />
  <row Id="5056" PostTypeId="1" CreationDate="2010-12-01T18:25:03.223" Score="2" ViewCount="301" Body="&lt;p&gt;Given the support vectors of a linear SVM, how can I compute the equation of the decision boundary?&lt;/p&gt;&#xA;" OwnerUserId="2221" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-12-03T08:06:06.830" LastActivityDate="2010-12-03T08:06:06.830" Title="Computing the decision boundary of a linear SVM model" Tags="&lt;machine-learning&gt;&lt;svm&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="1" />
  <row Id="5057" PostTypeId="2" ParentId="5056" CreationDate="2010-12-01T20:04:56.970" Score="1" ViewCount="" Body="&lt;p&gt;It's a linear combination of the support vectors where the coefficients are given by the Lagrange multipliers corresponding to these support vectors.  &lt;/p&gt;&#xA;" OwnerUserId="881" LastActivityDate="2010-12-01T20:04:56.970" />
  <row Id="5058" PostTypeId="1" CreationDate="2010-12-01T20:16:27.117" Score="2" ViewCount="163" Body="&lt;p&gt;In a social experiment that I was conducting, I was trying to count the number of people each user contacted in a period of 10 days. The population size was 100 for the experiment. Based on the values that I calculated, I fit a negative binomial distribution to the data (the Q-Q plot is given below).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Conventional wisdom says that most networks amongst humans follow a power law distribution. I am guessing that my population size is too small to make a full conclusion about anything but is there any kind of relation between a negative binomial distribution and a power law distribution? I am asking this because I read a few days back that Normal Distribution and Gamma distribution (whose discrete analogue is the negative binomial) have a special role in that many other distributions can be derived from the Gamma distribution. I am wondering if this is true even with the power law distribution. I am a beginner in statistics so kindly point me in the right direction if I am out of track.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/MMla7.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="2164" LastActivityDate="2010-12-02T04:00:56.167" Title="Is there any relation between Power Law and Negative Binomial distribution?" Tags="&lt;r&gt;&lt;distributions&gt;&lt;probability&gt;&lt;modeling&gt;" AnswerCount="1" CommentCount="5" />
  <row Id="5059" PostTypeId="2" ParentId="4551" CreationDate="2010-12-01T20:17:08.233" Score="6" ViewCount="" Body="&lt;p&gt;interpreting &lt;code&gt;Probability(data | hypothesis)&lt;/code&gt; as &lt;code&gt;Probability(hypothesis | data)&lt;/code&gt; without the application of Bayes' theorem.&lt;/p&gt;&#xA;" OwnerUserId="961" LastActivityDate="2010-12-01T20:17:08.233" CommunityOwnedDate="2010-12-01T20:17:08.233" />
  <row Id="5060" PostTypeId="2" ParentId="4551" CreationDate="2010-12-01T21:09:50.050" Score="2" ViewCount="" Body="&lt;p&gt;Probably not as applicable to psych stats (or is it? I'm not sure) but failing to account for a split plot design in an analysis of an experiment.  I've seem way too many people do this.&lt;/p&gt;&#xA;" OwnerUserId="1028" LastActivityDate="2010-12-01T21:09:50.050" CommunityOwnedDate="2010-12-01T21:09:50.050" CommentCount="1" />
  <row Id="5061" PostTypeId="2" ParentId="5026" CreationDate="2010-12-01T21:17:22.760" Score="3" ViewCount="" Body="&lt;p&gt;I'd add some observations to what's been said...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;AI is a very broad term for anything that has to do with machines doing reasoning-like or sentient-appearing activities, ranging from planning a task or cooperating with other entities, to learning to operate limbs to walk. A pithy definition is that AI is anything computer-related that we don't know how to do well yet. (Once we know how to do it well, it generally gets its own name and is no longer &quot;AI&quot;.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's my impression, contrary to Wikipedia, that Pattern Recognition and Machine Learning are the same field, but the former is practiced by computer-science folks while the latter is practiced by statisticians and engineers. (Many technical fields are discovered over and over by different subgroups, who often bring their own lingo and mindset to the table.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data Mining, in my mind anyhow, takes Machine Learning/Pattern Recognition (the techniques that work with the data) and wrap them in database, infrastructure, and data validation/cleaning techniques. &lt;/p&gt;&#xA;" OwnerUserId="1764" LastActivityDate="2010-12-01T21:17:22.760" CommunityOwnedDate="2010-12-01T21:17:22.760" CommentCount="3" />
  <row Id="5062" PostTypeId="1" AcceptedAnswerId="5063" CreationDate="2010-12-01T21:43:00.990" Score="1" ViewCount="204" Body="&lt;p&gt;I have data points from a half circle and I already know the radius. I want to find the circle which best fits the points using a fixed radius. How can I do this? If I solve the problem using a typical circle fit algorithm the radius is too unstable due to &quot;noise&quot;.&lt;/p&gt;&#xA;" OwnerUserId="2223" LastActivityDate="2010-12-01T22:06:55.510" Title="Geometric circle fitting with known radius" Tags="&lt;nonlinear-regression&gt;" AnswerCount="1" />
  <row Id="5063" PostTypeId="2" ParentId="5062" CreationDate="2010-12-01T22:06:55.510" Score="3" ViewCount="" Body="&lt;p&gt;It depends on how the points might depart from the circle.  If they do so through measurement error, a natural model is that their locations are binormally distributed with the coordinates $x$ and $y$ uncorrelated, of equal variances.  This leads to a difficult model to fit, but if the errors are not too great compared to the radius, we can approximate it closely as a normal distribution in the radial direction only (because tangential displacements do not move a point off the circle, to first order).  This suggests a model of the form&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Distance($(x_i, y_i)$, $(\alpha, \beta)$) = $r + \epsilon_i$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $r$ is given, the $(x_i, y_i)$ are the data, $(\alpha, \beta)$ are coordinates of the center (to be estimated), and the $\epsilon_i$ are identically and independently distributed with zero mean.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To estimate the parameters you could use least squares or maximum likelihood (among other approaches).  For example, least squares would seek values of $\alpha$ and $\beta$ minimizing&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\sum_i \left( \sqrt{(x_i - \alpha)^2 + (y_i - \beta)^2} - r \right)^2 \text{.}$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This problem has no closed-form solution but can readily be solved numerically.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a clever approach in the image processing world: create an image of the boundaries of the circles of radius $r$ centered at the data points.  In so doing, &lt;em&gt;add&lt;/em&gt; the intensities of the boundaries wherever they overlap.  Any grid point of greatest intensity is a reasonable solution.  (In effect this is a kernel density estimate.  You are using a kernel of the form $K(x,y)$ that is zero except for values of $(x,y)$ for which $x^2+y^2$ is close to $r^2$ and you are convolving that kernel with the data.)&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-12-01T22:06:55.510" CommentCount="4" />
  <row Id="5064" PostTypeId="2" ParentId="5026" CreationDate="2010-12-01T23:57:48.737" Score="3" ViewCount="" Body="&lt;p&gt;I'm most familiar with the machine-learning - data mining axis - so I'll concentrate on that:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Machine learning tends to be interested in inference in non-standard situations, for instance non-i.i.d. data, active learning, semi-supervised learning, learning with structured data (for instance strings or graphs).  ML also tends to be interested in theoretical bounds on what is learnable, which often forms the basis for the algorithms used (e.g. the support vector machine).  ML tends to be of a Bayesian nature.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data mining is interested in finding patterns in data that you don't already know about.  I'm not sure that is significantly different from exploratory data analysis in statistics, whereas in machine learning there is generally a more well-defined problem to solve.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ML tends to be more interested in small datasets where over-fitting is the problem and data mining tends to be interested in large-scale datasets where the problem is dealing with the quantities of data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Statistics and machine learning provides many of the basic tools used by data miners.&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2010-12-01T23:57:48.737" CommunityOwnedDate="2010-12-01T23:57:48.737" />
  <row Id="5065" PostTypeId="1" AcceptedAnswerId="5067" CreationDate="2010-12-02T01:24:18.253" Score="1" ViewCount="243" Body="&lt;p&gt;I want to find $\theta$ such that&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ \theta = argmin_{\theta} \left( \left|\left| Y - \sum_{i=1}^k \theta_i X_i \right|\right| \right) $&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $X_i$ and $Y$ are N x N matrices and $\theta$ is a weight vector that specifies how to linearly combine the $k$ $X$'s to approximate Y.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This smells like a linear optimization problem though I'm unskilled in this kind of math and can't seem to formulate it as a linear program.  I'd also be curious if anyone has any advice on how to learn this kind of math and/or problem formulation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="2224" LastEditorUserId="223" LastEditorDisplayName="" LastEditDate="2010-12-17T07:57:59.127" LastActivityDate="2010-12-17T07:57:59.127" Title="How can I find the best linear combination of a set of matrices to approximate a target matrix?" Tags="&lt;regression&gt;&lt;optimization&gt;&lt;matrix&gt;" AnswerCount="3" />
  <row Id="5066" PostTypeId="2" ParentId="5041" CreationDate="2010-12-02T02:45:59.310" Score="1" ViewCount="" Body="&lt;p&gt;I don't know much about these kinds of proofs, but I think John Langford's thesis might be a good reference. Here's a relevant page: &lt;a href=&quot;http://hunch.net/~jl/projects/prediction_bounds/prediction_bounds.html&quot; rel=&quot;nofollow&quot;&gt;http://hunch.net/~jl/projects/prediction_bounds/prediction_bounds.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and the probably relevant section of the thesis: &lt;a href=&quot;http://hunch.net/~jl/projects/prediction_bounds/thesis/mathml/thesisse15.xml#x22-300004.1&quot; rel=&quot;nofollow&quot;&gt;http://hunch.net/~jl/projects/prediction_bounds/thesis/mathml/thesisse15.xml#x22-300004.1&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2077" LastActivityDate="2010-12-02T02:45:59.310" />
  <row Id="5067" PostTypeId="2" ParentId="5065" CreationDate="2010-12-02T03:39:19.017" Score="2" ViewCount="" Body="&lt;p&gt;$\theta = argmin_{\theta}  (Y - \sum_{i=1}^k \theta_i X_i)$ would be an affine function in $\theta$ and hence an unconstrained linear program.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But $\theta = argmin_{\theta} || Y - \sum_{i=1}^k \theta_i X_i ||$ has an arbitrary norm.  Fortunately, norms are convex and convexity is preserved under compositions with affine functions, so the problem is simply an unconstrained convex optimization program.  They can be solved easily.  Here is an example from MATLAB using &lt;a href=&quot;http://cvxr.com/&quot; rel=&quot;nofollow&quot;&gt;cvx&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Initialize 5 random normal 2x2 matrices and try to regress $Y$ on the 4 $X_i$'s in the Euclidean norm in this example (to change this, just use a different norm() function, of which cvx has many).  If Y and the $X_i$'s are linearly independent, since there are only 4 degrees of freedom, we should be able to achieve 0 error, which we do.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;N=2;&#xA;Y=randn(N,N);&#xA;X1=randn(N,N);&#xA;X2=randn(N,N);&#xA;X3=randn(N,N);&#xA;X4=randn(N,N);&#xA;&#xA;cvx_begin&#xA;    variable theta(4);&#xA;    minimize(norm(Y-theta(1)*X1-theta(2)*X2-theta(3)*X3-theta(4)*X4))&#xA;cvx_end&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Status: Solved&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Optimal value (cvx_optval): +1.50457e-11&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;blockquote&gt;&#xA;    &lt;p&gt;norm(Y)&lt;/p&gt;&#xA;  &lt;/blockquote&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;ans = 1.3731&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The norm of Y is about 1.4, and the norm of the difference given our thetas is effectively zero.&lt;/p&gt;&#xA;" OwnerUserId="1815" LastEditorUserId="1815" LastEditorDisplayName="" LastEditDate="2010-12-02T03:47:30.290" LastActivityDate="2010-12-02T03:47:30.290" CommentCount="3" />
  <row Id="5068" PostTypeId="2" ParentId="5065" CreationDate="2010-12-02T03:54:35.287" Score="0" ViewCount="" Body="&lt;p&gt;Edit: Sorry, misunderstood the question. My Answer doesn't make sense in the context of my newfound understanding of the question.&lt;/p&gt;&#xA;" OwnerUserId="2166" LastActivityDate="2010-12-02T03:54:35.287" />
  <row Id="5069" PostTypeId="2" ParentId="5058" CreationDate="2010-12-02T04:00:56.167" Score="2" ViewCount="" Body="&lt;p&gt;There are many power-law distributions, so you have a lot of possible models.  You might start by trying to fit a &lt;a href=&quot;http://en.wikipedia.org/wiki/Logarithmic_distribution&quot; rel=&quot;nofollow&quot;&gt;log-series distribution&lt;/a&gt;, which is a limiting case of the negative binomial.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Don't think a priori that you have a mixture distribution as suggested by whuber until you've estimated model parameters and done at least a goodness of fit test.  Long-tail distributions, like power-law, log-series, Zipf, etc., typically have what look like outliers in the right-hand tail; their separation from the bulk of the observations is just an artifact of (relatively) small sample size.  Mixtures are a pain in the butt to estimate, since some regions overlap.  You can often avoid that sort of problem by stepping up your modeling one level with something like Poisson regression, assuming you have some covariate (predictor) data about each user -- this basically does the mixing for you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Johnson, Kemp, and Kotz reference given at the end of the referenced Wikipedia article has everything you'd ever want to know about all these distributions, including many methods of parameter estimation.&lt;/p&gt;&#xA;" OwnerUserId="1920" LastActivityDate="2010-12-02T04:00:56.167" CommentCount="2" />
  <row Id="5070" PostTypeId="1" AcceptedAnswerId="5071" CreationDate="2010-12-02T04:20:55.457" Score="2" ViewCount="220" Body="&lt;p&gt;If we have a loaded coin that plays out 75% heads, 25% tails, what would be the best way to bet on the outcome of each of $n$ trials? How could we maximize our probability of winning?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possible to generalize for a coin that's loaded $n:(100-n)$?&lt;/p&gt;&#xA;" OwnerUserId="2226" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-12-02T09:29:43.640" LastActivityDate="2011-02-02T00:04:57.403" Title="Maximizing probability of winning on loaded coin" Tags="&lt;probability&gt;&lt;dice&gt;" AnswerCount="3" />
  <row Id="5071" PostTypeId="2" ParentId="5070" CreationDate="2010-12-02T04:54:20.090" Score="5" ViewCount="" Body="&lt;p&gt;1/ do you already know the bias n?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2/ if yes (actually you need only to know which side of the coin is heavier) then you cannot do better than always bet on this side.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the long term, you'll have a n % success ratio. You cannot do better because the trials are independent.&lt;/p&gt;&#xA;" OwnerUserId="1709" LastActivityDate="2010-12-02T04:54:20.090" CommentCount="2" />
  <row Id="5072" PostTypeId="2" ParentId="5070" CreationDate="2010-12-02T05:34:57.873" Score="5" ViewCount="" Body="&lt;p&gt;In what follows, I will assume you mean someone gives you 1:1 odds on the loaded coin.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You are looking for the &lt;a href=&quot;http://en.wikipedia.org/wiki/Kelly_criterion&quot; rel=&quot;nofollow&quot;&gt;Kelly criterion&lt;/a&gt;, which states:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ f^* = \frac{ b p - q }{ b } $$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where (just copying from wikipedia) $f^*$ is the fraction of your bank roll, $b$ is the fraction payout (in your case presumably $b=1$, i.e. a dollar investment gives you a dollar return if heads is thrown) and $q = 1-p$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For your example, the fraction the Kelly criterion says to invest is $f^* = .75 - .25 = 0.5$.  i.e. The Kelly criterion tells you to invest half of your savings each time you play.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As intuition, you want to make sure you don't invest &lt;em&gt;all&lt;/em&gt; your money on a loaded coin as just one bad throw will deplete your savings.  Understanding what function to maximize for is what the Kelly criterion is instructive for.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Kelly criterion requires that you maximize your winnings based on:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ p\  \text{ln}(1 + b x) + (1-p)\ \text{ln}(1 - x) $$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maximizing for $x$ yields the equation for $f^*$ above.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unfortunately I am a little unclear as to why this particular equation is being maximized as opposed to some other equation.  I have heard an information theoretic argument for why this is so (notice that the equation above looks very much like an entropy equation) but, sadly, still remain puzzled.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Well, I feel pretty dumb.  As whuber points out in the comments, the Kelly criterion is quite easy to derive.  If we assume a you want to invest a constant proportion of your savings at each trial, call it $r$, then your winnings after $n$ trials, $W_n$, for an initial savings of $W_0$, will be:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ W_n = (1 + b r)^{p n} (1 - r)^{(1-p) n} W_0 $$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Taking logarithms, noticing that this equation, as a function of $r$, has one maximum, then setting the derivative equal to 0 and solving gives the Kelly criterion formula for $r = f^*$ as above.&lt;/p&gt;&#xA;" OwnerUserId="2168" LastEditorUserId="2168" LastEditorDisplayName="" LastEditDate="2010-12-02T15:02:08.493" LastActivityDate="2010-12-02T15:02:08.493" CommentCount="7" />
  <row Id="5073" PostTypeId="1" AcceptedAnswerId="5075" CreationDate="2010-12-02T06:09:21.087" Score="2" ViewCount="659" Body="&lt;p&gt;I have a data-frame whose first column is the name of an item and the second column is the frequency of that item in the dataset. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; names            freq&#xA;1 tomato           7&#xA;2 potato           4&#xA;3 cabbage          5&#xA;4 sukuma-wiki      8&#xA;5 terere           20&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I would like to have a stacked bar column that depicts the proportion of each entry on the chart. How do you handle coloring of the stacked bar when presented with over sixty entries? what is the easiest way to do this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you&lt;/p&gt;&#xA;" OwnerUserId="201" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-02T08:56:49.973" LastActivityDate="2010-12-02T08:56:49.973" Title="Stacked bar plot" Tags="&lt;r&gt;&lt;data-visualization&gt;&lt;barplot&gt;" AnswerCount="3" CommentCount="1" FavoriteCount="1" />
  <row Id="5074" PostTypeId="2" ParentId="5073" CreationDate="2010-12-02T06:27:26.500" Score="3" ViewCount="" Body="&lt;p&gt;For the coloring, either you specify a list of colors or you generate them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the latter, I suggest you execute this code&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;n = 32;&#xA;main.name = paste(&quot;color palettes; n=&quot;,n)&#xA;ch.col = c(&quot;rainbow(n, start=.7, end=.1)&quot;, &quot;heat.colors(n)&quot;, &quot;terrain.colors(n)&quot;,            &quot;topo.colors(n)&quot;, &quot;cm.colors(n)&quot;);&#xA;&#xA;nt &amp;lt;- length(ch.col)&#xA;i &amp;lt;- 1:n; &#xA;j &amp;lt;- n/nt; &#xA;d &amp;lt;- j/6; &#xA;dy &amp;lt;- 2*d;&#xA;&#xA;plot(i,i+d, type=&quot;n&quot;, yaxt=&quot;n&quot;, xaxt=&quot;n&quot;, ylab=&quot;&quot;, , xlab =&quot;&quot;, main=main.name)   #yaxt=&quot;n&quot; set no y axie label and tick.&#xA;for (k in 1:nt) {&#xA;rect(i-.5, (k-1)*j+ dy, i+.4, k*j, col = eval(parse(text=ch.col[k])), border = &quot;grey&quot;);&#xA;text(2.5*j, k * j + dy/2, ch.col[k])&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;taken from the blog &lt;a href=&quot;http://statisticsr.blogspot.com/2008/07/color-scale-in-r.html&quot; rel=&quot;nofollow&quot;&gt;http://statisticsr.blogspot.com/2008/07/color-scale-in-r.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Barplotting should be done with ?barplot&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;DF=data.frame(names=c(&quot;tomato&quot;, &quot;potato&quot;, &quot;cabbage&quot;, &quot;sukuma-wiki&quot;, &quot;terere&quot;), freq=c(7,4,5,8,20))&#xA;barplot(as.matrix(DF[,2]), col=heat.colors(length(DF[,2])), legend=DF[,1], xlim=c(0,9), width=2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1709" LastEditorUserId="1709" LastEditorDisplayName="" LastEditDate="2010-12-02T07:33:43.230" LastActivityDate="2010-12-02T07:33:43.230" CommentCount="3" />
  <row Id="5075" PostTypeId="2" ParentId="5073" CreationDate="2010-12-02T07:26:52.260" Score="7" ViewCount="" Body="&lt;p&gt;With 60 distinct categories, I feel you may have a hard time making that an effective graphic. You may want to consider a regular bar-chart that is sorted in ascending or descending order. Whether or not these are counts or percentages is up to you. Maybe something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(ggplot2)&#xA;df$names &amp;lt;- reorder(df$names, -df$freq) #Reorders into ascending order&#xA;qplot(x = names, y = freq, data = df, geom = &quot;bar&quot;) + coord_flip()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;EDIT:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To make a stacked bar chart with &lt;code&gt;ggplot&lt;/code&gt;, we set the &lt;code&gt;x  = 1&lt;/code&gt; since we will have only one column. We will use the &lt;code&gt;fill&lt;/code&gt; argument to add color:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;qplot(x = factor(1), y = freq, data = df, geom = &quot;bar&quot;, fill = names) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Also of interest: a stacked bar chart is pretty darn close to being a pie chart. You can transform the coordinate system of ggplot charts with  &lt;code&gt;+ coord_polar(theta = &quot;y&quot;)&lt;/code&gt; to make a pie chart from the stacked bar chart above.&lt;/p&gt;&#xA;" OwnerUserId="696" LastEditorUserId="696" LastEditorDisplayName="" LastEditDate="2010-12-02T07:56:40.507" LastActivityDate="2010-12-02T07:56:40.507" CommentCount="2" />
  <row Id="5076" PostTypeId="2" ParentId="5073" CreationDate="2010-12-02T07:35:19.660" Score="4" ViewCount="" Body="&lt;p&gt;I doubt you fill find a suitable range of distinct colours with so much categories. Anyway, here are some ideas:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;For stacked barchart, you need &lt;code&gt;barplot()&lt;/code&gt; with &lt;code&gt;beside=FALSE&lt;/code&gt; (which is the default) -- this is in base R (@Chase's solution with &lt;a href=&quot;http://cran.r-project.org/web/packages/ggplot2/index.html&quot; rel=&quot;nofollow&quot;&gt;ggplot2&lt;/a&gt; is good too)&lt;/li&gt;&#xA;&lt;li&gt;For generating a color ramp, you can use the &lt;a href=&quot;http://cran.r-project.org/web/packages/RColorBrewer/index.html&quot; rel=&quot;nofollow&quot;&gt;RColorBrewer&lt;/a&gt; package; the example shown by @fRed can be reproduced with &lt;code&gt;brewer.pal&lt;/code&gt; and any one of the diverging or sequential palettes. However, the number of colour is limited, so you will need to recycle them (e.g., every 6 items)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Here is an illustration:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(RColorBrewer)&#xA;x &amp;lt;- sample(LETTERS[1:20], 100, replace=TRUE)&#xA;tab &amp;lt;- as.matrix(table(x))&#xA;my.col &amp;lt;- brewer.pal(6, &quot;BrBG&quot;) # or brewer.pal(6, &quot;Blues&quot;)&#xA;barplot(tab, col=my.col)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;There is also the &lt;a href=&quot;http://cran.r-project.org/web/packages/colorspace/index.html&quot; rel=&quot;nofollow&quot;&gt;colorspace&lt;/a&gt; package, which has a nice accompagnying vignette about the design of good color schemes. Check also Ross Ihaka's course on &lt;a href=&quot;http://www.stat.auckland.ac.nz/~ihaka/courses/787/&quot; rel=&quot;nofollow&quot;&gt;Topic in Computational Data Analysis and Graphics&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, a better way to display such data is probably to use a so-called Cleveland dot plot, i.e.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dotchart(tab)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-12-02T07:35:19.660" CommentCount="3" />
  <row Id="5077" PostTypeId="1" CreationDate="2010-12-02T10:31:27.627" Score="4" ViewCount="358" Body="&lt;p&gt;I'm trying to implement the hit and run MCMC algorithm, but I'm having a bit of trouble understanding how to go about it. The general idea, is as follows:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;To generate a proposal jump in MH, we:&lt;/p&gt;&#xA;  &#xA;  &lt;ul&gt;&#xA;  &lt;li&gt;Generate a direction $d$ from a&#xA;  distribution on the surface of the&#xA;  unit sphere $\mathcal{O}$   &lt;/li&gt;&#xA;  &lt;li&gt;Generate&#xA;  a signed distance $\lambda$ along the&#xA;  constrained space.&lt;/li&gt;&#xA;  &lt;/ul&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;However, I have no idea of how I should go about implementing this in R (or any other language).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone have a snippet of code that would point me in the right direction?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;BTW, I not that interested in a library that does this method, I want to try and code it up myself. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks.&lt;/p&gt;&#xA;" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-12-02T11:21:44.393" LastActivityDate="2011-01-28T18:42:16.323" Title="Hit and run MCMC" Tags="&lt;r&gt;&lt;bayesian&gt;&lt;mcmc&gt;&lt;hit-and-run&gt;" AnswerCount="2" CommentCount="6" FavoriteCount="2" />
  <row Id="5078" PostTypeId="1" AcceptedAnswerId="5085" CreationDate="2010-12-02T10:32:07.580" Score="1" ViewCount="205" Body="&lt;p&gt;Sorry for a possibly ignorant question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have fit a mixed-effects model using the lmer function from the lme4 package, and the main fixed effect (a factor with three levels) in the model was significant according to a run with pvals.fnc (from the languageR package).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To illustrate the effect in an appealing way, I would like to plot three bars with the &quot;baseline condition&quot; (the intercept including the effect level 1), the &quot;experimental condition 1&quot; (the intercept + the effect level 2), the &quot;experimental condition 2&quot; (intercept + effect level 3), and the corresponding confidence intervals for these conditions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, how do I do that? The function plotLMER.fnc used to do that, but has stopped working for me (lme4_0.999375-32; languageR_1.0).&lt;br&gt;&#xA;Model: mylmer &amp;lt;- lmer(outcome ~ (1|subject) + (1|item) + Factor, data)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I have MCMC output in the form of:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;$fixed&#xA;              Estimate MCMCmean HPD95lower HPD95upper  pMCMC Pr(&amp;gt;|t|)&#xA;(Intercept)     0.4728   0.4718     0.2250     0.7368 0.0010   0.0008&#xA;Factor2        -0.0420  -0.0420    -0.1732     0.0931 0.5414   0.5710&#xA;Factor3        -0.1643  -0.1631    -0.3153    -0.0112 0.0328   0.0508&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My idea was to construct the conditions from this data by simply using the intercept as the baseline condition, and construct the experimental conditions by adding the effect of each factor level to the intercept. The new (intercept + effect combined) CI would be constructed by turning the HPD interval into a standard deviation and then use the square root of the sum of squared standard deviations (&lt;a href=&quot;http://www.cartage.org.lb/en/themes/sciences/chemistry/miscellenous/helpfile/Erroranalysis/AdditionSubtraction/AdditionSubtraction.htm&quot; rel=&quot;nofollow&quot;&gt;like here&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this approach appropriate? If I do it, the CIs become become fairly large and the effect no longer looks significant (conflicting with the pvals.fnc output).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a better way (code example would be great)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If not, is there a &quot;least worst&quot; solution that could satisfy readers who strongly request the standard bar plot with CIs?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Problems in the back of my head, which I am too ignorant to properly formulate:&lt;br&gt;&#xA;CI not equal to HPD.&lt;br&gt;&#xA;Transforming CI to SD is inappropriate, as CI may be asymmetric.&lt;br&gt;&#xA;Problems with determining CI at all in mixed models (&lt;a href=&quot;http://stats.stackexchange.com/questions/1797/what-would-a-confidence-interval-around-a-predicted-value-from-a-mixed-effects-mo&quot;&gt;like here&lt;/a&gt;).  &lt;/p&gt;&#xA;" OwnerUserId="2228" LastEditorUserId="2228" LastEditorDisplayName="" LastEditDate="2010-12-02T11:41:11.120" LastActivityDate="2010-12-02T12:27:31.840" Title="Confidence-intervals for conditions tested with a mixed-effects model" Tags="&lt;r&gt;&lt;confidence-interval&gt;&lt;mixed-model&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="1" />
  <row Id="5079" PostTypeId="1" AcceptedAnswerId="5084" CreationDate="2010-12-02T10:33:12.123" Score="4" ViewCount="156" Body="&lt;p&gt;I have some data which describe residential units for people with learning disability, variables like how nice the furnishings are, the level of psychiatric symptomology on the unit, how happy the staff are, stuff like that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to check to see if we are measuring the right things- e.g. do units with happier staff have nicer atmospheres, do units with nice furnishings have happy staff, that kind of thing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is that I only have data for 8 units (averages within each, e.g. 10 staff give their responses on how happy they are, this is the average for the unit) so I can't really use linear regression to see whether the things that we have measured affect each other. I've drawn some scatterplots for all of the data and on the whole I would say there does appear to be a linear relationship in the way I would expect. But as I say with 8 units it's not really going to generate any statistics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I had the bright idea of ordering each unit across all variables, and then comparing the rank orders somehow. If we are measuring the right things then the ranks should be similar, like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Unit 1 (ranks across all variables): 1,1,1,1,1,1,1,1,1&#xA;Unit 2: 2,2,2,2,2,2,2,2,2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;whereas if I'm wrong, and the variables aren't important to each other, I will get this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Unit 1: 1,2,3,4,5,6,7,8&#xA;Unit 2: 8,7,6,5,4,3,2,1&#xA;Unit 3: 4,5,6,7,8,1,2,3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's what I get:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Unit1    7    5  5.0    5    3    4    5    3&#xA;Unit2    6    2  4.0    6    5    3    2    5&#xA;Unit3    3    7  7.5    1    4    1    1    1&#xA;Unit4    4    4  3.0    7    6    7    7    8&#xA;Unit5    5    3  1.0    4    2    5    6    7&#xA;Unit6    2    6  6.0    8    8    8    8    6&#xA;Unit7    1    8  7.5    3    7    6    4    4&#xA;Unit8    8    1  2.0    2    1    2    3    2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It looks pretty good to me, except for the first column.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any thoughts on this? Is is a proper statistic that I haven't heard of? Or is there something reasonably robust I can do with these results?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sorry for the lengthy question, many thanks in advance!&lt;/p&gt;&#xA;" OwnerUserId="199" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-12-02T11:07:53.903" LastActivityDate="2010-12-02T12:44:04.133" Title="Compare rank orders of population members across different variables" Tags="&lt;ranks&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="5080" PostTypeId="2" ParentId="5056" CreationDate="2010-12-02T10:37:22.867" Score="4" ViewCount="" Body="&lt;p&gt;The &lt;a href=&quot;http://www-stat.stanford.edu/~tibs/ElemStatLearn/&quot; rel=&quot;nofollow&quot;&gt;Elements of Statistical Learning&lt;/a&gt;, from Hastie et al., has a complete chapter on support vector classifiers and SVMs (in your case, start page 418 on the 2nd edition). Another good tutorial is &lt;a href=&quot;http://www.ci.tuwien.ac.at/~meyer/svm/final.pdf&quot; rel=&quot;nofollow&quot;&gt;Support Vector Machines in R&lt;/a&gt;, by David Meyer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unless I misunderstood your question, the decision boundary (or hyperplane) is defined by $x^T\beta + \beta_0=0$ (with $\|\beta\|=1$, and $\beta_0$ the intercept term), or as @ebony said a linear combination of the support vectors. The margin is then $2/\|\beta\|$, following Hastie et al. notations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the on-line help of &lt;code&gt;ksvm()&lt;/code&gt; in the &lt;a href=&quot;http://cran.r-project.org/web/packages/kernlab/index.html&quot; rel=&quot;nofollow&quot;&gt;kernlab&lt;/a&gt; R package, but see also &lt;a href=&quot;http://www.jstatsoft.org/v11/i09/paper&quot; rel=&quot;nofollow&quot;&gt;kernlab – An S4 Package for Kernel Methods in R&lt;/a&gt;, here is a toy example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(101)&#xA;x &amp;lt;- rbind(matrix(rnorm(120),,2),matrix(rnorm(120,mean=3),,2))&#xA;y &amp;lt;- matrix(c(rep(1,60),rep(-1,60)))&#xA;svp &amp;lt;- ksvm(x,y,type=&quot;C-svc&quot;)&#xA;plot(svp,data=x)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Note that for the sake of clarity, we don't consider train and test samples.&#xA;Results are shown below, where color shading helps visualizing the fitted decision values; values around 0 are on the decision boundary. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/oNWf7.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Calling &lt;code&gt;attributes(svp)&lt;/code&gt; gives you attributes that you can access, e.g.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;alpha(svp)  # support vectors whose indices may be &#xA;            # found with alphaindex(svp)&#xA;b(svp)      # (negative) intercept &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So, to display the decision boundary, with its corresponding margin, let's try the following (in the rescaled space), which is largely inspired from a tutorial on SVM made some time ago by &lt;a href=&quot;http://cbio.ensmp.fr/~jvert/&quot; rel=&quot;nofollow&quot;&gt;Jean-Philippe Vert&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plot(scale(x), col=y+2, pch=y+2, xlab=&quot;&quot;, ylab=&quot;&quot;)&#xA;w &amp;lt;- colSums(coef(svp)[[1]] * x[unlist(alphaindex(svp)),])&#xA;b &amp;lt;- b(svp)&#xA;abline(b/w[1],-w[2]/w[1])&#xA;abline((b+1)/w[1],-w[2]/w[1],lty=2)&#xA;abline((b-1)/w[1],-w[2]/w[1],lty=2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And here it is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/ZvQpJ.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-12-02T10:37:22.867" CommentCount="1" />
  <row Id="5081" PostTypeId="1" CreationDate="2010-12-02T11:17:13.693" Score="1" ViewCount="259" Body="&lt;p&gt;Hello fellow Number crunchers&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As the headline suggest, I am looking for a java library for learning and inference of Bayesian Networks. I have already found some, but I am hoping for a recommendation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Requirements in a quick overview:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;written in Java (my overlord tells me that this is no point of discussion)&lt;/li&gt;&#xA;&lt;li&gt;configuration is possible via code (and not solely via a GUI).&lt;/li&gt;&#xA;&lt;li&gt;source code available&lt;/li&gt;&#xA;&lt;li&gt;project is still maintained&lt;/li&gt;&#xA;&lt;li&gt;the more powerful, the better&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Which one do you recommend ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: This should be cw.&lt;/p&gt;&#xA;" OwnerUserId="264" LastEditorUserId="264" LastEditorDisplayName="" LastEditDate="2010-12-02T11:55:34.457" LastActivityDate="2010-12-02T14:46:14.977" CommunityOwnedDate="2010-12-02T21:56:02.210" Title="Java library for Bayesian Networks" Tags="&lt;bayesian&gt;&lt;java&gt;&lt;library&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="5082" PostTypeId="2" ParentId="5081" CreationDate="2010-12-02T11:30:28.597" Score="2" ViewCount="" Body="&lt;p&gt;See the &lt;a href=&quot;http://weka.sourceforge.net/doc.dev/overview-summary.html&quot; rel=&quot;nofollow&quot;&gt;packages&lt;/a&gt; of the open source project Weka, which is a collection of machine learning algorithms for data mining tasks. The algorithms can either be applied directly to a dataset or called from your own Java code. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is the class for &lt;a href=&quot;http://weka.sourceforge.net/doc.dev/weka/classifiers/bayes/BayesNet.html&quot; rel=&quot;nofollow&quot;&gt;Bayes Network learning&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="339" LastActivityDate="2010-12-02T11:30:28.597" CommunityOwnedDate="2010-12-02T21:56:02.210" CommentCount="1" />
  <row Id="5083" PostTypeId="1" CreationDate="2010-12-02T11:33:03.493" Score="4" ViewCount="50" Body="&lt;p&gt;The signal to noise ratio is simple, and is usually defined in the context of simple Gaussian local-level models. In the cause of non-gaussian signal or noise models, do people do things more complicated then the ratio of the variances of the two distributions?&lt;/p&gt;&#xA;" OwnerUserId="996" LastActivityDate="2010-12-02T11:33:03.493" Title="Generalization of the Signal-Noise ratio for non-Gaussian processes" Tags="&lt;mixed-model&gt;&lt;signal-processing&gt;" FavoriteCount="1" />
  <row Id="5084" PostTypeId="2" ParentId="5079" CreationDate="2010-12-02T11:48:46.770" Score="4" ViewCount="" Body="&lt;p&gt;I don't know how useful the following approach is, but one might conceptualize the situation slightly differently: imagine the different variables are raters who simply order the units from &quot;best&quot; to &quot;worst&quot;. You expect that the rank order will be similar among &quot;raters&quot;. This seems to be an application for Kendall's concordance coefficient $W$ of inter-rater agreement. In R&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; rtr1    &amp;lt;- c(1, 6, 3, 2, 5, 4)  # rank order from &quot;rater&quot; 1&#xA;&amp;gt; rtr2    &amp;lt;- c(1, 5, 6, 2, 4, 3)  # &quot;rater&quot; 2&#xA;&amp;gt; rtr3    &amp;lt;- c(2, 3, 6, 5, 4, 1)  # &quot;rater&quot; 3&#xA;&amp;gt; ratings &amp;lt;- cbind(rtr1, rtr2, rtr3)&#xA;&amp;gt; library(irr)      # for kendall()&#xA;&amp;gt; kendall(ratings)&#xA; Kendall's coefficient of concordance W&#xA; Subjects = 6 &#xA;   Raters = 3 &#xA;        W = 0.568 &#xA;&#xA; Chisq(5) = 8.52 &#xA;  p-value = 0.130&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Edit: This is equivalent to the Friedman-Test for dependent samples:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; rtrAll &amp;lt;- c(rtr1, rtr2, rtr3)&#xA;&amp;gt; nBl    &amp;lt;- 3    # number of blocks / raters&#xA;&amp;gt; P      &amp;lt;- 6    # number of dependent samples / units&#xA;&amp;gt; IV     &amp;lt;- factor(rep(1:P, nBl))         # factor sample / unit&#xA;&amp;gt; blocks &amp;lt;- factor(rep(1:nBl, each=P))    # factor blocks / raters&#xA;&amp;gt; friedman.test(rtrAll, IV, blocks)&#xA;        Friedman rank sum test&#xA;data:  rtrAll, IV and blocks &#xA;Friedman chi-squared = 8.5238, df = 5, p-value = 0.1296&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1909" LastEditorUserId="1909" LastEditorDisplayName="" LastEditDate="2010-12-02T12:44:04.133" LastActivityDate="2010-12-02T12:44:04.133" CommentCount="3" />
  <row Id="5085" PostTypeId="2" ParentId="5078" CreationDate="2010-12-02T12:27:31.840" Score="1" ViewCount="" Body="&lt;p&gt;The &lt;a href=&quot;http://glmm.wikidot.com/faq&quot; rel=&quot;nofollow&quot;&gt;DRAFT r-sig-mixed-models FAQ&lt;/a&gt; details (in the &quot;Predictions and/or confidence (or prediction) intervals on predictions&quot; section) how to obtain predictions and confidence intervals for cells in the design of a mixed effects model. The ezPredict() function in the ez package wraps the code for the lme4 case (well, obtaining predictions and variances, leaving the user to decide their own CI). &lt;/p&gt;&#xA;" OwnerUserId="364" LastActivityDate="2010-12-02T12:27:31.840" CommentCount="1" />
  <row Id="5086" PostTypeId="2" ParentId="5081" CreationDate="2010-12-02T14:46:14.977" Score="2" ViewCount="" Body="&lt;p&gt;You can easily accomplish this with R though rJava (&lt;a href=&quot;http://rosuda.org/JRI/&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;JRI&lt;/strong&gt;&lt;/a&gt;, to be precise).  You should tell your overlord that you want to use the best tool for the job.&lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-12-02T14:46:14.977" CommunityOwnedDate="2010-12-02T21:56:02.210" CommentCount="1" />
  <row Id="5087" PostTypeId="1" AcceptedAnswerId="5088" CreationDate="2010-12-02T15:18:04.343" Score="4" ViewCount="150" Body="&lt;p&gt;There are numerous procedures for functional data clustering based on orthonormal basis functions. I have a series of models built with the GAMM models, using the &lt;code&gt;gamm()&lt;/code&gt; from the mgcv package in R. For fitting a long-term trend, I use a thin plate regression spline. Next to that, I introduce a CAR1 model in the random component to correct for autocorrelation. For more info, see eg the paper of Simon Wood on &lt;a href=&quot;http://r.789695.n4.nabble.com/attachment/2063352/0/tprs.pdf&quot; rel=&quot;nofollow&quot;&gt;thin plate regression splines&lt;/a&gt; or his &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1584884746&quot; rel=&quot;nofollow&quot;&gt;book on GAM models&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I'm a bit puzzled in how I get the correct coefficients out of the models. And I'm even less confident that the coefficients I can extract, are the ones I should use to cluster different models. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A simple example, using:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#runnable code&#xA;require(mgcv)&#xA;require(nlme)&#xA;library(RLRsim)&#xA;library(RColorBrewer)&#xA;&#xA;x1 &amp;lt;- 1:1000&#xA;x2 &amp;lt;- runif(1000,10,500)&#xA;&#xA;fx1 &amp;lt;- -4*sin(x1/50)&#xA;fx2 &amp;lt;- -10*(x2)^(1/4)&#xA;y &amp;lt;- 60+ fx1 + fx2 + rnorm(1000,0,5)&#xA;&#xA;test &amp;lt;- gamm(y~s(x1)+s(x2))&#xA;# end runnable code&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then I can construct the original basis using smoothCon :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#runnable code&#xA;um &amp;lt;- smoothCon(s(x1),data=data.frame(x1=x1),&#xA;         knots=NULL,absorb.cons=FALSE)&#xA;#end runnable code&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now,when I look at the basis functions I can extract using &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# runnable code&#xA;X &amp;lt;- extract.lmeDesign(test$lme)$X&#xA;Z &amp;lt;- extract.lmeDesign(test$lme)$Z&#xA;&#xA;op &amp;lt;- par(mfrow=c(2,5),mar=c(4,4,1,1))&#xA;plot(x1,X[,1],ylab=&quot;Basis function&quot;,xlab=&quot;X&quot;,type=&quot;l&quot;,lwd=2)&#xA;plot(x1,X[,2],ylab=&quot;Basis function&quot;,xlab=&quot;X&quot;,type=&quot;l&quot;,lwd=2)&#xA;plot(x1,Z[,8],ylab=&quot;Basis function&quot;,xlab=&quot;X&quot;,type=&quot;l&quot;,lwd=2)&#xA;plot(x1,Z[,7],ylab=&quot;Basis function&quot;,xlab=&quot;X&quot;,type=&quot;l&quot;,lwd=2)&#xA;plot(x1,Z[,6],ylab=&quot;Basis function&quot;,xlab=&quot;X&quot;,type=&quot;l&quot;,lwd=2)&#xA;plot(x1,Z[,5],ylab=&quot;Basis function&quot;,xlab=&quot;X&quot;,type=&quot;l&quot;,lwd=2)&#xA;plot(x1,Z[,4],ylab=&quot;Basis function&quot;,xlab=&quot;X&quot;,type=&quot;l&quot;,lwd=2)&#xA;plot(x1,Z[,3],ylab=&quot;Basis function&quot;,xlab=&quot;X&quot;,type=&quot;l&quot;,lwd=2)&#xA;plot(x1,Z[,2],ylab=&quot;Basis function&quot;,xlab=&quot;X&quot;,type=&quot;l&quot;,lwd=2)&#xA;plot(x1,Z[,1],ylab=&quot;Basis function&quot;,xlab=&quot;X&quot;,type=&quot;l&quot;,lwd=2)&#xA;par(op)&#xA;# end runnable code&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;they look already quite different. I can get the final coefficients used to build the smoother by&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#runnable code&#xA;Fcoef &amp;lt;- test$lme$coef$fixed&#xA;Rcoef &amp;lt;- unlist(test$lme$coef$random)&#xA;#end runnable code&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;but I'm far from sure these are the coefficients I look for. I fear I can't just use those coefficients as data in a clustering procedure. I would really like to know which coefficients are used to transform the basis functions from the ones I get with &lt;code&gt;smoothCon()&lt;/code&gt; to the ones I extract from the lme-part of the gamm-object. And if possible, where I can find them. I've read the related articles, but somehow I fail to figure it out myself. All help is appreciated.&lt;/p&gt;&#xA;" OwnerUserId="1124" LastActivityDate="2010-12-03T09:12:20.757" Title="Use coefficients of thin plate regression splines in a clustering method" Tags="&lt;r&gt;&lt;clustering&gt;&lt;gamm&gt;&lt;basis-functions&gt;&lt;mgcv&gt;" AnswerCount="1" FavoriteCount="0" />
  <row Id="5088" PostTypeId="2" ParentId="5087" CreationDate="2010-12-02T15:49:45.877" Score="3" ViewCount="" Body="&lt;p&gt;If I understand correctly, I think you want the coefficients from the &lt;code&gt;$gam&lt;/code&gt; component:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; coef(test$gam)&#xA;(Intercept)     s(x1).1     s(x1).2     s(x1).3     s(x1).4     s(x1).5 &#xA; 21.8323526   9.2169405  15.7504889  -3.4709907  16.9314057 -19.4909343 &#xA;    s(x1).6     s(x1).7     s(x1).8     s(x1).9     s(x2).1     s(x2).2 &#xA;  1.1124505  -3.3807996  21.7637766 -23.5791595   3.2303904  -3.0366406 &#xA;    s(x2).3     s(x2).4     s(x2).5     s(x2).6     s(x2).7     s(x2).8 &#xA; -2.0725621  -0.6642467   0.7347857   1.7232242  -0.5078875  -7.7776700 &#xA;    s(x2).9 &#xA;-12.0056347&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update 1:&lt;/strong&gt; To get at the basis functions we can use &lt;code&gt;predict(...., type = &quot;lpmatrix&quot;)&lt;/code&gt; to get $Xp$ the smoothing matrix:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Xp &amp;lt;- predict(test$gam, type = &quot;lpmatrix&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The fitted spline (e.g. for &lt;code&gt;s(x1)&lt;/code&gt;) can be recovered then using:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plot(Xp[,2:10] %*% coef(test$gam)[2:10], type = &quot;l&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can plot this ($Xp$) and see that it is similar to &lt;code&gt;um[[1]]$X&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;layout(matrix(1:2, ncol = 2))&#xA;matplot(um[[1]]$X, type = &quot;l&quot;)&#xA;matplot(Xp[,1:10], type = &quot;l&quot;)&#xA;layout(1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I pondered why these are not exactly the same. &lt;strike&gt;Is it because the original basis functions have been subject to the penalised regression during fitting???&lt;/strike&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update 2:&lt;/strong&gt; You can make them the same by including the identifiability constraints into your basis functions in &lt;code&gt;um&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;um2 &amp;lt;- smoothCon(s(x1), data=data.frame(x1=x1), knots=NULL, &#xA;                 absorb.cons=TRUE)&#xA;layout(matrix(1:2, ncol = 2))&#xA;matplot(um2[[1]]$X, type = &quot;l&quot;, main = &quot;smoothCon()&quot;)&#xA;matplot(Xp[,2:10], type = &quot;l&quot;, main = &quot;Xp matrix&quot;) ##!##&#xA;layout(1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Note I have not got the intercept in the line marked &lt;code&gt;##!##&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You ought to be able to get $Xp$ directly from function &lt;code&gt;PredictMat()&lt;/code&gt;, which is documented on same page as &lt;code&gt;smoothCon()&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1390" LastEditorUserId="1390" LastEditorDisplayName="" LastEditDate="2010-12-03T09:12:20.757" LastActivityDate="2010-12-03T09:12:20.757" CommentCount="3" />
  <row Id="5089" PostTypeId="2" ParentId="5065" CreationDate="2010-12-02T16:19:17.053" Score="3" ViewCount="" Body="&lt;p&gt;If your norm is a hilbert space norm (for example root mean square error, also called hilbert schmidt norm in the case of matrices or l^2 norm if you take them as vectors)  then obtaining the solution is a first year calculous exercise if you rephrase things using:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$c=(\langle Y,X_i \rangle)_{i=1,\dots,k}$ and $A=(\langle X_j,X_i \rangle)_{j,i=1,\dots,k}$ ($c$ is a vector and $A$ a $k\times k$ matrix). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The solution to your problem is given by solving $A\theta^*=c$&#xD;&#xA;no need for optimization ! &lt;/p&gt;&#xA;" OwnerUserId="223" LastActivityDate="2010-12-02T16:19:17.053" />
  <row Id="5090" PostTypeId="1" AcceptedAnswerId="5641" CreationDate="2010-12-02T21:44:09.290" Score="0" ViewCount="133" Body="&lt;p&gt;The R package &lt;a href=&quot;http://cran.r-project.org/web/packages/dlm/index.html&quot; rel=&quot;nofollow&quot;&gt;dlm&lt;/a&gt; implements filtering and smoothing (&lt;code&gt;dlmFilter&lt;/code&gt; and &lt;code&gt;dlmSmooth&lt;/code&gt;) for models with regression effects, but forecasting is not (yet) available for these models:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mod &amp;lt;- dlmModSeas(4)+dlmModReg(cbind(rnorm(100),rnorm(100)))&#xA;fi &amp;lt;- dlmFilter(rnorm(100),mod)&#xA;f &amp;lt;- dlmForecast(fi,nAhead=12)&#xA;Error in dlmForecast(fi, nAhead = 12): &#xA;  dlmForecast only works with constant models&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can I do this in R?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for your help!&lt;/p&gt;&#xA;" OwnerUserId="2234" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-12-02T22:55:07.853" LastActivityDate="2010-12-20T03:50:36.193" Title="Gaussian state space forecasting with regression effects" Tags="&lt;r&gt;&lt;forecasting&gt;&lt;dlm&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="2" />
  <row Id="5091" PostTypeId="2" ParentId="5048" CreationDate="2010-12-02T22:25:41.843" Score="1" ViewCount="" Body="&lt;p&gt;The correlation coefficient of two sets of values is one number. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The auto-correlation of one set of values is a function (see e.g. &lt;a href=&quot;http://en.wikipedia.org/wiki/Autocorrelation&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Autocorrelation&lt;/a&gt; ). Let's call the argument of the function &lt;code&gt;t&lt;/code&gt; (looks like it's the $\tau$ in your question), then the value of the auto-correlation function at &lt;code&gt;t&lt;/code&gt; is the correlation coefficient of the set of values and the set of values shifted by &lt;code&gt;t&lt;/code&gt; (I might be ignoring normalization factors here).&lt;/p&gt;&#xA;" OwnerUserId="961" LastActivityDate="2010-12-02T22:25:41.843" CommentCount="1" />
  <row Id="5092" PostTypeId="1" CreationDate="2010-12-02T23:24:13.180" Score="8" ViewCount="250" Body="&lt;p&gt;I am a stats newbie, so apologies in advance if I'm asking a braindead question. I have searched for answers to my question, but I find that many of the topics are either too specific, or quickly go beyond what I currently comprehend.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have some simulation work that includes large datasets which become infeasible to simulate exhaustively. For the smallest of my datasets, an exhaustive run presents the following distribution of results from a total of 9180900 tests.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Result/Frequency:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;0 7183804&lt;/li&gt;&#xA;&lt;li&gt;1 1887089&lt;/li&gt;&#xA;&lt;li&gt;2 105296&lt;/li&gt;&#xA;&lt;li&gt;3 4571&lt;/li&gt;&#xA;&lt;li&gt;4 140&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;What the numbers &lt;em&gt;mean&lt;/em&gt; does not matter; what matters is that the larger datasets I have can stretch into billions of tests, and become far too time consuming to run. I need to constrain the workload.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I feel I ought to be able to sample from the full set of tests to derive a distribution for the sample, and infer (within some bounds) that the results of an exhaustive simulation would exhibit roughly the same distribution. There is no bias inherent in the tests which are run, so uniformly randomly choosing inputs ought to provide a valid sample.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I do not yet understand is how I should go about selecting my sample size. In particular, the distribution exhibits a strange tail, and I fear that sampling too small will lose the lower frequencies. (The 140 occurrences of '4' account for only 0.0015% of the population!)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, my question is, what is the best way of calculating a sample size with which I can assert some level of goodness in my results?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or, am I asking the wrong question?&lt;/p&gt;&#xA;" OwnerUserId="2246" LastActivityDate="2010-12-05T05:29:13.617" Title="Representative sampling" Tags="&lt;distributions&gt;&lt;sample-size&gt;&lt;experiment-design&gt;&lt;sampling&gt;" AnswerCount="2" FavoriteCount="2" />
  <row Id="5093" PostTypeId="1" AcceptedAnswerId="5095" CreationDate="2010-12-03T00:12:49.363" Score="3" ViewCount="172" Body="&lt;p&gt;I am currently working on a RandomForest based prediction method using protein sequence data. I have generated two models first model (NF) using standard set of features and the second model (HF) using hybrid features. I have done Mathews Correlation Coefficient (MCC) and Accuracy calculation and the following are my results:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Model 1 (NF): Training Accuracy - 62.85% Testing Accuracy - 56.38 MCC - 0.1673&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Model 2 (HF): Training Accuracy - 60.34 Testing Accuracy - 61.78 MCC - 0.1856&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The testing data is an independent dataset (means not included in the training data).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since there is a trade-off in accuracy and MCC between the models am confused about the prediction power of the models. Could you please share your thoughts on which model I should consider for further analysis? Apart from Accuracy and MCC is there any other measure that I should consider for validation? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance. &lt;/p&gt;&#xA;" OwnerUserId="529" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-03T00:35:31.087" LastActivityDate="2010-12-03T00:53:44.013" Title="Statistical validation of RandomForest models" Tags="&lt;machine-learning&gt;&lt;cross-validation&gt;&lt;random-forest&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="5094" PostTypeId="2" ParentId="5093" CreationDate="2010-12-03T00:34:34.147" Score="4" ViewCount="" Body="&lt;p&gt;It just seems those two variants are equivalent; yet some better test should be made to confirm this, at least cross validation.&lt;br&gt;&#xA;Also if this NF and HF sets have some attributes in common, it may suggest that only this common part is useful -- I would invest some time in making feature selection.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-12-03T00:34:34.147" CommentCount="1" />
  <row Id="5095" PostTypeId="2" ParentId="5093" CreationDate="2010-12-03T00:53:44.013" Score="3" ViewCount="" Body="&lt;p&gt;Hi Khader, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I like the idea of parsimony- the smaller the number of variables in the model, the better. Unless you are driven theoretically of course. Feature selection refers to the process of choosing which variables to use in the model (getting the best combination of variables). There are lots of different options for feature selection (worth a read). With that said, there should be inbuilt within the rf algorithm, a variable importance measure that you can generate as a starting point (with that said, be very very careful with this because there are noted biases in this) - see Strobl et al in the R journal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I trust you have varied the number of variables randomly sampled at each node (this is mtry in R) and the depth of the trees and splitting criteria etc.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In terms of appearance, the second model looks slighly better to me, simply because of the reproduced accuracy in the test and train results. It always concerns me that if my test set accuracy is notably lower, there may be something wrong with the model. I trust you have made sure that your test and train set are balanced, at least on the dependent variable you are looking to classify. If this is binary (0,1) your models are not really doing much better than chance (50,50).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;An very important thing to look at is the sensitivity (the number of true positives in a binary task 0,1 that are correctly classified) and specificity (the number of true negatives in a binary task 0,1) that are both correctly classified. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If possible, I would compare this model against other machine learning algorithms such as boosted trees, support vector machines (which do ok in gene data) etc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not sure what package you are using - hope that helps if &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are using r - look up caret in cran (really good intro to some of the ideas here and great for getting out some alternative measures of performance).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Paul D &lt;/p&gt;&#xA;" OwnerUserId="2238" LastActivityDate="2010-12-03T00:53:44.013" CommentCount="4" />
  <row Id="5096" PostTypeId="2" ParentId="4904" CreationDate="2010-12-03T04:58:22.370" Score="0" ViewCount="" Body="&lt;p&gt;I will repost the answer I gave on math.stackexchange:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your question needs some more information:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How is their score generated (what kind of game is it)? What should your non-cheating data look like? How do people cheat? How will their score be different (in a statistical sense) when they are not cheating? Do you know roughly the proportion that are cheating? Or is that something you also want to find out?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would also look at outlier detection algorithms: wikipedia looks useful on this topic (link). Using a Q-Q plot on your data may also be useful if your non-cheating data should be approximately Normally distributed; points that are significantly above the line might be cheaters.&lt;/p&gt;&#xA;" OwnerUserId="1146" LastActivityDate="2010-12-03T04:58:22.370" />
  <row Id="5097" PostTypeId="1" AcceptedAnswerId="5099" CreationDate="2010-12-03T05:23:38.833" Score="3" ViewCount="184" Body="&lt;p&gt;This is a follow-up of the question I posted &lt;a href=&quot;http://stats.stackexchange.com/questions/5093/statistical-validation-of-randomforest-models&quot;&gt;earlier&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am assessing the two RF models which are generated using two different set of features &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;NF - Test_Accuracy &gt; Training accuracy (500 features)&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;HF - Test_Accuracy &amp;lt; Training accuracy (125 features)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Testing and Training is done using independent data sets and the accuracy is derived from the average of a 5-cross validation. The difference between the models are in the number of features. I am afraid there could be a possible &lt;a href=&quot;http://en.wikipedia.org/wiki/Overfitting&quot; rel=&quot;nofollow&quot;&gt;overfitting&lt;/a&gt; in one of the model (It is not clear to me which model could be overfitting, because I have used independent dataset and k-cross validation on the datasets). I would like to know what are the standard methods (tools/libraries) which can be used to assess overfitting. &lt;/p&gt;&#xA;" OwnerUserId="529" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-03T09:40:20.157" LastActivityDate="2010-12-03T09:40:20.157" Title="How to assess overfitting?" Tags="&lt;machine-learning&gt;&lt;random-forest&gt;" AnswerCount="1" />
  <row Id="5098" PostTypeId="2" ParentId="5092" CreationDate="2010-12-03T05:41:03.173" Score="5" ViewCount="" Body="&lt;p&gt;I think the answer to your question is a couple other questions: how rare does a given test outcome need to be before you don't care about it?  How certain do you want to be that you'll actually find at least test that comes out that way if it occurs right at the threshold where you've stopped caring about it.  Given those values you can do a power analysis.  I'm not 100% confident whether you need to do a multinomial (involving more than one outcome) power analysis or not, I'm guessing a binomial one (either the rare test or not) will work just fine, e.g. &lt;a href=&quot;http://statpages.org/proppowr.html&quot; rel=&quot;nofollow&quot;&gt;http://statpages.org/proppowr.html&lt;/a&gt;.  Alpha = .05, Power = 80%, Group on proportion 0, Group 1 proportion .0015.  Relative sample size, 1; total - just south of 13,000 tests.  At which the expected number of test 4s is ~20.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That will help you find the number of tests you need to have to detect one of those rare occurring results.  However if what you really care about is relative frequency, the problem is harder.  I'd conjecture that if you simply multiplied the resulting N from the power analysis by 20 or 30 you'd find a reasonable guess.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In practice, if you don't really need to decide the number of tests ahead of time, you might consider running tests until you get 20 or 30 result 4s.  By the time you've gotten that many 4s you should start to have a reasonable though not absolute estimate of their relative frequency IMO.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ultimately - there are trade-offs between number of tests run and accuracy.  You need to know how precise you want your estimates to be before you can really determine how many is &quot;enough&quot;.&lt;/p&gt;&#xA;" OwnerUserId="196" LastActivityDate="2010-12-03T05:41:03.173" CommentCount="1" />
  <row Id="5099" PostTypeId="2" ParentId="5097" CreationDate="2010-12-03T09:36:39.130" Score="4" ViewCount="" Body="&lt;p&gt;This result does not mean that you have overfitting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First of all, CV is more reliable than test set -- you can have (bad) luck in selecting test, what results in (pessimistic) optimistic bias with respect to true accuracy. CV effectively smooths this problem by repeating the procedure of selecting test. What's worse for using the test, RF is a stochastic algorithm and so two runs with different seed will give you different test accuracies, and the difference may be even bigger than that between CV and test.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second, you may use standard deviation of accuracy from all CV runs to test whether:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Your CV accuracy is really different from this on test.&lt;/li&gt;&#xA;&lt;li&gt;One of the feature sets you used is really better than the other. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-12-03T09:36:39.130" CommentCount="1" />
  <row Id="5100" PostTypeId="1" AcceptedAnswerId="5103" CreationDate="2010-12-03T12:32:23.160" Score="1" ViewCount="119" Body="&lt;p&gt;Hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;this is my first question on this site, so please be patient with me. I am doing a random walk, where I build a timeseries curve. I do that a preset number of times ( let's say 100 times ). Now I was wondering what should I do with all the generated curves. Eventually I want to have 1 curve that is the best representation. I tried taking the mean and median of the values for each point of time, but that gives me a rather tame and flat curve. What other options do I have? Your input is appreciated!&lt;/p&gt;&#xA;" OwnerUserId="2218" LastActivityDate="2010-12-03T14:00:26.023" Title="Random walk data post processing" Tags="&lt;random-variable&gt;" AnswerCount="3" CommentCount="3" />
  <row Id="5102" PostTypeId="2" ParentId="5100" CreationDate="2010-12-03T13:22:49.450" Score="2" ViewCount="" Body="&lt;p&gt;Plotting the mean or median for each timepoint sounds a sensible start. You could also plot a &lt;a href=&quot;http://en.wikipedia.org/wiki/Reference_range&quot; rel=&quot;nofollow&quot;&gt;reference range&lt;/a&gt; for each timepoint to show the variability across curves at each timepoint. You could also add a few (perhaps 5 or 10) randomly-chosen curves to illustrate the variability across timepoints within each curve. Should be perfectly possible to show all of those things on the same plot with a suitable choice of colours and line weights.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That should give a graphical depiction of the process's behaviou but doesn't really answer your requirement for 'one curve that is the best representation'. But to answer that we need to know what you mean by 'best' -- how will you &lt;em&gt;use&lt;/em&gt; this 'best curve'? The mean or median may &lt;em&gt;look&lt;/em&gt; too flat and boring to use it as the sole graphical display but may be the 'best' summary curve for quite a few purposes.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-12-03T13:22:49.450" CommentCount="1" />
  <row Id="5103" PostTypeId="2" ParentId="5100" CreationDate="2010-12-03T13:55:13.343" Score="3" ViewCount="" Body="&lt;p&gt;I'm somewhat confused - if these are random walks, isn't the expectation a flat, uninteresting line?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(123)&#xA;nWalks &amp;lt;- 1000&#xA;nTimes &amp;lt;- 100&#xA;mat &amp;lt;- matrix(c(rep(0, nWalks), rnorm(nWalks*(nTimes-1))),&#xA;              ncol = nTimes)&#xA;rwalks &amp;lt;- apply(mat, 1, cumsum)&#xA;matplot(rwalks, type = &quot;l&quot;)&#xA;## Stack the data for fitting a smoother&#xA;df &amp;lt;- stack(data.frame(rwalks))[, 2:1]&#xA;names(df) &amp;lt;- c(&quot;Walk&quot;,&quot;Xt&quot;)&#xA;df &amp;lt;- within(df, Time &amp;lt;- rep(seq_len(nTimes), nWalks))&#xA;## fit smother and predict&#xA;mod &amp;lt;- with(df, smooth.spline(Time, Xt))&#xA;pred &amp;lt;- predict(mod, x = 1:100)&#xA;lines(pred, col = &quot;red&quot;, lwd = 4)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This gives:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/Oi2Sk.png&quot; alt=&quot;1000 simulated random walks from origin 0, with summary smoothing spline (think, red line)&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Whilst the expectation doesn't do an awful lot, the range and the variance do slightly more-interesting things:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;layout(1:3)&#xA;plot(apply(rwalks, 1, mean), type = &quot;l&quot;, main = &quot;Mean&quot;,&#xA;     ylab = &quot;Mean&quot;)&#xA;plot(apply(rwalks, 1, var), type = &quot;l&quot;, main = &quot;Variance&quot;,&#xA;     ylab = &quot;Variance&quot;)&#xA;plot(apply(rwalks, 1, max) - apply(rwalks, 1, min), type = &quot;l&quot;,&#xA;     main = &quot;Range&quot;, ylab = &quot;Range&quot;)&#xA;layout(1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/K5jfR.png&quot; alt=&quot;Expectation, variance and range of simulated random walks&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The tiny fluctuations in the mean trace are due to sampling variation for the set of random walks I generated. it should settle down to be 0 if you up the number of random walks generated. (In any case, the value of the mean is effectively zero given the magnitude of the values the walks take.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Perhaps you could explain a bit more what you mean by &quot;best representation&quot;?&lt;/p&gt;&#xA;" OwnerUserId="1390" LastActivityDate="2010-12-03T13:55:13.343" CommentCount="3" />
  <row Id="5104" PostTypeId="2" ParentId="5100" CreationDate="2010-12-03T14:00:26.023" Score="0" ViewCount="" Body="&lt;p&gt;If you're simply looking for a way of representing a process for readers, look up &quot;diffusion modelling&quot; in the psychological literature and you'll find how those folks typically attempt to represent a stochastic process and typical exemplar.&lt;/p&gt;&#xA;" OwnerUserId="364" LastActivityDate="2010-12-03T14:00:26.023" />
  <row Id="5105" PostTypeId="2" ParentId="4259" CreationDate="2010-12-03T16:23:28.423" Score="0" ViewCount="" Body="&lt;p&gt;The best introductory Bayesian book I have found is &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0198518897&quot; rel=&quot;nofollow&quot;&gt;Data Analysis - A Bayesian Tutorial&lt;/a&gt;. It is quite practical.&lt;/p&gt;&#xA;" OwnerUserId="1146" LastActivityDate="2010-12-03T16:23:28.423" CommunityOwnedDate="2010-12-03T16:23:28.423" />
  <row Id="5106" PostTypeId="2" ParentId="58" CreationDate="2010-12-03T16:48:15.773" Score="0" ViewCount="" Body="&lt;p&gt;actually back propogation is the training or learning algorithm rather than the neural network itself it is the best known example of neural network training algorithm.&lt;/p&gt;&#xA;" LastActivityDate="2010-12-03T16:48:15.773" />
  <row Id="5107" PostTypeId="1" CreationDate="2010-12-03T17:40:07.863" Score="3" ViewCount="230" Body="&lt;p&gt;Say I have a set of sample points generated by a multivariate normal distribution D whose parameters I don't know.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to be able to measure the distance from an arbitrary point to the distribution D.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One way of doing this would be to get an estimate of the parameters of D, and use it to get the manalanobis distance to the center of D.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, this gets wrong as the size of the sample gets small, or as the number of parameters go up.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An additional information I can use is a bunch of other similar sets of sample points belonging to similar distributions - so I could use that to get a &quot;prior&quot; over the parameters of D, and then update that according to the sample points I know it generated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My intuition is that I could have a mahalanobis distance DC using the covariance matrix of my sample points, and a mahalanobis distance DP using the covariance matrix of all the data points I have (not only those belonging to my distribution), and use a weighted sum of them as my distance metric function of the size of my sample from D (if the size is small, I'd better use DP, if it's large, I can use DC). But I'm not sure how to formalize that exactly, and feel I'm missing some conceptual tools.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(this is similar to a standard classification problem, but here I'm not interested in class membership, only in distance to a given class - the other classes are only useful to provide priors)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, how would you formally describe this problem? Is there a standard solution?&lt;/p&gt;&#xA;" OwnerUserId="1737" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-12-03T20:47:09.717" LastActivityDate="2010-12-06T11:43:57.957" Title="(Mahalanobis) distance to a multivariate distribution of which I have few sample points" Tags="&lt;classification&gt;&lt;multivariate-analysis&gt;&lt;distance-functions&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="5109" PostTypeId="1" AcceptedAnswerId="5113" CreationDate="2010-12-03T20:34:20.547" Score="1" ViewCount="232" Body="&lt;p&gt;When minimizing a function by general Metropolis-Hastings algorithms, the function is viewed as an unnormalized density of some distribution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(1) As density functions are required to be nonnegative, I was wondering if there is some restriction on functions that can be minimized by Metropolis-Hastings algorithms?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(2) To minimize the following function:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;$f(x) = 10 \sin(0.3x) \sin(1.3x^2) + 0.001 x^4 + 0.2 x+ 80.$&#xD;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;How do I transform it into some other function before I can apply Metropolis-Hastings algorithms?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks and regards!&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;EDIT:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks! I agree Simulated Annealing might better handling such situation, but this is an assignment that requires to use Metropolitan-Hastings algorithms. So I am afraid I have not other choice.&lt;/p&gt;&#xA;" OwnerUserId="1005" LastEditorUserId="1005" LastEditorDisplayName="" LastEditDate="2010-12-05T00:35:31.170" LastActivityDate="2010-12-05T00:35:31.170" Title="Minimization of a function by Metropolis-Hastings algorithms" Tags="&lt;homework&gt;&lt;optimization&gt;&lt;monte-carlo&gt;&lt;metropolis-hastings&gt;&lt;simulated-annealing&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="5110" PostTypeId="2" ParentId="726" CreationDate="2010-12-03T21:05:59.607" Score="1" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;“Statistics is much like a streetlight. Not very enlightening, but nice for supporting you”&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://secure.wikimedia.org/wikipedia/en/wiki/Robert_Storm_Petersen#Quotes&quot; rel=&quot;nofollow&quot;&gt;Storm P&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2247" LastActivityDate="2010-12-03T21:05:59.607" CommunityOwnedDate="2010-12-03T21:05:59.607" CommentCount="1" />
  <row Id="5111" PostTypeId="1" CreationDate="2010-12-03T22:19:21.333" Score="6" ViewCount="223" Body="&lt;p&gt;I have a question regarding the sign test when the individual measurements may be correlated. Let me start off with some background. Suppose we have 4 Organisms (a,b,c,d),and we make measurements in two separate ways, say A and B. Our data may look as follows&lt;/p&gt;&#xA;&#xA;&lt;p&gt;a = 3 for measurement A and 1 for measurement B&lt;br&gt;&#xA;b = 4 for measurement A and 3 for measurement B&lt;br&gt;&#xA;c = 0 for A and 4 for B&lt;br&gt;&#xA;d = 2 for A and 0 for B&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We now take the difference between A and B: $2,1,-4,2$. Looking at the signs we get the pattern $++-+$. We want to test if there is any difference between method A and method B. Take:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$H_0$(null hypothesis) = distribution for A is equal to the distribution for B&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Under $H_0$ we would expect $\textrm{Pr}(A&amp;gt;B)=\textrm{Pr}(B&amp;gt;A)=.5$, therefore any pattern of $+$'s and $-$'s would be equally likely. i.e. $--+-$ is as likely to occur as&#xA;$+-++$ etc. Let $U =$ number of $+$'s (in our case $U=3$). Assuming $H_0$ one can show that $\textrm{Pr}(U\ge3) = (1+4)/2^4 = 5/16=0.3125$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, suppose the a and b are strongly positively correlated. Therefore not&#xA;all combinations of $+$'s and $-$'s would be equally likely.For example one would not expect to have a &gt; b for method A and a &amp;lt; b for method B. Therefore we would not expect sequences like $+-..$ or $-+..$ to occur. Taking this into account assuming $H_0$ it turns out that $\textrm{Pr}(U\ge3) = 3/8=0.375$, i.e. our p&#xA;value increases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I come to my question: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;If instead of 4 organisms, I have say 100&#xA;  organisms, and also suppose I have an upper bound on the number of&#xA;  correlations and the size of each correlation. Is there any way to&#xA;  construct an upper bound on the p value?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="2475" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-12-20T19:28:26.343" LastActivityDate="2010-12-21T18:22:10.200" Title="Nonparametric sign test for correlated variables" Tags="&lt;hypothesis-testing&gt;&lt;nonparametric&gt;" AnswerCount="2" FavoriteCount="2" />
  <row Id="5112" PostTypeId="2" ParentId="5109" CreationDate="2010-12-03T23:45:11.203" Score="2" ViewCount="" Body="&lt;p&gt;If you want to find the global minimum of a function, &lt;a href=&quot;http://en.wikipedia.org/wiki/Simulated_annealing&quot; rel=&quot;nofollow&quot;&gt;simulated annealing&lt;/a&gt; would be the algorithm to look at, in which case there is no need to view the function as an unnormalised density of any kind and no need to transform the function.&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2010-12-03T23:45:11.203" CommentCount="1" />
  <row Id="5113" PostTypeId="2" ParentId="5109" CreationDate="2010-12-03T23:50:56.123" Score="3" ViewCount="" Body="&lt;p&gt;You are rather looking for a simulated annealing, which is easier to understand when formulated in the original, physics way:&lt;br&gt;&#xA;Having&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;$x$ is a state of the system&lt;/li&gt;&#xA;&lt;li&gt;$f(x)$ is an energy of the system; energy is defined up to addition of a constant, so there is no problem with it being negative or positive -- the only constraint is the-lower-the-better&lt;/li&gt;&#xA;&lt;li&gt;$T$ is a temperature of the system, and $kT$ is this temperature expressed in energy units&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;the MH algorithm, formulated as&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;$x&amp;#39;=x+\text{random change}$&#xD;&lt;/li&gt;&#xA;&lt;li&gt;if $\big\{\exp{\frac{f(x)-f(x&amp;#39;)}{kT}}&amp;gt;R(0;1)\big\}$ than $x=x&amp;#39;$ &lt;/li&gt;&#xA;&lt;li&gt;goto 1&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;recreates the same occupancy of particular states (values of $x$) as for a system placed in temperature $T$, as shown by Metropolis in his pioneer work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Physical systems tend to spontaneously go into the lowest energy state and stay there, so after initial relaxation $x$ should generally oscillate around minimum. Those oscillations' amplitude depends on temperature, so simulated annealing reduces $T$ during run to reduce them and make more accurate localisation of minimum. Initially high temperature is required to prevent the system from landing and staying stuck for a very long time in some local minimum.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: If you were told that you should use just Metropolis-Hastings, it most probably means you should do this this way, only with constant temperature (constantly deceasing temperature is the only thing that simulated annealing adds to MH). Then you will need to make few experiments to figure out a good temperature, so that there will be quite good accuracy no stucking in the local minima.&lt;/p&gt;&#xA;" OwnerUserId="88" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-04T12:44:09.460" LastActivityDate="2010-12-04T12:44:09.460" CommentCount="3" />
  <row Id="5114" PostTypeId="1" AcceptedAnswerId="5411" CreationDate="2010-12-03T23:57:59.227" Score="7" ViewCount="227" Body="&lt;p&gt;I am trying to state a prior distribution for a Bayesian meta-analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have the following information about a random variable: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Two observations: 3.0, 3.6&lt;/li&gt;&#xA;&lt;li&gt;a scientist who studies the variable has told me that $P(X&amp;lt;2)=P(X&amp;gt;8)=0$, and that values as high as 6 have nonzero probability.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I have used the following approach to optimization (the mode of log-N = $e^{\mu-\sigma^2)}$:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;prior &amp;lt;- function(parms, x, alpha) {&#xA;  a &amp;lt;- abs(plnorm(x[1], parms[1], parms[2]) - (alpha/2))&#xA;  b &amp;lt;- abs(plnorm(x[2], parms[1], parms[2]) - (1-alpha/2))&#xA;  mode &amp;lt;- exp(parms[1] - parms[2]^2)&#xA;  c &amp;lt;- abs(mode-3.3)&#xA;  return(a + b + c)&#xA;}&#xA;v = nlm(prior,c(log(3.3),0.14),alpha=0.05,x=c(2.5,7.5))&#xA;x &amp;lt;- seq(1,10,0.1)&#xA;plot(x, dlnorm(x, v$estimate[1], v$estimate[2]))&#xA;abline(v=c(2.5,7.5), lty=2) #95%CI&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/hvKna.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the figure, you can see the distribution that this returns, but I would like to find something more like the red lines I have drawn in.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This provides the same shaped distribution using the lognormal, gamma, or the normal, and it results in a distribution with $P(X=5)&amp;lt;0.05$ and $P(X=6)&amp;lt; 0.01$, i.e.:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; plnorm(c(5,6), v$estimate[1],v$estimate[2])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Can anyone suggest alternatives? I would prefer to stick with a single distribution rather than a mixture.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="1381" LastEditorUserId="1381" LastEditorDisplayName="" LastEditDate="2010-12-15T17:34:52.937" LastActivityDate="2010-12-15T17:34:52.937" Title="Seeking a distribution, perhaps uncommon, consistent with two data points and expert constraints?" Tags="&lt;r&gt;&lt;distributions&gt;&lt;probability&gt;&lt;bayesian&gt;&lt;optimization&gt;" AnswerCount="4" CommentCount="11" />
  <row Id="5115" PostTypeId="1" AcceptedAnswerId="5124" CreationDate="2010-12-04T00:08:23.027" Score="12" ViewCount="1149" Body="&lt;p&gt;Who are the most important statisticians and what is it that made them famous? (Reply just one scientist per answer please)&lt;/p&gt;&#xA;" OwnerUserId="1808" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-12-13T23:12:56.030" LastActivityDate="2011-06-01T17:32:23.790" CommunityOwnedDate="2010-12-04T00:12:46.677" Title="Most famous statisticians" Tags="&lt;big-list&gt;&lt;history&gt;" AnswerCount="27" CommentCount="6" FavoriteCount="5" />
  <row Id="5116" PostTypeId="2" ParentId="5115" CreationDate="2010-12-04T00:18:06.090" Score="9" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Laplace&quot; rel=&quot;nofollow&quot;&gt;Pierre-Simon Laplace&lt;/a&gt; for work on fundamentals of (Bayesian) probability.&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2010-12-04T00:18:06.090" CommunityOwnedDate="2010-12-04T00:18:06.090" />
  <row Id="5117" PostTypeId="2" ParentId="5115" CreationDate="2010-12-04T00:36:08.460" Score="28" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Ronald_Fisher&quot; rel=&quot;nofollow&quot;&gt;Ronald Fisher&lt;/a&gt; for his fundamental contributions to the way we analyze data, whether it be the analysis of variance framework, maximum likelihood, permutation tests, or any number of other ground-breaking discoveries. &lt;/p&gt;&#xA;" OwnerUserId="1118" LastActivityDate="2010-12-04T00:36:08.460" CommunityOwnedDate="2010-12-04T00:36:08.460" CommentCount="5" />
  <row Id="5118" PostTypeId="2" ParentId="5115" CreationDate="2010-12-04T01:32:33.420" Score="13" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Karl_Pearson&quot; rel=&quot;nofollow&quot;&gt;Karl Pearson&lt;/a&gt; for his work on mathematical statistics.  Pearson correlation, Chi-square test, and principal components analysis are just a few of the incredibly important ideas that stem from his works.&lt;/p&gt;&#xA;" OwnerUserId="1028" LastActivityDate="2010-12-04T01:32:33.420" CommunityOwnedDate="2010-12-04T01:32:33.420" CommentCount="1" />
  <row Id="5119" PostTypeId="1" AcceptedAnswerId="5120" CreationDate="2010-12-04T02:13:27.347" Score="2" ViewCount="280" Body="&lt;p&gt;Hi there. I asked this question on mathoverflow yesterday and got a suggestion to try it here. I apologize if this is an easy question, but I can't seem to find an answer anywhere.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm trying to duplicate a macroeconomic paper that uses MCMC analysis to derive time series and parameter values. For the variance of a white noise term the authors of the original paper choose an inverse gamma prior of mean 0.5 and having a 90% confidence interval between 0.21 and 0.79.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm using Dynare to run my estimation and Dynare only accepts these kinds of parameters in terms of standard error. Therefore, my question basically amounts to &quot;how do I find the square root of an inverse gamma distribution&quot;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In case it's not clear what I'm asking, here's how the model equation appears in the paper:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$a_t=\rho_a\times{a_{t-1}}+\epsilon_a;\hspace{20 pt}\epsilon_a\sim{N}(0,\sigma_a)$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$a_t$ follows a simple AR(1) process and $\epsilon_a$ is meant to be white noise with mean 0 and &lt;em&gt;variance&lt;/em&gt; $\sigma_a$. The prior for $\sigma_a$ is supposed to follow an inverse gamma distribution with the mean and confidence interval above. My question then is how can I find the &quot;square root of the distribution&quot; so I can write an expression for the standard error of $\epsilon_a$, instead of its variance?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If it's not possible to come up with a simple expression for this, would you have any suggestions for roughly equivalent specifications of the prior?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance&lt;/p&gt;&#xA;" OwnerUserId="2251" LastActivityDate="2010-12-04T03:10:39.210" Title="Square root of inverse gamma distribution?" Tags="&lt;distributions&gt;&lt;probability&gt;&lt;bayesian&gt;" AnswerCount="1" />
  <row Id="5120" PostTypeId="2" ParentId="5119" CreationDate="2010-12-04T02:51:18.860" Score="3" ViewCount="" Body="&lt;p&gt;You need not worry about the square root of the error variance. Instead, sample the error variance directly conditional on everything else. The posterior of the error variance will also be an inverse gamma given your model assumptions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, I am not sure why you want to know the square root of the inverse-gamma.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: A more careful reading suggests that the issue is with the specific software you are using. The following link seems to be useful: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.dynare.org/phpBB3/viewtopic.php?f=1&amp;amp;t=1372&quot; rel=&quot;nofollow&quot;&gt;http://www.dynare.org/phpBB3/viewtopic.php?f=1&amp;amp;t=1372&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In particular, the pdf file on page 5 seems to provide some guidance as to how to specify an inverse-gamma prior for the variance in Dynare.&lt;/p&gt;&#xA;" LastActivityDate="2010-12-04T03:10:39.210" CommentCount="2" />
  <row Id="5121" PostTypeId="2" ParentId="22" CreationDate="2010-12-04T03:06:00.983" Score="3" ViewCount="" Body="&lt;p&gt;Just a little bit of fun...&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;A Bayesian is one who, vaguely expecting a horse, and catching a glimpse of a donkey, strongly believes he has seen a mule.&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;From this site:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www2.isye.gatech.edu/~brani/isyebayes/jokes.html&quot; rel=&quot;nofollow&quot;&gt;http://www2.isye.gatech.edu/~brani/isyebayes/jokes.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and from the same site, a nice essay...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;An Intuitive Explanation of Bayes' Theorem&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://yudkowsky.net/rational/bayes&quot; rel=&quot;nofollow&quot;&gt;http://yudkowsky.net/rational/bayes&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="485" LastEditorUserId="485" LastEditorDisplayName="" LastEditDate="2010-12-04T03:12:36.393" LastActivityDate="2010-12-04T03:12:36.393" />
  <row Id="5122" PostTypeId="2" ParentId="5115" CreationDate="2010-12-04T03:37:32.367" Score="10" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Francis_Galton&quot; rel=&quot;nofollow&quot;&gt;Francis Galton&lt;/a&gt; for creating correlation and promoting regression. &lt;/p&gt;&#xA;" OwnerUserId="74" LastActivityDate="2010-12-04T03:37:32.367" CommunityOwnedDate="2010-12-04T03:37:32.367" CommentCount="5" />
  <row Id="5123" PostTypeId="2" ParentId="5115" CreationDate="2010-12-04T03:38:44.393" Score="9" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Andrey_Markov&quot; rel=&quot;nofollow&quot;&gt;Andrey Markov&lt;/a&gt; for stochastic processes and markov chains. &lt;/p&gt;&#xA;" OwnerUserId="74" LastActivityDate="2010-12-04T03:38:44.393" CommunityOwnedDate="2010-12-04T03:38:44.393" CommentCount="3" />
  <row Id="5124" PostTypeId="2" ParentId="5115" CreationDate="2010-12-04T03:46:20.583" Score="19" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Thomas_Bayes&quot; rel=&quot;nofollow&quot;&gt;Thomas Bayes&lt;/a&gt; for inventing Bayes' theorem&lt;/p&gt;&#xA;" OwnerUserId="74" LastActivityDate="2010-12-04T03:46:20.583" CommunityOwnedDate="2010-12-04T03:46:20.583" CommentCount="3" />
  <row Id="5125" PostTypeId="2" ParentId="5115" CreationDate="2010-12-04T03:48:36.697" Score="12" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Carl_Friedrich_Gauss&quot; rel=&quot;nofollow&quot;&gt;Carl Gauss&lt;/a&gt; for least squares&lt;/p&gt;&#xA;" OwnerUserId="74" LastActivityDate="2010-12-04T03:48:36.697" CommunityOwnedDate="2010-12-04T03:48:36.697" />
  <row Id="5126" PostTypeId="2" ParentId="22" CreationDate="2010-12-04T06:54:30.117" Score="4" ViewCount="" Body="&lt;p&gt;The Bayesian is asked to make bets, which may include anything from which fly will crawl up a wall faster to which medicine will save most lives, or which prisoners should go to jail. He has a big box with a handle. He knows that if he puts absolutely everything he knows into the box, including his personal opinion, and turns the handle, it will make the best possible decision for him.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The frequentist is asked to write reports. He has a big black book of rules. If the situation he is asked to make a report on is covered by his rulebook, he can follow the rules and write a report so carefully worded that it is wrong, at worst, one time in 100 (or one time in 20, or one time in whatever the specification for his report says).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The frequentist knows (because he has written reports on it) that the Bayesian sometimes makes bets that, in the worst case, when his personal opinion is wrong, could turn out badly. The frequentist also knows (for the same reason) that if he bets against the Bayesian every time he differs from him, then, over the long run, he will lose.&lt;/p&gt;&#xA;" OwnerUserId="1789" LastActivityDate="2010-12-04T06:54:30.117" />
  <row Id="5128" PostTypeId="1" AcceptedAnswerId="5130" CreationDate="2010-12-04T09:05:32.740" Score="6" ViewCount="168" Body="&lt;p&gt;We are preparing a manuscript and the editor asked us to convert a figure with a boxplot into a table &quot;because of the more exact data content&quot;. While I think boxplots are pretty decent in revealing something about the data, I was wondering what you guys think about this? Do you (often) opt for tables over boxplots for presenting your data?&lt;/p&gt;&#xA;" OwnerUserId="144" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-04T15:54:10.600" LastActivityDate="2010-12-04T15:54:10.600" Title="Boxplots as tables" Tags="&lt;data-visualization&gt;&lt;boxplot&gt;&lt;tables&gt;" AnswerCount="2" FavoriteCount="4" />
  <row Id="5129" PostTypeId="2" ParentId="5128" CreationDate="2010-12-04T09:22:22.180" Score="3" ViewCount="" Body="&lt;p&gt;It depends on your objective, if it is a quick visualization I will stick with boxplot but if it is a more detailed examination I would stay with the data&lt;/p&gt;&#xA;" OwnerUserId="1808" LastActivityDate="2010-12-04T09:22:22.180" />
  <row Id="5130" PostTypeId="2" ParentId="5128" CreationDate="2010-12-04T09:23:44.360" Score="12" ViewCount="" Body="&lt;p&gt;I tend to think that boxplots will convey more effective information if there are numerous empirical distributions that you want to summarize into a single figure. If you only have two or three groups, editors may ask you to provide numerical summaries instead, either because it is more suitable for the journal policy, or because readers won't gain much insight into the data from a figure. If you provide the three quartiles, range, and optionally the mean $\pm$ SD, then an advertised reader should have a clear idea of the shape of the distribution (symmetry, presence of outlying values, etc.).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would suggest two critical reviews by Andrew Gelman (the first goes the other way around, but still it provides insightful ideas):&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Gelman, A, Pasarica, C, and Dodhia, R. &lt;a href=&quot;http://www.stat.columbia.edu/~gelman/research/published/dodhia.pdf&quot; rel=&quot;nofollow&quot;&gt;Let's practice what we preach&lt;/a&gt;. The American Statistician (2002) 56(2): 121-130. &lt;/li&gt;&#xA;&lt;li&gt;Gelman, A. &lt;a href=&quot;http://www.stat.columbia.edu/~gelman/research/published/tables4.pdf&quot; rel=&quot;nofollow&quot;&gt;Why Tables are Really Much Better than Graphs&lt;/a&gt;. (also &lt;a href=&quot;http://www.stat.columbia.edu/~cook/movabletype/archives/2009/04/why_tables_are.html&quot; rel=&quot;nofollow&quot;&gt;discussed on his blog&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-12-04T09:50:30.397" LastActivityDate="2010-12-04T09:50:30.397" />
  <row Id="5131" PostTypeId="2" ParentId="5115" CreationDate="2010-12-04T10:52:07.720" Score="5" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Edwin_Thompson_Jaynes&quot; rel=&quot;nofollow&quot;&gt;Edwin Thompson Jaynes&lt;/a&gt; for work on objective Bayesian methods, particularly MaxEnt and transformation groups.&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2010-12-04T10:52:07.720" CommunityOwnedDate="2010-12-04T10:52:07.720" CommentCount="1" />
  <row Id="5132" PostTypeId="2" ParentId="5115" CreationDate="2010-12-04T10:54:08.763" Score="4" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Harold_Jeffreys&quot; rel=&quot;nofollow&quot;&gt;Harold Jeffreys&lt;/a&gt; for revival of Bayesian interpretation of probability.&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2010-12-04T10:54:08.763" CommunityOwnedDate="2010-12-04T10:54:08.763" />
  <row Id="5133" PostTypeId="2" ParentId="5115" CreationDate="2010-12-04T11:07:26.660" Score="11" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Bradley_Efron&quot; rel=&quot;nofollow&quot;&gt;Bradley Efron&lt;/a&gt; for the &lt;a href=&quot;http://en.wikipedia.org/wiki/Bootstrapping_%28statistics%29&quot; rel=&quot;nofollow&quot;&gt;Bootstrap&lt;/a&gt; - one of the most useful techniques in computational statistics.&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2010-12-04T11:07:26.660" CommunityOwnedDate="2010-12-04T11:07:26.660" />
  <row Id="5134" PostTypeId="2" ParentId="4451" CreationDate="2010-12-04T11:27:56.983" Score="1" ViewCount="" Body="&lt;p&gt;Just found this: &lt;a href=&quot;http://snap.stanford.edu/data/twitter7.html&quot; rel=&quot;nofollow&quot;&gt;476 million Twitter tweets&lt;/a&gt; (via &lt;a href=&quot;https://twitter.com/#!/yarapavan&quot; rel=&quot;nofollow&quot;&gt;@yarapavan&lt;/a&gt;).&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-12-04T11:27:56.983" CommunityOwnedDate="2010-12-04T11:27:56.983" />
  <row Id="5135" PostTypeId="1" AcceptedAnswerId="5138" CreationDate="2010-12-04T11:28:14.300" Score="6" ViewCount="1198" Body="&lt;p&gt;Hi,&lt;br&gt;&#xA;the help pages in R assume I know what those numbers mean. I don't :)&#xA;I'm trying to really intuitively understand every number here. I will just post the output and comment on what I found out. There might (will) be mistakes, as I'll just write what I assume. Please correct me, and I will edit the wrong parts.&lt;br&gt;&#xA;Mainly I'd like to know what the t-value in the coefficients mean, and why they print the residual standard error. I hope someone can clarify that.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Call:&#xA;lm(formula = iris$Sepal.Width ~ iris$Petal.Width)&#xA;&#xA;Residuals:&#xA;     Min       1Q   Median       3Q      Max &#xA;-1.09907 -0.23626 -0.01064  0.23345  1.17532 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;A 5-point-summary of the residuals (Their mean is always 0, right?). The numbers can be used  (I'm guessing here) to quickly see if there are any big outliers. Also you can already see it here if the residuals are far from normal distributed (which they should be).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Coefficients:&#xA;                 Estimate Std. Error t value Pr(&amp;gt;|t|)    &#xA;(Intercept)       3.30843    0.06210  53.278  &amp;lt; 2e-16 ***&#xA;iris$Petal.Width -0.20936    0.04374  -4.786 4.07e-06 ***&#xA;---&#xA;Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Estimates $\hat{\beta_i}$ , computed by least squares regression. Also, the standard error $\sigma_{\beta_i}$ . I'd like to know how this is calculated.&lt;br&gt;&#xA;Also, no idea where the t value and the corresponding p come from. I know $\hat{\beta}$ should be normal distributed, but how is the t value calculated?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Residual standard error: 0.407 on 148 degrees of freedom&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;$\sqrt{ \frac{1}{n-p} \epsilon^T\epsilon }$ , I guess. But why do we calculate that, and what does it say us?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Multiple R-squared: 0.134,  Adjusted R-squared: 0.1282 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;$ R^2 = \frac{s_\hat{y}^2}{s_y^2} $ , which is $ \frac{\sum_{i=1}^n (\hat{y_i}-\bar{y})^2}{\sum_{i=1}^n (y_i-\bar{y})^2} $ . The ratio is close to 1 if the points lie on a straight line, and 0 if they are random.&lt;br&gt;&#xA;What is the adjusted R-squared?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;F-statistic: 22.91 on 1 and 148 DF,  p-value: 4.073e-06 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;F and p for the &lt;strong&gt;whole&lt;/strong&gt; model, not only for single $\beta_i$s as previous. The F value is $ \frac{s^2_{\hat{y}}}{\sum\epsilon_i} $ . The bigger it grows, the more unlikely it is that the $\beta$'s do not have any effect at all.&lt;/p&gt;&#xA;" OwnerUserId="2091" LastActivityDate="2010-12-04T15:53:02.953" Title="Interpretation of R's lm() output" Tags="&lt;r&gt;&lt;regression&gt;&lt;output&gt;" AnswerCount="1" CommentCount="4" FavoriteCount="9" />
  <row Id="5136" PostTypeId="1" AcceptedAnswerId="5163" CreationDate="2010-12-04T11:41:22.610" Score="2" ViewCount="145" Body="&lt;p&gt;The problem is to define when a variable might be considered as a latent variable. I am interested in how to describe a latent variable, and what are the properties of latent variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My twofold question is: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;When you try to explain what a latent variable is, what do you consider as the main differences between a manifest and a latent variable?&lt;/li&gt;&#xA;&lt;li&gt;When does factor analysis or item response modeling seem more appropriate ?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Example.&lt;/strong&gt;&#xA;On the one hand, if you want to measure fish weight without any instrument, you can devise items to measure it. In this case, do you rely on a latent variable model? On the other hand, social level is sometimes measured directly through answers to a survey with a linear model (or other models) applied to items such as &lt;em&gt;highest diploma&lt;/em&gt;, &lt;em&gt;numbers of books at home&lt;/em&gt;, &lt;em&gt;numbers of electronics devices&lt;/em&gt;, but not by considering a latent variable. But why can't we use a latent variable model in this case?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="1154" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-12-04T20:14:58.293" LastActivityDate="2010-12-05T12:21:37.620" Title="When do you consider a variable is a latent variable ?" Tags="&lt;psychometrics&gt;&lt;latent-variable&gt;" AnswerCount="2" CommentCount="3" FavoriteCount="1" />
  <row Id="5137" PostTypeId="2" ParentId="5011" CreationDate="2010-12-04T12:13:36.183" Score="3" ViewCount="" Body="&lt;p&gt;A quick google brings up this, which indicates that when working with circular data you'll need a different definition of 'bias' for a start:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;However, when using data on the circle, we cannot use distance in Euclidean space, so all differences &lt;em&gt;θ − θ&lt;sub&gt;i&lt;/sub&gt;&lt;/em&gt; should be replaced by considering the angle between two vectors:&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;$d_i\theta)= \| \theta -\theta_i \| = \min(|\theta-\theta_i|, 2π -|\theta-\theta_i|).$&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;-- Charles C Taylor. Automatic bandwidth selection for circular density estimation. &lt;em&gt;Computational Statistics &amp;amp; Data Analysis&lt;/em&gt;&#xA;Volume 52, Issue 7, 15 March 2008, Pages 3493-3500. doi: &lt;a href=&quot;http://dx.doi.org/10.1016/j.csda.2007.11.003&quot; rel=&quot;nofollow&quot;&gt;10.1016/j.csda.2007.11.003&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;He references these books:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;S. Rao Jammalamadaka and A. SenGupta, &lt;em&gt;Topics in Circular Statistics&lt;/em&gt;, World Scientific, Singapore (2001).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;K.V. Mardia and P.E. Jupp, &lt;em&gt;Directional Statistics&lt;/em&gt;, John Wiley, Chichester (1999).&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-12-04T12:13:36.183" />
  <row Id="5138" PostTypeId="2" ParentId="5135" CreationDate="2010-12-04T12:59:27.017" Score="16" ViewCount="" Body="&lt;p&gt;Five point summary: yes, the idea is to give a quick summary of the distribution. It should be roughly symmetrical about mean, the median should be close to 0, the 1Q and 3Q values should ideally be roughly of similar absolute magnitude etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each coefficient in the model is a Gaussian (Normal) random variable. The $\hat{\beta_i}$ is the estimate of the mean of the distribution of that random variable, and the standard error is the square root of the variance of that distribution. It is a measure of the uncertainty in the estimate of the $\hat{\beta_i}$. You can look at how these are computed (well the mathematical formulae used) on &lt;a href=&quot;http://en.wikipedia.org/wiki/Ordinary_least_squares&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;. Note that any self-respecting stats programme will &lt;strong&gt;not&lt;/strong&gt; use the standard mathematical equations to compute the $\hat{\beta_i}$ because doing them on a computer can lead to a large loss of precision in the computations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The $t$ statistics are the estimates ($\hat{\beta_i}$) divided by their standard errors ($\hat{\sigma_i}$), e.g. $t_i = \frac{\hat{\beta_i}}{\hat{\sigma_i}}$. Assuming you have the same model in object &lt;code&gt;mod&lt;/code&gt;as your Q:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; mod &amp;lt;- lm(Sepal.Width ~ Petal.Width, data = iris)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;then the $t$ values R reports are computed as:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; (tstats &amp;lt;- coef(mod) / sqrt(diag(vcov(mod))))&#xA;(Intercept) Petal.Width &#xA;  53.277950   -4.786461 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Where &lt;code&gt;coef(mod)&lt;/code&gt; are the $\hat{\beta_i}$, and &lt;code&gt;sqrt(diag(vcov(mod)))&lt;/code&gt; gives the square roots of the diagonal elements of the covariance matrix of the model parameters, which are the standard errors of the parameters ($\hat{\sigma_i}$).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The p-value is the probability of achieving a value of $t$ as larger or larger if the null hypothesis were true. Here the null hypothesis is the $\hat{\beta_i}$ are individually 0. They are computed as (using &lt;code&gt;tstats&lt;/code&gt; from above):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; 2 * pt(abs(tstats), df = df.residual(mod), lower.tail = FALSE)&#xA; (Intercept)  Petal.Width &#xA;1.835999e-98 4.073229e-06&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So we compute the upper tail probability of achieving the $t$ values we did from a $t$ distribution with degrees of freedom equal to the residual degrees of freedom of the model. This represents the probability of achieving a $t$ value greater than the absolute values of the observed $t$s. It is multiplied by 2, because of course $t$ can be large in the negative direction too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The residual standard error is an estimate of the parameter $\sigma$. The assumption in ordinary least squares is that the residuals are individually described by a Guassian (normal) distribution with mean 0 and standard deviation $\sigma$. The $\sigma$ relates to the constant variance assumption; each residual has the same variance and that variance is equal to $\sigma^2$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The adjusted $R^2$ is the same thing as $R^2$, but adjusted for the complexity of the model, i.e. the number of parameters in the model. Think about things for a minute. If we have a model with a single parameter. It will have a certain $R^2$. If we add another parameter to this model, the $R^2$ of the new model has to increase, &lt;strong&gt;even&lt;/strong&gt; if the added parameter has no statistical power. The adjusted $R^2$ tries to account for this, by including information on the number of parameters in the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Adjusted $R^2$ is computed as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$1 - (1 - R^2) \frac{n - 1}{n - p - 1}$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The $F$ is the ratio of two variances, the variance explained by the parameters in the model and the residual or unexplained variance. You can see this better if we get the ANOVA table for the model via &lt;code&gt;anova()&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; anova(mod)&#xA;Analysis of Variance Table&#xA;&#xA;Response: Sepal.Width&#xA;             Df  Sum Sq Mean Sq F value    Pr(&amp;gt;F)    &#xA;Petal.Width   1  3.7945  3.7945   22.91 4.073e-06 ***&#xA;Residuals   148 24.5124  0.1656                      &#xA;---&#xA;Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;First, notice the $F$s are the same in the ANOVA output and the &lt;code&gt;summary(mod)&lt;/code&gt; output. The &lt;code&gt;Mean Sq&lt;/code&gt; column contains the two variances and $3.7945 / 0.1656 = 22.91$. We can compute the probability of achieving an $F$ that large under the null hypothesis of no effect, from an $F$-distribution with 1 and 148 degrees of freedom. This is what is reported in the final column of the ANOVA table. In the simple case of a single, continuous predictor (as per your example), $F = t_{\mathrm{Petal.Width}}^2$, which is why the p-values are the same. This equivalence only holds in this simple case.&lt;/p&gt;&#xA;" OwnerUserId="1390" LastEditorUserId="1390" LastEditorDisplayName="" LastEditDate="2010-12-04T15:53:02.953" LastActivityDate="2010-12-04T15:53:02.953" CommentCount="6" />
  <row Id="5139" PostTypeId="2" ParentId="5136" CreationDate="2010-12-04T15:12:21.650" Score="1" ViewCount="" Body="&lt;p&gt;That is a modeling decision. One way to look at it can be illustrated by the following example. &#xA;A couple of hundreds electrodes are attached to the head to measure brain activity. Electricity, blood flow, whatever and you get lots of signals. These measurements that you get are observables. They are mixed in probably very non-linear way and are not useful.&lt;br&gt;&#xA;Latent or also hidden variables are modeling the individual variables that are responsible for their generations. They are supposed to be more pure, more interpretable. How to extract the signal that is causing the eye to blink, or to open the mouth, or emotions and many more complicated signals. Hope it helps to understand the intuition.&lt;/p&gt;&#xA;" OwnerUserId="2084" LastActivityDate="2010-12-04T15:12:21.650" CommentCount="1" />
  <row Id="5140" PostTypeId="2" ParentId="5115" CreationDate="2010-12-04T15:52:26.967" Score="6" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Leo_Breiman&quot; rel=&quot;nofollow&quot;&gt;Leo Breiman&lt;/a&gt; for bagging, and random forests.&lt;/p&gt;&#xA;" OwnerUserId="88" LastEditorUserId="74" LastEditorDisplayName="" LastEditDate="2011-05-21T20:49:32.533" LastActivityDate="2011-05-21T20:49:32.533" CommunityOwnedDate="2010-12-04T15:52:26.967" CommentCount="3" />
  <row Id="5141" PostTypeId="1" AcceptedAnswerId="5143" CreationDate="2010-12-04T18:18:59.190" Score="1" ViewCount="497" Body="&lt;p&gt;Hi,&#xA;Does anybody know how to bold the x and y axis labels (&quot;Single sample&quot; and &quot;Value axis&quot; in an R boxplot?&#xA;I appreciate your help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is what my boxplot looks &#xA;&lt;a href=&quot;http://img822.imageshack.us/img822/331/23807704.png&quot; rel=&quot;nofollow&quot;&gt;http://img822.imageshack.us/img822/331/23807704.png&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2241" LastActivityDate="2010-12-04T18:43:26.597" Title="Boxplot labels in R" Tags="&lt;r&gt;&lt;boxplot&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="5143" PostTypeId="2" ParentId="5141" CreationDate="2010-12-04T18:43:26.597" Score="2" ViewCount="" Body="&lt;p&gt;Using this example data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dat &amp;lt;- data.frame(values = c(rnorm(100, mean = 1), rnorm(100, mean = 3),&#xA;                             rnorm(100, mean = 4, sd = 3)),&#xA;                  groups = factor(rep(c(&quot;aaa&quot;,&quot;bbb&quot;,&quot;ccc&quot;), each = 100)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;There are several ways. One is to use &lt;code&gt;?plotmath&lt;/code&gt; functionality and the &lt;code&gt;plotmath&lt;/code&gt; &quot;function&quot; &lt;code&gt;bold()&lt;/code&gt; in an expression:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;boxplot(values ~ groups, data = dat,&#xA;        ylab = expression(bold(Value~axis)),&#xA;        xlab = expression(bold(Single~sample)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;or, similarly&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;boxplot(values ~ groups, data = dat,&#xA;        ylab = expression(bold(&quot;Value axis&quot;)),&#xA;        xlab = expression(bold(&quot;Single sample&quot;)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Another way is to leave the titles off the plot and then add them with the &lt;code&gt;title()&lt;/code&gt; function using the bold font:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;boxplot(values ~ groups, data = dat)&#xA;title(ylab = &quot;Value axis&quot;, xlab = &quot;Single sample&quot;, font.lab = 2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;We need graphical parameter &lt;code&gt;font.lab&lt;/code&gt; as this is the parameter that controls the axis labels. Read the entries in &lt;code&gt;?par&lt;/code&gt; for more info.&lt;/p&gt;&#xA;" OwnerUserId="1390" LastActivityDate="2010-12-04T18:43:26.597" CommentCount="1" />
  <row Id="5144" PostTypeId="1" AcceptedAnswerId="5146" CreationDate="2010-12-04T18:47:22.257" Score="2" ViewCount="102" Body="&lt;p&gt;I have an experiment with any possible (reasonable) number of parameters (independent variables). I run the experiment several times for each possible combination of my variables. The data I get will be generally numeric. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However I know nothing (and any assumptions are difficult) about the distribution of my data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I am interested in is a measure of how well do my parameters predict the data I get. Which statistic should I use? How do I calculate it (by hand, a link to a tutorial would be very sweet)?&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Edit&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;I am trying to solve this as generally as possible (hence the slightly non-specific description) for a piece of software I'm working on. To make it a bit more clear a bit of an example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have these parameters:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;decay: 0.1 | 0.2 | 0.3&#xA;particles: 10 | 100&#xA;velocity: 30 | 70&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This gives 12 combinations (3 * 2 * 2) and I'll measure my dependent variable (say temperature) five times for each combinations. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus my final dataset will have 60 measurements of temperature.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now suppose that temperature was in fact given by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$t = K(0.3v + 0.6d + \varepsilon)$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $t$ is temperature, $K$ is some constant, $v$ is velocity, $d$ is decay and $\varepsilon$ is some sort of random effect. Particles is completely unrelated to the measured temperature.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I'd like to perform a test that would tell me that velocity has a ~0.3 effect, decay ~0.6 and particles ~0 effect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However I may have more or less variables and more or less measurements.&lt;/p&gt;&#xA;" OwnerUserId="2261" LastEditorUserId="2261" LastEditorDisplayName="" LastEditDate="2010-12-06T03:00:55.350" LastActivityDate="2010-12-06T03:00:55.350" Title="Appropriate test for multivariate experiment result with unknown distributions" Tags="&lt;experiment-design&gt;" AnswerCount="1" CommentCount="4" />
  <row Id="5145" PostTypeId="2" ParentId="5115" CreationDate="2010-12-04T20:42:26.150" Score="16" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/William_Sealy_Gosset&quot; rel=&quot;nofollow&quot;&gt;William Sealy Gosset&lt;/a&gt; for Student's t-distribution and the statistically-driven improvement of beer.&lt;/p&gt;&#xA;" OwnerUserId="2077" LastActivityDate="2010-12-04T20:42:26.150" CommunityOwnedDate="2010-12-04T20:42:26.150" CommentCount="1" />
  <row Id="5146" PostTypeId="2" ParentId="5144" CreationDate="2010-12-04T21:13:59.517" Score="6" ViewCount="" Body="&lt;p&gt;Well, following your update, it seems you are dealing with a factorial experiment (&lt;em&gt;factorial&lt;/em&gt; means that every factors are crossed, or, in other words, each unit is subjected to every possible combination of your factors), with five replicates. Let assume that these are not the same statistical units whose temperature is repeatedly measured across each of the 12 combinations (for the sake of clarity).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An &lt;a href=&quot;http://en.wikipedia.org/wiki/Analysis_of_variance&quot; rel=&quot;nofollow&quot;&gt;ANalysis Of VAriance&lt;/a&gt; (ANOVA) seems to be the most appropriate method to deal with this design. Basically, it will allow you to estimate the contribution of each source of variance (decay, particles, and velocity) wrt. the total variance in the observed temperature. What is not explained by these factors is called the residual variance (what you call the 'random effect'). A full additive model (i.e., without modeling interaction between your factors) will read something like&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;y_{ijkl}=\mu + \alpha_i + \beta_j + \gamma_k + \varepsilon_{ijkl},&#xD;&#xA;$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $y_{ijkl}$ is the temperature for unit $l$ when considering levels $i=1\dots a$, $j=1\dots b$, and $k=1\dots c$, of factors $\alpha$ (decay), $\beta$ (particles), and $\gamma$ (velocity); the $\varepsilon_{ijk}$ are the residuals assumed to follow a gaussian distribution of unknown variance, $\sigma^2$. They can be viewed as random fluctuations around $\mu$, the overall mean, and reflect the between-unit variations that are not accounted for by the other factors. The $\alpha_i$, $\beta_j$, and $\gamma_k$ can be viewed as factor-specific deviations from the overall mean $\mu$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The so-called &lt;em&gt;main effect&lt;/em&gt; of decay, particles, and velocity will be estimated by forming a ratio between the variance that they account for (known as &lt;em&gt;mean squares&lt;/em&gt;) and the residual variance (what is left after considering all variance explained by those factors), which is known to follow a Fisher-Snedecor (F) distribution, with $d-1$ and $N-abc$ degrees of freedom, where $d=a$, $b$, or $c$ stands for the number of levels of $\alpha$ (decay), $\beta$ (particles), and $\gamma$ (velocity). A significant effect (following an &lt;a href=&quot;http://en.wikipedia.org/wiki/Statistical_hypothesis_testing&quot; rel=&quot;nofollow&quot;&gt;hypothesis test&lt;/a&gt; of a null effect, i.e. $H_0:\, \mu_i=\mu_j\,\, \forall i\neq j$ &lt;em&gt;vs.&lt;/em&gt; $H_1:$ at least two of the $\mu_i$'s differ) would indicate that the factor under consideration has a significant effect on the outcome. This is readily obtained by any statistical software. For instance, in R you would use something like&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;summary(aov(temperature ~ decay + particles + velocity, data=df))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;provided temperature and factor levels are organized in four columns, in a data.frame named &lt;code&gt;df&lt;/code&gt;, as suggested below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;t1 0.1 10 30&#xA;t2 0.1 10 30&#xA;t3 0.1 10 30&#xA;t4 0.1 10 30&#xA;t5 0.1 10 30&#xA;t6 0.2 10 30&#xA;t7 0.2 10 30&#xA;...&#xA;t60 0.3 100 70&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The effect of any of the three factors can also be summarized under an equation like the one you referred to sy simply calling (again under R):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;summary.lm(aov(temperature ~ decay + particles + velocity))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This follows from the fact that an ANOVA is nothing more than a &lt;a href=&quot;http://en.wikipedia.org/wiki/Linear_model&quot; rel=&quot;nofollow&quot;&gt;Linear Model&lt;/a&gt; that you may have heard about (think of a regression model where the explanatory variables are all categorical).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should you want to account for possible interactions between all three factors, you need to add three second-order and one three-order interaction terms. If any of these effects prove to be significant, this would mean that the effect of the corresponding factors cannot be considered in isolation one from the other (e.g., the effect of decay on temperature is not the same depending on the number of particles).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for references, I would suggest starting with on-line tutorial or textbook like &lt;a href=&quot;http://www.psych.nyu.edu/cohen/three_way_ANOVA.pdf&quot; rel=&quot;nofollow&quot;&gt;Three-Way ANOVA&lt;/a&gt;, by Barry Cohen, or &lt;a href=&quot;http://cran.r-project.org/doc/contrib/Faraway-PRA.pdf&quot; rel=&quot;nofollow&quot;&gt;Practical Regression and Anova using R&lt;/a&gt;, by John MainDonald (but see also other textbooks available on &lt;a href=&quot;http://cran.r-project.org/other-docs.html&quot; rel=&quot;nofollow&quot;&gt;CRAN documentation&lt;/a&gt;). The definitive reference is Montgomery, &lt;a href=&quot;http://bcs.wiley.com/he-bcs/Books?action=index&amp;amp;itemId=047148735X&amp;amp;bcsId=2172&quot; rel=&quot;nofollow&quot;&gt;Design and Analysis of Experiments&lt;/a&gt; (Wiley, 2005).&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-12-04T21:23:14.870" LastActivityDate="2010-12-04T21:23:14.870" CommentCount="2" />
  <row Id="5147" PostTypeId="1" AcceptedAnswerId="5148" CreationDate="2010-12-04T23:02:34.320" Score="1" ViewCount="120" Body="&lt;p&gt;When trying to find the mode of a nonnegative function $f$ (i.e. maximize the function), one way to do it is to sampling the function viewed as an unnormalized density of some distribution via MCMC.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose we have had a sufficiently long sequence of samples via this method, I was wondering how to determine the mode from the samples?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Specifically, as we know that a part taken from the end of the sequence will be assumed to be approximately subject to the distribution corresponding to the target function $f$. As far as I have thought already, there might be two choices for the estimation of the mode of the function: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;take the last sample in the original sequence, &lt;/li&gt;&#xA;&lt;li&gt;take a small subsequence from the end of the original sequence and evaluate the function f on every sample in the subsequence and pick the one with the maximal function value. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I saw the first choice from my class note, but the second was my first thought before looking at the note. So I was  wondering what choice for the mode of the target function might be reasonable or better and why? Possible to have any references on this? You don't have to limit your scope to the two I just mentioned.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks and regards!&lt;/p&gt;&#xA;" OwnerUserId="1005" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-05T00:01:21.373" LastActivityDate="2010-12-05T00:01:21.373" Title="Finding the mode of a function by MCMC sampling" Tags="&lt;mcmc&gt;&lt;optimization&gt;&lt;monte-carlo&gt;" AnswerCount="1" />
  <row Id="5148" PostTypeId="2" ParentId="5147" CreationDate="2010-12-04T23:24:09.753" Score="3" ViewCount="" Body="&lt;p&gt;The mode is indeed the maximum of f(x), so the value of x encountered during the simulation that gives the highest value of f(x) ought to be the best approximation of the mode.  AFAICS there is no good reason that the last sample should be the mode, unless you are performing simulated annealing and the temperature has fallen below the &quot;freezing point&quot; for the problem.  Likewise I don't see why only a small subset at the end of the run is important, the x maximising f(x) is the estimate of the mode regardless of where in the run it occurrs.&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2010-12-04T23:24:09.753" CommentCount="10" />
  <row Id="5149" PostTypeId="1" AcceptedAnswerId="5153" CreationDate="2010-12-05T02:33:01.707" Score="3" ViewCount="346" Body="&lt;p&gt;I have some time series data and want to test for the existence of and estimate the parameters of a linear trend in a dependent variable w.r.t. time, i.e. time is my independent variable.  The time points cannot be considered IID under the null of no trend.  Specifically, the error terms for points sampled near each other in time are positively correlated.  Error terms for samples obtained at sufficiently different times can be considered IID for all practical purposes.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I do not have a well-specified model of how the error terms are correlated for points close to each other in time.  All I know from domain knowledge is that they are positively correlated to some degree or another.  Other than this issue, I believe the assumptions of ordinarly least squares linear regression (homoskedasticity, linearity, normally distributed error terms) are met.  Modulo the correlated error term issue, OLS would solve my problem.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am a complete novice at dealing with time series data.  Is there any &quot;standard&quot; way to proceed in these circumstances?&lt;/p&gt;&#xA;" OwnerUserId="1347" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-05T12:49:08.100" LastActivityDate="2010-12-06T03:21:10.097" Title="Determining trend significance in a time series" Tags="&lt;time-series&gt;&lt;regression&gt;&lt;correlation&gt;" AnswerCount="4" FavoriteCount="2" />
  <row Id="5150" PostTypeId="1" CreationDate="2010-12-05T03:24:56.760" Score="4" ViewCount="225" Body="&lt;p&gt;I have a biometric authentication system that is using a person's gait to authenticate them. I extract features from gait, run it through a comparison versus a template and produce a similarity score (where if this similarity score is below a certain threshold, then the user is authenticated). So, I have 72 trials total (36 trials containing a positive case and 36 that contain a negative case). What I want to do is graph the ability of this system to authenticate people by illustrating it with a ROC graph. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unfortunately, I don't quite understand how to choose a threshold. Is there some mathematical procedure involved for choosing a threshold for the similarity scores? Do I just choose a bunch of different thresholds, and graph the corresponding ROC curves for all these different threshold values? The resulting similarity scores vary from [0.6,1.2] where the positive cases tend to lie around 0.6. All my coding is being done in Matlab. &lt;/p&gt;&#xA;" OwnerUserId="1224" LastActivityDate="2010-12-05T13:32:24.757" Title="Choosing the right threshold for a biometric trait authentication system" Tags="&lt;matlab&gt;&lt;mathematical-statistics&gt;&lt;roc&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="1" />
  <row Id="5151" PostTypeId="2" ParentId="125" CreationDate="2010-12-05T03:47:19.163" Score="0" ViewCount="" Body="&lt;p&gt;This book suggests it is aimed at entry level undergraduate level&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Biostatistics: A Bayesian Introduction.  By George G Woodsworth.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Published by John Wiley &amp;amp; Sons&lt;/p&gt;&#xA;" OwnerUserId="2030" LastActivityDate="2010-12-05T03:47:19.163" CommunityOwnedDate="2010-12-05T03:47:19.163" />
  <row Id="5152" PostTypeId="1" CreationDate="2010-12-05T04:46:48.430" Score="1" ViewCount="159" Body="&lt;p&gt;I am conducting a study on a cohort of people with a follow-up period of 7 years. I wish to use Cox Proportional Hazard model to estimate HR between an exposure and the length of time of an event. One missing information is the date of birth for the all subjects, but month and year are available.This prevents the calculation of exact age at the time of the study.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestions will be much appreciated? Any sensitivity analysis should be conducted?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks   &lt;/p&gt;&#xA;" OwnerUserId="2264" LastActivityDate="2010-12-07T02:59:36.707" Title="Missing values in survival analysis" Tags="&lt;survival&gt;&lt;missing-data&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="1" />
  <row Id="5153" PostTypeId="2" ParentId="5149" CreationDate="2010-12-05T05:13:40.530" Score="3" ViewCount="" Body="&lt;p&gt;What you are describing is commonly referred to as &lt;a href=&quot;http://en.wikipedia.org/wiki/Autocorrelation&quot; rel=&quot;nofollow&quot;&gt;auto correlated errors&lt;/a&gt;. I would suggest you look up resources on ARIMA modelling. ARIMA modelling will allow you to model the correlation in your error term, and hence allow you to assess your trend variable independent of this auto correlation (or other independent variables you are interested in).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My suggested reading for an into to ARIMA modelling would be&#xA;&lt;a href=&quot;http://books.google.com/books?id=D6-CAAAAIAAJ&amp;amp;q=Applied+Time+Series+Analysis+for+the+Social+Sciences&amp;amp;dq=Applied+Time+Series+Analysis+for+the+Social+Sciences&amp;amp;hl=en&amp;amp;ei=QaLNTKD-HcH88Aabt-Al&amp;amp;sa=X&amp;amp;oi=book_result&amp;amp;ct=result&amp;amp;resnum=1&amp;amp;ved=0CC8Q6AEwAA&quot; rel=&quot;nofollow&quot;&gt;Applied Time Series Analysis for the Social Sciences&lt;/a&gt; 1980&#xA;by R McCleary ; R A Hay ; E E Meidinger ; D McDowall&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But there are plenty of resources (time series analysis is a massive field of study). You would probably be able to turn up some good online resources with just a google search if you don't have access to an academic library. I just turned up this page, &lt;a href=&quot;http://www.statsoft.com/textbook/time-series-analysis/#arima&quot; rel=&quot;nofollow&quot;&gt;Statistica ARIMA&lt;/a&gt;, it has a brief but very concise description of ARIMA modelling as well as other methods for time series analysis.&lt;/p&gt;&#xA;" OwnerUserId="1036" LastActivityDate="2010-12-05T05:13:40.530" />
  <row Id="5155" PostTypeId="2" ParentId="5115" CreationDate="2010-12-05T05:18:37.237" Score="14" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/John_Tukey&quot; rel=&quot;nofollow&quot;&gt;John Tukey&lt;/a&gt; for Fast Fourier Transforms, exploratory data analysis, box plots, projection pursuit, jackknife. coined the words &quot;software&quot; and &quot;bit&quot;&lt;/p&gt;&#xA;" OwnerUserId="74" LastEditorUserId="74" LastEditorDisplayName="" LastEditDate="2011-06-01T17:32:23.790" LastActivityDate="2011-06-01T17:32:23.790" CommunityOwnedDate="2010-12-05T05:18:37.237" />
  <row Id="5156" PostTypeId="2" ParentId="5092" CreationDate="2010-12-05T05:29:13.617" Score="2" ViewCount="" Body="&lt;p&gt;I think that power analysis is too elaborate for what you're trying to do, and might let your down.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;With a sample size north of 9 million, I think your estimate for &lt;code&gt;p = Pr(X &amp;gt; 3) = 0.000015&lt;/code&gt; is pretty accurate.  So you can use that in a simple binomial(n, p) model to estimate a sample size.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say your goal is to observed at least one &quot;Large&quot; event with a probability of 99.9%.  Then &lt;code&gt;Pr(L &amp;gt; 0) = 1 - Pr(L = 0) = 1 - 0.999985^n = 0.999&lt;/code&gt; and your desired sample size is &lt;code&gt;n = ln(0.001)/ln(0.999985) = 460514&lt;/code&gt;.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, if you're feeling lucky and are willing to take a 10% chance of missing a Large event, you only need a sample size of n = 153505.  Tripling the sample size cuts your chance of missing the Large event by a factor of 100, so I'd go for the 460,000.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;BUT...if you're looking for FIVE's, their probability is just south of 1/9180902 and to observe at least one of THOSE with 99.9% probability, you'd need a sample size of about 63.4 million!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do heed DrKNexus' advice about updating your estimate of the probabilities for the Large events, since it might not be constant across all your datasets.&lt;/p&gt;&#xA;" OwnerUserId="1920" LastActivityDate="2010-12-05T05:29:13.617" CommentCount="1" />
  <row Id="5157" PostTypeId="2" ParentId="5149" CreationDate="2010-12-05T08:02:27.610" Score="1" ViewCount="" Body="&lt;p&gt;Along the lines of a previous answer, if all assumptions for OLS are met except for the fact that errors are correlated, maybe something as simple as a &lt;a href=&quot;http://en.wikipedia.org/wiki/Cochrane%E2%80%93Orcutt_estimation&quot; rel=&quot;nofollow&quot;&gt;Cochrane-Orcutt&lt;/a&gt; correction would be enough to solve your problem. &lt;/p&gt;&#xA;" OwnerUserId="892" LastActivityDate="2010-12-05T08:02:27.610" />
  <row Id="5158" PostTypeId="1" AcceptedAnswerId="5164" CreationDate="2010-12-05T09:40:51.387" Score="4" ViewCount="227" Body="&lt;p&gt;Hi,&#xA; I understand that when sampling from a finite population and our sample size is more than 5% of the population, we need to a correction on the sample's mean and standard error using this formula:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\hspace{10mm} FPC=\sqrt{\frac{N-n}{N-1}}$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where N is the population size and n is the sample size.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have 3 questions about this formula:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Why is the threshold set at 5%?&lt;/li&gt;&#xA;&lt;li&gt;How was the formula derived?&lt;/li&gt;&#xA;&lt;li&gt;Are there other online resources that comprehensively explain this formula besides &lt;a href=&quot;http://www.jstor.org/pss/2340569&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; paper?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Thank you,&lt;/p&gt;&#xA;" OwnerUserId="1636" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-05T12:43:49.383" LastActivityDate="2010-12-05T13:19:33.057" Title="Explanation of finite correction factor" Tags="&lt;sampling&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="1" />
  <row Id="5159" PostTypeId="1" AcceptedAnswerId="5169" CreationDate="2010-12-05T11:13:52.800" Score="2" ViewCount="187" Body="&lt;p&gt;Hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The height for 1000 students is approximately normal with a mean 174.5cm and a standard deviation of 6.9cm. If 200 random samples of size 25 are chosen from this population and the values of the mean are recorded to the nearest integer, determine the probability that the mean height for the students is more than 176cm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since the samples were rounded to the nearest integer, I should find $P(X&amp;gt;176.5)$ instead of $P(X&amp;gt;176)$. Is this how we account for the effect of rounding the observations?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: In light of whuber's answer:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The answer given by my module (no workings were provided):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\hspace{1cm} n=25; Normal$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\hspace{1cm} \mu_{\overline{x}}=174.5cm$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\hspace{1cm} \sigma_{\overline{x}}=6.9/5=1.38$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The answer is 0.1379. Which I'm pretty sure was found using $1-\phi(\dfrac{176-174.5}{1.38})$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So,&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Is this an acceptable answer?&lt;/li&gt;&#xA;&lt;li&gt;Since $n$ was less than 30, would it be ok to find the probability using a t-distribution?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Thank you.&lt;/p&gt;&#xA;" OwnerUserId="1636" LastEditorUserId="1636" LastEditorDisplayName="" LastEditDate="2010-12-07T19:18:04.460" LastActivityDate="2010-12-07T19:18:04.460" Title="Correction due to rounding error" Tags="&lt;homework&gt;" AnswerCount="2" />
  <row Id="5160" PostTypeId="1" CreationDate="2010-12-05T11:14:12.273" Score="4" ViewCount="627" Body="&lt;p&gt;Hello Members:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am looking for a good tutorial on clustering data in &lt;code&gt;R&lt;/code&gt; using hierarchical dirichlet process (HDP) (one of the recent and popular nonparametric Bayesian methods). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is &lt;code&gt;DPpackage&lt;/code&gt; (IMHO, the most comprehensive of all the available ones) in &lt;code&gt;R&lt;/code&gt; for nonparametric Bayesian analysis. But I am unable to understand the examples provided in &lt;code&gt;R News&lt;/code&gt; or in the package reference manual well enough to code HDP.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help or pointer is appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A C++ implementation of HDP for topic modeling is available &lt;a href=&quot;http://www.cs.princeton.edu/~blei/topicmodeling.html&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; (please look at the bottom for C++ code)&lt;/p&gt;&#xA;" OwnerUserId="1307" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-06T08:52:21.543" LastActivityDate="2010-12-20T14:22:03.660" Title="Nonparametric Bayesian analysis in R" Tags="&lt;r&gt;&lt;bayesian&gt;&lt;clustering&gt;&lt;nonparametric&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="5161" PostTypeId="2" ParentId="5159" CreationDate="2010-12-05T11:26:49.977" Score="3" ViewCount="" Body="&lt;p&gt;I understand the question as one where we know the theoretical distribution of students height with some precision (i.e., with one decimal place). In the present case, this is a gaussian distribution with parameters $\mathcal{N}(174.5;6.9^2)$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, we have empirical measurements of students height on small samples ($n=25$), but results are rounded to the nearest integer due to possible measurement error or imperfect measurement device.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, my understanding is that the question is really to assess $\Pr(X&amp;gt;176)$ or $\Pr(Z&amp;gt;\frac{176-174.5}{6.9})$ if you refer to the standardized $\mathcal{N}(0;1)$ distribution, and not $\Pr(X&amp;gt;176.5)$ as you suggested.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-12-05T11:51:26.913" LastActivityDate="2010-12-05T11:51:26.913" CommentCount="1" />
  <row Id="5162" PostTypeId="2" ParentId="5150" CreationDate="2010-12-05T11:46:24.613" Score="4" ViewCount="" Body="&lt;p&gt;Generally, the cut-off value is chosen such as to maximize the compromise between sensitivity (Se) and specificity (Sp). You can generate a regular sequence of thresholds and plot the resulting ROC curve, as shown below, based on the &lt;a href=&quot;http://cran.r-project.org/web/packages/DiagnosisMed/index.html&quot; rel=&quot;nofollow&quot;&gt;DiagnosisMed&lt;/a&gt; R package.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/yM82E.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Actually, the raw data looks like&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  test.values TP FN FP TN Sensitivity Specificity&#xA;1       0.037 51  0 97  0           1      0.0000&#xA;2       0.038 51  0 96  1           1      0.0103&#xA;3       0.039 51  0 91  6           1      0.0619&#xA;4       0.040 51  0 84 13           1      0.1340&#xA;5       0.041 51  0 74 23           1      0.2371&#xA;6       0.042 51  0 67 30           1      0.3093&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and the optimal threshold is found as&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   test.values TP FN FP TN Sensitivity Specificity&#xA;47       0.194 43  8  8 89      0.8431      0.9175&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To sum up, I would suggest to generate a regular sequence of possible thresholds and compute Se and Sp in each case; then, choose the one that maximize Se and (1-Sp) (or use other criteria if you want to minimize FP or FN rates).&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-12-05T13:32:24.757" LastActivityDate="2010-12-05T13:32:24.757" CommentCount="4" />
  <row Id="5163" PostTypeId="2" ParentId="5136" CreationDate="2010-12-05T12:21:37.620" Score="1" ViewCount="" Body="&lt;p&gt;The relevant section of the classical typology distinguishes between (observed) variables, latent variables, and parameters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regular variables are observed and have a distribution. Latent variables are not observed and have a distribution.  Parameters are not observed and do not have a distribution.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Parameters vs latent variables is indeed a modelling decision.  Consider a set of survey questions that tap an underlying scale.  If you expect that learning about one subject's position on the scale is potentially informative about another subject's position and you wish to be able to generalise to new subjects then you should treat position as a latent variable.  If not, you may as well treat it like a parameter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bringing up FA and IRT is a bit confusing because some measurement models aim to estimate subject parameters e.g. Rasch models, and some aim to estimate subject latent variables e.g. FA and IRT models.  All types of model have parameters in addition, associated with the items.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a survey context there are also indexes, constructed by combining several indicators (which are observed variables).  You should probably think of these as non-parametric &lt;em&gt;estimators&lt;/em&gt; of latent variables, for when you don't feel happy with measurement model parametric assumptions.  (Although personally I've never been particularly sure about their status)&lt;/p&gt;&#xA;" OwnerUserId="1739" LastActivityDate="2010-12-05T12:21:37.620" CommentCount="3" />
  <row Id="5164" PostTypeId="2" ParentId="5158" CreationDate="2010-12-05T12:32:46.047" Score="3" ViewCount="" Body="&lt;p&gt;The threshold is chosen such that it ensures convergence of the &lt;a href=&quot;http://en.wikipedia.org/wiki/Hypergeometric_distribution&quot; rel=&quot;nofollow&quot;&gt;hypergeometric distribution&lt;/a&gt; ($\sqrt{\frac{N-n}{N-1}}$ is its SD), instead of a binomial distribution (for sampling with replacement), to a normal distribution (this is the Central Limit Theorem, see e.g., &lt;a href=&quot;http://stat-www.berkeley.edu/~stark/SticiGui/Text/clt.htm&quot; rel=&quot;nofollow&quot;&gt;The Normal Curve, the Central Limit Theorem, and Markov's and Chebychev's Inequalities for Random Variables&lt;/a&gt;). In other words, when $n/N\leq 0.05$ (i.e., $n$ is not 'too large' compared to $N$), the FPC can safely be ignored; it is easy to see how the correction factor evolves with varying $n$ for a fixed $N$: with $N=10,000$, we have $\text{FPC}=.9995$ when $n=10$ while $\text{FPC}=.3162$ when $n=9,000$. When $N\to\infty$, the FPC approaches 1 and we are close to the situation of sampling with replacement (i.e., like with an infinite population).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To understand this results, a good starting point is to read some online tutorials on sampling theory where sampling is done without replacement (&lt;a href=&quot;http://en.wikipedia.org/wiki/Simple_random_sample&quot; rel=&quot;nofollow&quot;&gt;simple random sampling&lt;/a&gt;). This online tutorial on &lt;a href=&quot;http://www.stat.washington.edu/fritz/DATAFILES425_2009/Stat425Ranksum.pdf&quot; rel=&quot;nofollow&quot;&gt;Nonparametric statistics&lt;/a&gt; has an illustration on computing the expectation and variance for a total. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You will notice that some authors use $N$ instead of $N-1$ in the denominator of the FPC; in fact, it depends on whether you work with the sample or population statistic: for the variance, it will be $N$ instead of $N-1$ if you are interested in $S^2$ rather than $\sigma^2$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for online references, I can suggest you&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.idrc.ca/esaro/ev-104013-201-1-DO_TOPIC.html&quot; rel=&quot;nofollow&quot;&gt;Estimation and statistical inference&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://wmbriggs.com/public/HGDAmstat4.pdf&quot; rel=&quot;nofollow&quot;&gt;A new look at inference for the Hypergeometric Distribution&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.stat.washington.edu/fritz/DATAFILES/hypergeo.pdf&quot; rel=&quot;nofollow&quot;&gt;Finite Population Sampling with Application to the Hypergeometric Distribution&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://statmaster.sdu.dk/courses/st118/module02/index.html&quot; rel=&quot;nofollow&quot;&gt;Simple random sampling&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-12-05T13:19:33.057" LastActivityDate="2010-12-05T13:19:33.057" />
  <row Id="5165" PostTypeId="2" ParentId="5149" CreationDate="2010-12-05T13:22:35.273" Score="1" ViewCount="" Body="&lt;p&gt;Generalised least squares (GLS) is one potential option here. The OLS estimates of the parameters are given by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\hat{\beta} = (X^{T}\Sigma^{-1}X)^{-1}X^{T}\Sigma^{-1}y$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Normally we leave out $\Sigma$ as in OLS it is defined as $\sigma^2 \mathbf{I}$, i.e. an identity matrix multiplied by the estimated residual standard error. $\mathbf{I}$ is the assumption of uncorrelated errors; an observation is perfectly correlated with itself and is uncorrelated with any other observation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;GLS relaxes this indepence assumption by allowing $\Sigma$ to take different forms. Usually we choose a simple process to parametrise $\Sigma$, such as an AR(1). In an AR(1) the correlation between two errors at times $t$ and $s$ is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\mathrm{cor}(\varepsilon_s \varepsilon_t) = \left\lbrace \begin{array}{ll}&#xD;&#xA;            1            &amp;amp; \mathrm{if} \; s = t \\&#xD;&#xA;            \rho^{|t-s|} &amp;amp; \mathrm{else}     \\&#xD;&#xA;        \end{array}&#xD;&#xA;        \right. $$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which would give us the following error covariance matrix:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\mathbf{\Sigma} = \sigma^2 \left( \begin{array}{ccccc}&#xD;&#xA;                     1          &amp;amp; \rho       &amp;amp; \rho^2     &amp;amp; \cdots &amp;amp; \rho^{n-1} \\&#xD;&#xA;                     \rho       &amp;amp; 1          &amp;amp; \rho       &amp;amp; \cdots &amp;amp; \rho^{n-2} \\&#xD;&#xA;                     \rho^2     &amp;amp; \rho       &amp;amp; 1          &amp;amp; \cdots &amp;amp; \rho^{n-3} \\&#xD;&#xA;                     \vdots     &amp;amp; \vdots     &amp;amp; \vdots     &amp;amp; \ddots &amp;amp; \vdots     \\&#xD;&#xA;                     \rho^{n-1} &amp;amp; \rho^{n-2} &amp;amp; \rho^{n-3} &amp;amp; \cdots &amp;amp; 1          \\&#xD;&#xA;                 \end{array} \right)$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An additional parameter estimate is required, $\rho$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More complex processes for $\Sigma$ can be employed, including ARMA models. In R, these sorts of model s can be fitted using the &lt;code&gt;gls()&lt;/code&gt; function in package &lt;strong&gt;nlme&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are an R user, you might also take a look at the &lt;strong&gt;sandwich&lt;/strong&gt; package which allows for something similar to the above, but where you estimate the OLS model and then afterwards, estimate $\Sigma$ and use that as a plug-in value to correct the standard errors of the OLS parameters.&lt;/p&gt;&#xA;" OwnerUserId="1390" LastActivityDate="2010-12-05T13:22:35.273" />
  <row Id="5166" PostTypeId="1" CreationDate="2010-12-05T16:27:41.193" Score="0" ViewCount="163" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;Possible Duplicate:&lt;/strong&gt;&lt;br&gt;&#xA;  &lt;a href=&quot;http://stats.stackexchange.com/questions/3324/why-use-a-lagged-dv-as-an-instrumental-variable&quot;&gt;Why use a lagged DV as an instrumental variable?&lt;/a&gt;  &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&#xA;&#xA;&lt;p&gt;In a simultaneous equations model, can lagged dependent variable be IVs. I understand sometimes lagged independent variables can be IVs. But under what conditions the lagged dependent variables would be IVs? Any journal articles about this? Thanks!&lt;/p&gt;&#xA;" LastActivityDate="2010-12-05T16:27:41.193" ClosedDate="2010-12-05T19:25:38.410" Title="Can lagged dependent variables be good IVs?" Tags="&lt;instrumental-variables&gt;" CommentCount="1" />
  <row Id="5167" PostTypeId="1" CreationDate="2010-12-05T20:54:03.357" Score="1" ViewCount="190" Body="&lt;p&gt;I am sampling covariance matrix from a Inverse Wishart distribution. In one dimensional case, after doing sufficient iterations I am taking the mode value for variance (after removing the burn-in values). How to do the same in a multivariate case?&lt;/p&gt;&#xA;" OwnerUserId="2157" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-07T10:38:37.483" LastActivityDate="2010-12-07T10:38:37.483" Title="Sampling covariance matrix using Gibbs sampling" Tags="&lt;mcmc&gt;&lt;gibbs&gt;" AnswerCount="2" CommentCount="3" />
  <row Id="5168" PostTypeId="2" ParentId="5160" CreationDate="2010-12-05T20:54:42.357" Score="4" ViewCount="" Body="&lt;p&gt;Here are some online ressources I found interesting without going into detail (and I'm not a specialist of this topic):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.cse.buffalo.edu/faculty/mbeal/papers/hdp.pdf&quot; rel=&quot;nofollow&quot;&gt;Hierarchical Dirichlet Processes&lt;/a&gt;, by Teh et al. (2005)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.cs.cmu.edu/~kbe/dp_tutorial.pdf&quot; rel=&quot;nofollow&quot;&gt;Dirichlet Processes A gentle tutorial&lt;/a&gt;, by El-Arini (2008)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.mit.edu/~9.520/Classes/class21_dirichlet_2010.pdf&quot; rel=&quot;nofollow&quot;&gt;Bayesian Nonparametrics&lt;/a&gt;, by Rosasco (2010)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://learning.eng.cam.ac.uk/zoubin/talks/uai05tutorial-b.pdf&quot; rel=&quot;nofollow&quot;&gt;Non-parametric Bayesian Methods&lt;/a&gt;, by Ghahramani (2005)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The definitive reference seems to be&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;N. Hjort, C. Holmes, P. Müller, and S.&#xA;  Walker, editors. &lt;em&gt;&lt;a href=&quot;http://www.cambridge.org/gb/knowledge/isbn/item2707877/?site_locale=en_GB&quot; rel=&quot;nofollow&quot;&gt;Bayesian&#xA;  Nonparametrics&lt;/a&gt;&lt;/em&gt;. Number 28 in&#xA;  Cambridge Series in Statistical and&#xA;  Probabilistic Mathematics. Cambridge&#xA;  University Press, 2010.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;About R, there seems to be some other packages worth to explore if the &lt;a href=&quot;http://cran.r-project.org/web/packages/DPpackage/index.html&quot; rel=&quot;nofollow&quot;&gt;DPpackage&lt;/a&gt; does not suit your needs, e.g. &lt;a href=&quot;http://cran.r-project.org/web/packages/dpmixsim/&quot; rel=&quot;nofollow&quot;&gt;dpmixsim&lt;/a&gt;, &lt;a href=&quot;http://www.bioconductor.org/help/bioc-views/release/bioc/html/BHC.html&quot; rel=&quot;nofollow&quot;&gt;BHC&lt;/a&gt;, or &lt;a href=&quot;http://www.stat.washington.edu/hoff/Code/MBSC/&quot; rel=&quot;nofollow&quot;&gt;mbsc&lt;/a&gt; found on &lt;a href=&quot;http://www.rseek.org&quot; rel=&quot;nofollow&quot;&gt;Rseek.org&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-12-05T20:54:42.357" CommunityOwnedDate="2010-12-05T21:06:08.147" CommentCount="3" />
  <row Id="5169" PostTypeId="2" ParentId="5159" CreationDate="2010-12-05T21:40:11.867" Score="5" ViewCount="" Body="&lt;p&gt;I interpret this question as supposing that an experiment is conducted 200 times.  In this experiment, 25 people are independently drawn from the population (with replacement) and their average height is rounded to the nearest centimeter.  This process yields 200 whole numbers.  You seem to be asking, what is the chance that the average of these 200 numbers exceeds 176 cm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This interpretation requires us to deal with several phenomena: the sampling distribution of the mean, the effects of rounding, and the effect of repeated sampling.  Other interpretations are possible but it appears these issues will arise no matter what, so I hope the following analysis will illustrate useful, relevant techniques even if a different interpretation is intended.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;The &lt;strong&gt;sampling distribution of the mean&lt;/strong&gt; of 25 independent values (with replacement) has the same mean as the parent distribution and 1/25th of its variance.  It's also Normal.  Thus in this case it's a Normal(174.5, 6.9/5) distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Rounding&lt;/strong&gt; turns a continuous distribution (Normal(174.5, 6.9/5) in this case) into a &lt;em&gt;discrete&lt;/em&gt; distribution, because now the only possible values are 0, 1, ..., 174, 175, 176, ..., cm.  The probability of observing one of these values $y$ equals the probability that the true value lies between $y - 1/2$ and $y + 1/2$ and therefore is given by&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\Pr(Y) = \Phi(\frac{y + 1/2 - 174.5}{6.9/5}) - \Phi(\frac{y - 1/2 - 174.5}{6.9/5}).$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where, as usual, $\Phi$ is the cumulative distribution function for the standard Normal distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because normal distributions are symmetric, bias in rounding of values less than the mean ought to balance any bias in rounding of values greater than the mean.  The balancing will be perfect when the distribution's mode is a half integer, which is the case here.  Thus the mean of this &quot;discrete Normal&quot; distribution is &lt;em&gt;exactly&lt;/em&gt; 174.5.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The rounding will increase the variance&lt;/strong&gt;.  &lt;em&gt;As an approximation,&lt;/em&gt; people usually think of rounding as acting &lt;em&gt;at random&lt;/em&gt; to vary a number by some amount uniformly distributed between $-1/2$ and $+1/2$.  The variance of this uniform distribution is $1/12$, whence we can &lt;em&gt;estimate&lt;/em&gt; the variance of the discrete normal distribution as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\sqrt{sd^2 + 1/2} = \sqrt{(6.9/5)^2 + 1/12} = 1.40986\ 99703\ 63697\ 52354,$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;approximately.  This approximation works well when the rounding is small compared to the standard deviation of the true distribution, which is the case here.  In fact, exact calculations give a value of $1.40986\ 99703\ 63697\ 65285$, differing from the approximation by about $10^{-16}$.  That's more than accurate enough! -- But it was worthwhile checking.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now that we know the parameters of the distribution of rounded averages of 25-person samples--namely, a mean of 174.5 and SD of 1.40986--we determine as before that the expectation of &lt;strong&gt;the mean of 200 such rounded averages&lt;/strong&gt; is 174.5 and its SD is $1.40986/\sqrt{200}$ = $0.099693$.  Now this distribution is going to be extremely close to Normal, but not perfectly so: after all, its values must all be multiples of 1/200 = 0.005 cm.  If you like, you can introduce a &lt;strong&gt;continuity correction&lt;/strong&gt; in the usual way by observing that there can be no average between 176 and 176.005 cm, so you would compute the probability of a &lt;em&gt;Normal&lt;/em&gt; variate exceeding their midpoint of 176.0025 cm.  However, this makes no practical difference, because 176 - 174.5 cm is more than 15 standard deviations above the mean: it's virtually impossible that the average of 200 rounded values could exceed either 176.0025 or 176.  The exact value is approximately &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$1 - \Phi(\frac{176.0025 - 174.5}{0.099693})$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;which less than $10^{-53}$.  Because the original population is only &quot;approximately&quot; Normal, we shouldn't rely on any probability calculation this small.  We might just as well say &lt;strong&gt;the answer is &quot;essentially zero.&quot;&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-12-05T21:40:11.867" CommentCount="2" />
  <row Id="5170" PostTypeId="1" AcceptedAnswerId="5176" CreationDate="2010-12-05T23:12:33.357" Score="0" ViewCount="304" Body="&lt;p&gt;I am new to forecasting in R and am trying to automatically fit an ARIMA model to what I believe is a univariate dataset.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; str(p1.z)&#xA;'zoo' series from 2009-04-05 to 2010-10-31&#xA;  Data: int [1:83] 360 570 540 585 570 690 495 660 510 690 ...&#xA;  Index: Class 'Date'  num [1:83] 14339 14346 14353 14360 14367 ...&#xA;&#xA;&amp;gt;  head(p1.z) &#xA;  2009-04-05 2009-04-12 2009-04-19 2009-04-26 2009-05-03 2009-05-10 &#xA;         360        570        540        585        570        690&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But when I try to fit the model, I get the error as seen below.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; p1.arima &amp;lt;- auto.arima(p1.z)&#xA;Error in nsdiffs(xx) : Non seasonal data&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It is my understanding that the forecast package and the auto.arima function would be able to fit my data seasonal or not.  I am trying to learn time series forecasting and am using a dataset that appears to be ideal for this sort of task .  Also, the function ets() was able to find a model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help you can provide will be greatly appreciated&lt;/p&gt;&#xA;" OwnerUserId="569" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-12-06T10:26:02.110" LastActivityDate="2010-12-06T10:26:02.110" Title="Starting out with forecast package in R" Tags="&lt;r&gt;&lt;time-series&gt;&lt;forecasting&gt;" AnswerCount="1" />
  <row Id="5171" PostTypeId="1" AcceptedAnswerId="5258" CreationDate="2010-12-05T23:43:10.043" Score="5" ViewCount="261" Body="&lt;p&gt;I hope this isn't either far too basic or redundant. I have been looking around for guidance but so far I am still uncertain of how to proceed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My data consists of counts of a particular structure used in conversations between pairs of interlocutors. The hypothesis I want to test is the following: more frequent use of this structure by one speaker will tend to increase the frequency of the structure by the other speaker (i.e., this might be evidence of a priming effect).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I just have two vectors, the counts for speaker A and the counts for speaker B are the columns, and if they are lined up each row represents a particular conversation, like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A  B&lt;br&gt;&#xA;0  1&lt;br&gt;&#xA;0  2&lt;br&gt;&#xA;1  0&lt;br&gt;&#xA;3  1&lt;br&gt;&#xA;0  2&lt;br&gt;&#xA;2  0&lt;br&gt;&#xA;2  1&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are about 420 conversations (rows). There are lots of zeros in this data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What would be the best way to analyze this data? I am using R, if that makes a difference.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a plot of the frequencies (counts). The x-axis is number of uses by speaker A, the y-axis number of uses by speaker B. The distinction between speakers means only that speaker A spoke first, and there's no special reason why they did. Otherwise the distinction between speaker A and speaker B is basically meaningless:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://phonematic.com/convplot.jpg&quot; alt=&quot;Valid XHTML&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And this is frequency relative to number of sentences spoken by each speaker in each conversation. :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://phonematic.com/rs_plot.jpg&quot; alt=&quot;Valid XHTML&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(I should mention that I have thrown out conversations with no hits at all, i.e. {0,0}.)&lt;/p&gt;&#xA;" OwnerUserId="52" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-10T20:38:31.137" LastActivityDate="2011-05-10T20:38:31.137" Title="Testing paired frequencies for independence" Tags="&lt;categorical-data&gt;&lt;independence&gt;" AnswerCount="3" CommentCount="5" FavoriteCount="3" />
  <row Id="5172" PostTypeId="1" CreationDate="2010-12-06T00:17:18.907" Score="3" ViewCount="197" Body="&lt;p&gt;I have several sets of data, unfortunately the data comes to me in a &quot;summary&quot; form. My job is to consolidate the several data sources into one general summary. I'm currently using the median to summarise the data, but I don't know if this is statistically sound. Here's a description of my problem:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are $N_P$ samples, each with varying sample sizes, but all from a single population. Neither the sample size or the standard variation are known. Each sample can be divided into $N_Q$ disjoint groups (or qualities). From each sample, the only data that is known is what percent of the sample falls within a group (or category). For example, population $A$ contains, $x\%$ of $a$, $y\%$ of $b$ and $z\%$ of $c$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The different samples are not disjoint, so a single item might be in several of the samples; but I don't know how much overlapping there is. There are 5-8 different samples with 5-7 categories. An example (smaller) table is the following.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;            cat. a    cat. b    cat. c    &#xA;sample A    47.34%    30.05%    11.92%&#xA;sample B    41.60%    29.90%    11.90%&#xA;sample c    47.74%    29.67%    12.69%&#xA;--------    ------    ------    ------&#xA;median      47.34%    29.90%    11.92%&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now is it statistically sound to create this &quot;median&quot; summary, which takes each group from the different samples and finds the median? Maybe I should be using the mean? The problem I'm seeing is the &quot;median sample&quot; usually sums to less than 100%, even though the percentages from each sample sum to 100%. Should this matter?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Sample sizes: 100k - 100m&#xA;Population size: ~1 billion&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2271" LastEditorUserId="2271" LastEditorDisplayName="" LastEditDate="2010-12-06T05:02:30.237" LastActivityDate="2010-12-06T05:02:30.237" Title="Is taking the median of a set of percentages statistically sound?" Tags="&lt;sampling&gt;&lt;median&gt;&lt;population&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="1" />
  <row Id="5173" PostTypeId="1" AcceptedAnswerId="5175" CreationDate="2010-12-06T00:33:15.987" Score="5" ViewCount="218" Body="&lt;p&gt;&quot;Spurious regression&quot; (in the context of time series) and associated terms like unit root tests are something I've heard a lot about, but never understood.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why/when, intuitively, does it occur? (I believe it's when your two time series are cointegrated, i.e., some linear combination of the two is stationary, but I don't see why cointegration should lead to spuriousness.) What do you do to avoid it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm looking for a high-level understanding of what cointegration/unit root tests/Granger causality have to do with Spurious regression (those three are terms I remember being associated with spurious regression somehow, but I don't remember what exactly), so either a custom response or a link to references where I can learn more would be great.&lt;/p&gt;&#xA;" OwnerUserId="1106" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-12-06T09:08:35.157" LastActivityDate="2010-12-06T15:36:20.017" Title="Resources for learning about spurious time series regression" Tags="&lt;time-series&gt;&lt;regression&gt;&lt;cointegration&gt;" AnswerCount="2" FavoriteCount="2" />
  <row Id="5174" PostTypeId="2" ParentId="5173" CreationDate="2010-12-06T00:53:50.953" Score="7" ViewCount="" Body="&lt;p&gt;Let's start with the spurious regression.  Take or imagine two series which are both driven by a dominant time trend: for example US population and US consumption of &lt;em&gt;whatever&lt;/em&gt; (it doesn't matter what item you think about, be it soda or licorice or gas).  Both series will be &lt;em&gt;growing&lt;/em&gt; because of the common time trend. Now regress aggregate consumption on aggregate population size and presto, you have a great fit. (We could simulate that quickly in R too.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But it means nothing. There is no relationship (as we as the modelers know) -- yet the linear model sees a &lt;em&gt;fit&lt;/em&gt; (in the minimizing sum of squares sense) as &lt;em&gt;both series happen to both be uptrending&lt;/em&gt; without a causal link. We fell victim to a spurious regression.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What could or should be modeled is change in one series on change in the other, or maybe per capita consumption, or ...   All those changes make the variables stationary which helps to alleviate the issue.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, from 30,000 feet, unit roots and cointegration help you with formal inference in these case by providing rigorous statistical underpinning &#xA;(&lt;em&gt;Econometrica&lt;/em&gt; publications and a Nobel don't come easily) where none was available. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for question in good resources: it's tricky. I have read dozens of time series books, and most excel at the math and leave the intuition behind.  There is nothing like Kennedy's &lt;em&gt;Econometrics&lt;/em&gt; text for time series.  Maybe Walter Enders text comes closest. I will try to think of some more and update here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other than books, software for actually &lt;em&gt;doing&lt;/em&gt; this is important and R has what you need. The price is right too.&lt;/p&gt;&#xA;" OwnerUserId="334" LastEditorUserId="334" LastEditorDisplayName="" LastEditDate="2010-12-06T13:26:36.717" LastActivityDate="2010-12-06T13:26:36.717" />
  <row Id="5175" PostTypeId="2" ParentId="5173" CreationDate="2010-12-06T01:56:44.033" Score="8" ViewCount="" Body="&lt;p&gt;These concepts have been created to deal with &lt;strong&gt;regressions (for instance correlation) between non stationary series.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Clive Granger is the key author you should read.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cointegration has been introduced in 2 steps:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;1/ Granger, C., and P. Newbold (1974): &quot;Spurious Regression in Econometrics,&quot;&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this article, the authors point out that regression among non stationary variables should be conducted as regressions among &lt;strong&gt;changes&lt;/strong&gt; (or log changes) of the variables. Otherwise you might find high correlation without any real significance. (= spurious regression)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;2/ Engle, Robert F., Granger, Clive W. J. (1987) &quot;Co-integration and error correction: Representation, estimation and testing&quot;, Econometrica, 55(2), 251-276.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this article (for which Granger has been rewarded by the Nobel jury in 2003), the authors go further, and introduce cointegration as a way to study the error correction model that can exist between two non stationary variables.&lt;br&gt;&#xA;Basically the 1974 advice to regress the change in the time series may lead to unspecified regression models. You can indeed have variables whose changes are uncorrelated, but which are connected through an &quot;error correction model&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hence, you can have correlation without cointegration, and cointegration without correlation. The two are complementary.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If there was only one paper to read, I suggest you start with this one, which is a very good and nice introduction:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www-stat.wharton.upenn.edu/~steele/Courses/434/434Context/Co-integration/Murray93DrunkAndDog.pdf&quot; rel=&quot;nofollow&quot;&gt;(Murray 1993) Drunk and her dog&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1709" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-12-06T15:36:20.017" LastActivityDate="2010-12-06T15:36:20.017" />
  <row Id="5176" PostTypeId="2" ParentId="5170" CreationDate="2010-12-06T03:12:22.033" Score="3" ViewCount="" Body="&lt;p&gt;&lt;code&gt;ets()&lt;/code&gt; and &lt;code&gt;auto.arima()&lt;/code&gt; are not really set up to handle &lt;code&gt;zoo&lt;/code&gt; objects. Although &lt;code&gt;ets()&lt;/code&gt; is not returning an error, it will be ignoring any seasonality. &lt;code&gt;auto.arima()&lt;/code&gt; is failing because it is confused by the &lt;code&gt;zoo&lt;/code&gt; object with apparent seasonality. I will try to include better checking in a future version.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When using the forecast package, use &lt;code&gt;ts&lt;/code&gt; objects instead. In this example, &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x &amp;lt;- ts(x)&#xA;auto.arima(x)&#xA;ets(x)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;That will ignore the frequency component of &lt;code&gt;x&lt;/code&gt;. It looks like weekly data, so&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x &amp;lt;- ts(x,start=2009+(31+28+31+4)/365,f=52)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;will capture the frequency (and start period). However, note that &lt;code&gt;ets()&lt;/code&gt; will not handle weekly data and will return an error with this latter formulation.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-12-06T03:12:22.033" CommentCount="4" />
  <row Id="5177" PostTypeId="2" ParentId="5149" CreationDate="2010-12-06T03:21:10.097" Score="4" ViewCount="" Body="&lt;p&gt;To add to the existing answers, if you are using R a simple way to proceed is to allow the ARMA errors to be modelled automatically using &lt;code&gt;auto.arima()&lt;/code&gt;. If &lt;code&gt;x&lt;/code&gt; is your time series, then you can proceed as follows.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;t &amp;lt;- 1:length(x)&#xA;auto.arima(x,xreg=t,d=0)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This will fit the model $x_t = a + bt + e_t$ where $e_t\sim\text{ARMA}(p,q)$ and $p$ and $q$ are selected automatically using the AIC.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The resulting output will give the value of $b$ and its standard error. Here is an example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Series: x &#xA;ARIMA(3,0,0) with non-zero mean &#xA;&#xA;Call: auto.arima(x = x, xreg = t) &#xA;&#xA;Coefficients:&#xA;          ar1     ar2      ar3  intercept       t&#xA;      -0.3770  0.1454  -0.2351   563.9654  0.0376&#xA;s.e.   0.1107  0.1190   0.1145    11.4725  0.2378&#xA;&#xA;sigma^2 estimated as 5541:  log likelihood = -475.85&#xA;AIC = 963.7   AICc = 964.81   BIC = 978.21&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In this case, $p=3$ and $q=0$. The first three coefficients give the autoregressive terms, $a$ is the intercept and $b$ is in the &lt;code&gt;t&lt;/code&gt; column. In this (artificial) example, the slope is not significantly different from zero.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;code&gt;auto.arima&lt;/code&gt; function is using MLE rather than GLS, but the two are asymptotically equivalent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The use of a Cochrane-Orcutt procedure only works if the error is AR(1). So the above is much more general and flexible.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-12-06T03:21:10.097" CommentCount="2" />
  <row Id="5178" PostTypeId="2" ParentId="5172" CreationDate="2010-12-06T04:15:24.173" Score="1" ViewCount="" Body="&lt;p&gt;What you are doing does not makes sense if your goal is to categorize what proportion of the entire population (sample A + sample B + sample C) is in category a, b, and c. Consider the following contingency table:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   a  b  c             a    b    c&#xA;A  8; 1; 1         A  .8;  .1;  .1&#xA;B  7; 2; 1         B  .7;  .2;  .1&#xA;C  1; 13; 16       C  .03; .43; .53&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then, for example, the median of the category a probabilities is 0.7 and the mean is 0.51, but only 16/50 = 0.32 of the all the observations are in column a. Likewise, the median of the category c probabilities would be 0.1, but only 0.36 of the observations are in column c. Does the &quot;median summary&quot; you propose tell you anything meaningful in a situation such as this one? Unless you have the marginal counts of either the samples or the categories, or you are willing to make some assumptions about them, I don't think there is a whole lot you can do in this case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you have any specific goals in mind? Also, how many categories and samples do you have?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: Your sample/population phrasing is slightly confusing. It's better to say you &quot;have 3 samples, each which be sub-divided into 3 categories a,b, and c.&quot; The phrase &quot;sample population&quot; is troublesome, as is your reference to two different &quot;populations.&quot; &lt;/p&gt;&#xA;" OwnerUserId="2144" LastActivityDate="2010-12-06T04:15:24.173" CommentCount="4" />
  <row Id="5179" PostTypeId="2" ParentId="5171" CreationDate="2010-12-06T05:01:20.237" Score="0" ViewCount="" Body="&lt;p&gt;I would maybe start with a &lt;a href=&quot;http://en.wikipedia.org/wiki/Rank_correlation&quot; rel=&quot;nofollow&quot;&gt;rank correlation&lt;/a&gt; analysis.&lt;br&gt;&#xA;The issue is that you may have very low correlations as the effects you are trying to capture are small.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Both Kendall and Spearman correlation coefficients are implemented in R in&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cor(x=A, y=B, method = &quot;spearman&quot;)  &#xA;cor(x=A, y=B, method = &quot;kendall&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1709" LastActivityDate="2010-12-06T05:01:20.237" CommentCount="3" />
  <row Id="5180" PostTypeId="2" ParentId="5167" CreationDate="2010-12-06T06:10:49.867" Score="0" ViewCount="" Body="&lt;p&gt;Hello Saurabh:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I understand your question correctly:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Covariance matrix for 1-dim case reduces to the variance. Wishart Distribution (or Inv wishart distribution depending on your formulation) is a prior of covariance matrices, which for dimensions $\geq$ 2 correspond to multivariate case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I may have misunderstood you. Please correct me if that is the case. &lt;/p&gt;&#xA;" OwnerUserId="1307" LastActivityDate="2010-12-06T06:10:49.867" CommentCount="1" />
  <row Id="5181" PostTypeId="1" AcceptedAnswerId="5263" CreationDate="2010-12-06T10:30:50.827" Score="2" ViewCount="239" Body="&lt;p&gt;Say I'm doing stats on the height of adults from various countries.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I assume the heights of adults from one country are normally distributed, and ignore sex differences (I also ignore the fact that neighbouring countries tend to have similar populations).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a bunch of data by country, but for some countries I have very few data points, which can lead to quite big errors on my estimate of standard distributions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a way I can use the data from countries for which I have a lot of data to get better estimates - say if I notice standard deviation in those countries is always between 7 and 8.5 cm, but my dataset for Nepal (for which I have 9 samples) has a standard deviation of 9.5 cm, I should probably correct that downwards. But how? Is there a formula for this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When calculating the parameters for Nepal, shouldn't the data from the other countries allow me to have a &quot;prior&quot; distribution of expected means and deviations, which I would then update by taking the actual data from Nepal into account? How would I formalize this methodology?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(I got this while looking for a simple reduction of the problem that prompted &lt;a href=&quot;http://stats.stackexchange.com/questions/5107/mahalanobis-distance-to-a-multivariate-distribution-of-which-i-have-few-sample&quot;&gt;previous question&lt;/a&gt;, which didn't get an answer yet - I'm still mostly looking for good methodologies for thinking about this kind of problem).&lt;/p&gt;&#xA;" OwnerUserId="1737" LastActivityDate="2010-12-16T03:26:47.083" Title="Estimating distribution parameters from few data points" Tags="&lt;bayesian&gt;&lt;estimation&gt;&lt;normal-distribution&gt;&lt;uncertainty&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="5182" PostTypeId="2" ParentId="5171" CreationDate="2010-12-06T11:00:31.630" Score="2" ViewCount="" Body="&lt;p&gt;You seem to have ordered categorical data, therefore I suggest a linear-by-linear test as described by Agresti (2007, p229 ff). Function &lt;code&gt;lbl_test()&lt;/code&gt; of package &lt;code&gt;coin&lt;/code&gt; implements it in R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Agresti, A. (2007). Introduction to Categorical Data Analysis. 2nd Ed. Hoboken, New Jersey: John Wiley &amp;amp; Sons. Hoboken, NJ: Wiley.&lt;/p&gt;&#xA;" OwnerUserId="1909" LastActivityDate="2010-12-06T11:00:31.630" CommentCount="0" />
  <row Id="5183" PostTypeId="2" ParentId="5107" CreationDate="2010-12-06T11:43:57.957" Score="2" ViewCount="" Body="&lt;p&gt;If you have very little data, it is not that the distance estimate is wrong, but that your estimate is uncertain.  A Bayesian approach would seek to determine the posterior distribution of the distance between the arbitrary point and the multi-variate distribution, rather than a single point estimate, and then marginalise over that posterior in reaching your conclusion.  This posterior distribution reflects the uncertainty in estimating the mean and covariance matrix of the multivariate Gaussian distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would be wary of using an informative prior.  In a Bayesian analysis, the conclusions are only as strong as the prior assumtions on which they are based; if the prior is questionable - so is the posterior.  Without more information about the problem it is not possible to determine if such a prior is reasonable.&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2010-12-06T11:43:57.957" />
  <row Id="5184" PostTypeId="1" CreationDate="2010-12-06T14:04:10.083" Score="10" ViewCount="399" Body="&lt;p&gt;In a recent paper, I fitted a three-way fixed effects models. Since one of the factors wasn't significant (p &gt; 0.1), I removed it and refitted the model with two fixed effects and an interaction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've just had referees comments back, to quote:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;That time was not a significant factor&#xA;  in the 3-way ANOVA is not of itself a&#xA;  sufficient criterion for pooling the&#xA;  time factor: the standard text on this&#xA;  issue, Underwood 1997, argues that the&#xA;  p-value for a non-significant effect&#xA;  must be greater than 0.25 before&#xA;  treatment levels of a factor can be&#xA;  pooled. The authors should give the&#xA;  relevant p-value here, and justify&#xA;  their pooling with reference to&#xA;  Underwood 1997.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;My questions are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;I've never heard of the 0.25 rule. Has anyone else? I can understand not removing the factor if the p-value was close to the cut-off, but to have a &quot;rule&quot; seems a bit extreme.&lt;/li&gt;&#xA;&lt;li&gt;This referee states that &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0521556961&quot; rel=&quot;nofollow&quot;&gt;Underwood 1997&lt;/a&gt; is the standard text. Is it really? I've never heard of it. What would be the standard text (does such a thing exist)? Unfortunately, I don't have access to this Underwood, 1997.&lt;/li&gt;&#xA;&lt;li&gt;Any advice when responding to the referees.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Background: this paper was submitted to a non-statistical journal. When fitting the three-way model I checked for interaction effects.&lt;/p&gt;&#xA;" OwnerUserId="8" LastActivityDate="2010-12-06T14:57:16.090" Title="Removing factors from a 3-way ANOVA table" Tags="&lt;anova&gt;&lt;fixed-effects-model&gt;" AnswerCount="2" CommentCount="3" FavoriteCount="1" />
  <row Id="5185" PostTypeId="2" ParentId="5184" CreationDate="2010-12-06T14:53:21.063" Score="8" ViewCount="" Body="&lt;p&gt;I'm guessing the Underwood in question is &lt;em&gt;Experiments in Ecology&lt;/em&gt; (Cambridge Press 1991). Its a more-or-less standard reference in the ecological sciences, perhaps third behind Zar and Sohkol and Rohlf (and in my opinion the most 'readable' of the three)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you can find a copy, the relevant section your referee is citing is in 9.7 on p.273. There Underwood suggests a recommended pooling procedure (so not a 'rule' &lt;em&gt;per se&lt;/em&gt;) for non-significant factors. Its a 2-step procedure that frankly I don't quite understand, but the upshot is the p = 0.25 is suggested to reduce the probability of Type I error when pooling the non-significant factor (so nothing to do with 'time' in your example, it could be any non-sig factor). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The procedure doesn't actually appear to be Underwood's, he himself cites Winer et al 1991 (&lt;em&gt;Statistical Procedures in Experimental Design&lt;/em&gt; McGraw-Hill). You might try there if you can't find a copy of Underwood.&lt;/p&gt;&#xA;" OwnerUserId="1475" LastActivityDate="2010-12-06T14:53:21.063" CommentCount="2" />
  <row Id="5186" PostTypeId="2" ParentId="5184" CreationDate="2010-12-06T14:57:16.090" Score="6" ViewCount="" Body="&lt;p&gt;I loathe these sort of cut-off-based rules. I think it depends on design and what your &lt;em&gt;a priori&lt;/em&gt; hypotheses and expectations were. If you expecting the outcome to vary with time then I'd say you should keep time in, as you would for any other 'blocking' factor. On the other hand, if you were replicating the same experiments at different times and had no reason to think the outcome would vary with time but wished to check this was the case, then having done so and found little or no evidence for it varying with time, i'd say it's quite entirely reasonable to then drop time. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've never heard of Underwood before. It may be a standard text for 'Experiments in Ecology' (the book's title), but there's no obvious reason that experiments in ecology should be treated any differently from any other experiments in this respect, so to view it as &quot;&lt;em&gt;the&lt;/em&gt; standard text on this issue&quot; seems unjustified.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-12-06T14:57:16.090" CommentCount="3" />
  <row Id="5187" PostTypeId="1" AcceptedAnswerId="5188" CreationDate="2010-12-06T17:23:20.837" Score="2" ViewCount="149" Body="&lt;p&gt;Hello,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am new to Gibbs Sampling, and have been using WinBUGS, but I find that it is not well-suited towards storing/presenting results, so I have been calling it from R using the R2WinBUGS package.  The data is apparently stored as a &quot;bug&quot; class.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I converted it to coda to run diagnostics, and it displays each of the chains, but I am confused as to the $ extension for each individual chain.  I cannot find any good documentation for the &quot;coda&quot; class (the cran instructions are not helpful).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My code is below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; bugs.sim &amp;lt;- bugs(data, inits, parameter, &quot;gl.bug&quot;, n.chains = 5,&#xA;                   codaPkg = TRUE, DIC = FALSE, n.iter = 5000)&#xA;&amp;gt; codaobject &amp;lt;- read.bugs(bugs.sim)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As you can see, I have 5 chains, and I would like to take the mean and standard deviation of each.  How do I go about doing this?  I can use the codaobject to take the &lt;a href=&quot;http://www.people.fas.harvard.edu/~plam/teaching/methods/convergence/convergence_print.pdf&quot; rel=&quot;nofollow&quot;&gt;Geweke diagnostic&lt;/a&gt; of each chain, it displays each chain as &lt;code&gt;[[i]]&lt;/code&gt; ($i=1,\dots,5$).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance.  And any references to a detailed documentation for R2Winbugs, would also be greatly appreciated.&lt;/p&gt;&#xA;" OwnerUserId="2274" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-07T10:56:02.763" LastActivityDate="2010-12-07T15:57:53.917" Title="Using R2WinBUGS, how to extract information from each chain?" Tags="&lt;r&gt;&lt;mcmc&gt;&lt;bugs&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="5188" PostTypeId="2" ParentId="5187" CreationDate="2010-12-06T18:03:04.027" Score="3" ViewCount="" Body="&lt;p&gt;The object returned by &lt;code&gt;read.bugs&lt;/code&gt; is an object of S3 class &lt;code&gt;mcmc.list&lt;/code&gt;. &#xA;You can use the double brackets &lt;code&gt;[[&lt;/code&gt; to access the separate chains, i.e. the different &lt;code&gt;mcmc&lt;/code&gt;-objects that make up the larger &lt;code&gt;mcmc.list&lt;/code&gt; object, which really is simply a list of &lt;code&gt;mcmc&lt;/code&gt;-objects that inherits some information about thinning and chain length from its components.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More to the point, s.th. like &lt;code&gt;lapply(codaobject, function(x){ colMeans(x) })&lt;/code&gt; should return the posterior means for each parameter in each chain and &lt;code&gt;lapply(codaobject, function(x){ apply(x, 2, sd) })&lt;/code&gt; should give chain- and parameter-specific posterior sd's, since each chain is essentially just a numeric matrix with rows corresponding to the (saved) iterations and columns corresponding to the different params.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT:&#xA;I think Gelman/Hill's &quot;Bayesian Data Analysis&quot; contains some worked examples using R2WinBUGS.&lt;/p&gt;&#xA;" OwnerUserId="1979" LastEditorUserId="1979" LastEditorDisplayName="" LastEditDate="2010-12-06T18:10:12.523" LastActivityDate="2010-12-06T18:10:12.523" CommentCount="2" />
  <row Id="5189" PostTypeId="1" AcceptedAnswerId="5194" CreationDate="2010-12-06T18:40:58.743" Score="2" ViewCount="310" Body="&lt;p&gt;With some great help from this forum, I have been able to get up and running with some basic time series analysis in R.  Right now, my needs are mostly univariate time series.  Here is my question:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can read in daily data from database into a data frame.  I have two columns, date which is understood by R as POSIXct and the second which is the value of interest and numeric.  What is the best, most straightforward way to make this a ts object where R understands the start/end dates and represents daily observations dynamically?  It seems to me that I shouldn't be required to (when coercing my object to a ts object)  manually tell it the start and end dates when the data frame is already has it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For some context, I have been able to aggregate other data from daily to weekly, but find myself doing things in ways that just seem long and unnecessary considering R already understands my raw data as time.  As you can tell, I am new to R and time series in R, but I figure that since R is so powerful, there probably is a pretty easy way around my issues.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help you can provide will be very much appreciated.&lt;/p&gt;&#xA;" OwnerUserId="569" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-07T10:52:55.247" LastActivityDate="2010-12-07T10:52:55.247" Title="Getting started with time series in R" Tags="&lt;r&gt;&lt;time-series&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="3" />
  <row Id="5190" PostTypeId="2" ParentId="5181" CreationDate="2010-12-06T20:27:33.010" Score="3" ViewCount="" Body="&lt;p&gt;What you seem to be referring to is called &lt;a href=&quot;http://en.wikipedia.org/wiki/Shrinkage_%28statistics%29&quot; rel=&quot;nofollow&quot;&gt;&quot;shrinkage&quot;&lt;/a&gt;. This allows you to share strength across groups and is frequently used in hierarchical Bayesian models. The (very) basic idea is to impose a prior distribution over the entire population and place more weight on the prior for those groups that have few samples, much like you would for any Bayesian analysis. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The best reference I can think of for this is ch. 8 of Peter Hoff's book: &quot;A First Course in Bayesian Statistical Methods&quot;, but if you don't have access to this book, I'm sure any book on Bayesian statistics will talk about this (search for &quot;group comparison&quot;, &quot;hierarchical modeling&quot;, or &quot;shrinkage&quot;). I know that &lt;a href=&quot;http://faculty.chicagobooth.edu/robert.gramacy/papers.html&quot; rel=&quot;nofollow&quot;&gt;Robert Gramacy&lt;/a&gt; has done some work on this as well, so you may wish to check out his publications.&lt;/p&gt;&#xA;" OwnerUserId="1913" LastActivityDate="2010-12-06T20:27:33.010" />
  <row Id="5191" PostTypeId="1" AcceptedAnswerId="5193" CreationDate="2010-12-06T21:08:37.437" Score="2" ViewCount="41" Body="&lt;p&gt;This question is related to my previous  question &lt;a href=&quot;http://stats.stackexchange.com/questions/5011/bias-for-kernel-density-estimator-periodic-case&quot;&gt;Bias for kernel density estimator (periodic case)&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A kernel $K(x)$ is of the order $p$ if&#xA;$$\int_{-\infty}^{\infty}K(x)x^{j}=\delta_{0,j}\ j=0,...p-1$$ &#xA;$$\int_{-\infty}^{\infty}K(x)x^{p}\neq0\ $$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does it mean that for  the kernel  with period 1 the definition of  the order of the kernel is&#xA;$$\int_{0}^{1}K(x)Min(x,1-x)^{j}=\delta_{0,j}\ j=0,...p-1$$&#xD;&#xA;$$\int_{0}^{1}K(x)Min(x,1-x)^{p}\neq0\ $$&#xD;&lt;/p&gt;&#xA;" OwnerUserId="2189" LastActivityDate="2010-12-07T00:03:46.107" Title="Order of the kernel for periodic case" Tags="&lt;kernel&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="5192" PostTypeId="2" ParentId="5115" CreationDate="2010-12-06T21:24:56.347" Score="10" ViewCount="" Body="&lt;p&gt;George Box for his work on time series, designed experiments and elucidating the iterative nature of scientific discovery (proposing and testing models).&lt;/p&gt;&#xA;" OwnerUserId="2277" LastActivityDate="2010-12-06T21:24:56.347" CommunityOwnedDate="2010-12-06T21:24:56.347" CommentCount="1" />
  <row Id="5193" PostTypeId="2" ParentId="5191" CreationDate="2010-12-07T00:03:46.107" Score="2" ViewCount="" Body="&lt;p&gt;I think the correct analog of this definition in the periodic case is that coefficients $1$ through $p-1$ of the Fourier Series for $K$ all vanish.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The purpose of the definition of &lt;em&gt;order&lt;/em&gt; is to obtain estimates of the bias of the kernel estimator.  When $K$ &quot;kills&quot; powers $1$ through $p-1$ of $x$, then the bias will be approximately of order $h^p$ for a bandwidth $h$.  This is proven in Tsybakov's Proposition 1.2 by expanding the pdf in a power series: multiplication by $K$ kills off the terms through order $p-1$, leaving the Taylor error term of order $p$; elementary estimates of that integral finish the job.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The analog of a power series for periodic functions is the Fourier Series.  The analog is a perfect one: we can think of a periodic function as being defined on the unit circle in the complex plane.  It has a complex coordinate $q = e^{i x}$ (where now the period is $2\pi$ rather than $1$, but that's inconsequential).  Expanding $K(q)$ in a power series expresses it as a sum of powers of $q$.  However, from&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$q^j = (e^{i x})^j = e^{i x j} = \cos(j x) + i \sin(j x)$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;we see that this expansion is just the Fourier Series (both the sine and cosine terms).  Consequently you should be able to emulate the proof of Proposition 1.2 with very little change at all.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-12-07T00:03:46.107" />
  <row Id="5194" PostTypeId="2" ParentId="5189" CreationDate="2010-12-07T01:14:28.133" Score="2" ViewCount="" Body="&lt;p&gt;It seems like you need the package xts.&#xA;Create your time serie using &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;install.packages('xts')&#xA;library(xts)&#xA;X = xts(coredata(DF[,2]), order.by=DF[,1])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then you will be able to manipulate your data easily.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;to.weekly(X)  &#xA;to.monthly(X)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Please note that you will then manipulate xts objects and not ts. But no worries, you can go back to ts whenever needed.&lt;/p&gt;&#xA;" OwnerUserId="1709" LastEditorUserId="1709" LastEditorDisplayName="" LastEditDate="2010-12-07T02:22:22.400" LastActivityDate="2010-12-07T02:22:22.400" />
  <row Id="5195" PostTypeId="1" AcceptedAnswerId="5209" CreationDate="2010-12-07T01:29:37.223" Score="5" ViewCount="425" Body="&lt;p&gt;As title, I need to draw something like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/KYQ5V.jpg&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can ggplot, or other packages if ggplot is not capable, be used to draw something like this?&lt;/p&gt;&#xA;" OwnerUserId="588" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-07T10:53:58.223" LastActivityDate="2010-12-07T13:19:27.420" Title="How to draw funnel plot using ggplot2 in R?" Tags="&lt;r&gt;&lt;data-visualization&gt;&lt;ggplot2&gt;&lt;funnel-plot&gt;" AnswerCount="2" CommentCount="4" FavoriteCount="3" />
  <row Id="5196" PostTypeId="1" AcceptedAnswerId="5201" CreationDate="2010-12-07T01:31:14.380" Score="3" ViewCount="286" Body="&lt;p&gt;In order to calibrate a confidence level to a probability in supervised learning (say to map the confidence from an SVM or a decision tree using oversampled data) one method is to use Platt's Scaling (e.g., &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.60.5153&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;Obtaining Calibrated Probabilities from Boosting&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically one uses logistic regression to map $[-\infty;\infty]$ to $[0;1]$. The dependent variable is the true label and the predictor is the confidence from the uncalibrated model. What I don't understand is the use of a target variable other than 1 or 0. The method calls for creation of a new &quot;label&quot; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/ENsFb.jpg&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I don't understand is how this new target is useful. Isn't logistic regression simply going to treat the dependent variable as a binary label (regardless of what label is given)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found that in SAS changing the dependent from 1/0 to something else reverted back to the same model (using proc genmod). Perhaps my error or perhaps SAS's lack of versatility. I was able to change the model in R. As an example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data(ToothGrowth) attach(ToothGrowth) #1/0 coding &#xA;dep &amp;lt;-ifelse(supp == &quot;VC&quot;, 1,0) &#xA;OneZeroModel &amp;lt;-glm(dep~len,family=binomial) &#xA;OneZeroModel &#xA;predict(OneZeroModel) &#xA;&#xA;#Platt coding &#xA;dep2 &amp;lt;-ifelse(supp == &quot;VC&quot;, 31/32 ,1/32) &#xA;plattCodeModel &amp;lt;-glm(dep2~len,family=binomial) &#xA;plattCodeModel &#xA;predict(plattCodeModel) &#xA;&#xA;compare&amp;lt;-cbind(predict(OneZeroModel),predict(plattCodeModel)) &#xA;&#xA;plot(predict(OneZeroModel),predict(plattCodeModel))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2040" LastEditorUserId="2040" LastEditorDisplayName="" LastEditDate="2010-12-09T01:09:30.390" LastActivityDate="2010-12-09T01:09:30.390" Title="Why use Platt's scaling?" Tags="&lt;logistic-regression&gt;&lt;cross-validation&gt;&lt;calibration&gt;" AnswerCount="2" FavoriteCount="3" />
  <row Id="5197" PostTypeId="1" CreationDate="2010-12-07T02:08:09.843" Score="2" ViewCount="158" Body="&lt;p&gt;I have database of 78706 resident incidents in aged care facilities (5 years of data).&#xA;I want to to learn and implement a tool allowing analyzing these data using following attributes:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Resident&lt;/li&gt;&#xA;&lt;li&gt;Date/Time&lt;/li&gt;&#xA;&lt;li&gt;Location&lt;/li&gt;&#xA;&lt;li&gt;Result&lt;/li&gt;&#xA;&lt;li&gt;Injury&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I want to be able to get following assumptions from my system which will be passed to specialists for further research, decision making and action:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Examples of outputs:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Most of incidents in facility A with residents X, Y and Z&lt;/li&gt;&#xA;&lt;li&gt;Falls occur in North Wing between 2am and 5am&lt;/li&gt;&#xA;&lt;li&gt;Skin tears happen during showering in facility B&lt;/li&gt;&#xA;&lt;li&gt;Most incidents in a facility C related to repositioning&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;My question is &lt;strong&gt;not&lt;/strong&gt; what software package can help me but what &lt;strong&gt;type&lt;/strong&gt; of statistical analysis solves this problem - regression, cluster etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can you recommend some &lt;strong&gt;practical books&lt;/strong&gt; for a starter too?&lt;/p&gt;&#xA;" OwnerUserId="2279" LastActivityDate="2010-12-07T03:07:27.183" Title="What type of statistical analysis solves this problem?" Tags="&lt;regression&gt;&lt;clustering&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="5198" PostTypeId="2" ParentId="5152" CreationDate="2010-12-07T02:59:36.707" Score="1" ViewCount="" Body="&lt;p&gt;It is hard to imagine a situation when the effect of age with a precision of a month is not sufficient - even for babies after the first few months of life nobody uses weeks. For adults, even rounding to years should be just fine. &lt;/p&gt;&#xA;" OwnerUserId="279" LastActivityDate="2010-12-07T02:59:36.707" />
  <row Id="5199" PostTypeId="2" ParentId="5197" CreationDate="2010-12-07T03:07:27.183" Score="1" ViewCount="" Body="&lt;p&gt;You could consider &lt;a href=&quot;http://en.wikipedia.org/wiki/Association_rule_learning&quot; rel=&quot;nofollow&quot;&gt;association analysis&lt;/a&gt;.  If your time is discretized appropriately and the data support your 2nd example (Falls occur in North Wing between 2am and 5am), one possible learned rule that comes out of the analysis might be {North Wing, 2AM-5AM} =&gt; {Fall}.&lt;/p&gt;&#xA;" OwnerUserId="1815" LastActivityDate="2010-12-07T03:07:27.183" />
  <row Id="5200" PostTypeId="1" AcceptedAnswerId="5203" CreationDate="2010-12-07T03:47:50.017" Score="1" ViewCount="118" Body="&lt;p&gt;In Frank Schorfheide's class notes on &lt;a href=&quot;http://www.webpages.ttu.edu/pesummer/ECO%205328/notes/4%20likelihood%20dsge.pdf&quot; rel=&quot;nofollow&quot;&gt;likelihood functions of DSGE models&lt;/a&gt;, he expresses the value of the likelihood function for a given vector of parameters $\theta$, and time series $Y^T$ as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p(Y^{T}|\theta)=(2\pi)^{-nT/2}\left(\prod_{t=1}^{T}\left|F_{t|t-1}\right|\right)^{-1/2}exp\{-\frac{1}{2}\sum_{t=1}^{T}v_{t}F_{t|t-1}v_t\prime\}$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $v_t$ is the innovation in $y$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$v_t=y_t-\hat{y}_{t|t-1}$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and the marginal distribution of $y_t$ is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$y_t|Y^{t-1}\sim\mathcal{N}\left(\hat{y}_{t|t-1}, F_{t|t-1}\right)$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've just got a few questions about what these terms look like. First, does anyone have an idea what $n$ is in the exponent of the first term in the first equation? I think it might be a misprint, but I'm not sure. Second, what does $F_{t|t-1}$ look like? For an $n\times 1$ vector $y$ I'm picturing an $n\times n$ matrix, but what would the values of $F_{j,k}$ be equal to? I'm picturing the covariance between $y_{t,j}$ and $\hat{y}_{t|t-1,k}$ - is that correct? Lastly, I'm assuming from the results of my code that the value the likelihood function returns is a scalar, but it doesn't look like the formula produces one -- for an $n\times 1$ vector $y$, wouldn't the second term in the first equation be $n\times n$? Or do you think it's meant to be the determinant of $F$?&lt;/p&gt;&#xA;" OwnerUserId="2251" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-07T10:47:30.507" LastActivityDate="2010-12-07T10:47:30.507" Title="Likelihood function of DSGE model using Kalman filter" Tags="&lt;kalman-filter&gt;&lt;likelihood-function&gt;" AnswerCount="1" />
  <row Id="5201" PostTypeId="2" ParentId="5196" CreationDate="2010-12-07T07:43:17.147" Score="5" ViewCount="" Body="&lt;p&gt;I suggest to check out the &lt;a href=&quot;http://en.wikipedia.org/wiki/Logistic_regression&quot; rel=&quot;nofollow&quot;&gt;wikipedia page of logistic regression&lt;/a&gt;. It states that in case of a binary dependent variable logistic regression maps the predictors to the probability of occurrence of the dependent variable. Without any transformation, the probability used for training the model is either 1 (if y is positive in the training set) or 0 (if y is negative). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So: Instead of using the absolute values 1 for positive class and 0 for negative class when fitting $p_i=\frac{1}{(1+exp(A*f_i+B))}$ (where $f_i$ is the uncalibrated output of the SVM), Platt suggests to use the mentioned transformation to allow the opposite label to appear with some probability. In this way some regularization is introduced. When the size of the dataset reaches infinity, $y_+$ will become 1 and $y_{-}$ will become zero. For details, see the &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.41.1639&quot; rel=&quot;nofollow&quot;&gt;original paper of Platt&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="264" LastActivityDate="2010-12-07T07:43:17.147" CommentCount="4" />
  <row Id="5202" PostTypeId="2" ParentId="5115" CreationDate="2010-12-07T07:46:56.347" Score="2" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Teuvo_Kohonen&quot; rel=&quot;nofollow&quot;&gt;Teuvo Kohonen&lt;/a&gt; for invention of the &lt;a href=&quot;http://en.wikipedia.org/wiki/Self-Organizing_Map&quot; rel=&quot;nofollow&quot;&gt;Self-Organizing-Map&lt;/a&gt; (SOM).&lt;/p&gt;&#xA;" OwnerUserId="264" LastActivityDate="2010-12-07T07:46:56.347" CommunityOwnedDate="2010-12-07T07:46:56.347" CommentCount="5" />
  <row Id="5203" PostTypeId="2" ParentId="5200" CreationDate="2010-12-07T08:25:52.680" Score="2" ViewCount="" Body="&lt;p&gt;$n$ is the dimension of the observation vector, as you mention in your question.&#xA;$F$ is the covariance matrix of innovations; I think you are missing an exponent of -1 in the last term of the likelihood. It should read $v_t&amp;#39; F_{t|t-1}^{-1} v_t$ (using the convention that $v_t$ is a column vector; your notation&#xA;seems to assume the opposite).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you have in the second term of the likelihood is indeed the determinant of $F$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might want to peruse any among many excellent books on Kalman filter and state-space models, like &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0198523548&quot; rel=&quot;nofollow&quot;&gt;Durbin-Koopman&lt;/a&gt;, &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0486439380&quot; rel=&quot;nofollow&quot;&gt;Anderson-Moore&lt;/a&gt; or &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0521405734&quot; rel=&quot;nofollow&quot;&gt;Harvey&lt;/a&gt;. Or may be just look at the Wikipedia the topic on &lt;a href=&quot;http://en.wikipedia.org/wiki/Kalman_filter&quot; rel=&quot;nofollow&quot;&gt;Kalman filtering&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="892" LastActivityDate="2010-12-07T08:25:52.680" CommentCount="1" />
  <row Id="5204" PostTypeId="2" ParentId="5196" CreationDate="2010-12-07T08:57:03.053" Score="1" ViewCount="" Body="&lt;p&gt;Another method of avoiding over-fitting that I have found useful is to fit the univariate logistic regression model to the leave-out-out cross-validation output of the SVM, which can be approximated efficiently using the &lt;a href=&quot;http://dx.doi.org/10.1023/A:1012450327387&quot; rel=&quot;nofollow&quot;&gt;Span bound&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, if you want a classifier that produces estimates of the probability of class membership, then you would be better off using kernel logistic regression, which aims to do that directly.  The ouput of the SVM is designed for discrete classification and doesn't necessarily contain the information required for accurate estimation of probabilities away from the p=0.5 contour.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.gaussianprocess.org/gpml/&quot; rel=&quot;nofollow&quot;&gt;Gaussian process classifiers&lt;/a&gt; are another good option if you want a kernel based probabilistic classifier.&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2010-12-07T08:57:03.053" />
  <row Id="5205" PostTypeId="2" ParentId="5167" CreationDate="2010-12-07T09:59:50.810" Score="1" ViewCount="" Body="&lt;p&gt;You shouldn't need a Gibbs sampler: the mode of an &lt;a href=&quot;http://en.wikipedia.org/wiki/Inverse-Wishart_distribution&quot; rel=&quot;nofollow&quot;&gt;inverse-Wishart&lt;/a&gt; has a closed form. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, &lt;em&gt;independent&lt;/em&gt; random samples from the Cholesky factor of a Wishart can be obtained from the &lt;a href=&quot;http://en.wikipedia.org/wiki/Wishart_distribution#Bartlett_decomposition&quot; rel=&quot;nofollow&quot;&gt;Bartlett decomposition&lt;/a&gt;: as it is triangular, it can be inverted easily by forward subsitution to get the Cholesky factor of an inverse-Wishart.&lt;/p&gt;&#xA;" OwnerUserId="495" LastActivityDate="2010-12-07T09:59:50.810" />
  <row Id="5206" PostTypeId="1" AcceptedAnswerId="5208" CreationDate="2010-12-07T10:10:47.810" Score="5" ViewCount="244" Body="&lt;p&gt;This is the confidence interval estimated by prop.test&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;n &amp;lt;- 600; x &amp;lt;- 276; p &amp;lt;- 0.40&#xA;prop.test(x, n, p, alternative=&quot;two.sided&quot;, conf.level=0.95, correct=T)&#xA;95 percent confidence interval:&#xA; 0.4196787 0.5008409 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I tried to reproduce it, reading the code under prop.test. Here is a simplified way to get those two limits&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ESTIMATE &amp;lt;- x/n&#xA;YATES &amp;lt;- 0.5&#xA;conf.level &amp;lt;- 0.95&#xA;z &amp;lt;- qnorm((1 + conf.level)/2)&#xA;YATES &amp;lt;- min(YATES, abs(x - n * p)) &#xA;z22n &amp;lt;- z^2/(2 * n)&#xA;p.c &amp;lt;- ESTIMATE + YATES/n&#xA;(p.c + z22n + z * sqrt(p.c * (1 - p.c)/n + z22n/(2 * n)))/(1 + 2 * z22n)&#xA;[1] 0.5008409&#xA;p.c &amp;lt;- ESTIMATE - YATES/n&#xA;(p.c + z22n - z * sqrt(p.c * (1 - p.c)/n + z22n/(2 * n)))/(1 + 2 * z22n)&#xA;[1] 0.4196787&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Can you explain to me why the underlying probability of success (p) is used in line 5? or maybe you could suggest where can I find more info about this YATES correction that affects the ESTIMATE.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you&lt;/p&gt;&#xA;" OwnerUserId="339" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-07T10:37:21.260" LastActivityDate="2010-12-07T14:08:24.490" Title="Yates' continuity correction in confidence interval returned by prop.test" Tags="&lt;r&gt;&lt;confidence-interval&gt;&lt;yates-correction&gt;" AnswerCount="2" />
  <row Id="5207" PostTypeId="1" CreationDate="2010-12-07T10:35:08.577" Score="7" ViewCount="226" Body="&lt;p&gt;I tried to simulate from a bivariate density $p(x,y)$ using Metropolis algorithms in R and had no luck. The density can be expressed as  $p(y|x)p(x)$, where $p(x)$ is Singh-Maddala distribution &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$p(x)=\dfrac{aq x^{a-1}}{b^a (1 + (\frac{x}{b})^a)^{1+q}}$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;with parameters $a$, $q$, $b$, and $p(y|x)$ is log-normal with log-mean as a fraction of $x$, and log-sd a constant. To test whether my sample is the one I want, I looked at the marginal density of $x$, which should be $p(x)$. I tried different Metropolis algorithms from R packages MCMCpack, mcmc and dream. I discarded burn-in, used thinning, used samples with size up to million, but the resulting marginal density was never the one I supplied. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the final edition of my code I used:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;logvrls &amp;lt;- function(x,el,sdlog,a,scl,q.arg) {&#xA;    if(x[2]&amp;gt;0) {&#xA;         dlnorm(x[1],meanlog=el*log(x[2]),sdlog=sdlog,log=TRUE)+&#xA;         dsinmad(x[2],a=a,scale=scl,q.arg=q.arg,log=TRUE)&#xA;    }&#xA;    else -Inf    &#xA;}&#xA;&#xA;a &amp;lt;- 1.35&#xA;q &amp;lt;- 3.3&#xA;scale &amp;lt;- 10/gamma(1 + 1/a)/gamma(q - 1/a)*  gamma(q) &#xA;&#xA;Initvrls &amp;lt;- function(pars,nseq,meanlog,sdlog,a,scale,q) {&#xA;    cbind(rlnorm(nseq,meanlog,sdlog),rsinmad(nseq,a,scale,q))&#xA;}&#xA;&#xA;library(dream)&#xA;aa &amp;lt;- dream(logvrls,&#xA;        func.type=&quot;logposterior.density&quot;,&#xA;        pars=list(c(0,Inf),c(0,Inf)),&#xA;        FUN.pars=list(el=0.2,sdlog=0.2,a=a,scl=scale,q.arg=q),&#xA;        INIT=Initvrls,&#xA;        INIT.pars=list(meanlog=1,sdlog=0.1,a=a,scale=scale,q=q),&#xA;        control=list(nseq=3,thin.t=10)&#xA;        )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I've settled on dream package, since it samples until the convergence. I've tested whether I have the correct results in three ways. Using KS statistic, comparing quantiles, and estimating the parameters of Singh-Maddala distribution with maximum likelihood from the resulting sample:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ks.test(as.numeric(aa$Seq[[2]][,2]),psinmad,a=a,scale=scale,q.arg=q)&#xA;&#xA;lsinmad &amp;lt;- function(x,sample)&#xA;    sum(dsinmad(sample,a=x[1],scale=x[2],q.arg=x[3],log=TRUE))&#xA; optim(c(2,20,2),lsinmad,method=&quot;BFGS&quot;,sample=aa$Seq[[1]][,2])&#xA;&#xA; qq &amp;lt;- eq(0.025,.975,by=0.025)   &#xA; tst &amp;lt;- cbind(qq,&#xA;              sapply(aa$Seq,function(l)round(quantile(l[,2],qq),3)),&#xA;              round(qsinmad(qq,a,scale,q),3))&#xA; colnames(tst) &amp;lt;- c(&quot;Quantile&quot;,&quot;S1&quot;,&quot;S2&quot;,&quot;S3&quot;,&quot;True&quot;)&#xA;&#xA; library(ggplot2)&#xA; qplot(x=Quantile,y=value,&#xA;       data=melt(data.frame(tst),id=1), &#xA;       colour=variable,group=variable,geom=&quot;line&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When I look at the results of these comparisons, KS statistic almost always rejects the null hypothesis that sample is from Singh-Maddala distribution with supplied parameters. Maximum likelihood estimated parameters sometimes comes close to its true values, but usually too far out of comfort zone, to accept that sampling procedure was succesfull. Ditto for the quantiles, empirical quantiles are not too far, but too far away. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is what I am doing wrong? My own hypotheses:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;MCMC is not appropriate for this type of sampling&lt;/li&gt;&#xA;&lt;li&gt;MCMC cannot converge, due to theoretical reasons (the distribution function does not satisfy required properties, whatever they are)&lt;/li&gt;&#xA;&lt;li&gt;I do not use the Metropolis algorithm correctly&lt;/li&gt;&#xA;&lt;li&gt;My distribution tests are not correct, since I do not have independent sample.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2010-12-07T13:03:37.737" LastActivityDate="2011-05-12T19:03:23.007" Title="Sampling from bivariate distribution with known density using MCMC" Tags="&lt;sampling&gt;&lt;monte-carlo&gt;&lt;metropolis-hastings&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="1" />
  <row Id="5208" PostTypeId="2" ParentId="5206" CreationDate="2010-12-07T10:55:54.983" Score="3" ViewCount="" Body="&lt;p&gt;The help page indicates that &quot;Continuity correction is used only if it does not exceed the difference between sample and null proportions in absolute value.&quot; This is what line 5 is checking: &lt;code&gt;x/n&lt;/code&gt; is the empirical proportion, &lt;code&gt;p&lt;/code&gt; is the null proportion. (Actually, I find the &quot;if&quot; slightly misleading since it's more of a &quot;insofar as it does not exceed&quot; when looking at line 5.)&lt;/p&gt;&#xA;" OwnerUserId="1909" LastActivityDate="2010-12-07T10:55:54.983" />
  <row Id="5209" PostTypeId="2" ParentId="5195" CreationDate="2010-12-07T11:38:07.737" Score="6" ViewCount="" Body="&lt;p&gt;Although there's room for improvement, here is a small attempt with simulated (heteroscedastic) data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(ggplot2)&#xA;set.seed(101)&#xA;x &amp;lt;- runif(100, min=1, max=10)&#xA;y &amp;lt;- rnorm(length(x), mean=5, sd=0.1*x)&#xA;df &amp;lt;- data.frame(x=x*70, y=y)&#xA;m &amp;lt;- lm(y ~ x, data=df) &#xA;fit95 &amp;lt;- predict(m, interval=&quot;conf&quot;, level=.95)&#xA;fit99 &amp;lt;- predict(m, interval=&quot;conf&quot;, level=.999)&#xA;df &amp;lt;- cbind.data.frame(df, &#xA;                       lwr95=fit95[,&quot;lwr&quot;],  upr95=fit95[,&quot;upr&quot;],     &#xA;                       lwr99=fit99[,&quot;lwr&quot;],  upr99=fit99[,&quot;upr&quot;])&#xA;&#xA;p &amp;lt;- ggplot(df, aes(x, y)) &#xA;p + geom_point() + &#xA;    geom_smooth(method=&quot;lm&quot;, colour=&quot;black&quot;, lwd=1.1, se=FALSE) + &#xA;    geom_line(aes(y = upr95), color=&quot;black&quot;, linetype=2) + &#xA;    geom_line(aes(y = lwr95), color=&quot;black&quot;, linetype=2) +&#xA;    geom_line(aes(y = upr99), color=&quot;red&quot;, linetype=3) + &#xA;    geom_line(aes(y = lwr99), color=&quot;red&quot;, linetype=3)  + &#xA;    annotate(&quot;text&quot;, 100, 6.5, label=&quot;95% limit&quot;, colour=&quot;black&quot;, &#xA;             size=3, hjust=0) +&#xA;    annotate(&quot;text&quot;, 100, 6.4, label=&quot;99.9% limit&quot;, colour=&quot;red&quot;, &#xA;             size=3, hjust=0) +&#xA;    labs(x=&quot;No. admissions...&quot;, y=&quot;Percentage of patients...&quot;) +    &#xA;    theme_bw() &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/stUAo.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-12-07T11:38:07.737" />
  <row Id="5210" PostTypeId="2" ParentId="5195" CreationDate="2010-12-07T13:19:27.420" Score="4" ViewCount="" Body="&lt;p&gt;If you are looking for this (meta-analysis) type of &lt;a href=&quot;http://en.wikipedia.org/wiki/Funnel_plot&quot; rel=&quot;nofollow&quot;&gt;funnel plot&lt;/a&gt;, then the following might be a starting point:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(ggplot2)&#xA;&#xA;set.seed(1)&#xA;p &amp;lt;- runif(100)&#xA;number &amp;lt;- sample(1:1000, 100, replace = TRUE)&#xA;p.se &amp;lt;- sqrt((p*(1-p)) / (number))&#xA;df &amp;lt;- data.frame(p, number, p.se)&#xA;&#xA;## common effect (fixed effect model)&#xA;p.fem &amp;lt;- weighted.mean(p, 1/p.se^2)&#xA;&#xA;## lower and upper limits for 95% and 99.9% CI, based on FEM estimator&#xA;number.seq &amp;lt;- seq(0.001, max(number), 0.1)&#xA;number.ll95 &amp;lt;- p.fem - 1.96 * sqrt((p.fem*(1-p.fem)) / (number.seq)) &#xA;number.ul95 &amp;lt;- p.fem + 1.96 * sqrt((p.fem*(1-p.fem)) / (number.seq)) &#xA;number.ll999 &amp;lt;- p.fem - 3.29 * sqrt((p.fem*(1-p.fem)) / (number.seq)) &#xA;number.ul999 &amp;lt;- p.fem + 3.29 * sqrt((p.fem*(1-p.fem)) / (number.seq)) &#xA;dfCI &amp;lt;- data.frame(number.ll95, number.ul95, number.ll999, number.ul999, number.seq, p.fem)&#xA;&#xA;## draw plot&#xA;fp &amp;lt;- ggplot(aes(x = number, y = p), data = df) +&#xA;    geom_point(shape = 1) +&#xA;    geom_line(aes(x = number.seq, y = number.ll95), data = dfCI) +&#xA;    geom_line(aes(x = number.seq, y = number.ul95), data = dfCI) +&#xA;    geom_line(aes(x = number.seq, y = number.ll999, linetype = 2), data = dfCI) +&#xA;    geom_line(aes(x = number.seq, y = number.ul999, linetype = 2), data = dfCI) +&#xA;    geom_hline(aes(yintercept = p.fem), data = dfCI) +&#xA;    scale_y_continuous(limits = c(0,1.1)) +&#xA;  xlab(&quot;number&quot;) + ylab(&quot;p&quot;) + theme_bw() &#xA;fp&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/2mrP6.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="307" LastActivityDate="2010-12-07T13:19:27.420" />
  <row Id="5211" PostTypeId="1" CreationDate="2010-12-07T13:21:51.317" Score="0" ViewCount="151" Body="&lt;p&gt;Hi guys,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to determine effects on a brand using pre- and post-exposure measurement.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One dependent variable incorporates several items on Likert type scales (&lt;em&gt;strongly disagree/ strongly agree from 1-7&lt;/em&gt;) like attitude toward the brand, perceived value etc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is calculated as the mean of the post minus pre differences of all items. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another item I would like to include is measured differently in € (&lt;em&gt;e.g. € 1,50 in pre-condition and € 1,60 in post-condition&lt;/em&gt;). To overcome the difference in measurement problem I want to calculate the z-scores of all post-pre differences (&lt;em&gt;e.g. z-score of 10ct. with above example&lt;/em&gt;) of all items, and afterwards calculate the mean. Later on I want to perform anovas and t-tests with this standardized variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My question:&lt;/strong&gt; Can i beforehand assess the reliability of this measure with standard procedures (cronbachs alpha, exploratory factor analysis)? Is there anything I have to observe?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any insight is greatly appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance!  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Kind Regards &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alex&lt;/p&gt;&#xA;" OwnerUserId="2285" LastActivityDate="2010-12-07T13:21:51.317" Title="How to test reliability of dependent variable when z-score transformation is in effect" Tags="&lt;reliability&gt;&lt;within-subjects&gt;&lt;z-statistic&gt;" CommentCount="2" />
  <row Id="5212" PostTypeId="2" ParentId="5206" CreationDate="2010-12-07T14:08:24.490" Score="4" ViewCount="" Body="&lt;p&gt;On the second question of where you can find more info on this continuity correction (attributed to Yates in the help for &lt;code&gt;prop.test&lt;/code&gt; but not in the refs below, I think &lt;a href=&quot;http://en.wikipedia.org/wiki/Yates%27_correction_for_continuity&quot; rel=&quot;nofollow&quot;&gt;as Yates orginally proposed a continuity correction only to the chi-squared test for contingency tables&lt;/a&gt;):&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Newcombe RG. Two-sided confidence intervals for the single proportion: comparison of seven methods. Stat Med 1998; 17(8):857-872. PMID:&lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pubmed/9595616&quot; rel=&quot;nofollow&quot;&gt;9595616&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Brown LD, Cai TT, DasGupta A. Interval estimation for a binomial proportion (with Comments &amp;amp;  Rejoinder). Statistical Science 2001; 16(2):101-133. doi:&lt;a href=&quot;http://dx.doi.org/doi:10.1214/ss/1009213286&quot; rel=&quot;nofollow&quot;&gt;10.1214/ss/1009213286&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The continuity-corrected Wilson score interval is 'method 4' in Newcomb. Brown et al. consider only the uncorrected Wilson score interval in the main text, but George Casella suggests using the continuity-corrected version in his Comment (p121), which Brown et al. discuss in their Rejoinder (p130):&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Casella suggests the possibility of performing&#xA;  a continuity correction on the score statistic&#xA;  prior to constructing a confidence interval. We do&#xA;  not agree with this proposal from any perspective.&#xA;  These “continuity-corrected Wilson” intervals&#xA;  have extremely conservative coverage properties,&#xA;  though they may not in principle be guaranteed to&#xA;  be everywhere conservative. But even if one’s goal,&#xA;  unlike ours, is to produce conservative intervals,&#xA;  these intervals will be very inefficient at their normal&#xA;  level relative to Blyth–Still or even Clopper–&#xA;  Pearson.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The Clopper-Pearson 'exact' interval is provided by &lt;code&gt;binom.test&lt;/code&gt; in R. I'd suggest using that rather than &lt;code&gt;prop.test&lt;/code&gt; if you want a conservative interval, i.e. one that guarantees &lt;em&gt;at least&lt;/em&gt; 95% coverage. If you'd prefer an interval that has close to 95% coverage &lt;em&gt;on average&lt;/em&gt; (over p) and will therefore often be narrower, you could use &lt;code&gt;prop.test(…, correct=FALSE)&lt;/code&gt; to give the uncorrected Wilson score interval.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The standard textbook for such matters is Fleiss &lt;em&gt;Statistical Methods for Rates and Proportions&lt;/em&gt;. Newcomb references the original 1981 edition but &lt;a href=&quot;http://books.google.co.uk/books?id=a5LwdxF2d10C&quot; rel=&quot;nofollow&quot;&gt;the latest edition is the 3rd (2003)&lt;/a&gt;. I haven't checked it myself, however.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-12-07T14:08:24.490" CommentCount="1" />
  <row Id="5213" PostTypeId="1" AcceptedAnswerId="5221" CreationDate="2010-12-07T14:30:19.860" Score="6" ViewCount="151" Body="&lt;p&gt;Suppose I observe a sample $(y_i,x_i)$, $i=1,...,n$. Suppose that I know the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$y_i=\alpha_0+\alpha_1x_i+\varepsilon_i$, $i \in J\subset\{1,...,n\}$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$y_i=\beta_0+\beta_1x_i+\varepsilon_i$, $i \in J^c$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\varepsilon_i$ are i. i. d. and $J$ is not known in advance. Is it possible to estimate $\alpha_0,\alpha_1,\beta_0,\beta_1$? Or at least test the hypothesis that $J=\varnothing$?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If $J$ is known the problem is very easy to solve. Going through all the subsets is not feasible, since we have $2^n$ possible combinations. If we assume $J=\{1,...,k\}$ with unknown $k=1,...,n$, it is the classical change-point problem, for which many tests are available. I suspect that this maybe ill-posed problem, so I wanted to check before trying to solve it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a simple illustration of the problem:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;N &amp;lt;- 200&#xA;s1 &amp;lt;- sample(1:N,N %/% 2)&#xA;s2 &amp;lt;- (1:N)[!(1:N) %in% s1]&#xA;&#xA;x &amp;lt;- rnorm(N)&#xA;eps &amp;lt;- rnorm(N)&#xA;&#xA;ind &amp;lt;- 1:N&#xA;&#xA;y &amp;lt;- rep(NA,N*T)&#xA;y[ind %in% s1] &amp;lt;- 2+0.5*x[ind %in% s1]+eps[ind %in% s1]/5&#xA;y[ind %in% s2] &amp;lt;- 1+1*x[ind %in% s2]+eps[ind %in% s2]/5&#xA;y&#xA;&#xA;sal1 &amp;lt;- ind %in% s1&#xA;&#xA;plot(x, y)&#xA;points(x[sal1], y[sal1], col=2)&#xA;abline(2, 0.5, col=2)&#xA;abline(1, 1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Graphically it is more or less obvious that we have two different models. Maybe it is possible to use some classification or data-mining techniques for solving this problem?&lt;/p&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2010-12-07T14:46:48.717" LastActivityDate="2010-12-07T21:03:29.853" Title="Discerning between two different linear regression models in one sample" Tags="&lt;regression&gt;&lt;classification&gt;&lt;data-mining&gt;" AnswerCount="2" />
  <row Id="5214" PostTypeId="1" AcceptedAnswerId="5216" CreationDate="2010-12-07T15:32:41.797" Score="1" ViewCount="164" Body="&lt;p&gt;I have left censored data where the distribution is known (it's near enough lognormal, at least in theory).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd like to calculate some simple summary stats: geometric mean and standard deviation in this case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've previously used R's &lt;code&gt;NADA&lt;/code&gt; package for this but it is no longer on CRAN.  Is there an alternative available?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I contacted Lopaka Lee, the package maintainer, and he says that he's &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;working on updating the package for&#xA;  the upcoming release of R&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;so hopefully the package's absence is only temporary.&lt;/p&gt;&#xA;" OwnerUserId="478" LastEditorUserId="478" LastEditorDisplayName="" LastEditDate="2010-12-07T18:28:04.907" LastActivityDate="2010-12-07T18:28:04.907" Title="How do you calculate simple statistics for left censored data in R?" Tags="&lt;r&gt;&lt;censoring&gt;" AnswerCount="1" CommentCount="0" />
  <row Id="5216" PostTypeId="2" ParentId="5214" CreationDate="2010-12-07T15:54:24.510" Score="4" ViewCount="" Body="&lt;p&gt;I will let other suggest better alternatives to &lt;a href=&quot;http://www.practicalstats.com/nada/nada/downloads_files/NADAforR_Examples.pdf&quot; rel=&quot;nofollow&quot;&gt;NADA&lt;/a&gt;, but it seems the package is still available on CRAN, in the &lt;a href=&quot;http://cran.r-project.org/src/contrib/Archive&quot; rel=&quot;nofollow&quot;&gt;Archive&lt;/a&gt; section. The last version is from May, 2009.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Installation went fine for me, using &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;R CMD install NADA_1.5-2.tar.gz&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Under Windows, I guess you can just download the tgz and use built-in install facilities.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-12-07T15:54:24.510" CommentCount="3" />
  <row Id="5217" PostTypeId="2" ParentId="5187" CreationDate="2010-12-07T15:57:53.917" Score="1" ViewCount="" Body="&lt;p&gt;The contents of your chains are stored in three different formats.  Take a look at&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;bugs.sim$sims.array&#xA;bugs.sim$sims.list&#xA;bugs.sim$sims.matrix&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and read the Value section of &lt;code&gt;?bugs&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="478" LastActivityDate="2010-12-07T15:57:53.917" />
  <row Id="5218" PostTypeId="2" ParentId="4830" CreationDate="2010-12-07T16:08:40.270" Score="2" ViewCount="" Body="&lt;p&gt;In logistic regression, highly skewed distributions of outcome variables (where there are far more non-events to events or vis versa), the cut point or probability trigger does need to be adjusted, but it will not have much of an effect on overall classification efficieny.  This will always remain roughly the same, but you are currently under-classifying events since the &quot;chance&quot; probability in such a data set will always make you more likely to classify into non-events.  This needs to be adjusted for.  In fact, in such a situation it's not uncommon to see the overall efficiency of classification go down,since it was previously inflated by misscalculation due to chance.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Think of it this way, if you have an event where 90% don't do it and 10% do it, then if you put everyone into the &quot;don't do it&quot; group, you automatically get 90% right, and that was without even trying, just pure chance, inflated by the skewness of it's distribution.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The issue of interactions is unrelated to this skewing, and should be driven by theory.  You will most likely always improve classification by adding additional terms, including simply adding interactions, but you do so by often overfitting the model. You then have to go back and be able to interpret this.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Matt P&#xA;Data Analyst, University of Illinois Urbana Champaign&lt;/p&gt;&#xA;" OwnerUserId="2287" LastActivityDate="2010-12-07T16:08:40.270" />
  <row Id="5219" PostTypeId="2" ParentId="4830" CreationDate="2010-12-07T16:22:15.807" Score="0" ViewCount="" Body="&lt;p&gt;deleted, wouldn't have worked for your case, sorry, wasn't thinking.&#xA;-matt&lt;/p&gt;&#xA;" OwnerUserId="2287" LastEditorUserId="2287" LastEditorDisplayName="" LastEditDate="2010-12-07T16:30:00.257" LastActivityDate="2010-12-07T16:30:00.257" />
  <row Id="5220" PostTypeId="1" CreationDate="2010-12-07T16:37:20.697" Score="1" ViewCount="544" Body="&lt;p&gt;I have done a meta analysis and heterogeneity is too high.  I am working with (even/Total) for experimental and control groups to calculate the Odds Ratio.   I have done fixed-effect and random-effect modeling.  I now need to use meta-regression via SPSS.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone direct me to a good set of materials to learn how to do this?  I have read some links but none has really solved this question.&#xA;Thanks&lt;/p&gt;&#xA;" OwnerUserId="2288" LastEditorUserId="307" LastEditorDisplayName="" LastEditDate="2010-12-07T21:11:28.573" LastActivityDate="2011-04-22T01:22:57.970" Title="How to do meta-regression analysis with SPSS?" Tags="&lt;regression&gt;&lt;spss&gt;&lt;meta-analysis&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="1" />
  <row Id="5221" PostTypeId="2" ParentId="5213" CreationDate="2010-12-07T17:01:09.163" Score="5" ViewCount="" Body="&lt;p&gt;You need to model the observations as a mixture model. Define:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$p$ as the probability that a sample belongs to the first data generating process.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, the density function of $y_i$ is given by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$f(y_i|-) \sim p f_1(y_i|-) + (1-p) f_2(y_i|-)$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$f_1(.)$ is the density that arises because of the first data generating process and&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$f_2(.)$ is the density that arises because of the second data generating process.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can then either use maximum likelihood (see for example the &lt;a href=&quot;http://en.wikipedia.org/wiki/Expectation-maximization_algorithm&quot; rel=&quot;nofollow&quot;&gt;EM algorithm&lt;/a&gt;) or bayesian approaches to estimate the model.&lt;/p&gt;&#xA;" LastActivityDate="2010-12-07T17:01:09.163" CommentCount="1" />
  <row Id="5222" PostTypeId="2" ParentId="5220" CreationDate="2010-12-07T17:57:35.953" Score="7" ViewCount="" Body="&lt;p&gt;You could start with David B Wilson's website on &quot;&lt;a href=&quot;http://mason.gmu.edu/~dwilsonb/ma.html&quot; rel=&quot;nofollow&quot;&gt;meta-analysis stuff&lt;/a&gt;&quot;. He offers spss, stata, and sas macros for performing meta-analytic analyses (including meta-regression; metareg.sps) + PPT slides (analysis.ppt, interpretation.ppt).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another presentation I really like(d) was given by Marsh et al. „&lt;a href=&quot;http://www.self.ox.ac.uk/rdimaterials.htm&quot; rel=&quot;nofollow&quot;&gt;Meta-Analysis: Session 3.3 &amp;amp; 3.4: Teacher Expectancy Example&lt;/a&gt;”  (see &quot;Practical example - fixed, random, &amp;amp; multilevel Meta20 data&quot;). Unfortunately, the presentation seems to be no longer available... But you might want to check the &lt;a href=&quot;http://www.self.ox.ac.uk/addit.materials.htm&quot; rel=&quot;nofollow&quot;&gt;other resources&lt;/a&gt; (see esp. &quot;Advanced Meta-Anaylsis Seminar Presentations&quot;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, you can find a presentation on &quot;&lt;a href=&quot;http://www.campbellcollaboration.org/artman2/uploads/1/Pigott_random_mixed_effects.pdf&quot; rel=&quot;nofollow&quot;&gt;Random and Mixed-effects Modeling&lt;/a&gt;&quot; on the website of the Campbell Collaboration.&lt;/p&gt;&#xA;" OwnerUserId="307" LastEditorUserId="307" LastEditorDisplayName="" LastEditDate="2010-12-08T12:33:37.117" LastActivityDate="2010-12-08T12:33:37.117" CommentCount="1" />
  <row Id="5223" PostTypeId="2" ParentId="5077" CreationDate="2010-12-07T18:19:10.860" Score="3" ViewCount="" Body="&lt;p&gt;I didn't look at the paper you supplied, but let me have a go anyway:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have a $p$-dimensional parameter space you can generate a random direction $d$ uniformly distributed on the surface of the unit sphere with&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x &amp;lt;- rnorm(p)&#xA;d &amp;lt;- x/sqrt(sum(x^2))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(c.f. &lt;a href=&quot;http://en.wikipedia.org/wiki/Hypersphere#Generating_random_points&quot; rel=&quot;nofollow&quot;&gt;Wiki&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, use this to generate proposals for $d$ for rejection sampling (assuming you can actually evaluate the distribution for $d$). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming you start in position $x$ and have accepted a $d$, generate a proposal $y$ with&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; lambda &amp;lt;- r&amp;lt;SOMEDISTRIBUTION&amp;gt;(foo, bar)&#xA; y &amp;lt;- x + lambda * d&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and do a Metropolis-Hastings-Step to decide whether to move to $y$ or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, how well this can work will depend on the distribution of $d$ and how expensive it is to (repeatedly) evaluate its density in the rejection sampling step, but since generating proposals for $d$ is cheap you may get away with it.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Added for @csgillespie's benefit:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From what I was able to gather by some googling, hit-and-run MCMC is useful primarily for fast mixing if you have a (multivariate) target that has arbitrary and not necessarily connected support, because it enables you to move from any point in the support to any other in one step. More &lt;a href=&quot;http://books.google.com/books?id=1-ffZVmazvwC&amp;amp;pg=PA173&amp;amp;dq=hit-and-run++MCMC&amp;amp;hl=de&amp;amp;ei=jL0ATdG3JozLswbOwfzyDg&amp;amp;sa=X&amp;amp;oi=book_result&amp;amp;ct=result&amp;amp;resnum=2&amp;amp;ved=0CCwQ6AEwAQ#v=onepage&amp;amp;q=hit-and-run%20%20MCMC&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="1979" LastEditorUserId="1979" LastEditorDisplayName="" LastEditDate="2010-12-09T11:49:13.383" LastActivityDate="2010-12-09T11:49:13.383" CommentCount="4" />
  <row Id="5224" PostTypeId="1" CreationDate="2010-12-07T19:53:42.400" Score="1" ViewCount="76" Body="&lt;p&gt;Hello,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am interested in recent papers or books on image compression using Machine Learning-based techniques. I know about &lt;a href=&quot;http://www.google.ro/url?sa=t&amp;amp;source=web&amp;amp;cd=3&amp;amp;ved=0CCUQFjAC&amp;amp;url=http%3A%2F%2Fciteseerx.ist.psu.edu%2Fviewdoc%2Fdownload%3Fdoi%3D10.1.1.13.6973%26rep%3Drep1%26type%3Dpdf&amp;amp;rct=j&amp;amp;q=image%20compression%20with%20neural%20networks%20a%20survey&amp;amp;ei=1Yb-TIHXOo-r8QP2npGNCw&amp;amp;usg=AFQjCNFh_lTg5qw9UJN1_fwvhwWK37J7ng&amp;amp;cad=rja&quot; rel=&quot;nofollow&quot;&gt;Jiang's survey on Image compression using neural networks&lt;/a&gt;, but this article seems a little outdated (1999). Do you know some more recent resources (articles, books) that treat about this subject?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks,&#xA;Lucian&lt;/p&gt;&#xA;" OwnerUserId="976" LastActivityDate="2010-12-07T19:53:42.400" ClosedDate="2010-12-08T09:15:39.710" Title="Image compression by ML techniques" Tags="&lt;machine-learning&gt;&lt;compression&gt;" CommentCount="1" />
  <row Id="5225" PostTypeId="2" ParentId="5213" CreationDate="2010-12-07T21:03:29.853" Score="3" ViewCount="" Body="&lt;p&gt;The first hit on &lt;a href=&quot;http://Rseek.org&quot; rel=&quot;nofollow&quot;&gt;Rseek&lt;/a&gt; with keywords &quot;mixture regression&quot; brings up the &lt;code&gt;flexmix&lt;/code&gt; package, which does what you want. I seem to recall that there were other packages for this as well.&lt;/p&gt;&#xA;" OwnerUserId="279" LastActivityDate="2010-12-07T21:03:29.853" CommentCount="1" />
  <row Id="5226" PostTypeId="1" AcceptedAnswerId="5231" CreationDate="2010-12-08T00:13:18.207" Score="4" ViewCount="113" Body="&lt;p&gt;If $X=[x_1,x_2,...,x_n]^T$ is an $n$-dimensional random variable and we have &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$E\left\{X\right\} = M = \left[m_1,m_2,...,m_n\right]^T$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Cov\left\{X\right\} = \Sigma = diag\left(\lambda_1,\lambda_2,...,\lambda_n\right)$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;how can I express the following expectation in terms of $M$, $\Sigma$, and $n$ (and maybe raw $m_i$'s and $\lambda_i$'s)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$E\left\{ \left(X-M\right)^T\left(X-M\right)\left(X-M\right)^T\left(X-M\right)\right\}$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Supposing $x_i$'s are i.i.d and have normal distribution would be acceptable, but are these assumptions necessary?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;I know that $E\left\{ \left(X-M\right)^T\left(X-M\right)\right\} = \sum_{i=1}^n \left(\lambda_i\right)$ but don't think this would help in this case.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;In the section &lt;del&gt;&lt;em&gt;6.2.3 Cubic Forms&lt;/em&gt;&lt;/del&gt; &lt;em&gt;8.2.4 Quartic Forms&lt;/em&gt; of &lt;a href=&quot;http://matrixcookbook.com/&quot; rel=&quot;nofollow&quot;&gt;Matrix cookbook&lt;/a&gt; there is a formula for calculated quadratic expectations like this, but i don't want just a formula to solve it. I think there should be a simple question for this problem because the covariance matrix is diagonalized. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="2148" LastEditorUserId="2148" LastEditorDisplayName="" LastEditDate="2010-12-08T14:35:35.940" LastActivityDate="2010-12-08T14:35:35.940" Title="Expectation of $\left(X-M\right)^T\left(X-M\right)\left(X-M\right)^T\left(X-M\right)$" Tags="&lt;expected-value&gt;" AnswerCount="3" CommentCount="4" />
  <row Id="5227" PostTypeId="2" ParentId="5226" CreationDate="2010-12-08T01:02:29.333" Score="1" ViewCount="" Body="&lt;p&gt;I believe this depends on the kurtosis of $X$. If I am reading this correctly, and assuming the $X_i$ are independent, you are trying to find the expectation of $\sum_i (X_i - m_i)^4$. Because $X_i^4$ appears, you cannot find this expectation in terms of $M$ and $\Sigma$ without making further assumptions. (Even without the independence of the $X_i$, you will have $E[X_i^4]$ terms in your expectation.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you assume that the $X_i$ are normally distributed, you should find the expectation is equal to $3 \sum_i \lambda_i^2$.&lt;/p&gt;&#xA;" OwnerUserId="795" LastEditorUserId="795" LastEditorDisplayName="" LastEditDate="2010-12-08T03:03:57.627" LastActivityDate="2010-12-08T03:03:57.627" />
  <row Id="5228" PostTypeId="1" AcceptedAnswerId="5232" CreationDate="2010-12-08T01:21:07.327" Score="4" ViewCount="236" Body="&lt;p&gt;I am looking at the sample kurtosis of a fairly skewed random variable, and the results seem inconsistent. To simply illustrate the problem, I looked at the sample kurtosis of a log-normal RV. In R (which I am slowly learning): &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(moments); &#xA;&#xA;samp_size = 2048;&#xA;n_trial = 4096;&#xA;&#xA;kvals &amp;lt;- rep(NA,1,n_trial); #preallocate&#xA;for (iii in 1:n_trial) {&#xA;    kvals[iii] &amp;lt;- kurtosis(exp(rnorm(samp_size)));&#xA;}&#xA;print(summary(kvals));&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The summary I get is&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. &#xA;  11.87   28.66   39.32   59.17   61.70 1302.00 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;According to &lt;a href=&quot;http://en.wikipedia.org/wiki/Log_normal&quot; rel=&quot;nofollow&quot;&gt;Wikipedia&lt;/a&gt;, the kurtosis for this log-normal RV should be around 114. Clearly the sample kurtosis is biased.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Doing some research I found that sample kurtosis is biased for small sample sizes. I used the 'G2' estimator as provided by the &lt;code&gt;e1071&lt;/code&gt; package in CRAN, and got very similar results for this sample size. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The question&lt;/strong&gt;: which of the following characterize what is going on:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The standard error of the sample kurtosis is simply very large for this RV (even though the hand-wavey common estimate of the standard error is of order $1/\sqrt{n}$). Alternatively, I used too few samples (2048) in this study.&lt;/li&gt;&#xA;&lt;li&gt;These implementations of sample kurtosis suffer from numerical problems which might be corrected by &lt;em&gt;e.g.&lt;/em&gt; &lt;a href=&quot;http://people.xiph.org/~tterribe/notes/homs.html&quot; rel=&quot;nofollow&quot;&gt;Terriberry's method&lt;/a&gt; (in much the same way that Welford's method gives better results than the naive method for sample variance).&lt;/li&gt;&#xA;&lt;li&gt;I computed the population kurtosis incorrectly. (ouch)&lt;/li&gt;&#xA;&lt;li&gt;Sample kurtosis is inherently biased and you can never fix it for such small sample sizes.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="795" LastEditorUserId="1390" LastEditorDisplayName="" LastEditDate="2010-12-08T08:42:18.133" LastActivityDate="2010-12-08T08:42:18.133" Title="Is sample kurtosis hopelessly biased?" Tags="&lt;r&gt;&lt;unbiased-estimator&gt;&lt;kurtosis&gt;" AnswerCount="2" CommentCount="3" />
  <row Id="5229" PostTypeId="2" ParentId="3466" CreationDate="2010-12-08T01:31:49.233" Score="5" ViewCount="" Body="&lt;p&gt;Daniel B. Wright discusses this in section 5 of his article &lt;a href=&quot;http://www2.fiu.edu/~dwright/pdf/makefriends.pdf&quot; rel=&quot;nofollow&quot;&gt;Making Friends with your Data&lt;/a&gt;.&#xA;He suggests (p.130):&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The only procedure that&#xA;  is always correct in this situation is&#xA;  a scatterplot comparing the scores at&#xA;  time 2 with those at time 1 for the&#xA;  different groups. In most cases you&#xA;  should analyse the data in several&#xA;  ways. If the approaches give different&#xA;  results ... think more&#xA;  carefully about the model implied by&#xA;  each.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;He recommends the following articles as further reading:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Hand, D. J. (1994). Deconstructing statistical questions. Journal of the Royal Statistical Society: A, 157, 317–356.&lt;/li&gt;&#xA;&lt;li&gt;Lord, F. M. (1967). A paradox in the interpretation of group comparisons. Psychological Bulletin, 72, 304–305. &lt;a href=&quot;http://isites.harvard.edu/fs/docs/icb.topic477909.files/Lords_Paradox.pdf&quot; rel=&quot;nofollow&quot;&gt;Free PDF&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Wainer, H. (1991). Adjusting for differential base rates: Lord’s paradox again. Psychological Bulletin, 109, 147–151. &lt;a href=&quot;http://dionysus.psych.wisc.edu/Lit/Articles/WainerH1991a.pdf&quot; rel=&quot;nofollow&quot;&gt;Free PDF&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-12-25T06:55:23.157" LastActivityDate="2010-12-25T06:55:23.157" CommentCount="1" />
  <row Id="5231" PostTypeId="2" ParentId="5226" CreationDate="2010-12-08T04:20:55.597" Score="3" ViewCount="" Body="&lt;p&gt;Because $\left(X-M\right)^T\left(X-M\right) = \sum_i{(X_i - m_i)^2}$,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\left(X-M\right)^T\left(X-M\right)\left(X-M\right)^T\left(X-M\right) = \sum_{i,j}{(X_i - m_i)^2(X_j - m_j)^2} \text{.}$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are two kinds of expectations to obtain here.  Assuming the $X_i$ are independent and $i \ne j$, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\eqalign{&#xD;&#xA;E \left[ (X_i - m_i)^2(X_j - m_j)^2 \right] &amp;amp;= E\left[(X_i - m_i)^2\right] E\left[(X_j - m_j)^2\right] \cr&#xD;&#xA;&#xD;&#xA;&amp;amp;= \lambda_i \lambda_j .&#xD;&#xA;}$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When $i = j$,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\eqalign{&#xD;&#xA;E \left[ (X_i - m_i)^2(X_j - m_j)^2 \right] &amp;amp;= E\left[(X_i - m_i)^4\right] \cr&#xD;&#xA;&#xD;&#xA;&amp;amp;= 3 \lambda_i^2 \text{ for Normal variates} \cr&#xD;&#xA;&#xD;&#xA;&amp;amp;= \lambda_i \lambda_j + 2 \lambda_i^2 \text{.}&#xD;&#xA;}$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Whence the expectation equals&#xA;$$\eqalign{&#xD;&#xA;&amp;amp;\sum_{i, j} {\lambda_i \lambda_j} + 2 \sum_{i} {\lambda_i^2} \cr&#xD;&#xA;= &amp;amp;(\sum_{i}{\lambda_i})^2 + 2 \sum_{i} {\lambda_i^2}.&#xD;&#xA;}$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note where the assumptions of independence and Normality come in.  Minimally, we need to assume the &lt;em&gt;squares&lt;/em&gt; of the residuals are &lt;em&gt;mutually&lt;/em&gt; independent and we only need a formula for the central fourth moment; Normality is not necessasry.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-12-08T04:20:55.597" CommentCount="1" />
  <row Id="5232" PostTypeId="2" ParentId="5228" CreationDate="2010-12-08T04:51:09.113" Score="5" ViewCount="" Body="&lt;p&gt;There's a &lt;a href=&quot;http://www.mathworks.com/help/toolbox/stats/kurtosis.html&quot; rel=&quot;nofollow&quot;&gt;bias correction&lt;/a&gt;.  It's not huge.  I believe the sampling variance of the kurtosis is proportional to the &lt;em&gt;eighth&lt;/em&gt; (!) central moment, which can be enormous for a lognormal distribution.  You would need millions of trials (or far more) in a simulation to detect bias unless the CV is tiny.  (Plot a histogram of kvals to see how extraordinarily skewed they are.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The correct kurtosis is indeed about 113.9364.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as R style goes, it can be convenient to encapsulate the simulation in a function so you can easily modify the sample size or number of trials. &lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-12-08T04:51:09.113" CommentCount="1" />
  <row Id="5233" PostTypeId="1" AcceptedAnswerId="5236" CreationDate="2010-12-08T05:22:59.760" Score="4" ViewCount="163" Body="&lt;p&gt;Say I have a database of around a million words, and I want to get an intuitive idea about exactly how a particular, quite infrequent, word is distributed throughout this data. My goal is to be able to see clearly whether this word tends to cluster together, or whether it is relatively evenly spaced. What would be some good methods for visualizing this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, I have seen something that looks useful. It's basically a strip (long rectangle) in which each instance of something is represented by a very thin red vertical line. The problem is that I don't know what these are called, and therefore I can't figure out how to make something like this in R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help finding the right R function for that, or any other suggestions for good ways to visualize this sort of data, would be most appreciated.&lt;/p&gt;&#xA;" OwnerUserId="52" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-12-08T05:33:36.847" LastActivityDate="2010-12-08T19:43:07.323" Title="Visualizing the distribution of something within a very large body of data" Tags="&lt;r&gt;&lt;distributions&gt;&lt;data-visualization&gt;" AnswerCount="4" CommentCount="1" />
  <row Id="5234" PostTypeId="2" ParentId="5233" CreationDate="2010-12-08T05:32:42.603" Score="3" ViewCount="" Body="&lt;p&gt;With a 1200 dpi printer using the thinnest possible line (one pixel) for each word, your plot of a million words would still be almost 20 meters long!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maybe a &lt;a href=&quot;http://www.statmethods.net/graphs/density.html&quot; rel=&quot;nofollow&quot;&gt;density plot&lt;/a&gt; would be more helpful.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-12-08T05:32:42.603" CommentCount="3" />
  <row Id="5235" PostTypeId="1" AcceptedAnswerId="5245" CreationDate="2010-12-08T05:50:51.757" Score="0" ViewCount="649" Body="&lt;p&gt;In multiple linear regression, I can understand the correlations between residual and predictors are zero, but what is the expected correlation between residual and the criterion variable? Should it expected to be zero or highly correlated? What's the meaning of that?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="400" LastActivityDate="2011-04-23T02:29:59.547" Title="What is the expected correlation between residual and the criterion variable?" Tags="&lt;regression&gt;&lt;residuals&gt;" AnswerCount="2" CommentCount="3" FavoriteCount="1" />
  <row Id="5236" PostTypeId="2" ParentId="5233" CreationDate="2010-12-08T05:51:16.343" Score="2" ViewCount="" Body="&lt;p&gt;While Whuber is correct in principle you still might be able to see something because your word is very infrequent and you only want plots of the one word.  Something quite uncommon might only appear 30 times, probably not more than 500.&#xA;Let's say you convert your words into a single vector of words that's a million long.  You could easily construct a plot with basic R commands.  Let's call the vector 'words' and the rare item 'wretch'.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;n &amp;lt;- length(words)&#xA;plot(1:n, integer(n), type = 'n', xlab = 'index of word', ylab = '', main = 'instances of wretch', yaxt = 'n', frame.plot = TRUE)&#xA;wretch &amp;lt;- which(words %in% 'wretch')&#xA;abline(v = wretch, col = 'red', lwd = 0.2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You could change the line assigning wretch using a grep command if you need to account for variations of the word.  Also, the lwd in the abline command could be set thicker or thinner depending on the frequency of the word.  If you end up plotting 400 instances 0.2 will work fine.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried some density plots of this kind of data. I imported about 50,000 words of Shakespeare and finding patterns was easier for me in the code above than it was in the density plots.  I used a very common word that appeared in frequency 200x more than the mean frequency ('to') and the plots looked just fine.  I think you'll make a fine graph like this with rare instances in 1e6 words.&lt;/p&gt;&#xA;" OwnerUserId="601" LastEditorUserId="601" LastEditorDisplayName="" LastEditDate="2010-12-08T16:37:08.893" LastActivityDate="2010-12-08T16:37:08.893" />
  <row Id="5237" PostTypeId="2" ParentId="5233" CreationDate="2010-12-08T06:10:45.690" Score="1" ViewCount="" Body="&lt;p&gt;Hello Alan:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't know if this may be useful in your case, but in bioinformatics I often feel the need to visualize the distribution of gene counts in a give data set. This is definitely not as large as your data set, but I think the strategy can be followed for most of the large data sets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A typical strategy would be to find a predetermined number of clusters using, say, hierarchical clustering (or any other clustering procedure). Once you have the clusters, you can sample a gene from each of these clusters. Assuming that the gene is representative of the cluster, visualizing the count for the gene (in form of density plot, histogram, qq-plot, etc.) is equivalent to visualizing the behavior of the cluster. You can do the same for all the clusters. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically, you reduce the huge data set to clusters and then visualize the representatives from these clusters assuming &quot;on an average&quot; the clusters' behavior will remain &quot;more or less&quot; the same.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Warning&lt;/strong&gt;: This method is highly sensitive to a lot of things, a few are, clustering method, how many clusters you choose, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I believe visualizing all the words if the number of words is reasonably large (say $\geq$ 50) would be pretty difficult. As as whuber aptly points out, it may be almost impossible.&lt;/p&gt;&#xA;" OwnerUserId="1307" LastActivityDate="2010-12-08T06:10:45.690" />
  <row Id="5238" PostTypeId="1" CreationDate="2010-12-08T07:14:29.000" Score="4" ViewCount="341" Body="&lt;p&gt;When I work on data analysis projects I often store data in comma or tab-delimited (CSV, TSV) data files. While data often belongs in a dedicated database management system. &#xA;For many of my applications, this would be overdoing things.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can edit CSV and TSV files in Excel (or presumably another Spreadsheet program).&#xA;This has benefits:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;spreadsheets make it easy to enter data&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;There are also several problems:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Working with CSV and TSV files leads to a wide range of warning messages about various features being lost and how only the active sheet will be saved and so forth. Thus, it's annoying if you just want to open the file and make a little change.&lt;/li&gt;&#xA;&lt;li&gt;It does many &quot;supposedly intelligent&quot; conversions. &#xA;For example, if you enter 12/3, it will think that you want to enter a date.&#xA;&lt;strong&gt;UPDATE:&lt;/strong&gt; I should have mentioned that the date example is just one of many examples;&#xA;most problems seem to be related to inappropriate conversion.&#xA;In particular, text fields that look like numbers or dates cause problems.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Alternatively, I could work directly with the text file&#xA;in a standard text editor. This ensures that what I enter is what is recorded. &#xA;However it is a very awkward way to enter data &#xA;(columns don't line up; &#xA;it's difficult to enter data simply into multiple cells; etc.).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Question&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What is a good strategy for working with CSV or TSV data files?&#xA;i.e., what strategy makes it easy to enter and manipulate the data while also&#xA;ensuring that what you enter is actually interpreted correctly?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-12-08T10:46:35.740" LastActivityDate="2010-12-08T17:40:57.860" Title="Strategy for editing comma separated value (CSV) files" Tags="&lt;csv-file&gt;" AnswerCount="6" CommentCount="2" FavoriteCount="1" />
  <row Id="5239" PostTypeId="2" ParentId="5238" CreationDate="2010-12-08T07:31:04.013" Score="3" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Update:&lt;/strong&gt; [Having been going through a large backlog of email from R-Help] I am reminded of the thread on &quot;&lt;a href=&quot;http://thread.gmane.org/gmane.comp.lang.r.general/213174/focus=213179&quot; rel=&quot;nofollow&quot;&gt;The behaviour of &lt;code&gt;read.csv()&lt;/code&gt;&lt;/a&gt;&quot;. In this, Duncan Murdoch mentions that he prefers to use &lt;a href=&quot;http://en.wikipedia.org/wiki/Data_Interchange_Format&quot; rel=&quot;nofollow&quot;&gt;Data Interchange Format (DIF)&lt;/a&gt; files instead of csv for some of the reason Jeromy mentions. I just tried this and Gnumeric gets it wrong (loading 12/3 as a date), but OpenOffice.org reads this correctly and preserves the 12/3 information intact. (Anyone care to check this in MS Excel?)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;DIF files are plain text and can be read by spreadsheets and R (as long as you use a recent R revision (SVN revision &gt;= r53778)) will read the data in in the correct format.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Original&lt;/strong&gt;: I would try to avoid using a spreadsheet full stop for data editing / manipulation whenever possible. It is incredibly difficult, if not impossible, to document any changes you make to an existing data set so that pretty much rules it out from a reproducible research point of view. At most, I use a spreadsheet to quickly view existing data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For data processing, I tend to write an R script that will take the raw csv file and apply all the necessary processing steps required. I heavily comment that script to explain exactly what I am doing at each stage and why. My data analysis script would then call the data processing script which loads and processes the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For data entry, is it more hassle to enter the data in a text editor or in a spreadsheet? I suspect the problems you mention for the latter do not outweigh those of trying to enter CSV data into a text editor.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could try a better spreadsheet; OpenOffice.org refuses to stop formatting 12/3 as a date (or it converts it to the numeric representation) even if one formats the column as &quot;numeric&quot; first. Gnumeric on the other hand will leave 12/3 as it is &lt;strong&gt;if&lt;/strong&gt; you format the column as &quot;numeric&quot; first.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can force OpenOffice.org to not reformat 12/3 as a date by prepending a &lt;code&gt;'&lt;/code&gt; to the entries, i.e. &lt;code&gt;'12/3&lt;/code&gt; will get displayed as 12/3 in the spreadsheet and saved out as text. This is probably quite safe to use.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Not sure why you would want 12/3 stored numerically as 12/3 in the text file - how should something like R read this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your comment on warnings about losing features or only saving the active sheet aren't really problems are they? (If they are, then I want your problems in my life ;-)&lt;/p&gt;&#xA;" OwnerUserId="1390" LastEditorUserId="1390" LastEditorDisplayName="" LastEditDate="2010-12-08T10:35:34.760" LastActivityDate="2010-12-08T10:35:34.760" CommentCount="8" />
  <row Id="5240" PostTypeId="2" ParentId="4991" CreationDate="2010-12-08T07:40:03.023" Score="1" ViewCount="" Body="&lt;p&gt;Have a look at the &lt;a href=&quot;http://cran.r-project.org/web/packages/dlm/index.html&quot; rel=&quot;nofollow&quot;&gt;dlm&lt;/a&gt; package and its &lt;a href=&quot;http://cran.r-project.org/web/packages/dlm/vignettes/dlm.pdf&quot; rel=&quot;nofollow&quot;&gt;vignette&lt;/a&gt;. I think you might find what you are looking for from the vignette. The package authors have also written a book &lt;a href=&quot;http://amzn.com/0387772375&quot; rel=&quot;nofollow&quot;&gt;Dynamic Linear Models with R&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="214" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-12-14T21:58:57.510" LastActivityDate="2010-12-14T21:58:57.510" CommentCount="4" />
  <row Id="5241" PostTypeId="2" ParentId="5238" CreationDate="2010-12-08T07:51:55.477" Score="8" ViewCount="" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;If you are comfortable with R, you can create your basic data.frame and then use the fix() function on it to input data.  Along the same line as #5, once you set up the data.frame you can use a series of readLines(n=1) (or whatever) to get your data in, validate it, and the provide the opportunity to add the next row.  Then leave the fixing to fix().  See an implemented example below using scan().&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Another option in excel would be messy, but you could type in 12/9, then have another column evaluate =IFERROR(MONTH(DateEntryCell)/DAY(DataEntryCell),DataEntryCell).  But then you'll have to maintain the excel sheet AND the csv sheet and all of the complaining as you write the csv will persist.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;Alternatively, so long as your fields are relatively short and have a consistent length a regular text editor should serve you well with TSV.  You can always load it up in excel when you are done and make sure the number of columns for each row is what you expect it to be.&lt;/li&gt;&#xA;&lt;li&gt;Emacs is available on a number of platforms and probably has something just for this, e.g. &lt;a href=&quot;http://www.emacswiki.org/emacs/CsvMode&quot; rel=&quot;nofollow&quot;&gt;http://www.emacswiki.org/emacs/CsvMode&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;li&gt;If you are a hearty soul, programming something quick up in a programming language to do the data entry is trivial, the data editing will be a lot harder.&lt;/li&gt;&#xA;&lt;li&gt;A quick google search shows software with just this purpose, but no free software seemed to be any good.&lt;/li&gt;&#xA;&lt;li&gt;It sounds insane, but someone on superuser suggested editing tables in access and then exporting them as CSV... that is just crazy enough to work.&lt;/li&gt;&#xA;&lt;li&gt;It doesn't stop excel from complaining as you save as .csv, but you can type a single apostrophe before your data entry field and that makes it leave it alone in terms of auto-formatting.  Nicely, this (in Office 2007 at least) doesn't leave apostrophes in the csv file.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update:&lt;/strong&gt;&#xA;I've been poking around a lot on this problem because it is an issue I also have. So far the best/easiest solution for data-entry I've seen so far is &lt;a href=&quot;http://www.whitepeaksoftware.com/main/killink-csv-editor&quot; rel=&quot;nofollow&quot;&gt;KillinkCSV&lt;/a&gt;. It isn't &quot;free&quot; software, it is shareware with a 30 day trial duration and a reasonable price (~$27).  I'm not sure how much I trust it for editing existing CSVs though - I handed it an insanely large (and presumably well formatted) CSV and it failed to read all of the rows.  However it seemed to work well for one that was reasonably large (20 MB) and the problem with the larger file may be user error on my part.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;R Example:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#This function takes a what argument like in scan, a list with the types to be used, see usage example at the end of this code block&#xA;#data.entry will keep reading in values until the values it reads in matches what is in &quot;terminateon&quot;.&#xA;#limitations: Many - but notably, this tells you the cell AFTER you have a data entry error and the error messages are messy.&#xA;data.entry &amp;lt;- function(what,terminateon)&#xA;{&#xA;    CONTINUE &amp;lt;- TRUE #Make sure we start the loop&#xA;    data &amp;lt;- NULL #Create empty data so that the data.frame can define itself&#xA;    ti &amp;lt;- NULL&#xA;    while(CONTINUE)&#xA;    {&#xA;        ti &amp;lt;- NULL&#xA;        err &amp;lt;- NULL&#xA;        err &amp;lt;- try({ti &amp;lt;- as.data.frame(scan(what=what, n=length(what)))},silent=TRUE) #Try getting the data according to the parameters in 'what' one row at a time.&#xA;        if(is.null(ti)) #if the previous expression failed, ti will still be NULL, so an error occured, report it and move on.&#xA;        {&#xA;            print(&quot;Error in data entry!&quot;)&#xA;            print(err)&#xA;        } else&#xA;        {&#xA;            data &amp;lt;- rbind(data,ti) #If there wasn't an error, add ti to the previous value&#xA;        }&#xA;        if (!is.null(ti) &amp;amp; all(ti == terminateon)) &#xA;            {&#xA;                CONTINUE &amp;lt;- FALSE&#xA;                data &amp;lt;- data[-c(nrow(data)),]&#xA;            } #if we've recieved the final value we won't continue and the last row is invalid so we remove it&#xA;    }&#xA;    return(data)&#xA;}&#xA;&#xA;data.entry(list(x=integer(), y=numeric(), z=character()),terminateon=c(999,999,&quot;Z&quot;))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="196" LastEditorUserId="196" LastEditorDisplayName="" LastEditDate="2010-12-08T17:40:57.860" LastActivityDate="2010-12-08T17:40:57.860" CommentCount="3" />
  <row Id="5242" PostTypeId="2" ParentId="5228" CreationDate="2010-12-08T08:41:48.057" Score="4" ViewCount="" Body="&lt;p&gt;[Just on the R Style - @whuber has answered the Kurtsosis Q]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This was a bit too complicated to stick into a comment. For such simple loops like the one you use, we can combine @whuber's suggestion of encapsulating the simulation in a function with the &lt;code&gt;replicate()&lt;/code&gt; function. &lt;code&gt;replicate()&lt;/code&gt; takes care of allocation and running the loop for you. An example is given below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(moments)&#xA;foo &amp;lt;- function(size, trials, meanlog = 0, sdlog = 1) {&#xA;    replicate(trials,&#xA;              kurtosis(rlnorm(size, meanlog = meanlog, &#xA;                              sdlog = sdlog)))&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;We use it like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; set.seed(1)&#xA;&amp;gt; out &amp;lt;- foo(2048, 10000)&#xA;&amp;gt; summary(out)&#xA;   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. &#xA;  10.93   28.77   39.99   62.53   62.58 1557.00&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Note that I use the &lt;code&gt;rlnorm()&lt;/code&gt; function to generate the log-normal random variable. It is equivalent to &lt;code&gt;exp(rnorm())&lt;/code&gt; in your loop but uses the correct tool, and we allow our function to pass on user-specified parameters of the log-normal distribution.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; set.seed(123)&#xA;&amp;gt; exp(rnorm(1))&#xA;[1] 0.5709374&#xA;&amp;gt; set.seed(123)&#xA;&amp;gt; rlnorm(1)&#xA;[1] 0.5709374&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1390" LastActivityDate="2010-12-08T08:41:48.057" CommentCount="4" />
  <row Id="5243" PostTypeId="2" ParentId="5238" CreationDate="2010-12-08T09:09:34.283" Score="0" ViewCount="" Body="&lt;p&gt;I like Gnumeric because it does not try to be so much idiot-resistant as others (it doesn't shout about lost functionality) and works with large data... yet I think it is Linux-only.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-12-08T09:09:34.283" CommentCount="2" />
  <row Id="5244" PostTypeId="2" ParentId="5238" CreationDate="2010-12-08T09:09:51.760" Score="2" ViewCount="" Body="&lt;p&gt;After I asked this question, I started having a look at &lt;a href=&quot;http://csved.sjfrancke.nl/index.html&quot; rel=&quot;nofollow&quot;&gt;CSVed&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the website:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;CSVed is an easy and powerful CSV file&#xA;  editor, you can manipulate any CSV&#xA;  file, separated with any separator.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I'm not sure if anyone has experience with it.&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-12-08T09:09:51.760" CommentCount="1" />
  <row Id="5245" PostTypeId="2" ParentId="5235" CreationDate="2010-12-08T09:27:14.297" Score="2" ViewCount="" Body="&lt;p&gt;So, the residuals are your unexplained variance, the difference between your model's predictions and the actual outcome you're modeling. In practice, few models produced through linear regression will have all residuals close to zero unless linear regression is being used to analyze a mechanical or fixed process.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ideally, the residuals from your model should be random, meaning they should not be correlated with either your independent or dependent variables (what you term the criterion variable). In linear regression, your error term is normally distributed, so your residuals should also be normally distributed as well. If you have significant outliers, or If your residuals are correlated with either your dependent variable or your independent variables, then you have a problem with your model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have significant outliers and non-normal distribution of your residuals, then the outliers may be skewing your weights (Betas), and I would suggest calculating DFBETAS to check the influence of your observations on your weights. If your residuals are correlated with your dependent variable, then there is a significantly large amount of unexplained variance that you are not accounting for. You may also see this if you're analyzing repeated observations of the same thing, due to autocorrelation. This can be checked for by seeing if your residuals are correlated with your time or index variable. If your residuals are correlated with your independent variables, then your model is heteroskedastic (see: &lt;a href=&quot;http://en.wikipedia.org/wiki/Heteroscedasticity&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Heteroscedasticity&lt;/a&gt;). You should check (if you haven't already) if your input variables are normally distributed, and if not, then you should consider scaling or transforming your data (the most common kinds are log and square-root) in order to make it more normalized.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the case of both, your residuals, and your independent variables, you should take a QQ-Plot, as well as perform a Kolmogorov-Smirnov test (this particular implementation is sometimes referred to as the Lilliefors test) to make sure that your values fit a normal distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Three things that are quick and may be helpful in dealing with this problem, are examining the median of your residuals, it should be as close to zero as possible (the mean will almost always be zero as a result of how the error term is fitted in linear regression), a Durbin-Watson test for autocorrelation in your residuals (especially as I mentioned before, if you are looking at multiple observations of the same things), and performing a partial residual plot will help you look for heteroscedasticity and outliers.&lt;/p&gt;&#xA;" OwnerUserId="2166" LastEditorUserId="2166" LastEditorDisplayName="" LastEditDate="2010-12-08T09:35:02.833" LastActivityDate="2010-12-08T09:35:02.833" CommentCount="2" />
  <row Id="5246" PostTypeId="2" ParentId="5238" CreationDate="2010-12-08T09:42:47.957" Score="2" ViewCount="" Body="&lt;p&gt;Excel is not very CSV friendly. For example, if you were to enter &quot;1,300&quot; into Excel, and save that as a comma separated value, it would let you! This can be a big problem (I encounter it on a regular basis when receiving files from others).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I personally use OpenOffice.org Calc, I also use many of the solutions listed above, however many of these don't have the functionality and the ease of use that are required for regular editing. OOO Calc is much more intelligent than Excel, although being a spreadsheet program, you will still have to enter &quot;=12/3&quot; instead of &quot;12/3&quot; otherwise you will be entering a value, rather than a calculation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Give it a whirl, you won't be disappointed.&lt;/p&gt;&#xA;" OwnerUserId="2166" LastActivityDate="2010-12-08T09:42:47.957" />
  <row Id="5247" PostTypeId="1" AcceptedAnswerId="5261" CreationDate="2010-12-08T09:59:17.943" Score="3" ViewCount="130" Body="&lt;p&gt;I am going to host a training session to teach healthcare staff how to use control chart (c-chart to be specific), and I need to tell that why these rules (&lt;a href=&quot;http://en.wikipedia.org/wiki/Western_Electric_rules#Zone_rules&quot; rel=&quot;nofollow&quot;&gt;Western Electric Rules&lt;/a&gt;) are so called rules that when pattern is matched, outbreak can be considered as a probability.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have read the article &quot;&lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pubmed/11372500&quot; rel=&quot;nofollow&quot;&gt;Control Charts 101: A Guide to Health Care Applications&lt;/a&gt;&quot; a bit and it teaches me a bit, saying the chance for one data point to exceed UCL is less than 0.5%, I can calculate that using the following R code without problem:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1-pnorm(3, mean=0, sd=1)&#xA;[1] 0.001349898&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have written the following R code to verify the same thing in c-chart (i.e. poisson rather than normal distribution), for lambda ranged from 1.0 to 100.0&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;UCL &amp;lt;- integer(0)&#xA;&#xA;for (loop.UCL in (1:1000))&#xA;    {lambda &amp;lt;- loop.UCL/10&#xA;     result &amp;lt;- 1-ppois(lambda+ceiling(sqrt(lambda)*3),lambda)&#xA;     UCL &amp;lt;- c(UCL, result)&#xA;    }&#xA;&#xA;summary(UCL)&#xA;     Min.   1st Qu.    Median      Mean   3rd Qu. &#xA;0.0009679 0.0016160 0.0018260 0.0019960 0.0021220 &#xA;     Max. &#xA;0.0134600 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I think I can handle the first Western Electric Rules, but I am puzzled for the second and third.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone give me some insight or hint? Thanks!&lt;/p&gt;&#xA;" OwnerUserId="588" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-12-08T15:43:49.840" LastActivityDate="2010-12-08T16:12:12.467" Title="How to calculate the probability for pattern that violates &quot;control chart rules&quot; to occur?" Tags="&lt;r&gt;&lt;poisson&gt;&lt;control-chart&gt;" AnswerCount="1" />
  <row Id="5248" PostTypeId="2" ParentId="4989" CreationDate="2010-12-08T10:22:15.833" Score="2" ViewCount="" Body="&lt;p&gt;Looking at the &lt;a href=&quot;http://www.jstor.org/stable/full/1912352?seq=1&quot; rel=&quot;nofollow&quot;&gt;Heckman article&lt;/a&gt; I see no reason, why two selection rules cannot be applied. In the article Heckman postulates the model as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Y_{1i}=X_{1i}\beta_1+U_{1i}$&#xD;&#xA;$Y_{2i}=X_{2i}\beta_1+U_{2i}$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The goal is to estimate the first equation, but some data is missing. Which data is missing is controled by second equation. Then we have&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$E(Y_{1i}| X_{1i}, \text{sample selection rule})=X_{1i}\beta_1+E(U_{1i}|\text{sample selection rule})$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If sample selection rule is $Y_{2i}\ge 0$ we have&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$E(Y_{1i}|X_{1i},Y_{2i}\ge 0)=X_{1i}\beta_1+E(U_{1i}|U_{2i}\ge -X_{2i}\beta_2)$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And the article goes into details how this is estimated. From this I see no reason why we cannot add another sample selection rule $Y_{3i}\ge 0$, where:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Y_{3i}=X_{3i}\beta_3+U_{3i}$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then we get&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$E(Y_{1i}|X_{1i},Y_{2i}\ge 0,Y_{3i}\ge 0)=X_{1i}\beta_1+E(U_{1i}|U_{2i}\ge -X_{2i}\beta_2,U_{3i}\ge X_{3i}\beta_3)$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Everything depends what assumptions are laid on $U_{1i},U_{2i},U_{3i}$. For estimation we can form maximum likelihood, or look for analog of formulas with Mills ration for bivariate normal density for trivariate normal density. In the latter case proceed as in article.  I would be quite optimistic of this idea succeeding. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the panel data framework look at &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0262232197&quot; rel=&quot;nofollow&quot;&gt;Wooldridge book &quot;Econometric analysis of cross-section and panel data&quot;&lt;/a&gt;. It has whole chapter dedicated for sample selection problems for panel data. The way I see it, it is straightforward application of ideas in Heckman's article.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for the code in R, tough luck. If it is not implemented in plm package, it probably does not have readily available implementation.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2010-12-08T10:22:15.833" CommentCount="1" />
  <row Id="5249" PostTypeId="1" CreationDate="2010-12-08T10:43:56.587" Score="12" ViewCount="238" Body="&lt;p&gt;In a previous question I asked about tools for editing &lt;a href=&quot;http://stats.stackexchange.com/questions/5238/strategy-for-editing-comma-separated-value-csv-files&quot;&gt;CSV files&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://stats.stackexchange.com/questions/5238/strategy-for-editing-comma-separated-value-csv-files/5239#5239&quot;&gt;Gavin&lt;/a&gt;&#xA;linked to a comment on R Help by &lt;a href=&quot;http://thread.gmane.org/gmane.comp.lang.r.general/213174/focus=213179&quot; rel=&quot;nofollow&quot;&gt;Duncan Murdoch&lt;/a&gt;&#xA;suggesting that Data Interchange Format is a more reliable way to store data than CSV.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For some applications a dedicated database management system is what is needed.&#xA;However, for small scale data analysis projects something more light weight seems more suitable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider the following criteria for evaluating a file format:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;reliabile&lt;/strong&gt;: the data entered should stay true to what has been entered; &#xA;data should open consistently in different software; &lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;simple&lt;/strong&gt;: it would be nice if the file format is easy to understand and ideally be&#xA;readable with a simple text editor; it should be easy to write a simple program to read and write the format.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;open&lt;/strong&gt;: the format should be open&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;interoperable&lt;/strong&gt;: the file format should be supported by many systems&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I find tab and comma separated value formats fail on the reliability criterion.&#xA;Although I suppose I could blame the importing and exporting programs rather than the file format.&#xA;I often find myself having to make little adjustments to the options in &#xA;&lt;code&gt;read.table&lt;/code&gt; in order to prevent some strange character from breaking the loading of the data frame.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Questions&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Which file format best meets these needs?&lt;/li&gt;&#xA;&lt;li&gt;Is Data Interchange Format a better alternative? or does it have its own problems?&lt;/li&gt;&#xA;&lt;li&gt;Is there some other format that is preferable?&lt;/li&gt;&#xA;&lt;li&gt;Am I unfairly evaluating TSV and CSV? Is there a simple set of tips for working with such files that make the file format more reliable?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-12-08T12:39:51.427" LastActivityDate="2010-12-09T14:13:59.967" Title="Simple, reliable, open, and interoperable plain text format for storing data" Tags="&lt;csv-file&gt;" AnswerCount="4" CommentCount="6" FavoriteCount="1" />
  <row Id="5250" PostTypeId="1" AcceptedAnswerId="5325" CreationDate="2010-12-08T11:22:42.240" Score="4" ViewCount="632" Body="&lt;p&gt;I am trying to analyse some data using a mixed effect model. The data I collected represents the weight of some young animals of different genotype over time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using the approach proposed here:&#xA;&lt;a href=&quot;http://blog.gribblelab.org/2009/03/09/repeated-measures-anova-using-r/&quot; rel=&quot;nofollow&quot;&gt;http://blog.gribblelab.org/2009/03/09/repeated-measures-anova-using-r/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In particular I'm using solution #2&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I have something like&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(nlme)&#xA;model &amp;lt;- lme(weight ~ time * Genotype, random = ~1|Animal/time, &#xA;         data=weights)    &#xA;av &amp;lt;- anova(model)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, I would like to have some multiple comparisons.&#xA;Using &lt;code&gt;multcomp&lt;/code&gt; I can do:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(multcomp)&#xA;comp.geno &amp;lt;- glht(model, linfct=mcp(Genotype=&quot;Tukey&quot;))&#xA;print(summary(comp.geno))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And, of course, I could do the same with time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have two questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;How do I use &lt;code&gt;mcp&lt;/code&gt; to see the interaction between Time and Genotype?&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;When I rung &lt;code&gt;glht&lt;/code&gt; I get this warning:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;covariate interactions found -- default contrast might be inappropriate&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What does it mean? Can I safely ignore it? Or what should I do to avoid it?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Thank you&#xA;nico&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT:&lt;/strong&gt;&#xA;I found &lt;a href=&quot;http://cran.r-project.org/web/packages/multcomp/vignettes/generalsiminf.pdf&quot; rel=&quot;nofollow&quot;&gt;this PDF&lt;/a&gt; that says:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Because it is impossible to determine the parameters of interest automatically in this case, mcp() in multcomp will by default generate comparisons for the main effects only, &lt;strong&gt;ignoring covariates and interactions&lt;/strong&gt;. Since version 1.1-2, one can specify to average over interaction terms and covariates using arguments interaction_average = TRUE and covariate_average = TRUE respectively, whereas versions older than 1.0-0 automatically averaged over interaction terms. &lt;strong&gt;We suggest to the users, however, that they write out, manually, the set of contrasts they want.&lt;/strong&gt; One should do this whenever there is doubt about what the default contrasts measure, which typically happens in models with higher order interaction terms. We refer to Hsu (1996), Chapter~7, and Searle (1971), Chapter~7.3, for further discussions and examples on this issue.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I do not have access to those books, but maybe someone here has?&lt;/p&gt;&#xA;" OwnerUserId="582" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-12-08T19:11:45.767" LastActivityDate="2011-01-31T14:24:18.900" Title="Multiple comparisons on a mixed effects model" Tags="&lt;r&gt;&lt;anova&gt;&lt;mixed-model&gt;&lt;multiple-comparisons&gt;&lt;repeated-measures&gt;" AnswerCount="2" FavoriteCount="5" />
  <row Id="5251" PostTypeId="2" ParentId="5226" CreationDate="2010-12-08T11:31:14.750" Score="1" ViewCount="" Body="&lt;p&gt;If you lose iid and normality assumptions things can get ugly. In &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0471047457&quot; rel=&quot;nofollow&quot;&gt;Anderson book&lt;/a&gt; you can find explicit formulas for expectations of type&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\sum_{s,r,t,u}E(X_s-m)(X_r-m)(X_t-m)(X_u-m)$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;when $X=(x_1,...,x_n)$ is a sample from stationary process, with mean $m$. In general it is not possible to express such types of moments using only the second and first moments. If we have $cov(X_i,X_j)=0$, it does not guarantee that $cov(X_i^2,X_j^2)=0$ for example. It does only for normal variables, for which zero-correlation equals independence.    &lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2010-12-08T11:31:14.750" />
  <row Id="5252" PostTypeId="2" ParentId="5235" CreationDate="2010-12-08T12:15:49.997" Score="2" ViewCount="" Body="&lt;p&gt;In regression model&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$y_i=\mathbf{x}_i&amp;#39;\beta+u_i$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;the usual assumption is that $(y_i,\mathbf{x}_i,u_i)$, $i=1,...,n$ is a iid sample. Under assumptions that $E\mathbf{x}_iu_i=0$ and $E(\mathbf{x}_i\mathbf{x}_i&amp;#39;)$ has a full rank, ordinary least squares estimator&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\widehat{\beta}=(\sum_{i=1}^n\mathbf{x}_i\mathbf{x}_i&amp;#39;)^{-1}\sum_{i=1}\mathbf{x}_iy_i$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;is consistent and assymptoticaly normal. The expected covariance betweeen residual and response variable then is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Ey_iu_i=E(\mathbf{x}_i&amp;#39;\beta+u_i)u_i=Eu_i^2$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we furthermore assume that $E(u_i|\mathbf{x}_1,...,\mathbf{x}_n)=0$, we can calculate the expected covariance between $y_i$ and the estimate of regression residual:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align*}&#xA;Ey_i\widehat{u}_i&amp;amp;=Ey_i(y_i-\mathbf{x}_i'\widehat{\beta})\\&#xA;&amp;amp;=E(\mathbf{x}_i'\beta+u_i)(u_i-\mathbf{x}_i(\widehat{\beta}-\beta))\\&#xA;&amp;amp;=E(u_i^2)(1-E x_i (\sum_{i=1}^n\mathbf{x}_i\mathbf{x}_i')^{-1}\mathbf{x}_i)&#xA;\end{align*}&lt;/p&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2010-12-08T12:23:10.097" LastActivityDate="2010-12-08T12:23:10.097" CommentCount="1" />
  <row Id="5253" PostTypeId="1" AcceptedAnswerId="5256" CreationDate="2010-12-08T12:16:53.550" Score="4" ViewCount="816" Body="&lt;p&gt;I know it's a basic question, but I'm new to R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After reading a dataset:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dataset &amp;lt;- read.csv(&quot;forR.csv&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can I get R to give me the number of cases it contains?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, will the returned value include of exclude cases omitted with &lt;code&gt;na.omit(dataset)&lt;/code&gt;?&lt;/p&gt;&#xA;" OwnerUserId="1950" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-08T14:49:45.580" LastActivityDate="2010-12-08T14:51:02.570" Title="How do I get the number of rows of a data.frame in R?" Tags="&lt;r&gt;" AnswerCount="2" CommentCount="5" />
  <row Id="5255" PostTypeId="2" ParentId="5253" CreationDate="2010-12-08T12:29:53.513" Score="4" ViewCount="" Body="&lt;p&gt;Briefly:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Run &lt;code&gt;dim(dataset)&lt;/code&gt; to retrieve both &lt;em&gt;n&lt;/em&gt; and &lt;em&gt;k&lt;/em&gt;, you can also use &lt;code&gt;nrow(df)&lt;/code&gt; and &lt;code&gt;ncol(df)&lt;/code&gt; (and even &lt;code&gt;NROW(df)&lt;/code&gt; and &lt;code&gt;NCOL(df)&lt;/code&gt; -- variants are needed for other types too).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If you transform e.g. via &lt;code&gt;dataset &amp;lt;- na.omit(dataset)&lt;/code&gt;, then the cases are gone and are not counted.  But if you do &lt;em&gt;e.g.&lt;/em&gt; &lt;code&gt;summary(dataset)&lt;/code&gt; the NA cases are accounted for.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="334" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-08T14:51:02.570" LastActivityDate="2010-12-08T14:51:02.570" CommentCount="0" />
  <row Id="5256" PostTypeId="2" ParentId="5253" CreationDate="2010-12-08T12:34:11.843" Score="4" ViewCount="" Body="&lt;p&gt;&lt;code&gt;dataset&lt;/code&gt; will be a data frame. As I don't have &lt;code&gt;forR.csv&lt;/code&gt;, I'll make up a small data frame for illustration:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(1)&#xA;dataset &amp;lt;- data.frame(A = sample(c(NA, 1:100), 1000, rep = TRUE),&#xA;                      B = rnorm(1000))&#xA;&#xA;&amp;gt; head(dataset)&#xA;   A           B&#xA;1 26  0.07730312&#xA;2 37 -0.29686864&#xA;3 57 -1.18324224&#xA;4 91  0.01129269&#xA;5 20  0.99160104&#xA;6 90  1.59396745&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To get the number of cases, count the number of rows using &lt;code&gt;nrow()&lt;/code&gt; or &lt;code&gt;NROW()&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; nrow(dataset)&#xA;[1] 1000&#xA;&amp;gt; NROW(dataset)&#xA;[1] 1000&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To count the data after omitting the &lt;code&gt;NA&lt;/code&gt;, use the same tools, but wrap &lt;code&gt;dataset&lt;/code&gt; in &lt;code&gt;na.omit()&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; NROW(na.omit(dataset))&#xA;[1] 993&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The difference between &lt;code&gt;NROW()&lt;/code&gt; and &lt;code&gt;NCOL()&lt;/code&gt; and their lowercase variants (&lt;code&gt;ncol()&lt;/code&gt; and &lt;code&gt;nrow()&lt;/code&gt;) is that the lowercase versions will only work for objects that dimensions (arrays, matrices, data frames). The uppercase versions will work with vectors, which are treated as if they were a 1 column matrix, and are robust if you end up subsetting your data such that R drops an empty dimension.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively, use &lt;code&gt;complete.cases()&lt;/code&gt; and &lt;code&gt;sum&lt;/code&gt; it (&lt;code&gt;complete.cases()&lt;/code&gt; returns a logical vector [&lt;code&gt;TRUE&lt;/code&gt; or &lt;code&gt;FALSE&lt;/code&gt;] indicating if any observations are &lt;code&gt;NA&lt;/code&gt; for any rows.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; sum(complete.cases(dataset))&#xA;[1] 993&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1390" LastEditorUserId="1390" LastEditorDisplayName="" LastEditDate="2010-12-08T12:48:12.283" LastActivityDate="2010-12-08T12:48:12.283" />
  <row Id="5257" PostTypeId="2" ParentId="3845" CreationDate="2010-12-08T13:04:03.837" Score="2" ViewCount="" Body="&lt;p&gt;It is a known bug in plm package, when &lt;code&gt;effect=&quot;individual&quot; &lt;/code&gt; the pgmm crashes. The bug is fixed in plm version 1.2-7. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for simulation, you need to calculate estimates of individual effects, since they are not estimated in GMM. At the current moment, the plm does not have the functions for predicting GMM model. I have created and put them &lt;a href=&quot;https://github.com/mpiktas/vzplm&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;, but I do not advise to use them if you are not familiar with R. I have submitted these functions (forecast.pgmm and predict.pgmm) to plm maintainers, but they did not include them yet in plm package.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2010-12-08T13:04:03.837" CommentCount="2" />
  <row Id="5258" PostTypeId="2" ParentId="5171" CreationDate="2010-12-08T14:39:39.863" Score="6" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Poisson_regression&quot; rel=&quot;nofollow&quot;&gt;Log-linear models&lt;/a&gt; might be another option to look at, if you want to study your two-way data structure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you assume that the two samples are matched (i.e., there is some kind of dependency between the two series of locutions) and you take into consideration that data are actually counts that can be considered as scores or ordered responses (as suggested by @caracal), then you can also look at marginal models for matched-pairs, which usually involve the analysis of a square contingency table. It may not be necessarily the case that you end up with such a square Table, but we can also decide of an upper-bound for the number of, e.g. passive sentences. Anyway, models for matched pairs are well explained in Chapter 10 of Agresti, &lt;em&gt;&lt;a href=&quot;http://www.stat.ufl.edu/~aa/cda/cda.html&quot; rel=&quot;nofollow&quot;&gt;Categorical Data Analysis&lt;/a&gt;&lt;/em&gt;; relevant models for ordinal categories in square tables are testing for &lt;em&gt;quasi-symmetry&lt;/em&gt; (the difference in the effect of a category from one case to the other follows a linear trend in the category scores), &lt;em&gt;conditional symmetry&lt;/em&gt; ($\pi_{ab}&amp;lt;\pi_{ab}$ or $\pi_{ab}&amp;gt;\pi_{ab}$, $\forall a,b$), and &lt;em&gt;quasi-uniform association&lt;/em&gt; (linear-by-linear association off the main diagonal, which in the case of equal-interval scores means an uniform local association). Ordinal quasi-symmetry (OQS) is a special case of linear logit model, and it can be compared to a simpler model where only &lt;em&gt;marginal homogeneity&lt;/em&gt; holds with an LR test, because ordinal quasi-symmetry + marginal homogeneity $=$ symmetry.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Following Agresti's notation (p. 429), we consider $u_1\leq\dots\leq u_I$ ordered scores for variable $X$ (in rows) and variable $Y$ (in columns); $a$ or $b$ denotes any row or column. The OQS model reads as the following log-linear model:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;\log\mu_{ab}=\lambda+\lambda_a+\lambda_b+\beta u_b +\lambda_{ab}&#xD;&#xA;$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\lambda_{ab}=\lambda_{ba}$ for all $a&amp;lt;b$. Compared to the usual QS model for nominal data which is $\log\mu_{ab}=\lambda+\lambda_a^X+\lambda_b^Y+\lambda_{ab}$, where $\lambda_{ab}=0$ would mean independence between the two variables, in the OQS model we impose $\lambda_b^Y-\lambda_b^X=\beta u_b$ (hence introducing the idea of a linear trend). The equivalent logit representation is $\log(\pi_{ab}/\pi_{ba})=\beta(u_b-u_a)$, for $a\leq b$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If $\beta=0$, then we have symmetry as a special case of this model. If $\beta\neq 0$, then we have stochastically ordered margins, that is $\beta&amp;gt;0$ means that column mean is higher compared to row mean (and the greater $|\beta|$, the greater the differences between the two joint probabilities distributions $\pi_{ab}$ and $\pi_{ba}$ are, which will be reflected in the differences between row and column marginal distributions). A test of $\beta=0$ corresponds to a test of marginal homogeneity. The interpretation of the estimated $\beta$ is straightforward: the estimated probability that score on variable $X$ is $x$ units more positive than the score on $Y$ is $\exp(\hat\beta x)$ times the reverse probability. In your particular case, it means that $\hat\beta$ might allow to quantify the influence that one particular speaker exerts on the other.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of note, all R code was made available by Laura Thompson in her &lt;a href=&quot;https://home.comcast.net/~lthompson221/Splusdiscrete2.pdf&quot; rel=&quot;nofollow&quot;&gt;S Manual to Accompany Agresti's Categorical Data Analysis&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hereafter, I provide some example R code so that you can play with it on your own data. So, let's try to generate some data first:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(56)&#xA;d &amp;lt;- as.data.frame(replicate(2, rpois(420, 1.5)))&#xA;colnames(d) &amp;lt;- paste(&quot;S&quot;, 1:2, sep=&quot;&quot;)&#xA;d.tab &amp;lt;- table(d$S1, d$S2, dnn=names(d)) # or xtabs(~S1+S2, d)&#xA;library(vcdExtra)&#xA;structable(~S1+S2, data=d)&#xA;# library(ggplot2)&#xA;# ggfluctuation(d.tab, type=&quot;color&quot;) + labs(x=&quot;S1&quot;, y=&quot;S2&quot;) + theme_bw()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Visually, the cross-classification looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   S2  0  1  2  3  4  5  6&#xA;S1                        &#xA;0     17 35 31  8  7  3  0&#xA;1     41 41 30 23  7  2  0&#xA;2     19 43 18 18  5  0  1&#xA;3     11 21  9 15  2  1  0&#xA;4      0  3  4  1  0  0  0&#xA;5      1  0  0  2  0  0  0&#xA;6      0  0  0  1  0  0  0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, we can fit the OQS model. Unlike Laura Thompson which used the base &lt;code&gt;glm()&lt;/code&gt; function and a custom design matrix for symmetry, we can rely on the &lt;a href=&quot;http://cran.r-project.org/web/packages/gnm/index.html&quot; rel=&quot;nofollow&quot;&gt;gnm&lt;/a&gt; package; we need, however, to add a vector for numerical scores to estimate $\beta$ in the above model.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(gnm)&#xA;d.long &amp;lt;- data.frame(counts=c(d.tab), S1=gl(7,1,7*7,labels=0:6),&#xA;                     S2=gl(7,7,7*7,labels=0:6))&#xA;d.long$scores &amp;lt;- rep(0:6, each=7)&#xA;summary(mod.oqs &amp;lt;- gnm(counts~scores+Symm(S1,S2), data=d.long, &#xA;                       family=poisson))&#xA;anova(mod.oqs)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here, we have $\hat\beta=0.123$, and thus the probability that Speaker B scores 4 when Speaker A scores 3 is $\exp(0.123)=1.13$ times the probability that Speaker B have a score of 3 while Speaker A have a score of 4.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I recently came across the &lt;a href=&quot;http://cran.r-project.org/web/packages/catspec/index.html&quot; rel=&quot;nofollow&quot;&gt;catspec&lt;/a&gt; R package which seems to offer similar facilities, but I didn't try it. There was a good tutorial at UseR! 2009 about all this stuff: &lt;a href=&quot;http://www.r-project.org/conferences/useR-2009/tutorials/gnmCourse-3up.pdf&quot; rel=&quot;nofollow&quot;&gt;Introduction to Generalized Nonlinear Models in R&lt;/a&gt;, but see also the accompagnying vignette, &lt;a href=&quot;http://cran.r-project.org/web/packages/gnm/vignettes/gnmOverview.pdf&quot; rel=&quot;nofollow&quot;&gt;Generalized nonlinear models in R: An overview of the gnm package&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to grasp the idea with real data, there are a lot of examples with real data sets in the &lt;a href=&quot;http://cran.r-project.org/web/packages/vcdExtra/index.html&quot; rel=&quot;nofollow&quot;&gt;vcdExtra&lt;/a&gt; package from Michael Friendly. About the OQS model, Agresti used data on Premarital Sex and Extramarital sex (Table 10.5, p. 421). Results are discussed in §10.4.7 (p. 430), and $\hat\beta$ was estimated at -2.86. The code below allow (partly grabbed from Thompson's textbook) to reproduce these results. We would need to relevel factor levels so as to set the same baseline than Agresti.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;table.10.5 &amp;lt;- data.frame(expand.grid(PreSex=factor(1:4),&#xA;                                     ExSex=factor(1:4)),&#xA;                         counts=c(144,33,84,126,2,4,14,29,0,2,6,25,0,0,1,5))&#xA;table.10.5$scores &amp;lt;- rep(1:4,each=4)&#xA;summary(mod.oqs &amp;lt;- gnm(counts~scores+Symm(PreSex,ExSex), data=table.10.5, &#xA;                       family=poisson)) # beta = -2.857&#xA;anova(mod.oqs) # G^2(5)=2.10&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-12-08T19:16:01.067" LastActivityDate="2010-12-08T19:16:01.067" CommentCount="2" />
  <row Id="5259" PostTypeId="2" ParentId="726" CreationDate="2010-12-08T15:13:53.143" Score="2" ViewCount="" Body="&lt;p&gt;&lt;code&gt;Torture numbers, and they'll confess to anything.&lt;/code&gt;  ~Gregg Easterbrook&lt;/p&gt;&#xA;" OwnerUserId="1371" LastActivityDate="2010-12-08T15:13:53.143" CommunityOwnedDate="2010-12-08T15:13:53.143" CommentCount="2" />
  <row Id="5260" PostTypeId="1" AcceptedAnswerId="5273" CreationDate="2010-12-08T15:37:55.173" Score="1" ViewCount="114" Body="&lt;p&gt;Suppose $\hat{m} = \frac{1}{N}\sum_{i=1}^{N}(X_i)$ where $X_i \sim N(m,\sigma)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are the following steps correct?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Var\left\{(\hat{m}-m)^2\right\} = E\left\{(\hat{m}-m)^4\right\} - E^2\left\{(\hat{m}-m)^2\right\}$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$= 3E^2\left\{(\hat{m}-m)^2\right\} - E^2\left\{(\hat{m}-m)^2\right\}$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$= 2E^2\left\{(\hat{m}-m)^2\right\}$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and I know that $ E\left\{(\hat{m}-m)^2\right\} = \frac{1}{N^2}\sigma$. Then&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Var\left\{(\hat{m}-m)^2\right\} = 2\frac{1}{N^4}\sigma^2$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;However the textbook says (without any proving) that &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Var\left\{(\hat{m}-m)^2\right\} \tilde{} \frac{1}{N^2} $&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where am I going wrong?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update:&lt;/strong&gt;&#xA;as &lt;a href=&quot;http://stats.stackexchange.com/users/919/whuber&quot;&gt;whuber&lt;/a&gt; told in the comments, i was wrong about $ E\left\{(\hat{m}-m)^2\right\} $. This expectation equla to $\frac{1}{N}\sigma$ and not $\frac{1}{N^2}\sigma$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, the variance is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Var\left\{(\hat{m}-m)^2\right\} = 2E^2\left\{(\hat{m}-m)^2\right\} = 2\frac{1}{N^2}\sigma^2 \tilde{} \frac{1}{N^2}$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyway, the answer provided by &lt;a href=&quot;http://stats.stackexchange.com/users/2116/mpiktas&quot;&gt;mpiktas&lt;/a&gt; is also correct and i prefer to chose it as the best answer.&lt;/p&gt;&#xA;" OwnerUserId="2148" LastEditorUserId="2148" LastEditorDisplayName="" LastEditDate="2010-12-09T10:26:04.733" LastActivityDate="2010-12-09T10:26:04.733" Title="Calculating $Var\left\{(\hat{m}-m)^2\right\}$ for a univariate normal distribution" Tags="&lt;estimation&gt;&lt;variance&gt;&lt;expected-value&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="5261" PostTypeId="2" ParentId="5247" CreationDate="2010-12-08T16:12:12.467" Score="2" ViewCount="" Body="&lt;p&gt;Two figures of merit in control charting are (1) the expected length of time the process will appear to remain in control when in fact it is; and (2) the expected length of time it takes for an OOC condition to be detected after the process first moves out of control.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Under the usual assumptions--iid normally distributed values, no serial correlation, etc--we can reduce the first case to analyses of correlated coin flipping experiments.  An accurate solution takes some work; people usually run simulations.  However, each rule by itself has a simple interpretation:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Rule 1&lt;/strong&gt; characterizes each measurement by whether it lies beyond the interval $[-3\sigma, 3\sigma]$ (with 0.27% probability) or not.  It corresponds, then, to flipping a coin with $\Pr(\text{heads})$ =0.0027 and we want to know the expected number of flips before a &quot;heads&quot; (OOC condition) is observed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Rule 2&lt;/strong&gt;  characterizes each measurement by whether it exceeds $2\sigma$ or falls below $-2\sigma$.  This is like a &quot;3-sided&quot; coin: a multinomial distribution.  One face says &quot;above $2 \sigma$&quot; and occurs with probability 2.28%.  Let's call this &quot;heads 1&quot;.  Another face says &quot;below $-2 \sigma$&quot; and also occurs with probability 2.28%.  Call this &quot;heads 2&quot;.  The third face says &quot;between $-2 \sigma$ and $2 \sigma$&quot; and occurs with probability 95.45%.  The analogous question concerns the expected number of flips with this coin before a sequence of two heads &lt;em&gt;of the same type&lt;/em&gt; is observed.  The calculation might not be easy, but it's easy to see this event is fairly rare: the chance of either head appearing is just 4.55%, but given that it just appeared, the chance that a head of the same type immediately follows it is only 2.28%.  Thus, if we only had a pair of throws to consider, an OOC event of this type would occur with probability 4.55% * 2.28% * 2 (multiply by 2 to account for both types of &quot;heads&quot;) = 0.21%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Rule 3&lt;/strong&gt; can be analyzed in a similar manner (but is more complicated).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that the rules are interrelated: a single observation can violate two or even all three rules, even though all preceding observations were in control.  However, this has fairly low probability of occurring, so to a good approximation we can assume the rules are mutually exclusive (allowing us to sum their probabilities).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The purpose of rules 2 and 3 is to reduce the expected time needed to detect an OOC condition caused by a systematic change in the mean.  How they accomplish this is intuitively clear: a small increase in mean, for example, only slightly increases the chance of triggering rule 1, but greatly increases the chance of triggering rule 2.  For example, a one-sd increase in mean increases the chance of a rule 1 violation to $1 - \Phi(3-1) + \Phi(-3-1)$ = 2.28%, which is expected to take about 1/0.028 = 44 time steps to detect, but a rule 3 violation (four in a row above 1 sd) now has slightly greater than a $(1/2)^4$ = 6.25% chance of occurring, which will be detected almost three times quicker (around 16 time steps).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;In summary&lt;/strong&gt;, these rules can be understood by analyzing sequences of coin flips (or die rolls); each one corresponds to an event (or sequence of events) that is sufficiently rare that a process in control will go for a long time without triggering an OOC signal; and the combined set of rules is formulated to be able to detect relatively small shifts of the mean as quickly as possible.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-12-08T16:12:12.467" />
  <row Id="5262" PostTypeId="2" ParentId="5249" CreationDate="2010-12-08T16:32:54.793" Score="6" ViewCount="" Body="&lt;p&gt;I wonder if there is a criterion collision going on here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One complaint about file formats such as Excel, SQL, etc are that you have to define the datatypes in advance to have it behave well, which runs contrary to the &quot;something more light weight&quot; criterion (as I understand your restriction to be more time related than computationally related).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In contrast, the criteria that it not muck up the data, or allow the data to be mucked up, require some error checking. Unless you let the system auto-magically figure out the data types (which is essentially where Excel is failing you), there no way to have your cake and eat it too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;IMO, of the two, the second criterion is more important.  Data integrity, once violated, makes analysis difficult or impossible.  Lost observations or invalid values (if not properly checked) can mess up everything.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In regards to DIF, the actual raw text is not human readable and would be difficult (IMO) for humans to do data entry in.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;IMO, you should give delimited files a fair shake.  As mentioned above in the comments, the 'data mangling' is mostly the fault of a subset of tools you are using.  Well behaved programs should not mangle delimited files.  The greatest source of mangling is a poorly specified delimiter.  For example, if your data might have commas, a CSV is inappropriate.  If it might have tabs TSV is inappropriate.  For many (but not all) programs you can specify an alternate delimiter.  For example, I've used the tilde (~) in a couple difficult cases.&lt;/p&gt;&#xA;" OwnerUserId="196" LastEditorUserId="196" LastEditorDisplayName="" LastEditDate="2010-12-08T16:44:01.007" LastActivityDate="2010-12-08T16:44:01.007" CommentCount="1" />
  <row Id="5263" PostTypeId="2" ParentId="5181" CreationDate="2010-12-08T16:51:02.557" Score="5" ViewCount="" Body="&lt;p&gt;The correct answer here is &lt;a href=&quot;http://en.wikipedia.org/wiki/Hierarchical_Bayes_model&quot; rel=&quot;nofollow&quot;&gt;hierarchical modeling&lt;/a&gt; (also called multilevel modeling). What  you want to do is have the variance parameters drawn from a common prior distribution who's parameters are also estimated. Something like &lt;/p&gt;&#xA;&#xA;&lt;p&gt;CountryVariance_i ~ D(Location, Scale)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Location ~ D_2(LocationPriorParams)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Scale ~ D_3(ScalePriorParams)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where D, D_2, D_3 are whatever distributions would like. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This formalizes the notion that the variance parameters should be similar, and will do the correct kind of shrinkage. I believe that Gelman's  books (&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/158488388X&quot; rel=&quot;nofollow&quot;&gt;Bayesian Data Analysis&lt;/a&gt; and &lt;a href=&quot;http://www.stat.columbia.edu/~gelman/arm/&quot; rel=&quot;nofollow&quot;&gt;Data Analysis Using Regression and Multilevel/Hierarchical Models&lt;/a&gt;) talk quite a bit about this kind of thing (though perhaps not a lot for variance parameters). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't have a great introduction on hand, so I recommend looking at the wikipedia page and searching for &quot;Hierarchical modeling&quot; or something and clicking on various introductions (there are many). Look for graphs as these are helpful for understanding hierarchical models.&lt;/p&gt;&#xA;" OwnerUserId="1146" LastEditorUserId="1146" LastEditorDisplayName="" LastEditDate="2010-12-16T03:26:47.083" LastActivityDate="2010-12-16T03:26:47.083" />
  <row Id="5264" PostTypeId="2" ParentId="4451" CreationDate="2010-12-08T16:58:44.833" Score="0" ViewCount="" Body="&lt;p&gt;Check out kaggle.com , they have some contests about social networks and they give out datasets. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, Stanford's SNAP is a great resource. And it has research works to boot.&lt;/p&gt;&#xA;" OwnerUserId="2301" LastActivityDate="2010-12-08T16:58:44.833" CommunityOwnedDate="2010-12-08T16:58:44.833" />
  <row Id="5265" PostTypeId="2" ParentId="5238" CreationDate="2010-12-08T17:19:52.813" Score="2" ViewCount="" Body="&lt;p&gt;I suggest you look at google refine (http://code.google.com/p/google-refine/). I think is a very good tool for editing CSV files&lt;/p&gt;&#xA;" OwnerUserId="2302" LastActivityDate="2010-12-08T17:19:52.813" />
  <row Id="5266" PostTypeId="1" CreationDate="2010-12-08T17:34:11.617" Score="2" ViewCount="150" Body="&lt;p&gt;Hello!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a need to analyze an experiment that had a binomial outcome and categorical predictors. For context we sent out an email to our customers and encouraged them to register their account online (did register or did not register online). There are different nine groups that received the email. The groups are segmented based on their account opening date with our firm (1 week ago, 2 weeks ago, 3 weeks ago, etc...).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What method would I use to determine if the response from each of the nine groups is significantly different? We have used R and ran ANOVA in the past when our dependent is continuous but I am not sure how to apply the same procedure with a binomial. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you in advance for any help!&lt;/p&gt;&#xA;" OwnerUserId="2304" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-12-09T02:29:13.083" LastActivityDate="2010-12-09T14:38:43.207" Title="Analysis with binomial outcome and categorical predictor" Tags="&lt;r&gt;&lt;logistic-regression&gt;&lt;anova&gt;&lt;binomial&gt;" AnswerCount="2" />
  <row Id="5267" PostTypeId="2" ParentId="5266" CreationDate="2010-12-08T17:48:19.950" Score="2" ViewCount="" Body="&lt;p&gt;The model you re looking for is called a &lt;a href=&quot;http://en.wikipedia.org/wiki/Logistic_regression&quot; rel=&quot;nofollow&quot;&gt;logistic regression&lt;/a&gt;. &#xA;Look at the &lt;code&gt;glm&lt;/code&gt; function in R and use &lt;code&gt;family=binomial()&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1979" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-12-09T02:28:42.037" LastActivityDate="2010-12-09T02:28:42.037" />
  <row Id="5268" PostTypeId="1" CreationDate="2010-12-08T17:56:29.253" Score="2" ViewCount="143" Body="&lt;p&gt;(suggested post from cstheory)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am looking to cluster users together in a database, with each user represented by a number of features that are both discrete and continuous in nature. &quot;Similar&quot; users should be clustered together in a way that underlying strongly correlated features can be easily discovered. The number of clusters are unknown &lt;em&gt;a priori&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To make more concrete, let's say the features per user are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;gender (m/f)&lt;/li&gt;&#xA;&lt;li&gt;location (one of 10 cities)&lt;/li&gt;&#xA;&lt;li&gt;favorite color (red/green/blue).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Let's say that we have N users and that favorite color is a R.V. dependent on gender and city. How are we to discover possible strong correlations with gender and/or location and favorite colors?  There are a number of clustering techniques, from K-NN, k-means, matrix factorization, even PCA, but many seem to hide the underlying correlations that tie the users together. Any advice?&lt;/p&gt;&#xA;" OwnerUserId="2305" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-01-09T12:53:14.423" LastActivityDate="2011-01-09T13:00:17.960" Title="K-NN or matrix factorization for discovering correlated features?" Tags="&lt;clustering&gt;&lt;categorical-data&gt;&lt;archetypal&gt;" AnswerCount="0" CommentCount="3" FavoriteCount="1" />
  <row Id="5269" PostTypeId="2" ParentId="5260" CreationDate="2010-12-08T18:06:58.873" Score="2" ViewCount="" Body="&lt;p&gt;I think you intended to take the &lt;em&gt;mean&lt;/em&gt; of the $x_i$, instead you took the sum in the definition of $\hat{m}$. This makes the quantity $\hat{m} - m$ look weird.&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-12-08T18:06:58.873" CommentCount="1" />
  <row Id="5270" PostTypeId="1" AcceptedAnswerId="5276" CreationDate="2010-12-08T18:27:38.503" Score="2" ViewCount="242" Body="&lt;p&gt;I have some data which, after lots of searching, I concluded would probably benefit from using a linear mixed effects model. I think I have an interesting result here, but I am having a little trouble figuring out how to interpret all of the results. This is what I get from the summary() function in R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; summary(nonzero.lmer)&#xA;Linear mixed model fit by REML &#xA;Formula: relative.sents.A ~ relative.sents.B + (1 | id.A) + (1 | abstract) &#xA;   Data: nonzero &#xA;    AIC    BIC logLik deviance REMLdev&#xA; -698.8 -683.9  354.4   -722.6  -708.8&#xA;Random effects:&#xA; Groups   Name        Variance   Std.Dev. &#xA; id.A     (Intercept) 1.0790e-04 0.0103877&#xA; abstract (Intercept) 3.0966e-05 0.0055647&#xA; Residual             2.9675e-04 0.0172263&#xA;Number of obs: 146, groups: id.A, 97; abstract, 52&#xA;&#xA;Fixed effects:&#xA;                 Estimate Std. Error t value&#xA;(Intercept)      0.017260   0.003046   5.667&#xA;relative.sents.B 0.428808   0.080050   5.357&#xA;&#xA;Correlation of Fixed Effects:&#xA;            (Intr)&#xA;rltv.snts.B -0.742&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My question involves the relationship between the dependent variable (&quot;relative.sents.A&quot;) and &quot;relative.sents.B&quot; once the random factors are factored out. I gather that the t-value of 5.357 for relative.sents.B should be significant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But does this show what the direction of the effect is? I am thinking that because the coefficient for the slope is positive that this means that as relative.sents.B increases, so does my dependent variable. Is this correct?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The book I've been using briefly mentions that the correlation reported here is not a normal correlation, but goes into no details. Normally, I'd look there to figure out the direction and magnitude of the effect. Is that wrong?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I'm wrong on both counts, then what is a good (hopefully reasonably straightforward) way to discover the direction and size of the effect?&lt;/p&gt;&#xA;" OwnerUserId="52" LastActivityDate="2010-12-08T20:06:40.940" Title="Interpreting size and direction of fixed effects in a linear mixed effect model" Tags="&lt;r&gt;&lt;mixed-model&gt;" AnswerCount="1" />
  <row Id="5271" PostTypeId="1" AcceptedAnswerId="5272" CreationDate="2010-12-08T18:46:12.053" Score="1" ViewCount="359" Body="&lt;p&gt;I have fit a linear model and I now doing a lack-of-fit test. If the model has a good fit, the residual deviance should follow a $\chi^2$ distribution. The model I just ran has a very poor fit, and now I want to show this visually. I have used the following R code to plot the value of the residual deviance and the $\chi^2$ distribution it should come from (under the hypothesis of good fit).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df &amp;lt;- 384162&#xA;resdev &amp;lt;- 750749&#xA;x &amp;lt;- seq(from=350000, to=800000, by=1)&#xA;y &amp;lt;- dchisq(x, df=df)&#xA;plot(x, y, type='l')&#xA;abline(v=resdev, col='red')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Everything is working fine, except now the X-axis values are displaying in scientific notation. How to I control the formating of the axis values? I understand it may be crowded if I use normal notation, so can I format labels something like &lt;code&gt;350k&lt;/code&gt; for 350,000?&lt;/p&gt;&#xA;" OwnerUserId="1973" LastActivityDate="2010-12-08T19:08:27.073" Title="How to change X-axis format in plot" Tags="&lt;r&gt;" AnswerCount="1" />
  <row Id="5272" PostTypeId="2" ParentId="5271" CreationDate="2010-12-08T19:08:27.073" Score="1" ViewCount="" Body="&lt;p&gt;Look at  function &lt;code&gt;axis&lt;/code&gt;  help page. Something like the following should work:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;axis(1, at = x &amp;lt;- seq(3.5e5, 8e5, by = 5e4),  labels = paste(x/1000, &quot;k&quot;, sep = &quot;&quot; ))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2010-12-08T19:08:27.073" CommentCount="3" />
  <row Id="5273" PostTypeId="2" ParentId="5260" CreationDate="2010-12-08T19:35:22.073" Score="5" ViewCount="" Body="&lt;p&gt;If $\hat{m}=\frac{1}{n}\sum_{i=1}^nX_i$, where $X_i$ is iid normal sample, then $\hat{m}\sim N(m,\frac{\sigma^2}{n})$. Then $(\hat{m}-m)\sim N(0,\frac{\sigma^2}{n})$ and we can apply the results about normal distribution. We have&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align*}&#xA;Var((\hat{m}-m)^2)&amp;amp;=Var((N(0,\frac{\sigma^2}{n}))^2)\\&#xA;&amp;amp;=E(N(0,\frac{\sigma^2}{n}))^4-(E(N(0,\frac{\sigma^2}{n}))^2)^2\\&#xA;&amp;amp;=3\frac{\sigma^4}{n^2}-\frac{\sigma^4}{n^2}=2\frac{\sigma^4}{n^2}&#xA;\end{align*}&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2010-12-08T19:35:22.073" CommentCount="2" />
  <row Id="5274" PostTypeId="2" ParentId="5233" CreationDate="2010-12-08T19:43:07.323" Score="3" ViewCount="" Body="&lt;p&gt;Let's take a derivative (difference) here, so instead of working with location, you work directly with what you want: distance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say word FOO appears in the text 30 times. Calculate the distance (number of other words) between each consecutive occurrence of FOO, creating a vector of 29 distances. Then pick your plot: histogram, density, xy with log x, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This doesn't show you &lt;em&gt;where&lt;/em&gt; clusters are, but it does show clustering.&lt;/p&gt;&#xA;" OwnerUserId="1764" LastActivityDate="2010-12-08T19:43:07.323" CommentCount="3" />
  <row Id="5275" PostTypeId="1" AcceptedAnswerId="5277" CreationDate="2010-12-08T19:53:15.887" Score="4" ViewCount="249" Body="&lt;p&gt;Let $\{x_i\}_{i=1}^n$ be a sample from a multivariate Gaussian distribution ${\cal N}(0, \Sigma_X)$ and $\{y_i\}_{i=1}^m$ be a sample from ${\cal N}(0, \Sigma_Y)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there hypothesis tests for $\Sigma_X = \Sigma_Y$? Pointers to relevant literature would be very appreciated.&lt;/p&gt;&#xA;" OwnerUserId="168" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-12-08T20:19:25.480" LastActivityDate="2010-12-08T21:56:58.590" Title="Two-sample test for multivariate normal distributions under the assumption that means are the same" Tags="&lt;hypothesis-testing&gt;&lt;multivariate-analysis&gt;" AnswerCount="1" />
  <row Id="5276" PostTypeId="2" ParentId="5270" CreationDate="2010-12-08T20:06:40.940" Score="1" ViewCount="" Body="&lt;p&gt;If we assume that your model is adequate, then your reasoning is correct, if relative.sents.B increases by 1, relative.sents.A will increase by 0.43.  &lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2010-12-08T20:06:40.940" />
  <row Id="5277" PostTypeId="2" ParentId="5275" CreationDate="2010-12-08T20:52:08.077" Score="6" ViewCount="" Body="&lt;p&gt;The &lt;a href=&quot;http://en.wikipedia.org/wiki/Mauchly%27s_sphericity_test&quot; rel=&quot;nofollow&quot;&gt;Mauchly's test&lt;/a&gt; allows to test if a given covariance matrix is proportional to a reference (identity or other) and is available through &lt;code&gt;mauchly.test()&lt;/code&gt; under R. It is mostly used in repeated-measures design (to test (1) if the dependent variable VC matrices are equal or homogeneous, and (2) whether the correlations between the levels of the within-subjects variable are comparable--altogether, this is known as the &lt;em&gt;sphericity assumption&lt;/em&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Box’s M statistic is used (in MANOVA or LDA) to test for homogeneity of covariance matrices, but as it is very sensitive to normality it will often reject the null (&lt;a href=&quot;http://finzi.psych.upenn.edu/R/Rhelp02a/archive/33330.html&quot; rel=&quot;nofollow&quot;&gt;R code&lt;/a&gt; not available in standard packages).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Covariance structure models as found in &lt;a href=&quot;http://en.wikipedia.org/wiki/Structural_equation_modeling&quot; rel=&quot;nofollow&quot;&gt;Structural Equation Modeling&lt;/a&gt; are also an option for more complex stuff (although in multigroup analysis testing for the equality of covariances makes little sense if the variances are not equal), but I have no references to offer actually.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I guess any textbook on multivariate data analysis would have additional details on these procedures. I also found this article for the case where normality assumption is not met:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Aslam, S and Rocke, DM. &lt;a href=&quot;http://dmrocke.ucdavis.edu/papers/Aslam.Rocke.2005.pdf&quot; rel=&quot;nofollow&quot;&gt;A robust&#xA;  testing procedure for the equality of&#xA;  covariance matrices&lt;/a&gt;, Computational&#xA;  Statistics &amp;amp; Data Analysis 49 (2005)&#xA;  863-874&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-12-08T21:56:58.590" LastActivityDate="2010-12-08T21:56:58.590" />
  <row Id="5278" PostTypeId="1" AcceptedAnswerId="5280" CreationDate="2010-12-08T21:45:45.977" Score="11" ViewCount="550" Body="&lt;p&gt;What way (ways?) is there to visually explain what is ANOVA?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any references, link(s) (R packages?) will be welcomed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="253" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-12-12T11:43:56.130" LastActivityDate="2010-12-12T11:43:56.130" Title="How to visualize what ANOVA does?" Tags="&lt;data-visualization&gt;&lt;anova&gt;&lt;teaching&gt;" AnswerCount="7" CommentCount="2" FavoriteCount="7" />
  <row Id="5279" PostTypeId="2" ParentId="5249" CreationDate="2010-12-08T22:11:26.423" Score="3" ViewCount="" Body="&lt;p&gt;In all seriousness, I would consider RData files created by R itself as it fits&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;reliable (check)&lt;/li&gt;&#xA;&lt;li&gt;simple (call it a draw--the format is binary)&lt;/li&gt;&#xA;&lt;li&gt;open (check: doesn't get more open than R source code)&lt;/li&gt;&#xA;&lt;li&gt;interoperable (check: works everywhere R works)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Close enough for me.  If by systems you mean &lt;em&gt;applications&lt;/em&gt; rather than operating system then the last point is a fail.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Oh, and RData is &lt;em&gt;efficient&lt;/em&gt; as the files are now by default compressed (which used to be an option which was turned off by default).&lt;/p&gt;&#xA;" OwnerUserId="334" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-12-08T22:26:10.397" LastActivityDate="2010-12-08T22:26:10.397" CommentCount="5" />
  <row Id="5280" PostTypeId="2" ParentId="5278" CreationDate="2010-12-08T22:24:17.513" Score="15" ViewCount="" Body="&lt;p&gt;Personally, I like introducing linear regression and ANOVA by showing that it is all the same and that linear models amount to partition the total variance: We have some kind of variance in the outcome that can be explained by the factors of interest, plus the unexplained part (called the 'residual'). I generally use the following illustration (gray line for total variability, black lines for group- or individua value specific variability) :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/i4FuO.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also like the &lt;a href=&quot;http://cran.r-project.org/web/packages/heplots/index.html&quot; rel=&quot;nofollow&quot;&gt;heplots&lt;/a&gt; R package, from Michael Friendly and John Fox, but see also &lt;a href=&quot;http://socserv.mcmaster.ca/jfox/heplots-dsc-paper.pdf&quot; rel=&quot;nofollow&quot;&gt;Visual Hypothesis Tests in Multivariate Linear Models: The heplots Package for R&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Standard ways to explain what ANOVA actually does, especially in the Linear Model framework, are really well explained in &lt;a href=&quot;http://www.springer.com/statistics/statistical+theory+and+methods/book/978-0-387-95361-8&quot; rel=&quot;nofollow&quot;&gt;Plane answers to complex questions&lt;/a&gt;, by Christensen, but there are very few illustrations. Saville and Wood's &lt;a href=&quot;http://www.springer.com/statistics/book/978-0-387-97517-7&quot; rel=&quot;nofollow&quot;&gt;Statistical methods: The geometric approach&lt;/a&gt; has some examples, but mainly on regression. In Montgomery's &lt;a href=&quot;http://eu.wiley.com/WileyCDA/WileyTitle/productCd-EHEP000137.html&quot; rel=&quot;nofollow&quot;&gt;Design and Analysis of Experiments&lt;/a&gt;, which mostly focused on DoE, there are illustrations that I like, but see below&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/lhjff.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(these are mine :-)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I think you have to look for textbooks on Linear Models if you want to see how sum of squares, errors, etc. translates into a vector space, as shown on &lt;a href=&quot;http://en.wikipedia.org/wiki/Ordinary_least_squares#Geometric_approach&quot; rel=&quot;nofollow&quot;&gt;Wikipedia&lt;/a&gt;. &lt;a href=&quot;http://qed.econ.queensu.ca/pub/dm-book/&quot; rel=&quot;nofollow&quot;&gt;Estimation and Inference in Econometrics&lt;/a&gt;, by Davidson and MacKinnon, seems to have nice illustrations (the 1st chapter actually covers OLS geometry) but I only browse the French translation (available &lt;a href=&quot;http://russell.vcharite.univ-mrs.fr/EIE/&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;). &lt;a href=&quot;http://www.ef.uni-lj.si/projekti/phdnet/datoteke/Projection.pdf&quot; rel=&quot;nofollow&quot;&gt;The Geometry of Linear Regression&lt;/a&gt; has also some good illustrations. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ah, and I just remember this article by Robert Pruzek, &lt;a href=&quot;http://www.acase.org/rpruzek/anovagraphic.pdf&quot; rel=&quot;nofollow&quot;&gt;A new graphic for one-way ANOVA&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-12-10T13:56:33.720" LastActivityDate="2010-12-10T13:56:33.720" CommentCount="4" />
  <row Id="5281" PostTypeId="1" AcceptedAnswerId="5284" CreationDate="2010-12-09T00:42:19.260" Score="5" ViewCount="175" Body="&lt;p&gt;Suppose we are to estimate a expectation problem $E(f(X))$, where $X$ is a random variable with known distribution, by simulation and Large Law of numbers estimator. &lt;a href=&quot;http://en.wikipedia.org/wiki/Antithetic_variates&quot; rel=&quot;nofollow&quot;&gt;Antithetic method&lt;/a&gt; is a way to reduce variance of estimator in such cases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If $X$ is a 1D random variable with cdf $F$, antithetic method is applied as follows: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;get a sample U of the uniform distribution over $[0,1]$, then $X_1=F^{-1}(U)$  has F as cdf, and $Y_1=F^{-1}(1-F(X_1)) $also has F as cdf and $X_1$ and $Y_1$ have negative correlation. Then E(f(X)) is estimated by $\frac{\sum_{i=1}^N f(X_i)+f(Y_i)}{2N}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are my questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;if $X_1$ and $Y_1$ have negative correlation, then to reduce the variance of estimator, is it correct that $f(X_1)$ and $f(Y_1)$ must also have negative correlation? What is the condition on $f$ for this to be true?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If $X$ is a multivariate random variable, since its cdf $F$ has no quantile inverse F^{-1}, is it still possible to apply antithetic method in general cases? &#xA;If consider the special case where the components of $X$ are independent, is it possible to apply antithetic method? How if yes?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I notice that antithetic also increases the samples without actually doing more sampling of any distribution. I remember increasing sample size will also reduce variance of LLN estimators. If the variance of the estimator can be reduced by antithetic method, how much is contributed by increase of sample size and how much by introducing negative correlation between samples? If use iid samples with the same size as those samples after doubled by antithetic, which one will have less variance?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Thanks and regards!&lt;/p&gt;&#xA;" OwnerUserId="1005" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-12-09T02:27:09.217" LastActivityDate="2011-05-17T20:29:54.773" Title="Questions about antithetic variate method" Tags="&lt;expected-value&gt;&lt;antithetic&gt;&lt;variance-reduction&gt;" AnswerCount="1" />
  <row Id="5282" PostTypeId="2" ParentId="5160" CreationDate="2010-12-09T02:10:10.680" Score="2" ViewCount="" Body="&lt;p&gt;These two links provide some R (and C) code examples of implementing a DP normal mixture:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://ice.uchicago.edu/2008_presentations/Rossi/density_estimation_with_DP_priors.ppt&quot; rel=&quot;nofollow&quot;&gt;http://ice.uchicago.edu/2008_presentations/Rossi/density_estimation_with_DP_priors.ppt&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.duke.edu/~neelo003/r/DP02.r&quot; rel=&quot;nofollow&quot;&gt;http://www.duke.edu/~neelo003/r/DP02.r&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Found another reference.  Chapter 15 has DP winbugs code:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.ics.uci.edu/~wjohnson/BIDA/BIDABook.html&quot; rel=&quot;nofollow&quot;&gt;http://www.ics.uci.edu/~wjohnson/BIDA/BIDABook.html&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="2310" LastEditorUserId="2310" LastEditorDisplayName="" LastEditDate="2010-12-20T14:22:03.660" LastActivityDate="2010-12-20T14:22:03.660" CommentCount="1" />
  <row Id="5283" PostTypeId="2" ParentId="5278" CreationDate="2010-12-09T02:19:57.483" Score="6" ViewCount="" Body="&lt;p&gt;Check out Hadley Wickham's presentation on ggplot.&#xA;Starting on pages 23-40 of this document he describes an interesting approach to visualizing ANOVAs&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://had.co.nz/ggplot2/resources/2007-past-present-future.pdf&quot; rel=&quot;nofollow&quot;&gt;http://had.co.nz/ggplot2/resources/2007-past-present-future.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Link taken from:&#xA;&lt;a href=&quot;http://had.co.nz/ggplot2/&quot; rel=&quot;nofollow&quot;&gt;http://had.co.nz/ggplot2/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="995" LastActivityDate="2010-12-09T02:19:57.483" />
  <row Id="5284" PostTypeId="2" ParentId="5281" CreationDate="2010-12-09T02:26:44.333" Score="6" ViewCount="" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Yes.  There's no simple condition.  When $f$ is monotonic, $f(X_1)$ and $f(Y_1)$ will still be negatively correlated.  When $f$ is not monotonic, all bets are off.  For example, let $F$ be a uniform distribution on $[-1,1]$ and let $f(x) = x^2$.  Then $X_1 = -Y_1$, whence $f(X_1) = f(Y_1)$, implying $f(X_1)$ and $f(Y_1)$ are perfectly correlated: you gain no additional information about the expectation from $(X_1, Y_1)$ than you do from $X_1$ alone.  The cost of using the antithetic method in this extreme case is to double the sample size in order to achieve a given estimation variance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A practical example of the problem with non-monotonic $f$ appears &lt;a href=&quot;http://staff.polito.it/paolo.brandimarte/AntSamp.PDF&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Yes, in some cases.  Use the antithetic method on the components of $X$ separately.  This ought to work provided the components are not strongly correlated or when $F$ is symmetric.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Provided $f(X_1)$ and $f(Y_1)$ are negatively correlated, you get smaller estimation variance with the antithetic method.  As an extreme example of this, consider the case where $F$ is uniform on $[-1,1]$ and $f$ is the identity.  Then for any &lt;em&gt;single&lt;/em&gt; sample, $Y_1 = -X_1$ and their mean $(X_1+Y_1)/2 = 0$ estimates the mean of $F$ &lt;em&gt;exactly&lt;/em&gt;; whereas the mean of two independent samples $(X_1, X_2)$ has a variance of $1/6$.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;This technique seems to be related, at least in spirit, to &lt;a href=&quot;http://en.wikipedia.org/wiki/Latin_hypercube_sampling&quot; rel=&quot;nofollow&quot;&gt;Latin hypercube sampling&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-05-17T20:29:54.773" LastActivityDate="2011-05-17T20:29:54.773" CommentCount="1" />
  <row Id="5285" PostTypeId="2" ParentId="5278" CreationDate="2010-12-09T05:34:18.920" Score="1" ViewCount="" Body="&lt;p&gt;maybe the book the analysis of variance by henry scheffe, section 2.9 geometrical interpretations of F, orthogonality relations, will help?&lt;/p&gt;&#xA;" OwnerUserId="2312" LastActivityDate="2010-12-09T05:34:18.920" />
  <row Id="5286" PostTypeId="2" ParentId="5249" CreationDate="2010-12-09T05:38:20.340" Score="2" ViewCount="" Body="&lt;p&gt;In response to Dirk Eddelbuettel's answer, I suggest using the &lt;a href=&quot;http://www.hdfgroup.org/HDF5/whatishdf5.html&quot; rel=&quot;nofollow&quot;&gt;HDF5 file format&lt;/a&gt;. It is less simple than the RData format, or you might say, 'more rich', but certainly more interoperable (can be used in C, Java, Matlab, etc). I have found that I/O involving large HDF5 files is very fast.&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-12-09T05:38:20.340" CommentCount="3" />
  <row Id="5287" PostTypeId="2" ParentId="5114" CreationDate="2010-12-09T10:07:54.657" Score="1" ViewCount="" Body="&lt;p&gt;What about the &lt;a href=&quot;http://en.wikipedia.org/wiki/Kumaraswamy_distribution&quot; rel=&quot;nofollow&quot;&gt;Kumaraswamy&lt;/a&gt; distribution, which has the following pdf:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;f(x; a,b) = a b x^{a-1}{ (1-x^a)}^{b-1} &#xD;&#xA;$$&#xD;&#xA;for $a&amp;gt;0$, $b&amp;gt;0$, $0 &amp;lt; x &amp;lt; 1$. This distribution can be &lt;a href=&quot;http://en.wikipedia.org/wiki/Kumaraswamy_distribution#Generalizing_to_arbitrary_range&quot; rel=&quot;nofollow&quot;&gt;rescaled&lt;/a&gt; to have the required support.&lt;/p&gt;&#xA;" OwnerUserId="8" LastActivityDate="2010-12-09T10:07:54.657" />
  <row Id="5288" PostTypeId="1" AcceptedAnswerId="5313" CreationDate="2010-12-09T10:11:07.247" Score="3" ViewCount="216" Body="&lt;p&gt;I'm working in a piece of software designed for satellite image classification based on various features of objects in the image. The software provides various built-in features like the mean of the values in the object, the maximum and minimum of the values etc. However, I want to use the median of the values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't have access to the raw values in the object, all I have is the information below:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Mean&lt;/li&gt;&#xA;&lt;li&gt;Max&lt;/li&gt;&#xA;&lt;li&gt;Min&lt;/li&gt;&#xA;&lt;li&gt;Standard Deviation&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;And I can do arithmetic on those values using standard operators (+, -, /, *, ^ etc).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a way to calculate the median (or something closely approximating it) from just this information?&lt;/p&gt;&#xA;" OwnerUserId="261" LastActivityDate="2010-12-09T17:34:04.427" Title="Calculate median without access to raw data" Tags="&lt;descriptive-statistics&gt;&lt;averages&gt;" AnswerCount="2" />
  <row Id="5289" PostTypeId="2" ParentId="5288" CreationDate="2010-12-09T10:36:27.497" Score="6" ViewCount="" Body="&lt;p&gt;If you know underlying distribution of the data, you can. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, for normal distributed data, the mean and median are same (median=mode=mean).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or for exponential distribution with mean $\lambda^{-1}$ the median is $\lambda^{-1} ln(2)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;it is impossible to obtain median without having raw data or knowing the actual data distribution.&lt;/p&gt;&#xA;" OwnerUserId="2148" LastActivityDate="2010-12-09T10:36:27.497" CommentCount="7" />
  <row Id="5290" PostTypeId="1" AcceptedAnswerId="5338" CreationDate="2010-12-09T11:08:23.557" Score="0" ViewCount="157" Body="&lt;p&gt;I have a time series with an exponential moving average and I want to calculate a moving return of the EMA over the last m periods (something like a smoothed moving return).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Y(t) is the value of the time series at time period t&lt;/p&gt;&#xA;&#xA;&lt;p&gt;S(t) is the value of an EMA of Y at time period t&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now R(t) is the return of the EMA over the last m time periods:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;R(t) = S(t) / S(t-m) - 1&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is: how many time periods should the EMA calculation use for a given m?&#xA;Precisely, if the EMA is calculated using S(t) = alpha * Y(t) + (1-alpha) * S(t-1) and alpha is set by 2/(N+1), then how should N depend on m?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm assuming that N should be sufficiently less than m to prevent 'overlap' of Y values that are used in the calculation of S(t) and S(t-m).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any theories or best practices about this?&lt;/p&gt;&#xA;" OwnerUserId="2316" LastEditorUserId="2316" LastEditorDisplayName="" LastEditDate="2010-12-09T16:25:43.103" LastActivityDate="2010-12-10T05:49:12.900" Title="Moving return of exponential moving average -- choice of alpha" Tags="&lt;time-series&gt;&lt;exponential&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="1" />
  <row Id="5291" PostTypeId="2" ParentId="4030" CreationDate="2010-12-09T12:09:12.687" Score="4" ViewCount="" Body="&lt;p&gt;In &lt;a href=&quot;http://www.jstor.org/stable/2938278&quot; rel=&quot;nofollow&quot;&gt;Johansen article&lt;/a&gt; VECM is specified with dummy variables. If your exogenous variables are strictly exogenous I see no reason why you cannot use original Johansen VECM, &#xA;so look in the article how Johansen treats dummies.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;R package &lt;code&gt;vars&lt;/code&gt; implements Johansen approach, where you can supply the dummies.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2010-12-09T12:09:12.687" />
  <row Id="5292" PostTypeId="1" AcceptedAnswerId="5301" CreationDate="2010-12-09T13:49:10.580" Score="13" ViewCount="820" Body="&lt;p&gt;Is there any GUI for r that makes it easier for a beginner to start learning and programming in that language?&lt;/p&gt;&#xA;" OwnerUserId="1808" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-10T00:13:43.333" LastActivityDate="2011-03-02T05:24:33.013" CommunityOwnedDate="2010-12-09T14:32:50.613" Title="Any good GUI for R?" Tags="&lt;r&gt;&lt;gui&gt;" AnswerCount="10" CommentCount="1" FavoriteCount="4" />
  <row Id="5293" PostTypeId="1" CreationDate="2010-12-09T13:56:21.027" Score="5" ViewCount="373" Body="&lt;p&gt;Hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My name is Tuhin.&#xA;I came up with a couple of questions when I was doing an&#xA;analysis in R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I did a logistic regression analysis in R and tried to check&#xA;how good the model fits the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But, I got stuck as I could not get the pseudo R square value&#xA;for the model which could give me some idea about the variation&#xA;explained by the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could you please guide me on how to achieve this value (pseudo&#xA;R square for Logistic regression analysis).&#xA;It would also be helpful if you could show me a way to get the&#xA;Hosmer Lemeshow statistic for the model as well. I found out a&#xA;user defined function to do it, but if there is a quicker way&#xA;possible, it would be really helpful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would be very grateful if you can provide me the answers to&#xA;my queries.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Eagerly waiting for your response.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regards&lt;/p&gt;&#xA;" OwnerUserId="2318" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-12-09T15:33:07.360" LastActivityDate="2010-12-09T17:18:23.010" Title="Find out pseudo R square value for a Logistic Regression analysis" Tags="&lt;r&gt;&lt;logistic-regression&gt;&lt;goodness-of-fit&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="5294" PostTypeId="2" ParentId="5292" CreationDate="2010-12-09T14:04:18.603" Score="4" ViewCount="" Body="&lt;p&gt;List available &lt;a href=&quot;http://sciviews.org/_rgui/&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="22" LastActivityDate="2010-12-09T14:04:18.603" CommunityOwnedDate="2010-12-09T14:32:50.613" />
  <row Id="5295" PostTypeId="2" ParentId="5249" CreationDate="2010-12-09T14:13:59.967" Score="1" ViewCount="" Body="&lt;p&gt;I'm not quite sure why fixed text format with the appropriate meta data does not meet your criteria. It is not as simple to read as a delimiter but you need metadata to use the information anyway. The task of writing syntax to read the program simply depends on how large and complicated the structure of the dataset is. SPSS and Excel have a GUI to help with these tasks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are only two errors with CSV files I have come across: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Missing fields without delimiters (so every other field in that record is misplaced, I have also had this problem with missing tags in XML)&lt;/li&gt;&#xA;&lt;li&gt;A comma within a text string&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;(if you have encountered other problems feel free to give examples)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Two is solved with a more irregular delimiter as drnexus suggested (a pipe (|) is one I have encountered before, but a tilde (~) works just as well in that neither is likely to be included in string fields.) One is a problem not easily solved by whatever software you are using, and both are problems with the way people wrote the files to begin with, not the software used to read the files.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd also like to say I agree with drnexus on both this thread and his &lt;a href=&quot;http://stats.stackexchange.com/questions/5238/strategy-for-editing-comma-separated-value-csv-files/5241#5241&quot;&gt;response&lt;/a&gt; on your other recent thread about editing these files. You seem to be complaining about the software you use (particularly Excel) and asking to store data in a format that conforms to your ill behaved software. Maybe the question should be how to get Excel to stop auto-formatting plain text files. Your reliable criteria as it appears to me is a software problem with reading plain text files. I don't use R for data management, but I have not had that hard of a time reading delimited files in SPSS as you seem to be suggesting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the original files are not written properly what makes you expect any software to reliably read the file? And a specific file format will certainly not prevent you from incorrectly writing the data to whatever file type you choose to begin with.&lt;/p&gt;&#xA;" OwnerUserId="1036" LastActivityDate="2010-12-09T14:13:59.967" CommentCount="2" />
  <row Id="5296" PostTypeId="2" ParentId="5292" CreationDate="2010-12-09T14:32:36.933" Score="18" ViewCount="" Body="&lt;p&gt;This has been answered &lt;a href=&quot;http://stackoverflow.com/questions/1439059/best-ide-texteditor-for-r&quot;&gt;several times on StackOverflow&lt;/a&gt;.  The top selections on there seem to consistently be Eclipse with StatET or Emacs with ESS.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wouldn't say that there are any good gui's to make it easier to &lt;em&gt;learn&lt;/em&gt; the language.  The closest thing would be &lt;a href=&quot;http://cran.r-project.org/web/packages/Deducer/index.html&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;deducer&lt;/strong&gt;&lt;/a&gt; from Ian Fellows.  But there are plenty of other resources (books, papers, blogs, packages, etc.) available for learning.&lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2010-12-09T14:32:36.933" CommunityOwnedDate="2010-12-09T14:32:50.613" CommentCount="1" />
  <row Id="5297" PostTypeId="2" ParentId="5266" CreationDate="2010-12-09T14:38:43.207" Score="0" ViewCount="" Body="&lt;p&gt;As @fabians noted, this can be done with logistic regression. You should investigate the inclusion of your predictor as an ordinal variable (not categorical) because with 9 levels you likely could model it that way and gain power. Then you can test for a linear effect or a quadratic effect etc. instead of ignoring the ordering of the &quot;categories&quot;.&lt;/p&gt;&#xA;" OwnerUserId="2040" LastActivityDate="2010-12-09T14:38:43.207" />
  <row Id="5298" PostTypeId="2" ParentId="5293" CreationDate="2010-12-09T14:42:42.797" Score="5" ViewCount="" Body="&lt;p&gt;Take a look at the &lt;code&gt;lrm()&lt;/code&gt; function from the &lt;a href=&quot;http://cran.r-project.org/web/packages/Design/index.html&quot; rel=&quot;nofollow&quot;&gt;Design&lt;/a&gt; package. It features everything you need for fitting GLM. The Hosmer and Lemeshow test has limited power and depends on arbitrary discretization; it is discussed in Harrell, &lt;em&gt;Regression Modeling Strategies&lt;/em&gt; (p. 231) and on the &lt;a href=&quot;http://www.biostat.wustl.edu/archives/html/s-news/1999-04/msg00147.html&quot; rel=&quot;nofollow&quot;&gt;R-help&lt;/a&gt; mailing-list. There is also a comparison of GoF tests for logistic regression in &lt;a href=&quot;http://www.ds.unifi.it/cipollini/GLM/Doc/HHLcL1997_GofFit.pdf&quot; rel=&quot;nofollow&quot;&gt;A comparison of goodness-of-fit tests for the logistic regression model&lt;/a&gt;, Stat. Med. 1997 16(9):965.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an example of use:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(Design)  # depends on Hmisc&#xA;x1 &amp;lt;- rnorm(500)&#xA;x2 &amp;lt;- rnorm(500)&#xA;L  &amp;lt;- x1+abs(x2)&#xA;y  &amp;lt;- ifelse(runif(500)&amp;lt;=plogis(L), 1, 0)&#xA;f &amp;lt;- lrm(y ~ x1+x2, x=TRUE, y=TRUE)&#xA;resid(f, 'gof')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which yields something like&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Sum of squared errors     Expected value|H0                    SD &#xA;         100.33517914          100.37281429            0.37641975 &#xA;                    Z                     P &#xA;          -0.09998187            0.92035872 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;but see &lt;code&gt;help(residuals.lrm)&lt;/code&gt; for additional help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The following thread contains critical discussions that might also be helpful: Logistic Regression: &lt;a href=&quot;http://stats.stackexchange.com/questions/3559/logistic-regression-which-pseudo-r-squared-measure-is-the-one-to-report-cox-s/&quot;&gt;Which pseudo R-squared measure is the one to report (Cox &amp;amp; Snell or Nagelkerke)?&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-12-09T15:04:18.770" LastActivityDate="2010-12-09T15:04:18.770" />
  <row Id="5299" PostTypeId="1" CreationDate="2010-12-09T14:49:52.150" Score="1" ViewCount="157" Body="&lt;p&gt;I am looking for applied references to data augmentation (preferably with some written code).  Either online references are books would be great.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found this book online:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/142007749X&quot; rel=&quot;nofollow&quot;&gt;http://www.amazon.com/Bayesian-Missing-Data-Problems-Biostatistics/dp/142007749X/ref=sr_1_1?ie=UTF8&amp;amp;s=books&amp;amp;qid=1291905761&amp;amp;sr=1-1&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But with no reviews I am hesitant on purchasing it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit:  I have two variables X and Y.  Let's say X follows a mixture of normals and there is a logistic relationship between X and Y.  There is measurement error when observing X.  We observe 100 X Y pairs and need to estimate the function between the two.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a book on measurement error (John P. Buonaccorsi) the author recommends data augmentation (I believe the introduced variables are the true X means) for estimation.  However no details are given.  I am looking for simple examples (R code but doesn't really matter) to get started.&lt;/p&gt;&#xA;" OwnerUserId="2310" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-27T16:06:35.893" LastActivityDate="2011-03-27T16:06:35.893" Title="Data Augmentation Examples" Tags="&lt;mcmc&gt;&lt;error&gt;&lt;measurement&gt;&lt;mixture&gt;" AnswerCount="0" CommentCount="3" />
  <row Id="5300" PostTypeId="1" CreationDate="2010-12-09T14:56:16.567" Score="2" ViewCount="69" Body="&lt;p&gt;I have a set of data with 4 input variables and 1 result field (they're all numerical values). I want to determine the influence of variable #4 on the outcomes, so it makes sense to try to fix the other 3 inputs and the compare the results versus the value of var #4. I realize I'm probably using horrendous statistics terminology, but I was just looking for some quick help--and yes, this can be quick and dirty, it doesn't have to be professional level.&lt;/p&gt;&#xA;" OwnerUserId="2320" LastActivityDate="2010-12-09T15:34:47.920" Title="Fixing 3 variables to find the influence of a 4th?" Tags="&lt;correlation&gt;" AnswerCount="1" />
  <row Id="5301" PostTypeId="2" ParentId="5292" CreationDate="2010-12-09T15:05:19.457" Score="12" ViewCount="" Body="&lt;p&gt;I would second @Shane's recommendation for Deducer, and would also recommend &lt;a href=&quot;http://socserv.mcmaster.ca/jfox/Misc/Rcmdr/&quot; rel=&quot;nofollow&quot;&gt;the R Commander&lt;/a&gt; by John Fox.  The CRAN package is &lt;a href=&quot;http://cran.r-project.org/web/packages/Rcmdr/index.html&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.  It's called the R &quot;Commander&quot; because it returns the R commands associated with the point-and-click menu selections, which can be saved and run later from the command prompt.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this way, if you don't know how to do &lt;em&gt;something&lt;/em&gt; then you can find it in the menus and get an immediate response for the proper way to do &lt;em&gt;something&lt;/em&gt; with R code. It looks like Deducer operates similarly, though I haven't played with Deducer for a while.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The base R Commander is designed for beginner-minded tasks, but there are plugins available for some more sophisticated analyses (Deducer has plugins, too). Bear in mind, however, that no GUI can do &lt;em&gt;everything&lt;/em&gt;, and at some point the user will need to wean him/herself from pointing-and-clicking.  Some people (myself included) think that is a good thing.&lt;/p&gt;&#xA;" OwnerUserId="1108" LastEditorUserId="1108" LastEditorDisplayName="" LastEditDate="2010-12-10T00:42:49.693" LastActivityDate="2010-12-10T00:42:49.693" CommunityOwnedDate="2010-12-09T15:05:19.457" CommentCount="3" />
  <row Id="5302" PostTypeId="2" ParentId="5300" CreationDate="2010-12-09T15:09:13.803" Score="4" ViewCount="" Body="&lt;p&gt;Fit linear regression model:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Y=\beta_0+\beta_1X_1+\beta_2X_2+\beta_3X_3+\beta_4X_4$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $X_i$ are your input variables and $Y$ is the outcome. The interpretation of coefficient $\beta_4$ then is the size of change of Y if we change $X_4$ by one unit, holding other input variables constant.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-12-09T15:34:47.920" LastActivityDate="2010-12-09T15:34:47.920" CommentCount="1" />
  <row Id="5304" PostTypeId="1" AcceptedAnswerId="5320" CreationDate="2010-12-09T15:41:59.660" Score="5" ViewCount="430" Body="&lt;p&gt;Dear everyone - I've noticed something strange that I can't explain, can you? In summary: the manual approach to calculating a confidence interval in a logistic regression model, and the R function &lt;code&gt;confint()&lt;/code&gt; give different results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've been going through Hosmer &amp;amp; Lemeshow's &lt;em&gt;Applied logistic regression&lt;/em&gt; (2nd edition).  In the 3rd chapter there is an example of calculating the odds ratio and 95% confidence interval.  Using R, I can easily reproduce the model:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Call:&#xA;glm(formula = dataset$CHD ~ as.factor(dataset$dich.age), family = &quot;binomial&quot;)&#xA;&#xA;Deviance Residuals: &#xA;   Min      1Q  Median      3Q     Max  &#xA;-1.734  -0.847  -0.847   0.709   1.549  &#xA;&#xA;Coefficients:&#xA;                             Estimate Std. Error z value Pr(&amp;gt;|z|)    &#xA;(Intercept)                   -0.8408     0.2551  -3.296  0.00098 ***&#xA;as.factor(dataset$dich.age)1   2.0935     0.5285   3.961 7.46e-05 ***&#xA;---&#xA;Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 &#xA;&#xA;(Dispersion parameter for binomial family taken to be 1)&#xA;&#xA;    Null deviance: 136.66  on 99  degrees of freedom&#xA;Residual deviance: 117.96  on 98  degrees of freedom&#xA;AIC: 121.96&#xA;&#xA;Number of Fisher Scoring iterations: 4&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, when I calculate the confidence intervals of the parameters, I get a different interval to the one given in the text:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; exp(confint(model))&#xA;Waiting for profiling to be done...&#xA;                                 2.5 %     97.5 %&#xA;(Intercept)                  0.2566283  0.7013384&#xA;as.factor(dataset$dich.age)1 3.0293727 24.7013080&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Hosmer &amp;amp; Lemeshow suggest the following formula:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;e^{[\hat\beta_1\pm z_{1-\alpha/2}\times\hat{\text{SE}}(\hat\beta_1)]}&#xD;&#xA;$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and they calculate the confidence interval for &lt;code&gt;as.factor(dataset$dich.age)1&lt;/code&gt; to be (2.9, 22.9).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This seems straightforward to do in R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# upper CI for beta&#xA;exp(summary(model)$coefficients[2,1]+1.96*summary(model)$coefficients[2,2])&#xA;# lower CI for beta&#xA;exp(summary(model)$coefficients[2,1]-1.96*summary(model)$coefficients[2,2])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;gives the same answer as the book.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, any thoughts on why &lt;code&gt;confint()&lt;/code&gt; seems to give different results?  I've seen lots of examples of people using &lt;code&gt;confint()&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1991" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-12-09T18:25:58.977" LastActivityDate="2010-12-12T19:47:32.683" Title="Why is there a difference between manually calculating a logistic regression 95% confidence interval, and using the confint() function in R?" Tags="&lt;r&gt;&lt;logistic-regression&gt;&lt;confidence-interval&gt;" AnswerCount="3" />
  <row Id="5305" PostTypeId="1" CreationDate="2010-12-09T15:55:17.107" Score="0" ViewCount="222" Body="&lt;p&gt;I have an irregularly spaced &lt;code&gt;XTS&lt;/code&gt; time series (with &lt;code&gt;POSIXct&lt;/code&gt; values as index type).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I build a new time series sampled at a let's say 10 minute interval, but with each sample moment aligned to a round time (13:00:00, 13:10:00, 13:20:00, ...). If a resampling moment doesn't fall exactly on a original series value, I want to take the previous one.&lt;/p&gt;&#xA;" OwnerUserId="749" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-12-09T17:20:09.307" LastActivityDate="2010-12-09T18:58:39.610" Title="How to re-sample an XTS time series in R?" Tags="&lt;r&gt;&lt;time-series&gt;&lt;sampling&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="5306" PostTypeId="1" AcceptedAnswerId="5309" CreationDate="2010-12-09T15:55:42.223" Score="3" ViewCount="383" Body="&lt;p&gt;I'm trying to transition to R from using SPSS. In the past, I've setup my data in the wide format for my repeated measures ANOVAs in SPSS. How do I need to setup my data to run nlme()? The data is balanced. It has within-subjects variables of trial type(3 levels) each measured on 8 times, and a between-subjects factor with 2 levels. 2 separate analyses will be run; one with response time as the DV and another with accuracy as the DV. I know the data has to be in a long format but I'm not sure how the columns should be arranged. Should 1 column be the subject id, another the trial type, another the time, and then 1 for the DV? Does this matter? Any points in the right direction would be greatly appreciated. Thanks.&lt;/p&gt;&#xA;" OwnerUserId="2322" LastActivityDate="2010-12-09T18:55:54.040" Title="How do I setup up repeated measures data for analysis with nlme()?" Tags="&lt;r&gt;&lt;mixed-model&gt;&lt;dataset&gt;&lt;repeated-measures&gt;" AnswerCount="3" FavoriteCount="1" />
  <row Id="5307" PostTypeId="2" ParentId="5304" CreationDate="2010-12-09T16:17:46.617" Score="9" ViewCount="" Body="&lt;p&gt;I believe if you look into the help file for confint() you will find that the confidence interval being constructed is a &quot;profile&quot; interval instead of a Wald confidence interval (your formula from HL).  &lt;/p&gt;&#xA;" OwnerUserId="2040" LastActivityDate="2010-12-09T16:17:46.617" CommentCount="1" />
  <row Id="5308" PostTypeId="1" AcceptedAnswerId="5326" CreationDate="2010-12-09T16:34:50.800" Score="0" ViewCount="150" Body="&lt;p&gt;For this dataset:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data my;&#xA;    input x y;&#xA;    datalines;&#xA;    -122.413582861209 37.7828877716232&#xA;    -122.417876547159 37.7848288325307&#xA;;&#xA;proc print;&#xA;run;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The output and related tables are:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/lGS8r.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I import, save and use these values to their maximum precision?&lt;/p&gt;&#xA;" OwnerUserId="1077" LastActivityDate="2010-12-10T06:47:56.540" Title="Read decimal values in SAS" Tags="&lt;sas&gt;&lt;pc-sas&gt;" AnswerCount="1" />
  <row Id="5309" PostTypeId="2" ParentId="5306" CreationDate="2010-12-09T16:41:02.197" Score="4" ViewCount="" Body="&lt;p&gt;Yes, you need to set up your data so that each grouping factor, dependendent variable and covariate corresponds to a column and every row contains one observation (i.e. long format):&#xA;Everything that you enter into the model formulas for the random and fixed parts has to be a column in your data set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can use &lt;code&gt;reshape&lt;/code&gt; to get your data from wide to long format and back. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are transitioning to R and do a lot of mixed models, try to get a copy of Pinheiro/Bates &quot;Mixed-Effects Models in S and S-PLUS&quot;, it's a comprehensive reference from the guys who wrote &lt;code&gt;nlme&lt;/code&gt; with a lot of worked examples. &lt;/p&gt;&#xA;" OwnerUserId="1979" LastActivityDate="2010-12-09T16:41:02.197" />
  <row Id="5310" PostTypeId="2" ParentId="5305" CreationDate="2010-12-09T17:01:30.993" Score="1" ViewCount="" Body="&lt;pre&gt;&lt;code&gt;library(xts)&#xA;?endpoints&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Fo instance&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;tmp=zoo(rnorm(1000), as.POSIXct(&quot;2010-02-1&quot;)+(1:1000)*60)&#xA;tmp[endpoints(tmp, &quot;minutes&quot;, 20)]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;to subsample every 20 minutes&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might also want to check out to.minutes, to.daily, etc.&lt;/p&gt;&#xA;" OwnerUserId="300" LastActivityDate="2010-12-09T17:01:30.993" CommentCount="3" />
  <row Id="5311" PostTypeId="2" ParentId="5292" CreationDate="2010-12-09T17:11:31.780" Score="8" ViewCount="" Body="&lt;p&gt;I think that the command line is the best interface, and especially for the beginners. The sooner you'll start with console, the sooner you'll find out that this is the fastest, the most comfortable and what's most important the only fully non-limiting way of using R.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-12-09T17:11:31.780" CommunityOwnedDate="2010-12-09T17:11:31.780" CommentCount="5" />
  <row Id="5312" PostTypeId="2" ParentId="5293" CreationDate="2010-12-09T17:18:23.010" Score="0" ViewCount="" Body="&lt;p&gt;Pseudo R Square is very easy to calculate manually.  You just need to look up the -2LL value for the baseline based on the average probability of occurrence of the binomial event.  And, you need the -2LL value for the actual Logistic regression.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say the -2LL value for the baseline is 10 and for the Logistic regression model it is 5.  Then, the calculation of the Pseudo R Square is: (10 - 5)/10 = 50%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another common Pseudo R Square measure is the McFadden R Square that generates the same result.  Its calculation is: 1-(5/10) = 50%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Pseudo R Square measures do tell you how much your Logistic Regression model reduces the error vs simply guessing the average probability of occurrence for every observations.         &lt;/p&gt;&#xA;" OwnerUserId="1329" LastActivityDate="2010-12-09T17:18:23.010" />
  <row Id="5313" PostTypeId="2" ParentId="5288" CreationDate="2010-12-09T17:34:04.427" Score="3" ViewCount="" Body="&lt;p&gt;The question can be construed as requesting a nonparametric estimator of the median of a sample in the form f(min, mean, max, sd).  In this circumstance, by contemplating extreme (two-point) distributions, we can trivially establish that&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ 2\ \text{mean} - \text{max} \le \text{median} \le 2\ \text{mean} - \text{min}.$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There might be an improvement available by considering the constraint imposed by the known SD.  To make any more progress, additional assumptions are needed.  Typically, some measure of skewness is essential.  (In fact, skewness can be estimated from the deviation between the mean and the median relative to the sd, so one should be able to reverse the process.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One could, in a pinch, use these four statistics to obtain a &lt;a href=&quot;http://en.wikipedia.org/wiki/Principle_of_maximum_entropy&quot; rel=&quot;nofollow&quot;&gt;maximum-entropy&lt;/a&gt; solution and use its median for the estimator.  Actually, the min and max probably won't be any good, but in a satellite image there are fixed upper and lower bounds (e.g., 0 and 255 for an eight-bit image); these would constrain the maximum-entropy solution nicely.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's worth remarking that general-purpose image processing software is capable of producing far more information than this, so it could be worthwhile looking at other software solutions.  Alternatively, often one can trick the software into supplying additional information.  For example, if you could divide each apparent &quot;object&quot; into two pieces you would have statistics for the two halves.  That would provide useful information for estimating a median.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-12-09T17:34:04.427" />
  <row Id="5314" PostTypeId="2" ParentId="5292" CreationDate="2010-12-09T17:39:07.070" Score="3" ViewCount="" Body="&lt;p&gt;I used JGR for a short while, until it became apparent it would quickly consume all the memory on my system. I have not used it since, and recommend you do &lt;em&gt;not&lt;/em&gt; use it.&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-12-09T17:39:07.070" CommunityOwnedDate="2010-12-09T17:39:07.070" />
  <row Id="5315" PostTypeId="1" AcceptedAnswerId="5322" CreationDate="2010-12-09T17:56:04.053" Score="1" ViewCount="110" Body="&lt;p&gt;Dear All, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am relating age to a binary outcome in a logistic model (or, more to the point I would like to).  However, the distribution of the ages looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;nn &amp;lt;- 1000&#xA;age &amp;lt;- c(rpois(nn / 3, lambda = 0.5),&#xA;       rnorm(nn / 3, mean = 10, sd = 2),&#xA;       runif(nn / 3, min = 0, max = 15))&#xA;age &amp;lt;- age[which(age &amp;gt; 0)]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/TV4PL.png&quot; alt=&quot;alt text&quot;&gt;  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How would you approach this problem?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="1991" LastActivityDate="2010-12-10T02:23:20.637" Title="How would you deal with a bimodally distributed predictor variable in bivariate logistic regression?" Tags="&lt;distributions&gt;&lt;logistic-regression&gt;" AnswerCount="3" />
  <row Id="5316" PostTypeId="2" ParentId="5306" CreationDate="2010-12-09T17:56:56.857" Score="2" ViewCount="" Body="&lt;p&gt;It's unlikely reshape will help you out if you're coming straight from SPSS files because you not only need it set up differently, you probably need different data than you were using in SPSS.  In the standard repeated measures analysis you enter means of each individual condition for each subject into the analysis but for nlme you enter each individual data point.  You'll have to go back to the raw data files.  If those have a data point on each line and columns to identify the condition of each point then you can just basically concatenate each file together, identifying the separate subjects along the way.  Something like the following would work in that case&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;fList &amp;lt;- list.files('myDataF/')&#xA;dList &amp;lt;- lapply(fList, function(f) {x &amp;lt;- read.table(paste('myDataF', f, sep = '/'), header = TRUE)&#xA;    x$subj &amp;lt;- f&#xA;    return(x)})&#xA;myData &amp;lt;- do.call(rbind, dList)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(this could be even simpler if you knew that there was exactly the same number of lines in each file)&lt;/p&gt;&#xA;" OwnerUserId="601" LastActivityDate="2010-12-09T17:56:56.857" CommentCount="2" />
  <row Id="5317" PostTypeId="2" ParentId="5290" CreationDate="2010-12-09T18:00:15.903" Score="1" ViewCount="" Body="&lt;p&gt;This is actually a rather complex problem. There are a few directions you can look into. One way, typically recommended in the forecasting literature, is to optimize for the forecasting error. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have a specific application in mind you can define your own cost function to optimize. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A different view on this is to look at the EWMA as a state space model, then the problem is equivalent to setting up an appropriate Kalman filter which you can do with MLE, see for instance &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0198523548&quot; rel=&quot;nofollow&quot;&gt;Time Series Analysis by State Space Methods&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are other directions you can go, but I think this will give you an idea...&lt;/p&gt;&#xA;" OwnerUserId="300" LastActivityDate="2010-12-09T18:00:15.903" />
  <row Id="5318" PostTypeId="2" ParentId="5306" CreationDate="2010-12-09T18:55:54.040" Score="1" ViewCount="" Body="&lt;p&gt;NLME relies on a &quot;univariate&quot; as opposed to &quot;multi-variate&quot; data structure.  See the description below, copied from my response to another question here:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://stats.stackexchange.com/questions/3702/data-manipulation-in-r-for-functional-data-analysis/3705#3705&quot;&gt;Data manipulation in R for functional data analysis&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As to how you would get the data into R and into one of these formats, we'd need to know more about what your input file looks like and the format that it is in. However, here are some general tips on formatting the type of data that you have for analysis in any system.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Singer (Applied Longitudinal Data Analysis) suggests two generally useful layouts for the statistical analysis of longitudinal data: the person-level (mutlivariate) structure or the person-period (univariate) structure. The latter is generally preferrable for a number of reasons.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The person-level data structure (or the multivariate format) contains one row of data for each observational unit (such as persons) and a variable for each measurement period. Age would not be included in the data set and would be implicit in the levels of your time factor (e.g., in a repeated measures ANOVA). This structure can lead naturally to summaries that aren't very meaningful, is less efficient than it could be, and cannot account for your unequally spaced observations (differing age intervals between observations) or time-varying covariates.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That data setup might look something like this...&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Mutlivariate&#xA;    ID  Gender  height1 height2 height3 height4 height5&#xA;    1   Boy     76.2   74.6   78.2   77.7     76&#xA;    2   Boy     80.4   78.0   81.8   80.5     80&#xA;    3   Boy     83.3   82.0   85.4   83.3     83&#xA;    4   Girl    96.0   94.9   97.1   98.6     96&#xA;    5   Girl    87.7   90.0   89.6   90.3     89&#xA;    6   Girl    85.7   86.9   87.9   87.0     86&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;A preferable layout is often the person-period layout (or the univariate format) where each individual has a record for each time for which they were observed. The person-period dataset has a number of advantages. First, it leads to more natural summaries of the data, e.g. getting an average by group, by time or by group and time is now straight forward. Second, the dataset will accommodate entry of unequal intervals in the time dimension, such as you have here. In addition, if you have them, you could add columns for any other demographic covariates and these could differ over time. Also, data in this format is prepared for modern analytical techniques such as multilevel modeling. Finally, the univariate data structure is consistent with good practice in database design and normalization, increasing efficiency and making it appropriate for the typical query structure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The univariate layout would look something like this...&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Univariate&#xA;    ID  Age Gender  Height&#xA;    1   1   Boy     76.2&#xA;    1   1.5 Boy     74.6&#xA;    1   3   Boy     78.2&#xA;    1   5   Boy     77.7&#xA;    2   1   Girl    80.4&#xA;    2   1.5 Girl    81.8&#xA;    2   3   Girl    80.5&#xA;    2   5   Girl    80&#xA;    3   1   Boy     115.8&#xA;    3   1.5 Boy     112.3&#xA;    3   3   Boy     111.0&#xA;    3   5   Boy     104.1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="485" LastActivityDate="2010-12-09T18:55:54.040" />
  <row Id="5319" PostTypeId="2" ParentId="5305" CreationDate="2010-12-09T18:58:39.610" Score="1" ViewCount="" Body="&lt;p&gt;I'm still not sure what you're trying to do and I still think an example would help, but I thought I'd guess that you may be interested in &lt;code&gt;align.time&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Compare this:&#xA;tmp[endpoints(tmp, &quot;minutes&quot;, 20)]&#xA;# with this:&#xA;align.time( tmp[endpoints(tmp, &quot;minutes&quot;, 20)], n=60*20 )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1657" LastActivityDate="2010-12-09T18:58:39.610" CommentCount="2" />
  <row Id="5320" PostTypeId="2" ParentId="5304" CreationDate="2010-12-09T19:00:23.177" Score="7" ViewCount="" Body="&lt;p&gt;After having fetched the data from the &lt;a href=&quot;http://www.ats.ucla.edu/stat/Stata/examples/alr2/&quot; rel=&quot;nofollow&quot;&gt;accompagnying website&lt;/a&gt;, here is how I would do:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;chdage &amp;lt;- read.table(&quot;chdage.dat&quot;, header=F, col.names=c(&quot;id&quot;,&quot;age&quot;,&quot;chd&quot;))&#xA;chdage$aged &amp;lt;- ifelse(chdage$age&amp;gt;=55, 1, 0)&#xA;mod.lr &amp;lt;- glm(chd ~ aged, data=chdage, family=binomial)&#xA;summary(mod.lr)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The 95% CIs based on profile likelihood is obtained with&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(MASS)&#xA;exp(confint(mod.lr))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This often is the default if package &lt;code&gt;MASS&lt;/code&gt; is automatically loaded. In this case, I get &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;                2.5 %     97.5 %&#xA;(Intercept) 0.2566283  0.7013384&#xA;aged        3.0293727 24.7013080&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, if I want to compare with 95% Wald CIs (based on asymptotic normality) like the one you computed by hand, I would use &lt;code&gt;confint.default()&lt;/code&gt; instead; this yields:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;                2.5 %     97.5 %&#xA;(Intercept) 0.2616579  0.7111663&#xA;aged        2.8795652 22.8614705&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Wald CIs are good in most situation, although profile likelihood-based may be useful with complex sampling strategies. If you want to grasp the idea of how they work, here is a brief overview of the main principles: &lt;a href=&quot;http://people.upei.ca/hstryhn/stryhn208.pdf&quot; rel=&quot;nofollow&quot;&gt;Confidence intervals by the profile likelihood method, with applications in veterinary epidemiology&lt;/a&gt;. You can also take a look at Venables and Ripley's MASS book, §8.4, pp. 220-221.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-12-09T19:00:23.177" CommentCount="1" />
  <row Id="5321" PostTypeId="1" AcceptedAnswerId="5334" CreationDate="2010-12-09T19:07:48.340" Score="2" ViewCount="385" Body="&lt;p&gt;I'm implementing PCA using eigenvalue decomposition in matlab. I know matlab has PCA implemented, but it helps me understand all the technicalities when I write code.&#xA;I've been following the guidance from &lt;a href=&quot;http://books.google.com/books?id=bXzAlkODwa8C&amp;amp;lpg=PA647&amp;amp;dq=eigenvalue%20decomposition&amp;amp;pg=PA647#v=onepage&amp;amp;q&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;, but I'm getting different results in comparison to built-in function princomp. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could anybody look at it and point me in the right direction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's the code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;function [mu, Ev, Val ] = pca(data)&#xA;&#xA;% mu - mean image&#xA;% Ev - matrix whose columns are the eigenvectors corresponding to the eigen&#xA;% values Val &#xA;% Val - eigenvalues&#xA;&#xA;if nargin ~= 1&#xA; error ('usage: [mu,E,Values] = pca_q1(data)');&#xA;end&#xA;&#xA;mu = mean(data)';&#xA;&#xA;nimages = size(data,2);&#xA;&#xA;for i = 1:nimages&#xA; data(:,i) = data(:,i)-mu(i);&#xA;end&#xA;&#xA;L = data'*data;&#xA;[Ev, Vals]  = eig(L);    &#xA;[Ev,Vals] = sort(Ev,Vals);&#xA;&#xA;% computing eigenvector of the real covariance matrix&#xA;Ev = data * Ev;&#xA;&#xA;Val = diag(Vals);&#xA;Vals = Vals / (nimages - 1);&#xA;&#xA;% normalize Ev to unit length&#xA;proper = 0;&#xA;for i = 1:nimages&#xA; Ev(:,i) = Ev(:,1)/norm(Ev(:,i));&#xA; if Vals(i) &amp;lt; 0.00001&#xA;  Ev(:,i) = zeros(size(Ev,1),1);&#xA; else&#xA;  proper = proper+1;&#xA; end;&#xA;end;&#xA;&#xA;Ev = Ev(:,1:nimages);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1371" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-12-09T22:00:07.197" LastActivityDate="2010-12-12T12:52:20.237" Title="Eigenvalue decomposition in matlab" Tags="&lt;machine-learning&gt;&lt;pca&gt;&lt;matlab&gt;" AnswerCount="2" FavoriteCount="2" />
  <row Id="5322" PostTypeId="2" ParentId="5315" CreationDate="2010-12-09T19:35:12.803" Score="6" ViewCount="" Body="&lt;p&gt;Logistic (or other) regression does not make any assumptions about the distribution of the predictors, they are only assumed to be known exactly. So there is nothing stopping you from going ahead and using age in your regression model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The issue you have to actually worry about is whether the effect of age is linear on the logit scale, but that has nothing to do with its distribution.&lt;/p&gt;&#xA;" OwnerUserId="279" LastActivityDate="2010-12-09T19:35:12.803" CommentCount="2" />
  <row Id="5323" PostTypeId="2" ParentId="5315" CreationDate="2010-12-09T20:02:54.003" Score="2" ViewCount="" Body="&lt;p&gt;@andrew: I personally like a simple approach which is to create a logit plot where I would do something along the lines of binning age into 10 groups (deciles) and plotting the logit against these bins. This gives a sense for the linearity of the variable in the logit and may suggest a transformation. &lt;/p&gt;&#xA;" OwnerUserId="2040" LastActivityDate="2010-12-09T20:02:54.003" />
  <row Id="5324" PostTypeId="2" ParentId="5292" CreationDate="2010-12-09T21:31:56.837" Score="3" ViewCount="" Body="&lt;p&gt;Personally ESS, but as stated above i have found Rcmdr very easy to use. &lt;/p&gt;&#xA;" OwnerUserId="2262" LastActivityDate="2010-12-09T21:31:56.837" CommunityOwnedDate="2010-12-09T21:31:56.837" />
  <row Id="5325" PostTypeId="2" ParentId="5250" CreationDate="2010-12-09T21:44:24.397" Score="6" ViewCount="" Body="&lt;p&gt;If &lt;code&gt;time&lt;/code&gt; and &lt;code&gt;Genotype&lt;/code&gt; are both categorical predictors as they appear to be, and you are interested in comparing all time/Genotype pairs to each other, then you can just create one interaction variable, and use Tukey contrasts on it:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;weights$TimeGeno &amp;lt;- interaction(weigths$Time, weights$Geno)&#xA;model &amp;lt;- lme(weight ~ TimeGeno, random = ~1|Animal/time, data=weights) &#xA;comp.timegeno &amp;lt;- glht(model, linfct=mcp(TimeGeno=&quot;Tukey&quot;)) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you are interested in other contrasts, then you can use the fact that the &lt;code&gt;linfct&lt;/code&gt; argument can take a matrix of coefficients for the contrasts - this way you can set up exactly the comparisons you want.&lt;/p&gt;&#xA;" OwnerUserId="279" LastActivityDate="2010-12-09T21:44:24.397" CommentCount="1" />
  <row Id="5326" PostTypeId="2" ParentId="5308" CreationDate="2010-12-09T21:50:21.977" Score="4" ViewCount="" Body="&lt;p&gt;The full precision value is there, you just don't see it. SAS uses formats to control what it prints/shows, even if you don't notice it (there are default numeric and character formats).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Try using &lt;code&gt;FORMAT x y 17.12;&lt;/code&gt; either in the data step to set it permanently, or in the &lt;code&gt;PROC PRINT&lt;/code&gt; step to set it temporarily.&lt;/p&gt;&#xA;" OwnerUserId="279" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-12-10T06:47:56.540" LastActivityDate="2010-12-10T06:47:56.540" CommentCount="1" />
  <row Id="5327" PostTypeId="1" AcceptedAnswerId="6508" CreationDate="2010-12-10T00:54:26.633" Score="1" ViewCount="482" Body="&lt;p&gt;My (very basic) knowledge of the Tobit regression model isn't from a class, like I would prefer. Instead, I have picked up pieces of information here and there through several Internet searches. My best guess at the assumptions for truncated regression are that they are very similar to the ordinary least squares (OLS) assumptions. I have no idea if that is correct, though.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hence my question: &lt;em&gt;What are the assumptions I should check for when performing Tobit regression?&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;&lt;strong&gt;Note:&lt;/em&gt;&lt;/strong&gt; &lt;em&gt;The original form of this question referred to truncated regression, which was not the model I was using or asking about. I have corrected the question.&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="1583" LastEditorUserId="1583" LastEditorDisplayName="" LastEditDate="2011-01-24T20:30:24.627" LastActivityDate="2011-01-25T07:03:46.943" Title="What are the assumptions for applying a Tobit regression model?" Tags="&lt;regression&gt;&lt;assumptions&gt;" AnswerCount="3" CommentCount="5" />
  <row Id="5328" PostTypeId="2" ParentId="4451" CreationDate="2010-12-10T01:12:14.670" Score="0" ViewCount="" Body="&lt;p&gt;Facebook social graph, application installations and Last.fm users, events, groups at&#xA;&lt;a href=&quot;http://odysseas.calit2.uci.edu/research/&quot; rel=&quot;nofollow&quot;&gt;http://odysseas.calit2.uci.edu/research/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Two datasets (collected April-May 2009) which contain representative samples of ~1 million users Facebook-wide, with a few annotated properties: for each sampled user,  the friend list, privacy settings, and network membership are included. A third dataset ( collected Feb 2008) includes a bipartite graph that represents application installations by Facebook users. A fourth dataset with Daily Active Users and application installations over 6 months (collected Sept 2007-Feb 2008). A fifth dataset that includes a representative sample of Last.fm users obtained using multigraph sampling (collected July 2010).&lt;/p&gt;&#xA;" OwnerUserId="2326" LastActivityDate="2010-12-10T01:12:14.670" CommunityOwnedDate="2010-12-10T01:12:14.670" />
  <row Id="5329" PostTypeId="1" AcceptedAnswerId="5337" CreationDate="2010-12-10T01:40:31.767" Score="0" ViewCount="63" Body="&lt;p&gt;Hello every body,&#xA;I want to cluster a graph using the FCM algorithm, I used the adjacency matrix of the graph as the data, and the &quot;Euclidean&quot; distance  as metric.&#xA;The problem is that the adjacency matrix is full of zero's (depends actually on the degree of the node) and my network is a kinda huge more than 2000 node.&#xA;For the results I get many small size clusters and some big size clusters, that some time contain about 50% of the population and it is not right.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think my problem is from the data representation. is there any conditions that must be satisfied apart from the FCM algorithm conditions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you in advance.  &lt;/p&gt;&#xA;" OwnerUserId="2325" LastActivityDate="2010-12-10T04:39:47.820" Title="Are there any conditions on the data in Fuzzy c-mean clustering?" Tags="&lt;clustering&gt;" AnswerCount="1" />
  <row Id="5330" PostTypeId="1" AcceptedAnswerId="5335" CreationDate="2010-12-10T01:44:12.643" Score="1" ViewCount="100" Body="&lt;p&gt;Hi my apologies first, I'm a biologist and not so good in statistics. In my study I'm studying the effect of concentration of feed on growth of a certain specimen. I have with me the different concentrations and the growth data for each concentration. What do you suggest is the best statistical test to find out the best concentration (i.e the one having maximum growth). Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="2327" LastActivityDate="2010-12-10T08:09:31.127" Title="Which test to find out best concentration (the one having maximum effect)?" Tags="&lt;regression&gt;&lt;mathematical-statistics&gt;" AnswerCount="3" FavoriteCount="1" />
  <row Id="5331" PostTypeId="2" ParentId="5315" CreationDate="2010-12-10T02:23:20.637" Score="0" ViewCount="" Body="&lt;p&gt;If binning the age variable is an option, you might want to use a classification tree to get an idea of where to place the cutoffs.&lt;/p&gt;&#xA;" OwnerUserId="1945" LastActivityDate="2010-12-10T02:23:20.637" />
  <row Id="5332" PostTypeId="2" ParentId="5292" CreationDate="2010-12-10T02:48:55.077" Score="2" ViewCount="" Body="&lt;p&gt;I recommend &lt;a href=&quot;http://www.sciviews.org/Tinn-R/&quot; rel=&quot;nofollow&quot;&gt;Tinn-R&lt;/a&gt; (Which is the acronym for Tinn is not Notepad)&lt;/p&gt;&#xA;" OwnerUserId="1709" LastEditorUserId="1709" LastEditorDisplayName="" LastEditDate="2010-12-10T07:08:08.270" LastActivityDate="2010-12-10T07:08:08.270" CommunityOwnedDate="2010-12-10T02:48:55.077" CommentCount="2" />
  <row Id="5333" PostTypeId="1" AcceptedAnswerId="5374" CreationDate="2010-12-10T03:04:44.617" Score="1" ViewCount="385" Body="&lt;p&gt;I have a linear mixed-effect model which I hope will answer the question of whether an increase in the frequency of use of one word leads to an increase of the frequency of use of that word by another person in a conversation, factoring out random effects of subject and topic of conversation. The basic model I've come up with looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Linear mixed model fit by REML &#xA;Formula: relative.sents.A ~ relative.sents.B + (1 | id.A) + (1 | abstract) &#xA;   Data: nonzero &#xA;    AIC    BIC logLik deviance REMLdev&#xA; -698.8 -683.9  354.4   -722.6  -708.8&#xA;Random effects:&#xA; Groups   Name        Variance   Std.Dev. &#xA; id.A     (Intercept) 1.0790e-04 0.0103877&#xA; abstract (Intercept) 3.0966e-05 0.0055647&#xA; Residual             2.9675e-04 0.0172263&#xA;Number of obs: 146, groups: id.A, 97; abstract, 52&#xA;&#xA;Fixed effects:&#xA;                 Estimate Std. Error t value&#xA;(Intercept)      0.017260   0.003046   5.667&#xA;relative.sents.B 0.428808   0.080050   5.357&#xA;&#xA;Correlation of Fixed Effects:&#xA;            (Intr)&#xA;rltv.snts.B -0.742&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The &quot;dependent&quot; variable is relative frequency of use by one person, and the fixed variable is relative frequency of use by another. I decided to see what the R^2 would be:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; cor(nonzero$relative.sents.A, fitted(nonzero.lmer))^2&#xA;[1] 0.6705905&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To see what proportion of this is due to the fixed effect, I made a new model with only the random effects:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; summary(r.only.lmer)&#xA;Linear mixed model fit by REML &#xA;Formula: relative.sents.A ~ 1 + (1 | id.A) + (1 | abstract) &#xA;   Data: nonzero &#xA;    AIC    BIC logLik deviance REMLdev&#xA; -678.2 -666.3  343.1   -696.7  -686.2&#xA;Random effects:&#xA; Groups   Name        Variance   Std.Dev. &#xA; id.A     (Intercept) 1.2868e-04 0.0113435&#xA; abstract (Intercept) 7.8525e-06 0.0028022&#xA; Residual             3.7643e-04 0.0194017&#xA;Number of obs: 146, groups: id.A, 97; abstract, 52&#xA;&#xA;Fixed effects:&#xA;            Estimate Std. Error t value&#xA;(Intercept) 0.029149   0.002088   13.96&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;...and tried the same thing:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; cor(nonzero$relative.sents.A, fitted(r.only.lmer))^2&#xA;[1] 0.6882534&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To my surprise, without that fixed effect, R^2 seems to increase!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does this mean my model is useless? If so, any suggestions on what might be wrong? Or am I somehow misinterpreting these results? &lt;/p&gt;&#xA;" OwnerUserId="52" LastEditorUserId="52" LastEditorDisplayName="" LastEditDate="2010-12-10T18:15:33.003" LastActivityDate="2010-12-11T11:52:26.187" Title="Interpreting correlation from two linear mixed-effect models" Tags="&lt;r&gt;&lt;mixed-model&gt;" AnswerCount="1" CommentCount="4" FavoriteCount="3" />
  <row Id="5334" PostTypeId="2" ParentId="5321" CreationDate="2010-12-10T03:25:24.200" Score="3" ViewCount="" Body="&lt;p&gt;The line &lt;code&gt;[Ev,Vals] = sort(Ev,Vals);&lt;/code&gt; probably does not do what you think it should. The sort command would have to be applied separately. Moreover, &lt;code&gt;eig&lt;/code&gt; returns a diagonal matrix of eigenvalues, which you have to strip out to a vector. Probably you want to do the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[Ev, Vals]  = eig(L);&#xA;Vals = diag(Vals);        %strip out the eigenvalues;&#xA;[Vals,sidx] = sort(Vals); %do the sort&#xA;Ev = Ev(:,sidx);          %apply the sort to the eigenvectors as well.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I would guess this is actually unecessary, because &lt;code&gt;eig&lt;/code&gt; returns the eigenvectors, -values, in ascending order anyway.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On a stylistic note, you can subtract out the mean using &lt;code&gt;bsxfun&lt;/code&gt; as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X = bsxfun(@minus,X,mu);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;instead of calling the for-loop.&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-12-10T03:25:24.200" />
  <row Id="5335" PostTypeId="2" ParentId="5330" CreationDate="2010-12-10T03:30:21.893" Score="2" ViewCount="" Body="&lt;p&gt;It sounds like an ANOVA (analysis of variance) setting, except that usually ANOVA is used to ask the question whether there are differences in growth between the concentrations. If you used ANOVA and found that there is a difference, then it wouldn't indicate which is the &quot;best&quot; one, just that they are different. There are post-hoc tests to look at all the different pairs. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd take a different approach though:&#xA;Can the different concentration be considered as gradually increasing? (rather than just &quot;different&quot; from each other). What kind of relationship do you anticipate between the feed concentration and the growth? How many concentrations to you have? how many records per concentration? If you have more than 2-3 concentrations and sufficient data on each concentration level you could fit a parametric or nonparametric growth curve (x-axis is concentration, y-axis is growth). This would give you a clearer idea of what is going on, with less assumptions.&lt;/p&gt;&#xA;" OwnerUserId="1945" LastActivityDate="2010-12-10T03:30:21.893" />
  <row Id="5337" PostTypeId="2" ParentId="5329" CreationDate="2010-12-10T04:39:47.820" Score="0" ViewCount="" Body="&lt;p&gt;Different types of measures that form the matrix may be used to identify the cell values, where the measure controls how the clusters are formed. Some examples of values that can be used as measures include minimum distance from one node to the other, the degree of connectivity of each node and the amount of nodes needed to pass by to reach the other node.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I consider that minimum distance from one node to the other should be the best choice.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also try using the minkowsky distance which is the standard distance used in fcm, for calculating this distance you need a parameter which is the degree of fuzzyness of the distance, wwhen this parameter is 1 this distance is equal to the euclidian distan. &lt;/p&gt;&#xA;" OwnerUserId="1808" LastActivityDate="2010-12-10T04:39:47.820" CommentCount="4" />
  <row Id="5338" PostTypeId="2" ParentId="5290" CreationDate="2010-12-10T05:49:12.900" Score="1" ViewCount="" Body="&lt;p&gt;N itself is often set in an arbitrary manner that makes sense for the application. Frankly the alpha = 2/(N+1) is also somewhat arbitrary, &lt;a href=&quot;http://www.itl.nist.gov/div898/handbook/pmc/section4/pmc431.htm&quot; rel=&quot;nofollow&quot;&gt;http://www.itl.nist.gov/div898/handbook/pmc/section4/pmc431.htm&lt;/a&gt; suggests instead doing an optimization that minimizes the mean squared error. (Nevertheless, alpha = 2/(N+1) seems to be fairly standard practice.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So as to how n should depend on m, or vice versa, I say try to find something that makes sense given your particular data and that will help with interpreting your results. I wouldn't worry about overlap too much, as it's not like the data is independent--that's why you do the EMA in the first place. &lt;/p&gt;&#xA;" OwnerUserId="2073" LastActivityDate="2010-12-10T05:49:12.900" CommentCount="1" />
  <row Id="5339" PostTypeId="2" ParentId="5330" CreationDate="2010-12-10T06:07:01.160" Score="1" ViewCount="" Body="&lt;p&gt;If you have growth curves from each feed concentration, with time on the x-axis, you'll first want to estimate the growth from each curve. % change at the end points may be sufficient for your purposes, but to use all of your data consider looking to fitting some kind of model to each growth curve with a regression. Then you can estimate the parameters of the model and do a statistical comparison on those. Your institution might have access to graph pad, (can't link b/c not enough rep but google), and it is very intuitive.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This approach has some benefits. For example, one concentration might lead to faster growth initially but not peak as high, for whatever reason--maybe it becomes slightly toxic with time. This analysis could potentially detect that. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For more help we'd need to know the type of growth you expect from your specimen. If it's bacteria there are a ton of ways to do growth curves, see e.g. the classic paper by Zwietering et al here: &lt;a href=&quot;http://aem.asm.org/cgi/reprint/56/6/1875.pdf&quot; rel=&quot;nofollow&quot;&gt;http://aem.asm.org/cgi/reprint/56/6/1875.pdf&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="2073" LastActivityDate="2010-12-10T06:07:01.160" />
  <row Id="5340" PostTypeId="1" AcceptedAnswerId="5357" CreationDate="2010-12-10T06:17:00.830" Score="1" ViewCount="247" Body="&lt;p&gt;In Excel or Google Docs you can readily construct a Cumulative Distribution Function for the Normal distribution using the NORMDIST function (X, Mean, Standard Deviation).  I am trying to do the same for a Lognormal distribution using the LOGNORMALDIST function that has the same parameters (X, Mean, Standard Deviation).  I have played around with using X or LN(X) on the mentioned parameters. But, I get absurd results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you know how to generate a Lognormal Cumulative Distribution Function using LOGNORMALDIST? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance for any assistance.  I will readily reward anyone with a good answer with a positive vote.  &lt;/p&gt;&#xA;" OwnerUserId="1329" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-12-10T17:11:44.917" LastActivityDate="2010-12-10T17:11:44.917" Title="How to use the LOGNORMALDIST function to generate a Cumulative Distribution Function?" Tags="&lt;distributions&gt;&lt;excel&gt;&lt;p-values&gt;" AnswerCount="2" CommentCount="3" />
  <row Id="5341" PostTypeId="2" ParentId="5330" CreationDate="2010-12-10T08:09:31.127" Score="2" ViewCount="" Body="&lt;p&gt;Are you not looking after method for studying &lt;a href=&quot;http://en.wikipedia.org/wiki/Dose-response_relationship&quot; rel=&quot;nofollow&quot;&gt;dose-response relationship&lt;/a&gt;? From the description you gave, I guess your growth curves are highly non-linear. Moreover, the problem of using simple ANOVA to compare two growth curve (i.e., equivalent to a time course curve) is that it would be blind to the fact that the different doses are administered in a sequential order, though it should give a reasonable first idea of &lt;em&gt;some kind&lt;/em&gt; of differences between the two, if any. However, multiple comparisons are likely not to answer the question you really want to ask since we often expect very few differences at baseline and larger ones at intermediate concentration levels, whereas what you generally want to assess is what's the lowest concentration level that yields a significant difference between the two products.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are using R, the &lt;a href=&quot;http://cran.r-project.org/web/packages/drc/index.html&quot; rel=&quot;nofollow&quot;&gt;drc&lt;/a&gt; package (see also &lt;a href=&quot;http://www.bioassay.dk&quot; rel=&quot;nofollow&quot;&gt;www.bioassay.dk&lt;/a&gt;) allows to fit various models to DR data, and plot the resulting DR curve like the one shown below. It was generated from the on-line help with data &lt;code&gt;secalonic&lt;/code&gt; whose description is:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Data stem from an experiment assessing&#xA;  the inhibitory effect of secalonic&#xA;  acids on plant growth.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Gong, X. and Zeng, R. and Luo, S. and&#xA;  Yong, C. and Zheng, Q. (2004) Two new&#xA;  secalonic acids from &lt;em&gt;Aspergillus&#xA;  Japonicus&lt;/em&gt; and their allelopathic&#xA;  effects on higher plants, &lt;em&gt;Proceedings&#xA;  of International Symposium on&#xA;  Allelopathy Research and Application&lt;/em&gt;,&#xA;  27-29 April, Shanshui, Guangdong,&#xA;  China (Editors: R. Zeng and S. Luo),&#xA;  209-217.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Ritz, C (2009) Towards a unified&#xA;  approach to dose-response modeling in&#xA;  ecotoxicology To appear in &lt;em&gt;Environ&#xA;  Toxicol Chem&lt;/em&gt;.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/qM7CE.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-12-10T08:09:31.127" />
  <row Id="5342" PostTypeId="2" ParentId="5340" CreationDate="2010-12-10T08:19:06.807" Score="3" ViewCount="" Body="&lt;p&gt;On Microsoft Office for Mac 2008 &lt;code&gt;LOGNORMDIST(5;1;1)&lt;/code&gt; gives &lt;code&gt;0,728882893&lt;/code&gt; in Excel. On R &lt;code&gt;plnorm(5,1,1)&lt;/code&gt; gives &lt;code&gt;0.7288829&lt;/code&gt;. In R you need to supply mean and standard deviation on log scale, so it seems that in Excel you need to do the same.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2010-12-10T08:19:06.807" CommentCount="9" />
  <row Id="5343" PostTypeId="1" CreationDate="2010-12-10T08:20:25.247" Score="4" ViewCount="295" Body="&lt;p&gt;I am trying to predict the outcome of a random variable x, which is a real-valued number. In some cases I can observe another variable y1, which should approximate x. I model y1 as a Gaussian distribution with mean of 0 and an empirically estimated standard deviation, and use the probability density function of that Gaussian to predict x.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In some cases I also have a second observation y2, which I can model similarly as a Gaussian.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the appropriate way to combine y1 and y2 into an estimate on x? Should I add the distributions and model x as a mixture of Gaussians, or should I multiply them? Or, should I try both and pick the answer that maximizes the probability of sampling an independent set of data?&lt;/p&gt;&#xA;" OwnerUserId="2328" LastEditorUserId="2328" LastEditorDisplayName="" LastEditDate="2010-12-10T17:25:18.083" LastActivityDate="2010-12-12T19:04:16.523" Title="Combining Probability Density Functions" Tags="&lt;distributions&gt;&lt;probability&gt;&lt;normal-distribution&gt;" AnswerCount="1" CommentCount="6" />
  <row Id="5344" PostTypeId="1" AcceptedAnswerId="5345" CreationDate="2010-12-10T09:31:14.580" Score="9" ViewCount="1403" Body="&lt;p&gt;Hello-&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have fit a few mixed effects models (particularly longitudinal models) using lme4 in r but would like to really master the models and the code that goes with them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, before diving in with both feet (and buying some books) I want to be sure that I am learning the right library. I have used lme4 up to now because I just found it easier than nlme, but if nlme is better for my purposes then I feel I should use that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm sure neither is &quot;better&quot; in a simplistic way, but I would value some opinions or thoughts. My main criteria are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;easy to use (I'm a psychologist by training, and not particularly versed in statistics or coding, but I'm learning)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;good features for fitting longitudinal data (if there is a difference here- but this is what I mainly use them for)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;good (easy to interpret) graphical summaries, again not sure if there is a difference here but I often produce graphs for people even less technical than I, so nice clear plots are always good (I'm very fond of the xyplot function in lattice() for this reason).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;As usual, hope this question isn't too vague, and thanks in advance for any wisdom!&lt;/p&gt;&#xA;" OwnerUserId="199" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-10T11:03:56.030" LastActivityDate="2010-12-11T11:45:13.167" Title="How to choose nlme or lme4 R library for mixed effects models?" Tags="&lt;r&gt;&lt;mixed-model&gt;" AnswerCount="3" FavoriteCount="2" />
  <row Id="5345" PostTypeId="2" ParentId="5344" CreationDate="2010-12-10T10:40:25.643" Score="11" ViewCount="" Body="&lt;p&gt;Both packages use &lt;code&gt;Lattice&lt;/code&gt; as the backend, but &lt;code&gt;nlme&lt;/code&gt; has some nice features like &lt;code&gt;groupedData()&lt;/code&gt; and &lt;code&gt;lmList()&lt;/code&gt; that are lacking in &lt;code&gt;lme4&lt;/code&gt; (IMO). From a practical perspective, the two most important criteria seem, however, that&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;code&gt;lme4&lt;/code&gt; extends &lt;code&gt;nlme&lt;/code&gt; with other link functions: in &lt;code&gt;nlme&lt;/code&gt;, you cannot fit outcomes whose distribution is not gaussian, &lt;code&gt;lme4&lt;/code&gt; can be used to fit mixed-effects logitic regression, for example.&lt;/li&gt;&#xA;&lt;li&gt;in &lt;code&gt;nlme&lt;/code&gt;, it is possible to specify the variance-covariance matrix for the random effects (e.g. an AR(1)); it is not possible in &lt;code&gt;lme4&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Now, &lt;code&gt;lme4&lt;/code&gt; can easily handle very huge number of random effects (hence, No. individuals in a given study) thanks to its C part and the use of sparse matrices. The &lt;code&gt;nlme&lt;/code&gt; package has somewhat been superseded by &lt;code&gt;lme4&lt;/code&gt; so I won't expect people spending much time developing add-ons on top of &lt;code&gt;nlme&lt;/code&gt;. Personally, when I have a continuous response in my model, I tend to use both packages, but I'm now versed to the &lt;code&gt;lme4&lt;/code&gt; way for fitting GLMM.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rather than buying a book, take a look first at the Doug Bates' draft book on R-forge: &lt;a href=&quot;http://lme4.r-forge.r-project.org/book/&quot; rel=&quot;nofollow&quot;&gt;lme4: Mixed-effects Modeling with R&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-12-10T10:40:25.643" CommentCount="5" />
  <row Id="5346" PostTypeId="1" CreationDate="2010-12-10T10:54:27.050" Score="6" ViewCount="310" Body="&lt;p&gt;Hi all,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand that in One-Way ANOVA two alternative F-Ratios have been derived to be robust when homogeneity of variance has been violated. Tomarkin and Serlin (1986) review amongst other techniques the Brown-Forsythe and Welch F-Ratios and conclude that both control the type I error rate well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far I have only come across W and B-Fs F Ratios in One-Way ANOVAS. Am I able to use them in ANOVAS with two factors? And if no, why not?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="2285" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-10T11:03:12.177" LastActivityDate="2011-03-23T14:55:49.300" Title="Brown-Forsythe and Welch f-ratios in two-way ANOVAs?" Tags="&lt;anova&gt;&lt;homogeneity&gt;" AnswerCount="1" FavoriteCount="2" />
  <row Id="5347" PostTypeId="1" AcceptedAnswerId="5358" CreationDate="2010-12-10T11:06:05.337" Score="9" ViewCount="719" Body="&lt;p&gt;I am modeling a random variable ($Y$) which is the sum of some ~15-40k independent Bernoulli random variables ($X_i$), each with a different success probability ($p_i$). Formally, $Y=\sum X_i$ where $\Pr(X_i=1)=p_i$ and $\Pr(X_i=0)=1-p_i$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am interested in quickly answering queries such as $\Pr(Y&amp;lt;=k)$ (where $k$ is given).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently, I use random simulations to answer such queries. I randomly draw each $X_i$ according to its $p_i$, then sum all $X_i$ values to get $Y&amp;#39;$. I repeat this process a few thousand times and return the fraction of times $\Pr(Y&amp;#39;\leq k)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Obviously, this is not totally accurate (although accuracy greatly increases as the number of simulations increases). Also, it seems I have enough data about the distribution to avoid the use simulations. Can you think of a reasonable way to get the exact probability $\Pr(Y\leq k)$?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;p.s.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I use Perl &amp;amp; R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Following the responses I thought some clarifications might be needed. I will shortly describe the setting of my problem. Given is a circular genome with  circumference &lt;code&gt;c&lt;/code&gt; and a set of &lt;code&gt;n&lt;/code&gt; ranges mapped to it. For example, &lt;code&gt;c=3*10^9&lt;/code&gt; and &lt;code&gt;ranges={[100,200],[50,1000],[3*10^9-1,1000],...}&lt;/code&gt;. Note all ranges are closed (both ends are inclusive). Also note that we only deal with integers (whole units).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am looking for regions on the circle that are undercovered by the given &lt;code&gt;n&lt;/code&gt; mapped ranges. So to test whether a given a range of length &lt;code&gt;x&lt;/code&gt; on the circle is undercovered, I test the hypothesis that the &lt;code&gt;n&lt;/code&gt; ranges are mapped randomly. The probability a mapped range of length &lt;code&gt;q&amp;gt;x&lt;/code&gt; will fully cover the given range of length &lt;code&gt;x&lt;/code&gt; is &lt;code&gt;(q-x)/c&lt;/code&gt;. This probability becomes quite small when &lt;code&gt;c&lt;/code&gt; is large and/or &lt;code&gt;q&lt;/code&gt; is small. What I'm interested is the number of ranges (out of &lt;code&gt;n&lt;/code&gt;) which cover &lt;code&gt;x&lt;/code&gt;. This is how &lt;code&gt;Y&lt;/code&gt; is formed. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I test my null hypothesis vs. one sided alternative (undercoverage). Also note I am testing multiple hypothesis (different &lt;code&gt;x&lt;/code&gt; lengths), and sure to correct for this.&lt;/p&gt;&#xA;" OwnerUserId="634" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-04-13T22:54:38.963" LastActivityDate="2011-04-13T22:54:38.963" Title="How can I efficiently model the sum of Bernoulli random variables?" Tags="&lt;r&gt;&lt;distributions&gt;&lt;binomial&gt;&lt;random-variable&gt;" AnswerCount="7" CommentCount="6" FavoriteCount="1" />
  <row Id="5348" PostTypeId="2" ParentId="5347" CreationDate="2010-12-10T11:24:02.673" Score="5" ViewCount="" Body="&lt;p&gt;The sum of $n$ Bernoulli random variables is the &lt;a href=&quot;http://en.wikipedia.org/wiki/Binomial_distribution&quot; rel=&quot;nofollow&quot;&gt;binomial distribution&lt;/a&gt;. So in R to obtain $Pr(Y &amp;lt; k)$, just use the &lt;code&gt;pbinom&lt;/code&gt; function:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;##k=1000, n=2000 bernoulli, p=0.5&#xA;pbinom(1000, 2000, 0.5)&#xA;0.5089195&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, your scenario isn't that straightforward as $p$ isn't constant. Since your Bernoulli random variables are independent, I would replace $p$ with $\bar p$. Since $n$ is very large, the error associated with your estimate of $\bar p$ would be very small.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;In your case you might even consider using the &lt;a href=&quot;http://en.wikipedia.org/wiki/Binomial_distribution#Normal_approximation&quot; rel=&quot;nofollow&quot;&gt;Normal approximation&lt;/a&gt; to the binomial distribution:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pnorm(1000, 2000*0.5, 2000*0.25)&#xA;0.5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="8" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2010-12-10T11:52:43.960" LastActivityDate="2010-12-10T11:52:43.960" CommentCount="10" />
  <row Id="5349" PostTypeId="2" ParentId="5347" CreationDate="2010-12-10T12:06:50.643" Score="3" ViewCount="" Body="&lt;p&gt;With different $p_i$ your best bet I think is normal approximation. Let $B_n=\sum_{i=1}^np_i(1-p_i)$. Then&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align*}&#xA;B_n^{-1/2}\left(\sum_{i=1}^nX_i-\sum_{i=1}^np_i\right)\to N(0,1),&#xA;\end{align*}&#xA;as $n\to\infty$, provided that for each $\varepsilon&amp;gt;0$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align*}&#xA;B_n^{-1}\sum_{i=1}^nE\left((X_i-p_i)^2\mathbf{1}\{|X_i-p_i|&gt;\varepsilon B_n^{1/2}\}\right)\to 0,&#xA;\end{align*}&#xA;as $n\to\infty$, which for Bernouli variables will hold if $B_n\to\infty$. This is so called Lindeberg condition which is sufficient and necessary for convergence to the standard normal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Update:&#xA;The approximation error can be calculated from following inequality:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align*}&#xA;\sup_x|F_n(x)-\Phi(x)|\le AL_n,&#xA;\end{align*} &#xA;where&#xA;\begin{align*}&#xA;L_n=B_n^{-3/2}\sum_{i=1}^nE|X_i-p_i|^3&#xA;\end{align*}&#xA;and $F_n$ is the cdf of the scaled and centered sum of $X_i$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As whuber pointed out, the convergence can be slow for bad behaved $p_i$. For $p_i=\frac{1}{1+i}$ we have $B_n\approx \ln n$ and $L_n\approx (\ln n)^{-1/2}$. Then taking $n=2^{16}$ we get that maximum deviation from the standard normal cdf is a whopping 0.3.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2010-12-13T08:02:37.390" LastActivityDate="2010-12-13T08:02:37.390" CommentCount="9" />
  <row Id="5350" PostTypeId="2" ParentId="5344" CreationDate="2010-12-10T13:18:17.567" Score="12" ViewCount="" Body="&lt;p&gt;As chl pointed out, the main difference is what kind of variance-covariance structure you can specify for the random effects. In &lt;code&gt;lme4&lt;/code&gt; you can specify either:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;diagonal covariance structures (i.e., enforce mutually uncorrelated random effects via syntax like &lt;code&gt;~ (1 | group)+ (0 + x1 | group) + (0 + x2 | group)&lt;/code&gt;)&lt;/li&gt;&#xA;&lt;li&gt;or unstructured covariance matrices (i.e. all correlations are estimated, &lt;code&gt;~ (1  + x1 + x2 | group)&lt;/code&gt;)&lt;/li&gt;&#xA;&lt;li&gt;or partially diagonal, partially unstructured covariance (&lt;code&gt;y ~ (1 + x1 | group) + (0 + x2 | group)&lt;/code&gt;, where you would estimate a correlation between the random intercept and random slope for &lt;code&gt;x1&lt;/code&gt;, but no correlations between the random slope for &lt;code&gt;x2&lt;/code&gt; and the random intercept and between the random slope for &lt;code&gt;x2&lt;/code&gt; and the random slope for &lt;code&gt;x1&lt;/code&gt;).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;nlme&lt;/code&gt; offers a much broader class of covariance structures for the random effects. My experience is that the flexibility of &lt;code&gt;lme4&lt;/code&gt; is sufficient for most applications, however. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd also add a third difference in capabilities that may be more relevant for many longitudinal data situations: &lt;code&gt;nlme&lt;/code&gt;  let's you specify variance-covariance structures for the residuals (i.e. spatial or temporal autocorrelation or heteroskedasticity or covariate-dependent variability) in the &lt;code&gt;weights&lt;/code&gt; argument (c.f. &lt;code&gt;?varFunc&lt;/code&gt;), while &lt;code&gt;lme4&lt;/code&gt; only allows fixed prior weights for the observations. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A fourth difference is that it can be difficult to get &lt;code&gt;nlme&lt;/code&gt; to fit (partially) crossed random effects, while that's a non-issue in &lt;code&gt;lme4&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You'll probably be fine if you stick with &lt;code&gt;lme4&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1979" LastActivityDate="2010-12-10T13:18:17.567" CommentCount="1" />
  <row Id="5351" PostTypeId="1" AcceptedAnswerId="5428" CreationDate="2010-12-10T14:30:40.167" Score="5" ViewCount="512" Body="&lt;p&gt;I asked this &lt;a href=&quot;http://stackoverflow.com/questions/4403433/calculate-newy-west-standard-errors-without-an-an-lm-object-in-r&quot;&gt;question&lt;/a&gt; yesterday on StackOverflow, and got an answer, but we agreed that it seems a bit hackish and there may be a better way to look at it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The question: I would like calculate the Newey-West (HAC) standard errors for a vector (in this case a vector of stock returns). The function &lt;code&gt;NeweyWest()&lt;/code&gt; in the &lt;code&gt;sandwich&lt;/code&gt; package does this, but takes an &lt;code&gt;lm&lt;/code&gt; object as an input. The solution Joris Meys offered is to project the vector onto 1, which turns my vector into residuals to feed into &lt;code&gt;NeweyWest()&lt;/code&gt;. That is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;as.numeric(NeweyWest(lm(rnorm(100) ~ 1)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;for the variance of the mean.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should I be doing it like this? Or is there a way to more directly do what I want? Thanks!&lt;/p&gt;&#xA;" OwnerUserId="1445" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-12-11T17:18:10.267" LastActivityDate="2010-12-13T09:08:56.433" Title="Calculate Newey-West standard errors without an lm object in R" Tags="&lt;r&gt;&lt;standard-error&gt;&lt;autocorrelation&gt;&lt;heteroscedasticity&gt;" AnswerCount="1" CommentCount="4" />
  <row Id="5352" PostTypeId="1" CreationDate="2010-12-10T15:17:37.817" Score="3" ViewCount="78" Body="&lt;p&gt;I have a dataset where the difficulty level of my items and skill of my participants is unknown, but I would like to be able to extract some metric of how individual participants do on the test. What comes to mind initially is to use an item response theory model.  However, I have two a priori constraints that makes me question whether IRT will be able to adapt to my peculiar case, or whether there is a better approach to use.  As there are two constraints, I will split them into two questions.  If these questions are too simple, please refer me to an entry level IRT text which will provide the answers.  The first question is this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know that the difficulty of my items increases monotonically, by which I mean I know item A is harder than item B, and so on, but I do not know the extent to which item A is harder than item B. Is there a way to construct an item response theory model to adjust the model so that the item difficulty parameters are forced to reflect this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S. I was unable to think of appropriate tags for this question, if you have some suggestions please either make the edits yourself or suggest them in comments.&lt;/p&gt;&#xA;" OwnerUserId="196" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-12-10T21:50:32.603" LastActivityDate="2010-12-10T21:50:32.603" Title="Is there a way to place a priori constraints on item difficulty under item response theory?" Tags="&lt;psychometrics&gt;&lt;item-response-model&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="1" />
  <row Id="5353" PostTypeId="2" ParentId="5352" CreationDate="2010-12-10T16:15:26.540" Score="4" ViewCount="" Body="&lt;p&gt;Off the shelf, the various software offerings from SSI may do what you want. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I wanted to enforce the constraint, and I didn't have &lt;em&gt;too&lt;/em&gt; much data, then I'd switch into a BUGS / JAGS MCMC framework for modelling.  I'd use the LSAT example in book 2 of the worked examples (contained in the JAGS distribution or as a separate download from the BUGS people) as the code for standard '3 parameter' IRT model.  Then I'd add a line imposing the parameter constraint as described on p.35 of the current JAGS manual.  I'd then curl up with a good book, since convergence on such models tends to take ages. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or I might not bother.  Standard errors for such parameters tend to be pretty tiny for a respectable number of subjects, so the ordering constraint may not make any difference to your estimation certainty.  And if your parameters end up sitting on top of one another, they're probably trying to tell you your prior is wrong.&lt;/p&gt;&#xA;" OwnerUserId="1739" LastActivityDate="2010-12-10T16:15:26.540" CommentCount="3" />
  <row Id="5354" PostTypeId="1" AcceptedAnswerId="5355" CreationDate="2010-12-10T16:28:53.980" Score="5" ViewCount="573" Body="&lt;p&gt;I've got some data about airline flights (in a data frame called &lt;code&gt;flights&lt;/code&gt;) and I would like to see if the flight time has any effect on the probability of a significantly delayed arrival (meaning 10 or more minutes). I figured I'd use logistic regression, with the flight time as the predictor and whether or not each flight was significantly delayed (a bunch of Bernoullis) as the response. I used the following code...&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;flights$BigDelay &amp;lt;- flights$ArrDelay &amp;gt;= 10&#xA;delay.model &amp;lt;- glm(BigDelay ~ ArrDelay, data=flights, family=binomial(link=&quot;logit&quot;))&#xA;summary(delay.model)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;...but got the following output.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; flights$BigDelay &amp;lt;- flights$ArrDelay &amp;gt;= 10&#xA;&amp;gt; delay.model &amp;lt;- glm(BigDelay ~ ArrDelay, data=flights, family=binomial(link=&quot;logit&quot;))&#xA;Warning messages:&#xA;1: In glm.fit(x = X, y = Y, weights = weights, start = start, etastart = etastart,  :&#xA;  algorithm did not converge&#xA;2: In glm.fit(x = X, y = Y, weights = weights, start = start, etastart = etastart,  :&#xA;  fitted probabilities numerically 0 or 1 occurred&#xA;&amp;gt; summary(delay.model)&#xA;&#xA;Call:&#xA;glm(formula = BigDelay ~ ArrDelay, family = binomial(link = &quot;logit&quot;),&#xA;    data = flights)&#xA;&#xA;Deviance Residuals:&#xA;       Min          1Q      Median          3Q         Max&#xA;-3.843e-04  -2.107e-08  -2.107e-08   2.107e-08   3.814e-04&#xA;&#xA;Coefficients:&#xA;            Estimate Std. Error z value Pr(&amp;gt;|z|)&#xA;(Intercept)  -312.14     170.26  -1.833   0.0668 .&#xA;ArrDelay       32.86      17.92   1.833   0.0668 .&#xA;---&#xA;Signif. codes:  0 â***â 0.001 â**â 0.01 â*â 0.05 â.â 0.1 â â 1&#xA;&#xA;(Dispersion parameter for binomial family taken to be 1)&#xA;&#xA;    Null deviance: 2.8375e+06  on 2291292  degrees of freedom&#xA;Residual deviance: 9.1675e-03  on 2291291  degrees of freedom&#xA;AIC: 4.0092&#xA;&#xA;Number of Fisher Scoring iterations: 25&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What does it mean that the algorithm did not converge? I thought it be because the &lt;code&gt;BigDelay&lt;/code&gt; values were &lt;code&gt;TRUE&lt;/code&gt; and &lt;code&gt;FALSE&lt;/code&gt; instead of &lt;code&gt;0&lt;/code&gt; and &lt;code&gt;1&lt;/code&gt;, but I got the same error after I converted everything. Any ideas?&lt;/p&gt;&#xA;" OwnerUserId="1973" LastActivityDate="2010-12-10T17:34:21.217" Title="Logistic regression model does not converge" Tags="&lt;r&gt;&lt;logistic-regression&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="5355" PostTypeId="2" ParentId="5354" CreationDate="2010-12-10T16:36:49.853" Score="7" ViewCount="" Body="&lt;p&gt;First thought: &lt;em&gt;Perfect separation&lt;/em&gt;, meaning the predictor is 'too good', the logits go to +/- infinity and everything falls over.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second thought: Does the code do what you think it does? Your variable names don't seem to quite match your description.  You might elaborate what the data is more precisely, since it looks like you &lt;em&gt;might&lt;/em&gt; be trying to predict something with itself. &lt;/p&gt;&#xA;" OwnerUserId="1739" LastActivityDate="2010-12-10T16:36:49.853" CommentCount="4" />
  <row Id="5356" PostTypeId="2" ParentId="5354" CreationDate="2010-12-10T17:00:15.503" Score="5" ViewCount="" Body="&lt;p&gt;&lt;code&gt;glm()&lt;/code&gt; uses an iterative re-weighted least squares algorithm. The algorithm hit the maximum number of allowed iterations before signalling convergence. The default, documented in &lt;code&gt;?glm.control&lt;/code&gt; is 25. You pass control parameters as a list in the &lt;code&gt;glm&lt;/code&gt; call:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;delay.model &amp;lt;- glm(BigDelay ~ ArrDelay, data=flights, family=binomial,&#xA;                   control = list(maxit = 50))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As @Conjugate Prior says, you seem to be predicting the response with the data used to generate it. You have complete separation as any &lt;code&gt;ArrDelay &amp;lt; 10&lt;/code&gt; will predict &lt;code&gt;FALSE&lt;/code&gt; and any &lt;code&gt;ArrDelay &amp;gt;= 10&lt;/code&gt; will predict &lt;code&gt;TRUE&lt;/code&gt;. The other warning message tells you that the fitted probabilities for some observations were effectively 0 or 1 and that is a good indicator you have something wrong with the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The two can warnings go hand in hand. The likelihood function can be quite flat when some $\hat{\beta}_i$ get large, as in your example. If you allow more iterations, the model coefficients will diverge further if you have a separation issue.&lt;/p&gt;&#xA;" OwnerUserId="1390" LastEditorUserId="1390" LastEditorDisplayName="" LastEditDate="2010-12-10T17:34:21.217" LastActivityDate="2010-12-10T17:34:21.217" />
  <row Id="5357" PostTypeId="2" ParentId="5340" CreationDate="2010-12-10T17:11:32.343" Score="6" ViewCount="" Body="&lt;p&gt;I mistrust all but the lowest-level functions in Excel, and for good reason: many procedures that go beyond simple arithmetic operations have flaws or errors and most of them are poorly documented.  This includes &lt;em&gt;all&lt;/em&gt; the probability distribution functions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Numerical flaws are inevitable due to limitations in floating point accuracy.  For example, no matter what platform you use, if it uses double precision then don't try to compute the tail probability of a standard normal distribution for z = 50: the value equals $10^{-545}$, which underflows.  However, using NORMSDIST you'll get increasingly bad values once |z| exceeds 3 or so, and once |z| exceeds 8 Excel gives up and just returns zero.  Here is a tabulation of some of the errors (compared to Mathematica's answers):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\eqalign{&#xD;&#xA; Z &amp;amp;\quad 10^6\text{(Excel - True)/True} \cr&#xD;&#xA; -3 &amp;amp;\quad 50 \cr&#xD;&#xA; -4 &amp;amp;\quad 468 \cr&#xD;&#xA; -5 &amp;amp;\quad 1580\cr&#xD;&#xA; -6 &amp;amp;\quad 3582\cr&#xD;&#xA; -7 &amp;amp;\quad 6462\cr&#xD;&#xA; -8 &amp;amp;\quad 70789\cr&#xD;&#xA; -9 &amp;amp;\quad -1000000&#xD;&#xA;}$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, if you &lt;em&gt;must&lt;/em&gt; use Excel's statistical functions, severely limit the ones you do use; learn their flaws and foibles; work around those problems; and use the ones you are familiar with as building blocks for everything else.  In this spirit, I recommend computing lognormal probabilities in terms of NORMSDIST: just apply it to the log of the argument.  Specifically, in place of &lt;strong&gt;LOGNORMDIST(z, mu, sigma)&lt;/strong&gt; use &lt;strong&gt;NORMSDIST((LN(z) - mu)/sigma)&lt;/strong&gt;.  Once you have verified that these expressions do return the same values, you have also established exactly what LOGNORMDIST does: there's little possibility of confusion.  In particular, you can see that mu is the mean of the logarithms, not the geometric mean, and that sigma is the SD of the logs, not the geometric SD.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-12-10T17:11:32.343" CommentCount="1" />
  <row Id="5358" PostTypeId="2" ParentId="5347" CreationDate="2010-12-10T18:23:10.457" Score="8" ViewCount="" Body="&lt;p&gt;If it often resembles a &lt;a href=&quot;http://en.wikipedia.org/wiki/Poisson_distribution&quot; rel=&quot;nofollow&quot;&gt;Poisson&lt;/a&gt;, have you tried approximating it by a Poisson with parameter $\lambda = \sum p_i$ ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt;: I've found a theoretical result to justify this, as well as a name for the distribution of $Y$: it's called the &lt;a href=&quot;http://en.wikipedia.org/wiki/Poisson_binomial_distribution&quot; rel=&quot;nofollow&quot;&gt;Poisson binomial distribution&lt;/a&gt;. &lt;a href=&quot;http://en.wikipedia.org/wiki/Le_Cam%27s_theorem&quot; rel=&quot;nofollow&quot;&gt;Le Cam's inequality&lt;/a&gt; tells you how closely its distribution is approximated by the distribution of a Poisson with parameter $\lambda = \sum p_i$. It tells you the quality of this approx is governed by the sum of the squares of the $p_i$s, to paraphrase &lt;a href=&quot;http://www.jstor.org/stable/2325124&quot; rel=&quot;nofollow&quot;&gt;Steele (1994)&lt;/a&gt;. So if &lt;em&gt;all&lt;/em&gt; your $p_i$s are reasonably small, as it now appears they are, it should be a pretty good approximation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT 2&lt;/strong&gt;: How small is 'reasonably small'? Well, that depends how good you need the approximation to be! The &lt;a href=&quot;http://en.wikipedia.org/wiki/Le_Cam%27s_theorem&quot; rel=&quot;nofollow&quot;&gt;Wikipedia article on Le Cam's theorem&lt;/a&gt; gives the precise form of the result I referred to above: the sum of the absolute differences between the &lt;a href=&quot;http://en.wikipedia.org/wiki/Probability_mass_function&quot; rel=&quot;nofollow&quot;&gt;probability mass function&lt;/a&gt; (pmf) of $Y$ and the pmf of the above Poisson distribution is no more than twice the sum of the squares of the $p_i$s. Another result from &lt;a href=&quot;http://www.projecteuclid.org/euclid.pjm/1103038058&quot; rel=&quot;nofollow&quot;&gt;Le Cam (1960)&lt;/a&gt; may be easier to use: this sum is also no more than 18 times the largest $p_i$. There are quite a few more such results... see &lt;a href=&quot;http://www.jstor.org/stable/2030354&quot; rel=&quot;nofollow&quot;&gt;Serfling (1978)&lt;/a&gt; for one review.&lt;/p&gt;&#xA;" OwnerUserId="449" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-12-12T17:20:27.390" LastActivityDate="2010-12-12T17:20:27.390" CommentCount="13" />
  <row Id="5359" PostTypeId="1" AcceptedAnswerId="5365" CreationDate="2010-12-10T18:26:17.983" Score="2" ViewCount="130" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Diagonal elements of Psi (...)&#xA;   represent independent noise variances&#xA;   for each of the variables &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;C.M. Bishop, &lt;em&gt;Pattern Recognition and Machine Learning&lt;/em&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;...but I'm not clear on what does the Psi signify.&#xA;I've applied FA to a series of images, received Psi matrix and now struggling with interpreting the results.&lt;/p&gt;&#xA;" OwnerUserId="1371" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-11T12:52:08.337" LastActivityDate="2010-12-11T12:52:08.337" Title="What does the Psi term in factor analysis signify?" Tags="&lt;factor-analysis&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="5360" PostTypeId="1" AcceptedAnswerId="5361" CreationDate="2010-12-10T18:42:31.760" Score="8" ViewCount="511" Body="&lt;p&gt;I am fitting a stepwise logistic regression on a set of data in SPSS.  In the procedure, I am fitting my model to a random subset that is approx. 60% of the total sample, which is about 330 cases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I find interesting is that every time I re-sample my data, I am getting different variables popping in and out in the final model.  A few predictors are always present in the final model, but others pop in and out depending on the sample.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is this.  What is the best way to handle this?  I was hoping to see the convergence of predictor variables, but that isn't the case.  Some models make much more intuitive sense from an operational view (and would be easier to explain to the decision makers), and others fit the data slightly better.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In short, since variables are shuffling around, how would you recommend dealing with my situation?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="569" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-11T12:56:21.487" LastActivityDate="2010-12-12T11:40:31.173" Title="Stepwise logistic regression and sampling" Tags="&lt;logistic-regression&gt;&lt;spss&gt;&lt;stepwise-regression&gt;" AnswerCount="4" FavoriteCount="2" />
  <row Id="5361" PostTypeId="2" ParentId="5360" CreationDate="2010-12-10T19:05:02.207" Score="9" ViewCount="" Body="&lt;p&gt;Don't resample!  Create one random subsample once and for all.  Perform your analysis on it.  Validate the results against the held-out data.  It's likely most of the &quot;significant&quot; variables will turn out not to be significant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Focus on the variables that make sense, not those that fit the data a little better.  If you have more than a handful of variables for 330 records, you're at great risk of overfitting in the first place.  Consider using fairly severe entering and leaving criteria for the stepwise regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(I presume you have already carried out the analysis and exploration to identify appropriate re-expressions of the independent variables, that you have identified likely interactions, and that you have established that there really is an approximately linear relationship between the logit of the dependent variable and the regressors.  If not, do this essential preliminary work and only then return to the stepwise regression.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Be cautious about following generic advice like I just gave, by the way :-).  Your approach should depend on the purpose of the analysis (prediction?  extrapolation?  scientific understanding?  decision making?) as well as the nature of the data, the number of variables, etc.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-12-10T19:05:02.207" CommentCount="4" />
  <row Id="5362" PostTypeId="2" ParentId="5278" CreationDate="2010-12-10T19:15:07.527" Score="4" ViewCount="" Body="&lt;p&gt;Hello everyone.  Thank you for your great answer so far.  While they where very enlightening, I felt that using them for the course I am currently teaching (well, TA'ing) will be too much for my students. (I help teach the course BioStatistics for students from advanced degrees in medicine sciences)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, I ended up creating two images (Both are simulation based) which I think are useful example for explaining ANOVA.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;I would be happy to read comments or suggestions for improving them.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first image shows a simulation of 30 data points, separated to 3 plots (showing how the MST=Var is separated to the data that creates MSB and MSW:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The left plot shows a scatter plot of the data per group.  &lt;/li&gt;&#xA;&lt;li&gt;The middle one shows how the data we are going to use for MSB looks like.&lt;/li&gt;&#xA;&lt;li&gt;The right image shows how the data we are going to use for MSW looks like.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/wOyBm.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second image shows 4 plots, each one for a different combination of variance and expectancy for the groups while&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The first row of plots is for low variance, while the second row is for high(er) variance.&lt;/li&gt;&#xA;&lt;li&gt;The first column of plots is for equal expectancy between the groups, while the second column shows groups with (very) different expectancies.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/vnI1Q.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="253" LastEditorUserId="253" LastEditorDisplayName="" LastEditDate="2010-12-10T20:26:42.770" LastActivityDate="2010-12-10T20:26:42.770" CommentCount="3" />
  <row Id="5363" PostTypeId="1" CreationDate="2010-12-10T19:21:44.130" Score="2" ViewCount="153" Body="&lt;p&gt;Dear Room,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I got a slight confusion on the back propagation algorithm used in MLP. &#xA;The error is adjusted by the cost function. In BP, we are trying to adjust the weight of the hidden layers. The question is:&lt;br&gt;&#xA;(a) The output error I can understand, i.e., e=d-y.[Without the subscripts].&lt;br&gt;&#xA;(b) My question is how one gets the error of hidden layer. How to calculate it?&lt;br&gt;&#xA;(c) If I backpropagate it, should I use it as a cost function of adaptive filter or should I use a pointer (in C/C++) programming sense, to update the weight?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If any learned member of the room can answer this silly question, I'd be hugely helpful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Best Regards,&#xA;Subhabrata Banerjee,&#xA;Gurgaon,&#xA;Haryana,&#xA;India.&lt;/p&gt;&#xA;" OwnerUserId="2329" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-11T12:54:38.740" LastActivityDate="2010-12-12T11:13:06.010" Title="Backpropagation algorithm" Tags="&lt;machine-learning&gt;&lt;neural-networks&gt;" AnswerCount="1" CommentCount="3" FavoriteCount="1" />
  <row Id="5364" PostTypeId="1" AcceptedAnswerId="5385" CreationDate="2010-12-10T20:19:19.707" Score="4" ViewCount="274" Body="&lt;p&gt;The underlying model of &lt;a href=&quot;http://en.wikipedia.org/wiki/Partial_least_squares_regression&quot; rel=&quot;nofollow&quot;&gt;PLS&lt;/a&gt; is that a given $n \times m$ matrix $X$ and $n$ vector $y$ are related by &#xA;$$X = T P&amp;#39; + E,$$&#xD;&#xA;$$y = T q&amp;#39; + f,$$&#xD;&#xA;where $T$ is a latent $n \times k$ matrix, and $E, f$ are noise terms (sssuming $X, y$ are centered). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;PLS produces estimates of $T, P, q$, and a 'shortcut' vector of regression coefficients, $\hat{\beta}$ such that $y \sim X \hat{\beta}$. I would like to find the distribution of $\hat{\beta}$ under some simplifying assumptions, which should probably include the following:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The model is correct, &lt;em&gt;i.e.&lt;/em&gt; $X = T P&amp;#39; + E,y = T q&amp;#39; + f$ for unknown $T, P, q$;&lt;/li&gt;&#xA;&lt;li&gt;The number of latent factors, $k$, is known, and used in the PLS algorithm;&lt;/li&gt;&#xA;&lt;li&gt;The actual error terms are i.i.d. zero-mean normal with known variances;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;This question is somewhat underdefined because there are scores of variants of 'the' PLS algorithm, but I would accept results for any of them. I would also accept guidance on how to estimate the distribution of $\hat{\beta}$ via &lt;em&gt;e.g.&lt;/em&gt; a bootstrap, but perhaps that is a separate question.&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-12-16T22:27:47.410" Title="How to compute the confidence intervals on regression coefficients in PLS?" Tags="&lt;regression&gt;&lt;confidence-interval&gt;&lt;latent-variable&gt;&lt;pls&gt;" AnswerCount="2" CommentCount="0" />
  <row Id="5365" PostTypeId="2" ParentId="5359" CreationDate="2010-12-10T20:34:45.023" Score="6" ViewCount="" Body="&lt;p&gt;Following Bishop's notation, the FA model is written as (Eq. 12.64, p. 584):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;p(x|z)=\mathcal{N}(x|\mathbf{W}z+\boldsymbol{\mu},\boldsymbol{\psi})&#xD;&#xA;$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\boldsymbol{\psi}$ is a $D\times D$ diagonal matrix of so-called variable uniquenesses, that is the variance not accounted for by the latent factors, whereas $\mathbf{W}$ reflects factor loadings $\lambda_i$, that is the correlation of variable $i$ with factors represented in $z$ (more exactly, the square of $\lambda_i$ is the variance explained by the latent factor).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're not familiar with the FA literature, I would suggest &quot;lighter&quot; approach, e.g. William Revelle has good tutorials on his website &lt;a href=&quot;http://www.personality-project.org&quot; rel=&quot;nofollow&quot;&gt;personality-project.org&lt;/a&gt;; especially, I would suggest Chapter 6 of his forthcoming book on Psychometric methods entitled &lt;a href=&quot;http://personality-project.org/r/book/Chapter6.pdf&quot; rel=&quot;nofollow&quot;&gt;Constructs, Components, and Factor models&lt;/a&gt;. You will shortly understand the relations between PCA and FA. Specifically, with PCA, we are constructing linear combinations of observed variables (this yields a composite variable), whereas in FA we are expressing each variable as a weighted combination of hypothesized latent factors (where weights are called loadings) plus an error term (the $\boldsymbol{\psi}$ in the above formula). In sum, the FA model incorporates a model for noise--this is what is expressed in Equation 12.65; but see &lt;a href=&quot;http://stats.stackexchange.com/questions/1576/what-are-the-differences-between-factor-analysis-and-principal-component-analysis&quot;&gt;What are the differences between Factor Analysis and Principal Component Analysis&lt;/a&gt;, for additional discussion.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-12-10T20:42:01.910" LastActivityDate="2010-12-10T20:42:01.910" CommentCount="2" />
  <row Id="5366" PostTypeId="1" CreationDate="2010-12-10T20:53:42.207" Score="4" ViewCount="354" Body="&lt;p&gt;I need to cluster units into $k$ clusters to minimize within-group sum of squares (WSS), but I need to ensure that the clusters each contain at least $m$ units.  Any idea if any of R's clustering functions allow for clustering into $k$ clusters subject to a minimum cluster size constraint?  kmeans() does not seem to offer a size constraint option.&lt;/p&gt;&#xA;" OwnerUserId="96" LastActivityDate="2010-12-11T15:52:15.773" Title="Clustering (k-means, or otherwise) with a minimum cluster size constraint" Tags="&lt;r&gt;&lt;clustering&gt;" AnswerCount="3" />
  <row Id="5367" PostTypeId="2" ParentId="5366" CreationDate="2010-12-10T21:31:02.173" Score="1" ViewCount="" Body="&lt;p&gt;I think it would just be a matter of running the k means as part of an if loop with a test for cluster sizes, I.e. Count n in cluster k - also remember that k means will give different results for each run on the same data so you should probably be running it as part of a loop anyway to extract the &quot;best&quot; result&lt;/p&gt;&#xA;" OwnerUserId="2333" LastActivityDate="2010-12-10T21:31:02.173" CommentCount="1" />
  <row Id="5368" PostTypeId="1" CreationDate="2010-12-11T03:34:15.997" Score="2" ViewCount="216" Body="&lt;p&gt;I think that after apply PCA the covariance between components is reduced since the aim of PCA is to maximize variance.  Covariance is needed by Factor analysis and thus it does not make sense to operate on the data in PCA space.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does this make any sense? Does someone have any other explanation? &lt;/p&gt;&#xA;" OwnerUserId="2337" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-11T12:50:40.273" LastActivityDate="2010-12-13T04:35:59.257" Title="Why would it not make sense to apply factor analysis after PCA?" Tags="&lt;pca&gt;&lt;factor-analysis&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="5369" PostTypeId="2" ParentId="5368" CreationDate="2010-12-11T06:27:19.523" Score="4" ViewCount="" Body="&lt;p&gt;PCA actually ends up with orthogonal variables, therefore the covariance between the components should be 0. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It does not make sense to do factor analysis which selects the factors from a dataset after a procedure that makes factors from the dataset.&lt;/p&gt;&#xA;" OwnerUserId="1808" LastActivityDate="2010-12-11T06:27:19.523" CommentCount="4" />
  <row Id="5370" PostTypeId="2" ParentId="5366" CreationDate="2010-12-11T06:33:24.350" Score="2" ViewCount="" Body="&lt;p&gt;Use EM Clustering&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In EM clustering, the algorithm iteratively refines an initial cluster model to fit the data and determines the probability that a data point exists in a cluster. The algorithm ends the process when the probabilistic model fits the data. The function used to determine the fit is the log-likelihood of the data given the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If empty clusters are generated during the process, or if the membership of one or more of the clusters falls below a given threshold, the clusters with low populations are reseeded at new points and the EM algorithm is rerun.&lt;/p&gt;&#xA;" OwnerUserId="1808" LastActivityDate="2010-12-11T06:33:24.350" CommentCount="1" />
  <row Id="5371" PostTypeId="2" ParentId="5366" CreationDate="2010-12-11T08:04:01.883" Score="0" ViewCount="" Body="&lt;p&gt;How large is your data set? Maybe you could try to run a hierarchical clustering and then decide which clusters retain based on your dendrogram.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your data set is huge, you could also combine both clustering methods: an initial non-hierarchical clustering and then a hierarchical clustering using the groups from the non-hierarchical analysis. You can find an example of this approach in &lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pubmed/15385419&quot; rel=&quot;nofollow&quot;&gt;Martínez-Pastor et al (2005)&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="221" LastEditorUserId="221" LastEditorDisplayName="" LastEditDate="2010-12-11T15:52:15.773" LastActivityDate="2010-12-11T15:52:15.773" CommentCount="1" />
  <row Id="5372" PostTypeId="1" AcceptedAnswerId="5383" CreationDate="2010-12-11T11:31:49.120" Score="1" ViewCount="129" Body="&lt;p&gt;I need to detect Out Of Shelf situations in reatil stores. What I had in mind is to assume that purchase frequency of some article (SKU) has certain ditribution (I tried with Poisson) and then calculate probability (according to that distribution) of that specific article was not bought since last purchase, and if that probability is below some treshold (let say 1%) then it is a possible OOS situation.&lt;br&gt;&#xA;My question is:&lt;br&gt;&#xA;What distributions are commonly used in such problems and is there any way that one can test for several distributions and than pick the best one? Thing is, this should be done automatically due to large number of SKU/store combinations to deal with.&lt;/p&gt;&#xA;" OwnerUserId="2341" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-11T18:57:33.810" LastActivityDate="2010-12-11T20:40:51.063" Title="What distribution should be used for the detection of out of shelf situations in a retail stores?" Tags="&lt;distributions&gt;&lt;probability&gt;&lt;poisson&gt;" AnswerCount="1" />
  <row Id="5373" PostTypeId="2" ParentId="5344" CreationDate="2010-12-11T11:45:13.167" Score="5" ViewCount="" Body="&lt;p&gt;Others have summarized the differences very well. My impression is that &lt;code&gt;lme4&lt;/code&gt; is more suited for clustered data sets especially when you need to use crossed random effects. For repeated measures designs (including many longitudinal designs) however, &lt;code&gt;nlme&lt;/code&gt; is the tool since only &lt;code&gt;nlme&lt;/code&gt; supports specifying a correlation structure for the residuals. You do it using the &lt;code&gt;correlations&lt;/code&gt; or &lt;code&gt;cor&lt;/code&gt; argument with a &lt;code&gt;corStruct&lt;/code&gt; object. It also allows you to model heteroscedasticity using a &lt;code&gt;varFunc&lt;/code&gt; object.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Notice: I don't have enough rep to comment on the previous answers that's why I'm writing this as an answer.&lt;/p&gt;&#xA;" OwnerUserId="2020" LastActivityDate="2010-12-11T11:45:13.167" />
  <row Id="5374" PostTypeId="2" ParentId="5333" CreationDate="2010-12-11T11:52:26.187" Score="5" ViewCount="" Body="&lt;p&gt;You're misinterpreting these results, which is easy to do as with mixed models there's more than one type of 'fitted value' and the documentation of &lt;code&gt;lmer&lt;/code&gt; isn't as clear as it might be. Try using &lt;code&gt;fixed.effects()&lt;/code&gt; in place of &lt;code&gt;fitted()&lt;/code&gt; and you should get correlations which makes more intuitive sense if you're interested in the contribution of the fixed effects. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;code&gt;fitted()&lt;/code&gt; function of &lt;code&gt;lmer&lt;/code&gt; is documented as giving the 'conditional means'. I had to check the Theory.pdf vignette to work out that these &lt;em&gt;include the predictions of the modelled random effects&lt;/em&gt;. Your modelled random effect variances are, overall, smaller in the model including the fixed effect. But smaller random effects mean less shrinkage, i.e. the predicted random effect is closer to the observed residual. When calculating the correlation, it seems that in your case this smaller shrinkage just overcomes the improvement from the fixed effect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The interpretation of $R^2$ as 'proportion of variance explained' gets more complex with mixed models, as it depends whether you think of random effects as 'explaining' variance. Probably not, in most cases.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-12-11T11:52:26.187" CommentCount="4" />
  <row Id="5375" PostTypeId="2" ParentId="5278" CreationDate="2010-12-11T13:19:29.733" Score="5" ViewCount="" Body="&lt;p&gt;How about something like this?&#xA;&lt;img src=&quot;http://i.imgur.com/OL0HP.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Following Crawley (2005). Statistics. An introduction using R: Wiley.&lt;/p&gt;&#xA;" OwnerUserId="1050" LastActivityDate="2010-12-11T13:19:29.733" CommentCount="1" />
  <row Id="5376" PostTypeId="2" ParentId="5278" CreationDate="2010-12-11T14:43:17.020" Score="1" ViewCount="" Body="&lt;p&gt;It seems the ship has already sailed in terms of an answer, but I think that if this is an introductory course that most of the displays offered here are going to be too difficult to grasp for introductory students... or at the very least too difficult to grasp without an introductory display which provides a very simplified explanation of partitioning variance.  Show them how SST total increases with the number of subjects.  Then after showing it inflate for several subjects (maybe adding one in each group several times), explain that SST = SSB + SSW (though I prefer to call it SSE from the outset because it avoids confusion when you go to the within subjects test IMO).  Then show them a visual representation of the variance partitioning, e.g. a big square color coded such that you can see how SST is made of SSB and SSW.  Then, graphs similar to Tals or EDi may become useful, but I agree with EDi that the scale should be SS rather than MS for pedagogical purposes when first explaining things.&lt;/p&gt;&#xA;" OwnerUserId="196" LastEditorUserId="196" LastEditorDisplayName="" LastEditDate="2010-12-12T04:24:44.690" LastActivityDate="2010-12-12T04:24:44.690" />
  <row Id="5377" PostTypeId="2" ParentId="5347" CreationDate="2010-12-11T17:05:23.770" Score="3" ViewCount="" Body="&lt;p&gt;Well, based on your description and the discussion in the comments it is clear that $Y$ has mean $\sum_i p_i$ and variance $\sum_i p_{i}(1-p_{i})$.  The shape of $Y$'s distribution will ultimately depend on the behavior of $p_i$.  For suitably &quot;nice&quot; $p_i$ (in the sense that not too many of them are really close to zero), the distribution of $Y$ will be approximately normal (centered right at $\sum p_i$).  But as  $\sum_i p_i$ starts heading toward zero the distribution will be shifted to the left and when it crowds up against the $y$-axis it will start looking a lot less normal and a lot more Poisson, as @whuber and @onestop have mentioned.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;From your comment &quot;the distribution looks Poisson&quot; I suspect that this latter case is what's happening, but can't really be sure without some sort of visual display or summary statistics about the $p$'s.  Note however, as @whuber did, that with sufficiently pathological behavior of the $p$'s you can have all sorts of spooky things happen, like limits that are mixture distributions.  I doubt that is the case here, but again, it really depends on what your $p$'s are doing. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As to the original question of &quot;how to efficiently model&quot;, I was going to suggest a hierarchical model for you but it isn't really appropriate if the $p$'s are fixed constants.  In short, take a look at a histogram of the $p$'s and make a first guess based on what you see.  I would recommend the answer by @mpiktas (and by extension @csgillespie) if your $p$'s aren't too crowded to the left, and I would recommend the answer by @onestop if they are crowded left-ly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By the way, here is the R code I used while playing around with this problem: the code isn't really appropriate if your $p$'s are too small, but it should be easy to plug in different models for $p$ (including spooky-crazy ones) to see what happens to the ultimate distribution of $Y$.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(1)&#xA;M &amp;lt;- 5000&#xA;N &amp;lt;- 15000&#xA;p &amp;lt;- rbeta(N, shape1 = 1, shape2 = 10)&#xA;Y &amp;lt;- replicate(M, sum(rbinom(N, size = 1, prob = p)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now take a look at the results.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;hist(Y)&#xA;mean(Y)&#xA;sum(p)&#xA;var(Y)&#xA;sum(p*(1 - p))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Have fun; I sure did.&lt;/p&gt;&#xA;" OwnerUserId="1108" LastActivityDate="2010-12-11T17:05:23.770" CommentCount="2" />
  <row Id="5378" PostTypeId="2" ParentId="5278" CreationDate="2010-12-11T17:33:56.597" Score="3" ViewCount="" Body="&lt;p&gt;Since we gather certain types of nice graphs in this post, here is another one that I recently found and may help you understand how ANOVA works and how the F statistic is generated. The graphic was created using the &lt;a href=&quot;http://rgm2.lab.nig.ac.jp/RGM2/func.php?rd_id=granova%3agranova.1w&quot; rel=&quot;nofollow&quot;&gt;granova&lt;/a&gt; package in R. &#xA;&lt;img src=&quot;http://i.imgur.com/oxOOi.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="339" LastActivityDate="2010-12-11T17:33:56.597" CommentCount="2" />
  <row Id="5379" PostTypeId="2" ParentId="5360" CreationDate="2010-12-11T18:04:50.730" Score="6" ViewCount="" Body="&lt;p&gt;In general, there are two problems of feature selection:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;em&gt;minimal optimal&lt;/em&gt;, where you seek for smallest set of variables that give you the smallest error&lt;/li&gt;&#xA;&lt;li&gt;&lt;em&gt;all relevant&lt;/em&gt;, where you seek for all variables relevant in a problem&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The convergence of predictor selection is in a domain of the all relevant problem, which is hell hard and thus requires much more powerful tools than logistic regression, heavy computations and a very careful treatment. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But it seems you are doing the first problem, so you shouldn't worry about this. I can generally second whuber's answer, but I disagree with the claim that you should drop resampling -- here it won't be a method to stabilize feature selection, but nevertheless it will be a simulation for estimating performance of a coupled feature selection + training, so will give you an insight in confidence of your accuracy.&lt;/p&gt;&#xA;" OwnerUserId="88" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-11T23:40:23.283" LastActivityDate="2010-12-11T23:40:23.283" CommentCount="1" />
  <row Id="5380" PostTypeId="2" ParentId="5360" CreationDate="2010-12-11T18:34:21.803" Score="5" ViewCount="" Body="&lt;p&gt;An important question is &quot;why do why do you want a model with as few variables a possible?&quot;.  If you want to have as few variables as possible to minimize the cost of data collection for the operational use of your model, then the answers given by whuber and mbq are an excellent start.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If predictive performance is what is really important, then you are probably better off not doing any feature selection at all and use regularized logistic regression instead (c.f. ridge regression).  In fact if predictive performance was what was of primary importance, I would use bagged regularized logistic regression as a sort of &quot;belt-and-braces&quot; strategy for avoiding over-fitting a small dataset.  Millar in his &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1584881712&quot; rel=&quot;nofollow&quot;&gt;book&lt;/a&gt; on subset selection in regression gives pretty much that advice in the appendix, and I have found it to be excellent advice for problems with lots of features and not very many observations.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If understanding the data is important, then there is no need for the model used to understand the data to be the same one used to make predictions.  In that case, I would resample the data many times and look at the patterns of selected variables across samples to find which variables were informative (as mbq suggests, if feature selection is unstable, a single sample won't give the full picture), but I would still used the bagged regularized logistic regression model ensemble for predictions.&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2010-12-11T18:34:21.803" CommentCount="3" />
  <row Id="5381" PostTypeId="2" ParentId="5360" CreationDate="2010-12-11T19:10:31.897" Score="3" ViewCount="" Body="&lt;p&gt;You might glance at the paper &lt;a href=&quot;http://www.stats.ox.ac.uk/~meinshau/stability.pdf&quot; rel=&quot;nofollow&quot;&gt;Stability Selection&lt;/a&gt; by Meinshausen and Buhlmann in J R Statist. Soc B (2010) 72 Part 4, and the discussion after it. They consider what happens when you repeatedly divide your set of data points at random into two halves and look for features in each half. By assuming that what you see in a one half is independent of what you see in the matching other half you can prove bounds on the expected number of falsely selected variables.&lt;/p&gt;&#xA;" OwnerUserId="1789" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-12-12T11:40:31.173" LastActivityDate="2010-12-12T11:40:31.173" CommentCount="1" />
  <row Id="5382" PostTypeId="1" CreationDate="2010-12-11T19:54:14.823" Score="2" ViewCount="237" Body="&lt;p&gt;Hi! I'm a new guy here. Hopefully, I'm asking this question to right forum.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Problem:&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;We have data of a group of people (P1, P2, P3). They rank their expertise (1-10, where higher number is better) in a list of components (G1, G2, G3).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    P1  P2  P3&#xA;--------------&#xA;G1 | 8   4   7&#xA;G2 | 7   3   7&#xA;G3 | 9   6   5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Also, we have some data regarding work done by each person in each component. Example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For P1,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;     W  WD&#xA;----------&#xA;G1 | 0   0&#xA;G2 | 2   0&#xA;G3 | 8   2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where &lt;code&gt;W&lt;/code&gt; is total work allotted to user P1, and &lt;code&gt;WD&lt;/code&gt; is actual work done. W &gt;= WD &gt;= 0.&#xA;We have similar data for P2, and P3 users.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Point to note:&lt;/strong&gt; The user might have some level of expertise regardless of work done in a component. Example: P1 has ranked himself 8/10 even though he has not been given any task in G1 component (W = 0). P1 also has ranked himself 7/10 for G2 even though he has not finished any task in that component (WD = 0).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, we want to calculate effective rating of all users relative to the group of users, not self-ranking, considering their work data and self-ranking.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone suggest some mechanism to achieve this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks much in advance!&lt;/p&gt;&#xA;" OwnerUserId="2344" LastEditorUserId="2344" LastEditorDisplayName="" LastEditDate="2010-12-11T20:26:26.090" LastActivityDate="2010-12-20T07:25:21.537" Title="Normalizing rating in a group of people [finding effectiveness]" Tags="&lt;normalization&gt;&lt;rating&gt;" AnswerCount="3" CommentCount="5" FavoriteCount="1" />
  <row Id="5383" PostTypeId="2" ParentId="5372" CreationDate="2010-12-11T20:40:51.063" Score="4" ViewCount="" Body="&lt;p&gt;Poisson makes a certain amount of sense if &quot;purchase frequency&quot; is e.g. number of items bought per day.  Negative binomial would also make sense (probably more sensible, as it will allow for more variation). If you have a huge amount of data, then you could just use the empirical distribution (i.e. across days, what is the &lt;em&gt;empirical&lt;/em&gt; frequency of &amp;lt;= this number being bought?)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A couple of things that make sense to consider in this context (don't know how much effort you are going to be putting into this, nor what your statistical background is): (1) it would make sense to fit a mixed model to allow for variation among stores, SKUs, days, etc.; (2) you might want to use more than just zeros -- i.e. if there were really only 2 items there to begin with, then there will only be 2 bought, and you could potentially detect that that outcome was highly unlikely for that particular SKU/store/day combination ... and hence detect the OOS issue before the next day (when there were zero and hence zero bought).&lt;/p&gt;&#xA;" OwnerUserId="2126" LastActivityDate="2010-12-11T20:40:51.063" CommentCount="5" />
  <row Id="5384" PostTypeId="2" ParentId="5114" CreationDate="2010-12-11T20:49:36.103" Score="1" ViewCount="" Body="&lt;p&gt;Since the log-normal distribution has two parameters, you can't satisfactorily fit it to three constraints that don't naturally fit it. With extreme quantiles of 2.5 and 7.5, the mode is ~4, and there is not much you can do about it. Since the scale of the errors for &lt;code&gt;a&lt;/code&gt; and &lt;code&gt;b&lt;/code&gt; is much smaller than for &lt;code&gt;c&lt;/code&gt;, one of them will be pretty much ignored during the optimization.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a better fit, you can either pick a three-parameter distribution, for example the generalized gamma distribution (implemented in the &lt;code&gt;VGAM&lt;/code&gt; package), or add a shift parameter to the lognormal (or gamma, ...) distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a last note, since the distribution you are looking for is clearly not symmetric, the average of the two given observations is not the right value for the mode. I would  maximize the sum of the densities at 3.0 and 3.6 while maintaining the extreme quantiles at 2.5 and 7.5 - this is possible if you have three parameters.&lt;/p&gt;&#xA;" OwnerUserId="279" LastActivityDate="2010-12-11T20:49:36.103" />
  <row Id="5385" PostTypeId="2" ParentId="5364" CreationDate="2010-12-11T21:07:43.147" Score="4" ViewCount="" Body="&lt;p&gt;Do you know this article: &lt;a href=&quot;http://www.iasbs.ac.ir/chemistry/chemometrics/history/4th/5.pdf&quot; rel=&quot;nofollow&quot;&gt;PLS-regression: a basic tool of chemometrics&lt;/a&gt;? Deriving SE and CI for the PLS parameters is described in §3.11. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I generally rely on Bootstrap for computing CIs, as suggested in e.g., Abdi, H. &lt;a href=&quot;http://www.utdallas.edu/~herve/abdi-wireCS-PLS2010.pdf&quot; rel=&quot;nofollow&quot;&gt;Partial least squares regression and projection on latent structure regression (PLS Regression)&lt;/a&gt;. I seem to remember there are theoretical solutions discussed in Tenenhaus M. (1998) &lt;em&gt;La régression PLS: Théorie et pratique&lt;/em&gt; (Technip), but I cannot check for now as I don't have the book. For now, there are some useful R packages, like &lt;a href=&quot;http://cran.r-project.org/web/packages/plsRglm/index.html&quot; rel=&quot;nofollow&quot;&gt;plsRglm&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;P.S.&lt;/em&gt; I just discovered &lt;a href=&quot;http://128.84.158.119/pdf/1002.4112v1&quot; rel=&quot;nofollow&quot;&gt;Nicole Krämer&lt;/a&gt;'s article, in reference to the &lt;a href=&quot;http://cran.r-project.org/web/packages/plsdof/index.html&quot; rel=&quot;nofollow&quot;&gt;plsdof&lt;/a&gt; R package.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-12-11T21:07:43.147" />
  <row Id="5387" PostTypeId="1" AcceptedAnswerId="5398" CreationDate="2010-12-11T22:50:26.880" Score="3" ViewCount="447" Body="&lt;p&gt;In a logit model, is there a smarter way to determine the effect of an independent ordinal variable than to use dummy variables for each level?&lt;/p&gt;&#xA;" OwnerUserId="82" LastEditorUserId="82" LastEditorDisplayName="" LastEditDate="2010-12-11T23:15:04.053" LastActivityDate="2010-12-12T11:26:47.150" Title="Logit with ordinal independent variables" Tags="&lt;logistic-regression&gt;&lt;logit&gt;&lt;ordinal&gt;" AnswerCount="2" CommentCount="3" FavoriteCount="1" />
  <row Id="5388" PostTypeId="2" ParentId="5387" CreationDate="2010-12-11T23:29:31.963" Score="4" ViewCount="" Body="&lt;p&gt;it's perfectly fine to use a categorical predictor in a logit (or OLS) regression model if the levels are ordinal. But if you have a reason to treat each level as discrete (or if in fact your categorical variable is nominal rather than ordinal), then, as alternative to dummy coding, you can also use orthogonal contrast coding.  For very complete &amp;amp; accessible discussion, see Judd, C.M., McClelland, G.H. &amp;amp; Ryan, C.S. Data analysis : a model comparison approach, Edn. 2nd. (Routledge/Taylor and Francis, New York, NY; 2008), or just google &quot;contrast coding&quot;&lt;/p&gt;&#xA;" OwnerUserId="2347" LastActivityDate="2010-12-11T23:29:31.963" CommentCount="3" />
  <row Id="5389" PostTypeId="2" ParentId="5347" CreationDate="2010-12-11T23:50:58.420" Score="5" ViewCount="" Body="&lt;p&gt;@onestop provides good references.  The Wikipedia article on the Poisson binomial distribution gives a recursive formula for computing the exact probability distribution; it requires $O(n^2)$ effort.  Unfortunately, it's an alternating sum, so it will be numerically unstable: it's hopeless to do this computation with floating point arithmetic.  Fortunately, when the $p_i$ are small, you only need to compute a small number of probabilities, so the effort is really proportional to $O(n \log(\sum_i{p_i}))$.  The precision necessary to carry out the calculation with rational arithmetic (&lt;em&gt;i.e., exactly,&lt;/em&gt; so that the numerical instability is not a problem) grows slowly enough that the overall timing may still be approximately $O(n^2)$.  That's feasible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a test, I created an array of probabilities $p_i = 1/(i+1)$ for various values of $n$ up to $n = 2^{16}$, which is the size of this problem.  For small values of $n$ (up to $n = 2^{12}$) the timing for the exact calculation of probabilities was in seconds and scaled quadratically, so I ventured a calculation for $n = 2^{16}$ out to three SDs above the mean (probabilities for 0, 1, ..., 22 successes).  It took 80 minutes (with Mathematica 8), in line with the predicted time.  (The resulting probabilities are fractions whose numerators and denominators have about 75,000 digits apiece!)  This shows the calculation can be done.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An alternative is to run a long simulation (a million trials ought to do).  It only has to be done once, because the $p_i$ don't change.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-12-11T23:50:58.420" CommentCount="1" />
  <row Id="5390" PostTypeId="2" ParentId="1870" CreationDate="2010-12-12T01:27:00.017" Score="1" ViewCount="" Body="&lt;p&gt;In youtube there are a lot of videos related with SPSS, I usually learn a gui with them. I recommend you record videos to teach how to apply methods and encourage them to use the &quot;paste&quot; button to learn a little of syntax&lt;/p&gt;&#xA;" OwnerUserId="2348" LastEditorUserId="2348" LastEditorDisplayName="" LastEditDate="2010-12-12T01:39:25.333" LastActivityDate="2010-12-12T01:39:25.333" CommunityOwnedDate="2010-12-12T01:27:00.017" />
  <row Id="5391" PostTypeId="2" ParentId="1870" CreationDate="2010-12-12T02:13:30.870" Score="1" ViewCount="" Body="&lt;p&gt;Your method will certainly work for most introductory techniques you'd need to do. I am quite familiar with R, but I was required to learn Minitab to teach to non-statistics people. In one afternoon, I had a basic enough understanding of Minitab to explain how to use it in the limited framework of an introductory stats class. I figure SPSS would behave similarly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would suggest reviewing the course syllabus and trying to complete all techniques outlined in the syllabus. A large majority of your students' questions will pertain to doing tasks the teacher has asked them to do, and a smaller percentage of questions will be related to doing tasks in their other statistics endeavors. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You will be able to answer a large majority of student questions by knowing how to use SPSS to solve a class-related problem, so just playing around within SPSS will benefit you.&lt;/p&gt;&#xA;" OwnerUserId="1118" LastActivityDate="2010-12-12T02:13:30.870" CommunityOwnedDate="2010-12-12T02:13:30.870" />
  <row Id="5392" PostTypeId="1" AcceptedAnswerId="5407" CreationDate="2010-12-12T03:21:44.353" Score="6" ViewCount="154" Body="&lt;p&gt;Hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Standard deviation of random variable $A =\sigma_{1}=5$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Standard deviation of random variable $B=\sigma_{2}=4$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then the variance of A+B is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Var(w_{1}A+w_{2}B)= w_{1}^{2}\sigma_{1}^{2}+w_{2}^{2}\sigma_{2}^{2} +2w_{1}w_{2}p_{1,2}\sigma_{1}\sigma_{2}$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$p_{1,2}$ is the correlation between the two random variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$w_{1}$ is the weight of random variable A&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$w_{2}$ is the weight of random variable B&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$w_{1}+w_{2}=1$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The figure below plots the variance of A and B as the weight of A changes from 0 to 1, for the correlations -1 (yellow),0 (blue) and 1 (red).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/ATJSF.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How did the formula result in a straight line (red) when the correlation is 1? As far as I can tell, when $p_{1,2}=1$, the formula simplifies to:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Var(w_{1}A+w_{2}B)= w_{1}^{2}\sigma_{1}^{2}+w_{2}^{2}\sigma_{2}^{2} +2w_{1}w_{2}\sigma_{1}\sigma_{2}$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I express that in the form of $y=mx+c$?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you.&lt;/p&gt;&#xA;" OwnerUserId="1636" LastEditorUserId="1636" LastEditorDisplayName="" LastEditDate="2010-12-12T10:40:35.683" LastActivityDate="2010-12-12T21:40:58.743" Title="Variance of two weighted random variables" Tags="&lt;random-variable&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="5393" PostTypeId="2" ParentId="5347" CreationDate="2010-12-12T03:38:17.960" Score="1" ViewCount="" Body="&lt;p&gt;Hi David:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think other answers are great, but I didn't see any Bayesian ways of estimating your probability. The answer doesn't have an explicit form, but the probability can be simulated using R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the attempt:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ X_i | p_i \sim Ber(p_i)$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ p_i \sim Beta(\alpha, \beta) $$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using &lt;a href=&quot;http://en.wikipedia.org/wiki/Beta_distribution&quot; rel=&quot;nofollow&quot;&gt;wikipedia&lt;/a&gt; we can get estimates of $\hat{\alpha}$ and $\hat{\beta}$ (see parameter estimation section).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now you can generate draws for the $i^{th}$ step, generate $p_i$ from $Beta(\hat{\alpha},\hat{\beta})$ and then generate $ X_i$ from $Ber(p_i)$. After you have done this $N$ times you can get $Y = \sum X_i$. This is a single cycle for generation of Y, do this $M$(large) number of times and the histogram for $M$ Ys will be the estimate of density of Y. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Prob[Y \leq y] = \frac {\#Y \leq y} {M}$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;This analysis is valid only when $p_i$ are not fixed.&lt;/strong&gt; This is not the case here. But I will leave it here, in case someone has a similar question.  &lt;/p&gt;&#xA;" OwnerUserId="1307" LastEditorUserId="1307" LastEditorDisplayName="" LastEditDate="2010-12-13T00:00:30.870" LastActivityDate="2010-12-13T00:00:30.870" CommentCount="3" />
  <row Id="5395" PostTypeId="2" ParentId="5382" CreationDate="2010-12-12T08:57:36.583" Score="3" ViewCount="" Body="&lt;p&gt;It sounds like you have multiple sources of information about measurements of interest.&#xA;The aim is to combine the multiple sources so as to maximise measurement accuracy.&#xA;In general you could adopt a formula that weights the different sources of information based on the confidence you ascribe to the source.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It also seems that the confidence in the source might vary.&#xA;E.g., the more work data you have on someone, perhaps the more weight this information would be given for that individual.&#xA;You may also have evidence on the accuracy of an individual's self ratings.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, a simple strategy would to develop a weighted composite of available information where the size of the weight is relative to the informativeness of the source.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This question on &lt;a href=&quot;http://stats.stackexchange.com/questions/1848/bayesian-rating-system-with-multiple-categories-for-each-rating&quot;&gt;Bayesian rating systems&lt;/a&gt; might also be relevant.&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-12-12T08:57:36.583" CommentCount="2" />
  <row Id="5396" PostTypeId="1" CreationDate="2010-12-12T10:18:17.687" Score="1" ViewCount="106" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Does anyone know how to create a nice looking summary table in JMP for several variables? I use Tables&gt;Summary and the table I get has one row that includes for example mean(var1) mean(var2) mean(var3) median(var1) median(var2) median(var3) and so on. I would like each variable to have row of its own, with columns representing the statistics.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Does anyone know how to create a frequency table when creating a histogram in JMP? I get a histogram by using Analyze&gt;Distributions but I don't see any option of getting the frequency table.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Thank you,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Naama Wolf&lt;/p&gt;&#xA;" OwnerUserId="2352" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-12T13:02:56.367" LastActivityDate="2011-04-20T23:57:53.813" Title="Summary and frequency tables in JMP" Tags="&lt;data-visualization&gt;&lt;jmp&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="5397" PostTypeId="2" ParentId="5363" CreationDate="2010-12-12T11:13:06.010" Score="2" ViewCount="" Body="&lt;p&gt;I haven't dealt with Neural Networks for some years now, but I think you will find everything you need here:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://page.mi.fu-berlin.de/rojas/neural/chapter/K7.pdf&quot; rel=&quot;nofollow&quot;&gt;Neural Networks - A Systematic Introduction, Chapter 7: The backpropagation algorithm&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I apologize for not writing the direct answer here, but since I have to look up the details to remember (like you) and given that the answer without some backup may be even useless, I hope this is ok. However, if any questions remain, drop a comment and I'll see what I can do.&lt;/p&gt;&#xA;" OwnerUserId="264" LastActivityDate="2010-12-12T11:13:06.010" CommentCount="1" />
  <row Id="5398" PostTypeId="2" ParentId="5387" CreationDate="2010-12-12T11:21:34.503" Score="5" ViewCount="" Body="&lt;p&gt;To add to @dmk38's response, &quot;any set of scores gives a &lt;em&gt;valid&lt;/em&gt; test, provided they are constructed without consulting the results of the experiment. If the set of scores is poor, in that it badly distorts a numerical scale that really does underlie the ordered classification, the test will not be sensitive. The scores should therefore embody the best insight available about the way in which the classification was constructed and used.&quot; (Cochran, 1954, cited by Agresti, 2002, pp. 88-89). In other words, treating an ordered factor as a numerically scored variable is merely a modelling issue. Provided it makes sense, this will only impact the way you interpret the result, and there is no definitive rule of thumb on how to choose the best representation for an ordinal variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider the following example on maternal Alcohol consumption and presence or absence of congenital malformation (Agresti, &lt;a href=&quot;http://www.stat.ufl.edu/~aa/cda/cda.html&quot; rel=&quot;nofollow&quot;&gt;Categorical Data Analysis&lt;/a&gt;, Table 3.7 p.89):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;            0    &amp;lt;1 1-2 3-5 6+&#xA;Absent  17066 14464 788 126 37&#xA;Present    48    38   5   1  1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In this particular case, we can model the outcome using logistic regression or simple association table. Let's do it in R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;tab3.7 &amp;lt;- matrix(c(17066,48,14464,38,788,5,126,1,37,1), nr=2,&#xA;                 dimnames=list(c(&quot;Absent&quot;,&quot;Present&quot;),&#xA;                               c(&quot;0&quot;,&quot;&amp;lt;1&quot;,&quot;1-2&quot;,&quot;3-5&quot;,&quot;6+&quot;)))&#xA;library(vcd)&#xA;assocstats(tab3.7)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Usual $\chi^2$ (12.08, p=0.016751) or LR (6.20, p=0.184562) statistic (with 4 df) do not account for the ordered levels in Alcohol consumption.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Treating both variables as ordinal with equally spaced scores (this has no impact for binary variables, like malformation, and we choose the baseline as 0=absent), we could test for a linear by linear association. Let's first construct an exploded version of this contingency Table:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(reshape)&#xA;tab3.7.df &amp;lt;- untable(data.frame(malform=gl(2,1,10,labels=0:1), &#xA;                                alcohol=gl(5,2,10,labels=colnames(tab3.7))), &#xA;                     c(tab3.7))&#xA;# xtabs(~malform+alcohol, tab3.7.df) # check&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then we can test for a linear association using&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(coin)&#xA;#lbl_test(as.table(tab3.7))&#xA;lbl_test(malform ~ alcohol, data=tab3.7.df)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which yields $\chi^2(1)=1.83$ with $p=0.1764$. Note that this statistic is simply the correlation between the two series of scores (that Agresti called $M^2=(n-1)r^2$), which is readily computed as&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cor(sapply(tab3.7.df, as.numeric))[1,2]^2*(32574-1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As can be seen, there is not much evidence of a clear association between the two variables. As done by Agresti, if we choose to recode Alcohol levels as {0,0.5,1.5,4,7}, that is using mid-range values for an hypothesized continuous scale with the last score being somewhat purely arbitrary, then we would conclude to a larger effect of maternal Alcohol consumption on the development of congenital malformation:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;lbl_test(malform ~ alcohol, data=tab3.7.df,         &#xA;         scores=list(alcohol=c(0,0.5,1.5,4,7)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;yields a test statistic of 6.57 with an associated p-value of 0.01037. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are alternative coding schemes, including &lt;em&gt;midranks&lt;/em&gt; (in which case, we fall back to Spearman $\rho$ instead of Pearson $r$) that is discussed by Agresti, but I hope you catch the general idea here: It is best to select scores that actually reflect a reasonable measures of the distance between adjacent categories of your ordinal variable, and equal spacing is often a good compromise (in the absence of theoretical justification).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using the GLM approach, we would proceed as follows. But first check how Alcohol is encoded in R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;class(tab3.7.df$alcohol)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It is a simple unordered factor (&lt;code&gt;&quot;factor&quot;&lt;/code&gt;), hence a nominal predictor. Now, here are three models were we consider Alcohol as a nominal, ordinal or continuous predictor.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;summary(mod1 &amp;lt;- glm(malform ~ alcohol, data=tab3.7.df, &#xA;                    family=binomial))&#xA;summary(mod2 &amp;lt;- glm(malform ~ ordered(alcohol), data=tab3.7.df, &#xA;                    family=binomial))&#xA;summary(mod3 &amp;lt;- glm(malform ~ as.numeric(alcohol), data=tab3.7.df, &#xA;                    family=binomial))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The last case implicitly assumes an equal-interval scale, and the $\hat\beta$ is interpreted as @dmk38 did: it reflects the effect of a one-unit increase in Alcohol on the outcome through the logit link, that is the increase in probability of observing a malformation (compared to no malformation, i.e. the odds-ratio) is $\exp(\hat\theta)=\exp(0.228)=1.256$. The Wald test is not significant at the usual 5% level. In this case, the design matrix only includes 2 columns: the first is a constant column of 1's for the intercept, the second is the numerical value (1 to 5) for the predictor, as in a simple linear regression. In sum, this model tests for a linear effect of Alcohol on the outcome (on the logit scale).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, in the two other cases (&lt;code&gt;mod1&lt;/code&gt; and &lt;code&gt;mod2&lt;/code&gt;), we get different output because the design matrix used to model the predictor differs, as can be checked by using:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.matrix(mod1)&#xA;model.matrix(mod2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;We can see that the associated design matrix for &lt;code&gt;mod1&lt;/code&gt; includes dummy variables for the $k-1$ levels of Alcohol (0 is always the baseline) after the intercept term in the first column, whereas in the case of &lt;code&gt;mod2&lt;/code&gt; we have four columns of contrast-coded effects (after the column of 1's for the intercept). The coefficient for the category &quot;3-5&quot; is estimated at 1.03736 under &lt;code&gt;mod1&lt;/code&gt;, but 0.01633 under &lt;code&gt;mod2&lt;/code&gt;. Note that AIC and other likelihood-based measures remain, however, identical between these two models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can try assigning new scores to Alcohol and see how it will impact the predicted probability of a malformation.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-12-12T11:26:47.150" LastActivityDate="2010-12-12T11:26:47.150" CommentCount="4" />
  <row Id="5399" PostTypeId="1" CreationDate="2010-12-12T12:11:28.810" Score="3" ViewCount="527" Body="&lt;p&gt;Say we have two Gaussian random vectors $p(x_1) = N(0,\Sigma_1), p(x_2) = N(0,\Sigma_2)$, is there a well known result for the expectation of their product $E[x_1x_2^T]$ without assuming independence?&lt;/p&gt;&#xA;" OwnerUserId="2353" LastEditorUserId="2353" LastEditorDisplayName="" LastEditDate="2010-12-12T15:04:13.580" LastActivityDate="2010-12-21T05:30:02.967" Title="Expectation of product of Gaussian random variables" Tags="&lt;random-variable&gt;&lt;normal-distribution&gt;&lt;expected-value&gt;" AnswerCount="3" CommentCount="7" />
  <row Id="5400" PostTypeId="2" ParentId="5321" CreationDate="2010-12-12T12:52:20.237" Score="3" ViewCount="" Body="&lt;p&gt;(This was supposed to be a comment, but it got too long)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd say&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;L = data'*data;&#xA;[Ev, Vals]  = eig(L);    &#xA;[Ev,Vals] = sort(Ev,Vals);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;is pretty dodgy; it's a common notion in numerical linear algebra that forming matrices like $\mathbf A^T\mathbf A$ worsens the so-called &quot;conditioning&quot; of your problem, in that merely forming the matrix can cause the loss of significant figures, and any further operations with it, like trying to obtain the required eigenpairs, will result in a complete mess.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One possible way to show how &quot;squaring&quot; ruins things is to consider the so-called Lauchli matrix, which is returned by the MATLAB function &lt;code&gt;gallery('lauchli',n)&lt;/code&gt;, which returns an $(n+1)\times n$ matrix. If you attempt to square that and give that to an eigenroutine like &lt;code&gt;eig()&lt;/code&gt;... disaster! That's just for a small matrix; the larger the matrix you have, the more likely the conditioning of your matrix will be rather bad.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The best way of doing things, then, is to use the singular value decomposition. I discussed SVD and the Lauchli example in &lt;a href=&quot;http://math.stackexchange.com/questions/3869/3871#3871&quot;&gt;this m.SE answer&lt;/a&gt;, but for your purposes, here's what you might try replacing that snippet I quoted with (following shabbychef's answer):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[U, Vals, Ev]  = svd(data, 'econ');&#xA;Vals = diag(Vals);&#xA;Vals = Vals.^2&#xA;[Vals, sidx] = sort(Vals);&#xA;Ev = Ev(:,sidx);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;On the other hand, &lt;code&gt;svd()&lt;/code&gt; returns the singular values in decreasing order, so the sorting might no longer be required...&lt;/p&gt;&#xA;" OwnerUserId="830" LastActivityDate="2010-12-12T12:52:20.237" CommentCount="1" />
  <row Id="5401" PostTypeId="2" ParentId="5382" CreationDate="2010-12-12T13:51:48.310" Score="1" ViewCount="" Body="&lt;p&gt;I'm going to give a stab at an answer, if I misunderstand your question, please comment here or revise your question and I'll do my best to revise my answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I understand you correctly you are trying to find a users 'true level of expertise'.  An elusive quantity to be sure.  Given what you have written, I think you expect that true expertise has some relationship with actual amount of work done.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are some problems meeting the assumption of the technique I am going to suggest first, but I hope the violations will be tolerable.  A first step could be setting up a regression equation to predict work completed given work assigned for each component (G1, G2, G3, etc).  Residuals from this regression equation are deviations from the normal amount of productivity in that component.  Deviations from normal productivity (if you assume those who are more expert get a greater proportion of their work done) may be viewed as a measure of 'true expertise'.  It also sets each person's productivity in comparison to others (a stated goal).  Of course, I strongly suggest you view a scatter-plot of these data (and compare model fits) as I suspect that the relationship will not be linear and that you'll have have to fit a quadratic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not sure what you want to do with the self-ratings.  Of course, you should eye-ball the self ratings, if someone is giving themselves all high scores, that participant may need to be dropped. Another thing you might consider doing prior to analysis is subtracting each subject's mean rating of expertise from each of their ratings of expertise. Though in this approach you'll lose information about people who are overall more skilled than others. To validate that the self-ratings are meaningful you want to correlate them with work completed.  They should be somewhat positive given your assumption that those who complete more work on a given component tend to be (but are not necessarily) more expert.  If the self-ratings are shown to be a somewhat valid measure, then you can move on to correlating the self ratings with the above mentioned residual score.  Again, the correlation ought to be somewhat positive.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit 1:&lt;/strong&gt; You say you want to calculate the &quot;effective rating&quot; of all users relative to the group of users, &lt;em&gt;not&lt;/em&gt; self-ranking, considering their work data &lt;em&gt;and&lt;/em&gt; self-ranking.  I'm not sure how you can both consider self-ranking and not consider it at the same time.  If you mean that you want to create a measure of skill based on something other than self-rankings, but then combine it with the self-ranking data, you can simply sum the Z-score of some measure of skill (e.g. work done or the method I describe above) and the Z-score of self ratings.&lt;/p&gt;&#xA;" OwnerUserId="196" LastEditorUserId="196" LastEditorDisplayName="" LastEditDate="2010-12-12T14:00:57.243" LastActivityDate="2010-12-12T14:00:57.243" CommentCount="8" />
  <row Id="5402" PostTypeId="1" CreationDate="2010-12-12T14:10:55.290" Score="3" ViewCount="84" Body="&lt;p&gt;I have heard that many problems in Mathematical Statistics can be stated and solved in terms of Symplectic Geometry. Unfortunately this was a pretty vague statement and I am interested in something more concrete. Also there seem to be some books written form this perspective, but I couldn't find any.&lt;/p&gt;&#xA;" OwnerUserId="2354" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-12-12T18:20:14.517" LastActivityDate="2010-12-12T18:33:17.970" Title="References for use of symplectic geometry in statistics?" Tags="&lt;books&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="1" />
  <row Id="5403" PostTypeId="2" ParentId="5392" CreationDate="2010-12-12T15:24:15.363" Score="8" ViewCount="" Body="&lt;p&gt;It isn't linear.  The formula says it isn't linear.  Trust your mathematical instinct!  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It only appears linear in the graph because of the scale, with $\sigma_{1}=5$ and $\sigma_{2}=4$. Try it yourself: calculate the slopes at a few places and you will see that they differ.  You can exaggerate the difference by picking $\sigma_{1}=37$, say.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is some R code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;a &amp;lt;- 5; b &amp;lt;- 4; p &amp;lt;- 1&#xA;f &amp;lt;- function(w) w^2*a^2 + (1-w)^2*b^2 + 2*w*(1-w)*p*a*b&#xA;curve(f, from = 0, to = 1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you would like to check some slopes:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(f(0.5) - f(0.4)) / 0.1&#xA;(f(0.8) - f(0.7)) / 0.1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1108" LastEditorUserId="1108" LastEditorDisplayName="" LastEditDate="2010-12-12T15:29:50.743" LastActivityDate="2010-12-12T15:29:50.743" />
  <row Id="5404" PostTypeId="2" ParentId="5343" CreationDate="2010-12-12T15:52:47.647" Score="1" ViewCount="" Body="&lt;p&gt;I can tell you what I would do as a machine learner ;):&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Creating two models $M_1,M_2$ for x using $y_1,y_2$ respectively&lt;/li&gt;&#xA;&lt;li&gt;A prediction for x is calculated as the average of the predictions of $M_1$ and $M_2$&#xD;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;A &quot;model&quot; can be either a gaussian distribution (as described, if you know that x a) has a gaussian distribution and b) know that the function x to y is nearly the identity) or anything else, e.g. a simple linear regression model (if you are not sure whether there are additional factors).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;BUT:&lt;/strong&gt; If x is indeed a bimodal distribution with the two gaussians y, than the suggested approach would not make any sense. In this case I'd try a more generic approach, i.e. EM to see whether one or two gaussians are more appropriate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is quite hard to give a general answer to this questions, since it is not exactly clear what determines whether y1 or y2 can be observed. It could be that for both y x is missing completely at random (in this case y1 and y2 were just different random samples), but on the other hand it could be that for a certain fraction of x only y1 can be observed, but not y2, and vice versa for another fraction of x.&lt;/p&gt;&#xA;" OwnerUserId="264" LastEditorUserId="264" LastEditorDisplayName="" LastEditDate="2010-12-12T19:04:16.523" LastActivityDate="2010-12-12T19:04:16.523" />
  <row Id="5405" PostTypeId="2" ParentId="5327" CreationDate="2010-12-12T16:10:31.023" Score="2" ViewCount="" Body="&lt;p&gt;To echo Aniko's comment: The primary assumption is the existence of truncation.  This is not the same assumption as the two other possibilities that your post suggests to me: boundedness and sample selection.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have a fundamentally &lt;em&gt;bounded&lt;/em&gt; dependent variable rather than a truncated one you might want to move to a generalized linear model framework with one of the (less often chosen) distributions for Y e.g. log-normal, gamma, exponential, etc. which respect that lower bound.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively you might then ask yourself whether you think that the process that generates the zero observations in your model is the same as the one that generates the strictly positive values - prices in your application, I think.  If this is not the case then something from the class of &lt;em&gt;sample selection&lt;/em&gt; models, (e.g. Heckman models) might be appropriate.  In that case you'd be in the situation of specifying one model of being willing to pay any price at all, and another model of what price your subjects would pay if they wanted to pay something.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In short, you probably want to review the difference between assuming truncated, censored, bounded, and sample selected dependent variables.  Which one you want will come from the details of your application.  Once that first most important assumption is made you can more easily determine whether you like the specific assumptions of any model in your chosen class.  Some of the sample selection models have assumptions that are rather difficult to check...&lt;/p&gt;&#xA;" OwnerUserId="1739" LastActivityDate="2010-12-12T16:10:31.023" CommentCount="0" />
  <row Id="5407" PostTypeId="2" ParentId="5392" CreationDate="2010-12-12T17:13:53.450" Score="6" ViewCount="" Body="&lt;p&gt;Using $w_1 + w_2 = 1$, compute&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\eqalign{&#xD;&#xA;\text{Var}(w_1 A + w_2 B) &amp;amp;= \left( w_1 \sigma_1 + w_2 \sigma_2 \right)^2 \cr&#xD;&#xA;&amp;amp;= \left( w_1(\sigma_1 - \sigma_2) + \sigma_2 \right)^2 \text{.}&#xD;&#xA;} $$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This shows that when $\sigma_1 \ne \sigma_2$, the graph of the variance versus $w_1$ (shown sideways in the illustration) is a &lt;em&gt;parabola&lt;/em&gt; centered at $\sigma_2 / (\sigma_2 - \sigma_1)$.  No portion of any parabola is linear.  With $\sigma_1 = 5$ and $\sigma_2 = 4$, the center is at $-5$: way below the graph at the scale in which it is drawn.  Thus, you are looking at a small piece of a parabola, which will appear linear.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When $\sigma_1 = \sigma_2$, the variance &lt;em&gt;is&lt;/em&gt; a linear function of $w_1$.  In this case the plot would be a perfectly vertical line segment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;BTW, you &lt;em&gt;knew&lt;/em&gt; this answer already, without calculation, because basic principles imply the plot of variance cannot be a line unless it is vertical.  After all, there is no mathematical or statistical prohibition to restrict $w_1$ to lie between $0$ and $1$: &lt;em&gt;any&lt;/em&gt; value of $w_1$ determines a new random variable (a linear combination of the random variables A and B) and therefore must have a non-negative value for its variance.  Therefore &lt;em&gt;all these curves (even when extended to the full vertical range of&lt;/em&gt; $w_1$  &lt;em&gt;) must lie to the right of the vertical axis.&lt;/em&gt;  That precludes all lines except vertical ones.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Plot of the variance for $\rho = 1 - 2^{-k}, k = -1, 0, 1, \ldots, 10$:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/xnxgP.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-12-12T21:40:58.743" LastActivityDate="2010-12-12T21:40:58.743" />
  <row Id="5408" PostTypeId="2" ParentId="4360" CreationDate="2010-12-12T17:50:29.487" Score="7" ViewCount="" Body="&lt;p&gt;A &lt;em&gt;subjectivist&lt;/em&gt; Bayesian argument is practically the only way (from a statistical standpoint) you could go about understanding your &lt;em&gt;intuition&lt;/em&gt;, which is--properly speaking--the subject of a &lt;em&gt;psychological&lt;/em&gt; investigation, not a statistical one.  However, it is patently unfair--and therefore invalid--to use a Bayesian approach to argue that an investigator faked the data.  The logic of this is perfectly circular: it comes down to saying &quot;based on my prior beliefs about the outcome, I find your result incredible, and therefore you &lt;em&gt;must&lt;/em&gt; have cheated.&quot;  Such an illogical self-serving argument obviously wouldn't stand up in a courtroom or in a peer review process.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead, we could take a tip from &lt;a href=&quot;http://www.cs.brown.edu/courses/csci1950-l/presentations/REVISED--AFCB%202007%20Course%20Fisher%20and%20Mendel.ppt&quot; rel=&quot;nofollow&quot;&gt;Ronald Fisher's critique of Mendel's experiments&lt;/a&gt; and conduct a formal hypothesis test.  Of course it's invalid to test a &lt;em&gt;post hoc&lt;/em&gt; hypothesis based on the outcome.  But experiments have to be replicated to be believed: that's a tenet of the scientific method.  So, having seen one result we think might have been faked, we can formulate an appropriate hypothesis to test &lt;em&gt;future&lt;/em&gt; (or additional) results.  In this case the critical region would comprise a set of results extremely close to the expectation.  For instance, a test at the $\alpha$ = 5% level would view any result between 9,996 and 10,004 as suspect, because (a) this collection is close to our hypothesized &quot;faked&quot; results and (b) under the null hypothesis of &lt;em&gt;no faking&lt;/em&gt; (innocent until proven guilty in court!), a result in this range has only a 5% (actually 5.07426%) chance of occurring.  Furthermore, we can put this seemingly &lt;em&gt;ad hoc&lt;/em&gt; approach in a chi-square context (a la Fisher) simply by squaring the deviation between the observed proportion and the expected proportion, then invoking the &lt;a href=&quot;http://en.wikipedia.org/wiki/Neyman%E2%80%93Pearson_lemma&quot; rel=&quot;nofollow&quot;&gt;Neyman-Pearson lemma&lt;/a&gt; in a one-tailed test at the &lt;em&gt;low&lt;/em&gt; tail and applying the &lt;a href=&quot;http://en.wikipedia.org/wiki/Binomial_distribution#Normal_approximation&quot; rel=&quot;nofollow&quot;&gt;Normal approximation to the Binomial distribution&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Although such a test cannot &lt;em&gt;prove&lt;/em&gt; fakery, it can be applied to &lt;em&gt;future&lt;/em&gt; reports from that experimenter to assess the credibility of their claims, &lt;em&gt;without making untoward and unsupportable assumptions based on your intuition alone.&lt;/em&gt;  This is much more fair and rigorous than invoking a Bayesian argument to implicate someone who might be perfectly innocent and just happened to be so unlucky that they got a beautiful experimental result!&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-12-12T17:50:29.487" />
  <row Id="5409" PostTypeId="2" ParentId="5402" CreationDate="2010-12-12T18:01:19.650" Score="4" ViewCount="" Body="&lt;p&gt;A direct connection would be unexpected: the two fields appear to have little in common.  For example, &lt;a href=&quot;http://loveguests.com/books/science/mathematics-statistics/3103-an-introduction-to-symplectic-geometry.html&quot; rel=&quot;nofollow&quot;&gt;a modern introduction to symplectic geometry published by the American Mathematical Society&lt;/a&gt; appears to make no mention of mathematical statistics at all.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At best it seems any connection would come through mathematical physics.  A symplectic geometry on phase space naturally arises in the &lt;a href=&quot;http://en.wikipedia.org/wiki/Hamiltonian_mechanics#Geometry_of_Hamiltonian_systems&quot; rel=&quot;nofollow&quot;&gt;Hamiltonian formulation of classical mechanics&lt;/a&gt; and that in turn can be used to explore global properties of physical systems.  The study of periodic and near-periodic orbits becomes somewhat statistical (&lt;em&gt;e.g.&lt;/em&gt;, &lt;a href=&quot;http://en.wikipedia.org/wiki/Ergodic_theory&quot; rel=&quot;nofollow&quot;&gt;ergodic theorems&lt;/a&gt;).  When applied to a system with many degrees of freedom it would conceivably relate some aspects of symplectic geometry to thermodynamics, which is inherently a statistical theory&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-12-12T18:01:19.650" />
  <row Id="5410" PostTypeId="2" ParentId="5402" CreationDate="2010-12-12T18:33:17.970" Score="3" ViewCount="" Body="&lt;p&gt;I know nothing whatsoever about symplectic geometry, but a bit of googling brought up a &lt;a href=&quot;http://dx.doi.org/10.1016/S0378-3758%2897%2900006-2&quot; rel=&quot;nofollow&quot;&gt;1997 article in the &lt;em&gt;Journal of Statistical Planning &amp;amp; Inference&lt;/em&gt; by Barndorff-Nielsen &amp;amp; Jupp&lt;/a&gt;, which contains this quote:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Some other links between statistics and symplectic geometry have been discussed&#xA;  by Friedrich and Nakamura. Friedrich (1991) established some connections between&#xA;  expected (Fisher) information and symplectic structures. However, his approach and&#xA;  results are quite different from those considered here. Nakamura (1993, 1994) has&#xA;  shown that certain parametric statistical models in which the parameter space &lt;em&gt;M&lt;/em&gt; is an&#xA;  even-dimensional vector space (and so has the symplectic structure of the cotangent&#xA;  space of a vector space) give rise to completely integrable Hamiltonian systems on &lt;em&gt;M&lt;/em&gt;.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The cited refs are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Friedrich, T., 1991. Die Fisher-lnformation und symplectische Strukturen. &lt;em&gt;Math. Nachr.&lt;/em&gt; 153: 273-296.&lt;/li&gt;&#xA;&lt;li&gt;Nakamura, Y., 1993. Completely integrable gradient systems on the manifolds of Gaussian and multinomial&#xA;distributions. &lt;em&gt;Japan. J. Ind. Appl. Math.&lt;/em&gt; 10: 179-189.&lt;/li&gt;&#xA;&lt;li&gt;Nakamura, Y., 1994. Gradient systems associated with probability distributions. Japan. &lt;em&gt;J. Ind. Appl. Math.&lt;/em&gt; 11: 21-30.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The article's Introduction says B-N &amp;amp; others have used &lt;strong&gt;&lt;em&gt;differential&lt;/em&gt; geometry&lt;/strong&gt; as an approach to statistical asymptotics. Symplectic geometry is a branch of differential geometry (&lt;a href=&quot;http://en.wikipedia.org/wiki/Symplectic_geometry&quot; rel=&quot;nofollow&quot;&gt;according to Wikipedia&lt;/a&gt;). A &lt;a href=&quot;http://www.google.co.uk/search?tbs=bks%3A1&amp;amp;q=%22differential+geometry%22+statistics+OR+statistical+-mechanics+-thermodynamics&quot; rel=&quot;nofollow&quot;&gt;Google Books search&lt;/a&gt; finds several books about the application of &lt;em&gt;differential geometry&lt;/em&gt; to statistics and related fields such as econometrics.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-12-12T18:33:17.970" CommentCount="1" />
  <row Id="5411" PostTypeId="2" ParentId="5114" CreationDate="2010-12-12T18:38:47.193" Score="4" ViewCount="" Body="&lt;p&gt;If, given an answer to my comment above, you wish to bound the range of the distribution, why not simply fit a Beta distribution where you rescale to the unit interval?  In other words, if you know that the parameter of interest should fall between $[2 , 8]$, then why not define $Y = \frac{X - 5}{6} + \frac{1}{2} = \frac{X - 2}{6}$.  Where I've first centered the interval on zero, divided by the width so that Y will have a range of 1, and then added $\frac{1}{2}$ back so that the range of Y is $[0,1]$. (You can think of it either way: directly from $[2,8] \rightarrow [0,1]$ or from $[2,8] \rightarrow [-\frac{1}{2},\frac{1}{2}] \rightarrow [0,1]$, but I thought the latter might be easier at first).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, with two data points, you could fit a beta posterior with a uniform beta prior?&lt;/p&gt;&#xA;" OwnerUserId="1499" LastEditorUserId="1499" LastEditorDisplayName="" LastEditDate="2010-12-12T18:44:14.977" LastActivityDate="2010-12-12T18:44:14.977" />
  <row Id="5412" PostTypeId="2" ParentId="5304" CreationDate="2010-12-12T19:47:32.683" Score="3" ViewCount="" Body="&lt;p&gt;Following up: profile confidence intervals are more reliable (choosing the appropriate cutoff for the likelihood does involve an asymptotic (large sample) assumption, but this is a much weaker assumption than the quadratic-likelihood-surface assumption underlying the Wald confidence intervals). As far as I know, there is no argument for the Wald statistics over the profile confidence intervals except that the Wald statistics are much quicker to compute and may be &quot;good enough&quot; in many circumstances (but sometimes way off: look up the Hauck-Donner effect).&lt;/p&gt;&#xA;" OwnerUserId="2126" LastActivityDate="2010-12-12T19:47:32.683" />
  <row Id="5414" PostTypeId="1" AcceptedAnswerId="5417" CreationDate="2010-12-12T21:53:35.743" Score="11" ViewCount="295" Body="&lt;p&gt;What are some of the main pitfalls of using linear mixed-effects models? What are the most important things to test/watch out for in assessing the appropriateness of your model? When comparing models of the same dataset, what are the most important things to look for?&lt;/p&gt;&#xA;" OwnerUserId="52" LastActivityDate="2010-12-13T08:21:53.467" Title="Pitfalls of linear mixed models" Tags="&lt;mixed-model&gt;&lt;model-comparison&gt;" AnswerCount="3" FavoriteCount="9" />
  <row Id="5415" PostTypeId="2" ParentId="5399" CreationDate="2010-12-12T22:41:14.540" Score="5" ViewCount="" Body="&lt;p&gt;Yes, there is a well-known result.  Based on your edit, we can focus first on individual entries of the array $E[x_1 x_2^T]$.  Such an entry is the product of two variables of zero mean and finite variances, say $\sigma_1^2$ and $\sigma_2^2$.  The &lt;a href=&quot;http://en.wikipedia.org/wiki/Cauchy%E2%80%93Schwarz_inequality#Probability_theory&quot; rel=&quot;nofollow&quot;&gt;Cauchy-Schwarz Inequality&lt;/a&gt; implies the absolute value of the expectation of the product cannot exceed $|\sigma_1 \sigma_2|$.  In fact, every value in the interval $[-|\sigma_1 \sigma_2|, |\sigma_1 \sigma_2|]$ is possible because it arises for some binormal distribution.  &lt;strong&gt;Therefore, the $i,j$ entry of $E[x_1 x_2^T]$ must be less than or equal to $\sqrt{\Sigma_{1_{i,i}} \Sigma_{2_{j,j}}}$ in absolute value.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we now assume all variables are normal and that $(x_1; x_2)$ is multinormal, there will be further restrictions because the covariance matrix of $(x_1; x_2)$ must be positive semidefinite.  Rather than belabor the point, I will illustrate.  Suppose $x_1$ has &lt;em&gt;two&lt;/em&gt; components $x$ and $y$ and that $x_2$ has one component $z$.  Let $x$ and $y$ have unit variance and correlation $\rho$ (thus specifying $\Sigma_1$) and suppose $z$ has unit variance ($\Sigma_2$).  Let the expectation of $x z$ be $\alpha$ and that of $y z$ be $\beta$.  We have established that $|\alpha| \le 1$ and $|\beta| \le 1$.  However, not all combinations are possible: at a minimum, the &lt;strong&gt;determinant of the covariance matrix of $(x_1; x_2)$ cannot be negative.&lt;/strong&gt;  This imposes the non-trivial condition&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$1-\alpha ^2-\beta ^2+2 \alpha  \beta  \rho -\rho ^2 \ge 0.$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For any $-1 \lt \rho \lt 1$ this is an ellipse (along with its interior) inscribed within the $\alpha, \beta$ square $[-1, 1] \times [-1, 1]$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To obtain further restrictions, additional assumptions about the variables are necessary.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Plot of the permissible region&lt;/strong&gt; $(\rho, \alpha, \beta)$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/MBJhf.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-12-12T22:41:14.540" />
  <row Id="5416" PostTypeId="1" CreationDate="2010-12-12T23:04:28.360" Score="0" ViewCount="155" Body="&lt;p&gt;Hi&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Under what scenario is Root Mean Square normalization to be used?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks much.&lt;/p&gt;&#xA;" OwnerUserId="397" LastActivityDate="2010-12-12T23:04:28.360" ClosedDate="2010-12-14T15:16:35.377" Title="When to use Root Mean Square normalization?" Tags="&lt;normalization&gt;" CommentCount="1" />
  <row Id="5417" PostTypeId="2" ParentId="5414" CreationDate="2010-12-13T00:15:05.537" Score="6" ViewCount="" Body="&lt;p&gt;Hello Alan:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a good question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are some common pitfalls:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Using standard likelihood theory, we may derive a test to compare two nested&#xA;hypotheses, $H_0$ and $H_1$, by computing the likelihood ratio test statistic. The null distribution of this test statistic is approximately chi-squared with degrees of freedom equal to difference in the dimensions of the two parameters spaces. Unfortunately, this test is only approximate and requires several assumptions. One crucial assumption is that the parameters under the null are not on the boundary of the parameter space. Since we are often interested in testing hypotheses about the random effects that take the form:&#xA;$$H_0: \sigma^2=0$$&#xD;&#xA;&lt;em&gt;This a real concern.&lt;/em&gt; The way to get around this problem is using REML. But still, the p-values will tend to be larger than they should be. This means that if you observe a significant effect using the χ2 approximation, you can be fairly confident that it is actually significant. Small, but not significant, p-values might spur one to use more accurate, but time-consuming, bootstrap methods.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Comparing fixed effects: If you plan to use the likelihood ratio test to compare two nested models that differ only in their fixed effects, you cannot use the REML estimation method. The reason is that REML estimates the random effects by considering linear combinations of the data that remove the fixed effects. If these fixed effects are changed, the likelihoods of the two models will not be directly comparable.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;P-values: The p-values generated by the likelihood ratio test for fixed effects are approximate and unfortunately tend to be too small, thereby sometimes overstating the importance of some effects. We may use &lt;em&gt;nonparametric&lt;/em&gt; bootstrap methods to find more accurate p-values for the likelihood ratio test.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;There are other concerns about p-values for the fixed effects test which are highlighted by Dr. Doug Bates [&lt;a href=&quot;https://stat.ethz.ch/pipermail/r-help/2006-May/094765.html&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;].&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I am sure other members of the forum will have better answers. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Source: Extending linear models with R -- Dr. Julain Faraway.&lt;/p&gt;&#xA;" OwnerUserId="1307" LastActivityDate="2010-12-13T00:15:05.537" />
  <row Id="5418" PostTypeId="1" CreationDate="2010-12-13T00:56:07.200" Score="24" ViewCount="683" Body="&lt;p&gt;I'm planning to start writing R packages.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I thought it would be good to study the source code of existing packages&#xA;in order to learn the conventions of package construction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My criteria for good packages to study:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Simple statistical/technical ideas&lt;/strong&gt;: The point is to learn about the mechanics of package construction. Understanding the package should not require detailed highly domain specific knowledge about the actual topic of the package.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Simple and conventional coding style&lt;/strong&gt;: I'm looking for something a bit more than &lt;code&gt;Hello World&lt;/code&gt; but not a whole lot more. Idiosyncratic tricks and hacks would be distracting when first learning R packages.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Good coding style&lt;/strong&gt;: The code is well written. It reveals both an understanding of good coding, in general, and an awareness of the conventions of coding in R.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Questions:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Which packages would be good to study?&lt;/li&gt;&#xA;&lt;li&gt;Why would the suggested package source code be good to study&#xA;relative either to the criteria mentioned above or any other criteria that might be relevant?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update (13/12/2010)&lt;/strong&gt;&#xA;Following Dirk's comments I wanted to make it clear that no doubt many packages would be good to study first. I also agree that packages will provide models for different things (e.g., vignettes, S3 classes, S4 classes, unit testing, Roxygen, etc.).&#xA;Nonetheless, it would be interesting to read concrete suggestions about good packages &#xA;to start with and the reasons why they would be good packages to start with.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've also updated the question above to refer to &quot;packages&quot; rather than &quot;package&quot;.&lt;/p&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-12-13T03:21:06.557" LastActivityDate="2010-12-20T19:51:03.293" Title="First R packages source code to study in preparation for writing own package" Tags="&lt;r&gt;" AnswerCount="5" CommentCount="2" FavoriteCount="12" />
  <row Id="5419" PostTypeId="2" ParentId="5418" CreationDate="2010-12-13T01:42:30.347" Score="0" ViewCount="" Body="&lt;p&gt;i would recommend hadley's reshape package. you can find the source at &lt;a href=&quot;https://github.com/hadley/reshape&quot; rel=&quot;nofollow&quot;&gt;https://github.com/hadley/reshape&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="277" LastActivityDate="2010-12-13T01:42:30.347" CommentCount="2" />
  <row Id="5420" PostTypeId="2" ParentId="5414" CreationDate="2010-12-13T02:13:33.037" Score="4" ViewCount="" Body="&lt;p&gt;Modeling the variance structure is arguably the most powerful and important single feature of mixed models.  This extends beyond variance structure to include correlation among observations.  Care must be taken to build an appropriate covariance structure otherwise tests of hypotheses, confidence intervals, and estimates of treatment means may not be valid.  Often one needs knowledge of the experiment to specify the correct random effects.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;SAS for Mixed Models is my go to resource, even if I want to do the analysis in R.&lt;/p&gt;&#xA;" OwnerUserId="2310" LastActivityDate="2010-12-13T02:13:33.037" />
  <row Id="5421" PostTypeId="2" ParentId="5418" CreationDate="2010-12-13T02:47:06.790" Score="6" ViewCount="" Body="&lt;p&gt;Why not take an empirically-driven random sampling approach?  Just pick a few and see which work for you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Kidding aside, just look at a few packages you yourself use and are familiar with.  Downloading them is easy, or if you prefer you can also view them via a web interface at R-Forge, RForge, or Github.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You will most likely end up with different packages for different ideas.  Some may help you with the way they integrate, say, a vignette. Some may help with compiled code. Or unit tests. Or Roxygen.   There are about 2600 of them, so why obsess over a single best?&lt;/p&gt;&#xA;" OwnerUserId="334" LastActivityDate="2010-12-13T02:47:06.790" CommentCount="2" />
  <row Id="5422" PostTypeId="2" ParentId="3511" CreationDate="2010-12-13T03:03:02.913" Score="1" ViewCount="" Body="&lt;p&gt;You may check the paper: Efficient L1-regularized logistic regression, which is an IRLS-based algorithm for LASSO. &#xA;Regarding the implementation, the link may be useful for you&#xA;(http://ai.stanford.edu/~silee/softwares/irlslars.htm).&lt;/p&gt;&#xA;" OwnerUserId="2358" LastActivityDate="2010-12-13T03:03:02.913" />
  <row Id="5423" PostTypeId="2" ParentId="5368" CreationDate="2010-12-13T04:35:59.257" Score="0" ViewCount="" Body="&lt;p&gt;Factor Analysis differs from PCA by an extra &quot;rotation&quot; step. The purpose is to make the resulting factors more interpretable. check out my blog post for more details and a good resource: &lt;a href=&quot;http://blog.bzst.com/2009/03/principal-components-analysis-vs-factor.html&quot; rel=&quot;nofollow&quot;&gt;http://blog.bzst.com/2009/03/principal-components-analysis-vs-factor.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1945" LastActivityDate="2010-12-13T04:35:59.257" CommentCount="1" />
  <row Id="5424" PostTypeId="2" ParentId="4734" CreationDate="2010-12-13T04:50:07.943" Score="1" ViewCount="" Body="&lt;p&gt;For another dimension of the difference between the two, please see my blog post -- this is related to the goal of the study: &lt;a href=&quot;http://blog.bzst.com/2009/03/principal-components-analysis-vs-factor.html&quot; rel=&quot;nofollow&quot;&gt;http://blog.bzst.com/2009/03/principal-components-analysis-vs-factor.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1945" LastActivityDate="2010-12-13T04:50:07.943" />
  <row Id="5425" PostTypeId="2" ParentId="1995" CreationDate="2010-12-13T05:08:35.643" Score="4" ViewCount="" Body="&lt;p&gt;Here's another perspective on using multilevel vs. regression models: In an interesting paper by Afshartous and de Leeuw, they show that if the purpose of the modeling is predictive (that is, to predict new observations), the choice of model is different from when the goal is inference (where you try to match the model with the data structure). The paper that I am referring to is &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Afshartous, D., de Leeuw, J. (2005). Prediction in multilevel models. J. Educat. Behav. Statist. 30(2):109–139.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I just found another related paper by these authors here: &lt;a href=&quot;http://moya.bus.miami.edu/~dafshartous/Afshartous_CIS.pdf&quot; rel=&quot;nofollow&quot;&gt;http://moya.bus.miami.edu/~dafshartous/Afshartous_CIS.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1945" LastActivityDate="2010-12-13T05:08:35.643" />
  <row Id="5426" PostTypeId="2" ParentId="1995" CreationDate="2010-12-13T07:47:49.127" Score="2" ViewCount="" Body="&lt;p&gt;I learned from Snijders and Bosker, Multilevel Analysis: An introduction to basic and advanced multilevel modeling. It is very well pitched at the beginner I think, it must be because I am a thicko where these things are concerned and it made sense to me.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I second the Gelman and Hill as well, a truly brilliant book.&lt;/p&gt;&#xA;" OwnerUserId="199" LastActivityDate="2010-12-13T07:47:49.127" />
  <row Id="5427" PostTypeId="2" ParentId="5414" CreationDate="2010-12-13T08:21:53.467" Score="5" ViewCount="" Body="&lt;p&gt;The common pitfall which I see is the ignoring the variance of random effects. If it is large compared to residual variance or variance of dependent variable, the fit usually looks nice, but only because random effects account for all the variance. But since the graph of actual vs predicted looks nice you are inclined to think that your model is good. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Everything falls apart when such model is used for predicting new data. Usually then you can use only fixed effects and the fit can be very poor. &lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2010-12-13T08:21:53.467" />
  <row Id="5428" PostTypeId="2" ParentId="5351" CreationDate="2010-12-13T09:08:56.433" Score="6" ViewCount="" Body="&lt;p&gt;Suppose we have a regression&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align*}&#xA;y=X\beta+u&#xA;\end{align*}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then OLS estimate $\hat{\beta}$ is&#xA;\begin{align*}&#xA;\widehat{\beta}-\beta=(X'X)^{-1}X'u&#xA;\end{align*}&#xA;and assuming that $\hat{\beta}$ is unbiased estimate we have&#xA;\begin{align*}&#xA;D\widehat{\beta}=E\left[(X'X)^{-1}X'uu'X(X'X)^{-1}\right]&#xA;\end{align*}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The usual OLS assumptions are that $E(u|X)=0$ and $E(uu&amp;#39;|X)=\sigma^2I_n$ which gives us&#xA;\begin{align*}&#xA;D\widehat{\beta}=\sigma^2E(X'X)^{-1}&#xA;\end{align*}&#xA;This covariance matrix is usually reported in statistical packages.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If $u_i$ are heteroscedastic and (or) autocorellated, then $E(uu&amp;#39;|X)\neq\sigma^2I_n$ and the usual output gives misleading results. To get the correct results HAC standard errors are calculated. All the methods for HAC errors calculate &#xA;\begin{align*}&#xA;diag(E(X'X)^{-1}X'uu'X(X'X)^{-1}).&#xA;\end{align*}&#xA;They differ on their assumptions what $E(uu&amp;#39;|X)$ looks like. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So it is natural then that function &lt;code&gt;NeweyWest&lt;/code&gt; requests linear model. Newey-West method calculates the correct standard errors of linear model estimator. So your solution is perfectly correct if you assume that your stock returns follow the model&#xA;\begin{align}&#xA;r_t=\mu+u_t&#xA;\end{align}&#xA;and you want to estimate $D\mu$ guarding against irregularities in $u_t$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If on the other hand you want to estimate &quot;correct&quot; $Dr_t$ (whatever that means), you should check out volatility models, such as GARCH and its variants. They assume that&#xA;\begin{align*}&#xA;r_t=\sigma_t\varepsilon_t&#xA;\end{align*} &#xA;where $\varepsilon_t$ are iid normal. The goal is then to correctly estimate $\sigma_t$. Then $Dr_t=D\sigma_t$ and you have &quot;correct&quot; estimate of your variance, guarding against usual idiosyncrasies of stock returns such as volatility clustering, skewness and etc.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2010-12-13T09:08:56.433" CommentCount="2" />
  <row Id="5429" PostTypeId="2" ParentId="3849" CreationDate="2010-12-13T12:09:19.943" Score="3" ViewCount="" Body="&lt;p&gt;As suggested by &lt;a href=&quot;http://stats.stackexchange.com/users/919/whuber&quot;&gt;whuber&lt;/a&gt; &amp;amp; &lt;a href=&quot;http://stats.stackexchange.com/users/28/srikant-vadali&quot;&gt;Srikant Vadali&lt;/a&gt; I posted &lt;a href=&quot;http://www.stata.com/statalist/archive/2010-12/msg00443.html&quot; rel=&quot;nofollow&quot;&gt;this question&lt;/a&gt; on Statalist.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.maartenbuis.nl/&quot; rel=&quot;nofollow&quot;&gt;Maarten L. Buis&lt;/a&gt; &lt;a href=&quot;http://www.stata.com/statalist/archive/2010-12/msg00451.html&quot; rel=&quot;nofollow&quot;&gt;pointed&lt;/a&gt; that estout itself can be &lt;a href=&quot;http://repec.org/bocode/e/estout/advanced.html#advanced404&quot; rel=&quot;nofollow&quot;&gt;used&lt;/a&gt; for processing PCA (as well as factor analysis) results. Thanks Maarten.&lt;/p&gt;&#xA;" OwnerUserId="22" LastActivityDate="2010-12-13T12:09:19.943" CommentCount="1" />
  <row Id="5430" PostTypeId="1" CreationDate="2010-12-13T12:43:59.630" Score="4" ViewCount="181" Body="&lt;p&gt;This image show a histogram (200 bins) of accumulated distances from a radar distance meter (very noisy).&#xA;The peak around 7 meters is an object. At thought this looked kind of like a normal distribution, at least if you ignore values &amp;lt;4m (which for this application is reasonable).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have also achieved a reasonable fit with a &lt;a href=&quot;http://en.wikipedia.org/wiki/Log-normal_distribution&quot; rel=&quot;nofollow&quot;&gt;log-normal distribution&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I am trying to do is to filter out true distances based on the probability distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/tYgBf.png&quot; alt=&quot;alt text&quot;&gt;&#xA;&lt;strong&gt;EDIT:&lt;/strong&gt; log-normal distribution:&#xA;&lt;img src=&quot;http://i.imgur.com/9UMoY.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note:&lt;/strong&gt; this question can also be found at &lt;a href=&quot;http://math.stackexchange.com/questions/14162/is-this-a-probability-distribution&quot;&gt;math.stackexchange.com&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1411" LastEditorUserId="1411" LastEditorDisplayName="" LastEditDate="2010-12-13T14:18:05.003" LastActivityDate="2011-05-18T05:47:30.183" Title="How would you filter this?" Tags="&lt;distributions&gt;&lt;signal-processing&gt;" AnswerCount="1" CommentCount="13" />
  <row Id="5431" PostTypeId="1" CreationDate="2010-12-13T15:32:39.450" Score="2" ViewCount="258" Body="&lt;p&gt;Should small sample sizes of 5 and less be included in reports? &#xA;This is an area of medical training that is currently being reported only to the course level trainers and statistically not included in reports to upper management.   All of the data is mined from course-level surveys.&lt;/p&gt;&#xA;" OwnerUserId="2362" LastEditorUserId="223" LastEditorDisplayName="" LastEditDate="2010-12-17T07:43:57.623" LastActivityDate="2010-12-17T07:43:57.623" Title="Statistical significance with sample size of 5 or less" Tags="&lt;hypothesis-testing&gt;&lt;statistical-significance&gt;" AnswerCount="3" CommentCount="1" />
  <row Id="5432" PostTypeId="2" ParentId="5431" CreationDate="2010-12-13T15:48:08.347" Score="5" ViewCount="" Body="&lt;p&gt;If you are doing things properly the worst thing you can do is not to reject $H_0$ (does not mean $H_0$ is true, only mean you do not have enough evidence to reject) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyway, there can be something significant with 5 observations: &#xA; If you need to test, for example, wether your random variable has a negative mean under the assumption of variance one, then having 5 observation over 10 is highly significant .... &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just be carefull not to use a test that uses something like asymptotic normality or other assumption that cannot be satisfyed or checked with only five observations&lt;/p&gt;&#xA;" OwnerUserId="223" LastActivityDate="2010-12-13T15:48:08.347" />
  <row Id="5433" PostTypeId="2" ParentId="5418" CreationDate="2010-12-13T16:50:50.383" Score="9" ViewCount="" Body="&lt;p&gt;I would suggest looking at the &lt;a href=&quot;http://cran.r-project.org/web/packages/zoo/index.html&quot; rel=&quot;nofollow&quot;&gt;zoo&lt;/a&gt; package for the following reasons:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;It has several well-written vignettes;&lt;/li&gt;&#xA;&lt;li&gt;It uses a namespace using &lt;code&gt;useDynLib&lt;/code&gt;, &lt;code&gt;import&lt;/code&gt;, &lt;code&gt;export&lt;/code&gt;, and &lt;code&gt;S3method&lt;/code&gt;;&lt;/li&gt;&#xA;&lt;li&gt;It has several unit tests using &lt;code&gt;RUnit&lt;/code&gt;;&lt;/li&gt;&#xA;&lt;li&gt;It provides good examples of how to create/document S3 methods;&lt;/li&gt;&#xA;&lt;li&gt;It has some calls to C code via the &lt;code&gt;.Call&lt;/code&gt; interface;&lt;/li&gt;&#xA;&lt;li&gt;It contains a (plotting) demo;&lt;/li&gt;&#xA;&lt;li&gt;It aims to be consistent with the core R installation (e.g. functions behave similarly, it doesn't mask / override base functions, etc.)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;It doesn't use roxygen, which is very handy, but 7 out of 8 ain't bad. ;-)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To respond to your criteria:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The concept is simple: &lt;code&gt;zoo&lt;/code&gt; is a matrix-like class ordered by &lt;em&gt;something&lt;/em&gt;.  No domain-specific knowledge necessary.&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;zoo&lt;/code&gt; does seem to have a few idiosyncratic coding conventions, but nothing over-the-top that prevents understanding the code.&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;zoo&lt;/code&gt; aims to be as consistent with R as possible.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="1657" LastEditorUserId="1657" LastEditorDisplayName="" LastEditDate="2010-12-20T19:51:03.293" LastActivityDate="2010-12-20T19:51:03.293" CommentCount="3" />
  <row Id="5434" PostTypeId="1" AcceptedAnswerId="5437" CreationDate="2010-12-13T17:06:52.920" Score="3" ViewCount="220" Body="&lt;p&gt;I am doing multiple regression with some data (5 predictors, 1 response). Since the response is discrete and non-negative, I thought I would try Poisson regression. However, the data are significantly overdispersed (variance &gt; mean), so I am now trying negative binomial regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was able to fit the model with this code.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(MASS)&#xA;model.nb &amp;lt;- glm.nb(Response ~ Pred1 + Pred2 + Pred3 + Pred4 + Pred5 - 1, data=d)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now I would like to see if I can get a better fit by including interactions between the predictors. However, when I try to do so, I get the following error.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; model.nb.intr &amp;lt;- glm.nb(Response ~ Pred1 * Pred2 * Pred3 * Pred4 * Pred5 - 1, data=d)&#xA;Error: no valid set of coefficients has been found: please supply starting values&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Any ideas what may be causing this?&lt;/p&gt;&#xA;" OwnerUserId="1973" LastEditorUserId="1390" LastEditorDisplayName="" LastEditDate="2010-12-13T17:36:44.883" LastActivityDate="2010-12-14T22:23:25.407" Title="No valid coefficients for NegBin regression" Tags="&lt;r&gt;&lt;regression&gt;&lt;negative-binomial&gt;" AnswerCount="3" />
  <row Id="5435" PostTypeId="2" ParentId="5431" CreationDate="2010-12-13T17:17:46.440" Score="2" ViewCount="" Body="&lt;p&gt;It depends what you're testing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are testing for variance and your $H_0$ is that $V = 1$, but your values are $0, +1000, -1000, +2000, -2000$, then it's clear that you should reject $H_0$. If you are testing the hypothesis that the mean is not 2, then you do not have enough data to reject it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For most things, though, 5 is too little.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(I didn't run through the math above to build intuition. I'd generally recommend a Bayes ratio as the best quantitative test).&lt;/p&gt;&#xA;" OwnerUserId="2067" LastActivityDate="2010-12-13T17:17:46.440" />
  <row Id="5436" PostTypeId="2" ParentId="5434" CreationDate="2010-12-13T17:33:49.410" Score="7" ViewCount="" Body="&lt;p&gt;Your model is too complex for the computer to work out some reasonable starting values that do not lead to infinite deviance when doing the &lt;code&gt;glm.fit&lt;/code&gt; iterations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Have you got enough data to estimate all these interactions? Do you think it is plausible for all predictors to interact with each other? If not, think about which predictors might interact and include only those terms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The error asks you to supply some starting values for it to work from. For this, you need to supply a vector of parameter values as argument &lt;code&gt;start&lt;/code&gt;; from &lt;code&gt;?glm&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   start: starting values for the parameters in the linear predictor.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You need to supply 31 model parameters (I hope you have many 1000s of data points?) to &lt;code&gt;start&lt;/code&gt;, in this order:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; colnames(model.matrix(Y ~ Pred1*Pred2*Pred3*Pred4*Pred5 -1, data = DF))&#xA; [1] &quot;Pred1&quot;                         &quot;Pred2&quot;                        &#xA; [3] &quot;Pred3&quot;                         &quot;Pred4&quot;                        &#xA; [5] &quot;Pred5&quot;                         &quot;Pred1:Pred2&quot;                  &#xA; [7] &quot;Pred1:Pred3&quot;                   &quot;Pred2:Pred3&quot;                  &#xA; [9] &quot;Pred1:Pred4&quot;                   &quot;Pred2:Pred4&quot;                  &#xA;[11] &quot;Pred3:Pred4&quot;                   &quot;Pred1:Pred5&quot;                  &#xA;[13] &quot;Pred2:Pred5&quot;                   &quot;Pred3:Pred5&quot;                  &#xA;[15] &quot;Pred4:Pred5&quot;                   &quot;Pred1:Pred2:Pred3&quot;            &#xA;[17] &quot;Pred1:Pred2:Pred4&quot;             &quot;Pred1:Pred3:Pred4&quot;            &#xA;[19] &quot;Pred2:Pred3:Pred4&quot;             &quot;Pred1:Pred2:Pred5&quot;            &#xA;[21] &quot;Pred1:Pred3:Pred5&quot;             &quot;Pred2:Pred3:Pred5&quot;            &#xA;[23] &quot;Pred1:Pred4:Pred5&quot;             &quot;Pred2:Pred4:Pred5&quot;            &#xA;[25] &quot;Pred3:Pred4:Pred5&quot;             &quot;Pred1:Pred2:Pred3:Pred4&quot;      &#xA;[27] &quot;Pred1:Pred2:Pred3:Pred5&quot;       &quot;Pred1:Pred2:Pred4:Pred5&quot;      &#xA;[29] &quot;Pred1:Pred3:Pred4:Pred5&quot;       &quot;Pred2:Pred3:Pred4:Pred5&quot;      &#xA;[31] &quot;Pred1:Pred2:Pred3:Pred4:Pred5&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I would use the coefs from the first model to fill in the first 5 starting values and then what you do about the others is up to you. You could try starting them all off at 1 and see if that will get the model to fit?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might also benefit from code in the &lt;a href=&quot;http://cran.r-project.org/web/packages/pscl/index.html&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;pscl&lt;/strong&gt; package&lt;/a&gt; which can fit hurdle and zero-inflated models to count data.&lt;/p&gt;&#xA;" OwnerUserId="1390" LastEditorUserId="1390" LastEditorDisplayName="" LastEditDate="2010-12-13T17:46:32.053" LastActivityDate="2010-12-13T17:46:32.053" CommentCount="2" />
  <row Id="5437" PostTypeId="2" ParentId="5434" CreationDate="2010-12-13T17:57:44.083" Score="7" ViewCount="" Body="&lt;p&gt;Before jumping to a model that includes all interactions, you can try adding only the 2-way interactions:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.nb.intr &amp;lt;- glm.nb(Response ~ (Pred1 + Pred2 + Pred3 + Pred4 + Pred5)^2 - 1, data=d)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="279" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-12-13T19:32:30.540" LastActivityDate="2010-12-13T19:32:30.540" CommentCount="4" />
  <row Id="5438" PostTypeId="2" ParentId="5115" CreationDate="2010-12-13T18:20:38.607" Score="1" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://stat.stanford.edu/~donoho/&quot; rel=&quot;nofollow&quot;&gt;David Donoho&lt;/a&gt; development of multiscale ideas in statistics, and a lot of theoretically justified while practically very efficient ideas in very high dimensional statistics, CHA: computational harmonic analysis,... &lt;/p&gt;&#xA;" OwnerUserId="223" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-12-15T17:36:43.840" LastActivityDate="2010-12-15T17:36:43.840" CommunityOwnedDate="2010-12-13T18:20:38.607" CommentCount="2" />
  <row Id="5439" PostTypeId="2" ParentId="5431" CreationDate="2010-12-13T18:33:03.063" Score="2" ViewCount="" Body="&lt;p&gt;Five is pretty small, but not so small that it doesn't contain some information.  If you get 5 course evaluations that agree an instructor is ineffective, that's pretty strong evidence.  That said, the only time the evidence will be conclusive if it is rather extreme, e.g. all zeroes or all tens.  Still, it seems foolish to toss it out.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may also have a later opportunity to pool evidence, say if the same instructor or course is offered multiple times.  Then those fives start adding up.&lt;/p&gt;&#xA;" OwnerUserId="1920" LastActivityDate="2010-12-13T18:33:03.063" />
  <row Id="5440" PostTypeId="2" ParentId="5115" CreationDate="2010-12-13T18:39:58.043" Score="2" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Emanuel_Parzen&quot; rel=&quot;nofollow&quot;&gt;Emanuel Parzen&lt;/a&gt; for kernel density estimation&lt;/p&gt;&#xA;" OwnerUserId="223" LastActivityDate="2010-12-13T18:39:58.043" CommunityOwnedDate="2010-12-13T18:39:58.043" />
  <row Id="5441" PostTypeId="2" ParentId="5115" CreationDate="2010-12-13T18:42:29.023" Score="1" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Lucien_Le_Cam&quot; rel=&quot;nofollow&quot;&gt;Lucien Le Cam&lt;/a&gt; for his contribution to mathematical statistics. (maybe Local asymptotic normality and contiguity made him famous)&lt;/p&gt;&#xA;" OwnerUserId="223" LastActivityDate="2010-12-13T18:42:29.023" CommunityOwnedDate="2010-12-13T18:42:29.023" />
  <row Id="5442" PostTypeId="2" ParentId="1995" CreationDate="2010-12-13T19:08:53.720" Score="2" ViewCount="" Body="&lt;p&gt;Here's an example where a multilevel model might be &quot;essential.&quot; Suppose you want to rate the &quot;quality&quot; of the education provided by a set of schools using students' test scores.  One way to define school quality is in terms of average test performance after taking student characteristics into account.  You could conceptualized this as,&#xA;$$y_{is} = \alpha_s + X_{is}&amp;#39;\beta_s + \epsilon_{is},$$&#xD;&#xA;where $y_{is}$ is the continuous test score for student $i$ in school $s$, $X_{is}$ are student attributes centered at school means, $\beta_s$ is a school-specific coefficient on these attributes, $\alpha_s$ is a &quot;school effect&quot; that measures school quality, and $\epsilon_{is}$ are student level idiosyncrasies in test taking performance.  Interest here focuses on estimating the $\alpha_s$'s, which measure the &quot;added value&quot; that the school provides to students once their attributes are accounted-for.  You want to take student attributes into account, because you don't want to punish a good school that has to deal with students with certain disadvantages, therefore depressing average test scores despited the high &quot;added value&quot; that the school provides to its students.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;With the model in hand, the issue becomes one of estimation.  If you have lots of schools and lots of data for each school, the nice properties of OLS (see Angrist and Pischke, &lt;em&gt;Mostly Harmless...&lt;/em&gt;, for a current review) suggest that you would want to use that, with suitable adjustments to standard errors to account for dependencies, and using dummy variables and interactions to get at school level effects and school specific intercepts.  OLS may be inefficient, but it's so transparent that it might be easier to convince skeptical audiences if you use that.  But if your data are sparse in certain ways---particularly if you have few observations for some schools---you may want to impose more &quot;structure&quot; on the problem.  You may want to &quot;borrow strength&quot; from the larger-sample schools to improve the noisy estimates that you would get in the small-sample schools if the estimation were done with no structure.  Then, you might turn to a random effects model estimated via FGLS, or maybe an approximation to direct likelihood given a certain parametric model, or even Bayes on a parametric model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this example, the use of a multilevel model (however we decide to fit it, ultimately) is motivated by the direct interest in the school-level intercepts.  Of course, in other situations, these group level parameters may be nothing more than nuisance.  Whether or not you need to adjust for them (and, therefore, still work with some kind of multilevel model) depends on whether certain conditional exogeneity assumptions hold. On that, I would recommend consulting the econometric literature on panel data methods; most insights from there carry over to general grouped data contexts.&lt;/p&gt;&#xA;" OwnerUserId="96" LastActivityDate="2010-12-13T19:08:53.720" CommentCount="1" />
  <row Id="5443" PostTypeId="1" CreationDate="2010-12-13T19:24:57.693" Score="3" ViewCount="137" Body="&lt;p&gt;Hi all,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When building a CART model (specifically classification tree) using rpart (in R), it is sometimes obvious that there are variables (X's) that are meaningful for predicting some of the outcome (y) variables - while other predictors are relevant for other y's only.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How can it be estimated, which explanatory variable is &quot;used&quot; for which of the predicted value in the outcome variable?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an example code in which x2 is the only important variable for predicting &quot;b&quot; (one of the y outcomes).  There is no predicting variable for &quot;c&quot;, and x1 is a predictor for &quot;a&quot;, assuming that x2 permits it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can this situation be extracted from the fitted model? &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;N &amp;lt;- 200&#xA;set.seed(5123)&#xA;x1 &amp;lt;- runif(N)&#xA;x2 &amp;lt;- runif(N)&#xA;x3 &amp;lt;- runif(N)&#xA;y &amp;lt;- sample(letters[1:3], N, T)&#xA;y[x1 &amp;lt;.5] &amp;lt;- &quot;a&quot;&#xA;y[x2 &amp;lt;.1] &amp;lt;- &quot;b&quot;&#xA;&#xA;fit &amp;lt;- rpart(y ~ x1+x2)&#xA;fit2 &amp;lt;- prune(fit, cp= 0.07)&#xA;plot(fit2)&#xA;text(fit2, use.n=TRUE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="253" LastActivityDate="2010-12-16T02:34:59.040" Title="Estimate the &quot;meaningful&quot; predictors for a value in a CART model (rpart)" Tags="&lt;r&gt;&lt;cart&gt;&lt;rpart&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="5444" PostTypeId="1" AcceptedAnswerId="5445" CreationDate="2010-12-13T19:41:36.827" Score="2" ViewCount="99" Body="&lt;p&gt;&lt;strong&gt;Motivation&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a prior on a random variable $X\sim \beta^\prime(\alpha,\beta)$ that I would like to use in JAGS. JAGS does not support the $\beta^\prime$ distribution, but it does support the $F$ distributions, and the F distribution is related to the $\beta^\prime(\alpha,\beta)$ like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\text{if } X\sim \beta^\prime(\alpha,\beta) \text{ then } X\frac{\alpha}{\beta}\sim F(2\alpha, 2\beta)$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a transformation $$c,d = f(\alpha, \beta)$$ such that:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\text{if } X\sim \beta^\prime(\alpha,\beta)\text{ then } X\sim F(c, d)?$$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is an appropriate analytical approach or solution to this problem?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Current Approach&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My solution is to use simulation. Although sufficient for my application, a formal solution would be more satisfying.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   set.seed(0)&#xA;   alpha &amp;lt;- 2&#xA;   beta  &amp;lt;- 4&#xA;   Y &amp;lt;- rf(100000, 2*alpha, 2*beta) * ( beta / alpha )&#xA;   parms &amp;lt;-signif(fitdistr(Y, 'f', start = list(df1=1, df2=2))$estimate,2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt; whuber's answer states that there is no general transformation.&lt;/p&gt;&#xA;" OwnerUserId="1381" LastEditorUserId="1381" LastEditorDisplayName="" LastEditDate="2010-12-13T23:31:16.857" LastActivityDate="2010-12-13T23:31:16.857" Title="How do I transform the parameters of the F distribution?" Tags="&lt;data-transformation&gt;&lt;random-variable&gt;" AnswerCount="1" FavoriteCount="0" />
  <row Id="5445" PostTypeId="2" ParentId="5444" CreationDate="2010-12-13T20:30:30.397" Score="5" ViewCount="" Body="&lt;p&gt;I don't quite get it: why can't you just generate an $F_{2\alpha, 2\beta}$ variate and rescale it by $\beta/\alpha$?  I also don't see the connection between the code in your current approach and your question, because $Y$ appears to be an $F_{2\alpha, 2\beta}$ variate rescaled by $\alpha/\beta$ rather than  $\beta/\alpha$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Nevertheless, taking the question at face value, if there are parameters $c$ and $d$ corresponding to $\alpha$ and $\beta$, then&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The mean of $X$ must be $\beta/\alpha$ times the mean of $F_{2\alpha, 2\beta}$ and it is also equal to the mean of $F_{c,d}$.&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The mode of $X$ must be $\beta/\alpha$ times the mode of $F_{2\alpha, 2\beta}$ and it is also the mode of  $F_{c,d}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using standard &lt;a href=&quot;http://en.wikipedia.org/wiki/F-distribution&quot; rel=&quot;nofollow&quot;&gt;formulas&lt;/a&gt; and solving gives a unique solution&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\eqalign{&#xD;&#xA;c = &amp;amp; \frac{2 \left(\alpha^2+\alpha^2 \beta \right)}{\alpha -\alpha  \beta + 2 \alpha^2 \beta + 2 \beta ^2-2 \alpha  \beta^2} \cr&#xD;&#xA;&#xD;&#xA;d = &amp;amp; \frac{2 \beta^2}{\alpha -\alpha \beta +\beta^2}.&#xD;&#xA;}$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can also equate variances:&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The variance of $X$ must equal $(\beta/\alpha)^2$ times the variance of $F_{2\alpha, 2\beta}$ and it must equal the variance of $F_{c,d}$.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Except possibly for special values of $\alpha$ and $\beta$ (most notably, $\alpha = \beta$), this requirement conflicts with the preceding solution.  Therefore &lt;strong&gt;there does not exist any such transformation in general&lt;/strong&gt;.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-12-13T20:30:30.397" CommentCount="4" />
  <row Id="5446" PostTypeId="2" ParentId="5115" CreationDate="2010-12-13T22:10:48.643" Score="3" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/George_Dantzig&quot; rel=&quot;nofollow&quot;&gt;George Dantzig&lt;/a&gt; for the Simplex Method, and for being the student who mistook two open statistics problems that Neyman had written on the board for homework problems, and in his &quot;ignorance&quot; solving them. I'd vote for him just for the story.&lt;/p&gt;&#xA;" OwnerUserId="1764" LastActivityDate="2010-12-13T22:10:48.643" CommunityOwnedDate="2010-12-13T22:10:48.643" CommentCount="4" />
  <row Id="5447" PostTypeId="2" ParentId="5115" CreationDate="2010-12-13T23:21:55.837" Score="4" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Problem_of_points&quot; rel=&quot;nofollow&quot;&gt;Blaise Pascal and Pierre de Fermat&lt;/a&gt; for creating the theory of probability and inventing the idea of expected value (1654) in order to solve a problem grounded in statistical observations (from gambling).&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-12-13T23:21:55.837" CommunityOwnedDate="2010-12-13T23:21:55.837" />
  <row Id="5448" PostTypeId="1" CreationDate="2010-12-13T23:25:11.007" Score="2" ViewCount="169" Body="&lt;p&gt;I have conducted a confirmatory factor analysis to test the fit of a model of 5 factors with 5 items each to my sample. I used the modification indices to alter the model until I obtained statistics indicating an acceptable fit of the model to my data. As others have done in my area I then conducted a principal components analysis (PCA) with a varimax rotation to check the items loaded on the correct factors, however, not all of them did. Is this a usual result? Could this be because I have a small sample size (n96)? As the model is 'acceptable' in the confirmatory analysis should I simply work with the scale structures from those results and not do the PCA?&lt;/p&gt;&#xA;" OwnerUserId="2365" LastActivityDate="2010-12-14T09:50:58.897" Title="Confirmatory and explanatory factor analysis" Tags="&lt;factor-analysis&gt;" AnswerCount="1" />
  <row Id="5450" PostTypeId="1" AcceptedAnswerId="5455" CreationDate="2010-12-13T23:43:17.117" Score="8" ViewCount="380" Body="&lt;p&gt;In a regression, the interaction term wipes out both related direct effects. Do I drop the interaction or report the outcome? The interaction was not part of the original hypothesis. &lt;/p&gt;&#xA;" OwnerUserId="2367" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-12-14T08:35:11.153" LastActivityDate="2010-12-14T15:21:58.523" Title="What if interaction wipes out my direct effects in regression?" Tags="&lt;regression&gt;&lt;interaction&gt;" AnswerCount="3" CommentCount="4" FavoriteCount="2" />
  <row Id="5451" PostTypeId="2" ParentId="4551" CreationDate="2010-12-13T23:50:20.977" Score="2" ViewCount="" Body="&lt;p&gt;Requesting, and perhaps obtaining &lt;em&gt;The Flow Chart&lt;/em&gt;: That graphical thing where you say what the level of your variables are and what sort of relationship you're looking for, and you follow the arrows down to get a Brand Name Test or a Brand Name Statistic.  Sometimes offered with mysterious 'parametric' and 'non-parametric' paths.&lt;/p&gt;&#xA;" OwnerUserId="1739" LastActivityDate="2010-12-13T23:50:20.977" CommunityOwnedDate="2010-12-13T23:50:20.977" />
  <row Id="5452" PostTypeId="1" AcceptedAnswerId="5471" CreationDate="2010-12-14T00:09:21.473" Score="2" ViewCount="179" Body="&lt;p&gt;All,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am &lt;em&gt;trying&lt;/em&gt; to study the GBM package in R. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I. I wanted to try and figure out where the deviance, initial value, gradient  and terminal node estimates came from. Please see this snippet:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/xeCwF.jpg&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To start out, I was not sure where the Bernoulli deviance came from. I expected that the deviance, would be -2 multiplied by the log-likelihood&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/GNSRh.jpg&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;, ignoring the weight variable w. Can anyone suggest what I am doing (or missing) wrong to dervive what is shown in GBM?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;II. I was also confused on where the terminal node estimates came from?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help is very appreciated!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Brian&lt;/p&gt;&#xA;" OwnerUserId="2040" LastActivityDate="2010-12-14T13:20:06.913" Title="R Package GBM - Bernoulli Deviance" Tags="&lt;machine-learning&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="5453" PostTypeId="1" AcceptedAnswerId="5458" CreationDate="2010-12-14T01:45:45.857" Score="3" ViewCount="177" Body="&lt;p&gt;Let's say you have a process that generates data according to r = sin(t) + epsilon, where epsilon ~ N(0,V) is Gaussian noise. The unconditional variance of r is 0.5 + V. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say we're forecasting r with a model m, and that our forecast is &quot;perfect&quot; in that m = sin(t). Construct v = r - m, which is the forecast error, and will be ~ N(0,V). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;According to Bayes, we then have p(r|v) ~ p(v|r)*p(r), which is the the product of two Gaussian PDFs, one with variance V, the other (0.5+V). This product will itself be a Gaussian with total variance T = 1/(1/V + 1/(0.5+V)). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The funny thing is that T &amp;lt; V guaranteed, in fact T/V = (V+0.5)/(2V+0.5)! In other words, according to Bayes, the variance of your forecast error is less than the inherent noise in the data generation process itself!?! Isn't that impossible? Can anyone help me sort through this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you in advance,&#xA;-Jesse&lt;/p&gt;&#xA;" OwnerUserId="2379" LastActivityDate="2010-12-14T20:04:37.797" Title="Why does continuous Bayesian analysis seem to give this contradictory result?" Tags="&lt;bayesian&gt;&lt;inference&gt;" AnswerCount="4" CommentCount="7" />
  <row Id="5454" PostTypeId="2" ParentId="5418" CreationDate="2010-12-14T02:02:46.990" Score="5" ViewCount="" Body="&lt;p&gt;I do not consider myself an established R package developer but have recently undergone the process of writing and maintaining a package for my work environment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I had previously been writing / maintaining / updating a set of scripts that I would pass from project to project via the &lt;code&gt;source()&lt;/code&gt; function. The end result of this was that I'd end up with mostly redundant scripts hanging out in various places on our network drives. It was never clear where the most up-to-date set of scripts were located. I have since migrated to writing / maintaining a package utilizing roxygen. It has drastically simplified my life and made it easier to share my work with colleagues.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Based on your criteria above, I second the recommendation of reviewing the packages that Hadley has written. In particular, I think reading through the &lt;a href=&quot;https://github.com/hadley/devtools/wiki&quot; rel=&quot;nofollow&quot;&gt;devtools wiki&lt;/a&gt; would be very helpful. Hadley's code is well documented and several of his packages utilize roxygen. I think writing and maintaining one document for both R functions and R documentation is much easier than having them split out in two locations (.R and .RD files). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hadley's packages also serve some fairly basic concepts and are relatively easy to deparse (imho) if you are looking for pointers on the technical aspect ideas. I find myself digging through the &lt;a href=&quot;https://github.com/hadley/plyr&quot; rel=&quot;nofollow&quot;&gt;plyr&lt;/a&gt; source code when I'm looking for a pointer on roxygen documentation or other fundamental tasks. &lt;/p&gt;&#xA;" OwnerUserId="696" LastActivityDate="2010-12-14T02:02:46.990" CommentCount="3" />
  <row Id="5455" PostTypeId="2" ParentId="5450" CreationDate="2010-12-14T02:25:20.237" Score="11" ViewCount="" Body="&lt;p&gt;I think this one is tricky; as you hint, there's 'moral hazard' here: if you hadn't looked at the interaction at all, you'd be free and clear, but now that you have there is a suspicion of data-dredging if you drop it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The key is &lt;em&gt;probably&lt;/em&gt; a change in the meaning of your effects when you go from the main-effects-only to the interaction model. What you get for the 'main effects' depends very much on how your treatments and contrasts are coded. In R, the default is treatment contrasts with the first factor levels (the ones with the first names in alphabetical order unless you have gone out of your way to code them differently) as the baseline levels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say (for simplicity) that you have two levels, 'control' and 'trt', for each factor. Without the interaction, the meaning of the 'v1.trt' parameter (assuming treatment contrasts as is the default in R) is &quot;average difference between 'v1.control' and 'v1.trt' group&quot;; the meaning of the 'v2.trt' parameter is &quot;average difference between 'v2.control' and 'v2.trt'&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With the interaction, 'v1.trt' is the average difference between 'v1.control' and 'v1.trt' &lt;strong&gt;in the 'v2.control' group&lt;/strong&gt;, and similarly 'v2.trt' is the average difference between v2 groups in the 'v1.control' group. Thus, if you have fairly small treatment effects in each of the control groups, but a large effect in the treatment groups, you could easily see what you're seeing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The only way I can see this happening &lt;em&gt;without&lt;/em&gt; a significant interaction term, however, is if all the effects are fairly weak (so that what you really mean by &quot;the effect disappeared&quot; is that you went from p=0.06 to p=0.04, across the magic significance line).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another possibility is that you are 'using up too many degrees of freedom' -- that is, the parameter estimates don't actually change that much, but the residual error term is sufficiently inflated by having to estimate another 4 [ = (2-1)*(5-1)] parameters that your significant terms become non-significant. Again, I would only expect this with a small data set/relatively weak effects.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One possible solution is to move to sum contrasts, although this is also delicate -- you have to be convinced that 'average effect' is meaningful in your case. The very best thing is to plot your data and to look at the coefficients and understand what's happening in terms of the estimated parameters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope that helps.&lt;/p&gt;&#xA;" OwnerUserId="2126" LastActivityDate="2010-12-14T02:25:20.237" CommentCount="1" />
  <row Id="5456" PostTypeId="2" ParentId="5115" CreationDate="2010-12-14T03:02:22.327" Score="7" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Jerzy_Neyman&quot; rel=&quot;nofollow&quot;&gt;Jerzy Neyman&lt;/a&gt; and &lt;a href=&quot;http://en.wikipedia.org/wiki/E.S._Pearson&quot; rel=&quot;nofollow&quot;&gt;Egon Pearson&lt;/a&gt; for work on experimental design, hypothesis testing, confidence intervals, and the Neyman-Pearson lemma.&lt;/p&gt;&#xA;" OwnerUserId="2077" LastActivityDate="2010-12-14T03:02:22.327" CommunityOwnedDate="2010-12-14T03:02:22.327" CommentCount="1" />
  <row Id="5457" PostTypeId="1" AcceptedAnswerId="5459" CreationDate="2010-12-14T03:09:29.683" Score="5" ViewCount="299" Body="&lt;p&gt;When I tell my non-statistical friends I am a grad student pursuing a PhD in statistics they naturally say &quot;oh so you want to be a professor?&quot;.  I tell them no I actually plan to work in industry.  Then they respond with, &quot;and do what?&quot;.  I have not found a good response to this question.  I want to give them a range of interesting problems that statisticians work on, but my response is usually muddled or is too technical.  I think most people picture us working in excel and calculating means and standard deviations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is a good response to this question that is brief and interesting?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks! (not sure what a good existing tag is for this question)&lt;/p&gt;&#xA;" OwnerUserId="2310" LastActivityDate="2010-12-14T23:17:09.983" CommunityOwnedDate="2010-12-14T20:00:01.800" Title="What is it that a statistician does?" Tags="&lt;careers&gt;" AnswerCount="7" CommentCount="3" FavoriteCount="3" />
  <row Id="5458" PostTypeId="2" ParentId="5453" CreationDate="2010-12-14T03:28:01.720" Score="2" ViewCount="" Body="&lt;p&gt;Actually, the variances are zero: $V(v|r) = V(r|v) = 0$.$p(v|r)$ is a Dyrac function which has a peak at the right spot (where $r = v -m$).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you know $v$ or $r$, the other one is a deterministic function of the one you know.&lt;/p&gt;&#xA;" OwnerUserId="2067" LastEditorUserId="2067" LastEditorDisplayName="" LastEditDate="2010-12-14T16:28:03.290" LastActivityDate="2010-12-14T16:28:03.290" CommentCount="3" />
  <row Id="5459" PostTypeId="2" ParentId="5457" CreationDate="2010-12-14T03:54:53.847" Score="10" ViewCount="" Body="&lt;p&gt;The area I am most interested in is the realm of biostatistics. Statistics can be used in this regard to do anything from summarize the results of a drug trial, determining whether Prozac really is more effective than the placebo sugar pill, to tumor detection in cancer patients. Please check out this presentation I found: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.stat.ncsu.edu/biostat/biomed.pdf&quot; rel=&quot;nofollow&quot;&gt;What is Biostatistics?&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Remember, a statistician is a function that maps a set of data to a set of decisions.&lt;/p&gt;&#xA;" OwnerUserId="1118" LastActivityDate="2010-12-14T03:54:53.847" CommunityOwnedDate="2010-12-14T20:00:01.800" CommentCount="1" />
  <row Id="5460" PostTypeId="2" ParentId="5457" CreationDate="2010-12-14T04:18:06.930" Score="4" ViewCount="" Body="&lt;p&gt;A statistician performs the following generic steps:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Has a goal- test a theory, generate a prediction etc&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Obtains data (via surveys, experiments, thrid-parties etc) that is consistent with the above goal.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Develops a statistical model that says roughly: Variable of interest = function of covariates + random error.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Estimate the impact of covariates on the variable of interest using various tools from math/stats.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Use the resulting estimates to evaluate theory/generate predictions.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;You can contextualize the above generic steps to various settings. The context can be changed depending on the audience. For example:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;With doctors: discuss how you would assess the effectiveness of drugs&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;With a more general audience discuss how you would assess the winner in an upcoming election&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;With business professionals discuss how you would assess the relative impact of advertising designs etc.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" LastActivityDate="2010-12-14T04:18:06.930" CommunityOwnedDate="2010-12-14T20:00:01.800" />
  <row Id="5461" PostTypeId="1" CreationDate="2010-12-14T04:19:26.873" Score="1" ViewCount="114" Body="&lt;p&gt;The title might be a bit misleading. Unfortunately statistics is not my area of forte, so gentle guidance along the right path is much appreciated. That said, here's my problem:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm working on analyzing precise time measurements from an experimental process. I have 16 measurement groups. Of these, one should (on average) differ slightly in magnitude from the others. The remaining 15 should have similar or identical properties. I'm trying to determine which of these measurement groups is the odd one out. My first pass at the problem involved a naive data collection routine rotating between sample groups until enough samples from each group were collected.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My data is clearly not normal. Furthermore, sample values seem to fluctuate as a function of time (non-scientific visual inspection shows that larger values and smaller values tend to cluster across sample groups). The variable under consideration should not be fluctuating like this, so I believe this effect is attributable to some other element in an admittedly noisy environment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My first pass at analysis was to run a KS test comparing each sample group to the set consisting of all other collected data. This worked as expected: the sample which was known to vary showed a much lower p-value than the other candidates.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The KS test isn't ideal here. Despite a theoretical belief that the data is continuous, my measurement resolution limits dictate that there must be a number of duplicate data points (3 effective places of resolution and n between 10k and 1m per group).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unfortunately, this sample set was taken from a somewhat idealized experimental condition. I need to improve my analysis to include some larger elements of  noise. I have done the experiment in my real-world condition, and have been having trouble finding reliable ways to analyze my data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem seems to be related to the fact that while my effect remains constant over much larger (and thus longer-running) sampling regimes, the macro-level time changes seem to be interfering. As I move to an order of magnitude more samples in my real-world condition, rather than converging on one single value as different, the KS test starts to show that ALL the values differ (and the ordering is inconsistent).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've re-written my data collection to log both the timestamp for each sample, and several timing values taken over the course of my experimental procedure (originally I was just logging a single aggregate timing result by calculating a difference using these values). With this expanded data in hand, I've been looking for appropriate analysis techniques. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's where I'm stumped. I'm not sure what I should do with this timestamp data. I've turned a multiple sample comparison problem into a time sequence. If I can use the time data to even out my values, I'm still (at least potentially) left with a complicated multi-variable analysis. PCA assumes normality so that's out. ICA isn't appropriate because I know that most of my measured variables are not, in fact, independent of each other. Once I've sorted that out, I still need to compare to the overall group and figure out which one is different. Because each sample-per-group has a discrete timestamp, I don't have a simple way to compare across my sampling groups. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems like the Wilcoxon–Mann–Whitney (Wilcoxon rank-sum) test is likely to be of use here, but only after I've done work on the underlying data. Suggestions?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: for what it's worth, this is a practical problem. Approximate solutions and things that narrow the candidate field are perfectly reasonable. No points for excessively pretty P values here.&lt;/p&gt;&#xA;" OwnerUserId="1750" LastEditorUserId="1750" LastEditorDisplayName="" LastEditDate="2010-12-14T18:38:37.697" LastActivityDate="2010-12-15T03:04:59.567" Title="Removing macro-level time variance" Tags="&lt;time-series&gt;&lt;hypothesis-testing&gt;&lt;statistical-significance&gt;&lt;multiple-comparisons&gt;" AnswerCount="1" CommentCount="6" />
  <row Id="5462" PostTypeId="1" AcceptedAnswerId="5464" CreationDate="2010-12-14T05:23:40.037" Score="9" ViewCount="284" Body="&lt;p&gt;This is only for my high school senior project, so it doesn't have to be perfect. I'm doing a project on global warming, and I want to survey people for their opinions. I know that if I use a convenience sample of my classmates, I'll have a lot of bias. I was wondering if there was a site on the Internet where I could post my survey for random people to answer, so that I could get as close as possible to an SRS. If not, I'd be interested in some other suggestions for ways to carry out my survey. &lt;/p&gt;&#xA;" OwnerUserId="2371" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-12-14T08:35:51.997" LastActivityDate="2010-12-15T21:15:14.057" Title="Is there a site to post my survey to so I can get a sample representative of the population?" Tags="&lt;survey&gt;&lt;online&gt;" AnswerCount="7" />
  <row Id="5463" PostTypeId="2" ParentId="5462" CreationDate="2010-12-14T05:45:34.817" Score="3" ViewCount="" Body="&lt;p&gt;wow, i admire your ambition! To answer you question directly, no i'm not familiar with any site. Perhaps a message board you're familiar with? Has your instructor addressed the issue of bias? It may be enough that you recognize the issue of bias in your sample and speak to how this will affect your survey. Most properly randomized nationally representative surveys have an N of somewhere between 1800 and 2300. &lt;a href=&quot;http://www.norc.uchicago.edu/GSS+Website/&quot; rel=&quot;nofollow&quot;&gt;The GSS probably has a few questions regarding global warming&lt;/a&gt;.&#xA;However this will be inaccessible if you're not familiar with SPSS or STATA. I know others on here can help you better than I in regards to weighting responses to correct for bias. &lt;/p&gt;&#xA;" OwnerUserId="2262" LastActivityDate="2010-12-14T05:45:34.817" CommentCount="4" />
  <row Id="5464" PostTypeId="2" ParentId="5462" CreationDate="2010-12-14T06:13:02.323" Score="7" ViewCount="" Body="&lt;p&gt;For a high school project it will probably be difficult to gain a representative sample with your limited time and budget. I'd be somewhat skeptical an online vendor can achieve a random sample of anything. If I had to do it I think I would use the phone book and either call or go to the addresses to distribute the survey in your community. Although this isn't perfect (likley under-represents younger people, minorities, and people of lower socio-economic status) it makes the project more feasible in your situation (as opposed to conducting stratified sampling based on geographic regions). Both of these solutions are local samples, and would not get you a national sample. An agency I work for samples the local community based on parcel addresses and apartment listings obtained from the postal office (these sources of information are not always public though, and would require more front end work than opening up a phone book).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;idclark is right and the &lt;a href=&quot;http://www.norc.uchicago.edu/GSS+Website/Documentation/&quot; rel=&quot;nofollow&quot;&gt;General Social Survey&lt;/a&gt; has several variables addressing public opinion on global warming. You could either analyze that data directly (as it is open to the public) or if you are required to construct your own survey you could mimic the GSS's questions and then see how close your sample appears to be compared to the GSS. At the moment I am having a much easier time searching the GSS variable lists from the &lt;a href=&quot;http://www.icpsr.umich.edu/icpsrweb/ICPSR/ssvd/variables?study=General+Social+Surveys,+1972-2008+%5bCumulative+File%5d&quot; rel=&quot;nofollow&quot;&gt;ICPSR archive&lt;/a&gt; than I am from the actual GSS website. You can actually conduct basic data analysis online at ICPSR of the &lt;a href=&quot;http://www.icpsr.umich.edu/icpsrweb/ICPSR/studies/4697?archive=ICPSR&amp;amp;sdaAvailable=true&amp;amp;q=GSS&quot; rel=&quot;nofollow&quot;&gt;GSS data&lt;/a&gt; (you can basically do frequencies and cross-tabulations). If all you need are frequencies this circumvents the need to use stat software such as SPSS or Stata. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Getting a representative sample is difficult not only because true populations which to construct your sample are sometimes hard to come by, but just as importantly not everybody you do sample responds to the survey. I wouldn't be fixated on a national sample either, I'm sure you can construct just as interesting questions to answer by estimating public opinion of the local community, school mates, teachers, etc. What I find interesting is in comparison (like say comparing your school teachers opinions to those of the students). You could also conduct an experiment via surveys (like say manipulate how you present arguments for or against global warming and see how people differ in their opinions of global warming).&lt;/p&gt;&#xA;" OwnerUserId="1036" LastActivityDate="2010-12-14T06:13:02.323" CommentCount="1" />
  <row Id="5465" PostTypeId="1" CreationDate="2010-12-14T06:20:48.903" Score="14" ViewCount="1316" Body="&lt;p&gt;I am looking for some statistics (and probability, I guess) interview questions, from the most basic through the more advanced. Answers are not necessary (although links to specific questions on this site would do well).&lt;/p&gt;&#xA;" OwnerUserId="795" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-14T10:11:27.107" LastActivityDate="2011-02-11T15:01:37.667" CommunityOwnedDate="2010-12-14T08:48:45.083" Title="Statistics interview questions" Tags="&lt;big-list&gt;" AnswerCount="15" CommentCount="2" FavoriteCount="5" />
  <row Id="5466" PostTypeId="2" ParentId="5465" CreationDate="2010-12-14T07:08:35.253" Score="2" ViewCount="" Body="&lt;p&gt;I often ask &quot;how would you define/explain what forecasting is?&quot; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Answer to that type of very general question helps me to see if people are connected to a particular case of forecasting. There is not a right answer but answering this synthetically during an interview is not always easy:) &lt;/p&gt;&#xA;" OwnerUserId="223" LastEditorUserId="795" LastEditorDisplayName="" LastEditDate="2010-12-14T23:00:17.420" LastActivityDate="2010-12-14T23:00:17.420" CommunityOwnedDate="2010-12-14T09:37:12.100" />
  <row Id="5467" PostTypeId="2" ParentId="5462" CreationDate="2010-12-14T07:17:58.210" Score="0" ViewCount="" Body="&lt;p&gt;a sample representative of population cannot be obtained through internet as you will only get people interested in answering your survey online, which will give you a biased sample.&lt;/p&gt;&#xA;" OwnerUserId="1709" LastActivityDate="2010-12-14T07:17:58.210" CommentCount="2" />
  <row Id="5468" PostTypeId="2" ParentId="5448" CreationDate="2010-12-14T08:01:23.333" Score="1" ViewCount="" Body="&lt;p&gt;What is the rationale of applying an exploratory/unsupervised method (PCA or FA with VARIMAX rotation) after having tested a confirmatory model, especially if this is done on the same sample? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your CFA model, you impose constraints on your pattern matrix, e.g. some items are supposed to load on one factor but not on the others. A large modification index indicates that freeing a parameter or removing an equality constraint could result in better model fit. Item loadings are already available through your model fit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the contrary, in PCA or FA there is no such constraint, even following an orthogonal rotation (whose purpose is just to make factor more interpretable in that items would generally tend to load more heavily on a factor than on several ones). But, it is worth noting that these models are conceptually and mathematically different: the FA model is a measurement model, where we assume that there is some unique error attached to each item; this is not the case under the PCA framework. It is thus not surprising that you failed to replicate your factor structure, which may be an indication that there are possible item cross-loading, low item reliability, low stability in your factor structure, or the existence of a higher-order factor structure, that is enhanced by your low sample size.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In both case, but especially CFA, $N=96$ is a very limited sample size. Although some authors have suggested a ratio individuals:items of 5 to 10, this is merely the number of dimensions that is important. In your case, the estimation of your parameters will be noisy, and in the case of PCA you may expect fluctuations in your estimated loadings (just try bootstrap to get an idea of 95% CIs).&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-12-14T09:50:58.897" LastActivityDate="2010-12-14T09:50:58.897" CommentCount="1" />
  <row Id="5469" PostTypeId="2" ParentId="5457" CreationDate="2010-12-14T08:10:11.293" Score="2" ViewCount="" Body="&lt;p&gt;I'm not a statistician, but a lot of my work involves statistics, and I work in health care.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The two things that I spend most of my time doing are:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;a) examining the sizes of effects and trends and seeing if they are &quot;real&quot;&#xA;b) presenting very large datasets in a simple way so that managers and users of our services can understand them- usually in the form of graphs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having said that, I have NEVER successfully explained my job to anyone at a party, so I think I'm in the same boat as you! I love it when people say &quot;that sounds interesting&quot;- because it doesn't, not the way I tell it!&lt;/p&gt;&#xA;" OwnerUserId="199" LastActivityDate="2010-12-14T08:10:11.293" CommunityOwnedDate="2010-12-14T20:00:01.800" />
  <row Id="5470" PostTypeId="2" ParentId="5465" CreationDate="2010-12-14T08:12:55.613" Score="11" ViewCount="" Body="&lt;p&gt;Not sure what the job is, but I think &quot;Explain x to a novice&quot; would probably be good- &lt;/p&gt;&#xA;&#xA;&lt;p&gt;a) because they will probably need to do this in the job&lt;/p&gt;&#xA;&#xA;&lt;p&gt;b) it's a good test of understanding, I reckon.&lt;/p&gt;&#xA;" OwnerUserId="199" LastActivityDate="2010-12-14T08:12:55.613" CommunityOwnedDate="2010-12-14T09:37:12.100" CommentCount="4" />
  <row Id="5471" PostTypeId="2" ParentId="5452" CreationDate="2010-12-14T09:16:36.017" Score="4" ViewCount="" Body="&lt;p&gt;It is a mathematical trick. We have&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align*}&#xA;\log\frac{p_i}{1-p_i}=f(x_i)&#xA;\end{align*}&#xA;and from this we get&#xA;\begin{align*}&#xA;\frac{1}{1-p_i}=1+\exp(f(x_i))&#xA;\end{align*} &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The log likelihood is&#xA;\begin{align*}&#xA;\sum_{i=1}^n\left[y_i\log(p_i)+(1-y_i)\log(1-p_i)\right]&amp;amp;=\sum_{i=1}^n\left[y_i\log\frac{p_i}{1-p_i}+\log(1-p_i)\right]\\&#xA;&amp;amp;=\sum_{i=1}^n\left[y_if(x_i)-\log\frac{1}{1-p_i}\right]\\&#xA;&amp;amp;=\sum_{i=1}^n\left[y_if(x_i)-\log(1+\exp(f(x_i)))\right]\\&#xA;\end{align*}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Only some terms were rearranged. I hope I made clear how exactly it was done.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2010-12-14T13:20:06.913" LastActivityDate="2010-12-14T13:20:06.913" CommentCount="2" />
  <row Id="5472" PostTypeId="2" ParentId="5465" CreationDate="2010-12-14T09:17:59.400" Score="6" ViewCount="" Body="&lt;p&gt;Standard Q where I work is along the lines of:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Have a look at this output of a multiple logistic regression from a statistical package you claim to have used (preferably one we use too). XXX is the independent variable of principal interest. How woud you interpret the results for a colleague with knowledge of the subject matter but no formal statistical training? (If necessary prompt for separate interpretation of point estimate, CI, p-value).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="449" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-12-14T14:52:25.680" LastActivityDate="2010-12-14T14:52:25.680" CommunityOwnedDate="2010-12-14T09:17:59.400" CommentCount="1" />
  <row Id="5473" PostTypeId="2" ParentId="5450" CreationDate="2010-12-14T11:41:45.440" Score="2" ViewCount="" Body="&lt;p&gt;In a regular multiple regression with two quantitative predictor variables, including their interaction just means including their observation-wise product as an additional predictor variable: $Y = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \beta_3 (X_1 \cdot X_2) = (b_0 + b_2 X_2) + (b_1 + b_3 X_2) X_1$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This typically introduces high multicollinearity since the product will strongly correlate with both original variables. With multicollinearity, individual parameter estimates depend strongly on which other variables are considered - like in your case. As a counter-measure, centering the variables often reduces multicollinearity when the interaction is considered.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not sure if this directly applies to your case since you seem to have categorical predictors but use the term &quot;regression&quot; instead of &quot;ANOVA&quot;. Of course the latter case is essentially the same model, but only after choosing the contrast coding scheme as Ben explained.&lt;/p&gt;&#xA;" OwnerUserId="1909" LastActivityDate="2010-12-14T11:41:45.440" />
  <row Id="5474" PostTypeId="2" ParentId="5452" CreationDate="2010-12-14T12:41:25.990" Score="0" ViewCount="" Body="&lt;p&gt;Hi, I'm also studying the GBM package!&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;mpiktas, i think you forgot a log on the left hand side in the 2nd equation? I assume you substitute p_i with 1/(1+exp(-f(x_i))), but then in the 2nd row above there is log(1/...) = log(1+...), or am i wrong? Anyway, i think then you did it right in the third row...&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Can you tell us what is the motivation for choosing p_i = 1/(1+exp(-f(x_i)))? Where does that come form? The p_i should mirror the proportion of success, i.e. class proportions, right?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Thanks!&#xA;peter&lt;/p&gt;&#xA;" OwnerUserId="2375" LastActivityDate="2010-12-14T12:41:25.990" CommentCount="1" />
  <row Id="5475" PostTypeId="2" ParentId="5465" CreationDate="2010-12-14T12:57:48.713" Score="3" ViewCount="" Body="&lt;p&gt;I was asked once how I would explain the relevance of the central limit theorem to a class of freshmen in the social sciences that barely have knowledge about statistics.&lt;/p&gt;&#xA;" OwnerUserId="1934" LastActivityDate="2010-12-14T12:57:48.713" CommunityOwnedDate="2010-12-14T12:57:48.713" />
  <row Id="5476" PostTypeId="2" ParentId="5465" CreationDate="2010-12-14T13:12:15.970" Score="0" ViewCount="" Body="&lt;p&gt;Tell me how Fisher's exact test works, and what it's good for.&lt;/p&gt;&#xA;" OwnerUserId="1920" LastActivityDate="2010-12-14T13:12:15.970" CommunityOwnedDate="2010-12-14T13:12:15.970" CommentCount="2" />
  <row Id="5477" PostTypeId="2" ParentId="5462" CreationDate="2010-12-14T13:13:52.900" Score="0" ViewCount="" Body="&lt;p&gt;Marketing people are using self-selected samples in online-surveys all the time, so they probably have methods for cleaning their data. Sadly, I don't have a good pointer right now to look for their methods.&lt;/p&gt;&#xA;" OwnerUserId="1766" LastActivityDate="2010-12-14T13:13:52.900" />
  <row Id="5478" PostTypeId="2" ParentId="5465" CreationDate="2010-12-14T14:01:47.267" Score="5" ViewCount="" Body="&lt;p&gt;You might also want to reflect on whether the interview is the best medium for measuring the construct of interest.&#xA;If you want to measure prior knowledge of probability or statistics, you might be better off relying more on a written test.&#xA;You can ask more questions, and thus increase reliability of measurement. It's more standardised both in administration, and in scoring. And once the instrument is developed, it probably uses fewer resources to administer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could then use the interview as a more focussed tool looking at factors such as verbal and interpersonal skills.&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-12-14T14:01:47.267" CommunityOwnedDate="2010-12-14T14:01:47.267" CommentCount="1" />
  <row Id="5479" PostTypeId="1" AcceptedAnswerId="5505" CreationDate="2010-12-14T14:05:31.890" Score="4" ViewCount="202" Body="&lt;p&gt;I am teaching myself DLM's using R's &lt;code&gt;dlm&lt;/code&gt; package and have two strange results. I am modeling a time series using three combined elements: a trend (&lt;code&gt;dlmModPoly&lt;/code&gt;), seasonality (&lt;code&gt;dlmModTrig&lt;/code&gt;), and moving seasonality (&lt;code&gt;dlmModReg&lt;/code&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first strange result is with the &lt;code&gt;$f&lt;/code&gt; (one-step-ahead foreacast) result. Most of this forecast appears to be one month behind the actual data, which I believe I've seen in many examples of one-step-ahead forecasting online and in books. The strange thing is that the moving seasonality is NOT similarly lagged, but hits exactly where it should. Is this normal?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I use the result's &lt;code&gt;$m&lt;/code&gt; to manually assemble the componenet, everything lines up perfectly, so it's weird, though it makes sense in a way: the moving seasonality has exogenous data to help it while the rest of the forecast does not. (Still, it'd be nice to simply &lt;code&gt;lag&lt;/code&gt; the resulting &lt;code&gt;$f&lt;/code&gt; and see a nice match.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More troubling is the difference I see if I change the degree of &lt;code&gt;dlmModPoly&lt;/code&gt;'s polynomial (from 1 to 2) in an attempt to get a smoother level. This introduces a huge spike in all three components at month 9. The spikes all basically cancel out in the composite, but obviously make each piece, say the level or the seasonality, look rather ridiculous there.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this just one of those things that happens and I should be prepared to throw away the result's first year of data as &quot;break-in&quot;? Or is it an indication that something is wrong? (Even in the degree 1 polynomial case, the first year's moving seasonality's level is a bit unsettled, but no huge spike as when I use a degree 2 polynomial.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is my R code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;lvl0 &amp;lt;- log (my.data[1])&#xA;slp0 &amp;lt;- mean (diff (log (my.data)))&#xA;&#xA;buildPTR2 &amp;lt;- function (x)&#xA;   {&#xA;   pm &amp;lt;- dlmModPoly (order=1, dV=exp (x[1]), dW=exp (x[2]), m0=lvl0)&#xA;   tm &amp;lt;- dlmModTrig (s=12, dV=exp (x[1]), q=2, dW=exp (x[3:4]))&#xA;   rm &amp;lt;- dlmModReg (moving.season, dV=exp (x[1]))&#xA;&#xA;   ptrm &amp;lt;- pm + tm + rm&#xA;   return (ptrm)&#xA;   }&#xA;&#xA;mlptr2 &amp;lt;- dlmMLE (log (my.data), rep (1, 6), buildPTR2)&#xA;dptr2 &amp;lt;- buildPTR2 (mlptr2$par)&#xA;dptrf2 &amp;lt;- dlmFilter (log (my.data), dptr2)&#xA;&#xA;tsdiag (dptrf2)&#xA;&#xA;buildPTR3 &amp;lt;- function (x)&#xA;   {&#xA;   pm &amp;lt;- dlmModPoly (order=2, dV=exp (x[1]), dW=c(0, exp (x[2])), m0=c(lvl0, slp0))&#xA;   tm &amp;lt;- dlmModTrig (s=12, dV=exp (x[1]), q=2, dW=exp (x[3:4]))&#xA;   rm &amp;lt;- dlmModReg (moving.season, dV=exp (x[1]))&#xA;&#xA;   ptrm &amp;lt;- pm + tm + rm&#xA;   return (ptrm)&#xA;   }&#xA;&#xA;mlptr3 &amp;lt;- dlmMLE (log (my.data), rep (1, 8), buildPTR3)&#xA;dptr3 &amp;lt;- buildPTR3 (mlptr3$par)&#xA;dptrf3 &amp;lt;- dlmFilter (log (my.data), dptr3) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Per the follow-on question: the data itself is monthly data for 10 years, with each month being the weekly average attendance at a theatrical production. The data definitely has seasonal and moving seasonal effects. I want to model the trend and the seasonal effects to give the management some insight, and to prepare for forecasting. (Which is not directly possible with &lt;code&gt;dlm&lt;/code&gt; when you include a &lt;code&gt;dlmModReg&lt;/code&gt; component, though that's the next step.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(I am trying to use an order=2 polynomial component that I believe creates an IRW trend, which is supposed to be nicely smooth.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If it matters, my moving seasonality is a yearly Big Bash Gala event that can fall in two different months, and I indicate it with 0 for most months and 1 for months in which the Big Bash falls.&lt;/p&gt;&#xA;" OwnerUserId="1764" LastEditorUserId="1764" LastEditorDisplayName="" LastEditDate="2010-12-14T20:13:14.577" LastActivityDate="2010-12-15T06:14:58.613" Title="DLM results looking wonky" Tags="&lt;r&gt;&lt;time-series&gt;&lt;dlm&gt;" AnswerCount="2" CommentCount="3" FavoriteCount="1" />
  <row Id="5480" PostTypeId="2" ParentId="5457" CreationDate="2010-12-14T14:12:47.083" Score="2" ViewCount="" Body="&lt;p&gt;My attempt at a simple answer that is both applicable across sub-domains and understandable (in gist) to the lay-person: When science develops theories about the world, these theories are compared to real-world data. The role of the statistician is to assess how well one or more competing theories account for the data. This is achieved with mathematics that let the statistician quantify their uncertainty about the conclusions they draw from the comparison of theory to data.&lt;/p&gt;&#xA;" OwnerUserId="364" LastActivityDate="2010-12-14T14:12:47.083" CommunityOwnedDate="2010-12-14T20:00:01.800" />
  <row Id="5481" PostTypeId="2" ParentId="5450" CreationDate="2010-12-14T15:21:58.523" Score="3" ViewCount="" Body="&lt;p&gt;Are you sure the variables have been appropriately expressed?  Consider two independent variables $X_1$ and $X_2$.  The problem statement asserts that you are getting a good fit in the form&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Y = \beta_0 + \beta_{12} X_1 X_2 + \epsilon$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If there is some evidence that the variance of the residuals increases with $Y$, then a better model uses multiplicative error, of which one form is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Y = \beta_0 + \left( \beta_{12} X_1 X_2 \right) \delta$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This can be rewritten&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\log(Y - \beta_0) = \log(\beta_{12}) + \log(X_1) + \log(X_2) + \log(\delta);$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;that is, if you re-express your variables in the form&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\eqalign{&#xD;&#xA;\eta =&amp;amp; \log(Y - \beta_0) \cr&#xD;&#xA;\xi_1 =&amp;amp; \log(X_1)\cr&#xD;&#xA;\xi_2 =&amp;amp; \log(X_2)\cr&#xD;&#xA;\zeta =&amp;amp; \log(\delta) \sim N(0, \sigma^2)&#xD;&#xA;}$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;then the model is linear and likely has homoscedastic residuals:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\eta = \gamma_0 + \gamma_1 \xi_1 + \gamma_2 \xi_2 + \zeta,$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and &lt;em&gt;it may just so happen&lt;/em&gt; that $\gamma_1$ and $\gamma_2$ are both close to 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The value of $\beta_0$ can be discovered through standard methods of &lt;a href=&quot;http://www.library.wisc.edu/selectedtocs/bb548.pdf&quot; rel=&quot;nofollow&quot;&gt;exploratory data analysis&lt;/a&gt; or, sometimes, is indicated by the nature of the variable.  (For instance, it might be a theoretical minimum value attainable by $Y$.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively, suppose $\beta_0$ is positive and sizable (within the context of the data) but $\sqrt{\beta_0}$ is inconsequentially small.  Then the original fit can be re-expressed as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Y = (\theta_1 + X_1) (\theta_2 + X_2) + \epsilon$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\theta_1 \theta_2 = \beta_0$ and both $\theta_1$ and $\theta_2$ are small.  Here, the missing cross terms $\theta_1 X_2$ and $\theta_2 X_1$ are presumed small enough to be subsumed within the error term $\epsilon$.  Again, assuming a multiplicative error and taking logarithms gives a model &lt;em&gt;with only direct effects and no interaction.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This analysis shows how it is possible--even likely in some applications--to have a model in which the only effects appear to be interactions.  This arises when the variables (independent, dependent, or both) are presented to you in an unsuitable form and their logarithms are a more effective target for modeling.  The distributions of the variables and of the initial residuals provide the clues needed to determine whether this may be the case: skewed distributions of the variables and heteroscedasticity of the residuals (specifically, having variances roughly proportional to the predicted values) are the indicators.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-12-14T15:21:58.523" CommentCount="3" />
  <row Id="5482" PostTypeId="2" ParentId="5347" CreationDate="2010-12-14T16:46:07.687" Score="3" ViewCount="" Body="&lt;p&gt;(Because this is approach is independent of the other solutions posted, including one that I have posted, I'm offering it as a separate response).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;You can compute the exact distribution in seconds (or less) provided the sum of the p's is small.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We have already seen suggestions that the distribution might approximately be Gaussian (under some scenarios) or Poisson (under other scenarios).  Either way, we know its mean $\mu$ is the sum of the $p_i$ and its variance $\sigma^2$ is the sum of $p_i(1-p_i)$.  Therefore the distribution will be concentrated within a few standard deviations of its mean, say $z$ SDs with $z$ between 4 and 6 or thereabouts.  Therefore we need only compute the probability that the sum $X$ equals (an integer) $k$ for $k = \mu - z \sigma$ through $k = \mu + z \sigma$.  When most of the $p_i$ are small, $\sigma^2$ is approximately equal to (but slightly less than) $\mu$, so to be conservative we can do the computation for $k$ in the interval $[\mu - z \sqrt{\mu}, \mu + z \sqrt{\mu}]$.  For example, when the sum of the $p_i$ equals $9$ and choosing $z = 6$ in order to cover the tails well, we would need the computation to cover $k$ in $[9 - 6 \sqrt{9}, 9 + 6 \sqrt{9}]$ = $[0, 27]$, which is just 28 values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The distribution is computed recursively&lt;/strong&gt;.  Let $f_i$ be the distribution of the sum of the first $i$ of these Bernoulli variables.  For any $j$ from $0$ through $i+1$, the sum of the first $i+1$ variables can equal $j$ in two mutually exclusive ways: the sum of the first $i$ variables equals $j$ and the $i+1^\text{st}$ is $0$ or else the sum of the first $i$ variables equals $j-1$ and the $i+1^\text{st}$ is $1$.  Therefore&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$f_{i+1}(j) = f_i(j)(1 - p_{i+1}) + f_i(j-1) p_{i+1}.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;We only need to carry out this computation for integral&lt;/strong&gt; $j$ &lt;strong&gt;in the interval from&lt;/strong&gt; $\max(0, \mu - z \sqrt{\mu})$ &lt;strong&gt;to&lt;/strong&gt; $\mu + z \sqrt{\mu}.$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When most of the $p_i$ are tiny (but the $1 - p_i$ are still distinguishable from $1$ with reasonable precision), this approach is not plagued with the huge accumulation of floating point roundoff errors used in the solution I previously posted.  Therefore, extended-precision computation is not required.  For example, a double-precision calculation for an array of $2^{16}$ probabilities $p_i = 1/(i+1)$ ($\mu = 10.6676$, requiring calculations for probabilities of sums between $0$ and $31$) took 0.1 seconds with Mathematica 8 and 1-2 seconds with Excel 2002 (both obtained the same answers).  Repeating it with quadruple precision (in Mathematica) took about 2 seconds but did not change any answer by more than $3 \times 10^{-15}$.  Terminating the distribution at $z = 6$ SDs into the upper tail lost only $3.6 \times 10^{-8}$ of the total probability.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another calculation for an array of 40,000 double precision random values between 0 and 0.001 ($\mu = 19.9093$) took 0.08 seconds with Mathematica.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;This algorithm is parallelizable.&lt;/strong&gt; Just break the set of $p_i$ into disjoint subsets of approximately equal size, one per processor.  Compute the distribution for each subset, then convolve the results (using FFT if you like, although this speedup is probably unnecessary) to obtain the full answer.  This makes it practical to use even when $\mu$ gets large, when you need to look far out into the tails ($z$ large), and/or $n$ is large.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The timing for an array of $n$ variables with $m$ processors scales as $O(n(\mu + z \sqrt{\mu})/m)$.  Mathematica's speed is on the order of a million per second.  For example, with $m = 1$ processor, $n = 20000$ variates, a total probability of $\mu = 100$, and going out to $z = 6$ standard deviations into the upper tail, $n(\mu + z \sqrt{\mu})/m = 3.2$ million: figure a couple seconds of computing time.  If you compile this you might speed up the performance two orders of magnitude.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Incidentally, in these test cases, graphs of the distribution clearly showed some positive skewness: they aren't normal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the record, here is a Mathematica solution:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pb[p_, z_] := Module[&#xA;  {\[Mu] = Total[p]},&#xA;  Fold[#1 - #2 Differences[Prepend[#1, 0]] &amp;amp;, &#xA;   Prepend[ConstantArray[0, Ceiling[\[Mu] + Sqrt[\[Mu]] z]], 1], p]&#xA;  ]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(&lt;strong&gt;NB&lt;/strong&gt; The color coding applied by this site is meaningless for Mathematica code.  In particular, the gray stuff is &lt;em&gt;not&lt;/em&gt; comments: it's where all the work is done!)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An example of its use is&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pb[RandomReal[{0, 0.001}, 40000], 8]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-01-07T05:42:04.397" LastActivityDate="2011-01-07T05:42:04.397" CommentCount="1" />
  <row Id="5483" PostTypeId="1" AcceptedAnswerId="5485" CreationDate="2010-12-14T17:33:16.430" Score="3" ViewCount="109" Body="&lt;p&gt;Lets say for example a class of students and their grades are the data set.&#xA;Lets say there are around 35 students.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is what you know:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Your mark&#xA;The class average&#xA;The median mark&#xA;The standard deviation&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Are there any other conclusions about this data that can be made given this information?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="2380" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-12-15T04:35:56.323" LastActivityDate="2010-12-15T04:35:56.323" Title="What else can be deduced from the following class grade summary information?" Tags="&lt;r&gt;&lt;distributions&gt;&lt;dataset&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="5484" PostTypeId="2" ParentId="5457" CreationDate="2010-12-14T17:34:59.467" Score="1" ViewCount="" Body="&lt;p&gt;A statistician tells you what conclusions can be reached from a data set and, maybe even more important, what conclusions &lt;em&gt;cannot&lt;/em&gt; be reached.&lt;/p&gt;&#xA;" OwnerUserId="666" LastActivityDate="2010-12-14T17:34:59.467" CommunityOwnedDate="2010-12-14T20:00:01.800" />
  <row Id="5485" PostTypeId="2" ParentId="5483" CreationDate="2010-12-14T17:40:38.690" Score="4" ViewCount="" Body="&lt;p&gt;You can probably also compute upper bounds on the number of students who did better or worse than you did. The population version would be via &lt;a href=&quot;http://en.wikipedia.org/wiki/Chebyshev%27s_inequality&quot; rel=&quot;nofollow&quot;&gt;Chebyshev's inequality&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if $X_{me}$ is 'my' score, $s^2$ is the sample variance, $\bar{X}$ is the sample mean, and $X_i$ are the class scores, including 'mine', we have&#xA;$$s^2 = \frac{1}{n-1}\sum_{i} (X_i - \bar{X})^2$$&#xD;&#xA;Now partition the sum into parts which are more extreme than my score and those which are less extreme:&#xA;$$(n-1) s^2 = \sum_{i \in I} (X_i - \bar{X})^2 + \sum_{i \not\in I}(X_i - \bar{X})^2$$,&#xA;where $I$ is the set of indices $i$ such that $|X_i - \bar{X}| \ge |X_{me} - \bar{X}|$. Now bound the summands in the right sum by zero from below, and bound those in the left sum by the condition defining $I$:&#xA;$$(n-1) s^2 \ge \sum_{i \in I} (X_{me} - \bar{X})^2 + \sum_{i \not\in I}0 = |I| (X_{me} - \bar{X})^2$$&#xD;&#xA;And thus you have&#xA;$$|I| \le (n-1)s^2 / (X_{me} - \bar{X})^2$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now $I$ includes all those in the class that did better than I did, if I beat the mean, and thus you have an upper bound on that number in that case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using the sample median and a sample version of &lt;a href=&quot;http://en.wikipedia.org/wiki/Gauss%27s_inequality&quot; rel=&quot;nofollow&quot;&gt;Gauss' Inequality&lt;/a&gt;, you may be able to prove a tighter bound, but with some more assumptions on the distribution, and the proof is not as clear. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(&lt;em&gt;n.b.&lt;/em&gt; This is, I believe, essentially how Chebyshev's inequality is proved.)&lt;/p&gt;&#xA;" OwnerUserId="795" LastEditorUserId="795" LastEditorDisplayName="" LastEditDate="2010-12-14T22:52:58.770" LastActivityDate="2010-12-14T22:52:58.770" CommentCount="4" />
  <row Id="5486" PostTypeId="2" ParentId="5453" CreationDate="2010-12-14T17:44:17.730" Score="0" ViewCount="" Body="&lt;p&gt;whuber, &#xA;a derivation of the product of two Gaussian pdfs is from &lt;a href=&quot;https://people.ok.ubc.ca/jbobowsk/phys327/Gaussian%20Convolution.pdf&quot; rel=&quot;nofollow&quot;&gt;https://people.ok.ubc.ca/jbobowsk/phys327/Gaussian%20Convolution.pdf&lt;/a&gt;. The formula for the variance I gave, 1 / (1 / (V+0.5) + 1/V) seems to be accurate. Another way of writing this same expression is V*(V+0.5)/(V + (V + 0.5)) from which my equation for T/V follows directly - showing that T &amp;lt; V. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Perhaps you were citing the variance for the convolution of two Gaussian pdfs? The sum of the variances is correct for the convolution. I believe Bayes specifies the product, not the convolution, however, or do I have that wrong?&lt;/p&gt;&#xA;" OwnerUserId="2379" LastEditorUserId="2379" LastEditorDisplayName="" LastEditDate="2010-12-14T17:51:17.260" LastActivityDate="2010-12-14T17:51:17.260" CommentCount="3" />
  <row Id="5487" PostTypeId="1" CreationDate="2010-12-14T18:30:49.127" Score="3" ViewCount="406" Body="&lt;p&gt;Hi to everyone.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to calculate the sample size required for an observational study in which incidence of disease is 19-29%. Population affected is 600,000 people.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The study has two samples similar on baseline characteristics treated &#xA; with two different drugs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Statistical analysis will be chi-square and Fisher's test.&#xA;I have to demonstrate non-inferiority of one drugs with respect of the other one.&#xA;I don't have any other info.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone help me to calculate the sample sizes required in order to obtain power of 0.80&#xA;with an alpha 0.05?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks a lot in advance&lt;/p&gt;&#xA;" OwnerUserId="2381" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-12-15T04:32:31.740" LastActivityDate="2010-12-15T04:32:31.740" Title="Sample size calculation for study aiming to demonstrate non-inferiority of a drug" Tags="&lt;sample-size&gt;&lt;epidemiology&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="5488" PostTypeId="2" ParentId="5453" CreationDate="2010-12-14T18:56:12.360" Score="0" ViewCount="" Body="&lt;p&gt;mpiktas,&#xA;actually, if you set r = c + epsilon with c a constant, you get the same problem. In fact, that is a simpler statement of this issue. In that case p(r|v) will have a variance exactly half of the variance of epsilon.&lt;/p&gt;&#xA;" OwnerUserId="2379" LastActivityDate="2010-12-14T18:56:12.360" />
  <row Id="5489" PostTypeId="2" ParentId="5453" CreationDate="2010-12-14T20:04:37.797" Score="1" ViewCount="" Body="&lt;p&gt;As per whuber's suggestion, I am posting the summary of mistakes in a separate answer.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;The distribution of $t$ is not stated. If $t$ is not a random variable, then the mean of $r$ is $\sin t$ and unconditional variance hence is $V$, not $\frac{1}{2}+V$. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If we assume that $t$ is uniformly distributed in interval $[0,2\pi]$ and independent from $\varepsilon$ as per shabbychef suggestion, then unconditional variance is $\frac{1}{2}+V$. But then distribution of $r$ is certainly not normal. So the product of $p(v|r)$ and $p(r)$ will not be Gaussian, and following variance calculations are incorrect.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2010-12-14T20:04:37.797" />
  <row Id="5490" PostTypeId="1" CreationDate="2010-12-14T20:08:27.367" Score="2" ViewCount="115" Body="&lt;p&gt;Assume we have a sensor field with dimension M*M. In order to apply any data compression technique, first I want to know what is the compression limit or minimum entropy of the entire sensor field. How could I compute the minimum entropy or compression limit for the sensor field? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;or &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Actually I want to have the theoretical compression limit. Let's put the problem for an image. I want to know whether there are any mathematical methods to calculate the theoretical compression limit. Please let me know or suggest any readings to formulate the problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="2384" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-14T21:36:25.123" LastActivityDate="2010-12-14T21:36:25.123" Title="How to compute theoretical compression limit?" Tags="&lt;entropy&gt;&lt;compression&gt;" AnswerCount="1" />
  <row Id="5491" PostTypeId="2" ParentId="4991" CreationDate="2010-12-14T20:29:05.030" Score="2" ViewCount="" Body="&lt;p&gt;If you have time varying parameters and want to do things sequentially (filtering), then SMC makes the most sense. MCMC is better when you want to condition on all of the data, or you have unknown static parameters that you want to estimate. Particle filters have issues with static parameters (degeneracy). &lt;/p&gt;&#xA;" OwnerUserId="643" LastActivityDate="2010-12-14T20:29:05.030" CommentCount="1" />
  <row Id="5492" PostTypeId="2" ParentId="5490" CreationDate="2010-12-14T20:33:56.257" Score="2" ViewCount="" Body="&lt;p&gt;The &quot;entropy&quot; is defined only within the &lt;a href=&quot;http://en.wikipedia.org/wiki/Entropy_%28information_theory%29#Data_compression&quot; rel=&quot;nofollow&quot;&gt;context of a probabilistic model&lt;/a&gt; for the data.  If you characterize the image as a set of $M^2$ distinct &quot;characters&quot; and assume the frequencies of those characters adequately reflect their probabilities, then you need only apply the formula&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Entropy = Sum (over all characters $c$) of [-log(probability of $c$) * probability of $c$].&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A standard (but by no means the only) estimate of the probability of a character in a set of $N = M^2$ characters is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Estimated probability of $c$ = (Number of occurrences of $c$) / $N$.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-12-14T20:33:56.257" CommentCount="5" />
  <row Id="5493" PostTypeId="2" ParentId="5487" CreationDate="2010-12-14T20:42:14.083" Score="4" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://www.psycho.uni-duesseldorf.de/aap/projects/gpower/&quot; rel=&quot;nofollow&quot;&gt;G*Power&lt;/a&gt; is a commonly-recommended program for sample size calculations.  I've only dabbled with it a couple of times in the past, but it's more than capable of handling the situation you describe.&lt;/p&gt;&#xA;" OwnerUserId="71" LastActivityDate="2010-12-14T20:42:14.083" CommentCount="2" />
  <row Id="5494" PostTypeId="2" ParentId="5115" CreationDate="2010-12-14T21:11:31.903" Score="1" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://www.cs.uic.edu/~wilkinson/&quot; rel=&quot;nofollow&quot;&gt;Leland Wilkinson&lt;/a&gt; for his contribution to statistical graphics.&lt;/p&gt;&#xA;" OwnerUserId="609" LastActivityDate="2010-12-14T21:11:31.903" CommunityOwnedDate="2010-12-14T21:11:31.903" CommentCount="1" />
  <row Id="5495" PostTypeId="2" ParentId="5434" CreationDate="2010-12-14T22:23:25.407" Score="3" ViewCount="" Body="&lt;p&gt;If you can't get satisfaction with R you can fit this model and more complicated &#xA;ones with AD Model Builder which is free software available at &lt;a href=&quot;http://admb-project.org&quot; rel=&quot;nofollow&quot;&gt;http://admb-project.org&lt;/a&gt;. ADMB permits you to model the over dispersion in a variety of ways,&#xA;rather than being confined to the GLM paradigm. I can advise you if you are interested.&lt;/p&gt;&#xA;" OwnerUserId="1585" LastActivityDate="2010-12-14T22:23:25.407" CommentCount="2" />
  <row Id="5496" PostTypeId="2" ParentId="5457" CreationDate="2010-12-14T23:13:34.167" Score="2" ViewCount="" Body="&lt;p&gt;A statistician is a numerical detective, uncovering the stories hidden in a mass of data.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-12-14T23:13:34.167" CommunityOwnedDate="2010-12-14T23:13:34.167" />
  <row Id="5497" PostTypeId="2" ParentId="5457" CreationDate="2010-12-14T23:17:09.983" Score="1" ViewCount="" Body="&lt;p&gt;The TV show Numb3rs is useful as many people have seen it. I tell them that I'm like the guys on Numb3rs except I deal with solving business problems rather than crimes. (Substitute &quot;business problems&quot; for whatever field you work in.) That usually gets the response &quot;Wow, cool!&quot; which is better than what I used to get.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-12-14T23:17:09.983" CommunityOwnedDate="2010-12-14T23:17:09.983" CommentCount="1" />
  <row Id="5498" PostTypeId="2" ParentId="5479" CreationDate="2010-12-14T23:41:45.517" Score="4" ViewCount="" Body="&lt;p&gt;So you have monthly data with trend and seasonality and you want to both analyse the trend/seasonal components and produce forecasts. These are two separate tasks. While you can do both with &lt;code&gt;dlm&lt;/code&gt;, there are simpler approaches if you separate the tasks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For studying the trend and seasonality, I suggest using STL via the &lt;code&gt;stl()&lt;/code&gt; function in R. It is robust and has nice graphics that are easy to explain to non-statisticians.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For forecasting, I would use a seasonal ARIMA model. To start with, ignore the Big Bash Gala event, and try fitting a seasonal ARIMA model using auto.arima() from the &lt;code&gt;forecast&lt;/code&gt; package. The automatic seasonal differencing selection is not very good, so I would use&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;fit1 &amp;lt;- auto.arima(x,D=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where &lt;code&gt;x&lt;/code&gt; is your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, to add in the BBG effect, set up a dummy regression variable (&lt;code&gt;z&lt;/code&gt;) taking value 1 when BBG occurs in the month and 0 otherwise. Add that in to your model using &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;fit2 &amp;lt;- auto.arima(x,xreg=z,D=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Produce forecasts from each model using the &lt;code&gt;forecast()&lt;/code&gt; function. For &lt;code&gt;fit2&lt;/code&gt;, you will need future values of &lt;code&gt;z&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even if you want to stick with &lt;code&gt;dlm&lt;/code&gt;, the above approaches will provide useful comparative benchmarks.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-12-14T23:41:45.517" CommentCount="2" />
  <row Id="5499" PostTypeId="2" ParentId="5483" CreationDate="2010-12-15T03:00:22.257" Score="4" ViewCount="" Body="&lt;p&gt;Since the data set is grades, you probably also know the minimum and maximum possible scores, e.g. 0 and 100.  Given the median and first two moments, you can fit a variety of models to the sample statistics, say a scaled beta distribution.  This would give you tighter estimates than Chebyshev's inequality, although you'd be making some serious regularity and smoothness assumptions.  (The nifty thing with a beta distribution is that the mean and variance completely define it, so you could use the median as a validation statistic.)  Sounds like fun!&lt;/p&gt;&#xA;" OwnerUserId="1920" LastActivityDate="2010-12-15T03:00:22.257" />
  <row Id="5500" PostTypeId="2" ParentId="5461" CreationDate="2010-12-15T03:04:59.567" Score="2" ViewCount="" Body="&lt;p&gt;From what I can tell, your problems seem to be:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) smooth the time series data to remove correlated fluctuations. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Identify which of the inputs differs, using the smoothed data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You're seem to be worried about not being able to solve (2) once you solve (1). But let's solve 1 first and then worry about 2, right? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's one idea. You say you're sampling in a round robin fashion and you have 16 inputs. So maybe treat each 16 draws as one &quot;round&quot;, sum up all of the values in each round, and divide each value in the 16-draw round by that sum to normalize. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems that this would work if your time series data is correlated on a longer time scale than 16 data points. If the data is correlated on a much longer time scale you could even normalize in larger chunks, like 160 or 1,600 data points, to maximize noise reduction (and comp. efficiency). &lt;/p&gt;&#xA;" OwnerUserId="2073" LastActivityDate="2010-12-15T03:04:59.567" CommentCount="1" />
  <row Id="5501" PostTypeId="2" ParentId="5462" CreationDate="2010-12-15T03:22:35.440" Score="2" ViewCount="" Body="&lt;p&gt;Your best bet is mechanical turk, &lt;a href=&quot;https://www.mturk.com/mturk/welcome&quot; rel=&quot;nofollow&quot;&gt;https://www.mturk.com/mturk/welcome&lt;/a&gt;. It will cost you some money, but not much. If your questions are short and can be done in ~ 1 min you can easily charge 10 cents or so per answer, so if you want 50 responses it will cost only $5. You can get the data formatted in .xml, which you can import into R or even (shudder) Excel. The people you sample will be much more varied than you'd get in a high school class. And you can ask for demographic info (age, gender, nationality, etc), as questions in your survey. Also making the survey is easy, just adjust amazon's templates. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;See &lt;a href=&quot;http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1601785&quot; rel=&quot;nofollow&quot;&gt;http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1601785&lt;/a&gt; for more issues with the biases you mention, the WEIRD study. Also see why people at mech turk participate: &lt;a href=&quot;http://behind-the-enemy-lines.blogspot.com/2008/09/why-people-participate-on-mechanical.html&quot; rel=&quot;nofollow&quot;&gt;http://behind-the-enemy-lines.blogspot.com/2008/09/why-people-participate-on-mechanical.html&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="2073" LastActivityDate="2010-12-15T03:22:35.440" />
  <row Id="5502" PostTypeId="1" AcceptedAnswerId="5515" CreationDate="2010-12-15T04:04:07.567" Score="5" ViewCount="317" Body="&lt;p&gt;Is there a principled way to estimate factor scores when you have ordinal, discrete variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have $n$ ordinal, discrete, variables. If I make the assumption that underlying each response is a continuous, normally distributed variable, then I can calculate an $n\times n$ polychoric correlation matrix. I can then run a factor analysis on this matrix and get factor loadings for each variable. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I combine the factor loadings and the variables to estimate the factor scores. The typical ways to estimate scores would appear to require that I treat the ordinal data as interval.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suppose I might need to dig deeper into the guts of polychoric correlation to figure out a link function.&lt;/p&gt;&#xA;" OwnerUserId="82" LastEditorUserId="82" LastEditorDisplayName="" LastEditDate="2010-12-16T05:25:16.043" LastActivityDate="2010-12-16T05:25:16.043" Title="Factor scores from discrete, ordinal responses" Tags="&lt;factor-analysis&gt;&lt;ordinal&gt;" AnswerCount="3" FavoriteCount="1" />
  <row Id="5503" PostTypeId="1" AcceptedAnswerId="5522" CreationDate="2010-12-15T04:40:11.687" Score="3" ViewCount="172" Body="&lt;p&gt;Hello All:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think, I read this quote some where:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;For every field &quot;x&quot; there exists a field &quot;computational x&quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Has anyone else read this or remembers reading anything close to this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I remember correctly, it was by Dr. Jan de Leeuw.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone please tell if my memory fails me here? (I could not find any link after a lot of googling)&lt;/p&gt;&#xA;" OwnerUserId="1307" LastActivityDate="2010-12-15T19:22:15.603" Title="Is there a quote like this from some statistician?" Tags="&lt;reproducible-research&gt;" AnswerCount="2" CommentCount="4" />
  <row Id="5504" PostTypeId="1" AcceptedAnswerId="5516" CreationDate="2010-12-15T06:00:52.117" Score="2" ViewCount="196" Body="&lt;p&gt;The annual returns on stocks and treasury bonds over the next 12 months are uncertain.  Suppose that these returns can be described by normal distributions with stocks having a mean of 15% and a standard deviation of 20%, and bonds having a mean of 6% and a standard deviation of 9%.  Which asset is more likely to have a negative return?  &lt;/p&gt;&#xA;" OwnerUserId="2385" LastActivityDate="2010-12-15T17:16:05.057" Title="Normal distribution probability" Tags="&lt;homework&gt;" AnswerCount="3" CommentCount="1" FavoriteCount="1" />
  <row Id="5505" PostTypeId="2" ParentId="5479" CreationDate="2010-12-15T06:14:58.613" Score="0" ViewCount="" Body="&lt;p&gt;Difficult to diagnose without looking at the results, but here is a wild guess: if in the fit the variance associated to the level component is large, that component will &quot;follow&quot; your data. The filtered estimates will nearly coincide with observations, and the forecasts will appear to lag them --which, as I understand, is what you observe--.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have no explanation for what you mention in your third paragraph.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would think that the behaviour you note in the fourth paragraph is not so abnormal: you have monthly data and the Kalman filter has to settle down. For a transient period (at least a year) you may look at a wandering state. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My understanding is that the local linear trend (dlmModPoly(order=2,...)) needs&#xA;not give a smoother fit than the local level model (dlmModPoly(order=1,...)). It all depends on the values of the fitted variances of the level and slope components of the state.&lt;/p&gt;&#xA;" OwnerUserId="892" LastActivityDate="2010-12-15T06:14:58.613" CommentCount="3" />
  <row Id="5506" PostTypeId="2" ParentId="5504" CreationDate="2010-12-15T06:37:51.053" Score="5" ViewCount="" Body="&lt;p&gt;Here is the R code to quickly solve this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   &amp;gt; pnorm(0, mean=15, sd=20)&#xA;        [1] 0.2266274&#xA;&#xA;   &amp;gt; pnorm(0, mean=6, sd=9) &#xA;        [1] 0.2524925&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So bonds will be more likely to have a negative return&lt;/p&gt;&#xA;" OwnerUserId="2144" LastActivityDate="2010-12-15T06:37:51.053" CommentCount="1" />
  <row Id="5507" PostTypeId="2" ParentId="5504" CreationDate="2010-12-15T07:24:41.860" Score="3" ViewCount="" Body="&lt;p&gt;Stock: P( (X-15)/20 &amp;lt; (0-15)/20 ) = P(Z&amp;lt;-3/4) = .2266,&#xA;Bond: P( (Y-6)/9 &amp;lt; (0-6)/9 ) = P(Z&amp;lt;-2/3) = .2525&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More likely the bond.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that the price is log-normally distributed.  &lt;/p&gt;&#xA;" OwnerUserId="2387" LastActivityDate="2010-12-15T07:24:41.860" CommentCount="3" />
  <row Id="5508" PostTypeId="1" AcceptedAnswerId="5540" CreationDate="2010-12-15T07:57:00.543" Score="2" ViewCount="147" Body="&lt;p&gt;Hi there,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm using multinomial probit to estimate some parameters, and I keep seeing references to the fact that MNP was considered computationally &quot;intractible&quot; relative to binomial probit up until the early 21st century. The question is: why? I get that adding variables makes things take longer (I've got a background in CS), but for the life of me I can't see why estimation should be any worse than, say, $O(n^3)$ in the &quot;nomial-ness&quot; of the model. That is, when you add a new choice, you can update your simulations based on transformations of the joint error terms, and from there it's binomial probit with a couple indicators thrown in. Is there something deeper going on in the background that I'm not taking into consideration?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks,&#xA;Kyle&lt;/p&gt;&#xA;" OwnerUserId="53" LastActivityDate="2010-12-15T23:46:54.900" Title="Computational considerations of multinomial probit versus binomial probit" Tags="&lt;maximum-likelihood&gt;&lt;multinomial&gt;" AnswerCount="2" CommentCount="3" />
  <row Id="5509" PostTypeId="2" ParentId="5503" CreationDate="2010-12-15T08:05:29.210" Score="4" ViewCount="" Body="&lt;p&gt;Maybe you are after this talk?&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;a href=&quot;http://www.biostat.jhsph.edu/~rpeng/ENAR2009/lecture-slides.pdf&quot; rel=&quot;nofollow&quot;&gt;Tutorial: Methods for Reproducible&#xA;  Research&lt;/a&gt;, by Roger D. Peng (slide&#xA;  3)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Also, papers on Reproducible research written by de Leeuw that I am aware of are &lt;a href=&quot;http://preprints.stat.ucla.edu/301/301.pdf&quot; rel=&quot;nofollow&quot;&gt;Reproducible Research: the Bottom Line&lt;/a&gt;, and &lt;a href=&quot;http://preprints.stat.ucla.edu/570/statsoft.pdf&quot; rel=&quot;nofollow&quot;&gt;Statistical Software -- Overview&lt;/a&gt;. But a quick check didn't reveal any citation like the one you show.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-12-15T17:44:09.527" LastActivityDate="2010-12-15T17:44:09.527" CommentCount="6" />
  <row Id="5510" PostTypeId="2" ParentId="3313" CreationDate="2010-12-15T08:30:42.657" Score="0" ViewCount="" Body="&lt;p&gt;Hi Adal,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;did you convert the aforementioned library to R? I need to convert the same library into R and wanted to ask if you might share your results?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="2389" LastActivityDate="2010-12-15T08:30:42.657" />
  <row Id="5511" PostTypeId="2" ParentId="5115" CreationDate="2010-12-15T08:53:41.307" Score="4" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/W._Edwards_Deming&quot; rel=&quot;nofollow&quot;&gt;W. Edwards Deming&lt;/a&gt; for promoting statistical process control&lt;/p&gt;&#xA;" OwnerUserId="74" LastActivityDate="2010-12-15T08:53:41.307" CommunityOwnedDate="2010-12-15T08:53:41.307" />
  <row Id="5512" PostTypeId="2" ParentId="5114" CreationDate="2010-12-15T12:27:20.673" Score="0" ViewCount="" Body="&lt;p&gt;You could also try the triangular distribution.  To fit this, you basically specify a lower bound (this would be X=2), an upper bound (this would be X=8), and a &quot;most likely&quot; value.  The wikepedia page &lt;a href=&quot;http://en.wikipedia.org/wiki/Triangular_distribution&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Triangular_distribution&lt;/a&gt; has more info on this distribution.  If there is not much faith in the &quot;most likely&quot; value (as it appears to be, prior to observing any data), it may be a good idea to place a non-informative prior distribution on it, and then use the two data points to estimate this value.  One good one is the jeffrey's prior, which for this problem would be p(c) = 1/(pi*sqrt((c-2)*(c-8))), where &quot;c&quot; is the &quot;most likely value&quot; (consistent with the wikipedia notation).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given this prior, you can work out the posterior distribution of c analytically, or via simulation.  The analytic form of the likelihood is not particularly nice, so simulation seems to be more attractive.  This example is particularly well suited to rejection sampling (see wiki page for a general description of rejection sampling), because the maximised likelihood is 1/3^n regardless of the value of c, which provides the &quot;upper bound&quot;.  So you generate a &quot;candidate&quot; from the jeffrey's prior (call it c_i), and then evaluate the likelihood at this candidate L(x1,..,xn|c_i), and divide by the maximised likelihood, to give (3^n)*L(x1,..,xn|c_i).  You then generate a U(0,1) random variable, and if u is less than (3^n)*L(x1,..,xn|c_i), then accept c_i as a posterior sampled value, otherwise throw away c_i and start again.  Repeat this process until you have enough accepted samples (100, 500, 1,000, or more depending on how accurate you want).  Then just take the sample average of whatever function of c you are interested in (the likelihood of a new observation is an obvious candidate for your application).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An alternative to accept-reject is to use the value of the likelihood as a weight (and don't generate the u), and then proceed with taking weighted averages using all candidates, rather than un-weighted averages with the accepted candidates&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2010-12-15T12:27:20.673" />
  <row Id="5513" PostTypeId="2" ParentId="1906" CreationDate="2010-12-15T12:47:51.133" Score="0" ViewCount="" Body="&lt;p&gt;Predictive Analytics World: pawcon.com&lt;/p&gt;&#xA;" OwnerUserId="2393" LastActivityDate="2010-12-15T12:47:51.133" CommunityOwnedDate="2010-12-15T12:47:51.133" />
  <row Id="5514" PostTypeId="1" CreationDate="2010-12-15T14:18:11.087" Score="2" ViewCount="95" Body="&lt;p&gt;Could anyone kindly provide an explanation (mathematically or non-mathematically) about the non-existence of the intercept term in &lt;a href=&quot;http://www.ats.ucla.edu/stat/sas/library/logistic.pdf&quot; rel=&quot;nofollow&quot;&gt;conditional logistic regression&lt;/a&gt;? Is the interpretation of the coefficients similar to that of (unconditional) &lt;a href=&quot;http://en.wikipedia.org/wiki/Logistic_regression&quot; rel=&quot;nofollow&quot;&gt;logistic regression&lt;/a&gt;? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you for your help. &lt;/p&gt;&#xA;" OwnerUserId="2264" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-12-15T17:28:37.433" LastActivityDate="2010-12-17T13:15:20.630" Title="Queries on conditional logistic regression" Tags="&lt;logistic-regression&gt;&lt;survival&gt;&lt;epidemiology&gt;" AnswerCount="2" />
  <row Id="5515" PostTypeId="2" ParentId="5502" CreationDate="2010-12-15T14:54:08.093" Score="4" ViewCount="" Body="&lt;p&gt;It's commonplace to extract factor scores from ordinal-variable indicators. Researchers using likert measures do it all the time. Because factor scores are based on covariance, it's usually not that big a deal that the &quot;intervals&quot; might not be uniform within and across items, particularly if the items are comparable &amp;amp; use reasonably-compact scales (e.g., 5 or 7 pt &quot;agree/disagree&quot; likert items): all the subjects are responding to the same items, and if the items are indeed valid measures of some latent variable, the responses should should display a uniform covariance pattern. See Gorsuch, R. L. (1983). Factor Analysis. Hillsdale, NJ: Lawrence Erlbaum. 2nd. ed., pp. 119-20.  But if bothers you to assume the responses for you ordinal variables are linear -- or even more important, if you want factor scores that aren't linear but reflect recurring nonlinear associations among categorical items (as you would be doing if your variables were nominal or qualitative)-- you should use a nonlinear scaling alternative to conventional factor analysis, such as latent class analysis or item response theory. (There is of course a family resemblance between this query and your query on use of ordinal predictors in logit regression models; maybe I can once again inspire chi or someone else who knows more than I to treat us to an even more fine-grained account of why you needn't worry-- or maybe why you should.) &lt;/p&gt;&#xA;" OwnerUserId="2347" LastActivityDate="2010-12-15T14:54:08.093" />
  <row Id="5516" PostTypeId="2" ParentId="5504" CreationDate="2010-12-15T15:14:55.810" Score="7" ViewCount="" Body="&lt;p&gt;The objective of this exercise is to help you develop your ability to reason with probability distributions.  You would like to get to the point where your reflex is to think through such problems this way:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;A negative return is anything less than 0%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;For &lt;strong&gt;stocks&lt;/strong&gt;, 0% is three-quarters of a standard deviation (&lt;em&gt;i.e.&lt;/em&gt;, three quarters of 20%) less than the mean (15%).  Therefore we can think of the chance of a negative return as the area to the left of -3/4 under a standard bell curve.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;For &lt;strong&gt;bonds&lt;/strong&gt;, 0% is only two-thirds of a standard deviation less than its mean (0% - 6% = -2/3 of 9%).  The chance of a negative return is represented by the area to the left of -2/3 under the same bell curve.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Because negative two-thirds (bonds) is greater than negative three-quarters (stocks), negative returns occupy more of the left tail of the bond return distribution.  That makes a negative bond return more likely.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;The whole point is to translate information about means and standard deviations into areas under a distribution function.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Notice that the only calculations required in this case are simple ones; with textbook problems like this, you can do them in your head.  In &quot;real world&quot; problems you can usually still think a problem through by approximating the calculations.  This gives you the ability to think on your feet, which is one key to mastering any subject.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-12-15T17:16:05.057" LastActivityDate="2010-12-15T17:16:05.057" CommentCount="1" />
  <row Id="5517" PostTypeId="1" AcceptedAnswerId="9680" CreationDate="2010-12-15T16:18:49.643" Score="5" ViewCount="479" Body="&lt;p&gt;The library languageR provides a method (pvals.fnc) to do MCMC significance testing of the fixed effects in a mixed effect regression model fit using lmer.  However, pvals.fnc gives an error when the lmer model includes random slopes.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a way to do an MCMC hypothesis test of such models?&lt;br&gt;&#xA;If so, how?  (To be accepted an answer should have a worked example in R)&#xA;If not, is there a conceptual/computation reason why there is no way?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This question might be related to &lt;a href=&quot;http://stats.stackexchange.com/questions/152/is-there-a-standard-method-to-deal-with-label-switching-problem-in-mcmc-estimatio&quot;&gt;this one&lt;/a&gt; but I didn't understand the content there well enough to be certain.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit 1&lt;/strong&gt;: A proof of concept showing that pvals.fnc() still does 'something' with lme4 models, but that it doesn't do anything with random slope models.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(lme4)&#xA;library(languageR)&#xA;#the example from pvals.fnc&#xA;data(primingHeid) &#xA;# remove extreme outliers&#xA;primingHeid = primingHeid[primingHeid$RT &amp;lt; 7.1,]&#xA;# fit mixed-effects model&#xA;primingHeid.lmer = lmer(RT ~ RTtoPrime * ResponseToPrime + Condition + (1|Subject) + (1|Word), data = primingHeid)&#xA;mcmc = pvals.fnc(primingHeid.lmer, nsim=10000, withMCMC=TRUE)&#xA;#Subjects are in both conditions...&#xA;table(primingHeid$Subject,primingHeid$Condition)&#xA;#So I can fit a model that has a random slope of condition by participant&#xA;primingHeid.lmer.rs = lmer(RT ~ RTtoPrime * ResponseToPrime + Condition + (1+Condition|Subject) + (1|Word), data = primingHeid)&#xA;#However pvals.fnc fails here...&#xA;mcmc.rs = pvals.fnc(primingHeid.lmer.rs)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It says: Error in pvals.fnc(primingHeid.lmer.rs) : &#xA;  MCMC sampling is not yet implemented in lme4_0.999375&#xA;  for models with random correlation parameters&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Additional question:  Is pvals.fnc performing as expected for random intercept model?  Should the outputs be trusted?&lt;/p&gt;&#xA;" OwnerUserId="196" LastEditorUserId="196" LastEditorDisplayName="" LastEditDate="2010-12-18T19:38:33.103" LastActivityDate="2011-04-18T10:35:40.950" Title="How can one do an MCMC hypothesis test on a mixed effect regression model with random slopes?" Tags="&lt;r&gt;&lt;mixed-model&gt;&lt;statistical-significance&gt;&lt;monte-carlo&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="3" />
  <row Id="5518" PostTypeId="2" ParentId="1906" CreationDate="2010-12-15T17:28:47.470" Score="1" ViewCount="" Body="&lt;p&gt;SIAM's &lt;a href=&quot;http://www.siam.org/meetings/sdm11/&quot; rel=&quot;nofollow&quot;&gt;Data Mining Conference&lt;/a&gt;, SDM11.&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-12-15T17:28:47.470" CommunityOwnedDate="2010-12-15T17:28:47.470" />
  <row Id="5519" PostTypeId="2" ParentId="5514" CreationDate="2010-12-15T17:31:04.437" Score="3" ViewCount="" Body="&lt;p&gt;Conditional logistic regression compares cases to controls.  The coefficients multiply &lt;em&gt;differences&lt;/em&gt; in factor values between cases and controls.  The intercept terms cancel in the likelihood and therefore play no identifiable role in the model.  See the section on &quot;Conditional Logistic Regression&quot; in &lt;a href=&quot;http://www.ats.ucla.edu/stat/sas/library/logistic.pdf&quot; rel=&quot;nofollow&quot;&gt;Ying So's tutorial for SAS&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-12-15T17:31:04.437" />
  <row Id="5520" PostTypeId="1" AcceptedAnswerId="5568" CreationDate="2010-12-15T18:08:15.290" Score="4" ViewCount="157" Body="&lt;p&gt;I am wondering if there is any reasonably simple way of calculating the following problem:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Drawing, with replacement, $n$ balls from a bin of $N$ different colored balls, with a known probability of drawing each color of ball, what is the expected number of &quot;unique&quot; balls, &lt;em&gt;i.e.,&lt;/em&gt; balls with no other ball of the same color?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;e.g.&lt;/em&gt;&lt;br&gt;&#xA;$P(red) = 0.25$&lt;br&gt;&#xA;$P(blue) = 0.3$&lt;br&gt;&#xA;$P(green) = 0.2$&lt;br&gt;&#xA;$P(yellow) = 0.25$  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some example outcomes with 5 balls:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\{red, red, green, blue, yellow\}$ - 3 unique balls&lt;br&gt;&#xA;$\{red, red, green, green, blue\}$ - 1 unique ball&lt;br&gt;&#xA;$\{blue, blue, blue,  yellow, yellow\}$ - 0 unique balls  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or, with 3 balls:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\{red, green, blue\}$ - 3 unique&lt;br&gt;&#xA;$\{red, red, blue\}$ - 1 unique&lt;br&gt;&#xA;$\{red, red, red\}$ - 0 unique  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For 1 ball, it's trivially 1; for 2 balls, it's 1 - the probability of the outcomes where the two balls are the same color * 2 balls, after that it starts getting more complicated.&lt;/p&gt;&#xA;" OwnerUserId="2395" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-12-16T22:12:01.823" LastActivityDate="2010-12-21T15:01:39.427" Title="Expected number of uniques in a non-uniformly distributed population" Tags="&lt;expected-value&gt;&lt;multinomial&gt;" AnswerCount="3" CommentCount="1" FavoriteCount="1" />
  <row Id="5521" PostTypeId="2" ParentId="5502" CreationDate="2010-12-15T19:05:19.570" Score="3" ViewCount="" Body="&lt;p&gt;The 'principled' approach (that is to say the a priori defensible approach that may not empirically make much difference) is to use a &lt;em&gt;graded response model&lt;/em&gt;, a rather useful member of the IRT family often used for Likert type items.  The R package &lt;strong&gt;ltm&lt;/strong&gt; makes this very straightforward.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You're then assuming there is a ordinal logistic regression relationship between the unobserved trait and each of your indicators.  Choosing this model class allows you to take the ordinal nature of the indicators seriously and provides information about what part of the trait each item is most informative about.  Like factor analysis, it gives you a standard error for the score, although FA people seem to ignore these for some reason.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, choosing this model class limits your ability to do all the classic factor analysis stuff like rotating things until you like the look of them.  I think this is a plus, but reasonable people disagree.  If you're doing that sort of thing to find out how many 'scales' you have, you'll want to look at the Mokken procedures that try to identify scales, since the FA 'fit another dimension and rotate to simple structure' won't work.&lt;/p&gt;&#xA;" OwnerUserId="1739" LastActivityDate="2010-12-15T19:05:19.570" CommentCount="3" />
  <row Id="5522" PostTypeId="2" ParentId="5503" CreationDate="2010-12-15T19:22:15.603" Score="3" ViewCount="" Body="&lt;p&gt;Well here's one place de Leeuw says it: &lt;a href=&quot;http://preprints.stat.ucla.edu/491/useR.pdf&quot; rel=&quot;nofollow&quot;&gt;http://preprints.stat.ucla.edu/491/useR.pdf&lt;/a&gt;  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It might also be found in a more formal document, but nothing in my collection...&lt;/p&gt;&#xA;" OwnerUserId="1739" LastActivityDate="2010-12-15T19:22:15.603" CommentCount="7" />
  <row Id="5524" PostTypeId="2" ParentId="5465" CreationDate="2010-12-15T19:33:16.230" Score="2" ViewCount="" Body="&lt;p&gt;For an observational data context: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider this regression model applied to this substantive problem.  What, if anything, in it can be interpreted causally?  [Further probe] What would you need to learn to change your opinion? &lt;/p&gt;&#xA;" OwnerUserId="1739" LastActivityDate="2010-12-15T19:33:16.230" CommunityOwnedDate="2010-12-15T19:33:16.230" />
  <row Id="5525" PostTypeId="1" AcceptedAnswerId="5526" CreationDate="2010-12-15T19:41:44.733" Score="5" ViewCount="195" Body="&lt;p&gt;Is there a way to get the number of parameters of a linear model like that?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model &amp;lt;- lm(Y~X1+X2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I would like to get the number 3 somehow (intercept + X1 + X2). I looked for something like this in the structures that &lt;code&gt;lm&lt;/code&gt;, &lt;code&gt;summary(model)&lt;/code&gt; and &lt;code&gt;anova(model)&lt;/code&gt; return, but I didn't figure it out. In case I don't get an answer, I'll stick on&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dim(model.matrix(model))[2]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Thank you&lt;/p&gt;&#xA;" OwnerUserId="632" LastActivityDate="2010-12-16T23:16:31.660" Title="Get the number of parameters of a linear model" Tags="&lt;r&gt;&lt;regression&gt;" AnswerCount="4" />
  <row Id="5526" PostTypeId="2" ParentId="5525" CreationDate="2010-12-15T19:56:17.260" Score="8" ViewCount="" Body="&lt;p&gt;Try something like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; x &amp;lt;- replicate(2, rnorm(100))&#xA;&amp;gt; y &amp;lt;- 1.2*x[,1]+rnorm(100)&#xA;&amp;gt; summary(lm.fit &amp;lt;- lm(y~x))&#xA;&amp;gt; length(lm.fit$coefficients)&#xA;[1] 3&#xA;&amp;gt; # or&#xA;&amp;gt; length(coef(lm.fit))&#xA;[1] 3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can have a better idea of what an R object includes with&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; str(lm.fit)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-12-15T19:56:17.260" CommentCount="2" />
  <row Id="5527" PostTypeId="2" ParentId="5508" CreationDate="2010-12-15T20:02:45.727" Score="2" ViewCount="" Body="&lt;p&gt;The currently popular method of fitting multinomial probit models is &lt;a href=&quot;http://jblevins.org/notes/msl&quot; rel=&quot;nofollow&quot;&gt;maximum simulated likelihood&lt;/a&gt; using the Geweke–Hajivassiliou–Keane algorithm (&lt;a href=&quot;http://www.jstor.org/stable/1913710&quot; rel=&quot;nofollow&quot;&gt;Geweke 1989&lt;/a&gt;; &lt;a href=&quot;http://www.jstor.org/stable/2999576&quot; rel=&quot;nofollow&quot;&gt;Hajivassiliou and McFadden 1998&lt;/a&gt;; &lt;a href=&quot;http://www.jstor.org/stable/2109768&quot; rel=&quot;nofollow&quot;&gt;Keane and Wolpin 1994&lt;/a&gt;). So the algorithm dates from the late 1990s. If you've thought up a more efficient method I suggest you submit it to &lt;em&gt;Econometrica&lt;/em&gt;.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-12-15T20:02:45.727" />
  <row Id="5528" PostTypeId="2" ParentId="5465" CreationDate="2010-12-15T20:06:07.013" Score="2" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;How do you numericize something that&#xA;  is not numerical?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Example, &lt;a href=&quot;http://www.springerlink.com/content/g71368g57x013j48/&quot; rel=&quot;nofollow&quot;&gt;&quot;Automatic Feature Extraction for Classifying Audio Data&quot;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rationale: Can they figure out how to analyze something statistically that is not already in a big table?&lt;/p&gt;&#xA;" OwnerUserId="74" LastActivityDate="2010-12-15T20:06:07.013" CommunityOwnedDate="2010-12-15T20:06:07.013" />
  <row Id="5529" PostTypeId="2" ParentId="5465" CreationDate="2010-12-15T20:08:32.143" Score="2" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;How do you prevent over-fitting when&#xA;  you are creating a statistical model?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Good answer: cross-validation&lt;/p&gt;&#xA;" OwnerUserId="74" LastActivityDate="2010-12-15T20:08:32.143" CommunityOwnedDate="2010-12-15T20:08:32.143" />
  <row Id="5530" PostTypeId="2" ParentId="5465" CreationDate="2010-12-15T20:10:26.060" Score="3" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Here is a big data set. What is your&#xA;  plan for dealing with outliers? How&#xA;  about missing values? How about transformations?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Can they deal with real-world data?&lt;/p&gt;&#xA;" OwnerUserId="74" LastActivityDate="2010-12-15T20:10:26.060" CommunityOwnedDate="2010-12-15T20:10:26.060" />
  <row Id="5531" PostTypeId="2" ParentId="5465" CreationDate="2010-12-15T20:14:20.347" Score="1" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Here is a &lt;a href=&quot;http://en.wikipedia.org/wiki/Tinkertoy&quot; rel=&quot;nofollow&quot;&gt;TinkerToy&lt;/a&gt; set. Show me&#xA;  how Euclidean distance works in three&#xA;  dimensions. Now show me how multiple regression works.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Can they explain how statistics works in the physical world?&lt;/p&gt;&#xA;" OwnerUserId="74" LastActivityDate="2010-12-15T20:14:20.347" CommunityOwnedDate="2010-12-15T20:14:20.347" CommentCount="1" />
  <row Id="5532" PostTypeId="1" AcceptedAnswerId="5541" CreationDate="2010-12-15T20:21:02.263" Score="2" ViewCount="118" Body="&lt;p&gt;&lt;em&gt;Working off a fairly limited statistical understanding, so apologies in advance if this question is ludicrously basic.  This has to have happened to other people who have successfully solved this issue, but searching for what I think are relevant terms is getting me things like &quot;Karlin's Conjecture for Random Replacement Sampling Plans,&quot; and I'm starting to panic.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Attempting to contact 1000 people for a survey.  Plan was to get a list of the target population, draw a random sample of 1000, then contact those people.  Problem was that the original list contained people who weren't actually part of the population, so our random sample of 1000 only contained around 650 people we could actually contact.  For time &amp;amp; budgetary reasons, filtering the entire original list so it only contains our target population is impossible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do we fix this?  One option is to take a second random sample of 1000, filter for the people who actually are in our target population, then contact everyone who fits the bill (guessing around 1300 total from both samples?).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another option is to take a second random sample of 1000, filter for the people who actually are in our target population, take the resulting list of 1300 and randomly contact 1000 of them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A third option is to throw the original sample back in to the list, take a larger initial random sample (1500-2000 people?), and contact everyone from that sample who meets our criteria.  For time and sanity reasons we'd like to avoid this if possible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All of these feel somehow &quot;off&quot; to me, given my incredibly basic statistical education.  So, gurus: what am I missing?  Is there a fourth option that makes more sense?  Will our results be statistically valid given any of the above approaches?  And does this affect our analysis in any way?&lt;/p&gt;&#xA;" OwnerUserId="2398" LastActivityDate="2010-12-16T02:22:22.583" Title="Survey Sampling: What am I missing with this plan?" Tags="&lt;sampling&gt;&lt;survey&gt;" AnswerCount="2" />
  <row Id="5533" PostTypeId="2" ParentId="5462" CreationDate="2010-12-15T20:36:20.493" Score="0" ViewCount="" Body="&lt;p&gt;If local-only is an option, consider going to places where the &quot;common man&quot; goes, such as the DMV, grocery store, a variety of malls in town (especially at xmas time). Protip: bring a pretty girl along to ask people to do the survey. Sounds sexist but it works :)&lt;/p&gt;&#xA;" OwnerUserId="74" LastActivityDate="2010-12-15T20:36:20.493" />
  <row Id="5534" PostTypeId="1" AcceptedAnswerId="5607" CreationDate="2010-12-15T20:52:37.257" Score="6" ViewCount="216" Body="&lt;p&gt;I was reading Robert Serfling's 1980 book &quot;Approximation Theorems of Mathematical Statistics&quot; and came across the following construction of the Dvoretzky–Kiefer–Wolfowitz inequality for arbitrary distributions $F$,  which DKW prove for distributions on $[0,1]$.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Given independent $X_i$ with d.f. F and defined on a common probability space, one can construct independent uniform $[0,1]$ variates $Y_i$ such that $\mathbf{P}[X_i = F^{-1}(Y_i)] =1,\forall i$.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Why? Is this true for arbitrary distributions (including discontinuous ones)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Secondly,&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Let $G$ denote the uniform $[0,1]$ distribution and $G_n$ the sample distribution function of the $Y_i$s. Then $F(x)=G(F(x))$ and with probability $1$, $F_n(x) = G_n(F(x))$.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Why?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I don't understand quantile functions as well I would like, and so am having some trouble following these arguments. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: All of this is on page 59 of the book.&lt;/p&gt;&#xA;" OwnerUserId="2399" LastActivityDate="2010-12-17T20:13:50.217" Title="Transforming arbitrary distributions to distributions on $[0,1]$" Tags="&lt;distributions&gt;" AnswerCount="3" FavoriteCount="1" />
  <row Id="5535" PostTypeId="2" ParentId="5462" CreationDate="2010-12-15T21:15:14.057" Score="0" ViewCount="" Body="&lt;p&gt;Just a thought, but it might make sense to implement an inexpensive Google Ad Words campaign where at least the participants would be coming directly from Google search and you can somewhat control some sort of stratification.  Of course, it is never possible to get a true sample.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Along these lines, there is some work being done nowadays to adjust for selection bias of Web Surveys via propensity scores. This is being done by some of the majors, including Harris.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.amstat.org/sections/srms/proceedings/y2004/files/Jsm2004-000032.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.amstat.org/sections/srms/proceedings/y2004/files/Jsm2004-000032.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-Ralph Winters&lt;/p&gt;&#xA;" OwnerUserId="2400" LastActivityDate="2010-12-15T21:15:14.057" />
  <row Id="5536" PostTypeId="2" ParentId="5525" CreationDate="2010-12-15T21:19:18.657" Score="0" ViewCount="" Body="&lt;p&gt;I think you could use the component &lt;code&gt;lm.fit$rank&lt;/code&gt; or else subtract &lt;code&gt;lm.fit$df.residual&lt;/code&gt; from the sample size to get what you want. (I assume you want the number of free parameters.)&lt;/p&gt;&#xA;" OwnerUserId="892" LastEditorUserId="892" LastEditorDisplayName="" LastEditDate="2010-12-16T16:11:57.860" LastActivityDate="2010-12-16T16:11:57.860" />
  <row Id="5538" PostTypeId="2" ParentId="5534" CreationDate="2010-12-15T22:09:06.303" Score="11" ViewCount="" Body="&lt;p&gt;This is merely saying that $F(x) = \Pr[X \le x] = \Pr[F(X) \le F(x)]$ which is exactly what it means for $F(X)$ to have a uniform distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;OK, let's go a little slower.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For continuous distributions, forget for a moment that the CDF $F$ is a CDF and think of it as just a nonlinear way to re-express the values of $X$.  In fact, to make the distinction clear, suppose that $G$ is &lt;em&gt;any&lt;/em&gt; monotonically increasing way of re-expressing $X$.  Let $Y$ be the name of its re-expressed value.  $G^{-1}$, by definition, is the &quot;back transform&quot;: it expresses $Y$ back in terms of the original $X$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the distribution of $Y$?  As always, we discover this by picking an arbitrary value that $Y$ might take on, say $y$, and ask for the chance that $Y$ is less than or equal to $y$.  Back-transform this question in terms of the original way of expressing $X$: we are inquiring about the chance that $X$ is less than or equal to $x = G^{-1}(y)$.  Now take $G$ to be $F$ and remember that $F$ is the CDF of $X$: by definition, the chance that $X$ is less than or equal to any $x$ is $F(x)$.  In this case,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$F(x) = F(G^{-1}(y)) = F(F^{-1}(y)) = y.$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We have established that the CDF of $Y$ is $\Pr[Y \le y] = y$, the uniform distribution on $[0,1]$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It can help to look at this graphically.  Draw the graph of $F$.  As $X$ ranges over the reals, $F$ ranges between $0$ and $1$.  The function $F$ is constructed &lt;em&gt;specifically&lt;/em&gt; so that the distribution of $F(x)$ is uniform.  That is, if you want to pick a random value for $X$, pick a &lt;em&gt;uniformly&lt;/em&gt; random value along the y-axis between $0$ and $1$ and find the value of $X$ where $F(X)$ equals that random height.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the continuous case we have $X = F^{-1}(Y)$ so clearly $\Pr[X = F^{-1}(Y)] = 1$.  In the discontinuous case there's no difficulty, either, provided we define $F^{-1}$ appropriately.  But if there's a jump in $X$ from $x_0$ to $x_1 \gt x_0$, all we can say in general is that the event where $x_0 \le X \lt x_1$ has zero probability, not that it's impossible for $X$ to lie in this interval.  For this reason we cannot assert that $X = F^{-1}(Y)$ everywhere, but we can assert that the event that this equality does not hold has probability zero, because it consists of at most a countable number of jumps.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(For a practical example of why this arcane technical distinction is important, consider a gambling problem.  Suppose you will play Roulette repeatedly with a fixed bet until either you are broke or you double your money.  Let $X$ be the random variable representing your net gain &lt;em&gt;if you ever go broke or double your money&lt;/em&gt;.  Otherwise, define $X$ to be any number you want.  $X$ has a Bernoulli distribution because there is some chance $p$ you will go broke, the chance of doubling your money is $1-p$ (work it out!), and the chance of playing forever is zero.  Nevertheless, playing forever is a possibility: it is part of the mathematical set of possible outcomes.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a simple exercise in learning to reason with the uniform probability transform, graph $F$ for a Bernoulli($p$) variable.  The graph equals $0$ for all $x \lt 0$, jumps to $1-p$ at $0$, is horizontal again for $0 \lt x \lt 1$, then jumps to $1$ at $x=1$ and stays there for all greater $x$.  A uniform variate on the interval $[0,1]$ on the y-axis will cover the initial jump with a probability $p$; $F^{-1}$ maps this down to $x = 0$.  Otherwise, the variate covers the final jump with the remaining probability $1-p$ and $F^{-1}$ maps this down to $1$.  We see, then, how a uniform distribution of $Y$ reproduces this simple discrete distribution function.  Illustrations of CDFs of discrete and continuous/discrete distributions appear on &lt;a href=&quot;http://www.quantdec.com/envstats/notes/class_05/distributions.htm&quot; rel=&quot;nofollow&quot;&gt;this page&lt;/a&gt; I wrote for a stats course long ago.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-12-15T22:09:06.303" CommentCount="0" />
  <row Id="5539" PostTypeId="2" ParentId="5525" CreationDate="2010-12-15T22:24:19.247" Score="2" ViewCount="" Body="&lt;p&gt;May be it's a little bit hackish but you can do :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;n &amp;lt;- length(coefficients(model))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2028" LastActivityDate="2010-12-15T22:24:19.247" />
  <row Id="5540" PostTypeId="2" ParentId="5508" CreationDate="2010-12-15T23:46:54.900" Score="3" ViewCount="" Body="&lt;p&gt;It boils down to how you feel about assuming the &lt;em&gt;Independence of Irrelevant Alternatives&lt;/em&gt; (IIA) as an assumption about choice behaviour.  So the first thing to do is look that up.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Multinomial logit assumes IIA and multinomial probit does not.  The computational price of &lt;em&gt;not&lt;/em&gt; assuming it is what gets expensive.  Almost any econometrics text will cover the details, but assuming you already understand logistic regression, the intuitive picture is this: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In latent variable formulation, logistic regression models need to integrate over a latent distribution to get the probability of a 1 rather than a 0. In choice modelling contexts this is thought of as the expected utility of choosing 1 rather than 0, although we only see the final (stochastic) choice.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A similar situation arises when there are multiple choices.  If we are happy to assume IIA, that is: that a third choice does not affect your relative preferences over two existing choices, then the integrals are separable and straightforward.  If you are not happy to assume IIA (and that can be reasonable when the new choice is a plausible substitute for one of the existing ones) then you will have to estimate an arbitrary (unobserved) covariance structure over all the options and then do the multidimensional integrations to get your choice probabilities out.  It is these integrations or replacements for them that cause the computational problem.&lt;/p&gt;&#xA;" OwnerUserId="1739" LastActivityDate="2010-12-15T23:46:54.900" CommentCount="1" />
  <row Id="5541" PostTypeId="2" ParentId="5532" CreationDate="2010-12-16T00:48:28.910" Score="5" ViewCount="" Body="&lt;p&gt;This is a very common situation, and usually one just plans for a larger initial sample size, so that once ineligible subjects are thrown out, the desired sample size is reached. In your case it appears that ~65% of the original list qualifies. So to get 350 more qualifying subjects you need to draw a random sample of about 540. In fact, depending on how you do the filtering, you could keep drawing additional random samples of size 1 until you get the required number of qualifying subjects.&lt;/p&gt;&#xA;" OwnerUserId="279" LastActivityDate="2010-12-16T00:48:28.910" CommentCount="1" />
  <row Id="5542" PostTypeId="1" AcceptedAnswerId="5548" CreationDate="2010-12-16T01:07:58.040" Score="2" ViewCount="145" Body="&lt;h2&gt;Background&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I am generally interested in learning appropriate methods of using data to specify priors. A &lt;a href=&quot;http://stats.stackexchange.com/q/1/1381&quot;&gt;previous question&lt;/a&gt; asks how to elicit priors from experts and received some good recommendations. Here, I am interested in learning how to specify a prior using data. I plan to use these priors in a meta-analysis to synthesize additional data that I collect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;update&lt;/em&gt;  Although John provides a 'correct' answer, in my case, it would require substantial modification of the original model to implement, so I would prefer to find a way to estimate the prior as a discrete step.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Questions&lt;/h2&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;What is the best way to specify such a data informed prior?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If I am working with parameters for a particular species (monkeys), and this species belongs to a group of organisms(primates), and data are available for primates but not for monkeys, would it be appropriate to fit a distribution based on the primate data?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;h2&gt;Example cases, first with proposed solution&lt;/h2&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;I have 100 observations from 100 primate species of primate thumb length:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(0)&#xA;thumb &amp;lt;- rgamma(100, 4, 0.1)&#xA;library(MASS)&#xA;fitdistr(thumb, 'gamma')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Indeed, when there is no apriori reason to select a particular distribution, the distribution can be chosen by maximum likelihood:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for(dist in c('gamma', 'lognormal', 'weibull') {&#xA;logLik(fitdistr(thumb, dist))&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I have collected 50 means, standard errors, and sample sizes from 50 different primate species, and 50 independent observations from another 50 species of eye diameter:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;eye &amp;lt;- data.frame( diameter = rgamma(100, 4, 0.1),&#xA;                   se       = c(rlnorm(50, 0.5,1), rep(NA, 50)),&#xA;                   n        = c(rep(1:5, 10), rep(1, 50)))      &#xA;eye &amp;lt;- signif(eye, 3)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can I incorporate the sample statistics into my calculation of a prior?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="1381" LastEditorUserId="1381" LastEditorDisplayName="" LastEditDate="2010-12-17T20:04:44.930" LastActivityDate="2011-01-18T15:16:19.887" Title="What methods can be used to specify priors from data?" Tags="&lt;probability&gt;&lt;bayesian&gt;&lt;meta-analysis&gt;&lt;prior&gt;" AnswerCount="2" />
  <row Id="5543" PostTypeId="1" AcceptedAnswerId="5564" CreationDate="2010-12-16T01:48:12.853" Score="9" ViewCount="240" Body="&lt;p&gt;I have found some distributions for which BUGS and R have different parameterizations: Normal, log-Normal, and Weibull.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For each of these, I gather that the second parameter used by R needs to be inverse transformed (1/parameter) before being used in BUGS (or JAGS in my case). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone know of a comprehensive list of these transformations that currently exists?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The closest I can find would be comparing the distributions in table 7 of the &lt;a href=&quot;http://sourceforge.net/projects/mcmc-jags/files/Manuals/2.x/jags_user_manual.pdf&quot; rel=&quot;nofollow&quot;&gt;JAGS 2.2.0 user manual&lt;/a&gt; with the results of &lt;code&gt;?rnorm&lt;/code&gt; etc. and perhaps a few probability texts. This approach appears to require that the transformations will need to be deduced from the pdfs separately. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would prefer to avoid this task (and possible errors) if it has already been done, or else start the list here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Based on Ben's suggestions, I have written the following function to transform a dataframe of parameters from R to BUGS parameterizations.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;##' convert R parameterizations to BUGS paramaterizations&#xA;##' &#xA;##' R and BUGS have different parameterizations for some distributions. &#xA;##' This function transforms the distributions from R defaults to BUGS &#xA;##' defaults. BUGS is an implementation of the BUGS language, and these &#xA;##' transformations are expected to work for bugs.&#xA;##' @param priors data.frame with colnames c('distn', 'parama', 'paramb')&#xA;##' @return priors with jags parameterizations&#xA;##' @author David LeBauer&#xA;&#xA;r2bugs.distributions &amp;lt;- function(priors) {&#xA;&#xA;  sd2tau &amp;lt;- priors$distn %in% c('lnorm', 'norm')&#xA;  priors$paramb[sd2tau] &amp;lt;-  1/priors$paramb[sd2tau]^2&#xA;&#xA;  scale2rate &amp;lt;- priors$dist %in% c('weibull')&#xA;  priors$paramb[scale2rate] &amp;lt;-  1 / priors$paramb[scale2rate]&#xA;&#xA;  reverse.order &amp;lt;- priors$dist %in% c('binom')&#xA;  priors$paramb[reverse.order] &amp;lt;-  priors$parama[reverse.order]&#xA;  priors$parama[reverse.order] &amp;lt;-  priors$paramb[reverse.order]&#xA;&#xA;  priors$distn[priors$distn == 'weibull'] &amp;lt;- 'weib'&#xA;  priors$distn[priors$distn == 'binom']   &amp;lt;- 'bin'&#xA;  priors$distn[priors$distn == 'chisq']   &amp;lt;- 'chisqr'&#xA;  priors$distn[priors$distn == 'nbinom']  &amp;lt;- 'negbin'&#xA;&#xA;  return(priors)&#xA;}&#xA;&#xA;##' @examples&#xA;##' priors &amp;lt;- data.frame(distn = c('weibull', 'lnorm', 'norm', 'gamma'),&#xA;##'                     parama = c(1, 1, 1, 1),&#xA;##'                     paramb = c(2, 2, 2, 2))&#xA;##' r2bugs.distributions(priors)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1381" LastEditorUserId="1381" LastEditorDisplayName="" LastEditDate="2011-01-14T22:48:45.363" LastActivityDate="2011-01-16T16:01:06.980" Title="For which distributions are the parameterizations in BUGS and R different?" Tags="&lt;r&gt;&lt;bugs&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="2" />
  <row Id="5544" PostTypeId="2" ParentId="5502" CreationDate="2010-12-16T02:03:23.547" Score="1" ViewCount="" Body="&lt;p&gt;Can I just clarify something here please, do you have items scored on different scales you need to pre-process and combine (interval, ordinal, nominal), or are you looking to do a factor analysis on just ordinal scale variables?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If it is the latter - here is one approach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://cran.r-project.org/web/packages/Zelig/vignettes/factor.ord.pdf&quot; rel=&quot;nofollow&quot;&gt;http://cran.r-project.org/web/packages/Zelig/vignettes/factor.ord.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2238" LastActivityDate="2010-12-16T02:03:23.547" />
  <row Id="5545" PostTypeId="1" CreationDate="2010-12-16T02:04:39.983" Score="5" ViewCount="127" Body="&lt;p&gt;I have to run repeated correlations. If one of them reach the significant value (as given in the Pearson table), how can I be sure that it isn't actually a false positive?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm pretty new in the statistical analysis, so I apologize if my question seems naive.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for any answers.&lt;/p&gt;&#xA;" OwnerUserId="2402" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-12-16T09:59:10.713" LastActivityDate="2010-12-16T09:59:10.713" Title="How to prevent false positive findings with repeated correlations?" Tags="&lt;correlation&gt;&lt;multiple-comparisons&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="5546" PostTypeId="2" ParentId="5532" CreationDate="2010-12-16T02:22:22.583" Score="1" ViewCount="" Body="&lt;p&gt;I hate answering a question with a list of questions, but I hope thinking about some of these issues will give you some insight as to the best way to proceed. The first two questions are mainly for clarification as it wasn't clear from reading your question:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Have you already obtained a list of 1000 people?&lt;/li&gt;&#xA;&lt;li&gt;Have you already collected 650 samples?&lt;/li&gt;&#xA;&lt;li&gt;If you have already collected your survey data, did all 650 people who qualify for your survey actually complete it? Even if they qualify, I have to imagine some # of those people will not complete your survey for a variety of reasons. You'll need to account for some additional attrition to reach your goal of n = 1000.&lt;/li&gt;&#xA;&lt;li&gt;What are the costs associated with obtaining a larger list?&lt;/li&gt;&#xA;&lt;li&gt;Are the costs of collecting the data sufficiently high enough to dissuade you from obtaining &quot;extra&quot; sample above you target of n = 1000?&lt;/li&gt;&#xA;&lt;li&gt;Do you really need n = 1000? What are the negative effects of n = 900? n = 700?&lt;/li&gt;&#xA;&lt;li&gt;Are there high start up costs to collecting the data in phases?&lt;/li&gt;&#xA;&lt;li&gt;How is your survey administered? Can you build in the filtering system into the survey? i.e. &quot;If answer to question 1 == yes, then terminate&quot;?&lt;/li&gt;&#xA;&lt;li&gt;Does your survey contain any temporal elements that may introduce bias if you conduct the survey in two/three/four phases? i.e. are people going to respond or behave differently if they answer the survey in December vis a vis March?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Without knowing more of the specifics of your survey and the population in question, the most straight forward approach in my opinion would be to obtain a random sample that is sufficiently large enough to account for the people who won't qualify and those who won't take your survey. This sounds like option three that you described above. If you already have n = 650, then I would recommend obtaining a list large enough to collect the remaining 350 samples. Does your survey contain demographic data or other benchmarks that you can use to compare across groups (waves of data collection) to see if you have introduced any bias into your data?&lt;/p&gt;&#xA;" OwnerUserId="696" LastActivityDate="2010-12-16T02:22:22.583" />
  <row Id="5547" PostTypeId="2" ParentId="5443" CreationDate="2010-12-16T02:34:59.040" Score="1" ViewCount="" Body="&lt;p&gt;Tal,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My understanding based on this article, is that you cannot obtain the individual variables related to each separate class when the class is a factor. However, using rpart.control, you are able to identify the variables that are important at each node.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://cran.r-project.org/web/packages/caret/vignettes/caretVarImp.pdf&quot; rel=&quot;nofollow&quot;&gt;http://cran.r-project.org/web/packages/caret/vignettes/caretVarImp.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Recursive Partitioning: The reduction in the loss function (e.g. mean squared error)&#xA;attributed to each variable at each split is tabulated and the sum is returned. Also, since there may be candidate variables that are important but are not used in a split, the top competing variables are also tabulated at each split. This can be turned on using the &lt;strong&gt;maxcompete argument in rpart.control&lt;/strong&gt;. This method does not currently provide class specific measures of importance when the response is a factor.&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="2238" LastActivityDate="2010-12-16T02:34:59.040" CommentCount="1" />
  <row Id="5548" PostTypeId="2" ParentId="5542" CreationDate="2010-12-16T03:31:59.010" Score="5" ViewCount="" Body="&lt;p&gt;If you have all this data, I think the best answer is to actually fit a single large model, using Hierarchical Modeling rather than do it in two steps (generating a prior then fitting a model). This is basically the &lt;a href=&quot;http://stats.stackexchange.com/questions/5181/estimating-distribution-parameters-from-few-data-points/5263#5263&quot;&gt;answer I gave to this question&lt;/a&gt;. I explain this a little bit more there. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a hierarchical model you model each of the parameters you are interested in (for example, the location and scale parameters of the thumb lengths for a species) are drawn from a common prior distribution. The hyperparameters of the hierarchical model parameterize the common prior distribution of the parameters for the species', and you estimate the hyperparameters at the same time as the parameters you're interested in. The hyperparameters of course need their own prior distribution, but these can be relatively diffuse.&lt;/p&gt;&#xA;" OwnerUserId="1146" LastActivityDate="2010-12-16T03:31:59.010" CommentCount="3" />
  <row Id="5549" PostTypeId="2" ParentId="5520" CreationDate="2010-12-16T05:25:07.710" Score="1" ViewCount="" Body="&lt;p&gt;Maybe I'm being naive here, but would the Multinomial Distribution not work for this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P(1 unique) = P(1 Blue) + P(1 Red) + P(1 Blue), there are probably a lot of details that would need to be fill in, like P(1 Blue) = The multinomial distribution for all possible combinations of the other ball combinations where there's one blue.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Multinomial_distribution&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Multinomial_distribution&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2387" LastActivityDate="2010-12-16T05:25:07.710" CommentCount="1" />
  <row Id="5550" PostTypeId="1" AcceptedAnswerId="5553" CreationDate="2010-12-16T07:56:16.983" Score="3" ViewCount="168" Body="&lt;p&gt;I want to perform model comparison according to several criteria using R.&#xA;My dataframe's name is &lt;code&gt;df&lt;/code&gt; &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;head(df)&#xA;&#xA;   Y  X1  X2   X3   X4&#xA;1 18 307 130 3504 12.0&#xA;2 15 350 165 3693 11.5&#xA;3 18 318 150 3436 11.0&#xA;4 16 304 150 3433 12.0&#xA;5 17 302 140 3449 10.5&#xA;6 15 429 198 4341 10.0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to compare all possible combinations with Y as the dependent variable. I'll give you what I've done so far to get the R.squared for all possible models. I would like your comments for better coding, especially on getting all those model formulas&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# First get all possible combinations of the 4 inependent variables&#xA;comb_list &amp;lt;- lapply(1:4,function(i) combn(4,i))&#xA;&#xA;# now create the formulas&#xA;comb_list_forms &amp;lt;- lapply(unlist(&#xA;  sapply(1:length(comb_list), function(i)&#xA;  sapply(1:dim(comb_list[[i]])[2], function(x) {nam &amp;lt;- names(df[1+comb_list[[i]][,x]])&#xA;  formul &amp;lt;- &quot;Y~&quot;&#xA; sapply(1:length(nam), function(y) formul &amp;lt;&amp;lt;- paste(formul,nam[y],sep=&quot;+&quot;))&#xA; formul &amp;lt;- sub(&quot;~+&quot;, &quot;~&quot;, formul,fixed = T)&#xA; }))),as.formula)&#xA;&#xA;# finally get r.squared&#xA;attach(df)&#xA;models.r.sq &amp;lt;- sapply(comb_list_forms, function(i) summary(lm(i))$r.squared)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Thank you&lt;/p&gt;&#xA;" OwnerUserId="632" LastEditorUserId="632" LastEditorDisplayName="" LastEditDate="2010-12-16T08:18:23.023" LastActivityDate="2011-02-15T11:31:07.203" Title="Automating model selection criteria production" Tags="&lt;r&gt;&lt;model-selection&gt;" AnswerCount="4" />
  <row Id="5551" PostTypeId="2" ParentId="5520" CreationDate="2010-12-16T07:57:26.163" Score="2" ViewCount="" Body="&lt;p&gt;I will give the outline of the solution. Numbers of each coloured  ball in a draw follows multinomial distribution as tshauck pointed out. Let $R$ denote the number of red balls, $G$ - number of green balls, $B$ - number of blue balls and $Y$ - number of yellow balls in the draw of the size $n$. Then the probability that in random draw we have exactly $x_1$ red balls, $x_2$ green balls, $x_3$ blue balls and $x_4$ yellow ball is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(R=x_1,G=x_2,B=x_3,Y=x_4)=\frac{n!}{x_1!x_2!x_3!x_4!}p_r^{x_1}p_g^{x_2}p_b^{x_3}p_y^{x_4}$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $p_r$ is probability of picking red ball, $p_g$ - green, $p_b$ - blue, $p_y$ - yellow.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Denote the number of unique balls in a draw by $U$. Then $U=f(R,G,B,Y)$. Since you have the distribution of vector $(R,G,B,Y)$ you can calculate distribution of $U$. Since we have four colours $U$ can get values $0,1,2,3,4$. So to get probability that $U=0$ you need to find all the possible combinations of $(R,G,B,Y)$ for which $U=0$ and add the probabilities of these combinations. So when $U=0$? When&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;All the balls appear more than once: $R&amp;gt;1$, $G&amp;gt;1$, $B&amp;gt;1$, $Y&amp;gt;1$&#xD;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;One colour is absent and all others appear more than once&lt;/p&gt;&#xA;&#xA;&lt;p&gt;a. $R=0$, $G&amp;gt;1$, $B&amp;gt;1$, $Y&amp;gt;1$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;b. $R&amp;gt;1$, $G=0$, $B&amp;gt;1$, $Y&amp;gt;1$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;c. $R&amp;gt;1$, $G&amp;gt;1$, $B=0$, $Y&amp;gt;1$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;d. $R&amp;gt;1$, $G&amp;gt;1$, $B&amp;gt;1$, $Y=0$&#xD;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Two colours are absent  and all others appear more than once. $6$ cases&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Three colours are absent, all the draw is of one colour. $4$ cases&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;All the four cases are mutually exclusive, so you can add the probabilities.  Cases of $U=1,2,3,4$ can be treated similarly. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This of course is not an elegant solution, but it I do not see why it should not work. I suggest asking this in &lt;a href=&quot;http://math.stackexchange.com&quot;&gt;math.stackexchange.com&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update 1&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This approach is for calculating the probability distribution of $U$. For expected value of $U$ - whuber's answer is the right one. &lt;/p&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2010-12-16T18:46:12.313" LastActivityDate="2010-12-16T18:46:12.313" CommentCount="3" />
  <row Id="5552" PostTypeId="2" ParentId="4267" CreationDate="2010-12-16T08:03:12.553" Score="0" ViewCount="" Body="&lt;p&gt;At this year's &lt;a href=&quot;http://nips.cc/&quot; rel=&quot;nofollow&quot;&gt;NIPS&lt;/a&gt; there was a short paper on &lt;a href=&quot;http://nlp.fi.muni.cz/~xrehurek/nips/rehurek_nips.pdf&quot; rel=&quot;nofollow&quot;&gt;distributed, very large-scale SVD that works in a single pass over a streaming input matrix&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The paper's more implementation-oriented but puts things into perspective with real wall-clock times and all. The table near the beginning is a good survey too.&lt;/p&gt;&#xA;" OwnerUserId="2404" LastEditorUserId="223" LastEditorDisplayName="" LastEditDate="2010-12-16T08:38:52.443" LastActivityDate="2010-12-16T08:38:52.443" CommunityOwnedDate="2010-12-16T08:03:12.553" CommentCount="2" />
  <row Id="5553" PostTypeId="2" ParentId="5550" CreationDate="2010-12-16T08:10:19.380" Score="5" ViewCount="" Body="&lt;p&gt;I think package &lt;a href=&quot;http://cran.r-project.org/web/packages/leaps/index.html&quot; rel=&quot;nofollow&quot;&gt;leaps&lt;/a&gt; will help you. It is designed specifically to select a &quot;best&quot; subset of variables for linear regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for the formula creation, you can use foreach package like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; #Create the list where each list element contains one combination&#xA;  xcomb &amp;lt;- foreach(i=1:4, .combine=c) %do% { &#xA;       combn(paste(&quot;X&quot;, 1:4, sep=&quot;&quot;), i, simplify=FALSE)&#xA; }&#xA; #Convert list elements to formula   &#xA; formlist &amp;lt;- lapply(xcomb, function(l) formula(paste(&quot;Y&quot;, paste(l, collapse=&quot;+&quot;), sep=&quot;~&quot;)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2010-12-16T13:20:56.190" LastActivityDate="2010-12-16T13:20:56.190" CommentCount="2" />
  <row Id="5554" PostTypeId="2" ParentId="5545" CreationDate="2010-12-16T08:17:59.500" Score="5" ViewCount="" Body="&lt;p&gt;I think  you should consider &lt;a href=&quot;http://en.wikipedia.org/wiki/Bonferroni_correction&quot; rel=&quot;nofollow&quot;&gt;Bonferroni&lt;/a&gt; correction. This seems a &lt;a href=&quot;http://en.wikipedia.org/wiki/Multiple_comparisons&quot; rel=&quot;nofollow&quot;&gt;multiple comparison&lt;/a&gt; problem, so Bonferroni is only one of the solutions, but it is the easiest to implement. &lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2010-12-16T08:17:59.500" CommentCount="6" />
  <row Id="5555" PostTypeId="1" AcceptedAnswerId="5557" CreationDate="2010-12-16T09:44:12.623" Score="0" ViewCount="87" Body="&lt;p&gt;Hi there, I've been having some trouble with this story problem.  Any help you could give me would be really appreciated.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A store manager monitors the store's temperature by taking 4 independent temperature readings at 4 locations where, if the system is working correctly, the temp. is normally distributed with a mean of 68 degrees F and a standard deviation of 3 degrees.  What's the mean and standard deviation of the average of these 4 readings if the system is working correctly?  What upper and lower limits should the manager set for the average of these 4 readings for there to be a .01 probability of violating a limit when the system is working correctly?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for any advice/help you can give me!&lt;/p&gt;&#xA;" OwnerUserId="2385" LastActivityDate="2010-12-16T11:12:31.337" Title="Probability and limits" Tags="&lt;distributions&gt;&lt;standard-deviation&gt;&lt;homework&gt;&lt;mean&gt;" AnswerCount="1" />
  <row Id="5556" PostTypeId="2" ParentId="5550" CreationDate="2010-12-16T10:29:14.630" Score="2" ViewCount="" Body="&lt;p&gt;This is another way to get the formulas &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;comb_list_forms &amp;lt;- unlist(lapply(1:length(comb_list), function(i)&#xA;  lapply(1:dim(comb_list[[i]])[2], function(x) &#xA;    as.formula(paste(&quot;Y ~ &quot;, paste(names(df[1+comb_list[[i]][,x]]), collapse= &quot;+&quot;))))))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="339" LastActivityDate="2010-12-16T10:29:14.630" />
  <row Id="5557" PostTypeId="2" ParentId="5555" CreationDate="2010-12-16T11:07:02.010" Score="5" ViewCount="" Body="&lt;p&gt;The two key elements that you need to be familiar with before working through this exercise are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The &lt;a href=&quot;http://en.wikipedia.org/wiki/Sampling_distribution&quot; rel=&quot;nofollow&quot;&gt;sampling distribution&lt;/a&gt; of a mean: in your case, the average of your four measurements will follow a normal distribution, with a mean that equals that of the parent distribution but with a standard deviation of $3/\sqrt{4}$.&lt;/li&gt;&#xA;&lt;li&gt;How to translate probabilistic assertions in terms of the underlying &lt;a href=&quot;http://en.wikipedia.org/wiki/Probability_density_function&quot; rel=&quot;nofollow&quot;&gt;Probability Density Function&lt;/a&gt;: given that there is an infinite number of possible PDFs for a distribution described by two parameters (its location and shape), it is often more convenient to work with the standardized normal distribution which is simply $\mathcal{N}(0;1)$, because if $X\sim \mathcal{N}(\mu; \sigma)$, then we know that $Z=\frac{X-\mu}{\sigma}\sim\mathcal{N}(0;1)$.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Then, you just have to figure out how to find $z_1$ and $z_2$ such that $\Pr(z_1\leq Z\leq z_2)=1-0.01$. It often helps to draw the graph of $\Pr(Z\leq z)$ which is bell-shaped, centered on its mean, and whose total area equals 1. For instance, for a given quantile $z_1$, $\Pr(Z&amp;lt;z_1)=p_1$ where $p_1$ is the shaded area shown below (here, $z_1=-1$, that is 1 standard deviation below the mean):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/Bua04.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As the total area equals 1, the remaining (unshaded) area equals $1-p_1$. Likewise, you may readily express any bounded area as a sum or difference of such inequalities.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-12-16T11:12:31.337" LastActivityDate="2010-12-16T11:12:31.337" CommentCount="3" />
  <row Id="5558" PostTypeId="2" ParentId="5550" CreationDate="2010-12-16T12:46:10.393" Score="2" ViewCount="" Body="&lt;p&gt;You could consider using &lt;code&gt;step&lt;/code&gt; on the full model. This uses AIC rather than $R^2$ as the criterion for selection as this is considered to be superior.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In any case, the functions &lt;code&gt;add1&lt;/code&gt; and &lt;code&gt;drop1&lt;/code&gt; that &lt;code&gt;step&lt;/code&gt; uses may be of use.&lt;/p&gt;&#xA;" OwnerUserId="229" LastActivityDate="2010-12-16T12:46:10.393" />
  <row Id="5559" PostTypeId="1" CreationDate="2010-12-16T13:15:13.730" Score="1" ViewCount="203" Body="&lt;p&gt;What is the best way to divide Cross Validated/Stack Overflow/Server fault reputations into separate bins, considering reputation systems tend to be highly skewed?  For example: Suppose I want a set of 64 bins - 0-250,250-500,500-100...10,000-11,000. Is there a method that takes into account the number of users that have a specific reputation wen specifying the bin limits or can I just assign arbitrary bins?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;slotishtype&lt;/p&gt;&#xA;" OwnerUserId="2405" LastEditorUserId="2405" LastEditorDisplayName="" LastEditDate="2011-01-20T10:18:47.440" LastActivityDate="2011-01-20T18:06:28.053" Title="How to divide ordinal set into bins?" Tags="&lt;ordinal&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="5560" PostTypeId="1" CreationDate="2010-12-16T14:57:00.873" Score="2" ViewCount="88" Body="&lt;p&gt;I have a random events generator. I know in advance the set of event that can be generated (in my case I have only three possible events). The probabilities of the events are not known. I need to estimate these probabilities. For that I run an experiment. For example I generate 20 events. So, I have a sequence of events. For example: &lt;code&gt;a,a,b,a,c,c,c,a,b,....c&lt;/code&gt;. Having the sequence I can count for the number of every event (so, in my case I get three integer numbers $n_1$, $n_2$ and $n_3$).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can calculate the probabilities of every event in the following way:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\nu_1 = \frac{n_1}{n_1+n_2+n_3}$&#xD;&#xA;$\nu_2 = \frac{n_2}{n_1+n_2+n_3}$&#xD;&#xA;$\nu_3 = \frac{n_3}{n_1+n_2+n_3}$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But these values are approximate. For example, I can have probability of 0.25 for the event &lt;code&gt;a&lt;/code&gt;, so if I generate 20 events, I have to get approximately 5 &lt;code&gt;a&lt;/code&gt;-events. But just by chance I can get 10 or 0 &lt;code&gt;a&lt;/code&gt;-events.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, I want to have a density distribution of probabilities. Since, $\nu_1$, $\nu_2$ and $\nu_3$ are dependent (the sum of them is equal to one) I going to use $\nu_1$ and $\nu_2$. So, I want to have an explicit form for the density distribution of $\nu_1$ and $\nu_2$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\rho (\nu_1,\nu_2) = F (n_1,n_2,n_3)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anybody know where I can get it?&lt;/p&gt;&#xA;" OwnerUserId="2407" LastActivityDate="2010-12-16T16:39:56.680" Title="How to estimate the likelihood function for random generator of three events?" Tags="&lt;distributions&gt;&lt;probability&gt;&lt;measurement&gt;&lt;likelihood-function&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="5561" PostTypeId="2" ParentId="5550" CreationDate="2010-12-16T15:04:03.347" Score="4" ViewCount="" Body="&lt;p&gt;Ok, here is another way (which is certainly not as elegant or concise than the other ones, but it has the merit of not requiring Emacs to check parenthesis matches :-)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say we have a vector of predictors of interest like this&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; Xs &amp;lt;- paste(&quot;X&quot;, 1:4, sep=&quot;&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then, we can just use&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; allXs &amp;lt;- lapply(seq(along=Xs), &#xA;                  function(x) combn(Xs, x, simplify=FALSE, &#xA;                                    FUN=paste, collapse=&quot;+&quot;))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; unlist(allXs)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;gives you all 15 combinations of the X's.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another option is to just change the right-hand side of a formula, say &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; fm &amp;lt;- as.formula(paste(&quot;Y ~ &quot;, paste(Xs, collapse= &quot;+&quot;)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;so as to reflect the different combinations that you enumerate in your &lt;code&gt;comb_list&lt;/code&gt; object. This can be done using the &lt;a href=&quot;http://cran.r-project.org/web/packages/Formula/index.html&quot; rel=&quot;nofollow&quot;&gt;Formula&lt;/a&gt; package:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; fm &amp;lt;- as.formula(paste(&quot;Y ~ &quot;, paste(Xs, collapse= &quot;|&quot;)))&#xA;&amp;gt; fm2 &amp;lt;- Formula(fm)&#xA;&amp;gt; foo &amp;lt;- function(x) formula(fm2, rhs=x, collapse=TRUE)&#xA;&amp;gt; foo(1:2)&#xA;Y ~ X1 + X2&#xA;&amp;lt;environment: 0x102593040&amp;gt;&#xA;&amp;gt; foo(c(1,3,4))&#xA;Y ~ X1 + X3 + X4&#xA;&amp;lt;environment: 0x1021d02a0&amp;gt;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-12-16T15:04:03.347" CommentCount="1" />
  <row Id="5562" PostTypeId="1" CreationDate="2010-12-16T15:09:52.723" Score="0" ViewCount="121" Body="&lt;p&gt;Allo All,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am looking at 5000 pricepoints of hourly time series each over 3 months of time.&#xA; my problem is &#xA;      a) identify and &#xA;      b) rank them by a suitable co-integration type measure a set of pairs(2-tuples) of pricepoints which tend to Move in the OPPOSITE direction on an hourly basis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Atleast 60% of the time they pricepoints will be near zero and then rest they will ideally move in the opposite direction though not the same amount and atleast they should not much move in the same direction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So&#xA;a) what is a good measure to be used here &#xA;b) once we identify the measure what is a good heursistic to restrict the search space to less than the 5000C2 &#xA;c) what would be a good R package.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Appreciate ur consideration in earnest&#xA;Yet another student of statistics!&lt;/p&gt;&#xA;" OwnerUserId="1896" LastActivityDate="2010-12-16T15:09:52.723" ClosedDate="2011-04-11T01:48:10.543" Title="identifying pairs(by a cointegration measure) in a pair trading type problem" Tags="&lt;time-series&gt;&lt;correlation&gt;&lt;clustering&gt;&lt;cointegration&gt;" CommentCount="2" FavoriteCount="2" />
  <row Id="5563" PostTypeId="1" AcceptedAnswerId="5565" CreationDate="2010-12-16T15:13:16.647" Score="0" ViewCount="154" Body="&lt;p&gt;Is there a 'limit' function in R? &lt;code&gt;help(lim)&lt;/code&gt; or &lt;code&gt;help(limit)&lt;/code&gt; weren't useful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And in general, is R suitable for mathematical computations?&lt;/p&gt;&#xA;" OwnerUserId="1700" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-12-16T15:20:42.767" LastActivityDate="2010-12-16T20:10:54.397" Title="Is there a limit function in R?" Tags="&lt;r&gt;&lt;mathematics&gt;" AnswerCount="2" CommentCount="4" />
  <row Id="5564" PostTypeId="2" ParentId="5543" CreationDate="2010-12-16T15:25:54.663" Score="13" ViewCount="" Body="&lt;p&gt;I don't know of a canned list. Here is my list (edits provided by original questioner):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Normal and log-normal&lt;/strong&gt;  are parameterized in terms of $\tau$ (precision) rather&#xA;than $\sigma$ or $\sigma^2$ (std. dev. or variance); $\tau = 1/\sigma^2 =&#xD;&#xA;1/\mbox{var}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Beta, Poisson, Exponential, Uniform&lt;/strong&gt; are all the same&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Negative binomial&lt;/strong&gt; in BUGS has only the discrete parameterization&#xA;(size,prob), not the &quot;ecological&quot; (size,mu, where size can be&#xA;non-integer) parameterization.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Weibull&lt;/strong&gt; in BUGS is (&lt;code&gt;shape&lt;/code&gt;,&lt;code&gt;rate&lt;/code&gt;), in R is (&lt;code&gt;shape&lt;/code&gt;,&lt;code&gt;scale&lt;/code&gt;); rate = 1/scale&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Gamma&lt;/strong&gt; in BUGS is (&lt;code&gt;shape&lt;/code&gt;,&lt;code&gt;rate&lt;/code&gt;). This is the default in R, but R&#xA;also allows (shape,scale) [if the scale argument is named]; rate =&#xA;1/scale&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Order matters&lt;/strong&gt;, especially in BUGS (which doesn't have named&#xA;arguments), e.g. R &lt;code&gt;dbinom(x,size,prob)&lt;/code&gt; vs BUGS &lt;code&gt;dbin(p,n)&lt;/code&gt; [same&#xA;parameters, opposite order].&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Name differences&lt;/strong&gt;: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Binomial&lt;/strong&gt;: R=&lt;code&gt;dbinom&lt;/code&gt;,BUGS=&lt;code&gt;dbin&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Chi-squared&lt;/strong&gt;: R=&lt;code&gt;dchisq&lt;/code&gt;,BUGS=&lt;code&gt;dchisqr&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Weibull&lt;/strong&gt;: R=&lt;code&gt;dweibull&lt;/code&gt;,BUGS=&lt;code&gt;dweib&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Negative binomial&lt;/strong&gt;: R=&lt;code&gt;dnbinom&lt;/code&gt;, BUGS=&lt;code&gt;dnegbin&lt;/code&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="2126" LastEditorUserId="2126" LastEditorDisplayName="" LastEditDate="2011-01-16T16:01:06.980" LastActivityDate="2011-01-16T16:01:06.980" CommentCount="2" />
  <row Id="5565" PostTypeId="2" ParentId="5563" CreationDate="2010-12-16T15:29:34.803" Score="1" ViewCount="" Body="&lt;p&gt;Following the advice of &lt;a href=&quot;http://tolstoy.newcastle.edu.au/R/e11/help/10/07/2296.html&quot; rel=&quot;nofollow&quot;&gt;this answer&lt;/a&gt; on the r-help mailing list, you can find examples in the sympy help file in the &lt;a href=&quot;http://rsympy.googlecode.com&quot; rel=&quot;nofollow&quot;&gt;rSymPy&lt;/a&gt; and in the Ryacas vignette in the &lt;a href=&quot;http://Ryacas.googlecode.com&quot; rel=&quot;nofollow&quot;&gt;Ryacas package&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, all of the work that I do in R requires mathematical computations, and R does an excellent job of facilitating this work. &lt;/p&gt;&#xA;" OwnerUserId="1381" LastActivityDate="2010-12-16T15:29:34.803" />
  <row Id="5566" PostTypeId="1" CreationDate="2010-12-16T15:36:19.483" Score="1" ViewCount="196" Body="&lt;p&gt;I flip a coin 20 times and get 14 heads. I want to calculate the p-value of the hypothesis that my coin is fair. What probability should I calculate? In Wikipedia it is written that I need to calculate the probability to get 14 &lt;strong&gt;or more&lt;/strong&gt; heads in 20 flips. Why is it 14 &quot;or more&quot;? Why not 14 &lt;strong&gt;or less&lt;/strong&gt;?&lt;/p&gt;&#xA;" OwnerUserId="2407" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-16T20:03:10.180" LastActivityDate="2010-12-16T20:03:10.180" Title="Testing if a coin is fair" Tags="&lt;maximum-likelihood&gt;&lt;p-value&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="5568" PostTypeId="2" ParentId="5520" CreationDate="2010-12-16T16:20:37.650" Score="6" ViewCount="" Body="&lt;p&gt;Let $X_i$ be the random variable equal to $1$ when there is exactly one ball of color $i$ ($i = 1, 2, \ldots, m$; to avoid confusion I write $m$ instead of $N$).  The count of color $i$ follows a Binomial($p_i$, $n$) distribution, implying the expectation of $X_i$ is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\eqalign{&#xD;&#xA;\mathbb{E}[X_i] = &amp;amp;\sum_{j=0}^{n} \binom{n}{j} p_i^j (1-p_i)^{n-j} X_i(j) \cr&#xD;&#xA;= &amp;amp;\binom{n}{1} p_i (1-p_i)^{n-1} \cr&#xD;&#xA;= &amp;amp;n p_i (1-p_i)^{n-1}.&#xD;&#xA;}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The number of unique colors is the sum of the $X_i$.  Because expectation is linear we obtain&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\eqalign{&#xD;&#xA;\mathbb{E}[X] = &amp;amp;\sum_{i=1}^{m}\mathbb{E}[X_i] = \sum_{i=1}^{m}n p_i (1-p_i)^{n-1} \cr&#xD;&#xA;= &amp;amp;n\sum_{i=1}^{m} p_i (1-p_i)^{n-1}.&#xD;&#xA;}$$&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-12-21T15:01:39.427" LastActivityDate="2010-12-21T15:01:39.427" CommentCount="2" />
  <row Id="5569" PostTypeId="2" ParentId="5560" CreationDate="2010-12-16T16:39:56.680" Score="4" ViewCount="" Body="&lt;p&gt;You should look at the &lt;a href=&quot;http://en.wikipedia.org/wiki/Multinomial_distribution&quot; rel=&quot;nofollow&quot;&gt;multinomial distribution&lt;/a&gt;.&lt;/p&gt;&#xA;" LastActivityDate="2010-12-16T16:39:56.680" />
  <row Id="5570" PostTypeId="2" ParentId="5566" CreationDate="2010-12-16T16:42:57.647" Score="7" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;It's neither&lt;/strong&gt; because the alternative to being fair is that the coin favors heads &lt;em&gt;or&lt;/em&gt; tails.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You are free to invent any test you like.  For example, I could (idiosyncratically) decide the coin is unfair if and only if the number of heads is either 6 or 15 (the &quot;critical region&quot;), because this event occurs with only 5% chance when the coin is fair.  &lt;em&gt;The key question is how well does a test perform&lt;/em&gt;.  The &lt;a href=&quot;http://en.wikipedia.org/wiki/Neyman%E2%80%93Pearson_lemma&quot; rel=&quot;nofollow&quot;&gt;Neyman-Pearson lemma&lt;/a&gt; shows that this particular one I just invented is a poor test.  A good test is one whose critical region not only is unlikely when the null is true, but is also highly likely when the null is false.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is no one best critical region for this kind of two-sided test, but a reasonable compromise is to adopt a procedure that will detect deviations from fairness in both directions.  That suggests a critical region that contains the most extreme possibilities: a bunch near 0 and a bunch near 20.  A good choice at the 5% level is to consider any outcome of 15 or more &lt;em&gt;or&lt;/em&gt; 5 or less to be significant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let us then adopt the best &lt;em&gt;symmetric&lt;/em&gt; test for a fair coin.  This means we want the critical region to include $20-i$ heads whenever it includes $i$ heads (that is, $20-i$ tails).  This treats heads and tails on an equal footing.  &lt;em&gt;It is only in the context of a particular test (like this one) that a p-value has any meaning.&lt;/em&gt;  The p-value corresponding to an outcome of 14 is, by definition, the smallest significance of any such test that includes 14 in its critical region.  By symmetry it must include 20 - 14 = 6 and by the Neyman-Pearson lemma it must include all values larger than 14 and all values less than 6.  The chance of this under the null is 11.53%.  This chance increases uniformly as the chance of heads deviates more and more from 1/2 &lt;em&gt;in either direction.&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-12-16T17:54:03.310" LastActivityDate="2010-12-16T17:54:03.310" CommentCount="1" />
  <row Id="5571" PostTypeId="1" AcceptedAnswerId="5579" CreationDate="2010-12-16T18:04:08.740" Score="8" ViewCount="373" Body="&lt;p&gt;I recently fit 4 multiple regression models for the same predictor/response data. Two of the models I fit with Poisson regression.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.pois &amp;lt;- glm(Response ~ P1 + P2 +...+ P5, family=poisson(), ...)&#xA;model.pois.inter &amp;lt;- glm(Response ~ (P1 + P2 +...+ P5)^2, family=poisson(), ...)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Two of the models I fit with negative binomial regression.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(MASS)&#xA;model.nb &amp;lt;- glm.nb(Response ~ P1 + P2 +...+ P5, ...)&#xA;model.nb.inter &amp;lt;- glm.nb(Response ~ (P1 + P2 +...+ P5)^2, ...)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is there a statistical test I can use to compare these models? I've been using the AIC as a measure of the fit, but AFAIK this doesn't represent an actual test.&lt;/p&gt;&#xA;" OwnerUserId="1973" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-12-16T22:03:34.217" LastActivityDate="2010-12-17T07:52:18.490" Title="Comparing regression models" Tags="&lt;regression&gt;&lt;aic&gt;&lt;model-comparison&gt;&lt;likelihood-ratio&gt;" AnswerCount="2" CommentCount="7" />
  <row Id="5572" PostTypeId="1" AcceptedAnswerId="5584" CreationDate="2010-12-16T18:16:48.147" Score="4" ViewCount="124" Body="&lt;p&gt;I am planing a pre-post treatment-control design study with a large number of pre-treatment measurements. I have subjects divided into a control group and a treatment group. For both groups, I will collect hourly data for one year prior to the start of the treatment and then continue collecting data for another year. This will yield approximately 9000 pre-treatment measurements and 9000 post treatment measurements for each subject.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The treatment is something that cannot be stopped once it is started, so a cross-over design could only be of the form AA/AB, which won't take advantage of the benefits of that type of design.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The psychological and bio-medical literature suggests using an ANCOVA model, where the pre-treatment data is used as a covariate in the model. Putting 9000 covariates in a model seems totally ridiculous. Also, reducing the pre-treatment data to a summary statistic doesn't take advantage of the large number of measurements.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm sure that this must have come up before, any ideas? References to published results would be especially helpful.&lt;/p&gt;&#xA;" OwnerUserId="743" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-17T21:50:50.030" LastActivityDate="2010-12-17T21:50:50.030" Title="Taking advantage of many pre-treatment measurements" Tags="&lt;mixed-model&gt;&lt;experiment-design&gt;&lt;repeated-measures&gt;&lt;panel-data&gt;" AnswerCount="2" CommentCount="4" />
  <row Id="5573" PostTypeId="1" AcceptedAnswerId="5577" CreationDate="2010-12-16T20:06:26.387" Score="4" ViewCount="116" Body="&lt;p&gt;I would like to create an &quot;equilibrated histogram&quot; with roughly the same number of data points in each bin. The second restriction I have is that I should have only 4 bins. Given the following list of numbers below, how can I achieve that?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-2.153, -1.732, -1.699, -1.559, -1.355, -1.306, -1.151, -1.129, -0.636, 0.4085, 0.5408, 0.5731, 0.5842, 0.6206, 0.8175, 0.8274, 0.8710, 1.3214, 1.5552, 2.2342&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="64" LastEditorUserId="696" LastEditorDisplayName="" LastEditDate="2010-12-16T20:49:03.377" LastActivityDate="2010-12-16T23:28:25.913" Title="How to build an &quot;equilibrated histogram&quot;?" Tags="&lt;r&gt;&lt;histogram&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="1" />
  <row Id="5574" PostTypeId="2" ParentId="5563" CreationDate="2010-12-16T20:10:54.397" Score="1" ViewCount="" Body="&lt;p&gt;As @David mentioned, R is not doing symbolic calculations on its own, and such jobs should be done in other programs or with wrapper packages.&lt;br&gt;&#xA;Yet, calculating limit numerically is usually quite easy -- in most cases you just need to find argument $n$ large enough so the difference between $f(n+1)-f(n)$ is smaller than the desired accuracy -- this can be done in a simple loop.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-12-16T20:10:54.397" />
  <row Id="5575" PostTypeId="2" ParentId="5573" CreationDate="2010-12-16T20:14:35.177" Score="6" ViewCount="" Body="&lt;p&gt;You are looking for &lt;a href=&quot;http://en.wikipedia.org/wiki/Quantile&quot; rel=&quot;nofollow&quot;&gt;quantiles&lt;/a&gt;; in R there is a function &lt;code&gt;quantile&lt;/code&gt; that will calculate them for you; &lt;code&gt;Hmisc&lt;/code&gt; R package provides &lt;code&gt;cut2&lt;/code&gt; function which explicitly calculates such &quot;equilibrated bins&quot;. &lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-12-16T20:14:35.177" CommentCount="1" />
  <row Id="5576" PostTypeId="2" ParentId="2828" CreationDate="2010-12-16T20:16:41.043" Score="0" ViewCount="" Body="&lt;p&gt;Maybe this is part of the answer you're looking for:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;books.nips.cc/papers/files/nips22/NIPS2009_1210.pdf&lt;/p&gt;&#xA;" OwnerUserId="2410" LastActivityDate="2010-12-16T20:16:41.043" CommentCount="1" />
  <row Id="5577" PostTypeId="2" ParentId="5573" CreationDate="2010-12-16T20:48:42.173" Score="5" ViewCount="" Body="&lt;p&gt;To follow up on @mbq's suggestion, here's the code to do that with R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; require(Hmisc)&#xA;&amp;gt; &#xA;&amp;gt; x &amp;lt;- c(-2.153, -1.732, -1.699, -1.559, -1.355&#xA;+ , -1.306, -1.151, -1.129, -0.636, 0.4085&#xA;+ , 0.5408, 0.5731, 0.5842, 0.6206, 0.8175&#xA;+ , 0.8274, 0.8710, 1.3214, 1.5552, 2.2342&#xA;+ )&#xA;&amp;gt; eqBins &amp;lt;- cut2(x, g = 4)&#xA;&amp;gt; &#xA;&amp;gt; #what are the bins and how many in each?&#xA;&amp;gt; as.data.frame(table(eqBins))&#xA;           eqBins Freq&#xA;1 [-2.153,-1.306)    5&#xA;2 [-1.306, 0.541)    5&#xA;3 [ 0.541, 0.827)    5&#xA;4 [ 0.827, 2.234]    5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="696" LastEditorUserId="696" LastEditorDisplayName="" LastEditDate="2010-12-16T23:28:25.913" LastActivityDate="2010-12-16T23:28:25.913" CommentCount="2" />
  <row Id="5578" PostTypeId="2" ParentId="5364" CreationDate="2010-12-16T21:07:01.267" Score="1" ViewCount="" Body="&lt;p&gt;I discovered a paper by Reiss, &lt;em&gt;et. al.&lt;/em&gt;, &#xA;&lt;a href=&quot;http://dx.doi.org/10.1016/j.chemolab.2009.11.003&quot; rel=&quot;nofollow&quot;&gt;Partial least squares confidence interval calculation for industrial end-of-batch quality prediction&lt;/a&gt;, in which appears the quote:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The PLS prediction should be accompanied by an online conﬁdence&#xA;  interval to indicate the accuracy of the prediction. The formulation of&#xA;  the conﬁdence interval for the PLS prediction is an area of study that&#xA;  has not concluded a “gold standard”.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This paper contains a reference to the 'excellent survey of such work', &lt;a href=&quot;http://dx.doi.org/10.1016/S0169-7439%2801%2900204-0&quot; rel=&quot;nofollow&quot;&gt;Standard error of prediction for multiway PLS&lt;/a&gt;, by Faber and Bro, and a paper by Faber and Kowalski, &#xA;&lt;a href=&quot;http://http%3A%2F%2Fdx.doi.org%2Fdoi%3A10.1002%2F%28SICI%291099-128X%28199705%2911%3A3%3C181%3A%3AAID-CEM459%3E3.0.CO%3B2-7&quot; rel=&quot;nofollow&quot;&gt;Propagation of measurement errors for the validation of predictions obtained by principal component regression and partial least squares&lt;/a&gt;.&#xA;I will summarize these results as they become available...&lt;/p&gt;&#xA;" OwnerUserId="795" LastEditorUserId="795" LastEditorDisplayName="" LastEditDate="2010-12-16T22:27:47.410" LastActivityDate="2010-12-16T22:27:47.410" CommentCount="1" />
  <row Id="5579" PostTypeId="2" ParentId="5571" CreationDate="2010-12-16T21:47:11.920" Score="7" ViewCount="" Body="&lt;p&gt;You can compare the negative binomial model to the corresponding Poisson model with a likelihood ratio test. A Poisson model is equivalent to A negative binomial model with an overdispersion parameter of zero. Therefore they are nested models and likelihood ratios are valid. The complication is that the overdispersion parameter is restricted to be non-negative, i.e. it logically can't be less than zero, so the null hypothesis is on the boundary of the parameter space. This means that instead of comparing twice the log-likelihood to a chi-squared distribution with one degree of freedom, you need to compare it to a mixture distribution consisting of equal parts of a chi-squared with 1 d.f. and a point mass at zero (a chi-squared distribution with zero degrees of freedom). What that means in practice is that you can calculate the p-value using the chi-squared with 1 d.f. and then halve it. For more details and background, see Case 5 of &lt;a href=&quot;http://www.jstor.org/stable/2289471&quot; rel=&quot;nofollow&quot;&gt;Self &amp;amp; Liang &lt;em&gt;JASA&lt;/em&gt; 1987; &lt;strong&gt;82&lt;/strong&gt;:605-610.&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that some statistical software packages, such as Stata, will do this all for you automatically when you fit a negative binomial model. In fact i've shamelessly cribbed much of the above from the Stata help system -- if you have Stata see &lt;code&gt;&lt;a href=&quot;http://www.stata.com/help.cgi?j_chibar&quot; rel=&quot;nofollow&quot;&gt;help j_chibar&lt;/a&gt;&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="449" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-12-17T07:52:18.490" LastActivityDate="2010-12-17T07:52:18.490" />
  <row Id="5580" PostTypeId="2" ParentId="5525" CreationDate="2010-12-16T23:00:58.127" Score="2" ViewCount="" Body="&lt;p&gt;A more general approach is to use the &lt;code&gt;logLik()&lt;/code&gt; function.  It returns an object with the attribute &lt;code&gt;df&lt;/code&gt; that gives the fitted models degrees of freedom.  The benefit of this approach is that it works with many other model classes (including &lt;code&gt;glm&lt;/code&gt;).  In the case of ordinary linear regression (&lt;code&gt;lm&lt;/code&gt;) this corresponds to the number of parameters + 1 for the  estimate of the error variance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the &lt;code&gt;logLik&lt;/code&gt; documentation:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;For &quot;lm&quot; fits it is assumed that the scale has been estimated (by maximum likelihood or REML), and all the constants in the log-likelihood are included.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;You can get the number of observations this way too.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; X1 &amp;lt;- rnorm(10)&#xA;&amp;gt; X2 &amp;lt;- rnorm(10)&#xA;&amp;gt; Y &amp;lt;- X1 + X2 + rnorm(10)&#xA;&amp;gt; model &amp;lt;- lm(Y~X1+X2)&#xA;&amp;gt; ll &amp;lt;- logLik(model)&#xA;&amp;gt; attributes(ll)&#xA;$nall&#xA;[1] 10&#xA;&#xA;$nobs&#xA;[1] 10&#xA;&#xA;$df&#xA;[1] 4&#xA;&#xA;$class&#xA;[1] &quot;logLik&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1670" LastEditorUserId="1670" LastEditorDisplayName="" LastEditDate="2010-12-16T23:16:31.660" LastActivityDate="2010-12-16T23:16:31.660" CommentCount="1" />
  <row Id="5581" PostTypeId="2" ParentId="4551" CreationDate="2010-12-16T23:13:28.913" Score="9" ViewCount="" Body="&lt;p&gt;Failing to look at (plot) the data.&lt;/p&gt;&#xA;" OwnerUserId="1670" LastActivityDate="2010-12-16T23:13:28.913" CommunityOwnedDate="2010-12-16T23:13:28.913" CommentCount="2" />
  <row Id="5582" PostTypeId="1" CreationDate="2010-12-16T23:14:05.047" Score="0" ViewCount="207" Body="&lt;p&gt;I have some hourly/daily temperature data at: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://data.barrycarter.info/&quot; rel=&quot;nofollow&quot;&gt;http://data.barrycarter.info/&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;and want to approximate it using a &quot;simple&quot; mathematical function. Notes: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you look at the mean daily temperature graph for KABQ, it looks &#xA;like a sine wave with a period of one year. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Similarly, the hourly temperature for a given day in KABQ also looks &#xA;like a sine wave with a period of one day. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, closer inspection (Fourier analysis) shows that they're &#xA;not. There are fairly strong components of frequency 2/year and 3/year &#xA;for the daily temperature, and the hourly temperature also has strong &#xA;non-single-period terms. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a function that describes the daily mean temperature and (a &#xA;separate function) the hourly mean temperature somewhat accurately?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The function would have constants (parameters) that would change for &#xA;non-KABQ locations. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I realize I can keep taking more Fourier terms to increase accuracy, &#xA;but I was hoping for something more elegant. For example, maybe the &#xA;graph is a parametrized version of sin^2(x) or some other &#xA;&quot;well-known&quot; function. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: Below's a hopefully better example of what I mean. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You take the Fourier transform of some data and the first 15 terms are: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;2.82831 &#xA;3.36979 + 0.322402 I &#xA;3.79182 + 2.34689 I &#xA;0.73987 + 4.71806 I &#xA;-3.16205 + 0.709054 I &#xA;2.76656 - 1.86274 I &#xA;-0.762084 + 3.94561 I &#xA;1.98004 - 2.6786 I &#xA;-2.51665 + 2.21764 I &#xA;3.6131 + 2.29278 I &#xA;2.79673 - 2.55569 I &#xA;-0.455474 - 2.7534 I &#xA;-1.51359 - 1.18634 I &#xA;-1.31561 - 0.177807 I &#xA;-0.928778 + 0.218559 I &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You immediately recognize this as the Fourier series for cos(x^2), and &#xA;model your data appropriately. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this case, using just the first few Fourier terms for approximation &#xA;is a bad idea, since the data follows a much &quot;better&quot; pattern. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;That's what I want. Fourier series is one of many methods to analyze &#xA;data. I want my approximation to be method neutral in some sense. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I guess I'm sort of asking: given a Fourier series, how do I find a &#xA;nearby &quot;well known&quot; function. &lt;/p&gt;&#xA;" OwnerUserId="1566" LastEditorUserId="1566" LastEditorDisplayName="" LastEditDate="2010-12-23T23:57:56.107" LastActivityDate="2010-12-23T23:57:56.107" ClosedDate="2011-04-24T18:13:33.177" Title="Approximating temperature data beyond Fourier series" Tags="&lt;curve-fitting&gt;" CommentCount="7" />
  <row Id="5583" PostTypeId="2" ParentId="5572" CreationDate="2010-12-17T04:14:41.050" Score="1" ViewCount="" Body="&lt;p&gt;Hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Excuse my previous post. I now see that you are not referring to 9000 different covariates. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I have written does not apply to your situation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sincerest apologies.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Paul &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a lot of discussion about matching and dimensionality reduction on pre-treatment covariates that may be worthwhile examining - i.e. propensity weighting via logistic regression and establishing balance on the pre-treatment covariates vis a vis different matching approaches.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please refer to the following ............... &lt;a href=&quot;http://gking.harvard.edu/matchit&quot; rel=&quot;nofollow&quot;&gt;http://gking.harvard.edu/matchit&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This approach is easily executed in r, but the number of variables you have would be looking to use would be very unlikely to work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cheers Paul&lt;/p&gt;&#xA;" OwnerUserId="2238" LastEditorUserId="2238" LastEditorDisplayName="" LastEditDate="2010-12-17T05:49:56.120" LastActivityDate="2010-12-17T05:49:56.120" CommentCount="1" />
  <row Id="5584" PostTypeId="2" ParentId="5572" CreationDate="2010-12-17T05:13:59.897" Score="3" ViewCount="" Body="&lt;p&gt;This is not a complete answer, but just a few thoughts:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;More pre-treatment measures should increase the reliability of your measurement of baseline differences. Increasing reliability of measuring baseline differences should increase your statistical power in detecting group differences (assuming a real effect exists) using the pre-post control design.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;9000 pre-treatment measures is a lot. Such a design would usually imply that you are interested in the temporal dynamics of some phenomena. Nonetheless, if you are just using measurements as an indicator of baseline differences, then there would be a number of strategies for incorporating this into your model. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The simplest strategy would be to take the mean for each participant.&lt;/li&gt;&#xA;&lt;li&gt;If there is trend in participant data, then an estimate of the individual's score just before the intervention may be more of interest. &lt;/li&gt;&#xA;&lt;li&gt;Even more sophisticated would be to develop a model for each individual of what their score would be on the dependent variable following the intervention based on some projection using the pre-treatment measures. This might be more relevant if there was some form of seasonal or other systematic effect operating in different ways for different individuals.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You may also want to read this &lt;a href=&quot;http://stats.stackexchange.com/questions/3466/best-practice-when-analysing-pre-post-treatment-control-designs&quot;&gt;earlier question on strategies for analysing such designs&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-12-17T05:13:59.897" CommentCount="1" />
  <row Id="5585" PostTypeId="2" ParentId="604" CreationDate="2010-12-17T05:45:18.537" Score="1" ViewCount="" Body="&lt;p&gt;I am having troubling following your reasoning, but here are some things you should consider.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Generally, the harder you fit a model to your training data, the worse the model will perform on independent validation data sets. By over-fitting the model to the training set, you risk capturing predictor-response relationships that are particular to the training set you are using. These relationships are likely due to random chance. When building a model for classification, you want to only capture the predictor-response relationships that are common to all training sets. This is requires careful selection of the right size model (big enough to capture the true predictor-response relationship, small enough to not to overfit to your particular training set.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, the fact that a linear regression gives an R^2 of 1 doesn't mean much. For example, I can generate a 101 X 100 matrix of N(0,1) observations, take the first column to be the &quot;response&quot;, and the other 100 columns to be &quot;predictors.&quot; This will give me an R^2 value of 1, even though the &quot;response&quot; and &quot;predictors&quot; are independent (assuming the rows/columns are linearly independent, which they will be with probability 1 if they are all N(0,1) observations.)  So in your n=5 observation, p=20 predictor case, you can choose any 5 predictors and get a perfect fit. R^2 is generally a pretty poor model assessment metric. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, unless you are certain the conditional distribution of the predictors is multivariate normal and that the predictors have a common covariance matrix, LDA may not be the best choice here. There are several better nonparametric/semiparametric methods available.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maybe you can clarify your post a little bit to get a better response.&lt;/p&gt;&#xA;" OwnerUserId="2144" LastActivityDate="2010-12-17T05:45:18.537" />
  <row Id="5586" PostTypeId="1" CreationDate="2010-12-17T06:31:41.993" Score="1" ViewCount="92" Body="&lt;p&gt;I have several OLS models with robust s.e.'s that predict an outcome variable Y&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Model 1&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Y=B_0 +B_1X_1$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Model 2&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Y=B_0 + B_1X_1 + B_2X_2$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Model 3&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Y=B_0 +B_1X_1 + B_2X_2 +B_3X_3$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am interested in giving an average effect for $B_1$ across Models 1-3 with an accompanied 95% CI.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can I just take the average of $B_1$'s across Models 1-3 and the average of standard errors to construct my confidence interval?  What is this called?     &lt;/p&gt;&#xA;" OwnerUserId="2414" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-17T12:15:36.593" LastActivityDate="2010-12-17T12:15:36.593" Title="Average effect across multiple linear models?" Tags="&lt;regression&gt;&lt;statistical-significance&gt;&lt;best-practices&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="5587" PostTypeId="2" ParentId="4987" CreationDate="2010-12-17T06:44:15.940" Score="0" ViewCount="" Body="&lt;p&gt;Hi, Legend.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As it happens, I'm working on a very similar problem, with about the same experience level. Have you taken into account that your data appear to be left- or zero-truncated? Also, in addition to negative-binomial, have you considered the beta binomial or geometric distributions for your count data?&lt;/p&gt;&#xA;" OwnerUserId="2415" LastActivityDate="2010-12-17T06:44:15.940" />
  <row Id="5588" PostTypeId="2" ParentId="5514" CreationDate="2010-12-17T06:57:01.517" Score="1" ViewCount="" Body="&lt;p&gt;Conditional logit models for unit $i$ in group $j$ of size $G$, $P(Y_{ij}=1|\sum_{k=1}^GY_{gj}=M)$ for some $0&amp;lt;M&amp;lt;G$. So, suppose you have a group of size 2 in which there is one success.  Then, group member number 1's contribution to the likelihood is given by $A^{Y_{1j}}B^{1-Y_{1j}}$, where,&lt;br&gt;&#xA;$$&#xD;&#xA;\begin{matrix}&#xD;&#xA;A &amp;amp; = P(Y_{1j}=1|Y_{1j}+Y_{2j}=1) = \frac{P(Y_{ij}=1 \cap Y_{1j}+Y_{2j}=1)}{P(Y_{1j}+Y_{2j}=1)} \\ &#xD;&#xA;&amp;amp; = \frac{P(Y_{1j}=1 \cap Y_{2j}=0)}{P(Y_{1j}=1 \cap Y_{2j}=0)+P(Y_{1j}=0 \cap Y_{2j}=1)}.&#xD;&#xA;\end{matrix}&#xD;&#xA;$$&#xD;&#xA;The term $B$ is derived in a similar manner.  Specifying the response probabilities in terms of a logit model,&#xA;$$&#xD;&#xA;\begin{matrix}&#xD;&#xA;A &amp;amp; = \frac{\frac{\exp(\alpha + \beta&amp;#39;x_{1j})}{1+\exp(\alpha + \beta&amp;#39;x_{1j})}\frac{1}{1+\exp(\alpha + \beta&amp;#39;x_{2j})}}{\frac{\exp(\alpha + \beta&amp;#39;x_{1j})}{1+\exp(\alpha + \beta&amp;#39;x_{1j})}\frac{1}{1+\exp(\alpha + \beta&amp;#39;x_{2j})} + \frac{1}{1+\exp(\alpha + \beta&amp;#39;x_{1j})}\frac{\exp(\alpha + \beta&amp;#39;x_{2j})}{1+\exp(\alpha + \beta&amp;#39;x_{2j})}}\\&#xD;&#xA;&amp;amp; = \frac{\exp(\alpha)\exp(\beta&amp;#39;x_{1j})}{\exp(\alpha)\exp(\beta&amp;#39;x_{1j})+\exp(\alpha)\exp(\beta&amp;#39;x_{2j})}\\&#xD;&#xA;&amp;amp; = \frac{\exp(\beta&amp;#39;x_{1j})}{\exp(\beta&amp;#39;x_{1j})+\exp(\beta&amp;#39;x_{2j})},&#xD;&#xA;\end{matrix}&#xD;&#xA;$$ &#xA;that is, the intercept cancels out due to the conditioning.  In fact, all additively separable group specific effects cancel in this way, which is why conditional logit is sometimes called &quot;fixed effects logit&quot; model (with &quot;fixed effects&quot; defined in the econometrics sense, not the mixed models sense).  This property is also why conditional logit is so useful for &lt;em&gt;matched&lt;/em&gt; case control data, effectively partialling out potential sources of confounding due to heterogeneity across matching strata.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The coefficients are indeed on the &quot;usual&quot; logit scale.&lt;/p&gt;&#xA;" OwnerUserId="96" LastEditorUserId="96" LastEditorDisplayName="" LastEditDate="2010-12-17T13:15:20.630" LastActivityDate="2010-12-17T13:15:20.630" CommentCount="1" />
  <row Id="5589" PostTypeId="2" ParentId="5586" CreationDate="2010-12-17T07:11:15.503" Score="2" ViewCount="" Body="&lt;p&gt;If these 3 models are estimated from independent samples, then you can assume that $\beta_1$ are independent for these 3 models. Then you can average them. The standard error of the average then will be the square root from the average of the squares of the standard errors. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However you should check if you do not have &lt;a href=&quot;http://en.wikipedia.org/wiki/Omitted-variable_bias&quot; rel=&quot;nofollow&quot;&gt;omited-variables problem&lt;/a&gt;. If you do, then one of the $\beta_1$ in your models is biased, so it should be discarded. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course if you have all the data for these 3 models, I suggest estimate Model 3 with all the data and then use the coefficient $\beta_1$ with its standard error, assuming that your model is adequate.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2010-12-17T07:11:15.503" CommentCount="3" />
  <row Id="5590" PostTypeId="2" ParentId="5571" CreationDate="2010-12-17T07:16:03.467" Score="2" ViewCount="" Body="&lt;p&gt;I believe &lt;code&gt;anova()&lt;/code&gt; in R can be used for this. Despite its name, it's a likelihood ratio test. Crawley in his &lt;a href=&quot;http://www.bio.ic.ac.uk/research/mjcraw/therbook/index.htm&quot; rel=&quot;nofollow&quot;&gt;The R Book&lt;/a&gt; has some examples of usage.&lt;/p&gt;&#xA;" OwnerUserId="144" LastActivityDate="2010-12-17T07:16:03.467" />
  <row Id="5591" PostTypeId="1" CreationDate="2010-12-17T10:36:49.853" Score="9" ViewCount="496" Body="&lt;p&gt;Roughly speaking p-value gives a probability of the observed outcome of an experiment given the hypothesis (model). Having this probability (p-value) we want to judge our hypothesis (how likely it is). But wouldn't it be more natural to calculate the probability of the hypothesis given the observed outcome?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In more details. We have a coin. We through it 20 times and we get 14 heads (14 out of 20 is what I call &quot;outcome of experiment&quot;). Now, our hypothesis is that the coin is fair (probabilities of head and tail are equal to each other). Now we calculate the p-value, that is equal to the probability to get 14 or more heads in 20 flips of coin. OK, now we have this probability (0.058) and we want to use this probability to judge our model (how is it likely that we have a fair coin). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But if we want to estimate the probability of the model, why we do not calculate the probability of the model given the experiment? Why do we calculate the probability of the experiment given the model (p-value)?&lt;/p&gt;&#xA;" OwnerUserId="2407" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-12-17T10:51:16.163" LastActivityDate="2010-12-18T08:39:00.030" Title="Why do people use p-value?" Tags="&lt;likelihood-function&gt;&lt;p-value&gt;" AnswerCount="4" CommentCount="4" FavoriteCount="7" />
  <row Id="5592" PostTypeId="2" ParentId="5591" CreationDate="2010-12-17T11:06:10.647" Score="8" ViewCount="" Body="&lt;p&gt;Computing the probability that the hypothesis is correct doesn't fit well within the frequentist definition of a probability (a long run frequency), which was adopted to avoid the supposed subjectivity of the Bayesian definition of a probability.  The truth of a particular hypothesis is not a random variable, it is either true or it isn't and has no long run frequency.  It is indeed more natural to be interested in the probability of the truth of the hypothesis, which is IMHO why p-values are often misinterpreted as the probability that the null hypothesis is true.  Part of the difficulty is that from Bayes rule, we know that to compute the posterior probability that a hypothesis is true, you need to start with a prior probability that the hypothesis is true.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A Bayesian &lt;em&gt;would&lt;/em&gt; compute the probability that the hypothesis is true, given the data (and his/her prior belief).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Essentially in deciding between frequentist and Bayesian approaches is a choice whether the supposed subjectivity of the Bayesian approach is more abhorrent than the fact that the frequentist approach generally does not give a direct answer to the question you actually want to ask - but there is room for both.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the case of asking whether a coin is fair, i.e. the probability of a head is equal to the probability of a tail, we also have an example of a hypothesis that we know in the real world is almost certainly false right from the outset.  The two sides of the coin are non-symmetric, so we should expect a slight assymetry in the probabilities of heads and tails, so if the coin &quot;passes&quot; the test, it just means we don't have enough observations to confirm what we know to be true - that the coin is very slightly biased!&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2010-12-17T11:06:10.647" CommentCount="6" />
  <row Id="5593" PostTypeId="2" ParentId="5534" CreationDate="2010-12-17T11:10:52.003" Score="0" ViewCount="" Body="&lt;p&gt;whuber, thank you so much for your careful answer. It is appreciated. The answer to my question does indeed lie in the last paragraph of your reply - now if I could only wrap my head around it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is throwing me off is the following example which I have recreated from Galen Shorack's &quot;Probability for Statisticians&quot; (page 111). Here, the Lebesgue measure of the set $[X\neq F^{-1}Y]$ is not zero. Would you agree? I am referring to the points in the interval $(2,3)$ and in $(3,3.5)$ for which the inverse transformation does not bring any points back.&#xA;Thank you again for looking into this.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&#xA;\documentclass[a4paper]{article}&#xA;% Graphics&#xA;\usepackage{graphics}&#xA;\usepackage{graphicx}&#xA;\usepackage{pstricks}&#xA;\usepackage{pst-plot}&#xA;\usepackage{pstricks-add}&#xA;\usepackage{epstopdf}&#xA;\begin{document}&#xA;&#xA;% An arbitrary CDF&#xA;\begin{figure}[htbp]&#xA;\begin{center}&#xA;\begin{psgraph}[arrows=](0,0)(-1.5,-.3)(6,1.2){.5\textwidth}{2.5cm}&#xA;    \psplot[algebraic, linecolor=black]{-1}{1}{.025*x^2+.1*x+.175} % {this goes from .1 --&gt; .3}&#xA;    \psplot[algebraic, linecolor=black]{1}{2}{-.1*x^2+.4*x+.1}  % {this goes form .4 --&gt; .5}&#xA;    \psline[linecolor=black](2,.5)(3,.5)  % this stays at .5&#xA;    \psline[linecolor=black](3,.6)(3.5,.6)  % this stays at .6&#xA;    \psplot[algebraic, linecolor=black]{3.5}{5}{  -0.1333*(x-5)^2+.9}  % this goes from .6 --&gt; .9&#xA;    \psdots[dotstyle=*](3,.6)(1,0.4)&#xA;     \end{psgraph}&#xA;\end{center}&#xA;\caption{Arbitrary CDF with discontinuities and flat sections}&#xA;\label{fig:cdf}&#xA;\end{figure}&#xA;&#xA;% The quantile function&#xA;\begin{figure}[htbp]&#xA;\begin{center}&#xA;\begin{psgraph}[arrows=](0,0)(-.3, -1.5)(1.2, 6){2cm}{5cm}&#xA;% inverses using the Matlab finverse symbolic toolbox function&#xA;\psplot[algebraic, linecolor=black]{.1}{.3}{20*(x/10 - 3/400)^(1/2) - 2} &#xA;\psline[linecolor=black](.3, 1)(.4, 1)  &#xA;\psplot[algebraic, linecolor=black]{.4}{.5}{-((x-0.5)/(-0.1))^(0.5)+2}  &#xA;\psline[linecolor=black](.5, 3)(.6, 3)&#xA;\psplot[algebraic, linecolor=black]{.6}{.9}{-((x-.9)/(-0.1333))^(0.5)+5}&#xA;\psdots[dotstyle=*](.6, 3)(.5, 2)&#xA;\end{psgraph}&#xA;\end{center}&#xA;\caption{Quantile function of CDF in figure (\protect \ref{fig:cdf})}&#xA;\label{fig:qf}&#xA;\end{figure}&#xA;\end{document}&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;PS. I could not use the comment box for the reply as I needed to use the &lt;code&gt;&amp;lt;code&lt;/code&gt;&gt;&lt;/code&gt; environment.&lt;/p&gt;&#xA;" OwnerUserId="2399" LastActivityDate="2010-12-17T11:10:52.003" CommentCount="1" />
  <row Id="5594" PostTypeId="2" ParentId="490" CreationDate="2010-12-17T11:13:27.427" Score="1" ViewCount="" Body="&lt;p&gt;If you are only interested in generalization performance, you are probably better off not performing any feature selection and using regularization instead (e.g. ridge regression).  There have been several open &lt;a href=&quot;http://clopinet.com/isabelle/Projects/NIPS2003&quot; rel=&quot;nofollow&quot;&gt;challenges&lt;/a&gt; in the machine learning community on feature selection, and methods that rely on regularization rather than feature selection generally perform at least as well, if not better.&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2010-12-17T11:13:27.427" CommunityOwnedDate="2010-12-17T11:13:27.427" />
  <row Id="5596" PostTypeId="2" ParentId="5115" CreationDate="2010-12-17T12:20:38.070" Score="3" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Adolphe_Quetelet&quot; rel=&quot;nofollow&quot;&gt;Adolphe Quetelet&lt;/a&gt; for his work on the &quot;average man&quot;, and for pioneering the use of statistics in the social sciences.  Before him, statistics were largely confined to the physical sciences (astronomy, in particular).&lt;/p&gt;&#xA;" OwnerUserId="1108" LastActivityDate="2010-12-17T12:20:38.070" CommunityOwnedDate="2010-12-17T12:20:38.070" CommentCount="1" />
  <row Id="5597" PostTypeId="1" AcceptedAnswerId="5600" CreationDate="2010-12-17T12:27:48.893" Score="15" ViewCount="259" Body="&lt;p&gt;As a biologist, many of the research projects I work on at some point involve collaboration with a statistician, whether it be for simple advice or for implementing and testing a model for my data. My statistics colleagues admit that they do a significant amount of collaboration, insomuch that the tenure review process only considers papers on which they are the first or last author.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What would make me (or any other scientist) a better collaborator? What would make it easier for you (as a statistician) to work with me? Specifically, what is one statistics concept you wish all of your scientist collaborators already understood?&lt;/p&gt;&#xA;" OwnerUserId="1973" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-17T13:32:23.420" LastActivityDate="2010-12-18T21:22:28.910" Title="Statistics collaboration" Tags="&lt;science&gt;&lt;collaboration&gt;" AnswerCount="4" CommentCount="1" FavoriteCount="1" />
  <row Id="5598" PostTypeId="2" ParentId="5597" CreationDate="2010-12-17T13:49:05.330" Score="2" ViewCount="" Body="&lt;p&gt;Having no preconceived ideas about the method you should use solely based on papers. Their ideas, logic or methods may be faulty. You want to think about your problem and use the most appropriate set of tools. This reminds me of reproducing cited information without checking the source.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, paper with methods (or logic) that differs from the rest of literature may hinder or cull a review process because &quot;it's not the norm&quot;.&lt;/p&gt;&#xA;" OwnerUserId="144" LastActivityDate="2010-12-17T13:49:05.330" />
  <row Id="5599" PostTypeId="2" ParentId="5597" CreationDate="2010-12-17T14:01:35.943" Score="6" ViewCount="" Body="&lt;p&gt;I think the concept  that few scientists grasp is this: A statistical result can really only be taken at face value when the statistical methods were chosen in advance while the experiment was being planned (or while preliminary data were collected to polish methods). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You are likely to be mislead if you first analyze the data this way, then that way, then try something else, then analyze only a subset of data, then analyze only that subset after removing an obvious outlier..... and only stop when the results match your preconceptions or has lot of asterisks. That is a fine way to generate an hypothesis, but not an appropriate way to test one.&lt;/p&gt;&#xA;" OwnerUserId="25" LastEditorUserId="25" LastEditorDisplayName="" LastEditDate="2010-12-18T03:00:48.173" LastActivityDate="2010-12-18T03:00:48.173" CommentCount="3" />
  <row Id="5600" PostTypeId="2" ParentId="5597" CreationDate="2010-12-17T15:00:51.533" Score="9" ViewCount="" Body="&lt;p&gt;My answer is from the point of view of an UK academic statistician. In particular, as an academic that gets judged on advances in statistical methodology.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;What would make me (or any other&#xA;  scientist) a better collaborator? &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;To be blunt - money. My time isn't free and I (as an academic) don't get employed to carry out standard statistical analysis. Even being first/last author on a paper that uses standard methodology is worth very little to me (in terms promotion and my personal research). Paying for my time will buy me out of administrative or teaching duties. Payment could be through a joint grant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the UK, every five or so years academics have to submit their four best papers. My papers are judged on their contribution to the statistical literature. It sucks, but that's the way it is. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now it may well be that you have a very interesting problem which would lead to advances in statistical techniques. However, just think about the size of your statistics department compared to the rest of the Uni. There probably won't be enough statisticians to go around.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In saying that, I do try and do some &quot;statistical consultancy&quot; once a year to broaden my interests and to help for teaching purposes. This year I did some &lt;a href=&quot;http://csgillespie.wordpress.com/2010/12/08/new-paper-survival-analysis/&quot; rel=&quot;nofollow&quot;&gt;survival analysis&lt;/a&gt;. However, I've never advertised this fact and I still get half dozen requests each year for help!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sorry for being so negative :(&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Specifically, what is one statistics&#xA;  concept you wish all of your scientist&#xA;  collaborators already understood?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;That statisticians do statistical research. As one of my collaborators said:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Surely there's nothing left to solve in statistics?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="8" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-12-18T21:22:28.910" LastActivityDate="2010-12-18T21:22:28.910" CommentCount="2" />
  <row Id="5601" PostTypeId="1" AcceptedAnswerId="5652" CreationDate="2010-12-17T15:25:31.870" Score="1" ViewCount="138" Body="&lt;p&gt;Hi All-&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am trying to view the output from the GBM package for boosted trees in R. Below I am fitting a single tree without any sampling in order to compare the tree to the complete dataset. First, create the data set:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  set.seed(1973)&#xA;&#xA;############## CREATE DATA#############################################&#xA;N &amp;lt;- 1000&#xA;X1 &amp;lt;- runif(N)&#xA;X2 &amp;lt;- 2*runif(N)&#xA;X3 &amp;lt;- ordered(sample(letters[1:4],N,replace=TRUE),levels=letters[4:1])&#xA;X4 &amp;lt;- factor(sample(letters[1:6],N,replace=TRUE))&#xA;X5 &amp;lt;- factor(sample(letters[1:3],N,replace=TRUE))&#xA;X6 &amp;lt;- 3*runif(N)&#xA;mu &amp;lt;- c(-1,0,1,2)[as.numeric(X3)]&#xA;&#xA;SNR &amp;lt;- 10 # signal-to-noise ratio&#xA;Y &amp;lt;- X1**1.5 + 2 * (X2**.5) + mu&#xA;sigma &amp;lt;- sqrt(var(Y)/SNR)&#xA;Y &amp;lt;- Y + rnorm(N,0,sigma)&#xA;&#xA;# introduce some missing values&#xA;X1[sample(1:N,size=500)] &amp;lt;- NA&#xA;X4[sample(1:N,size=300)] &amp;lt;- NA&#xA;&#xA;data &amp;lt;- data.frame(Y=Y,X1=X1,X2=X2,X3=X3,X4=X4,X5=X5,X6=X6)&#xA;########################################################################&#xA;&#xA;#Fit model##############################################################&#xA;gbm1 &amp;lt;- gbm(Y~X1+X2+X3+X4+X5+X6, # formula&#xA;    data=data,                   # dataset&#xA;    var.monotone=c(0,0,0,0,0,0), # -1: monotone decrease,&#xA;                                 # +1: monotone increase,&#xA;                                 #  0: no monotone restrictions&#xA;    distribution=&quot;gaussian&quot;,     # bernoulli, adaboost, gaussian,&#xA;                                 # poisson, coxph, and quantile available&#xA;    n.trees=1,                   # number of trees&#xA;    shrinkage=1,                 # shrinkage or learning rate,&#xA;                                 # 0.001 to 0.1 usually work&#xA;    interaction.depth=1,         # 1: additive model, 2: two-way interactions, etc.&#xA;    bag.fraction = 1,            # subsampling fraction, 0.5 is probably best&#xA;    train.fraction = 1,          # fraction of data for training,&#xA;                                 # first train.fraction*N used for training&#xA;    n.minobsinnode = 10,         # minimum total weight needed in each node&#xA;    keep.data=TRUE,              # keep a copy of the dataset with the object&#xA;    verbose=TRUE)                # print out progress&#xA;&#xA;###########################################################################&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Next, look at the tree. This suggests, I think, a split on X2 at the value 1.5. However, this suggests 522 records one direction and 478 the other. Looking at the data, this record split does not correspond to the counts. Any insight? Is this a bug?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pretty.gbm.tree(gbm1,i.tree = 1)&#xA;length(d&amp;lt;-subset(data, data$X2&amp;gt;1.50,3)[,1])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2040" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-12-20T19:02:28.863" LastActivityDate="2010-12-20T19:02:28.863" Title="How to view GBM package trees?" Tags="&lt;r&gt;" AnswerCount="1" />
  <row Id="5602" PostTypeId="1" CreationDate="2010-12-17T17:21:32.020" Score="1" ViewCount="140" Body="&lt;p&gt;I have data which has several properties  (metadata, as key value pairs, where the keyspace is shared over the whole dataset) per object.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I took a sample of objects and divided them in n groups according to an unknown algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What statistical methods or algorithm exists to find the relevant properties and their weight for the division to&#xA;receive a similar grouping of the data than the &quot;unknown algorithm&quot; did?&lt;/p&gt;&#xA;" OwnerUserId="2423" LastEditorUserId="2423" LastEditorDisplayName="" LastEditDate="2010-12-21T14:58:31.110" LastActivityDate="2010-12-21T14:58:31.110" Title="Methods of grouping sets of data" Tags="&lt;classification&gt;&lt;data-mining&gt;" AnswerCount="3" CommentCount="1" />
  <row Id="5603" PostTypeId="1" AcceptedAnswerId="5640" CreationDate="2010-12-17T17:24:10.373" Score="2" ViewCount="215" Body="&lt;p&gt;How does one approach the problem of modeling a &quot;birth-death process&quot; where the arrivals are dependent on the current state in the following way: if the population is above a certain point, the probability of an arrival decreases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically, I'm interested in complicating (slightly) an existing model of &quot;births&quot; that just has Poisson distributed arrivals and thinking of adding in the idea that there's a &quot;saturation point&quot; above which arrivals are less likely (waiting until the population drops back below the point). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should I be reading about nth-order Markov processes? Or should I be looking at queueing theory?&lt;/p&gt;&#xA;" OwnerUserId="446" LastEditorUserId="446" LastEditorDisplayName="" LastEditDate="2010-12-23T15:53:46.737" LastActivityDate="2010-12-24T02:13:25.680" Title="Modeling a birth-death process that is not memoryless" Tags="&lt;stochastic-processes&gt;" AnswerCount="3" CommentCount="2" FavoriteCount="1" />
  <row Id="5604" PostTypeId="1" CreationDate="2010-12-17T19:04:04.553" Score="1" ViewCount="393" Body="&lt;p&gt;Hello, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a data set that includes the number of visits to a website. Here are some descriptive statistics for my data&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Median: 4&lt;br&gt;&#xA;Mean: 14.1352&lt;br&gt;&#xA;SD: 121.8119  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Clearly, there are some huge values (individuals who have visited the site thousands of times.) To remove these outliers I considered simply removing any data that falls outside more than 3.5 standard deviations from the mean. The result is that I discovered there is still a significant fat tail with my data. After removal of data that falls outside more than 3.5 standard deviations my descriptive statistics adjust to&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Median: 4&lt;br&gt;&#xA;Mean: 10.2201&lt;br&gt;&#xA;SD: 19.7492  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also explored using a winsorized mean but again since my data is asymmetric I feel like my descriptive statistics are biased. Is there a method that I can use to reevaluate my data to provide descriptive statistics that would represent a ‘majority’ of the population? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As I understand the concept of bootstrapping, I could sample my population and then resample and generation thousands of populations that may represent my population differently based on the resample of my original sample population. Would this method be appropriate? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any other ideas or direction?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance for any help. Any references or examples with R would be very much appreciated as well. &lt;/p&gt;&#xA;" OwnerUserId="2427" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-17T22:46:49.800" LastActivityDate="2010-12-18T20:51:52.010" Title="Removing outliers from asymmetric data" Tags="&lt;r&gt;&lt;outliers&gt;&lt;descriptive-statistics&gt;" AnswerCount="3" CommentCount="2" FavoriteCount="1" />
  <row Id="5605" PostTypeId="2" ParentId="5591" CreationDate="2010-12-17T19:10:47.387" Score="0" ViewCount="" Body="&lt;p&gt;&quot;Roughly speaking p-value gives a probability of the observed outcome of an experiment given the hypothesis (model).&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;but it doesn't.  Not even roughly - this fudges an essential distinction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The model is not specified, as Raskolnikov points out, but let's assume you mean a binomial model (independent coin tosses, fixed unknown coin bias).  The hypothesis is the claim that the relevant parameter in this model, the bias or probability of heads, is 0.5.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Having this probability (p-value) we want to judge our hypothesis (how likely it is)&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We may indeed want to make this judgement but a p-value will not (and was not designed to) help us do so.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;But wouldn't it be more natural to calculate the probability of the hypothesis given the observed outcome?&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Perhaps it would.  See all the discussion of Bayes above.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;[...] Now we calculate the p-value, that is equal to the probability to get 14 or more heads in 20 flips of coin.  OK, now we have this probability (0.058) and we want to use this probability to judge our model (how is it likely that we have a fair coin).&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;'of our hypothesis, assuming our model to be true', but essentially: yes.  Large p-values indicate that the coin's behaviour is consistent with the hypothesis that it is fair.  (They are also typically consistent with the hypothesis being false but so close to being true we do not have enough data to tell; see 'statistical power'.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;But if we want to estimate the probability of the model, why we do not calculate the probability of the model given the experiment? Why do we calculate the probability of the experiment given the model (p-value)?&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We actually don't calculate the probability of the experimental results given the hypothesis in this setup.  After all, the probability is only about 0.176 of seeing exactly 10 heads when the hypothesis is true, and that's the &lt;em&gt;most&lt;/em&gt; probable value.  This isn't a quantity of interest at all.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is also relevant that we don't usually estimate the probability of the model either.  Both frequentist and Bayesian answers typically assume the model is true and make their inferences about its parameters.  Indeed, not all Bayesians &lt;em&gt;would&lt;/em&gt; even in principle be interested in the probability of the model, that is: the probability that the whole situation was well modelled by a binomial distribution.  They might do a lot of model checking, but never actually ask how likely the binomial was in the space of other possible models.  Bayesians who care about Bayes Factors are interested, others not so much.&lt;/p&gt;&#xA;" OwnerUserId="1739" LastActivityDate="2010-12-17T19:10:47.387" CommentCount="1" />
  <row Id="5607" PostTypeId="2" ParentId="5534" CreationDate="2010-12-17T20:13:50.217" Score="3" ViewCount="" Body="&lt;p&gt;It’s much easier to simultaneously construct $X_i$ and $Y_i$ having the desired properties,&#xA;by first letting $Y_i$ be i.i.d. Uniform$[0,1]$ and then taking $X_i = F^{-1}(Y_i)$.  This is the basic method for generating random variables with arbitrary distributions.&#xA;The other direction, where you are first given $X_i$ and then asked to construct $Y_i$, is more difficult, but is still possible for all distributions.  You just have to be careful with how you define $Y_i$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Attempting to define $Y_i$ as $Y_i = F(X_i)$ fails to produce uniformly distributed $Y_i$ when $F$ has jump discontinuities.  You have to spread the point masses in the distribution of $X_i$ across the the gaps created by the jumps.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let $$D = \{x : F(x) \neq \lim_{z \to x^-} F(z)\}$$ denote the set of jump discontinuities of $F$.  ($\lim_{z\to x^-}$ denotes the limit from the left.  All distributions functions are right continuous, so the main issue is left discontinuities.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let $U_i$ be i.i.d. Uniform$[0,1]$ random variables, and define&#xA;$$Y_i = &#xD;&#xA;\begin{cases}&#xD;&#xA;F(X_i), &amp;amp; \text{if }X_i \notin D \\&#xD;&#xA;U_i F(X_i) + (1-U_i) \lim_{z \to X_i^-} F(z), &amp;amp; \text{otherwise.}&#xD;&#xA;\end{cases}&#xD;&#xA;$$&#xD;&#xA;The second part of the definition fills in the gaps uniformly.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The quantile function $F^{-1}$ is not a genuine inverse when $F$ is not 1-to-1. Note that if $X_i \in D$ then $F^{-1}(Y_i) = X_i$, because the pre-image of the gap is the corresponding point of discontinuity.  For the continuous parts where $X_i \notin D$, the flat sections of $F$ correspond to intervals where $X_i$ has 0 probability so they don’t really matter when considering $F^{-1}(Y_i)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second part of your question follows from similar reasoning after the first part which asserts that $X_i = F^{-1}(Y_i)$ with probability 1.  The empirical CDFs are defined as &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$G_n(y) = \frac{1}{n} \sum_{i=1}^n 1_{\{Y_i \leq y\}}$$&#xD;&#xA;$$F_n(x) = \frac{1}{n} \sum_{i=1}^n 1_{\{X_i \leq x\}}$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;so&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;\begin{align}&#xD;&#xA;G_n(F(x)) &#xD;&#xA;&amp;amp;= \frac{1}{n} \sum_{i=1}^n 1_{\{Y_i \leq F(x) \}} &#xD;&#xA;= \frac{1}{n} \sum_{i=1}^n 1_{\{F^{-1}(Y_i) \leq x \}} &#xD;&#xA;= \frac{1}{n} \sum_{i=1}^n 1_{\{X_i \leq x \}} &#xD;&#xA;= F_n(x)&#xD;&#xA;\end{align}&#xD;&#xA;$$&#xD;&#xA;with probability 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It should be easy to convince yourself that $Y_i$ has Uniform$[0,1]$ distribution by looking at pictures.  Doing so rigorously is tedious, but can be done.  We have to verify that $P(Y_i \leq u) = u$ for all $u \in (0,1)$.  Fix such $u$ and let $x^* = \inf\{x : F(x) \geq u \}$ — this is just the value of quantile function at $u$.  It’s defined this way to deal with flat sections.  We’ll consider two separate cases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First suppose that $F(x^*) = u$.  Then &#xA;$$&#xD;&#xA;Y_i \leq u&#xD;&#xA;\iff Y_i \leq F(x^*)&#xD;&#xA;\iff F(X_i) \leq F(x^*).&#xD;&#xA;$$&#xD;&#xA;Since $F$ is a non-decreasing function and $F(x^*) = u$,&#xA;$$&#xD;&#xA;F(X_i) \leq F(x^*) \iff X_i \leq x^* .&#xD;&#xA;$$&#xD;&#xA;Thus, &#xA;$$&#xD;&#xA;P[Y_i \leq u]&#xD;&#xA;= P[X_i \leq x^*]&#xD;&#xA;= F(x^*)&#xD;&#xA;= u .&#xD;&#xA;$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now suppose that $F(x^*) \neq u$.  Then necessarily $F(x^*) &amp;gt; u$, and $u$ falls inside one of the gaps.  Moreover, $x^* \in D$, because otherwise $F(x^*) = u$ and we have a contradiction.&#xA;Let $u^* = F(x^*)$ be the upper part of the gap.  Then by the previous case, &#xA;$$&#xD;&#xA;\begin{align}&#xD;&#xA;P[Y_i \leq u]&#xD;&#xA;&amp;amp;= P[Y_i \leq u^*] - P[u &amp;lt; Y_i \leq u^*]\\&#xD;&#xA;&amp;amp;= u^* - P[u &amp;lt; Y_i \leq u^*].&#xD;&#xA;\end{align}&#xD;&#xA;$$&#xD;&#xA;By the way $Y_i$ is defined,  $P(Y_i = u^*) = 0$ and &#xA;$$&#xD;&#xA;\begin{align}&#xD;&#xA;P[u &amp;lt; Y_i \leq u^*] &#xD;&#xA;&amp;amp;= P[u &amp;lt; Y_i &amp;lt; u^*] \\&#xD;&#xA;&amp;amp;= P[u &amp;lt; Y_i &amp;lt; u^* , X_i = x^*] \\&#xD;&#xA;&amp;amp;= u^* - u .&#xD;&#xA;\end{align}&#xD;&#xA;$$&#xD;&#xA;Thus, $P[Y_i \leq u] = u$.&lt;/p&gt;&#xA;" OwnerUserId="1670" LastActivityDate="2010-12-17T20:13:50.217" CommentCount="2" />
  <row Id="5608" PostTypeId="2" ParentId="5591" CreationDate="2010-12-17T20:55:13.000" Score="5" ViewCount="" Body="&lt;p&gt;As a former academic who moved into practice, I'll take a shot. People use p-values because they are useful. You can't see it in textbooky examples of coin flips. Sure they're not really solid foundationally, but maybe that is not as necessary as we like to think when we're thinking academically.&#xA;In the world of data,  we're surrounded by a literally infinite number of possible things to look into next. With p-value computations all you need as an idea of what is uninteresting and a numerical heuristic for what sort of data might be interesting (well, plus a probability model for uninteresting). Then individually or collectively we can scan things pretty simple, rejecting the bulk of the uninteresting. The p-value allows us to say &quot;If I don't put much priority on thinking about  this otherwise, this data gives me no reason to change&quot;.&#xA;I agree p-values can be misinterpreted and overinterpreted, but they're still an important part of statistics. &lt;/p&gt;&#xA;" OwnerUserId="2134" LastActivityDate="2010-12-17T20:55:13.000" />
  <row Id="5609" PostTypeId="2" ParentId="5399" CreationDate="2010-12-17T21:44:43.637" Score="2" ViewCount="" Body="&lt;p&gt;There are no strong results and it does not depend on Gaussianity.  In the case where $x_1$ and $x_2$ are scalars, you are asking if knowing the variance of the variables implies something about their covariance.  whuber’s answer is right.  The Cauchy-Schwarz Inequality and positive semidefiniteness constrain the possible values.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The simplest example is that the squared covariance of a pair of variables can never exceed the product of their variances.  For covariance matrices there is a generalization.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider the block partitioned covariance matrix of $[x_1 \ x_2]$,&#xA;$$&#xD;&#xA;\left[&#xD;&#xA;\begin{array}{cc}&#xD;&#xA;\Sigma_{11} &amp;amp; \Sigma_{12} \\&#xD;&#xA;\Sigma_{21} &amp;amp; \Sigma_{22}&#xD;&#xA;\end{array}&#xD;&#xA;\right].&#xD;&#xA;$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then&#xA;$$\Vert \Sigma_{12} \Vert_q^2 \leq \Vert \Sigma_{11} \Vert_q \Vert \Sigma_{22} \Vert_q$$&#xD;&#xA;for all &lt;a href=&quot;http://en.wikipedia.org/wiki/Schatten_norm&quot; rel=&quot;nofollow&quot;&gt;Schatten q-norms&lt;/a&gt;.  Positive (semi)definiteness of the covariance matrix also provides the constraint that &#xA;$$&#xD;&#xA;\Sigma_{11} - \Sigma_{12} \Sigma_{22}^{-1} \Sigma_{21}&#xD;&#xA;$$&#xD;&#xA;must be positive (semi)definite.  $\Sigma_{22}^{-1}$ is the (Moore-Penrose) inverse of $\Sigma_{22}$.&lt;/p&gt;&#xA;" OwnerUserId="1670" LastEditorUserId="1670" LastEditorDisplayName="" LastEditDate="2010-12-17T22:05:36.543" LastActivityDate="2010-12-17T22:05:36.543" />
  <row Id="5610" PostTypeId="2" ParentId="4364" CreationDate="2010-12-17T22:00:34.923" Score="2" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Stein%27s_lemma&quot; rel=&quot;nofollow&quot;&gt;Stein’s Lemma&lt;/a&gt; provides a very useful characterization.  $Z$ is standard Gaussian iff &#xA;$$E f’(Z) = E Z f(Z)$$&#xD;&#xA;for all absolutely continuous functions $f$ with $E|f’(Z)| &amp;lt; \infty$.&lt;/p&gt;&#xA;" OwnerUserId="1670" LastEditorUserId="1670" LastEditorDisplayName="" LastEditDate="2010-12-17T23:24:38.617" LastActivityDate="2010-12-17T23:24:38.617" CommunityOwnedDate="2010-12-17T22:00:34.923" CommentCount="2" />
  <row Id="5611" PostTypeId="2" ParentId="411" CreationDate="2010-12-17T22:33:50.943" Score="2" ViewCount="" Body="&lt;p&gt;I think you have to consider the theoretical vs applied advantages of the different notions of distance.  Mathematically natural objects don’t necessarily translate well into application. &#xA;Kolmogorov-Smirnov is the most well-known for application, and is entrenched in testing for goodness of fit.  I suppose that one of the reasons for this is that when the underlying distribution $F$ is continuous the distribution of the statistic is independent of $F$.&#xA;Another is that it can be easily inverted to give confidence bands for the CDF.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But it’s often used in a different way where $F$ is estimated by $\hat{F}$, and the test statistic takes the form&#xA;$$\sup_x | F_n(x) - \hat{F}(x)|.$$&#xD;&#xA;The interest is in seeing how well $\hat{F}$ fit the data and acting as if $\hat{F} = F$, even though the asymptotic theory does not necessarily apply.&lt;/p&gt;&#xA;" OwnerUserId="1670" LastEditorUserId="1670" LastEditorDisplayName="" LastEditDate="2010-12-17T22:42:22.920" LastActivityDate="2010-12-17T22:42:22.920" />
  <row Id="5612" PostTypeId="2" ParentId="5604" CreationDate="2010-12-17T23:02:12.257" Score="0" ViewCount="" Body="&lt;p&gt;Dason, I am attempted to develop a clean data set because I do know in fact that some of my data has been tainted. More detail is that some of the visits in my data for a particular ID are from screen scrapers (IP filtering is not working because I do not own the data nor would it be retroactive)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For this reason I am attempting to use a statistical method to scrub out the bad data and create a usable data set. Mostly just looking for clean descriptive statistics of my data.  &lt;/p&gt;&#xA;" OwnerUserId="2432" LastActivityDate="2010-12-17T23:02:12.257" CommentCount="1" />
  <row Id="5613" PostTypeId="2" ParentId="5591" CreationDate="2010-12-18T05:56:43.220" Score="2" ViewCount="" Body="&lt;p&gt;Your question is a great example of frequentist reasoning and is, actually quite natural.  I've used this example in my classes to demonstrate the nature of hypothesis tests.  I ask for a volunteer to predict the results of a coin flip.  No matter what the result, I record a &quot;correct&quot; guess.  We do this repeatedly until the class becomes suspicious.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, they have a null model in their head.  They assume the coin is fair.  Given that assumption of 50% correct when is everything is fair, every successive correct guess arouses more suspicion that the fair coin model is incorrect.  A few correct guesses and they accept the role of chance.  After 5 or 10 correct guesses, the class always begins to suspect that the chance of a fair coin is low.  Thus it is with the nature of hypothesis testing under the frequentist model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is a clear and intuitive representation of the frequentist take on hypothesis testing.  It is the probability of the observed data given that the null is true.  It is actually quite natural as demonstrated by this easy experiment.  We take it for granted that the model is 50-50 but as evidence mounts, I reject that model and suspect that there is something else at play.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, if the probability of what I observe is low given the model I assume (the p-value) then I have some confidence in rejecting my assumed model.  Thus, a p-value is a useful measure of evidence against my assumed model taking into account the role of chance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A disclaimer: I took this exercise from a long forgotten article in, what I recall, was one of the ASA journals.&lt;/p&gt;&#xA;" OwnerUserId="485" LastEditorUserId="485" LastEditorDisplayName="" LastEditDate="2010-12-18T06:11:01.040" LastActivityDate="2010-12-18T06:11:01.040" CommentCount="1" />
  <row Id="5614" PostTypeId="1" CreationDate="2010-12-18T06:56:55.773" Score="5" ViewCount="257" Body="&lt;p&gt;While it is easier to use the Pearson chi-square/Cressie-Read type test, I would like to test the equality of proportions in $k$ categories across two groups using a Kolmogorov-Smirnov type test of the form proposed  by &lt;a href=&quot;http://www.jstor.org/stable/1268631&quot; rel=&quot;nofollow&quot;&gt;Pettitt  &amp;amp; Stephens (1977)&lt;/a&gt; (see also &lt;a href=&quot;http://www4.gu.edu.au:8080/adt-root/uploads/approved/adt-QGU20031006.143823/public/03Chapter2.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In particular as the authors of that paper point out, it may have some power against trending alternatives. So their one-sample nominal/categorical Kolmogorov-Smirnov test has the form:&#xA;$$ D_n = \sup_{\pi}\sup_{1 \leq j \leq k}\vert \sum_{i=1}^j(f_{exp,\pi(i)}-f_{obs,\pi(i)})\vert$$&#xD;&#xA;where $\pi$ is a permutation of the order of the categories, $f_{.,i}$ are the observed and expected frequencies (or equivalently, proportion of observations) in category $i$. This can be written equivalently as:&#xA;$$ D_n = \frac{1}{2} \sum_{i=1}^k\vert f_{exp,i}-f_{obs,i} \vert$$&#xD;&#xA;I would like to extend this to a two-sample case using a randomising/permutation procedure, such:&#xA;$$ D_n^{(r)} = \frac{1}{2} \sum_{i=1}^k\vert f^{(r)}_{\text{group1},i}-f^{(r)}_{\text{group2},i} \vert,\, r=1,\dots,R $$&#xD;&#xA;where $.^{(r)}$ denotes a statistic calculated based on the $r^{\text{th}}$ permutation of the categorical variable. Reject if the value of the original statistic is larger than the value of $95\%$ of the permuted statistics.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any comments as to the pros/cons/validity of such a procedure are very welcome. Thanks.&lt;/p&gt;&#xA;" OwnerUserId="2399" LastActivityDate="2010-12-19T06:48:42.843" Title="Two-sample permutation Kolmogorov-Smirnov tests" Tags="&lt;hypothesis-testing&gt;" AnswerCount="1" />
  <row Id="5615" PostTypeId="2" ParentId="5604" CreationDate="2010-12-18T07:14:17.327" Score="3" ViewCount="" Body="&lt;p&gt;Don't remove any outliers until you explore the data a bit further. I suggest that you should do a log transform on the data and see whether it becomes more nearly symmetrical--the outliers may not be as extreme as you think. (Log values make perfect sense if there is some sort of power law at play.)&lt;/p&gt;&#xA;" OwnerUserId="1679" LastActivityDate="2010-12-18T07:14:17.327" />
  <row Id="5617" PostTypeId="1" AcceptedAnswerId="5618" CreationDate="2010-12-18T13:50:36.403" Score="5" ViewCount="618" Body="&lt;p&gt;Let's say we have two factors (A and B), each with two levels (A1, A2  and  B1, B2) and a response variable (y).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The when performing a two way ANOVA of the type:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;y~A+B+A*B&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;We are testing three null hypothesis:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;There is no difference in the means&#xA;of factor A &lt;/li&gt;&#xA;&lt;li&gt;There is no difference&#xA;in means of factor B &lt;/li&gt;&#xA;&lt;li&gt;There is no interaction between factors A and B&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;When written down, the first two hypothesis are easy to formulate (for 1 it is $H_0:\; \mu_{A1}=\mu_{A2}$)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But how should hypothesis 3 be formulated?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;edit&lt;/strong&gt;: and how would it be formulated for the case of more then two levels?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="253" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-12-18T16:12:53.090" LastActivityDate="2010-12-20T09:24:11.373" Title="What is the NULL hypothesis for interaction in a two-way ANOVA?" Tags="&lt;hypothesis-testing&gt;&lt;anova&gt;" AnswerCount="2" CommentCount="3" FavoriteCount="1" />
  <row Id="5618" PostTypeId="2" ParentId="5617" CreationDate="2010-12-18T14:14:19.383" Score="6" ViewCount="" Body="&lt;p&gt;An interaction tells us that the levels of factor A have different effects based on what level of factor B you're applying.  So we can test this through a linear contrast.  Let C = (A1B1 - A1B2) - (A2B1 - A2B2)  where A1B1 stands for the mean of the group that received A1 and B1 and so on.  So here we're looking at A1B1 - A1B2 which is the effect that factor B is having when we're applying A1.  If there is no interaction this should be the same as the effect B is having when we apply A2: A2B1 - A2B2.  If those are the same then their difference should be 0 so we could use the tests:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$H_0: C = 0\quad\text{vs.}\quad H_A: C \neq 0.$&lt;/p&gt;&#xA;" OwnerUserId="1028" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-12-18T15:16:35.160" LastActivityDate="2010-12-18T15:16:35.160" CommentCount="2" />
  <row Id="5619" PostTypeId="1" CreationDate="2010-12-18T15:33:30.063" Score="2" ViewCount="453" Body="&lt;p&gt;I know that $r$ is itself a measure of the effect size, but I would like to know if using Spearman's rank test I can argue that the relation between X and Y is significant with $r = 0.33$ and that the effect is medium, as I do with Pearson test.&lt;/p&gt;&#xA;" OwnerUserId="2438" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-12-18T16:07:11.407" LastActivityDate="2010-12-21T17:02:01.640" Title="Effect size of Spearman's rank test" Tags="&lt;correlation&gt;&lt;effect-size&gt;" AnswerCount="2" CommentCount="3" />
  <row Id="5620" PostTypeId="1" AcceptedAnswerId="5621" CreationDate="2010-12-18T15:34:36.363" Score="0" ViewCount="76" Body="&lt;p&gt;I'm reading &quot;&lt;a href=&quot;http://www-stat.stanford.edu/~hastie/local.ftp/Springer/ESLII_print4.pdf&quot; rel=&quot;nofollow&quot;&gt;The Elements of Statistical Learning&lt;/a&gt;&quot; and early on there are references to &lt;strong&gt;p-vectors&lt;/strong&gt; (page 10) and &lt;strong&gt;K-vectors&lt;/strong&gt; (page 12). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What exactly is meant by a p-vector and K-vector?&lt;/p&gt;&#xA;" OwnerUserId="1212" LastActivityDate="2010-12-19T05:18:34.387" Title="p-vector and K-vector" Tags="&lt;mathematical-statistics&gt;" AnswerCount="2" />
  <row Id="5621" PostTypeId="2" ParentId="5620" CreationDate="2010-12-18T15:53:41.117" Score="4" ViewCount="" Body="&lt;p&gt;It's merely some generic notation for a vector of $p$ attributes or variables observed on $i=1,\dots, N$ individuals, so that you can define $X^T = (X_1,X_2,\dots,X_p)$ as a vector of inputs, in the feature (or input) space (and each individual will have one such vector of observed inputs). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The $K$ notation seems to be reserved to the output space: in a classical linear regression model where $Y=X\beta$, Y is a scalar ($K=1$), whereas in a multivariate setting (say, you record weight, height, and color) it could be a $K$-vector (i.e., 3-vector with my example).&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-12-18T15:53:41.117" CommentCount="1" />
  <row Id="5622" PostTypeId="2" ParentId="5617" CreationDate="2010-12-18T17:38:48.130" Score="10" ViewCount="" Body="&lt;p&gt;I think it's important to clearly separate the hypothesis and its corresponding test. For the following, I assume a balanced, between-subjects CRF-$pq$ design (equal cell sizes, Kirk's notation: Completely Randomized Factorial design).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Y_{ijk}$ is observation $i$ in treatment $j$ of factor $A$ and treatment $k$ of factor $B$ with $1 \leq i \leq n$, $1 \leq j \leq p$ and $1 \leq k \leq q$. The model is $Y_{ijk} = \mu_{jk} + \epsilon_{i(jk)}, \quad \epsilon_{i(jk)} \sim N(0, \sigma_{\epsilon}^2)$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Design:&#xA;$\begin{array}{r|ccccc|l}&#xD;&#xA;~      &amp;amp; B 1      &amp;amp; \ldots   &amp;amp; B k      &amp;amp; \ldots   &amp;amp; B q      &amp;amp; ~\\\hline&#xD;&#xA;A 1    &amp;amp; \mu_{11} &amp;amp; \ldots   &amp;amp; \mu_{1k} &amp;amp; \ldots   &amp;amp; \mu_{1q} &amp;amp; \mu_{1.}\\&#xD;&#xA;\ldots &amp;amp; \ldots   &amp;amp; \ldots   &amp;amp; \ldots   &amp;amp; \ldots   &amp;amp; \ldots   &amp;amp; \ldots\\&#xD;&#xA;A j    &amp;amp; \mu_{j1} &amp;amp; \ldots   &amp;amp; \mu_{jk} &amp;amp; \ldots   &amp;amp; \mu_{jq} &amp;amp; \mu_{j.}\\&#xD;&#xA;\ldots &amp;amp; \ldots   &amp;amp; \ldots   &amp;amp; \ldots   &amp;amp; \ldots   &amp;amp; \ldots   &amp;amp; \ldots\\&#xD;&#xA;A p    &amp;amp; \mu_{p1} &amp;amp; \ldots   &amp;amp; \mu_{pk} &amp;amp; \ldots   &amp;amp; \mu_{pq} &amp;amp; \mu_{p.}\\\hline&#xD;&#xA;~      &amp;amp; \mu_{.1} &amp;amp; \ldots   &amp;amp; \mu_{.k} &amp;amp; \ldots   &amp;amp; \mu_{.q} &amp;amp; \mu&#xD;&#xA;\end{array}$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\mu_{jk}$ is the expected value in cell $jk$, $\epsilon_{i(jk)}$ is the error associated with the measurement of person $i$ in that cell. The $()$ notation indicates that the indices $jk$ are fixed for any given person $i$ because that person is observed in only one condition. A few definitions for the effects:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\mu_{j.} = \frac{1}{q} \sum_{k=1}^{q} \mu_{jk}$ (average expected value for treatment $j$ of factor $A$)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\mu_{.k} = \frac{1}{p} \sum_{j=1}^{p} \mu_{jk}$ (average expected value for treatment $k$ of factor $B$)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\alpha_{j} = \mu_{j.} - \mu$ (effect of treatment $j$ of factor $A$, $\sum_{j=1}^{p} \alpha_{j} = 0$)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\beta_{k} = \mu_{.k} - \mu$ (effect of treatment $k$ of factor $B$, $\sum_{k=1}^{q} \beta_{k} = 0$)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$(\alpha \beta)_{jk} = \mu_{jk} - (\mu + \alpha_{j} + \beta_{k}) = \mu_{jk} - \mu_{j.} - \mu_{.k} + \mu$&lt;br&gt;&#xA;(interaction effect for the combination of treatment $j$ of factor $A$ with treatment $k$ of factor $B$, $\sum_{j=1}^{p} (\alpha \beta)_{jk} = 0 \, \wedge \, \sum_{k=1}^{q} (\alpha \beta)_{jk} = 0)$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\alpha_{j}^{(k)} = \mu_{jk} - \mu_{.k}$&lt;br&gt;&#xA;(conditional main effect for treatment $j$ of factor $A$ within fixed treatment $k$ of factor $B$, $\sum_{j=1}^{p} \alpha_{j}^{(k)} = 0 \, \wedge \, \frac{1}{q} \sum_{k=1}^{q} \alpha_{j}^{(k)} = \alpha_{j} \quad \forall \, j, k)$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\beta_{k}^{(j)} = \mu_{jk} - \mu_{j.}$&lt;br&gt;&#xA;(conditional main effect for treatment $k$ of factor $B$ within fixed treatment $j$ of factor $A$, $\sum_{k=1}^{q} \beta_{k}^{(j)} = 0 \, \wedge \, \frac{1}{p} \sum_{j=1}^{p} \beta_{k}^{(j)} = \beta_{k} \quad \forall \, j, k)$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With these definitions, the model can also be written as:&#xA;$Y_{ijk} = \mu + \alpha_{j} + \beta_{k} + (\alpha \beta)_{jk} + \epsilon_{i(jk)}$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This allows us to express the null hypothesis of no interaction in several equivalent ways:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;$H_{0_{I}}: \sum_{j}\sum_{k} (\alpha \beta)^{2}_{jk} = 0$&lt;br&gt;&#xA;(all individual interaction terms are $0$, such that $\mu_{jk} = \mu + \alpha_{j} + \beta_{k} \, \forall j, k$. This means that treatment effects of both factors - as defined above - are additive everywhere.)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;$H_{0_{I}}: \alpha_{j}^{(k)} - \alpha_{j}^{(k&amp;#39;)} = 0 \quad \forall \, j \, \wedge \, \forall \, k, k&amp;#39; \quad (k \neq k&amp;#39;)$&lt;br&gt;&#xA;(all conditional main effects for any treatment $j$ of factor $A$ are the same, and therefore equal $\alpha_{j}$. This is essentially Dason's answer.)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;$H_{0_{I}}: \beta_{k}^{(j)} - \beta_{k}^{(j&amp;#39;)} = 0 \quad \forall \, j, j&amp;#39; \, \wedge \, \forall \, k \quad (j \neq j&amp;#39;)$&lt;br&gt;&#xA;(all conditional main effects for any treatment $k$ of factor $B$ are the same, and therefore equal $\beta_{k}$.)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;$H_{0_{I}}$: In a diagramm which shows the expected values $\mu_{jk}$ with the levels of factor $A$ on the $x$-axis and the levels of factor $B$ drawn as separate lines, the $q$ different lines are parallel.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="1909" LastEditorUserId="1909" LastEditorDisplayName="" LastEditDate="2010-12-20T09:24:11.373" LastActivityDate="2010-12-20T09:24:11.373" CommentCount="2" />
  <row Id="5623" PostTypeId="2" ParentId="5619" CreationDate="2010-12-18T17:55:17.180" Score="2" ViewCount="" Body="&lt;p&gt;With increasing sample size $n$, $r_{z} = \sqrt{n-1} r_{S}$ is asymptotically $N(0, 1)$ distributed (standard normal distribution). In R&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;rSz   &amp;lt;- sqrt(n-1) * rS&#xA;(pVal &amp;lt;- 1-pnorm(rSz))   # one-sided p-value, test for positive rank correlation&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1909" LastActivityDate="2010-12-18T17:55:17.180" />
  <row Id="5624" PostTypeId="2" ParentId="5604" CreationDate="2010-12-18T20:51:52.010" Score="1" ViewCount="" Body="&lt;p&gt;The answer you get depends on the question you ask. Dason asked what you are trying to do; I would also ask this question.  You say you know some of your data is &quot;tainted&quot; but do you know that this taint applies to the high values?  Many sorts of counts have long tails, with no bad data at all.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you just want to summarize central tendency, you could give mean, median and various trimmed means (the median is just the 50% trimmed mean, after all).  Or, perhaps better, you could supply a box plot or dot plot or density plot of the data.  Perhaps you do want to take the log; that often makes sense for counts, where you are interested in multiplicative rather than additive differences (e.g. Mary viewed the site 10 times as often as Jim, who viewed it twice as much as Bob)&lt;/p&gt;&#xA;" OwnerUserId="686" LastActivityDate="2010-12-18T20:51:52.010" />
  <row Id="5625" PostTypeId="2" ParentId="5597" CreationDate="2010-12-18T20:55:08.027" Score="6" ViewCount="" Body="&lt;p&gt;To get a good answer, you must write a good question. Answering a statistics question without context is like boxing blindfolded. You might knock your opponent out, or you might break your hand on the ring post.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What goes into a good question?&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Tell us the PROBLEM you are trying to solve. That is, the substantive problem, not the statistical aspects.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Tell us what math and statistics you know. If you’ve had one course in Introductory Stat, then it won’t make sense for us to give you an answer full of mixed model theory and matrix algebra. On the other hand, if you’ve got several courses or lots of experience, then we can assume you know some basics.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Tell us what data you have, where it came from, what is missing, how many variables, what are the Dependent Variables (DVs) and Independent Variables (IVs) – if any, and anything else we need to know about the data. Also tell us which (if any) statistical software you use.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Are you thinking of hiring a consultant, or do you just want pointers in some direction?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;THEN, and ONLY THEN tell us what you’ve tried, why you aren’t happy, and so on.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="686" LastActivityDate="2010-12-18T20:55:08.027" CommentCount="2" />
  <row Id="5626" PostTypeId="2" ParentId="452" CreationDate="2010-12-19T00:59:28.370" Score="2" ViewCount="" Body="&lt;p&gt;In &lt;em&gt;Introductory Econometrics&lt;/em&gt; (Woolridge, 2009 edition page 268) this question is addressed.  Woolridge says that when using robust standard errors, the t-statistics obtained only have distributions which are similar to the exact t-distributions if the sample size is large. If the sample size is small, the t-stats obtained using robust regression might have distributions that are not close to the t distribution and this could throw off inference.&lt;/p&gt;&#xA;" OwnerUserId="2444" LastActivityDate="2010-12-19T00:59:28.370" />
  <row Id="5627" PostTypeId="2" ParentId="4364" CreationDate="2010-12-19T02:56:46.690" Score="1" ViewCount="" Body="&lt;p&gt;Its characteristic function has the same form as its pdf.  I am not sure of another distribution which does that. &lt;/p&gt;&#xA;" OwnerUserId="2447" LastActivityDate="2010-12-19T02:56:46.690" CommunityOwnedDate="2010-12-19T02:56:46.690" />
  <row Id="5628" PostTypeId="2" ParentId="5517" CreationDate="2010-12-19T03:07:09.180" Score="3" ViewCount="" Body="&lt;p&gt;Here's (at least most of) a solution with &lt;code&gt;MCMCglmm&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First fit the equivalent intercept-variance-only model with &lt;code&gt;MCMCglmm&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(MCMCglmm)&#xA;primingHeid.MCMCglmm = MCMCglmm(fixed=RT ~ RTtoPrime * ResponseToPrime&#xA;      + Condition,&#xA;  random=~Subject+Word, data = primingHeid)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Comparing fits between &lt;code&gt;MCMCglmm&lt;/code&gt; and &lt;code&gt;lmer&lt;/code&gt;, first retrieving my hacked version of &lt;code&gt;arm::coefplot&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;source(url(&quot;http://www.math.mcmaster.ca/bolker/R/misc/coefplot_new.R&quot;))&#xA;## combine estimates of fixed effects and variance components&#xA;pp &amp;lt;- as.mcmc(with(primingHeid.MCMCglmm,cbind(Sol,VCV)))&#xA;## extract coefficient table&#xA;cc1 &amp;lt;- coeftab(primingHeid.MCMCglmm,ptype=c(&quot;fixef&quot;,&quot;vcov&quot;))&#xA;## strip fixed/vcov indicators to make names match with lmer output&#xA;rownames(cc1) &amp;lt;- gsub(&quot;(Sol|VCV).&quot;,&quot;&quot;,rownames(cc1))&#xA;## fixed effects -- v. similar&#xA;coefplot(list(cc1[1:5,],primingHeid.lmer))&#xA;## variance components -- quite different.  Worth further exploration?&#xA;coefplot(list(cc1[6:8,],coeftab(primingHeid.lmer,ptype=&quot;vcov&quot;)),&#xA;    xlim=c(0,0.16),&#xA;         cex.pts=1.5)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now try it with random slopes:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;primingHeid.rs.MCMCglmm = MCMCglmm(fixed=RT ~ RTtoPrime * ResponseToPrime&#xA; + Condition,&#xA;  random=~Subject+Subject:Condition+Word, data = primingHeid)        &#xA;summary(primingHeid.rs.MCMCglmm)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This does give some sort of &quot;MCMC p-values&quot; ... you'll have to explore for yourself and see whether the whole thing makes sense ...&lt;/p&gt;&#xA;" OwnerUserId="2126" LastActivityDate="2010-12-19T03:07:09.180" CommentCount="1" />
  <row Id="5629" PostTypeId="1" AcceptedAnswerId="5745" CreationDate="2010-12-19T04:20:06.560" Score="1" ViewCount="204" Body="&lt;p&gt;Is there a software package that supports, or could support, through operator overloading or extensions, code such as the following;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x = rand_arr(10) ; array 10 elements long&#xA;y = rand_arr(10)&#xA;z = x + y ; elemental addition (z[0]=x[0]+y[0];z[1]=x[1]+y[1],...)&#xA;print z&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If the above, or something similar, is valid in this hypothetical language, does it also support, or could it support via operator overloading / custom classes / extensions / whatever, the following additional syntax:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x.err[*] = 0.1 % a fixed error&#xA;y.err = rand_arr(10) % a dynamic noisy error range&#xA;z = x + y&#xA;print z.err&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Or whatever the implementation dictates, but the point being that the error is tracked for me through the &lt;code&gt;z = x+y&lt;/code&gt; without work on my part.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I could imagine default support as per a standard (or customizable) function for &lt;code&gt;+-/*e^&lt;/code&gt; and &lt;code&gt;log()&lt;/code&gt; operators, and support for extensions or hooks to add support for additional functions (system, and user) such as &lt;code&gt;z = smooth(x,3)&lt;/code&gt; and &lt;code&gt;z = my_func(x^2)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This would be a nice feature for a language.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Error Explanation&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;There is a discussion of &lt;a href=&quot;http://www.rit.edu/cos/uphysics/uncertainties/Uncertaintiespart2.html#propagation&quot; rel=&quot;nofollow&quot;&gt;ERROR PROPAGATION&lt;/a&gt; here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the error tracking, I'm assuming that &lt;code&gt;x.err&lt;/code&gt; is the error associated with each measurement, in this case, a constant value of &lt;code&gt;0.1&lt;/code&gt;. &lt;code&gt;y.err&lt;/code&gt; is similar, but each measurement is part of some noise. Given a standard definition of error propagation, such as the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;When adding x + y with errors dx and dy, the solution has error dx + dy&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Or in code&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;z = x + y =&amp;gt; dz = dx + dy&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I can add lines of code to track dz, but is there a language that supports, or could support, dz without me doing extra work?&lt;/p&gt;&#xA;" OwnerUserId="957" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-21T20:02:06.103" LastActivityDate="2010-12-25T13:20:55.960" Title="Implementing error propagation" Tags="&lt;error-propagation&gt;" AnswerCount="3" CommentCount="4" />
  <row Id="5630" PostTypeId="2" ParentId="5629" CreationDate="2010-12-19T05:08:59.810" Score="4" ViewCount="" Body="&lt;p&gt;It's difficult to determine what you're asking for because you haven't specified the semantics of &quot;x.err&quot; etc., but it sounds like you might be interested in &lt;a href=&quot;http://en.wikipedia.org/wiki/Interval_arithmetic#Implementations&quot; rel=&quot;nofollow&quot;&gt;interval arithmetic&lt;/a&gt;.  &lt;a href=&quot;http://www.cs.utep.edu/interval-comp/intlang.html&quot; rel=&quot;nofollow&quot;&gt;Implementations&lt;/a&gt; are available in Lisp, Numerica, Maple, Matlab, and Mathematica.  &lt;a href=&quot;http://www.cs.utep.edu/interval-comp/intsoft.html&quot; rel=&quot;nofollow&quot;&gt;Libraries&lt;/a&gt; are available for ADA, Fortran, C++, etc.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Edit&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;In light of comments (scattered between here and SO), it appears the OP is asking about implementing automatic &lt;a href=&quot;http://www.rit.edu/cos/uphysics/uncertainties/Uncertaintiespart2.html#propagation&quot; rel=&quot;nofollow&quot;&gt;error propagation&lt;/a&gt;. In principle that's no harder than implementing, say, a complex number class or interval arithmetic, which are straightforward exercises in any OO system.  The data structure would be a tuple $(x, \epsilon)$ where $x$ represents a real number and $\epsilon \ge 0$ quantifies its &quot;error&quot;.  The usual numbers would be embedded into this structure via $x \to (x, 0)$.  The semantics would include&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\eqalign{&#xD;&#xA;(x, \epsilon) + (y, \delta) = &amp;amp;(x+y, (\epsilon^2 + \delta^2)^{1/2}) \cr&#xD;&#xA;(x, \epsilon) - (y, \delta) = &amp;amp;(x-y, (\epsilon^2 + \delta^2)^{1/2}) \cr&#xD;&#xA;(x, \epsilon) \times (y, \delta) = &amp;amp;(x y,((y \epsilon)^2 + (x \delta)^2)^{1/2}) \cr&#xD;&#xA;\text{etc.} \cr&#xD;&#xA;f((x, \epsilon), (y, \delta)) = &amp;amp;(f(x, y), || (\frac{\partial f}{\partial x}\epsilon, \frac{\partial f}{\partial y}\delta)||). \cr&#xD;&#xA;}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(The last generalizes everything preceding it and points the way to implementing powers, exponentials, logs, trig functions, &lt;em&gt;etc&lt;/em&gt;.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Such an implementation could truly be handy for those who understand deeply what's going on.  For others--who might not be aware of or appreciate the importance of assumptions of independence, of relatively small errors, of differentiability of functions, &lt;em&gt;etc.&lt;/em&gt;--it would be truly dangerous.  As a simple example, when $(x, \epsilon)$ and $(y, \delta)$ are perfectly negatively correlated (implying $\epsilon = \delta$), the correct operation is $(x, \epsilon) + (y, \delta) = (x + y, 0)$.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-12-21T19:22:49.363" LastActivityDate="2010-12-21T19:22:49.363" CommentCount="6" />
  <row Id="5631" PostTypeId="2" ParentId="5629" CreationDate="2010-12-19T05:12:26.403" Score="2" ViewCount="" Body="&lt;p&gt;I recommend you look into &lt;a href=&quot;http://www.r-project.org/&quot; rel=&quot;nofollow&quot;&gt;R&lt;/a&gt;, the open source software for statistical computing. R supports many vectorized functions to deal with element by element computation of objects. Chapter 3 of &lt;a href=&quot;http://lib.stat.cmu.edu/S/Spoetry/Tutor/R_inferno.pdf&quot; rel=&quot;nofollow&quot;&gt;The R Inferno&lt;/a&gt; provides a good outline on vectorization in R. From your post above, I'm assuming you have some background in computer programming, and R's command line interface will probably feel right at home to you. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's your pseudo code above implemented in R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(42)    #Makes random numbers reproducible.&#xA;x &amp;lt;- sample(1:20, size = 10, replace = FALSE)&#xA;y &amp;lt;- sample(1:20, size = 10, replace = FALSE)&#xA;z &amp;lt;- x + y&#xA;z&#xA;&amp;gt; z&#xA; [1] 29 32 23 20 19 23 35  4 23 19&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm not entirely sure what you are trying to do with the error coding, but R has the capability to support just about any mathematical or statistical computation you can dream up. If someone else hasn't already implemented a solution for your problem, you can leverage the fact R is a programming language and write your own code to solve it. It is relatively easy to interface with other languages such as C, C++, python from R so there are plenty of options there.&lt;/p&gt;&#xA;" OwnerUserId="696" LastActivityDate="2010-12-19T05:12:26.403" />
  <row Id="5632" PostTypeId="2" ParentId="5620" CreationDate="2010-12-19T05:18:34.387" Score="4" ViewCount="" Body="&lt;p&gt;In mathematics and physics, the &quot;x&quot; in &quot;x-vector&quot; stands for the dimension of the vector.  The meanings of $K$ and $p$ were previously established.  Typically a &quot;p-vector&quot; is written as a column vector and a &quot;p-covector&quot; would be written as a row vector.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-12-19T05:18:34.387" />
  <row Id="5633" PostTypeId="2" ParentId="5614" CreationDate="2010-12-19T06:15:26.400" Score="3" ViewCount="" Body="&lt;p&gt;The answer depends on the nature of the data generation process and on the alternative hypothesis you have in mind.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your test is a kind of unweighted chi-square.  Because of this lack of weighting, changes that principally affect the less-populated categories will be difficult to detect.  For example, your test is going to be &lt;em&gt;much&lt;/em&gt; less powerful than the chi-square test for a uniform shift in location, which is detected primarily by noticing that almost all the probability in one tail gets shifted into the other tail.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, suppose your categories are integer ranges $[i, i+1)$ indexed by $i$ and you are observing normal variates of unit variance but unknown mean.  100 observations of a standard normal variate, say, will mainly occupy categories $-2$ through $1$, although you can expect a few to occupy categories $-3$ and $2$.  Even for a whopping big shift of $5$ standard errors (&lt;em&gt;i.e.&lt;/em&gt;, a change in mean of $5/\sqrt{100} = 0.5$), the power of your K-D-like test is only about 50% (when $\alpha = 0.05$).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is difficult to conceive of a setting where this test will be more powerful than the chi-square test.  If you think you are in such a situation, perform some simulations to find out what the power is and how it compares to the standard alternative tests.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-12-19T06:48:42.843" LastActivityDate="2010-12-19T06:48:42.843" CommentCount="2" />
  <row Id="5634" PostTypeId="1" AcceptedAnswerId="5635" CreationDate="2010-12-19T15:53:17.357" Score="2" ViewCount="180" Body="&lt;p&gt;So I have data like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Cost    20  30  10   5&#xA;Rating   5   3   2   5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to make a chart of rating vs. cost, so the points would be &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[(5,20), (3,30), (2,10), (5,5)]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I can't seem to get excel to do anything other than put the two rows as independent series. Am I missing something, or I do have to pivot the data somehow to make it do that?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Actually, I'm using an old-ish version of &lt;a href=&quot;http://www.apple.com/iwork/numbers/&quot; rel=&quot;nofollow&quot;&gt;Numbers.app&lt;/a&gt; on OS X, but I'm hoping the concept will be the same. I have access to excel if need be.)&lt;/p&gt;&#xA;" OwnerUserId="1531" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-12-19T18:14:33.343" LastActivityDate="2010-12-19T18:14:33.343" Title="In Excel, how do I plot two rows against each other?" Tags="&lt;data-visualization&gt;&lt;excel&gt;" AnswerCount="1" />
  <row Id="5635" PostTypeId="2" ParentId="5634" CreationDate="2010-12-19T15:56:51.113" Score="4" ViewCount="" Body="&lt;p&gt;Select the two rows and do a scatterplot which I think is called an XY plot in Excel (sorry, I run a Linux machine, so I do not have Excel installed).&lt;/p&gt;&#xA;" OwnerUserId="582" LastActivityDate="2010-12-19T15:56:51.113" CommentCount="5" />
  <row Id="5636" PostTypeId="2" ParentId="3497" CreationDate="2010-12-19T18:04:59.073" Score="4" ViewCount="" Body="&lt;p&gt;The answer to your original question is yes, because the classical theory applies under your sampling scheme.  You don’t need any assumptions on the original data matrix.  All of the randomness (implicitly behind standard errors and consistency) comes from your scheme for sampling $N$ rows from the data matrix.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Think of your entire dataset (100M rows) as being the population.  Each estimate (assuming your sample of size $N$ is a simple random sample of the rows) is a consistent estimate of the regression coefficients (say, $\hat{\beta}_*$) computed from the entire data set.  Moreover, it is approximately Normal with mean equal to $\hat{\beta}_*$ and some covariance.  The usual estimate of the covariance of the estimate is also consistent.  If you repeat this $M$ times and average those $M$ estimates, then the resulting estimate (say, $\hat{\beta}_{avg}$) will also be approximately Normal.  You can treat those $M$ estimates as being nearly independent (uncorrelated) as long as $N$ and $M$ are small relative to 100M.  That’s an important assumption.  The idea being that sampling without replacement is approximately the same as sampling with replacement when the sample size is small compared to the population size.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That being said, I think that your problem really is one of how to efficiently approximate the regression estimate ($\hat{\beta}_*$) computed from the entire data set.  There is a difference between (1) averaging $M$ estimates based on samples of size $N$ and (2) one estimate based on a sample of size $MN$.  The MSE of (2) will generally be smaller than the MSE of (1).  They would only be equal if the estimate was linear in the data, but that is not the case.  I assume you are using least squares.  The least squares estimate is linear in the $Y$ (response) vector, but not the $X$ (covariates) matrix.  You are randomly sampling $Y$ and $X$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(1) and (2) are both simple schemes, but not necessarily efficient. (Though it may not matter since you only have 30 variables.)  There are better ways.  Here is one example: &lt;a href=&quot;http://arxiv.org/abs/0710.1435&quot; rel=&quot;nofollow&quot;&gt;http://arxiv.org/abs/0710.1435&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1670" LastEditorUserId="1670" LastEditorDisplayName="" LastEditDate="2010-12-19T21:49:54.970" LastActivityDate="2010-12-19T21:49:54.970" />
  <row Id="5637" PostTypeId="2" ParentId="5418" CreationDate="2010-12-19T18:29:21.010" Score="1" ViewCount="" Body="&lt;p&gt;Another piece of advise might be to look at packages yours will be depending on or interacting with, especially if these implement some &lt;a href=&quot;http://stats.stackexchange.com/questions/5418/first-r-packages-source-code-to-study-in-preparation-for-writing-own-package/5433#5433&quot;&gt;items&lt;/a&gt; &lt;a href=&quot;http://stats.stackexchange.com/users/1657/joshua-ulrich&quot;&gt;Joshua Ulrich&lt;/a&gt; mentioned or have been written by renowned authors. It might be helpful to learn how things are done in your field, to ensure some compatibility. Often people will have thought about certain issues and reading their solution migth be helpful. &lt;/p&gt;&#xA;" OwnerUserId="1355" LastActivityDate="2010-12-19T18:29:21.010" />
  <row Id="5638" PostTypeId="1" AcceptedAnswerId="5639" CreationDate="2010-12-20T02:40:39.920" Score="3" ViewCount="108" Body="&lt;p&gt;Hi All:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose I have a vector and I don't know, apriori, its unique elements (here: 1 and 2).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;vec &amp;lt;-&#xA;  c(1, 1, 1, 2, 2, 2, 2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I was interested in knowing is there a better way (or elegant way) of getting the number of unique elements in &lt;code&gt;vec&lt;/code&gt; i.e. the same result as &lt;code&gt;table(vec)&lt;/code&gt;. It doesn't matter if its a data.frame or a named vector. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;R&amp;gt; table(vec)&#xA;vec&#xA;1 2 &#xA;3 4 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Reason:&lt;/em&gt; I was curious to know if there is a better way. Also, I noticed that there is a &lt;code&gt;for&lt;/code&gt; loop in the &lt;code&gt;base&lt;/code&gt; implementation (in addition to .C call). I don't know if it's a big concern, but when I do something like &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;R&amp;gt; table(rep(1:1000,100000))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;R takes really long time. I am sure it's because of the huge number 100000. But is there a way of making it faster?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt; This also does a good job in addition to &lt;code&gt;Chase's&lt;/code&gt; answer.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;R&amp;gt; rle(sort(sampData))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1307" LastEditorUserId="1307" LastEditorDisplayName="" LastEditDate="2010-12-20T05:14:34.403" LastActivityDate="2010-12-20T13:08:25.967" Title="Is there a better way of obtaining the same output as table(vec) where vec is a vector?" Tags="&lt;r&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="5639" PostTypeId="2" ParentId="5638" CreationDate="2010-12-20T03:18:59.257" Score="7" ViewCount="" Body="&lt;p&gt;This is an interesting problem - I'm curious to see other thoughts on this. Looking at the source for &lt;code&gt;table()&lt;/code&gt; reveals that it builds off of &lt;code&gt;tabulate()&lt;/code&gt;. &lt;code&gt;tabulate()&lt;/code&gt; has a few quirks apparently, namely that it only deals with positive integers and returns an integer vector without names. We can use &lt;code&gt;unique()&lt;/code&gt; on our vector to apply the &lt;code&gt;names()&lt;/code&gt;. If you need to tabulate zero or negative values, I guess going back and reviewing &lt;code&gt;table()&lt;/code&gt; would be necessary as &lt;code&gt;tabulate()&lt;/code&gt; doesn't seem to do that per the examples on the help page.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;table2 &amp;lt;- function(data) {&#xA;    x &amp;lt;- tabulate(data)&#xA;    y &amp;lt;- sort(unique(data))&#xA;    names(x) &amp;lt;- y&#xA;    return(x)   &#xA;    }&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And a quick test:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; set.seed(42)&#xA;&amp;gt; sampData &amp;lt;- sample(1:5, 10000000, TRUE, prob = c(.3,.25, .2, .15, .1))&#xA;&amp;gt; &#xA;&amp;gt; system.time(table(sampData))&#xA;   user  system elapsed &#xA;  4.869   0.669   5.503 &#xA;&amp;gt; system.time(table2(sampData))&#xA; user  system elapsed &#xA;0.410   0.200   0.605&#xA;&amp;gt; &#xA;&amp;gt; table(sampData)&#xA;sampData&#xA;      1       2       3       4       5 &#xA;2999200 2500232 1998652 1500396 1001520 &#xA;&amp;gt; table2(sampData)&#xA;      1       2       3       4       5 &#xA;2999200 2500232 1998652 1500396 1001520 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;EDIT: I just realized there is a &lt;code&gt;count()&lt;/code&gt; function in &lt;code&gt;plyr&lt;/code&gt; which is another alternative to &lt;code&gt;table()&lt;/code&gt;. In the test above, it performs better than &lt;code&gt;table()&lt;/code&gt;, and slightly worse than the hack-job solution I put together:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(plyr)&#xA; system.time(count(sampData))&#xA;   user  system elapsed &#xA;  1.620   0.870   2.483&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="696" LastEditorUserId="696" LastEditorDisplayName="" LastEditDate="2010-12-20T13:08:25.967" LastActivityDate="2010-12-20T13:08:25.967" CommentCount="5" />
  <row Id="5640" PostTypeId="2" ParentId="5603" CreationDate="2010-12-20T03:39:18.017" Score="3" ViewCount="" Body="&lt;p&gt;If this is for a particular application, I would first consider the question: &quot;do you need to fit such a complex model?&quot; which is code for, what other, more simple methods have you tried prior to this one?  Have you looked at a plot of the &quot;births &amp;amp; deaths&quot; data that you have?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would advise that you look up some of the work on &quot;state space&quot; theory, and kalman filtering as a starter.  It sounds like you basically have an autoregressive process that isn't a simple random walk.  These can usually be dealt with via the Kalman filter (as long as you are willing to assume normality of the errors).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A simple way (I think) is to consider a &quot;regression&quot; of the births/deaths against the current state.  You could just use OLS as a simple start (to figure out what's going on), but this ignores that the errors from the regression are correlated (and not independent as in OLS).  This will have the effect of OLS standard errors being either too small or too big, depending on the direction of the correlation (the OLS estimates are still &quot;unbiased&quot;)&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2010-12-20T03:39:18.017" CommentCount="1" />
  <row Id="5641" PostTypeId="2" ParentId="5090" CreationDate="2010-12-20T03:50:36.193" Score="0" ViewCount="" Body="&lt;p&gt;Here's the solution I came up with: The trick is to add NAs to the end of the observation data. When seeing NA as a response variable the Kalman filter algorithm will simply predict the next value and not update the state vector. This is exactly what we want to make our forecast.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;nAhead &amp;lt;- 12&#xA;mod &amp;lt;- dlmModSeas(4)+dlmModReg(cbind(rnorm(100+nAhead),rnorm(100+nAhead)))&#xA;fi &amp;lt;- dlmFilter(c(rnorm(100),rep(NA,nAhead)),mod)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is this correct?&lt;/p&gt;&#xA;" OwnerUserId="2451" LastActivityDate="2010-12-20T03:50:36.193" />
  <row Id="5642" PostTypeId="2" ParentId="2182" CreationDate="2010-12-20T05:45:35.210" Score="2" ViewCount="" Body="&lt;p&gt;It’s easier to explain in terms of standard deviations, rather than confidence intervals.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your friend’s conclusion is basically correct &lt;strong&gt;under the simplest model&lt;/strong&gt; where you have simple random sampling and two candidates.  Now the sample proportions satisfy $p_A + p_B = 1$ so that $p_B = 1 - p_A$.  Thus, &#xA;$$Var(p_A - p_B) = Var(2 p_A - 1) = 4 Var(p_A)$$&#xA;and so &#xA;$$SD(p_A - p_B) = 2 SD(p_A).$$&#xA;What makes this simple relationship possible is that $p_A$ and $p_B$ are perfectly negatively correlated, because in general&#xA;$$Var(p_A - p_B) = Var(p_A) + Var(p_B) - 2 Cov(p_A, p_B).$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Outside this simple model&lt;/strong&gt;, if $p_A + p_B = 1$ does not hold in general, then &lt;strong&gt;you must take into account the correlation&lt;/strong&gt; between $p_A$ and $p_B$ that is not included in the margin of error.  It is possible for $SD(p_A - p_B) \ll 2 SD(p_A)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But all this nuance seems to indicate that the polling organizations should report the margin of error on the difference.  Where’s Nate Silver?&lt;/p&gt;&#xA;" OwnerUserId="1670" LastEditorUserId="1670" LastEditorDisplayName="" LastEditDate="2010-12-22T06:32:25.093" LastActivityDate="2010-12-22T06:32:25.093" />
  <row Id="5644" PostTypeId="2" ParentId="5382" CreationDate="2010-12-20T07:25:21.537" Score="0" ViewCount="" Body="&lt;p&gt;I worked on it and thought this Bayesian equation will be useful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rating&lt;sub&gt;TM&lt;/sub&gt; = SR&lt;sub&gt;TM&lt;/sub&gt;/(1 + AWD/WD&lt;sub&gt;TM&lt;/sub&gt;) + MSRT/(1 + WD&lt;sub&gt;TM&lt;/sub&gt;/AWD)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The variables are:&lt;br/&gt;&#xA;SR = Team member's self rating&lt;br/&gt;&#xA;AWD = Average work done by team&lt;br/&gt;&#xA;WD = Work done by team member&lt;br/&gt;&#xA;MSRT = Mean self rating of team&lt;br/&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(TM: TeamMember)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please comment if you think this is not right. Thanks!&lt;/p&gt;&#xA;" OwnerUserId="2344" LastActivityDate="2010-12-20T07:25:21.537" CommentCount="3" />
  <row Id="5645" PostTypeId="2" ParentId="2982" CreationDate="2010-12-20T07:31:41.610" Score="3" ViewCount="" Body="&lt;p&gt;$L_1$ penalization is part of an optimization problem.  Soft-thresholding is part of an algorithm.  Sometimes $L_1$ penalization leads to soft-thresholding.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For regression, $L_1$ penalized least squares (Lasso) results in soft-thresholding when the columns of the $X$ matrix are orthogonal (assuming the rows correspond to different samples).  It is really straight-forward to derive when you consider the special case of mean estimation, where the $X$ matrix consists of a single $1$ in each row and zeroes everywhere else.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the general $X$ matrix, computing the Lasso solution via cyclic coordinate descent results in essentially iterative soft-thresholding.  See &lt;a href=&quot;http://projecteuclid.org/euclid.aoas/1196438020&quot; rel=&quot;nofollow&quot;&gt;http://projecteuclid.org/euclid.aoas/1196438020&lt;/a&gt; .&lt;/p&gt;&#xA;" OwnerUserId="1670" LastEditorUserId="1670" LastEditorDisplayName="" LastEditDate="2010-12-21T01:03:16.813" LastActivityDate="2010-12-21T01:03:16.813" CommentCount="1" />
  <row Id="5646" PostTypeId="2" ParentId="2854" CreationDate="2010-12-20T07:59:32.343" Score="3" ViewCount="" Body="&lt;p&gt;First I would like to point out that to get the object &lt;code&gt;ks3&lt;/code&gt; you do not need to do that much of the book-keeping code. Use the features of &lt;code&gt;ddply&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ks3 &amp;lt;- ddply(o,.(ai,Gs),function(d){&#xA;     temp &amp;lt;- ols(value~as.numeric(bc)+as.numeric(age),data=d,x=T,y=T)&#xA;     t2 &amp;lt;- bootcov(temp,B=1000,coef.reps=T) &#xA;     data.frame(t2$boot.Coef,ai=d$ai[1],gc2=d$Gs[1])&#xA;})&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Also &lt;code&gt;scales=free&lt;/code&gt; works much better if you want to see what kind of graph you intend to provide. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Concerning your first question, if you estimate the standard errors of coefficient using bootstrap, you are absolutely correct in using estimated percentiles. There is no need to use the percentiles of bivariate normal, since then you assume normality and bootstrap is used exactly for the purpose of not attaching oneself to some theoretical distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For your second question, first you need to define what do you mean by bias-corrected percentiles. Furthermore your question implies that &lt;code&gt;bootcov&lt;/code&gt; returns non bias-corrected percentiles, but it does not return any percentiles at all.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;&lt;strong&gt;Update&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we know that coefficients are normal we can use confidence ellipses. The mean and covariance matrix are estimated using bootstrap. The following code checks which of the observations of the bivariate sample falls into 0.95 confidence ellipse region:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; confeps &amp;lt;- function(x, level=0.95, t=qchisq(level,2)) { &#xA;      cv &amp;lt;- cov(x)&#xA;      ch &amp;lt;- t(chol(x))&#xA;      x &amp;lt;- sweep(x, 2, apply(x, 2, mean), &quot;-&quot;)&#xA;      y &amp;lt;- solve(ch,t(x))&#xA;      tt &amp;lt;- y[1, ]^2 + y[2, ]^2&#xA;      tt &amp;lt; t&#xA;    }&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here is the example how this function works:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;      bs &amp;lt;- cbind(a &amp;lt;- rnorm(1000), 2-a/3+rnorm(1000)/5)&#xA;      mn &amp;lt;- apply(bs, 2, mean)&#xA;      require(ellipse)&#xA;      eps &amp;lt;- ellipse(cov(bs), centre=mn, npoints=1000)&#xA;      plot(eps, type=&quot;l&quot;, xlim=range(bs[,1]), ylim=range(bs[, 2]))&#xA;      col &amp;lt;- rep(1, nrow(x))&#xA;      col[!confeps(bs)] &amp;lt;- 2&#xA;      points(bs[, 1], bs[, 2], col=col)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2010-12-24T11:45:34.523" LastActivityDate="2010-12-24T11:45:34.523" CommentCount="3" />
  <row Id="5647" PostTypeId="1" CreationDate="2010-12-20T12:30:03.800" Score="2" ViewCount="211" Body="&lt;p&gt;Dear all,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am fitting a conditional logistic regression model with 1:4 controls using &lt;code&gt;R&lt;/code&gt;. I wish to obtain &lt;code&gt;AIC&lt;/code&gt; from the model. How can I extract the appropriate parameters based on object m?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(survival)&#xA;m&amp;lt;-clogit(cc~exp+ factor1+ factor2 + strata(stratum),data=data1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Thank you for your help.&lt;/p&gt;&#xA;" OwnerUserId="2264" LastActivityDate="2010-12-20T12:40:11.920" Title="How to obtain AIC with conditional logistic regression using R?" Tags="&lt;r&gt;&lt;logistic-regression&gt;" AnswerCount="1" />
  <row Id="5648" PostTypeId="2" ParentId="5647" CreationDate="2010-12-20T12:40:11.920" Score="3" ViewCount="" Body="&lt;p&gt;It seems you must do it manually, so something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;2*length(m$coefficients)-2*(m$loglik[2])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-12-20T12:40:11.920" CommentCount="2" />
  <row Id="5649" PostTypeId="2" ParentId="5602" CreationDate="2010-12-20T14:10:31.630" Score="2" ViewCount="" Body="&lt;p&gt;Since your metadata is probably discrete, I suggest using &lt;a href=&quot;http://en.wikipedia.org/wiki/Classification_tree&quot; rel=&quot;nofollow&quot;&gt;classification trees&lt;/a&gt;. Note that from your example it is highly likely that your unknown algorithm is random, i.e. there is no algorithm. &lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2010-12-20T14:10:31.630" />
  <row Id="5650" PostTypeId="1" CreationDate="2010-12-20T14:49:58.113" Score="6" ViewCount="138" Body="&lt;p&gt;Hi All,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am currently working on some time series data, I know I can use&#xA;LOESS/ARIMA model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The  data is written to a vector whose length is 1000, which is a queue,&#xA;updating every 15 minutes,  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus the old data will pop out while the new data push in the vector.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can rerun the whole model on a scheduler, e.g. retrain the model every 15&#xA;minutes, that is,  Use the whole 1000 value to train the LOESS model, However it is very inefficient, as every time only one&#xA;value is insert while another 999 vlaues still same as last time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So how can I achieve better performance?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks&lt;/p&gt;&#xA;" OwnerUserId="2454" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-20T21:37:58.697" LastActivityDate="2010-12-20T21:37:58.697" Title="Incremental learning for LOESS time series model" Tags="&lt;time-series&gt;" CommentCount="3" FavoriteCount="1" />
  <row Id="5651" PostTypeId="2" ParentId="5602" CreationDate="2010-12-20T16:55:46.273" Score="1" ViewCount="" Body="&lt;p&gt;This isn't a clustering, it's (supervised) classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are a lot of methods to do classification, such as naïve bayes for binary features and linear discriminants for continuous, or neural networks for some opaque combination of both.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the example you mention, the feasibility of what you suggest depends a lot on the metadata - for example, if it's been tagged (=classified) properly, you could expect to be able to find some tags they like/dislike. However, the problem you stated is more general. Discovering the exact algorithm is unlikely, and in the example one doesn't even exist.&lt;/p&gt;&#xA;" OwnerUserId="2456" LastActivityDate="2010-12-20T16:55:46.273" />
  <row Id="5652" PostTypeId="2" ParentId="5601" CreationDate="2010-12-20T18:12:43.150" Score="3" ViewCount="" Body="&lt;p&gt;This is not a bug. The model is stored using a 0-based index. So SplitVar=0 is X1, SplitVar=1 is X2, and SplitVar=2 is X3. So this split corresponds to a split on X3. Since X3 is an ordinal factor and the split is at 1.5, this corresponds to splitting levels 0&amp;amp;1 from 2&amp;amp;3.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&gt; sum(data$X3&amp;lt;=&quot;c&quot;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[1] 522&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&gt; sum(data$X3&gt;=&quot;b&quot;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[1] 478&lt;/p&gt;&#xA;" OwnerUserId="2457" LastActivityDate="2010-12-20T18:12:43.150" CommentCount="1" />
  <row Id="5653" PostTypeId="2" ParentId="5111" CreationDate="2010-12-20T19:26:32.303" Score="2" ViewCount="" Body="&lt;p&gt;Under one interpretation of your situation there is no need to modify the p values at all.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, let's posit that a sequence of (unknown) bivariate distributions $p_i(x,y)$ govern $A$ and $B$ for each organism $i$.  That is, $\Pr(A=x, B=y) = p_i(x,y)$ for all possible outcomes $(x,y)$ of $(A,B)$.  To test whether the measurement procedures $A$ and $B$ differ, a reasonable null hypothesis is that these distributions are all &lt;em&gt;symmetric&lt;/em&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$H_0: p_i(x,y) = p_i(y,x) \text{ for all } i, x, y.$$&#xD;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The sign statistic (difference between number of $+$ and number of $-$ results) is still a reasonable one to use in this test.  (It actually tests the null hypothesis $H_0: \Pr(A&amp;lt;B) = \Pr(B&amp;lt;A)$.)  Its distribution depends on the chances of &lt;em&gt;ties&lt;/em&gt;; namely on the values $t_i = \sum_{x}p_i(x,x)$ (one for each organism $i$).  The question, which appears not to contemplate the possibility of ties at all, suggests their chances are fairly small.  In any case, the symmetry assumption in the null implies the chance of organism $i$ yielding a $+$ sign equals the chance of organism $i$ yielding a $-$ sign and the assumption that ties are unlikely implies both these chances are close to $1/2$.  This implies the distribution of the sign statistic is binomial, as usual, &lt;em&gt;despite any correlation&lt;/em&gt; (or lack thereof) between $A$ and $B$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If there is a substantial chance of ties, it looks like you cannot make any progress towards quantitative bounds until you specify something about those chances.  For example, if you provide an upper bound for the $t_i$ you can say something about the distribution of the sign statistic.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-12-20T19:26:32.303" />
  <row Id="5654" PostTypeId="1" CreationDate="2010-12-20T22:24:03.600" Score="7" ViewCount="319" Body="&lt;p&gt;I'm trying to help a scientist design a study for the occurrence of salmonella microbes.  He would like to compare an experimental antimicrobial formulation against a chlorine (bleach) at poultry farms.  Because background rates of salmonella differ over time, he plans to measure % poultry w/salmonella before treatment, and after treatment. So the measurement will be the difference of before/after % salmonella for the experimental vs. chlorine formulas.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone advise on how to estimate the sample sizes necessary?  Let's say the background rate is 50%; after bleach it's 20%; and we want to detect whether the experimental formulation changes the rate by +/- 10%.  thank you&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT:&#xA;What I'm struggling with is how to incorporate the background rates. Let's call them p3 and p4, the &quot;before&quot; salmonella rates for bleach and experimental samples, respectively. So the statistic to be estimated is the difference of differences: Experimental(After-Before) - Bleach(After-Before) = (p0-p2) - (p3-p1). To fully account for the sampling variation of &quot;before&quot; rates p2 and p3 in the sample-size calculation --- is it as simple as using p0(1-p0)+p1(1-p1)+p2(1-p2)+p3(1-p3) wherever there's a variation term in the sample-size equation? Let all samples sizes be equal, n1 = n2 = n.&lt;/p&gt;&#xA;" OwnerUserId="2473" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-22T11:04:25.630" LastActivityDate="2010-12-22T11:04:25.630" Title="Sample size for proportions in repeated measures" Tags="&lt;sample-size&gt;&lt;repeated-measures&gt;&lt;proportion&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="5655" PostTypeId="2" ParentId="1875" CreationDate="2010-12-21T00:33:47.953" Score="0" ViewCount="" Body="&lt;p&gt;How about just binning the given predictions and taking the observed fractions as your estimate for each bin?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can generalise this to a continuous model by weighing all the observations around your value of interest (say the prediction by tomorrow) by a Gaussian and seeing what the weighted average is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can guess a width to get you a given fraction of your data (or, say, never less than 100 points for a good estimate). Alternatively use a method such as cross-validation of max-likelihood to get the Gaussian width.&lt;/p&gt;&#xA;" OwnerUserId="2067" LastActivityDate="2010-12-21T00:33:47.953" />
  <row Id="5656" PostTypeId="1" AcceptedAnswerId="5662" CreationDate="2010-12-21T01:35:25.853" Score="5" ViewCount="300" Body="&lt;p&gt;All,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Wondering if anyone has run across a package/function in R that will combine levels of a factor whose proportion of all the levels in a factor is less than some threshold? Specifically, one of the first steps in data preparation I conduct is to collapse sparse levels of factors together (say into a level called 'Other') that do not constitute at least, say, 2% of the total. This is done unsupervised and is done when the objective is to model some activity in marketing (not fraud detection, where those very small occurrences could be extremely important). I am looking for a function that will collapse levels until some threshold proportion is met. Does anyone know of anything?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&#xA;Brian &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks to these great suggestions I wrote a function pretty easily. I did realize though that it was possible to collapse levels with proportion &amp;lt; the minimum and still have that recoded level be &amp;lt; the minimum, requiring the addition of the lowest level with proportion &gt; the minimum. Likely can be more efficient but it appears to work. The next enhancement would be to figure out how to capture the &quot;rules&quot; for applying the collapse logic to new data (a validation set or future data).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;collapseFactors&amp;lt;- function(tableName,minPercent=5,fillIn =&quot;RECODED&quot; )&#xA;{&#xA;    for (i in 1:ncol(tableName))&#xA;        {   &#xA;&#xA;            if(is.factor(tableName[,i]) == TRUE) #process just factors&#xA;            {&#xA;&#xA;&#xA;                sortedTable&amp;lt;-sort(prop.table(table(tableName[,i])))&#xA;                numberToCollapse&amp;lt;-length(sortedTable[sortedTable&amp;lt;(minPercent/100)])&#xA;&#xA;                if (sum(sortedTable[1:numberToCollapse])&amp;lt;(minPercent/100))&#xA;                    {&#xA;                        numberToCollapse=numberToCollapse+1 #add next level if &amp;lt; minPercent&#xA;                    }&#xA;&#xA;                if(numberToCollapse&amp;gt;1) #if not &amp;gt;1 then nothing to collapse&#xA;                {&#xA;                    lf &amp;lt;- names(sortedTable[1:numberToCollapse])&#xA;                    levels(tableName[,i])[levels(tableName[,i]) %in% lf] &amp;lt;- fillIn&#xA;                }&#xA;            }#end if a factor&#xA;&#xA;&#xA;        }#end for loop&#xA;&#xA;    return(tableName)&#xA;&#xA;}#end function&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2040" LastEditorUserId="2040" LastEditorDisplayName="" LastEditDate="2010-12-22T01:57:10.257" LastActivityDate="2010-12-22T01:57:10.257" Title="R package for combining factor levels for datamining?" Tags="&lt;r&gt;" AnswerCount="3" FavoriteCount="1" />
  <row Id="5657" PostTypeId="2" ParentId="5656" CreationDate="2010-12-21T02:33:01.607" Score="3" ViewCount="" Body="&lt;p&gt;I wrote a quick function that will accomplish this goal. I'm a novice R user, so it may be slow with large tables.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Merge.factors &amp;lt;- function(x, p) { &#xA;    #Combines factor levels in x that are less than a specified proportion, p.&#xA;    t &amp;lt;- table(x)&#xA;    y &amp;lt;- subset(t, prop.table(t) &amp;lt; p)&#xA;    z &amp;lt;- subset(t, prop.table(t) &amp;gt;= p)&#xA;    other &amp;lt;- rep(&quot;Other&quot;, sum(y))&#xA;    new.table &amp;lt;- c(z, table(other))&#xA;    new.x &amp;lt;- as.factor(rep(names(new.table), new.table))&#xA;    return(new.x)&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As an example of it in action:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; a &amp;lt;- rep(&quot;a&quot;, 100)&#xA;&amp;gt; b &amp;lt;- rep(&quot;b&quot;, 1000)&#xA;&amp;gt; c &amp;lt;- rep(&quot;c&quot;, 1000)&#xA;&amp;gt; d &amp;lt;- rep(&quot;d&quot;, 1000)&#xA;&amp;gt; e &amp;lt;- rep(&quot;e&quot;, 400)&#xA;&amp;gt; f &amp;lt;- rep(&quot;f&quot;, 100)&#xA;&amp;gt; x &amp;lt;- factor(c(a, b, c, d, e, f))&#xA;&amp;gt; summary(x)&#xA;   a    b    c    d    e    f &#xA; 100 1000 1000 1000  400  100 &#xA;&amp;gt; prop.table(table(x))&#xA;x&#xA;         a          b          c          d          e          f &#xA;0.02777778 0.27777778 0.27777778 0.27777778 0.11111111 0.02777778 &#xA;&amp;gt; &#xA;&amp;gt; w &amp;lt;- Merge.factors(x, .05)&#xA;&amp;gt; summary(w)&#xA;    b     c     d     e Other &#xA; 1000  1000  1000   400   200 &#xA;&amp;gt; class(w)&#xA;[1] &quot;factor&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1118" LastEditorUserId="1118" LastEditorDisplayName="" LastEditDate="2010-12-21T06:58:11.473" LastActivityDate="2010-12-21T06:58:11.473" CommentCount="2" />
  <row Id="5658" PostTypeId="2" ParentId="5656" CreationDate="2010-12-21T04:51:40.117" Score="4" ViewCount="" Body="&lt;p&gt;The only problem with Christopher answer is that it will mix up the original ordering of the factor. Here is my fix:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; Merge.factors &amp;lt;- function(x, p) {&#xA;     t &amp;lt;- table(x)&#xA;     levt &amp;lt;- cbind(names(t), names(t)) &#xA;     levt[t/sum(t)&amp;lt;p, 2] &amp;lt;- &quot;Other&quot;&#xA;     change.levels(x, levt)&#xA; }&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where &lt;code&gt;change.levels&lt;/code&gt; is the following function. I wrote it some time ago, so I suspect there might be better ways of achieving what it does.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; change.levels &amp;lt;- function(f, levt) {&#xA;     ##Change the the names of the factor f levels from&#xA;     ##substitution table levt.&#xA;     ## In the first column there are the original levels, in&#xA;     ## the second column -- the substitutes&#xA;     lv &amp;lt;- levels(f)&#xA;     if(sum(sort(lv) != sort(levt[, 1]))&amp;gt;0)&#xA;     stop (&quot;The names from substitution table does not match given level names&quot;)&#xA;     res &amp;lt;- rep(NA, length(f))&#xA;&#xA;     for(i in lv) {&#xA;          res[f==i] &amp;lt;- as.character(levt[levt[, 1]==i, 2])&#xA;     }&#xA;     factor(res)&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2010-12-21T04:51:40.117" />
  <row Id="5659" PostTypeId="2" ParentId="5399" CreationDate="2010-12-21T05:30:02.967" Score="0" ViewCount="" Body="&lt;p&gt;suppose $(X,Y)$ is bivariate normal with zero means and correlation $\rho$. then &lt;/p&gt;&#xA;&#xA;&lt;p&gt;${\mathrm E} XY= cov(X,Y)= \rho\sigma_X\sigma_Y$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;all of the entries in the matrix $x_1x_2^T$ are of the form $XY$.&lt;/p&gt;&#xA;" OwnerUserId="1112" LastActivityDate="2010-12-21T05:30:02.967" CommentCount="2" />
  <row Id="5660" PostTypeId="2" ParentId="5602" CreationDate="2010-12-21T08:36:12.760" Score="0" ViewCount="" Body="&lt;p&gt;You can use SOM which is a kind of supervised clustering. Or as sesqu said their are tons of other algorithms such as support vector machines, regression or logistic regression. Neural Networks, etc ..&lt;/p&gt;&#xA;" OwnerUserId="1808" LastActivityDate="2010-12-21T08:36:12.760" />
  <row Id="5661" PostTypeId="2" ParentId="1875" CreationDate="2010-12-21T08:54:45.653" Score="2" ViewCount="" Body="&lt;p&gt;The &lt;a href=&quot;http://docs.lib.noaa.gov/rescue/mwr/078/mwr-078-01-0001.pdf&quot; rel=&quot;nofollow&quot;&gt;Brier Score&lt;/a&gt; approach is very simple and the most directly applicable way verify accuracy of a predicted outcome versus binary event.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Don't rely on just formulas ...plot the scores for different periods of time, data, errors, [weighted] rolling average of data, errors ... it's tough to say what visual analysis might reveal ... after you think you see something, you will better know what kind of hypothesis test to perform until AFTER you &lt;strong&gt;look&lt;/strong&gt; at the data.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Brier Score inherently assumes stability of the variation/underlying distributions weather and technology driving the forecasting models, lack of linearity, no bias, lack of change in bias ... it assumes that same general level of accuracy/inaccuracy is consistent.  As climate changes in ways that are not yet understood, the accuracy of weather predictions would decrease; conversely, the scientists feeding information to the weatherman have more resources, more complete models, more computing power so perhaps the accuracy of the predictions would increase.  Looking at the errors would tell something about stability, linearity and bias of the forecasts ... you may not have enough data to see trends; you may learn that stability, linearity and bias are not an issue.  You may learn that weather forecasts are getting more accurate ... or not.&lt;/p&gt;&#xA;" OwnerUserId="2342" LastEditorUserId="2342" LastEditorDisplayName="" LastEditDate="2010-12-21T22:21:01.123" LastActivityDate="2010-12-21T22:21:01.123" />
  <row Id="5662" PostTypeId="2" ParentId="5656" CreationDate="2010-12-21T10:16:38.273" Score="6" ViewCount="" Body="&lt;p&gt;It seems it's just a matter of &quot;releveling&quot; the factor; no need to compute partial sums or make a copy of the original vector. E.g.,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(101)&#xA;a &amp;lt;- factor(LETTERS[sample(5, 150, replace=TRUE, &#xA;                           prob=c(.1, .15, rep(.75/3,3)))])&#xA;p &amp;lt;- 1/5&#xA;lf &amp;lt;- names(which(prop.table(table(a)) &amp;lt; p))&#xA;levels(a)[levels(a) %in% lf] &amp;lt;- &quot;Other&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here, the original factor levels are distributed as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; A  B  C  D  E &#xA;18 23 35 36 38 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and then it becomes&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Other     C     D     E &#xA;   41    35    36    38 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It may be conveniently wrapped into a function. There is a &lt;code&gt;combine_factor()&lt;/code&gt; function in the &lt;a href=&quot;http://cran.r-project.org/web/packages/reshape/index.html&quot; rel=&quot;nofollow&quot;&gt;reshape&lt;/a&gt; package, so I guess it could be useful too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, as you seem interested in data mining, you might have a look at the &lt;a href=&quot;http://caret.r-forge.r-project.org/Classification_and_Regression_Training.html&quot; rel=&quot;nofollow&quot;&gt;caret&lt;/a&gt; package. It has a lot of useful features for data preprocessing, including functions like &lt;code&gt;nearZeroVar()&lt;/code&gt; that allows to flag predictors with very imbalanced distribution of observed values (See the vignette, &lt;a href=&quot;http://cran.r-project.org/web/packages/caret/vignettes/caretMisc.pdf&quot; rel=&quot;nofollow&quot;&gt;example data, pre-processing functions, visualizations and other functions&lt;/a&gt;, p. 5, for example of use).&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-12-21T10:16:38.273" CommentCount="10" />
  <row Id="5663" PostTypeId="2" ParentId="5115" CreationDate="2010-12-21T11:49:21.513" Score="4" ViewCount="" Body="&lt;p&gt;Florence Nightingale for being &quot;a true pioneer in the graphical representation of statistics&quot; and developing the polar area diagram. Yes, &lt;em&gt;that&lt;/em&gt; Florence Nightingale!&lt;/p&gt;&#xA;" OwnerUserId="2468" LastActivityDate="2010-12-21T11:49:21.513" CommunityOwnedDate="2010-12-21T11:49:21.513" />
  <row Id="5664" PostTypeId="1" AcceptedAnswerId="5665" CreationDate="2010-12-21T13:18:15.667" Score="5" ViewCount="201" Body="&lt;p&gt;I'm a complete newbie to statistics (although I find it really interesting!), and I have taken the task of distributing feedback to speakers of a conference I'm co-organizing. Each speaker was given a grade on the scale 1-5 from the participants, and we combine feedback from all participants into a mean score, for instance 3.56. We can then order the speakers by mean score.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In addition to giving the speakers their mean score, we also want to give them a clue about how they did compared to the other speakers. To avoid discouraging the speakers who did the worst (we want people to try again!), we came up with giving back a more fuzzy metric. We want to divide the speakers into four groups: 10% best/20%/30%/40% worst. Does there exist a name for this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt;:&#xA;I'll try an example. If I have ten talks with scores of { 1.2, 1.3, 2.1, 2.4, 2.7, 3.0, 3.2, 4.1, 4.2, 4.5}, I would divide them up like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;10% best: 4.5&lt;br&gt;&#xA;20% &quot;next best&quot;: 4.2, 4.1&lt;br&gt;&#xA;30% &quot;next worst&quot;: 3.2, 3.0, 2.7&lt;br&gt;&#xA;40% worst: 2.4, 2.1, 1.3, 1.2  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I want to know is if there is a name for these ranges.&lt;/p&gt;&#xA;" OwnerUserId="2470" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-12-22T03:18:16.753" LastActivityDate="2010-12-22T19:37:14.630" Title="Does there exist a name for comparing individual grades to a set of grades?" Tags="&lt;basic-concepts&gt;" AnswerCount="4" CommentCount="2" FavoriteCount="1" />
  <row Id="5665" PostTypeId="2" ParentId="5664" CreationDate="2010-12-21T13:37:09.843" Score="5" ViewCount="" Body="&lt;p&gt;If I understand you correctly, you may refer to &lt;a href=&quot;http://en.wikipedia.org/wiki/Percentile&quot; rel=&quot;nofollow&quot;&gt;Percentiles&lt;/a&gt;, perhaps espacially Quartiles.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Perhaps you can elaborate a little more on which percentages should be enclosed in each bin, to get a more accurate answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE:&lt;/strong&gt; Based on the comments below &lt;em&gt;decile&lt;/em&gt; seems to be the term you want. For your data these can easily achieved via &lt;code&gt;R&lt;/code&gt; (note the differences from your example):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; x = c(1.2, 1.3, 2.1, 2.4, 2.7, 3.0, 3.2, 4.1, 4.2, 4.5)&#xA;&amp;gt; quantile(x,c(0.9,0.8,0.7,0.6))&#xA; 90%  80%  70%  60% &#xA;4.23 4.12 3.47 3.08&#xA;&amp;gt; x[x &amp;gt; quantile(x,0.9)]&#xA;[1] 4.5&#xA;&amp;gt; x[x &amp;gt; quantile(x,0.8) &amp;amp; x &amp;lt; quantile(x,0.9)]&#xA;[1] 4.2&#xA;&amp;gt; x[x &amp;gt; quantile(x,0.7) &amp;amp; x &amp;lt; quantile(x,0.8)]&#xA;[1] 4.1&#xA;&amp;gt; x[x &amp;lt; quantile(x,0.7)]&#xA;[1] 1.2 1.3 2.1 2.4 2.7 3.0 3.2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can then tell the speakers you are in the 10th decile, 9th decile, 8th decile, or 7th and worse decile (or something with above the 90th percentile, ...). But from my point, the problem will always be to name the catch all (i.e. worst) category.&lt;/p&gt;&#xA;" OwnerUserId="442" LastEditorUserId="442" LastEditorDisplayName="" LastEditDate="2010-12-21T14:34:48.360" LastActivityDate="2010-12-21T14:34:48.360" CommentCount="7" />
  <row Id="5666" PostTypeId="2" ParentId="5654" CreationDate="2010-12-21T14:10:44.480" Score="2" ViewCount="" Body="&lt;p&gt;Let's take a stab at a first-order approximation assuming simple random sampling and a constant proportion of infection for any treatment.  Assume the sample size is large enough that a normal approximation can be used in a hypothesis test on proportions so we can calculate a z statistic like so&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$z = \frac{p_t - p_0}{\sqrt{p_0(1-p_0)(\frac{1}{n_1}+\frac{1}{n_2})}}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is the sample statistic for a two-sample test, new formula vs. bleach, since we expect the effect of bleach to be random as well as the effect of the new formula.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then let $n = n_1 = n_2$, since balanced experiments have the greatest power, and use your specifications that $|p_t - p_0| \geq 0.1$, $p_0 = 0.2$.  To attain a test statistic $|z| \geq 2$ (Type I error of about 5%), this works out to $n \approx 128$.  This is a reasonable sample size for the normal approximation to work, but it's definitely a lower bound.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd recommend doing a similar calculation based on the desired power for the test to control Type II error, since an underpowered design has a high probability of missing an actual effect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you've done all this basic spadework, start looking at the stuff &lt;strong&gt;whuber&lt;/strong&gt; addresses.  In particular, it's not clear from your problem statement whether the samples of poultry measured are different groups of subjects, or the same groups of subjects.  If they're the same, you're into paired t test or repeated measures territory, and you need someone smarter than me to help out!&lt;/p&gt;&#xA;" OwnerUserId="1920" LastActivityDate="2010-12-21T14:10:44.480" CommentCount="2" />
  <row Id="5667" PostTypeId="2" ParentId="726" CreationDate="2010-12-21T15:48:06.727" Score="4" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;The primary product of a research&#xA;  inquiry is one or more measures of&#xA;  effect size, not p values.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Cohen, J. (1990). &lt;a href=&quot;http://www.cps.nova.edu/marker/whatIhavelearnedsofarcohen.pdf&quot; rel=&quot;nofollow&quot;&gt;Things I have learned (so far)&lt;/a&gt;. &lt;em&gt;American Psychologist&lt;/em&gt;, 45, 1304-1312.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-12-21T15:48:06.727" CommunityOwnedDate="2010-12-21T15:48:06.727" />
  <row Id="5668" PostTypeId="2" ParentId="5664" CreationDate="2010-12-21T16:02:19.503" Score="3" ViewCount="" Body="&lt;p&gt;You have the answer you asked for, but along with how to communicate this information you might also want to think about how to asses the reliability &amp;amp; precision of the scores. If the evaluators aren't really using the same standards, the scores will furnish a misleading measure of the quality of the speakers no matter how you decide to categorize them.  Also, even if the evaluators are reliable in this sense, your rankings should be sensitive to the standard error in their measurements (likely to be large if you have only a modest number of evaluators): you don't want to imply that there are meaningful differences among speakers whose scores differ by amounts that are comparable to the level of background noise in your data. If you aren't in a position to furnish genuinely informative quantitative feedback, you are better off, in my view, picking one or two evaluators you trust to give the speakers' qualitative feedback informed by the evaluators' own observations &amp;amp; by their assessments of whatever evidence they have on the reactions of others.&lt;/p&gt;&#xA;" OwnerUserId="2347" LastEditorUserId="2347" LastEditorDisplayName="" LastEditDate="2010-12-21T18:29:28.220" LastActivityDate="2010-12-21T18:29:28.220" CommentCount="2" />
  <row Id="5669" PostTypeId="2" ParentId="859" CreationDate="2010-12-21T16:08:11.367" Score="1" ViewCount="" Body="&lt;p&gt;Look into models with &lt;strong&gt;spatially correlated errors&lt;/strong&gt; (and spatially correlated covariates).  A brief introduction, with references to &lt;a href=&quot;http://geodacenter.asu.edu/&quot; rel=&quot;nofollow&quot;&gt;GeoDa&lt;/a&gt;, is available &lt;a href=&quot;http://www.s4.brown.edu/s4/courses/SO261-John/lab9.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.  There are plenty of texts; good ones are by &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0471002550&quot; rel=&quot;nofollow&quot;&gt;Noel Cressie&lt;/a&gt;, &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0521774373&quot; rel=&quot;nofollow&quot;&gt;Robert Haining&lt;/a&gt;, and &lt;a href=&quot;http://www2.fiu.edu/~tardanic/spatial.pdf&quot; rel=&quot;nofollow&quot;&gt;Fotheringham et al&lt;/a&gt; (the last link goes to a summary, not a book site).  Some R code has recently been emerging but I'm unfamiliar with it.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-12-21T16:08:11.367" />
  <row Id="5670" PostTypeId="2" ParentId="5619" CreationDate="2010-12-21T16:44:46.750" Score="3" ViewCount="" Body="&lt;p&gt;I see no obvious reason not to do so. As far as I know, we usually make a distinction between two kind of effect size (ES) measures for qualifying the strength of an observed association: ES based on $d$ (difference of means) and ES based on $r$ (correlation). The latter includes Pearson's $r$, but also Spearman's $\rho$, Kendall's $\tau$, or the multiple correlation coefficient.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for their interpretation, I think it mainly depends on the field you are working in: A correlation of .20 would certainly not be interpreted in the same way in psychological &lt;em&gt;vs.&lt;/em&gt; software engineering studies. Don't forget that Cohen's three-way classification--small, medium, large--was based on behavioral data, as discussed in Kraemer et al. (2003), p. 1526. In their Table 1, they made no distinction about the different types of ES measures belonging to the $r$ family. There have by no way an absolute meaning and should be interpreted with reference to established results or literature review.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to add some other references that provide useful reviews of common ES measures and their interpretation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;References&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Helena C. Kraemer, George A. Morgan, Nancy L. Leech, Jeffrey A. Gliner, Jerry J. Vaske, and Robert J. Harmon (2003). &lt;a href=&quot;http://psy6023.alliant.wikispaces.net/file/view/Kraemer2003.pdf&quot; rel=&quot;nofollow&quot;&gt;Measures of Clinical Significance&lt;/a&gt;. &lt;em&gt;J Am Acad Child Adolesc Psychiatry&lt;/em&gt;, 42(12), 1524-1529.&lt;/li&gt;&#xA;&lt;li&gt;Christopher J. Ferguson (2009). &lt;a href=&quot;http://www.tamiu.edu/~cferguson/Ferguson%20PPRP.pdf&quot; rel=&quot;nofollow&quot;&gt;An Effect Size Primer: A Guide for Clinicians and Researchers&lt;/a&gt;. &lt;em&gt;Professional Psychology: Research and Practice&lt;/em&gt;, 40(5), 532-538.&lt;/li&gt;&#xA;&lt;li&gt;Edward F. Fern and Kent B. Monroe (1996). &lt;a href=&quot;http://www.wisedata.hk/readings/Topic%2015_Secondary%20data/3.%20Effect%20size%20and%20significance%20testing/Effect-Size%20Estimates_%20Issues%20and%20Problems%20in%20Interpretation.pdf&quot; rel=&quot;nofollow&quot;&gt;Effect-Size Estimates: Issues and Problems in Interpretation&lt;/a&gt;. &lt;em&gt;Journal of Consumer Research&lt;/em&gt;, 23, 89-105.&lt;/li&gt;&#xA;&lt;li&gt;Daniel J. Denis (2003). &lt;a href=&quot;http://theoryandscience.icaap.org/content/vol4.1/02_denis.html&quot; rel=&quot;nofollow&quot;&gt;Alternatives to Null Hypothesis Significance Testing&lt;/a&gt;. &lt;em&gt;Theory and Science&lt;/em&gt;, 4(1).&lt;/li&gt;&#xA;&lt;li&gt;Paul D. Ellis (2010). &lt;em&gt;&lt;a href=&quot;http://www.cambridge.org/gb/knowledge/isbn/item2704152/?site_locale=en_GB&quot; rel=&quot;nofollow&quot;&gt;The Essential Guide to Effect Sizes&lt;/a&gt;&lt;/em&gt;. Cambridge University Press. -- just browsed the TOC&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-12-21T17:02:01.640" LastActivityDate="2010-12-21T17:02:01.640" CommentCount="5" />
  <row Id="5671" PostTypeId="2" ParentId="5207" CreationDate="2010-12-21T17:16:52.633" Score="1" ViewCount="" Body="&lt;p&gt;In fact, you should not do MCMC, since your problem is so much simpler. Try this algorithm:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Step 1: Generate a X from Log Normal &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Step 2: Keeping this X fixed, generate a Y from the Singh Maddala.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Voila! Sample Ready!!!&lt;/p&gt;&#xA;" OwnerUserId="2472" LastActivityDate="2010-12-21T17:16:52.633" CommentCount="3" />
  <row Id="5672" PostTypeId="2" ParentId="5054" CreationDate="2010-12-21T17:23:16.220" Score="1" ViewCount="" Body="&lt;p&gt;Maybe I'm missing something here, but if you plot the number of times these hashtags are mentioned over time, shouldn't that tell you something?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, maybe you need automated processing. In that case fit splines to these series, and take the derivative. (They are easy : just look up what the Function Data Analysts do.) Sharply trending topics will have high derivatives. How high, will come from your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do tell if this worked or not?&lt;/p&gt;&#xA;" OwnerUserId="2472" LastActivityDate="2010-12-21T17:23:16.220" />
  <row Id="5674" PostTypeId="2" ParentId="5111" CreationDate="2010-12-21T18:22:10.200" Score="0" ViewCount="" Body="&lt;p&gt;A well thought out answer! Thanks.&lt;/p&gt;&#xA;" OwnerUserId="2475" LastActivityDate="2010-12-21T18:22:10.200" CommentCount="1" />
  <row Id="5675" PostTypeId="1" AcceptedAnswerId="5676" CreationDate="2010-12-21T19:37:50.770" Score="2" ViewCount="396" Body="&lt;p&gt;When computing a confidence interval of slope in linear regression, should you use the z- or t-statistic?&lt;/p&gt;&#xA;" OwnerUserId="1395" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-21T20:59:47.620" LastActivityDate="2010-12-22T07:04:10.727" Title="Confidence interval of slope in linear regression" Tags="&lt;regression&gt;&lt;confidence-interval&gt;" AnswerCount="3" CommentCount="1" />
  <row Id="5676" PostTypeId="2" ParentId="5675" CreationDate="2010-12-21T19:43:52.090" Score="5" ViewCount="" Body="&lt;p&gt;If you're doing linear regression using least squares, you should use base confidence intervals on Student's &lt;em&gt;t&lt;/em&gt;-distribution.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-12-21T19:43:52.090" CommentCount="1" />
  <row Id="5677" PostTypeId="2" ParentId="5675" CreationDate="2010-12-21T20:10:22.910" Score="3" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Rule of thumb: Use Student's &lt;em&gt;t&lt;/em&gt; distribution if you must estimate the variance.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since the distribution's variance is estimated (not known), you should use Student's &lt;em&gt;t&lt;/em&gt; distribution rather than the standard normal distribution (&lt;em&gt;z&lt;/em&gt;), which requires a known variance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Although the &lt;em&gt;t&lt;/em&gt; distribution becomes almost exactly the same as the &lt;em&gt;z&lt;/em&gt; distribution when the degrees of freedom (think size of the sample) are large, it is (in my experience) quite rare that the &lt;em&gt;z&lt;/em&gt; distribution is used instead of the &lt;em&gt;t&lt;/em&gt; distribution in cases like this.&lt;/p&gt;&#xA;" OwnerUserId="1583" LastEditorUserId="1583" LastEditorDisplayName="" LastEditDate="2010-12-21T21:18:12.697" LastActivityDate="2010-12-21T21:18:12.697" CommentCount="8" />
  <row Id="5678" PostTypeId="2" ParentId="5054" CreationDate="2010-12-21T20:21:54.227" Score="1" ViewCount="" Body="&lt;p&gt;As far as I can tell from my reading, a common method for determining a time series' trend is to smooth the series, perhaps in an iterated fashion, as in:&#xA;&lt;a href=&quot;http://www.sbp.org.pk/departments/stats/sam.pdf&quot; rel=&quot;nofollow&quot;&gt;A Pakistan SBP paper&lt;/a&gt;. In the Seasonal Adjustment Methodology section, it describes how X-12 ARIMA does it, though they also use a seasonal factor which perhaps you could also use or perhaps you could simply ignore.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other links might include &lt;a href=&quot;http://www.bankofengland.co.uk/mfsd/iadb/notesiadb/seasonal_adjustment.htm&quot; rel=&quot;nofollow&quot;&gt;A Bank of England web page&lt;/a&gt; and &lt;a href=&quot;http://www.census.gov/ts/papers/jbes98.pdf&quot; rel=&quot;nofollow&quot;&gt;A US Census Bureau paper&lt;/a&gt; (pages 8-12).&lt;/p&gt;&#xA;" OwnerUserId="1764" LastEditorUserId="1764" LastEditorDisplayName="" LastEditDate="2010-12-21T20:55:04.573" LastActivityDate="2010-12-21T20:55:04.573" />
  <row Id="5679" PostTypeId="2" ParentId="5675" CreationDate="2010-12-21T21:10:43.467" Score="2" ViewCount="" Body="&lt;p&gt;Depends on assumptions on your disturbances. If they are normal and homoscedastic, then yes use t-statistic. In economic applications though these assumptions rarely hold, so in that case I would suggest using z-statistic with &lt;a href=&quot;http://en.wikipedia.org/wiki/White_standard_errors&quot; rel=&quot;nofollow&quot;&gt;robust standard errors&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2010-12-22T07:04:10.727" LastActivityDate="2010-12-22T07:04:10.727" CommentCount="1" />
  <row Id="5680" PostTypeId="1" AcceptedAnswerId="5946" CreationDate="2010-12-21T21:38:00.003" Score="6" ViewCount="677" Body="&lt;p&gt;I have analyzed an experiment with a repeated measures ANOVA. The ANOVA is a 3x2x2x2x3 with 2 between-subject factors and 3 within (N = 189). Error rate is the dependent variable. The distribution of error rates has a skew of 3.64 and a kurtosis of 15.75. The skew and kurtosis are the result of 90% of the error rate means being 0. Reading some of the previous threads on normality tests here has me a little confused. I thought that if you had data that was not normally distributed it was in your best interest to transform it if possible, but it seems that a lot of people think analyzing non-normal data with an ANOVA or a T-test is acceptable. Can I trust the results of the ANOVA?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(FYI, In the future I intend to analyze this type of data in R with mixed-models with a binomial distribution)&lt;/p&gt;&#xA;" OwnerUserId="2322" LastActivityDate="2011-01-04T19:19:39.637" Title="Can I trust ANOVA results for a non-normally distributed DV?" Tags="&lt;anova&gt;&lt;normality&gt;" AnswerCount="4" CommentCount="8" />
  <row Id="5681" PostTypeId="1" CreationDate="2010-12-22T00:04:01.290" Score="1" ViewCount="95" Body="&lt;p&gt;Measurement outcome is a continuous number measured in $p$ locations.&#xA;This yields a vector $X_1$ to $X_p$ per subject. These vectors are obtained on $N$ subjects.&#xA;In addition these $N$ subjects come from $k$ groups.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What method should I use to compare the average vector across the entire study population to an hypothesized null?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And how to compare between the $k$ groups the per group average vector overall and for individual $X_i$ locations?&lt;/p&gt;&#xA;" OwnerUserId="2478" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-22T11:31:41.020" LastActivityDate="2010-12-22T11:47:26.107" Title="Comparing a vector of averages to a null one" Tags="&lt;multiple-comparisons&gt;&lt;multivariate-analysis&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="5682" PostTypeId="1" CreationDate="2010-12-22T01:19:57.423" Score="2" ViewCount="373" Body="&lt;p&gt;I was wondering if anyone had experience with SAS/IML and R, and can give some pointers as to the relative advantages/disadvantages of the two. I've used R extensively for programming and statistical analyses, but haven't had much experience with IML. However, as this company is a SAS shop, I'm likely to have to use it (the alternative is to program everything in base SAS, which sometimes leads to true horrors).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="1569" LastActivityDate="2011-01-13T20:49:25.670" Title="SAS/IML compared to R" Tags="&lt;r&gt;&lt;sas&gt;" AnswerCount="3" />
  <row Id="5683" PostTypeId="2" ParentId="5682" CreationDate="2010-12-22T01:26:07.817" Score="2" ViewCount="" Body="&lt;p&gt;I have never used it, but I know for new versions of IML, you can call R routines. Maybe start by looking at &lt;a href=&quot;http://support.sas.com/rnd/app/studio/statr.pdf&quot; rel=&quot;nofollow&quot;&gt;http://support.sas.com/rnd/app/studio/statr.pdf&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2040" LastActivityDate="2010-12-22T01:26:07.817" />
  <row Id="5684" PostTypeId="2" ParentId="5664" CreationDate="2010-12-22T03:23:21.193" Score="3" ViewCount="" Body="&lt;p&gt;In addition to the existing answers, you may find it useful to read up about test norms, a well established topic in psychology and education.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.eric.ed.gov/ERICWebPortal/search/detailmini.jsp?-nfpb=true&amp;amp;_&amp;amp;ERICExtSearch_SearchValue_0=ED104921&amp;amp;ERICExtSearch_SearchType_0=no&amp;amp;accno=ED104921&quot; rel=&quot;nofollow&quot;&gt;Test Norms: Their Use and Interpretation.&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://psychassessment.com.au/PDF/Chapter%2004.pdf&quot; rel=&quot;nofollow&quot;&gt;http://psychassessment.com.au/PDF/Chapter%2004.pdf&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.google.com.au/search?hl=en&amp;amp;source=hp&amp;amp;q=test+norms&amp;amp;btnG=Google+Search&amp;amp;aq=f&amp;amp;aqi=&amp;amp;aql=&amp;amp;oq=&amp;amp;gs_rfai=&quot; rel=&quot;nofollow&quot;&gt;Google 'test norms'&lt;/a&gt; &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-12-22T03:23:21.193" CommentCount="1" />
  <row Id="5685" PostTypeId="2" ParentId="5681" CreationDate="2010-12-22T04:48:57.077" Score="1" ViewCount="" Body="&lt;p&gt;It may not make sense to take an unweighted average of all $N$ observations if the subjects are divided among $k$ subpopulations [=groups] - unless the sample sizes in the groups are proportional to the sizes of the subpopulations. Otherwise, what characteristic[s] of interest would an unweighted average estimate? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your sample design seems to be that of a 1-way MANOVA - so you could finesse the issue of unequal treatment [=group] means by including [vector] group-effect terms in a MANOVA model. Then the logical thing is to test first for no group differences [with the appropriate MANOVA procedure] - and if it appears there are none - the $N$ observations can then be averaged and tested using Hotelling's-$T^2$.&lt;/p&gt;&#xA;" OwnerUserId="1112" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-22T11:47:26.107" LastActivityDate="2010-12-22T11:47:26.107" CommentCount="2" />
  <row Id="5686" PostTypeId="1" CreationDate="2010-12-22T06:52:16.957" Score="7" ViewCount="340" Body="&lt;p&gt;I would like to know why some languages like R has both NA and NaN. What are the differences or are they equally the same? Is it really needed to have NA?&lt;/p&gt;&#xA;" OwnerUserId="2479" LastActivityDate="2010-12-23T17:23:28.107" Title="What is the difference between NaN and NA?" Tags="&lt;r&gt;" AnswerCount="3" CommentCount="3" />
  <row Id="5687" PostTypeId="2" ParentId="5686" CreationDate="2010-12-22T07:08:13.557" Score="13" ViewCount="" Body="&lt;p&gt;Hello User2479&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;?is.nan&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;?is.na&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;?NA &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;?NaN&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Should answer your question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But, in short: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;NaN means $\frac {0} {0}$ -- Stands for Not a Number&lt;/p&gt;&#xA;&#xA;&lt;p&gt;NA  is generally interpreted as a missing value and has various forms - NA_integer_, NA_real_, etc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, NaN $\neq$ NA and there is a need for NaN and NA.&lt;/p&gt;&#xA;" OwnerUserId="1307" LastActivityDate="2010-12-22T07:08:13.557" CommentCount="2" />
  <row Id="5688" PostTypeId="2" ParentId="5686" CreationDate="2010-12-22T07:09:16.243" Score="4" ViewCount="" Body="&lt;p&gt;NA is for missing data. NaN, as J.M. said is for arithmetic purpose. NaN is usually the product of some arithmetic operation, such as &lt;code&gt;0/0&lt;/code&gt;. NA usually is declared in advance, or is a product of operation when you try to access something that is not there:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; &amp;gt; a &amp;lt;- c(1,2)&#xA; &amp;gt; a[3]&#xA;   [1] NA &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2010-12-22T07:09:16.243" />
  <row Id="5689" PostTypeId="2" ParentId="5680" CreationDate="2010-12-22T07:40:28.613" Score="0" ViewCount="" Body="&lt;p&gt;You state;  FYI, In the future I intend to analyze this type of data in R with mixed-models with a binomial distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That's not ANOVA by a long shot, either, then.  Is it?&lt;/p&gt;&#xA;" OwnerUserId="2480" LastActivityDate="2010-12-22T07:40:28.613" CommentCount="2" />
  <row Id="5690" PostTypeId="1" AcceptedAnswerId="9097" CreationDate="2010-12-22T09:55:50.503" Score="4" ViewCount="418" Body="&lt;p&gt;I'm looking into median survival using Kaplan-Meier in different states for a type of cancer. There are quite big differences between the states. How can i compare the median survival between all the states and determine which ones are significantly different from the mean median survival all across the country?&lt;/p&gt;&#xA;" OwnerUserId="1291" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-29T01:09:38.913" LastActivityDate="2011-04-29T01:09:38.913" Title="How to compare median survival between groups?" Tags="&lt;multiple-comparisons&gt;&lt;survival&gt;" AnswerCount="2" CommentCount="3" FavoriteCount="1" />
  <row Id="5691" PostTypeId="1" AcceptedAnswerId="5692" CreationDate="2010-12-22T11:13:56.733" Score="2" ViewCount="214" Body="&lt;p&gt;I wish to create a confidence interval for a statistic calculated on a table (let's say the p.value of chisqr.test).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For this, I might sample a bootstrap sample with the same size, from a table with the proportions of the table I have.  And my questions are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Is this a correct procedure to use, or is there some catch here that I might be missing?&lt;/li&gt;&#xA;&lt;li&gt;How can a table be &quot;sampled&quot; in R?  (my first instinct is to &quot;open up&quot; the table using melt, then expend the table to rows, sample them, and fold the results back to a table.  Is there a better way?)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;p.s: I am aware of the &quot;simulate.p.value&quot; switch in chisqr.test - my question is more general - does it make sense to use bootstrap in such a way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sample table if someone wishes to show something:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(x &amp;lt;- matrix(c(12, 5, 7, 7), ncol = 2))&#xA;prop.table(x)&#xA;&#xA;# undoing a table&#xA;as.data.frame(as.table(x)) # but how do we open up the rows now? - **update**: this was answered in the comments thanks to chl.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="253" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-12-22T14:20:56.223" LastActivityDate="2010-12-25T04:24:25.403" Title="Is it o.k. to bootstrap sample of a table from its proportions - and how to do so (in R)?" Tags="&lt;r&gt;&lt;chi-squared&gt;&lt;bootstrap&gt;&lt;contingency-tables&gt;" AnswerCount="3" CommentCount="5" />
  <row Id="5692" PostTypeId="2" ParentId="5691" CreationDate="2010-12-22T11:46:08.653" Score="3" ViewCount="" Body="&lt;p&gt;Hello Tal:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let me see if I understand you correctly:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You have a contingency table say $M$ by $N$ and you calculate a statistic based on this table. You want a CI for this statistic and you don't have a theoretical CI (or may be you don't want to use it, due to some reason). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Generally, if you have a contingency table, the columns represent the possible levels of the response variable (here: $Y$ will have $N$ levels) and the rows represent the levels of the explanatory variables (here: $X$ will have $M$ levels). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now if you want to bootstrap, you need to make sure that you couple $Y$s and $X$s together. That is, shuffle $(X_i, Y_i)$ together for $i$ from $1$ to total number of observations (note this will be greater than $MN$ unless all the entries in the table are $1$).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically, what you can do is translate your table to a &quot;hypothetical&quot; data set set by using row numbers as levels for your $X$s and the column number as the value for the corresponding $Y$s.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To make it mathematically clear:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lets say your $(i, j)^{th}$ entry in the table is $a$, then you need to create, $a$ data points for which the value of $Y$ is $j$ (remember column number for $Y$ value) and $X$ value is $i$. Do the same for $i = 1 \ldots m$ and $j = 1 \ldots n$. Now you have a &quot;hypothetical&quot; data set. Bootstrap from this data set as you would do in linear regression, say $B$ number of times. Each of the newly generated data set is a bootstrap sample. For each of the bootstrapped sample calculate the corresponding contingency table (say for $Y=1$ how many $X$s=1, how may $X$s=2, ... , this will give you the (1,1), (2,1),... entry for your table). Calculate the statistic for each of the $B$ tables and you have the desired bootstrap distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't know if this is what you meant by bootstrapping the table. Please let me know if you thought otherwise.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;HTH&lt;/p&gt;&#xA;&#xA;&lt;p&gt;S.&lt;/p&gt;&#xA;" OwnerUserId="1307" LastActivityDate="2010-12-22T11:46:08.653" CommentCount="1" />
  <row Id="5693" PostTypeId="2" ParentId="4687" CreationDate="2010-12-22T12:52:08.763" Score="3" ViewCount="" Body="&lt;p&gt;First of all, your usage of the term &quot;prior probability&quot; seems to be wrong. For any node N with discrete values $n_i$ the probability that a certain value of N occurs &lt;em&gt;a priori&lt;/em&gt; is $p(N=n_i)$. If a node has no parents, one is interested in calculate this prob. But if a node has parents P, one is interested in calculating the &lt;em&gt;conditional probability&lt;/em&gt;, i.e. the probability that a certain value of N occurs &lt;em&gt;given&lt;/em&gt; its parents. This is $p(N=n_i|P)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regarding your actual question: How to calculate the conditional probabilities of a child node if the parents are continuous and the child node is a) discrete or b) continuous. I will to explain one approach using a concrete example, but I hope the general idea will be clear though.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider this network:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/HkcnB.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where subsidy and buys are discrete nodes meanwhile harvest and cost are continuous. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;a)&lt;/strong&gt; p(Cost|Subsidy, Harvest)&#xA;Two options:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Discretize Harvest and treat it as discrete =&gt; maybe information loss&lt;/li&gt;&#xA;&lt;li&gt;Model a mapping from the current value of harvest to a parameter describing the distribution of Cost.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Details to 2.:&#xA;Let's assume cost can be modeled using a normal distribution. In this case it is common to fix the variance and map the value of Harvest linearly to the mean of gaussian. The parent subsidy (binary) only add the constraint to create a separate distribution for subsidy=true and subsidy=false.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Result:&#xA;$p(Cost=c|Harvest=h,Subsidy=true)=N(\alpha_1 * h + \beta_1,\sigma_1^2)(c)$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$p(Cost=c|Harvest=h,Subsidy=false)=N(\alpha_2 * h + \beta_2,\sigma_2^2)(c)$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;for some $\alpha$s and $\beta$s. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;b)&lt;/strong&gt; p(Buys|Cost)&#xA;In this case on need to map the probability of occurrence of certain costs to the probability of Buy=True (p(Buy=False) = 1-p(Buy=True)). (Note that this is the same task as in logistic regression). One approach is if the parent has a normal distribution, to calculate the integral from 0 to x of a standard normal distribution, where x is the z-transformed value of the parent. In our example:&#xA;$p(Buys=true|Cost=c) = integral(0,\frac{-c+\mu}{\sigma})$ with $\mu$=mean and $\sigma$=standard deviation of the Cost distribution. In this case $-c+\mu$ instead of $\mu-c$, because an observation (knowledge extracted from data !) is, that the lower the cost the more probable is a buy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the case of a non-binary discrete child-node, one approach is to transform the one multi-value-problem into multiple binary problems. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Example-Source / Further reading:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;&quot;Artificial Intelligence: A modern approach&quot;&lt;/em&gt; by Russell and Norvig&lt;/p&gt;&#xA;" OwnerUserId="264" LastEditorUserId="264" LastEditorDisplayName="" LastEditDate="2010-12-22T13:04:42.997" LastActivityDate="2010-12-22T13:04:42.997" CommentCount="3" />
  <row Id="5694" PostTypeId="2" ParentId="5691" CreationDate="2010-12-22T13:05:52.840" Score="1" ViewCount="" Body="&lt;p&gt;I once asked similar &lt;a href=&quot;http://stackoverflow.com/questions/2281561/random-sample-from-given-bivariate-discrete-distribution&quot;&gt;question on stackoverflow&lt;/a&gt;. Basically you sample from table the same way you sample vector. &lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2010-12-22T13:05:52.840" CommentCount="1" />
  <row Id="5695" PostTypeId="2" ParentId="2891" CreationDate="2010-12-22T14:21:26.737" Score="1" ViewCount="" Body="&lt;p&gt;If these are exclusive behaviours - they must either turn left or right and can't go straight on, stop or anything else - then you have data that you might assume are binomially distributed: in the example you give there are then 1042 left turns (the 'ones') in 1084 'runs' which you are implicitly assuming to be independent observations.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Testing&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could either use a binomial distribution to test whether this is consistent with a particular true proportion of left turns, say 90%.  In R, the test against 90% is &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;binom.test(1042, n=1084, p=.90)&lt;/code&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;and is rejected (and &lt;code&gt;chisq.test&lt;/code&gt; agrees).  You probably have covariates though and perhaps non-independent draws via clustering etc.  For these, switch to a binomial logistic regression framework.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Plotting&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For this data you should probably plot the &lt;em&gt;empirical logits&lt;/em&gt;: here log(1042/42) = 3.211228, or their posterior means (see below).  Visually this quantity represents a proportional increase and decrease in the counts as an equally sized increment up or down.  A symmetrical representation in proportional rather than absolute terms is usually what you want for this sort of data.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can get a quite a well-behaved confidence interval for the empirical logit via a Bayesian argument: the posterior distribution, assuming an invariant 'Jeffreys' Prior of $\text{Beta}(0.5,0.5)$, of the empirical logit is Normal with mean $\mu = \log{(\text{left}/\text{right})}$ and standard deviation $\sigma_\mu = (\text{left} + 0.5)^{-1} + (\text{right} + 0.5)^{-1}$.&lt;/p&gt;&#xA;" OwnerUserId="1739" LastActivityDate="2010-12-22T14:21:26.737" CommentCount="1" />
  <row Id="5696" PostTypeId="1" CreationDate="2010-12-22T15:41:17.387" Score="4" ViewCount="144" Body="&lt;p&gt;I have multiple independent coders who are trying to identify events in a time series -- in this case, watching video of face-to-face conversation and looking for particular nonverbal behaviors (e.g., head nods) and coding the time and category of each event.  This data could reasonable be treated as a discrete-time series with a high sampling rate (30 frames/second) or as a continuous-time series, whichever is easier to work with.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd like to compute some measure of inter-rater reliability, but I expect there to be some uncertainty in &lt;em&gt;when&lt;/em&gt; events occurred; that is, I expect that one coder might, for example, code that a particular movement began a quarter second later than other coders thought it started.  These are rare events, if that helps; typically at least several seconds (hundreds of video frames) between events.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a good way of assessing inter-rater reliability that looks at both of these kinds of agreement and disagreement: (1) do raters agree on &lt;em&gt;what&lt;/em&gt; event occurred (if any), and (2) do they agree on &lt;em&gt;when&lt;/em&gt; it occurred?  The second is important to me because I'm interested in looking at the timing of these events relative to other things happening in the conversation, like what people are saying.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Standard practice in my field seems to be to divide things up into time slices, say 1/4 of a second or so, aggregate the events each coder reported per time slice, then compute Cohen's kappa or some similar measure.  But the choice of slice duration is ad-hoc, and I don't get a good idea of uncertainty in time of events.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Best thought I have so far is that I could compute some kind of reliability curve; something like kappa as a function of the size of the window within which I consider two events as being coded at the same time.  I'm not really sure where to go from there, though...&lt;/p&gt;&#xA;" OwnerUserId="2484" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-12-22T15:59:33.153" LastActivityDate="2011-05-29T23:51:42.810" Title="Interrater reliability for events in a time series with uncertainty about event time" Tags="&lt;time-series&gt;&lt;reliability&gt;&lt;inter-rater&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="5697" PostTypeId="2" ParentId="3173" CreationDate="2010-12-22T16:13:40.510" Score="0" ViewCount="" Body="&lt;p&gt;Hi All&#xA;I would like to know why it must by add the term (&lt;strong&gt;player - 1&lt;/strong&gt; | month) in the model in the place of (player|month) directly&#xA;Thanks&lt;/p&gt;&#xA;" OwnerUserId="2486" LastActivityDate="2010-12-22T16:13:40.510" CommentCount="1" />
  <row Id="5698" PostTypeId="1" AcceptedAnswerId="5699" CreationDate="2010-12-22T16:16:54.973" Score="1" ViewCount="109" Body="&lt;p&gt;Please forgive me if this is not the right Stack Exchange (and for inventing terms).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For discrete random variables X and Y, the mutual information of X and Y can be defined as follows:&#xA;$I(X;Y) = \sum_{y \in Y} \sum_{x \in X} &#xD;&#xA;                 p(x,y) \log{ \left( \frac{p(x,y)}{p_1(x)\,p_2(y)}&#xD;&#xA;                              \right) }, \,\!$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I will define the mutual information of a &quot;cell&quot; $x_0$ to be:&#xA;$CI(x_0,Y) = \sum_{y \in Y} p(x_0,y) \log{ \left( \frac{p(x_0,y)}{p_1(x_0)\,p_2(y)}&#xD;&#xA;                              \right) }, \,\!$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not sure if this quantity goes by another name.  Essentially I'm restricting focus to a single state of variable X (and then the full MI can be calculated by summing all the cell MIs).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question: is it guaranteed that $CI(x_0,Y) \ge 0$?  We know $I(X;Y)\ge0$ and we know that the pointwise mutual information can be negative.  I feel like CI should be nonnegative and that I might be missing some obvious proof.&lt;/p&gt;&#xA;" OwnerUserId="2485" LastEditorUserId="2485" LastEditorDisplayName="" LastEditDate="2010-12-22T16:51:18.393" LastActivityDate="2010-12-22T18:17:14.713" Title="Can the mutual information of a &quot;cell&quot; be negative?" Tags="&lt;probability&gt;&lt;mutual-information&gt;&lt;information-theory&gt;&lt;probability-theory&gt;" AnswerCount="1" />
  <row Id="5699" PostTypeId="2" ParentId="5698" CreationDate="2010-12-22T17:16:24.587" Score="0" ViewCount="" Body="&lt;p&gt;Answered on &lt;a href=&quot;http://cstheory.stackexchange.com/questions/3939/can-the-mutual-information-of-a-cell-be-negative/3943#3943&quot;&gt;Theoretical Computer Science&lt;/a&gt;.  CI cannot be negative.&lt;/p&gt;&#xA;" OwnerUserId="2485" LastEditorUserId="2485" LastEditorDisplayName="" LastEditDate="2010-12-22T18:17:14.713" LastActivityDate="2010-12-22T18:17:14.713" />
  <row Id="5700" PostTypeId="1" CreationDate="2010-12-22T17:44:21.793" Score="4" ViewCount="370" Body="&lt;p&gt;Greetings,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm performing research that will help determine the size of observed space and the time elapsed since the big bang. Hopefully you can help!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have data conforming to a piecewise linear function on which I want to perform two linear regressions. There is a point at which the slope and intercept change, and I need to (write a program to) find this point.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thoughts?&lt;/p&gt;&#xA;" OwnerUserId="2488" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-02-28T12:34:24.513" LastActivityDate="2011-02-28T12:34:24.513" Title="Finding the change point in data from a piecewise linear function" Tags="&lt;regression&gt;&lt;change-point&gt;" AnswerCount="5" CommentCount="1" FavoriteCount="4" />
  <row Id="5701" PostTypeId="2" ParentId="5700" CreationDate="2010-12-22T17:56:04.617" Score="3" ViewCount="" Body="&lt;p&gt;If the number of points is not too big, you may try all possibilities. Let's assume that the points are $X_i=(x_i,y_i)$ where $i=1,..,N$. Than, you may loop with $j$ from $2$ to $N-2$ and fit two lines to both $\{X_1,...,X_j\}$ and $\{X_{(j+1)},...,X_N\}$. Finally, you pick $j$ for which the sum of sum of squared residuals for both lines is minimal.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2010-12-22T17:56:04.617" />
  <row Id="5702" PostTypeId="2" ParentId="5700" CreationDate="2010-12-22T17:56:27.893" Score="5" ViewCount="" Body="&lt;p&gt;R package &lt;a href=&quot;http://cran.r-project.org/web/packages/strucchange/index.html&quot; rel=&quot;nofollow&quot;&gt;strucchange&lt;/a&gt; might help you. Look at the vignette, it has a nice overview how to solve similar problems. &lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2010-12-22T17:56:27.893" />
  <row Id="5703" PostTypeId="1" CreationDate="2010-12-22T17:57:02.523" Score="7" ViewCount="158" Body="&lt;p&gt;Given $n$-vectors $x, y_1, y_2$ such that the Spearman correlation coefficient of $x$ and $y_i$ is $\rho_i = \rho(x,y_i)$, are there known bounds on the Spearman coefficient of $x$ with $y_1 + y_2$, in terms of the $\rho_i$ (and $n$, presumably)? That is, can one find (non-trivial) functions $l(\rho_1,\rho_2,n), u(\rho_1,\rho_2,n)$ such that &#xA;$$l(\rho_1,\rho_2,n) \le \rho(x,y_1+y_2) \le u(\rho_1,\rho_2,n)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;edit&lt;/strong&gt;: per @whuber's example in the comment, it appears that in the general case, only the trivial bounds $l = -1, u = 1$ can be made. Thus, I would like to further impose the constraint:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;$y_1, y_2$ are permutations of the integers $1 \ldots n$.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="795" LastEditorUserId="795" LastEditorDisplayName="" LastEditDate="2010-12-22T21:15:59.063" LastActivityDate="2010-12-23T20:53:49.907" Title="Are there bounds on the Spearman correlation of a sum of two variables?" Tags="&lt;correlation&gt;&lt;spearman&gt;&lt;bounds&gt;" AnswerCount="1" CommentCount="9" />
  <row Id="5704" PostTypeId="1" CreationDate="2010-12-17T10:02:03.917" Score="0" ViewCount="123" Body="&lt;p&gt;I have recently implemented on Maximum Likelihood Estimation (MLE). I have taken the standard from Simon Haykin's book on Neural Networks. I have seen that he has treated MLE in the book as an Hierarchical Mixture Model (HME). As in India, this is a standard book so I have implemented the same. My community of teachers are happy with the process. Before testing the results, I have tried to read some other literature and came up with some other interpretation as many people like Andrew Moore's website on Machine Learning does not show it as an HME model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question here are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Is there any confusion in standard for MLE?  &lt;/li&gt;&#xA;&lt;li&gt;Am I taking the right path?  &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="2329" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-27T16:04:04.740" LastActivityDate="2011-03-27T16:04:04.740" Title="A question on maximum likelihood estimation as a Hierarchical Mixture Model" Tags="&lt;machine-learning&gt;&lt;maximum-likelihood&gt;&lt;neural-networks&gt;&lt;mixture&gt;" CommentCount="2" FavoriteCount="1" />
  <row Id="5705" PostTypeId="2" ParentId="4687" CreationDate="2010-12-22T18:30:25.820" Score="1" ViewCount="" Body="&lt;p&gt;Consider two simple cases, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) a real valued variable X is the parent of another real valued variable Y&lt;br&gt;&#xA;2) a real valued variable X is the parent of a discrete valued variable Y&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assume that the Bayes net is a directed graph X -&gt; Y.  The Bayes net is fully specified, in both cases, when P(X) and P(Y | X) are specified.  Or, strictly speaking when P(X | a) and P(Y | X, b) are specified where a and b are the parameters governing the marginal distribution of X and the conditional distribution of Y given X, respectively.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Parametric Assumptions&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we are happy for some reason to assume that P(X | a) is Normal, so a contains its mean and its variance.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider case 1.  Perhaps we are happy to assume that P(Y | X, b) is also Normal and that the conditional relationship is linear in X.  Then b contains regression coefficients, as we normally understand them.  That means here: an intercept, a slope parameter, and a conditional variance.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now consider case 2.  Perhaps we are willing to assume that P(Y | X, b) is Binomial and that the conditional relationship is linear in X.  Then b contains the coefficients from a logistic regression model: here that means an intercept and a slope parameter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In case 1 if we observe Y and try to infer X then we simply use Bayes theorem with the parametric assumptions noted above.  The result is rather well known: the (posterior) mean of X given Y is a weighted average of the prior mean of X (gotten from a) and the observed value of Y, where the weights are a function of the standard deviations of X (from a) and Y given X (from b).  More certainty about X pulls the posterior mean towards its mean.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In case 2 there may be no handy closed form for P(X | Y) so we might proceed by drawing a sample from P(X), making use of your assumptions about P(Y | X) and normalising the result.  That is then a sample from P(X | Y).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In both cases, all we are doing is computing Bayes theorem, for which we need some parametric assumptions about the relevant distributions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All this gets more complicated when you want to learn a and b, rather than assuming them, but that is, I think, beyond your question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Sampling&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When thinking about continuous parents it may be helpful to think about &lt;em&gt;generating&lt;/em&gt; data from the network.  To make on data point form the network in case 1 (with the assumptions as above) we would first sample from P(X) - a Normal distribution - then take that value and generate a conditional distribution over Y by plugging the value we generated into the regression model of P(Y | X) as an independent variable.  That conditional distribution is also Normal (by assumption) so we then sample a Y value from that distribution.  We now have a new data point.  The same process applies when P(Y | X) is a logistic regression model.  The model gives us a probability of 1 vs. 0 given the value of X we sampled.  We then sample from that distribution - essentially tossing a coin biased according to the model probabilities - and we have a new data point.  When we think about inference in a Bayes net, we are, conceptually speaking, just reversing this process.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Perhaps that helps?&lt;/p&gt;&#xA;" OwnerUserId="1739" LastActivityDate="2010-12-22T18:30:25.820" CommentCount="1" />
  <row Id="5706" PostTypeId="1" CreationDate="2010-12-22T18:52:56.557" Score="3" ViewCount="110" Body="&lt;p&gt;Suppose I have a fleet of two-car trains riding around, and that each car is equipped with a data recording device. Unfortunately, some of the recording devices aren't working. I don't know either the exact size of the fleet or the percentage of failed recording devices. I'd like to make a reasonable guess about how many cars I'm missing data from.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In particular:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Total fleet size $S$ is unknown.&lt;/li&gt;&#xA;&lt;li&gt;Failure rate (failed units / total cars) $F$ is unknown.&lt;/li&gt;&#xA;&lt;li&gt;I have data from both cars from $S_2$ trains and from exactly one car from $S_1$ trains.&lt;/li&gt;&#xA;&lt;li&gt;Therefore, I know that there are at least $S_1$ failed units. What I don't know is $S_0$, the number of trains with failed units on both cars.&lt;/li&gt;&#xA;&lt;li&gt;Let's assume that the distribution of cars with failed units is random, and whether one car has a failed unit is independent of whether it's mate does.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Does the following make sense for a first-order approximation?&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Guess that the failure rate $F&amp;#39;$ is equal to the proportion of missing cars that I know about to total cars that I know about: $F&amp;#39; = S_1 / (2*(S_2 + S_1))$&lt;/li&gt;&#xA;&lt;li&gt;Assume that the likelihood of a train having two failed units is $F&amp;#39;^2$.&lt;/li&gt;&#xA;&lt;li&gt;Therefore $S_0 = F&amp;#39;^2 * (S_0 + S_1 + S_2)$&lt;/li&gt;&#xA;&lt;li&gt;Therefore $S_0 = (S_2 + S_1) * F&amp;#39;^2 / (1 - F&amp;#39;^2)$&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="2491" LastEditorUserId="2491" LastEditorDisplayName="" LastEditDate="2010-12-23T14:41:47.153" LastActivityDate="2010-12-23T14:41:47.153" Title="Extrapolating the amount of data missing from the amount of data partially missing" Tags="&lt;data-mining&gt;&lt;missing-data&gt;&lt;transportation&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="5707" PostTypeId="2" ParentId="5700" CreationDate="2010-12-22T19:16:53.440" Score="3" ViewCount="" Body="&lt;p&gt;This is an (offline) changepoint detection problem.  Our &lt;a href=&quot;http://stats.stackexchange.com/questions/2432/loess-that-allows-discontinuities/2445#2445&quot;&gt;previous discussion&lt;/a&gt; provides references to journal articles and R code.  Look first at the &lt;a href=&quot;http://www.jstor.org/pss/2290726&quot; rel=&quot;nofollow&quot;&gt;Barry and Hartigan&lt;/a&gt; &quot;product partition model,&quot; because it handles changes in slope and has efficient implementations.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-12-22T19:16:53.440" />
  <row Id="5708" PostTypeId="2" ParentId="5664" CreationDate="2010-12-22T19:37:14.630" Score="1" ViewCount="" Body="&lt;p&gt;I'd like to add another cautionary note, and a suggestion. When asked for a 1-5 rating, I usually think up some scale, like:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;worst ever&lt;/li&gt;&#xA;&lt;li&gt;real bad&lt;/li&gt;&#xA;&lt;li&gt;ok&lt;/li&gt;&#xA;&lt;li&gt;great&lt;/li&gt;&#xA;&lt;li&gt;awesome&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;If your raters did similarly, taking an average is somewhat questionable; the difference between &quot;worst ever&quot; and &quot;real bad&quot; may be larger than the difference between &quot;great&quot; and &quot;awesome&quot;. In jargon, your data may be somewhere between ordinal and interval (see &lt;a href=&quot;http://en.wikipedia.org/wiki/Level_of_measurement&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Level_of_measurement&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For this reason, I'd suggest that you give your speakers a histogram of their ratings as part of their feedback. And maybe even rank speakers by percent of their ratings above, say, 3, rather than by their mean rating.&lt;/p&gt;&#xA;" OwnerUserId="2489" LastActivityDate="2010-12-22T19:37:14.630" CommentCount="3" />
  <row Id="5709" PostTypeId="2" ParentId="5706" CreationDate="2010-12-22T20:17:21.017" Score="6" ViewCount="" Body="&lt;p&gt;This is a quick partial response to outline some options and correct some errors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You are implicitly seeking a &lt;a href=&quot;http://en.wikipedia.org/wiki/Method_of_moments_%28statistics%29&quot; rel=&quot;nofollow&quot;&gt;method of moments estimator&lt;/a&gt;.  Under your assumptions, letting $f$ be the failure rate and $n$ be the fleet size, the expectations of the $S_i$ (which are governed by a multinomial distribution) are&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\eqalign{&#xD;&#xA;\mathbb{E}_{f;n}[S_0] = &amp;amp;f^2 n \cr&#xD;&#xA;\mathbb{E}_{f;n}[S_1] = &amp;amp;2 f (1-f) n \cr&#xD;&#xA;\mathbb{E}_{f;n}[S_2] = &amp;amp;(1-f)^2 n.&#xD;&#xA;}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the algebraic relation&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\mathbb{E}_{f;n}[S_0] = \frac{\mathbb{E}_{f;n}[S_1]^2}{4 \mathbb{E}_{f;n}[S_2]}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(which implies the failure rate should be about &lt;em&gt;twice&lt;/em&gt; your estimate) and assuming $S_2 \ne 0$ you can derive the method of moments estimator&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\hat{S}_0 = \frac{S_1^2}{4 S_2}.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is likely to be biased, especially if $S_2$ is small, which impels us to consider other estimators.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More generally, this problem can be thought of as looking for a &quot;good&quot; estimator for $S_0$ of the form $\hat{S}_0 = t(S_1, S_2)$ in an experiment in which the outcomes are the sum of $n$ Binomial($1-f$, $2$) variables and $S_1$ and $S_2$ are the counts of the single and double &quot;successes,&quot; respectively.  To this end you need to supply a &lt;a href=&quot;http://en.wikipedia.org/wiki/Loss_function&quot; rel=&quot;nofollow&quot;&gt;loss function&lt;/a&gt; $\Lambda(s,t)$ and analyze the &lt;em&gt;statistical risk&lt;/em&gt; $r$, which is the expected loss&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$r_t(f,n) = \mathbb{E}_{f;n}[\Lambda(S_0, t(S_1, S_2))].$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The loss function quantifies the cost of estimating that $S_0$ equals $t(S_1, S_2)$; usually the estimate is not perfect and there is a cost associated with that.  The risk for a particular estimator $t$ is the expected loss; it depends on the unknown failure rate $f$ and the unknown fleet size $n$.  Thus the exercise comes down to finding procedures with acceptable risk &lt;em&gt;functions.&lt;/em&gt;  If you have some quantitative information about the likely values of $f$ and $n$ you can exploit that either directly, by limiting the domain of the risk to the likely values, or with a Bayesian analysis in which you compute the expected value of $r_t$ under some assumed prior distribution of $(f,n)$.  At this point the risk becomes solely a function of the procedure $t$ and it's &quot;merely&quot; a matter of finding a risk-minimizing procedure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In any event, to make further progress you need to supply a loss function (or some reasonable approximation thereof).  I would hesitate to recommend or use the method of moments estimator (derived above) without knowing &lt;em&gt;something&lt;/em&gt; about your loss.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-12-22T20:17:21.017" CommentCount="2" />
  <row Id="5710" PostTypeId="2" ParentId="423" CreationDate="2010-12-22T23:06:03.883" Score="5" ViewCount="" Body="&lt;p&gt;From &lt;a href=&quot;http://www.smbc-comics.com&quot; rel=&quot;nofollow&quot;&gt;SMBC&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/isA5v.gif&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="1106" LastActivityDate="2010-12-22T23:06:03.883" CommunityOwnedDate="2010-12-22T23:06:03.883" CommentCount="2" />
  <row Id="5712" PostTypeId="2" ParentId="5686" CreationDate="2010-12-23T01:12:45.503" Score="1" ViewCount="" Body="&lt;p&gt;I think of NA standing for 'Not Available', while NaN is 'Not a Number', although this is more mnemonic than explanation. By the way, I know of no language other than R (perhaps Splus?) that has both. Matlab, for example, has only NaN.&lt;/p&gt;&#xA;" OwnerUserId="795" LastEditorUserId="795" LastEditorDisplayName="" LastEditDate="2010-12-23T17:23:28.107" LastActivityDate="2010-12-23T17:23:28.107" CommentCount="3" />
  <row Id="5713" PostTypeId="1" CreationDate="2010-12-23T04:43:03.667" Score="4" ViewCount="417" Body="&lt;p&gt;Occasionally I see in literature that a categorical variable such as sex is “partialled” or “regressed” out in (fixed-effects or mixed-effects) regression analysis. I'm troubled with the following practical issues involved in such a statement: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(1) Usually the coding method is not mentioned in the paper. Such a variable has to be coded with quantitative values, and I feel the sensible way should be effect coding (e.g., male = 1, female = -1) so that partialling can be achieved with other effects interpreted at the grand mean of both sex groups. A different coding may render a different (and unwanted) interpretation. For example, dummy coding (e.g., male = 0, female = 1) would leave other effects associated with males, not the grand mean. Even centering this dummy-coded variable might not work well for their partialling purpose if there is unequal number of subjects across the two groups. Am I correct? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(2) If the effect of such a categorical variable is included in the model, examining its effects first seems necessary and should be discussed in the context because of its consequence on the interpretation of other effects. What troubles me is that sometimes the authors don't even mention the significance of sex effect, let alone any model building process. If the sex effect exists, a natural follow-up question is whether any interactions exist between sex and other variables in the model? If no sex effect and no interactions exist, sex should be removed from the model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(3) If sex is considered of no interest to those authors, what is the point of including it in the model in the first place without checking its effects? Does the inclusion of such a categorical variable (and costing one degree of freedom on the fixed effect of sex) gain anything for their partialling purpose when sex effect exists (my limited experience says essentially no)?&lt;/p&gt;&#xA;" OwnerUserId="1513" LastEditorUserId="1513" LastEditorDisplayName="" LastEditDate="2010-12-23T04:53:54.900" LastActivityDate="2010-12-24T00:48:56.240" Title="Partialling or regressing out a categorical variable?" Tags="&lt;regression&gt;" AnswerCount="4" CommentCount="1" />
  <row Id="5714" PostTypeId="2" ParentId="2419" CreationDate="2010-12-23T05:18:33.843" Score="2" ViewCount="" Body="&lt;p&gt;I had good success with the tree-based learners in &lt;a href=&quot;http://luispedro.org/software/milk&quot; rel=&quot;nofollow&quot;&gt;Milk: Machine Learning Toolkit for Python&lt;/a&gt;.  It seems to be under active development, but the documentation was a bit sparse when I was using it.  The test suite (github.com/luispedro/milk/blob/master/tests/test_adaboost.py) contains a &quot;boosted stump&quot; though, which could get you going pretty quickly:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import milk.supervised.tree&#xA;import milk.supervised.adaboost&#xA;&#xA;def test_learner():&#xA;    from milksets import wine&#xA;    learner = milk.supervised.adaboost.boost_learner(milk.supervised.tree.stump_learner())&#xA;    features, labels = wine.load()&#xA;    features = features[labels &amp;lt; 2]&#xA;    labels = labels[labels &amp;lt; 2] == 0&#xA;    labels = labels.astype(int)&#xA;    model = learner.train(features, labels)&#xA;    train_out = np.array(map(model.apply, features))&#xA;    assert (train_out == labels).mean() &amp;gt; .9&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2498" LastActivityDate="2010-12-23T05:18:33.843" CommentCount="2" />
  <row Id="5715" PostTypeId="2" ParentId="5700" CreationDate="2010-12-23T11:04:55.210" Score="1" ViewCount="" Body="&lt;p&gt;Also the &lt;a href=&quot;http://cran.r-project.org/web/packages/segmented/index.html&quot; rel=&quot;nofollow&quot;&gt;segmented&lt;/a&gt; package has helped me with similar problems in the past.&lt;/p&gt;&#xA;" OwnerUserId="1291" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-12-23T21:31:31.840" LastActivityDate="2010-12-23T21:31:31.840" />
  <row Id="5716" PostTypeId="2" ParentId="5713" CreationDate="2010-12-23T11:47:46.440" Score="3" ViewCount="" Body="&lt;p&gt;I don't think (1) makes any difference. The idea is to partial out from the response &lt;em&gt;and&lt;/em&gt; the other predictors the effects of Sex. It doesn't matter if you code 0, 1 (Treatment contrasts) or 1, -1 (Sum to zero contrasts) as the models represent the same &quot;amount&quot; of information which is then removed. Here is an example in R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(1)&#xA;dat &amp;lt;- data.frame(Size = c(rnorm(20, 180, sd = 5), &#xA;                           rnorm(20, 170, sd = 5)),&#xA;                  Sex = gl(2,20,labels = c(&quot;Male&quot;,&quot;Female&quot;)))&#xA;&#xA;options(contrasts = c(&quot;contr.treatment&quot;, &quot;contr.poly&quot;))&#xA;r1 &amp;lt;- resid(m1 &amp;lt;- lm(Size ~ Sex, data = dat))&#xA;options(contrasts = c(&quot;contr.sum&quot;, &quot;contr.poly&quot;))&#xA;r2 &amp;lt;- resid(m2 &amp;lt;- lm(Size ~ Sex, data = dat))&#xA;options(contrasts = c(&quot;contr.treatment&quot;, &quot;contr.poly&quot;))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;From these two models, the residuals are the same and it is this information one would then take into the subsequent model (plus the same thing removing Sex effect form the other covariates):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; all.equal(r1, r2)&#xA;[1] TRUE&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I happen to agree with (2), but on (3) if Sex is no interest to the researchers, they might still want to control for Sex effects, so my null model would be one that includes Sex and I test alternatives with additional covariates plus Sex. Your point about interactions and testing for effects of the non-interesting variables is an important and valid observation.&lt;/p&gt;&#xA;" OwnerUserId="1390" LastActivityDate="2010-12-23T11:47:46.440" CommentCount="0" />
  <row Id="5717" PostTypeId="2" ParentId="5713" CreationDate="2010-12-23T15:31:46.357" Score="0" ViewCount="" Body="&lt;p&gt;It looks like I can't add a long comment directly to Dr. Simpson's answer. Sorry I have to put my response here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I really appreciate your response, Dr. Simpson! I should clarify my arguments a little bit. What I'm having trouble with the partialling business is not a theoretical but a practical issue. Suppose a linear regression model is of the following form &lt;/p&gt;&#xA;&#xA;&lt;p&gt;y = a + b * Sex + other fixed effects + residuals  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I totally agree that, from the theoretical perspective, regardless how we quantify the Sex variable, we would have the same residuals. Even if I code the subjects with some crazy numbers such as male = 10.7 and female = 53.65, I would still get the same residuals as &lt;code&gt;r1&lt;/code&gt; and &lt;code&gt;r2&lt;/code&gt; in your example. However, what matters in those papers is not about the residuals. Instead, the focus is on the interpretation of the intercept &lt;code&gt;a&lt;/code&gt; and other fixed effects in the model above, and this may invite problem when partialling. With such a focus in mind, how Sex is coded does seem to have a big consequence on the interpretation of all other effects in the above model. With dummy coding (&lt;code&gt;options(contrasts = c(&quot;contr.treatment&quot;, &quot;contr.poly&quot;))&lt;/code&gt; in R), all other effects except 'b' should be interpreted as being associated with sex group with code &quot;0&quot; (males). With effect coding (&lt;code&gt;options(contrasts = c(&quot;contr.sum&quot;, &quot;contr.poly&quot;))&lt;/code&gt; in R), all other effects except &lt;code&gt;b&lt;/code&gt; are the average effects for the whole population regardless the sex.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using your example, the model simplifies to&lt;/p&gt;&#xA;&#xA;&lt;p&gt;y = a + b * Sex + residuals.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem can be clearly seen with the following about the estimate of the intercept &lt;code&gt;a&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; summary(m1)&#xA;&#xA;Call: lm(formula = Size ~ Sex, data = dat)&#xA;&#xA;...&#xA;&#xA;Coefficients:&#xA;            Estimate Std. Error t value Pr(&amp;gt;|t|)    &#xA;(Intercept) 180.9526     0.9979 181.332  &amp;lt; 2e-16 ***&#xA;&#xA;&amp;gt; summary(m2)&#xA;&#xA;Call: lm(formula = Size ~ Sex, data = dat)&#xA;&#xA;...&#xA;&#xA;Coefficients:&#xA;            Estimate Std. Error t value Pr(&amp;gt;|t|)    &#xA;(Intercept) 175.4601     0.7056 248.659  &amp;lt; 2e-16 ***&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Finally it looks like I have to agree that my original argument (3) might not be valid. Continuing your example, &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; options(contrasts = c(&quot;contr.sum&quot;, &quot;contr.poly&quot;))&#xA;&amp;gt; m0 &amp;lt;- lm(Size ~ 1, data = dat)&#xA;&amp;gt; summary(m0)&#xA;&#xA;Call: lm(formula = Size ~ 1, data = dat)&#xA;&#xA;...&#xA;&#xA;Coefficients:&#xA;            Estimate Std. Error t value Pr(&amp;gt;|t|)    &#xA;(Intercept)  175.460      1.122   156.4   &amp;lt;2e-16 ***&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It seems that including Sex in the model does not change the effect estimate, but it does increase the statistical power since more variability in the data is accounted for through the Sex effect. My previous illusion in argument (3) may have come from a dataset with a huge sample size in which adding Sex in the model didn't really change much for the significance of other effects. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, in the conventional balanced ANOVA-type analysis, a between-subjects factor such as Sex does not have consequence on those effects unrelated to the factor because of the orthogonal partitioning of the variances?&lt;/p&gt;&#xA;" OwnerUserId="1513" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-12-23T21:28:57.707" LastActivityDate="2010-12-23T21:28:57.707" CommentCount="1" />
  <row Id="5718" PostTypeId="2" ParentId="4172" CreationDate="2010-12-23T16:59:38.090" Score="0" ViewCount="" Body="&lt;p&gt;Another &lt;a href=&quot;http://stats.stackexchange.com/questions/527/what-ways-are-there-to-show-two-analytical-methods-are-equivalent/&quot;&gt;question&lt;/a&gt; seems to have covered similar ground, at least in regards to similarity of discrete values.  There I suggest regressing regression would be usable in theory for the GPS position, though I imagine a solution that respects the two dimensionality of the data would be preferable.  @chi has a better answer on that same question (that includes citations). Similarity of distributions seems to be a question for a K-S test (I wonder if there is a multivariate version).  For the categorization data it seems like Baaysean approach could be useful.  Given categorization A at time slice 1 using Method 1... what is the Pr(Cat B) @ time slice 1 using Method 1?  You might also do a lagged version of this to look at whether one method is picking up categorization changes before or after the other.&lt;/p&gt;&#xA;" OwnerUserId="196" LastActivityDate="2010-12-23T16:59:38.090" />
  <row Id="5719" PostTypeId="2" ParentId="5713" CreationDate="2010-12-23T17:05:00.970" Score="0" ViewCount="" Body="&lt;p&gt;Remember though that error will be reduced by adding any addtional factors.  Even if gender is insignficant in your model it may still be useful in the study.  Signficance can be found in any factor if the sample size is large enough.  Conversly, if the sample size is not large enough a signficant effect may not be testable.  Hence good model building and power analysis.&lt;/p&gt;&#xA;" OwnerUserId="2507" LastActivityDate="2010-12-23T17:05:00.970" />
  <row Id="5720" PostTypeId="2" ParentId="5603" CreationDate="2010-12-23T17:10:23.330" Score="2" ViewCount="" Body="&lt;p&gt;Frankly, what you have is still a Markov Chain, as vqv has very rightly pointed out. (PS I tried to add this as a comment but where is the button??)&lt;/p&gt;&#xA;" OwnerUserId="2472" LastActivityDate="2010-12-23T17:10:23.330" CommentCount="2" />
  <row Id="5721" PostTypeId="2" ParentId="5700" CreationDate="2010-12-23T18:45:57.660" Score="0" ViewCount="" Body="&lt;p&gt;What is wrong with doing simple non-linear least squares in this case?  Am I missing something obvious?&lt;/p&gt;&#xA;" OwnerUserId="1777" LastActivityDate="2010-12-23T18:45:57.660" />
  <row Id="5722" PostTypeId="2" ParentId="5703" CreationDate="2010-12-23T19:11:42.133" Score="4" ViewCount="" Body="&lt;p&gt;Spearman's rank correlation is just the Pearson product-moment correlation between the ranks of the variables. Shabbychef's extra constraint means that $y_1$ and $y_2$ are the same as their ranks and that there are no ties, so they have equal standard deviation $\sigma_y$ (say). If we also replace x by its ranks, the problem becomes the equivalent problem for the Pearson product-moment correlation.&lt;br&gt;&#xA;By definition of the Pearson product-moment correlation,&#xA;$$\begin{align}&#xD;&#xA;\rho(x,y_1+y_2) &#xD;&#xA;  &amp;amp;= \frac{\operatorname{Cov}(x,y_1+y_2)}&#xD;&#xA;          {\sigma_x \sqrt{\operatorname{Var}(y_1+y_2)}} \\&#xD;&#xA;  &amp;amp;= \frac{\operatorname{Cov}(x,y_1) + \operatorname{Cov}(x,y_2)}&#xD;&#xA;          {\sigma_x \sqrt{\operatorname{Var}(y_1)+\operatorname{Var}(y_2)&#xD;&#xA;                          + 2\operatorname{Cov}(y_1,y_2)}} \\&#xD;&#xA;  &amp;amp;= \frac{\rho_1\sigma_x\sigma_y + \rho_2\sigma_x\sigma_y}&#xD;&#xA;          {\sigma_x \sqrt{2\sigma_y^2 + 2\sigma_y^2\rho(y_1,y_2)}} \\&#xD;&#xA;  &amp;amp;= \frac{\rho_1 + \rho_2}&#xD;&#xA;          {\sqrt{2}\left(1+\rho(y_1,y_2)\right)^{1/2}}. \\&#xD;&#xA;\end{align}$$&#xA;For any set of three variables, if we know two of their three correlations we can put bounds on the third correlation (see e.g. &lt;a href=&quot;http://www.informaworld.com/smpp/content~content=a906010843&quot; rel=&quot;nofollow&quot;&gt;Vos 2009&lt;/a&gt;, or from &lt;a href=&quot;http://en.wikipedia.org/wiki/Partial_correlation#Using_recursive_formula&quot; rel=&quot;nofollow&quot;&gt;the formula for partial correlation&lt;/a&gt;):&#xA;$$\rho_1\rho_2 - \sqrt{1-\rho_1^2}\sqrt{1-\rho_2^2} \leq \rho(y_1,y_2) \leq &#xD;&#xA;  \rho_1\rho_2 + \sqrt{1-\rho_1^2}\sqrt{1-\rho_2^2} $$&#xA;Therefore&#xA;$$\frac{\rho_1 + \rho_2}&#xD;&#xA;       {\sqrt{2}\left(1+\rho_1\rho_2 + \sqrt{1-\rho_1^2}\sqrt{1-\rho_2^2}\right)^{1/2}}&#xD;&#xA;  \leq \rho(x,y_1+y_2) \leq&#xD;&#xA; \frac{\rho_1 + \rho_2}&#xD;&#xA;       {\sqrt{2}\left(1+\rho_1\rho_2 - \sqrt{1-\rho_1^2}\sqrt{1-\rho_2^2}\right)^{1/2}}&#xD;&#xA;$$&#xA;if  $\rho_1 + \rho_2 \geq 0$; if $\rho_1 + \rho_2 \le 0$ you need to switch the bounds around.&lt;/p&gt;&#xA;" OwnerUserId="449" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-12-23T20:53:49.907" LastActivityDate="2010-12-23T20:53:49.907" CommentCount="5" />
  <row Id="5723" PostTypeId="2" ParentId="5682" CreationDate="2010-12-23T20:42:30.510" Score="3" ViewCount="" Body="&lt;p&gt;You might want to pick up (or look at) a copy of Rick Wicklin's book: Statistical Programming with SAS IML software&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.etymonline.com/index.php?search=hysterical&amp;amp;searchmode=none&quot; rel=&quot;nofollow&quot;&gt;http://www.etymonline.com/index.php?search=hysterical&amp;amp;searchmode=none&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;He also has a blog about IML.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And, on SAS' site, there is a section about IML:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://support.sas.com/forums/forum.jspa?forumID=47&quot; rel=&quot;nofollow&quot;&gt;http://support.sas.com/forums/forum.jspa?forumID=47&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And you will want IMLStudio, which offers a multiple window view that is much easier to integrate with Base SAS than the old IML was.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have used Base SAS and SAS Stat a lot.  I've only barely looked at IML.  But, from what I've seen, your knowledge of R should help.  &lt;/p&gt;&#xA;" OwnerUserId="686" LastActivityDate="2010-12-23T20:42:30.510" CommentCount="1" />
  <row Id="5724" PostTypeId="1" CreationDate="2010-12-23T22:25:19.767" Score="1" ViewCount="76" Body="&lt;p&gt;Hello everyone,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm new here and wondering if anyone could give me some hints on how to estimate the time varying coefficient and state variable together. Here is my model:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;observation equation: $Y(t)= A(t)X(t)+ w(t)$,&lt;br&gt;&#xA;state equation: $X(t)=\phi X(t-1)+v(t)$, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;here I have time varying coefficient $A(t)$, it doesn't depend on any predetermined parameter $\theta$, for example. If I treat $A(t)$ as another state variable, then it is nonlinear state space, I have no idea how to estimate multiplicative state variables. Any hints? Thank you.&lt;/p&gt;&#xA;" OwnerUserId="2510" LastEditorUserId="2510" LastEditorDisplayName="" LastEditDate="2010-12-24T13:05:52.223" LastActivityDate="2010-12-24T13:05:52.223" Title="Multiplicative unobservable component in state space model" Tags="&lt;time-series&gt;" AnswerCount="1" />
  <row Id="5725" PostTypeId="2" ParentId="5713" CreationDate="2010-12-24T00:48:56.240" Score="1" ViewCount="" Body="&lt;p&gt;It's true that the choice of coding method influences how you interpret the model coefficients. In my experience though (and I realise this can depend on your field), dummy coding is so prevalent that people don't have a huge problem dealing with it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this example, if male = 0 and female = 1, then the intercept is basically the mean response for males, and the Sex coefficient is the impact on the response due to being female (the &quot;female effect&quot;). Things get more complicated once you are dealing with categorical variables with more than two levels, but the interpretation scheme extends in a natural way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What this ultimately means is that you should be careful that any substantive conclusions you draw from the analysis don't depend on the coding method used.&lt;/p&gt;&#xA;" OwnerUserId="1569" LastActivityDate="2010-12-24T00:48:56.240" />
  <row Id="5726" PostTypeId="2" ParentId="5603" CreationDate="2010-12-24T02:13:25.680" Score="1" ViewCount="" Body="&lt;p&gt;You might find Scott and Smyth '03 of interest: &lt;a href=&quot;http://www.datalab.uci.edu/papers/ScottSmythV7.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.datalab.uci.edu/papers/ScottSmythV7.pdf&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;They discuss markov modulated poisson processes, which varies the rate parameter based on which markov state it is in. You could have two states, one below and one above the given population of interest. &lt;/p&gt;&#xA;" OwnerUserId="2073" LastActivityDate="2010-12-24T02:13:25.680" />
  <row Id="5727" PostTypeId="1" AcceptedAnswerId="5741" CreationDate="2010-12-24T02:40:14.103" Score="4" ViewCount="272" Body="&lt;p&gt;I had a question on calcuating James-Stein Shrinkage factor in the &lt;a href=&quot;http://www-stat.stanford.edu/~ckirby/brad/other/Article1977.pdf&quot; rel=&quot;nofollow&quot;&gt;1977 Scientific American paper by Bradley Efron and Carl Morris&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I gathered the data for the baseball players and it is given below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Name, avg45, avgSeason    &#xA;Clemente, 0.400, 0.346    &#xA;Robinson, 0.378, 0.298    &#xA;Howard, 0.356, 0.276    &#xA;Johnstone, 0.333, 0.222    &#xA;Berry, 0.311, 0.273    &#xA;Spencer, 0.311, 0.270    &#xA;Kessinger, 0.289, 0.263    &#xA;Alvarado, 0.267, 0.210    &#xA;Santo, 0.244, 0.269    &#xA;Swoboda, 0.244, 0.230    &#xA;Unser, 0.222, 0.264    &#xA;Williams, 0.222, 0.256    &#xA;Scott, 0.222, 0.303    &#xA;Petrocelli, 0.222, 0.264    &#xA;Rodriguez, 0.222, 0.226    &#xA;Campaneris, 0.200, 0.285    &#xA;Munson, 0.178, 0.316    &#xA;Alvis, 0.156, 0.200&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;avg45 is the average after 45 at bats and is denoted as y in the article. avgSeason is the end of the season average.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The James-Stein estimator for the average ($z$) is given by&#xA;$ z = \bar{y} + c (y-\bar{y})$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and the the shrinkage factor $c$ is given by (page 5 of Scientific American 1977 article)&#xA;$ c = 1 - \frac{(k-3) \sigma^2} {\sum (y - \bar{y})^2} $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$k$ = number of unknown means. Here there are 18 players so $k = 18$. I can calculate $\sum (y - \bar{y})^2$ using avg45 values. But I don't know how to calculate $\sigma^2$. The authors say $c = 0.212$ for the given data set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried using both $\sigma_{x}^2$ and $\sigma_{y}^2$ for $\sigma^2$ but they don't give the correct answer of $c = 0.212$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anybody be kind enough to let me know how to calculate $\sigma^2$ for this data set?&lt;/p&gt;&#xA;" OwnerUserId="2513" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-24T10:04:04.993" LastActivityDate="2010-12-24T18:23:29.703" Title="James-Stein estimator: How to calculate $\sigma^2$ in shrinkage factor?" Tags="&lt;estimation&gt;&lt;shrinkage&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="2" />
  <row Id="5728" PostTypeId="1" AcceptedAnswerId="5764" CreationDate="2010-12-24T06:49:14.123" Score="1" ViewCount="184" Body="&lt;p&gt;Hi, I am running a model for a problem in insurance domain. The final results show some false positive x and some false negative y. I am using SAS Enterprise Miner for this. Can somebody suggest me how to reduce false positive? I know for this i have to increase the false negative. I want to know two things:  &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Is there any option in e-miner where I can give more weight to false negative and less to false positive?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Is there any general approach in modeling which tells us any ways to reduce false negatives or is it just a hit and trial approach?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="1763" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-24T09:52:56.380" LastActivityDate="2010-12-26T20:58:12.193" Title="Reducing false positive rate" Tags="&lt;data-mining&gt;&lt;sas&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="5729" PostTypeId="2" ParentId="5728" CreationDate="2010-12-24T08:47:24.223" Score="2" ViewCount="" Body="&lt;p&gt;What you can do if you do not find the weight option is create the same effect yourself, by increasing the amount of the positives, for example you can give as an input to the algorithm 2 times each of the known positives an leave the negatives as they where. You can even increase it 10 times, it is a matter of experimenting to get as near as you can to the best possible result.&lt;/p&gt;&#xA;" OwnerUserId="1808" LastActivityDate="2010-12-24T08:47:24.223" CommentCount="2" />
  <row Id="5730" PostTypeId="2" ParentId="4753" CreationDate="2010-12-24T09:01:39.010" Score="0" ViewCount="" Body="&lt;p&gt;I had the same problem with a sparce matrix in NLP and what we did was select the columns that where more useful to clasify our rows (that gave more information for discerning the result), if you want I can explain it to you in more detail but it is really simple you can figure it out. &#xA;But your problem does not seem to be a classification one, I actually am a little confused about what you said about above the diagonal and below it. But I was thinking that you can use the &lt;a href=&quot;http://en.wikipedia.org/wiki/Apriori_algorithm&quot; rel=&quot;nofollow&quot;&gt;Apriori&lt;/a&gt; data mining algorithm to discover the more important alliances between any number of items.&lt;/p&gt;&#xA;" OwnerUserId="1808" LastActivityDate="2010-12-24T09:01:39.010" CommentCount="1" />
  <row Id="5731" PostTypeId="2" ParentId="5724" CreationDate="2010-12-24T09:22:15.193" Score="1" ViewCount="" Body="&lt;p&gt;I think this is related to &lt;a href=&quot;http://stats.stackexchange.com/questions/4334/state-space-form-of-time-varying-ar1&quot;&gt;a previously asked question&lt;/a&gt;. One of the answers suggested to use the &lt;a href=&quot;http://admb-project.org&quot; rel=&quot;nofollow&quot;&gt;AD Model Builder software&lt;/a&gt;. Although I haven't used it myself, looking at the manual it looks like an alternative.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wonder though if your problem is sufficiently specified. How does the coefficient At change? You need to put some structure on it, perhaps a smoothness constraint, it it is to be estimated at all.&lt;/p&gt;&#xA;" OwnerUserId="892" LastActivityDate="2010-12-24T09:22:15.193" CommentCount="2" />
  <row Id="5732" PostTypeId="1" CreationDate="2010-12-24T09:55:08.067" Score="1" ViewCount="215" Body="&lt;p&gt;Why the test function in Kolmogorov-Smirnov test takes the supremum of the set of differences, not maximum? When maximum (greatest element) of a set does not exist then supremum may exist, but if greatest element exist then supremum is same as greatest element. I want to know why here greatest element does not exist?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;br&gt;&#xA;Prasenjit &lt;/p&gt;&#xA;" OwnerUserId="2516" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-24T10:03:05.737" LastActivityDate="2010-12-24T12:34:18.007" Title="Why &quot;supremum&quot;, not &quot;maximum&quot; in Kolmogorov-Smirnov test?" Tags="&lt;kolmogorov-smirnov&gt;" AnswerCount="1" />
  <row Id="5733" PostTypeId="1" CreationDate="2010-12-24T10:35:28.497" Score="5" ViewCount="218" Body="&lt;p&gt;I'm working on a Monte Carlo function for valuing several assets with partially correlated returns.  Currently, I just generate a covariance matrix and feed to the the &lt;code&gt;rmvnorm()&lt;/code&gt; function in R.  (Generates correlated random values.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, looking at the distributions of returns of an asset, it is not normally distributed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is really a two part question:&lt;br&gt;&#xA;1) How can I estimate some kind of PDF or CDF when all I have is some real-world data without a known distribution?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) How can I generate correlated values like rmvnorm, but for this unknown (and non-normal) distribution?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="2566" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-26T13:42:16.817" LastActivityDate="2011-03-04T19:06:54.170" Title="Generate random multivariate values from empirical data" Tags="&lt;mcmc&gt;&lt;monte-carlo&gt;&lt;pdf&gt;" AnswerCount="5" FavoriteCount="2" />
  <row Id="5734" PostTypeId="1" CreationDate="2010-12-24T10:46:30.457" Score="1" ViewCount="580" Body="&lt;p&gt;I try to implement a time series data analysis project, but I have to do in Java, C# or Python, is there any good libary such like LOESS, ARIMA in R you can recommend? Many thanks&lt;/p&gt;&#xA;" OwnerUserId="2454" LastActivityDate="2011-01-20T06:28:47.500" Title="Is there any library like LOESS or ARIMA in Java/C# or Python?" Tags="&lt;time-series&gt;" AnswerCount="4" />
  <row Id="5735" PostTypeId="2" ParentId="5733" CreationDate="2010-12-24T12:16:09.760" Score="2" ViewCount="" Body="&lt;p&gt;Regarding the first question, you might consider resampling your data. There would be a problem in case your data were correlated over time (rather than contemporaneously correlated), in which case you would need something like a block bootstrap. But for returns data, a simple bootstrap is probably fine.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I guess the answer to the second question is very much dependent on the target distribution.&lt;/p&gt;&#xA;" OwnerUserId="892" LastActivityDate="2010-12-24T12:16:09.760" />
  <row Id="5736" PostTypeId="2" ParentId="5733" CreationDate="2010-12-24T12:23:59.530" Score="2" ViewCount="" Body="&lt;p&gt;The answer to the first question is that you build a model. In your case this means choosing a distribution and estimating its parameters. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you have the distribution you can sample from it using Gibbs or Metropolis algorithms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the side note, do you really need to sample from this distribution? Usually the interest is in some characteristic of the distribution. You can estimate it using empirical distribution via bootstrap, or again build a model for this characteristic. &lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2010-12-24T12:23:59.530" CommentCount="4" />
  <row Id="5737" PostTypeId="2" ParentId="5732" CreationDate="2010-12-24T12:34:18.007" Score="5" ViewCount="" Body="&lt;p&gt;Suppose a difference is an increasing continuous function on open interval and zero everywhere else. Then the maximum will not exist, but supremum will.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2010-12-24T12:34:18.007" />
  <row Id="5738" PostTypeId="2" ParentId="5727" CreationDate="2010-12-24T12:55:41.967" Score="6" ViewCount="" Body="&lt;p&gt;I am not quite sure about the $c = 0.212$, but the following article provides a much more detailed description of these data:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Efron, B., &amp;amp; Morris, C. (1975). Data analysis using Stein's estimator and its generalizations. Journal of the American Statistical Association, 70(350), 311-319 &lt;a href=&quot;http://www.uni-leipzig.de/~strimmer/lab/courses/ss06/seminar/papers/A/efron-morris1975.pdf&quot; rel=&quot;nofollow&quot;&gt;(link to pdf)&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On page 312, you will see that Efron &amp;amp; Morris use an arc-sin transformation of these data, so that the variance of the batting averages is approximately unity:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; dat &amp;lt;- read.table(&quot;data.txt&quot;, header=T, sep=&quot;,&quot;)&#xA;&amp;gt; yi  &amp;lt;- dat$avg45&#xA;&amp;gt; k   &amp;lt;- length(yi)&#xA;&amp;gt; yi  &amp;lt;- sqrt(45) * asin(2*yi-1)&#xA;&amp;gt; c   &amp;lt;- 1 - (k-3)*1 / sum((yi - mean(yi))^2)&#xA;&amp;gt; c&#xA;[1] 0.2091971&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then they use c=.209 for the computation of the $z$ values, which we can easily back-transform:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; zi  &amp;lt;- mean(yi) + c * (yi - mean(yi))&#xA;&amp;gt; round((sin(zi/sqrt(45)) + 1)/2,3) ### back-transformation&#xA;[1] 0.290 0.286 0.282 0.277 0.273 0.273 0.268 0.264 0.259&#xA;[10] 0.259 0.254 0.254 0.254 0.254 0.254 0.249 0.244 0.239&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So these are the values of the Stein estimator. For Clemente, we get .290, which is quite close to the .294 from the 1977 article.&lt;/p&gt;&#xA;" OwnerUserId="1934" LastEditorUserId="1934" LastEditorDisplayName="" LastEditDate="2010-12-24T13:49:58.580" LastActivityDate="2010-12-24T13:49:58.580" />
  <row Id="5739" PostTypeId="2" ParentId="4334" CreationDate="2010-12-24T13:12:37.723" Score="0" ViewCount="" Body="&lt;p&gt;Would it possible to treat Y(t-1) as an exogenous variable, and estimate this state space model with kalman filter. Then the routine for estimating the coefficient is very standard.&lt;/p&gt;&#xA;" OwnerUserId="2510" LastActivityDate="2010-12-24T13:12:37.723" CommentCount="1" />
  <row Id="5740" PostTypeId="2" ParentId="5734" CreationDate="2010-12-24T14:10:09.720" Score="5" ViewCount="" Body="&lt;p&gt;R is an open source project, so you can look at the file &lt;code&gt;src/library/stats/src/loessc.c&lt;/code&gt; which implements the C-level computation behind &lt;code&gt;loess()&lt;/code&gt;.  You should be able to use that for an extension module to the other languages you listed. Or, and this may be easier, you some of the existing ways to access R from Java, Python or C#.&lt;/p&gt;&#xA;" OwnerUserId="334" LastActivityDate="2010-12-24T14:10:09.720" CommentCount="3" />
  <row Id="5741" PostTypeId="2" ParentId="5727" CreationDate="2010-12-24T16:06:05.953" Score="5" ViewCount="" Body="&lt;p&gt;The parameter $\sigma^2$ is the (unknown) common variance of the vector components, each of which we assume are normally distributed.  For the baseball data we have $45 \cdot Y_i \sim \mathsf{binom}(45,p_i)$, so the normal approximation to the binomial distribution gives (taking $ \hat{p_{i}} = Y_{i}$)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;\hat{p}_{i}\approx \mathsf{norm}(\mathtt{mean}=p_{i},\mathtt{var} = p_{i}(1-p_{i})/45).&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Obviously in this case the variances are not equal, yet if they had been equal to a common value then we could estimate it with the pooled estimator&#xA;$$&#xD;&#xA;\hat{\sigma}^2 = \frac{\hat{p}(1 - \hat{p})}{45},&#xD;&#xA;$$&#xA;where $\hat{p}$ is the grand mean&#xA;$$&#xD;&#xA;\hat{p} = \frac{1}{18\cdot 45}\sum_{i = 1}^{18}45\cdot{Y_{i}}=\overline{Y}.&#xD;&#xA;$$&#xA;It looks as though this is what Efron and Morris have done (in the 1977 paper).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can check this with the following R code. Here are the data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;y &amp;lt;- c(0.4, 0.378, 0.356, 0.333, 0.311, 0.311, 0.289, 0.267, 0.244, 0.244, 0.222, 0.222, 0.222, 0.222, 0.222, 0.2, 0.178, 0.156)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and here is the estimate for $\sigma^2$:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;s2 &amp;lt;- mean(y)*(1 - mean(y))/45&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which is $\hat{\sigma}^2 \approx 0.004332392$.  The shrinkage factor in the paper is then&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1 - 15*s2/(17*var(y))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which gives $c \approx 0.2123905$.  Note that in the second paper they made a transformation to sidestep the variance problem (as @Wolfgang said).  Also note in the 1975 paper they used $k - 2$ while in the 1977 paper they used $k - 3$.&lt;/p&gt;&#xA;" OwnerUserId="1108" LastEditorUserId="1108" LastEditorDisplayName="" LastEditDate="2010-12-24T18:23:29.703" LastActivityDate="2010-12-24T18:23:29.703" />
  <row Id="5742" PostTypeId="1" AcceptedAnswerId="5743" CreationDate="2010-12-24T17:07:49.823" Score="2" ViewCount="110" Body="&lt;h2&gt;Background&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I have a data set $Y$:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(0)&#xA;predictor &amp;lt;- c(rep(5,10), rep(10,10), rep(15,10), rep(20,10)) + rnorm(40)&#xA;response  &amp;lt;- c(rnorm(10,1), rnorm(10,4), rnorm(10,2), rnorm(10,1))&#xA;plot(predictor, response)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and a set of models g$_i$:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;fits &amp;lt;- list()&#xA;fits[['null']]      &amp;lt;- lm(response ~ 1)&#xA;fits[['linear']]    &amp;lt;- lm(response ~ poly(predictor, 1))&#xA;fits[['quadratic']] &amp;lt;- lm(response ~ poly(predictor, 2))&#xA;fits[['cubic']]     &amp;lt;- lm(response ~ poly(predictor, 3))&#xA;fits[['Monod']]     &amp;lt;- nls(response ~ a*predictor/(b+predictor), &#xA;                          start = list(a=1, b=1))&#xA;fits[['log']]       &amp;lt;- lm(response ~ poly(log(predictor + 1), 1))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h2&gt;Problem&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I can find the best fit using &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(plyr)    &#xA;ldply(fits, AIC)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am revising a manuscript with results originally presented as raw AIC values, but I find the actual AIC values are fairly uninformative and difficult to interpret&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Question&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Can I calculate the probability of each model given the set of models e.g. g$_i$ $$\frac{P(Y|\textrm{g}_i)}{\sum_{j=1:n}P(Y|\textrm{g}_j)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An approach that most similar to using AIC wins, because I would like to minimize the amount of revisions to the methods and results that will be required.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Other considerations:&lt;/h2&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;I have considered using ANOVA although this does not appear to have as straightforward of an interpretation as the above, or does it?:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;anova(fits[[1]],fits[[2]],fits[[3]],fits[[4]],fits[[5]],fits[[6]])&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;A Bayesian approach would require more work on both implementing statistics and because it would fundamentally change the interpretation from the likelihood interpretation of $P(Y|g)$ to the Bayesian $P(g|Y)$.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="1381" LastEditorUserId="1381" LastEditorDisplayName="" LastEditDate="2010-12-25T15:35:17.650" LastActivityDate="2010-12-25T15:35:17.650" Title="How can I calculate the probability of model $g_i$ given a set of $n$ models and AIC values?" Tags="&lt;probability&gt;&lt;model-selection&gt;&lt;maximum-likelihood&gt;" AnswerCount="1" CommentCount="8" />
  <row Id="5743" PostTypeId="2" ParentId="5742" CreationDate="2010-12-24T20:56:03.353" Score="1" ViewCount="" Body="&lt;p&gt;Although AIC may not be suitable in this context, Aikake weights provide the ratio of $$\frac{L(g_i|Y)}{\sum_{j=1:n}{L(g_j|Y)}}$$&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Solution&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;This can be calculated from AIC in this way for each model $i$ (closely following &lt;a href=&quot;http://books.google.com/books?id=BQYR6js0CC8C&amp;amp;pg=PA75&amp;amp;lpg=PA75&amp;amp;dq=burnham+and+anderson+akaike&amp;amp;source=bl&amp;amp;ots=i8bVidhbWF&amp;amp;sig=8-a_gaD1QiL_JoPICrzqA_gDyJM&amp;amp;hl=en&amp;amp;ei=AwYVTYunPIeenAfSyIi8CQ&amp;amp;sa=X&amp;amp;oi=book_result&amp;amp;ct=result&amp;amp;resnum=4&amp;amp;ved=0CC8Q6AEwAw#v=onepage&amp;amp;q&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;Burnham and Anderson, 2002&lt;/a&gt;):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\Delta_i = AIC_i - AIC_{min}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $AIC_{min}$ is the best fit model&lt;/p&gt;&#xA;&#xA;&lt;p&gt;normalizing these by the sum of the likelihoods gives the Aikake Weight ($W_i$) for each model&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$W_i=\frac{exp(-1/2\Delta_i)}{\sum_{j=1:n}{exp(-1/2\Delta_j)}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;which &lt;a href=&quot;http://faculty.washington.edu/skalski/classes/QERM597/papers/Johnson%20and%20Omland.pdf&quot; rel=&quot;nofollow&quot;&gt;Johnson and Olmland (2004)&lt;/a&gt; interpret as&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;the probability that model &lt;em&gt;i&lt;/em&gt; is the best model for the observed data given the candidate set of models.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Burnham and Anderson state that this approach applies for AIC$_c$, QAIC, QAIC$_c$, and TIC. &lt;/p&gt;&#xA;&#xA;&lt;h2&gt;References&lt;/h2&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Johnson and Olmland, 2004. Model selection in ecology and evolution. TREE 19(2) dx.doi.org/doi:10.1016/j.tree.2003.10.013&lt;/li&gt;&#xA;&lt;li&gt;Burnham, K.P. and Anderson, D.R. (2002) Model Selection and Multimodel Inference: A Practical Information-Theoretic Approach, Springer&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="1381" LastEditorUserId="1381" LastEditorDisplayName="" LastEditDate="2010-12-24T21:25:57.317" LastActivityDate="2010-12-24T21:25:57.317" CommentCount="2" />
  <row Id="5744" PostTypeId="2" ParentId="5691" CreationDate="2010-12-25T04:24:25.403" Score="0" ViewCount="" Body="&lt;p&gt;how to bootstrap a table ought to depend on how it was obtained. there are a few ways this can be done. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;for example, one can sample n individuals and classify all of them into an $r\times c$ table. in this scenario, neither row nor column totals are fixed. it would then seem plausible that a bootstrap sample would take a sample of size n  with replacement from the original n respondents and classify them into an $r\times c$ table.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;in another scenario, the row totals for the table could have been fixed in advance. in that case, one would draw with replacement a sample of  $n_{i+}$ individuals from the $n_{i+}$ individuals in row $i$, separately for each row.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;does this make sense? [apologies if this restates what has already been said. i don't quite grasp all of it.]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;i would like to reiterate @whuber's query on why you are asking this question. one usually does not speak of obtaining a CI for a statistic [such as a p-value]. can you clarify this a bit?&lt;/p&gt;&#xA;" OwnerUserId="1112" LastActivityDate="2010-12-25T04:24:25.403" CommentCount="1" />
  <row Id="5745" PostTypeId="2" ParentId="5629" CreationDate="2010-12-25T13:15:31.367" Score="1" ViewCount="" Body="&lt;p&gt;See Wikipedia's&#xA;&lt;a href=&quot;http://en.wikipedia.org/wiki/List_of_uncertainty_propagation_software&quot; rel=&quot;nofollow&quot;&gt;List of uncertainty propagation software&lt;/a&gt;,&#xA;in particular Python&#xA;&lt;a href=&quot;http://packages.python.org/uncertainties&quot; rel=&quot;nofollow&quot;&gt;uncertainties&lt;/a&gt; .&lt;br&gt;&#xA;There's even a conference:&#xA;&lt;a href=&quot;http://probabilistic-programming.org/wiki/NIPS&quot; rel=&quot;nofollow&quot;&gt;http://probabilistic-programming.org/wiki/NIPS&lt;/a&gt;*2008_Workshop&lt;br&gt;&#xA;(cut-paste that with the &quot;*2008_Workshop&quot;.)&lt;br&gt;&#xA;(Beware -- random variables that propagate through a network and converge&#xA;and correlate get messy.)&lt;/p&gt;&#xA;" OwnerUserId="557" LastEditorUserId="557" LastEditorDisplayName="" LastEditDate="2010-12-25T13:20:55.960" LastActivityDate="2010-12-25T13:20:55.960" CommentCount="1" />
  <row Id="5746" PostTypeId="1" CreationDate="2010-12-25T15:29:04.023" Score="3" ViewCount="159" Body="&lt;p&gt;The &lt;a href=&quot;http://en.wikipedia.org/wiki/Secretary_problem&quot; rel=&quot;nofollow&quot;&gt;Secretary problem&lt;/a&gt;&#xA;has an algorithm for fixed N and immediate accept/reject&#xA;(that is, reject reject ... accept one, stop).&#xA;There are several variants;&#xA;in mine, secretaries or samples come from a real-valued source X&lt;sub&gt;j&lt;/sub&gt;,&#xA;payoff is from best-so-far not last,&#xA;and each sample costs $ c:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;maximize payoff = X&lt;sub&gt;bestsofar&lt;/sub&gt; - c * Nsample&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's a picture of random walks of this kind: either go up to a&#xA;high-water mark (new maximum) of X&lt;sub&gt;1&lt;/sub&gt; X&lt;sub&gt;2&lt;/sub&gt; ..., or down c:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/NxQrC.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone point me to stop-or-keep-looking rules for optimal stopping&#xA;for problems like this ?&#xA;Perhaps one could combine two kinds of rule:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;sample at least ..., take the best after that (Secretary problem)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;stop when the peak - current exceeds some &amp;Delta; (Allaart)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;As @whuber points out, one needs &lt;em&gt;some&lt;/em&gt; model of the distribution of the samples&#xA;to define the problem. This I don't have.&#xA;Nonetheless, statisticians must have looked at problems of this kind --&#xA;sequential sampling, sequential design of experiments, optimal stopping,&#xA;optimal learning ?&#xA;Then please help me rephrase my question to ask for a tutorial on ...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Added links 29Dec, thanks @James; readers please extend &amp;mdash;&lt;br&gt;&#xA;&lt;a href=&quot;http://en.wikipedia.org/wiki/Optimal_stopping&quot; rel=&quot;nofollow&quot;&gt;Optimal stopping&lt;/a&gt;&lt;br&gt;&#xA;Hill, &lt;a href=&quot;http://www.americanscientist.org/issues/feature/2009/2/knowing-when-to-stop/1&quot; rel=&quot;nofollow&quot;&gt;Knowing when to stop&lt;/a&gt;&#xA;2009, 3p, excellent&lt;br&gt;&#xA;Allaart,&#xA;&lt;a href=&quot;http://www.cas.unt.edu/~allaart/papers/crwmax.pdf&quot; rel=&quot;nofollow&quot;&gt;Stopping the maximum of a correlated random walk with cost for observation&lt;/a&gt;&#xA;2004, 12p&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Sorry to keep changing this &amp;mdash; trying to converge to a standard formulation.)&lt;/p&gt;&#xA;" OwnerUserId="557" LastEditorUserId="557" LastEditorDisplayName="" LastEditDate="2010-12-29T13:00:33.357" LastActivityDate="2011-01-05T12:29:37.580" Title="Optimal stopping from an unknown distribution" Tags="&lt;sampling&gt;&lt;search-theory&gt;&lt;optimal-stopping&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="5747" PostTypeId="1" AcceptedAnswerId="5753" CreationDate="2010-12-25T19:24:45.540" Score="11" ViewCount="544" Body="&lt;p&gt;I know empirically that is the case.  I have just developed models that run into this conundrum.  I also suspect it is not necessarily a yes/no answer. I mean by that if both A and B are correlated with C, this may have some implication regarding the correlation between A and B.  But, this implication may be weak.  It may be just a sign direction and nothing else.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is what I mean...  Let's say A and B both have a 0.5 correlation with C.  Given that, the correlation between A and B could well be 1.0.  I think it also could be 0.5 or even lower.  But, I think it is unlikely that it would be negative.  Do you agree with that?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, is there an implication if you are considering the standard Pearson Correlation Coefficient or instead the Spearman (rank) Correlation Coefficient?  My recent empirical observations were associated with the Spearman Correlation Coefficient.   &lt;/p&gt;&#xA;" OwnerUserId="1329" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-01-12T16:15:01.780" LastActivityDate="2011-01-12T16:17:07.083" Title="If A and B are correlated with C, why are A and B not necessarily correlated?" Tags="&lt;correlation&gt;&lt;cross-correlation&gt;" AnswerCount="5" CommentCount="2" FavoriteCount="4" />
  <row Id="5748" PostTypeId="2" ParentId="5747" CreationDate="2010-12-25T19:30:52.613" Score="6" ViewCount="" Body="&lt;p&gt;I will leave the statistical demonstration to those who are better suited than me for it... but intuitively say that event A generates a process X that contributes to the generation of event C. Then A is correlated to C (through X). B, on the other hand generates Y, that also shapes C. Therefore A is correlated to C, B is correlated to C but A and B are not correlated.&lt;/p&gt;&#xA;" OwnerUserId="582" LastEditorUserId="582" LastEditorDisplayName="" LastEditDate="2010-12-26T08:17:28.360" LastActivityDate="2010-12-26T08:17:28.360" CommentCount="3" />
  <row Id="5749" PostTypeId="1" CreationDate="2010-12-25T21:13:06.783" Score="3" ViewCount="81" Body="&lt;p&gt;If you want to regress y on x, where multiple y's are observed at each x, is it ever better to instead take the mean at each x, and the use those means for the regression?  Does it depend on the distributional assumptions?&lt;/p&gt;&#xA;" OwnerUserId="2524" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-26T13:45:12.120" LastActivityDate="2010-12-26T13:45:12.120" Title="When is it better to average observations at the same abscissa?" Tags="&lt;regression&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="5750" PostTypeId="1" AcceptedAnswerId="5754" CreationDate="2010-12-25T22:16:04.897" Score="14" ViewCount="514" Body="&lt;p&gt;I have a several hundreds of measurements. Now I am considering utilizing some kind of software to correlate every measure with every measure. This means that there are hundreds of correlations. Among these there should (statistically) be a high correlation, even if the data is completely random (each measure has only about 100 datapoints).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I find a correlation, how do I include the information about how hard I looked for a correlation into it? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not at a high level in statistics, so please bear with me.&lt;/p&gt;&#xA;" OwnerUserId="888" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-26T13:45:32.963" LastActivityDate="2011-01-02T11:57:26.020" Title="Look and you shall find (a correlation)" Tags="&lt;correlation&gt;&lt;multiple-comparisons&gt;" AnswerCount="5" CommentCount="1" FavoriteCount="7" />
  <row Id="5751" PostTypeId="2" ParentId="5747" CreationDate="2010-12-25T22:17:12.493" Score="4" ViewCount="" Body="&lt;p&gt;I think it's better to ask &quot;why SHOULD they be correlated?&quot; or, perhaps &quot;Why should have any particular correlation?&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The following R code shows a case where x1 and x2 are both correlated with Y, but have 0 correlation with each other&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x1 &amp;lt;- rnorm(100)&#xA;x2  &amp;lt;- rnorm(100)&#xA;y &amp;lt;- 3*x1 + 2*x2 + rnorm(100, 0, .3)&#xA;&#xA;cor(x1,y)&#xA;cor(x2,y)&#xA;cor(x1,x2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The correlation with Y can be made stronger by reducing the .3 to .1 or whatever&lt;/p&gt;&#xA;" OwnerUserId="686" LastActivityDate="2010-12-25T22:17:12.493" CommentCount="2" />
  <row Id="5752" PostTypeId="2" ParentId="5750" CreationDate="2010-12-25T22:22:20.943" Score="5" ViewCount="" Body="&lt;p&gt;This is an example of multiple comparisons.  There's a large literature on this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have, say, 100 variables, then  you will have 100*99/2 =4950 correlations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the data are just noise, then you would expect 1 in 20 of these to be significant at p = .05.  That's 247.5&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Before going farther, though, it would be good if you could say WHY you are doing this.  What are these variables, why are you correlating them, what is your substantive idea?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or, are you just fishing for high correlations?&lt;/p&gt;&#xA;" OwnerUserId="686" LastActivityDate="2010-12-25T22:22:20.943" CommentCount="1" />
  <row Id="5753" PostTypeId="2" ParentId="5747" CreationDate="2010-12-25T23:54:38.263" Score="9" ViewCount="" Body="&lt;p&gt;Because correlation is a mathematical property of multivariate distributions, some insight can be had purely through calculations, regardless of the statistical genesis of those distributions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the &lt;strong&gt;Pearson correlations&lt;/strong&gt;, consider &lt;strong&gt;multinormal variables&lt;/strong&gt; $X$, $Y$, $Z$.  These are useful to work with because any non-negative definite matrix actually is the covariance matrix of some multinormal distributions, thereby resolving the existence question.  If we stick to matrices with $1$ on the diagonal, the off-diagonal entries of the covariance matrix will be their correlations.  Writing the correlation of $X$ and $Y$ as $\rho$, the correlation of $Y$ and $Z$ as $\tau$, and the correlation of $X$ and $Z$ as $\sigma$, we compute that&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;$1 + 2 \rho \sigma \tau - \left(\rho^2 + \sigma^2 + \tau^2\right) \ge 0$ (because this is the determinant of the correlation matrix and it cannot be negative).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;When $\sigma = 0$ this implies that $\rho^2 + \tau^2 \le 1$.  To put it another way: when both $\rho$ and $\tau$ are large in magnitude, $X$ and $Z$ &lt;em&gt;must&lt;/em&gt; have nonzero correlation.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If $\rho^2 = \tau^2 = 1/2$, then any non-negative value of $\sigma$ (between $0$ and $1$ of course) is possible.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;When $\rho^2 + \tau^2 \lt 1$, negative values of $\sigma$ are allowable.   For example, when $\rho = \tau = 1/2$, $\sigma$ can be anywhere between $-1/2$ and $1$.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;These considerations imply there are indeed some constraints on the mutual correlations.  The constraints (which depend only on the non-negative definiteness of the correlation matrix, not on the actual distributions of the variables) can be tightened depending on assumptions about the univariate distributions.  For instance, it's easy to see (and to prove) that when the distributions of $X$ and $Y$ are not in the same location-scale family, their correlations must be &lt;em&gt;strictly&lt;/em&gt; less than $1$ in size.  (Proof: a correlation of $\pm 1$ implies $X$ and $Y$ are linearly related a.s.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as &lt;strong&gt;Spearman rank correlations&lt;/strong&gt; go, consider three trivariate observations $(1,1,2)$, $(2,3,1)$, and $(3,2,3)$ of $(X, Y, Z)$.  Their mutual rank correlations are $1/2$, $1/2$, and $-1/2$.  Thus even the sign of the rank correlation of $Y$ and $Z$ can be the reverse of the signs of the correlations of $X$ and $Y$ and $X$ and $Z$.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-01-12T16:17:07.083" LastActivityDate="2011-01-12T16:17:07.083" CommentCount="5" />
  <row Id="5754" PostTypeId="2" ParentId="5750" CreationDate="2010-12-26T00:13:23.897" Score="24" ViewCount="" Body="&lt;p&gt;This is an excellent question, worthy of someone who is a clear statistical thinker, because it recognizes a subtle but important aspect of multiple testing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are &lt;a href=&quot;http://www.technion.ac.il/docs/sas/stat/chap43/sect14.htm&quot; rel=&quot;nofollow&quot;&gt;standard methods to adjust the p-values&lt;/a&gt; of multiple correlation coefficients (or, equivalently, to broaden their confidence intervals), such as the Bonferroni and Sidak methods (&lt;em&gt;q.v.&lt;/em&gt;).  However, these are far too conservative with large correlation matrices due to the inherent mathematical relationships that must hold among correlation coefficients in general.  (For some examples of such relationships see the &lt;a href=&quot;http://stats.stackexchange.com/q/5747/919&quot;&gt;recent question and the ensuing thread&lt;/a&gt;.)  One of the best approaches for dealing with this situation is to conduct a &lt;a href=&quot;http://en.wikipedia.org/wiki/Resampling_%28statistics%29&quot; rel=&quot;nofollow&quot;&gt;permutation (or resampling) test&lt;/a&gt;.  It's easy to do this with correlations: in each iteration of the test, just randomly scramble the order of values of each of the fields (thereby destroying any inherent correlation) and recompute the full correlation matrix.  Do this for several thousand iterations (or more), then summarize the distributions of the entries of the correlation matrix by, for instance, giving their 97.5 and 2.5 percentiles: these would serve as mutual symmetric two-sided 95% confidence intervals under the null hypothesis of no correlation.  (The first time you do this with a large number of variables you will be astonished at how high some of the correlation coefficients can be even when there is no inherent correlation.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When reporting the results, no matter what computations you do, you should include the following:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;The size of the correlation matrix (&lt;em&gt;i.e.&lt;/em&gt;, how many variables you have looked at).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;How you determined the p-values or &quot;significance&quot; of any of the correlation coefficients (&lt;em&gt;e.g.&lt;/em&gt;, left them as-is, applied a Bonferroni correction, did a permutation test, or whatever).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Whether you looked at alternative measures of correlation, such as &lt;a href=&quot;http://en.wikipedia.org/wiki/Spearman%27s_rank_correlation_coefficient&quot; rel=&quot;nofollow&quot;&gt;Spearman rank correlation&lt;/a&gt;.  If you did, also indicate why you chose the method you are actually reporting on and using.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-12-26T00:13:23.897" CommentCount="2" />
  <row Id="5755" PostTypeId="2" ParentId="5749" CreationDate="2010-12-26T00:19:39.053" Score="6" ViewCount="" Body="&lt;p&gt;When you use the means you are removing much of the variation of the y's around their averages.  This would be incorrect if you are assessing the relationships between the individual y's and the x's.  In particular it will cause you to overestimate the correlation between the y's and the x's and give you too much confidence in the estimated regression coefficients.  If your purpose is to assess the relationship between the &lt;em&gt;average&lt;/em&gt; y and the x's, that might be ok.  (One subtlety occurs when the numbers of y's involved in the averages vary, because then you should be using &lt;a href=&quot;http://www.itl.nist.gov/div898/handbook/pmd/section1/pmd143.htm&quot; rel=&quot;nofollow&quot;&gt;weighted least squares&lt;/a&gt; methods rather than ordinary least squares.)  Just beware of the &lt;a href=&quot;http://en.wikipedia.org/wiki/Ecological_fallacy&quot; rel=&quot;nofollow&quot;&gt;ecological fallacy&lt;/a&gt;, which is the mistake of using the aggregated (i.e., averaged) y's in place of their individual values.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;These considerations are independent of any distributional assumptions.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-12-26T00:19:39.053" />
  <row Id="5756" PostTypeId="2" ParentId="5747" CreationDate="2010-12-26T07:26:29.060" Score="4" ViewCount="" Body="&lt;p&gt;Correlation is the cosine of the angle between two vectors. In the situation described, (A,B,C) is a triple of observations, made n times, each observation being a real number. The correlation between A and B is the cosine of the angle between $V_A=A-E(A)$ and $V_B=B-E(B)$ as measured in n-dimensional euclidean space.  So our situation reduces to considering 3 vectors $V_A$, $V_B$ and $V_C$ in n dimensional space. We have 3 pairs of vectors and therefore 3 angles. If two of the angles are small (high correlation) then the third one will also be small. But to say &quot;correlated&quot; is not much of a restriction: it means that the angle is between 0 and $\pi/2$. In general this gives no restriction at all on the third angle. Putting it another way, start with any angle less than $\pi$ between $V_A$ and $V_B$  (any correlation except -1). Let $V_C$ bisect the angle between $V_A$ and $V_B$. Then C will be correlated with both A and B.&lt;/p&gt;&#xA;" OwnerUserId="2526" LastEditorUserId="2526" LastEditorDisplayName="" LastEditDate="2010-12-26T07:54:47.530" LastActivityDate="2010-12-26T07:54:47.530" CommentCount="2" />
  <row Id="5757" PostTypeId="1" CreationDate="2010-12-26T11:41:21.370" Score="5" ViewCount="174" Body="&lt;p&gt;In a double-blind study, when are deviations from the control group considered statistically significant? And is this related to the number of samples?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I realise that every experiment is different and statistical significance should depend on the deviations in measurements and the size of the sample group, but I'm hoping there is an intuitive rule-of-thumb formula that can &quot;flag&quot; an event as interesting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a light stats background in engineering, but am by no means a stats guru. A worked example would be much appreciated to help me understand and apply it to every-day things.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update with example:&lt;/strong&gt; OK, here's a (not so simple) thought experiment of what I mean. Suppose I want to measure the toxicity of additives in a village's water supply by comparing mortality rates over time. Given the village's population, natality and mortality rates over several years and the date upon which an additive was introduced into the water supply (disregard quantity), when would a rise in the mortality rate become interesting?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Intuitively, if the mortality rate remains between 0.95% and 1.25% for 10 years, and suddenly spikes to 2.00%, then surely this would be an interesting event &lt;em&gt;if&lt;/em&gt; an additive was added that year (assume short-term toxic effects). Obviously there could be other explanations for the rise, but let's focus on statistical significance. Now, how about if it rises to 1.40%? Is that statistically significant? Where do you draw the line?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm starting to get the feeling that I need to &quot;choose&quot; a critical region, which feel less authoritative. Can a Gaussian distribution guide me on this? What other information do I need to determine statistical significance?&lt;/p&gt;&#xA;" OwnerUserId="1497" LastEditorUserId="1497" LastEditorDisplayName="" LastEditDate="2010-12-26T16:39:54.303" LastActivityDate="2010-12-26T16:39:54.303" Title="When is a deviation statistically significant?" Tags="&lt;algorithms&gt;&lt;statistical-significance&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="5758" PostTypeId="2" ParentId="4949" CreationDate="2010-12-26T12:51:17.367" Score="0" ViewCount="" Body="&lt;p&gt;It seems that you can go about this in two ways, depending on what model assumptions you are happy to make.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Generative Approach&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming a &lt;em&gt;generative&lt;/em&gt; model for the data, you also need to know the prior probabilities of each class for an analytic statement of the classification error.  Look up &lt;a href=&quot;http://en.wikipedia.org/wiki/Linear_discriminant_analysis&quot; rel=&quot;nofollow&quot;&gt;Discriminant Analysis&lt;/a&gt; to get the optimal decision boundary in closed form, then compute the areas on the wrong sides of it for each class to get the error rates.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I assume this is the approach intended by your invocation of &lt;em&gt;the&lt;/em&gt; Bayes classifier, which is defined only when everything about the data generating process is specified.  Since this is seldom possible it is always also worth considering the&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Discrimination Approach&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you don't want to or cannot specify the prior class probabilities, you can take advantage of the fact that the discriminant function can under many circumstances (roughly, exponential family class conditional distributions) be modelled &lt;em&gt;directly&lt;/em&gt; by a logistic regression model.  The error rate calculation is then the one for the relevant logistic regression model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a comparison of approaches and a discussion of error rates, &lt;a href=&quot;http://www.cs.berkeley.edu/~jordan/papers/uai.ps&quot; rel=&quot;nofollow&quot;&gt;Jordan 1995&lt;/a&gt; and &lt;a href=&quot;http://www.cs.berkeley.edu/~jordan/papers/ng-jordan-nips01.ps&quot; rel=&quot;nofollow&quot;&gt;Jordan 2001&lt;/a&gt; and references may be of interest.&lt;/p&gt;&#xA;" OwnerUserId="1739" LastActivityDate="2010-12-26T12:51:17.367" />
  <row Id="5759" PostTypeId="2" ParentId="4396" CreationDate="2010-12-26T13:37:32.393" Score="0" ViewCount="" Body="&lt;p&gt;I think one option is to split a SPSS file in groups. The menu approach is Data &gt; Split file.&#xA;I have worked with the rank option, but I think that is better to split the file, for doing other analysis by groups. &lt;/p&gt;&#xA;" OwnerUserId="2348" LastActivityDate="2010-12-26T13:37:32.393" />
  <row Id="5760" PostTypeId="2" ParentId="5757" CreationDate="2010-12-26T15:52:51.497" Score="6" ViewCount="" Body="&lt;p&gt;This question gets to the heart of statistical thinking by recognizing that both (a) &quot;every experiment is different,&quot; implying no single &quot;cookbook&quot; recipe will suffice to assess experimental results in all cases and (b) &quot;significance should depend on the deviations in measurements,&quot; pointing towards the importance of probability theory in modeling the deviations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unfortunately (a) indicates that a universal &quot;simple formula&quot; is not possible.  However, &lt;strong&gt;some things can be said in general&lt;/strong&gt;.  These include&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Deviations in measurements can be attributed partly to predictable phenomena as determined by properties of the subjects or the experiments.  For example, weights of people depend (on average) partly on their gender.  Deviations that are not predictable or determinate are usually modeled as &lt;a href=&quot;http://en.wikipedia.org/wiki/Random_variable&quot; rel=&quot;nofollow&quot;&gt;random variables&lt;/a&gt;.  How you analyze the deviations into these deterministic and random components is a probability model.  Probability models can be as simple as textbook descriptions of dice and coins, but for &lt;a href=&quot;http://www.fs.fed.us/psw/publications/documents/psw_gtr066/psw_gtr066.pdf&quot; rel=&quot;nofollow&quot;&gt;realistic situations&lt;/a&gt; they can be quite complex.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Statistical &quot;significance&quot; measures the chance, when (hypothetically) the treatment has no effect, that some measure of difference between treatment and control groups would cause us to infer the groups are indeed different.  This description is lengthy because so much is involved in its preparation: measuring the results, expressing the differences between the groups in some way (the &lt;a href=&quot;http://en.wikipedia.org/wiki/Test_statistic&quot; rel=&quot;nofollow&quot;&gt;test statistic&lt;/a&gt;), and selecting an inference procedure based on that statistic.  Each of these things is under the control of the experimenter or observer and each is important.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;A simple textbook example&lt;/strong&gt; concerns a controlled experiment in which only two outcomes are possible for each subject, such as &quot;dies&quot; and &quot;lives&quot;.  With good experimental design--double-blinding, randomization of subjects, careful measurement, etc.--we can view the experimental outcomes behaving random draws of  &lt;a href=&quot;http://www.quantdec.com/envstats/notes/class_06/tutorial.htm&quot; rel=&quot;nofollow&quot;&gt;tickets from a box&lt;/a&gt;, where each ticket is labeled with one of the two outcomes.  This is the probability model.  The test statistic is usually the difference in proportions between the two groups (&lt;em&gt;e.g.&lt;/em&gt;, the difference between their mortality rates).  Statistical theory, in the form of the &lt;a href=&quot;http://en.wikipedia.org/wiki/Neyman%E2%80%93Pearson_lemma&quot; rel=&quot;nofollow&quot;&gt;Neyman-Pearson Lemma&lt;/a&gt;, tells us to base our determination of the experimental result on whether this difference exceeds some predetermined threshold.  Probability theory allows us to analyze this tickets-in-a-box model to come up with an appropriate threshold (the test's &lt;a href=&quot;http://www.stats.gla.ac.uk/steps/glossary/hypothesis_testing.html#cr&quot; rel=&quot;nofollow&quot;&gt;critical region&lt;/a&gt;).  The theory shows precisely how that threshold depends on the sizes of the control and treatment groups.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;To go further&lt;/strong&gt;, you need to learn some basic probability theory and see applications in some exemplary cases.  This will accustom you to the habit of &quot;thinking statistically&quot; about everyday things.  Two &lt;em&gt;great&lt;/em&gt; resources are &lt;a href=&quot;http://www.larrygonick.com/html/pub/books/sci7.html&quot; rel=&quot;nofollow&quot;&gt;Smith &amp;amp; Gonick's Cartoon Guide to Statistics&lt;/a&gt; and the classic Freedman &lt;em&gt;et al.&lt;/em&gt; textbook &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0393970833&quot; rel=&quot;nofollow&quot;&gt;Statistics&lt;/a&gt;, used at Berkeley for several generations.  (The link goes to an older edition which you can inspect online and buy used for almost nothing.)&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-12-26T15:52:51.497" CommentCount="1" />
  <row Id="5761" PostTypeId="2" ParentId="5746" CreationDate="2010-12-26T17:02:26.383" Score="4" ViewCount="" Body="&lt;p&gt;Not an answer, but maybe this helps to clarify the question...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't think the secretary problem is without dependence on the underlying distribution.  For example, if the observations aren't stationary, the 37% approach is not likely to be optimal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a concrete example, a person making monthly purchases in an index fund for their retirement account could use the 37% rule to try to buy in at the low of the each month. In practice, this strategy doesn't work well because prices tend to trend upward. If some poster can cite work which deals with optimal stopping rules when the observations display trends, I'd be grateful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Perhaps a more helpful response to the original question is this paper &lt;a href=&quot;http://www.sie.arizona.edu/MURI/cd/content/Bearden%20Skip%20the%20Sq%20Rt%20of%20n%20Thrusts%20A%20and%20C.pdf&quot; rel=&quot;nofollow&quot;&gt;Skip the Square Root of n: A New Secretary Problem&lt;/a&gt; which deals with the objective of maximizing the expected rank of the chosen element, rather than the probability of choosing the top ranked element as in the classical secretary problem.&lt;/p&gt;&#xA;" OwnerUserId="2527" LastActivityDate="2010-12-26T17:02:26.383" CommentCount="2" />
  <row Id="5762" PostTypeId="1" AcceptedAnswerId="5784" CreationDate="2010-12-26T17:08:51.420" Score="2" ViewCount="323" Body="&lt;p&gt;My project is about whether early language scores (on a reading task) can predict later scores on a social attribution task. I have to conduct a logistic regression for my project but I am stuck on which one is appropriate. This is because I have conducted a binary logistic and multinomial logistic regression but I always get warnings and I do not know what to do. Any help will be much appreciated. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's just that my supervisor specifically instructed me to use a logistic regression, sorry I forgot to add that there are three groups involved autism, language impaired and typically developing kids and I need to look at whether langauge scores of the autistic kids and language impaired kids are also a signifcant predictor of scoring low on a social attribution task in later life. Please can you help with this??&lt;/p&gt;&#xA;" OwnerUserId="2528" LastEditorUserId="2528" LastEditorDisplayName="" LastEditDate="2010-12-26T23:14:08.420" LastActivityDate="2010-12-28T15:30:54.830" Title="What logistic regression is best to use?" Tags="&lt;logistic-regression&gt;&lt;homework&gt;" AnswerCount="3" />
  <row Id="5763" PostTypeId="2" ParentId="5762" CreationDate="2010-12-26T17:29:24.403" Score="6" ViewCount="" Body="&lt;p&gt;Logistic regression is to be used when the outcome of interest is a binary variable (e.g., success/failure), whereas multinomial logistic is reserved for the case of a multi-category response variable (e.g., blue/red/green). In both cases, the response variable to be predicted is a categorical variable. The predictors might be categorical or continuous variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From what you described, you are interested in predicting scores on a social attribution task based on observed scores on a reading scores. If both sets of scores are numerical, then it is simply a linear regression model. In R, it is something like&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# fake data&#xA;x &amp;lt;- rnorm(100)&#xA;y &amp;lt;- 1.2*x + rnorm(100)&#xA;# model fit&#xA;summary(lm(y ~ x))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The output reads:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Call:&#xA;lm(formula = y ~ x)&#xA;&#xA;Residuals:&#xA;    Min      1Q  Median      3Q     Max &#xA;-2.6635 -0.6227 -0.1589  0.6360  2.2494 &#xA;&#xA;Coefficients:&#xA;            Estimate Std. Error t value Pr(&amp;gt;|t|)    &#xA;(Intercept) -0.03769    0.10042  -0.375    0.708    &#xA;x            1.31590    0.10796  12.188   &amp;lt;2e-16 ***&#xA;---&#xA;Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 &#xA;&#xA;Residual standard error: 1.003 on 98 degrees of freedom&#xA;Multiple R-squared: 0.6025, Adjusted R-squared: 0.5985 &#xA;F-statistic: 148.6 on 1 and 98 DF,  p-value: &amp;lt; 2.2e-16 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here an increase of one unit in $x$ is associated to an increase of 1.32 units in $y$. This is basically what the regression coefficient tells you when you assume a model like $\mathbb{E}(y|x)=\beta_0+\beta_1x+\varepsilon$, where $\varepsilon$ are independent and identically distributed as a centered gaussian with variance $\sigma^2$ (unknown).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, you may still be interested in a binary outcome. For instance, I can define &quot;Success&quot; as $y&amp;gt;0$. In this case, a logistic regression would look like&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;yb &amp;lt;- ifelse(y&amp;gt;0, 1, 0)&#xA;summary(glm(yb ~ x, family=binomial))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where the regression coefficient gives you the log of the odds of passing the exam (compared to failing to reach a score of 0):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Coefficients:&#xA;            Estimate Std. Error z value Pr(&amp;gt;|z|)    &#xA;(Intercept)  -0.5047     0.3194  -1.580    0.114    &#xA;x             3.3959     0.6871   4.943  7.7e-07 ***&#xA;---&#xA;Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some pointers with multinomial logistic regression have already been given. What I wonder is the outcome you want to model: If you're interested in whether the autistic group has lower scores on the attribution task (as would be expected from the literature), then in a logistic model both scores will be continuous predictors, and reading scores will be considered as a covariate; in other words, you model the odds of being in one of the diagnostic class as a function of attrition task, after adjusting for baseline differences in language proficiency. But it seems it would make sense only if the diagnostic categories are not &lt;em&gt;a priori&lt;/em&gt; defined and it does not directly answer the question you are asking (whether language is a predictor for attrition that may act differentially according to the diagnostic group); otherwise, I would rather model the attrition scores as a function of a grouping variable (diagnostic category) + reading scores, which is basically an &lt;a href=&quot;http://en.wikipedia.org/wiki/Analysis_of_covariance&quot; rel=&quot;nofollow&quot;&gt;ANCOVA&lt;/a&gt; model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Both models are available in R: &lt;code&gt;mlogit()&lt;/code&gt; in the package &lt;a href=&quot;http://cran.r-project.org/web/packages/mlogit/index.html&quot; rel=&quot;nofollow&quot;&gt;mlogit&lt;/a&gt; for multinomial logistic regression, and &lt;code&gt;lm()&lt;/code&gt; for ANCOVA.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-12-28T08:33:23.127" LastActivityDate="2010-12-28T08:33:23.127" CommentCount="2" />
  <row Id="5764" PostTypeId="2" ParentId="5728" CreationDate="2010-12-26T20:51:31.283" Score="3" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Regarding first (and second) question:&lt;/strong&gt; A general approach to reduce misclassifications error by iteratively training models and reweighting rows (based on classification error) is &lt;a href=&quot;http://en.wikipedia.org/wiki/Boosting&quot; rel=&quot;nofollow&quot;&gt;Boosting&lt;/a&gt;. I think you might find that technique interesting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Regarding second question:&lt;/strong&gt; The question sounds a little bit naive to me (but I maybe did not understand your true intention), since reducing misclassification error = improving model performance is one of &lt;em&gt;the&lt;/em&gt; challenges in Data Mining / Machine Learning. So if there were a general all-time working strategy, we all would have been replaced by machines (earlier than we will anyways). So I think that &lt;em&gt;yes&lt;/em&gt;, the general approach here is &lt;strong&gt;educated&lt;/strong&gt; trial and error. I suggest this question, &lt;a href=&quot;http://stats.stackexchange.com/questions/4830/better-classification-of-default-in-logistic-regression&quot;&gt;Better Classification of default in logistic regression&lt;/a&gt;, which may give you some ideas for both questioning and model improvement.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suggest to play around a little bit and then come back to ask more specific questions. General questions regarding model improvement are hard to answer without data and/or additional information of the circumstances. Good luck !&lt;/p&gt;&#xA;" OwnerUserId="264" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-12-26T20:58:12.193" LastActivityDate="2010-12-26T20:58:12.193" />
  <row Id="5765" PostTypeId="1" CreationDate="2010-12-27T05:22:20.193" Score="2" ViewCount="127" Body="&lt;p&gt;Given a list of numbers, is it possible to find out (or in other words, is there a statistical measure to tells the) the closeness of the numbers&#xA;(do note that i am not talking about correlation - this would be for 2 sequences - something like correlation between height and weight).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am looking for something like a closeness coefficient for a given series of numbers,&#xA;so given a series [0,10,20,30,40] - the 'closeness coefficient' gives me the spread of the numbers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It would also be nice if the 'closeness coefficient' depicts the 'density' of the numbers - but if this is a different computable statistical measure, then it shouldnt be a problem.&lt;/p&gt;&#xA;" OwnerUserId="2535" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-27T10:16:02.793" LastActivityDate="2010-12-27T10:16:02.793" Title="Measure of closeness" Tags="&lt;measurement&gt;" AnswerCount="2" CommentCount="3" />
  <row Id="5766" PostTypeId="2" ParentId="5765" CreationDate="2010-12-27T05:34:18.727" Score="8" ViewCount="" Body="&lt;p&gt;The simplest would be the standard deviation. Other measures of 'scale' include the MAD (median absolute deviation), the IQR (interquartile range), Winsorized standard deviation, etc. You might also be interested in the &lt;a href=&quot;http://en.wikipedia.org/wiki/Index_of_dispersion&quot; rel=&quot;nofollow&quot;&gt;Index of Disperson&lt;/a&gt;, the &lt;a href=&quot;http://en.wikipedia.org/wiki/Coefficient_of_variation&quot; rel=&quot;nofollow&quot;&gt;Coefficient of Variation&lt;/a&gt;, or other measures of &lt;a href=&quot;http://en.wikipedia.org/wiki/Statistical_dispersion&quot; rel=&quot;nofollow&quot;&gt;'dispersion'&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="795" LastEditorUserId="795" LastEditorDisplayName="" LastEditDate="2010-12-27T05:47:20.770" LastActivityDate="2010-12-27T05:47:20.770" />
  <row Id="5767" PostTypeId="2" ParentId="5765" CreationDate="2010-12-27T06:36:10.327" Score="1" ViewCount="" Body="&lt;p&gt;I'd use standard deviation, like shabbychef said. See &lt;a href=&quot;http://en.wikipedia.org/wiki/Variance&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Variance&lt;/a&gt; to learn more.&lt;/p&gt;&#xA;" OwnerUserId="2489" LastActivityDate="2010-12-27T06:36:10.327" />
  <row Id="5768" PostTypeId="1" CreationDate="2010-12-27T09:00:13.790" Score="3" ViewCount="220" Body="&lt;p&gt;I have been confused with the problem for a very long time, and hope that somebody here can help me out~&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a experiment installed in split-plot design, with 3 temperal groups as blocks, and 2 factors (each have 3 fixed levels) combinations in each of the block, and I have 4 replicates in each treatments. I used to build the model like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model&amp;lt;-aov(Var~Block+A*B+Error(Block/A/B),data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where A and B are fixed factors of my focus. It seems that R will calculate the Block as an error term, where only DF, SS and MS were reported, but not F-ratio and p value. My question is: is it make sense for me to estimate the effect of block(i.e. different time period) in R?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Acutally, the 4 replication were sampled in four continuous days to develop a temporal sequence for another analysis. I would like to know how could I test the sampling effect, is it another block, or could be treated as a fixed factor nested within the treatment?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance~~&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Marco&lt;/p&gt;&#xA;" OwnerUserId="2536" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-01-14T19:33:30.747" LastActivityDate="2011-01-14T19:33:30.747" Title="Can I test the effect of Block in a split plot design in R?" Tags="&lt;r&gt;&lt;experiment-design&gt;&lt;split-plot&gt;" AnswerCount="1" />
  <row Id="5769" PostTypeId="2" ParentId="5734" CreationDate="2010-12-27T15:20:52.803" Score="1" ViewCount="" Body="&lt;p&gt;With respect to loess the &lt;a href=&quot;http://biopython.org/wiki/Biopython&quot; rel=&quot;nofollow&quot;&gt;BioPython&lt;/a&gt; project has a &lt;a href=&quot;http://www.koders.com/python/fid5A91A606E15507B6823DEC7A059488A6624C4832.aspx?s=python&quot; rel=&quot;nofollow&quot;&gt;lowess() function.&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for ARIMA model fitting, &lt;a href=&quot;http://www.vni.com/campaigns/pyimslstudioeval/&quot; rel=&quot;nofollow&quot;&gt;PyIMSL Studio&lt;/a&gt; contains a number of very useful time series analysis functions, including ones for &lt;a href=&quot;http://www.vni.com/products/imsl/documentation/PyIMSL/stat/wwhelp/wwhimpl/js/html/wwhelp.htm#href=Csch8.9.9.html&quot; rel=&quot;nofollow&quot;&gt;automatically fitting the best ARIMA model&lt;/a&gt; and &lt;a href=&quot;http://www.vni.com/products/imsl/documentation/PyIMSL/stat/wwhelp/wwhimpl/js/html/wwhelp.htm#href=Csch8.9.7.html&quot; rel=&quot;nofollow&quot;&gt;identifying outliers&lt;/a&gt; from it. These same functions are available in the other IMSL libraries if you choose to instead code in C, C#, Java, or Fortran.&lt;/p&gt;&#xA;" OwnerUserId="1080" LastEditorUserId="1080" LastEditorDisplayName="" LastEditDate="2010-12-27T16:36:11.353" LastActivityDate="2010-12-27T16:36:11.353" CommentCount="2" />
  <row Id="5770" PostTypeId="2" ParentId="5768" CreationDate="2010-12-27T16:46:32.043" Score="2" ViewCount="" Body="&lt;p&gt;Hello Marco:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think you know this already, but you can calculate the F Statistic by the ratio of MS and the F Statistic follows $F_{df1, df2}$ where df1 and df2 are degrees of freedom of the numerator and denominator.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;About your estimating the block effect question: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, I don't understand what is your block effect? Normally in split plot design time is &lt;em&gt;not&lt;/em&gt; the block, but &lt;em&gt;sometimes&lt;/em&gt; it does introduce correlation (like block). Generally, the blocks are the subjects (but if they are blocks as in RBD or samples as in CRD depends on the design)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I understand your question correctly:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;your &lt;em&gt;whole plot&lt;/em&gt; treatment is the &lt;em&gt;fixed factor treatment&lt;/em&gt; and your split plot treatment is &lt;em&gt;t&lt;/em&gt; and you have four observation per individual (lets assume a block). You have got a longitudinal mixed effects model (aka repeated measures). If you have access to SAS can easily handle that using the glimmix (or mixed procedure). All you need to do is impose a temporal correlation (generally AR (1)). If you want to use R, please refer to George Casella's &lt;a href=&quot;http://www.stat.ufl.edu/~casella/StatDesign/WebRPrograms/&quot; rel=&quot;nofollow&quot;&gt;website&lt;/a&gt;, an incredible resource for R related experimental design programs. The relevant example for you will be &lt;strong&gt;Hypertension.R&lt;/strong&gt; His book's 5th chapter is my favorite piece on split plot design.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please let me know if I misunderstood you.&lt;/p&gt;&#xA;" OwnerUserId="1307" LastActivityDate="2010-12-27T16:46:32.043" CommentCount="5" />
  <row Id="5771" PostTypeId="1" AcceptedAnswerId="5780" CreationDate="2010-12-27T19:50:12.313" Score="1" ViewCount="128" Body="&lt;p&gt;Suppose I am doing some experimental procedure on two treatment groups. The procedure has several stages, each of which may fail. Failure at any stage halts the experiment. If all stages are passed then there is some useful result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Although I'm primarily interested in the final result, the treatments &lt;strong&gt;might&lt;/strong&gt; also entail different failure rates along the way. I'd like to quantify this, and since we're looking at simple counts it seems like a chi square or Fisher exact test would be appropriate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I want to use such a test as it were &lt;em&gt;recursively&lt;/em&gt;, to the groups passing each stage, do I need to apply some correction for multiple comparisons?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That is, supposing the groups progressed like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;             Group_A         Group_B&#xA;Start        100             100&#xA;Stage_1      90              95&#xA;Stage_2      80              85&#xA;Stage_3      60              75&#xA;Stage_4      55              30&#xA;Results      ...             ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Does it make sense to do a sequence of 2x2 tests of the form:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;             Group_A         Group_B&#xA;Passed_N     X               Y&#xA;Failed_N     Started_N-X     Started_N-Y&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I feel like I should just &lt;em&gt;know&lt;/em&gt; the answer, but I can't figure out whether  this counts as doing repeated tests or not. The populations are &lt;em&gt;somewhat&lt;/em&gt; distinct each time, but heavily overlapping.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, would it make a difference if I had physical reasons to suppose that only stage 4 should be at all affected by the treatments? Could I just choose to ignore any differences in passage through the other stages in that case?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Feel free also to post answers like &quot;ZOMG, don't use that sort of test here, use XXXX, in manner YYYY, for reasons ZZZZ.&quot;)&lt;/p&gt;&#xA;" OwnerUserId="174" LastActivityDate="2010-12-28T10:48:09.420" Title="Multiple comparisons on nested subsets of data" Tags="&lt;hypothesis-testing&gt;&lt;multiple-comparisons&gt;&lt;chi-squared&gt;" AnswerCount="1" />
  <row Id="5772" PostTypeId="2" ParentId="5762" CreationDate="2010-12-28T02:35:22.147" Score="1" ViewCount="" Body="&lt;p&gt;Good explanation.  Remember in your program output for the multilogistic model will only give estimates for two models even though there are three categories.  The model that produces the largest estimate has the higheset odds from coming from that group.  The last group can be determined by 1-estimate1-estimate2.  From the top of my head, I think the following holds:  exp(model1)/(1-exp(model2))=chance of model 1 and exp(model2)/(1-exp(model2))=chance of model 2.&lt;/p&gt;&#xA;" OwnerUserId="2539" LastActivityDate="2010-12-28T02:35:22.147" CommentCount="2" />
  <row Id="5773" PostTypeId="2" ParentId="5747" CreationDate="2010-12-28T02:45:45.653" Score="11" ViewCount="" Body="&lt;p&gt;I'm on an annual fishing trip right now.  There is a correlation between the time of day I fish and the amount of fish I catch.  There is also a correlation between the size of the bait I use and the amount of fish I catch.  There is no correlation between the size of the bait and the time of day.&lt;/p&gt;&#xA;" OwnerUserId="2539" LastActivityDate="2010-12-28T02:45:45.653" CommentCount="1" />
  <row Id="5774" PostTypeId="1" CreationDate="2010-12-28T03:47:52.190" Score="9" ViewCount="732" Body="&lt;p&gt;I have a dataset that has both continuous and categorical data. I am analyzing by using PCA and am wondering if it is fine to include the categorical variables as a part of the analysis. My understanding is that PCA can only be applied to continuous variables. Is that correct? If it cannot be used for categorical data, what alternatives exist for their analysis? &lt;/p&gt;&#xA;" OwnerUserId="2540" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-28T07:18:52.687" LastActivityDate="2010-12-28T07:41:44.453" Title="Can principal component analysis be applied to categorical data?" Tags="&lt;pca&gt;&lt;categorical-data&gt;" AnswerCount="2" FavoriteCount="4" />
  <row Id="5775" PostTypeId="2" ParentId="5750" CreationDate="2010-12-28T05:36:53.667" Score="6" ViewCount="" Body="&lt;p&gt;From your follow up response to Peter Flom's question, it sounds like you might be better served by techniques that look at higher level structure in your correlation matrix.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Techniques like factor analysis, PCA, multidimensional scaling, and cluster analysis of variables can be used to group your variables into sets of relatively more related variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, you may want to think theoretically about what kind of structure should be present. When your number of variables is large and the number of observations is small, you are often better relying more on prior expectations.&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2010-12-28T05:36:53.667" CommentCount="1" />
  <row Id="5776" PostTypeId="1" CreationDate="2010-12-28T06:35:28.500" Score="2" ViewCount="219" Body="&lt;p&gt;There are 8 people playing poker.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, the odds of winning the entire round = 1/8&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2 rounds are played, and Bill wins both rounds.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are the odds this was random?  (Hypothesis test?)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;NullH = Bill has no added skill.  (Got lucky)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;AltH = Bill has skill.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;p = .13  = 1/8&lt;/p&gt;&#xA;&#xA;&lt;p&gt;q = .87 = 7/8&lt;/p&gt;&#xA;&#xA;&lt;p&gt;n = 2&lt;/p&gt;&#xA;&#xA;&lt;p&gt;SD = sqrt(pq/n) = .23&lt;/p&gt;&#xA;&#xA;&lt;p&gt;actual (p-hat) = 1&lt;/p&gt;&#xA;&#xA;&lt;p&gt;z = 3.74&lt;/p&gt;&#xA;&#xA;&lt;p&gt;p-value = 0%&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Conclusion: Odds of winning 2 out of 2 rounds randomly is unlikely.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Reject null hypothesis.  Bill has skill.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this right?&#xA;Thanks!!&lt;/p&gt;&#xA;" OwnerUserId="1279" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-28T09:39:31.057" LastActivityDate="2010-12-30T06:36:57.283" Title="Odds of winning 2 trials in a row" Tags="&lt;hypothesis-testing&gt;" AnswerCount="3" FavoriteCount="1" />
  <row Id="5777" PostTypeId="2" ParentId="5774" CreationDate="2010-12-28T07:09:51.967" Score="12" ViewCount="" Body="&lt;p&gt;Although a PCA applied on binary data would yield results comparable to those obtained from a &lt;a href=&quot;http://en.wikipedia.org/wiki/Multiple_correspondence_analysis&quot; rel=&quot;nofollow&quot;&gt;Multiple Correspondence Analysis&lt;/a&gt; (factor scores and eigenvalues are linearly related), there are more appropriate techniques to deal with mixed data types, namely Multiple Factor Analysis for mixed data available in the &lt;a href=&quot;http://cran.r-project.org/web/packages/FactoMineR/index.html&quot; rel=&quot;nofollow&quot;&gt;FactoMineR&lt;/a&gt; R package (&lt;code&gt;AFDM()&lt;/code&gt;). If your variables can be considered as structured subsets of descriptive attributes, then &lt;a href=&quot;http://factominer.free.fr/advanced-methods/multiple-factor-analysis.html&quot; rel=&quot;nofollow&quot;&gt;Multiple Factor Analysis&lt;/a&gt; (&lt;code&gt;MFA()&lt;/code&gt;) is also an option.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The challenge with categorical variables is to find a suitable way to represent distances between variable categories and individuals in the factorial space. To overcome this problem, you can look for a non-linear transformation of each variable--whether it be nominal, ordinal, polynomial, or numerical--with optimal scaling. This is well explained in &lt;a href=&quot;http://www.jstatsoft.org/v31/i04/paper&quot; rel=&quot;nofollow&quot;&gt;Gifi Methods for Optimal Scaling in R: The Package homals&lt;/a&gt;, and an implementation is available in the corresponding R package &lt;a href=&quot;http://cran.r-project.org/web/packages/homals/index.html&quot; rel=&quot;nofollow&quot;&gt;homals&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-12-28T07:09:51.967" />
  <row Id="5778" PostTypeId="2" ParentId="5774" CreationDate="2010-12-28T07:41:44.453" Score="3" ViewCount="" Body="&lt;p&gt;A Google search &quot;pca for discrete variables&quot; gives this &lt;a href=&quot;http://www.unc.edu/~skolenik/talks/Gustavo-Stas-PCA-generic.pdf&quot; rel=&quot;nofollow&quot;&gt;nice overview&lt;/a&gt;. To add to chl answer, the PC analysis is really analysis of eigenvectors of covariance matrix. So the problem is how to calculate the &quot;correct&quot; covariance matrix. One of the approaches is to use&#xA;&lt;a href=&quot;http://en.wikipedia.org/wiki/Polychoric_correlation&quot; rel=&quot;nofollow&quot;&gt;polychoric correlation&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2010-12-28T07:41:44.453" CommentCount="2" />
  <row Id="5779" PostTypeId="2" ParentId="5776" CreationDate="2010-12-28T07:59:47.490" Score="1" ViewCount="" Body="&lt;p&gt;What you are observing is a random variable with &lt;a href=&quot;http://en.wikipedia.org/wiki/Binomial_distribution&quot; rel=&quot;nofollow&quot;&gt;binomial distribution&lt;/a&gt;, where number of trials is 2 and success probability $1/8$. You then can calculate p-value directly, without resorting to normal approximation. Your statistic is then the number of successes, which under null hypothesis is distributed as $Bin(n,p)$, with $n=2$, and $p=1/8$. The p-value is then&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(\hat{X}\ge 2)=P(\hat{X}=2)={2\choose 2}\left(\frac{1}{8}\right)^2\left(1-\frac{1}{8}\right)^0=\left(\frac{1}{8}\right)^2=0.015625$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So yes the null hypothesis should be rejected.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2010-12-28T09:10:10.303" LastActivityDate="2010-12-28T09:10:10.303" CommentCount="7" />
  <row Id="5780" PostTypeId="2" ParentId="5771" CreationDate="2010-12-28T10:48:09.420" Score="1" ViewCount="" Body="&lt;p&gt;Hi walkytalky.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming I understood your question correctly, I think what you are doing is fine.  However, it does sound like you should correct your P values for multiple comparisons (for example using holm method with p.adjust function in R).  The reason for the adjustment is that you are searching for &quot;interesting results&quot; over the entire sequence of stages.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, instead of using Fisher's test you might want to use &lt;a href=&quot;http://www.r-statistics.com/2010/02/barnards-exact-test-a-powerful-alternative-for-fishers-exact-test-implemented-in-r/&quot; rel=&quot;nofollow&quot;&gt;Bernard's exact test&lt;/a&gt; (which is more powerful then fisher).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another way I can think of doing this is to have for each subject the maximum stage it reached, and then do a wilcox.test on the ranks of the stages patients reached (between groups A and B).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope it helps...&lt;/p&gt;&#xA;" OwnerUserId="253" LastActivityDate="2010-12-28T10:48:09.420" CommentCount="3" />
  <row Id="5781" PostTypeId="2" ParentId="5733" CreationDate="2010-12-28T10:48:46.020" Score="1" ViewCount="" Body="&lt;p&gt;I'm with @mpiktas in that I also think you need a model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think the standard method here would be to estimate a copula to capture the dependence structure between the different assets and use e.g. skew-normal- or t-distributed marginal distributions for the different assets. That gives you a very general model class (more general that assuming e.g. a multivariate t-distribution) that is pretty much the standard for your kind of task (e.g. I think Basel II requires financial institutions to use copula-methods to estimate their VaR). There's a &lt;code&gt;copula&lt;/code&gt; package for R.&lt;/p&gt;&#xA;" OwnerUserId="1979" LastActivityDate="2010-12-28T10:48:46.020" />
  <row Id="5782" PostTypeId="1" AcceptedAnswerId="5790" CreationDate="2010-12-28T14:13:16.227" Score="3" ViewCount="552" Body="&lt;p&gt;Lets say we have random variable $X$ with known variance and mean. The question is: what is the variance of $f(X)$ for some given function f. The only general method that I'm aware of is the delta method, but it gives only aproximation. Now I'm interested in $f(x)=\sqrt{x}$, but it'd be also nice to know some general methods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit 29.12.2010&lt;/strong&gt;&lt;br&gt;&#xA;I've done some calculations using Taylor series, but I'm not sure whether they are correct, so I'd be glad if someone could &lt;strong&gt;confirm&lt;/strong&gt; them.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;First we need to approximate $E[f(X)]$&lt;br&gt;&#xA;$E[f(X)] \approx E[f(\mu)+f&amp;#39;(\mu)(X-\mu)+\frac{1}{2}\cdot f&amp;#39;&amp;#39;(\mu)(X-\mu)^2]=f(\mu)+\frac{1}{2}\cdot f&amp;#39;&amp;#39;(\mu)\cdot Var[X]$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now we can approximate $D^2 [f(X)]$&lt;br&gt;&#xA;$E[(f(X)-E[f(X)])^2] \approx E[(f(\mu)+f&amp;#39;(\mu)(X-\mu)+\frac{1}{2}\cdot f&amp;#39;&amp;#39;(\mu)(X-\mu)^2 -E[f(X)])^2]$  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using the approximation of $E[f(X)]$ we know that $f(\mu)-Ef(x) \approx -\frac{1}{2}\cdot f&amp;#39;&amp;#39;(\mu)\cdot Var[X]$&lt;br&gt;&#xA;We also now that $E[X-\mu]=0$  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using those two we get:&lt;br&gt;&#xA;$D^2[f(X)] \approx \frac{1}{4}\cdot f&amp;#39;&amp;#39;(\mu)^2\cdot Var[X]^2-\frac{1}{2}\cdot f&amp;#39;&amp;#39;(\mu)^2\cdot Var[X]^2 + f&amp;#39;(\mu)^2\cdot Var[X]+\frac{1}{4}f&amp;#39;&amp;#39;(\mu)^2\cdot E[(X-\mu)^4]$&lt;br&gt;&#xA;$D^2 [f(X)] \approx \frac{1}{4}\cdot f&amp;#39;&amp;#39;(\mu)^2 \cdot [D^4 X-(D^2 X)^2]+f&amp;#39;(\mu)\cdot D^2 X$&lt;/p&gt;&#xA;" OwnerUserId="1643" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-13T07:21:34.183" LastActivityDate="2011-05-13T07:21:34.183" Title="Variance of a function of one random variable" Tags="&lt;variance&gt;&lt;random-variable&gt;&lt;delta-method&gt;" AnswerCount="2" CommentCount="3" />
  <row Id="5783" PostTypeId="2" ParentId="5782" CreationDate="2010-12-28T14:59:28.877" Score="4" ViewCount="" Body="&lt;p&gt;To know the first two moments of X (mean and variance) is not enough, if the function f(x) is arbitrary (non linear). Not only for computing the variance of the transformed variable Y, but also for its mean.&#xA;To see this -and perhaps to attack your problem- you can assume that your transformation function has a Taylor expansion around the mean of X and work from there.&lt;/p&gt;&#xA;" OwnerUserId="2546" LastActivityDate="2010-12-28T14:59:28.877" />
  <row Id="5784" PostTypeId="2" ParentId="5762" CreationDate="2010-12-28T15:30:54.830" Score="1" ViewCount="" Body="&lt;p&gt;It sounds like either your professor gave you an incorrect recommendation, or you misunderstood, or you have not communicated what you  want in a clear way.  Something is wrong, in any case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are trying to predict a numeric score (like IQ, SAT, etc) then you do not want logistic regression of any type. You probably want linear regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But perhaps that is not what you  want to do.  Perhaps you want to use the scores to predict the group that the person is in; since there are three groups, you do not want &quot;ordinary&quot; or binomial logistic regression, but you might want either ordinal or multinomial logistic, depending on whether the outcome is ordered or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One additional possibility is that your professor expects you to turn the numerical value of score into a categorical one; this is nearly always a bad idea, but it is often done, nonetheless.  In this case, you would probably want ordinal logistic regression&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you  are using SAS, you may find a presentation of mine informative; it is at www.nesug.org/Proceedings/nesug10/sa/sa03.pdf&lt;/p&gt;&#xA;" OwnerUserId="686" LastActivityDate="2010-12-28T15:30:54.830" CommentCount="1" />
  <row Id="5785" PostTypeId="2" ParentId="5776" CreationDate="2010-12-28T16:24:26.547" Score="6" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;No, this is not right, because no observational context has been provided.&lt;/strong&gt;  All the following scenarios are consistent with the information given:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Two games will be played.  &lt;em&gt;Beforehand&lt;/em&gt;, you hypothesize that Bill will win both.  Assuming the results are independent and Bill has 1/8 chance of winning, the chance of this occurring is 1/64 = &lt;strong&gt;about 1.5%&lt;/strong&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Again two games will be played.  &lt;em&gt;Beforehand&lt;/em&gt;, you hypothesize that &lt;em&gt;somebody&lt;/em&gt; will win both games.  Under the same assumptions the chance of this occurring is 1/64 &lt;em&gt;for each player&lt;/em&gt;.  Because all eight possibilities (for the eight players) are mutually exclusive, the chances add, giving a probability of 1/8 = &lt;strong&gt;12.5%&lt;/strong&gt; for this outcome.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;This time an indefinite number of games will be played, with the series stopping when Bill wins two in a row.  The chance that the series does end is &lt;strong&gt;100%&lt;/strong&gt; and we observe that Bill won the last two games.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Again an indefinite number of games will be played, with the series stopping when &lt;em&gt;anybody&lt;/em&gt; wins two in a row.  The chance the series ends is 100% unless there is perfect anti-correlation among the winners.  Assuming all players have equal chances of winning, the chance that Bill is the one to end it is 1/8 = &lt;strong&gt;12.5%&lt;/strong&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Because neither a clear null hypothesis nor an alternative have been specified, this is not a well-defined hypothesis testing situation.  Therefore the idea of &quot;p-value&quot; is &lt;em&gt;meaningless&lt;/em&gt; (and is unnecessary anyway).  In scenario #1 the low probability provides some evidence--not much--that Bill is not just winning independently at random.  Some alternative explanations include (a) some other players have essentially no chances of winning and Bill just got lucky; (b) the cards are not shuffled well between games, causing the winner of one to be the likely winner of the next, and Bill just got lucky in the first game; (c) in some games certain players have higher chances of winning not through skill but due to their position in the deal, and Bill happened to be in a good position in one or both games.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We cannot generally conclude that the winner of two successive games has &quot;skill&quot;.  For instance, he might have a partner who is setting him up to win.  Or he might be the mark in a group of card sharks who are letting him win to get his confidence, &lt;em&gt;etc.&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-12-28T16:24:26.547" CommentCount="1" />
  <row Id="5786" PostTypeId="1" CreationDate="2010-12-28T16:32:06.750" Score="5" ViewCount="190" Body="&lt;p&gt;Recently I have used Platt's scaling of SVM-outputs to estimate probabilities of default-events. More direct alternatives seem to be &quot;Kernel logistic Regression&quot; (KLR) and the related &quot;Import Vector Machine&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone tell which kernel method giving probability-outputs is currently state of the art? Does an R-implementation of KLR exist?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you very much for your help!&lt;/p&gt;&#xA;" OwnerUserId="2549" LastActivityDate="2010-12-28T22:30:00.733" Title="Which kernel method gives the best probability outputs?" Tags="&lt;logistic-regression&gt;&lt;kernel&gt;&lt;svm&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="1" />
  <row Id="5787" PostTypeId="2" ParentId="5734" CreationDate="2010-12-28T16:45:03.330" Score="0" ViewCount="" Body="&lt;p&gt;As far as I know there aren't many options. Libraries for time series analysis free or otherwise are hard to come by. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are doing this for work you can have a look at &lt;a href=&quot;http://www.nag.com&quot; rel=&quot;nofollow&quot;&gt;NAG&lt;/a&gt;. It's commercial and not Java native. But you can use JNI with their C library. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;They have a &lt;a href=&quot;http://www.nag.com/numeric/CL/nagdoc_cl09/xhtml/G13/g13conts.xml#g13&quot; rel=&quot;nofollow&quot;&gt;time series chapter&lt;/a&gt; which does quite a bit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the free side there is &lt;a href=&quot;http://gretl.sourceforge.net&quot; rel=&quot;nofollow&quot;&gt;gretl&lt;/a&gt; looks good, but I have no direct experience with it. If anyone does, I'd like to hear their opinions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Have a look at &lt;a href=&quot;http://stackoverflow.com/questions/950004/java-library-to-do-time-series-analysis&quot;&gt;this question&lt;/a&gt; on stackoverflow as well.&lt;/p&gt;&#xA;" OwnerUserId="300" LastActivityDate="2010-12-28T16:45:03.330" />
  <row Id="5788" PostTypeId="1" AcceptedAnswerId="5789" CreationDate="2010-12-28T17:00:55.580" Score="4" ViewCount="352" Body="&lt;p&gt;I am attempting to examine the change in slope between a predictor and response over two years.  In year 1, it is definitely positive. (Linear regression, the 95% CI of the slope doesn't overlap 0).  In year 2, the point estimate of the slope is close to 0 (0.002) and the CI overlaps 0.  This is what I would expect if the slope was, well, actually 0.  And given that any test of the slope will suggest that I cannot reject that it is 0 - great!  Although, yes, I know, that doesn't confirm that the slope is 0.  Not falling into that trap.  But, with a slope value very close to 0, and a CI that overlaps 0...it all seems like the slope has declined.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I now have a colleague who thinks I cannot say that the value of the slope declined between year 1 and year 2.  Rather, they think that I can only say that there is no relationship between the predictor and response in year 2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Am I correct in interpreting my results that the slope of the year 2 relationship is indeed ~0?  Or is there a problem with my interpretation?&lt;/p&gt;&#xA;" OwnerUserId="101" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-29T09:44:26.790" LastActivityDate="2010-12-29T09:44:26.790" Title="How to tell if the slope of a line is 0 or there is just no relationship?" Tags="&lt;regression&gt;&lt;statistical-significance&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="5789" PostTypeId="2" ParentId="5788" CreationDate="2010-12-28T17:54:58.717" Score="6" ViewCount="" Body="&lt;p&gt;Well, your colleague definitely is incorrect, as many standard examples will show.  For example, for the data $(x, x(1-x))$ with values of $x$ equal to $0, 1/n, 2/n, \ldots, 1$, the slope is exactly zero but obviously there's a strong (quadratic) relationship between &quot;predictor&quot; ($x$) and &quot;response&quot; ($x(1-x)$).  Your colleague would be correct to say &quot;there is no &lt;em&gt;significant&lt;/em&gt; evidence of a &lt;em&gt;linear&lt;/em&gt; relationship in year 2.&quot;  But you can do more than that, as you know.  You can assess the change between the two years.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Asserting that the &quot;slope has declined&quot; compares two &lt;em&gt;estimates&lt;/em&gt; to each other.  As such, you need to account for the uncertainty in &lt;em&gt;both&lt;/em&gt; estimates, not just in the one for year two.  From the mere fact that one slope is significantly different from zero and the other slope is not, you cannot legitimately conclude their difference is significant.  However, a valid comparison is simple to do: associated with each slope is a standard error (a routine part of regression software output) and a degree of freedom (usually equal to the number of data values minus the number of parameters).  &lt;em&gt;Assuming no temporal correlation&lt;/em&gt; in the error terms, you can compute the standard error for the difference in slopes as usual: take the root of the sum of squares of their SEs.  Refer the t-statistic (&lt;em&gt;i.e.&lt;/em&gt;, the ratio of the slope difference to the SE of their difference) to Student's t distribution.  The degree of freedom to use equals the sum of the DFs of the individual slopes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The usual caveats and assumptions about tests hold, of course.  But presumably, because you're already using linear regressions in each of the years separately, you have checked that suitable assumptions are reasonable for these data.  About the only additional thing you might want to check is that the variances of the residuals are approximately the same in the two years.  (You could use an F-test for this.)  If not, that's worth exploring further in order to understand just how the change came about.  Assuming it was abrupt and there's no sign of temporal trends in the residuals (no &lt;a href=&quot;http://en.wikipedia.org/wiki/Heteroscedasticity&quot; rel=&quot;nofollow&quot;&gt;heteroscedasticity&lt;/a&gt;), you might consider using a &lt;a href=&quot;http://en.wikipedia.org/wiki/Student%27s_t-test#Unequal_sample_sizes.2C_unequal_variance&quot; rel=&quot;nofollow&quot;&gt;t-test for unequal variances&lt;/a&gt; instead of the simpler test described above.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-12-28T17:54:58.717" CommentCount="5" />
  <row Id="5790" PostTypeId="2" ParentId="5782" CreationDate="2010-12-28T19:23:58.687" Score="5" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've underestimated Taylor expansions. They actually work. I assumed that integral of the remainder term can be unbounded, but with a little work it can be shown that this is not the case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Taylor expansion works for functions in bounded closed interval. For random variables with finite variance  &lt;a href=&quot;http://en.wikipedia.org/wiki/Tchebycheff_inequality&quot; rel=&quot;nofollow&quot;&gt;Chebyshev inequality&lt;/a&gt; gives&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(|X-EX|&amp;gt;c)\le \frac{Var(X)}{c}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So for any $\varepsilon&amp;gt;0$ we can find large enough $c$ so that &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(X\in [EX-c,EX+c])=P(|X-EX|\le c)&amp;lt;1-\varepsilon$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First let us estimate $Ef(X)$. We have&#xA;\begin{align}&#xA;Ef(X)=\int_{|x-EX|\le c}f(x)dF(x)+\int_{|x-EX|&gt;c}f(x)dF(x)&#xA;\end{align}&#xA;where $F(x)$ is the distribution function for $X$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since the domain of the first integral is interval $[EX-c,EX+c]$ which is bounded closed interval we can apply Taylor expansion:&#xA;\begin{align}&#xA;f(x)=f(EX)+f'(EX)(x-EX)+\frac{f''(EX)}{2}(x-EX)^2+\frac{f'''(\alpha)}{3}(x-EX)^3&#xA;\end{align}&#xA;where $\alpha\in [EX-c,EX+c]$, and the equality holds for all $x\in[EX-c,EX+c]$. I took only 4 terms in the Taylor expansion, but in general we can take as many as we like, as long as function $f$ is smooth enough.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Substituting this formula to the previous one we get&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align}&#xA;Ef(X)&amp;amp;=\int_{|x-EX|\le c}f(EX)+f'(EX)(x-EX)+\frac{f''(EX)}{2}(x-EX)^2dF(x)\\&#xA;&amp;amp;+\int_{|x-EX|\le c}\frac{f'''(\alpha)}{3}(x-EX)^3dF(x)&#xA;+\int_{|x-EX|&gt;c}f(x)dF(x)&#xA;\end{align}&#xA;Now we can increase the domain of the integration to get the following formula&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align}&#xA;Ef(X)&amp;amp;=f(EX)+\frac{f''(EX)}{2}E(X-EX)^2+R_3\\&#xA;\end{align}&#xA;where&#xA;\begin{align}&#xA;R_3&amp;amp;=\frac{f'''(\alpha)}{3}E(X-EX)^3+\\&#xA;&amp;amp;+\int_{|x-EX|&gt;c}\left(f(EX)+f'(EX)(x-EX)+\frac{f''(EX)}{2}(x-EX)^2+f(X)\right)dF(x)&#xA;\end{align}&#xA;Now under some moment conditions we can show that the second term of this remainder term is as large as $P(|X-EX|&amp;gt;c)$ which is small. Unfortunately the first term remains and so the quality of the approximation depends on $E(X-EX)^3$ and the behaviour of third derivative of $f$ in bounded intervals. Such approximation should work best for random variables with $E(X-EX)^3=0$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now for the variance we can use Taylor approximation for $f(x)$, subtract the formula for $Ef(x)$ and square the difference. Then&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$E(f(x)-Ef(x))^2=(f&amp;#39;(EX))^2Var(X)+T_3$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $T_3$ involves moments $E(X-EX)^k$ for $k=4,5,6$. We can arrive at this formula also by using only first-order Taylor expansion, i.e. using only the first and second derivatives. The error term would be similar.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other way is to expand $f^2(x)$:&#xA;\begin{align}&#xA;f^2(x)&amp;amp;=f^2(EX)+2f(EX)f'(EX)(x-EX)\\&#xA;&amp;amp;+(f'(EX))^2+f(EX)f''(EX)^2+\frac{(f^2(\beta))'''}{3}(X-EX)^3&#xA;\end{align}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Similarly we get then&#xA;\begin{align*}&#xA;Ef^2(x)=f^2(EX)+[(f'(EX))^2+f(EX)f''(EX)]Var(X)+\tilde{R}_3&#xA;\end{align*}&#xA;where $\tilde{R}_3$ is similar to $R_3$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The formula for variance then becomes&#xA;\begin{align}&#xA;Var(f(X))=[f'(EX)]^2Var(X)+\frac{[f''(EX)]^2}{4}Var^2(X)+\tilde{T}_3&#xA;\end{align}&#xA;where $\tilde{T}_3$ have only third moments.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2010-12-30T08:54:55.643" LastActivityDate="2010-12-30T08:54:55.643" CommentCount="7" />
  <row Id="5791" PostTypeId="1" AcceptedAnswerId="5823" CreationDate="2010-12-28T19:53:11.963" Score="4" ViewCount="203" Body="&lt;h2&gt;Background&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I have a model with 17 parameters, and I currently use the coefficient of variation ($\text{CV}=\sigma/\mu$) to summarize the prior and posterior distributions of each parameter. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;All of the parameters are &gt; 0. I would also like to summarize these pdfs on a normalized scale (in this case standard deviation normalized by the mean) so that they can be compared to each other, and with other statistics presented in similar adjacent plots (sensitivity, explained variance). I will include density plots for each parameter separately, but I would like to summarize them here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the sensitivity of the CV to $\mu$ causes the following confusion that, although easily explained in text, would be preferable to avoid.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;the posterior CV of one parameter is greater than the prior because the mean has decreased more than the variance (parameter &lt;code&gt;O&lt;/code&gt; in figure).&lt;/li&gt;&#xA;&lt;li&gt;one of the parameters (&lt;code&gt;N&lt;/code&gt;) is in units of temperature. It has a 95% prior CI of (8,12 Celsius $\simeq$ 281-285K); when I present the data in units of Kelvin which is only defined for positive values, the CV is &amp;lt;1%, if presented as C, the CV is closer to 40%. To me, it seems that neither of these CVs provides an intuitive representation of the CI.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;h2&gt;Question&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Are there better ways to present this information, either as a CV or as another statistic?&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Figure&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;As an example, this is the type of plot that I am planning to present, with posterior CV in black and prior CV in grey. For scale, the CV of parameter &lt;code&gt;O&lt;/code&gt; is 1.6.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/CVeO1.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="1381" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-01-10T17:42:26.800" LastActivityDate="2011-01-10T17:42:26.800" Title="Alternatives to using Coefficient of Variation to summarize a set of parameter distributions?" Tags="&lt;distributions&gt;&lt;data-visualization&gt;&lt;summary-statistics&gt;" AnswerCount="2" CommentCount="6" />
  <row Id="5792" PostTypeId="1" AcceptedAnswerId="5793" CreationDate="2010-12-28T19:56:05.327" Score="2" ViewCount="81" Body="&lt;p&gt;How can I extract the R-square from a fit &lt;a href=&quot;http://cran.r-project.org/web/packages/rpart/index.html&quot; rel=&quot;nofollow&quot;&gt;rpart&lt;/a&gt; model?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;rsq.rpart(fit)&lt;/code&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;plots the two graphs, but I simply want to extract the R-square value for the full tree.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I assume this is fairly obvious, but numerous searches didn't really lend anything useful.  Any help you can provide will be greatly appreciated.&lt;/p&gt;&#xA;" OwnerUserId="569" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-12-28T20:31:25.117" LastActivityDate="2010-12-28T20:31:25.117" Title="R-square from rpart model" Tags="&lt;r&gt;&lt;rpart&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="5793" PostTypeId="2" ParentId="5792" CreationDate="2010-12-28T20:28:57.350" Score="3" ViewCount="" Body="&lt;p&gt;The advantage of R is that most of the time you can easily access the source code. So in your case, start with&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; rsq.rpart&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(without parenthesis) to see what the function actually does. The $R^2$ values are obtained as&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;tmp &amp;lt;- printcp(fit)&#xA;rsq.val &amp;lt;- 1-tmp[,c(3,4)]  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where for each row (aka, No. splits) we have the &quot;apparent&quot; and &quot;relative&quot; (wrt. cross-validation) statistics.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-12-28T20:28:57.350" CommentCount="3" />
  <row Id="5795" PostTypeId="2" ParentId="5786" CreationDate="2010-12-28T22:01:12.497" Score="1" ViewCount="" Body="&lt;p&gt;I guess you know that the kernel for logistic regression is a non parametric one, so first of all you have that restriction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regarding the R package the one I know and works pretty well is &lt;a href=&quot;http://cran.r-project.org/web/packages/np/index.html&quot; rel=&quot;nofollow&quot;&gt;np&lt;/a&gt;: Nonparametric kernel smoothing methods for mixed data types&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This package provides a variety of nonparametric (and semiparametric) kernel methods that seamlessly handle a mix of continuous, unordered, and ordered factor data types. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regarding the state of the art kernell I can recomend to experiment with the ones described in this &lt;a href=&quot;http://www.cs.mcgill.ca/~icml2009/papers/505.pdf&quot; rel=&quot;nofollow&quot;&gt;paper&lt;/a&gt; from 2009. Read it carefully to choose the one that is best and more actual for you.&lt;/p&gt;&#xA;" OwnerUserId="1808" LastActivityDate="2010-12-28T22:01:12.497" CommentCount="1" />
  <row Id="5796" PostTypeId="2" ParentId="5786" CreationDate="2010-12-28T22:30:00.733" Score="2" ViewCount="" Body="&lt;p&gt;Gaussian process classification (using Expectation Propagation) is probably the state-of-the-art in machine learning.  There is an excellent &lt;a href=&quot;http://www.gaussianprocess.org/gpml/chapters/&quot; rel=&quot;nofollow&quot;&gt;book&lt;/a&gt; by Rasmussen and Williams (downloadable for free), the &lt;a href=&quot;http://www.gaussianprocess.org/gpml/&quot; rel=&quot;nofollow&quot;&gt;website&lt;/a&gt; for which has a very good MATLAB implementation.  More software, books, papers etc. &lt;a href=&quot;http://www.gaussianprocess.org/&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.  However, in practice, KLR will probably work just as well for most problems, the major difficulty is in selecting the kernel and regularisation parameters, which is probably best done by cross-validation, although leave-one-out cross-validation can be approximated very efficiently, see &lt;a href=&quot;http://dx.doi.org/10.1007/s10994-008-5055-9&quot; rel=&quot;nofollow&quot;&gt;Cawley and Talbot (2008).&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2010-12-28T22:30:00.733" CommentCount="5" />
  <row Id="5797" PostTypeId="1" AcceptedAnswerId="5824" CreationDate="2010-12-28T22:45:59.357" Score="5" ViewCount="118" Body="&lt;p&gt;Given $K$ possible 'treatments' of some kind, and independent observations of some response under those treatments, say $X_{i,k}$ for $i=1,\ldots,n_k$ and $k=1,\ldots,K$, I am faced with the classical data-mining dilemma. The task is to simultaneously:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Find the treatment, say $k^*$, which maximizes effect, and&lt;/li&gt;&#xA;&lt;li&gt;Estimate the effect size of treatment $k^*$.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Here you can think of effect size as a measure of location, say mean. The naive approach to this problem is as follows:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Estimate the effect size of each treatment. For example, if the effect size is the mean, let $\bar{X_k} = (\sum_i X_{i,k}) / n_k$.&lt;/li&gt;&#xA;&lt;li&gt;Pick the treatment which maximizes the estimated effect, &lt;em&gt;i.e.&lt;/em&gt; let $k^* = \arg\max_k \bar{X_k}$.&lt;/li&gt;&#xA;&lt;li&gt;Estimate the effect size of $k^*$ by $\bar{X_{k^*}}$.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The problem, of course, is that the estimate of the optimal effect size is positively biased because the choice of optimal treatment is not independent of the estimated effect sizes, which are unconditionally unbiased.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Additionally, the goals are often somewhat broader: instead of picking the treatment which maximizes effect, we are tasked with sorting the treatments by estimated effect size and estimating the effect sizes. Because of this, I am looking for some kind of function $f$ which 'unbiases' the estimated effect sizes, but preserves monotonicity. That is, $f$ takes a $K$-vector to a $K$-vector, and&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;$f$ preserves monotonicity: if $\bar{X_{i_1}} \le \bar{X_{i_2}} \le \ldots \le \bar{X_{i_K}}$ and $(Y_1,Y_2,\ldots,Y_K) = f(\bar{X_1},\bar{X_2},\ldots,\bar{X_K})$, then $Y_{i_1} \le Y_{i_2} \le \ldots \le Y_{i_K}$.&lt;/li&gt;&#xA;&lt;li&gt;$f$ (approximately, possibly under weird assumptions) unbiases the effect size estimates: if $(Y_1,Y_2,\ldots,Y_K) = f(\bar{X_1},\bar{X_2},\ldots,\bar{X_K})$, and $Y_{i_1} \le Y_{i_2} \le \ldots \le Y_{i_K}$, then $Y_{i_j}$ is a (nearly) unbiased estimate of the effect size of the $j$th smallest effect treatment.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Probably such a function would have to be aware of some additional information, like the number of observations $n_k$ and some measure of the spread of the response under each treatment. Probably the monotonicity requirement is questionable if the standard errors of the effect estimates vary wildly. Sweep that under the rug if possible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am also interested in approaches to this problem when the response observations are not independent, rather they are paired..&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2011-01-05T01:32:02.787" Title="Is there a bias correction for effect size in a data mining context?" Tags="&lt;multiple-comparisons&gt;&lt;multivariate-analysis&gt;&lt;data-mining&gt;&lt;unbiased-estimator&gt;&lt;bias-correction&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="5798" PostTypeId="1" CreationDate="2010-12-29T00:04:34.793" Score="-3" ViewCount="290" Body="&lt;p&gt;Hi all,&#xA;I am interested in calculating the probability that species x falls into at least one shared category in two distinct populations, using the R statistical software program. &#xA;Any help will be greatly appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you,&#xA;R Newbie&lt;/p&gt;&#xA;" OwnerUserId="2556" LastActivityDate="2010-12-29T04:17:31.153" Title="How to calculate Probability using R software" Tags="&lt;r&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="5799" PostTypeId="2" ParentId="5750" CreationDate="2010-12-29T01:22:48.120" Score="3" ViewCount="" Body="&lt;p&gt;Perhaps you could do a preliminary analysis on a random subset of the data to form hypotheses, and then test those few hypotheses of interest using the rest of the data. That way you would not have to correct for nearly as many multiple tests. (I think...)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, if you use such a procedure you will be reducing the size of the dataset used for the final analysis and so reduce your power to find real effects. However, corrections for multiple comparisons reduce power as well and so I'm not sure that you would necessarily lose anything.&lt;/p&gt;&#xA;" OwnerUserId="1679" LastActivityDate="2010-12-29T01:22:48.120" CommentCount="1" />
  <row Id="5800" PostTypeId="2" ParentId="5798" CreationDate="2010-12-29T03:02:00.247" Score="0" ViewCount="" Body="&lt;p&gt;Definitely more information is needed.  If the two populations are assumed to come from a multivariate normal distribution, you could use the lda() or qda() function in R.  This will classify a set of observations based on the highest probability of coming from a particular population.  Here's how to compute these mathematically:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;mean of group 1 = mu.1.b&#xA;mean of group 2 = mu.2.b&#xA;inverse of pooled covariance of both groups = S.p.b.inv&#xA;prior probability of coming from population 1 or 2 = p1 or p2&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;t(mu.1.b)%*%S.p.b.inv%*%c(x1[i],x2[j])-.5*t(mu.1.b)%*%S.p.b.inv%*%mu.1.b + log(p1)&#xA;&#xA;t(mu.2.b)%*%S.p.b.inv%*%c(x1[i],x2[j])-.5*t(mu.2.b)%*%S.p.b.inv%*%mu.2.b + log(p2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is for the linear.  You can do a very similar thing for the qda analysis to determine the probability of coming from each population.  The qda function does not assume equal covariance matrices between the two populations.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can extract these probabilities with the &lt;code&gt;predict()&lt;/code&gt; function with the &lt;code&gt;$post&lt;/code&gt; statment.  &lt;code&gt;predict(lda(inputdatahere))$post&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="2539" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2010-12-29T04:17:31.153" LastActivityDate="2010-12-29T04:17:31.153" />
  <row Id="5801" PostTypeId="1" AcceptedAnswerId="5805" CreationDate="2010-12-29T06:09:32.537" Score="1" ViewCount="89" Body="&lt;p&gt;Sorry if this is a n00b question, I'm just trying to wrap my head around the problem. I am trying to refute conventional wisdom here, so any help is greatly appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And now for the question:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, I flip a coin 100 times and I end up with 50 consecutive heads followed by 50 consecutive tails. How would I calculate the probability or likelihood that the next time I replay the scenario that I will end up with the exact same result? Would the equation be different if we were using a ternary or quaternary event instead of a binary event?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Conventional, non-statistical, wisdom would assume that coming to the same result on consecutive attempts would be incredibly remote, but that the odds of eventually getting an identical result would increase as more attempts are made, i.e. 50/50, 50/50 would be implausible, but 50/50, 49/51, 51/49 ... 50/50 would be plausible.&lt;/p&gt;&#xA;" OwnerUserId="2131" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2010-12-29T06:13:55.593" LastActivityDate="2010-12-29T09:30:18.017" Title="Is it possible to predict the likelihood of an order of random events?" Tags="&lt;probability&gt;&lt;randomness&gt;" AnswerCount="2" />
  <row Id="5802" PostTypeId="2" ParentId="4356" CreationDate="2010-12-29T06:58:48.683" Score="3" ViewCount="" Body="&lt;p&gt;Rpart only provides univariate splits. I believe, based upon your question, that you are not entirely familiar with the difference between a univariate partitioning method and a multivariate partitioning method. I have done my best to explain this below, as well as provide some references for further research and to suggest some R packages to implement these methods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rpart is a tree based classifier that uses recursive partitioning. With partitioning methods you must define the points within your data at which a split is to be made. The rpart algorithm in R does this by finding the variable and the point which best splits (and thus reduces) the RSS. Because the splits only happen along one variable at a time, these are univariate splits. A Multivariate Split is typically defined as a simultaneous partitioning along multiple axis (hence multivariate), i.e. the first rpart node might split along Age&gt;35, the second node might split along Income &gt;25,000, and the third node might split along Cities west of the Mississippi. The second and third nodes are split on smaller subsets of the overall data, so in the second node the income criterion best splits the RSS only for those people who have an age of over 35, it does not apply to observations not found in this node, the same applies for the Cities criterion. One could continue doing this until there is a node for each observation in your dataset (rpart uses a minimum bucket size function in addition to a minimum node size criterion and a cp parameter which is the minimum the r-squared value must increase in order to continue fitting). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A multivariate method, such as Patient Rule Induction Method (the prim package in R) would simultaneously split by selecting, for example, All Observations where Income was Greater than 22,000, Age&gt;32, and Cities West of Atlanta. The reason why the fit might be different is because the calculation for the fit is multivariate instead of univariate, the fit of these three criterion is calculated based upon the simultaneous fit of the three variables on all observations meeting these criterion rather than iteratively partitioning based upon univariate splits (as with rpart).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are varying beliefs in regards to the effectiveness of univariate versus multivariate partitioning methods. Generally what I have seen in practice, is that most people prefer univariate partitioning (such as rpart) for explanatory purposes (it is only used in prediction when dealing with a problem where the structure is very well defined and the variation among the variables is fairly constant, this is why these are often used in medicine). Univariate tree models are typically combined with ensemble learners when used for prediction (i.e. a Random Forest). People who do use multivariate partitioning or clustering (which is very closely related to multivariate partitioning) often do so for complex problems that univariate methods fit very poorly, and do so mainly for prediction, or to group observations into categories.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I highly recommend Julian Faraway's book Extending the Linear Model with R. Chapter 13 is dedicated entirely to the use of Trees (all univariate). If you're interested further in multivariate methods, Elements of Statistical Learning by Hastie et. al, provides an excellent overview of many multivariate methods, including PRIM (although Friedman at Stanford has his original article on the method posted on his website), as well as clustering methods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In regards to R Packages to utilize these methods, I believe you're already using the rpart package, and I've mentioned the prim package above. There are various built in clustering routines, and I am quite fond of the party package mentioned by another person in this thread, because of its implementation of conditional inference in the decision tree building process. The optpart package lets you perform multivariate partitioning, and the mvpart package (also mentioned by someone else) lets you perform multivariate rpart trees, however I personally prefer using partDSA, which lets you combine nodes further down in your tree to help prevent partitioning of similar observations, if I feel rpart and party are not adequate for my modeling purposes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: In my example of an rpart tree in paragraph 2, I describe how partitioning works with node numbers, if one were to draw out this tree, the partitioning would proceed to the left if the rule for the split was true, however in R I believe the split actually proceeds to the right if the rule is true.&lt;/p&gt;&#xA;" OwnerUserId="2166" LastEditorUserId="2166" LastEditorDisplayName="" LastEditDate="2010-12-30T06:06:56.613" LastActivityDate="2010-12-30T06:06:56.613" CommentCount="6" />
  <row Id="5805" PostTypeId="2" ParentId="5801" CreationDate="2010-12-29T09:29:59.850" Score="2" ViewCount="" Body="&lt;p&gt;You have some event with certain probability of success. So the probability that the next time this event happens is the success probability. In your case you have flipped coin 100 times with 50 consecutive heads and 50 consecutive tails. The probability of this event happening is $1/2^{100}$. If you repeat this experiment independent of the first one, the probability that it will happen remains the same. &#xA;Although you generated 100 random &lt;a href=&quot;http://en.wikipedia.org/wiki/Bernoulli_distribution&quot; rel=&quot;nofollow&quot;&gt;Bernoulli&lt;/a&gt; variables with probability $p=1/2$, your random variable of interest is a Bernoulli with probability $p=1/2^{100}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The probability that that eventually you get the same result increases with number of trials. This number of trials needed for success is random variable with &lt;a href=&quot;http://en.wikipedia.org/wiki/Geometric_distribution&quot; rel=&quot;nofollow&quot;&gt;geometric distribution&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you use ternary or quartenary event the same holds, as long as your a looking for specific event to happen. Then calculate probability of this event and again you have a Bernoulli random variable with this probability. &lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2010-12-29T09:29:59.850" />
  <row Id="5806" PostTypeId="2" ParentId="5801" CreationDate="2010-12-29T09:30:18.017" Score="3" ViewCount="" Body="&lt;p&gt;The probability of 50 tails followed by 50 heads, assuming balanced iid bernoulli coins, is simply $\prod_{i=1}^{100} P(Event_i=Predicted_i)=\frac 1 2 ^{50} \times \frac 1 2 ^{50} = 2^{-100}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;50/50 is just as unlikely as 49/51, if order must be preserved.&lt;/p&gt;&#xA;" OwnerUserId="2456" LastActivityDate="2010-12-29T09:30:18.017" />
  <row Id="5807" PostTypeId="1" CreationDate="2010-12-29T14:31:10.063" Score="3" ViewCount="225" Body="&lt;p&gt;Colleagues of mine recently presented a work where they calibrate boosted regression trees (BRT) models on small data sets ($n= 30$). They validated the models using leave-one-out cross validation (LOOCV) using R2, RMSPE and RPD indices. They also provided these indices computed by training and validating the model on the full dataset. The R2, RMSPE and RPD values obtained through LOOCV were almost strictly equal to the R2, RMSPE and RPD values obtained when validating on the training data set. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My questions are :&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Is such a results expected for LOOCV on BRT?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Is this because BRT is relatively insensitive to outliers (and to single individuals?) that excluding one individual during LOOCV does not make a difference, providing nearly similar calibrated models with same performance metrics on the excluded individuals?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;In that case does LOOCV for BRT makes any sense, compared to repeated k-fold CV with $k &amp;lt; n$?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Thank you in advance&lt;/p&gt;&#xA;" OwnerUserId="2561" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-29T17:28:11.153" LastActivityDate="2010-12-29T21:20:27.750" Title="Leave-one-out cross validation and boosted regression trees" Tags="&lt;regression&gt;&lt;machine-learning&gt;&lt;cross-validation&gt;" AnswerCount="1" />
  <row Id="5808" PostTypeId="1" AcceptedAnswerId="5809" CreationDate="2010-12-29T15:14:45.243" Score="3" ViewCount="367" Body="&lt;p&gt;One of the ANCOVA assumptions, as I read it &lt;a href=&quot;http://www.google.gr/url?sa=t&amp;amp;source=web&amp;amp;cd=2&amp;amp;sqi=2&amp;amp;ved=0CCoQFjAB&amp;amp;url=http%3A%2F%2F161.111.161.171%2Festadistica2004%2FANOVA%2FANCOVA.pdf&amp;amp;rct=j&amp;amp;q=ancova%20assumptions%20covariates%20independent%20randomization%20&amp;amp;ei=0UobTcOkKoey8gOq9rCeBQ&amp;amp;usg=AFQjCNFzfXuU154mlSPmMA8oN48fxXhRiw&amp;amp;sig2=6udO2dbErcUJeB_qi4kd_w&amp;amp;cad=rja&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;, says that:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;&lt;strong&gt;Independent variables orthogonal to covariates&lt;/strong&gt;. In traditional ANCOVA, the independents area assumed to be orthogonal to the factors. If the covariate is influenced by the categorical independents, then the control adjustment ANCOVA makes on the dependent variable prior to assessing the effects of the categorical independents will be biased since some indirect effects of the independents will be removed from the dependent.&lt;/em&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;See also the third assumption &lt;a href=&quot;http://en.wikiversity.org/wiki/Advanced_ANOVA/ANCOVA&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could you tell me how should I verify that this assumption is met, in a given model containing three factors and one covariate?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can refer to SPSS or R ways of doing this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you very much&lt;/p&gt;&#xA;" OwnerUserId="339" LastActivityDate="2010-12-29T17:12:47.310" Title="Checking factor / covariate independence in ANCOVA" Tags="&lt;r&gt;&lt;spss&gt;&lt;ancova&gt;&lt;assumptions&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="5809" PostTypeId="2" ParentId="5808" CreationDate="2010-12-29T16:02:23.433" Score="4" ViewCount="" Body="&lt;p&gt;One approach is to see if the covariate is correlated with the predictor variables. That is, if the ANCOVA is given by:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;predicted ~ covariate + predictor1*predictor2*predictor3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then first assess whether the covariate and the various predictor effects/interactions are correlated:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;covariate ~ predictor1*predictor2*predictor3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you find that the covariate is correlated with any of the predictor variables or their interaction, then you're violating the assumption you cite. If the covariate is categorical with more than 2 levels, you'll have to assess the correlations via multinomial regression.&lt;/p&gt;&#xA;" OwnerUserId="364" LastEditorUserId="364" LastEditorDisplayName="" LastEditDate="2010-12-29T16:35:50.477" LastActivityDate="2010-12-29T16:35:50.477" CommentCount="6" />
  <row Id="5810" PostTypeId="2" ParentId="5808" CreationDate="2010-12-29T16:39:58.993" Score="0" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt; As pointed out in the comments below, this is an answer to a different question, one about ensuring that the effect of the covariate is the same in all groups. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You have to check for interactions of the covariate and the factors. So if the ANCOVA model was&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;lm1 &amp;lt;- lm(outcome ~ covariate + factor1*factor2*factor3)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;then the you can add all the interactions and look at&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;lm2 &amp;lt;- lm(outcome ~ covariate*factor1*factor2*factor3)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;followed by an F-test:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;anova(lm1, lm2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It might also make sense to add fewer interactions (especially if you don't have a lot of data), so that the large number of high order interactions don't eat up the power.&lt;/p&gt;&#xA;" OwnerUserId="279" LastEditorUserId="279" LastEditorDisplayName="" LastEditDate="2010-12-29T17:12:47.310" LastActivityDate="2010-12-29T17:12:47.310" CommentCount="2" />
  <row Id="5812" PostTypeId="2" ParentId="5807" CreationDate="2010-12-29T17:27:19.787" Score="2" ViewCount="" Body="&lt;p&gt;It is hard to tell without data, but the set may be &quot;too homogeneous&quot; to make LOO work -- imagine you have a set $X$ and you duplicate all objects to make a set $X_d$ -- while BRT usually have very good accuracy on its train, it is pretty obvious that LOO on $X_d$ will probably give identical results to test-on-train.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So if the accuracy if good I would even try resampling CV (on each of let's say 10 folds you make train of an equal size to the full set by sampling objects with replacement and test from objects that were not placed in train -- this should spit them in about 1:2 proportion) on this data to verify this result. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: More precise algorithm of resampling CV  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given a dataset with $N$ objects and $M$ attributes:  &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Training set is made by randomly selecting $N$ objects from the original set &lt;strong&gt;with replacement&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;li&gt;The objects that were not selected in the step 1 form the test set (this is roughly $\frac{1}{3}N$ objects) &lt;/li&gt;&#xA;&lt;li&gt;Classifier is trained on a train set and tested on test set, and the measured error is gathered&lt;/li&gt;&#xA;&lt;li&gt;Steps 1-3 are repeated $T$ times, where $T$ is more less arbitrary, say 10, 15 or 30&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="88" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-29T21:20:27.750" LastActivityDate="2010-12-29T21:20:27.750" CommentCount="8" />
  <row Id="5813" PostTypeId="1" AcceptedAnswerId="5816" CreationDate="2010-12-29T17:48:41.317" Score="1" ViewCount="95" Body="&lt;p&gt;I am a beginner level programmer preparing for the interview in medical research company. Job sounds damn interesting and I would like to get there.&#xA;To show my skills and interest, I want to write a program related to the topic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think, statistical analysis is quite used in that field, isn't it?&#xA;What would you suggest, as idea for demo program? Calculation of basic statistical parameters of input data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;UPDATE: @chl: my background: beginner Silverlight programmer, with basic (college) level of statistics; my intentions: small application, that shows I know Silverlight &amp;amp; I have interest in medical research&lt;/p&gt;&#xA;&#xA;&lt;p&gt;@onestop:&#xA;company's area of activity: research in proteins, metabolics, central nervous system; job specification: I have just basic requirements, and there is Silverlight knowledge listed&lt;/p&gt;&#xA;" OwnerUserId="2564" LastEditorUserId="2564" LastEditorDisplayName="" LastEditDate="2010-12-29T22:09:28.347" LastActivityDate="2010-12-29T22:09:28.347" Title="Demo for bioinformatics" Tags="&lt;bioinformatics&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="5814" PostTypeId="1" CreationDate="2010-12-29T18:22:44.217" Score="2" ViewCount="99" Body="&lt;p&gt;&lt;a href=&quot;http://ops.fhwa.dot.gov/publications/fhwahop08024/chapter6.htm&quot; rel=&quot;nofollow&quot;&gt;Traffic light synchronization&lt;/a&gt; nowadays is not a tedious project, Image: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/KtnTF.png&quot; alt=&quot;image&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a personal research project I'm trying to build a statistical model to solve such problems in two and higher dimensions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to synchronize &lt;code&gt;x&lt;/code&gt; 4 legged (approaches) intersections such that:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;The intersections are all located within the same area in a grid type format. I'd like to think of this as a 1/0 matrix, where 1 indicates a signalized intersection and 0 doesn't:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Matrix Example&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[1 1 1 1]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[0 1 0 1]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[1 1 1 1]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This assumes that all nodes are connected in a vertical or horizontal but not diagonal manner and that they could have a direction of either one or two way.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Each Intersection has a traffic light on each approach&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;The allowable movements per leg are left, right, and Through (lrt) and no U-Turns&lt;/li&gt;&#xA;&lt;li&gt;Each Leg can be one lane, i.e. one lane that allows for lrt or approaches can have combined, through and right (1 lane), and left (1 lane, or dedicated such as alpha left lanes, beta through lanes, and gamma right lanes. Some intersections should allow for restricted or no movement, as in an intersection could be setup as &quot;no-left turns allowed&quot;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Each lane should have a distinct vehicle flow (vehicles / minutes (Time)) per movement, so a two lane (left + through, and right) approach could have the following movement:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Left + Through lane: 10 vehicles / minute (through) and 5 vehicles per minute turning left&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Right lane: 15 vehicles per minute&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Assume green cycle time (time from beginning of green to beginning second green to depend on the vehicle volumes, yet could only be between t1 and t2 seconds. Green time = red time + yellow time). Yellow time should be assumed as a constant either 1, 2, or 3 seconds (depending on the jurisdiction). if pedestrian crossing is warranted, the red time should be assumed to be &quot;white go ped: 4 seconds x lane&quot; + &quot;blinking ped: 4 seconds x lane&quot;. remember that the number of lanes would be from the opposing approaches combined.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;The system should optimize green time so that maximum flow is passed through the system at all intersections depending no the existing volumes &lt;strong&gt;and&lt;/strong&gt; on historical values.&lt;/li&gt;&#xA;&lt;li&gt;Although there are historical data on the number of vehicles that enter the green network, I assume it is best to use a Monte Carlo simulation to simulate vehicle entry into the system and then calibrated to historical data. &lt;/li&gt;&#xA;&lt;li&gt;The end result should be an equation or system of equations that will give you the green time for each intersection depending on volumes and what to expect from the other intersections.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I know this might not be the clearest question, but I'm happy to answer any questions and or comments.&lt;/p&gt;&#xA;" OwnerUserId="59" LastActivityDate="2010-12-29T18:22:44.217" Title="Help in setting up and solving a transportation / traffic problem" Tags="&lt;monte-carlo&gt;&lt;transportation&gt;" CommentCount="2" />
  <row Id="5816" PostTypeId="2" ParentId="5813" CreationDate="2010-12-29T20:03:22.763" Score="1" ViewCount="" Body="&lt;p&gt;Write a script that aligns short DNA fragments (25-100 basepairs) to an existing genome. &lt;/p&gt;&#xA;" OwnerUserId="1795" LastActivityDate="2010-12-29T20:03:22.763" CommentCount="3" />
  <row Id="5817" PostTypeId="1" AcceptedAnswerId="5822" CreationDate="2010-12-29T20:17:43.350" Score="2" ViewCount="242" Body="&lt;p&gt;I want to know if a covariate for each subject interacts with three types of trials, and the difficulty of those trials. My dependent measures are accuracy and response times (RT). For this question, I’d like to focus on RTs. Traditionally, people in my field have dichotomized the covariate of interest and used ANOVAs for analysis. I would like to treat the covariate as the continuous variable it is, and treat the subjects as random effects. I want to analyze this using mixed-models in R(nlme).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first 2 trial types can be either easy or hard and the third trial type is a combination of the first 2. These trials can be easy-easy, easy-hard, hard-easy, hard-hard.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I expect people who have higher scores on the covariate to show a smaller difference between hard and easy RTs for at least 1 trial type.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a repeated-measures design with each subject completing 3 blocks of 40 trials of each of the trial types (for trialtypes 1 &amp;amp; 2: 20 easy, 20 hard; for trialtype3, 10 easy-easy, 10 easy-hard, 10 hard-hard, 10 hard-easy). Stated differently, each subject completes 3 blocks of 120 trials with the various trialtypes randomly ordered. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Only RTs for correct trials will be analyzed (resulting in an unbalanced design for RT data). Besides a counter-balancing of response keys, this is a completely within-subjects design.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To summarize, what is the model (or models) that will allow me to test for interactions between trialtypes, difficulty, and the covariate using nlme in R? &lt;/p&gt;&#xA;" OwnerUserId="2322" LastActivityDate="2010-12-29T22:11:52.237" Title="What is my model statement for mixed models (nlme) in R?" Tags="&lt;r&gt;&lt;mixed-model&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="5818" PostTypeId="2" ParentId="5696" CreationDate="2010-12-29T20:25:54.147" Score="0" ViewCount="" Body="&lt;p&gt;Here's a couple of ways to think about.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A) You could treat each full sequence of codings as a ordered set of events (i.e. [&quot;head nod&quot;, &quot;head shake&quot;, &quot;head nod&quot;, &quot;eyebrow raised&quot;] and [&quot;head nod&quot;, &quot;head shake&quot;, &quot;eyebrow raised&quot;]), then align the sequences using an algorithm that made sense to you ( &lt;a href=&quot;http://en.wikipedia.org/wiki/Sequence_alignment&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Sequence_alignment&lt;/a&gt; ). You could then compute inter coder reliability for the entire sequence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;B) Then, again using the aligned sequences, you could compare when they said an event happened, given that they both observed the event.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2)&#xA;Alternately, you could model this as a Hidden Markov Model, and use something like the Baumn-Welch algorithm to impute the probabilities that, given some actual event, each coder actually coded the data correctly. &lt;a href=&quot;http://en.wikipedia.org/wiki/Baum-Welch_algorithm&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Baum-Welch_algorithm&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="82" LastEditorUserId="82" LastEditorDisplayName="" LastEditDate="2010-12-30T21:38:28.320" LastActivityDate="2010-12-30T21:38:28.320" />
  <row Id="5819" PostTypeId="1" AcceptedAnswerId="5820" CreationDate="2010-12-29T20:27:46.593" Score="2" ViewCount="105" Body="&lt;p&gt;I am using &lt;a href=&quot;http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.gaussian_kde.html&quot; rel=&quot;nofollow&quot;&gt;&lt;code&gt;scipy.stats.gaussian_kde&lt;/code&gt;&lt;/a&gt; to estimate a pdf for some data. The problem is that the resulting pdf takes values larger than 1. As far as I understand, this should not happen. Am I mistaken? If so why?&lt;/p&gt;&#xA;" OwnerUserId="977" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-12-30T15:28:38.317" LastActivityDate="2010-12-30T15:28:38.317" Title="Kernel density estimate takes values larger than 1" Tags="&lt;distributions&gt;&lt;probability&gt;&lt;estimation&gt;&lt;python&gt;" AnswerCount="1" />
  <row Id="5820" PostTypeId="2" ParentId="5819" CreationDate="2010-12-29T20:33:48.510" Score="5" ViewCount="" Body="&lt;p&gt;You are mistaken. The CDF should not be greater than 1, but the PDF may be. Think, for example, of the PDF of a Gaussian random variable with mean zero and standard deviation $\sigma$:&#xA;$$f(x) = \frac{1}{\sqrt{2\sigma\pi}}\exp(-\frac{x^2}{2\sigma^2})$$&#xA;if you make $\sigma$ very small, then for $x = 0$, the PDF is arbitrarily large! &lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-12-29T20:33:48.510" CommentCount="3" />
  <row Id="5821" PostTypeId="2" ParentId="5733" CreationDate="2010-12-29T21:56:14.677" Score="0" ViewCount="" Body="&lt;p&gt;Hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The distributions do not appear to fit any known distribution.  I think it would be very dangerous to assume a parametric and then use that for monte carlo estimation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Isn't there some kind of bootstrap or &quot;empirical monte carlo&quot; method I can look at?&lt;/p&gt;&#xA;" OwnerUserId="2566" LastActivityDate="2010-12-29T21:56:14.677" CommentCount="4" />
  <row Id="5822" PostTypeId="2" ParentId="5817" CreationDate="2010-12-29T22:11:52.237" Score="0" ViewCount="" Body="&lt;p&gt;(This isn't an answer, simply an attempt at clarification of the question. I would have made this a comment to the question, but I haven't had success putting code in a comment)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Am I correct in interpreting your design as captured by the data frame resulting from the following (ignoring the arbitrary number of subjects and trials):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;a = data.frame(expand.grid(&#xA;    s = factor(1:10)&#xA;    , trial = 1:20&#xA;    , task1 = c('absent','easy','hard')&#xA;    , task2 = c('absent','easy','hard')&#xA;)); &#xA;a = a[!(a$task1=='absent' &amp;amp; a$task2=='absent'),]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Thus, &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;table(a$task1,a$task2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;yields&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;       easy absent hard&#xA;easy    200    200  200&#xA;absent  200      0  200&#xA;hard    200    200  200&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm more familiar with lme4 than nlme, but I know that the absent/absent cell will cause lmer to fail if you try a model with a task1:task2 interaction. Somehow we need to get lmer to recognize that this is a nested model with two tasks that are presented either singly or combined...&lt;/p&gt;&#xA;" OwnerUserId="364" LastActivityDate="2010-12-29T22:11:52.237" CommentCount="1" />
  <row Id="5823" PostTypeId="2" ParentId="5791" CreationDate="2010-12-29T22:49:05.940" Score="4" ViewCount="" Body="&lt;p&gt;It seems to me that CV is inappropriate here. I think you may be better off separating the change in location from the change in dispersion. In addition, the distributions you mention in your comment to the question are, for most parameter values, skewed (positively, except for the beta distribution). That makes me question whether the standard deviation is the best choice for a measure of dispersion; perhaps the interquartile range (IQR) might be better, or possibly the median absolute deviation? Similarly, rather than the mean, I might consider the median or the mode as the measure of location. The choice might in practice be determined by ease of computation as well as the field of application, the details of the model...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say you choose to use the IQR and the mode. You could summarise the change in dispersion using the ratio of posterior to prior IQRs, probably plotted on a log scale as that's usually appropriate for ratios. You could summarise the change in location using the ratio of the difference between posterior and prior modes to the prior IQR, or to the posterior IQR, or perhaps to the geometric mean of the posterior and prior IQRs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These are just some quick ideas that came to mind. I can't claim any strong underpinnings for them, or even any great personal attachment to them.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-12-29T22:49:05.940" CommentCount="2" />
  <row Id="5824" PostTypeId="2" ParentId="5797" CreationDate="2010-12-29T23:55:41.287" Score="0" ViewCount="" Body="&lt;p&gt;Ok, duh, one approach would be to use the James-Stein shrinkage. This will not, I believe, unbias the estimates, but will reduce the mean squared error.&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-12-29T23:55:41.287" />
  <row Id="5825" PostTypeId="2" ParentId="5791" CreationDate="2010-12-30T04:50:05.367" Score="4" ViewCount="" Body="&lt;p&gt;Some alternatives, which have the same 'flavor' as CV are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The coefficient of L-variation, see &lt;em&gt;e.g.&lt;/em&gt; &lt;a href=&quot;http://www.hydrol-earth-syst-sci-discuss.net/7/5467/2010/hessd-7-5467-2010.pdf&quot; rel=&quot;nofollow&quot;&gt;Viglione&lt;/a&gt;. The second sample L-moment takes the place of the sample standard deviation.&lt;/li&gt;&#xA;&lt;li&gt;The $\gamma$-Winsorized standard deviation divided by the $\gamma$-trimmed mean. You probably want to divide this by $1 - 2\gamma$ to get a consistent scaling across different values of $\gamma$. I would try $\gamma = 0.2$.&lt;/li&gt;&#xA;&lt;li&gt;The square root of the number of samples times the McKean-Schrader estimate of the standard error of the median, divided by the sample median.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="795" LastActivityDate="2010-12-30T04:50:05.367" CommentCount="5" />
  <row Id="5826" PostTypeId="1" AcceptedAnswerId="5830" CreationDate="2010-12-30T05:53:33.917" Score="2" ViewCount="256" Body="&lt;p&gt;I was wondering if anybody had experience in how to set the knot points when using cubic regression splines.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some background: I have a response and predictor variable, and I want to determine the trend relationship between the two. To see what it looks like without making too many assumptions, I've fit a smoothing spline curve using the &lt;code&gt;gam&lt;/code&gt; function in R. The trend is obviously not linear, but otherwise well-behaved: smooth, and not too wiggly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd now like to model this trend using a simple, cubic regression spline (there are various practical issues with using the &lt;code&gt;gam&lt;/code&gt; fit, or I'd just use that). Of course, using a regression spline requires the knots to be specified in advance. It's not too hard to do that with linear splines: I'd insert a knot where the slope of the smooth fit changes substantially, eg around local minima/maxima. However, cubic splines appear to be a more complicated story. Any guidance on where I should put the knots would be much appreciated.&lt;/p&gt;&#xA;" OwnerUserId="1569" LastActivityDate="2010-12-30T10:11:57.400" Title="Knot selection for cubic regression splines" Tags="&lt;regression&gt;&lt;splines&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="5827" PostTypeId="1" CreationDate="2010-12-30T06:07:23.883" Score="5" ViewCount="245" Body="&lt;p&gt;We work on cervical cancer detection and we started accumulating data in 2006 by performing HPV testing on all our samples. We have accumulated a data set with almost 400,000 women. Data enters the database on a daily basis. Most women come for a yearly smear so most women are more than one time in the database (2-13x). Most women are HPV negative (appr 85%) but some become HPV positive during this observation period.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would be very interested in looking at the rate of infection (incidence) of HPV16. I suppose I will have to look at the women with at least 2 samples and restricting to women going from a negative sample to a positive sample. I have a few questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;How do I start? Do I use a survival analysis and how do I do this specifically in Stata or Prism5?&lt;/li&gt;&#xA;&lt;li&gt;Do I censor some women and how (these women who are HPV16+ on the first sample, we don't know when they acquired this)?&lt;/li&gt;&#xA;&lt;li&gt;What do I do with all the HPV negative women, do they stay in the analysis?&lt;/li&gt;&#xA;&lt;li&gt;How would I compare the incidence rate according to age? I would specifically like to know if young women (&amp;lt;30 years) have a different pattern when compared to women of &gt;30 years.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="2571" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-12-30T09:49:09.903" LastActivityDate="2010-12-30T16:21:13.550" Title="How do I calculate and compare incidence of infection between age-groups" Tags="&lt;epidemiology&gt;&lt;stata&gt;&lt;disease&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="5828" PostTypeId="2" ParentId="4356" CreationDate="2010-12-30T06:31:03.853" Score="1" ViewCount="" Body="&lt;p&gt;Multivariate splits as defined in the CART book aren't implemented in &lt;code&gt;rpart&lt;/code&gt;. The CART software package from Salford Systems has this feature, but AFAIK it uses a proprietary algorithm licensed from Breiman, Friedman et al.&lt;/p&gt;&#xA;" OwnerUserId="1569" LastActivityDate="2010-12-30T06:31:03.853" />
  <row Id="5829" PostTypeId="2" ParentId="5776" CreationDate="2010-12-30T06:36:57.283" Score="0" ViewCount="" Body="&lt;p&gt;This a job for Bayes Theorem, not for null-hypothesis testing. You've given us some information, so we should now simply determine how much more or less consistent that information is with the hypothesis than with its negation &amp;amp; adjust our priors accordingly. Based on what you've told us I conclude that the likelihood &quot;Bill is skilled&quot; is low--somewhere in the neighborhood of 10% (that is, I'll &lt;em&gt;bet&lt;/em&gt; he is &quot;lucky&quot; unless you offer me odds better than 9:1 against &quot;skilled&quot;). Here's why:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Neither the failure to make hypotheses in advance of the 2 games, the lack of information about how many games have been played or will be played after the 2 in question, nor the possibility that someone is cheating, etc. really matter. Having been supplied no information bearing on the situation, we should assume an 8-player poker game drawn at random from the universe of such games being played. That deals w/ w/ all the &quot;what ifs&quot; &amp;amp; &quot;could bes&quot; that Whuber draws our attention to: b/c the proportion of games that are fixed, or involve 1 avg player competing w/ 7 imbeciles, or are played w/ unshuffled decks etc., is tiny in comparison to the proportion of games that are fair &amp;amp; involve &quot;run of the mill players,&quot; you will just be making your life complicated if you assume this game involves anything other than a &quot;normal game&quot; (if you have a different sense of how the universe of 8-person games is populated, then modify this part of the analysis accordingly; I'm just trying to demonstrate how to think about this problem!).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Based on billions of games &amp;amp; meticulous record-keeping, I put the likelihood at about 0.10 that any randomly selected player in a normal game of poker is a &quot;skilled&quot; (again, if you have a different sense of what the talent distribution looks like, substitute your own estimate here).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;In a normal game of poker, the likelihood that a skilled poker player will win any 2 consecutive hands against 7 randomly selected players is only a scintilla higher than the likelihood that an unskilled one will. Poker is a game of skill, yes, but variance is super high (if you think otherwise, fine, but in that case you definitely are not an experienced poker player). If you had said &quot;Bill made 2 consecutive 40-foot jump shots,&quot; or &quot;won 2 consecutive olympic marathons,&quot; in contrast, then my priors about how much better he is than the average baseketball player or the average marathon runner would shift much much more dramatically-- those are lower-variance indicators of those types of skill.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Because the likelihood ratio for the hypothesis and the negation of the hypothesis is thus very close to 1, you won't do much better here than going with your priors. Again, mine is that there is 1/10 chance that Bill is skilled.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="2347" LastActivityDate="2010-12-30T06:36:57.283" />
  <row Id="5830" PostTypeId="2" ParentId="5826" CreationDate="2010-12-30T09:08:39.313" Score="3" ViewCount="" Body="&lt;p&gt;This is a tricky problem and most people just select the knots by trial and error. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One approach which is growing in popularity is to use penalized regression splines instead. Then knot selection has little effect provided you have lots of knots. The coefficients are constrained to avoid any coefficient being too large. It turns out that this is equivalent to a mixed effects model where the spline coefficients are random. Then the whole problem can be solved using REML without worrying about knot selection or a smoothing parameter. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since you use R, you can fit such a model using the &lt;code&gt;spm()&lt;/code&gt; function in the &lt;a href=&quot;http://cran.r-project.org/web/packages/SemiPar/index.html&quot; rel=&quot;nofollow&quot;&gt;SemiPar package&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2010-12-30T09:08:39.313" CommentCount="2" />
  <row Id="5831" PostTypeId="1" AcceptedAnswerId="5883" CreationDate="2010-12-30T10:11:11.710" Score="12" ViewCount="259" Body="&lt;p&gt;I think it is fair to say statistics is an applied science so when averages and standard deviations are calculated it is because someone is looking to make some decisions based on those numbers. Part of being a good statistician then I would hope is being able to &quot;sense&quot; when the sample data can be trusted and when some statistical test is completely misrepresenting the true data we're interested in. Being a programmer that is interested in analysis of big data sets I'm relearning some statistics and probability theory but I can't shake this nagging feeling that all the books I've looked at are kind of like politicians that get up on stage and say a whole bunch of things and then append the following disclaimer at the end of their speech &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Now, I'm not saying that this is good or bad but the numbers say it's good so you should vote for me anyway.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Maybe you get that but maybe you don't so here's a question. Where do I go to find war stories by statisticians where some decisions were based on some statistical information that later turned out to be completely wrong?&lt;/p&gt;&#xA;" OwnerUserId="2293" LastActivityDate="2011-01-25T06:09:58.707" CommunityOwnedDate="2010-12-31T10:46:17.657" Title="Statistics, war stories, data intuition" Tags="&lt;probability&gt;&lt;summary-statistics&gt;&lt;intuition&gt;&lt;learning&gt;&lt;history&gt;" AnswerCount="2" CommentCount="0" FavoriteCount="4" />
  <row Id="5832" PostTypeId="2" ParentId="5826" CreationDate="2010-12-30T10:11:57.400" Score="2" ViewCount="" Body="&lt;p&gt;It depends what you mean by &quot;not &lt;em&gt;too&lt;/em&gt; wiggly&quot;, but you might like to take a look at fractional polynomials for a simpler approach to fitting smooth curves that are not linear but not 'wiggly'. See &lt;a href=&quot;http://www.jstor.org/stable/2986270&quot; rel=&quot;nofollow&quot;&gt;Royston &amp;amp; Altman 1994&lt;/a&gt; and the &lt;a href=&quot;http://cran.r-project.org/web/packages/mfp/index.html&quot; rel=&quot;nofollow&quot;&gt;mfp package&lt;/a&gt; in R or the &lt;a href=&quot;http://www.stata.com/help.cgi?fracpoly&quot; rel=&quot;nofollow&quot;&gt;fracpoly command&lt;/a&gt; in Stata.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-12-30T10:11:57.400" />
  <row Id="5833" PostTypeId="2" ParentId="3589" CreationDate="2010-12-30T10:58:21.897" Score="0" ViewCount="" Body="&lt;p&gt;Well a lot depends on the assumptions you make.  If you assume that the data is stationary then more data for series one will give you abetter estimate of its volatility.  This estimate can be used to improve the correlation estimate.&#xA;So the follwoing statment is incorrect:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;The history of Y's price before X went public is useless for assessing their subsequent correlation&quot;&lt;/p&gt;&#xA;" OwnerUserId="2574" LastActivityDate="2010-12-30T10:58:21.897" />
  <row Id="5834" PostTypeId="1" CreationDate="2010-12-30T10:58:50.857" Score="2" ViewCount="192" Body="&lt;p&gt;In a Markov chain, a state $j$ is transient if $f_{jj}&amp;lt;1$ ($f_{jj}$ is probability of ever visiting state $j$ starting from state $j$ ).&#xA;Suppose, I have an irreducible transient DTMC (means all states are transient). Now, I want to prove that for any $i,j$ in $S$ ($S$ is DTMC state space), $f_{ij}$ (i.e probability of ever reaching state $j$ starting from state $i$) is less than 1. It is clear that $f_{ii}&amp;lt;1$ and $f_{jj}&amp;lt;1$. But, how to prove that $f_{ij}&amp;lt;1$ for any $i,j$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&#xA;Prasenjit&lt;/p&gt;&#xA;" OwnerUserId="2516" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-12-30T17:17:09.033" LastActivityDate="2010-12-31T16:46:13.443" Title="First passage time distribution in a irreducible transient discrete-time Markov chain (DTMC)" Tags="&lt;markov-process&gt;" AnswerCount="1" CommentCount="4" />
  <row Id="5835" PostTypeId="2" ParentId="5827" CreationDate="2010-12-30T12:58:23.283" Score="0" ViewCount="" Body="&lt;p&gt;I don't use Stata or Prism 5, so I can't help with that part, but a survival analysis does sound appropriate, at least from what you've said.  The most commonly used method is Cox proportional hazard, which is certainly available in Stata.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instructions on how to code the data are program specific, so I will leave it to a Stata expert to help with that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Survival analysis can deal with various kinds of censoring.  In particular, you have two types to deal with: 1) Some women were positive when you first saw them, and 2) Some women were negative when you last saw them.  But, again, how to do this will be dependent on the program you use.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can include age as a time varying covariate.&lt;/p&gt;&#xA;" OwnerUserId="686" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2010-12-30T16:21:13.550" LastActivityDate="2010-12-30T16:21:13.550" CommentCount="3" />
  <row Id="5836" PostTypeId="1" AcceptedAnswerId="5838" CreationDate="2010-12-30T15:06:34.093" Score="6" ViewCount="314" Body="&lt;p&gt;This is more of a &quot;how to use R&quot; question than an actual hardcore statistics question, but I think the concentration of R masters here makes this a good forum for it. I'm refreshing a time series graphing package that currently uses gnuplot. The first step is getting somewhere close to the current graphs, and then I hope to be able to add more statistical analysis from R later.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently, the gnuplot-generated graphs look like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/EkuiN.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The corresponding graph from R looks like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/mqWEr.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm reasonably happy with this. What's missing is the legend stuff from below the gnuplot graph. I can add a legend saying &quot;In&quot; and &quot;Out&quot;, but how to best present the average, maximal and minimal values? I could add horizontal lines to the graph and print the values inside the graph area, but the placement might interfere with the actual graph.&lt;/p&gt;&#xA;" OwnerUserId="2443" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-12-30T16:27:12.743" LastActivityDate="2011-01-02T22:56:50.777" Title="Annotating graphs in R" Tags="&lt;r&gt;&lt;time-series&gt;&lt;data-visualization&gt;&lt;ggplot2&gt;" AnswerCount="4" CommentCount="3" FavoriteCount="4" />
  <row Id="5837" PostTypeId="2" ParentId="5827" CreationDate="2010-12-30T15:34:32.293" Score="2" ViewCount="" Body="&lt;p&gt;Survival analysis is a good idea, but:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Do not discard anybody - even those with only one data point. The fact that they did or did not have HPV at that age is informative.&lt;/li&gt;&#xA;&lt;li&gt;Your data is interval censored: left-censored for those who are positive at first measurement, right-censored for those who stay negative, and censored between the two measurement times for those who change status.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I would set up age 0 (or some other fixed value before the earliest measurement) as the starting point for time, and then estimate the hazard of the infection non-parametrically - Stata probably has a routine for that, but if not R certainly does. The result would be the age-dependent instantenous probability of infection - pretty much what you are looking for.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to estimate how covariates affect this hazard, then interval-censored regression is needed - standard Cox regression does not handle this situation.&lt;/p&gt;&#xA;" OwnerUserId="279" LastActivityDate="2010-12-30T15:34:32.293" CommentCount="1" />
  <row Id="5838" PostTypeId="2" ParentId="5836" CreationDate="2010-12-30T16:23:03.247" Score="5" ViewCount="" Body="&lt;p&gt;A quick and dirty way to paste some text and numerical results along the labels of your legend is to simply rename the factor levels. For instance,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df &amp;lt;- data.frame(x=rnorm(100), y=rnorm(100), f=gl(2,50))&#xA;df$f2 &amp;lt;- df$f&#xA;levels(df$f2) &amp;lt;- paste(levels(df$f), tapply(df$y, df$f, mean), sep=&quot;: &quot;)&#xA;p &amp;lt;- ggplot(data=df) + geom_point(aes(x=x, y=y, color=f2))&#xA;p + opts(legend.position = 'bottom', legend.title=NULL)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can add whatever you want into the new labels, such as mean, min, max, etc. (e.g., create a custom function, inspired from &lt;code&gt;summary()&lt;/code&gt; that returns the values you want, and append them to &lt;code&gt;c(&quot;In&quot;,&quot;Out&quot;)&lt;/code&gt;).&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2010-12-30T16:23:03.247" CommentCount="1" />
  <row Id="5839" PostTypeId="2" ParentId="5836" CreationDate="2010-12-30T16:59:30.710" Score="5" ViewCount="" Body="&lt;p&gt;I'm sure that there's a more elegant way to do this but you can try this :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Data &amp;lt;- data.frame(serie1 = cumsum(rnorm(100)), serie2 = rnorm(100), temps = as.Date(1:100, origin = &quot;2000-01-01&quot;))&#xA;&#xA; label &amp;lt;- c(&quot;In traffic [max 2.0G bps] [Avg 1.1G bps] [95% 1.8G bps] [Min 569.4M bps]&quot;, &quot;Out traffic [max 2.0G bps] [Avg 672M bps] [95% 2.1G bps] [Min 154.3M bps]&quot;)&#xA;&#xA;p &amp;lt;- ggplot(melt(Data, id = &quot;temps&quot;), aes(temps, value, group = variable, colour = variable))&#xA;p &amp;lt;- p + geom_line() +xlab(&quot;&quot;) + opts(legend.position = &quot;bottom&quot;, legend.title = NULL)&#xA;p &amp;lt;- p + scale_colour_manual(values = c(&quot;red&quot;, &quot;blue&quot;), breaks = c(&quot;serie1&quot;, &quot;serie2&quot;), labels = label)&#xA;&#xA;print(p)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2028" LastActivityDate="2010-12-30T16:59:30.710" CommentCount="2" />
  <row Id="5840" PostTypeId="2" ParentId="5836" CreationDate="2010-12-30T17:10:42.580" Score="5" ViewCount="" Body="&lt;p&gt;Sorry this is an answer -- I guess I haven't earned the &quot;comment&quot; button yet...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All I was going to suggest is that &quot;how to&quot; questions like this one might be a better fit for StackOverflow. There are R pros there who answer these sorts of questions before you can hit 'send'! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't know if there's an official FAQ or guidance on SO vs. CV, but since Hadley himself answers a lot of the ggplot2 questions on SO, that's where I go first.&lt;/p&gt;&#xA;" OwnerUserId="2306" LastActivityDate="2010-12-30T17:10:42.580" CommentCount="1" />
  <row Id="5841" PostTypeId="2" ParentId="5327" CreationDate="2010-12-30T17:17:29.253" Score="0" ViewCount="" Body="&lt;p&gt;@Firefeather: Does your data contain (and can only really ever contain) only positive values? If so, model it using a generalized linear model with gamma error and log link. If it contains zeros then you could consider a two stage (logistic regression for probability of zero and gamma regression for the positive values). This latter scenario can also be modeled as a single regression using a zero inflated gamma. Some great explanations of this were given on a SAS list a few years ago. Start here if interested and search for follow-ups.&#xA;&lt;a href=&quot;http://listserv.uga.edu/cgi-bin/wa?A2=ind0805A&amp;amp;L=sas-l&amp;amp;P=R20779&quot; rel=&quot;nofollow&quot;&gt;link text&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Might help point you in another direction if the truncated regression turns out implausible.&lt;/p&gt;&#xA;" OwnerUserId="2040" LastActivityDate="2010-12-30T17:17:29.253" />
  <row Id="5842" PostTypeId="1" AcceptedAnswerId="5846" CreationDate="2010-12-30T17:59:52.847" Score="6" ViewCount="151" Body="&lt;p&gt;I have a load of point data in 3D, and I wish to cluster them like so:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Each cluster contains points all of which are at most distance $d$ from another point in the cluster. &lt;/li&gt;&#xA;&lt;li&gt;All points in two distinct clusters are at least distance $d$ from each other.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The distances used are Euclidean distances.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this a well known clustering algorithm? If so, what is is called? Thanks.&lt;/p&gt;&#xA;" OwnerUserId="2578" LastActivityDate="2010-12-30T20:04:33.040" Title="Cluster data points by distance between clusters" Tags="&lt;clustering&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="5844" PostTypeId="1" CreationDate="2010-12-30T19:53:26.203" Score="2" ViewCount="161" Body="&lt;p&gt;Working on probabilistic outputs of kernel methods I found the formulation of the SVM as a Penalized Method using the Binomial Deviance (described for example in &quot;&lt;a href=&quot;http://www-stat.stanford.edu/~tibs/ElemStatLearn/&quot; rel=&quot;nofollow&quot;&gt;The Elements of Statistical Learning Theory&lt;/a&gt;, 2nd edition&quot; by Hastie et al., pp 426--430). The model is called &quot;Kernel Logistic Regression (KLR)&quot;. References on &quot;The well studied KLR&quot; lead to literature on smoothing splines, but KLR is never mentioned there.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the connection between KLR and Smoothing Splines?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you in advance for your answer!&lt;/p&gt;&#xA;" OwnerUserId="2549" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-30T20:40:41.867" LastActivityDate="2010-12-31T00:02:24.833" Title="What is the connection between Kernel Logistic Regression and Smoothing Splines?" Tags="&lt;machine-learning&gt;&lt;logistic-regression&gt;&lt;smoothing&gt;&lt;kernel&gt;&lt;svm&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="5845" PostTypeId="1" AcceptedAnswerId="5849" CreationDate="2010-12-30T19:53:29.647" Score="3" ViewCount="152" Body="&lt;p&gt;Can the group recommend a good introduction text/resource to applied resampling techniques? Specifically, I am interested in alternatives to classical parametric tests (e.g. t tests, ANOVA, ANCOVA) for comparing groups when assumptions such as normality are clearly violated. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;An example problem type I would like to educate myself as to a better way to solve may involve something such as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I)&lt;br&gt;&#xA;2 Groups: Treatment and Control  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Dependent Var: Change in account balance dollars after intervention&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Covariate: Pre intervention account balance dollars.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Issue with applying ANCOVA: Many subjects will not have any change (many zeros). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;II)&lt;br&gt;&#xA;2 Groups: Treatment and Control&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Dependent Var: new accounts added&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Covariate: Pre intervention number of accounts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;*Many subjects will not have any added account (many zeros). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can I use a bootstrap? A permutation test? This is the type of analysis I would like to apply nonparametric resampling methods to.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;brian&lt;/p&gt;&#xA;" OwnerUserId="2040" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-30T20:39:02.897" LastActivityDate="2010-12-30T20:39:02.897" Title="Good text for resampling?" Tags="&lt;books&gt;&lt;bootstrap&gt;&lt;resampling&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="5846" PostTypeId="2" ParentId="5842" CreationDate="2010-12-30T20:04:33.040" Score="4" ViewCount="" Body="&lt;p&gt;Create a graph in which the points are nodes and two points are connected with an edge if and only if they lie within distance $d$ of each other.  Stated in these terms, your criteria become&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Every node in a cluster of two or more nodes is connected to at least one other node in that cluster.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;No two points in any disjoint clusters can be connected to each other.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In short, you want to compute the &lt;strong&gt;&lt;em&gt;connected components&lt;/em&gt;&lt;/strong&gt; of this graph.  Linear-time algorithms are well known and simple to execute, as described in the &lt;a href=&quot;http://en.wikipedia.org/wiki/Connected_component_%28graph_theory%29&quot; rel=&quot;nofollow&quot;&gt;Wikipedia article&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To do this efficiently for a lot of points in 3D, you want to limit the amount of distance calculation you perform.  Data structures such as &lt;a href=&quot;http://en.wikipedia.org/wiki/Octree&quot; rel=&quot;nofollow&quot;&gt;octrees&lt;/a&gt;, or even simpler ones (such as exploiting a &lt;a href=&quot;http://en.wikipedia.org/wiki/Lexicographical_order&quot; rel=&quot;nofollow&quot;&gt;lexicographic sorting&lt;/a&gt; of the points) work well.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-12-30T20:04:33.040" />
  <row Id="5847" PostTypeId="1" AcceptedAnswerId="5864" CreationDate="2010-12-30T20:08:04.423" Score="3" ViewCount="277" Body="&lt;p&gt;So, working in web analytics, it's common to use a chi-square test for A/B tests to test statistical significance. I often use a calculator like this: &lt;a href=&quot;http://www.usereffect.com/split-test-calculator&quot; rel=&quot;nofollow&quot;&gt;http://www.usereffect.com/split-test-calculator&lt;/a&gt; , so if each variation gets 1000 impressions, and A has 100 conversions and B has 200 conversions, I know that, with a 99.9% confidence interval (assuming randomized samples) that B is the winning variation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But occasionally, I have values that create percentages over 100%, and seem to throw these calculations off.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, if I'm testing two donation pages, and I want to use the $ values for conversions, how would I go about that? My goal is to make a statistically informed decision about which page is performing better, in terms of its ability to raise more money.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I have 1000 impressions of each variation, and one variation raises &lt;code&gt;$2000&lt;/code&gt; off of 200 contributions, and the othe other raises &lt;code&gt;$2500&lt;/code&gt; off of 100 contributions, what calculation can I use to know if the difference is meaningful? &lt;/p&gt;&#xA;" OwnerUserId="142" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2010-12-30T20:42:34.797" LastActivityDate="2010-12-31T12:10:01.110" Title="Statistical significance (chi squared) for percentages over 100%?" Tags="&lt;chi-squared&gt;&lt;testing&gt;" AnswerCount="1" CommentCount="5" />
  <row Id="5848" PostTypeId="2" ParentId="5845" CreationDate="2010-12-30T20:13:12.627" Score="2" ViewCount="" Body="&lt;p&gt;Phillip Good, &lt;em&gt;Permutation, Parametric, and Bootstrap Tests of Hypotheses&lt;/em&gt; (3rd Edition).  Springer, 2005.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This book is mathematically easy, accessible, and covers a wide range of applications.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-12-30T20:13:12.627" CommentCount="4" />
  <row Id="5849" PostTypeId="2" ParentId="5845" CreationDate="2010-12-30T20:18:40.273" Score="3" ViewCount="" Body="&lt;p&gt;As for a good reference, I would recommend Philip Good, &lt;em&gt;&lt;a href=&quot;http://www.springer.com/birkhauser/applied+probability+and+statistics/book/978-0-8176-4386-7&quot; rel=&quot;nofollow&quot;&gt;Resampling Methods: A Practical Guide to Data Analysis&lt;/a&gt;&lt;/em&gt; (Birkhäuser Boston, 2005, 3rd ed.) for an applied companion textbook. And here is &lt;a href=&quot;http://www-stat.wharton.upenn.edu/~stine/mich/bibliography.pdf&quot; rel=&quot;nofollow&quot;&gt;An Annotated Bibliography for Bootstrap Resampling&lt;/a&gt;. &lt;a href=&quot;http://www.creative-wisdom.com/pub/mirror/resampling_methods.pdf&quot; rel=&quot;nofollow&quot;&gt;Resampling methods: Concepts, Applications, and Justification&lt;/a&gt; also provides a good start.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are many R packages that facilitate the use of resampling techniques:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://cran.r-project.org/web/packages/boot/index.html&quot; rel=&quot;nofollow&quot;&gt;boot&lt;/a&gt;, for bootstraping -- but see also P. Burns, &lt;a href=&quot;http://lib.stat.cmu.edu/S/Spoetry/Tutor/bootstrap_resampling.html&quot; rel=&quot;nofollow&quot;&gt;The Statistical Bootstrap and Other Resampling Methods&lt;/a&gt;, for illustrations&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://cran.r-project.org/web/packages/coin/index.html&quot; rel=&quot;nofollow&quot;&gt;coin&lt;/a&gt;, for permutation tests (bit see the accompagnying &lt;a href=&quot;http://cran.r-project.org/web/packages/coin/vignettes/coin.pdf&quot; rel=&quot;nofollow&quot;&gt;vignette&lt;/a&gt; which includes extensive help) &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;(There are many other packages...)&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-12-30T20:24:55.710" LastActivityDate="2010-12-30T20:24:55.710" CommentCount="7" />
  <row Id="5850" PostTypeId="1" AcceptedAnswerId="5851" CreationDate="2010-12-30T21:57:40.627" Score="4" ViewCount="116" Body="&lt;p&gt;Given a random variable $X_1$ drawn from a distribution with cdf $F$, and random variables $X_2, \cdots,X_n$ drawn from another distribution with cdf $G$, what is the formula for the probability that $X_1$ is the $k$-th out of the $n$ variables, when they are sorted in order.  If it helps, assume that $F$ and $G$ are both defined on [0,1], with positive density everywhere.&lt;/p&gt;&#xA;" OwnerUserId="2582" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-12-31T10:33:41.713" LastActivityDate="2010-12-31T10:33:41.713" Title="Order statistics: probability random variable is k-th out of n when ordered" Tags="&lt;order-statistics&gt;" AnswerCount="1" />
  <row Id="5851" PostTypeId="2" ParentId="5850" CreationDate="2010-12-30T22:57:51.933" Score="5" ViewCount="" Body="&lt;p&gt;By computing the probability that $k-1$ of the $G$-distributed $X$'s are less than or equal to $X_1$, conditional on the value of $X_1 = t$, and integrating over all $t$ we obtain&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\int_{-\infty}^{\infty}{\binom{n-1}{k-1} G(t)^{k-1}(1-G(t))^{n-k}dF(t)}.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This of course assumes independence of all the $X_i$.  Given no assumptions about any relationship about $F$ and $G$, this expression should not simplify in general.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-12-30T22:57:51.933" />
  <row Id="5852" PostTypeId="2" ParentId="5844" CreationDate="2010-12-31T00:02:24.833" Score="1" ViewCount="" Body="&lt;p&gt;There is a book on &quot;Nonparametric Regression and Generalized Linear Models: A Roughness Penalty Approach&quot; by &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0412300400&quot; rel=&quot;nofollow&quot;&gt;Green and Silverman&lt;/a&gt;, that is probably a good start, but my copy is in my office, so I can't get it until the new year.  Essentially IIRC, the link between kernel methods and smoothing splines is that a regularisation term is used that penalises particular properties of the function implemented by the model, commonly (as the name suggests) the roughness (as measured by the second derivative or curvature).  For kernel methods, the regularisation operator depends on the choice of kernel, but the regularisation operator is not dependent on the particular sample of data, which is the connection with splines rather than other non-parametric models.  Smoothing splines can be used with more or less any loss function (including the logistic loss), just as kernel methods can use more or less any (convex) loss.  Often the squared error or hinge losses are not the best ones, but they get most of the attention, and you can sometimes incorporate useful expert knowledge about the task via the loss just as in GLMs.  Hopefully I can give a better answer once I have a chance to refer to my books!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S. there is a &lt;a href=&quot;http://dx.doi.org/10.1109/5.58326&quot; rel=&quot;nofollow&quot;&gt;paper&lt;/a&gt; on regularisation networks by Poggio and Girosi that may well be quite relevant as well.&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2010-12-31T00:02:24.833" />
  <row Id="5853" PostTypeId="1" AcceptedAnswerId="5876" CreationDate="2010-12-31T05:49:57.543" Score="2" ViewCount="311" Body="&lt;p&gt;This relates to my programming question here.&#xA;&lt;a href=&quot;http://stackoverflow.com/questions/4560554/bell-curve-gaussian-algorithm-python-and-or-c&quot;&gt;http://stackoverflow.com/questions/4560554/bell-curve-gaussian-algorithm-python-and-or-c&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On Answers.com, I found this simple example: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Find the arithmetic mean (average)&#xA;=&gt; Sum of all values in the set, divided by the numbers of elements in the set&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Find the sum of the squares of all values in the set&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Divide output of (2) over the numbers of elements in the set&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Substract the square of mean (1) from the output of (3)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Take the square root of the outcome of (4)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Set A={1,3,4,5,7}&#xA;1. (1+3+4+5+7)/5 = 4&#xA;2. (1*1+3*3+4*4+5*5+7*7)=1+9+16+25+49 = 100&#xA;3. 100 / 5 = 20&#xA;4. 20 - 4*4=20-16 = 4&#xA;5. SQRT(4) = 2&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Read more: &lt;a href=&quot;http://wiki.answers.com/Q/How_to_find_the_variance_of_a_set_of_numbers#ixzz19fIdujyG&quot; rel=&quot;nofollow&quot;&gt;http://wiki.answers.com/Q/How_to_find_the_variance_of_a_set_of_numbers#ixzz19fIdujyG&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now given all that, how can I fit the above data to a bell curve (such as a credit score) ranging from 200 to 800.  Obviously the number 5 in the above set would be 500.  But then what is the formula for determining what 3 should be on the same scale.  Even though the original set Set A={1,3,4,5,7} is not a bell-curve, I want to force it into a bell-curve.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Imagine these are scores of 5 people.  Next month the scores might change as follows: &#xA;Set A2={1,2,4,5,9} (one guys loses a point, and the top guy gains two more points - the rich get richer and the poor get poorer). &#xA;Then perhaps a new guy comes into the set: Set A3={1,2,4,5,8,9}.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks,&#xA;Neal &lt;/p&gt;&#xA;" OwnerUserId="2585" LastEditorUserId="2585" LastEditorDisplayName="" LastEditDate="2010-12-31T15:03:31.523" LastActivityDate="2011-01-09T16:55:51.487" Title="Forcing a set of numbers to a gaussian bell-curve" Tags="&lt;algorithms&gt;&lt;normal-distribution&gt;" AnswerCount="2" CommentCount="4" />
  <row Id="5854" PostTypeId="1" CreationDate="2010-12-31T06:38:50.327" Score="8" ViewCount="385" Body="&lt;p&gt;I am working on creating a website, which displays the census data for a user selected Polygons &amp;amp; would like to graphically show the distribution of various parameters (one graph per parameter).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data usually has the following properties:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The sample size tend to be large (say around 10,000 data points)&lt;/li&gt;&#xA;&lt;li&gt;The range in values tends to be quire large (for example, the minimum population can be less than 100 &amp;amp; the maximum can be something like 500,000)&lt;/li&gt;&#xA;&lt;li&gt;q1 usually is close to the minimum (say 200) while q2 &amp;amp; q3 will be within 10,000&lt;/li&gt;&#xA;&lt;li&gt;It doesn't look anything like a normal distribution&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I am not a statistician and hence my description might not be exactly clear.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to show this distribution on a graph, which will be seen by citizens (the layman, if you like).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would have best liked to use a histogram, but it is not possible due to the large range of values, due to which making bins is not really easy &amp;amp; straight forward.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From what little I know about statistics,  a box plot is what is often used to show this kind of data, but I feel that for a layperson, deciphering the Box plot is not easy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are my options to show this data in an easy to understand manner?&lt;/p&gt;&#xA;" OwnerUserId="2586" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-12-31T09:59:05.407" LastActivityDate="2011-01-04T12:37:08.117" Title="What are some alternatives to a boxplot?" Tags="&lt;distributions&gt;&lt;data-visualization&gt;&lt;boxplot&gt;" AnswerCount="7" CommentCount="3" FavoriteCount="4" />
  <row Id="5856" PostTypeId="2" ParentId="5854" CreationDate="2010-12-31T09:18:51.613" Score="4" ViewCount="" Body="&lt;p&gt;Here is a &lt;a href=&quot;http://www.mathworks.com/matlabcentral/fileexchange/23661-distribution-plot&quot; rel=&quot;nofollow&quot;&gt;matlab function&lt;/a&gt; for plotting multiple histograms side-by-side in 2D as an alternative to box-plot. See the picture on the top. And here is &lt;a href=&quot;http://www.mathworks.com/matlabcentral/fileexchange/26508-notboxplot-alternative-to-box-plots&quot; rel=&quot;nofollow&quot;&gt;another one&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;a href=&quot;http://pubs.amstat.org/doi/abs/10.1198/000313008X370843&quot; rel=&quot;nofollow&quot;&gt;density strip&lt;/a&gt; is another alternative to box-plot. It is a shaded monochrome strip whose darkness at a point is proportional to the probability density of the quantity at that point. This is an &lt;a href=&quot;http://rgm2.lab.nig.ac.jp/RGM_results/denstrip%3adenstrip/denstrip_006_big.png&quot; rel=&quot;nofollow&quot;&gt;R implementation&lt;/a&gt; of the density strip &lt;/p&gt;&#xA;" OwnerUserId="339" LastEditorUserId="339" LastEditorDisplayName="" LastEditDate="2010-12-31T09:31:58.393" LastActivityDate="2010-12-31T09:31:58.393" CommentCount="2" />
  <row Id="5857" PostTypeId="2" ParentId="5854" CreationDate="2010-12-31T09:37:33.170" Score="3" ViewCount="" Body="&lt;p&gt;How about using quantiles? It is not necessary to present a graph then, only a table. For village census I think the users will be most interested how many there are villages of certain size, so giving for example deciles will tell them them information such as $x\%$ of all the villages are smaller than the certain number. For deciles $x=0,10,20,...,100$. You can graph this table with the percents on a x-axis and the deciles on the y-axis.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2010-12-31T09:37:33.170" CommentCount="3" />
  <row Id="5858" PostTypeId="2" ParentId="4949" CreationDate="2010-12-31T09:41:57.923" Score="4" ViewCount="" Body="&lt;p&gt;There's no closed form, but you could do it numerically.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a concrete example, consider two Gaussians with following parameters&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\mu_1=\left(\begin{matrix}&#xD;&#xA;-1\\\\&#xD;&#xA;-1&#xD;&#xA;\end{matrix}\right),&#xD;&#xA;\mu_2=\left(\begin{matrix}&#xD;&#xA;1\\\\&#xD;&#xA;1&#xD;&#xA;\end{matrix}\right)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\Sigma_1=\left(\begin{matrix}&#xD;&#xA;2&amp;amp;1/2\\\\&#xD;&#xA;1/2&amp;amp;2&#xD;&#xA;\end{matrix}\right),\ \Sigma_2=\left(\begin{matrix}&#xD;&#xA;1&amp;amp;0\\\\&#xD;&#xA;0&amp;amp;1&#xD;&#xA;\end{matrix}\right)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bayes optimal classifier boundary will correspond to the point where two densities are equal&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://yaroslavvb.com/upload/save/bayes-optimal1.png&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since your classifier will pick the most likely class at every point, you need to integrate over the density that is not the highest one for each point. For the problem above, it corresponds to volumes of following regions&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://yaroslavvb.com/upload/save/bayes-optimal2.png&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://yaroslavvb.com/upload/save/bayes-optimal3.png&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can integrate two pieces separately using some numerical integration package. For the problem above I get &lt;code&gt;0.253579&lt;/code&gt; using following Mathematica code&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dens1[x_, y_] = PDF[MultinormalDistribution[{-1, -1}, {{2, 1/2}, {1/2, 2}}], {x, y}];&#xA;dens2[x_, y_] = PDF[MultinormalDistribution[{1, 1}, {{1, 0}, {0, 1}}], {x, y}];&#xA;piece1 = NIntegrate[dens2[x, y] Boole[dens1[x, y] &amp;gt; dens2[x, y]], {x, -Infinity, Infinity}, {y, -Infinity, Infinity}];&#xA;piece2 = NIntegrate[dens1[x, y] Boole[dens2[x, y] &amp;gt; dens1[x, y]], {x, -Infinity, Infinity}, {y, -Infinity, Infinity}];&#xA;piece1 + piece2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="511" LastActivityDate="2010-12-31T09:41:57.923" />
  <row Id="5859" PostTypeId="1" CreationDate="2010-12-31T10:10:49.467" Score="6" ViewCount="323" Body="&lt;p&gt;I am confused about how to decide whether to treat time as continuous or discrete in survival analysis. Specifically, I want to use survival analysis to identify child- and household-level variables that have the largest discrepancy in their impact on boys' versus girls' survival (up to age 5). I have a dataset of child ages (in months) along with an indicator for whether the child is alive, the age at death (in months), and other child- and household-level variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since time is recorded in months and all children are under age 5, there are many tied survival times (often at half-year intervals: 0mos, 6mos, 12mos, etc). Based on what I have read about survival analysis, having many tied survival times makes me think I should be treating time as discrete. However, I have read several other studies where survival time is in, for example, person-years (and so surely there are tied survival times) and continuous-time methods like Cox proportional hazards are used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are the criteria I should use to decide whether to treat time as continuous or discrete? For my data and question, using some continuous-time model (Cox, Weibull, etc) makes intuitive sense to me, but the discrete nature of my data and the amount of tied survival times seem to suggest otherwise.&lt;/p&gt;&#xA;" OwnerUserId="2587" LastActivityDate="2011-01-03T16:07:14.337" Title="Survival analysis: continuous vs discrete time" Tags="&lt;survival&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="5860" PostTypeId="2" ParentId="5854" CreationDate="2010-12-31T10:12:59.793" Score="1" ViewCount="" Body="&lt;p&gt;I rather like &lt;a href=&quot;http://en.wikipedia.org/wiki/Violin_plot&quot; rel=&quot;nofollow&quot;&gt;violin plots&lt;/a&gt; myself, as this gives an idea of the shape of the distribution.  However if the large range of values is the issue, then maybe it would be best to plot the log of the data rather than the raw values, that would then make choosing the box sizes for histograms etc.  As the display is for laymen, don't mention logs and mark the axis 10, 100, 1000, 10000, 100000, 1000000 etc.&lt;/p&gt;&#xA;" OwnerUserId="887" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-12-31T11:57:20.967" LastActivityDate="2010-12-31T11:57:20.967" />
  <row Id="5861" PostTypeId="2" ParentId="5854" CreationDate="2010-12-31T10:17:27.987" Score="8" ViewCount="" Body="&lt;p&gt;A boxplot isn't that complicated. After all, you just need to compute the three &lt;a href=&quot;http://en.wikipedia.org/wiki/Quartile&quot; rel=&quot;nofollow&quot;&gt;quartiles&lt;/a&gt;, and the min and max which define the range; a subtlety arises when we want to draw the whiskers and various methods have been proposed. For instance, in a &lt;a href=&quot;http://mathworld.wolfram.com/Box-and-WhiskerPlot.html&quot; rel=&quot;nofollow&quot;&gt;Tukey boxplot&lt;/a&gt; values outside 1.5 times the inter-quartile from the first or third quartile would be considered as outliers and displayed as simple points. See also &lt;a href=&quot;http://j.mp/hS5RJW&quot; rel=&quot;nofollow&quot;&gt;Methods for Presenting Statistical Information: The Box Plot for a good overview&lt;/a&gt;, by Kristin Potter. The &lt;a href=&quot;http://cran.r-project.org/&quot; rel=&quot;nofollow&quot;&gt;R&lt;/a&gt; software implements a slightly different rule but the source code is available if you want to study it (see the &lt;code&gt;boxplot()&lt;/code&gt; and &lt;code&gt;boxplot.stats()&lt;/code&gt; functions). However, it is not very useful when the interest is in identifying outliers from a very skewed distribution (but see, &lt;a href=&quot;ftp://ftp.win.ua.ac.be/pub/preprints/04/AdjBox04.pdf&quot; rel=&quot;nofollow&quot;&gt;An adjusted boxplot for skewed distributions&lt;/a&gt;, by Hubert and Vandervieren, CSDA 2008 52(12)).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as online visualization is concerned, I would suggest taking a look at &lt;a href=&quot;http://vis.stanford.edu/protovis/&quot; rel=&quot;nofollow&quot;&gt;Protovis&lt;/a&gt; which is a plugin-free js toolbox for interactive web displays. The &lt;a href=&quot;http://vis.stanford.edu/protovis/ex/&quot; rel=&quot;nofollow&quot;&gt;examples&lt;/a&gt; page has very illustrations of what can be achieved with it, in very few lines.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2010-12-31T10:29:34.243" LastActivityDate="2010-12-31T10:29:34.243" CommentCount="5" />
  <row Id="5862" PostTypeId="2" ParentId="5854" CreationDate="2010-12-31T11:13:59.307" Score="5" ViewCount="" Body="&lt;p&gt;I'd suggest you persevere with histograms. They're much more widely understood than the alternatives. Use a log scale to cope with the large range of values. Here's an example I cooked up in a couple of minutes in Stata: &lt;img src=&quot;http://i.imgur.com/kCbit.png&quot; alt=&quot;Histogram with log scale on the value axis&quot;&gt;&lt;br&gt;&#xA;I admit that the x-axis numerical labels weren't entirely straightforward or automatic, but as you're building a website I'm sure your programming skills are up to the challenge!&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-12-31T11:13:59.307" CommentCount="3" />
  <row Id="5863" PostTypeId="2" ParentId="5854" CreationDate="2010-12-31T11:14:35.397" Score="2" ViewCount="" Body="&lt;p&gt;If you are targeting the general population (i.e. a non statistical-savvy audience) you should focus on eye-candy rather than statistical accuracy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Forget about boxplots, let alone violin plots (I personally find them very difficult to read)! If you'd ask the average street man what a quantile is, you would mostly get some wide eyed silence... &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should use barplots, bubble charts, maybe some pie charts (brrrr). Forget about error bars (although I would put SD in text somewhere where applicable).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Use colors, shapes, thick lines, 3D. You should make each chart unique and immediately easy to understand, even without having to read all the legends/axes etc.&#xA;Make a smart use of maps by coloring them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.informationisbeautiful.net/&quot; rel=&quot;nofollow&quot;&gt;Information is beautiful&lt;/a&gt; is a very good resource to get ideas. Look at this chart for instance: &lt;a href=&quot;http://www.informationisbeautiful.net/visualizations/caffeine-and-calories/&quot; rel=&quot;nofollow&quot;&gt;Caffeine and Calories&lt;/a&gt;: anyone can understand it, and it's pleasing to the eye.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And, of course, have a look at Edward Tufte's work.&lt;/p&gt;&#xA;" OwnerUserId="582" LastActivityDate="2010-12-31T11:14:35.397" CommentCount="4" />
  <row Id="5864" PostTypeId="2" ParentId="5847" CreationDate="2010-12-31T12:10:01.110" Score="1" ViewCount="" Body="&lt;p&gt;I assume you have data on the amount of each donation, not just the two totals for each variation. You don't want to use &lt;a href=&quot;http://en.wikipedia.org/wiki/Pearson%27s_chi-square_test&quot; rel=&quot;nofollow&quot;&gt;Pearson's chi-square test&lt;/a&gt;; that's only appropriate for counts or proportions, not for amounts. You want an independent (unpaired) two-sample test for equality of location. The classic such '&lt;a href=&quot;http://en.wikipedia.org/wiki/Parametric_statistics&quot; rel=&quot;nofollow&quot;&gt;parametric&lt;/a&gt;' test is &lt;a href=&quot;http://en.wikipedia.org/wiki/T-test#Independent_two-sample_t-test&quot; rel=&quot;nofollow&quot;&gt;Student's &lt;em&gt;t&lt;/em&gt;-test&lt;/a&gt;. As it's quite plausible that the variances differ as well as the means, so I'd suggest you use &lt;strong&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Welch%27s_t-test&quot; rel=&quot;nofollow&quot;&gt;Welch's &lt;em&gt;t&lt;/em&gt;-test&lt;/a&gt;&lt;/strong&gt;, which is a slight variation that allows for unequal variances. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You &lt;em&gt;might&lt;/em&gt; also consider '&lt;a href=&quot;http://en.wikipedia.org/wiki/Non-parametric_statistics&quot; rel=&quot;nofollow&quot;&gt;non-parametric&lt;/a&gt;' ('distribution-free') tests, of which the obvious one is the &lt;a href=&quot;http://en.wikipedia.org/wiki/Mann-Whitney_U_test&quot; rel=&quot;nofollow&quot;&gt;Mann–Whitney U test&lt;/a&gt;, but as your outcome of interest is clearly the difference in mean donation and your sample size is reasonably large, you're almost certainly better off sticking with parametric tests such as the above &lt;em&gt;t&lt;/em&gt;-tests that use the mean as the location parameter and so directly address the null hypothesis of equality of means of the two samples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By the way, don't forget to include the people who saw either variation of the donation page but didn't donate anything as donations of value zero.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2010-12-31T12:10:01.110" CommentCount="1" />
  <row Id="5865" PostTypeId="2" ParentId="4551" CreationDate="2010-12-31T13:17:04.073" Score="0" ViewCount="" Body="&lt;p&gt;Using Analysis of Covariance (ANCOVA) to try to &quot;control for&quot; or &quot;regress out&quot; the influence of a covariate that is known to be correlated with, or affect the influence of, other predictor variables. More discussion at this question: &lt;a href=&quot;http://stats.stackexchange.com/questions/5808/checking-factor-covariate-independence-in-ancova&quot;&gt;Checking factor/covariate independence in ANCOVA&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="364" LastActivityDate="2010-12-31T13:17:04.073" CommunityOwnedDate="2010-12-31T13:17:04.073" />
  <row Id="5866" PostTypeId="2" ParentId="5859" CreationDate="2010-12-31T14:30:55.177" Score="1" ViewCount="" Body="&lt;p&gt;I suspect if you use continuous time models you will want to use interval censoring, reflecting the fact that you don't know the exact time of failure, just an interval in which the failure ocurred.  If you fit parametric regression models with interval censoring using maximum likelihhod the tied survival times is not an issue IIRC.&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2010-12-31T14:30:55.177" />
  <row Id="5867" PostTypeId="2" ParentId="2234" CreationDate="2010-12-31T14:56:21.053" Score="3" ViewCount="" Body="&lt;p&gt;There are around 100 classification and regression models which are trainable via the caret package. Any of the classification models will be an option for you (as opposed to regression models, which require a continuous response). For example to train a random forest:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(caret)&#xA;train(response~., data, method=&quot;rf&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;See the caret model training vignette which comes with the distribution for a full list of the models available. It is split into dual-use and classification models (both of which you can use) and regression-only (which you can't). caret will automatically train the parameters for your chosen model for you.&lt;/p&gt;&#xA;" OwnerUserId="2589" LastActivityDate="2010-12-31T14:56:21.053" CommunityOwnedDate="2010-12-31T14:56:21.053" />
  <row Id="5868" PostTypeId="1" AcceptedAnswerId="5869" CreationDate="2010-12-31T15:07:55.817" Score="2" ViewCount="77" Body="&lt;p&gt;With real-valued $X_1, X_2, \ldots$, define&lt;br&gt;&#xA;$Max_n := \max(X_1,\ldots,X_n)$ record value or high-water mark&lt;br&gt;&#xA;$NextMax_n :=$ the next greater high water, $Max_{n+m} &amp;gt; Max_n$&lt;br&gt;&#xA;$Up_n := NextMax_n - Max_n$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone suggest an introduction to estimating $NextMax$ and $Up$, for&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;$X$ i.i.d. from a known distribution,  &lt;/li&gt;&#xA;&lt;li&gt;nonparametric, estimate-as-you-go?  &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Either a book chapter or an online stats course would do nicely.&lt;/p&gt;&#xA;" OwnerUserId="557" LastEditorUserId="557" LastEditorDisplayName="" LastEditDate="2011-01-01T15:44:40.887" LastActivityDate="2011-01-01T15:44:40.887" Title="Basics of extreme values / high-water marks?" Tags="&lt;books&gt;&lt;extreme-value&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="5869" PostTypeId="2" ParentId="5868" CreationDate="2010-12-31T15:19:49.277" Score="2" ViewCount="" Body="&lt;p&gt;Stuart Coles' &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1852334592&quot; rel=&quot;nofollow&quot;&gt;book&lt;/a&gt; on extreme value statistics would probably be a good place to start.&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2010-12-31T15:19:49.277" CommentCount="2" />
  <row Id="5870" PostTypeId="1" AcceptedAnswerId="5875" CreationDate="2010-12-31T15:25:01.957" Score="3" ViewCount="141" Body="&lt;p&gt;Lets say we have three independent variables:&lt;br&gt;&#xA;$$\eqalign{&#xD;&#xA;X_{1}\sim &amp;amp;B(n,\frac{1}{2}+\beta) \cr &#xD;&#xA;X_{2}\sim &amp;amp;B(n,\frac{1}{2}) \cr&#xD;&#xA;X_{3}\sim &amp;amp;B(n,\frac{1}{2}-\beta).&#xD;&#xA;}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm looking for the distribution of &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\max(X_{1},X_{2},X_{3})-\min(X_{1},X_{2},X_{3}).$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For sufficiently large $n$ we have &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\Pr(\max(X_{1},X_{2},X_{3})=k)\approx \Pr(X_{1}=k)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and similarly &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\Pr(\min(X_{1},X_{2},X_{3})=k)\approx \Pr(X_{3}=k).$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So using those two we have:  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\eqalign{&#xD;&#xA;&amp;amp;\Pr(\max(X_{1},X_{2},X_{3})-\min(X_{1},X_{2},X_{3})=k) \cr&#xD;&#xA;&amp;amp;\approx \Pr(X_{1}-X_{3}=k) \cr&#xD;&#xA;&amp;amp;=(\frac{1}{2}+\beta)^k  (\frac{1}{2}-\beta)^{2n+k}\sum\limits_{i=k}^{n}\binom{n}{i}\binom{n}{i-k} \left(\frac{1+2\beta}{1-2\beta} \right)^{2i}&#xD;&#xA;}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;but this doesn't seem to be easy to approximate...  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was thinking about another approach. Binomial variables can be approximated by normal distributions and the sum of normally distributed variables is well known, so it would be easy to calculate. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The question&lt;/strong&gt;: Are there any inequalities that could give a better approximation? I'm aware only of the Chernoff bound but it is not very helpful here.&lt;/p&gt;&#xA;" OwnerUserId="1643" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2010-12-31T21:45:34.447" LastActivityDate="2010-12-31T21:45:34.447" Title="The range of non-identically distributed binomial variables" Tags="&lt;distributions&gt;&lt;random-variable&gt;&lt;approximation&gt;" AnswerCount="2" CommentCount="5" />
  <row Id="5871" PostTypeId="2" ParentId="5834" CreationDate="2010-12-31T15:42:51.813" Score="3" ViewCount="" Body="&lt;p&gt;Your claim is false: there exist transient Markov chains such that $f_{ij}=1$ for some (but not all) states $i$ and $j$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, assume that the state space is the union of the discrete halfline $\mathbb{Z}_+$ and of a discrete circle $\mathbb{Z}/N\mathbb{Z}$ with $N\ge3$, the halfline and the circle meeting at $0$. Write $c(k)$ for the $k$th state on the circle, counted clockwise and starting from $0$, thus $c(0)=c(N)=0$ but $c(k)$ for $1\le k\le N-1$ is not on the halfline $\mathbb{Z}_+$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The transitions are as follows. If one is at $i$ in $\mathbb{Z}_+$ with $i\ne0$, one moves to $i+1$ or to $i-1$ with probability $p$ or $1-p$, respectively. If one is at $0$, one moves to $1$ or to $c(1)$, both with positive probability. If one is at $c(k)$ with $1\le k\le N-1$, one moves to $c(k+1)$ with probability $1$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In words, while on the halfline, one performs a biased random walk and while on the circle, one moves on the circle clockwise and deterministically until one is back at $0$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For every $p&amp;gt;1/2$, this Markov chain is transient. Nevertheles, for every $k$ and $\ell$ such that $1\le k&amp;lt;\ell\le N-1$, starting from $c(k)$, one hits $c(\ell)$ with full probability hence $f_{c(k)c(\ell)}=1$.&lt;/p&gt;&#xA;" OwnerUserId="2592" LastEditorUserId="2592" LastEditorDisplayName="" LastEditDate="2010-12-31T16:46:13.443" LastActivityDate="2010-12-31T16:46:13.443" CommentCount="4" />
  <row Id="5872" PostTypeId="2" ParentId="5870" CreationDate="2010-12-31T15:58:09.770" Score="3" ViewCount="" Body="&lt;p&gt;Call $R_n=\max(X_1,X_2,X_3)-\min(X_1,X_2,X_3)$. The usual LLN shows that $R_n/n$ converges to $2\beta$ almost surely and in $L^1$. (The &lt;b&gt;distribution&lt;/b&gt; of $R_n$ for a given fixed $n$ is an altogether different story.)&lt;/p&gt;&#xA;" OwnerUserId="2592" LastActivityDate="2010-12-31T15:58:09.770" CommentCount="10" />
  <row Id="5873" PostTypeId="1" CreationDate="2010-12-31T16:02:54.980" Score="4" ViewCount="122" Body="&lt;p&gt;I would like to test a hypothesis that a database contains data that are correct in 95% or more of the samples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found this &lt;a href=&quot;http://stattrek.com/Lesson5/Proportion.aspx?Tutorial=AP&quot; rel=&quot;nofollow&quot;&gt;Hypothesis test for a proportion&lt;/a&gt; experiment design that looked appropriate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I will randomly sample one-tenth of the population and mark each datum as correct or incorrect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a summary of the design:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;State the hypotheses: (Null Hypothesis: P &gt;= 0.95, Alternative: P &amp;lt; 0.95)&lt;/li&gt;&#xA;&lt;li&gt;Formulate an analysis plan (Significance level is 0.05)&lt;/li&gt;&#xA;&lt;li&gt;Analyze sample data (calculate the &amp;sigma; and compute the z-score test statistic (z))&lt;/li&gt;&#xA;&lt;li&gt;Interpret results (Reject the null hypothesis iff the P-value is less than the significance level (0.05).)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I have a handful of questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Is this an appropriate test to use if there is no magnitude of error (only that the datum is correct or not)?&lt;/li&gt;&#xA;&lt;li&gt;The StatTrek page says to stop after 10 successes and 10 failures, but to sample no more than 10% of the population. Are 10, 10, and 10 just helpful rules of thumb, or is there some other guiding principle? &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="2591" LastActivityDate="2011-01-07T20:05:42.093" Title="Experiment design for proportion" Tags="&lt;experiment-design&gt;&lt;proportion&gt;" AnswerCount="1" CommentCount="6" />
  <row Id="5874" PostTypeId="1" AcceptedAnswerId="5878" CreationDate="2010-12-31T16:15:37.393" Score="3" ViewCount="113" Body="&lt;p&gt;The greatest gain of the statistics classes in both school and university seems to be that I now have an inkling of which QA site to use for this question. :)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm a programmer and I'm making a questionnaire with a financial background. There will be many (probably over 100) questions, which will be presented to the user one-by-one, like in a &lt;a href=&quot;http://en.wikipedia.org/wiki/Wizard_(software)&quot; rel=&quot;nofollow&quot;&gt;wizard&lt;/a&gt;. Since it answering them all will be quite time consuming (likely over an hour), it would be best if the user interface could show the estimated time remaining - both for the current question and all remaining together.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As there is no way of determining beforehand how long a question will take, my idea was to gather statistical data about it (especially during alpha/beta stages). The naive approach would be to just average the times taken for each question, but (thanks again to the above mentioned statistics classes) I'm aware that this isn't really the best way to do it. There are all kinds of normal distribution curves, error calculations, approximation methods, and other arcane mathematical artifacts that I could not even begin to dream about.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some more information:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Questions are of very varying lengths.&lt;/strong&gt; There are simple yes/no questions, and there are questions that will make you scratch your head. There are questions that ask to enter your contact details, and there are questions that require to meticulously fill a 30-row table. Some questions will ask you to calculate stuff. Some questions will require you to fetch some paperwork from your employer or something. Which brings up another point:&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;The whole process can be suspended/saved and resumed at a later time.&lt;/strong&gt; This shouldn't happen very often, but given the length of the process, and that some people don't pay attention to the pre-requirements, it can happen.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Some questions may not be applicable based on answers to previous questions.&lt;/strong&gt; For example, there might be a row of questions which are only applicable to businesses and not private persons.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Some questions are essentialy duplicated based on previous questions.&lt;/strong&gt; For example, if you entered that in the past year you worked for 4 employers, you will need to enter the same data for each of them.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;I can detect:&lt;/strong&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;When the user starts/finishes a question.&lt;/strong&gt; Hence, I can calculate the duration.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;When the user goes idle.&lt;/strong&gt; This is rather imprecise. I can only detect when there is no keyboard/mouse activity. This can be because the user has gone away, but it can also mean that he is digging through the papers looking for numbers. Or (since it will be a webpage), maybe he simply switched to another program - or another tab in the browser.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Which questions have been answered, which definately will need to be answered, which definately won't need to be answered, and which &lt;em&gt;might&lt;/em&gt; need to be answered.&lt;/strong&gt; For the last type of questions, I'm not sure if it might be possible to calculate a meaningful probability based on previous answers. Perhaps.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;So... any ideas on how to tackle this estimate? Remember - I'm going for maximum accuracy (or is it precision? better make it both), and would rather have better estimates than lower resource usage.&lt;/p&gt;&#xA;" OwnerUserId="2590" LastActivityDate="2010-12-31T21:35:13.707" Title="How to best estimate the time remaining for a variable-length questionnaire?" Tags="&lt;estimation&gt;" AnswerCount="2" CommentCount="4" />
  <row Id="5875" PostTypeId="2" ParentId="5870" CreationDate="2010-12-31T17:11:08.333" Score="4" ViewCount="" Body="&lt;p&gt;For $n \le 20$ just compute the distribution by brute force.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, with $n=3$ the pdf is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\begin{array}{}&#xD;&#xA; k=0: &amp;amp;-\frac{7}{64} \left(-1+4 \beta ^2\right)^3 \cr&#xD;&#xA; k=1: &amp;amp;\frac{39}{256} \left(1-4 \beta ^2\right)^2 \left(3+4 \beta ^2\right) \cr &#xD;&#xA; k=2: &amp;amp;-\frac{3}{128} \left(-15-92 \beta ^2+560 \beta ^4+192 \beta ^6\right) \cr &#xD;&#xA; k=3: &amp;amp;\frac{1}{256} \left(21+564 \beta ^2+1392 \beta ^4+448 \beta ^6\right)&#xD;&#xA;\end{array}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This was produced by the Mathematica code&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Clear[f, i, q];&#xA;f[n_, p_, k_] := Binomial[n, k] p^k (1 - p)^(n - k);&#xA;i[k1_, k2_, k3_, k_] := If[Max[{k1, k2, k3}] - Min[{k1, k2, k3}] == k, 1, 0];&#xA;q[n_, k_, p_] := &#xA;  Sum[f[n, 1/2 + p, k1] f[n, 1/2, k2] f[n, 1/2 - p, k3] &#xA;     i[k1, k2, k3, k], {k1, 0, n}, {k2, Max[0, k1 - k], Min[n, k1 + k]}, &#xA;     {k3, Max[0, k1 - k, k2 - k], Min[n, k1 + k, k2 + k]}];&#xA;With[{n=3}, Table[q[n, k, \[Beta]], {k,0,n}]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Calculations of the entire distribution for $n=20$ take four seconds.  Here are plots for $\beta = 0, 1/6, 1/3$.  (As $\beta$ increases, the distribution shifts to the right.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/W9ewt.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-12-31T17:11:08.333" CommentCount="2" />
  <row Id="5876" PostTypeId="2" ParentId="5853" CreationDate="2010-12-31T17:47:11.673" Score="6" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;A scaled range&lt;/strong&gt;, like 200 to 800 (for SATs, &lt;em&gt;e.g.&lt;/em&gt;), &lt;strong&gt;is just a change of units of measurement&lt;/strong&gt;.  (It works exactly like changing temperatures in Fahrenheit to those in Celsius.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The middle value of 500 is intended to correspond to the average of the data.  The range is intended to correspond to about 99.7% of the data &lt;em&gt;when the data do follow a Normal distribution&lt;/em&gt; (&quot;Bell curve&quot;).  It is &lt;em&gt;guaranteed&lt;/em&gt; to include 8/9 of the data (&lt;a href=&quot;http://en.wikipedia.org/wiki/Chebyshev%27s_inequality&quot; rel=&quot;nofollow&quot;&gt;Chebyshev's Inequality&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this case, the formula 1-5 computes the standard deviation of the data.  This is simply a new unit of measurement for the original data.  It needs to correspond to 100 units in the new scale.  Therefore, to convert an original value to the scaled value,&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Subtract the average.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Divide by the standard deviation.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Multiply by 100.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Add 500.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If the result lies beyond the range $[200, 800]$ you can either use it as-is or &quot;clamp&quot; it to the range by rounding up to 200, down to 800.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;In the example&lt;/strong&gt;, using data $\{1,3,4,5,7\}$, the average is $4$ and the SD is $2$.  Therefore, upon rescaling, $1$ becomes $(1 - 4)/2 * 100 + 500 = 350$.  The entire rescaled dataset, computed similarly, is $\{350, 450, 500, 550, 650\}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;When the original data are distributed in a distinctly non-normal way, you need another approach&lt;/strong&gt;.  You no longer compute an average or SD.  Instead, put all the scores in order, from 1st (smallest) up to $n$th (largest).  These are their &lt;em&gt;ranks&lt;/em&gt;.  Convert any rank $i$ into its &lt;em&gt;percentage&lt;/em&gt; $(i-1/2)/n$.  (In the example, $n=5$ and data are already in rank order $i=1,2,3,4,5$.  Therefore their percentages are $1/10, 3/10, 5/10, 7/10, 9/10$, often written equivalently as $10\%, 30\%$, etc.)  Corresponding to any percentage (between $0$ and $1$, necessarily) is a &lt;em&gt;normal quantile&lt;/em&gt;.  It is computed with the &lt;a href=&quot;http://en.wikipedia.org/wiki/Quantile_function&quot; rel=&quot;nofollow&quot;&gt;normal quantile function&lt;/a&gt;, which is closely related to the &lt;a href=&quot;http://en.wikipedia.org/wiki/Normal_distribution#Quantile_function&quot; rel=&quot;nofollow&quot;&gt;error function&lt;/a&gt;.  (Simple &lt;a href=&quot;http://en.wikipedia.org/wiki/Normal_distribution#Numerical_approximations_for_the_normal_CDF&quot; rel=&quot;nofollow&quot;&gt;numerical approximations&lt;/a&gt; are straightforward to code.)  Its values, which typically will be between -3 and 3, have to be rescaled (just as before) to the range $[200, 800]$.  Do this by first multiplying the normal quantile by 100 and then adding 500.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The normal quantile function is available in many computing platforms, including spreadsheets (Excel's &lt;strong&gt;normsinv&lt;/strong&gt;, for instance).  &lt;strong&gt;For example,&lt;/strong&gt; the normal quantiles (or &quot;normal scores&quot;) for the data $\{1,3,4,5,7\}$ are $\{372, 448, 500, 552, 628\}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This &quot;&lt;a href=&quot;http://en.wikipedia.org/wiki/Normal_score&quot; rel=&quot;nofollow&quot;&gt;normal scoring&lt;/a&gt;&quot; approach will always give scores between 200 and 800 when you have 370 or fewer values.  When you have 1111 or fewer values, all but the highest and lowest will have scores between 200 and 800.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-01-09T16:55:51.487" LastActivityDate="2011-01-09T16:55:51.487" CommentCount="3" />
  <row Id="5877" PostTypeId="2" ParentId="5853" CreationDate="2010-12-31T18:58:15.357" Score="1" ViewCount="" Body="&lt;p&gt;You could try this approach - normalise your data set to range between the values -1 and +1 thus:- &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(((individual_value - min_of_all_values) / (max_of_all_values - min_of-all_values))-0.5)*2. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This will convert every value in your data set to a value between -1 and +1, with the actual maximum and minimum values being set to +1 and -1 respectively, and then reset these +1 and -1 values to be +0.9999 and -0.9999 (necessary for following calculations.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then apply the &lt;a href=&quot;http://en.wikipedia.org/wiki/Fisher_transformation&quot; rel=&quot;nofollow&quot;&gt;Fisher Transformation&lt;/a&gt; to each of the above normalised values to &quot;force it&quot; to approximately conform to a normal distribution, and then &quot;un-normalise&quot; each of these Fisher Transform values to range in value between 200 and 800 thus:- &lt;/p&gt;&#xA;&#xA;&lt;p&gt;((Fish_value - min_all_Fish_values) / (max_all_Fish_values - min_all_Fish_values))*600 + 200&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The maximum Fisher Transform value will be set to exactly 800, the minimum Fisher Transform value will be set to exactly 200, and all the other values will lie between these two extremes, according to an approximate normal distribution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Referencing your original question on SO and the issue of scalability, the advantage of this approach is that provided any new data point is not itself a new maximum or minimum for the data set as a whole you can apply the above calculations to the new data point to get its score between 200 and 800 without affecting any of the existing scores of the original data set. If a new data point is a new maximum or minimum you will have to recalculate the scores for the whole data set with this new &quot;normalising&quot; maximum or minimum value. &lt;/p&gt;&#xA;" OwnerUserId="226" LastActivityDate="2010-12-31T18:58:15.357" />
  <row Id="5878" PostTypeId="2" ParentId="5874" CreationDate="2010-12-31T19:06:34.287" Score="2" ViewCount="" Body="&lt;p&gt;I do not see why averages should not work.  You might want to use &lt;a href=&quot;http://en.wikipedia.org/wiki/Winsorized_mean&quot; rel=&quot;nofollow&quot;&gt;winsorized&lt;/a&gt; or &lt;a href=&quot;http://en.wikipedia.org/wiki/Trimmed_mean&quot; rel=&quot;nofollow&quot;&gt;trimmed&lt;/a&gt; averages  to  guard against outliers, which are likely due to inability to tell exactly whether user is idle or not. Then for the estimated time remaining show the sum of the averages for each question. You could build more complicated and precise methods if you collected data on the completed time together with user characteristics. For that though you will probably need another questionaire, so unless you are expecting a lot of feedback in alpha and beta phases, I suggest using the simple averages. Sometimes simplest methods work the best.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2010-12-31T20:34:31.280" LastActivityDate="2010-12-31T20:34:31.280" CommentCount="6" />
  <row Id="5879" PostTypeId="2" ParentId="5874" CreationDate="2010-12-31T21:35:13.707" Score="2" ViewCount="" Body="&lt;p&gt;As @mpiktas points out, simple is good.  But I can't help thinking there will be substantial variation among users.  Roughly, we can conceive of the time to complete each question as depending on the question and on the user, so the trick is to tease out the question-specific timing factors from the experience of previous users and to estimate the user-specific timing factor from the user's experience with the questions they have already tackled.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This opens up the possibilities greatly.  A good answer would rely on exploratory analysis of preliminary data.  In general, though, a combination of (1) determining a suitable way to express question completion times (logarithms?  square roots?) and (2) robust analysis of variance techniques (such as &lt;a href=&quot;http://www.itl.nist.gov/div898/software/dataplot/refman1/ch3/median_p.pdf&quot; rel=&quot;nofollow&quot;&gt;median polish&lt;/a&gt;) or &lt;a href=&quot;http://en.wikipedia.org/wiki/Iteratively_reweighted_least_squares&quot; rel=&quot;nofollow&quot;&gt;IRLS&lt;/a&gt; regression is likely to do far better than ignoring the user-specific factors (&quot;simple averaging&quot;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you have estimated question-specific timing factors, which could be as simple as computing median or mean times per question, you could easily estimate the timing factor for a new user by comparing their times to typical times for the questions they have already attempted.  For instance (this is truly simplistic, but gives the flavor of the idea) if they take twice as long to complete the first five questions as the sum of the typical completion times for those questions, you could estimate that this user will take twice as long as the average to complete the next 95 questions.  You can then add the average completion times for those 95 questions and double the sum.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2010-12-31T21:35:13.707" CommentCount="8" />
  <row Id="5880" PostTypeId="1" AcceptedAnswerId="5881" CreationDate="2011-01-01T09:48:45.877" Score="5" ViewCount="188" Body="&lt;p&gt;Hi,&lt;br&gt;&#xA;The probability of that a number between 1 to x is prime is $\frac{1}{\ln{x}}$ as per prime number theorem and also the total number of prime numbers between $1$ to $x$ will be $\frac{x}{\ln{x}}$.&#xA;But if we select $n$ (32 bit) random numbers, what is the probability that $p$ of them are primes?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or Simply put&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the probability of picking $p$ primes from $n$ random numbers (32 bit).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;TIA..&lt;/p&gt;&#xA;" OwnerUserId="2597" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-01-01T18:07:23.237" LastActivityDate="2011-01-27T23:45:44.303" Title="What is the probability of picking p primes from n random numbers?" Tags="&lt;probability&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="5881" PostTypeId="2" ParentId="5880" CreationDate="2011-01-01T10:20:34.620" Score="12" ViewCount="" Body="&lt;p&gt;There are 203,280,221 primes less than $2^{32}$. (&lt;a href=&quot;http://www.ieeta.pt/~tos/primes.html&quot; rel=&quot;nofollow&quot;&gt;Source&lt;/a&gt;). So the probability that a random 32-bit number is prime is $203,280,221 / 2^{32} \approx 0.04733$. Assuming you want selection &lt;em&gt;with&lt;/em&gt; replacement, i.e. the same number can be picked more than once, the probability of picking $p$ primes from $n$ 32-bit random numbers is, from the &lt;a href=&quot;http://en.wikipedia.org/wiki/Binomial_distribution#Probability_mass_function&quot; rel=&quot;nofollow&quot;&gt;probability mass function of the binomial distribution&lt;/a&gt;,&#xA;$$\frac {n!}{p!(n-p)!} 0.04733^p (1-0.04733)^{n-p} . $$&lt;/p&gt;&#xA;" OwnerUserId="449" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-01-27T23:45:44.303" LastActivityDate="2011-01-27T23:45:44.303" CommentCount="1" />
  <row Id="5882" PostTypeId="2" ParentId="1980" CreationDate="2011-01-01T10:35:17.683" Score="1" ViewCount="" Body="&lt;p&gt;Robert Gentleman wrote a paper called &lt;a href=&quot;http://www.bepress.com/sagmb/vol4/iss1/art2/&quot; rel=&quot;nofollow&quot;&gt;&quot;Reproducible Research: A Bioinformatics Case Study&quot;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It implements a short set of analyses as an R Package and uses Sweave.&#xA;It also discusses the use of Sweave more generally.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See the &lt;a href=&quot;http://www.bepress.com/sagmb/vol4/iss1/art2/&quot; rel=&quot;nofollow&quot;&gt;&quot;Related Files&quot;&lt;/a&gt; section of the article page for an archive file of all files and folders used.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Reference:&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Gentleman, Robert (2005) &quot;Reproducible Research: A Bioinformatics Case Study,&quot; Statistical Applications in Genetics and Molecular Biology: Vol. 4 : Iss. 1, Article 2. &lt;/li&gt;&#xA;&lt;li&gt;DOI: 10.2202/1544-6115.1034&lt;/li&gt;&#xA;&lt;li&gt;Available at: &lt;a href=&quot;http://www.bepress.com/sagmb/vol4/iss1/art2&quot; rel=&quot;nofollow&quot;&gt;http://www.bepress.com/sagmb/vol4/iss1/art2&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastActivityDate="2011-01-01T10:35:17.683" />
  <row Id="5883" PostTypeId="2" ParentId="5831" CreationDate="2011-01-01T22:49:18.977" Score="6" ViewCount="" Body="&lt;p&gt;This isn't exactly what you're asking for, but I think you'd like David Freedman's work. He calls BS on misapplication of statistical tests, etc... See here: &lt;a href=&quot;http://www.stat.berkeley.edu/~freedman/&quot; rel=&quot;nofollow&quot;&gt;http://www.stat.berkeley.edu/~freedman/&lt;/a&gt;. One of my favorites is “What is the probability of an earthquake?”.&lt;/p&gt;&#xA;" OwnerUserId="2489" LastActivityDate="2011-01-01T22:49:18.977" CommunityOwnedDate="2011-01-01T22:49:18.977" CommentCount="1" />
  <row Id="5885" PostTypeId="1" CreationDate="2011-01-02T03:16:43.820" Score="6" ViewCount="201" Body="&lt;p&gt;Any suggestion for a good source to learn MCMC methods?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;thanks.&lt;/p&gt;&#xA;" OwnerUserId="2601" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-01-02T11:22:32.533" LastActivityDate="2011-01-02T21:39:14.060" CommunityOwnedDate="2011-01-02T11:22:43.313" Title="MCMC method -- good sources" Tags="&lt;books&gt;&lt;mcmc&gt;" AnswerCount="5" CommentCount="2" FavoriteCount="5" />
  <row Id="5886" PostTypeId="2" ParentId="5885" CreationDate="2011-01-02T07:36:06.087" Score="2" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://books.google.co.uk/books?id=TRXrMWY_i2IC&quot; rel=&quot;nofollow&quot;&gt;Gilks W.R., Richardson S., Spiegelhalter D.J. &lt;em&gt;Markov Chain Monte Carlo in Practice&lt;/em&gt;. Chapman &amp;amp; Hall/CRC, 1996.&lt;/a&gt;&lt;br&gt;&#xA;A relative oldie now, but still a goodie.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-01-02T07:36:06.087" CommunityOwnedDate="2011-01-02T11:22:43.313" />
  <row Id="5887" PostTypeId="1" AcceptedAnswerId="5888" CreationDate="2011-01-02T08:31:58.303" Score="7" ViewCount="143" Body="&lt;p&gt;We have a set biological samples that was pretty expensive to obtain. We put these samples through a series of tests to generate data that are used for building a predictive model. For this purpose we have divided the samples to training (70%) and testing (30%) sets. We have successfully created a model and applied it on the testing set to discover that the performance was &quot;less than optimal&quot;. The experimentalists now want to improve the biological tests in order to create a better model. Provided that we cannot obtain new samples, would you suggest us to re-shuffle the samples to create new training and validation sets or to stick with the original division. (We don't have any indication that the division was a problematic one).&lt;/p&gt;&#xA;" OwnerUserId="2604" LastActivityDate="2011-01-02T20:40:16.190" Title="Should I re-shuffle my data?" Tags="&lt;modeling&gt;&lt;classification&gt;&lt;experiment-design&gt;&lt;bootstrap&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="1" />
  <row Id="5888" PostTypeId="2" ParentId="5887" CreationDate="2011-01-02T09:22:46.873" Score="9" ViewCount="" Body="&lt;p&gt;As you already use a holdout sample, I would say you should keep it and build your new models on the same training sample so that all models will consider the same relationships between features. In addition, if you perform feature selection, the samples must be left out before any of these filtering stages; that is, feature selection must be included in the cross-validation loop.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of note, there are more powerful methods than a 0.67/0.33 split for model selection, namely k-fold cross-validation or leave-one-out.  See &lt;em&gt;e.g.&lt;/em&gt; &lt;a href=&quot;http://www-stat.stanford.edu/~tibs/ElemStatLearn/&quot; rel=&quot;nofollow&quot;&gt;The Elements of Statistical Learning&lt;/a&gt; (§7.10, pp. 241-248), &lt;a href=&quot;http://www.modelselection.org/&quot; rel=&quot;nofollow&quot;&gt;www.modelselection.org&lt;/a&gt; or &lt;a href=&quot;http://www.i-journals.org/ss/viewarticle.php?id=54&amp;amp;layout=abstract&quot; rel=&quot;nofollow&quot;&gt;A survey of cross-validation procedures for model selection&lt;/a&gt; by Arlot and Celisse (more advanced mathematical background required).&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-01-02T20:40:16.190" LastActivityDate="2011-01-02T20:40:16.190" />
  <row Id="5889" PostTypeId="2" ParentId="5885" CreationDate="2011-01-02T09:32:02.837" Score="7" ViewCount="" Body="&lt;p&gt;For online tutorials, there are&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.soton.ac.uk/~sks/utrecht/&quot; rel=&quot;nofollow&quot;&gt;A tutorial in MCMC&lt;/a&gt;, by Sahut (2000)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://kmh-lanl.hansonhub.com/talks/maxent00b.pdf&quot; rel=&quot;nofollow&quot;&gt;Tutorial on Markov Chain Monte Carlo&lt;/a&gt;, by Hanson (2000)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://civs.ucla.edu/old/MCMC/MCMC_tutorial.htm&quot; rel=&quot;nofollow&quot;&gt;Markov Chain Monte Carlo for Computer Vision&lt;/a&gt;, by Zhu et al. (2005)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.worldscibooks.com/etextbook/5904/5904_intro.pdf&quot; rel=&quot;nofollow&quot;&gt;Introduction to Markov Chain Monte Carlo simulations and their statistical analysis&lt;/a&gt;, by Berg (2004).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://projecteuclid.org/DPubS?service=UI&amp;amp;version=1.0&amp;amp;verb=Display&amp;amp;handle=euclid.ss/1177011137&quot; rel=&quot;nofollow&quot;&gt;Practical Markov Chain Monte Carlo&lt;/a&gt;, by Geyer (&lt;em&gt;Stat. Science&lt;/em&gt;, 1992), is also a good starting point, and you can look at the &lt;a href=&quot;http://cran.r-project.org/web/packages/MCMCpack/index.html&quot; rel=&quot;nofollow&quot;&gt;MCMCpack&lt;/a&gt; or &lt;a href=&quot;http://cran.r-project.org/web/packages/mcmc/&quot; rel=&quot;nofollow&quot;&gt;mcmc&lt;/a&gt; R packages for illustrations.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-01-02T12:08:40.193" LastActivityDate="2011-01-02T12:08:40.193" CommunityOwnedDate="2011-01-02T11:22:43.313" />
  <row Id="5890" PostTypeId="2" ParentId="5885" CreationDate="2011-01-02T10:17:49.443" Score="3" ViewCount="" Body="&lt;p&gt;I haven't read it (yet), but if you're into R, there is Christian P. Robert book:&#xA;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1441915753&quot; rel=&quot;nofollow&quot;&gt;Introducing Monte Carlo Methods with R (Use R)&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know of it from following his (very good) &lt;a href=&quot;http://xianblog.wordpress.com/&quot; rel=&quot;nofollow&quot;&gt;blog&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="253" LastActivityDate="2011-01-02T10:17:49.443" CommunityOwnedDate="2011-01-02T11:22:43.313" />
  <row Id="5891" PostTypeId="2" ParentId="5885" CreationDate="2011-01-02T11:20:10.533" Score="2" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://books.google.co.uk/books?id=yPvECi_L3bwC&quot; rel=&quot;nofollow&quot;&gt;Dani Gamerman &amp;amp; Hedibert F. Lopes. &lt;em&gt;Markov Chain Monte Carlo: Stochastic Simulation for Bayesian Inference&lt;/em&gt; (2nd ed.). Boca Raton, FL: Champan &amp;amp; Hall/CRC, 2006. 344 pp. ISBN 0-412-81820-5&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-- a more recently updated book than Gilks, Richardson &amp;amp; Spiegelhalter. I haven't read it myself, but it was well &lt;a href=&quot;http://pubs.amstat.org/doi/abs/10.1198/tech.2008.s542&quot; rel=&quot;nofollow&quot;&gt;reviewed in &lt;em&gt;Technometrics&lt;/em&gt; in 2008&lt;/a&gt;, and the first edition also got a good &lt;a href=&quot;http://www.jstor.org/stable/2988381&quot; rel=&quot;nofollow&quot;&gt;review in &lt;em&gt;The Statistician&lt;/em&gt;&lt;/a&gt; back in 1998.&lt;/p&gt;&#xA;" OwnerUserId="449" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-01-02T12:06:00.227" LastActivityDate="2011-01-02T12:06:00.227" CommunityOwnedDate="2011-01-02T11:22:43.313" />
  <row Id="5892" PostTypeId="2" ParentId="5750" CreationDate="2011-01-02T11:57:26.020" Score="0" ViewCount="" Body="&lt;p&gt;This is not really an answer, but how does one go about using less conservative P-value adjustments (e.g. that proposed by Benjamini and Hochberg) when you're dealing with a correlation matrix? I'm having computational issues with R at the moment in this respect. Not sure how to proceed with performing the P-value adjustment whilst preserving the structure of the correlation matrix.... i.e. the adjustment is based on fewer than half of the actual numbers present in the correlation matrix! but I don't know how to inform R about this.&lt;/p&gt;&#xA;" OwnerUserId="2606" LastActivityDate="2011-01-02T11:57:26.020" CommentCount="1" />
  <row Id="5893" PostTypeId="1" CreationDate="2011-01-02T17:04:44.293" Score="6" ViewCount="211" Body="&lt;p&gt;A name: first, possibly a middle, and surname.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm curious about how much information you can mine out of a name, using publicly available datasets. I know that you can get the following with anywhere between a low-high probability (depending on the input) using US census data:&#xA;1) Gender.&#xA;2) Race.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Facebook for instance, used exactly that to find out, with a decent level of accuracy, the racial distribution of users of their site (https://www.facebook.com/note.php?note_id=205925658858).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What else can be mined? I'm not looking for anything specific, this is a very open-ended question to assuage my curiousity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My examples are US specific, so we'll assume that the name is the name of someone located in the US; but, if someone knows of publicly available datasets for other countries, I'm more than open to them too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not quite sure if this is the right place for this, if it's not, I'd appreciate if someone could point me to a more appropriate place.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope this is an interesting question, and this is the appropriate place!&lt;/p&gt;&#xA;" OwnerUserId="2608" LastActivityDate="2011-01-03T20:34:28.793" Title="How much information can you mine out of a name?" Tags="&lt;dataset&gt;&lt;data-mining&gt;&lt;census&gt;" AnswerCount="7" CommentCount="3" FavoriteCount="3" />
  <row Id="5894" PostTypeId="1" AcceptedAnswerId="5897" CreationDate="2011-01-02T17:52:18.390" Score="4" ViewCount="104" Body="&lt;p&gt;I am new to statistics so I apologize for the simplicity of my question. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a bag with 27 numbers inside; I will pull one number out until I get number 17. THe numbers will be returned to the bag after every try so the prob of success will always be 1/27.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am trying to determine on average how many pulls I will need.  &lt;/p&gt;&#xA;" OwnerUserId="2610" LastActivityDate="2011-01-03T00:10:55.017" Title="Number of tried for a favorable result" Tags="&lt;binomial&gt;&lt;negative-binomial&gt;" AnswerCount="4" />
  <row Id="5895" PostTypeId="2" ParentId="5894" CreationDate="2011-01-02T17:59:57.440" Score="2" ViewCount="" Body="&lt;p&gt;It is easier to think backwards. What is the chance of NOT choosing 17? That chance is 26/27. What is the chance of not choosing 17 twice in a row? The answer is (26/27)&lt;em&gt;(26/27). How about three times in a row?  The answer is (26/27)&lt;/em&gt;(26/27) *(26/27). More generally, the answer is (26/27)^K, where K is the number of times you try and &quot;^&quot; means &quot;to the power of&quot;. You can compute that probability for various values of K, and find a value of K that gives an answer close to 0.50. &lt;/p&gt;&#xA;" OwnerUserId="25" LastActivityDate="2011-01-02T17:59:57.440" />
  <row Id="5896" PostTypeId="2" ParentId="5894" CreationDate="2011-01-02T19:07:29.217" Score="3" ViewCount="" Body="&lt;p&gt;On average, you need 27 tries.&lt;/p&gt;&#xA;" OwnerUserId="2592" LastActivityDate="2011-01-02T19:07:29.217" CommentCount="3" />
  <row Id="5897" PostTypeId="2" ParentId="5894" CreationDate="2011-01-02T19:14:42.883" Score="2" ViewCount="" Body="&lt;p&gt;Harvey explained the logic.  But the name of the distribution you're interested in is the geometric distribution.&lt;/p&gt;&#xA;" OwnerUserId="1028" LastActivityDate="2011-01-02T19:14:42.883" CommentCount="1" />
  <row Id="5898" PostTypeId="2" ParentId="5893" CreationDate="2011-01-02T20:43:26.670" Score="2" ViewCount="" Body="&lt;p&gt;From the first name predict region, age, first generation immigrant status. From the last name you could predict geographical location of original patronym. For the full name you could predict social and economic status (Thurston Howell III).&lt;/p&gt;&#xA;" OwnerUserId="82" LastActivityDate="2011-01-02T20:43:26.670" />
  <row Id="5899" PostTypeId="1" AcceptedAnswerId="5905" CreationDate="2011-01-02T21:14:13.577" Score="3" ViewCount="180" Body="&lt;p&gt;I am sorry for the vague title - its because I don't know what I am talking about.&lt;br&gt;&#xA;Imagine a committee of people in charge of hiring a CEO. A committee member can vote &quot;No hire&quot; (+0), &quot;Maybe&quot;(+1), &quot;Hire&quot;(+2) for a potential CEO candidate. Each CEO is scored based on the votes, and the CEO with the highest score is hired.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, imagine a situation where two candidates are tied... both of them have a score of 5. However, candidate1 had 2Hires and 1Maybe (ie, 2+2+1), whereas candidate2 had 5Maybes (ie, 1+1+1+1+1).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is my question:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Intuitively, I think we can say that we are more confident about candidate1 eventho the scores are tied. What specific domain of statistics am I dealing with here (so that I can investigate on my own)&lt;/li&gt;&#xA;&lt;li&gt;In statistics, how does one go about resolving such situations? And I am not just merely talking about resolving a tie... but rather, how to resolve this in a generic manner?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Thank you.&lt;/p&gt;&#xA;" OwnerUserId="2605" LastActivityDate="2011-02-03T12:51:24.553" Title="A basic voting system question" Tags="&lt;basic-concepts&gt;" AnswerCount="3" CommentCount="3" FavoriteCount="1" />
  <row Id="5900" PostTypeId="2" ParentId="5885" CreationDate="2011-01-02T21:39:14.060" Score="2" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://www.crcpress.com/product/isbn/9781420079418;jsessionid=AYSwK8euwh4-aVy7BXE2wQ%2a%2a&quot; rel=&quot;nofollow&quot;&gt;Handbook of Markov Chain Monte Carlo, Steve Brooks, Andrew Gelman, Galin Jones and Xiao-Li Meng, eds. 2011 CRC Press.&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.stat.columbia.edu/~gelman/research/published/GelmanMCMCchapter4.pdf&quot; rel=&quot;nofollow&quot;&gt;Chapter 4&lt;/a&gt;, 'Inference from simulations and monitoring convergence' by Gelman and Shirley, is available online.&lt;/p&gt;&#xA;" OwnerUserId="1381" LastActivityDate="2011-01-02T21:39:14.060" CommunityOwnedDate="2011-01-02T21:39:14.060" CommentCount="1" />
  <row Id="5901" PostTypeId="2" ParentId="5894" CreationDate="2011-01-02T21:55:46.263" Score="0" ViewCount="" Body="&lt;p&gt;Edit: I made a presumption before where after 27 tries it would then stop searching thus giving a bias on the results&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Number of repeats&#xA;iters = 10000&#xA;results = rep(0,iters)&#xA;# Ball to search for&#xA;searchNumber = 27&#xA;&#xA;for(i in 1:iters) {&#xA;    # A random permutation:&#xA;    balls = sample(1:27,iters, replace=T)&#xA;&#xA;    # Number of times picking a ball&#xA;    tries = 1&#xA;&#xA;    while(tries&amp;lt;iters) {&#xA;        if(balls[tries] == searchNumber) {&#xA;            results[i] = tries&#xA;            break&#xA;        }&#xA;        tries = tries + 1&#xA;    }&#xA;}&#xA;&#xA;hist(results, breaks=100, col=&quot;blue&quot;)&#xA;mean(results)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Edit: with a mean / average of around 27&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Edit: In my example we are looking for 27 but the results are the same, it could be any number between 1-27&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="2612" LastEditorUserId="2612" LastEditorDisplayName="" LastEditDate="2011-01-03T00:10:55.017" LastActivityDate="2011-01-03T00:10:55.017" CommentCount="2" />
  <row Id="5902" PostTypeId="2" ParentId="5836" CreationDate="2011-01-02T22:56:50.777" Score="0" ViewCount="" Body="&lt;p&gt;Try the &lt;a href=&quot;http://directlabels.r-forge.r-project.org/&quot; rel=&quot;nofollow&quot;&gt;directlabels&lt;/a&gt; package for some more control.  As far as the caption goes, if you're automatically generating the graphs, you might simply use it as caption text (in LaTeX or HTML or what have you).  Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="317" LastActivityDate="2011-01-02T22:56:50.777" />
  <row Id="5903" PostTypeId="1" CreationDate="2011-01-02T23:59:51.453" Score="7" ViewCount="438" Body="&lt;p&gt;Given two arrays x and y, both of length n, I fit a model y = a + b*x and want to calculate a 95% confidence interval for the slope. This is (b - delta, b + delta) where b is found in the usual way and&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;delta = qt(0.975,df=n-2)*se.slope&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and se.slope is the standard error in the slope. One way to get the standard error of the slope from R is &lt;code&gt;summary(lm(y~x))$coef[2,2]&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now suppose I write the likelihood of the slope given x and y, multiply this by a &quot;flat&quot; prior and use a MCMC technique to draw a sample &lt;em&gt;m&lt;/em&gt; from the posterior distribution. Define&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;lims = quantile(m,c(0.025,0.975))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My question: is &lt;code&gt;(lims[[2]]-lims[[1]])/2&lt;/code&gt; approximately equal to delta as defined above?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Addendum&lt;/strong&gt; Below is a simple JAGS model where these two seem to be different. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model {&#xA; for (i in 1:N) {&#xA;  y[i] ~ dnorm(mu[i], tau)&#xA;  mu[i] &amp;lt;- a + b * x[i]&#xA; }&#xA; a ~ dnorm(0, .00001)&#xA; b ~ dnorm(0, .00001)&#xA; tau &amp;lt;- pow(sigma, -2)&#xA; sigma ~ dunif(0, 100)&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I run the following in R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;N &amp;lt;- 10&#xA;x &amp;lt;- 1:10&#xA;y &amp;lt;- c(30.5,40.6,20.5,59.1,52.5,&#xA;       96.0,121.4,78.9,112.1,128.4)&#xA;lin &amp;lt;- lm(y~x)&#xA;&#xA;#Calculate delta for a 95% confidence interval on the slope&#xA;delta.lm &amp;lt;- qt(0.975,df=N-2)*summary(lin)$coef[2,2]&#xA;&#xA;library('rjags')&#xA;jags &amp;lt;- jags.model('example.bug', data = list('x' = x,'y' = y,'N' = N),&#xA;                   n.chains = 4,n.adapt = 100)&#xA;update(jags, 1000)&#xA;params &amp;lt;- jags.samples(jags,c('a', 'b', 'sigma'),7500)&#xA;lims &amp;lt;- quantile(params$b,c(0.025,0.975))&#xA;delta.bayes &amp;lt;- (lims[[2]]-lims[[1]])/2&#xA;&#xA;cat(&quot;Classical confidence region: +/-&quot;,round(delta.lm, digits=4),&quot;\n&quot;)&#xA;cat(&quot;Bayesian confidence region:  +/-&quot;,round(delta.bayes,digits=4),&quot;\n&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Classical confidence region: +/- 4.6939&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bayesian confidence region:  +/- 5.1605&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rerunning this multiple times, the Bayesian confidence region is consistently wider than the classical one. So is this due to the priors I've chosen?&lt;/p&gt;&#xA;" OwnerUserId="2617" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-01-16T14:51:59.780" LastActivityDate="2011-01-16T14:52:45.937" Title="Confidence intervals for regression parameters: Bayesian vs. classical" Tags="&lt;r&gt;&lt;regression&gt;&lt;bayesian&gt;&lt;confidence-interval&gt;&lt;frequentist&gt;" AnswerCount="4" FavoriteCount="1" />
  <row Id="5904" PostTypeId="2" ParentId="5903" CreationDate="2011-01-03T01:53:27.830" Score="4" ViewCount="" Body="&lt;p&gt;Hi Ringold:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you sample from the posterior of b | y and calculate lims (as you define) it should be same as (b - delta, b + delta). Specifically, if you calculate the posterior distribution of b | y under a flat prior, it is same as the classical sampling distribution of b.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For more details refer to: Gelman et al. (2003). Bayesian Data Analysis. CRC Press. Section 3.6&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ringold, the behavior observed by you is consistent with the Bayesian idea. The Bayesian Credible Interval (CI) is generally wider than the classical ones. And the reason is, as you correctly guessed, the hyperpriors taken into account the variability because of the unknown parameters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For simple scenarios like these (NOT IN GENERAL):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Baysian CI &gt; Empirical Bayesian CI &gt; Classical CI ; &gt; == wider &lt;/p&gt;&#xA;" OwnerUserId="1307" LastEditorUserId="1307" LastEditorDisplayName="" LastEditDate="2011-01-03T13:56:55.263" LastActivityDate="2011-01-03T13:56:55.263" CommentCount="1" />
  <row Id="5905" PostTypeId="2" ParentId="5899" CreationDate="2011-01-03T01:59:54.670" Score="8" ViewCount="" Body="&lt;p&gt;To give some context, I don't view this as a &quot;statistical&quot; question as much of a &quot;group preference&quot; question. Economists and policy wonks do a lot of thinking about questions of how to convert individual preferences into a &quot;will of the people.&quot; You will find lots of interesting reading if you search the web for &quot;political economy&quot; and &quot;voting system&quot; together.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Reasonable people will disagree on which candidate &quot;wins&quot; for the example you gave. There is no objectively correct voting system. Baked into the idea of a voting system are assumptions about fairness and representation. Each voting system has advantageous and disadvantageous properties. (Digging into the pros/cons of various voting systems is super interesting but would take many pages. I recommend by starting with Wikipedia's entry on &quot;Voting Systems&quot;. Also be sure to read about Arrow's Impossibility Theorem.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Don't expect to find a universally accepted &quot;best&quot; voting system. Rather, pick a voting system that seems most reasonable for your domain (i.e. the upsides of the system outweigh the downsides). Make sure that your constituency buys-in to the voting system if you want the results to be taken seriously.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Resolving this described situation cannot be done with statistics alone. You will need to choose a voting system -- that choice will drive the ballot choice and how the ballots are converted into results (e.g. one or more winners, some sort of ranking, or some sort of scoring).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a big question. I think you'll enjoy digging into these topics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, for what it is worth, I have strong reservations about the described ballot (0 for no hire, +1 for maybe, +2 for hire) as a way for a board to pick a CEO.&lt;/p&gt;&#xA;" OwnerUserId="660" LastEditorUserId="660" LastEditorDisplayName="" LastEditDate="2011-02-02T15:44:38.890" LastActivityDate="2011-02-02T15:44:38.890" CommentCount="2" />
  <row Id="5906" PostTypeId="1" CreationDate="2011-01-02T17:12:10.053" Score="0" ViewCount="118" Body="&lt;p&gt;While not directly related to programming, I figured that the programmers on here would be just as curious as I was about this question. Feel free to close the question if it does not meet with the guidelines.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A name: first, possibly a middle, and surname.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm curious about how much information you can mine out of a name, using publicly available datasets. I know that you can get the following with anywhere between a low-high probability (depending on the input) using US census data: 1) Gender. 2) Race.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Facebook for instance, used exactly that to find out, with a decent level of accuracy, the racial distribution of users of their site (https://www.facebook.com/note.php?note_id=205925658858).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What else can be mined? I'm not looking for anything specific, this is a very open-ended question to assuage my curiousity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My examples are US specific, so we'll assume that the name is the name of someone located in the US; but, if someone knows of publicly available datasets for other countries, I'm more than open to them too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope this is an interesting question!&lt;/p&gt;&#xA;" LastActivityDate="2011-01-03T11:13:42.550" Title="How much information can you mine out of a name?" Tags="&lt;dataset&gt;&lt;data-mining&gt;" AnswerCount="0" CommentCount="0" FavoriteCount="0" />
  <row Id="5907" PostTypeId="2" ParentId="5893" CreationDate="2011-01-02T19:30:45.810" Score="1" ViewCount="" Body="&lt;p&gt;The amount of information that can be found varies wildly, from just race and gender, to all sorts of personal info. Your best bet at getting the information would be social network sites like facebook, as they generally provide more information than cencus databases.&lt;/p&gt;&#xA;" LastActivityDate="2011-01-03T11:13:42.550" />
  <row Id="5908" PostTypeId="2" ParentId="5893" CreationDate="2011-01-02T19:41:43.203" Score="1" ViewCount="" Body="&lt;p&gt;There's quite a wide range of information you can get depending on the sources you use. Census data is an obvious one. You can also get information from Facebook, MySpace and other social networking sites. You could also probably search public news archives for mentions of their name. Maybe even those ubclained property sites that some states have.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want a real world world example of what can be done, take a look at pipl.com&lt;/p&gt;&#xA;" LastActivityDate="2011-01-03T11:13:42.550" />
  <row Id="5909" PostTypeId="2" ParentId="5893" CreationDate="2011-01-02T20:43:28.110" Score="7" ViewCount="" Body="&lt;p&gt;This is not a serious answer, but I just remembered something from a book I read a year ago. There is a chapter in &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0060731338&quot; rel=&quot;nofollow&quot;&gt;Freakonomics&lt;/a&gt; devoted to what you can tell about a person from the name. The chapter is based on the author's research paper &lt;em&gt;The causes and consequences of distinctively black names&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think I've found an excerpt or summary of it in &lt;a href=&quot;http://www.slate.com/id/2116449/&quot; rel=&quot;nofollow&quot;&gt;this article&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The data show that, on average, a&#xA;  person with a distinctively black&#xA;  name—whether it is a woman named Imani&#xA;  or a man named DeShawn—does have a&#xA;  worse life outcome than a woman named&#xA;  Molly or a man named Jake. But it&#xA;  isn't the fault of his or her name. If&#xA;  two black boys, Jake Williams and&#xA;  DeShawn Williams, are born in the same&#xA;  neighborhood and into the same&#xA;  familial and economic circumstances,&#xA;  they would likely have similar life&#xA;  outcomes. But the kind of parents who&#xA;  name their son Jake don't tend to live&#xA;  in the same neighborhoods or share&#xA;  economic circumstances with the kind&#xA;  of parents who name their son DeShawn.&#xA;  And that's why, on average, a boy&#xA;  named Jake will tend to earn more&#xA;  money and get more education than a&#xA;  boy named DeShawn. DeShawn's name is&#xA;  an indicator—but not a cause—of his&#xA;  life path.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" LastActivityDate="2011-01-03T11:13:42.550" />
  <row Id="5910" PostTypeId="2" ParentId="5893" CreationDate="2011-01-03T01:47:42.210" Score="1" ViewCount="" Body="&lt;p&gt;You probably could find out:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Profession and possibly job history, if one participates in any professional discussions (current job usually can be found out from either domain name in email or signature, search would reveal past ones too)&lt;/li&gt;&#xA;&lt;li&gt;Relatives, if one maintains profile on social networks.&lt;/li&gt;&#xA;&lt;li&gt;Current location, at least up to the city. &lt;/li&gt;&#xA;&lt;li&gt;Ethnic background, if one has distinct name (i.e., somebody named &quot;Lubomir&quot; is probably connected to one of the Slavic European countries, etc.). &lt;/li&gt;&#xA;&lt;li&gt;Birth date from social networks - people tend to congratulate a person on or around his birth date, and if you're lucky you also get the year when one turns 25, 30, 35 etc. as one of the people congratulating would probably mention it if not the person in question. &lt;/li&gt;&#xA;&lt;li&gt;Educational background - from LinkedIn, etc.&lt;/li&gt;&#xA;&lt;li&gt;Hobbies, favorite sports teams, etc.&lt;/li&gt;&#xA;&lt;li&gt;If one is a pet lover, he'd probably have all his pets on the social networks too.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Which btw means you should &lt;em&gt;never ever&lt;/em&gt; use anything from the list above for your passwords, secret questions, etc.&lt;/p&gt;&#xA;" LastActivityDate="2011-01-03T11:13:42.550" CommentCount="2" />
  <row Id="5911" PostTypeId="2" ParentId="5893" CreationDate="2011-01-03T03:24:33.477" Score="2" ViewCount="" Body="&lt;p&gt;Just to add in to other suggestions here, one of the largest sources for family data is the raft of genealogy sites out there.  I think most western people are probably listed by some family member, distant or otherwise on a few of them and any such inclusion comes with a usually comprehensive family tree attached, complete with places, birth details, etc.  Very informative.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you cross match that data with friend graphs in Facebook, as people tend to add siblings/cousins (and parents/children on occasion), then use the locational data with electoral roles and directories, you can usually pinpoint people even with common names, and get a surprisingly large amount of data on them.&lt;/p&gt;&#xA;" LastActivityDate="2011-01-03T11:13:42.550" />
  <row Id="5912" PostTypeId="1" CreationDate="2011-01-03T13:13:27.213" Score="5" ViewCount="219" Body="&lt;p&gt;A recent question about &lt;a href=&quot;http://stats.stackexchange.com/questions/2234/alternatives-to-logistic-regression-in-r&quot;&gt;alternatives to logistic regression in R&lt;/a&gt; yielded a variety of answers including randomForest, gbm, rpart, bayesglm, and generalized additive models.  What are the practical and interpretation differences between these methods and logistic regression?  What assumptions do they make (or not make) relative to logistic regression?  Are the suitable for hypothesis testing?  Etc.&lt;/p&gt;&#xA;" OwnerUserId="196" LastActivityDate="2011-01-05T15:55:08.137" Title="What are the practical &amp; interpretation differences between alternatives and logistic regression?" Tags="&lt;r&gt;&lt;hypothesis-testing&gt;&lt;logistic-regression&gt;&lt;random-forest&gt;" AnswerCount="1" FavoriteCount="2" />
  <row Id="5913" PostTypeId="1" AcceptedAnswerId="5921" CreationDate="2011-01-03T13:17:49.960" Score="5" ViewCount="434" Body="&lt;p&gt;I’m writing some code &lt;em&gt;(JavaScript)&lt;/em&gt; to compare benchmark results. I’m using the &lt;a href=&quot;http://frank.mtsu.edu/~dkfuller/notes302/welcht.pdf&quot; rel=&quot;nofollow&quot;&gt;Welch T-test&lt;/a&gt; because the variance and/or sample size between benchmarks is most likely different. The critical value is pulled from a T-distribution table at 95% confidence &lt;em&gt;(two-sided)&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Welch formula is pretty straight-forward, but I am fuzzy on interpreting a significant result. I am not sure if the critical value should be divided by 2 or not. Help clearing that up is appreciated. Also should I be &lt;a href=&quot;https://developer.mozilla.org/en/JavaScript/Reference/Global_Objects/Math/round&quot; rel=&quot;nofollow&quot;&gt;rounding&lt;/a&gt; the degrees of freedom, &lt;code&gt;df&lt;/code&gt;, to lookup the critical value or would &lt;a href=&quot;https://developer.mozilla.org/en/JavaScript/Reference/Global_Objects/Math/ceil&quot; rel=&quot;nofollow&quot;&gt;&lt;code&gt;Math.ceil&lt;/code&gt;&lt;/a&gt; or &lt;a href=&quot;https://developer.mozilla.org/en/JavaScript/Reference/Global_Objects/Math/floor&quot; rel=&quot;nofollow&quot;&gt;&lt;code&gt;Math.floor&lt;/code&gt;&lt;/a&gt; be more appropriate?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  /**&#xA;   * Determines if the benchmark's hertz is higher than another.&#xA;   * @member Benchmark&#xA;   * @param {Object} other The benchmark to compare.&#xA;   * @returns {Number} Returns `1` if higher, `-1` if lower, and `0` if indeterminate.&#xA;   */&#xA;  function compare(other) {&#xA;    // use welch t-test&#xA;    // http://frank.mtsu.edu/~dkfuller/notes302/welcht.pdf&#xA;    // http://www.public.iastate.edu/~alicia/stat328/Regression%20inference-part2.pdf&#xA;    var a = this.stats,&#xA;        b = other.stats,&#xA;        pow = Math.pow,&#xA;        bitA = a.variance / a.size,&#xA;        bitB = b.variance / b.size,&#xA;        df = pow(bitA + bitB, 2) / ((pow(bitA, 2) / a.size - 1) + (pow(bitB, 2) / b.size - 1)),&#xA;        t = (a.mean - b.mean) / Math.sqrt(bitA + bitB),&#xA;        c = getCriticalValue(Math.round(df));&#xA;&#xA;    // check if t-statistic is significant&#xA;    return Math.abs(t) &amp;gt; c / 2 ? (t &amp;gt; 0 ? 1 : -1) : 0;&#xA;  }&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update:&lt;/strong&gt; Thanks for all the replies so far! My colleague &lt;a href=&quot;http://stats.stackexchange.com/questions/5913/interpreting-two-sided-two-sample-welch-t-test/5961#5961&quot;&gt;posted some more info here&lt;/a&gt;, in case that affects the advice.&lt;/p&gt;&#xA;" OwnerUserId="2616" LastEditorUserId="2616" LastEditorDisplayName="" LastEditDate="2011-01-04T13:18:52.173" LastActivityDate="2011-01-04T18:48:40.033" Title="Interpreting two-sided, two-sample, Welch T-Test" Tags="&lt;distributions&gt;&lt;t-test&gt;&lt;javascript&gt;" AnswerCount="4" FavoriteCount="1" />
  <row Id="5915" PostTypeId="2" ParentId="5913" CreationDate="2011-01-03T13:56:58.953" Score="6" ViewCount="" Body="&lt;p&gt;Dividing by 2 is for p-values. If you compare critical values the division by 2 is not necessary. The function &lt;code&gt;getCriticalValue&lt;/code&gt; should be the &lt;a href=&quot;http://en.wikipedia.org/wiki/Quantile_function&quot; rel=&quot;nofollow&quot;&gt;quantile function&lt;/a&gt; of Student's t distribution. Thus it should take 2 values, the probability and the degrees of freedom. If you want 2-sided hypothesis as indicated by your code, then you need 0.975 quantile. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the rounding, since the degrees of freedom are positive &lt;code&gt;Math.round&lt;/code&gt; looks good.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-01-03T13:56:58.953" CommentCount="3" />
  <row Id="5916" PostTypeId="2" ParentId="5913" CreationDate="2011-01-03T14:16:02.377" Score="6" ViewCount="" Body="&lt;p&gt;It's not absolutely necessary to round the degrees of freedom to an integer. &lt;a href=&quot;http://en.wikipedia.org/wiki/Student%27s_t-distribution&quot; rel=&quot;nofollow&quot;&gt;Student's &lt;em&gt;t&lt;/em&gt;-distribution&lt;/a&gt; can be defined for all positive real values of this parameter. Restricting it to a positive integer may make the critical value easier to calculate though, depending on how you're doing that. And it will make very little difference in practice with any reasonable sample sizes.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-01-03T14:16:02.377" CommentCount="7" />
  <row Id="5917" PostTypeId="2" ParentId="5899" CreationDate="2011-01-03T15:06:52.347" Score="2" ViewCount="" Body="&lt;p&gt;You're asking an intriguing question. I agree with the comments that are showing some apprehension at the &quot;one-man-one-vote&quot; system. I also agree that knowing the basic statistics (like standard deviation and mean) will not give you an insight into the will of the voters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to play off of David James's answer, keying in on stakeholders. Instead of a vote, perhaps you could give stakeholders a virtual account, which they must &quot;spend&quot; on the candidates.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If they had &lt;code&gt;$100&lt;/code&gt; each, then perhaps one stakeholder would show a strong preference for candidate A by spending all &lt;code&gt;$100&lt;/code&gt; on him or her. Another stakeholder might like candidate B slightly more than candidate A and spend &lt;code&gt;$60/$40&lt;/code&gt;. A third stakeholder might find all three candidates equally (un)appealing and spend &lt;code&gt;$33/$33/$34.&lt;/code&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A variation would be to give different stakeholders accounts of different sizes. For example, perhaps the exiting CEO gets &lt;code&gt;$200&lt;/code&gt; and a worker's representative gets &lt;code&gt;$150.&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could even ask for an open vote, where each stakeholder explains his reasoning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Highest earner wins the position. Or maybe the top two get the most careful look and a runoff.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This betting technique is an adaptation of what is done in &lt;em&gt;Blind Man's Bluff: The Untold Story of American Submarine Espionage&lt;/em&gt; (1998, S. Sontag and C.Drew) A B-52 bomber collided with an air tanker, and they lost an H-bomb.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Craven asked a group of submarine and salvage experts to place Las-Vegas-style bets on the probability of each of the different scenarios that might describe the bomb's loss.... Each scenario left the weapon in a different location.... He was relying on Bayes' theorem of subjective probability. (pp. 58-59)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Whatever you choose, please make sure that the rules are clear before you start voting on candidates. A perception that the rules changed will not help the transition.&lt;/p&gt;&#xA;" OwnerUserId="2591" LastActivityDate="2011-01-03T15:06:52.347" CommentCount="1" />
  <row Id="5918" PostTypeId="1" AcceptedAnswerId="5988" CreationDate="2011-01-03T15:08:29.897" Score="6" ViewCount="324" Body="&lt;p&gt;Note: Case is n&gt;&gt;p&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am reading Elements of Statistical Learning and there are various mentions about the &quot;right&quot; way to do cross validation( e.g. page 60, page 245). Specifically, my question is how to evaluate the final model (without a separate test set) using k-fold CV or bootstrapping when there has been a model search? It seems that in most cases (ML algorithms without embedded feature selection) there will be&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;A feature selection step&lt;/li&gt;&#xA;&lt;li&gt;A meta parameter selection step (e.g. the cost parameter in SVM).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;My Questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;I have seen that the feature selection step can be done where feature selection is done on the whole training set and held aside. Then, using k-fold CV, the feature selection algorithm is used in each fold (getting different features possibly chosen each time) and the error averaged. Then, you would use the features chosen using all the data (that were set aside) to train the final mode, but use the error from the cross validation as an estimate of future performance of the model. &lt;em&gt;IS THIS CORRECT?&lt;/em&gt;&lt;/li&gt;&#xA;&lt;li&gt;When you are using cross validation to select model parameters, then how to estimate model performance afterwards? &lt;em&gt;IS IT THE SAME PROCESS AS #1 ABOVE OR SHOULD YOU USE NESTED CV LIKE SHOWN ON PAGE 54 (&lt;a href=&quot;http://cseweb.ucsd.edu/~elkan/291/dm.pdf&quot; rel=&quot;nofollow&quot;&gt;pdf&lt;/a&gt;) OR SOMETHING ELSE?&lt;/em&gt;&lt;/li&gt;&#xA;&lt;li&gt;When you are doing both steps (feature and parameter setting).....then what do you do? complex nested loops?&lt;/li&gt;&#xA;&lt;li&gt;If you have a separate hold out sample, does the concern go away and you can use cross validation to select features and parameters (without worry since your performance estimate will come from a hold out set)?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Many thanks to anyone who can clear this up! Sorry for a wordy question but I wanted to make sure I explained the confusion points (many ;-))&#xA;Brian&lt;/p&gt;&#xA;" OwnerUserId="2040" LastActivityDate="2011-01-04T21:16:40.400" Title="Cross Validation (error generalization) after model selection" Tags="&lt;machine-learning&gt;&lt;model-selection&gt;&lt;data-mining&gt;&lt;cross-validation&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="2" />
  <row Id="5919" PostTypeId="2" ParentId="5903" CreationDate="2011-01-03T15:28:05.297" Score="0" ViewCount="" Body="&lt;p&gt;Now I'm confused. First you said that the posterior distribution of b|y under a flat prior is the same as the classical sampling distribution of b. Then you said the Bayesian CI is wider than the classical one. How could it be wider if the distributions are the same?&lt;/p&gt;&#xA;" OwnerUserId="2617" LastActivityDate="2011-01-03T15:28:05.297" CommentCount="3" />
  <row Id="5920" PostTypeId="2" ParentId="5912" CreationDate="2011-01-03T15:39:49.363" Score="5" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Disclaimer:&lt;/strong&gt; It is certainly far from being a full answer to the question!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think there are at least two levels to consider before establishing a distinction between all such methods:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;whether &lt;em&gt;a single model is fitted&lt;/em&gt; or not:&lt;/strong&gt; This helps opposing methods like logistic regression vs. RF or &lt;a href=&quot;http://en.wikipedia.org/wiki/Gradient_boosting&quot; rel=&quot;nofollow&quot;&gt;Gradient Boosting&lt;/a&gt; (or more generally &lt;a href=&quot;http://en.wikipedia.org/wiki/Ensemble_learning&quot; rel=&quot;nofollow&quot;&gt;Ensemble methods&lt;/a&gt;), and also put emphasis on parameters estimation (with associated asymptotic or bootstrap confidence intervals) vs. classification or prediction accuracy computation;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;whether &lt;em&gt;all variables are considered&lt;/em&gt; or not:&lt;/strong&gt; This is the basis of feature selection, in the sense that penalization or regularization allows to cope with &quot;irregular&quot; data sets (e.g., large $p$ and/or small $n$) and improve generalizability of the findings.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Here are few other points that I think are relevant to the question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In case we consider several models--the same model is fitted on different subsets (individuals and/or variables) of the available data, or different competitive models are fitted on the same data set--, &lt;a href=&quot;http://en.wikipedia.org/wiki/Cross-validation_%28statistics%29&quot; rel=&quot;nofollow&quot;&gt;cross-validation&lt;/a&gt; can be used to avoid overfitting and perform model or feature selection, although CV is not limited to this particular cases (it can be used with &lt;a href=&quot;http://en.wikipedia.org/wiki/Generalized_additive_model&quot; rel=&quot;nofollow&quot;&gt;GAMs&lt;/a&gt; or penalized GLMs, for instance). Also, there is the traditional interpretation issue: more complex models often implies more complex interpretation (more parameters, more stringent assumptions, etc.).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Gradient boosting and RFs overcome the limitations of a single decision tree, thanks to &lt;a href=&quot;http://en.wikipedia.org/wiki/Boosting&quot; rel=&quot;nofollow&quot;&gt;Boosting&lt;/a&gt; whose main idea is to combine the output of several weak learning algorithms in order to build a more accurate and stable decision rule, and &lt;a href=&quot;http://en.wikipedia.org/wiki/Bootstrap_aggregating&quot; rel=&quot;nofollow&quot;&gt;Bagging&lt;/a&gt; where we &quot;average&quot; results over resampled data sets. Altogether, they are often viewed as some kind of black boxes in comparison to more &quot;classical&quot; models where clear specifications for the model are provided (I can think of three classes of models: &lt;a href=&quot;http://en.wikipedia.org/wiki/Parametric_model&quot; rel=&quot;nofollow&quot;&gt;parameteric&lt;/a&gt;, &lt;a href=&quot;http://en.wikipedia.org/wiki/Semiparametric_model&quot; rel=&quot;nofollow&quot;&gt;semi-parametric&lt;/a&gt;, &lt;a href=&quot;http://en.wikipedia.org/wiki/Non-parametric_model&quot; rel=&quot;nofollow&quot;&gt;non-parametric&lt;/a&gt;), but I think the discussion held under this other thread &lt;a href=&quot;http://stats.stackexchange.com/questions/6/the-two-cultures-statistics-vs-machine-learning&quot;&gt;The Two Cultures: statistics vs. machine learning?&lt;/a&gt; provide interesting viewpoints.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are a couple of papers about feature selection and some ML techniques:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Saeys, Y, Inza, I, and Larrañaga, P. &lt;a href=&quot;http://bioinformatics.oxfordjournals.org/content/23/19/2507.abstract&quot; rel=&quot;nofollow&quot;&gt;A review of feature selection techniques in bioinformatics&lt;/a&gt;, Bioinformatics (2007) 23(19): 2507-2517.&lt;/li&gt;&#xA;&lt;li&gt;Dougherty, ER, Hua J, and Sima, C. &lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2766788/&quot; rel=&quot;nofollow&quot;&gt;Performance of Feature Selection Methods&lt;/a&gt;, Current Genomics (2009) 10(6): 365–374.&lt;/li&gt;&#xA;&lt;li&gt;Boulesteix, A-L and Strobl, C. &lt;a href=&quot;http://www.biomedcentral.com/1471-2288/9/85/abstract&quot; rel=&quot;nofollow&quot;&gt;Optimal classifier selection and negative bias in error rate estimation: an empirical study on high-dimensional prediction&lt;/a&gt;, BMC Medical Research Methodology (2009) 9:85.&lt;/li&gt;&#xA;&lt;li&gt;Caruana, R and Niculescu-Mizil, A. &lt;a href=&quot;http://www.cs.cornell.edu/~caruana/ctp/ct.papers/caruana.icml06.pdf&quot; rel=&quot;nofollow&quot;&gt;An Empirical Comparison of Supervised Learning Algorithms&lt;/a&gt;. Proceedings of the 23rd International Conference on Machine Learning (2006).&lt;/li&gt;&#xA;&lt;li&gt;Friedman, J, Hastie, T, and Tibshirani, R. &lt;a href=&quot;http://projecteuclid.org/DPubS?service=UI&amp;amp;version=1.0&amp;amp;verb=Display&amp;amp;handle=euclid.aos/1016218223&quot; rel=&quot;nofollow&quot;&gt;Additive logistic regression: A statistical view of boosting&lt;/a&gt;,  Ann. Statist. (2000) 28(2):337-407. (With discussion)&lt;/li&gt;&#xA;&lt;li&gt;Olden, JD, Lawler, JJ, and Poff, NL. &lt;a href=&quot;http://www.fish.washington.edu/research/oldenlab/pdf/2008/QuarterReviewBiol_2008.pdf&quot; rel=&quot;nofollow&quot;&gt;Machine learning methods without tears: a primer for ecologists&lt;/a&gt;, Q Rev Biol. (2008) 83(2):171-93.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;And of course, &lt;a href=&quot;http://www-stat.stanford.edu/~tibs/ElemStatLearn/&quot; rel=&quot;nofollow&quot;&gt;The Elements of Statistical Learning&lt;/a&gt;, by Hastie and coll., is full of illustrations and references. Also be sure to check the &lt;a href=&quot;http://www.autonlab.org/tutorials/&quot; rel=&quot;nofollow&quot;&gt;Statistical Data Mining Tutorials&lt;/a&gt;, from Andrew Moore.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-01-05T15:55:08.137" LastActivityDate="2011-01-05T15:55:08.137" />
  <row Id="5921" PostTypeId="2" ParentId="5913" CreationDate="2011-01-03T15:45:13.360" Score="8" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;(1a)&lt;/strong&gt; You don't need the Welch test to cope with different sample sizes.  That's automatically handled by the Student t-test.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;(1b)&lt;/strong&gt; If you think there's a real chance the variances in the two populations are strongly different, then you are assuming &lt;em&gt;a priori&lt;/em&gt; that the two populations differ.  It might not be a difference of location--that's what a t-test evaluates--but it's still an important difference nonetheless.  Don't paper it over by adopting a test that ignores this difference!  (Differences in variance often arise where one sample is &quot;contaminated&quot; with a few extreme results, simultaneously shifting the location and increasing the variance.  Because of the large variance it can be difficult to detect the shift in location (&lt;em&gt;no matter how great it is&lt;/em&gt;) in a small to medium size sample, because the increase in variance is roughly proportional to the &lt;em&gt;squared&lt;/em&gt; change in location.  This form of &quot;contamination&quot; occurs, for instance, when only a fraction of an experimental group responds to the treatment.)  Therefore you should consider a more appropriate test, such as a &lt;a href=&quot;http://books.google.com/books?id=u97pzxRjaCQC&amp;amp;pg=PA384&amp;amp;lpg=PA384&amp;amp;dq=statistics+slippage+test&amp;amp;source=bl&amp;amp;ots=ixvyigvJR3&amp;amp;sig=Z-cjW8Bbvae-I5Q8SCZfw0P3qqA&amp;amp;hl=en&amp;amp;ei=K-4hTd2nAcH68AbU0YCqDg&amp;amp;sa=X&amp;amp;oi=book_result&amp;amp;ct=result&amp;amp;resnum=8&amp;amp;ved=0CFAQ6AEwBw#v=onepage&amp;amp;q=statistics%20slippage%20test&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;slippage test&lt;/a&gt;.  Even better would be a less automated graphical approach using exploratory data analysis techniques.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;(2)&lt;/strong&gt; Use a &lt;a href=&quot;http://en.wikipedia.org/wiki/Two-tailed_test&quot; rel=&quot;nofollow&quot;&gt;two-sided test&lt;/a&gt; when a change of average in either direction (greater or lesser) is possible.  Otherwise, when you are testing only for an increase or decrease in average, use a one-sided test.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;(3)&lt;/strong&gt; Rounding would be incorrect and you shouldn't have to do it: most algorithms for computing t distributions don't care whether the DoF is an integer.  Rounding is not a big deal, but if you're using a t-test in the first place, you're concerned about small sample sizes (for otherwise the simpler &lt;a href=&quot;http://en.wikipedia.org/wiki/Z-test&quot; rel=&quot;nofollow&quot;&gt;z-test&lt;/a&gt; will work fine) and even small changes in DoF can matter a little.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-01-03T17:38:42.127" LastActivityDate="2011-01-03T17:38:42.127" CommentCount="1" />
  <row Id="5922" PostTypeId="1" AcceptedAnswerId="6045" CreationDate="2011-01-03T15:47:12.577" Score="6" ViewCount="138" Body="&lt;p&gt;A colleague in applied statistics sent me this:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&quot;I was wondering if you know any way&#xA;  to find out  the true dimension of the&#xA;  domain of a function.  For example, a&#xA;  circle is a one dimensional function &#xA;  in a two dimensional space. If I do&#xA;  not know how to draw,  is there a&#xA;  statistic that I can compute that&#xA;  tells me that  it is a one dimensional&#xA;  object in a two dimensional space?  I&#xA;  have to do this in high dimensional&#xA;  situations so cannot  draw pictures.&#xA;  Any help will be greatly appreciated.&quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The notion of dimension here is obviously ill-defined. I mean,&#xA;I could run a curve through any finite collection of points in &#xA;high dimensional space, and say that my data is one-dimensional.&#xA;But, depending on the configuration, there may be an easier or &#xA;more efficient way to describe the data as a higher dimensional set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Such issues must have been considered in the statistics literature,&#xA;but I'm not familiar with it. Any pointers or suggestions? Thanks!&lt;/p&gt;&#xA;" OwnerUserId="1157" LastActivityDate="2011-01-08T20:17:36.033" Title="Estimating the dimension of a data set" Tags="&lt;large-data&gt;" AnswerCount="3" />
  <row Id="5923" PostTypeId="2" ParentId="5903" CreationDate="2011-01-03T15:54:07.543" Score="4" ViewCount="" Body="&lt;p&gt;The 'problem' is in the prior on sigma.  Try a &lt;em&gt;less&lt;/em&gt; informative setting&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;tau ~ dgamma(1.0E-3,1.0E-3)&#xA;sigma &amp;lt;- pow(tau, -2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;in your jags file.  Then update a bunch&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;update(10000)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;grab the parameters, and summarise your quantity of interest.  It should line up reasonably well with the classic version. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Clarification: The updating is just to make sure you get where you're going whatever choice of prior you decide on, although chains for models like this one with diffuse priors and random starting values do take longer to converge.  In real problems you'd check convergence before summarising anything, but convergence is not the main issue in your example I don't think.&lt;/p&gt;&#xA;" OwnerUserId="1739" LastEditorUserId="1739" LastEditorDisplayName="" LastEditDate="2011-01-03T16:35:15.193" LastActivityDate="2011-01-03T16:35:15.193" CommentCount="1" />
  <row Id="5924" PostTypeId="2" ParentId="5922" CreationDate="2011-01-03T16:03:25.970" Score="6" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Principal_component_analysis&quot; rel=&quot;nofollow&quot;&gt;Principal Components Analysis&lt;/a&gt; of &lt;em&gt;local&lt;/em&gt; data is a good point of departure.  We have to take some care, though, to distinguish &lt;em&gt;local&lt;/em&gt; (intrinsic) from &lt;em&gt;global&lt;/em&gt; (extrinsic) dimension.  In the example of points on a circle, the local dimension is 1, but overall the points within the circle lie in a 2D space.  To apply PCA to this, the trick is to &lt;em&gt;localize&lt;/em&gt;: select one data point and extract only those that are close to it.  Apply PCA to this subset.  The number of large eigenvalues will suggest the intrinsic dimension.  Repeating this at other data points will indicate whether the data exhibit a constant intrinsic dimension throughout.  If so, each of the PCA results provides a partial &lt;a href=&quot;http://en.wikipedia.org/wiki/Differentiable_manifold#Atlases&quot; rel=&quot;nofollow&quot;&gt;atlas&lt;/a&gt; of the manifold.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-01-03T16:03:25.970" CommentCount="1" />
  <row Id="5925" PostTypeId="2" ParentId="5859" CreationDate="2011-01-03T16:07:14.337" Score="3" ViewCount="" Body="&lt;p&gt;The choice of the survival model should be guided by the underlying phenomenon. In this case it appears to be continuous, even if the data is collected in a somewhat discrete manner. A resolution of one month would be just fine over a 5-year period.&#xA;However, the large number of ties at 6 and 12 months makes one wonder wether you really have a 1-month precision (the ties at 0 are expected - that's a special value where relatively lot of deaths actually happen). I am not quite sure what you can do about that as this most likely reflects after-the-fact rounding rather than interval censoring.&lt;/p&gt;&#xA;" OwnerUserId="279" LastActivityDate="2011-01-03T16:07:14.337" />
  <row Id="5926" PostTypeId="1" CreationDate="2011-01-03T17:12:09.590" Score="7" ViewCount="306" Body="&lt;p&gt;The dependent variables in a MANOVA should not be &quot;too strongly correlated.&quot;  But how strong a correlation is too strong? It would be interesting to get people's opinions on this issue. For instance, would you proceed with MANOVA in the following situations?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;a) Y1 and Y2 are correlated with r=0.3 and p&amp;lt;0.005&lt;/p&gt;&#xA;&#xA;&lt;p&gt;b) Y1 and Y2 are correlated with r=0.7 and p=0.049&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some representative quotes in response to onestop:&lt;br&gt;&#xA;&lt;em&gt;&quot;MANOVA works well in situations where there are moderate correlations between DVs&quot; (course notes from San Francisco State Uni)&lt;br&gt;&#xA;&quot;The dependent variables are correlated which is appropriate for Manova&quot; (United States EPA Stats Primer)&lt;br&gt;&#xA;&quot;The dependent variables should be related conceptually, and they should&#xA;be correlated with one another at a low to moderate level.&quot; (Course notes from Northern Arizona University)&lt;br&gt;&#xA;&quot;DVs correlated from about .3 to about.7 are eligible&quot; (Maxwell 2001, Journal of Consumer Psychology)&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;n.b. I'm not referring to the assumption that the intercorrelation between Y1 and Y2 should be the same across all levels of independent variables, simply to this apparent grey area about the actual magnitude of the intercorrelation.&lt;/p&gt;&#xA;" OwnerUserId="266" LastEditorUserId="266" LastEditorDisplayName="" LastEditDate="2011-01-04T10:44:10.297" LastActivityDate="2011-04-05T18:33:24.237" Title="MANOVA and correlations between dependent variables: how strong is too strong?" Tags="&lt;correlation&gt;&lt;anova&gt;&lt;multivariate-analysis&gt;&lt;rule-of-thumb&gt;&lt;manova&gt;" AnswerCount="3" CommentCount="3" />
  <row Id="5927" PostTypeId="1" CreationDate="2011-01-03T18:01:37.370" Score="2" ViewCount="98" Body="&lt;p&gt;Are there meaningful ways to quantify how &quot;flat&quot; the log likelihood function is around the MLE when the parameter has more than one dimension? In particular is the determinant of the Hessian a reasonable measure?&lt;/p&gt;&#xA;" OwnerUserId="1004" LastActivityDate="2011-01-03T18:34:48.363" Title="A measure of the &quot;flatness&quot; of log likelihood at the MLE" Tags="&lt;maximum-likelihood&gt;" AnswerCount="1" CommentCount="5" />
  <row Id="5928" PostTypeId="2" ParentId="4445" CreationDate="2011-01-03T18:23:17.097" Score="0" ViewCount="" Body="&lt;p&gt;Whuber, would you be willing to provide an answer to your own question here? It seems that you have gained some insight into this problems since initially posing it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sorry to enter this as a question instead of a comment, but I don't have enough reputation to comment. &lt;/p&gt;&#xA;" OwnerUserId="82" LastActivityDate="2011-01-03T18:23:17.097" CommentCount="4" />
  <row Id="5929" PostTypeId="2" ParentId="5927" CreationDate="2011-01-03T18:25:09.920" Score="4" ViewCount="" Body="&lt;p&gt;You might find that the &lt;a href=&quot;http://en.wikipedia.org/wiki/Fisher_information&quot; rel=&quot;nofollow&quot;&gt;Fisher Information&lt;/a&gt; has some properties you like.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Its the expectation that gives Fisher Information its interpretation as the 'informativeness' of a measurement.  But if you're just looking for something geometrically descriptive then your suggestion seems like a reasonable one to me.&lt;/p&gt;&#xA;" OwnerUserId="1739" LastEditorUserId="1739" LastEditorDisplayName="" LastEditDate="2011-01-03T18:34:48.363" LastActivityDate="2011-01-03T18:34:48.363" />
  <row Id="5931" PostTypeId="2" ParentId="4519" CreationDate="2011-01-03T19:53:37.627" Score="4" ViewCount="" Body="&lt;p&gt;I may, after all this time, finally have understood the question.  The data, if I'm correct, are a set of tuples $(i, j, y(i,j))$ where $i$ is one player, $j \ne i$ is another player, and $y(i,j)$ is the number of attacks of $i$ on $j$.  In this notation the objective is to relate $y(i,j)$ to $y(j,i)$.  There are some natural ways to do this, including:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Analyze the data set $\{(y(i,j), y(j,i))\}$ by means of a scatterplot or PCA (to find the principal eigenvalue).  Note this is &lt;em&gt;not&lt;/em&gt; a regression situation because both components of each ordered pair are observed: neither can be considered under the control of an experimenter nor observed without error.  It is this scatterplot, I believe, that appears triangular.  This already suggests that any attempt to describe it succinctly, such as by means of a principal direction, is doomed.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Model $y(i,j)$ in terms of characteristics of $i$ and $j$.  This is a classic regression situation.  The solution provides an indirect, but possibly powerful, way to relate $y(i,j)$ to $y(j,i)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this case, also consider re-expressing the data in terms of &lt;em&gt;relative&lt;/em&gt; numbers of attacks.  That is, instead of using $y(i,j)$ use $x(i,j) = y(i,j)/\sum_{j}{y(i,j)}$.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-01-03T19:53:37.627" />
  <row Id="5932" PostTypeId="2" ParentId="4844" CreationDate="2011-01-03T20:11:46.050" Score="5" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;First&lt;/strong&gt;, it's almost impossible to drive a car &quot;randomly.&quot;  Did you periodically consult a random number generator to determine what direction to head in next?  I don't think so.  This calls into question the use of any statistical procedure that assumes randomness (even if it isn't simple and could lead to dependence).  &quot;Arbitrariness&quot; and &quot;randomness&quot; are fundamentally different: it's important not to confuse the two.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Second&lt;/strong&gt;, a confidence interval applies to inferences about a population or process.  In this case it seems to have something to do with temperature, but temperature where?  When?  Unless this is clearly defined you can develop no valid information relating your data to a target of inference.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Third&lt;/strong&gt;, independence is not absolutely necessary.  This problem sounds similar to standard environmental and &lt;a href=&quot;http://findarticles.com/p/articles/mi_m2120/is_n4_v76/ai_17087662/&quot; rel=&quot;nofollow&quot;&gt;ecological sampling&lt;/a&gt; plans where something is observed along a transect: animals are counted from an airplane, water temperature is logged from a boat, and so on.  When randomness is incorporated in the sampling plan (&quot;design based&quot;), you can use various &lt;a href=&quot;http://projecteuclid.org/DPubS/Repository/1.0/Disseminate?view=body&amp;amp;id=pdf_1&amp;amp;handle=euclid.lnms/1215458846&quot; rel=&quot;nofollow&quot;&gt;probability sampling&lt;/a&gt; estimators and their confidence intervals.  Otherwise, you need to use &lt;a href=&quot;http://en.wikipedia.org/wiki/Geostatistics&quot; rel=&quot;nofollow&quot;&gt;geostatistical&lt;/a&gt; methods (&quot;model based&quot;).  These view the temperatures as samples of a spatially continuous field of temperatures.  They attempt to deduce some characteristics of this field from the sampled temperatures, characteristics such as the type, extent, and direction of spatial correlation.  From this various inferences can be made about temperatures in unsampled locations.  Geostatistical methods exist even for these binary (thresholded) data, where they are known as &quot;&lt;a href=&quot;http://www.spatialanalysisonline.com/output/html/IndicatorKriging.html&quot; rel=&quot;nofollow&quot;&gt;indicator Kriging&lt;/a&gt;.&quot;  Such analyses intimately involve the relative geometries of the sample locations and the field to be estimated.  Thus, in this case, &lt;em&gt;an accurate record of the car's location at each sample is needed&lt;/em&gt;.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-01-03T20:11:46.050" />
  <row Id="5933" PostTypeId="2" ParentId="5893" CreationDate="2011-01-03T20:34:28.793" Score="1" ViewCount="" Body="&lt;p&gt;The last chapter of &lt;em&gt;Freakonomics&lt;/em&gt; (2005, Steven D. Levitt and Stephen J. Dubner) has a fascinating discussion about names, particularly as they relate to socio-economic status and race.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;They have a list of first names that might or might not correlate well with FB's analysis of last names. They also describe how name choice is changing diachronically (across time).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Who knows -- the parents' selection name might be more accurate than what people report on the census. &lt;/p&gt;&#xA;" OwnerUserId="2591" LastActivityDate="2011-01-03T20:34:28.793" />
  <row Id="5934" PostTypeId="1" AcceptedAnswerId="5942" CreationDate="2011-01-03T21:54:58.037" Score="0" ViewCount="348" Body="&lt;p&gt;Over the holidays I played a dice game where each player had 3 to 7 d6 to roll each turn.  The game gave certain advantages to doubles and triples.  I wanted to know the odds of rolling doubles or triples given N dice to understand the importance of &quot;upgrading&quot; to more dice.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is really about how to reason about such problems:  It's easy to see that the odds of rolling doubles with 2 d6 are 1/6 (first die can be anything, and 1/6 of the time the second will match).  It's also easy to see that with 7 dice the odds are 1.0 (6 dice could be 1..6, but the 7th must match one of them).  The middle cases are fuzzier to me.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I ended up writing a program to enumerate all of the possibilities (even 6^7 is only 279,936 cases to evaluate) because that was the only way I felt I could verify any closed formula I came up with.  This also enabled me to distinguish cases of multiple doubles (or triples).  I would like to know how to derive closed forms and how to validate such solutions without brute-force simulation.&lt;/p&gt;&#xA;" OwnerUserId="2622" LastActivityDate="2011-01-04T01:04:18.533" Title="Odds of rolling doubles or triples given N 6-sided dice" Tags="&lt;probability&gt;&lt;games&gt;&lt;dice&gt;" AnswerCount="3" />
  <row Id="5935" PostTypeId="1" AcceptedAnswerId="5936" CreationDate="2011-01-03T22:04:01.343" Score="4" ViewCount="267" Body="&lt;p&gt;I am analyzing an experimental data set. The data consists of a paired vector of treatment type and a binomial outcome:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Treatment    Outcome&#xA;A            1&#xA;B            0&#xA;C            0&#xA;D            1&#xA;A            0&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In the outcome column, 1 denotes a success and 0 denotes a failure. I'd like to figure out if the treatment significantly varies the outcome. There are 4 different treatments with each experiment repeated a large number of times (2000 for each treatment).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is, can I analyze the binary outcome using ANOVA? Or should I be using a chi-square test to check the binomial data? It seems like chi-square assumes the proportion would be be evenly split, which isn't the case. Another idea would be to summarize the data using the proportion of successes versus failures for each treatment and then to use a proportion test.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm curious to hear your recommendations for tests that make sense for these sorts of binomial success/failure experiments.&lt;/p&gt;&#xA;" OwnerUserId="2624" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-01-03T22:24:48.337" LastActivityDate="2011-01-03T23:47:35.070" Title="ANOVA on binomial data" Tags="&lt;anova&gt;&lt;experiment-design&gt;&lt;binomial&gt;" AnswerCount="2" />
  <row Id="5936" PostTypeId="2" ParentId="5935" CreationDate="2011-01-03T22:21:28.357" Score="5" ViewCount="" Body="&lt;p&gt;No to ANOVA, which assumes a normally distributed outcome variable (among other things). There are &quot;old school&quot; transformations to consider, but I would prefer logistic regression (equivalent to a chi square when there is only one independent variable, as in your case). The advantage of using logistic regression over a chi square test is that you can easily use a linear contrast to compare specific levels of the treatment if you find a significant result to the overall test (type 3). For example A versus B, B versus C etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Update Added for clarity:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Taking data at hand (the post doc data set from &lt;a href=&quot;http://ftp.sas.com/samples/A55770&quot; rel=&quot;nofollow&quot;&gt;Allison&lt;/a&gt;) and using the variable cits as follows, this was my point:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;       postdocData$citsBin&amp;lt;-ifelse(postdocData$cits &amp;gt;2, 3, postdocData$cits)&#xA;       postdocData$citsBin&amp;lt;-as.factor(postdocData$citsBin)&#xA;       ordered(postdocData$citsBin, levels = c(&quot;0&quot;, &quot;1&quot;, &quot;2&quot;,&quot;3&quot;))&#xA;       contrasts(postdocData$citsBin) &amp;lt;- contr.treatment(4, base = 4) #set the 4th level as the reference&#xA;       contrasts(postdocData$citsBin)&#xA;&#xA;        #  1 2 3&#xA;        #0 1 0 0&#xA;        #1 0 1 0&#xA;        #2 0 0 1&#xA;        #3 0 0 0&#xA;&#xA;    #fit the univariate logistic regression model&#xA;    model.1&amp;lt;-glm(pdoc~citsBin, data=postdocData, family=binomial(link=&quot;logit&quot;))&#xA;&#xA;    library(car) #John Fox package&#xA;    #type 3 analysis (SAS verbiage)&#xA;    car::Anova(model.1,test=&quot;LR&quot;, type=&quot;III&quot;) &#xA;                  #Response: pdoc&#xA;                  #        LR Chisq Df Pr(&amp;gt;Chisq)&#xA;                  #citsBin   1.7977  3     0.6154&#xA;&#xA;&#xA;chisq.test(table(postdocData$citsBin,postdocData$pdoc)) #X-squared = 1.7957, df = 3, p-alue= 0.6159&#xA;&#xA;#then can test differences in levels, such as&#xA;&#xA;#contrast cits =0 minus cits=1 = 0&#xA;# Ho: Beta_1 - Beta_2 =0&#xA;    cVec&amp;lt;-c(0,1,-1,0)&#xA;    car::linearHypothesis(model.1,cVec, verbose=TRUE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2040" LastEditorUserId="2040" LastEditorDisplayName="" LastEditDate="2011-01-03T23:47:35.070" LastActivityDate="2011-01-03T23:47:35.070" CommentCount="2" />
  <row Id="5937" PostTypeId="1" AcceptedAnswerId="5940" CreationDate="2011-01-03T22:23:41.990" Score="10" ViewCount="237" Body="&lt;p&gt;When teaching an introductory level class, the teachers I know tend to invent some numbers and a story in order to exemplify the method they are teaching. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I would prefer is to tell a real story with real numbers.  However, these stories needs to relate to a very tiny dataset, which enables manual calculations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestions for such datasets will be very welcomed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some sample topics for the tiny datasets:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;correlation/regression (basic)&lt;/li&gt;&#xA;&lt;li&gt;ANOVA (1/2 ways)&lt;/li&gt;&#xA;&lt;li&gt;z/t tests - one/two un/paired samples&lt;/li&gt;&#xA;&lt;li&gt;comparisons of proportions - two/multi way tables&lt;/li&gt;&#xA;&lt;li&gt;So on...&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="253" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2011-01-03T23:01:01.980" LastActivityDate="2011-01-04T04:04:56.267" CommunityOwnedDate="2011-01-03T23:01:01.980" Title="Tiny (real) datasets for giving examples in class?" Tags="&lt;dataset&gt;&lt;teaching&gt;&lt;big-list&gt;" AnswerCount="5" CommentCount="2" FavoriteCount="2" />
  <row Id="5938" PostTypeId="2" ParentId="5937" CreationDate="2011-01-03T22:58:26.610" Score="7" ViewCount="" Body="&lt;p&gt;For two-way tables, I like the data on gender and survival of the titanic passengers:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;       | Alive  Dead | Total&#xA;-------+-------------+------&#xA;Female | 308    154  |  462&#xA;Male   | 142    709  |  851&#xA;-------+-------------+------&#xA;Total  | 450    863  | 1313&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;With this data, one can discuss things like the chi-square test for independence and measure of assocation, such as the relative rate and the odds ratio. For example, female passengers were ~4 times more likely to survive than male passengers. At the same time, male passengers were ~2.5 times more likely to die than female passengers. The odds ratio for survival/dying is always 10 though.&lt;/p&gt;&#xA;" OwnerUserId="1934" LastActivityDate="2011-01-03T22:58:26.610" CommentCount="1" />
  <row Id="5939" PostTypeId="2" ParentId="1164" CreationDate="2011-01-03T23:00:01.527" Score="4" ViewCount="" Body="&lt;p&gt;Wooldridge &quot;Introductory Econometrics - A Modern Approach&quot; 2E p.261.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If Heteroskedasticity-robust standard errors are valid more often than the usual OLS standard errors, why do we bother we the usual standard errors at all?...One reason they are still used in cross sectional work is that, if the homoskedasticity assumption holds and the erros are normally distributed, then the usual t-statistics have exact t distributions, regardless of the sample size. The robust standard errors and robust t statistics are justified only as the sample size becomes large. With small sample sizes, the robust t statistics can have distributions that are not very close to the t distribution, and that could throw off our inference. In large sample sizes, we can make a case for always reporting only the Heteroskedasticity-robust standard errors in cross-sectional applications, and this practice is being followed more and more in applied work.&lt;/p&gt;&#xA;" OwnerUserId="2625" LastActivityDate="2011-01-03T23:00:01.527" />
  <row Id="5940" PostTypeId="2" ParentId="5937" CreationDate="2011-01-03T23:03:39.257" Score="9" ViewCount="" Body="&lt;p&gt;The &lt;a href=&quot;http://lib.stat.cmu.edu/DASL/&quot; rel=&quot;nofollow&quot;&gt;data and story library&lt;/a&gt; seems to have what you need. You can search it for particular data sets.&lt;/p&gt;&#xA;" OwnerUserId="1381" LastActivityDate="2011-01-03T23:03:39.257" CommunityOwnedDate="2011-01-03T23:03:39.257" CommentCount="1" />
  <row Id="5941" PostTypeId="2" ParentId="5937" CreationDate="2011-01-03T23:07:43.137" Score="4" ViewCount="" Body="&lt;p&gt;The &lt;a href=&quot;http://www.amstat.org/publications/jse/jse_data_archive.htm&quot; rel=&quot;nofollow&quot;&gt;Journal of Statistical Education&lt;/a&gt; has an archive of educational data sets.&lt;/p&gt;&#xA;" OwnerUserId="1381" LastActivityDate="2011-01-03T23:07:43.137" CommunityOwnedDate="2011-01-03T23:07:43.137" />
  <row Id="5942" PostTypeId="2" ParentId="5934" CreationDate="2011-01-03T23:12:30.220" Score="3" ViewCount="" Body="&lt;p&gt;You are looking at a much more manageable version of the famous &lt;a href=&quot;http://en.wikipedia.org/wiki/Birthday_problem&quot; rel=&quot;nofollow&quot;&gt;birthday problem&lt;/a&gt;, which has 365 choices for each &quot;die&quot;. The link describes the solution as well. In short, the trick for counting duplicates is to rather counting the cases without a duplicate - that is much simpler.&lt;/p&gt;&#xA;" OwnerUserId="279" LastActivityDate="2011-01-03T23:12:30.220" CommentCount="3" />
  <row Id="5943" PostTypeId="2" ParentId="5937" CreationDate="2011-01-03T23:15:30.717" Score="13" ViewCount="" Body="&lt;p&gt;There's a book called &quot;A Handbook of Small Datasets&quot; by D.J. Hand, F. Daly, A.D. Lunn, K.J. McConway and E. Ostrowski.  The Statistics department at NCSU have electronically posted the datasets from this book &lt;a href=&quot;http://www.stat.ncsu.edu/sas/sicl/data/&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The website above gives only the data; you would need to read the book to get the story behind the numbers, that is, any story beyond what you can glean from the data set's title.  But, they are &lt;em&gt;small&lt;/em&gt;, and they are &lt;em&gt;real&lt;/em&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1108" LastActivityDate="2011-01-03T23:15:30.717" CommunityOwnedDate="2011-01-03T23:15:30.717" CommentCount="1" />
  <row Id="5944" PostTypeId="2" ParentId="5935" CreationDate="2011-01-03T23:27:03.747" Score="2" ViewCount="" Body="&lt;p&gt;Hello Speciousfool:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to differ from what you think about Chi-Sq test. It is applicable even if the data is not binomial. It's based on the asymptotic normality of mle (in most of the cases). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would do a logistic regression like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\log \frac {\hat{\pi}} {1-\hat{\pi}} = \beta_0 + \beta_1 \times D_1 + \beta_2 \times D_2$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$D_1$ and $D_2$ are dummy variables. $D_1 = D_2 = 0 \implies A, D_1 = 1, D_2 = 0 \implies B, D_1 = 1 D_2 = 1 \implies C$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$H_o : \beta_0 = \beta_1 = \beta_2 = 0$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is the ANOVA equivalent if there is a relation or not. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$H_o : \beta_0 = 0$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is the test is A has some effect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$H_o : \beta_1 - \beta_0 = 0$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is the test is B has some effect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$H_o : \beta_2 - (\frac {\beta_0+\beta_1} {2}) = 0$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is the test is C has some effect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now you can do further contrasts to find our what you are interested in. It is still a chi-sq test, but with different degrees of freedom (3, 1, 1, and 1, respectively)&lt;/p&gt;&#xA;" OwnerUserId="1307" LastEditorUserId="1307" LastEditorDisplayName="" LastEditDate="2011-01-03T23:40:57.383" LastActivityDate="2011-01-03T23:40:57.383" CommentCount="1" />
  <row Id="5945" PostTypeId="1" AcceptedAnswerId="5950" CreationDate="2011-01-03T23:31:11.083" Score="4" ViewCount="194" Body="&lt;p&gt;SVMs for classification make intuitive sense to me: I understand how minimizing $||\theta||^2$ yields the maximum margin. However, I don't understand that objective in the context of regression. Various texts (&lt;a href=&quot;http://kernelsvm.tripod.com/&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.99.2073&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;) describe this as maximizing &quot;flatness.&quot; Why would we want to do that? What in regression is the equivalent to the concept of &quot;margin&quot;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://agbs.kyb.tuebingen.mpg.de/km/bb/showthread.php?tid=991&quot; rel=&quot;nofollow&quot;&gt;Here&lt;/a&gt; are a few attempted answers, but none that really helped my understanding.&lt;/p&gt;&#xA;" OwnerUserId="1720" LastActivityDate="2011-01-14T17:48:48.753" Title="Understanding SVM regression: objective function and &quot;flatness&quot;" Tags="&lt;regression&gt;&lt;svm&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="5946" PostTypeId="2" ParentId="5680" CreationDate="2011-01-04T00:00:42.387" Score="3" ViewCount="" Body="&lt;p&gt;Like other parametric tests, the analysis of variance assumes that the data fit the normal distribution. If your measurement variable is not normally distributed, you may be increasing your chance of a false positive result if you analyze the data with an anova or other test that assumes normality. Fortunately, an anova is not very sensitive to moderate deviations from normality; simulation studies, using a variety of non-normal distributions, have shown that the false positive rate is not affected very much by this violation of the assumption (Glass et al. 1972, Harwell et al. 1992, Lix et al. 1996). This is because when you take a large number of random samples from a population, the means of those samples are approximately normally distributed even when the population is not normal.&#xA;It is possible to test the goodness-of-fit of a data set to the normal distribution. I do not suggest that you do this, because many data sets that are significantly non-normal would be perfectly appropriate for an anova.&#xA;Instead, if you have a large enough data set, I suggest you just look at the frequency histogram. If it looks more-or-less normal, go ahead and perform an anova. If it looks like a normal distribution that has been pushed to one side, like the sulphate data above, you should try different data transformations and see if any of them make the histogram look more normal. If that doesn't work, and the data still look severely non-normal, it's probably still okay to analyze the data using an anova. However, you may want to analyze it using a non-parametric test. Just about every parametric statistical test has a non-parametric substitute, such as the Kruskal–Wallis test instead of a one-way anova, Wilcoxon signed-rank test instead of a paired t-test, and Spearman rank correlation instead of linear regression. These non-parametric tests do not assume that the data fit the normal distribution. They do assume that the data in different groups have the same distribution as each other, however; if different groups have different shaped distributions (for example, one is skewed to the left, another is skewed to the right), a non-parametric test may not be any better than a parametric one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;References&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Glass, G.V., P.D. Peckham, and J.R. Sanders. 1972. Consequences of failure to meet assumptions underlying fixed effects analyses of variance and covariance. Rev. Educ. Res. 42: 237-288.&lt;/li&gt;&#xA;&lt;li&gt;Harwell, M.R., E.N. Rubinstein, W.S. Hayes, and C.C. Olds. 1992. Summarizing Monte Carlo results in methodological research: the one- and two-factor fixed effects ANOVA cases. J. Educ. Stat. 17: 315-339.&lt;/li&gt;&#xA;&lt;li&gt;Lix, L.M., J.C. Keselman, and H.J. Keselman. 1996. Consequences of assumption violations revisited: A quantitative review of alternatives to the one-way analysis of variance F test. Rev. Educ. Res. 66: 579-619.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="2627" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-01-04T19:19:39.637" LastActivityDate="2011-01-04T19:19:39.637" CommentCount="1" />
  <row Id="5947" PostTypeId="2" ParentId="5934" CreationDate="2011-01-04T00:35:45.310" Score="3" ViewCount="" Body="&lt;p&gt;You're really asking for instruction in &lt;a href=&quot;http://en.wikipedia.org/wiki/Combinatorics&quot; rel=&quot;nofollow&quot;&gt;combinatorics&lt;/a&gt;, which is a vast field.  For this particular problem, a good method, if somewhat abstract, is to use &lt;a href=&quot;http://en.wikipedia.org/wiki/Probability-generating_function&quot; rel=&quot;nofollow&quot;&gt;generating functions&lt;/a&gt;.  The probability generating function (pgf) for a d$n$, where $n = 1, 2, \ldots$, is obtained by averaging $n$ variables.  Their names don't matter, so let's call them $x_1, x_2, \ldots, x_n$.  The pgf for $k$ rolls is just the $k^{\text{th}}$ power of the pgf for 1 roll.  So, if you expand&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\left(\frac{x_1 + x_2 + \cdots + x_n}{n}\right)^k$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;you can read the answer to any question by inspecting the coefficients.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, to find the probability of rolling a full house (three of one number, two of another) in Yahtzee ($n=6, k=5$) look at all the terms in $((x_1 + \cdots + x_6)/6)^5$ involving a third power and a second power.  There are 15 of them, each with the coefficient 5/3888, and each counts distinctly different outcomes, so the chance of a full house is the sum of these chances, equal to 15*5/3888.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's straightforward to write code that multiplies and adds polynomials of this sort.  The number of calculations for seven rolls of your d6 is just a few thousand, far smaller than the hundreds of thousands needed to enumerate and tally all possible combinations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Mathematical analysis of generating functions often gives closed-form formulas for the coefficients.  Many examples are available; you can check some of them out on the &lt;a href=&quot;http://math.stackexchange.com/questions/tagged/generating-functions&quot;&gt;math site&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-01-04T00:35:45.310" CommentCount="1" />
  <row Id="5948" PostTypeId="2" ParentId="5934" CreationDate="2011-01-04T01:04:18.533" Score="3" ViewCount="" Body="&lt;p&gt;You have posed the kind of combinatorics question that profs will ask you as an undergrad.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's reduce the problem to doubles of two dice. Instead of asking about doubles, let's ask how &lt;em&gt;not&lt;/em&gt; to roll doubles. There are 6 ways to roll the first die and 5 ways (excluding the first die's roll) to roll the second die. Because we are &lt;em&gt;not&lt;/em&gt; rolling doubles, we could pair a 1 like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1 2&#xA;1 3&#xA;1 4&#xA;1 5&#xA;1 6&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You could continue this pattern for 6 x 5 or 30 non-doubles. There are 6 * 6 = 36 ways to roll a pair of dice, so 36 - (6 * 5) gives you 6 ways to roll doubles.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That was trivial, but let's build up to 3 dice.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How many ways are there to &lt;em&gt;not&lt;/em&gt; roll 3 dice that are the same? The answer is 6 * 5 * 4 = 120, because there are 6 ways to roll the first die, 5 ways to roll the second (and not duplicate the first), and 4 ways to roll the third die (and not duplicate the first or the second).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are the ways to not roll a double or triple with three dice, when the first die is a 1.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1   2   3&#xA;1   2   4&#xA;1   2   5&#xA;1   2   6&#xA;1   3   2&#xA;1   3   4&#xA;1   3   5&#xA;1   3   6&#xA;1   4   2&#xA;1   4   3&#xA;1   4   5&#xA;1   4   6&#xA;1   5   2&#xA;1   5   3&#xA;1   5   4&#xA;1   5   6&#xA;1   6   2&#xA;1   6   3&#xA;1   6   4&#xA;1   6   5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;There are 6 * 6 * 6 = 216 ways to roll three dice, and 6 * 5 * 4 ways to roll non-duplicates, so there are 216 - 120 = 96 ways to roll doubles OR triples. &lt;/p&gt;&#xA;" OwnerUserId="2591" LastActivityDate="2011-01-04T01:04:18.533" CommentCount="3" />
  <row Id="5949" PostTypeId="2" ParentId="5937" CreationDate="2011-01-04T04:04:56.267" Score="2" ViewCount="" Body="&lt;p&gt;Probably such an obvious answer that it does not really need to be mentioned, but for correlation or linear regression &lt;a href=&quot;http://en.wikipedia.org/wiki/Anscombe%27s_quartet&quot; rel=&quot;nofollow&quot;&gt;Anscombe's quartet&lt;/a&gt; is a logical choice. Although it is not a real story with real data I think it is such a simple example it would reasonably fit into your criteria.&lt;/p&gt;&#xA;" OwnerUserId="1036" LastActivityDate="2011-01-04T04:04:56.267" CommunityOwnedDate="2011-01-04T04:04:56.267" CommentCount="1" />
  <row Id="5950" PostTypeId="2" ParentId="5945" CreationDate="2011-01-04T05:22:41.427" Score="5" ViewCount="" Body="&lt;p&gt;One way that I think about the flatness is that it makes my predictions less sensitive to perturbations in the features. That is, if I am constructing a model of the form&#xA;$$y = x^\top \theta + \epsilon,$$&#xA;where my feature vector $x$ has already been normalized, then smaller values in $\theta$ mean my model is less sensitive to errors in measurement/random shocks/non-stationarity of the features, $x$. Given two models (&lt;em&gt;i.e.&lt;/em&gt; two possible values of $\theta$) which explain the data equally well, I prefer the 'flatter' one. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also think of Ridge Regression as peforming the same thing without the kernel trick or the SVM 'tube' regression formulation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;edit&lt;/strong&gt;: In response to @Yang's comments, some more explanation:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Consider the linear case: $y = x^\top \theta + \epsilon$. Suppose the $x$ are drawn i.i.d. from some distribution, independent of $\theta$. By the dot product identity, we have $y = ||x|| ||\theta|| \cos\psi + \epsilon$, where $\psi$ is the angle between $\theta$ and $x$, which is probably distributed under some spherically uniform distribution. Now note: the 'spread' (&lt;em&gt;e.g.&lt;/em&gt; the sample standard deviation) of our predictions of $y$ is proportional to $||\theta||$. To get good MSE with the latent, noiseless versions of our observations, we want to shrink that $||\theta||$. &lt;em&gt;c.f.&lt;/em&gt; &lt;a href=&quot;http://en.wikipedia.org/wiki/James%E2%80%93Stein_estimator&quot; rel=&quot;nofollow&quot;&gt;James Stein estimator&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;li&gt;Consider the linear case with lots of features. Consider the models $y = x^\top \theta_1 + \epsilon$, and $y = x^\top \theta_2 + \epsilon$. If $\theta_1$ has more zero elements in it than $\theta_2$, but about the same explanatory power, we would prefer it, base on Occam's razor, since it has dependencies on fewer variables (&lt;em&gt;i.e.&lt;/em&gt; we have 'done feature selection' by setting some elements of $\theta_1$ to zero). Flatness is kind of a continuous version of this argument. If each marginal of $x$ has unit standard deviation, and $\theta_1$ has &lt;em&gt;e.g.&lt;/em&gt; 2 elements which are 10, and the remaining $n-2$ are smaller than 0.0001, depending on your tolerance of noise, this is effectively 'selecting' the two features, and zeroing out the remaining ones.&lt;/li&gt;&#xA;&lt;li&gt;When the kernel trick is employed, you are performing a linear regression in an high (sometimes infinite) dimensional vector space. Each element of $\theta$ now corresponds to one of your &lt;em&gt;samples&lt;/em&gt;, not your &lt;em&gt;features&lt;/em&gt;. If $k$ elements of $\theta$ are non-zero, and the remaining $m-k$ are zero, the features corresponding to the $k$ non-zero elements of $\theta$ are called your 'support vectors'. To store your SVM model, say on disk, you need only keep those $k$ feature vectors, and you can throw the rest of them away. Now flatness &lt;em&gt;really&lt;/em&gt; matters, because having $k$ small reduces storage and transmission, &lt;em&gt;etc&lt;/em&gt;, requirements. Again, depending on your tolerance for noise, you can probably zero out all elements of $\theta$ but the $l$ largest, for some $l$, after performing an SVM regression. Flatness here is equivalent to parsimony with respect to the number of support vectors.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="795" LastEditorUserId="795" LastEditorDisplayName="" LastEditDate="2011-01-14T17:48:48.753" LastActivityDate="2011-01-14T17:48:48.753" CommentCount="8" />
  <row Id="5951" PostTypeId="2" ParentId="5922" CreationDate="2011-01-04T05:28:07.677" Score="4" ViewCount="" Body="&lt;p&gt;I'm not sure about the 'domain of a function' part, but &lt;a href=&quot;http://en.wikipedia.org/wiki/Hausdorff_dimension&quot; rel=&quot;nofollow&quot;&gt;Hausdorff Dimension&lt;/a&gt; seems to answer this question. It has the odd property of agreeing with simple examples (&lt;em&gt;e.g.&lt;/em&gt; the circle has Hausdorff Dimension 1), but of giving non-integral results for some sets ('fractals').&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2011-01-04T05:28:07.677" CommentCount="2" />
  <row Id="5952" PostTypeId="1" AcceptedAnswerId="5956" CreationDate="2011-01-04T08:14:55.153" Score="8" ViewCount="361" Body="&lt;p&gt;Is there a way of plotting the regression line of a piecewise model like this, other than using &lt;code&gt;lines&lt;/code&gt; to plot each segment separately, or using &lt;code&gt;geom_smooth(aes(group=Ind), method=&quot;lm&quot;, fill=FALSE)&lt;/code&gt; ?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;m.sqft &amp;lt;- mean(sqft)&#xA;model &amp;lt;- lm(price~sqft+I((sqft-m.sqft)*Ind))&#xA;# sqft, price: continuous variables, Ind: if sqft&amp;gt;mean(sqft) then 1 else 0&#xA;&#xA;plot(sqft,price)&#xA;abline(reg = model)&#xA;Warning message:&#xA;In abline(reg = model) :&#xA;  only using the first two of 3regression coefficients&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Thank you.&lt;/p&gt;&#xA;" OwnerUserId="339" LastEditorUserId="339" LastEditorDisplayName="" LastEditDate="2011-01-05T10:31:19.180" LastActivityDate="2011-01-05T10:31:19.180" Title="Plotting a piecewise regression line" Tags="&lt;r&gt;&lt;data-visualization&gt;&lt;regression&gt;" AnswerCount="1" FavoriteCount="2" />
  <row Id="5954" PostTypeId="1" CreationDate="2011-01-04T09:29:01.153" Score="17" ViewCount="355" Body="&lt;p&gt;How can a regression model be any use if you don't know the function you are trying to get the parameters for?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I saw a piece of research that said that mothers who breast fed their children were less likely to suffer diabetes in later life. The research was from a survey of some 1000 mothers and controlled for miscellaneous factors and a loglinear model was used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now does this mean that they reckon all the factors that determine the likelihood of diabetes fit in a nice function (exponential presumably) that translates neatly into a linear model with logs and that whether the woman breast fed turned out to be statistically significant?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm missing something I'm sure but, how the hell do they know the model? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="2629" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-01-04T16:47:00.697" LastActivityDate="2011-01-06T14:09:59.477" Title="Understanding regressions - the role of the model" Tags="&lt;regression&gt;&lt;epidemiology&gt;" AnswerCount="5" CommentCount="1" FavoriteCount="4" />
  <row Id="5955" PostTypeId="2" ParentId="5954" CreationDate="2011-01-04T09:45:18.260" Score="21" ViewCount="" Body="&lt;p&gt;It helps to view regression as a linear approximation of the true form. Suppose the true relationship is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$y=f(x_1,...,x_k)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;with $x_1,...,x_k$ factors explaining the $y$.  Then first order Taylor approximation of $f$ around zero is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$f(x_1,...,x_k)=f(0,...,0)+\sum_{i=1}^{k}\frac{\partial f(0)}{\partial x_k}x_k+\varepsilon,$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\varepsilon$ is the approximation error. Now denote $\alpha_0=f(0,...,0)$ and $\alpha_k=\frac{\partial{f}(0)}{\partial x_k}$ and you have a regression:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$y=\alpha_0+\alpha_1 x_1+...+\alpha_k x_k + \varepsilon$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So although you do not know the true relationship, if $\varepsilon$ is small you get approximation, from which you can still deduce useful conclusions.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-01-04T09:45:18.260" CommentCount="1" />
  <row Id="5956" PostTypeId="2" ParentId="5952" CreationDate="2011-01-04T09:58:53.000" Score="6" ViewCount="" Body="&lt;p&gt;The only way I know how to do this easily is to predict from the model across the range of &lt;code&gt;sqft&lt;/code&gt; and plot the predictions. There isn't a general way with &lt;code&gt;abline&lt;/code&gt; or similar. You might also take a look at the &lt;strong&gt;&lt;a href=&quot;http://cran.r-project.org/web/packages/segmented/index.html&quot; rel=&quot;nofollow&quot;&gt;segmented&lt;/a&gt;&lt;/strong&gt; package which will fit these models and provide the plotting infrastructure for you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Doing this via predictions and base graphics. First, some dummy data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(1)&#xA;sqft &amp;lt;- runif(100)&#xA;sqft &amp;lt;- ifelse((tmp &amp;lt;- sqft &amp;gt; mean(sqft)), 1, 0) + rnorm(100, sd = 0.5)&#xA;price &amp;lt;- 2 + 2.5 * sqft&#xA;price &amp;lt;- ifelse(tmp, price, 0) + rnorm(100, sd = 0.6)&#xA;DF &amp;lt;- data.frame(sqft = sqft, price = price,&#xA;                 Ind = ifelse(sqft &amp;gt; mean(sqft), 1, 0))&#xA;rm(price, sqft)&#xA;plot(price ~ sqft, data = DF)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Fit the model:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mod &amp;lt;- lm(price~sqft+I((sqft-mean(sqft))*Ind), data = DF)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Generate some data to predict for and predict:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;m.sqft &amp;lt;- with(DF, mean(sqft))&#xA;pDF &amp;lt;- with(DF, data.frame(sqft = seq(min(sqft), max(sqft), length = 200)))&#xA;pDF &amp;lt;- within(pDF, Ind &amp;lt;- ifelse(sqft &amp;gt; m.sqft, 1, 0))&#xA;pDF &amp;lt;- within(pDF, price &amp;lt;- predict(mod, newdata = pDF))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Plot the regression lines:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ylim &amp;lt;- range(pDF$price, DF$price)&#xA;xlim &amp;lt;- range(pDF$sqft, DF$sqft)&#xA;plot(price ~ sqft, data = DF, ylim = ylim, xlim = xlim)&#xA;lines(price ~ sqft, data = pDF, subset = Ind &amp;gt; 0, col = &quot;red&quot;, lwd = 2)&#xA;lines(price ~ sqft, data = pDF, subset = Ind &amp;lt; 1, col = &quot;red&quot;, lwd = 2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You could code this up into a simple function - you only need the steps in the two preceding code chunks - which you can use in place of &lt;code&gt;abline&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;myabline &amp;lt;- function(model, data, ...) {&#xA;    m.sqft &amp;lt;- with(data, mean(sqft))&#xA;    pDF &amp;lt;- with(data, data.frame(sqft = seq(min(sqft), max(sqft),&#xA;                                            length = 200)))&#xA;    pDF &amp;lt;- within(pDF, Ind &amp;lt;- ifelse(sqft &amp;gt; m.sqft, 1, 0))&#xA;    pDF &amp;lt;- within(pDF, price &amp;lt;- predict(mod, newdata = pDF))&#xA;    lines(price ~ sqft, data = pDF, subset = Ind &amp;gt; 0, ...)&#xA;    lines(price ~ sqft, data = pDF, subset = Ind &amp;lt; 1, ...)&#xA;    invisible(model)&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ylim &amp;lt;- range(pDF$price, DF$price)&#xA;xlim &amp;lt;- range(pDF$sqft, DF$sqft)&#xA;plot(price ~ sqft, data = DF, ylim = ylim, xlim = xlim)&#xA;myabline(mod, DF, col = &quot;red&quot;, lwd = 2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Via the &lt;strong&gt;segmented&lt;/strong&gt; package&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(segmented)&#xA;mod2 &amp;lt;- lm(price ~ sqft, data = DF)&#xA;mod.s &amp;lt;- segmented(mod2, seg.Z = ~ sqft, psi = 0.5,&#xA;                   control = seg.control(stop.if.error = FALSE))&#xA;plot(price ~ sqft, data = DF)&#xA;plot(mod.s, add = TRUE)&#xA;lines(mod.s, col = &quot;red&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;With these data it doesn't estimate the breakpoint at &lt;code&gt;mean(sqft)&lt;/code&gt;, but the &lt;code&gt;plot&lt;/code&gt; and &lt;code&gt;lines&lt;/code&gt; methods in that package might help you implement something more generic than &lt;code&gt;myabline&lt;/code&gt; to do this job for you diretcly from the fitted &lt;code&gt;lm()&lt;/code&gt; model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt; If you want segmented to estimate the location of the breakpoint, then set the &lt;code&gt;'psi'&lt;/code&gt; argument to &lt;code&gt;NA&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mod.s &amp;lt;- segmented(mod2, seg.Z = ~ sqft, psi = NA,&#xA;                   control = seg.control(stop.if.error = FALSE))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then &lt;code&gt;segmented&lt;/code&gt; will try &lt;code&gt;K = 10&lt;/code&gt; quantiles of &lt;code&gt;sqft&lt;/code&gt;, with &lt;code&gt;K&lt;/code&gt; being set in &lt;code&gt;seg.control()&lt;/code&gt; and which defaults to &lt;code&gt;10&lt;/code&gt;. See &lt;code&gt;?seg.control&lt;/code&gt; for more.&lt;/p&gt;&#xA;" OwnerUserId="1390" LastEditorUserId="1390" LastEditorDisplayName="" LastEditDate="2011-01-04T12:26:42.980" LastActivityDate="2011-01-04T12:26:42.980" CommentCount="11" />
  <row Id="5957" PostTypeId="2" ParentId="5954" CreationDate="2011-01-04T10:38:37.040" Score="8" ViewCount="" Body="&lt;p&gt;An excellent first question! I agree with mpiktas's answer, i.e. the short answer is &quot;they don't, but they hope to have an approximation to the right model that gives approximately the right answer&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the jargon of epidemiology, this model uncertainty one source of what's known as '&lt;strong&gt;residual confounding&lt;/strong&gt;'. See &lt;a href=&quot;http://pmean.com/10/ResidualConfounding.html&quot; rel=&quot;nofollow&quot;&gt;Steve Simon's page 'What is residual confounding?'&lt;/a&gt; for a good short description, or &lt;a href=&quot;http://onlinelibrary.wiley.com/doi/10.1002/sim.4780111308/abstract&quot; rel=&quot;nofollow&quot;&gt;Heiko Becher's 1992 paper in &lt;em&gt;Statistics in Medicine&lt;/em&gt;&lt;/a&gt; (subscription req'd) for a longer, more mathematical treatment, or &lt;a href=&quot;http://aje.oxfordjournals.org/content/166/6/646.abstract&quot; rel=&quot;nofollow&quot;&gt;Fewell, Davey Smith &amp;amp; Sterne's more recent paper in the &lt;em&gt;American Journal of Epidemiology&lt;/em&gt;&lt;/a&gt; (subscription req'd).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is one reason that epidemiology of small effects is difficult and the findings often controversial - if the measured effect size is small, it's hard to rule out residual confounding or other sources of bias as the explanation.&lt;/p&gt;&#xA;" OwnerUserId="449" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-01-04T15:50:53.127" LastActivityDate="2011-01-04T15:50:53.127" />
  <row Id="5958" PostTypeId="2" ParentId="5954" CreationDate="2011-01-04T12:06:35.013" Score="11" ViewCount="" Body="&lt;p&gt;The other side of the answer, complementary to mpiktas's answer but not mentioned so far, is: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;They don't, but as soon as they assume some model structure, they can &lt;em&gt;check it&lt;/em&gt; against the data&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The two basic things that could go wrong are:  The &lt;em&gt;form&lt;/em&gt; of the function, e.g. it's not even linear in logs.  So you'd start by plotting the an appropriate residual against the expected values.  Or the choice of &lt;em&gt;conditional distribution&lt;/em&gt;, e.g. the observed counts overdispersed relative to Poisson.  So you'd test against an Negative Binomial version of the same model, or see if extra covariates account for the extra variation.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You'd also want to check for outliers, influential observations, and a host of other things.  A reasonable place to read about checking these kinds of model problems is ch.5 of Cameron and Trivedi 1998.  (There is surely a better place for epidemiologically oriented researchers to start - perhaps other folk can suggest it.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If these diagnostics indicated the model failed to fit the data, you'd change the relevant aspect of the model and start the whole process again.&lt;/p&gt;&#xA;" OwnerUserId="1739" LastActivityDate="2011-01-04T12:06:35.013" />
  <row Id="5959" PostTypeId="2" ParentId="5854" CreationDate="2011-01-04T12:37:08.117" Score="2" ViewCount="" Body="&lt;p&gt;You might also want to have a look at &lt;a href=&quot;http://www.jstatsoft.org/v28/c01/paper&quot; rel=&quot;nofollow&quot;&gt;beanplots&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/W4Bf8.jpg&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[&lt;a href=&quot;http://jnlnet.wordpress.com/2008/11/30/beanplots/&quot; rel=&quot;nofollow&quot;&gt;Source&lt;/a&gt;]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Implemented in &lt;a href=&quot;http://cran.r-project.org/web/packages/beanplot/index.html&quot; rel=&quot;nofollow&quot;&gt;R package&lt;/a&gt; by Peter Kampstra.  &lt;/p&gt;&#xA;" OwnerUserId="22" LastActivityDate="2011-01-04T12:37:08.117" />
  <row Id="5960" PostTypeId="1" AcceptedAnswerId="5966" CreationDate="2011-01-04T13:03:59.157" Score="5" ViewCount="728" Body="&lt;p&gt;I understand that once we plot the values as a chart, we can identify a bimodal distribution by observing the twin-peaks, but how does one find it programmatically? (i am looking for an algorithm)&lt;/p&gt;&#xA;" OwnerUserId="2535" LastActivityDate="2011-02-06T07:53:58.403" Title="How to Identify a Bimodal Distribution" Tags="&lt;distributions&gt;" AnswerCount="5" CommentCount="1" FavoriteCount="1" />
  <row Id="5961" PostTypeId="2" ParentId="5913" CreationDate="2011-01-04T13:04:57.233" Score="1" ViewCount="" Body="&lt;p&gt;I'm working with the OP on the benchmarking project and wanted to thank you all for clearing some things up. Also I wanted to provide a bit more information in case that affects the advice.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The sample size ranges from 5 - 700+ (as many as can be completed in 8 seconds or until the margin of error is at or below 1%. The critical values are pulled from an object for simplicity &lt;em&gt;(because other calculations determine the degrees of freedom as sample-size minus 1)&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  /**&#xA;   * T-Distribution two-tailed critical values for 95% confidence&#xA;   * http://www.itl.nist.gov/div898/handbook/eda/section3/eda3672.htm&#xA;   */&#xA;  T_DISTRIBUTION = {&#xA;    '1':  12.706, '2':  4.303, '3':  3.182, '4':  2.776 /* , ... */&#xA;  }&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;h2&gt;Update&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I checked and the difference between variances seems rather high.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Variances:&#xA;    4,474,400,141.236059&#xA;    3,032,977,106.8208385&#xA;  226,854,226,665.14194&#xA;       24,612,581.169126578&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;We are testing the operations per second of various code snippets &lt;em&gt;(some are slower, lower ops/sec, others faster, higher ops/sec)&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also we used to simply compare the overlap between means &lt;code&gt;+-&lt;/code&gt; margin of error of each but it was suggested that a t-test is better because it can find results with statistical significance.&lt;/p&gt;&#xA;" OwnerUserId="2634" LastEditorUserId="2634" LastEditorDisplayName="" LastEditDate="2011-01-04T18:48:40.033" LastActivityDate="2011-01-04T18:48:40.033" CommentCount="11" />
  <row Id="5962" PostTypeId="1" AcceptedAnswerId="5992" CreationDate="2011-01-04T14:06:34.460" Score="5" ViewCount="150" Body="&lt;p&gt;I have 12 positive training sets (cancer cells treated with drugs with each of 12 different mechanisms of action).  For each of these positive training sets, I would like to train a support-vector machine to distinguish it from a negative set of equal size sampled from the experiment.  Each set has between 1000 and 6000 cells, and there are 476 features (image features) of each cell, each scaled linearly to [0, 1].&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I use &lt;a href=&quot;http://www.csie.ntu.edu.tw/~cjlin/libsvm/&quot; rel=&quot;nofollow&quot;&gt;LIBSVM&lt;/a&gt; and the Gaussian RGB kernel.  Using five-fold crossvalidation, I have done a grid search for log₂ C ∈ [-5, 15] and log₂ ɣ ∈ [-15, 3].  The results are as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/8GiiB.png&quot; alt=&quot;Results of grid search&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was disappointed that there is not a single set of parameters that give high accuracies for all 12 classification problems.  I was also surprised that the grids do not generally show a high-accuracy region surrounded by lower accuracies.  Does this just mean that I need to expand the search parameter space, or is the grid search an indication that something else is wrong?&lt;/p&gt;&#xA;" OwnerUserId="220" LastActivityDate="2011-01-05T00:57:32.817" Title="Should an SVM grid search show a high-accuracy region with low accuracies around?" Tags="&lt;svm&gt;" AnswerCount="1" CommentCount="4" />
  <row Id="5963" PostTypeId="2" ParentId="5960" CreationDate="2011-01-04T14:58:43.317" Score="0" ViewCount="" Body="&lt;p&gt;The definition in wiki is slightly confusing to me.  The probability of a continous data set having just one mode is zero.  A simple way to program a bimodal distrubiton is with two seperate normal distributions centered differently.  This creates two peaks or what wiki calls modes.  You can actually use almost any two distributions, but one of the harder statistical opportunities is to find how the data set was formed after combining the two random data distributions.&lt;/p&gt;&#xA;" OwnerUserId="2539" LastActivityDate="2011-01-04T14:58:43.317" CommentCount="2" />
  <row Id="5964" PostTypeId="2" ParentId="5954" CreationDate="2011-01-04T15:34:50.540" Score="4" ViewCount="" Body="&lt;p&gt;There is the famous quote &quot;Essentially, all models are wrong, but some are useful&quot; of &lt;a href=&quot;http://en.wikiquote.org/wiki/George_Box&quot; rel=&quot;nofollow&quot;&gt;George Box&lt;/a&gt;. When fitting models like this, we try to (or should) think about the data generation process and the physical, real world, relationships between the response and covariates. We try to express these relationships in a model that fits the data. Or to put it another way, is consistent with the data. As such an empirical model is produced.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Whether it is useful or not is determined later - does it give good, reliable predictions, for example, for women not used to fit the model? Are the model coefficients interpretable and of scientific use? Are the effect sizes meaningful?&lt;/p&gt;&#xA;" OwnerUserId="1390" LastActivityDate="2011-01-04T15:34:50.540" />
  <row Id="5966" PostTypeId="2" ParentId="5960" CreationDate="2011-01-04T16:44:31.697" Score="7" ViewCount="" Body="&lt;p&gt;Identifying a mode for a continuous distribution requires smoothing or binning the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Binning&lt;/strong&gt; is typically too procrustean: the results often depend on where you place the bin cutpoints.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Kernel_smoother&quot; rel=&quot;nofollow&quot;&gt;Kernel smoothing&lt;/a&gt;&lt;/strong&gt; (specifically, in the form of &lt;a href=&quot;http://en.wikipedia.org/wiki/Kernel_density_estimation&quot; rel=&quot;nofollow&quot;&gt;kernel density estimation&lt;/a&gt;) is a good choice.  Although many kernel shapes are possible, typically the result does not depend much on the shape.  It depends on the kernel bandwidth.  Thus, people either use an &lt;a href=&quot;http://www.math.nyu.edu/mfdd/riedel.save/cip3.pdf&quot; rel=&quot;nofollow&quot;&gt;adaptive kernel smooth&lt;/a&gt; or conduct a sequence of kernel smooths for varying fixed bandwidths in order to check the stability of the modes that are identified.  Although using an adaptive or &quot;optimal&quot; smoother is attractive, be aware that most (all?) of these are designed to achieve a balance between precision and average accuracy: they are not designed to optimize estimation of the location of modes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as implementation goes, kernel smoothers locally shift and scale a predetermined function to fit the data.  Provided that this basic function is differentiable--Gaussians are a good choice because you can differentiate them as many times as you like--then all you have to do is replace it by its derivative to obtain the derivative of the smooth.  Then it's simply a matter of applying a standard zero-finding procedure to detect and test the critical points.  (&lt;a href=&quot;http://en.wikipedia.org/wiki/Brent%27s_method&quot; rel=&quot;nofollow&quot;&gt;Brent's method&lt;/a&gt; works well.)  Of course you can do the same trick with the second derivative to get a quick test of whether any critical point is a local maximum--that is, a mode.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-01-04T16:44:31.697" CommentCount="1" />
  <row Id="5967" PostTypeId="1" AcceptedAnswerId="6002" CreationDate="2011-01-04T16:55:36.283" Score="4" ViewCount="301" Body="&lt;p&gt;We are working on a multivariate linear regression model.  Our objective is to forecast the quarterly % growth in mortgage loans outstanding. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The independent variables are: &#xA;1) Dow Jones level.&#xA;2) % change in Dow Jones over past quarter.&#xA;3) Case Shiller housing price index.&#xA;4) % change in Case Shiller housing price index over past quarter. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a stepwise regression process, all above variables were selected.  And, variables 1) and 3) were surprisingly significant.  But, that is when used in combination.  Somehow, there is something about the difference in those two indeces that does partly explain the % change in mortgage loans outstanding. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For my part, I find variables 1) and 3) problematic.  This is because I believe they are heteroskedastic.  And, their respective coefficients confidence interval are therefore unreliable.  I also think they may have multicollinearity issues with their related variables based on % change.  They may also cause some autocorrelation issues.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, at first it seems some of my concerns may be overstated.  After graphing the residuals of the whole model they seem OK.  They don't trend upward.  So, it appears the heteroskedasticity is not an issue for the whole model.  Multicollinearity is not too bad.  The variable with the highest VIF is around 5 much lower than the usual threshold of 10. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Nevertheless, I am still somewhat concerned that even though the whole model seems OK; the specific regression coefficients of the mentioned variables may not be (or more specifically the related confidence intervals). &lt;/p&gt;&#xA;" OwnerUserId="1329" LastActivityDate="2011-01-05T12:06:14.560" Title="Can you use heteroskedastic time series variables within a regression model?" Tags="&lt;regression&gt;&lt;multicollinearity&gt;&lt;stepwise-regression&gt;&lt;heteroskedasticity&gt;" AnswerCount="3" CommentCount="3" FavoriteCount="2" />
  <row Id="5968" PostTypeId="2" ParentId="3140" CreationDate="2011-01-04T17:01:30.713" Score="1" ViewCount="" Body="&lt;pre&gt;&lt;code&gt;load fisheriris&#xA;indices = crossvalind('Kfold',species,10);&#xA;cp = classperf(species); % initializes the CP object&#xA;for i = 1:10&#xA;    test = (indices == i); train = ~test;&#xA;    class = knnclassify(meas(test,:),meas(train,:),species(train));&#xA;    % updates the CP object with the current classification results&#xA;    classperf(cp,class,test);  &#xA;end&#xA;cp.CorrectRate % queries for the correct classification rate&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2637" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-01-04T19:11:00.380" LastActivityDate="2011-01-04T19:11:00.380" CommentCount="1" />
  <row Id="5969" PostTypeId="1" AcceptedAnswerId="6001" CreationDate="2011-01-04T17:37:50.460" Score="7" ViewCount="294" Body="&lt;p&gt;I do quasi-experimental individual differences psychology research. I examine how people who differ in a cognitive ability (that I measure) perform on another task that always at least involves within-subject manipulations (and sometimes between-subject)–DVs are usually response time and/or accuracy. For this question I'd like to focus on response times (let's assume they are normally distributed). I then infer from the ability-task relations what it means theoretically for the cognitive ability. The nature of this work is correlational and involves repeated measures where each subject completes many task trials(mostly I’m not interested in changes over time, just the overall difference).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Researchers in my field often create categorical variables from the cognitive ability scores and compare the upper and lower quartiles of the distribution with a repeated-measures ANOVA. Because the cognitive ability is measured continuously, I am looking for an analytic strategy that treats the cognitive ability in this way. I’ve recently been investigating mixed-models, thinking that I can treat the people as a random effect grouping variable and the cognitive ability score as a random effect nested within people.  I would like to examine interactions between this nested random effect (cognitive ability) and the fixed effects of the experimental treatments by doing model comparisons.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does this seem like a reasonable analytic strategy? Am I thinking about this right? What are some other ways (the simpler, the better) that I can take advantage of repeated measures -remove experimental subject variance- while also maintaining the cognitive ability measure as a continuous measure? Any citations or examples in R are appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a typical experiment, I would expect there to be anywhere from 1-3 categorical IVs with 2-4 levels(measured by multiple trials), and 1 continuous IV(cognitive ability). The exact nature of the categorical variables changes from study to study. The designs are fully crossed.&lt;/p&gt;&#xA;" OwnerUserId="2322" LastEditorUserId="2322" LastEditorDisplayName="" LastEditDate="2011-01-05T15:21:18.433" LastActivityDate="2011-01-06T22:33:36.143" Title="How should I analyze repeated-measures individual differences experiments?" Tags="&lt;repeated-measures&gt;" AnswerCount="1" CommentCount="6" FavoriteCount="3" />
  <row Id="5970" PostTypeId="2" ParentId="5680" CreationDate="2011-01-04T17:44:03.453" Score="1" ViewCount="" Body="&lt;p&gt;Juan has offered a lot.  A simplified and slightly more structured answer (via an annotated flow chart) is available at &lt;a href=&quot;http://yellowbrickstats.com&quot; rel=&quot;nofollow&quot;&gt;yellowbrickstats.com&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2669" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-01-04T19:18:58.203" LastActivityDate="2011-01-04T19:18:58.203" />
  <row Id="5971" PostTypeId="1" CreationDate="2011-01-04T17:45:49.020" Score="1" ViewCount="106" Body="&lt;p&gt;I'm doing some software testing where we are measuring specific latencies.  Generally we run the same test several times to just eyeball the results and make sure that they are consistent across runs.  I usually do this by plotting the cumulative distributions for each run together on a graph, and look for an anomalies.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What more quantitative methods exist for doing this sort of thing?  I would ultimately like to be able to run my tests multiple times, and determine some sort of error bound, so I can say something like &quot;I change variable X and the median time changed Y, but that is not significant&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Apologies for any vague handwaving above, I am not a statistician :)&lt;/p&gt;&#xA;" OwnerUserId="2638" LastActivityDate="2011-01-04T20:03:10.713" Title="What is a good measure (or set of measures) for the difference between two sample sets?" Tags="&lt;statistical-significance&gt;" AnswerCount="3" FavoriteCount="2" />
  <row Id="5972" PostTypeId="1" AcceptedAnswerId="5986" CreationDate="2011-01-04T17:50:44.507" Score="5" ViewCount="217" Body="&lt;p&gt;Let's say we have two biased coins. The probability of tossing a head on the first coin is $\alpha$ and the probability of tossing a head on the second coin is $1-\alpha$. We toss both coins $n$ times and we say that there is a success when there is a head on both coins. If we denote this random variable by $X$ then&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$X\sim B(n,\alpha-\alpha^2).$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The question is how to properly estimate $\alpha$.  The problem with a 'standard' approach ($\alpha - \alpha^2=\frac{k}{n}$ where $k$ is number of successes) is that the number of successes might be greater than $0.25n$ and in such a case we obtain that $\alpha$ is a complex number.  We can treat such cases as $\alpha=0.5$, but then this estimator is not unbiased anymore.&lt;/p&gt;&#xA;" OwnerUserId="1643" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-01-13T23:11:34.813" LastActivityDate="2011-01-13T23:11:34.813" Title="Estimation of probability of a success in binomial distribution" Tags="&lt;estimation&gt;&lt;binomial&gt;&lt;unbiased-estimator&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="5973" PostTypeId="2" ParentId="5918" CreationDate="2011-01-04T18:25:18.910" Score="2" ViewCount="" Body="&lt;p&gt;I have been doing an extensive cross-validation analysis on a data set that cost millions to acquire, and there is no external validation set available. In this case, I performed extensive nested cross validation to ensure validity. I selected features and optimized parameters only from the respective training sets. This is computationally expensive for large data sets, but it's what I had to do to maintain validity. However, there are complications that come with it...for example, different features are selected in each training set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my answer is that in cases where you don't have feasible access to an external data set, this is a reasonable way to go. When you do have an external data set, you can pretty much go to town however you want on the main data set and then test once on the external data set.&lt;/p&gt;&#xA;" OwnerUserId="2643" LastActivityDate="2011-01-04T18:25:18.910" CommentCount="5" />
  <row Id="5974" PostTypeId="2" ParentId="5971" CreationDate="2011-01-04T18:26:12.340" Score="1" ViewCount="" Body="&lt;p&gt;what kind of output are you getting? are you using regression? if you're trying to measure which set is best to use, you can use an F test using the sample errors, but I'm not sure what exactly you're taking about.&lt;/p&gt;&#xA;" OwnerUserId="2644" LastActivityDate="2011-01-04T18:26:12.340" CommentCount="2" />
  <row Id="5975" PostTypeId="1" CreationDate="2011-01-04T18:26:50.577" Score="2" ViewCount="111" Body="&lt;p&gt;Hi all,&#xA;This probably seems like a really strange question, but let me try to explain what I want to do; hopefully it will make sense.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a data set with a couple dozen variables, such as age, level of education, self-assessed (via a Likert scale) measures of technical ability, experience, willingness to share personal information, etc etc. Using this data, I want to estimate &quot;likelihood of participation&quot; in using a not-yet-developed social network. Since the site doesn't exist yet, I don't have a simple dependent variable with a &quot;1&quot; or &quot;0&quot; indicating whether they've used it or not, so I can't take the usual approach with regression and estimate what impact each variable had on their participation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there some way to &quot;create&quot; a dependent variable that models &quot;likely vs. not likely&quot; (or something like that), and do the regression against that? I'm really lost...would be open to any ideas. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks,&#xA;Matt&lt;/p&gt;&#xA;" OwnerUserId="2641" LastActivityDate="2011-01-05T10:02:04.577" Title="Regression with an unknown dependent variable - estimating &quot;likelihood&quot; to do something" Tags="&lt;regression&gt;" AnswerCount="3" />
  <row Id="5976" PostTypeId="2" ParentId="5975" CreationDate="2011-01-04T18:35:41.397" Score="2" ViewCount="" Body="&lt;p&gt;The simplest approach would be to do a literature search and see if someone else developed a model with your variables, and then generate predictions based on those prior models. Another possibility would be to make your own model from another data source that does have the required information, and then use that to generate predictions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Obviously you can't do much with your data besides generate predictions with my suggestion, although that may be all you want from the context of your question.&lt;/p&gt;&#xA;" OwnerUserId="1036" LastActivityDate="2011-01-04T18:35:41.397" />
  <row Id="5977" PostTypeId="2" ParentId="5967" CreationDate="2011-01-04T18:43:15.373" Score="0" ViewCount="" Body="&lt;p&gt;well, to try and help, with heteroskedasticity present, the least squares estimators are still unbiased but no longer most efficient, they no longer have smallest variance violating part of the Gauss Markov theorum. I could be wrong, but I believe you're standard errors are more effected by multicollinearity than heterosked. Multicoll inflates your standard errors, T values become smaller and leading to increased likelihood of retaining false null hypotheses. &lt;/p&gt;&#xA;" OwnerUserId="2644" LastActivityDate="2011-01-04T18:43:15.373" CommentCount="3" />
  <row Id="5978" PostTypeId="2" ParentId="798" CreationDate="2011-01-04T18:57:29.233" Score="0" ViewCount="" Body="&lt;p&gt;I think Sturges' rule can be used for n &amp;lt; 200; where n is the number of observations&lt;/p&gt;&#xA;" OwnerUserId="2535" LastActivityDate="2011-01-04T18:57:29.233" />
  <row Id="5979" PostTypeId="2" ParentId="5680" CreationDate="2011-01-04T19:01:59.580" Score="2" ViewCount="" Body="&lt;p&gt;Specifically regarding error rates as a DV, &lt;a href=&quot;http://linkinghub.elsevier.com/retrieve/pii/S0749596X07001283&quot; rel=&quot;nofollow&quot;&gt;Dixon (2008)&lt;/a&gt; very cogently demonstrates that null hypothesis testing via ANOVA can cause both increased false alarm rates (calling effects &quot;significant&quot; when they're not) &lt;em&gt;and&lt;/em&gt; increased miss rates (missing real effects). He also shows that mixed effects modelling, specifying binomially distributed error, is the more appropriate approach to analyzing rate data. &lt;/p&gt;&#xA;" OwnerUserId="364" LastActivityDate="2011-01-04T19:01:59.580" />
  <row Id="5980" PostTypeId="2" ParentId="5967" CreationDate="2011-01-04T19:07:38.950" Score="1" ViewCount="" Body="&lt;p&gt;You are trying to mix level and flow data in one regression. Just notice, that the percent change could be considered as a certain transformation of the level variable (like returns and log-returns in financial econometrics). So since your dependent variable is % change, when you include levels, they try to &quot;mimic&quot; stationarity instead of stochastic movement, and likely may form co-integrating relationship (try to test this anyway). Another &quot;trick&quot; to make level series stationary is to try fractions if they have some sense or consider differences (like interest-rate term-structure). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;And I do agree that it is not the heteroscedasticity that forms the problem. Think carefully about the theory you are building your forecasting model on. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good luck. &lt;/p&gt;&#xA;" OwnerUserId="2645" LastEditorUserId="2645" LastEditorDisplayName="" LastEditDate="2011-01-04T20:24:38.007" LastActivityDate="2011-01-04T20:24:38.007" CommentCount="6" />
  <row Id="5982" PostTypeId="1" AcceptedAnswerId="5996" CreationDate="2011-01-04T19:21:07.543" Score="1" ViewCount="166" Body="&lt;p&gt;We are trying to make simulation experiment involving a common stochastic trend, that is described by the random walk (or $I(1)$ process) $Y_t = Y_{t-1} + \varepsilon_t$, where innovations $\varepsilon_t$ ~ $N(0,1)$. However when could we be sure that the past innovations are more or less reasonably included into the stochastic trend?  Is there any good proxy for random-walk's burn-in period? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've looked for R's burn-in default suggestion for $AR(p)$ part: &lt;code&gt;ceiling(6/log(minroots))&lt;/code&gt;, meaning that in case of unit root we get infinity here and roughly tried to take close to unity root 1.0001 (actually it is the same like taking 60000 at once). So any reasonable suggestions or rule of thumbs you do use in practice? &lt;/p&gt;&#xA;" OwnerUserId="2645" LastEditorUserId="2645" LastEditorDisplayName="" LastEditDate="2011-01-21T08:46:29.903" LastActivityDate="2011-01-21T08:46:29.903" Title="Burn-in period for random walk" Tags="&lt;time-series&gt;&lt;simulation&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="5983" PostTypeId="2" ParentId="5971" CreationDate="2011-01-04T19:30:05.887" Score="1" ViewCount="" Body="&lt;p&gt;The general approach is to model your latencies in each condition with some distribution, like a Gaussian, and test whether the means of those two distributions are different, e.g. using a &lt;a href=&quot;http://en.wikipedia.org/wiki/Student%27s_t-test&quot; rel=&quot;nofollow&quot;&gt;t-test&lt;/a&gt;. This lets you pick a significance threshold (p-value) and test for it automatically.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Gaussian isn't a perfect model for this case, because it's symmetric, and you know that your latencies are the sum of the actual time to run your code in isolation, plus some random latency due to the OS swapping, waiting for I/O, ... in other words, you know that your code will never randomly run faster than the time to run it in isolation. But, it probably won't matter. I'd just use a t-test.&lt;/p&gt;&#xA;" OwnerUserId="2489" LastActivityDate="2011-01-04T19:30:05.887" />
  <row Id="5984" PostTypeId="2" ParentId="1164" CreationDate="2011-01-04T19:39:25.350" Score="1" ViewCount="" Body="&lt;p&gt;My knowledge of robust estimators is solely in regards to robust standard errors for regression parameters so my comment will only be in regards to those. I would suggest people read this article,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On The So-Called &quot;Huber Sandwich Estimator&quot; and &quot;Robust Standard Errors&quot;&#xA;by: Freedman, A. David&#xA;The American Statistician, Vol. 60, No. 4. (November 2006), pp. 299-302. &lt;a href=&quot;http://dx.doi.org/10.1198/000313006X152207&quot; rel=&quot;nofollow&quot;&gt;doi:10.1198/000313006X152207&lt;/a&gt; (&lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.119.135&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;PDF Version&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Particular what I am concerned about with these approaches is not that they are wrong, but they are simply distracting from bigger problems. Thus I entirely agree with Robin Girard's answer and his mention of &quot;no free lunch&quot;.&lt;/p&gt;&#xA;" OwnerUserId="1036" LastActivityDate="2011-01-04T19:39:25.350" />
  <row Id="5985" PostTypeId="2" ParentId="5971" CreationDate="2011-01-04T20:03:10.713" Score="1" ViewCount="" Body="&lt;p&gt;I would take your latency responses from running the operation and perform some tests for normality.  If these hold true, then you can create some confidence intervals using a standard normal table.  If your tests for normality don't hold true, then a t-table would be more appropriate.  You said that you normally plot your latency variables to find any anomalies.  How does your plot normally look?  Are these repeated operations being performed with the system being held constant?  Do you want the system held constant during your experiment?  Is the speed of the system being taken into account.  These are all things that could be considered to reduce your error in modeling of the time latency variables.  These things will provide for a better confidence interval.&lt;/p&gt;&#xA;" OwnerUserId="2539" LastActivityDate="2011-01-04T20:03:10.713" />
  <row Id="5986" PostTypeId="2" ParentId="5972" CreationDate="2011-01-04T20:23:32.117" Score="6" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;The estimator is biased&lt;/strong&gt;, regardless.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note first that $\alpha$ is not identifiable because you cannot distinguish between $\alpha$ and $1-\alpha$.  Let's accommodate this problem by allowing that we don't care which coin is which and stipulating (arbitrarily, but with no loss of generality), that $0 \le \alpha \le 1/2$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's reasonable, and conventional, to fix the estimator $g$ as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\eqalign{&#xD;&#xA;g(k,n) =&amp;amp; \frac{1 - \sqrt{\delta}}{2} \cr&#xD;&#xA;\delta =&amp;amp; \max(0, 1 - 4 k / n) &#xD;&#xA;}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;No matter what you do, though, this will be a nonlinear function of the outcome $k$ and therefore is certain to be biased for almost all $\alpha$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;A better approach&lt;/strong&gt; is to search for estimators $h(k,n)$ among some functional class of estimators (such as ones that are linear in $k$) that minimize the expectation of some loss function.  In many situations an estimator that works well for quadratic loss also works well for many reasonable types of losses, so let's look at this.  What we're talking about, then, is (for each $n$) to minimize the expectation $\mathbb{E}[(h(k,n) - \alpha)^2]$ among all estimators $h$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Let's look graphically&lt;/strong&gt; at what's going on.  The bias of any estimator $h$ of the parameter $\alpha$ is the difference between its expectation and the parameter, $\mathbb{E}[h(k,n) - \alpha]$.  We can study &lt;em&gt;any&lt;/em&gt; proposed estimator, then, by graphing its bias (if we really care about that) and its loss.  For any value of $n$ they are functions of $\alpha$, which (of course) is unknown.  That's why we have to look at the entire graph.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are the bias (blue, dashed) and square root of the expected quadratic loss (red) for $g$ when $n=16$:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/YIiBd.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(I use the root of the loss because this is directly comparable to the bias.)&#xA;For example, $g$ is unbiased for $\alpha \approx 1/3$ but otherwise is biased, with the size of the bias largest for $\alpha = 1/2$.  The root expected loss is roughly between 0.15 and 0.2 provided $\alpha$ exceeds $1/6$, approximately.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an alternative, consider linear estimators $h_\lambda(k,n)$ of the form $h_\lambda(k,n) = \lambda(n) k/n$.  Here is a plot of $h_2$ also for $n=16$ (but please note the change in scale on the vertical axis):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/1sohe.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For most $\alpha$ its bias exceeds that of $g$, but for some $\alpha$ (near 0.4) it actually has less bias.  For a wide range of $\alpha$, though, its root expected loss is &lt;em&gt;less&lt;/em&gt; than that of $g$.  Provided $\alpha \gt 1/5$ or so, this simple estimator clearly outperforms the &quot;obvious&quot; one!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is not necessarily &quot;the best&quot; linear estimator, however.  To illustrate, here is a plot of $h_{4/3}$:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/jcSnS.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It outperforms both $g$ and $h_2$ for $1/8 \lt \alpha \lt 3/8$, approximately.  Note, though, that $g$ outperforms the $h_{\lambda}$ for sufficiently small $\alpha$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These considerations suggest there is value in knowing something about what $\alpha$ might be: that will tell you which portions of the loss graphs to focus on in selecting among alternative estimators.  If, in addition, you have a prior distribution for $\alpha$ you can compute the expected loss (this is now a single &lt;em&gt;number&lt;/em&gt;) and use that to compare estimators: your task becomes one of finding an estimator with lowest possible expected loss.  This, of course, is a &lt;em&gt;Bayesian&lt;/em&gt; estimator.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regardless, &lt;strong&gt;using plots of expected loss is a standard and effective way to compare estimators and to choose ones that are appropriate for any particular problem&lt;/strong&gt;.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-01-13T22:10:04.197" LastActivityDate="2011-01-13T22:10:04.197" CommentCount="6" />
  <row Id="5987" PostTypeId="1" AcceptedAnswerId="5989" CreationDate="2011-01-04T20:58:27.577" Score="3" ViewCount="205" Body="&lt;p&gt;What is the worst classier that learns badly in practical problems?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit:&#xA;Especially bad on test data..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="2599" LastEditorUserId="2599" LastEditorDisplayName="" LastEditDate="2011-01-05T13:27:22.290" LastActivityDate="2011-01-05T13:27:22.290" Title="Worst classifier" Tags="&lt;machine-learning&gt;&lt;classification&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="1" />
  <row Id="5988" PostTypeId="2" ParentId="5918" CreationDate="2011-01-04T21:04:30.177" Score="2" ViewCount="" Body="&lt;p&gt;The key thing to remember is that for cross-validation to give an (almost) unbiased performance estimate &lt;em&gt;every&lt;/em&gt; step involved in fitting the model must also be performed independently in each fold of the cross-validation procedure.  The best thing to do is to view feature selection, meta/hyper-parameter setting and optimising the parameters as integral parts of model fitting and never do any one of these steps without doing the other two.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The optimistic bias that can be introduced by departing from that recipe can be surprisingly large, as demonstrated by &lt;a href=&quot;http://jmlr.csail.mit.edu/papers/v11/cawley10a.html&quot; rel=&quot;nofollow&quot;&gt;Cawley and Talbot&lt;/a&gt;, where the bias introduced by an apparently benign departure was larger than the difference in performance between competing classifiers.  Worse still biased protocols favours bad models most strongly, as they are more sensitive to the tuning of hyper-parameters and hence are more prone to over-fitting the model selection criterion!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Answers to specific questions:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The procedure in step 1 is valid becuase feature selection is performed separately in each fold, so what you are cross-validating is whole procedure used to fit the final model.  The cross-validation estimate will have a slight pessimistic bias as the dataset for each fold is slightly smaller than the whole dataset used for the final model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For 2, as cross-validation is used to select the model parameters then you need to repeat that procedure independently in each fold of the cross-validation used for performance estation, you you end up with nested cross-validation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For 3, essentially, yes you need to do nested-nested cross-validation.  Essentially you need to repeat in each fold of the outermost cross-validation (used for performance estimation) &lt;em&gt;everything&lt;/em&gt; you intend to do to to fit the final model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For 4 - yes, if you have a separate hold-out set, then that will give an unbiased estimate of performance without needing an additional cross-validation.&lt;/p&gt;&#xA;" OwnerUserId="887" LastEditorUserId="887" LastEditorDisplayName="" LastEditDate="2011-01-04T21:16:40.400" LastActivityDate="2011-01-04T21:16:40.400" />
  <row Id="5989" PostTypeId="2" ParentId="5987" CreationDate="2011-01-04T21:15:11.963" Score="4" ViewCount="" Body="&lt;p&gt;It is usually the statistician using the classifier that is the problem (for picking the wrong tool for the particular problem at hand, or for using it incorrectly).  The &quot;no free lunch&quot; theorems show there is no a-priori distinction between classifiers (which works best/worst depends on the data), so I'd say there isn't a worst classifier in a general sense.  However, some classifiers are easier to shoot yourself in the foot with than others, in which case, I would nominate artificial neural networks (the over-fit very easily and need proper architecture selection and none of the solutions to such problems are reliable).&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2011-01-04T21:15:11.963" CommentCount="4" />
  <row Id="5990" PostTypeId="2" ParentId="5987" CreationDate="2011-01-04T21:17:03.833" Score="7" ViewCount="" Body="&lt;p&gt;Consider the binary case.  If you don't know the proportions of the two classes, then the worst you can do is to &lt;strong&gt;flip a fair coin&lt;/strong&gt; in each case: the expected error rate is $1/2$.  If you do know the proportions, and the smaller of the two is $p$, say, then you should always classify objects in that category: the expected error rate is $1-p \gt 1/2$.  (However, it could be argued that this procedure is &lt;em&gt;not&lt;/em&gt; worse.  After observing it sufficiently long, you would notice its error rate is significantly greater than $1/2$ and then you would have the option of &lt;em&gt;negating&lt;/em&gt; it--that is, assigning the opposite classification--and that would have an error rate of only $p \lt 1/2$.)&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-01-04T21:17:03.833" CommentCount="7" />
  <row Id="5991" PostTypeId="2" ParentId="5975" CreationDate="2011-01-04T22:49:00.363" Score="5" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;&quot;Data! Data! Data! I can't make bricks&#xA;  without clay.&quot;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;              Sherlock Holmes&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Sorry, you just can't do it. If you make up some measure that combines some of your variables, then guess what: those exact variables will be predictive of it.&lt;/p&gt;&#xA;" OwnerUserId="279" LastActivityDate="2011-01-04T22:49:00.363" CommentCount="1" />
  <row Id="5992" PostTypeId="2" ParentId="5962" CreationDate="2011-01-05T00:57:32.817" Score="3" ViewCount="" Body="&lt;p&gt;The optimal values for the hyper-parameters will be different for different learning taks, you need to tune them separately for every problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reason you don't get a single optimum is becuase both the kernel parameter and the regularisation parameter control the complexity of the model.  If C is small you get a smooth model, likewise if the kernel with is broad, you will get a smooth model (as the basis functions are not very local).  This means that different combinations of C and the kernel width lead to similarly complex models, with similar performance (which is why you get the diagonal feature in many of the plots you have).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The optimum also depends on the particular sampling of the training set.  It is possible to over-fit the cross-validation error, so choosing the hyper-parameters by cross-validation can actually make performance worse if you are unlucky.  See &lt;a href=&quot;http://jmlr.csail.mit.edu/papers/v11/cawley10a.html&quot; rel=&quot;nofollow&quot;&gt;Cawley and Talbot&lt;/a&gt; for some discussion of this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The fact that there is a broad plateau of values for the hyper-parameters where you get similarly good values is actually a good feature of support vector machines as it suggests that they are not overly vulnerable to over-fitting in model selection.  If you had a sharp peak at the optimal values, that would be a bad thing as the peak would be difficult to find using a finite dataset which would provide an unreliable indication of where that peak actually resides.&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2011-01-05T00:57:32.817" />
  <row Id="5993" PostTypeId="2" ParentId="5797" CreationDate="2011-01-05T01:32:02.787" Score="1" ViewCount="" Body="&lt;p&gt;This answer may be way, way off base as I don't understand the medical context of your question and the nature of the medical test results you allude to, but it might be possible to estimate data mining bias by some sort of Monte Carlo permutation of your results. The type of approach I'm thinking of is taken from a book, Evidence Based Technical Analysis, which has a companion website &lt;a href=&quot;http://www.evidencebasedta.com/&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Although this is written with the financial markets in mind it is, in my opinion, a book about Monte Carlo techniques as much as it is about markets. &lt;/p&gt;&#xA;" OwnerUserId="226" LastActivityDate="2011-01-05T01:32:02.787" CommentCount="1" />
  <row Id="5994" PostTypeId="2" ParentId="118" CreationDate="2011-01-05T02:44:49.123" Score="-1" ViewCount="" Body="&lt;p&gt;Thinker Fisher:  We square the difference of the absolute values from the mean to get the mean and take the square root back at the end rather than use the absolute value (without squaring) of the difference to get the expected value (mean) because by squaring the absolute values we thin out the tails of the distribution.  This gives the illusion of a &quot;normal distribution&quot; with squared values creating a tighter cluster around the mean rather than a flatter bell curve with thicker tails and greater variance.  In effect, the math was created to increase the illusion of certainty, a requirement of forecast engines to &quot;control risk&quot; in spite of the impact of dramatically increasing risk by eliminating uncertainty. That is why we are always blindsided by uncertainty; we have effectively eliminated it from the math.&lt;/p&gt;&#xA;" OwnerUserId="2647" LastActivityDate="2011-01-05T02:44:49.123" CommentCount="2" />
  <row Id="5995" PostTypeId="1" AcceptedAnswerId="6014" CreationDate="2011-01-05T05:27:36.953" Score="3" ViewCount="261" Body="&lt;p&gt;I apologize in advance for the vague title, but I couldn't think of anything better.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have two datasets, where one is a very small subset of the other. The percentage of people who have a specific attribute in the large dataset is x%. The percentage of people who have the same attribute in the subset is y%. The subset contains people most likely to do some action X.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance: I have a list of people who have bought my software. I have a list of people who have renewed. The attribute % for the bought is 20, for the renewed is 60%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The buyer list size is say 100,000, the people who have renewed is 500.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My questions are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Is the likelihood of that attribute affecting the user's behaviour simply 3x?&lt;/li&gt;&#xA;&lt;li&gt;I know correlation is not causation, but at what point does this change? I.e, in the above case, there's a 3x difference, is that statistically significant enough?&lt;/li&gt;&#xA;&lt;li&gt;Does the huge difference between buyers/renewed introduce some sort of bias into the dataset tha makes the attribute percentage useless as a form of determining people who will renew?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Thank you.&lt;/p&gt;&#xA;" OwnerUserId="2648" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-01-05T07:40:00.260" LastActivityDate="2011-01-05T19:37:25.097" Title="Analyzing the difference between two datasets where one is a subset of the other" Tags="&lt;statistical-significance&gt;&lt;data-mining&gt;&lt;unbiased-estimator&gt;" AnswerCount="3" />
  <row Id="5996" PostTypeId="2" ParentId="5982" CreationDate="2011-01-05T05:51:44.557" Score="4" ViewCount="" Body="&lt;p&gt;Burn-in doesn’t make sense here.  The random walk you describe does not have a stationary distribution.&lt;/p&gt;&#xA;" OwnerUserId="1670" LastActivityDate="2011-01-05T05:51:44.557" CommentCount="10" />
  <row Id="5997" PostTypeId="1" AcceptedAnswerId="5998" CreationDate="2011-01-05T05:55:56.113" Score="4" ViewCount="156" Body="&lt;p&gt;I'm a beginner to statistics (taken only one college course), but I have a background in programming. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I just started playing with a Bayesian classifier library for Ruby and I am looking for ideas for things to analyze.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Right now I'm messing around with Tweet categorization, but do you have any ideas?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More importantly, how can I learn about which types of data lend themselves to naive Bayesian classification. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="2649" LastActivityDate="2011-03-26T14:30:20.593" Title="What kinds of things can I predict with a naive Bayesian classifier?" Tags="&lt;bayesian&gt;&lt;naive-bayes&gt;" AnswerCount="2" />
  <row Id="5998" PostTypeId="2" ParentId="5997" CreationDate="2011-01-05T07:35:46.257" Score="8" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://www-stat.stanford.edu/~tibs/ElemStatLearn/&quot; rel=&quot;nofollow&quot;&gt;The Elements of Statistical Learning&lt;/a&gt;, by Hastie et al. has a lot of illustrations of Machine Learning applications, and all &lt;a href=&quot;http://www-stat.stanford.edu/~tibs/ElemStatLearn/data.html&quot; rel=&quot;nofollow&quot;&gt;data sets&lt;/a&gt; are available on the companion website, including data on spam as on the &lt;a href=&quot;http://classifier.rubyforge.org/&quot; rel=&quot;nofollow&quot;&gt;Ruby Classifier&lt;/a&gt; webpage. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for a gentle introduction to Bayes classifier, I would suggest to look at the following tutorial from Andrew Moore: &lt;a href=&quot;http://www.autonlab.org/tutorials/naive.html&quot; rel=&quot;nofollow&quot;&gt;A Short Intro to Naive Bayesian Classifiers&lt;/a&gt; (many &lt;a href=&quot;http://www.autonlab.org/tutorials/&quot; rel=&quot;nofollow&quot;&gt;other tutorials&lt;/a&gt; are also available).&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-01-05T07:35:46.257" />
  <row Id="5999" PostTypeId="2" ParentId="5975" CreationDate="2011-01-05T10:02:04.577" Score="3" ViewCount="" Body="&lt;p&gt;I agree with @Aniko that you won't be able to &quot;predict&quot; anything without an outcome. Now @Andy suggestion makes sense, provided you find a users database sharing similar characteristics. As an example of related studies, I guess you might find interesting google hits on users' characteristics in studies on Twitter, Facebook or other social networks or community driven sites. Here is a couple of references that I just found: &lt;a href=&quot;http://www.amath.unc.edu/Faculty/mucha/Reprints/facebook.pdf&quot; rel=&quot;nofollow&quot;&gt;Comparing community structure to characteristics in online collegiate social networks&lt;/a&gt; (Traud et al.,  &lt;a href=&quot;http://arxiv.org/abs/0809.0690&quot; rel=&quot;nofollow&quot;&gt;arXiv:0809.0690&lt;/a&gt;); &lt;a href=&quot;http://faculty.poly.edu/~onov/Nov%20Wattal%20CHI%202009%20Social%20Computing%20and%20Privacy%20Concerns.pdf&quot; rel=&quot;nofollow&quot;&gt;Social Computing Privacy Concerns: Antecedents &amp;amp; Effects&lt;/a&gt; (Nov and Wattal, CHI 2009).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Aside from users' characteristics, willingness to particpate or contribute to a social network is also a function of its characteristics (number of members, contact frequency, content quality, etc.).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, you still can do some kind of exploratory analysis on your data set. Specifically, a cluster analysis would help to highlight groups of individuals sharing similar response profiles. Another approach would be to use &lt;a href=&quot;http://en.wikipedia.org/wiki/Multiple_correspondence_analysis&quot; rel=&quot;nofollow&quot;&gt;Multiple correspondence analysis&lt;/a&gt; to uncover potentially interesting patterns in your data. Those features might be related to already published results in a further step. Obviously, you can't regress an outcome that is still to be observed from this, but you will certainly get a better idea of how structured your data are.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-01-05T10:02:04.577" />
  <row Id="6000" PostTypeId="2" ParentId="5995" CreationDate="2011-01-05T10:12:10.967" Score="0" ViewCount="" Body="&lt;p&gt;For question 2 (significance), could you simply use a randomised resampling procedure and calculate the percentage of subsets of n=500 where the attribute % is ≥60?&lt;/p&gt;&#xA;" OwnerUserId="266" LastActivityDate="2011-01-05T10:12:10.967" CommentCount="1" />
  <row Id="6001" PostTypeId="2" ParentId="5969" CreationDate="2011-01-05T10:55:52.227" Score="6" ViewCount="" Body="&lt;p&gt;There were already some useful comments, that are probably waiting for some updates in the question, so I will just drop some general online references:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://magnuson.psy.uconn.edu/mirman/R/Baayen.pdf?bcsi_scan_D99544420D78AF92=0&amp;amp;bcsi_scan_filename=Baayen.pdf&quot; rel=&quot;nofollow&quot;&gt;Practical Data Analysis for the Language Sciences with R&lt;/a&gt;, Baayen (2008)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://nlp.stanford.edu/~manning/courses/ling289/Jaeger07catdata.pdf&quot; rel=&quot;nofollow&quot;&gt;Categorical data analysis: Away from ANOVAs (transformation or not) and towards logit mixed models&lt;/a&gt;, Jaeger (J. Mem. Language 2008 59(4))&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Examples using R may be found on Doug Bates' &lt;a href=&quot;http://lme4.r-forge.r-project.org/&quot; rel=&quot;nofollow&quot;&gt;lme4 - Mixed-effects models project&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-01-05T10:55:52.227" />
  <row Id="6002" PostTypeId="2" ParentId="5967" CreationDate="2011-01-05T12:06:14.560" Score="3" ViewCount="" Body="&lt;p&gt;Your dependent variable is growth. For economic time-series data it is more likely that growth will be a stationary process. This means that it will have constant mean. The level data on the other hand is usually non-stationary. Since your model is linear regression you assume that the true data generating process is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Y_t=\alpha_0+X_{1t}\alpha_1+...+X_{kt}\alpha_k+u_t$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $u_t$ is a white noise, $Y_t$ is stationary and $X_{kt}$ are non-stationary. Now stationarity implies that&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$EY_t=const=\alpha_0+\alpha_1EX_{1t}+...+\alpha_kEX_{kt}$$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now $EX_{kt}$ are functions of time which for non-stationary processes changes with time. So you are implying that&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\alpha_0+\alpha_1\mu_1(t)+...+\alpha_k\mu_k(t)=const$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;for some non-constant functions $\mu_k(t)$. This places quite severe restrictions on non-stationary processes $X_{kt}$. For example if we have only one independent variable this restriction becomes&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\alpha_1\mu_1(t)=const-\alpha_0$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;so either $\mu_1$ is constant or $\alpha_1$ is zero. In first case this contradicts the presumption, that $X_1$ is non-stationary in the second case the regression model is of no use. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So in general this is why it is not a good idea to mix levels and growths in regression, unless you are really sure that they are all stationary.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another problem with time-series regression that for certain class &lt;a href=&quot;http://en.wikipedia.org/wiki/Order_of_integration&quot; rel=&quot;nofollow&quot;&gt;of non-stationarity processes&lt;/a&gt; the regression can be &lt;a href=&quot;http://en.wikipedia.org/wiki/Spurious_correlation&quot; rel=&quot;nofollow&quot;&gt;spurious&lt;/a&gt;. In this case you cannot trust the least squares estimates $\hat{\alpha}_k$, since in spurious regression case their distribution is not normal and does not tend to normal, so usual regression statistics do not apply. For example you can get that $\alpha_k$ is significantly non-zero, when it is actually zero. So before the regression it is always a good idea to test whether your variables are not integrated using some variant of &lt;a href=&quot;http://en.wikipedia.org/wiki/Dickey%E2%80%93Fuller_test&quot; rel=&quot;nofollow&quot;&gt;Dickey-Fuller test&lt;/a&gt;. I strongly suspect that Dow Jones index is integrated process.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now as others pointed out, heteroscedasticity in the independent regression variable is harmless. The problems can arise if regression errors are heteroscedastic. Then the least squares estimates will be consistent, but inefficient, also the standard errors should &lt;a href=&quot;http://en.wikipedia.org/wiki/White_standard_errors&quot; rel=&quot;nofollow&quot;&gt;be adjusted&lt;/a&gt;, for hypothesis testing.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-01-05T12:06:14.560" CommentCount="1" />
  <row Id="6003" PostTypeId="2" ParentId="5995" CreationDate="2011-01-05T12:07:25.033" Score="2" ViewCount="" Body="&lt;p&gt;I interpret &quot;has attribute&quot; as &quot;the value for this attribute is not missing&quot;. As you already pointed out, the question now is whether the value &quot;missing&quot; has a importance regarding the target &quot;renewed&quot; (to add another (fictional, but plausible) example: missing entries for attribute &quot;income&quot; in a customer database may occur with a higher probability when the corresponding customers have high income). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say $q$ the missing ratio and $p=1-q$ the not-missing ratio.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;No, not plainly 3. This is because the other value of the attribute may have some influence on the outcome (&quot;renewed&quot;), too. To measure the importance one can e.g. use InformationGainRatio. I recommend to perform a discretization first (if the attribute we are talking about is numerical) and (most important) treat the value &quot;missing&quot; as an additional value, don't just ignore it. Now you can calculate the InformationGainRatio for the whole set and the subset to see the difference in importance. Another option is do apply a Logistic Regression and check the coefficient of the attribute (but I do not have that much experience in this area). &lt;/li&gt;&#xA;&lt;li&gt;See the answer of Freya Harrison. You can perform a statistical test using the normal distribution approximation to the binomial distribution with the Null-Hypothesis $H_0:=p&amp;lt;=0.2$ and Alternative-Hypothesis $H_1:= p&amp;gt;0.2$. The result will tell you whether the difference is statistical significant.&lt;/li&gt;&#xA;&lt;li&gt;I don't think so, because a set of 500 is not small (beside: it is not necessarily true that the whole set would have less information if you restrict it to a sample of 20000). There is still some variance, but useless is the attribute (given that the difference is, even if not significant, quite big) definitively not. In such cases a good old repetitive crossvalidation tells us (I'd choose k around 5 in your case) whether one can perform a prediction based on the attribute or not.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="264" LastActivityDate="2011-01-05T12:07:25.033" />
  <row Id="6004" PostTypeId="2" ParentId="5746" CreationDate="2011-01-05T12:29:37.580" Score="0" ViewCount="" Body="&lt;p&gt;Also not an answer, but a back-of-the envelope calculation for the simpler case of i.i.d. X&lt;sub&gt;j&lt;/sub&gt; from a known distribution, to help my (non-statistician's) intuition:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example Monte Carlo, to show the wide variation in run lengths, payoffs and costs:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# sample exponential*100 until &amp;gt; the first 3&#xA;# c0 = break-even cost, peak / n&#xA;&#xA;15  [ 54 127   0  36  16  10  21  42  51  77  54 116  23 210]&#xA;17  [24 31 68  5 85]&#xA;22  [345  38 118 209 225   9   4  19 210  10  55 317  76 118  38 116 180   2 139 450]&#xA;23  [  3 111  54  82  15  22 161]&#xA;23  [ 35  34  14   2 114]&#xA;&#xA;24  [ 16  89 120  11  53 119  53   5  77 109  72 289]&#xA;33  [ 88 234  15  15 165  51  18 262]&#xA;40  [138  33 156  11  59 239]&#xA;45  [ 98 139  43  31 226]&#xA;54  [ 43 139 129 215]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For X&lt;sub&gt;j&lt;/sub&gt; i.i.d. from a known distribution,&#xA;one can calculate the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Emax3 = Expected max( X&lt;sub&gt;1&lt;/sub&gt; .. X&lt;sub&gt;3&lt;/sub&gt; )  # or 10 or &amp;radic; n ...&lt;br&gt;&#xA;p = Prob { X &gt; Emax3 }&lt;br&gt;&#xA;Ntopeak ~ 1/p, the expected number of draws to the first X &gt; Emax&lt;br&gt;&#xA;Epeak = its expected value&lt;/p&gt;&#xA;&#xA;&lt;p&gt;then compare the break-even cost c0 = Epeak / (3 + Ntopeak)&#xA;to the ante you can afford.&#xA;For exponentials, as above, I get&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;expon 3:&#xA;Emax3 158&#xA;Ntopeak 5&#xA;Epeak quartiles [187 227 296]&#xA;c0 ~ [187 227 296] / (3 + 5) = [23 28 37]&#xA;    # but Epeak dist / Ntopeak dist ?&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For a given distribution and cost-per-sample c,&#xA;one could optimize the number of initial samples (here 3)&#xA;by brute force.&#xA;But the high variance in such results &#xA;makes them hard to communicate, even misleading.&lt;/p&gt;&#xA;" OwnerUserId="557" LastActivityDate="2011-01-05T12:29:37.580" />
  <row Id="6005" PostTypeId="1" AcceptedAnswerId="6009" CreationDate="2011-01-05T13:36:36.637" Score="1" ViewCount="182" Body="&lt;p&gt;I am slightly new to writing functions in R. &#xA;Here I have a basic function that searches for a pattern and returns the indexes of where the occurence occurs given a list dataset&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#this functions takes a pattern and prints the indexes for the matches&#xA;find_domain &amp;lt;- function(pattern,list) grep(pattern,list,ignore.case=T)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Sometimes during the analysis i found that i need to do that for a larger dataset. The idea is to look for occurence of a domain (pattern) in each of the d7_dataset rows(since each row represents a protein sequence and each column is a domain) such that i can then count the number of sequences that contain or have a given domain(pattern in this case) So i wrote this&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;seqs &amp;lt;- c()&#xA;for (i in 1:nrow(d7_dataset)){&#xA;       pos &amp;lt;- find_domain(&quot;CIDRγ13&quot;,d7_dataset[i,])&#xA;       if (!is.na(pos[1])) &#xA;        seqs &amp;lt;- c(seqs,1)  &#xA;     }&#xA;&#xA;total_seqs &amp;lt;- sum(seqs)&#xA;total_seqs &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However this seems like it can be written as a single function so that i can eliminate duplication and simplify it such that it is generic enough to apply to multiple datasets.&#xA;Any ideas on how to condence it to a single function?&#xA;In a nutshell, Given a pattern or list of patterns  and a dataset(as a dataframe) to search, return the number of sequences(rows in the dataset) that contain the given pattern.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More or less like this&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;results &amp;lt;– search(a list of patterns to search,dataset)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;results should look like&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pattern  count&#xA; pat1     40&#xA; pat2      3&#xA; pat3      0&#xA; .         . &#xA; .         .&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It would be very good if the pattern argument would accept a list to search&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you&lt;/p&gt;&#xA;" OwnerUserId="201" LastActivityDate="2011-01-05T16:59:03.550" Title="An R function for performing searches" Tags="&lt;r&gt;&lt;dataset&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="6007" PostTypeId="2" ParentId="6005" CreationDate="2011-01-05T13:53:05.480" Score="3" ViewCount="" Body="&lt;p&gt;You should first replace your for loop with something like &lt;code&gt;apply(d7_dataset, 1, foo)&lt;/code&gt;, where &lt;code&gt;foo()&lt;/code&gt; is either your function or something along those lines, e.g. &lt;code&gt;gregexpr()&lt;/code&gt;. The result of &lt;code&gt;gregexpr()&lt;/code&gt; is a list of numeric vectors with attributes similar to &lt;code&gt;regexpr()&lt;/code&gt; but giving all matches in each element.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On a related point, there was another function that was proposed as a more user-friendly alternative to &lt;code&gt;gregexpr()&lt;/code&gt;: &lt;a href=&quot;http://www.r-bloggers.com/developing-a-user-friendly-regular-expression-function-easygregexpr/&quot; rel=&quot;nofollow&quot;&gt;easyGregexpr&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The following example gives you the number of matches for each row when considering a list of three motifs (in a 10x100 matrix):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dd &amp;lt;- replicate(100, replicate(10, paste(sample(letters[1:4], 2),    &#xA;                                         collapse=&quot;&quot;)))&#xA;pat &amp;lt;- list(&quot;aa&quot;, &quot;ab&quot;, &quot;cd&quot;)&#xA;foo &amp;lt;- function(d, p) apply(d, 1, function(x) length(grep(p, x)))&#xA;lapply(pat, function(x) foo(dd, x))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-01-05T16:59:03.550" LastActivityDate="2011-01-05T16:59:03.550" CommentCount="2" />
  <row Id="6008" PostTypeId="1" AcceptedAnswerId="6011" CreationDate="2011-01-05T13:56:46.507" Score="2" ViewCount="266" Body="&lt;p&gt;I am trying to do Bayesian posterior predictive checking, whereby I calculate the DIC for my fitted model, and compare to DIC from data simulated from the fitted model. I can get the DIC out of winBUGS, however I am not sure how to calculate the likelihood (for the DIC) outside of winBUGS (i.e., without fitting new models). All of the literature is quite general with respect to discrepancy functions for model checking, and have notation like p(D|theta) which I understand, but doesn't help me when I want to actually do the calculation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given a set of parameter values, and a Y_rep data set - how do I calculate the likelihood? &lt;/p&gt;&#xA;" OwnerUserId="2654" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-01-05T15:59:50.490" LastActivityDate="2011-01-05T16:01:11.860" Title="How to calculate likelihood for a bayesian model?" Tags="&lt;bayesian&gt;&lt;likelihood-function&gt;" AnswerCount="1" />
  <row Id="6009" PostTypeId="2" ParentId="6005" CreationDate="2011-01-05T14:06:47.837" Score="5" ViewCount="" Body="&lt;p&gt;If your dataset is a matrix, then &lt;code&gt;grep&lt;/code&gt; will work directly:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; set.seed(1)&#xA;&amp;gt; bb=matrix(letters[sample(1:20, 100, rep=TRUE)], nrow=20)&#xA;&amp;gt; grep(&quot;b&quot;, bb)&#xA; [1] 10 55 56 69 92&#xA;&amp;gt; bb[grep(&quot;b&quot;, bb)]&#xA;[1] &quot;b&quot; &quot;b&quot; &quot;b&quot; &quot;b&quot; &quot;b&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you have multiple patterns, then use &lt;code&gt;ldply&lt;/code&gt; from package &lt;a href=&quot;http://cran.r-project.org/web/packages/plyr/index.html&quot; rel=&quot;nofollow&quot;&gt;plyr&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; ldply(c(&quot;a&quot;, &quot;b&quot;, &quot;z&quot;), function(l) data.frame(pattern = l, count = length(grep(l, bb))))&#xA;    pattern count&#xA;  1       a     2&#xA;  2       b     5&#xA;  3       z     0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Your function &lt;code&gt;search&lt;/code&gt; then can look like this&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; search &amp;lt;- function(patlist,data) {&#xA;      if(!is.matrix(data)) data &amp;lt;- as.matrix(data)&#xA;      ldply(patlist, function(l) data.frame(pattern = l, count = length(grep(l, bb))))&#xA; }&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Note that this will not work for &lt;code&gt;data.frame&lt;/code&gt;. So if that is possible you will need to convert it to the matrix, as I did in the code above.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-01-05T14:06:47.837" CommentCount="5" />
  <row Id="6011" PostTypeId="2" ParentId="6008" CreationDate="2011-01-05T15:33:28.930" Score="3" ViewCount="" Body="&lt;p&gt;The &lt;a href=&quot;http://en.wikipedia.org/wiki/Likelihood_function&quot; rel=&quot;nofollow&quot;&gt;likelihood&lt;/a&gt; is numerically equal to $\operatorname{P}(D\vert\theta)$, where D is the data vector and theta the parameter vector. Well, strictly speaking it's only equal up to a multiplicative factor, but most software packages including WinBUGS define it as being equal - see &lt;a href=&quot;http://www.mrc-bsu.cam.ac.uk/bugs/winbugs/dicpage.shtml#q1&quot; rel=&quot;nofollow&quot;&gt;Q1 of their FAQ on DIC&lt;/a&gt;. The distinction between likelihood and probability is made because instead of the probability of the &lt;em&gt;data&lt;/em&gt; for given values of the &lt;em&gt;parameters&lt;/em&gt;, the likelihood is considered to be a function of the &lt;em&gt;parameters&lt;/em&gt; for a fixed set of &lt;em&gt;data&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's usually analytically simpler and more computationally stable to work with the &lt;a href=&quot;http://en.wikipedia.org/wiki/Likelihood_function#Log-likelihood&quot; rel=&quot;nofollow&quot;&gt;log-likelihood&lt;/a&gt;, and that's what you'll need to calculate the DIC anyway.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;ADDED&lt;/strong&gt; In response to user2654's comment below:&lt;br&gt;&#xA;$\operatorname{P}(D\vert\theta)$ depends on the model for the data. The simplest case is when the data are &lt;a href=&quot;http://en.wikipedia.org/wiki/Iid&quot; rel=&quot;nofollow&quot;&gt;independent and identically distributed&lt;/a&gt; (iid), in which case $\operatorname{P}(D\vert\theta)$  becomes a product over all the data points, and therefore the log-likelihood is a sum over all the data points. The probability for each data point is calculated from the &lt;a href=&quot;http://en.wikipedia.org/wiki/Probability_density_function&quot; rel=&quot;nofollow&quot;&gt;probability density function&lt;/a&gt; or &lt;a href=&quot;http://en.wikipedia.org/wiki/Probability_mass_function&quot; rel=&quot;nofollow&quot;&gt;probability mass function&lt;/a&gt; defined by your model.&lt;/p&gt;&#xA;" OwnerUserId="449" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-01-05T16:01:11.860" LastActivityDate="2011-01-05T16:01:11.860" CommentCount="3" />
  <row Id="6013" PostTypeId="1" AcceptedAnswerId="6015" CreationDate="2011-01-05T19:17:42.883" Score="7" ViewCount="393" Body="&lt;p&gt;I have a python script that creates a list of lists of server uptime and performance data, where each sub-list (or 'row') contains a particular cluster's stats.  For example, nicely formatted it looks something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;-------  -------------  ------------  ----------  -------------------&#xA;Cluster  %Availability  Requests/Sec  Errors/Sec  %Memory_Utilization&#xA;-------  -------------  ------------  ----------  -------------------&#xA;ams-a    98.099          1012         678          91&#xA;bos-a    98.099          1111         12           91&#xA;bos-b    55.123          1513         576          22&#xA;lax-a    99.110          988          10           89&#xA;pdx-a    98.123          1121         11           90&#xA;ord-b    75.005          1301         123          100&#xA;sjc-a    99.020          1000         10           88&#xA;...(so on)...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So in list form, it might look like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[[ams-a,98.099,1012,678,91],[bos-a,98.099,1111,12,91],...]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My question:  What's the best way to determine the outliers in each column?  Or are outliers not necessarily the best way to attack the problem of finding 'badness'?  In the data above, I'd definitely want to know about bos-b and ord-b, as well as ams-a since it's error rate is so high, but the others can be discarded.  Depending on the column, since higher is not necessarily worse, nor is lower, I'm trying to figure out the most efficient way to do this.  Seems like numpy gets mentioned a lot for this sort of stuff, but not sure where to even start with it (sadly, I'm more sysadmin than statistician...).  When I asked over at Stack Overflow, someone mentioned using numpy's scoreatpercentile function and throw out anything over 99th percentile - does that seem like a good idea?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Cross-posted from stackoverflow, here: &lt;a href=&quot;http://stackoverflow.com/questions/4606288&quot;&gt;http://stackoverflow.com/questions/4606288&lt;/a&gt;)&lt;/p&gt;&#xA;" OwnerUserId="2659" LastActivityDate="2011-01-06T04:54:35.017" Title="Finding outliers in a data set" Tags="&lt;outliers&gt;&lt;percentile&gt;" AnswerCount="3" FavoriteCount="3" />
  <row Id="6014" PostTypeId="2" ParentId="5995" CreationDate="2011-01-05T19:37:25.097" Score="5" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Make a table&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;              No renewal Renewal  Total&#xA;              ---------- ------- ------&#xA;Attribute  No      79800     200  80000&#xA;          Yes      19700     300  20000&#xA;---------------------------------------&#xA;Total              99500     500 100000&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;The computations are&lt;/em&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Number of non-renewers = 100,000 - 500 = 99,500.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&quot;Attribute&quot; is 20% of all buyers = 0.20 * 100,000 = 20,000.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Non-&quot;attribute&quot; is therefore 100,000 - 20,000 - 80,000 of all buyers.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&quot;Attribute&quot; is 60% of all renewers = 0.60 * 500 = 300.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Therefore, 20,000 - 300 = 19,700 of all non-renewers have the attribute.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Non-&quot;attribute&quot; for renewers is 500 - 300 = 200.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Therefore, 80,000 - 200 = 79,800 of all non-renewers do not have the attribute.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Conduct a &lt;a href=&quot;http://en.wikipedia.org/wiki/Pearson%27s_chi-square_test#Test_of_independence&quot; rel=&quot;nofollow&quot;&gt;chi-square test of independence&lt;/a&gt;.&lt;/strong&gt;  This is valid because each of the cells in your table has a large count.  (&quot;Large&quot; typically means 5 or greater.)  &lt;em&gt;Here are the calculations&lt;/em&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;80% of all buyers are &quot;no attribute&quot; and 99.5% do not renew.  Therefore we expect .80 * .995 * 100,000 = 79,600 to be non-renewers with &quot;no attribute&quot;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Similarly, the rest of the table of expected values is&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;               No renewal  Renewal&#xA;                ---------- -------&#xA;  Attribute  No      79600     400&#xA;            Yes      19900     100&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The residuals (differences between the tables) are 79800 - 79600 = 200, 200 - 400 = -200, 19700 - 19900 = -200, and 300 - 100 = 200.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The chi-square terms are the &lt;em&gt;squared residuals divided by the expectations.&lt;/em&gt;  Specifically, these equal $200^2/79600 = 0.5025$, $(-200)^2/400 = 100$, $(-200)^2/19900 = 2.0101$, and $200^2/100 = 400$.  The chi-square statistic is their sum, 0.5025 + 100 + 2.0101 + 400 = &lt;strong&gt;502.5126&lt;/strong&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;There is only &lt;strong&gt;one degree of freedom&lt;/strong&gt;: given the marginal percentages (80% and 20% for rows, 99.5% and 0.5% for columns), the entire table is determined by the number of buyers (100,000).  That's a single value, so there's one DoF.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The chance of a chi-square variate exceeding 502.5126 is astronomically small (less than 2.3E-111).  It's best to use good statistical software to compute this value, but even Excel's calculation =CHIDIST(1, 502.5126) is close enough.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Now we can answer the questions.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Your data say nothing about the &quot;attribute&quot; affecting behavior.  This is an observational study.  All it finds is that there is an &lt;em&gt;association&lt;/em&gt; between the &quot;attribute&quot; and renewal and (thanks to the chi-square calculation) we cannot attribute that association to randomness.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If 55.65% of all buyers had the &quot;attribute&quot; and 60% of all renewers continued to have the attribute, the chi-square test would still be (barely) significant at the 5% level.  &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;It is not clear what you mean by &quot;huge difference.&quot;  Is it the proportion having the &quot;attribute&quot;?  Their actual numbers?  And what do you mean by &quot;bias&quot;?  That would depend on how you plan to use these results.  If you want to predict future renewal rates, there are all kinds of potential biases resulting from ways in which these particular 100,000 customers could differ from future customers.  But that question is not settled with a statistical analysis: it's really a matter of (marketing) faith.  &lt;em&gt;What the data say&lt;/em&gt; is that renewal is convincingly associated with the &quot;attribute&quot; within this particular population of 100,000 people.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-01-05T19:37:25.097" CommentCount="2" />
  <row Id="6015" PostTypeId="2" ParentId="6013" CreationDate="2011-01-05T20:04:41.887" Score="10" ViewCount="" Body="&lt;p&gt;Based on the way you phrase the question&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;are outliers not necessarily the best&#xA;  way to attack the problem of finding&#xA;  'badness'?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It is not clear that you are looking for outliers. For example, it seems that you are interested in machines performing above/below some threshold. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an example, if all of your servers were at 98 $\pm$ 0.1 % availability, a server at 100% availability would be an outlier, as would a server at 97.6% availability. But these may be within your desired limits.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, there may be good reasons apriori to want to be notified of any server at less than 95% availability, whether or not there is one or many servers below this threshold.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For this reason, a search for outliers may not provide the information that you are interested in. The thresholds could be determined statistically based on historical data, e.g. by modeling error rate as poisson or percent availability as beta variables. In an applied setting, these thresholds could probably be determined based on performance requirements.&lt;/p&gt;&#xA;" OwnerUserId="1381" LastEditorUserId="1381" LastEditorDisplayName="" LastEditDate="2011-01-05T20:14:04.220" LastActivityDate="2011-01-05T20:14:04.220" CommentCount="1" />
  <row Id="6016" PostTypeId="2" ParentId="6013" CreationDate="2011-01-05T20:59:42.467" Score="3" ViewCount="" Body="&lt;p&gt;A simple way to find anomalous servers would be to assume they are identically distributed, estimate the population parameters, and sort them according to their likelihoods, ascending. Column likelihoods would be combined either with their product or their minimum (or some other T-norm). This works pretty well as long as outliers are rare. For outlier detection itself, stable population parameters are usually estimated iteratively by dropping any discovered outliers, but that's not vital as long as you're manually inspecting the list and thereby avoiding thresholding.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the likelihoods, you might try Beta for the proportions and Poisson for the rates.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As pointed out by David, outlier detection is not quite the same as reliability analysis, which would flag all servers that exceed some threshold. Furthermore, some people would approach the problem trough loss functions - defining the pain you feel when some server is at 50% availability or 500 error rate, and then rank them according to that pain.&lt;/p&gt;&#xA;" OwnerUserId="2456" LastActivityDate="2011-01-05T20:59:42.467" />
  <row Id="6017" PostTypeId="2" ParentId="2691" CreationDate="2011-01-05T21:11:38.220" Score="3" ViewCount="" Body="&lt;p&gt;From someone who has used PCA a lot (and tried to explain it to a few people as well) here's an example from my own field of neuroscience.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When we're recording from a person's scalp we do it with 64 electrodes. So, in effect we have 64 numbers in a list that represent the voltage given off by the scalp. Now since we record with microsecond precision, if we have a 1-hour experiment (often they are 4 hours) then that gives us 10e6 * 60^3 == 216,000,000,000 time points at which a voltage was recorded at each electrode so that now we have a 216,000,000,000 x 64 matrix. Since a major assumption of PCA is that your variables are correlated, it is a great technique to reduce this ridiculous amount of data to an amount that is tractable. As has been said numerous times already, the eigenvalues represent the amount of variance explained by the variables (columns). In this case an eigenvalue represents the variance in the voltage at a particular point in time contributed by a particular electrode. So now we can say, &quot;Oh, well electrode &lt;code&gt;x&lt;/code&gt; at time point &lt;code&gt;y&lt;/code&gt; is what we should focus on for further analysis because that is where the most change is happening&quot;. Hope this helps. Loving those regression plots!&lt;/p&gt;&#xA;" OwnerUserId="2660" LastActivityDate="2011-01-05T21:11:38.220" />
  <row Id="6018" PostTypeId="1" CreationDate="2011-01-06T00:08:12.783" Score="0" ViewCount="49" Body="&lt;p&gt;Assume a consumer can independently exist in only 1 of 4 non-uniformally distributed buying states at any given point during a 1 month period. The overall averages for each buying state are known only at month end. What's the recommended approach to simulate the buying state allocations for 1,000 consumers on any single day using the prior month's performance?&lt;/p&gt;&#xA;" OwnerUserId="2663" LastActivityDate="2011-01-06T00:08:12.783" ClosedDate="2011-04-11T01:49:27.357" Title="Simulating 1,000 Consumers Likely Segment Allocation: Best Function?" Tags="&lt;distributions&gt;&lt;forecasting&gt;&lt;markov-process&gt;&lt;simulation&gt;" CommentCount="1" />
  <row Id="6019" PostTypeId="2" ParentId="6013" CreationDate="2011-01-06T04:54:35.017" Score="2" ViewCount="" Body="&lt;p&gt;Identifying a given data point as an outlier implies that there is some data generating process or model from which the data are expected to come from. It sounds like you are not sure what those models are for the given metrics and clusters you are concerned about. So, here is what I would consider exploring: &lt;a href=&quot;http://controls.engin.umich.edu/wiki/index.php/SPC%3a_Basic_control_charts%3a_theory_and_construction,_sample_size,_x-bar,_r_charts,_s_charts&quot; rel=&quot;nofollow&quot;&gt;statistical process control charts&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The idea here would be to collect the&lt;br&gt;&#xA;   - %Availability&lt;br&gt;&#xA;   - Requests/Sec&lt;br&gt;&#xA;   - Errors/Sec&lt;br&gt;&#xA;   - %Memory_Utilization&lt;/p&gt;&#xA;&#xA;&lt;p&gt;metrics for each of your clusters. For each metric, create a subset of the data that only includes values that are &quot;reasonable&quot; or in control. Build the charts for each metric based on this in-control data. Then you can start feeding live data to your charting code and visually assess if the metrics are in control or not. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, visually doing this for multiple metrics across many clusters may not be feasible, but this could be a good way to start to learn about the dynamics you are faced with. You might then create a notification service for clusters with metrics that go out of control. Along these lines, I have played with using neural networks to automatically classify control chart patterns as being OK vs some specific flavor of out-of-control (e.g. %availability trending down or cyclic behavior in errors/sec). Doing this gives you the advantages of statistical process control charts (long used in manufacturing settings) but eases the burden of having to spend lots of time actually looking at charts, since you can train a neural network to classify patterns based upon your expert interpretation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for code, there is the &lt;a href=&quot;http://pypi.python.org/pypi/spc/0.3&quot; rel=&quot;nofollow&quot;&gt;spc package on pypi&lt;/a&gt; but I do not have any experience using this. My toy example of using neural networks (naive Bayes too) &lt;a href=&quot;http://forums.vni.com/showthread.php?t=4215&quot; rel=&quot;nofollow&quot;&gt;can be found here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1080" LastActivityDate="2011-01-06T04:54:35.017" CommentCount="1" />
  <row Id="6020" PostTypeId="1" AcceptedAnswerId="6031" CreationDate="2011-01-06T05:27:21.180" Score="1" ViewCount="196" Body="&lt;p&gt;I'm a programmer with little statistical background, and I'm trying to create something similar to what facebook did recently (with other data):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.facebook.com/notes/facebook-data-team/whats-on-your-mind/477517358858&quot; rel=&quot;nofollow&quot;&gt;http://www.facebook.com/notes/facebook-data-team/whats-on-your-mind/477517358858&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That is to be able to find correlation between one variable (age) and bunch of other variables or categories (type of words).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the first chart, age vs. word type, I'm guessing they started out with data that looked like:&#xA;(header)&#xA;Status Update, Age, Word Cat1 percentage, Word Cat2 percentage, Word Cat3 percentage, etc..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and 1 million of such rows.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From what I understand correlation allows you to compare a variable to another variable, so how can I compare one variable (age) to a bunch of others?&lt;/p&gt;&#xA;" OwnerUserId="2664" LastActivityDate="2011-01-06T12:50:15.300" Title="Doing correlation on one variable vs many" Tags="&lt;r&gt;&lt;correlation&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="6021" PostTypeId="1" AcceptedAnswerId="6030" CreationDate="2011-01-06T06:44:19.350" Score="8" ViewCount="249" Body="&lt;p&gt;Hi, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;THis is a data visualization question. I have a database that contains some data that is constantly revised (online update).&#xA;What is the best way in R to update a graph every let say 5 or 10 seconds. (without plotting again the all thing is possible)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;fRed&lt;/p&gt;&#xA;" OwnerUserId="1709" LastEditorUserId="1709" LastEditorDisplayName="" LastEditDate="2011-05-09T04:55:43.877" LastActivityDate="2011-05-09T04:55:43.877" Title="R: update a graph dynamically" Tags="&lt;r&gt;&lt;data-visualization&gt;" AnswerCount="2" FavoriteCount="3" />
  <row Id="6022" PostTypeId="1" AcceptedAnswerId="6038" CreationDate="2011-01-06T08:11:22.483" Score="4" ViewCount="274" Body="&lt;p&gt;What methods can I use to infer a distribution if I know only three percentiles?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, I know that in a certain data set, the fifth percentile is 8,135, the 50th percentile is 11,259, and the 95th percentile is 23,611.  I want to be able to go from any other number to its percentile.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's not  my data, and those are all the statistics I have.  It's clear that the distribution isn't normal.  The only other information I have is that this data represents government per-capita funding for different school districts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know enough about statistics to know that this problem has no definite solution, but not enough to know how to go about finding good guesses.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would a lognormal distribution be appropriate?  What tools can I use to perform the regression (or do I need to do it myself)?&lt;/p&gt;&#xA;" OwnerUserId="2665" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-01-07T13:49:30.270" LastActivityDate="2011-01-07T13:49:30.270" Title="Estimating a distribution based on three percentiles" Tags="&lt;r&gt;&lt;regression&gt;&lt;percentile&gt;" AnswerCount="5" CommentCount="1" />
  <row Id="6023" PostTypeId="2" ParentId="6021" CreationDate="2011-01-06T08:14:27.433" Score="6" ViewCount="" Body="&lt;p&gt;For offline visualization, you can generate PNG files and convert them to an animated GIF using &lt;a href=&quot;http://www.imagemagick.org/&quot; rel=&quot;nofollow&quot;&gt;ImageMagick&lt;/a&gt;. I used it for demonstration (this redraw all data, though):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;source(url(&quot;http://aliquote.org/pub/spin_plot.R&quot;))&#xA;dd &amp;lt;- replicate(3, rnorm(100))&#xA;spin.plot(dd)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This generates several PNG files, prefixed with &lt;code&gt;fig&lt;/code&gt;. Then, on an un*x shell, &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;convert -delay 20 -loop 0 fig*.png sequence.gif&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;gives this animation (which is inspired from &lt;em&gt;Modern Applied Biostatistical Methods using S-Plus&lt;/em&gt;, S. Selvin, 1998):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/Bjauf.gif&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another option which looks much more promising is to rely on the &lt;a href=&quot;http://cran.r-project.org/web/packages/animation/index.html&quot; rel=&quot;nofollow&quot;&gt;animation&lt;/a&gt; package. There is an example with a &lt;a href=&quot;http://animation.yihui.name/ts%3amoving_window_ar&quot; rel=&quot;nofollow&quot;&gt;Moving Window Auto-Regression&lt;/a&gt; that should let you go started with.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-01-06T08:14:27.433" />
  <row Id="6024" PostTypeId="1" CreationDate="2011-01-06T08:51:17.610" Score="5" ViewCount="124" Body="&lt;p&gt;Given observations of $\{y, x_1, x_2, \cdots, x_n\}$, we can always do a linear regression and get all the coefficients $\{c_i\}$ for the model&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$y = c_0 + c_1 x_1 + \cdots + c_n x_n.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, this may not be the best answer. Let me explain it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When we are doing a regression, we have estimates $\{d_i\}$ for the standard deviations of the coefficients $\{c_i\}$ and it may turn out, in my particular problem, that most of these coefficients have low $t$-values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, in my problem, I already know the underlying model is more like&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$y = \Sigma_i c_i (x_{m_i} - x_{n_i})$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;such as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$y = c_1 (x_3 - x_4) + c_2 (x_1 - x_9)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and the problem is I don't know $\{m_i\}$ and $\{n_i\}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That is, in my strange case, if I already know it is of the form $y = c_1 (x_3 - x_4) + c_2 (x_1 - x_9)$, then when I find $c_1$ and $c_2$, I will find them to have high $t$-values. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Nevertheless I don't know $(x_3 - x_4)$ and $(x_1 - x_9)$ are the &quot;special&quot; combinations. And if I just solve $y = c_1 x_1 + \cdots + c_9 x_9$, I will find all $\{c_i\}$ to have low $t$-values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(The reason for this strange phenomenon is, my $\{x_i\}$ have significant correlations with each other.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems that I can solve the model&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$y = c_{1,2} (x_1 - x_2) + c_{1,3} (x_1 - x_3) + \cdots + c_{4,6} (x_4 - x_6) + \cdots$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and find all $\{c_{i,j}\}$ with high $t$-values. But then there will be $36$ coefficients $\{c_{i,j}\}$ instead of $9$. I wonder if there are faster methods?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you.&lt;/p&gt;&#xA;" OwnerUserId="2666" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-01-06T16:02:04.530" LastActivityDate="2011-01-07T05:26:54.567" Title="Effectively fitting this kind of model: $y = c_1 (x_3 - x_4) + c_2 (x_1 - x_9)$" Tags="&lt;regression&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="6025" PostTypeId="2" ParentId="6022" CreationDate="2011-01-06T08:56:43.687" Score="4" ViewCount="" Body="&lt;p&gt;For a lognormal the ratio of the 95th percentile to the median is the same as the ratio of the median to the 5th percentile. That's not even nearly true here so lognormal wouldn't be a good fit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You have enough information to fit a distribution with three parameters, and you clearly need a skew distribution. For analytical simplicity, I'd suggest the &lt;a href=&quot;http://en.wikipedia.org/wiki/Shifted_log-logistic_distribution#Shifted_log-logistic_distribution&quot; rel=&quot;nofollow&quot;&gt;shifted log-logistic distribution&lt;/a&gt; as its &lt;a href=&quot;http://en.wikipedia.org/wiki/Quantile_function&quot; rel=&quot;nofollow&quot;&gt;quantile function&lt;/a&gt; (i.e. the inverse of its cumulative distribution function) can be written in a reasonably simple closed form, so you should be able to get closed-form expressions for its three parameters in terms of your three quantiles with a bit of algebra (i'll leave that as an exercise!). This distribution is used in flood frequency analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This isn't going to give you any indication of the uncertainty in the estimates of the other quantiles though. I don't know if you need that, but as a statistician I feel I should be able to provide it, so I'm not really satisfied with this answer. I certainly wouldn't use this method, or probably &lt;em&gt;any&lt;/em&gt; method, to extrapolate (much) outside the range of the 5th to 95th percentiles.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-01-06T08:56:43.687" CommentCount="5" />
  <row Id="6026" PostTypeId="1" AcceptedAnswerId="6028" CreationDate="2011-01-06T09:12:48.257" Score="6" ViewCount="794" Body="&lt;p&gt;Hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm a medical student trying to understand statistics(!) - so please be gentle! ;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm writing an essay containing a fair amount of statistical analysis including survival analysis (Kaplan-Meier, Log-Rank and Cox regression).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I ran a Cox regression on my data trying to find out if I can find a significant difference between the deaths of patients in two groups (high risk or low risk patients).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I added several covariates to the Cox regression to control for their influence. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Risk (Dichotomous)&#xA;Gender (Dichotomous)&#xA;Age at operation (Integer level)&#xA;Artery occlusion (Dichotomous)&#xA;Artery stenosis (Dichotomous)&#xA;Shunt used in operation (Dichotomous)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I removed Artery occlusion from the covariates list because its SE was extremely high (976). All other SEs are between 0,064 and 1,118. This is what I get:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;                    B       SE      Wald    df  Sig.    Exp(B)  95,0% CI for Exp(B)&#xA;                                                                Lower   Upper&#xA;    risk            2,086   1,102   3,582   1   ,058    8,049   ,928    69,773&#xA;    gender         -,900    ,733    1,508   1   ,220    ,407    ,097    1,710&#xA;    op_age          ,092    ,062    2,159   1   ,142    1,096   ,970    1,239&#xA;    stenosis        ,231    ,674    ,117    1   ,732    1,259   ,336    4,721&#xA;    op_shunt        ,965    ,689    1,964   1   ,161    2,625   ,681    10,119&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I know that risk is only borderline-significant at 0,058. But besides that how do I interpret the Exp(B) value? I read an article on logistic regression (which is somewhat similar to Cox regression?) where the Exp(B) value was interpreted as: &quot;Being in the high-risk group includes an 8-fold increase in possibility of the outcome,&quot; which in this case is death. Can I say that my high-risk patients are 8 times as likely to die earlier than ... what?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please help me! ;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By the way I'm using SPSS 18 to run the analysis.&lt;/p&gt;&#xA;" OwnerUserId="2652" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-01-07T03:19:15.610" LastActivityDate="2011-02-25T09:50:26.273" Title="How do I interpret Exp(B) in Cox Regression?" Tags="&lt;regression&gt;" AnswerCount="2" />
  <row Id="6027" PostTypeId="2" ParentId="6022" CreationDate="2011-01-06T10:12:31.150" Score="2" ViewCount="" Body="&lt;p&gt;About the only things you can infer from the data is that the distribution is nonsymmetric. You can't even tell whether those quantiles came from a fitted distribution or just the ecdf.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If they came from a fitted distribution, you could try all the distributions you can think of and see if any match. If not, there's not nearly enough information. You could interpolate a 2nd degree polynomial or a 3rd degree spline for the quantile function and use that, or come up with a theory as to the distribution family and match quantiles, but any inferences you would make with these methods would be deeply suspect.&lt;/p&gt;&#xA;" OwnerUserId="2456" LastActivityDate="2011-01-06T10:12:31.150" CommentCount="2" />
  <row Id="6028" PostTypeId="2" ParentId="6026" CreationDate="2011-01-06T10:49:53.407" Score="10" ViewCount="" Body="&lt;p&gt;Generally speaking, $\exp(\hat\beta_1)$ is the ratio of the hazards between two individuals whose values of $x_1$ differ by one unit when all other covariates are held constant. The parallel with other linear models is that in Cox regression the hazard function is modeled as $h(t)=h_0(t)\exp(\beta&amp;#39;x)$, where $h_0(t)$ is the baseline hazard. This is equivalent to say that $\log(\text{group hazard}/\text{baseline hazard})=\log\big((h(t)/h_0(t)\big)=\sum_i\beta_ix_i$. Then, a unit increase in the independent variable $i$ is associated with $\beta_i$ increase in the log hazard rate.&#xA;The regression coefficient allow thus to quantify the log of the hazard in the treatment group (compared to the control or placebo group), accounting for the covariates included in the model; it is interpreted as a relative risk (assuming no time-varying coefficients).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the case of logistic regression, the regression coefficient reflects the log of the &lt;a href=&quot;http://en.wikipedia.org/wiki/Odds_ratio#Role_in_logistic_regression&quot; rel=&quot;nofollow&quot;&gt;odds-ratio&lt;/a&gt;, hence the interpretation as an k-fold increase in risk. So yes, the interpretation of hazard ratios shares some resemblance with the interpretation of odds ratios.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Be sure to check Dave Garson's website where there is some good material on &lt;a href=&quot;http://faculty.chass.ncsu.edu/garson/PA765/cox.htm&quot; rel=&quot;nofollow&quot;&gt;Cox Regression&lt;/a&gt; with SPSS.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-01-06T10:49:53.407" CommentCount="5" />
  <row Id="6029" PostTypeId="2" ParentId="6020" CreationDate="2011-01-06T12:00:24.480" Score="2" ViewCount="" Body="&lt;p&gt;Correlation is a rather vague word meaning the fact that one variable is dependent on the other; in many cases this is just a synonym for &lt;a href=&quot;http://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient&quot; rel=&quot;nofollow&quot;&gt;Pearson correlation coefficient&lt;/a&gt;, which assumes linear dependence (i.e. $y=A \cdot x+B$), so things like &quot;when x increses, y increases&quot; (cor-&gt; +1) or &quot;when x increases, y decreases&quot; (cor-&gt; -1).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regression on the other hand is a problem of finding a function that describes dependence between variables; in a linear problem it would be finding values of A and B, but in general it can be anything -- finding a period of some periodic phenomenon by fitting it with $y=\sin(\omega t+\phi)$ is also a regression.&#xA;Yet regression can be used to assess correlation -- if some regression model  fits the data well and in a significant way, it means there is correlation; this can be numerised with explained variance for instance. (In general it is a long and complex story)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That's all for oversized intro; going back to your question, this Facebook page shows Pearson correlations, which can be calculated without any regression with &lt;code&gt;cor&lt;/code&gt; R function (also the formula is quite simple and available on the Wiki page). And as David wrote, counting correlation between $N$ variables boils down to counting correlations of all possible pairs.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2011-01-06T12:00:24.480" />
  <row Id="6030" PostTypeId="2" ParentId="6021" CreationDate="2011-01-06T12:10:37.987" Score="6" ViewCount="" Body="&lt;p&gt;Assuming you want to update R &lt;code&gt;windows()&lt;/code&gt; or &lt;code&gt;x11()&lt;/code&gt; graph, you can use functions like &lt;code&gt;points()&lt;/code&gt; and &lt;code&gt;lines()&lt;/code&gt; to add new points or extend lines on a graph without redraw; yet note that this won't change the axes range to accommodate points that may go out of view. In general it is usually a good idea to make the plotting itself instantaneous -- for instance by moving computation effort into making some reduced middle representation which can be plotted rapidly, like density map instead of huge number of points or reducing resolution of line plots (this may be complex though).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For holding R session for a certain time without busy wait, use &lt;code&gt;Sys.sleep()&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2011-01-06T12:10:37.987" CommentCount="4" />
  <row Id="6031" PostTypeId="2" ParentId="6020" CreationDate="2011-01-06T12:21:07.763" Score="5" ViewCount="" Body="&lt;p&gt;To help you get started with the visualization, here is a snippet of R code with simulated data (a matrix with age and counts for 20 words, arranged in columns, for 100 subjects). The computation are done as proposed my @mbq (correlation).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;n &amp;lt;- 100  # No. subjects&#xA;k &amp;lt;- 20   # No. words&#xA;words &amp;lt;- paste(&quot;word&quot;, 1:k, sep=&quot;&quot;)&#xA;df &amp;lt;- data.frame(age=rnorm(n, mean=25, sd=5), &#xA;                 replicate(k, sample(1:10, n, rep=T)))&#xA;colnames(df)[2:(k+1)] &amp;lt;- words&#xA;robs &amp;lt;- sort(cor(as.matrix(df))[-1,1])&#xA;&#xA;library(lattice)&#xA;my.cols &amp;lt;- colorRampPalette(c(&quot;red&quot;,&quot;blue&quot;))&#xA;res &amp;lt;- data.frame(robs=robs, x=seq(1,20), y=rep(1,20))&#xA;&#xA;trellis.par.set(clip=list(panel=&quot;off&quot;), axis.line=list(col=&quot;transparent&quot;))&#xA;levelplot(robs~y*x, data=res, col.regions=my.cols, &#xA;          colorkey=F, xlab=&quot;&quot;, ylab=&quot;&quot;, scales=list(draw=F),&#xA;          panel=function(...) {&#xA;            panel.levelplot(...)&#xA;            panel.text(x=rep(1, k), y=seq(1, k), lab=rownames(res))&#xA;         })&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/n23ot.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The above picture was saved as PDF, setting the margins to 1, and croped with &lt;code&gt;pdfcrop&lt;/code&gt; from my TeXLive distribution.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pdf(&quot;1.pdf&quot;)&#xA;op &amp;lt;- par(mar=c(1,1,1,1))&#xA;(...)&#xA;par(op)&#xA;dev.off()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I guess it would not be too difficult to make a similar looking chart with &lt;code&gt;barchart()&lt;/code&gt; from &lt;a href=&quot;http://cran.r-project.org/web/packages/lattice/index.html&quot; rel=&quot;nofollow&quot;&gt;lattice&lt;/a&gt;, or &lt;code&gt;ggfluctuation()&lt;/code&gt; or any other &lt;code&gt;qplot()&lt;/code&gt; from &lt;a href=&quot;http://cran.r-project.org/web/packages/ggplot2/index.html&quot; rel=&quot;nofollow&quot;&gt;ggplot2&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-01-06T12:50:15.300" LastActivityDate="2011-01-06T12:50:15.300" CommentCount="1" />
  <row Id="6032" PostTypeId="2" ParentId="6022" CreationDate="2011-01-06T13:10:16.490" Score="1" ViewCount="" Body="&lt;p&gt;The use of quantiles to estimate parameters of a priori distributions is discussed in the literature on human response time measurement as &quot;quantile maximum probability estimation&quot; (QMPE, though originally erroneously dubbed &quot;quantile maximum likelihood estimation&quot;, QMLE), discussed at length by &lt;a href=&quot;http://hdl.handle.net/1959.13/27899&quot; rel=&quot;nofollow&quot;&gt;Heathcote and colleagues&lt;/a&gt;. You could fit a number of different a priori distributions (ex-Gaussian, shifted Lognormal, Wald, and Weibull) then compare the sum log likelihoods of the resulting best fits for each distribution to find the distribution flavor that seems to yield the best fit.&lt;/p&gt;&#xA;" OwnerUserId="364" LastActivityDate="2011-01-06T13:10:16.490" CommentCount="3" />
  <row Id="6033" PostTypeId="1" CreationDate="2011-01-06T13:53:16.580" Score="5" ViewCount="266" Body="&lt;p&gt;I came across a picture of an application prototype that finds significant changes (&quot;trends&quot; - not spikes/outliers) in traffic data:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/sGYeF.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to write a program (Java, optionally R) that is able to do the same - but because my statistic skills are a little rusty, I need to dig into this topic again.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;What approach/algorithm should I use/research therefore?&lt;/em&gt; &lt;/p&gt;&#xA;" OwnerUserId="2667" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-02-28T12:37:08.800" LastActivityDate="2011-04-01T09:08:51.430" Title="Detect changes in time series" Tags="&lt;time-series&gt;&lt;change-point&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="1" />
  <row Id="6035" PostTypeId="2" ParentId="5954" CreationDate="2011-01-06T14:09:59.477" Score="0" ViewCount="" Body="&lt;p&gt;Thank you all very much.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to spend a little time thinking about your answers and perhaps, if you don't mind my try writing them in my terms for your views. I like this description of the process as coming from the Taylor series. I have had to pick up my knowledge of regression haphazzardly and through Economics and Mathematics for Economists and the link with Taylor is noteable by it's absense.&lt;/p&gt;&#xA;" OwnerUserId="2629" LastActivityDate="2011-01-06T14:09:59.477" CommentCount="1" />
  <row Id="6036" PostTypeId="2" ParentId="726" CreationDate="2011-01-06T14:22:03.547" Score="1" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Anyone who considers arithmetical&#xA;  methods of producing random digits is,&#xA;  of course, in a state of sin.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;-- Von Neumann&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-01-06T14:22:03.547" CommunityOwnedDate="2011-01-06T14:22:03.547" />
  <row Id="6037" PostTypeId="2" ParentId="4551" CreationDate="2011-01-06T15:28:05.167" Score="3" ViewCount="" Body="&lt;p&gt;While I can relate to much of what Michael Lew says, abandoning p-values in favor of likelihood ratios still misses a more general problem--that of overemphasizing probabilistic results over effect sizes, which are required to give a result substantive meaning.  This type of error comes in all shapes and sizes and I find it to be the most insidious statistical mistake.  Drawing on J. Cohen and M. Oakes and others, I've written a piece on this at &lt;a href=&quot;http://integrativestatistics.com/insidious.htm&quot; rel=&quot;nofollow&quot;&gt;http://integrativestatistics.com/insidious.htm&lt;/a&gt; .&lt;/p&gt;&#xA;" OwnerUserId="2669" LastActivityDate="2011-01-06T15:28:05.167" CommunityOwnedDate="2011-01-06T15:28:05.167" CommentCount="4" />
  <row Id="6038" PostTypeId="2" ParentId="6022" CreationDate="2011-01-06T16:29:31.403" Score="7" ViewCount="" Body="&lt;p&gt;Using a purely statistical method to do this work will provide absolutely no additional information about the distribution of school spending: the result will merely reflect an arbitrary choice of algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;You need more data&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is easy to come by: use data from previous years, from comparable districts, whatever.  For example, federal spending on 14866 school districts in 2008 is available from the &lt;a href=&quot;http://www.census.gov/govs/school/&quot; rel=&quot;nofollow&quot;&gt;Census site&lt;/a&gt;.  It shows that across the country, total per-capita (enrolled) federal revenues were approximately lognormally distributed, but breaking it down by state shows substantial variation (&lt;em&gt;e.g.&lt;/em&gt;, log spending in Alaska has negative skew while log spending in Colorado has strong positive skew).  Use those data to characterize the likely form of distribution and then fit your quantiles to that form.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're even close to the right distributional form, then you should be able to reproduce the quantiles accurately by fitting one or at most two parameters.  The best technique for finding the fit will depend on what distributional form you use, but--far more importantly--it will depend on &lt;strong&gt;what you intend to use the results for&lt;/strong&gt;.  Do you need to estimate an average spending amount?  Upper and lower limits on spending?  Whatever it is, you want to adopt some measure of goodness of fit that will give you the best chance of making good decisions with your results.  For example, if your interest is focused in the upper 10% of all spending, you will want to fit the 95th percentile accurately and you might care little about fitting the 5th percentile.  No sophisticated fitting technique will make these considerations for you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course no one can legitimately guarantee that this data-informed, decision-oriented method will perform any better (or any worse) than some statistical recipe, but--unlike a purely statistical approach--this method has a basis grounded in reality, with a focus on your needs, giving it some credibility and defense against criticism.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-01-06T19:32:30.900" LastActivityDate="2011-01-06T19:32:30.900" CommentCount="5" />
  <row Id="6039" PostTypeId="1" CreationDate="2011-01-06T17:11:11.887" Score="2" ViewCount="120" Body="&lt;p&gt;I am kinda new to stats and understand random sampling, however I am just learning PCA and wondering if it is just a more sophisticated form of sampling? In other words if I have a large data set. and take a random sample would I then apply PCA to it? Or just apply PCA to the data set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for any help..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Mike&lt;/p&gt;&#xA;" OwnerUserId="2674" LastActivityDate="2011-01-07T01:34:31.820" Title="Random Sampling and PCA" Tags="&lt;pca&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="6040" PostTypeId="2" ParentId="6024" CreationDate="2011-01-06T17:24:19.430" Score="4" ViewCount="" Body="&lt;p&gt;You can not use the 36-coefficient model, and not because it's going to be slow. Speed is the least of your worries here. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The real trouble is that you've taken an already under-determined problem (because of the correlations), and converted it into a problem which is severely under-determined for &lt;em&gt;any&lt;/em&gt; data, because of linear dependencies. Simply put, $x_1-x_2=(x_1-x_3)-(x_2-x_3)$, so you can only determine 2 out of the 3 coefficients for these terms in the best case. The only way to fix this will be to prescribe some artificial regularization condition, like having minimal $\sum c_i^2$, or whatever may be right in your case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems to me that you might be better off if you start by analyzing the correlation matrix and first figuring out which terms of the form $c_i-c_j$ should really appear in your problem.&lt;/p&gt;&#xA;" OwnerUserId="2658" LastActivityDate="2011-01-06T17:24:19.430" CommentCount="1" />
  <row Id="6041" PostTypeId="2" ParentId="6039" CreationDate="2011-01-06T17:38:35.500" Score="2" ViewCount="" Body="&lt;p&gt;Well, if you were interested in cross validation (i.e. how much your model will predict on a different sample), then you could use PCA on a random subset and then fit that model to the rest of your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That being said, PCA is a tool for summarising a covariance matrix in a smaller matrix, so it may not be the best thing to use. Factor Analysis is a better approach if you want to figure out what is happening i&lt;/p&gt;&#xA;" OwnerUserId="656" LastActivityDate="2011-01-06T17:38:35.500" />
  <row Id="6042" PostTypeId="1" AcceptedAnswerId="6043" CreationDate="2011-01-06T17:52:31.127" Score="3" ViewCount="157" Body="&lt;p&gt;(I'm asking this question for a friend,  honest...)&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Is there an easy way to convert from&#xA;  an SPSS file to a SAS file, which&#xA;  preserves the formats AND labels?&#xA;  Saving as a POR file gets me the&#xA;  labels (I think) but not the POR file.&#xA;  I tried to save to a SAS7dat file but&#xA;  it didn't work. Thanks,&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="253" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-01-07T11:01:23.480" LastActivityDate="2011-01-07T11:01:23.480" Title="Converting an SPSS file to a SAS file?" Tags="&lt;spss&gt;&lt;sas&gt;" AnswerCount="2" CommentCount="3" />
  <row Id="6043" PostTypeId="2" ParentId="6042" CreationDate="2011-01-06T18:47:42.000" Score="1" ViewCount="" Body="&lt;p&gt;I would just suggest they make the syntax to relabel and reformat the variables. You can use the command, &lt;code&gt;display dictionary.&lt;/code&gt; in PASW (aka SPSS) to output the dictionary in a table that you can copy and paste the variable names and labels. Looking at this &lt;a href=&quot;http://www.ats.ucla.edu/stat/sas/modules/labels.htm&quot; rel=&quot;nofollow&quot;&gt;example&lt;/a&gt; of making SAS labels it should be as simple as pasting the text in the appropriate place.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Formats may be slightly harder, but I could likely give a suggestion if pointed to a code sample of formats in SAS (if copy and paste from the display dictionary command won't suffice for value labels or data formats).&lt;/p&gt;&#xA;" OwnerUserId="1036" LastEditorUserId="1036" LastEditorDisplayName="" LastEditDate="2011-01-06T18:59:07.933" LastActivityDate="2011-01-06T18:59:07.933" CommentCount="2" />
  <row Id="6044" PostTypeId="1" AcceptedAnswerId="6062" CreationDate="2011-01-06T19:35:49.930" Score="-1" ViewCount="396" Body="&lt;p&gt;Hi there. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As the title says, I'd like to calculate the percentage difference for two sets of points. For example, suppose I have $S_{1}=\{(1,x_{1}),(2,x_{2}),(3,x_{3})\}$ and $S_{2}=\{(1,y_{1}),(2,y_{2}),(3,y_{3})\}$. How can I know the difference in percentage between both sets of data. What is the correct way to do that? Is that kind of assessment meaningful to establish to which degree of precision a set of data is preferred over the other?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my particular case, $S_{1}$ is simply a set of numerical results obtained by &lt;a href=&quot;http://en.wikipedia.org/wiki/Direct_simulation_Monte_Carlo&quot; rel=&quot;nofollow&quot;&gt;DSMC&lt;/a&gt; and $S_{2}$ was obtained by a theoretical result. I'd like to quantify how much difference exist between each other in order to establish when it is convenient to use one or the other.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By &quot;difference in percentage&quot; I mean &lt;a href=&quot;http://en.wikipedia.org/wiki/Percent_difference&quot; rel=&quot;nofollow&quot;&gt;percent difference&lt;/a&gt;. Hopefully that clarifies a bit the question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another way to formulate my question would be: How can I arrive to conclusions such as &quot;The results from experiment A are inaccurate by 10% with respect to experiment B&quot;, when experiment A and B are a set of values.&lt;/p&gt;&#xA;" OwnerUserId="2676" LastEditorUserId="2676" LastEditorDisplayName="" LastEditDate="2011-01-07T01:03:12.160" LastActivityDate="2011-01-07T13:45:52.677" Title="Calculate percentage difference for two sets of points" Tags="&lt;percentile&gt;" AnswerCount="2" CommentCount="9" />
  <row Id="6045" PostTypeId="2" ParentId="5922" CreationDate="2011-01-06T19:40:32.547" Score="6" ViewCount="" Body="&lt;p&gt;See &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Levina, E. and Bickel, P. (2004) “Maximum Likelihood Estimation of Intrinsic Dimension.” Advances in Neural Information Processing Systems 17&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://books.nips.cc/papers/files/nips17/NIPS2004_0094.pdf&quot; rel=&quot;nofollow&quot;&gt;http://books.nips.cc/papers/files/nips17/NIPS2004_0094.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Their idea is that if the data are sampled from a smooth density in $R^m$ embedded in $R^p$ with $m &amp;lt; p$, then locally the number of data points in a small ball of radius $t$ behaves roughly like a poisson process.  The rate of the process is related to the volume of the ball which in turn is related to the intrinsic dimension.&lt;/p&gt;&#xA;" OwnerUserId="1670" LastEditorUserId="1670" LastEditorDisplayName="" LastEditDate="2011-01-08T20:17:36.033" LastActivityDate="2011-01-08T20:17:36.033" CommentCount="2" />
  <row Id="6046" PostTypeId="1" CreationDate="2011-01-06T19:49:26.740" Score="3" ViewCount="275" Body="&lt;p&gt;Hi,&#xA;I have animals, that could be virgin or mated (reproductive state is the fixed factor), which I've stimulated sequentially with 4 different doses of an odour (doses are the repeated measures, the same animal was blown with 4 increasing doses of the same odorant). Then, I measure the neuronal response (variable: number of spikes) of each animal to each dose of the odorant.  This might be a typical case or repeated measures, however I have some missing values for the doses, I have not all the doses completed for some animals. For example, for animal 1, I missed recording 1 out of the 4 doses.&#xA;what can I do? I have two statistical packages: SPSS 16 or Statistical.&#xA;thanks for your help!&lt;/p&gt;&#xA;" OwnerUserId="2677" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2011-01-07T09:36:30.217" LastActivityDate="2011-01-10T14:53:32.060" Title="How to solve a case of unbalanced repeated measures?" Tags="&lt;repeated-measures&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="1" />
  <row Id="6047" PostTypeId="1" AcceptedAnswerId="6060" CreationDate="2011-01-06T19:50:41.317" Score="10" ViewCount="123" Body="&lt;p&gt;Obviously events A and B are independent iff Pr$(A\cap B)$ = Pr$(A)$Pr$(B)$.  Let's define a related quantity Q:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Q\equiv\frac{\mathrm{Pr}(A\cap B)}{\mathrm{Pr}(A)\mathrm{Pr}(B)}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So A and B are independent iff Q = 1 (assuming the denominator is nonzero).  Does Q actually have a name though?  I feel like it refers to some elementary concept that is escaping me right now and that I will feel quite silly for even asking this.&lt;/p&gt;&#xA;" OwnerUserId="2485" LastActivityDate="2011-01-07T09:38:47.577" Title="Does this quantity related to independence have a name?" Tags="&lt;probability&gt;&lt;terminology&gt;&lt;independence&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="0" />
  <row Id="6048" PostTypeId="2" ParentId="6046" CreationDate="2011-01-06T21:06:18.200" Score="2" ViewCount="" Body="&lt;p&gt;Mixed effects analysis (available in &lt;a href=&quot;http://www.r-project.org&quot; rel=&quot;nofollow&quot;&gt;R&lt;/a&gt; via the &lt;a href=&quot;http://cran.r-project.org/web/packages/lme4/index.html&quot; rel=&quot;nofollow&quot;&gt;lme4&lt;/a&gt; package, free as always) can handle missing data like this. My understanding is (possibly erroneous? Mixed effects modelling experts please feel free to provide correction) of how this is achieved is as follows: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given a set of predictor variables that are combined in some way to form a matrix that I usually call the &quot;predictor design&quot; (for example, for a completely crossed design of 3 variables with 2 levels each, the predictor design is a 2x2x2 matrix in which you expect to have each cell filled with an observed response), and given a set of &quot;units of observation&quot; (ex. individual human participants in an experiment), missing data can occur if the experiment fails to obtain an observation for a given unit in a given cell of the predictor design.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the missing data occurs in a cell of the design that involves a continuous predictor variable over which the model would typically attempt to fit a slope, the model simply goes ahead and fits the slope across those cells without missing data. I &lt;em&gt;think&lt;/em&gt; that the model also takes into account the fact that there is missing data in determining the degree of influence that given unit's data has on inference relating to that slope; that is, slopes from units with lots of missing data will by definition be based on fewer observations and thereby are expected to be less reliable estimates, a circumstance that the model takes into account when combining that estimate with those from other units with possibly more reliable estimates.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the slope cannot be computed (ex. one or fewer cells have data), or if the missing data occurs in a cell of the design that involves a categorical predictor variable, then that given unit will not influence inference of that particular slope/effect, but remains in the model for influence of slopes/effects for which the unit does have sufficient data. (Hm, now that I think of it, I'm not sure how this works in the context of missing data for cells that constitute the intercept level of contrasts when treatment contrasts are used...)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note, however, that if you are missing information about the predictor variables (eg, you have a response, but you're unsure where in the predictor design it should fall), the model will &lt;strong&gt;not&lt;/strong&gt; attempt to impute this information and will instead simply ignore that value.&lt;/p&gt;&#xA;" OwnerUserId="364" LastEditorUserId="364" LastEditorDisplayName="" LastEditDate="2011-01-10T14:53:32.060" LastActivityDate="2011-01-10T14:53:32.060" CommentCount="5" />
  <row Id="6049" PostTypeId="2" ParentId="6044" CreationDate="2011-01-06T22:25:22.827" Score="1" ViewCount="" Body="&lt;p&gt;It seems to me like you need to formulate a question you want your data to answer. Let me suggest a few (perhaps you can edit your post to reflect what questions make sense for your data):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;As the DSMC value increases, does the theoretical result also increase?&lt;/li&gt;&#xA;&lt;li&gt;If I know the value of the theoretical result, how accurately can I estimate the value of DSMC?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If the points (1,x1) and (1,y1) refer to the same measurement or the same run of the experiment, or one is an estimate of the other. One natural way to see how related they are related is to plot {(x1,y1),(x2,y2) ...}.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can read about Pearson correlation and Kendall tau and Spearman rho here:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Correlation_and_dependence&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Correlation_and_dependence&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1540" LastActivityDate="2011-01-06T22:25:22.827" CommentCount="2" />
  <row Id="6050" PostTypeId="1" AcceptedAnswerId="6052" CreationDate="2011-01-07T00:15:28.330" Score="6" ViewCount="253" Body="&lt;p&gt;I'm doing a simple AIC-based backward elimination model where some variables are categorical variables with multiple levels. These variables are modeled as a set of dummy variables. When doing backward elimination, should I be removing all the levels of a variable together? Or should I treat each dummy variable separately? And why?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a related question, step in R handles each dummy variable separately when doing backward elimination. If I wanted to remove an entire categorical variable at once, can I do that using step? Or are there alternatives to step which can handle this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="2308" LastActivityDate="2011-01-07T01:23:04.920" Title="How should I handle categorical variables with multiple levels when doing backward elimination?" Tags="&lt;model-selection&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="6051" PostTypeId="2" ParentId="4600" CreationDate="2011-01-07T00:54:44.693" Score="0" ViewCount="" Body="&lt;p&gt;In an anova context, the partial eta squared will tell what % of the Y variance is explained by a given X when controlling for all other X's.  In a regression context, you could refer to the squared partial correlation of the X of interest.  &lt;/p&gt;&#xA;" OwnerUserId="2669" LastActivityDate="2011-01-07T00:54:44.693" />
  <row Id="6052" PostTypeId="2" ParentId="6050" CreationDate="2011-01-07T01:23:04.920" Score="5" ViewCount="" Body="&lt;p&gt;I think you'd have to remove the entire categorical variable.  Imagine a logistic regression in which you're trying to predict if a person has a disease or not.  Country of birth might have a major impact on that, so you include it in your model.  If the specific USAmerican origin didn't have any impact on AIC and you dropped it, how would you calculate $\hat{y}$ for an American?  R uses reference contrasts for factors by default, so I think they'd just be calculated at the reference level (say, Botswana), if at all.  That's probably not going to end well...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A better option would be to sort out sensible encodings of country of birth beforehand - collapsing into region, continent, etc. and finding which of those is most suitable for your model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, there are many ways to misuse stepwise variable selection, so make sure that you're doing it properly.  There's plenty about that on this site, though; searching for &quot;stepwise&quot; brings up some good results.  &lt;a href=&quot;http://stats.stackexchange.com/questions/5360/stepwise-logistic-regression-and-sampling&quot;&gt;This is particularly pertinent&lt;/a&gt;, with lots of good advice in the answers.&lt;/p&gt;&#xA;" OwnerUserId="71" LastActivityDate="2011-01-07T01:23:04.920" CommentCount="4" />
  <row Id="6053" PostTypeId="2" ParentId="6039" CreationDate="2011-01-07T01:34:31.820" Score="1" ViewCount="" Body="&lt;p&gt;Principal Components Analysis is a way of distilling a large set of variables into a few topics or themes or fundamentals.  It's dimension reduction.  The only resemblance I see to sampling is that sampling also involves a kind of reduction.  &lt;/p&gt;&#xA;" OwnerUserId="2669" LastActivityDate="2011-01-07T01:34:31.820" />
  <row Id="6054" PostTypeId="2" ParentId="868" CreationDate="2011-01-07T01:55:24.827" Score="0" ViewCount="" Body="&lt;p&gt;One picayune thing that could matter down the road is, in your equation&#xA;P = 1 / [1 + e^(-35 + (4*4 + 4*4 + 3*0 + 1*1)] &lt;/p&gt;&#xA;&#xA;&lt;p&gt;you've misplaced the &quot;-&quot;:  it needs to go outside the parenthesis.  So it'd be&#xA;P = 1 / [1 + e^-(a + B1*X1 + B2*X2 + B3*X3...+ Bn*Xn)].&lt;/p&gt;&#xA;" OwnerUserId="2669" LastActivityDate="2011-01-07T01:55:24.827" />
  <row Id="6055" PostTypeId="2" ParentId="6042" CreationDate="2011-01-07T02:17:31.810" Score="0" ViewCount="" Body="&lt;p&gt;SPSS writes SAS7dat format files normally with no problems.  When you it did not work, what actually happened?&lt;/p&gt;&#xA;" OwnerUserId="2679" LastActivityDate="2011-01-07T02:17:31.810" CommentCount="1" />
  <row Id="6056" PostTypeId="1" CreationDate="2011-01-07T03:23:36.777" Score="3" ViewCount="129" Body="&lt;p&gt;Hi stackexchange users:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I try to install rpy2 in my system,&#xA;(I compile R with --enable-R-shlib and with --enable-BLAS-shlib flags)&#xA;but when I try in python console&lt;/p&gt;&#xA;&#xA;&lt;p&gt;import rpy2&#xA;import rpy2.robjects&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I got:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Traceback (most recent call last):&#xA;    File &quot;&quot;, line 1, in&#xA;    File &quot;/usr/lib/python2.6/dist-packages/rpy2/robjects/&lt;strong&gt;init&lt;/strong&gt;.py&quot;,&#xA; line 14, in&#xA;      import rpy2.rinterface as rinterface&#xA;    File &quot;/usr/lib/python2.6/dist-packages/rpy2/rinterface/&lt;strong&gt;init&lt;/strong&gt;.py&quot;,&#xA; line 75, in&#xA;      from rpy2.rinterface.rinterface import *&#xA; ImportError: libRblas.so: cannot open shared object file: No such file&#xA; or directory&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The rpy2 directory is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;rpy2.&lt;strong&gt;path&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;['/usr/lib/python2.6/dist-packages/rpy2']&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My R version is:&#xA; R version 2.12.1 Patched (2011-01-04 r53913)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My R home is:&#xA; /usr/bin/R&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My ubuntu version is:&#xA; Linux kenneth-desktop 2.6.32-27-generic #49-Ubuntu SMP Thu Dec 2&#xA; 00:51:09 UTC 2010 x86_64 GNU/Linux&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My Python version is:&#xA; Python 2.6.5 (r265:79063, Apr 16 2010, 13:57:41)&#xA; [GCC 4.4.3] on linux2&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I install rpy2 from source (sudo python setup.py build install)&#xA;I got:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;running build&#xA;running build_py&#xA;running build_ext&#xA;Configuration for R as a library:&#xA;   include_dirs: ('/usr/lib64/R/include',)&#xA;   libraries: ('R', 'Rblas', 'Rlapack')&#xA;   library_dirs: ('/usr/lib64/R/lib',)&#xA;   extra_link_args: ()&#xA;  # OSX-specific (included in extra_link_args)&#xA;   framework_dirs: ()&#xA;   frameworks: ()&#xA;running install&#xA;running install_lib&#xA;running install_data&#xA;running install_egg_info&#xA;Removing /usr/local/lib/python2.6/dist-packages/rpy2-2.1.9.egg-info&#xA;Writing /usr/local/lib/python2.6/dist-packages/rpy2-2.1.9.egg-info&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What am I doing wrong?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you for your help.&lt;/p&gt;&#xA;" OwnerUserId="2680" LastActivityDate="2011-01-07T03:52:44.340" Title="Problems with libRblas.so on ubuntu with rpy2" Tags="&lt;r&gt;&lt;python&gt;" AnswerCount="1" />
  <row Id="6057" PostTypeId="2" ParentId="6047" CreationDate="2011-01-07T03:33:37.813" Score="0" ViewCount="" Body="&lt;p&gt;Maybe you are asking how this quantity is related to the Odds Ratio,&#xA;as a quantity for measuring independence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think you are searching for &quot;Relation to statistical independence&quot;. See &lt;a href=&quot;http://en.wikipedia.org/wiki/Odds_ratio&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Odds_ratio&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2680" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2011-01-07T09:38:47.577" LastActivityDate="2011-01-07T09:38:47.577" />
  <row Id="6058" PostTypeId="2" ParentId="6056" CreationDate="2011-01-07T03:52:44.340" Score="3" ViewCount="" Body="&lt;p&gt;It looks like you tried to do things locally but didn't quite get there. I happen to maintain the Debian packages of R (which get rebuilt for Ubuntu and are accessible at &lt;a href=&quot;http://cran.r-project.org/bin/linux/ubuntu/&quot; rel=&quot;nofollow&quot;&gt;CRAN&lt;/a&gt;. These builds use external BLAS.  rpy2 then builds just fine as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would recommend that you read the &lt;a href=&quot;http://cran.r-project.org/bin/linux/ubuntu/&quot; rel=&quot;nofollow&quot;&gt;README&lt;/a&gt;, try to install &lt;code&gt;r-base-core&lt;/code&gt; and &lt;code&gt;r-base-dev&lt;/code&gt; from the repositories and then try to install &lt;code&gt;rpy2&lt;/code&gt; from source.  Or live with the slightly older &lt;code&gt;rpy2&lt;/code&gt; package in Ubuntu.&lt;/p&gt;&#xA;" OwnerUserId="334" LastActivityDate="2011-01-07T03:52:44.340" />
  <row Id="6059" PostTypeId="2" ParentId="6024" CreationDate="2011-01-07T05:07:13.817" Score="5" ViewCount="" Body="&lt;p&gt;(This response picks up where @AVB, who has provided useful comments, left off by suggesting we need to figure out which differences $X_i - X_j$ ought to be included among the independent variables.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The big question here is &lt;strong&gt;what is an &lt;em&gt;effective&lt;/em&gt; method&lt;/strong&gt; to identify the model.  Later we can worry about faster methods.  (But regression is so fast that you could process dozens of variables for millions of records in a matter of seconds.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To make sure I'm not going astray, and to illustrate the procedure, I &lt;strong&gt;simulated a dataset&lt;/strong&gt; like yours, only a little simpler.  It consists of 60 independent draws from a common multivariate normal distribution with five unit-variance variables $Z_1, Z_2, Z_3, Z_4,$ and $Y$.  The first two variables are independent of the second two and have correlation coefficient 0.9.  The second two variables have correlation coefficient -0.9.  The correlations between $Z_i$ and $Y$ are 0.5, 0.5, 0.5, and -0.5.  Then--this changes nothing essential but makes the data a little more interesting--I rescaled the variables, thus: $X_1 = Z_1, X_2 = 2 Z_2, X_3 = 3 Z_3, X_4 = 4 Z_4$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's begin by establishing that this simulation emulates the stated problem.  Here is a scatterplot matrix.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/Y3AqA.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The full regression of $Y$ against the $X_i$ is highly significant ($F(4, 55) = 15.28,\ p &amp;lt; 0.0001$) but all four t-values equal 1.24 ($p = 0.222$), which is not significant at all.  The estimated coefficients are 0.26, 0.13, 0.088, and -0.066 (rounded to two sig figs).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Here is my proposal&lt;/strong&gt;: systematically combine variables in pairs (six pairs in this case, 36 pairs for nine variables), &lt;em&gt;one pair at a time.&lt;/em&gt;  Regress a pair &lt;em&gt;along with all remaining variables,&lt;/em&gt; seeking highly significant results for the pairs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is a &quot;pair&quot;?  It is the linear combination suggested by the estimated coefficients.  In this case, they are&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\eqalign{&#xD;&#xA;X_{12} =&amp;amp; X_1 / 0.26 &amp;amp;+ X_2 / 0.13 \cr&#xD;&#xA;X_{13} =&amp;amp; X_1 / 0.26 &amp;amp;+ X_3 / 0.088 \cr&#xD;&#xA;X_{14} =&amp;amp; X_1 / 0.26 &amp;amp;- X_4 / 0.066 \cr&#xD;&#xA;X_{23} =&amp;amp; X_2 / 0.13 &amp;amp;+ X_3 / 0.088 \cr&#xD;&#xA;X_{24} =&amp;amp; X_2 / 0.13 &amp;amp;- X_4 / 0.066 \cr&#xD;&#xA;X_{34} =&amp;amp; X_3 / 0.088 &amp;amp;- X_4 / 0.066 \text{.}&#xD;&#xA;}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, with $\hat{\beta}_i$ representing the estimated coefficient of $X_i$ in this full regression, the pairs are defined by&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$X_{ij} = X_i / \hat{\beta}_i + X_j / \hat{\beta}_j\text{.}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is so systematic that it's straightforward to script.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &quot;identification regressions&quot; are the model&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Y \sim X_{12} + X_3 + X_4$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;along with the five additional permutations thereof, one for each pair.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You are looking for results where $X_{ij}$ becomes significant: &lt;em&gt;ignore the significance of the remaining&lt;/em&gt; $X_k$.  To see what's going on, I will list the results of all six identification regressions for the simulation.  As a shorthand, I list the variables followed by a vector of their &lt;em&gt;t-values&lt;/em&gt; only:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\eqalign{&#xD;&#xA;X_{12}, X_3, X_4:&amp;amp;\ (5.50, 1.24, -1.24) \cr&#xD;&#xA;X_{13}, X_2, X_4:&amp;amp;\ (1.36, 4.94, -1.13) \cr&#xD;&#xA;X_{14}, X_2, X_3:&amp;amp;\ (1.31, 5.16, 1.17) \cr&#xD;&#xA;X_{23}, X_1, X_4:&amp;amp;\ (1.64, 3.10, -1.09) \cr&#xD;&#xA;X_{24}, X_1, X_3:&amp;amp;\ (1.50, 4.15, 1.07) \cr&#xD;&#xA;X_{34}, X_1, X_2:&amp;amp;\ (5.56, 1.25, 1.25)&#xD;&#xA;}&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you can see from the first component of each vector (the t-value for the pair), precisely two disjoint pairs exhibit significant t-statistics: $X_{12}$, with $t = 5.50\ (p \lt 0.001)$, and $X_{34}$, with $t = 5.56\ (p \lt 0.001)$.  The model thus identified is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Y \sim X_{12} + X_{34}\text{.}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(In general, we would also include--provisionally--any remaining $X_i$ not participating in any of the pairs.  There aren't any in this case.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The regression results are&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\eqalign{&#xD;&#xA;\hat{\beta_{12}} &amp;amp;= 0.027\ (t = 5.54,\ p \lt 0.001) \cr&#xD;&#xA;\hat{\beta_{34}} &amp;amp;= 0.0055\ (t = 5.58,\ p \lt 0.001), \cr&#xD;&#xA;F(2, 57) &amp;amp;= 30.92\ (p \lt 0.0001).&#xD;&#xA;}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Translating back to the original $X_i$, the model is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\eqalign{&#xD;&#xA;Y &amp;amp;= 0.027(X_1 / 0.26 + X_2 / 0.13) + 0.0055(X_3 / 0.088 - X_4 / 0.066) \cr&#xD;&#xA;  &amp;amp;= 0.103 X_1 + 0.206 X_2 + 0.0629 X_3 - 0.0839 X_4 \cr&#xD;&#xA;  &amp;amp;= 0.103 (Z_1 + Z_2) + 0.021 (Z_3 - Z_4) \text{.}&#xD;&#xA;}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(The last line shows how this all relates to form of the original question.)  That's exactly the form used in the simulation: $Z_1$ and $Z_2$ enter with the same coefficient and $Z_3$ and $Z_4$ enter with opposite coefficients.  This method got the right answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to &lt;strong&gt;share a cool observation&lt;/strong&gt; in this regard.  First, here's the scatterplot matrix for the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/kR5Dd.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Notice how $X_{12}$ and $X_{34}$ look uncorrelated.  Furthermore, $Y$ is only weakly correlated with these variables.  Doesn't look like much of a relationship, does it?  Now consider an alternative set of pairs, $X_{13}$ and $X_{24}$.  The regression of $Y$ on these is still highly significant ($F(2, 57) = 16.61\ (p \lt 0.0001).$  Moreover, the coefficient of $X_{24}$ is significant ($t = 2.39,\ p = 0.020$) even though that of $X_{13}$ is not ($t = 0.24,\ p = 0.812$).  But look at the scatterplot matrix!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/11E4s.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Clearly $X_{13}$ and $X_{24}$ are strongly correlated.  But, &lt;em&gt;even though this is the wrong model,&lt;/em&gt; $Y$ is also visibly correlated with these two variables, much more so than in the preceding scatterplot matrix!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The lesson here&lt;/strong&gt; is that mere bivariate plots can be deceiving in a multiple regression setting: to analyze the relationship between any candidate independent variable (such as $X_{12}$) and the independent variable ($Y$), we must make sure to &quot;factor out&quot; all other independent variables.  (This is done by regressing $Y$ on all other independent variables and, separately, regressing $X_{12}$ on all the others.  Then one looks at a scatterplot of the &lt;em&gt;residuals&lt;/em&gt; of the first regression against the &lt;em&gt;residuals&lt;/em&gt; of the second regression.  It's a theorem that the slope in this &lt;em&gt;bivariate&lt;/em&gt; regression equals the coefficient of $X_{12}$ in the full &lt;em&gt;multivariate&lt;/em&gt; regression of $Y$ against all the variables.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This insight shows why we might want to systematically perform the &quot;identification regressions&quot; I have proposed, rather than using graphical methods or attempting to combine many of the pairs in one model.  Each identification regression assesses the strength of the contribution of a proposed linear combination of variables (a &quot;pair&quot;) &lt;em&gt;in the context of all the remaining independent variables.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that although correlated variables were involved, &lt;em&gt;correlation is not an essential feature of the problem or of the solution.&lt;/em&gt;  Even where you don't expect the original variables $X_i$ to be strongly correlated, you could expect a model to have (unknown) linear constraints among the variables.  &lt;strong&gt;That&lt;/strong&gt; is the important issue to cope with.  The presence of correlation only means that it can be problematic to identify such pairs solely by inspecting the original regression results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Following the procedure I have proposed does not guarantee you will find a unique solution.  It's conceivable, for instance, that you will find so many highly significant pairs that they are linearly dependent, forcing you to select among them by some other criterion. Nevertheless, the results you get ought to limit the sets of pairs you need to examine; they can be obtained with a straightforward procedure without intervention; and--if this simulation is any guide--they have a good chance of producing effective results.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-01-07T05:26:54.567" LastActivityDate="2011-01-07T05:26:54.567" />
  <row Id="6060" PostTypeId="2" ParentId="6047" CreationDate="2011-01-07T08:54:25.257" Score="8" ViewCount="" Body="&lt;p&gt;I think that you are looking for &lt;code&gt;Lift&lt;/code&gt; (or improvement). Lift is the ratio of the probability that A and B occur together to the multiple of the two individual probabilities for A and B. It is used to interpret the importance of a rule in &lt;a href=&quot;http://en.wikipedia.org/wiki/Association_rule_learning#Useful_Concepts&quot; rel=&quot;nofollow&quot;&gt;association rule mining&lt;/a&gt;. Lift is a way to measure how much better a model is over benchmark and it is defined as the confidence divided by the benchmark, where any value that is greater that one suggest that there is some usefulness to the rule. See &lt;a href=&quot;http://maya.cs.depaul.edu/~classes/ect584/WEKA/associate.html&quot; rel=&quot;nofollow&quot;&gt;this page&lt;/a&gt; also as another example.&lt;/p&gt;&#xA;" OwnerUserId="339" LastActivityDate="2011-01-07T08:54:25.257" CommentCount="2" />
  <row Id="6061" PostTypeId="1" CreationDate="2011-01-07T10:27:57.527" Score="5" ViewCount="60" Body="&lt;p&gt;I'm searching for some good reading material on &lt;a href=&quot;http://en.wikipedia.org/wiki/Multifractal&quot; rel=&quot;nofollow&quot;&gt;multifractal analysis&lt;/a&gt;. Preferably something accessible that doesn't put the stress too much on mathematical proofs but rather on applications. As long as it gives a good review of the status of the field, the interesting results and applications, I would be happy. Also, if anything related to multifractal analysis and statistics or time series comes up, I'll take it as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Books, papers, internetpages, videos, etc... accepted!&lt;/p&gt;&#xA;" OwnerUserId="2036" LastEditorUserId="2036" LastEditorDisplayName="" LastEditDate="2011-02-18T16:32:56.213" LastActivityDate="2011-02-18T16:32:56.213" Title="Need good material on multifractal analysis" Tags="&lt;books&gt;&lt;dataset&gt;&lt;fractals&gt;" CommentCount="2" />
  <row Id="6062" PostTypeId="2" ParentId="6044" CreationDate="2011-01-07T11:26:21.673" Score="1" ViewCount="" Body="&lt;p&gt;First, let's compare two lists of numbers &amp;mdash; are they from the same distribution ?&#xA;For example, how close are the lists of 20 numbers, &quot;|&quot; marks,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;||||||.||.||...||.....||.|................|.....................................&#xA;|||.|...|..|...|.......||...|...|...|.....|..|.................|.....|..........&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;? To see, visualize, how such lists differ&#xA;(whether real, simulated or theoretical),&lt;br&gt;&#xA;make a &lt;a href=&quot;http://en.wikipedia.org/wiki/QQ_plot&quot; rel=&quot;nofollow&quot;&gt;QQ plot&lt;/a&gt;:&#xA;sort X, sort Y, plot the pairs (X&lt;sub&gt;j&lt;/sub&gt;, Y&lt;sub&gt;j&lt;/sub&gt;),&#xA;see how close that curve is to the line X = Y.&#xA;Also, &lt;a href=&quot;http://stats.stackexchange.com/search?q=qq+plot&quot;&gt;search QQ plot&lt;/a&gt; here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A &lt;a href=&quot;http://en.wikipedia.org/wiki/K-S_test&quot; rel=&quot;nofollow&quot;&gt;K-S test&lt;/a&gt;&#xA;gives a number from 0, X and Y identical,&#xA;to 1, way off; you could flip this around to 100 % down to 0 %.&#xA;However a QQ plot shows more directly where X and Y differ.&lt;/p&gt;&#xA;" OwnerUserId="557" LastEditorUserId="557" LastEditorDisplayName="" LastEditDate="2011-01-07T13:45:52.677" LastActivityDate="2011-01-07T13:45:52.677" CommentCount="2" />
  <row Id="6063" PostTypeId="1" CreationDate="2011-01-07T11:46:10.653" Score="3" ViewCount="292" Body="&lt;p&gt;I have been running 3-level multilevel models with &lt;a href=&quot;http://www.ssicentral.com/hlm/&quot; rel=&quot;nofollow&quot;&gt;HLM&lt;/a&gt;, and my main interest is in some cross-level interaction effects that I am finding. My concern is that the effect sizes of these interactions appear to be small – I am wondering whether they are really meaningful. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am turning to you to ask whether anyone could advice me on how to evaluate the size of this kind of effect. Are there any benchmarks for interaction effects in regression analyses? Would these be appropriate for multilevel models, too? I believe that it is not uncommon that the explained variance does not increase much when one adds in interaction terms in multiple regression, and that this would be even more the case for multilevel models when the variance that one is trying to explain is the level 1 variance. Is that right? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance!&lt;/p&gt;&#xA;" OwnerUserId="2686" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-01-07T12:31:44.423" LastActivityDate="2011-01-10T00:19:12.147" Title="Evaluating effect sizes of interactions in multiple regression" Tags="&lt;regression&gt;&lt;effect-size&gt;&lt;multilevel-analysis&gt;" AnswerCount="2" />
  <row Id="6065" PostTypeId="2" ParentId="6022" CreationDate="2011-01-07T13:49:08.650" Score="4" ViewCount="" Body="&lt;p&gt;As @whuber pointed out, statistical methods do not exactly work here. You need to infer the distribution from other sources. When you know the distribution you have a non-linear equation solving exercise. Denote by $f$ the quantile function of your chosen probability distribution with parameter vector $\theta$. What you have is the following nonlinear system of equations:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align*}&#xA;q_{0.05}&amp;amp;=f(0.05,\theta) \\&#xA;q_{0.5}&amp;amp;=f(0.5,\theta) \\&#xA;q_{0.95}&amp;amp;=f(0.95,\theta)\\&#xA;\end{align*}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $q$ are your quantiles. You need to solve this system to find $\theta$. Now for practically for any 3-parameter distribution you will find values of parameters satisfying this equation. For 2-parameter and 1-parameter distributions this system is overdetermined, so there are no exact solutions. In this case you can search for a set of parameters which minimizes the discrepancy:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align*}&#xA;(q_{0.05}-f(0.05,\theta))^2+ (q_{0.5}-f(0.5,\theta))^2 + (q_{0.95}-f(0.95,\theta))^2 &#xA;\end{align*}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here I chose the quadratic function, but you can chose whatever you want. According to @whuber comments you can assign weights, so that more important quantiles can be fitted more accurately.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For four and more parameters the system is underdetermined, so infinite number of solutions exists. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is some sample R code illustrating this approach. For purposes of demonstration I generate the quantiles from Singh-Maddala distribution from &lt;a href=&quot;http://cran.r-project.org/web/packages/VGAM/index.html&quot; rel=&quot;nofollow&quot;&gt;VGAM&lt;/a&gt; package. This distribution has 3 parameters and is used in income distribution modelling.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; q &amp;lt;- qsinmad(c(0.05,0.5,0.95),2,1,4)&#xA; plot(x&amp;lt;-seq(0,2,by=0.01), dsinmad(x, 2, 1, 4),type=&quot;l&quot;)&#xA; points(p&amp;lt;-c(0.05, 0.5, 0.95), dsinmad(p, 2, 1, 4))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/XHQjV.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now form the function which evaluates the non-linear system of equations: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; fn &amp;lt;- function(x,q) q-qsinmad(c(0.05, 0.5, 0.95), x[1], x[2], x[3])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Check whether true values satisfy the equation:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; &amp;gt; fn(c(2,1,4),q)&#xA;   [1] 0 0 0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For solving the non-linear equation system I use function &lt;code&gt;nleqslv&lt;/code&gt; from package &lt;a href=&quot;http://cran.r-project.org/web/packages/nleqslv/index.html&quot; rel=&quot;nofollow&quot;&gt;nlqeslv&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; &amp;gt; sol &amp;lt;- nleqslv(c(2.4,1.5,4.3),fn,q=q)&#xA; &amp;gt; sol$x       &#xA;  [1] 2.000000 1.000000 4.000001&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As we see we get the exact solution. Now let us try to fit log-normal distribution to these quantiles. For this we will use the &lt;code&gt;optim&lt;/code&gt; function.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; &amp;gt; ofn &amp;lt;- function(x,q)sum(abs(q-qlnorm(c(0.05,0.5,0.95),x[1],x[2]))^2)&#xA; &amp;gt; osol &amp;lt;- optim(c(1,1),ofn)&#xA; &amp;gt; osol$par&#xA;   [1] -0.905049  0.586334&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now plot the result &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  plot(x,dlnorm(x,osol$par[1],osol$par[2]),type=&quot;l&quot;,col=2)&#xA;  lines(x,dsinmad(x,2,1,4))&#xA;  points(p,dsinmad(p,2,1,4))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/VnbG5.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From this we immediately see that the quadratic function is not so good. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-01-07T13:49:08.650" CommentCount="1" />
  <row Id="6066" PostTypeId="1" CreationDate="2011-01-07T14:49:08.103" Score="2" ViewCount="135" Body="&lt;p&gt;I have carried out a stepwise logistic regression in JMP. Then (using the proper button in the program window), I have choosen to build a nominal logistic regression model using (only) the variables indentified by the stepwise procedure.&#xA;Anyhow, comparing the summary tables of the stepwise regression and the nominal one, I have recognized that the regression coefficients are not the same, and also the p-values are not the same. There is even a variable which passes from a p-value of 0.02 to a p-value of 0.19 (much greater that 0.10, the threesold value I have choosen before stepwise procedure to retain variables in the model!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How is it possible?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I could use the values in the stepwise summary, but it does not contains any data allowing to build the confidence intervals. So, in suborder my question is: how can I calculate the confidence intervals using only the data reported in JMP stepwise regression summary?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: I have recognized just a minute ago that the differences refer to categorical variables which have yield more than one significative comparison.&#xA;For example, on stepwise regression details I read variable1 is included in the model three times (and passed three times to the nominal regression procedure): A-B versus C-D-E-F-G, C-D versus E-F-G, E-F versus G. Anyhow, such variable1 is reported only one time in regression summary, which cites only the first comparison (A-B versus C-D-E-F-G). It remains a mistery for me why.&lt;/p&gt;&#xA;" OwnerUserId="1219" LastEditorUserId="1219" LastEditorDisplayName="" LastEditDate="2011-01-07T15:33:35.757" LastActivityDate="2011-01-07T15:33:35.757" Title="Discrepancy between stepwise and nominal logistic regression results in JMP" Tags="&lt;logistic-regression&gt;&lt;stepwise-regression&gt;&lt;jmp&gt;" CommentCount="2" />
  <row Id="6067" PostTypeId="1" AcceptedAnswerId="6086" CreationDate="2011-01-07T16:48:03.487" Score="5" ViewCount="390" Body="&lt;p&gt;Okay, so I think I have a decent enough sample, taking into account the 20:1 rule of thumb: a fairly large sample (N=374) for a total of 7 candidate predictor variables. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My problem is the following: whatever set of predictor variables I use, the classifications never get better than a specificity of 100% and a sensitivity of 0%. However unsatisfactory, this could actually be the best possible result, given the set of candidate predictor variables (from which I can't deviate). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But, I couldn't help but think I could do better, so I noticed that the categories of the dependent variable were quite unevenly balanced, almost 4:1. Could a more balanced subsample improve classifications?&lt;/p&gt;&#xA;" OwnerUserId="2690" LastActivityDate="2011-01-08T16:01:51.453" Title="Does an unbalanced sample matter when doing logistic regression?" Tags="&lt;logistic-regression&gt;&lt;sample-size&gt;" AnswerCount="3" CommentCount="2" />
  <row Id="6069" PostTypeId="2" ParentId="6067" CreationDate="2011-01-07T17:26:12.403" Score="1" ViewCount="" Body="&lt;p&gt;Yes, it is possible that they could.&lt;/p&gt;&#xA;" OwnerUserId="601" LastActivityDate="2011-01-07T17:26:12.403" />
  <row Id="6070" PostTypeId="2" ParentId="6063" CreationDate="2011-01-07T17:27:20.800" Score="0" ViewCount="" Body="&lt;p&gt;Pursuant to my discussion on the conceptual overlap between effects size and likelihood ratios &lt;a href=&quot;http://stats.stackexchange.com/questions/4551/what-are-common-statistical-sins/6037#6037&quot;&gt;here&lt;/a&gt;, I wonder if the likelihood ratio for each effect against its respective null might serve as a useful metric to achieve the aims sought by those who conventionally employ effect size measures.&lt;/p&gt;&#xA;" OwnerUserId="364" LastActivityDate="2011-01-07T17:27:20.800" />
  <row Id="6071" PostTypeId="1" CreationDate="2011-01-07T18:02:14.250" Score="4" ViewCount="202" Body="&lt;p&gt;I'm not sure if this is precisely a measurement error model or not.  I'm working on performing meta analysis, and the model I'm starting with is fairly basic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{aligned}&#xA;X_i &amp;amp;= \mu_i + e_i \\&#xA;Y_i &amp;amp;= \beta \mu_i + g_i + \delta_i&#xA;\end{aligned}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The random components are $e_i$, $g_i$, and $\delta_i$,and the variance is known for $e_i$ and $\delta_i$.  This falls under a measurement error model with measurement in both the predictor and response.  How would I fit this model in R?&lt;/p&gt;&#xA;" OwnerUserId="1364" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-06-02T22:04:29.907" LastActivityDate="2011-06-02T22:04:29.907" Title="Classical measurement error model in R " Tags="&lt;r&gt;&lt;mixed-model&gt;&lt;meta-analysis&gt;&lt;random-effects-model&gt;" AnswerCount="1" CommentCount="6" />
  <row Id="6072" PostTypeId="2" ParentId="6067" CreationDate="2011-01-07T18:29:10.353" Score="5" ViewCount="" Body="&lt;p&gt;The problem is not that the classes are imbalance per se, it is that there may not be sufficient patterns belonging to the minority class to adequately represent its distribution.  This means that the problem can arise for any classifier (even if you have a synthetic problem and you know you have the true model), not just logistic regression.  The good thing is that as more data become available, the &quot;class imbalance&quot; problem usually goes away.  Having said which 4:1 is not that imbalanced.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you use a balanced dataset, the important thing is to remember that the output of the model is now an estimate of the a-posteriori probability assuming the classes are equally common, and so you may end up biasing the model too far.  I would weight the patterns belonging to each class differently and choose the weights by minimising the cross-entropy on a test set with the correct operational class frequencies.&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2011-01-07T18:29:10.353" />
  <row Id="6073" PostTypeId="2" ParentId="5873" CreationDate="2011-01-07T20:05:42.093" Score="1" ViewCount="" Body="&lt;p&gt;Thank you, whuber, for making me aware of Wald's Sequential Probability Ratio Test (SPRT). At your recommendation, I will relist this &lt;a href=&quot;http://quantitativeskills.com/sisa/statistics/sprt.htm&quot; rel=&quot;nofollow&quot;&gt;Quantitative Skills site&lt;/a&gt;. They will give you an out-of-the-box table to determine whether to continue or stop testing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also took the time to research that site's references, and was directed toward a comprehensive article that is intended for medical testing, but is easily transferable to other domains. It is &lt;em&gt;Increasing Efficiency in Evaluation Research: The Use of Sequential Analysis&lt;/em&gt; (Howe, Holly L., American Journal of Public Health July 1982, Vol. 72, No. 7, pp. 690-697.) This article may be downloaded in its entirety.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since I have not seen SPRT in my stats courses, I will provide a cookbook that I hope will he helpful for the stackexchange community.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For my null hypothesis, I tested for a level of 95% correct. If, however, the level was below 80%, it would be a cause for concern. So I have&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$p_1 = .95$ (null hypothesis), and $p_2 = .80$ (alternative hypothesis)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I will use $\alpha = 0.05$ and $\beta = 0.10$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Howe shows a graph with two parallel lines, with plots of the cumulative errors. Testing continues while the cumulative errors (and in my case, cumulative count of correct data points) lie between the two lines.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the cumulative errors exceed either line, then either:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;accept the null hypothesis (if cumulative error count falls below the bottom line,  $d_1$), or&lt;/li&gt;&#xA;&lt;li&gt;reject the null hypothesis (if cumulative error count exceeds the top line, $d_2$).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Here are the equations. I am adding a denominator because it is used several times.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$denom = log\left [ \left (  \frac{p_2}{p_1}\right )(\frac{1 - p_1}{1 - p_2}) \right ]$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The slopes of the lines are the same, and represented by s.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$s = \frac{log\left ( \frac{1 - p_1}{1 - p_2} \right )}{denom}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The intercepts, $h_1$ and $h_2$, are computed as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$h_1 = \frac{log\left ( \frac{1 - \alpha }{\beta }\right )}{denom}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$h_2 = \frac{log\left ( \frac{1 - \beta }{\alpha  }\right )}{denom}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I set up a spreadsheet with data point N going from 1 to 50. Then I added two columns for acceptance threshold ($d_1$) and rejection threshold ($d_2$).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$d_1 = -h_1 + sN$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$d_2 = h_2 + sN$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my experiment,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$denom = -0.67669$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$h_1 = -1.44485$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$h_2 = -1.85501$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The values of $d_1$ at N=2, N=5, N=10 are 3.224, 5.893, 10.342.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I then added columns for success and cumSuccess. I picked data points until the cumulative number exceeded the acceptance threshold, and I accepted the null hypothesis.&lt;/p&gt;&#xA;" OwnerUserId="2591" LastActivityDate="2011-01-07T20:05:42.093" />
  <row Id="6074" PostTypeId="1" CreationDate="2011-01-07T22:08:40.717" Score="5" ViewCount="1288" Body="&lt;p&gt;Context: I am a programmer with some (half-forgotten) experience in statistics from uni courses. Recently I stumbled upon &lt;a href=&quot;http://akinator.com&quot; rel=&quot;nofollow&quot;&gt;http://akinator.com&lt;/a&gt; and spent some time trying to make it fail. And who wasn't? :)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've decided to find out how it could work. After googling and reading related blog posts and adding some of my (limited) knowledge into resulting mix I come up with the following model (I'm sure that I'll use the wrong notation, please don't kill me for that):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are Subjects (S) and Questions (Q). Goal of the predictor is to select the subject S which has the greatest aposterior probability of being the subject that user is thinking about, given questions and answers collected so far.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let game G be a set of questions asked and answers given: $\{q_1, a_1\}, \{q_2, a_2\} ... \{q_n, a_n\}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then predictor is looking for $P(S|G) = \frac{P(G|S) * P(S)}{P(G)}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Prior for subjects ($P(S)$) could be just the number of times subject has been guessed divided by total number of games.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Making an assumption that all answers are independent, we could compute the likelihood of subject S given the game G like so: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(G|S) = \prod_{i=1..n} P(\{q_i, a_i\} | S)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We could calculate the $P(\{q_i, a_i\} | S)$ if we keep track of which questions and answers were given when the used have though of given subject:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P({q, a} | S) = \frac{answer\ a\ was\ given\ to\ question\ q\ in\ the\ game\ when\ S\ was\ the\ subject}{number\ of\ times\ q\ was\ asked\ in\ the\ games\ involving\ S}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, $P(S|G)$ defines a probability distribution over subjects and when we need to select the next question we have to select the one for which the expected change in the entropy of this distribution is maximal:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$argmax_j (H[P(S|G)] - \sum_{a=yes,no,maybe...} H[P(S|G \vee \{q_j, a\})]$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've tried to implement this and it works. But, obviously, as the number of subjects goes up, performance degrades due to the need to recalculate the $P(S|G)$ after each move and calculate updated distribution $P(S|G \vee \{q_j, a\})$ for question selection.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suspect that I simply have chosen the wrong model, being constrained by the limits of my knowledge. Or, maybe, there is an error in the math. Please enlighten me: what should I make myself familiar with, or how to change the predictor so that it could cope with millions of subjects and thousands of questions?&lt;/p&gt;&#xA;" OwnerUserId="2696" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-01-07T23:40:20.670" LastActivityDate="2011-01-15T19:41:21.880" Title="Akinator.com and Naive Bayes classifier" Tags="&lt;machine-learning&gt;&lt;naive-bayes&gt;" AnswerCount="2" CommentCount="4" FavoriteCount="4" />
  <row Id="6075" PostTypeId="2" ParentId="3779" CreationDate="2011-01-07T23:19:55.983" Score="9" ViewCount="" Body="&lt;p&gt;It is very hard to draw a rack that does not contain any valid word in&#xA;Scrabble and its variants. Below is an R program I wrote to estimate the&#xA;probability that the initial 7-tile rack does not contain a valid word. It&#xA;uses a monte carlo approach and the &lt;a href=&quot;http://newtoyinc.com/&quot; rel=&quot;nofollow&quot;&gt;Words With Friends&lt;/a&gt; lexicon (I&#xA;couldn’t find the official Scrabble lexicon in an easy format). Each trial&#xA;consists of drawing a 7-tile rack, and then checking if the rack contains a&#xA;valid word.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Minimal words&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You don’t have to scan the entire lexicon to check if the rack contains a&#xA;valid word. You just need to scan a &lt;strong&gt;minimal lexicon&lt;/strong&gt; consisting of&#xA;&lt;em&gt;minimal&lt;/em&gt; words. A word is minimal if it contains no other word as a subset.&#xA;For example 'em’ is a minimal word; 'empty’ is not. The point of this is that&#xA;if a rack contains word &lt;em&gt;x&lt;/em&gt; then it must also contain any subset of &lt;em&gt;x&lt;/em&gt;. In&#xA;other words: a rack contains no words iff it contains no minimal words.&#xA;Luckily, most words in the lexicon are not minimal, so they can be eliminated.&#xA;You can also merge permutation equivalent words. I was able to reduce the&#xA;Words With Friends lexicon from 172,820 to 201 minimal words.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Wildcards can be easily handled by treating racks and words as distributions&#xA;over the letters. We check if a rack contains a word by subtracting one&#xA;distribution from the other. This gives us the number of each letter missing&#xA;from the rack. If the sum of those number is $\leq$ the number of wildcards,&#xA;then the word is in the rack.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The only problem with the monte carlo approach is that the event that we are&#xA;interested in is very rare. So it should take many, many trials to get an&#xA;estimate with a small enough standard error. I ran my program (pasted at the bottom) with&#xA;$N=100,000$ trials and got an &lt;strong&gt;estimated probability of 0.004 that the&#xA;initial rack does not contain a valid word&lt;/strong&gt;. The estimated standard error of&#xA;that estimate is 0.0002. It took just a couple minutes to run on my Mac Pro,&#xA;including downloading the lexicon.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I’d be interested in seeing if someone can come up with an efficient exact&#xA;algorithm. A naive approach based on inclusion-exclusion seems like it could&#xA;involve a combinatorial explosion.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Inclusion-exclusion&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think this is a bad solution, but here is an incomplete sketch anyway.&#xA;In principle you can write a program to do the calculation, but the &#xA;specification would be tortuous.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The probability we wish to calculate is&#xA;$$&#xD;&#xA;P(k\text{-tile rack does not contain a word})&#xD;&#xA;=&#xD;&#xA;1 - P(k\text{-tile rack contains a word})&#xD;&#xA;.&#xD;&#xA;$$&#xA;The event inside the probability on the right side is a union of events:&#xA;$$&#xD;&#xA;P(k\text{-tile rack contains a word})&#xD;&#xA;=&#xD;&#xA;P\left(\cup_{x \in M} \{ k\text{-tile rack contains }x \} \right),&#xD;&#xA;$$&#xA;where $M$ is a minimal lexicon.&#xA;We can expand it using the inclusion-exclusion formula.&#xA;It involves considering all possible intersections of the events above.&#xA;Let $\mathcal{P}(M)$ denote the power set of $M$, i.e. the set of all&#xA;possible subsets of $M$.  Then &#xA;$$&#xD;&#xA;\begin{align}&#xD;&#xA;  &amp;amp;P(k\text{-tile rack contains a word}) \\&#xD;&#xA;  &amp;amp;= &#xD;&#xA;  P\left(\cup_{x \in M} \{ k\text{-tile rack contains }x \} \right) \\&#xD;&#xA;  &amp;amp;=&#xD;&#xA;  \sum_{j=1}^{|M|}&#xD;&#xA;  (-1)^{j-1}&#xD;&#xA;  \sum_{S \in \mathcal{P}(M) : |S| = j}&#xD;&#xA;  P\left( \cap_{x \in S} \{ k\text{-tile rack contains }x \} \right)&#xD;&#xA;\end{align}&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The last thing to specify is how to calculate the probability on the last line above.&#xA;It involves a multivariate hypergeometric.&lt;br&gt;&#xA;$$\cap_{x \in S} \{ k\text{-tile rack contains }x \}$$&#xA;is the event that the rack contains every word in $S$.  This is a pain to deal with because of wildcards.  We'll have to &#xA;consider, by conditioning, each of the following cases: the rack contains no&#xA;wildcards, the rack contains 1 wildcard, the rack contains 2 wildcards, ...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then &#xA;$$&#xD;&#xA;\begin{align}&#xD;&#xA;&amp;amp;P\left( \cap_{x \in S} \{ k\text{-tile rack contains }x \} \right)&#xD;&#xA;\\&#xD;&#xA;&amp;amp;= \sum_{w=0}^{n_{*}} &#xD;&#xA;P\left( \cap_{x \in S} \{ k\text{-tile rack contains }x \} | &#xD;&#xA;k\text{-tile rack contains } w \text{ wildcards}&#xD;&#xA;\right) \\&#xD;&#xA;&amp;amp;\quad \times&#xD;&#xA;P(k\text{-tile rack contains } w \text{ wildcards})&#xD;&#xA;.&#xD;&#xA;\end{align}&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm going to stop here, because the expansions are tortuous to write out and&#xA;not at all enlightening. It's easier to write a computer program to do it. But&#xA;by now you should see that the inclusion-exclusion approach is intractable. It&#xA;involves $2^{|M|}$ terms, each of which is also very complicated. For the&#xA;lexicon I considered above $2^{|M|} \approx 3.2 \times 10^{60}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Scanning all possible racks&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think this is computationally easier, because there are fewer possible racks than possible subsets of minimal words.  We successively reduce the&#xA;set of possible $k$-tile racks until we get the set of racks which contain no&#xA;words. For Scrabble (or Words With Friends) the number of possible 7-tile&#xA;racks is in the tens of billions. Counting the number of those that do not&#xA;contain a possible word should be doable with a few dozen lines of R code. But I&#xA;think you should be able to do better than just enumerating all possible&#xA;racks. For instance, 'aa' is a minimal word. That immediately eliminates all&#xA;racks containing more than one 'a’.  You can repeat with other words.  Memory shouldn’t be an issue for modern computers.  A 7-tile Scrabble rack requires fewer than 7 bytes of storage.  At worst we would use a few gigabytes to store all possible racks, but I don’t think that’s a good idea either.  Someone may want to think more about this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Monte Carlo R program&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# &#xA;#  scrabble.R&#xA;#  &#xA;#  Created by Vincent Vu on 2011-01-07.&#xA;#  Copyright 2011 Vincent Vu. All rights reserved.&#xA;# &#xA;&#xA;# The Words With Friends lexicon&#xA;# http://code.google.com/p/dotnetperls-controls/downloads/detail?name=enable1.txt&amp;amp;can=2&amp;amp;q=&#xA;url &amp;lt;- 'http://dotnetperls-controls.googlecode.com/files/enable1.txt'&#xA;lexicon &amp;lt;- scan(url, what=character())&#xA;&#xA;# Words With Friends&#xA;letters &amp;lt;- c(unlist(strsplit('abcdefghijklmnopqrstuvwxyz', NULL)), '?')&#xA;tiles &amp;lt;- c(9, 2, 2, 5, 13, 2, 3, 4, 8, 1, 1, 4, 2, 5, 8, 2, 1, 6, 5, 7, 4, &#xA;           2, 2, 1, 2, 1, 2)&#xA;names(tiles) &amp;lt;- letters&#xA;&#xA;# Scrabble&#xA;# tiles &amp;lt;- c(9, 2, 2, 4, 12, 2, 3, 2, 9, 1, 1, 4, 2, 6, 8, 2, 1, 6, 4, 6, 4, &#xA;#            2, 2, 1, 2, 1, 2)&#xA;&#xA;&#xA;# Reduce to permutation equivalent words&#xA;sort.letters.in.words &amp;lt;- function(x) {&#xA;  sapply(lapply(strsplit(x, NULL), sort), paste, collapse='')&#xA;}&#xA;&#xA;min.dict &amp;lt;- unique(sort.letters.in.words(lexicon))&#xA;min.dict.length &amp;lt;- nchar(min.dict)&#xA;&#xA;# Find all minimal words of length k by elimination&#xA;# This is held constant across iterations:&#xA;#   All words in min.dict contain no other words of length k or smaller&#xA;k &amp;lt;- 1&#xA;while(k &amp;lt; max(min.dict.length))&#xA;{&#xA;  # List all k-letter words in min.dict&#xA;  k.letter.words &amp;lt;- min.dict[min.dict.length == k]&#xA;&#xA;  # Find words in min.dict of length &amp;gt; k that contain a k-letter word&#xA;  for(w in k.letter.words)&#xA;  {&#xA;    # Create a regexp pattern&#xA;    makepattern &amp;lt;- function(x) {&#xA;      paste('.*', paste(unlist(strsplit(x, NULL)), '.*', sep='', collapse=''), &#xA;            sep='')&#xA;    }&#xA;    p &amp;lt;- paste('.*', &#xA;               paste(unlist(strsplit(w, NULL)), &#xA;                     '.*', sep='', collapse=''), &#xA;               sep='')&#xA;&#xA;    # Eliminate words of length &amp;gt; k that are not minimal&#xA;    eliminate &amp;lt;- grepl(p, min.dict) &amp;amp; min.dict.length &amp;gt; k&#xA;    min.dict &amp;lt;- min.dict[!eliminate]&#xA;    min.dict.length &amp;lt;- min.dict.length[!eliminate]&#xA;  }&#xA;  k &amp;lt;- k + 1&#xA;}&#xA;&#xA;# Converts a word into a letter distribution&#xA;letter.dist &amp;lt;- function(w, l=letters) {&#xA;  d &amp;lt;- lapply(strsplit(w, NULL), factor, levels=l)&#xA;  names(d) &amp;lt;- w&#xA;  d &amp;lt;- lapply(d, table)&#xA;  return(d)&#xA;}&#xA;&#xA;# Sample N racks of k tiles&#xA;N &amp;lt;- 1e5&#xA;k &amp;lt;- 7&#xA;rack &amp;lt;- replicate(N,&#xA;                  paste(sample(names(tiles), size=k, prob=tiles), &#xA;                        collapse=''))&#xA;&#xA;contains.word &amp;lt;- function(rack.dist, lex.dist)&#xA;{&#xA;  # For each word in the lexicon, subtract the rack distribution from the &#xA;  # letter distribution of the word.  Positive results correspond to the &#xA;  # number of each letter that the rack is missing.&#xA;  y &amp;lt;- sweep(lex.dist, 1, rack.dist)&#xA;&#xA;  # If the total number of missing letters is smaller than the number of &#xA;  # wildcards in the rack, then the rack contains that word&#xA;  any(colSums(pmax(y,0)) &amp;lt;= rack.dist[names(rack.dist) == '?'])&#xA;}&#xA;&#xA;# Convert rack and min.dict into letter distributions&#xA;min.dict.dist &amp;lt;- letter.dist(min.dict)&#xA;min.dict.dist &amp;lt;- do.call(cbind, min.dict.dist)&#xA;rack.dist &amp;lt;- letter.dist(rack, l=letters)&#xA;&#xA;# Determine if each rack contains a valid word&#xA;x &amp;lt;- sapply(rack.dist, contains.word, lex.dist=min.dict.dist)&#xA;&#xA;message(&quot;Estimate (and SE) of probability of no words based on &quot;, &#xA;        N, &quot; trials:&quot;)&#xA;message(signif(1-mean(x)), &quot; (&quot;, signif(sd(x) / sqrt(N)), &quot;)&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1670" LastEditorUserId="1670" LastEditorDisplayName="" LastEditDate="2011-01-08T20:45:41.183" LastActivityDate="2011-01-08T20:45:41.183" CommentCount="5" />
  <row Id="6076" PostTypeId="1" AcceptedAnswerId="6085" CreationDate="2011-01-07T23:22:09.210" Score="0" ViewCount="227" Body="&lt;p&gt;I'm a statistics newbie (medical student) trying my luck with a Cox regression for a survival analysis on the outcome of a specific type of operation. And I'm trying to determine which variables to control for. And how to handle the age of the patient at operation-time (measure-start)...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently I've set the following as covariates in my analysis (PASW/SPSS):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Risk (Low/High)&#xA;Gender (Female/Male)&#xA;Shunt used in operation (False/True)&#xA;Artery stenosis (False/True)&#xA;Age at operation (Integer value)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I've set the 4 first variables as categories (dummy coded as 0 and 1), and specified the first value (0) as the reference value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But what about the patients age at the operation? I assume this must be controlled for? But the age at measure-start (operation) isn't time-dependent (is it?), but are always the same value, right? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does this mean I have to control for the patients real age aswell? And does this mean I have to use the &lt;code&gt;COX with Time-dependent Covariables&lt;/code&gt; function instead of the regular &lt;code&gt;COX Regression&lt;/code&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for any help!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regards&#xA;Alex&lt;/p&gt;&#xA;" OwnerUserId="2652" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-01-07T23:43:23.037" LastActivityDate="2011-01-08T15:28:00.433" Title="How to handle age at measure-start in Cox regression?" Tags="&lt;regression&gt;&lt;spss&gt;" AnswerCount="1" />
  <row Id="6077" PostTypeId="1" AcceptedAnswerId="6084" CreationDate="2011-01-07T23:48:05.703" Score="1" ViewCount="257" Body="&lt;p&gt;Another newbie question here (probably piece of cake for you guys).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I run a Cox Regression and one of my covariates come out as significant:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;                                                    95,0% CI for Exp(B)&#xA;        B       SE      Wald    df  Sig.    Exp(B)  Lower   Upper&#xA;risk    2,224   1,107   4,036   1   ,045    9,244   1,056   80,950&#xA;&amp;lt;5 covariates removed for clarity&amp;gt;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But my Omnibus Tests of Model Coefficients(a) isn't significant. Change from previous step: Chi-square=10,290. df=6. Sig.=0,113.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does this mean my model is simply too bad to use for anything? Can I say anything credible about my data after running this analysis?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for any clues that'll take me closer to the beauty of applied statistics... ;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regards&#xA;Alex&lt;/p&gt;&#xA;" OwnerUserId="2652" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-01-07T23:57:54.437" LastActivityDate="2011-01-08T15:20:27.973" Title="Interpretation of log likelihood and covariate significance in Cox regression" Tags="&lt;regression&gt;" AnswerCount="1" />
  <row Id="6078" PostTypeId="1" AcceptedAnswerId="6088" CreationDate="2011-01-08T02:52:10.047" Score="2" ViewCount="115" Body="&lt;p&gt;I want to do a chi-squared test on data that looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;A B&#xA;0 0 &#xA;1 0&#xA;0 1&#xA;1 1&#xA;8 0&#xA;3 4&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can think of each pair as one trial with two participants. In each trial, there are a different number of observations for each participant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have binned each data pair like so: I counted how many pairs have 0 for both pairs (e.g. 0-0), how many have exactly one 0 (e.g., 0-1, 1-0, 8-0, etc.), and how many have greater than 0 for both (e.g., 1-1, 3-4, etc.). This gives me the following counts: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Two zeroes: 227&lt;br&gt;&#xA;One zero: 277&lt;br&gt;&#xA;No zeroes: 146  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is that I am not sure how to calculate expected values here. Each pair represents the number of times something appears within an arbitrary number of observations; each pair represents a different number of observations, and each element of the pair does too. So, for instance, for a given 0-1 pair the first participant might have given 200 observations (with no hits), and the second might have yielded 150 observations (with 1 hit). Another 0-1 pair might have yielded 100 observations and 50 observations, respectively.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, in this case, just totaling the overall number of hits and dividing that by the total number of observations won't get me the right expected values... &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Am I missing something obvious here? For instance if we know that the relative frequency of hits per observation is 0.01217, and if there are a different number of observations for each trial, is there a simple way to get expected values in a problem like this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is what I want to test: whether the hits between participants in each trial are independent. I expect that as the hits of one participant in one pair increases, so will the other. I realize that there might be better ways to test this, but my committee have asked me to use a chi-sq test, if possible. So what I expect to find is that the number of trials in the 0-0 and in the no-zero bins will be higher than would be expected if hits are distributed evenly.&lt;/p&gt;&#xA;" OwnerUserId="52" LastEditorUserId="52" LastEditorDisplayName="" LastEditDate="2011-01-08T05:20:53.410" LastActivityDate="2011-01-08T19:36:42.637" Title="Expected values for chi-squared test on binned paired counts" Tags="&lt;chi-squared&gt;&lt;expected-value&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="6079" PostTypeId="1" AcceptedAnswerId="6090" CreationDate="2011-01-08T03:03:48.000" Score="3" ViewCount="358" Body="&lt;p&gt;I did a linear regression in R and got the following result:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;                        Estimate Std. Error t value Pr(&amp;gt;|t|)    &#xA;(Intercept)        192116.40    6437.27  29.844  &amp;lt; 2e-16 ***&#xA;cdd                   272.74      26.94  10.123 1.56e-09 ***&#xA;pmax(hdd - 450, 0)     61.73      22.54   2.738   0.0123 *  &#xA;---&#xA;Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 &#xA;&#xA;Residual standard error: 16500 on 21 degrees of freedom&#xA;Multiple R-squared: 0.8454, Adjusted R-squared: 0.8307 &#xA;F-statistic: 57.41 on 2 and 21 DF,  p-value: 3.072e-09 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My question regards the R-squared value, 0.83 and what it means if I want to specify approximate percentage contributions of each (monthly) variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT:&#xA;See data, below. Say I take the first 12 hdd and cdd data points, and calculate the sum of the 12 predictions (i.e. the first year's total prediction), using the coefficients, above. The baseline (intercept) contribution to the year would be approximately 12 * 192116.40 = 2305397, right? Similarly, the cdd contribution to the year would be approximately 1608 * 272.74 = 438565.9, and hdd would be (after my hand-made hinge function) approximately 1329 * 61.73 = 82039.17. Summing the three values yields 2826002, which is within 1.3% of actual total usage (2862840, the sum of the first 12 elec's).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can I then say that cdd contributes 438565.9/2826002= 0.1551895, or approximately 16% of the yearly total? Or do I need to take that and compensate for the adjusted R-squared: 0.1551895*0.8307= 0.1289159 (i.e. multiply by the adjusted R-squares), for approximately 13% of the total? Or is none of this correct reasoning?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My data is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;     elec  hdd cdd&#xA;1  235940  880   3&#xA;2  205380  772   4&#xA;3  211780  551   9&#xA;4  192220  281  68&#xA;5  221440  165 119&#xA;6  304840   15 364&#xA;7  283160    4 434&#xA;8  300440   11 339&#xA;9  272900   42 214&#xA;10 204220  322  44&#xA;11 201060  592   8&#xA;12 229460  784   2&#xA;13 214520 1064   0&#xA;14 197900  719   2&#xA;15 186660  618  15&#xA;16 195340  332  88&#xA;17 241200  109 159&#xA;18 260700   18 282&#xA;19 299940   29 367&#xA;20 293240    2 426&#xA;21 268740   51 159&#xA;22 208380  319  36&#xA;23 183820  452   7&#xA;24 231360  903   0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(The monthly billing cycle for elec can be anywhere from 29 to 32 days, so that injects a lot of variance right there. I do not yet have all of the billing cycle lengths to to a trading day kind of adjustment.)&lt;/p&gt;&#xA;" OwnerUserId="1764" LastEditorUserId="1764" LastEditorDisplayName="" LastEditDate="2011-01-10T18:19:19.323" LastActivityDate="2011-01-10T18:19:19.323" Title="R-squared result in linear regression and &quot;unexplained variance&quot;" Tags="&lt;regression&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="2" />
  <row Id="6080" PostTypeId="1" CreationDate="2011-01-08T06:24:18.810" Score="3" ViewCount="140" Body="&lt;p&gt;Friends,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm trying to fit the GAMLSS library's Sichel distribution to some zero-truncated data, but the only way to get the function to work is to include the zero-class anyway but give it a frequency of 0, which doesn't take into account the zero-truncated nature of my data. Can anyone suggest a way to properly &quot;redistribute&quot; the zero-class's probability to the remaining probabilities (or some other, better, course of action using Sichel)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you run the following example, you'll see that &lt;code&gt;sum(pdf2)&lt;/code&gt; equals 1, but that the zero class that has a probability in my case of 0 is still allocated around 27% of the cum probability: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Counts = data.frame(n = c(0,1,2,3,4,5,6,7,8,9,10),&#xA;                    freq = c(0,182479,76986,44859,24315,49,100,490,106,0,2))&#xA;&#xA;gamlss(n~1,family=SICHEL, control=gamlss.control(n.cyc=50),data=Counts )&#xA;&#xA;pdf2 = dSICHEL(x=with(Counts, n), mu = 1.610, sigma = 98.43, nu = 3.315)&#xA;&#xA;print( with(Counts, cbind(n, freq, fitted=pdf2*sum(freq))), dig=9)&#xA;&#xA;sum(pdf2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Thanks,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;--Jeff&lt;/p&gt;&#xA;" OwnerUserId="2702" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-01-09T05:37:51.467" LastActivityDate="2011-05-27T06:50:41.300" Title="Zero-truncated Sichel distribution in R" Tags="&lt;r&gt;&lt;distributions&gt;&lt;count-data&gt;" AnswerCount="3" FavoriteCount="1" />
  <row Id="6081" PostTypeId="1" AcceptedAnswerId="6083" CreationDate="2011-01-08T11:26:07.737" Score="3" ViewCount="536" Body="&lt;p&gt;So assuming that there is a point in testing the normality assumption for anova (see &lt;a href=&quot;http://stats.stackexchange.com/questions/2492/normality-testing-essentially-useless&quot;&gt;1&lt;/a&gt; and &lt;a href=&quot;http://stats.stackexchange.com/questions/2824/checking-anova-assumptions&quot;&gt;2&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can it be tested in R?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would expect to do something like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;## From Venables and Ripley (2002) p.165.&#xA;utils::data(npk, package=&quot;MASS&quot;)&#xA;npk.aovE &amp;lt;- aov(yield ~  N*P*K + Error(block), npk)&#xA;residuals(npk.aovE)&#xA;qqnorm(residuals(npk.aov))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Which doesn't work, since &quot;residuals&quot; don't have a method (nor predict, for that matter) for &#xA;the case of repeated measures anova.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So what should be done in this case?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can the residuals simply be extracted from the same fit model without the Error term?  I am not familiar enough with the literature to know if this is valid or not, thanks in advance for any suggestion.&lt;/p&gt;&#xA;" OwnerUserId="253" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-01-08T11:41:28.580" LastActivityDate="2011-01-08T19:46:46.810" Title="Testing the normality assumption for repeated measures anova? (in R)" Tags="&lt;r&gt;&lt;anova&gt;&lt;normality&gt;&lt;repeated-measures&gt;" AnswerCount="3" FavoriteCount="1" />
  <row Id="6082" PostTypeId="2" ParentId="6079" CreationDate="2011-01-08T11:28:51.557" Score="3" ViewCount="" Body="&lt;p&gt;R^2 is the percent of variance in the DV accounted for by the whole model.  That is, your intercept and your IVS combined account for that much of the variance, using the linear regression model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case, you got an R^2 of 0.85, indicating that intercept, plus cdd plus pmax combined account for 85% of the variance in the DV.  The other 15% is error.  That is, variance that is not accounted for by the model&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You cannot tell, from the information given, how much of this 85% is contributed by each.  In order to do this, you would have to run more models:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;DV ~ .   (intercept alone)&lt;br&gt;&#xA;DV ~ cdd&lt;br&gt;&#xA;DV ~ pmax&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each of these would have an R^2, and you could then tell how much each adds.&lt;/p&gt;&#xA;" OwnerUserId="686" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-01-08T13:07:34.513" LastActivityDate="2011-01-08T13:07:34.513" CommentCount="3" />
  <row Id="6083" PostTypeId="2" ParentId="6081" CreationDate="2011-01-08T14:26:28.970" Score="3" ViewCount="" Body="&lt;p&gt;You may not get a simple response to &lt;code&gt;residuals(npk.aovE)&lt;/code&gt; but that does not mean there are no residuals in that object. Do &lt;code&gt;str&lt;/code&gt; and see that within the levels there are still residuals. I would imagine you were most interested in the &quot;Within&quot; level&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; residuals(npk.aovE$Within)&#xA;          7           8           9          10          11          12 &#xA; 4.68058815  2.84725482  1.56432584 -5.46900749 -1.16900749 -3.90234083 &#xA;         13          14          15          16          17          18 &#xA; 5.08903669  1.28903669  0.35570336 -3.27762998 -4.19422371  1.80577629 &#xA;         19          20          21          22          23          24 &#xA;-3.12755705  0.03910962  2.60396981  1.13730314  2.77063648  4.63730314 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My own training and practice has not been to use normality testing, instead to use QQ plots and parallel testing with robust methods.&lt;/p&gt;&#xA;" OwnerUserId="2129" LastActivityDate="2011-01-08T14:26:28.970" CommentCount="2" />
  <row Id="6084" PostTypeId="2" ParentId="6077" CreationDate="2011-01-08T15:20:27.973" Score="2" ViewCount="" Body="&lt;p&gt;Credibility depends on context and you've not offered any. If the other 5 degrees of freedom are in the model for good reasons but you aren't particularly interested in estimating the effects, only interested in the impact of &quot;risk&quot;, then you may have something interesting but the really wide confidence interval prevents me from applying the label &quot;credible&quot;. However, the likelihood ratio test for the addition of &quot;risk&quot; to the model would be a more solid estimate of its effect independent of the other covariates. Your Wald test is fairly borderline.&lt;/p&gt;&#xA;" OwnerUserId="2129" LastActivityDate="2011-01-08T15:20:27.973" CommentCount="4" />
  <row Id="6085" PostTypeId="2" ParentId="6076" CreationDate="2011-01-08T15:28:00.433" Score="2" ViewCount="" Body="&lt;p&gt;You are right that you don't need to use a time-dependent model for this situation. If, however, age would be expected to have a dominant effect (as might be expected),  you may be safer using an interval-censored model. With R, that is setup with three arguments to the Surv object, but I don't have any experience with SPSS's facilities for survival analysis.&lt;/p&gt;&#xA;" OwnerUserId="2129" LastActivityDate="2011-01-08T15:28:00.433" CommentCount="2" />
  <row Id="6086" PostTypeId="2" ParentId="6067" CreationDate="2011-01-08T16:01:51.453" Score="5" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Balance in the Training Set&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For logistic regression models unbalanced training data affects only the estimate of the model intercept (although this of course skews all the predicted probabilities, which in turn compromises your predictions). Fortunately the intercept correction is straightforward:   Provided you know, or can guess, the true proportion of 0s and 1s and know the proportions in the training set you can apply a rare events correction to the intercept.  Details are in &lt;a href=&quot;http://pan.oxfordjournals.org/content/9/2/137.abstract&quot; rel=&quot;nofollow&quot;&gt;King and Zheng (2001)&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These 'rare event corrections' were designed for case control research designs, mostly used in epidemiology, that select cases by choosing a fixed, usually balanced number of 0 cases and 1 cases, and then need to correct for the resulting sample selection bias.  Indeed, you might train your classifier the same way.  Pick a nice balanced sample and then correct the intercept to take into account the fact that you've selected on the dependent variable to learn more about rarer classes than a random sample would be able to tell you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Making Predictions&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On a related but distinct topic: Don't forget that you should be thresholding intelligently to make predictions.  It is not always best to predict 1 when the model probability is greater 0.5.  Another threshold may be better.  To this end you should look into the Receiver Operating Characteristic (ROC) curves of your classifier, not just its predictive success with a default probability threshold.&lt;/p&gt;&#xA;" OwnerUserId="1739" LastActivityDate="2011-01-08T16:01:51.453" CommentCount="2" />
  <row Id="6087" PostTypeId="2" ParentId="6081" CreationDate="2011-01-08T19:33:45.350" Score="1" ViewCount="" Body="&lt;p&gt;I think that the normality assumption can be assessed for each of the repeated measures, before performing the analysis. I would reshape the dataframe so that each column corresponds to a repeated measure, and then perform a shapiro.test to each one of those columns. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;apply(cast(melt(npk,measure.vars=&quot;yield&quot;), ...~N+P+K)[-c(1:2)],2,function(x) shapiro.test(x)$p.value)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="339" LastActivityDate="2011-01-08T19:33:45.350" CommentCount="3" />
  <row Id="6088" PostTypeId="2" ParentId="6078" CreationDate="2011-01-08T19:36:42.637" Score="2" ViewCount="" Body="&lt;p&gt;First of all, if your counts come from a different number of trials, then you cannot just ignore that. Apparently your data is better represent as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Pair Subject NTrials Hits&#xA;1       A     200     0&#xA;1       B     150     0&#xA;2       A     100     1&#xA;2       B     215     0&#xA;etc&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Second, there does not seem to be any reason to recode the number of hits as 0 or &gt;0 - you are throwing away data, and making the analysis more difficult (and probably less meaningful).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the analysis approach, I don't think you can make do with a simple chi-square test. You will need some sort of binomial regression, probably with a random effect accounting for the within-pair correlation. In fact, a test for the presence of this random effect would answer the question of the presence of correlation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively, you could calculate the probability of a hit for each participant, and then use some sort of weighted correlation, with each observation weighted by its inverse variance (np(1-p)).&lt;/p&gt;&#xA;" OwnerUserId="279" LastActivityDate="2011-01-08T19:36:42.637" />
  <row Id="6089" PostTypeId="2" ParentId="6081" CreationDate="2011-01-08T19:46:46.810" Score="0" ViewCount="" Body="&lt;p&gt;Another option would be to use the &lt;code&gt;lme&lt;/code&gt; function of the &lt;code&gt;nlme&lt;/code&gt; package (and then pass the obtained model to &lt;code&gt;anova&lt;/code&gt;). You can use &lt;code&gt;residuals&lt;/code&gt; on its output.&lt;/p&gt;&#xA;" OwnerUserId="582" LastActivityDate="2011-01-08T19:46:46.810" />
  <row Id="6090" PostTypeId="2" ParentId="6079" CreationDate="2011-01-08T20:05:44.290" Score="7" ViewCount="" Body="&lt;p&gt;$R^2$ is the squared correlation of the OLS prediction $\hat{Y}$ and the DV $Y$. In a multiple regression with three predictors $X_{1}, X_{2}, X_{3}$:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# generate some data&#xA;&amp;gt; N  &amp;lt;- 100&#xA;&amp;gt; X1 &amp;lt;- rnorm(N, 175, 7)                                 # predictor 1&#xA;&amp;gt; X2 &amp;lt;- rnorm(N,  30, 8)                                 # predictor 2&#xA;&amp;gt; X3 &amp;lt;- abs(rnorm(N, 60, 30))                            # predictor 3&#xA;&amp;gt; Y  &amp;lt;- 0.5*X1 - 0.3*X2 - 0.4*X3 + 10 + rnorm(N, 0, 10)  # DV&#xA;&amp;gt; fitX123 &amp;lt;- lm(Y ~ X1 + X2 + X3)  # regression&#xA;&amp;gt; summary(fitX123)$r.squared       # R^2&#xA;[1] 0.6361916&#xA;&#xA;&amp;gt; Yhat &amp;lt;- fitted(fitX123)          # OLS prediction Yhat&#xA;&amp;gt; cor(Yhat, Y)^2&#xA;[1] 0.6361916&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;$R^2$ is also equal to the variance of $\hat{Y}$ divided by the variance of $Y$. In that sense, it is the &quot;variance accounted for by the predictors&quot;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; var(Yhat) / var(Y)&#xA;[1] 0.6361916&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The squared semi-partial correlation of $Y$ with a predictor $X_{1}$ is equal to the increase in $R^2$ when adding $X_{1}$ as a predictor to the regression with all remaining predictors. This may be taken as the unique contribution of $X_{1}$ to the proportion of variance explained by all predictors. Here, the semi-partial correlation is the correlation of $Y$ with the residuals from regression where $X_{1}$ is the predicted variable and $X_{2}$ and $X_{3}$ are the predictors.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# residuals from regression with DV X1 and predictors X2, X3&#xA;&amp;gt; X1.X23 &amp;lt;- residuals(lm(X1 ~ X2 + X3))&#xA;&amp;gt; (spcorYX1.X23 &amp;lt;- cor(Y, X1.X23))   # semi-partial correlation of Y with X1&#xA;[1] 0.3172553&#xA;&#xA;&amp;gt; spcorYX1.X23^2                     # squared semi-partial correlation&#xA;[1] 0.1006509&#xA;&#xA;&amp;gt; fitX23 &amp;lt;- lm(Y ~ X2 + X3)          # regression with DV Y and predictors X2, X3&#xA;&#xA;# increase in R^2 when changing to full regression&#xA;&amp;gt; summary(fitX123)$r.squared - summary(fitX23)$r.squared&#xA;[1] 0.1006509&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1909" LastEditorUserId="1909" LastEditorDisplayName="" LastEditDate="2011-01-08T22:10:36.727" LastActivityDate="2011-01-08T22:10:36.727" CommentCount="5" />
  <row Id="6091" PostTypeId="1" AcceptedAnswerId="6094" CreationDate="2011-01-08T21:00:42.343" Score="1" ViewCount="80" Body="&lt;p&gt;I have a bunch of data like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;P1 [1, 2, 3, 4, 5, 6, 7, 8, 10]&#xA;P2 [5, 8, 10, 12, 20]&#xA;P3 [10, 201, 440]&#xA;P4 [1, 2, 10]&#xA;P5 [1, 2]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Right now it's ranked by the size of its data set (as shown above), so values that are heavily represented on the chart are on top e.g., P1 has the most)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm trying to rank this data so that it favors those that have lower values in data set. So for instance P5 (with low values) would be pushed higher than it's now.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, P2 could be ranked higher because although it has less values than P1, the values are mostly in the higher range.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically I want the data to be sorted by the size of it and its overall tendency to have lower values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can average it all out but I don't wanna give weight to data with small set (e.g., 1 or 2 values)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So something like this would be nice (guessing):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;P1 [1, 2, 3, 4, 5, 6, 7, 8, 10]&#xA;P5 [1, 2]&#xA;P4 [1, 2, 10]&#xA;P2 [1, 5, 8, 10, 12, 20]&#xA;P3 [10, 201, 440, 500]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm looking for an R solution.&lt;/p&gt;&#xA;" OwnerUserId="2664" LastActivityDate="2011-01-08T23:36:27.173" Title="Good way to rank/sort data that falls in the lower range" Tags="&lt;r&gt;&lt;distributions&gt;&lt;mean&gt;" AnswerCount="1" />
  <row Id="6092" PostTypeId="1" CreationDate="2011-01-08T23:01:11.587" Score="3" ViewCount="65" Body="&lt;p&gt;Is there a way to use &lt;code&gt;install.packages()&lt;/code&gt; so that it will build and install both 32-bit and 64-bit versions of a package from &lt;strong&gt;source&lt;/strong&gt; by default?  I have multiple computers, all running Mac OS X on 32-bit and 64-bit Intel CPUs.  I keep installed packages in my home directory, and I like the convenience of syncing my home directory between my computers.  Right now I have to launch R twice—once for each architecture.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It’s not a big deal, just annoying to do each and every time.&lt;/p&gt;&#xA;" OwnerUserId="1670" LastEditorUserId="1670" LastEditorDisplayName="" LastEditDate="2011-01-08T23:07:08.573" LastActivityDate="2011-01-08T23:07:08.573" Title="Use install.packages() to build and install all available architectures by default?" Tags="&lt;r&gt;" CommentCount="2" />
  <row Id="6093" PostTypeId="1" CreationDate="2011-01-08T23:09:08.913" Score="0" ViewCount="259" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;Possible Duplicate:&lt;/strong&gt;&lt;br&gt;&#xA;  &lt;a href=&quot;http://stats.stackexchange.com/questions/421/what-book-would-you-recommend-for-non-statisticians&quot;&gt;What book would you recommend for non-statisticians?&lt;/a&gt;  &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&#xA;&#xA;&lt;p&gt;Hello all, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which is a good book for self-learning of statistics? I tried the &quot;Head first statistics&quot; which starts very well, but when talking on t-distribution and some more complex issues, its quality degrades (just my opinion). Also, smth like &quot;for dummies&quot; might be too simplistic. I started now &quot;Statistics in plain english&quot;, it might be a good startup for intuition, but I guess it will not suffice later on. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, I considered Feller's volumes on probability theory as a very good start for studying that domain. I am looking for similar consistent books in the domain of Statistics. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="976" LastActivityDate="2011-01-09T01:02:26.773" ClosedDate="2011-01-09T12:50:39.447" Title="Good book on statistics" Tags="&lt;books&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="2" />
  <row Id="6094" PostTypeId="2" ParentId="6091" CreationDate="2011-01-08T23:36:27.173" Score="3" ViewCount="" Body="&lt;p&gt;If you take the average of each row, divide by its length, and order the rows by that value, it produces the results you wanted:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; ranks &amp;lt;- unlist(lapply(p,mean))/unlist(lapply(p,length))&#xA;&amp;gt; ranks&#xA;        P1         P2         P3         P4         P5 &#xA; 0.5679012  2.2000000 72.3333333  1.4444444  0.7500000 &#xA;&amp;gt; p[order(ranks)]&#xA;$P1&#xA;[1]  1  2  3  4  5  6  7  8 10&#xA;&#xA;$P5&#xA;[1] 1 2&#xA;&#xA;$P4&#xA;[1]  1  2 10&#xA;&#xA;$P2&#xA;[1]  5  8 10 12 20&#xA;&#xA;$P3&#xA;[1]  10 201 440`&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="697" LastActivityDate="2011-01-08T23:36:27.173" CommentCount="2" />
  <row Id="6095" PostTypeId="2" ParentId="6093" CreationDate="2011-01-09T01:02:26.773" Score="1" ViewCount="" Body="&lt;p&gt;If you liked Feller, then you definitely need to think about a real stats book. My suggestion as a self-learner at perhaps slightly less sophisticated level than you would be to look for Cox and Hinkley's &quot;Theoretical Statistics&quot;. Very readable for a stats text. Not much, if any, in the way of problems sets, though.&lt;/p&gt;&#xA;" OwnerUserId="2129" LastActivityDate="2011-01-09T01:02:26.773" />
  <row Id="6096" PostTypeId="1" CreationDate="2011-01-09T08:20:48.563" Score="4" ViewCount="111" Body="&lt;p&gt;I'm working on benchmarking the speed of various JavaScript methods.&#xA;Part of the benchmark process requires repeating a test for a minimum time &lt;em&gt;(to reduce the percentage uncertainty to at or less than 1%)&lt;/em&gt;. There is some overhead on each test &lt;em&gt;(the cost of the loop, incrementing a counter variable and so on)&lt;/em&gt;. I currently benchmark an empty test to get the overhead cost. The result has it's own mean and margin of error. How should I apply this calibration? Do I take the calibration mean and subtract it from every measured value that compose other benchmark samples, or should I subtract it from the other benchmark's mean value even with differences in margin of error, or is their some formula to follow?&lt;/p&gt;&#xA;" OwnerUserId="2634" LastEditorUserId="2634" LastEditorDisplayName="" LastEditDate="2011-01-09T17:29:55.527" LastActivityDate="2011-01-09T17:29:55.527" Title="Correct way to calibrate means" Tags="&lt;calibration&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="1" />
  <row Id="6097" PostTypeId="1" CreationDate="2011-01-09T11:03:31.153" Score="5" ViewCount="61" Body="&lt;p&gt;Say I have a process that gives me 3 outputs: $O^1$, $O^2$ and $O^3$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The outputs are generated from a semi-deterministic process, i.e. there is a deterministic component in the outputs, along with a random component.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In particular, having $n$ measurements over time, the outputs $O_j \quad j=1,2,...,n$ are -at least in part- dependent on the previous outputs. So $O_j = f(O_{j-1}, O_{j-2}, O_{j-3}) + \epsilon$ (I'm not interested in going farther away than 2 or 3 measurements, $\epsilon$ is the random component).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So now I have a set of ~150 consecutive measurements, how can I predict what are the likely outputs in the future?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can easily calculate the distribution of values following a certain output, for instance I could say that if $O^1_j$ is between 50 and 60 I have a certain probability of $O^1_{j+1}$ of being between 30 and 40, by looking at the measurements that I took in the past. I did construct some pdf for the distribution of these probabilities, but now I'm a bit stuck, especially because there is probably an interaction between the three outputs (so for instance, updating my previous statement $O^1_j = f(O^1_{j-l}, O^2_{j-l}, O^3_{j-l}) + \epsilon \quad\quad l=1,2,3$)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've been reading about Bayesian predictors and I tought they could be applied here, but I don't know enough about the topic to determine if this is a good choice or if there is something easier/more appropriate. I will appreciate any suggestion!&lt;/p&gt;&#xA;" OwnerUserId="582" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2011-01-09T23:19:23.807" LastActivityDate="2011-01-09T23:19:23.807" Title="Predicting a semi-deterministic process" Tags="&lt;time-series&gt;&lt;probability&gt;&lt;bayesian&gt;&lt;forecasting&gt;&lt;predictive-models&gt;" AnswerCount="1" CommentCount="4" />
  <row Id="6098" PostTypeId="2" ParentId="6046" CreationDate="2011-01-09T12:27:42.207" Score="4" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Repeated Measures&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Personally I would pursue a hierarchical model where the basic observations are, for each animal, the 4 (or fewer) levels of odour and the corresponding neuronal responses.  And the predictor for the per animal intercept and slope on this relationship is the animal's reproductive status.  (Here I'm assuming that your interest is in the effect of reproductive status on these aspects of the response function and to what extent it is distinguishable from individual variation.)  That would give you nice interpretable animal level regression parameters, e.g. moving from virgin to mated animals drops the predicted firing rate by x and increases the effect of one unit increase in odour dose by z.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Failing that, a mixed model with reproductive fixed effect would probably also work.  Actually I think that's all SPSS 16 offers you anyway.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wouldn't immediately worry about missing data in this framework.  Just try it and then check for robustness of the results, as Rob suggests. The more basic problem is knowing what SPSS is telling you when you fit one of these models.  For that, you'll want to read up a bit first.  Other folk here may have preferred introductions to mixed models - mine are all R-oriented and therefore not so helpful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Spikes&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are working with spike counts they are probably conditionally Poisson distributed (and don't forget the offset, if the exposure during measurement varies).  If you don't have the option to specify that fact you might need to fit the model in appropriately adjusted log counts or suchlike.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Missing Data&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have enough animals missing a few measurements for some of them might be ok.  For a lot of missing, there won't be much information as Rob also points out.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you (or your audience) worry about the missing data, you could do multiple imputation first.  If I remember right, and I don't really use SPSS for anything, 16 makes you use AMOS for multiple imputation, but later version have it built into the missing value module.  So that might be an option.&lt;/p&gt;&#xA;" OwnerUserId="1739" LastActivityDate="2011-01-09T12:27:42.207" />
  <row Id="6100" PostTypeId="2" ParentId="6096" CreationDate="2011-01-09T13:24:09.993" Score="4" ViewCount="" Body="&lt;p&gt;There is something called &quot;small error propagation&quot;, and it says that the error of a function $f$ of variables $x_1,x_2,\cdots,x_n$ with errors $\Delta x_1,\Delta x_2,\cdots,\Delta x_n$ equals&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\Delta f=\sqrt{\sum_i\left(\frac{\partial f}{\partial x_i}\Delta x_i\right)^2}$,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;so for $f(a,b):=a-b$ the error is $\Delta f=\sqrt{\Delta a^2+\Delta b^2}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, subtract the means and report this euclidean length of errors as a final error.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2011-01-09T13:24:09.993" CommentCount="5" />
  <row Id="6101" PostTypeId="1" CreationDate="2011-01-09T16:51:41.767" Score="2" ViewCount="114" Body="&lt;p&gt;Hi, I have a real-life situation that can be solved using Queueing Theory.&lt;br&gt;&#xA;This should be easy for someone in the field.  Any pointers would be appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Scenario:&lt;/strong&gt;&lt;br&gt;&#xA;There is a single Queue and &lt;strong&gt;N&lt;/strong&gt; Servers.&lt;br&gt;&#xA;When a server becomes free, the Task at the front of the queue gets serviced.&lt;br&gt;&#xA;The mean service time is &lt;strong&gt;T&lt;/strong&gt; seconds.&lt;br&gt;&#xA;The mean inter-Task arrival time is &lt;strong&gt;K&lt;/strong&gt; * &lt;strong&gt;T&lt;/strong&gt; (where &lt;strong&gt;K&lt;/strong&gt; &gt; 1)&lt;br&gt;&#xA;(assume Poisson or Gaussian distributions, whichever is easier to analyze.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question:&lt;/strong&gt;&lt;br&gt;&#xA;At steady state, what is the length of the queue?  (in terms of &lt;strong&gt;N&lt;/strong&gt;, &lt;strong&gt;K&lt;/strong&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Related Question:&lt;/strong&gt;&lt;br&gt;&#xA;What is the expected delay for a Task to be completed?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the real-life situation I am trying to model:&lt;br&gt;&#xA;I have an Apache web server with 25 worker processes.&lt;br&gt;&#xA;At steady-state there are 125 requests in the queue.&lt;br&gt;&#xA;I want to have a theoretical basis to help me optimize resources&#xA;and understand quantitatively how adding more worker processes&#xA;affects the queue length and delay.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know the single queue, single server, Poisson distribution is well analyzed.&lt;br&gt;&#xA;I don't know the more general solution for &lt;strong&gt;N&lt;/strong&gt; servers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;thanks in advance,&lt;br&gt;&#xA;-- David Jones&lt;br&gt;&#xA;dxjones@gmail.com&lt;/p&gt;&#xA;" OwnerUserId="2711" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-04-13T10:33:54.043" LastActivityDate="2011-04-13T10:33:54.043" Title="Queueing Theory: How to estimate steady-state queue length for single queue, N servers?" Tags="&lt;poisson&gt;&lt;networks&gt;&lt;simulation&gt;&lt;queueing&gt;" CommentCount="1" />
  <row Id="6102" PostTypeId="1" AcceptedAnswerId="6105" CreationDate="2011-01-09T18:51:29.387" Score="3" ViewCount="113" Body="&lt;p&gt;I'm currently applying   the Roy Zelner test of poolability as shown&#xA;in the excellent &lt;a href=&quot;http://doc.rero.ch/lm.php?url=1000,42,6,20080417092244-DQ/wp0804.pdf&quot; rel=&quot;nofollow&quot;&gt;article of Andrea Vaona&lt;/a&gt;,&#xA;in fact I'm working with panel N=17 T=5, and my model  looks like&#xA;this : &#xA;$$Y_{it}= a_0+B_1X_1+B_2X_2+B_3X_3+B_4X_4+e_{it}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is the following. When I'm  testing  for coefficient equality of the unpooled data (the last stage), many of my&#xA;constraints are getting dropped. This impacts the degrees  of freedom of&#xA;$\chi^2$, and I would like to understand the reason?&#xA;Is this because the time dimension of my panel is too small? or&#xA;because the number of my constraints is too high?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ama&lt;/p&gt;&#xA;" OwnerUserId="1251" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2011-01-13T20:54:35.390" LastActivityDate="2011-01-13T20:54:35.390" Title="Why are my constraints getting dropped?" Tags="&lt;regression&gt;&lt;panel-data&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="6103" PostTypeId="2" ParentId="5026" CreationDate="2011-01-09T19:59:14.237" Score="3" ViewCount="" Body="&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Statistics&lt;/strong&gt; is concerned with probabilistic models, specifically inference on these models using data.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Machine Learning&lt;/strong&gt; is concerned with predicting a particular outcome given some data. Almost any reasonable machine learning method can be formulated as a formal probabilistic model, so in this sense machine learning is very much the same as statistics, but it differs in that it generally doesn't care about parameter estimates (just prediction) and it focuses on computational efficiency and large datasets.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Data Mining&lt;/strong&gt; is (as I understand it) applied machine learning. It focuses more on the practical aspects of deploying machine learning algorithms on large datasets. It is very much similar to machine learning.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Artificial Intelligence&lt;/strong&gt; is anything that is concerned with (some arbitrary definition of) intelligence in computers. So, it includes &lt;em&gt;a lot&lt;/em&gt; of things.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In general, probabilistic models (and thus statistics) have proven to be the most effective way to formally structure knowledge and understanding in a machine, to such an extent that all three of the others (AI, ML and DM) are today mostly subfields of statistics. Not the first discipline to become a shadow arm of statistics... (Economics, psychology, bioinformatics, etc.)&lt;/p&gt;&#xA;" OwnerUserId="2713" LastActivityDate="2011-01-09T19:59:14.237" CommunityOwnedDate="2011-01-09T19:59:14.237" />
  <row Id="6104" PostTypeId="1" AcceptedAnswerId="6121" CreationDate="2011-01-09T20:08:12.767" Score="1" ViewCount="197" Body="&lt;p&gt;I start with three independent random variables, $X_1, X_2, X_3$. They are each normally distributed with:  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$X_i \sim N(\mu_i, \sigma^2), i = 1, 2, 3.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I then have three transformations, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\eqalign{&#xD;&#xA;Y_1 &amp;amp;= -X_1/\sqrt{2} + X_2/\sqrt{2}  \cr&#xD;&#xA;Y_2 &amp;amp;= -X_1/\sqrt{3} - X_2/\sqrt{3} + X_3/\sqrt{3} \cr &#xD;&#xA;Y_3 &amp;amp;= X_1/\sqrt{6} + X_2/\sqrt{6} + 2X_3 / \sqrt{6}  \cr&#xD;&#xA;}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am supposed to show that when $\mu_i = 0,$ $i = 1, 2, 3,$ $(Y_1^2 + Y_2^2 + Y_3^2)/\sigma^2 \sim \chi^2(3)$. I have also shown the transformations to preserve the independence, as the transformation matrix is orthogonal. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have already shown that the expectations of $Y_1, Y_2, Y_3$ is 0 and their variances are all the same. Using the normal pdf, I have shown that: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Y_i^2 \sim \frac{1}{2\pi\sigma^2} \exp(-2x^2 / 2\sigma^2).$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I thought about applying a substitution of $z = 2x^2 / \sigma^2$ to get the exponent into a similar form as the chi-square's $\exp(-x/2)$ form, but I'm stuck on what to do with the constants outside to get them to look similar. Could someone offer a hand?&lt;/p&gt;&#xA;" OwnerUserId="1118" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-01-10T02:02:53.867" LastActivityDate="2011-01-10T08:10:03.080" Title="Proving that the squares of normal rv's is Chi-square distributed" Tags="&lt;distributions&gt;&lt;probability&gt;&lt;homework&gt;&lt;chi-squared&gt;&lt;mathematical-statistics&gt;" AnswerCount="2" CommentCount="6" />
  <row Id="6105" PostTypeId="2" ParentId="6102" CreationDate="2011-01-09T20:15:40.220" Score="5" ViewCount="" Body="&lt;p&gt;You have a panel data regression&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$y_{it}=x_{it}&amp;#39;\beta+u_{it},$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $x_{it}$ in your case is $(1,X_1,X_2,X_3,X_4)$. Poolability tests test whether  alternative model is actually correct:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$y_{it}=x_{it}&amp;#39;\beta_i+u_{it}.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the null hypothesis is that $\beta_i=\beta$. To test this hypothesis we need to estimate $\hat{\beta_i}$. In your case, you need to estimate 17 $\beta_i$. Since $T=5$, your are estimating regressions with 5 parameters having 5 observations. This of course gives you a lot of problems, since the usual practice for statistical packages in this case is to drop some of the variables from the regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general if $T$ is small do not test whether you can pool the data. Simply use panel data regression and check whether the resulting model is appropriate. &lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-01-09T20:15:40.220" CommentCount="3" />
  <row Id="6106" PostTypeId="2" ParentId="6097" CreationDate="2011-01-09T20:48:21.907" Score="4" ViewCount="" Body="&lt;p&gt;If you want to forecast time-series data, first you need to check whether it is &lt;a href=&quot;http://en.wikipedia.org/wiki/Stationary_process&quot; rel=&quot;nofollow&quot;&gt;stationary&lt;/a&gt;. Basically this means checking whether data has trends. If for example some time trend is present, you can concern yourself only with its forecast, because time-trends usually dominate everything else. For stationary time series it is good to use &lt;a href=&quot;http://en.wikipedia.org/wiki/Box-Jenkins&quot; rel=&quot;nofollow&quot;&gt;Box-Jenkins&lt;/a&gt; approach. This in the end will give you some kind of  &lt;a href=&quot;http://en.wikipedia.org/wiki/ARMA&quot; rel=&quot;nofollow&quot;&gt;ARMA&lt;/a&gt; model (autoregressive model suggested by @whuber is a subset of this model). Since you have three time series you may look into &lt;a href=&quot;http://en.wikipedia.org/wiki/Vector_autoregression&quot; rel=&quot;nofollow&quot;&gt;VAR&lt;/a&gt; models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you use R, then first step can be performed by function &lt;code&gt;stl&lt;/code&gt;, it is function from standard R. Autoregressive models can be fit automatically  by &lt;code&gt;auto.arima&lt;/code&gt; in package &lt;a href=&quot;http://cran.r-project.org/web/packages/forecast/&quot; rel=&quot;nofollow&quot;&gt;forecast&lt;/a&gt;. This function can either fit your desired model, or find the best specification for certain definition of best. You might look into that package more, since it is specially designed for forecasting time series. For VAR model use &lt;code&gt;VAR&lt;/code&gt; function from &lt;a href=&quot;http://cran.r-project.org/web/packages/vars/index.html&quot; rel=&quot;nofollow&quot;&gt;vars&lt;/a&gt; package. This package has a nice vignette describing its capabilities.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-01-09T20:48:21.907" CommentCount="1" />
  <row Id="6107" PostTypeId="2" ParentId="6104" CreationDate="2011-01-09T21:09:26.587" Score="1" ViewCount="" Body="&lt;p&gt;Have you tried simply multiplying out the squared Y^2's in terms of the X[1:3] terms. I suspect that when you are all done that you will see that you simply have: (1/2 +1/3 +1/6)* X1^2 + (1/2 +1/3 +1/6)*X2^2 + (1/2 +1/3 +1/6)*X3^2 . This, of course, assumes that X1X3=X3X1, i.e. that your random variable algebra is commutative, but unless you are working on  complex variables in particle physics, that assumption should hold. So far I have gotten about halfway there, and my approach seems to be holding up. It would seem to be useful that you go through the exercise, rather than for me to display it.&lt;/p&gt;&#xA;" OwnerUserId="2129" LastActivityDate="2011-01-09T21:09:26.587" CommentCount="4" />
  <row Id="6108" PostTypeId="1" AcceptedAnswerId="6116" CreationDate="2011-01-09T21:57:48.190" Score="1" ViewCount="112" Body="&lt;p&gt;Suppose I have a biased coin (heads with probability p), and I keep flipping it until I get t tails. So I had to flip n times in total to get t tails. How do I find a (frequentist) confidence interval for the probability of heads p?&lt;/p&gt;&#xA;" OwnerUserId="1106" LastActivityDate="2011-01-10T01:31:24.363" Title="Confidence interval for success probability in negative binomial experiment" Tags="&lt;confidence-interval&gt;&lt;negative-binomial&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="6109" PostTypeId="1" AcceptedAnswerId="6117" CreationDate="2011-01-09T22:08:34.650" Score="1" ViewCount="122" Body="&lt;p&gt;This is a softmax probability distribution:&lt;br&gt;&#xA;$$P(i| w_1, w_2, \ldots, w_n) = \frac{exp(w_i)}{\sum_{i=1}^n exp(w_i)}.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It known also as Boltzmann distribution. It is used in generalized Bradley-Terry model and in multinomial logistic regression. There are efficient minorization-maximization algorithms for infering $\vec{w}$ from data through Maximal Likelihood principle.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm looking for similar distribution but extended with &quot;variance&quot; parameter vector.&#xA;A parameter that would represent the confidence in $\vec{w}$.&#xA;Then I would like to infer both $\vec{w}$ and its confidence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anybody of you happen to know such distribution or a research on the topic?&lt;/p&gt;&#xA;" OwnerUserId="217" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-01-10T02:14:39.147" LastActivityDate="2011-01-11T20:44:19.740" Title="Is there a SoftMax distribution with confidence parameters?" Tags="&lt;distributions&gt;&lt;maximum-likelihood&gt;&lt;bradley-terry-model&gt;&lt;softmax&gt;" AnswerCount="3" />
  <row Id="6110" PostTypeId="2" ParentId="6109" CreationDate="2011-01-09T22:28:29.373" Score="1" ViewCount="" Body="&lt;p&gt;The Maxwell distribution is the classical limit under conditions of high temperature and non-interacting wave functions of both Fermi-Dirac statistics and Bose-Einstein statistics. I would expect that you would want to look at the F-D statistics if you are interested in higher variance, since Bose-Einstein statistics lead to aggregations of particles, whereas F-D statistics (and the Pauli exclusion principle) are what keep neutron stars from further collapse:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(F-D without normalization): \begin{align}&#xA; P(i| w_1, w_2, \ldots, w_n) = \frac{exp(w_i)}{\sum_{i=1}^n exp(w_i) +1}&#xA;\end{align}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: The Bradley-Terry model is a special case of a more general paired-choice model that was &lt;a href=&quot;http://biomet.oxfordjournals.org/content/77/2/265.abstract&quot; rel=&quot;nofollow&quot;&gt;proposed by Stern&lt;/a&gt;:&lt;br&gt;&#xA;In The &lt;a href=&quot;http://eom.springer.de/default.htm&quot; rel=&quot;nofollow&quot;&gt;Springer Encyclopedia of Mathematics&lt;/a&gt; you read: &quot;H. Stern has considered, [a6], models for paired comparison experiments based on comparison of gamma random variables. Different values of the shape parameter yield different models, including the Bradley–Terry model and the Thurstone model. Likelihood methods can be used to estimate the parameters of the models. The likelihood equations must be solved with iterative methods.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;H. Stern,   &quot;A continuum of paired comparison models&quot;  Biometrika , 77  (1990)  pp. 265–273&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://biomet.oxfordjournals.org/content/77/2/265.abstract&quot; rel=&quot;nofollow&quot;&gt;http://biomet.oxfordjournals.org/content/77/2/265.abstract&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2129" LastEditorUserId="2129" LastEditorDisplayName="" LastEditDate="2011-01-11T20:44:19.740" LastActivityDate="2011-01-11T20:44:19.740" CommentCount="3" />
  <row Id="6111" PostTypeId="1" CreationDate="2011-01-09T22:42:13.717" Score="2" ViewCount="253" Body="&lt;p&gt;Here is the problem: A survey contains 7 binary questions (Yes/No responses). If two people are answering the survey, what is the probability for their answers on 4 or more of the questions to match? In other words, if we have four or more matching answers, we can consider the overall survey response to be similar for both people.&lt;/p&gt;&#xA;" OwnerUserId="2715" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-01-11T08:38:38.173" LastActivityDate="2011-01-11T08:38:38.173" Title="The probability for two people to provide identical answers on survey questions" Tags="&lt;probability&gt;&lt;permutation&gt;&lt;independence&gt;" AnswerCount="2" CommentCount="3" />
  <row Id="6112" PostTypeId="2" ParentId="6111" CreationDate="2011-01-09T23:22:43.173" Score="0" ViewCount="" Body="&lt;p&gt;If for each question the probability of selecting the same answer is equal to 0.5, the answer is the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\sum_{i=4}^7{\binom{7}{i}p^i(1-p)^{7-i}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $p=0.5.$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this case it is a &lt;a href=&quot;http://en.wikipedia.org/wiki/Binomial_distribution&quot; rel=&quot;nofollow&quot;&gt;binomial distribution&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1540" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-01-10T01:54:31.270" LastActivityDate="2011-01-10T01:54:31.270" CommentCount="2" />
  <row Id="6113" PostTypeId="2" ParentId="6063" CreationDate="2011-01-10T00:19:12.147" Score="1" ViewCount="" Body="&lt;p&gt;The effect sizes of interactions in a multivariate regression can be assessed in same way as the effect sizes of any other predictor. The common thing is to look at the incremental contribution to R^2 (semi-partial R^2), but there are other possibilities, including Cohen's f^2 for nested models (this is a likelihood ratio test). Chapter 9 of Cohen's Statistical Power Analysis for the Behavioral Sciences has a very good discussion. It is true, too, that the effect sizes of product-term interactions tend to be &quot;small&quot; in terms of incremental addition to R^2. But the &lt;em&gt;practical&lt;/em&gt; effect of such an interaction can be very large. This point is especially important to bear in mind when the interaction involves some sort of treatment or intervention -- e.g., in a drug trial, the practical effect of an interaction between the treatment &amp;amp; some individual characteristic of a patient might contribute only a small amount to mode R^2 but have a very appreciable effect on the clinical outcomes. See Rosenthal, R. &amp;amp; Rubin, D.B. A Note on Percent Variance Explained as A Measure of the Importance of Effects. J Appl Soc Psychol 9, 395-396 (1979); Abelson, R.P. A Variance Explanation Paradox: When a Little is a Lot. Psychological Bulletin 97, 129-133 (1985). Both this possibility &amp;amp; the challenge of trying to interpret (or explicate) the simultaneous importance of the coefficients for the predictor, moderator &amp;amp; product-interaction in a regression output, tend to make reporting of the interaction's &quot;effect size&quot; uninformative; better, I'd say, is to illustrate (graphically) the effect size of the interaction in practical terms -- that is, by showing how changes in meaningful levels of the predictor (&quot;high exposure vs. low exposure&quot;) and moderator (&quot;being a man vs. being a woman&quot;) affect the outcome variable expressed in units that make sense given the context (&quot;additional yrs of life&quot;). I don't have as much experience w/ multilvel modeling, but I do know that the strategy I'm describing is the basic philosophy of Gelman, A. &amp;amp; Hill, J. Data Analysis Using Regression and Multilevel/Hierarchical Models. (Cambridge University Press, Cambridge ; New York; 2007)-- the greatest work on regression, in my opinion, after Cohen &amp;amp; Cohen!&lt;/p&gt;&#xA;" OwnerUserId="2347" LastActivityDate="2011-01-10T00:19:12.147" />
  <row Id="6114" PostTypeId="2" ParentId="6111" CreationDate="2011-01-10T01:07:09.440" Score="2" ViewCount="" Body="&lt;p&gt;I assume that the survey will be answered independently by the participants. First, you need estimates for the baseline probabilities $p_{i}$ that an answer $i$ will be answered &quot;yes&quot;. The probability of two persons answering &quot;yes&quot; for question $i$ is then $p_{i}^{2}$. Likewise, the probability of two persons answering &quot;no&quot; for question $i$ is $(1-p_{i})^{2}$, hence the probability of agreement is $p_{i}^{2} + (1-p_{i})^{2}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you assume that all $p_{i} = 0.5$, then you get the answer given by carlosdc since $0.5^{2} + (1-0.5)^{2} = 0.5$. If you allow the $p_{i}$ to vary, an answer can probably be given in closed form as well, but with only 7 questions, it's easy to simply enumerate all possibilities to get 4 or more agreements, and calculate the probability for each case.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; n &amp;lt;- 7            # number of questions&#xA;&amp;gt; p &amp;lt;- rep(0.5, n)  # probabilities p_i, here: set all to 0.5&#xA;# p &amp;lt;- c(0.4, 0.4, 0.4, 0.4, 0.1, 0.1, 0.1) # alternative: let p_i vary&#xA;&amp;gt; k &amp;lt;- 4:7          # number of agreements to check&#xA;# k &amp;lt;- 0:7          # check: result (total probability) should be 1&#xA;&#xA;# vector to hold probability for each number of agreements&#xA;&amp;gt; res &amp;lt;- numeric(length(k))&#xA;&#xA;# function to calculate the probability for an event with agreement on the&#xA;# questions x and disagreement on the remaining questions&#xA;&amp;gt; getP &amp;lt;- function(x) {&#xA;+     tf &amp;lt;- 1:n %in% x              # convert numerical to logical index vector&#xA;+     pp &amp;lt;- p[tf]^2 + (1-p[tf])^2   # probabilities of agreeing on questions x&#xA;+&#xA;+     # probabilities of disagreeing on remaining questions&#xA;+     qq &amp;lt;- 1 - (p[!tf]^2 + (1-p[!tf])^2)&#xA;+     prod(pp) * prod(qq)           # total probability&#xA;+ }&#xA;&#xA;# for each number of agreements: calculate probability&#xA;&amp;gt; for(i in seq(along=res)) {&#xA;+     # all choose(n, k) possibilities to have k agreements&#xA;+     poss &amp;lt;- combn(1:n, k[i])&#xA;+&#xA;+     # probability for each of those possibilities, edit: take 0-length into account&#xA;+     if (length(poss) &amp;gt; 0) {&#xA;+         res[i] &amp;lt;- sum(apply(poss, 2, getP))&#xA;+     } else {&#xA;+         res[i] &amp;lt;- getP(numeric(0))&#xA;+     }&#xA;+ }&#xA;&#xA;&amp;gt; res                # probability for 4, 5, 6, 7 agreements&#xA;[1] 0.2734375 0.1640625 0.0546875 0.0078125&#xA;&#xA;&amp;gt; dbinom(k, n, 0.5)  # check: all p_i = 0.5 -&amp;gt; binomial distribution&#xA;[1] 0.2734375 0.1640625 0.0546875 0.0078125&#xA;&#xA;&amp;gt; sum(res)           # probability for 4 or more agreements&#xA;[1] 0.5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The R code could certainly be simplified, also &lt;code&gt;prod()&lt;/code&gt; might be worse in terms of error propagation with small numbers than &lt;code&gt;exp(sum(log()))&lt;/code&gt;, although I'm not sure on that one.&lt;/p&gt;&#xA;" OwnerUserId="1909" LastEditorUserId="1909" LastEditorDisplayName="" LastEditDate="2011-01-10T19:04:34.480" LastActivityDate="2011-01-10T19:04:34.480" CommentCount="13" />
  <row Id="6115" PostTypeId="2" ParentId="6109" CreationDate="2011-01-10T01:26:19.127" Score="1" ViewCount="" Body="&lt;p&gt;Hello Lew:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not sure, but I think your probability model is a special case of &lt;a href=&quot;http://en.wikipedia.org/wiki/Multinomial_logit&quot; rel=&quot;nofollow&quot;&gt;Multinomial logit&lt;/a&gt; model with no covariates and only the intercept terms ($w_i$ will be the intercepts).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is model is a special case of GLM and hence there exits an iteratively weighted least square method (IRWLS) to get the maximum likelihood estimates of $w_i$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you don't want to code the IRWLS algorithm yourself, please check the &lt;code&gt;polr&lt;/code&gt; function in the &lt;code&gt;MASS&lt;/code&gt; library in R to accomplish the ML estimation of $w_i$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As @whuber points out, unless you adopt a Bayesian approach, there is no distribution for the parameters (assumed fixed), but there exits one for the estimates (here: ML) as they are estimated from the data, hence the randomness.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;HTH&lt;/p&gt;&#xA;&#xA;&lt;p&gt;S.&lt;/p&gt;&#xA;" OwnerUserId="1307" LastEditorUserId="1307" LastEditorDisplayName="" LastEditDate="2011-01-10T02:51:03.933" LastActivityDate="2011-01-10T02:51:03.933" />
  <row Id="6116" PostTypeId="2" ParentId="6108" CreationDate="2011-01-10T01:31:24.363" Score="3" ViewCount="" Body="&lt;p&gt;Hello Grautur:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are looking for ML estimation (which may or may not be what you want) please check the &lt;code&gt;fitdistr&lt;/code&gt; function in the &lt;code&gt;MASS&lt;/code&gt;library in &lt;code&gt;R&lt;/code&gt;. This function can estimate the unknown parameters using ML estimation. But please be careful about the parametrization of the negative binomial distribution of &lt;code&gt;MASS&lt;/code&gt;. Apart from estimating the unknown parameters, it gives the confidence intervals for the estimates.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;HTH&#xA;S.&lt;/p&gt;&#xA;" OwnerUserId="1307" LastActivityDate="2011-01-10T01:31:24.363" />
  <row Id="6117" PostTypeId="2" ParentId="6109" CreationDate="2011-01-10T02:14:16.637" Score="6" ViewCount="" Body="&lt;p&gt;This question appears to confuse two distinct things.  Any additional parameter in the model would (by definition) describe the distribution of $i$, &lt;strong&gt;not&lt;/strong&gt; the distributions of any of the $w_i$.  Unless you adopt a Bayesian prior for $\vec{w}$ (which does not seem to be part of this question), the parameters do not have any distribution at all: they are what they are.  When you use a particular &lt;em&gt;procedure&lt;/em&gt; to &lt;em&gt;estimate&lt;/em&gt; $\vec{w}$, however, then the &lt;em&gt;estimates&lt;/em&gt; $(\hat{w_i})$ do have a distribution.  It makes sense to talk about the variance of that distribution.  It can be estimated in standard ways, such as the &lt;a href=&quot;http://en.wikipedia.org/wiki/Maximum_likelihood&quot; rel=&quot;nofollow&quot;&gt;inverse of the expectation of the Hessian of the log likelihood&lt;/a&gt;.  It is unnecessary--and meaningless--to introduce yet another parameter to capture that information.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-01-10T02:14:16.637" CommentCount="2" />
  <row Id="6118" PostTypeId="1" CreationDate="2011-01-10T05:09:45.743" Score="0" ViewCount="130" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;Possible Duplicate:&lt;/strong&gt;&lt;br&gt;&#xA;  &lt;a href=&quot;http://stats.stackexchange.com/questions/421/what-book-would-you-recommend-for-non-statisticians&quot;&gt;What book would you recommend for non-statisticians?&lt;/a&gt;  &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&#xA;&#xA;&lt;p&gt;Hi, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone recommend a good statistic textbook for me? I am a comparative ophthalmology research and do not have any academic statistic training.  I need to understand this and come up to speed quickly.  Thank you very much. &lt;/p&gt;&#xA;" OwnerUserId="2716" LastActivityDate="2011-01-10T05:09:45.743" ClosedDate="2011-01-10T07:03:41.650" Title="Recommending a good statistic textbook" Tags="&lt;books&gt;" CommentCount="1" />
  <row Id="6119" PostTypeId="1" AcceptedAnswerId="6125" CreationDate="2011-01-10T05:50:59.077" Score="6" ViewCount="182" Body="&lt;p&gt;I'm analyzing people based on their twitter stream. We are using a 'word bag' model of users, which basically amounts to counting how often each word appears in a persons twitter stream (and then using that as a proxy for a more normalized 'probability they will use a given word' in a particular length of text). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Due to constraints further down the pipeline, we cannot retain full data on usage of all words for all users, so we are trying to find the most 'symbolically efficient' words to retain in our analysis. That is, we're trying to retain a subset of dimensions, which, knowing their values would allow a hypothetical seer to most accurately model the probabilities of all words (including any we left out of the analysis).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So a principal components analysis (PCA) type approach seems an appropriate first step. (happily ignoring for now the fact that PCA would also 'rotate' us into dimensions that don't correspond to any particular word).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I am reading that &lt;a href=&quot;http://www.useit.com/alertbox/zipf.html&quot; rel=&quot;nofollow&quot;&gt;&quot;Zipf distributions .. characterize the use of words in a natural language (like English) &quot;&lt;/a&gt; and as far as I know, PCA analysis makes various assumptions about the data being normally distributed. So, I'm wondering whether the fundamental assumptions of the PCA analysis will be sufficiently far 'off' from reality to be a ral problem. That is, does PCA rely on the data being 'close to' Gaussian Normal for it to work at all well?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If this is a problem as I suspect, are there any other recommendations? That is, some other approach worth investigating that is 'equivalent' to PCA in some way but more appropriate for Zipf or power law distributed data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that I am a programmer, not a statistician, so apologies if I messed up my terminology in the above. (Corrections of course welcomed!)&lt;/p&gt;&#xA;" OwnerUserId="2717" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2011-03-15T19:05:03.893" LastActivityDate="2011-03-15T19:05:03.893" Title="Is principal components analysis valid if the distribution(s) are Zipf like? What would be similar to PCA but suited to non gaussian data?" Tags="&lt;pca&gt;&lt;normal-distribution&gt;&lt;zipf&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="1" />
  <row Id="6121" PostTypeId="2" ParentId="6104" CreationDate="2011-01-10T08:10:03.080" Score="1" ViewCount="" Body="&lt;p&gt;We have $X_1\sim N(\mu_1,\sigma^2)$ and $X_2\sim N(\mu_2,\sigma^2)$, hence &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$EY_1=E(-X_1/\sqrt{2}+X_2/\sqrt{2})=-1/\sqrt{2}EX_1+1/\sqrt{2}EX_2=0$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align*}&#xA;EY_1^2&amp;amp;=E(-X_1/\sqrt{2}+X_2/\sqrt{2})^2\\&#xA;&amp;amp;=E(X_1/\sqrt{2})^2-2E(X_1X_2/2)+E(X_2/\sqrt{2})^2\\&#xA;&amp;amp;=1/2\sigma^2+1/2\sigma^2=\sigma^2&#xA;\end{align*}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hence $Y_1\sim N(0,\sigma^2)$ since it is the linear combination of normal variables. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Similarly we get $Y_2\sim N(0,\sigma^2)$  and $Y_3\sim N(0,\sigma^2)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$EY_1Y_2=1/\sqrt{6}E(X_1)^2-1/\sqrt{6}EX_2^2=0$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and similarly $EY_2Y_3=EY_1Y_3=0$, hence $Y_1$, $Y_2$ and $Y_3$ are independent, since for normal variables independece coincided with zero correlation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having established that we have &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$(Y_1^2+Y_2^2+Y_3^2)/\sigma^2=\left(\frac{Y_1}{\sigma^2}\right)^2+\left(\frac{Y_2}{\sigma^2}\right)^2+\left(\frac{Y_3}{\sigma^2}\right)^2=Z_1^2+Z_2^2+Z_3^2$$,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $Z_i=Y_i/\sigma$. Since $Y_i\sim N(0,\sigma^2)$, we have $Z_i\sim N(0,1)$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We have showed that our quantity of interest is a sum of squares of 3 independent standard normal variables, which by definition is $\chi^2$ with 3 degrees of freedom. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As I've said in the comments you do not need to calculate the densities. If you on the other hand want to do that, your formula is wrong. Here is why. Denote by $G(x)$ distribution of $Y_1^2$ and $F(x)$ the distribution of $Y_1$. Then we have&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$G(x)=P(Y_1^2&amp;lt;x)=P(-\sqrt{x}&amp;lt;Y_1&amp;lt;\sqrt{x})=F(\sqrt{x})-F(-\sqrt{x})$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now the density of $Y_1^2$ is $G&amp;#39;(x)$, so&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$G&amp;#39;(x)=\frac{1}{2\sqrt{x}}(F&amp;#39;(\sqrt{x})+F&amp;#39;(-\sqrt{x})$$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We have that&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$F&amp;#39;(x)=\frac{1}{\sigma\sqrt{2\pi}}e^{-\frac{x^2}{\sigma^2}},$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;so&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$G&amp;#39;(x)=\frac{1}{\sigma\sqrt{2\pi x}}e^{-\frac{x}{2}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If $\sigma^2=1$ we have a pdf of $\chi^2$ with one degree of freedom. (Note that for $Z_1$ instead of $Y_1$ the calculation is similar and $\sigma^2=1$ ) As @whuber pointed out, this is &lt;a href=&quot;http://en.wikipedia.org/wiki/Gamma_distribution&quot; rel=&quot;nofollow&quot;&gt;gamma&lt;/a&gt; distribution, and sums of independent gamma distributions is again gamma, the exact formula is provided in the wikipedia page.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-01-10T08:10:03.080" CommentCount="1" />
  <row Id="6122" PostTypeId="1" AcceptedAnswerId="6124" CreationDate="2011-01-10T09:54:02.767" Score="1" ViewCount="335" Body="&lt;p&gt;I have almost two questions. I need a single covariate logistic regression (LR) for each of my variables. Should I do it manually in SPSS, selecting each variable and do logistic regression? Is there a &quot;for each&quot; cycle to do it? I should switch to R language to have what I want. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the multivariables (multi covariates) LR, could I have missing values?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!!&lt;/p&gt;&#xA;" OwnerUserId="2719" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-01-10T10:37:10.423" LastActivityDate="2011-01-10T15:04:25.203" Title="Simple and multiple logistic regression" Tags="&lt;r&gt;&lt;logistic-regression&gt;&lt;spss&gt;" AnswerCount="3" />
  <row Id="6123" PostTypeId="1" AcceptedAnswerId="6140" CreationDate="2011-01-10T10:06:20.720" Score="2" ViewCount="72" Body="&lt;p&gt;I have a sample survey of a population whose distributions of certain characteristics are not identical to the distributions of the overall population.  For example, the age of my respondents may be biased downward, or the incomes in my sample may be too high compared to the population (or my theoretical population distribution, if the actual population distribution is unknown).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know that it is possible to calculate a weighting coefficient that can adjust the distribution of the sample to match the population distribution on one dimension, but is it possible to adjust for two dimensions (e.g. age and income)?  If so, couldn't there be a situation where there is no solution (no single coefficient that will adjust the sample distribution to the population distribution on both dimensions?)&lt;/p&gt;&#xA;" OwnerUserId="1195" LastActivityDate="2011-01-10T20:17:54.023" Title="Is it possible to weight survey observations to more than one distribution?" Tags="&lt;distributions&gt;&lt;survey&gt;" AnswerCount="2" />
  <row Id="6124" PostTypeId="2" ParentId="6122" CreationDate="2011-01-10T10:33:39.490" Score="4" ViewCount="" Body="&lt;p&gt;If I understand you correctly, you want to fit two successive simple logistic regression model. I don't know if there's a specific instruction in SPSS that allows to switch the covariate of interest or cycle through them, but I guess you can run the two models in succession. In R, if your data are organized in a matrix or data.frame, this is easily done as&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X &amp;lt;- replicate(2, rnorm(100))  # two random deviates&#xA;y &amp;lt;- rnorm(100)&#xA;apply(X, 2, function(x) lm(y ~ x))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;About your second question, models like this are generally estimated using listwise deletion: any individuals having one or more missing observations on the covariates are deleted before estimating model parameters. Again, in R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X[2,2] &amp;lt;- NA&#xA;summary(lm(y ~ X))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;shows that one observation has been deleted, yielding 96 DF (instead of 97).&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-01-10T10:33:39.490" CommentCount="1" />
  <row Id="6125" PostTypeId="2" ParentId="6119" CreationDate="2011-01-10T11:31:41.053" Score="4" ViewCount="" Body="&lt;p&gt;PCA is probably fine to use on your data, as it appears that it does not make any assumptions about the structure of the data. See link for a good introduction. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Principal_component_analysis#cite_note-4&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Principal_component_analysis#cite_note-4&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The note leads to a short PDF tutorial on PCA. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps. &lt;/p&gt;&#xA;" OwnerUserId="656" LastActivityDate="2011-01-10T11:31:41.053" CommentCount="2" />
  <row Id="6126" PostTypeId="2" ParentId="6123" CreationDate="2011-01-10T11:36:02.833" Score="1" ViewCount="" Body="&lt;p&gt;Just asking for a clarification : why would you want to do this? As I understand it, to do this properly, you need to have some concrete benchmark via which you are quantifying the bias.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The most important thing is how appropriate is the benchmark chosen? (Are we standardizing to the wrong base?)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even if we have the correct benchmark, is it true that the the benchmark is &lt;em&gt;exactly&lt;/em&gt; true? (The issue of matching each variable to the benchmark)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you ask me, just proceed with the sample, and quantify separately for each variable the apparent bias from your sample to the &quot;benchmark&quot; population. I'd assume that this would be a more informative and accurate picture.&lt;/p&gt;&#xA;" OwnerUserId="2472" LastActivityDate="2011-01-10T11:36:02.833" CommentCount="1" />
  <row Id="6127" PostTypeId="1" AcceptedAnswerId="6128" CreationDate="2011-01-10T12:10:29.000" Score="9" ViewCount="599" Body="&lt;p&gt;I have data from an experiment which I analyzed by t-tests. The dependent variable is interval scaled and the data is either non-paired (i.e., 2 groups) or paired (i.e., within-subjects). &#xA;E.g. (within subjects):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x1 &amp;lt;- c(99, 99.5, 65, 100, 99, 99.5, 99, 99.5, 99.5, 57, 100, 99.5, &#xA;99.5, 99, 99, 99.5, 89.5, 99.5, 100, 99.5)&#xA;y1 &amp;lt;- c(99, 99.5, 99.5, 0, 50, 100, 99.5, 99.5, 0, 99.5, 99.5, 90, &#xA;80, 0, 99, 0, 74.5, 0, 100, 49.5)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, data is not normal so one reviewer asked us to use other statistics bu tthe t-test. However, as one can easily see, the data is not only not normally distributed, but the distributions are not equal between conditions:&#xA;&lt;img src=&quot;http://i.imgur.com/5y0kx.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, the usual nonparametric suspects, Mann-Whitney-U-Test (non-paired) and Wilcoxon Test (paired), can not be used as they require equal distributions between conditions. Hence, I decided that some resampling or permutation test would be the best.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Now, I am looking for a R implementation of a permutation t-test equivalent, or any other advise on what to do with the data.&lt;/strong&gt;&lt;br&gt;&#xA;I figured that there are some R-packages that could do this for me, e.g., coin, perm, exactRankTest, ..., but don't know which oine to pick. So, if somebody with some experience in using these tests could give me some kick-start, this would be ubercool.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE:&lt;/strong&gt; It would be ideal if you could provide an example of how to report the results from this test.&lt;/p&gt;&#xA;" OwnerUserId="442" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-01-10T13:50:15.787" LastActivityDate="2011-01-10T18:49:31.047" Title="Which permutation test implementation in R to use instead of t-tests (paired and non-paired)?" Tags="&lt;r&gt;&lt;nonparametric&gt;&lt;permutation&gt;" AnswerCount="3" CommentCount="1" FavoriteCount="2" />
  <row Id="6128" PostTypeId="2" ParentId="6127" CreationDate="2011-01-10T13:22:10.797" Score="8" ViewCount="" Body="&lt;p&gt;It shouldn't matter that much since the test statistic will always be the difference in means (or something equivalent). Small differences can come from the implementation of Monte-Carlo methods. Trying the three packages with your data with a one-sided test for two independent variables:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; DV &amp;lt;- c(x1, y1)&#xA;&amp;gt; IV &amp;lt;- factor(rep(c(&quot;A&quot;, &quot;B&quot;), c(length(x1), length(y1))))&#xA;&amp;gt; library(coin)                    # for oneway_test(), pvalue()&#xA;&amp;gt; pvalue(oneway_test(DV ~ IV, alternative=&quot;greater&quot;,&#xA;+                    distribution=approximate(B=9999))&#xA;[1] 0.00330033&#xA;&#xA;&amp;gt; library(perm)                    # for permTS()&#xA;&amp;gt; permTS(DV ~ IV, alternative=&quot;greater&quot;, method=&quot;exact.mc&quot;,&#xA;+        control=permControl(nmc=10^4-1))$p.value&#xA;[1] 0.003&#xA;&#xA;&amp;gt; library(exactRankTests)          # for perm.test()&#xA;&amp;gt; perm.test(DV ~ IV, paired=FALSE, alternative=&quot;greater&quot;, exact=TRUE)$p.value&#xA;[1] 0.003171822&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To check the exact p-value with a manual calculation of all permutations, I'll restrict the data to the first 9 values.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; x1 &amp;lt;- x1[1:9]&#xA;&amp;gt; y1 &amp;lt;- y1[1:9]&#xA;&amp;gt; DV &amp;lt;- c(x1, y1)&#xA;&amp;gt; IV &amp;lt;- factor(rep(c(&quot;A&quot;, &quot;B&quot;), c(length(x1), length(y1))))&#xA;&amp;gt; pvalue(oneway_test(DV ~ IV, alternative=&quot;greater&quot;, distribution=&quot;exact&quot;))&#xA;[1] 0.0945907&#xA;&#xA;&amp;gt; permTS(DV ~ IV, alternative=&quot;greater&quot;, exact=TRUE)$p.value&#xA;[1] 0.0945907&#xA;&#xA;# perm.test() gives different result due to rounding of input values&#xA;&amp;gt; perm.test(DV ~ IV, paired=FALSE, alternative=&quot;greater&quot;, exact=TRUE)$p.value&#xA;[1] 0.1029412&#xA;&#xA;# manual exact permutation test&#xA;&amp;gt; idx  &amp;lt;- seq(along=DV)                 # indices to permute&#xA;&amp;gt; idxA &amp;lt;- combn(idx, length(x1))        # all possibilities for different groups&#xA;&#xA;# function to calculate difference in group means given index vector for group A&#xA;&amp;gt; getDiffM &amp;lt;- function(x) { mean(DV[x]) - mean(DV[!(idx %in% x)]) }&#xA;&amp;gt; resDM    &amp;lt;- apply(idxA, 2, getDiffM)  # difference in means for all permutations&#xA;&amp;gt; diffM    &amp;lt;- mean(x1) - mean(y1)       # empirical differencen in group means&#xA;&#xA;# p-value: proportion of group means at least as extreme as observed one&#xA;&amp;gt; (pVal &amp;lt;- sum(resDM &amp;gt;= diffM) / length(resDM))&#xA;[1] 0.0945907&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;coin&lt;/code&gt; and &lt;code&gt;exactRankTests&lt;/code&gt; are both from the same author, but &lt;code&gt;coin&lt;/code&gt; seems to be more general and extensive - also in terms of documentation. &lt;code&gt;exactRankTests&lt;/code&gt; is not actively developed anymore. I'd therefore choose &lt;code&gt;coin&lt;/code&gt; (also because of informative functions like &lt;code&gt;support()&lt;/code&gt;), unless you don't like to deal with S4 objects.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: for two dependent variables, the syntax is&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; id &amp;lt;- factor(rep(1:length(x1), 2))    # factor for participant&#xA;&amp;gt; pvalue(oneway_test(DV ~ IV | id, alternative=&quot;greater&quot;,&#xA;+                    distribution=approximate(B=9999)))&#xA;[1] 0.00810081&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1909" LastEditorUserId="1909" LastEditorDisplayName="" LastEditDate="2011-01-10T18:49:31.047" LastActivityDate="2011-01-10T18:49:31.047" CommentCount="15" />
  <row Id="6129" PostTypeId="2" ParentId="6127" CreationDate="2011-01-10T13:49:44.313" Score="1" ViewCount="" Body="&lt;p&gt;Are these scores proportions? If so, you certainly shouldn't be using a gaussian parametric test, and while you could go ahead with a non-parametric approach like a permutation test or bootstrap of the means, I'd suggest that you'll get more statistical power by employing a suitable non-gaussian parametric approach. Specifically, any time you can compute a proportion measure within a unit of interest (ex. participant in an experiment), you can and probably should use a mixed effects model that specifies observations with binomially distributed error. See &lt;a href=&quot;http://linkinghub.elsevier.com/retrieve/pii/S0749596X07001283&quot; rel=&quot;nofollow&quot;&gt;Dixon 2004&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="364" LastActivityDate="2011-01-10T13:49:44.313" CommentCount="3" />
  <row Id="6130" PostTypeId="2" ParentId="6122" CreationDate="2011-01-10T14:15:34.053" Score="1" ViewCount="" Body="&lt;p&gt;Hi Simone,&#xA;SPSS has some scripting facilities (syntax, sax basic, pyhton, ...). I myself have so far only used syntax. Maybe the link below can help you to construct a loop that does the job, it points to the excellent website of the UCLA. The problem solved there is only slightly different from yours so chances are high that you can modify it according to your needs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.ats.ucla.edu/stat/spss/faq/looping_parallel_lists.html&quot; rel=&quot;nofollow&quot;&gt;http://www.ats.ucla.edu/stat/spss/faq/looping_parallel_lists.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;psj&lt;/p&gt;&#xA;" OwnerUserId="1573" LastActivityDate="2011-01-10T14:15:34.053" CommentCount="1" />
  <row Id="6131" PostTypeId="2" ParentId="6122" CreationDate="2011-01-10T14:45:42.287" Score="4" ViewCount="" Body="&lt;p&gt;Here is some SPSS code to loop through each of your models.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;*creating a simulated dataset.&#xA;input program.&#xA;loop #i = 1 to 100.&#xA;compute yvar = RV.BERNOULLI(.5).&#xA;compute xvar1 = RV.NORMAL(100,10).&#xA;compute xvar2 = RV.NORMAL(100,10).&#xA;compute xvar3 = RV.NORMAL(100,10).&#xA;compute listwise = RV.BERNOULLI(.1).&#xA;end case.&#xA;end loop.&#xA;end file.&#xA;end input program.&#xA;execute.&#xA;&#xA;*using these variables you can run the regressions one equation at a time.&#xA;LOGISTIC REGRESSION VARIABLES YVAR&#xA;  /SELECT=LISTWISE EQ 0&#xA;  /METHOD=ENTER XVAR1.&#xA;&#xA;*or you can create a macro and loop through your independent variable list.&#xA;DEFINE !logit_loop (dep = !TOKENS(1)&#xA;                             /ind = !CMDEND )&#xA;!DO !i !IN !ind&#xA;LOGISTIC REGRESSION VARIABLES !dep&#xA;  /SELECT=listwise = 0 &#xA;  /METHOD=ENTER !i.&#xA;!DOEND.&#xA;!ENDDEFINE.&#xA;!logit_loop dep = yvar ind = xvar1 xvar2 xvar3.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Note the loop does a selection on a variable named listwise = 0 , otherwise missing values for any variables would just be dropped (and potentially produce equations with different sets). While looping through your list is pretty simple in SPSS, where I think R is more convienant is the fact that the elements of your different objects are readily available in R (and so it is easier to access them and manipulate them). I'll update with an example later, but basically all the SPSS code does is produce alot of output which you have to read to find the values your interested in. In R you produce objects that have attributes you can extract. You can do the same thing in SPSS, but it is IMO more difficult and requires going into the output XML and parsing the info you want (which I don't know how to do). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So long story short if you are alright with just viewing the output this is easily accomplished in SPSS (or PASW now). If you want to produce a program to summarize specific elements of those models, I think it is easier to learn a solution in R.&lt;/p&gt;&#xA;" OwnerUserId="1036" LastEditorUserId="1036" LastEditorDisplayName="" LastEditDate="2011-01-10T15:04:25.203" LastActivityDate="2011-01-10T15:04:25.203" CommentCount="7" />
  <row Id="6132" PostTypeId="2" ParentId="5045" CreationDate="2011-01-10T16:46:43.897" Score="4" ViewCount="" Body="&lt;p&gt;Somewhere there's a saying &quot;Model what you're interested in.&quot;  I think you need to determine at what level you're most interested in the parameters.  I'd also like to see you define what you mean by &quot;better result.&quot;  If it were me I'd probably want to obtain best estimates at the more specific level so that I could isolate the contributions of different, specific indices as much as possible.  But whatever you do, watch for collinearity.  I'd take a close look at a correlation matrix and/or at a printout of zero-order, partial, and part correlations, and be prepared to run several iterations of regression before drawing any conclusions about the relative contributions of different variables.  I'd also want to obtain partial regression plots to see more about the nature of each X-Y relationship than a coefficient alone can give.&lt;/p&gt;&#xA;" OwnerUserId="2669" LastActivityDate="2011-01-10T16:46:43.897" CommentCount="2" />
  <row Id="6133" PostTypeId="1" AcceptedAnswerId="6165" CreationDate="2011-01-10T16:58:37.693" Score="2" ViewCount="116" Body="&lt;p&gt;I think people here could guide me in solving a problem related to anomaly detection in Computer Science. The term &lt;em&gt;anomaly&lt;/em&gt; here refers to some undesired event occuring in the system like a virus infection.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I could get to know about it from more than one source. For example, after having extracted a value from two different data structures, if there is a difference it is certain that virus infection is there.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to remove the false positive cases, information is gathered from different data structures or mechanisms. Hence, certain information are less trusted and certain information are more trusted.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am looking for a mathematical method, that could easily handle this type of situations. Whether Fuzzy/Genetic Algorithm/Neural Net fits here? Found in some places they are using normality-based approach (using z-score). Please help.&lt;/p&gt;&#xA;" OwnerUserId="2721" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-01-10T18:57:52.517" LastActivityDate="2011-01-11T14:50:31.043" Title="Which method to use for anomaly detection?" Tags="&lt;data-mining&gt;&lt;computational-statistics&gt;&lt;computing&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="6134" PostTypeId="2" ParentId="6127" CreationDate="2011-01-10T17:06:35.393" Score="8" ViewCount="" Body="&lt;p&gt;A few comments are, I believe, in order.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) I would encourage you to try multiple visual displays of your data, because they can capture things that are lost by (graphs like) histograms, and I also strongly recommend that you plot on side-by-side axes.  In this case, I do not believe the histograms do a very good job of communicating the salient features of your data.  For example, take a look at side-by-side boxplots:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;boxplot(x1, y1, names = c(&quot;x1&quot;, &quot;y1&quot;))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/rztQJ.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or even side-by-side stripcharts:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;stripchart(c(x1,y1) ~ rep(1:2, each = 20), method = &quot;jitter&quot;, group.names = c(&quot;x1&quot;,&quot;y1&quot;), xlab = &quot;&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/NTnmm.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Look at the centers, spreads, and shapes of these!  About three-quarters of the $x1$ data fall well above the median of the $y1$ data.  The spread of $x1$ is tiny, while the spread of $y1$ is huge. Both $x1$ and $y1$ are highly left-skewed, but in different ways.  For example, $y1$ has five (!) repeated values of zero.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) You didn't explain in much detail where your data come from, nor how they were measured, but this information is very important when it comes time to select a statistical procedure.  Are your two samples above independent?   Are there any reasons to believe that the marginal distributions of the two samples should be the same (except for a difference in location, for example)?  What were the considerations &lt;em&gt;prior&lt;/em&gt; to the study that led you to look for evidence of a difference between the two groups? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) The t-test is not appropriate for these data because the marginal distributions are markedly non-normal, with extreme values in both samples.  If you like, you could appeal to the CLT (due to your moderately-sized sample) to use a $z$-test (which would be similar to a z-test for large samples), but given the skewness (in both variables) of your data I would not judge such an appeal very convincing.  Sure, you can use it anyway to calculate a $p$-value, but what does that do for you?  If the assumptions aren't satisfied then a $p$-value is just a statistic; it doesn't tell what you (presumably) want to know: whether there is evidence that the two samples come from different distributions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4) A permutation test would also be inappropriate for these data.  The single and often-overlooked assumption for permutation tests is that the two samples are &lt;em&gt;exchangeable&lt;/em&gt; under the null hypothesis.  That would mean that they have identical marginal distributions (under the null).  But you are in trouble, because the graphs suggest that the distributions differ both in location and scale (and shape, too).  So, you can't (validly) test for a difference in location because the scales are different, and you can't (validly) test for a difference in scale because the locations are different. Oops.  Again, you can do the test anyway and get a $p$-value, but so what?  What have you really accomplished?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;5) In my opinion, these data are a perfect (?) example that a well chosen picture is worth 1000 hypothesis tests.  We don't need statistics to tell the difference between a pencil and a barn.  The appropriate statement in my view for these data would be &quot;These data exhibit marked differences with respect to location, scale, and shape.&quot;  You could follow up with (robust) descriptive statistics for each of those to quantify the differences, and explain what the differences mean in the context of your original study.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;6) Your reviewer is probably (and sadly) going to insist on some sort of $p$-value as a precondition to publication.  Sigh!  If it were me, given the differences with respect to &lt;em&gt;everything&lt;/em&gt; I would probably use a nonparametric Kolmogorov-Smirnov test to spit out a $p$-value that demonstrates that the distributions are different, and then proceed with descriptive statistics as above.  You would need to add some noise to the two samples to get rid of ties.  (And of course, this all assumes that your samples are independent which you didn't state explicitly.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This answer is a lot longer than I originally intended it to be.  Sorry about that.&lt;/p&gt;&#xA;" OwnerUserId="1108" LastActivityDate="2011-01-10T17:06:35.393" CommentCount="5" />
  <row Id="6135" PostTypeId="2" ParentId="3779" CreationDate="2011-01-10T17:41:58.967" Score="7" ViewCount="" Body="&lt;p&gt;This is a (long!) comment on the nice work @vqv has posted in this thread.  It aims to obtain a definitive answer.  He has done the hard work of simplifying the dictionary.  All that remains is to exploit it to the fullest.  His results suggest that &lt;strong&gt;a brute-force solution is feasible&lt;/strong&gt;.  After all, including a wildcard, there are at most $27^7 = 10,460,353,203$ words one can make with 7 characters, and it looks like less than 1/10000 of them--say, around a million--will fail to include some valid word.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first step is to augment the minimal dictionary with a wildcard character, &quot;?&quot;.  22 of the letters appear in two-letter words (all but c, q, v, z).  Adjoin a wildcard to those 22 letters and add these to the dictionary: {a?, b?, d?, ..., y?} are now in.  Similarly we can inspect the minimal three-letter words, causing some additional words to appear in the dictionary.  Finally, we add &quot;??&quot; to the dictionary.  After removing repetitions that result, it contains 342 minimal words.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An elegant way to proceed--one that uses a very small amount of encoding indeed--is to &lt;strong&gt;view this problem as an algebraic one&lt;/strong&gt;.  A word, considered as an unordered set of letters, is just a monomial.  For example, &quot;spats&quot; is the monomial $a p s^2 t$.  The dictionary therefore is a collection of monomials.  It looks like&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\{a^2, a b, a d, ..., o z \psi, w x \psi, \psi^2\}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(where, to avoid confusion, I have written $\psi$ for the wildcard character).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;A rack contains a valid word if and only if that word divides the rack.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A more abstract, but extremely powerful, way to say this is that the dictionary generates an ideal $I$ in the polynomial ring $R = \mathbb{Z}[a, b, \ldots, z, \psi]$ and that the racks with valid words become zero in the quotient ring $R/I$, whereas racks without valid words remain nonzero in the quotient.  If we form the sum of all racks in $R$ and compute it in this quotient ring, then &lt;em&gt;the number of racks without words equals the number of distinct monomials in the quotient.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furthermore, the sum of all racks in $R$ is straightforward to express.  Let $\alpha = a + b + \cdots + z + \psi$ be the sum of all letters in the alphabet.  $\alpha^7$ contains one monomial for each rack.  (As an added bonus, its coefficients count the number of ways each rack can be formed, allowing us to compute its probability if we like.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;As a simple example&lt;/strong&gt; (to see how this works), suppose (a) we don't use wildcards and (b) all letters from &quot;a&quot; through &quot;x&quot; are considered words.  Then the only possible racks from which words cannot be formed must consist entirely of y's and z's.  We compute $\alpha=(a+b+c+\cdots+x+y+z)^7$ modulo the ideal generated by $\{a,b,c, \ldots, x\}$ one step at a time, thus:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\eqalign{&#xD;&#xA;\alpha^0 &amp;amp;= 1 \cr&#xD;&#xA;\alpha^1 &amp;amp;= a+b+c+\cdots+x+y+z \equiv y+z \mod I \cr&#xD;&#xA;\alpha^2 &amp;amp;\equiv (y+z)(a+b+\cdots+y+z) \equiv (y+z)^2 \mod I \cr&#xD;&#xA;\cdots &amp;amp;\cr&#xD;&#xA;\alpha^7 &amp;amp;\equiv (y+z)^6(a+b+\cdots+y+z) \equiv (y+z)^7 \mod I \text{.}&#xD;&#xA;}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can read off the chance of getting a non-word rack from the final answer, $y^7 + 7 y^6 z + 21 y^5 z^2 + 35 y^4 z^3 + 35 y^3 z^4 + 21 y^2 z^5 + &#xD;&#xA; 7 y z^6 + z^7$: each coefficient counts the ways in which the corresponding rack can be drawn.  For example, there are 21 (out of 26^7 possible) ways to draw 2 y's and 5 z's because the coefficient of $y^2 z^5$ equals 21.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From elementary calculations, it is obvious this is the correct answer.  The whole point is that this procedure works regardless of the contents of the dictionary.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Notice how reducing the power modulo the ideal at each stage reduces the computation: that's the shortcut revealed by this approach.  (End of example.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Polynomial algebra systems implement these calculations&lt;/strong&gt;.  For instance, here is &lt;em&gt;Mathematica&lt;/em&gt; code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;alphabet =  a + b + c + d + e + f + g + h + i + j + k + l + m + n + o + &#xA;            p + q + r + s + t + u + v + w + x + y + z + \[Psi];&#xA;dictionary = {a^2, a b, a d, a e, ..., w z \[Psi], \[Psi]^2};&#xA;next[pp_] := PolynomialMod[pp alphabet, dictionary];&#xA;nonwords = Nest[next, 1, 7];&#xA;Length[nonwords]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(The dictionary can be constructed in a straightforward manner from @vqv's min.dict; I put a line here showing that it is short enough to be specified directly if you like.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The output--which takes ten minutes of computation--is 577958.  (&lt;strong&gt;NB&lt;/strong&gt; In an earlier version of this message I had made a tiny mistake in preparing the dictionary and obtained 577940.  I have edited the text to reflect what I hope are now the correct results!)  A little less than the million or so I expected, but of the same order of magnitude.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To compute the &lt;em&gt;chance&lt;/em&gt; of obtaining such a rack, we need to account for the number of ways in which the rack can be drawn.  As we saw in the example, this equals its coefficient in $\alpha^7$.  The chance of drawing &lt;em&gt;some&lt;/em&gt; such rack is the sum of all these coefficients, easily found by setting all the letters equal to 1:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;nonwords /. (# -&amp;gt; 1) &amp;amp; /@ (List @@ alphabet)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The answer equals 1066056120, giving a chance of 10.1914% of drawing a rack from which no valid word can be formed (if all letters are equally likely).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When the probabilities of the letters vary, just replace each letter with its chance of being drawn:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;tiles = {9, 2, 2, 4, 12, 2, 3, 2, 9, 1, 1, 4, 2, 6, 8, 2, 1, 6, 4, 6, &#xA;         4, 2, 2, 1, 2, 1, 2};&#xA;chances = tiles / (Plus @@ tiles);&#xA;nonwords /. (Transpose[{List @@ alphabet, chances}] /. {a_, b_} -&amp;gt; a -&amp;gt; b)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The output is &lt;strong&gt;1.079877553303%, the &lt;em&gt;exact&lt;/em&gt; answer&lt;/strong&gt; (albeit using an approximate model, drawing &lt;em&gt;with&lt;/em&gt; replacement).  Looking back, it took four lines to enter the data (alphabet, dictionary, and alphabet frequencies) and only three lines to do the work: describe how to take the next power of $\alpha$ modulo $I$, take the 7th power recursively, and substitute the probabilities for the letters.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-01-10T23:05:05.543" LastActivityDate="2011-01-10T23:05:05.543" CommentCount="10" />
  <row Id="6136" PostTypeId="5" CreationDate="2011-01-10T17:44:45.473" Score="0" ViewCount="0" Body="&lt;p&gt;SPSS (Statistical Package for the Social Sciences) is a proprietary cross-platform general-purpose statistical software package. &lt;a href=&quot;http://www.spss.com/&quot; rel=&quot;nofollow&quot;&gt;SPSS's homepage&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Versions 17 and 18 of SPSS were renamed with the acronym PASW (for Predictive Analytics SoftWare), but with version 19 it has been renamed to SPSS (see &lt;a href=&quot;http://en.wikipedia.org/wiki/SPSS&quot; rel=&quot;nofollow&quot;&gt;Wikipedia&lt;/a&gt; for a history of the name changes). One unique aspect of SPSS software compared to other popular propriety software packages (such as Stata or SAS) is the built in functionality to call Python or R commands within syntax. Otherwise it is largely comparable to other general proprietary and freeware packages (such as R), although it differs in some advanced statistical capabilities and aspects of data manipulation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suggested readings on using SPSS and learning the command syntax are two online PDF's.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.spss.com/sites/dm-book/&quot; rel=&quot;nofollow&quot;&gt;Programming and Data Management for&#xA;IBM Statistics&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://szoc.tarstud.hu/upload/szoc_szakirany_programmingwithspsssyntaxandmacros.pdf&quot; rel=&quot;nofollow&quot;&gt;Programming with&#xA;SPSS Syntax and Macros&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Other useful print versions are&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://books.google.com/books?id=mf8pEAAzN4gC&amp;amp;dq=An+Intermediate+Guide+to+SPSS+Programming%3a+Using+Syntax+for+Data+Management&amp;amp;source=gbs_navlinks_s&quot; rel=&quot;nofollow&quot;&gt;An Intermediate Guide to SPSS&#xA;Programming: Using Syntax for Data&#xA;Management&lt;/a&gt; by Sarah Boslaugh (For&#xA;Data Management)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://books.google.com/books?id=a6FLF1YOqtsC&amp;amp;dq=discovering+statistics+using+SPSS&amp;amp;source=gbs_navlinks_s&quot; rel=&quot;nofollow&quot;&gt;Discovering statistics using SPSS: and sex and drugs and rock 'n' roll&lt;/a&gt;. by Andy Field (For Statistical Analysis)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Other forums entirely devoted to the software are (and suggested material to search when encountering a problem with SPSS are):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://spssx-discussion.1045642.n5.nabble.com/&quot; rel=&quot;nofollow&quot;&gt;University of Georgia SPSSX discussion list&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://groups.google.com/group/comp.soft-sys.stat.spss/topics&quot; rel=&quot;nofollow&quot;&gt;SPSS google group forum&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.ibm.com/developerworks/spssdevcentral&quot; rel=&quot;nofollow&quot;&gt;SPSS developer central&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Other suggested webpages are &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;UCLA's Academic Technology Services &lt;a href=&quot;http://www.ats.ucla.edu/stat/spss/&quot; rel=&quot;nofollow&quot;&gt;SPSS Resources webpage&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://spsstools.net/&quot; rel=&quot;nofollow&quot;&gt;Raynald's SPSS tools&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://sites.google.com/a/lakeheadu.ca/bweaver/Home/statistics/spss/my-spss-page&quot; rel=&quot;nofollow&quot;&gt;Bruce Weaver's SPSS page&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://jeromyanglim.blogspot.com/&quot; rel=&quot;nofollow&quot;&gt;Jeromy Anglim's blog&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.gnu.org/software/pspp/&quot; rel=&quot;nofollow&quot;&gt;PSPP&lt;/a&gt; is a free-ware, open source alternative largely mimicking the look and functionality of SPSS.&lt;/p&gt;&#xA;" OwnerUserId="-1" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-01-19T11:54:02.143" LastActivityDate="2011-01-19T11:54:02.143" />
  <row Id="6137" PostTypeId="4" CreationDate="2011-01-10T17:44:45.473" Score="0" ViewCount="0" Body="SPSS (Statistical Package for the Social Sciences) is a proprietary cross-platform general-purpose statistical software package. " OwnerUserId="-1" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-01-10T17:44:45.473" LastActivityDate="2011-01-10T17:44:45.473" />
  <row Id="6138" PostTypeId="2" ParentId="6074" CreationDate="2011-01-10T17:46:20.557" Score="5" ViewCount="" Body="&lt;p&gt;I don’t think it is really a classification problem.  20 questions is often characterized as a &lt;em&gt;compression&lt;/em&gt; problem.  This actually matches better with the last part of your question where you talk about entropy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See Chapter 5.7 (&lt;a href=&quot;http://books.google.com/books?id=EuhBluW31hsC&amp;amp;lpg=PP1&amp;amp;dq=cover%20and%20thomas%20information%20theory&amp;amp;pg=PA120&quot; rel=&quot;nofollow&quot;&gt;Google books&lt;/a&gt;) of &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cover, T.M. and Joy, A.T. (2006) Elements of Information Theory&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and also &lt;a href=&quot;http://en.wikipedia.org/wiki/Huffman_coding&quot; rel=&quot;nofollow&quot;&gt;Huffman coding&lt;/a&gt;.  This paper I found on arXiv may be of interest as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Gill, J.T. and Wu, W. (2010) &quot;Twenty Questions Games Always End With Yes”&#xA;&lt;a href=&quot;http://arxiv.org/abs/1002.4907&quot; rel=&quot;nofollow&quot;&gt;http://arxiv.org/abs/1002.4907&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For simplicity assume yes/no questions (whereas akinator.com allows allows maybe, don’t know). &#xA;Assume that every possible subject (what akinator.com knows) can be uniquely identified by a sequence of yes/no questions and answers — essentially a binary vector.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The questions that are (and their answers) asked define a recursive partitioning of the space of subjects.  This partitioning also corresponds to a tree structure.  The interior vertices of the tree correspond to questions, and the leaves correspond to subjects.  The depth of the leaves is exactly the number of questions required to uniquely identify the subject.  You can (trivially) identify every known subject by asking every possible question.  That’s not interesting because there are potentially hundreds of questions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The connection with Huffman coding is that it gives an optimal way (under a certain probabilistic model) to construct the tree so that the average depth is (nearly) minimal.  In other words, it tells you how to arrange the sequence of questions (construct a tree) so that the number of questions you need to ask is on average small.  It uses a greedy approach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is, of course, more to akinator.com than this, but the basic idea is that you can think of the problem in terms of a tree and trying to minimize the average depth of its leaves.&lt;/p&gt;&#xA;" OwnerUserId="1670" LastEditorUserId="1670" LastEditorDisplayName="" LastEditDate="2011-01-14T20:29:06.633" LastActivityDate="2011-01-14T20:29:06.633" CommentCount="8" />
  <row Id="6139" PostTypeId="1" CreationDate="2011-01-10T09:54:12.360" Score="5" ViewCount="238" Body="&lt;p&gt;Hi, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm studying Biomedical Computer science and I have to research a paper about genotype-phenotype association.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this paper the authors use a correlation analysis by first calculating the Pearson correlation and then calculating the hypergeometric distribution to filter out insignificant associations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.biomedcentral.com/1471-2164/7/257&quot; rel=&quot;nofollow&quot;&gt;http://www.biomedcentral.com/1471-2164/7/257&lt;/a&gt;&lt;br&gt;&#xA;Under Methods/Associating genes to phenotypes&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;While the correlation measures the strength of association between an organism's genomic  content and  its phenotype,  we  also  applied  another  method,  exploiting  the hypergeometric  distribution  function,  to  determine  the significance of these&lt;br&gt;&#xA;  associations [...] where a result smaller or equal to 20% response is considered negative. So for a given gene found in M  species,  the  hypergeometric  function  provides  the probability by random chance that the gene is found in m species which contain  the COG and are also positive  in the laboratory test.&lt;br&gt;&#xA;  The following criteria were applied to the correlated data set. The intersection between a specific COG and a phenotype had to contain at least 3 organisms, and for any intersection, 30% of the microbes had to share the COG. The scores were adjusted using the standard Bonferroni error correction for multiple testing.&lt;br&gt;&#xA;  Since the Bonferroni correction  is  one  of  the most  conservative,  it  is  likely  that some biologically relevant associations were unnecessarily discarded. In this case $\alpha$ was set as less than equal to 0.01, therefore, any hypergeometric distribution score less than  or  equal  to  0.0001 was  deemed  significant. Using these criteria, we set a 0.8 and a 0.9 correlation threshold to assess the significance of the COG-phenotype associations.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;My question is: Is this a valid scientific correlation analysis or not? Are there any reservations?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, can you give me an idea for a good statistics book for science?&lt;/p&gt;&#xA;" LastActivityDate="2011-01-10T19:50:38.643" Title="Is a correlation analysis with Pearson's correlation and Bonferroni Method a valid approach to find correlations between two sets of data" Tags="&lt;correlation&gt;" CommentCount="2" />
  <row Id="6140" PostTypeId="2" ParentId="6123" CreationDate="2011-01-10T20:17:54.023" Score="1" ViewCount="" Body="&lt;p&gt;The answer to the question, bracketing for the moment whether it is a good idea, is yes, there are ways to do this.  If you have information on the joint distribution of the population over all the variables of interest (e.g. from a census), you can &quot;poststratify&quot; to that joint distribution, in the case of all discrete variables, or use &quot;calibration&quot; for more general combinations of discrete and continuous variables.  If you have only information on the marginal distribution and you are willing to believe that an assumption of independence over the dimensions won't introduce too much bias, then you can &quot;rake&quot; to the margins.  You can combine post-stratification, calibration, and raking to attend to arbitrary combinations of population information.  Use a software package to do it though (e.g. the &quot;survey&quot; package in R or the svy suite of commands in Stata), because the computations can be tricky.  For accessible introductions to these concepts with worked examples in R and Stata, respectively, see the following two references:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lee and Forthofer. Analyzing Complex Survey Data. Sage.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lumley. Complex Surveys: A Guide to Analysis Using R. Wiley.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, @Mohit asked why you would want to do this.  Indeed, you should think about it.  Intuitively, it would seem that you are reducing the bias in estimating population parameters from your sample.  But there are cases in which such weighting can exacerbate biases; search for &quot;non-ignorable&quot; missingness or sample selection problems.&lt;/p&gt;&#xA;" OwnerUserId="96" LastActivityDate="2011-01-10T20:17:54.023" CommentCount="2" />
  <row Id="6141" PostTypeId="1" CreationDate="2011-01-10T21:29:47.717" Score="3" ViewCount="274" Body="&lt;p&gt;Hello&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am now writing my bachelors thesis and I have come across some difficulties. I am about to do some panel regressions with time and entity fixed effects and I would therefore like to use the plm package. But when I do add fixed effects and want to have heteroscedasticity robust standard errors they seem to be incorrect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone know why the HC standard errors differ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is my code&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Load data&#xA;load(file=&quot;panel&quot;)&#xA;attach(panel)&#xA;&#xA;# Load packages&#xA;library(lmtest)&#xA;library(plm)&#xA;&#xA;&#xA;# Create two models. The lm.model is a linear model and as the&#xA;# LAND variable is a factor variable representing countries&#xA;# (Land = Country in Swedish) this model will have entity fixed&#xA;# effects. In the plm.model the plm package is used and&#xA;# individual effects and within model is turned on (which is&#xA;# the same as entity fixed effects)&#xA;lm.model&amp;lt;-lm(NETTOSPARANDE ~ EURO + LAND, data=panel)&#xA;plm.model&amp;lt;-plm(NETTOSPARANDE ~ EURO, index=c(&quot;LAND&quot;,&quot;ÅR&quot;), effect=&quot;individual&quot;, model=&quot;within&quot;, data=panel)&#xA;&#xA;# When looking at the coefficents without heteroscadisity robust&#xA;# standard errors they are identical. They do also have the same&#xA;# value in stata.&#xA;coeftest(lm.model)[1:2,]&#xA;coeftest(plm.model)&#xA;&#xA;# But when looking at the coefficents using heteroscadisity&#xA;# robust standard errors the lm.model and the plm.model produces&#xA;# different standard errors.&#xA;coeftest(lm.model, vcov.=vcovHC(lm.model, method=&quot;white2&quot;, type=&quot;HC1&quot;))[1:2,]&#xA;coeftest(plm.model, vcov.=vcovHC(plm.model, method=&quot;white2&quot;, type=&quot;HC1&quot;))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you want to test the data it can be found here (the panel file)&#xA;  [1]: &lt;a href=&quot;https://sourceforge.net/projects/emumoralhazard/files/&quot; rel=&quot;nofollow&quot;&gt;https://sourceforge.net/projects/emumoralhazard/files/&lt;/a&gt; &lt;em&gt;R-data&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And here is my output&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1&amp;gt; # Load data&#xA;1&amp;gt; load(file=&quot;panel&quot;)&#xA;&#xA;1&amp;gt; attach(panel)&#xA;&#xA;1&amp;gt; # Load packages&#xA;1&amp;gt; library(lmtest)&#xA;Loading required package: zoo&#xA;&#xA;1&amp;gt; library(plm)&#xA;Loading required package: kinship&#xA;Loading required package: survival&#xA;Loading required package: splines&#xA;Loading required package: nlme&#xA;Loading required package: lattice&#xA;[1] &quot;kinship is loaded&quot;&#xA;Loading required package: Formula&#xA;Loading required package: MASS&#xA;Loading required package: sandwich&#xA;&#xA;1&amp;gt; # Create two models. The lm.model is a linear model and as the&#xA;1&amp;gt; # LAND variabel is a factor variable representing countries&#xA;1&amp;gt; # (Land = Country in swedish) this model will have entity fixed&#xA;1&amp;gt; # effects. In the plm.model the plm package is used and&#xA;1&amp;gt; # individual effects and within model is turned on (which is&#xA;1&amp;gt; # the same as entity fixed effects)&#xA;1&amp;gt; lm.model&amp;lt;-lm(NETTOSPARANDE ~ EURO + LAND, data=panel)&#xA;&#xA;1&amp;gt; plm.model&amp;lt;-plm(NETTOSPARANDE ~ EURO, index=c(&quot;LAND&quot;,&quot;ÅR&quot;), effect=&quot;individual&quot;, model=&quot;within&quot;, data=panel)&#xA;&#xA;1&amp;gt; # When looking at the coefficients without heteroscedasticity robust&#xA;1&amp;gt; # standard errors they are identical. They do also have the same&#xA;1&amp;gt; # value in Stata.&#xA;1&amp;gt; coeftest(lm.model)[1:2,]&#xA;             Estimate Std. Error   t value     Pr(&amp;gt;|t|)&#xA;(Intercept) -3.731024  0.7731778 -4.825570 1.726921e-06&#xA;EURO1        2.187170  0.4076720  5.365024 1.112984e-07&#xA;&#xA;1&amp;gt; coeftest(plm.model)&#xA;&#xA;t test of coefficients:&#xA;&#xA;      Estimate Std. Error t value  Pr(&amp;gt;|t|)    &#xA;EURO1  2.18717    0.40767   5.365 1.113e-07 ***&#xA;---&#xA;Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 &#xA;&#xA;&#xA;1&amp;gt; # But when looking at the coefficients using heteroscedasticity &#xA;1&amp;gt; # robust standard errors the lm.model and the plm.model produces&#xA;1&amp;gt; # different standard errors.&#xA;1&amp;gt; coeftest(lm.model, vcov.=vcovHC(lm.model, method=&quot;white2&quot;, type=&quot;HC1&quot;))[1:2,]&#xA;             Estimate Std. Error    t value     Pr(&amp;gt;|t|)&#xA;(Intercept) -3.731024  0.3551280 -10.506138 5.102122e-24&#xA;EURO1        2.187170  0.3386029   6.459395 2.009894e-10&#xA;&#xA;1&amp;gt; coeftest(plm.model, vcov.=vcovHC(plm.model, method=&quot;white2&quot;, type=&quot;HC1&quot;))&#xA;&#xA;t test of coefficients:&#xA;&#xA;      Estimate Std. Error t value  Pr(&amp;gt;|t|)    &#xA;EURO1  2.18717    0.33849  6.4615 1.983e-10 ***&#xA;---&#xA;Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2724" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-01-10T23:31:21.637" LastActivityDate="2011-01-10T23:31:21.637" Title="Why do I get different heteroscedasticity robust standard errors in R when using the plm package?" Tags="&lt;r&gt;&lt;robust&gt;&lt;panel-data&gt;&lt;heteroscedasticity&gt;&lt;fixed-effects-model&gt;" AnswerCount="2" />
  <row Id="6142" PostTypeId="1" AcceptedAnswerId="6144" CreationDate="2011-01-10T21:51:11.223" Score="1" ViewCount="94" Body="&lt;p&gt;I tried to group a set of elements and to calculate the row means&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If my list has more than one element, it works fine:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;tapply(colnames(myMA), c(1,1,1,2,2,2,3,3,4,4), list)    &#xA;myMAmean &amp;lt;- sapply(myList, function(x) rowMeans(myMA[,x]))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, in my data some of the rows are unique:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;myList &amp;lt;- tapply(colnames(myMA), c(1,1,1,2,2,2,3,3,4,5), list)&#xA;myMAmean &amp;lt;- sapply(myList, function(x) rowMeans(myMA[,x]))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Notice the 4 &amp;amp; 5 it makes them as unique so if I run this it says&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&quot;Error in rowMeans(myMA[, x]) : &#xA;  'x' must be an array of at least two dimensions&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I don't want to run a loop. Is there any solutions for this?&lt;/p&gt;&#xA;" OwnerUserId="2725" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-01-10T22:03:29.930" LastActivityDate="2011-01-10T22:03:29.930" Title="How to calculate the rowMeans with some single rows in data?" Tags="&lt;r&gt;" AnswerCount="1" />
  <row Id="6143" PostTypeId="2" ParentId="6141" CreationDate="2011-01-10T22:01:37.537" Score="0" ViewCount="" Body="&lt;p&gt;@Skolnick -&#xA;After an admittedly quick look at your problem, it seems that plm() truncates and/or rounds results to less digits than lm().  I think this is the most likely cause of the differences in your heteroscedasticity-robust standard error estimates.  Explore summary(plm.model) and summary(lm.model) for differences in residual quantiles.  &lt;/p&gt;&#xA;" OwnerUserId="2727" LastActivityDate="2011-01-10T22:01:37.537" />
  <row Id="6144" PostTypeId="2" ParentId="6142" CreationDate="2011-01-10T22:01:55.583" Score="5" ViewCount="" Body="&lt;p&gt;You can prevent the loss of the dimension attribute when using &quot;[&quot; with drop=FALSE:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;tapply(colnames(myMA), c(1,1,1,2,2,2,3,3,4,4), list)&#xA;myMAmean &amp;lt;- sapply(myList, function(x) rowMeans(myMA[,x, drop=FALSE]))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2129" LastActivityDate="2011-01-10T22:01:55.583" CommentCount="1" />
  <row Id="6145" PostTypeId="2" ParentId="6141" CreationDate="2011-01-10T22:06:42.507" Score="2" ViewCount="" Body="&lt;p&gt;Though numerically both of your model coefficients estimates coincide, you are actually fitting two different models: least square dummy variables and fixed effects.&#xA;They differ on their assumptions, so the robust standard errors are calculated differently. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Function &lt;code&gt;vcovHC&lt;/code&gt; is a wrapper function, different functions are actually used in case of &lt;code&gt;lm&lt;/code&gt; model and &lt;code&gt;plm&lt;/code&gt;. In the first case the function &lt;code&gt;vcovHC&lt;/code&gt; comes from the &lt;code&gt;sandwich&lt;/code&gt; package, in the second from the &lt;code&gt;plm&lt;/code&gt; package. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The difference in results arises because  &lt;code&gt;vcovHC&lt;/code&gt; for &lt;code&gt;lm&lt;/code&gt; object does not exploit panel data structure. Comparing&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plm:::vcovHC.plm&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;vcovHC.default&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;reveals that for panel data there is additional argument &lt;code&gt;cluster = c(&quot;group&quot;, &quot;time&quot;)&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Look into &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0262232588&quot; rel=&quot;nofollow&quot;&gt;Wooldridge book&lt;/a&gt; for more explanation why least squares dummy variables and fixed effects regressions differ. It has whole chapter dedicated for that. &lt;/p&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-01-10T22:29:15.153" LastActivityDate="2011-01-10T22:29:15.153" CommentCount="3" />
  <row Id="6146" PostTypeId="1" CreationDate="2011-01-10T22:38:31.137" Score="3" ViewCount="160" Body="&lt;p&gt;I want to find the conjugate prior distribution for the &lt;a href=&quot;https://secure.wikimedia.org/wikipedia/en/wiki/Fisher%27s_noncentral_hypergeometric_distribution&quot; rel=&quot;nofollow&quot;&gt;Fisher's noncentral hypergeometric distribution&lt;/a&gt;? Basically I want to integrate the parameters out of the distributions to arrive at the Bayesian Likelihood of observing some data with the noncentral hypergeometric distribution.&lt;/p&gt;&#xA;" OwnerUserId="2728" LastEditorUserId="2728" LastEditorDisplayName="" LastEditDate="2011-01-10T22:44:35.567" LastActivityDate="2011-01-18T14:52:28.410" Title="What is the conjugate of the noncentral hypergeometric distribution?" Tags="&lt;bayesian&gt;" AnswerCount="2" />
  <row Id="6147" PostTypeId="1" CreationDate="2011-01-10T23:15:27.453" Score="0" ViewCount="107" Body="&lt;p&gt;I'm brushing up my stats for an AI course that is going to show me the Gaussian mixture model. My prof said I'm supposed to know how to &quot;calculate the probability of a multivariate and continuous event X under the normal distribution&quot;. I have forgot this multivariate density function. Can someone please remind/show me how to do this? Thank you.&lt;/p&gt;&#xA;" OwnerUserId="2729" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-01-10T23:21:22.787" LastActivityDate="2011-01-10T23:21:22.787" Title="How to calculate probability for a multivariate/continuous event" Tags="&lt;distributions&gt;&lt;probability&gt;&lt;homework&gt;&lt;normal-distribution&gt;" CommentCount="2" />
  <row Id="6148" PostTypeId="2" ParentId="6080" CreationDate="2011-01-11T00:11:47.023" Score="0" ViewCount="" Body="&lt;p&gt;Follow-up:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;The SICHEL() function is in the gamlss.dist package.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Would a simple (and correct!) method to adjust the &gt; 0 probabilities be to divide them by (1-.27)?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;--Jeff&lt;/p&gt;&#xA;" OwnerUserId="2730" LastActivityDate="2011-01-11T00:11:47.023" />
  <row Id="6149" PostTypeId="2" ParentId="6146" CreationDate="2011-01-11T00:20:06.003" Score="1" ViewCount="" Body="&lt;p&gt;Perhaps a conjugate prior does not exist for the noncentral hypergeometric distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If someone wants to confirm this, it is worth noting that the conjugate to a univariate hypergeometric distribution is the beta-binomial; the conjugate to a multivariate hypergeometric distribution is the Dirichlet-Multinomial (from &lt;a href=&quot;http://www.johndcook.com/CompendiumOfConjugatePriors.pdf&quot; rel=&quot;nofollow&quot;&gt;Fink, 1997&lt;/a&gt;). For more details on the derivation, see &lt;a href=&quot;http://www.informaworld.com/smpp/content~content=a780123192~db=all&quot; rel=&quot;nofollow&quot;&gt;Dyer and Pierce, 1993&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Dyer, Danny and Pierce, Rebecca L. &quot;On the Choice of the Prior Distribution in Hypergeometric Sampling&quot;. Communications in Statistics-Theory and Methods, 1993, 22(8), 2125-214&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Fink, D. 1997. A Compendium of Conjugate Priors.&lt;/p&gt;&#xA;" OwnerUserId="1381" LastEditorUserId="1381" LastEditorDisplayName="" LastEditDate="2011-01-11T19:20:21.523" LastActivityDate="2011-01-11T19:20:21.523" CommentCount="2" />
  <row Id="6150" PostTypeId="1" AcceptedAnswerId="6177" CreationDate="2011-01-11T01:06:58.773" Score="4" ViewCount="123" Body="&lt;p&gt;&lt;strong&gt;Problem&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to plot the variance explained by each of 30 parameters, for example as a barplot with a different bar for each parameter, and variance on the y axis:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/AERls.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the variances are are strongly skewed toward small values, including 0, as can be seen in the histogram below:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/tVWYB.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I transform them by $log(x+1)$, it will be easier to see differences among the small values (histogram and barplot below):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/imlTM.png&quot; alt=&quot;alt text&quot;&gt;&lt;img src=&quot;http://i.imgur.com/s8rEU.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Plotting on a log-scale is common, but is plotting $log(x+1)$ similarly reasonable? &lt;/p&gt;&#xA;" OwnerUserId="1381" LastEditorUserId="1381" LastEditorDisplayName="" LastEditDate="2011-01-11T04:03:42.273" LastActivityDate="2011-01-11T21:19:35.873" Title="Is visualization sufficient rationale for transforming data?" Tags="&lt;data-visualization&gt;&lt;data-transformation&gt;" AnswerCount="2" />
  <row Id="6151" PostTypeId="1" CreationDate="2011-01-11T01:09:34.687" Score="0" ViewCount="376" Body="&lt;p&gt;I want to test if there is a rivalry among two siblings in a family.  I have 15 questions in my study and I let my 100 respondents ( distributed equally to two siblings)   ranked them 1 to 15. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How should I analyse this data?&lt;/p&gt;&#xA;" OwnerUserId="2731" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2011-01-13T20:58:11.047" LastActivityDate="2011-01-13T22:33:59.487" Title="Analysing questionnaire data" Tags="&lt;hypothesis-testing&gt;&lt;correlation&gt;&lt;multiple-comparisons&gt;&lt;statistical-significance&gt;&lt;questionnaires&gt;" AnswerCount="2" CommentCount="3" />
  <row Id="6152" PostTypeId="1" AcceptedAnswerId="6175" CreationDate="2011-01-11T01:45:42.400" Score="4" ViewCount="352" Body="&lt;p&gt;Hi All,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a (I suspect) simple question. I have time series cross section data on voting behavior in the Council of the European Union (the monthly number of yes, no and abstentions for each member state from 1999 to 2007). So basically the variables are counts, thus a Poisson/negative binomial regression would be appropriate, possibly with lagged dependent variables on the right hand side to control for time dependencies. I have seen papers with people using such negative binomial models to forecast, for instance the number of monthly legislative acts adopted in the future, and I have three questions in this regard:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;How can i run a negative binomial regression on panel data without making any inferential mistakes?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;How can I use a negative binomial model with lags to forecast future values of the dependent variable.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Can this be done in R?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Thomas&lt;/p&gt;&#xA;" OwnerUserId="2704" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-01-11T10:24:08.690" LastActivityDate="2011-01-11T20:57:05.410" Title="Time Series Cross Section Forecasting with R" Tags="&lt;r&gt;&lt;time-series&gt;&lt;forecasting&gt;&lt;panel-data&gt;&lt;negative-binomial&gt;" AnswerCount="2" FavoriteCount="3" />
  <row Id="6153" PostTypeId="2" ParentId="6150" CreationDate="2011-01-11T02:57:05.127" Score="3" ViewCount="" Body="&lt;p&gt;It can be reasonable. The better question to ask is whether 1 is the proper number to add. What was your minimum? If it was 1 to begin with, then you are imposing a particular interval between items with value of zero and those with value 1. Depending on the domain of study it may make more sense to choose 0.5 or 1/e as the offset. The implication of transforming to a log scale is that you now have a ratio scale.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I am bothered by the plots. I would ask whether a model that has most of the explained variance in the tail of a skewed distribution  be considered to have desirable statistical properties. I think not.&lt;/p&gt;&#xA;" OwnerUserId="2129" LastActivityDate="2011-01-11T02:57:05.127" CommentCount="1" />
  <row Id="6154" PostTypeId="1" AcceptedAnswerId="6162" CreationDate="2011-01-11T03:27:10.977" Score="1" ViewCount="116" Body="&lt;p&gt;Anyone know why when you run a SEM model in SAS using proc calis (or tcalis) you do not get p-values for the parameter estimates?  It does supply a t-value however.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Two popular SEM packages in R, 'sem' and 'lavaan', both give p-values for the estimates but they use a Z test statistic.&lt;/p&gt;&#xA;" OwnerUserId="2310" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-01-11T04:22:03.180" LastActivityDate="2011-01-11T11:37:56.767" Title="Proc Calis (or TCalis) and p-values" Tags="&lt;r&gt;&lt;modeling&gt;&lt;sas&gt;&lt;p-values&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="6155" PostTypeId="1" AcceptedAnswerId="6157" CreationDate="2011-01-11T04:57:29.320" Score="3" ViewCount="499" Body="&lt;p&gt;Hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not sure the subject enters into the CrossValidated interest. You'll tell me.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have to study a graph (from the &lt;a href=&quot;http://en.wikipedia.org/wiki/Graph_theory&quot; rel=&quot;nofollow&quot;&gt;graph theory&lt;/a&gt;)&#xA;ie. &#xA;I have a certain number of dots that are connected.&#xA;I have a table with all the dots and the dots each one is dependant on.&#xA;(I have also another table with the implications)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My questions are:&lt;br&gt;&#xA;Is there a good software (or a R package) to study that easily?&lt;br&gt;&#xA;Is there an easy way to display the graph?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&#xA;fRed&lt;/p&gt;&#xA;" OwnerUserId="1709" LastEditorUserId="1709" LastEditorDisplayName="" LastEditDate="2011-02-28T06:05:59.463" LastActivityDate="2011-02-28T06:14:53.413" Title="Graph theory -- analysis and visualization" Tags="&lt;r&gt;&lt;data-visualization&gt;&lt;graph-theory&gt;" AnswerCount="4" CommentCount="2" FavoriteCount="1" />
  <row Id="6156" PostTypeId="2" ParentId="6155" CreationDate="2011-01-11T07:47:02.697" Score="2" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://igraph.sourceforge.net/index.html&quot; rel=&quot;nofollow&quot;&gt;iGraph&lt;/a&gt; is a very interesting cross-language (R, Python, Ruby, C) library.&#xA;It allows you to work with unidirected and directed graphs and has quite a few analysis algorithms already implemented.&lt;/p&gt;&#xA;" OwnerUserId="582" LastActivityDate="2011-01-11T07:47:02.697" CommentCount="2" />
  <row Id="6157" PostTypeId="2" ParentId="6155" CreationDate="2011-01-11T07:51:00.740" Score="5" ViewCount="" Body="&lt;p&gt;There are various packages for representing directed and undirected graphs, incidence/adjacency matrix, etc. in addition to &lt;a href=&quot;http://cran.r-project.org/web/packages/graph/index.html&quot; rel=&quot;nofollow&quot;&gt;graph&lt;/a&gt;; look for example at the &lt;a href=&quot;http://cran.r-project.org/web/views/gR.html&quot; rel=&quot;nofollow&quot;&gt;gR&lt;/a&gt; Task view. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For visualization and basic computation, I think the &lt;a href=&quot;http://cran.r-project.org/web/packages/igraph/index.html&quot; rel=&quot;nofollow&quot;&gt;igraph&lt;/a&gt; package is the reliable one, in addition to &lt;a href=&quot;http://www.bioconductor.org/packages/release/bioc/html/Rgraphviz.html&quot; rel=&quot;nofollow&quot;&gt;Rgraphviz&lt;/a&gt; (on BioC as pointed out by @Rob). Be aware that for the latter to be working properly, &lt;a href=&quot;http://www.graphviz.org/&quot; rel=&quot;nofollow&quot;&gt;graphviz&lt;/a&gt; must be installed too. The &lt;a href=&quot;http://cran.r-project.org/web/packages/igraph/index.html&quot; rel=&quot;nofollow&quot;&gt;igraph&lt;/a&gt; package has nice algorithms for creating good layouts, much like &lt;a href=&quot;http://www.graphviz.org/&quot; rel=&quot;nofollow&quot;&gt;graphviz&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an example of use, starting from a fake adjacency matrix:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;adj.mat &amp;lt;- matrix(sample(c(0,1), 9, replace=TRUE), nr=3)&#xA;g &amp;lt;- graph.adjacency(adj.mat)&#xA;plot(g)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/Vjn2D.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-01-11T07:51:00.740" CommentCount="1" />
  <row Id="6158" PostTypeId="1" AcceptedAnswerId="6161" CreationDate="2011-01-11T07:56:27.927" Score="1" ViewCount="71" Body="&lt;p&gt;I collected some data with an instruments with 1Hz sampling clock, now I want to low-pass filter the data to separate the mean and fluctuation part (Reynolds decomposition).&#xA;How can I design a low-pass filter with a cutoff period of 20 minutes?&lt;/p&gt;&#xA;" OwnerUserId="1637" LastActivityDate="2011-01-11T10:21:42.920" Title="Low-pass Filter" Tags="&lt;time-series&gt;" AnswerCount="1" />
  <row Id="6160" PostTypeId="2" ParentId="6155" CreationDate="2011-01-11T09:38:04.657" Score="4" ViewCount="" Body="&lt;p&gt;Another option is the statnet package. Statnet has functions for all the commonly used measures in SNA, and can also estimate ERG models. If you have your data in an edge list, read in the data as follows (assuming your data frame is labelled &quot;edgelist&quot;):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;net &amp;lt;- as.network(edgelist, matrix.type = &quot;edgelist&quot;, directed = TRUE) #if the network is directed, otherwise: directed = FALSE&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If your data is in an adjacancy matrix you replace the matrix.type argument with &quot;adjacancy&quot;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;net &amp;lt;- as.network(edgelist, matrix.type = &quot;adjacancy&quot;, directed = TRUE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The statnet package has some very nice plotting capabilities. To do a simple plot simply type:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;gplot(net)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To scale the nodes according to their betweenness centrality, simply do:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;bet &amp;lt;- betweenness(net)&#xA;gplot(net, vertex.cex = bet)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;By default the gplot function uses the Fructherman-Rheingold algorithm for placing the nodes, however this can be controlled from the mode option, for instance to use MDS for the placement of nodes type:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;gplot(net, vertex.cex, mode = &quot;mds&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;or to use a circle layout:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;gplot(net, vertex.cex, mode = &quot;circle&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;There are many more possibilities, and this &lt;a href=&quot;http://students.washington.edu/mclarkso/documents/gplot%20Ver2.pdf&quot; rel=&quot;nofollow&quot;&gt;guide&lt;/a&gt; covers most of the basic options.&#xA;For a self contained example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;net &amp;lt;- rgraph(20) #generate a random network with 20 nodes&#xA;bet &amp;lt;- betweenness(net) #calculate betweenness scores&#xA;gplot(net) #a simple plot&#xA;gplot(net, vertex.cex = bet/3) #nodes scaled according to their betwenness centrality, the measure is divided by 3 so the nodes dont become to big.&#xA;gplot(net, vertex.cex = bet/3, mode = &quot;circle&quot;) #with a circle layout&#xA;gplot(net, vertex.cex = bet/3, mode = &quot;circle&quot;, label = 1:20) #with node labels&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Best, Thomas&lt;/p&gt;&#xA;" OwnerUserId="2704" LastEditorUserId="2704" LastEditorDisplayName="" LastEditDate="2011-01-11T09:54:37.487" LastActivityDate="2011-01-11T09:54:37.487" CommentCount="1" />
  <row Id="6161" PostTypeId="2" ParentId="6158" CreationDate="2011-01-11T10:21:42.920" Score="3" ViewCount="" Body="&lt;p&gt;Well, you have to use Fourier transform, either in a dirty way by zeroing elements with frequencies you want to filter out or (in a more accurate way, with moving window) by convolution with filter, like in &lt;a href=&quot;https://ccrma.stanford.edu/~jos/sasp/Example_1_Low_Pass_Filtering.html&quot; rel=&quot;nofollow&quot;&gt;this MATLAB code example&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2011-01-11T10:21:42.920" CommentCount="2" />
  <row Id="6162" PostTypeId="2" ParentId="6154" CreationDate="2011-01-11T11:37:56.767" Score="2" ViewCount="" Body="&lt;p&gt;@mpiktas is right and knowing the value of the test statistic ($t$ or $z$) allows you to know which parameter estimate is significant at the desired $\alpha$ level. In practice, the $t$-statistic is equivalent to a $z$-score for large samples (which is often the case in SEM), and the significance thresholds are 1.96 and 2.58 for the .05 and .01 $\alpha$ levels. Most of the time, $p$-values are interesting when comparing models; as shown in this nice tutorial on &lt;a href=&quot;http://www.sma.org.sg/smj/4612/4612bs1.pdf&quot; rel=&quot;nofollow&quot;&gt;Structural equation modeling&lt;/a&gt; using SAS, by Y H Chan, giving $t$- or $z$-statistic with associated critical values at 5% should be enough, IMO.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-01-11T11:37:56.767" CommentCount="1" />
  <row Id="6163" PostTypeId="1" AcceptedAnswerId="6166" CreationDate="2011-01-11T13:53:36.093" Score="2" ViewCount="178" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Deming_regression&quot; rel=&quot;nofollow&quot;&gt;Deming Regression&lt;/a&gt; is a regression technique taking into account uncertainty in both the explanatory and dependent variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Although I have found some interesting references on the calculation of this property in &lt;a href=&quot;http://iopscience.iop.org/0957-0233/18/11/025&quot; rel=&quot;nofollow&quot;&gt;matlab&lt;/a&gt; and in &lt;a href=&quot;http://www.mail-archive.com/r-help@r-project.org/msg85070.html&quot; rel=&quot;nofollow&quot;&gt;R&lt;/a&gt; I'm stuck when I try to calculate the standard prediction error. The error on the model estimate is given in both methods, but I wonder if I can use that for prediction by using the variances of their prediction.&#xA;Eg: &lt;code&gt;var(y_pred) = var(a*x+b) = E[a]^2*var(x) + E[x]^2*a+var(b)&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="2732" LastActivityDate="2011-01-11T20:09:50.207" Title="What is the prediction error while using deming regression (weighted total least squares)" Tags="&lt;regression&gt;&lt;variance&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="1" />
  <row Id="6164" PostTypeId="1" CreationDate="2011-01-11T14:13:18.877" Score="0" ViewCount="385" Body="&lt;p&gt;Hi, I have used lag functions of lag values in estimating GARCH model in eviews but dont know how to interprete it and how to describe which model  i.e. ARCH, GARCH or TGARCH is better on the basis of information criterion.&lt;/p&gt;&#xA;" OwnerUserId="2733" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-01-11T19:01:38.427" LastActivityDate="2011-01-11T19:01:38.427" ClosedDate="2011-04-11T01:53:43.923" Title="Output of GARCH model (eviews output) using lag of log values" Tags="&lt;garch&gt;&lt;eviews&gt;" CommentCount="1" />
  <row Id="6165" PostTypeId="2" ParentId="6133" CreationDate="2011-01-11T14:50:31.043" Score="1" ViewCount="" Body="&lt;p&gt;Read this: &lt;a href=&quot;http://www-users.cs.umn.edu/~kumar/papers/anomaly-survey.php&quot; rel=&quot;nofollow&quot;&gt;Anomaly Detection : A Survey (2009)&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="635" LastActivityDate="2011-01-11T14:50:31.043" />
  <row Id="6166" PostTypeId="2" ParentId="6163" CreationDate="2011-01-11T15:11:25.573" Score="4" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt; I've updated the answer to reflect the discussions in the comments.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The model is given as&#xA;\begin{align*}&#xA;y&amp;amp;=y^{*}+\varepsilon\\&#xA;x&amp;amp;=x^{*}+\eta\\&#xA;y^{*}&amp;amp;=\alpha+x^{*}\beta&#xA;\end{align*}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So when forecasting with a new value $x$ we can forecast either $y$ or $y^{*}$. Their forecasts coincide $\hat{y}=\hat{y}^{*}=\hat{\alpha}+\hat{\beta}x$  but their error variances will be different:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Var(\hat{y})=Var(\hat{y}^{*})+Var(\varepsilon)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To get $Var(\hat{y}^{*})$ write&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align*}&#xA;\hat{y}^{*}-y^{*}&amp;amp;=\hat{\alpha}-\alpha+\hat{\beta} (x^{*}+\eta)-\beta x^{*}\\&#xA;&amp;amp;=(\hat{\alpha}-\alpha)+(\hat{\beta}-\beta) x^{*}+ \hat{\beta}\eta&#xA;\end{align*}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align*}&#xA;Var(\hat{y}^{*})=E(\hat{y}^{*}-y^{*})^2&amp;amp;=D(\hat{\alpha}-\alpha)+D(\hat{\beta}-\beta) (x^{*})^2+ E\hat{\beta}^2D\eta\\&#xA;&amp;amp;+cov(\hat{\alpha}-\alpha,\hat{\beta}-\beta)x^{*}&#xA;\end{align*}&lt;/p&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-01-11T20:09:50.207" LastActivityDate="2011-01-11T20:09:50.207" CommentCount="5" />
  <row Id="6167" PostTypeId="1" CreationDate="2011-01-11T15:59:24.120" Score="3" ViewCount="50" Body="&lt;p&gt;I have large comparison data in form &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a pairwise comparison data each data point compares two alternatives.&lt;br&gt;&#xA;For instance:&lt;br&gt;&#xA;A &gt; B (A is preferred to B, A and B are classes, not numbers)&lt;br&gt;&#xA;A &gt; B&lt;br&gt;&#xA;B &gt; A&lt;br&gt;&#xA;B &gt; C&lt;br&gt;&#xA;A &gt; C&lt;br&gt;&#xA;etc ...  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In short we can write numbers of preferences in data set:&lt;br&gt;&#xA;A vs B 999:1&lt;br&gt;&#xA;X vs A 500:500&lt;br&gt;&#xA;X vs B 500:500  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bradley-Terry model models pairwise preference by assigning on parameter to each class:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ P(A &amp;gt; B\; |\; \vec{w} ) = \frac{w_A}{w_A + w_B} $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Parameters can be estimated from data through maximal likelihood.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm looking for extension of Bradley-Terry model (or a completely new model) that would be able to model situations like the one above. I.e. A is always strongly preferred to B: $P(A&amp;gt;B) = 0.999$ but $ P(X&amp;lt;A) = P(X&amp;lt;B) = 0.5 $.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;B-T model cannot represent that. Do you have any ideas how to create better model ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS&#xA;The model will be applied to a data of size $10^8$ so it would be good to have simple maximal likelihood algorithm.&lt;/p&gt;&#xA;" OwnerUserId="217" LastActivityDate="2011-01-11T15:59:24.120" Title="How to model pairwise preference with both strong and weak preferences?" Tags="&lt;modeling&gt;&lt;bradley-terry-model&gt;&lt;pairwise-comparison&gt;" CommentCount="1" />
  <row Id="6169" PostTypeId="1" AcceptedAnswerId="6173" CreationDate="2011-01-11T19:10:46.507" Score="6" ViewCount="247" Body="&lt;p&gt;What are the appropriate uses for a grouped vs a stacked bar plot?&lt;/p&gt;&#xA;" OwnerUserId="559" LastEditorUserId="559" LastEditorDisplayName="" LastEditDate="2011-01-12T00:58:18.377" LastActivityDate="2011-01-13T11:56:13.593" Title="Group vs Stacked Bar Plots" Tags="&lt;data-visualization&gt;&lt;barplot&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="3" />
  <row Id="6170" PostTypeId="1" AcceptedAnswerId="6174" CreationDate="2011-01-11T20:03:11.703" Score="3" ViewCount="56" Body="&lt;p&gt;I am trying to model future sales data for products which have very low sales volume.  I am a programmer with a smattering in statistics, so I apologise in advance if this qustion is naive!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is what distribution is most suitable for my sales profile and is it possible to verify the distribution.  The exact framing of the problem is that we might have a product that has say 6 units of inventory and we may sell 8 units a year with a &quot;standard deviation&quot; of 5 units (i.e. the sales are lumpy so we calculate a standard deviation, but its not really a normal distribution)...we want to say with a certain probability how many days inventory we have left.  For high volume products we can assume the normal distribution and its pretty easy to back out the days inventory left.  However for low volume products we can't assume normal distribution (as obviously it is bounded by the fact that we can't have less than 0 sales).  I have looked at Poisson, but I am not sure if that is the most suitable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could someone point me to some resources that would help me identify the right model/technique to use?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Mike.&lt;/p&gt;&#xA;" OwnerUserId="2734" LastActivityDate="2011-01-11T20:37:46.450" Title="Suitable Distribution/Methodology for Estimating Data Points" Tags="&lt;distributions&gt;&lt;model-selection&gt;" AnswerCount="1" />
  <row Id="6171" PostTypeId="1" CreationDate="2011-01-11T20:16:36.053" Score="0" ViewCount="79" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;Possible Duplicate:&lt;/strong&gt;&lt;br&gt;&#xA;  &lt;a href=&quot;http://stats.stackexchange.com/questions/4756/confidence-interval-for-bernoulli-sampling&quot;&gt;Confidence interval for Bernoulli sampling&lt;/a&gt;  &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&#xA;&#xA;&lt;p&gt;I have a set of datapoints that are all 0 or 1, and the average is important to me.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I calculate my confidence in the average?  How many samples do I need to be confident that the average is not unduly influenced by chance?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To rephrase with a concrete example, consider a taking a test.  With each question on the test being graded as correct (1) or incorrect (0), how do I calculate my confidence that the test grade (percent correct, or the average) represents their true ability in the topic?  How long should the test be make sure it's close to their true ability?&lt;/p&gt;&#xA;" OwnerUserId="2735" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-01-11T20:27:54.903" LastActivityDate="2011-01-12T03:00:32.967" ClosedDate="2011-01-12T03:58:35.237" Title="For a univariate, binary dataset, how do I calculate my confidence in the average?" Tags="&lt;confidence-interval&gt;&lt;statistical-significance&gt;&lt;binomial&gt;&lt;averages&gt;" AnswerCount="0" CommentCount="0" />
  <row Id="6172" PostTypeId="2" ParentId="6152" CreationDate="2011-01-11T20:18:36.863" Score="3" ViewCount="" Body="&lt;p&gt;May be you can take a look at the &lt;strong&gt;pglm&lt;/strong&gt; package (from the same author of plm), use the family &lt;strong&gt;negbin&lt;/strong&gt;.  You can also try from a Bayesian point of view the MCMCglmm package.&lt;/p&gt;&#xA;" OwnerUserId="2028" LastActivityDate="2011-01-11T20:18:36.863" CommentCount="3" />
  <row Id="6173" PostTypeId="2" ParentId="6169" CreationDate="2011-01-11T20:30:15.043" Score="10" ViewCount="" Body="&lt;p&gt;I think grouped bars are preferable to stacked bars in most situations because they retain information about the sizes of the groups and stay readable even when you have multiple nominal categories.  For me, the segments of stacked bars get difficult to compare beyond two categories - and even with just two categories, they can be quite deceptive if your groups are of very different sizes.  I'd prefer a frequency table over a stacked bar plot any day.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should also consider a stacked &lt;em&gt;series&lt;/em&gt; of bar plots, with each group in a separate plot and each plot stacked on top of the last (there's an example on &lt;a href=&quot;http://www.robertluttman.com/vms/Week5/page9.htm&quot; rel=&quot;nofollow&quot;&gt;this page&lt;/a&gt;, referred to as &quot;layered histograms&quot;, which I find confusing).  This is probably what I use most often.  The &lt;code&gt;lattice&lt;/code&gt; and &lt;code&gt;ggplot2&lt;/code&gt; packages in R both support this kind of plotting.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Historical note: &lt;a href=&quot;http://www.worsleyschool.net/science/files/bargraphs/page.html&quot; rel=&quot;nofollow&quot;&gt;histograms != bar plots&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="71" LastEditorUserId="71" LastEditorDisplayName="" LastEditDate="2011-01-12T18:50:28.300" LastActivityDate="2011-01-12T18:50:28.300" CommentCount="5" />
  <row Id="6174" PostTypeId="2" ParentId="6170" CreationDate="2011-01-11T20:37:46.450" Score="2" ViewCount="" Body="&lt;p&gt;You are looking for a class of models known as 'purchase incidence'. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A poisson distribution with the rate of sales $\lambda$ such that $Y_{t}=\lambda t$ is the number of units sold during time period $t$ is a good place to start.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(Y_{t}=y_{t})=\frac{e^{-\lambda t}(\lambda t)^{y_{t}}}{y_{t}!}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, if you have 3 items left and 10 before restocking, the chance that you will have an item for each customer who wants to purchase one is $$1-P(Y_t=[0,1,2,3])$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Additional parameters can be added as needed to model known processes that cause the distribution of sales to vary with time. For example, heterogeneity in $\lambda$ can be modeled as $\lambda\sim\text{Gamma}(a,b)$. You can find a description of this and related models in chapter 12 of Leeflang 2000 &quot;Building models for marketing decisions&quot;&lt;/p&gt;&#xA;" OwnerUserId="1381" LastActivityDate="2011-01-11T20:37:46.450" CommentCount="5" />
  <row Id="6175" PostTypeId="2" ParentId="6152" CreationDate="2011-01-11T20:57:05.410" Score="3" ViewCount="" Body="&lt;p&gt;After a bit of research, I can give a partial answer. In his &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0262232197&quot; rel=&quot;nofollow&quot;&gt;book&lt;/a&gt;  Wooldridge discusses Poisson and negative binomial regressions for cross-section and panel data. But for regression with lagged variables he only discusses Poisson regression. Maybe negative binomial is discussed in the new edition. The main conclusion is that for random effects Poisson regression with lagged random variable can be estimated by mixed effects Poisson regression model. The detailed description can be found &lt;a href=&quot;http://www.cemmap.ac.uk/wps/cwp0218.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;. The mixed effects Poisson regression in R can be estimated with &lt;strong&gt;glmer&lt;/strong&gt; from package &lt;strong&gt;lme4&lt;/strong&gt;. To adapt it to work with panel data, you will need to create lagged variable explicitly. Then your estimation command should look something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;glmer(y~lagy+exo+(1|Country),data,family=quasipoisson)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You should also look  into &lt;strong&gt;gplm&lt;/strong&gt; package suggested by @dickoa. But be sure to check, whether it supports lagged variables. Yves Croissant, the creator of gplm and plm packages writes wonderful code, but unfortunately in my personal experience, the code is not tested enough, so bugs crop up more frequently than in standard R packages. &lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-01-11T20:57:05.410" CommentCount="3" />
  <row Id="6176" PostTypeId="1" CreationDate="2011-01-11T21:01:20.807" Score="8" ViewCount="314" Body="&lt;p&gt;I'm interested in learning more about nonparametric Bayesian (and related) techniques. My background is in computer science and I have never taken a course on measure theory or probability theory. Can any one recommend a readable introduction to these concepts to get me started?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I should mention that my background is in computer science, though I have had a limited amount of formal training in probability and statistics.&lt;/p&gt;&#xA;" OwnerUserId="1913" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-20T09:06:02.027" LastActivityDate="2011-05-20T09:06:02.027" Title="Readable introduction to measure theory" Tags="&lt;books&gt;&lt;big-list&gt;&lt;theory&gt;" AnswerCount="4" CommentCount="2" FavoriteCount="1" />
  <row Id="6177" PostTypeId="2" ParentId="6150" CreationDate="2011-01-11T21:19:35.873" Score="6" ViewCount="" Body="&lt;p&gt;This has been called a &quot;&lt;strong&gt;started logarithm&lt;/strong&gt;&quot; by some (&lt;em&gt;e.g.&lt;/em&gt;, John Tukey).  (For some examples, Google &lt;em&gt;john tukey &quot;started log&quot;&lt;/em&gt;.) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's perfectly fine to use.  In fact, you could expect to have to use a nonzero starting value to account for rounding of the dependent variable.  For example, rounding the dependent variable to the nearest integer effectively lops off 1/12 from its true variance, suggesting a reasonable start value should be at least 1/12.  (That value doesn't do a bad job with these data.  Using other values above 1 don't really change the picture much; they just raise all the values in the bottom right plot almost uniformly.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are deeper reasons to use the logarithm (or started log) to assess variance: for example, the slope of a plot of variance against estimated value on a log-log scale estimates a &lt;a href=&quot;http://staff.ustc.edu.cn/~zwp/teach/Reg/Boxcox.pdf&quot; rel=&quot;nofollow&quot;&gt;Box-Cox parameter for stabilizing the variance&lt;/a&gt;.  Such power-law fits of variance to some related variable are often observed.  (This is an empirical statement, not a theoretical one.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your purpose is to &lt;em&gt;present&lt;/em&gt; the variances, proceed with care.  Many audiences (apart from scientific ones) cannot understand a logarithm, much less a started one.  Using a start value of 1 at least has the merit of being a little simpler to explain and interpret than some other start value.  Something to consider is to plot their roots, which are the standard deviations, of course.  It would look something like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/gNlrW.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regardless, if your purpose is to explore the data, learn from them, fit a model, or evaluate a model, don't let anything get in the way of finding reasonable graphical representations of your data and derived values such as these variances.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-01-11T21:19:35.873" CommentCount="1" />
  <row Id="6178" PostTypeId="2" ParentId="6151" CreationDate="2011-01-11T22:34:30.497" Score="0" ViewCount="" Body="&lt;p&gt;I agree with David that your question needs some better explanation.  Perhaps you should start by giving us a snippet of your data set up so that we can see how it's structured.  Also indicate what variable links the individuals into sibling groups.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It sounds like you &lt;em&gt;might&lt;/em&gt; want to take a look at ranked order logit (type &quot;help rologit&quot;) if your data &amp;amp; analysis are something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;*****************************! Begin Example&#xA;&#xA;*-----run this code  your Stata do-file editor&#xA; clear&#xA;inp id ranking   option   x1 x2   male&#xA;1   4  1  1 0  0 &#xA;1   2  2  0 1  0&#xA;1   3  3  0 0  0&#xA;1   1  4  1 1  0&#xA;2   1  1  3 0  0&#xA;2   3  2  0 1  0&#xA;2   3  3  2 1  0&#xA;2   4  4  1 2  0&#xA;3   1  1  3 1  1&#xA;3   3  2  1 1  1&#xA;3   4  4  0 1  1&#xA;4   2  1  1 1  1&#xA;4   1  2  1 1  1&#xA;4   0  3  0 1  1&#xA;4   0  4  1 0  1&#xA;end&#xA;&#xA;rologit ranking x1 x2 male, group(id)&#xA;****************************! End Example&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;- Eric  &lt;br&gt; &#xA;eric.a.booth@gmail.com&lt;/p&gt;&#xA;" OwnerUserId="1033" LastActivityDate="2011-01-11T22:34:30.497" />
  <row Id="6179" PostTypeId="2" ParentId="4220" CreationDate="2011-01-11T22:56:03.347" Score="0" ViewCount="" Body="&lt;p&gt;The point value at a particular parameter value of a probability density plot would be a likelihood, right? If so, then the statement might be corrected by simply changing P(height|male) to L(height|male).&lt;/p&gt;&#xA;" OwnerUserId="1679" LastActivityDate="2011-01-11T22:56:03.347" />
  <row Id="6180" PostTypeId="1" CreationDate="2011-01-11T23:16:47.687" Score="3" ViewCount="140" Body="&lt;p&gt;Confidence intervals for binomial proportions have irregular coverage over the range of possible population parameters (e.g. see Brown et al. 2001 &lt;a href=&quot;http://projecteuclid.org/DPubS?service=UI&amp;amp;version=1.0&amp;amp;verb=Display&amp;amp;handle=euclid.ss/1009213286&quot; rel=&quot;nofollow&quot;&gt;http://projecteuclid.org/DPubS?service=UI&amp;amp;version=1.0&amp;amp;verb=Display&amp;amp;handle=euclid.ss/1009213286&lt;/a&gt;). How can I formally and usefully describe the properties of the confidence intervals?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say I toss a coin ten times and obtain seven heads. Are the following statements accurate?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the Clopper-Pearson method:&lt;br&gt;&#xA;The interval 0.3475–0.9333 has been generated by a method that will on at least 95% of occasions, for any true population proportion, contain the true population proportion. The long-run frequency with which this method would yield confidence intervals containing the true population proportion pertaining to this particular experiment is at least 95%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the Wilson's scores method:&lt;br&gt;&#xA;The interval 0.3968–0.8922 has been generated by a method that will on 95% of occasions, averaged over all population proportions, contain the true population proportion. The long-run frequency with which this method would yield confidence intervals containing the true population proportion pertaining to this particular experiment may be more or less than 95%.&lt;/p&gt;&#xA;" OwnerUserId="1679" LastActivityDate="2011-01-15T15:02:55.903" Title="Statement of result for binomial confidence intervals" Tags="&lt;confidence-interval&gt;&lt;binomial&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="6181" PostTypeId="1" CreationDate="2011-01-12T01:10:42.207" Score="1" ViewCount="297" Body="&lt;p&gt;I am using IDL regression function to compute the multiple linear correlation coefficient...&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x = [transpose(anom1),transpose(anom2),transpose(anom3)]&#xA;coef = regress(x,y, const=a0, correlation=corr, mcorrelation=mcorr, sigma=stderr)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;mcorr&lt;/code&gt; is returned with values between 0.0 and 1.0. Clearly, the result of the following IDL code I dug out of the can can't be negative...&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mcorrelation = SQRT(mult_covariance)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But, could you help me verify this? I would like a citation or to see 'the math' behind calculating &lt;code&gt;mcorrelation&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2738" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2011-01-13T17:49:23.117" LastActivityDate="2011-01-19T09:30:53.440" Title="Can the multiple linear correlation coefficient be negative?" Tags="&lt;regression&gt;&lt;r-squared&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="6182" PostTypeId="1" AcceptedAnswerId="6185" CreationDate="2011-01-12T01:15:14.267" Score="3" ViewCount="152" Body="&lt;p&gt;I've never taken a stats course so I really don't know where to begin on this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using microcontroller that according to the datasheet, has a Flash memory that can be reprogrammed a minimum of 10,000 times, with 100,000 times being a more typical number.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These numbers seem low to me (based on other devices using a similar technology which claim lifetimes of an order of magnitude higher), so I decided to take one of our products and do destructive testing on it.  I started testing last Friday night and it is already up to 600,000 events without an error (as determined by calculating a 16-bit CRC over the 4K bytes of data test being Flashed).  Based on this device far surpassing the &quot;typical&quot; value in the datasheet, I have started tests on three more devices.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So at some point I will have a failure value for all four devices.  How can I use that data to predict an expected lifetime x for the device with a 99.9% certainty?  (Not sure if I am saying that right, what I mean is that only 0.1% of devices will fail before x events).&lt;/p&gt;&#xA;" OwnerUserId="2698" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-01-12T08:04:13.837" LastActivityDate="2011-01-12T16:33:25.283" Title="Predicting number of events with 99.9% probability based on tests of four devices" Tags="&lt;probability&gt;" AnswerCount="1" CommentCount="5" />
  <row Id="6184" PostTypeId="2" ParentId="4756" CreationDate="2011-01-12T02:05:53.253" Score="5" ViewCount="" Body="&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;em&gt;If&lt;/em&gt; the average, $\hat{p}$, is not near $1$ or $0$, and sample size $n$ is sufficiently large (i.e. $n\hat{p}&amp;gt;5$ and $n(1-\hat{p})&amp;gt;5$, the confidence interval can be estimated by a normal distribution and the confidence interval constructed thus:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\hat{p}\pm z_{1-\alpha/2}\sqrt{\frac{\hat{p}(1-\hat{p})}{n}}$$ &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;em&gt;If&lt;/em&gt; $\hat{p} = 0$ and $n&amp;gt;30$, the $95\%$ confidence interval is approximately $[0,\frac{n}{3}]$ &lt;a href=&quot;http://www.jstor.org/pss/2685405&quot; rel=&quot;nofollow&quot;&gt;(Javanovic and Levy, 1997)&lt;/a&gt;; the opposite holds for $\hat{p}=1$. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;em&gt;Else&lt;/em&gt; &lt;a href=&quot;http://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval&quot; rel=&quot;nofollow&quot;&gt;Wikipedia&lt;/a&gt; provides a good overview and points to Agresti and Couli (1998) and Ross (2003) for details about the use of estimates other than the normal approximation, the Wilson score, Clopper-Pearson, or Agresti-Coull intervals. These can be more accurate when above assumptions about $n$ and $\hat{p}$ are not met.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.jstor.org/stable/2685469&quot; rel=&quot;nofollow&quot;&gt;Agresti, Alan; Coull, Brent A. (1998). &quot;Approximate is better than 'exact' for interval estimation of binomial proportions&quot;. The American Statistician 52: 119–126.&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.jstor.org/pss/2685405&quot; rel=&quot;nofollow&quot;&gt;Jovanovic, B. D. and P. S. Levy, 1997. A Look at the Rule of Three. The American Statistician Vol. 51, No. 2, pp. 137-139&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://dx.doi.org/10.1016%2FS0010-4825%2803%2900019-2&quot; rel=&quot;nofollow&quot;&gt;Ross, T. D. (2003). &quot;Accurate confidence intervals for binomial proportion and Poisson rate estimation&quot;. Computers in Biology and Medicine 33: 509–531.&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1381" LastEditorUserId="1381" LastEditorDisplayName="" LastEditDate="2011-01-12T02:58:18.993" LastActivityDate="2011-01-12T02:58:18.993" CommentCount="2" />
  <row Id="6185" PostTypeId="2" ParentId="6182" CreationDate="2011-01-12T05:18:54.760" Score="4" ViewCount="" Body="&lt;p&gt;@whuber makes an important point about assumptions,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is common to assume that failure occurs either because a part is defective or because it has worn out. This means that there are two processes causing failure, and defective units fail shortly after deployment whereas wearing out happens over time. It is difficult (or impossible) to justify parameterizing both of these processes if $n=4$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Case 1&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a simple case, assume that defective units can be eliminated (e.g. by an initial screening or 'burn in'), and all units fail due to a constant wearing process. This assumption may be valid since the data sheet says that the minimum expected lifetime is $10,000$ events. One might model this process as an exponential distribution:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example: assume that your four test units fail after $[5,10,20,40]x10^{5}$ reprogrammings. You can find the maximum likelihood estimate for the exponential distribution in R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(MASS)&#xA;fitdistr(c(5, 10, 20, 40)*1e+5, 'exponential')&#xA;lambda &amp;lt;- fitdistr(c(5, 10, 20, 40)*1e+5, 'exponential')$estimate&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To find the time during which only 1/1000 units will fail, estimate the 0.1th quantile of this distribution.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;qexp(0.001, lambda)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In this example, the result is an expected lifetime of 1875.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;edit&lt;/em&gt; As whuber points out, $n=4$ is still to small for this problem, because it is very difficult to estimate small probabilities, and distribution tails are very sensitive to assumptions about the probability distributions used to model the data, and to the data itself. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since you have good prior knowledge from the product datasheet, you could incorporate this information into your analysis as a prior on lambda. The gamma distribution is a conjugate prior, $\lambda\sim \text{Gamma}(\alpha, \beta)$, but I can not find a parameterization appropriate to the prior information, with $\text{median}\simeq 100,000$, $P(Y&amp;lt;10,000)\simeq 0)$ and nonzero density above $10,000$. Once you determine the model to use, it would be worth asking a separate question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Case 2&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you assume that failure is caused either by defect or by wear, you could consider the Weibull and Generalized Exponential distributions (Gupta and Kunda 1999, 2001, 2007). The field of survival analysis provides many options, but as whuber points out, $n=4$ is insufficient to justify a model other than, perhaps, the single parameter exponential.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://home.iitk.ac.in/~kundu/paper47.pdf&quot; rel=&quot;nofollow&quot;&gt;Gupta, R. D. and Kundu, D. (1999). Generalized exponential distributions&quot;, Australian and New Zealand Journal of Statistics, vol. 41, 173 - 188.&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://home.iitk.ac.in/~kundu/paper62.pdf&quot; rel=&quot;nofollow&quot;&gt;Gupta, R. D. and Kundu, D. (2001), Generalized exponential distributions: different methods of estimation&quot;, Journal of Statistical Computation and Simulation. vol. 69, 315-338&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://home.iitk.ac.in/~kundu/paper113.pdf&quot; rel=&quot;nofollow&quot;&gt;Gupta, R. D. and Kundu, D. 2007. Generalized Exponential Distribution: existing results and some recent developments. Journal of Statistical Planning and Inference&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1381" LastEditorUserId="1381" LastEditorDisplayName="" LastEditDate="2011-01-12T16:33:25.283" LastActivityDate="2011-01-12T16:33:25.283" CommentCount="10" />
  <row Id="6187" PostTypeId="1" AcceptedAnswerId="6188" CreationDate="2011-01-12T06:05:54.813" Score="2" ViewCount="251" Body="&lt;p&gt;Still learning basic functions in R,&#xA;The subset function seems to only filter based a condition based on single column with or without multiple conditions?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I easily filter data from a dataframe?&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;when you are provided with multiple conditions&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;When the condition needs to be applied across the available columns.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Example:&#xA;Given a data frame containing&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;name    D1      D2     D3      D4&#xA;julius  &quot;A&quot;     &quot;A&quot;    &quot;B&quot;     &quot;B&quot;&#xA;cate    &quot;D&quot;     &quot;E&quot;     &quot;A&quot;     &quot;C&quot;&#xA;karo    &quot;A&quot;     &quot;D&quot;     &quot;C&quot;     &quot;E&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;say I want to filter this dataframe so that only names where any of the D1 to D4 is an 'E'&#xA;then I should have,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;name    D1      D2     D3      D4&#xA;cate    &quot;D&quot;     &quot;E&quot;     &quot;A&quot;     &quot;C&quot;&#xA;karo    &quot;A&quot;     &quot;D&quot;     &quot;C&quot;     &quot;E&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Say that the D1 can be a big list of columns, how or what is the recommended approach to perform this filter?&#xA;Thank you&lt;/p&gt;&#xA;" OwnerUserId="201" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-01-12T08:02:30.840" LastActivityDate="2011-01-12T12:36:43.463" Title="Filtering a dataframe" Tags="&lt;r&gt;&lt;dataframe&gt;" AnswerCount="1" />
  <row Id="6188" PostTypeId="2" ParentId="6187" CreationDate="2011-01-12T07:35:13.187" Score="10" ViewCount="" Body="&lt;p&gt;If you want to combine several filters in &lt;strong&gt;subset&lt;/strong&gt; function use logical operators:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; subset(data, D1 == &quot;E&quot; | D2 == &quot;E&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;will select those rows for which either column D1 or column D2 has value &quot;E&quot;. Look at the help pages for available logical operators:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; &amp;gt; ?&quot;|&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For your second question what you need is to filter the rows. This can be achieved in the following way &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; collist &amp;lt;- c(&quot;D1&quot;,&quot;D2&quot;,&quot;D3&quot;,&quot;D4&quot;)&#xA; sel &amp;lt;- apply(data[,collist],1,function(row) &quot;E&quot; %in% row)&#xA; data[sel,]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The first argument to &lt;strong&gt;apply&lt;/strong&gt; suplies the columns on which we need to filter. The second argument is 1, meaning that we are looping through rows of the data. The third argument is unnamed one-line function which returns &lt;strong&gt;TRUE&lt;/strong&gt; if &quot;E&quot; is present in the row and &lt;strong&gt;FALSE&lt;/strong&gt; if the &quot;E&quot; is not present. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The result of the apply function will be logical vector &lt;strong&gt;sel&lt;/strong&gt;, which has the length the same as number of rows in data. We then use this vector to select the necessary rows.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The same can be achieved with grep:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sel &amp;lt;- apply(data[,collist],1,function(row) length(grep(&quot;E&quot;,row))&amp;gt;0)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;in R &lt;strong&gt;grep&lt;/strong&gt; with default arguments returns the numbers of elements in the supplied vector which have the matching pattern. &lt;/p&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-01-12T12:36:43.463" LastActivityDate="2011-01-12T12:36:43.463" CommentCount="6" />
  <row Id="6189" PostTypeId="1" CreationDate="2011-01-12T07:59:44.047" Score="5" ViewCount="527" Body="&lt;p&gt;I have daily measurements of nitrogen dioxide for one year (365 days) and the &lt;strong&gt;interquartile&lt;/strong&gt; (IQR) is 24 microgram per cubic meter. What does &quot;24&quot; mean in this context, apart from the definition of IQR which is the difference between &lt;strong&gt;the 25th and 75th percentile&lt;/strong&gt;? How would you explain this figure to a journalist, for example?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks   &lt;/p&gt;&#xA;" OwnerUserId="2742" LastActivityDate="2011-01-16T14:11:18.313" Title="What is the interpretation of interquartile range?" Tags="&lt;summary-statistics&gt;" AnswerCount="4" CommentCount="1" />
  <row Id="6190" PostTypeId="2" ParentId="6189" CreationDate="2011-01-12T08:17:42.690" Score="10" ViewCount="" Body="&lt;p&gt;From definition, this defines the range witch holds 75-25=50 per cent of all measured values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strike&gt;: (median-24/2,median+24/2). Median should be written somewhere near this IQR.&lt;/strike&gt;&lt;br&gt;&#xA;The above was false of course, it seems I was still sleeping when writing this;  sorry for confusion. It is true that IQR is width of a range which holds 50% of data, but it is not centered in median -- one needs to know both Q1 and Q3 to localize this range.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general IQR can be seen as a nonparametric (=when we don't assume that the distribution is Gaussian) equivalent to standard deviation -- both measure spread of the data. (Equivalent not equal, for SD, (mean-$\sigma$,mean+$\sigma$) holds 68.2% of perfectly normally distributed data).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: As for example, this is how it looks on normal data; red lines show $\pm 1\sigma$, the range showed by the box on boxplot shows IRQ, the histogram shows the data itself:&#xA;&lt;img src=&quot;http://i.imgur.com/0ivsM.png&quot; alt=&quot;alt text&quot;&gt;&lt;br&gt;&#xA;you can see both show spread pretty good; $\pm 1\sigma$ range holds 68.3% of data (as expected). Now for non-normal data&lt;br&gt;&#xA;&lt;img src=&quot;http://i.imgur.com/q0MV5.png&quot; alt=&quot;alt text&quot;&gt;&lt;br&gt;&#xA;the sd spread is widened due to long, asymmetric tail and $\pm 1\sigma$ holds 90.5% of data! (IRQ holds 50% in both cases by definition)&lt;/p&gt;&#xA;" OwnerUserId="88" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-01-12T11:23:15.383" LastActivityDate="2011-01-12T11:23:15.383" CommentCount="1" />
  <row Id="6191" PostTypeId="2" ParentId="6176" CreationDate="2011-01-12T08:42:00.717" Score="4" ViewCount="" Body="&lt;p&gt;After some research, I ended up buying this when I thought I needed to know something about measure-theoretic probability:  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://books.google.co.uk/books?id=PYNAgzDffDMC&quot; rel=&quot;nofollow&quot;&gt;Jeffrey Rosenthal. &lt;em&gt;A First Look at Rigorous Probability Theory&lt;/em&gt;. World Scientific 2007. ISBN 9789812703712.&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I haven't read much of it, however, as my personal experience is in accord with &lt;a href=&quot;http://www.stat.columbia.edu/~cook/movabletype/archives/2009/05/the_benefits_of.html&quot; rel=&quot;nofollow&quot;&gt;Stephen Senn's quip&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-01-12T08:42:00.717" CommentCount="3" />
  <row Id="6192" PostTypeId="2" ParentId="6189" CreationDate="2011-01-12T08:54:45.957" Score="5" ViewCount="" Body="&lt;p&gt;The interquartile range is an &lt;strong&gt;interval&lt;/strong&gt;, not a scalar. You should always report &lt;em&gt;both&lt;/em&gt; numbers, not just the difference between them. You can then explain it by saying that half the sample readings were between these two values, a quarter were smaller than the lower quartile, and a quarter higher than the upper quartile.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-01-12T08:54:45.957" CommentCount="6" />
  <row Id="6193" PostTypeId="2" ParentId="6176" CreationDate="2011-01-12T12:01:00.673" Score="1" ViewCount="" Body="&lt;p&gt;Personally, I've found Kolmogorov's original &lt;em&gt;&lt;a href=&quot;http://clrc.rhul.ac.uk/resources/fop/Theory%20of%20Probability%20%28small%29.pdf&quot; rel=&quot;nofollow&quot;&gt;Foundations of the Theory of Probability&lt;/a&gt;&lt;/em&gt; to be fairly readable, at least compared to most measure theory texts. Although it obviously doesn't contain any later work, it does give you an idea of most of the important concepts (sets of measure zero, conditional expectation, etc.). It is also mercifully brief, at only 84 pages. &lt;/p&gt;&#xA;" OwnerUserId="495" LastActivityDate="2011-01-12T12:01:00.673" CommentCount="1" />
  <row Id="6195" PostTypeId="1" AcceptedAnswerId="6198" CreationDate="2011-01-12T12:44:59.007" Score="6" ViewCount="301" Body="&lt;p&gt;Hi everyone! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I asked &lt;a href=&quot;http://math.stackexchange.com/questions/17200/how-to-model-prices&quot;&gt;this question&lt;/a&gt; on the matemathics stackexchange site and was recommended to ask here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm working on a hobby project and would need some help with the following problem.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;A bit of context&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Let's say there is a collection of items with a description of features and a price. Imagine a list of cars and prices. All cars have a list of features, e.g. engine size, color, horse power, model, year etc. For each make, something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Ford:&#xA;V8, green, manual, 200hp, 2007, $200&#xA;V6, red, automatic, 140hp, 2010, $300&#xA;V6, blue, manual, 140hp, 2005, $100&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Going even further, the list of cars with prices is published with some time-interval which means we have access to historical price data. Might not always include exactly the same cars.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Problem&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;I would like to understand how to model prices for any car based on this base information, most importantly cars not in the initial list.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Ford, v6, red, automatic, 130hp, 2009&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For the above car, it's almost the same as one in the list, just slightly different in horse power and year. To price this, what is needed?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;What I'm looking for is something practical and simple, but I would also like to hear about more complex approaches how to model something like this.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;What I've tried&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Here is what I've been experimenting with so far:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) using historical data to lookup car X. If not found, no price. This  is of course very limited and one can only use this in combination with some time decay to alter prices for known cars over time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) using a car feature weighting scheme together with a priced sample car. Basically that there is a base price and features just alter that with some factor. Based on this any car's price is derived.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first proved to be not enough and the second proved to not be always correct and I might not have had the best approach to using the weights. This also seems to be a bit heavy on  maintaining weights, so that's why I thought maybe there is some way to use the historical data as statistics in some way to get weights or to get something else. I just don't know where to start.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Other important aspects&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;integrate into some software project I have. Either by using existing libraries or writing algorithm myself.&lt;/li&gt;&#xA;&lt;li&gt;fast recalculation when new historical data comes in.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Any suggestions how a problem like this could be approached?&lt;/strong&gt; All ideas are more than welcome. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks a lot in advance and looking forward to reading your suggestions!&lt;/p&gt;&#xA;" OwnerUserId="2746" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-01-13T07:54:04.290" LastActivityDate="2011-01-13T18:40:03.760" Title="How to model prices?" Tags="&lt;regression&gt;&lt;forecasting&gt;&lt;econometrics&gt;" AnswerCount="5" FavoriteCount="1" />
  <row Id="6196" PostTypeId="2" ParentId="6195" CreationDate="2011-01-12T15:00:00.930" Score="1" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;What I'm looking for is something practical and simple, but I would also like to hear about more complex approaches how to model something like this.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;After some sort of a discussion, here is my complete view of the things&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The problem&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Aim:&lt;/strong&gt; to understand how to price the cars in a better way&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Context:&lt;/strong&gt; in their decision process people solve several questions: do I need a car, if I do, what attributes I prefer most (including the price, because, being rational, I would like to have a car with best quality/price ratio), compare the number of attributes between different cars and choosing valuing them &lt;strong&gt;jointly&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the seller position, I would like to set the price as high as possible, and sell the car as quickly as possible. So if I set the price too high and am waiting for months it could be considered as &lt;em&gt;not demanded&lt;/em&gt; on the market and marked with 0 comparing to very demanded attribute sets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Observations:&lt;/strong&gt; real deals that relates the attributes of a particular car with the price set within the bargaining process (regarding the previous remark it is important to know how long it take to set the deal). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Pros:&lt;/strong&gt; you do observe the things that were actually bought on the market, so you are not guessing if there exist a person with high enough reservation price that wants to buy&#xA;a particular car&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Cons:&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;your assumption is that market is efficient, meaning the prices you observe are close to equilibrium&lt;/li&gt;&#xA;&lt;li&gt;you ignore the variants of car attributes that were not purchased or took too long to set the deal, meaning your insights &lt;em&gt;are biased&lt;/em&gt;, so you actually do work with latent variable models&lt;/li&gt;&#xA;&lt;li&gt;Observing the data for a long time you need to deflate them, though the inclusion of the car age partly compensates this.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Solution methods&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first one, as suggested by whuber, is the &lt;strong&gt;classical least squares regression model&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Pros:&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;indeed the simplest solution as it is the work-horse of econometrics&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Cons:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;ignores that you do observe the things &lt;strong&gt;incompletely&lt;/strong&gt; (&lt;em&gt;latent variables&lt;/em&gt;)&lt;/li&gt;&#xA;&lt;li&gt;acts as the regressors are independent one of the other, so the basic model ignores the fact that you may like &lt;em&gt;blue Ford&lt;/em&gt; differently from &lt;em&gt;blue Mercedes&lt;/em&gt;, but it is not the sum of marginal influence that comes from blue and Ford&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In case of classical regression, since you are not limited in the degrees of freedom, to try also different interaction terms. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore more complicated solution would be either &lt;a href=&quot;http://en.wikipedia.org/wiki/Tobit_model&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;tobit&lt;/strong&gt;&lt;/a&gt; or &lt;strong&gt;Heckman model&lt;/strong&gt;, you may want to consult A.C. Cameron and P.K. Trivedi &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0521848059&quot; rel=&quot;nofollow&quot;&gt;Microeconometrics: methods and applications&lt;/a&gt; for more details on core methods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Pros:&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;you do separate the fact that people may not like some sets of attributes at all, or some set of attributes has a small probability to be bought &lt;em&gt;from&lt;/em&gt; the actual price setting&lt;/li&gt;&#xA;&lt;li&gt;your results are not biased (or at least less than in the first case)&lt;/li&gt;&#xA;&lt;li&gt;in case of Heckman you separate the reasons that motivates to buy the particular car from the pricing decision of how much I would like to pay for this car: the first one is influenced by individual preferences, the second one by budget constraint&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Cons:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Both models are more &lt;em&gt;data greedy&lt;/em&gt;, i.e. we need to observe either the time length between the ask and bid to equalize (if it is fairly short put 1, else 0), or to observe the sets that were ignored by the market &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;And, finally, if you simply interested in how price influences the probability to be bought you may work with some kind of &lt;strong&gt;logit&lt;/strong&gt; models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We agreed, that &lt;strong&gt;conjoint analysis&lt;/strong&gt; is not suitable here, because you do have different context and observations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good luck.&lt;/p&gt;&#xA;" OwnerUserId="2645" LastEditorUserId="2645" LastEditorDisplayName="" LastEditDate="2011-01-13T09:31:44.543" LastActivityDate="2011-01-13T09:31:44.543" CommentCount="10" />
  <row Id="6197" PostTypeId="2" ParentId="6189" CreationDate="2011-01-12T15:32:09.447" Score="6" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;This is a simple question asking for a simple answer.&lt;/strong&gt;  Here is a list of statements, starting with the most basic, and proceeding with more precise qualifications.  &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;The IQR is the spread of the middle half of the data.&lt;/strong&gt;&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;&lt;em&gt;Without making assumptions about how the data are distributed&lt;/em&gt;, the IQR quantifies the amount by which individual values typically vary.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;The IQR is related to the well-known &quot;standard deviation&quot; (SD): when the data follow a &quot;bell curve,&quot; the IQR is about 35% greater than the SD.  (Equivalently, the SD is about three-quarters of the IQR.)&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;As a rule of thumb, data values that deviate from the middle value by more than twice the IQR deserve individual attention.  They are called &quot;outliers.&quot;  Data values that deviate from the middle value by more than 3.5 times the IQR are usually scrutinized closely.  They are sometimes called &quot;far outliers.&quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-01-12T15:32:09.447" />
  <row Id="6198" PostTypeId="2" ParentId="6195" CreationDate="2011-01-12T15:58:50.810" Score="8" ViewCount="" Body="&lt;p&gt;&quot;Practical&quot; and &quot;simple&quot; suggest &lt;strong&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Least_squares&quot; rel=&quot;nofollow&quot;&gt;least squares regression&lt;/a&gt;.&lt;/strong&gt;  It's easy to set up, easy to do with lots of software (R, Excel, Mathematica, &lt;em&gt;any&lt;/em&gt; statistics package), easy to interpret, and can be extended in many ways depending on how accurate you want to be and how hard you're willing to work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This approach is essentially your &quot;weighting scheme&quot; (2), but it finds the weights easily, guarantees as much accuracy as possible, and is easy and fast to update.  There are &lt;em&gt;loads&lt;/em&gt; of libraries to perform least squares calculations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It will help to include not only the variables you listed--engine type, power, etc--but also &lt;em&gt;age&lt;/em&gt; of car.  Furthermore, make sure to adjust prices for inflation.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-01-12T15:58:50.810" CommentCount="3" />
  <row Id="6201" PostTypeId="2" ParentId="6195" CreationDate="2011-01-12T20:01:09.850" Score="3" ViewCount="" Body="&lt;p&gt;I agree with @whuber, that linear regression is a way to go, but care must be taken when interpreting results. The problem is that in economics the price is always related to demand. If demand goes up, prices go up, if demand goes down, prices go down. So the price is determined by demand and in return demand is determined by price. So if we model price as a regression from some attributes without the demand there is a real danger that the regression estimates will be wrong due to &lt;a href=&quot;http://en.wikipedia.org/wiki/Omitted-variable_bias&quot; rel=&quot;nofollow&quot;&gt;omitted-variable bias&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-01-12T20:01:09.850" CommentCount="7" />
  <row Id="6202" PostTypeId="2" ParentId="6181" CreationDate="2011-01-12T20:18:55.417" Score="1" ViewCount="" Body="&lt;p&gt;r can indeed be negative - if two variables are negatively related.  r^2 can only be between 0 and 1, for the simple reason that it is the square of a real number.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if we correlated income and time spent in jail throughout life, I would guess we would get a negative correlation (I haven't done this, I'm just guessing).  &lt;/p&gt;&#xA;" OwnerUserId="686" LastActivityDate="2011-01-12T20:18:55.417" CommentCount="2" />
  <row Id="6203" PostTypeId="2" ParentId="6195" CreationDate="2011-01-12T21:05:47.813" Score="1" ViewCount="" Body="&lt;p&gt;Besides what have been said, and not really quite different from some of the suggestions already made, you might want to have a look at the vast literature on &lt;a href=&quot;http://en.wikipedia.org/wiki/Hedonic_regression&quot; rel=&quot;nofollow&quot;&gt;hedonic pricing models&lt;/a&gt;. What it boils down to is a regression model trying to explain the price of a composite good as a function of its attributes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This would allow you to price a car knowing its attributes (horse power, size, brand, etc.), even if an exactly similar mix of attributes is not present in your sample. It is a very popular approach for valuation of essentially non replicable assets --like real state properties. If you Google for &quot;hedonic models&quot; you will find many references and examples.&lt;/p&gt;&#xA;" OwnerUserId="892" LastActivityDate="2011-01-12T21:05:47.813" CommentCount="1" />
  <row Id="6204" PostTypeId="1" AcceptedAnswerId="6210" CreationDate="2011-01-13T00:26:45.187" Score="2" ViewCount="309" Body="&lt;p&gt;How can I increase the space between the bars in a bar plot in &lt;a href=&quot;http://had.co.nz/ggplot2/&quot; rel=&quot;nofollow&quot;&gt;ggplot2&lt;/a&gt; ?&lt;/p&gt;&#xA;" OwnerUserId="2725" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2011-01-13T17:46:20.420" LastActivityDate="2011-01-13T17:46:20.420" Title="How to increase the space between the bars in a bar plot in ggplot2?" Tags="&lt;r&gt;&lt;ggplot2&gt;&lt;barplot&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="1" />
  <row Id="6206" PostTypeId="1" AcceptedAnswerId="6207" CreationDate="2011-01-13T01:43:15.577" Score="3" ViewCount="285" Body="&lt;p&gt;I made a logistic regression model using glm in R.  I have two independent variables.  How can I plot the decision boundary of my model in the scatter plot of the two variables.  For example, how can I plot a figure like:&#xA;&lt;a href=&quot;http://onlinecourses.science.psu.edu/stat557/node/55&quot; rel=&quot;nofollow&quot;&gt;http://onlinecourses.science.psu.edu/stat557/node/55&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="2755" LastActivityDate="2011-01-13T02:51:15.190" Title="How to plot decision boundary in R for logistic regression model?" Tags="&lt;r&gt;&lt;logistic-regression&gt;" AnswerCount="1" />
  <row Id="6207" PostTypeId="2" ParentId="6206" CreationDate="2011-01-13T02:44:44.073" Score="5" ViewCount="" Body="&lt;pre&gt;&lt;code&gt;set.seed(1234)&#xA;&#xA;x1 &amp;lt;- rnorm(20, 1, 2)&#xA;x2 &amp;lt;- rnorm(20)&#xA;&#xA;y &amp;lt;- sign(-1 - 2 * x1 + 4 * x2 )&#xA;&#xA;y[ y == -1] &amp;lt;- 0&#xA;&#xA;df &amp;lt;- cbind.data.frame( y, x1, x2)&#xA;&#xA;mdl &amp;lt;- glm( y ~ . , data = df , family=binomial)&#xA;&#xA;slope &amp;lt;- coef(mdl)[2]/(-coef(mdl)[3])&#xA;intercept &amp;lt;- coef(mdl)[1]/(-coef(mdl)[3]) &#xA;&#xA;library(lattice)&#xA;xyplot( x2 ~ x1 , data = df, groups = y,&#xA;   panel=function(...){&#xA;       panel.xyplot(...)&#xA;       panel.abline(intercept , slope)&#xA;       panel.grid(...)&#xA;       })&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/Pp8xK.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I must remark that perfect separation occurs here, therefore the &lt;code&gt;glm&lt;/code&gt; function gives you a warning. But that is not important here as the purpose is to illustrate how to draw the linear boundary and the observations colored according to their covariates.&lt;/p&gt;&#xA;" OwnerUserId="1307" LastEditorUserId="1307" LastEditorDisplayName="" LastEditDate="2011-01-13T02:51:15.190" LastActivityDate="2011-01-13T02:51:15.190" CommentCount="4" />
  <row Id="6208" PostTypeId="1" AcceptedAnswerId="6211" CreationDate="2011-01-13T04:06:59.613" Score="7" ViewCount="338" Body="&lt;p&gt;I developed the ez package for R as a means to help folks transition from stats packages like SPSS to R. This is (hopefully) achieved by simplifying the specification of various flavours of ANOVA, and providing SPSS-like output (including effect sizes and assumption tests), among other features. The &lt;code&gt;ezANOVA()&lt;/code&gt; function mostly serves as a wrapper to &lt;code&gt;car::Anova()&lt;/code&gt;, but the current version of &lt;code&gt;ezANOVA()&lt;/code&gt; implements only type-II sums of squares, whereas &lt;code&gt;car::Anova()&lt;/code&gt; permits specification of either type-II or -III sums of squares. As I possibly should have expected, several users have requested that I provide an argument in &lt;code&gt;ezANOVA()&lt;/code&gt; that lets the user request type-II or type-III. I have been reticent to do so and outline my reasoning below, but I would appreciate the community's input on my or any other reasoning that bears on the issue.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Reasons for &lt;em&gt;not&lt;/em&gt; including a &quot;SS_type&quot; argument in &lt;code&gt;ezANOVA()&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The difference between type I, II, and III sum squares only crops up when data are unbalanced, in which case I'd say that more benefit is derived from ameliorating imbalance by further data collection than fiddling with the ANOVA computation.&lt;/li&gt;&#xA;&lt;li&gt;The difference between type II and III applies to lower-order effects that are qualified by higher-order effects, in which case I consider the lower-order effects scientifically uninteresting. (But see below for possible complication of the argument)&lt;/li&gt;&#xA;&lt;li&gt;For those rare circumstances when (1) and (2) don't apply (when further data collection is impossible and the researcher has a valid scientific interest in a qualified main effect that I can't currently imagine), one can relatively easily modify the &lt;code&gt;ezANOVA()&lt;/code&gt; source or employ &lt;code&gt;car::Anova()&lt;/code&gt; itself to achieve type III tests. In this way, I see the extra effort/understanding required to obtain type III tests as a means by which I can ensure that only those that really know what they're doing go that route.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Now, the most recent type-III requestor pointed out that argument (2) is undermined by consideration of circumstances where extant but &quot;non-significant&quot; higher-order effects can bias computation of sums of squares for lower-order effects. In such cases it's imaginable that a researcher would look to the higher-order effect, and seeing that it is &quot;non-significant&quot;, turn to attempting interpretation of the lower-order effects that, unbeknownst to the researcher, have been compromised. My initial reaction is that this is not a problem with sums of squares, but with p-values and the tradition of null hypothesis testing. I suspect that a more explicit measure of evidence, such as the likelihood ratio, might be more likely to yield a less ambiguous picture of the models supported consistent with the data. However, I haven't done much thinking on the consequence of unbalanced data for the computation of likelihood ratios (which indeed involve sums of squares), so I'll have to give this some further thought.&lt;/p&gt;&#xA;" OwnerUserId="364" LastEditorUserId="364" LastEditorDisplayName="" LastEditDate="2011-01-13T04:25:43.920" LastActivityDate="2011-02-09T23:41:42.233" Title="Should I include an argument to request type-III sums of squares in ezANOVA?" Tags="&lt;r&gt;&lt;anova&gt;&lt;sums-of-squares&gt;" AnswerCount="3" CommentCount="2" FavoriteCount="3" />
  <row Id="6209" PostTypeId="2" ParentId="1525" CreationDate="2011-01-13T04:13:48.283" Score="0" ViewCount="" Body="&lt;p&gt;Proportion and probability, both are calculated from the total but the value of proportion is certain while that of probability is no0t certain..&lt;/p&gt;&#xA;" OwnerUserId="2757" LastActivityDate="2011-01-13T04:13:48.283" />
  <row Id="6210" PostTypeId="2" ParentId="6204" CreationDate="2011-01-13T08:16:00.253" Score="9" ViewCount="" Body="&lt;p&gt;You can always play with the &lt;code&gt;width&lt;/code&gt; parameter, as shown below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df &amp;lt;- data.frame(x=factor(LETTERS[1:4]), y=sample(1:100, 4))&#xA;library(ggplot2)&#xA;ggplot(data=df, aes(x=x, y=y, width=.5)) + &#xA;  geom_bar(stat=&quot;identity&quot;, position=&quot;identity&quot;) +&#xA;  opts(title=&quot;width = .5&quot;) + labs(x=&quot;&quot;, y=&quot;&quot;) +&#xA;  theme_bw()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Compare with the following other settings for &lt;code&gt;width&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/KsHJm.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far, so good. Now, suppose we have two factors. In case you would like to play with evenly spaced juxtaposed bars (like when using &lt;code&gt;space&lt;/code&gt; together with &lt;code&gt;beside=TRUE&lt;/code&gt; in &lt;code&gt;barplot()&lt;/code&gt;), it's not so easy using &lt;code&gt;geom_bar(position=&quot;dodge&quot;)&lt;/code&gt;: you can change bar width, but not add space in between adjacent bars (and I didn't find a convenient solution on Google). I ended up with something like that:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df &amp;lt;- data.frame(g=gl(2, 1, labels=letters[1:2]), y=sample(1:100, 4))&#xA;x.seq &amp;lt;- c(1,2,4,5)&#xA;ggplot(data=transform(df, x=x.seq), aes(x=x, y=y, width=.85)) +&#xA;  geom_bar(stat=&quot;identity&quot;, aes(fill=g)) + labs(x=&quot;&quot;, y=&quot;&quot;) + &#xA;  scale_x_discrete(breaks = NA) + &#xA;  geom_text(aes(x=c(sum(x.seq[1:2])/2, sum(x.seq[3:4])/2), y=0, &#xA;                label=c(&quot;X&quot;,&quot;Y&quot;)), vjust=1.2, size=8)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The vector used for the $x$-axis is &quot;injected&quot; in the data.frame, so that so you change the outer spacing if you want, while &lt;code&gt;width&lt;/code&gt; allows to control for inner spacing. Labels for the $x$-axis might be enhanced by using &lt;code&gt;scale_x_discrete()&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/BoUeM.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-01-13T14:12:40.397" LastActivityDate="2011-01-13T14:12:40.397" />
  <row Id="6211" PostTypeId="2" ParentId="6208" CreationDate="2011-01-13T11:14:47.550" Score="3" ViewCount="" Body="&lt;p&gt;Just to amplify - I am the most recent requestor, I believe.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In specific comment on Mike's points:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;It's clearly true that the I/II/III difference only applies with correlated predictors (of which unbalanced designs are the most common example, certainly in factorial ANOVA) - but this seems to me to be an argument that dismisses the analysis of the unbalanced situation (and hence any Type I/II/III debate). It may be imperfect, but that's the way things happen (and in many contexts the costs of further data collection outweigh the statistical problem, caveats notwithstanding).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;This is completely fair and represents the meat of most of the &quot;II versus III, favouring II&quot; arguments I've come across. The best summary I've encountered is Langsrud (2003) &quot;ANOVA for unbalanced data: Use Type II instead of Type III sums of squares&quot;, Statistics and Computing 13: 163-167 (I have a PDF if the original is hard to find). He argues (taking the two-factor case as the basic example) that if there's an interaction, there's an interaction, so consideration of main effects is usually meaningless (an obviously fair point) - and if there's no interaction, the Type II analysis of main effects is more powerful than the Type III (undoubtedly), so you should always go with Type II. I've seen other arguments (e.g. Venables, Fox) that emphasize the meaning (or lack of) of considering hypotheses about main effects in the presence of interactions, and/or/equivalently suggesting that the Type III assumptions about the null hypothesis are often not sensible (e.g. Langsrud).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;And I agree with this: if you have an interaction but have some question about the main effect as well, then you're probably into do-it-yourself territory.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Clearly there are those who just want Type III because SPSS does it, or some other reference to statistical Higher Authority. I am not wholly against this view, if it comes down to a choice of a lot of people sticking with SPSS (which I have some things against, namely time, money, and licence expiry conditions) and Type III SS, or a lot of people shifting to R and Type III SS. However, this argument is clearly a lame one statistically.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the argument that I found rather more substantial in favour of Type III is that made independently by Myers &amp;amp; Well (2003, &quot;Research Design and Statistical Analysis&quot;, pp. 323, 626-629) and Maxwell &amp;amp; Delaney (2004, &quot;Designing Experiments and Analyzing Data: A Model Comparison Perspective&quot;, pp. 324-328, 332-335). That is as follows:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;if there's an interaction, all methods give the same result for the interaction sum of squares&lt;/li&gt;&#xA;&lt;li&gt;Type II assumes that there's no interaction for its test of main effects; type III doesn't&lt;/li&gt;&#xA;&lt;li&gt;Some (e.g. Langsrud) argue that if the interaction is not significant, then you're justified in assuming that there isn't one, and looking at the (more powerful) Type II main effects&lt;/li&gt;&#xA;&lt;li&gt;But if the test of the interaction is underpowered, yet there is an interaction, the interaction may come out &quot;non-significant&quot; yet still lead to a violation of the assumptions of the Type II main effects test, biasing those tests to be too liberal.&lt;/li&gt;&#xA;&lt;li&gt;Myers &amp;amp; Well cite Appelbaum/Cramer as the primary proponents of the Type II approach, and go on [p323]: &quot;... More conservative criteria for nonsignificance of the interaction could be used, such as requiring that the interaction not be significant at the .25 level, but there is insufficient understanding of the consequences of even this approach. As a general rule, Type II sums of sqaures should not be calculated unless there is strong a priori reason to assume no interaction effects, and a clearly nonsignificant interaction sum of squares.&quot; They cite [p629] Overall, Lee &amp;amp; Hornick 1981 as a demonstration that interactions that do not approach significance can bias tests of main effects. Maxwell &amp;amp; Delaney [p334] advocate the Type II approach if the population interaction is zero, for power, and the Type III approach if it isn't [for the interpretability of means derived from this approach]. They too advocate using Type III in the real-life situation (when you're making inferences about the presence of the interaction from the data) because of the problem of making a type 2 [underpowered] error in the interaction test and thus accidentally violating the assumptions of the Type II SS approach; they then make similar further points to Myers &amp;amp; Well, and note the long debate on this issue!&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;So my interpretation (and I'm no expert!) is that there's plenty of Higher Statistical Authority on both sides of the argument; that the usual arguments put forward aren't about the usual situation that would give rise to problems (that situation being the common one of interpreting main effects with a non-significant interaction); and that there are fair reasons to be concerned about the Type II approach in that situation (and it comes down to a power versus potential over-liberalism thing).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For me, that's enough to wish for the Type III option in ezANOVA, as well as Type II, because (for my money) it's a superb interface to R's ANOVA systems. R is some way from being easy to use for novices, in my view, and the &quot;ez&quot; package, with ezANOVA and the rather lovely effect plotting functions, goes a long way towards making R accessible to a more general research audience. Some of my thoughts-in-progress (and a nasty hack for ezANOVA) are at &lt;a href=&quot;http://www.psychol.cam.ac.uk/statistics/R/anova.html&quot; rel=&quot;nofollow&quot;&gt;http://www.psychol.cam.ac.uk/statistics/R/anova.html&lt;/a&gt; .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would be interested to hear everyone's thoughts!&lt;/p&gt;&#xA;" OwnerUserId="2761" LastActivityDate="2011-01-13T11:14:47.550" CommentCount="3" />
  <row Id="6212" PostTypeId="2" ParentId="6208" CreationDate="2011-01-13T11:33:00.883" Score="4" ViewCount="" Body="&lt;p&gt;Caveat: a purely non-statistical answer. I prefer to work with one function (or at least one package) when doing the same type of analysis (e.g., ANOVA). Up to now, I consistently use &lt;code&gt;Anova()&lt;/code&gt; since I prefer its syntax for specifying models with repeated measures  - compared to &lt;code&gt;aov()&lt;/code&gt;, and lose little (SS type I) with non-repeated measures. &lt;code&gt;ezANOVA()&lt;/code&gt; is nice for the added benefit of effect sizes. But What I don't particularly like is having to deal with 3 different functions to do essentially the same type of analysis, just because one of them implements feature X (but not Y), and the other one Y (but not X).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For ANOVA, I can choose between &lt;code&gt;oneway()&lt;/code&gt;, &lt;code&gt;lm()&lt;/code&gt;, &lt;code&gt;aov()&lt;/code&gt;, &lt;code&gt;Anova()&lt;/code&gt;, &lt;code&gt;ezANOVA()&lt;/code&gt;, and probably others. When teaching R, it already is a pain to explain the different options, how they relate to each other (&lt;code&gt;aov()&lt;/code&gt; is a wrapper for &lt;code&gt;lm()&lt;/code&gt;), and which function does what: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;oneway()&lt;/code&gt; only for single factor designs but with option &lt;code&gt;var.equal=FALSE&lt;/code&gt;. No such option in &lt;code&gt;aov()&lt;/code&gt; and others, but those functions also for multifactorial designs.&lt;/li&gt;&#xA;&lt;li&gt;syntax for repeated measures a bit complicated in &lt;code&gt;aov()&lt;/code&gt;, better in &lt;code&gt;Anova()&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;convenient SS type I only in &lt;code&gt;aov()&lt;/code&gt;, not in &lt;code&gt;Anova()&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;convenient SS type II and III  only in &lt;code&gt;Anova()&lt;/code&gt;, not in &lt;code&gt;aov()&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;convenient effect size measures in &lt;code&gt;ezANOVA()&lt;/code&gt;, not in others&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;It would be neat to only have to teach one function with one consistent syntax that does it all. Without convenient SS type III, &lt;code&gt;ezANOVA()&lt;/code&gt; can't be that function for me because I know that students will be asked to use them at some point (&quot;just cross-check these results that John Doe got with SPSS&quot;). I feel it's better to have the option to make the choice oneself without having to learn yet another syntax for specifying models. The &quot;I know what's best for you&quot; attitude may have its merits, but can be over-protective.&lt;/p&gt;&#xA;" OwnerUserId="1909" LastEditorUserId="1909" LastEditorDisplayName="" LastEditDate="2011-01-13T12:55:49.043" LastActivityDate="2011-01-13T12:55:49.043" CommentCount="5" />
  <row Id="6213" PostTypeId="2" ParentId="6169" CreationDate="2011-01-13T11:56:13.593" Score="0" ViewCount="" Body="&lt;p&gt;I don't think there are any appropriate uses of stacked bar charts; grouped bar charts are better, but both are inferior to other plots, depending on what aspect of your data you want to emphasize, and how much data you have.&lt;/p&gt;&#xA;" OwnerUserId="686" LastActivityDate="2011-01-13T11:56:13.593" CommentCount="1" />
  <row Id="6214" PostTypeId="1" AcceptedAnswerId="6216" CreationDate="2011-01-13T12:07:18.603" Score="7" ViewCount="188" Body="&lt;p&gt;How should I define a model formula in R, when one (or more) exact linear restrictions binding the coefficients is available. As an example, say that you know that b1 = 2*b0 in a simple linear regression model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you!&lt;/p&gt;&#xA;" OwnerUserId="339" LastActivityDate="2011-01-13T13:19:49.620" Title="Fitting models in R where coefficients are subject to restriction(s)" Tags="&lt;r&gt;&lt;regression&gt;&lt;modeling&gt;" AnswerCount="1" FavoriteCount="2" />
  <row Id="6215" PostTypeId="2" ParentId="6189" CreationDate="2011-01-13T12:51:14.577" Score="1" ViewCount="" Body="&lt;p&gt;Roughly speaking, I would say to a journalist that I could declare the daily level of nitrogen dioxide being sure, after discarding the highest values and the lowest values, that in each one of one-half of the days in that year the observed value is not beyond a distance of IQR/2 from the declared level.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if your first quartile and third quartile are 100 and 124, you could say that daily level is 112 (average of 100 and 124) and assure your interlocutor that in half of the days the error you make isn't greater than 12. &lt;/p&gt;&#xA;" OwnerUserId="1219" LastEditorUserId="1219" LastEditorDisplayName="" LastEditDate="2011-01-16T14:11:18.313" LastActivityDate="2011-01-16T14:11:18.313" CommentCount="2" />
  <row Id="6216" PostTypeId="2" ParentId="6214" CreationDate="2011-01-13T12:53:38.177" Score="5" ViewCount="" Body="&lt;p&gt;Suppose your model is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ Y(t) = \beta_0 + \beta_1 \cdot X_1(t) + \beta_2 \cdot X_2(t) + \varepsilon(t)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and you are planning to restrict the coefficients, for instance like:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ \beta_1 = 2 \beta_2$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;inserting the restriction, rewriting the original regression model you will get&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ Y(t) = \beta_0 + 2 \beta_2 \cdot X_1(t) + \beta_2 \cdot X_2(t) + \varepsilon(t) $ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ Y(t) = \beta_0 + \beta_2 (2 \cdot X_1(t) + X_2(t)) + \varepsilon(t)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;introduce a new variable $Z(t) = 2 \cdot X_1(t) + X_2(t)$ and your model with restriction will be&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ Y(t) = \beta_0 + \beta_2 Z(t) + \varepsilon(t)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this way you can handle any exact restrictions, because the number of &lt;em&gt;equal signs&lt;/em&gt; reduces the number of unknown parameters by the same number.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Playing with R formulas you can do directly by I() function&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;lm(formula = Y ~ I(1 + 2*X1) + X2 + X3 - 1, data = &amp;lt;your data&amp;gt;) &#xA;lm(formula = Y ~ I(2*X1 + X2) + X3, data = &amp;lt;your data&amp;gt;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2645" LastEditorUserId="2645" LastEditorDisplayName="" LastEditDate="2011-01-13T13:19:49.620" LastActivityDate="2011-01-13T13:19:49.620" CommentCount="3" />
  <row Id="6217" PostTypeId="1" CreationDate="2011-01-13T13:03:43.850" Score="3" ViewCount="39" Body="&lt;p&gt;In their great book &quot;Wavelet methods for time series analysis&quot; (2006), Percival &amp;amp; Walden state on p. 83 that the first-round pyramid algorithm scaling filter coefficients&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\tilde{V}_{i,t}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;can be approximated by&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\frac{1}{N} \sum_{k=-\frac{N}{4}+1}^{\frac{N}{4}}{\chi_{k}e^{i 2 \pi t k/N}}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;arguing that $2^{1/2} \tilde{V}_{i,t}$ &quot;is formed by filtering [the time series] $\{X_{t}\}$ with the low-pass filter $\{g_l\}$ with nominal pass-band [-1/4, 1/4] ...&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why is it $\tilde{V}_{i,t}$ and not $2^{1/2} \tilde{V}_{i,t}$ which is approximated by the above expression?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My understanding is that the approximation is due to limiting the frequency domain of the inverse DFT according to the pass-band of the filter?&lt;/p&gt;&#xA;" OwnerUserId="2765" LastActivityDate="2011-01-13T13:03:43.850" Title="Wavelet analysis, scaling filter: where does the square root of 2 go to?" Tags="&lt;wavelet&gt;" FavoriteCount="1" />
  <row Id="6218" PostTypeId="1" CreationDate="2011-01-13T14:17:53.720" Score="2" ViewCount="122" Body="&lt;p&gt;I built a fixed effects model using time series cross sectional data.  The model was built using PROC MIXED in SAS.  There are 2 effect coded attributes in the model and inteactions. One dummy variable has 12 levels ($MO$), the other has 16 levels ($MSA$). My question has to do with interpreting and graphing the interaction effects.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The attributes are as follows:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The dependent variable is log transformed&lt;/li&gt;&#xA;&lt;li&gt;$D_1$ is log transformed and centered (subtracted the mean)&lt;/li&gt;&#xA;&lt;li&gt;$M_2$ is centered&lt;/li&gt;&#xA;&lt;li&gt;$W_3$ is centered&lt;/li&gt;&#xA;&lt;li&gt;$A_4$ is centered&lt;/li&gt;&#xA;&lt;li&gt;$MO$ is an effect coded categorical dummy variable&lt;/li&gt;&#xA;&lt;li&gt;$MSA$ is an effect coded categorical dummy variable&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;(A) I’d like to graph the interactions to try to understand the model better.  For the interaction concerning (the ln transformed) $D_1$ can I reduce the regression formula as one normally would or do I need to account for the fact that $D_1$ is ln transformed and $\beta_0$ represents the geometric mean?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The interaction term for $D_1\times MO_1$ would reduce to&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align*}&#xA;\ln(y) &amp;amp;= \beta_0 +  \beta_1 D_1 + \beta_5 MO_1 + \beta_{61} D_1\times MO_1 \\ &#xA;        &amp;amp; =(\beta_0 +  \beta_5) MO_1+ (\beta_1 + \beta_{61} MO_1) D_1 &#xA;\end{align*}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(B) Do I use &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\beta_0+\beta_1 D_1+\beta_5 MO_1+\beta_{61}D_1\times MO_1$$  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;or &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$(\beta_0 +  \beta_5) MO_1+ (\beta_1 + \beta_{61}MO_1)D_1$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;to interpret the interaction. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(C) How do I interpret the interaction?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(D) How do I interpret the coefficients of the appropriate equation?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(E) What is the formula used to determine the beta coefficient of the omitted dummy variable?&lt;/p&gt;&#xA;" OwnerUserId="2766" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-01-13T19:56:18.500" LastActivityDate="2011-01-13T19:56:18.500" Title="Interactions in Log Transformed DV" Tags="&lt;sas&gt;&lt;interaction&gt;" CommentCount="1" />
  <row Id="6219" PostTypeId="1" AcceptedAnswerId="6220" CreationDate="2011-01-13T14:51:20.863" Score="2" ViewCount="333" Body="&lt;p&gt;A physics application I'm using reports for a first order fit of the three points below as $11.388612x - 301.878$.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   x, y    &#xA;  35, 0&#xA;  430, 4861&#xA;  656, 7000&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It also shows a field labeled: &quot;RMS: 329.499&quot;  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How is that RMS calculated? I tried  RMSD as defined &lt;a href=&quot;http://en.wikipedia.org/wiki/Root_mean_square_deviation&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; but didn't get the same value.&lt;/p&gt;&#xA;" OwnerUserId="2767" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2011-01-13T15:57:40.513" LastActivityDate="2011-01-13T16:54:03.113" Title="What does RMS stand for?" Tags="&lt;regression&gt;&lt;curve-fitting&gt;" AnswerCount="3" CommentCount="3" />
  <row Id="6220" PostTypeId="2" ParentId="6219" CreationDate="2011-01-13T15:55:41.003" Score="2" ViewCount="" Body="&lt;p&gt;That's the root mean square error (RMSE) of the regression.&lt;br&gt;&#xA;$$RMSE = \sqrt{\frac{1}{n-k}\sum{(y_i-\hat{y_i})^2}},$$&lt;br&gt;&#xA;where $y_i$ is the observed and $\hat{y_i}$ the fitted value for observation $i$, $n$ is the number of observations, and $k$ is the number of parameters fitted (including the constant).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I just tried fitting a straight line by &lt;a href=&quot;http://en.wikipedia.org/wiki/Simple_linear_regression&quot; rel=&quot;nofollow&quot;&gt;simple linear regression&lt;/a&gt; in another statistics package and got an RMSE of 329.499751.&lt;/p&gt;&#xA;" OwnerUserId="449" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-01-13T16:54:03.113" LastActivityDate="2011-01-13T16:54:03.113" CommentCount="6" />
  <row Id="6221" PostTypeId="2" ParentId="6219" CreationDate="2011-01-13T15:55:47.193" Score="6" ViewCount="" Body="&lt;p&gt;RMS stands for the root mean square error. It's calculated in the following way.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;First we calculate the residuals: -96.72, 265.77, -169.05&lt;/li&gt;&#xA;&lt;li&gt;Next we calculate the squared &lt;a href=&quot;http://en.wikipedia.org/wiki/Errors_and_residuals_in_statistics&quot; rel=&quot;nofollow&quot;&gt;residuals&lt;/a&gt;: -96.72$^2$,  265.77$^2$, -169.05$^2$&lt;/li&gt;&#xA;&lt;li&gt;Then we sum and divide by $n-2=1$&lt;/li&gt;&#xA;&lt;li&gt;Take the square root.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Further info&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A residual is simply the $observed - fitted$. So when x = 35, the observed is 0 and the fitted value is &lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{equation}&#xA;11.388612\times 35 - 301.878 = 96.72&#xA;\end{equation}&#xA;The residual is then: $0 - 96.72 = -96.72$&lt;/p&gt;&#xA;" OwnerUserId="8" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2011-01-13T16:42:04.813" LastActivityDate="2011-01-13T16:42:04.813" CommentCount="4" />
  <row Id="6222" PostTypeId="2" ParentId="6219" CreationDate="2011-01-13T15:59:58.307" Score="4" ViewCount="" Body="&lt;p&gt;It's the &lt;a href=&quot;http://en.wikipedia.org/wiki/Root_mean_square&quot; rel=&quot;nofollow&quot;&gt;RMS&lt;/a&gt; (root mean square) of the residuals of the linear regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; x &amp;lt;-c(35, 430, 656)&#xA;&amp;gt; y &amp;lt;- c(0, 4861, 7000)&#xA;&amp;gt; mod &amp;lt;- lm(y~x)&#xA;&amp;gt; mod&#xA;&#xA;Call:&#xA;lm(formula = y ~ x)&#xA;&#xA;Coefficients:&#xA;(Intercept)            x  &#xA;    -301.88        11.39  &#xA;&#xA;&amp;gt; sqrt(sum(resid(mod)^2))&#xA;[1] 329.4998&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="582" LastEditorUserId="582" LastEditorDisplayName="" LastEditDate="2011-01-13T16:07:27.117" LastActivityDate="2011-01-13T16:07:27.117" CommentCount="2" />
  <row Id="6224" PostTypeId="1" CreationDate="2011-01-13T16:45:02.370" Score="3" ViewCount="367" Body="&lt;p&gt;Hi All,&lt;br&gt;&#xA;I used lmer function in lme4 package in order to assess the effects of 2 categorical fixed effects (1º Animal Group: rodents and ants; 2º Microhabitat: bare soil and under cover) on seed predation (a count dependent variable). I have 2 Site, 10 tree per site and 4 seed station per tree. Site and Tree are my (philosophically) random factors, but given that I have only two level for Site, it must be treat Site as fix factor. I have questions about how to interpret the results:&lt;br&gt;&#xA;1) I made a model selection criterion based on QAICc, but the best model (lower QAICc) does not result in any significant fixed effect and other models with higher QAIC (e.g. the Full Model) did find significant fixed factors. This make sense?&lt;br&gt;&#xA;2) given a fixed factor important to the model, How do I distinguish which level of fixed factor is influencing the response variable.&lt;br&gt;&#xA;Finally, correlation between the fixed factors implies an incorrect estimation of the model?&lt;br&gt;&#xA;Thanks in advance and sorry for my english&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    FullModel=lmer(SeedPredation ~ AnimalGroup*Microhabitat*Site + (1|Site:Tree) + (1|obs), data=datos,  family=&quot;poisson&quot;) &#xA;&#xA;    QAICc(FM)104.9896&#xA;&#xA;    enterGeneralized linear mixed model fit by the Laplace approximation &#xA;Formula: SP ~ AG * MH * Site + (1 | Site:Tree) + (1 | obs) &#xA;   Data: datos &#xA;   AIC   BIC logLik deviance&#xA; 101.8 125.6  -40.9     81.8&#xA;Random effects:&#xA; Groups    Name        Variance Std.Dev.&#xA; obs       (Intercept) 0.20536  0.45317 &#xA; Site:Tree (Intercept) 1.19762  1.09436 &#xA;Number of obs: 80, groups: obs, 80; Site:Tree, 20&#xA;&#xA;Fixed effects:&#xA;                 Estimate Std. Error z value Pr(&amp;gt;|z|)  &#xA;(Intercept)       0.01161    0.47608   0.024   0.9805  &#xA;AGR             -18.97679 3130.76500  -0.006   0.9952  &#xA;MHUC             -1.60704    0.63626  -2.526   0.0115 *&#xA;Site2            -0.91424    0.74506  -1.227   0.2198  &#xA;AGR:MHUC         19.92369 3130.76508   0.006   0.9949  &#xA;AGR:Site2         1.02241 4431.84919   0.000   0.9998  &#xA;MHUC:Site2        1.80029    0.86235   2.088   0.0368 *&#xA;AGR:MHUC:Site2   -3.49042 4431.84933  -0.001   0.9994  &#xA;---&#xA;Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 &#xA;&#xA;Correlation of Fixed Effects:&#xA;            (Intr) AGR    MHUC   Site2  AGR:MHUC AGR:S2 MHUC:S&#xA;AGR          0.000                                            &#xA;MHUC        -0.281  0.000                                     &#xA;Site2       -0.639  0.000  0.180                              &#xA;AGR:MHUC     0.000 -1.000  0.000  0.000                       &#xA;AGR:Site2    0.000 -0.706  0.000  0.000  0.706                &#xA;MHUC:Site2   0.208  0.000 -0.738 -0.419  0.000    0.000       &#xA;AGR:MHUC:S2  0.000  0.706  0.000  0.000 -0.706   -1.000  0.000 code here&#xA;&#xA;BestModel=lmer(SP ~ AG * MH + (1|Site:Tree) + (1|obs), data=datos,  family = &quot;poisson&quot;) &#xA;&#xA;QAICc(M) 101.4419&#xA;&#xA;Generalized linear mixed model fit by the Laplace approximation &#xA;Formula: SP ~ AG + AG:MH + (1 | Site:Tree) + (1 | obs) &#xA;   Data: datos &#xA;   AIC   BIC logLik deviance&#xA; 100.3 114.6 -44.15     88.3&#xA;Random effects:&#xA; Groups    Name        Variance Std.Dev.&#xA; obs       (Intercept) 0.76027  0.87194 &#xA; Site:Tree (Intercept) 1.14358  1.06938 &#xA;Number of obs: 80, groups: obs, 80; Site:Tree, 20&#xA;&#xA;Fixed effects:&#xA;             Estimate Std. Error z value Pr(&amp;gt;|z|)&#xA;(Intercept)   -0.5153     0.4061  -1.269    0.205&#xA;AGR          -18.7146  2603.4397  -0.007    0.994&#xA;AGA:MHUC      -0.7301     0.5045  -1.447    0.148&#xA;AGR:MHUC      17.7221  2603.4397   0.007    0.995&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2486" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-01-13T21:10:39.160" LastActivityDate="2011-01-18T14:54:42.040" Title="GLMM output interpretation (correct text)" Tags="&lt;r&gt;&lt;mixed-model&gt;" AnswerCount="1" CommentCount="5" />
  <row Id="6225" PostTypeId="1" AcceptedAnswerId="6240" CreationDate="2011-01-13T16:46:34.553" Score="11" ViewCount="587" Body="&lt;p&gt;As the question states - Is it possible to prove the null hypothesis? From my (limited) understanding of hypothesis, the answer is no but I can't come up with a rigorous explanation for it. Does the question have a definitive answer? &lt;/p&gt;&#xA;" OwnerUserId="2198" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-02-12T05:51:37.453" LastActivityDate="2011-03-07T01:52:33.090" Title="Is it possible to prove a null hypothesis?" Tags="&lt;hypothesis-testing&gt;&lt;proof&gt;&lt;equivalence&gt;" AnswerCount="7" CommentCount="3" FavoriteCount="4" />
  <row Id="6226" PostTypeId="1" AcceptedAnswerId="6233" CreationDate="2011-01-13T16:52:47.640" Score="3" ViewCount="157" Body="&lt;p&gt;I'm doing a time series analysis. I'm doing most of my analysis in R, where I can use &quot;NA&quot; to represent &quot;not available&quot; (e.g. a missing data point). But I'm doing some data preparation in OpenOffice; currently, I'm leaving cells blank for missing data. Is there a better way to &quot;declare&quot; that a cell is NA?&lt;/p&gt;&#xA;" OwnerUserId="660" LastEditorUserId="660" LastEditorDisplayName="" LastEditDate="2011-01-13T18:21:28.980" LastActivityDate="2011-01-13T19:38:36.953" Title="Representing missing data in an OpenOffice spreadsheet" Tags="&lt;r&gt;&lt;missing-data&gt;&lt;open-office&gt;&lt;na&gt;" AnswerCount="2" CommentCount="7" />
  <row Id="6227" PostTypeId="2" ParentId="6225" CreationDate="2011-01-13T16:54:05.220" Score="6" ViewCount="" Body="&lt;p&gt;Yes there is a definitive answer.  That answer is: No, there isn't a way to prove a null hypothesis.  The best you can do, as far as I know, is throw confidence intervals around your estimate and demonstrate that the effect is so small that it might as well be essentially non-existent.&lt;/p&gt;&#xA;" OwnerUserId="196" LastActivityDate="2011-01-13T16:54:05.220" CommentCount="1" />
  <row Id="6229" PostTypeId="2" ParentId="6225" CreationDate="2011-01-13T17:43:22.683" Score="7" ViewCount="" Body="&lt;p&gt;Answer from the mathematical side : it is possible if and only if &quot;hypotheses are mutually singular&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If by &quot;prove&quot; you mean have a rule that can &quot;accept&quot; (should I say that:) )  $H_0$ with a probability to make a mistake that is zero, then you are searching what could be called &quot;ideal test&quot; and this exists:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are testing wether  a random variable $X$ is drawn from $P_0$ or from $P_1$ (i.e testing $H_0: X\leadsto P_0$ versus $H_1: X\leadsto P_1$) then there exists an ideal test &lt;strong&gt;if and only if&lt;/strong&gt; $P_1\bot P_0$ ($P_1$ and $P_0$ are &quot;mutually singular&quot;).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you don't know what &quot;mutually singular&quot; means I can give you an example: $\mathcal{U}[0,1]$ and $\mathcal{U}[3,4]$ (uniforms on $[0,1]$ and $[3,4]$) are mutually singular.  This means if you want to test &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$H_0: X\leadsto \mathcal{U}[0,1]$ versus $H_1: X\leadsto \mathcal{U}[3,4]$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;then there exist an ideal test (guess what it is :) ) : a test that is never wrong ! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If $P_1$ and $P_0$ are not mutually singular, then this does not exist (this results from the &quot;only if part&quot;)! &lt;/p&gt;&#xA;" OwnerUserId="223" LastEditorUserId="223" LastEditorDisplayName="" LastEditDate="2011-02-09T15:52:44.957" LastActivityDate="2011-02-09T15:52:44.957" CommentCount="2" />
  <row Id="6230" PostTypeId="2" ParentId="6195" CreationDate="2011-01-13T18:40:03.760" Score="1" ViewCount="" Body="&lt;p&gt;It looks like a linear regression problem me too, but what about K nearest neighbors &lt;a href=&quot;http://en.wikipedia.org/wiki/K-nearest_neighbor_algorithm&quot; rel=&quot;nofollow&quot;&gt;KNN&lt;/a&gt;.  You can come up with a distance formula between each car and compute the price as the average between the K (say 3) nearest.  A distance formula can be euclidian based like the difference in cylinders plus the difference in doors, plus difference in horsepower and so on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you go with linear regresion I would suggest a couple things:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Scale the dollar value up to modern day to account for inflation.&lt;/li&gt;&#xA;&lt;li&gt;Divide your data into epochs.  I'll bet you'll find you will need one model for pre ww2 and post ww2 for example.  This is just a hunch though.&lt;/li&gt;&#xA;&lt;li&gt;Cross validate your model to avoid over fitting.  Divide your data into 5 chunks.  Train on 4 and urn the model on the 5th chunk.  Sum up the errors, rinse, repeat for the other chunks.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Another idea is to made a hybrid between models.  Use regresion and KNN both as datapoints and create the final price as the weighted average or something.&lt;/p&gt;&#xA;" OwnerUserId="2769" LastActivityDate="2011-01-13T18:40:03.760" />
  <row Id="6231" PostTypeId="2" ParentId="6226" CreationDate="2011-01-13T18:46:07.780" Score="4" ViewCount="" Body="&lt;p&gt;This won't be specific to OpenOffice, but I have found the ways different spreadsheet software and more traditional stat packages (such as R or SPSS) handle missing data are not always intuitive or even uniform within the software. Anytime I have missing data I typically check certain functions with toy data to see how they are handled.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In Excel IMO it is a waste of time to make explicit missing data declarations as it makes it more difficult to use the built in functions (especially if you use integers to represent missing data). Hence if you asked for Excel I would just say leave the cells blank (although I can't say for sure if this translates directly to OpenOffice).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As always, you will probably be well served to learn to do the data manipulation in R that you are currently doing in spreadsheet software.&lt;/p&gt;&#xA;" OwnerUserId="1036" LastActivityDate="2011-01-13T18:46:07.780" />
  <row Id="6232" PostTypeId="1" CreationDate="2011-01-13T19:28:37.650" Score="4" ViewCount="251" Body="&lt;p&gt;We are analysing the effects of different harvesting intensities on soil nutrients. Data was collected over three different years. The first year was before treatments were applied, and the second and third years were 1 year and 6 years post treatment.  There are 4 stand types (each replicated 3 times), 6 treatments (retention harvesting of all different intensities) and 6 plots within each stand type*treatment. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am having some trouble wrapping my head around this question and am wondering which test I should be using?&lt;/p&gt;&#xA;" OwnerUserId="2771" LastEditorUserId="5" LastEditorDisplayName="" LastEditDate="2011-01-13T19:38:15.753" LastActivityDate="2011-01-15T12:30:49.283" Title="Which Test do I use? ANCOVA, repeated measures, multi-way ANOVA?" Tags="&lt;anova&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="6233" PostTypeId="2" ParentId="6226" CreationDate="2011-01-13T19:38:36.953" Score="4" ViewCount="" Body="&lt;p&gt;If you want to import the data to R leave the cells blank. If file is saved as csv and imported to R, blank cells will be represented as NA automatically. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to do some analysis in OpenOffice, I think you will find that @Andy W advice useful. Built-in OpenOffice functions may behave weirdly if you use some custom NA declaration. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally as @cgillespie pointed out it is better to do all data preparation in R. Even if it is slightly harder. The foremost reason for that is that this way you will be able to track changes to original data, which is highly desirable for debugging purposes. Furthermore if you work out the preparation with care it will be very easy to include new data. For this reason alone I now always do my data preparation only in R. I advise to look into &lt;a href=&quot;http://cran.r-project.org/web/packages/reshape/index.html&quot; rel=&quot;nofollow&quot;&gt;package reshape&lt;/a&gt;, it saves a lot of time for me.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-01-13T19:38:36.953" CommentCount="2" />
  <row Id="6234" PostTypeId="1" AcceptedAnswerId="6235" CreationDate="2011-01-13T20:08:50.767" Score="8" ViewCount="267" Body="&lt;p&gt;Is there a visualization model that is good for showing the intersection overlap of many sets?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am thinking something like Venn diagrams but that somehow might lend itself better to a larger number of sets such as 10 or more. Wikipedia does show some higher set Venn diagrams but even the 4 set diagrams are a lot to take in.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My guess as to the final result of the data would be that many of the sets won't overlap so it is possible that Venn diagrams would be fine -- but I would like to find a computer tool that will be able to generate that. Its looks to me like Google charts doesn't allow that many sets.&lt;/p&gt;&#xA;" OwnerUserId="2772" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-01-14T12:51:34.317" LastActivityDate="2011-02-20T09:48:31.850" Title="Visualizing the intersections of many sets" Tags="&lt;data-visualization&gt;&lt;dataset&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="1" />
  <row Id="6235" PostTypeId="2" ParentId="6234" CreationDate="2011-01-13T20:38:45.660" Score="11" ViewCount="" Body="&lt;p&gt;When you have a large number of sets, I would try something that is more linear and shows the links directly (like a network graph).  Flare and Protovis both have utilities to handle these visualizations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See &lt;a href=&quot;http://stats.stackexchange.com/questions/3158/what-is-this-type-of-circular-link-visualization-called/3159#3159&quot;&gt;this question for some examples&lt;/a&gt; like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/vo8GG.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="5" LastEditorUserId="5" LastEditorDisplayName="" LastEditDate="2011-01-13T20:44:23.957" LastActivityDate="2011-01-13T20:44:23.957" CommentCount="5" />
  <row Id="6236" PostTypeId="2" ParentId="5682" CreationDate="2011-01-13T20:49:25.670" Score="7" ViewCount="" Body="&lt;p&gt;Full disclosure: I work at SAS.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The IML blog is &lt;a href=&quot;http://blogs.sas.com/iml&quot; rel=&quot;nofollow&quot;&gt;http://blogs.sas.com/iml&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Both languages are matrix-vector languages with a rich run-time library and the ability to write your own functions. For data analysis tasks and matrix computations, they both provide the neccessary tools to help you analyze your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The SAS/IML syntax is very similar to the SAS DATA step, so it appeals to SAS programmers.  You can also call all of the SAS DATA step functions, and you can call any SAS procedure from within SAS/IML by using the SUBMIT/ENDSUBMIT statements.  The SAS/IML Studio application is very nice for developing programs and for creating graphics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The R community creates and shares a large number of packages, including packages written by top academic researchers.  New statistical methods appear in R very quickly. The R community has many help and discussion lists. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The SAS/IML language does not contain every statistical analysis (as a built-in function) because the assumption is that you will call SAS/STAT or SAS/ETS procedures when you need a specialized analysis. For example, SAS/IML does not have functions for mixed modelling, but you can prepare the data in SAS/IML, call the MIXED or GLIMMIX procedure, and then use IML some more to manipulate or modify the output from the procedure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In chapter 11 (and 16) of my book, I show how to call R from SAS/IML, transfer data back and forth, and generally show how to get the best of both worlds.&lt;/p&gt;&#xA;" OwnerUserId="2773" LastActivityDate="2011-01-13T20:49:25.670" />
  <row Id="6237" PostTypeId="2" ParentId="6151" CreationDate="2011-01-13T22:33:59.487" Score="2" ViewCount="" Body="&lt;p&gt;In the spirit of an &lt;a href=&quot;http://stats.stackexchange.com/questions/3006/factor-analysis-of-dyadic-data/3010#3010&quot;&gt;earlier response&lt;/a&gt;, you might be interested in David A Kenny's webpage on &lt;a href=&quot;http://davidakenny.net/dyad.htm&quot; rel=&quot;nofollow&quot;&gt;dyadic analysis&lt;/a&gt;, and models for matched pairs (See Agresti, &lt;a href=&quot;http://www.stat.ufl.edu/~aa/cda/cda.html&quot; rel=&quot;nofollow&quot;&gt;Categorical Data Analysis&lt;/a&gt;, Chapter 10, or this nice &lt;a href=&quot;http://www.ed.uiuc.edu/courses/EdPsy490AT/lectures/9matched_02.pdf&quot; rel=&quot;nofollow&quot;&gt;handout&lt;/a&gt;, by C J Anderson). There's also a lot of literature on sib-pair study design, in genetics, epidemiology, and psychometrics. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you may know, studies on sibling rivalry also suggest that parents' attitude might play a role, but also that generally sibling relationships in early adulthood might be characterized by independent dimensions (warmth, conflict, and rivalry, according to Stocker et al., 1997). So it may be interesting for you to look at what has been done in psychometrics, especially whether your items share some similarity with previous studies or not. The very first hit on Google with &lt;code&gt;siblings rivalry scale statistical analysis&lt;/code&gt; was a study on &lt;a href=&quot;http://www.kon.org/urc/v8/tsang.html&quot; rel=&quot;nofollow&quot;&gt;The Effects of Working Mothers on Sibling Rivalry&lt;/a&gt; which offers some clues on how to handle such data (although I still think that model for matched pairs are better than the $\chi^2$-based approach used in this study).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;References&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Stocker, CM, Lanthier, RP, Furman, W (1997). &lt;a href=&quot;http://abutler.net/trc/publications/pdfs/Sibling%20relationships%20in%20early%20adulthood%2057.pdf&quot; rel=&quot;nofollow&quot;&gt;Sibling Relationships in Early Adulthood&lt;/a&gt;. &lt;em&gt;Journal of Family Psychology&lt;/em&gt;, 11(2), 210-221.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-01-13T22:33:59.487" CommentCount="1" />
  <row Id="6238" PostTypeId="2" ParentId="6180" CreationDate="2011-01-13T22:52:29.760" Score="0" ViewCount="" Body="&lt;p&gt;When using confidence intervals, my response is always as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In repeated sampling &lt;strong&gt;% of intervals so constructed will contain _&lt;/strong&gt;. Thus, we have &lt;strong&gt;% confidence that the true &lt;em&gt;&lt;/strong&gt; lies within &lt;strong&gt;&lt;/em&gt;__&lt;em&gt;&lt;/strong&gt;-&lt;/em&gt;&lt;strong&gt;&lt;em&gt;_&lt;/em&gt;&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One example is &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In repeated sampling, 95% of all intervals so constructed will contain Mu, the true population mean. Thus we have 95% confidence that the population GPA lies between 2.72 and 3.07. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't see anything particularly wrong with your response, except I would imagine the last sentence should be 95%, not more or less, or at least. &lt;/p&gt;&#xA;" OwnerUserId="2644" LastActivityDate="2011-01-13T22:52:29.760" CommentCount="1" />
  <row Id="6239" PostTypeId="1" AcceptedAnswerId="6241" CreationDate="2011-01-13T23:11:58.450" Score="4" ViewCount="141" Body="&lt;p&gt;I have a time series and I want to subset it while keeping it as a time series, preserving the start, end, and frequency.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, let's say I have a time series:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; qs &amp;lt;- ts(101:110, start=c(2009, 2), frequency=4)&#xA;&amp;gt; qs&#xA;     Qtr1 Qtr2 Qtr3 Qtr4&#xA;2009       101  102  103&#xA;2010  104  105  106  107&#xA;2011  108  109  110     &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now I will subset it:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; qs[time(qs) &amp;gt;= 2010 &amp;amp; time(qs) &amp;lt; 2011]&#xA;[1] 104 105 106 107&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Notice that I got the correct results, but I lost the &quot;wrappings&quot; from the time series (namely start, end, frequency.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm looking for a function for this. Isn't subsetting a time series is a common scenario? Since I haven't found one yet, here is a function I wrote:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;subset.ts &amp;lt;- function(data, start, end) {&#xA;  ks &amp;lt;- which(time(data) &amp;gt;= start &amp;amp; time(data) &amp;lt; end)&#xA;  vec &amp;lt;- data[ks]&#xA;  ts(vec, start=start(data) + c(0, ks[1] - 1), frequency=frequency(data))&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'd like to hear about improvements or cleaner ways to do this. In particular, I don't like the way I'm hard-coding start and end. I'd rather let the user specify an arbitrary boolean condition.&lt;/p&gt;&#xA;" OwnerUserId="660" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-01-14T06:24:35.500" LastActivityDate="2011-01-14T09:35:20.220" Title="Subsetting R time series vectors" Tags="&lt;r&gt;&lt;time-series&gt;" AnswerCount="2" />
  <row Id="6240" PostTypeId="2" ParentId="6225" CreationDate="2011-01-13T23:51:59.263" Score="9" ViewCount="" Body="&lt;p&gt;If you are talking about the real world &amp;amp; not formal logic, the answer is &lt;em&gt;of course.&lt;/em&gt; &quot;Proof&quot; of anything by empirical means depends on the strength of the inference one can make, which in turn is determined by validity of the testing process as evaluated in light of everything one knows about how the world  works (i.e., theory). Whenever one accepts that certain empirical results justify  rejecting the &quot;null&quot; hypothesis, one is necessarily making judgments of this sort (validity of design; world works in certain way), so having to make the analogous assumptions necessary to justify inferring &quot;proof of the null&quot; is &lt;em&gt;not&lt;/em&gt; problematic at all. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So what are the analogous assumptions? Here is an example of &quot;proving the null&quot; that is commonplace in health science &amp;amp; in social science.  (1) Define &quot;null&quot; or &quot;no effect&quot; in some way that is practically meaningful. Let's say that I believe that I should conduct myself as if there is no meaningful difference between 2 treatments, t1 &amp;amp; t2, for a disease unless one gives a 3% better chance of recovery than the other. (2) Figure out a valid design for testing whether there is any effect-- in this case, whether there is a difference in recovery likelihood between t1 &amp;amp; t2. (3) Do a &lt;em&gt;power&lt;/em&gt; analysis to determine whether what sample size is necessary to generate a sufficiently high likelihood-- one that I am confident relying on given what's at stake -- that I would see the meaningful effect, 3% in my example, &lt;em&gt;assuming&lt;/em&gt; it exists. Usually people say power is sufficient if the likelihood of observing a specified effect at a specified alpha is at least 0.80, but the right level of confidence is really a matter of how averse you are to error -- same as it is when you select p-value threshold for &quot;rejecting the null.&quot;(4) Perform the empirical test &amp;amp; observe the effect. If it is below the specified &quot;meaningful difference&quot; value -- 3% in my example -- you've &quot;proven&quot; that there is &quot;no effect.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a good treatment of this matter, see Streiner, D.L. &lt;a href=&quot;http://ww1.cpa-apc.org/Publications/Archives/CJP/2003/december/streiner.pdf&quot; rel=&quot;nofollow&quot;&gt;Unicorns Do Exist: A Tutorial on “Proving” the Null Hypothesis&lt;/a&gt;. Canadian Journal of Psychiatry 48, 756-761 (2003).&lt;/p&gt;&#xA;" OwnerUserId="2347" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-01-14T09:17:37.343" LastActivityDate="2011-01-14T09:17:37.343" CommentCount="4" />
  <row Id="6241" PostTypeId="2" ParentId="6239" CreationDate="2011-01-14T00:01:03.723" Score="13" ViewCount="" Body="&lt;p&gt;Use the &lt;code&gt;window&lt;/code&gt; function:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; window(qs, 2010, c(2010, 4))&#xA;     Qtr1 Qtr2 Qtr3 Qtr4&#xA;2010  104  105  106  107&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="5" LastActivityDate="2011-01-14T00:01:03.723" CommentCount="2" />
  <row Id="6242" PostTypeId="2" ParentId="1455" CreationDate="2011-01-14T07:14:24.423" Score="0" ViewCount="" Body="&lt;p&gt;what is meant by logistic emperical transformation&lt;/p&gt;&#xA;" OwnerUserId="2777" LastActivityDate="2011-01-14T07:14:24.423" CommentCount="1" />
  <row Id="6243" PostTypeId="1" CreationDate="2011-01-14T08:09:04.757" Score="2" ViewCount="138" Body="&lt;p&gt;When computing the difference between means I can resolve a new mean, variance, standard error of the mean, and margin of error, but is there a way to compute a new/composite sample size value?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I ask because I use the difference between means as a way to calibrate the mean and later when comparing other calibrated means sample size comes into the formulas.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(&lt;a href=&quot;http://stattrek.com/AP-Statistics-4/Difference-Means.aspx&quot; rel=&quot;nofollow&quot;&gt;http://stattrek.com/AP-Statistics-4/Difference-Means.aspx&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Update&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;In the code example below I subtract the calibrations mean from the benchmarks mean time to execute a test. The calibration is an empty test to clock the cost of the benchmarks overhead.&#xA;The only part I am missing is how to create a composite sample size (if that is possible).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;       mean -= cs.mean;&#xA;       variance += cs.variance;&#xA;       sd = sqrt(variance);&#xA;       moe = sd * getCriticalValue(getDegreesOfFreedom(me, cal));&#xA;       rme = (moe / mean) * 100;&#xA;       extend(me.stats, {&#xA;         'ME': moe,&#xA;         'RME': rme,&#xA;         'deviation': sd,&#xA;         'mean': mean,&#xA;         'size': ???,&#xA;         'variance': variance&#xA;       });&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2634" LastEditorUserId="2634" LastEditorDisplayName="" LastEditDate="2011-01-14T20:24:56.150" LastActivityDate="2011-01-15T20:28:57.713" Title="Computing the difference between means and resolving a new/composite sample size" Tags="&lt;sample-size&gt;&lt;mean&gt;" AnswerCount="3" CommentCount="8" />
  <row Id="6244" PostTypeId="2" ParentId="6239" CreationDate="2011-01-14T09:35:20.220" Score="1" ViewCount="" Body="&lt;p&gt;Also useful, if you are combining multiple time series and don't want to have to have to &lt;code&gt;window&lt;/code&gt; every one to get them to match, &lt;code&gt;ts.union&lt;/code&gt; and &lt;code&gt;ts.intersect&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1195" LastActivityDate="2011-01-14T09:35:20.220" />
  <row Id="6245" PostTypeId="1" CreationDate="2011-01-14T10:56:04.463" Score="4" ViewCount="92" Body="&lt;p&gt;In a field experiment involving crops, what is the difference in considering block as random or otherwise as fixed factor?&#xA;As far as I understood, random means that conclusion can be extended to other levels not included in the study; fixed factor on the contrary restricts the analysis only to a specific experiment.&lt;/p&gt;&#xA;" OwnerUserId="2779" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-01-14T19:32:48.250" LastActivityDate="2011-01-14T19:32:48.250" Title="How can I treat blocks in a split plot design?" Tags="&lt;anova&gt;&lt;mixed-model&gt;&lt;random-variable&gt;&lt;experiment-design&gt;&lt;split-plot&gt;" AnswerCount="1" />
  <row Id="6246" PostTypeId="2" ParentId="1455" CreationDate="2011-01-14T12:10:06.210" Score="1" ViewCount="" Body="&lt;p&gt;Generally, with a large sample size it is assumed as reasonable approximation that all estimators (or some opportune functions of them) have a normal distribution.  So, if you only need the &lt;em&gt;p&lt;/em&gt;-value corresponding to the given confidence interval, you can simply proceed as follows:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;transform OR and the corresponding (c1,c2) CI to ln(OR) and&#xA;(ln(c1),ln(c2))  [OR domain is (0,+inf) while ln(OR) domain is (-inf,+inf)]&lt;/li&gt;&#xA;&lt;li&gt;since the length of every CI depends on its level alpha&#xA;and on estimator standard deviation, calculate&#xA;sd(OR)=(ln(c2)-ln(c1))/(z(alpha/2)*2) [Pr(Z&gt;z(alpha/2))=alpha/2; z(0.05/2)=1.96]&lt;/li&gt;&#xA;&lt;li&gt;calculate the &lt;em&gt;p&lt;/em&gt;-value corresponding to the (standardized&#xA;normal) test statistic z=ln(OR)/sd(OR)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="1219" LastEditorUserId="1219" LastEditorDisplayName="" LastEditDate="2011-01-14T12:15:16.330" LastActivityDate="2011-01-14T12:15:16.330" CommentCount="1" />
  <row Id="6247" PostTypeId="1" AcceptedAnswerId="6251" CreationDate="2011-01-14T13:47:37.963" Score="3" ViewCount="171" Body="&lt;p&gt;I have a problem. I am creating a wide barplot (50x grouped 4 columns) and printing it to ps. However the figure does not fit to printed ps - the x axis goes out of a page. It would fit if only I could remove a large left margin on yaxis. But I have no idea how to move picture left and reduce the margin. I have spend some time on asking uncle google but I have not found anything. Could somebody help me?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is my R script:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mx &amp;lt;- matrix( c(5,3,8,9,5,3,8,9),nr=4)&#xA;&#xA;postscript(file=&quot;&amp;lt;fileName&amp;gt;.ps&quot;);&#xA;&#xA;barplot(mx, beside=T,  col=gray.colors(4), cex.axis = 1.4, cex.names=1.2, xlim=c(1,60), width=0.264,  xaxs = &quot;r&quot;, yaxs = &quot;r&quot;)&#xA;&#xA;legend(&quot;topleft&quot;, c( &quot;A1&quot;,  &quot;A2&quot;,&quot;B1&quot;,&quot;B2&quot;), pch=15, &#xA;        cex=1.1,&#xA;       col=gray.colors(4), &#xA;       bty=&quot;n&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1389" LastEditorUserId="1389" LastEditorDisplayName="" LastEditDate="2011-01-14T15:39:25.757" LastActivityDate="2011-01-14T15:39:25.757" Title="Postscript in R: How to remove a large left margin (on y axis)" Tags="&lt;r&gt;&lt;data-visualization&gt;&lt;barplot&gt;" AnswerCount="3" CommentCount="1" />
  <row Id="6248" PostTypeId="2" ParentId="6247" CreationDate="2011-01-14T13:52:32.007" Score="2" ViewCount="" Body="&lt;p&gt;Excerpt from the help page of function &lt;strong&gt;par&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;mar&lt;/code&gt;: A numerical vector of the form &lt;code&gt;c(bottom, left, top, right)&lt;/code&gt;&#xA;          which gives the number of lines of margin to be specified on&#xA;          the four sides of the plot.  The default is &lt;code&gt;c(5, 4, 4, 2) +&#xA;          0.1&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So try something like&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;par(mar=c(5,1,4,2))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;before calling &lt;strong&gt;barplot&lt;/strong&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-01-14T13:52:32.007" CommentCount="4" />
  <row Id="6249" PostTypeId="2" ParentId="6247" CreationDate="2011-01-14T13:53:55.810" Score="2" ViewCount="" Body="&lt;p&gt;You could look after the par() function's mar(gin) paramater. A nice brief can be found in &lt;a href=&quot;http://research.stowers-institute.org/efg/R/Graphics/Basics/mar-oma/index.htm&quot; rel=&quot;nofollow&quot;&gt;efg's Research Notes&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2714" LastActivityDate="2011-01-14T13:53:55.810" />
  <row Id="6250" PostTypeId="2" ParentId="26" CreationDate="2011-01-14T13:54:47.790" Score="0" ViewCount="" Body="&lt;p&gt;Here is a good explanation:&#xA;&lt;a href=&quot;http://amarsagoo.blogspot.com/2007/09/making-sense-of-standard-deviation.html&quot; rel=&quot;nofollow&quot;&gt;http://amarsagoo.blogspot.com/2007/09/making-sense-of-standard-deviation.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="311" LastActivityDate="2011-01-14T13:54:47.790" />
  <row Id="6251" PostTypeId="2" ParentId="6247" CreationDate="2011-01-14T13:59:06.693" Score="3" ViewCount="" Body="&lt;p&gt;I quite like the suggestion made by &lt;a href=&quot;http://www.stat.columbia.edu/~cook/movabletype/archives/2010/10/could_someone_p.html&quot; rel=&quot;nofollow&quot;&gt;Andrew Gelman&lt;/a&gt; for the default setting of &lt;code&gt;par&lt;/code&gt;. Namely, &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;par(mar=c(3,3,2,1), mgp=c(2,.7,0), tck=-.01)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="8" LastActivityDate="2011-01-14T13:59:06.693" />
  <row Id="6252" PostTypeId="1" CreationDate="2011-01-14T14:06:06.030" Score="5" ViewCount="215" Body="&lt;p&gt;I have a clustering algorithm (not k-means) with input parameter $k$ (number of clusters). After performing clustering I'd like to get some quantitative measure of quality of this clustering. &#xA;The clustering algorithm has one important property. For $k=2$ if I feed $N$ data points without any significant distinction among them to this algorithm as a result I will get one cluster containing $N-1$ data points and one cluster with $1$ data point. Obviously this is not what I want. So I want to calculate this quality measure to estimate reasonability of this clustering. Ideally I will be able to compare this measures for different $k$. So I will run clustering in the range of $k$ and choose the one with the best quality.&#xA;How do I calculate such quality measure?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;UPDATE:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's an example when $(N-1, 1)$ is a bad clustering. Let's say there are 3 points on a plane forming equilateral triangle. Splitting these points into 2 clusters is obviously worse than splitting them into 1 or 3 clusters.&lt;/p&gt;&#xA;" OwnerUserId="255" LastEditorUserId="255" LastEditorDisplayName="" LastEditDate="2011-01-14T19:33:53.787" LastActivityDate="2011-01-28T18:28:55.360" Title="Clustering quality measure" Tags="&lt;clustering&gt;" AnswerCount="4" CommentCount="0" FavoriteCount="1" />
  <row Id="6253" PostTypeId="1" CreationDate="2011-01-14T15:02:59.787" Score="5" ViewCount="165" Body="&lt;p&gt;You are in an exam, and are presented with the following question:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Write down what mark do you expect to&#xA;  take in this exam... If you get it&#xA;  right in range of +/-10 % then you&#xA;  will take 10% bonus... if wrong (or&#xA;  not answered) you will lose 5%&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Assume that you have no idea of how you are going to perform in this exam. How would you choose a mark that maximizes your expected return mark?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words, I need to help deriving an equation for the expected mark given a decided mark, if that is possible...&lt;/p&gt;&#xA;" OwnerUserId="2599" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2011-01-14T15:24:06.853" LastActivityDate="2011-01-15T21:41:48.193" Title="Increasing Exam Expected Mark" Tags="&lt;probability&gt;&lt;expected-value&gt;" AnswerCount="3" CommentCount="2" FavoriteCount="1" />
  <row Id="6254" PostTypeId="1" CreationDate="2011-01-14T15:13:26.037" Score="2" ViewCount="66" Body="&lt;p&gt;Hi&#xA;I have a very imbalanced sample set, e.g. 99% true and 1% false. Is it reasonable to select a balanced subset with a 50/50-distribution for neural network trainining? the reason for this is, that i guess training on the original dataset may induce a bias on the true-samples.&#xA;Can you suggest my some literature that covers this topic especially for neural netowrks?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you very much,&#xA;Julian&lt;/p&gt;&#xA;" OwnerUserId="2781" LastActivityDate="2011-01-14T15:41:24.823" Title="Balanced sampling for network training?" Tags="&lt;sampling&gt;&lt;neural-networks&gt;" AnswerCount="1" />
  <row Id="6255" PostTypeId="2" ParentId="6253" CreationDate="2011-01-14T15:17:31.510" Score="5" ViewCount="" Body="&lt;p&gt;First a couple of assumptions:&#xA;1. All marks are equally likely.&#xA;1. If you guess your mark to be 95 and you get 95, your return mark is 100 not&#xA;   105.&#xA;1. Similarly, if your exam mark is 1 and you guess 50 (say), then your return&#xA;   mark is 0 not -4.&#xA;1. I'm only considering discrete marks, that is, values 0, ..., 100.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose your guessed mark is $g=50$. Then your expected return mark is:&#xA;$$&#xD;&#xA;\frac{\sum_{i=0}^{34} + \sum_{i=50}^{70} + \sum_{i=56}^{95}}{101} = 49.15482&#xD;&#xA;$$&#xA;This is for a particularly $g$. We need to repeat this for all $g$. Using the R&#xA;code at the end, we get the following plot:&#xA;&lt;img src=&quot;http://i.imgur.com/0fwic.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since all marks are equally likely you get a plateau with edge effects. If you  really have &lt;em&gt;absolutely&lt;/em&gt; no idea of what mark you will get, then a sensible strategy would be to maximise the chance of passing the exam. If the pass mark is 40%, then set your guess mark at 35%. This now means that to pass the exam, you only need to get above 35% but more importantly your strategy for sitting the exam is to answer every question to the best of your ability. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your guess mark was 30%, and towards the end of the exam you thought that you would score 42%, you are now in the strange position of deciding whether to intentionally make an error (as 42% results in a return mark of 37%).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: I think in most real life situations you would have some idea of how you would get on. For example, do you really think that you have equal probability of getting between 0-10%, 11-20%, ..., 90-100% in your exam.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;R code&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;f =function(s) {&#xA;  mark = 0&#xA;  for(i in 0:100){&#xA;    if(i &amp;lt; (s-10) | i &amp;gt; (s + 10))&#xA;      mark = mark +  max(0, i-5)&#xA;    else&#xA;      mark = mark + min(i+10, 100)&#xA;  }&#xA;  return(mark/101)&#xA;}&#xA;s = 0:100&#xA;y = sapply(s, f)&#xA;plot(s, y)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="8" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2011-01-15T21:41:48.193" LastActivityDate="2011-01-15T21:41:48.193" CommentCount="2" />
  <row Id="6256" PostTypeId="2" ParentId="6254" CreationDate="2011-01-14T15:41:24.823" Score="1" ViewCount="" Body="&lt;p&gt;Yes, it is reasonable to select a balanced dataset, however if you do your model will probably over-predict the minority class in operation (or on the test set).  This is easily overcome by using a threshold probability that is not 0.5.  The best way to choose the new threshold is to optimise on a validation sample that has the same class frequencies as encountered in operation (or in the test set).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rather than re-sample the data, a better thing to do would be to give different weights to the positive and negative examples in the training criterion.  This has the advantage that you use all of the available training data.  The reason that a class imbalance leads to difficulties is not the imbalance per se.  It is more that you just don't have enough examples from the minority class to adequately represent its underlying distribution.  Therefore if you resample rather than re-weight, you are solving the problem by making the distribution of the majority class badly represented as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some may advise simply using a different threshold rather than reweighting or resampling.  The problem with that approach is that with ANN the hidden layer units are optimised to minimise the training criterion, but the training criterion (e.g. sum-of-squares or cross-entropy) depends on how the behaviour of the model away from the decision boundary rather than only near the decision boundary.  As as result hidden layer units may be assigned to tasks that reduce the value of the training criterion, but do not help in accurate classification.  Using re-weighted training patterns helps here as it tends to focus attention more on the decision boundary, and so the allocation of hidden layer resources &lt;em&gt;may&lt;/em&gt; be better.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For references, a google scholar search for &quot;Nitesh Chawla&quot; would be a good start, he has done a fair amount of very solid work on this.&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2011-01-14T15:41:24.823" CommentCount="1" />
  <row Id="6257" PostTypeId="2" ParentId="6252" CreationDate="2011-01-14T16:47:59.270" Score="3" ViewCount="" Body="&lt;p&gt;Since clustering is unsupervised, it's hard to know a priori what the best clustering is. This is research topic. Gary King, a well-known quantitative social scientist, has a &lt;a href=&quot;http://gking.harvard.edu/publications/general-purpose-computer-assisted-clustering-methodology&quot; rel=&quot;nofollow&quot;&gt;forthcoming article&lt;/a&gt; on this topic. &lt;/p&gt;&#xA;" OwnerUserId="2784" LastActivityDate="2011-01-14T16:47:59.270" CommentCount="2" />
  <row Id="6258" PostTypeId="2" ParentId="6245" CreationDate="2011-01-14T17:22:49.567" Score="2" ViewCount="" Body="&lt;p&gt;Dear User2779:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The way you are thinking is one of the ways most people interpret blocks. But the bigger picture which sometimes people don't notice is: &lt;strong&gt;Blocks are a way to model a correlation structure. They let us &quot;eliminate&quot; or control for factors which we know influence the outcomes but are not really of interest.&lt;/strong&gt; However, your conclusion about fixed factors may not be true in general. If the sampling is random and the effects can be considered fixed (or the individual variances are not very large), fixed factor analysis can be generalized to the population. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On a different note, if you read Casella's &lt;em&gt;Statistical Design&lt;/em&gt;, he points out that blocks need not be treated as random. It does makes sense to treat them as random, but not all the time. In most of the cases, thinking about blocks as a tool to impose a correlation structure or control for the &quot;unknown&quot; factors helps.&lt;/p&gt;&#xA;" OwnerUserId="1307" LastEditorUserId="1307" LastEditorDisplayName="" LastEditDate="2011-01-14T17:30:57.563" LastActivityDate="2011-01-14T17:30:57.563" CommentCount="1" />
  <row Id="6260" PostTypeId="2" ParentId="6253" CreationDate="2011-01-14T18:37:22.030" Score="5" ViewCount="" Body="&lt;p&gt;I'm not sure if this would be a funny game or your professor is mildly sadistic. It would be torturous for students who are right on the edge of passing (which we may expect them to be the worst guessers!) Sorry not an answer but I couldn't help myself.&#xA;&lt;img src=&quot;http://i.imgur.com/MuDBB.jpg&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="1036" LastActivityDate="2011-01-14T18:37:22.030" />
  <row Id="6261" PostTypeId="2" ParentId="6232" CreationDate="2011-01-14T19:57:45.567" Score="1" ViewCount="" Body="&lt;p&gt;Sounds like a mixed effects ANOVA.  If you have a continuous treatment variable (i.e., harvesting intensity), then an ANCOVA is warranted (or, really, just a mixed effects/hierarchical general linear model) (or generalized linear model if your response variable better fits that framework).  &lt;/p&gt;&#xA;" OwnerUserId="101" LastActivityDate="2011-01-14T19:57:45.567" />
  <row Id="6262" PostTypeId="2" ParentId="6074" CreationDate="2011-01-14T21:57:51.143" Score="7" ViewCount="" Body="&lt;p&gt;This game looks similar to 20 questions at &lt;a href=&quot;http://20q.net&quot; rel=&quot;nofollow&quot;&gt;http://20q.net&lt;/a&gt;, which the creator reports is based on a neural network.&#xA;Here's one way to structure such network, similar to the neural network described in  &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.60.3182&quot; rel=&quot;nofollow&quot;&gt;Concept description vectors and the 20 question game&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You'd have&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;A fixed number of questions, with some questions marked as &quot;final&quot; questions. &lt;/li&gt;&#xA;&lt;li&gt;One input unit per question, where &lt;code&gt;0/1&lt;/code&gt; represents &lt;code&gt;no/yes&lt;/code&gt; answer. Initially set to &lt;code&gt;0.5&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;One output unit per question, sigmoid squished into 0..1 range&lt;/li&gt;&#xA;&lt;li&gt;Hidden layer connecting all input units to all output units.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Input units for questions that have been answered are set to 0 or 1, and the assumption is that neural network has been trained to make output units output values close to 1 for questions that have &quot;Yes&quot; answer given a set of existing answers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At each stage you would pick the question which &lt;code&gt;NN&lt;/code&gt; is the least sure about, ie, corresponding output unit is close to &lt;code&gt;0.5&lt;/code&gt;, ask the question, and set corresponding input unit to the answer. At the last stage you pick an output unit from the &quot;final question&quot; list with value closest to &lt;code&gt;1&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each game of 20 questions gives 20 datapoints which you can use to update &lt;code&gt;NN&lt;/code&gt;'s weights with back-propagation, ie, you update the weights to make the outputs of current neural network match the true answer given all the previous questions asked.&lt;/p&gt;&#xA;" OwnerUserId="511" LastEditorUserId="511" LastEditorDisplayName="" LastEditDate="2011-01-15T19:41:21.880" LastActivityDate="2011-01-15T19:41:21.880" />
  <row Id="6263" PostTypeId="1" CreationDate="2011-01-14T22:22:30.160" Score="0" ViewCount="132" Body="&lt;p&gt;I'm going to enter a MSc program to study ML. What are the best schools to do that in Europe?&#xA;I've heard of UCL. What are the strong others?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If any student of UCL here, how hard was to get in?&#xA;(In June I will graduate from Ural State University in Russia)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cheers!&lt;/p&gt;&#xA;" OwnerUserId="2785" LastActivityDate="2011-01-14T22:22:30.160" ClosedDate="2011-01-15T10:51:10.813" Title="Where to study machine learning in Europe?" Tags="&lt;machine-learning&gt;" FavoriteCount="1" />
  <row Id="6265" PostTypeId="1" CreationDate="2011-01-15T00:02:52.923" Score="2" ViewCount="440" Body="&lt;p&gt;Hi Everyone,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like some help with the following problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have 40 subjects. On each subject I take a measurement at 25 body&#xA;sites.  The measurement is a continuous variable that varies between 0 and 1 and appears to be normally distributed.  I want do a test to see if body site statistically significantly affects the measurement.  I also have demographic data on the subjects but haven't tried to analyze that yet.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think the correct thing to do would be to use repeated measures ANOVA or MANOVA to do the analysis because the data is paired.  The problem is that I have missing data and my basic understanding is that I would have to exclude any subject that has data missing for even one body site.  If I did this, I would only have 6 subjects left.  If I excluded any body site with data missing then I would only have 8 body sites left.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my thought was that I could use regular ANOVA but just lose some power to see a difference.  However, this seems like it violates the assumption of independence for ANOVA.  So my main questions are as follows:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Does using Regular ANOVA on matched data violate the independence assumption, and if so is the only consequence a decreased likelihood of rejecting the null hypothesis or is there&#xA;more to it than that? Also does anyone know of a reference I could site regarding this?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Does anyone have any suggestions for a better way to analyze my data?  By the way I usually use JMP or SPSS for analysis.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Billy&lt;/p&gt;&#xA;" OwnerUserId="2788" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-01-15T07:57:33.163" LastActivityDate="2011-01-16T10:24:07.540" Title="Consequence of violating independence assumption of ANOVA" Tags="&lt;anova&gt;&lt;repeated-measures&gt;&lt;missing-data&gt;&lt;manova&gt;" AnswerCount="2" CommentCount="3" />
  <row Id="6266" PostTypeId="2" ParentId="6253" CreationDate="2011-01-15T03:50:50.530" Score="2" ViewCount="" Body="&lt;p&gt;Use the bootstrap! Take lots of practice exams and estimate what your score will be on the real exam. If it does not improve your estimate, it will probably be good preparation!&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2011-01-15T03:50:50.530" CommentCount="1" />
  <row Id="6267" PostTypeId="2" ParentId="6252" CreationDate="2011-01-15T04:15:14.430" Score="1" ViewCount="" Body="&lt;p&gt;Here you have a couple of measures, but there are many more:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;SSE: sum of the square error from the items of each cluster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Inter cluster distance: sum of the square distance between each cluster centroid.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Intra cluster distance for each cluster: sum of the square distance from the items of each cluster to its centroid.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maximum Radius: largest distance from an instance to its cluster centroid.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Average Radius: sum of the largest distance from an instance to its cluster centroid divided by the number of clusters.&lt;/p&gt;&#xA;" OwnerUserId="1808" LastActivityDate="2011-01-15T04:15:14.430" CommentCount="2" />
  <row Id="6268" PostTypeId="1" CreationDate="2011-01-15T06:36:34.710" Score="8" ViewCount="215" Body="&lt;p&gt;I'm searching how to (visually) explain simple linear correlation to first year students.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The classical way to visualize would be to give an Y~X scatter plot with a straight regression line.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Recently, I came by the idea of extending this type of graphics by adding to the plot 3 more images, leaving me with: the scatter plot of y~1, then of y~x, resid(y~x)~x and lastly of residuals(y~x)~1 (centered to the mean)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an example of such a visualization:&#xA;&lt;img src=&quot;http://i.imgur.com/Pe2ul.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And the R code to produce it:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(345)&#xA;x &amp;lt;- runif(50) * 10&#xA;y &amp;lt;- x +rnorm(50)&#xA;&#xA;&#xA;layout(matrix(c(1,2,2,2,2,3 ,3,3,3,4), 1,10))&#xA;plot(y~rep(1, length(y)), axes = F, xlab = &quot;&quot;, ylim = range(y))&#xA;points(1,mean(y), col = 2, pch = 19, cex = 2)&#xA;plot(y~x, ylab = &quot;&quot;, )&#xA;abline(lm(y~x), col = 2, lwd = 2)&#xA;&#xA;plot(c(residuals(lm(y~x)) + mean(y))~x, ylab = &quot;&quot;, ylim = range(y))&#xA;abline(h =mean(y), col = 2, lwd = 2)&#xA;&#xA;plot(c(residuals(lm(y~x)) + mean(y))~rep(1, length(y)), axes = F, xlab = &quot;&quot;, ylab = &quot;&quot;, ylim = range(y))&#xA;points(1,mean(y), col = 2, pch = 19, cex = 2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Which leads me to my question: I would appreciate any &lt;strong&gt;suggestions on how this graph can be enhanced&lt;/strong&gt; (either with text, marks, or any other type of relevant visualizations). Adding relevant R code will also be nice.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One direction is to add some information of the R^2 (either by text, or by somehow adding lines presenting the magnitude of the variance before and after the introduction of x)&#xA;Another option is to highlight one point and showing how it is &quot;better explained&quot; thanks to the regression line.  Any input will be appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cheers, Tal&lt;/p&gt;&#xA;" OwnerUserId="253" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-01-15T12:36:51.127" LastActivityDate="2011-01-19T19:08:24.507" Title="How to present the gain in explained variance thanks to the correlation of Y and X?" Tags="&lt;r&gt;&lt;data-visualization&gt;&lt;regression&gt;&lt;correlation&gt;" AnswerCount="4" CommentCount="2" FavoriteCount="5" />
  <row Id="6269" PostTypeId="2" ParentId="6268" CreationDate="2011-01-15T09:00:39.090" Score="1" ViewCount="" Body="&lt;p&gt;Not answering to your exact question, but the followings could be interesting by visualizing one possible pitfall of linear correlations based on an &lt;a href=&quot;http://stackoverflow.com/questions/4666590/remove-outliers-from-correlation-coefficient-calculation/4668720#4668720&quot;&gt;answer&lt;/a&gt; from &lt;a href=&quot;http://stackoverflow.com/q/4666590/564164&quot;&gt;stackoveflow&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;par(mfrow=c(2,1))&#xA;&#xA;set.seed(1)&#xA;x &amp;lt;- rnorm(1000)&#xA;y &amp;lt;- rnorm(1000)&#xA;plot(y~x, ylab = &quot;&quot;, main=paste('1000 random values (r=', round(cor(x,y), 4), ')',  sep=''))&#xA;abline(lm(y~x), col = 2, lwd = 2)&#xA;&#xA;x &amp;lt;- c(x, 500)&#xA;y &amp;lt;- c(y, 500)&#xA;cor(x,y)&#xA;plot(y~x, ylab = &quot;&quot;, main=paste('1000 random values and (500, 500) (r=', round(cor(x,y), 4), ')',  sep=''))&#xA;abline(lm(y~x), col = 2, lwd = 2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/LcRHP.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://stackoverflow.com/questions/4666590/remove-outliers-from-correlation-coefficient-calculation/4667300#4667300&quot;&gt;@Gavin Simpson's&lt;/a&gt; and &lt;a href=&quot;http://stackoverflow.com/questions/4666590/remove-outliers-from-correlation-coefficient-calculation/4675675#4675675&quot;&gt;@bill_080's answer&lt;/a&gt; also includes nice plots of correlation in the same topic.&lt;/p&gt;&#xA;" OwnerUserId="2714" LastActivityDate="2011-01-15T09:00:39.090" />
  <row Id="6270" PostTypeId="2" ParentId="6252" CreationDate="2011-01-15T10:16:53.073" Score="3" ViewCount="" Body="&lt;p&gt;The choice of metric rather depends on what you consider the purpose of clustering to be.  Personally I think clustering ought to be about identifying different groups of observations that were each generated by a different data generating process.  So I would test the quality of a clustering by generating data from known data generating processes and then calculate how often patterns are misclassified by the clustering.  Of course this involved making assumtions about the distribution of patterns from each generating process, but you can use datasets designed for supervised classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Others view clustering as attempting to group together points with similar attribute values, in which case measures such as SSE etc are applicable.  However I find this definition of clustering rather unsatisfactory, as it only tells you something about the particular sample of data, rather than something generalisable about the underlying distributions.  How methods deal with overlapping clusters is a particular problem with this view (for the &quot;data generating process&quot; view it causes no real problem, you just get probabilities of cluster membership).&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2011-01-15T10:16:53.073" CommentCount="1" />
  <row Id="6271" PostTypeId="2" ParentId="2691" CreationDate="2011-01-15T12:25:12.660" Score="0" ViewCount="" Body="&lt;p&gt;Why so eigenvalues/eigenvectors ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When doing PCA, you want to compute some orthogonal basis by maximizing the projected variance on each basis vector.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having computed previous basis vectors, you want the next one to be:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;orthogonal to the previous&lt;/li&gt;&#xA;&lt;li&gt;norm 1&lt;/li&gt;&#xA;&lt;li&gt;maximizing projected variance, i.e with maximal covariance norm&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;This is a constrained optimization problem, and the Lagrange multipliers (here's for the geometric intuition, see wikipedia page) tell you that the gradients of the objective (projected variance) and the constraint (unit norm) should be &quot;parallel&quot; at the optimium. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is the same as saying that the next basis vector should be an eigenvector of the covariance matrix. The best choice at each step is to pick the one with the largest eigenvalue among the remaining ones.&lt;/p&gt;&#xA;" OwnerUserId="2792" LastEditorUserId="2792" LastEditorDisplayName="" LastEditDate="2011-01-15T12:31:57.020" LastActivityDate="2011-01-15T12:31:57.020" />
  <row Id="6273" PostTypeId="2" ParentId="6243" CreationDate="2011-01-15T13:46:04.417" Score="1" ViewCount="" Body="&lt;p&gt;I just think you are making it just too complex -- it is a JS benchmark for other programmers, not a clinical trial or Higgs boson search that will be peer reviewed by bloodthirsty referees and later have a great impact.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just make a non-small number of repetitions (30), of both test and empty test, subtract the means, calculate error of this mean difference as I wrote you before and publish just this -- adding dozens of other measures will only confuse the readers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, for each comparisons, you will have two means $X$ and $Y$ and two errors of that means, respectively $\Delta X$ and $\Delta Y$, so calculate $T$ as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$T=\frac{X-Y}{\sqrt{(\Delta X)^2+(\Delta Y)^2}}$,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;assume $\infty$ degrees of freedom for t-test and you're done. (If you don't believe that $30=\infty$, see this: &lt;a href=&quot;http://en.wikipedia.org/wiki/Student&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Student&lt;/a&gt;'s_t-distribution )&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2011-01-15T13:46:04.417" CommentCount="1" />
  <row Id="6274" PostTypeId="2" ParentId="6243" CreationDate="2011-01-15T13:58:40.187" Score="1" ViewCount="" Body="&lt;p&gt;&lt;em&gt;How&lt;/em&gt; are you comparing the calibrated means? If you're looking at differences between them, and testing if that's zero using a $t$-test, then surely the mean of the empty loop ($\bar{x}_0$, say) will simply cancel out:&#xA;$$(\bar{x}_1 - \bar{x}_0) - (\bar{x}_2 - \bar{x}_0) = \bar{x}_1 - \bar{x}_2$$&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-01-15T13:58:40.187" CommentCount="1" />
  <row Id="6275" PostTypeId="1" AcceptedAnswerId="6278" CreationDate="2011-01-15T14:01:55.437" Score="5" ViewCount="342" Body="&lt;p&gt;Hi all,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am currently collecting data for an experiment into psychosocial characteristics associated with the experience of pain. As part of this, I am collecting GSR and BP measurements electronically from my participants, along with various self-report and implicit measures. I have a psychological background and am comfortable with factor analysis, linear models and experimental analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is what are good (preferably free) resources available for learning about time series analysis. I am a total newb when it comes to this area, so any help would be greatly appreciated. I have some pilot data to practice on, but would like to have my analysis plan worked out in detail before i finish collected data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the provided references were also R related, that would be wonderful. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edited: to change grammar and to add 'self report and implicit measures'&lt;/p&gt;&#xA;" OwnerUserId="656" LastEditorUserId="656" LastEditorDisplayName="" LastEditDate="2011-01-15T19:36:20.873" LastActivityDate="2011-01-15T19:36:20.873" Title="Good Introductions to Time Series (with R)" Tags="&lt;r&gt;&lt;time-series&gt;&lt;physiological&gt;&lt;measurements&gt;" AnswerCount="2" FavoriteCount="5" />
  <row Id="6276" PostTypeId="2" ParentId="6275" CreationDate="2011-01-15T14:14:37.693" Score="2" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://www.stat.pitt.edu/stoffer/tsa3/&quot; rel=&quot;nofollow&quot;&gt;Time Series Analysis and Its Applications: With R Examples&lt;/a&gt; by Robert H. Shumway and David S. Stoffer would be a great resource for the subject, but you may find a lot of useful blog entries (e.g. my favorite one: &lt;a href=&quot;http://learnr.wordpress.com/&quot; rel=&quot;nofollow&quot;&gt;learnr&lt;/a&gt;) and tutorials (e.g. &lt;a href=&quot;http://www.stat.pitt.edu/stoffer/tsa2/R_time_series_quick_fix.htm&quot; rel=&quot;nofollow&quot;&gt;from the linked homepage&lt;/a&gt;) also on the Internet freely available .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On David Stoffer's homepage (linked above) you can find the example datasets used in the book's chapters, and others from the first and second editions with even sample chapters also.&lt;/p&gt;&#xA;" OwnerUserId="2714" LastEditorUserId="2714" LastEditorDisplayName="" LastEditDate="2011-01-15T14:20:01.030" LastActivityDate="2011-01-15T14:20:01.030" CommentCount="3" />
  <row Id="6277" PostTypeId="2" ParentId="6268" CreationDate="2011-01-15T14:27:25.763" Score="1" ViewCount="" Body="&lt;p&gt;I'd have two two-panel plots, both have the xy plot on the left, and a histogram on the right. In the first plot, a horizontal line is placed at the mean of y and lines extend from this to each point, representing the residuals of y values from the mean. The histogram with this simply plots these residuals. Then in the next pair, the xy plot contains a line representing the linear fit and again vertical lines representing the residuals, which are represented in a histogram to the right. Keep x axis of the histograms constant to highlight the shift to lower values in the linear fit relative to the mean &quot;fit&quot;.&lt;/p&gt;&#xA;" OwnerUserId="364" LastActivityDate="2011-01-15T14:27:25.763" />
  <row Id="6278" PostTypeId="2" ParentId="6275" CreationDate="2011-01-15T14:46:59.737" Score="9" ViewCount="" Body="&lt;p&gt;This is a very large subject and there are many good books that cover it.  These are both good, but Cryer is my favorite of the two:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Cryer. &quot;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0387759581&quot; rel=&quot;nofollow&quot;&gt;Time Series Analysis: With Applications in R&lt;/a&gt;&quot; is a classic on the subject, updated to include R code.&lt;/li&gt;&#xA;&lt;li&gt;Shumway and Stoffer. &quot;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0387293175&quot; rel=&quot;nofollow&quot;&gt;Time Series Analysis and Its Applications: With R Examples&lt;/a&gt;&quot;.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;A good free resource is &lt;a href=&quot;http://zoonek2.free.fr/UNIX/48_R/15.html&quot; rel=&quot;nofollow&quot;&gt;Zoonekynd's ebook, especially the time series section&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My first suggestion for seeing the R packages would be the free ebook &lt;a href=&quot;https://www.rmetrics.org/ebooks-tseries&quot; rel=&quot;nofollow&quot;&gt;&quot;A Discussion of Time Series Objects for R in Finance&quot;&lt;/a&gt; from Rmetrics.  It gives lots of examples comparing the different time series packages and discusses some of the considerations, but it doesn't provide any theory.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Eric Zivot's &lt;a href=&quot;http://books.google.com/books?id=sxODP2l1mX8C&quot; rel=&quot;nofollow&quot;&gt;&quot;Modeling financial time series with S-PLUS&quot;&lt;/a&gt; and Ruey Tsay's &quot;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0471690740&quot; rel=&quot;nofollow&quot;&gt;Analysis of Financial Time Series&lt;/a&gt;&quot; (available in the TSA package on CRAN) are directed and financial time series but both provide good general references.  I strongly recommend looking at &lt;a href=&quot;http://faculty.chicagobooth.edu/ruey.tsay/teaching/&quot; rel=&quot;nofollow&quot;&gt;&lt;b&gt;Ruey Tsay's homepage&lt;/b&gt;&lt;/a&gt; because it covers all these topics, and provides the necessary R code.  In particular, look at the &lt;a href=&quot;http://faculty.chicagobooth.edu/ruey.tsay/teaching/bs41202/sp2009/&quot; rel=&quot;nofollow&quot;&gt;&quot;Analysis of Financial Time Series&quot;&lt;/a&gt;, and &lt;a href=&quot;http://faculty.chicagobooth.edu/ruey.tsay/teaching/mts/sp2009/&quot; rel=&quot;nofollow&quot;&gt;&quot;Multivariate Time Series Analysis&quot;&lt;/a&gt; courses.&lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2011-01-15T14:46:59.737" CommentCount="2" />
  <row Id="6279" PostTypeId="1" CreationDate="2011-01-15T14:48:52.947" Score="2" ViewCount="390" Body="&lt;p&gt;I am a very beginner of statistic. Recently a project require me to analyse data using logistic regression &amp;amp; SPSS within a specific time frame. Although I have read few books, but still very blur on how to start off. Can someone guide me through? What is the 1st ste and what next?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyway, I have started some. Once entered the data into SPSS, I have done crosstab (categorical IV), descriptive (continuous IV) and spearman correlation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, I proceed to test for nonlinearity by transforming into Ln which give me some problems. I have re-coded all zero cells to a small value (0.0001) to enable the Ln transformation. Then, I re-test the nonlinearity.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Question:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) The only solution for violation is to transform the variable from continuous to categorical? I got one violation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) One Exp(B) is extremely large (15203.835). What does this means? Why?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) There is one interaction has Exp(B) = 0.00. Why?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks. &lt;/p&gt;&#xA;" OwnerUserId="2793" LastActivityDate="2011-01-16T04:48:53.307" Title="Steps of data analysis using logistic regression" Tags="&lt;logistic-regression&gt;" AnswerCount="2" CommentCount="5" />
  <row Id="6280" PostTypeId="2" ParentId="6180" CreationDate="2011-01-15T15:02:55.903" Score="3" ViewCount="" Body="&lt;p&gt;You ask how to 'formally and usefully' present your conclusions&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Formally: Your answer is an accurate summary of some of the results from Brown et al. as I understand them.  (I note you do not offer their preferred small n method).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Usefully: I wonder who you audience is.  For professional statisticians, you could state your two intervals directly with only citations to the original papers - no further exposition needed.  For an applied audience, you would surely rather pick an interval on whatever substantive grounds you (or they) have, e.g. a preference for conservative coverage or good behavior for small proportions, etc., and just present that interval alone, noting its nominal and perhaps also its actual coverage much as you do above, perhaps with a footnote to the effect that other intervals are possible.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As it stands you offer a choice of intervals but not much explicit guidance for an applied audience to make use of that information.  In short, for that sort of audience I would suggest either more information about the implications of choice of interval.  Or less!&lt;/p&gt;&#xA;" OwnerUserId="1739" LastActivityDate="2011-01-15T15:02:55.903" CommentCount="3" />
  <row Id="6281" PostTypeId="1" AcceptedAnswerId="6284" CreationDate="2011-01-15T15:03:39.233" Score="2" ViewCount="233" Body="&lt;p&gt;I'm working through a practice problem for my Stats homework. We're using Confidence Intervals to find a range that the true mean lies within. I'm having trouble understanding how to find the required sample size to estimate the true mean within something like +- 0.5%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand how to work the problem when the range is given as a number, such as +- 0.5 mm. How do I handle percentages?&lt;/p&gt;&#xA;" OwnerUserId="2794" LastEditorUserId="2129" LastEditorDisplayName="" LastEditDate="2011-01-15T18:10:18.550" LastActivityDate="2011-01-18T10:51:39.927" Title="How to use Confidence Intervals to find the true mean within a percentage" Tags="&lt;confidence-interval&gt;&lt;homework&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="6284" PostTypeId="2" ParentId="6281" CreationDate="2011-01-15T18:37:14.230" Score="4" ViewCount="" Body="&lt;p&gt;I am not sure what kind of variable is being audited, so I give 2 alternatives:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;To be able to compute the required sample size to give an acceptable estimate to a &lt;strong&gt;continuous variable&lt;/strong&gt; (= given confidence interval) you have to know a few parameters: mean, standard deviation (and to be precise: population size). If you do not know these, you have to be able to give an accurate estimate to those (based on e.g. researches in the past). &#xA;$$n=\left(\frac{Z_{c}\sigma}{E}\right)^2,$$&#xA;where $n$ is sample size, $Z_{c}$ is choosen from standard normal distributon table based on $\alpha$ and $\sigma$ is the standard deviation.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I could image that the variable being examined is a &lt;strong&gt;discrete&lt;/strong&gt; one, and the confidence interval shows that how many percent of the population is about to choose one category based on the sample (proportion). That way the required sample size could be computed easily with:$$n=p(1-p)\left(\frac{Z_{c}}{E}\right)^2$$ where $n$ is sample size, $p$ is proportion in population, $Z_{c}$ is choosen from standard normal distributon table based on $\alpha$, and $E$ is the margin of error.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Note: you can find a lot of online calculators also (&lt;a href=&quot;http://www.rad.jhmi.edu/jeng/javarad/samplesize/&quot; rel=&quot;nofollow&quot;&gt;e.g.&lt;/a&gt;). Worth reading &lt;a href=&quot;http://www.ltcconline.net/greenl/courses/201/estimation/ciprop.htm&quot; rel=&quot;nofollow&quot;&gt;this article&lt;/a&gt; also.&lt;/p&gt;&#xA;" OwnerUserId="2714" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-01-16T07:38:37.187" LastActivityDate="2011-01-16T07:38:37.187" CommentCount="2" />
  <row Id="6285" PostTypeId="2" ParentId="6243" CreationDate="2011-01-15T20:28:57.713" Score="0" ViewCount="" Body="&lt;p&gt;Thanks @mbq and @onestop. After running some tests the calibration was a wash. Subtracting the means raised the margin or error to the point that the calibrated and none calibrated test results where indeterminately different.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;@mbq I will take your advice and reduce the critical value lookup to 30 &lt;em&gt;(as infinity)&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When comparing against other benchmarks I think I can even avoid the walsh t-test because the variances are so low &lt;em&gt;(all below 1)&lt;/em&gt;. Some examples are the variances of benchmarks are:&#xA;&lt;code&gt;1.1733873442408789e-16&lt;/code&gt;, &lt;code&gt;0.000012223589489868368&lt;/code&gt;, &lt;code&gt;3.772214786601029e-19&lt;/code&gt;, and &lt;code&gt;6.607725046958527e-16&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2634" LastActivityDate="2011-01-15T20:28:57.713" />
  <row Id="6286" PostTypeId="2" ParentId="6265" CreationDate="2011-01-15T21:30:57.260" Score="1" ViewCount="" Body="&lt;p&gt;You should consider using mixed-effect / multi-level models. The techniques used to fit these models work fine with unbalanced design, which is how it will interpret your missing data. As long as the data is missing at random, this is a reasonable way to proceed. SPSS is able to fit linear mixed-effect models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Mixed-effect models also allow continuous covariates, so you could add, e.g., age as a person-level predictor.&lt;/p&gt;&#xA;" OwnerUserId="2739" LastActivityDate="2011-01-15T21:30:57.260" CommentCount="0" />
  <row Id="6287" PostTypeId="1" AcceptedAnswerId="6289" CreationDate="2011-01-15T23:09:06.293" Score="3" ViewCount="147" Body="&lt;p&gt;I'm fooling around with threshold time series models.  While I was digging through what others have done, I ran across the CDC's site for flu data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.cdc.gov/flu/weekly/&quot; rel=&quot;nofollow&quot;&gt;http://www.cdc.gov/flu/weekly/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;About 1/3 of the way down the page is a graph titled &quot;Pneumonia and Influenza Mortality....&quot;.  It shows the actuals in red, and two black seasonal series.   The top seasonal series is labeled &quot;Epidemic Threshold&quot; and appears to be some constant percent/amount above the &quot;Seasonal Baseline&quot; series.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My first question is:   Is that really how they determine when to publicly say we're in an epidemic (some percent above baseline)?  It looks to me like they're in the noise range, not to mention the &quot;other factors&quot; influence that is obviously not accounted for in that baseline series.  To me, there are way too many false positives.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My second question is:   Can you point me to any real world examples/publications of threshold models (hopefully in R)?&lt;/p&gt;&#xA;" OwnerUserId="2775" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-01-16T12:23:18.243" LastActivityDate="2011-01-16T12:23:18.243" Title="Threshold models and flu epidemic recognition" Tags="&lt;r&gt;&lt;time-series&gt;&lt;epidemiology&gt;&lt;threshold&gt;&lt;models&gt;" AnswerCount="1" />
  <row Id="6288" PostTypeId="2" ParentId="6279" CreationDate="2011-01-15T23:48:40.987" Score="0" ViewCount="" Body="&lt;p&gt;Generally large beta coefficients signal multi-collinearity. You should look for marginals that are zero in your cross-tabulations. You should also pay attention to mpiktas's comment. Testing for linearity (and transforming to categorical) is not generally needed if you have been setting up your data correctly.&lt;/p&gt;&#xA;" OwnerUserId="2129" LastActivityDate="2011-01-15T23:48:40.987" CommentCount="6" />
  <row Id="6289" PostTypeId="2" ParentId="6287" CreationDate="2011-01-16T01:18:19.567" Score="3" ViewCount="" Body="&lt;p&gt;The CDC uses the epidemic threshold of &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;1.645 standard deviations above the baseline for that time of year.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The definition may have multiple sorts of detection or mortality endpints. (The one you are pointing to is pneumonia and influenza mortality. the lower black curve is not really a series, but rather a modeled seasonal mean, and the upper black curve is  1.645 sd's above that mean).&#xA;&lt;a href=&quot;http://www.cdc.gov/mmwr/PDF/ss/ss5107.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.cdc.gov/mmwr/PDF/ss/ss5107.pdf&lt;/a&gt;&lt;br&gt;&#xA;&lt;a href=&quot;http://www.cdc.gov/flu/weekly/pdf/overview.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.cdc.gov/flu/weekly/pdf/overview.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; pnorm(1.645)&#xA;[1] 0.950015&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So it's a 95% threshold. (And it does look as though about 1 out of 20 weeks are over the threshold. You pick your thresholds, not to be perfect, but to have the sensitivity you deem necessary.) The seasonal adjustment model appears to be sinusoidal. There is an &lt;a href=&quot;http://search.r-project.org/cgi-bin/namazu.cgi?query=seasonal+model+epidemic&amp;amp;max=100&amp;amp;result=normal&amp;amp;sort=score&amp;amp;idxname=functions&amp;amp;idxname=vignettes&amp;amp;idxname=views&quot; rel=&quot;nofollow&quot;&gt;R &quot;flubase&quot; package&lt;/a&gt; that should be consulted.&lt;/p&gt;&#xA;" OwnerUserId="2129" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-01-16T12:20:33.207" LastActivityDate="2011-01-16T12:20:33.207" CommentCount="3" />
  <row Id="6290" PostTypeId="2" ParentId="6268" CreationDate="2011-01-16T04:00:33.937" Score="1" ViewCount="" Body="&lt;p&gt;I think what you propose is good, but I would do it in three different examples&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) X and Y are completely unrelated.  Simply remove &quot;x&quot; from the r code that generates y (y1&amp;lt;-rnorm(50))&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) The example you posted (y2 &amp;lt;- x+rnorm(50))&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) The X are Y are the same variable.  Simply remove &quot;rnorm(50)&quot; from the r code that generates y (y3&amp;lt;-x)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This would more explicitly show how increasing the correlation decreases the variability in the residuals.  You would just need to make sure that the vertical axis doesn't change with each plot, which may happen if you're using default scaling.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So you could compare three plots r1 vs x, r2 vs x and r3 vs x.  I am using &quot;r&quot; to indicate the residuals from the fit using y1, y2, and y3 respectively.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My R skills in plotting are quite hopeless, so I can't offer much help here.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-01-16T04:00:33.937" />
  <row Id="6291" PostTypeId="2" ParentId="6279" CreationDate="2011-01-16T04:48:53.307" Score="1" ViewCount="" Body="&lt;p&gt;Hi lcl23,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The large value of &quot;B&quot; would be a coefficient of a variable usually called &quot;X&quot; in your model.  Usually, &quot;X&quot; has a real world meaning (could be income, could be a measured volume of something, etc.).  So the job is to interpret this &quot;B&quot; in terms of &quot;X&quot;.  The usual definition (in ordinary least squares) is that a ONE UNIT increase in &quot;X&quot; corresponds to a &quot;B&quot; increase in &quot;Y&quot; (where &quot;Y&quot; is the dependent variable which you are modeling).  It is similar (but not exact) when interpreting for a logistic regression: a ONE UNIT increase in &quot;X&quot; corresponds to a B increase in the log-odds (which is &quot;Y&quot; in this case).  Therefore EXP(B) tells you the proportional increase in the odds for a ONE UNIT increase in &quot;X&quot;.  So the question which may help is &quot;What does a ONE UNIT increase in X mean in the real world?&quot;  This may make the apparent extreme value seem more sensible.  Another thing to ask is what is the range of X values in your data? for if this is much smaller than 1 then EXP(B) is effectively extrapolating well beyond what you have observed in the &quot;X space&quot; (something which is generally not recommended because the relationship may be different).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Put more succinctly the numerical value of your betas is related to the scale at which you measure your X variables (or independent variables).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The easiest mathematical way to see this is the form of the simple ordinary least squares estimate which is &lt;/p&gt;&#xA;&#xA;&lt;p&gt;B=(standard deviation of Y) / (standard deviation of X) * (correlation between Y and X)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This result is not exact in logistic regression, but it does approximately happen.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would also recommend that replacing the observed proportions with (r_i+1)/(n_i+2) is not a bad way to go before plugging them into the logistic function, because it can help guard against creating extreme logit values (which are only an artifact of your choice of &quot;small number&quot;).  Extreme logit values can create outliers and influential points in the regression, and this can make your regression coefficients highly sensitive to these observations, or to be more precise, to your particular choice of &quot;small number&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This has been called &quot;Laplace's rule of succession&quot; (add 1 success and 1 failure to what was observed) and it effectively &quot;pulls&quot; the proportion back towards 1/2, with less &quot;pulling&quot; the greater n_i is.  In Bayesian language it corresponds to the prior information that it is possible for both of the binary outcomes to occur.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-01-16T04:48:53.307" CommentCount="1" />
  <row Id="6292" PostTypeId="2" ParentId="6265" CreationDate="2011-01-16T10:24:07.540" Score="3" ViewCount="" Body="&lt;p&gt;Billy,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the comment about the data being rejected if the light is below a certain intensity, does this correspond to why you have missing records? Because if you do, then you do not have total &quot;missingness&quot;, but rather &quot;censoring&quot; because you know that the response is below a certain threshold.  This does have an impact because the likelihood function becomes a product of the density functions for those observed value, and a product of the cumulative density functions for those values which were &quot;rejected&quot;.  I would say that this is a case of the data being &quot;Not Missing At Random&quot; (NMAR).  Basically this means the cause of the missingness is related to the actual value that is missing, which in this case, the value is below a certain threshold.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One thing I find interesting about NMAR is it often comes across as a &quot;bad&quot; thing.  My view is that it is actually a good thing (from a statistical point of view) because you haven't completely lost the record, it is still giving you some information.  The only bad thing is that the &quot;standard&quot; mathematics usually don't work as elegantly, and standard software can't be used as easily.  You just need to work harder to extract the information.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another thing you make reference to is the &quot;mean intensity&quot; of the light.  It seems odd to talk of &quot;mean&quot; when referring to a single measurement.  Or is this the mean of the light over the area of the spot?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Survival analysis with censored data may be a useful place to start, since this is a similar problem mathematically.  It may be a bit of a hurdle to &quot;translate&quot; the survival application into a relevant one for your analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My advice would be to start simple, and build complexity as you go.  It seems as though you have already done this a bit (by excluding the demographic data initially).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One way to do this is to do a simple one-way ANOVA only using complete cases with the classification over the 25 sites.  one-way unbalanced ANOVA is quite easy to implement (but not as a regression! lack of balance makes ANOVA different from its standard regression representation!).  It resembles the two sample t-test with different sample sizes.&#xA;Another option is to analyse the data 2 body sites at a time using complete &quot;paired&quot; cases.  This would give you a kind of &quot;ordering&quot; of the body types.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This analysis is all of the &quot;exploratory&quot; type, because it essentially &quot;throws away&quot; part of the information in exchange for a more simple view of the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It does seem as though this would be quite an involved analysis to take all of the information into account (particularly the censoring).  The multi-level approach suggested by @BR would be a useful approximation to this (by &quot;throwing away&quot; the threshold information).  The multiple imputation approach suggested by Christopher Aden is another way to go, where you get the advantage of the easier to understand ANOVA, while properly taking account of the uncertainty due to missing values.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-01-16T10:24:07.540" CommentCount="1" />
  <row Id="6293" PostTypeId="2" ParentId="6281" CreationDate="2011-01-16T11:31:53.067" Score="1" ViewCount="" Body="&lt;p&gt;It does seem a bit odd for this problem, because there does not appear to be a pivotal statistic or if there is, it isn't the usual Z or T statistic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's why I think this is the case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem of estimating the population mean, say $\mu$, to within $\pm $ 0.5% obviously depends on the value of $\mu$ (a pivotal statistic would NOT depend on $\mu$).  To estimate $\mu$ within an absolute amount, say $\pm $1, is independent of the actual value of $\mu$ (in the normally distributed case).  To put it another way, the width of the standard &quot;Z&quot; confidence interval does not depend on $\mu$, it only depends on the population standard deviation, say $\sigma$, the sample size n, and the level of confidence, expressed by the value Z.  You can call the length of this interval $ L=L(\sigma,n,Z)=\frac{2 \sigma Z}{\sqrt{n}} $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now we want an interval which is $0.01 \mu $ wide (equal length either side of $\mu$).  So the required equation that we need to solve is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ L=0.01 \mu=\frac{2 \sigma Z}{\sqrt{n}} $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Re-arranging for n gives&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ n = (\frac{2 \sigma Z}{0.01 \mu})^2 = 40,000 Z^2 (\frac{\sigma}{\mu})^2 $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using Z=1.96 to have a 95% CI gives&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ n = 153,664 * (\frac{\sigma}{\mu})^2 $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So that you need some prior information about the ratio $\frac{\sigma}{\mu}$ (by &quot;prior information&quot; I mean you need to know something about the ratio $\frac{\sigma}{\mu}$ in order to solve the problem).  If $\frac{\sigma}{\mu}$ is not known with certainty, then the &quot;optimal sample size&quot; also cannot be known with certainty.  The best way to go from here is to specify a probability distribution for $\frac{\sigma}{\mu}$ and then take the expected value of $(\frac{\sigma}{\mu})^2$ and put this into the above equation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What happens if we only require $\pm 0.005 $ (rather than $\pm 0.005 \mu$)  is that $\mu$ in the above equations for n disappears.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastEditorUserId="2392" LastEditorDisplayName="" LastEditDate="2011-01-18T10:51:39.927" LastActivityDate="2011-01-18T10:51:39.927" CommentCount="2" />
  <row Id="6294" PostTypeId="1" AcceptedAnswerId="6295" CreationDate="2011-01-16T11:51:10.763" Score="5" ViewCount="226" Body="&lt;p&gt;What analyses can be used to find an interaction effect in a 2-factor design, with one ordinal and one categorical factor, with binary-valued data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Specifically, are there any types of analyses that are capable of dealing with a 2 factor design 5(ordinal) x 2(categorical), where the outcomes are either true or false?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One could do a 5x2 chi square analysis, but it loses the power of the ordinality of the one factor. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively, one could run independent logistic/probit regressions, but then there is the question of testing for the interaction effect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any thoughts or suggestions that would put me in the right direction would be helpful.&lt;/p&gt;&#xA;" OwnerUserId="2800" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-01-16T12:26:17.087" LastActivityDate="2011-01-16T13:54:38.253" Title="Interaction between ordinal and categorical factor" Tags="&lt;interaction&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="1" />
  <row Id="6295" PostTypeId="2" ParentId="6294" CreationDate="2011-01-16T12:43:29.283" Score="3" ViewCount="" Body="&lt;p&gt;I'd stick with logistic or probit regression, enter both factors as covariates, but enter the ordinal factor as if it was continuous. To test for interaction, do a &lt;a href=&quot;http://en.wikipedia.org/wiki/Likelihood-ratio_test&quot; rel=&quot;nofollow&quot;&gt;likelihood-ratio test&lt;/a&gt; comparing models with and without an interaction between the two factors. This test will have a single degree of freedom and therefore retain good power. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;After using this to decide whether or not you want to include an interaction between the two factors, you can then move on to decide how best to code the 5-level factor in your final model. It could make sense to keep treating it as if it were continuous, or you might wish to code it as four dummy (indicator) variables, or you choose to collapse it into fewer levels, or use some other type of contrast. The choice probably depends on the scientific meaning of the model and its intended use, as well as the fit of the various models.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-01-16T12:43:29.283" CommentCount="5" />
  <row Id="6297" PostTypeId="2" ParentId="2715" CreationDate="2011-01-16T13:48:53.937" Score="6" ViewCount="" Body="&lt;p&gt;Always ask yourself &quot;what do these results mean and how will they be used?&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Usually the purpose of using statistics is to assist in making decisions under uncertainty.  So it is important to have at the front of your mind &quot;What decisions will be made as a result of this analysis and how will this analysis influence these decisions?&quot; (e.g. publish an article, recommend a new method be used, provide $X in funding to Y, get more data, report an estimated quantity as E, etc.etc.....)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you don't feel that there is any decision to be made, then one wonders why you are doing the analysis in the first place (as it is quite expensive to do analysis).  I think of statistics as a &quot;nuisance&quot; in that it is a means to an end, rather than an end itself.  In my view we only quantify uncertainty so that we can use this to make decisions which account for this uncertainty in a precise way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think this is one reason why keeping things simple is a good policy in general, because it is usually much easier to relate a simple solution to the real world (and hence to the environment in which the decision is being made) than the complex solution.  It is also usually easier to understand the limitations of the simple answer.  You then move to the more complex solutions when you understand the limitations of the simple solution, and how the complex one addresses them.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-01-16T13:48:53.937" CommunityOwnedDate="2011-01-16T13:48:53.937" />
  <row Id="6298" PostTypeId="1" AcceptedAnswerId="10760" CreationDate="2011-01-16T14:01:00.537" Score="12" ViewCount="369" Body="&lt;p&gt;&lt;a href=&quot;http://code.google.com/intl/pl-PL/apis/predict/&quot; rel=&quot;nofollow&quot;&gt;Google Prediction API&lt;/a&gt; is a cloud service where user can submit some training data to train some mysterious classifier and later ask it to classify incoming data, for instance to implement spam filters or predict user preferences.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But what is behind the scenes?&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2011-05-13T10:24:17.390" Title="What is behind Google Prediction API?" Tags="&lt;machine-learning&gt;" AnswerCount="1" CommentCount="4" FavoriteCount="4" />
  <row Id="6299" PostTypeId="2" ParentId="5903" CreationDate="2011-01-16T14:47:21.463" Score="1" ViewCount="" Body="&lt;p&gt;Hi Ringold,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given that simple linear regression is analytically identical between classical and Bayesian analysis with Jeffrey's prior, both of which are analytic, it seems a bit odd to resort to a numerical method such as MCMC to do the Bayesian analysis.  MCMC is just a numerical integration tool, which allows Bayesian methods to be used in more complicated problems which are analytically intractable, just the same as Newton-Rhapson or Fisher Scoring are numerical methods for solving classical problems which are intractable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The posterior distribution p(b|y) using the Jeffrey's prior p(a,b,s) proportional to 1/s (where s is the standard deviation of the error) is a student t distribution with location b_ols, scale se_b_ols (&quot;ols&quot; for &quot;ordinary least squares&quot; estimate), and n-2 degrees of freedom.  But the sampling distribution of b_ols is also a student t with location b, scale se_b_ols, and n-2 degrees of freedom.  Thus they are identical except that b and b_ols have been swapped, so when it comes to creating the interval, the &quot;est +- bound&quot; of the confidence interval gets reversed to a &quot;est -+ bound&quot; in the credible interval.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the confidence interval and credible interval are analytically identical, and it matters not which method is used (provided there is no additional prior information) - so take the method which is computationally cheaper (e.g. the one with fewer matrix inversions).  What your result with MCMC shows is that the particular approximation used with MCMC gives a credible interval which is too wide compared to the exact analytic credible interval.  This is probably a good thing (although we would want the approximation to be better) that the approximate Bayesian solution appears more conservative than the exact Bayesian solution.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastEditorUserId="2392" LastEditorDisplayName="" LastEditDate="2011-01-16T14:52:45.937" LastActivityDate="2011-01-16T14:52:45.937" CommentCount="1" />
  <row Id="6300" PostTypeId="2" ParentId="6225" CreationDate="2011-01-16T15:35:37.147" Score="0" ViewCount="" Body="&lt;p&gt;For me, the decision theoretical framework presents the easiest way to understand the &quot;null hypothesis&quot;.  It basically says that there must be at least two alternatives: the Null hypothesis, and at least one alternative.  Then the &quot;decision problem&quot; is to accept one of the alternatives, and reject the others (although we need to be precise about what we mean by &quot;accepting&quot; and &quot;rejecting&quot; the hypothesis).  I see the question of &quot;can we prove the null hypothesis?&quot; as analogous to &quot;can we always make the correct decision?&quot;.  From a decision theory perspective the answer is clearly yes if &lt;/p&gt;&#xA;&#xA;&lt;p&gt;1)there is no uncertainty in the decision making process, for then it is a mathematical exercise to work out what the correct decision is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2)we accept all the other premises/assumptions of the problem.  The most critical one (I think) is that the hypothesis we are deciding between are exhaustive, and one (and only one) of them must be true, and the others must be false.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From a more philosophical standpoint, it is not possible to &quot;prove&quot; anything, in the sense that the &quot;proof&quot; depends entirely on the assumptions / axioms which lead to that &quot;proof&quot;.  I see proof as a kind of logical equivalence rather than a &quot;fact&quot; or &quot;truth&quot; in the sense that if the proof is wrong, the assumptions which led to it are also wrong.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Applying this to the &quot;proving the null hypothesis&quot; I can &quot;prove&quot; it to be true by simply assuming that it is true, or by assuming that it is true if certain conditions are meet (such as the value of a statistic).&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-01-16T15:35:37.147" />
  <row Id="6301" PostTypeId="2" ParentId="4663" CreationDate="2011-01-16T17:42:01.520" Score="6" ViewCount="" Body="&lt;p&gt;When used in stage-wise mode, the LARS algorithm is a greedy method that does not yield a provably consistent estimator (in other, it does not converge on average to a stable result when you increase the number of samples). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the contrary, the LASSO (and thus the LARS algorithm when used in LASSO mode) solves a convex data fitting problem. In particular, this problem (the L1 penalized linear estimator) has plenty of nice proved properties (consistency, sparsistency).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would thus try to always use the LARS in LASSO mode (or use another solver for LASSO), unless you have very good reasons to prefer stage wise.&lt;/p&gt;&#xA;" OwnerUserId="1265" LastActivityDate="2011-01-16T17:42:01.520" CommentCount="1" />
  <row Id="6302" PostTypeId="1" AcceptedAnswerId="6303" CreationDate="2011-01-16T20:56:43.303" Score="6" ViewCount="297" Body="&lt;p&gt;&lt;em&gt;NOTE: I purposely did not label the axis due to pending publications. The line colors represent the same data in all three plots.&lt;/em&gt;&lt;br&gt;&#xA;I fitted my data using a negative binomial distribution to generate a pdf. I am happy with the pdf and meets my research needs. PDF plot:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/ttDnS.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;For when reporting the CDF, should I use the empirical or fitted CDF?&lt;/strong&gt; There are slight differences between the empirical and fitted CDF, specifically at x = 40, the yellow and cyan lines intersect in the empirical distribution, but not the fitted.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Empirical:&lt;br&gt;&#xA;&lt;img src=&quot;http://i.imgur.com/j0ELI.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Negative Binomial CDF:&#xA;&lt;img src=&quot;http://i.imgur.com/13yyc.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="559" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-01-16T23:09:09.193" LastActivityDate="2011-01-18T12:42:18.770" Title="Use Empirical CDF vs Distribution CDF?" Tags="&lt;distributions&gt;&lt;data-visualization&gt;&lt;pdf&gt;&lt;cdf&gt;" AnswerCount="2" CommentCount="5" />
  <row Id="6303" PostTypeId="2" ParentId="6302" CreationDate="2011-01-16T23:07:41.130" Score="5" ViewCount="" Body="&lt;p&gt;Personally, I'd favour instead showing the &lt;em&gt;fit&lt;/em&gt; of the theoretical to the empirical distribution using a set of &lt;a href=&quot;http://en.wikipedia.org/wiki/P-P_plot&quot; rel=&quot;nofollow&quot;&gt;P-P plots&lt;/a&gt; or &lt;a href=&quot;http://en.wikipedia.org/wiki/Q-Q_plot&quot; rel=&quot;nofollow&quot;&gt;Q-Q plots&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="449" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-01-17T11:19:01.070" LastActivityDate="2011-01-17T11:19:01.070" CommentCount="3" />
  <row Id="6304" PostTypeId="1" AcceptedAnswerId="6307" CreationDate="2011-01-17T03:34:38.283" Score="9" ViewCount="210" Body="&lt;p&gt;Let $t_i$ be drawn i.i.d from a Student t distribution with $n$ degrees of freedom, for moderately sized $n$ (say less than 100). Define&#xA;$$T = \sum_{1\le i \le k} t_i^2$$&#xA;Is $T$ distributed nearly as a chi-square with $k$ degrees of freedom? Is there something like the Central Limit Theorem for the sum of squared random variables?&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2011-01-17T21:54:42.793" Title="What is the sum of squared t variates?" Tags="&lt;chi-squared&gt;&lt;central-limit-theorem&gt;&lt;students-t&gt;" AnswerCount="3" CommentCount="2" FavoriteCount="1" />
  <row Id="6305" PostTypeId="2" ParentId="6304" CreationDate="2011-01-17T04:07:32.863" Score="5" ViewCount="" Body="&lt;p&gt;I'll answer second question. The central limit theorem is for any iid sequence, squared or not squared. So in your case if $k$ is sufficiently large we have&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\dfrac{T-kE(t_1)^2}{\sqrt{kVar(t_1^2)}}\sim N(0,1)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $Et_1^2$ and $Var(t_1^2)$ is respectively the mean and variance of squared Student t distribution with $n$ degrees of freedom. Note that $t_1^2$ is distributed as F distribution with $1$ and $n$ degrees of freedom. So we can grab the formulas for mean and variance from &lt;a href=&quot;http://en.wikipedia.org/wiki/F-distribution&quot; rel=&quot;nofollow&quot;&gt;wikipedia page&lt;/a&gt;. The final result then is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\dfrac{T-k\frac{n}{n-2}}{\sqrt{k\frac{2n^2(n-1)}{(n-2)^2(n-4)}}}\sim N(0,1)$&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-01-17T04:07:32.863" CommentCount="5" />
  <row Id="6306" PostTypeId="1" CreationDate="2011-01-17T09:18:09.130" Score="18" ViewCount="362" Body="&lt;p&gt;Although this question is somewhat subjective, I hope it&#xA;qualifies&#xA;as a good subjective question according to the &lt;a href=&quot;http://blog.stackoverflow.com/2010/09/good-subjective-bad-subjective/&quot; rel=&quot;nofollow&quot;&gt;faq guidelines&lt;/a&gt;.&#xA;It is based on a question that Olle Häggström asked me a year ago&#xA;and although I have some thoughts about it I do not have a&#xA;definite answer and I would appreciate some help from others.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Background:&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;A paper entitled &quot;Equidistant letter sequences in the book of Genesis,&quot; by&#xA;D. Witztum, E. Rips and Y. Rosenberg&#xA;made the extraordinary claim that the Hebrew text of the&#xA;Book of Genesis encodes events which&#xA;did not occur until millennia after the text was written. The paper was&#xA;published by &quot;Statistical Science&quot; in 1994 (Vol. 9 429-438), and was offered&#xA;as a &quot;challenging puzzle&quot; whose solution may contribute to the field of statistics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In reply, &lt;a href=&quot;http://cs.anu.edu.au/~bdm/dilugim/StatSci/StatSci.pdf&quot; rel=&quot;nofollow&quot;&gt;another paper&lt;/a&gt; entitled &quot;Solving the Bible code puzzle&quot;&#xA;by B. McKay, D. Bar-Natan, M. Bar-Hillel and G. Kalai appeared in&#xA;Statistical science in 1999 (Vol. 14 (1999) 150-173). The new paper&#xA;argues that Witztum,&#xA;Rips and Rosenberg's case is fatally defective,&#xA;indeed that their result merely reflects on&#xA;the choices made in designing their experiment and&#xA;collecting the data for it. The paper presents&#xA;extensive evidence in support of that conclusion.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(My own interests which are summarized in Section 8&#xA;of our paper are detailed&#xA;in another &lt;a href=&quot;http://www.ma.huji.ac.il/~kalai/ratio.ps&quot; rel=&quot;nofollow&quot;&gt;technical report&lt;/a&gt; with Bar Hillel and Mckay&#xA;entitled &quot;The two famous rabbis experiments: how&#xA;similar is too similar?&quot; See also &lt;a href=&quot;http://www.ma.huji.ac.il/~kalai/bc.html&quot; rel=&quot;nofollow&quot;&gt;this site&lt;/a&gt;.)&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;The questions:&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Olle Häggström's specific question was: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&quot;I once suggested that your paper&#xA;  might be useful in a statistics course&#xA;  on advanced undergraduate level, for&#xA;  the purpose of illustrating the&#xA;  pitfalls of data mining and related&#xA;  techniques. Would you agree?&quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In addition to Olle's question let me ask a more general question.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Is there something related to&#xA;  statistics that we have learned,&#xA;  (including perhaps some interesting&#xA;  questions to ask) from the Bible Code&#xA;  episode.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Just to make it clear, my question is restricted to&#xA;insights &lt;strong&gt;related to statistics&lt;/strong&gt; and not to any other aspect of this episode.&lt;/p&gt;&#xA;" OwnerUserId="1148" LastActivityDate="2011-01-23T22:22:22.243" Title="Are there statistical lessons from the &quot;Bible Code&quot; episode." Tags="&lt;hypothesis-testing&gt;&lt;data-mining&gt;" CommentCount="3" FavoriteCount="4" />
  <row Id="6307" PostTypeId="2" ParentId="6304" CreationDate="2011-01-17T10:44:06.310" Score="8" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Answering the first question.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We could start from the fact noted by mpiktas, that $t^2 \sim F(1, n)$. And then try a more simple step at first - search for the distribution of a sum of two random variables distributed by $F(1,n)$. This could be done either by calculating the convolution of two random variables, or calculating the product of their characteristic functions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;a href=&quot;http://cowles.econ.yale.edu/P/cp/p05b/p0560.pdf&quot; rel=&quot;nofollow&quot;&gt;article&lt;/a&gt; by P.C.B. Phillips shows that my first guess about &quot;[confluent] hypergeometric functions involved&quot; was indeed true. It means that the solution will be not trivial, and the brute-force is complicated, but necessary condition to answer your question. So since $n$ is fixed and you sum up t-distributions, we can't say for sure what the final result will be. Unless someone has a good skill playing with products of confluent hypergeometric functions.&lt;/p&gt;&#xA;" OwnerUserId="2645" LastActivityDate="2011-01-17T10:44:06.310" CommentCount="1" />
  <row Id="6308" PostTypeId="1" AcceptedAnswerId="6310" CreationDate="2011-01-17T11:11:10.560" Score="10" ViewCount="403" Body="&lt;p&gt;I am referring to this article: &lt;a href=&quot;http://www.nytimes.com/2011/01/11/science/11esp.html&quot; rel=&quot;nofollow&quot;&gt;http://www.nytimes.com/2011/01/11/science/11esp.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Consider the following experiment. Suppose there was reason to believe that a coin was slightly weighted toward heads. In a test, the coin comes up heads 527 times out of 1,000.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Is this significant evidence that the&#xA;  coin is weighted?&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Classical analysis says yes. With a&#xA;  fair coin, the chances of getting 527&#xA;  or more heads in 1,000 flips is less&#xA;  than 1 in 20, or 5 percent, the&#xA;  conventional cutoff. To put it another&#xA;  way: the experiment finds evidence of&#xA;  a weighted coin “with 95 percent&#xA;  confidence.”&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Yet many statisticians do not buy it.&#xA;  One in 20 is the probability of&#xA;  getting any number of heads above 526&#xA;  in 1,000 throws. That is, it is the&#xA;  sum of the probability of flipping&#xA;  527, the probability of flipping 528,&#xA;  529 and so on.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;But the experiment did not find all of&#xA;  the numbers in that range; it found&#xA;  just one — 527. It is thus more&#xA;  accurate, these experts say, to&#xA;  calculate the probability of getting&#xA;  that one number — 527 — if the coin is&#xA;  weighted, and compare it with the&#xA;  probability of getting the same number&#xA;  if the coin is fair.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Statisticians can show that this ratio&#xA;  cannot be higher than about 4 to 1,&#xA;  according to Paul Speckman, a&#xA;  statistician, who, with Jeff Rouder, a&#xA;  psychologist, provided the example.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;First question:&lt;/strong&gt; This is new to me. Has anybody a reference  where I can find the exact calculation and/or can YOU help me by giving me the exact calculation yourself and/or can you point me to some material where I can find similar examples?&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Bayes devised a way to update the&#xA;  probability for a hypothesis as new&#xA;  evidence comes in.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;So in evaluating the strength of a&#xA;  given finding, Bayesian (pronounced&#xA;  BAYZ-ee-un) analysis incorporates&#xA;  known probabilities, if available,&#xA;  from outside the study.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;It might be called the “Yeah, right”&#xA;  effect. If a study finds that kumquats&#xA;  reduce the risk of heart disease by 90&#xA;  percent, that a treatment cures&#xA;  alcohol addiction in a week, that&#xA;  sensitive parents are twice as likely&#xA;  to give birth to a girl as to a boy,&#xA;  the Bayesian response matches that of&#xA;  the native skeptic: Yeah, right. The&#xA;  study findings are weighed against&#xA;  what is observable out in the world.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;In at least one area of medicine —&#xA;  diagnostic screening tests —&#xA;  researchers already use known&#xA;  probabilities to evaluate new&#xA;  findings. For instance, a new&#xA;  lie-detection test may be 90 percent&#xA;  accurate, correctly flagging 9 out of&#xA;  10 liars. But if it is given to a&#xA;  population of 100 people already known&#xA;  to include 10 liars, the test is a lot&#xA;  less impressive.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;It correctly identifies 9 of the 10&#xA;  liars and misses one; but it&#xA;  incorrectly identifies 9 of the other&#xA;  90 as lying. Dividing the so-called&#xA;  true positives (9) by the total number&#xA;  of people the test flagged (18) gives&#xA;  an accuracy rate of 50 percent. The&#xA;  “false positives” and “false&#xA;  negatives” depend on the known rates&#xA;  in the population.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Second question:&lt;/strong&gt; How do you exactly judge if a new finding is &quot;real&quot; or not with this method? And: Isn't this as arbitrary as the 5%-barrier because of the use of some preset prior probability?&lt;/p&gt;&#xA;" OwnerUserId="230" LastActivityDate="2011-02-08T20:25:07.877" Title="Article about misuse of statistical method in NYTimes" Tags="&lt;hypothesis-testing&gt;&lt;bayesian&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="3" />
  <row Id="6309" PostTypeId="1" AcceptedAnswerId="6311" CreationDate="2011-01-17T11:25:59.540" Score="6" ViewCount="250" Body="&lt;p&gt;I was reading through a paper and I saw a table with a comparison between PPV (Positive Predictive Value) and NPV (Negative Predictive Value). They did some kind of statistical test for them, this is a sketch of the table:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;PPV    NPV    p-value&#xA;65.9   100    &amp;lt; 0.00001&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Every rows refers to a particular contingency table. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What kind of hypothesis test did they do?&#xA;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="2719" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-01-17T14:54:17.517" LastActivityDate="2011-01-17T14:54:17.517" Title="Statistical test for Positive and Negative Predictive Value" Tags="&lt;epidemiology&gt;&lt;contingency-tables&gt;&lt;p-values&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="6310" PostTypeId="2" ParentId="6308" CreationDate="2011-01-17T13:22:38.390" Score="19" ViewCount="" Body="&lt;p&gt;I will answer the first question in detail.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;With a fair coin, the chances of&#xA;  getting 527 or more heads in 1,000&#xA;  flips is less than 1 in 20, or 5&#xA;  percent, the conventional cutoff.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;For a fair coin the number of heads in 1000 trials follows the &lt;a href=&quot;http://en.wikipedia.org/wiki/Binomial_distribution&quot; rel=&quot;nofollow&quot;&gt;binomial distribution&lt;/a&gt; with number of trials $n=1000$ and probability $p=1/2$. The probability of getting more than 527 heads is then &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(B(1000,1/2)&amp;gt;=527)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This can be calculated with any statistical software package. R gives us&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; pbinom(526,1000,1/2,lower.tail=FALSE)&#xA;   0.04684365&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So the probability that with fair coin we will get more than 526 heads is approximately 0.047, which is close to 5% cuttoff mentioned in the article.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The following statement&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;To put it another way: the experiment&#xA;  finds evidence of a weighted coin&#xA;  “with 95 percent confidence.”&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;is debatable. I would be reluctant to say it, since 95% confidence can be interpreted in several ways.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Next we turn to &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;But the experiment did not find all of&#xA;  the numbers in that range; it found&#xA;  just one — 527. It is thus more&#xA;  accurate, these experts say, to&#xA;  calculate the probability of getting&#xA;  that one number — 527 — if the coin is&#xA;  weighted, and compare it with the&#xA;  probability of getting the same number&#xA;  if the coin is fair.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Here we compare two events $B(1000,1/2)=527$ -- fair coin, and $B(1000,p)=527$ -- weighted coin. Substituting the &lt;a href=&quot;http://en.wikipedia.org/wiki/Binomial_distribution&quot; rel=&quot;nofollow&quot;&gt;formulas&lt;/a&gt; for probabilities of these events and noting that the binomial coefficient cancels out we get&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{P(B(1000,p)=527)}{P(B(1000,1/2)=527)}=\frac{p^{527}(1-p)^{473}}{(1/2)^{1000}}.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a function of $p$, thus we cand find minima or maxima of it. From the article we may infer that we need maxima:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Statisticians can show that this ratio&#xA;  cannot be higher than about 4 to 1,&#xA;  according to Paul Speckman, a&#xA;  statistician, who, with Jeff Rouder, a&#xA;  psychologist, provided the example.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;To make maximisation easier take logarithm of ratio, calculate the derivative with respect to $p$ and equate it to zero. The solution will be&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p=\frac{527}{1000}.$$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can check that it is really a maximum using &lt;a href=&quot;http://en.wikipedia.org/wiki/Second_derivative_test&quot; rel=&quot;nofollow&quot;&gt;second derivative test&lt;/a&gt; for example. Substituting it to the formula we get&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{(527/1000)^{527}(473/1000)^{473}}{(1/2)^{1000}}\approx 4.3$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the ratio is 4.3 to 1, which agrees with the article.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-01-17T19:48:13.903" LastActivityDate="2011-01-17T19:48:13.903" CommentCount="6" />
  <row Id="6311" PostTypeId="2" ParentId="6309" CreationDate="2011-01-17T13:57:31.230" Score="8" ViewCount="" Body="&lt;p&gt;Assuming a cross-classification like the one shown below (here, for a screening instrument)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/HCmDE.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;we can define four measures of screening accuracy and predictive power:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;em&gt;Sensitivity&lt;/em&gt; (se), a/(a + c), i.e. the probability of the screen providing a positive result given that disease is present;&lt;/li&gt;&#xA;&lt;li&gt;&lt;em&gt;Specificity&lt;/em&gt; (sp), d/(b + d), i.e. the probability of the screen providing a negative result given that disease is absent;&lt;/li&gt;&#xA;&lt;li&gt;&lt;em&gt;Positive predictive value&lt;/em&gt; (PPV), a/(a+b), i.e. the probability of patients with positive test results who are correctly diagnosed (as positive);&lt;/li&gt;&#xA;&lt;li&gt;&lt;em&gt;Negative predictive value&lt;/em&gt; (NPV), d/(c+d), i.e. the probability of patients with negative test results who are correctly diagnosed (as negative).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Each four measures are simple proportions computed from the observed data. A suitable statistical test would thus be a &lt;a href=&quot;http://en.wikipedia.org/wiki/Binomial_test&quot; rel=&quot;nofollow&quot;&gt;binomial (exact) test&lt;/a&gt;, which should be available in most statistical packages, or many online calculators. The tested hypothesis is whether the observed proportions significantly differ from 0.5 or not. I found, however, more interesting to provide confidence intervals rather than a single significance test, since it gives an information about the precision of measurement. Anyway, for reproducing the results you shown, you need to know the total margins of your two-way table (you only gave the PPV and NPV as %).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an example, suppose that we observe the following data (the CAGE questionnaire is a screening questionnaire for alcohol):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/Tsz0E.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;then in R the PPV would be computed as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; binom.test(99, 142)&#xA;&#xA;    Exact binomial test&#xA;&#xA;data:  99 and 142 &#xA;number of successes = 99, number of trials = 142, p-value = 2.958e-06&#xA;alternative hypothesis: true probability of success is not equal to 0.5 &#xA;95 percent confidence interval:&#xA; 0.6145213 0.7714116 &#xA;sample estimates:&#xA;probability of success &#xA;             0.6971831 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you are using SAS, then you can look at the Usage Note 24170: &lt;a href=&quot;http://support.sas.com/kb/24/170.html&quot; rel=&quot;nofollow&quot;&gt;How can I estimate sensitivity, specificity, positive and negative predictive values, false positive and negative probabilities, and the likelihood ratios?&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To compute confidence intervals, the gaussian approximation, $p \pm 1.96 \times \sqrt{p(1-p)/n}$ (1.96 being the quantile of the standard normal distribution at $p=0.975$ or $1-\alpha/2$ with $\alpha=5$%), is used in practice, especially when the proportions are quite small or large (which is often the case here).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For further reference, you can look at&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Newcombe, RG. &lt;a href=&quot;http://www.stats.org.uk/statistical-inference/Newcombe1998.pdf&quot; rel=&quot;nofollow&quot;&gt;Two-Sided Confidence&#xA;  Intervals for the Single Proportion:&#xA;  Comparison of Seven Methods&lt;/a&gt;.&#xA;  &lt;em&gt;Statistics in Medicine&lt;/em&gt;, 17, 857-872 (1998).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-01-17T14:10:13.687" LastActivityDate="2011-01-17T14:10:13.687" CommentCount="4" />
  <row Id="6312" PostTypeId="1" CreationDate="2011-01-17T14:15:04.320" Score="10" ViewCount="153" Body="&lt;p&gt;A signal detection experiment typically presents the observer (or diagnostic system) with either a signal or a non-signal, and the observer is asked to report whether they think the presented item is a signal or non-signal. Such experiments yield data that fill a 2x2 matrix:&#xA;&lt;img src=&quot;http://i.imgur.com/JktFQ.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Signal detection theory represents such data as representing a scenario where the &quot;signal/non-signal&quot; decision is based on a continuum of signal-ness on which the signal trials generally have a higher value than the non-signal trials, and the observer simply chooses a criterion value above which they will report &quot;signal&quot;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/Vb2UM.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the diagram above, the green and red distributions represents the &quot;signal&quot; and &quot;non-signal&quot; distributions, respectively and the grey line represents a given observer's chosen criterion. To the right of the grey line, the area under the green curve represents the hits and the area under the red curve represents the false alarms; to the left of the grey line, the area under the green curve represens misses and the area under the red curve represents correct rejections.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As may be imagined, according to this model, the proportion of responses that fall into each cell of the 2x2 table above is determined by:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The relative proportion of trials sampled from the green and red distributions (base rate)&lt;/li&gt;&#xA;&lt;li&gt;The criterion chosen by the observer&lt;/li&gt;&#xA;&lt;li&gt;The separation between the distributions&lt;/li&gt;&#xA;&lt;li&gt;The variance of each distribution&lt;/li&gt;&#xA;&lt;li&gt;Any departure from equality of variance between distributions (equality of variance is depicted above)&lt;/li&gt;&#xA;&lt;li&gt;The shape of each distribution (both are Gaussian above)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Often the influence of #5 and #6 can only be assessed by getting the observer to make decisions across a number of different criteria levels, so we'll ignore that for now. Additionally, #3 and #4 only make sense relative to one another (eg. how big is the separation relative to the variability of the distributions?), summarized by a measure of  &quot;discriminability&quot; (also known as d'). Thus, signal detection theory proscribes estimation of two properties from signal detection data: criterion &amp;amp; discriminability.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I have often noticed that research reports (particularly from the medical field) fail to apply the signal detection framework and instead attempt to analyze quantities such as &quot;Positive predictive value&quot;, &quot;Negative predictive value&quot;, &quot;Sensitivity&quot;, and &quot;Specificity&quot;, all of which represent different marginal values from the 2x2 table above (&lt;a href=&quot;http://en.wikipedia.org/wiki/Positive_predictive_value&quot; rel=&quot;nofollow&quot;&gt;see here for elaboration&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What utility do these marginal properties provide? My inclination is to disregard them completely because they confound the theoretically independent influences of criterion and discriminability, but possibly I simply lack the imagination to consider their benefits.&lt;/p&gt;&#xA;" OwnerUserId="364" LastActivityDate="2011-01-18T17:29:07.060" Title="Is it valid to analyze signal detection data without employing metrics derived from signal detection theory?" Tags="&lt;diagnostic&gt;&lt;signal-detection&gt;" AnswerCount="3" CommentCount="2" FavoriteCount="1" />
  <row Id="6314" PostTypeId="1" AcceptedAnswerId="10976" CreationDate="2011-01-17T15:12:50.590" Score="11" ViewCount="394" Body="&lt;p&gt;In 1999, Beyer et al. asked,&#xA;&lt;a href=&quot;http://www.cis.temple.edu/~vasilis/Courses/CIS750/Papers/beyer99when_17.pdf&quot; rel=&quot;nofollow&quot;&gt;When is &quot;Nearest Neighbor&quot; meaningful?&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there better ways of analyzing and visualizing&#xA;the effect of distance flatness on NN search since 1999?&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Does [a given] data set provide meaningful answers to the 1-NN problem?&#xA;  The 10-NN problem? The 100-NN problem?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;How would you experts approach this question today?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;hr&gt;&#xA;Edits Monday 24 Jan:  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How about &quot;distance whiteout&quot; as a shorter name for &quot;distance flatness &#xA;with increasing dimension&quot; ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An easy way to look at &quot;distance whiteout&quot;&#xA;is to run 2-NN,&#xA;and plot distances to the nearest neighbor and second-nearest neighbors.&#xA;The plot below shows dist&lt;sub&gt;1&lt;/sub&gt; and dist&lt;sub&gt;2&lt;/sub&gt;&#xA;for a range of nclusters and dimensions, by Monte Carlo.&#xA;This example shows pretty good distance contrast&#xA;for the scaled absolute difference |dist&lt;sub&gt;2&lt;/sub&gt; - dist&lt;sub&gt;1&lt;/sub&gt;|.&#xA;(The relative differences |dist&lt;sub&gt;2&lt;/sub&gt; / dist&lt;sub&gt;1&lt;/sub&gt;|&#xA;&amp;rarr; 1 as dimension &amp;rarr; &amp;infin;, so become useless.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Whether absolute errors or relative errors should be used&#xA;in a given context&#xA;depends of course on the &quot;real&quot; noise present: difficult.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suggestion: always run 2-NN;&#xA;2 neighbors are useful when they're close, and useful when not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/N23Vs.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="557" LastEditorUserId="557" LastEditorDisplayName="" LastEditDate="2011-01-24T17:13:36.567" LastActivityDate="2011-05-19T18:10:44.623" Title="When is &quot;Nearest Neighbor&quot; meaningful, today?" Tags="&lt;machine-learning&gt;&lt;k-nearest-neighbour&gt;" AnswerCount="2" CommentCount="5" FavoriteCount="8" />
  <row Id="6315" PostTypeId="2" ParentId="6312" CreationDate="2011-01-17T15:24:33.570" Score="3" ViewCount="" Body="&lt;p&gt;The Positive Predictive Influence (PPV) is not a good measure, not only because it confounds both mechanisms (discriminability and response bias), but also because of item base-rates. It is preferable to use the posterior probabilities, like P(signal|&quot;yes&quot;), which account for item base-rates:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(signal|yes) = \frac{P(signal)P(Hit)}{P(signal)P(Hit)+P(noise)P(False Alarm)}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;but... what is it good for?? well, it is useful for adjusting the response criteria in order to maximize/minimize a the probability of a specific outcome. So, it is complementary to the sensitivity and response bias measures in the sense that it helps to summarize the outcomes of changes in response bias.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A word of advice: if you are sticking with 2x2 outcome matrix that basically only allows you to get a sensitivity measure like d', don't even bother with SDT and just use Hits-False Alarms. Both measures (d' and (H-F)) have a correlation of .96 (no matter what BS detection theorists might come up with)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;hope this helps&#xA;cheers&lt;/p&gt;&#xA;" OwnerUserId="447" LastEditorUserId="447" LastEditorDisplayName="" LastEditDate="2011-01-17T15:38:40.497" LastActivityDate="2011-01-17T15:38:40.497" />
  <row Id="6316" PostTypeId="2" ParentId="256" CreationDate="2011-01-17T15:36:54.080" Score="3" ViewCount="" Body="&lt;p&gt;Boosting employs shrinkage through the learning rate parameter, which, coupled with &lt;em&gt;k&lt;/em&gt;-fold cross validation, OOB predictions or independent test set, determine the number of trees one should keep in the ensemble.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We want a model that learns slowly, hence there is a trade-off in terms of the complexity of each individual model and the number of models to include. The guidance I have seen suggests you should set the learning rate as low as is feasibly possible (given compute time and storage space requirements), whilst the complexity of each tree should be selected on basis of whether interactions are allowed, and to what degree, the more complex the tree, the more complex the interactions that can be represented.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The learning rate is chosen in the range 0, ... 1. Smaller values preferred. This is a weighting applied to each tree to down weight the contribution of each model to the fitted values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;k&lt;/em&gt;-fold CV (or OOB predictions or independent test set) is used to decide when the boosted model has started to overfit. Essentially it is this that stops us boosting to the perfect model, but it is better to learn slowly so we have a large ensemble of models contributing to the fitted model.&lt;/p&gt;&#xA;" OwnerUserId="1390" LastActivityDate="2011-01-17T15:36:54.080" />
  <row Id="6317" PostTypeId="1" AcceptedAnswerId="6323" CreationDate="2011-01-17T15:44:45.267" Score="5" ViewCount="88" Body="&lt;p&gt;In boosting, each additional tree is fitted to the unexplained variation in the response that is currently un-modelled. If we are using squared-error loss, this amounts to fitting on the residuals from the aggregation of the trees fitted up to this point. I am not clear on whether it is at this point that the shrinkage (learning rate) is applied? E.g. we fit the first tree, and compute fitted values. Do we now shrink (down weight) these fitted values by the learning rate yielding a new set of fitted values, from which we compute the residuals that are used as the responses for the second tree in the ensemble, and iterate through many trees?&lt;/p&gt;&#xA;" OwnerUserId="1390" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-01-18T00:42:38.193" LastActivityDate="2011-01-18T00:42:38.193" Title="When is the shrinkage applied in Friedman's stochastic gradient boosting machine?" Tags="&lt;machine-learning&gt;&lt;boosting&gt;" AnswerCount="1" FavoriteCount="2" />
  <row Id="6318" PostTypeId="1" CreationDate="2011-01-17T16:07:47.570" Score="8" ViewCount="202" Body="&lt;p&gt;In response to a growing body of statisticians and researchers that criticize the utility of null-hypothesis testing (NHT) for science as a cumulative endeavour, the American Psychological Association Task Force on Statistical Inference avoided an outright ban on NHT, but instead suggested that researchers report effect sizes in addition to p-values derived from NHT. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, effect sizes are not easily accumulated across studies. Meta-analytic approaches can accumulate distributions of effect sizes, but effect sizes are typically computed as a ratio between raw effect magnitude and unexplained &quot;noise&quot; in the data of a given experiment, meaning that the distribution of effect sizes is affected not only by the variability in the raw magnitude of the effect across studies, but also variability in the manifestation of noise across studies.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In contrast, an alternative measure of effect strength, likelihood ratios, permit both intuitive interpretation on a study-by-study basis, and can be easily aggregated across studies for meta-analysis. Within each study, the likelihood represents the weight of evidence for a model containing a given effect relative to a model that does not contain the effect, and could typically be reported as, for example, &quot;Computation of a likelihood ratio for the effect of X revealed 8 times more evidence for the effect than for its respective null&quot;. Furthermore, the likelihood ratio also permits intuitive representation of the strength of null findings insofar as likelihood ratios below 1 represent scenarios where the null is favoured and taking the reciprocal of this value represents the weight of evidence for the null over the effect. Notably, the likelihood ratio is represented mathematically as the ratio of unexplained variances of the two models, which differ only in the variance explained by the effect and thus is not a huge conceptual departure from an effect size. On the other hand, computation of a meta-analytic likelihood ratio, representing the weight of evidence for an effect across studies, is simply a matter of taking the product of likelihood ratios across studies.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, I argue that for science seeking to establish the degree of gross evidence in favour of a effect/model, likelihood ratios are the way to go. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are more nuanced cases where models are differentiable only in the specific size of an effect, in which case some sort of representation of the interval over which we believe the data are consistent with effect parameter values might be preferred. Indeed, the APA task force also recommends reporting confidence intervals, which can be used to this end, but I suspect that this is also an ill-considered approach. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Confidence intervals are lamentably often misinterpreted (&lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pubmed/16392994&quot; rel=&quot;nofollow&quot;&gt;by students and researchers alike&lt;/a&gt;). I also fear that their ability for use in NHT (by assessment of inclusion of zero within the CI) will only serve to further delay the extinction of NHT as an inferential practice. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead, when theories are differentiable only by the size of effects, I suggest that Bayesian approach would be more appropriate, where the prior distribution of each effect is defined by each model separately, and the resulting posterior distributions are compared.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does this approach, replacing p-values, effect sizes and confidence intervals with likelihood ratios and, if necessary, Bayesian model comparison, seem sufficient? Does it miss out on some necessary inferential feature that the here-maligned alternatives provide?&lt;/p&gt;&#xA;" OwnerUserId="364" LastEditorUserId="364" LastEditorDisplayName="" LastEditDate="2011-01-17T17:45:06.630" LastActivityDate="2011-01-18T15:31:45.893" Title="Do likelihood ratios and Bayesian model comparison provide superior &amp; sufficient alternatives to null-hypothesis testing?" Tags="&lt;bayesian&gt;&lt;confidence-interval&gt;&lt;effect-size&gt;&lt;inference&gt;" AnswerCount="1" CommentCount="8" FavoriteCount="2" />
  <row Id="6319" PostTypeId="2" ParentId="1964" CreationDate="2011-01-17T17:18:11.883" Score="1" ViewCount="" Body="&lt;p&gt;On Matlab File Exchange, there is a kde function that provides the optimal bandwidth with the assumption that a Gaussian kernel is used: &lt;a href=&quot;http://www.mathworks.com/matlabcentral/fileexchange/14034-kernel-density-estimator&quot; rel=&quot;nofollow&quot;&gt;Kernel Density Estimator&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even if you don't use Matlab, you can parse through this code for it's method of calculating the optimal bandwidth. This is a highly rated function on file exchange and I have used it many times. &lt;/p&gt;&#xA;" OwnerUserId="559" LastActivityDate="2011-01-17T17:18:11.883" />
  <row Id="6320" PostTypeId="1" CreationDate="2011-01-17T18:32:55.550" Score="3" ViewCount="110" Body="&lt;p&gt;I have a very large number of observations. Observations arrive sequentially. Each observation is an $n$-dimensional vector (with $n \ge 100$), is independent from the others and is drawn from the same unknown distribution. Is there an optimal policy to estimate the unknown distribution, given some space bounds on the number of observations that can be stored? I would leave the estimation criteria open-ended, (in terms of expected or minimax error, asymptotic consistency, aysmptotic efficiency etc.).&lt;/p&gt;&#xA;" OwnerUserId="30" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-01-17T18:50:47.337" LastActivityDate="2011-02-06T01:23:06.167" Title="Estimating probability distribution function of a data stream" Tags="&lt;estimation&gt;&lt;multivariable&gt;&lt;streaming-data&gt;" AnswerCount="1" CommentCount="5" FavoriteCount="1" />
  <row Id="6321" PostTypeId="1" AcceptedAnswerId="6322" CreationDate="2011-01-17T19:06:08.327" Score="6" ViewCount="189" Body="&lt;p&gt;I have two variables, and I can calculate e.g. the Pearson correlation between them, but I would like to know something analogous to what a t-test would give me (i.e. some notion of how significant the correlation is). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does such a thing exist?&lt;/p&gt;&#xA;" OwnerUserId="900" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-01-18T00:39:18.570" LastActivityDate="2011-01-21T10:05:07.917" Title="Assessing significance of correlation" Tags="&lt;correlation&gt;&lt;statistical-significance&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="6322" PostTypeId="2" ParentId="6321" CreationDate="2011-01-17T19:09:49.817" Score="8" ViewCount="" Body="&lt;p&gt;Yes, you can get a $p$-value for testing the null hypothesis that the Pearson correlation is zero. See &lt;a href=&quot;http://en.wikipedia.org/wiki/Pearson%27s_correlation#Inference&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Pearson%27s_correlation#Inference&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-01-17T19:09:49.817" CommentCount="1" />
  <row Id="6323" PostTypeId="2" ParentId="6317" CreationDate="2011-01-17T19:35:52.177" Score="3" ViewCount="" Body="&lt;p&gt;Using trees, the shrinkage takes place at the update stage of the algorithm, when the new function $f(x)_k$ is created as the function prior step ($f(x)_{k-1}$) + the new decision tree output ($p(x)_k$). This new tree output ($p(x)_k$) is scaled by the learning rate parameter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See for example the implementation in R &lt;a href=&quot;http://cran.r-project.org/web/packages/gbm/vignettes/gbm.pdf&quot; rel=&quot;nofollow&quot;&gt;GBM&lt;/a&gt; on page 6.  &lt;/p&gt;&#xA;" OwnerUserId="2040" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-01-18T00:42:11.000" LastActivityDate="2011-01-18T00:42:11.000" />
  <row Id="6324" PostTypeId="2" ParentId="6312" CreationDate="2011-01-17T19:36:23.677" Score="2" ViewCount="" Body="&lt;p&gt;This might be an over-simplification, but specificity and sensitivity are measures of performance, and are used when there isn't any objective knowledge of the nature of the signal. I mean your density vs. signalness plot assumes one variable that quantifies signalness. For very high dimensional, or infinite-dimensional data, and without a rigorous, provable theory of the mechanism of the generation of the signal, the selection of the variable is non-trivial. The question then arises, is why, after selecting such a variable, are its statistical properties, like the mean and variance for the signal and non-signal not quantified. In many cases, the variable may not be just normal, Poisson, or exponentially distributed. It may even be non-parametric, in which case quantifying the separation as mean difference over variance etc., does not make much sense. Also, a lot of literature in the biomedical field is focussed on applications, and ROC, specificity-sensitivity etc., can be used as objective criteria for comparing the approaches in terms of the limited nature of the problem, and basically that is all that is required. Sometimes people may not be interested in describing, say the actual discrete version log-gamma distribution of the ratio of gene1 vs gene2 transcript abundance in diseased vs control subjects, rather the only thing of importance is whether this is elevated and how much variance of the phenotype or probability of disease it explains.&lt;/p&gt;&#xA;" OwnerUserId="2728" LastEditorUserId="2728" LastEditorDisplayName="" LastEditDate="2011-01-18T17:29:07.060" LastActivityDate="2011-01-18T17:29:07.060" />
  <row Id="6325" PostTypeId="2" ParentId="6304" CreationDate="2011-01-17T20:28:09.393" Score="8" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;It's not even a close approximation.&lt;/strong&gt;  For small $n$, the expectation of $T$ equals $\frac{k n}{n-2}$ whereas the expectation of $\chi^2(k)$ equals $k$.  When $k$ is small (less than 10, say) histograms of $\log(T)$ and of $\log(\chi^2(k))$ don't even have the same shape, indicating that shifting and rescaling $T$ still won't work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Intuitively, for small degrees of freedom Student's $t$ is heavy tailed.  Squaring it emphasizes that heaviness.  The sums therefore will be more skewed--usually &lt;em&gt;much&lt;/em&gt; more skewed--than sums of squared normals (the $\chi^2$ distribution).  Calculations and simulations bear this out.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;h3&gt;Illustration (as requested)&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/AcMKs.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each histogram depicts an independent simulation of 100,000 trials with the specified degrees of freedom ($n$) and summands ($k$), standardized as described by @mpiktas.  The value of $n=9999$ on the bottom row approximates the $\chi^2$ case.  Thus you can compare $T$ to $\chi^2$ by scanning down each column.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that standardization is not possible for $n \lt 5$ because the appropriate moments do not even exist.  The lack of stability of shape (as you scan from left to right across any row or from top to bottom down any column) is even more marked for $n \le 4$.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-01-17T21:54:42.793" LastActivityDate="2011-01-17T21:54:42.793" CommentCount="4" />
  <row Id="6326" PostTypeId="1" AcceptedAnswerId="6335" CreationDate="2011-01-18T00:45:45.837" Score="4" ViewCount="172" Body="&lt;p&gt;I have two treatments A &amp;amp; B. Here are my groups, where X represents the appropriate control for that particular treatment:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Group 1: XX&#xA;Group 2: AX&#xA;Group 3: XB&#xA;Group 4: AB&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The hypothesis is that the treatment B will have an effect, but that that effect will no longer be apparent when combined with treatment A.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, if run my experiment and run an ANOVA on the data, and the results of the analysis show that only Group 3 was significantly different than the others, is it correct to say that &quot;treatment B had an effect and that effect was lost when combined with treatment A&quot;? Or, do would I also need to show a significant different between XB and AB?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you! =)&lt;/p&gt;&#xA;" OwnerUserId="2816" LastActivityDate="2011-01-18T11:28:01.773" Title="Interpreting interactions between two treatments" Tags="&lt;anova&gt;" AnswerCount="2" />
  <row Id="6327" PostTypeId="2" ParentId="6312" CreationDate="2011-01-18T01:23:18.533" Score="2" ViewCount="" Body="&lt;p&gt;You're comparing &quot;What is the probability that a positive test outcome is correct given a known prevalence and test criterion?&quot; with &quot;What is the sensitivity and bias of an unknown system to various signals of this type?&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems to me that the two both use some similar theory but they really have very different purposes.  With the medical tests criterion is irrelevant.  It can be set to a known value in many cases.  So, determining the criterion of the test is pointless afterwards.  Signal detection theory is best for systems where criterion is unknown.  Furthermore, prevalence, or signal, tends to be a fixed (and often very small) value.  With SDT you often work out a mean d' over varying signals modelling a very complex situation as a few simple descriptors.  When both the criterion and signal are fixed known quantities can SDT tell you anything interesting?  It seems like a lot of mathematical sophistication to deal with a fundamentally simpler problem.  &lt;/p&gt;&#xA;" OwnerUserId="601" LastActivityDate="2011-01-18T01:23:18.533" />
  <row Id="6328" PostTypeId="2" ParentId="3328" CreationDate="2011-01-18T01:42:16.450" Score="0" ViewCount="" Body="&lt;p&gt;If you keep the log likelihoods, you can just select the one with the highest value. Also, if your interest is primarily the mode, just doing an optimization to find the point with the highest log likelihood would suffice.&lt;/p&gt;&#xA;" OwnerUserId="1146" LastActivityDate="2011-01-18T01:42:16.450" />
  <row Id="6329" PostTypeId="1" AcceptedAnswerId="6332" CreationDate="2011-01-18T02:20:54.247" Score="1" ViewCount="289" Body="&lt;p&gt;I've been using the ets() and auto.arima() functions from the &lt;a href=&quot;http://robjhyndman.com/software/forecast/&quot; rel=&quot;nofollow&quot;&gt;forecast package&lt;/a&gt; to forecast a large number of univariate time series.  I've been using the following function to choose between the 2 methods, but I was wondering if CrossValidated had any better (or less naive) ideas for automatic forecasting.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;auto.ts &amp;lt;- function(x,ic=&quot;aic&quot;) {&#xA;    XP=ets(x, ic=ic) &#xA;    AR=auto.arima(x, ic=ic)&#xA;&#xA;    if (get(ic,AR)&amp;lt;get(ic,XP)) {&#xA;        model&amp;lt;-AR&#xA;    }&#xA;    else {&#xA;        model&amp;lt;-XP&#xA;    }&#xA;        model&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;/edit: What about this function?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;auto.ts &amp;lt;- function(x,ic=&quot;aic&quot;,holdout=0) {&#xA;    S&amp;lt;-start(x)[1]+(start(x)[2]-1)/frequency(x) #Convert YM vector to decimal year&#xA;    E&amp;lt;-end(x)[1]+(end(x)[2]-1)/frequency(x)&#xA;    holdout&amp;lt;-holdout/frequency(x) #Convert holdout in months to decimal year&#xA;    fitperiod&amp;lt;-window(x,S,E-holdout) #Determine fit window&#xA;&#xA;    if (holdout==0) {&#xA;        testperiod&amp;lt;-fitperiod&#xA;    }&#xA;    else {&#xA;        testperiod&amp;lt;-window(x,E-holdout+1/frequency(x),E) #Determine test window&#xA;    }&#xA;&#xA;    XP=ets(fitperiod, ic=ic)&#xA;    AR=auto.arima(fitperiod, ic=ic)&#xA;&#xA;    if (holdout==0) {&#xA;        AR_acc&amp;lt;-accuracy(AR)&#xA;        XP_acc&amp;lt;-accuracy(XP)&#xA;    }&#xA;    else {&#xA;        AR_acc&amp;lt;-accuracy(forecast(AR,holdout*frequency(x)),testperiod)&#xA;        XP_acc&amp;lt;-accuracy(forecast(XP,holdout*frequency(x)),testperiod)&#xA;    }&#xA;&#xA;    if (AR_acc[3]&amp;lt;XP_acc[3]) { #Use MAE&#xA;        model&amp;lt;-AR&#xA;    }&#xA;    else {&#xA;        model&amp;lt;-XP&#xA;    }&#xA;    model&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The &quot;holdout&quot; is the number of periods you wish to use as an out of sample test.  The function then calculates a fit window and a test window based on this parameter.  Then it runs the auto.arima and ets functions on the fit window, and chooses the one with the lowest MAE in the test window.  If the holdout is equal to 0, it tests the in-sample fit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a way to automatically update the chosen model with the complete dataset, once it has been selected?&lt;/p&gt;&#xA;" OwnerUserId="2817" LastEditorUserId="2817" LastEditorDisplayName="" LastEditDate="2011-01-19T18:39:32.660" LastActivityDate="2011-01-19T18:39:32.660" Title="Combining auto.arima() and ets() from the forecast package" Tags="&lt;r&gt;&lt;time-series&gt;&lt;forecasting&gt;&lt;exponential&gt;&lt;arima&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="6330" PostTypeId="1" AcceptedAnswerId="6348" CreationDate="2011-01-18T02:50:01.373" Score="5" ViewCount="498" Body="&lt;p&gt;I have previously used &lt;a href=&quot;http://www.forecastpro.com/&quot; rel=&quot;nofollow&quot;&gt;forecast pro&lt;/a&gt; to forecast univariate time series, but am switching my workflow over to R.  The forecast package for R contains a lot of useful functions, but one thing it doesn't do is any kind of data transformation before running auto.arima().  In some cases forecast pro decides to log transform data before doing forecasts, but I haven't yet figured out why.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my question is: when should I log-transform my time series before trying ARIMA methods on it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;/edit: after reading your answers, I'm going to use something like this, where x is my time series:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(lmtest)&#xA;if ((gqtest(x~1)$p.value &amp;lt; 0.10) {&#xA;    x&amp;lt;-log(x)&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Does this make sense?&lt;/p&gt;&#xA;" OwnerUserId="2817" LastEditorUserId="2817" LastEditorDisplayName="" LastEditDate="2011-01-19T20:07:15.060" LastActivityDate="2011-02-23T20:20:26.387" Title="When to log transform a time series before fitting an ARIMA model" Tags="&lt;r&gt;&lt;time-series&gt;&lt;data-transformation&gt;&lt;forecasting&gt;&lt;arima&gt;" AnswerCount="4" FavoriteCount="2" />
  <row Id="6331" PostTypeId="2" ParentId="6326" CreationDate="2011-01-18T03:26:40.937" Score="2" ViewCount="" Body="&lt;p&gt;If in post hoc testing Group 3's mean was significantly different from all the others' then you've already shown that XB is different from AB. Am I missing something?  Your statement about B's effect (and its being lost when combined with A's) would be correct.&lt;/p&gt;&#xA;" OwnerUserId="2669" LastActivityDate="2011-01-18T03:26:40.937" CommentCount="2" />
  <row Id="6332" PostTypeId="2" ParentId="6329" CreationDate="2011-01-18T03:35:29.263" Score="5" ViewCount="" Body="&lt;p&gt;The likelihoods from the two model classes, and hence the AIC values, are not comparable due to different initialization assumptions. So your function is not valid. I suggest you try out the two model classes on your series and see which gives the best out-of-sample forecasts.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2011-01-18T03:35:29.263" CommentCount="5" />
  <row Id="6333" PostTypeId="2" ParentId="6330" CreationDate="2011-01-18T03:41:29.677" Score="8" ViewCount="" Body="&lt;p&gt;Plot a graph of the data against time. If it looks like the variation increases with the level of the series, take logs. Otherwise model the original data.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2011-01-18T03:41:29.677" CommentCount="1" />
  <row Id="6334" PostTypeId="2" ParentId="6330" CreationDate="2011-01-18T03:45:46.210" Score="0" ViewCount="" Body="&lt;p&gt;You might want to log-transform series when they are somehow naturally geometric or where the time value of an investment implies that you will be comparing to a minimal risk bond that has a positive return. This will make them more &quot;linearizable&quot;, and therefore suitable for a simple differencing recurrence relationship.&lt;/p&gt;&#xA;" OwnerUserId="2129" LastActivityDate="2011-01-18T03:45:46.210" />
  <row Id="6335" PostTypeId="2" ParentId="6326" CreationDate="2011-01-18T11:28:01.773" Score="4" ViewCount="" Body="&lt;p&gt;If I understand you correctly, your design is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\begin{array}{rcccl}&#xD;&#xA;~     &amp;amp; B_{X}    &amp;amp; B_{B}    &amp;amp; M     \\\hline&#xD;&#xA;A_{X} &amp;amp; \mu_{11} &amp;amp; \mu_{12} &amp;amp; \mu_{1.} \\&#xD;&#xA;A_{A} &amp;amp; \mu_{21} &amp;amp; \mu_{22} &amp;amp; \mu_{2.} \\\hline&#xD;&#xA;M     &amp;amp; \mu_{.1} &amp;amp; \mu_{.2} &amp;amp; \mu&#xD;&#xA;\end{array}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first part of your hypothesis (effect of treatment B within control group of A) then means that $H_{1}^{1}: \mu_{12} - \mu_{11} &amp;gt; 0$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second part of your hypothesis (no effect of treatment B within treatment A) would then be $H_{1}^{2}: \mu_{22} - \mu_{21} = 0$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So your composite hypothesis is $H_{1}: H_{1}^{1} \wedge H_{1}^{2}$. The problem is with the second part because a non-significant post-hoc test for $H_{0}: \mu_{22} - \mu_{11} = 0$ doesn't mean that there is no effect - your test simply might not have enough power to detect the difference.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could still test the hypothesis $H_{1}&amp;#39;: (\mu_{12} - \mu_{11}) &amp;gt; (\mu_{22} - \mu_{21})$, i.e., an interaction contrast. However, this tests the weaker hypothesis that B has a bigger effect within A's control group than within treatment A.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not sure what you mean by &quot;the results of the analysis show that only Group 3 was significantly different than the others&quot;. I don't understand how exactly you would test that. You could test $\mu_{12} \neq \frac{1}{3} (\mu_{11} + \mu_{21} + \mu_{22})$, but that is a weaker hypothesis (Group 3 is different from the average of the remaining groups).&lt;/p&gt;&#xA;" OwnerUserId="1909" LastActivityDate="2011-01-18T11:28:01.773" CommentCount="2" />
  <row Id="6336" PostTypeId="2" ParentId="6302" CreationDate="2011-01-18T12:42:18.770" Score="2" ViewCount="" Body="&lt;p&gt;The empirical CDF needs to be treated with care at the end points of the data, and in other places where there is &quot;sparse&quot; data.  This is because they tend to make weak structural assumptions about what goes on &quot;in between&quot; each data point.  It would also be a good idea to have &quot;dots&quot; for the empirical CDF plot rather than lines, or have the dots superimposed over the lines, so that it is easier to see where most of the data actually is.  Another alternative is to put the &quot;dots&quot; for the data over the fitted CDF plot, although there may be too much going on in the plot.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maybe its a plotting difficulty, but the empirical CDF should look like a staircase or step function (horizontal lines with &quot;jumps&quot; at the observed values).  The empirical plots above do not look this way, they appear &quot;smoothed&quot;.  Maybe they are a &quot;non-parametric&quot; CDF using some kind of plot smoother?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If it is a &quot;non-parametric&quot; CDF then you are basically comparing between to models: the negative binomial and the non-parametric one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My advice: have a separate plot for each data (each colour on a new graph), and then put the empirical CDF as &quot;dots&quot; where the data was observed, and the fitted negative binomial CDF as a smooth line on the same plot.  This would look similar to a regression-style scatter plot with a fitted line.  An example of the kind of plot I am talking (which has R-code to create it) is here &lt;a href=&quot;http://stats.stackexchange.com/questions/6268/how-to-present-the-gain-in-explained-variance-thanks-to-the-correlation-of-y-and/6290#6290&quot;&gt;How to present the gain in explained variance thanks to the correlation of Y and X?&lt;/a&gt;)&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-01-18T12:42:18.770" CommentCount="4" />
  <row Id="6337" PostTypeId="1" AcceptedAnswerId="6338" CreationDate="2011-01-18T13:04:50.173" Score="4" ViewCount="102" Body="&lt;p&gt;I have various clinical data on participants in a study. I'm looking at a continuous variable (&quot;A&quot;) and a (binary) categorical variable (group) (&quot;O&quot;). I used a Wilcoxon test in R (the data are not normally distributed) to see if &quot;A&quot; is significantly different between the two groups. I got a borderline p-value of 0.054.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I run the Wilcoxon again but include only the males (30 of 72), the p-value is ~0.3; for females only it's ~0.25.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How is it possible that there is no difference in &quot;A&quot; between the groups for males and females separately, but when combined there is a difference?&lt;/p&gt;&#xA;" OwnerUserId="2824" LastEditorUserId="2824" LastEditorDisplayName="" LastEditDate="2011-01-18T15:09:21.593" LastActivityDate="2011-01-18T15:09:21.593" Title="Subsets not significantly different but superset is" Tags="&lt;r&gt;&lt;statistical-significance&gt;&lt;nonparametric&gt;&lt;wilcoxon&gt;" AnswerCount="2" />
  <row Id="6338" PostTypeId="2" ParentId="6337" CreationDate="2011-01-18T13:45:26.073" Score="6" ViewCount="" Body="&lt;p&gt;It seems to be a question of &lt;a href=&quot;http://en.wikipedia.org/wiki/Statistical_power&quot; rel=&quot;nofollow&quot;&gt;test power&lt;/a&gt;. If you only look at a subset you have a lot less participants and therefore a lot less power to find an effect of similar size.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With a reduced sample size you can only find a much bigger effect. So it is NOT recommended to only look at the subsets in this case. Unless there is an interaction (i.e., do the results point into the same direction for both men and women?).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furthermore, there is no need to use a Wilcoxon test only because your data is not normally distributed (unless it &lt;strong&gt;heavily&lt;/strong&gt; deviates). Probably you can still use the t.test (for example one of the user here, whuber, recently advocated the t.test in a similar case, because the normally assumption does not necessarily hold for the data but for the sampling distribution. quoting him: &quot;The reason is that the sampling distributions of the means are approximately normal, even though the distributions of the data are not&quot;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, if you still don't want to use the t.test there are more powerful 'assumption free' parametric alternatives, especially permutation tests. See the answer to my question here (whubers quote is also from there): &lt;a href=&quot;http://stats.stackexchange.com/questions/6127/which-permutation-test-implementation-in-r-to-use-instead-of-t-tests-paired-and&quot;&gt;Which permutation test implementation in R to use instead of t-tests (paired and non-paired)?&lt;/a&gt;&lt;br&gt;&#xA;In my case the results were even a little bit better (i.e., smaller p) than when using the t.test. So I would recommend this permutation test based on the &lt;code&gt;coin&lt;/code&gt; package. I could provide you with the necessary r-commands if you provide some sample data in your question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update: The effect of outliers on the t-test&lt;/strong&gt;&lt;br&gt;&#xA;If you look at the help of t.test in R &lt;code&gt;?t.test&lt;/code&gt;, you will find the following example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;t.test(1:10,y=c(7:20))      # P = .00001855&#xA;t.test(1:10,y=c(7:20, 200)) # P = .1245    -- NOT significant anymore&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Although in the second case you have a much extremer difference in the means, the outlier leads to the counterintuitive finding that the data is not significant anymore. Hence, a method to deal with outliers (e.g. Winsorizing, &lt;a href=&quot;http://www.unt.edu/rss/class/mike/5700/articles/robustAmerPsyc.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;) is recommended for parametric tests as the t if the data permits.&lt;/p&gt;&#xA;" OwnerUserId="442" LastEditorUserId="442" LastEditorDisplayName="" LastEditDate="2011-01-18T14:49:41.990" LastActivityDate="2011-01-18T14:49:41.990" CommentCount="10" />
  <row Id="6339" PostTypeId="2" ParentId="6337" CreationDate="2011-01-18T14:24:30.237" Score="6" ViewCount="" Body="&lt;p&gt;This is not necessarily an issue of statistical power; it could also be an example of &lt;a href=&quot;http://en.wikipedia.org/wiki/Confounding&quot; rel=&quot;nofollow&quot;&gt;confounding&lt;/a&gt;.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example:  &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;One category of $O$ is more common in males but the other is more common in females  &lt;/li&gt;&#xA;&lt;li&gt;The distribution of $A$ differs between males and females   &lt;/li&gt;&#xA;&lt;li&gt;&lt;em&gt;Within each sex separately&lt;/em&gt;, the distribution of $A$ is exactly the same for both $O$ categories&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Then there will still be an &lt;em&gt;overall&lt;/em&gt; difference in the distribution of $A$ between the $O$ categories.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-01-18T14:24:30.237" CommentCount="4" />
  <row Id="6340" PostTypeId="2" ParentId="6146" CreationDate="2011-01-18T14:52:28.410" Score="2" ViewCount="" Body="&lt;p&gt;From the statement of the question, it seems as though you don't require conjugacy per se, rather you would like an analytical solution to your integration.  From the form of the distribution, it would appear at first glance that the analytics of most solutions would be rather messy and difficult to interpret.  The &quot;analytic&quot; solution would likely involve non-analytic functions (such as gamma function, beta function, confluent hypergeometric function), which may require numerical evaluation anyway.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;May be quicker to use MCMC, rejection sampling or some other numerical technique to do the integration.  But this means you need to choose a prior for you parameters (ideally, one which describes what you know about them).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One choice which comes to my mind, is the product of the beta binomial prior for the &quot;central hypergeometric&quot; part, and a beta distribution of the second kind for the odds ratio parameter.  (beta distribution of the second kind is the distribution of the odds ratio of a beta distributed random variable, similar to a F distribution).  But this is somewhat an arbitrary choice, only based on the conjugacy for the central hypergeometric distribution, and a &quot;heavy-tailed&quot; distribution (possibly robust? definitely more robust than gamma or inverse gamma) for the odds ratio parameter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, which parameters are you integrating out? and which parameters are you taking the likelihood of?&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-01-18T14:52:28.410" CommentCount="5" />
  <row Id="6341" PostTypeId="2" ParentId="6224" CreationDate="2011-01-18T14:54:42.040" Score="-1" ViewCount="" Body="&lt;p&gt;Silvina: con respecto a lapregunta 1 no te puedo ayudar.con respecto a la 2 usa la funcion pvals.fnc del paquete languageR (un muchacho llamado Bodo Winter tiene un buen tutorial en la red), Con respecto a la correlación entre factores fijos tus valores son bajos, lo ideal es que sean 0, ya que sino indican problemas de colinealidad&lt;/p&gt;&#xA;" OwnerUserId="2826" LastActivityDate="2011-01-18T14:54:42.040" CommentCount="2" />
  <row Id="6342" PostTypeId="1" AcceptedAnswerId="6344" CreationDate="2011-01-18T15:01:29.900" Score="3" ViewCount="107" Body="&lt;p&gt;I have continuous data &quot;A&quot;, binary categorical data &quot;O&quot;, gender/sex and age for several participants in a study.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A linear model in R shows no correlation between A and age. I would now like to group A into groups by age and see if there is a difference between the groups. I know about 'hist' and 'split' in R, but these do not do what I need.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(1) How can I divide/split A into groups based on age (18 to 27, 28 to 37, etc.).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(2) Once I've done that, I would use a $\chi^2$ test?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(3) Could I also test O in the same groups, using counts? &lt;/p&gt;&#xA;" OwnerUserId="2824" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-01-18T15:07:26.673" LastActivityDate="2011-01-18T15:31:51.173" Title="Splitting one variable according to bins from another variable" Tags="&lt;r&gt;&lt;regression&gt;&lt;histogram&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="6343" PostTypeId="2" ParentId="5542" CreationDate="2011-01-18T15:16:19.887" Score="1" ViewCount="" Body="&lt;p&gt;A useful way to incorporate data into a prior distribution is the principle of maximum entropy.  You basically provide constraints that the prior distribution is to satisfy (e.g. mean, variance, etc.,etc.) and then choose the distribution which is most &quot;spread out&quot; that satisfies these constraints.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The distribution generally has the form  $p(x) \propto exp(...)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edwin Jaynes was the originator of this principle, so searching for his work is a good place to start.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See the wiki page (http://en.wikipedia.org/wiki/Principle_of_maximum_entropy) and links therein for a more detailed description.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-01-18T15:16:19.887" />
  <row Id="6344" PostTypeId="2" ParentId="6342" CreationDate="2011-01-18T15:24:02.170" Score="3" ViewCount="" Body="&lt;pre&gt;&lt;code&gt;&amp;gt; A   &amp;lt;- round(rnorm(100, 100, 15), 2)       # generate some data&#xA;&amp;gt; age &amp;lt;- sample(18:65, 100, replace=TRUE)&#xA;&amp;gt; sex &amp;lt;- factor(sample(0:1, 100, replace=TRUE), labels=c(&quot;f&quot;, &quot;m&quot;))&#xA;&#xA;# 1) bin age into 4 groups of similar size&#xA;&amp;gt; ageFac &amp;lt;- cut(age, breaks=quantile(age, probs=seq(from=0, to=1, by=0.25)),&#xA;+               include.lowest=TRUE)&#xA;&#xA;&amp;gt; head(ageFac)&#xA;[1] (26,36.5] (26,36.5] (36.5,47] [18,26]   [18,26]   [18,26]  &#xA;Levels: [18,26] (26,36.5] (36.5,47] (47,65]&#xA;&#xA;&amp;gt; table(ageFac)   # check group size&#xA;ageFac&#xA;[18,26] (26,36.5] (36.5,47]   (47,65] &#xA;     27        23        26        24&#xA;&#xA;# 2) test continuous DV in age-groups&#xA;&amp;gt; anova(lm(A ~ ageFac))&#xA;Analysis of Variance Table&#xA;Response: A&#xA;          Df  Sum Sq Mean Sq F value Pr(&amp;gt;F)&#xA;ageFac     3    15.8   5.272  0.0229 0.9953&#xA;Residuals 96 22099.2 230.200               &#xA;&#xA;# 3) chi^2-test for equal distributions of sex in age-groups    &#xA;&amp;gt; addmargins(table(sex, ageFac))&#xA;     ageFac&#xA;sex   [18,26] (26,36.5] (36.5,47] (47,65] Sum&#xA;  f        11        10        12      11  44&#xA;  m        16        13        14      13  56&#xA;  Sum      27        23        26      24 100&#xA;&#xA;&amp;gt; chisq.test(table(sex, ageFac))&#xA;        Pearson's Chi-squared test&#xA;data:  table(sex, ageFac) &#xA;X-squared = 0.2006, df = 3, p-value = 0.9775&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1909" LastEditorUserId="1909" LastEditorDisplayName="" LastEditDate="2011-01-18T15:31:51.173" LastActivityDate="2011-01-18T15:31:51.173" CommentCount="3" />
  <row Id="6345" PostTypeId="2" ParentId="6318" CreationDate="2011-01-18T15:31:45.893" Score="3" ViewCount="" Body="&lt;p&gt;The main advantages of a Bayesian approach, at least to me as a researcher in Psychology are:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) lets you accumulate evidence in favor of the null&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) circumvents the theoretical and practical problems of sequential testing&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) is not vulnerable to reject a null just because of a huge N (see previous point)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4) is better suited when working with small effects (with large effects both Frequentist and Bayesian methods tend to agree pretty much all the time)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;5) allows one to do hierarchical modeling in a feasible way. For instance, introducing item and participant effects in some model classes like Multinomial Processing Tree models would need to be done in a Bayesian framework otherwise computing time would be insanely long.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;6) gets you &quot;real&quot; confidence intervals&lt;/p&gt;&#xA;&#xA;&lt;p&gt;7) You require 3 things: the likelihood, the priors, and probability of the data. the first you get from your data, the second you make up, and the third you don't need at all given proportionality. Ok, maybe I exaggerate a little ;-)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Overall, one can invert your question: Does this all mean that classical frequentist stats are not sufficient? I think that saying &quot;no&quot; is too harsh a verdict. Most problems can be somewhat avoided if one goes beyond p-values and looks at stuff like effect sizes, the possibility of item effects, and consistently replicate findings (too many one-experiment papers get published!).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But not everything is that easy with Bayes. Take for instance model selection with non-nested models. In these cases, the priors are extremely important as they greatly affect results, and sometimes you dont have that much knowledge on most of the models you wanna work with in order to get your priors right. Also, takes reaaaally long....&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I leave two references for anybody that might be interested in diving into Bayes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.ejwagenmakers.com/BayesCourse/BayesBookWeb.pdf&quot; rel=&quot;nofollow&quot;&gt;&quot;A Course in Bayesian Graphical Modeling for Cognitive Science&quot;&lt;/a&gt; by Lee and Wagenmakers&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/047014114X&quot; rel=&quot;nofollow&quot;&gt;&quot;Bayesian Modeling Using WinBUGS&quot;&lt;/a&gt; by Ntzoufras&lt;/p&gt;&#xA;" OwnerUserId="447" LastActivityDate="2011-01-18T15:31:45.893" />
  <row Id="6347" PostTypeId="1" CreationDate="2011-01-18T16:19:49.377" Score="3" ViewCount="152" Body="&lt;p&gt;PCA based filtering is used to identify and eliminate noise in data. This would basically involve computing the PCs and using the top k PCs to denoise the data. What if I know for sure that only the extremely small values in my matrix are noise? Now, a value may be small w.r.t the entire matrix but not small w.r.t a particular row/column. Can I achieve this with some transformation of the input matrix followed by a PCA based filter? &lt;/p&gt;&#xA;" OwnerUserId="2806" LastActivityDate="2011-01-28T00:18:46.790" Title="PCA Based Filtering but only filter out small values" Tags="&lt;pca&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="1" />
  <row Id="6348" PostTypeId="2" ParentId="6330" CreationDate="2011-01-18T18:43:46.330" Score="7" ViewCount="" Body="&lt;p&gt;Some caveats before to proceed. As I often suggest to my students, use &lt;code&gt;auto.arima()&lt;/code&gt; things only as a first approximation to your final result or if you want to have parsimonious model when you check that your rival theory-based model do better.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Data&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You have clearly to start from the description of time series data you are working with. In macro-econometrics you usually work with aggregated data, and geometric means (surprisingly) have more empirical evidence for macro time series data, probably because most of them decomposable into &lt;em&gt;exponentially growing trend&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By the way Rob's suggestion &quot;visually&quot; works for time series with &lt;em&gt;clear seasonal part&lt;/em&gt;, as slowly varying annual data is less clear for the increases in variation. Luckily exponentially growing trend is usually seen (if it seems to be linear, than no need for logs).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Model&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your analysis is based on some theory that states that some &lt;em&gt;weighted geometric mean&lt;/em&gt; $Y(t) = X_1^{\alpha_1}(t)...X_k^{\alpha_k}(t)\varepsilon(t)$ more known as the &lt;em&gt;multiplicative regression model&lt;/em&gt; is the one you have to work with. Then you usually move to a &lt;em&gt;log-log regression model&lt;/em&gt;, that is linear in parameters and most of your variables, but some growth rates, are transformed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In financial econometrics logs are a common thing due to the popularity of log-returns, because...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Log transformations have nice properties&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In log-log regression model it is the interpretation of estimated parameter, say $\alpha_i$ as the &lt;em&gt;elasticity&lt;/em&gt; of $Y(t)$ on $X_i(t)$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In error-correction models we have an empirically stronger assumption that &lt;em&gt;proportions are more stable&lt;/em&gt; (&lt;em&gt;stationary&lt;/em&gt;) than the absolute differences.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In financial econometrics it is &lt;em&gt;easy to aggregate the log-returns over time&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are many other reasons not mentioned here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Finally&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that log-transformation is usually applied to non-negative (level) variables. If you observe the differences of two time series (net export, for instance) it is not even possible to take the log, you have either to search for original data in levels or assume the form of common trend that was subtracted.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[&lt;strong&gt;addition after edit&lt;/strong&gt;] If you still want a &lt;em&gt;statistical criterion&lt;/em&gt; for when to do log transformation a simple solution would be any test for heteroscedasticity. In the case of increasing variance I would recommend  &lt;a href=&quot;http://en.wikipedia.org/wiki/Goldfeld%E2%80%93Quandt_test&quot; rel=&quot;nofollow&quot;&gt;Goldfeld-Quandt Test&lt;/a&gt; or similar to it. In R it is located in &lt;code&gt;library(lmtest)&lt;/code&gt; and is denoted by &lt;code&gt;gqtest(y~1)&lt;/code&gt; function. Simply regress on intercept term if you don't have any regression model, &lt;code&gt;y&lt;/code&gt; is your dependent variable. &lt;/p&gt;&#xA;" OwnerUserId="2645" LastEditorUserId="2645" LastEditorDisplayName="" LastEditDate="2011-01-19T09:49:07.870" LastActivityDate="2011-01-19T09:49:07.870" CommentCount="4" />
  <row Id="6349" PostTypeId="2" ParentId="256" CreationDate="2011-01-18T18:44:34.727" Score="0" ViewCount="" Body="&lt;p&gt;&quot;Why doesn't it boost very weak classifiers &quot;to infinity&quot; (perfectness)?&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because weak classifiers do not exist!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If they did, then boosting would get perfect results always. Since it doesn't, weak classifiers do not exist (i.e., there is no method that, given a weighted set of arbitrary data points and labels always returns a classifier that is better than trivial).&lt;/p&gt;&#xA;" OwnerUserId="2067" LastActivityDate="2011-01-18T18:44:34.727" CommentCount="4" />
  <row Id="6350" PostTypeId="1" AcceptedAnswerId="6351" CreationDate="2011-01-18T19:07:59.347" Score="9" ViewCount="677" Body="&lt;p&gt;The &lt;a href=&quot;http://en.wikipedia.org/wiki/Anova#Assumptions_of_ANOVA&quot; rel=&quot;nofollow&quot;&gt;Wikipedia page on ANOVA lists three assumptions&lt;/a&gt;, namely:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Independence of cases – this is an assumption of the model that simplifies the statistical analysis.&lt;/li&gt;&#xA;&lt;li&gt;Normality – the distributions of the residuals are normal.&lt;/li&gt;&#xA;&lt;li&gt;Equality (or &quot;homogeneity&quot;) of variances, called homoscedasticity...&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Point of interest here is the second assumption. Several sources list the assumption differently. Some say normality of the raw data, some claim of residuals.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Several questions pop up:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;are normality and normal distribution of residuals the same person (based on Wikipedia entry, I would claim normality is a property, and does not pertain residuals directly (but can be a property of residuals (deeply nested text within brackets, freaky)))?&lt;/li&gt;&#xA;&lt;li&gt;if not, which assumption should hold? One? Both?&lt;/li&gt;&#xA;&lt;li&gt;if the assumption of normally distributed residuals is the right one, are we making a grave mistake by checking only the histogram of raw values for normality?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="144" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-29T00:53:02.283" LastActivityDate="2011-04-29T00:53:02.283" Title="ANOVA assumption normality/normal distribution of residuals" Tags="&lt;anova&gt;&lt;assumptions&gt;&lt;residuals&gt;&lt;normallity&gt;" AnswerCount="3" CommentCount="4" FavoriteCount="2" />
  <row Id="6351" PostTypeId="2" ParentId="6350" CreationDate="2011-01-18T19:45:40.100" Score="7" ViewCount="" Body="&lt;p&gt;Let's assume this is a &lt;a href=&quot;http://en.wikipedia.org/wiki/Analysis_of_variance#Fixed-effects_models_.28Model_1.29&quot; rel=&quot;nofollow&quot;&gt;fixed effects&lt;/a&gt; model.  (The advice doesn't really change for random-effects models, it just gets a little more complicated.)&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;No, &lt;strong&gt;normality and normal distribution of residual are not the same&lt;/strong&gt;.  Suppose you measured yield from a crop with and without a fertilizer application.  In plots without fertilizer the yield ranged from 70 to 130.  In two plots with fertilizer the yield ranged from 470 to 530.  The distribution of results is strongly non-normal: it's clustered at two locations related to the fertilizer application.  Suppose further the average yields are 100 and 500, respectively.  Then &lt;em&gt;all&lt;/em&gt; residuals range from -30 to +30.  They might (or might not) be normally distributed, but obviously this is a completely different distribution.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;The distribution of the &lt;em&gt;residuals&lt;/em&gt; matters&lt;/strong&gt;, because they reflect the random part of the model.  Note also that the p-values are computed from F (or t) statistics and those depend on residuals, not on the original values.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If there are significant and important effects in the data (as in this example), then &lt;strong&gt;you &lt;em&gt;might&lt;/em&gt; be making a &quot;grave&quot; mistake&lt;/strong&gt;.  You could, by luck, make the correct determination: that is, by looking at the raw data you will seing a mixture of distributions and this can look normal (or not).  The point is that what you're looking it is not relevant.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;ANOVA residuals don't have to be anywhere close to normal in order to fit the model.  However, near-normality of the residuals is &lt;em&gt;essential&lt;/em&gt; for p-values computed from the F-distribution to be meaningful.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-01-18T19:45:40.100" CommentCount="4" />
  <row Id="6352" PostTypeId="2" ParentId="6350" CreationDate="2011-01-18T20:01:21.290" Score="3" ViewCount="" Body="&lt;p&gt;In the one-way case with $p$ groups of size $n_{j}$:&#xA;$F = \frac{SS_{b} / df_{b}}{SS_{w} / df_{w}}$ where&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$SS_{b} = \sum_{j=1}^{p}{n_{j} (M - M_{j}})^{2}$ and&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$SS_{w} = \sum_{j=1}^{p}\sum_{i=1}^{n_{j}}{(y_{ij} - M_{j})^{2}}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$F$ follows an $F$-distribution if $SS_{b} / df_{b}$ and $SS_{w} / df_{w}$ are independent, $\chi^{2}$-distributed variables with $df_{b}$ and $df_{w}$ degrees of freedom, respectively. This is the case when $SS_{b}$ and $SS_{w}$ are the sum of squared independent normal variables with mean $0$ and equal scale. Thus $M-M_{j}$ and $y_{ij}-M_{j}$ must be normally distributed. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$y_{i(j)} - M_{j}$ is the residual from the full model ($Y = \mu_{j} + \epsilon = \mu + \alpha_{j} + \epsilon$), $y_{i(j)} - M$ is the residual from the restricted model ($Y = \mu + \epsilon$). The difference of these residuals is $M - M_{j}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT to reflect clarification by @onestop: under $H_{0}$ all true group means are equal (and thus equal to $M$), thus normality of the group-level residuals $y_{i(j)} - M_{j}$ implies normality of $M - M_{j}$ as well. The DV values themselves need not be normally distributed.&lt;/p&gt;&#xA;" OwnerUserId="1909" LastEditorUserId="1909" LastEditorDisplayName="" LastEditDate="2011-01-19T09:43:20.550" LastActivityDate="2011-01-19T09:43:20.550" CommentCount="2" />
  <row Id="6353" PostTypeId="1" CreationDate="2011-01-18T20:04:35.030" Score="3" ViewCount="807" Body="&lt;p&gt;It looks like you can use coding for one categorical variable, but I have two categorical and one continuous predictor variable. Can i use multiple regression for this in SPSS and if so how?&#xA;thanks!&lt;/p&gt;&#xA;" OwnerUserId="2829" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-01-19T00:38:17.517" LastActivityDate="2011-01-25T04:29:36.703" Title="Can I use multiple regression when I have mixed categorical and continuous predictors?" Tags="&lt;regression&gt;&lt;spss&gt;&lt;categorical-data&gt;&lt;continuous-data&gt;" AnswerCount="3" CommentCount="3" />
  <row Id="6354" PostTypeId="1" CreationDate="2011-01-18T20:46:44.443" Score="2" ViewCount="275" Body="&lt;p&gt;Most methods for &lt;a href=&quot;http://eu.wiley.com/WileyCDA/WileyTitle/productCd-0470090162.html&quot; rel=&quot;nofollow&quot;&gt;symbolic data analyis&lt;/a&gt; are currently implemented in the SODAS software.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any R packages for symbolic data except clamix and clusterSim?&lt;/p&gt;&#xA;" OwnerUserId="2831" LastActivityDate="2011-01-21T18:46:58.413" Title="R package for symbolic data analysis" Tags="&lt;r&gt;&lt;clustering&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="6355" PostTypeId="1" CreationDate="2011-01-18T20:52:59.467" Score="2" ViewCount="108" Body="&lt;p&gt;I understand the premise of kNN algorithm for spatial data.  And I know I can extend that algorithm to be used on any continuous data variable (or nominal data with Hamming Distance).  However, what strategies are used when dealing with higher dimensional data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, say I have a table of data (x[1], x[2], x[3], ..., x[n]) and I want to build a set of classifiers to predict one of those columns (say x[n]).  Using kNN algorithm I would pick any two columns from the remaining columns (x[1]-x[n-1]) to train against.  So say I could pick x[1] and x[2] and build a classifier off those.  Or I could pick x[1] and x[4], or I could pick x[5] and x[8], etc.  I could even pick just a single column and build a classifiers off that, or 3 columns and build a classifiers off that.  Is there an advantage to using higher dimensions (2D, 3D, etc) or should you just build x-1 single dimension classifiers and aggregate their predictions in some way?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since building all of these classifiers from all potential combinations of the variables  would be computationally expensive.  How could I optimize this search to find the the best kNN classifiers from that set?  And, once I find a series of classifiers what's the best way to combine their output to a single prediction?  Voting might be the simplest answer to this question.  Or weighting each vote by error rates from the training data for each classifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do most implementations apply kNN to a more generalized learning?&lt;/p&gt;&#xA;" OwnerUserId="1929" LastActivityDate="2011-01-18T23:06:15.697" Title="Help understand kNN for multi-dimensional data" Tags="&lt;machine-learning&gt;&lt;k-nearest-neighbour&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="1" />
  <row Id="6356" PostTypeId="2" ParentId="726" CreationDate="2011-01-18T21:34:07.010" Score="0" ViewCount="" Body="&lt;p&gt;&lt;em&gt;We may regard the present state of the universe as the effect of its past and the cause of its future. An intellect which at a certain moment would know all forces that set nature in motion, and all positions of all items of which nature is composed, if this intellect were also vast enough to submit these data to analysis, it would embrace in a single formula the movements of the greatest bodies of the universe and those of the tiniest atom; for such an intellect nothing would be uncertain and the future just like the past would be present before its eyes.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Peirre Simon Laplace.  Also known as Laplace's demon&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-01-18T21:34:07.010" CommunityOwnedDate="2011-01-18T21:34:07.010" />
  <row Id="6357" PostTypeId="2" ParentId="726" CreationDate="2011-01-18T22:11:11.050" Score="4" ViewCount="" Body="&lt;p&gt;&lt;em&gt;One sees, from this Essay, that the theory of probabilities is basically just common sense reduced to calculus; it makes one appreciate with exactness that which accurate minds feel with a sort of instinct, often without being able to account for it.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another one from Laplace&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-01-18T22:11:11.050" CommunityOwnedDate="2011-01-18T22:11:11.050" CommentCount="2" />
  <row Id="6358" PostTypeId="1" AcceptedAnswerId="6361" CreationDate="2011-01-18T22:46:06.467" Score="4" ViewCount="154" Body="&lt;p&gt;Thanks in advance for bearing with me, I am not a statistician of any kind and don't know how to describe what I'm imagining, so Google isn't helping me here...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm including a rating system in a web application I'm working on. Each user can rate each item exactly once.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was imagining a scale with 4 values: &quot;strongly dislike&quot;, &quot;dislike&quot;, &quot;like&quot;, and &quot;strongly like&quot;, and I had planned on assigning these values of -5, -2, +2, and +5 respectively.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, if every item was going to have the same number of ratings, then I would be quite comfortable with this scoring system as clearly differentiating the most liked and least liked items. However, the items will not have the same number of ratings, and the disparity between the number of votes on different photos may be quite dramatic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In that case, comparing the cumulative scores on two items means that an old item with a lot of mediocre ratings is going to have a much higher score than an exceptional new item with many fewer votes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, the first obvious thing I thought of us to take an average... but now if an item has only one rating of &quot;+5&quot; it has a better average than an item that has a score of 99 &quot;+5&quot; ratings and 1 &quot;+2&quot; rating. Intuitively that isn't an accurate representation of the popularity of an item.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I imagine this problem is common and you guys don't need me to belabor it with more examples, so I'll stop at this point and elaborate in comments if needed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My questions are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;What is this kind of problem called, and is there a term for the techniques used to solve it? I'd like to know this so I can read up on it.&lt;/li&gt;&#xA;&lt;li&gt;If you happen to know of any lay-friendly resources on the subject, I'd very much appreciate a link.&lt;/li&gt;&#xA;&lt;li&gt;Finally, I'd appreciate any other suggestions about how to effectively collect and analyze this kind of data.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Thank you very much for your time!&lt;/p&gt;&#xA;" OwnerUserId="2832" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-29T00:41:17.380" LastActivityDate="2011-04-29T00:41:17.380" Title="Weight a rating system to favor items rated highly by more people over items rated highly by fewer people?" Tags="&lt;scales&gt;&lt;rating&gt;&lt;scores&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="6359" PostTypeId="2" ParentId="6355" CreationDate="2011-01-18T23:06:15.697" Score="1" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Is there an advantage to using higher dimensions (2D, 3D, etc) or should you just build x-1 single dimension classifiers and aggregate their predictions in some way?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This depends on whether your features are informative or not. Do you suspect that some features will not be useful in your classification task? To gain a better idea of your data, you can also try to compute pairwise correlation or mutual information between the response variable and each of your features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To combine all (or a subset) of your features, you can try computing the L1 (Manhattan), or L2 (Euclidean) distance between the query point and each 'training' point as a starting point. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Since building all of these classifiers from all potential combinations of the variables would be computationally expensive. How could I optimize this search to find the the best kNN classifiers from that set?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This is the problem of feature subset selection. There is a lot of academic work in this area (see Guyon, I., &amp;amp; Elisseeff, A. (2003). An Introduction to Variable and Feature Selection. Journal of Machine Learning Research, 3, 1157-1182. for a good overview).&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;And, once I find a series of classifiers what's the best way to combine their output to a single prediction?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This will depend on whether or not the selected features are independent or not. In the case that features are independent, you can weight each feature by its mutual information (or some other measure of informativeness) with the response variable (whatever you are classifying on). If some features are dependent, then a single classification model will probably work best.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;How do most implementations apply kNN to a more generalized learning?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;By allowing the user to specify their own distance matrix between the set of points. kNN works well when an appropriate distance metric is used.&lt;/p&gt;&#xA;" OwnerUserId="1913" LastActivityDate="2011-01-18T23:06:15.697" />
  <row Id="6360" PostTypeId="2" ParentId="6350" CreationDate="2011-01-18T23:06:31.050" Score="4" ViewCount="" Body="&lt;p&gt;Standard Classical one-way ANOVA can be viewed as an extension to the classical &quot;2-sample T-test&quot; to an &quot;n-sample T-test&quot;.  This can be seen from comparing a one-way ANOVA with only two groups to the classical 2-sample T-test.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think where you are getting confused is that (under the assumptions of the model) the residuals and the raw data are BOTH normally distributed.  However the raw data consist of normal distributions with &lt;em&gt;different&lt;/em&gt; means (unless all the effects are exactly the same) but the &lt;em&gt;same&lt;/em&gt; variance.  The residuals on the other hand have the &lt;em&gt;same normal distribution&lt;/em&gt;.  This comes from the third assumption of homoscedasticity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is because the normal distribution is decomposable into a mean and variance components.  If $Y_{ij}$ has a normal distribution with mean $\mu_{j}$ and variance $\sigma^2$ can be written as $Y_{ij}=\mu_{j}+\sigma\epsilon_{ij}$ where $\epsilon_{ij}$ has a standard normal distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While ANOVA is derivable from the assumption of normality, I think (but am unsure) it can be replaced by an assumption of linearity (along the &lt;em&gt;Best Linear Unbiased Estimator&lt;/em&gt; (BLUE) lines of estimation, where &quot;BEST&quot; is interpreted as minimum mean square error).  I believe this basically involves replacing the distribution for $\epsilon_{ij}$ with &lt;em&gt;any&lt;/em&gt; mutually independent distribution (over all &lt;em&gt;i&lt;/em&gt; and &lt;em&gt;j&lt;/em&gt;) which has mean 0 and variance 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In terms of looking at your raw data, it should look normal &lt;em&gt;when plotted separately for each factor level in your model&lt;/em&gt;.  This means plotting $Y_{ij}$ for each &lt;em&gt;j&lt;/em&gt; on a separate graph.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-01-18T23:06:31.050" CommentCount="1" />
  <row Id="6361" PostTypeId="2" ParentId="6358" CreationDate="2011-01-18T23:42:31.533" Score="9" ViewCount="" Body="&lt;p&gt;One way you can combat this is to use proportions in each category, which does not require you to put numbers in for each category (you can leave it as 80% rated as &quot;strongly likes&quot;).  However proportions do suffer from the &lt;em&gt;small number of ratings&lt;/em&gt; issue.  This shows up in your example the Photo with 1 +5 rating would get a higher average score (and proportion) than one with the 99 +5 and 1 +2 rating.  This doesn't fit well with my intuition (and I suspect most peoples).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One way to get around this small sample size issue is to use a Bayesian technique known as &quot;&lt;a href=&quot;http://en.wikipedia.org/wiki/Rule_of_succession&quot; rel=&quot;nofollow&quot;&gt;Laplace's rule of succession&lt;/a&gt;&quot; (searching this term may be useful).  It simply involves adding 1 &quot;observation&quot; to each category before calculating the probabilities.  If you wanted to take an average for a numerical value, I would suggest a &lt;em&gt;weighted average&lt;/em&gt; where the weights are the probabilities calculated by the rule of succession.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the mathematical form, let $n_{sd},n_{d},n_{l},n_{sl}$ denote the number of responses of &quot;strongly dislike&quot;, &quot;dislike&quot;, &quot;like&quot;, and &quot;strongly like&quot; respectively (in the two examples, $n_{sl}=1,n_{sd}=n_{d}=n{l}=0$ and $n_{sl}=99,n_{l}=1,n_{sd}=n_{d}=0$).  You then calculate the probability (or weight) for strongly like as &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Pr(\text{&amp;quot;Strongly Like&amp;quot;}) = \frac{n_{sl}+1}{n_{sd}+n_{d}+n_{l}+n_{sl}+4}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the two examples you give, they give probabilities of &quot;strongly like&quot; as&#xA;$\frac{1+1}{1+0+0+0+4}=\frac{2}{5}$ and $\frac{99+1}{99+1+0+0+4}=\frac{100}{104}$  which I think agree more closely with &quot;common sense&quot;.  Removing the added constants give $\frac{1}{1}$ and $\frac{99}{100}$ which makes the first outcome seem higher than it should be (at least to me anyway).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The respective scores are just given by the weighted average, which I have written below as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Score=\begin{array}{1 1} 5\frac{n_{sl}+1}{n_{sd}+n_{d}+n_{l}+n_{sl}+4}+2\frac{n_{l}+1}{n_{sd}+n_{d}+n_{l}+n_{sl}+4} \\ - 2\frac{n_{d}+1}{n_{sd}+n_{d}+n_{l}+n_{sl}+4}&#xD;&#xA;-5\frac{n_{sd}+1}{n_{sd}+n_{d}+n_{l}+n_{sl}+4}\end{array}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or more succinctly as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Score=\frac{5 n_{sl}+ 2 n_{l} - 2 n_{d} - 5 n_{sd}}{n_{sd}+n_{d}+n_{l}+n_{sl}+4}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which gives scores in the two examples of $\frac{5}{5}=1$ and $\frac{497}{104}\sim 4.8$.  I think this shows an appropriate difference between the two cases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This may have been a bit &quot;mathsy&quot; so let me know if you need more explanation.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-01-19T00:02:12.053" LastActivityDate="2011-01-19T00:02:12.053" CommentCount="11" />
  <row Id="6362" PostTypeId="2" ParentId="6358" CreationDate="2011-01-18T23:59:50.293" Score="0" ViewCount="" Body="&lt;p&gt;I'd take a graphical approach.  The x-axis could be average rating and the y could be number of ratings.  I used to do this with sports statistics to compare the contribution of young phenoms with that of veteran stars.  The nearer a point is to the upper right corner, the closer to the ideal.  Of course, deciding on the &quot;best&quot; item would still be a subjective decision, but this would provide some structure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to plot average rating against another variable, then you could set up number of ratings as the third variable using bubble size, in a bubble plot--e.g., in XL or SAS.&lt;/p&gt;&#xA;" OwnerUserId="2669" LastActivityDate="2011-01-18T23:59:50.293" />
  <row Id="6363" PostTypeId="2" ParentId="6353" CreationDate="2011-01-19T00:09:20.683" Score="1" ViewCount="" Body="&lt;p&gt;You definitely can, by following the same method you'd use for the first categorical predictor.  Create dummy variables just as you would for the first such variable. But it's often easier to use SPSS's Unianova command.  You can look this up in any printed or pdf'd Syntax Guide, or you can access it through Analyze...General Linear Model...Univariate.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Despite being a little more complicated, the Regression command has a number of advantages over Unianova, though.  The chief one is that you can choose 'missing pairwise' (you don't have to lose a case simply because it's missing a value for one or two predictors).  You can also get many valuable diagnostics such as partial plots and influence statistics.&lt;/p&gt;&#xA;" OwnerUserId="2669" LastActivityDate="2011-01-19T00:09:20.683" CommentCount="4" />
  <row Id="6364" PostTypeId="1" CreationDate="2011-01-19T01:00:36.910" Score="4" ViewCount="622" Body="&lt;p&gt;&lt;strong&gt;Motivation&lt;/strong&gt;: I'm writing a state estimator in MATLAB (the unscented Kalman filter), which calls for the update of the (upper-triangular) square-root of a covariance matrix $S$ at every iteration (that is, for a covariance matrix $P$, it is true that $P=SS^{T}$). In order for me to perform the requisite calculations, I need to do a Rank-1 Cholesky Update and Downdate using MATLAB's &lt;code&gt;cholupdate&lt;/code&gt; function. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Problem&lt;/strong&gt;: Unfortunately, during the course of the iterations, this matrix $S$ can sometimes lose positive definiteness. The Cholesky downdate fails on non-PD matrices.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My question is&lt;/strong&gt;: are there any simple and reliable ways in MATLAB to make $S$ positive-definite?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(&lt;strong&gt;or more generally, is there a good way of making any given covariance $X$ matrix positive-definite?&lt;/strong&gt;)&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Notes&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;$S$ is full rank&lt;/li&gt;&#xA;&lt;li&gt;I've tried the eigendecomposition approach (which did not work). This basically involved finding $S = VDV^{T}$, setting all negative elements of $V,D = 1 \times 10^{-8}$, and reconstructing a new $S&amp;#39; = V&amp;#39; D&amp;#39; V&amp;#39;^{T}$ where $V&amp;#39;,D&amp;#39;$ are matrices with only positive elements.&lt;/li&gt;&#xA;&lt;li&gt;I am aware of the Higham approach (which is implemented in R as &lt;code&gt;nearpd&lt;/code&gt;), but it seems to only project to the nearest PSD matrix. I need a PD matrix for the Cholesky update.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="2833" LastEditorUserId="2833" LastEditorDisplayName="" LastEditDate="2011-01-19T01:54:42.757" LastActivityDate="2011-01-19T04:35:06.207" Title="Making square-root of covariance matrix positive-definite (Matlab)" Tags="&lt;matlab&gt;&lt;variance-covariance&gt;&lt;numerics&gt;" AnswerCount="3" CommentCount="2" FavoriteCount="1" />
  <row Id="6365" PostTypeId="2" ParentId="6364" CreationDate="2011-01-19T01:40:09.367" Score="2" ViewCount="" Body="&lt;p&gt;in Matlab:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;help cholupdate&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I get &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;CHOLUPDATE Rank 1 update to Cholesky factorization.&#xA;    If R = CHOL(A) is the original Cholesky factorization of A, then&#xA;    R1 = CHOLUPDATE(R,X) returns the upper triangular Cholesky factor of A + X*X',&#xA;    where X is a column vector of appropriate length.  CHOLUPDATE uses only the&#xA;    diagonal and upper triangle of R.  The lower triangle of R is ignored.&#xA;&#xA;    R1 = CHOLUPDATE(R,X,'+') is the same as R1 = CHOLUPDATE(R,X).&#xA;&#xA;    R1 = CHOLUPDATE(R,X,'-') returns the Cholesky factor of A - X*X'.  An error&#xA;    message reports when R is not a valid Cholesky factor or when the downdated&#xA;    matrix is not positive definite and so does not have a Cholesky factorization.&#xA;&#xA;    [R1,p] = CHOLUPDATE(R,X,'-') will not return an error message.  If p is 0&#xA;    then R1 is the Cholesky factor of A - X*X'.  If p is greater than 0, then&#xA;    R1 is the Cholesky factor of the original A.  If p is 1 then CHOLUPDATE failed&#xA;    because the downdated matrix is not positive definite.  If p is 2, CHOLUPDATE&#xA;    failed because the upper triangle of R was not a valid Cholesky factor.&#xA;&#xA;    CHOLUPDATE works only for full matrices.&#xA;&#xA;    See also chol.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="795" LastActivityDate="2011-01-19T01:40:09.367" CommentCount="3" />
  <row Id="6366" PostTypeId="1" AcceptedAnswerId="6369" CreationDate="2011-01-19T01:49:58.940" Score="5" ViewCount="114" Body="&lt;p&gt;I have a question about the AR(1) model.  Expressed mathematically as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ Z_{t} = \rho Z_{t-1} + \epsilon_{t}, t=1,..,T$$&#xA;$$ \epsilon_{t} \sim iid \ N(0,1) $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is about the &quot;transformation group&quot; method of creating non-informative priors, which I believe initially was suggested by Edwin Jaynes (and is discussed in Chapter 12 of his book &lt;em&gt;Probability Theory: The Logic of Science&lt;/em&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One possible suggestion for a transformation group is to consider &quot;reversing&quot; the time series and then rescaling.  Thus my transformation group is the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\rho^{(1)} = \rho^{-1}$$&#xA;$$Z_{t}^{(1)} = \rho^{(1)}Z_{T-t+1}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using the original AR distribution, you can show that this transformation basically just &quot;shuffles&quot; the $\epsilon_{t}$ terms, which by definition of the model are exchangeable.  So, estimating $\rho$ using $Z_{t}$ is equivalent to estimating $\rho^{(1)}$ using $Z_{t}^{(1)}$ (&lt;em&gt;i.e.&lt;/em&gt;, the joint distribution of the noise is the same in both cases).  Thus the prior for $\rho^{(1)}$ must be the probability transform of the prior for $\rho$.  Or, in mathematical terms, the prior must satisfy the following functional equation:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$f(\rho)=|{\frac{\partial \rho^{-1}}{\partial \rho}}| f(\rho^{-1})=\rho^{-2}f(\rho^{-1})$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unfortunately this does not describe a unique function.  In fact, any function with the following form will satisfy the above functional equation:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;  f(\rho) = (constant) \times \begin{bmatrix}&#xD;&#xA;       \ \rho^{2b} (1-\rho^{2})^{a} &amp;amp; |\rho|&amp;lt;1 \\&#xD;&#xA;       \ \rho^{-2(b+a+1)} (\rho^{2}-1)^{a} &amp;amp; |\rho|&amp;gt;1           &#xD;&#xA;     \end{bmatrix}.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For $a &amp;gt; -1$ and $b&amp;gt;-\frac{1}{2}$ this distribution is proper, with the normalizing constant being the reciprocal of $2\beta(b+\frac{1}{2},a+1)$ where $\beta(a,b)$ is the &quot;beta integral&quot;.  Note that this class includes the &quot;symmetric reference prior&quot; recommended in &lt;em&gt;Berger, J. O. and Yang, R. (1994). Noninformative priors and Bayesian testing for the AR(1) model. Econometric Theory 10 461–482.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Usually specifying a transformation group makes the solution unique, so I am perplexed as to how this group of transformations does not produce a unique solution.  Have I done something wrong in the process of creating the transformation group?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please let me know if I need to explain my question better.  It is quite a doozy!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;UPDATE:Perhaps there is no transformation group which uniquely determines the prior in this case?&lt;/p&gt;&#xA;" OwnerUserId="2392" LastEditorUserId="2392" LastEditorDisplayName="" LastEditDate="2011-01-19T08:28:39.373" LastActivityDate="2011-01-19T08:28:39.373" Title="Non-informative priors for the AR(1) model" Tags="&lt;time-series&gt;&lt;bayesian&gt;&lt;prior&gt;&lt;transformation-groups&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="1" />
  <row Id="6367" PostTypeId="2" ParentId="6364" CreationDate="2011-01-19T02:43:49.377" Score="2" ViewCount="" Body="&lt;p&gt;Here is code I've used in the past (using the SVD approach). I know you said you've tried it already, but it has always worked for me so I thought I'd post it to see if it was helpful. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;function [sigma] = validateCovMatrix(sig)&#xA;&#xA;% [sigma] = validateCovMatrix(sig)&#xA;%&#xA;% -- INPUT --&#xA;% sig:      sample covariance matrix&#xA;%&#xA;% -- OUTPUT --&#xA;% sigma:    positive-definite covariance matrix&#xA;%&#xA;&#xA;EPS = 10^-6;&#xA;ZERO = 10^-10;&#xA;&#xA;sigma = sig;&#xA;[r err] = cholcov(sigma, 0);&#xA;&#xA;if (err ~= 0)&#xA;    % the covariance matrix is not positive definite!&#xA;    [v d] = eig(sigma);&#xA;&#xA;    % set any of the eigenvalues that are &amp;lt;= 0 to some small positive value&#xA;    for n = 1:size(d,1)&#xA;        if (d(n, n) &amp;lt;= ZERO)&#xA;            d(n, n) = EPS;&#xA;        end&#xA;    end&#xA;    % recompose the covariance matrix, now it should be positive definite.&#xA;    sigma = v*d*v';&#xA;&#xA;    [r err] = cholcov(sigma, 0);&#xA;    if (err ~= 0)&#xA;        disp('ERROR!');&#xA;    end&#xA;end&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1913" LastActivityDate="2011-01-19T02:43:49.377" CommentCount="2" />
  <row Id="6368" PostTypeId="1" AcceptedAnswerId="6378" CreationDate="2011-01-19T04:09:38.677" Score="2" ViewCount="231" Body="&lt;p&gt;Hi, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please reference this question (stats.stackexchange.com) &lt;a href=&quot;http://bit.ly/got8Bs&quot; rel=&quot;nofollow&quot;&gt;http://bit.ly/got8Bs&lt;/a&gt; for study design and initial reply from caracal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question now is: How can I test for an interaction contrast between two groups as (s)he suggested? I'm on the science side of things (minimal stats expertise) and use Graphpad Prism for basic analyses. I also have a primitive understanding of R. Is it possible to test H1′:(μ12−μ11)&gt;(μ22−μ21) using either of these programs?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks again!&lt;/p&gt;&#xA;" OwnerUserId="2816" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-01-19T09:33:06.637" LastActivityDate="2011-01-19T13:06:52.697" Title="Test for interaction contrasts using Prism or R" Tags="&lt;r&gt;&lt;interaction&gt;&lt;contrasts&gt;" AnswerCount="1" />
  <row Id="6369" PostTypeId="2" ParentId="6366" CreationDate="2011-01-19T04:29:23.700" Score="4" ViewCount="" Body="&lt;p&gt;This transformation group is discrete and finite: it contains exactly two elements, the identity and inverting $\rho$.  It's simply not big enough to determine a prior.  In fact, you can choose any measurable function $f$ defined on $[-1,1]$ provided (a) it is integrable and (b) $\rho^{-2}f(1/\rho)d\rho$ is integrable on $[1, \infty]$.  The latter restricts $f$ only in a neighborhood of $0$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;BTW, for this model to be practical you need to introduce a nuisance parameter $\sigma$: $\epsilon_t \sim N(0, \sigma^2)$.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-01-19T04:29:23.700" CommentCount="3" />
  <row Id="6370" PostTypeId="2" ParentId="6364" CreationDate="2011-01-19T04:35:06.207" Score="0" ViewCount="" Body="&lt;p&gt;One alternative way to compute the Cholesky factorisation is by fixing the diagonal elements of S to 1, and then introducing a diagonal matrix D, with positive elements.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This avoids the need to take square roots when doing the computations, which can cause problems when dealing with &quot;small&quot; numbers (i.e. numbers small enough so that the rounding which occurs due to floating point operations matters).  The &lt;a href=&quot;http://en.wikipedia.org/wiki/Cholesky_decomposition#Avoiding_taking_square_roots&quot; rel=&quot;nofollow&quot;&gt;wikipedia page&lt;/a&gt; has what this adjusted algorithm looks like.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So instead of $P=SS^T$ you get $P=RDR^T$ with $S=RD^{\frac{1}{2}}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-01-19T04:35:06.207" CommentCount="1" />
  <row Id="6371" PostTypeId="1" CreationDate="2011-01-19T05:52:52.100" Score="4" ViewCount="144" Body="&lt;p&gt;I have a finite-state, time-independent Markov chain with two absorbing states which models educational outcomes (the absorbing states are completing and not completing). The transition probabilities are estimated by taking the proportions of people who move from one state to another at successive time points (based on a census of the population at two successive time points), and I have calculated the stationary vector.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, since this needs to be done with several different cuts of the data, I would like to know if there is any way of associating a confidence interval to the entries of the stationary vector, to aid in identifying significant differences.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The article&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Karson, M. J. and Wrobleski, W. J. (1976),&lt;br/&gt;&#xA;   CONFIDENCE INTERVALS FOR ABSORBING MARKOV CHAIN PROBABILITIES APPLIED TO LOAN PORTFOLIOS.&lt;br/&gt; Decision Sciences, 7: 10–17.&lt;br/&gt; doi: 10.1111/j.1540-5915.1976.tb00653.x&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;looks helpful, but I'm not sure if it what I need. So my questions are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Is there a way to estimate confidence intervals for the stationary vector?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If yes, &lt;strike&gt; and it is in the cited article, do I just need to grit my teeth and push through it, or&lt;/strike&gt; is there a more modern treatment?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strike&gt;(a long shot, but could save me some work) Is there is a macro or similar for SAS to estimate said confidence intervals?&lt;/strike&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="1144" LastEditorUserId="1144" LastEditorDisplayName="" LastEditDate="2011-01-24T03:09:08.333" LastActivityDate="2011-01-29T21:39:40.393" Title="Can we get confidence intervals for entries in stationary vector for an absorbing, time-independent Markov chain?" Tags="&lt;confidence-interval&gt;&lt;sas&gt;&lt;markov-process&gt;" AnswerCount="1" CommentCount="7" />
  <row Id="6373" PostTypeId="2" ParentId="2356" CreationDate="2011-01-19T08:05:54.043" Score="3" ViewCount="" Body="&lt;p&gt;Keith Winstein,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;EDIT: Just to clarify, this answer describes the example given in Keith Winstein Answer on the King with the cruel statistical game.  The Bayesian and Frequentist answers both use the same information, which is to ignore the information on the number of fair and unfair coins when constructing the intervals.  If this information is not ignored, the the frequentist should use the integrated Beta-Binomial Likelihood as the sampling distribution in constructing the Confidence interval, in which case the Clopper-Pearson Confidence Interval is not appropriate, and needs to be modified. A similar adjustment should occur in the Bayesian solution.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;EDIT: I have also clarified the initial use of the clopper Pearson Interval.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;EDIT: alas, my alpha is the wrong way around, and my clopper pearson interval is incorrect.  My humblest apologies to @whuber, who correctly pointed this out, but who I initially disagreed with and ignored.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The CI Using the Clopper Pearson method is very good&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you only get one observation, then the Clopper Pearson Interval can be evaluated analytically.  Suppose the coin is comes up as &quot;success&quot; (heads) you need to choose $\theta$ such that &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$[Pr(Bi(1,\theta)\geq X)\geq\frac{\alpha}{2}] \cap [Pr(Bi(1,\theta)\leq X)\geq\frac{\alpha}{2}]$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When $X=1$ these probabilities are $Pr(Bi(1,\theta)\geq 1)=\theta$ and $Pr(Bi(1,\theta)\leq 1)=1$, so the Clopper Pearson CI implies that $\theta\geq\frac{\alpha}{2}$ (and the trivially always true $1\geq\frac{\alpha}{2}$) when $X=1$.  When $X=0$ these probabilities are $Pr(Bi(1,\theta)\geq 0)=1$ and $Pr(Bi(1,\theta)\leq 0)=1-\theta$, so the Clopper Pearson CI implies that $1-\theta \geq\frac{\alpha}{2}$, or $\theta\leq 1-\frac{\alpha}{2}$ when $X=0$.  So for a 95% CI we get $[0.025,1]$ when $X=1$, and $[0,0.975]$ when $X=0$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, one who uses the Clopper Pearson Confidence Interval will &lt;em&gt;never ever&lt;/em&gt; be beheaded.  Upon observing the interval, it is basically the whole parameter space.  But the C-P interval is doing this by giving 100% coverage to a supposedly 95% interval!  Basically, the Frequentists &quot;cheats&quot; by giving a 95% confidence interval more coverage than he/she was asked to give (although who wouldn't cheat in such a situation? if it were me, I'd give the whole [0,1] interval).  If the king asked for an &lt;strong&gt;exact&lt;/strong&gt; 95% CI, this frequentist method would fail regardless of what actually happened (perhaps a better one exists?).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What about the Bayesian Interval?  (specifically the Highest Posterior Desnity (HPD) Bayesian Interval)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because we know &lt;em&gt;a priori&lt;/em&gt; that both heads and tails can come up, the uniform prior is a reasonable choice.  This gives a posterior distribution of $(\theta|X)\sim Beta(1+X,2-X)$ .  Now, all we need to do now is create an interval with 95% posterior probability.  Similar to the clopper pearson CI, the Cummulative Beta distribution is analytic here also, so that $Pr(\theta \geq \theta^{e} | x=1) = 1-(\theta^{e})^{2}$ and $Pr(\theta \leq \theta^{e} | x=0) = 1-(1-\theta^{e})^{2}$ setting these to 0.95 gives $\theta^{e}=\sqrt{0.05}\approx 0.224$ when $X=1$ and $\theta^{e}= 1-\sqrt{0.05}\approx 0.776$ when $X=0$.  So the two credible intervals are $(0,0.776)$ when $X=0$ and $(0.224,1)$ when $X=1$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus the Bayesian will be beheaded for his HPD Credible interval in the case when he gets the bad coin &lt;em&gt;and&lt;/em&gt; the Bad coin comes up tails which will occur with a chance of $\frac{1}{10^{12}+1}\times\frac{1}{10}\approx 0$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First observation, the Bayesian Interval is smaller than the confidence interval.  Another thing is that the Bayesian would be closer to the actual coverage stated, 95%, than the frequentist.  In fact, the Bayesian is just about as close to the 95% coverage as one can get in this problem.  And contrary to Keith's statement, if the bad coin is chosen, 10 Bayesians out of 100 will on average lose their head (not all of them, because the bad coin must come up heads for the interval to not contain $0.1$).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Interestingly, if the CP-interval for 1 observation was used repeatedly (so we have N such intervals, each based on 1 observation), and the true proportion was anything between $0.025$ and $0.975$, then coverage of the 95% CI will always be 100%, and not 95%!  This clearly depends on the true value of the parameter! So this is at least one case where repeated use of a confidence interval does not lead to the desired level of confidence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To quote a &lt;em&gt;genuine&lt;/em&gt; 95% confidence interval, then &lt;em&gt;by definition&lt;/em&gt; there should be some cases (i.e. at least one) of the observed interval &lt;strong&gt;which do not contain the true value of the parameter&lt;/strong&gt;.  Otherwise, how can one justify the 95% tag?  Would it not be just a valid or invalid to call it a 90%, 50%, 20%, or even 0% interval?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I do not see how simply stating &quot;it actually means 95% or more&quot; without a complimentary restriction is satisfactory.  This is because the obvious mathematical solution is the whole parameter space, and the problem is trivial.  suppose I want a 50% CI? if it only bounds the false negatives then the whole parameter space is a valid CI using only this criteria.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Perhaps a better criterion is (and this is what I believe is implicit in the definition by Kieth) &quot;as close to 95% as possible, without going below 95%&quot;.  The Bayesian Interval would have a coverage closer to 95% than the frequentist (although not by much), and would not go under 95% in the coverage ($\text{100%}$ coverage when $X=0$, and $100\times\frac{10^{12}+\frac{9}{10}}{10^{12}+1}\text{%} &amp;gt; \text{95%}$ coverage when $X=1$).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In closing, it does seem a bit odd to ask for an interval of uncertainty, and then evaluate that interval by the using the true value which we were uncertain about.  A &quot;fairer&quot; comparison, for both confidence and credible intervals, to me seems like &lt;em&gt;the truth of the statement of uncertainty given with the interval&lt;/em&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastEditorUserId="2392" LastEditorDisplayName="" LastEditDate="2011-01-27T15:52:09.183" LastActivityDate="2011-01-27T15:52:09.183" CommentCount="5" />
  <row Id="6374" PostTypeId="2" ParentId="6181" CreationDate="2011-01-19T09:30:53.440" Score="1" ViewCount="" Body="&lt;p&gt;The multiple correlation in standard linear regression cannot be negative, the maths are easy to show it, although it depends on what &quot;multiple correlation&quot; is taken to mean.  The usual way you would calculate $R^{2}$ is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$R^2=\frac{SSR}{TSS}$$&#xA;where&#xA;$$ SSR = \sum_{i} (\hat{Y_i}-\bar{Y})^2$$&#xA;and&#xA;$$ TSS = \sum_{i} (Y_i-\bar{Y})^2$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since sums of squares can never be negative, neither can the $R^2$ value, as long as its calculated this way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, $R^2$ calculated this way can be greater than 1, if you use an estimator which does not have the observed residuals sum to zero.  Or mathematically, $R^2$ will be necessarily bounded by 1 if&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\sum_{i} (\hat{Y_i}-Y_i)=0$$&#xA;and&#xA;$$\sum_{i} (\hat{Y_i}-Y_i)\hat{Y_i}=0$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or in words, the average of the residuals is equal to 0, and the fitted values are uncorrelated with the residuals over the whole data set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is because you can expand TSS as follows&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ TSS = \sum_{i} (Y_i-\bar{Y})^2 = \sum_{i} ([Y_i-\hat{Y_i}]-[\bar{Y}-\hat{Y_i}])^2$$&#xA;$$=\sum_{i} (Y_i-\hat{Y_i})^2-2\sum_{i} [Y_i-\hat{Y_i}][\bar{Y}-\hat{Y_i}]+\sum_{i} (\bar{Y}-\hat{Y_i})^2$$&#xA;$$=\sum_{i} (Y_i-\hat{Y_i})^2-2\bar{Y}\sum_{i} [Y_i-\hat{Y_i}]+2\sum_{i} [Y_i-\hat{Y_i}]\hat{Y_i}+\sum_{i} (\bar{Y}-\hat{Y_i})^2$$&#xA;$$=\sum_{i} (Y_i-\hat{Y_i})^2+\sum_{i} (\bar{Y}-\hat{Y_i})^2$$&#xA;$$\implies TSS=SSR+\sum_{i} (Y_i-\hat{Y_i})^2 \geq SSR \geq 0$$&#xA;$$\implies 1 \geq \frac{SSR}{TSS}=R^2 \geq 0$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The constraints listed are always satisfied by the usual OLS estimators (in fact they form part of the equations that &lt;em&gt;define&lt;/em&gt; OLS estimation)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$R^2$ can go negative if it is calculated by $1-\frac{SSE}{TSS}$ Where $SSE=\sum_{i} (Y_i-\hat{Y_i})^2$ instead of the way I described.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a (silly) example of $R^2&amp;gt;1$, you can put as the estimate $\hat{Y_i}=\bar{Y_i}+TSS$ So That $SSR=n(TSS)^2$ and $R^2=n(TSS)$  Which will exceed 1 for big enough n or TSS.  To make $R^2$ go negative, set $\hat{Y_i}=Y_i+TSS$ so that $SSE=n(TSS)^2$ and $R^2=1-n(TSS)$ which will be less than 0 for big enough n and TSS &lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-01-19T09:30:53.440" />
  <row Id="6375" PostTypeId="1" CreationDate="2011-01-19T10:07:22.923" Score="7" ViewCount="140" Body="&lt;p&gt;Hi all,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to learn a bit more on interactive data visualization (zooming, pointing, brushing, point-mapping and so on).  I would welcome any:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Tutorial/guide/book(?)/video on &lt;strong&gt;how to use&lt;/strong&gt; such methods for statistical exploration.&lt;/li&gt;&#xA;&lt;li&gt;Pointers for good/interesting &lt;strong&gt;interactive data-viz packages&lt;/strong&gt; (in R, and outside of it)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Just to start the ball rolling, I know that in R there are various ways to get interactive visualization, like &lt;a href=&quot;http://www.ggobi.org/rggobi/&quot; rel=&quot;nofollow&quot;&gt;rggobi&lt;/a&gt;, the new &lt;a href=&quot;http://www.r-bloggers.com/r-and-google-visualization-api/&quot; rel=&quot;nofollow&quot;&gt;googleViz R package&lt;/a&gt;, the &lt;a href=&quot;http://cran.r-project.org/web/packages/animation/index.html&quot; rel=&quot;nofollow&quot;&gt;animation package&lt;/a&gt; and some others.  But if there are other packages worth exploring (offering things that R doesn't), I would be glad to know about them (like jmp, mathlab, spss, sas, excel, and so on).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;p.s: this is the first question to use the tag &quot;interactive-visualization&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cheers, Tal&lt;/p&gt;&#xA;" OwnerUserId="253" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-01-20T22:17:22.283" LastActivityDate="2011-01-20T22:17:22.283" Title="Resources for learning to use (/create) dynamic (/interactive) statistical visualization" Tags="&lt;data-visualization&gt;&lt;interactive-visualization&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="1" />
  <row Id="6376" PostTypeId="2" ParentId="6375" CreationDate="2011-01-19T10:43:48.173" Score="6" ViewCount="" Body="&lt;p&gt;Apart from &lt;a href=&quot;http://vis.stanford.edu/protovis/&quot; rel=&quot;nofollow&quot;&gt;Protovis&lt;/a&gt; (HTML+JS) or &lt;a href=&quot;http://mayavi.sourceforge.net/&quot; rel=&quot;nofollow&quot;&gt;Mayavi&lt;/a&gt; (Python), I would recommend &lt;a href=&quot;http://processing.org/&quot; rel=&quot;nofollow&quot;&gt;Processing&lt;/a&gt; which is&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;an open source programming language&#xA;  and environment for people who want to&#xA;  create images, animations, and&#xA;  interactions. Initially developed to&#xA;  serve as a software sketchbook and to&#xA;  teach fundamentals of computer&#xA;  programming within a visual context.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;There are a lot of open-source scripts on &lt;a href=&quot;http://www.openprocessing.org/&quot; rel=&quot;nofollow&quot;&gt;http://www.openprocessing.org/&lt;/a&gt;, and a lot of &lt;a href=&quot;http://processing.org/learning/books/&quot; rel=&quot;nofollow&quot;&gt;related books&lt;/a&gt; that deal with Processing but also data visualization. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know there is a project to provide an R interface, &lt;a href=&quot;https://r-forge.r-project.org/projects/rprocessing/&quot; rel=&quot;nofollow&quot;&gt;rprocessing&lt;/a&gt;, but I don't know how it goes. There's also an interface with clojure/incanter (see e.g., &lt;a href=&quot;http://data-sorcery.org/2009/08/30/processing-intro/&quot; rel=&quot;nofollow&quot;&gt;Creating Processing Visualizations with Clojure and Incanter&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are many online resources, among which Stanford class notes, e.g. &lt;a href=&quot;https://graphics.stanford.edu/wikis/cs448b-10-fall&quot; rel=&quot;nofollow&quot;&gt;CS448B&lt;/a&gt;, or &lt;a href=&quot;http://fellinlovewithdata.com/guides/7-classic-foundational-vis-papers&quot; rel=&quot;nofollow&quot;&gt;7 Classic Foundational Vis Papers You Might not Want to Publicly Confess you Don’t Know&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-01-19T10:52:57.817" LastActivityDate="2011-01-19T10:52:57.817" />
  <row Id="6377" PostTypeId="1" CreationDate="2011-01-19T12:41:46.500" Score="3" ViewCount="129" Body="&lt;p&gt;I have hourly data for a variable for several years. I want to analyse each month separately. How do I arrange the data for the same month of different years?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, suppose I have Jan 1997, Jan 1998m and Jan 1999. Do I have to find a mean for say 18 Jan 1997, 18 Jan 1998 and 18 Jan1999. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Hours Jan 96 Jan97 Jan98 Jan99&#xA;&#xA;01:00 3 2 4 7 02:00 5 2 6 5 . . . . . . . . . . . 23:00 NA 5 7 3 00:00 01:00 . . .&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The dots each represent a value and the data is for all days of the month. NA means that the value is missing.&lt;/p&gt;&#xA;" OwnerUserId="2959" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-03T20:54:19.110" LastActivityDate="2011-03-03T20:54:19.110" Title="Arranging hourly data for several years" Tags="&lt;time-series&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="1" />
  <row Id="6378" PostTypeId="2" ParentId="6368" CreationDate="2011-01-19T12:52:07.167" Score="1" ViewCount="" Body="&lt;p&gt;Since you've got a 2x2 design, there is only one (unique) interaction contrast. In that case, the interaction-test in the two-factorial ANOVA is equivalent to the two-sided test of the given contrast. Since you've got a one-sided a-priori hypothesis, you can take the interaction p-value of the ANOVA output and divide it by 2. If you want to test other (non-interaction) contrasts as well, you may have to think about p-value adjustment. In R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; Nj  &amp;lt;- c(41, 37, 42, 40)     # generate some data in a 2x2 design&#xA;&amp;gt; DVa &amp;lt;- rnorm(Nj[1], 0,   1)&#xA;&amp;gt; DVb &amp;lt;- rnorm(Nj[2], 0.3, 1)&#xA;&amp;gt; DVc &amp;lt;- rnorm(Nj[3], 0.6, 1)&#xA;&amp;gt; DVd &amp;lt;- rnorm(Nj[4], 1.0, 1)&#xA;&amp;gt; DV  &amp;lt;- c(DVa, DVb, DVc, DVd)&#xA;&amp;gt; IV1 &amp;lt;- factor(rep(1:2, c(sum(Nj[1:2]), sum(Nj[3:4]))))&#xA;&amp;gt; IV2 &amp;lt;- factor(rep(c(1:2, 1:2), Nj))&#xA;&amp;gt; summary(aov(DV ~ IV1*IV2))    # ANOVA with interaction test&#xA;             Df  Sum Sq Mean Sq F value    Pr(&amp;gt;F)    &#xA;IV1           1  20.959 20.9593 21.8302 6.392e-06 ***&#xA;IV2           1   2.358  2.3577  2.4556   0.11913    &#xA;IV1:IV2       1   3.172  3.1722  3.3040   0.07103 .  &#xA;Residuals   156 149.777  0.9601                      &#xA;&#xA;# test single contrast in associated one-way design&#xA;&amp;gt; IV &amp;lt;- interaction(IV1, IV2)    # factor for associated one-way design&#xA;&amp;gt; cc &amp;lt;- c(-1/2, 1/2, 1/2, -1/2)  # contrast coefficients&#xA;&amp;gt; library(multcomp)              # for glht()&#xA;&#xA;# test single contrast, p-value = 0.5 * that of ANOVA interaction test&#xA;&amp;gt; summary(glht(aov(DV ~ IV), linfct=mcp(IV=cc), alternative=&quot;less&quot;))&#xA;         Simultaneous Tests for General Linear Hypotheses&#xA;Multiple Comparisons of Means: User-defined Contrasts&#xA;Fit: aov(formula = DV ~ IV)&#xA;Linear Hypotheses:&#xA;       Estimate Std. Error t value Pr(&amp;lt;t)  &#xA;1 &amp;gt;= 0  -0.2819     0.1551  -1.818 0.0355 *&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1909" LastEditorUserId="1909" LastEditorDisplayName="" LastEditDate="2011-01-19T13:06:52.697" LastActivityDate="2011-01-19T13:06:52.697" CommentCount="1" />
  <row Id="6379" PostTypeId="1" CreationDate="2011-01-19T13:07:36.177" Score="4" ViewCount="110" Body="&lt;p&gt;I have a set of players. They play with each other (pairwise). Pairs of player are chosen randomly. In one game, one player wins and another one lose. The player play with each other a limited number of games (some players play more games some less). So, I have data (who wins whom and how many times). Now I assume that every player has ranking that determines the probability to win.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;And now I want to check if this assumption is actually truth.&lt;/strong&gt; Of course, I can use &lt;a href=&quot;http://en.wikipedia.org/wiki/Elo_rating_system&quot; rel=&quot;nofollow&quot;&gt;Elo rating system&lt;/a&gt; or &lt;a href=&quot;http://en.wikipedia.org/wiki/PageRank&quot; rel=&quot;nofollow&quot;&gt;PageRank algorithm&lt;/a&gt; to calculate rating of every player. But by calculating ratings I do not prove that they (ratings) actually exist or that they mean anything.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words, I want to have a way to prove (or to check) that player do have different strength. How can I do it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;ADDED&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To be more specific. I have 8 players and only 18 games. So, there a lot of pair of players that did not play with each other and there a lot of pairs that played only one with each other. As a consequence, I cannot estimate the probability of win for a given pair of players. I also see, for example that there is a player that won 6 times in 6 games. But may be it is just a coincidence. &lt;/p&gt;&#xA;" OwnerUserId="2407" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-01-19T15:58:12.263" LastActivityDate="2011-01-19T18:05:02.633" Title="How to prove that Elo rating or Page ranking have a meaning for my set?" Tags="&lt;rating&gt;&lt;elo&gt;&lt;pagerank&gt;&lt;ranking&gt;" AnswerCount="3" CommentCount="1" FavoriteCount="2" />
  <row Id="6380" PostTypeId="1" AcceptedAnswerId="6383" CreationDate="2011-01-19T13:25:00.700" Score="8" ViewCount="437" Body="&lt;p&gt;I am using GNU R at a Ubuntu-Lucid PC which has 4 CPUs.&#xA;In order to use all 4 CPUs, I installed the &quot;r-cran-multicore&quot; package.&#xA;As the package's manual lacks of practical examples that I understand, I need advice in how to optimize my script in order to make use of all 4 CPUs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My dataset is a data.frame (called P1) that has 50,000 rows and 1600 cols. For each row, I'd like to calc the maximun, sum and mean. My script looks as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;p1max &amp;lt;- 0&#xA;p1mean &amp;lt;- 0&#xA;p1sum &amp;lt;-0&#xA;plength &amp;lt;- length(P1[,1])&#xA;for(i in 1:plength){&#xA;   p1max &amp;lt;- c(p1max, max(P1[i,]))&#xA;   p1mean &amp;lt;- c(p1mean, mean(P1[i,]))&#xA;   p1sum &amp;lt;- c(p1sum, sum(P1[i,]))&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Could anyone please tell me how to modify and run the script in order to use all 4 CPUs?&lt;/p&gt;&#xA;" OwnerUserId="2838" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-01-19T21:44:00.163" LastActivityDate="2011-01-19T21:44:00.163" Title="How to optimize my R script in order to use &quot;multicore&quot;" Tags="&lt;r&gt;" AnswerCount="3" CommentCount="4" FavoriteCount="5" />
  <row Id="6381" PostTypeId="2" ParentId="6380" CreationDate="2011-01-19T13:37:31.400" Score="0" ViewCount="" Body="&lt;p&gt;Have a look at the &lt;a href=&quot;http://cran.r-project.org/web/packages/snow/index.html&quot; rel=&quot;nofollow&quot;&gt;snow&lt;/a&gt; and &lt;a href=&quot;http://cran.r-project.org/web/packages/snowfall/index.html&quot; rel=&quot;nofollow&quot;&gt;snowfall&lt;/a&gt; packages. Plenty of examples with those...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to speed up that specific code rather than learning about R and parallelism you should do this&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;P1 = matrix(rnorm(1000), ncol=10, nrow=10&#xA;apply(P1, 1, max)&#xA;apply(P1, 1, mean)&#xA;apply(P1, 1, sum)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="300" LastEditorUserId="300" LastEditorDisplayName="" LastEditDate="2011-01-19T14:20:03.513" LastActivityDate="2011-01-19T14:20:03.513" CommentCount="3" />
  <row Id="6382" PostTypeId="2" ParentId="6353" CreationDate="2011-01-19T14:03:26.643" Score="2" ViewCount="" Body="&lt;ol&gt;&#xA;&lt;li&gt;If this is an SPSS syntax question, the answer is just put the categorical variable, coded appropriately, into the variable list for &quot;independent variables&quot; along with the continuous one. &lt;/li&gt;&#xA;&lt;li&gt;On the statistics: Is your categorical variable binary? If so, you need to use a dummy or other valid contrast code. If it is not binary, is your categorical variable ordinal or nominal? If nominal, then again, you must use some contrasting code strategy--in effect modeling the impact of each level of the variable on the outcome or &quot;dependent&quot; variable. If the categorical variable is ordinal, then &lt;em&gt;most likely&lt;/em&gt; the sensible thing to do is to enter it as-is into the model, just as you would with a continuous predictor (i.e., &quot;independent&quot;) variable. You would be assuming, in that case, that the increments between levels of the categorical predictor (&quot;indepdent&quot;) variable; only rarely will this be a mistake, but when it is, you should again use a contrast code &amp;amp; model the impact of each level. This question comes up in this forum quite often -- here is a good &lt;a href=&quot;http://stats.stackexchange.com/questions/2619/continuous-and-categorical-variable-data-analysis&quot;&gt;analaysis&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;How to handle missing data is, in my view, a completely separate matter. My understanding is that pairwise deletion is not viewed as a valid approach for multivariate regression. Listwise is pretty common but can can also bias results &amp;amp; certainly is a shame. Multiple imputation is a thing of beauty.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="2347" LastActivityDate="2011-01-19T14:03:26.643" />
  <row Id="6383" PostTypeId="2" ParentId="6380" CreationDate="2011-01-19T14:25:28.853" Score="7" ViewCount="" Body="&lt;p&gt;Use &lt;a href=&quot;http://cran.r-project.org/web/packages/foreach/index.html&quot; rel=&quot;nofollow&quot;&gt;foreach&lt;/a&gt; and &lt;a href=&quot;http://cran.r-project.org/web/packages/doMC/index.html&quot; rel=&quot;nofollow&quot;&gt;doMC&lt;/a&gt;. The detailed explanation can be &lt;a href=&quot;http://cran.r-project.org/web/packages/doMC/vignettes/gettingstartedMC.pdf&quot; rel=&quot;nofollow&quot;&gt;found here&lt;/a&gt;. Your script will change very little, the line &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for(i in 1:plength){&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;should be changed to&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;foreach(i=1:plength) %dopar% { &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The prerequisites for any multitasking script using these packages are&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(foreach)&#xA;library(doMC)&#xA;registerDoMC()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Note of caution. According to the documentation you cannot use this in GUI. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for your problem, do you really need multitasking? Your data.frame takes about 1.2GB of RAM, so it should fit into your memory. So you can simply use apply:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;p1smry &amp;lt;- apply(P1,1,summary)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The result will be a matrix with summaries of each row.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also use function &lt;strong&gt;mclapply&lt;/strong&gt; which is in the package multicore. Then your script might look like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;loopfun &amp;lt;- function(i) {&#xA;     summary(P1[i,])&#xA;}&#xA;&#xA;res &amp;lt;- mclapply(1:nrow(P1),loopfun)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This will return the list, where i-th element will be the summary of i-th row. You can convert it to matrix using &lt;strong&gt;sapply&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mres &amp;lt;- sapply(res,function(x)x)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-01-19T14:34:42.153" LastActivityDate="2011-01-19T14:34:42.153" CommentCount="1" />
  <row Id="6384" PostTypeId="2" ParentId="6380" CreationDate="2011-01-19T14:54:32.927" Score="11" ViewCount="" Body="&lt;p&gt;You've already got an answer as to how to use more than one core, but the real problem is with the way you have written your loops. &lt;strong&gt;Never extend your result vector/object at each iteration of a loop&lt;/strong&gt;. If you do this, you force R to copy your result vector/object and extend it which all takes time. Instead, preallocate enough storage space before you start the loop and fill in as you go along. Here is an example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(1)&#xA;p1 &amp;lt;- matrix(rnorm(10000), ncol=100)&#xA;system.time({&#xA;p1max &amp;lt;- p1mean &amp;lt;- p1sum &amp;lt;- numeric(length = 100)&#xA;for(i in seq_along(p1max)){&#xA;   p1max[i] &amp;lt;- max(p1[i,])&#xA;   p1mean[i] &amp;lt;- mean(p1[i,])&#xA;   p1sum[i ]&amp;lt;- sum(p1[i,])&#xA;}&#xA;})&#xA;&#xA;   user  system elapsed &#xA;  0.005   0.000   0.005&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Or you can do these things via &lt;code&gt;apply()&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;system.time({&#xA;p1max2 &amp;lt;- apply(p1, 1, max)&#xA;p1mean2 &amp;lt;- apply(p1, 1, mean)&#xA;p1sum2 &amp;lt;- apply(p1, 1, sum)&#xA;})&#xA;   user  system elapsed &#xA;  0.007   0.000   0.006 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But note that this is &lt;strong&gt;no&lt;/strong&gt; faster than doing the loop properly and sometimes slower.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, always be on the lookout for vectorised code. You can do row sums and means using &lt;code&gt;rowSums()&lt;/code&gt; and &lt;code&gt;rowMeans()&lt;/code&gt; which are quicker than either the loop or &lt;code&gt;apply&lt;/code&gt; versions:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;system.time({&#xA;p1max3 &amp;lt;- apply(p1, 1, max)&#xA;p1mean3 &amp;lt;- rowMeans(p1)&#xA;p1sum3 &amp;lt;- rowSums(p1)&#xA;})&#xA;&#xA;   user  system elapsed &#xA;  0.001   0.000   0.002 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If I were a betting man, I would have money on the third approach I mention beating &lt;code&gt;foreach()&lt;/code&gt; or the other multi-core options in a speed test on your matrix because they would have to speed things up considerably to justify the overhead incurred in setting up the separate processes that are farmed out the the different CPU cores.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Update: Following the comment from @shabbychef is it faster to do the sums once and reuse in the computation of the mean?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;system.time({&#xA;    p1max4 &amp;lt;- apply(p1, 1, max)&#xA;    p1sum4 &amp;lt;- rowSums(p1)&#xA;    p1mean4 &amp;lt;- p1sum4 / ncol(p1)&#xA;    })&#xA;&#xA;   user  system elapsed &#xA;  0.002   0.000   0.002&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Not in this test run, but this is far from exhaustive...&lt;/p&gt;&#xA;" OwnerUserId="1390" LastEditorUserId="1390" LastEditorDisplayName="" LastEditDate="2011-01-19T21:32:27.480" LastActivityDate="2011-01-19T21:32:27.480" CommentCount="7" />
  <row Id="6385" PostTypeId="2" ParentId="6379" CreationDate="2011-01-19T15:34:31.023" Score="0" ViewCount="" Body="&lt;p&gt;You want to test the hypothesis that probability of a result depends on the matchup. $H_0$, then, is that every game is essentially a coin flip.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A simple test for this would be calculating the proportion of times the player with more previous games played will win, and comparing that to the binomial cumulative distribution function. That should show the existence of some kind of effect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're interested about the quality of the Elo rating system for your game, a simple method would be to run a 10-fold crossvalidation on the predictive performance of the Elo model (which actually assumes outcomes aren't iid, but I'll ignore that) and comparing that to a coin flip.&lt;/p&gt;&#xA;" OwnerUserId="2456" LastActivityDate="2011-01-19T15:34:31.023" CommentCount="1" />
  <row Id="6386" PostTypeId="2" ParentId="6377" CreationDate="2011-01-19T15:40:54.390" Score="3" ViewCount="" Body="&lt;p&gt;Could you give an example of what your dataset looks like?  You're probably looking for the &lt;a href=&quot;http://gbi.agrsci.dk/~shd/misc/Rdocs/reshape.pdf&quot; rel=&quot;nofollow&quot;&gt;reshape command&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lets say you have a dataframe with 2 columns, Date and X, where X is the time series you want to analyze.  This dataset is in &quot;long&quot; form, with one observation per row.  To convert it to &quot;wide&quot; form, where each month is a separate column, do something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Date&amp;lt;-c('2009-01-01','2009-02-01','2010-01-01','2010-02-01')&#xA;X&amp;lt;-c(1,2,3,4)&#xA;Year&amp;lt;-as.numeric(format(as.POSIXct(Date), &quot;%Y&quot;))&#xA;Month&amp;lt;-as.numeric(format(as.POSIXct(Date), &quot;%m&quot;))&#xA;&#xA;df&amp;lt;-as.data.frame(cbind(Year,Month,X))&#xA;df&#xA;df&amp;lt;-reshape(df,v.names='X',idvar=&quot;Year&quot;,timevar='Month',direction='wide')&#xA;df&#xA;&#xA;mean(df$X.1)  #January mean&#xA;mean(df$X.2)  #Feb mean&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2817" LastActivityDate="2011-01-19T15:40:54.390" />
  <row Id="6387" PostTypeId="1" AcceptedAnswerId="6397" CreationDate="2011-01-19T16:20:10.837" Score="4" ViewCount="131" Body="&lt;p&gt;I would be very glad if someone can point me out to statistics and econometrics distance learning courses like &lt;a href=&quot;http://www2.statistics.com&quot; rel=&quot;nofollow&quot;&gt;http://www2.statistics.com&lt;/a&gt;.&#xA;Thanks in advance&lt;/p&gt;&#xA;" OwnerUserId="1251" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-01-19T19:10:37.907" LastActivityDate="2011-01-19T19:10:37.907" CommunityOwnedDate="2011-01-19T18:49:49.017" Title="Statistics and econometrics distance learning" Tags="&lt;teaching&gt;&lt;econometrics&gt;" AnswerCount="2" FavoriteCount="2" />
  <row Id="6388" PostTypeId="2" ParentId="6379" CreationDate="2011-01-19T16:23:40.973" Score="2" ViewCount="" Body="&lt;p&gt;If you want to test the null hypothesis that each player is equally likely to win or lose each game, I think you want a test of &lt;strong&gt;symmetry of the contingency table&lt;/strong&gt; formed by tabulating winners against losers. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Set up the data so that you have two variables, 'winner' and 'loser' containing the ID of the winner and loser for each game, i.e. each 'observation' is a game. You can then construct a contingency table of winner vs loser. Your null hypothesis is that you'd expect this table to be symmetric (on average over repeated tournaments). In your case, you'll get an 8×8 table where most of the entries are zero (corresponding to players that never met), ie. the table will be &lt;em&gt;very&lt;/em&gt; sparse, so an 'exact' test will almost certainly be necessary rather than one relying on asymptotics. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Such an exact test is available in Stata with the &lt;a href=&quot;http://www.stata.com/help.cgi?symmetry&quot; rel=&quot;nofollow&quot;&gt;symmetry command&lt;/a&gt;. In this case, the syntax would be:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;symmetry winner loser, exact&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;No doubt it's also implemented in other statistics packages that I'm less familiar with.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-01-19T16:23:40.973" CommentCount="2" />
  <row Id="6389" PostTypeId="2" ParentId="6387" CreationDate="2011-01-19T16:24:23.583" Score="2" ViewCount="" Body="&lt;p&gt;Stanford has some good programs as part of &lt;a href=&quot;http://scpd.stanford.edu/&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;SCPD&lt;/strong&gt;&lt;/a&gt;.  In particular, you might be interested in the &lt;a href=&quot;http://scpd.stanford.edu/public/category/courseCategoryCertificateProfile.do?method=load&amp;amp;certificateId=1209602#searchResults&quot; rel=&quot;nofollow&quot;&gt;&quot;Data Mining and Applications Graduate Certificate&quot;&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2011-01-19T16:24:23.583" CommunityOwnedDate="2011-01-19T18:49:49.017" />
  <row Id="6390" PostTypeId="2" ParentId="5559" CreationDate="2011-01-19T17:28:12.457" Score="2" ViewCount="" Body="&lt;p&gt;One approach would be to use &lt;a href=&quot;http://en.wikipedia.org/wiki/V-optimal_histograms&quot; rel=&quot;nofollow&quot;&gt;v-optimal histograms&lt;/a&gt;, which are histograms that select bin sizes in order to minimize the sum of squared errors of the representation. Another alternative is equi-depth histograms which may be useful if you'd like each bin to contain (approximately) the same number of elements.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As Whuber mentioned, it would be helpful to first identify what it is that qualifies as a &quot;good&quot; binning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the revised question, it sounds like slotishtype is looking for an equi-depth histogram. Here is some example Matlab code for computing the bins, hope this helps.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;% INPUT: &#xA;% v     input vector of values (sorted)&#xA;% n     number of bins youd like to split the data into&#xA;%&#xA;% OUTPUT: &#xA;% domain    provides the representative value for each bin&#xA;% rep       provides the range of each bin&#xA;%&#xA;&#xA;% sample input from a skewed distribution&#xA;v = sort(exprnd(1, 1000, 1));&#xA;% number of bins&#xA;n = 5;&#xA;&#xA;N = length(v);&#xA;d = zeros(N, 1);&#xA;&#xA;[sv ix] = sort(v);&#xA;% figure out how many elements we'll put into each bin&#xA;bin_size = round(N / n);&#xA;rep = zeros(n, 3);&#xA;&#xA;s = 1;&#xA;for i = 1:n&#xA;    e = min(N, s + bin_size - 1);&#xA;    % compute the representative of this bin &#xA;    % (I use the mean, but you could change this depending on your problem)&#xA;    rep(i, 1) = mean(sv(s:e));&#xA;&#xA;    % this is just to show the range of values each bin takes on&#xA;    rep(i, 2) = min(sv(s:e));&#xA;    rep(i, 3) = max(sv(s:e));&#xA;&#xA;    % d builds up a representation of the original data using the bin&#xA;    % representatives for each value (vector quantization)&#xA;    d(ix(s:e)) = rep(i, 1);&#xA;&#xA;    s = e + 1;&#xA;end&#xA;domain = unique(rep(:, 1));&#xA;&#xA;fprintf('Bin representative vlaues:\n');&#xA;rep&#xA;&#xA;fprintf('Range of each bin:\n');&#xA;rep(:, 3) - rep(:, 2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1913" LastEditorUserId="1913" LastEditorDisplayName="" LastEditDate="2011-01-20T18:06:28.053" LastActivityDate="2011-01-20T18:06:28.053" CommentCount="5" />
  <row Id="6391" PostTypeId="1" CreationDate="2011-01-19T17:33:49.987" Score="3" ViewCount="83" Body="&lt;p&gt;Hi&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wonder how to reason when selecting samples. I am doing a panel data regression analysis about how the euro membership correlates with the budget deficit in the member countriues.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using R that is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;BUDGETDEFICIT ~ EURODUMMY + some_controll_variables&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;EURODUMMY is the variable of interest. Using panel data I have anual observations for many countriies. My question is how selecting what sample to use effects the regression analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should I only use countries that has introduced the Euro?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is adding other European countries or western like countries benefitial as they add more datapoints and/or acts like a reference gropup?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or is id only bad for the data to add countries that has not introduced the euro as they have no variation in the variable of interest (EURODUMMY)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other comments regarding this problem is also welcome.&lt;/p&gt;&#xA;" OwnerUserId="2724" LastActivityDate="2011-01-20T10:01:57.660" Title="Sample selection and variation in the variable of interest when using panel data" Tags="&lt;r&gt;&lt;sampling&gt;&lt;panel-data&gt;" AnswerCount="1" />
  <row Id="6392" PostTypeId="1" AcceptedAnswerId="6394" CreationDate="2011-01-19T17:55:22.683" Score="3" ViewCount="55" Body="&lt;p&gt;I am in the process of modeling the random variable X as the days from issuing an invoice until its payment.  This variable is dependent on the days of credit for the invoice, so there are distinct X's depending on the days of credit, let's call them X_c.  I'm starting with the most common, X_30, which should be centered around 30 with a very heavy tail and a very rapid ramp up of the density getting to 30. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A basic histogram looks like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/p10ia.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have lots of data to try and fit but would like to get some pointer into which distributions might model this better from a conceptual point of view.&lt;/p&gt;&#xA;" OwnerUserId="2808" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-01-19T19:17:37.303" LastActivityDate="2011-01-19T19:17:37.303" Title="Where to start in estimating the days until payment of an invoice?" Tags="&lt;distributions&gt;&lt;modeling&gt;&lt;model-selection&gt;&lt;survival&gt;&lt;goodness-of-fit&gt;" AnswerCount="1" />
  <row Id="6393" PostTypeId="2" ParentId="6379" CreationDate="2011-01-19T18:05:02.633" Score="2" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;You need a probability model.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The idea behind a ranking system is that a single number adequately characterizes a player's ability.  We might call this number their &quot;strength&quot; (because &quot;rank&quot; already means something specific in statistics).  We would predict that player A will beat player B when strength(A) exceeds strength(B).  But this statement is too weak because (a) it is not quantitative and (b) it does not account for the possibility of a weaker player occasionally beating a stronger player.  We can overcome both problems by &lt;em&gt;supposing the probability that A beats B depends only on the difference in their strengths.&lt;/em&gt;  If this is so, then we can re-express all the strengths is necessary so that &lt;em&gt;the difference in strengths equals the log odds of a win.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Specifically, this model is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$logit(\Pr(A \text{ beats } B)) = \lambda_A - \lambda_B$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where, by definition, $logit(p) = \log(p) - \log(1-p)$ is the log odds and I have written $\lambda_A$ for player A's strength, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This model has as many parameters as players (but there is one less degree of freedom, because it can only identify &lt;em&gt;relative&lt;/em&gt; strengths, so we would fix one of the parameters at an arbitrary value).  It is a kind of &lt;a href=&quot;http://en.wikipedia.org/wiki/Generalized_linear_model&quot; rel=&quot;nofollow&quot;&gt;generalized linear model&lt;/a&gt; (in the Binomial family, with logit link).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The parameters can be estimated by &lt;a href=&quot;http://en.wikipedia.org/wiki/Maximum_likelihood&quot; rel=&quot;nofollow&quot;&gt;Maximum Likelihood&lt;/a&gt;.  The same theory provides a means to erect confidence intervals around the parameter estimates and to test hypotheses (such as whether the strongest player, according to the estimates, is significantly stronger than the estimated weakest player).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Specifically, the likelihood of a set of games is the product&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\prod_{\text{all games}}{\frac{\exp(\lambda_{\text{winner}} - \lambda_{\text{loser}})}{1 + \exp(\lambda_{\text{winner}} - \lambda_{\text{loser}})}}.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After fixing the value of one of the $\lambda$, the estimates of the others are the values that maximize this likelihood.  Thus, varying any of the estimates reduces the likelihood from its maximum.  If it is reduced too much, it is not consistent with the data.  In this fashion we can find confidence intervals for all the parameters: they are the limits in which varying the estimates does not overly decrease the log likelihood.  General hypotheses can similarly be tested: a hypothesis constrains the strengths (such as by supposing they are all equal), this constraint limits how large the likelihood can get, and if this restricted maximum falls too far short of the actual maximum, the hypothesis is rejected.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;In this particular problem there are 18 games and 7 free parameters.  In general that is too many parameters: there is so much flexibility that the parameters can be quite freely varied without changing the maximum likelihood much.  Thus, applying the ML machinery is likely to prove the obvious, which is that there likely are not enough data to have confidence in the strength estimates.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-01-19T18:05:02.633" />
  <row Id="6394" PostTypeId="2" ParentId="6392" CreationDate="2011-01-19T18:30:35.307" Score="4" ViewCount="" Body="&lt;p&gt;As this is an example of &lt;a href=&quot;http://en.wikipedia.org/wiki/Time_to_event_analysis&quot; rel=&quot;nofollow&quot;&gt;time to event analysis&lt;/a&gt;, the obvious starting point would be the &lt;a href=&quot;http://en.wikipedia.org/wiki/Weibull_distribution&quot; rel=&quot;nofollow&quot;&gt;Weibull distribution&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-01-19T18:30:35.307" CommentCount="4" />
  <row Id="6395" PostTypeId="2" ParentId="4245" CreationDate="2011-01-19T18:34:36.517" Score="-1" ViewCount="" Body="&lt;p&gt;marginal mean is the mean of means&lt;/p&gt;&#xA;" OwnerUserId="2840" LastActivityDate="2011-01-19T18:34:36.517" CommentCount="1" />
  <row Id="6396" PostTypeId="2" ParentId="4840" CreationDate="2011-01-19T18:52:37.563" Score="2" ViewCount="" Body="&lt;p&gt;Try the forecasting package for r. Specifically, the auto.arima() and ets() functions will model the seasonality and trend in the ozone data and allow you to make monthly predictions for future.&lt;/p&gt;&#xA;" OwnerUserId="2817" LastActivityDate="2011-01-19T18:52:37.563" />
  <row Id="6397" PostTypeId="2" ParentId="6387" CreationDate="2011-01-19T19:03:46.913" Score="2" ViewCount="" Body="&lt;p&gt;Here are a few links:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.stat.tamu.edu/dist/&quot; rel=&quot;nofollow&quot;&gt;http://www.stat.tamu.edu/dist/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.stat.colostate.edu/distance_degree.html&quot; rel=&quot;nofollow&quot;&gt;http://www.stat.colostate.edu/distance_degree.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.worldcampus.psu.edu/AppliedStatisticsCertificate.shtml&quot; rel=&quot;nofollow&quot;&gt;http://www.worldcampus.psu.edu/AppliedStatisticsCertificate.shtml&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2775" LastActivityDate="2011-01-19T19:03:46.913" CommunityOwnedDate="2011-01-19T19:03:46.913" />
  <row Id="6398" PostTypeId="2" ParentId="6268" CreationDate="2011-01-19T19:08:24.507" Score="3" ViewCount="" Body="&lt;p&gt;Here are some suggestions (about your plot, not about how I would illustrate correlation/regression analysis):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The two univariate plots you show in the right and left margins may be simplified with a call to &lt;code&gt;rug()&lt;/code&gt;;&lt;/li&gt;&#xA;&lt;li&gt;I find more informative to show a density plot of $X$ and $Y$ or a boxplot, at risk of being evocative of the idea of a bi-normality assumption which makes no sense in this context;&lt;/li&gt;&#xA;&lt;li&gt;In addition to the regression line, it is worth showing a non-parametric estimate of the trend, like a loess (this is good practice and highly informative about possible local non linearities);&lt;/li&gt;&#xA;&lt;li&gt;Points might be highlighted (with varying color or size) according to Leverage effect or Cook distances, i.e. any of those measures that show how influential individual values are on the estimated regression line. I'll second @DWin's comment and I think it is better to highlight how individual points &quot;degrade&quot; goodness-of-fit or induce some kind of departure from the linearity assumption.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Of note, this graph assumes X and Y are non-paired data, otherwise I would stick to a Bland-Altman plot ($(X-Y)$ against $(X+Y)/2$), in addition to scatterplot.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-01-19T19:08:24.507" />
  <row Id="6400" PostTypeId="1" AcceptedAnswerId="6402" CreationDate="2011-01-19T20:48:08.387" Score="4" ViewCount="263" Body="&lt;p&gt;I am conducting a mulitple first order regression analysis of genetic data. The vectors of y-values do not all follow a normal distribution, therefore I need to implement a non-parametric regression using ranks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is the &lt;code&gt;lm()&lt;/code&gt; function in R suitable for this, i.e.,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;lin.reg &amp;lt;- lm(Y~X*Z)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where Y, X and Z are vectors of ordinal categorical variables?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am interested in the p-value assigned to the coefficient of the interaction term in the first order model. The &lt;code&gt;lm()&lt;/code&gt; function obtains this from a t-test, i.e., is the interaction coefficient significantly different from zero.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is the automatic implementation of a t-test to determine this p-value appropriate when the regression model is carried out on data as described?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sample data for clarity:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Y &amp;lt;- c(4, 1, 2, 3) # A vector of ranks&#xA;X &amp;lt;- c(0, 2, 1, 1) # A vector of genotypes (0 = aa, 1 = ab, 2 = bb)&#xA;Z &amp;lt;- c(2, 2, 1, 0)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2842" LastEditorUserId="2842" LastEditorDisplayName="" LastEditDate="2011-01-19T21:10:30.230" LastActivityDate="2011-01-21T17:19:22.187" Title="Non-parametric regression" Tags="&lt;r&gt;&lt;regression&gt;&lt;nonparametric&gt;&lt;genetics&gt;" AnswerCount="1" CommentCount="5" FavoriteCount="2" />
  <row Id="6401" PostTypeId="1" AcceptedAnswerId="6403" CreationDate="2011-01-20T00:58:16.340" Score="2" ViewCount="151" Body="&lt;p&gt;Hi All,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am currently estimating a bunch of ARMA models, and using them to predict subsets of my data. In order to evaluate their predictive accuracy I would like to make some ROC plots, however since all of my variables are continuous, I wonder how this could be done in R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Best, Thomas&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S: I have looked at the ROCR package, but this seems to only work for dichotomous variables.&lt;/p&gt;&#xA;" OwnerUserId="2704" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2011-01-20T03:58:52.230" LastActivityDate="2011-01-20T03:58:52.230" Title="ROC plot for continuous data in R" Tags="&lt;r&gt;&lt;forecasting&gt;&lt;predictive-models&gt;&lt;roc&gt;" AnswerCount="1" />
  <row Id="6402" PostTypeId="2" ParentId="6400" CreationDate="2011-01-20T03:11:20.827" Score="1" ViewCount="" Body="&lt;p&gt;If your response variable is ordinal, you may want to consider and &quot;ordered logistic regression&quot;.  This is basically where you model the cumulative probabilities {in the simple example, you would model $Pr(Y\leq 1),Pr(Y\leq 2),Pr(Y\leq 3)$}.  This incorporates the ordering of the response into the model, &lt;em&gt;without&lt;/em&gt; the need for an arbitrary assumption which transforms the ordered response into a numerical one (although having said that, this can be a useful first step in exploratory analysis, or in selecting which $X$ and $Z$ variables are not necessary)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a way that you can get the glm() function in R to give you the MLE's for this model (other wise you would need to write your own algorithm to get the MLEs).  You define a new set of variables, say $W$, where these are defined as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$W_{1jk} = \frac{Y_{1jk}}{\sum_{i=1}^{i=I} Y_{ijk}}$$&#xA;$$W_{2jk} = \frac{Y_{2jk}}{\sum_{i=2}^{i=I} Y_{ijk}}$$&#xA;$$...$$&#xA;$$W_{I-1,jk} = \frac{Y_{I-1,jk}}{\sum_{i=I-1}^{i=R} Y_{ijk}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $i=1,..,I$ indexes the $Y$ categories, $j=1,..,J$ indexes the $X$ categories, and $k=1,..,K$ indexes the $Z$ categories.  Then fit a glm() of W on X and Z using the complimentary log-log link function.  Denoting $\theta_{ijk}=Pr(Y_{ijk}\leq i)$ as the cumulative probability, the MLE's of the theta's (assuming a multi-nomial distribution for $Y_{ijk}$ values) is then&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\hat{\theta}_{ijk}=\hat{W}_{ijk}+\hat{\theta}_{(i-1)jk}(1-\hat{W}_{ijk}) \ \ \ i=1,\dots ,I-1$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $\hat{\theta}_{0jk}=0$ and $\hat{\theta}_{Ijk}=1$ and $\hat{W}_{ijk}$ are the fitted values from the glm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can then use the deviance table (use the anova() function on the glm object) to assess the significance of the regressor variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;EDIT: one thing I forgot to mention in my original answer was that in the glm() function, you need to specify weights when fitting the model to&lt;/em&gt; $W$, &lt;em&gt;which are equal to the denominators in the respective fractions defining each&lt;/em&gt; $W$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could also try a Bayesian approach, but you would most likely need to use sampling techniques to get your posterior, and using the multinomial likelihood (but parameterised with respect to $\theta_{ijk}$, so the likelihood function will have &lt;em&gt;differences&lt;/em&gt; of the form $\theta_{ijk}-\theta_{i-1,jk}$), the MLE's are a good &quot;first crack&quot; at genuinely fitting the model, and give an approximate Bayesian solution (as you may have noticed, I prefer Bayesian inference)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This method is in my lecture notes, so I'm not really sure how to reference it (there are no references given in the notes) apart from what I've just said.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just another note, I won't harp on it, but I p-values are not all they are cracked up to be.  A good post discussing this can be found &lt;a href=&quot;http://stats.stackexchange.com/questions/6308/article-about-misuse-of-statistical-method-in-nytimes&quot;&gt;here&lt;/a&gt;.  I like Harlod Jeffrey's quote above p-values (from his book &lt;em&gt;probability theory&lt;/em&gt;) &quot;A null hypothesis may be rejected because it did not predict something that was not observed&quot; (this is because p-values ask for the probability of events &lt;em&gt;more extreme&lt;/em&gt; than what was observed).&lt;/p&gt;&#xA;" OwnerUserId="2392" LastEditorUserId="2392" LastEditorDisplayName="" LastEditDate="2011-01-21T17:19:22.187" LastActivityDate="2011-01-21T17:19:22.187" CommentCount="9" />
  <row Id="6403" PostTypeId="2" ParentId="6401" CreationDate="2011-01-20T03:23:58.857" Score="3" ViewCount="" Body="&lt;p&gt;Well, that &lt;em&gt;is&lt;/em&gt; the basis for ROC curves. You see what proportion of correct predictions (i.e. yes or no) are at a variety of predictor levels. The analog of an ROC curve for continuous outcomes would be a validation plot. You develop a prediction score on a training set and validate it on a test set. Or you develop it on the full set and then use bootstrap methods to create neo-samples for validation.&lt;/p&gt;&#xA;" OwnerUserId="2129" LastActivityDate="2011-01-20T03:23:58.857" />
  <row Id="6404" PostTypeId="1" AcceptedAnswerId="6411" CreationDate="2011-01-20T03:49:06.957" Score="8" ViewCount="207" Body="&lt;p&gt;Anyone that follows baseball has likely heard about the out-of-nowhere MVP-type performance of Toronto's Jose Bautista. In the four years previous, he hit roughly 15 home runs per season. Last year he hit 54, a number surpassed by only 12 players in baseball history. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In 2010 he was paid 2.4 million and he's asking the team for 10.5 million for 2011. They're offering 7.6 million. If he can repeat that in 2011, he'll be easily worth either amount. But what are the odds of him repeating? How hard can we expect him to regress to the mean? How much of his performance can we expect was due to chance? What can we expect his regression-to-the-mean adjusted 2010 totals to be? How do I work it out?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've been playing around with the Lahman Baseball Database and squeezed out a query that returns home run totals for all players in the previous five seasons who've had at least 50 at-bats per season.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The table looks like this (notice Jose Bautista in row 10)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;     first     last hr_2006 hr_2007 hr_2008 hr_2009 hr_2010&#xA;1    Bobby    Abreu      15      16      20      15      20&#xA;2   Garret Anderson      17      16      15      13       2&#xA;3  Bronson   Arroyo       2       1       1       0       1&#xA;4  Garrett   Atkins      29      25      21       9       1&#xA;5     Brad   Ausmus       2       3       3       1       0&#xA;6     Jeff    Baker       5       4      12       4       4&#xA;7      Rod  Barajas      11       4      11      19      17&#xA;8     Josh     Bard       9       5       1       6       3&#xA;9    Jason Bartlett       2       5       1      14       4&#xA;10    Jose Bautista      16      15      15      13      54&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and the full result (232 rows) is available &lt;a href=&quot;http://datalove.org/files/regress_hr.csv&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I really don't know where to start. Can anyone point me in the right direction? Some relevant theory, and R commands would be especially helpful. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks kindly&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tommy&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: The example is a little contrived. Home runs definitely aren't the best indicator of a player's worth, and home run totals don't consider the varying number of chances per season that a batter has the chance to hit home runs (plate appearances). Nor does it reflect that some players play in more favourable stadiums, and that league average home runs change year over year. Etc. Etc. If I can grasp the theory behind accounting for regression to the mean, I can use it on more suitable measures than HRs.&lt;/p&gt;&#xA;" OwnerUserId="827" LastActivityDate="2011-01-21T15:32:30.207" Title="Measuring Regression to the Mean in Hitting Home Runs" Tags="&lt;r&gt;&lt;regression&gt;&lt;modeling&gt;" AnswerCount="4" CommentCount="4" FavoriteCount="3" />
  <row Id="6405" PostTypeId="2" ParentId="5734" CreationDate="2011-01-20T06:28:47.500" Score="1" ViewCount="" Body="&lt;p&gt;Gretl is a very good and standard open source time series library.&lt;/p&gt;&#xA;" OwnerUserId="2845" LastActivityDate="2011-01-20T06:28:47.500" />
  <row Id="6406" PostTypeId="2" ParentId="6404" CreationDate="2011-01-20T09:36:25.980" Score="2" ViewCount="" Body="&lt;p&gt;You need additional data on the players and their characteristics in the time-span you have data about home-runs. For the first step add some time-varying characteristics such as players age or experience. Then you can use HLM or panel data models. You will need to prepare data in the form: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    First Last  Year HR Experience Age&#xA;1.  Bobby Abreu 2005 15     6      26&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The most simple model would then be (the function &lt;strong&gt;lme&lt;/strong&gt; is from package &lt;strong&gt;nlme&lt;/strong&gt;)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;lme(HR~Experience,random=~Experience|Year,data=your_data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This model will heavily rely on assumption that each player's home-run number relies only on experience allowing some variability. It probably will not be very accurate, but you at least will get a feel how unlikely are Jose Bautista's numbers compared to average player. This model can be further improved by adding other player's characteristics.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-01-20T09:36:25.980" CommentCount="4" />
  <row Id="6407" PostTypeId="2" ParentId="6391" CreationDate="2011-01-20T10:01:57.660" Score="2" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Model and data source&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As in any other applications you should start from the plausible model, that would fit most of the countries in your cross-sectional dimension. Suppose you do have a quite universal theory that fits most of the countries in the world. Then you could try to include as many countries as possible using either &lt;a href=&quot;http://data.worldbank.org/&quot; rel=&quot;nofollow&quot;&gt;World Bank&lt;/a&gt;, &lt;a href=&quot;http://www.imfstatistics.org/imf/logon.aspx&quot; rel=&quot;nofollow&quot;&gt;IFS &lt;/a&gt; statistics or simply &lt;a href=&quot;http://pwt.econ.upenn.edu/php_site/pwt_index.php&quot; rel=&quot;nofollow&quot;&gt;Penn World table&lt;/a&gt; data. However you may also limit yourself to more homogeneous samples like EU data taken from &lt;a href=&quot;http://epp.eurostat.ec.europa.eu/portal/page/portal/eurostat/home/&quot; rel=&quot;nofollow&quot;&gt;Eurostat&lt;/a&gt; or &lt;a href=&quot;http://www.oecd.org/document/39/0,3746,en_2649_201185_46462759_1_1_1_1,00.html&quot; rel=&quot;nofollow&quot;&gt;OECD&lt;/a&gt; statistics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Cross-section of countries&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since in panel data models (you may also consider &lt;a href=&quot;http://cran.r-project.org/doc/contrib/Fox-Companion/appendix-mixed-models.pdf&quot; rel=&quot;nofollow&quot;&gt;linear mixed effects&lt;/a&gt; models as an option, since you also include interaction terms) under you (mostly) quantitative hypothesis &quot;was there a significant (both statistically and economically) impact of belonging to euro-zone club on the budget deficit?&quot; you have to balance the number of countries that are in the euro-zone, adopted it from a certain year, and are out of the club. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Should I only use countries that have introduced the Euro?&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Is adding other European countries or western like countries beneficial as they add more data points and/or acts like a reference group?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Regarding these particular questions, in my opinion, the inclusion of EU countries and some OECD rivals to France, Germany and Italy, would be sufficient for your analysis. You have not to limit only to euro-zone club, because of selection bias. Well some countries are adopted euro within the time dimension, but it is better to compare with the countries that are not so restricting their fiscal policy due to strict Maastricht criteria they obliged to follow.  &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Or is id only bad for the data to add countries that has not introduced the euro as they have no variation in the variable of interest (EURODUMMY)?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Dummy is a variable that has two values $0$ and $1$, so if you include only (not in this case though) $1$ you will face pure multicollinearity problem with the common intercept term, that is present by default in &lt;code&gt;plm&lt;/code&gt; model.&lt;/p&gt;&#xA;" OwnerUserId="2645" LastActivityDate="2011-01-20T10:01:57.660" CommentCount="2" />
  <row Id="6408" PostTypeId="2" ParentId="6354" CreationDate="2011-01-20T13:29:45.717" Score="0" ViewCount="" Body="&lt;p&gt;Just from reading the link, it seems to me that there is probably some packages that do what you want in the machine learning view on CRAN &lt;a href=&quot;http://cran.r-project.org/web/views/MachineLearning.html&quot; rel=&quot;nofollow&quot;&gt;http://cran.r-project.org/web/views/MachineLearning.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="656" LastActivityDate="2011-01-20T13:29:45.717" />
  <row Id="6409" PostTypeId="2" ParentId="1475" CreationDate="2011-01-20T14:06:55.660" Score="2" ViewCount="" Body="&lt;p&gt;(Months later,)&#xA;a nice way to picture k-clusters and to see the effect of various k&#xA;is to build a&#xA;&lt;a href=&quot;http://en.wikipedia.org/wiki/Minimum_spanning_tree&quot; rel=&quot;nofollow&quot;&gt;Minimum Spanning Tree&lt;/a&gt;&#xA;and look at the longest edges. For example,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/03Bv3.png&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here there are 10 clusters, with 9 longest edges&#xA;855 899 942 954 1003 1005 1069 1134 1267.&lt;br&gt;&#xA;For 9 clusters, collapse the cyan 855 edge; for 8, the purple 899;&#xA;and so on.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The single-link k-clustering algorithm ... is precisely Kruskal's algorithm&#xA;  ... equivalent to finding an MST and deleting the k-1 most expensive edges.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&amp;mdash; Wayne, &#xA;&lt;a href=&quot;http://www.cs.princeton.edu/~wayne/kleinberg-tardos/04greedy-2x2.pdf&quot; rel=&quot;nofollow&quot;&gt;Greedy Algorithms&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;22000 points, 242M pairwise distances, take ~ 1 gigabyte (float32):&#xA;might fit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To view a high-dimensional tree or graph in 2d,&#xA;see Multidimensional Scaling (also from Kruskal),&#xA;and the huge literature on dimension reduction.&#xA;However, in dim &gt; 20 say, most distances will be near the median,&#xA;so I believe dimension reduction &lt;em&gt;cannot&lt;/em&gt; work there.&lt;/p&gt;&#xA;" OwnerUserId="557" LastEditorUserId="557" LastEditorDisplayName="" LastEditDate="2011-01-20T15:30:01.123" LastActivityDate="2011-01-20T15:30:01.123" />
  <row Id="6410" PostTypeId="1" CreationDate="2011-01-20T15:40:31.200" Score="6" ViewCount="143" Body="&lt;p&gt;Hi all,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've a question which probably is going to show my ignorance about statistics :). I have a large set of machines that produce iron bars of certain lengths. For each machine, I have ran experiments and have a list of lengths. From those I can calculate a mean and sample standard deviation. I don't really care about their means and I am mainly focused on the variation. Therefore, I basically only record a sample standard deviation per machine. I think the results of each machine follow a normal distribution. So far so good :)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I now want to combine these variations into a single number. Therefore, I calculate the quadratic average of each machine variation, let's call it X. In the next step, I also would like to give an estimate for the spread around X. What is this number called and what's the best way to calculate it?  I'm not sure it's related to the confidence interval of a standard deviation and I don't know whether the measurements are independent (a design fault would show up in all, a construction maybe only in some).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Example.&lt;/strong&gt; I'll try to clarify with an example. Suppose I measure 3 machines and find that they produce lengths of&lt;br&gt;&#xA;M1: 100 +/- 7&lt;br&gt;&#xA;M2: 120 +/- 8&lt;br&gt;&#xA;M3: 130 +/- 9&lt;br&gt;&#xA;where the numbers behind the +/-'s are the sample standard deviations of observed values on that single machine. As said before, I don't care about the means but only in the spread, so I define {X_1, X_2, X_3} = {7,8,9}. Their quadratic average is X = RMS(X_i) = $\sqrt{194}$ and I think of X as an indication of the average spread of a machine in my park.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose that I would have found {X_1, X_2, X_3} = {3,8,11}. Their quadratic average is the same $\sqrt{194}$, but the spread around it is obviously bigger. My confidence in the correctness of $\sqrt{194}$ as the average spread of a machine should therefore be lower (I'd like to test some more machines, for instance) and I would like to express this in a number.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Questions&lt;/strong&gt;&#xA;Some answer to questions: they aren't identical; if some machine really misbehaves I could see it directly from the machine test (i.e. I would see it from a large X_i), but I wouldn't detect a small misbehavior. Furthermore, the sample amount for each machine could be different (I have more tests on my old machine wrt my new machines).&lt;/p&gt;&#xA;" OwnerUserId="2848" LastEditorUserId="2848" LastEditorDisplayName="" LastEditDate="2011-01-22T07:16:28.483" LastActivityDate="2011-01-22T07:16:28.483" Title="Reliability of mean of standard deviations" Tags="&lt;standard-deviation&gt;&lt;reliability&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="6411" PostTypeId="2" ParentId="6404" CreationDate="2011-01-20T15:55:04.220" Score="3" ViewCount="" Body="&lt;p&gt;I think that there's definitely a Bayesian shrinkage or prior correction that could help prediction but you might want to also consider another tack...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Look up players in history, not just the last few years, who've had breakout seasons after a couple in the majors (dramatic increases perhaps 2x) and see how they did in the following year.  It's possible the probability of maintaining performance there is the right predictor.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There's a variety of ways to look at this problem but as mpiktas said, you're going to need more data.  If you just want to deal with recent data then you're going to have to look at overall league stats, the pitchers he's up against, it's a complex problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And then there's just considering Bautista's own data.  Yes, that was his best year but it was also the first time since 2007 he had over 350 ABs (569).  You might want to consider converting the percentage increase in performance.&lt;/p&gt;&#xA;" OwnerUserId="601" LastActivityDate="2011-01-20T15:55:04.220" />
  <row Id="6412" PostTypeId="1" CreationDate="2011-01-20T15:59:21.423" Score="3" ViewCount="229" Body="&lt;p&gt;What is your favorite free tool on Linux for multivariate logistic regression?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Possibilities I've seen:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.r-project.org&quot; rel=&quot;nofollow&quot;&gt;R&lt;/a&gt; (see &lt;a href=&quot;http://nlp.stanford.edu/~manning/courses/ling289/logistic.pdf&quot; rel=&quot;nofollow&quot;&gt;paper&lt;/a&gt;).  &lt;a href=&quot;http://stackoverflow.com/questions/3439248/logistic-regression-in-r-sas-like-output&quot;&gt;This question&lt;/a&gt; says use &lt;a href=&quot;http://cran.r-project.org/package=Design&quot; rel=&quot;nofollow&quot;&gt;design&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;li&gt;Can you use &lt;a href=&quot;http://docs.scipy.org/doc/scipy/reference/stats.html#statistical-functions&quot; rel=&quot;nofollow&quot;&gt;SciPy&lt;/a&gt;?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Other choices?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do people have experience with large data?&lt;/p&gt;&#xA;" OwnerUserId="2849" LastActivityDate="2011-01-20T21:55:22.140" CommunityOwnedDate="2011-01-20T17:13:38.950" Title="Free tools on Linux for multivariate logistic regression?" Tags="&lt;r&gt;&lt;logistic-regression&gt;&lt;python&gt;" AnswerCount="3" CommentCount="4" />
  <row Id="6413" PostTypeId="2" ParentId="6412" CreationDate="2011-01-20T16:41:35.723" Score="0" ViewCount="" Body="&lt;p&gt;I'm not sure, but i think PSPP does logistic regression. Not entirely certain if it can handle multivariate logistic regression.&lt;/p&gt;&#xA;" OwnerUserId="656" LastActivityDate="2011-01-20T16:41:35.723" CommunityOwnedDate="2011-01-20T17:13:38.950" />
  <row Id="6414" PostTypeId="2" ParentId="1812" CreationDate="2011-01-20T17:27:34.687" Score="3" ViewCount="" Body="&lt;p&gt;I find myself routinely using parallel analysis (O'Connor, 2000). This solves the problem of how many factors to extract nicely.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;See: &lt;a href=&quot;https://people.ok.ubc.ca/brioconn/nfactors/nfactors.html&quot; rel=&quot;nofollow&quot;&gt;https://people.ok.ubc.ca/brioconn/nfactors/nfactors.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;O'Connor, B. P. (2000). SPSS and SAS programs for determining the number of components using parallel analysis and Velicer's MAP test. Behavior Research Methods, Instrumentation, and Computers, 32, 396-402.&lt;/p&gt;&#xA;" OwnerUserId="2851" LastActivityDate="2011-01-20T17:27:34.687" CommentCount="1" />
  <row Id="6415" PostTypeId="2" ParentId="6412" CreationDate="2011-01-20T17:50:00.817" Score="4" ViewCount="" Body="&lt;p&gt;If you want to use Python, you can use the &lt;a href=&quot;http://scikit-learn.sourceforge.net/&quot; rel=&quot;nofollow&quot;&gt;scikit-learn&lt;/a&gt;, which relies on Scipy. See the documentation for an example of the logistic regression:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.sourceforge.net/auto_examples/glm/plot_logistic_path.html&quot; rel=&quot;nofollow&quot;&gt;http://scikit-learn.sourceforge.net/auto_examples/glm/plot_logistic_path.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This implementation is based on liblinear, thus it scales reasonably well. In addition, it implements L1 and L2 penalization, for sparse or shrunk regression, when dealing with high-dimensional data.&lt;/p&gt;&#xA;" OwnerUserId="1265" LastActivityDate="2011-01-20T17:50:00.817" CommunityOwnedDate="2011-01-20T17:50:00.817" CommentCount="2" />
  <row Id="6416" PostTypeId="2" ParentId="6410" CreationDate="2011-01-20T20:18:45.753" Score="1" ViewCount="" Body="&lt;p&gt;If you want to test whether the variances of several machines deviates from the other variances combining them into average will not help you. The problem is that these differing variances will skew your average. To test whether there are different variances you can use &lt;a href=&quot;http://en.wikipedia.org/wiki/Bartlett%27s_test&quot; rel=&quot;nofollow&quot;&gt;Bartlet's test&lt;/a&gt;. It is sensitive to normality, but since you said that your data is normal this should not be a problem, though it would be a good idea to test that. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now if you can assume that all the machines are similar in sense that they can have different means but similar variance, the problem is very simple. If you assume that machines are independent treat the variances from each machine as a random sample. Then estimate the mean and standard deviation of this sample. For large number of machines, the normal approximation will kick in, so it will not matter whether you use standard deviations or the variances. In both cases the sample mean will estimate average statistic of your choice, and standard deviation of the sample will estimate average spread of statistic of your choice. The 95% confidence interval will then be $\mu\pm 1.96\sigma$. &lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-01-20T20:18:45.753" CommentCount="2" />
  <row Id="6417" PostTypeId="2" ParentId="3519" CreationDate="2011-01-20T21:11:26.137" Score="1" ViewCount="" Body="&lt;p&gt;In my opinion, Matlab is an ugly language. Perhaps it's gotten default arguments and named arguments in its core by now, but many examples you find online do the old &quot;If there are 6 arguments, this, else if there are 5 arguments this and that...&quot; and named arguments are just vectors with alternating strings (names) and values. That's so 1970's that I simply can't use it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;R may have its issues, and it is also old, but it was built on a foundation (Scheme/Lisp) that was forward-looking and has held up rather well in comparison.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That said, Matlab is much faster if you like to code with loops, etc. And it has much better debugging facilities. And more interactive graphics. On the other hand, what passes for documenting your code/libraries is rather laughable compared to R and you pay a pretty penny to use Matlab.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All IMO.&lt;/p&gt;&#xA;" OwnerUserId="1764" LastActivityDate="2011-01-20T21:11:26.137" />
  <row Id="6418" PostTypeId="1" CreationDate="2011-01-20T21:52:48.840" Score="4" ViewCount="108" Body="&lt;p&gt;I need an idea for a new rating system.. the problem in the ordinary one (just average votes) is that it does not count how many votes... &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example consider these 2 cases:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3 people voted 5/5&lt;/p&gt;&#xA;&#xA;&lt;p&gt;500 people voted 4/5&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The ordinary voting systems just take the average, leading to the first one to be better..&#xA;However, I want the second one to get a higher rating, because many people have voted for the second...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help? &lt;/p&gt;&#xA;" OwnerUserId="2599" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-01-22T14:04:39.037" LastActivityDate="2011-01-22T14:13:48.103" Title="Rating system taking account of number of votes" Tags="&lt;rating&gt;" AnswerCount="4" CommentCount="2" FavoriteCount="3" />
  <row Id="6419" PostTypeId="2" ParentId="6412" CreationDate="2011-01-20T21:55:22.140" Score="1" ViewCount="" Body="&lt;p&gt;R is a full statistical computing environment, featuring a programming language especially designed and optimized to this purpose and an enormous an high quality library tightly covering the whole are of data science.&#xA;SciPy is just a BLAS for Python with some support to basic statistics. &lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2011-01-20T21:55:22.140" CommunityOwnedDate="2011-01-20T21:55:22.140" CommentCount="1" />
  <row Id="6420" PostTypeId="2" ParentId="6418" CreationDate="2011-01-20T22:00:51.767" Score="1" ViewCount="" Body="&lt;p&gt;You could use a system like &lt;a href=&quot;http://blog.reddit.com/2009/10/reddits-new-comment-sorting-system.html&quot; rel=&quot;nofollow&quot;&gt;reddit's &quot;best&quot; algorithm&lt;/a&gt; for sorting comments:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;This algorithm treats the vote count&#xA;  as a statistical sampling of a&#xA;  hypothetical full vote by everyone,&#xA;  much as in an opinion poll. It uses&#xA;  this to calculate the 95% confidence&#xA;  score for the comment. That is, it&#xA;  gives the comment a provisional&#xA;  ranking that it is 95% sure it will&#xA;  get to. The more votes, the closer the&#xA;  95% confidence score gets to the&#xA;  actual score&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So in the case of 3 people voting 5/5, you might be 95% sure the &quot;actual&quot; rating is at least a 1, whereas in the case of 500 people voting you might be 95% sure the &quot;actual&quot; rating is at least a 4/5.&lt;/p&gt;&#xA;" OwnerUserId="2817" LastActivityDate="2011-01-20T22:00:51.767" />
  <row Id="6421" PostTypeId="1" CreationDate="2011-01-21T02:53:54.717" Score="12" ViewCount="529" Body="&lt;p&gt;I still remember the Annals of Statistics paper on Boosting by Friedman-Hastie-Tibshirani, and the comments on that same issues by other authors (including Freund and Schapire). At that time, clearly Boosting was viewed as a breakthrough in many respects: computationally feasible, an ensemble method, with excellent yet mysterious performance. Around the same time, SVM came of age, offering a &lt;em&gt;framework&lt;/em&gt; underpinned by solid theory and with plenty of variants and applications. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;That was in the marvelous 90s. In the past 15 years, it seems to me that a lot of Statistics has been a cleaning and detailing operation, but with few truly new views.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I'll ask two questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Have I missed some revolutionary/seminal paper?&lt;/li&gt;&#xA;&lt;li&gt;If not, are there new approaches&#xA;that you think have the potential to&#xA;change the viewpoint of statistical&#xA;inference?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Rules: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;One answer per post; &lt;/li&gt;&#xA;&lt;li&gt;References or links welcome.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;P.S.: I have a couple of candidates for promising breakthroughs. I will post them later.&lt;/p&gt;&#xA;" OwnerUserId="30" LastActivityDate="2011-01-30T04:04:07.940" CommunityOwnedDate="2011-01-21T03:10:15.437" Title="What are the breakthroughs in Statistics of the past 15 years?" Tags="&lt;mathematical-statistics&gt;&lt;breakthroughs&gt;" AnswerCount="8" CommentCount="5" FavoriteCount="5" />
  <row Id="6422" PostTypeId="2" ParentId="6418" CreationDate="2011-01-21T03:18:02.573" Score="1" ViewCount="" Body="&lt;p&gt;You may run into problems implied by the &lt;a href=&quot;http://en.wikipedia.org/wiki/Gibbard-Satterthwaite_theorem&quot; rel=&quot;nofollow&quot;&gt;Gibbard Satterthwaite&lt;/a&gt; Theorem or &lt;a href=&quot;http://en.wikipedia.org/wiki/Arrow%27s_impossibility_theorem&quot; rel=&quot;nofollow&quot;&gt;Arrow's Impossibility Theorem&lt;/a&gt;, or any of the results of voting theory...&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2011-01-21T03:18:02.573" />
  <row Id="6423" PostTypeId="2" ParentId="6418" CreationDate="2011-01-21T03:48:51.013" Score="5" ViewCount="" Body="&lt;p&gt;You are talking about a &lt;a href=&quot;http://en.wikipedia.org/wiki/Shrinkage_estimator&quot; rel=&quot;nofollow&quot;&gt;shrinkage estimator&lt;/a&gt;. Imdb is possibly the most famous example of this, how they calculate which movies will make it onto the &lt;a href=&quot;http://www.imdb.com/chart/top&quot; rel=&quot;nofollow&quot;&gt;top 250&lt;/a&gt;. It relies on the equation, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;weighted rating (WR) = (v ÷ (v+m)) × R + (m ÷ (v+m)) × C , where:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;* R = average for the movie (mean) = (Rating)&#xA;* v = number of votes for the movie = (votes)&#xA;* m = minimum votes required to be listed in the Top 250 (currently 3000)&#xA;* C = the mean vote across the whole report (currently 6.9)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;They call this a &quot;true bayesian rating&quot; and that's true in the sense that our prior for the parameter &quot;average rating&quot; is that it is the same as for all other movies. This prior is then updated based on the &quot;likelihood,&quot; which is the average rating for that movie, which has more strength if it has more votes. But I'm not sure whether this technically qualifies as bayesian, because neither the prior nor the posterior is a distribution... Can anyone clarify this?&lt;/p&gt;&#xA;" OwnerUserId="2073" LastActivityDate="2011-01-21T03:48:51.013" />
  <row Id="6424" PostTypeId="2" ParentId="6421" CreationDate="2011-01-21T08:38:17.833" Score="6" ViewCount="" Body="&lt;p&gt;I'm not sure if you would call it a &quot;breakthrough&quot; per se, But the Publishing of &lt;em&gt;Probability Theory: The Logic of Science&lt;/em&gt; By Edwin Jaynes and Larry Bretthorst may be noteworthy.  Some of the things they do here are:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) show equivalence between some iterative &quot;seasonal adjustment&quot; schemes and Bayesian &quot;nuisance parameter&quot; integration.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) resolved the so called &quot;Marginalisation Paradox&quot; - thought to be the &quot;death of bayesianism&quot; by some, and the &quot;death of improper priors&quot; by others.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) the idea that probability &lt;em&gt;describes a state of knowledge&lt;/em&gt; about a proposition being true or false, as opposed to describing a &lt;em&gt;physical property of the world&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first three chapters of this book are available for free &lt;a href=&quot;http://bayes.wustl.edu/etj/prob&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-01-21T08:38:17.833" CommunityOwnedDate="2011-01-21T08:38:17.833" />
  <row Id="6425" PostTypeId="2" ParentId="6421" CreationDate="2011-01-21T09:42:03.843" Score="7" ViewCount="" Body="&lt;p&gt;As an applied statistician and occasional minor software author, i'd say:  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/WinBUGS&quot; rel=&quot;nofollow&quot;&gt;WinBUGS&lt;/a&gt;&lt;/strong&gt; (released 1997)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's based on BUGS, which was released more than 15 years ago (1989), but it's WinBUGS that made Bayesian analysis of realistically complex models available to a far wider user base. See e.g. &lt;a href=&quot;http://dx.doi.org/10.1002/sim.3680&quot; rel=&quot;nofollow&quot;&gt;Lunn, Spiegelhalter, Thomas &amp;amp; Best (2009)&lt;/a&gt; (and the discussion on it in &lt;a href=&quot;http://onlinelibrary.wiley.com/doi/10.1002/sim.v28:25/issuetoc&quot; rel=&quot;nofollow&quot;&gt;Statistics in Medicine vol. 28 issue 25&lt;/a&gt;).&lt;/p&gt;&#xA;" OwnerUserId="449" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-01-21T13:44:29.173" LastActivityDate="2011-01-21T13:44:29.173" CommunityOwnedDate="2011-01-21T09:42:03.843" />
  <row Id="6426" PostTypeId="2" ParentId="6421" CreationDate="2011-01-21T09:55:15.793" Score="1" ViewCount="" Body="&lt;p&gt;The Expectation-Propagation algorithm for Bayesian inference, especially in Gaussian Process classification, was arguably a significant breakthrough, as it provides an efficient analytic approximation method that works almost as well as computationally expensive sampling based approaches (unlike the usual Laplace approximation).  See the work of Thomas Minka and others on the &lt;a href=&quot;http://research.microsoft.com/en-us/um/people/minka/papers/ep/roadmap.html&quot; rel=&quot;nofollow&quot;&gt;EP roadmap&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="887" LastEditorUserId="887" LastEditorDisplayName="" LastEditDate="2011-01-21T14:03:33.117" LastActivityDate="2011-01-21T14:03:33.117" CommunityOwnedDate="2011-01-21T09:55:15.793" CommentCount="1" />
  <row Id="6427" PostTypeId="2" ParentId="6321" CreationDate="2011-01-21T10:05:07.917" Score="1" ViewCount="" Body="&lt;p&gt;As an alternative, &lt;a href=&quot;http://www2.warwick.ac.uk/fac/sci/statistics/crism/research/2005/05-15/05-15wv1.pdf&quot; rel=&quot;nofollow&quot;&gt;this beautiful paper&lt;/a&gt; (a bit technical/mathsy, so beware) allows you to test for &lt;em&gt;any&lt;/em&gt; correlation (not just $\rho=0$) using a Non-informative Bayesian Decision Theoretic approach.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-01-21T10:05:07.917" />
  <row Id="6428" PostTypeId="2" ParentId="6404" CreationDate="2011-01-21T10:44:27.050" Score="3" ViewCount="" Body="&lt;p&gt;You &lt;em&gt;can&lt;/em&gt; fit a model to this data alone and get predictions that account for regression to the mean by using mixed (multilevel) models. Predictions from such models account for regression to the mean. Even without knowing next to nothing about baseball I don't find results I got terribly believable, since, as you say, the model really needs to take account of other factors, such as plate appearances.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think a Poisson mixed-effects model would be more suitable than a linear mixed model as the number of home runs is a count. Looking at the &lt;a href=&quot;http://datalove.org/files/regress_hr.csv&quot; rel=&quot;nofollow&quot;&gt;data you provided&lt;/a&gt;, a histogram of &lt;code&gt;hr&lt;/code&gt; shows it is strongly positively skewed, suggesting  so a linear mixed model isn't going to work well, and includes a fairly large number of zeroes, with or without log-transforming hr first.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's some code using the &lt;code&gt;lmer&lt;/code&gt; function from the &lt;a href=&quot;http://cran.r-project.org/package=lme4&quot; rel=&quot;nofollow&quot;&gt;lme4&lt;/a&gt; package. Having created an ID variable to identify each player and reshaped the data to 'long' format as mpiktas indicated in his answer, (i did that in Stata as i'm no good at data management in R, but you could do it in a spreadsheet package):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Year.c &amp;lt;- Year - 2008   # centering y eases computation and interpretation&#xA;(M1 &amp;lt;- lmer(HR ~ Year.c + (Year.c|ID), data=baseball.long, family=poisson(log), nAGQ=5))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This fits a model with a log-link giving an exponential dependence of hit-rate on year, which is allowed to vary between players. Other link functions are possible, though the identity link gave an error due to negative fitted values. A sqrt link worked ok though, and has lower BIC and AIC than the model with the log link, so it may be a better fit. The predictions for hit-rate in 2011 are sensitive to the chosen link function, particularly for players such as Bautista whose hit-rate has changed a lot recently. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm afraid I haven't managed to actually get such predictions out of &lt;code&gt;lme4&lt;/code&gt; though. I'm more familiar with Stata, which makes it very easy to get predictions for observations with missing values for the outcome, although &lt;a href=&quot;http://www.stata.com/help.cgi?xtmelogit&quot; rel=&quot;nofollow&quot;&gt;xtmelogit&lt;/a&gt; doesn't appear to offer any choice of link function other than log, which gave a prediction of 50 for Bautista's home runs in 2011. As I said, I don't find that terribly believable. I'd be grateful someone could show how to generate predictions for 2011 from the above &lt;code&gt;lmer&lt;/code&gt; models above.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An &lt;a href=&quot;http://en.wikipedia.org/wiki/AR%281%29&quot; rel=&quot;nofollow&quot;&gt;autoregressive model&lt;/a&gt; such as AR(1) for the player-level errors might be interesting too, but I don't know how to combine such a structure with a Poisson mixed model.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-01-21T10:44:27.050" CommentCount="3" />
  <row Id="6430" PostTypeId="2" ParentId="1812" CreationDate="2011-01-21T11:18:22.393" Score="3" ViewCount="" Body="&lt;p&gt;I would have to second chl's suggestion of the psych package, its extremely useful and has implementations of the MAP and parallel analysis criteria for number of factors. In my own experience, i have found that if you create factor analysis solutions for all the numbers between those returned by MAP and parallel analysis, you normally can find a relatively optimum solution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would also second the use of OpenMx for confirmatory factor analysis, as it seems to give the best results of all of them, and is much, much better for badly behaved matrices (as mine tend to be). The syntax is also quite nice, once you get used to it. The only issue that i have with it is that the optimiser is not open source, and thus it is not available on CRAN. Apparently they are working on an open source implementation of the optimiser, so that may not be an issue for much longer. &lt;/p&gt;&#xA;" OwnerUserId="656" LastActivityDate="2011-01-21T11:18:22.393" CommentCount="1" />
  <row Id="6431" PostTypeId="2" ParentId="2356" CreationDate="2011-01-21T11:21:13.240" Score="5" ViewCount="" Body="&lt;p&gt;I said earlier that I would have a go at answering the question, so here goes...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Jaynes was being a little naughty in his paper in that a frequentist confidence interval isn't defined as an interval where we might expect the true value of the statistic to lie with high (specified) probability, so it isn't unduly surprising that contradictions arise if they are interpreted as if they were.  The problem is that this is often the way confidence intervals are used in practice, as an interval highly likely to contain the true value (given what we can infer from our sample of data) is what we often want.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The key issue for me is that when a question is posed, it is best to have a direct answer to that question.  Whether Bayesian credible intervals are worse than frequentist confidence intervals depends on what question was actually asked.  If the question asked was:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(a) &quot;Give me an interval where the true value of the statistic lies with probability p&quot;, then it appears a frequentist cannot actually answer that question directly (and this introduces the kind of problems that Jaynes discusses in his paper), but a Bayesian can, which is why a Bayesian credible interval is superior to the frequentist confidence interval in the examples given by Jaynes.  But this is only becuase it is the &quot;wrong question&quot; for the frequentist.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(b) &quot;Give me an interval where, were the experiment repeated a large number of times, the true value of the statistic would lie within p*100% of such intervals&quot; then the frequentist answer is just what you want.  The Bayesian may also be able to give a direct answer to this question (although it may not simply be the obvious credible interval).  Whuber's comment on the question suggests this is the case.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So essentially, it is a matter of correctly specifying the question and properly intepreting the answer.  If you want to ask question (a) then use a Bayesian credible interval, if you want to ask question (b) then use a frequentist confidence interval.&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2011-01-21T11:21:13.240" CommentCount="3" />
  <row Id="6432" PostTypeId="2" ParentId="6421" CreationDate="2011-01-21T13:40:53.907" Score="2" ViewCount="" Body="&lt;p&gt;The introduction of the &quot;intrinsic discrepancy&quot; loss function and other &quot;parameterisation free&quot; loss functions into decision theory.  It has many other &quot;nice&quot; properties, but I think the best one is as follows: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;if the best estimate of $\theta$ using the intrinsic discrepancy loss function is $\theta^{e}$, then the best estimate of any one-to-one function of $\theta$, say $g(\theta)$ is simply $g(\theta^{e})$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think this is very cool! (e.g. best estimate of log-odds is log(p/(1-p)), best estimate of variance is square of standard deviation, etc. etc.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The catch? the intrinsic discrepancy can be quite difficult to work out! (it involves min() funcion, a likelihood ratio, and integrals!)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &quot;counter-catch&quot;? you can &quot;re-arrange&quot; the problem so that it is easier to calculate!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &quot;counter-counter-catch&quot;? figuring out how to &quot;re-arrange&quot; the problem can be difficult!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are some references I know of which use this loss function.  While I very much like the &quot;intrinsic estimation&quot; parts of these papers/slides, I have some reservations about the &quot;reference prior&quot; approach that is also described.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.stat.duke.edu/research/conferences/valencia/IntStatRev.pdf&quot; rel=&quot;nofollow&quot;&gt;Bayesian Hypothesis Testing:A Reference Approach&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.uv.es/~bernardo/2005Test.pdf&quot; rel=&quot;nofollow&quot;&gt;Intrinsic Estimation&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.uv.es/~bernardo/2007BayesAnalysis.pdf&quot; rel=&quot;nofollow&quot;&gt;Comparing Normal Means: New Methods for an Old Problem&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.uv.es/~bernardo/JMBSlidesV9.pdf&quot; rel=&quot;nofollow&quot;&gt;Integrated Objective Bayesian Estimation and Hypothesis Testing&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-01-21T13:40:53.907" CommunityOwnedDate="2011-01-21T13:40:53.907" />
  <row Id="6433" PostTypeId="1" CreationDate="2011-01-21T15:15:09.900" Score="1" ViewCount="127" Body="&lt;p&gt;I have samples of Bernoulli distributed variable that are neither the first nor the second i in iid. My goal is to model their sum. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I got from Wikipedia that I can use the poisson binomial distribution to make up for one of the i's, but then I have to keep all the inidividual probabilities.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It would probably also be possible to throw the central limit theorem against it somehow to model it as a Gaussian, but I wonder if I can do better.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any results on how well a binomial distribution fits the sum of non identically non independently distributed Bernoulli samples. Especially if I can get some bounds on the accuracy wrt the correlation of the samples or something like that.&lt;/p&gt;&#xA;" OwnerUserId="2860" LastActivityDate="2011-01-21T15:15:09.900" Title="Approximate a poisson binomial distribution with a binomial distribution?" Tags="&lt;binomial&gt;" CommentCount="2" />
  <row Id="6434" PostTypeId="2" ParentId="6404" CreationDate="2011-01-21T15:32:30.207" Score="1" ViewCount="" Body="&lt;p&gt;You might want to check out &lt;a href=&quot;http://www.insidethebook.com/ee/&quot; rel=&quot;nofollow&quot;&gt;The Book Blog.&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tom Tango and the other authors of &quot;The Book: Playing the Percentages in Baseball&quot; are probably the best sources of sabermetrics out there.  In particular, they love regression to the mean.  They came up with a forecasting system designed to be the most basic acceptable system (Marcel), and it relies almost exclusively on regression to the mean.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Off the top of my head, I suppose one method would be to use such a forecast to estimate true talent, and then find an appropriate distribution around that mean talent.  Once you have that, each plate appearance will be like a Bernoulli trial, so the binomial distribution could take you the rest of the way.&lt;/p&gt;&#xA;" OwnerUserId="2485" LastActivityDate="2011-01-21T15:32:30.207" />
  <row Id="6435" PostTypeId="1" AcceptedAnswerId="6436" CreationDate="2011-01-21T15:37:09.363" Score="5" ViewCount="156" Body="&lt;p&gt;I have a function like this dummy-one:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;FUN &amp;lt;- function(x, parameter){&#xA;  if (parameter == 1){&#xA;      z &amp;lt;- DO SOMETHING WITH &quot;x&quot;}&#xA;  if (parameter ==2){&#xA;      z &amp;lt;- DO OTHER STUFF WITH &quot;x&quot;}&#xA;return(z)&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, I would like to use the function on a dataset using apply.&#xA;The problem is, that &lt;code&gt;apply(data,1,FUN(parameter=1))&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;wont work, as FUN doesn't know what &quot;x&quot; is.&#xA;Is there a way to tell apply to call FUN with &quot;x&quot; as the current row/col?&#xA;`&lt;/p&gt;&#xA;" OwnerUserId="2838" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-01-21T18:23:46.240" LastActivityDate="2011-01-21T18:23:46.240" Title="Passing arguments to iterated function through apply" Tags="&lt;r&gt;" AnswerCount="2" CommentCount="0" FavoriteCount="1" />
  <row Id="6436" PostTypeId="2" ParentId="6435" CreationDate="2011-01-21T15:55:53.000" Score="7" ViewCount="" Body="&lt;p&gt;You want &lt;code&gt;apply(data,1,FUN,parameter=1)&lt;/code&gt;.  Note the &lt;code&gt;...&lt;/code&gt; in the function definition:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; args(apply)&#xA;function (X, MARGIN, FUN, ...) &#xA;NULL&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and the corresponding entry in the documentation:  &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;...: optional arguments to ‘FUN’.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="1657" LastActivityDate="2011-01-21T15:55:53.000" />
  <row Id="6437" PostTypeId="2" ParentId="6435" CreationDate="2011-01-21T15:57:41.810" Score="5" ViewCount="" Body="&lt;p&gt;You can make an anonymous function within the call to &lt;code&gt;apply&lt;/code&gt; so that FUN will know what &quot;x&quot; is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;apply(data, 1, function(x) FUN(x, parameter = 1))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;See &lt;code&gt;?apply&lt;/code&gt; for examples at the bottom that use this method.&lt;/p&gt;&#xA;" OwnerUserId="696" LastActivityDate="2011-01-21T15:57:41.810" CommentCount="5" />
  <row Id="6438" PostTypeId="1" AcceptedAnswerId="6439" CreationDate="2011-01-21T17:57:41.657" Score="4" ViewCount="53" Body="&lt;p&gt;Suppose you wanted to estimate the $q$ quantile of a distribution by observing $n$ independent draws from that distribution, but with $q &amp;lt; \frac{1}{n}$. What methods are available, and for what ranges of the 'undersampling coefficient' $q n$ are they expected to work?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can imagine some parametric approaches: a fully parametric approach would assume the form of the distribution, then fit &lt;em&gt;e.g.&lt;/em&gt; the scale and location, and estimate the quantile from the fitted distribution.  A slightly less parametric form would try to estimate the form (&lt;em&gt;e.g.&lt;/em&gt; fit to a power law) of the left tail, and use that to extrapolate. Are either of these sensible? Do they work? What are some other approaches?&lt;/p&gt;&#xA;" OwnerUserId="795" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-01-21T18:41:43.440" LastActivityDate="2011-01-21T18:41:43.440" Title="Quantile extrapolation?" Tags="&lt;estimation&gt;&lt;quantiles&gt;&lt;parametric&gt;&lt;extreme-value&gt;&lt;extrapolation&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="6439" PostTypeId="2" ParentId="6438" CreationDate="2011-01-21T18:30:10.430" Score="1" ViewCount="" Body="&lt;p&gt;If you are also interested in quantiles with $q&amp;gt;1-1/n$ there is no definitive answer. You need to supply more details, since for distributions with heavy tails the estimation of such quantiles involves quite complicated mathematics. Try google search for tail index estimation and you will get plethora of links.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-01-21T18:30:10.430" CommentCount="1" />
  <row Id="6440" PostTypeId="2" ParentId="6354" CreationDate="2011-01-21T18:46:58.413" Score="2" ViewCount="" Body="&lt;p&gt;I've just found out that the package &lt;a href=&quot;http://keii.ae.jgora.pl/symbolicDA/index.html&quot; rel=&quot;nofollow&quot;&gt;symbolicDA&lt;/a&gt; is about to appear on CRAN in a few months' time.&lt;/p&gt;&#xA;" OwnerUserId="2831" LastActivityDate="2011-01-21T18:46:58.413" CommentCount="1" />
  <row Id="6441" PostTypeId="1" CreationDate="2011-01-21T20:07:12.527" Score="1" ViewCount="107" Body="&lt;p&gt;Hello,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a question regarding regression analysis on a dataset were the input values generate different results over time:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;e.g.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1 2&#xA;2 2&#xA;3 5&#xA;4 1&#xA;2 5&#xA;3 8&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How would I go about doing the regression on such a dataset, since the values change?&lt;/p&gt;&#xA;" OwnerUserId="2865" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-01-21T20:43:16.063" LastActivityDate="2011-01-24T14:11:27.283" Title="Regression with repeated data" Tags="&lt;regression&gt;&lt;repeated-measures&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="6442" PostTypeId="1" AcceptedAnswerId="6443" CreationDate="2011-01-21T20:10:12.493" Score="3" ViewCount="150" Body="&lt;pre&gt;&lt;code&gt;AIC(lm(Fertility ~ ., data=swiss))&#xA;[1] 326.0716&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;ok, since AIC is calculated as&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;-2*logLik(lm(Fertility ~ ., data=swiss)) + 2*7&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Why does stepAIC return a smaller AIC?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;stepAIC(lm(Fertility ~ ., data=swiss))&#xA;Start:  AIC=190.69&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="339" LastActivityDate="2011-01-21T20:58:46.413" Title="Question on AIC and stepAIC" Tags="&lt;r&gt;&lt;regression&gt;&lt;aic&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="6443" PostTypeId="2" ParentId="6442" CreationDate="2011-01-21T20:58:46.413" Score="5" ViewCount="" Body="&lt;p&gt;Read &lt;code&gt;?AIC&lt;/code&gt;. From the Details section we have:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; The log-likelihood and hence the AIC is only defined up to an&#xA; additive constant.  Different constants have conventionally be&#xA; used for different purposes and so ‘extractAIC’ and ‘AIC’ may give&#xA; different values (and do for models of class ‘&quot;lm&quot;’: see the help&#xA; for ‘extractAIC’).&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So you are seeing the effect of different additive constants, as &lt;code&gt;stepAIC()&lt;/code&gt; (in the MASS package) is using &lt;code&gt;extractAIC()&lt;/code&gt; to compute the AIC of the models.&lt;/p&gt;&#xA;" OwnerUserId="1390" LastActivityDate="2011-01-21T20:58:46.413" />
  <row Id="6444" PostTypeId="1" AcceptedAnswerId="6446" CreationDate="2011-01-21T21:14:02.597" Score="5" ViewCount="194" Body="&lt;p&gt;Thus can we have random variables $X_1 , ... , X_n$ , each with zero mean and unit variance, such that for a sizeable representative sample:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(1) in the least-squares regression equation for $X_1$ against $X_2 ,..., X_n$ , the variable $X_n$ has a large coefficient, and its contribution is rated highly significant, while the other variables have markedly smaller coefficients and lower significance;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(2) in the least-squares regression for $X_n$ against $X_1 ,..., X_{n-1}$ , the variable $X_1$ has a small coefficient and low significance?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reason for this question is that this effect appears in some real data (using SPSS), with five variables. But the result seems paradoxical; so perhaps we are doing something wrong.&lt;/p&gt;&#xA;" OwnerUserId="2866" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-01-22T15:50:09.937" LastActivityDate="2011-02-03T11:10:49.133" Title="Can statistical prediction be asymmetric?" Tags="&lt;regression&gt;&lt;spss&gt;" AnswerCount="2" CommentCount="3" />
  <row Id="6445" PostTypeId="2" ParentId="6441" CreationDate="2011-01-21T21:40:08.813" Score="2" ViewCount="" Body="&lt;p&gt;What do you mean by &quot;the input values generate different results over time?&quot;  This happens a lot in regression analysis, and isn't typically a problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could start by &lt;a href=&quot;http://www.r-project.org/&quot; rel=&quot;nofollow&quot;&gt;loading your data into R&lt;/a&gt;, and running a simple linear model.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x&amp;lt;-c(1,2,3,4,2,3)&#xA;y&amp;lt;-c(2,2,5,1,5,8)&#xA;&#xA;model&amp;lt;-lm(y~x)&#xA;summary(model)&#xA;&#xA;plot(x,y)&#xA;lines(x,fitted(model))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In the case of your example data, a simple linear model is terrible.  Can you be more specific about what you're trying to do?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;/edit: In response to suncoolso: once you've fit a simple linear model, you can use the &quot;gls&quot; command from the &quot;nlme&quot; package to fit a simple linear model with an autoregressive correlation structure.&lt;/p&gt;&#xA;" OwnerUserId="2817" LastEditorUserId="2817" LastEditorDisplayName="" LastEditDate="2011-01-24T14:11:27.283" LastActivityDate="2011-01-24T14:11:27.283" CommentCount="1" />
  <row Id="6446" PostTypeId="2" ParentId="6444" CreationDate="2011-01-22T00:44:35.250" Score="7" ViewCount="" Body="&lt;p&gt;The coefficient of $X_n$ (and its significance) in the regression of $X_1$ on $X_2, \ldots, X_n$ can be computed by first obtaining the residuals $Y_1$ for the regression of $X_1$ on $X_2, \ldots, X_{n-1}$ and obtaining the residuals $Y_n$ for the regression of $X_n$ on $X_2, \ldots, X_{n-1}$.  Then you regress $Y_1$ on $Y_n$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Similarly, the coefficient of $X_1$ in the regression of $X_n$ on $X_1, \ldots, X_{n-1}$ is computed by regressing $Y_n$ on $Y_1$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This reduces the question to one of ordinary regression (of one variable against another).  The coefficients are not related to one another in a simple manner, because the variance of $Y_1$ does not have to equal the variance of $Y_n$, but the t-statistics will be identical.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems the software has not done what you expected.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-01-22T00:44:35.250" CommentCount="2" />
  <row Id="6447" PostTypeId="2" ParentId="1966" CreationDate="2011-01-22T00:50:41.847" Score="1" ViewCount="" Body="&lt;p&gt;A nice short paper by Jose Bernardo &lt;a href=&quot;http://www.uv.es/~bernardo/Kernel.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; gives a useful Bayesian method to estimate a density.  But as with most things Bayesian, the computational cost must be paid for this method.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-01-22T00:50:41.847" CommunityOwnedDate="2011-01-22T00:50:41.847" />
  <row Id="6448" PostTypeId="1" AcceptedAnswerId="6450" CreationDate="2011-01-22T03:43:13.820" Score="5" ViewCount="201" Body="&lt;p&gt;I think this is a fairly stupid question, but I just realize that I don't quite understand the term continuity correction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I use R and found the same syntax &lt;code&gt;correct=TRUE&lt;/code&gt; for both &lt;code&gt;chisq.test&lt;/code&gt; and &lt;code&gt;mcnemar.test&lt;/code&gt;. Are they referring to different continuity correction method? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have heard of the yate's continuity for pearson chi-square test is not very popular now as it may &quot;over-adjust&quot; the result, then how about that for McNemar's chi-square test?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="588" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-01-22T11:15:19.630" LastActivityDate="2011-01-23T00:15:16.413" Title="Continuity correction for Pearson and McNemar's chi-square test" Tags="&lt;r&gt;&lt;chi-squared&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="6449" PostTypeId="2" ParentId="6444" CreationDate="2011-01-22T04:44:22.490" Score="7" ViewCount="" Body="&lt;p&gt;Your are trying to estimate model&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align}&#xA;X_1&amp;amp;=\alpha_0+\alpha_2X_2+...+\alpha_nX_n+\varepsilon,\\&#xA;X_n&amp;amp;=\beta_0+\beta_1X_1+...+\beta_{n-1}X_{n-1}+\eta.&#xA;\end{align}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For such model ordinary least squares will give biased estimates. Assuming that $X_2,...,X_{n-1}$ are either deterministic or independent from $\varepsilon$ and $\eta$ we have &lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align}&#xA;EX_n\varepsilon&amp;amp;=E\left(\beta_0+\beta_1X_1+...+\beta_{n-1}X_{n-1}+\eta\right)\varepsilon\\&#xA;&amp;amp;=\beta_1EX_1\varepsilon+E\varepsilon\eta=\\&#xA;&amp;amp;=\beta_1E(\alpha_0+\alpha_2X_2+...+\alpha_nX_n+\varepsilon)\varepsilon+E\varepsilon\eta\\&#xA;&amp;amp;=\alpha_n\beta_1EX_n\varepsilon+\beta_1E\varepsilon^2 +E\varepsilon\eta.&#xA;\end{align}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So&#xA;\begin{align}&#xA;EX_n\varepsilon=\frac{\beta_1E\varepsilon^2+E\varepsilon\eta}{1-\alpha_n\beta_1}&#xA;\end{align}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The main assumption of linear regression is that the error is not correlated to the regressors. Without this assumption the estimates are biased and inconsistent. As we see in this case the assumption is violated, so it is entirely natural to expect unexpected results. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The are ways to get statistically sound estimates of the coefficients. This is an extremely common problem in econometrics called &lt;a href=&quot;http://en.wikipedia.org/wiki/Endogeneity_%28economics%29&quot; rel=&quot;nofollow&quot;&gt;endogeneity&lt;/a&gt;. Most popular solution is &lt;a href=&quot;http://en.wikipedia.org/wiki/Two-stage_least_squares&quot; rel=&quot;nofollow&quot;&gt;two-stage least squares.&lt;/a&gt; In general these type of models are called &lt;a href=&quot;http://en.wikipedia.org/wiki/Simultaneous_equations_model&quot; rel=&quot;nofollow&quot;&gt;simultaneous equations&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the comments below the attention was drawn to the fact that the OP is just trying to fit two regressions and the model given above might be inappropriate. @onestop provided excellent reference about data modelling vs algorithmic modelling cultures. Nonetheless there is an important point I want to make in this case. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Taking into account @whuber answer we can restrict ourselves to the case where we have only two variables, $Y$ and $X$. Suppose having the sample $(y_i,x_i)$ we fit two linear regressions $Y$ vs $X$ and $X$ vs $Y$. The least squares estimates are the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align}&#xA;\beta_{YX}&amp;amp;=\frac{\sum_{i=1}^nx_iy_i}{\sum_{i=1}x_i^2} \\&#xA;\beta_{XY}&amp;amp;=\frac{\sum_{i=1}^nx_iy_i}{\sum_{i=1}y_i^2}&#xA;\end{align}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now from the formulas it is clear that $\beta_{XY}$ and $\beta_{YX}$ coincide only in the case when $\sum_{i=1}^ny_i^2=\sum_{i=1}^nx_i^2$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The corresponding $t$-statistics are then&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align}&#xA;t_{YX}&amp;amp;=\frac{\beta_{YX}}{\sigma_{\beta_{XY}}}, \quad \sigma_{\beta_{YX}}^2=\frac{ \sigma_{YX}^2}{\sum_{i=1}^nx_i^2}, \quad \sigma_{YX}^2=\frac{1}{n-1}\sum_{i=1}^n(y_i-\beta_{YX}x_i)^2 \\&#xA;t_{XY}&amp;amp;=\frac{\beta_{XY}}{\sigma_{XY}}, \quad \quad \sigma_{\beta_{XY}}^2=\frac{ \sigma_{XY}^2}{\sum_{i=1}^nx_i^2}, \quad  \sigma_{XY}^2=\frac{1}{n-1}\sum_{i=1}^n(y_i-\beta_{XY}x_i)^2&#xA;\end{align}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After a bit of manipulation we get the amazing fact (in my opinion) that&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align}&#xA;t_{XY}=t_{YX}=\frac{\sum_{i=1}^nx_iy_i}{\sqrt{\frac{1}{n-1}\left(\sum_{i=1}^ny_i^2\sum_{i=1}^nx_i^2-\left(\sum_{i=1}^nx_iy_i\right)^2\right)}}&#xA;\end{align}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So this illustrates that OP clearly has a problem, since the $t$-statistics must coincide in this case. This of course was pointed out by @whuber. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far so good. The problem arises when we want to get the distribution of this statistic. This is needed since $t$-statistic formally test the hypothesis that the coefficient $\beta_{XY}$ is zero. Without assuming any model, suppose that we want to test the null hypothesis that $cov(X,Y)=0$. We can also assume that we have $EX=EY=0$. Rewrite the $t$ statistic as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align}&#xA;t=\frac{\frac{1}{\sqrt{n-1}}\sum_{i=1}^nx_iy_i}{\sqrt{\frac{1}{n-1}\sum_{i=1}^ny_i^2\frac{1}{n-1}\sum_{i=1}^nx_i^2-\left(\frac{1}{n-1}\sum_{i=1}^nx_iy_i\right)^2}}&#xA;\end{align}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now since we have a sample due to law of large numbers we have&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align*}&#xA;\frac{1}{n-1}\sum_{i=1}^ny_i^2\frac{1}{n-1}\sum_{i=1}^nx_i^2\xrightarrow{P}var(X)var(Y)\\&#xA;\frac{1}{n-1}\sum_{i=1}^nx_iy_i\xrightarrow{P}cov(X,Y),&#xA;\end{align*}&#xA;where $\xrightarrow{P}$ denotes convergence in probability.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So denominator of $t$-statistic converges to $\sqrt{var(X)var(Y)}$ under null hypothesis of $cov(X,Y)=0$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Under null hypothesis due to central limit theorem we have that&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align}&#xA;\frac{1}{\sqrt{n-1}}\sum_{i=1}^nx_iy_i\xrightarrow{D}N(0,var(XY)),&#xA;\end{align} &#xA;where $\xrightarrow{D}$ denotes convergence in distribution. So we get that under null-hypothesis of no correlation the $t$-statistic converges to&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align}&#xA;t\xrightarrow{D}N\left(0,\frac{var(XY)}{var(X)var(Y)}\right)&#xA;\end{align} &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now if $(X,Y)$ are bivariate normal we have that $\frac{var(XY)}{var(X)var(Y)}=1$, under null hypothesis of $cov(X,Y)=0$, since zero correlation implies independence for normal random variables. The quantity $\frac{var(XY)}{var(X)var(Y)}$ is one, when $X$ is deterministic as is usually the case in linear regression, but then the null hypothesis $cov(X,Y)=0$ makes no sense. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now if we assume that we have a model $Y=X\beta+\varepsilon$, under null hypothesis that $\beta=0$ and usual assumption $E(\varepsilon^2|X)=\sigma^2$ we get that&#xA;\begin{align}&#xA;t\xrightarrow{D}N(0,1),&#xA;\end{align}&#xA;since &#xA;\begin{align}&#xA;\frac{var(\varepsilon X)}{var(X)var(\varepsilon)}=\frac{\sigma^2var(X)}{\sigma^2var(X)}=1&#xA;\end{align}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But in the OP question we have &lt;strong&gt;two&lt;/strong&gt; regressions, so if we allow model for one, we &lt;strong&gt;must allow&lt;/strong&gt; model for the other and we arrive at the problem which I described in my initial answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I hope that my lengthy update illustrates that if we do not assume usual model in doing regression, we cannot assume that usual statistics will have the same distributions. &lt;/p&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-02-03T11:10:49.133" LastActivityDate="2011-02-03T11:10:49.133" CommentCount="7" />
  <row Id="6450" PostTypeId="2" ParentId="6448" CreationDate="2011-01-22T10:20:15.813" Score="7" ViewCount="" Body="&lt;p&gt;Another reason that continuity corrections for contingency tables have gone out of fashion is that they only make a noticeable difference when the cell counts are small, and modern computing power has made it feasible to calculate 'exact' p-values for such tables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Exact tests for 2×2 tables for &lt;strong&gt;R&lt;/strong&gt; are provided by the &lt;a href=&quot;http://cran.r-project.org/web/packages/exact2x2/&quot; rel=&quot;nofollow&quot;&gt;exact2x2&lt;/a&gt; package written by &lt;a href=&quot;http://www.niaid.nih.gov/about/organization/dcr/BRB/staff/Pages/michael.aspx&quot; rel=&quot;nofollow&quot;&gt;Michael Fay&lt;/a&gt;. There's an accompanying &lt;a href=&quot;http://cran.r-project.org/web/packages/exact2x2/vignettes/exactMcNemar.pdf&quot; rel=&quot;nofollow&quot;&gt;vignette about the exact McNemar's test and matching confidence intervals&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="449" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-01-23T00:15:16.413" LastActivityDate="2011-01-23T00:15:16.413" CommentCount="3" />
  <row Id="6451" PostTypeId="2" ParentId="6418" CreationDate="2011-01-22T14:13:48.103" Score="1" ViewCount="" Body="&lt;p&gt;There is a simple (also to implement) heuristic to first seed the pool of votes with small number of dummy votes with average voting, and later replace it with incoming votes.&lt;br&gt;&#xA;So, for instance, new object appears and you give it few votes rating it 2.5/5 (this is the best you can tell about it at the zero-knowledge point). Then the first vote comes, let's say 5/5, but it is somewhat tempered by the rest of the initial pool and the objects mean is slightly above 2.5. Than next votes come and the mean gradually moves from the initial guess  to the real average which than has time to stabilize. Finally this algorithm converges to normal vote mean.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2011-01-22T14:13:48.103" />
  <row Id="6453" PostTypeId="1" AcceptedAnswerId="6456" CreationDate="2011-01-22T17:18:05.033" Score="4" ViewCount="145" Body="&lt;p&gt;I have run experiments on a group of users under two conditions, measuring the time it took users to finish their experiments. I used a cross-over design where half of the users started in the first conditions to end with the second, and the other half of the users did the other way around. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I analyze the data provided in a few different ANOVAs and find different p-values for my hypotheses. Some are below 0.05, some are below 0.01, some are over 0.05.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do I need to fix an alpha level of statistical significance to be used in all my analysis, or can I report something like 'Hypothesis A is proven true at alpha level 0.05, while Hypothesis B is true at alpha level 0.01 (thus, possibly a stronger proof)'?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't know if I am being clear enough here. Let me know and I'll add details if needed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="1320" LastEditorUserId="1320" LastEditorDisplayName="" LastEditDate="2011-01-22T19:35:29.030" LastActivityDate="2011-01-22T19:35:29.030" Title="How to fix the threshold for statistical validity of p-values produced by ANOVAs?" Tags="&lt;hypothesis-testing&gt;&lt;anova&gt;&lt;mixed-model&gt;&lt;statistical-significance&gt;&lt;p-value&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="6454" PostTypeId="1" CreationDate="2011-01-22T17:36:57.363" Score="2" ViewCount="144" Body="&lt;p&gt;Is it possible to do meta analysis of only two studies. What will be limitation of such analysis. &lt;/p&gt;&#xA;" OwnerUserId="2874" LastActivityDate="2011-01-22T23:56:52.097" Title="Is it possible to do meta analysis of only two studies" Tags="&lt;meta-analysis&gt;" AnswerCount="2" />
  <row Id="6455" PostTypeId="1" CreationDate="2011-01-22T18:02:39.427" Score="8" ViewCount="534" Body="&lt;p&gt;After an ARMA model is fit to a time series, it is common to check the residuals via the Ljung-Box portmanteau test (among other tests).  The Ljung-Box test returns a p value.  It has a parameter, &lt;em&gt;h&lt;/em&gt;, which is the number of lags to be tested.  Some texts recommend using &lt;em&gt;h&lt;/em&gt;=20; others recommend using &lt;em&gt;h&lt;/em&gt;=ln(n); most do not say what &lt;em&gt;h&lt;/em&gt; to use.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rather than using a single value for &lt;em&gt;h&lt;/em&gt;, suppose that I do the Ljung-Box test for all &lt;em&gt;h&lt;/em&gt;&amp;lt;50, and then pick the &lt;em&gt;h&lt;/em&gt; which gives the minimum p value.  Is that approach reasonable?  What are the advantages and disadvantages?  (One obvious disadvantage is increased computation time, but that is not a problem here.)  Is there literature on this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To elaborate slightly....  If the test gives p&gt;0.05 for all &lt;em&gt;h&lt;/em&gt;, then obviously the time series (residuals) pass the test.  My question concerns how to interpret the test if p&amp;lt;0.05 for some values of &lt;em&gt;h&lt;/em&gt; and not for other values.&lt;/p&gt;&#xA;" OwnerUserId="2875" LastEditorUserId="2875" LastEditorDisplayName="" LastEditDate="2011-01-23T19:37:13.097" LastActivityDate="2011-03-02T23:55:45.330" Title="How many lags to use in the Ljung-Box test of a time series?" Tags="&lt;time-series&gt;" AnswerCount="3" CommentCount="4" FavoriteCount="2" />
  <row Id="6456" PostTypeId="2" ParentId="6453" CreationDate="2011-01-22T18:09:44.463" Score="6" ViewCount="" Body="&lt;p&gt;Hey, but it seems you already looked at the results!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Usually, the risk of falsely rejecting the null (Type I error, or $\alpha$) should be decided before starting the analysis. Power might also be fixed to a given value (e.g., 0.80). At least, this is the &quot;Neyman-Pearson&quot; approach. For example, you might consider a risk of 5% ($\alpha=0.05$) for all your hypotheses, and if the tests are not independent you should consider correcting for multiple comparisons, using any single-step or step-down methods you like. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When reporting your results, you should indicate the Type I (and II, if applicable) error you considered (before seeing the results!), corrected or not for multiple comparisons, and give your p-values as p&amp;lt;.001 or p=.0047 for example. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, I would say that your tests allow you to reject a given null hypothesis not to prove Hypothesis A or B. Moreover, what you describe as 0.001 being a somewhat stronger indication of an interesting deviation from the null than 0.05 is more in light with the Fisher approach to &lt;a href=&quot;http://en.wikipedia.org/wiki/Statistical_hypothesis_testing&quot; rel=&quot;nofollow&quot;&gt;statistical hypothesis testing&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-01-22T18:09:44.463" />
  <row Id="6457" PostTypeId="2" ParentId="6453" CreationDate="2011-01-22T18:30:38.843" Score="5" ViewCount="" Body="&lt;p&gt;My advice would be to tread carefully with p-values if you didn't have a specific hypothesis in mind before you started the experiment.  Adjusting p-values for multiple and &quot;vaguely specified&quot; hypothesis (e.g. not specifying the alternative hypothesis) is difficult.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suppose the &quot;purist&quot; would tell you that this should be fixed &lt;em&gt;prior&lt;/em&gt; to looking at the data (one of my lecturers call not doing this &lt;em&gt;intellectual dishonesty&lt;/em&gt;), but I would only say this is appropriate for &quot;confirmatory analysis&quot; where a well defined model (or set of models) has been set prior to the data being seen.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the analysis is more &quot;exploratory&quot; then I would not worry about precise level so much, rather try to find relationships and try to explain why they may be there (i.e. use the analysis to build a model).  tentative hypothesis testing may be useful as an initial guide, but you would need to get more data to confirm your hypothesis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A useful way to &quot;get more data&quot; without running another experiment is to &quot;lock up&quot; some portion of your data and use the rest to &quot;explore&quot; and then once you are confident of a potentially useful model, &quot;test&quot; your theory with the data you &quot;locked up&quot;.  NOTE: you can on do the &quot;test&quot; once!&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-01-22T18:30:38.843" CommentCount="5" />
  <row Id="6458" PostTypeId="2" ParentId="6454" CreationDate="2011-01-22T19:45:21.480" Score="6" ViewCount="" Body="&lt;p&gt;Yes, it is &lt;a href=&quot;http://stats.stackexchange.com/q/3652/1381&quot;&gt;possible&lt;/a&gt;, but whether it is appropriate depends on the intent of your analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Meta-analysis is a method of combining information from different sources, so it is technically possible to do a meta-analysis of only two studies - even of multiple results within a single paper. The key concern is not &lt;em&gt;if&lt;/em&gt; you can do this, but that the method is appropriate for the questions that you have and the conclusions that you want to make, and that you acknowledgE the limitations of your analysis. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, the typical use of meta-analysis is to quantitatively synthesize previous studies on a particular subject, such as the effects of some medical intervention. In this context, it is important to make your criteria for study selection before the analysis and then find all studies available that meet those criteria. These criteria might limit the scope of your search to publications in English, in a particular journal or set of journals, those that use particular methods, etc. In practice, it is necessary to be familiar with the studies you are interested in to state these criteria. However, if you non-randomly select two papers from among many that have been published, it would introduce bias into your study. If only two studies have been published, it might be hard to justify any conclusions from a meta-analysis but it could still be done.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, I have used the meta-analytical approach to synthesize data from a single study, for example if summary statistics are reported for subgroups but I am interested in finding the overall mean and variance. I don't always call this a meta-analysis in mixed company, so as not to confuse this application of the method with the more common use of meta-analysis as a comprehensive review &lt;em&gt;sensu stricto&lt;/em&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1381" LastEditorUserId="1381" LastEditorDisplayName="" LastEditDate="2011-01-22T22:42:40.807" LastActivityDate="2011-01-22T22:42:40.807" CommentCount="2" />
  <row Id="6459" PostTypeId="2" ParentId="6455" CreationDate="2011-01-22T20:35:47.630" Score="1" ViewCount="" Body="&lt;p&gt;Before you zero-in on the &quot;right&quot; h (which appears to be more of an opinion than a hard rule), make sure the &quot;lag&quot; is correctly defined.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.stat.pitt.edu/stoffer/tsa2/Rissues.htm&quot; rel=&quot;nofollow&quot;&gt;http://www.stat.pitt.edu/stoffer/tsa2/Rissues.htm&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Quoting the section below Issue 4 in the above link:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;....The p-values shown for the Ljung-Box statistic plot are incorrect because the degrees of freedom used to calculate the p-values are lag instead of lag - (p+q). That is, the procedure being used does NOT take into account the fact that the residuals are from a fitted model. And YES, at least one R core developer knows this....&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit (01/23/2011): Here's an article by Burns that might help:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://lib.stat.cmu.edu/S/Spoetry/Working/ljungbox.pdf&quot; rel=&quot;nofollow&quot;&gt;http://lib.stat.cmu.edu/S/Spoetry/Working/ljungbox.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2775" LastEditorUserId="2775" LastEditorDisplayName="" LastEditDate="2011-01-23T17:48:49.720" LastActivityDate="2011-01-23T17:48:49.720" CommentCount="3" />
  <row Id="6461" PostTypeId="2" ParentId="6454" CreationDate="2011-01-22T23:56:52.097" Score="0" ViewCount="" Body="&lt;p&gt;If you compute a &lt;a href=&quot;http://pbr.psychonomic-journals.org/content/11/5/791.full.pdf&quot; rel=&quot;nofollow&quot;&gt;likelihood ratio&lt;/a&gt; for the effect of interest in each study, you can simply multiply them together to obtain the aggregate weight of evidence for the effect.&lt;/p&gt;&#xA;" OwnerUserId="364" LastActivityDate="2011-01-22T23:56:52.097" CommentCount="5" />
  <row Id="6462" PostTypeId="1" CreationDate="2011-01-23T04:07:55.287" Score="3" ViewCount="32" Body="&lt;p&gt;Hi, I have tested the &lt;a href=&quot;http://en.wikipedia.org/wiki/Cluster_Analysis#Fuzzy_c-means_clustering&quot; rel=&quot;nofollow&quot;&gt;fuzzy C-means (FCM)&lt;/a&gt; algorithm using the R function &lt;code&gt;fanny&lt;/code&gt; from the &lt;code&gt;cluster&lt;/code&gt; package and I have wrote my own FCM algorithm to have more control on the distance function. The problem is that my final membership function converges many times to the 1/K, I mean almost all the values in the membership matrix are near to 1/K.&#xA;I am wondering what are the conditions that influence the final result. I am using  fuzzy parameter m=2.&lt;/p&gt;&#xA;" OwnerUserId="2325" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-01-23T14:32:40.043" LastActivityDate="2011-01-23T14:32:40.043" Title="In which case does FCM membership converge to 1/K?" Tags="&lt;clustering&gt;" CommentCount="3" />
  <row Id="6463" PostTypeId="1" AcceptedAnswerId="6465" CreationDate="2011-01-23T07:41:38.150" Score="4" ViewCount="149" Body="&lt;p&gt;What software does one use to create a map that describes conditions over an arbitrary area ? The example I am thinking of would be a map of the US where states that voted one way or another would be red or blue. Obviously I can get a map of the US and use a graphics program to manually paint in the states, but I am figuring there should be some sort of software to do that in a somewhat automated fashion. Apart from a for-pay web app, I haven't been able to find anything. I am very new to the field so I'm not familiar with the available tools.&lt;/p&gt;&#xA;" OwnerUserId="2878" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-01-23T10:03:27.403" LastActivityDate="2011-01-24T19:35:16.437" Title="What software is used for maps of the US (or other arbitrary areas)?" Tags="&lt;data-visualization&gt;" AnswerCount="4" CommentCount="3" FavoriteCount="1" />
  <row Id="6464" PostTypeId="2" ParentId="6463" CreationDate="2011-01-23T08:13:06.243" Score="3" ViewCount="" Body="&lt;p&gt;You should try R software with package &lt;a href=&quot;http://cran.r-project.org/web/packages/spdep/index.html&quot; rel=&quot;nofollow&quot;&gt;spdep&lt;/a&gt;. see the vignette &lt;a href=&quot;http://cran.r-project.org/web/packages/spdep/vignettes/sids.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="223" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-01-23T09:26:32.053" LastActivityDate="2011-01-23T09:26:32.053" />
  <row Id="6465" PostTypeId="2" ParentId="6463" CreationDate="2011-01-23T09:26:00.510" Score="12" ViewCount="" Body="&lt;p&gt;In addition to the package that @robin pointed too, you should look at the &lt;a href=&quot;http://cran.r-project.org/web/views/Spatial.html&quot; rel=&quot;nofollow&quot;&gt;Spatial&lt;/a&gt; Task View on CRAN. What you are describing is known as a &lt;a href=&quot;http://en.wikipedia.org/wiki/Choropleth_map&quot; rel=&quot;nofollow&quot;&gt;choropleth map&lt;/a&gt;, as illustrated here: &lt;a href=&quot;http://www.r-bloggers.com/choropleth-maps-of-presidential-voting/&quot; rel=&quot;nofollow&quot;&gt;Choropleth Maps of Presidential Voting&lt;/a&gt;, or &lt;a href=&quot;http://www.r-bloggers.com/u-s-unemployment-data-animated-choropleth-maps/&quot; rel=&quot;nofollow&quot;&gt;U.S. Unemployment Data: Animated Choropleth Maps&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In R, they can be handled using &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;the base graphics routines (with &lt;a href=&quot;http://cran.r-project.org/web/packages/maps/index.html&quot; rel=&quot;nofollow&quot;&gt;maps&lt;/a&gt;),&lt;/li&gt;&#xA;&lt;li&gt;the lattice way (see &lt;code&gt;mapplot()&lt;/code&gt; in &lt;a href=&quot;http://cran.r-project.org/web/packages/latticeExtra/index.html&quot; rel=&quot;nofollow&quot;&gt;latticeExtra&lt;/a&gt;),&lt;/li&gt;&#xA;&lt;li&gt;the ggplot2 way (see e.g., &lt;a href=&quot;http://blog.revolutionanalytics.com/2009/11/choropleth-challenge-result.html&quot; rel=&quot;nofollow&quot;&gt;Choropleth Challenge Results&lt;/a&gt; for example code).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Other softwares that allow to deal with geographical maps include &lt;a href=&quot;http://rosuda.org/mondrian/Mondrian.html&quot; rel=&quot;nofollow&quot;&gt;Mondrian&lt;/a&gt;, &lt;a href=&quot;http://www.qgis.org/&quot; rel=&quot;nofollow&quot;&gt;Quantum GIS&lt;/a&gt; (and I guess many other GIS programs), but data format may vary from one software to the other.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-01-23T09:26:00.510" CommentCount="1" />
  <row Id="6466" PostTypeId="2" ParentId="577" CreationDate="2011-01-23T14:11:09.733" Score="1" ViewCount="" Body="&lt;p&gt;AIC should rarely be used, as it is really only valid asymptotically.  It is almost always better to use AICc (AIC with a &lt;b&gt;c&lt;/b&gt;orrection for finite sample size).  AIC tends to overparameterize: that problem is greatly lessened with AICc.  The main exception to using AICc is when the underlying distributions are heavily leptokurtic.  For more on this, see the book &lt;em&gt;Model Selection&lt;/em&gt; by Burnham &amp;amp; Anderson.&lt;/p&gt;&#xA;" OwnerUserId="2875" LastActivityDate="2011-01-23T14:11:09.733" CommentCount="1" />
  <row Id="6467" PostTypeId="2" ParentId="6463" CreationDate="2011-01-23T14:31:32.623" Score="2" ViewCount="" Body="&lt;p&gt;I've not tried &lt;a href=&quot;http://www.crack4mac.com/&quot; rel=&quot;nofollow&quot;&gt;CrackMaps&lt;/a&gt; but it looks very interesting. Only available for the Mac, I presume.&lt;/p&gt;&#xA;" OwnerUserId="2617" LastActivityDate="2011-01-23T14:31:32.623" CommentCount="1" />
  <row Id="6468" PostTypeId="1" CreationDate="2011-01-23T15:16:47.363" Score="3" ViewCount="138" Body="&lt;p&gt;This is a question that landed on my desk, and I don't have the requisite experience to reply.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A researcher has been asked to perform (by a journal reviewer) a two-way factorial ANCOVA on some microrarray data (a single array will be a single measurement of ~30,000 entities from a single sample in this case).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The suggestion was to use ethnicity and age as co-variables.  I understand that ANCOVA requires at least one categorical and one continuous predictor variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The researcher has 3 ethnic groupings in his samples - 6 of group A, 2 of group B and 1 of group C.  These samples are spread across three treatments (X, Y and Z), with 3 samples per treatment.  This means the treatments cannot be split evenly by ethnicity, group C will only appear in &lt;em&gt;one&lt;/em&gt; of the treatments.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this sample size/grouping issue a barrier to running a two-way factorial ANCOVA?  Is there another test that might be more appropriate under the circumstances?&lt;/p&gt;&#xA;" OwnerUserId="133" LastActivityDate="2011-01-23T17:48:31.583" Title="Requirements to perform a two-way factorial ANCOVA" Tags="&lt;ancova&gt;" AnswerCount="1" />
  <row Id="6469" PostTypeId="1" AcceptedAnswerId="6470" CreationDate="2011-01-23T15:36:47.640" Score="4" ViewCount="295" Body="&lt;p&gt;How do I fit a linear model with autocorrelated errors in R? In stata I would use the &lt;code&gt;prais&lt;/code&gt; command, but I can't find an R equivalent...&lt;/p&gt;&#xA;" OwnerUserId="2817" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-01-23T17:35:50.843" LastActivityDate="2011-01-24T04:48:03.363" Title="Simple linear model with autocorrelated errors in R" Tags="&lt;r&gt;&lt;time-series&gt;&lt;autocorrelation&gt;" AnswerCount="3" FavoriteCount="2" />
  <row Id="6470" PostTypeId="2" ParentId="6469" CreationDate="2011-01-23T15:52:41.160" Score="5" ViewCount="" Body="&lt;p&gt;Have a look at &lt;code&gt;gls&lt;/code&gt; (generalized least squares) from the package &lt;a href=&quot;http://cran.r-project.org/web/packages/nlme/index.html&quot; rel=&quot;nofollow&quot;&gt;nlme&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can set a correlation profile for the errors in the regression, e.g. ARMA, etc:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; gls(Y ~ X, correlation=corARMA(p=1,q=1))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;for ARMA(1,1) errors.&lt;/p&gt;&#xA;" OwnerUserId="300" LastActivityDate="2011-01-23T15:52:41.160" CommentCount="1" />
  <row Id="6471" PostTypeId="2" ParentId="6469" CreationDate="2011-01-23T16:11:36.907" Score="2" ViewCount="" Body="&lt;p&gt;Use function &lt;strong&gt;gls&lt;/strong&gt; from package &lt;strong&gt;nlme&lt;/strong&gt;. Here is the example.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;##Generate data frame with regressor and AR(1) error. The error term is &#xA;## \eps_t=0.3*\eps_{t-1}+v_t&#xA;df &amp;lt;- data.frame(x1=rnorm(100), err=filter(rnorm(100)/5,filter=0.3,method=&quot;recursive&quot;))&#xA;&#xA;##Create ther response&#xA;df$y &amp;lt;- 1 + 2*df$x + df$err&#xA;&#xA;###Fit the model&#xA;gls(y~x, data=df, corr=corAR1(0.5,form=~1))&#xA;&#xA;Generalized least squares fit by REML&#xA;  Model: y ~ x &#xA;  Data: df &#xA;  Log-restricted-likelihood: 9.986475&#xA;&#xA; Coefficients:&#xA; (Intercept)           x &#xA;   1.040129    2.001884 &#xA;&#xA; Correlation Structure: AR(1)&#xA; Formula: ~1 &#xA; Parameter estimate(s):&#xA;     Phi &#xA; 0.2686271 &#xA;Degrees of freedom: 100 total; 98 residual&#xA;Residual standard error: 0.2172698 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Since model is fitted using maximum likelihood you need to supply starting values. The default starting value is 0, but as always it is good to try several values to ensure the convergence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As Dr. G pointed out you can also use other correlation structures, namely ARMA. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that in general least squares estimates are consistent if covariance matrix of regression errors is not multiple of identity matrix, so if you fit model with specific covariance structure, first you need to test whether it is appropriate.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-01-23T18:56:29.413" LastActivityDate="2011-01-23T18:56:29.413" CommentCount="1" />
  <row Id="6472" PostTypeId="2" ParentId="6468" CreationDate="2011-01-23T17:48:31.583" Score="2" ViewCount="" Body="&lt;p&gt;A total sample size of nine is simply too small to do a two-way factorial ANCOVA or a two way-factorial ANOVA. I would have thought it's stretching things to even do a one-way ANOVA on nine observations, especially with 30,000 different outcomes. I think you need a bigger sample if you wish to make any statistical inferences.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-01-23T17:48:31.583" CommentCount="1" />
  <row Id="6473" PostTypeId="1" AcceptedAnswerId="6474" CreationDate="2011-01-23T18:44:29.417" Score="1" ViewCount="158" Body="&lt;p&gt;Guys,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was wondering if anyone has ported the examples from Durbin &amp;amp; Koopman &quot;Time Series Analysis by State Space Methods&quot; to R?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can find RATS code for the examples online and obviosly SsfPack/Ox but no signs of an R companion for this book...&lt;/p&gt;&#xA;" OwnerUserId="300" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-01-23T19:54:57.243" LastActivityDate="2011-01-23T19:54:57.243" Title="R examples for Durbin &amp; Koopman &quot;Time Series Analysis by State Space Methods&quot;" Tags="&lt;r&gt;&lt;books&gt;" AnswerCount="1" />
  <row Id="6474" PostTypeId="2" ParentId="6473" CreationDate="2011-01-23T18:53:28.767" Score="5" ViewCount="" Body="&lt;p&gt;There is a package in CRAN (&lt;a href=&quot;http://cran.r-project.org/web/packages/KFAS/index.html&quot; rel=&quot;nofollow&quot;&gt;KFAS&lt;/a&gt;) which implements a good portion of the algorithms described in Durbin &amp;amp; Koopman, including &quot;exact&quot; non-informative distributions for the state vector, or parts of it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Although it is not paticularly tied to Durbin &amp;amp; Koopman's book, you might also be interested in package &lt;a href=&quot;http://cran.r-project.org/web/packages/dlm/index.html&quot; rel=&quot;nofollow&quot;&gt;dlm&lt;/a&gt; and the companion book &lt;a href=&quot;http://www.amazon.com/s/ref=nb_sb_noss?url=search-alias%3Dstripbooks&amp;amp;field-keywords=Petris&quot; rel=&quot;nofollow&quot;&gt;Dynamic Linear Models with R&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="892" LastActivityDate="2011-01-23T18:53:28.767" CommentCount="2" />
  <row Id="6475" PostTypeId="1" AcceptedAnswerId="6491" CreationDate="2011-01-23T19:48:59.000" Score="3" ViewCount="442" Body="&lt;p&gt;I have used a three-way ANOVA to analyze the the effect of genes, transcription factors, and different conditions on the gene expression. Now I have 9 elements, SSa, SSb, SSc, SSab, SSbc, SSac, SSabc, SSe. How can I interpret these values? I appreciate if you introduce me some resources, or share your own experience with interpreting 3-way ANOVA.&#xA;Best&lt;/p&gt;&#xA;" OwnerUserId="2885" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-01-23T19:55:49.120" LastActivityDate="2011-01-24T16:01:01.520" Title="How to interpret a three-way ANOVA?" Tags="&lt;anova&gt;&lt;interpretation&gt;" AnswerCount="1" CommentCount="3" FavoriteCount="1" />
  <row Id="6476" PostTypeId="2" ParentId="6469" CreationDate="2011-01-23T21:22:19.533" Score="4" ViewCount="" Body="&lt;p&gt;In addition to the &lt;code&gt;gls()&lt;/code&gt; function from &lt;code&gt;nlme&lt;/code&gt;, you can also use the &lt;code&gt;arima()&lt;/code&gt; function in the &lt;code&gt;stats&lt;/code&gt; package using MLE. Here is an example with both functions.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x &amp;lt;- 1:100&#xA;e &amp;lt;- 25*arima.sim(model=list(ar=0.3),n=100)&#xA;y &amp;lt;- 1 + 2*x + e&#xA;&#xA;###Fit the model using gls()&#xA;require(nlme)&#xA;(fit1 &amp;lt;- gls(y~x, corr=corAR1(0.5,form=~1)))&#xA;Generalized least squares fit by REML&#xA;  Model: y ~ x &#xA;  Data: NULL &#xA;  Log-restricted-likelihood: -443.6371&#xA;&#xA;Coefficients:&#xA;(Intercept)           x &#xA;   4.379304    1.957357 &#xA;&#xA;Correlation Structure: AR(1)&#xA; Formula: ~1 &#xA; Parameter estimate(s):&#xA;      Phi &#xA;0.3637263 &#xA;Degrees of freedom: 100 total; 98 residual&#xA;Residual standard error: 22.32908 &#xA;&#xA;###Fit the model using arima()&#xA;(fit2 &amp;lt;- arima(y, xreg=x, order=c(1,0,0)))&#xA;&#xA;Call:&#xA;arima(x = y, order = c(1, 0, 0), xreg = x)&#xA;&#xA;Coefficients:&#xA;         ar1  intercept       x&#xA;      0.3352     4.5052  1.9548&#xA;s.e.  0.0960     6.1743  0.1060&#xA;&#xA;sigma^2 estimated as 423.7:  log likelihood = -444.4,  aic = 896.81 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The advantage of the arima() function is that you can fit a much larger variety of ARMA error processes. If you use the auto.arima() function from the forecast package, you can automatically identify the ARMA error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(forecast)    &#xA;fit3 &amp;lt;- auto.arima(y, xreg=x)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="159" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2011-01-24T04:48:03.363" LastActivityDate="2011-01-24T04:48:03.363" CommentCount="2" />
  <row Id="6477" PostTypeId="1" AcceptedAnswerId="6486" CreationDate="2011-01-23T21:42:19.447" Score="5" ViewCount="401" Body="&lt;p&gt;I have run Levene's and Bartlett's test on groups of data from one of my experiments to validate that I am not violating ANOVA's assumption of homogeneity of variances. I'd like to check with you guys that I'm not making any wrong assumptions, if you don't mind :D&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The p-value returned by both of those tests is the probability that my data, if it were generated again using equal variances, would be the same. Thus, using those tests, to be able to say that I do not violate ANOVA's assumption of homogeneity of variances, I would only need a p-value that is higher than a chosen alpha level (say 0.05)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;E.g., with the data I am currently using, the Bartlett's test returns p=0.57, while the Levene's test (well they call it a Brown-Forsythe Levene-type test) gives a p=0.95. That means, no matter which test I use, I can say that the data I meet the assumption. Am I making any mistake?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="1320" LastEditorUserId="1320" LastEditorDisplayName="" LastEditDate="2011-01-23T23:15:38.383" LastActivityDate="2011-01-24T15:20:28.567" Title="Interpretation of the p-values produced by Levene's or Bartlett's test for homogeneity of variances" Tags="&lt;anova&gt;&lt;homogeneity&gt;&lt;levenes-test&gt;" AnswerCount="3" />
  <row Id="6478" PostTypeId="1" AcceptedAnswerId="6485" CreationDate="2011-01-23T22:06:03.373" Score="4" ViewCount="250" Body="&lt;p&gt;When building a CART model (specifically classification tree) using rpart (in R), it is often interesting to know what is the importance of the various variables introduced to the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, my question is:  &lt;strong&gt;What common measures exists for ranking/measuring variable importance of participating variables in a CART model?  And how can this be computed using R (for example, when using the rpart package)&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, here is some dummy code, created so you might show your solutions on it.  This example is structured so that it is clear that variable x1 and x2 are &quot;important&quot; while (in some sense) x1 is more important then x2 (since x1 should apply to more cases, thus make more influence on the structure of the data, then x2).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(31431)&#xA;n &amp;lt;- 400&#xA;x1 &amp;lt;- rnorm(n)&#xA;x2 &amp;lt;- rnorm(n)&#xA;x3 &amp;lt;- rnorm(n)&#xA;x4 &amp;lt;- rnorm(n)&#xA;x5 &amp;lt;- rnorm(n)&#xA;&#xA;X &amp;lt;- data.frame(x1,x2,x3,x4,x5)&#xA;&#xA;y &amp;lt;- sample(letters[1:4], n, T)&#xA;y &amp;lt;- ifelse(X[,2] &amp;lt; -1 , &quot;b&quot;, y)&#xA;y &amp;lt;- ifelse(X[,1] &amp;lt; 0 , &quot;a&quot;, y)&#xA;&#xA;require(rpart)&#xA;fit &amp;lt;- rpart(y~., X)&#xA;plot(fit); text(fit)&#xA;&#xA;info.gain.rpart(fit) # your function - telling us on each variable how important it is&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(references are always welcomed)&lt;/p&gt;&#xA;" OwnerUserId="253" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2011-01-27T10:42:22.663" LastActivityDate="2011-01-27T10:42:22.663" Title="How to measure/rank &quot;variable importance&quot; when using CART? (specifically using {rpart} from R)" Tags="&lt;r&gt;&lt;classification&gt;&lt;model-selection&gt;&lt;cart&gt;&lt;rpart&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="2" />
  <row Id="6479" PostTypeId="2" ParentId="6477" CreationDate="2011-01-23T22:20:05.120" Score="3" ViewCount="" Body="&lt;p&gt;You're on &quot;the right side of the p-value.&quot;  I'd just adjust your statement slightly to say that, IF the groups had equal variances in their populations, this result of p=0.95 indicates that random sampling using these n-sizes would produce variances this far apart or farther 95% of the time.  In other words, strictly speaking it's correct to phrase the result in terms of what it says about the null hypothersis, but not in terms of what it says about the future.&lt;/p&gt;&#xA;" OwnerUserId="2669" LastEditorUserId="2669" LastEditorDisplayName="" LastEditDate="2011-01-23T22:39:18.393" LastActivityDate="2011-01-23T22:39:18.393" CommentCount="1" />
  <row Id="6480" PostTypeId="1" AcceptedAnswerId="6482" CreationDate="2011-01-24T00:56:13.530" Score="6" ViewCount="136" Body="&lt;p&gt;I have run 10,000 random samples (910 data points each) on a data set of about 75,000 data points. I would like to make a continuous distribution out of this so that I can test the probability of getting the results of a particular non-random sample which I made based on theoretical concerns. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For each random sample (and for the &quot;real&quot; sample), I collected the number of hits, the number of hits + misses (this number varies somewhat for reasons which I don't think are important), and the relative frequency of hits (hits / hits+misses). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ideally, I'd like to take the relative frequencies and turn it into a continuous distribution (I assume it will be roughly normal), so that I can then see how likely the &quot;real&quot; relative frequency would be (using something simple like a T-test). But I'm not sure how to go about doing that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, is there an easier way to test the probability of obtaining my actual results just given a long file of the results of each random sample?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I assume there's some kind of R function that would make this fairly straightforward. Any hints?&lt;/p&gt;&#xA;" OwnerUserId="52" LastEditorUserId="144" LastEditorDisplayName="" LastEditDate="2011-01-24T09:37:48.097" LastActivityDate="2011-01-24T18:42:45.843" Title="How can I make a continuous distribution out of simulation results in R?" Tags="&lt;r&gt;&lt;distributions&gt;&lt;simulation&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="6481" PostTypeId="1" AcceptedAnswerId="6510" CreationDate="2011-01-24T06:40:36.760" Score="1" ViewCount="82" Body="&lt;p&gt;As title, I have hesitating whether I should use ordinal logistic regression or not, but I don't think I have time to understand that and to figure out how to work it out in R, can I just ignore it? Will the consequence be serious (i.e. seriously under/over-estimate the effect size)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="588" LastActivityDate="2011-01-25T07:54:33.577" Title="Consequence of ignoring the order of a categorical variable with different levels in logistic regression" Tags="&lt;logistic-regression&gt;&lt;ordinal&gt;" AnswerCount="1" CommentCount="4" />
  <row Id="6482" PostTypeId="2" ParentId="6480" CreationDate="2011-01-24T07:42:57.160" Score="2" ViewCount="" Body="&lt;p&gt;Hi Alan,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It sounds like what you're describing is a bootstrap simulation in order to estimate the distribution of a statistic (the relative frequencies).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The package I'd suggest you to look into is the &lt;a href=&quot;http://cran.r-project.org/web/packages/boot/index.html&quot; rel=&quot;nofollow&quot;&gt;boot package:&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;functions and datasets for&#xA;  bootstrapping from the book &quot;Bootstrap&#xA;  Methods and Their Applications&quot; by A.&#xA;  C. Davison and D. V. Hinkley (1997,&#xA;  CUP).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It should hold many functions needed for what (it seems to me) that you are trying to do.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are some &lt;a href=&quot;http://www.r-bloggers.com/bootstrapping-and-the-boot-package-in-r/&quot; rel=&quot;nofollow&quot;&gt;tutorials on the boot package&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Best,&#xA;Tal&lt;/p&gt;&#xA;" OwnerUserId="253" LastActivityDate="2011-01-24T07:42:57.160" />
  <row Id="6483" PostTypeId="1" CreationDate="2011-01-24T07:48:41.997" Score="6" ViewCount="107" Body="&lt;p&gt;Hello,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've items that have a geo-spatial position and a temporal origin. For both dimensions, I build clusters so far.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm now in search of a way to merge this different clusters forming spatio-temporal clusters. Of course, I want to prevent calculating completely new clusters from the scratch and rather use the existing information through the previous clustering.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any algorithm how to build 3d clusters by merging a previous 1d and a 2d clustering process?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="2880" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-01-24T19:33:04.137" LastActivityDate="2011-06-02T22:31:51.410" Title="Merging spatial and temporal clusters" Tags="&lt;time-series&gt;&lt;clustering&gt;&lt;spatial&gt;" AnswerCount="1" CommentCount="3" FavoriteCount="3" />
  <row Id="6484" PostTypeId="1" CreationDate="2011-01-24T09:23:24.853" Score="3" ViewCount="111" Body="&lt;p&gt;I am using binary logistic regression to test this model: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;logit(ρ1) = α + β1(INDDIR) + β2(INDCHAIR) + β3(BOARDSIZE) + β4(DIRSHIP) + β5(MEETING) + β6(EXPERT) + β7(INSTI) + β8(DEBT) + β9(LnSIZE) + β10(BIG4)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have some doubt on the casewise list (outliers). The first time I run the analysis, there were 44 outliers. I deleted all these 44 cases. Then, I re-run the analysis. Again, 11 outliers were appeared. I deleted all these 11 cases then re-run. Again and again, 6 outliers were found! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, should I delete all outliers or it would be a never ending story?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please help. Thanks. &lt;/p&gt;&#xA;" OwnerUserId="2793" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-01-24T11:10:04.533" LastActivityDate="2011-01-24T11:10:04.533" Title="Problem with casewise list in logistic regression" Tags="&lt;logistic-regression&gt;&lt;spss&gt;" CommentCount="7" />
  <row Id="6485" PostTypeId="2" ParentId="6478" CreationDate="2011-01-24T10:47:25.223" Score="6" ViewCount="" Body="&lt;p&gt;Variable importance might generally be computed based on the corresponding reduction of predictive accuracy when the predictor of interest is removed (with a permutation technique, like in Random Forest) or some measure of decrease of node impurity, but see (1) for an overview of available methods. An obvious alternative to CART is RF of course (&lt;a href=&quot;http://cran.r-project.org/web/packages/randomForest/index.html&quot; rel=&quot;nofollow&quot;&gt;randomForest&lt;/a&gt;, but see also &lt;a href=&quot;http://cran.r-project.org/web/packages/party/index.html&quot; rel=&quot;nofollow&quot;&gt;party&lt;/a&gt;). With RF, the Gini importance index is defined as the averaged Gini decrease in node impurities over all trees in the forest (it follows from the fact that the Gini impurity index for a given parent node is larger than the value of that measure for its two daughter nodes, see e.g. (2)).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know that Carolin Strobl and coll. have contributed a lot of simulation and experimental studies on (conditional) variable importance in RFs and CARTs (e.g., (3-4), but there are many other ones, or her thesis, &lt;a href=&quot;http://edoc.ub.uni-muenchen.de/8904/1/Strobl_Carolin.pdf&quot; rel=&quot;nofollow&quot;&gt;Statistical Issues in Machine Learning – Towards Reliable Split Selection and Variable Importance Measures&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To my knowledge, the &lt;a href=&quot;http://caret.r-forge.r-project.org/Classification_and_Regression_Training.html&quot; rel=&quot;nofollow&quot;&gt;caret&lt;/a&gt; package (5) only considers a loss function for the regression case (i.e., mean squared error). Maybe it will be added in the near future (anyway, an example with a classification case by k-NN is available in the on-line help for &lt;code&gt;dotPlot&lt;/code&gt;). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, Noel M O'Boyle seems to have some R code for &lt;a href=&quot;http://www.redbrick.dcu.ie/~noel/R_classification.html&quot; rel=&quot;nofollow&quot;&gt;Variable importance in CART&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;References&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Sandri and Zuccolotto. &lt;a href=&quot;http://www.msandri.it/docs/Sandri_Zuccolotto_2.pdf&quot; rel=&quot;nofollow&quot;&gt;A bias correction algorithm for the Gini variable importance measure in classification trees&lt;/a&gt;. 2008&lt;/li&gt;&#xA;&lt;li&gt;Izenman. &lt;em&gt;Modern Multivariate Statistical Techniques&lt;/em&gt;. Springer 2008&lt;/li&gt;&#xA;&lt;li&gt;Strobl, Hothorn, and Zeilis. &lt;a href=&quot;http://journal.r-project.org/archive/2009-2/RJournal_2009-2_Strobl~et~al.pdf&quot; rel=&quot;nofollow&quot;&gt;Party on!&lt;/a&gt;. &lt;em&gt;R Journal&lt;/em&gt; 2009 1/2&lt;/li&gt;&#xA;&lt;li&gt;Strobl, Boulesteix, Kneib, Augustin, and Zeilis. &lt;a href=&quot;http://www.biomedcentral.com/1471-2105/9/307&quot; rel=&quot;nofollow&quot;&gt;Conditional variable importance for random forests&lt;/a&gt;. &lt;em&gt;BMC Bioinformatics&lt;/em&gt; 2008, 9:307&lt;/li&gt;&#xA;&lt;li&gt;Kuhn. &lt;a href=&quot;http://www.jstatsoft.org/v28/i05/paper&quot; rel=&quot;nofollow&quot;&gt;Building Predictive Models in R Using the caret Package&lt;/a&gt;. &lt;em&gt;JSS&lt;/em&gt; 2008 28(5)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-01-24T10:47:25.223" CommentCount="2" />
  <row Id="6486" PostTypeId="2" ParentId="6477" CreationDate="2011-01-24T12:53:41.863" Score="4" ViewCount="" Body="&lt;p&gt;The p-value of your significance test can be interpreted as the probability of observing the value of the relevant statistic &lt;em&gt;as or more extreme&lt;/em&gt; than the value you actually observed, given that the null hypothesis is true.  (note that the p-value makes no reference to what values of the statistic are likely under the &lt;em&gt;alternative hypothesis&lt;/em&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;EDIT: in mathematical terminology, this can be written as:&lt;/em&gt;&#xA;$$p-value = Pr(T &amp;gt; T_{obs} | H_{0})$$&#xA;&lt;em&gt;where&lt;/em&gt; $T$ &lt;em&gt;is some function of the data (the &quot;statistic&quot;) and&lt;/em&gt; $T_{obs}$ &lt;em&gt;is the actual value of&lt;/em&gt; $T$ &lt;em&gt;observed;&lt;/em&gt; $H_{0}$ &lt;em&gt;denotes the conditions implied by the null hypothesis on the sampling distribution of&lt;/em&gt; $T$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can &lt;em&gt;never&lt;/em&gt; be sure that you're assumptions hold true, only whether or not &lt;em&gt;the data you observed is consistent with your assumptions&lt;/em&gt;.  A p-value gives a rough measure of this consistency.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A p-value does not give the probability that the same data will be observed, only the probability that the value of the statistic is as or more extreme to the value observed, given the null hypothesis.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastEditorUserId="2392" LastEditorDisplayName="" LastEditDate="2011-01-24T14:56:01.570" LastActivityDate="2011-01-24T14:56:01.570" CommentCount="2" />
  <row Id="6487" PostTypeId="1" CreationDate="2011-01-24T13:12:50.933" Score="1" ViewCount="157" Body="&lt;p&gt;Looking to estimate a &lt;code&gt;VARX(p,q)&lt;/code&gt; type VECM in R if possible.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd like to estimate a VECM with p lags (lags relative to the level, not diff of the vars) on the endogenous variables and q lags on the exogenous components. Any ideas?  At the moment I'm using the &lt;code&gt;ca.jo()&lt;/code&gt; function and adding the contemporaneous and lagged exogenous variables to the &lt;code&gt;dumvar=&lt;/code&gt; option.  I do not feel confident about this approach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can I do this in R?  Should I switch software?  &lt;/p&gt;&#xA;" OwnerUserId="2892" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-01-24T15:23:46.473" LastActivityDate="2011-01-24T15:23:46.473" Title="Lagged Exogenous Variables in VECM with R" Tags="&lt;r&gt;&lt;time-series&gt;&lt;cointegration&gt;" AnswerCount="1" />
  <row Id="6488" PostTypeId="2" ParentId="6487" CreationDate="2011-01-24T13:23:25.360" Score="4" ViewCount="" Body="&lt;p&gt;If your variables are exogenous and their lags are exogenous, then there is no problem in treating them as dummy variables. So you can add them to &lt;code&gt;dumvar&lt;/code&gt; option. My answer &lt;a href=&quot;http://stats.stackexchange.com/questions/4030/finding-coefficients-for-vecm-exogenous-variables/5291#5291&quot;&gt;to similar question&lt;/a&gt; applies.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-01-24T13:23:25.360" CommentCount="1" />
  <row Id="6489" PostTypeId="2" ParentId="2950" CreationDate="2011-01-24T15:19:25.240" Score="0" ViewCount="" Body="&lt;p&gt;I wouldnt cluster or classify the data at all. Since youve got ratio scaled data (isotopes) my method of choice would be PCA (Principal Component Analysis). By colouring your points in the PCA diagram according to your &quot;eco-type&quot; you could see their dispersal within the isotope ratio variation. Further, you will see the influence of each parameter (isotope ratios) in the biplot - the result will contain much more information than a classification.&lt;/p&gt;&#xA;" OwnerUserId="2893" LastActivityDate="2011-01-24T15:19:25.240" />
  <row Id="6490" PostTypeId="2" ParentId="6477" CreationDate="2011-01-24T15:20:28.567" Score="2" ViewCount="" Body="&lt;p&gt;While the previous comments are 100% correct, the plots produced for model objects in R provide a graphic summary of this question. Personally, i always find the plots much more useful than the p value, as one can transform the data afterward and spot the changes immediately in the plot.&lt;/p&gt;&#xA;" OwnerUserId="656" LastActivityDate="2011-01-24T15:20:28.567" CommentCount="1" />
  <row Id="6491" PostTypeId="2" ParentId="6475" CreationDate="2011-01-24T16:01:01.520" Score="2" ViewCount="" Body="&lt;p&gt;Just a (not so) brief note on the question as posed, before I give a way to intepret, you must be careful about &lt;em&gt;which&lt;/em&gt; sums of squares you have calculated, for they relate to different kinds of null hypothesis.  For example, the term $SSa$ may be for just the effect $a$ assuming everything else is in the model, or it may be any effect including $a$, which is $a+ab+ac+abc$, or some other null hypothesis which refers to effect $a$.  The standard literature refers to four different &lt;em&gt;Types&lt;/em&gt; of sums of squares, corresponding to different ways of analysing the data (e.g. total contribution, sequential contribution, marginal contribution).  You should investigate whatever software you are using to see what sums of squares it gives you.  If you are using R it is most likely &lt;em&gt;sequential&lt;/em&gt; which means basically &quot;in the order specified when you wrote the model&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There probably is no &quot;simple&quot; way to interpret a 3-way (or higher) ANOVA, but here's how I do it.  I think of it like a contingency table (or &quot;contingency cube&quot; in 3-way case).  You have a factor for the row, a factor for the column, and a factor for the &quot;depth&quot;.  As you move around the cells in the table, the mean changes and we can observe patterns in how the mean varies as we move around the table in a &quot;structured way&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the factors are &quot;independent&quot; in their association with the response, then this indicates that the mean value only depends on which row, column, and depth we are in.  For instance, if you change the row (but leave the column and depth fixed) then the effect of the column is unchanged, and the effect of the depth is unchanged.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If all factors are present, then it matters which particular cell the observation is in (i.e. each cell has a unique mean under the model).  For instance, if you change the row (but leave the column and depth fixed) then the effect of the column always changes, and the effect of the depth always changes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the &quot;in-between&quot; interpretation where 2-way interactions are present but 3-way interaction are not, it is a bit more tricky to interpret.  This goes along the lines of if you change the row (but leave the column and depth fixed) then the effect column-by-row always changes, and the effect of the depth-by-row always changes, but the effect of the depth-by-column remains the same.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So like I said, probably not &quot;simple&quot;, but this is how I would interpret interactions in a 3-way ANOVA.  And finally, the $SSe$ term is simply the variation which could not be explained by the full 3-way model (residual or error variance).  The sums of squares are devices to make testing a &quot;null hypothesis&quot; more straight forward.  But in multi-way ANOVA, there are many null hypothesis one can test, and so no &quot;automatic&quot; procedure know what the user will want in their particular case, so they give a default.  It is important to understand what the default sums of squares can be used to test, and then if any of those tests are the one you want to do.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-01-24T16:01:01.520" />
  <row Id="6492" PostTypeId="1" AcceptedAnswerId="6679" CreationDate="2011-01-24T17:55:40.930" Score="5" ViewCount="110" Body="&lt;p&gt;I want to use a t distribution to model short interval asset returns in a bayesian model. I'd like to estimate both the degrees of freedom (along with other parameters in my model) for the distribution.  I know that asset returns are quite non-normal, but I don't know too much beyond that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is an appropriate, mildly informative prior distribution for the degrees of freedom in such a model?&lt;/p&gt;&#xA;" OwnerUserId="1146" LastEditorUserId="1146" LastEditorDisplayName="" LastEditDate="2011-01-24T18:03:05.840" LastActivityDate="2011-01-28T19:39:49.070" Title="What's a good prior distribution for degrees of freedom in a t distribution?" Tags="&lt;distributions&gt;&lt;bayesian&gt;&lt;modeling&gt;&lt;prior&gt;" AnswerCount="2" CommentCount="5" FavoriteCount="2" />
  <row Id="6493" PostTypeId="1" AcceptedAnswerId="6506" CreationDate="2011-01-24T18:02:29.257" Score="3" ViewCount="87" Body="&lt;p&gt;I have been using log normal distributions as prior distributions for scale parameters (for normal distributions, t distributions etc.) when I have a rough idea about what the scale should be, but want to err on the side of saying I don't know much about it. I use it because the that use makes intuitive sense to me, but I haven't seen others use it. Are there any hidden dangers to this? &lt;/p&gt;&#xA;" OwnerUserId="1146" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-29T01:00:57.067" LastActivityDate="2011-04-29T01:00:57.067" Title="Mildly informative prior distributions for scale parameters" Tags="&lt;distributions&gt;&lt;bayesian&gt;&lt;modeling&gt;&lt;prior&gt;&lt;maximum-entropy&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="6494" PostTypeId="2" ParentId="6480" CreationDate="2011-01-24T18:42:45.843" Score="2" ViewCount="" Body="&lt;p&gt;Sounds like you want something like a &lt;a href=&quot;http://en.wikipedia.org/wiki/Kernel_density_estimation&quot; rel=&quot;nofollow&quot;&gt;Kernel Density Estimate&lt;/a&gt; of the distribution. In R I think you want &lt;a href=&quot;http://stat.ethz.ch/R-manual/R-patched/library/stats/html/density.html&quot; rel=&quot;nofollow&quot;&gt;density&lt;/a&gt; function. &lt;/p&gt;&#xA;" OwnerUserId="1146" LastActivityDate="2011-01-24T18:42:45.843" CommentCount="1" />
  <row Id="6496" PostTypeId="2" ParentId="6463" CreationDate="2011-01-24T19:35:16.437" Score="2" ViewCount="" Body="&lt;p&gt;Self promotion: &lt;a href=&quot;http://www.jmp.com&quot; rel=&quot;nofollow&quot;&gt;JMP&lt;/a&gt; (commercial desktop software) does choropleths of US states, US counties, world countries and regions of select other countries.&lt;/p&gt;&#xA;" OwnerUserId="1191" LastActivityDate="2011-01-24T19:35:16.437" />
  <row Id="6497" PostTypeId="2" ParentId="6492" CreationDate="2011-01-24T20:25:26.457" Score="0" ViewCount="" Body="&lt;p&gt;Here's an article that might help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.portfolioprobe.com/2011/01/12/the-number-1-novice-quant-mistake/&quot; rel=&quot;nofollow&quot;&gt;http://www.portfolioprobe.com/2011/01/12/the-number-1-novice-quant-mistake/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2775" LastActivityDate="2011-01-24T20:25:26.457" CommentCount="3" />
  <row Id="6498" PostTypeId="1" AcceptedAnswerId="6503" CreationDate="2011-01-25T02:19:35.050" Score="7" ViewCount="244" Body="&lt;p&gt;This may be hard to find, but I'd like to read a &lt;strong&gt;well-explained ARIMA example&lt;/strong&gt; that&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;uses minimal math&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;extends the discussion beyond building a model into using that model to forecast specific cases&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;uses graphics as well as numerical results to characterize the fit between forecasted and actual values.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="2669" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-29T23:03:48.523" LastActivityDate="2011-03-31T22:03:58.057" Title="Seeking certain type of ARIMA explanation" Tags="&lt;time-series&gt;&lt;arima&gt;" AnswerCount="4" FavoriteCount="1" />
  <row Id="6499" PostTypeId="1" AcceptedAnswerId="6548" CreationDate="2011-01-25T04:16:28.640" Score="5" ViewCount="178" Body="&lt;p&gt;Assume I have a random vector $X = \{x_1, x_2, ..., x_N\}$, composed of i.i.d. binomially distributed values.  If it would simplify the problem substantially, we can approximate them as normally distributed.  Given that all other parameters are fixed, I want to know how $E[min(X)]$ (the expected value of the smallest number in the vector $X$) scales with $N$.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't care about a precise answer.  I just want to know how it scales, i.e. linearly (obviously not), exponentially, power law, etc.&lt;/p&gt;&#xA;" OwnerUserId="1347" LastEditorUserId="1347" LastEditorDisplayName="" LastEditDate="2011-01-25T04:23:44.263" LastActivityDate="2011-01-27T20:44:19.810" Title="How do extreme values scale with sample size?" Tags="&lt;binomial&gt;&lt;normal-distribution&gt;&lt;approximation&gt;&lt;extreme-value&gt;" AnswerCount="3" CommentCount="1" />
  <row Id="6500" PostTypeId="2" ParentId="6498" CreationDate="2011-01-25T04:27:55.573" Score="6" ViewCount="" Body="&lt;p&gt;I tried to do that in chapter 7 of my &lt;a href=&quot;http://robjhyndman.com/forecasting/&quot; rel=&quot;nofollow&quot;&gt;1998 textbook&lt;/a&gt; with Makridakis &amp;amp; Wheelwright. Whether I succeeded or not I'll leave others to judge. You can read some of the chapter online via &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0471532339&quot; rel=&quot;nofollow&quot;&gt;Amazon&lt;/a&gt; (from p311). Search for &quot;ARIMA&quot; in the book to persuade Amazon to show you the relevant pages.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2011-01-25T04:27:55.573" CommentCount="1" />
  <row Id="6501" PostTypeId="2" ParentId="6353" CreationDate="2011-01-25T04:29:36.703" Score="1" ViewCount="" Body="&lt;p&gt;A simple way to turn categorical variables into a set of dummy variables for use in models in SPSS is using the do repeat syntax. This is the simplest to use if your categorical variables are in numeric order.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;*making vector of dummy variables.&#xA;vector dummy(3,F1.0).&#xA;*looping through dummy variables using do repeat, in this example category would be the categorical variable to recode. &#xA;do repeat dummy = dummy1 to dummy3 /#i = 1 to 3.&#xA;compute dummy = 0.&#xA;if category = #i dummy = 1.&#xA;end repeat.&#xA;execute. &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Otherwise you can simply run a set of if statements to make your dummy variables. My current version (16) has no native ability to specify a set of dummy variables automatically in the regression command (like you can in Stata using the &lt;a href=&quot;http://www.stata.com/support/faqs/data/dummy.html&quot; rel=&quot;nofollow&quot;&gt;xi command&lt;/a&gt;) but I wouldn't be surprised if this is available in some newer version. Also take note of dmk38's point #2, this coding scheme is assuming nominal categories. If your variable is ordinal more discretion can be used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also agree with dmk38 and the talk about regression being better because of its ability to specify missing data in a particular manner is a completely separate issue.  &lt;/p&gt;&#xA;" OwnerUserId="1036" LastActivityDate="2011-01-25T04:29:36.703" />
  <row Id="6502" PostTypeId="1" AcceptedAnswerId="6512" CreationDate="2011-01-25T04:55:04.823" Score="4" ViewCount="131" Body="&lt;p&gt;In a LASSO regression scenario where&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$y= X \beta + \epsilon$,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and the LASSO estimates are given by the following optimization problem&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ \min_\beta ||y - X \beta|| + \tau||\beta||_1$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any distributional assumptions regarding the $\epsilon$? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In an OLS scenario, one would expect that the $\epsilon$ are independent and normally distributed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does it make any sense to analyze the residuals in a LASSO regression?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know that the LASSO estimate can be obtained as the posterior mode under independent double-exponential priors for the $\beta_j$. But I haven't found any standard &quot;assumption checking phase&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance (:&lt;/p&gt;&#xA;" OwnerUserId="2902" LastActivityDate="2011-01-25T08:13:53.123" Title="LASSO assumptions" Tags="&lt;regression&gt;&lt;lasso&gt;&lt;assumptions&gt;&lt;residuals&gt;" AnswerCount="1" />
  <row Id="6503" PostTypeId="2" ParentId="6498" CreationDate="2011-01-25T05:14:30.753" Score="3" ViewCount="" Body="&lt;p&gt;My suggested reading for an intro to ARIMA modelling would be&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://books.google.com/books?id=D6-CAAAAIAAJ&amp;amp;q=Applied+Time+Series+Analysis+for+the+Social+Sciences&amp;amp;dq=Applied+Time+Series+Analysis+for+the+Social+Sciences&amp;amp;hl=en&amp;amp;ei=QaLNTKD-HcH88Aabt-Al&amp;amp;sa=X&amp;amp;oi=book_result&amp;amp;ct=result&amp;amp;resnum=1&amp;amp;ved=0CC8Q6AEwAA&quot; rel=&quot;nofollow&quot;&gt;Applied Time Series Analysis for the Social Sciences&lt;/a&gt; 1980&#xA;by R McCleary ; R A Hay ; E E Meidinger ; D McDowall&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is aimed at social scientists so the mathematical demands are not too rigorous. Also for shorter treatments I would suggest two Sage Green Books (although they are entirely redundant with the McCleary book),&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.sagepub.com/books/Book442?series=Series486&amp;amp;&quot; rel=&quot;nofollow&quot;&gt;Interrupted Time Series Analysis&lt;/a&gt;&#xA;by David McDowall, Richard McCleary,&#xA;Errol Meidinger, and Richard A. Hay,&#xA;Jr&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.sagepub.com/books/Book2605?series=Series486&amp;amp;&quot; rel=&quot;nofollow&quot;&gt;Time Series Analysis&lt;/a&gt; by Charles&#xA;W. Ostrom&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The Ostrom text is only ARMA modelling and does not discuss forecasting. I don't think they would meet your requirement for graphing forecast error either. I'm sure you could dig up more useful resources by examining questions tagged with time-series on this forum as well.&lt;/p&gt;&#xA;" OwnerUserId="1036" LastEditorUserId="1036" LastEditorDisplayName="" LastEditDate="2011-01-25T05:24:27.843" LastActivityDate="2011-01-25T05:24:27.843" CommentCount="1" />
  <row Id="6504" PostTypeId="2" ParentId="5831" CreationDate="2011-01-25T05:38:56.410" Score="3" ViewCount="" Body="&lt;p&gt;You might check out a recent presentation on SSRN by Bernard Black, &quot;Bloopers: How (Mostly) Smart People Get Causal Inference Wrong.&quot;&#xA;&lt;a href=&quot;http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1663404&quot; rel=&quot;nofollow&quot;&gt;http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1663404&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I will say that I also admire David Freedman and appreciate his work. Though I was a UC Berkeley grad student while he was here, he passed away before I had a chance to take his course. You might have a look at his collected works edited by a few other Berkeley professors: &quot;Statistical Models and Causal Inference: A Dialogue with the Social Sciences.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Charlie&lt;/p&gt;&#xA;" OwnerUserId="401" LastEditorUserId="401" LastEditorDisplayName="" LastEditDate="2011-01-25T06:09:58.707" LastActivityDate="2011-01-25T06:09:58.707" CommunityOwnedDate="2011-01-25T05:38:56.410" />
  <row Id="6505" PostTypeId="1" AcceptedAnswerId="6511" CreationDate="2011-01-25T05:51:13.933" Score="2" ViewCount="628" Body="&lt;p&gt;I'm not sure if I'm asking something stupid or off topic here, but I can't think where can I ask this question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;suppose I am going to do a univariate logistic regression on several independent variables, like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mod.a &amp;lt;- glm(x ~ a, data=z, family=binominal(&quot;logistic&quot;))&#xA;mod.b &amp;lt;- glm(x ~ b, data=z, family=binominal(&quot;logistic&quot;))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I did a model comparison (likelihood ratio test) to see if the model is better than the null model by this command&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1-pchisq(mod.a$null.deviance-mod.a$deviance, mod.a$df.null-mod.a$df.residual)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then I built another model with all variables in it&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mod.c &amp;lt;- glm(x ~ a+b, data=z, family=binomial(&quot;logistic&quot;))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In order to see if the variable is statistically significant in the multivariate model, I used the &lt;code&gt;lrtest&lt;/code&gt; command from &lt;code&gt;epicalc&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;lrtest(mod.c,mod.a) ### see if variable b is statistically significant after adjustment of a&#xA;lrtest(mod.c,mod.b) ### see if variable a is statistically significant after adjustment of b&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I wonder if the &lt;code&gt;pchisq&lt;/code&gt; method and the &lt;code&gt;lrtest&lt;/code&gt; method are equivalent for doing loglikelihood test? As I dunno how to use &lt;code&gt;lrtest&lt;/code&gt; for univate logistic model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Please kindly let me know if I asked the wrong question here, thanks!)&lt;/p&gt;&#xA;" OwnerUserId="588" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-01-25T08:13:33.250" LastActivityDate="2011-01-25T08:49:04.917" Title="Likelihood ratio test in R" Tags="&lt;r&gt;&lt;logistic-regression&gt;&lt;diagnostic&gt;" AnswerCount="2" CommentCount="3" FavoriteCount="1" />
  <row Id="6506" PostTypeId="2" ParentId="6493" CreationDate="2011-01-25T06:05:53.627" Score="4" ViewCount="" Body="&lt;p&gt;I would recommend using a &quot;Beta distribution of the second kind&quot; ($Beta_2$ for short) for a &lt;em&gt;mildly informative&lt;/em&gt; distribution, and to use the conjugate inverse gamma distribution if you have &lt;em&gt;strong&lt;/em&gt; prior beliefs.  The reason I say this is that the conjugate prior is &lt;em&gt;non-robust&lt;/em&gt; in the sense that, if the prior and data conflict, the prior has an &lt;em&gt;unbounded&lt;/em&gt; influence on the posterior distribution.  Such behaviour is what I would call &quot;dogmatic&quot;, and not justified by &lt;em&gt;mild&lt;/em&gt; prior information.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The property which determines &lt;em&gt;robustness&lt;/em&gt; is the tail-behaviour of the prior and of the likelihood.  A very good article outlining the technical details is &lt;a href=&quot;http://ba.stat.cmu.edu/journal/2006/vol01/issue01/andrade.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.  For example, a likelihood can be chosen (e.g. t-distribution) such that as an observation $y_i \rightarrow \infty$ (i.e. becomes arbitrarily large) it is discarded from the analysis of a location parameter (much in the same way that you would intuitively do with such an observation).  The rate of &quot;discarding&quot; depends on how heavy the tails of the distribution are.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some slides which show an application in the heirarchical modelling context can be found &lt;a href=&quot;http://www-stat.wharton.upenn.edu/statweb/Conference/OBayes09/slides/Luis%20Pericchi.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; (shows the mathematical form of the $Beta_2$ distribution), with a paper &lt;a href=&quot;http://www-stat.wharton.upenn.edu/statweb/Conference/OBayes09/AbstractPapers/Pericchi_Paper.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are not in the hierarchical modeling context, then I would suggest comparing the posterior (or whatever results you are creating) but use the &lt;em&gt;Jeffreys prior&lt;/em&gt; for a scale parameter, which is given by $p(\sigma)\propto\frac{1}{\sigma}$.  This can be created as a limit of the $Beta_2$ density as both its parameters converge to zero. For an approximation you could use small values.  But I would try to work out the solution &lt;em&gt;analytically&lt;/em&gt; if at all possible (and if not a complete analytical solution, get the analytical solution as far progressed as you possibly can), because you will not only save yourself some computational time, but you are also likely to &lt;em&gt;understand&lt;/em&gt; what is happening in your model better.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A further alternative is to use specify your prior information in the form of constraints (mean equal to $M$, variance equal to $V$, IQR equal to $IQR$, etc. with the values of $M,V,IQR$ specified by yourself), and then use the &lt;a href=&quot;http://en.wikipedia.org/wiki/Principle_of_maximum_entropy&quot; rel=&quot;nofollow&quot;&gt;maximum entropy distribution&lt;/a&gt; (search any work by Edwin Jaynes or Larry Bretthorst for a good explanation of what Maximum Entropy is and what it is not) with respect to the jeffrey's &quot;invariant measure&quot; $m(\sigma)=\frac{1}{\sigma}$.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;MaxEnt is the &quot;rolls royce&quot; version, while the $Beta_2$ is more a &quot;sedan&quot; version.  The reason for this is that the MaxEnt distribution &quot;assumes the least&quot; subject to the constraints you have put into it (e.g. no constraints means you just get the jeffreys prior), whereas the $Beta_2$ distribution may contain some &quot;hidden&quot; features which may or may not be desirable in your specific case (e.g. if the prior information is &lt;em&gt;more reliable&lt;/em&gt; than the data, then $Beta_2$ is bad).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The other nice property of MaxEnt distribution is that &lt;em&gt;if there are no unspecified constraints operating in the data generating mechanism&lt;/em&gt; then the MaxEnt distribution is &lt;em&gt;overwhelmingly&lt;/em&gt; the most likely distribution that you will see (we're talking odds way over billions and trillions to one).  Therefore, if the distribution you see is not the MaxEnt one, then there is likely &lt;em&gt;additional constraints which you have not specified&lt;/em&gt; operating on the true process, and the observed values can provide a clue as to what that constraint might be.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastEditorUserId="2392" LastEditorDisplayName="" LastEditDate="2011-01-25T13:03:33.710" LastActivityDate="2011-01-25T13:03:33.710" CommentCount="1" />
  <row Id="6507" PostTypeId="2" ParentId="6499" CreationDate="2011-01-25T07:00:26.590" Score="1" ViewCount="" Body="&lt;p&gt;The distribution of the minimum of &lt;em&gt;any&lt;/em&gt; set of N iid random variables is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$f_{min}(x)=Nf(x)[1-F(x)]^{N-1}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $f(x)$ is the pdf and $F(x)$ is the cdf (this is sometime called a $Beta-F$ distribution, because it is a compound of a Beta distribution and an arbitrary distribution).  Hence the expectation (in this particular case) is given by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$E[min(X)] = N\sum_{x=0}^{x=n} xf(x)[1-F(x)]^{N-1}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which means that $E[min(X)]=NE(x_1[1-F(x_1)]^{N-1})$.  Using the &quot;delta method&quot; to approximation this expectation $E[g(x)]\approx g(E[X])$ gives&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$E[min(X)]=NE(x_1[1-F(x_1)]^{N-1})\approx N(E(x_1)[1-F(E(x_1))]^{N-1})$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Substituting $np=E[x_1]$ then gives the approximation:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$E[min(X)]\approx Nnp[1-F(np)]^{N-1}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that $F(np)\approx \frac{1}{2}$ (via normal approx.) to give&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$E[min(X)]\approx \frac{Nnp}{2^{N-1}}$$&lt;/p&gt;&#xA;" OwnerUserId="2392" LastEditorUserId="2392" LastEditorDisplayName="" LastEditDate="2011-01-25T23:25:12.873" LastActivityDate="2011-01-25T23:25:12.873" CommentCount="9" />
  <row Id="6508" PostTypeId="2" ParentId="5327" CreationDate="2011-01-25T07:03:46.943" Score="1" ViewCount="" Body="&lt;p&gt;If we go for a simple answer, the excerpt from the &lt;a href=&quot;http://books.google.com/books?id=cdBPOJUP4VsC&amp;amp;printsec=frontcover&amp;amp;dq=wooldridge+econometrics&amp;amp;hl=fr&amp;amp;ei=1HM-TfauIIfqOZqBqY8L&amp;amp;sa=X&amp;amp;oi=book_result&amp;amp;ct=result&amp;amp;resnum=2&amp;amp;ved=0CC8Q6AEwAQ#v=onepage&amp;amp;q=wooldridge%20econometrics&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;Wooldridge book&lt;/a&gt; (page 533) is very appropriate:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;... both heteroskedasticity and nonnormality result in the Tobit estimator $\hat{\beta}$ being inconsistent for $\beta$. This inconsistency occurs because the derived density of $y$ given $x$ hinges crucially on $y^*|x\sim\mathrm{Normal}(x\beta,\sigma^2)$. This nonrobustness of the Tobit estimator shows that data censoring can be very costly: in the absence of censoring ($y=y^*$) $\beta$ could be consistently estimated under $E(u|x)=0$ [or even $E(x'u)=0$].&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The notations in this excerpt comes from Tobit model:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align}&#xA;y^{*}&amp;amp;=x\beta+u, \quad u|x\sim N(0,\sigma^2)\\&#xA;y^{*}&amp;amp;=\max(y^*,0)&#xA;\end{align}&#xA;where $y$ and $x$ are observed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To sum up the difference between least squares and Tobit regression is the inherent assumption of normality in the latter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also I always thought that the &lt;a href=&quot;http://www.jstor.org/pss/1914031&quot; rel=&quot;nofollow&quot;&gt;original article of Amemyia&lt;/a&gt; was quite nice in laying out the theoretical foundations of the Tobit regression. &lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-01-25T07:03:46.943" CommentCount="1" />
  <row Id="6509" PostTypeId="2" ParentId="6499" CreationDate="2011-01-25T07:31:29.650" Score="2" ViewCount="" Body="&lt;p&gt;The table in &lt;a href=&quot;http://books.google.com/books?id=dVP-RTea5wcC&amp;amp;lpg=PP1&amp;amp;dq=a%20first%20course%20in%20order%20statistics&amp;amp;hl=fr&amp;amp;pg=PA69#v=onepage&amp;amp;q&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;this page&lt;/a&gt; of &lt;a href=&quot;http://books.google.com/books?id=dVP-RTea5wcC&amp;amp;printsec=frontcover&amp;amp;dq=a+first+course+in+order+statistics&amp;amp;hl=fr&amp;amp;ei=93s-TaLkGZOWsgOvyOWkBQ&amp;amp;sa=X&amp;amp;oi=book_result&amp;amp;ct=result&amp;amp;resnum=1&amp;amp;ved=0CCcQ6AEwAA#v=onepage&amp;amp;q&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;this book&lt;/a&gt; might help you. The explicit formulas for the expectation of minimum of sample of binomial distributions is given &lt;a href=&quot;http://books.google.com/books?id=dVP-RTea5wcC&amp;amp;lpg=PP1&amp;amp;dq=a%20first%20course%20in%20order%20statistics&amp;amp;hl=fr&amp;amp;pg=PA68#v=onepage&amp;amp;q&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;in the page before&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-01-25T07:31:29.650" />
  <row Id="6510" PostTypeId="2" ParentId="6481" CreationDate="2011-01-25T07:48:25.753" Score="2" ViewCount="" Body="&lt;p&gt;If you need to quickly use an ordinal variable as an input, I recommend using it as a factor. The order of the factor is important for the interpretation of the results, but it's rather simple to reorder to the factor (using the factor command). The way the results are reported is the first group in the factor is held constant as the intercept and the coefficients for the subsequent groups are the difference between your reference group (the first item in the factor) and the other group. For example, if you three races coded, White, Black, and Asian. Since R does things alphabetically by default, Asian will become your reference group if you use &quot;+factor(race)&quot; which means you won't see a coefficient for Asian, but you will see a coefficient for Black and for White. These coefficients will be for the Difference between White and Asian and the Difference between Black and Asian.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An easy way to think of this is each level in the factor is treated as a binary variable. Each variable is assigned a coefficient, but the input is binary, meaning if your observation is Asian, Asian=1, Black=0, White=0, so it doesn't matter what the coefficients are for Black or White if your observation is Asian because any coefficient multiplied by zero will still be zero. Thus they're all mutually exclusive.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This also works for ordinal things, such as high, medium and low income. Depending on what your independent variable is and how the data was collected this may be completely suitable for your needs. It is important to note that this does not work well for hierarchical variables. If your input(s) are hierarchical in nature, I highly recommend using a nested mixed random effects model (lme4 package).&lt;/p&gt;&#xA;" OwnerUserId="2166" LastEditorUserId="2166" LastEditorDisplayName="" LastEditDate="2011-01-25T07:54:33.577" LastActivityDate="2011-01-25T07:54:33.577" CommentCount="2" />
  <row Id="6511" PostTypeId="2" ParentId="6505" CreationDate="2011-01-25T08:01:14.260" Score="4" ViewCount="" Body="&lt;p&gt;Basically, yes, provided you use the correct difference in log-likelihood:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; library(epicalc)&#xA;&amp;gt; model0 &amp;lt;- glm(case ~ induced + spontaneous, family=binomial, data=infert)&#xA;&amp;gt; model1 &amp;lt;- glm(case ~ induced, family=binomial, data=infert)&#xA;&amp;gt; lrtest (model0, model1)&#xA;Likelihood ratio test for MLE method &#xA;Chi-squared 1 d.f. =  36.48675 , P value =  0 &#xA;&amp;gt; model1$deviance-model0$deviance&#xA;[1] 36.48675&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and &lt;em&gt;not the deviance for the null model&lt;/em&gt; which is the same in both cases. The number of df is the number of parameters that differ between the two nested models, here df=1. BTW, you can look at the source code for &lt;code&gt;lrtest()&lt;/code&gt; by just typing&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; lrtest&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;at the R prompt.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-01-25T08:01:14.260" CommentCount="3" />
  <row Id="6512" PostTypeId="2" ParentId="6502" CreationDate="2011-01-25T08:13:53.123" Score="2" ViewCount="" Body="&lt;p&gt;I am not an expert on LASSO, but here is my take. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;First note that OLS is pretty robust to violations of indepence and normality. Then judging from the Theorem 6 &lt;a href=&quot;https://webspace.utexas.edu/xh588/www/papers/Lasso-NIPS.pdf&quot; rel=&quot;nofollow&quot;&gt;in this article&lt;/a&gt; and the discussion above it I guess, that in LASSO regression we are more concerned not with the distribution of $\varepsilon_i$, but in the joint distribution of $(y_i,x_i)$. The theorem relies on the assumption that $(y_i,x_i)$ is a sample, so this is comparable to usual OLS assumptions. But LASSO is less restrictive, it does not constrain $y_i$ to be generated from the linear model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To sum up, the answer to your first question is no. There are no distributional assumptions on $\varepsilon$, all distributional assumptions are on $(y,X)$. Furthermore they are weaker, since in LASSO nothing is postulate on conditional distribution $(y|X)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having said that,  the answer to the second question is then also no. Since the $\varepsilon$ does not play any role it does not make any sense to analyse them the way you analyse them in OLS (normality tests, heteroscedasticity, Durbin-Watson, etc). You should however analyse them in context how good the model fit was.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-01-25T08:13:53.123" />
  <row Id="6513" PostTypeId="1" AcceptedAnswerId="6539" CreationDate="2011-01-25T08:17:54.077" Score="2" ViewCount="381" Body="&lt;p&gt;I want to predict inter-day electricity load. My data are electricity loads for 11 months, sampled in 30 minute intervals. I also got the weather-specific data from a meteorological station (temperature, relative humidity, wind direction, wind speed, sunlight). From this, I want to predict the electricity load until the end of the day. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can run my algorithm until 10:00 of the present day and after that it should give the prediction of loads in 30 minute intervals. So, it should tell the load at 10:30, 11:00, 11:30 and so on until 24:00.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My first attempt was to create a &lt;strong&gt;linear model&lt;/strong&gt; in R.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;BP.TS &amp;lt;- ts(Buying.power, frequency = 48)&#xA;a &amp;lt;- data.frame(&#xA;Time, BP.TS, Weekday, Pressure, Temperature, RelHumidity, AvgWindSpeed, AvgWindDirection, MaxWindSpeed, MaxWindDirection, SunLightTime,&#xA;m, Buying.2dayago, AfterHolidayAndBPYesterday8, MovingAvgLast7DaysMidnightTemp&#xA;)&#xA;a &amp;lt;- a[(6*48+1):nrow(a),]&#xA;&#xA;start = 9716&#xA;steps.ahead = 21&#xA;par(mfrow=c(5,2))&#xA;for (i in 1:10) {&#xA;    train &amp;lt;- a[1:(start+(i-1)*48),]&#xA;    test &amp;lt;- a[((i-1)*48+start+1):((i-1)*48+start+steps.ahead),]&#xA;    summary(reg &amp;lt;- lm(log(BP.TS)~., data=train, na.action=NULL))&#xA;    pred &amp;lt;- exp(predict(reg, test))&#xA;&#xA;    plot(test$BP.TS, type=&quot;o&quot;)&#xA;    lines(pred, col=2)&#xA;    cat(&quot;MAE&quot;, mean(abs(test$BP.TS - pred)), &quot;\n&quot;)&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is not very succesful. Now I try to model the data with ARIMA. I used auto.arima() from the &lt;a href=&quot;http://cran.r-project.org/web/packages/forecast/index.html&quot; rel=&quot;nofollow&quot;&gt;forecast package&lt;/a&gt;. These are the results I got:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; auto.arima(BP.TS)&#xA;Series: BP.TS &#xA;ARIMA(2,0,1)(1,1,2)[48]                    &#xA;&#xA;Call: auto.arima(x = BP.TS) &#xA;&#xA;Coefficients:&#xA;         ar1      ar2     ma1    sar1     sma1    sma2&#xA;      1.1816  -0.2627  -0.554  0.4381  -1.2415  0.3051&#xA;s.e.  0.0356   0.0286   0.033  0.0952   0.0982  0.0863&#xA;&#xA;sigma^2 estimated as 256118:  log likelihood = -118939.7&#xA;AIC = 237893.5   AICc = 237893.5   BIC = 237947&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now if I try something like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;reg = arima(train$BP.TS, order=c(2,0,1), xreg=cbind(&#xA;train$Time, &#xA;train$Weekday, &#xA;train$Pressure, &#xA;train$Temperature, &#xA;train$RelHumidity, &#xA;train$AvgWindSpeed, &#xA;train$AvgWindDirection, &#xA;train$MaxWindSpeed, &#xA;train$MaxWindDirection, &#xA;train$SunLightTime,&#xA;train$Buying.2dayago,&#xA;train$MovingAvgLastNDaysLoad,&#xA;train$X1, train$X2, train$X3, train$X4, train$X5, train$X6, train$X7, train$X8, train$X9, &#xA;train$X11, train$X12, train$X13, train$X14, train$X15, train$X16, train$X17, train$X18, &#xA;train$MovingAvgLast7DaysMidnightTemp&#xA;))&#xA;&#xA;p &amp;lt;- predict(reg, n.ahead=21, newxreg=cbind(&#xA;test$Time, &#xA;test$Weekday, &#xA;test$Pressure, &#xA;test$Temperature, &#xA;test$RelHumidity, &#xA;test$AvgWindSpeed, &#xA;test$AvgWindDirection, &#xA;test$MaxWindSpeed, &#xA;test$MaxWindDirection, &#xA;test$SunLightTime,&#xA;test$Buying.2dayago,&#xA;test$MovingAvgLastNDaysLoad,&#xA;test$X1, test$X2, test$X3, test$X4, test$X5, test$X6, test$X7, test$X8, test$X9, &#xA;test$X11, test$X12, test$X13, test$X14, test$X15, test$X16, test$X17, test$X18, &#xA;test$MovingAvgLast7DaysMidnightTemp&#xA;))&#xA;&#xA;plot(test$BP.TS, type=&quot;o&quot;, ylim=c(6300,8300))&#xA;par(new=T)&#xA;plot(p$pred, col=2, ylim=c(6300,8300))&#xA;cat(&quot;MAE&quot;, mean(p$se), &quot;\n&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I get even worse results. Why? I ran out of ideas, so please help. If there is additional information I need to give, please ask.&lt;/p&gt;&#xA;" OwnerUserId="2889" LastActivityDate="2011-01-26T04:19:31.357" Title="Predicting daily electricity load - fitting time series" Tags="&lt;r&gt;&lt;time-series&gt;&lt;regression&gt;&lt;predictive-models&gt;&lt;arima&gt;" AnswerCount="2" CommentCount="5" FavoriteCount="4" />
  <row Id="6514" PostTypeId="1" CreationDate="2011-01-25T08:31:41.517" Score="1" ViewCount="102" Body="&lt;p&gt;I have a problem with my mixed effects model where I found that both subjective evaluations of my stimuli (valence - positive or negative and self-relevance - highly self-relevant or not self-relevant) are highly correlated with one another. Theoretically this is not too surprising but poses a problem in my model. Thus can I take out one of these two variables in my model? When I run the model on solely one of the two variables, I obtain more or less the same t-values of my significant interactions. I also have age as a categorical variable and two dimensional continuous variables in my model. My dependent variable is response times. &lt;/p&gt;&#xA;" OwnerUserId="2934" LastActivityDate="2011-01-25T08:31:41.517" Title="Problem of two highly correlated continuous variables in a mixed effects model" Tags="&lt;mixed-model&gt;&lt;continuous-data&gt;" CommentCount="4" />
  <row Id="6515" PostTypeId="2" ParentId="6505" CreationDate="2011-01-25T08:49:04.917" Score="3" ViewCount="" Body="&lt;p&gt;An alternative is the &lt;code&gt;lmtest&lt;/code&gt; package, which has an &lt;code&gt;lrtest()&lt;/code&gt; function which accepts a single model. Here is the example from &lt;code&gt;?lrtest&lt;/code&gt; in the &lt;code&gt;lmtest&lt;/code&gt; package, which is for an LM but there are methods that work with GLMs:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; require(lmtest)&#xA;Loading required package: lmtest&#xA;Loading required package: zoo&#xA;&amp;gt; ## with data from Greene (1993):&#xA;&amp;gt; ## load data and compute lags&#xA;&amp;gt; data(&quot;USDistLag&quot;)&#xA;&amp;gt; usdl &amp;lt;- na.contiguous(cbind(USDistLag, lag(USDistLag, k = -1)))&#xA;&amp;gt; colnames(usdl) &amp;lt;- c(&quot;con&quot;, &quot;gnp&quot;, &quot;con1&quot;, &quot;gnp1&quot;)&#xA;&amp;gt; fm1 &amp;lt;- lm(con ~ gnp + gnp1, data = usdl)&#xA;&amp;gt; fm2 &amp;lt;- lm(con ~ gnp + con1 + gnp1, data = usdl)&#xA;&amp;gt; ## various equivalent specifications of the LR test&#xA;&amp;gt;&#xA;&amp;gt; ## Compare two nested models&#xA;&amp;gt; lrtest(fm2, fm1)&#xA;Likelihood ratio test&#xA;&#xA;Model 1: con ~ gnp + con1 + gnp1&#xA;Model 2: con ~ gnp + gnp1&#xA;  #Df  LogLik Df  Chisq Pr(&amp;gt;Chisq)    &#xA;1   5 -56.069                         &#xA;2   4 -65.871 -1 19.605  9.524e-06 ***&#xA;---&#xA;Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 &#xA;&amp;gt;&#xA;&amp;gt; ## with just one model provided, compare this model to a null one&#xA;&amp;gt; lrtest(fm2)&#xA;Likelihood ratio test&#xA;&#xA;Model 1: con ~ gnp + con1 + gnp1&#xA;Model 2: con ~ 1&#xA;  #Df   LogLik Df  Chisq Pr(&amp;gt;Chisq)    &#xA;1   5  -56.069                         &#xA;2   2 -119.091 -3 126.04  &amp;lt; 2.2e-16 ***&#xA;---&#xA;Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1390" LastActivityDate="2011-01-25T08:49:04.917" CommentCount="1" />
  <row Id="6516" PostTypeId="1" CreationDate="2011-01-25T09:44:42.360" Score="5" ViewCount="57" Body="&lt;p&gt;I'm currently working on audio data trying to perform a recognition for given classes (for example grinding coffee etc.). However, I have some trouble distinguishing the null class from interesting sound segments.&#xA;Currently, I simply look at the audio intensity. As I have a limited, known number of classes I want to detect, I thought about saving some aggregate of the signals (mean fft) comparing the unclassified signal to it. If it is close enough to one of the saved aggregates do a classification, if not just drop it.&#xA;My approach seems to me quite naive. Therefore, input/ideas appreciated ;)&lt;/p&gt;&#xA;" OwnerUserId="2904" LastActivityDate="2011-01-25T15:11:37.393" Title="Ideas for Segmenting Audio Data" Tags="&lt;machine-learning&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="6517" PostTypeId="2" ParentId="6516" CreationDate="2011-01-25T10:33:14.700" Score="2" ViewCount="" Body="&lt;p&gt;There are interesting ideas you have already started to dig in. You should just dig a little more. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Using decomposition on a meaningfull basis like fft or wavelet transform is interesting. This is &lt;strong&gt;alternative representation.&lt;/strong&gt; You can think of a lot of alternative representations... wavelet is my favorite for audio data.  &lt;/li&gt;&#xA;&lt;li&gt;Using &quot;directly&quot; the whole signal to try to discriminate if it is from the null class is not a good idea, and as you already started with, you need to build up statistical summary. This can be one summary but there can be a few more. You have to think wether this statistical summary contains interesting information for the discimination you want to do or not. For example I doubt that the mean of the FFT is interesting in your case. This is &lt;strong&gt;dimensionality reduction.&lt;/strong&gt; &lt;/li&gt;&#xA;&lt;li&gt;There exists automatic ways to find meaningfull statistical summary. For example you can build up automatically a large number of candidate (with for example linear combination of your wavelet coefficient, or simply the coefficient themselves, or some other combination, or wavelet coefficient and fft coeff, ...) and measure there discrimination power somehow and keep the one that have best dicrimination power. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="223" LastActivityDate="2011-01-25T10:33:14.700" />
  <row Id="6518" PostTypeId="1" AcceptedAnswerId="6523" CreationDate="2011-01-25T11:40:59.857" Score="9" ViewCount="165" Body="&lt;p&gt;Has anyone written a brief survey of the various approaches to statistics? To a first approximation you have frequentist and Bayesian statistics. But when you look closer you also have other approaches like likelihoodist and empirical Bayes. And then you have subdivisions within groups such as subjective Bayes objective Bayes within Bayesian statistics etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A survey article would be good. It would be even better if it included a diagram. &lt;/p&gt;&#xA;" OwnerUserId="319" LastEditorUserId="319" LastEditorDisplayName="" LastEditDate="2011-01-27T21:06:54.677" LastActivityDate="2011-01-27T21:06:54.677" CommunityOwnedDate="2011-01-25T14:43:09.693" Title="Statistical landscape" Tags="&lt;bayesian&gt;&lt;frequentist&gt;&lt;philosophical&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="6519" PostTypeId="1" CreationDate="2011-01-25T12:11:54.427" Score="3" ViewCount="75" Body="&lt;p&gt;We are measuring data with a high sample rate (20 kHz) and calculated a big standard error due to our system setup. Currently we are only interested in slow signals (in the order of Hz's) is it valid to use averaging (once per 20.000 samples) and thus lower our standard error with an order 20.000?&lt;/p&gt;&#xA;" OwnerUserId="2907" LastActivityDate="2011-01-25T16:32:18.687" Title="What does averaging do for the noise level of my signal" Tags="&lt;standard-error&gt;&lt;measurement&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="6520" PostTypeId="1" AcceptedAnswerId="6522" CreationDate="2011-01-25T12:26:40.787" Score="4" ViewCount="124" Body="&lt;p&gt;Hi All,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have 100 geographical regions in a country. For each region the total number of houses and the number of vacant houses have been collected yearly over 20 years. I have also some other economic indicators at the country level (GDP, interest rate etc.). Now, given the forecasts for these indicators for next year, I want to forecast next year vacancy rate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have first used an auto-regressive mixed-effect model in &lt;code&gt;R&lt;/code&gt; (package lme4) where the vacancy rate (computed as the ratio of vacant houses over the total number of houses) in a region depends on the last year's vacancy rate, the mean vacancy rate of neighboring regions, GDP and interest rate. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem with this model is that the vacancy rate can go outside the range [0,1], which obviously does not make sense. I need to restrict the range of the vacancy rate: a simple fix is restricting &lt;em&gt;ex post&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anybody have experience with such models? I think that I can use some mixed multinomial logit model probably.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would appreciate if you could provide guidance along with some &lt;code&gt;R&lt;/code&gt; code.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regards&lt;/p&gt;&#xA;" OwnerUserId="1443" LastEditorUserId="1443" LastEditorDisplayName="" LastEditDate="2011-02-07T09:08:55.787" LastActivityDate="2011-02-07T09:08:55.787" Title="Modeling vacancy rate" Tags="&lt;time-series&gt;&lt;mixed-model&gt;&lt;forecasting&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="6521" PostTypeId="2" ParentId="6520" CreationDate="2011-01-25T12:43:29.727" Score="3" ViewCount="" Body="&lt;p&gt;It would seem to make sense to use a &lt;em&gt;generalized&lt;/em&gt; linear mixed model with &lt;code&gt;family=binomial&lt;/code&gt; and a logit or probit link. This would restrict your fitted values to the range (0,1). I don't know whether you can combine that with an autoregressive error structure in &lt;code&gt;lmer4&lt;/code&gt; though.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-01-25T12:43:29.727" CommentCount="4" />
  <row Id="6522" PostTypeId="2" ParentId="6520" CreationDate="2011-01-25T12:53:36.730" Score="4" ViewCount="" Body="&lt;p&gt;One of the tricks in modelling percentages is to use the &lt;a href=&quot;http://en.wikipedia.org/wiki/Logit&quot; rel=&quot;nofollow&quot;&gt;logit transformation&lt;/a&gt;. Then instead of modelling percentage $p_i$ as linear function you model the logit transform of this percentage:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align}&#xA;y_i=\log\frac{p_i}{1-p_i}&#xA;\end{align}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In R you will need to create new transformed variable and use it as a dependent variable in lmer. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might look into modelling directly the number of empty houses instead of percentages, then you will not have a problem with non-sensical values. I suggest using log transformation for that.  This of course means that you might get more non-vacant houses than there are houses, but this can be used as an indicator of model inadequacies. If on the other hand you have for some regions full booking in historical data meaning that demand was larger than supply, you might want to look into censored regression models.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-01-25T12:53:36.730" />
  <row Id="6523" PostTypeId="2" ParentId="6518" CreationDate="2011-01-25T13:11:39.397" Score="2" ViewCount="" Body="&lt;p&gt;Here's one I found via a Google Image search, but maybe it's &lt;em&gt;too&lt;/em&gt; brief, and the diagram too simple: &lt;a href=&quot;http://labstats.net/articles/overview.html&quot; rel=&quot;nofollow&quot;&gt;http://labstats.net/articles/overview.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-01-25T13:11:39.397" CommentCount="1" />
  <row Id="6524" PostTypeId="1" CreationDate="2011-01-25T13:28:53.477" Score="1" ViewCount="242" Body="&lt;p&gt;fT(t;B,C) = exp(-t/C)-exp(-t/B) / C-B where our mean is C+B and t&gt;0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;so far i have found my log likelihood functions and differentiated them as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;dl/dB = sum[t*exp(t/C) / (B^2(exp(t/c)-exp(t/B)))] +n/(C-B) = 0&lt;/p&gt;&#xA;&#xA;&lt;p&gt;i have also found a similar dl/dC.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have now been asked to comment what you can find in the way of sufficient statistics for estimating these parameters and why there is no simple way of using Maximum Likelihood for estimation in the problem. I am simply unsure as to what to comment upon. Any help would be appreciated. Thanks, Rachel&lt;/p&gt;&#xA;" OwnerUserId="2908" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-01-25T14:36:03.493" LastActivityDate="2011-04-26T02:41:34.593" Title="Maximum likelihood and sufficient statistics" Tags="&lt;homework&gt;&lt;maximum-likelihood&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="6525" PostTypeId="1" CreationDate="2011-01-25T13:59:38.873" Score="2" ViewCount="197" Body="&lt;p&gt;I'm very new to R Programming. So please excuse for such a simple doubt.&#xA;&lt;img src=&quot;http://i.imgur.com/pcmle.png&quot; alt=&quot;How to plots this graph in R?&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to plot the above graph. The x &amp;amp; y values are sequence from 0 to 2560. I want plot a a curve on the points where x*y=10^6.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are the line required in R Programming Language.&lt;/p&gt;&#xA;" OwnerUserId="1706" LastEditorUserId="1706" LastEditorDisplayName="" LastEditDate="2011-01-25T14:54:24.167" LastActivityDate="2011-01-25T15:18:53.783" Title="How can I plot this simple graph (Refer Image) in R?" Tags="&lt;r&gt;" AnswerCount="2" CommentCount="3" FavoriteCount="0" />
  <row Id="6526" PostTypeId="2" ParentId="6518" CreationDate="2011-01-25T14:00:23.943" Score="2" ViewCount="" Body="&lt;p&gt;Here's a longer article. No diagram and isn't exactly a survey, more, as the abstract puts it, of 'an idiosyncratic walk through some of these issues':&lt;/p&gt;&#xA;&#xA;&lt;p&gt;M. J. Bayarri and J. O. Berger (2004). &lt;a href=&quot;http://www.jstor.org/stable/4144373&quot; rel=&quot;nofollow&quot;&gt;&quot;The Interplay of Bayesian and Frequentist Analysis&quot;&lt;/a&gt;. &lt;em&gt;Statistical Science&lt;/em&gt; &lt;strong&gt;19&lt;/strong&gt; (1):58-80. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(access requires a JSTOR subscription)&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-01-25T14:00:23.943" CommentCount="2" />
  <row Id="6527" PostTypeId="2" ParentId="6525" CreationDate="2011-01-25T14:08:35.037" Score="6" ViewCount="" Body="&lt;p&gt;I think all you need is:   &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; curve(1e6/x,0,2560)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;EDIT in light of comments:&lt;br&gt;&#xA;Or perhaps: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plot(...&amp;lt;your data&amp;gt;...)&#xA;curve(1e6/x, 1e6/2560,2560, add=TRUE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="449" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-01-25T15:18:53.783" LastActivityDate="2011-01-25T15:18:53.783" CommentCount="7" />
  <row Id="6528" PostTypeId="2" ParentId="6525" CreationDate="2011-01-25T14:12:36.200" Score="0" ViewCount="" Body="&lt;p&gt;The easiest way is to compute all &lt;em&gt;y&lt;/em&gt; values at every given &lt;em&gt;x&lt;/em&gt; values, like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df &amp;lt;- data.frame(x=1:2560)&#xA;df$y &amp;lt;- 10^6/df$x&#xA;# the latter equivalent to:&#xA;# df &amp;lt;- within(df,y&amp;lt;-10^6/x)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And after plot the dataframe:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plot(df, type=&quot;l&quot;, main=expression(f(x) == frac(10^{6},x)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/NkgVw.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="2714" LastEditorUserId="2714" LastEditorDisplayName="" LastEditDate="2011-01-25T15:10:03.390" LastActivityDate="2011-01-25T15:10:03.390" CommentCount="2" />
  <row Id="6529" PostTypeId="2" ParentId="6455" CreationDate="2011-01-25T14:15:47.943" Score="3" ViewCount="" Body="&lt;p&gt;The answer definitely depends on: &#xA;What are actually trying to use the $Q$ test for? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The common reason is: to be more or less &lt;em&gt;confident&lt;/em&gt; about joint statistical significance of the null hypothesis of no autocorrelation up to lag $h$ (alternatively assuming that you have something close to &lt;em&gt;a weak white noise&lt;/em&gt;) and to build a &lt;em&gt;parsimonious&lt;/em&gt; model, having as little number of parameters as possible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Usually time series data has natural seasonal pattern, so the practical rule-of-thumb would be to set $h$ to twice this value. Another one is the forecasting horizon, if you use the model for forecasting needs. Finally if you find some significant departures at latter lags try to think about the corrections (could this be due to some seasonal effects, or the data was not corrected for outliers).&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Rather than using a single value for h, suppose that I do the Ljung-Box test for all h&amp;lt;50, and then pick the h which gives the minimum p value.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It's a &lt;em&gt;joint significance&lt;/em&gt; test, so if the choice of $h$ is data-driven, then why should I care about some small (occasional?) departures at any lag less than $h$, supposing that it is much less than $n$ of course (the power of the test you mentioned). Seeking to find a simple yet relevant model I suggest the information criteria as described below.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;My question concerns how to interpret the test if $p&amp;lt;0.05$ for some values of $h$ and not for other values.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So it will depend on how far from the present it happens. Disadvantages of far departures: more parameters to estimate, less degrees of freedom, worse predictive power of the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Try to estimate the model including the MA and\or AR parts at the lag where the departure occurs AND additionally look at one of information criteria (either AIC or BIC depending on the sample size) this would bring you more insights on what model is more parsimonious. Any out-of-sample prediction exercises are also welcome here.&lt;/p&gt;&#xA;" OwnerUserId="2645" LastActivityDate="2011-01-25T14:15:47.943" CommentCount="1" />
  <row Id="6530" PostTypeId="2" ParentId="6519" CreationDate="2011-01-25T14:57:37.033" Score="2" ViewCount="" Body="&lt;p&gt;Averaging is just a low pass filter so if you want to filter out the high frequency component of your signal, it will do that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Obviously different averaging techniques and different parameters will filter out high frequency components in a different way. See for instance &lt;a href=&quot;http://ptolemy.eecs.berkeley.edu/eecs20/week12/freqResponseRA.html&quot; rel=&quot;nofollow&quot;&gt;this illustration of the frequency response of a simple moving average&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="300" LastActivityDate="2011-01-25T14:57:37.033" />
  <row Id="6531" PostTypeId="2" ParentId="6516" CreationDate="2011-01-25T15:11:37.393" Score="1" ViewCount="" Body="&lt;p&gt;I think the R package &lt;a href=&quot;http://www.ggobi.org/rggobi/introduction.pdf&quot; rel=&quot;nofollow&quot;&gt;rggobi&lt;/a&gt; is exactly what you're looking for.  Audo recognition is even their &lt;a href=&quot;http://www.ggobi.org/book/2007-infovis/05-clustering.pdf&quot; rel=&quot;nofollow&quot;&gt;example problem&lt;/a&gt;!&lt;/p&gt;&#xA;" OwnerUserId="2817" LastActivityDate="2011-01-25T15:11:37.393" />
  <row Id="6532" PostTypeId="2" ParentId="6483" CreationDate="2011-01-25T16:14:11.083" Score="1" ViewCount="" Body="&lt;p&gt;I don’t know if I understood your question correctly, but I’ll give it a try.&#xA;Any hierarchical agglomerative algorithm can do the job. Remember that agglomerative algorithms proceed by “pasting” observations to a cluster and treating the cluster as a single unit. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would suggest this:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Substitute  your 1d or 2d observations with the clusters’ centroids.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Use a distance to assign the temporal observations into the 1d/2d cluster. (Euclidean distance might work).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Hope this works =)&lt;/p&gt;&#xA;" OwnerUserId="2902" LastEditorUserId="2902" LastEditorDisplayName="" LastEditDate="2011-01-25T16:19:54.587" LastActivityDate="2011-01-25T16:19:54.587" CommentCount="1" />
  <row Id="6533" PostTypeId="2" ParentId="6519" CreationDate="2011-01-25T16:32:18.687" Score="1" ViewCount="" Body="&lt;p&gt;Yes, you can reduce the noise by averaging or in general by reducing the sampling rate of the signal. This is a very common technique in signal processing commonly refereed to as &lt;a href=&quot;http://en.wikipedia.org/wiki/Oversampling&quot; rel=&quot;nofollow&quot;&gt;oversampling&lt;/a&gt;. It is widely used in sigma-delta converters. Oversampling basically means that the sampling rate of the signal is much higher than the needed Nyquist limit (i.e. 2 times higher than bandwidth of the signal). By low-pass filtering and decimating the signal, you reduce the standard deviation of the noise by the decimation factor (assuming a uniform distribution for the noise).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having said that, I doubt in your example you can &lt;em&gt;downsample&lt;/em&gt; by a factor of 20000. From your description it seems that your &quot;signal&quot; include frequency up to few Hertz. Notice that to get best/correct results you should sample your signal higher than the Nyquist limit. For example if you want to preserve everything below 20 Hz, you should not decimate by a factor more than 500 (i.e. 20000 / 500 = 40 Hz = 2 x 20 Hz). If you are not much into signal processing, to do the downsampling correctly I suggest using a ready made package so that you know the lowpass-filter is implemented correctly. In MATLAB you can use the &lt;a href=&quot;http://www.mathworks.com/help/toolbox/signal/resample.html&quot; rel=&quot;nofollow&quot;&gt;resample&lt;/a&gt; function in the signal-processing toolbox. In R you may use the &lt;a href=&quot;http://rgm2.lab.nig.ac.jp/RGM2/R_man-2.9.0/library/signal/man/decimate.html&quot; rel=&quot;nofollow&quot;&gt;decimate&lt;/a&gt; function from the signal package.&lt;/p&gt;&#xA;" OwnerUserId="2020" LastActivityDate="2011-01-25T16:32:18.687" />
  <row Id="6534" PostTypeId="1" AcceptedAnswerId="6536" CreationDate="2011-01-25T16:44:25.457" Score="1" ViewCount="609" Body="&lt;p&gt;So, I have a data set of percentages like so:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;100   /   10000   = 1% (0.01)&#xA;2     /     5     = 40% (0.4)&#xA;4     /     3     = 133% (1.3) &#xA;1000  /   2000    = 50% (0.5)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to find the standard deviation of the percentages, but weighted for their data volume. ie, the first and last data points should dominate the calculation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I do that? And is there a simple way to do it in Excel?&lt;/p&gt;&#xA;" OwnerUserId="142" LastEditorUserId="142" LastEditorDisplayName="" LastEditDate="2011-01-25T23:42:29.083" LastActivityDate="2011-01-25T23:42:29.083" Title="How do I calculate a weighted standard deviation? In Excel?" Tags="&lt;standard-deviation&gt;&lt;excel&gt;&lt;weighted-mean&gt;" AnswerCount="1" />
  <row Id="6535" PostTypeId="1" CreationDate="2011-01-25T17:06:41.823" Score="2" ViewCount="96" Body="&lt;p&gt;Is there any downside to using a repeated-measure ANOVA when you have a within-subjects 2-level factor where 1 level is  represented by more trials than the other. I know in ANOVA we are comparing means, but does the the tighter standard error on the one level throw off the ANOVA? For example, if my IV is stimulus congruency and I present congruent trials 80 times while only presenting incongruent trials 20 times, can I just run the the ANOVA including all of the trials or should I run the analysis by selecting the same amount of trials for each factor level (i.e. select only 20 of the congruent trials to pair with the 20 incongruent trials)? If this is OK for a strictly within-subjects design, does it cause problems if a balanced or unbalanced between-subjects variable is added to the design?&lt;/p&gt;&#xA;" OwnerUserId="2322" LastEditorUserId="2322" LastEditorDisplayName="" LastEditDate="2011-01-25T19:52:40.277" LastActivityDate="2011-01-25T19:52:40.277" Title="Does a within-subjects factor with unequally represented levels mess up a repeated-measures ANOVA?" Tags="&lt;anova&gt;&lt;repeated-measures&gt;&lt;standard-error&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="6536" PostTypeId="2" ParentId="6534" CreationDate="2011-01-25T18:22:20.437" Score="5" ViewCount="" Body="&lt;p&gt;The formula for weighted standard deviation is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ \sqrt{ \frac{ \sum_{i=1}^N w_i (x_i - \bar{x}^*)^2 }{ \frac{(M-1)}{M} \sum_{i=1}^N w_i } } $,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$N$ is the number of observations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$M$ is the number of nonzero weights.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$w_i$ are the weights&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$x_i$ are the observations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\bar{x}^*$ is the weighted mean.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Remember that the formula for weighted mean is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\bar{x}^* = \sum_{i=1}^N w_i x_i$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Use the appropriate weights to get the desired result. In your case I would suggest to use $\frac{\mbox{Number of cases in segment}}{\mbox{Total number of cases}}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To do this in Excel, you need to calculate the weighted mean first. Then calculate the $(x_i - \bar{x}^*)^2$ in a separate column. The rest must be very easy.&lt;/p&gt;&#xA;" OwnerUserId="2902" LastActivityDate="2011-01-25T18:22:20.437" />
  <row Id="6537" PostTypeId="2" ParentId="6535" CreationDate="2011-01-25T18:58:46.707" Score="1" ViewCount="" Body="&lt;p&gt;I think there are ways to do unbalanced repeated measures ANOVA; maybe look at the books &quot;Variance Components&quot; by Searle, Casella, and McCulloch, and &quot;Components of Variance&quot; by Cox and Solomon. It also seems to be in some books about multilevel / hierarchical / mixed effect models (e.g. Raudenbush and Bryk), where it is treated as a special case of a hierarchical / multilevel / mixed effect model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not sure what programs support unbalanced repeated measures anova (&lt;a href=&quot;http://www.ats.ucla.edu/stat/sas/library/repeated_ut.htm&quot; rel=&quot;nofollow&quot;&gt;&quot;Unbalanced designs create special difficulties for the analysis of variance.&quot;&lt;/a&gt;), but most statistical programs support mixed effect models. Is there any reason that would be insufficient?&lt;/p&gt;&#xA;" OwnerUserId="2739" LastActivityDate="2011-01-25T18:58:46.707" CommentCount="4" />
  <row Id="6538" PostTypeId="1" CreationDate="2011-01-25T19:03:49.097" Score="13" ViewCount="579" Body="&lt;p&gt;I know people love to close duplicates so I am &lt;strong&gt;not&lt;/strong&gt; asking for a reference to &lt;em&gt;start&lt;/em&gt; learning statistics (as &lt;a href=&quot;http://stats.stackexchange.com/questions/414/introduction-to-statistics-for-mathematicians&quot;&gt;here&lt;/a&gt;). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a doctorate in mathematics but never learned statistics. What is the shortest route to the equivalent knowledge to a top notch BS statistics degree and how do I measure when I have acheived that. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If a list of books would suffice (assuming I do the exercises lets say), that's terrific. Yes, I expect working out problems to be an implicit part of learning it but I want to fast track as much as realistically possible. I am not looking for an insanely rigorous treatment unless that is part of what statistical majors generally learn.&lt;/p&gt;&#xA;" OwnerUserId="2912" LastActivityDate="2011-06-05T14:57:27.867" CommunityOwnedDate="2011-01-25T21:13:21.110" Title="Mathematician wants the equivalent knowledge to a quality stats degree" Tags="&lt;mathematics&gt;&lt;learning&gt;" AnswerCount="7" CommentCount="5" FavoriteCount="17" />
  <row Id="6539" PostTypeId="2" ParentId="6513" CreationDate="2011-01-25T19:30:59.893" Score="3" ViewCount="" Body="&lt;p&gt;I've played around with electrical demand models, and I can tell you that it's a good idea to start &quot;zoomed out&quot;.   Each region has its own characteristics, but the general idea is the same.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Electric demand is a function of many variables.  Starting with the slowest moving terms.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;General Economic Activity is the slowest moving term (typically the 3 to 8 year time frame).  This term is typically related to Gross Domestic Product for the area.  Electrical Demand may generally grow faster than GDP, but the electrical demand &quot;ups&quot; during good economic times, and demand &quot;downs&quot; during recessions provide an obvious link to GDP.   See the blue line in the first graph below.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Next, is the Seasonal Term (annual time frame).  For instance in the U.S., the Summer Peak shows up in August, the Winter Peak shows up in January, the Spring Trough shows up in April and the Fall Trough shows up in November.   See the red line in top two graphs below.   In the second graph, I have shown the Seasonal Term to be constant for each month, but you can easily improve that by a linear or non-linear relationship for each month (monthly time frame).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;You are now down to the daily time frame.  The bottom graph shows the Electrical Demand for Texas for one 24 hour period (12/22/2010).  The Day-time Peak was at 7:00PM (19:00) and the Night-time Trough was at 4:00AM (04:00).  This time frame is where you want to consider holidays, weekends, weather, etc.   However, keep in mind that those other variables (in 1 and 2 above) are also affecting your results.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;So, from your description, you have data for 11 months.   Look at the first graph below and assume that you have data for 11 months.   Is that enough to get an idea of the Seasonal Term for the year?   I would use a minimum of 10 years of monthly data to get a feel for the Seasonal Term.   The idea here is to tweek the structure of your daily model differently during months of &quot;rapid seasonal change&quot; versus months of &quot;slow seasonal change&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Next, I would play around with the size and structure of the &quot;data window&quot; that you will use to estimate your daily model.   For example, will you get a better daily model if you include daily fall and winter data when estimating a summer daily model?   Or, is it better to use 10 rescaled &quot;summer data windows&quot;, one for each year in 10 years of data, when estimating a summer daily model?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you get all of the deterministic terms working well, then, and only then would I go after the ARIMA terms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/dviD3.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/zk4Lk.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/d3dQM.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="2775" LastActivityDate="2011-01-25T19:30:59.893" CommentCount="6" />
  <row Id="6540" PostTypeId="1" AcceptedAnswerId="6555" CreationDate="2011-01-25T19:34:09.613" Score="1" ViewCount="190" Body="&lt;p&gt;Let say I have this kind of data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1 01/1/1980&#xA;2 01/2/1999&#xA;3 03/12/2000&#xA;-1 03/6/2005&#xA;-5 07/07/2007&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;how can I calculate the Present Value (PV) to them in respect to current date, let say with 5% interest rate, in spreadsheet? &lt;code&gt;Current date&lt;/code&gt; means today.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;[Update]&lt;/strong&gt; I am probably misunderstanding terms &lt;code&gt;PV&lt;/code&gt; and &lt;code&gt;FV&lt;/code&gt;. I am trying to find some ready function similar to &lt;a href=&quot;http://docs.google.com/support/bin/answer.py?hl=en&amp;amp;answer=155151&quot; rel=&quot;nofollow&quot;&gt;this one&lt;/a&gt;. Whatever method you use it must work with the above data. Please, stop spam.&lt;/p&gt;&#xA;" OwnerUserId="2914" LastEditorUserId="2914" LastEditorDisplayName="" LastEditDate="2011-01-28T21:13:35.683" LastActivityDate="2011-01-28T21:13:35.683" Title="How do calculate Present Value in Google Spreadsheet?" Tags="&lt;econometrics&gt;&lt;google-spreadsheet&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="1" />
  <row Id="6541" PostTypeId="2" ParentId="6538" CreationDate="2011-01-25T19:38:57.673" Score="1" ViewCount="" Body="&lt;p&gt;I come from a computer science background focusing on machine learning. &#xA;However, I really started to understand (and more important to apply) statistics after taking a Pattern Recognition course using Bishop's Book&#xA;&lt;a href=&quot;http://research.microsoft.com/en-us/um/people/cmbishop/prml/&quot; rel=&quot;nofollow&quot;&gt;http://research.microsoft.com/en-us/um/people/cmbishop/prml/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;here are some course slides from MIT:&lt;br&gt;&#xA;&lt;a href=&quot;http://www.ai.mit.edu/courses/6.867-f03/lectures.html&quot; rel=&quot;nofollow&quot;&gt;http://www.ai.mit.edu/courses/6.867-f03/lectures.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This will just give you the background (+ some matlab code) to use statistics for real work problems and is definitely more on the applied side. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Yet, it highly depends on what you want to do with your knowledge.&#xA;To get a measure for how good you are you might want to browse the open course ware of some university for advanced statistics courses, to check if you know the topics covered.&#xA;Just my 5 cent.&lt;/p&gt;&#xA;" OwnerUserId="2904" LastActivityDate="2011-01-25T19:38:57.673" />
  <row Id="6542" PostTypeId="2" ParentId="6538" CreationDate="2011-01-25T19:44:38.850" Score="1" ViewCount="" Body="&lt;p&gt;E.T. Jaynes &quot;Probability Theory: The Logic of Science: Principles and Elementary Applications Vol 1&quot;, Cambridge University Press, 2003 is pretty much a must read for the Bayesian side of statistics, at about the right level.  I'm looking forward to reccomendations for the frequentist side of things (I have loads of monographs, but very few good general texts).&lt;/p&gt;&#xA;" OwnerUserId="887" LastEditorUserId="887" LastEditorDisplayName="" LastEditDate="2011-01-25T19:54:09.400" LastActivityDate="2011-01-25T19:54:09.400" CommentCount="6" />
  <row Id="6543" PostTypeId="2" ParentId="6538" CreationDate="2011-01-25T19:48:44.050" Score="4" ViewCount="" Body="&lt;p&gt;I can't speak for the more rigorous schools, but I am doing a B.S. in General Statistics (the most rigorous at my school) at University of California, Davis, and there is a fairly heavy amount of reliance on rigor and derivation. A doctorate in math will be helpful, insomuch as you will have a very strong background in real analysis and linear algebra--useful skills in statistics. My statistics program has about 50% of the coursework going to support the fundamentals (linear algebra, real analysis, calculus, probability, estimation), and the other 50% goes towards specialized topics that rely on the fundamentals (nonparametrics, computation, ANOVA/Regression, time series, Bayesian analysis).&lt;br&gt;&#xA;Once you get the fundamentals, jumping to the specifics is usually not too difficult. Most of the individuals in my classes struggle with the proofs and real analysis, and easily grasp the statistical concepts, so coming from a math background will most definitely help. That being said, the following two texts have pretty good coverage of many topics covered in statistics. Both were recommended in the link you provided, by the way, so I wouldn't say your question and the one you linked are necessarily uncorrelated. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0691080046&quot; rel=&quot;nofollow&quot;&gt;Mathematical Methods of Statistics&lt;/a&gt;, by Harald Cramer  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1441923225&quot; rel=&quot;nofollow&quot;&gt;All of Statistics: A Concise Course in Statistical Inference&lt;/a&gt;, by Larry Wasserman&lt;/p&gt;&#xA;" OwnerUserId="1118" LastEditorUserId="1118" LastEditorDisplayName="" LastEditDate="2011-01-25T22:18:51.977" LastActivityDate="2011-01-25T22:18:51.977" CommentCount="2" />
  <row Id="6544" PostTypeId="1" AcceptedAnswerId="6784" CreationDate="2011-01-25T19:56:10.017" Score="4" ViewCount="365" Body="&lt;p&gt;I have a dataset that I want to fit a simple linear model to, but I want to include the lag of the dependent variable as one of the regressors. Then I want to predict future values of this time series using forecasts I already have for the independent variables. The catch is: how do I incorporate the lag into my forecast?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's an example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#A function to calculate lags&#xA;lagmatrix &amp;lt;- function(x,max.lag){embed(c(rep(NA,max.lag),x),max.lag)}&#xA;lag &amp;lt;- function(x,lag) {&#xA; out&amp;lt;-lagmatrix(x,lag+1)[,lag]&#xA; return(out[1:length(out)-1])&#xA;}&#xA;&#xA;y&amp;lt;-arima.sim(model=list(ar=c(.9)),n=1000) #Create AR(1) dependant variable&#xA;A&amp;lt;-rnorm(1000) #Create independant variables&#xA;B&amp;lt;-rnorm(1000)&#xA;C&amp;lt;-rnorm(1000)&#xA;Error&amp;lt;-rnorm(1000)&#xA;y&amp;lt;-y+.5*A+.2*B-.3*C+.1*Error #Add relationship to independant variables &#xA;&#xA;#Fit linear model&#xA;lag1&amp;lt;-lag(y,1)&#xA;model&amp;lt;-lm(y~A+B+C+lag1)&#xA;summary(model)&#xA;&#xA;#Forecast linear model&#xA;A&amp;lt;-rnorm(50) #Assume we know 50 future values of A, B, C&#xA;B&amp;lt;-rnorm(50)&#xA;C&amp;lt;-rnorm(50)&#xA;lag1&amp;lt;-  #################This is where I'm stuck##################&#xA;&#xA;newdata&amp;lt;-as.data.frame(cbind(A,B,C,lag1))&#xA;predict.lm(model,newdata=newdata)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2817" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-01-26T09:11:58.750" LastActivityDate="2011-02-01T16:19:03.493" Title="Predicting from a simple linear model with lags in R" Tags="&lt;r&gt;&lt;time-series&gt;&lt;regression&gt;&lt;predictive-models&gt;" AnswerCount="3" FavoriteCount="4" />
  <row Id="6545" PostTypeId="2" ParentId="725" CreationDate="2011-01-25T20:40:17.520" Score="1" ViewCount="" Body="&lt;p&gt;Hi all,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a very late response, so this may no longer be of interest, but I am working on putting together an R library with various hyperspectral image processing capabilities. At the moment my focus has been on endmember detection and unmixing. If this is still something which is of interest please let me know. My hope is to publish a beta version to CRAN or R-Forge in the near future but I would be happy to send out the code itself.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Best,&#xA;Dan&lt;/p&gt;&#xA;" OwnerUserId="2915" LastActivityDate="2011-01-25T20:40:17.520" CommentCount="1" />
  <row Id="6548" PostTypeId="2" ParentId="6499" CreationDate="2011-01-25T21:10:34.973" Score="6" ViewCount="" Body="&lt;p&gt;Assume that the random variables $x_k$ are i.i.d., nonnegative, integer valued, bounded by $n$, and such that $P(x_k=0)$ and $P(x_k=1)$ are both positive. For every $N\ge1$, let&#xA;$$&#xD;&#xA;X_N= \min\{x_1,\ldots,x_N\}.&#xD;&#xA;$$ &#xA;Then, when $N\to+\infty$,&#xA;$$&#xD;&#xA;E(X_N)=c^N(1+o(1)),&#xD;&#xA;$$ &#xA;where $c&amp;lt;1$ is independent of $N$ and given by &#xA;$$&#xD;&#xA;c=P(x_k\ge1).&#xD;&#xA;$$&#xA;Hence $E(X_N)$ is exponentially small. When each $x_k$ is Binomial $(n,p)$ with $n\ge1$ and $p$ in $(0,1)$ fixed, the result holds with $c=1-(1-p)^n$. &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;To see this, note that $[X_N\ge i]=[x_1\ge i]\cap\cdots\cap[x_N\ge i]$ for every $i$ and that, since $X_N$ is nonnegative and integer valued, $E(X_N)$ is the sum over $i\ge1$ of $P(X_N\ge i)$, hence&#xA;$$&#xD;&#xA;E(X_N)=\sum_{i\ge 1}P(x_1\ge i)^N.&#xD;&#xA;$$&#xA;For every $i\ge n+1$, $P(x_1\ge i)=0$. For every $2\le i\le n$, $0\le P(x_1\ge i)\le P(x_1\ge 2)$. Hence&#xA;$$&#xD;&#xA;c^N\le E(X_N)\le c^N+(n-1)d^N,&#xD;&#xA;$$ &#xA;with&#xA;$$&#xD;&#xA;c=P(x_1\ge1),\quad d=P(x_1\ge 2).&#xD;&#xA;$$&#xA;Because $P(x_k=1)$ is positive, one knows that $d&amp;lt;c$, hence $E(X_N)\sim c^N$ when $N\to+\infty$.&lt;/p&gt;&#xA;" OwnerUserId="2592" LastEditorUserId="2592" LastEditorDisplayName="" LastEditDate="2011-01-27T20:44:19.810" LastActivityDate="2011-01-27T20:44:19.810" CommentCount="5" />
  <row Id="6549" PostTypeId="1" AcceptedAnswerId="6551" CreationDate="2011-01-25T22:33:24.837" Score="1" ViewCount="180" Body="&lt;p&gt;Given a univariate sample $\vec X = X_1, ..., X_n$ with standard deviation 1 and a strictly monotone transformation $t: R \to R$ with the property that the standard deviation of $t(\vec X)$ is also 1 (where $t(\vec X)$ is $t$ applied to each $X_i$). If one know fits a normal distribution to $\vec X$ and $t(\vec X)$, one observes the following fact: the likelihood at the respective MLEs of $\vec X$ and $t(\vec X)$ is the same. The reason for that is, that the standard deviation is the MLE for the $\sigma$ parameter of the normal distribution, so just the mean changes, which does not change the likelihood.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In some computations with &lt;em&gt;multivariate&lt;/em&gt; data I observed the same fact, using transformations such that $\det(cov(t(\vec X)))=1)$. But I did not see a straightforward proof for that! Note that in the multivariate case, the MLEs for the variance parameters do change, even if the above determinant is always 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am sure I am not the first to notice, but my (somewhat limited) literature on the multivariate normal distribution does not give me a clue how to prove the above statement, i.e. that the multivariate normal likelihood does only depend on the determinant of the $cov(t(\vec X))$, regardless of choice of $t$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A proof boils down to looking at the terms of the form $x&amp;#39;\Sigma x$ for $\det \Sigma  =  1$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you have more clue?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks, Philipp&lt;/p&gt;&#xA;" OwnerUserId="2916" LastActivityDate="2011-01-27T03:32:30.250" Title="Likelihood at MLE and transformations, the multivariate normal case" Tags="&lt;normal-distribution&gt;&lt;multivariate-analysis&gt;&lt;maximum-likelihood&gt;" AnswerCount="1" />
  <row Id="6550" PostTypeId="1" AcceptedAnswerId="6553" CreationDate="2011-01-25T23:01:12.220" Score="3" ViewCount="138" Body="&lt;p&gt;I have about 500 text documents stored in a directory.  I want to randomly select one to read in the contents (this process will be repeated several times).  The names of the files are not sequential.  By this I mean they are not named something like 1.txt, 2.txt, 3.txt,...   The names are basically random characters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I do this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="2310" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-01-26T09:11:37.347" LastActivityDate="2011-01-26T09:11:37.347" Title="Reading random text file in R" Tags="&lt;r&gt;&lt;dataset&gt;" AnswerCount="2" />
  <row Id="6551" PostTypeId="2" ParentId="6549" CreationDate="2011-01-25T23:39:15.153" Score="0" ViewCount="" Body="&lt;p&gt;This is due to the &lt;em&gt;invariance&lt;/em&gt; property of MLEs.  It is an elementary exercise in calculus to show that if $E_{MLE}$ maximises the function $F(E)$ then &lt;em&gt;any monotonic function&lt;/em&gt; of $E$, say $g=g(E)$ will also be maximised at $g_{MLE}=g(E_{MLE})$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$F(E)=F[g^{-1}(E)]$ and therefore the maximum occurs at $F_{MAX}=F(E_{MLE})=F[g^{-1}(E_{MLE})]$, which means that $g_{MLE}=g(E_{MLE})$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If $g(.)$ were not monotone (1-to-1 so that $g^{-1}(.)$ is well defined), then the above argument would not hold.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: I think this is why the MLE's change for the determinant transformation, perhaps the $t(.)$ function is not a  1-to-1 function.  It is all about the functional properties of your $t(.)$ function (whether it is 1-to-1 or not).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT#2: After some discussion, the transformation $t(.)$ may be 1-to-1, but if the inverse transformation $t^{-1}(.)$ doesn't have a Jacobian of 1, then the value of the likelihood at its maximum will be different.  Fixing the variance to be 1 may one way of achieving this property.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The class of transformations $t(X)$ has a approximate (exact for normals?) covariance of $$cov[t(X)]\approx (\frac{\partial t(x)}{\partial t})_{x=X_{MLE}}^T cov[X](\frac{\partial t(x)}{\partial t})_{x=X_{MLE}}$$.&lt;br&gt;&#xA;Fixing the determinant to 1 fixes (approximately) &#xA;$$|\frac{\partial t(x)}{\partial t}|^{-2}_{x=X_{MLE}}\approx |cov[X]|$$&#xA;Now if $t(.)$ is invertible, the jacobian of the inverse transformation is just the inverse of the jacobian.  SO we have:&#xA;$$|\frac{\partial t(x)}{\partial t}|^{-1}_{x=X_{MLE}}=|\frac{\partial t^{-1}(x)}{\partial t}|_{x=X_{MLE}}\approx |cov[X]|^{\frac{1}{2}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now in the normal distribution, the likelihood evaluated at the MLE, the exponential part just becomes 1.  So the maximised likelihood  for X is given by&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$f_X(x_{MLE}) = \frac{1}{(2\pi)^{\frac{N}{2}} |cov[X]|^{\frac{1}{2}}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;using the Jacobian rule for transforming to $T=t(X)$ gives&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$f_T(t_{MLE}) = \frac{|\frac{\partial t^{-1}(x)}{\partial t}|_{x=X_{MLE}}}{(2\pi)^{\frac{N}{2}} |cov(X)|} \approx\frac{|cov[X]|^{\frac{1}{2}}}{(2\pi)^{\frac{N}{2}} |cov[X]|^{\frac{1}{2}}} = (2\pi)^{-\frac{N}{2}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which is a constant, as your analysis shows (you could even check my calculations, by comparing the likelihood you get with $(2\pi)^{-\frac{N}{2}}$, and see if it is right).  I would imagine that either the approximation is exact for normals, or exact for certain transformations $t(.)$, or becomes exact as $N\rightarrow\infty$, or the approximation error is in decimal places which you checked (or can't calculate), OR the computer you use to get the MLEs of $T$ uses the same approximation technique as I have (called the delta method).&lt;/p&gt;&#xA;" OwnerUserId="2392" LastEditorUserId="2392" LastEditorDisplayName="" LastEditDate="2011-01-27T03:32:30.250" LastActivityDate="2011-01-27T03:32:30.250" CommentCount="5" />
  <row Id="6552" PostTypeId="2" ParentId="6550" CreationDate="2011-01-25T23:39:45.370" Score="2" ViewCount="" Body="&lt;p&gt;Do a list.files and then access each file by its index.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; lisfil &amp;lt;- list.files(&quot;E:\\Data&quot;) #Replace E:\\Data with your directory name&#xA; lenlisfil &amp;lt;- length(lisfil)&#xA; for (i in 1:lenlisfil) {&#xA;    #Do something to the current file such as:&#xA;    print(lisfil[i])&#xA; }&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2775" LastActivityDate="2011-01-25T23:39:45.370" CommentCount="3" />
  <row Id="6553" PostTypeId="2" ParentId="6550" CreationDate="2011-01-25T23:48:34.877" Score="5" ViewCount="" Body="&lt;p&gt;First, select one file randomly from the current directory:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;file &amp;lt;- sample(list.files(), 1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then do whatever you want  with it, e.g. read all lines:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;readLines(file)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you want to repeat this several times, you could do it in one step:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;readLines(sample(list.files(), 1))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2714" LastActivityDate="2011-01-25T23:48:34.877" />
  <row Id="6554" PostTypeId="1" AcceptedAnswerId="6573" CreationDate="2011-01-25T23:48:47.747" Score="5" ViewCount="88" Body="&lt;p&gt;If I have a parameter with a prior $X\sim N(\mu,\sigma)$ and three data points, $x_1,x_2,x_3$, how can I calculate the posterior in R or BUGS?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an attempt at doing this in BUGS, that I have gotten stuck on:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data:&#xA;list(x=1.4, 2.1, 1.1, mu = 10, tau=1/10)&#xA;&#xA;model {&#xA;  for (i in 1:3) { &#xA;      x[i] ~ dnorm(mu, tau)&#xA;  }&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Any help appreciated&lt;/p&gt;&#xA;" OwnerUserId="2750" LastActivityDate="2011-01-26T15:45:58.250" Title="Getting started on BUGS, need help implementing a simple model" Tags="&lt;r&gt;&lt;bayesian&gt;&lt;bugs&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="6555" PostTypeId="2" ParentId="6540" CreationDate="2011-01-26T00:05:52.667" Score="2" ViewCount="" Body="&lt;p&gt;This sounds more like you want the &lt;em&gt;Future Value&lt;/em&gt;.  The &lt;em&gt;Present Value&lt;/em&gt; is usually defined by valuing cash flows which occur &lt;em&gt;in the future&lt;/em&gt;, not in the past.  But it amounts to the same equations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyways, you multiply the values in each period by $(1+r)^{t_0-t}$ and just add the results for your PV.  $r$ is your interest rate (with annual compounding), $t$ is the actual time of the cash flow (measured in years), and $t_0$ is where the &lt;em&gt;Present&lt;/em&gt; is (also measured in years).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So to update from 01/01/1980 to 01/01/2011 is just $1\times (1+0.05)^{2011-1980}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now if you want to get pedantic and include the &lt;em&gt;days&lt;/em&gt; in your calculations, you just need to count the &quot;left over&quot; days and divide them by $365$ (or $365.24$ if you've gone insane with details), and put this as a decimal or fraction addition to $t$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So for updating from 03/06/2005 to 01/01/2011 we have $31+28+31+30+31+3=154$ days between 01/01/2005 and 03/06/2005 so this gives an update of $-1\times (1+0.05)^{2011-(2005+\frac{154}{365})}$&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-01-26T00:05:52.667" CommentCount="3" />
  <row Id="6556" PostTypeId="2" ParentId="6524" CreationDate="2011-01-26T00:41:37.533" Score="0" ViewCount="" Body="&lt;p&gt;The &quot;failsafe&quot; way to find a sufficient statistic in just about &lt;em&gt;any&lt;/em&gt; problem:  Calculate the Bayesian posterior distribution (up to prop constant) using a uniform prior.  If a sufficient statistic exists, this method will tell you what it is.  So basically, strip all multiplicative factors (those factors which do not involve your &lt;em&gt;parameters&lt;/em&gt;, but may involve functions of the data) from your likelihood function.  I would suggest that the &quot;sum function&quot; stuff is your sufficient statistic (although the notation was not clear at the time I answered this question).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;NOTE: the use of a uniform prior may make the posterior &lt;em&gt;improper&lt;/em&gt; but it will show you what the functions are sufficient for your problem.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastEditorUserId="2392" LastEditorDisplayName="" LastEditDate="2011-01-26T00:46:55.203" LastActivityDate="2011-01-26T00:46:55.203" CommentCount="7" />
  <row Id="6557" PostTypeId="1" AcceptedAnswerId="6615" CreationDate="2011-01-26T01:16:56.443" Score="4" ViewCount="244" Body="&lt;p&gt;Let's say I have data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x1 &amp;lt;- rnorm(100,2,10)&#xA;x2 &amp;lt;- rnorm(100,2,10)&#xA;y &amp;lt;- x1+x2+x1*x2+rnorm(100,1,2)&#xA;dat &amp;lt;- data.frame(y=y,x1=x1,x2=x2)&#xA;res &amp;lt;- lm(y~x1*x2,data=dat)&#xA;summary(res)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to plot the continuous by continuous interaction such that x1 is on the X axis and x2 is represented by 3 lines, one which represents x2 at a Z-score of 0, one at Z-score of +1, and another at a Z-score of -1, with each line a separate color and labelled.  How can I do this using ggplot2?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, it might look something like this (though of course with different colored lines rather than different line types):&#xA;&lt;img src=&quot;http://i.imgur.com/eUZJG.png&quot; alt=&quot;Example Image&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="196" LastEditorUserId="196" LastEditorDisplayName="" LastEditDate="2011-01-26T01:48:30.457" LastActivityDate="2011-01-27T11:45:40.877" Title="How can one plot continuous by continuous interactions in ggplot2?" Tags="&lt;r&gt;&lt;regression&gt;&lt;ggplot2&gt;&lt;interaction&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="2" />
  <row Id="6558" PostTypeId="2" ParentId="6513" CreationDate="2011-01-26T01:18:49.687" Score="0" ViewCount="" Body="&lt;p&gt;@Grega: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I ran out of room in the comment area, so I started another answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From your comment, that's the problem.  Each system/grid has its own &quot;signature&quot;.  It's a combination of system dynamics, age, weather, local economics, cultures, traditions, etc.  In Japan, its worse.  Some sections of the country run at 50Hz and other sections run at 60Hz.  These grids connect at high voltage DC stations, which means some areas behave like electrical &quot;islands&quot; (totally different behavior than their neighbors just a few miles away).  If you're zeroed-in on one factory, the predictability goes down even more.   Fewer users means higher uncertainty.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;No matter how you do this, it's going to be messy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would filter out the daily/weekday/weekend/holiday component to get a Seasonal Term.  How?  A 31 day CENTERED moving average?  51 day CMA?  XX day CMA?   You'll have to experiment with that, but I would make it a variable so you can tweek it later.  Whatever filter you end up with, keep in mind that it stops short of either end of your data (a 31 day centered moving average will start on the 16th day of your raw data and end on the 16th day from the end of your raw data).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Next, the best you can do with 11 months of data is to &quot;make up&quot; month 12 (draw a line from your filtered series at its end to its beginning).   Next, subtract your Seasonal Term (the filtered data) from your raw data to get a residual.   Fit the residual to your weather data, allowing for factors for day-of-the-week and holidays.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some factors that you may need to add are:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) A &quot;production run&quot; factor.   Whatever they make at this factory, if they switch from one product/category to another, the power demand required to make one product may be different than what is required for another product.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) A &quot;change over&quot; factor.   This is when they shift from one product/category to another.   Sometimes it takes days of preparation for the switch.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) A &quot;work shift&quot; factor.   If they have three shifts per day, power demand for the late shift will probably be significantly different than the day shifts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good luck.   As you probably know, this kind of a problem can get real frustrating.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;====== Edit to answer Grega's first comment (01/25/2011) ====================&lt;/p&gt;&#xA;&#xA;&lt;p&gt;@Grega: Answering your first comment, I'm afraid it doesn't.  The idea behind a model like this would be to have multiple &quot;similar instances&quot; of your 32 future points, so you can fit those points, and then predict new points.  You don't have typical &quot;similar instances&quot; because yesterday was not the same day-of-the-week as today.  You have to use last week's same day-of-the-week and the previous week's same day-of-the-week, etc.   By the time you get several &quot;similar instances&quot; (say 20), you're typically more than 20 weeks in the past (a holiday may screw up one or more of your weeks).  At that point, you're in a whole different season of the year.   So, in order to use those days in another season, you need to remove the Seasonal Term from the raw data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's a sloppy situation, but it's the best you can do with 11 months of data.&lt;/p&gt;&#xA;" OwnerUserId="2775" LastEditorUserId="2775" LastEditorDisplayName="" LastEditDate="2011-01-26T04:19:31.357" LastActivityDate="2011-01-26T04:19:31.357" CommentCount="5" />
  <row Id="6559" PostTypeId="2" ParentId="6544" CreationDate="2011-01-26T02:15:57.690" Score="0" ViewCount="" Body="&lt;p&gt;Ok, I answered my own problem, but my solution could use more testing and probably isn't perfect.  Suggestions would be appreciated!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First of all I used a modified version of the parseCall function available &lt;a href=&quot;http://rguha.net/code/R/parseCall.R&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;parseCall &amp;lt;- function(obj) {&#xA;    if (class(obj) != 'call') {&#xA;        stop(&quot;Must supply a 'call' object&quot;)&#xA;    }&#xA;&#xA;    srep &amp;lt;- deparse(obj)&#xA;    if (length(srep) &amp;gt;1) srep &amp;lt;- paste(srep,sep='',collapse='')&#xA;&#xA;    fname &amp;lt;- unlist(strsplit(srep,&quot;\\(&quot;))[1]&#xA;&#xA;    func &amp;lt;- unlist(strsplit(srep, paste(fname,&quot;\\(&quot;,sep='')))[2]&#xA;    func &amp;lt;- unlist(strsplit(func,&quot;&quot;))&#xA;    func &amp;lt;- paste(func[-length(func)],sep='',collapse=&quot;&quot;)&#xA;&#xA;    func &amp;lt;- unlist(strsplit(func,&quot;,&quot;))&#xA;&#xA;    vals &amp;lt;- list()&#xA;    nms &amp;lt;- c()&#xA;    cnt &amp;lt;- 1&#xA;    for (args in func) {&#xA;        arg &amp;lt;- unlist(strsplit(args,&quot;=&quot;))[1]&#xA;        val &amp;lt;- unlist(strsplit(args,&quot;=&quot;))[2]&#xA;&#xA;        arg &amp;lt;- gsub(&quot; &quot;, &quot;&quot;, arg)&#xA;        val &amp;lt;- gsub(&quot; &quot;, &quot;&quot;, val)&#xA;&#xA;        vals[[cnt]] &amp;lt;- val&#xA;        nms[cnt] &amp;lt;- arg&#xA;        cnt &amp;lt;- cnt + 1&#xA;    }&#xA;    names(vals) &amp;lt;- nms&#xA;    return(vals)&#xA;&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This function returns the dependent variable of a linear regression&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;getDepVar &amp;lt;- function(call) {&#xA;    call&amp;lt;-parseCall(call)&#xA;    formula&amp;lt;-call$formula&#xA;    out&amp;lt;-unlist(strsplit(formula,&quot;~&quot;)[1])&#xA;    return(out[1])&#xA;    }&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And finally, this function does the magic:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ipredict &amp;lt;-function(model,newdata) {&#xA;    Y&amp;lt;-getDepVar(model$call)&#xA;    P&amp;lt;-predict(model,newdata=newdata)&#xA;    for (i in seq(1,dim(newdata)[1])) {&#xA;        if (is.na(newdata[i,Y])) {&#xA;            newdata[i,Y]&amp;lt;-predict(model,newdata=newdata[1:i,])[i]&#xA;            P[i]&amp;lt;-newdata[i,Y]&#xA;        }&#xA;    }&#xA;    return(P)&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Example usage (based on my question):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#A function to calculate lags&#xA;lagmatrix &amp;lt;- function(x,max.lag){embed(c(rep(NA,max.lag),x),max.lag)}&#xA;lag &amp;lt;- function(x,lag) {&#xA;    out&amp;lt;-lagmatrix(x,lag+1)[,lag]&#xA;    return(out[1:length(out)-1])&#xA;}&#xA;&#xA;y&amp;lt;-arima.sim(model=list(ar=c(.9)),n=1000) #Create AR(1) dependant variable&#xA;A&amp;lt;-rnorm(1000) #Create independant variables&#xA;B&amp;lt;-rnorm(1000)&#xA;C&amp;lt;-rnorm(1000)&#xA;Error&amp;lt;-rnorm(10)&#xA;y&amp;lt;-y+.5*A+.2*B-.3*C+.1*Error #Add relationship to independant variables &#xA;&#xA;#Fit linear model&#xA;model&amp;lt;-lm(y~A+B+C+I(lag(y,1)))&#xA;summary(model)&#xA;&#xA;#Forecast linear model&#xA;A&amp;lt;-c(A,rnorm(50)) #Assume we know 50 future values of A, B, C&#xA;B&amp;lt;-c(B,rnorm(50))&#xA;C&amp;lt;-c(C,rnorm(50))&#xA;y&amp;lt;-c(y,rep(NA,50))&#xA;&#xA;newdata&amp;lt;-as.data.frame(cbind(y,A,B,C))&#xA;ipredict(model,newdata=newdata)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2817" LastActivityDate="2011-01-26T02:15:57.690" CommentCount="5" />
  <row Id="6560" PostTypeId="1" AcceptedAnswerId="6564" CreationDate="2011-01-26T02:28:03.963" Score="5" ViewCount="92" Body="&lt;p&gt;I'm looking for some help devising a hypothesis test for the following situation.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;I have a radioactive source that spits out a particle every so often.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Also, I have two particle detectors: a red particle detector, and a green particle detector.  Whenever the red particle detector detects a particle, it flashes a red light; let $R$ denote the event that the particle was detected by the red detector, and $r$ the complement event that the particle was not detected by the red detector.  Whenever the green particle detector detects a particle, it flashes a green light; let $G$ be the event that the green detector detects the particle, and $g$ that it does not.  Thus, each emitted particle falls into one of four categories:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;detected by both detectors ($RG$),&lt;/li&gt;&#xA;&lt;li&gt;detected by the red detector but not the green detector ($Rg$),&lt;/li&gt;&#xA;&lt;li&gt;detected by the green detector but not the red detector ($rG$), or&lt;/li&gt;&#xA;&lt;li&gt;not detected by either detector ($rg$).&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Each time a particle is emitted, the red detector has some probability of detecting the particle, and the green detector has some probability of detecting the particle.  (They will never trigger a false detection when no particle is present.)  I know that each particle is handled identically and independently of all other particles, but I don't know whether the two detectors are independent of each other.  It's possible that they are independent (i.e., $\Pr[RG] = \Pr[R] \Pr[G]$), or that they are correlated (i.e., $\Pr[RG] \ne \Pr[R] \Pr[G]$); I don't know which is the case, a priori.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I keep a count of the number of $RG$-detections (i.e., number of times when both detectors detected something), number of $Rg$-detections (i.e., number of times when the red detector detected something, but not the green one), and the number of $rG$-detections.  Unfortunately, I have no way to measure the number of $rg$-situations, since those particles aren't detected by either of the detector.  At the end of the experiment, I have three non-negative integers, representing these counts.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I want to test the hypothesis $H$ that the two detectors are independent, i.e., that event $R$ is independent of event $G$.  Can anyone help suggest a way to compute the $p$-value of this hypothesis, given 3 numbers from such an experiment?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd be perfectly satisfied with a computer algorithm/procedure to compute the $p$-value.  I don't need a simple formula; something that could be computed by a computer would be sufficient.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Here's another way to view this.  We could form a 2x2 contingency table, such as this one:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;     G |  g&#xA;  ---------&#xA;R | 17   22&#xA;r | 12    ?&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;recording that we saw 17 $RG$-events, 22 $Rg$-events, and so on.  Unfortunately, the lower-right cell is empty, since we don't know how many $rg$-particles were emitted.  If we had counts for all four cells, presumably we could use Fisher's exact test, but we don't.  Also, we aren't given $\Pr[R]$ or $\Pr[G]$ (I guess they are nuisance parameters) or the total number of particles emitted.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestions?&lt;/p&gt;&#xA;" OwnerUserId="2921" LastActivityDate="2011-01-27T23:42:20.253" Title="Test for independence, when I'm missing one bin of a 2x2 contingency" Tags="&lt;hypothesis-testing&gt;" AnswerCount="1" />
  <row Id="6561" PostTypeId="2" ParentId="6557" CreationDate="2011-01-26T02:28:45.770" Score="2" ViewCount="" Body="&lt;p&gt;Computing the estimates for y with Z-score of 0 (&lt;em&gt;y0&lt;/em&gt; column), -1 (&lt;em&gt;y1m&lt;/em&gt; column) and 1 (&lt;em&gt;y1p&lt;/em&gt; column):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dat$y0 &amp;lt;- res$coefficients[[1]] + res$coefficients[[2]]*dat$x1 + res$coefficients[[3]]*0 + res$coefficients[[4]]*dat$x1*0&#xA;dat$y1m &amp;lt;- res$coefficients[[1]] + res$coefficients[[2]]*dat$x1 + res$coefficients[[3]]*-1 + res$coefficients[[4]]*dat$x1*-1&#xA;dat$y1p &amp;lt;- res$coefficients[[1]] + res$coefficients[[2]]*dat$x1 + res$coefficients[[3]]*1 + res$coefficients[[4]]*dat$x1*1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Plotting the lines with base &lt;code&gt;plot()&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plot(dat$x1, dat$y0, type=&quot;l&quot;, xlab=&quot;x1&quot;, ylab=&quot;Estimates&quot;)&#xA;lines(dat$x1, dat$y1m, col=&quot;red&quot;)&#xA;lines(dat$x1, dat$y1p, col=&quot;blue&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/tGsxq.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To use ggplot, you may call &lt;a href=&quot;http://had.co.nz/ggplot2/geom_abline.html&quot; rel=&quot;nofollow&quot;&gt;geom_line&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ggplot(dat, aes(x1, y0)) + geom_line() +&#xA;    geom_line(aes(x1, y1m), color=&quot;red&quot;) +&#xA;    geom_line(aes(x1, y1p), color=&quot;blue&quot;) +&#xA;    theme_bw() + opts(title=&quot;&quot;) + xlab(&quot;x1&quot;) + ylab(&quot;Estimates&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/OWPWg.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="2714" LastActivityDate="2011-01-26T02:28:45.770" CommentCount="3" />
  <row Id="6562" PostTypeId="1" AcceptedAnswerId="6566" CreationDate="2011-01-26T02:41:40.690" Score="3" ViewCount="146" Body="&lt;p&gt;I have read an &lt;a href=&quot;http://nlp.stanford.edu/~manning/courses/ling289/logistic.pdf&quot; rel=&quot;nofollow&quot;&gt;article&lt;/a&gt; from Christopher Manning, and saw an interesting code for collapsing categorical variable in an logistic regression model:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;glm(ced.del ~ cat + follows + I(class == 1), family=binomial(&quot;logit&quot;))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Does the &lt;code&gt;I(class == 1)&lt;/code&gt; means that the &lt;code&gt;class&lt;/code&gt; variable has been recoded into either it is &lt;code&gt;1&lt;/code&gt; or it is not &lt;code&gt;1&lt;/code&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After that, I am thinking of modifying it a bit:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;glm(ced.del ~ cat + follows + I(class %in% C(1,2)), family=binomial(&quot;logit&quot;))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am planning to merge the variable &lt;code&gt;class&lt;/code&gt; from &lt;code&gt;c(1,2,3,4)&lt;/code&gt; into two groups, one group contains &lt;code&gt;c(1,2)&lt;/code&gt;, another group contains &lt;code&gt;c(3,4)&lt;/code&gt;, can the code above give me the result I want?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="588" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-01-26T09:10:28.580" LastActivityDate="2011-01-26T09:10:28.580" Title="Collapsing categorical data easily for regression in R" Tags="&lt;r&gt;&lt;logistic-regression&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="6563" PostTypeId="1" CreationDate="2011-01-26T02:53:22.350" Score="4" ViewCount="110" Body="&lt;p&gt;There is one variable in my data have 80% of missing data. The data is missing because of non-existence (i.e. how much bank loan the company owes). I came across an article saying that dummy variable adjustment method is the solution for this problem. Meaning that I need to transform this continuous variable to categorical? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this the only solution? I do not want to drop this variable as I think theoretically, it is important to my research question.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks. &lt;/p&gt;&#xA;" OwnerUserId="2793" LastActivityDate="2011-01-26T03:34:37.983" Title="80% of missing data in a single variable" Tags="&lt;missing-data&gt;" AnswerCount="1" />
  <row Id="6564" PostTypeId="2" ParentId="6560" CreationDate="2011-01-26T03:22:22.153" Score="7" ViewCount="" Body="&lt;p&gt;If the missing count were close to 22*12/17, the table would appear independent.  This is consistent with your observations.  If the missing count is far from this value, the table would exhibit strong lack of dependence.  This, too, is consistent with your observations.  Evidently your data cannot discriminate the two cases: independence or lack thereof are &lt;em&gt;unidentifiable&lt;/em&gt;.  Therefore, your only hope is to adopt additional assumptions, such as a prior for the missing count (equivalently, for the total number of emitted particles).&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-01-26T03:22:22.153" CommentCount="2" />
  <row Id="6565" PostTypeId="2" ParentId="6563" CreationDate="2011-01-26T03:27:01.367" Score="7" ViewCount="" Body="&lt;p&gt;Are the data &quot;missing&quot; in the sense of being &lt;em&gt;unknown&lt;/em&gt; or does it just mean there is no loan (so the loan amount is zero)?  It sounds like the latter, in which case you need an &lt;em&gt;additional&lt;/em&gt; binary dummy to indicate whether there is a loan.  No transformation of the loan amount is needed (apart, perhaps, from a continuous re-expression, such as a root or started log, which might be indicated by virtue of other considerations).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This works well in a regression.  A simple example is a conceptual model of the form&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\text{dependent variable (Y) = loan amount (X) + constant.}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With the addition of a loan indicator ($I$), the regression model is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Y = \beta_I I + \beta_X X + \beta_0 + \epsilon$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;with $\epsilon$ representing random errors with zero expectations.  The coefficients are interpreted as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\beta_0$ is the expectation of $Y$ for no-loan situations, because those are characterized by $X = 0$ and $I = 0$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\beta_X$ is the marginal change in $Y$ with respect to the amount of the loan ($X$).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\beta_I + \beta_0$ is the intercept for the cases with loans.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-01-26T03:34:37.983" LastActivityDate="2011-01-26T03:34:37.983" CommentCount="5" />
  <row Id="6566" PostTypeId="2" ParentId="6562" CreationDate="2011-01-26T09:07:10.353" Score="4" ViewCount="" Body="&lt;p&gt;This shows what the first is doing:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; class &amp;lt;- sample(1:5, 10, replace = TRUE)&#xA;&amp;gt; class == 1&#xA; [1]  TRUE FALSE FALSE FALSE  TRUE  TRUE  TRUE FALSE  TRUE  TRUE&#xA;&amp;gt; as.numeric(class == 1)&#xA; [1] 1 0 0 0 1 1 1 0 1 1&#xA;&amp;gt; ## or better,&#xA;&amp;gt; factor(as.numeric(class == 1))&#xA; [1] 1 0 0 0 1 1 1 0 1 1&#xA;Levels: 0 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;class == 1&lt;/code&gt; is creating an indicator variable as you surmise. The &lt;code&gt;I(....)&lt;/code&gt; bit is due to us having to insulate certain operations from R's formula parsing code.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Similarly, &lt;code&gt;class %in% c(1,2)&lt;/code&gt; is doing the same thing but matching on &lt;code&gt;1&lt;/code&gt; &lt;em&gt;or&lt;/em&gt; &lt;code&gt;2&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; class %in% c(1,2)&#xA; [1]  TRUE FALSE FALSE  TRUE  TRUE  TRUE  TRUE FALSE  TRUE  TRUE&#xA;&amp;gt; as.numeric(class %in% c(1,2))&#xA; [1] 1 0 0 1 1 1 1 0 1 1&#xA;&amp;gt; ## etc&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and it works within the modelling calls:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; set.seed(1)&#xA;&amp;gt; dat &amp;lt;- data.frame(Y = rnorm(100), X1 = rnorm(100), &#xA;+                   class = sample(1:5, 100, replace = TRUE))&#xA;&amp;gt; mod &amp;lt;- lm(Y ~ X1 + I(class %in% c(1,4)), data = dat)&#xA;&amp;gt; coef(mod)&#xA;              (Intercept)                        X1 I(class %in% c(1, 4))TRUE &#xA;                  0.138287085              -0.003827624              -0.072059592 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;but I think it is neater if you create a new variable for the class compression you want and use that directly in your model:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; dat &amp;lt;- within(dat, class2 &amp;lt;- factor(class %in% c(1, 4),&#xA;+                                     levels = c(FALSE,TRUE)))&#xA;&amp;gt; mod2 &amp;lt;- lm(Y ~ X1 + class2, data = dat)&#xA;&amp;gt; coef(mod2)&#xA; (Intercept)           X1   class2TRUE &#xA; 0.138287085 -0.003827624 -0.072059592&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1390" LastActivityDate="2011-01-26T09:07:10.353" />
  <row Id="6567" PostTypeId="2" ParentId="4466" CreationDate="2011-01-26T10:02:07.550" Score="3" ViewCount="" Body="&lt;p&gt;I would recomend two things in addition to the excellent answers already present;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;At Key points in your code, dump out the current data as a flat file, suitably named and described in comments, thus highlighting if one package has produced differing results where the differences have been introduced.  These data files, as well as the original input and the resulting output should be included in your 'reproducible research set'&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Include some testing of the packages concerned within your code, for instance using something like &lt;a href=&quot;http://cran.r-project.org/web/packages/testthat/index.html&quot; rel=&quot;nofollow&quot;&gt;TestThat&lt;/a&gt;.  The hard part is making small, reproducible tests that are likely to highlight any changes in what a package does &lt;em&gt;that relate&lt;/em&gt; to your analysis.  This would at least highlight to another person that there is some difference in the environments.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="114" LastActivityDate="2011-01-26T10:02:07.550" />
  <row Id="6568" PostTypeId="2" ParentId="6176" CreationDate="2011-01-26T11:09:56.750" Score="4" ViewCount="" Body="&lt;p&gt;For a &lt;em&gt;really short&lt;/em&gt; introduction (seven page pdf), there's also this, intended to allow you to follow papers that use a bit of measure theory :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;&lt;a href=&quot;http://www.ee.washington.edu/research/guptalab/publications/MeasureTheoryTutorial.pdf&quot; rel=&quot;nofollow&quot;&gt;A Measure Theory Tutorial (Measure Theory for Dummies)&lt;/a&gt;&lt;/em&gt;. Maya R. Gupta.&#xA;Dept of Electrical Engineering, University of Washington, 2006.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;He gives some refs at the end and says &quot;one of the friendliest books is Resnick’s, which teaches measure theoretic graduate level probability with the assumption that you do not have a B.A. in mathematics.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;S. I. Resnick, &lt;em&gt;&lt;a href=&quot;http://books.google.co.uk/books?id=T9-PMDSVDNsC&quot; rel=&quot;nofollow&quot;&gt;A probability path&lt;/a&gt;&lt;/em&gt;, Birkhäuser, 1999. 453 pages.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-01-26T11:09:56.750" CommentCount="1" />
  <row Id="6570" PostTypeId="1" AcceptedAnswerId="6576" CreationDate="2011-01-26T13:24:23.333" Score="4" ViewCount="138" Body="&lt;p&gt;I want to conduct a poll on the quality of a product containing several questions with five possible answers:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Very poor&lt;/li&gt;&#xA;&lt;li&gt;Poor&lt;/li&gt;&#xA;&lt;li&gt;OK / No opinion&lt;/li&gt;&#xA;&lt;li&gt;Good&lt;/li&gt;&#xA;&lt;li&gt;Very good&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;A colleague has advised me to ditch option 3 (OK / No opinion) to force people to choose.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which will produce the most reliable / useful data? Is there a preferred option or is it dependent on other factors (if so, what)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand that Gallup usually has five options, hence why I choose five.&lt;/p&gt;&#xA;" OwnerUserId="1259" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-01-26T15:12:29.600" LastActivityDate="2011-01-27T16:04:35.303" Title="Should a multiple choice poll contain a neutral response?" Tags="&lt;polling&gt;" AnswerCount="5" CommentCount="3" />
  <row Id="6571" PostTypeId="1" CreationDate="2011-01-26T14:02:10.713" Score="2" ViewCount="106" Body="&lt;p&gt;Dear all,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let us say we have some demographic time series data which tells us how many hours people spend in front of a computer screen each day, grouped by age and gender:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(42)&#xA;dates = seq(as.Date(&quot;2011/1/1&quot;), by=&quot;day&quot;, length.out = 100)&#xA;male.age1 = round(runif(100, min = 1, max = 10))&#xA;female.age1 = round(runif(100, min = 1, max = 10))&#xA;male.age2 = round(runif(100, min = 1, max = 10))&#xA;female.age2 = round(runif(100, min = 1, max = 10))&#xA;df = data.frame(dates = dates, male.age1 = male.age1, female.age1 = female.age1, male.age2 = male.age2, female.age2 = female.age2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; df[1:5,]&#xA;       dates male.age1 female.age1 male.age2 female.age2&#xA;1 2011-01-01         9           7         9           5&#xA;2 2011-01-02         9           3         6           5&#xA;3 2011-01-03         4           3         9           2&#xA;4 2011-01-04         8           5         5           4&#xA;5 2011-01-05         7           9         2           9&#xA;etc...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am just wondering what kind of analysis would be appropriate for this type of data, and if you could give me some examples? My initial thinking is trying to see how certain types of people act on certain dates, though I don't know how to go about that. I think what I'm finding hard is that the dates are not really independent of each other (i.e. what happens today might effect what happens tomorrow).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks in advance for your time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: This isn't homework or anything like that, just curiosity and trying to learn R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note 2: Is there any benefit in reshaping the data as follows (I came across the reshape package a few days ago, and it looks really cool):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(reshape)&#xA;library(reshape2)&#xA;y = melt(df, id = &quot;dates&quot;)&#xA;y$gender = NA&#xA;ind = grep(&quot;female&quot;, y$variable, fixed = TRUE)&#xA;y[ind, &quot;gender&quot;] = &quot;female&quot;&#xA;y[-ind, &quot;gender&quot;] = &quot;male&quot;&#xA;y$gender = factor(y$gender)&#xA;levels(y$variable) &amp;lt;- c(&quot;age1&quot;, &quot;age2&quot;, &quot;age3&quot;, &quot;age4&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Which gives:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; y[with(y, order(dates)), ][1:10,]&#xA;         dates variable value gender&#xA;1   2011-01-01     age1     9   male&#xA;101 2011-01-01     age2     7 female&#xA;201 2011-01-01     age3     9   male&#xA;301 2011-01-01     age4     5 female&#xA;2   2011-01-02     age1     9   male&#xA;102 2011-01-02     age2     3 female&#xA;202 2011-01-02     age3     6   male&#xA;302 2011-01-02     age4     5 female&#xA;3   2011-01-03     age1     4   male&#xA;103 2011-01-03     age2     3 female&#xA;    etc...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="539" LastEditorUserId="539" LastEditorDisplayName="" LastEditDate="2011-01-26T14:07:52.420" LastActivityDate="2011-01-26T19:19:10.297" Title="What types of analysis are appropriate for demographic time series data?" Tags="&lt;r&gt;&lt;time-series&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="6572" PostTypeId="2" ParentId="6570" CreationDate="2011-01-26T15:32:22.127" Score="0" ViewCount="" Body="&lt;p&gt;The fact that you are forcing the respondent for a positive answer or negative answer, in this situation, is not correct; the respondent may be undecided and it is a possibility; it is more when it is a new product. If you are developing an instrument to measure quality it is better that you use 5 options as given by you. &lt;/p&gt;&#xA;" OwnerUserId="2926" LastActivityDate="2011-01-26T15:32:22.127" CommentCount="2" />
  <row Id="6573" PostTypeId="2" ParentId="6554" CreationDate="2011-01-26T15:40:51.817" Score="3" ViewCount="" Body="&lt;p&gt;(Disclaimer: I am by no means very experienced in Bayesian stats or using BUGS!)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What parameters would you like to estimate? What does &quot;mu = 10, tau=1/10&quot; mean? Are these numbers considered to be priors? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let us assume your interested in estimating $\mu$ and $\sigma^2$. (Keep in mind that Win/OpenBUGS use the precision, i.e prec = 1/$\sigma^2$). So, your BUGS model could look like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model {&#xA;  for (i in 1:3) { &#xA;      x[i] ~ dnorm(mu, prec)&#xA;  }&#xA;&#xA;## priors (will have strong impact on the parameter estimation)  &#xA;prec ~ dgamma(0.1, 0.001)&#xA;mu ~ dnorm(2.0, 0.0001)&#xA;var &amp;lt;- 1/prec&#xA;}&#xA;&#xA;## data vector (BUGS follows the S/R notation, i.e. use the c() function)&#xA;list(x=c(1.4, 2.1, 1.1))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am using OpenBUGS (10,000 iterations, burn-in: 5000; starting values were generated by OpenBUGS) and here are my results:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;        mean    sd      MC_error    val2.5pc    median  val97.5pc   start   sample&#xA;mu      1.544   0.6666      0.01039     0.3894  1.535       2.728   5000    5002&#xA;prec    4.155   3.942       0.06772     0.1263  3.024       14.38   5000    5002&#xA;var     1.31    5.893       0.1191      0.06952 0.331       7.917   5000    5002&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You might realize that the variance estimator has been heavily affected by the choice of the prior.     &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As @whuber already mentioned, I strongly recommend that you check out the many examples that come with any of the BUGS packages. You also might be interested in &lt;a href=&quot;http://arcue.botany.unimelb.edu.au/bayescode.html&quot; rel=&quot;nofollow&quot;&gt;&quot;Bayesian Methods for Ecology&quot;&lt;/a&gt; or &lt;a href=&quot;http://stat-athens.aueb.gr/~jbn/winbugs_book/&quot; rel=&quot;nofollow&quot;&gt;&quot;Bayesian Modeling Using WinBUGS: An introduction&quot;&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="307" LastEditorUserId="307" LastEditorDisplayName="" LastEditDate="2011-01-26T15:45:58.250" LastActivityDate="2011-01-26T15:45:58.250" CommentCount="5" />
  <row Id="6575" PostTypeId="1" AcceptedAnswerId="6589" CreationDate="2011-01-26T15:51:26.317" Score="4" ViewCount="81" Body="&lt;p&gt;Consider the following picture representing the experimental data sequence obtained by two 1D-sensors (each point of the sequence is plotted on XY plane according to the respective sensor reading):&lt;br&gt;&#xA;&lt;img src=&quot;http://i.imgur.com/U6fWb.png&quot; alt=&quot;experimental raw data&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's visually obvious that two modes have been registered. Let's assume that generally those two modes interfere, so there's no easy possibility to separate them by isolating certain sequence segments. I try the classic principal component analysis by finding the covariance matrix, then finding the set of eigenvalues and corresponding eigenvectors:&lt;br&gt;&#xA;&lt;img src=&quot;http://i.imgur.com/osv6x.png&quot; alt=&quot;experimental data pca&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;White box dimensions represents the magnitude if the eigenvalues, box orientation represents the direction of eigenvectors.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's clear that PCA first component deviates slightly from the high-magnitude mode direction, while the second component deviates greatly due to skewness of the lower-magnitude mode original direction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is known that PCA, being based on eigenvectors, results in orthogonal basis of primary components. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there other elegant methods (or PCA-derived methods) to obtain the non-orthogonal basis of primary skewed components?&lt;/p&gt;&#xA;" OwnerUserId="2820" LastActivityDate="2011-01-26T20:30:43.377" Title="How to do primary component analysis on multi-mode data with non-orthogonal primary components?" Tags="&lt;pca&gt;&lt;multivariate-analysis&gt;&lt;mode&gt;" AnswerCount="1" />
  <row Id="6576" PostTypeId="2" ParentId="6570" CreationDate="2011-01-26T16:03:55.117" Score="5" ViewCount="" Body="&lt;p&gt;Neutral points can mean many different things to many people. The way you labeled the middle choice yourself reflects this uncertainty. Some reasons for choosing the neutral point from the perspective of a participant:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;I don't care to really think about my answer to this question (I just want to get paid and leave)&lt;/li&gt;&#xA;&lt;li&gt;I have no strong opinion on this question&lt;/li&gt;&#xA;&lt;li&gt;I don't understand the question, but don't want to ask (I just want to get paid and leave)&lt;/li&gt;&#xA;&lt;li&gt;with regards to the given aspect, the product is truly medium in quality, i.e., it neither excels nor falls short of my expectations&lt;/li&gt;&#xA;&lt;li&gt;with regards to the given aspect, the product has some high-quality features, and some low-quality features&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Without further qualification, the people who choose the middle category can thus represent a very heterogeneous collection of attitudes / cognitions. With good labeling, some of this confusion can be avoided.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also present a separate &quot;no answer&quot; category. However, participants often interpret such a category as a signal to only provide an answer if they feel very confident in their choice. In other words, participants then tend to choose &quot;no answer&quot; because they feel they're not well-informed enough to make a choice that meets the questionnaire-designers quality standards.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;IMHO there's no right answer to your question. You have to be very careful in labeling some or all of the presented choices, do lots of pre-testing with additional free interviews of participants on how they perceived the options. If you're really pragmatic, you just choose a standard label-set for which you can cite an article that everybody else always cites and be done with it.&lt;/p&gt;&#xA;" OwnerUserId="1909" LastActivityDate="2011-01-26T16:03:55.117" />
  <row Id="6577" PostTypeId="1" CreationDate="2011-01-26T16:07:42.770" Score="2" ViewCount="101" Body="&lt;p&gt;Hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've got some data that has this basic shape (using R):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df &amp;lt;- data.frame(group=sample(LETTERS, 500, T, log(2:27)), &#xA;                 type=sample(c(&quot;x&quot;,&quot;y&quot;), 500, T, c(.4,.6)),&#xA;                 value=sample(0:20, 500, T))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to investigate the ratios between &lt;code&gt;x&lt;/code&gt; and &lt;code&gt;y&lt;/code&gt; within each group.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One way would be to first compute the mean of &lt;code&gt;x&lt;/code&gt; and &lt;code&gt;y&lt;/code&gt; within each group, then use the &lt;code&gt;compressed.ratio&lt;/code&gt; function I wrote (does it have a name?  I just made it up) to map the ratio between the means from the interval [0,Inf] onto the interval [-1,1] so that it can be plotted symmetrically in &lt;code&gt;x&lt;/code&gt; and &lt;code&gt;y&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;compressed.ratio &amp;lt;- function(x, y) (x-y)/(x+y)&#xA;df.means &amp;lt;- ddply(df, .(group,type), &#xA;            function(df) data.frame(mean=mean(df$value),n=nrow(df)))&#xA;with(df.means, plot(unique(group), &#xA;               compressed.ratio(mean[type==&quot;x&quot;], mean[type==&quot;y&quot;]), ylim=c(-1,1)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In addition to this, I'd like to show something that gets at the amount of variation within each group, and also shows where there might be problems with very small numbers of samples in a given group.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I haven't thought of a good way to do these - the obvious way to show uncertainty due to sample size would be to use standard-error bars, but I'm not sure how to compute the standard-error of a ratio between two groups of quantities.  Would it be appropriate to compute the ratios of each &lt;code&gt;x&lt;/code&gt; to &lt;code&gt;mean(y)&lt;/code&gt;, and then &lt;code&gt;mean(x)&lt;/code&gt; to each &lt;code&gt;y&lt;/code&gt;, and treat those as &lt;code&gt;x+y&lt;/code&gt; separate measurements?  Or maybe to do some kind of random simulation, doing draws from the &lt;code&gt;x&lt;/code&gt; and &lt;code&gt;y&lt;/code&gt; pools and taking their ratios?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, does anyone know some kind of visual standard way to show both the standard-deviation and standard-error in the same graph?  Maybe a thick error bar and thin whiskers?&lt;/p&gt;&#xA;" OwnerUserId="1434" LastEditorUserId="1434" LastEditorDisplayName="" LastEditDate="2011-01-27T19:53:37.743" LastActivityDate="2011-01-27T19:53:37.743" Title="Investigate ratios between two groups" Tags="&lt;r&gt;&lt;statistical-significance&gt;" AnswerCount="1" />
  <row Id="6578" PostTypeId="2" ParentId="6080" CreationDate="2011-01-26T16:20:22.180" Score="0" ViewCount="" Body="&lt;p&gt;By zero-truncated, do you mean that any data that would have had a 0 as a response is just missing? In that case, can't you just put the 0s in?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or do you mean that some proportion of the time, instead of getting a sensical answer, you get a 0 instead? That sounds like zero-inflation to me. In that case, there are zero-inflated poisson and similar in GAMLSS.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't know of a zero-inflated Sichel, and there's nothing in GAMLSS for it, but is there a particularly good reason for using the Sichel distribution for your data? Does it reflect the underlying process particularly well? (I believe that the Sichel represents a mixture model of Poissons, where the meta-distribution is distributed as Inverse Gaussian...)&lt;/p&gt;&#xA;" OwnerUserId="6" LastActivityDate="2011-01-26T16:20:22.180" />
  <row Id="6579" PostTypeId="1" CreationDate="2011-01-26T16:24:01.237" Score="3" ViewCount="127" Body="&lt;p&gt;I am doing an experiment whereby I have 4 different conditions.  Within each condition, I do 4-7 technical replicates (cell counts in 4-7 high powered fields).  I have also repeated the experiment 3 times (3 biological replicate (rats)).  What test will compare the 4 different conditions, but will also take into account the variation between biological replicates?  I am trying to do this in graphpad prism 5.0.&lt;/p&gt;&#xA;" OwnerUserId="2928" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-01-26T18:46:56.767" LastActivityDate="2011-01-26T18:46:56.767" Title="Statistical test for multiple biological replicates" Tags="&lt;anova&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="6580" PostTypeId="1" AcceptedAnswerId="6602" CreationDate="2011-01-26T16:25:10.783" Score="4" ViewCount="124" Body="&lt;p&gt;&lt;strong&gt;Question&lt;/strong&gt;&lt;br&gt;&#xA;Is there such concept in econometrics/statistics as a derivative of parameter $\hat{b_{p}}$ in a linear model with respect to some observation $X_{ij}$?&lt;br&gt;&#xA;By derivative I mean $\frac{\partial \hat{b_{p}}}{\partial X_{ij}}$ - how would change parameter $\hat{b_{p}}$ if we changed $X_{ij}$?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Motivation&lt;/strong&gt;&lt;br&gt;&#xA;I was thinking about a situation when we have some uncertainty in data (e.g. results from a survey) and we have enough money to obtain precise results only in a single observation, which observation should we choose?&lt;br&gt;&#xA;My intuition is saying that we should choose the observation that might change parameters the most, which is equivalent to the highest value of the derivative. If there are any other concepts feel free to write about them.&lt;/p&gt;&#xA;" OwnerUserId="1643" LastEditorUserId="1643" LastEditorDisplayName="" LastEditDate="2011-01-26T19:06:17.867" LastActivityDate="2011-01-27T04:21:12.800" Title="Derivative of a linear model" Tags="&lt;regression&gt;&lt;uncertainty&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="2" />
  <row Id="6581" PostTypeId="1" AcceptedAnswerId="6610" CreationDate="2011-01-26T16:27:34.040" Score="5" ViewCount="372" Body="&lt;p&gt;What is &quot;Deviance,&quot; how is it calculated, and what are its uses in different fields in statistics?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In particular, I'm personally interested in its uses in CART (and its implementation in rpart in R). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm asking this since the &lt;a href=&quot;http://en.wikipedia.org/wiki/Deviance_%28statistics%29&quot; rel=&quot;nofollow&quot;&gt;wiki-article&lt;/a&gt; seems somewhat lacking and your insights will be most welcomed.&lt;/p&gt;&#xA;" OwnerUserId="253" LastActivityDate="2011-01-29T19:24:21.040" Title="What is Deviance? (specifically in CART/rpart)" Tags="&lt;r&gt;&lt;cart&gt;&lt;rpart&gt;&lt;deviance&gt;" AnswerCount="3" FavoriteCount="2" />
  <row Id="6582" PostTypeId="1" CreationDate="2011-01-26T17:23:46.763" Score="4" ViewCount="306" Body="&lt;p&gt;When deconstructing my mixed effects model, I found a three-way significant interaction. I calculated my p-value by using maximum likelihood ratio tests allowing for a comparison of the fit of the two models (the model with all predictors minus the model with all predictors but the predictor of interest - in this case, the three-way interaction). When I conduct follow-up comparisons of the three-way interaction, do I need to correct the alpha level of significance with Bonferroni correction?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for all input!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: (merged from answer --mbq)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to look at the significance of the three-way interaction and then...I wanted to look at any other significant interactions within that first three-way interaction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I use the same dataset...the model is a crossed random effects of participants and items (the data is comprised of repeated observations (response times) with Valence (positive and negative) and age as a between subjects factor as well as two continuous predictor variables (attachment dimensions). Thus my model is Valence x Age x Attachment anxiety x Attachment avoidance. I found that Valence x Age x Attachment avoidance is significant. However, I want to examine this interaction further. I did this by examining the same model but just for young adults vs old adults separately. Thus, I found with older adults a significant interaction of Valence and Attachment avoidance. However, when I calculated the p-value (as described above) of this two-way interaction, can I take the p-value as is or do I need to correct with Bonferroni? And if so, how? I hope this is clearer? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically I want to examine the 'direction' of my three-way interaction and test whether or not the differences within that interaction is significant.&lt;/p&gt;&#xA;" OwnerUserId="2934" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-01-26T20:57:29.597" LastActivityDate="2011-05-27T01:50:33.833" Title="Conducting planned comparisons in mixed model using lmer" Tags="&lt;mixed-model&gt;&lt;interaction&gt;&lt;model-comparison&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="1" />
  <row Id="6583" PostTypeId="2" ParentId="6579" CreationDate="2011-01-26T18:45:42.947" Score="1" ViewCount="" Body="&lt;p&gt;Sounds like you'll need to do a &lt;strong&gt;two-way analysis of variance&lt;/strong&gt; to me. I'm assuming the 'technical replicates' are 3 repeats of the same measurement procedure in the same rat with the same condition, and all the rats are subjected to all the conditions. The rats are then a 'blocking' factor, and the condition is your 'treatment' factor.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My only niggling doubt is: how did you decide how many technical replicates to do in each case?&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-01-26T18:45:42.947" />
  <row Id="6584" PostTypeId="2" ParentId="6570" CreationDate="2011-01-26T18:49:01.063" Score="0" ViewCount="" Body="&lt;p&gt;If you wish to detect overt opinions then put in a neutral option.  If you wish to detect any potential positive or negative bias then leave it out.  As caracal said, label things as unambiguously as possible with respect to what you wish the options to reflect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've seen studies where only the form of response was changed.  When there were only two options, like / dislike, then two stimuli were rated as very strongly liked in roughly equal proportions.  When subjects were subsequently given an infinite rating scale with neither like nor dislike in the middle the rating differences between the two stimuli were vast (75% of the scale vs. 4%).  This suggests that with a limited scale and no neutral option you can detect very small biases as large effects so you should be careful in interpreting such scales and use them judiciously.&lt;/p&gt;&#xA;" OwnerUserId="601" LastEditorUserId="601" LastEditorDisplayName="" LastEditDate="2011-01-26T18:56:44.410" LastActivityDate="2011-01-26T18:56:44.410" />
  <row Id="6585" PostTypeId="2" ParentId="6580" CreationDate="2011-01-26T19:14:09.810" Score="6" ViewCount="" Body="&lt;p&gt;I guess this would come under the heading of &lt;strong&gt;regression diagnostics&lt;/strong&gt;. I haven't seen this precise statistic before, but something that comes fairly close is &lt;strong&gt;DFBETA&lt;/strong&gt;&lt;sub&gt;&lt;em&gt;ij&lt;/em&gt;&lt;/sub&gt;, which is the the change in regression coefficient &lt;em&gt;i&lt;/em&gt; when the &lt;em&gt;j&lt;/em&gt;&lt;sup&gt;th&lt;/sup&gt; observation is &lt;em&gt;omitted&lt;/em&gt; divided by the estimated standard error of coefficient &lt;em&gt;i&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The book that defined this and many other regression diagnostics (perhaps &lt;em&gt;too&lt;/em&gt; many) is:  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Belsley, D. A., E. Kuh, and R. E. Welsch. (1980). &lt;em&gt;&lt;a href=&quot;http://books.google.co.uk/books?id=GECBEUJVNe0C&quot; rel=&quot;nofollow&quot;&gt;Regression Diagnostics: Identifying Influential Data and Sources of Collinearity&lt;/a&gt;.&lt;/em&gt;  New York: Wiley. ISBN 0471691178&lt;/p&gt;&#xA;" OwnerUserId="449" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-01-26T19:21:21.437" LastActivityDate="2011-01-26T19:21:21.437" CommentCount="1" />
  <row Id="6586" PostTypeId="2" ParentId="6570" CreationDate="2011-01-26T19:16:51.430" Score="2" ViewCount="" Body="&lt;p&gt;I try to avoid questions with more than two answers, as it is impossible to compare them between users. (good vs. very good can be very subjective).&#xA;I rephrase most questions into binary type (giving though the possibility to be indiffernt):&#xA;&quot;Would you use the product everyday?&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Yes No Indifferent&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Would you recommend the product to your friends?&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found results obtained with this method to be way more consistent with the feeling I got from later interviews and performance tests. However, my work so far focuses on Human Computer Interaction questionnaires.&#xA;The best is anyways to conduct real person interviews, as you learn more from them. Of course, they are also very time consuming :(&lt;/p&gt;&#xA;" OwnerUserId="2904" LastActivityDate="2011-01-26T19:16:51.430" />
  <row Id="6587" PostTypeId="2" ParentId="6571" CreationDate="2011-01-26T19:19:10.297" Score="2" ViewCount="" Body="&lt;p&gt;If you're trying to learn time series in R, I would suggest you to use real data and not simulated data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is so because in time series there many effects due to time, such as seasonality and trend.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would suggest you to take a look at&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;?ts&#xA;?ts.plot&#xA;?decompose&#xA;?arima&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you want to study this simulated data sets, you may find useful grouping your data as time series using male.age1, female.age1, male.age2, female.age2. For example&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;m.age1.series &amp;lt;- ts( male.age1, start=c(01,01) , frequency=30 )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This will create a time series object that you may analyze.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Take a look at the following link:&#xA;&lt;a href=&quot;http://www.stat.pitt.edu/stoffer/tsa2/R_time_series_quick_fix.htm&quot; rel=&quot;nofollow&quot;&gt;http://www.stat.pitt.edu/stoffer/tsa2/R_time_series_quick_fix.htm&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope it helps! :)&lt;/p&gt;&#xA;" OwnerUserId="2902" LastActivityDate="2011-01-26T19:19:10.297" />
  <row Id="6588" PostTypeId="1" AcceptedAnswerId="6596" CreationDate="2011-01-26T20:01:23.850" Score="5" ViewCount="115" Body="&lt;p&gt;I have a data set with lots of zeros that looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(1)&#xA;x &amp;lt;- c(rlnorm(100),rep(0,50))&#xA;hist(x,probability=TRUE,breaks = 25)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I would like to draw a line for its density, but the &lt;code&gt;density()&lt;/code&gt; function uses a moving window that calculates negative values of x.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;lines(density(x), col = 'grey')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;There is a &lt;code&gt;density(... from, to)&lt;/code&gt; arguments, but these seem to only truncate the calculation, not alter the window so that the density at 0 is consistent with the data as can be seen by the following plot : &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;lines(density(x, from = 0), col = 'black')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(if the interpolation was changed, I would expect that the black line would have higher density at 0 than the grey line)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there alternatives to this function that would provide a better calculation of the density at zero?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/1IQTU.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="2750" LastEditorUserId="2750" LastEditorDisplayName="" LastEditDate="2011-01-27T17:51:29.550" LastActivityDate="2011-01-27T17:51:29.550" Title="How can I estimate the density of a zero-inflated parameter in R?" Tags="&lt;r&gt;&lt;probability&gt;" AnswerCount="3" />
  <row Id="6589" PostTypeId="2" ParentId="6575" CreationDate="2011-01-26T20:30:43.377" Score="1" ViewCount="" Body="&lt;p&gt;There are factor analysis techniques that allow oblique rotation, not just the orthogonal rotation that PCA uses. Take a look at direct oblimin rotation or promax rotation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Not sure what statistical application you are using. In R, the psych and HDMD packages have commands that allow oblique rotations.&lt;/p&gt;&#xA;" OwnerUserId="2933" LastActivityDate="2011-01-26T20:30:43.377" CommentCount="1" />
  <row Id="6591" PostTypeId="2" ParentId="6080" CreationDate="2011-01-26T20:38:00.620" Score="0" ViewCount="" Body="&lt;p&gt;@Harlan:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By zero-truncated, I mean that the probability of a zero-count is zero. My data are the count of items purchased &lt;em&gt;per transaction&lt;/em&gt; at a convenience store over a five-day period--obviously there's at least one item being bought per purchase (but usually fewer than, say, 20).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The schema looks something like:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;DATE, TRANSACTION_ID, COUNT_OF_PURCHASES&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As to Sichel, I do have reason to believe that that distribution might provide a good fit, but I'm trying several others as well (e.g., ZTNB).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for your reply.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;--Jeff&lt;/p&gt;&#xA;" OwnerUserId="2730" LastEditorUserId="2730" LastEditorDisplayName="" LastEditDate="2011-01-27T06:16:24.633" LastActivityDate="2011-01-27T06:16:24.633" CommentCount="3" />
  <row Id="6592" PostTypeId="1" CreationDate="2011-01-26T21:02:50.433" Score="4" ViewCount="129" Body="&lt;p&gt;I'm not sure that I've titled this question correctly, but here is my query. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose you are given a set of measurements and the uncertainty (variance) associated with each. The task is to statistically figure out how many different objects were likely measured and finally, to combine measurements into a single estimate for each. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second part is easy enough - an uncertainty-weighted mean would do it - but I am having difficulty understanding how to sort out how many objects were measured. If there were just two objects, ANOVA would work. But what if there are an unknown number of objects? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an aside, I'm aware of a Baysian technique in which one considers each measurement in turn, building a set of hypotheses for each measurement that doesn't fall within the confidence interval of an existing hypothesis and combining it into the hypothesis when they do. But I think this method is dependent on the order the measurements are considered and therefore imparts a kind of time dependence on measurements that have none. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I feel like this is something that's commonly done and I should know how to do, but I'm stumped so any thoughts you all might have would be much appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks,&#xA;Val &lt;/p&gt;&#xA;" OwnerUserId="2932" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-01-26T23:07:52.737" LastActivityDate="2011-05-26T23:50:32.723" Title="Multiple hypothesis ANOVA" Tags="&lt;anova&gt;&lt;clustering&gt;&lt;meta-analysis&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="6593" PostTypeId="2" ParentId="6581" CreationDate="2011-01-26T22:13:54.307" Score="3" ViewCount="" Body="&lt;p&gt;Hello Tal:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Deviance is the likelihood-ratio statistic for testing the null hypothesis that the model holds agains the general alternative (i.e., the saturated model). For some Poisson and binomial GLMs, the number  of observations $N$ stays fixed as the individual counts increase in size. Then the deviance has a &lt;em&gt;chi-squared asymptotic null distribution&lt;/em&gt;. The degrees of freedom = N - p, where p is the number of model parameters; i.e., it is equal to the numbers of free parameters in the saturated and unsaturated models. The deviance then provides a test for the model fit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Deviance = -2[L(\hat{\mathbf{\mu}} | \mathbf{y})-L(\mathbf{y}|\mathbf{y})]$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, most of the times, you want to test if you need to drop some variables. Say there are two models $M_1$ and $M_2$ with $p_1$ and $p_2$ parameters, respectively, and you need to test which of these two is better. Assume $M_1$ is a special case of  $M_2$ i.e. nested models.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In that case, the difference of deviance is taken:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\Delta Deviance = -2[L(\hat{\mathbf{\mu}_1} | \mathbf{y})-L(\hat{\mathbf{\mu}_2}|\mathbf{y})]$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Notice that the log likelihood of the saturated model cancels and the degree of freedom of $\Delta Deviance$ changes to $p_2-p_1$. This is what we use most often when we need to test if some of the parameters are 0 or not. But when you fit &lt;code&gt;glm&lt;/code&gt; in &lt;code&gt;R&lt;/code&gt; the deviance output is for the saturated model vs the current model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to read in greater details: cf: Categorical Data Analysis by Alan Agresti, pp 118.&lt;/p&gt;&#xA;" OwnerUserId="1307" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-01-26T22:20:50.633" LastActivityDate="2011-01-26T22:20:50.633" CommentCount="3" />
  <row Id="6594" PostTypeId="2" ParentId="4466" CreationDate="2011-01-26T22:17:22.510" Score="0" ViewCount="" Body="&lt;p&gt;Have you consider Unit Testing with RUnit to verify theoretical behavior?&lt;/p&gt;&#xA;" OwnerUserId="2935" LastActivityDate="2011-01-26T22:17:22.510" />
  <row Id="6595" PostTypeId="2" ParentId="6588" CreationDate="2011-01-26T22:19:10.947" Score="0" ViewCount="" Body="&lt;p&gt;You may try lowering bandwidth (blue line is for &lt;code&gt;adjust=0.5&lt;/code&gt;),&#xA;&lt;img src=&quot;http://i.imgur.com/79Uly.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;but probably KDE is just not the best method to deal with such data.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2011-01-26T22:19:10.947" CommentCount="2" />
  <row Id="6596" PostTypeId="2" ParentId="6588" CreationDate="2011-01-26T22:39:44.857" Score="7" ViewCount="" Body="&lt;p&gt;The density is infinite at zero because it includes a discrete spike. You need to estimate the spike using the proportion of zeros, and then estimate the positive part of the density assuming it is smooth. KDE will cause problems at the left hand end because it will put some weight on negative values. One useful approach is to transform to logs, estimate the density using KDE, and then transform back. See &lt;a href=&quot;http://www.jstor.org/pss/2290569&quot; rel=&quot;nofollow&quot;&gt;Wand, Marron &amp;amp; Ruppert (JASA 1991)&lt;/a&gt; for a reference. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The following R function will do the transformed density:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;logdensity &amp;lt;- function (x, bw = &quot;SJ&quot;) &#xA;{&#xA;    y &amp;lt;- log(x)&#xA;    g &amp;lt;- density(y, bw = bw, n = 1001)&#xA;    xgrid &amp;lt;- exp(g$x)&#xA;    g$y &amp;lt;- c(0, g$y/xgrid)&#xA;    g$x &amp;lt;- c(0, xgrid)&#xA;    return(g)&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then the following will give the plot you want:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(1)&#xA;x &amp;lt;- c(rlnorm(100),rep(0,50))&#xA;hist(x,probability=TRUE,breaks = 25)&#xA;fit &amp;lt;- logdensity(x[x&amp;gt;0]) # Only take density of positive part&#xA;lines(fit$x,fit$y*mean(x&amp;gt;0),col=&quot;red&quot;) # Scale density by proportion positive&#xA;abline(v=0,col=&quot;blue&quot;) # Add spike at zero.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/OUfZq.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="159" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2011-01-26T23:02:53.137" LastActivityDate="2011-01-26T23:02:53.137" CommentCount="3" />
  <row Id="6597" PostTypeId="2" ParentId="6592" CreationDate="2011-01-26T22:41:50.113" Score="0" ViewCount="" Body="&lt;p&gt;One approach would be a &lt;strong&gt;finite mixture model&lt;/strong&gt; with an unknown number of components. A set of measurements and their variances sounds like &lt;strong&gt;meta-analysis&lt;/strong&gt;. I suggest you have at look at &lt;a href=&quot;http://www.charite.de/biometrie/schlattmann/book/&quot; rel=&quot;nofollow&quot;&gt;Peter Schlattmann's webpage for his book 'Medical Applications of Finite Mixture Models'&lt;/a&gt;, which includes meta-analysis amongst its applications. The book is not cheap, but if you download the R or SAS code from that webpage and have a look at the documentation you &lt;em&gt;may&lt;/em&gt; be able to manage without it.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-01-26T22:41:50.113" CommentCount="1" />
  <row Id="6598" PostTypeId="2" ParentId="6588" CreationDate="2011-01-26T23:05:31.587" Score="3" ViewCount="" Body="&lt;p&gt;I'd agree with Rob Hyndman that you need to deal with the zeroes separately. There are a few methods of dealing with a kernel density estimation of a variable with bounded support, including 'reflection', 'rernormalisation' and 'linear combination'. These don't appear to have been implemented in R's &lt;code&gt;density&lt;/code&gt; function, but are available in &lt;a href=&quot;http://fmwww.bc.edu/RePEc/bocode/k/kdens.pdf&quot; rel=&quot;nofollow&quot;&gt;Benn Jann's &lt;code&gt;kdens&lt;/code&gt; package for Stata&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-01-26T23:05:31.587" />
  <row Id="6599" PostTypeId="1" AcceptedAnswerId="6665" CreationDate="2011-01-26T23:48:06.297" Score="5" ViewCount="125" Body="&lt;p&gt;I have read Alexandru Niculescu-Mizil and Rich Caruana's paper &quot;&lt;a href=&quot;http://aaaipress.org/Papers/Workshops/2007/WS-07-05/WS07-05-006.pdf&quot; rel=&quot;nofollow&quot;&gt;Obtaining Calibrated Probabilities from Boosting&lt;/a&gt;&quot; and the discussion in &lt;a href=&quot;http://stats.stackexchange.com/questions/5196/why-use-platts-scaling&quot;&gt;this&lt;/a&gt; thread. However, I am still having trouble understanding and implementing &lt;strong&gt;logistic&lt;/strong&gt; or &lt;strong&gt;Platt's scaling&lt;/strong&gt; to calibrate the output of my multi-class boosting classifier (gentle-boost with decision stumps).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am somewhat familiar with generalized linear models, and I think I understand how the logistic and Platt's calibration methods work in the binary case, but am not sure I know how to extend the method described in the paper to the multi-class case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The classifier I am using outputs the following: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;$f_{ij}$ = Number of votes that the classifier casts for class $j$ for the sample $i$ that is being classified&lt;/li&gt;&#xA;&lt;li&gt;$y_i$ = Estimated class &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;At this point I have the following questions:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Q1:&lt;/strong&gt; Do I need to use a multinomial logit to estimate probabilities? or can I still do this with logistic regression (e.g. in a &lt;em&gt;1-vs-all&lt;/em&gt; fashion)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Q2:&lt;/strong&gt; How should I define the intermediate target variables (e.g. as in Platt's scaling) for the multi-class case?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Q3:&lt;/strong&gt; I understand this might be a lot to ask, but would anybody be willing to sketch out the pseudo-code for this problem? (on a more practical level, I am interested in a solution in Matlab).&lt;/p&gt;&#xA;" OwnerUserId="2798" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-01-27T20:50:06.290" LastActivityDate="2011-01-28T13:21:02.197" Title="Calibrating a multi-class boosted classifier" Tags="&lt;machine-learning&gt;&lt;boosting&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="3" />
  <row Id="6600" PostTypeId="2" ParentId="6582" CreationDate="2011-01-27T01:07:12.153" Score="0" ViewCount="" Body="&lt;p&gt;It sounds like you basically have a problem of &lt;em&gt;model choice&lt;/em&gt;.  I think this is best treated as a &lt;em&gt;decision problem&lt;/em&gt;.  You want to act &lt;em&gt;as if&lt;/em&gt; the final model you select &lt;em&gt;is the true model&lt;/em&gt;, so that you can make conclusions about your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So in decision theory, you need to specify a &lt;em&gt;loss function&lt;/em&gt;, which says how you are going to rank each model, and a &lt;em&gt;set of alternative models&lt;/em&gt; which you are going to decide between.  See &lt;a href=&quot;http://www.uv.es/~bernardo/2010Valencia9.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;http://www.uv.es/~bernardo/JMBSlidesV9.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; for a decision theoretical approach to hypothesis testing in inference. And &lt;a href=&quot;http://www.uv.es/~bernardo/Kernel.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; is one which uses a decision theory approach to choose a model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It sounds like you want to use the &lt;em&gt;p-value&lt;/em&gt; as your loss function (because that's how you want to compare the models).  So if this is your criterion, then you pick the model with the smallest p-value.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But the criterion needs to apply to something which the models have in common, an &quot;obvious&quot; choice based on a statistic which measures how well the model fits the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One example is the sum of squared errors for predicting a new set of observations &lt;em&gt;which were not included in the model fitting&lt;/em&gt; (based on the idea that a &quot;good&quot; model should reproduce the data it is supposed to be describing).  So, what you can do is, for each model:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) randomly split your data into two parts, a &quot;model part&quot; big enough for your model, and a &quot;test&quot; part to check predictions (which particular partition should not matter if the model is a good model).  The &quot;model&quot; set is usually larger than the &quot;test&quot; set (at least 10 times larger, depending on how much data you have)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Fit the model to the &quot;model data&quot;, and then use it to predict the &quot;test&quot; data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) Calculate the sum of squared error for prediction in the &quot;test&quot; data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4) repeat 1-3 as many times as you feel necessary for your data (just in case you did a &quot;bad&quot; or &quot;unlucky&quot; partition), and take the average of the sum of squared error value in step 3).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It does seem as though you have already defined a class of alternative models that you are willing to consider.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just a side note: &lt;em&gt;Any&lt;/em&gt; procedure that you use to select the model, should go into step 1, including &quot;automatic&quot; model selection procedures.  This way you properly account for the &quot;multiple comparisons&quot; that the automatic procedure does.  Unfortunately, you need to have an alternative (maybe one is &quot;foward selection&quot; one is &quot;forward stepwise&quot; one is &quot;backward selection&quot;, etc.).  To &quot;keep things fair&quot; you could keep the same set of partitions for all models.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-01-27T01:07:12.153" CommentCount="5" />
  <row Id="6601" PostTypeId="1" AcceptedAnswerId="6605" CreationDate="2011-01-27T03:57:01.977" Score="14" ViewCount="663" Body="&lt;p&gt;This is a similar question to the one &lt;a href=&quot;http://stats.stackexchange.com/questions/155/what-is-your-favorite-laymans-explanation-for-a-difficult-statistical-concept&quot;&gt;here&lt;/a&gt;, but different enough I think to be worthwhile asking.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I thought I'd put as a starter, what I think one of the hardest to grasp is.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Mine is the difference between &lt;em&gt;probability&lt;/em&gt; and &lt;em&gt;frequency&lt;/em&gt;.  One is at the level of &quot;knowledge of reality&quot; (probability), while the other is at the level &quot;reality itself&quot; (frequency).  This almost always makes me confused if I think about it too much.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edwin Jaynes Coined a term called the &quot;mind projection fallacy&quot; to describe getting these things mixed up.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any thoughts on any other tough concepts to grasp?&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-01-30T00:05:34.997" CommunityOwnedDate="2011-01-27T04:24:01.987" Title="What is the hardest statistical concept to grasp?" Tags="&lt;teaching&gt;" AnswerCount="11" CommentCount="5" FavoriteCount="8" />
  <row Id="6602" PostTypeId="2" ParentId="6580" CreationDate="2011-01-27T04:21:12.800" Score="5" ViewCount="" Body="&lt;p&gt;@onestop points in the right direction.  Belsley, Kuh, and Welsch describe this approach on pp. 24-26 of their book.  To differentiate with respect to an &lt;em&gt;observation&lt;/em&gt; (and not just one of its attributes), they introduce a weight, perform weighted least squares, and differentiate with respect to the weight.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Specifically, let $\mathbb{X} = X_{ij}$ be the design matrix, let $\mathbf{x}_i$ be the $i$th observation, let $e_i$ be its residual, let $w_i$ be the weight, and define $h_i$ (the $i$th diagonal entry in the hat matrix) to be $\mathbf{x}_i (\mathbb{X}^T \mathbb{X})^{-1} \mathbf{x}_i^T$.  They compute&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{\partial b(w_i)}{\partial w_i} = \frac{(\mathbb{X}^T\mathbb{X})^{-1} \mathbf{x}_i^T e_i}{\left[1 - (1 - w_i)h_i\right]^2},$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;whence&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{\partial b(w_i)}{\partial w_i}\Bigg|_{w_i=1} = (\mathbb{X}^T\mathbb{X})^{-1} \mathbf{x}_i^T e_i.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is interpreted as a way to &quot;identify influential observations, ... provid[ing] a means for examining the sensitivity of the regression coefficients to a slight change in the weight given to the ith observation.  Large values of this derivative indicate observations that have large influence on the calculated coefficients.&quot;  They suggest it can be used as an alternative to the DFBETA diagnostic.  (DFBETA measures the change in $b$ when observation $i$ is completely deleted.)  The relationship between the influence and DFBETA is that DFBETA equals the influence divided by $1 - h_i$ [equation 2.1 p. 13].&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-01-27T04:21:12.800" CommentCount="3" />
  <row Id="6603" PostTypeId="2" ParentId="6176" CreationDate="2011-01-27T04:35:11.757" Score="0" ViewCount="" Body="&lt;p&gt;Jumping straight into non-parametric Bayesian analysis is quite a big first leap!  Maybe get a bit of parametric Bayes under your belt first?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Three books which you may find useful from the Bayesian part of things are:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) &lt;em&gt;Probability Theory: The Logic of Science&lt;/em&gt; by E. T. Jaynes, Edited by G. L. Bretthorst (2003)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) &lt;em&gt;Bayesian Theory&lt;/em&gt; by Bernardo, J. M. and Smith, A. F. M. (1st ed 1994, 2nd ed 2007).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) &lt;em&gt;Bayesian Decision Theory&lt;/em&gt; J. O. Berger (1985)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A good place to see &lt;em&gt;recent applications&lt;/em&gt; of Bayesian statistics is the FREE journal called &lt;a href=&quot;http://ba.stat.cmu.edu/&quot; rel=&quot;nofollow&quot;&gt;Bayesian Analysis&lt;/a&gt;, with articles from 2006 to present.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-01-27T04:35:11.757" />
  <row Id="6604" PostTypeId="1" CreationDate="2011-01-27T05:51:07.740" Score="3" ViewCount="254" Body="&lt;p&gt;Hello,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am running a bivariate correlation analysis in SPSS, and I am performing multiple comparisons (there are 8 variables in total). I want to correct for multiple comparisons because I am aware that any 'significant' results could simply be flukes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the Bonferroni correction is not appropriate in this case (it is too strict).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone know how to correct for multiple comparisons in SPSS?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider some sample data as follows. There are 4 'independent variables' and 4 'dependent variables'.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Independent variables:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Blood flow through middle cerebral artery&lt;/li&gt;&#xA;&lt;li&gt;Blood flow through anterior cerebral artery&lt;/li&gt;&#xA;&lt;li&gt;Blood flow through posterior cerebral artery&lt;/li&gt;&#xA;&lt;li&gt;Blood flow through anterior communicating artery. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Dependent variables&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Performance on cognitive test #1&lt;/li&gt;&#xA;&lt;li&gt;Performance on cognitive test #2&lt;/li&gt;&#xA;&lt;li&gt;Performance on cognitive test #3&lt;/li&gt;&#xA;&lt;li&gt;Performance on cognitive test #4&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The 4 'independent' variables are &lt;strong&gt;not&lt;/strong&gt; uncorrelated to each other. The 4 'dependent' variables are also &lt;strong&gt;not&lt;/strong&gt; going to be uncorrelated to each other (i.e., if a person does well on one test, chances are they will also do well on another test).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(I realize that it is wrong to call these variables 'independent' and 'dependent', since correlation does not prove causality, but this is the way that I have framed them in my mind)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suppose that there are 2 problems here:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;how to correct for multiple comparisons (from a statistics point of view) and&lt;/li&gt;&#xA;&lt;li&gt;how to actually implement this in SPSS (a practical problem).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Any help would be much appreciated (especially for problem #2).&lt;/p&gt;&#xA;" OwnerUserId="2938" LastActivityDate="2011-01-27T10:46:55.900" Title="Correcting for multiple comparisons when running a bivariate correlation in SPSS" Tags="&lt;correlation&gt;&lt;multiple-comparisons&gt;&lt;spss&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="6605" PostTypeId="2" ParentId="6601" CreationDate="2011-01-27T06:06:18.073" Score="20" ViewCount="" Body="&lt;p&gt;for some reason, people have difficulty grasping what a p-value really is.&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2011-01-27T06:06:18.073" CommunityOwnedDate="2011-01-27T06:06:18.073" CommentCount="37" />
  <row Id="6606" PostTypeId="2" ParentId="6601" CreationDate="2011-01-27T06:46:32.313" Score="13" ViewCount="" Body="&lt;p&gt;Similar to shabbychef's answer, it is difficult to understand the meaning of a confidence interval in frequentist statistics. I think the biggest obstacle is that a confidence interval doesn't answer the question that we would like to answer. We'd like to know, &quot;what's the chance that the true value is inside this particular interval?&quot; Instead, we can only answer, &quot;what's the chance that a randomly chosen interval created in this way contains the true parameter?&quot; The latter is obviously less satisfying.&lt;/p&gt;&#xA;" OwnerUserId="401" LastActivityDate="2011-01-27T06:46:32.313" CommunityOwnedDate="2011-01-27T06:46:32.313" CommentCount="1" />
  <row Id="6607" PostTypeId="2" ParentId="6601" CreationDate="2011-01-27T07:51:33.693" Score="5" ViewCount="" Body="&lt;p&gt;What do the different distributions really represent, besides than how they are used.&lt;/p&gt;&#xA;" OwnerUserId="1808" LastActivityDate="2011-01-27T07:51:33.693" CommunityOwnedDate="2011-01-27T07:51:33.693" CommentCount="2" />
  <row Id="6608" PostTypeId="2" ParentId="6581" CreationDate="2011-01-27T08:47:16.440" Score="5" ViewCount="" Body="&lt;p&gt;It might be a bit clearer if we think about a perfect model with as many parameters as observations such that it explains all variance in the response. This is the saturated model. Deviance simply measures the difference in &quot;fit&quot; of a candidate model and that of the saturated model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a regression tree, the saturated model would be one that had as many terminal nodes (leaves) as observations so it would perfectly fit the response. The deviance of a simpler model can be computed as the node residual sums of squares, summed over all nodes. In other words, the sum of squared differences between predicted and observed values. This is the same sort of error (or deviance) used in least squares regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a classification tree, residual sums of squares is not the most appropriate measure of lack of fit. Instead, there is an alternative measure of deviance, plus trees can be built minimising an entropy measure or the Gini index. The latter is the default in &lt;code&gt;rpart&lt;/code&gt;. The Gini index is computed as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$D_i = 1 - \sum_{k = 1}^{K} p_{ik}^2$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $p_{ik}$ is the observed proportion of class $k$ in node $i$. This measure is summed of all terminal $i$ nodes in the tree to arrive at a deviance for the fitted tree model.&lt;/p&gt;&#xA;" OwnerUserId="1390" LastActivityDate="2011-01-27T08:47:16.440" CommentCount="1" />
  <row Id="6609" PostTypeId="1" CreationDate="2011-01-27T08:53:01.223" Score="6" ViewCount="299" Body="&lt;h3&gt;Question:&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Can you do a repeated measures multinomial logistic regression using SPSS?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;Context:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;I need to do a regression on data at two points of time and I think this maybe the only way to go(?).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To elaborate: I work for a national health service supporting individuals with psychosis. I want to investigate whether any factors (age, gender etc) predict the vocational outcome of a client group (3 categories: partial vocation, full vocation or no vocation) at entry of service and at 18 months into the service. I want to consider whether there is a difference at 18 months compared to entry to the service in regards to vocational status.&lt;/p&gt;&#xA;" OwnerUserId="2939" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-02-25T03:26:34.467" LastActivityDate="2011-02-25T03:26:34.467" Title="How to do a repeated measures multinomial logistic regression using SPSS?" Tags="&lt;logistic-regression&gt;&lt;spss&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="6610" PostTypeId="2" ParentId="6581" CreationDate="2011-01-27T09:05:42.833" Score="8" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Deviance and GLM&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Formally, one can views deviance as a sort of distance between two probabilistic models; in an GLM context, it amounts to two times the ratio of log-likelihood between two nested models $\ell_1/\ell_0$ where $\ell_0$ is the &quot;smaller&quot; model, that is a linear restriction on model parameters (cf. the &lt;a href=&quot;http://j.mp/awJEkH&quot; rel=&quot;nofollow&quot;&gt;Neyman–Pearson lemma&lt;/a&gt;), as @suncoolsu said. As such, it can be used to perform &lt;em&gt;model comparison&lt;/em&gt;. It can also be seen it as a generalization of the RSS used in OLS estimation (ANOVA, regression), for it provides a measure of &lt;em&gt;goodness-of-fit&lt;/em&gt; of the model being evaluated when compared to the null model (intercept only). It works with LM too:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; x &amp;lt;- rnorm(100)&#xA;&amp;gt; y &amp;lt;- 0.8*x+rnorm(100)&#xA;&amp;gt; lm.res &amp;lt;- lm(y ~ x)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The residuals SS (RSS) is computed as $\hat\varepsilon^t\hat\varepsilon$, which is readily obtained as:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; t(residuals(lm.res))%*%residuals(lm.res)&#xA;         [,1]&#xA;[1,] 98.66754&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;or from the (unadjusted) $R^2$ &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; summary(lm.res)&#xA;&#xA;Call:&#xA;lm(formula = y ~ x)&#xA;&#xA;(...)&#xA;&#xA;Residual standard error: 1.003 on 98 degrees of freedom&#xA;Multiple R-squared: 0.4234, Adjusted R-squared: 0.4175 &#xA;F-statistic: 71.97 on 1 and 98 DF,  p-value: 2.334e-13 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;since $R^2=1-\text{RSS}/\text{TSS}$ where $\text{TSS}$ is the total variance. Note that it is directly available in an ANOVA table, like&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; summary.aov(lm.res)&#xA;            Df Sum Sq Mean Sq F value    Pr(&amp;gt;F)    &#xA;x            1 72.459  72.459  71.969 2.334e-13 ***&#xA;Residuals   98 98.668   1.007                      &#xA;---&#xA;Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, look at the deviance:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; deviance(lm.res)&#xA;[1] 98.66754&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In fact, for linear models the deviance equals the RSS (you may recall that OLS and ML estimates coincide in such a case).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Deviance and CART&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can see CART as a way to allocate already $n$ labeled individuals into arbitrary classes (in a classification context). Trees can be viewed as providing a probability model for individuals class membership. So, at each node $i$, we have a probability distribution $p_{ik}$ over the classes. What is important here is that the leaves of the tree give us a random sample $n_{ik}$ from a multinomial distribution specified by $p_{ik}$. We can thus define the deviance of a tree, $D$, as the sum over all leaves of&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$D_i=-2\sum_kn_{ik}\log(p_{ik}),$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;following Venables and Ripley's notations (&lt;a href=&quot;http://www.stats.ox.ac.uk/pub/MASS4/&quot; rel=&quot;nofollow&quot;&gt;MASS&lt;/a&gt;, Springer 2002, 4th ed.). If you have access to this essential reference for R users (IMHO), you can check by yourself how such an approach is used for splitting nodes and fitting a tree to observed data (p. 255 ff.); basically, the idea is to minimize, by pruning the tree, $D+\alpha \#(T)$ where $\#(T)$ is the number of nodes in the tree $T$. Here we recognize the &lt;em&gt;cost-complexity trade-off&lt;/em&gt;. Here, $D$ is equivalent to the concept of node impurity (i.e., the heterogeneity of the distribution at a given node) which are based on a measure of entropy or information gain, or the well-known Gini index, defined as $1-\sum_kp_{ik}^2$ (the unknown proportions are estimated from node proportions).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With a regression tree, the idea is quite similar, and we can conceptualize the deviance as sum of squares defined for individuals $j$ by&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$D_i=\sum_j(y_j-\mu_i)^2,$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;summed over all leaves. Here, the probability model that is considered within each leaf is a gaussian $\mathcal{N}(\mu_i,\sigma^2)$. Quoting Venables and Ripley (p. 256), &quot;$D$ is the usual scaled deviance for a gaussian GLM. However, the distribution at internal nodes of the tree is then a mixture of normal distributions, and so $D_i$ is only appropriate at the leaves. The tree-construction process has to be seen as a &lt;em&gt;hierarchical refinement of probability models, very similar to forward variable selection in regression&lt;/em&gt;.&quot; Section 9.2 provides further detailed information about &lt;code&gt;rpart&lt;/code&gt; implementation, but you can already look at the &lt;code&gt;residuals()&lt;/code&gt; function for &lt;code&gt;rpart&lt;/code&gt; object, where &quot;deviance residuals&quot; are computed as the square root of minus twice the logarithm of the fitted model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.stanford.edu/class/stats315b/minitech.pdf&quot; rel=&quot;nofollow&quot;&gt;An introduction to recursive partitioning using the rpart routines&lt;/a&gt;, by Atkinson and Therneau, is also a good start. For more general review (including bagging), I would recommend&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Moissen, G.G. (2008). &lt;a href=&quot;http://www.fs.fed.us/rm/pubs_other/rmrs_2008_moisen_g001.pdf&quot; rel=&quot;nofollow&quot;&gt;Classification and Regression Trees&lt;/a&gt;. &lt;em&gt;Ecological Informatics&lt;/em&gt;, pp. 582-588.&lt;/li&gt;&#xA;&lt;li&gt;Sutton, C.D. (2005). &lt;a href=&quot;http://mason.gmu.edu/~csutton/vt6.pdf&quot; rel=&quot;nofollow&quot;&gt;Classification and Regression Trees, Bagging,&#xA;and Boosting&lt;/a&gt;, in &lt;em&gt;Handbook of Statistics, Vol. 24&lt;/em&gt;, pp. 303-329, Elsevier.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-01-29T19:24:21.040" LastActivityDate="2011-01-29T19:24:21.040" CommentCount="6" />
  <row Id="6611" PostTypeId="2" ParentId="6540" CreationDate="2011-01-27T10:46:01.347" Score="1" ViewCount="" Body="&lt;p&gt;The question does not state the precise intervals or yields so the $H_{0}$ hypothesis must be conservative with infinite intervals and yields with pessimist approximations, &lt;code&gt;.*logic&lt;/code&gt;'s suggestion won't qualify. Confidence interval not calculated. So:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$lim_{ m \rightarrow \infty } \left[ 1+\frac{r}{m} \right] ^{mt}=e^{rt}$,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where &lt;code&gt;r&lt;/code&gt; is the rate p.a. and &lt;code&gt;t&lt;/code&gt; is the time. The sum is $\sum_{k=1}^{n} x_{k} e^{rt}$. If you have data of different signs, you must calculate positive numbers to one sum and negative numbers to one sum, this way you get proper upper/lower bounds. The exponent is $r_{k} MINUS(timestamp_{1}, timestamp_{2})/365$, the &lt;code&gt;MINUS&lt;/code&gt; -function returns days between the timestamps and $r_{k}$ is the rate. The terms &lt;code&gt;FV&lt;/code&gt; and &lt;code&gt;PV&lt;/code&gt; are:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$FV = x_{0} \left( 1+r \right)^{n} + x_{1} \left(1+r \right)^{n-1}+...+x_{n}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$PV = x_{0} + \frac{x_{1}}{1+r} +...+ \frac{x_{n}}{\left( 1+r\right)}^{n}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;so $FV$ is with the sum -formula, while you just take reciprocal with $PV$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Example&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1 01/1/1980&#xA;2 01/2/1999&#xA;3 03/12/2000&#xA;-1 03/6/2005&#xA;-5 07/07/2007&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Work in progress.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Trying to find one-liner to operate over the data:  $if (term_{k} &amp;gt; 0) : sum (positive_{k}e^{MINUS(t_{k},t_{0})/365})$&lt;/p&gt;&#xA;" OwnerUserId="2914" LastEditorUserId="2914" LastEditorDisplayName="" LastEditDate="2011-01-28T21:04:15.727" LastActivityDate="2011-01-28T21:04:15.727" CommentCount="4" />
  <row Id="6612" PostTypeId="2" ParentId="6604" CreationDate="2011-01-27T10:46:55.900" Score="3" ViewCount="" Body="&lt;p&gt;This first part of my response won't address your two questions directly since what I am suggesting departs from your correlational approach. If I understand you correctly, you have two blocks of variables, and they play an asymmetrical role in the sense that one of them is composed of response variables (performance on four cognitive tests) whereas the other includes explanatory variables (measures of blood flow at several locations). So, a nice way to answer your question of interest would be to look at &lt;a href=&quot;http://en.wikipedia.org/wiki/Partial_least_squares_regression&quot; rel=&quot;nofollow&quot;&gt;PLS regression&lt;/a&gt;. As detailed in an earlier response of mine, &lt;a href=&quot;http://stats.stackexchange.com/questions/4517/regression-with-multiple-dependent-variables&quot;&gt;Regression with multiple dependent variables?&lt;/a&gt;, the correlation between factor scores on the first dimension will reflect the overall link between these two blocks, and a closer look at the weighted combination of variables in each block (i.e., loadings) would help interpreting the contribution of each variable of the $X$ block in predicting the $Y$ block. The &lt;a href=&quot;http://faculty.chass.ncsu.edu/garson/PA765/pls.htm&quot; rel=&quot;nofollow&quot;&gt;SPSS implementation&lt;/a&gt; is detailed on Dave Garson's website. This prevents from using any correction for multiple comparisons.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Back to your specific questions, yes the Bonferroni correction is known to be conservative and step-down methods are to be preferred (instead of correcting the p-values or the test statistic in one shot for all the tests, we adapt the threshold depending on the previous HT outcomes, in a sequential manner).&#xA;Look into SPSS documentation (or &lt;a href=&quot;http://www.uky.edu/ComputingCenter/SSTARS/www/documentation/MultipleComparisons_3.htm#b13&quot; rel=&quot;nofollow&quot;&gt;Pairwise Comparisons in SAS and SPSS&lt;/a&gt;) to find a suitable one, e.g. Bonferroni-Holm.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-01-27T10:46:55.900" />
  <row Id="6613" PostTypeId="1" CreationDate="2011-01-27T11:03:38.850" Score="3" ViewCount="127" Body="&lt;pre&gt;&lt;code&gt;x &amp;lt;- read.table('file_name',header=TRUE, row.names=1)&#xA;y &amp;lt;- t(x)&#xA;y &amp;lt;- data.frame(y)&#xA;row.names(y) &amp;lt;- names(x)&#xA;names(y) &amp;lt;- row.names(x)&#xA;library(corrplot)&#xA;corr &amp;lt;- cor(y) &#xA;par(ask = TRUE)&#xA;corrplot(corr, order = &quot;hclust&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm trying use corrplot on my dataset. The original dataset has 25000 rows and 100 columns. I transposed to 25000 columns and 100 rows and finally applied corrplot. And failed!&#xA;If any help would be appreciated. [note:1st column is and first row are names and others are values (like -0.9 or 2.5 or 14449.45 etc.)]&lt;/p&gt;&#xA;" OwnerUserId="2895" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-01-27T17:53:25.883" LastActivityDate="2011-01-27T17:53:25.883" Title="Error in cor(y): allocMatrix: too many elements specified" Tags="&lt;r&gt;&lt;correlation&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="6614" PostTypeId="2" ParentId="6601" CreationDate="2011-01-27T11:08:52.113" Score="4" ViewCount="" Body="&lt;p&gt;From my personal experience the concept of &lt;a href=&quot;http://en.wikipedia.org/wiki/Likelihood_function&quot; rel=&quot;nofollow&quot;&gt;likelihood&lt;/a&gt; can also cause quite a lot of stir, especially for non-statisticians. As wikipedia says, it is very often mixed up with the concept of probability, which is not exactly correct.&lt;/p&gt;&#xA;" OwnerUserId="22" LastActivityDate="2011-01-27T11:08:52.113" CommunityOwnedDate="2011-01-27T11:08:52.113" />
  <row Id="6615" PostTypeId="2" ParentId="6557" CreationDate="2011-01-27T11:45:40.877" Score="5" ViewCount="" Body="&lt;p&gt;Here's my version with your simulated data set:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x1 &amp;lt;- rnorm(100,2,10)&#xA;x2 &amp;lt;- rnorm(100,2,10)&#xA;y &amp;lt;- x1+x2+x1*x2+rnorm(100,1,2)&#xA;dat &amp;lt;- data.frame(y=y,x1=x1,x2=x2)&#xA;res &amp;lt;- lm(y~x1*x2,data=dat)&#xA;z1 &amp;lt;- z2 &amp;lt;- seq(-1,1)&#xA;newdf &amp;lt;- expand.grid(x1=z1,x2=z2)&#xA;&#xA;library(ggplot2)&#xA;p &amp;lt;- ggplot(data=transform(newdf, yp=predict(res, newdf)), &#xA;            aes(y=yp, x=x1, color=factor(x2))) + stat_smooth(method=lm)&#xA;p + scale_colour_discrete(name=&quot;x2&quot;) + &#xA;  labs(x=&quot;x1&quot;, y=&quot;mean of resp&quot;) + &#xA;  scale_x_continuous(breaks=seq(-1,1)) + theme_bw()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I let you manage the details about x/y-axis labels and legend positioning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/rPz0z.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-01-27T11:45:40.877" CommentCount="2" />
  <row Id="6616" PostTypeId="1" CreationDate="2011-01-27T12:39:56.637" Score="1" ViewCount="79" Body="&lt;p&gt;Hello,&#xA;I'm doing some data analysis and need some help.&#xA;I've got a bunch of time series $q(t)$, from which I compute mean first passage times, ie. the time $\tau$ from $q&amp;gt;q_0$ to $q_0$, resulting in profiles $\tau(q)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Over each of these $\tau(q)$ sets I compute a (weighted) average $\tau_{mean}$. Together these $\tau_{mean}$ make up a curve $\tau_{mean}(\eta)$, as I'm varying some external parameter $\eta$ for each time series.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I did some fits for this $\tau_{mean}(\eta)$ curve, first a simple linear fit $\tau_{mean}(\eta) = \tau_i + \tau_w\cdot\eta$ and a power law fit $\tau_{mean}(\eta) = \tau_i\cdot\eta^\alpha$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to compare those fits to each other and of course somehow the errors resulting from the averaging above should be somehow incorporated in the goodness-of-fit test. I'm a bit puzzled how to do this. I was already reading some stuff on hierarchial linear models, but I'm no statistician and I'm still very confused.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another problem is that I also did the power law fit using linear regression, which should be avoided as stated here: &lt;a href=&quot;http://arxiv.org/abs/0706.1062&quot; rel=&quot;nofollow&quot;&gt;http://arxiv.org/abs/0706.1062&lt;/a&gt;&#xA;But to be honest, I don't really understand how to incorporate the concepts presented there into my problem. As I understand, there they mainly deal with power-laws in the long-tail of some data whereas I have a complete power law fit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope you guys can help me with my problem. Thanks very much, Greetings&#xA;Julius&lt;/p&gt;&#xA;" OwnerUserId="2943" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-01-27T17:59:14.597" LastActivityDate="2011-01-27T17:59:14.597" Title="Goodness of fit for multi-level-data fits" Tags="&lt;goodness-of-fit&gt;&lt;multilevel-analysis&gt;&lt;fitting&gt;" />
  <row Id="6617" PostTypeId="2" ParentId="6601" CreationDate="2011-01-27T13:22:27.327" Score="3" ViewCount="" Body="&lt;p&gt;I think the question is interpretable in two ways, which will give very different answers:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) For people studying statistics, particularly at a relatively advanced level, what is the hardest concept to grasp?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Which statistical concept is misunderstood by the most people?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For 1) I don't know the answer at all.  Something from measure theory, maybe?  Some type of integration?  I don't know.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For 2) p-value, hands down.&lt;/p&gt;&#xA;" OwnerUserId="686" LastActivityDate="2011-01-27T13:22:27.327" CommunityOwnedDate="2011-01-27T13:22:27.327" CommentCount="1" />
  <row Id="6618" PostTypeId="2" ParentId="6601" CreationDate="2011-01-27T13:29:56.687" Score="3" ViewCount="" Body="&lt;p&gt;I think that very few scientists understand this basic point: It is only possible to interpret results of statistical analyses at face value, if every step was planned in advance. Specifically:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Sample size has to be picked in advance. It is not ok to keep analyzing the data as more subjects are added, stopping when the results looks good. &lt;/li&gt;&#xA;&lt;li&gt;Any methods used to normalize the data or exclude outliers must also be decided in advance. It isn't ok to analyze various subsets of the data until you find results you like.&lt;/li&gt;&#xA;&lt;li&gt;And finally, of course, the statistical methods must be decided in advance. Is it not ok to analyze the data via parametric and nonparametric methods, and pick the results you like. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Exploratory methods can be useful to, well, explore. But then you can't turn around and run regular statistical tests and interpret the results in the usual way.&lt;/p&gt;&#xA;" OwnerUserId="25" LastEditorUserId="25" LastEditorDisplayName="" LastEditDate="2011-01-28T14:51:32.457" LastActivityDate="2011-01-28T14:51:32.457" CommunityOwnedDate="2011-01-27T13:29:56.687" CommentCount="7" />
  <row Id="6619" PostTypeId="1" AcceptedAnswerId="6620" CreationDate="2011-01-27T13:40:46.323" Score="2" ViewCount="64" Body="&lt;p&gt;...2 to 5 questions answered correctly, out of 20 of them? Each question has 5 choices. Probability of getting one right is 1/5. Probability of getting exactly 1 right is ${20 \choose 1} p^1  q^{19}$, with $p=P(\mathrm{right})$ and $q=P(\mathrm{wrong})$  (which I managed to understand and calculate). However how do I calculate for the problem above?&lt;/p&gt;&#xA;" OwnerUserId="1833" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-01-27T16:41:24.930" LastActivityDate="2011-01-27T16:41:24.930" Title="Probability of getting between..." Tags="&lt;probability&gt;&lt;homework&gt;&lt;binomial&gt;" AnswerCount="1" CommentCount="4" />
  <row Id="6620" PostTypeId="2" ParentId="6619" CreationDate="2011-01-27T13:56:41.140" Score="4" ViewCount="" Body="&lt;p&gt;Hint: sum up  the probabilities. The probability that exactly $k$ answers are answered correctly is $${20 \choose k}\left(\frac{1}{5}\right)^k\left(\frac{4}{5}\right)^{20-k}.$$ In your case you have $k=2,3,4,5$.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-01-27T13:56:41.140" CommentCount="4" />
  <row Id="6621" PostTypeId="2" ParentId="6577" CreationDate="2011-01-27T14:51:14.593" Score="1" ViewCount="" Body="&lt;p&gt;Why not display the raw data?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df &amp;lt;- data.frame(group=sample(LETTERS, 500, T, log(2:27)), &#xA;                 x=sample(0:20, 500, T), &#xA;                 y=sample(0:20, 500, T))&#xA;df$ratio &amp;lt;- with(df, (x-y)/(x+y))&#xA;&#xA;library(ggplot2)&#xA;qplot(group, ratio, data = df) + &#xA;  stat_summary(fun.y = mean, colour = &quot;red&quot;, size = 2, geom = &quot;point&quot;)&#xA;&#xA;qplot(group, ratio, data = df) + &#xA;  stat_summary(fun.data = &quot;mean_cl_boot&quot;, colour = &quot;red&quot;, geom = &quot;crossbar&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="46" LastActivityDate="2011-01-27T14:51:14.593" CommentCount="2" />
  <row Id="6622" PostTypeId="2" ParentId="6613" CreationDate="2011-01-27T14:53:11.207" Score="0" ViewCount="" Body="&lt;p&gt;I would guess that you don't have enough memory.  The correlation matrix for 25,000 columns will be 25,000 x 25000 which is about 4 gig. &lt;code&gt;(25000 ^ 2 * 8 ) / 1024 ^ 3)&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="46" LastActivityDate="2011-01-27T14:53:11.207" CommentCount="3" />
  <row Id="6623" PostTypeId="2" ParentId="4466" CreationDate="2011-01-27T14:57:16.553" Score="1" ViewCount="" Body="&lt;p&gt;Good suggestions, I've got plenty of things to look into now.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Remember, one extremely important consideration is making sure that the work is &quot;correct&quot; in the first place.  This is the role that tools like &lt;a href=&quot;http://en.wikipedia.org/wiki/Sweave&quot; rel=&quot;nofollow&quot;&gt;Sweave&lt;/a&gt; play, by increasing the chances that what you did, and what you said you did, are the same thing.&lt;/p&gt;&#xA;" OwnerUserId="1434" LastActivityDate="2011-01-27T14:57:16.553" CommentCount="1" />
  <row Id="6624" PostTypeId="1" AcceptedAnswerId="6631" CreationDate="2011-01-27T15:04:03.230" Score="4" ViewCount="145" Body="&lt;p&gt;I've asked the &lt;a href=&quot;http://math.stackexchange.com/questions/19180/detect-abnormal-points-in-point-cloud&quot;&gt;same question&lt;/a&gt; at Math SE, but the suggestion is that probably this question belongs here. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given a list of &lt;a href=&quot;http://en.wikipedia.org/wiki/Point_cloud&quot; rel=&quot;nofollow&quot;&gt;point cloud&lt;/a&gt; in terms of $(x,y,z)$ how to determine abnormal points? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The motivation is this. We need to reconstruct a terrain surface out from those point cloud, which the surveyors obtain when doing field survey. The surveyors would take an equipment and record a sufficient sample of the $x,y,z$ of a terrain. Those points will be recorded into a CAD program.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is that the CAD file can be corrupted from time to time with the introduction of &quot;abnormal&quot; points. Those points do not fit into the terrain surface generally, and tend to have erroneous $z$ value ( i.e., the $z$ value is outside of the normal range). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am aware that the definition of abnormal points is a bit loose; and I can't come up with a rigorous definition of it. However, I know what is an abnormal point when I see the drawing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given all these constraint, is there any algorithm to detect these kinds of abnormal points? &lt;/p&gt;&#xA;" OwnerUserId="175" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-01-27T17:54:02.543" LastActivityDate="2011-01-28T09:14:13.320" Title="Detecting abnormal points in point cloud" Tags="&lt;outliers&gt;&lt;spatial&gt;" AnswerCount="4" CommentCount="0" FavoriteCount="1" />
  <row Id="6626" PostTypeId="2" ParentId="6252" CreationDate="2011-01-27T15:37:09.260" Score="0" ViewCount="" Body="&lt;p&gt;As others have pointed out, there are many measures of clustering &quot;quality&quot;;&#xA;most programs minimize SSE.&#xA;No single number can tell much about noise in the data,&#xA;or noise in the method,&#xA;or flat minima &amp;mdash; low points in Saskatchewan.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So first try to visualize, get a feel for,&#xA;a given clustering, before reducing it to &quot;41&quot;.&#xA;Then make 3 runs: do you get SSEs 41, 39, 43 or 41, 28, 107 ?&#xA;What are the cluster sizes and radii ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Added:) Take a look at silhouette plots and silhouette scores, e.g. in&#xA;the book by Izenman,&#xA;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0387781889&quot; rel=&quot;nofollow&quot;&gt;Modern Multivariate Statistical Techniques&lt;/a&gt;&#xA;(2008, 731p, isbn 0387781889).&lt;/p&gt;&#xA;" OwnerUserId="557" LastEditorUserId="557" LastEditorDisplayName="" LastEditDate="2011-01-28T18:28:55.360" LastActivityDate="2011-01-28T18:28:55.360" />
  <row Id="6627" PostTypeId="2" ParentId="6613" CreationDate="2011-01-27T15:42:03.393" Score="1" ViewCount="" Body="&lt;p&gt;The &quot;allocMatrix: too many elements specified&quot; error is thrown in on line 170 of &lt;code&gt;R/src/main/array.c&lt;/code&gt; when nrow x ncol is greater than &lt;code&gt;INT_MAX&lt;/code&gt; (+2,147,483,647).  &lt;code&gt;INT_MAX&lt;/code&gt; is defined in the C standard library file &quot;limits.h&quot; and it is the same in the 32-bit and 64-bit toolchain used to build R, so no amount of RAM on a current 64-bit R build will solve your problem.&lt;/p&gt;&#xA;" OwnerUserId="1657" LastActivityDate="2011-01-27T15:42:03.393" />
  <row Id="6628" PostTypeId="2" ParentId="6570" CreationDate="2011-01-27T16:04:35.303" Score="2" ViewCount="" Body="&lt;p&gt;I think this whole &quot;force people to choose&quot; thing is just a complete red herring. People say it to me all the time. To me it sounds like &quot;force people to state the capital of Uzbekistan&quot;. They don't know, and forcing them won't make them know any better.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With that mini-rant over, my only sensible contribution is to say that you should always pilot surveys whenever you can. Pilot both versions, see who uses the &quot;don't know&quot; category in the one where it's included, and look at the distribution of responses. And talk to the people who filled it out. &quot;Were you sure of your answer?&quot; &quot;What made you say 'don't know' here&quot;- that kind of thing.&lt;/p&gt;&#xA;" OwnerUserId="199" LastActivityDate="2011-01-27T16:04:35.303" />
  <row Id="6629" PostTypeId="2" ParentId="6624" CreationDate="2011-01-27T16:10:51.673" Score="1" ViewCount="" Body="&lt;p&gt;Are the points relatively dense on your surface? Then I would suggest counting the number of points in a sphere around every point. Choose the radius of your sphere to be a bit less than the distance the &quot;abnormal&quot; points have to the regular surface - maybe half of what they typically have. Then throw out the points where the number of other points inside that sphere is very low. (I don't know if your outliers occur in small groups or if they are isolated points; this technique should work for either case.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If a naive implementation picks out the correct points but is too slow, and you're struggling to come up with a faster algorithm to do the same, then let us know. I'm sure we could come up with something :)&lt;/p&gt;&#xA;" OwnerUserId="2898" LastActivityDate="2011-01-27T16:10:51.673" CommentCount="4" />
  <row Id="6630" PostTypeId="1" CreationDate="2011-01-27T16:18:30.880" Score="9" ViewCount="205" Body="&lt;p&gt;I need some guidance on the appropriate level of pooling to use for difference of means tests on time series data.  I am concerned about temporal and sacrificial pseudo-replication, which seem to be in tension on this application.  This is in reference to a mensural study rather than a manipulative experiment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Consider a monitoring exercise&lt;/strong&gt;:  A system of sensors measures dissolved oxygen (DO) content at many locations across the width and depth of a pond.  Measurements for each sensor are recorded twice daily, as DO is known to vary diurnally.  The two values are averaged to record a daily value.  Once a week, the daily results are aggregated spatially to arrive at a single weekly DO concentration for the whole pond.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Those weekly results are reported periodically, and further aggregated – weekly results are averaged to give a monthly DO concentration for the pond.  The monthly results are averaged to give an annual value.  The annual averages are themselves averaged to report decadal DO concentrations for the pond.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The goal is to answer questions such as:  Was the pond's DO concentration in year X higher, lower, or the same as the concentration in year Y?  Is the average DO concentration of the last ten years different than that of the prior decade?  The DO concentrations in a pond respond to many inputs of large magnitude, and thus vary considerably.  A significance test is needed.  The method is to use a T-test comparison of means.  Given that the decadal values are the mean of the annual values, and the annual values are the mean of the monthly values, this seems appropriate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Here’s the question&lt;/strong&gt; – you can calculate the decadal means and the T-values of those means from the monthly DO values, or from the annual DO values.  The mean doesn’t change of course, but the width of the confidence interval and the T-value does.  Due to the order of magnitude higher N attained by using monthly values, the CI often tightens up considerably if you go that route.  This can give the opposite conclusion vs using the annual values with respect to the statistical significance of an observed difference in the means, using the same test on the same data.  &lt;strong&gt;What is the proper interpretation of this discrepancy?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you use the monthly results to compute the test stats for a difference in decadal means, are you running afoul of temporal pseudoreplication?  If you use the annual results to calc the decadal tests, are you sacrificing information and thus pseudoreplicating?&lt;/p&gt;&#xA;" OwnerUserId="2948" LastActivityDate="2011-05-06T12:32:01.043" Title="What temporal resolution for time series significance test?" Tags="&lt;time-series&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="1" />
  <row Id="6631" PostTypeId="2" ParentId="6624" CreationDate="2011-01-27T16:39:42.483" Score="6" ViewCount="" Body="&lt;p&gt;An outlier detector for your irregular (&quot;vector&quot;) point data is available in GRASS as &lt;a href=&quot;http://grass.osgeo.org/grass64/manuals/html64_user/v.outlier.html&quot; rel=&quot;nofollow&quot;&gt;v.outlier&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An overview of spatial outlier detection methods appears in a &lt;a href=&quot;http://www.geo.upm.es/postgrado/CarlosLopez/papers/AHybridApproachToDetectSpatialTemporalOutliers.pdf&quot; rel=&quot;nofollow&quot;&gt;2004 paper by Cheng and Li&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.casa.arizona.edu/data/rnr_420/hutchinson_article.pdf&quot; rel=&quot;nofollow&quot;&gt;An older method&lt;/a&gt;, specialized for topographic data, relies on &quot;drainage enforcement&quot; (making the water flow downhill continuously without accumulating in sinks).  That can find some of the outliers, but probably not all of them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A more generic method is to adapt a local indicator of spatial variability, such as a &lt;a href=&quot;http://en.wikipedia.org/wiki/Indicators_of_spatial_association&quot; rel=&quot;nofollow&quot;&gt;local Moran's I&lt;/a&gt; statistic, to identify points that are &quot;too far&quot; away from the surface.  &lt;a href=&quot;http://geodacenter.asu.edu/software/downloads&quot; rel=&quot;nofollow&quot;&gt;GeoDa&lt;/a&gt; can compute such statistics.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-01-27T16:39:42.483" />
  <row Id="6632" PostTypeId="2" ParentId="6624" CreationDate="2011-01-27T16:41:47.253" Score="1" ViewCount="" Body="&lt;p&gt;You could fit some sort of smooth function for $z(x,y)$, perhaps using &lt;a href=&quot;http://en.wikipedia.org/wiki/Local_regression&quot; rel=&quot;nofollow&quot;&gt;locally weighted scatterplot smoothing&lt;/a&gt; (LOWESS or LOESS), then look for points where the residual for $z$ (i.e. the difference between the observed and fitted values) is greater than some fixed multiple of the standard error of prediction. That should be straightforward e.g. in &lt;strong&gt;&lt;code&gt;R&lt;/code&gt;&lt;/strong&gt; using the &lt;code&gt;loess&lt;/code&gt; function in the standard &lt;code&gt;stats&lt;/code&gt; package.&lt;/p&gt;&#xA;" OwnerUserId="449" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-01-28T09:14:13.320" LastActivityDate="2011-01-28T09:14:13.320" CommentCount="3" />
  <row Id="6634" PostTypeId="2" ParentId="6624" CreationDate="2011-01-27T17:48:28.370" Score="0" ViewCount="" Body="&lt;p&gt;I think this problem relies only in the outliers of variable $z$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The surveyor scans a grid of $x$,$y$ points that are &quot;well-behaved&quot;. On the other hand $z$ points may contain abnormal values (in statistics we call them outliers).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would suggest to explore the values of $z$, and the plot of $(x,y,z)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From those plots it will be clear that abnormal values of $z$ occur isolated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lets suppose that we have a rectangular grid of points $x_k, y_k$, at each point of the grid we have a value of $z$ that we will denote as $z_{k,k}$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, if we think $z_k$ is an abnormal point, we expect a low correlation among $(x_k,y_k,z_{k,k})$ and $(x_{k+1},y_{k},z_{k+1,k})$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, we expect a low correlation between $(x_k,y_k,z_{k,k})$ and its neighbors $\mathcal{N}(k,k)$. A way to measure the spatial correlation between the point $(k,k)$ and its neighborhood is the empirical variogram defined by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\hat{\gamma}(k,k) = \frac{1}{\#\mathcal{N}(k,k)} \sum_{(i,j), (p,q) \in \mathcal{N}(k,k)} | z_{i,j} - z_{p,q} |^2$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you calculate $\hat{\gamma}(k,k)$ for the whole grid you can be sure that outliers in the empirical variogram are indeed abnormal points.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A boxplot can be useful to identify the outliers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using the variogram is a way to ensure that you are actually reading an abnormal point. Suppose that your surveyors are scanning a slope, then you will notice that the $z_{k,k}$ has high values, but also their neighbors. In case the point is abnormal only $z_{k,k}$ will have high values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;NOTE: If you're sure that your surveyors are analyzing a rather flat surface, get rid of the variogram and make a boxplot of $z$, any outlier identified by the boxplot is an abnormal point.&lt;/p&gt;&#xA;" OwnerUserId="2902" LastActivityDate="2011-01-27T17:48:28.370" CommentCount="3" />
  <row Id="6635" PostTypeId="2" ParentId="6601" CreationDate="2011-01-27T17:57:12.117" Score="6" ViewCount="" Body="&lt;p&gt;Tounge firmly in cheek: For frequentists, the Bayesian concept of probability; for Bayesians, the frequentist concept of probability.  ;o)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Both have merit of course, but it can be very difficult to understand why one framework is interesting/useful/valid if your grasp of the other is too firm.  Cross-validated is a good remedy as asking questions and listening to answers is a good way to learn.&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2011-01-27T17:57:12.117" CommunityOwnedDate="2011-01-27T17:57:12.117" CommentCount="2" />
  <row Id="6636" PostTypeId="1" AcceptedAnswerId="6666" CreationDate="2011-01-27T18:42:03.703" Score="5" ViewCount="144" Body="&lt;p&gt;(redirected here from mathoverflow.net)&#xA;Hello, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;At work I was asked the probability of a user hitting an outage on the website. I have some following metrics. Total system downtime = 500,000 seconds a year. Total amount of seconds a year = 31,556,926 seconds. Thus, p of system down = 0.159 or 1.59%&#xA;We can also assume that downtime occurs evenly for a period of approximately 2 hours per week.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, here is the tricky part. We have a metric for amount of total users attempting to use the service = 16,000,000 during the same time-frame. However, these are subdivided, in the total time spent using the service. So, lets say we have 7,000,000 users that spend between 0 - 30 seconds attempting to use the service. So for these users what is the probability of hitting the system when it is unavailable? (We can assume an average of 15 seconds spent total if this simplifies things)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I looked up odds ratios and risk factors, but I am not sure how to calculate the probability of the event occurring at all.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S. I was given a possible answer, at &lt;a href=&quot;http://mathoverflow.net/questions/52816/probability-calculation-system-uptime-likelihood-of-occurence&quot; rel=&quot;nofollow&quot;&gt;http://mathoverflow.net/questions/52816/probability-calculation-system-uptime-likelihood-of-occurence&lt;/a&gt; and was following the advice on posting the question in the most appropriate forum.&lt;/p&gt;&#xA;" OwnerUserId="2950" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-29T00:51:46.293" LastActivityDate="2011-04-29T00:51:46.293" Title="Probability calculation, system uptime, likelihood of occurence" Tags="&lt;probability&gt;&lt;odds-ratio&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="2" />
  <row Id="6637" PostTypeId="1" AcceptedAnswerId="6639" CreationDate="2011-01-27T19:15:38.773" Score="3" ViewCount="96" Body="&lt;p&gt;Hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let us take two formulations of the L2-SVM optimization problem, one constrained: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$min_{\alpha,b} ||w||_2^2 + C \sum_{i=1}^n {\xi_{i}^2}$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;s.t $ y_i(w^T x_i +b) \geq 1 - \xi_i$&lt;br&gt;&#xA;and $\xi_i \geq 0 \forall i$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and one unconstrained:  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$min_{\alpha,b} ||w||_2^2 + C \sum_{i=1}^n max(0,1 - y_i (w^T x_i + b))^2$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the difference between those two formulations of the optimization problem? Is one better than the other?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope I didn't make any mistake in the equations. Thanks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Update : I took the unconstrained formulation from &lt;a href=&quot;http://www.kyb.mpg.de/publications/attachments/primal_%5b0%5d.pdf&quot; rel=&quot;nofollow&quot;&gt;Olivier Chapelle's work&lt;/a&gt;. It seems that people use the unconstrained optimization problem when they want to work on the primal and the other way around when they want to work on the dual, I was wondering why?&lt;/p&gt;&#xA;" OwnerUserId="1320" LastEditorUserId="1320" LastEditorDisplayName="" LastEditDate="2011-01-27T21:30:56.150" LastActivityDate="2011-01-27T21:30:56.150" Title="Constrained versus unconstrained formulation of SVM optimisation" Tags="&lt;optimization&gt;&lt;svm&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="6638" PostTypeId="1" AcceptedAnswerId="6641" CreationDate="2011-01-27T19:27:56.060" Score="4" ViewCount="108" Body="&lt;p&gt;I have been running a linear regression where my dependent variable is a composite. By this I mean that it is built up of components that are added and multiplied together. Specifically, for the composite variable A:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;A = (B*C + D*E + F*G + H*I + J*K + L*M)*(1 - N)*(1 + O*P)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;None of the component variables are used as independent variables (the only independent variables are dummy variables). The component variables are mostly (though not completely) independent of one another.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently I just run a regression with A as the DV, to estimate each dummy variable's effect on A. But I would also like to estimate each dummy variable's effect on the separate components of A (and in the future I hope to try applying separate priors for each component). To do this I have been running several separate regressions, each with a different one of the components as the DV (and using the same IVs for all the regressions). If I do this, should I expect that for a given dummy IV, I could recombine the coefficient estimates from all the separate regressions (using the formula listed above) and get the same value as I get for that IV when I run the composite A regression? Am I magnifying the coefficient standard errors by running all these separate regressions and then trying to recombine the values (there is a lot of multicollinearity in the dummy variables)? Is there some other structure than linear regression that would be better for a case like this?&lt;/p&gt;&#xA;" OwnerUserId="1090" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-01-27T19:39:57.683" LastActivityDate="2011-01-27T20:29:25.263" Title="Composite dependent variable" Tags="&lt;regression&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="6639" PostTypeId="2" ParentId="6637" CreationDate="2011-01-27T19:48:05.287" Score="2" ViewCount="" Body="&lt;p&gt;It seems to me that at the solution of the first problem, the inequality constraint becomes an equality, i.e. $1 - \xi_i = y_i(w^Tx_i + b)$, because we are minimising the $\xi_i$s and the smallest value that satisfies the constraint ocurrs at equality.  So as $\xi_i \geq 0$, $\xi_i = max(0, 1 - y_i(w^Tx_i+b))$, which substituting gives something rather similar to your second formulation.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having checked the paper by Chapelle, it looks like the second formulation is missing a &quot;1 -&quot; in the second half of the max operation (see definition of L(.,.) following equation 2.8).  In that case both formulations are identical, they are both equivalent representations of the primal optimisation problem (the dual formulation is in terms of the Lagrange multipliers $\alpha_i$).  The advantages and disadvantages are therefore purely computational.&lt;/p&gt;&#xA;" OwnerUserId="887" LastEditorUserId="887" LastEditorDisplayName="" LastEditDate="2011-01-27T21:07:00.313" LastActivityDate="2011-01-27T21:07:00.313" />
  <row Id="6640" PostTypeId="1" CreationDate="2011-01-27T19:52:14.727" Score="1" ViewCount="91" Body="&lt;p&gt;I have what seems like a fairly common business statistics scenario: I need to compare one group of stores to another group of stores and be able to say if their difference in sales is statistically different.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example:&#xA;Group A ($n_A$ = 30 stores) participated in a promotion and saw an avg sales increase for this month compared to the same month last year of $\bar{x}_A$ and a standard deviation of $s_A$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Group B ($n_B$ = 50 stores) did not participate in the promotion and had avg sales increase of $\bar{x}_B$ and corresponding $s_B$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I realize there are a number of other variables but, in theory, I should be able to say with some certainty that there is or isn't any difference between stores that took or did not participate in a promotion, right?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can I do a standard comparison of means test? Does it make a difference that these stores comprise of the entire population? Or is it not the entire population and I should be looking at average increase for multiple months and multiple years?&lt;/p&gt;&#xA;" OwnerUserId="2951" LastActivityDate="2011-01-27T22:45:31.127" Title="How should I compare average store sales change across time?" Tags="&lt;multiple-comparisons&gt;&lt;averages&gt;&lt;business-intelligence&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="6641" PostTypeId="2" ParentId="6638" CreationDate="2011-01-27T20:29:25.263" Score="3" ViewCount="" Body="&lt;p&gt;(1)  &lt;em&gt;Should I expect to obtain the same fits using the two models?&lt;/em&gt;  No.  Let's look at what's going on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(a) In the regression of $A$ directly--I'll call it the &quot;monolithic model,&quot; the model is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$A_j = \sum{\beta_i X_{ij}} + \epsilon_j,$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;with the cases indexed by $j$, the variables (including a constant, if any) indexed by $i$, and with the $\epsilon_j$ random variables of zero mean.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(b) In the &quot;composite model&quot; you have a series of regressions.  To be systematic, let's rename the variables $B$ though $P$ as $B_1, B_2, \ldots, B_{15}$ and write &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$A = (B C + \cdots + L M)(1 - N)(1 + O P) = f(B_1, B_2, \ldots, B_{15}) = f(\mathbf{B}).$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each component model in the composite is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$B_{kj} = \sum_{i}{\gamma_{ki} X_{ij}} + \delta_{kj}, \quad k=1,2,\ldots,15.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$A_j = f(\mathbf{B}_j) = f(\sum_{i}{\gamma_{ki} X_{ij}} + \delta_{kj}).$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To see how this differs from the monolithic model, let's consider the simpler case where $A = B_1 B_2 = f(B_1,B_2)$, which gives&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\eqalign{&#xD;&#xA;A_j &amp;amp;= f(B_{1j}, B_{2j}) \cr&#xD;&#xA;    &amp;amp;=\left(\sum_{i}{\gamma_{1i} X_{ij}} + \delta_{1j}\right)\left(\sum_{l}{\gamma_{2l} X_{lj}} + \delta_{2j}\right) \cr&#xD;&#xA;    &amp;amp;=\sum_{il}{\gamma_{1i}\gamma_{2l}X_{ij}X_{lj}} + \sum_{i}{\left(\gamma_{1i}\delta_{2j} + \gamma_{2i}\delta_{1j}\right) X_{ij}} + \delta_{1j}\delta_{2j}.&#xD;&#xA;}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note the differences:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;If the error terms $\delta_{ij}$ are not independent, the expectation of $\delta_{1j}\delta_{2j}$ will be nonzero, introducing a bias.&lt;/li&gt;&#xA;&lt;li&gt;If the error terms are independent, the expectation of $\delta_{1j}\delta_{2j}$ is zero, which is good, but the expectation of $A$ equals $\sum_{il}{\gamma_{1i}\gamma_{1l}X_{ij}X_{lj}}$.  In this model there are only interaction terms!&lt;/li&gt;&#xA;&lt;li&gt;If you include &lt;em&gt;all&lt;/em&gt; interaction terms in the monolithic model and the $\delta$s are independent, then the interaction coefficients for $X_iX_l$ can be compared to the coefficients $\gamma_{1i}\gamma_{2l}+\gamma_{2i}\gamma_{1l}$ that appear in the composite model.  However, we cannot expect equality, because there are fewer $\gamma$ parameters in the composite model than there would be interaction parameters in the monolithic model.  (In other words, the structure of the composite model introduces algebraic relationships among the coefficients that the monolithic model cannot enforce.)&lt;/li&gt;&#xA;&lt;li&gt;The distribution of the random part of this model, $\sum_{i}{\left(\gamma_{1i}\delta_{2j} + \gamma_{2i}\delta_{1j}\right) X_{ij}} + \delta_{1j}\delta_{2j}$, depends on the data $X_{ij}$, on the parameters $\gamma_{ki}$, and on the products of $\delta_{kj}$.  As such it will likely be heteroscedastic--that is, depending on the values of the variables--and have a different (and complex) distributional shape even when the $\delta_{kj}$ have a &quot;nice&quot; shape (such as normal).&lt;/li&gt;&#xA;&lt;li&gt;The analog of $\beta_j$ in the composite model, $\gamma_{1i}\delta_{2j} + \gamma_{2i}\delta_{1j}$, depends on the residuals in the component models of the composite.  Therefore we would not expect these to have equal values.  In fact, they would only tend (probabilistically) to be equal when the $\beta_j$ are zero, because the expectation of the analog is zero.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Whence &lt;strong&gt;the monolithic and composite models are almost entirely different.&lt;/strong&gt;  There seems to be no valid way even to compare coefficients and where they can be compared we cannot expect them to be equal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(2) &lt;em&gt;Am I magnifying the coefficient standard errors?&lt;/em&gt;  By the same reasoning, the standard errors cannot even be compared.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is nothing overtly wrong with using either model, but they say two different things about the behavior of $A$.  The choice between them should not be made on the basis of coefficient standard errors but on the basis of which (if either) appears to be more appropriate and useful for the intended purpose.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(3) &lt;em&gt;Are there alternatives to linear regression?&lt;/em&gt;  Of course.  You could extend the models in many nonlinear ways.  But normally we begin with the simplest models that can reasonably be expected to serve our analytical purposes.  Note that the monolithic model uses far fewer parameters than the composite model and therefore is &lt;em&gt;a fortiori&lt;/em&gt; simpler.  &lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-01-27T20:29:25.263" CommentCount="1" />
  <row Id="6642" PostTypeId="1" AcceptedAnswerId="6676" CreationDate="2011-01-27T20:52:04.727" Score="4" ViewCount="131" Body="&lt;p&gt;Let $S_n = \frac{1}{n}\sum_{i=1}^n X_i$, and $T_n = \frac{1}{n}\sum_{j=1}^nY_i$, where &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The $X_i$ are iid, the $Y_i$ are iid (with a different law)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$X_i$, and $Y_i$ are dependent &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For $i\neq j$, $X_i$ and $Y_j$ are independent. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a central limit type result for $S_n^2 - T_n^2$?&lt;/p&gt;&#xA;" OwnerUserId="2952" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-01-28T13:05:46.070" LastActivityDate="2011-01-28T18:48:44.800" Title="Limiting distribution of a squared sum of random variables" Tags="&lt;central-limit-theorem&gt;&lt;delta-method&gt;" AnswerCount="1" CommentCount="11" />
  <row Id="6643" PostTypeId="1" AcceptedAnswerId="6645" CreationDate="2011-01-27T21:09:18.177" Score="3" ViewCount="86" Body="&lt;p&gt;Is there a closed form solution for this inverse CDF?&lt;/p&gt;&#xA;" OwnerUserId="2808" LastEditorUserId="2808" LastEditorDisplayName="" LastEditDate="2011-01-28T14:01:35.860" LastActivityDate="2011-01-28T14:44:30.703" Title="What is the closed form solution for the inverse CDF for Epanechnikov" Tags="&lt;distributions&gt;&lt;kernel&gt;&lt;cdf&gt;" AnswerCount="1" />
  <row Id="6644" PostTypeId="2" ParentId="6640" CreationDate="2011-01-27T21:51:47.840" Score="2" ViewCount="" Body="&lt;p&gt;If all of the stores were included in the study rather than a sample, then you could make conclusions without using probability statements or statistics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But if you want to use a subsample to make inference about the larger population or make forecasts, then the use of statistics is appropriate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A standard comparison of means test for which your data meet the assumptions would be appropriate. For example, if you want to make inference about the effect of promotion on the larger population of stores, e.g. to evaluate the null hypothesis that there is no effect, the &lt;a href=&quot;http://en.wikipedia.org/wiki/Student%27s_t-test&quot; rel=&quot;nofollow&quot;&gt;student's t test&lt;/a&gt; with unequal sample sizes and unequal variance, a.k.a. &lt;a href=&quot;http://en.wikipedia.org/wiki/Welch%27s_t_test&quot; rel=&quot;nofollow&quot;&gt;Welch's t-test&lt;/a&gt; is a widely used and robust method.&lt;/p&gt;&#xA;" OwnerUserId="1381" LastEditorUserId="1381" LastEditorDisplayName="" LastEditDate="2011-01-27T22:45:31.127" LastActivityDate="2011-01-27T22:45:31.127" CommentCount="4" />
  <row Id="6645" PostTypeId="2" ParentId="6643" CreationDate="2011-01-27T22:41:50.163" Score="3" ViewCount="" Body="&lt;p&gt;You mean for a random variable with a single Epanechnikov kernel as PDF? Well, the PDF is $\frac{3}{4}(1-u^2)$, so the CDF is $\frac{1}{4}(2 + 3 u - u^3)$. Inverting this in Maple leads to three solutions, of which  $u = -1/2\,{\frac { \left( 1-2\,t+2\,i\sqrt {t}\sqrt {1-t} \right) ^{2/3}+1 +i\sqrt {3} \left( 1-2\,t+2\,i\sqrt {t}\sqrt {1-t} \right) ^{2/3}-i&#xD;&#xA;\sqrt {3}}{\sqrt [3]{1-2\,t+2\,i\sqrt {t}\sqrt {1-t}}}}$ seems to be the right one (where the third roots return the main branch). Of course this is a real value for real values of $t$ between 0 and 1; I currently don't have time to make this come out right but I'll try and revisit in a couple of days. If someone else sees it, it would be great if you could leave a comment.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Note whuber's comment below for a much nicer formula.&lt;/p&gt;&#xA;" OwnerUserId="2898" LastEditorUserId="2898" LastEditorDisplayName="" LastEditDate="2011-01-28T14:44:30.703" LastActivityDate="2011-01-28T14:44:30.703" CommentCount="5" />
  <row Id="6646" PostTypeId="2" ParentId="6601" CreationDate="2011-01-27T23:29:07.293" Score="8" ViewCount="" Body="&lt;p&gt;What is the meaning of &quot;degrees of freedom&quot;? How about df that are not whole numbers?&lt;/p&gt;&#xA;" OwnerUserId="2954" LastEditorUserId="-1" LastEditorDisplayName="" LastEditDate="2011-01-29T19:13:03.407" LastActivityDate="2011-01-29T19:13:03.407" CommunityOwnedDate="2011-01-27T23:29:07.293" />
  <row Id="6647" PostTypeId="1" CreationDate="2011-01-27T23:33:18.597" Score="1" ViewCount="56" Body="&lt;p&gt;I have 100,000 metal parts on a production line a week. Parts can either be fine, or defective. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Random parts are automatically pushed onto a conveyor belt to be checked by a human. We then do a simplistic calculation like: After checking 1000 parts, 5 are defective, 5/1000 = 0.5% defective, so potentially 0.5% * 100,000 = 500 are defective in all.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We assume the human checking is flawless.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone give any pointers on the type of statistics required to calculate &quot;how many parts need to be checked to be 90% confident of the defect rate&quot;?&lt;/p&gt;&#xA;" OwnerUserId="2955" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-01-27T23:39:16.577" LastActivityDate="2011-01-27T23:39:16.577" Title="Sampling and confidence calculations" Tags="&lt;binomial&gt;&lt;sample-size&gt;" CommentCount="2" />
  <row Id="6648" PostTypeId="1" AcceptedAnswerId="6649" CreationDate="2011-01-27T23:34:01.300" Score="2" ViewCount="64" Body="&lt;p&gt;An average of &lt;strong&gt;n&lt;/strong&gt; birds fly through an area, in an hour, following a Poisson process. (I think this means the hours don't matter; there's no influence in the number of birds that fly through, at different parts of the day, hypothetically. Correct me if I'm wrong.)&#xA;P1 is the probability that exactly &lt;strong&gt;m&lt;/strong&gt; birds fly through between 12:00-14:00 (2 continuous hours). P2 is the probability that exactly &lt;strong&gt;m&lt;/strong&gt; birds fly through between 15:00-16:00 and 17:00-18:00 (a total of 2 hours still, but not together). Please notice m is the same for both situations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are the probabilities of P1 and P2 the same or I'm assuming something that's wrong?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Please tell me if you need more information, I can make it up)&lt;/p&gt;&#xA;" OwnerUserId="1833" LastActivityDate="2011-01-27T23:42:22.013" Title="Poisson process, time and probabilities" Tags="&lt;probability&gt;&lt;poisson&gt;" AnswerCount="1" />
  <row Id="6649" PostTypeId="2" ParentId="6648" CreationDate="2011-01-27T23:42:22.013" Score="4" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Yes, they are the same.&lt;/strong&gt;  Another crucial assumption in the Poisson process is that what happens now is independent of what happened a moment ago or what will happen in the next moment (or at any other moment, for that matter).  Therefore the distribution of events during any (measurable) period of time depends only on the length of time, not on how it is broken up.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-01-27T23:42:22.013" CommentCount="1" />
  <row Id="6650" PostTypeId="2" ParentId="6601" CreationDate="2011-01-27T23:45:50.293" Score="3" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Fiducial_inference&quot; rel=&quot;nofollow&quot;&gt;Fiducial inference&lt;/a&gt;&lt;/strong&gt;. Even Fisher admitted he didn't understand what it does, and he invented it.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-01-27T23:45:50.293" CommunityOwnedDate="2011-01-27T23:45:50.293" />
  <row Id="6651" PostTypeId="2" ParentId="6347" CreationDate="2011-01-28T00:18:46.790" Score="3" ViewCount="" Body="&lt;p&gt;PCA depends on the scaling of your columns. If you perform a PCA on matrix $X$, then rescale each column to be norm 1 (&lt;em&gt;i.e.&lt;/em&gt; divide by the two norm of each column), then perform a PCA on the transformed $X$, you will get different answers. I believe this is part of what 'small w.r.t. a particular row/column' is referring to in the original question. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the small elements in the matrix should contribute very little to a PCA/downsample/reconstruction operation. Perhaps you would be better served by determining, perhaps in a semiautomatic way, &lt;em&gt;how many PCs&lt;/em&gt; to take in the representation. For this purpose, you might want to look at the scaling of the eigenvalues you get from the SVD, and look for the 'knee' in the &lt;a href=&quot;http://janda.org/workshop/factor%20analysis/SPSS%20run/SPSS08.htm&quot; rel=&quot;nofollow&quot;&gt;scree plot&lt;/a&gt;, or if you know something about the small noise, you can rely on the distribution of eigenvalues of the Gramian matrix. For normally distributed noise, with equal variances, they should follow a &lt;a href=&quot;http://en.wikipedia.org/wiki/Marchenko%E2%80%93Pastur_distribution&quot; rel=&quot;nofollow&quot;&gt;Marchenko-Pastur Distribution&lt;/a&gt;, up to scaling. This will give you the limits of the eigenvalues you expect to see in the pure noise situation. Basing your PC cutoff on that limit may be fruitful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sorry this is somewhat vague, I do not think I fully understand what technically is desired from the OP.&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2011-01-28T00:18:46.790" CommentCount="1" />
  <row Id="6652" PostTypeId="1" AcceptedAnswerId="6801" CreationDate="2011-01-28T00:23:50.893" Score="17" ViewCount="628" Body="&lt;p&gt;I know roughly and informally what a confidence interval is.  However, I can't seem to wrap my head around one rather important detail:  According to Wikipedia:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;A confidence interval does not predict that the true value of the parameter has a particular probability of being in the confidence interval given the data actually obtained.  &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I've also seen similar points made in several places on this site.  A more correct definition, also from Wikipedia, is:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;if confidence intervals are constructed across many separate data analyses of repeated (and possibly different) experiments, the proportion of such intervals that contain the true value of the parameter will approximately match the confidence level&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Again, I've seen similar points made in several places on this site.  I don't get it.  If, under repeated experiments, the fraction of computed confidence intervals that contain the true parameter $\theta$ is $(1 - \alpha)$, then how can the probability that $\theta$ is in the confidence interval computed for the actual experiment be anything other than $(1 - \alpha)$?  I'm looking for the following in an answer:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Clarification of the distinction between the incorrect and correct definitions above.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;A formal, precise definition of a confidence interval that clearly shows why the first definition is wrong.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;A concrete example of a case where the first definition is spectacularly wrong, even if the underlying model is correct.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="1347" LastActivityDate="2011-05-12T02:58:02.370" Title="What, precisely, is a confidence interval?" Tags="&lt;confidence-interval&gt;&lt;definition&gt;" AnswerCount="5" CommentCount="5" FavoriteCount="3" />
  <row Id="6653" PostTypeId="1" AcceptedAnswerId="6839" CreationDate="2011-01-28T00:24:32.027" Score="14" ViewCount="492" Body="&lt;p&gt;Hi All,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am hoping that I can ask this question the correct way.  I have access to play-by-play data, so it's more of an issue with best approach and constructing the data properly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I am looking to do is to calculate the probability of winning an NHL game given the score and time remaining in regulation.  I figure I could use a logistic regression, but I am not sure what the dataset should look like.  Would I have multiple observations per game and for every slice of time I am interested in?  Would I have one observation per game and fit seperate models per slice of time? Is logisitic regression even the right way to go?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help you can provide will be very much appreciated!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Best regards.&lt;/p&gt;&#xA;" OwnerUserId="569" LastEditorUserId="264" LastEditorDisplayName="" LastEditDate="2011-02-02T11:01:33.630" LastActivityDate="2011-02-21T19:22:09.767" Title="Logistic Regression and Dataset Structure" Tags="&lt;time-series&gt;&lt;probability&gt;&lt;logistic-regression&gt;" AnswerCount="4" CommentCount="3" FavoriteCount="3" />
  <row Id="6654" PostTypeId="2" ParentId="6652" CreationDate="2011-01-28T03:47:24.143" Score="8" ViewCount="" Body="&lt;p&gt;There are many issues concerning confidence intervals, but let's focus on the quotations.  The problem lies in possible misinterpretations rather than being a matter of correctness.  When people say a &quot;parameter has a particular probability of&quot; something, they are thinking of &lt;em&gt;the parameter&lt;/em&gt; as being a random variable.  This is not the point of view of a (classical) confidence interval procedure, for which the random variable is the interval itself and the parameter is determined, not random, yet unknown.  This is why such statements are frequently attacked.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Mathematically, if we let $t$ be any procedure that maps data $\mathbf{x} = (x_i)$ to subsets of the parameter space and if (no matter what the value of the parameter $\theta$ may be) the assertion $\theta \in t(\mathbf{x})$ defines an event $A(\mathbf{x})$, then--by definition--it has a probability $\Pr_{\theta}\left( A(\mathbf{x}) \right)$ for any possible value of $\theta$.  When $t$ is a confidence interval procedure with confidence $1-\alpha$ then this probability is supposed to have an infimum (over all parameter values) of $1-\alpha$.  (Subject to this criterion, we usually select procedures that optimize some additional property, such as producing short confidence intervals or symmetric ones, but that's a separate matter.)  The Weak Law of Large Numbers then justifies the second quotation.  That, however, is not a definition of confidence intervals: it is merely a property they have.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think this analysis has answered question 1, shows that the premise of question 2 is incorrect, and makes question 3 moot.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-01-28T03:47:24.143" CommentCount="9" />
  <row Id="6655" PostTypeId="1" AcceptedAnswerId="6661" CreationDate="2011-01-28T06:12:21.280" Score="6" ViewCount="148" Body="&lt;p&gt;There are quite a few methods for parameter estimation out there. MLE, UMVUE, MoM, decision-theoretic, and others all seem like they have a fairly logical case for why they are useful for parameter estimation. Is any one method better than the others, or is it just a matter of how we define what the &quot;best fitting&quot; estimator is (similar to how minimizing orthogonal errors produces different estimates from an ordinary least squares approach)?&lt;/p&gt;&#xA;" OwnerUserId="1118" LastActivityDate="2011-01-30T03:41:02.227" Title="How do I know which method of parameter estimation to choose?" Tags="&lt;estimation&gt;&lt;maximum-likelihood&gt;&lt;mathematical-statistics&gt;&lt;unbiased-estimator&gt;&lt;method-of-moments&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="2" />
  <row Id="6656" PostTypeId="1" CreationDate="2011-01-28T06:51:57.627" Score="2" ViewCount="85" Body="&lt;p&gt;Hello,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am studying a dynamical system that takes as an initial condition a list. I want to analyze the evolution of Shannon's entropy in this system. I know the maximum entropy (50) and the minimum (0). Pure random conditions have almost maximum entropy, and so it is hard to analyze changes in it unless it decreases. I set up the list to have an initial value of 25 (average between maximum and minimum), so there is an equal amount to expand in either direction. Is this statistically sound?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="2976" LastActivityDate="2011-05-30T13:29:35.353" Title="Median entropy to observe evolution of system?" Tags="&lt;entropy&gt;" AnswerCount="2" CommentCount="6" />
  <row Id="6657" PostTypeId="2" ParentId="6371" CreationDate="2011-01-28T06:54:14.890" Score="1" ViewCount="" Body="&lt;p&gt;So, as said in the comments, the Markov chain you consider has some absorbing states (and is irreducible, presumably), hence its stationary distribution is concentrated on these absorbing states. Therefore the issue is to compute some confidence intervals for the only two non zero coordinates of the stationary vector, one for each of the absorbing states. I would call these entries &lt;em&gt;absorption probabilities&lt;/em&gt; rather than stationary vector because there is not much really stationary here, but anyway...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since there are two absorbing states, $a$ and $b$ say, you are interested in $u_c=P_c[$The chain is absorbed at $a]$, for a given initial state $c\ne a$, $b$. I gather you observe the number $N_t(x)$ of particles at state $x$ and time $t$, for every state $x$ (or every $x\ne a$, $b$?), at two different times $t_1$ and $t_2$. How to estimate $u_c$ from these counts? Surely I am missing the obvious but, even replacing (a multiple of) each $N_t(x)$ by the exact value of $P_c[$The chain is at $x$ at time $t]$, I do not see how to compute $u_c$ from these quantities. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Dimensional analysis shows that for $n$ states, $N_t(x)$ at every state $x$ and two different times yields $2(n−1)$ independent parameters and the transition matrix has $n(n−1)$ independent parameters, a fact which seems to indicate that it would be impossible to identify the latter from the former as soon as $n\ge3$. OK, this argument is too sloppy to be really conclusive but... &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Caveat: I did not read, and have no access to, the paper by Karson and Wrobleski.)&lt;/p&gt;&#xA;" OwnerUserId="2592" LastEditorUserId="2592" LastEditorDisplayName="" LastEditDate="2011-01-29T21:39:40.393" LastActivityDate="2011-01-29T21:39:40.393" CommentCount="4" />
  <row Id="6658" PostTypeId="1" CreationDate="2011-01-28T06:55:48.787" Score="3" ViewCount="465" Body="&lt;p&gt;I want to compare sixteen Case Fatality Rates (deaths per 100 cases) of a particular disease from sixteen different populations across 7 years. Each population received the same treatment but some regions did not implement it properly. As a result, I am trying to show the effectiveness the treatment had in each of the regions to prove a hypothesis that &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;More deaths in some regions were&#xA;  because of not giving treatment&#xA;  thoroughly.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Can someone suggest me a way to do this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update:&lt;/strong&gt; &#xA;Below is the data from 2003 to 2010 if that helps:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;SI.    2003              2004              2005               2006            2007              2008           2009            2010    &#xA;-----------------------------------------------------------------------------------------------------------------------------------------------&#xA;       Cases   Deaths    Cases   Deaths    Cases    Deaths    Cases   Deaths  Cases   Deaths    Cases  Deaths  Cases   Deaths  Cases   Deaths&#xA;1      31      4         7       3         34       0         11      0       22      0         6      0       14      0       132     5&#xA;2      109     49        235     64        145      52        392     119     424     133       319    99      462     92      562     125&#xA;3      6       2         85      28        192      64        21      3       336     164       203    45      325     95      50      7&#xA;4      12      5         17      0         6        0         1       0       0       0         0      0       0       0       x       x&#xA;5      0       0         0       0         4        0         0       0       27      0         39     0       66      3       80      0&#xA;6      104     67        37      27        46       39        2       1       32      18        13     3       12      10      2       1&#xA;7      226     10        181     6         122      10        73      3       32      1         3      0       246     8       138     1&#xA;S      17      2         9       1         1        0         3       3       2       0         2      0       3       0       19      5&#xA;9      475     115       22      0         51       0         1       0       0       0         24     0       1       0       34      17&#xA;10     1       0         0       0         1        0         0       0       65      0         4      0       6       0       118     15&#xA;11     0       0         0       0         0        0         0       0       7       0         0      0       9       2       11      6&#xA;12     0       0         0       0         1        0         0       0       0       0         0      0       0       0       x       x&#xA;13     0       0         0       0         0        0         0       0       0       0         12     0       0       0       7       0&#xA;14     163     36        88      9         51       11        18      1       37      0         144    0       265     8       458     7&#xA;15     1124    237       1030    228       6061     1500      2320    528     3024    645       3012   537     3073    556     3540    494&#xA;16     2       1         3       1         12       6         0       0       16      2         58     0       0       0       1       0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2956" LastEditorUserId="2956" LastEditorDisplayName="" LastEditDate="2011-01-30T06:01:05.940" LastActivityDate="2011-01-31T18:48:45.210" Title="How can we compare multiple proportions from multiple independent populations to evaluate implementation of a treatment?" Tags="&lt;hypothesis-testing&gt;&lt;spss&gt;&lt;proportion&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="1" />
  <row Id="6660" PostTypeId="1" CreationDate="2011-01-28T08:24:29.473" Score="5" ViewCount="84" Body="&lt;p&gt;as question, since we can do the conversion from odds ratio &lt;code&gt;(p1/q1)/(p2/q2)&lt;/code&gt; to relative risk &lt;code&gt;(p1/(p1+q1))/(p2/(p2+q2))&lt;/code&gt; fairly easily, I wonder if there is anything that I need to pay attention before doing this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is obvious that if I am doing a case-control study, I shouldn't do a conversion, because I never know the relative risk from this kind of study, but anything other things that I need to consider?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="588" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-01-28T10:04:02.257" LastActivityDate="2011-01-28T10:04:02.257" Title="Prerequisite for conversion from odds ratio to relative risk to be valid" Tags="&lt;relative-risk&gt;&lt;odds&gt;" CommentCount="1" FavoriteCount="2" />
  <row Id="6661" PostTypeId="2" ParentId="6655" CreationDate="2011-01-28T09:10:11.987" Score="5" ViewCount="" Body="&lt;p&gt;There's a slight confusion of two things here: &lt;em&gt;methods for deriving&lt;/em&gt; estimators, and &lt;em&gt;criteria for evaluating&lt;/em&gt; estimators. Maximum likelihood (ML) and method-of-moments (MoM) are ways of deriving estimators; Uniformly minimum variance unbiasedness (UMVU) and decision theory are criteria for evaluating different estimators once you have them, but they won't tell you how to derive them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of the methods for deriving estimators, ML usually produces estimators that are more efficient (i.e. lower variance) than MoM &lt;em&gt;if&lt;/em&gt; you know the model under which your data were derived (the 'data-generating process' (DGP) in the jargon). But MoM makes fewer assumptions about the model; as its name implies, it only uses one or more moments, usually just the mean or just the mean and variance, so it's &lt;em&gt;sometimes&lt;/em&gt; more robust if you're not sure about the DGP. There can be more than one MoM estimator for the same problem, while if you know the DGP, there is only one ML estimator.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of the methods for evaluating estimators, decision-theoretic depends on having a loss function to use to judge your estimator, although the results can be fairly robust to a range of 'reasonable' loss functions.  UMVU estimators often don't even exist; in many cases there &lt;em&gt;is&lt;/em&gt; no unbiased estimator that always has minimum variance. And the criterion of unbiasedness is also of questionable usefulness, as it's not invariant to transformations. For example, would you prefer an unbiased estimator of the odds ratio, or of the log odds ratio? The two will be different.&lt;/p&gt;&#xA;" OwnerUserId="449" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-01-28T09:26:27.437" LastActivityDate="2011-01-28T09:26:27.437" CommentCount="2" />
  <row Id="6662" PostTypeId="2" ParentId="6601" CreationDate="2011-01-28T11:07:15.030" Score="2" ViewCount="" Body="&lt;p&gt;Confidence interval in non-Bayesian tradition is a difficult one. &lt;/p&gt;&#xA;" OwnerUserId="1966" LastActivityDate="2011-01-28T11:07:15.030" CommunityOwnedDate="2011-01-28T11:07:15.030" />
  <row Id="6664" PostTypeId="1" CreationDate="2011-01-28T13:07:55.577" Score="6" ViewCount="373" Body="&lt;p&gt;I have serial hematological measurements data and I have plotted their means and SE in Stata. On the y-axis I have for example hemoglobin and time (visit days) on the x-axis hence I can visualize hemoglobin levels with time (whether it is decreasing or in increasing). The level decreases up to sometime and increases again. What test can use to test whether this is significant or not? Stata and R related answers are welcomed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Julie&lt;/p&gt;&#xA;" OwnerUserId="2961" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-01-28T13:09:32.870" LastActivityDate="2011-01-28T13:49:56.673" Title="Statistical test for trend (continuous variable) in Stata or R" Tags="&lt;r&gt;&lt;stata&gt;" AnswerCount="1" />
  <row Id="6665" PostTypeId="2" ParentId="6599" CreationDate="2011-01-28T13:21:02.197" Score="1" ViewCount="" Body="&lt;p&gt;This is a topic of practical interest to me as well so I did a little research. Here are two papers by an author that is often listed as a reference in these matters.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.research.ibm.com/people/z/zadrozny/kdd2002-Transf.pdf&quot; rel=&quot;nofollow&quot;&gt;Transforming classifier scores into&#xA;accurate multiclass probability&#xA;estimates&lt;/a&gt; &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.research.ibm.com/people/z/zadrozny/bzadnips.pdf&quot; rel=&quot;nofollow&quot;&gt;Reducing multiclass&#xA;to binary by coupling probability&#xA;estimates&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The gist of the technique advocated here is to reduce the multiclass problem to a binary one (e.g. one versus the rest, AKA one versus all), use a technique like Platt (preferrably using a test set) to claibrate the binary scores/probabilities and then combine these using a techique as discussed in the papers (one is an extenstion of a Hastie et al process of &quot;coupling&quot;). In the first link, the best results were found by simply normalizing the binary probabilities to that they sum to 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would love to hear other advice and if any of these tecnhiqes have been implmented in R. &lt;/p&gt;&#xA;" OwnerUserId="2040" LastActivityDate="2011-01-28T13:21:02.197" />
  <row Id="6666" PostTypeId="2" ParentId="6636" CreationDate="2011-01-28T13:24:37.943" Score="4" ViewCount="" Body="&lt;p&gt;Okay, so here is my answer that I promised.  I initially thought it would be quickish, but my answer has become quite large, so at the begining, I state my general results first, and leave the gory details down the bottom for those who want to see it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I must thank @terry felkrow for this fascinating question - if I could give you +10 I would!  This basically is a prime example of the slickness and elegance of Bayesian and Maximum Entropy methods.  I have had much fun working it out!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;SUMMARY&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Exact result&#xA;$$Pr(\theta \in (0,S)|F_{obs},T_U,T_D)=1-\frac{T_U}{T_U+T_D}\Bigg(\frac{T_U}{T_U+S}\Bigg)^{F_{obs}+1}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $\theta$ is the time of the first down time (in seconds) observed by the user, $T_U$ is the number of &quot;up time&quot; seconds observed , $T_D$ is the number of &quot;down time&quot; seconds observed, and $F_{obs}$ is the number of &quot;down periods&quot; (F for &quot;failures&quot;; $\frac{T_D}{F_{obs}}$ is the average number of seconds spent in &quot;down time&quot;) observed&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For your case, $F_{obs}$ is not given, but I would guess that you could find out what it was (which is why I gave the answer for known $F_{obs}$).  Now because you know $T_D$, this tells you a bit about $F_{obs}$, and you should be able to pose an &quot;Expected Value&quot; or educated guess of $F_{obs}$, call it $\hat{F}$.  Now using the geometric distribution with probability parameter $p=\frac{1}{\hat{F}}$ (this is the Maximum Entropy distribution for fixed mean equal to $\hat{F}$), to integrate out $F_{obs}$ gives the probability of (see details for the maths):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Pr(\theta \in (0,S)|\hat{F},T_U,T_D)=1-\frac{\Bigg(\frac{T_U}{T_U+T_D}\Bigg)\Bigg(\frac{T_U}{T_U+S}\Bigg)^2}{\hat{F}-(\hat{F}-1)\Bigg(\frac{T_U}{T_U+S}\Bigg)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So for your specific case, the table below shows various bounds for different $F$, assuming it is known (column 2) or &quot;expected&quot; (column 3).  Can see that the knowing $F_{obs}$ comparing to knowing a &quot;rough&quot; guess $\hat{F}$ only matters when it is very large, (i.e. when the observed average down time is 1 second or less).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ &#xD;&#xA;\begin{array}{c|c}&#xD;&#xA;  F &amp;amp; Pr(\theta \geq \text{S}|F_{obs},T_U,T_D) &amp;amp; Pr(\theta \in (0,S)|\hat{F},T_U,T_D) \\&#xD;&#xA;  \hline&#xD;&#xA;  1,000,000 &amp;amp; 0.625 &amp;amp; 0.499 \\&#xD;&#xA;  \hline&#xD;&#xA;  500,000 &amp;amp; 0.393 &amp;amp; 0.336 \\&#xD;&#xA;  \hline&#xD;&#xA;  250,000 &amp;amp; 0.227 &amp;amp; 0.207 \\&#xD;&#xA;  \hline&#xD;&#xA;  125,000 &amp;amp; 0.128 &amp;amp; 0.122 \\&#xD;&#xA;  \hline&#xD;&#xA;  62,500 &amp;amp; 0.074 &amp;amp; 0.072 \\&#xD;&#xA;  \hline&#xD;&#xA;  31,250 &amp;amp; 0.045 &amp;amp; 0.045 \\&#xD;&#xA;  \hline&#xD;&#xA;  15,685 &amp;amp; 0.031 &amp;amp; 0.030 \\&#xD;&#xA;  \hline&#xD;&#xA;  7,812 &amp;amp; 0.023 &amp;amp; 0.023 \\&#xD;&#xA;  \hline&#xD;&#xA;  1 &amp;amp; 0.016 &amp;amp; 0.016&#xD;&#xA; \end{array}&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;DETAILS&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is based on example 3 in the paper below&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Jaynes, E. T., 1976. `Confidence Intervals vs Bayesian Intervals,' in Foundations of Probability Theory, Statistical Inference, and Statistical Theories of Science, W. L. Harper and C. A. Hooker (eds.), D. Reidel, Dordrecht, p. 175; &lt;a href=&quot;http://bayes.wustl.edu/etj/articles/confidence.pdf&quot; rel=&quot;nofollow&quot;&gt;pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It supposes that the probability that a machine will operate without failure for a time $t$, is given by&#xA;$$Pr(\theta \geq t)=e^{-\lambda t};\ \  0&amp;lt;t,\lambda &amp;lt; \infty$$&#xA;Where $\lambda$ is an unknown &quot;rate of failure&quot;, to be estimated from some data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I will use this to model the failure times in 2 separate cases.  Where &quot;failure&quot; indicates going from &quot;working&quot; to &quot;down time&quot;, and the other way around.  You can think of this like modeling two &quot;memoryless&quot; proceedures.  We first &quot;wait&quot; for the down time, from time $t=t_{0u}=0$, to time $t=t_{1d}$ (so that there was $t_1$ seconds of uninterupted &quot;operating&quot; time).  This has a failure rate of $\lambda_d$ At time $t=t_{1d}$ a new process takes over and now we &quot;wait&quot; for the down time to &quot;fail&quot; at time $t=t{1u}$.  It is also supposed that the rate of failure is constant over time, and that the process has &lt;em&gt;independent increments&lt;/em&gt; (i.e. if you know where the process is at time $t=s$, then all other information about the process prior to time $t&amp;lt;s$ is irrelevant).  This is what is known as a &lt;em&gt;first order Markov process&lt;/em&gt;, also known as a &quot;memoryless&quot; process (for obvious reasons).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Okay, the problem goes as follows, Jaynes eq (8) gives the density that $r$ units out of $n$ will fail at the times $t_1 ,t_2 ,\dots,t_r$, and the remaining (n-r) do not fail at time t as&#xA;$$p(t_1 ,t_2 ,\dots,t_r | \lambda,n)=[\lambda^r exp(-\lambda \sum_{i}t_i)][exp(-(n-r)\lambda t)]$$&#xA;Then assigning a uniform prior (the particular prior you use won't matter in your case because you have so much data, the likelihood will dominate any reasonably &quot;flat&quot; prior) to $\lambda$, this give the a posterior predictive distribution of (see Jaynes paper for details, eq (9)-(13)):&#xA;$$Pr(\theta\geq\theta_0|n,t_1 ,\dots,t_r)=\int_0^{\infty}Pr(\theta\geq\theta_0|\lambda)p( \lambda | t_1 ,t_2 ,\dots,t_r,n)d\lambda=\Bigg(\frac{T}{T+\theta_0}\Bigg)^{r+1}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $T=\sum_{i}t_i + (n-r)t$ is the total time the devices operated without failure.  This indicates that you only needed to know the total &quot;failure free time&quot;, which you have both given as $T_D=500,000$ and $T_U=31,556,926-500,000=31,056,926$.  Also for you problem we always observed either $n$ or $n-1$ &quot;failures&quot; by time $t$, depending on whether the system was &quot;down&quot; or &quot;up&quot; at time $t$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now if you knew what $F_{obs}$ was, then you just plug in $r=F_{obs}$ to the above equation.  The probability that a user will not be in the &quot;down time&quot; in the first $S$ seconds given that the system was &quot;up&quot; when they started is then&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Pr(\theta\geq S|[\text{Up at start} ],F_{obs},T_U)=\Bigg(\frac{T_U}{T_U+30}\Bigg)^{F_{obs}+1}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But the story is not yet finished, because we can marginalise (remove conditions) further.  To make the equations shorter, let $A$ stand for &lt;em&gt;the system was up when the user started&lt;/em&gt;, and let $B$ stand for &lt;em&gt;no down time in&lt;/em&gt; $S$ &lt;em&gt;seconds&lt;/em&gt;.  Then, by the law of total probability, we have&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Pr(B|F_{obs},T_U,T_D)=Pr(B|F_{obs},T_U,T_D,A)Pr(A|F_{obs},T_U,T_D)$$&#xA;$$+Pr(B|F_{obs},T_U,T_D,\overline{A})Pr(\overline{A}|T_U,T_D)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now $\overline{A}$ means that the system was down when the user started, so that it is impossible for $B$ to be true (i.e. no down time) when $\overline{A}$ is true.  Thus, $Pr(B|F_{obs},T_U,T_D,\overline{A})=0$, and we just have to multiply by $Pr(A|F_{obs},T_U,T_D)$.  This is given by $\frac{T_U}{T_U+T_D}$, because none of information contained in $F_{obs},T_U,T_D$ give any reason to favor any particular time over any other time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Pr(\theta\geq S|F_{obs},T_U,T_D)=\frac{T_U}{T_U+T_D}\Bigg(\frac{T_U}{T_U+S}\Bigg)^{F_{obs}+1}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Taking 1 minus this gives the desired result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;NOTE: We may have additional knowledge which would favor certain times, such as knowing what time of day is more likely to have a system outage, or we may believe that system outage is related to the number of users; this analysis ignores such information, and so could be improved upon by taking it into account.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;NOTE: if you only knew the a rough guess of $F_{obs}$, say $\hat{F}$, you could (in theory) use the geometric distribution (has largest entropy for fixed mean) for $F_{obs}$ with probability parameter $p=\frac{1}{\hat{F}}$ and marginalise over $F_{obs}$ to give:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Pr(\theta \geq S|T_U,T_D)=\frac{T_U}{T_U+T_D}\sum_{i=1}^{i=\infty} p(1-p)^{i-1}\Bigg(\frac{T_U}{T_U+S}\Bigg)^{i+1}$$&#xA;$$=\frac{T_U}{T_U+T_D}\Bigg(\frac{T_U}{T_U+S}\Bigg)\sum_{i=1}^{i=\infty} p(1-p)^{i-1}\Bigg(\frac{T_U}{T_U+S}\Bigg)^{i}$$&#xA;$$=\frac{T_U}{T_U+T_D}\Bigg(\frac{T_U}{T_U+S}\Bigg)\sum_{i=1}^{i=\infty} p(1-p)^{i-1} exp\Bigg(i log\Bigg[\frac{T_U}{T_U+S}\Bigg]\Bigg)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now the summation is just the moment generating function, $m_{X}(t)=E[exp(tX)]$, evaluated at $t=log\Bigg[\frac{T_U}{T_U+S}\Bigg]$.  The &lt;a href=&quot;http://en.wikipedia.org/wiki/Geometric_distribution&quot; rel=&quot;nofollow&quot;&gt;mgf for the geometric distribution&lt;/a&gt; is given by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$m_{X}(t)=E[exp(tX)]=\frac{pe^t}{1-(1-p)e^t}$$&#xA;$$\rightarrow m_{X}(log\Bigg[\frac{T_U}{T_U+S}\Bigg])=\frac{p\Bigg[\frac{T_U}{T_U+S}\Bigg]}{1-(1-p)\Bigg[\frac{T_U}{T_U+S}\Bigg]}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And this gives a marginal probability of (noting $p=\frac{1}{\hat{F}}$):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Pr(\theta \geq S|T_U,T_D)=\frac{T_U}{T_U+T_D}\Bigg(\frac{T_U}{T_U+S}\Bigg)\frac{\frac{1}{\hat{F}}\Bigg[\frac{T_U}{T_U+S}\Bigg]}{1-(1-\frac{1}{\hat{F}})\Bigg[\frac{T_U}{T_U+S}\Bigg]}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rearranging terms gives the final result:&#xA;$$Pr(\theta \in (0,S)|T_U,T_D)=1-Pr(\theta \geq S|T_U,T_D)=1-\frac{\Bigg(\frac{T_U}{T_U+T_D}\Bigg)\Bigg(\frac{T_U}{T_U+S}\Bigg)^2}{\hat{F}-(\hat{F}-1)\Bigg(\frac{T_U}{T_U+S}\Bigg)}$$&lt;/p&gt;&#xA;" OwnerUserId="2392" LastEditorUserId="2392" LastEditorDisplayName="" LastEditDate="2011-01-28T16:22:23.360" LastActivityDate="2011-01-28T16:22:23.360" CommentCount="8" />
  <row Id="6667" PostTypeId="2" ParentId="6664" CreationDate="2011-01-28T13:49:56.673" Score="2" ViewCount="" Body="&lt;p&gt;It seems that your problem can be stated as change-point problem. R packages dealing with such type of problems are &lt;a href=&quot;http://cran.r-project.org/web/packages/segmented/index.html&quot; rel=&quot;nofollow&quot;&gt;segmented&lt;/a&gt; and &lt;a href=&quot;http://cran.r-project.org/web/packages/strucchange/index.html&quot; rel=&quot;nofollow&quot;&gt;strucchange&lt;/a&gt;. Since you want to look into changes in time trend (and time trends always need special treatment in linear regression), I suggest differencing your hemoglobin level data and then testing whether there is a change in mean. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Look also into answers for this &lt;a href=&quot;http://stats.stackexchange.com/questions/5700/finding-the-change-point-in-data-from-a-piecewise-linear-function&quot;&gt;question&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-01-28T13:49:56.673" CommentCount="1" />
  <row Id="6668" PostTypeId="1" CreationDate="2011-01-28T15:13:59.893" Score="4" ViewCount="71" Body="&lt;p&gt;I calculated the quantiles for an Epanechnikov kernel which I'm using to estimate the density of a sample. What I need is to find the sample quantiles knowing that it is composed of many Epanechnikov kernels. Is there a way to calculate at wich data points the different quantiles are using the kernel inverse CDF formula?&lt;/p&gt;&#xA;" OwnerUserId="2953" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-01-28T16:00:29.460" LastActivityDate="2011-01-28T16:00:29.460" Title="How can I convert kernel quantiles into sample quantiles?" Tags="&lt;kernel&gt;&lt;quantiles&gt;&lt;cdf&gt;" CommentCount="2" />
  <row Id="6669" PostTypeId="2" ParentId="6652" CreationDate="2011-01-28T15:32:33.077" Score="0" ViewCount="" Body="&lt;p&gt;From a &lt;em&gt;theoretical perspective&lt;/em&gt; Questions 2 and 3 are based on the incorrect assumption that the definitions are wrong.  So I am in agreement with @whuber's answer in that respect, and @whuber's answer to question 1 does not require any additional input from me.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, from a more &lt;em&gt;practical perspective&lt;/em&gt; a confidence interval can be given its intuitive definition (Probability of containing the true value) when it is numerically identical with a Bayesian credible interval based on the same information (i.e. a non-informative prior).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But this is somewhat disheartening for the die hard anti-bayesian, because in order to verify the conditions to give his CI the interpretation he/she want to give it, they must work out the Bayesian solution, for which the intuitive interpretation automatically holds!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The easiest example is a $1-\alpha$ confidence interval for the normal mean with a known variance $\overline{x}\pm \sigma Z_{\alpha/2} $, and a $1-\alpha$ posterior credible interval $\overline{x}\pm \sigma Z_{\alpha/2} $.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not exactly sure of the conditions, but I know the following are important for the intuitive interpretation of CIs to hold:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) a Pivot statistic exists, whose distribution is independent of the parameters (do exact pivots exist outside normal and chi-square distributions?)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) there are no nuisance parameters, (except in the case of a Pivotal statistic, which is one of the few &lt;em&gt;exact&lt;/em&gt; ways one has to handle nuisance parameters when making CIs)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) a sufficient statistic exists for the parameter of interest, and the confidence interval uses the sufficient statistic&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4) the sampling distribution of the sufficient statistic and the posterior distribution have some kind of symmetry between the sufficient statistic and the parameter.  In the normal case the sampling distribution the symmetry is in $(\overline{x}|\mu,\sigma)\sim N(\mu,\frac{\sigma}{\sqrt{n}})$ while $(\mu|\overline{x},\sigma)\sim N(\overline{x},\frac{\sigma}{\sqrt{n}})$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These conditions are usually difficult to find, and usually it is quicker to work out the Bayesian interval, and compare it.  An interesting exercise may also be to try and answer the question &quot;for what prior is my CI also a Credible Interval?&quot;  You may discover some hidden assumptions about your CI procedure by looking at this prior.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-01-28T15:32:33.077" />
  <row Id="6670" PostTypeId="1" CreationDate="2011-01-28T15:37:17.283" Score="5" ViewCount="163" Body="&lt;p&gt;I want to calculate a better bandwidh for my kernel density estimator, which is an Epanechnikov. I use Silverman's formula which involves the standard deviation of the sample, the sample size and a constant, but I'm getting a very smooth curve in most cases and I would prefer if it were more balanced. Thank you for any help you can give me.&lt;/p&gt;&#xA;" OwnerUserId="2953" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2011-01-28T22:00:21.917" LastActivityDate="2011-01-30T04:40:55.097" Title="Which is the formula from Silverman to calculate the bandwidth in a kernel density estimation?" Tags="&lt;estimation&gt;&lt;kernel&gt;&lt;smoothing&gt;&lt;kde&gt;" AnswerCount="4" CommentCount="1" />
  <row Id="6671" PostTypeId="2" ParentId="6670" CreationDate="2011-01-28T16:00:09.583" Score="5" ViewCount="" Body="&lt;p&gt;To shamelessly quote the Stata manual entry for &lt;a href=&quot;http://www.stata.com/help.cgi?kdensity&quot; rel=&quot;nofollow&quot;&gt;&lt;code&gt;kdensity&lt;/code&gt;&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The optimal width is the width that would minimize the &lt;a href=&quot;http://en.wikipedia.org/wiki/Mean_integrated_squared_error&quot; rel=&quot;nofollow&quot;&gt;mean integrated squared error&lt;/a&gt; if the data were Gaussian and a Gaussian kernel were used, so it is not optimal in any global sense. In fact, for multimodal and highly skewed densities, this width is usually too wide and oversmooths the density (Silverman 1992).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Bernard_Silverman&quot; rel=&quot;nofollow&quot;&gt;Silverman, B. W.&lt;/a&gt; 1992. &lt;em&gt;&lt;a href=&quot;http://books.google.com/books?id=e-xsrjsL7WkC&quot; rel=&quot;nofollow&quot;&gt;Density Estimation for Statistics and Data Analysis&lt;/a&gt;&lt;/em&gt;. London: Chapman &amp;amp; Hall. ISBN 9780412246203&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The formula Stata give for the optimal bandwidth $h$ is:&lt;br&gt;&#xA;$$h = \frac{0.9m}{n^{1/5}} \quad \mbox{with } m = \min\left(\sqrt{\operatorname{Var}(X)},\frac{\operatorname{IQR}(X)}{1.349}\right),$$&#xA;where $n$ is the number of observations on $X$, $\operatorname{Var}(X)$ is its variance and $\operatorname{IQR}(X)$ its interquartile range.&lt;/p&gt;&#xA;" OwnerUserId="449" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-01-28T16:22:45.787" LastActivityDate="2011-01-28T16:22:45.787" CommentCount="1" />
  <row Id="6672" PostTypeId="2" ParentId="6652" CreationDate="2011-01-28T17:14:04.500" Score="0" ViewCount="" Body="&lt;p&gt;Okay, I realize that when you calculate a 95% confidence interval for a parameter using classical frequentist methods, it doesn't mean that there is a 95% probability that the parameter lies within that interval. And yet ... when you approach the problem from a Bayesian perspective, and calculate a 95% credible interval for the parameter, you get (assuming a non-informative prior) &lt;em&gt;exactly the same interval&lt;/em&gt; that you get using the classical approach. So, if I use classical statistics to calculate the 95% confidence interval for (say) the mean of a data set, then it &lt;em&gt;is&lt;/em&gt; true that there's a 95% probability that the parameter lies in that interval.&lt;/p&gt;&#xA;" OwnerUserId="2617" LastActivityDate="2011-01-28T17:14:04.500" CommentCount="4" />
  <row Id="6674" PostTypeId="2" ParentId="726" CreationDate="2011-01-28T18:01:15.537" Score="3" ViewCount="" Body="&lt;p&gt;I just can't help myself, this is a provocative quote from E. T. Jaynes:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Many of us have already explored the road you are following, and we know what you will find at the end of it.  It doesn't matter how many new words you drag into the discussion to avoid having to utter the word 'probability' in a sense different from frequency: likelihood, confidence, significance, propensity, support, credibility, acceptability, indifference, consonance, tenability; and so on, until the resources of the good Dr Roget are exhausted.  All of these are attempts to represent degrees of plausibility by real numbers, and they are covered automatically by Cox's theorems.  It doesn't matter which approach you happen to like philosophically; by the time you have made your methods fully consistent, you will be forced, kicking and screaming, back to the ones given by Laplace.  Until you have achieved mathematical equivalence with Laplace's methods, it will be possible, by looking in specific problems with Galileo's magnification, to exhibit the defects in your methods.&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-01-28T18:01:15.537" CommunityOwnedDate="2011-01-28T18:01:15.537" />
  <row Id="6675" PostTypeId="2" ParentId="5077" CreationDate="2011-01-28T18:42:16.323" Score="1" ViewCount="" Body="&lt;p&gt;I came across your question when I was looking for the original reference for Hit-and-Run.  Thanks for that!  I just put together a proof-of-concept implementation of hit-and-run for PyMC at the end of &lt;a href=&quot;http://healthyalgorithms.wordpress.com/2011/01/28/mcmc-in-python-pymc-step-methods-and-their-pitfalls&quot; rel=&quot;nofollow&quot;&gt;this recent blog&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2498" LastActivityDate="2011-01-28T18:42:16.323" />
  <row Id="6676" PostTypeId="2" ParentId="6642" CreationDate="2011-01-28T18:42:37.477" Score="3" ViewCount="" Body="&lt;p&gt;If $X_i$ and $Y_j$ are dependent for $i=j$, but independent for  $i\neq j$ we have an iid sample from bivariate distribution $Z_i=(X_i,Y_i)$. Then central limit theorem gives us&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align}&#xA;\sqrt{n}\left(\frac{1}{n}\sum_{i=1}^nZ_i -EZ_1\right)\xrightarrow{D}N(0,\Sigma)&#xA;\end{align}&#xA;with $\Sigma=cov(Z_1)$ and $\xrightarrow{D}$ indicating convergence in distribution. Note that&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$(S_n,T_n)=\frac{1}{n}\sum_{i=1}^nZ_i.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now we can use delta method, which states that if $r_n(U_n-\theta)\xrightarrow{D}U$ for numbers $r_n\to\infty$, then $r_n(\phi(U_n)-\phi(\theta))\xrightarrow{D}\phi&amp;#39;_\theta(U)$. This statement can be found &lt;a href=&quot;http://books.google.com/books?id=UEuQEM5RjWgC&amp;amp;lpg=PR1&amp;amp;dq=asymptotic%20statistics%20vaart&amp;amp;hl=fr&amp;amp;pg=PA26#v=onepage&amp;amp;q&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;. Delta method is also described &lt;a href=&quot;http://en.wikipedia.org/wiki/Delta_method&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In our case now we have $r_n=\sqrt{n}$, $\phi(x,y)=x^2-y^2$, $U_n=(S_n,T_n)$ and $\theta=(EX_1,EY_1)$. We have &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\phi_{\theta}&amp;#39;=(2EX_1,-2EY_1)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$U=(U_1,U_2)\sim N(0,\Sigma)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally we get&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align}&#xA;\sqrt{n}\left(S_n^2-T_n^2-(EX_1)^2+(EY_1)^2\right)\xrightarrow{D} &#xA;2U_1EX_1-2U_2EY_1:=V&#xA;\end{align}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since we know that $(U_1,U_2)\sim N(0,\Sigma)$ we get that&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$V\sim N(0,4(EX_1,EY_1)\Sigma(EX_1,EY_1)&amp;#39;).$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that I basically redid the example after the theorem in the link. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The final answer then depends on $\Sigma=cov((X_1,Y_1))$, but this should be known to original poster.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-01-28T18:48:44.800" LastActivityDate="2011-01-28T18:48:44.800" CommentCount="2" />
  <row Id="6677" PostTypeId="2" ParentId="6670" CreationDate="2011-01-28T19:00:00.760" Score="2" ViewCount="" Body="&lt;p&gt;I second @onestop, but quote Wilcox, 'Introduction to Robust Estimation and Hypothesis Testing', 2nd edition, page 50:&#xA;$$&#xD;&#xA;h = 1.06\frac{A}{n^{1/5}}, \qquad A = \min{\left(s,\frac{IQR(x)}{1.34}\right)},&#xD;&#xA;$$&#xA;where $s$ is the sample standard deviation.&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2011-01-28T19:00:00.760" CommentCount="4" />
  <row Id="6678" PostTypeId="2" ParentId="726" CreationDate="2011-01-28T19:20:55.060" Score="0" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Everybody knows that probability and statistics are the same thing, and statistics is nothing but correlation. Now the correlation is just the cosine of an angle, thus all is trivial. &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;-- Emil Artin, according to Kai Lai Chung in&#xA;&lt;a href=&quot;http://books.google.com/books?id=safNnEOICL8C&quot; rel=&quot;nofollow&quot;&gt;Elementary probability theory&lt;/a&gt; (right, Artin might not been known primarily as a statistician)&lt;/p&gt;&#xA;" OwnerUserId="2592" LastEditorUserId="2592" LastEditorDisplayName="" LastEditDate="2011-01-28T19:29:25.247" LastActivityDate="2011-01-28T19:29:25.247" CommunityOwnedDate="2011-01-28T19:20:55.060" />
  <row Id="6679" PostTypeId="2" ParentId="6492" CreationDate="2011-01-28T19:39:49.070" Score="1" ViewCount="" Body="&lt;p&gt;On page 372 of &lt;a href=&quot;http://www.stat.columbia.edu/~gelman/arm/&quot; rel=&quot;nofollow&quot;&gt;ARM&lt;/a&gt;, Gelman and Hill mention using a uniform distribution on the inverse of DF between 1/DF = .5 and 1/DF = 0. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Specifically, in BUGS, they use: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;nu.y &amp;lt;- 1/nu.inv.y &#xA;nu.inv.y ~ dunif(0,.5)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1146" LastActivityDate="2011-01-28T19:39:49.070" />
  <row Id="6680" PostTypeId="1" AcceptedAnswerId="6685" CreationDate="2011-01-28T19:46:47.740" Score="2" ViewCount="130" Body="&lt;p&gt;Hi &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to convert the table A to table B. How can I do that using R?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;TABLE A&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Y 10&#xA;Y 12&#xA;Y 18 &#xA;X 22 &#xA;X 12&#xA;Z 11&#xA;Z 15&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;TABLE B&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X 22 12&#xA;Y 10 12 18&#xA;Z 11 15 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2725" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-01-28T22:23:20.910" LastActivityDate="2011-01-28T23:08:18.223" Title="Need to convert duplicate column elements to a unique element in R" Tags="&lt;r&gt;" AnswerCount="2" CommentCount="3" FavoriteCount="2" />
  <row Id="6681" PostTypeId="2" ParentId="726" CreationDate="2011-01-28T20:23:49.323" Score="4" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;[Statistics are] the only tools by which an opening can be cut through the formidable thicket of difficulties that bars the path of those who pursue the science of man.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;-- Sir Francis &lt;a href=&quot;http://galton.org/&quot; rel=&quot;nofollow&quot;&gt;Galton&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="2592" LastActivityDate="2011-01-28T20:23:49.323" CommunityOwnedDate="2011-01-28T20:23:49.323" />
  <row Id="6682" PostTypeId="2" ParentId="6680" CreationDate="2011-01-28T20:47:51.733" Score="3" ViewCount="" Body="&lt;p&gt;Here is the possible solution with taking a clarification in comment in mind, but still I think that the question is invalid as it stands.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;##This gives the table A with both columns in character, which we will use&#xA;df&amp;lt;-rbind(c(&quot;Y&quot;,10),c(&quot;Y&quot;,12),c(&quot;Y&quot;,18),c(&quot;X&quot;,22), c(&quot;X&quot;,12), c(&quot;Z&quot;,11), c(&quot;Z&quot;,15))&#xA;&amp;gt; df&#xA;     [,1] [,2]&#xA;[1,] &quot;Y&quot;  &quot;10&quot;&#xA;[2,] &quot;Y&quot;  &quot;12&quot;&#xA;[3,] &quot;Y&quot;  &quot;18&quot;&#xA;[4,] &quot;X&quot;  &quot;22&quot;&#xA;[5,] &quot;X&quot;  &quot;12&quot;&#xA;[6,] &quot;Z&quot;  &quot;11&quot;&#xA;[7,] &quot;Z&quot;  &quot;15&quot;&#xA;&#xA;##break the table into the list&#xA;l &amp;lt;-tapply(df[,2],df[,1],function(l)l)&#xA;&#xA;##calculate the maximum length of the list element&#xA;n &amp;lt;- max(sapply(l,length))&#xA;&#xA;##pad the elements of the list with empty strings&#xA;tbB&amp;lt;-t(sapply(l,function(x){res &amp;lt;- rep(&quot;&quot;,n);res[1:length(x)]&amp;lt;-x;res}))&#xA;&amp;gt; tbB&#xA;  [,1] [,2] [,3]&#xA;X &quot;22&quot; &quot;12&quot; &quot;&quot;  &#xA;Y &quot;10&quot; &quot;12&quot; &quot;18&quot;&#xA;Z &quot;11&quot; &quot;15&quot; &quot;&quot;  &#xA;&#xA;##Comma separated file &#xA;write.table(tbB,file=&quot;tableB.csv&quot;,quote=FALSE,col.names=FALSE,sep=&quot;,&quot;)&#xA;&#xA;&#xA;##Tab delimited &#xA;write.table(tbB,file=&quot;tableB.txt&quot;,quote=FALSE,col.names=FALSE,sep=&quot;\t&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If we read the table A from the file, then &lt;code&gt;df&lt;/code&gt; is a data.frame with first column a &lt;code&gt;factor&lt;/code&gt; and second column &lt;code&gt;numeric&lt;/code&gt; (this is a default R behaviour). To apply the code above it is necessary to transform the columns to &lt;code&gt;character&lt;/code&gt;: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df &amp;lt;- sapply(df,as.character) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-01-28T20:47:51.733" CommentCount="5" />
  <row Id="6683" PostTypeId="2" ParentId="6670" CreationDate="2011-01-28T21:58:45.647" Score="2" ViewCount="" Body="&lt;p&gt;I asked a &lt;a href=&quot;http://stats.stackexchange.com/questions/168/choosing-a-bandwidth-for-kernel-density-estimators&quot;&gt;similar question&lt;/a&gt; a few months ago. Rob Hyndman provided an excellent &lt;a href=&quot;http://stats.stackexchange.com/questions/168/choosing-a-bandwidth-for-kernel-density-estimators/179#179&quot;&gt;answer&lt;/a&gt; that recommends the Sheather-Jones method.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One addition point. In R, for the &lt;code&gt;density&lt;/code&gt; function, you set the bandwidth explicitly via the &lt;code&gt;bw&lt;/code&gt; argument. However, I often find that the &lt;code&gt;adjust&lt;/code&gt; argument is more helpful. The &lt;code&gt;adjust&lt;/code&gt; argument scales the value of the bandwidth. So &lt;code&gt;adjust=2&lt;/code&gt; means double the bandwidth.&lt;/p&gt;&#xA;" OwnerUserId="8" LastActivityDate="2011-01-28T21:58:45.647" CommentCount="1" />
  <row Id="6684" PostTypeId="1" AcceptedAnswerId="6718" CreationDate="2011-01-28T22:32:16.647" Score="1" ViewCount="259" Body="&lt;p&gt;&lt;strong&gt;Note: I've updated the example case code, there were some errors in the previous version&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cross posted to R-help, because I half suspect this is 'unexpected behaviour'.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to predict values from an existing lm (linear model, e.g.&#xA;lm.obj) result in R using a new set of predictor variables (e.g.&#xA;newdata).  Specifically, I am interested in the predicted y value at the mean, 1 SD above of the mean, and 1 SD below the mean for each predictor. However, it seems that because my linear models was made by calling scale() on the target predictor that predict exits with an error, &quot;Error in scale(xxA, center = 9.7846094491829, scale =&#xA;0.959413568556403) : object 'xxA' not found&quot;.  By debugging predict, I&#xA;can see that the error occurs in a call to model.frame.  By debugging&#xA;model frame I can see the error occurs with this command:  variables&#xA;&amp;lt;- eval(predvars, data, env); it seems likely that the error is&#xA;because predvars looks like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;list(scale(xxA, center = 10.2058714830537, scale = 0.984627257169526),&#xA;   scale(xxB, center = 20.4491690881149, scale = 1.13765718273923))&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An example case:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   dat &amp;lt;- data.frame(xxA = rnorm(20,10), xxB = rnorm(20,20))&#xA;   dat$out &amp;lt;- with(dat,xxA+xxB+xxA*xxB+rnorm(20,20))&#xA;   lm.res.scale &amp;lt;- lm(out ~ scale(xxA)*scale(xxB),data=dat)&#xA;   my.data &amp;lt;- lm.res.scale$model #load the data from the lm object&#xA;   newdata &amp;lt;- expand.grid(X1=c(-1,0,1),X2=c(-1,0,1))&#xA;   names(newdata) &amp;lt;- c(&quot;scale(xxA)&quot;,&quot;scale(xxB)&quot;)&#xA;   newdata$Y &amp;lt;- predict(lm.res.scale,newdata)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is there something I could do before passing newdata or lm.obj to&#xA;predict() that would prevent the error?  I tried:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the help file it looks&#xA;like I might be able to do something with the terms, argument but I&#xA;haven't quite figured out what I would need to do. Alternatively, is&#xA;there a fix for model.frame that would prevent the error?  Should&#xA;predict() behave this way?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Additional Details:&lt;/strong&gt;&#xA;However, I really want a solution that, in one step will provide values like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;coef(lm.res.scale)[1]+&#xA;coef(lm.res.scale)[2]*newdata[,1]+&#xA;coef(lm.res.scale)[3]*newdata[,2]+&#xA;coef(lm.res.scale)[4]*newdata[,1]*newdata[,2]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I think that should be exactly what predict() should do.  That is, I think my example code should be equivalent to:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   dat &amp;lt;- data.frame(xxA = rnorm(20,10), xxB = rnorm(20,20))&#xA;   dat$out &amp;lt;- with(dat,xxA+xxB+xxA*xxB+rnorm(20,20))&#xA;   #rescaling outside of lm&#xA;   X1 &amp;lt;- with(dat,as.vector(scale(xxA)))&#xA;   X2 &amp;lt;- with(dat,as.vector(scale(xxB)))&#xA;   y  &amp;lt;- with(dat,out)&#xA;   lm.res.correct &amp;lt;- lm(y~X1*X2)&#xA;   my.data &amp;lt;- lm.res.correct$model #load the data from the lm object&#xA;   newdata &amp;lt;- expand.grid(X1=c(-1,0,1),X2=c(-1,0,1))&#xA;   #No need to rename newdata as it matches my lm object already&#xA;   newdata$Y &amp;lt;- predict(lm.res.correct,newdata)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Notably, adjusting my formula to include as.vector() does not solve the problem with my attempt to use predict() directly with newdata.&lt;/p&gt;&#xA;" OwnerUserId="196" LastEditorUserId="196" LastEditorDisplayName="" LastEditDate="2011-01-29T02:00:13.550" LastActivityDate="2011-01-30T20:04:27.090" Title="How can one use the predict function on a lm object where the IVs have been dynamically scaled?" Tags="&lt;r&gt;&lt;regression&gt;" AnswerCount="2" CommentCount="3" />
  <row Id="6685" PostTypeId="2" ParentId="6680" CreationDate="2011-01-28T23:08:18.223" Score="2" ViewCount="" Body="&lt;p&gt;Similar to what @mpiktas proposed, but using &lt;code&gt;aggregate&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Construct the data frame&#xA;let &amp;lt;- c(&quot;Y&quot;, &quot;Y&quot;, &quot;Y&quot;, &quot;X&quot;, &quot;X&quot;, &quot;Z&quot;, &quot;Z&quot;)&#xA;num &amp;lt;- c(10, 12, 18, 22, 12, 11, 15)&#xA;&#xA;df &amp;lt;- data.frame(let, num)&#xA;&#xA;# Aggregate data by the first column.&#xA;# Do not apply any transformation to the data (use the identity function)&#xA;ag &amp;lt;- aggregate(df$num, FUN=identity, by=list(let))&#xA;# Find the line with maximum number of elements&#xA;maxlen &amp;lt;- max(sapply(ag$x, length)))&#xA;# Transform the list to a matrix&#xA;res &amp;lt;- t(sapply(ag$x, function(x){c(x, rep(NA, maxlen-length(x)))}))&#xA;row.names(res) &amp;lt;- ag[,1]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Outputs:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1&amp;gt; res&#xA;  [,1] [,2] [,3]&#xA;X   22   12   NA&#xA;Y   10   12   18&#xA;Z   11   15   NA&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="582" LastActivityDate="2011-01-28T23:08:18.223" />
  <row Id="6686" PostTypeId="2" ParentId="6684" CreationDate="2011-01-29T00:28:42.120" Score="1" ViewCount="" Body="&lt;p&gt;See if this console scrape shows what you might have wanted. Changed the names of the new data columns:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; dat &amp;lt;- data.frame(xxA = rnorm(20,10), xxB = rnorm(10,20))&#xA;&amp;gt;    dat$out &amp;lt;- with(dat,xxA+xxB+xxA*xxB+rnorm(20,20))&#xA;&amp;gt;    xVar &amp;lt;- &quot;scale(xxA)&quot;&#xA;&amp;gt;    traceVar &amp;lt;- &quot;scale(xxB)&quot;&#xA;&amp;gt;    DVname &amp;lt;- &quot;out&quot;&#xA;&amp;gt;    lm.obj &amp;lt;- lm.res.scale &amp;lt;- lm(out ~ scale(xxA)*scale(xxB),data=dat)&#xA;&amp;gt;    my.data &amp;lt;- lm.obj$model #load the data from the lm object&#xA;&amp;gt;    X1 &amp;lt;- my.data[,xVar]&#xA;&amp;gt;    X2 &amp;lt;- my.data[,traceVar]&#xA;&amp;gt;    DV &amp;lt;- lm.obj$model[,DVname]&#xA;&amp;gt;    newdata &amp;lt;- expand.grid(X1=c(-1,0,1),X2=c(-1,0,1))&#xA;&amp;gt;    newdata$xxA &amp;lt;- newdata$X1 * sd(my.data[,xVar])&#xA;&amp;gt;    newdata$xxB &amp;lt;- newdata$X2 * sd(my.data[,traceVar])&#xA;&amp;gt;    names(newdata) &amp;lt;- names(dat) #have to rename to original variable names for predict to work&#xA;&amp;gt;    newdata$Y &amp;lt;- predict(lm.obj,newdata)&#xA;&amp;gt; str(newdata)&#xA;'data.frame':   9 obs. of  5 variables:&#xA; $ xxA: num  -1 0 1 -1 0 1 -1 0 1&#xD;&#xA;     $ xxB: num  -1 -1 -1 0 0 0 1 1 1&#xA; $ out: num  -1 0 1 -1 0 1 -1 0 1&#xD;&#xA;     $ NA : num  -1 -1 -1 0 0 0 1 1 1&#xA; $ Y  : num  47.7 44.7 41.8 46.5 44.7 ...&#xD;&#xA;     - attr(*, &amp;quot;out.attrs&amp;quot;)=List of 2&#xD;&#xA;      ..$ dim     : Named int  3 3&#xA;  .. ..- attr(*, &quot;names&quot;)= chr  &quot;X1&quot; &quot;X2&quot;&#xA;  ..$ dimnames:List of 2&#xA;  .. ..$ X1: chr  &quot;X1=-1&quot; &quot;X1= 0&quot; &quot;X1= 1&quot;&#xA;  .. ..$ X2: chr  &quot;X2=-1&quot; &quot;X2= 0&quot; &quot;X2= 1&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt;&#xA;With your requested revisions,  this is what I would suggest:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;newdat &amp;lt;- expand.grid(X1=c(-1,0,1),X2=c(-1,0,1))&#xA;newdat$xxA &amp;lt;- mean(dat$xxA) + newdat$X1*sd(dat$xxA)&#xA;newdat$xxB &amp;lt;- mean(dat$xxB) + newdat$X2*sd(dat$xxB)&#xA; newdata$Y &amp;lt;- predict(lm.res.scale,newdat[, c(&quot;xxA&quot;,&quot;xxB&quot;) ] )&#xA; newdat$Y &amp;lt;- predict(lm.res.scale,newdat[, c(&quot;xxA&quot;,&quot;xxB&quot;) ] )&#xA; newdat&#xA;&#xA;  X1 X2       xxA      xxB        Y&#xA;1 -1 -1  8.931712 18.59229 214.0096&#xA;2  0 -1  9.830117 18.59229 231.4839&#xA;3  1 -1 10.728522 18.59229 248.9583&#xA;4 -1  0  8.931712 19.77820 225.3516&#xA;5  0  0  9.830117 19.77820 243.9916&#xA;6  1  0 10.728522 19.77820 262.6315&#xA;7 -1  1  8.931712 20.96410 236.6936&#xA;8  0  1  9.830117 20.96410 256.4992&#xA;9  1  1 10.728522 20.96410 276.3047&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Notice you don't need to do all those back-flips to match the model terms inside the lm-object. You only need to provide it with properly named data on the scale of the original data. The predict machinery will take care of the scaling.&lt;/p&gt;&#xA;" OwnerUserId="2129" LastEditorUserId="2129" LastEditorDisplayName="" LastEditDate="2011-01-29T12:04:57.473" LastActivityDate="2011-01-29T12:04:57.473" CommentCount="1" />
  <row Id="6687" PostTypeId="1" CreationDate="2011-01-29T03:33:43.937" Score="2" ViewCount="41" Body="&lt;p&gt;I am quitting job and therefore won't have access to the excel addin that generates an array of regression statistics. The function is set up like this: LINEREG(independentarray, dependentarray,statistics), where &quot;statistics is 1 if you want the function to produce a full set of statistics including coefficient estimates, estimate standard errors, r-squared, F-stats etc. Using this, I can then INDEX(LINEREG(X,Y,S),1,2) etc to pull individual components out of the results. This is useful for doing rolling historical analysis and back tests etc. Are there add-ins that I can buy or download for this purpose? &lt;/p&gt;&#xA;" OwnerUserId="2969" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-01-29T03:54:52.597" LastActivityDate="2011-01-29T03:54:52.597" Title="How can I pull pull elements out of an array containing multiple regression results?" Tags="&lt;regression&gt;&lt;excel&gt;" CommentCount="1" />
  <row Id="6688" PostTypeId="1" CreationDate="2011-01-29T04:44:13.400" Score="7" ViewCount="176" Body="&lt;p&gt;Suppose I want to build a binary classifier.  I have several thousand features and only a few 10s of samples.  From domain knowledge, I have a good reason to believe that the class label can be accurately predicted using only a few features, but I have no idea &lt;em&gt;which&lt;/em&gt; ones.  I also want the final decision rule to be easy to interpret/explain, further necessitating a small number of features.  Certain subsets of my features are highly correlated, so selecting the most predictive few independently wouldn't work.  I also want to be able to meaningfully do hypothesis testing on my features.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is the following stepwise regression procedure reasonable under these conditions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Given the features already in the model (or just the intercept on the first iteration), select the feature that produces the largest log likelihood ratio when added to the model.  Use the likelihood ratio chi-square test to calculate a nominal P-value for each hypothesis test performed in this selection.  The null here is that adding the extra variable to the model provides no additional predictive ability.  The alternative is that it does increase predictive abilityl&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Treat the hypotheses tested in Step 1 of each iteration as a family and calculate the false discovery rate for the smallest P-value (for the feature selected) using something like Benjamini-Hochberg.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Goto 1 unless some stopping criteria are met.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Report the false discovery rates for the individual features, but &lt;strong&gt;&lt;em&gt;not&lt;/em&gt;&lt;/strong&gt; the P-value for the model as a whole (since this will be massively inflated).  Each of these multiple testing corrected P-values represents the statistical significance of that feature &lt;em&gt;given&lt;/em&gt; all of the features previously added to the model.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Does doing something like this under these circumstances successfully avoid all of the typical criticisms of stepwise regression?  Are the false discovery rates calculated in this way reasonable?&lt;/p&gt;&#xA;" OwnerUserId="1347" LastEditorUserId="1347" LastEditorDisplayName="" LastEditDate="2011-01-29T14:07:37.390" LastActivityDate="2011-01-29T16:25:14.063" Title="Sane stepwise regression?" Tags="&lt;regression&gt;&lt;logistic-regression&gt;&lt;multiple-comparisons&gt;&lt;stepwise-regression&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="2" />
  <row Id="6689" PostTypeId="2" ParentId="6421" CreationDate="2011-01-29T05:40:49.240" Score="5" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Least-angle_regression&quot; rel=&quot;nofollow&quot;&gt;LARS&lt;/a&gt; gets my vote. It combines linear regression with variable selection. Algorithms to compute it usually give you a collection of $k$ linear models, the $i$th one of which has nonzero coefficients for only $i$ regressors, so you can easily look at models of different complexity.&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2011-01-29T05:40:49.240" CommunityOwnedDate="2011-01-29T05:40:49.240" CommentCount="2" />
  <row Id="6690" PostTypeId="1" AcceptedAnswerId="6691" CreationDate="2011-01-29T06:45:35.563" Score="4" ViewCount="104" Body="&lt;p&gt;If I see &lt;code&gt;Rmath.h&lt;/code&gt; in &lt;code&gt;/usr/share/R/include&lt;/code&gt;, the signature of the function &lt;code&gt;dpois&lt;/code&gt; or &lt;code&gt;Rf_dpois&lt;/code&gt; is  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;double dpois(double, double, int); &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, if I do &lt;code&gt;?dpois&lt;/code&gt; in &lt;code&gt;R&lt;/code&gt;, I see:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dpois(x, lambda, log = FALSE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Are these both the same thing? If yes, can someone please clarify?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also I tried to find the code for &lt;code&gt;dpois&lt;/code&gt; at &lt;a href=&quot;http://svn.r-project.org/R/trunk/src/library/stats/&quot; rel=&quot;nofollow&quot;&gt;R-svn&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone please tell how to find the code for functions like these?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you for any help or pointers.&lt;/p&gt;&#xA;" OwnerUserId="1307" LastActivityDate="2011-01-29T16:31:29.507" Title="What is the difference between Rf_dpois in Rmath.h and the dpois that I use directly in R?" Tags="&lt;r&gt;" AnswerCount="3" CommentCount="1" FavoriteCount="2" />
  <row Id="6691" PostTypeId="2" ParentId="6690" CreationDate="2011-01-29T06:55:11.320" Score="3" ViewCount="" Body="&lt;p&gt;Yes they are the same.  I am guessing that  &lt;code&gt;x&lt;/code&gt; is double because of &lt;a href=&quot;http://en.wikipedia.org/wiki/Type_conversion&quot; rel=&quot;nofollow&quot;&gt;type conversions&lt;/a&gt;. If &lt;code&gt;integer&lt;/code&gt; is multiplied by &lt;code&gt;double&lt;/code&gt; in C, the &lt;code&gt;double&lt;/code&gt; is converted to integer. There are checks in the code that &lt;code&gt;x&lt;/code&gt; is really integer, though it is of type &lt;code&gt;double&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The code for this function is in &lt;code&gt;src/nmath/dpois.c&lt;/code&gt;. I found it by doing &lt;code&gt;grep -R &quot;dpois&quot;&lt;/code&gt; in directory with extracted R source code. This will work from the terminal in Linux and in Mac OS X. In Windows you will need to install grep. I suggest installing &lt;a href=&quot;http://www.murdoch-sutherland.com/Rtools/index.html&quot; rel=&quot;nofollow&quot;&gt;Rtools&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note.&lt;/strong&gt; The code is very short, but I intentionally do not post it here, since I do not know for sure if I will not break some licence by doing that.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-01-29T08:11:23.197" LastActivityDate="2011-01-29T08:11:23.197" CommentCount="3" />
  <row Id="6692" PostTypeId="2" ParentId="6421" CreationDate="2011-01-29T08:53:30.500" Score="8" ViewCount="" Body="&lt;p&gt;The answer is so simple that i have to write all this gibberish to make CV let me post it: &lt;a href=&quot;http://www.r-project.org&quot; rel=&quot;nofollow&quot;&gt;R&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="704" LastEditorUserId="704" LastEditorDisplayName="" LastEditDate="2011-01-29T10:15:28.620" LastActivityDate="2011-01-29T10:15:28.620" CommunityOwnedDate="2011-01-29T08:53:30.500" />
  <row Id="6693" PostTypeId="2" ParentId="6690" CreationDate="2011-01-29T14:05:27.887" Score="3" ViewCount="" Body="&lt;p&gt;In reference to the question on how to find the source for headers of other functions, see this article in R News by Uwe Ligges:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Uwe Ligges. R Help Desk: &lt;a href=&quot;http://cran.r-project.org/doc/Rnews/Rnews_2006-4.pdf&quot; rel=&quot;nofollow&quot;&gt;Accessing the sources&lt;/a&gt;. R News, 6(4):43-45, October 2006.&lt;/p&gt;&#xA;" OwnerUserId="1390" LastActivityDate="2011-01-29T14:05:27.887" CommentCount="1" />
  <row Id="6694" PostTypeId="2" ParentId="6688" CreationDate="2011-01-29T16:25:14.063" Score="1" ViewCount="" Body="&lt;p&gt;For the purposes of my answer, I will denote the binary variable of interest as $Y_i \text{  ;}(i=1,\dots,n)$ and the predictors $X_{ij} \text{  ;} (j=1,\dots,p)$ and assume that $Y$ has values of $Y=0$ and $Y=1$.  It will also be convenient to define $\gamma_m$ to indicate the model $m \text{  ;}(m=1,..,M)$, such that $\gamma_m^TX_{ij}$ is equal to $X_{ij}$ if the &lt;em&gt;jth&lt;/em&gt; variable is in the &lt;em&gt;mth&lt;/em&gt; model, and $0$ otherwise.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would make a modification to your method, and give a rationale.  You are using a classifier model, which means you want to predict the value of a categorical variable into the future - so you should really be defining a &lt;em&gt;prediction rule&lt;/em&gt; (given a new set of predictors $X_{j}$, how will you predict whether $Y=1$ or $Y=0$).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I would suggest evaluating the prediction directly, rather than the likelihood ratio.  However, the observation predicted should not be included in the estimation of the model (because this is exactly the situation you will face when actually using your model).  So have a new step 1) (bold is my suggested change).&#xA;&lt;em&gt;1) Given the features already in the model (or just the intercept on the first iteration), select the feature that produces&lt;/em&gt; &lt;strong&gt;the best predictions&lt;/strong&gt; &lt;em&gt;when added to the model.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now you need to decide &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;what you want &quot;best&quot; to mean mathematically&lt;/li&gt;&#xA;&lt;li&gt;how to split your data into &quot;fitting&quot; and &quot;predicting&quot; parts &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I will make a suggestion for each:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;An intuitive definition for a &quot;good&quot; classifier (and also computationally simple) is the proportion of correct classifications it makes.  However, you may have some additional knowledge of the specific consequences of making a correct or incorrect classification (e.g. predicting correctly when $Y=1$ is twice as important than when $Y=0$).  In this case you should incorporate this knowledge into the definition of &quot;good&quot;.  But for the equations in my answer I will use $F=\frac{C}{C+I}$ as the criterion ($F$=&quot;fraction&quot; or &quot;frequency&quot; $C$=&quot;correct&quot; $I$=&quot;incorrect&quot;)&lt;/li&gt;&#xA;&lt;li&gt;Because you don't have a lot of data, you need as much as possible to fit the model, so a simple drop one jacknife procedure can be used.  You leave observation $1$ out, fit the model with observations $2,\dots,n$, and use this to predict observation $1$.  Then you leave observation $2$ out, fit the model with observations $1,3,\dots,n$, and use this to predict observation $2$; and so on until each observation has been &quot;left out&quot; and predicted.  You will then have $n$ predictions, and you can now calculate $F=\frac{C}{n}$, the fraction of correctly predicted values for the particular model.  Subscript this for the particular model $F_m$.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;You would then calculate $F_m$ for each model $(m=1,\dots,M)$, and pick the model which predicts the best $m=\text{argmax}_{m\in M} F_m$.  Note that the good thing about the above method is that you do not need to worry about how many variables are in your model or how correlated these variables are (unless it makes it impossible to actually fit the model).  This is because the model is fit separately to the prediction, so bias due to over-fitting, or degradation due to numerical instability will show up in the poorer predictions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a step-wise situation it is done sequentially, so at the $sth$ you have $M_s=p+1$ models to choose between: one each for &quot;removing&quot; each $X_{j}$ which is in the model, one for &quot;adding&quot; each $X_{j}$ which is not in the model, and one for keeping the model unchanged (you stop the procedure when you choose this model, and this is your final model).  If there is a tie, you need an additional criteria to split the winners (or you could let you algorithm &quot;branch&quot; off, and see where each &quot;branch&quot; ends up, then take the &quot;branch&quot; which had the best predictions at its final step)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Step-wise can be risky because you may find &quot;local maximums&quot; instead of &quot;global maximums&quot;, especially because you have such a large number of predictors (this is a big &quot;space&quot; to optimise over, and is probably multi-modal - meaning there are many &quot;best&quot; models)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The good thing about this is that the model you choose has a clear, directly relevant interpretation:  The model which predicted the &lt;em&gt;highest&lt;/em&gt; proportion of results correctly, out of the alternatives considered.  And you have a clear measure of exactly how good your binary classifier is (it classified $100F$ percent correctly).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think you will find this a &lt;strong&gt;lot&lt;/strong&gt; easier to justify your choice of final model to a non-statistician, rather than trying to explain why the p-value indicates the model is good.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And for hypothesis testing, you can declare any effect remaining in your final model as &quot;significant&quot; in that the relationships contained in this model we able to re-produce the data ($Y$) the most effectively.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Two final remarks:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;You could also use this machinery to decide if step-wise is better than forward selection (only add variables) or backward selection (start from full model, and only remove variables).&lt;/li&gt;&#xA;&lt;li&gt;You could fit the full model (or any model with $p\geq n$) by &quot;ridging&quot; the model, which amounts to adding a small number to the diagonal elements of the $X^TX$ matrix, or $X^TWX$ for GLMs before inverting when calculating your betas, to give $(X^TX+\lambda I)^{-1}X^TY$ or $(X^TWX+\lambda I)^{-1}X^TWY$.  Basically, $\lambda$ constrains the sum of squares of the betas to be less than a particular value, increasing the value of $\lambda$ decreases this constraint (which is a &quot;smooth&quot; model selection procedure in its own right, if you think about it).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-01-29T16:25:14.063" CommentCount="5" />
  <row Id="6695" PostTypeId="2" ParentId="6690" CreationDate="2011-01-29T16:31:29.507" Score="5" ViewCount="" Body="&lt;p&gt;You could always test things, courtesy of &lt;a href=&quot;http://cran.r-project.org/packages=inline&quot; rel=&quot;nofollow&quot;&gt;inline&lt;/a&gt; and &lt;a href=&quot;http://cran.r-project.org/packages=Rcpp&quot; rel=&quot;nofollow&quot;&gt;Rcpp&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here we pass three lines of code to C++ to receive the two parameters, and then pass those to &lt;code&gt;Rf_dpois&lt;/code&gt; (while setting the &lt;code&gt;log&lt;/code&gt; argument to false):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;R&amp;gt; library(Rcpp)&#xA;R&amp;gt; library(inline)&#xA;R&amp;gt; src &amp;lt;- &quot;double x = Rcpp::as&amp;lt;double&amp;gt;(xs);&#xA;+         double y = Rcpp::as&amp;lt;double&amp;gt;(ys);&#xA;+         return(Rcpp::wrap(Rf_dpois(x, y, false)));&quot;&#xA;R&amp;gt; fun &amp;lt;- cxxfunction(signature(xs=&quot;numeric&quot;,ys=&quot;numeric&quot;), src, plugin = &quot;Rcpp&quot;)&#xA;R&amp;gt; fun(2, 0.5)&#xA;[1] 0.0758163&#xA;R&amp;gt; dpois(2, 0.5)&#xA;[1] 0.0758163&#xA;R&amp;gt; fun(1.0, 0.5)&#xA;[1] 0.303265&#xA;R&amp;gt; dpois(1.0, 0.5)&#xA;[1] 0.303265&#xA;R&amp;gt; &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The code to &lt;code&gt;dpois()&lt;/code&gt; is in &lt;code&gt;R-2.12.1/src/nmath/dpois.c&lt;/code&gt;. And yes, it can be freely copied.&lt;/p&gt;&#xA;" OwnerUserId="334" LastActivityDate="2011-01-29T16:31:29.507" CommentCount="1" />
  <row Id="6697" PostTypeId="2" ParentId="6421" CreationDate="2011-01-29T18:37:00.597" Score="3" ViewCount="" Body="&lt;p&gt;Just falling within the 15 year window, I believe, are the algorithms for controlling &lt;a href=&quot;http://en.wikipedia.org/wiki/False_discovery_rate&quot; rel=&quot;nofollow&quot;&gt;False Discovery Rate&lt;/a&gt;. I like the 'q-value' approach. &lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2011-01-29T18:37:00.597" CommunityOwnedDate="2011-01-29T18:37:00.597" CommentCount="1" />
  <row Id="6698" PostTypeId="1" AcceptedAnswerId="6699" CreationDate="2011-01-29T20:30:28.627" Score="2" ViewCount="102" Body="&lt;p&gt;Let's say you regress monthly stock returns, and the regression model has a standard error around such monthly stock returns of 2%.  Next, you forecast the next year annual stock return based on the regressed monthly returns.  What is the standard error around this annual stock return? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some may think this is not a very good example.  Don't get hung up on what is the proper way to model stock returns.  This has nothing to do with the question.  The question is simply figuring out the calculation to convert a monthly standard error into an annual one.  &lt;/p&gt;&#xA;" OwnerUserId="1329" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-04-28T22:57:34.777" LastActivityDate="2011-04-28T23:46:12.220" Title="How do you compute the annual standard error of a regression model when the model itself is based on monthly observations?" Tags="&lt;standard-error&gt;" AnswerCount="2" />
  <row Id="6699" PostTypeId="2" ParentId="6698" CreationDate="2011-01-29T20:32:23.547" Score="1" ViewCount="" Body="&lt;p&gt;The general rule is $\sqrt N$. So from daily data, annual volatility would be estimated as &lt;code&gt;sd(v)* sqrt(255)&lt;/code&gt;.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hence for your example, multiply the estimated error by the square root of twelve.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Edit:&lt;/em&gt; In response to the follow-up comment, here is a complete worked example with real data. The &lt;em&gt;annual&lt;/em&gt; volatility comes out to be around 16% which seems about right.  You could modify this for rolling volatilities, or weekly data, or ....&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;R&amp;gt; library(tseries)&#xA;Loading required package: quadprog&#xA;Loading required package: zoo&#xA;&#xA;    ‘tseries’ version: 0.10-22&#xA;&#xA;    ‘tseries’ is a package for time series analysis and computational finance.&#xA;&#xA;    See ‘library(help=&quot;tseries&quot;)’ for details.&#xA;&#xA;R&amp;gt; SP500 &amp;lt;- get.hist.quote(&quot;^GSPC&quot;, &quot;2000-01-01&quot;, &quot;2011-01-29&quot;, quote=&quot;Close&quot;, compression=&quot;m&quot;)&#xA;trying URL 'http://chart.yahoo.com/table.csv?s=^GSPC&amp;amp;a=0&amp;amp;b=01&amp;amp;c=2000&amp;amp;d=0&amp;amp;e=29&amp;amp;f=2011&amp;amp;g=m&amp;amp;q=q&amp;amp;y=0&amp;amp;z=^GSPC&amp;amp;x=.csv'&#xA;Content type 'text/csv' length unknown&#xA;opened URL&#xA;.......&#xA;downloaded 8162 bytes&#xA;&#xA;time series starts 2000-01-03&#xA;time series ends   2011-01-03&#xA;R&amp;gt; head(SP500)&#xA;             Close&#xA;2000-01-03 1394.46&#xA;2000-02-01 1366.42&#xA;2000-03-01 1498.58&#xA;2000-04-03 1452.43&#xA;2000-05-01 1420.60&#xA;2000-06-01 1454.60&#xA;R&amp;gt; sd(diff(log(SP500)))&#xA;    Close &#xA;0.0478781 &#xA;R&amp;gt; sd(diff(log(SP500)))*sqrt(12)&#xA;   Close &#xA;0.165855 &#xA;R&amp;gt; &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="334" LastEditorUserId="334" LastEditorDisplayName="" LastEditDate="2011-01-29T21:53:52.430" LastActivityDate="2011-01-29T21:53:52.430" CommentCount="4" />
  <row Id="6700" PostTypeId="2" ParentId="6658" CreationDate="2011-01-29T20:49:17.667" Score="1" ViewCount="" Body="&lt;p&gt;If I'm reading you right (and changing Tal's 4 to a 5), then at &lt;a href=&quot;http://en.wikipedia.org/wiki/Statistical_hypothesis_testing&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Statistical_hypothesis_testing&lt;/a&gt; &#xA;if you scroll halfway down you'll find the formula for &quot;Two-proportion z-test, pooled for d0 = 0.&quot;  I would think you'd want to do such a test for each of the five years, then choose a meta-analytic method of pooling the results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(You can also use an online calculator for each test.  &lt;a href=&quot;http://www.dimensionresearch.com/resources/calculators/ztest.html&quot; rel=&quot;nofollow&quot;&gt;http://www.dimensionresearch.com/resources/calculators/ztest.html&lt;/a&gt; and &lt;a href=&quot;http://www.surveystar.com/our_services/ztest.htm&quot; rel=&quot;nofollow&quot;&gt;http://www.surveystar.com/our_services/ztest.htm&lt;/a&gt; are not perfect but each looks serviceable.)&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;In light of further comments...From the research question you've posed, it sounds as if regional differences per se are not important.  Therefore you could simplify a great deal by collapsing across thorough-treatment regions and not-thorough-treatment regions, yielding two sets of regions for which to test the difference in proportions.  You could do this for each of the years on which you have a substantial amount of data.  Then you could pool the different years' test results using a standard meta-analytic method, and you would have a single answer to your question of whether the two levels of implementation show significantly different results.&lt;/p&gt;&#xA;" OwnerUserId="2669" LastEditorUserId="2669" LastEditorDisplayName="" LastEditDate="2011-01-31T18:48:45.210" LastActivityDate="2011-01-31T18:48:45.210" CommentCount="8" />
  <row Id="6701" PostTypeId="2" ParentId="949" CreationDate="2011-01-29T22:03:58.937" Score="2" ViewCount="" Body="&lt;p&gt;Unfortunately, I cannot post this as a comment due to lack of reputation, so please don't down-vote me just for that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm wondering if the OP is only interested in the case where the predictors are binary or if, perhaps, he's interested in cases where they may be continuous or take on other discrete values?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One example that doesn't quite answer the OP's question, but is related, and which I like, deals with item-preference rankings obtained via paired comparisons. The Bradley--Terry model can be expressed as a logistic regression where&#xA;$$&#xD;&#xA;\mathrm{logit}( \Pr(Y_{ij} = 1) ) = \alpha_i - \alpha_j ,&#xD;&#xA;$$&#xA;where the $\alpha_i$ is an &quot;affinity&quot;, &quot;popularity&quot;, or &quot;strength&quot; parameter of item $i$ and $Y_{ij} = 1$ indicates item $i$ was preferred over item $j$ in a paired comparison.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If a full round-robin of comparisons is performed (i.e., a pairwise preference is recorded for each unordered $(i,j)$ pair), then it turns out that the rank order of the MLEs $\hat{\alpha}_i$ correspond to the rank order of $S_i = \sum_{j \neq i} Y_{ij}$, the sum total of times each object was preferred over another.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you think of sports and imagine a full round-robin tournament, then it says that the B--T model ranks the players/teams according to their winning percentage. Whether this is an encouraging or disappointing result depends on your point of view, I suppose.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(The rank-ordering result does not hold, in general, when a full round-robin is not played.)&lt;/p&gt;&#xA;" OwnerUserId="2970" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-01-30T03:26:47.567" LastActivityDate="2011-01-30T03:26:47.567" CommentCount="1" />
  <row Id="6702" PostTypeId="1" AcceptedAnswerId="10490" CreationDate="2011-01-29T22:51:20.317" Score="3" ViewCount="203" Body="&lt;p&gt;Here's what I want to do, but there seem to be no &lt;code&gt;predict&lt;/code&gt; method for the mlogit.  Any ideas?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(mlogit)&#xA;data(&quot;Fishing&quot;, package = &quot;mlogit&quot;)&#xA;Fish &amp;lt;- mlogit.data(Fishing, varying = c(2:9), shape = &quot;wide&quot;, choice = &quot;mode&quot;)&#xA;&#xA;Fish_fit&amp;lt;-Fish[-1,]&#xA;Fish_test&amp;lt;-Fish[1,]&#xA;m &amp;lt;- mlogit(mode ~price+ catch | income, data = Fish_fit)&#xA;predict(m,newdata=Fish_test)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2817" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-01-30T14:15:55.020" LastActivityDate="2011-05-08T02:23:47.900" Title="Predict after running the mlogit function in R" Tags="&lt;r&gt;&lt;logistic-regression&gt;&lt;logit&gt;&lt;multinomial&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="2" />
  <row Id="6703" PostTypeId="2" ParentId="6609" CreationDate="2011-01-29T23:59:29.663" Score="2" ViewCount="" Body="&lt;p&gt;One way is to build an SPSS PLUM or NOMREG model that checks for an interaction between each predictor and a binary predictor, “time.”  In that scenario you'd use just a single column for all the values of your outcome variable.  For 1/2 the data set, time would be marked 0, and for the other half it'd be marked 1.  Essentially you’d be treating time as if it were like gender or any other binary predictor that potentially could interact with other predictors.&lt;/p&gt;&#xA;" OwnerUserId="2669" LastEditorUserId="2669" LastEditorDisplayName="" LastEditDate="2011-02-25T03:13:15.720" LastActivityDate="2011-02-25T03:13:15.720" />
  <row Id="6704" PostTypeId="2" ParentId="6601" CreationDate="2011-01-30T00:05:34.997" Score="4" ViewCount="" Body="&lt;p&gt;Conditional probability probably leads to most &lt;em&gt;mistakes&lt;/em&gt; in everyday experience. There are many harder concepts to grasp, of course, but people usually don't have to worry about them--this one they can't get away from &amp;amp; is a source of rampant misadventure. &lt;/p&gt;&#xA;" OwnerUserId="2347" LastActivityDate="2011-01-30T00:05:34.997" CommunityOwnedDate="2011-01-30T00:05:34.997" CommentCount="1" />
  <row Id="6705" PostTypeId="1" CreationDate="2011-01-30T01:20:12.840" Score="6" ViewCount="133" Body="&lt;p&gt;I'm doing an autocorrelation analysis for a spatially distributed collection of observations. To perform my analysis, I am using Moran's I statistic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My questions are: &lt;strong&gt;(1)&lt;/strong&gt; What are the implications and benefits of using different weighting functions, i.e. $d^{-1}$, $d^{-2}$, $\exp(-d)$, and &lt;strong&gt;(2)&lt;/strong&gt; Is there any (perhaps informal) answer to which of the possible weighting functions is used most frequently in the geo-statistics literature (and for what purposes)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for why I care: I am trying to explore whether there is clustering in my data set at different scales of structure, following some of the methodology of &lt;a href=&quot;http://www.esajournals.org/doi/abs/10.1890/0012-9658%282000%29081%5B0773%3ASDPPIT%5D2.0.CO%3B2&quot; rel=&quot;nofollow&quot;&gt;Fauchald 2000&lt;/a&gt;. I am plotting Moran's I versus aggregation scale. The interesting thing that the resulting correlation curves show very different qualitative behavior when calculating using $d^{-1}$ and $d^{-2}$ weighting functions ($d^{-1}$ has a discontinuity point, for example). I'm having a hard time understanding why this would be true -- does anyone have experience with this who may be able to point me to some references?&lt;/p&gt;&#xA;" OwnerUserId="1283" LastActivityDate="2011-05-03T08:46:51.997" Title="Choice of weight function in Moran's I" Tags="&lt;clustering&gt;&lt;autocorrelation&gt;&lt;spatial&gt;&lt;scale-invariance&gt;" AnswerCount="2" FavoriteCount="0" />
  <row Id="6706" PostTypeId="1" AcceptedAnswerId="6708" CreationDate="2011-01-30T01:23:50.887" Score="2" ViewCount="267" Body="&lt;p&gt;I am doing the following to fit my data using an exponential function:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Define the data&#xA;data &amp;lt;- c(67, 81, 93, 65, 18, 44, 31, 103, 64, 19, 27, 57, 63, 25, 22, 150, &#xA;          31, 58, 93, 6, 86, 43, 17, 9, 78, 23, 75, 28, 37, 23, 108, 14, 137, &#xA;          69, 58, 81, 62, 25, 54, 57, 65, 72, 17, 22, 170, 95, 38, 33, 34, 68, &#xA;          38, 117, 28, 17, 19, 25, 24, 15, 103, 31, 33, 77, 38, 8, 48, 32, 48, &#xA;          26, 63, 16, 70, 87, 31, 36, 31, 38, 91, 117, 16, 40, 7, 26, 15, 89, &#xA;          67, 7, 39, 33, 58)&#xA;&#xA;# Fit the data to a model&#xA;params = fitdistr(data, &quot;exponential&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I got the following for the params:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; params&#xA;      rate    &#xA;  0.019694623 &#xA; (0.002087626)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to draw a QQ-plot to see how good the fit was. I am guessing I need to generate exponentially distributed data using the parameter generated and then use some function to draw the QQ plot but am not sure how to go about doing this. Can someone tell me how to do this in R? &lt;/p&gt;&#xA;" OwnerUserId="2164" LastActivityDate="2011-02-07T18:28:04.613" Title="How do I generate a QQ-Plot for data fitted using fitdistr?" Tags="&lt;r&gt;&lt;distributions&gt;&lt;goodness-of-fit&gt;&lt;exponential&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="6707" PostTypeId="1" AcceptedAnswerId="6727" CreationDate="2011-01-30T01:50:47.830" Score="2" ViewCount="137" Body="&lt;p&gt;I've seen at least 3 sources on time series* state that the component of a series that is variously called random, stochastic, or noise (something clearly separate from any deterministic, patterned component) itself consists of 2 parts, a systematic part and an unsystematic part.  I can't for the life of me figure out how the random part can in turn have a systematic part.  I can see how the &lt;em&gt;series&lt;/em&gt; can have unsystematic and systematic parts, but not how the random component of the series could have both of these.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;*For example, Burns and Grove, The Practice of Nursing Research (2005, p. 475-6).&lt;/p&gt;&#xA;" OwnerUserId="2669" LastActivityDate="2011-01-30T22:16:38.797" Title="A &quot;systematic&quot; part of a random time series component?" Tags="&lt;time-series&gt;" AnswerCount="3" />
  <row Id="6708" PostTypeId="2" ParentId="6706" CreationDate="2011-01-30T01:53:42.750" Score="3" ViewCount="" Body="&lt;p&gt;Try the following code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;simdata &amp;lt;- qexp(ppoints(length(data)), rate = params$estimate)&#xA;qqplot(data, simdata)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(Inspired by the base R implementation of &lt;code&gt;qqnorm&lt;/code&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS When using non-base R functions, you should state what library they come from.  I had to Google to discover that &lt;code&gt;fitdistr&lt;/code&gt; is from MASS.&lt;/p&gt;&#xA;" OwnerUserId="2975" LastActivityDate="2011-01-30T01:53:42.750" CommentCount="5" />
  <row Id="6709" PostTypeId="2" ParentId="6655" CreationDate="2011-01-30T03:41:02.227" Score="0" ViewCount="" Body="&lt;p&gt;I would suggest that the type of estimator depends on a few things:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;What are the consequences of getting the estimate wrong? (e.g. is it less bad if your estimator is too high, compared to being too low? or are you indifferent about the direction of error?  if an error is twice as big, is this twice as bad? is it percentage error or absolute error that is important? Is the estimation only intermediate step that is required for prediction? is large sample behaviour more or less important than small sample behaviour?)&lt;/li&gt;&#xA;&lt;li&gt;What is your prior information about the quantity you are estimating? (e.g. how is the data functionally related to your quantity? do you know if the quantity is positive? discrete? have you estimated this quantity before? how much data do you have? Is there any &quot;group invariance&quot; structure in your data?)&lt;/li&gt;&#xA;&lt;li&gt;What software do you have? (e.g. no good suggesting MCMC if you don't have the software to do it, or using a GLMM if you don't know how to do it.)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The first two points are context specific, and by thinking about &lt;em&gt;your specific application&lt;/em&gt;, you will generally be able to define certain &lt;em&gt;properties&lt;/em&gt; that you would like your estimator to have.  You then choose the estimator which you can actually calculate, which has as many of the properties which you want it to have.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think the lack of context that a teaching course has with estimation, means that often &quot;default&quot; criterion are used, similarly for prior information (the most obvious &quot;default&quot; being that you know the sampling distribution of your data).  Having said that, some of the default methods are good, especially if you don't know enough about the context.  But if you &lt;em&gt;do&lt;/em&gt; know the context, and you &lt;em&gt;have the tools&lt;/em&gt; to incorporate that context, then you should, for otherwise you may get counter-intuitive results (because of what you ignored).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The I'm not a big fan of MVUE as a general rule, because you often need to sacrifice too much variance to get unbiased-ness.  For example, imagine you are throwing darts at a dartboard, and you want to hit to the bulls-eye.  Supposing that the maximum deviation from the bulls-eye is 6cm for a particular throwing strategy, but the center of the dart points is 1cm above of the bullseye.  This is not MVUE, because the center should be on the bullseye.  But suppose that in order to shift the distribution down 1cm (on the average), you have to increase your radius to at least 10cm (so the maximum error is now 10cm, and not 6cm). This is the kind of thing that can happen with MVUE, &lt;em&gt;unless the variance is already small&lt;/em&gt;.  Suppose I was a much more accurate throw, and could narrow my error to 0.1cm.  Now the bias really matters, because I will never hit the bullseye!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In short, for me, bias only matters when it is small compared to the variance.  And you will usually only get small variances when you have a large sample.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-01-30T03:41:02.227" />
  <row Id="6710" PostTypeId="2" ParentId="6421" CreationDate="2011-01-30T04:04:07.940" Score="2" ViewCount="" Body="&lt;p&gt;Adding my own 5 cents, I believe the most significant breakthrough of the past 15 years has been Compressed Sensing. LARS, LASSO, and a host of other algorithms fall in this domain, in that Compressed Sensing explains why they work and extends them to other domains.&lt;/p&gt;&#xA;" OwnerUserId="30" LastActivityDate="2011-01-30T04:04:07.940" CommunityOwnedDate="2011-01-30T04:04:07.940" />
  <row Id="6712" PostTypeId="2" ParentId="6670" CreationDate="2011-01-30T04:40:55.097" Score="0" ViewCount="" Body="&lt;p&gt;What I usually do is to calculate the plugin bandwidth using Silverman's formula (h_p) and then crossvalidate in the range of [h_p/5, 5h_p] to find the optimal bandwidth. This crossvalidation can be done either by using leave-one-out least squares crossvalidation or by leave-one-out-likelihood crossvalidation.&lt;/p&gt;&#xA;" OwnerUserId="2964" LastActivityDate="2011-01-30T04:40:55.097" CommentCount="1" />
  <row Id="6713" PostTypeId="2" ParentId="6656" CreationDate="2011-01-30T05:49:25.220" Score="0" ViewCount="" Body="&lt;p&gt;Dividing by $\log{2}$ is simply for setting the base to 2, because $\log_b{x}=\frac{\log_e{x}}{\log_e{b}}$. As for multiplying by 50, I should have worded it better. There are two possible states for each element of a list for a list of length 100. So, assuming a frequency for each state is 50 (and a probability of 0.5), the formula simplifies to the one I gave above. Am I doing something wrong? Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="2976" LastActivityDate="2011-01-30T05:49:25.220" CommentCount="2" />
  <row Id="6714" PostTypeId="2" ParentId="6652" CreationDate="2011-01-30T06:15:56.497" Score="2" ViewCount="" Body="&lt;p&gt;I wouldn't call the definition of CIs as wrong, but they are easy to mis-interpret, due to there being more than one definition of probability.  CIs are based on the following definition of Probability (Frequentist or ontological)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(1)&lt;em&gt;probability of a proposition=long run proportion of times that proposition is observed to be true, conditional on the data generating process&lt;/em&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, in order to be conceptually valid in using a CI, you &lt;em&gt;must&lt;/em&gt; accept this definition of probability.  If you don't, then your interval is not a CI, from a theoretical point of view.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is why the definition used the word &lt;strong&gt;proportion&lt;/strong&gt; and NOT the word &lt;strong&gt;probability&lt;/strong&gt;, to make it clear that the &quot;long run frequency&quot; definition of probability is being used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The main alternative definition of Probability (Epistemological or probability as an extension of deductive Logic or Bayesian) is &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(2)&lt;em&gt;probability of a proposition = rational degree of belief that the proposition is true, conditional on a state of knowledge&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;People often intuitively get both of these definitions mixed up, and use whichever interpretation happens to appeal to their intuition.  This can get you into all kinds of confusing situations (especially when you move from one paradigm to the other).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That the two approaches often lead to the same result, means that in some cases we have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;rational degree of belief that the proposition is true, conditional on a state of knowledge = long run proportion of times that proposition is observed to be true, conditional on the data generating process&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The point is that it does not hold &lt;em&gt;universally&lt;/em&gt;, so we cannot expect the two different definitions to always lead to the same results.  So, unless you actually work out the Bayesian solution, and then find it to be the same interval, you cannot give the interval given by the CI the interpretation as a probability of containing the true value.  And if you do, then the interval is not a Confidence Interval, but a Credible Interval.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastEditorUserId="2392" LastEditorDisplayName="" LastEditDate="2011-02-01T11:04:32.597" LastActivityDate="2011-02-01T11:04:32.597" CommentCount="17" />
  <row Id="6715" PostTypeId="1" CreationDate="2011-01-30T06:23:56.603" Score="7" ViewCount="84" Body="&lt;p&gt;Please pardon me if this question is not clear. I am not sure if I am using the right terminologies.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have conducted an experiment in different environments multiple times. So my data looks something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Environment1  1.2  2.1  1.1  1.5  1.6&#xA;Environment2  4.2  2.6  3.5  2.5  2.9&#xA;Environment3  7.2  4.6  5.3  4.5  1.6&#xA;Environment4  0.0  0.0  1.2  15.0 0.0&#xA;Environment5  3.2  2.4  7.2  5.5  6.6&#xA;Environment6  23.2  32.1  18.1  1.5  19.6&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I can clearly see (or maybe my intuition says) that the experiment was not conducted properly in Environment4 (too low and fluctuating a lot) and Environment5 (way too high) but am not sure how to prove this. Am I supposed to rely on hypothesis-testing with the hypothesis:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The experiment was not conducted properly in Environments 4 and 6.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;and then use some procedure to prove this? Or is there a standard way of showing this? Can someone please help me how to approach this kind of problems? I am using R.&lt;/p&gt;&#xA;" OwnerUserId="2164" LastEditorUserId="2164" LastEditorDisplayName="" LastEditDate="2011-01-30T06:32:25.987" LastActivityDate="2011-01-30T06:43:32.580" Title="Can someone help me understand what type of problem I am looking at? Not sure if this classifies as hypothesis-testing." Tags="&lt;r&gt;&lt;distributions&gt;&lt;hypothesis-testing&gt;&lt;statistical-significance&gt;&lt;experiment-design&gt;" AnswerCount="1" CommentCount="5" />
  <row Id="6716" PostTypeId="2" ParentId="6715" CreationDate="2011-01-30T06:43:32.580" Score="1" ViewCount="" Body="&lt;p&gt;You can do a &lt;a href=&quot;http://en.wikipedia.org/wiki/Student%27s_t-test&quot; rel=&quot;nofollow&quot;&gt;student test&lt;/a&gt; to see if the mean is different between the group 4,6 and the rest. Even if your sample size is small you will conclude in a difference. Note that it will tell you that group 4,6 is significantly different in average from the rest but it won't tell you that &quot;The experiment was not conducted properly in Environments 4 and 6&quot; which can't be answered without a knowledge of what &quot;properly&quot; means in the observations. &lt;/p&gt;&#xA;" OwnerUserId="223" LastActivityDate="2011-01-30T06:43:32.580" CommentCount="4" />
  <row Id="6717" PostTypeId="2" ParentId="6707" CreationDate="2011-01-30T07:00:30.623" Score="2" ViewCount="" Body="&lt;p&gt;&quot;random&quot; is often used as if it was a real property of the data under study, where it should be replaced with &quot;uncertain&quot;.  To give an example, if I ask you what how much money you earned over the past month, and you don't tell me, it is not &quot;random&quot;, but just uncertain.  However, treating the uncertainty as if it was random allows you to make some useful conclusions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The noise is not &quot;random&quot; per se, but given that we usually have limited knowledge of how each particular piece of &quot;noise&quot; is generated, assuming that it is random can be useful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now whenever you fit a model to some data, you will have residuals from that model.  And if treating the &quot;noise&quot; as if it was random was a good idea, then the residuals from the model should be consistent with whatever definition of &quot;randomness&quot; you have used in fitting the model.  If they are not, then basically the time series is telling you that the &quot;randomness&quot; you assumed is not a good description of what it actually happening, and it gives you a clue as to what a better description might be.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if I fit a linear relationship for the systematic part, but it is actually quadratic, then the so-called &quot;random&quot; noise will not look random at all, rather it will contain the squared component of the systemtatic part.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To make it even more concrete, suppose that your response $Y$ is a deterministic function of $X$, say $Y=3+2X+X^2$.  Now, because you don't know this function you suppose that $Y=\alpha+\beta X + error$, and because you have no reason to doubt the model prior to seeing the data, you assume that the error is just &quot;random noise&quot; (usually $N(0,\sigma^2)$).  However, once you actually fit your data and look at the residuals, they will all line up as an exact quadratic function of the residuals.  Thus there is a systematic component to the &quot;noise&quot; (in fact the &quot;noise&quot; is entirely systematic).  This is basically &quot;Nature&quot; telling &quot;You&quot; that you model is wrong, and gives a clue as to how it could be improved.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The same kind of thing is happening in the time series.  you could just replace the model above with $Y_{1}=1,Y_{2}=6,Y_{3}=0.5,Y_{4}=10,Y_{5}=3,Y_{6}=10$ and for $t\geq7$ have $Y_t=10+2 Y_{t-1} -5Y_{t-1}^3 + 2Y_{t-5}$  and the same kind of thing would happen.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-01-30T07:00:30.623" />
  <row Id="6718" PostTypeId="2" ParentId="6684" CreationDate="2011-01-30T07:50:33.400" Score="2" ViewCount="" Body="&lt;p&gt;When you use the &lt;code&gt;predict&lt;/code&gt; with &lt;code&gt;newdata&lt;/code&gt; argument you must supply the the data.frame with the same column names. In your code you have&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;newdata &amp;lt;- expand.grid(X1=c(-1,0,1),X2=c(-1,0,1))&#xA;names(newdata) &amp;lt;- c(&quot;scale(xxA)&quot;,&quot;scale(xxB)&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But the formula supplied to lm object is&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;out ~ scale(xxA)*scale(xxB)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So when you call the predict, it tries to find objects &lt;code&gt;xxA&lt;/code&gt; and &lt;code&gt;xxB&lt;/code&gt; in your data and apply function &lt;code&gt;scale&lt;/code&gt; as per your initial request. But all R finds are objects &lt;code&gt;scale(xxA)&lt;/code&gt; and &lt;code&gt;scale(xxB)&lt;/code&gt;. So naturally it produces the error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now if you supply correctly named &lt;code&gt;newdata&lt;/code&gt; &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;newdata &amp;lt;- expand.grid(xxA=c(-1,0,1),xxB=c(-1,0,1)) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and try to use it for prediction&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;newdata$Y &amp;lt;- predict(lm.res.scale,newdata)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;R will remember the how it scaled original data and apply the same scaling to your new data. In this case supplied value -1 for &lt;code&gt;xxA&lt;/code&gt; will be subtracted the original mean of &lt;code&gt;xxA&lt;/code&gt; and divided by the original standard value of &lt;code&gt;xxA&lt;/code&gt;. If you want to get prediction of 1 S.D below the mean, you will need to supply this value. In your case then newdata should look like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;newdata &amp;lt;- expand.grid(xxA=mean(dat$xxA)+sd(dat$xxA)*c(-1,0,1),xxB=mean(dat$xxB)+sd(dat$xxB)*c(-1,0,1))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I gathered all the solutions in one place to compare:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;##Prepare data&#xA;set.seed(1)&#xA;dat &amp;lt;- data.frame(xxA = rnorm(20,10), xxB = rnorm(20,20))&#xA;dat$out &amp;lt;- with(dat,xxA+xxB+xxA*xxB+rnorm(20,20))&#xA;dat &amp;lt;- within(dat,{&#xA;    X1 &amp;lt;- as.numeric(scale(xxA))&#xA;    X2 &amp;lt;- as.numeric(scale(xxB))&#xA;    })&#xA;&#xA;##Estimate the models&#xA;lm.res.scale &amp;lt;- lm(out ~ scale(xxA)*scale(xxB),data=dat)&#xA;lm.res.correct &amp;lt;- lm(out~X1*X2,data=dat)&#xA;lm.mod &amp;lt;- lm(out ~ I(scale(xxA))*I(scale(xxB)), data=dat)&#xA;rms.res &amp;lt;- Glm(out ~ scale(xxA)*scale(xxB),data=dat)&#xA;&#xA;&#xA;##Build data for prediction&#xA;newdata &amp;lt;- expand.grid(xxA=c(-1,0,1),xxB=c(-1,0,1))&#xA;newdata$X1&amp;lt;-newdata$xxA&#xA;newdata$X2&amp;lt;-newdata$xxB&#xA;&#xA;##Gather the predictions&#xA;newdata$Yscaled &amp;lt;- predict(lm.res.scale,newdata)&#xA;newdata$Ycorrect &amp;lt;- predict(lm.res.correct,newdata)&#xA;newdata$YwithI &amp;lt;- predict(lm.mod,newdata)&#xA;newdata$Ywithrms &amp;lt;- Predict(rms.res,xxA=c(-1,0,1),xxB=c(-1,0,1),conf.int=FALSE)[,3]&#xA;&#xA;##Build alternative data for prediction&#xA;newdata2 &amp;lt;- expand.grid(xxA=mean(dat$xxA)+sd(dat$xxA)*c(-1,0,1),xxB=mean(dat$xxB)+sd(dat$xxB)*c(-1,0,1))&#xA;&#xA;#Predict&#xA;newdata$Yorigsc &amp;lt;- predict(lm.res.scale,newdata2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I used &lt;code&gt;set.seed&lt;/code&gt; so the results should be the same if you try to repeat it. The newdata looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; newdata&#xA;  xxA xxB X1 X2  Yscaled Ycorrect   YwithI Ywithrms  Yorigsc&#xA;1  -1  -1 -1 -1 25.79709 225.9562 221.7517 221.7517 225.9562&#xA;2   0  -1  0 -1 25.63030 244.5181 243.0404 243.0404 244.5181&#xA;3   1  -1  1 -1 25.46351 263.0800 264.3291 264.3291 263.0800&#xA;4  -1   0 -1  0 25.36341 234.6981 231.7012 231.7012 234.6981&#xA;5   0   0  0  0 26.21499 254.0704 254.0704 254.0704 254.0704&#xA;6   1   0  1  0 27.06657 273.4427 276.4396 276.4396 273.4427&#xA;7  -1   1 -1  1 24.92972 243.4400 241.6507 241.6507 243.4400&#xA;8   0   1  0  1 26.79967 263.6227 265.1004 265.1004 263.6227&#xA;9   1   1  1  1 28.66962 283.8054 288.5501 288.5501 283.8054&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As expected &lt;code&gt;Yscaled&lt;/code&gt; produces not the result we need since the original scaling &#xA;is applied. In the case when we scale data before &lt;code&gt;lm&lt;/code&gt; (&lt;code&gt;Ycorrect&lt;/code&gt;) and when we supply alternative unscaled values (&lt;code&gt;Yorigsc&lt;/code&gt;) results coincide and are the ones needed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now the other prediction methods give different results. This happens since R is forced to forget the original scaling using formula&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;out ~ I(scale(xxA))*I(scale(xxB))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;or package &lt;code&gt;rms&lt;/code&gt;. But when we use predict, the values are still scaled, but now according to supplied values of &lt;code&gt;xxA&lt;/code&gt; and &lt;code&gt;xxB&lt;/code&gt;. This is best illustrated by following statement, which in some way mimics what predict does with the data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; eval(expression(cbind(scale(xxA),scale(xxB))),env=as.list(newdata))&#xA;           [,1]      [,2]&#xA; [1,] -1.154701 -1.154701&#xA; [2,]  0.000000 -1.154701&#xA; [3,]  1.154701 -1.154701&#xA; [4,] -1.154701  0.000000&#xA; [5,]  0.000000  0.000000&#xA; [6,]  1.154701  0.000000&#xA; [7,] -1.154701  1.154701&#xA; [8,]  0.000000  1.154701&#xA; [9,]  1.154701  1.154701&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;We can see that in this case, scaling does not change original values too much, but this is even worse, since the values from predict look reasonable, when in fact they are wrong.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-01-30T20:04:27.090" LastActivityDate="2011-01-30T20:04:27.090" CommentCount="5" />
  <row Id="6719" PostTypeId="2" ParentId="6707" CreationDate="2011-01-30T09:21:36.527" Score="1" ViewCount="" Body="&lt;p&gt;Systematic and unsystematic are rather ambiguous terms. One of the possible explanations is given by @probabilityislogic. Another may be given &lt;a href=&quot;http://en.wikipedia.org/wiki/Decomposition_of_time_series&quot; rel=&quot;nofollow&quot;&gt;here.&lt;/a&gt; Since the context you gave is time series, I think this might be related to  &lt;a href=&quot;http://en.wikipedia.org/wiki/Wold%27s_theorem&quot; rel=&quot;nofollow&quot;&gt;Wold's theorem&lt;/a&gt;.  Unfortunately wikipedia text captures the essence, but does not go into the details of which part is systematic and non systematic. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I did not manage to find appropriate link to refer to, so I will try give some explanation based on the &lt;a href=&quot;http://www.setbook.com.ua/books/120541.html?PHPSESSID=i41mmv16i1t034pb3u0o0pkkf1&quot; rel=&quot;nofollow&quot;&gt;book&lt;/a&gt; I have. This subject is also discussed &lt;a href=&quot;http://books.google.com/books?id=BjaHPwAACAAJ&amp;amp;dq=cramer+leadbetter&amp;amp;hl=fr&amp;amp;ei=pGpFTdDQOZChOtzGuJEC&amp;amp;sa=X&amp;amp;oi=book_result&amp;amp;ct=result&amp;amp;resnum=1&amp;amp;ved=0CCkQ6AEwAA&quot; rel=&quot;nofollow&quot;&gt;in this book&lt;/a&gt;. I will not give precise and rigorous definitions, since they involve Hilbert spaces and other graduate mathematics stuff, which I think is not really necessary to get the point across. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each &lt;a href=&quot;http://en.wikipedia.org/wiki/Stationary_process&quot; rel=&quot;nofollow&quot;&gt;covariance-stationary&lt;/a&gt; process $\{X_t,t\in \mathbb{Z}\}$ can be uniquely decomposed into two stationary proceses: $X_t=M_t+N_t$, singular $M_t$ and regular $N_t$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Singular and regular processes are defined via their prediction properties. In stationary process theory the prediction of process $X_t$ at time $t$ is formed from linear span of its history $(X_s,s&amp;lt;t)$. Singular processes are processes for which the prediction error:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$E(\hat{X}_t-X_t)^2$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;is zero. Such processes sometimes are called deterministic, and in your context can be also called systematic. The most simple example of such process is $X_t=\eta$ for all $t$ and $\eta$ some random variable. Then the linear prediction of $X_t$ based on its history will always be $\eta$. The error of such prediction as defined above would be zero. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regular stationary processes on the other hand cannot be predicted without error from their history. It can be shown that the stationary process $N_t$ is regular if and only if it admits  $MA(\infty)$ decomposition. This means that there exists &lt;a href=&quot;http://en.wikipedia.org/wiki/White_noise&quot; rel=&quot;nofollow&quot;&gt;white-noise&lt;/a&gt; sequence $(\varepsilon_t)$ such that&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$N_t=\sum_{t=0}^{\infty}c_n\varepsilon_{t-n}.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where coefficients $c_n$ are such, that the equality holds. These processes sometimes are called non-deterministic, or probably non-systematic in your case.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-01-30T13:51:17.730" LastActivityDate="2011-01-30T13:51:17.730" CommentCount="3" />
  <row Id="6720" PostTypeId="1" AcceptedAnswerId="6726" CreationDate="2011-01-30T11:56:51.913" Score="4" ViewCount="197" Body="&lt;p&gt;An exponential model is a model described by following equation:&lt;br&gt;&#xA;$$\hat{y_{i}}=\beta_{0}\cdot e^{\beta_{1}x_{1i}+\ldots+\beta_{k}x_{ki}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The most common approach used to estimate such model is linearization, which can be done easily by calculating logarithms of both sides. What are the other approaches? I'm especially interested in those which can handle $y_{i}=0$ in some observations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update 31.01.2011&lt;/strong&gt;&lt;br&gt;&#xA;I'm aware of the fact that this model can't produce zero. I'll elaborate a bit what I'm modeling and why I choose this model. Let's say we want to predict how much money does a client spend in a shop. Of course many clients are just looking and they don't buy anything, that why there are 0. I didn't want to use linear model because it produces a lot of negative values, which doesn't make any sense. The other reason is that this model works really good, much better than the linear. I've used genetic algorithm to estimate those parameters so it wasn't 'scientific' approach. Now I'd like to know how to deal with problem using more scientific methods. It can be also assumed that most, or even all, of the variables are binary variables.&lt;/p&gt;&#xA;" OwnerUserId="1643" LastEditorUserId="1643" LastEditorDisplayName="" LastEditDate="2011-01-31T09:50:09.067" LastActivityDate="2011-01-31T14:48:08.047" Title="Estimation of exponential model" Tags="&lt;estimation&gt;&lt;nonlinear-regression&gt;" AnswerCount="3" CommentCount="1" FavoriteCount="2" />
  <row Id="6721" PostTypeId="2" ParentId="6720" CreationDate="2011-01-30T12:30:39.593" Score="7" ViewCount="" Body="&lt;p&gt;This is a &lt;a href=&quot;http://en.wikipedia.org/wiki/Generalized_linear_model&quot; rel=&quot;nofollow&quot;&gt;generalized linear model&lt;/a&gt; (GLM) with a log &lt;a href=&quot;http://en.wikipedia.org/wiki/Generalized_linear_model#Link_function&quot; rel=&quot;nofollow&quot;&gt;link function&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any probability distribution on $[0,\infty)$ with non-zero density at zero will handle $y_i=0$ in some observations; the most common would be the Poisson distribution, resulting in &lt;a href=&quot;http://en.wikipedia.org/wiki/Poisson_regression&quot; rel=&quot;nofollow&quot;&gt;Poisson regression&lt;/a&gt;, a.k.a. log-linear modelling. Another choice would be a &lt;a href=&quot;http://en.wikipedia.org/wiki/Negative_binomial_distribution&quot; rel=&quot;nofollow&quot;&gt;negative binomial distribution&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you don't have count data, or if $y_i$ takes non-integer values, you can still use the framework of generalized linear models without fully specifying a distribution for $\operatorname{P}(y_i|\bf{x})$ but instead only specifying the relationship between its mean and variance using &lt;a href=&quot;http://en.wikipedia.org/wiki/Quasi-likelihood&quot; rel=&quot;nofollow&quot;&gt;quasi-likelihood&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="449" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-01-30T12:50:51.010" LastActivityDate="2011-01-30T12:50:51.010" CommentCount="2" />
  <row Id="6722" PostTypeId="2" ParentId="6720" CreationDate="2011-01-30T14:04:34.183" Score="3" ViewCount="" Body="&lt;p&gt;You can always use &lt;a href=&quot;http://en.wikipedia.org/wiki/Non-linear_least_squares&quot; rel=&quot;nofollow&quot;&gt;non-linear least squares&lt;/a&gt;. Then your model will be:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$y_i=\beta_0\exp(\beta_1x_{1i}+...+\beta_kx_{ki})+\varepsilon_i$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The zeroes in $y_i$ then will be treated as deviations from the non-linear trend.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-01-30T14:04:34.183" CommentCount="2" />
  <row Id="6723" PostTypeId="1" AcceptedAnswerId="6752" CreationDate="2011-01-30T15:14:03.557" Score="2" ViewCount="270" Body="&lt;p&gt;Given the dataset &lt;a href=&quot;http://dl.dropbox.com/u/138632/statistics/R%20data/cars.txt&quot; rel=&quot;nofollow&quot;&gt;cars.txt&lt;/a&gt;, we want to formulate a good regression model for the Midrange Price using the variables Horsepower, Length, Luggage, Uturn, Wheelbase, and Width. Both:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;using all possible subsets selection, and&lt;/li&gt;&#xA;&lt;li&gt;using an automatic selection technique.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;For the first part, we do in R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cars &amp;lt;- read.table(file=file.choose(), header=TRUE)&#xA;names(cars)&#xA;&#xA;#regression&#xA;attach(cars)&#xA;leap &amp;lt;- leaps(x=cbind(cars$Horsepower, cars$Length, cars$Luggage, cars$Uturn, cars$Wheelbase, cars$Width), &#xA;y=cars$MidrangePrice, method=c(&quot;r2&quot;), nbest=3)&#xA;combine &amp;lt;- cbind(leap$which,leap$size, leap$r2)&#xA;n &amp;lt;- length(leap$size)&#xA;dimnames(combine) &amp;lt;- list(1:n,c(&quot;horsep&quot;,&quot;length&quot;,&quot;Luggage&quot;,&quot;Uturn&quot;,&quot;Wheelbase&quot;,&quot;Width&quot;,&quot;size&quot;,&quot;r2&quot;))&#xA;round(combine, digits=3)&#xA;&#xA;leap.cp &amp;lt;- leaps(x=cbind(cars$Horsepower, cars$Length, cars$Luggage, cars$Uturn, cars$Wheelbase, cars$Width), &#xA;y=cars$MidrangePrice, nbest=3)&#xA;combine.cp &amp;lt;- cbind(leap.cp$which,leap.cp$size, leap.cp$Cp)&#xA;dimnames(combine.cp) &amp;lt;- list(1:n,c(&quot;horsep&quot;,&quot;length&quot;,&quot;Luggage&quot;,&quot;Uturn&quot;,&quot;Wheelbase&quot;,&quot;Width&quot;,&quot;size&quot;,&quot;cp&quot;))&#xA;round(combine.cp, digits=3)&#xA;plot(leap.cp$size, leap.cp$Cp, ylim=c(1,7))&#xA;abline(a=0, b=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Am I correct in my interpretation that the most adequate model is one with 4 parameters (the three variables Horsepower, Wheelbase and Width) because it has the lowest Mallows' Cp value?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the second part, we can choose between the forward, backward or stepwise selection models:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#stepwise selection methods&#xA;#forward&#xA;slm.foward &amp;lt;- step(lm(cars$MidrangePrice ~1, data=cars), scope=~cars$Horsepower + cars$Length + cars$Luggage + cars$Uturn + cars$Wheelbase + cars$Horsepower+ cars$Width, direction=&quot;forward&quot;)&#xA;&#xA;#backward&#xA;reg.lm1 &amp;lt;- lm(cars$MidrangePrice ~ cars$Horsepower + cars$Length + cars$Luggage + cars$Uturn + cars$Wheelbase + cars$Horsepower + cars$Width)&#xA;slm.backward &amp;lt;- step(reg.lm1, direction=&quot;backward&quot;)&#xA;&#xA;&#xA;#stepwise&#xA;reg.lm1 &amp;lt;- lm(cars$MidrangePrice ~ cars$Horsepower + cars$Length + cars$Luggage + cars$Uturn + cars$Wheelbase + cars$Horsepower + cars$Width)&#xA;slm.stepwise &amp;lt;- step(reg.lm1,direction=&quot;both&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How do I interpret the results I get from this R code?&lt;/p&gt;&#xA;" OwnerUserId="2980" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2011-05-27T04:02:47.707" LastActivityDate="2011-05-27T04:02:47.707" Title="Regression selection using all possible subsets selection and automatic selection techniques" Tags="&lt;r&gt;&lt;regression&gt;&lt;model-selection&gt;&lt;stepwise-regression&gt;" AnswerCount="2" CommentCount="4" />
  <row Id="6724" PostTypeId="2" ParentId="4556" CreationDate="2011-01-30T17:39:27.313" Score="2" ViewCount="" Body="&lt;p&gt;The &quot;mechanical&quot; result of just plugging in $z = 1$ into the transfer response is essentially a product of two facts. The steady-state gain is (usually, I believe) defined as the (magnitude of the) limiting response as $t \to \infty$ of the system to a unit-step input.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The so-called final-value theorem states that, &lt;em&gt;if the limit&lt;/em&gt; $\lim_{n \to \infty} y(n)$ &lt;em&gt;exists&lt;/em&gt;, then $\lim_{n \to \infty} y(n) = \lim_{z \to 1} (1-z^{-1}) Y(z)$ where $y(n)$ is the time-domain output and $Y(z)$ is its corresponding $z$-transform.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, for the steady-state gain, the input is a unit-step function, so $x(n) = 1$ for each $n \geq 0$. Hence,&#xA;$$&#xD;&#xA;X(z) = \sum_{n=0}^\infty x(n) z^{-n} = \sum_{n=0}^\infty z^{-n} = \frac{1}{1-z^{-1}} .&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using the transfer equation, we get that the $z$-transform of the output is&#xA;$$&#xD;&#xA;Y(z) = X(z) H(z) = \frac{H(z)}{1-z^{-1}} .&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Assuming that the limit $\lim_{n\to\infty} y(n)$ exists) we have that&#xA;$$&#xD;&#xA;\lim_{n \to \infty} y(n) = \lim_{z \to 1} (1-z^{-1}) Y(z) = \lim_{z \to 1}\, H(z) .&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The left-hand side is the steady-state value of a step-response (i.e., it is the value of the response as time goes to $\infty$ of a one-unit constant input), and so the steady-state gain is $|\lim_{n \to \infty} y(n)| = \lim_{n \to \infty} |y(n)|$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Technically, you need to check that the limit exists (which I've tried to emphasize). It seems to me that a sufficient condition would be that all the poles of the transfer response be &lt;em&gt;strictly&lt;/em&gt; inside the unit circle. (&lt;strong&gt;Caveat lector&lt;/strong&gt;: I haven't checked that closely at all.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If this does not sufficiently clarify things, you might try doing Google searches on terms like &quot;dc gain&quot; and &quot;final-value theorem&quot;, which are closely related to what you want.&lt;/p&gt;&#xA;" OwnerUserId="2970" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-01-30T18:04:18.460" LastActivityDate="2011-01-30T18:04:18.460" />
  <row Id="6725" PostTypeId="1" CreationDate="2011-01-30T17:56:44.463" Score="2" ViewCount="57" Body="&lt;p&gt;I have to implement a genetic algorithm and therefore select the &quot;fittest&quot; of all possible members of a generation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have the following arrays of fitnesses, their weighed equivalents and a cumulative array.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Fitnesses: [0.2, 0.0, 0.2, 0.0, 0.0, 0.0]&#xA;Weighed Fitnesses: [0.5, 0.0, 0.5, 0.0, 0.0, 0.0]&#xA;Cumulative Fitnesses: [0.5, 0.5, 1.0, 1.0, 1.0, 1.0]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now I have to choose five new members, with a higher probability of the chosen member being fitter than the others, i.e. the probability of being chosen proportional to the fitness.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How would I go about doing this by using a random number?&lt;/p&gt;&#xA;" OwnerUserId="1205" LastActivityDate="2011-01-31T00:08:48.627" Title="Pick a random item with the help of an array of cumulative probabilities" Tags="&lt;random-variable&gt;&lt;random-generation&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="6726" PostTypeId="2" ParentId="6720" CreationDate="2011-01-30T17:57:06.893" Score="8" ViewCount="" Body="&lt;p&gt;There are several issues here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(1) &lt;strong&gt;The model needs to be explicitly probabilistic&lt;/strong&gt;.  In almost all cases there will be &lt;em&gt;no&lt;/em&gt; set of parameters for which the lhs matches the rhs for all your data: there will be residuals.  You need to make assumptions about those residuals.  Do you expect them to be zero on the average?  To be symmetrically distributed?  To be approximately normally distributed?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are two models that agree with the one specified but allow drastically different residual behavior (and therefore will typically result in different parameter estimates).  You can vary these models by varying assumptions about the joint distribution of the $\epsilon_{i}$:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\text{A:}\ y_{i} =\beta_{0} \exp{\left(\beta_{1}x_{1i}+\ldots+\beta_{k}x_{ki} + \epsilon_{i}\right)}$$&#xA;$$\text{B:}\ y_{i} =\beta_{0} \exp{\left(\beta_{1}x_{1i}+\ldots+\beta_{k}x_{ki}\right)} + \epsilon_{i}.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Note that these are models for the &lt;em&gt;data&lt;/em&gt; $y_i$; there usually is no such thing as an &lt;em&gt;estimated&lt;/em&gt; data value $\hat{y_i}$.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(2) The need to handle zero values for the y's implies &lt;strong&gt;the stated model (A) is both wrong and inadequate&lt;/strong&gt;, because it cannot produce a zero value no matter what the random error equals.  The second model above (B) allows for zero (or even negative) values of y's.  However, one should not choose a model solely on such a basis.  To reiterate #1: it is important to model the errors reasonably well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(3) &lt;strong&gt;Linearization changes the model&lt;/strong&gt;.  Typically, it results in models like (A) but not like (B).  It is used by people who have analyzed their data enough to know this change will not appreciably affect the parameter estimates and by people who are ignorant of what is happening.  (It is hard, many times, to tell the difference.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(4) &lt;strong&gt;A common way to handle the possibility of a zero value&lt;/strong&gt; is to propose that $y$ (or some re-expression thereof, such as the square root) has a strictly positive chance of equally zero.  Mathematically, we are mixing a point mass (a &quot;delta function&quot;) in with some other distribution.  These models look like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\eqalign{&#xD;&#xA;f(y_i) &amp;amp;\sim F(\mathbf{\theta}); \cr&#xD;&#xA;\theta_j &amp;amp;= \beta_{j0} + \beta_{j1} x_{1i} + \cdots + \beta_{jk} x_{ki}&#xD;&#xA;}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\Pr_{F_\theta}[f(Y) = 0] = \theta_{j+1} \gt 0$ is one of the parameters implicit in the vector $\mathbf{\theta}$, $F$ is some family of distributions parameterized by $\theta_1, \ldots, \theta_j$, and $f$ is the reexpression of the $y$'s (the &quot;link&quot; function of a generalized linear model: see onestop's reply).  (Of course, then, $\Pr_{F_\theta}[f(Y) \le t]$ = $(1 - \theta_{j+1})F_\theta(t)$ when $t \ne 0$.)  Examples are the &lt;a href=&quot;http://rss.acs.unt.edu/Rdoc/library/pscl/html/zeroinfl.html&quot; rel=&quot;nofollow&quot;&gt;zero-inflated Poisson and Negative Binomial models&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(5) &lt;strong&gt;The issues of constructing a model and fitting it are related but different&lt;/strong&gt;.  As a simple example, even an ordinary regression model $Y = \beta_0 + \beta_1 X + \epsilon$ can be fit in many ways by means of least squares (which gives the same parameter estimates as Maximum Likelihood and almost the same standard errors), &lt;a href=&quot;http://en.wikipedia.org/wiki/Iteratively_reweighted_least_squares&quot; rel=&quot;nofollow&quot;&gt;iteratively reweighted least squares&lt;/a&gt;, various other forms of &quot;&lt;a href=&quot;http://en.wikipedia.org/wiki/Robust_regression&quot; rel=&quot;nofollow&quot;&gt;robust least squares&lt;/a&gt;,&quot; etc.  The choice of fitting is often based on convenience, expedience (&lt;em&gt;e.g.&lt;/em&gt;, availability of software), familiarity, habit, or convention, but at least some thought should be given to what is appropriate for the assumed distribution of the error terms $\epsilon_i$, to what the &lt;a href=&quot;http://en.wikipedia.org/wiki/Loss_function&quot; rel=&quot;nofollow&quot;&gt;loss function&lt;/a&gt; for the problem might reasonably be, and to the possibility of exploiting additional information (such as a &lt;a href=&quot;http://en.wikipedia.org/wiki/Prior_probability&quot; rel=&quot;nofollow&quot;&gt;prior distribution&lt;/a&gt; for the parameters).&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-01-31T14:48:08.047" LastActivityDate="2011-01-31T14:48:08.047" CommentCount="3" />
  <row Id="6727" PostTypeId="2" ParentId="6707" CreationDate="2011-01-30T22:16:38.797" Score="3" ViewCount="" Body="&lt;p&gt;The Burns reference that you are quoting seems to dividing the stochastic part into autocorrelation error, which is a byproduct of any time series analysis (and is systematic), vs. truly random error which is uncontrollable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-Ralph Winters &lt;/p&gt;&#xA;" OwnerUserId="3489" LastActivityDate="2011-01-30T22:16:38.797" CommentCount="1" />
  <row Id="6728" PostTypeId="1" CreationDate="2011-01-30T23:22:12.240" Score="7" ViewCount="355" Body="&lt;p&gt;I was trying to fit my data into various models and figured out that the &lt;code&gt;fitdistr&lt;/code&gt; function from library &lt;code&gt;MASS&lt;/code&gt; of &lt;code&gt;R&lt;/code&gt; gives me &lt;code&gt;Negative Binomial&lt;/code&gt; as the best-fit. Now from the &lt;a href=&quot;http://en.wikipedia.org/wiki/Negative_binomial_distribution&quot; rel=&quot;nofollow&quot;&gt;wiki&lt;/a&gt; page, the definition is given as:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;NegBin(r,p) distribution describes the probability of k failures and r&#xA;  successes in k+r Bernoulli(p) trials&#xA;  with success on the last trial.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Using &lt;code&gt;R&lt;/code&gt; to perform model fitting gives me two parameters &lt;code&gt;mean&lt;/code&gt; and &lt;code&gt;dispersion parameter&lt;/code&gt;. I am not understanding how to interpret these because I cannot see these parameters on the wiki page. All I can see is the following formula:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/Tpnyi.png&quot; alt=&quot;Negative Binomial Distribution Formula&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where &lt;code&gt;k&lt;/code&gt; is the number of observations and &lt;code&gt;r=0...n&lt;/code&gt;. Now how do I relate these with the parameters given by &lt;code&gt;R&lt;/code&gt;? The help file does not provide much information either. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, just to say a few words about my experiment: In a social experiment that I was conducting, I was trying to count the number of people each user contacted in a period of 10 days. The population size was 100 for the experiment. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, if the model fits the Negative Binomial, I can blindly say that it follows that distribution but I really want to understand the intuitive meaning behind this. What does it mean to say that the number of people contacted by my test subjects follows a negative binomial distribution? Can someone please help clarify this?&lt;/p&gt;&#xA;" OwnerUserId="2164" LastActivityDate="2011-02-01T04:59:52.097" Title="Understanding the parameters inside the Negative Binomial Distribution" Tags="&lt;r&gt;&lt;distributions&gt;&lt;modeling&gt;&lt;negative-binomial&gt;" AnswerCount="2" FavoriteCount="2" />
  <row Id="6729" PostTypeId="2" ParentId="6728" CreationDate="2011-01-30T23:58:00.913" Score="5" ViewCount="" Body="&lt;p&gt;You should look further down the wikipedia page, to where it says &quot;gamma-Poisson mixture&quot;. While the definition you cite (which I call the &quot;coin-flipping&quot; definition since I usually define it for classes as &quot;suppose you want to flip a coin until you get k heads&quot;) is easier to derive and makes more sense in an introductory probability or mathematical statistics context, the gamma-Poisson mixture is (in my experience) a much more generally useful way to think about the distribution in applied contexts. (In particular, this definition allows non-integer values of the dispersion/size parameter.) In this context, your dispersion parameter describes the distribution of a hypothetical Gamma distribution that underlies your data and describes unobserved variation among individuals in their intrinsic level of contact. In particular, it is the shape parameter of the Gamma, and it may be helpful in thinking about this to know that the coefficient of variation of a Gamma distribution with shape parameter $\theta$ is $1/\sqrt{\theta}$; as $\theta$ becomes large the latent variability disappears and the distribution becomes Poisson.&lt;/p&gt;&#xA;" OwnerUserId="2126" LastActivityDate="2011-01-30T23:58:00.913" />
  <row Id="6730" PostTypeId="2" ParentId="6725" CreationDate="2011-01-31T00:08:48.627" Score="1" ViewCount="" Body="&lt;p&gt;Whuber answer is of course very good and simple, but if You are looking for something faster (binary search approach needs $O(logn)$ steps) then You can look at &lt;strong&gt;Walker's alias method&lt;/strong&gt;, it is described well in Knuths Art of Computer Programming, unfortunatelly I can't tell You the exact page because I have only Polish version of this book. You can also read &lt;a href=&quot;http://www.cs.utk.edu/~vose/Publications/random.pdf&quot; rel=&quot;nofollow&quot;&gt;this article&lt;/a&gt;. These method allows to draw a random number from any finite discrete distribution in constant time.&lt;/p&gt;&#xA;" OwnerUserId="1643" LastActivityDate="2011-01-31T00:08:48.627" CommentCount="1" />
  <row Id="6731" PostTypeId="1" AcceptedAnswerId="6732" CreationDate="2011-01-31T01:11:51.263" Score="6" ViewCount="367" Body="&lt;p&gt;I have a design matrix of p regressors, n observations, and I am trying to compute the sample variance-covariance matrix of the parameters.  I am trying to directly calculate it using svd.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using R, when I take svd of the design matrix, I get three components: a matrix $U$ which is $n \times p$, a matrix $D$ which is $1\times 3$ (presumably eigenvalues), and a matrix $V$ which is $3\times 3$.  I diagonalized $D$, making it a $3\times 3$ matrix with 0's in the off-diagonals.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Supposedly, the formula for covariance is: $V D^2 V&amp;#39;$, however, the matrix does not match, nor is it even close to R's built in function, &lt;code&gt;vcov&lt;/code&gt;.&#xA;Does anyone have any advice/references?  I admit that I am a bit unskilled in this area.&lt;/p&gt;&#xA;" OwnerUserId="2984" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-01-31T07:17:26.280" LastActivityDate="2011-01-31T07:17:26.280" Title="Using Singular Value Decomposition to Compute Variance Covariance Matrix from linear regression model" Tags="&lt;r&gt;&lt;regression&gt;" AnswerCount="1" FavoriteCount="2" />
  <row Id="6732" PostTypeId="2" ParentId="6731" CreationDate="2011-01-31T03:04:22.890" Score="11" ViewCount="" Body="&lt;p&gt;First, recall that under assumptions of multivariate normality of the linear-regression model, we have that&#xA;$$&#xD;&#xA;\hat{\beta} \sim \mathcal{N}( \beta, \sigma^2 (X^T X)^{-1} ) .&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, if $X = U D V^T$ where the right-hand side is the SVD of X, then we get that $X^T X = V D U^T U D V = V D^2 V^T$. Hence,&#xA;$$&#xD;&#xA;(X^T X)^{-1} = V D^{-2} V^T .&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We're still missing the estimate of the variance, which is&#xA;$$&#xD;&#xA;\hat{\sigma}^2 = \frac{1}{n - p} (y^T y - \hat{\beta}^T X^T y) .&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Though I haven't checked, hopefully &lt;strong&gt;vcov&lt;/strong&gt; returns $\hat{\sigma}^2 V D^{-2} V^T$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note:&lt;/strong&gt; You wrote $V D^2 V^T$, which is $X^T X$, but we need the inverse for the variance-covariance matrix. Also note that in $R$, to do this computation you need to do&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;vcov.matrix &amp;lt;- var.est * (v %*% d^(-2) %*% t(v))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;observing that for matrix multiplication we use &lt;code&gt;%*%&lt;/code&gt; instead of just &lt;code&gt;*&lt;/code&gt;. &lt;code&gt;var.est&lt;/code&gt; above is the estimate of the variance of the noise.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Also, I've made the assumptions that $X$ is full-rank and $n \geq p$ throughout. If this is not the case, you'll have to make minor modifications to the above.)&lt;/p&gt;&#xA;" OwnerUserId="2970" LastActivityDate="2011-01-31T03:04:22.890" CommentCount="3" />
  <row Id="6733" PostTypeId="1" AcceptedAnswerId="6735" CreationDate="2011-01-31T03:08:41.140" Score="4" ViewCount="80" Body="&lt;p&gt;Say I have a list with $n$ elements (say number $1$'s) and I want to do 1000 random changes to them, such as a $+1$. If I (1) picked a random element, (2) changed it, and then (3) did these two steps (1)+(2) another 999 times, this would probably be completely random, right? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, what if, whenever I change an element, I take it from the list and reinsert the changed element at the end of the list (i.e. this is the new step (2)) – would the changes also be completely random then?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have the feeling that it might decrease the probability for already changed elements to get changed again, but can't quite put my finger on the cause. Also, consider this variant: instead of inserting them in the end, I insert them in a random new location after the change. How would that influence the probability for picking any element?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(PS. If someone knows good tags for this, please add them and delete this paragraph...)&lt;/p&gt;&#xA;" OwnerUserId="2440" LastActivityDate="2011-01-31T04:16:40.627" Title="Picking random elements from a list – still random if they are reinserted at the end?" Tags="&lt;probability&gt;" AnswerCount="1" />
  <row Id="6734" PostTypeId="1" AcceptedAnswerId="6739" CreationDate="2011-01-31T03:14:14.757" Score="6" ViewCount="378" Body="&lt;p&gt;Hi All,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have been reading the description of ridge regression in Applied Linear Statistical Models 5th Ed chapter 11. The ridge regression is done on a data set available at &lt;a href=&quot;http://www.cst.cmich.edu/users/lee1c/spss/V16_materials/DataSets_v16/BodyFat-TxtFormat.txt&quot; rel=&quot;nofollow&quot;&gt;body fat data&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The textbook matches the output in SAS, where the back transformed coefficients are given in the fitted model as Y=-7.3978+0.5553*X1+0.3681*X2-0.1917*X3.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is shown from SAS as:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;proc reg data = ch7tab1a outest = temp outstb noprint;&#xA;  model y = x1-x3 / ridge = 0.02;&#xA;run;&#xA;quit;&#xA;proc print data = temp;&#xA;  where _ridge_ = 0.02 and y = -1;&#xA;  var y intercept x1 x2 x3;&#xA;run;&#xA;Obs     Y    Intercept       X1         X2         X3&#xA;&#xA; 2     -1     -7.40343    0.55535    0.36814    -0.19163&#xA; 3     -1      0.00000    0.54633    0.37740    -0.13687&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But R gives very different coefficients:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data&amp;lt;-read.table(&quot;http://www.cst.cmich.edu/users/lee1c/spss/V16_materials/DataSets_v16/BodyFat-TxtFormat.txt&quot;,sep=&quot; &quot;,header=FALSE)&#xA;data&amp;lt;-data[,c(1,3,5,7)]&#xA;colnames(data)&amp;lt;-c(&quot;x1&quot;,&quot;x2&quot;,&quot;x3&quot;,&quot;y&quot;)&#xA;ridge&amp;lt;-lm.ridge(y ~ ., data, lambda=0.02)   &#xA;ridge$coef&#xA;coef(ridge)&#xA;&#xA;&amp;gt;   ridge$coef&#xA;       x1        x2        x3 &#xA;10.126984 -4.682273 -3.527010 &#xA;&amp;gt;   coef(ridge)&#xA;                   x1         x2         x3 &#xA;42.2181995  2.0683914 -0.9177207 -0.9921824 &#xA;&amp;gt; &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Anyone help me with understanding why?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks! Brian&lt;/p&gt;&#xA;" OwnerUserId="2040" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-01-31T10:42:18.220" LastActivityDate="2011-01-31T10:42:18.220" Title="Difference between ridge regression implementation in R and SAS" Tags="&lt;r&gt;&lt;sas&gt;&lt;ridge-regression&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="6735" PostTypeId="2" ParentId="6733" CreationDate="2011-01-31T03:24:39.737" Score="6" ViewCount="" Body="&lt;p&gt;It's necessary to be a little bit careful about what you mean by &quot;random&quot; and &quot;completely random&quot; here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you are describing, at least in your first scheme is a set of random draws such that the resulting vector is multinomially distributed with n = 1000 and probabilities $p_1 = p_2 = \cdots = p_n = 1/n$. Even in your first scheme, the resulting variables are &lt;strong&gt;not&lt;/strong&gt; independent. Marginally, they do all have the same distribution and, intuitively (though I didn't check carefully), it seems that the vector of counts would have a property called &lt;em&gt;exchangeability&lt;/em&gt; which, in some sense, is &quot;&lt;em&gt;almost&lt;/em&gt; independent and identically distributed&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the second scheme, where you move each new selection to the end of the list after incrementing it, the reordered variables are not even exchangeable any more, nor do they have the same marginal distribution. Intuitively, this is because the ones being moved to the end of the list will tend to have larger values than ones at the beginning of the list. This is related to, but not quite, what are called &lt;em&gt;order statistics&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would have to think about the last scheme (i.e., reinserting in a random location) more carefully. On the surface, I believe you would essentially end up with the same distribution as in the first case. But, I've not checked that even remotely formally, so take that with a hefty dose of salt. &lt;/p&gt;&#xA;" OwnerUserId="2970" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-01-31T04:16:40.627" LastActivityDate="2011-01-31T04:16:40.627" CommentCount="3" />
  <row Id="6736" PostTypeId="1" CreationDate="2011-01-31T05:39:54.900" Score="3" ViewCount="110" Body="&lt;p&gt;What properties do these measures have and how can I determine which one is better for a given purpose? What are extreme cases where they differ a lot?&lt;/p&gt;&#xA;" OwnerUserId="2440" LastActivityDate="2011-01-31T10:33:32.287" Title="Properties of Battacharyya distance vs Kullback-Leibler divergence" Tags="&lt;distributions&gt;&lt;probability&gt;&lt;kullback-leibler&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="6737" PostTypeId="2" ParentId="2356" CreationDate="2011-01-31T06:44:48.373" Score="1" ViewCount="" Body="&lt;p&gt;This is a &quot;fleshed out&quot; example given in a book written by Larry Wasserman &lt;a href=&quot;http://research.rmutp.ac.th/research/A%20Concise%20Course%20in%20Statistical%20Inference.pdf&quot; rel=&quot;nofollow&quot;&gt;All of statistics&lt;/a&gt; on Page 216 (&lt;em&gt;12.8 Strengths and Weaknesses of Bayesian&#xA;Inference&lt;/em&gt;).  I basically provide what Wasserman doesn't in his book 1) an explanation for what is actually happening, rather than a throw away line; 2) the frequentist answer to the question, which Wasserman conveniently does not give; and 3) a demonstration that the equivalent confidence calculated &lt;em&gt;using the same information&lt;/em&gt; suffers from the same problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this example, he states the following situation&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;An observation, X, with a Sampling distribution: $(X|\theta)\sim N(\theta,1)$&lt;/li&gt;&#xA;&lt;li&gt;Prior distribution of $(\theta)\sim N(0,1)$ (he actually uses a general $\tau^2$ for the variance, but his diagram specialises to $\tau^2=1$)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;He then goes to show that, using a Bayesian 95% credible interval in this set-up eventually has 0% frequentist coverage when the true value of $\theta$ becomes arbitrarily large.  For instance, he provides a graph of the coverage (p218), and checking by eye, when the true value of $\theta$ is 3, the coverage is about 35%.  He then goes on to say:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;...What should we conclude from all this? The important thing&#xA;is to understand that frequentist and Bayesian methods are answering&#xA;dierent questions. To combine prior beliefs with data&#xA;in a principled way, use bayesian inference. To construct procedures&#xA;with guaranteed long run performance, such as condence&#xA;intervals, use frequentist methods...&lt;/em&gt; (p217)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And then moves on without any disection or explanation of &lt;em&gt;why&lt;/em&gt; the Bayesian method performed apparently so bad.  Further, he does not give a answer from the frequentist approach, just a broad brush statement about &quot;the long-run&quot; - a classical political tactic (emphasise your strength + others weakness, but never compare like for like).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I will show how the problem &lt;em&gt;as stated&lt;/em&gt; $\tau=1$ can be formulated in frequentist/orthodox terms, and then show that the result using confidence intervals gives &lt;em&gt;precisely the same answer as the Bayesian one&lt;/em&gt;.  Thus any defect in the Bayesian (real or perceived) is not corrected by using confidence intervals.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Okay, so here goes.  The first question I ask is what state of knowledge is described by the prior $\theta\sim N(0,1)$?  If one was &quot;ignorant&quot; about $\theta$, then the appropriate way to express this is $p(\theta)\propto 1$.  Now suppose that we were ignorant, and we observed $Y\sim N(\theta,1)$, independently of $X$.  What would our posterior for $\theta$ be?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p(\theta|Y)\propto p(\theta)p(Y|\theta)\propto exp\Big(-\frac{1}{2}(Y-\theta)^2\Big)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus $(\theta|Y)\sim N(Y,1)$.  This means that the prior distribution given in Wassermans example, is equivalent to having observed an iid copy of $X$ equal to $0$.  Frequentist methods cannot deal with a prior, but it can be thought of as having made 2 observations from the sampling distribution, one equal to $0$, and one equal to $X$.  Both problems are entirely equivalent, and we can actually give the frequentist answer for the question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because we are dealing with a normal distribution with known variance, the mean is a sufficient statistic for constructing a confidence interval for $\theta$.  The mean is equal to $\overline{x}=\frac{0+X}{2}=\frac{X}{2}$ and has a sampling distribution&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$(\overline{x}|\theta)\sim N(\theta,\frac{1}{2})$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus an $(1-\alpha)\text{%}$ CI is given by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{1}{2}X\pm Z_{\alpha/2}\frac{1}{\sqrt{2}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But, using The results of example 12.8 for Wasserman, he shows that the posterior $(1-\alpha)\text{%}$ credible interval for $\theta$ is given by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$cX\pm \sqrt{c}Z_{\alpha/2}$$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $c=\frac{\tau^{2}}{1+\tau^{2}}$.  Thus, plugging in the value at $\tau^{2}=1$ gives $c=\frac{1}{2}$ and the credible interval becomes:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{1}{2}X\pm Z_{\alpha/2}\frac{1}{\sqrt{2}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which are exactly the same as the confidence interval!  So any defect in the coverage exhibited by the Bayesian method, is not corrected by using the frequentist confidence interval! [If the frequentist chooses to ignore the prior, then to be a fair comparison, the Bayesian should also ignore this prior, and use the ignorance prior $p(\theta)\propto 1$, and the two intervals will still be equal - both $X \pm Z_{\alpha/2})$].&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So what the hell is going on here?&#xA;The problem is basically one of non-robustness of the normal sampling distribution.  because the problem is equivalent to having already observed a iid copy, $X=0$.  If you have observed $0$, then this is &lt;em&gt;extremely unlikely&lt;/em&gt; to have occurred if the true value is $\theta=4$ (probability that $X\leq 0$ when $\theta=4$ is 0.000032).  This explains why the coverage is so bad for large &quot;true values&quot;, because they effectively make the implicit observation contained in the prior an &lt;em&gt;outlier&lt;/em&gt;.  In fact you can show that this example is basically equivalent to showing that the arithmetic mean has an unbounded influence function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Generalisation.&#xA;Now some people may say &quot;but you only considered $\tau=1$, which may be a special case&quot;.  This is not true: any value of $\tau^2=\frac{1}{N}$ $(N=0,1,2,3,\dots)$ can be interpreted as observing $N$ iid copies of $X$ which were all equal to $0$, in addition to the $X$ of the question.  The confidence interval will have the same &quot;bad&quot; coverage properties for large $\theta$.  But this becomes increasingly unlikely if you keep observing values of $0$ (and no rational person would continue to worry about large $\theta$ when you keep seeing $0$).&lt;/p&gt;&#xA;" OwnerUserId="2392" LastEditorUserId="2392" LastEditorDisplayName="" LastEditDate="2011-01-31T07:09:29.793" LastActivityDate="2011-01-31T07:09:29.793" CommentCount="3" />
  <row Id="6738" PostTypeId="2" ParentId="6736" CreationDate="2011-01-31T07:43:50.690" Score="1" ViewCount="" Body="&lt;p&gt;First properties are explained competently &lt;a href=&quot;http://en.wikipedia.org/wiki/Kullback-Leibler_divergence&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;  and&#xA;&lt;a href=&quot;http://en.wikipedia.org/wiki/Bhattacharyya_distance&quot; rel=&quot;nofollow&quot;&gt;there&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which one is better suited to a given purpose will depend on said given purpose so you might think about rephrasing this part of your question.&lt;/p&gt;&#xA;" OwnerUserId="2592" LastActivityDate="2011-01-31T07:43:50.690" CommentCount="8" />
  <row Id="6739" PostTypeId="2" ParentId="6734" CreationDate="2011-01-31T08:18:17.773" Score="4" ViewCount="" Body="&lt;p&gt;Though ridge regression looks &lt;a href=&quot;http://en.wikipedia.org/wiki/Ridge_regression&quot; rel=&quot;nofollow&quot;&gt;at first like simple algorithm&lt;/a&gt; the devil is in the details. Apparently original variables are scaled, and parameter $\lambda$ is not the parameter you would think it is given the original description. From what I gathered reading &lt;a href=&quot;http://books.google.com/books?id=zWpegyjze34C&amp;amp;lpg=PP1&amp;amp;dq=Measurement%2C%20Regression%20and%20Calibration&amp;amp;hl=fr&amp;amp;pg=PA51#v=onepage&amp;amp;q&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;the reference&lt;/a&gt; given in R help page of &lt;code&gt;lm.ridge&lt;/code&gt; there is no one agreed way of doing ridge regression. So the difference in results can only be explained by different algorithms used by R and SAS. Hopefully someone more knowledgeable can give more detailed answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can see what kind of algorithm is applied in R by looking at the source of &lt;code&gt;lm.ridge&lt;/code&gt;. Just type &lt;code&gt;lm.ridge&lt;/code&gt; in the R prompt.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-01-31T08:18:17.773" CommentCount="4" />
  <row Id="6740" PostTypeId="1" AcceptedAnswerId="6742" CreationDate="2011-01-31T08:43:05.747" Score="5" ViewCount="135" Body="&lt;p&gt;I am reading the second edition of Categorical Data Analysis by Alan Agresti, and somehow stuck in the following second paragraph:&#xA;&lt;img src=&quot;http://i.imgur.com/ahQDT.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't quite understand why $\beta\pi(\hat{x})(1 - \pi(\hat{x}))$ will give the probability when $x = 26.3$, can anyone enlighten me? Thanks.&lt;/p&gt;&#xA;" OwnerUserId="588" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-01-31T14:50:46.843" LastActivityDate="2011-01-31T14:50:46.843" Title="Alternative ways for interpretation of odds" Tags="&lt;logistic-regression&gt;" AnswerCount="1" />
  <row Id="6742" PostTypeId="2" ParentId="6740" CreationDate="2011-01-31T11:19:42.163" Score="5" ViewCount="" Body="&lt;p&gt;The answer is near the bottom of &lt;a href=&quot;http://books.google.co.uk/books?id=hpEzw4T0sPUC&amp;amp;pg=PA166&quot; rel=&quot;nofollow&quot;&gt;p166&lt;/a&gt;. It's using a linear approximation (what social scientists would call a 'marginal effect'). A small change $\delta x$ in $x$ gives a change in probability of:&#xA;$$\delta\pi \approx \frac{\partial \pi(x)}{\partial x} \delta x.$$&#xA;With $\operatorname{logit}(\pi(x)) = \alpha + \beta x$, it's straightforward to show that&#xA;$ \partial \pi(x) / \partial x = \beta \pi(x)(1-\pi(x))$.&lt;/p&gt;&#xA;" OwnerUserId="449" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-01-31T11:27:32.697" LastActivityDate="2011-01-31T11:27:32.697" />
  <row Id="6743" PostTypeId="2" ParentId="5250" CreationDate="2011-01-31T12:19:34.623" Score="1" ViewCount="" Body="&lt;p&gt;Have a look at InvivoStat (&lt;a href=&quot;http://invivostat.co.uk&quot; rel=&quot;nofollow&quot;&gt;invivostat.co.uk&lt;/a&gt;), it should do the type of analysis that you are looking for&lt;/p&gt;&#xA;" OwnerUserId="2988" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-01-31T14:24:18.900" LastActivityDate="2011-01-31T14:24:18.900" CommentCount="1" />
  <row Id="6744" PostTypeId="1" AcceptedAnswerId="6750" CreationDate="2011-01-31T13:12:42.683" Score="2" ViewCount="103" Body="&lt;p&gt;Hi,&lt;br&gt;&#xA;I have no math or stats background, so this may be a real dumbo question.&lt;br&gt;&#xA;I'm developing a time estimation app which would let user enter her worst case and best case estimates, and then select a probability curve which she feels looks realistic.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I needed to draw some kind of curve that sums up to 1, so I came up with a normal distribution. I'm not sure that it captures how we give time estimates but this is not what question is about.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want user to be able to move finger on the chart, so the curve would bend accordingly. I assume it would feel nice if curve's peak followed the finger, but then I have the following question: is it possible to get normal distribution curve by arbitrary peak point? If not, which curve is good for this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE:&lt;/strong&gt;&lt;br&gt;&#xA;I see that I can obtain &lt;em&gt;μ&lt;/em&gt; from &lt;em&gt;x&lt;/em&gt; location.&lt;br&gt;&#xA;Can I get &lt;em&gt;σ&lt;/em&gt; for an arbitrary &lt;em&gt;y&lt;/em&gt; though (combined with known &lt;em&gt;μ&lt;/em&gt;)?&lt;/p&gt;&#xA;" OwnerUserId="2989" LastEditorUserId="2989" LastEditorDisplayName="" LastEditDate="2011-01-31T13:25:01.897" LastActivityDate="2011-01-31T15:35:37.323" Title="Get distribution curve characteristics by selecting peak point?" Tags="&lt;distributions&gt;&lt;estimation&gt;&lt;normal-distribution&gt;" AnswerCount="3" />
  <row Id="6745" PostTypeId="2" ParentId="6744" CreationDate="2011-01-31T13:20:40.813" Score="0" ViewCount="" Body="&lt;p&gt;See the &lt;a href=&quot;http://en.wikipedia.org/wiki/Normal_distribution&quot; rel=&quot;nofollow&quot;&gt;article on the normal distribution in Wikipedia&lt;/a&gt;: the parameter &lt;em&gt;μ&lt;/em&gt; is the mean and also the location of the peak.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-01-31T13:20:40.813" CommentCount="1" />
  <row Id="6746" PostTypeId="1" CreationDate="2011-01-31T14:03:05.300" Score="5" ViewCount="112" Body="&lt;p&gt;I'm working with a two-state process with $x_t$ in $\{1, -1\}$ for $t = 1, 2, \ldots$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The autocorrelation function is indicative of a process with long-memory, i.e. it displays a power law decay with an exponent &amp;lt; 1. You can simulate a similar series in R with:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; library(fArma)&#xA;&amp;gt; x&amp;lt;-fgnSim(10000,H=0.8)&#xA;&amp;gt; x&amp;lt;-sign(x)&#xA;&amp;gt; acf(x)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My question: is there a canonical way to optimally predict the next value in the series given just the autocorrelation function? One way to predict is simply to use&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\hat{x}(t) = x(t-1)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;which has a classification rate of $(1 + \rho_1) / 2$, where $\rho$ is the lag-1 autocorrelation, but I feel like it must be possible to do better by taking into account the long-memory structure.&lt;/p&gt;&#xA;" OwnerUserId="2425" LastActivityDate="2011-01-31T14:03:05.300" Title="Predicting long-memory processes" Tags="&lt;time-series&gt;&lt;predictive-models&gt;&lt;autocorrelation&gt;" CommentCount="4" FavoriteCount="2" />
  <row Id="6747" PostTypeId="1" AcceptedAnswerId="6749" CreationDate="2011-01-31T14:10:18.723" Score="2" ViewCount="61" Body="&lt;p&gt;I want to measure the correlation between two 1D point processes $x$ and $y$. Ordinarily I could use the bivariate K-function&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$K(t) = \frac{T}{n_xn_y} \sum_{i=1}^{n_x} \sum_{j=1}^{n_y} w(x_i,y_j) I[d(x_i,y_j)&amp;lt;t]$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $n_x$ is the number of observations in $x$ and $n_y$ is the number of observations in $y$. Deviation from $K(t)=t$ is an indication of correlation between the two point processes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, my analysis is complicated by the fact that the unconditional distributions of the point processes are non-uniform. They each have a characteristic distribution of intervals between observations, which could be different for $x$ and $y$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To make things more complicated, the intensities vary over the time period (for example, the intensity might be low around midnight and higher during daylight hours).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming I have enough data to get good estimates of the unconditional distributions of $x$ and $y$, and of the way that intensities vary across time, is there a way to take this into account?&lt;/p&gt;&#xA;" OwnerUserId="2425" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-01-31T15:14:46.450" LastActivityDate="2011-02-01T12:09:07.593" Title="Correlation of two 1d point processes with non-uniform unconditional distribution" Tags="&lt;correlation&gt;&lt;cross-correlation&gt;&lt;point-process&gt;" AnswerCount="2" />
  <row Id="6748" PostTypeId="2" ParentId="6705" CreationDate="2011-01-31T14:37:55.117" Score="2" ViewCount="" Body="&lt;p&gt;Although not within the domain of geostatistics, for question #2, I would casually say the most frequent weighting function used in my field (Criminology) would be a a binary weighting scheme. Although I have rarely seen a good theoretical or empirical argument to use one weighting scheme over another (or how one defines a neighbor in a binary weighting scheme either). It may simply be because of historical preference and convienance that such a scheme is typically used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a distinction that should be drawn between data driven approaches to constructing spatial weights and the theory based approach to deriving spatial weights. You are currently performing the former, and in this approach you are implicitly treating the estimation of spatial weights as a problem of measurement error, and hence should use techniques to validate your measurements (which is considerably complicated due to the endogeneity of the spatial weights). Using a weighting scheme based on some of the chance variation in the data and using it in subsequent causal models is synonymous with other fallacies related to inference and data snooping. Unfortunately I have no good references of spatial weight models validated in any meaningful way besides the extent of the auto-correlation, which to be frank isn't all that convincing of an empirical argument. Spatial dependence can be the result of either causal processes (i.e. the value at one point in space affects the value at another point in space), or it can be the result of other measurement errors (i.e. the measured support of the data do not match the support of the processes that generate those phenomena). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is oppossed to theory based construction of spatial weights (or &quot;model-driven&quot; in Luc Anselin's terminology), in which one specifies the weight matrix &lt;em&gt;a priori&lt;/em&gt; to estimating a model. I did not read the Fauchald paper you cited, but it appears in the abstract they have plausible theoretical explanations for the observed patterns based on some optimal foraging strategy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For readings I would suggest Luc Anselin's book, &lt;a href=&quot;http://books.google.com/books?id=3dPIXClv4YYC&amp;amp;dq=anselin&amp;amp;source=gbs_navlinks_s&quot; rel=&quot;nofollow&quot;&gt;Spatial Econometrics: Methods and Models&lt;/a&gt; (1988), particularly chapters 2 and 3 will be of most interest. Also as another work with a similar viewpoint to mine (although it will likely be of less interest) is an essay piece by Gary King, &quot;&lt;a href=&quot;http://dx.doi.org/10.1016/0962-6298%2895%2900079-8&quot; rel=&quot;nofollow&quot;&gt;Why context should not count&lt;/a&gt;&quot;. I would also suggest another paper as it appears they had similar goals to yours, and defined the weights for a lattice system based on variogram estimates (&lt;a href=&quot;http://dx.doi.org/10.1080/17538940903253898&quot; rel=&quot;nofollow&quot;&gt;Negreiros, 2010&lt;/a&gt;). &lt;/p&gt;&#xA;" OwnerUserId="1036" LastEditorUserId="1036" LastEditorDisplayName="" LastEditDate="2011-01-31T14:43:16.333" LastActivityDate="2011-01-31T14:43:16.333" CommentCount="1" />
  <row Id="6749" PostTypeId="2" ParentId="6747" CreationDate="2011-01-31T15:14:21.927" Score="2" ViewCount="" Body="&lt;p&gt;Yes.  The key is that how you &lt;em&gt;measure&lt;/em&gt; correlation does not change, but the expected value does depend on the hypothesized underlying process.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$K$ (and cross-$K$, which is what you are looking at) is defined generally for inhomogeneous point processes.  For a good discussion with examples see &lt;a href=&quot;http://www.stat.iastate.edu/preprint/articles/2001-18.pdf&quot; rel=&quot;nofollow&quot;&gt;Philip Dixon's analysis&lt;/a&gt;.  Because all his examples are 2D, the specific formulas for expectations do not apply directly to your 1D data, but all the general ideas do apply.  Note that with non-uniform distributions the definition (and computation) of $K$ do not change, but the expectation does change.  With non-stationary distributions you also have to take care to distinguish the two forms of cross-K between your two series of data.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-01-31T15:14:21.927" CommentCount="2" />
  <row Id="6750" PostTypeId="2" ParentId="6744" CreationDate="2011-01-31T15:34:01.507" Score="3" ViewCount="" Body="&lt;p&gt;Because normal distributions are symmetric, and it's natural to think of &quot;best&quot; and &quot;worst&quot; cases symmetrically (&lt;em&gt;e.g.&lt;/em&gt;, the chance of the best case is no greater than 5% and the chance of the worst case is no less than the same percent), they are not likely to work well in this application.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In effect you have three degrees of freedom to specify a distribution.  Pick any family that requires three parameters and is suited to distributions of &quot;time&quot;.  If you are estimating time to complete a task, people often have used &lt;a href=&quot;http://en.wikipedia.org/wiki/Beta_distribution&quot; rel=&quot;nofollow&quot;&gt;scaled Beta distributions&lt;/a&gt; for this purpose.  Another good choice is the &lt;a href=&quot;http://en.wikipedia.org/wiki/Weibull_distribution&quot; rel=&quot;nofollow&quot;&gt;Weibull distribution&lt;/a&gt;, which often arises in such applications, and possibly the &lt;a href=&quot;http://en.wikipedia.org/wiki/Gamma_distribution&quot; rel=&quot;nofollow&quot;&gt;Gamma distribution&lt;/a&gt;.  The latter two require only two parameters, but many people have introduced a third location parameter that shifts the distribution (usually to the right), so-called &quot;three-parameter Weibull&quot; and &quot;three-parameter Gamma&quot;.  You can also embed both these distributions into a larger three-parameter family that includes the normal, called a &quot;&lt;a href=&quot;http://www.weibull.com/LifeDataWeb/generalized_gamma_distribution.htm&quot; rel=&quot;nofollow&quot;&gt;generalized Gamma&lt;/a&gt;&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Typically some latitude is allowed for the time to exceed the &quot;worst&quot; case, so you would interpret the worst case as, say, somewhere between the 80th and 99th percentile of the distribution rather than it maximum (and similarly, &lt;em&gt;mutatis mutandis&lt;/em&gt;, for the best case).  Thus, whatever family you pick, you need to be able to find its usual parameters (such as $\alpha$, $\beta$, and the scale factor for the scaled Beta distribution), &lt;em&gt;given&lt;/em&gt; the values of a lower percentile (the best case), an upper percentile (the worst case), and the mode.  This can always be done (provided the percentiles and modes can be modeled by the chosen distributional family at all), but the amount of coding work--and the details of the mathematics--will vary with the distributional family.  Because this can challenge one's numerical programming skills, many applications use approximations, often gross ones (given the qualitative nature of the elicited information).  A common one occurs in &lt;a href=&quot;http://en.wikipedia.org/wiki/Beta_distribution#Task_duration_modeling&quot; rel=&quot;nofollow&quot;&gt;task duration modeling&lt;/a&gt;, where scaled Beta distributions are estimated by means of formulas that are not exactly right but are extremely simple (they are linear combinations of the three values).&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-01-31T15:34:01.507" CommentCount="1" />
  <row Id="6751" PostTypeId="2" ParentId="6744" CreationDate="2011-01-31T15:35:37.323" Score="1" ViewCount="" Body="&lt;p&gt;The normal curve is not going to work for you:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;it is not constrained by a minimum/maximum&lt;/li&gt;&#xA;&lt;li&gt;it is symmetric around the peak&lt;/li&gt;&#xA;&lt;li&gt;it has a very restrictive shape&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;There are loads of other statistical distributions that would probably fit your needs better. I would especially look at distributions confined to the 0-1 interval (and then rescaled to your min-max interval) such as the &lt;a href=&quot;http://en.wikipedia.org/wiki/Beta_distribution&quot; rel=&quot;nofollow&quot;&gt;Beta distribution&lt;/a&gt;. The &quot;peak&quot; of the distribution is called &quot;mode&quot; in stat-speak. The formula for the mode is quite straightforward for the Beta. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note, however, that this distribution has two parameters, and you probably want to allow some way of changing both.&lt;/p&gt;&#xA;" OwnerUserId="279" LastActivityDate="2011-01-31T15:35:37.323" CommentCount="2" />
  <row Id="6752" PostTypeId="2" ParentId="6723" CreationDate="2011-01-31T15:40:37.687" Score="3" ViewCount="" Body="&lt;p&gt;For the second part, you must interpret the output as the steps towards your final model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, in the forward case you begin with&#xA;    Start:  AIC=377.95&#xA;    cars$MidrangePrice ~ 1&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;              Df Sum of Sq    RSS    AIC&#xA;+ cars$Horsepower  1    4979.3 3054.9 300.66&#xA;+ cars$Wheelbase   1    3172.3 4862.0 338.76&#xA;+ cars$Length      1    2448.8 5585.4 350.14&#xA;+ cars$Width       1    1969.2 6065.0 356.89&#xA;+ cars$Uturn       1    1450.2 6584.0 363.63&#xA;+ cars$Luggage     1    1079.6 6954.7 368.12&#xA;&amp;lt;none&amp;gt;                         8034.2 377.95&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Your current model is only considering the constant &lt;code&gt;cars$MidrangePrice ~ 1&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each row in the table indicates that in case you add that variable (for example, Horsepower), you will get the following results rearding &lt;code&gt;Sq&lt;/code&gt; &lt;code&gt;RSS&lt;/code&gt;(Residual Sum of Squares) and &lt;code&gt;AIC&lt;/code&gt; (Akaike Information Criterion).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the other cases you must read the results the same way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps :)&lt;/p&gt;&#xA;" OwnerUserId="2902" LastActivityDate="2011-01-31T15:40:37.687" />
  <row Id="6753" PostTypeId="1" CreationDate="2011-01-31T16:41:29.663" Score="6" ViewCount="208" Body="&lt;p&gt;... instead of e.g. the popular Equal-Width-Histograms. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Additional question: What is a good/robust rule of the thumb to calculate the number of bins for equal frequency histograms (like the &lt;a href=&quot;http://stats.stackexchange.com/questions/798/calculating-optimal-number-of-bins-in-a-histogram-for-n-where-n-ranges-from-30-3/862#862&quot;&gt;Freedmann-Diaconis-Rule&lt;/a&gt; for equal-width).&lt;/p&gt;&#xA;" OwnerUserId="264" LastActivityDate="2011-05-14T22:46:17.517" Title="When to use Equal-Frequency-Histograms" Tags="&lt;histogram&gt;&lt;rule-of-thumb&gt;" AnswerCount="1" CommentCount="9" FavoriteCount="1" />
  <row Id="6754" PostTypeId="1" CreationDate="2011-01-31T16:46:08.220" Score="2" ViewCount="79" Body="&lt;p&gt;Hi everyone,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am a beginner in statistics, and I want to learn machine learning :). Therefore, I have gathered some sample data to practice. But, the problem is I want to create a feature (or attribute), which is common for every entity. But, this feature is observed and measured for entity X in Y meters and Z seconds, and for entity W in M meters and L seconds, and there are many observations like described before. Which  algorithm should I apply to create a common feature for all entities? BTW, I am using WEKA.&lt;/p&gt;&#xA;" OwnerUserId="2170" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-01-31T18:21:57.650" LastActivityDate="2011-01-31T18:21:57.650" Title="What should I do to compare different sets of data?" Tags="&lt;multiple-comparisons&gt;&lt;feature-selection&gt;&lt;normalization&gt;&lt;weka&gt;" CommentCount="4" />
  <row Id="6755" PostTypeId="1" AcceptedAnswerId="6757" CreationDate="2011-01-31T17:46:43.547" Score="4" ViewCount="172" Body="&lt;p&gt;The data files from &lt;a href=&quot;http://www.csie.ntu.edu.tw/~cjlin/libsvm/&quot; rel=&quot;nofollow&quot;&gt;http://www.csie.ntu.edu.tw/~cjlin/libsvm/&lt;/a&gt; are in 'svm' format.  I am trying to read this in to sparse matrix representation in R.  Is there an easy/efficient way to do this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is what I am doing now: read in file line by line (800,000 lines), for each line separate classes, values, and cols.  Store the classes as a list and the features as a .csr sparse matrix (1 row), then rbind the feature row with all previous rows.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is terribly inefficient and basically won't finish (12 minutes for 1000 lines). I think it comes from rbinding the sparse matrices once the number of rows starts to get large.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: the matrix (800000*48000) is too big to build and then convert to sparse format.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="2310" LastEditorUserId="2310" LastEditorDisplayName="" LastEditDate="2011-01-31T18:40:30.193" LastActivityDate="2011-02-01T18:58:10.837" Title="Reading in SVM files in R (libsvm)" Tags="&lt;svm&gt;&lt;matrix&gt;&lt;dataset&gt;" AnswerCount="2" />
  <row Id="6757" PostTypeId="2" ParentId="6755" CreationDate="2011-01-31T19:01:59.317" Score="3" ViewCount="" Body="&lt;p&gt;The &lt;code&gt;e1071&lt;/code&gt; package has a means for exporting to the libsvm &quot;svm&quot; format in &lt;a href=&quot;http://ugrad.stat.ubc.ca/R/library/e1071/html/write.svm.html&quot; rel=&quot;nofollow&quot;&gt;the &lt;code&gt;write.svm&lt;/code&gt; function&lt;/a&gt;.  But to the best of my knowledge, there is no &lt;code&gt;read.svm&lt;/code&gt; function.&lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2011-01-31T19:01:59.317" CommentCount="1" />
  <row Id="6758" PostTypeId="1" AcceptedAnswerId="6775" CreationDate="2011-01-31T19:58:00.220" Score="2" ViewCount="185" Body="&lt;p&gt;I've fit a model with several independent variables, one of which is the lag of the dependent variable, using the dynlm package.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming I have 1-step-ahead forecasts for my independent variables, how do I get 1-step-ahead forecasts for my dependent variables?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(dynlm)&#xA;&#xA;y&amp;lt;-arima.sim(model=list(ar=c(.9)),n=10) #Create AR(1) dependant variable&#xA;A&amp;lt;-rnorm(10) #Create independant variables&#xA;B&amp;lt;-rnorm(10)&#xA;C&amp;lt;-rnorm(10)&#xA;y&amp;lt;-y+.5*A+.2*B-.3*C #Add relationship to independant variables &#xA;data=cbind(y,A,B,C)&#xA;&#xA;#Fit linear model&#xA;model&amp;lt;-dynlm(y~A+B+C+L(y,1),data=data)&#xA;&#xA;#Forecast&#xA;A&amp;lt;-c(A,rnorm(1)) #Assume we already have 1-step forecasts for A,B,C&#xA;B&amp;lt;-c(B,rnorm(1))&#xA;C&amp;lt;-c(C,rnorm(1))&#xA;y=window(y,end=end(y)+c(1,0),extend=TRUE)&#xA;newdata&amp;lt;-cbind(y,A,B,C)&#xA;predict(model,newdata)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And here is an example using the dyn package, which works.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(dyn)&#xA;&#xA;#Fit linear model&#xA;model&amp;lt;-dyn$lm(y~A+B+C+lag(y,-1),data=data)&#xA;&#xA;#Forecast&#xA;predict(model,newdata)the dyn packages, which works:&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2817" LastEditorUserId="2817" LastEditorDisplayName="" LastEditDate="2011-01-31T23:33:00.067" LastActivityDate="2011-02-01T08:03:30.793" Title="1-step-ahead predictions with dynlm R package" Tags="&lt;r&gt;&lt;dynamic-regression&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="2" />
  <row Id="6759" PostTypeId="1" AcceptedAnswerId="6800" CreationDate="2011-01-31T19:58:19.363" Score="3" ViewCount="237" Body="&lt;p&gt;Hi I have a table like &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Coulmn1 Column 2&#xA;&#xA;A    1&#xA;&#xA;A   1&#xA;&#xA;A   2&#xA;&#xA;B   4&#xA;&#xA;B   1&#xA;&#xA;B   1&#xA;&#xA;C   2&#xA;&#xA;C   2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to remove the duplicates based on both the columns so that my final output will be&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;A   1&#xA;&#xA;A   2&#xA;&#xA;B   4&#xA;&#xA;B   1&#xA;&#xA;C   2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;order is really not a matter for me&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank in advance&lt;/p&gt;&#xA;" OwnerUserId="2725" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-02-01T08:42:45.233" LastActivityDate="2011-02-02T09:27:56.653" Title="Removing duplicated rows from R data frame" Tags="&lt;r&gt;" AnswerCount="2" FavoriteCount="0" />
  <row Id="6760" PostTypeId="2" ParentId="6759" CreationDate="2011-01-31T20:22:27.047" Score="7" ViewCount="" Body="&lt;p&gt;You are looking for &lt;code&gt;unique()&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;a &amp;lt;- c(rep(&quot;A&quot;, 3), rep(&quot;B&quot;, 3), rep(&quot;C&quot;,2))&#xA;b &amp;lt;- c(1,1,2,4,1,1,2,2)&#xA;df &amp;lt;-data.frame(a,b)&#xA;unique(df)&#xA;&#xA;&amp;gt; unique(df)&#xA;  a b&#xA;1 A 1&#xA;3 A 2&#xA;4 B 4&#xA;5 B 1&#xA;7 C 2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="307" LastActivityDate="2011-01-31T20:22:27.047" CommentCount="1" />
  <row Id="6761" PostTypeId="2" ParentId="6544" CreationDate="2011-01-31T21:04:56.823" Score="2" ViewCount="" Body="&lt;p&gt;It's clear the solution I posted previously is inadequate and inelegant.  Here is my second attempt, which 100% solves my problem.  Please let me know if you spot any bugs!  I will cross post to stack overflow, if you all think that would be a better place to get comments on my code.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#A function to iteratively predict a time series&#xA;ipredict &amp;lt;-function(model, newdata, interval = &quot;none&quot;,&#xA;        level = 0.95, na.action = na.pass, weights = 1) {&#xA;    P&amp;lt;-predict(model,newdata=newdata,interval=interval,&#xA;        level=level,na.action=na.action,weights=weights)&#xA;    for (i in seq(1,dim(newdata)[1])) {&#xA;        if (is.na(newdata[i])) {&#xA;            if (interval==&quot;none&quot;) {&#xA;                P[i]&amp;lt;-predict(model,newdata=newdata,interval=interval,&#xA;                    level=level,na.action=na.action,weights=weights)[i]&#xA;                newdata[i]&amp;lt;-P[i]&#xA;            }&#xA;            else{&#xA;                P[i,]&amp;lt;-predict(model,newdata=newdata,interval=interval,&#xA;                    level=level,na.action=na.action,weights=weights)[i,]&#xA;                newdata[i]&amp;lt;-P[i,1]&#xA;            }&#xA;        }&#xA;    }&#xA;    P_end&amp;lt;-end(P)[1]*frequency(P)+(end(P)[2]-1) #Convert (time,period) to decimal time&#xA;    P&amp;lt;-window(P,end=P_end-1*frequency(P)) #Drop last observation, which is NA&#xA;    return(P)&#xA;}&#xA;&#xA;&#xA;#Example usage:&#xA;library(dyn)&#xA;y&amp;lt;-arima.sim(model=list(ar=c(.9)),n=10) #Create AR(1) dependant variable&#xA;A&amp;lt;-rnorm(10) #Create independant variables&#xA;B&amp;lt;-rnorm(10)&#xA;C&amp;lt;-rnorm(10)&#xA;Error&amp;lt;-rnorm(10)&#xA;y&amp;lt;-y+.5*A+.2*B-.3*C+.1*Error #Add relationship to independant variables &#xA;data=cbind(y,A,B,C)&#xA;&#xA;#Fit linear model&#xA;model.dyn&amp;lt;-dyn$lm(y~A+B+C+lag(y,-1),data=data)&#xA;summary(model.dyn)&#xA;&#xA;#Forecast linear model&#xA;A&amp;lt;-c(A,rnorm(5))&#xA;B&amp;lt;-c(B,rnorm(5))&#xA;C&amp;lt;-c(C,rnorm(5))&#xA;y=window(y,end=end(y)+c(5,0),extend=TRUE)&#xA;newdata&amp;lt;-cbind(y,A,B,C)&#xA;P1&amp;lt;-ipredict(model.dyn,newdata)&#xA;P2&amp;lt;-ipredict(model.dyn,newdata,interval=&quot;prediction&quot;)&#xA;&#xA;#Plot&#xA;plot(y)&#xA;lines(P1,col=2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2817" LastActivityDate="2011-01-31T21:04:56.823" CommentCount="8" />
  <row Id="6762" PostTypeId="1" AcceptedAnswerId="6766" CreationDate="2011-01-31T22:49:27.690" Score="4" ViewCount="188" Body="&lt;p&gt;We have n actors. Each actor chooses from n*2 actions. How can I calculate the probability that at least one actor will choose a unique outcome?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, say we have 5 pickup artists in a town with 10 bars. Each PUA chooses a bar at random. What are the odds that at least one PUA will have a bar to himself/herself? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is relevant to some work I'm doing on scheduling in a distributed system. I know that the relevant equation should probably start with $\binom{n + 2n -1}{n}$, but then what?&lt;/p&gt;&#xA;" OwnerUserId="2995" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-02-02T19:25:00.480" LastActivityDate="2011-02-06T14:48:56.133" Title="Probability of at least one unique outcome" Tags="&lt;probability&gt;&lt;combination&gt;" AnswerCount="2" FavoriteCount="0" />
  <row Id="6763" PostTypeId="1" CreationDate="2011-01-31T23:06:13.850" Score="2" ViewCount="136" Body="&lt;p&gt;I have 80 chemical batches, each of which has 8 associated measurements like pH, viscosity, etc.  Is there a way to use Principal Components Analysis or Factor Analysis to group the similar batches together?  I know I can use cluster analysis, but I want to have a different visual representation of batch groups other than the dendogram.  I'd like to have a bi-plot.  I tried to perform a PCA, but got an error message that there cannot be more variables than observations.  My matrix has the 80 batches as the column names, and the 8 measurements as the row names.  When this matrix is transposed, PCA just groups the like measurements, not the batches.&lt;/p&gt;&#xA;" OwnerUserId="2473" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-02-01T00:14:02.047" LastActivityDate="2011-05-02T13:52:06.720" Title="Statistically group chemical batches" Tags="&lt;pca&gt;&lt;factor-analysis&gt;" AnswerCount="4" CommentCount="2" />
  <row Id="6766" PostTypeId="2" ParentId="6762" CreationDate="2011-02-01T00:44:41.830" Score="5" ViewCount="" Body="&lt;p&gt;This is not a simple calculation in general, though the probability will be almost 1 if $n$ is any substantial size.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To take your example of five people and 10 bars, there are $10^5$, i.e 100,000 possible equally-likely distributions.  All of these will have at least one person alone, except the 10 cases where all five are in a single bar or the 900 cases where three are in a particular bar and two in another.  That leaves 99090 distributions which satisfy your condition, more than 99%.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that 99090 is not divisible by $\binom{5 + 10 - 1}{5} = 2002$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For $n=1$ to $6$, the fractions seem to be $\frac{2}{2},$ $\frac{12}{16},$ $\frac{210}{216},$ $\frac{3920}{4096},$ $\frac{99090}{100000}$ and $\frac{2962872}{2985984}$, where the denominator in each case is $(2n)^n$. I would expect the probabilities then to keep rising as $n$ increases beyond 6.    &lt;/p&gt;&#xA;" OwnerUserId="2958" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-02-01T14:56:57.243" LastActivityDate="2011-02-01T14:56:57.243" CommentCount="1" />
  <row Id="6767" PostTypeId="2" ParentId="6763" CreationDate="2011-02-01T00:49:31.910" Score="0" ViewCount="" Body="&lt;p&gt;You need to be more specific about what you mean by &quot;similar&quot;, and provide some type of example.   Maybe the following will help as an example of 8 batches with 3 measurements.    Feel free to change it if you think it will help.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;texinp &amp;lt;- &quot;&#xA;Batch M1 M2 M3&#xA;1  0.01  14  -123&#xA;2  0.03  23    30&#xA;3  0.02  12    23&#xA;4  0.01  18  -107&#xA;5  0.04  49   110&#xA;6  0.01   8  -101&#xA;7  0.02  11    11&#xA;8  0.01  14  -115&quot;&#xA;&#xA;#Read the data into a data frame&#xA;df &amp;lt;- read.table(textConnection(texinp), header=TRUE)&#xA;df&#xA;&#xA;#Plot some stuff&#xA;plot(df$M1, df$M2)&#xA;plot(df$M2, df$M3)&#xA;plot(df$M1, df$M3)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you're not using R, show the example in whatever you're using.&lt;/p&gt;&#xA;" OwnerUserId="2775" LastEditorUserId="2775" LastEditorDisplayName="" LastEditDate="2011-02-01T02:04:59.757" LastActivityDate="2011-02-01T02:04:59.757" />
  <row Id="6768" PostTypeId="2" ParentId="6763" CreationDate="2011-02-01T01:03:26.380" Score="0" ViewCount="" Body="&lt;p&gt;You'll be hard-pressed to show an 8-D representation of sets of similar batches using anything but a dreary table.  But, along the lines of Bill's point I think, if you're willing to select the 3 most interesting or most discriminating dimensions, you could show where each batch falls within a cube defined by those 3.  Perhaps better is to first do a cluster analysis and then show where each cluster falls within such a cube.  Some software (such as SPSS) will allow you to assign each batch a color or a symbol according to its cluster, and you could draw spikes from each point to its cluster's centroid to create a nice, vivid effect.   &lt;/p&gt;&#xA;" OwnerUserId="2669" LastActivityDate="2011-02-01T01:03:26.380" />
  <row Id="6770" PostTypeId="1" AcceptedAnswerId="7149" CreationDate="2011-02-01T01:51:58.023" Score="2" ViewCount="254" Body="&lt;p&gt;I'm writing a program for an assignment for my Probability and Statistics course. This program performs several millions of simulated coin tosses, and then analyses various properties of the generated sequence. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of the things it does is count how many runs does the generated sequence contain, and their lengths. One result for 10.000.000 tosses is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;Number of runs of length 1 is: 2500986&#xA;Number of runs of length 2 is: 1246647&#xA;Number of runs of length 3 is: 625656&#xA;Number of runs of length 4 is: 311689&#xA;Number of runs of length 5 is: 156673&#xA;Number of runs of length 6 is: 78464&#xA;Number of runs of length 7 is: 39253&#xA;Number of runs of length 8 is: 19547&#xA;Number of runs of length 9 is: 9866&#xA;Number of runs of length 10 or more is: 9818&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, my problem is following: I need to compare the result acquired by simulation to theoretical expectation of sorts (my English ain't that good, I hope you understand what I mean). Basically, i need that same result as above, calculated on paper. I've spent over a week trying to figure this out, and I'm getting desperate, as my deadline is approaching, and I have no idea how to solve this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Pattern in the results obviously exists, but I just cant figure out a formula. Anyone, help please? :D&lt;/p&gt;&#xA;" OwnerUserId="2996" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-02-01T09:26:52.710" LastActivityDate="2011-02-13T11:02:14.957" Title="Expected number of runs of length n in coin toss" Tags="&lt;homework&gt;&lt;binomial&gt;&lt;simulation&gt;" AnswerCount="2" CommentCount="6" />
  <row Id="6771" PostTypeId="2" ParentId="6728" CreationDate="2011-02-01T04:54:02.557" Score="1" ViewCount="" Body="&lt;p&gt;@Legend:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As I mentioned in my earlier post to you, I'm working on getting my head around fitting a distribution to count data also. Here's among what I've learned:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When the variance is greater than the mean, overdispersion is evident and thus the negative binomial distribution is likely appropriate. If the variance and mean are the same, the Poisson distribution is suggested, and when the variance is less than the mean, it's the binomial distribution that's recommended.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With the count data you're working on, you're using the &quot;ecological&quot; parameterization of the Negative Binomial function in R. Section 4.5.1.3 (Page 165) of the following freely-available book speaks to this specifically (in the context of R, no less!) and, I hope, might address some of your questions: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.math.mcmaster.ca/~bolker/emdbook/book.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.math.mcmaster.ca/~bolker/emdbook/book.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you come to conclude that your data are zero-truncated (i.e., the probability of 0 observations is 0), then you might want to check out the zero-truncated flavor of the NBD that's in the R VGAM package (my meager point allocation prohibits me from posting the URL here).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's an example of its application:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(VGAM)&#xA;&#xA;someCounts = data.frame(n = c(1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16),&#xA;                     freq = c(182479,76986,44859,24315,16487,15308,5736,2843,1370,1115,1127,49,100,490,106,2))&#xA;&#xA;fit = vglm(n ~ 1, posnegbinomial, control = vglm.control(maxit = 1000), weights=freq, data=someCounts)&#xA;&#xA;Coef(fit)&#xA;&#xA;pdf2 = dposnegbin(x=with(someCounts, n), munb=0.8344248, size=0.4086801)&#xA;&#xA;print( with(someCounts, cbind(n, freq, fitted=pdf2*sum(freq))), dig=9)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I hope this is helpful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;--Jeff&lt;/p&gt;&#xA;" OwnerUserId="2730" LastEditorUserId="2730" LastEditorDisplayName="" LastEditDate="2011-02-01T04:59:52.097" LastActivityDate="2011-02-01T04:59:52.097" />
  <row Id="6772" PostTypeId="1" AcceptedAnswerId="8644" CreationDate="2011-02-01T05:58:26.000" Score="4" ViewCount="219" Body="&lt;p&gt;Not sure if you can help me but here's the problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have two binomial proportions A and B (95% CI) -&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$A = 2\% \pm 0.2\%,\quad B = 3\% \pm 0.2\%.$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words, B's proportion is 50% higher than A's.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Business people tend to prefer speaking about this difference in terms of a percentage increase. In other words, B's proportion is 50% higher than A's. How do I go about calculating a Confidence Interval for the 50% increase of B's proportion over A.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've done some research into Fieller’s formula which is said to provide exact percent effect. However, besides not understanding Fieller's theorem, I don't know how I would go about calculating this in R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone help? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt; E.g., if the CI for those intervals is 95%, the CI for the 50% increase = 98.52%.&lt;/p&gt;&#xA;" OwnerUserId="2595" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-03-22T22:11:13.873" LastActivityDate="2011-03-22T22:11:13.873" Title="How to compute Confidence Interval associated to a Binomial proportion's increase?" Tags="&lt;r&gt;&lt;confidence-interval&gt;&lt;binomial&gt;&lt;proportion&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="6773" PostTypeId="1" AcceptedAnswerId="6872" CreationDate="2011-02-01T06:35:55.837" Score="4" ViewCount="156" Body="&lt;p&gt;I have a logistic regression model in GeoBugs to estimate predictors of prevalence of a disease. Can anyone tell me if there is a simple way to determine the lower and upper bounds of phi for spatial.exp.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is part of the model:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Infected[i] ~ dbin(p[i], tested[i])&#xA;logit(p[i]) &amp;lt;- alpha + beta1*covar1[i] + beta2*covar2[i]  + u[i] &#xA;mu[i] &amp;lt;- 0&#xA;u[1:N] ~ spatial.exp(mu[], x[], y[], tau, phi, 1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This how I have specified the priors:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;phi ~ dunif(0.25, 20)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;At the moment my selection has been arbitrary. The GeoBugs manual only goes as far as saying that the bounds of the distribution are determined by the data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I would like to know is there a dummies version of the best way to select the values. &lt;/p&gt;&#xA;" OwnerUserId="1834" LastEditorUserId="1834" LastEditorDisplayName="" LastEditDate="2011-02-04T04:23:15.950" LastActivityDate="2011-02-04T04:27:12.527" Title="Geostatistical analysis using spatial.exp in WinBugs" Tags="&lt;bayesian&gt;&lt;logistic-regression&gt;&lt;spatial&gt;&lt;bugs&gt;" AnswerCount="1" CommentCount="0" />
  <row Id="6774" PostTypeId="2" ParentId="6763" CreationDate="2011-02-01T07:30:18.610" Score="0" ViewCount="" Body="&lt;p&gt;I'm not pretty sure, but i think that if you more variable than observation  the Partial Least Square (PLS) regression is the right tool for you, and it's commonly used by chemometrician.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you know this site is R biased and i use R too, so i'll recommend to take a look at pls package package which can also do principal component regression, and it also have a biplot.mvr (S3 method for biplot) function to make a biplot.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A good starting point can be &lt;a href=&quot;http://mevik.net/work/software/pls.html&quot; rel=&quot;nofollow&quot;&gt;http://mevik.net/work/software/pls.html&lt;/a&gt; , is the website of package's author and have some useful link like the journal of statistical software article about the R package pls.&lt;/p&gt;&#xA;" OwnerUserId="2028" LastEditorUserId="2028" LastEditorDisplayName="" LastEditDate="2011-02-01T07:55:29.913" LastActivityDate="2011-02-01T07:55:29.913" />
  <row Id="6775" PostTypeId="2" ParentId="6758" CreationDate="2011-02-01T07:39:48.617" Score="2" ViewCount="" Body="&lt;p&gt;Congratulations, you have found a bug. Prediction for &lt;code&gt;dynlm&lt;/code&gt; with new data is broken if lagged variables are used.  To see why look at the output of&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;predict(model)&#xA;predict(model,newdata=data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The results should be the same, but they are not. Without &lt;code&gt;newdata&lt;/code&gt; argument, the &lt;code&gt;predict&lt;/code&gt; function basically grabs &lt;code&gt;model&lt;/code&gt; element from the &lt;code&gt;dynlm&lt;/code&gt; output. With &lt;code&gt;newdata&lt;/code&gt; argument &lt;code&gt;predict&lt;/code&gt; tries to form new model matrix from &lt;code&gt;newdata&lt;/code&gt;. Since this involves parsing formula supplied to &lt;code&gt;dynlm&lt;/code&gt; and the formula has function &lt;code&gt;L&lt;/code&gt;, which is defined only internaly in function &lt;code&gt;dynlm&lt;/code&gt;, the incorrect model matrix is formed. If you try to debug, you will see, that the lagged dependent variable is not being lagged in the case of &lt;code&gt;newdata&lt;/code&gt; argument is supplied.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you can do is to lag the dependent variable and include it in the &lt;code&gt;newdata&lt;/code&gt;. Here is the code illustrating this approach. I use &lt;code&gt;set.seed&lt;/code&gt; so it would be easily reproducible.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(dynlm)&#xA;&#xA;set.seed(1)&#xA;y&amp;lt;-arima.sim(model=list(ar=c(.9)),n=10) #Create AR(1) dependant variable&#xA;A&amp;lt;-rnorm(10) #Create independant variables&#xA;B&amp;lt;-rnorm(10)&#xA;C&amp;lt;-rnorm(10)&#xA;y&amp;lt;-y+.5*A+.2*B-.3*C #Add relationship to independant variables &#xA;data=cbind(y,A,B,C)&#xA;&#xA;#Fit linear model&#xA;model&amp;lt;-dynlm(y~A+B+C+L(y,1),data=data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here is the buggy behaviour:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; predict(model)&#xA;       2        3        4        5        6        7        8        9       10 &#xA;3.500667 2.411196 2.627915 2.813815 2.468595 1.733852 2.114553 1.423225 1.470738 &#xA;&amp;gt; predict(model,newdata=data)&#xA;        1         2         3         4         5         6         7         8         9        10 &#xA;2.1628335 3.7063579 2.9781417 2.1374301 3.2582376 1.9534558 1.3670995 2.4547626 0.8448223 1.8762437 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Form the &lt;code&gt;newdata&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#Forecast fix.&#xA;A&amp;lt;-c(A,rnorm(1)) #Assume we already have 1-step forecasts for A,B,C&#xA;B&amp;lt;-c(B,rnorm(1))&#xA;C&amp;lt;-c(C,rnorm(1))&#xA;&#xA;newdata&amp;lt;-ts(cbind(A,B,C),start=start(y),freq=frequency(y))&#xA;&#xA;newdata&amp;lt;-cbind(lag(y,-1),newdata)&#xA;colnames(newdata) &amp;lt;- c(&quot;y&quot;,&quot;A&quot;,&quot;B&quot;,&quot;C&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Comnpare forecast with model fit:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; predict(model)&#xA;       2        3        4        5        6        7        8        9       10 &#xA;3.500667 2.411196 2.627915 2.813815 2.468595 1.733852 2.114553 1.423225 1.470738 &#xA;&amp;gt; predict(model,newdata=newdata)&#xA;       1        2        3        4        5        6        7        8        9       10       11 &#xA;      NA 3.500667 2.411196 2.627915 2.813815 2.468595 1.733852 2.114553 1.423225 1.470738 1.102367 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As you can see for historical data the forecast coincides and the last element contains the 1-step ahead forecast.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-02-01T08:03:30.793" LastActivityDate="2011-02-01T08:03:30.793" />
  <row Id="6776" PostTypeId="1" CreationDate="2011-02-01T09:07:09.633" Score="3" ViewCount="164" Body="&lt;p&gt;Hello,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would appreciate some help getting some EM stuff straight. So, say I generate data in R as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;N       &amp;lt;- 100&#xA;epsilon &amp;lt;- rnorm(N)&#xA;X       &amp;lt;-  10*runif(N)&#xA;beta.0  &amp;lt;- 10&#xA;beta.1  &amp;lt;-  3&#xA;sigma   &amp;lt;- 2&#xA;Y       &amp;lt;-  beta.0 + beta.1 * X + sigma * epsilon&#xA;epsilon2 &amp;lt;- rnorm(N)&#xA;X2 &amp;lt;- 10*runif(N)&#xA;Y2 &amp;lt;-  3 - X2 + 0.25 * epsilon2&#xA;Y.mix &amp;lt;- c(Y, Y2)&#xA;X.mix &amp;lt;- c(X, X2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, in expectation maximization, in the first step, I have some prior probability, say 0.5, of the data being from either one or the other distribution. So, using EM I know I can estimate the mean and variance of the two mixtures. From looking at a density plot, it seems like the means are at about -2 and 30 for the data I simulated. But, at what stage in EM do I back out the betas? I want to recover the slope, intercept, and sd deviation parameters for the 2 regression-type equations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for an clarification.&lt;/p&gt;&#xA;" OwnerUserId="2693" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-27T16:04:19.703" LastActivityDate="2011-05-26T22:50:31.847" Title="Understanding expectation maximization for simple 2 linear mixture case" Tags="&lt;r&gt;&lt;machine-learning&gt;&lt;mixture&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="6777" PostTypeId="1" CreationDate="2011-02-01T10:56:25.047" Score="3" ViewCount="83" Body="&lt;p&gt;I would like to apply Bayesian network on some data. However, some of the variables are related to time. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;E.g.  Number of time he/she visit library.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As the value can be defined as Total number of this person visits, Average weekly number of he/she visit, Daily number of visits, or even Number of visits in the last 3 days etc...&#xA;I do not want to include all these different variations in my network&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then how should I treat feature like this properly?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks for your help&lt;/p&gt;&#xA;" OwnerUserId="2454" LastActivityDate="2011-02-01T17:08:20.320" Title="How should I deal with features with time related values in Bayesian network?" Tags="&lt;time-series&gt;&lt;bayesian&gt;" AnswerCount="1" />
  <row Id="6778" PostTypeId="2" ParentId="6747" CreationDate="2011-02-01T12:09:07.593" Score="0" ViewCount="" Body="&lt;p&gt;It's a little weird to answer my own question, but...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To address the non-stationarity issue, it seems as though it's not particularly tricky to compute the expected $K$-function under the hypothesis of independent event times. In particular, let's say that we're looking at point processes on the interval $I=[0,T]$, and our processes $X$ and $Y$ have unconditional distributions $f_X(x)$ and $f_Y(y)$. We'll also denote the unit ball in one dimension by $B(x,t) = \{y\in I:d(x,y)&amp;lt;t\}$. Then&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$K_{XY}(t) = \mathbb{E}[\textrm{number of Y points in a neighbourhood t of a randomly chosen X point}]$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$K_{XY}(t) = \mathbb{E}_X[\mathbb{E}_Y[\textrm{number of Y points within t of x}]]$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$K_{XY}(t) = \int_I dx \int_{B(x,t)} dy \, f_X(x) f_Y(y)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So deviations from this function in the estimated $\hat{K}_{XY}(t)$ will be an indicator of correlation between the point processes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does this sound reasonable?&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Edit: In fact, I think this all holds if you take $I\subseteq\mathbb{R}^n$ and interpret $x$, $y$ as vectors.&lt;/p&gt;&#xA;" OwnerUserId="2425" LastActivityDate="2011-02-01T12:09:07.593" CommentCount="2" />
  <row Id="6779" PostTypeId="1" AcceptedAnswerId="6782" CreationDate="2011-02-01T12:51:07.040" Score="5" ViewCount="148" Body="&lt;p&gt;Apologies in advance - I'm a bit of a newbie with R but I've been Google-ing this for weeks and I haven't found a straight answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The problem&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically, I'm looking to plot proportions with their confidence intervals on a box and whiskers plot using the R environment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;i.e. Plot the results of binom.test(100,1000) and binom.test(125,1000)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the best package (I don't have a preference either way) and way to go about doing this?&lt;/p&gt;&#xA;" OwnerUserId="2595" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-02-01T14:57:09.207" LastActivityDate="2011-02-01T14:57:09.207" Title="Plotting binomial proportions on a box &amp; whiskers plot using R" Tags="&lt;r&gt;&lt;data-visualization&gt;&lt;boxplot&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="6780" PostTypeId="1" AcceptedAnswerId="6781" CreationDate="2011-02-01T13:24:57.997" Score="4" ViewCount="374" Body="&lt;p&gt;I have several query frequencies, and I need to estimate the coefficient of Zipf's law. These are the top frequencies: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;26486&#xA;12053&#xA;5052&#xA;3033&#xA;2536&#xA;2391&#xA;1444&#xA;1220&#xA;1152&#xA;1039&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2998" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2011-03-15T19:04:43.443" LastActivityDate="2011-03-15T19:04:43.443" Title="How to calculate Zipf's law coefficient from a set of top frequencies?" Tags="&lt;distributions&gt;&lt;estimation&gt;&lt;pareto-distribution&gt;&lt;zipf&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="1" />
  <row Id="6781" PostTypeId="2" ParentId="6780" CreationDate="2011-02-01T14:12:29.077" Score="7" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt; I've updated the code with maximum likelihood estimator as per @whuber suggestion. Minimizing sum of squares of differences between log theoretical probabilities and log frequencies though gives an answer would be a statistical procedure if it could be shown that it is some kind of M-estimator. Unfortunately I could not think of any which could give the same results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is my attempt. I calculate logarithms of the frequencies and try to fit them to logarithms of theoretical probabilities &lt;a href=&quot;http://en.wikipedia.org/wiki/Zipf%27s_law&quot; rel=&quot;nofollow&quot;&gt;given by this formula&lt;/a&gt;. The final result seems reasonable. Here is my code in R.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;fr&amp;lt;-c(26486, 12053, 5052, 3033, 2536, 2391, 1444, 1220, 1152, 1039)&#xA;&#xA;p&amp;lt;-fr/sum(fr)&#xA;&#xA;lzipf &amp;lt;- function(s,N) -s*log(1:N)-log(sum(1/(1:N)^s))&#xA;&#xA;opt.f&amp;lt; -function(s) sum((log(p)-lzipf(s,10))^2)&#xA;&#xA;opt &amp;lt;- optimize(opt.f,c(0.5,10))&#xA;&#xA;&amp;gt; opt&#xA;$minimum&#xA;[1] 1.463946&#xA;&#xA;$objective&#xA;[1] 0.1346248&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The best quadratic fit then is $s=1.47$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The maximum likelihood in R can be performed with &lt;code&gt;mle&lt;/code&gt; function (from &lt;code&gt;stats4&lt;/code&gt; package), which helpfully calculates standard errors (if correct negative maximum likelihood function is supplied):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ll &amp;lt;-function(s) sum(fr*(s*log(1:10)+log(sum(1/(1:10)^s))))&#xA;&#xA;fit &amp;lt;- mle(ll,start=list(s=1))&#xA;&#xA;&amp;gt; summary(fit)&#xA;Maximum likelihood estimation&#xA;&#xA;Call:&#xA;mle(minuslogl = ll, start = list(s = 1))&#xA;&#xA;Coefficients:&#xA;  Estimate  Std. Error&#xA;s 1.451385 0.005715046&#xA;&#xA;-2 log L: 188093.4 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here is the graph of the fit in log-log scale (again as @whuber suggested):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;s.sq &amp;lt;- opt$minimum&#xA;s.ll &amp;lt;- coef(fit)&#xA;&#xA;plot(1:10,p,log=&quot;xy&quot;)&#xA;lines(1:10,exp(lzipf(s.sq,10)),col=2)&#xA;lines(1:10,exp(lzipf(s.ll,10)),col=3)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Red line is sum of squares fit, green line is maximum-likelihood fit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/ChSRC.png&quot; alt=&quot;Log-log graph of fits&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-02-01T20:47:01.160" LastActivityDate="2011-02-01T20:47:01.160" CommentCount="4" />
  <row Id="6782" PostTypeId="2" ParentId="6779" CreationDate="2011-02-01T14:28:40.207" Score="6" ViewCount="" Body="&lt;p&gt;This really does look to be more appropriate to an [r] tagged question in SO but there seems to be a surprisingly wide degree of tolerance for such questions in CV, so here goes. The answer also assumes you did not really want to use the existing &lt;code&gt;boxplot()&lt;/code&gt; function which is what would be used to construct a true box-and-whiskers plot (and which doesn't really use binomial CI's, but rather uses the Tukey defined &lt;code&gt;fivenum()&lt;/code&gt; function ). The plotrix and gplots packages both have what appears to be the same plotCI function;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;est1 &amp;lt;- binom.test(100,1000) ; est2 &amp;lt;- binom.test(125,1000)&#xA;estvals &amp;lt;- rbind( c(est1$estimate, est1$conf.int[1], est1$conf.int[2]),&#xA;                  c(est2$estimate, est2$conf.int[1], est2$conf.int[2]))&#xA;plotCI(x = c(1,2), estvals[,1], ui=estvals[,3], li=estvals[,2], &#xA;         xlim=c(0.5,2.5), ylim=c(0, 0.15) , xaxt=&quot;n&quot;, xlab=&quot;Groups&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The basic function is the &lt;code&gt;arrows()&lt;/code&gt; plotting function with an argument that splays the points of the arrows to 90 degrees in both directions.&lt;/p&gt;&#xA;" OwnerUserId="2129" LastEditorUserId="2129" LastEditorDisplayName="" LastEditDate="2011-02-01T14:34:04.300" LastActivityDate="2011-02-01T14:34:04.300" CommentCount="1" />
  <row Id="6783" PostTypeId="1" CreationDate="2011-02-01T15:38:30.733" Score="1" ViewCount="65" Body="&lt;p&gt;Hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I receive 1000 &lt;em&gt;points&lt;/em&gt; per day from installations who produces electricity. &#xA;Every installation must proportionally produce the same amount of energy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have to spot failures in those data. The actual method : &lt;br/&gt;&#xA;1. I make an average. &lt;br/&gt;&#xA;2. I make a second average where I remove the installation who are 30% lower than the first average&lt;br/&gt;&#xA;3. I compare every installation with the filtered average. If an installation is lower than x % from the filtered average, there is a problem.&lt;br/&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there some better model for that kind of purpose ? Do you have books reference about that ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance&lt;/p&gt;&#xA;" OwnerUserId="3000" LastEditorUserId="3000" LastEditorDisplayName="" LastEditDate="2011-02-07T13:42:35.550" LastActivityDate="2011-02-07T13:42:35.550" Title="Failure detection method" Tags="&lt;diagnostic&gt;&lt;failure&gt;" CommentCount="5" />
  <row Id="6784" PostTypeId="2" ParentId="6544" CreationDate="2011-02-01T16:19:03.493" Score="2" ViewCount="" Body="&lt;p&gt;One more method, which has been suggested in other topics, is to just use the arima function with xregs.  Arima seems to be able to make forecasts from a new set of xregs just fine.&lt;/p&gt;&#xA;" OwnerUserId="2817" LastActivityDate="2011-02-01T16:19:03.493" CommentCount="1" />
  <row Id="6785" PostTypeId="2" ParentId="6777" CreationDate="2011-02-01T17:08:20.320" Score="1" ViewCount="" Body="&lt;p&gt;I am afraid you won't like this answer, but it depends on what you want to do. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Non-Time-Series-Modelling-Point of View&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an example: Suppose you want to predict whether a certain person is visiting the library at the next day. The starting point for your data is hence the data-id + person_id + label (visit=true/false). Now the question is which predictors are needed. Can you tell me which of this predictors is meaningful or not (without performing any modelling) ? &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;average-number-of-visits per week: Persons visiting the library more often have a higher chance of visiting the next day&lt;/li&gt;&#xA;&lt;li&gt;number-of-visits-in-the-last/days/hours: Some people go one only once per week to the library, but not always on the same day. So depending on the total-visit-count per week + whether the customer has visited the library recently, one could say whether the same person will come in again the next day.&lt;/li&gt;&#xA;&lt;li&gt;daily-number-of-visits: Every day multiple visits ? Or only multiple visits on the one weekday where the person comes in ?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;and so on ... I think you get the idea. I would try to generate a good amount of predictors and then apply subset selection techniques to find the best combination (yes, I am a machine learner).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Time-Series-Modelling-Point of View&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I admit I do not know that much about time-series analysis. I can only imagine that drawing one timeline per customer (e.g. in days or hours on the time/x-axis + plus a peak everytime the customer visits) and classifying this timelines (in context of the example in the first section) might help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;PS:&lt;/strong&gt; I am aware that this answer is not as complete as it should be. However, I hope to stimulate the discussion that way so one can come up with the true answer.&lt;/p&gt;&#xA;" OwnerUserId="264" LastActivityDate="2011-02-01T17:08:20.320" CommentCount="7" />
  <row Id="6786" PostTypeId="2" ParentId="6780" CreationDate="2011-02-01T17:22:39.173" Score="8" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;There are several issues before us in &lt;em&gt;any&lt;/em&gt; estimation problem:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Estimate the parameter.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Assess the quality of that estimate.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Explore the data.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Evaluate the fit.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;For those who would use statistical methods for understanding and communication, &lt;em&gt;the first should never be done without the others.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For &lt;strong&gt;estimation&lt;/strong&gt; it is convenient to use &lt;strong&gt;maximimum likelihood&lt;/strong&gt;.  The frequencies are so large we can expect the well-known asymptotic properties to hold.  The log likelihood for data $\{f_i\},\ i=1,\ldots,n$ is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\Lambda = -s \sum_{j=1}^n{f_j \log(j)} - \left(\sum_{j=1}^n{f_j}\right) \log\left(\sum_{i=1}^n{i^{-s}}\right).$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Numerical minimization with these data yields $\hat{s} = 1.45041$ and $\Lambda(\hat{s}) = -94046.7$.  This is significantly better (but just barely so) than the least squares solution (based on log frequencies) of $\hat{s}_{ls} = 1.463946$ with $\Lambda(\hat{s}_{ls}) = -94049.5$.  (The optimization can be done with a minor change to the elegant, clear &lt;strong&gt;R&lt;/strong&gt; code provided by mpiktas.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ML will also estimate &lt;strong&gt;confidence limits&lt;/strong&gt; for $s$ in the usual ways.  The chi-square approximation gives $[1.43922, 1.46162]$ (if I did the calculations correctly :-).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given the nature of Zipf's law, &lt;em&gt;the right way to graph this fit is on a log-log plot&lt;/em&gt;, where the fit will be linear (by definition):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/G1kEQ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To evaluate the goodness of fit and explore the data, &lt;strong&gt;look at the residuals&lt;/strong&gt; (data/fit, log-log axes again):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/mYmkx.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is not too great: although there's no evident serial correlation or heteroscedasticity in the residuals, they typically are around 10% (away from 1.0).  With frequencies in the thousands, we wouldn't expect deviations by more than a few percent.  The &lt;strong&gt;goodness of fit&lt;/strong&gt; is readily tested with &lt;em&gt;chi square&lt;/em&gt;.  We obtain $\chi^2 = 656.476$ with 10 - 1 = 9 degrees of freedom; this is &lt;strong&gt;highly significant evidence of departures from Zipf's Law&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Because the residuals appear random, in some applications we might be content to accept Zipf's Law (and our estimate of the parameter) as an &lt;em&gt;acceptable albeit rough description of the frequencies&lt;/em&gt;.  This analysis shows, though, that it would be a mistake to suppose this estimate has any explanatory or predictive value for the dataset examined here.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-02-01T19:21:57.197" LastActivityDate="2011-02-01T19:21:57.197" CommentCount="15" />
  <row Id="6787" PostTypeId="1" CreationDate="2011-02-01T17:54:27.250" Score="3" ViewCount="87" Body="&lt;p&gt;I've identified a Single Nucleotide Polymorphism (SNP) that is associated with a variable of interest (lets just call it height) using a Genome Wide Association Study (GWAS). The SNP can be of the values A/A, A/G or G/G in each of the 69 samples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also have gene expression data for the 69 samples and it appears that the SNP is affecting the expression of a large number (~500) genes. I'm wondering if there is a convenient statistical package/technique available to assess if the variable (height) becomes independent of genotype (A/A, A/G or G/G) conditioning on the expression of each of the ~500 genes who's expression are associated with the SNP (i.e. assessing conditional independence of each of the ~500 genes)?&lt;/p&gt;&#xA;" OwnerUserId="3001" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-02-01T22:17:04.600" LastActivityDate="2011-02-01T22:17:04.600" Title="Assessing conditional independence of genes in Trans-eQTL cluster" Tags="&lt;regression&gt;&lt;independence&gt;&lt;genetics&gt;" AnswerCount="1" />
  <row Id="6788" PostTypeId="2" ParentId="6770" CreationDate="2011-02-01T18:12:51.627" Score="1" ViewCount="" Body="&lt;p&gt;You have n independent Bernoulli rv, $X_1, X_2, ..., X_n$ with $P(X_i=1)=P(X_i=0)=p=1/2$.  You generate n bits $b_1, b_2, ..., b_n$, as observations on this sequence. Let N be the rv giving the number of runs of one, for example, in the simulated sequence of bits. &#xA;$N=\sum_{i=1}^n Y_i$, where $Y_i$ is the random variable that takes 1 as value if $b_i$ is the starting bit of a run of ones and 0 otherwise. $E(Y_1)=P(Y_1=1)=p$ while for $i=2,...,n$ $P(Y_i=1)=P(X_{i-1}=0, X_i=1)=(1-p)p.$ Hence the expected value of N, is $E(N)=p+(n-1)(1-p)/p$&lt;/p&gt;&#xA;" OwnerUserId="2376" LastEditorUserId="2376" LastEditorDisplayName="" LastEditDate="2011-02-02T15:56:31.237" LastActivityDate="2011-02-02T15:56:31.237" CommentCount="2" />
  <row Id="6789" PostTypeId="2" ParentId="6755" CreationDate="2011-02-01T18:58:10.837" Score="2" ViewCount="" Body="&lt;p&gt;I found a way that is at least now feasible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead of the sparseM package I use the Matrix package to build the sparse matrices.  Store the entries and columns in separate lists and then build the matrix by:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data=sparseMatrix(i=rep(1,length[[1]]),j=columns[[loop]],&#xA;    x=entires[[loop]],dims=c(120000,47235))&#xA;for(loop in 2:120000){&#xA;  if(loop %% 1000==0){&#xA;      print(loop)&#xA;      print(Sys.time())&#xA;  }&#xA;  data[loop,columns[[loop]]]=entries[[loop]]  &#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This still takes awhile (about 2 hours) but at least it works.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I store the corresponding classes in another list not in the sparse matrix.&lt;/p&gt;&#xA;" OwnerUserId="2310" LastActivityDate="2011-02-01T18:58:10.837" />
  <row Id="6790" PostTypeId="2" ParentId="6787" CreationDate="2011-02-01T22:15:50.637" Score="4" ViewCount="" Body="&lt;p&gt;I would recommend looking at the &lt;a href=&quot;http://www.bioconductor.org/packages/2.3/bioc/html/snpMatrix.html&quot; rel=&quot;nofollow&quot;&gt;snpMatrix&lt;/a&gt; R package. Within the &lt;code&gt;snp.lhs.tests()&lt;/code&gt; function, height will be the phenotype (or outcome), your SNP data will be stored in a vector (&lt;code&gt;snp.data&lt;/code&gt;), and your gene expression levels will enter the model as covariates. I never used this kind of covariates (in the GWAS I am familiar with we adjust for population stratification and subject-specific covariates), so I can't give an example of use. Just give it a try to see if it suits your needs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Otherwise, a larger modeling framework is available in the &lt;a href=&quot;http://www.bioconductor.org/packages/2.2/bioc/html/GGtools.html&quot; rel=&quot;nofollow&quot;&gt;GGtools&lt;/a&gt; package, from the &lt;a href=&quot;http://www.bioconductor.org&quot; rel=&quot;nofollow&quot;&gt;Bioconductor&lt;/a&gt; project too.  It was developed by Vince J. Carey for dealing specifically with SNP and gene expression data, in GWAS or eQTL studies. There are numerous examples of use on the web (look up for tutorial by VJ Carey with &lt;code&gt;GGtools&lt;/code&gt;). However, it's mainly to model gene expression as a function of observed genotypes (chromosome- or genome-wide).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Caution.&lt;/strong&gt; Please note that &lt;code&gt;GGtools&lt;/code&gt; package has a lot of dependencies, in particular &lt;code&gt;GGBase&lt;/code&gt; (which provides specific environments for holding genetic data) and &lt;code&gt;snpMatrix&lt;/code&gt; (which an provides an efficient storage of SNP data). You will also need to install the base packages from the Bioconductor repository (this is not part of CRAN); detailed instructions are available &lt;a href=&quot;http://www.bioconductor.org/install/index.html&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-02-01T22:15:50.637" />
  <row Id="6791" PostTypeId="1" AcceptedAnswerId="6797" CreationDate="2011-02-01T23:36:42.177" Score="6" ViewCount="354" Body="&lt;p&gt;Let Year 1 be last year's data and Year 2 be this year's data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose that in Year 1, you had a likert scale that was 1-9 (Categorical/Ordinal) and that in Year 2, for the same question you had a likert scale that was 1-5 (Categorical/Ordinal). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What would be some of the things that you would try (if at all) to compare the two years worth of data? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I've done so far: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Compared distributions (shape, skew, and kurtosis, statistically equal)&lt;/li&gt;&#xA;&lt;li&gt;Rescaled 1-9 to 1-5 and the changes YoY in frequencies match logical expectations derived from industry news/events and qualitative research findings. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Note: This is not homework. It also may not have a definite answer. But, I need a hand! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance!&lt;/p&gt;&#xA;" OwnerUserId="776" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-03-30T18:42:33.780" LastActivityDate="2011-03-31T17:12:31.247" Title="How to compare Likert scales with varying number of categories across time?" Tags="&lt;scales&gt;&lt;likert&gt;" AnswerCount="3" CommentCount="3" FavoriteCount="2" />
  <row Id="6792" PostTypeId="2" ParentId="5070" CreationDate="2011-02-02T00:04:57.403" Score="0" ViewCount="" Body="&lt;p&gt;now assume its a loaded die instead of a coin..can you derive the equation from that?&lt;/p&gt;&#xA;" OwnerUserId="3005" LastActivityDate="2011-02-02T00:04:57.403" CommentCount="2" />
  <row Id="6793" PostTypeId="2" ParentId="6762" CreationDate="2011-02-02T02:44:08.977" Score="1" ViewCount="" Body="&lt;p&gt;Below, I briefly sketch out one &quot;back of the envelope&quot; calculation, which is in no way optimal, but does at least show that the probability converges to one exponentially fast. Before getting into the (gory) details, we give the answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;\Pr(\text{at least one actor chooses a unique outcome}) \geq 1 - (e/4)^{n/2} .&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As mentioned before, comparing this result to the above for small $n$ shows that the result is not super sharp. But, it does provide a &lt;em&gt;guarantee&lt;/em&gt; for all $n$ and shows that as $n$ grows, the probability goes to 1 at least &lt;em&gt;exponentially&lt;/em&gt; fast. (The actual rate is probability more like $e^{b n \log n + o(n)}$).&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;The idea is to turn the problem into an occupancy or coupon-collecting problem. If you want a more precise answer, you can refer to the (extensive) literature on these subjects, but to prove a faster rate takes messier analysis. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let $U_n$ be the &lt;em&gt;proportion&lt;/em&gt; of actors that choose a unique outcome (when we have $n$ total actors), which we use below. (Technically, the analysis below is for even $n$, but that's a minor point. The same result holds for odd $n$; just replace $n/2$ by $(n-1)/2$ and $3n/2$ by $(3n+1)/2$ in the relevant places.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Restating the problem in terms of balls and bins. We have $n$ balls and $2n$ bins. The &quot;balls&quot; are the &quot;actors&quot; and the &quot;bins&quot; are the possible outcomes. We want to know the probability that at least one bin has only a single ball in it. The probability that any particular bin has only one ball is $\left(1 - \frac{1}{2n}\right)^n$. So, the expected &lt;em&gt;proportion&lt;/em&gt; of balls in a bin by themselves (i.e., actors that choose a unique outcome) is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;\mathbb{E}(U_n) = \left(1 - \frac{1}{2n}\right)^n \to e^{-1/2} \approx 60.65\%&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This gives us pretty good hope that the probability of at least one ball being in a bin by itself is (very) high.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To get a bound on this probability, note that if each of the &lt;em&gt;occupied&lt;/em&gt; bins has at least two balls in it, then at most $n/2$ bins can be occupied. So&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;\Pr(\text{no bin has exactly 1 ball}) \leq \Pr(\text{no more than n/2 bins are occupied})&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, if no more than $n/2$ bins are occupied, then there &lt;em&gt;must&lt;/em&gt; be some subset of $3n/2$ bins that are &lt;em&gt;all&lt;/em&gt; empty. There are ${2n \choose 3n/2} = {2n \choose n/2}$ such subsets, and so by the union bound, we get&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;\Pr(\text{some subset of 3n/2 bins is empty}) \leq {2n \choose n/2} \left(1 - \frac{3n/2}{2n}\right)^{n} = {2n \choose n/2} 4^{-n}&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To finish up, we use the elementary inequality&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;{2n \choose n/2} \leq \left(\frac{2n e}{n/2}\right)^{n/2} = (4e)^{n/2} .&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Putting it all together, we get&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;\Pr(\text{no bin has exactly 1 ball}) \leq (4 e)^{n/2} 4^{-n} = (e/4)^{n/2} .&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hence,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;\Pr(\text{at least one actor chooses a unique outcome}) \geq 1 - (e/4)^{n/2} .&#xD;&#xA;$$&lt;/p&gt;&#xA;" OwnerUserId="2970" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-02-06T14:48:56.133" LastActivityDate="2011-02-06T14:48:56.133" />
  <row Id="6794" PostTypeId="1" CreationDate="2011-02-02T03:16:45.343" Score="9" ViewCount="215" Body="&lt;p&gt;A few days ago I saw a post on how to setup a SweaveR, which would allow for a user to directly export things like tables, graphs, etc. into Latex. I couldn't quite follow the directions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone give step-by-step  instructions on how to do it on both, Mac and Windows?&lt;/p&gt;&#xA;" OwnerUserId="3008" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-06-05T14:41:45.250" LastActivityDate="2011-06-05T14:41:45.250" Title="Setting up Sweave, R, Latex, Eclipse StatET" Tags="&lt;r&gt;&lt;sweave&gt;&lt;latex&gt;&lt;statet&gt;" AnswerCount="4" CommentCount="3" FavoriteCount="2" />
  <row Id="6795" PostTypeId="1" AcceptedAnswerId="6933" CreationDate="2011-02-02T03:46:06.413" Score="7" ViewCount="269" Body="&lt;p&gt;I'm trying to solve a problem for least angle regression (LAR). This is a problem &lt;strong&gt;3.23&lt;/strong&gt; on page &lt;strong&gt;97&lt;/strong&gt; of &lt;a href=&quot;http://www.stanford.edu/~hastie/local.ftp/Springer/ESLII_print5.pdf#page=115&quot; rel=&quot;nofollow&quot;&gt;Hastie et al., Elements of Statistical Learning, 2nd. ed. (5th printing)&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider a regression problem with all variables and response having mean zero and standard deviation one.  Suppose also that each variable has identical absolute correlation with the response:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$&#xD;&#xA;\frac{1}{N} | \left \langle \bf{x}_j, \bf{y} \right \rangle | = \lambda, j = 1, ..., p&#xD;&#xA;$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let $\hat{\beta}$ be the least squares coefficient of $\mathbf{y}$ on $\mathbf{X}$ and let $\mathbf{u}(\alpha)=\alpha \bf{X} \hat{\beta}$ for $\alpha\in[0,1]$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am asked to show that&#xA;$$&#xD;&#xA;\frac{1}{N} | \left \langle \bf{x}_j, \bf{y}-u(\alpha) \right \rangle | = (1 - \alpha) \lambda, j = 1, ..., p&#xD;&#xA;$$&#xA;and I am having problems with that. Note that this can basically  says that the correlations of each $x_j$ with the residuals remain equal&#xA;in magnitude as we progress toward $u$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also do not know how to show that  the correlations are equal to:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\lambda(\alpha) = \frac{(1-\alpha)}{\sqrt{(1-\alpha)^2 + \frac{\alpha (2-\alpha)}{N} \cdot RSS}} \cdot \lambda$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any pointers would be greatly appreciated!&lt;/p&gt;&#xA;" OwnerUserId="988" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-02-10T07:55:06.183" LastActivityDate="2011-05-24T22:56:27.137" Title="Least angle regression keeps the correlations monotonically decreasing and tied?" Tags="&lt;regression&gt;&lt;machine-learning&gt;&lt;correlation&gt;&lt;homework&gt;" AnswerCount="1" CommentCount="11" FavoriteCount="3" />
  <row Id="6796" PostTypeId="2" ParentId="6791" CreationDate="2011-02-02T04:21:28.807" Score="3" ViewCount="" Body="&lt;p&gt;[Technically you've got survey items, not Likert scales; the latter are fashioned from multiple items.  See, for example, Paul Spector's &lt;em&gt;Summated Rating Scale Construction&lt;/em&gt; {Sage}.]  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The steps you take will need to depend on the audience for which you're reporting.  If it's academic and rigorous, like a dissertation committee, you may face special challenges.  If it's not, and if it's comfortable with the common 1-5 format, why not rescale to fit that and then report means and standard deviations (especially since shapes, skew, and kurtosis are no different from year to year.  I presume the distributions are normal enough that means accurately express central tendency?).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;--&gt;Why am I treating your variables as interval-level ones?  Purists may say that ordinal-level variables should not be reported via means or s.d.  Well, your comments suggest, despite your use of &quot;categorical/ordinal,&quot; that you are dealing with an ordinal level of measurement which you actually feel comfortable treating as interval-level.  After all, why otherwise would you assess skewness or kurtosis.  I'm guessing that your audience, too, will be ok with and will be able to relate to interval-level statistics such as means.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It sounds good that you have already explored the data graphically.  If you want to go beyond assessing the magnitude of the difference and conduct an hypothesis test, why not do a T-test (independent or correlated, depending on your data) comparing the 1-5 scores pre and the 1-5 scores post, and yielding a confidence interval for the mean difference.  Here I'm assuming you've got random samples from a population.&lt;/p&gt;&#xA;" OwnerUserId="2669" LastEditorUserId="2669" LastEditorDisplayName="" LastEditDate="2011-03-31T15:24:12.113" LastActivityDate="2011-03-31T15:24:12.113" CommentCount="1" />
  <row Id="6797" PostTypeId="2" ParentId="6791" CreationDate="2011-02-02T04:48:17.157" Score="5" ViewCount="" Body="&lt;p&gt;This is not a complete answer; just a few points:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;If you can administer both versions of the scale to a subsample, you could estimate what corresponding scores are on the two response formats.&#xA;Then you could apply a conversion formula that is empirically justified.&#xA;I can think of a number of ways of doing this. &#xA;I'd be interested if anyone has an academic paper on best practice for doing this.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If you do a simple rescaling (1 = 1; 2 = 3; 3 = 5; 4 = 7; 5 = 9), there is no guaranty that this is justifiable. &#xA;As a broad statement (at least within my experience in organisational settings) changes in item wording and changes in scale options are likely to have a greater effect on responses than any actual change in the attribute of interest.&#xA;At the very least you should check whether the scale anchors used are roughly equivalent across the two response formats.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastActivityDate="2011-02-02T04:48:17.157" CommentCount="1" />
  <row Id="6798" PostTypeId="1" AcceptedAnswerId="6799" CreationDate="2011-02-02T05:18:40.943" Score="1" ViewCount="46" Body="&lt;p&gt;I have several time series (generated from a numerical model) that go through an initial stage of spinup, followed by a period of dynamic equilibrium, that presumably exists for all times beyond the spinup period. &lt;a href=&quot;http://26.media.tumblr.com/tumblr_lfyplfPp1J1qgj7hyo1_500.png&quot; rel=&quot;nofollow&quot;&gt;See this link for an example.&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While I can usually estimate the spinup time and the equilibrium by inspection of the curves, I would like to know is if there is some standard method from time series analysis for determining whether a time series has passed through some early deterministic stage and has approached a later stage of noise around some mean value (Something like a AR(1) process, if I am using the terminology correctly).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Summary:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there an objective method to determine the point upon which a time series has approached equilibrium (e.g. phrased as a probability from a hypothesis test for each point) and is there similarly an objective way to state the mean value during this equilibrium period?&lt;/p&gt;&#xA;" OwnerUserId="3010" LastActivityDate="2011-02-02T09:04:46.853" Title="Objective measure of relaxation time towards equilibrium for a time series" Tags="&lt;time-series&gt;&lt;mean&gt;" AnswerCount="1" />
  <row Id="6799" PostTypeId="2" ParentId="6798" CreationDate="2011-02-02T09:04:46.853" Score="1" ViewCount="" Body="&lt;p&gt;Since it seems that you have a time trend and then constant mean, &lt;a href=&quot;http://en.wikipedia.org/wiki/KPSS_test&quot; rel=&quot;nofollow&quot;&gt;KPSS test&lt;/a&gt; is appropriate. Here is the example with R code.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x&amp;lt;-rnorm(1000,sd=10)&#xA;y&amp;lt;-c(rep(0.3,200),rep(180,800))+c(rep(0,20),(1:180),rep(0,800))+x&#xA;&#xA;plot(1:1000,y,type=&quot;l&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/g28qI.png&quot; alt=&quot;Example time series&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Test for stationarity&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; kpss.test(y)&#xA;&#xA;    KPSS Test for Level Stationarity&#xA;&#xA;data:  y &#xA;KPSS Level = 5.5237, Truncation lag parameter = 7, p-value = 0.01&#xA;&#xA;Message d'avis :&#xA;In kpss.test(y) : p-value smaller than printed p-value&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The null hypothesis that the series are stationary is rejected. Test the series after the spinoff.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; kpss.test(y[300:1000])&#xA;&#xA;    KPSS Test for Level Stationarity&#xA;&#xA;data:  y[300:1000] &#xA;KPSS Level = 0.1797, Truncation lag parameter = 6, p-value = 0.1&#xA;&#xA;Message d'avis :&#xA;In kpss.test(y[300:1000]) : p-value greater than printed p-value&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The null hypothesis of stationarity is accepted.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is also possible to detect when the change occurs by using &lt;code&gt;breakpoints&lt;/code&gt; function from package strucchange. It is rather slow for large data sets.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; breakpoints(y~t,data=df)&#xA;&#xA;     Optimal 2-segment partition: &#xA;&#xA;Call:&#xA;breakpoints.formula(formula = y ~ t, data = df)&#xA;&#xA;Breakpoints at observation number:&#xA;204 &#xA;&#xA;Corresponding to breakdates:&#xA;204 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In general, testing for stationarity is a complicated procedure, so my approach is not guaranteed to work in general. The main difficulty is that different tests have power over different alternatives. So the most appropriate test should be chosen on the basis what might be the deviation from the stationarity. &lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-02-02T09:04:46.853" CommentCount="2" />
  <row Id="6800" PostTypeId="2" ParentId="6759" CreationDate="2011-02-02T09:27:56.653" Score="5" ViewCount="" Body="&lt;p&gt;&lt;code&gt;unique()&lt;/code&gt; indeed answers your question, but another related and interesting function to achieve the same end is &lt;code&gt;duplicated()&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It gives you the possibility to look up which rows are duplicated.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;a &amp;lt;- c(rep(&quot;A&quot;, 3), rep(&quot;B&quot;, 3), rep(&quot;C&quot;,2))&#xA;b &amp;lt;- c(1,1,2,4,1,1,2,2)&#xA;df &amp;lt;-data.frame(a,b)&#xA;&#xA;duplicated(df)&#xA;[1] FALSE  TRUE FALSE FALSE FALSE  TRUE FALSE  TRUE&#xA;&#xA;&amp;gt; df[duplicated(df), ]&#xA;  a b&#xA;2 A 1&#xA;6 B 1&#xA;8 C 2&#xA;&#xA;&amp;gt; df[!duplicated(df), ]&#xA;  a b&#xA;1 A 1&#xA;3 A 2&#xA;4 B 4&#xA;5 B 1&#xA;7 C 2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="679" LastActivityDate="2011-02-02T09:27:56.653" />
  <row Id="6801" PostTypeId="2" ParentId="6652" CreationDate="2011-02-02T09:31:07.053" Score="2" ViewCount="" Body="&lt;p&gt;I found this thought experiment helpful when thinking about confidence intervals. It also answers your question 3.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let $X\sim U(0,1)$ and $Y=X+a$. Consider two observations of $Y$ taking the values $y_1$ and $y_2$, and let $y_l=\min(y_1,y_2)$ and $y_u=\max(y_1,y_2)$. Then $[y_l,y_u]$ is a 50% confidence interval for $a$ (since the interval includes $a$ if $x_1&amp;lt;1/2$ and $x_2&amp;gt;1/2$ or if $x_1&amp;gt;1/2$ and $x_2&amp;lt;1/2$, each of which has probability $1/4$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, if $y_u-y_l&amp;gt;1/2$ then we know that the probability that the interval contains $\theta$ is $1$, not $1/2$. The subtlety is that a $z\%$ confidence interval for a parameter means that the endpoints of the interval (which are random variables) lie either side of the parameter with probability $z\%$ &lt;em&gt;before you calculate the interval&lt;/em&gt;, not that the probability of the parameter lying within the interval is $z\%$ &lt;em&gt;after you have calculated the interval&lt;/em&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2425" LastActivityDate="2011-02-02T09:31:07.053" CommentCount="1" />
  <row Id="6802" PostTypeId="1" AcceptedAnswerId="6805" CreationDate="2011-02-02T10:24:07.393" Score="4" ViewCount="117" Body="&lt;p&gt;I want to compare few methods of variable selection. I want to do it using simulations. I'm aware of the fact that it won't be an ultimate answer to question 'Which method is the best one?', but I'm looking just form some hint. To do such simulations I need a method to draw a 'random linear model'. Is there any well accepted algorithm of drawing 'random linear model'? By well accepted I mean method that was used for example in some scientific paper.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was thinking about following simple approach:&lt;br&gt;&#xA;1) Choose $n$ and $k$, which denotes number of observations and number of variables.&lt;br&gt;&#xA;2) Generate random matrix $X$ by drawing each element using uniform distribution $(0,1)$.&lt;br&gt;&#xA;3) Generate parameters using uniform distribution $(0,1)$.&lt;br&gt;&#xA;4) Generate residuals using Normal Distribution $(0,\sigma^2)$ for some fixed and arbitrary chosen $\sigma^2$&lt;br&gt;&#xA;5) Calculate $Y=X\beta + \epsilon$&lt;/p&gt;&#xA;" OwnerUserId="1643" LastActivityDate="2011-02-02T16:36:59.593" Title="Concept of a random linear model" Tags="&lt;random-generation&gt;&lt;linear-model&gt;" AnswerCount="2" />
  <row Id="6803" PostTypeId="2" ParentId="6794" CreationDate="2011-02-02T10:47:15.130" Score="2" ViewCount="" Body="&lt;p&gt;For me, I found that Eclipse was overkill for creation of scientific papers. &#xA;So, for Windows, what i did was the following:&#xA;Install Miktex 2.8 (? not sure of version). Make sure that you install Miktex into a directory such as C:\Miktex, as Latex hates file paths with spaces in them. Make sure to select the option to install packages on the fly. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also make sure that R is installed somewhere that Latex can find it i.e. in a path with no spaces.&#xA;I installed TechNix center as my program to write documents in, but there are many others such as WinEdt, eclipse, texmaker, or indeed Emacs. &#xA;Now, make sure that you have \usepackage{Sweave} and usepackage{graphicx} in your preamble. &#xA;As I'm sure you know, you need to put &amp;lt;&gt;= at the start of your R chunk, and end it with @.&#xA;You will need either the package xtable or Hmisc to convert R objects to a latex format.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I like xtable, but you will probably need to do quite a bit of juggling of objects to get it into a form that xtable will accept (lm outputs, data frames, matrices). &#xA;When inserting a table make sure to put the results=tex option into your preamble for the code chunk, and if you need a figure, ensure that the fig=TRUE option is also there. &#xA;You can also only generate one figure per chunk, so just bear that in mind. &#xA;Something to be very careful with is that the R code is at the extreme left of the page, as if it is enclosed in an environment then it will be ignored (this took me a long time to figure out). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You need to save the file as .Rnw - make sure that whatever tex program you use does not append a .tex after this, as this will cause problems. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then either run R CMD Sweave foo.Rnw from the command line, or from within R run Sweave(&quot;foo.Rnw&quot;). Inevitably it will fail at some point (especially if you haven't done this before) so just debug your .Rnw file, rinse and repeat. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If it is the first time you have done this, it may prove easier to code all the R analyses from within r, and then use print statements to insert them into LaTex. I wouldn't recommend this as a good idea though, as if you discover that your datafile has errors at the end of this procedure (as i did last weekend) then you will need to rerun all of your analyses, which if you could properly from within latex from the beginning, can be avoided. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, Sweave computations can take some time, so you may wish to use the R package cacheSweave to save rerunning analyses. Apparently the R package highlight allows for colour coding of R code in documents, but i have not used this. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've never used latex or R on a Mac, so i will leave that explanation to someone else. Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="656" LastActivityDate="2011-02-02T10:47:15.130" />
  <row Id="6804" PostTypeId="2" ParentId="6794" CreationDate="2011-02-02T12:04:48.830" Score="7" ViewCount="" Body="&lt;p&gt;I use Eclipse/StatEt to produce document with Sweave and LaTex, and find Eclipse perfect as an editing environment.I can recomend the following guides;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.splusbook.com/R_Eclipse_StatET.pdf&quot; rel=&quot;nofollow&quot;&gt;Longhow Lam's pdf guide&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://jeromyanglim.blogspot.com/2010/02/getting-started-with-sweave-r-latex.html&quot; rel=&quot;nofollow&quot;&gt;Jeromy Anglim's post&lt;/a&gt; (Note, this includes info on installing the RJ package required for the latest versions of StatEt)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I also use &lt;a href=&quot;http://miktex.org/&quot; rel=&quot;nofollow&quot;&gt;MikTex&lt;/a&gt; on Windows and find everythign works really well once it's setup.  There's a &lt;a href=&quot;http://stackoverflow.com/search?q=%5Br%5D+statet&quot;&gt;few good questions and answers on stackoverflow&lt;/a&gt; as well&lt;/p&gt;&#xA;" OwnerUserId="114" LastActivityDate="2011-02-02T12:04:48.830" />
  <row Id="6805" PostTypeId="2" ParentId="6802" CreationDate="2011-02-02T13:02:11.713" Score="9" ViewCount="" Body="&lt;p&gt;One problem with the approach you outline is that &lt;em&gt;the regressors $x_i$ will (on average) be uncorrelated&lt;/em&gt;, and one situation in which variable selection methods have difficulty is highly correlated regressors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not sure the concept of a 'random' linear model is very useful here, as you have to decide on a probability distribution over your model space, which seems arbitrary. I'd rather think of it as an experiment, and apply the principles of good experimental design.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Postscript&lt;/strong&gt;: Here's one reference but i'm sure there are others:&lt;br&gt;&#xA;Andrea Burton, Douglas G. Altman, Patrick Royston, and Roger L. Holder. The design of simulation studies in medical statistics. &lt;em&gt;Statistics in Medicine&lt;/em&gt; &lt;strong&gt;25&lt;/strong&gt;(24):4279-4292, 2006. &lt;a href=&quot;http://dx.doi.org/10.1002/sim.2673&quot; rel=&quot;nofollow&quot;&gt;DOI:10.1002/sim.2673&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See also this related letter:&lt;br&gt;&#xA;Hakan Demirtas. &lt;em&gt;Statistics in Medicine&lt;/em&gt; &lt;strong&gt;26&lt;/strong&gt;(20):3818-3821, 2007. &lt;a href=&quot;http://dx.doi.org/10.1002/sim.2876&quot; rel=&quot;nofollow&quot;&gt;DOI:10.1002/sim.2876&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just found a commentary on a similar topic:&lt;br&gt;&#xA;G. Maldonado and S. Greenland. The importance of critically interpreting simulation studies. &lt;em&gt;Epidemiology&lt;/em&gt; &lt;strong&gt;8&lt;/strong&gt; (4):453-456, 1997. &lt;a href=&quot;http://www.jstor.org/stable/3702591&quot; rel=&quot;nofollow&quot;&gt;http://www.jstor.org/stable/3702591&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="449" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-02-02T16:28:59.487" LastActivityDate="2011-02-02T16:28:59.487" CommentCount="3" />
  <row Id="6806" PostTypeId="1" AcceptedAnswerId="6812" CreationDate="2011-02-02T14:02:57.167" Score="3" ViewCount="112" Body="&lt;p&gt;I have a dataframe  df as shown below&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  name  position            &#xA;1 HLA   1:1-15            &#xA;2 HLA   1:2-16 &#xA;3 HLA   1:3-17         &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I would like to split the position column into two more columns based on the &quot;:&quot; character such that i get&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;name    seq    position            &#xA;1 HLA   1       1-15            &#xA;2 HLA   1       2-16 &#xA;3 HLA   1       3-17 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So i thought this would do the trick,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df &amp;lt;- transform(df,pos = as.character(position)) &#xA;&#xA;df_split&amp;lt;- strsplit(df$pos, split=&quot;:&quot;)&#xA;&#xA;#found this hack from an old mailing list post&#xA;df &amp;lt;- transform(df, seq_name= sapply(df_split, &quot;[[&quot;, 1),pos2= sapply(df_split, &quot;[[&quot;, 2))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;however I get an error &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Error in strsplit(df$pos, split = &quot;:&quot;) : non-character argument&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What could be wrong?&#xA;How do you achieve this in R. I have simplified my case here, actuality the dataframe runs to over a hundred thousand rows.&lt;/p&gt;&#xA;" OwnerUserId="201" LastActivityDate="2011-02-02T14:57:24.317" Title="Splitting a numeric column for a dataframe" Tags="&lt;r&gt;&lt;dataframe&gt;&lt;dataset&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="6807" PostTypeId="1" AcceptedAnswerId="6808" CreationDate="2011-02-02T14:26:45.833" Score="6" ViewCount="247" Body="&lt;p&gt;Hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have recently graduated with my masters degree on medical and biological modeling, accompanied with engineering mathematics as a background. Even though my education program included a significant amount of courses on mathematical statistics (see below for a list), which I managed with pretty high grades,  I frequently end up completely lost staring down on both theory and applications of statistics. I have to say, compared to &quot;pure&quot; mathematics, statistics really makes little sense to me. Especially the notations and language used by most statisticians (including my past lecturers) is annoyingly convoluted and almost none of the resources I have seen so far (including wikipedia) had simple examples that one could easily relate to, and associate to the theory given...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This being the background; I also realize the bitter reality that I cannot have career as a researcher/engineer without a firm grip on statistics, especially within the field of bioinformatics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was hoping that I could get some tips from more experienced statisticians/mathematicians. How can I overcome this problem I have mentioned above? DO you know of any good resources; such as books, e-books, open courses (via iTunes or OpenCourseware for ex) etc.. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT:&lt;/strong&gt; As I have mentioned I am quite biased (negatively) towards a majority of the literature under general title of statistics, and since I can't buy a number of large (and expensive) coursebooks per branch of statistics, what I would need in terms of a book is something similar to what &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0716783398&quot; rel=&quot;nofollow&quot;&gt;Tipler &amp;amp; Mosca&lt;/a&gt;&#xA;is for Physics, but instead for statistics. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For those who don't know about Tipler; it's a large textbook that covers a wide majority of the subjects that one might encounter during higher studies, and presents them each from basic introduction to slightly deeper in detail. Basically a perfect reference book, bought it during my first year in uni, still use it every once in a while.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;The courses I have taken on statistics:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;a large introduction course, &lt;/li&gt;&#xA;&lt;li&gt;stationary stochastic processes,&lt;/li&gt;&#xA;&lt;li&gt;Markov processes,&lt;/li&gt;&#xA;&lt;li&gt;Monte Carlo methods&lt;/li&gt;&#xA;&lt;li&gt;Survival analysis&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="3014" LastEditorUserId="3014" LastEditorDisplayName="" LastEditDate="2011-02-02T16:20:25.740" LastActivityDate="2011-02-03T18:24:47.533" Title="Making sense out of statistics theory and applications" Tags="&lt;mathematical-statistics&gt;&lt;bioinformatics&gt;&lt;computational-statistics&gt;" AnswerCount="7" FavoriteCount="1" />
  <row Id="6808" PostTypeId="2" ParentId="6807" CreationDate="2011-02-02T14:41:39.990" Score="2" ViewCount="" Body="&lt;p&gt;Hello Posdef:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can completely understand your situation. Even though I am PhD student, I find it hard sometimes to related theory and application. If you are willing to immerse yourself in understanding theory, it is definitely rewarding when you think about real world problems. But the process may be frustrating.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of the many references that I like is Gelman and Hill's &lt;a href=&quot;http://www.stat.columbia.edu/~gelman/arm/&quot; rel=&quot;nofollow&quot;&gt;Data Analysis Using Hierarchical/Multilevel Models&lt;/a&gt;. They avoid the theory where they can express the underlying concept using simulations. It will definitely benefit you as you have experience in MCMC etc. As you say, you are working in bioinformatics, probably Harrell's &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0387952322&quot; rel=&quot;nofollow&quot;&gt;Regression Modeling Strategies&lt;/a&gt; is a great reference too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I will make this a community wiki and let others add to it.&lt;/p&gt;&#xA;" OwnerUserId="1307" LastEditorUserId="1307" LastEditorDisplayName="" LastEditDate="2011-02-02T23:37:15.957" LastActivityDate="2011-02-02T23:37:15.957" CommunityOwnedDate="2011-02-02T14:41:39.990" CommentCount="4" />
  <row Id="6809" PostTypeId="1" AcceptedAnswerId="6811" CreationDate="2011-02-02T14:42:08.477" Score="5" ViewCount="231" Body="&lt;p&gt;I'm considering building MATLAB and R interfaces to &lt;a href=&quot;http://en.wikipedia.org/wiki/Ross_Quinlan&quot; rel=&quot;nofollow&quot;&gt;Ross Quinlan&lt;/a&gt;'s &lt;a href=&quot;http://rulequest.com/download.html&quot; rel=&quot;nofollow&quot;&gt;C5.0&lt;/a&gt; (for those not familiar with it, C5.0 is a decision tree algorithm and software package; an extension of &lt;a href=&quot;http://en.wikipedia.org/wiki/C4.5&quot; rel=&quot;nofollow&quot;&gt;C4.5&lt;/a&gt;), and I am trying to get a sense of the components I would need to write.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The only documentation I found for C5.0 is &lt;a href=&quot;http://www.rulequest.com/see5-win.html&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;, which is a tutorial for See5 (a Windows interface to C5.0?) . The &lt;a href=&quot;http://rulequest.com/GPL/C50.tgz&quot; rel=&quot;nofollow&quot;&gt;tar&lt;/a&gt; file comes with a Makefile, but no Readme files or any additional documentation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;From what I read in the tutorial above, C5.0 uses an ASCII-based representation to handle inputs and outputs, and I am also considering building an interface that passes binary data directly between MATLAB or R and C5.0. Is C5.0's data representation used by any other machine-learning/classification software? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Has anybody tried building a MATLAB or an R interface to ID3, C4.5 or C5.0 before?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="2798" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-02-02T19:14:18.960" LastActivityDate="2011-02-28T14:09:51.923" Title="Building MATLAB and R interfaces to Ross Quinlan's C5.0" Tags="&lt;r&gt;&lt;machine-learning&gt;&lt;matlab&gt;" AnswerCount="4" FavoriteCount="1" />
  <row Id="6810" PostTypeId="2" ParentId="6806" CreationDate="2011-02-02T14:44:34.900" Score="1" ViewCount="" Body="&lt;p&gt;The &quot;trick&quot; is to use &lt;code&gt;do.call&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; a &amp;lt;- data.frame(x = c(&quot;1:1-15&quot;, &quot;1:2-16&quot;, &quot;1:3-17&quot;))&#xA;&amp;gt; a&#xA;       x&#xA;1 1:1-15&#xA;2 1:2-16&#xA;3 1:3-17&#xA;&amp;gt; a$x &amp;lt;- as.character(a$x)&#xA;&amp;gt; a.split &amp;lt;- strsplit(a$x, split = &quot;:&quot;)&#xA;&amp;gt; tmp &amp;lt;-do.call(rbind, a.split)&#xA;&amp;gt; data.frame(a, tmp)&#xA;       x X1   X2&#xA;1 1:1-15  1 1-15&#xA;2 1:2-16  1 2-16&#xA;3 1:3-17  1 3-17&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="307" LastActivityDate="2011-02-02T14:44:34.900" />
  <row Id="6811" PostTypeId="2" ParentId="6809" CreationDate="2011-02-02T14:54:16.900" Score="8" ViewCount="" Body="&lt;p&gt;That sounds like a great idea, especially as the page you link to shows that C5.0 is now under GPL.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have some experience wrapping C/C++ software to R using &lt;a href=&quot;http://dirk.eddelbuettel.com/code/rcpp.html&quot; rel=&quot;nofollow&quot;&gt;Rcpp&lt;/a&gt;; I would be happy to help. &lt;/p&gt;&#xA;" OwnerUserId="334" LastActivityDate="2011-02-02T14:54:16.900" CommentCount="1" />
  <row Id="6812" PostTypeId="2" ParentId="6806" CreationDate="2011-02-02T14:57:24.317" Score="5" ViewCount="" Body="&lt;pre&gt;&lt;code&gt;df_split&amp;lt;- strsplit(as.character(df$position), split=&quot;:&quot;)&#xA;df &amp;lt;- transform(df, seq_name= sapply(df_split, &quot;[[&quot;, 1),pos2= sapply(df_split, &quot;[[&quot;, 2))&#xA;&amp;gt; &#xA;&amp;gt; df&#xA;  name position    pos seq_name pos2&#xA;1  HLA   1:1-15 1:1-15        1 1-15&#xA;2  HLA   1:2-16 1:2-16        1 2-16&#xA;3  HLA   1:3-17 1:3-17        1 3-17&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2129" LastActivityDate="2011-02-02T14:57:24.317" CommentCount="2" />
  <row Id="6813" PostTypeId="2" ParentId="6807" CreationDate="2011-02-02T15:01:24.270" Score="1" ViewCount="" Body="&lt;p&gt;As an alternative to Regression Modeling Strategies, and for a more practical approach, &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/007310874X&quot; rel=&quot;nofollow&quot;&gt;Applied Linear Statistical Models&lt;/a&gt; is very good from my point of view.&lt;/p&gt;&#xA;" OwnerUserId="3016" LastActivityDate="2011-02-02T15:01:24.270" />
  <row Id="6814" PostTypeId="1" CreationDate="2011-02-02T15:07:42.593" Score="7" ViewCount="321" Body="&lt;p&gt;Hello,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a question about the Kullback-Leibler divergence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone explain why the &quot;distance&quot; between the blue density and the &quot;red&quot; density is smaller than the distance between the &quot;green&quot; curve and the &quot;red&quot; one?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/Zrm76.jpg&quot; alt=&quot;Graph of three pdfs&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Best regards,&#xA;Marco&lt;/p&gt;&#xA;" OwnerUserId="3016" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-02-02T18:30:12.660" LastActivityDate="2011-02-03T06:02:09.353" Title="Kullback-Leibler divergence - interpretation" Tags="&lt;kullback-leibler&gt;" AnswerCount="2" CommentCount="5" FavoriteCount="2" />
  <row Id="6815" PostTypeId="2" ParentId="6807" CreationDate="2011-02-02T15:52:41.277" Score="1" ViewCount="" Body="&lt;p&gt;Are you familiar with &lt;em&gt;Bayesian Data Analysis&lt;/em&gt; (by Gelman, Carlin, Stern, and Rubin)? Maybe that's what you need a dose of.&lt;/p&gt;&#xA;" OwnerUserId="1047" LastActivityDate="2011-02-02T15:52:41.277" />
  <row Id="6816" PostTypeId="2" ParentId="6809" CreationDate="2011-02-02T15:54:30.040" Score="4" ViewCount="" Body="&lt;p&gt;Interfacing C/C++ code to MATLAB is pretty straightforward, all you have to do is create a MEX gateway function to handle the conversion of parameters and return parameters.  I have experience in making MEX files to do this sort of thing and would be happy to help.&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2011-02-02T15:54:30.040" CommentCount="1" />
  <row Id="6817" PostTypeId="1" CreationDate="2011-02-02T16:20:15.653" Score="2" ViewCount="77" Body="&lt;p&gt;My intuition says that the third equation must be &quot;the length of the gradient squared less than epsilon&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$x_{k+1} = x_k - f(x_k)$&lt;br&gt;&#xA;$x_{k+1} = x_k + 1$&lt;br&gt;&#xA;$|f(x_k)|^2 &amp;lt; \epsilon$  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I am not sure whether it is the standard form.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How would you write the standard form of the gradient method, and particularly its ending rule?&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="3017" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-02-03T05:07:19.070" LastActivityDate="2011-02-03T05:07:19.070" Title="Optimizing the ending rule of gradient method" Tags="&lt;optimization&gt;" AnswerCount="2" CommentCount="6" />
  <row Id="6818" PostTypeId="2" ParentId="6802" CreationDate="2011-02-02T16:36:59.593" Score="2" ViewCount="" Body="&lt;p&gt;To address @onestop's objection to non-correlated regressors, you could do the following:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Choose $n, k, l$, where $l$ is the number of latent factors. &lt;/li&gt;&#xA;&lt;li&gt;Choose $\sigma_i$, the amount of 'idiosyncratic' volatility in the regressors.&lt;/li&gt;&#xA;&lt;li&gt;Draw a $k \times l$ matrix, $F$, of exposures, uniformly on $(0,1)$. (you may want to normalize to sum 1 across rows of $F$.)&lt;/li&gt;&#xA;&lt;li&gt;Draw a $n \times l$ matrix, $W$, of latent regressors as standard normals.&lt;/li&gt;&#xA;&lt;li&gt;Let $X = W F^\top + \sigma_i E$ be the regressors, where $E$ is an $n\times k$ matrix drawn from a standard normal.&lt;/li&gt;&#xA;&lt;li&gt;Proceed as before: draw $k$ vector $\beta$ uniformly on $(0,1)$.&lt;/li&gt;&#xA;&lt;li&gt;draw $n$ vector $\epsilon$ as a normal with variance $\sigma^2$.&lt;/li&gt;&#xA;&lt;li&gt;Let $y = X\beta + \epsilon$.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="795" LastActivityDate="2011-02-02T16:36:59.593" CommentCount="1" />
  <row Id="6819" PostTypeId="2" ParentId="6814" CreationDate="2011-02-02T17:23:44.250" Score="4" ViewCount="" Body="&lt;p&gt;KL divergence measures how difficult it is to fake one distribution with another one. Assume that you draw an i.i.d. sample of size $n$ from the red distribution and that $n$ is large. It may happen that the empirical distribution of this sample mimicks the blue distribution. This is rare but this &lt;em&gt;may&lt;/em&gt; happen... albeit with a probability which is vanishingly small, and which behaves like $\mathrm{e}^{-nH}$. The exponent $H$ is the KL divergence of the blue distribution with respect to the red one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having said that, I wonder why your KL divergences are ranked in the order you say they are.&lt;/p&gt;&#xA;" OwnerUserId="2592" LastActivityDate="2011-02-02T17:23:44.250" CommentCount="1" />
  <row Id="6820" PostTypeId="1" CreationDate="2011-02-02T18:05:08.037" Score="4" ViewCount="38" Body="&lt;p&gt;Let's say that there is a table like the one below, where for each mutant (A, B, C, etc.) protein or peptide we have a binding and a functional information (true/false) column, and for each amino acid in that protein, we have some kind of amino acid property like hydrophobicity:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Protein Binding Functional     1  2  3  4  5  6  7  8  9 10  &#xA;A          0         0        13 96 39 77 70 94 96 29 22 82    &#xA;B          0         1        94 45  2  2 11 46 50 77  7 99    &#xA;C          0         1        66 71 97 37 14 77 89 92 12 72    &#xA;D          1         1        11  8 94 73 16 53  2 27 54 97    &#xA;E          1         1        31 62 49 51  2 86 91 49 61  7    &#xA;F          1         0         2 42 65 42 54 41 45  9 71 20    &#xA;G          0         0        26 44 56 65 61 43 56 90 70 86    &#xA;H          0         1        54 99 68 64 94 81 85  0 50 84    &#xA;I          1         1        27 52 76 12 46 38 24 74 11 90    &#xA;J          1         1         1 58 77 50 72 51 87 99 47 67  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What statistical tests would you recommend to:  &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;See if the difference between means for each mutant is statistically significant.   &lt;/li&gt;&#xA;&lt;li&gt;See if the the aminoacid property can somewhere be related to binding and functional properties.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="2909" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-02-02T19:06:48.170" LastActivityDate="2011-02-02T19:06:48.170" Title="Statistical analysis of mutant proteins sequences" Tags="&lt;anova&gt;&lt;multiple-comparisons&gt;&lt;genetics&gt;" CommentCount="2" />
  <row Id="6821" PostTypeId="2" ParentId="6814" CreationDate="2011-02-02T18:11:19.850" Score="10" ViewCount="" Body="&lt;p&gt;Because I compute slightly different values of the KL divergence than reported here, let's start with my attempt at reproducing the graphs of these PDFs:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/HRUxe.png&quot; alt=&quot;PDFs for red, blue, and green&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The KL distance from $F$ to $G$ is the expectation, &lt;em&gt;under the probability law&lt;/em&gt; $F$, of the difference in logarithms of their PDFs.  Let us therefore look closely at the &lt;em&gt;log&lt;/em&gt; PDFs.  The values near 0 matter a lot:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/1QMbk.png&quot; alt=&quot;Log PDFs in the interval from 0 to 0.10&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Mathematica&lt;/em&gt; computes that KL(red, blue) = 0.574461 and KL(red, green) = 0.641924.  In the graph it is clear that between 0 and 0.02, approximately, log(green) differs far more from log(red) than does log(blue).  Moreover, in this range there is still substantially large probability density for red: its logarithm is greater than -1 (so the density is greater than about 1/2).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Take a look at the differences in logarithms.  Now the blue curve is the difference log(red) - log(blue) and the green curve is log(red) - log(green).  The KL divergences (w.r.t. red) are the expectations (according to the red pdf) of these functions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/WsfNq.png&quot; alt=&quot;Log PDFs for red:blue and red:green, over the interval from 0 to 0.04&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Note the change in horizontal scale, which now focuses more closely near 0.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Very roughly, it looks like a typical vertical distance between these curves is around 10 over the interval from 0 to 0.02, while a typical value for the red pdf is about 1/2.  Thus, this interval alone should add about 10 * 0.02 /2 = 0.1 to the KL divergences.  This just about explains the difference of .067.  Yes, it's true that the blue logarithms are further away than the green logs for larger horizontal values, but the differences are not as extreme and the red PDF decays quickly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In brief, extreme differences in the left tails of the blue and green distributions, for values between 0 and 0.02, explain why KL(red, green) exceeds KL(red, blue).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Incidentally, KL(blue, red) = 0.454776 and KL(green, red) = 0.254469.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;h3&gt;Code&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Specify the distributions&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;red = GammaDistribution[1/.85, 1];&#xA;green = InverseGaussianDistribution[1, 1/3.];&#xA;blue = InverseGaussianDistribution[1, 1/5.];&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Compute KL&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Clear[kl];&#xA;(* Numeric integation between specified endpoints. *)&#xA;kl[pF_, qF_, l_, u_] := Module[{p, q},&#xA;   p[x_] := PDF[pF, x];&#xA;   q[x_] := PDF[qF, x];&#xA;   NIntegrate[p[x] (Log[p[x]] - Log[q[x]]), {x, l, u},  &#xA;    Method -&amp;gt; &quot;LocalAdaptive&quot;]&#xA;   ];&#xA;(* Integration over the entire domain. *)&#xA;kl[pF_, qF_] := Module[{p, q},&#xA;   p[x_] := PDF[pF, x];&#xA;   q[x_] := PDF[qF, x];&#xA;   Integrate[p[x] (Log[p[x]] - Log[q[x]]), {x, 0, \[Infinity]}]&#xA;   ];&#xA;&#xA;kl[red, blue]&#xA;kl[red, green]&#xA;kl[blue, red, 0, \[Infinity]]&#xA;kl[green, red, 0, \[Infinity]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Make the plots&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Clear[plot];&#xA;plot[{f_, u_, r_}] := &#xA;  Plot[Evaluate[f[#, x] &amp;amp; /@ {blue, red, green}], {x, 0, u}, &#xA;   PlotStyle -&amp;gt; {{Thick, Darker[Blue]}, {Thick, Darker[Red]}, &#xA;     {Thick, Darker[Green]}},&#xA;   PlotRange -&amp;gt; r,&#xA;   Exclusions -&amp;gt; {0},&#xA;   ImageSize -&amp;gt; 400&#xA;   ];&#xA;Table[&#xA;  plot[f], {f, {{PDF, 4, {Full, {0, 3}}}, {Log[PDF[##]] &amp;amp;, &#xA;     0.1, {Full, Automatic}}}}&#xA;  ] // TableForm&#xA;&#xA;Plot[{Log[PDF[red, x]] - Log[PDF[blue, x]], &#xA;  Log[PDF[red, x]] - Log[PDF[green, x]]}, {x, 0, 0.04}, &#xA; PlotRange -&amp;gt; {Full, Automatic}, &#xA; PlotStyle -&amp;gt; {{Thick, Darker[Blue]}, {Thick, Darker[Green]}}]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-02-03T06:02:09.353" LastActivityDate="2011-02-03T06:02:09.353" CommentCount="6" />
  <row Id="6822" PostTypeId="2" ParentId="6807" CreationDate="2011-02-02T19:13:18.257" Score="1" ViewCount="" Body="&lt;p&gt;Everyone learns differently, but I think it's safe to say that examples, examples, examples, help a lot in statistics.    My suggestion would be to learn R (just the basics are enough to help a lot) and then you can try any and every example until your eyes bleed.  You can sort it, fit it, plot it, you name it.   And, since R is geared toward statistics, as you learn R, you'll be learning statistics.   Those books that you listed can then be attacked from a &quot;show me&quot; point of view.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since R is free, and a lot of source material is free, all you need to invest is your time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.mayin.org/ajayshah/KB/R/index.html&quot; rel=&quot;nofollow&quot;&gt;http://www.mayin.org/ajayshah/KB/R/index.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://math.illinoisstate.edu/dhkim/rstuff/rtutor.html&quot; rel=&quot;nofollow&quot;&gt;http://math.illinoisstate.edu/dhkim/rstuff/rtutor.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.cyclismo.org/tutorial/R/&quot; rel=&quot;nofollow&quot;&gt;http://www.cyclismo.org/tutorial/R/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.stat.pitt.edu/stoffer/tsa2/R_time_series_quick_fix.htm&quot; rel=&quot;nofollow&quot;&gt;http://www.stat.pitt.edu/stoffer/tsa2/R_time_series_quick_fix.htm&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.statmethods.net/about/books.html&quot; rel=&quot;nofollow&quot;&gt;http://www.statmethods.net/about/books.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are many good books on R that you can buy, here's one that I've used:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0387954759&quot; rel=&quot;nofollow&quot;&gt;http://www.amazon.com/Introductory-Statistics-R-Peter-Dalgaard/dp/0387954759&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit============&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I forgot to add a couple of links.   If you're using Windows, a good editor to feed R is Tinn-R (someone else can add links for editors on a Mac, or Linux).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.sciviews.org/Tinn-R/&quot; rel=&quot;nofollow&quot;&gt;http://www.sciviews.org/Tinn-R/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://cran.r-project.org/web/packages/TinnR/&quot; rel=&quot;nofollow&quot;&gt;http://cran.r-project.org/web/packages/TinnR/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2775" LastEditorUserId="2775" LastEditorDisplayName="" LastEditDate="2011-02-02T20:50:36.100" LastActivityDate="2011-02-02T20:50:36.100" CommentCount="1" />
  <row Id="6823" PostTypeId="2" ParentId="6809" CreationDate="2011-02-02T20:05:04.330" Score="2" ViewCount="" Body="&lt;p&gt;The C5.0 (Linux) documentation is at &lt;a href=&quot;http://rulequest.com/see5-unix.html&quot; rel=&quot;nofollow&quot;&gt;http://rulequest.com/see5-unix.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="3021" LastActivityDate="2011-02-02T20:05:04.330" />
  <row Id="6824" PostTypeId="1" AcceptedAnswerId="6826" CreationDate="2011-02-02T20:21:45.820" Score="4" ViewCount="280" Body="&lt;p&gt;Can an effect size be calculated for an interaction effect in general and more specifically using the F-statistic and its associated degrees of freedom?  If yes, should this be done and what is the appropriate interpretation of the effect size given the nature of interaction effects?  Initially, I assumed the answer is &quot;no.&quot; However, a quick Google search turned up statements about computing effect sizes for interaction terms.  Any assistance in clarifying this issue will be greatly appreciated!&lt;/p&gt;&#xA;" OwnerUserId="3022" LastActivityDate="2011-02-03T00:02:26.337" Title="Calculating and interpreting effect sizes for interaction terms" Tags="&lt;interaction&gt;" AnswerCount="2" />
  <row Id="6825" PostTypeId="2" ParentId="6817" CreationDate="2011-02-02T20:59:15.827" Score="3" ViewCount="" Body="&lt;p&gt;Interpreting your function $f(x)$ as a (scaled) version of the gradient, your termination rule is equivalent to $|f(x)| &amp;lt; \sqrt{\epsilon}$, &lt;em&gt;i.e.&lt;/em&gt; &quot;terminate when you've taken a step that is too small.&quot; This seems perfectly reasonable. &lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2011-02-02T20:59:15.827" CommentCount="1" />
  <row Id="6826" PostTypeId="2" ParentId="6824" CreationDate="2011-02-02T23:06:25.757" Score="1" ViewCount="" Body="&lt;p&gt;Looking into classic old texts (like Geoffrey Keppel's Design and Analysis:  A Researcher's Handbook and Fredric Wolf's Meta-Analysis: Quantitative Methods for Research Synthesis), I've seen several options, including &lt;em&gt;omega, phi&lt;/em&gt;, and the square of each.  But most widely interpretable and simplest to obtain from most software packages' output is the incremental contribution that the interaction makes to &lt;em&gt;r&lt;/em&gt;-squared.  Partial &lt;em&gt;eta&lt;/em&gt; squared (explained variance not shared with any other predictor in the model) is another option, and in fact for an interaction tested in a sequential model, it should be the same as the increment in &lt;em&gt;r&lt;/em&gt;-squared.  I realize I'm not answering your question about specifically using F and df; if that is that essential to you maybe others can address it.  Wolf shows how to convert &lt;em&gt;F&lt;/em&gt; to &lt;em&gt;r&lt;/em&gt; for the 2-group case only, and I'm not the strongest when it comes to formulas.&lt;/p&gt;&#xA;" OwnerUserId="2669" LastActivityDate="2011-02-02T23:06:25.757" />
  <row Id="6827" PostTypeId="1" AcceptedAnswerId="6831" CreationDate="2011-02-02T23:37:14.350" Score="2" ViewCount="149" Body="&lt;p&gt;Hi, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given a table comprising three columns,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1 2 0.05&#xA;1 3 0.04&#xA;2 3 0.001&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and a matrix with defined dimensions,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mat &amp;lt;- matrix(NA, nrow = 3, ncol = 3)&#xA;&#xA;    [,1] [,2] [,3]&#xA;[1,]   NA   NA   NA&#xA;[2,]   NA   NA   NA&#xA;[3,]   NA   NA   NA&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is there an efficient way to populate the matrix with the entries in the third column of the table with R, without having to iterate over the table, isolate the indices and insert value in a for loop?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="2842" LastActivityDate="2011-02-03T02:21:42.227" Title="Efficient way to populate matrix in R?" Tags="&lt;r&gt;&lt;matrix&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="6828" PostTypeId="2" ParentId="6817" CreationDate="2011-02-02T23:58:42.023" Score="2" ViewCount="" Body="&lt;p&gt;A simplified version of the gradient descent algorithm is as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You begin with $k=0$, $x_k$, $\alpha_k$ and a backtracking parameter $c \in (0,1)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The descent direction is given by $p_k = - \nabla f(x_k)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each iterate is computed as follows, $x_{k+1} = x_k + \alpha p_k$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Remember that $\alpha$ is interpreted as step-length and must satisfy the sufficient descent condition, $f(x_k + p_k ) \leq f(x_k) + c \alpha_k \nabla f(x_k)^T p_k$. The value of $\alpha$ must be calculated using the backtracking procedure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The algorithm is as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data: k=0; x; alpha; c; tol&#xA;begin:&#xA;compute the gradient;&#xA;while( norm(grad) &amp;gt; tol ){&#xA;  compute alpha using backtracking;&#xA;  p = -grad;&#xA;  x = x + alpha*p;&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The parameter &lt;code&gt;tol&lt;/code&gt; is the tolerance for the stopping rule, it basically ensures that your gradient is close enough to zero. If the gradient is close to zero you can proof that your current iteration is close to a local minimum.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope this answers your question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For further reference you may check Numerical Optimization by Nocedal &amp;amp; Wright.   &lt;/p&gt;&#xA;" OwnerUserId="2902" LastEditorUserId="2902" LastEditorDisplayName="" LastEditDate="2011-02-03T00:07:26.593" LastActivityDate="2011-02-03T00:07:26.593" CommentCount="1" />
  <row Id="6829" PostTypeId="2" ParentId="6824" CreationDate="2011-02-03T00:02:26.337" Score="3" ViewCount="" Body="&lt;p&gt;Yes, an effect size for an interaction can be computed, though I don't think I know any measures of effect size that you can compute simply from the F and df values; usually you need various sums-of-squares values to do the computations. If you have the raw data, the &quot;ezANOVA&quot; function in the &lt;a href=&quot;http://cran.r-project.org/web/packages/ez/index.html&quot; rel=&quot;nofollow&quot;&gt;&quot;ez&quot; package&lt;/a&gt; for R will give you &lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pubmed/14664681&quot; rel=&quot;nofollow&quot;&gt;generalized eta square&lt;/a&gt;, a measure of effect size that, unlike partial-eta square, generalizes across design types (eg. within-Ss designs vs between-Ss designs).&lt;/p&gt;&#xA;" OwnerUserId="364" LastActivityDate="2011-02-03T00:02:26.337" />
  <row Id="6830" PostTypeId="2" ParentId="6827" CreationDate="2011-02-03T00:54:32.690" Score="0" ViewCount="" Body="&lt;pre&gt;&lt;code&gt; mat &amp;lt;- matrix(c(1, 1, 2, 2, 3, 3, 0.05, 0.04, 0.001), nrow = 3, ncol = 3)&#xA; mat&#xA;&#xA; mat1 &amp;lt;- matrix(c(rep(NA,6), 0.05, 0.04, 0.001), nrow = 3, ncol = 3)&#xA; mat1&#xA;&#xA; mat2 &amp;lt;- matrix(NA, nrow = 3, ncol = 3)&#xA; mat2&#xA; mat2[,3] &amp;lt;- c(0.05, 0.04, 0.001)&#xA; mat2&#xA;&#xA;&#xA;&#xA; df &amp;lt;- data.frame(x=c(1, 1, 2), y=c(2, 3, 3), z=c(0.05, 0.04, 0.001))&#xA; df&#xA;&#xA; mat3 &amp;lt;- matrix(NA, nrow = 3, ncol =3)&#xA; mat3&#xA; mat3[,3] &amp;lt;- df$z&#xA; mat3&#xA;&#xA; mat4 &amp;lt;- matrix(NA, nrow = 3, ncol =3)&#xA; mat4&#xA; mat4[,3] &amp;lt;- unlist(df[3])&#xA; mat4&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2775" LastEditorUserId="2775" LastEditorDisplayName="" LastEditDate="2011-02-03T02:21:42.227" LastActivityDate="2011-02-03T02:21:42.227" />
  <row Id="6831" PostTypeId="2" ParentId="6827" CreationDate="2011-02-03T01:38:16.820" Score="2" ViewCount="" Body="&lt;p&gt;Generally tables are handled as matrices or arrays and matrix indexing allows two column arguments as arrays so:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; inp.mtx &amp;lt;- as.matrix(inp)&#xA;&amp;gt; mat[inp.mtx[,1:2] ]&amp;lt;- inp.mtx[,3]&#xA;&amp;gt; mat&#xA;     [,1] [,2]  [,3]&#xA;[1,]   NA 0.05 0.040&#xA;[2,]   NA   NA 0.001&#xA;[3,]   NA   NA    NA&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2129" LastActivityDate="2011-02-03T01:38:16.820" />
  <row Id="6832" PostTypeId="1" AcceptedAnswerId="6836" CreationDate="2011-02-03T02:24:22.080" Score="2" ViewCount="87" Body="&lt;p&gt;I'm building a database of economic and political indicators provided by the World Bank and other similar sources. I would like to run uni- and multivariate analysis of these indicators over the past 20 years of data. I'll be using JMP to analyze the data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's an excerpt of the database structure:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ID          | Pop. | GDP | (20ish other columns)&#xA;------------------------------------------------&#xA;Afghanistan | xx   | xx  | xx&#xA;Albania     | xx   | xx  | xx &#xA;Algeria     | xx   | xx  | xx&#xA;Andorra     | xx   | xx  | xx&#xA;etc.        | xx   | xx  | xx&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What would be the best way to input this data so that JMP can deal with all 20 years of data? Should I have 20 separate tables, one per year? Or should I have a column for each year all in one table (&lt;code&gt;GDP_1990&lt;/code&gt;, &lt;code&gt;GDP_1991&lt;/code&gt;, &lt;code&gt;GDP_1992&lt;/code&gt;, etc.)&lt;/p&gt;&#xA;" OwnerUserId="3025" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-02-03T05:54:20.473" LastActivityDate="2011-02-03T05:54:20.473" Title="How should I design a database for multi-year data analysis with JMP?" Tags="&lt;jmp&gt;&lt;dataset&gt;" AnswerCount="1" />
  <row Id="6833" PostTypeId="2" ParentId="6794" CreationDate="2011-02-03T02:39:36.663" Score="1" ViewCount="" Body="&lt;p&gt;I installed this suite quite recently and followed the instructions as per instructions &lt;a href=&quot;http://www.r-bloggers.com/getting-started-with-sweave-r-latex-eclipse-statet-texlipse/&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are links to all required software components required. I use MiKTex for all LaTex components.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are a few pitfalls if you are planning to use 64-bit windows as you will need the additional 64-bit java runtime. This is quite easy to overcome if you go to java.com in a 64-bit IE and verify your installation, it will point you to the 64-bit installer which is otherwise difficult to find.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To avoid mucking around with path variables I simply extracted the eclipse folder in C:\Program Files as this is where java lives and 64-bit R. From here the configuration options in eclipse can easily run automatically and find the appropriate parameters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="1834" LastActivityDate="2011-02-03T02:39:36.663" />
  <row Id="6834" PostTypeId="1" CreationDate="2011-02-03T03:44:59.920" Score="5" ViewCount="101" Body="&lt;p&gt;What are the major machine learning theories that maybe used by Twitter for suggesting followers?&lt;/p&gt;&#xA;" OwnerUserId="3026" LastActivityDate="2011-02-03T15:22:11.077" Title="What are the major machine learning theories that maybe used by Twitter for suggesting followers?" Tags="&lt;machine-learning&gt;" AnswerCount="1" />
  <row Id="6835" PostTypeId="1" CreationDate="2011-02-03T04:02:57.213" Score="0" ViewCount="182" Body="&lt;p&gt;What are the best quantitative models for trend detection?&#xA;I.e. market trend.&lt;/p&gt;&#xA;" OwnerUserId="3027" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-02-03T10:40:11.013" LastActivityDate="2011-03-03T15:32:02.253" Title="Trend detection quantitative models" Tags="&lt;time-series&gt;&lt;trend&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="6836" PostTypeId="2" ParentId="6832" CreationDate="2011-02-03T05:54:02.283" Score="3" ViewCount="" Body="&lt;p&gt;20 separate tables means 20 times as much work for everything you do: forget that!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I notice your &quot;database structure&quot; does not seem to include an attribute for year.  How is that information going to get into JMP?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are two useful structures for these data: &quot;long&quot; and &quot;wide&quot;.  The long structure consists of tuples of (ID, Pop., GDP, ..., Year).  The wide structure lists all years at once in each row: (ID, Pop_1990, ..., Pop_2009, GDP_1990, ..., ...).  Depending on the software, some procedures are easiest (or feasible only) with one structure and other procedures need the other structure.  Typically, tasks that query or sort the data by year (e.g., do separate regressions by year) benefit from the long structure and tasks that make pairwise comparisons (e.g., a scatterplot matrix of GDP over all years) benefit from a wide structure.  You will need both structures.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good stats software supports interconversion between long and wide structures.  It's usually easiest to convert long to wide, and there are powerful reasons on the database management side in favor of long, so as a rule, I start with the long format in all projects.  I can't remember what JMP does--I recall it's somewhat limited--so you might need to enlist the help of a real database management system, which is not a bad idea anyway.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I notice your question is specifically about &lt;em&gt;inputting&lt;/em&gt; data.  That issue is really separate from how the data are to be maintained.  For example, if you are manually transcribing tables, by far the best procedure is to create computer-readable copies of those tables.  This makes checking the input easier and more reliable.  Then write little scripts, if you have to, to assemble, restructure, and clean the input as needed.  In particular, do not let the structure most convenient for getting the data into JMP determine how you will manage the data throughout the project!  By the same token, do not let the format of typical summaries or reports determine the database structure, either.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-02-03T05:54:02.283" CommentCount="3" />
  <row Id="6837" PostTypeId="2" ParentId="6807" CreationDate="2011-02-03T08:54:35.680" Score="1" ViewCount="" Body="&lt;p&gt;All statistics problems essentialy boils to following 4 steps (which I borrowed from @whuber &lt;a href=&quot;http://stats.stackexchange.com/questions/6780/zipfs-law-coefficient/6786#6786&quot;&gt;answer on another question&lt;/a&gt;):&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Estimate the parameter.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Assess the quality of that estimate.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Explore the data.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Evaluate the fit.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;You can exchange word &lt;strong&gt;parameter&lt;/strong&gt; with word &lt;strong&gt;model&lt;/strong&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Statistics books usually present the first two points for various situations. The problem that each real world application requires different approach, hence different model, so a large part of the books end up cataloguing these different models. This has undesired effect that it is easy to lose yourself in the details and miss the big picture. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The big picture book which I heartily recommend is &lt;a href=&quot;http://books.google.com/books?id=UEuQEM5RjWgC&amp;amp;lpg=PR1&amp;amp;dq=asymptotic%20statistics&amp;amp;hl=fr&amp;amp;pg=PR1#v=onepage&amp;amp;q&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;Asymptotic statistics&lt;/a&gt;. It gives a rigorous treatment of the topic and is mathematically &quot;pure&quot;. Though its title mentions asymptotic statistics, the big untold secret is that majority of classical statistics methods are in essence based on asymptotic results. &lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-02-03T08:54:35.680" />
  <row Id="6838" PostTypeId="1" CreationDate="2011-02-03T08:58:53.177" Score="10" ViewCount="108" Body="&lt;p&gt;&lt;strong&gt;The data:&lt;/strong&gt; I have worked recently on analysing the stochastic properties of a spatio-temporal field of wind power production forecast errors. Formally, it can be said to be a process $$ \left (\epsilon^p_{t+h|t} \right )_{t=1\dots,T;\; h=1,\dots,H,\;p=p_1,\dots,p_n}$$ &#xA;indexed twice in time (with $t$ and $h$) and once in space ($p$) with $H$ being the number of look ahead times (equals something around $24$, regularly sampled) , $T$ being the number of &quot;forecast times&quot; (i.e. times at which the forecast is issued, around 30000 in my case, regularly sampled), and $n$ being a number of spatial positions (not gridded, around 300 in my case). Since this is a weather related process, I also have plenty of weather forecast, analysis, meteorological measurments that can be used. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question:&lt;/strong&gt; Can you describe me the exploratory analysis that you would perform on this type of data to understand the nature of the interdependence structure (that might not be linear) of the process in order to propose a fine modelling of it. &lt;/p&gt;&#xA;" OwnerUserId="223" LastEditorUserId="223" LastEditorDisplayName="" LastEditDate="2011-03-13T20:33:09.153" LastActivityDate="2011-03-13T20:33:09.153" Title="Exploratory analysis of spatio-temporal forecast errors" Tags="&lt;forecasting&gt;&lt;data-mining&gt;&lt;stochastic-processes&gt;&lt;spatial&gt;&lt;spatio-temporal&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="4" />
  <row Id="6839" PostTypeId="2" ParentId="6653" CreationDate="2011-02-03T09:30:48.723" Score="8" ViewCount="" Body="&lt;p&gt;Do a logistic regression with covariates &quot;play time&quot; and &quot;goals(home team) - goals(away team)&quot;. You will need an interaction effect of these terms since a 2 goal lead at half-time will have a much smaller effect than a 2 goal lead with only 1 minute left. Your response is &quot;victory (home team)&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Don't just assume linearity for this, fit a smoothly varying coefficient model for the effect of &quot;goals(home team) - goals(away team)&quot;, e.g. in R you could use &lt;code&gt;mgcv&lt;/code&gt;'s &lt;code&gt;gam&lt;/code&gt; function with a model formula  like &lt;code&gt;win_home ~ s(time_remaining, by=lead_home)&lt;/code&gt;. Make &#xA;&lt;code&gt;lead_home&lt;/code&gt; into a factor, so that you get a different effect of &lt;code&gt;time_remaining&lt;/code&gt; for every value of &lt;code&gt;lead_home&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would create multiple observations per game, one for every slice of time you are interested in. &lt;/p&gt;&#xA;" OwnerUserId="1979" LastActivityDate="2011-02-03T09:30:48.723" CommentCount="4" />
  <row Id="6840" PostTypeId="1" AcceptedAnswerId="6850" CreationDate="2011-02-03T09:31:02.100" Score="4" ViewCount="238" Body="&lt;p&gt;Hello everyone!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to be sure I am able to compute the KL divergence based on a sample.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assume the data come from a Gamma distribution with shape=1/.85 and scale=.85. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;set.seed(937) &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;theta &amp;lt;- .85&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;x &amp;lt;- rgamma(1000, shape=1/theta, scale=theta)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Based on that sample, I would like to compute the KL-divergence from the real underlying distribution to an inverse gaussian distribution with mean 1 (mu=1) and precision 0.832 (lambda=0.832).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I obtain KL = 1.286916. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can you confirm I compute it well ?&#xA;Thank you !&lt;/p&gt;&#xA;" OwnerUserId="3019" LastActivityDate="2011-02-03T17:44:15.207" Title="Estimate the Kullback-Leibler divergence" Tags="&lt;kullback-leibler&gt;" AnswerCount="1" />
  <row Id="6841" PostTypeId="1" CreationDate="2011-02-03T10:18:25.350" Score="3" ViewCount="167" Body="&lt;p&gt;Hi All,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have fitted a dynamic panel data model with Arellano-Bond estimator in gretl, here is the output:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Model 5: 2-step dynamic panel, using 2332 observations&#xA;Included 106 cross-sectional units&#xA;H-matrix as per Ox/DPD&#xA;Dependent variable: trvr&#xA;&#xA;              coefficient   std. error     z       p-value &#xA;  ---------------------------------------------------------&#xA;  Dtrvr(-1)    0.895381     0.0248490    36.03    2.55e-284 ***&#xA;  const        0.0230952    0.00226823   10.18    2.39e-024 ***&#xA;  x1          -0.0263556    0.00836633   -3.150   0.0016    ***&#xA;  x2           0.127888     0.0171532     7.456   8.94e-014 ***&#xA;&#xA;Sum squared resid    605.9396   S.E. of regression   0.510180&#xA;&#xA;Number of instruments = 256&#xA;Test for AR(1) errors: z = -4.29161 [0.0000]&#xA;Test for AR(2) errors: z = 1.62503 [0.1042]&#xA;Sargan over-identification test: Chi-square(252) = 105.139 [1.0000]&#xA;Wald (joint) test: Chi-square(3) = 2333.35 [0.0000]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have 2 questions about the results:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;How do I assess the fit?&lt;/li&gt;&#xA;&lt;li&gt;How can I simulate from the model?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Regards&lt;/p&gt;&#xA;" OwnerUserId="1443" LastActivityDate="2011-02-03T10:18:25.350" Title="Simulate Arellano-Bond" Tags="&lt;panel-data&gt;&lt;simulation&gt;&lt;gretl&gt;" CommentCount="10" />
  <row Id="6842" PostTypeId="2" ParentId="6835" CreationDate="2011-02-03T11:44:39.530" Score="2" ViewCount="" Body="&lt;p&gt;Without more detail it's hard to give you a comprehensive response, but you might for example look at the Hurst exponent to detect if a series displays trending characteristics. There are many R packages which compute the Hurst exponent - in my opinion the best collection can be found in the package fArma.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are many methods you could use to detect when a specific series is trending. A simple and on-line method is to take an exponential moving average of lagged returns.&lt;/p&gt;&#xA;" OwnerUserId="2425" LastActivityDate="2011-02-03T11:44:39.530" />
  <row Id="6843" PostTypeId="2" ParentId="5899" CreationDate="2011-02-03T12:51:24.553" Score="0" ViewCount="" Body="&lt;p&gt;A little OT, but one of my favourite nuggets of science is Arrow's theorem, so in case you're not familiar here's the wikipedia page:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Arrow&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Arrow&lt;/a&gt;'s_impossibility_theorem&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And all from a PhD thesis, too. Quite inspiring really. Mine was rubbish.&lt;/p&gt;&#xA;" OwnerUserId="199" LastActivityDate="2011-02-03T12:51:24.553" />
  <row Id="6844" PostTypeId="2" ParentId="6807" CreationDate="2011-02-03T13:00:51.660" Score="1" ViewCount="" Body="&lt;p&gt;I personally loved &lt;a href=&quot;http://statwww.epfl.ch/davison/SM/&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; which had a really good mix of theory and application (with lots of examples). It was a good match with casella and berger for a more theory oriented approach. And for a broad brush overview &lt;a href=&quot;http://books.google.co.uk/books?id=th3fbFI1DaMC&amp;amp;lpg=PP1&amp;amp;ots=eMrRH_Smn2&amp;amp;dq=wasserman%20all%20of%20statistics&amp;amp;pg=PP1#v=onepage&amp;amp;q&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="3033" LastActivityDate="2011-02-03T13:00:51.660" CommentCount="2" />
  <row Id="6845" PostTypeId="2" ParentId="672" CreationDate="2011-02-03T13:13:45.523" Score="1" ViewCount="" Body="&lt;p&gt;Bayes' theorem relates two ideas: probability and likelihood. Probability says: given this model, these are the outcomes. So: given a fair coin, I'll get heads 50% of the time. Likelihood says: given these outcomes, this is what we can say about the model. So: if you toss a coin 100 times and get 88 heads (to pick up on a previous example and make it more extreme), then the likelihood that the fair coin model is correct is not so high.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of the standard examples used to illustrate Bayes' theorem is the idea of testing for a disease: if you take a test that's 95% accurate for a disease that 1 in 10000 of the population have, and you test positive, what are the chances that you have the disease?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The naive answer is 95%, but this ignores the issue that 5% of the tests on 9999 out of 10000 people will give a false positive. So your odds of having the disease are far lower than 95%.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My use of the vague phrase &quot;what are the chances&quot; is deliberate. To use the probability/likelihood language: the probability that the test is accurate is 95%, but what you want to know is the likelihood that you have the disease.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Slightly off topic: The other classic example which Bayes theorem is used to solve in all the textbooks is the Monty Hall problem: You're on a quiz show. There is a prize behind one of three doors. You choose door one. The host opens door three to reveal no prize. Should you change to door two given the chance?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I like the rewording of the question (courtesy of the reference below): you're on a quiz show. There is a prize behind one of a million doors. You choose door one. The host opens all the other doors except door 104632 to reveal no prize. Should you change to door 104632?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My favourite book which discusses Bayes' theorem, very much from the Bayesian perspective, is &quot;Information Theory, Inference and Learning Algorithms &quot;, by David J. C. MacKay. It's a Cambridge University Press book, ISBN-13: 9780521642989. My answer is (I hope) a distillation of the kind of discussions made in the book. (Usual rules apply: I have no affiliations with the author, I just like the book).&lt;/p&gt;&#xA;" OwnerUserId="3034" LastActivityDate="2011-02-03T13:13:45.523" />
  <row Id="6847" PostTypeId="2" ParentId="672" CreationDate="2011-02-03T15:12:13.333" Score="1" ViewCount="" Body="&lt;p&gt;I really like Kevin Murphy's intro the to Bayes Theorem&#xA;&lt;a href=&quot;http://www.cs.ubc.ca/~murphyk/Bayes/bayesrule.html&quot; rel=&quot;nofollow&quot;&gt;http://www.cs.ubc.ca/~murphyk/Bayes/bayesrule.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The quote here is from an economist article:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.cs.ubc.ca/~murphyk/Bayes/economist.html&quot; rel=&quot;nofollow&quot;&gt;http://www.cs.ubc.ca/~murphyk/Bayes/economist.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The essence of the Bayesian approach is to provide a mathematical rule explaining how you should change your existing beliefs in the light of new evidence. In other words, it allows scientists to combine new data with their existing knowledge or expertise. The canonical example is to imagine that a precocious newborn observes his first sunset, and wonders whether the sun will rise again or not. He assigns equal prior probabilities to both possible outcomes, and represents this by placing one white and one black marble into a bag. The following day, when the sun rises, the child places another white marble in the bag. The probability that a marble plucked randomly from the bag will be white (ie, the child's degree of belief in future sunrises) has thus gone from a half to two-thirds. After sunrise the next day, the child adds another white marble, and the probability (and thus the degree of belief) goes from two-thirds to three-quarters. And so on. Gradually, the initial belief that the sun is just as likely as not to rise each morning is modified to become a near-certainty that the sun will always rise.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="2904" LastActivityDate="2011-02-03T15:12:13.333" />
  <row Id="6848" PostTypeId="2" ParentId="2296" CreationDate="2011-02-03T15:18:48.797" Score="4" ViewCount="" Body="&lt;p&gt;Here are a few suggestions, more from the statistics literature, with an eye toward applications in finance:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Geweke, J., &amp;amp; Zhou, G. (1996). Measuring the pricing error of the arbitrage pricing theory. Review of Financial Studies, 9(2), 557. Soc Financial Studies. Retrieved January 29, 2011, from &lt;a href=&quot;http://rfs.oxfordjournals.org/content/9/2/557.abstract&quot; rel=&quot;nofollow&quot;&gt;http://rfs.oxfordjournals.org/content/9/2/557.abstract&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might start here - a detailed discussion of identifiability issues (related to and including the sign indeterminacy you describe)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Aguilar, O., &amp;amp; West, M. (2000). Bayesian Dynamic Factor Models and Portfolio Allocation. Journal of Business &amp;amp; Economic Statistics, 18(3), 338. doi: 10.2307/1392266.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Lopes, H. F., &amp;amp; West, M. (2004). Bayesian model assessment in factor analysis. Statistica Sinica, 14(1), 41â68. Citeseer. Retrieved September 19, 2010, from &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.10.8242&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.10.8242&amp;amp;rep=rep1&amp;amp;type=pdf&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good luck!&lt;/p&gt;&#xA;" OwnerUserId="26" LastActivityDate="2011-02-03T15:18:48.797" />
  <row Id="6849" PostTypeId="2" ParentId="6834" CreationDate="2011-02-03T15:22:11.077" Score="3" ViewCount="" Body="&lt;p&gt;Three different approaches come to my mind&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;decisions based solely on the following / follower lists. If a lot of people you are following, follow a particular person, the chance is high you might be interested in this persons tweet.&lt;/li&gt;&#xA;&lt;li&gt;using links and hashtags. Assuming you link very often to specific websites or use specific links you might be interested in people doing the same.&lt;/li&gt;&#xA;&lt;li&gt;doing some kind of document clustering approaches on the tweets, to figure out who's writing similar things and suggest him or her.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The problem with recommender systems to me is that most machine learning algorithms are just using some kind of similarity measure to give suggestions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Check out &quot;recommender systems&quot; and &quot;document clustering&quot; as search keywords to get some more ideas.&lt;/p&gt;&#xA;" OwnerUserId="2904" LastActivityDate="2011-02-03T15:22:11.077" />
  <row Id="6850" PostTypeId="2" ParentId="6840" CreationDate="2011-02-03T15:32:19.690" Score="3" ViewCount="" Body="&lt;p&gt;&lt;em&gt;Mathematica,&lt;/em&gt; using &lt;em&gt;symbolic&lt;/em&gt; integration (not an approximation!), reports a value that equals 1.6534640367102553437 to 20 decimal digits.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;red = GammaDistribution[20/17, 17/20];&#xA;gray = InverseGaussianDistribution[1, 832/1000];&#xA;kl[pF_, qF_] := Module[{p, q},&#xA;   p[x_] := PDF[pF, x];&#xA;   q[x_] := PDF[qF, x];&#xA;   Integrate[p[x] (Log[p[x]] - Log[q[x]]), {x, 0, \[Infinity]}]&#xA;   ];&#xA;kl[red, gray]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In general, using a small Monte-Carlo sample is inadequate for computing these integrals.  As we saw in &lt;a href=&quot;http://stats.stackexchange.com/q/6814/919&quot;&gt;another thread&lt;/a&gt;, the value of the KL divergence can be dominated by the integral in short intervals where the base PDF is nonzero and the other PDF is close to zero.  A small sample can miss such intervals entirely and it can take a large sample to hit them enough times to obtain an accurate result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Take a look at the Gamma PDF (red, dashed) and the log PDF (gray) for the Inverse Gaussian near 0:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/68GVN.png&quot; alt=&quot;Gamma pdf&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/x47Zd.png&quot; alt=&quot;Log of Inverse Gamma pdf&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this case, the Gamma PDF stays high near 0 while the log of the Inverse Gaussian PDF diverges there.  Virtually all of the KL divergence is contributed by values in the interval [0, 0.05], which has a probability of 3.2% under the Gamma distribution.  The number of elements in a sample of $N = 1000$ Gamma variates which fall in this interval therefore has a Binomial(.032, $N$) distribution.  Its standard deviation equals $.032(1 - .032)/\sqrt{N}$ = 0.55%.  Thus, as a rough estimate, we can't expect your integral to have a relative accuracy much better than (3.2% - 2*0.55%) / 3.2% = give or take around 40%, because the number of times it samples this critical interval has this amount of error.  That accounts for the difference between your result and &lt;em&gt;Mathematica's&lt;/em&gt;.  To get this error down to 1%--a mere two decimal places of precision--you would need to multiply your sample approximately by $(40/1)^2 = 1600$: that is, you would need a few million values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a histogram of &lt;em&gt;the natural logarithms&lt;/em&gt; of 1000 independent estimates of the KL divergence.  Each estimate averages 1000 values randomly obtained from the Gamma distribution.  The correct value is shown as the dashed red line.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/l16Go.png&quot; alt=&quot;Histogram&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Histogram[Log[Table[&#xA; Mean[Log[PDF[red, #]/PDF[gray, #]] &amp;amp; /@ RandomReal[red, 1000]], {i,1,1000}]]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Although &lt;em&gt;on the average&lt;/em&gt; this Monte-Carlo method is unbiased, most of the time (87% in this simulation) the estimate is too low.  To make up for this, on occasion the overestimate can be gross: the largest of these estimates is 18.98. (The wide spread of values shows that the estimate of 1.286916 actually has &lt;em&gt;no&lt;/em&gt; reliable decimal digits!)  Because of this huge skewness in the distribution, the situation is actually much worse than I previously estimated with the simple binomial thought experiment.  The average of these simulations (comprising 1000*1000 values total) is just 1.21, still about 25% less than the true value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For computing the KL divergence in general, you need to use &lt;a href=&quot;http://en.wikipedia.org/wiki/Adaptive_quadrature&quot; rel=&quot;nofollow&quot;&gt;adaptive quadrature&lt;/a&gt; or exact methods.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-02-03T17:44:15.207" LastActivityDate="2011-02-03T17:44:15.207" CommentCount="1" />
  <row Id="6851" PostTypeId="1" CreationDate="2011-02-03T15:45:53.743" Score="2" ViewCount="144" Body="&lt;p&gt;Can the KS3D2 test as suggested by &lt;a href=&quot;http://adsabs.harvard.edu/abs/1987MNRAS.225..155F&quot; rel=&quot;nofollow&quot;&gt;Fasano and Franceschini (1987)&lt;/a&gt; be used when one of the three variables take discrete values between 0-40?  The other two variables are continuous.  &lt;/p&gt;&#xA;" OwnerUserId="3039" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-02-03T16:14:17.177" LastActivityDate="2011-02-03T20:34:22.620" Title="3d Kolmogorov-Smirnov test" Tags="&lt;kolmogorov-smirnov&gt;" AnswerCount="1" />
  <row Id="6852" PostTypeId="1" AcceptedAnswerId="6879" CreationDate="2011-02-03T16:40:05.237" Score="0" ViewCount="159" Body="&lt;p&gt;I have a folder with a hundred comma separated files (CSVs) of which the filename equals a company stock exchange followed by it's symbol, delimited with a &quot;_&quot;, eg: NASDAQ_MSFT.csv&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each file contains historical daily stock information, eg. a csv file looks like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Date,Open,High,Low,Close,Volume&#xA;29-Dec-00,21.97,22.91,21.31,21.69,93999000&#xA;28-Dec-00,22.56,23.12,21.94,22.28,75565800&#xA;27-Dec-00,23.06,23.41,22.50,23.22,66881000&#xA;...&#xA;5-Jan-00,55.56,58.19,54.69,56.91,62712600&#xA;4-Jan-00,56.78,58.56,55.94,56.31,52866600&#xA;3-Jan-00,58.69,59.31,56.00,58.34,51680600&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, I want to analyse this information (analyse the companies as if the company name is a column field rather than a new list of fields). But there are a couple of issues:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Each company has it's own file with the symbol and exchange as filename&lt;/li&gt;&#xA;&lt;li&gt;Some companies start at different dates than others. Eg. some might have a range of 30 days while others have a range of 30 months (every step is still 1 day difference though).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I am using SPSS as my analysis tool. My question is, how can I import these files to perform senseful analystical operations on them?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, I wish to see the average slope of the open price of all companies together, etc.&lt;/p&gt;&#xA;" OwnerUserId="3040" LastActivityDate="2011-02-04T14:35:35.567" Title="Strategy for analysing stock history data from multiple files with SPSS" Tags="&lt;distributions&gt;&lt;spss&gt;" AnswerCount="1" />
  <row Id="6853" PostTypeId="1" AcceptedAnswerId="6858" CreationDate="2011-02-03T18:14:11.723" Score="3" ViewCount="199" Body="&lt;p&gt;Given $X_1$ and $X_2$ normal random variables with correlation coefficient $\rho$, how do I find the correlation between following lognormal random variables $Y_1$ and $Y_2$?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Y_1 = a_1 \exp(\mu_1 T + \sqrt{T}X_1)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Y_2 = a_2 \exp(\mu_2 T + \sqrt{T}X_2)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, if $X_1 = \sigma_1 Z_1$ and $X_2 = \sigma_1 Z_2$, where $Z_1$ and $Z_2$ are standard normals, from the linear transformation property, we get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Y_1 = a_1 \exp(\mu_1 T + \sqrt{T}\sigma_1 Z_1)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Y_2 = a_2 \exp(\mu_2 T + \sqrt{T}\sigma_2 (\rho Z_1 + \sqrt{1-\rho^2}Z_2)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, how to go from here to compute correlation between $Y_1$ and $Y_2$?&lt;/p&gt;&#xA;" OwnerUserId="862" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-02-03T18:28:40.687" LastActivityDate="2011-02-03T20:17:19.980" Title="Correlation of log-normal random variables" Tags="&lt;correlation&gt;&lt;random-variable&gt;&lt;lognormal&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="6854" PostTypeId="2" ParentId="6807" CreationDate="2011-02-03T18:24:47.533" Score="1" ViewCount="" Body="&lt;p&gt;I think the most important thing here is to develop an intuition about statistics and some general statistical concepts.  Perhaps the best way to do this is to have some domain that you can &quot;own.&quot;  This can provide a positive feedback loop where understanding about the domain helps you to understand more about the underlying statistics, which helps you to understand more about the domain, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For me that domain was baseball stats.  I understood that a batter that goes 3 for 4 in a game is not a &quot;true&quot; .750 hitter.  This helps to understand the more general point that the sample data is not the same as the underlying distribution.  I also know he is probably closer to an average player than to a .750 hitter, so this helps to understand concepts like regression to the mean.  From there I can get to full-blown Bayesian inference where my prior probability distribution had a mean of that of the mean baseball player, and I now have 4 new samples with which to update my posterior distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't know what that domain is for you, but I would guess it would be more helpful than a mere textbook.  Examples help to understand the theory, which helps to understand the examples.  A textbook with examples is nice, but unless you can make those examples &quot;yours&quot; then I wonder if you will get enough from them.&lt;/p&gt;&#xA;" OwnerUserId="2485" LastActivityDate="2011-02-03T18:24:47.533" />
  <row Id="6855" PostTypeId="1" CreationDate="2011-02-03T18:55:04.247" Score="3" ViewCount="191" Body="&lt;p&gt;I used cforest and randomForest for a 300 rows and 9 columns dataset and received good (almost overfitted - error equal to zero) results for randomForest and big prediction errors for cforest classifiers. What is the main difference between these two procedures?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I admit that for cforest I used any possible input parameters combination e.g. the best one, but still with big classification errors, was &lt;code&gt;cforest_control(savesplitstats = TRUE, ntree=100, mtry=8, mincriterion=0, maxdepth=400, maxsurrogate = 1)&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For very big datasets (about 10000 rows and 192 columns) randomForest and cforest have almost the same errors (the former slightly better on the same level as radial kernel svms), but for the mentioned small one for my surprise there is no way to improve cforest prediction accuracy...&lt;/p&gt;&#xA;" OwnerUserId="3041" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-02-04T06:56:45.530" LastActivityDate="2011-02-04T06:56:45.530" Title="cforest and randomForest classification prediction error" Tags="&lt;r&gt;&lt;machine-learning&gt;&lt;classification&gt;&lt;random-forest&gt;" CommentCount="8" />
  <row Id="6856" PostTypeId="1" AcceptedAnswerId="6859" CreationDate="2011-02-03T19:00:03.880" Score="10" ViewCount="331" Body="&lt;p&gt;Since regression modeling is often more &quot;art&quot; than science, I often find myself testing many iterations of a regression structure. What are some efficient ways to summarize the information from these multiple model runs in an attempt to find the &quot;best&quot; model? One approach I've used is to put all the models into a list and run &lt;code&gt;summary()&lt;/code&gt; across that list, but I imagine there are more efficient ways to compare?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sample code &amp;amp; models:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ctl &amp;lt;- c(4.17,5.58,5.18,6.11,4.50,4.61,5.17,4.53,5.33,5.14)&#xA;trt &amp;lt;- c(4.81,4.17,4.41,3.59,5.87,3.83,6.03,4.89,4.32,4.69)&#xA;group &amp;lt;- gl(2,10,20, labels=c(&quot;Ctl&quot;,&quot;Trt&quot;))&#xA;weight &amp;lt;- c(ctl, trt)&#xA;&#xA;lm1 &amp;lt;- lm(weight ~ group)&#xA;lm2 &amp;lt;- lm(weight ~ group - 1)&#xA;lm3 &amp;lt;- lm(log(weight) ~ group - 1)&#xA;&#xA;#Draw comparisions between models 1 - 3?&#xA;&#xA;models &amp;lt;- list(lm1, lm2, lm3)&#xA;&#xA;lapply(models, summary)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="696" LastActivityDate="2011-02-03T20:23:39.190" Title="Aggregating results from linear model runs R" Tags="&lt;r&gt;&lt;regression&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="9" />
  <row Id="6858" PostTypeId="2" ParentId="6853" CreationDate="2011-02-03T19:27:54.193" Score="3" ViewCount="" Body="&lt;p&gt;I assume that $X_1\sim N(0,\sigma_1^2)$ and $X_2\sim N(0,\sigma_2^2)$. Denote $Z_i=\exp(\sqrt{T}X_i)$. Then&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align}&#xA;\log(Z_i)\sim N(0,T\sigma_i^2)&#xA;\end{align}&#xA;so $Z_i$ are &lt;a href=&quot;http://en.wikipedia.org/wiki/Lognormal_distribution&quot; rel=&quot;nofollow&quot;&gt;log-normal&lt;/a&gt;. Thus &lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align}&#xA;EZ_i&amp;amp;=\exp\left(\frac{T\sigma_i^2}{2}\right)\\&#xA;var(Z_i)&amp;amp;=(\exp(T\sigma_i^2)-1)\exp(T\sigma_i^2)&#xA;\end{align}&#xA;and&#xA;\begin{align}&#xA;EY_i&amp;amp;=a_i\exp(\mu_iT)EZ_i\\&#xA;var(Y_i)&amp;amp;=a_i^2\exp(2\mu_iT)var(Z_i)&#xA;\end{align}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then using the formula &lt;a href=&quot;http://en.wikipedia.org/wiki/Multivariate_normal_distribution&quot; rel=&quot;nofollow&quot;&gt;for m.g.f of multivariate normal&lt;/a&gt; we have&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align}&#xA;EY_1Y_2&amp;amp;=a_1a_2\exp((\mu_1+\mu_2)T)E\exp(\sqrt{T}X_1+\sqrt{T}X_2)\\&#xA;&amp;amp;=a_1a_2\exp((\mu_1+\mu_2)T)\exp\left(\frac{1}{2}T(\sigma_1^2+2\rho\sigma_1\sigma_2+\sigma_2^2)\right)&#xA;\end{align}&#xA;So&#xA;\begin{align}&#xA;cov(Y_1,Y_2)&amp;amp;=EY_1Y_2-EY_1EY_2\\&#xA;&amp;amp;=a_1a_2\exp((\mu_1+\mu_2)T)\exp\left(\frac{T}{2}(\sigma_1^2+\sigma_2^2)\right)(\exp(\rho\sigma_1\sigma_2T)-1)&#xA;\end{align}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now the correlation of $Y_1$ and $Y_2$ is covariance divided by square roots of variances:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align}&#xA;\rho_{Y_1Y_2}=\frac{\exp(\rho\sigma_1\sigma_2T)-1}{\sqrt{\left(\exp(\sigma_1^2T)-1\right)\left(\exp(\sigma_2^2T)-1\right)}}&#xA;\end{align}&lt;/p&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-02-03T20:17:19.980" LastActivityDate="2011-02-03T20:17:19.980" CommentCount="6" />
  <row Id="6859" PostTypeId="2" ParentId="6856" CreationDate="2011-02-03T19:32:10.660" Score="15" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://tables2graphs.com/doku.php?id=04_regression_coefficients&quot; rel=&quot;nofollow&quot;&gt;Plot them!&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://svn.cluelessresearch.com/tables2graphs/longley.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or, if you must, use tables:&#xA;The &lt;a href=&quot;http://cran.r-project.org/web/packages/apsrtable/index.html&quot; rel=&quot;nofollow&quot;&gt;apsrtable&lt;/a&gt; package or the &lt;code&gt;mtable&lt;/code&gt; function in the &lt;a href=&quot;http://cran.r-project.org/web/packages/memisc/index.html&quot; rel=&quot;nofollow&quot;&gt;memisc&lt;/a&gt; package.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using &lt;code&gt;mtable&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; mtable123 &amp;lt;- mtable(&quot;Model 1&quot;=lm1,&quot;Model 2&quot;=lm2,&quot;Model 3&quot;=lm3,&#xA;     summary.stats=c(&quot;sigma&quot;,&quot;R-squared&quot;,&quot;F&quot;,&quot;p&quot;,&quot;N&quot;))&#xA;&#xA;&amp;gt; mtable123&#xA;&#xA;Calls:&#xA;Model 1: lm(formula = weight ~ group)&#xA;Model 2: lm(formula = weight ~ group - 1)&#xA;Model 3: lm(formula = log(weight) ~ group - 1)&#xA;&#xA;=============================================&#xA;                 Model 1   Model 2   Model 3 &#xA;---------------------------------------------&#xA;(Intercept)      5.032***                    &#xA;                (0.220)                      &#xA;group: Trt/Ctl  -0.371                       &#xA;                (0.311)                      &#xA;group: Ctl                 5.032***  1.610***&#xA;                          (0.220)   (0.045)  &#xA;group: Trt                 4.661***  1.527***&#xA;                          (0.220)   (0.045)  &#xA;---------------------------------------------&#xA;sigma             0.696      0.696     0.143 &#xA;R-squared         0.073      0.982     0.993 &#xA;F                 1.419    485.051  1200.388 &#xA;p                 0.249      0.000     0.000 &#xA;N                20         20        20     &#xA;=============================================&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="375" LastEditorUserId="375" LastEditorDisplayName="" LastEditDate="2011-02-03T19:47:52.257" LastActivityDate="2011-02-03T19:47:52.257" CommentCount="9" />
  <row Id="6860" PostTypeId="1" AcceptedAnswerId="6864" CreationDate="2011-02-03T19:51:29.630" Score="1" ViewCount="216" Body="&lt;p&gt;I am doing some histogram rowstacked plots using the gnuplot command&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plot 'file.dat' using (100.*$2/$6) title column(2), for [i=3:5] '' using (100.*column(i)/column(6)) title column(i)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to change all the colors used for fill the bars in the histogram. For change the first box color, I simply add &lt;em&gt;lt rgb &quot;color&quot;&lt;/em&gt; before the comma, but I don't know how to modify colors in the loop to make each box have a different color.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Modifying the line's style does nothing.&lt;/p&gt;&#xA;" OwnerUserId="3043" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-02-03T22:35:53.690" LastActivityDate="2011-02-03T23:52:06.230" Title="Fill colors in Gnuplot" Tags="&lt;data-visualization&gt;&lt;gnuplot&gt;" AnswerCount="1" />
  <row Id="6861" PostTypeId="2" ParentId="2296" CreationDate="2011-02-03T20:06:33.490" Score="1" ViewCount="" Body="&lt;p&gt;This article deals with Bayesian estimation of dynamical hierarchical factor model:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;E. Moench, S. Ng, S. Potter. Dynamic Hierarchical Factor Models, Federal Reserve Bank of New York, 2009, Report No. 412. &lt;a href=&quot;http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1523787&quot; rel=&quot;nofollow&quot;&gt;link&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Naturally it can be adapted for non-hierarchical case. As usual you will find more references on topic by perusing the references in the article.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-02-03T20:06:33.490" />
  <row Id="6862" PostTypeId="2" ParentId="6856" CreationDate="2011-02-03T20:10:28.257" Score="6" ViewCount="" Body="&lt;p&gt;The following doesn't answer exactly the question. It may give you some ideas, though. It's something I recently did in order to assess the fit of several regression models using one to four independent variables (the dependent variable was in the first column of the df1 dataframe). &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# create the combinations of the 4 independent variables&#xA;library(foreach)&#xA;xcomb &amp;lt;- foreach(i=1:4, .combine=c) %do% {combn(names(df1)[-1], i, simplify=FALSE) }&#xA;&#xA;# create formulas&#xA;formlist &amp;lt;- lapply(xcomb, function(l) formula(paste(names(df1)[1], paste(l, collapse=&quot;+&quot;), sep=&quot;~&quot;)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The contents of as.character(formlist) was&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; [1] &quot;price ~ sqft&quot;                     &quot;price ~ age&quot;                     &#xA; [3] &quot;price ~ feats&quot;                    &quot;price ~ tax&quot;                     &#xA; [5] &quot;price ~ sqft + age&quot;               &quot;price ~ sqft + feats&quot;            &#xA; [7] &quot;price ~ sqft + tax&quot;               &quot;price ~ age + feats&quot;             &#xA; [9] &quot;price ~ age + tax&quot;                &quot;price ~ feats + tax&quot;             &#xA;[11] &quot;price ~ sqft + age + feats&quot;       &quot;price ~ sqft + age + tax&quot;        &#xA;[13] &quot;price ~ sqft + feats + tax&quot;       &quot;price ~ age + feats + tax&quot;       &#xA;[15] &quot;price ~ sqft + age + feats + tax&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then I collected some useful indices &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# R squared&#xA;models.r.sq &amp;lt;- sapply(formlist, function(i) summary(lm(i))$r.squared)&#xA;# adjusted R squared&#xA;models.adj.r.sq &amp;lt;- sapply(formlist, function(i) summary(lm(i))$adj.r.squared)&#xA;# MSEp&#xA;models.MSEp &amp;lt;- sapply(formlist, function(i) anova(lm(i))['Mean Sq']['Residuals',])&#xA;&#xA;# Full model MSE&#xA;MSE &amp;lt;- anova(lm(formlist[[length(formlist)]]))['Mean Sq']['Residuals',]&#xA;&#xA;# Mallow's Cp&#xA;models.Cp &amp;lt;- sapply(formlist, function(i) {&#xA;SSEp &amp;lt;- anova(lm(i))['Sum Sq']['Residuals',]&#xA;mod.mat &amp;lt;- model.matrix(lm(i))&#xA;n &amp;lt;- dim(mod.mat)[1]&#xA;p &amp;lt;- dim(mod.mat)[2]&#xA;c(p,SSEp / MSE - (n - 2*p))&#xA;})&#xA;&#xA;df.model.eval &amp;lt;- data.frame(model=as.character(formlist), p=models.Cp[1,],&#xA;r.sq=models.r.sq, adj.r.sq=models.adj.r.sq, MSEp=models.MSEp, Cp=models.Cp[2,])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The final dataframe was&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;                      model p       r.sq   adj.r.sq      MSEp         Cp&#xA;1                price~sqft 2 0.71390776 0.71139818  42044.46  49.260620&#xA;2                 price~age 2 0.02847477 0.01352823 162541.84 292.462049&#xA;3               price~feats 2 0.17858447 0.17137907 120716.21 351.004441&#xA;4                 price~tax 2 0.76641940 0.76417343  35035.94  20.591913&#xA;5            price~sqft+age 3 0.80348960 0.79734865  33391.05  10.899307&#xA;6          price~sqft+feats 3 0.72245824 0.71754599  41148.82  46.441002&#xA;7            price~sqft+tax 3 0.79837622 0.79446120  30536.19   5.819766&#xA;8           price~age+feats 3 0.16146638 0.13526220 142483.62 245.803026&#xA;9             price~age+tax 3 0.77886989 0.77173666  37884.71  20.026075&#xA;10          price~feats+tax 3 0.76941242 0.76493500  34922.80  21.021060&#xA;11     price~sqft+age+feats 4 0.80454221 0.79523470  33739.36  12.514175&#xA;12       price~sqft+age+tax 4 0.82977846 0.82140691  29640.97   3.832692&#xA;13     price~sqft+feats+tax 4 0.80068220 0.79481991  30482.90   6.609502&#xA;14      price~age+feats+tax 4 0.79186713 0.78163109  36242.54  17.381201&#xA;15 price~sqft+age+feats+tax 5 0.83210849 0.82091573  29722.50   5.000000&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Finally, a Cp plot (using library wle)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/JbQyk.png&quot; alt=&quot;&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="339" LastEditorUserId="339" LastEditorDisplayName="" LastEditDate="2011-02-03T20:23:39.190" LastActivityDate="2011-02-03T20:23:39.190" CommentCount="1" />
  <row Id="6863" PostTypeId="2" ParentId="6851" CreationDate="2011-02-03T20:34:22.620" Score="2" ViewCount="" Body="&lt;p&gt;Kolmogorov-Smirnov 3d test is when you have the sample of 3d vectors. The idea is to compare the sample distribution to a model distribution. So the main question is how does model distribution looks like. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now KS-test compares the cumulative distributions of sample distribution and the model distribution. The 3d test does the same. If you worry about discrete-values look at the bottom of page 6 of your reference. The authors test the behaviour of their statistic when the model distribution (they call it parent distribution) is constant in certain cubes. This means that the the testable distribution is discrete. So the answer to your question seems to be yes, discreteness is not a problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To make sure you can always do some Monte-Carlo simulations where you control the model distribution and choose it to be similar to the one you want to test, and see how the statistic performs.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-02-03T20:34:22.620" />
  <row Id="6864" PostTypeId="2" ParentId="6860" CreationDate="2011-02-03T23:52:06.230" Score="2" ViewCount="" Body="&lt;p&gt;I was able to find the answer by myself. Is pretty easy in fact if you know about the ternary operator in gnuplot.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first thing you should do is create a function for set the color you want for each box. A valid function could be:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;colorfunc(x) = x == 3 ? &quot;#C17D11&quot; : x == 4 ? &quot;#73d216&quot; : &quot;#3465A4&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then, in the for-loop part of the plot command you simply add &lt;em&gt;lt rgb colorfunc(i)&lt;/em&gt;. The plot command should be&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;plot 'file.dat' using (100.*$2/$6)&#xA;  title column(2), for [i=3:5] '' using&#xA;  (100.*column(i)/column(6)) title&#xA;  column(i) lt rgb colorfunc(i)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;and each color box will have the desired color. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I prefer to use HTML code in the color function, but you can use any of the valid ways to specify a color in gnuplot.&lt;/p&gt;&#xA;" OwnerUserId="3043" LastActivityDate="2011-02-03T23:52:06.230" />
  <row Id="6865" PostTypeId="1" CreationDate="2011-02-03T23:57:29.870" Score="5" ViewCount="652" Body="&lt;p&gt;How can I test effects in a Split-Plot ANOVA using suitable model comparisons for use with the &lt;code&gt;X&lt;/code&gt; and &lt;code&gt;M&lt;/code&gt; arguments of &lt;code&gt;anova.mlm()&lt;/code&gt; in R? I'm familiar with &lt;code&gt;?anova.mlm&lt;/code&gt; and Dalgaard (2007)[1]. Unfortunately it only brushes Split-Plot Designs. Doing this in a fully randomized design with two within-subjects factors:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;N  &amp;lt;- 20  # 20 subjects total&#xA;P  &amp;lt;- 3   # levels within-factor 1&#xA;Q  &amp;lt;- 3   # levels within-factor 2&#xA;DV &amp;lt;- matrix(rnorm(N* P*Q), ncol=P*Q)           # random data in wide format&#xA;id &amp;lt;- expand.grid(IVw1=gl(P, 1), IVw2=gl(Q, 1)) # intra-subjects layout of data matrix&#xA;&#xA;library(car)        # for Anova()&#xA;fitA &amp;lt;- lm(DV ~ 1)  # between-subjects design: here no between factor&#xA;resA &amp;lt;- Anova(fitA, idata=id, idesign=~IVw1*IVw2)&#xA;summary(resA, multivariate=FALSE, univariate=TRUE)  # all tests ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The following model comparisons lead to the same results. The restricted model doesn't include the effect in question but all other effects of the same order or lower, the full model adds the effect in question.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;anova(fitA, idata=id, M=~IVw1 + IVw2, X=~IVw2, test=&quot;Spherical&quot;) # IVw1&#xA;anova(fitA, idata=id, M=~IVw1 + IVw2, X=~IVw1, test=&quot;Spherical&quot;) # IVw2&#xA;anova(fitA, idata=id, M=~IVw1 + IVw2 + IVw1:IVw2,&#xA;                      X=~IVw1 + IVw2, test=&quot;Spherical&quot;)          # IVw1:IVw2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;A Split-Splot design with one within and one between-subjects factor:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;idB  &amp;lt;- subset(id, IVw2==1, select=&quot;IVw1&quot;)          # use only first within factor&#xA;IVb  &amp;lt;- gl(2, 10, labels=c(&quot;A&quot;, &quot;B&quot;))               # between-subjects factor&#xA;fitB &amp;lt;- lm(DV[ , 1:P] ~ IVb)                        # between-subjects design&#xA;resB &amp;lt;- Anova(fitB, idata=idB, idesign=~IVw1)&#xA;summary(resB, multivariate=FALSE, univariate=TRUE)  # all tests ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;These are the &lt;code&gt;anova()&lt;/code&gt; commands to replicate the tests, but I don't know why they work. &lt;strong&gt;Why do the tests of the following model comparisons lead to the same results?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;anova(fitB, idata=idB, X=~1, test=&quot;Spherical&quot;) # IVw1, IVw1:IVb&#xA;anova(fitB, idata=idB, M=~1, test=&quot;Spherical&quot;) # IVb&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Two within-subjects factors and one between-subjects factor:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;fitC &amp;lt;- lm(DV ~ IVb)  # between-subjects design&#xA;resC &amp;lt;- Anova(fitC, idata=id, idesign=~IVw1*IVw2)&#xA;summary(resC, multivariate=FALSE, univariate=TRUE)  # all tests ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How do I replicate the results given above with the corresponding model comparisons for use with the &lt;code&gt;X&lt;/code&gt; and &lt;code&gt;M&lt;/code&gt; arguments of &lt;code&gt;anova.mlm()&lt;/code&gt;? What is the logic behind these model comparisons?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: suncoolsu pointed out that for all practical purposes, data from these designs should be analyzed using mixed models. However, I'd still like to understand how to replicate the results of &lt;code&gt;summary(Anova())&lt;/code&gt; with &lt;code&gt;anova.mlm(..., X=?, M=?)&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[1]: &lt;a href=&quot;http://lib.stat.cmu.edu/R/CRAN/doc/Rnews/Rnews_2007-2.pdf&quot; rel=&quot;nofollow&quot;&gt;Dalgaard, P. 2007. New Functions for Multivariate Analysis. R News, 7(2), 2-7.&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1909" LastEditorUserId="1909" LastEditorDisplayName="" LastEditDate="2011-02-07T20:12:30.183" LastActivityDate="2011-02-07T20:12:30.183" Title="Split-Plot ANOVA: model comparison tests in R" Tags="&lt;r&gt;&lt;anova&gt;&lt;multivariate-analysis&gt;&lt;repeated-measures&gt;&lt;split-plot&gt;" AnswerCount="1" CommentCount="3" FavoriteCount="2" />
  <row Id="6866" PostTypeId="2" ParentId="1637" CreationDate="2011-02-04T01:29:59.663" Score="1" ViewCount="" Body="&lt;p&gt;One obvious point that everyone's overlooked:  With ANOVA you're testing the null that the mean is identical regardless of the values of your explanatory variables.  With a T-Test you can also test the one-sided case, that the mean is specifically greater given one value of your explanatory variable than given the other.&lt;/p&gt;&#xA;" OwnerUserId="1347" LastActivityDate="2011-02-04T01:29:59.663" />
  <row Id="6867" PostTypeId="1" CreationDate="2011-02-04T01:39:21.263" Score="0" ViewCount="232" Body="&lt;p&gt;Hey everyone, thanks for taking the time to look at this question. This is pretty simple, and I understand statistics quite well, but I think I'm not wrapping my head around the words, it seems to me as if there's something missing. Here's the question.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The manager of the aerospace division&#xA;  of General Aeronautics has estimated&#xA;  the price it can charge for providing&#xA;  satellite launching services to&#xA;  commercial firms. Her most optimistic&#xA;  estimate (a price not expected to be&#xA;  exceeded more than 10 percent of the&#xA;  time) is 2 million. Her most&#xA;  pessimistic estimate (a lower price&#xA;  than this one is not expected more&#xA;  than 10 percent of the time) is 1&#xA;  million. The expected value estimate&#xA;  is 1.5 million. The price distribution&#xA;  is believed to be approximately&#xA;  normal.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;What is the expected price?&lt;br&gt;&#xA;  What is&#xA;  the std dev of the launch price?&lt;br&gt;&#xA;  What&#xA;  is the probability of receiving a&#xA;  price less than 1.2 million?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So what is the actual probability of a 2 million dollar price? It just says that they don't expect to exceed 2 million more than 10% of the time. Does that mean 90% of the time they expect the price to be less than 2 million? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for your suggestions.&lt;/p&gt;&#xA;" OwnerUserId="2644" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-21T19:59:06.003" LastActivityDate="2011-05-21T20:48:51.000" Title="Fitting normal distribution to given quantiles" Tags="&lt;homework&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="6868" PostTypeId="2" ParentId="6867" CreationDate="2011-02-04T02:13:30.190" Score="0" ViewCount="" Body="&lt;p&gt;actually i think i answered it, i was over complicating it. the probability under thethe std normal curve associated with a greater value than 2 mill is 10%. So the z value associated with 90% (to the left of 2mill) is 1.285. So;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1.285=(value-mean)/std dev The mean is given as 1.5 which is also the expected price right? So the std dev works to be .389. This means the probability of getting less than 1.2mill 22.9%. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sound right to anyone?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="2644" LastActivityDate="2011-02-04T02:13:30.190" CommentCount="1" />
  <row Id="6869" PostTypeId="2" ParentId="6865" CreationDate="2011-02-04T02:36:57.687" Score="6" ViewCount="" Body="&lt;p&gt;Split-plot designs originated in agriculture, hence the name. But they frequently occur and I would say -- the workhorse of most of the clinical trials. The main plot is treated with a level of one factor while the levels of some other factor are allowed to vary with the subplots. The design arises as a result of restrictions on a full randomization. For example: a field may be divided into four subplots. It may be possible to plant different varieties in subplots, but only one type of irrigation may be used for the whole field. Not the distinction between &lt;em&gt;splits&lt;/em&gt; and &lt;em&gt;blocks&lt;/em&gt;. Blocks are features of the experimental units which we have the option to take advantage of in the experimental design, because we know they are there. Splits, on the other hand, impose restriction on what assignments of factors are possible. They impose requirements on the design that prevent a complete randomization. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;They are used a lot in clinical trials where when one factor is easy to change while another factor takes much more time to change. If the experimenter must do all runs for each level of the hard-to-change factor consecutively, a split plot design results with the hard-to-change factor representing the whole plot factor.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an example: In an agricultural field trial, the objective was to determine the effects of two crop varieties and four different irrigation methods. Eight fields were available, but only one type of irrigation may be applied to each field. The fields may be divided into two parts with a different variety in each part. The whole plot factor is the irrigation, which should be randomly assigned to fields. Within each field, the variety is assigned.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is how you do this in &lt;code&gt;R&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;install.packages(&quot;faraway&quot;)&#xA;data(irrigation)&#xA;summary(irrigation)&#xA;&#xA;library(lme4)&#xA;&#xA;R&amp;gt; (lmer(yield ~ irrigation * variety + (1|field), data = irrigation))&#xA;Linear mixed model fit by REML &#xA;Formula: yield ~ irrigation * variety + (1 | field) &#xA;   Data: irrigation &#xA;  AIC  BIC logLik deviance REMLdev&#xA; 65.4 73.1  -22.7     68.6    45.4&#xA;Random effects:&#xA; Groups   Name        Variance Std.Dev.&#xA; field    (Intercept) 16.20    4.02    &#xA; Residual              2.11    1.45    &#xA;Number of obs: 16, groups: field, 8&#xA;&#xA;Fixed effects:&#xA;                       Estimate Std. Error t value&#xA;(Intercept)               38.50       3.02   12.73&#xA;irrigationi2               1.20       4.28    0.28&#xA;irrigationi3               0.70       4.28    0.16&#xA;irrigationi4               3.50       4.28    0.82&#xA;varietyv2                  0.60       1.45    0.41&#xA;irrigationi2:varietyv2    -0.40       2.05   -0.19&#xA;irrigationi3:varietyv2    -0.20       2.05   -0.10&#xA;irrigationi4:varietyv2     1.20       2.05    0.58&#xA;&#xA;Correlation of Fixed Effects:&#xA;            (Intr) irrgt2 irrgt3 irrgt4 vrtyv2 irr2:2 irr3:2&#xA;irrigation2 -0.707                                          &#xA;irrigation3 -0.707  0.500                                   &#xA;irrigation4 -0.707  0.500  0.500                            &#xA;varietyv2   -0.240  0.170  0.170  0.170                     &#xA;irrgtn2:vr2  0.170 -0.240 -0.120 -0.120 -0.707              &#xA;irrgtn3:vr2  0.170 -0.120 -0.240 -0.120 -0.707  0.500       &#xA;irrgtn4:vr2  0.170 -0.120 -0.120 -0.240 -0.707  0.500  0.500&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Basically, what this model says is, irrigation and variety are fixed effects and variety is nested within irrigation. The fields are the random effects and pictorially will be something like&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;h2&gt;  I_1   |   I_2    |   I_3   |  I_4&lt;/h2&gt;&#xA;  &#xA;  &lt;h2&gt;V_1  V_2  |  V_1  V_2  |  V_1  V_2 | V_1  V_2&lt;/h2&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;But this was a special variant with fixed whole plot effect and subplot effect. There can be variants in which one or more are random. There can be more complicated designs like split-split .. plot designs. Basically, you can go &lt;em&gt;wild&lt;/em&gt; and crazy. But given the underlying structure and distribution ( ie fixed or random, nested or crossed, .. ) is clearly understood, a &lt;code&gt;lmer-Ninja&lt;/code&gt; will have no troubles in modeling. May be interpretation will be a mess.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regarding comparisons, say you have &lt;code&gt;lmer1&lt;/code&gt; and &lt;code&gt;lmer2&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;anova(lmer1, lmer2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;will give you the appropriate test based on the &lt;em&gt;chi-sq test statistic&lt;/em&gt;  with degrees of freedom equal to the difference of parameters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;cf: &#xA;Faraway, J., Extending Linear Models with R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Casella, G., Statistical Design&lt;/p&gt;&#xA;" OwnerUserId="1307" LastEditorUserId="1307" LastEditorDisplayName="" LastEditDate="2011-02-04T05:53:02.263" LastActivityDate="2011-02-04T05:53:02.263" CommentCount="1" />
  <row Id="6870" PostTypeId="1" AcceptedAnswerId="7461" CreationDate="2011-02-04T02:47:18.727" Score="5" ViewCount="134" Body="&lt;p&gt;I have a question on how to fit a censoring problem in JAGS.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I observe a bivariate mixture normal where the X values have measurement error.  I would like to model the true underlying 'means' of the observed censored values.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align*}&#xA;\lceil x_{true}+\epsilon \rceil = x_{observed} \&#xA;\epsilon \sim N(0,sd=.5)&#xA;\end{align*}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is what I have now:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; for (i in 1:n){&#xA;   x[i,1:2]~dmnorm(mu[z[i],1:2], tau[z[i],1:2,1:2])&#xA;   z[i]~dcat(prob[ ])&#xA; }&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Y also has measurement error.  What I want to do is something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; for (i in 1:n){&#xA;   x_obs[i] ~ dnorm(x_true[i],prec_x)I(x_true[i],)&#xA;   y_obs[i] ~ dnorm(y_true[i],prec_y)&#xA;   c(x_true[i]:y_true[i])~dmnorm(mu[ z [ i ],1:2], tau[z[i],1:2,1:2])&#xA;   z[i]~dcat(prob[ ])&#xA; }&#xA;&#xA; #priors for measurement error&#xA; e_x~dunif(.1,.9)&#xA; prec_x&amp;lt;-1/pow(e_x,2)&#xA; e_y~dunif(2,4)&#xA; prec_y&amp;lt;-1/pow(e_y,2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Obviously the c command is not valid in JAGS.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="2310" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2011-02-15T18:24:29.840" LastActivityDate="2011-02-21T18:53:41.467" Title="Censoring/Truncation in JAGS" Tags="&lt;mcmc&gt;&lt;censoring&gt;&lt;truncation&gt;&lt;jags&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="1" />
  <row Id="6871" PostTypeId="2" ParentId="6870" CreationDate="2011-02-04T04:14:18.500" Score="4" ViewCount="" Body="&lt;p&gt;Perhaps this is what you are looking for:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x_obs[i] ~ dnorm(x_true[i],prec_x)T(x_true[i], )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;JAGS has options for both censoring and truncation. It sounds like you want truncation, since you know &lt;em&gt;a-priori&lt;/em&gt; that the observation lies within a particular range &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Read the &lt;a href=&quot;http://sourceforge.net/projects/mcmc-jags/files/Manuals/2.x/jags_user_manual.pdf/download&quot; rel=&quot;nofollow&quot;&gt;user's manual&lt;/a&gt; for more details about how jags uses truncation and censoring.&lt;/p&gt;&#xA;" OwnerUserId="1381" LastEditorUserId="1381" LastEditorDisplayName="" LastEditDate="2011-02-21T18:21:36.643" LastActivityDate="2011-02-21T18:21:36.643" />
  <row Id="6872" PostTypeId="2" ParentId="6773" CreationDate="2011-02-04T04:27:12.527" Score="1" ViewCount="" Body="&lt;p&gt;I have worked this out myself. &#xA;The lower bound for phi can be estiamted from &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;-ln(0.5)/(max separating distance between points)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To find the max separating distance I used the following code in R. My data are in a flat file with x and y coords renamed to long and lat respectively:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data &amp;lt;- read.csv(file=&quot;file.csv&quot;, header=T, sep=&quot;,&quot;)&#xA;coords &amp;lt;- data.frame(data$long,data$lat) &#xA;library(sp)&#xA;pointDist &amp;lt;- apply(coords, 1, function(eachPoint) spDistsN1(as.matrix(coords), eachPoint, longlat=TRUE))&#xA;distances &amp;lt;- as.vector(pointDist)&#xA;max(distances)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1834" LastActivityDate="2011-02-04T04:27:12.527" />
  <row Id="6873" PostTypeId="2" ParentId="6838" CreationDate="2011-02-04T10:48:43.283" Score="2" ViewCount="" Body="&lt;p&gt;It seems to me that you have enough data to model the dependence on space-time and meteorological influences of both the bias of forecast errors (i.e. tendency to systematically over-/underestimate [first moment]) and their variance [second moment].&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For exploration of the bias, I'd just do a lot of scatterplots, heatmaps or hexbin plots.&#xA;For exploration of the variability, I'd just square the original errors and then again do a lot of scatterplots, heatmaps or hexbin plots. This is of course not entirely unproblematic if you have lots of bias, but it may still help to see patterns of covariate-influenced heteroskedasticity. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Colleagues of mine did a nice techreport that details a very flexible method for fitting these kind of models (also allows for modelling of higher moments, if necessary) that also has a good &lt;code&gt;R&lt;/code&gt;-implementation &lt;a href=&quot;https://r-forge.r-project.org/projects/gamboostlss/&quot; rel=&quot;nofollow&quot;&gt;gamboostLSS&lt;/a&gt; based on &lt;code&gt;mboost&lt;/code&gt;: &lt;a href=&quot;http://epub.ub.uni-muenchen.de/11938/&quot; rel=&quot;nofollow&quot;&gt;Mayr, Andreas; Fenske, Nora; Hofner, Benjamin; Kneib, Thomas and Schmid, Matthias (2010): GAMLSS for high-dimensional data – a flexible approach based on boosting.&lt;/a&gt;. Assuming you have access to machines with a lot of RAM (your datasets seems to be BIG), you can estimate all kinds of semiparametric effects (like smooth surface estimators for spatial effects or the joint effect of $t$ and $h$, tensor product splines for tempo-spatial effects or smooth interactions of meteorological effects etc..) for the different moments and perform term selection at the same time in order to get a parsimonious and interpretable model. The hope would be that the terms in this model are sufficient to account for the spatio-temporal autocorrelation structure of the forecast errors, but you should probably check the residuals of these models for autocorrelation (i.e. look at some variograms and ACFs). &lt;/p&gt;&#xA;" OwnerUserId="1979" LastActivityDate="2011-02-04T10:48:43.283" CommentCount="1" />
  <row Id="6874" PostTypeId="1" AcceptedAnswerId="6884" CreationDate="2011-02-04T12:01:50.607" Score="5" ViewCount="204" Body="&lt;p&gt;There are well-known on-line formulas for computing exponentially weighted moving averages and standard deviations of a process $(x_n)_{n=0,1,2,\dots}$. For the mean,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\mu_n = (1-\alpha) \mu_{n-1} + \alpha x_n$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and for the variance&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\sigma_n^2 = (1-\alpha) \sigma_{n-1}^2 + \alpha(x_n - \mu_{n-1})(x_n - \mu_n)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;from which you can compute the standard deviation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there similar formulas for on-line computation of exponential weighted third- and fourth-central moments? My intuition is that they should take the form&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$M_{3,n} = (1-\alpha) M_{3,n-1} + \alpha f(x_n,\mu_n,\mu_{n-1},S_n,S_{n-1})$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$M_{4,n} = (1-\alpha) M_{4,n-1} + \alpha f(x_n,\mu_n,\mu_{n-1},S_n,S_{n-1},M_{3,n},M_{3,n-1})$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;from which you could compute the skewness $\gamma_n = M_{3,n} / \sigma_n^3$ and the kurtosis $k_n = M_{4,n}/\sigma_n^4$ but I've not been able to find simple, closed-form expression for the functions $f$ and $g$.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt; Some more information. The updating formula for moving variance is a special case of the formula for the exponential weighted moving covariance, which can be computed via&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$C_n(x,y) = (1-\alpha) C_{n-1}(x,y) + \alpha (x_n - \bar{x}_n) (y_n - \bar{y}_{n-1})$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\bar{x}_n$ and $\bar{y}_n$ are the exponential moving means of $x$ and $y$. The asymmetry between $x$ and $y$ is illusory, and disappears when you notice that $y-\bar{y}_n = (1-\alpha) (y-\bar{y}_{n-1})$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Formulas like this can be computed by writing the central moment as an expectation $E_n(\cdot)$, where weights in the expectation are understood to be exponential, and using the fact that for any function $f(x)$ we have&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$E_n(f(x)) = \alpha f(x_n) + (1-\alpha) E_{n-1}(f(x))$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's easy to derive the updating formulas for the mean and variance using this relation, but it's proving to be more tricky for the third and fourth central moments.&lt;/p&gt;&#xA;" OwnerUserId="2425" LastEditorUserId="2425" LastEditorDisplayName="" LastEditDate="2011-02-04T12:49:50.233" LastActivityDate="2011-02-04T15:38:21.347" Title="Exponential weighted moving skewness/kurtosis" Tags="&lt;moments&gt;&lt;online&gt;&lt;kurtosis&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="6875" PostTypeId="2" ParentId="4350" CreationDate="2011-02-04T12:53:22.900" Score="1" ViewCount="" Body="&lt;p&gt;check this out and all other questions asked its author on this exchange. thanks. &lt;a href=&quot;http://stats.stackexchange.com/questions/2981/interpreting-output-of-igraphs-fastgreedy-community-clustering-method&quot;&gt;Weighted Community clustering&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="3038" LastActivityDate="2011-02-04T12:53:22.900" />
  <row Id="6876" PostTypeId="1" AcceptedAnswerId="6877" CreationDate="2011-02-04T13:09:58.190" Score="5" ViewCount="343" Body="&lt;p&gt;I've recently been looking for top of the line statisticians in a recruiting process for our company. Myself I'm a Physics Engineering major. I gather that great mathematical statisticians have studied a bit different courses, and much more in depth. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When evaluating a candidate, and courses are good indicators of this person being excellent?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Preferably we're talking graduate or post-graduate level.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;edit: We're looking to fill roles of data miners, statistical modeling and data visualization. Thx Chris for the suggestion to clarify.&lt;/p&gt;&#xA;" OwnerUserId="3048" LastEditorUserId="3048" LastEditorDisplayName="" LastEditDate="2011-02-04T13:51:02.023" LastActivityDate="2011-02-21T19:35:32.613" Title="What university level statistics courses are considered advanced/hard?" Tags="&lt;homework&gt;" AnswerCount="3" CommentCount="4" FavoriteCount="3" />
  <row Id="6877" PostTypeId="2" ParentId="6876" CreationDate="2011-02-04T13:24:32.357" Score="8" ViewCount="" Body="&lt;p&gt;It really depends what your company is doing. Are you looking for machine learning experts? Data visualisation experts? Data mining experts?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I interview statistics PhDs I like to ask them questions about linear regression, as I feel that anyone claiming to be an expert in statistics should at the very minimum be able to explain linear regression to me, and it's surprising how many can't.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Apart from that I'd consider it to be a good sign if they can have a good discussion about model selection/validation procedures, the concept of training and validation sets, cross-validation etc. If they know about classification algorithms (k-NN, SVM, decision trees etc) and can discuss their strengths/weaknesses that's even better.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I find that the particular courses they've studied are rarely a good indicator, and are only really useful for steering the discussion in the interview. If they're claiming to have studied something on their CV, I expect them to be able to discuss it at length.&lt;/p&gt;&#xA;" OwnerUserId="2425" LastActivityDate="2011-02-04T13:24:32.357" CommentCount="4" />
  <row Id="6878" PostTypeId="2" ParentId="6874" CreationDate="2011-02-04T13:28:52.210" Score="0" ViewCount="" Body="&lt;p&gt;I think that the following updating formula works for the third moment, although I'd be glad to have someone check it:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$M_{3,n} = (1-\alpha)M_{3,n-1} + \alpha \Big[ x_n(x_n-\mu_n)(x_n-2\mu_n) - x_n\mu_{n-1}(\mu_{n-1}-2\mu_n) - \dots$&#xA;$\dots - \mu_{n-1}(\mu_n-\mu_{n-1})^2 - 3(x_n-\mu_n) \sigma_{n-1}^2 \Big]$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Updating formula for the kurtosis still open...&lt;/p&gt;&#xA;" OwnerUserId="2425" LastActivityDate="2011-02-04T13:28:52.210" />
  <row Id="6879" PostTypeId="2" ParentId="6852" CreationDate="2011-02-04T14:35:35.567" Score="2" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Getting the data into SPSS&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Typically when you organize your data, the rows in the data matrix correspond to the units of analysis, and the columns correspond to attributes of those units. Given the description of your data, you have basically three possible units of analysis, companies, days, and company*days. Given the format of your CSV files it will be easiest to either import the data as &quot;days&quot; or &quot;company*days&quot;. I'm going to suggest you import it as &quot;company*days&quot; as it is easier to transform the data to the other units of analysis later on if needed in that format.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So to import the data, you can use this syntax (just replace with the right variable names and formats). I just generated this example through the GUI.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;GET DATA&#xA;  /TYPE=TXT&#xA;  /FILE='H:\NASDAQ_MSFT.csv'&#xA;  /DELCASE=LINE&#xA;  /DELIMITERS=&quot;,&quot;&#xA;  /ARRANGEMENT=DELIMITED&#xA;  /FIRSTCASE=2&#xA;  /IMPORTCASE=ALL&#xA;  /VARIABLES=&#xA;  time F3.0&#xA;  panel F1.0&#xA;  price F16.15.&#xA;EXECUTE.&#xA;Dataset Name NASDAQ_MSFT.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I would then make a variable that identifies what company the dataset correponds to.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;string company (A100).&#xA;compute company = &quot;NASDAQ_MSFT&quot;.&#xA;execute.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;After you have loaded the csv files into SPSS datasets, you can then concatenate them together with the add files command. Note this adds new rows to the dataset, not columns, hence the need to make a variable that identifies each company.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ADD FILES file = 'NASDAQ_MSFT'&#xA;/FILE = 'OTHER_DATASET'&#xA;/FILE = 'OTHER_DATASET'.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This will obviously be tedious, but if it only needs to be done once I would just write a macro to semi automate the job. If this needs to be done periodically I would suggest posting the question to one of the other SPSS forums (links provided in the &lt;a href=&quot;http://stats.stackexchange.com/tags/spss/info&quot;&gt;SPSS tag wiki&lt;/a&gt;), I wouldn't be surprised in someone has already made a tool via Python to accomplish this (i.e. getting all the files in one directory and doing whatever you want with them).&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Analyzing the data&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To be able to give useful advice you need to be able to formulate more specific questions. Different hypotheses would require different answers to your second question. Below I have posted a simulated example getting the average slope against time for all companies (via the split file command) and two plotting examples I cooked up with the GUI. If they are not clear enough just say so in a comment and I will add further explanation. For documentation on SPSS's graph language and other syntax see &lt;a href=&quot;http://support.spss.com/productsext/statistics/documentation/19/clientindex.html&quot; rel=&quot;nofollow&quot;&gt;this page&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;*SIMULATING DATA.&#xA;input program.&#xA;loop #company = 1 to 10.&#xA;loop time = 1 to 100.&#xA;compute company = #company.&#xA;compute price = RV.NORMAL(#company,.10).&#xA;variable level company (NOMINAL).&#xA;end case.&#xA;end loop.&#xA;end loop.&#xA;end file.&#xA;end input program.&#xA;execute.&#xA;&#xA;*USING split file to get the average slope against time for every company.&#xA;sort cases by company.&#xA;split file by company.&#xA;REGRESSION&#xA;  /MISSING LISTWISE&#xA;  /STATISTICS COEFF OUTS R ANOVA&#xA;  /CRITERIA=PIN(.05) POUT(.10)&#xA;  /NOORIGIN &#xA;  /DEPENDENT price&#xA;  /METHOD=ENTER time.&#xA;split file off.&#xA;&#xA;&#xA;*Multiple lines on one chart via Chart Builder GUI, note the panel variable needs to be coded as nominal.&#xA;GGRAPH&#xA;  /GRAPHDATASET NAME=&quot;graphdataset&quot; VARIABLES=time price company MISSING=LISTWISE REPORTMISSING=NO&#xA;  /GRAPHSPEC SOURCE=INLINE.&#xA;BEGIN GPL&#xA;  SOURCE: s=userSource(id(&quot;graphdataset&quot;))&#xA;  DATA: time=col(source(s), name(&quot;time&quot;))&#xA;  DATA: price=col(source(s), name(&quot;price&quot;))&#xA;  DATA: panel=col(source(s), name(&quot;company&quot;), unit.category())&#xA;  GUIDE: axis(dim(1), label(&quot;time&quot;))&#xA;  GUIDE: axis(dim(2), label(&quot;price&quot;))&#xA;  GUIDE: legend(aesthetic(aesthetic.color.interior), label(&quot;company&quot;))&#xA;  ELEMENT: line(position(time*price), color.interior(panel), missing.wings())&#xA;END GPL.&#xA;&#xA;*Small multiples via the IGRAPH GUI.&#xA;IGRAPH&#xA;  /VIEWNAME='Line Chart'&#xA;  /X1=VAR(time) TYPE=SCALE&#xA;  /Y=VAR(price) TYPE=SCALE&#xA;  /PANEL=VAR(company)&#xA;  /COORDINATE=VERTICAL&#xA;  /YLENGTH=5.2&#xA;  /X1LENGTH=6.5&#xA;  /CHARTLOOK='NONE'&#xA;  /CATORDER VAR(company) (ASCENDING VALUES OMITEMPTY)&#xA;  /LINE(MEAN) KEY=ON STYLE=LINE DROPLINE=OFF INTERPOLATE=STRAIGHT BREAK=MISSING.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1036" LastActivityDate="2011-02-04T14:35:35.567" CommentCount="1" />
  <row Id="6880" PostTypeId="1" CreationDate="2011-02-04T14:40:58.553" Score="0" ViewCount="210" Body="&lt;p&gt;There are a few things that are heavily used in the practice of statistics, data analysis, etc. and that form the common canon of education, yet everybody knows they are not exactly true, useful, well-behaved, or empirically supported.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So here is the question: Which is the single worst idea still actively propagated?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please make it one suggestion per post.&lt;/p&gt;&#xA;" OwnerUserId="3044" LastActivityDate="2011-02-05T15:22:55.597" Title="What are the most dangerous concepts in the practice of data analysis?" Tags="&lt;best-practices&gt;&lt;basic-concepts&gt;" AnswerCount="0" CommentCount="0" FavoriteCount="0" />
  <row Id="6881" PostTypeId="2" ParentId="4551" CreationDate="2011-02-04T14:54:21.060" Score="3" ViewCount="" Body="&lt;p&gt;That the p-value is the probability that the null hypothesis is true and (1-p) is the probability that the alternative hypothesis is true, of that failing to reject the null hypothesis means the alternative hypothesis is false etc.&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2011-02-04T14:54:21.060" />
  <row Id="6882" PostTypeId="2" ParentId="4551" CreationDate="2011-02-04T15:06:00.230" Score="2" ViewCount="" Body="&lt;p&gt;Using pie charts to illustrate relative frequencies. More &lt;a href=&quot;http://psychology.wikia.com/wiki/Pie_chart&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="609" LastActivityDate="2011-02-04T15:06:00.230" />
  <row Id="6883" PostTypeId="1" CreationDate="2011-02-04T15:33:38.487" Score="1" ViewCount="153" Body="&lt;p&gt;Along the same lines as &lt;a href=&quot;http://stats.stackexchange.com/questions/6856/aggregating-results-from-linear-model-runs-r&quot;&gt;this question&lt;/a&gt;, is there a nice way to display regression results in MATLAB from a single or many regressions in table or graph form?&lt;/p&gt;&#xA;" OwnerUserId="2864" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-02-04T18:12:17.163" LastActivityDate="2011-05-14T11:34:41.720" Title="Displaying regression results in MATLAB" Tags="&lt;regression&gt;&lt;matlab&gt;" AnswerCount="2" />
  <row Id="6884" PostTypeId="2" ParentId="6874" CreationDate="2011-02-04T15:38:21.347" Score="2" ViewCount="" Body="&lt;p&gt;The formulas are straightforward but they are not as simple as intimated in the question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let $Y$ be the previous EWMA and let $X = x_n$, which is presumed independent of $Y$.  By &lt;a href=&quot;http://en.wikipedia.org/wiki/Moving_average&quot; rel=&quot;nofollow&quot;&gt;definition&lt;/a&gt;, the new weighted average is $Z = \alpha X + (1 - \alpha)Y$ for a constant value $\alpha$.  For notational convenience, set $\beta = 1-\alpha$.  Let $F$ denote the CDF of a random variable and $\phi$ denote its &lt;a href=&quot;http://en.wikipedia.org/wiki/Moment-generating_function&quot; rel=&quot;nofollow&quot;&gt;moment generating function&lt;/a&gt;, so that&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\phi_X(t) = \mathbb{E}_F[\exp(t X)] = \int_\mathbb{R}{\exp(t x) dF_X(x)}.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0340614307&quot; rel=&quot;nofollow&quot;&gt;Kendall and Stuart&lt;/a&gt;, let $\mu_k^&amp;#39;(Z)$ denote the non-central moment of order $k$ for the random variable $Z$; that is, $\mu_k^&amp;#39;(Z) = \mathbb{E}[Z^k]$.  The &lt;a href=&quot;http://en.wikipedia.org/wiki/Skewness&quot; rel=&quot;nofollow&quot;&gt;skewness&lt;/a&gt; and &lt;a href=&quot;http://en.wikipedia.org/wiki/Kurtosis&quot; rel=&quot;nofollow&quot;&gt;kurtosis&lt;/a&gt; are expressible in terms of the $\mu_k^&amp;#39;$ for $k = 1,2,3,4$; for example, the skewness is defined as $\mu_3 / \mu_2^{3/2}$ where &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\mu_3 = \mu_3^&amp;#39;  - 3 \mu_2^&amp;#39;\mu_1^&amp;#39; + 2{\mu_1^&amp;#39;}^3 \text{ and }\mu_2 = \mu_2^&amp;#39; - {\mu_1^&amp;#39;}^2$$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;are the third and second central moments, respectively.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By standard elementary results, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\eqalign{&#xD;&#xA;&amp;amp;1 + \mu_1^&amp;#39;(Z) t + \frac{1}{2!} \mu_2^&amp;#39;(Z) t^2 + \frac{1}{3!} \mu_3^&amp;#39;(Z) t^3 + \frac{1}{4!} \mu_4^&amp;#39;(Z) t^4 +O(t^5) \cr&#xD;&#xA;&amp;amp;= \phi_Z(t) \cr&#xD;&#xA;&amp;amp;= \phi_{\alpha X}(t) \phi_{\beta Y}(t) \cr&#xD;&#xA;&amp;amp;= \phi_X(\alpha t) \phi_Y(\beta t) \cr&#xD;&#xA;&amp;amp;= (1 + \mu_1^&amp;#39;(X) \alpha t + \frac{1}{2!} \mu_2^&amp;#39;(X) \alpha^2 t^2 + \cdots)&#xD;&#xA;(1 + \mu_1^&amp;#39;(Y) \beta t + \frac{1}{2!} \mu_2^&amp;#39;(Y) \beta^2 t^2 + \cdots).&#xD;&#xA;}&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To obtain the desired non-central moments, multiply the latter power series through fourth order in $t$ and equate the result term-by-term with the terms in $\phi_Z(t)$.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-02-04T15:38:21.347" />
  <row Id="6885" PostTypeId="2" ParentId="4551" CreationDate="2011-02-04T15:38:59.070" Score="4" ViewCount="" Body="&lt;p&gt;The most dangerous trap I encountered when working on a predictive model is not to reserve a test dataset early on so as to dedicate it to the &quot;final&quot; performance evaluation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's really easy to overestimate the predictive accuracy of your model if you have a chance to somehow use the testing data when tweaking the parameters, selecting the prior, selecting the learning algorithm stopping criterion...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To avoid this issue, before starting your work on a new dataset you should split your data as:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;development set&lt;/li&gt;&#xA;&lt;li&gt;evaluation set&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Then split your development set as a &quot;training development set&quot; and &quot;testing development set&quot; where you use the training development set to train various models with different parameters and select the bests according to there performance on the testing development set. You can also do grid search with cross validation but only on the development set. Never use the evaluation set while model selection is not 100% done.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once your are confident with the model selection and parameters, perform a 10 folds cross-validation on the evaluation set to have an idea of the &quot;real&quot; predictive accuracy of the selected model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also if you data is temporal, it is best to choose the development / evaluation split on a time code: &quot;It's hard to make predictions - especially about the future.&quot;&lt;/p&gt;&#xA;" OwnerUserId="2150" LastActivityDate="2011-02-04T15:38:59.070" />
  <row Id="6886" PostTypeId="1" CreationDate="2011-02-04T16:07:50.967" Score="1" ViewCount="101" Body="&lt;p&gt;Suppose I have a neural network, with input variables $a, b, c, d, f, g$ and output variables$ m, n, o, p, q$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given different input values, the neural network will output corresponding $m, n, o, p, q$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I want find out the best input values which can maxmize $m, n$, while minimize $o,p,q$ with different weights as well. So how can I find the best $a, b, c, d, f, g$?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently I use a simple way, which calculate $x= w_1 m + w_2n+w_3 \frac{1}{o}+w_4 \frac{1}{p}+w_5 \frac{1}{q}$, then find the input to get maxmization of x.  However this simple method assume $m, n, o, p, q$ are independent, which is not the case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So how should I solve this problem? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks.&lt;/p&gt;&#xA;" OwnerUserId="2454" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-02-04T18:10:34.560" LastActivityDate="2011-02-24T17:37:38.757" Title="How to find the best input value for this simple problem?" Tags="&lt;optimization&gt;&lt;neural-networks&gt;" AnswerCount="4" CommentCount="1" />
  <row Id="6887" PostTypeId="2" ParentId="4551" CreationDate="2011-02-04T16:40:59.897" Score="0" ViewCount="" Body="&lt;p&gt;Perhaps the poor teaching of statistics to end consumers. The fact is that most courses have given a medieval menu, not including new theoretical developments, computational and best practices, insufficient teaching of modern and complete analysis of real data sets, at least in poor and developing countries, what is the situation in developed countries?&lt;/p&gt;&#xA;" OwnerUserId="523" LastActivityDate="2011-02-04T16:40:59.897" />
  <row Id="6888" PostTypeId="2" ParentId="6876" CreationDate="2011-02-04T17:00:11.510" Score="0" ViewCount="" Body="&lt;p&gt;Chris really nailed the data minining stuff.  If you need someone who can also look at experimental data, you can stop all but the most versatile of statisticians dead in their tracks by asking them to explain a &lt;strong&gt;split-plot experiment&lt;/strong&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1920" LastActivityDate="2011-02-04T17:00:11.510" />
  <row Id="6889" PostTypeId="2" ParentId="6876" CreationDate="2011-02-04T17:22:27.417" Score="2" ViewCount="" Body="&lt;p&gt;I agree with Chris on most of what he says. Additionally, I'd like to add that without knowing the institutions or universities in detail, just looking at grades would be very misleading. I could easily give &lt;a href=&quot;http://stats.stackexchange.com/questions/6807/making-sense-out-of-statistics-theory-and-applications&quot;&gt;a relevant example&lt;/a&gt;; I have recently graduated with a masters in engineering mathematics; and taken a variety of statistics courses (with good grades) but I couldnt work in any statistics intensive job right now. That doesn't mean that my uni sucks, but mostly that I didn't manage to learn much out of my statistics courses during university...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Apart from the candidate's knowledge on statistics, I'd also highly value good communication skills; as any cross-disciplinary project eventually boils down to communication problems between experts of different fields. Any test on how well the candidate can share his expertise with others should be a good measure on that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furthermore, good computer/programming skills (and no just R is not enough, IMHO) is surely a big plus. If the person has some background in mathematical modeling, it'd be a cherry on the cake :) &lt;/p&gt;&#xA;" OwnerUserId="3014" LastActivityDate="2011-02-04T17:22:27.417" CommentCount="2" />
  <row Id="6890" PostTypeId="1" AcceptedAnswerId="6893" CreationDate="2011-02-04T17:50:56.660" Score="3" ViewCount="348" Body="&lt;p&gt;I have dendrogram and a distance matrix. I wish to compute a heatmap -- without re-doing the distance matrix and clustering.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a function in R that permits this?&lt;/p&gt;&#xA;" OwnerUserId="2842" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-02T09:07:10.713" LastActivityDate="2011-05-02T09:07:10.713" Title="Plotting a heatmap given a dendrogram and a distance matrix in R" Tags="&lt;r&gt;&lt;data-visualization&gt;&lt;clustering&gt;" AnswerCount="3" />
  <row Id="6891" PostTypeId="1" CreationDate="2011-02-04T18:50:12.683" Score="2" ViewCount="142" Body="&lt;p&gt;I have a series of measurements in which something is sampled and placed into either category 1 or category 2.  I have 3x2x3 factors for each set of measurements, and each set of measurements is done independently 3 times.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is a good statistical test to look examine the combined effects of each factor, both independently and interactively with each other?&lt;/p&gt;&#xA;" OwnerUserId="1327" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-02-04T19:32:26.510" LastActivityDate="2011-03-21T04:22:44.343" Title="What is a good statistical test for independent replicates?" Tags="&lt;anova&gt;&lt;repeated-measures&gt;&lt;statistical-significance&gt;" AnswerCount="0" CommentCount="4" />
  <row Id="6892" PostTypeId="2" ParentId="6890" CreationDate="2011-02-04T19:23:34.453" Score="1" ViewCount="" Body="&lt;p&gt;You might try looking in the maptree or ape packages. What are you trying to do?&lt;/p&gt;&#xA;" OwnerUserId="1475" LastActivityDate="2011-02-04T19:23:34.453" />
  <row Id="6893" PostTypeId="2" ParentId="6890" CreationDate="2011-02-04T19:29:58.010" Score="8" ViewCount="" Body="&lt;p&gt;I don't know a specific function for that. The ones I used generally take raw data or a distance matrix. However, it would not be very difficult to hack already existing code, without knowing more than basic R. Look at the source code for the &lt;code&gt;cim()&lt;/code&gt; function in the &lt;a href=&quot;http://cran.r-project.org/web/packages/mixOmics/index.html&quot; rel=&quot;nofollow&quot;&gt;mixOmics&lt;/a&gt; package for example (I choose this one because source code is very easy to read; you will find other functions on the &lt;a href=&quot;http://www.bioconductor.org&quot; rel=&quot;nofollow&quot;&gt;Bioconductor&lt;/a&gt; project). The interesting parts of the code are l. 92-113, where they assign the result of HC to &lt;code&gt;ddc&lt;/code&gt;, and around l. 193-246 where they devised the plotting regions (you should input the values of your distance matrix in place of mat when they call &lt;code&gt;image()&lt;/code&gt;). HTH&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A recent Google search on a related subject lead me to &lt;code&gt;dendrogramGrob()&lt;/code&gt; from the &lt;a href=&quot;http://latticeextra.r-forge.r-project.org/&quot; rel=&quot;nofollow&quot;&gt;latticeExtra&lt;/a&gt; package. Assuming you already have your sorted dendrogram object, you can skip the first lines of the example code from the on-line help and get something like this (here, with the &lt;code&gt;mtcars&lt;/code&gt; dataset):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/tFFAS.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-30T08:25:17.213" LastActivityDate="2011-04-30T08:25:17.213" CommentCount="0" />
  <row Id="6894" PostTypeId="2" ParentId="6886" CreationDate="2011-02-04T19:34:43.890" Score="1" ViewCount="" Body="&lt;p&gt;Your question is not properly thought out. There are many different neural network models, but all I know of require a real-valued objective function to optimize for. That means you can't ask that specific question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To maximize $\sum w_i x_i$, simply set $w_i=\infty\cdot sign(x_i)$ or something comparable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A basic neural network training algorithm for the multilayer perceptron network is gradient descent for the objective function, averaged over the training set.&lt;/p&gt;&#xA;" OwnerUserId="2456" LastEditorUserId="2456" LastEditorDisplayName="" LastEditDate="2011-02-04T19:42:33.363" LastActivityDate="2011-02-04T19:42:33.363" />
  <row Id="6895" PostTypeId="2" ParentId="3176" CreationDate="2011-02-04T20:01:34.703" Score="1" ViewCount="" Body="&lt;p&gt;It seems that you are fitting &lt;a href=&quot;http://en.wikipedia.org/wiki/Simultaneous_equations_model&quot; rel=&quot;nofollow&quot;&gt;simultaneous equation model&lt;/a&gt; in &lt;a href=&quot;http://en.wikipedia.org/wiki/Panel_data&quot; rel=&quot;nofollow&quot;&gt;panel data&lt;/a&gt; setting. In this case using different dummies for different equations is entirely appropriate from statistical point of view. The justification should come from economic model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The thing you should worry most is finding appropriate instrumental variables for your endogenous variables. Simultaneous equation estimates for panel data are covered &lt;a href=&quot;http://books.google.com/books?id=yTVSqmufge8C&amp;amp;lpg=PP1&amp;amp;dq=baltagi%20panel%20data&amp;amp;hl=fr&amp;amp;pg=PA113#v=onepage&amp;amp;q&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;in Baltagi book &quot;Econometric analysis of panel data&quot;&lt;/a&gt;. You can also use Arrelano-Bond type estimates which are covered in the same book.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-02-04T20:01:34.703" />
  <row Id="6896" PostTypeId="1" CreationDate="2011-02-04T21:33:28.603" Score="8" ViewCount="242" Body="&lt;h3&gt;Context&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;I have two sets of data that I want to compare. Each data element in both sets is a vector containing 22 angles (all between $-\pi$ and $\pi$). The angles relate to a given human pose configuration, so a pose is defined by 22 joint angles.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I am ultimately trying to do is determine the &quot;closeness&quot; of the two sets of data. So for each pose (22D vector) in one set, I want to find its nearest neighbour in the other set, and create a distance plot for each of the closest pairs.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Questions&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Can I simply use Euclidean distance? &#xA;&lt;ul&gt;&#xA;&lt;li&gt;To be meaningful, I assume that the distance metric would need to be defined as: $\theta = |\theta_1 - \theta_2| \quad mod \quad \pi$, where $|...|$ is absolute value and mod is modulo. Then using the resulting 22 thetas, I can perform the standard Euclidean distance calculation, $\sqrt{t_1^2 + t_2^2 + \ldots + t_{22}^2}$.&lt;/li&gt;&#xA;&lt;li&gt;Is this correct?&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;Would another distance metric be more useful, such as chi-square, or Bhattacharyya, or some other metric? If so, could you please provide some insight as to why.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="3052" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-03-07T06:47:23.443" LastActivityDate="2011-03-07T17:08:51.720" Title="Best distance measure to use" Tags="&lt;measurement&gt;&lt;distance-functions&gt;&lt;directional-statistics&gt;" AnswerCount="1" CommentCount="14" FavoriteCount="2" />
  <row Id="6897" PostTypeId="1" AcceptedAnswerId="6898" CreationDate="2011-02-04T22:16:28.863" Score="3" ViewCount="203" Body="&lt;p&gt;Here is my example. Supose we evaluate a characteristic using two different methods (a and b) and we want to study if both methods performs in a same way. We also know that these two measures have been recorded from two different groups, and the mean values for each one of these groups are highly different. Our data set could be as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;a &amp;lt;- c(22,34,56,62,27,53)&#xA;b &amp;lt;- c(42.5,43,58.6,55,31.2,51.75)&#xA;group &amp;lt;- factor(c(1,1,2,2,1,2), labels=c('bad','good'))&#xA;dat &amp;lt;- data.frame(a, b, group)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The association between a and b could be calculated as:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;lm1 &amp;lt;- lm(a ~ b, data=dat)&#xA;summary(lm1)&#xA;&#xA;Call:&#xA;lm(formula = a ~ b, data = dat)&#xA;&#xA;Residuals:&#xA;      1       2       3       4       5       6 &#xA;-13.810  -2.533  -3.106   8.103   7.541   3.806 &#xA;&#xA;Coefficients:&#xA;            Estimate Std. Error t value Pr(&amp;gt;|t|)  &#xA;(Intercept) -25.6865    19.7210  -1.302   0.2627  &#xA;b             1.4470     0.4117   3.514   0.0246 *&#xA;---&#xA;Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 &#xA;&#xA;Residual standard error: 9.271 on 4 degrees of freedom&#xA;Multiple R-squared: 0.7554, Adjusted R-squared: 0.6942 &#xA;F-statistic: 12.35 on 1 and 4 DF,  p-value: 0.02457&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As we can see, it seems to be a high association between both measures. However, if we perform the same analysis for each group separately, this association disappears. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;lm2 &amp;lt;- lm(a ~ b, data=dat, subset=dat$class=='bad')&#xA;summary(lm2)&#xA;&#xA;Call:&#xA;lm(formula = a ~ b, data = dat, subset = dat$group == &quot;bad&quot;)&#xA;&#xA;Residuals:&#xA;      1       2       5 &#xA;-6.0992  5.8407  0.2584 &#xA;&#xA;Coefficients:&#xA;            Estimate Std. Error t value Pr(&amp;gt;|t|)&#xA;(Intercept)  22.9931    35.1657   0.654    0.631&#xA;b             0.1201     0.8953   0.134    0.915&#xA;&#xA;Residual standard error: 8.449 on 1 degrees of freedom&#xA;Multiple R-squared: 0.01769,    Adjusted R-squared: -0.9646 &#xA;F-statistic: 0.01801 on 1 and 1 DF,  p-value: 0.915&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;lm3 &amp;lt;- lm(a ~ b, data=dat, subset=dat$class=='good')&#xA;summary(lm3)&#xA;&#xA;Call:&#xA;lm(formula = a ~ b, data = dat, subset = dat$group == &quot;good&quot;)&#xA;&#xA;Residuals:&#xA;     3      4      6 &#xA;-2.394  5.047 -2.652 &#xA;&#xA;Coefficients:&#xA;            Estimate Std. Error t value Pr(&amp;gt;|t|)&#xA;(Intercept)  34.9361    70.4238   0.496    0.707&#xA;FIV           0.4003     1.2761   0.314    0.806&#xA;&#xA;Residual standard error: 6.184 on 1 degrees of freedom&#xA;Multiple R-squared: 0.08959,    Adjusted R-squared: -0.8208 &#xA;F-statistic: 0.09841 on 1 and 1 DF,  p-value: 0.8065 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How should we assess the association between the two methods? We should take into account the group factor? Maybe it is a trivial question, but I have doubts about how to deal with this problem.&lt;/p&gt;&#xA;" OwnerUserId="221" LastActivityDate="2011-02-05T19:53:05.150" Title="If correlation between two variables is affected by a factor, how should I evaluate this correlation?" Tags="&lt;correlation&gt;" AnswerCount="2" />
  <row Id="6898" PostTypeId="2" ParentId="6897" CreationDate="2011-02-04T22:50:54.100" Score="6" ViewCount="" Body="&lt;p&gt;This might be a case of locally uncorrelated, but globally correlated variables. The variance in each group might be limited because of group homogeneity, therefore there is no evidence for a relationship within each group. But globally, with the full variance, the relationship can be strong. A schematic illustration of the joint distribution within three groups, and the resulting global joint distribution:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/aKSOU.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: Your question also seems to be if the global correlation is still &quot;real&quot;, even if the theoretical correlation within each group is 0. Random variables are defined on a probability space $&amp;lt;\Omega, P&amp;gt;$ where $\Omega$ is the set of all outcomes (think of different observable persons in your case), and $P$ is a probability measure. If your natural population $\Omega$ includes members from all groups, then: yes, the variables are &quot;really&quot; correlated. Otherwise, if the members of different groups do not form a natural common $\Omega$, but each belong to separate populations, then: no, the variables are uncorrelated.&lt;/p&gt;&#xA;" OwnerUserId="1909" LastEditorUserId="1909" LastEditorDisplayName="" LastEditDate="2011-02-05T19:53:05.150" LastActivityDate="2011-02-05T19:53:05.150" CommentCount="1" />
  <row Id="6899" PostTypeId="1" CreationDate="2011-02-05T01:46:59.883" Score="2" ViewCount="56" Body="&lt;p&gt;&lt;a href=&quot;http://www.biomedcentral.com/1471-2105/10/318&quot; rel=&quot;nofollow&quot;&gt;Kavosh&lt;/a&gt; is a recent package designed for &lt;a href=&quot;http://en.wikipedia.org/wiki/Network_motif&quot; rel=&quot;nofollow&quot;&gt;network motif&lt;/a&gt; discovery.  To give a comparison, Kavosh generates a collection of similar networks using an MCMC process.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The networks in consideration are directed graphs without loops or multiple edges.  &lt;em&gt;Similar&lt;/em&gt; networks have the same vertex set and the same in-degrees and out-degrees.  Ideally we want to sample uniformly at random from the set of similar graphs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As I understand it, network motif detection programs typically use a well-studied switching process:  take two directed edges (a,c) and (b,d) uniformly at random, then replace with (a,d) and (b,c).  Reject if a loop or multiple edge is formed.  (see e.g. &lt;a href=&quot;http://arxiv.org/abs/cond-mat/0312028&quot; rel=&quot;nofollow&quot;&gt;http://arxiv.org/abs/cond-mat/0312028&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Judging from its source code, Kavosh seems to speed-up this process in the following way.  For all vertices v do:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Let a=v.&lt;/li&gt;&#xA;&lt;li&gt;b is a random vertex b&amp;lt;&gt;a.&lt;/li&gt;&#xA;&lt;li&gt;c is a random out-neighbour of a.&lt;/li&gt;&#xA;&lt;li&gt;d is a random out-neighbour of c.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;And as before, replace edges (a,c) and (b,d) with (a,d) and (b,c).  Reject if a loop or multiple edge is formed.  Repeat this whole process three times (so there can be up to 3|V| switches applied in total, where |V| is the number of vertices).  [For some reason I'm unsure of, the last two steps are also repeated up to three times.]&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;Question&lt;/strong&gt;: What effects could one expect to see as a result of this change?  How concerned should one be about this change?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The Kavosh paper does not say much about it:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In our approach, similar to Milo's&#xA;  random model [17,18] switching&#xA;  operations are applied on the edges of&#xA;  the input network repeatedly, until&#xA;  the network is well randomized.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="386" LastActivityDate="2011-02-05T01:46:59.883" Title="Kavosh uses a different switching process in its MCMC; how concerned should I be?" Tags="&lt;algorithms&gt;&lt;mcmc&gt;&lt;networks&gt;&lt;graph-theory&gt;" />
  <row Id="6900" PostTypeId="2" ParentId="6886" CreationDate="2011-02-05T10:22:08.123" Score="-1" ViewCount="" Body="&lt;p&gt;I disagree. This problem is not very related to neural network, basically given 5 values, how to find the optimization, if these 5 values are dependent and will change to different directions&lt;/p&gt;&#xA;" OwnerUserId="3060" LastEditorUserId="3060" LastEditorDisplayName="" LastEditDate="2011-02-05T13:58:07.273" LastActivityDate="2011-02-05T13:58:07.273" CommentCount="4" />
  <row Id="6901" PostTypeId="2" ParentId="5960" CreationDate="2011-02-05T10:43:56.113" Score="1" ViewCount="" Body="&lt;p&gt;Dear Jan,&#xA;a programmatic code to test for multimodality is available at www.estima.com and is called MODES.SRC. You can easily convert it to Matlab, R or other softwares. Regards, guido.&lt;/p&gt;&#xA;" OwnerUserId="3061" LastActivityDate="2011-02-05T10:43:56.113" />
  <row Id="6902" PostTypeId="2" ParentId="1595" CreationDate="2011-02-05T10:49:45.910" Score="3" ViewCount="" Body="&lt;p&gt;I haven't seen the &lt;a href=&quot;http://scikit-learn.sourceforge.net/&quot; rel=&quot;nofollow&quot;&gt;scikit-learn&lt;/a&gt; explicitly mentioned in the answers above. It's a Python package for machine learning in Python. It's fairly young but growing extremely rapidly (disclaimer: I am a scikit-learn developer). It's goals are to provide standard machine learning algorithmic tools in a unified interface with a focus on speed, and usability. As far as I know, you cannot find anything similar in Matlab. It's strong points are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;A &lt;a href=&quot;http://scikit-learn.sourceforge.net/user_guide.html&quot; rel=&quot;nofollow&quot;&gt;detailed documentation&lt;/a&gt;, with &lt;a href=&quot;http://scikit-learn.sourceforge.net/auto_examples/index.html&quot; rel=&quot;nofollow&quot;&gt;many examples&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;High quality standard &lt;a href=&quot;http://scikit-learn.sourceforge.net/supervised_learning.html&quot; rel=&quot;nofollow&quot;&gt;supervised learning&lt;/a&gt; (regression/classification) tools. Specifically: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;very versatile &lt;a href=&quot;http://scikit-learn.sourceforge.net/modules/svm.html&quot; rel=&quot;nofollow&quot;&gt;SVM&lt;/a&gt; (based on libsvm, but with integration of external patches, and a lot of work on the Python binding)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.sourceforge.net/modules/linear_model.html&quot; rel=&quot;nofollow&quot;&gt;penalized linear models&lt;/a&gt; (&lt;a href=&quot;http://scikit-learn.sourceforge.net/modules/linear_model.html#lasso&quot; rel=&quot;nofollow&quot;&gt;Lasso&lt;/a&gt;, &lt;a href=&quot;http://scikit-learn.sourceforge.net/auto_examples/logistic_l1_l2_coef.html&quot; rel=&quot;nofollow&quot;&gt;sparse logistic regression&lt;/a&gt;...) with efficient implementations.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The ability to perform &lt;a href=&quot;http://scikit-learn.sourceforge.net/modules/grid_search.html&quot; rel=&quot;nofollow&quot;&gt;model selection&lt;/a&gt; by cross-validation using multiple CPUs&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.sourceforge.net/unsupervised_learning.html&quot; rel=&quot;nofollow&quot;&gt;Unsupervised learning&lt;/a&gt; to explore the data or do a first dimensionality reduction, that can easily be chained to supervised learning.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Open source, BSD licensed. If you are not in a purely academic environment (I am in what would be a national lab in the state) this matters a lot as Matlab costs are then very high, and you might be thinking of deriving products from your work.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Matlab is a great tool, but in my own work, scipy+scikit-learn is starting to give me an edge on Matlab because Python does a better job with memory due to it's view mechanism (and I have big data), and because the scikit-learn enables me to very easily compare different approaches.&lt;/p&gt;&#xA;" OwnerUserId="1265" LastActivityDate="2011-02-05T10:49:45.910" CommunityOwnedDate="2011-02-05T10:49:45.910" />
  <row Id="6903" PostTypeId="1" AcceptedAnswerId="6910" CreationDate="2011-02-05T12:07:23.020" Score="-1" ViewCount="231" Body="&lt;p&gt;Which software is better? R, Excel or others.&#xA;Would appreciate recommendations of good resource websites/books that you found useful.&lt;/p&gt;&#xA;" OwnerUserId="3062" LastActivityDate="2011-02-05T14:08:19.613" ClosedDate="2011-02-06T13:02:49.250" Title="How do I do a Monte Carlo simulation?" Tags="&lt;modeling&gt;" AnswerCount="3" CommentCount="1" FavoriteCount="1" />
  <row Id="6904" PostTypeId="2" ParentId="4551" CreationDate="2011-02-05T12:26:58.653" Score="2" ViewCount="" Body="&lt;p&gt;In similar vein to @dirkan - The use of p-values as a formal measure of evidence of the null hypothesis being true.  It does have some good heuristic and intuitively good features, but is essentially an incomplete measure of evidence because it makes no reference to the alternative hypothesis.  While the data may be unlikely under the null (leading to a small p-value), the data may be &lt;em&gt;even more unlikely&lt;/em&gt; under the alternative hypothesis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The other problem with p-values, which also relates to some styles of hypothesis testing, is there is no principle telling you which statistic you should choose, apart from the very vague &quot;large value&quot; $\rightarrow$ &quot;unlikely if null hypothesis is true&quot;.  Once again, you can see the incompleteness showing up, for you should also have &quot;large value&quot; $\rightarrow$ &quot;likely if &lt;strong&gt;alternative&lt;/strong&gt; hypothesis is true&quot; as an additional heuristic feature of the test statistic.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-02-05T12:26:58.653" />
  <row Id="6905" PostTypeId="2" ParentId="4551" CreationDate="2011-02-05T12:40:27.353" Score="0" ViewCount="" Body="&lt;p&gt;Using statistics/probability in hypothesis testing to measure the &quot;absolute truth&quot;.  Statistics simply cannot do this, they can only be of use in deciding between &lt;em&gt;alternatives&lt;/em&gt;, which must be specified from &quot;outside&quot; the statistical paradigm.  Statements such as &quot;the null hypothesis is proved true by the statistics&quot; are just incorrect; statistics can only tell you &quot;the null hypothesis is favoured by the data, compared to the alternative hypothesis&quot;.  If you then &lt;em&gt;assume&lt;/em&gt; that either the null hypothesis or the alternative must be true, you can say &quot;the null proved true&quot;, but this is only a trivial consequence of your assumption, not anything demonstrated by the data.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-02-05T12:40:27.353" />
  <row Id="6906" PostTypeId="2" ParentId="6903" CreationDate="2011-02-05T12:53:55.560" Score="0" ViewCount="" Body="&lt;p&gt;To my point of view, R is a good choice. Further, most textbooks on R have a chapter devoted to simulation. Regards&lt;/p&gt;&#xA;" OwnerUserId="3019" LastActivityDate="2011-02-05T12:53:55.560" />
  <row Id="6907" PostTypeId="1" AcceptedAnswerId="6919" CreationDate="2011-02-05T13:01:14.260" Score="9" ViewCount="405" Body="&lt;p&gt;Hello!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Look at this picture:&#xA;&lt;img src=&quot;http://i.imgur.com/Io7JY.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we draw a sample from the red density then some values are expected to be less than 0.25 whereas it is impossible to generate such a sample from the blue distribution. As a consequence, the Kullback-Leibler distance from the red density to the blue density is infinity. However, the two curves are not that distinct, in some &quot;natural sense&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is my question: Does it exist an adaptation of the Kullback-Leibler distance that would allow a finite distance between these two curves?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you!&#xA;Marco&lt;/p&gt;&#xA;" OwnerUserId="3019" LastActivityDate="2011-02-27T03:38:48.520" Title="An adaptation of the Kullback-Leibler distance?" Tags="&lt;kullback-leibler&gt;" AnswerCount="4" CommentCount="2" FavoriteCount="3" />
  <row Id="6908" PostTypeId="2" ParentId="6903" CreationDate="2011-02-05T13:19:22.340" Score="2" ViewCount="" Body="&lt;p&gt;If you are not already using R, then BUGS might be worth a look. The current version is OpenBugs &lt;a href=&quot;http://mathstat.helsinki.fi/openbugs/Home.html&quot; rel=&quot;nofollow&quot;&gt;http://mathstat.helsinki.fi/openbugs/Home.html&lt;/a&gt;. On Linux, i believe that JAGS is preferred. The syntax is the same for both: &lt;a href=&quot;http://www-ice.iarc.fr/~martyn/software/jags/&quot; rel=&quot;nofollow&quot;&gt;http://www-ice.iarc.fr/~martyn/software/jags/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="656" LastActivityDate="2011-02-05T13:19:22.340" />
  <row Id="6909" PostTypeId="2" ParentId="6907" CreationDate="2011-02-05T13:23:23.520" Score="0" ViewCount="" Body="&lt;p&gt;Yes there does, Bernardo and Reuda defined something called the &quot;intrinsic discrepancy&quot; which for all purposes is a &quot;symmetrised&quot; version of the KL-divergence.  Taking the KL divergence from $P$ to $Q$ to be $\kappa(P \mid Q)$  The intrinsic discrepancy is given by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\delta(P,Q)\equiv \min \big[\kappa(P \mid Q),\kappa(Q \mid P)\big]$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Searching intrinsic discrepancy (or bayesian reference criterion) will give you some articles on this measure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case, you would just take the KL-divergence which is finite.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another alternative measure to KL is Hellinger distance&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;EDIT: clarification, some comments raised suggested that the intrinsic discrepancy will not be finite when one density 0 when the other is not.  This is not true if the operation of evaluating the zero density is carried out as a limit&lt;/em&gt; $Q\rightarrow 0$ &lt;em&gt;or&lt;/em&gt; $P\rightarrow 0$ &lt;em&gt;.  The limit is well defined, and it is equal to $0$ for one of the KL divergences, while the other one will diverge.  To see this note:&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\delta(P,Q)\equiv \min \Big[\int P \,\log \big(\frac{P}{Q}\big),\int Q \log \big(\frac{Q}{P}\big)\Big]$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Taking limit as $P\rightarrow 0$ over a region of the integral, the second integral diverges, and the first integral converges to $0$ over this region (assuming the conditions are such that one can interchange limits and integration).  This is because $\lim_{z\rightarrow 0} z \log(z) =0$. Because of the symmetry in $P$ and $Q$ the result also holds for $Q$.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-02-06T17:03:49.210" LastActivityDate="2011-02-06T17:03:49.210" CommentCount="16" />
  <row Id="6910" PostTypeId="2" ParentId="6903" CreationDate="2011-02-05T14:08:19.613" Score="1" ViewCount="" Body="&lt;p&gt;As a math-based engineering graduate, I'd say all Monte Carlo simulations I have done have been based on MATLAB and I felt pretty comfortable in that environment. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Otherwise, if you are familiar with R-environment it might be preferable to MATLAB, as R is pretty much aimed at statistics use, though I can't really say how easy it is to implement something in R, as my experience is very little in R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A) I wouldn't use Excel for anything more than making spreadsheets for presenting/overlooking data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;B) If you'd like to read about Monte Carlo based methods you can look at &lt;a href=&quot;http://mathworld.wolfram.com/MonteCarloMethod.html&quot; rel=&quot;nofollow&quot;&gt;MathWorld&lt;/a&gt; and follow the references there, AND/OR &lt;a href=&quot;http://www.vertex42.com/ExcelArticles/mc/MonteCarloSimulation.html&quot; rel=&quot;nofollow&quot;&gt;this introduction&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good luck&lt;/p&gt;&#xA;" OwnerUserId="3014" LastActivityDate="2011-02-05T14:08:19.613" />
  <row Id="6911" PostTypeId="2" ParentId="4551" CreationDate="2011-02-05T15:22:55.597" Score="2" ViewCount="" Body="&lt;p&gt;Maybe stepwise regression and other forms of testing after model selection&lt;/p&gt;&#xA;" OwnerUserId="2126" LastActivityDate="2011-02-05T15:22:55.597" />
  <row Id="6912" PostTypeId="1" CreationDate="2011-02-05T15:42:04.080" Score="4" ViewCount="322" Body="&lt;p&gt;Hello all,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I came by the post &quot;&lt;a href=&quot;http://www.r-bloggers.com/post-hoc-pairwise-comparisons-of-two-way-anova/&quot; rel=&quot;nofollow&quot;&gt;Post-hoc Pairwise Comparisons of Two-way ANOVA&lt;/a&gt;&quot; (responding to &lt;a href=&quot;http://rtutorialseries.blogspot.com/2011/01/r-tutorial-series-two-way-anova-with.html&quot; rel=&quot;nofollow&quot;&gt;this post&lt;/a&gt;), which shows the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dataTwoWayComparisons &amp;lt;- read.csv(&quot;http://www.dailyi.org/blogFiles/RTutorialSeries/dataset_ANOVA_TwoWayComparisons.csv&quot;)&#xA;&#xA;model1 &amp;lt;- aov(StressReduction~Treatment+Age, data =dataTwoWayComparisons)&#xA;summary(model1) # Treatment is signif&#xA;&#xA;pairwise.t.test(dataTwoWayComparisons$StressReduction, dataTwoWayComparisons$Treatment, p.adj = &quot;none&quot;)&#xA;# no signif pair&#xA;&#xA;TukeyHSD(model1, &quot;Treatment&quot;)&#xA;# mental-medical   is the signif pair.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(Output is attached bellow)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could someone please explain why the Tukey HSD is able to find a significant pairing while the paired (unadjusted pvalue) t-test fails in doing so?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Here is the code output&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; model1 &amp;lt;- aov(StressReduction~Treatment+Age, data =dataTwoWayComparisons)&#xA;&amp;gt; summary(model1) # Treatment is signif&#xA;            Df Sum Sq Mean Sq F value    Pr(&amp;gt;F)    &#xA;Treatment    2     18   9.000      11 0.0004883 ***&#xA;Age          2    162  81.000      99     1e-11 ***&#xA;Residuals   22     18   0.818                      &#xA;---&#xA;Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 &#xA;&amp;gt; &#xA;&amp;gt; pairwise.t.test(dataTwoWayComparisons$StressReduction, dataTwoWayComparisons$Treatment, p.adj = &quot;none&quot;)&#xA;&#xA;        Pairwise comparisons using t tests with pooled SD &#xA;&#xA;data:  dataTwoWayComparisons$StressReduction and dataTwoWayComparisons$Treatment &#xA;&#xA;         medical mental&#xA;mental   0.13    -     &#xA;physical 0.45    0.45  &#xA;&#xA;P value adjustment method: none &#xA;&amp;gt; # no signif pair&#xA;&amp;gt; &#xA;&amp;gt; TukeyHSD(model1, &quot;Treatment&quot;)&#xA;  Tukey multiple comparisons of means&#xA;    95% family-wise confidence level&#xA;&#xA;Fit: aov(formula = StressReduction ~ Treatment + Age, data = dataTwoWayComparisons)&#xA;&#xA;$Treatment&#xA;                 diff         lwr        upr     p adj&#xA;mental-medical      2  0.92885267 3.07114733 0.0003172&#xA;physical-medical    1 -0.07114733 2.07114733 0.0702309&#xA;physical-mental    -1 -2.07114733 0.07114733 0.0702309&#xA;&#xA;&amp;gt; # mental-medical   is the signif pair.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="253" LastActivityDate="2011-02-05T16:31:07.540" Title="How could a Tukey HSD test be more signif then the uncorrected P value of t.test?" Tags="&lt;r&gt;&lt;multiple-comparisons&gt;&lt;t-test&gt;&lt;post-hoc&gt;&lt;tukeyhsd&gt;" AnswerCount="1" />
  <row Id="6913" PostTypeId="1" AcceptedAnswerId="6928" CreationDate="2011-02-05T16:29:45.263" Score="6" ViewCount="346" Body="&lt;p&gt;I recently read the advice that you should generally use median not mean to eliminate outliers.&#xA;Example:&#xA;The following article&#xA;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1420064932&quot; rel=&quot;nofollow&quot;&gt;http://www.amazon.com/Forensic-Science-Introduction-Scientific-Investigative/product-reviews/1420064932/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;has 16 reviews at the moment:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;review= c(5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 4, 4, 3, 2, 1, 1)&#xA;summary(review)  ## &quot;ordinary&quot; summary&#xA;&#xA;Min. 1st Qu.  Median    Mean 3rd Qu.    Max. &#xA;1.000   3.750   5.000   4.062   5.000   5.000 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Because they use &lt;strong&gt;Mean&lt;/strong&gt; the article gets 4 stars but if they would use &lt;strong&gt;Median&lt;/strong&gt; it would get 5 stars.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Isn't the median a 'fairer' judge?&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;An experiment shows that median's error is always bigger than mean. &lt;strong&gt;Is median worse?&lt;/strong&gt;&#xA;&lt;code&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(foreach)&#xA;&#xA;#the overall population of bookjudgments&#xA;n&amp;lt;-5&#xA;p&amp;lt;-0.5&#xA;expected.value&amp;lt;-n*p&#xA;peoplesbelieve &amp;lt;-rbinom(10^6,n, p)&#xA;&#xA;#16 ratings made for 100 books&#xA;ratings &amp;lt;- foreach(i=1:100, .combine=rbind) %do% sample(peoplesbelieve,16)&#xA;stat &amp;lt;- foreach(i=1:100, .combine=rbind) %do% c(mean=mean(ratings[i,]), median=median(ratings[i,]))&#xA;&#xA;#which mean square error is bigger? Mean's or Median's?&#xA;meansqrterror.mean&amp;lt;-mean((stat[,&quot;mean&quot;]-expected.value)^2)&#xA;meansqrterror.median&amp;lt;-mean((stat[,&quot;median&quot;]-expected.value)^2)&#xA;&#xA;res&amp;lt;-paste(&quot;mean MSE&quot;,meansqrterror.mean)&#xA;res&amp;lt;-paste(res, &quot;| median MSE&quot;, meansqrterror.median)&#xA;print(res)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="778" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-02-12T05:48:27.287" LastActivityDate="2011-02-12T05:48:27.287" Title="Is median fairer than mean?" Tags="&lt;mean&gt;&lt;median&gt;&lt;averages&gt;" AnswerCount="4" CommentCount="6" FavoriteCount="2" />
  <row Id="6914" PostTypeId="2" ParentId="6912" CreationDate="2011-02-05T16:31:07.540" Score="7" ViewCount="" Body="&lt;p&gt;Because your pairwise $t$-test above is not adjusted for age, and age explains a lot of the variance in StressReduction.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-02-05T16:31:07.540" CommentCount="1" />
  <row Id="6915" PostTypeId="2" ParentId="6913" CreationDate="2011-02-05T16:46:57.890" Score="6" ViewCount="" Body="&lt;p&gt;If the only choices are integers in the range 1 to 5, can any really be considered an outlier? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm sure that with small sample sizes, popular outlier tests will fail, but that just points out the problems inherent in small samples. Indeed, given a sample of 5, 5, 5, 5, 5, 1, Grubbs' test reports 1 as an outlier at $\alpha = 0.05$. The same test for the data you give above does not identify the 1's as outliers.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Grubbs test for one outlier&#xA;&#xA;data:  review  G = 2.0667, U = 0.6963,&#xA;p-value = 0.2153 alternative&#xA;hypothesis: lowest value 1 is an outlier&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="597" LastEditorUserId="597" LastEditorDisplayName="" LastEditDate="2011-02-05T16:54:18.127" LastActivityDate="2011-02-05T16:54:18.127" CommentCount="2" />
  <row Id="6916" PostTypeId="1" CreationDate="2011-02-05T16:48:59.650" Score="10" ViewCount="166" Body="&lt;p&gt;I'm teaching an intro stats class and was reviewing the types of sampling, including systematic sampling where you sample every kth individual or object. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A student asked if sampling every person with a particular characteristic would accomplish the same thing. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, would sampling every person with a blue t-shirt be random enough and provide enough of a representation of the whole population? At least, if you're asking a question other than &quot;What color t-shirt do you prefer wearing?&quot; My sense is no, but I wondered if anyone here had any thoughts on this.&lt;/p&gt;&#xA;" OwnerUserId="1490" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-02-06T10:59:39.137" LastActivityDate="2011-02-06T10:59:39.137" Title="Is &quot;every blue t-shirted person&quot; a systematic sample?" Tags="&lt;sampling&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="2" />
  <row Id="6917" PostTypeId="2" ParentId="6897" CreationDate="2011-02-05T17:07:20.920" Score="0" ViewCount="" Body="&lt;p&gt;Therefore, it is important to evaluate whether the homogeneity of groups is due to low number of data, or actually these groups are quite homogeneous and different. In the first case, we could ensure the presence of a high correlation even when this has not been observed for each group separately.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But what would happen in the second case? If even using a large number of data we could not observe a correlation within each group, we could say that this correlation exist?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Perhaps the value of one of these measures will be only useful for predicting membership in one of the two groups, but not to predict the value of the other measure.&lt;/p&gt;&#xA;" OwnerUserId="221" LastActivityDate="2011-02-05T17:07:20.920" />
  <row Id="6918" PostTypeId="2" ParentId="6916" CreationDate="2011-02-05T17:20:22.780" Score="12" ViewCount="" Body="&lt;p&gt;The answer, in general, to your question is &quot;no&quot;. Obtaining a random sample from a population (especially of humans) is notoriously difficult. By conditioning on a particular characteristic, you're &lt;em&gt;by definition&lt;/em&gt; not obtaining a random sample. How much bias this introduces is another matter altogether. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a slightly absurd example, you wouldn't want to sample this way at, say, a football game between the Bears and the Packers, even if your population was &quot;football fans&quot;. (Bears fans may have different characteristics than other football fans, even when the quantity you are interested in may not seem directly related to football.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are many famous examples of hidden bias resulting from obtaining samples in this way. For example, in recent US elections in which phone polls have been conducted, it is believed that people owning only a cell phone and no landline are (perhaps dramatically) underrepresented in the sample. Since these people also tend to be, by and large, younger than those with landlines, a biased sample is obtained. Furthermore, younger people have very different political beliefs than older populations. So, this is a simple example of a case where, even when the sample was not &lt;em&gt;intentionally&lt;/em&gt; conditioned on a particular characteristic, it still happened that way. And, even though the poll had &lt;em&gt;nothing to do&lt;/em&gt; with the conditioning characteristic either (i.e., whether or not one uses a landline), the effect of the conditioning characteristic on the poll's conclusions was significant, both statistically and practically.  &lt;/p&gt;&#xA;" OwnerUserId="2970" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-02-05T17:33:53.097" LastActivityDate="2011-02-05T17:33:53.097" />
  <row Id="6919" PostTypeId="2" ParentId="6907" CreationDate="2011-02-05T18:14:56.510" Score="8" ViewCount="" Body="&lt;p&gt;You might look at Chapter 3 of Devroye, Gyorfi, and Lugosi, &lt;em&gt;A Probabilistic Theory of Pattern Recognition&lt;/em&gt;, Springer, 1996. See, in particular, the section on $f$-divergences.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$f$-Divergences can be viewed as a generalization of Kullback--Leibler (or, alternatively, KL can be viewed as a special case of an $f$-Divergence).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The general form is &#xA;$$&#xD;&#xA;D_f(p, q) = \int q(x) f\left(\frac{p(x)}{q(x)}\right) \, \lambda(dx) ,&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\lambda$ is a measure that dominates the measures associated with $p$ and $q$ and $f(\cdot)$ is a convex function satisfying $f(1) = 0$. (If $p(x)$ and $q(x)$ are densities with respect to Lebesgue measure, just substitute the notation $dx$ for $\lambda(dx)$ and you're good to go.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We recover KL by taking $f(x) = x \log x$. We can get the Hellinger difference via $f(x) = (1 - \sqrt{x})^2$ and we get the &lt;em&gt;total-variation&lt;/em&gt; or $L_1$ distance by taking $f(x) = \frac{1}{2} |x - 1|$. The latter gives&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;D_{\mathrm{TV}}(p, q) = \frac{1}{2} \int |p(x) - q(x)| \, dx&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that this last one at least gives you a finite answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In another little book entitled &lt;em&gt;Density Estimation: The $L_1$ View&lt;/em&gt;, Devroye argues strongly for the use of this latter distance due to its many nice invariance properties (among others). This latter book is probably a little harder to get a hold of than the former and, as the title suggests, a bit more specialized.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Addendum&lt;/strong&gt;: Via &lt;a href=&quot;http://stats.stackexchange.com/questions/7630/clustering-jsd-or-jsd2&quot;&gt;this question&lt;/a&gt;, I became aware that it appears that the measure that @Didier proposes is (up to a constant) known as the Jensen-Shannon Divergence. If you follow the link to the answer provided in that question, you'll see that it turns out that the square-root of this quantity is actually a metric and was previously recognized in the literature to be a special case of an $f$-divergence. I found it interesting that we seem to have collectively &quot;reinvented&quot; the wheel (rather quickly) via the discussion of this question. The interpretation I gave to it in the comment below @Didier's response was also previously recognized. All around, kind of neat, actually.&lt;/p&gt;&#xA;" OwnerUserId="2970" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-02-27T03:38:48.520" LastActivityDate="2011-02-27T03:38:48.520" CommentCount="5" />
  <row Id="6920" PostTypeId="1" CreationDate="2011-02-05T18:25:52.210" Score="8" ViewCount="318" Body="&lt;p&gt;I'm analysing some data where I would like to perform ordinary linear regression, however this is not possible as I am dealing with an on-line setting with a continuous stream of input data (which will quickly get too large for memory) and need to update parameter estimates while this is being consumed. i.e. I cannot just load it all into memory and perform linear regression on the entire data set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm assuming a simple linear multivariate regression model, i.e.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;y = Ax + b + e&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What's the best algorithm for creating a continuously updating estimate of the linear regression parameters A and b? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ideally:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;I'd like an algorithm that is most O(N*M) space and time complexity per update, where N is the dimensionality of the independent variable (x) and M is the dimensionality of the dependent variable (y).&lt;/li&gt;&#xA;&lt;li&gt;I'd like to be able to&#xA;specify some parameter to determine&#xA;how much the parameters are updated&#xA;by each new sample, e.g. 0.000001&#xA;would mean that the next sample would&#xA;provide one millionth of the&#xA;parameter estimate. This would give&#xA;some kind of exponential decay for&#xA;the effect of samples in the distant&#xA;past.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="2942" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-02-06T21:30:40.103" LastActivityDate="2011-03-24T10:57:37.647" Title="Efficient online linear regression" Tags="&lt;time-series&gt;&lt;regression&gt;&lt;algorithms&gt;&lt;real-time&gt;" AnswerCount="4" FavoriteCount="1" />
  <row Id="6921" PostTypeId="2" ParentId="1595" CreationDate="2011-02-05T18:31:42.033" Score="1" ViewCount="" Body="&lt;p&gt;great overview so far. I'm using python (specifically scipy + matplotlib) as a matlab replacement since 3 years working at University. I sometimes still go back because I'm familiar with specific libraries e.g. the matlab wavelet package is purely awesome.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I like the &lt;a href=&quot;http://enthought.com/&quot; rel=&quot;nofollow&quot;&gt;http://enthought.com/&lt;/a&gt; python distribution. It's commercial, yet free for academic purposes and, as far as I know, completely open-source. As I'm working with a lot of students, before using enthought it was sometimes troublesome for them to install numpy, scipy, ipython etc. Enthought provides an installer for Windows, Linux and Mac.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Two other packages worth mentioning:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;ipython (comes already with enthought) great advanced shell. a good intro is on showmedo &lt;a href=&quot;http://showmedo.com/videotutorials/series?name=PythonIPythonSeries&quot; rel=&quot;nofollow&quot;&gt;http://showmedo.com/videotutorials/series?name=PythonIPythonSeries&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;nltk - the natural language toolkit &lt;a href=&quot;http://www.nltk.org/&quot; rel=&quot;nofollow&quot;&gt;http://www.nltk.org/&lt;/a&gt; great package in case you want to do some statistics /machine learning on any corpus. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="2904" LastActivityDate="2011-02-05T18:31:42.033" CommunityOwnedDate="2011-02-05T18:31:42.033" />
  <row Id="6922" PostTypeId="2" ParentId="6920" CreationDate="2011-02-05T18:52:53.013" Score="5" ViewCount="" Body="&lt;p&gt;I think recasting your linear regression model into a &lt;a href=&quot;http://faculty.chicagobooth.edu/hedibert.lopes/teaching/MCMCtutorial/unit10-dynamicmodels.pdf&quot; rel=&quot;nofollow&quot;&gt;state-space model&lt;/a&gt; will give you what you are after. If you use R, you may want to use package &lt;a href=&quot;http://cran.r-project.org/web/packages/dlm/index.html&quot; rel=&quot;nofollow&quot;&gt;dlm&lt;/a&gt;&#xA;and have a look at the &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0387772375&quot; rel=&quot;nofollow&quot;&gt;companion book&lt;/a&gt; by Petris et al.&lt;/p&gt;&#xA;" OwnerUserId="892" LastActivityDate="2011-02-05T18:52:53.013" CommentCount="2" />
  <row Id="6923" PostTypeId="2" ParentId="6920" CreationDate="2011-02-05T19:09:20.667" Score="8" ViewCount="" Body="&lt;p&gt;Maindonald describes a sequential method based on &lt;a href=&quot;http://en.wikipedia.org/wiki/Givens_rotation&quot; rel=&quot;nofollow&quot;&gt;Givens rotations&lt;/a&gt;.  (A Givens rotation is an orthogonal transformation of two vectors that zeros out a given entry in one of the vectors.)  At the previous step you have decomposed the &lt;a href=&quot;http://en.wikipedia.org/wiki/Design_matrix&quot; rel=&quot;nofollow&quot;&gt;design matrix&lt;/a&gt; $\mathbf{X}$ into a triangular matrix $\mathbf{T}$ via an orthogonal transformation $\mathbf{Q}$ so that $\mathbf{Q}\mathbf{X} = (\mathbf{T}, \mathbf{0})^&amp;#39;$.  (It's fast and easy to get the regression results from a triangular matrix.)  Upon adjoining a new row $v$ below $\mathbf{X}$, you effectively extend $(\mathbf{T}, \mathbf{0})^&amp;#39;$ by a nonzero row, too, say $t$.  The task is to zero out this row while keeping the entries in the position of $\mathbf{T}$ diagonal.  A sequence of Givens rotations does this: the rotation with the first row of $\mathbf{T}$ zeros the first element of $t$; then the rotation with the second row of $\mathbf{T}$ zeros the second element, and so on.  The effect is to premultiply $\mathbf{Q}$ by a series of rotations, which does not change its orthogonality.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When the design matrix has $p+1$ columns (which is the case when regressing on $p$ variables plus a constant), the number of rotations needed does not exceed $p+1$ and each rotation changes two $p+1$-vectors.    The storage needed for $\mathbf{T}$ is $O((p+1)^2)$.  Thus this algorithm has a computational cost of $O((p+1)^2)$ in both time and space.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A similar approach lets you determine the effect on regression of deleting a row.  Maindonald gives formulas; so do &lt;a href=&quot;http://books.google.com/books?id=GECBEUJVNe0C&amp;amp;printsec=frontcover&amp;amp;dq=Belsley,+Kuh,+%26+Welsch&amp;amp;source=bl&amp;amp;ots=b6k5lVb7E3&amp;amp;sig=vOuq7Ehg3OrO01CepQ8p-DZ1Bww&amp;amp;hl=en&amp;amp;ei=m7hNTeyfNsSAlAeY3eXcDw&amp;amp;sa=X&amp;amp;oi=book_result&amp;amp;ct=result&amp;amp;resnum=5&amp;amp;ved=0CDMQ6AEwBA#v=onepage&amp;amp;q&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;Belsley, Kuh, &amp;amp; Welsh&lt;/a&gt;.  Thus, if you are looking for a moving window for regression, you can retain data for the window within a circular buffer, adjoining the new datum and dropping the old one with each update.  This doubles the update time and requires additional $O(k (p+1))$ storage for a window of width $k$.  It appears that $1/k$ would be the analog of the influence parameter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For exponential decay, I think (speculatively) that you could adapt this approach to weighted least squares, giving each new value a weight greater than 1.  There shouldn't be any need to maintain a buffer of previous values or delete any old data.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;References&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;J. H. Maindonald, &lt;em&gt;Statistical Computation.&lt;/em&gt;  J. Wiley &amp;amp; Sons, 1984.  Chapter 4.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;D. A. Belsley, E. Kuh, R. E. Welsch, &lt;em&gt;Regression Diagnostics: Identifying Influential Data and Sources of Collinearity.&lt;/em&gt;  J. Wiley &amp;amp; Sons, 1980. &lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-02-05T20:59:18.570" LastActivityDate="2011-02-05T20:59:18.570" CommentCount="15" />
  <row Id="6924" PostTypeId="2" ParentId="6907" CreationDate="2011-02-05T19:26:34.743" Score="4" ViewCount="" Body="&lt;p&gt;The &lt;a href=&quot;http://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test&quot; rel=&quot;nofollow&quot;&gt;Kolmogorov distance&lt;/a&gt; between two distributions $P$ and $Q$ is the sup norm of their CDFs.  (This is the largest vertical discrepancy between the two graphs of the CDFs.)  It is used in distributional testing where $P$ is an hypothesized distribution and $Q$ is the empirical distribution function of a dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is hard to characterize this as an &quot;adaptation&quot; of the KL distance, but it does meet the other requirements of being &quot;natural&quot; and finite.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Incidentally, because the KL divergence is not a true &quot;distance,&quot; we don't have to worry about preserving all the axiomatic properties of a distance.  We can maintain the non-negativity property while making the values finite by applying any monotonic transformation $\mathbb{R_+} \to [0,C]$ for some finite value $C$.  The inverse tangent will do fine, for instance.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-02-05T19:26:34.743" CommentCount="12" />
  <row Id="6925" PostTypeId="2" ParentId="5960" CreationDate="2011-02-05T20:23:18.553" Score="2" ViewCount="" Body="&lt;p&gt;There is a well-known paper by Silverman that deals with this issue. It employs kernel-density estimation. See&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;B. W. Silverman, Using kernel density estimates to investigate multimodality, &lt;em&gt;J. Royal&#xA;  Stat. Soc. B&lt;/em&gt;, vol. 43, no. 1, 1981, pp. 97-99.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Note that there are some errors in the tables of the paper. This is just a starting point, but a pretty good one. It provides a well-defined algorithm to use, in the event that's what you're most looking for. You might look on Google Scholar at papers that cite it for more &quot;modern&quot; approaches.&lt;/p&gt;&#xA;" OwnerUserId="2970" LastActivityDate="2011-02-05T20:23:18.553" />
  <row Id="6926" PostTypeId="2" ParentId="6916" CreationDate="2011-02-05T21:42:45.253" Score="6" ViewCount="" Body="&lt;p&gt;As long as the distribution of the characteristic you are using to select units into the sample is orthogonal to the distribution of the characteristic of the population you want to estimate, you can obtain an unbiased estimate of the population quantity by conditioning selection on it. The sample is not strictly a &lt;em&gt;random&lt;/em&gt; sample. But people tend to overlook that random samples are good because the random variable used to select units into sample is orthogonal to the distribution of the population characteristic, not because it is random. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just think about drawing randomly from a Bernoulli with P(invlogit(x_i)) where x_i in [-inf, inf] is a feature of unit i such that Cov(x, y)!=0, and y is the population characteristic whose mean you want to estimate. The sample is &quot;random&quot; in the sense that you are randomizing before selecting into sample. But the sample does not yield an unbiased estimate of the population mean of y.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you need is conditioning selection into sample on a variable that is &lt;em&gt;as good as randomly assigned&lt;/em&gt;. I.e., that is orthogonal to the variable on which the quantity of interest depends. Randomizing is good because it insures orthogonality, not because of randomization itself.&lt;/p&gt;&#xA;" OwnerUserId="3069" LastActivityDate="2011-02-05T21:42:45.253" CommentCount="1" />
  <row Id="6927" PostTypeId="1" AcceptedAnswerId="6936" CreationDate="2011-02-05T21:46:17.677" Score="7" ViewCount="145" Body="&lt;p&gt;I'm doing a simulation study which requires bootstrapping estimates obtained from a generalized linear mixed model (actually, the product of two estimates for fixed effects, one from a GLMM and one from an LMM). To do the study well would require about 1000 simulations with 1000 or 1500 bootstrap replications each time. This takes a significant amount of time on my computer (many days). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;How can I speed up the computation of these fixed effects?&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To be more specific, I have subjects who are measured repeatedly in three ways, giving rise to variables X, M, and Y, where X and M are continuous and Y is binary. We have two regression equations &#xA;$$M=\alpha_0+\alpha_1X+\epsilon_1$$&#xA;$$Y^*=\beta_0+\beta_1X+\beta_2M+\epsilon_2$$&#xA;where Y$^*$ is the underlying latent continuous variable for $Y$ and the errors are not iid.&lt;br&gt;&#xA;The statistic we want to bootstrap is $\alpha_1\beta_2$. Thus, each bootstrap replication requires fitting an LMM and a GLMM. My R code is (using lme4)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    stat=function(dat){&#xA;        a=fixef(lmer(M~X+(X|person),data=dat))[&quot;X&quot;]&#xA;        b=fixef(glmer(Y~X+M+(X+M|person),data=dat,family=&quot;binomial&quot;))[&quot;M&quot;]&#xA;        return(a*b)&#xA;    }&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I realize that I get the same estimate for $\alpha_1$ if I just fit it as a linear model, so that saves some time, but the same trick doesn't work for $\beta_2$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do I just need to buy a faster computer? :)&lt;/p&gt;&#xA;" OwnerUserId="2739" LastActivityDate="2011-02-06T15:16:20.523" Title="How can I speed up calculation of the fixed effects in a GLMM?" Tags="&lt;r&gt;&lt;mixed-model&gt;" AnswerCount="3" CommentCount="6" />
  <row Id="6928" PostTypeId="2" ParentId="6913" CreationDate="2011-02-05T22:13:43.713" Score="10" ViewCount="" Body="&lt;p&gt;The problem is that you haven't really defined what it means to have a good or fair rating. You suggest in a comment on @Kevin's answer that you don't like it if one bad review takes down an item. But comparing two items where one has a &quot;perfect record&quot; and the other has one bad review, maybe that difference should be reflected.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There's a whole (high-dimensional) continuum between median and mean. You can order the votes by value, then take a weighted average with the weights depending on the position in that order. The mean corresponds to all weights being equal, the median corresponds to only one or two entries in the middle getting nonzero weight, a trimmed average corresponds to giving all except the first and last couple the same weight, but you could also decide to weight the $k$th out of $n$ samples with weight $\frac{1}{1 + (2 k - 1 - n)^2}$ or $\exp(-\frac{(2k - 1 - n)^2}{n^2})$, to throw something random in there. Maybe such a weighted average where the outliers get less weight, but still a nonzero amount, could combine good properties of median and mean?&lt;/p&gt;&#xA;" OwnerUserId="2898" LastActivityDate="2011-02-05T22:13:43.713" />
  <row Id="6929" PostTypeId="2" ParentId="6927" CreationDate="2011-02-05T23:20:50.217" Score="2" ViewCount="" Body="&lt;p&gt;It could possibly be a faster computer.  But here is one trick which may work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Generate a simultation of $Y^*$, but only conditional on $Y$, then just do OLS or LMM on the simulated $Y^*$ values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Supposing your link function is $g(.)$.  this says how you get from the probability of $Y=1$ to the $Y^*$ value, and is most likely the logistic function $g(z)=log \Big(\frac{z}{1-z}\Big)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So if you assume a bernouli sampling distribution for $Y\rightarrow Y\sim Bernoulli(p)$, and then use the jeffreys prior for the probability, you get a beta posterior for $p\sim Beta(Y_{obs}+\frac{1}{2},1-Y_{obs}+\frac{1}{2})$.  Simulating from this should be like lighting, and if it isn't, then you need a faster computer.  Further, the samples are independent, so no need to check any &quot;convergence&quot; diagnostics such as in MCMC, and you probably don't need as many samples - 100 may work fine for your case.  If you have binomial $Y&amp;#39;s$, then just replace the $1$ in the above posterior with $n_i$, the number of trials of the binomial for each $Y_i$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So you have a set of simulated values $p_{sim}$.  You then apply the link function to each of these values, to get $Y_{sim}=g(p_{sim})$.  Fit a LMM to $Y_{sim}$, which is probably quicker than the GLMM program.  You can basically ignore the original binary values (but don't delete them!), and just work with the &quot;simulation matrix&quot; ($N\times S$, where $N$ is the sample size, and $S$ is the number of simulations).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So in your program, I would replace the $gmler()$ function with the $lmer()$ function, and $Y$ with a single simultation, You would then create some sort of loop which applies the $lmer()$ function to each simulation, and then takes the average as the estimate of $b$.  Something like &#xA;$$a=\dots$$&#xA;$$b=0$$&#xA;$$do \ s=1,\dots,S$$&#xA;$$b_{est}=lmer(Y_s\dots)$$&#xA;$$b=b+\frac{1}{s}(b_{est}-b)$$&#xA;$$end$$&#xA;$$return(a*b)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let me know if I need to explain anything a bit clearer&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-02-05T23:20:50.217" CommentCount="1" />
  <row Id="6930" PostTypeId="2" ParentId="6776" CreationDate="2011-02-06T01:02:12.610" Score="0" ViewCount="" Body="&lt;p&gt;You would just do this as part of the algorithm.  So you would&#xA;a) select observing $Y$ or $Y_2$, by your random probability&#xA;b) fit the relevant regression, depending on which part of the mixture you chose.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are also mixing over $X$ as well, this doesn't change the above method, you just act as if the mixing $X.mix$ corresponds to whatever you chose for $Y.mix$ (i.e. if you chose $Y$, then assume $X$, if you chose $Y_2$ then assume $X_2$)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You do this once for each Y.  Then you need a way to relate your predictions back to the original probability of being in each mixture, which you haven't specified.  Otherwise, there is nothing to plug back into the start of the algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One way to think about it, is that an observation with a low residual, is likely to match the relevant pair (i.e. you actually chose $Y$ and $X$, or $Y_2$ and $X_2$).  Further, the large sampled $Y_2$ values are likely to be wrong, and the small sampled $Y$ values are likely to be wrong.  Using this kind of information will help you update the mixing probabilities.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-02-07T10:44:03.220" LastActivityDate="2011-02-07T10:44:03.220" />
  <row Id="6931" PostTypeId="2" ParentId="6320" CreationDate="2011-02-06T01:23:06.167" Score="1" ViewCount="" Body="&lt;p&gt;If you have no reason to suspect more than one mode in your data, then a multivariate normal distribution is not a bad first go.  You just calculate the mean vector, and covariance matrix, and there's your PDF.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However this is a very rough answer, and it seems to me that with 100 observations, you will probably have multi-modal data.  Normal just looks like one big smooth multi-dimensional mountain.  You data would probably look more like a multi-dimensional mountain range (with lots of local humps).&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-02-06T01:23:06.167" />
  <row Id="6932" PostTypeId="2" ParentId="1595" CreationDate="2011-02-06T04:09:24.287" Score="0" ViewCount="" Body="&lt;p&gt;Perhaps not directly related, but R has a nice GUI environment for interactive sessions (edit: on Mac/Windows). IPython is very good but for an environment closer to Matlab's you might try Spyder or IEP. I've had better luck of late using IEP, but Spyder looks more promising.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;IEP:&#xA;&lt;a href=&quot;http://code.google.com/p/iep/&quot; rel=&quot;nofollow&quot;&gt;http://code.google.com/p/iep/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Spyder:&#xA;&lt;a href=&quot;http://packages.python.org/spyder/&quot; rel=&quot;nofollow&quot;&gt;http://packages.python.org/spyder/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And the IEP site includes a brief comparison of related software: &lt;a href=&quot;http://code.google.com/p/iep/wiki/Alternatives&quot; rel=&quot;nofollow&quot;&gt;http://code.google.com/p/iep/wiki/Alternatives&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="26" LastActivityDate="2011-02-06T04:09:24.287" CommunityOwnedDate="2011-02-06T04:09:24.287" />
  <row Id="6933" PostTypeId="2" ParentId="6795" CreationDate="2011-02-06T05:55:09.133" Score="14" ViewCount="" Body="&lt;p&gt;This is problem &lt;strong&gt;3.23&lt;/strong&gt; on page &lt;strong&gt;97&lt;/strong&gt; of &lt;a href=&quot;http://www.stanford.edu/~hastie/local.ftp/Springer/ESLII_print5.pdf#page=115&quot; rel=&quot;nofollow&quot;&gt;Hastie et al., &lt;em&gt;Elements of Statistical Learning&lt;/em&gt;, 2nd. ed. (5th printing)&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The key to this problem is a good understanding of ordinary least squares (i.e., linear regression), particularly the orthogonality of the fitted values and the residuals. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Orthogonality lemma&lt;/strong&gt;: Let $X$ be the $n \times p$ design matrix, $y$ the response vector and $\beta$ the (true) parameters. Assuming $X$ is full-rank (which we will throughout), the OLS estimates of $\beta$ are $\hat{\beta} = (X^T X)^{-1} X^T y$. The fitted values are $\hat{y} = X (X^T X)^{-1} X^T y$. Then $\langle \hat{y}, y-\hat{y} \rangle = \hat{y}^T (y - \hat{y}) = 0$. That is, the fitted values are &lt;em&gt;orthogonal&lt;/em&gt; to the residuals. This follows since $X^T (y - \hat{y}) = X^T y - X^T X (X^T X)^{-1} X^T y = X^T y - X^T y = 0$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, let $x_j$ be a column vector such that $x_j$ is the $j$th column of $X$. The assumed conditions are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;$\frac{1}{N} \langle x_j, x_j \rangle = 1$ for each $j$, $\frac{1}{N} \langle y, y \rangle = 1$,&lt;/li&gt;&#xA;&lt;li&gt;$\frac{1}{N} \langle x_j, 1_p \rangle = \frac{1}{N} \langle y, 1_p \rangle = 0$ where $1_p$ denotes a vector of ones of length $p$, and&lt;/li&gt;&#xA;&lt;li&gt;$\frac{1}{N} | \langle x_j, y \rangle | = \lambda$ for all $j$.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Note that &lt;strong&gt;in particular&lt;/strong&gt;, the last statement of the &lt;strong&gt;orthogonality lemma&lt;/strong&gt; is identical to $\langle x_j, y - \hat{y} \rangle = 0$ for all $j$.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The correlations are tied&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, $u(\alpha) = \alpha X \hat{\beta} = \alpha \hat{y}$. So,&#xA;$$&#xD;&#xA;\langle x_j, y - u(a) \rangle = \langle x_j, (1-\alpha) y + \alpha y - \alpha \hat{y} \rangle = (1-\alpha) \langle x_j, y \rangle + \alpha \langle x_j, y - \hat{y} \rangle ,&#xD;&#xA;$$&#xA;and the second term on the right-hand side is zero by the &lt;strong&gt;orthogonality lemma&lt;/strong&gt;, so&#xA;$$&#xD;&#xA;\frac{1}{N} | \langle x_j, y - u(\alpha) \rangle | = (1-\alpha) \lambda ,&#xD;&#xA;$$&#xA;as desired. The absolute value of the correlations are just&#xA;$$&#xD;&#xA;\hat{\rho}_j(\alpha) = \frac{\frac{1}{N} | \langle x_j, y - u(\alpha) \rangle |}{\sqrt{\frac{1}{N} \langle x_j, x_j \rangle }\sqrt{\frac{1}{N} \langle y - u(\alpha), y - u(\alpha) \rangle }} = \frac{(1-\alpha)\lambda}{\sqrt{\frac{1}{N} \langle y - u(\alpha), y - u(\alpha) \rangle }} &#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note&lt;/strong&gt;: The right-hand side above is independent of $j$ and the numerator is just the same as the covariance since we've assumed that all the $x_j$'s and $y$ are centered (so, in particular, no subtraction of the mean is necessary).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What's the point?&lt;/strong&gt; As $\alpha$ increases the response vector is modified so that it inches its way toward that of the (&lt;em&gt;restricted!&lt;/em&gt;) least-squares solution obtained from incorporating only the first $p$ parameters in the model. This simultaneously modifies the estimated parameters since they are simple inner products of the predictors with the (modified) response vector. The modification takes a special form though. It keeps the (magnitude of) the correlations between the predictors and the modified response the same throughout the process (even though the &lt;em&gt;value&lt;/em&gt; of the correlation is changing). Think about what this is doing geometrically and you'll understand the name of the procedure!&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Explicit form of the (absolute) correlation&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's focus on the term in the denominator, since the numerator is already in the required form. We have&#xA;$$&#xD;&#xA;\langle y - u(\alpha), y - u(\alpha) \rangle = \langle (1-\alpha) y + \alpha y - u(\alpha), (1-\alpha) y + \alpha y - u(\alpha) \rangle .&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Substituting in $u(\alpha) = \alpha \hat{y}$ and using the linearity of the inner product, we get&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;\langle y - u(\alpha), y - u(\alpha) \rangle = (1-\alpha)^2 \langle y, y \rangle + 2\alpha(1-\alpha) \langle y, y - \hat{y} \rangle + \alpha^2 \langle y-\hat{y}, y-\hat{y} \rangle .&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Observe that&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;$\langle y, y \rangle = N$ by assumption,&lt;/li&gt;&#xA;&lt;li&gt;$\langle y, y - \hat{y} \rangle = \langle y - \hat{y}, y - \hat{y} \rangle + \langle \hat{y}, y - \hat{y} \rangle = \langle y - \hat{y}, y - \hat{y}\rangle$, by applying the &lt;strong&gt;orthogonality lemma&lt;/strong&gt; (yet again) to the second term in the middle; and,&lt;/li&gt;&#xA;&lt;li&gt;$\langle y - \hat{y}, y - \hat{y} \rangle = \mathrm{RSS}$ by definition.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Putting this all together, you'll notice that we get&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;\hat{\rho}_j(\alpha) = \frac{(1-\alpha) \lambda}{\sqrt{ (1-\alpha)^2 + \frac{\alpha(2-\alpha)}{N} \mathrm{RSS}}} =  \frac{(1-\alpha) \lambda}{\sqrt{ (1-\alpha)^2 (1 - \frac{\mathrm{RSS}}{N}) + \frac{1}{N} \mathrm{RSS}}}&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To wrap things up, $1 - \frac{\mathrm{RSS}}{N} = \frac{1}{N} (\langle y, y, \rangle - \langle y - \hat{y}, y - \hat{y} \rangle ) \geq 0$ and so it's clear that $\hat{\rho}_j(\alpha)$ is monotonically decreasing in $\alpha$ and $\hat{\rho}_j(\alpha) \downarrow 0$ as $\alpha \uparrow 1$.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Epilogue&lt;/strong&gt;: Concentrate on the ideas here. There is really only one. The &lt;strong&gt;orthogonality lemma&lt;/strong&gt; does almost all the work for us. The rest is just algebra, notation, and the ability to put these last two to work.&lt;/p&gt;&#xA;" OwnerUserId="2970" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-05-24T22:56:27.137" LastActivityDate="2011-05-24T22:56:27.137" CommentCount="8" />
  <row Id="6934" PostTypeId="2" ParentId="5960" CreationDate="2011-02-06T07:53:58.403" Score="0" ViewCount="" Body="&lt;p&gt;You can't identify the number of modes (there's no limit to the number of tiny wiggles a distribution could have), but you can get a lower bound.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;e.g. see the first article here:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.google.com/search?q=lower+bound+number+of+modes+of+a+density+donoho&quot; rel=&quot;nofollow&quot;&gt;http://www.google.com/search?q=lower+bound+number+of+modes+of+a+density+donoho&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="805" LastActivityDate="2011-02-06T07:53:58.403" />
  <row Id="6935" PostTypeId="2" ParentId="6927" CreationDate="2011-02-06T13:22:55.083" Score="3" ViewCount="" Body="&lt;p&gt;Two other possibilities too consider, before buying a new computer.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Parallel computing - bootstrapping is easy to run in parallel. If your computer is reasonably new, you probably have four cores. Take a look the the &lt;a href=&quot;http://www.rforge.net/doc/packages/multicore/multicore.html&quot; rel=&quot;nofollow&quot;&gt;multicore&lt;/a&gt; library in R.&lt;/li&gt;&#xA;&lt;li&gt;Cloud computing is also a possibility and reasonably cheap. I have colleagues that have used the amazon cloud for running R scripts. They found that it was quite cost effective.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="8" LastActivityDate="2011-02-06T13:22:55.083" CommentCount="1" />
  <row Id="6936" PostTypeId="2" ParentId="6927" CreationDate="2011-02-06T15:16:20.523" Score="3" ViewCount="" Body="&lt;p&gt;It should help to specify starting values, though it's hard to know how much. As you're doing simulation and bootstrapping, you should know the 'true' values or the un-bootstrapped estimates or both. Try using those in the &lt;code&gt;start =&lt;/code&gt; option of &lt;code&gt;glmer&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could also consider looking into the whether the tolerances for declaring convergence are stricter than you need. I'm not clear how to alter them from the &lt;code&gt;lme4&lt;/code&gt; documentation though.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-02-06T15:16:20.523" CommentCount="1" />
  <row Id="6937" PostTypeId="2" ParentId="6907" CreationDate="2011-02-06T15:47:14.310" Score="6" ViewCount="" Body="&lt;p&gt;The Kullback-Leibler divergence $\kappa(P|Q)$ of $P$ with respect to $Q$ is infinite when $P$ is not absolutely continuous with respect to $Q$, that is, when there exists a measurable set $A$ such that $Q(A)=0$ and $P(A)\ne0$. Furthermore the KL divergence is not symmetric, in the sense that in general $\kappa(P|Q)\ne\kappa(Q|P)$. Recall that&#xA;$$&#xD;&#xA;\kappa(P|Q)=\int P\log\left(\frac{P}{Q}\right).&#xD;&#xA;$$&#xA;A way out of both these drawbacks, still based on KL divergence, is to introduce the midpoint &#xA;$$R=\frac12(P+Q).&#xD;&#xA;$$ &#xA;Thus $R$ is a probability measure, and $P$ and $Q$ are always absolutely continuous with respect to $R$. Hence one can consider a &quot;distance&quot; between $P$ and $Q$, still based on KL divergence but using $R$, defined as&#xA;$$&#xD;&#xA;\eta(P,Q)=\kappa(P|R)+\kappa(Q|R).&#xD;&#xA;$$&#xA;Then $\eta(P,Q)$ is nonnegative and finite for every $P$ and $Q$, $\eta$ is symmetric in the sense that $\eta(P,Q)=\eta(Q,P)$ for every $P$ and $Q$, and $\eta(P,Q)=0$ iff $P=Q$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An equivalent formulation is&#xA;$$&#xD;&#xA;\eta(P,Q)=2\log(2)+\int \left(P\log(P)+Q\log(Q)-(P+Q)\log(P+Q)\right).&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Addendum 1&lt;/strong&gt; The introduction of the midpoint of $P$ and $Q$ is not arbitrary in the sense that &#xA;$$&#xD;&#xA;\eta(P,Q)=\min [\kappa(P|\cdot)+\kappa(Q|\cdot)],&#xD;&#xA;$$ &#xA;where the minimum is over the set of probability measures. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Addendum 2&lt;/strong&gt; @cardinal remarks that $\eta$ is also an $f$-divergence, for the convex function&#xA;$$&#xD;&#xA;f(x)=x\log(x)−(1+x)\log(1+x)+(1+x)\log(2).&#xD;&#xA;$$&lt;/p&gt;&#xA;" OwnerUserId="2592" LastEditorUserId="2592" LastEditorDisplayName="" LastEditDate="2011-02-06T20:06:31.427" LastActivityDate="2011-02-06T20:06:31.427" CommentCount="2" />
  <row Id="6938" PostTypeId="2" ParentId="6913" CreationDate="2011-02-06T15:51:31.447" Score="8" ViewCount="" Body="&lt;p&gt;The answer you get depends on the question you ask.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Mean and median answer different questions.  So they give different answers.  It's not that one is &quot;fairer&quot; than another.  Medians are often used with highly skewed data (such as income).  But, even there, sometimes the mean is best.  And sometimes you don't want ANY measure of central tendency.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In addition, whenever you give a measure of central tendency, you should give some measure of spread.  The most common pairings are mean-standard deviation and median-interquartile range.  In these data, giving just a median of 5 is, I think, misleading, or, at least, uninformative.  The median would also be 5 if every single vote was a 5.  &lt;/p&gt;&#xA;" OwnerUserId="686" LastActivityDate="2011-02-06T15:51:31.447" CommentCount="1" />
  <row Id="6939" PostTypeId="1" AcceptedAnswerId="6942" CreationDate="2011-02-06T17:33:19.747" Score="1" ViewCount="123" Body="&lt;p&gt;Suppose I have the following simplified distribution:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;time | value&#xA;1    | 2&#xA;2    | 4&#xA;3    | 8&#xA;4    | 16&#xA;&#xA;1    | 1&#xA;2    | 3&#xA;3    | 9&#xA;4    | 27&#xA;&#xA;1    | 40&#xA;2    | 20&#xA;3    | 10&#xA;4    | 5&#xA;&#xA;1    | 12&#xA;2    | 1&#xA;3    | 99&#xA;4    | 23423&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;These are all part of the same dataset (so one x can have multiple values here, eg. time = 1 corresponds with value = 2,1,40,12). I separated them because the first 3 have an obvious pattern within their slope (2, 3 and 0.5) and the last does not have a pattern in its slope. Time is in days, and the values represent a quantity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, how do I use SPSS to find the elements of a distribution that together form a pattern (any kind of pattern) in their slope? And how can I make it also include elements that almost follow this pattern, but have slight variations (a variation that I can set)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any feedback is appreciated. Thanks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It looks like SPSS is not the best tool for the task. I am interested in the R language however.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If anyone could recommend any books regarding this field of pattern recognition in R, that would be great.&lt;/p&gt;&#xA;" OwnerUserId="3040" LastEditorUserId="3040" LastEditorDisplayName="" LastEditDate="2011-02-06T18:16:45.200" LastActivityDate="2011-02-06T18:42:53.183" Title="How to detect patterns between fields of a distribution in SPSS?" Tags="&lt;distributions&gt;&lt;spss&gt;&lt;pattern-recognition&gt;" AnswerCount="1" CommentCount="6" />
  <row Id="6940" PostTypeId="1" AcceptedAnswerId="6953" CreationDate="2011-02-06T18:13:50.133" Score="4" ViewCount="375" Body="&lt;p&gt;I hope this isn't a silly question. Let's say I have some arbitrary continuous distribution. I also have a statistic, and I'd like to use this arbitrary distribution to get a p-value for this statistic. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I realize that in R it's easy to do this as long as your distribution fits one of the built-in ones, like if it's normal. But is there an easy way to do this with any given distribution, without making that kind of assumption?&lt;/p&gt;&#xA;" OwnerUserId="52" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-02-07T10:49:56.943" LastActivityDate="2011-02-07T10:49:56.943" Title="Calculating p-value from an arbitrary distribution" Tags="&lt;r&gt;&lt;distributions&gt;&lt;p-value&gt;" AnswerCount="2" CommentCount="3" />
  <row Id="6941" PostTypeId="2" ParentId="6940" CreationDate="2011-02-06T18:33:44.633" Score="5" ViewCount="" Body="&lt;p&gt;Hello Alan:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Yes, it is possible to use any arbitrary distribution to get a p-value for &lt;em&gt;any statistic&lt;/em&gt;. Theoretically and practically you can calculate (one-sided) p-value by this formula. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\mathrm{p-value} = P[T &amp;gt; T_{observed} | H_0 \quad \mathrm{holds}]$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $T$ is the test-statistic of interest and $T_{observed}$ is the value that you have calculated for the observed data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you know the theoretical distribution of $T$ under $H_0$, great! Otherwise, you can use MCMC simulation to generate from the &lt;em&gt;null distribution&lt;/em&gt; of $T$ and calculate the Monte Carlo integral to obtain &lt;em&gt;p-value&lt;/em&gt;. Numerical integration techniques will also work in case you don't want to use (may be) easier Monte Carlo methods (especially in R; in Mathematica integration may be easier, but I have no experience using it)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The only assumption you are making here is -- you know the &lt;em&gt;null distribution&lt;/em&gt; of T (which may not be in the standard R random number generator formats). That's it -- as long as you know the null distribution, p-value can be calculated.&lt;/p&gt;&#xA;" OwnerUserId="1307" LastEditorUserId="1307" LastEditorDisplayName="" LastEditDate="2011-02-06T18:39:15.453" LastActivityDate="2011-02-06T18:39:15.453" CommentCount="4" />
  <row Id="6942" PostTypeId="2" ParentId="6939" CreationDate="2011-02-06T18:42:53.183" Score="1" ViewCount="" Body="&lt;p&gt;I think I understand what you are after, but I might be wrong - just to clarify things in advance :)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to find the &quot;distribution&quot; of your data, than &lt;a href=&quot;http://www.r-project.org/&quot; rel=&quot;nofollow&quot;&gt;R&lt;/a&gt; could do this easily, I have no idea about Spss. Though I am not sure about your are really after distributions, as those would only show the probabilities of certain values in your data series not dealing with the order, &lt;code&gt;fitdistr&lt;/code&gt; from &lt;a href=&quot;http://cran.r-project.org/web/packages/MASS/index.html&quot; rel=&quot;nofollow&quot;&gt;MASS&lt;/a&gt; and &lt;code&gt;fitdist&lt;/code&gt; from &lt;a href=&quot;http://cran.r-project.org/web/packages/fitdistrplus/index.html&quot; rel=&quot;nofollow&quot;&gt;fitdistrplus&lt;/a&gt; package will be your friend. Also, &lt;a href=&quot;http://cran.r-project.org/doc/contrib/Ricci-distributions-en.pdf&quot; rel=&quot;nofollow&quot;&gt;Vito Ricci's paper&lt;/a&gt; worths reading in the issue available on CRAN.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A small example assuming you have a data table (&lt;code&gt;data&lt;/code&gt;) with your data, and would like to fit the first columns data to normal distribution:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(fitdistrplus)&#xA;fitdist(data[,1],&quot;norm&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you would like to get the &quot;slopes&quot; of the pattern of a data in a row, as I suppose you are really after, than you need to set up some linear models based on your data. I am sure Spss can do the trick also, but in R look for &lt;code&gt;lm&lt;/code&gt; and &lt;code&gt;glm&lt;/code&gt; functions. See the manual of the &lt;a href=&quot;http://stat.ethz.ch/R-manual/R-devel/library/stats/html/lm.html&quot; rel=&quot;nofollow&quot;&gt;lm&lt;/a&gt; to fit linear models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A small example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# make up a demo dataset from day 1 to day 10 with 10 values&#xA;data &amp;lt;- data.frame(time=1:10, data=c(4.17,5.58,5.18,6.11,4.50,4.61,5.17,4.53,5.33,5.14))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This would look like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; data&#xA;   time data&#xA;1     1 4.17&#xA;2     2 5.58&#xA;3     3 5.18&#xA;4     4 6.11&#xA;5     5 4.50&#xA;6     6 4.61&#xA;7     7 5.17&#xA;8     8 4.53&#xA;9     9 5.33&#xA;10   10 5.14&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And fit a simple model on it:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; lm(data)&#xA;&#xA;Call:&#xA;lm(formula = data)&#xA;&#xA;Coefficients:&#xA;(Intercept)         data  &#xA;     4.6613       0.1667  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Which shows the slope being around 0.16667.&lt;/p&gt;&#xA;" OwnerUserId="2714" LastActivityDate="2011-02-06T18:42:53.183" CommentCount="3" />
  <row Id="6943" PostTypeId="1" AcceptedAnswerId="6944" CreationDate="2011-02-06T23:31:42.537" Score="4" ViewCount="298" Body="&lt;p&gt;I am unable to get the &lt;code&gt;zscore&lt;/code&gt; function to work. When I try to call it, I get the following error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;could not find function zscore&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The online documentation online states that it is in basic R library. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Searching online reveals that other people had the same problem some time ago - though on macs. Does anyone know how to fix this problem or use other alternative function?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I saw somewhere they were using &lt;code&gt;scale()&lt;/code&gt;. This works but I am not sure whether it is the correct substitute to &quot;zscore&quot;.&lt;/p&gt;&#xA;" OwnerUserId="3008" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2011-02-06T23:47:13.467" LastActivityDate="2011-02-06T23:47:13.467" Title="zscore function in R" Tags="&lt;r&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="6944" PostTypeId="2" ParentId="6943" CreationDate="2011-02-06T23:43:55.260" Score="5" ViewCount="" Body="&lt;p&gt;As the &lt;code&gt;zscore&lt;/code&gt; function you are looking for can be found in the R.basic package made by &lt;a href=&quot;http://www.braju.com/R/&quot; rel=&quot;nofollow&quot;&gt;Henrik Bengtsson&lt;/a&gt;, which cannot be found on CRAN. To install use:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;install.packages(c(&quot;R.basic&quot;), contriburl=&quot;http://www.braju.com/R/repos/&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;See &lt;a href=&quot;http://stats.stackexchange.com/q/3890/2714&quot;&gt;this similar topic&lt;/a&gt; for more details.&lt;/p&gt;&#xA;" OwnerUserId="2714" LastActivityDate="2011-02-06T23:43:55.260" />
  <row Id="6945" PostTypeId="1" AcceptedAnswerId="6947" CreationDate="2011-02-07T00:15:04.510" Score="3" ViewCount="126" Body="&lt;p&gt;In a lottery, 1/10 of the 50 000 000 tickets give a prize. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the minimum amount of tickets one should buy to have at least a 50% chance to win? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would be very glad if you could explain your methodology when resolving this. Please consider this as homework if you will.&lt;/p&gt;&#xA;" OwnerUserId="1833" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-02-12T05:55:21.817" LastActivityDate="2011-02-12T05:55:21.817" Title="Minimum tickets required for specified probability of winning lottery" Tags="&lt;probability&gt;&lt;homework&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="6946" PostTypeId="1" AcceptedAnswerId="6948" CreationDate="2011-02-07T00:23:26.160" Score="5" ViewCount="239" Body="&lt;p&gt;During an article revision the authors found, in average, 1.6 errors by page. Assuming the errors happen randomly following a Poisson process, what is the probability of finding 5 errors in 3 consecutive pages?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please explain your methodology, as the main purpose of the question is not getting the answer but the &quot;how to&quot;. Please consider this as homework if you will.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="1833" LastEditorUserId="1833" LastEditorDisplayName="" LastEditDate="2011-02-07T03:27:36.310" LastActivityDate="2011-02-12T13:25:08.530" Title="How do you solve a Poisson process problem" Tags="&lt;probability&gt;&lt;homework&gt;&lt;poisson&gt;" AnswerCount="2" CommentCount="6" FavoriteCount="1" />
  <row Id="6947" PostTypeId="2" ParentId="6945" CreationDate="2011-02-07T01:43:02.600" Score="9" ViewCount="" Body="&lt;p&gt;I should really ask what your thoughts are so far on this. This problem is very closely related to the &quot;birthday problem&quot;. The easiest way to do these problems is to count the possible ways of either winning or losing. Usually one is &lt;strong&gt;much&lt;/strong&gt; easier to do than the other, so the key is to find the right one. Before we get into the actual calculation, let's start with some heuristics.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Heuristics&lt;/strong&gt;: Let $n$ be the total number of tickets and $m$ be the number of winning tickets. In this case $n = 50\,000\,000$ and $m = 5\,000\,000$. When $n$ is very large then purchasing multiple distinguishable tickets is almost the same as sampling &lt;em&gt;with&lt;/em&gt; replacement from the population of tickets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's suppose that, instead of having to purchase $k$ separate tickets, we purchased a ticket, looked to see if it was a winner and then returned it to the lottery. We then repeat this procedure where each such draw is independent from all of the previous ones. Then the probability of winning after purchasing $k$ tickets is just&#xA;$$&#xA;\Pr( \text{we won} \mid \text{purchased $k$ tickets} ) = 1 - \left(\frac{n-m}{n}\right)^k .&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For our case then, the right-hand side is $1 - (9/10)^k$ and so we set this equal to $1/2$ and solve for $k$ in order to get the number of tickets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But, we're actually sampling without replacement. Below, we'll go through the development, with the point being that the heuristics above are more than good enough for the present problem and many similar ones.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;There are $50\,000\,000$ tickets. Of these $5\,000\,000$ are winning ones and $45\,000\,000$ are losing ones. We seek&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;\Pr(\text{we win}) \geq 1/2 \&amp;gt;,&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;or, equivalently,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;\Pr(\text{we lose}) \leq 1/2 .&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The probability that we lose is simply the probability that we hold &lt;strong&gt;none&lt;/strong&gt; of the winning tickets. Let $k$ be the number of tickets we purchase. If $k = 1$, then $\Pr(\text{we lose}) = 45\,000\,000 / 50\,000\,000 = 9/10 \geq 1/2$, so that won't do. If we choose two tickets then there are $45\,000\,000 \cdot 44\,999\,999$ ways to choose two losing tickets and there are $50\,000\,000 \cdot 49\,999\,999$ ways to choose &lt;em&gt;any&lt;/em&gt; two tickets. So,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;\Pr( \text{we lose} ) = \frac{45\,000\,000 \cdot 44\,999\,999}{50\,000\,000 \cdot 49\,999\,999} \approx 0.9^2 = 0.81.&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's generalize now. Let $m$ be the number of winning tickets and $n$ the total number of tickets, as before. Then, if we purchase $k$ tickets,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;\Pr(\text{we lose}) = \frac{(n-m) (n-m-1) \cdots (n-m-k+1)}{n (n-1) \cdots (n-k+1)} .&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It would be tedious, but we can now just start plugging in values for $k$ until we get the probability below $1/2$. We can use a &quot;trick&quot;, though, to get close to the answer, especially when $n$ is very large and $m$ is relatively small with respect to $n$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that $\frac{n-m-k}{n-k} \leq \frac{n-m}{n}$ for all $k &amp;lt; m &amp;lt; n$. Hence&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;\Pr(\text{we lose}) = \frac{(n-m) (n-m-1) \cdots (n-m-k+1)}{n (n-1) \cdots (n-k+1)} \leq \left(1 - \frac{m}{n} \right)^k ,&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and so we need to solve the equation $\left(1 - \frac{m}{n} \right)^k = \frac{1}{2}$ for $k$. But,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;k = \frac{\log \frac{1}{2}}{\log (1 - \frac{m}{n})} \&amp;gt;,&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and so when $m/n = 1/10$, we get $k \approx 6.58$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;So&lt;/strong&gt;, $k = 7$ tickets ought to do the trick. If you plug it in to the exact equation above, you'll get that&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;\Pr( \text{we win} \mid \text{k=7} ) \approx 52.2\%&#xD;&#xA;$$&lt;/p&gt;&#xA;" OwnerUserId="2970" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-02-07T03:21:01.393" LastActivityDate="2011-02-07T03:21:01.393" CommentCount="1" />
  <row Id="6948" PostTypeId="2" ParentId="6946" CreationDate="2011-02-07T03:06:02.280" Score="5" ViewCount="" Body="&lt;p&gt;The two most important characteristics of a Poisson process with rate $\lambda$ are&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;For any interval $(s, t)$, the number of arrivals within the interval follows a Poisson distribution with mean $\lambda (t-s)$.&lt;/li&gt;&#xA;&lt;li&gt;The number of arrivals in disjoint intervals are independent of one another. So, if $s_1 &amp;lt; t_1 &amp;lt; s_2 &amp;lt; t_2$, then the number of arrivals in $(s_1, t_1)$ and $(s_2, t_2)$ are independent of one another (and have means of $\lambda (t_1 - s_1)$ and $\lambda (t_2 - s_2)$, respectively).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;For this problem let &quot;time&quot; be denoted in &quot;pages&quot;. And so the Poisson process has rate $\lambda = 1.6 \text{ errors/page}$. Suppose we are interested in the probability that there are $x$ errors in three (&lt;em&gt;prespecified!&lt;/em&gt;) pages. Call the random variable corresponding to the number of errors $X$. Then, $X$ has a Poisson distribution with mean $\lambda_3 = 3 \lambda = 3 \cdot 1.6 = 4.8$. And so&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\Pr(\text{$x$ errors in three pages}) = \Pr(X = x) = \frac{e^{-\lambda_3} \lambda_3^x}{x!},&#xA;$$&#xA;so, for $x = 5$, we get&#xA;$$&#xD;&#xA;\Pr(X = 5) = \frac{e^{-\lambda_3} \lambda_3^5}{5!}  = \frac{e^{-4.8} 4.8^5}{5!} \approx 0.175&#xD;&#xA;$$&lt;/p&gt;&#xA;" OwnerUserId="2970" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-02-07T03:42:29.363" LastActivityDate="2011-02-07T03:42:29.363" CommentCount="1" />
  <row Id="6949" PostTypeId="1" CreationDate="2011-02-07T03:13:07.937" Score="3" ViewCount="94" Body="&lt;p&gt;I am looking for a book/case study etc on how to build a fraud detection model at the transaction level. Something applied rather than theoretical would be really helpful.&lt;/p&gt;&#xA;" OwnerUserId="3081" LastActivityDate="2011-04-13T17:16:56.247" Title="Any good reference books/material to help me build a txn level fraud detection model?" Tags="&lt;fraud&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="1" />
  <row Id="6950" PostTypeId="1" AcceptedAnswerId="6952" CreationDate="2011-02-07T03:45:54.400" Score="3" ViewCount="135" Body="&lt;p&gt;I have a large amount of variables (24) to predict a Y/N value, and I would like help for writting a procedure that automatically tries the different results of the factor selection to see how good the regression turns out to be, and of course I want to save the best model for later use.&lt;/p&gt;&#xA;" OwnerUserId="1808" LastActivityDate="2011-02-07T10:24:37.227" Title="How to do a logistic regression with the results of different factor analysis methods" Tags="&lt;r&gt;&lt;logistic-regression&gt;&lt;factor-analysis&gt;" AnswerCount="1" CommentCount="3" FavoriteCount="1" />
  <row Id="6951" PostTypeId="2" ParentId="6946" CreationDate="2011-02-07T07:36:25.897" Score="3" ViewCount="" Body="&lt;p&gt;If I were his instructor I'd prefer the following explanation, which obviously is equivalent to that given by @cardinal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let $N_t$ be the Poisson process of counting the number of errors on consequtive pages, of rate $\lambda=1.6/page$. One supposes that  we observe the process&#xA;at integer moments t (here t being assimilated with number of pages). &#xA;Because $P(N_t=k)=e^{-\lambda t}  (\lambda t)^k/k!$, we have $P(N_3=5)=e^{-(3*1.6)}(4.8)^5/5!$.&lt;/p&gt;&#xA;" OwnerUserId="2376" LastEditorUserId="2376" LastEditorDisplayName="" LastEditDate="2011-02-12T13:25:08.530" LastActivityDate="2011-02-12T13:25:08.530" CommentCount="1" />
  <row Id="6952" PostTypeId="2" ParentId="6950" CreationDate="2011-02-07T10:24:37.227" Score="3" ViewCount="" Body="&lt;p&gt;First of all, consider if the factor analysis is the right way to do feature extraction. I would suggest to use principal component analysis to make dimension reduction first and then use extracted features as predictor variables. Depends on your settings you should also use appropriate cross-validation regime to access your prediction.&lt;/p&gt;&#xA;" OwnerUserId="609" LastActivityDate="2011-02-07T10:24:37.227" CommentCount="4" />
  <row Id="6953" PostTypeId="2" ParentId="6940" CreationDate="2011-02-07T10:27:29.180" Score="5" ViewCount="" Body="&lt;p&gt;If you have a &lt;a href=&quot;http://en.wikipedia.org/wiki/Cumulative_distribution_function&quot; rel=&quot;nofollow&quot;&gt;cumulative distribution function&lt;/a&gt; $F$, then calculating the $p$-value for given statistic $T$ is simply $1-F(T)$. This is straightforward in R. If you have &lt;a href=&quot;http://en.wikipedia.org/wiki/Density_function&quot; rel=&quot;nofollow&quot;&gt;probability density function&lt;/a&gt; on the other hand, then $F(x)=\int_{-\infty}^xp(t)dt$. You can find this integral analytically or numerically. In R this will look like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dF &amp;lt;- function(x)dnorm(x)&#xA;pF &amp;lt;- function(q)integrate(dF,-Inf,q)$value &#xA;&#xA;&amp;gt; pF(1)&#xA;[1] 0.8413448&#xA;&amp;gt; pnorm(1)&#xA;[1] 0.8413447&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can tune &lt;code&gt;integrate&lt;/code&gt; for better accuracy. This of course may fail for specific cases, when the integral does not behave well, but it should work for majority of the density functions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can of course pass parameters into &lt;code&gt;pF&lt;/code&gt;, if you have several parameter values to try-out and do not want to redefine &lt;code&gt;dF&lt;/code&gt; each time. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dF &amp;lt;- function(x,mean=0,sd=1)dnorm(x,mean=mean,sd=sd)&#xA;pF &amp;lt;- function(q,mean=0,sd=1)integrate(dF,-Inf,q,mean=mean,sd=sd)$value &#xA;&#xA;&amp;gt; pF(1,1,1)&#xA;[1] 0.5&#xA;&amp;gt; pnorm(1,1,1)&#xA;[1] 0.5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Of course you can also use Monte-Carlo methods as detailed by @suncoolsu, this would be just another numerical method for integration.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-02-07T10:27:29.180" CommentCount="5" />
  <row Id="6954" PostTypeId="1" AcceptedAnswerId="6959" CreationDate="2011-02-07T10:49:26.900" Score="5" ViewCount="113" Body="&lt;p&gt;Hi everyone !&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In R there is a function &lt;a href=&quot;http://127.0.0.1:14633/library/stats/html/nlm.html&quot; rel=&quot;nofollow&quot;&gt;nlm()&lt;/a&gt; which carries out a minimization of a function f using the Newton-Raphson algorithm. In particular, that function outputs the value of the variable code defined as follows:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;code an integer indicating why the optimization process terminated.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;1: relative gradient is close to zero, current iterate is probably solution.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;2: successive iterates within tolerance, current iterate is probably solution.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;3: last global step failed to locate a point lower than estimate. Either estimate is an approximate local minimum of the function or steptol is too small.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;4: iteration limit exceeded.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;5: maximum step size stepmax exceeded five consecutive times. Either the function is unbounded below, becomes asymptotic to a finite value from above in some direction or stepmax is too small. &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Can someone explain me (maybe using a simple illustration with a function of only one variable) to what correspond situations 1-5?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, situation 1 might correspond to the following picture:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/2bfVh.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you in advance!&lt;/p&gt;&#xA;" OwnerUserId="3019" LastEditorUserId="3019" LastEditorDisplayName="" LastEditDate="2011-02-07T12:32:44.283" LastActivityDate="2011-02-26T15:47:54.760" Title="The code variable in the nlm() function" Tags="&lt;r&gt;&lt;minimum&gt;" AnswerCount="1" />
  <row Id="6955" PostTypeId="1" CreationDate="2011-02-07T11:25:49.097" Score="1" ViewCount="72" Body="&lt;p&gt;I create SAS output in .rtf format and the File Download warning pops up each time. I need to turn this off for all files that I create. How?&lt;/p&gt;&#xA;" OwnerUserId="3085" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-02-07T15:16:13.880" LastActivityDate="2011-02-18T22:56:20.907" Title="Getting rid of file download warning in SAS" Tags="&lt;sas&gt;" AnswerCount="2" />
  <row Id="6956" PostTypeId="1" AcceptedAnswerId="6961" CreationDate="2011-02-07T12:01:37.240" Score="1" ViewCount="190" Body="&lt;p&gt;I was looking at howto figure out the optimal lag order for an ADF-test when I came across &lt;a href=&quot;http://homepages.strath.ac.uk/~hbs96127/mlecture12.ppt&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; PPT presentation. There are two examples of a ADF-test model reduction, one on p23 and one on p36.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have two questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;How do I come up with the F numbers of the reduction? For example, on p23 it says &quot;F(1,487) = 0.92201 [0.3374]&quot;? I understand what an F number is, but what calculation lies behind the 0.92201 number (and 0.3374)?&lt;/li&gt;&#xA;&lt;li&gt;How do I interpret the numbers? For example, why is model 4 the preferred one on p23?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I'm not a complete noob, but I certainly wouldn't mind a nicely laid out pedagogical answer. Thanks!&lt;/p&gt;&#xA;" OwnerUserId="3086" LastActivityDate="2011-02-07T14:46:03.480" Title="How to interpret ADF-test optimal lag model reduction?" Tags="&lt;optimization&gt;&lt;test&gt;&lt;cointegration&gt;" AnswerCount="1" />
  <row Id="6957" PostTypeId="2" ParentId="6653" CreationDate="2011-02-07T12:15:54.877" Score="6" ViewCount="" Body="&lt;p&gt;I would  start simulating the data from a toy model. Something like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;n.games &amp;lt;- 1000&#xA;n.slices &amp;lt;- 90&#xA;&#xA;score.away &amp;lt;- score.home &amp;lt;- matrix(0, ncol=n.slices, nrow=n.games)&#xA;&#xA;for (j in 2:n.slices) {&#xA;  score.home[ ,j] &amp;lt;- score.home[ , j-1] + (runif(n.games)&amp;gt;.97)&#xA;  score.away[ ,j] &amp;lt;- score.away[ , j-1] + (runif(n.games)&amp;gt;.98)&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now we have something to play with. You could also use the raw data, but I find  simulating the data very helpful to think things through.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Next I would just plot the data, that is, plot time of the game versus lead home, with the color scale corresponding to the observed probability of winning.   &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;score.dif &amp;lt;- score.home-score.away&#xA;&#xA;windf &amp;lt;- data.frame(game=1:n.games, win=score.home[ , n.slices] &amp;gt; score.away[, n.slices])&#xA;&#xA;library(reshape)&#xA;library(ggplot2)&#xA;&#xA;dnow &amp;lt;- melt(score.dif)&#xA;names(dnow) &amp;lt;- c('game', 'time', 'dif')&#xA;dnow &amp;lt;- merge(dnow, windf)&#xA;&#xA;res &amp;lt;- ddply(dnow, c('time', 'dif'), function(x) c(pwin=sum(x$win)/nrow(x)))&#xA;&#xA;qplot(time, dif, fill=pwin, data=res, geom='tile') + scale_color_gradient2() &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This will help you find the support of your data, and give you a raw idea of what the probabilities look like. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/isZMF.png&quot; alt=&quot;Plot&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="375" LastEditorUserId="375" LastEditorDisplayName="" LastEditDate="2011-02-07T13:08:39.737" LastActivityDate="2011-02-07T13:08:39.737" />
  <row Id="6958" PostTypeId="1" CreationDate="2011-02-07T12:26:51.370" Score="4" ViewCount="328" Body="&lt;p&gt;I have fitted the ARIMA models to the original time series, and the best model is ARIMA(1,1,0). Now I want to simulate the series from that model. I wrote the simple AR(1) model, but I couldn't understand how to adjust the difference within the model ARI(1,1,0). &#xA;The following R code for AR(1) series is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;phi= -0.7048                                 &#xA;z=rep(0,100)                                 &#xA;e=rnorm(n=100,0,0.345)                       &#xA;cons=2.1                                     &#xA;z[1]=4.1&#xA;for (i in 2:100) z[i]=cons+phi*z[i-1]+e[i]   &#xA;plot(ts(Y))                &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How do i include the difference term ARI(1,1) in above code. &#xA;Any one help me in this regard.&lt;/p&gt;&#xA;" OwnerUserId="3084" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2011-03-30T03:13:24.613" LastActivityDate="2011-03-30T03:13:24.613" Title="Simulation of ARIMA (1,1,0) series" Tags="&lt;r&gt;&lt;time-series&gt;&lt;arima&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="6959" PostTypeId="2" ParentId="6954" CreationDate="2011-02-07T12:51:40.707" Score="5" ViewCount="" Body="&lt;p&gt;These situations are understood more clearly when having in mind what minimisation or maximisation really is and how optimisation works. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose we have function $f$ which has local minimum at $x_0$. Optimisation methods try to construct the sequence $x_i$ which converges to $x_0$. It is always shown that in theory the sequence constructed converges to the point of local minimum for some class of functions $f$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To obtain next candidate in iteration $i$ can be a lengthy process, so it is usual that all algorithms limit the number of iterations. This corresponds to situation &lt;strong&gt;4&lt;/strong&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then for each $x$ close to $x_0$ we have that $f(x)&amp;gt;f(x_0)$. So if $f(x_i)&amp;gt;f(x_{i-1})$ this is an indication that we reached the minimum. This corresponds to situation &lt;strong&gt;3&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now if function $f$ has a derivative at $x_0$ then necessarily $\nabla f(x_0)=0$.  Newton-Raphson method calculates gradient at each step, so if $\nabla f(x_i)\approx 0$, $x_i$ is probably a solution, which corresponds to situation &lt;strong&gt;1&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each convergent sequence of real vectors is &lt;a href=&quot;http://en.wikipedia.org/wiki/Cauchy_sequence&quot; rel=&quot;nofollow&quot;&gt;Cauchy&lt;/a&gt; sequence and vice versa, roughly meaning that if  $x_i$ is close to $x_0$, then $x_i$ is close to $x_{i+1}$ and vice versa, where $i$ is the iteration number. So if $|x_i-x_{i-1}|&amp;lt;\varepsilon$, and we know that in theory $x_i$ converges to $x_0$, then we should be close to the minimum point. This corresponds to situation &lt;strong&gt;2&lt;/strong&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Converging sequences have the property that they contract, i.e. if we are close to convergence all the remaining elements of the sequence are contained in small area. So if the sequence which in theory should converge starts to take large steps this is an indication that there is no convergence probably. This corresponds to situation &lt;strong&gt;5&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note&lt;/strong&gt; Strict mathematical definitions were left out intentionally.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-02-07T12:51:40.707" CommentCount="2" />
  <row Id="6960" PostTypeId="2" ParentId="6958" CreationDate="2011-02-07T13:32:42.747" Score="9" ViewCount="" Body="&lt;p&gt;If you want to simulate ARIMA you can use &lt;code&gt;arima.sim&lt;/code&gt; in R, there is no need to do it by hand. This will generate the series you want.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;e &amp;lt;- rnorm(100,0,0.345) &#xA;arima.sim(n=100,model=list(ar=-0.7048,order=c(1,1,0)),start.innov=4.1,n.start=1,innov=2.1+e)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can look at the code of how this is achieved by typing &lt;code&gt;arima.sim&lt;/code&gt; in R command line. &#xA;Alternatively if you do it yourself, the function you are probably looking is &lt;code&gt;diffinv&lt;/code&gt;. It computes the inverse of lagged differences.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For recursive sequences &lt;code&gt;R&lt;/code&gt; has a nice function &lt;code&gt;filter&lt;/code&gt;. So instead of using loop &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;z &amp;lt;- rep(NA,100)&#xA;z[1] &amp;lt;- 4.1&#xA;for (i in 2:100) z[i]=cons+phi*z[i-1]+e[i]   &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;you can write &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;filter(c(4.1,2.1+e),filter=-0.7048,method=&quot;recursive&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This will give the identical result to &lt;code&gt;arima.sim&lt;/code&gt; example above:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;diffinv(filter(c(4.1,2.1+e),filter=-0.7048,method=&quot;recursive&quot;)[-1])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-02-08T19:58:59.113" LastActivityDate="2011-02-08T19:58:59.113" />
  <row Id="6961" PostTypeId="2" ParentId="6956" CreationDate="2011-02-07T13:56:16.377" Score="2" ViewCount="" Body="&lt;p&gt;For page 23 you have simple &lt;a href=&quot;http://en.wikipedia.org/wiki/F-test&quot; rel=&quot;nofollow&quot;&gt;F statistics&lt;/a&gt; for regression (scroll down to regression problems in the link for the formula). So the number 0.92201 comes from this calculation (in R):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; (963.09-961.27)/(961.27/487)&#xA;[1] 0.922051&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The value in the brackets is p-value of F-statistic with degrees of freedom 1 and 487:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; 1-pf((963.09-961.27)/(961.27/487),1,487)&#xA;[1] 0.3374135&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now all these statistic test whether the additional parameters in the model are zero or not. So the model 4 comes up as the best, because compared to it, hypothesis that additional parameters in models 1 to 3 are zero is not rejected. On the other hand the hypothesis that additional parameter in model 4 is zero compared to model 5 is not rejected.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note under null hypothesis of unit root, the asymptotic statistics on coefficients of lagged differences are the same as in usual regression. The point of including lagged differences is to control for serial correlation, the question is how many. This example illustrates how you can choose how many to include. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my opinion though it is rather futile exercise. Why recreate ADF test by hand, when there is readily available packages to do it for you? The goal of ADF is to test whether we have unit root or not. Based on that information some kind of model is built for the variable tested, which in practice never has the form used in ADF test. This of course is my personal opinion. As always your mileage may vary.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-02-07T14:46:03.480" LastActivityDate="2011-02-07T14:46:03.480" CommentCount="1" />
  <row Id="6962" PostTypeId="2" ParentId="6955" CreationDate="2011-02-07T14:48:36.677" Score="2" ViewCount="" Body="&lt;p&gt;It's under the Recycle Bin's properties. Just right-click Recycle Bin,&#xA;properties, and you can turn off the confirmation there.&lt;/p&gt;&#xA;" OwnerUserId="2310" LastActivityDate="2011-02-07T14:48:36.677" CommentCount="3" />
  <row Id="6963" PostTypeId="1" AcceptedAnswerId="7617" CreationDate="2011-02-07T15:53:15.410" Score="8" ViewCount="193" Body="&lt;p&gt;It is easy to produce a random variable with Dirichlet distribution using Gamma variables with the same scale parameter. If:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ X_i \sim \text{Gamma}(\alpha_i, \beta) $ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ \left(\frac{X_1}{\sum_j X_j},\; \ldots\; , \frac{X_n}{\sum_j X_j}\right) \sim \text{Dirichlet}(\alpha_1,\;\ldots\;,\alpha_n) $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Problem&lt;/em&gt;&#xA;What happens if the scale parameters are not equal?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ X_i \sim \text{Gamma}(\alpha_i, \beta_i) $ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then what is the distribution this variable?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ \left(\frac{X_1}{\sum_j X_j},\; \ldots\; , \frac{X_n}{\sum_j X_j}\right) \sim \; ? $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For me it would be sufficient to know the expected value of this distribution.&lt;br&gt;&#xA;I need a approximate closed algebraic formula that can be evaluated very very quickly by a computer.&lt;br&gt;&#xA;Let's say approximation with accurancy of 0.01 is sufficient.&lt;br&gt;&#xA;You can assume that:  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ \alpha_i, \beta_i \in \mathbb{N} $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Note&lt;/em&gt; In short, the task is to find an approximation of this integral: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ f(\vec{\alpha}, \vec{\beta}) = \int_{\mathbb{R}^n_+} \;\frac{x_1}{\sum_j x_j} \cdot \prod_j \frac{\beta_j^{\alpha_j}}{\Gamma(\alpha_j)} x_j^{\alpha_j - 1} e^{-\beta_j x_j} \;\;  dx_1\ldots dx_n$&lt;/p&gt;&#xA;" OwnerUserId="217" LastEditorUserId="217" LastEditorDisplayName="" LastEditDate="2011-02-07T17:13:02.550" LastActivityDate="2011-03-08T07:45:43.053" Title="What is the expected value of modified Dirichlet distribution? (integration problem)" Tags="&lt;expected-value&gt;&lt;dirichlet-distribution&gt;&lt;gamma-distribution&gt;&lt;integral&gt;" AnswerCount="1" CommentCount="6" FavoriteCount="3" />
  <row Id="6964" PostTypeId="1" CreationDate="2011-02-07T17:16:09.403" Score="1" ViewCount="96" Body="&lt;p&gt;I just ran a vendor-supplied program that converts data from their proprietary binary format to what they call a &quot;Matlab format&quot;. There are two resulting files from the conversion, a &quot;filename.dat&quot; and a &quot;filename.m&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &quot;filename.m&quot; file seems to have column heading information in it. The second &quot;filename.dat&quot; file is &gt;13Gigs in size and I can't load it into anything at the moment to inspect it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I go about loading this &quot;Matlab format&quot; data into R? &lt;/p&gt;&#xA;" OwnerUserId="2768" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2011-02-07T18:11:40.303" LastActivityDate="2011-02-07T18:11:40.303" Title="How to pull data formatted for Matlab into R?" Tags="&lt;r&gt;&lt;matlab&gt;" AnswerCount="1" />
  <row Id="6965" PostTypeId="2" ParentId="6964" CreationDate="2011-02-07T17:26:07.803" Score="2" ViewCount="" Body="&lt;p&gt;This doesn't sound like &quot;Matlab format&quot; (which would be a .mat file).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rather, the .m-file is a Matlab function or script that is most likely used to read the data in the .dat file. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Have a look at the Matlab &lt;a href=&quot;http://www.mathworks.com/help/techdoc/&quot; rel=&quot;nofollow&quot;&gt;documentation&lt;/a&gt; to find out how the .m file reads the data, so that you can port the code to R.&lt;/p&gt;&#xA;" OwnerUserId="198" LastActivityDate="2011-02-07T17:26:07.803" />
  <row Id="6966" PostTypeId="1" CreationDate="2011-02-07T18:05:50.610" Score="6" ViewCount="470" Body="&lt;p&gt;Why continue to teach and use hypothesis testing (with all its difficult concepts and which are among the most statistical sins) for problems where there is an interval estimator (confidence, bootstrap, credibility or whatever)? What is the best explanation (if any) to be given to students? only tradition? The views will be very welcome.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance&lt;/p&gt;&#xA;" OwnerUserId="523" LastActivityDate="2011-02-08T17:52:56.230" Title="Why continue to teach and use hypothesis testing?" Tags="&lt;hypothesis-testing&gt;&lt;teaching&gt;&lt;best-practices&gt;" AnswerCount="6" CommentCount="4" FavoriteCount="2" />
  <row Id="6967" PostTypeId="1" AcceptedAnswerId="6973" CreationDate="2011-02-07T18:20:52.983" Score="2" ViewCount="154" Body="&lt;p&gt;/edit: To clarify: The mtable function from the &lt;a href=&quot;http://cran.r-project.org/web/packages/memisc/index.html&quot; rel=&quot;nofollow&quot;&gt;memisc&lt;/a&gt; package does exactly what I need, but unfortunately does not work with arima models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Similar to &lt;a href=&quot;http://stats.stackexchange.com/questions/6856/aggregating-results-from-linear-model-runs-r&quot;&gt;this question&lt;/a&gt;: I have multiple Arima models, some of which I've also fit with dependent variables. I'd like an easy way to make a table/graph of the coefficients in each model, as well as summary statistics about each model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is some example code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sim &amp;lt;- arima.sim(list(order = c(1,1,0), ar = 0.7), n = 200)&#xA;&#xA;ar1&amp;lt;-arima(sim,order=c(1,1,0))&#xA;ar2&amp;lt;-arima(sim,order=c(2,1,0))&#xA;ar3&amp;lt;-arima(sim,order=c(3,1,0))&#xA;ar4&amp;lt;-arima(sim,order=c(2,2,1))&#xA;&#xA;#Try mtable&#xA;library(memisc)&#xA;mtable(&quot;Model 1&quot;=ar1,&quot;Model 2&quot;=ar2,&quot;Model 3&quot;=ar3,&quot;Model 4&quot;=ar4)&#xA;#&amp;gt;Error in UseMethod(&quot;getSummary&quot;) : &#xA;#  no applicable method for 'getSummary' applied to an object of class &quot;Arima&quot;&#xA;&#xA;#Try  apsrtable&#xA;library(apsrtable)&#xA;apsrtable(&quot;Model 1&quot;=ar1,&quot;Model 2&quot;=ar2,&quot;Model 3&quot;=ar3,&quot;Model 4&quot;=ar4)&#xA;#&amp;gt;Error in est/x$se : non-numeric argument to binary operator&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2817" LastEditorUserId="2817" LastEditorDisplayName="" LastEditDate="2011-02-07T23:22:29.097" LastActivityDate="2011-02-08T00:26:12.043" Title="Aggregating results from Arima runs R" Tags="&lt;r&gt;&lt;time-series&gt;&lt;regression&gt;&lt;arima&gt;" AnswerCount="1" CommentCount="5" FavoriteCount="2" />
  <row Id="6968" PostTypeId="2" ParentId="6966" CreationDate="2011-02-07T18:24:00.687" Score="9" ViewCount="" Body="&lt;p&gt;For my part I teach hypothesis testing for a number of reasons.  One is historical, that they'll have to understand a large body of prior research they read and understand the hypothesis testing point of view.  A second is that even in modern times it's still used by some researchers, often implicitly, when performing other kinds of statistically analyses.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But when I teach it I teach it in the framework of model building, that there is a null and alternative model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The sins of hypothesis testing are not inherent in the math, or a variety of ways the underlying logic can be taught.  Where they lie is in over-reliance (over- a lot of things) and misinterpretation.  If a vast majority of naive researchers exclusively used interval estimation with no recognition of any of the relationships to these things we call hypotheses we might call that a sin.&lt;/p&gt;&#xA;" OwnerUserId="601" LastActivityDate="2011-02-07T18:24:00.687" CommentCount="2" />
  <row Id="6969" PostTypeId="2" ParentId="6706" CreationDate="2011-02-07T18:28:04.613" Score="1" ViewCount="" Body="&lt;p&gt;You could also look &lt;a href=&quot;http://www.wessa.net/rwasp_fitdistrexp.wasp&quot; rel=&quot;nofollow&quot;&gt;Wessa P., (2008), Maximum-likelihood Exponential Distribution Fitting&lt;/a&gt; which has a nice, documented online execution of the problem with the R source code attached to the actual plot.&lt;/p&gt;&#xA;" OwnerUserId="2714" LastActivityDate="2011-02-07T18:28:04.613" />
  <row Id="6970" PostTypeId="2" ParentId="6653" CreationDate="2011-02-07T19:16:46.093" Score="0" ViewCount="" Body="&lt;p&gt;I would think of it as a hierarchical model with play nested within ??? period/goal/line/game&lt;/p&gt;&#xA;" OwnerUserId="3091" LastActivityDate="2011-02-07T19:16:46.093" CommentCount="1" />
  <row Id="6971" PostTypeId="1" AcceptedAnswerId="6984" CreationDate="2011-02-07T22:37:55.593" Score="3" ViewCount="80" Body="&lt;p&gt;In order to test the Matrix package in R, I ran the following piece of code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(Matrix)&#xA;&#xA;p &amp;lt;- 50; n &amp;lt;- 500&#xA;a0 &amp;lt;- sample(1:(n*p), n*p*0.1)&#xA;d1 &amp;lt;- (a0-1) %% n&#xA;d2 &amp;lt;- floor((a0-d1)/n)&#xA;d0 &amp;lt;- cbind(d1+1, d2+1)&#xA;x0 &amp;lt;- Matrix(0, n, p, forceCheck=TRUE)&#xA;x0[d0] &amp;lt;- rnorm(n*p*0.1, 10, 1)&#xA;x1 &amp;lt;- matrix(rnorm(n*p), n, p)&#xA;&#xA;fx01 &amp;lt;- function(ll, x0) tcrossprod(x0, rnorm(ncol(x0)))&#xA;fx02 &amp;lt;- function(ll, x1) crossprod(t(x1), rnorm(ncol(x1)))&#xA;&#xA;system.time(lapply(1:1000, fx01, x0=x0))&#xA;system.time(lapply(1:1000, fx02, x1=x1))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is it normal that the first function takes 4-5 times more time to run than the second when the second does not use a sparse matrix ?&lt;/p&gt;&#xA;" OwnerUserId="603" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-02-08T08:59:04.167" LastActivityDate="2011-02-08T08:59:04.167" Title="Function timings in the Matrix package" Tags="&lt;r&gt;&lt;optimization&gt;&lt;matrix&gt;" AnswerCount="1" />
  <row Id="6972" PostTypeId="1" AcceptedAnswerId="6985" CreationDate="2011-02-07T22:56:45.337" Score="2" ViewCount="165" Body="&lt;p&gt;X is the weight of a baby when born (in gram). If the distribution of X is N(3315, 575) and Y is the number of babies with weight lower than 2719 in a random sample of 25, then P[Y≤4] is about...?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm trying to figure how to solve this. What kind of formula do I use? I suppose N is a normal distribution, that 3315 is the mean and that 575 the variance. Please correct me if I'm wrong.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: As stated in the comments 575 should be the standard deviation. I'm not interested in the answer like someone suggested. If you teach me how should I approach this it is more than enough.&lt;/p&gt;&#xA;" OwnerUserId="1833" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-02-08T06:04:57.733" LastActivityDate="2011-02-08T08:31:43.127" Title="Normal distribution and weight of babies problem" Tags="&lt;probability&gt;&lt;homework&gt;&lt;binomial&gt;&lt;normal-distribution&gt;" AnswerCount="1" CommentCount="7" FavoriteCount="3" />
  <row Id="6973" PostTypeId="2" ParentId="6967" CreationDate="2011-02-08T00:26:12.043" Score="6" ViewCount="" Body="&lt;p&gt;You need to write a function called &lt;code&gt;getSummary.Arima&lt;/code&gt; for use with the &lt;code&gt;memisc&lt;/code&gt; package. Check out &lt;code&gt;getSummary.lm&lt;/code&gt; for an example.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2011-02-08T00:26:12.043" CommentCount="3" />
  <row Id="6974" PostTypeId="1" AcceptedAnswerId="6977" CreationDate="2011-02-08T01:57:28.050" Score="4" ViewCount="204" Body="&lt;p&gt;My problem is this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are 52 people divided into 5 groups: A, B, C, D, and E. From stage 1 to stage 2, they either stay in the same groups, or change groups. The following table shows how many people change from their stage 1 group to stage 2 group for all pairs.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Stage 1                   Stage 2&#xA;         Group A   Group B   Group C   Group D   Group E&#xA;---------------------------------------------------------&#xA;Group A    18         0        0         0         2&#xA;Group B     0         6        0         0         3&#xA;Group C     4         0        2         0         2&#xA;Group D     0         0        0         0         0&#xA;Group E     4         2        0         1         8&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;E.g. 18 people change from group A to group A (i.e. they retain their group); 2 + 3 + 2 people change from groups A, B, C, D to group E.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, my claims are these:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;65% people (18 + 6 + 2 + 0 + 8 = 34 out of the total 52, the diagonal) do not change groups.&lt;/li&gt;&#xA;&lt;li&gt;15% people (8 out of 52) change from some other group to group A.&lt;/li&gt;&#xA;&lt;li&gt;13% people (7 out of 52) change from some other group to group E.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I want to attest these claims with the correct statistical test. Which one is the right one to use? I'm aware the numbers are small, so any test might be unreliable, but I'm curious to know what test it will be any way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas?&lt;/p&gt;&#xA;" OwnerUserId="3097" LastEditorUserId="3097" LastEditorDisplayName="" LastEditDate="2011-02-08T02:08:40.863" LastActivityDate="2011-02-09T04:24:16.253" Title="What is the right statistical test to use when attesting claims like &quot;60% people switched over from group A to group B&quot;?" Tags="&lt;hypothesis-testing&gt;&lt;classification&gt;&lt;statistical-significance&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="1" />
  <row Id="6975" PostTypeId="1" AcceptedAnswerId="7001" CreationDate="2011-02-08T02:23:56.847" Score="3" ViewCount="81" Body="&lt;p&gt;I'm trying to figure out the best way of creating groups in a dataset with many dimensions. I have 1000 measurements, and each measurement has 40 dimensions. The measurements are of neighborhoods with physical and socio-economic data. For example I have measures like the number of people in each area, the population density, the number of buildings, average household size etc, average income. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm quite new to this field, so I'm not sure what is the best way of going about this. Two thoughts I had were using CART (specifically the &lt;em&gt;rpart&lt;/em&gt; package in &lt;em&gt;R&lt;/em&gt;), principal component analysis, and k-means clustering. Ideally I would do all of this analysis in &lt;em&gt;R&lt;/em&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to end up with 6 to 8 &quot;typologies&quot; or representative groupings of the data. They do not necessarily have to include all 40 dimensions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would appreciate any advice on the best way of doing this.&lt;/p&gt;&#xA;" OwnerUserId="2635" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-02-08T16:21:20.827" LastActivityDate="2011-02-08T21:25:23.370" Title="Creating groups with multi-dimensional data?" Tags="&lt;classification&gt;&lt;pca&gt;&lt;factor-analysis&gt;&lt;cart&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="6976" PostTypeId="1" CreationDate="2011-02-08T02:24:03.660" Score="8" ViewCount="185" Body="&lt;p&gt;This question might sound very broad, but here is what I am looking for. I know there are many excellent books about econometric methods, and many excellent expository articles about econometric techniques. There are even excellent &lt;em&gt;reproducible&lt;/em&gt; examples of econometrics, as described in this CrossValidated &lt;a href=&quot;http://stats.stackexchange.com/questions/1980/complete-substantive-examples-of-reproducible-research-using-r&quot;&gt;question&lt;/a&gt;. In fact the examples in this question come very close to what I am looking for; the only thing missing in those examples is that they are only &lt;em&gt;research&lt;/em&gt; reports, without any mention of how the results of the study fared in a &lt;em&gt;real-world application&lt;/em&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I am looking for are documented/reproducible examples of real-world applications of econometric theory that ideally have the following characteristics:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;They should be &lt;strong&gt;reproducible&lt;/strong&gt;, i.e. contain detailed description of (and pointers to) data, econometric techniques, and code. Ideally the code would be in the R language.&lt;/li&gt;&#xA;&lt;li&gt;There should be detailed documentation showing that the technique &lt;strong&gt;succeeded in the real world&lt;/strong&gt;, according to a well-quantified measure of success (e.g. &quot;the technique helped increase revenues because it enabled improved forecasting of demand, and here are the numbers involved&quot;)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I am using the term &lt;em&gt;econometric&lt;/em&gt; quite broadly here -- I mean any sort of &lt;em&gt;data-mining&lt;/em&gt;, &lt;em&gt;statistical data-analysis&lt;/em&gt;, &lt;em&gt;predictiion&lt;/em&gt;, &lt;em&gt;forecasting&lt;/em&gt; or &lt;em&gt;machine-learning&lt;/em&gt; technique.&#xA;One immediate problem in finding such examples:  many successful applications of econometrics are done in a for-profit setting and are therefore proprietary, so if a technique worked well, it probably will &lt;em&gt;not&lt;/em&gt; be published (this is especially true in the case of proprietary trading strategies), but I am nevertheless hoping there are published examples that have at least property (2) above if not both (1) and (2).&lt;/p&gt;&#xA;" OwnerUserId="2544" LastActivityDate="2011-02-08T12:46:40.163" Title="Documented/reproducible examples of successful real-world applications of econometric methods?" Tags="&lt;r&gt;&lt;machine-learning&gt;&lt;forecasting&gt;&lt;data-mining&gt;&lt;econometrics&gt;" AnswerCount="1" CommentCount="6" FavoriteCount="2" />
  <row Id="6977" PostTypeId="2" ParentId="6974" CreationDate="2011-02-08T02:55:24.297" Score="6" ViewCount="" Body="&lt;p&gt;I think what you're probably looking for are the so-called &lt;strong&gt;mover-stayer&lt;/strong&gt; models. The basic model is a loglinear model that allows for an &quot;inflated&quot; diagonal and usually involves some form of symmetry on the off-diagonals.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A good place to start is A. Agresti, &lt;em&gt;Categorical Data Analysis&lt;/em&gt;, 2nd ed., pp. 423--428. (That's the big green book, not the smaller blue or white/brown one with almost the same name!)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might also look at J. K. Lindsey, &lt;em&gt;Applying generalized linear models&lt;/em&gt;, Springer, 1997, pp. 33--35.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Addendum&lt;/strong&gt;: The log-linear version of the &lt;strong&gt;mover-stayer&lt;/strong&gt; model is also often referred to as a &lt;strong&gt;quasi-independence&lt;/strong&gt; model. Two forms are popular. The more general form is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;\log \mu_{ij} = \mu + \alpha_i + \beta_j + \delta_i \mathbf{1}_{(i = j)} ,&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\mathbf{1}_{(x \in A)}$ is the indicator function that $x \in A$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's interpret this model. First off, if we drop the last term, then we recover the model for independence of the rows and columns. This would mean assuming that people change groups in such a way that the beginning and ending groups were independent of each other, even though each particular group (before and after) may have different marginal distributions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By throwing in the $\delta_i$ into the model, we are instead assuming there are two groups in the population. There are the &lt;strong&gt;movers&lt;/strong&gt; who will tend to move from one group to another distinct group and there are the &lt;strong&gt;stayers&lt;/strong&gt; who tend to be happy with their first choice. When the population breaks down into two such groups, we expect to see an inflated diagonal due to the &lt;strong&gt;stayers&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The model is also called a &lt;strong&gt;quasi-independence&lt;/strong&gt; model since it is a model for independence &lt;em&gt;on the off-diagonals&lt;/em&gt;. This means that, among the &lt;strong&gt;movers&lt;/strong&gt;, there is no interaction between the group they started in and the one they end up in (&quot;&lt;strong&gt;drifters&lt;/strong&gt;&quot; might be a more evocative term).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Notice that if we have a separate $\delta_i$ for each $i$, then the diagonal cells will be fit perfectly by this model. A common variant is to replace $\delta_i$ by $\delta$, i.e., each group has a common affinity to &quot;stay&quot; versus &quot;move&quot;.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Here is some $R$ code to fit such a model&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Mover-stayer model example&#xA;&#xA;Y &amp;lt;- rbind( c(18,0,0,0,2),&#xA;            c( 0,6,0,0,3),&#xA;            c( 4,0,2,0,2),&#xA;            c( 0,0,0,0,0),&#xA;            c( 4,2,0,1,8))&#xA;&#xA;grp &amp;lt;- c(&quot;A&quot;, &quot;B&quot;, &quot;C&quot;, &quot;D&quot;, &quot;E&quot;)&#xA;colnames(Y) &amp;lt;- grp&#xA;rownames(Y) &amp;lt;- grp&#xA;&#xA;X &amp;lt;- expand.grid( row=grp, col=grp)&#xA;X &amp;lt;- cbind( X, sapply(grp, function(r) { X[,1]==r &amp;amp; X[,2]==r }) )&#xA;&#xA;y &amp;lt;- as.vector(Y)&#xA;&#xA;df &amp;lt;- data.frame(y,X)&#xA;&#xA;# General model&#xA;move.stay &amp;lt;- glm(y~., family=&quot;poisson&quot;, data=df)&#xA;move.stay.summary &amp;lt;- summary(move.stay)&#xA;&#xA;# Common diagonal term&#xA;move.stay.const &amp;lt;- glm(y~row+col+I(A+B+C+D+E), family=&quot;poisson&quot;, data=df)&#xA;move.stay.const.summary &amp;lt;- summary(move.stay.const)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;At this point, you can do lots of things. If you need a formal hypothesis test, than a deviance test might be appropriate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your particular data certainly has the peculiarity of a row of all zeros. It's interesting that there was a &quot;group&quot; choice that contained no one in it to start with.&lt;/p&gt;&#xA;" OwnerUserId="2970" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-02-09T04:24:16.253" LastActivityDate="2011-02-09T04:24:16.253" CommentCount="5" />
  <row Id="6978" PostTypeId="1" CreationDate="2011-02-08T04:09:23.683" Score="9" ViewCount="127" Body="&lt;p&gt;This was inspired by &lt;a href=&quot;http://stats.stackexchange.com/questions/6920/efficient-online-linear-regression&quot;&gt;Efficient online linear regression&lt;/a&gt;, which I found very interesting.  Are there any texts or resources devoted to large-scale statistical computing, by which computing with datasets too large to fit in main memory, and perhaps too varied to effectively subsample.  For example, is it possible to fit mixed effects models in an online fashion?  Has anyone looked into the effects of replacing the standard 2nd order optimization techniques for MLE with 1st order, SGD-type techniques?&lt;/p&gt;&#xA;" OwnerUserId="1777" LastActivityDate="2011-02-08T04:09:23.683" Title="Online, scalable statistical methods" Tags="&lt;online&gt;&lt;computing&gt;&lt;resources&gt;" CommentCount="4" FavoriteCount="3" />
  <row Id="6979" PostTypeId="2" ParentId="6966" CreationDate="2011-02-08T05:07:01.333" Score="1" ViewCount="" Body="&lt;p&gt;Hypothesis testing is a useful way to frame a lot of questions: is the effect of a treatment zero or nonzero? The ability between statements such as these and a statistical model or procedure (including the construction of an interval estimator) is important for practitioners I think.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It also bears mentioning that a confidence interval (in the traditional sense) isn't inherently any less &quot;sin-prone&quot; than hypothesis testing - how many intro stats students know the real definition of a confidence interval?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Perhaps the problem isn't hypothesis testing or interval estimation as it is the classical versions of the same; the Bayesian formulation avoids these quite nicely.&lt;/p&gt;&#xA;" OwnerUserId="26" LastActivityDate="2011-02-08T05:07:01.333" CommentCount="11" />
  <row Id="6980" PostTypeId="2" ParentId="6966" CreationDate="2011-02-08T05:10:14.317" Score="8" ViewCount="" Body="&lt;p&gt;This is my personal opinion, so I'm not sure it properly qualifies as an &quot;answer&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Why should we teach hypothesis testing?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One very big reason, in short, is that, in all likelihood, in the time it takes you to read this sentence, hundreds, if not thousands (or millions) of hypothesis tests have been conducted within a 10ft radius of where you sit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your cell phone is &lt;strong&gt;definitely&lt;/strong&gt; using a likelihood ratio test to decide whether or not it is within range of a base station. Your laptop's WiFi hardware is doing the same in communicating with your router.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The microwave you used to auto-reheat that two-day old piece of pizza used a hypothesis test to decide when your pizza was hot enough.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your car's traction control system kicked in when you gave it too much gas on an icy road, or the tire-pressure warning system let you know that your rear passenger-side tire was abnormally low, and your headlights came on automatically at around 5:19pm as dusk was setting in.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your iPad is rendering this page in landscape format based on (noisy) accelerometer readings.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your credit card company shut off your card when &quot;you&quot; purchased a flat-screen TV at a Best Buy in Texas and a $2000 diamond ring at Zales in a Washington-state mall within a couple hours of buying lunch, gas, and a movie near your home in the Pittsburgh suburbs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The hundreds of thousands of bits that were sent to render this webpage in your browser each &lt;em&gt;individually&lt;/em&gt; underwent a hypothesis test to determine whether they were most likely a 0 or a 1 (in addition to some amazing error-correction).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Look to your right just a little bit at those &quot;related&quot; topics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All of these things &quot;happened&quot; due to &lt;strong&gt;hypothesis tests&lt;/strong&gt;. For many of these things some interval estimate of some parameter could be calculated. But, especially for automated industrial processes, the use and understanding of hypothesis testing is crucial.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;On a more theoretical statistical level, the important concept of statistical power arises rather naturally from a decision-theoretic / hypothesis-testing framework. Plus, I believe &quot;even&quot; a pure mathematician can appreciate the beauty and simplicity of the Neyman--Pearson lemma and its proof.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is not to say that hypothesis testing is taught, or understood, well. By and large, it's not. And, while I would agree that---particularly in the medical sciences---reporting of interval estimates along with effect sizes and notions of practical vs. statistical significance are almost universally preferable to any formal hypothesis test, this does not mean that hypothesis testing and the related concepts are not important and interesting in their own right.&lt;/p&gt;&#xA;" OwnerUserId="2970" LastActivityDate="2011-02-08T05:10:14.317" CommentCount="1" />
  <row Id="6981" PostTypeId="2" ParentId="6966" CreationDate="2011-02-08T05:41:56.397" Score="0" ViewCount="" Body="&lt;p&gt;In teaching Neyman Pearson hypothesis testing to early statistics students, I have often tried to locate it in its original setting: that of making decisions.  Then the infrastructure of type 1 and type 2 errors all makes sense, as does the idea that you might &lt;em&gt;accept&lt;/em&gt; the null hypothesis.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We have to make a decision, we think that the outcome of our decision can be improved by knowledge of a parameter, we only have an estimate of that parameter.  We still have to make a decision.  Then what is the best decision to make in the context of having an estimate of the parameter?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems to me that in its original setting (making decisions in the face of uncertainty) the NP hypothesis test makes perfect sense.  See e.g. N &amp;amp; P 1933, particularly p. 291.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Neyman and Pearson. On the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character (1933) vol. 231 pp. 289-337&lt;/p&gt;&#xA;" OwnerUserId="187" LastActivityDate="2011-02-08T05:41:56.397" CommentCount="1" />
  <row Id="6982" PostTypeId="2" ParentId="6974" CreationDate="2011-02-08T06:55:16.950" Score="0" ViewCount="" Body="&lt;p&gt;I would start with building the sums:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Stage 1                   Stage 2&#xA;            A   B   C   D   E | Sum&#xA;-----------------------------------&#xA;Group A    18   0   0   0   2 |  20&#xA;Group B     0   6   0   0   3 |   9&#xA;Group C     4   0   2   0   2 |   8&#xA;Group D     0   0   0   0   0 |   0&#xA;Group E     4   2   0   1   8 |  15&#xA;-----------------------------------&#xA;Sum        26   8   2   1  15 |  52 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As long as you don't tell us, that this is just a sample from a bigger population, I don't see any statistic involved, just frequency distributions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;From A to B went 0 person, not much to calculate here. 2 Persons went from A to E. A had initially 20 members (am I right?) and 2/20 is 10%. 10 Percent of the A group went to E. How many people have been later in group A, or where in total is not of interest here. &#xA;If you ask 'How many from group E came from A, then you would use the total of E in stage 2; there are 15 people, two of them came from A: 2/15.&lt;/p&gt;&#xA;" OwnerUserId="3100" LastActivityDate="2011-02-08T06:55:16.950" />
  <row Id="6983" PostTypeId="2" ParentId="6966" CreationDate="2011-02-08T08:12:23.203" Score="0" ViewCount="" Body="&lt;p&gt;I think it depends on &lt;em&gt;which&lt;/em&gt; hypothesis testing you are talking about.  The &quot;classical&quot; hypothesis testing (Neymann-Pearson) is said to be defective because it does not appropriately condition on what &lt;em&gt;actually happened when you did the test&lt;/em&gt;.  It instead is design to work &quot;regardless&quot; of what you actually saw in the long run.  But failing to condition can lead to misleading results in the individual case.  This is simply because the procedure &quot;does not care&quot; about the individual case, on the long run.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hypothesis testing can be cast in the decision theoretical framework, which I think is a much better way to understand it.  You can restate the problem as two decisions:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) &quot;I will act as if $H_0$ is true&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) &quot;I will act as if $H_A$ is true&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The decision framework is much easier to understand, because it clearly separates out the concepts of &quot;what you will do?&quot; and &quot;what is the truth?&quot; (via your prior information)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could even apply &quot;decision theory&quot; (DT) to your question.  But in order to stop hypothesis testing, DT says you must have an alternative decision available to you.  So the question is: if hypothesis testing is abandoned, what is to take its place?  I can't think of an answer to this question.  I can only think of alternative ways to do hypothesis testing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(NOTE: in the context of hypothesis testing, the data, sampling distribution, prior distribution, and loss function are all &lt;em&gt;prior information&lt;/em&gt; because they are obtained &lt;em&gt;prior&lt;/em&gt; to making the decision)&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-02-08T08:12:23.203" CommentCount="1" />
  <row Id="6984" PostTypeId="2" ParentId="6971" CreationDate="2011-02-08T08:29:44.753" Score="3" ViewCount="" Body="&lt;p&gt;There is an error in definition of &lt;code&gt;fx01&lt;/code&gt;, the matrices are not compatible the way you defined. If we define&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;fx01 &amp;lt;- function(ll, x0) tcrossprod(rnorm(ncol(x0)),x0)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then on my Macbook Pro with 2.53 GHz Intel Core 2 Duo processor with 4GB RAM and R 2.11.1, I get&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; system.time(lapply(1:1000, fx02, x1=x1))&#xA;   user  system elapsed &#xA;  0.459   0.092   0.546 &#xA;&amp;gt; system.time(lapply(1:1000, fx02, x1=x0))&#xA;   user  system elapsed &#xA;  0.632   0.005   0.633 &#xA;&amp;gt; system.time(lapply(1:1000, fx01, x0=x1))&#xA;   user  system elapsed &#xA;  0.078   0.001   0.079 &#xA;&amp;gt; system.time(lapply(1:1000, fx01, x0=x0))&#xA;   user  system elapsed &#xA;  3.810   0.014   3.805 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now this might give a hint (I cropped the output to make it clearer):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; showMethods(&quot;crossprod&quot;)&#xA;x=&quot;dgCMatrix&quot;, y=&quot;numeric&quot;&#xA;    (inherited from: x=&quot;CsparseMatrix&quot;, y=&quot;numeric&quot;)&#xA;&#xA;&amp;gt; showMethods(&quot;tcrossprod&quot;)&#xA;x=&quot;numeric&quot;, y=&quot;dgCMatrix&quot;&#xA;    (inherited from: x=&quot;numeric&quot;, y=&quot;Matrix&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So I suspect that in &lt;code&gt;tcrossprod&lt;/code&gt; there is a conversion of &lt;code&gt;dgCMatrix&lt;/code&gt; (class of &lt;code&gt;x0&lt;/code&gt;) to &lt;code&gt;Matrix&lt;/code&gt;, meaning we convert sparse matrix to usual matrix. There is no such conversion in case of &lt;code&gt;crossproduct&lt;/code&gt;. This might explain the difference, since sometimes unnecessary conversions are costly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Further inspection confirms this, some sparse matrix code is used:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; getMethod(&quot;crossprod&quot;,signature=c(x=&quot;CsparseMatrix&quot;, y=&quot;numeric&quot;))&#xA;Method Definition:&#xA;&#xA;function (x, y = NULL) &#xA;.Call(Csparse_dense_crossprod, x, y)&#xA;&amp;lt;environment: namespace:Matrix&amp;gt;&#xA;&#xA;Signatures:&#xA;        x               y        &#xA;target  &quot;CsparseMatrix&quot; &quot;numeric&quot;&#xA;defined &quot;CsparseMatrix&quot; &quot;numeric&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;No sparse matrix code is used:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; getMethod(&quot;tcrossprod&quot;,signature=c(x=&quot;numeric&quot;, y=&quot;Matrix&quot;))&#xA;Method Definition:&#xA;&#xA;function (x, y = NULL) &#xA;{&#xA;    dim(x) &amp;lt;- c(1L, length(x))&#xA;    callGeneric(x, y)&#xA;}&#xA;&amp;lt;environment: namespace:Matrix&amp;gt;&#xA;&#xA;Signatures:&#xA;        x         y       &#xA;target  &quot;numeric&quot; &quot;Matrix&quot;&#xA;defined &quot;numeric&quot; &quot;Matrix&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Side note:&lt;/em&gt; This is a first time I managed to get code from S4 methods (thanks for the question which forced me to figure it out), so I still may be wrong.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-02-08T08:43:50.513" LastActivityDate="2011-02-08T08:43:50.513" CommentCount="6" />
  <row Id="6985" PostTypeId="2" ParentId="6972" CreationDate="2011-02-08T08:31:43.127" Score="2" ViewCount="" Body="&lt;p&gt;It may be easier to first consider equality for general $Y=y$ and the to &quot;add up&quot; for $y=0,1,2,3,4$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because the question as stated does not assume any (logical) dependence between babies in the sample, we can say the all have the same probability of being under the threshold (because we do not know how to differentiate them, given what information there is in the question).  Call this probability $P=Pr[X&amp;lt;2719|I]$ (where $I$ denotes the information in the question).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now this is basically a binomial probability that you are asking for, with $25$ samples, and the probability that you see $y$ exactly is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Pr[Y=y|I]=\frac{25!}{y!(25-y)!}P^y (1-P)^{25-y}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And so you simply add this up for your answer:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Pr[Y\leq 4|I]=\sum_{y=0}^{4}\frac{25!}{y!(25-y)!}P^y (1-P)^{25-y}$$&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-02-08T08:31:43.127" CommentCount="1" />
  <row Id="6986" PostTypeId="1" AcceptedAnswerId="6992" CreationDate="2011-02-08T12:25:54.363" Score="3" ViewCount="119" Body="&lt;p&gt;If I have a dataset of $N$ pairs ($x_i$,$y_i$) where each $y_i$ has an individual error $\sigma_{y_i}$, I can incorporate this into regression by using the inverse of this as weights.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I now do logarithmic regression $\ln y_i = b_0 + b_1\,\ln x_i + \epsilon_i$, how do I incorporate now the errors? Are the weights just plain the same as in the linear case or do I have to convert them using the lognormal distribution: $Var = \mathrm{e}^{2\mu+\sigma^{2}}(\mathrm{e}^{\sigma^{2}}-1)$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for your help.&lt;/p&gt;&#xA;" OwnerUserId="3104" LastActivityDate="2011-02-08T16:34:48.667" Title="Logarithmic regression with individual errors" Tags="&lt;regression&gt;&lt;lognormal&gt;" AnswerCount="2" />
  <row Id="6987" PostTypeId="2" ParentId="6976" CreationDate="2011-02-08T12:46:40.163" Score="2" ViewCount="" Body="&lt;p&gt;As mentioned in the comments, travel demand forecasts often use inputs from discrete choice models (multinomial logit, nested logit, mixed logit, etc) to assist in the development of behavioral mode choice or route assignment in their travel demand forecasts. DCM certainly has many applications outside travel demand forecasting, but it has been used in the transportation industry for 30+ years so there should be lots of good examples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for reproducible examples:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://biogeme.epfl.ch/swissmetro.php&quot; rel=&quot;nofollow&quot;&gt;Biogeme&lt;/a&gt; is an open source piece of software that is optimized for estimating logit models. The website provides the data, code, and a paper write up describing their methods.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://r-forge.r-project.org/forum/forum.php?forum_id=2439&quot; rel=&quot;nofollow&quot;&gt;travelR&lt;/a&gt; is a project to make travel demand forecasts with R. There was a presentation at useR! 2010 about the project, abstract &lt;a href=&quot;http://user2010.org/abstracts/Raw.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; and slides &lt;a href=&quot;http://user2010.org/slides/Raw.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;. There is also a webinar coming up in a few weeks regarding R and travel demand forecasting that I'll find the link to and update here for those interested.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://amonline.trb.org/&quot; rel=&quot;nofollow&quot;&gt;Transportation Review Board Conference&lt;/a&gt; has made a list of all papers available online this year. I don't have a specific paper to link to, but there are several committee's worth of papers regarding the application of choice models in the transportation context.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="696" LastActivityDate="2011-02-08T12:46:40.163" CommentCount="2" />
  <row Id="6988" PostTypeId="2" ParentId="58" CreationDate="2011-02-08T12:47:57.683" Score="7" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;The back propagation algorithm is a &lt;a href=&quot;http://en.wikipedia.org/wiki/Gradient_descent&quot; rel=&quot;nofollow&quot;&gt;gradient descent&lt;/a&gt; algorithm for fitting a neural network model.&lt;/strong&gt; (as mentionned by @Dikran) Let me explain how. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Formally:&lt;/strong&gt; Using the calculation of the gradient at the end of this post within equation [1] below (that is a definition of the gradient descent) gives the back propagation algorithm as a particular case of the use of a gradient descent.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;A neural network model&lt;/strong&gt;&#xA;Formally, we fix ideas with a simple single layer model:  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ f(x)=g(A^1(s(A^2(x)))) $$&#xA;where $g:\mathbb{R} \rightarrow \mathbb{R}$ and $s:\mathbb{R}^M\rightarrow \mathbb{R}^M$ are known with for all $m=1\dots,M$, $s(x)[m]=\sigma(x[m])$,  and $A^1:\mathbb{R}^M\rightarrow \mathbb{R}$, $A^2\mathbb{R}^p\rightarrow \mathbb{R}^M$ are unknown affine functions. The function $\sigma:\mathbb{R}\rightarrow \mathbb{R}$ is called activation function in the framework of classification. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;A quadratic Loss function&lt;/strong&gt; is taken to fix ideas. &#xA;Hence the input $(x_1,\dots,x_n)$ vectors of $\mathbb{R}^p$ can be fitted to the real output  $(y_1,\dots,y_n)$ of $\mathbb{R}$ (could be vectors) by minimizing the empirical loss: &#xA; $$\mathcal{R}_n(A^1,A^2)=\sum_{i=1}^n (y_i-f(x_i))^2\;\;\;\;\;\;\; [1]$$&#xA;with respect to the choice of $A^1$ and $A^2$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Gradient_descent&quot; rel=&quot;nofollow&quot;&gt;Gradient descent&lt;/a&gt;&lt;/strong&gt;&#xA;A grandient descent for minimizing $\mathcal{R}$ is an algorithm that iterate:&#xA;    $$\mathbf{a}_{l+1}=\mathbf{a}_l-\gamma_l \nabla \mathcal{R}(\mathbf{a}_l),\ l \ge 0.$$&#xA;for well chosen step sizes $(\gamma_l)_l$ (also called learning rate in the framework of back propagation). It requires the calculation of the gradient of $\mathcal{R}$. In the considered case $\mathbf{a}_l=(A^1_{l},A^2_{l})$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Gradient of $\mathcal{R}$&lt;/strong&gt; (for the simple considered neural net model) &#xA;Let us denote,  by $\nabla_1 \mathcal{R}$ the gradient of $\mathcal{R}$ as a function of $A^1$, and $\nabla_2\mathcal{R}$ the gradient of $\mathcal{R}$ as a function of $A^2$. Standard calculation (using the rule for derivation of composition of functions) and the use of the notation $z_i=A^1(s(A^2(x_i)))$ give&#xA;$$\nabla_1 \mathcal{R}[1:M] =-2\times \sum_{i=1}^n z_i g&amp;#39;(z_i) (y_i-f(x_i))$$&#xA;for all $m=1,\dots,M$&#xA; $$\nabla_2 \mathcal{R}[1:p,m] =-2\times \sum_{i=1}^n x_i g&amp;#39;(z_i) z_i[m]\sigma^&amp;#39;(A^2(x_i)[m]) (y_i-f(x_i))$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here I used the R notation: $x[a:b]$ is the vector composed of the coordinates of $x$ from index $a$ to index $b$.&lt;/p&gt;&#xA;" OwnerUserId="223" LastEditorUserId="223" LastEditorDisplayName="" LastEditDate="2011-02-17T12:16:10.580" LastActivityDate="2011-02-17T12:16:10.580" />
  <row Id="6989" PostTypeId="1" AcceptedAnswerId="6994" CreationDate="2011-02-08T13:02:55.313" Score="7" ViewCount="288" Body="&lt;p&gt;I want to fit a multilevel GLMM with a Poisson distribution (with over-dispersion) using R. At the moment I am using &lt;a href=&quot;http://cran.r-project.org/web/packages/lme4/index.html&quot; rel=&quot;nofollow&quot;&gt;lme4&lt;/a&gt; but I noticed that recently the &lt;code&gt;quasipoisson&lt;/code&gt; family was removed. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've seen elsewhere that you can model additive over-dispersion for binomial distributions by adding a random intercept with one level per observation. Does this apply to the poisson distribution as well? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a better way to do it? Are there other packages that you would recommend? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks.&lt;/p&gt;&#xA;" OwnerUserId="1871" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-18T09:15:24.363" LastActivityDate="2011-04-18T09:15:24.363" Title="How do I fit a multilevel model for over-dispersed poisson outcomes? " Tags="&lt;r&gt;&lt;mixed-model&gt;&lt;poisson&gt;&lt;lmer&gt;&lt;overdispersion&gt;" AnswerCount="3" FavoriteCount="2" />
  <row Id="6990" PostTypeId="1" CreationDate="2011-02-08T13:16:39.953" Score="1" ViewCount="252" Body="&lt;p&gt;How do I carry out a repeated measures analysis using R?&lt;/p&gt;&#xA;" OwnerUserId="3107" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-02-09T04:55:12.440" LastActivityDate="2011-02-09T06:40:13.363" Title="Repeated measures using R" Tags="&lt;r&gt;&lt;anova&gt;&lt;repeated-measures&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="2" />
  <row Id="6991" PostTypeId="2" ParentId="6986" CreationDate="2011-02-08T15:26:07.017" Score="3" ViewCount="" Body="&lt;p&gt;It is good to remember what is the purpose of the weighting. Suppose we have linear regression&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align}&#xA;\mathbf{y}=\mathbf{X}\mathbf{\beta}+\mathbf{\varepsilon}&#xA;\end{align}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this case $\sigma_{y_i}=\sigma_{\varepsilon_i}=\sigma_i$ and we can write that &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\Omega=E(\varepsilon\varepsilon&amp;#39;)=diag(\sigma_i)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now weighted regression with weights inverse to $\sigma_i$ is &lt;a href=&quot;http://en.wikipedia.org/wiki/Generalized_least_squares&quot; rel=&quot;nofollow&quot;&gt;generalised least squares estimate&lt;/a&gt; with $\Omega$. Generalised least squares estimates are best linear unbiased estimates if variance of errors is $\Omega$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So you need to know the variance of errors, not the $y$. If you fit logarithmic regression and you know that $y$ is log normal, you need to weight with the variance of $\log y$, which in your case will correspond to variance of $\epsilon$. Then your weighted least squares estimates will be best linear unbiased estimates, and you will retain the same property as in linear case.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-02-08T15:26:07.017" CommentCount="2" />
  <row Id="6992" PostTypeId="2" ParentId="6986" CreationDate="2011-02-08T16:34:48.667" Score="2" ViewCount="" Body="&lt;p&gt;If the variance of $y_i$ is $\sigma_i^2,$ then, by the &lt;a href=&quot;http://en.wikipedia.org/wiki/Delta_method&quot; rel=&quot;nofollow&quot;&gt;delta method&lt;/a&gt;, the variance of $\log(y_i)$ is approximately $\sigma_i^2 / y_i^2 .$ So if you're doing &lt;a href=&quot;http://en.wikipedia.org/wiki/Weighted_least_squares#Weighted_least_squares&quot; rel=&quot;nofollow&quot;&gt;weighted least squares&lt;/a&gt; using inverse-variance weights, instead of weighting by $1/\sigma_i^2 ,$ you need to weight by $y_i^2 / \sigma_i^2 .$&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-02-08T16:34:48.667" CommentCount="2" />
  <row Id="6993" PostTypeId="2" ParentId="6989" CreationDate="2011-02-08T17:01:35.940" Score="2" ViewCount="" Body="&lt;p&gt;I think that the glmmADMB package is exactely what you are looking for.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;install.packages(&quot;glmmADMB&quot;,&#xA;  repos=&quot;http://r-forge.r-project.org&quot;)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;But in a bayesian point of view you can use the &lt;strong&gt;MCMCglmm&lt;/strong&gt; package or the &lt;strong&gt;BUGS/JAGS&lt;/strong&gt; software, they are very flexible and you can fit this kind of model. (and the syntax is close to the R one)&lt;/p&gt;&#xA;" OwnerUserId="2028" LastActivityDate="2011-02-08T17:01:35.940" />
  <row Id="6994" PostTypeId="2" ParentId="6989" CreationDate="2011-02-08T17:06:29.380" Score="8" ViewCount="" Body="&lt;p&gt;Hello George:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can fit multilevel GLMM with a Poisson distribution (with over-dispersion) using R in multiple ways. Few &lt;code&gt;R&lt;/code&gt; packages are: &lt;code&gt;lme4&lt;/code&gt;, &lt;code&gt;MCMCglmm&lt;/code&gt;, &lt;code&gt;arm&lt;/code&gt;, etc. A good reference to see is &lt;a href=&quot;http://www.stat.columbia.edu/~gelman/arm/&quot; rel=&quot;nofollow&quot;&gt;Gelman and Hill (2007)&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I will give an example of doing this using &lt;code&gt;rjags&lt;/code&gt; package in &lt;code&gt;R&lt;/code&gt;. It is an interface between &lt;code&gt;R&lt;/code&gt; and &lt;code&gt;JAGS&lt;/code&gt; (like &lt;code&gt;OpenBUGS&lt;/code&gt; or &lt;code&gt;WinBUGS&lt;/code&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$n_{ij} \sim \mathrm{Poisson}(\theta_{ij})$$&#xA;$$\log \theta_{ij} = \beta_0 + \beta_1 \mbox{ } \mathtt{Treatment}_{i} + \delta_{ij}$$&#xA;$$\delta_{ij} \sim N(0, \sigma^2_{\epsilon})$$ &#xA;$$i=1 \ldots I, \quad j = 1\ldots J$$&#xA;$\mathtt{Treatment}_i = 0 \mbox{ or } 1, \ldots, J-1 \mbox{ if the } i^{th} \mbox{ observation belongs to treatment group } 1 \mbox{, or, } 2, \ldots, J$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The $\delta_{ij}$ part in the code above models overdispersion. But there is no one stopping you from modeling correlation between individuals (you don't believe that individuals are really independent) and within individuals (repeated measures). Also, the rate parameter may be scaled by some other constant as in &lt;code&gt;rate models&lt;/code&gt;. Please see Gelman and Hill (2007) for more reference. Here is the &lt;code&gt;JAGS&lt;/code&gt; code for the simple model:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data{&#xA;        for (i in 1:I){         &#xA;            ncount[i,1] &amp;lt;- obsTrt1[i]&#xA;            ncount[i,2] &amp;lt;- obsTrt2[i]&#xA;                ## notice I have only 2 treatments and I individuals &#xA;    }                               &#xA;}&#xA;&#xA;model{&#xA;    for (i in 1:I){ &#xA;        nCount[i, 1] ~ dpois( means[i, 1] )&#xA;        nCount[i, 2] ~ dpois( means[i, 2] )&#xA;&#xA;        log( means[i, 1] ) &amp;lt;- mu + b * trt1[i] + disp[i, 1]&#xA;        log( means[i, 2] ) &amp;lt;- mu + b * trt2[i] + disp[i, 2]&#xA;&#xA;        disp[i, 1] ~ dnorm( 0, tau)&#xA;        disp[i, 2] ~ dnorm( 0, tau)&#xA;&#xA;    }&#xA;&#xA;    mu  ~ dnorm( 0, 0.001)&#xA;    b   ~ dnorm(0, 0.001)&#xA;    tau ~ dgamma( 0.001, 0.001)&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here is the &lt;code&gt;R&lt;/code&gt; code to implement use it (say it is named: &lt;code&gt;overdisp.bug&lt;/code&gt;)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dataFixedEffect &amp;lt;- list(&quot;I&quot;       = 10,&#xA;                        &quot;obsTrt1&quot; = obsTrt1 , #vector of n_i1&#xA;                        &quot;obsTrt2&quot; = obsTrt2,  #vector of n_i2&#xA;                        &quot;trt1&quot;    = trt1,     #vector of 0&#xA;                        &quot;trt2&quot;    = trt2,     #vector of 1&#xA;                       )&#xA;&#xA;initFixedEffect &amp;lt;- list(mu = 0.0 , b = 0.0, tau = 0.01)&#xA;&#xA;simFixedEffect &amp;lt;- jags.model(file     = &quot;overdisp.bug&quot;,&#xA;                             data     = dataFixedEffect,&#xA;                             inits    = initFixedEffect,&#xA;                             n.chains = 4,&#xA;                             n.adapt  = 1000)&#xA;&#xA;sampleFixedEffect &amp;lt;- coda.samples(model          = simFixedEffect,&#xA;                                  variable.names = c(&quot;mu&quot;, &quot;b&quot;, &quot;means&quot;),&#xA;                                  n.iter         = 1000)&#xA;&#xA;meansTrt1 &amp;lt;- as.matrix(sampleFixedEffect[ , 2:11])&#xA;meansTrt2 &amp;lt;- as.matrix(sampleFixedEffect[ , 12:21])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can play around with your parameters' posteriors and you can introduce more parameters to make you modeling more precise (&lt;em&gt;we like to think this&lt;/em&gt;). Basically, you get the idea.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For more details on using &lt;code&gt;rjags&lt;/code&gt; and &lt;code&gt;JAGS&lt;/code&gt;, please see &lt;a href=&quot;http://www.johnmyleswhite.com/notebook/2010/08/20/using-jags-in-r-with-the-rjags-package/&quot; rel=&quot;nofollow&quot;&gt;John Myles White's page&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1307" LastEditorUserId="1307" LastEditorDisplayName="" LastEditDate="2011-02-08T17:15:16.543" LastActivityDate="2011-02-08T17:15:16.543" CommentCount="3" />
  <row Id="6995" PostTypeId="2" ParentId="6990" CreationDate="2011-02-08T17:47:25.943" Score="3" ViewCount="" Body="&lt;p&gt;There are &lt;a href=&quot;http://www.r-statistics.com/2010/04/repeated-measures-anova-with-r-tutorials/&quot; rel=&quot;nofollow&quot;&gt;many tutorials&lt;/a&gt; you can go through.&lt;/p&gt;&#xA;" OwnerUserId="253" LastActivityDate="2011-02-08T17:47:25.943" CommentCount="2" />
  <row Id="6996" PostTypeId="2" ParentId="6966" CreationDate="2011-02-08T17:52:56.230" Score="2" ViewCount="" Body="&lt;p&gt;If I were a hardcore Frequentist I would remind you that confidence intervals are quite regularly just inverted hypothesis tests, i.e. when the 95% interval is simply another way of describing all the points that a test involving your data wouldn't reject at the .05 level.  In these situations a preference for one over the other is question of exposition rather than method.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, exposition is important of course, but I think that would be a pretty good argument.  It's neat and clarifying to explain the two approaches as restatements of the same inference from different points of view.  (The fact that not all interval estimators &lt;em&gt;are&lt;/em&gt; inverted tests is then an inelegant but not particularly awkward fact, pedagogically speaking).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Much more serious implications come from the decision to condition on the observations, as pointed out above.  However, even in retreat the Frequentist could always observe that there are plenty of situations (perhaps not a majority) where conditioning on the observations would be unwise or unilluminating.  For those, the HT/CI setup is (not 'are') exactly what is wanted, and should be taught as such.&lt;/p&gt;&#xA;" OwnerUserId="1739" LastActivityDate="2011-02-08T17:52:56.230" CommentCount="2" />
  <row Id="6997" PostTypeId="2" ParentId="6208" CreationDate="2011-02-08T18:23:45.137" Score="0" ViewCount="" Body="&lt;p&gt;This has been an eye-opening debate on the type-II/III issue for me. Thanks for everyone's the effort in providing the discussion. I'd come round to the view of promoting type-II consistently over type-III, but had a weak grasp on the argument - I just relied on the advice in John fox's regression book (car) that advised type-III tests were rarely interpretable (well, I think he said that...).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyway- ezANOVA is really useful for permitting access to R functionality that would otherwise be impossible for undergraduates I teach in psychology. I provide online R modules, one featuring ezANOVA to demonstrate mixed ANOVA designs (though it seems pre version 3 may have been buggy for this ... doh!)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Try it out here:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.wessa.net/rwasp_Mixed%20Model%20ANOVA.wasp&quot; rel=&quot;nofollow&quot;&gt;http://www.wessa.net/rwasp_Mixed%20Model%20ANOVA.wasp&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;after the module loads (~10s) find the compute button (half way down the page) and it will run ezANOVA and associated tables and plots. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ian&lt;/p&gt;&#xA;" OwnerUserId="3112" LastActivityDate="2011-02-08T18:23:45.137" CommentCount="2" />
  <row Id="6998" PostTypeId="1" AcceptedAnswerId="6999" CreationDate="2011-02-08T18:39:48.527" Score="3" ViewCount="69" Body="&lt;p&gt;The geometric mean is the appropriate measure of central tendency for log-normally distributed variables. However, the arithmetic mean still has some use in relation to log-normal variables - in inferring totals from survey data, for instance. It is my understanding that the arithmetic mean of the sample of a log-normal distribtion is the same as the arithmetic mean of the population, with normal errors providing the sample is large enough for CLT to be invoked.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you operate a simple missing data imputation method of replacing with a mean, you would inflate the geometric mean if you used the arithmetic mean (and give high leverage to imputed data), and deflate the arithmetic mean (and therefore the total) if you used the geometric mean.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is what is most appropriate to use, the geometric mean or the arithmetic mean? Or is such a simple imputation method completely inappropriate for log-normal data?&lt;/p&gt;&#xA;" OwnerUserId="229" LastActivityDate="2011-02-08T18:48:31.850" Title="Means and imputation of log-normal variables" Tags="&lt;mean&gt;&lt;data-imputation&gt;&lt;lognormal&gt;" AnswerCount="1" />
  <row Id="6999" PostTypeId="2" ParentId="6998" CreationDate="2011-02-08T18:48:31.850" Score="5" ViewCount="" Body="&lt;p&gt;Replacing missing data with the mean is worse than useless as a method of imputation. See e.g. &lt;a href=&quot;http://missingdata.lshtm.ac.uk/index.php?option=com_content&amp;amp;view=article&amp;amp;id=68%3asimple-mean-imputation&amp;amp;catid=39%3asimple-ad-hoc-methods-for-coping-with-missing-data&amp;amp;Itemid=96&quot; rel=&quot;nofollow&quot;&gt;this page on missingdata.org.uk&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-02-08T18:48:31.850" CommentCount="1" />
  <row Id="7001" PostTypeId="2" ParentId="6975" CreationDate="2011-02-08T21:25:23.370" Score="1" ViewCount="" Body="&lt;p&gt;You may want to start using a dimensionality reduction,maybe using &lt;a href=&quot;http://en.wikipedia.org/wiki/Principal_components_analysis&quot; rel=&quot;nofollow&quot;&gt;PCA&lt;/a&gt; or &lt;a href=&quot;http://en.wikipedia.org/wiki/Multidimensional_scaling&quot; rel=&quot;nofollow&quot;&gt;MDS&lt;/a&gt;, which may give a clearer picture.&#xA;I would also try &lt;a href=&quot;http://en.wikipedia.org/wiki/Cluster_analysis&quot; rel=&quot;nofollow&quot;&gt;clustering&lt;/a&gt;, whether k-means or a more sophisticated method.&lt;/p&gt;&#xA;" OwnerUserId="2314" LastActivityDate="2011-02-08T21:25:23.370" />
  <row Id="7004" PostTypeId="1" AcceptedAnswerId="7008" CreationDate="2011-02-09T00:48:49.097" Score="4" ViewCount="440" Body="&lt;h2&gt;Background&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I have a variable with an unknown distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have 500 samples, but I would like demonstrate the precision with which I can calculate variance, e.g. to argue that a sample size of 500 is sufficient. I am also interested in knowing the minimum sample size that would be required to estimate variance with a precision of $X\%$. &lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Questions&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;How can I calculate&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;the precision of my estimate of the variance given a sample size of $n=500$? of $n=N$?&lt;/li&gt;&#xA;&lt;li&gt;How can I calculate the minimum number of samples required to estimate the variance with a precision of $X$?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;h2&gt;Example&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Figure 1&lt;/strong&gt; density estimate of the parameter based on the 500 samples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/QeKvr.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Figure 2&lt;/strong&gt; Here is a plot of sample size on the x-axis vs. estimates of variance on the y axis that I have calculated using subsamples from the sample of 500. The idea is that the estimates will converge to the true variance as n increases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the estimates are not valid independent since the samples used to estimate variance for $n \in [10,125,250,500]$ are not independent of each other or of the samples used to calculate variance at $n\in [20,40,80]$  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/bmVh4.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="2750" LastEditorUserId="2750" LastEditorDisplayName="" LastEditDate="2011-02-09T05:44:25.160" LastActivityDate="2011-02-09T13:24:07.000" Title="Calculating required sample size, precision of variance estimate?" Tags="&lt;estimation&gt;&lt;random-variable&gt;&lt;variance&gt;&lt;sampling&gt;&lt;sample-size&gt;" AnswerCount="3" FavoriteCount="1" />
  <row Id="7005" PostTypeId="2" ParentId="7004" CreationDate="2011-02-09T02:12:51.547" Score="3" ViewCount="" Body="&lt;p&gt;For i.i.d. random variables $X_1, \dotsc, X_n$, the unbiased estimator for the variance $s^2$ (the one with denominator $n-1$) has variance $\sigma^4 \left(\frac{2}{n-1} + \frac{\kappa}{n}\right)$, where $\kappa$ is the kurtosis of the distribution (reference: &lt;a href=&quot;http://en.wikipedia.org/wiki/Sample_variance#Distribution_of_the_sample_variance&quot; rel=&quot;nofollow&quot;&gt;Wikipedia&lt;/a&gt;). So now you need to estimate the kurtosis of your distribution as well. You can use a quantity sometimes described as $G_2$ for that; &lt;a href=&quot;http://en.wikipedia.org/wiki/Sample_variance#Distribution_of_the_sample_variance&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; is another Wikipedia link with a formula. I would assume that if you use $s$ as an estimate for $\sigma$ and $G_2$ as an estimate for $\kappa$, that you get a reasonable estimate for $\mathrm{Var}\  s^2$, although I don't see a guarantee that it is unbiased. See if it matches with the variance among the subsets of your 500 data points reasonably, and if it does don't worry about it anymore :)&lt;/p&gt;&#xA;" OwnerUserId="2898" LastActivityDate="2011-02-09T02:12:51.547" CommentCount="4" />
  <row Id="7007" PostTypeId="1" CreationDate="2011-02-09T03:36:52.297" Score="3" ViewCount="156" Body="&lt;h2&gt;Context:&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;A sample of people were asked to provide ratings of their impression of &lt;a href=&quot;http://en.wikipedia.org/wiki/Jeepney&quot; rel=&quot;nofollow&quot;&gt;jeepneys&lt;/a&gt; using a Likert Scale. Age, gender, household income, and so on was also obtained.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Dangerous:&lt;/strong&gt;     strongly agree; agree;  neutral; disagree;  strongly disagree&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Convenient:&lt;/strong&gt;    strongly agree; agree;  neutral; disagree;  strongly disagree&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Comfortable:&lt;/strong&gt;   strongly agree; agree;  neutral; disagree;  strongly disagree&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Age:&lt;/strong&gt;       $\quad&amp;lt;18\quad\quad$   $19-25\quad\quad$   $25\geq\quad$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Gender:&lt;/strong&gt;   male;   female&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Household Income:&lt;/strong&gt;   $\quad&amp;lt;3000\quad\quad$  $3001-5000\quad\quad$   $5000\geq\quad$&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Questions:&lt;/h2&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What statistical analyses would you recommend? &lt;/li&gt;&#xA;&lt;li&gt;How could I implement this in stata?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="3118" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-02-12T05:47:26.930" LastActivityDate="2011-02-12T05:47:26.930" Title="Statistical Analysis for Perception Survey" Tags="&lt;survey&gt;&lt;stata&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="7008" PostTypeId="2" ParentId="7004" CreationDate="2011-02-09T04:05:39.063" Score="5" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Learning a variance is hard.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It takes a (perhaps surprisingly) large number of samples to estimate a variance well in many cases. Below, I'll show the development for the &quot;canonical&quot; case of an i.i.d. normal sample.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose $Y_i$, $i=1,\ldots,n$ are independent $\mathcal{N}(\mu, \sigma^2)$ random variables. We seek a $100(1-\alpha)\%$ confidence interval for the variance such that the width of the interval is $\rho s^2$, i.e., the width is $100\rho \%$ of the point estimate. For example, if $\rho = 1/2$, then the width of the CI is half the value of the point estimate, e.g., if $s^2 = 10$, then the CI would be something like $(8,\,13)$, having a width of 5. Note the asymmetry around the point estimate, as well. ($s^2$ is the unbiased estimator for the variance.) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;The&quot; (rather, &quot;a&quot;) confidence interval for $s^2$ is&#xA;$$&#xD;&#xA;\frac{(n-1) s^2}{\chi_{(n-1)}^{2\;(1-\alpha/2)}} \leq \sigma^2 \leq \frac{(n-1) s^2}{\chi_{(n-1)}^{2\;(\alpha/2)}} \&amp;gt;,&#xD;&#xA;$$&#xA;where $\chi_{(n-1)}^{2\;\beta}$ is the $\beta$ quantile of the chi-squared distribution with $n-1$ degrees of freedom. (This arises from the fact that $(n-1)s^2/\sigma^2$ is a pivotal quantity in a Gaussian setting.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We want to minimize the width so that&#xA;$$&#xD;&#xA;L(n) = \frac{(n-1) s^2}{\chi_{(n-1)}^{2\;(\alpha/2)}} - \frac{(n-1) s^2}{\chi_{(n-1)}^{2\;(1-\alpha/2)}} &amp;lt; \rho s^2 \&amp;gt;,&#xD;&#xA;$$&#xA;so we are left to solve for $n$ such that&#xA;$$&#xD;&#xA;(n-1) \left(\frac{1}{\chi_{(n-1)}^{2\;(\alpha/2)}} - \frac{1}{\chi_{(n-1)}^{2\;(1-\alpha/2)}} \right) &amp;lt; \rho .&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the case of a 99% confidence interval, we get $n = 65$ for $\rho = 1$ and $n = 5321$ for $\rho = 0.1$. This last case yields an interval that is (&lt;em&gt;still!&lt;/em&gt;) 10% as large as the point estimate of the variance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your chosen confidence level is less than 99%, then the same width interval will be obtained for a lower value of $n$. But, $n$ may still may be larger than you would have guessed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A plot of the sample size $n$ versus the proportional width $\rho$ shows something that looks asymptotically linear on a log-log scale; in other words, a power-law--like relationship. We can estimate the power of this power-law relationship (crudely) as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;\hat{\alpha} \approx \frac{\log 0.1 - \log 1}{\log 5321 - \log 65} = \frac{-\log 10}{\log \frac{5231}{65}} \approx -0.525 ,&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;which is, unfortunately, decidedly slow!&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;This is sort of the &quot;canonical&quot; case to give you a feel for how to go about the calculation. Based on your plots, your data don't look particularly normal; in particular, there is what appears to be noticeable skewness.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But, this should give you a ballpark idea of what to expect. Note that to answer your second question above, it is necessary to fix some confidence level first, which I've set to 99% in the development above for demonstration purposes.&lt;/p&gt;&#xA;" OwnerUserId="2970" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-02-09T13:24:07.000" LastActivityDate="2011-02-09T13:24:07.000" CommentCount="2" />
  <row Id="7009" PostTypeId="1" AcceptedAnswerId="7010" CreationDate="2011-02-09T04:10:04.727" Score="1" ViewCount="71" Body="&lt;p&gt;I have the pdf&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$f(y ; \theta) = \frac{1}{\theta} \exp( \frac{-y}{\theta}), \ y &amp;gt; 0$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and I'm supposed to determine if the following two estimators are unbiased or not:  $ \hat \theta = nY_{min} $ and $ \hat \theta = \frac{1}{n}\sum_{i=1}^n Y_i $.  I'm running into some problems because when I try to find the expected value of $ Y_{min} $ and $ Y_i $, the integral is undefined.&lt;/p&gt;&#xA;" OwnerUserId="2387" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2011-02-09T09:32:50.203" LastActivityDate="2011-02-09T23:10:24.540" Title="Seeing if estimators are unbiased" Tags="&lt;homework&gt;&lt;unbiased-estimator&gt;" AnswerCount="2" />
  <row Id="7010" PostTypeId="2" ParentId="7009" CreationDate="2011-02-09T04:26:13.793" Score="1" ViewCount="" Body="&lt;p&gt;Hello Tshauk:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please see the following Wikipedia &lt;a href=&quot;http://en.wikipedia.org/wiki/Exponential_distribution&quot; rel=&quot;nofollow&quot;&gt;article&lt;/a&gt; and it shows that both the integral exist. Also, if you look lower in the article, you will also see that both the estimators are unbiased. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Y_{min}$ is also exponentially distributed with mean parameter $\frac {\theta} {n}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, $E[nY_{min}]=n\times\frac {\theta} {n} = \theta$. Similarly, $E[\frac{1}{n}\sum_{i=1}^n Y_i] = \frac{n \theta}{n}=\theta$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;I am sure both the integrals exists&lt;/em&gt; &lt;/p&gt;&#xA;" OwnerUserId="1307" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2011-02-09T09:33:03.997" LastActivityDate="2011-02-09T09:33:03.997" CommentCount="1" />
  <row Id="7011" PostTypeId="2" ParentId="3531" CreationDate="2011-02-09T04:39:53.350" Score="1" ViewCount="" Body="&lt;p&gt;WinBUGS has support for RJMCMC with an addon. I've used it for GLMs, including those with an intrinsic CAR component. Not R, obviously, but through R2WinBUGS you can patch them together.&lt;/p&gt;&#xA;" OwnerUserId="26" LastActivityDate="2011-02-09T04:39:53.350" />
  <row Id="7012" PostTypeId="2" ParentId="6990" CreationDate="2011-02-09T06:40:13.363" Score="0" ViewCount="" Body="&lt;p&gt;This post explains it quite well: &lt;a href=&quot;http://blog.gribblelab.org/2009/03/09/repeated-measures-anova-using-r/&quot; rel=&quot;nofollow&quot;&gt;http://blog.gribblelab.org/2009/03/09/repeated-measures-anova-using-r/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="582" LastActivityDate="2011-02-09T06:40:13.363" />
  <row Id="7013" PostTypeId="2" ParentId="841" CreationDate="2011-02-09T08:45:33.313" Score="2" ViewCount="" Body="&lt;p&gt;The difficulty clearly comes because $X$ and $Y$ are corellated (I assume $(X,Y)$ is jointly gaussian, as Aniko) and you can't make a difference (as in @svadali's answer) or a ratio (as in Standard Fisher-Snedecor &quot;F-test&quot;) because those would be of dependent $\chi^2$ distribution, and because you don't know what this dependence is which make it difficult to derive the distribution under $H_0$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My answer relies on Equation (1) below.&lt;/strong&gt; Because the difference in variance can be factorized with a difference in eigenvalues and a difference in rotation angle the test of equality can be declined into two tests. I show that it is possible to use the Fisher-Snedecor Test &lt;strong&gt;together&lt;/strong&gt; with a test on the slope such as the one suggested by @shabbychef because of a simple property of 2D gaussian vectors. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Fisher-Snedecor Test:&lt;/strong&gt;&#xA;If for $i=1,2$  $(Z^i_{1},\dots,Z^i_{n_i} )$ iid gaussian random variables with empirical unbiased variance $\hat{\lambda}^2_i$ and true variance $\lambda^2_i$, then it is possible to test if $\lambda_1=\lambda_2$ using the fact that, under the null,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It uses the fact that $$R=\frac{\hat{\lambda}_X^2}{\hat{\lambda}_Y^2}$$ follows a &lt;a href=&quot;http://en.wikipedia.org/wiki/F-distribution&quot; rel=&quot;nofollow&quot;&gt;Fisher-Snedecor distribution&lt;/a&gt; $F(n_1-1,n_2-1)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;A simple property of 2D gaussian vector&lt;/strong&gt;&#xA;Let us denote by &#xA;$$R(\theta) = \begin{bmatrix} \cos \theta &amp;amp; -\sin \theta \\ \sin \theta &amp;amp; \cos \theta \\ \end{bmatrix} &#xD;&#xA;$$&#xA;It is clear that there exists $\lambda_1,\lambda_2&amp;gt;0$ $\epsilon_1$, $\epsilon_2$ two independent gaussian $\mathcal{N}(0,\lambda_i^2)$ such that&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\begin{bmatrix} X \\ Y  \end{bmatrix}  = R(\theta)\begin{bmatrix} \epsilon_1 \\ \epsilon_2  \end{bmatrix}  &#xD;&#xA;$$&#xA;and that we have &#xA;$$Var(X)-Var(Y)=(\lambda_1^2-\lambda_2^2)(\cos^2 \theta -\sin^2 \theta) \;\; [1]$$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Testing of $Var(X)=Var(Y)$ can be done through testing if (&#xA;$\lambda_1^2=\lambda_2^2$ or $\theta=\pi/4 \; mod \; [\pi/2]$)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Conclusion (Answer to the question)&lt;/strong&gt;&#xA;Testing for $\lambda_1^2=\lambda_2^2$ is easely done by using ACP (to decorrelate) and Fisher Scnedecor test. Testing $\theta=\pi/4 [mod \; \pi/2]$ is done by testing if $|\beta_1|=1$ in the linear regression $ Y=\beta_1 X+\sigma\epsilon$ (I assume $Y$ and $X$ are centered). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Testing wether $\left ( \lambda_1^2=\lambda_2^2 \text{ or }\theta=\pi/4 [mod \; \pi/2]\right )$ at level $\alpha$ is done by testing if $\lambda_1^2=\lambda_2^2$ at level $\alpha/3$ or if $|\beta_1|=1$ at level $\alpha/3$. &lt;/p&gt;&#xA;" OwnerUserId="223" LastEditorUserId="223" LastEditorDisplayName="" LastEditDate="2011-03-13T20:31:13.040" LastActivityDate="2011-03-13T20:31:13.040" />
  <row Id="7014" PostTypeId="2" ParentId="6949" CreationDate="2011-02-09T10:50:10.537" Score="-1" ViewCount="" Body="&lt;p&gt;The articles referred to here may be relevant. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.finance30.com/forum/topics/template-for-benfords-first&quot; rel=&quot;nofollow&quot;&gt;http://www.finance30.com/forum/topics/template-for-benfords-first&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2030" LastActivityDate="2011-02-09T10:50:10.537" />
  <row Id="7015" PostTypeId="1" AcceptedAnswerId="7017" CreationDate="2011-02-09T11:01:45.140" Score="2" ViewCount="122" Body="&lt;ul&gt;&#xA;&lt;li&gt;Is linear regression only suitable for variables with a normal&lt;br&gt;&#xA;distribution?&lt;/li&gt;&#xA;&lt;li&gt;If so, is there an alternative nonparametric test to test mediation or moderation? &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="3119" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-02-09T11:10:17.940" LastActivityDate="2011-02-09T20:01:38.423" Title="Normal distribution necessary to assess moderating and mediating effects?" Tags="&lt;distributions&gt;&lt;mediation&gt;&lt;moderator&gt;&lt;multiple-regression&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="7017" PostTypeId="2" ParentId="7015" CreationDate="2011-02-09T12:17:35.667" Score="4" ViewCount="" Body="&lt;p&gt;For those not familiar with the language, moderation and mediation were both discussed in Barron and Kenny's influential article (&lt;a href=&quot;http://www.public.asu.edu/~davidpm/classes/psy536/Baron.pdf&quot; rel=&quot;nofollow&quot;&gt;free pdf&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Mediation&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;With regards to mediation, bootstrapping is often used where normality does not seem like a reasonable assumption.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;For SPSS and perhaps other implementations, check out the macros on the website of &lt;a href=&quot;http://www.afhayes.com/spss-sas-and-mplus-macros-and-code.html&quot; rel=&quot;nofollow&quot;&gt;Andrew Hayes&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;For R  have a look at the &lt;a href=&quot;http://cran.r-project.org/web/packages/mediation/mediation.pdf&quot; rel=&quot;nofollow&quot;&gt;mediation package&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;Moderation&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;With regard to moderator regression, you could also explore bootstrapping options if you were concerned with the accuracy of the p-values you were obtaining.&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2011-02-09T12:17:35.667" CommentCount="5" />
  <row Id="7018" PostTypeId="2" ParentId="7004" CreationDate="2011-02-09T12:18:26.897" Score="0" ViewCount="" Body="&lt;p&gt;Just be aware that if a component of your unknown distribution is a Cauchy distribution, the variance is undefined.&lt;/p&gt;&#xA;" OwnerUserId="1920" LastActivityDate="2011-02-09T12:18:26.897" />
  <row Id="7019" PostTypeId="1" CreationDate="2011-02-09T13:16:41.170" Score="1" ViewCount="88" Body="&lt;p&gt;Hey all,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a data set with a range of 0 to 65,000.  The vast majority of data points (it is a huge sample) are concentrated between 0 and 1000.  There is only one point that has 65,000.  I want to plot this using a semi-logarithmic plot.  However, I would like the graph to have around 50 points.  If I use scales like 2,4,6,8,16,32,64,128,256 etc I have only a couple of points.  But If I use a multiple of 1.1 or 1.5 etc, the scales are not integers.  Is there a standard way to ensure you more concentrated intervals at a lower level on the scale?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cheers,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;s&lt;/p&gt;&#xA;" OwnerUserId="2405" LastActivityDate="2011-04-11T01:13:15.487" Title="Log-scale with concentrated data using integers" Tags="&lt;scales&gt;&lt;logarithm&gt;" AnswerCount="2" />
  <row Id="7020" PostTypeId="1" AcceptedAnswerId="7021" CreationDate="2011-02-09T14:22:32.753" Score="6" ViewCount="161" Body="&lt;p&gt;Suppose I will be getting some samples from a binomial distribution.  One way to model my prior knowledge is with a Beta distribution with parameters $\alpha$ and $\beta$.  As I understand it, this is equivalent to having seen &quot;heads&quot; $\alpha$ times in $\alpha + \beta$ trials.  As such, a nice shortcut to doing the full-blown Bayesian inference is to use $\frac{h+\alpha}{n+\alpha+\beta}$ as my new mean for the probability of &quot;heads&quot; after having seen $h$ heads in $n$ trials.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now suppose I have more than two states, so I will be getting some samples from a multinomial distribution.  Suppose I want to use a Dirichlet distribution with parameter $\alpha$ as a prior.  Again as a shortcut I can treat this as prior knowledge of event $i$'s probability as being equivalent to $\frac{\alpha_i}{\sum \alpha_j}$, and if I witness event $i$ $h$ times in $n$ trials my posterior for $i$ becomes $\frac{h + \alpha_i}{n + \sum \alpha_j}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now in the binomial case, it works out that prior knowledge of &quot;heads&quot; occurring $\alpha$ times in $\alpha + \beta$ trials is equivalent to &quot;tails&quot; occurring $\beta$ times in $\alpha + \beta$ trials.  Logically I don't believe I can have stronger knowledge of &quot;heads&quot; likelihood than of &quot;tails.&quot;  This gets more interesting with more than two outcomes though.  If I have say a 6-sided die, I can imagine my prior knowledge of side &quot;1&quot; being equivalent to 10 ones in 50 trials and my prior knowledge of side &quot;2&quot; as being equivalent to 15 twos in 100 trials.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So after all of that introduction, my question is how I can properly model such asymmetric prior knowledge in the multinomial case?  It seems as though if I'm not careful I can easily get illogical results due to total probability/likelihood not summing to 1.  Is there some way I can still use the Dirichlet shortcut, or do I need to sacrifice this altogether and use some other prior distribution entirely?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please forgive any confusion caused by potential abuses in notation or terminology above.&lt;/p&gt;&#xA;" OwnerUserId="2485" LastEditorUserId="2485" LastEditorDisplayName="" LastEditDate="2011-02-09T23:27:35.783" LastActivityDate="2011-02-09T23:27:35.783" Title="Bayesian inference for multinomial distribution with asymmetric prior knowledge?" Tags="&lt;probability&gt;&lt;bayesian&gt;&lt;prior&gt;&lt;multinomial&gt;&lt;dirichlet-distribution&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="7021" PostTypeId="2" ParentId="7020" CreationDate="2011-02-09T14:51:50.390" Score="1" ViewCount="" Body="&lt;p&gt;Dear Michael McGowan:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You have framed your question very well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think what you are looking for here is a case of hierarchical modeling. And you may want to model multiple layers of hierarchy (at the moment you only talk about priors). Having another layer of hyper-priors for the hyper--parameters lets you model the additional variabilities in hyper-parameters (as you are concerned about the variability issues of hyper-parameters). It also makes your modeling flexible and robust (may be slower).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Specifically in your case, you may benefit by having priors for the Dirichlet distribution parameters (Beta is a special case). This &lt;a href=&quot;http://www.stat.columbia.edu/~cook/movabletype/archives/2009/04/conjugate_prior.html&quot; rel=&quot;nofollow&quot;&gt;post&lt;/a&gt; by Gelman talks about how to impose priors on the parameters of Dirichlet distribution. He also cites on of his papers in a journal of toxicology.&lt;/p&gt;&#xA;" OwnerUserId="1307" LastEditorUserId="1307" LastEditorDisplayName="" LastEditDate="2011-02-09T17:09:59.697" LastActivityDate="2011-02-09T17:09:59.697" CommentCount="1" />
  <row Id="7022" PostTypeId="1" CreationDate="2011-02-09T16:50:00.400" Score="2" ViewCount="160" Body="&lt;p&gt;Given a set of data (~5000 values) I'd like to draw random samples from the same distribution as the original data. The problem is there is no way to know for sure what distribution the original data comes from. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It makes sense to use normal distribution in my case, although I'd like to be able to motivate that decision, and of course I also need to estimate the $(\mu,\sigma)$ pair. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any idea on how to accomplish this, preferably within Java environment. I have been using &lt;a href=&quot;http://commons.apache.org/math/apidocs/org/apache/commons/math/distribution/NormalDistributionImpl.html&quot; rel=&quot;nofollow&quot;&gt;Apache Commons Math&lt;/a&gt; and recently stumbled upon &lt;a href=&quot;http://acs.lbl.gov/software/colt/&quot; rel=&quot;nofollow&quot;&gt;Colt&lt;/a&gt; library. I was hoping to get it done without bothering with MATLAB and R. &lt;/p&gt;&#xA;" OwnerUserId="3014" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-02-09T17:13:52.090" LastActivityDate="2011-02-09T19:37:08.393" Title="Parameter estimation for normal distribution in Java" Tags="&lt;estimation&gt;&lt;normal-distribution&gt;&lt;java&gt;" AnswerCount="1" CommentCount="6" />
  <row Id="7024" PostTypeId="2" ParentId="7019" CreationDate="2011-02-09T17:35:34.287" Score="2" ViewCount="" Body="&lt;p&gt;Can you just use a scale comprised of powers of (1+r) for some small r, and round to the nearest integer? For example, in R, with r = 0.25:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; x &amp;lt;- unique(round(1.25^(0:50)))&#xA;&amp;gt; x&#xA; [1]     1     2     3     4     5     6     7     9    12    15    18    23    28    36    44    56    69    87   108   136   169   212   265   331   414&#xA;[26]   517   646   808  1010  1262  1578  1972  2465  3081  3852  4815  6019  7523  9404 11755 14694 18367 22959 28699 35873 44842 56052 70065&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2425" LastActivityDate="2011-02-09T17:35:34.287" CommentCount="1" />
  <row Id="7025" PostTypeId="1" AcceptedAnswerId="7027" CreationDate="2011-02-09T18:35:23.570" Score="1" ViewCount="91" Body="&lt;p&gt;Hi I have a table with comma delimited columns and I need to convert the comma delimited values to new rows. for exmaple the given table is &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Name     Start   End &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A 1,2,3 4,5,6&lt;/p&gt;&#xA;&#xA;&lt;p&gt;B 1,2 4,5&lt;/p&gt;&#xA;&#xA;&lt;p&gt;C 1,2,3,4 6,7,8,9   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to convert it like&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Name Start End&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A 1 4&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A 2 5  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A 3 6&lt;/p&gt;&#xA;&#xA;&lt;p&gt;B 1 4&lt;/p&gt;&#xA;&#xA;&lt;p&gt;B 2 5 &lt;/p&gt;&#xA;&#xA;&lt;p&gt;C 1 6&lt;/p&gt;&#xA;&#xA;&lt;p&gt;C 2 7&lt;/p&gt;&#xA;&#xA;&lt;p&gt;C 3 8   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;C 4  9&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can do that using VB script but I need to solve it using R&#xA;Can anyone solve this?&lt;/p&gt;&#xA;" OwnerUserId="2725" LastActivityDate="2011-02-09T19:29:49.670" Title="Separate Comma Delimited Cells To New Rows" Tags="&lt;r&gt;" AnswerCount="2" CommentCount="4" />
  <row Id="7026" PostTypeId="2" ParentId="7007" CreationDate="2011-02-09T19:12:23.193" Score="1" ViewCount="" Body="&lt;p&gt;There are all kinds of analyses you can do. It really depends on what you want to know.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let me show you, with the example of gender: If you want to know wheter there are gender differences, you can apply a variance test (univariate or multivariate) on the likert scales with the demographic variable as independent factors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you just want to see, what scale is distributed how in each gender, then just split the dataset into two.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to know whether there is some correlation between the demographic variables calculate a correlation matrix.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The gist is: You can do anything, but that won't help you and you'll just get lost in the data, if you don't really know what you want to learn. So ask yourself:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What is my goal?&lt;/li&gt;&#xA;&lt;li&gt;What do I need to know to reach my goal?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Then ask again with more specifics.&lt;/p&gt;&#xA;" OwnerUserId="1435" LastActivityDate="2011-02-09T19:12:23.193" CommentCount="3" />
  <row Id="7027" PostTypeId="2" ParentId="7025" CreationDate="2011-02-09T19:26:27.087" Score="2" ViewCount="" Body="&lt;p&gt;Here's an approach that should work for you. I'm assuming that your three input vectors are in different objects. We are going to create a list of those inputs and write a function that process each object and returns them in the form of a &lt;code&gt;data.frame&lt;/code&gt; with plyr. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The things to take note of here are the splitting of the character vector into it's component parts, then using &lt;code&gt;as.numeric&lt;/code&gt; to convert the numbers from the character form when they were split. Since R fills matrices by column, we define a 2 column matrix and let R fill the values for us. We then retrieve the Name column and put it all together in a &lt;code&gt;data.frame&lt;/code&gt;. &lt;code&gt;plyr&lt;/code&gt; is nice enough to process the list and convert it into a &lt;code&gt;data.frame&lt;/code&gt; for us automatically.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(plyr)&#xA;&#xA;a &amp;lt;- paste(&quot;A&quot;,1, 2,3,4,5,6, sep = &quot;,&quot;, collapse = &quot;&quot;)&#xA;b &amp;lt;- paste(&quot;B&quot;,1, 2,4,5, sep = &quot;,&quot;, collapse = &quot;&quot;)&#xA;c &amp;lt;- paste(&quot;C&quot;,1, 2,3,4,6,7,8,9, sep = &quot;,&quot;, collapse = &quot;&quot;)&#xA;&#xA;input &amp;lt;- list(a,b,c)&#xA;&#xA;splitter &amp;lt;- function(x) {&#xA;    x &amp;lt;- unlist(strsplit(x, &quot;,&quot;))&#xA;    out &amp;lt;- data.frame(x[1], matrix(as.numeric(x[-1]), ncol = 2))&#xA;    colnames(out) &amp;lt;- c(&quot;Name&quot;, &quot;Start&quot;, &quot;End&quot;)&#xA;    return(out)&#xA;}&#xA;&#xA;&#xA;ldply(input, splitter)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And the output:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; ldply(input, splitter)&#xA; Name Start End&#xA;1    A     1   4&#xA;2    A     2   5&#xA;3    A     3   6&#xA;4    B     1   4&#xA;5    B     2   5&#xA;6    C     1   6&#xA;7    C     2   7&#xA;8    C     3   8&#xA;9    C     4   9&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="696" LastActivityDate="2011-02-09T19:26:27.087" />
  <row Id="7028" PostTypeId="2" ParentId="7025" CreationDate="2011-02-09T19:29:49.670" Score="3" ViewCount="" Body="&lt;p&gt;You might have asked this question on &lt;a href=&quot;http://stackoverflow.com/&quot;&gt;SO&lt;/a&gt; as there is no issue dealing with statistics :)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyway, I made up a quite complicated and ugly solution which might work for you:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# load your data&#xA;x &amp;lt;- structure(list(Name = c(&quot;A&quot;, &quot;B&quot;, &quot;C&quot;), Start = c(&quot;1,2,3&quot;, &quot;1,2&quot;, &#xA;&quot;1,2,3,4&quot;), End = c(&quot;4,5,6&quot;, &quot;4,5&quot;, &quot;6,7,8,9&quot;)), .Names = c(&quot;Name&quot;, &#xA;&quot;Start&quot;, &quot;End&quot;), row.names = c(NA, -3L), class = &quot;data.frame&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Which looks like in R like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; x&#xA;  Name   Start     End length&#xA;1    A   1,2,3   4,5,6      3&#xA;2    B     1,2     4,5      2&#xA;3    C 1,2,3,4 6,7,8,9      4&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Data transformation with the help of &lt;code&gt;strsplit&lt;/code&gt; calls:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data &amp;lt;- data.frame(cbind(&#xA;    rep(x$Name,as.numeric(lapply(strsplit(x$Start,&quot;,&quot;), length))),&#xA;    unlist(lapply(strsplit(x$Start,&quot;,&quot;), cbind)),&#xA;    unlist(lapply(strsplit(x$End,&quot;,&quot;), cbind))&#xA;    ))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Naming the new data frame:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;names(data) &amp;lt;- c(&quot;Name&quot;, &quot;Start&quot;, &quot;End&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Which looks like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; data&#xA;  Name Start End&#xA;1    A     1   4&#xA;2    A     2   5&#xA;3    A     3   6&#xA;4    B     1   4&#xA;5    B     2   5&#xA;6    C     1   6&#xA;7    C     2   7&#xA;8    C     3   8&#xA;9    C     4   9&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2714" LastActivityDate="2011-02-09T19:29:49.670" CommentCount="2" />
  <row Id="7029" PostTypeId="1" AcceptedAnswerId="7039" CreationDate="2011-02-09T19:34:28.307" Score="16" ViewCount="632" Body="&lt;p&gt;I ran across this density the other day.  Has someone given this a name?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$f(x) = \log(1 + x^{-2}) / 2\pi$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The density is infinite at the origin and it also has fat tails.  I saw it used as a prior distribution in a context where many observations were expected to be small, though large values were expected as well.&lt;/p&gt;&#xA;" OwnerUserId="319" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-02-12T05:07:24.860" LastActivityDate="2011-02-12T05:07:24.860" Title="Does the distribution $\log(1 + x^{-2}) / 2\pi$ have a name?" Tags="&lt;distributions&gt;&lt;probability&gt;" AnswerCount="2" CommentCount="5" FavoriteCount="1" />
  <row Id="7030" PostTypeId="2" ParentId="7022" CreationDate="2011-02-09T19:37:08.393" Score="1" ViewCount="" Body="&lt;p&gt;How big are the samples that you need? If substantially smaller than the 5000 points you have, say maximum 100 points or so, you could just take a random subset of your sample. Then you don't even need to assume normality - it's guaranteed to come from the distribution you want!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Otherwise, it seems that the &lt;code&gt;org.apache.commons.math.stat.descriptive.moment&lt;/code&gt; package has a &lt;a href=&quot;http://commons.apache.org/math/apidocs/org/apache/commons/math/stat/descriptive/moment/Mean.html&quot; rel=&quot;nofollow&quot;&gt;Mean&lt;/a&gt; and &lt;a href=&quot;http://commons.apache.org/math/apidocs/org/apache/commons/math/stat/descriptive/moment/StandardDeviation.html&quot; rel=&quot;nofollow&quot;&gt;StandardDeviation&lt;/a&gt; class which use the correct formulas. These should give you $\mu$ and $\sigma$, respectively.&lt;/p&gt;&#xA;" OwnerUserId="2898" LastActivityDate="2011-02-09T19:37:08.393" CommentCount="1" />
  <row Id="7031" PostTypeId="2" ParentId="7015" CreationDate="2011-02-09T20:01:38.423" Score="1" ViewCount="" Body="&lt;p&gt;Many of us use linear regression in rough-and-ready fashion to learn about the relative importance of predictors, to assess the shape of relationships, and so on.  But if one wants to make strict probabilistic inferences one needs to satisfy the set of standard assumptions entailed in such regression.  The most important of these are random sampling and independent observations.  Another one is that the regression residuals (not any particular X or Y variable) should be normally distributed.  Without satisfying all of the standard assumptions, one cannot vouch for the accuracy of standard errors or p-values.&lt;/p&gt;&#xA;" OwnerUserId="2669" LastActivityDate="2011-02-09T20:01:38.423" />
  <row Id="7032" PostTypeId="1" CreationDate="2011-02-09T20:20:17.480" Score="1" ViewCount="117" Body="&lt;p&gt;Why is it that &quot;missing data&quot; and &quot;outliers&quot; can affect the performance of least square estimation?&lt;/p&gt;&#xA;" OwnerUserId="3125" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-02-12T05:03:00.997" LastActivityDate="2011-02-12T05:12:06.707" Title="Effect of missing data and outliers on least square estimation" Tags="&lt;regression&gt;&lt;estimation&gt;&lt;least-squares&gt;" AnswerCount="3" CommentCount="2" />
  <row Id="7033" PostTypeId="2" ParentId="7029" CreationDate="2011-02-09T23:08:02.450" Score="4" ViewCount="" Body="&lt;p&gt;Perhaps not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I could not find it in this fairly extensive list of distributions:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.math.wm.edu/~leemis/2008amstat.pdf&quot; rel=&quot;nofollow&quot;&gt;Leemis and McQuestion 2008 Univariate Distribution Relationships. &lt;em&gt;American Statistician&lt;/em&gt; 62(1) 45:53&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2750" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-02-10T20:59:20.080" LastActivityDate="2011-02-10T20:59:20.080" />
  <row Id="7034" PostTypeId="2" ParentId="7009" CreationDate="2011-02-09T23:10:24.540" Score="0" ViewCount="" Body="&lt;p&gt;The fact that the sample mean is an unbiased estimator is obtained combining these two facts:&#xA;1. The sample mean is an unbiased estimator of the population mean&#xA;2. The population mean is equal to theta&lt;/p&gt;&#xA;" OwnerUserId="3127" LastActivityDate="2011-02-09T23:10:24.540" />
  <row Id="7036" PostTypeId="1" AcceptedAnswerId="7041" CreationDate="2011-02-10T01:24:08.580" Score="4" ViewCount="186" Body="&lt;p&gt;This question may have been asked before, but I couldn't find it.   So, here goes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From about 3000 data points that can be characterized as &quot;wins&quot; or &quot;losses&quot; (binomial), it turns out that there are 52.8% dumb luck wins.   This is my dependent variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also have some additional data that may help in predicting the above wins that could be considered an independent variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The question I'm trying to answer is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If my independent variable can be used to predict 55% wins, how many trials are required (giving me 55% wins) for me to be 99% sure that this wasn't dumb luck?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The following R code is purposely hacky so I can see everything that is happening.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#Run through a set of trial sizes&#xA;numtri &amp;lt;- seq(2720, 2840, 1)&#xA;&#xA;#For a 52.8% probability of dumb luck wins, and the current trial size,&#xA;#calculate the number of wins at the 99% limit&#xA;numwin &amp;lt;- qbinom(p=0.99, size=numtri, prob=0.528)&#xA;&#xA;#Divide the number of wins at the 99% limit by the trial size to&#xA;#get the percent wins.&#xA;perwin &amp;lt;- numwin/numtri&#xA;&#xA;#Plot the percent wins versus the trial size, looking for the&#xA;#&quot;predicted&quot; 55% value. See the plot below.&#xA;plot(numtri, perwin, type=&quot;b&quot;, main=&quot;Looking for 0.55&quot;)&#xA;grid()&#xA;&#xA;#Draw a red line at the 55% level to show which trial sizes are correct&#xA;abline(h=0.55, lwd=2, col=&quot;red&quot;)&#xA;&#xA;#More than one answer? Why?........Integer issues&#xA;head(numtri)&#xA;head(numwin)&#xA;head(perwin)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;From the graph, the answer is: 2740 &amp;lt;= numtri &amp;lt;= 2820 &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you can guess, I'm also looking for the required number of trials for 56% wins, 57% wins, 58% wins, etc.   So, I'll be automating the process.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Back to my question.   Does anyone see a problem with the code, and if not, does anyone have a way to cleanly sneak up on the right and left edges of the &quot;answer&quot;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/2CSHV.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit (02/13/2011) ==================================================&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Per whuber's answer below, I now realize that to compare my alternate 55% wins (which came from the data) with my 52.8% &quot;dumb luck&quot; null (which also came from the data), I have to deal with the fuzz from both measured values.  In other words, to be &quot;99% sure&quot; (whatever that means), the 1% tail of BOTH proportions needs to be compared.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For me to get comfortable with whuber's formula for N, I had to reproduce his algebra.  In addition, because I know that I'll use this framework for more complicated problems, I bootstrapped the calculations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Below are some of the results.   The upper left graph was produced during the bootstrap process where N (the number of trials) was at 9000.  As you can see, the 1% tail for the null extends further into the alternate than the 1% for the alternate.   The conclusion?   9000 trials is not enough.   The lower left graph is also during the bootstrap process where N was at 13,000.   In this case, the 1% tail for the null falls into an area of the alternate that is less than the required 1% value.   The conclusion?  13,000 trials is more than enough (too many) for the 1% level.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The upper right graph is where N=11109 (whuber's calculated value) and the 1% tail of the null extends the right amount into the alternate, aligning the 1% tails.   The conclusion?  11109 trials are required for the 1% significance level (I am now comfortable with whuber's formula).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The lower right graph is where I used whuber's formula for N and varied the alternate q (at both the 1% and 5% significance levels) so I could have a reference of what might be an &quot;acceptable zone&quot; for alternates.    In addition, I overlaid some actual alternates versus their associated number of data points (the blue dots).    One point falls well below the 5% level, and even though it might provide 72% &quot;wins&quot;, there simply aren't enough data points to differentiate it from 52.8% &quot;dumb luck&quot;.   So, I'll probably drop that point and the two others below 5% (and the independent variables that were used to select those points).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/sEsZx.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="2775" LastEditorUserId="2775" LastEditorDisplayName="" LastEditDate="2011-02-13T20:36:13.740" LastActivityDate="2011-02-13T20:36:13.740" Title="Number of trials required from a binomial distribution to get the desired odds" Tags="&lt;r&gt;&lt;binomial&gt;&lt;p-value&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="7037" PostTypeId="2" ParentId="7032" CreationDate="2011-02-10T02:16:59.203" Score="3" ViewCount="" Body="&lt;p&gt;If you're using R, try the following example.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(tcltk)&#xA;demo(tkcanvas)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Move the dots around to create all of the outliers you want.  The regression will keep up with you.&lt;/p&gt;&#xA;" OwnerUserId="2775" LastActivityDate="2011-02-10T02:16:59.203" />
  <row Id="7038" PostTypeId="2" ParentId="7036" CreationDate="2011-02-10T02:20:51.027" Score="4" ViewCount="" Body="&lt;pre&gt;&lt;code&gt;numtri[c(min(which(perwin &amp;lt;= 0.55)),max(which(perwin &amp;gt;= 0.55)))]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="159" LastActivityDate="2011-02-10T02:20:51.027" CommentCount="1" />
  <row Id="7039" PostTypeId="2" ParentId="7029" CreationDate="2011-02-10T02:41:06.410" Score="10" ViewCount="" Body="&lt;p&gt;Indeed, even the first moment does not exist.  The CDF of this distribution is given by&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$F(x) = 1/2 + \left(\arctan(x) - x \log(\sin(\arctan(x)))\right)/\pi$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;for $x \ge 0$ and, by symmetry, $F(x) = 1 - F(|x|)$ for $x \lt 0$.  Neither this nor any of the obvious transforms look familiar to me.  (The fact that we can obtain a closed form for the CDF in terms of elementary functions already severely limits the possibilities, but the somewhat obscure and complicated nature of this closed form quickly rules out standard distributions or power/log/exponential/trig transformations of them.  The arctangent is, of course, the CDF of a Cauchy (Student $t_1$) distribution, exhibiting this CDF as a (substantially) perturbed version of the Cauchy distribution, shown as red dashes.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/GrcDB.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-02-10T02:41:06.410" CommentCount="9" />
  <row Id="7040" PostTypeId="1" AcceptedAnswerId="7042" CreationDate="2011-02-10T05:59:08.420" Score="3" ViewCount="282" Body="&lt;p&gt;In Meta analysis, how to interpret the Egger’s linear regression method intercept (B0) 10.34631, 95% confidence interval (1.05905, 19.63357), with t=3.54535, df=3. The 1-tailed p-value (recommended) is 0.01911, and the 2-tailed p-value is 0.03822. I am a medical doctor. &#xA;&lt;strong&gt;*&lt;em&gt;Updated&lt;/em&gt;*&lt;/strong&gt;&#xA;The data are comparison of Regions 1 and 2&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;          Region 1      Region 2         Region 3           &#xA;        Cases   Dead    Cases   Dead     Cases  Dead    Total cases Total dead cases&#xA;2006    2320    528     1484    108       73    3       3877            639&#xA;2007    3024    645     1592    75        32    1       4648            721&#xA;2008    3012    537     1920    53         3    0       4935            590&#xA;2009    3073    556     1477    40       246    8       4796            604&#xA;2010    3540    494     1460    26       138    1       5138            521&#xA;Total  14969    2760    7933    302      492    13     23394            3075&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2956" LastEditorUserId="2956" LastEditorDisplayName="" LastEditDate="2011-02-10T15:35:02.927" LastActivityDate="2011-02-10T15:35:02.927" Title="Egger’s linear regression method intercept in meta analysis" Tags="&lt;meta-analysis&gt;&lt;funnel-plot&gt;&lt;publication-bias&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="1" />
  <row Id="7041" PostTypeId="2" ParentId="7036" CreationDate="2011-02-10T07:24:36.527" Score="5" ViewCount="" Body="&lt;p&gt;I wonder what it means to be &quot;99% sure.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The code seems to equate &quot;dumb luck&quot; with $p$ = 52.8% probability of wins.  Let's imagine conducting $N$ trials, during which we observe $k$ wins.  Suppose, for instance, $N$ = 1000 and you observe $k$ = 530 wins.  That's greater than the expected number $p N$ = 528, but it's so close that a sceptic would argue it could be due to chance: &quot;dumb luck.&quot;  So, &lt;strong&gt;you need a test.&lt;/strong&gt;  The best test uses a threshold $t$ (which depends, obviously, on $N$ and $p$ and will be a bit greater than $N p$).  The chance--still assuming $p$ = 52.8%--of having more than $t$ wins needs to be so low that if you do observe more than $t$ wins, both you and a reasonable sceptic will allow that $p$ is probably greater than 52.8%.  This chance is usually called $\alpha$, the &quot;false positive rate.&quot;  Your first task is to set this threshold.  How will you do it?  That is, &lt;em&gt;what value of&lt;/em&gt; $\alpha$ &lt;em&gt;do you want to use?&lt;/em&gt;  Does &quot;99% sure&quot; mean that $1 - \alpha$ should equal 99%?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second task is to consider your chances of beating the threshold &lt;em&gt;if in fact&lt;/em&gt; $p$ &lt;em&gt;equals 55%&lt;/em&gt; (or whatever you choose).  Does &quot;99% sure&quot; mean you want to have a 99% chance of having more than $t$ wins in $N$ trials when $p$ = 55%?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, it seems to me there are &lt;strong&gt;two&lt;/strong&gt; chances involved here: one in determining the test to distinguish &quot;dumb luck&quot; and another one that will subsequently determine how many trials you need.  If this analysis correctly reflects the situation, the problem is easily solved.  Note a few things:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;There are plenty of uncertainties about this procedure and some of them are not captured by the statistical model.  Therefore we shouldn't aim for too much precision.  There's really no difference between, say, 2742 and 2749 trials.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;In case it turns out hundreds or thousands of trials are needed, the (usual) Normal approximation to the Binomial distribution will be highly accurate.  Point #1 justifies using an approximation.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Consequently, there is a simple formula to compute the number of trials directly: no need to explore hundreds of possibilities!&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The formula alluded to in #3 is this: in $N$ trials with probability $p$ of winning, the standard error of the proportion of wins is &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$s = \sqrt{p(1-p)/N}.$$  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The threshold therefore should be &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$t(N) = N(p + z_\alpha s)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $z_\alpha$ is the upper $100 - 100\alpha$ percentile of the standard normal distribution.  If the true probability of a win actually is $q$ (such as $q$ = 55%), the chance of exceeding $t(N)$ in $N$ trials equals&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\Phi\left(\frac{(q-t(N)/N)\sqrt{N}}{\sqrt{q(1-q)}}\right)&#xD;&#xA;= \Phi\left(\frac{(q-p)\sqrt{N} - z_\alpha \sqrt{p(1-p)}}{\sqrt{q(1-q)}}\right)&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\Phi$ denotes the standard normal cumulative distribution function.  Equating this with your desired degree of certainty (such as $\beta$ = 99%) and solving this equation for $N$ (rounding up to the nearest integer) gives the solution&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$N = \left(\frac{z_\alpha \sqrt{p(1-p)} + z_\beta \sqrt{q(1-q)}}{q-p}\right)^2$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(assuming $q \gt p$).&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;h3&gt;Example&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;For $p$ = 52.8%, $q$ = 55%, $\alpha$ = 1%, and $\beta$ = 99%, which give $z_\alpha$ = $z_\beta$ = 2.32635, I obtain $N$ = 11109 with this formula.  In 11109 trials you would expect to win about 5866 times by &quot;dumb luck&quot; and maybe as many as $t$ = 5988 times due to chance variation.  If the true winning percentage is 55%, you expect to win about 6110 times, but there's a 1% chance you will win 5988 or fewer times, and thereby (due to &quot;bad luck&quot;) not succeed in showing (with 99% confidence) that &quot;dumb luck&quot; isn't operating.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;R code for the example&lt;/h3&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#&#xA;# Specify inputs.&#xA;#&#xA;p &amp;lt;- 52.8/100     # Null hypothesis&#xA;q &amp;lt;- 55/100       # Alternative&#xA;alpha &amp;lt;- 1/100    # Test size&#xA;beta &amp;lt;- 99/100    # Chance of confirming the alternative&#xA;#&#xA;# Compute number of trials.&#xA;#&#xA;s.p &amp;lt;- sqrt(p*(1-p))&#xA;s.q &amp;lt;- sqrt(q*(1-q))&#xA;z.alpha &amp;lt;- qnorm(1 - alpha)&#xA;z.beta &amp;lt;- qnorm(beta)&#xA;delta &amp;lt;- q - p&#xA;N &amp;lt;- ceiling(((z.alpha * s.p + z.beta * s.q)/delta)^2)&#xA;#&#xA;# Explain.&#xA;#&#xA;t &amp;lt;- floor(N*(p + z.alpha*se.p))   # Threshold&#xA;m &amp;lt;- round(N*p)                    # Expectation (null)&#xA;k &amp;lt;- round(N*q)                    # Expectation (alternate)&#xA;size &amp;lt;- pnorm((t + 1/2 - N*p)/(s.p * sqrt(N))) # Actual test size&#xA;sureness &amp;lt;- 1 - pnorm((N*q - (t + 1/2))/(s.q * sqrt(N)))&#xA;print(list(N=N, threshold=t, expectation=m, alternate=k,&#xA;  size=size, sureness=sureness))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-02-10T14:25:46.007" LastActivityDate="2011-02-10T14:25:46.007" CommentCount="4" />
  <row Id="7042" PostTypeId="2" ParentId="7040" CreationDate="2011-02-10T08:09:11.320" Score="3" ViewCount="" Body="&lt;p&gt;I suppose you are not interested in &quot;hardcore&quot; statistical explanation. So, more the intercept deviates from zero, the more pronounced the asymmetry. If the &lt;em&gt;p&lt;/em&gt;-value of the intercept is 0.1 or smaller, the asymmetry is considered to be statistically significant. More &lt;a href=&quot;http://goo.gl/PlNhR&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="609" LastActivityDate="2011-02-10T08:09:11.320" CommentCount="2" />
  <row Id="7043" PostTypeId="2" ParentId="7032" CreationDate="2011-02-10T08:13:06.480" Score="1" ViewCount="" Body="&lt;p&gt;I'm not sure about the &quot;missing data&quot;, but I can give an answer on &quot;outliers&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is basically due to the &quot;unbounded&quot; influence that a single observation can have in least squares (or at least in conventional least squares).  A very, very simple example of least squares should show this.  Suppose you only estimate an intercept $\mu$ using data $Y_i \ (i=1,\dots,n)$.  The least square equation is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\sum_{i=1}^{n} (Y_i-\mu)^2$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which is minimised by choosing $\hat{\mu}=n^{-1}\sum_{i=1}^{n} Y_i=\overline{Y}$.  Now suppose I add one extra observation to the sample, equal to $X$, how will the estimate change?  The new best estimate using $n+1$ observations is just &#xA;$$\hat{\mu}_{+1}=\frac{X+\sum_{i=1}^{n} Y_i}{n+1}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rearranging terms gives&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\hat{\mu}_{+1}=\hat{\mu}+\frac{1}{n+1}(X-\hat{\mu})$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now for a given sample $\hat{\mu}$ and $n$ are fixed.  So I can essentially &quot;choose&quot; $X$ to get any new average that I want!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using the same argument, you can show that deleting the $j$th observation has a similar effect:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\hat{\mu}_{-j}=\hat{\mu}+\frac{-1}{n-1}(Y_{j}-\hat{\mu})$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And similarly (a bit tediously), you can show that removing $M$ observations gives:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\hat{\mu}_{-M}=\hat{\mu}+\frac{-M}{n-M}(\overline{Y}_{M}-\hat{\mu})$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $\overline{Y}_{M}$ is the average of the observations that you removed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The same kind of thing happens in general least squares, the estimate &quot;chases&quot; the outliers.  If you are worried about this, then &quot;least absolute deviations&quot; may be a better way to go (but this can be less efficient if you don't have any outliers).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Influence functions are a good way to study this stuff (outliers and robustness).  For example, you can get an approximate change in the variance $s^2=n^{-1}\sum_{i=1}^{n}(Y_i-\overline{Y})^2$ as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$s^2_{-j} = s^2 +\frac{-1}{n-1}((Y_j-\overline{Y})^2-s^2) + O(n^{-2})$$&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-02-10T08:13:06.480" />
  <row Id="7045" PostTypeId="1" AcceptedAnswerId="10475" CreationDate="2011-02-10T11:40:35.520" Score="2" ViewCount="220" Body="&lt;p&gt;Is there a package or library that can help me suggest a formula given the independent variables which will work well in glm, for example this formula can be something like x^2+log(y)+Z, it does not necessarily need to be the standard linear model x+y+z in order to explain a variable.&lt;/p&gt;&#xA;" OwnerUserId="1808" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-02-10T13:59:01.247" LastActivityDate="2011-05-07T18:54:02.790" Title="How to obtain in R a good formula for glm (general linear models) to predict a binomial variable?" Tags="&lt;r&gt;&lt;logistic-regression&gt;&lt;glm&gt;" AnswerCount="1" CommentCount="6" />
  <row Id="7046" PostTypeId="2" ParentId="7040" CreationDate="2011-02-10T12:11:08.863" Score="3" ViewCount="" Body="&lt;p&gt;@Andrej has already given an answer, i.e there is evidence of funnel plot asymmetry. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;@DrWho, I would be interested in the reference that suggests using a one-tailed test.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The following can give you an idea of the underlying logic of applying this regression model to test for publication bias:  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most of these regression approaches are using the so-called standard normal deviate (SND) which is defined as effect size divided by its standard error ($ES_i / SE_i$).  The inverse standard error (“precision”) serves as predictor variable. Then, an unweighted OLS regression is estimated.  When there is no evidence of funnel plot asymmetry, the intercept should not significantly differ from zero, i.e. $H_0: b_0 = 0$. In other words, the intercepts provide a measure of funnel plot asymmetry (Sterne/Egger 2005: 101). This is due to two reasons: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(1) Since the standard error depends on sample size, the inverse standard error for small studies will be close to zero. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(2) Even though small studies may produce large effect sizes, the SND will be small since the standard error will be large. Again, for small studies, the SND will be close to zero. For large studies, however, we will observe large SNDs and the inverse standard errors will also be large (Egger et al 1997: 629f.). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Egger, M., G. Davey Smith, M. Schneider, C. Minder (1997), Bias in meta-analysis detected by a simple, graphical test British Medical Journal 315: 629-634.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sterne, J.A.C., B.J. Becker, M. Egger (2005), The funnel plot. S. 75-98 in: H.R. Rothstein, A.J. Sutton, M. Borenstein (Hrsg.), Publication Bias in Meta-Analysis. Prevention, Assessment and Adjustments, The Atrium, Southern Gate, Chichester: John Wiley &amp;amp; Sons, Ltd.&lt;/p&gt;&#xA;" OwnerUserId="307" LastActivityDate="2011-02-10T12:11:08.863" CommentCount="3" />
  <row Id="7047" PostTypeId="1" AcceptedAnswerId="7067" CreationDate="2011-02-10T12:18:06.050" Score="2" ViewCount="95" Body="&lt;p&gt;Hey guys. I got two images from video frames. They have a certain portion of overlap. After warping one of them, I'm currently trying to blend them together. In other words, I would like to stitch them together. But I don't know how to accomplish that. Can anybody please give me some help? Thank you!&lt;br&gt;&#xA; Let's say the image data is store in 'image1_warped' and 'image2'. Appreciated your help!&lt;/p&gt;&#xA;" OwnerUserId="3133" LastActivityDate="2011-02-10T21:15:49.847" Title="Matlab image blending" Tags="&lt;matlab&gt;&lt;image-processing&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="7048" PostTypeId="1" AcceptedAnswerId="7416" CreationDate="2011-02-10T14:49:20.220" Score="9" ViewCount="250" Body="&lt;p&gt;Hello all,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While preparing for a talk I will give soon,  I recently started digging into two major (Free) tools for interactive data visualization: &lt;a href=&quot;http://www.ggobi.org/&quot; rel=&quot;nofollow&quot;&gt;GGobi&lt;/a&gt; and &lt;a href=&quot;http://rosuda.org/mondrian/&quot; rel=&quot;nofollow&quot;&gt;mondrian&lt;/a&gt; - both offer a great range of capabilities (even if they're a bit buggy).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wish to ask for your help in articulating (both to myself, and for my future audience) &lt;strong&gt;When is it helpful to use interactive plots? Either for data exploration (for ourselves) and data presentation (for a &quot;client&quot;)?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For when explaining the data to a client, I can see the value of animation for:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Using &quot;identify/linking/brushing&quot; for seeing which data point in the graph is what.&lt;/li&gt;&#xA;&lt;li&gt;Presenting a sensitivity analysis of the data (e.g: &quot;if we remove this point, here is what we will get)&lt;/li&gt;&#xA;&lt;li&gt;Showing the effect of different groups in the data (e.g: &quot;let's look at our graphs for males and now for the females&quot;)&lt;/li&gt;&#xA;&lt;li&gt;Showing the effect of time (or age, or in general, offering another dimension to the presentation)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;For when exploring the data ourselves, I can see the value of identify/linking/brushing when exploring an outlier in a dataset we are working on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But other then these two examples, I am not sure what other practical use these techniques offer.  Especially for our own data exploration!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It could be argued that the interactive part is good for exploring (For example) a different behavior of different groups/clusters in the data.  But when (in practice) I approached such situation, what I tended to do was to run the relevant statistical procedures (and post-hoc tests) - and what I found to be significant I would then plot with colors clearly dividing the data to the relevant groups.  From what I've seen, this is a safer approach then &quot;wondering around&quot; the data (which could easily lead to data dredging (were the scope of the multiple comparison needed for correction is not even clear).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd be very happy to read your experience/thoughts on this matter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(this question can be a wiki - although it is not subjective and a well thought-out answer will gladly win my &quot;answer&quot; mark :) )&lt;/p&gt;&#xA;" OwnerUserId="253" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-02-20T14:30:18.040" LastActivityDate="2011-02-20T14:30:18.040" Title="When is interactive data visualization useful to use?" Tags="&lt;data-visualization&gt;&lt;data-mining&gt;&lt;interactive-visualization&gt;&lt;ggobi&gt;&lt;mondrian&gt;" AnswerCount="3" CommentCount="1" FavoriteCount="5" />
  <row Id="7049" PostTypeId="1" CreationDate="2011-02-10T14:59:26.717" Score="3" ViewCount="241" Body="&lt;p&gt;Hi everyone, my question is as follows: Can the poisson distribution be used to analyze continuous data as well as discrete data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a few data sets where response variables are continuous, but resemble a poisson distribution rather than a normal distribution. However, the poisson distribution is a discrete distribution and is usually concerned with numbers or counts. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="3136" LastActivityDate="2011-02-11T01:29:36.383" Title="Using poisson regression for continuous data?" Tags="&lt;distributions&gt;&lt;regression&gt;&lt;poisson&gt;&lt;continuous-data&gt;" AnswerCount="2" CommentCount="3" FavoriteCount="1" />
  <row Id="7050" PostTypeId="2" ParentId="7048" CreationDate="2011-02-10T15:13:17.287" Score="5" ViewCount="" Body="&lt;p&gt;Dynamic linking of graphics is natural and effective for &lt;strong&gt;exploratory spatial data analysis&lt;/strong&gt;, or &lt;a href=&quot;http://www.ncgia.ucsb.edu/giscc/units/u128/u128_f.html&quot; rel=&quot;nofollow&quot;&gt;ESDA&lt;/a&gt;.  ESDA systems typically link one or more quantitative maps (such as &lt;a href=&quot;http://en.wikipedia.org/wiki/Choropleth_map&quot; rel=&quot;nofollow&quot;&gt;choropleth maps&lt;/a&gt;) with tabular views and statistical graphics of the underlying data.  Some such capabilities have been a part of a few desktop GIS systems for about 15 years, particularly &lt;a href=&quot;http://en.wikipedia.org/wiki/ArcView_3.x&quot; rel=&quot;nofollow&quot;&gt;ArcView 3&lt;/a&gt; (a discontinued commercial product).  The free &lt;a href=&quot;http://geodacenter.asu.edu/software/downloads&quot; rel=&quot;nofollow&quot;&gt;GeoDa&lt;/a&gt; software provides some of these capabilities within an environment designed for spatial data exploration and statistical analysis.  It's clunky, with an idiosyncratic interface and unpolished graphics, but fairly bug free.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This use of EDA circumvents the objection that statistical testing may be better than interactive exploration because in many (most?) situations there is no clear statistical model, there is no obvious (or even appropriate) statistical test, and hypothesis testing is often irrelevant: people need to see &lt;em&gt;what occurs&lt;/em&gt;, &lt;em&gt;where it occurs&lt;/em&gt;, and to observe the &lt;em&gt;statistical relationships&lt;/em&gt; among variables &lt;em&gt;in a spatial context.&lt;/em&gt;  Not all data analysis is, or should even consist of, formal procedures!&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-02-10T15:13:17.287" CommentCount="1" />
  <row Id="7051" PostTypeId="2" ParentId="7049" CreationDate="2011-02-10T15:15:28.673" Score="5" ViewCount="" Body="&lt;p&gt;If you're talking about using a Poisson response in a generalized linear model, then yes, if you are willing to make the assumption that the variance of each observation is equal to its mean.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you don't want to do that, another alternative may be to transform the response (e.g. take logs).&lt;/p&gt;&#xA;" OwnerUserId="495" LastActivityDate="2011-02-10T15:15:28.673" CommentCount="5" />
  <row Id="7052" PostTypeId="2" ParentId="6225" CreationDate="2011-02-10T15:29:44.943" Score="0" ViewCount="" Body="&lt;p&gt;Technically, no, a null hypothesis cannot be proven.  For any fixed, finite sample size, there will always be some small but nonzero effect size for which your statistical test has virtually no power.  More practically, though, you can prove that you're within some small epsilon of the null hypothesis, such that deviations less than this epsilon are not practically significant.&lt;/p&gt;&#xA;" OwnerUserId="1347" LastActivityDate="2011-02-10T15:29:44.943" />
  <row Id="7053" PostTypeId="1" AcceptedAnswerId="7064" CreationDate="2011-02-10T15:35:57.853" Score="5" ViewCount="97" Body="&lt;p&gt;I’m reviewing an article, and can’t give details but here is the situation, and it’s got me puzzled&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Patients were divided into 4 categories (call them A B C and D), which were exhaustive and exclusive.  Adjusted hazard ratios were computed for these four groups for all patients and for two subgroups of patients (call them Lung and Heart).  The two subgroups were also exhaustive and exclusive.  For the full sample, the hazard ratios for B C and D (compared to A) were 5.08, 4.39, and 1.81.  For the lung group, they were 1.15, 1.16 and 1.07.  For the heart group they were 2.55, 1.69 and 1.28.   That is, they were lower for both subgroups than for the whole population.  The lung group and heart group are not the same size; from the info provided, it’s not clear whether A B C and D are equally divided across heart and lung groups.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So …. This seems like Simpson’s paradox.  But I’ve not seen that term used for survival analysis.  I don’t see why it could NOT be so applied.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My feeling here is to suggest that heart and lung should ONLY be presented separately, but I’m not completely sure.    &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Excuse me if I am missing something obvious – dealing with some family health problems and not sleeping that well)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for any insights or thoughts&lt;/p&gt;&#xA;" OwnerUserId="686" LastActivityDate="2011-02-10T20:17:34.793" Title="Question about combining hazard ratios - Maybe Simpson's paradox?" Tags="&lt;survival&gt;&lt;simpsons-paradox&gt;" AnswerCount="2" />
  <row Id="7054" PostTypeId="1" AcceptedAnswerId="7131" CreationDate="2011-02-10T15:38:40.207" Score="4" ViewCount="95" Body="&lt;p&gt;Can we do a meta-analysis of data of 3 regions. A particular disease was treated with the same treatment but implemented thoroughly in 2 regions and not so thoroughly in 1 region. How to proceed with analysis? What is the best way to analyse these data. The data are of a disease (cases and deaths) as  follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;          Region 1      Region 2         Region 3           &#xA;        Cases   Dead    Cases   Dead     Cases  Dead    Total cases     Total dead cases&#xA;2006    2320    528     1484    108       73    3       3877            639&#xA;2007    3024    645     1592    75        32    1       4648            721&#xA;2008    3012    537     1920    53         3    0       4935            590&#xA;2009    3073    556     1477    40       246    8       4796            604&#xA;2010    3540    494     1460    26       138    1       5138            521&#xA;Total  14969    2760    7933    302      492    13     23394            3075&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;&lt;strong&gt;Updated&lt;/em&gt;&lt;/strong&gt; These are unpublished data.&lt;/p&gt;&#xA;" OwnerUserId="2956" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-02-12T05:52:05.690" LastActivityDate="2011-02-12T16:39:30.410" Title="Meta-analysis of 3 regions' data for 5 years" Tags="&lt;meta-analysis&gt;&lt;longitudinal&gt;" AnswerCount="1" CommentCount="8" FavoriteCount="1" />
  <row Id="7056" PostTypeId="1" AcceptedAnswerId="7075" CreationDate="2011-02-10T16:11:00.267" Score="1" ViewCount="67" Body="&lt;p&gt;readHTMLTable seems pretty robust, but when I try to use it on this page, I get an error.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas what it means or how I could get around it?  The page's biggest table is nested inside another table... is that a problem? &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; tables&amp;lt;-readHTMLTable(myURL, header=NA,a.data.frame=TRUE)&#xA;Error in htmlParse(doc) : &#xA;  error in creating parser for http://www.ncbi.nlm.nih.gov/genomes/lproks.cgi&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1463" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-02-10T17:53:56.070" LastActivityDate="2011-02-10T23:28:32.947" Title="Error creating parser in readHTMLTable" Tags="&lt;r&gt;&lt;dataset&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="7057" PostTypeId="1" CreationDate="2011-02-10T16:23:48.187" Score="4" ViewCount="421" Body="&lt;p&gt;Hello, I would like to get the coefficients for the LASSO problem &#xA;||Y-Xb||+lambda*sum(|b_j|)&#xA;The problem is that glmnet and lars functions give different answers. For the glmnet function I ask for the coefficients of lambda/length(Y) instead of just lambda, but I still get different answers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this expected? What is the relationship between the lars lambda and glmnet lambda? I understand that glmnet is faster for LASSO problems but I would like to know which method is more powerful?&lt;/p&gt;&#xA;" OwnerUserId="3139" LastActivityDate="2011-04-12T20:08:59.603" Title="GLMNET or LARS for computing LASSO solutions?" Tags="&lt;r&gt;&lt;machine-learning&gt;&lt;regression&gt;&lt;lasso&gt;&lt;regularization&gt;" AnswerCount="3" CommentCount="2" FavoriteCount="2" />
  <row Id="7058" PostTypeId="1" AcceptedAnswerId="7169" CreationDate="2011-02-10T16:35:26.570" Score="7" ViewCount="76" Body="&lt;p&gt;I have three features that I use to solve a classification problem. Originally, these features produced boolean values, so I could evaluate their redundancy by looking at how much the sets of positive and negative classifications overlap. Now I have extended the features to produce real values (scores) instead, and I would like to analyze their redundancy again, but I am at a complete loss on how to do that. Can anyone provide me with a pointer or idea on how to go about that?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know this question is very vague, that is because I do not have a very strong grasp of statistics. So, if you do not have an answer for me, maybe you have some questions that can help me understand better myself.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt; I am currently browsing Wikipedia on the subject, I have the feeling that what I want is a correlation coefficient, but I am still unsure if this is the right approach, and which of the many available coefficients is appropriate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit 2:&lt;/strong&gt; In the boolean case, I first created for each feature the set of samples for which it was true. Then, the correlation between two features was the size of intersection of these sets over the size of the union of these sets. If this value is 1, they are completely redundant, because always the same. If it is 0, they are never the same.&lt;/p&gt;&#xA;" OwnerUserId="977" LastEditorUserId="977" LastEditorDisplayName="" LastEditDate="2011-02-10T17:33:23.853" LastActivityDate="2011-02-13T22:04:12.390" Title="How to quantify redundancy of features?" Tags="&lt;correlation&gt;&lt;feature-selection&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="7059" PostTypeId="1" CreationDate="2011-02-10T18:02:16.093" Score="5" ViewCount="216" Body="&lt;p&gt;I am managing many people entering data into a database. I have a log of user, date, time, table, and action that each person makes:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;records &amp;lt;- data.frame(user = c('bob', 'bob', 'jane', 'jane', 'bob', 'bob', 'bob', 'jane', 'jane', 'bob'),&#xA;                      date = c(&quot;2010-06-24&quot;, &quot;2010-06-28&quot;, &quot;2010-06-29&quot;, &quot;2010-06-30&quot;, &quot;2010-07-01&quot;, &quot;2010-07-02&quot;, &quot;2010-07-05&quot;, &quot;2010-07-06&quot;, &quot;2010-07-07&quot;, &quot;2010-07-09&quot;), &#xA;                      time = c(&quot;01:40:08&quot;, &quot;01:40:18&quot;, &quot;01:40:28&quot;, &quot;01:40:37&quot;, &quot;01:40:44&quot;, &quot;01:40:52&quot;, &quot;01:40:59&quot;, &quot;01:56:26&quot;, &quot;02:16:37&quot;, &quot;03:55:06&quot;),&#xA;                      table = c(rep('table1',5), rep('table2',5)),&#xA;                      action = c('create', 'create', 'create', 'update', 'create', 'update', 'update', 'create', 'create', 'create')) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For a non-trivial example, the actual &lt;code&gt;records&lt;/code&gt; dataframe with 10,000 entries can be downloaded as an .Rdata file &lt;a href=&quot;http://dl.dropbox.com/u/18092793/records.Rdata&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;, and then:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;load('records.Rdata')&#xA;library(ggplot2)&#xA;qplot(date, table, data = records, color = user, geom='jitter')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can I visualize, overall and for each table:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;the amount of time each person works per week&lt;/li&gt;&#xA;&lt;li&gt;the type and number or frequency of actions that they made.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;?&lt;/p&gt;&#xA;" OwnerUserId="1381" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-02-11T08:08:51.310" LastActivityDate="2011-02-12T21:34:48.187" Title="How can I effectively summarize and visualize time series of employee activities?" Tags="&lt;r&gt;&lt;time-series&gt;&lt;data-visualization&gt;" AnswerCount="1" CommentCount="7" FavoriteCount="1" />
  <row Id="7061" PostTypeId="1" AcceptedAnswerId="7078" CreationDate="2011-02-10T19:10:16.860" Score="2" ViewCount="155" Body="&lt;p&gt;Given two basketball players.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;John made 38/50 free throws.&#xA;Mike made 80/100 free throws.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is probability that Mike is better at free throws than John?&lt;/p&gt;&#xA;" OwnerUserId="3143" LastActivityDate="2011-02-11T04:02:45.067" Title="Binomial Probability Question" Tags="&lt;probability&gt;&lt;binomial&gt;" AnswerCount="3" CommentCount="4" FavoriteCount="2" />
  <row Id="7062" PostTypeId="2" ParentId="7057" CreationDate="2011-02-10T19:19:24.763" Score="0" ViewCount="" Body="&lt;p&gt;LASSO is non-unique in the case where multiple features have perfect collinearity.  Here's a simple thought experiment to prove it.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say you have three random vectors $y$, $x_1$, $x_2$.  You're trying to predict $y$ from $x_1$, $x_2$.  Now assume $y$ = $x1$ = $x2$.  An optimal LASSO solution would be $\beta_1 = 1 - P$, $\beta_2 = 0$, where $P$ is the effect of LASSO penalty.  However, also optimal would be $\beta_1 = 0$, $\beta_2 - 1 - P$.&lt;/p&gt;&#xA;" OwnerUserId="1347" LastEditorUserId="1347" LastEditorDisplayName="" LastEditDate="2011-02-10T20:30:18.040" LastActivityDate="2011-02-10T20:30:18.040" CommentCount="1" />
  <row Id="7063" PostTypeId="2" ParentId="7053" CreationDate="2011-02-10T20:13:28.700" Score="1" ViewCount="" Body="&lt;p&gt;Yes.  It is certainly possible that this is due to something like Simpson's paradox.  If the data looked like &#xA;$$\begin{array}{rrrrrr}&#xD;&#xA;\textit{Organ}&amp;amp;\textit{Outcome}&amp;amp;A&amp;amp;B&amp;amp;C&amp;amp;D\\&#xD;&#xA;\textrm{Lung}&amp;amp;\textrm{Bad}&amp;amp;371&amp;amp;2727&amp;amp;2374&amp;amp;418\\&#xD;&#xA;\textrm{Lung}&amp;amp;\textrm{Good}&amp;amp;556&amp;amp;3199&amp;amp;2740&amp;amp;558\\&#xD;&#xA;\textrm{Heart}&amp;amp;\textrm{Bad}&amp;amp;214&amp;amp;245&amp;amp;195&amp;amp;273\\&#xD;&#xA;\textrm{Heart}&amp;amp;\textrm{Good}&amp;amp;8859&amp;amp;3828&amp;amp;4691&amp;amp;8752\\&#xD;&#xA;\end{array}$$&#xA;then I think you would get something like your hazard ratios (if that means ratios of bad outcomes/totals fractions).  Many other patterns of numbers would too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are reviewing the article, it seems reasonable to ask for the underlying numbers to be presented.  If they look anything like mine, then it does seem a little strange to add Lung and Heart numbers without a good reason.&lt;/p&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-02-10T20:13:28.700" />
  <row Id="7064" PostTypeId="2" ParentId="7053" CreationDate="2011-02-10T20:17:34.793" Score="2" ViewCount="" Body="&lt;p&gt;Strictly, &lt;a href=&quot;http://en.wikipedia.org/wiki/Simpson%27s_paradox&quot; rel=&quot;nofollow&quot;&gt;Simpson's paradox&lt;/a&gt; refers to a &lt;em&gt;reversal&lt;/em&gt; in the direction of effect, which hasn't happened here as all the hazard ratios are above 1, so I'd refer to this by the more general term &lt;strong&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Confounding&quot; rel=&quot;nofollow&quot;&gt;confounding&lt;/a&gt;&lt;/strong&gt;. You can certainly have confounding in survival analysis. I agree it appears sensible to only present the heart and lung results separately.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-02-10T20:17:34.793" CommentCount="1" />
  <row Id="7066" PostTypeId="2" ParentId="7061" CreationDate="2011-02-10T21:11:36.403" Score="1" ViewCount="" Body="&lt;p&gt;I think what you want to do is compare the predictive distributions of both players. The predictive distribution describes the probability that Mike/John will make his next shot given the data (integrating out the parameters). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is some Matlab code you can play with:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;clear;&#xA;clc;&#xA;rng = linspace(0, 1, 100);&#xA;&#xA;% data&#xA;j = [38 50];&#xA;m = [80 100];&#xA;&#xA;% priors (I assume uniform, but you can integrate external knowledge here)&#xA;a0 = 1;&#xA;b0 = 1;&#xA;&#xA;% posterior distribution for each player&#xA;post_j = [a0 + j(1), b0 + j(2) - j(1)];&#xA;post_m = [a0 + m(1), b0 + m(2) - m(1)];&#xA;&#xA;% visualize&#xA;postj = betapdf(rng, a0 + j(1), b0 + j(2) - j(1));&#xA;postm = betapdf(rng, a0 + m(1), b0 + m(2) - m(1));&#xA;figure(1); plot(rng, postj, 'r',...&#xA;                rng, postm, 'k');&#xA;title('Posterior Distributions');&#xA;legend('Jon', 'Mike');&#xA;xlabel('Theta');&#xA;&#xA;%% SAMPLING FROM PREDICTIVE &#xA;&#xA;SAMPLES = 5000;&#xA;&#xA;predj = zeros(SAMPLES, 1);&#xA;predm = zeros(SAMPLES, 1);&#xA;for i = 1:SAMPLES&#xA;    pj = betarnd(post_j(1), post_j(2));&#xA;    lj = binornd(1, pj);&#xA;    predj(i) = lj;&#xA;&#xA;&#xA;    pm = betarnd(post_m(1), post_m(2));&#xA;    lm = binornd(1, pm);&#xA;    predm(i) = lm;&#xA;end&#xA;&#xA;% Comparison of (sampled) predictive distributions:&#xA;fprintf('P(d_john | Dj) &amp;gt; P(d_mike | Dm) = %.3f\n', sum(predj &amp;gt; predm) / SAMPLES);&#xA;fprintf('P(d_john | Dj) &amp;lt; P(d_mike | Dm) = %.3f\n', sum(predj &amp;lt; predm) / SAMPLES);&#xA;fprintf('P(d_john | Dj) = P(d_mike | Dm) = %.3f\n', sum(predj == predm) / SAMPLES);&#xA;&#xA;pred_j = sum(predj) / SAMPLES;&#xA;pred_m = sum(predm) / SAMPLES;&#xA;&#xA;fprintf('Probability that John will make his next shot: %.4f\n', pred_j);&#xA;fprintf('Probability that Mike will make his next shot: %.4f\n', pred_m);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1913" LastActivityDate="2011-02-10T21:11:36.403" CommentCount="7" />
  <row Id="7067" PostTypeId="2" ParentId="7047" CreationDate="2011-02-10T21:15:49.847" Score="1" ViewCount="" Body="&lt;p&gt;I don't know algorithms off the top of my head, but I would start by having a look at &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.86.8364&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;Survey of image registration techniques&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1913" LastActivityDate="2011-02-10T21:15:49.847" CommentCount="1" />
  <row Id="7068" PostTypeId="2" ParentId="7059" CreationDate="2011-02-10T21:27:30.477" Score="1" ViewCount="" Body="&lt;p&gt;Below the code to plot the numbers of actions per week/per user:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;load(&quot;records.Rdata&quot;)&#xA;library(ggplot2)&#xA;records$posdate &amp;lt;- as.POSIXlt(records$date,format=&quot;%Y-%m-%d&quot;)&#xA;records$week &amp;lt;- as.numeric(format(records$posdate,&quot;%W&quot;)) #changed from previous hack!&#xA;numberActions &amp;lt;- by(records$action,records[,c(&quot;user&quot;,&quot;week&quot;)],function(x) length(x[x!=&quot;Login&quot;]))&#xA;numberActions &amp;lt;- melt(t(numberActions[1:7,]),&quot;week&quot;)&#xA;colnames(numberActions)[2] &amp;lt;- &quot;user&quot;&#xA;ggplot(numberActions,aes(week,value,group=user,col=user))+geom_line()+geom_point(size=3)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I hope this helps. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the second part you can use &lt;code&gt;plyr&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;numberActions &amp;lt;- ddply(records,c(&quot;user&quot;,&quot;week&quot;),function(x) table(x$action))&#xA;numberActions &amp;lt;- melt(numberActions,c(&quot;user&quot;,&quot;week&quot;))&#xA;numberActions &amp;lt;- numberActions[numberActions$value!=0 &amp;amp; numberActions$variable!=&quot;Login&quot;,]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and then the usual &lt;code&gt;ggplot&lt;/code&gt;...&lt;/p&gt;&#xA;" OwnerUserId="1443" LastEditorUserId="1443" LastEditorDisplayName="" LastEditDate="2011-02-11T09:08:07.167" LastActivityDate="2011-02-11T09:08:07.167" CommentCount="2" />
  <row Id="7069" PostTypeId="1" CreationDate="2011-02-10T21:33:27.913" Score="2" ViewCount="91" Body="&lt;p&gt;Is it possible to correct for violating the assumption of independence for nonparametric tests? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a categorical independent variable and a categorical and binary dependent variable, and each subject was exposed to multiple levels of treatment. I have done some preliminary analysis with binary logistic regression and post-hoc chi-square tests, but I think these are inappropriate unless I can correct for my non-independent data. Info on this has been surprisingly difficult to find, and help is much appreciated.&lt;/p&gt;&#xA;" OwnerUserId="3145" LastActivityDate="2011-02-10T22:48:44.833" Title="Is it possible to correct for violating the assumption of independence for nonparametric tests?" Tags="&lt;nonparametric&gt;&lt;non-independent&gt;" AnswerCount="1" />
  <row Id="7070" PostTypeId="1" AcceptedAnswerId="7071" CreationDate="2011-02-10T22:13:25.977" Score="1" ViewCount="338" Body="&lt;p&gt;What is the best way of defining white noise process so it is intuitive and easy to understand&lt;/p&gt;&#xA;" OwnerUserId="333" LastActivityDate="2011-02-11T09:13:59.350" Title="What is white noise process?" Tags="&lt;time-series&gt;" AnswerCount="2" />
  <row Id="7071" PostTypeId="2" ParentId="7070" CreationDate="2011-02-10T22:23:25.730" Score="4" ViewCount="" Body="&lt;p&gt;A white noise process is one with a mean zero and no correlation between its values at different times. See the &lt;a href=&quot;http://en.wikipedia.org/wiki/White_noise#White_random_process_.28white_noise.29&quot; rel=&quot;nofollow&quot;&gt;'white random process' section of Wikipedia's article on white noise&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-02-10T22:23:25.730" CommentCount="5" />
  <row Id="7072" PostTypeId="2" ParentId="7069" CreationDate="2011-02-10T22:30:01.883" Score="2" ViewCount="" Body="&lt;p&gt;You may be having trouble finding info on this because you're looking for it under 'nonparametric tests'. This wouldn't usually be included under the (rather confusing) 'nonparametric' heading. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Logistic regression is sensible, but you'll need one of its extensions to deal with correlated outcomes, such as conditional (fixed-effects) logistic regression, random-effects logistic regression, or mixed-effects logistic regression. It's hard to say which is appropriate without knowing more about your data and the question(s) you wish to address.&lt;/p&gt;&#xA;" OwnerUserId="449" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-02-10T22:48:44.833" LastActivityDate="2011-02-10T22:48:44.833" CommentCount="3" />
  <row Id="7074" PostTypeId="1" AcceptedAnswerId="7083" CreationDate="2011-02-10T23:25:06.553" Score="2" ViewCount="138" Body="&lt;p&gt;Is there a formal statistical test to test if process is a white noise?&lt;/p&gt;&#xA;" OwnerUserId="333" LastEditorUserId="1036" LastEditorDisplayName="" LastEditDate="2011-02-12T15:39:03.473" LastActivityDate="2011-02-12T15:39:03.473" Title="Formal statistical test for whether a process is a white noise" Tags="&lt;time-series&gt;&lt;white-noise&gt;" AnswerCount="1" CommentCount="4" />
  <row Id="7075" PostTypeId="2" ParentId="7056" CreationDate="2011-02-10T23:28:32.947" Score="1" ViewCount="" Body="&lt;p&gt;The table in question seems to need use of a Javascript button to download.  If you just want this particular data then save it as the offered &lt;em&gt;untitle.txt&lt;/em&gt; or something else (it is basically tab-delimited) and then &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;tables           &amp;lt;- read.delim(&quot;untitle.txt&quot;, skip=1)&#xA;colnames(tables) &amp;lt;- c(colnames(tables)[-1],&quot;junk&quot;)&#xA;tables$junk      &amp;lt;- NULL&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;seems to give you what I believe you want. The second and third lines of the R code are to deal with &quot;## Columns:&quot; which starts the second line of the file and upsets &lt;em&gt;read.delim()&lt;/em&gt; when trying to get the column names.  &lt;/p&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-02-10T23:28:32.947" CommentCount="1" />
  <row Id="7076" PostTypeId="2" ParentId="7049" CreationDate="2011-02-11T01:29:36.383" Score="3" ViewCount="" Body="&lt;p&gt;The key assumption of a generalized linear model that's relevant here is the relationship between the variance and mean of the response, given the values of the predictors. When you specify a Poisson distribution, what this implies is that you are assuming the conditional variance is equal to the conditional mean.* The actual &lt;em&gt;shape&lt;/em&gt; of the distribution doesn't matter as much: it could be Poisson, or gamma, or normal, or anything else as long as that mean-variance relationship holds.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;* You can relax the assumption that the variance equals the mean to one of proportionality, and still usually get good results.&lt;/p&gt;&#xA;" OwnerUserId="1569" LastActivityDate="2011-02-11T01:29:36.383" />
  <row Id="7077" PostTypeId="2" ParentId="7061" CreationDate="2011-02-11T03:47:17.390" Score="0" ViewCount="" Body="&lt;p&gt;This is very much a problem in Bayesian inference.  You appear to have taken the first step in realizing that the sample is not the same as the underlying probability distribution.  Even though Mike had a higher mean in his sample, John might have a higher mean for his true talent distribution.  When applying Bayesian inference, we start with a prior distribution, use the evidence to update our knowledge, and come out with a posterior distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The point I alluded to in my comment is that the prior knowledge can matter a great deal.  If we happen to know that the &quot;John&quot; happens to be basketball Hall of Famer John Stockton and the &quot;Mike&quot; happens to be me, then 100 shots are not going to be nearly enough to convince you that I'm better.  You prior distribution for John Stockton's true talent would probably be somewhat tight with a mean near .8 or so (he was .8261 for his career), whereas who knows for me...it might be .3 for all I know.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are estimating the true talent probabilities of making a free throw, you at least know that the values must lie in (0, 1), so a minimal prior distribution for each would be the uniform probability distribution on (0, 1).  This is equivalent to using the Beta distribution with parameters $\alpha = \beta = 1$.  Through some calculations I won't get into (along with Bayes theorem), it turns out that the posterior distribution is also a Beta distribution with parameters $\alpha + i$ and $\beta + j$ (assuming the player made $i$ shots in $i + j$ attempts).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So our posterior distribution for John is $Beta(39, 13)$ and for Mike it is $Beta(81, 21)$.  So with these priors our posterior estimate of John's mean talent is $\frac{39}{52} = .75$ and for Mike it is $\frac{81}{102} \approx .794.$  I believe finding the answer to how likely Mike is better in a nice closed form is messy, and I don't have the tools handy to calculate the exact answer, but you would integrate/sum the portions of those curves where Mike is better to find the answer.  Nick's code looks like it will probably get you a nice approximation.  I can tell you that with the prior I chose Mike will be higher (since they had the same prior and Mike had the higher sample mean, it does pass the smell test that Mike will have a higher posterior mean).  &lt;/p&gt;&#xA;" OwnerUserId="2485" LastActivityDate="2011-02-11T03:47:17.390" />
  <row Id="7078" PostTypeId="2" ParentId="7061" CreationDate="2011-02-11T03:55:13.450" Score="1" ViewCount="" Body="&lt;p&gt;Okay. I think I figured out the general answer:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given a sample of n/N made, the probability that the population success rate is greater than x is defined by the posterior probability distribution:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;$\frac{\int_{x}^1 r^n(1-r)^{N-n}\ dr}{\int_{0}^1 r^n(1-r)^{N-n}\ dr}$&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So for my example the chance that Mike's free throw percentage is greater than x is:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;$\frac{\int_{x}^1 r^{80}(1-r)^{20}\ dr}{\int_{0}^1 r^{80}(1-r)^{20}\ dr}$&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;We then need to integrate that over the probability distribution for John's free throw percentages:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;$\frac{\int_{0}^1 \int_{x}^1 x^{38}(1-x)^{12} r^{80}(1-r)^{20}\ dr\ dx}{\int_{0}^1 x^{38}(1-x)^{12}\ dx \int_{0}^1 r^{80}(1-r)^{20}\ dr}$&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Which as whuber said is ~ 72.66%&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the WolframAlpha solution:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Integrate[y^38 (1 - y)^12 x^80 (1 - x)^20, {x, 0, 1}, {y, 0, x}]/(Integrate[y^38 (1 - y)^12, {y, 0, 1}] Integrate[x^80 (1 - x)^20, {x, 0, 1}])&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="3143" LastEditorUserId="3143" LastEditorDisplayName="" LastEditDate="2011-02-11T04:02:45.067" LastActivityDate="2011-02-11T04:02:45.067" CommentCount="2" />
  <row Id="7079" PostTypeId="1" CreationDate="2011-02-11T06:27:08.743" Score="4" ViewCount="607" Body="&lt;ol&gt;&#xA;&lt;li&gt;How do you carry out analysis of covariance using R? &lt;/li&gt;&#xA;&lt;li&gt;How do you interpret the results? &lt;/li&gt;&#xA;&lt;li&gt;A practical example will be highly appreciated.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="3107" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-02-12T05:49:02.013" LastActivityDate="2011-02-12T05:49:02.013" Title="Analysis of covariance in R" Tags="&lt;r&gt;&lt;ancova&gt;" AnswerCount="1" />
  <row Id="7080" PostTypeId="1" CreationDate="2011-02-11T06:33:47.117" Score="0" ViewCount="84" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;Possible Duplicate:&lt;/strong&gt;&lt;br&gt;&#xA;  &lt;a href=&quot;http://stats.stackexchange.com/questions/871/correct-formatting-capitalization-italicization-hyphenation-of-p-value&quot;&gt;Correct formatting (capitalization, italicization, hyphenation) of &amp;ldquo;p-value&amp;rdquo;?&lt;/a&gt;  &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&#xA;&#xA;&lt;p&gt;What is the difference belween p,P and italic &lt;em&gt;p&lt;/em&gt; in statistics?&lt;/p&gt;&#xA;" OwnerUserId="2956" LastActivityDate="2011-02-11T06:33:47.117" ClosedDate="2011-02-11T07:29:34.140" Title="difference belween p,P and italic p in statistics" Tags="&lt;probability&gt;" CommentCount="0" />
  <row Id="7081" PostTypeId="2" ParentId="7070" CreationDate="2011-02-11T07:35:13.443" Score="1" ViewCount="" Body="&lt;p&gt;I myself usually think of white noise as an iid sequence with zero mean. At different times values of the process are then independent of each other, which is much stronger requirement than correlation zero. What is the best with this definition that it works in any context. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Side note.&lt;/em&gt; I only explained my intuition, the correct definition of white noise is given by @onestop. The definition I gave is mathematically defined as white noise in strict sense.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-02-11T09:13:59.350" LastActivityDate="2011-02-11T09:13:59.350" />
  <row Id="7082" PostTypeId="2" ParentId="7079" CreationDate="2011-02-11T07:44:39.227" Score="4" ViewCount="" Body="&lt;p&gt;Here is a &lt;a href=&quot;http://www3.imperial.ac.uk/pls/portallive/docs/1/1171922.PDF&quot; rel=&quot;nofollow&quot;&gt;detailed presentation&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="339" LastActivityDate="2011-02-11T07:44:39.227" CommentCount="2" />
  <row Id="7083" PostTypeId="2" ParentId="7074" CreationDate="2011-02-11T07:47:54.650" Score="6" ViewCount="" Body="&lt;p&gt;In time-series analysis usually &lt;a href=&quot;http://en.wikipedia.org/wiki/Ljung-Box_test&quot; rel=&quot;nofollow&quot;&gt;Ljung-Box test&lt;/a&gt; is used. Note though that it tests the correlations. If the correlations are zero, but variance varies, then the process is not white noise, but Ljung-Box test will fail to reject the null-hypothesis. Here is an example in R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; Box.test(c(rnorm(100,0,1),rnorm(100,0,10)),type=&quot;Ljung-Box&quot;)&#xA;&#xA;    Box-Ljung test&#xA;&#xA;data:  c(rnorm(100, 0, 1), rnorm(100, 0, 10)) &#xA;X-squared = 0.4771, df = 1, p-value = 0.4898&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here is the plot of the process:&#xA;&lt;img src=&quot;http://i.imgur.com/DiFBx.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is more discussion &lt;a href=&quot;http://stats.stackexchange.com/questions/6455/how-many-lags-to-use-in-the-ljung-box-test-of-a-time-series&quot;&gt;about this test&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-02-11T07:47:54.650" CommentCount="1" />
  <row Id="7084" PostTypeId="1" AcceptedAnswerId="7543" CreationDate="2011-02-11T09:09:36.410" Score="2" ViewCount="311" Body="&lt;h3&gt;Background:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;In connection with the question &lt;a href=&quot;http://stats.stackexchange.com/questions/7022/parameter-estimation-for-normal-distribution-in-java&quot;&gt;here&lt;/a&gt; I came upon a more interesting question. I believe the question is large and distinct enough to have it's own thread. Of course I might be mistaken, in that case I apologize for the clutter...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a dataset of experimental values, coming from an unknown distribution. The data is then used in an analysis, and as a measure of significance of the results of the analysis, I would like to sample random data from the same distribution and run my analysis on these datasets (say 1000 times the size of the original dataset) to see if the results from experimental values show significant deviation from randomized data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was thinking about drawing samples from a normal distribution, as it feels most natural that the samples come from some normal distribution. However I need to back up my assumption of normality somehow. I was originally thinking of using some sort of a normality test, but after some reading on the matter such as &lt;a href=&quot;http://stats.stackexchange.com/questions/2/what-is-normality&quot;&gt;What is 'normality?'&lt;/a&gt; and &lt;a href=&quot;http://stats.stackexchange.com/questions/2492/normality-testing-essentially-useless&quot;&gt;Normality testing: 'Essentially useless?'&lt;/a&gt; threads, and of course the Wikipedia article on &lt;a href=&quot;http://en.wikipedia.org/wiki/Normality_test&quot; rel=&quot;nofollow&quot;&gt;normality tests&lt;/a&gt;; I feel like these tests are not an accepted way of validating normality assumptions. &lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Question:&lt;/h3&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;How can normality be validated without using visual cues such as QQ plots? (the validation will be a part of larger software) &lt;/li&gt;&#xA;&lt;li&gt;Can a &quot;goodness of fit&quot; score be calculated? &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;&lt;strong&gt;EDIT:&lt;/strong&gt;&lt;/em&gt; As requested, I'll try and add some details about the data. The data at hand are from a biological experiment, however the instrumentation has high variation between the runs. The analysis I've mentioned takes the measured values and using a mathematical model evaluates functional meaning of the measured data. To do so, I need to see how unrelated/uncorrelated, made-up data rates in the same analysis, hence the intent to model by randomized values. I hope this clarifies my point of view. &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;em&gt;EDIT2:&lt;/em&gt;&lt;/strong&gt; There has been a series of questions, asking for clarification on the question. Both here and in the comments below I tried to explain my situation to my best ability. It seems like we are suffering from a communication mismatch... I dunno how I can give an example without writing up a long table and complicating things further OR brutally simplifying the bigger picture. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have no doubt that everyone who took their time and supply a reply has the best intentions, but I really appreciate if you could focus on the question at hand instead of inquiring further and further into the motivations behind why I need to do things this way and not another way. &lt;/p&gt;&#xA;" OwnerUserId="3014" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-02-23T16:22:29.813" LastActivityDate="2011-02-23T18:11:56.453" Title="Justifying normal approximation of experimental data" Tags="&lt;normality&gt;&lt;test&gt;&lt;goodness-of-fit&gt;&lt;approximation&gt;" AnswerCount="4" CommentCount="5" />
  <row Id="7085" PostTypeId="2" ParentId="6225" CreationDate="2011-02-11T09:27:37.847" Score="0" ViewCount="" Body="&lt;p&gt;There is a case where a proof is possible.  Suppose you have a school and your null hypothesis is that the numbers of boys and of girls is equal.  As the sample size increases, the uncertainty in the ratio of boys to girls tends to reduce, eventually reaching certainty (which is what I assume you mean by proof) when the whole pupil population is sampled.    &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But if you do not have a finite population, or if you are sampling with replacement and cannot spot resampled individuals, then you cannot reduce the uncertainty to zero with a finite sample. &lt;/p&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-02-11T09:27:37.847" />
  <row Id="7086" PostTypeId="2" ParentId="7084" CreationDate="2011-02-11T09:31:33.927" Score="5" ViewCount="" Body="&lt;p&gt;If your ultimate aim is, as you say, &quot;to see if the results from experimental values show significant deviation from randomized data&quot; then you'd be better to directly use the observed distribution by performing a &lt;a href=&quot;http://en.wikipedia.org/wiki/Resampling_%28statistics%29#Permutation_tests&quot; rel=&quot;nofollow&quot;&gt;permutation (re-randomization) test&lt;/a&gt;. This avoids making any assumptions about the 'true' distribution from which the data are drawn.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-02-11T09:31:33.927" CommentCount="5" />
  <row Id="7087" PostTypeId="2" ParentId="7057" CreationDate="2011-02-11T09:52:26.037" Score="0" ViewCount="" Body="&lt;p&gt;@deps_stats I am afraid that the size of my dataset is so large that LARS can not handle it, whereas on the other hand glmnet can handle my large dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;@mpiktas I want to find the solution of (Y-Xb)^2+L\sum|b_j|&#xA;but when I ask from the two algorithms(lars &amp;amp; glmnet) for their calculated coefficients for that particular L, I get different answers...and I wondering is that correct/ expected? or I am just using a wrong lambda for the two functions.&lt;/p&gt;&#xA;" OwnerUserId="3139" LastActivityDate="2011-02-11T09:52:26.037" CommentCount="4" />
  <row Id="7088" PostTypeId="1" AcceptedAnswerId="7119" CreationDate="2011-02-11T11:41:17.083" Score="3" ViewCount="122" Body="&lt;p&gt;I am doing some class work and i was wondering if there is a universally accepted format for analyzing and presenting the data in a word document for data gathered from questionnaires.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="3150" LastActivityDate="2011-02-12T05:18:18.587" Title="Academic data analysis format for questionaires" Tags="&lt;questionnaires&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="1" />
  <row Id="7089" PostTypeId="1" CreationDate="2011-02-11T12:30:24.763" Score="4" ViewCount="205" Body="&lt;p&gt;I need a formula for the probability of an event in a n-variate Bernoulli distribution $X\in\{0,1\}^n$ with given $P(X_i=1)=p_i$ probabilities for a single element and for pairs of elements $P(X_i=1 \wedge X_j=1)=p_{ij}$. Equivalently I could give mean and covariance of $X$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I already learned that there exist many $\{0,1\}^n$ distributions having the properties just as there are many distributions having a given mean and covariance. I am looking for a canonical one on $\{0,1\}^n$, just as the Gaussian is a canonical distribution for $R^n$ and a given mean and covariance.&lt;/p&gt;&#xA;" OwnerUserId="3152" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-02-11T12:51:48.473" LastActivityDate="2011-02-14T12:48:07.520" Title="Probability formula for a multivariate-bernoulli distribution" Tags="&lt;multivariate-analysis&gt;&lt;discrete-data&gt;" AnswerCount="2" />
  <row Id="7090" PostTypeId="1" CreationDate="2011-02-11T13:11:47.690" Score="2" ViewCount="70" Body="&lt;p&gt;Hello !&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When generating survival times to simulate Cox proportional hazards models, does it matter to generate them in days or in years?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In theory, I guess it does not matter. But in practice? Is there a preference regarding computational issues?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you!&#xA;Marco&lt;/p&gt;&#xA;" OwnerUserId="3019" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-02-13T14:35:26.377" LastActivityDate="2011-02-13T14:35:26.377" Title="Generating survival times in days or in years" Tags="&lt;survival&gt;&lt;hazard&gt;&lt;units&gt;" CommentCount="4" />
  <row Id="7091" PostTypeId="2" ParentId="7089" CreationDate="2011-02-11T13:19:26.237" Score="3" ViewCount="" Body="&lt;p&gt;The random variable taking values in $\{0,1\}^n$ is a discrete random variable. Its distribution  is fully described by probabilities &#xA;$p_{\mathbf{i}}=P(X=\mathbf{i})$ with $\mathbf{i}\in\{0,1\}^n$. The probabilities $p_{i}$ and $p_{ij}$ you give are sums of $p_{\mathbf{i}}$ for certain indexes $\mathbf{i}$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now it seems that you want to describe $p_{\mathbf{i}}$ by only using $p_i$ and $p_{ij}$. It is not possible without assuming certain properties on $p_{\mathbf{i}}$. To see that try to derive &lt;a href=&quot;http://en.wikipedia.org/wiki/Characteristic_function_%28probability_theory%29&quot; rel=&quot;nofollow&quot;&gt;characteristic function&lt;/a&gt; of $X$. If we take $n=3$ we get&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align}&#xA;Ee^{i(t_1X_1+t_2X_2+t_3X_3)}&amp;amp;=p_{000}+p_{100}e^{it_1}+p_{010}e^{it_2}+p_{001}e^{it_3}\\&#xA;&amp;amp;+p_{110}e^{i(t_1+t_2)}+p_{101}e^{i(t_1+t_3)}+p_{010}e^{i(t_2+t_3)}+p_{111}e^{i(t_1+t_2+t_3)}&#xA;\end{align}&#xA;It is not possible rearrange this expression so that $p_{\mathbf{i}}$ dissapear. For the gaussian random variable the characteristic function depends only on mean and covariance parameters. Characteristic functions uniquely define distributions, so this is why Gaussian can be described uniquely by using only mean and covariance. As we see for random variable $X$ this is not the case.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-02-11T13:34:42.417" LastActivityDate="2011-02-11T13:34:42.417" />
  <row Id="7092" PostTypeId="2" ParentId="5465" CreationDate="2011-02-11T13:26:25.447" Score="0" ViewCount="" Body="&lt;p&gt;How will you improve the revenue of a book seller, say amazon ? Explain different ways.&lt;/p&gt;&#xA;" OwnerUserId="3153" LastActivityDate="2011-02-11T13:26:25.447" CommunityOwnedDate="2011-02-11T13:26:25.447" />
  <row Id="7093" PostTypeId="2" ParentId="5465" CreationDate="2011-02-11T13:31:52.880" Score="-1" ViewCount="" Body="&lt;p&gt;How will you count the number of sandal wood trees in Bangalore ?&lt;/p&gt;&#xA;" OwnerUserId="3153" LastActivityDate="2011-02-11T13:31:52.880" CommunityOwnedDate="2011-02-11T13:31:52.880" />
  <row Id="7094" PostTypeId="2" ParentId="7089" CreationDate="2011-02-11T13:39:43.833" Score="1" ViewCount="" Body="&lt;p&gt;I don't know what the resulting distribution is called, or if it even has a name, but it strikes me the obvious way to set this up is to think of the model you'd use to model a 2×2×2×…×2 table using a log-linear (Poisson regression) model. As you know the 1st-order interactions only, it's then natural to assume that all higher-order interactions are zero.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using the questioner's notation, this gives the model:&#xA;$$P(X_1=x_1, X_2=x_2,\ldots,X_n=x_n) = &#xD;&#xA;\prod_i \left[ p_i^{x_i}(1-p_i)^{1-x_i} &#xD;&#xA;\prod_{j&amp;lt;i} \left(\frac{p_{ij}}{p_i p_j}\right)^{x_i x_j} \right] $$&lt;/p&gt;&#xA;" OwnerUserId="449" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-02-14T12:48:07.520" LastActivityDate="2011-02-14T12:48:07.520" CommentCount="3" />
  <row Id="7095" PostTypeId="2" ParentId="5465" CreationDate="2011-02-11T13:40:00.073" Score="0" ViewCount="" Body="&lt;p&gt;We are running a customer service centre. We are getting 1 million calls per month. How do we reduce it to ten thousand ?&lt;/p&gt;&#xA;" OwnerUserId="3153" LastActivityDate="2011-02-11T13:40:00.073" CommunityOwnedDate="2011-02-11T13:40:00.073" CommentCount="3" />
  <row Id="7096" PostTypeId="1" CreationDate="2011-02-11T13:48:51.977" Score="1" ViewCount="85" Body="&lt;p&gt;Hello everybody on the site,&#xA;Do any of you know whether R has a package/routine for estimating spline function with unknown number of knots?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank in advance.&lt;/p&gt;&#xA;" OwnerUserId="3154" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2011-02-12T00:53:09.813" LastActivityDate="2011-02-12T00:53:09.813" Title="Spline function with unknown knots" Tags="&lt;r&gt;&lt;splines&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="7097" PostTypeId="2" ParentId="7096" CreationDate="2011-02-11T13:56:45.177" Score="2" ViewCount="" Body="&lt;p&gt;Have a look at the facilities for fitting smoothing splines in the &lt;code&gt;gam&lt;/code&gt; package.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-02-11T13:56:45.177" CommentCount="1" />
  <row Id="7098" PostTypeId="2" ParentId="7048" CreationDate="2011-02-11T14:21:54.300" Score="4" ViewCount="" Body="&lt;p&gt;Hi Tal,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To me interactive visualization is useful only for my own exploration, or when working with a very hands-on client. When dealing with a final presentation, I prefer to choose the static graph that best makes my point. Otherwise clients can get totally distracted by the gee-whiz factor. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The biggest benefit I get from it is a level of speed that frees me to examine far more than I would have if I stopped to program a solution. JMP is one of my favorite tools for this since it integrates so much of what I want into a single interface. I think most people who are good statistical programmers try something like JMP (or GGobi, etc.) for far too short a period to really get good at it. JMP in particular will give you the impression you know it by just looking over the menus. However, working through the manual is really required to uncover all its power. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You mentioned my main concern about this level of speed though: you end up with absolutely no idea of what your p-values mean. In just a few minutes you can examine hundreds of relationships visually. Doing hypothesis testing after all that is totally misleading, but I see people do that all the time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A feature I love in GGobi is its projection pursuit, in which you specify what type of pattern you seek in a high dimensional space and then you sit back and watch it &quot;pursue&quot; that goal. Great stuff!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cheers,&#xA;Bob Muenchen&lt;/p&gt;&#xA;" OwnerUserId="3155" LastActivityDate="2011-02-11T14:21:54.300" CommentCount="2" />
  <row Id="7099" PostTypeId="2" ParentId="5465" CreationDate="2011-02-11T15:01:37.667" Score="2" ViewCount="" Body="&lt;p&gt;Two questions I've been asked:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1)  You fit a multiple regression to examine the effect of a particular variable a worker in another department is interested in.  The variable comes back insignificant, but your co-worker says that this is impossible as it is known to have an effect.  What would you say/do?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2)  You have 1000 variables and 100 observations.  You would like to find the significant variables for a particular response.  What would you do?&lt;/p&gt;&#xA;" OwnerUserId="2310" LastActivityDate="2011-02-11T15:01:37.667" CommunityOwnedDate="2011-02-11T15:01:37.667" />
  <row Id="7100" PostTypeId="1" CreationDate="2011-02-11T15:52:57.197" Score="3" ViewCount="115" Body="&lt;p&gt;I am doing research and trying to use a questionnaire that is available in English in another  language. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Is there a systematic approach for validating a questionnaire in another language? &lt;/li&gt;&#xA;&lt;li&gt;What statistical test should I be undertake and why?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="3157" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-02-12T04:36:55.440" LastActivityDate="2011-02-12T09:00:13.563" Title="Validating an existing questionnaire into another language" Tags="&lt;correlation&gt;&lt;factor-analysis&gt;&lt;reliability&gt;&lt;validation&gt;" AnswerCount="3" />
  <row Id="7101" PostTypeId="1" CreationDate="2011-02-11T08:01:40.737" Score="3" ViewCount="141" Body="&lt;p&gt;I am interested in applying Bayesian additive regression trees (BART) for classification analysis of gene expression data. I am relatively new to R (and Bioconductor packages) and I am unable to find some code or vignette that I can use to learn from. I will be thankful if someone can point me in a good direction.&lt;/p&gt;&#xA;" OwnerUserId="4045" LastActivityDate="2011-02-11T20:04:18.110" Title="Bayesian additive regression trees (BART) for classification analysis of gene expression data" Tags="&lt;r&gt;&lt;classification&gt;&lt;bayesian&gt;&lt;regression&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="7102" PostTypeId="1" AcceptedAnswerId="7104" CreationDate="2011-02-11T16:05:49.630" Score="3" ViewCount="151" Body="&lt;p&gt;I'm fitting a 4 parameter nonlinear regression model to multiple datasets, some of which fail to converge, however, the parameters output after a failure provide a fit that looks good, if not exceptional to my (and other's) eyes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've explored convergence criteria and they do converge eventually but the visual fit is terrible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any precedent for taking the visual fit ad ignoring convergence or are there some other things I can try?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm fitting a model of the form (sqrt(c1*x+c2*exp(x)^y)/sqrt(exp(x)^y)+c3 where y is a known constant, using nlminb in R.&lt;/p&gt;&#xA;" OwnerUserId="3158" LastActivityDate="2011-02-11T21:06:12.897" Title="Non-linear regression fails to converge, but fit appears good" Tags="&lt;r&gt;&lt;curve-fitting&gt;&lt;nonlinear-regression&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="7103" PostTypeId="1" CreationDate="2011-02-11T17:29:15.263" Score="6" ViewCount="100" Body="&lt;p&gt;&lt;strong&gt;Example of problem:&lt;/strong&gt; Part of our research team is working on providing operationally wind power forecast. Usually, since there are different time scalse that interest forecast user, a forecast is issued every 15 min (it has even happened that 5 seconds was requirer) for every 15 minutes ahead up to serveral days. Obviously when you contract with a wind farm owner he has many wind farms, can be 10 or even 1000 wind farms depending of who you are working with. It is likelly that this number is going to be bigger in the near future. Now if the forecast user wants not only the point forecast but the forecast of 100 intesreting quantile (i.e. the whole marginal distribution) and if he wants you to store everything in a data base year after year, then you start tinking ..... ok that makes a lot of data. I'm not an international meteorological institute and cannot afford that. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit (according to @whuber's comment):&lt;/strong&gt; to make it critical it can be tough of with a 5 second time frame. But anyway this is not meant to be the question, just an example (people coule provide other examples)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My question is&lt;/strong&gt; : is there a compression theory/best practice for time series of distributions (i.e function of time with values in a space of distribution). I agree that you could work quantile by quantile and apply simple compression algorithm dedicated to signals (could be based on wavelet) but I am searhing for something more dedicated. For example if you know everything is well approximated by gaussians, the quantile by quantile approach would be stupid !&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit (according to @whuber's comment):&lt;/strong&gt; The question is really how to integrate the temporal dimension in a compression scheme,   hence nI don't want to make a choice of a subset of quantiles that would be the same for ALL time. &#xA;The remark with gaussian is not dedicated to the example, it is an example to illustrate a case when very good compression is required. An other trivial remark (from the theoretical side) would be: imagine you have a very complicated distribution that cannot be summarized with few quantiles and that you observe it for 1000 successive times. It would be sad to store it for each of these times. &lt;/p&gt;&#xA;" OwnerUserId="223" LastEditorUserId="223" LastEditorDisplayName="" LastEditDate="2011-02-12T07:23:26.243" LastActivityDate="2011-03-14T12:55:11.880" Title="Compression theory, practice, for time series with values in a space of distributions (say of a real random variable)" Tags="&lt;distributions&gt;&lt;time-series&gt;&lt;signal-processing&gt;&lt;quantiles&gt;&lt;compression&gt;" AnswerCount="2" CommentCount="4" FavoriteCount="1" />
  <row Id="7104" PostTypeId="2" ParentId="7102" CreationDate="2011-02-11T17:44:46.580" Score="6" ViewCount="" Body="&lt;p&gt;I will assume the values of all the variables and constants are such that there won't be problems with obtaining square roots of negative numbers.  Then&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{\sqrt{c_1 x + c_2 \exp(x)^y}}{\sqrt{\exp(x)^y}} + c_3 =\sqrt{c_2 + c_1 x \exp(-y x)} + c_3.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When $y \gt 0$ then eventually, for sufficiently large $x$, $\frac{c_1}{c_2} x \exp(-y x)$ gets small and the fit can be closely approximated as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$(\sqrt{c_2} + c_3) + \frac{c_1}{2 c_2} x \exp(-y x).$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Otherwise, when $y \lt 0$ and $c_1 \ne 0$, eventually (for largish positive $x$) the value of $c_2$ is inconsequential compared to the exponential and the fit simplifies to&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\sqrt{c_1 x} \exp(-y x / 2) + c_3.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In either case there are redundant parameters: $\sqrt{c_2} + c_3$ can be kept constant while varying $c_2$ because $c_1$ can also be varied to keep $\frac{c_1}{2 c_2}$ constant.  In the second case $c_2$ can be freely varied throughout a wide range.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, in many circumstances you should &lt;em&gt;expect&lt;/em&gt; not to be able to identify at least one of these parameters.  It sounds like your data are in one of these cases.  If so, you might consider eliminating one of the $c_i$ from the model altogether.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively, consider a reparameterization of the function in the form&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\alpha \sqrt{1 + \beta y x \exp(-y x)} + \gamma - \alpha$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;($\alpha = \sqrt{c_2}$, $\beta = c_1/(c_2 y)$, $\gamma = c_3+\sqrt{c_2}$).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This might provide enough clues to the minimizer to help it converge.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-02-11T21:06:12.897" LastActivityDate="2011-02-11T21:06:12.897" CommentCount="4" />
  <row Id="7105" PostTypeId="2" ParentId="3" CreationDate="2011-02-11T17:52:56.833" Score="3" ViewCount="" Body="&lt;p&gt;This falls on the outer limits of 'statistical analysis', but &lt;a href=&quot;http://ccsl.mae.cornell.edu/eureqa&quot; rel=&quot;nofollow&quot;&gt;Eureqa&lt;/a&gt; is a very user friendly program for data-mining nonlinear relationships in data via genetic programming. Eureqa is not as general purpose, but it does what it does fairly well, and the GUI is quite intuitive. It can also take advantage of the available computing power via the eureqa server.&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2011-02-11T17:52:56.833" CommunityOwnedDate="2011-02-11T17:52:56.833" />
  <row Id="7106" PostTypeId="2" ParentId="3" CreationDate="2011-02-11T18:44:38.463" Score="2" ViewCount="" Body="&lt;p&gt;People have already named the tools that I regularly use, so I'll have to venture a bit farther afield to something I recently discovered and have just begun to learn...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ADMB (AD Model Builder), which does non-linear modeling based on the AUTODIF library, with MCMC and a few other features thrown in. It preprocesses and compiles the model down to a C++ executable and compiles it as a standalone app, which is supposed to be way faster than equivalent models implemented in R, MATLAB, etc. &lt;a href=&quot;http://admb-project.org/&quot; rel=&quot;nofollow&quot;&gt;ADMB Project&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It started and is still most popular in the fisheries world, but looks quite interesting for other purposes. It does not have graphing or other features of R, and would most likely be used in conjunction with R.&lt;/p&gt;&#xA;" OwnerUserId="1764" LastActivityDate="2011-02-11T18:44:38.463" CommunityOwnedDate="2011-02-11T18:44:38.463" />
  <row Id="7107" PostTypeId="2" ParentId="7100" CreationDate="2011-02-11T18:46:41.513" Score="1" ViewCount="" Body="&lt;p&gt;I found some good ideas in the short Sage book, Translating Questionnaires and Other Research Instruments, at&#xA;&lt;a href=&quot;http://www.uk.sagepub.com/books/Book5861&quot; rel=&quot;nofollow&quot;&gt;http://www.uk.sagepub.com/books/Book5861&lt;/a&gt; .  I wouldn't call it the most systematic or the most entertaining read, but it was helpful and it's fairly up-to-date and inexpensive.&lt;/p&gt;&#xA;" OwnerUserId="2669" LastActivityDate="2011-02-11T18:46:41.513" />
  <row Id="7109" PostTypeId="2" ParentId="7101" CreationDate="2011-02-11T20:04:18.110" Score="2" ViewCount="" Body="&lt;p&gt;I would suggest looking at the &lt;a href=&quot;http://cran.r-project.org/web/packages/BayesTree/index.html&quot; rel=&quot;nofollow&quot;&gt;BayesTree&lt;/a&gt; package, from CRAN. I have no experience with it, so I cannot say if there are better option from there. Try looking at the &lt;a href=&quot;http://cran.r-project.org/web/views/MachineLearning.html&quot; rel=&quot;nofollow&quot;&gt;Machine Learning&lt;/a&gt; Task View, or directly through &lt;a href=&quot;http://www.rseek.org/&quot; rel=&quot;nofollow&quot;&gt;www.rseek.org&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't know anything approaching in Bioconductor, but if the above package suits your needs I guess you won't have any problem with gene expression data. I know the &lt;a href=&quot;http://www.bioconductor.org/help/bioc-views/release/bioc/html/CMA.html&quot; rel=&quot;nofollow&quot;&gt;CMA&lt;/a&gt; package offer a full-featured pipeline for supervised classification (see e.g., &lt;a href=&quot;http://www.ibe.med.uni-muenchen.de/organisation/mitarbeiter/020_professuren/boulesteix/pdf/cma.pdf&quot; rel=&quot;nofollow&quot;&gt;CMA - A comprehensive Bioconductor package for super- vised classification with high dimensional data&lt;/a&gt;, by Slawski et al.). Maybe you can plug BART method in addition to the available methods?&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-02-11T20:04:18.110" />
  <row Id="7110" PostTypeId="1" AcceptedAnswerId="7147" CreationDate="2011-02-11T22:51:58.117" Score="5" ViewCount="252" Body="&lt;p&gt;What is/are the difference(s) between a longitudinal design and a time series?&lt;/p&gt;&#xA;" OwnerUserId="2956" LastActivityDate="2011-02-13T11:51:25.673" Title="Difference between longitudinal design and time series" Tags="&lt;time-series&gt;&lt;longitudinal&gt;" AnswerCount="3" CommentCount="1" FavoriteCount="2" />
  <row Id="7111" PostTypeId="1" AcceptedAnswerId="7114" CreationDate="2011-02-11T22:56:39.283" Score="5" ViewCount="346" Body="&lt;p&gt;Here's my understanding so far: To perform PCA, you have to subtract the means of each column from the data, compute the correlation coefficient matrix and then find the eigenvectors and eigenvalues. Well, rather, this is what I did to implement it in Python, except it only works with small matrices because the method to find the correlation coefficient matrix (corrcoef) doesn't let me use an array with high dimensionality. Since I have to use it for images, my current implementation doesn't really help me.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've read that it's possible to just take your data and do a (data . data.T) / lines instead of doing a (data.T . data) / lines, but that doesn't work for me. Well, I'm not exactly sure I understand what it means, besides the fact that it's supposed to be a matrix n x n instead of p x p (p &gt;&gt; n). I read up about those in the eigenfaces tutorials but none of them seemed to explain it in such a way I could really get it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In short, is there a simple algorithmic description of this method so that I can follow it?&lt;/p&gt;&#xA;" OwnerUserId="3164" LastEditorUserId="3164" LastEditorDisplayName="" LastEditDate="2011-02-11T23:23:00.880" LastActivityDate="2011-03-09T01:13:10.640" Title="PCA for images/arrays with high dimensionality?" Tags="&lt;pca&gt;&lt;python&gt;" AnswerCount="4" FavoriteCount="3" />
  <row Id="7112" PostTypeId="1" AcceptedAnswerId="7118" CreationDate="2011-02-11T23:09:54.510" Score="8" ViewCount="228" Body="&lt;p&gt;Hi All,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have 2 simple questions about linear regression:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;When is it advised to standardize the explanatory variables?&lt;/li&gt;&#xA;&lt;li&gt;Once estimation is carried out with standardized values, how can one predict with new values (how one should standardize the new values)? &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Some references would be helpful. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regards&lt;/p&gt;&#xA;" OwnerUserId="1443" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-02-12T04:29:03.503" LastActivityDate="2011-02-14T07:45:27.760" Title="When and how to use standardized explanatory variables in linear regression" Tags="&lt;regression&gt;&lt;predictive-models&gt;&lt;predictor&gt;&lt;standarization&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="7113" PostTypeId="2" ParentId="7110" CreationDate="2011-02-12T00:20:17.573" Score="3" ViewCount="" Body="&lt;p&gt;A &lt;a href=&quot;http://en.wikipedia.org/wiki/Time_series&quot; rel=&quot;nofollow&quot;&gt;time series&lt;/a&gt; is simple a sequence of data points spaced out over time, usually with regular time intervals.  A &lt;a href=&quot;http://en.wikipedia.org/wiki/Longitudinal_study&quot; rel=&quot;nofollow&quot;&gt;longitudinal design&lt;/a&gt; is rather more specific, keeping the same sample for each observation over time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An example of a time series might be unemployment measured every month using a labour force survey with a new sample each time; this would be a sequence of cross-sectional designs.  But it could be anything such as your personal savings each year, which would also be longitudinal.  Or it might simply follow a particular cohort of people growing older, such as the television documentary &lt;a href=&quot;http://en.wikipedia.org/wiki/Seven_Up!&quot; rel=&quot;nofollow&quot;&gt;&lt;em&gt;Seven Up!&lt;/em&gt;&lt;/a&gt; and the sequels every seven years after that - the latest was &lt;em&gt;49 Up&lt;/em&gt; in 2005, so there should be another edition next year.  Longitudinal designs tend to tell you more about ways in which typical individuals change over time, but might (depending on the details of the design and whether the sample is refreshed) say less about how the population as a whole changes.   &lt;/p&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-02-12T00:20:17.573" CommentCount="1" />
  <row Id="7114" PostTypeId="2" ParentId="7111" CreationDate="2011-02-12T01:19:41.250" Score="5" ViewCount="" Body="&lt;p&gt;The easiest way to do standard PCA is to center the columns of your data matrix (assuming the columns correspond to different variables) by subtracting the column means, and then perform an SVD.  The left singular vectors, multiplied by the corresponding singular value, correspond to the (estimated) principal components.  The right singular vectors correspond to the (estimated) principal component directions — these are the same as the eigenvectors given by PCA.  The singular values correspond to the standard deviations of the principal components (multiplied by a factor of root n, where n is the number of rows in your data matrix) — the same as the square root of the eigenvalues given by PCA.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to do PCA on the correlation matrix, you will need to standardize the columns of your data matrix before applying the SVD.  This amounts to subtracting the means (centering) and then dividing by the standard deviations (scaling).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This will be the most efficient approach if you want the full PCA.  You can verify with some algebra that this gives you the same answer as doing the spectral decomposition of the sample covariance matrix.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: There are also efficient methods for computing a partial SVD, when you only need a few of the PCs.  Some of these are variants of the power iteration.  The [Lanczos algorithm][1] is one example that is also related to partial least squares.  If your matrix is huge, you may be better off with an approximate method.  There are also statistical reasons for regularizing PCA when this is the case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[1]: &lt;a href=&quot;http://en.wikipedia.org/wiki/Lanczos_algorithm&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Lanczos_algorithm&lt;/a&gt; “Lanczos algorithm&quot;&lt;/p&gt;&#xA;" OwnerUserId="1670" LastEditorUserId="1670" LastEditorDisplayName="" LastEditDate="2011-02-27T20:09:28.270" LastActivityDate="2011-02-27T20:09:28.270" />
  <row Id="7115" PostTypeId="1" AcceptedAnswerId="7188" CreationDate="2011-02-12T03:28:54.630" Score="3" ViewCount="86" Body="&lt;p&gt;I'm wondering how far along the natural language processing is in determining the semantic distance between two excerpts of text.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, consider the following phrases&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Early today, I got up and washed my car.&lt;/li&gt;&#xA;&lt;li&gt;I cleaned my truck up this morning.&lt;/li&gt;&#xA;&lt;li&gt;Bananas are an excellent source of potassium.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Clearly (to the human reader) the first two statements are much more similar to one another than they are to the third statement.  Does there exist any methodology that would allow a computer to draw the same conclusion?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Ultimately I'm trying to group Twitter tweets according to semantic content.)&lt;/p&gt;&#xA;" OwnerUserId="1026" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-02-12T12:46:47.723" LastActivityDate="2011-02-14T07:51:24.210" Title="Semantic distance between excerpts of text" Tags="&lt;clustering&gt;&lt;classification&gt;&lt;text-mining&gt;" AnswerCount="2" CommentCount="0" FavoriteCount="1" />
  <row Id="7116" PostTypeId="2" ParentId="7111" CreationDate="2011-02-12T03:43:19.633" Score="4" ViewCount="" Body="&lt;p&gt;It sounds like what you want is the NIPALS algorithm for performing PCA. It's a very popular algorithm among statisticians. It has many advantages:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Computationally less expensive than SVD or eigenvalue decomposition methods if only the first few components are required.&lt;/li&gt;&#xA;&lt;li&gt;Has more modest storage requirements in general because the covariance matrix is never formed. This is a very important property for very large datasets.&lt;/li&gt;&#xA;&lt;li&gt;Can handle missing data in the dataset (though that's not an issue in your problem, since you're dealing with images).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Description&lt;/strong&gt;&lt;br&gt;&#xA;&lt;a href=&quot;http://en.wikipedia.org/wiki/Non-linear_iterative_partial_least_squares&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Non-linear_iterative_partial_least_squares&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Algorithm&lt;/strong&gt;&lt;br&gt;&#xA;Here's a simple and excellent description of the algorithm (in section 1.2)&lt;br&gt;&#xA;&lt;a href=&quot;http://stats4.eng.mcmaster.ca/w/mediafiles/mediawiki/f/f7/Section-Extra-Class-1.pdf&quot; rel=&quot;nofollow&quot;&gt;http://stats4.eng.mcmaster.ca/w/mediafiles/mediawiki/f/f7/Section-Extra-Class-1.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Remember to mean-center-scale first before doing PCA as it is scale-sensitive.&lt;/p&gt;&#xA;" OwnerUserId="2833" LastActivityDate="2011-02-12T03:43:19.633" />
  <row Id="7117" PostTypeId="2" ParentId="7115" CreationDate="2011-02-12T03:50:03.520" Score="2" ViewCount="" Body="&lt;p&gt;Check out the work by &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.89.3355&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;Jones &amp;amp; Mewhort (2007)&lt;/a&gt;. This &lt;a href=&quot;http://www.springerlink.com/content/qr50708142958x51/&quot; rel=&quot;nofollow&quot;&gt;more recent work&lt;/a&gt; may also be of interest, particularly their &lt;a href=&quot;http://www.casstools.org&quot; rel=&quot;nofollow&quot;&gt;online tool&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="364" LastActivityDate="2011-02-12T03:50:03.520" />
  <row Id="7118" PostTypeId="2" ParentId="7112" CreationDate="2011-02-12T04:27:47.997" Score="11" ViewCount="" Body="&lt;p&gt;Although terminology is a contentious topic, I prefer to call &quot;explanatory&quot; variables, &quot;predictor&quot; variables.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;When to standardise the predictors:&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;A lot of software for performing multiple linear regression will provide standardised coefficients which are equivalent to unstandardised coefficients where you manually standardise predictors and the response variable (of course, it sounds like you are talking about only standardising predictors).&lt;/li&gt;&#xA;&lt;li&gt;My opinion is that standardisation is a useful tool for making regression equations more meaningful.&#xA;This is particularly true in cases where the metric of the variable lacks meaning to the person interpreting the regression equation (e.g., a psychological scale on an arbitrary metric).&#xA;It can also be used to facilitate comparability of the relative importance of predictor variables (although other more sophisticated approaches exist for assessing relative importance; &lt;a href=&quot;http://jeromyanglim.blogspot.com/2009/09/variable-importance-and-multiple.html&quot; rel=&quot;nofollow&quot;&gt;see my post for a discussion&lt;/a&gt;).&#xA;In cases where the metric does have meaning to the person interpreting the regression equation, unstandardised coefficients are often more informative.&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I also think that relying on standardised variables may take attention away from the fact that we have not thought about how to make the metric of a variable more meaningful to the reader.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Andrew Gelman has a fair bit to say on the topic.&#xA;See his &lt;a href=&quot;http://www.stat.columbia.edu/~gelman/standardize/&quot; rel=&quot;nofollow&quot;&gt;page on standardisation&lt;/a&gt; for example and &lt;a href=&quot;http://www.stat.columbia.edu/~gelman/research/published/standardizing7.pdf&quot; rel=&quot;nofollow&quot;&gt;Gelman (2008, Stats Med, FREE PDF)&lt;/a&gt; in particular.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;Prediction based on standarisation:&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;I would not use standardised regression coefficients for prediction.&lt;/li&gt;&#xA;&lt;li&gt;You can always convert standardised coefficients to unstandardised coefficients if you know the mean and standard deviation of the predictor variable in the original sample.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-02-12T11:13:03.750" LastActivityDate="2011-02-12T11:13:03.750" CommentCount="4" />
  <row Id="7119" PostTypeId="2" ParentId="7088" CreationDate="2011-02-12T05:00:37.070" Score="2" ViewCount="" Body="&lt;p&gt;I'll do my best to interpret your question.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Style Rules&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Many journals and disciplines adopt a style guide (e.g., APA, Chicago, MLA, etc.). &#xA;If one applies to you, then you are likely to have many relevant rules to guide you in the presentation of tables, figures, and much more.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;Table and Figure Design&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;There are many common tables and figures that I've seen for summarising information about responses of a sample to a questionnaire.&#xA;Much of it depends on the nature of the questionnaire items.&lt;/li&gt;&#xA;&lt;li&gt;If you have a bunch of likert items, it's common to present a table where for each item  percentages for each response option is given along with means.&lt;/li&gt;&#xA;&lt;li&gt;If you have categorical predictors (e.g., gender, race, income category, etc.) you might want to present a table comparing means for the groups along with tests of statistical significance.&lt;/li&gt;&#xA;&lt;li&gt;If you have scales (i.e., items are combined to form a composite), you may want to present a table of these means.&lt;/li&gt;&#xA;&lt;li&gt;If you have benchmarks or norms, you may want to incorporate these into the table to aid in interpretation.&lt;/li&gt;&#xA;&lt;li&gt;Much more could be said, but it depends a lot on your design, your research question, and other considerations.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;File Formats and Document Creation Tools&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Your question uses the term &quot;word&quot; document.&#xA;Arguably, this confuses two issues.&#xA;&lt;ul&gt;&#xA;&lt;li&gt;First, there is the issue of file formats. You could create your document using many file formats. Word is only one of a class of word processor file formats. I prefer to use LaTeX, Markdown, or HTML depending on my purpose and then, often, generate a PDF for sharing. Sweave (or some equivalent weaving format) in particular is worth considering if you want to make your report more reproducible.&lt;/li&gt;&#xA;&lt;li&gt;Second, there is the issue of the document creation tool. &#xA;MS Word is only one tool. I prefer to use a good text editor to write one of the above mentioned plain text formats. &lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;The main point is that regardless of your creation tool or file format, you should be able to produce a report that looks a certain way (although some tools and file formats facilitate given aims better than others; LaTeX in particular produces more attractive word spacing than MS Word, for example).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastActivityDate="2011-02-12T05:00:37.070" />
  <row Id="7120" PostTypeId="2" ParentId="7088" CreationDate="2011-02-12T05:00:38.817" Score="1" ViewCount="" Body="&lt;p&gt;Jeremy has offered many constructive suggestions.  I'll add the point that &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&quot;If there were one approach that was&#xA;  clearly superior, then the Law of the&#xA;  Statistical Jungle would dictate that&#xA;  it would survive, and all of the other&#xA;  techniques would exist only as&#xA;  historical footnotes. The continued&#xA;  survival of all of them, and the&#xA;  constant introduction of new ones,&#xA;  indicates that no clearly superior&#xA;  solution exists; hence, the&#xA;  unfortunate need to think about what&#xA;  we're doing.&quot; --- Geoffrey Norman and David Streiner&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="2669" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-02-12T05:18:18.587" LastActivityDate="2011-02-12T05:18:18.587" />
  <row Id="7121" PostTypeId="2" ParentId="7032" CreationDate="2011-02-12T05:12:06.707" Score="1" ViewCount="" Body="&lt;p&gt;An graphical example on outliers that requires no software and can be read in 2 minutes is &lt;a href=&quot;http://en.wikipedia.org/wiki/Anscombe&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Anscombe&lt;/a&gt;'s_quartet&lt;/p&gt;&#xA;" OwnerUserId="2669" LastActivityDate="2011-02-12T05:12:06.707" />
  <row Id="7122" PostTypeId="2" ParentId="6913" CreationDate="2011-02-12T05:31:00.017" Score="1" ViewCount="" Body="&lt;p&gt;Just a quick thought:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you assume that each rating is drawn from a latent continuous variable, then you could define the median of this underlying continuous variable of interest as your value of interest, rather than the mean of this underlying distribution. Where the distribution is symmetric, then the mean and the median would ultimately be estimating the same quantities. Where the distribution is skewed, the median would differ from the mean.&#xA;In this case, to my mind, the median would correspond more with what we think of as the typical value. This goes some way to understanding why median income and median house prices are typically reported rather than the mean.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, when you have a small number of discrete values, the median performs poorly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Perhaps, you could use some density estimation procedure and then take the median of that, or use some interpolated median.&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2011-02-12T05:31:00.017" />
  <row Id="7123" PostTypeId="2" ParentId="7110" CreationDate="2011-02-12T05:41:19.063" Score="3" ViewCount="" Body="&lt;p&gt;If we think of designs made up of $n$ cases measured on $k$ occasions, then the following loose definition seems to me to be descriptive of the distinction:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;longitudinal designs: high $n$, low $k$&lt;/li&gt;&#xA;&lt;li&gt;time series: low $n$, high $k$&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Of course, this raises the question of what is high and what is low.&#xA;Summarising my own rough sense of these fuzzy definitions, prototypical examples of:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;time series might have $n$ = 1, 2, or 5 and $k$ = 20, 50, 100, or 1000, and&lt;/li&gt;&#xA;&lt;li&gt;longitudinal designs might have $n$ = 10, 50, 100, 1000 and $k$ = 2, 3, 5, 10, 20&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update:&lt;/strong&gt;&#xA;Following up on &lt;strong&gt;Dr Who's&lt;/strong&gt; question about what is the purpose of the distinction, I don't have an authoritative answer, but here are a few thoughts:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;terminology evolves in disciplines concerned with particular substantive problems &lt;/li&gt;&#xA;&lt;li&gt;time series&#xA;&lt;ul&gt;&#xA;&lt;li&gt;often concerned with forecasting future time points&lt;/li&gt;&#xA;&lt;li&gt;often concerned with modelling various cyclical and trend processes&lt;/li&gt;&#xA;&lt;li&gt;often concerned with describing temporal dynamics in great detail&lt;/li&gt;&#xA;&lt;li&gt;often studies phenomena where the particular thing measured is of specific interest (e.g., unemployment rate, stock market indices, etc.)&lt;/li&gt;&#xA;&lt;li&gt;temporal indices are often pre-existing&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;longitudinal designs:&#xA;&lt;ul&gt;&#xA;&lt;li&gt;often use samples of cases as exemplars of a population in order to make inferences about the population (e.g., sample of children to study how children change in general)&lt;/li&gt;&#xA;&lt;li&gt;often concerned with fairly general temporal processes like growth, variability and relatively simple functional change models&lt;/li&gt;&#xA;&lt;li&gt;study is often specifically designed to have a given number of time points.&lt;/li&gt;&#xA;&lt;li&gt;often interested in variation in change processes&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Given the differences in the actual temporal dynamics, and the particular combination of $k$ and $n$ this creates different statistical modelling challenges.&#xA;For example, with high $n$ and low $k$ multilevel models are often used that borrow strength from the typical change process to describe the individual change process.&#xA;These different disciplines, modelling challenges, and literatures encourage the creation of distinct terminology.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyway, that's my impression. Perhaps others have greater insight.&lt;/p&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-02-13T11:51:25.673" LastActivityDate="2011-02-13T11:51:25.673" CommentCount="2" />
  <row Id="7124" PostTypeId="2" ParentId="7100" CreationDate="2011-02-12T08:45:04.460" Score="0" ViewCount="" Body="&lt;p&gt;The procedure i have normally seen followed is to translate the questionnaire from english, and then have it back translated by someone else. If the two english translations match up, then you are good to go, otherwise repeat until they do. &lt;/p&gt;&#xA;" OwnerUserId="656" LastActivityDate="2011-02-12T08:45:04.460" />
  <row Id="7125" PostTypeId="2" ParentId="7100" CreationDate="2011-02-12T09:00:13.563" Score="2" ViewCount="" Body="&lt;p&gt;I don't know what your questionnaire aims to assess. In Health-related Quality-of-Life studies, for example, there are a certain number of recommendations for translation issues that were discussed in the following papers (among others):&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Marquis et al., Translating and evaluating questionnaires: Cultural issues for international research, in Fayers &amp;amp; Hays (eds.), &lt;em&gt;Assessing Quality of Life in Clinical Trials (2nd ed.)&lt;/em&gt;, Oxford, pp. 77-93 (2005).&lt;/li&gt;&#xA;&lt;li&gt;Hui and Triandis, Measurement in cross-cultural psychology: A review and comparison of strategies. &lt;em&gt;Journal of Cross-Cultural Psychology&lt;/em&gt;, 16(2), 131-152 (1985).&lt;/li&gt;&#xA;&lt;li&gt;Mathias et al., Rapid translation of quality of life measures for international clinical trials: Avoiding errors in the minimalist approach. &lt;em&gt;Quality of Life Research&lt;/em&gt;, 3, 403-412 (1994).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Translation can be done simultaneously in several languages, as was the case for the WHOQOL questionnaire, or in a primary language (e.g., English) for the SF-36 followed by translation in other target languages. There were many papers related to translation issues in either case that you will probably find on &lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pubmed/&quot; rel=&quot;nofollow&quot;&gt;Pubmed&lt;/a&gt;. Various procedures have been developed to ensure consistent translation, but forward/backward translation is most commonly found in the above studies.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In any case, most common issues when translating items (as single entities, and as a whole, that is at the level of the questionnaire) have to do with the equivalence of the hypothetical concept(s) that is/are supposed to be covered. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Otherwise, the very first things I would look at would be:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Am I measuring the same concepts? -- this is a question related to &lt;em&gt;validity&lt;/em&gt;;&lt;/li&gt;&#xA;&lt;li&gt;Are the scores delivered in the foreign language reliable enough? -- this has to do with &lt;em&gt;scores reliability&lt;/em&gt;;&lt;/li&gt;&#xA;&lt;li&gt;Are the items behaving as expected for everybody, i.e. without differential effects depending on country or native language? -- this is merely related to &lt;em&gt;differential item functioning&lt;/em&gt; (DIF), which is said to occur when the probability of endorsing a particular item differs according to a subject-specific covariate (e.g., age, gender, country), when holding subject trait constant.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Some of the common techniques used to assess those properties were discussed in an earlier related question, &lt;a href=&quot;http://stats.stackexchange.com/questions/4239/validating-questionnaires&quot;&gt;Validating questionnaires&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;About DIF specifically, here are some examples of subtle variation across subject-specific characteristics: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;In psychiatric studies, &quot;I feel sad&quot; / &quot;Able to enjoy life&quot; have been shown to highlight gender-related DIF (Crane et al., 2007).&lt;/li&gt;&#xA;&lt;li&gt;In personality assessment, there are well-known age and gender-effect on the NEO-PI questionnaire (reviewed in Kulas et al., 2008).&lt;/li&gt;&#xA;&lt;li&gt;In Health-related Quality-of-Life, items like &quot;Did you worry?&quot; / &quot;Did you feel depressed?&quot; have been shown to exhibit country-related DIF effect (Petersen&#xA;et al. 2003).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;A good starting point is this review paper by Jeanne Teresi in 2004: &lt;a href=&quot;http://outcomes.cancer.gov/conference/irt/teresi.pdf&quot; rel=&quot;nofollow&quot;&gt;Differential Item Functioning and Health Assessment&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-02-12T09:00:13.563" />
  <row Id="7126" PostTypeId="2" ParentId="7103" CreationDate="2011-02-12T11:19:00.807" Score="1" ViewCount="" Body="&lt;p&gt;Your distribution is parametric, and you should just store the parameters that are sufficient statistics, if you can identify them. That includes the distribution family. For a time series, you can take advantage of autocorrelation and store the parameters of the predictive distribution conditional on its previous values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The  entropy of the prior (predictive) distribution of the parameters determines the upper bound for compression strength, but you may not need to compress them further. If you do, use arithmetic compression. Decreasing entropy, say by discretizing quantiles, will give greater compression.&lt;/p&gt;&#xA;" OwnerUserId="2456" LastEditorUserId="2456" LastEditorDisplayName="" LastEditDate="2011-02-12T11:26:39.763" LastActivityDate="2011-02-12T11:26:39.763" CommentCount="6" />
  <row Id="7127" PostTypeId="2" ParentId="7111" CreationDate="2011-02-12T12:46:44.533" Score="2" ViewCount="" Body="&lt;p&gt;To add on Gilead's answer, they are computationally less expensive algorithms for truncated PCAs. NIPALS is indeed very popular, but I have had a lot of success with approximate methods that perform a succession of fits on partial data (what is often called PCA by random projection). This was discussed in a &lt;a href=&quot;http://metaoptimize.com/qa/questions/3258/randomized-methods-for-computing-svd#3506&quot; rel=&quot;nofollow&quot;&gt;metaoptimize&lt;/a&gt; thread.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you mention Python, let me point out that the algorithm is implemented in the &lt;a href=&quot;http://scikit-learn.sourceforge.net/&quot; rel=&quot;nofollow&quot;&gt;scikit-learn&lt;/a&gt;: the &lt;a href=&quot;http://scikit-learn.sf.net/dev/modules/decompositions.html#approximate-pca&quot; rel=&quot;nofollow&quot;&gt;ApproximatePCA&lt;/a&gt; class. In particular, it is used in an example demoing &lt;a href=&quot;http://scikit-learn.sf.net/dev/auto_examples/applications/plot_face_recognition.html&quot; rel=&quot;nofollow&quot;&gt;eigenfaces&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1265" LastEditorUserId="1265" LastEditorDisplayName="" LastEditDate="2011-02-27T17:26:41.607" LastActivityDate="2011-02-27T17:26:41.607" CommunityOwnedDate="2011-02-12T12:47:14.063" />
  <row Id="7128" PostTypeId="1" CreationDate="2011-02-12T15:22:27.697" Score="4" ViewCount="123" Body="&lt;h3&gt;Study:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;I simulated some surfaces materials at audio and haptic level, and I asked subjects to evaluate on a 9-point Likert scale the degree of coherence between the two stimuli.&#xA;For example there are stimuli with metal at auditory level and snow at haptic level, or wood both at auditory and haptic level.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So it is like a dissimilarity rating experiment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The experiment had only 12 participants, so for each stimulus I have 12 response (no repeated measures involved).&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Question:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;How should I analyse the following experiment?&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Initial Thoughts:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;So far the only way I am thinking to use is ANOVA.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Update&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Hello again,&#xA;I used the cmdscale command in R, but I did not get thecperceptual map I would love to see.&#xA;I as you an help! Maybe the problem is that I misunderstood if my table is well built for the purpose of the analsys.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let´s summarize my experiment, my goals and the problem:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I simulated some surfaces materials at audio and haptic level, and I asked &#xA;subjects to evaluate on a 9-point Likert scale the degree of coherence between &#xA;the two stimuli. For example there are trials where at auditory level a metal&#xA;surface was simulated and at haptic level the snow surface, or wood both at &#xA;auditory and haptic level.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The experiment had only 12 participants, so for each trial I have 12 response &#xA;(no repeated measures involved).&#xA;In total there were 36 trials, and each trial was evaluated only once by each&#xA;of the 12 participants. So each subject provided 36 ratings on the 9 point Likert &#xA;scale, one for each trial.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically, these are my data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; WD   MT   SW   GR   SN   DL&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;WD 7.00 6.50 4.91 4.83 5.50 5.00&lt;/p&gt;&#xA;&#xA;&lt;p&gt;MT 7.33 6.91 2.08 3.16 4.25 3.25&lt;/p&gt;&#xA;&#xA;&lt;p&gt;SW 2.91 1.75 7.91 6.25 6.83 5.41&lt;/p&gt;&#xA;&#xA;&lt;p&gt;GR 2.91 2.66 6.25 6.41 7.25 6.75&lt;/p&gt;&#xA;&#xA;&lt;p&gt;SN 4.00 4.00 5.58 6.00 7.00 6.58&lt;/p&gt;&#xA;&#xA;&lt;p&gt;DL 3.91 3.08 5.16 6.25 6.50 6.83&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the rows the haptic stimuli and on the columns the auditory stimuli. In each&#xA;cell there is the average score for each trial (e.g. the trial GR-MT is 2.66, that &#xA;is the average score given by participants to the trial where the material &quot;gravel&quot;&#xA;was provided at haptic level and the material &quot;metal&quot; was provided ar auditory level).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I want to analyze the data in the correct ways, and as said MDS is the best &#xA;analysis instead of ANOVA as I was thinking.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My first goal is to print a perceptual map where to place the pairs of &#xA;audio-haptic stimuli (e.g. WD-WD, MT-DL, etc.) and see how far are the trials &#xA;from each other.&#xA;I used cmdscale in R but I did not get the wanted result. Any suggestion?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My second goal would be to find some p-values like I normally get with ANOVA.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example I would like to understand if having the coherent couple of stimuli&#xA;SW-SW (which means &quot;snow&quot; both at audio and haptic level) produces significant&#xA;differences n the evaluations rather than the couple SW-MT (which means &quot;snow&quot; &#xA;at audio and &quot;metal&quot; at haptic level) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Again I would like to undestand if there is any statistical difference between&#xA;all the couples of stimuli corresponding to solid surfaces (like the couples MT-MT, &#xA;MT-WD, WD-WD, MT-MT) and all the couples where a solid surface and a aggregate surface&#xA;are presented (like the couples MT-SN, or WD-GR, etc.).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;...I want to get as many information as possible from that table.&#xA;I really thanks anyone who can provide any suggestion or useful information.&lt;/p&gt;&#xA;" OwnerUserId="3180" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-02-15T01:10:26.023" LastActivityDate="2011-02-16T01:26:19.933" Title="Dissimilarity rating experiment: how to analyze the results" Tags="&lt;rating&gt;" AnswerCount="3" CommentCount="3" />
  <row Id="7129" PostTypeId="1" AcceptedAnswerId="7130" CreationDate="2011-02-12T15:39:25.123" Score="4" ViewCount="65" Body="&lt;p&gt;When presenting data using a percentage, is it a good thing to have decimal places, say 2 decimal places instead of round off to whole numbers.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, instead of 23.43%, you round off to 23%   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am looking at this from the perspective of whether the 2 decimal places accuracy will make much difference since we are dealing with percentage and not raw data value.&lt;/p&gt;&#xA;" OwnerUserId="546" LastActivityDate="2011-02-12T16:12:18.203" Title="Should percentages have decimal places?" Tags="&lt;aggregation&gt;" AnswerCount="1" />
  <row Id="7130" PostTypeId="2" ParentId="7129" CreationDate="2011-02-12T16:12:18.203" Score="5" ViewCount="" Body="&lt;p&gt;It depends on the size of the differences between classes. In most applications, saying the 73% prefer option A and 27% prefer option B is perfectly acceptable. But if you're dealing in an election where candidate X has 50.15% of votes and candidate Y has 49.86%, the decimal places are very much necessary.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, you need to take care to make sure that all classes add up to 100%. In my electoral example above, they add up to 100.01%. In that case you might even consider adding a third decimal place.&lt;/p&gt;&#xA;" OwnerUserId="666" LastActivityDate="2011-02-12T16:12:18.203" />
  <row Id="7131" PostTypeId="2" ParentId="7054" CreationDate="2011-02-12T16:39:30.410" Score="1" ViewCount="" Body="&lt;p&gt;In light of the clarifications in the comments to the question, I'd suggest using logistic regression with indicator variables for each region and a linear trend over year, and also consider adding an interaction between year and region to allow the linear trend to differ between regions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want something simpler, you could apply Pearson's chi-squared test to the data for each year separately. It looks like there is very strong evidence that region 1 has higher death rates than either of the other two regions in all years except 2008, when region 3 had too few cases to be able to estimate the proportion of deaths with any precision.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Statistics can say nothing about the &lt;em&gt;reason&lt;/em&gt; that region 1 differs from the others, of course.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-02-12T16:39:30.410" CommentCount="1" />
  <row Id="7132" PostTypeId="1" AcceptedAnswerId="7135" CreationDate="2011-02-12T16:46:03.520" Score="3" ViewCount="184" Body="&lt;p&gt;When conducting a t-test we can always use a Welch approximation of the df, and let go of the assumption of equal variances.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there something similar for (even one way) anova?  (any R implementation of that?)&lt;/p&gt;&#xA;" OwnerUserId="253" LastActivityDate="2011-02-13T10:51:01.757" Title="Is there an anova procedure that doesn't assume equal variance?" Tags="&lt;r&gt;&lt;anova&gt;&lt;variance&gt;&lt;degrees-of-freedom&gt;" AnswerCount="2" />
  <row Id="7133" PostTypeId="2" ParentId="7128" CreationDate="2011-02-12T16:46:10.710" Score="1" ViewCount="" Body="&lt;p&gt;ANOVA sounds reasonable to me &amp;mdash; two-way ANOVA, to be specific, treating subjects as the blocking factor.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-02-12T16:46:10.710" CommentCount="3" />
  <row Id="7134" PostTypeId="1" CreationDate="2011-02-12T17:12:53.863" Score="5" ViewCount="112" Body="&lt;p&gt;Sometimes in reports I include a disclaimer about the p-values and other inferential statistics I've provided.  I say that since the sample wasn't random, then such statistics would not strictly apply.  My specific wording is usually given in a footnote:  &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&quot;While, strictly speaking, inferential&#xA;  statistics are only applicable in the&#xA;  context of random sampling, we follow&#xA;  convention in reporting significance&#xA;  levels and/or confidence intervals as&#xA;  convenient yardsticks even for&#xA;  nonrandom samples. See Michael Oakes's&#xA;  &lt;em&gt;Statistical inference: A commentary for the social and behavioural&#xA;  sciences&lt;/em&gt;  (NY: Wiley, 1986).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;On a couple of occasions--once for a peer-reviewed paper, once or twice in a non-academic setting--the editor or reviewer objected to this disclaimer, calling it confusing, and felt that the inferential findings should simply stand as written (and be given the mantle of authority).  Has anyone else encountered this problem and found a good solution?  On the one hand, people's understanding of p-values is generally dismal, even in the context of random sampling, so perhaps it doesn't matter much what we say.  On the other, to contribute further to misunderstandings seems to make one part of the problem. I should add that I frequently deal with survey studies, where random assignment does not apply and where Monte Carlo simulations would often fail to address the issue of representativeness.&lt;/p&gt;&#xA;" OwnerUserId="2669" LastEditorUserId="2669" LastEditorDisplayName="" LastEditDate="2011-02-12T23:25:01.687" LastActivityDate="2011-02-12T23:25:01.687" Title="Accomodating entrenched views of p-values" Tags="&lt;inference&gt;&lt;p-values&gt;" AnswerCount="1" CommentCount="3" FavoriteCount="2" />
  <row Id="7135" PostTypeId="2" ParentId="7132" CreationDate="2011-02-12T17:26:21.177" Score="2" ViewCount="" Body="&lt;p&gt;The latest version of &lt;a href=&quot;http://cran.r-project.org/web/packages/ez/index.html&quot; rel=&quot;nofollow&quot;&gt;ez&lt;/a&gt; lets you pass a &lt;code&gt;white.adjust&lt;/code&gt; argument to &lt;code&gt;car::Anova()&lt;/code&gt;, which implements a correction for heteroscedasticity. See &lt;code&gt;?car::Anova()&lt;/code&gt; for details.&lt;/p&gt;&#xA;" OwnerUserId="364" LastActivityDate="2011-02-12T17:26:21.177" />
  <row Id="7136" PostTypeId="2" ParentId="7134" CreationDate="2011-02-12T17:29:41.060" Score="4" ViewCount="" Body="&lt;p&gt;The use of inferential statistics can be justified not only based on a population model, but also based on a randomization model. The latter does not make any assumptions about the way the sample has been obtained. In fact, Fisher was the one that suggested that the randomization model should be the basis for statistical inference (as opposed to Neyman and Pearson). See, for example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ernst, M. D. (2004). Permutation methods: A basis for exact inference. Statistical Science, 19, 676-685. &lt;a href=&quot;http://projecteuclid.org/euclid.ss/1113832732&quot; rel=&quot;nofollow&quot;&gt;[link (open access)]&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ludbrook, J. &amp;amp; Dudley, H. (1998). Why permutation tests are superior to t and F tests in biomedical research. American Statistician, 52, 127-132. &lt;a href=&quot;http://www.jstor.org/stable/2685470&quot; rel=&quot;nofollow&quot; title=&quot;link&quot;&gt;[link (if you have JSTOR access)]&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I somehow doubt though that the editors or reviewers in question were using this as the reason for calling your disclaimer &quot;confusing&quot;.&lt;/p&gt;&#xA;" OwnerUserId="1934" LastActivityDate="2011-02-12T17:29:41.060" CommentCount="2" />
  <row Id="7138" PostTypeId="2" ParentId="7132" CreationDate="2011-02-12T17:54:14.780" Score="5" ViewCount="" Body="&lt;p&gt;There is a function named &lt;code&gt;oneway.test()&lt;/code&gt; in the base &lt;code&gt;stats&lt;/code&gt; package, which implements Welch correction for a one-way ANOVA. Its use is similar to the standard &lt;code&gt;t.test()&lt;/code&gt; function. It is also referred to as O'Brien transformation (Biometrics 40 (1984), 1079--1087) and might be applied with two or more independent samples.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-02-13T10:51:01.757" LastActivityDate="2011-02-13T10:51:01.757" />
  <row Id="7139" PostTypeId="1" CreationDate="2011-02-12T18:47:04.910" Score="4" ViewCount="131" Body="&lt;p&gt;I came across this study as part of a mock exam paper and was confused to say the least.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Context:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;The study investigates cognitive and behavioural factor related to the experience of anxiety in MRI scanners.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Participants completed the following questionnaires 5 mins after the scan:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;a measure of the frequency of anxiety experienced in the scanner&lt;/li&gt;&#xA;&lt;li&gt;a measure of the frequency of physical symptoms of panic&lt;/li&gt;&#xA;&lt;li&gt;a measure of the frequency of coping strategy employment&lt;/li&gt;&#xA;&lt;li&gt;a measure of the frequency of claustrophobic-related thoughts&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Participants were also asked to fill in and submit general measures at home after the scan:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;level of claustrophobic fears regarding restriction and suffocation&lt;/li&gt;&#xA;&lt;li&gt;depression&lt;/li&gt;&#xA;&lt;li&gt;anxiety&lt;/li&gt;&#xA;&lt;li&gt;health anxiety&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The return rate was 75%, i.e. only 97 of the original 130 participants completed the second batch of questionnaires.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Overall, the level of anxiety in the scanner was RELATED TO:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;level of claustrophobic fears regarding restriction and suffocation&lt;/li&gt;&#xA;&lt;li&gt;frequency of coping strategy to manage scanner anxiety&lt;/li&gt;&#xA;&lt;li&gt;no. symptoms of panic during the scan&lt;/li&gt;&#xA;&lt;li&gt;no. claustrophobia-related thoughts about the scan.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;No stats are given.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Questions:&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What type of design is this?&lt;/li&gt;&#xA;&lt;li&gt;Is it possible to understand the causes of anxiety experienced in MRI using this type of design? &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;Initial Thoughts:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;In my mind, causality can not be inferred because there is no comparison with a control group, and correlation would probably have been used rather than AVOVA. Is this the answer to the questions above or have I missed the point?&lt;/p&gt;&#xA;" OwnerUserId="3171" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-02-13T10:46:39.673" LastActivityDate="2011-02-14T08:32:27.437" Title="Can causality be inferred in a study with an experience followed by two sets of measures" Tags="&lt;causal-inference&gt;" AnswerCount="3" />
  <row Id="7140" PostTypeId="2" ParentId="7139" CreationDate="2011-02-12T23:40:45.743" Score="0" ViewCount="" Body="&lt;p&gt;I think you're on the right track.  Before drawing any conclusions about causality, I'd want to know...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;...to what degree the 97 were representative of the 130, and the 130, to the population of interest.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;...the magnitude of any relationships found as well as their statistical sig.  (Whether correlation or anova was used is not, I think, directly relevant to your question.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;...the psychometric properties (validity and reliability indicators) of all measures used (perhaps one would want to adjust for reliability attenuation, for instance).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;...what the subjects' characteristics were at baseline, i.e., before undergoing the scanner experience.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;...as you've said, how a control group would compare.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The bottom line is, this is a correlational/observational study and, certainly as reported, says little about causal relationships.&lt;/p&gt;&#xA;" OwnerUserId="2669" LastActivityDate="2011-02-12T23:40:45.743" CommentCount="1" />
  <row Id="7141" PostTypeId="1" CreationDate="2011-02-13T03:57:04.277" Score="6" ViewCount="116" Body="&lt;p&gt;Suppose I draw $n$ observations $X_1,X_2,\ldots,X_n$ independently from a distribution where  $X_i \sim \mathcal{N}(\mu_i,\sigma^2)$, where the mean is assumed to be Lipschitz: $\left| \mu_i - \mu_{i+1}\right| \le \gamma,$ with $\gamma$ known. I want to test the null hypothesis:&#xA;$$H_0: \mu_n = 0$$&#xA;against the local alternative $H_1: \mu_n &amp;gt; 0$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A few questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Is this a known problem? If so, for what terms should I google search?&lt;/li&gt;&#xA;&lt;li&gt;What is a reasonable way to perform this test? I can imagine there might be some way to do it by downweighting less recent observations--maybe throw the innovations into the noise term, then use a weighted t-test? The problem with that approach is that the change in mean may be deterministic, so I should pose the problem as a minimax problem maybe. Perhaps another approach would be to ignore the non-stationarity and use a regular t-test on some subset of the observations $X_j,X_{j+1},\ldots,X_n$, with $n-j$ chosen by some function of $\gamma, \sigma$ (probably their ratio?), and accept that the test will not maintain the nominal Type I rate. What are some better ideas?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit&lt;/strong&gt;: per the excellent comments so far, I believe this question can be reformulated as follows: Let $V$ be the convex set in $\mathbb{R}^n$ defined by:&#xA;$$&#xD;&#xA;V = \left\{\mathbf{x} | x_n = 0, x_{i-1} \le x_i + \gamma, x_{i-1} \ge x_i - \gamma, i = 2,\ldots,n  \right\}&#xD;&#xA;$$&#xA;Observing vector $X = f + \sigma\epsilon$, where the elements of $\epsilon$ are drawn independently from a standard Gaussian, test the null hypothesis:&#xA;$$H_0: f \in V$$&#xA;In this sense, it is similar to the paper of Baraud &lt;em&gt;et. al.&lt;/em&gt; mentioned by @Robin Girard, except the Baraud paper provides a method for the case where $V$ is a linear subspace of $\mathbb{R}^n$, &lt;em&gt;not&lt;/em&gt; a convex polytope.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Under this formulation, it seems that, under the null, the squared distance from $X$ to $f$ should be distributed as a Chi-squared, up to the scaling factor of $\sigma^2$. Then the distance from $X$ to $V$ should be no greater than that. The proposed test would then be to find the projection of $X$ onto $V$, and test the distance from $X$ to that projection based on the Chi-squared bound. I imagine this test will have fairly low power. Perhaps truncating the observation to a final subset of the $X_i$ would have higher power...&lt;/p&gt;&#xA;" OwnerUserId="795" LastEditorUserId="795" LastEditorDisplayName="" LastEditDate="2011-02-15T03:25:37.087" LastActivityDate="2011-05-14T07:46:07.330" Title="Location test under a bounded non-stationarity?" Tags="&lt;time-series&gt;&lt;hypothesis-testing&gt;&lt;stationarity&gt;&lt;location&gt;&lt;lipschitz&gt;" AnswerCount="1" CommentCount="18" FavoriteCount="2" />
  <row Id="7142" PostTypeId="2" ParentId="7139" CreationDate="2011-02-13T05:34:38.733" Score="1" ViewCount="" Body="&lt;h3&gt;Design:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;The study is an observational design.&#xA;Some may call it a correlational design, but I'm not a fan of such terminology because it can encourage the false assumption that the ability to draw a causal inference is related to the statistical test used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Following up on this point, ANOVA does not permit causal inference any more than correlation.&#xA;The issue is whether a variable was experimentally manipulated.&#xA;It's just that most of the time variables manipulated are categorical and not numeric, so we end up using ANOVAs to analyse their effect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a general rule validating a causal proposition is easier where you experimentally manipulate the variable of interest.&#xA;However, I would argue that correlations drawn from an observational design are still informative, particularly if they are combined with solid theoretical knowledge about the phenomena.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Causal Inference:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Many of the things related to anxiety experienced in the scanner are themselves related to the concept of anxiety. For example, panic, claustrophobia related thoughts and so on. Thus, showing that they are correlated, to my mind does not suggest causality. Rather, I would say that they are syndrome of co-occurring mental states that presumably have some common cause.&#xA;I suppose this would be an example of me using prior knowledge of the phenomena and combining this with the correlational evidence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The study is also limited to examining what causes individual differences in MRI anxiety.&#xA;It does not examine how MRI implementation could be altered to make it more or less anxiety evoking.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I suppose you could use the reported levels of anxiety in the scanner as evidence that the scanner causes anxiety.&#xA;Presumably we have an implicit control group in our minds that says most of the time people are not anxious about being in a scanner (particularly if they are not actually in a scanner).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't think the return rate is a major issue, although it might be worth considering whether this is biasing your results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some of the other variables are presumably getting at the question of whether anxiety experienced in a scanner is related (or even caused) by a general disposition of the person.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The fact that the general measures of anxiety, depression, and so on are measured after the MRI study is potentially a confound, although a researcher would probably argue that such measures are relatively stable over time.&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2011-02-13T05:34:38.733" CommentCount="1" />
  <row Id="7144" PostTypeId="2" ParentId="7111" CreationDate="2011-02-13T07:21:25.333" Score="0" ViewCount="" Body="&lt;p&gt;What you're doing right now is close, but you need to make sure you multiply the eigenvectors of (data . data.T) / lines by data.T, in order to get the eigenvectors of (data.T . data) / lines.&lt;/p&gt;&#xA;" OwnerUserId="1106" LastEditorUserId="1106" LastEditorDisplayName="" LastEditDate="2011-03-09T01:13:10.640" LastActivityDate="2011-03-09T01:13:10.640" CommentCount="1" />
  <row Id="7146" PostTypeId="1" AcceptedAnswerId="7148" CreationDate="2011-02-13T07:56:23.410" Score="11" ViewCount="297" Body="&lt;p&gt;Hi i am analaysing wind data for estimating energy from a wind turbine.&lt;br&gt;&#xA;I have taken 10 years of wind data and graphed a histogram;&lt;br&gt;&#xA;my second stage was to fit a Weibull distribution to the data.&lt;br&gt;&#xA;I used R with the package &lt;code&gt;lmom&lt;/code&gt; to compute the Weibul shape and scale &#xA;this is the code i used:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;library(lmom)    &#xA;wind.moments&amp;lt;-samlmu(as.numeric(pp$WS))      &#xA;moments&amp;lt;-pelwei(wind.moments)     &#xA;x.wei&amp;lt;-rweibull(n=length(pp$WS), shape=moments[&quot;delta&quot;], scale=moments[&quot;beta&quot;])    &#xA;hist(as.numeric(pp$WS), freq=FALSE)    &#xA;lines(density(x.wei), col=&quot;red&quot;, lwd=4)    &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It seems like there is some lag between the data and the density function; can you help me with this? &#xA;Another question is can you help me in calculating the anual energy from the density function?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/AqVij.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;br&gt;&#xA;thank you &lt;/p&gt;&#xA;" OwnerUserId="3178" LastEditorUserId="3178" LastEditorDisplayName="" LastEditDate="2011-02-13T08:47:00.057" LastActivityDate="2011-04-04T15:12:40.560" Title="Analysing wind data with R" Tags="&lt;r&gt;&lt;distributions&gt;" AnswerCount="4" CommentCount="10" FavoriteCount="3" />
  <row Id="7147" PostTypeId="2" ParentId="7110" CreationDate="2011-02-13T08:38:52.733" Score="2" ViewCount="" Body="&lt;p&gt;I will add that in time series context it is usually assumed that data observed is a realisation of stochastic process. Hence in time series a lot of attention is given to properties of stochastic processes, such as stationarity, ergodicity, etc. In longitudinal context in my understanding  data comes from usual samples (by sample I mean sequence of iid variables) observed at different points in time, so  classical statistic methods are applied, since they always assume that sample is observed. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For short answer, one might say that time series are studied in econometrics, longitudinal design -- in statistics. But that does not answer the question, just shifts it to another question. On the other hand a lot of short answers do exactly that.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-02-13T08:38:52.733" CommentCount="2" />
  <row Id="7148" PostTypeId="2" ParentId="7146" CreationDate="2011-02-13T09:51:27.640" Score="5" ViewCount="" Body="&lt;p&gt;&lt;code&gt;lmom&lt;/code&gt; function &lt;code&gt;pelwei&lt;/code&gt; fits a &lt;em&gt;three&lt;/em&gt; parameter Weibull distribution, with &lt;em&gt;location&lt;/em&gt;, scale and shape parameters. &lt;code&gt;rweibull&lt;/code&gt; generates random numbers for a &lt;em&gt;two&lt;/em&gt;-parameter Weibull distribution. You need to subtract the location parameter &lt;code&gt;moments[&quot;zeta&quot;]&lt;/code&gt;. That should give a &lt;em&gt;better&lt;/em&gt; fit, but it doesn't appear it will give a &lt;em&gt;good&lt;/em&gt; fit to your particular data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I notice &lt;a href=&quot;http://www.reuk.co.uk/Wind-Speed-Distribution-Weibull.htm&quot; rel=&quot;nofollow&quot;&gt;http://www.reuk.co.uk/Wind-Speed-Distribution-Weibull.htm&lt;/a&gt; says &quot;Wind speeds in most of the world can be modelled using the Weibull Distribution.&quot;. Perhaps you're just unlucky and live in a part of the world where they can't!&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-02-13T09:51:27.640" CommentCount="5" />
  <row Id="7149" PostTypeId="2" ParentId="6770" CreationDate="2011-02-13T10:28:48.027" Score="1" ViewCount="" Body="&lt;p&gt;You can simply use the geometric distribution, which gives you the probability of a given number of Bernouilli trials before a success or in these case before the series is broken. The distribution is given as follow :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P = (1-p)^(k) * p&lt;/p&gt;&#xA;&#xA;&lt;p&gt;with p = .5 for a fair coin and k = 1 : 9&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The idea is to compute the joint probability of k consecutive head/tail and one tail/head : &lt;/p&gt;&#xA;&#xA;&lt;p&gt;P{tail, tail,..., head} = P{tail}^k * P{head}.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then you multiply this probability by 10^7 the number of trials in your simulation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The results are the following :&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;2,500,000&lt;/li&gt;&#xA;&lt;li&gt;1,250,000&lt;/li&gt;&#xA;&lt;li&gt;625,000&lt;/li&gt;&#xA;&lt;li&gt;312,500&lt;/li&gt;&#xA;&lt;li&gt;156,250&lt;/li&gt;&#xA;&lt;li&gt;etc. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;which are quite close of your simulated results...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the case of a run of length 10 or more, you have to do 1-(others prob, including k=0) which gives 9765.625.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can compute these probabilities easily in R with P &amp;lt;- dgeom(1:9, prob=.5) and for P(10) : P &amp;lt;- 1- sum(dgeom(0:9, prob=.5))&lt;/p&gt;&#xA;" OwnerUserId="3108" LastEditorUserId="3108" LastEditorDisplayName="" LastEditDate="2011-02-13T11:02:14.957" LastActivityDate="2011-02-13T11:02:14.957" />
  <row Id="7150" PostTypeId="2" ParentId="7146" CreationDate="2011-02-13T11:21:23.203" Score="5" ViewCount="" Body="&lt;p&gt;I recreated your plot with data from &lt;a href=&quot;http://hawaii.gov/dbedt/ert/winddata/krab0192.txt&quot; rel=&quot;nofollow&quot;&gt;http://hawaii.gov/dbedt/ert/winddata/krab0192.txt&lt;/a&gt; (I took the 1200 measurements). I got a decent fit of the data, generally using your code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(lmom)&#xA;&#xA;daten &amp;lt;- read.delim(&quot;wind.txt&quot;)&#xA;wind.avg &amp;lt;- na.omit(as.numeric(daten[,&quot;X12&quot;]))&#xA;wind.moments&amp;lt;-samlmu(wind.avg)&#xA;moments&amp;lt;-pelwei(wind.moments)&#xA;x.wei&amp;lt;-rweibull(n=length(wind.avg), shape=moments[&quot;delta&quot;], scale=moments[&quot;beta&quot;])&#xA;hist(as.numeric(wind.avg), freq=FALSE)&#xA;lines(density(x.wei), col=&quot;red&quot;, lwd=4)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/JRb11.png&quot; alt=&quot;Wind Plot&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sorry, I'm not shure were your problem could be, but I think you should be able to fit weibull to your data. What makes me suspicious is the bell-curve of your density plot, I have no idea where that came from.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are the moments I generated:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;wind.moments&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;       l_1         l_2         t_3         t_4 &#xA;15.17287544  4.80372580  0.14963501  0.06954438&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;moments&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;     zeta      beta     delta &#xA; 0.516201 16.454233  1.745413 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;WTR to the annual output: I suppose I'd generate discrete values for the probability density function, multiply these values with the output function and sum it up. Alternatively, you could just use your raw data, multiply the values with the output function, sum it up and calculate the annual average, you should control for seasonality in a suitable way (e. g. make sure to use whole years, or to weight accordingly).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the uncontrolled output (using the formula from &lt;a href=&quot;http://www.articlesbase.com/diy-articles/determining-wind-turbine-annual-power-output-a-simple-formula-based-upon-blade-diameter-and-average-wind-speed-at-your-location-513080.html&quot; rel=&quot;nofollow&quot;&gt;http://www.articlesbase.com/diy-articles/determining-wind-turbine-annual-power-output-a-simple-formula-based-upon-blade-diameter-and-average-wind-speed-at-your-location-513080.html&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;years  &amp;lt;- length(wind.avg)/365&#xA;diameter &amp;lt;- 150&#xA;Power = (0.01328*diameter^2)*((wind.avg)^3)&#xA;(annual.power &amp;lt;- sum(Power)/years)&#xA;[1] 791828306&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1766" LastEditorUserId="1766" LastEditorDisplayName="" LastEditDate="2011-02-14T12:42:02.070" LastActivityDate="2011-02-14T12:42:02.070" />
  <row Id="7151" PostTypeId="2" ParentId="7128" CreationDate="2011-02-13T11:22:57.903" Score="1" ViewCount="" Body="&lt;p&gt;It's hard to name an appropriate method without knowing the research question you're trying to answer. With that in mind, multidimensional scaling (MDS) takes measures of global dissimilarity between pairs of stimuli as input data: observers are asked to rate the similarity of two stimuli without being given explicit criteria for that judgement.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;MDS tries to place the stimuli within a (low-dimensional) space such that the distance between the stimuli represent their dissimilarity. One is typically interested in the number of dimensions of the resulting space, and tries to infer some meaning about these dimensions. There are many variants of MDS. In R, check out the functions &lt;code&gt;cmdscale()&lt;/code&gt; as well as &lt;code&gt;isoMDS()&lt;/code&gt; and &lt;code&gt;sammon()&lt;/code&gt; from package &lt;code&gt;MASS&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1909" LastEditorUserId="1909" LastEditorDisplayName="" LastEditDate="2011-02-13T12:34:08.807" LastActivityDate="2011-02-13T12:34:08.807" CommentCount="2" />
  <row Id="7152" PostTypeId="1" AcceptedAnswerId="7470" CreationDate="2011-02-13T13:14:55.147" Score="0" ViewCount="183" Body="&lt;p&gt;If a cell has zero value, kindly guide me how to deal with it in statistical calculations.&lt;/p&gt;&#xA;" OwnerUserId="2956" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-02-22T00:57:17.433" LastActivityDate="2011-02-22T00:57:17.433" Title="Dealing with cell values of zero" Tags="&lt;sample-size&gt;" AnswerCount="2" CommentCount="3" FavoriteCount="2" />
  <row Id="7153" PostTypeId="1" AcceptedAnswerId="7163" CreationDate="2011-02-13T14:05:19.347" Score="1" ViewCount="71" Body="&lt;p&gt;Hi !&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to have a good idea on how to design clinical trials in oncology. In that issue, I am looking for a compact book that could give me a good overview, with the emphasis on statistical considerations.&#xA;Would you have a recommendation for me?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you in advance,&#xA;Marco&lt;/p&gt;&#xA;" OwnerUserId="3019" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-02-13T23:51:13.627" LastActivityDate="2011-02-13T23:51:13.627" Title="Book on designing clinical trials in oncology" Tags="&lt;books&gt;&lt;experiment-design&gt;&lt;clinical-trials&gt;" AnswerCount="1" CommentCount="4" />
  <row Id="7155" PostTypeId="1" AcceptedAnswerId="7158" CreationDate="2011-02-13T15:07:40.937" Score="19" ViewCount="429" Body="&lt;p&gt;People often talk about dealing with outliers in statistics.  The thing that bothers me about this is that, as far as I can tell, the definition of an outlier is completely subjective.  For example, if the true distribution of some random variable is very heavy-tailed or bimodal, any standard visualization or summary statistic for detecting outliers will incorrectly remove parts of the distribution you want to sample from.  What is a rigorous definition of an outlier, if one exists, and how can outliers be dealt with without introducing unreasonable amounts of subjectivity into an analysis?&lt;/p&gt;&#xA;" OwnerUserId="1347" LastActivityDate="2011-03-08T05:30:48.113" Title="Rigorous definition of an outlier?" Tags="&lt;outliers&gt;&lt;definition&gt;" AnswerCount="6" CommentCount="4" FavoriteCount="2" />
  <row Id="7156" PostTypeId="1" CreationDate="2011-02-13T16:00:25.593" Score="3" ViewCount="179" Body="&lt;p&gt;I have a dataset with yearly levels of corruption in a number of countries, as well as whether they changed their government that year.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;year, corruption, change of president&#xA;2001, 5, 0&#xA;2002, 7, 1&#xA;2003, 8, 0&#xA;etc.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to test whether corruption is affected by a change in power (defined as the election of a new president who isn't part of the same political party as the previous one).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The idea is to either look at the slope of corruption/year for the two years leading into the election, and the two years after, e.g. $t-2$, $t-1$ and $t$ for the before slope.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The other idea is to look at the average level of corruption three years before and after.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The rate might make more sense since there are more things that affect corruption and different countries may be on different trajectories. However, there could also be some benefit to just looking at average levels three years before and after.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any thoughts on which one I should look at, and also how to go about measuring this?&lt;/p&gt;&#xA;" OwnerUserId="3182" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-02-13T22:06:14.257" LastActivityDate="2011-02-13T22:06:14.257" Title="Difference between two slopes" Tags="&lt;hypothesis-testing&gt;" CommentCount="6" FavoriteCount="2" />
  <row Id="7157" PostTypeId="2" ParentId="7155" CreationDate="2011-02-13T16:03:53.543" Score="4" ViewCount="" Body="&lt;p&gt;You are correct that removing outliers can look like a subjective exercise but that doesn't mean that it's wrong.  The compulsive need to always have a rigorous mathematical reason for every decision regarding your data analysis is often just a thin veil of artificial rigour over what turns out to be a subjective exercise anyway.  This is especially true if you want to apply the same mathematical justification to every situation you come across.  (If there were bulletproof clear mathematical rules for everything then you wouldn't need a statistician.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, in your long tail distribution situation, there's no guaranteed method to just decide from the numbers whether you've got one underlying distribution of interest with outliers or two underlying distributions of interest with outliers being part of only one of them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Selecting the best methods in general for analysis is often subjective.  Whether it's unreasonably subjective depends on the explanation for the decision and on the outlier.&lt;/p&gt;&#xA;" OwnerUserId="601" LastEditorUserId="601" LastEditorDisplayName="" LastEditDate="2011-02-13T17:24:37.450" LastActivityDate="2011-02-13T17:24:37.450" />
  <row Id="7158" PostTypeId="2" ParentId="7155" CreationDate="2011-02-13T16:32:51.340" Score="8" ViewCount="" Body="&lt;p&gt;As long as your data comes from a know distribution with known properties, you can rigorously define an outlier as an event that is too unlikely to have been generated by the observed process (if you consider &quot;too unlikely&quot; to be non-rigorous, then all hypothesis testing is).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, this approach is problematic on two levels: It assumes that the data comes from a known distribution with known properties, and it brings the risk that outliers are looked at as data points that were smuggled into your data set by some magical faeries. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the absence of magical data faeries, all data comes from your experiment, and thus it is actually not possible to have outliers, just weird results. These can come from recording errors (e.g. a 400000 bedroom house for 4 dollars), systematic measurement issues (the image analysis algorithm reports huge areas if the object is too close to the border) experimental problems (sometimes, crystals precipitate out of the solution, which give very high signal), or features of your system (a cell can sometimes divide in three instead of two), but they can also be the result of a mechanism that no one's ever considered because it's rare and you're doing research, which means that some of the stuff you do is simply not known yet. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ideally, you take the time to investigate every outlier, and only remove it from your data set once you understand why it doesn't fit your model. This is time-consuming and subjective in that the reasons are highly dependent on the experiment, but the alternative is worse: If you don't understand where the outliers came from, you have the choice between letting outliers &quot;mess up&quot; your results, or defining some &quot;mathematically rigorous&quot; approach to hide your lack of understanding. In other words, by pursuing &quot;mathematical rigorousness&quot; you choose between not getting a significant effect and not getting into heaven. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If all you have is a list of numbers without knowing where they come from, you have no way of telling whether some data point is an outlier, because you can always assume a distribution where all data are inliers. &lt;/p&gt;&#xA;" OwnerUserId="198" LastEditorUserId="198" LastEditorDisplayName="" LastEditDate="2011-02-14T04:09:34.757" LastActivityDate="2011-02-14T04:09:34.757" CommentCount="5" />
  <row Id="7159" PostTypeId="1" CreationDate="2011-02-13T16:53:46.697" Score="3" ViewCount="140" Body="&lt;p&gt;Hi all,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even though this is not directly a question related to statistical analyses, I thought this group might have some useful insights. My school windows-xp based workstation takes several hours to run simple logistic regression models with bootstrap corrections and bootstrap procedures for model selection (R code adapted from Dr. Frank Harrell's book on logistic regression). Applying and getting a computing account with the University's computing department has been going extremely slowly (it's been over a month since I started the process). I need to get these analyses done as quickly as possible. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a result, I am considering getting a personal linux laptop from System76 (Pangolin). &lt;a href=&quot;https://www.system76.com/product_info.php?cPath=28&amp;amp;products_id=100&quot; rel=&quot;nofollow&quot;&gt;https://www.system76.com/product_info.php?cPath=28&amp;amp;products_id=100&lt;/a&gt;  I am hoping to utilize the R multicore package targeted at linux. However, I am not sure about which intel processor I should opt for. The i5 processors have 2 cores and higher clock speed but the i7 processors have 4 cores and lower clock speeds. The price difference between these two groups is around $500. I am really confused about this. Any insights would be immensely helpful. Thanks very much for your time!&lt;/p&gt;&#xA;" OwnerUserId="3183" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-02-13T17:30:58.240" LastActivityDate="2011-02-13T17:30:58.240" Title="Computing platform for R: System76 Linux laptop with dual core i5 or quad core i7?" Tags="&lt;logistic-regression&gt;&lt;model-selection&gt;&lt;bootstrap&gt;&lt;computing&gt;&lt;computational-statistics&gt;" CommentCount="4" FavoriteCount="1" />
  <row Id="7160" PostTypeId="2" ParentId="7152" CreationDate="2011-02-13T17:16:00.367" Score="7" ViewCount="" Body="&lt;p&gt;Zeros in tables are sometimes classified as structural, i.e.zero by design or by definition, or as random, i.e. a possible value that was observed. In the case of a study where no instances were observed despite being possible, the question often comes up: What is the one-sided 95% confidence interval above zero? This can be sensibly answered. It is, for instance, addressed in  &lt;a href=&quot;http://www.med.mcgill.ca/epidemiology/hanley/reprints/If_Nothing_Goes_1983.pdf&quot; rel=&quot;nofollow&quot;&gt;&quot;If Nothing Goes Wrong, Is Everything All Right? Interpreting Zero Numerators&quot; Hanley and Lippman-Hand.  JAMA. 1983;249(13):1743-45.&lt;/a&gt; Their bottom line was that the upper end of the confidence interval around the observed value of zero was 3/n where n was the number of observations. This &quot;rule of 3&quot; has been further addressed in later analyses and to my surprise I found it even has a &lt;a href=&quot;http://en.wikipedia.org/wiki/Rule_of_three_%28medicine%29&quot; rel=&quot;nofollow&quot;&gt;Wikipedia page&lt;/a&gt;. The best discussion I found was by &lt;a href=&quot;http://www.jstor.org/pss/2685405&quot; rel=&quot;nofollow&quot;&gt;Jovanovic and Levy in the American Statistician&lt;/a&gt;. That does not seem to be available in full-text in the searches, but can report after looking through it a second time that they modified the formula to be 3/(n+2) after sensible Bayesian considerations, which tightens up the CI a bit. There is a more recent review in &lt;a href=&quot;http://dx.doi.org/10.1111/j.1751-5823.2009.00078.x&quot; rel=&quot;nofollow&quot;&gt;International Statistical Review (2009), 77, 2, 266–275&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Addenda: After looking more closely at the last citation, above I also remember finding the extensive discussion in Agresti &amp;amp; Coull &quot;The American Statistician&quot;, Vol. 52, No. 2 (May, 1998), pp. 119-126 informative. The &quot;Agresti-Coull&quot; intervals are incorporated into various SAS and R functions. One R function with it is binom.confint {package:binom} by Sundar Dorai-Raj.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;: There are several methods for dealing with situations where an accumulation of &quot;zero&quot; observations distort an otherwise nice, tractable distribution of say costs or health-care usage patterns. These include zero-inflated and hurdle models as described by Zeileis in &lt;a href=&quot;http://cran.r-project.org/web/packages/pscl/vignettes/countreg.pdf&quot; rel=&quot;nofollow&quot;&gt;&quot;Regression Models for Count Data in R&quot;&lt;/a&gt;. Searching Google also demonstrates that Stata and SAS have facilities to handle such models.&lt;/p&gt;&#xA;" OwnerUserId="2129" LastEditorUserId="2129" LastEditorDisplayName="" LastEditDate="2011-02-13T20:51:24.727" LastActivityDate="2011-02-13T20:51:24.727" CommentCount="1" />
  <row Id="7162" PostTypeId="2" ParentId="7155" CreationDate="2011-02-13T19:30:44.873" Score="3" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Definition 1:&lt;/strong&gt; As already mentioned, an outlier in a group of data reflecting the same process (say process A) is an observation (or a set of observations) that is &lt;strong&gt;unlikely&lt;/strong&gt; to be a result of  process A. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This definition certainly involves an estimation of the likelyhood function of the process A (hence a model) and setting what unlikely means (i.e. deciding where to stop...). This definition is at the root of the answer I gave &lt;a href=&quot;http://stats.stackexchange.com/questions/213/what-is-the-best-way-to-identify-outliers-in-multivariate-data/386#386&quot;&gt;here&lt;/a&gt;. It is more related to ideas of &lt;strong&gt;hypothesis testing&lt;/strong&gt; of significance or &lt;strong&gt;goodness of fit&lt;/strong&gt;.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Definition 2&lt;/strong&gt; An outlier is an observation $x$ in a group of observations $G$ such that when modeling the group of observation with a given model the accuracy is higher if $x$ is removed and treated separately (with a mixture, in the spirit of what I mention &lt;a href=&quot;http://stats.stackexchange.com/questions/1444/how-should-i-transform-non-negative-data-including-zeros/1445#1445&quot;&gt;here&lt;/a&gt;) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This definition involves a &quot;given model&quot; and a measure of accuracy. I think this definition is more from the practical side and is more at the origin of outliers. At the Origin, outliers detection was a tool for &lt;strong&gt;robust statistic&lt;/strong&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Obviously these definition can be made very similar if you understand that calculating likelihood in the first definition involves modeling and calculation of a score :) &lt;/p&gt;&#xA;" OwnerUserId="223" LastEditorUserId="223" LastEditorDisplayName="" LastEditDate="2011-02-14T16:40:50.507" LastActivityDate="2011-02-14T16:40:50.507" />
  <row Id="7163" PostTypeId="2" ParentId="7153" CreationDate="2011-02-13T19:34:36.337" Score="1" ViewCount="" Body="&lt;p&gt;I asked a very similar question several months back in this post: &lt;a href=&quot;http://stats.stackexchange.com/questions/2770/good-text-on-clinical-trials&quot;&gt;Good Text on Clinical Trials&lt;/a&gt;.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I decided to go with &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0471727814&quot; rel=&quot;nofollow&quot;&gt;Clinical Trials: A Methodologic Perspective&lt;/a&gt;, by Steven Piantadosi I have been incredibly happy with the text. Now it's not aimed at oncology exactly, so this may not be the perfect text for you, but a lot of the fundamentals of clinical trials are addressed in the text, and from there, there are probably scores of journal articles related to oncology trials. &lt;/p&gt;&#xA;" OwnerUserId="1118" LastActivityDate="2011-02-13T19:34:36.337" CommentCount="1" />
  <row Id="7164" PostTypeId="1" CreationDate="2011-02-13T20:15:29.043" Score="4" ViewCount="44" Body="&lt;p&gt;I have a logistic regression model: $softmax(WX)$ where $W$ is my parameter matrix and $X$ is my input. I want a density function over the outputs of that model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say I know that my $X$ are distributed according to some density $p$. From the &lt;a href=&quot;http://en.wikipedia.org/wiki/Probability_density_function#Dependent_variables_and_change_of_variables&quot; rel=&quot;nofollow&quot;&gt;change of variables&lt;/a&gt; I know that the density of the outputs will be something like $p&amp;#39;(x) = p(x) det|J^{-1}|$ where $J = \frac{\partial{softmax(WX)}}{\partial{X}}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, this holds only for a bijective function $softmax(WX)$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What can I do if this is not the case - especially if $W$ is not square?&lt;/p&gt;&#xA;" OwnerUserId="2860" LastActivityDate="2011-04-14T04:00:01.310" Title="Change of variable with non bijective function" Tags="&lt;distributions&gt;&lt;probability&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="7165" PostTypeId="1" AcceptedAnswerId="7168" CreationDate="2011-02-13T20:24:38.900" Score="7" ViewCount="171" Body="&lt;p&gt;What book is the most thorough treatment of fundamental concepts in statistics? &#xA;I am not asking for a book on details of the methods of calculations and procedures, I am mainly interested in a book that thoroughly explains the foundational concepts ... an intuitive/illustrated/visual approach to the core ideas ... rather than load of mathematics equations etc. Size of the book is no problem ... even a large multi-volume text would do ... heck even a web resource would do.&lt;/p&gt;&#xA;" OwnerUserId="1620" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2011-02-13T23:28:51.173" LastActivityDate="2011-02-14T20:08:32.530" Title="A resource on concepts underlying statistics, not the techniques used in applied stats" Tags="&lt;books&gt;" AnswerCount="4" FavoriteCount="4" />
  <row Id="7166" PostTypeId="1" CreationDate="2011-02-13T21:04:03.320" Score="7" ViewCount="86" Body="&lt;p&gt;Hello,&#xA;a friend of mine has asked me to help him with predictive modelling of car traffic in a medium sized parking garage. The garage has its busy and easy days, its peak hours, dead hours opening hours (it is opened during 12 hours during weekdays and during 8 hours during weekends). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The goal is to predict how many cars will enter the garage during a given day (say, tomorrow) and how are these cars supposed to be distributed over the day.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please point me to general references (preferably, publicly available) to strategies and techniques.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you&lt;/p&gt;&#xA;" OwnerUserId="3184" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-04-13T10:34:55.570" LastActivityDate="2011-04-13T10:34:55.570" Title="General approaches to model car traffic in a parking garage" Tags="&lt;time-series&gt;&lt;multivariate-analysis&gt;&lt;predictive-models&gt;&lt;queueing&gt;" AnswerCount="2" CommentCount="4" />
  <row Id="7167" PostTypeId="2" ParentId="7146" CreationDate="2011-02-13T21:20:12.683" Score="3" ViewCount="" Body="&lt;p&gt;Here's a recent post at SO on wind turbines.  My answer on that link has three links that you might be interested in:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://stackoverflow.com/questions/4843194/r-language-sorting-data-into-ranges-averaging-ignore-outliers/4844783#4844783&quot;&gt;http://stackoverflow.com/questions/4843194/r-language-sorting-data-into-ranges-averaging-ignore-outliers/4844783#4844783&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I just checked one of the Weibull links in the above SO answer.  For some reason, the link is down.   Here are some links that provide the same basic information:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.gso.uri.edu/ozone/&quot; rel=&quot;nofollow&quot;&gt;http://www.gso.uri.edu/ozone/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.weru.ksu.edu/new_weru/publications/pdf/Comparison%20of%20the%20Weibull%20model%20with%20measured%20wind%20speed%20distributions%20for%20stochastic%20wind%20genera.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.weru.ksu.edu/new_weru/publications/pdf/Comparison%20of%20the%20Weibull%20model%20with%20measured%20wind%20speed%20distributions%20for%20stochastic%20wind%20genera.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.kfupm.edu.sa/ri/publication/cer/41_JP_Weibull_parameters_for_wind_speed_distribution_in_Saudi_Arabia.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.kfupm.edu.sa/ri/publication/cer/41_JP_Weibull_parameters_for_wind_speed_distribution_in_Saudi_Arabia.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://journal.dogus.edu.tr/13026739/2008/cilt9/sayi1/M00195.pdf&quot; rel=&quot;nofollow&quot;&gt;http://journal.dogus.edu.tr/13026739/2008/cilt9/sayi1/M00195.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.eurojournals.com/ejsr_26_1_01.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.eurojournals.com/ejsr_26_1_01.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, from the power generated from wind, the seasonality is obvious.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/FPnON.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/8mI1n.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="2775" LastEditorUserId="2775" LastEditorDisplayName="" LastEditDate="2011-02-14T01:24:19.100" LastActivityDate="2011-02-14T01:24:19.100" />
  <row Id="7168" PostTypeId="2" ParentId="7165" CreationDate="2011-02-13T21:40:37.220" Score="5" ViewCount="" Body="&lt;p&gt;It's hard to know exactly what you're looking for based on your post. Maybe you can edit it to clarify a little. I will say that to really understand statistics well, then you'll need to learn some math.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For fairly broad, low-level, introductory concepts, both&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Gonick and Smith, &lt;em&gt;A Cartoon Guide to Statistics&lt;/em&gt;, and&lt;/li&gt;&#xA;&lt;li&gt;D. Huff, &lt;em&gt;How to Lie with Statistics&lt;/em&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;are light, easy reads that present a lot of the core ideas. Another book directed at a more &quot;popular&quot; audience that I think every person should read is J. A. Paulos' &lt;em&gt;Innumeracy&lt;/em&gt;. It is not about probability or statistics, per se, and has more elementary probability than statistics, but it's framed in a way that I think most people can easily relate to.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have some calculus background and want to understand (introductory, frequentist) theoretical statistics, find a copy of Mood, Graybill and Boes, &lt;em&gt;Introduction to the Theory of Statistics&lt;/em&gt;, 3rd. ed. It's old, but in my opinion, still better than any of the more &quot;modern&quot; treatments. But, it's a book for which you'll have to be comfortable with mathematical notation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a &quot;modern&quot; view of applied statistics and the interface between it and machine learning, along with good examples, and good intuition, then Hastie et al., &lt;em&gt;Elements of Statistical Learning&lt;/em&gt;, is the most popular choice. Many people also tend to like Harrell's &lt;em&gt;Regression Modeling Strategies&lt;/em&gt;, which is a solid book, though I'm apparently not quite as big of a fan as others tend to be. Again, in both cases, you'll need to at least be comfortable with some calculus, linear algebra, and standard math notation.&lt;/p&gt;&#xA;" OwnerUserId="2970" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-02-13T22:37:46.510" LastActivityDate="2011-02-13T22:37:46.510" CommentCount="2" />
  <row Id="7169" PostTypeId="2" ParentId="7058" CreationDate="2011-02-13T22:04:12.390" Score="2" ViewCount="" Body="&lt;p&gt;This sounds like a problem of feature selection, if this is the case, I think you want to compute the &lt;a href=&quot;http://en.wikipedia.org/wiki/Mutual_information&quot; rel=&quot;nofollow&quot;&gt;mutual information&lt;/a&gt; between all subsets of features and the classification output. The subset with the highest mutual information will be the set of features that contains the most 'information' about the resulting classification of the record.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you only have 3 features, you can compute all possible subsets in a reasonable amount of time, if your feature set grows larger, you'll have to approximate this (typically using a greedy approach: take feature with the highest MI at each step).&lt;/p&gt;&#xA;" OwnerUserId="1913" LastActivityDate="2011-02-13T22:04:12.390" CommentCount="2" />
  <row Id="7170" PostTypeId="2" ParentId="6753" CreationDate="2011-02-13T22:14:43.607" Score="1" ViewCount="" Body="&lt;p&gt;Equi-depth histograms are a solution to the problem of &lt;a href=&quot;http://en.wikipedia.org/wiki/Quantization_%28signal_processing%29&quot; rel=&quot;nofollow&quot;&gt;quantization&lt;/a&gt; (mapping continuous values to discrete values).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For finding the best number of bins, I think it really depends on what you are trying to do with the histogram. In general I think it would be best to ensure your error of choice was below some threshold (eg. Sum of squared errors &amp;lt; THRESH) and bin the values in that manner.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively, the number of bins can be passed in as a parameter (if you're concerned about the space consumption of the histogram).&lt;/p&gt;&#xA;" OwnerUserId="1913" LastActivityDate="2011-02-13T22:14:43.607" CommentCount="2" />
  <row Id="7171" PostTypeId="2" ParentId="7165" CreationDate="2011-02-13T22:17:14.337" Score="4" ViewCount="" Body="&lt;p&gt;If you're interested in the philosophy of Statistics, you can't do much better than &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0805805281&quot; rel=&quot;nofollow&quot;&gt;Abelson's &quot;Statistics As Principled Argument&quot;&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="666" LastActivityDate="2011-02-13T22:17:14.337" />
  <row Id="7172" PostTypeId="1" CreationDate="2011-02-13T22:41:18.760" Score="4" ViewCount="53" Body="&lt;p&gt;I came by Bernard Silverman concept of &quot;rabbit ears hunting&quot; interactive procedure for density estimation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Might you offer:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;A recommended reference for this procedure?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;An explanation/justification for when it should &quot;work&quot;?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Any other suggested ad-hoc interactive criteria for density estimation?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Many thanks,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tal&lt;/p&gt;&#xA;" OwnerUserId="253" LastActivityDate="2011-02-13T22:41:18.760" Title="When/why should &quot;rabbit ears hunting&quot; density estimation &quot;work&quot;?" Tags="&lt;kernel&gt;&lt;kde&gt;&lt;density-function&gt;" CommentCount="3" FavoriteCount="0" />
  <row Id="7173" PostTypeId="1" AcceptedAnswerId="7174" CreationDate="2011-02-13T23:42:23.543" Score="2" ViewCount="260" Body="&lt;p&gt;I have a case-control study in which the cases are firms with health insurance and the controls are firms with no health insurance. I am studying the factors affecting enrolment in health insurance and was therefore using a logistic regression, which includes several covariates on firm characteristics that were measured in a survey. I have randomly sampled the firms from a database that includes two strata: insured and uninsured firms. I selected 65 from each group. However, within the group I also sampled from four strata  that correspond to industry. I am therefore wondering if I need to use conditional logistic regression, as opposed to unconditional logistic regression. However, I was under the impression that conditional logistic regression was for matched case-control studies or panel studies. In other feedback I've received I've been told that because I sampled on the outcome, I need to use the conditional model. Could someone please help me figure out which m? Any references would also be much appreciated. Thank you. &lt;/p&gt;&#xA;" OwnerUserId="834" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-02-16T22:08:54.653" LastActivityDate="2011-02-16T22:08:54.653" Title="How to decide between a logistic regression or conditional logistic regression?" Tags="&lt;logistic-regression&gt;&lt;survey&gt;" AnswerCount="1" CommentCount="4" FavoriteCount="1" />
  <row Id="7174" PostTypeId="2" ParentId="7173" CreationDate="2011-02-13T23:50:05.563" Score="3" ViewCount="" Body="&lt;p&gt;I don't agree that you sampled on the outcome, since you sampled on company and enrollment is your outcome. You may want to deal with the company as a random effect and the other features as fixed effects. So I am suggesting yet a third alternative: generalized mixed models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After clarification:&#xA;If the outcome is company enrollment rather than employee enrollment, then it is an ordinary case-control study for which unconditional logistic regression should be the standard approach. Conditional logistic regression is not necessary unless there were further conditions on the sampling regarding other company features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Further clarification:&#xA;If you were using R, then the package to identify and install would be not surprisingly: &quot;sampling&quot; by Thomas Lumley. It provides for the appropriate incorporation of the two-way sampling strategy you have outlined in the design phase prior to estimation with the svyglm() function. Stata also has a set of survey functions and I imagine they can also be used with the general linear modeling functions it provides. SAS didn't have such facilities in the past so the SUDAAN program was needed as an added (expensive) purchase, but I have a vague memory that this may have changed with its latest releases. (I don't know about SPSS with regard to sampling support for GLM models.)&lt;/p&gt;&#xA;" OwnerUserId="2129" LastEditorUserId="2129" LastEditorDisplayName="" LastEditDate="2011-02-16T21:06:49.663" LastActivityDate="2011-02-16T21:06:49.663" CommentCount="4" />
  <row Id="7175" PostTypeId="1" AcceptedAnswerId="7182" CreationDate="2011-02-14T00:21:17.413" Score="4" ViewCount="280" Body="&lt;p&gt;I'm experimenting with classifying data into groups. I'm quite new to this topic, and trying to understand the output of some of the analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using examples from &lt;a href=&quot;http://www.statmethods.net/advstats/cluster.html&quot; rel=&quot;nofollow&quot;&gt;Quick-R&lt;/a&gt;, several &lt;code&gt;R&lt;/code&gt; packages are suggested. I have tried using two of these packages (&lt;code&gt;fpc&lt;/code&gt; using the &lt;code&gt;kmeans&lt;/code&gt; function,  and &lt;code&gt;mclust&lt;/code&gt;). One aspect of this analysis that I do not understand is the comparison of the results.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# comparing 2 cluster solutions&#xA;library(fpc)&#xA;cluster.stats(d, fit1$cluster, fit2$cluster)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I've read through the relevant parts of the &lt;code&gt;fpc&lt;/code&gt; &lt;a href=&quot;http://cran.r-project.org/web/packages/fpc/fpc.pdf&quot; rel=&quot;nofollow&quot;&gt;manual&lt;/a&gt; and am still not clear on what I should be aiming for. For example, this is the output of comparing two different clustering approaches:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;$n&#xA;[1] 521&#xA;&#xA;$cluster.number&#xA;[1] 4&#xA;&#xA;$cluster.size&#xA;[1] 250 119  78  74&#xA;&#xA;$diameter&#xA;[1]  5.278162  9.773658 16.460074  7.328020&#xA;&#xA;$average.distance&#xA;[1] 1.632656 2.106422 3.461598 2.622574&#xA;&#xA;$median.distance&#xA;[1] 1.562625 1.788113 2.763217 2.463826&#xA;&#xA;$separation&#xA;[1] 0.2797048 0.3754188 0.2797048 0.3557264&#xA;&#xA;$average.toother&#xA;[1] 3.442575 3.929158 4.068230 4.425910&#xA;&#xA;$separation.matrix&#xA;          [,1]      [,2]      [,3]      [,4]&#xA;[1,] 0.0000000 0.3754188 0.2797048 0.3557264&#xA;[2,] 0.3754188 0.0000000 0.6299734 2.9020383&#xA;[3,] 0.2797048 0.6299734 0.0000000 0.6803704&#xA;[4,] 0.3557264 2.9020383 0.6803704 0.0000000&#xA;&#xA;$average.between&#xA;[1] 3.865142&#xA;&#xA;$average.within&#xA;[1] 1.894740&#xA;&#xA;$n.between&#xA;[1] 91610&#xA;&#xA;$n.within&#xA;[1] 43850&#xA;&#xA;$within.cluster.ss&#xA;[1] 1785.935&#xA;&#xA;$clus.avg.silwidths&#xA;         1          2          3          4 &#xA;0.42072895 0.31672350 0.01810699 0.23728253 &#xA;&#xA;$avg.silwidth&#xA;[1] 0.3106403&#xA;&#xA;$g2&#xA;NULL&#xA;&#xA;$g3&#xA;NULL&#xA;&#xA;$pearsongamma&#xA;[1] 0.4869491&#xA;&#xA;$dunn&#xA;[1] 0.01699292&#xA;&#xA;$entropy&#xA;[1] 1.251134&#xA;&#xA;$wb.ratio&#xA;[1] 0.4902123&#xA;&#xA;$ch&#xA;[1] 178.9074&#xA;&#xA;$corrected.rand&#xA;[1] 0.2046704&#xA;&#xA;$vi&#xA;[1] 1.56189&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;My primary question here is to better understand how to interpret the results of this cluster comparison.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Previously, I had asked more about the effect of scaling data, and calculating a distance matrix. However that was answered clearly by mariana soffer, and I'm just reorganizing my question to emphasize that I am interested in the intrepretation of my output which is a comparison of two different clustering algorithms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Previous part of question&lt;/em&gt;: &#xA;If I am doing any type of clustering, should I always scale data? For example, I am using the function &lt;code&gt;dist()&lt;/code&gt; on my scaled dataset as input to the &lt;code&gt;cluster.stats()&lt;/code&gt; function, however I don't fully understand what is going on. I read about &lt;code&gt;dist()&lt;/code&gt; &lt;a href=&quot;http://stat.ethz.ch/R-manual/R-patched/library/stats/html/dist.html&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; and it states that:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;this function computes and returns the distance matrix computed by using the specified distance measure to compute the distances between the rows of a data matrix.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="2635" LastEditorUserId="2635" LastEditorDisplayName="" LastEditDate="2011-02-20T21:59:36.737" LastActivityDate="2011-02-20T21:59:36.737" Title="Understanding comparisons of clustering results" Tags="&lt;r&gt;&lt;clustering&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="1" />
  <row Id="7176" PostTypeId="1" AcceptedAnswerId="7189" CreationDate="2011-02-14T00:28:54.207" Score="6" ViewCount="91" Body="&lt;p&gt;Everything I've read about combining errors in quadrature when multiplying or dividing quantities with associated errors says that this works for &quot;small error&quot;. What about large error? Say I want to compute A/B where A is +/- 1% and B is +/- 50%, can I still reasonably add the errors in quadrature?&lt;/p&gt;&#xA;" OwnerUserId="629" LastActivityDate="2011-02-14T07:51:51.503" Title="Propagation of large errors" Tags="&lt;error-propagation&gt;" AnswerCount="3" CommentCount="3" />
  <row Id="7177" PostTypeId="2" ParentId="7176" CreationDate="2011-02-14T02:10:59.663" Score="1" ViewCount="" Body="&lt;p&gt;The formula for error propagation&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\sigma_f^2 = \Sigma (\frac{\delta f}{\delta x} \sigma_x)^2$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;works exactly for normally distributed errors and linear functions $f(x_1,x_2,...)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since (most) functions can be linearly approximated, the above also works for small errors. For large errors, a symmetric distribution of $x$ leads to an asymmetric distribution of the error in $f$ (e.g. if $f(x)=x^{10}$, $f(0)=0$, $f(1)=1$, $f(2)=1024$, so the formula won't hold if $x=1$ and $\sigma_x=1$).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With large error, you may be able to calculate the transformation of the error distribution, otherwise you can perform a Monte Carlo simulation to estimate the distribution of $\sigma_f$.&lt;/p&gt;&#xA;" OwnerUserId="198" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-02-14T07:19:16.113" LastActivityDate="2011-02-14T07:19:16.113" />
  <row Id="7178" PostTypeId="2" ParentId="7176" CreationDate="2011-02-14T02:30:12.930" Score="0" ViewCount="" Body="&lt;p&gt;The first problem with large errors is that the expected value of the multiplication or division of the uncertain values will not be the multiplication or the division of the expected values.  So while it is true that $E[X+Y]=E[X]+E[Y]$ and $E[X-Y]=E[X]-E[Y]$, it would usually not be true to say $E[XY]=E[X]E[Y]$ or $E[X/Y]=E[X]/E[Y]$, though they will be close for small errors.  But for large errors, that effect will disrupt your propagation of error calculations.     &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second problem will be that the propagation of errors is asymmetric in multiplication and division, and that also becomes more important as the relative errors increase. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose for example you had $A$ being 270, 540 or 810 and $B$ being 3, 6 or 9.  Then $A/B$ could be 30, 45, 60, 90 (three ways), 135, 180, or 270.  While 90 may be the mode and median as well as 540/6, the mean is 110, and 30 is much closer to 90 (or to 110) than 270 is.     &lt;/p&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-02-14T02:30:12.930" />
  <row Id="7179" PostTypeId="1" AcceptedAnswerId="7193" CreationDate="2011-02-14T04:33:34.633" Score="2" ViewCount="55" Body="&lt;p&gt;Sorry if this is similar to another question, but I'm still trying to learn data analysis and I don't know what to search for.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I'd like to do is take some information that a potential customer enters into a web form and figure out the probability that they'll become a customer... giving our sales team an idea of which leads are most valuable. Specifically, being a travel company, there are criteria like origin and destination, date of travel vs. date they fill the form, and even how much/little they fill out the 3 subjective questions (about 100-150 characters seems ideal).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not even sure what the terminology is, so if anyone can provide some direction about where I can go to learn about this, I'd be very appreciative.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="3187" LastActivityDate="2011-02-14T09:37:18.190" Title="Resources for beginners - how to determine probability of user action based on certain criteria?" Tags="&lt;dataset&gt;&lt;resources&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="7180" PostTypeId="1" CreationDate="2011-02-14T05:32:01.177" Score="8" ViewCount="373" Body="&lt;p&gt;I have a dataset of project case studies for a new type of research method for Government agencies to support decision making activities.  My task is to develop an estimation method based on past experience for future projects for estimation purposes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My dataset is limited to 50 cases.  I have 30+ (potential) predictors recorded and one response variable (i.e. hours taken to complete the project).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Not all predictors are significant, using step-wise selection techniques I'm expecting number of prediction variables is likely to be in the 5-10 variable range.  Although I'm struggling to get a predictor set using the standard appraoches in tools like PASW (SPSS).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm well aware of all the material talking about rules of thumb for sample sizes and predictor variable to case ratios.  My dilemma is that it's taken close to 10 years to collect 50 cases as it is, so it's about as good as it will get.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is what should I do to get the most out of this small sample set?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That is any good references for dealing with small smaple sets?  Changes in p-value significance?  Changes to step-wise selection approaches?  Use of transforms such as centre-ing or log?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any advice is appreciated.&lt;/p&gt;&#xA;" OwnerUserId="3189" LastActivityDate="2011-02-14T06:47:37.777" Title="Multiple regression with small data sets" Tags="&lt;regression&gt;&lt;small-sample&gt;" AnswerCount="1" />
  <row Id="7181" PostTypeId="2" ParentId="7180" CreationDate="2011-02-14T05:53:15.473" Score="3" ViewCount="" Body="&lt;p&gt;Hello Shane:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you want to select a few predictors from your data set, I would suggest a simple linear regression with $L_1$ penalty or using the &lt;em&gt;LASSO&lt;/em&gt; (penalized linear regression). Your case is suited for regression with &lt;a href=&quot;http://en.wikipedia.org/wiki/Lasso_%28statistics%29#LASSO_method&quot; rel=&quot;nofollow&quot;&gt;LASSO&lt;/a&gt; penalty as your sample size, $n = 50$, and the number of predictors, $p=30$. Changing the tuning parameter will select the number of predictors you want to choose.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you can give details about the distribution of your variables, I can be more specific. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't use SPSS, but this can be done easily in &lt;code&gt;R&lt;/code&gt; using the &lt;code&gt;glmnet&lt;/code&gt; function in the package of the same name. If you look in the manual, it contains a generic example (very first one, for &lt;em&gt;gaussian case&lt;/em&gt;) which will solve your problem. I am sure, similar solution &lt;em&gt;must&lt;/em&gt; exist in SPSS. &lt;/p&gt;&#xA;" OwnerUserId="1307" LastEditorUserId="1307" LastEditorDisplayName="" LastEditDate="2011-02-14T06:47:37.777" LastActivityDate="2011-02-14T06:47:37.777" CommentCount="9" />
  <row Id="7182" PostTypeId="2" ParentId="7175" CreationDate="2011-02-14T06:03:07.077" Score="5" ViewCount="" Body="&lt;p&gt;First let me tell you that I am not going to explain exactly all the measures here, but I am going to give you an idea about how to compare how good is the clustering (lets assume we are comparing 2 clustering methods of the same amount of clusters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1.For example the bigger the diameter of the cluster, the worst the clustering, because the points that belong to it are more scattered.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2.The higher the average distance of each clustering, the worst the clustering method (Lets assume that the average distance is the averaged distance of each point from the cluster to the center of the cluster)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then there are this 2 metrics that are the most used, check the links to understand what they stand for:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;inter-cluster distance (the higher the better, is the summatory of the distance between the different cluster centroids)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;intra-cluster distance (the lower the better, is the summatory of the distance between the cluster members to the center of the cluster)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also for understanding better the metrics above check &lt;a href=&quot;http://www.stanford.edu/~maureenh/quals/html/ml/node82.html&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then you should read the manual of the library and functions you are using to understand which measure represent which one of these, or if it is not here what is the meaning of it, but I wouldn't bother I would stick with the ones I stated here)-&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lets go on with the questions you made:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) regarding scaling data: Yes you should always scale the data for clustering, otherwise the different scales of the different dimensions (variables) will have different influences in how  data is clustered, the higher the values from the variable, the more influential will be that variable in how the clustering is done, while indeed they should all have the same (unless for some particular strange reason you do not want it that way)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) The distance functions computes all the distances from one point(instance) to another. The most common distant measure is euclidean, so for example lets suppose you wan't to measure the distance from instance 1 to instance 2 (lets assume you only have 2 instances for the sake of simplicity). Also lets assume that each instance has 3 values &lt;code&gt;(x1,x2,x3)&lt;/code&gt;, so &lt;code&gt;I1=0.3,0.2,0.5&lt;/code&gt; and &lt;code&gt;I2=0.3,0.3,0.4&lt;/code&gt; so the euclidean distance from I1 and I2 would be: &lt;code&gt;sqrt((0.3-0.2)^2+(0.2-0.3)^2+(0.5-0.4)^2)=0.17&lt;/code&gt;, hence the distance matrix will result in&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    i1    i2&#xA;i1  0     0.17&#xA;i2  0.17  0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;notice that the distance matrix is always simmetrical.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the euclidean distance formula is not the only one that exists, there are many other distances that can be used to calculate this matrix. Check for example in wikipedia &lt;a href=&quot;http://en.wikipedia.org/wiki/Manhattan_distance&quot; rel=&quot;nofollow&quot;&gt;Manhattain Distance&lt;/a&gt; and how to calculate it. At the end of the wikipedia page for &lt;a href=&quot;http://en.wikipedia.org/wiki/Euclidean_distance&quot; rel=&quot;nofollow&quot;&gt;Euclidean Distance&lt;/a&gt; you can also check it's formula and at the end check which other distances exists.&lt;/p&gt;&#xA;" OwnerUserId="1808" LastActivityDate="2011-02-14T06:03:07.077" CommentCount="3" />
  <row Id="7183" PostTypeId="1" AcceptedAnswerId="7184" CreationDate="2011-02-14T06:03:25.210" Score="2" ViewCount="234" Body="&lt;p&gt;I have a matrix with hundreds of rows and tens of columns and wish to plot a heatmap. If I use the native R function:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;heatmap(matrix(sample(1:10000),nrow=1000,ncol=10))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I get a figure with illegible row titles. I assume the image is produced to match the specifications of the current plotting device. I'd like control over the height of the rows, even if this can't be displayed on my current device. Simply writing to a tall PNG just adds whitespace above/below the image:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;png( '/looks_the_same_with_whitespace.png', width=500, height=1500)&#xA;heatmap(matrix(sample(1:10000),nrow=1000,ncol=10))&#xA;dev.off()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is there a clean way to do this, perhaps by fooling R into thinking I have a very tall monitor? A function from a well-supported library is also a fine answer.&lt;/p&gt;&#xA;" OwnerUserId="1699" LastActivityDate="2011-02-14T06:34:36.140" Title="Increasing row height in R heatmap() function" Tags="&lt;r&gt;&lt;data-visualization&gt;" AnswerCount="1" />
  <row Id="7184" PostTypeId="2" ParentId="7183" CreationDate="2011-02-14T06:34:36.140" Score="2" ViewCount="" Body="&lt;p&gt;Hello David:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A while ago, I also faced the same problem. &lt;code&gt;heatmap.2&lt;/code&gt; function from &lt;code&gt;gplots&lt;/code&gt; package solved the problem. However, I don't quite see how looking at so many genes (or rowlabs) is beneficial. But as you asked you would do something like this: The key is change the &lt;code&gt;layout&lt;/code&gt; (see ?heatmap.2, ?layout) of the heatmap which draws heatmap on a 2x2 grid on the plotting device (the example in ?layout explains this much better). After that, you may want to change the &lt;code&gt;cex&lt;/code&gt; of &lt;code&gt;rowlabs&lt;/code&gt; by changing &lt;code&gt;cexRow&lt;/code&gt; accordingly. Your situation might need a bit of playing around with the values to get the desired result. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(gplots)&#xA;&#xA;row.scaled.expr &amp;lt;- matrix(sample(1:10000),nrow=1000,ncol=10)&#xA;&#xA;png( 'heatmap_without_whitespace_smaller_row_lab.png', width=500, height=1500)&#xA;heatmap.2(row.scaled.expr, dendrogram ='row',&#xA;                 Colv=FALSE, col=greenred(800), &#xA;                 key=FALSE, keysize=1.0, symkey=FALSE, density.info='none',&#xA;                 trace='none', colsep=1:10,&#xA;                 sepcolor='white', sepwidth=0.05,&#xA;                 scale=&quot;none&quot;,cexRow=0.2,cexCol=2,&#xA;                 labCol = colnames(row.scaled.expr),                 &#xA;                 hclustfun=function(c){hclust(c, method='mcquitty')},&#xA;                 lmat=rbind( c(0, 3), c(2,1), c(0,4) ), lhei=c(0.25, 4, 0.25 ),                 &#xA;                 )&#xA;dev.off()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1307" LastActivityDate="2011-02-14T06:34:36.140" CommentCount="1" />
  <row Id="7185" PostTypeId="1" AcceptedAnswerId="7199" CreationDate="2011-02-14T06:46:27.827" Score="3" ViewCount="265" Body="&lt;p&gt;Hi all,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone tell me the difference between using aov and lme for analyzing longitudinal data and how to interpret results from these two methods?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Below, I analyze the same dataset using aov and lme and got 2 different result. with aov, i got a significant result in the time by treatment interaction factor. But fitting a linear mixed model, time by treatment interaction is insignificant.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; UOP.kg.aov &amp;lt;- aov(UOP.kg~time*treat+Error(id), raw3.42)&#xA;&amp;gt; summary(UOP.kg.aov)&#xA;&#xA;Error: id&#xA;          Df  Sum Sq Mean Sq F value Pr(&amp;gt;F)&#xA;treat      1   0.142  0.1421  0.0377 0.8471&#xA;Residuals 39 147.129  3.7725               &#xA;&#xA;Error: Within&#xA;            Df  Sum Sq Mean Sq  F value  Pr(&amp;gt;F)    &#xA;time         1 194.087 194.087 534.3542 &amp;lt; 2e-16 ***&#xA;time:treat   1   2.077   2.077   5.7197 0.01792 *  &#xA;Residuals  162  58.841   0.363                     &#xA;---&#xA;Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1&#xA;&#xA;&amp;gt; UOP.kg.lme &amp;lt;- lme(UOP.kg~time*treat, random=list(id=pdDiag(~time)), na.action=na.omit, raw3.42)&#xA;&amp;gt; summary(UOP.kg.lme)&#xA;Linear mixed-effects model fit by REML&#xA; Data: raw3.42 &#xA;       AIC      BIC    logLik&#xA;  225.7806 248.9037 -105.8903&#xA;&#xA;Random effects:&#xA; Formula: ~time | id&#xA; Structure: Diagonal&#xA;        (Intercept)      time  Residual&#xA;StdDev:   0.6817425 0.5121545 0.1780466&#xA;&#xA;Fixed effects: UOP.kg ~ time + treat + time:treat &#xA;                 Value Std.Error  DF   t-value p-value&#xA;(Intercept)  0.5901420 0.1480515 162  3.986059  0.0001&#xA;time         0.8623864 0.1104533 162  7.807701  0.0000&#xA;treat       -0.2144487 0.2174843  39 -0.986042  0.3302&#xA;time:treat   0.1979578 0.1622534 162  1.220053  0.2242&#xA; Correlation: &#xA;           (Intr) time   treat &#xA;time       -0.023              &#xA;treat      -0.681  0.016       &#xA;time:treat  0.016 -0.681 -0.023&#xA;&#xA;Standardized Within-Group Residuals:&#xA;         Min           Q1          Med           Q3          Max &#xA;-3.198315285 -0.384858426  0.002705899  0.404637305  2.049705655 &#xA;&#xA;Number of Observations: 205&#xA;Number of Groups: 41 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1663" LastActivityDate="2011-02-14T15:35:47.063" Title="What is the difference between using aov() and lme() in analyzing a longitudinal dataset?" Tags="&lt;r&gt;&lt;mixed-model&gt;&lt;repeated-measures&gt;&lt;longitudinal&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="7186" PostTypeId="2" ParentId="3" CreationDate="2011-02-14T07:39:27.737" Score="3" ViewCount="" Body="&lt;p&gt;First of all let me tell you that in my opinion the best tool of all by far is R, which has tons of libraries and utilities I am not going to enumerate here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Let me expand the discussion about weka&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a library for R, which is called RWeka, which you can easily install in R, and use many of the functionalities from this great program along with the ones in R, let me give you a code example for doing a simple decision tree read from a standard database that comes with this package (it is also very easy to draw the resulting tree but I am going to let you do the research about how to do it, which is in the RWeka documentation:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(RWeka)&#xA;iris &amp;lt;- read.arff(system.file(&quot;arff&quot;, &quot;iris.arff&quot;, package = &quot;RWeka&quot;))&#xA;classifier &amp;lt;- IBk(class ~., data = iris)&#xA;summary(classifier)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;There are also several python libraries for doing this&lt;/strong&gt; (python is very very easy to learn)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First let me enumerate the packages you can use, I am not going to go in detail about them;&#xA;Weka(yes you have a library for python), NLKT (the most famous open source package for textmining besides datamining), &lt;a href=&quot;http://www.astro.cornell.edu/staff/loredo/statpy/&quot; rel=&quot;nofollow&quot;&gt;statPy&lt;/a&gt;, &lt;a href=&quot;http://scikit-learn.sourceforge.net/index.html&quot; rel=&quot;nofollow&quot;&gt;sickits&lt;/a&gt;, and scipy. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is also orange which is excelent ( I will also talk about it later), here is a code example for doing a tree from the data in the table cmpart1, that also performs 10 folds validation, you can also graph the tree &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import orange, orngMySQL, orngTree&#xA;&#xA;data = orange.ExampleTable(&quot;c:\\python26\\orange\\cmpart1.tab&quot;)&#xA;&#xA;domain=data.domain&#xA;n=10&#xA;buck=len(data)/n&#xA;l2=[]&#xA;for i in range(n):&#xA;    tmp=[]&#xA;    if i==n-1:&#xA;        tmp=data[n*buck:]&#xA;    else:&#xA;        tmp=data[buck*i:buck*(i+1)]&#xA;    l2.append(tmp)&#xA;&#xA;train=[]&#xA;test=[]&#xA;di={'yy':0,'yn':0,'ny':0,'nn':0}&#xA;for i in range(n):&#xA;    train=[]&#xA;    test=[]&#xA;    for j in range(n):&#xA;        if j==i:&#xA;            test=l2[i]&#xA;        else:&#xA;            train.extend(l2[j])&#xA;    print &quot;-----&quot;&#xA;    trai=orange.Example(domain, train)&#xA;    tree = orngTree.TreeLearner(train)&#xA;    for ins in test:&#xA;        d1= ins.getclass()&#xA;        d2=tree(ins)&#xA;        print d1&#xA;        print d2&#xA;        ind=str(d1)+str(d2)&#xA;        di[ind]=di[ind]+1&#xA;print di&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;To end with some other packages I used and found interesting&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://orange.biolab.si/&quot; rel=&quot;nofollow&quot;&gt;Orange&lt;/a&gt;:data visualization and analysis for novice and experts. Data mining through visual programming or Python scripting. Components for machine learning. Extensions for bioinformatics and text mining. (I personally recomend this, I used it a lot integrating it in python and it was excelent) I can send you some python code if you want me to. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.lcb.uu.se/tools/rosetta/index.php&quot; rel=&quot;nofollow&quot;&gt;ROSETTA&lt;/a&gt;: toolkit for analyzing tabular data within the framework of rough set theory. ROSETTA is designed to support the overall data mining and knowledge discovery process: From initial browsing and preprocessing of the data, via computation of minimal attribute sets and generation of if-then rules or descriptive patterns, to validation and analysis of the induced rules or patterns.(This I also enjoyed using very much)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.keel.es&quot; rel=&quot;nofollow&quot;&gt;KEEL&lt;/a&gt;:assess evolutionary algorithms for Data Mining problems including regression, classification, clustering, pattern mining and so on. It allows us to perform a complete analysis of any learning model in comparison to existing ones, including a statistical test module for comparison. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.itl.nist.gov/div898/software/dataplot/homepage.htm&quot; rel=&quot;nofollow&quot;&gt;DataPlot&lt;/a&gt;: for scientific visualization, statistical analysis, and non-linear modeling. The target Dataplot user is the researcher and analyst engaged in the characterization, modeling, visualization, analysis, monitoring, and optimization of scientific and engineering processes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.statpages.org/miller/openstat/&quot; rel=&quot;nofollow&quot;&gt;Openstats&lt;/a&gt;: Includes A Statistics and Measurement Primer, Descriptive Statistics, Simple Comparisons, Analyses of Variance, Correlation, Multiple Regression, Interrupted Time Series, Multivariate Statistics, Non-Parametric Statistics, Measurement, Statistical Process Control, Financial Procedures, Neural Networks,&#xA;Simulation&lt;/p&gt;&#xA;" OwnerUserId="1808" LastActivityDate="2011-02-14T07:39:27.737" CommunityOwnedDate="2011-02-14T07:39:27.737" CommentCount="1" />
  <row Id="7187" PostTypeId="2" ParentId="7112" CreationDate="2011-02-14T07:45:27.760" Score="-1" ViewCount="" Body="&lt;p&gt;Let me reply with a short answer dough it might overlap with the excelent answer written before.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Always standarize, that allows you to interpret the regression, specially the coefficients of the regression better.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;For the new data which is not standarize, I recomend you to store the values you used for each variable to be standarized, such as the maximum and the minimum, and then do the same transformation you did in the hole dataset before but just for this single instance.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="1808" LastActivityDate="2011-02-14T07:45:27.760" />
  <row Id="7188" PostTypeId="2" ParentId="7115" CreationDate="2011-02-14T07:51:24.210" Score="1" ViewCount="" Body="&lt;p&gt;Lets suppose we can calculate the distance from one noun to another in the following way. Use the Worldnet (which I guess you know), and utilize a function that exists, but you can build it yourself, that counts for how many points of the taxonomy of words you need to get from one word to another (for example from cat to dog you might have 4 but from nail to music you might have 25) Then using this numbers calculated among the nouns of the sentences just invent a metric (for example use simply the average of distance between the nouns, or use the minimum distance between the nouns) that will help you carry on your task.&lt;/p&gt;&#xA;" OwnerUserId="1808" LastActivityDate="2011-02-14T07:51:24.210" CommentCount="3" />
  <row Id="7189" PostTypeId="2" ParentId="7176" CreationDate="2011-02-14T07:51:51.503" Score="1" ViewCount="" Body="&lt;p&gt;For large error, the standard error of $A/B$ depends on the &lt;em&gt;distributions&lt;/em&gt; of $A$ and $B$, not just on their standard errors. The distribution of $A/B$ is known as a &lt;a href=&quot;http://en.wikipedia.org/wiki/Ratio_distribution&quot; rel=&quot;nofollow&quot;&gt;ratio distribution&lt;/a&gt;, but &lt;em&gt;which&lt;/em&gt; ratio distribution depends on the distributions of $A$ and $B$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we assume that $A$ and $B$ both have Gaussian (normal) distributions, then $A/B$ has a &lt;a href=&quot;http://en.wikipedia.org/wiki/Ratio_distribution#Gaussian_ratio_distribution&quot; rel=&quot;nofollow&quot;&gt;Gaussian ratio distribution&lt;/a&gt;, for which a closed form exists but is rather complicated. In general, this will be an asymmetric distribution, so it is not well characterised simply by its mean and standard deviation. However, it is possible to find a confidence interval for $A/B$ using &lt;a href=&quot;http://en.wikipedia.org/wiki/Fieller%27s_theorem&quot; rel=&quot;nofollow&quot;&gt;Fieller's theorem&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-02-14T07:51:51.503" />
  <row Id="7190" PostTypeId="1" CreationDate="2011-02-14T08:16:36.003" Score="2" ViewCount="82" Body="&lt;p&gt;I have a set of integer that I am trying to see use different methods to categorize them into four groups, and the 2x2 table for the outcome of the 2 methods is displayed as below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;                method_B&#xA;     method_A   0   1   2   3&#xA;            0 182  11   0   0&#xA;            1  41 127   2   0&#xA;            2   0  12  18   0&#xA;            3   0   0   0   4&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am thinking of using &lt;code&gt;chisq.test&lt;/code&gt; for comparing whether the difference in grouping distribution produced by 2 methods are likely to be caused by chance, but I am worrying about too many zeroes in the 2x2 table.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should I use &lt;code&gt;chi.sq&lt;/code&gt; test? Or is there any equivalent &lt;code&gt;fisher.test&lt;/code&gt; for my case?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S. I tried &lt;code&gt;fisher.test&lt;/code&gt; but it gives me &lt;code&gt;FEXACT ERROR 501&lt;/code&gt;, which I have no idea what it is.&lt;/p&gt;&#xA;" OwnerUserId="588" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-02-14T08:18:23.990" LastActivityDate="2011-02-14T08:35:26.457" Title="Statistical test for difference in categorization method" Tags="&lt;categorical-data&gt;&lt;chi-squared&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="7191" PostTypeId="2" ParentId="7139" CreationDate="2011-02-14T08:32:27.437" Score="0" ViewCount="" Body="&lt;p&gt;Here is another question from this websites that says among other things the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What can a statistical model say about causation?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This led to his motto:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;NO CAUSATION WITHOUT MANIPULATION&lt;/p&gt;&#xA;&#xA;&lt;p&gt;which emphasized the importance of restrictions around experiments that consider causation. Andrew Gelman makes a similar point:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;To find out what happens when you change something, it is necessary to change it.&quot;...There are things you learn from perturbing a system that you'll never find out from any amount of passive observation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;check it out: &lt;a href=&quot;http://stats.stackexchange.com/questions/2245/statistics-and-causal-inference&quot;&gt;Statistics and causal inference?&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1808" LastActivityDate="2011-02-14T08:32:27.437" />
  <row Id="7192" PostTypeId="2" ParentId="7190" CreationDate="2011-02-14T08:35:26.457" Score="2" ViewCount="" Body="&lt;p&gt;I don't think either chi-squared test or Fisher's exact test is useful here. It's pretty obvious that the results from method A and from method B aren't independent of each other. A statistic to &lt;em&gt;quantify how well&lt;/em&gt; the two methods agree is more useful. The obvious choice is &lt;a href=&quot;http://en.wikipedia.org/wiki/Cohen%27s_kappa&quot; rel=&quot;nofollow&quot;&gt;Cohen's kappa&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-02-14T08:35:26.457" />
  <row Id="7193" PostTypeId="2" ParentId="7179" CreationDate="2011-02-14T09:37:18.190" Score="2" ViewCount="" Body="&lt;p&gt;Your problem can be described as &quot;binary classification problem&quot; with the response variable &quot;become_customer&quot; $\in$ {yes,no}.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as see, your next steps should be:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;create a tabular dataset with one row = one visitor and columns = predictors + response variable. Predictor variables reflect the input into the forms (as described by you), meanwhile response is &quot;yes&quot;, if the visitor became a customer, else &quot;no&quot;.&lt;/li&gt;&#xA;&lt;li&gt;learn and validate (keyword: Crossvalidation; metrics like AUC, Precision, Recall etc.) an expressive (non black-box) classification model. On suggestion is &quot;Decision Trees&quot;, which are mainly used in the area of Direct Marketing / Performance Marketing since they can give you a description of potential customers.&lt;/li&gt;&#xA;&lt;li&gt;By calibrating the output of the scores of the classification model you can additionally get an estimate of the conversion rate for each customer.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="264" LastActivityDate="2011-02-14T09:37:18.190" CommentCount="3" />
  <row Id="7194" PostTypeId="2" ParentId="7165" CreationDate="2011-02-14T11:08:24.803" Score="1" ViewCount="" Body="&lt;p&gt;I like Kennedy's Guide to Econometrics, which treats every topic on three levels, the first of which is a non-technical description, in so far as this is possible.&lt;/p&gt;&#xA;" OwnerUserId="1766" LastActivityDate="2011-02-14T11:08:24.803" CommentCount="3" />
  <row Id="7195" PostTypeId="2" ParentId="7166" CreationDate="2011-02-14T11:47:39.873" Score="5" ViewCount="" Body="&lt;p&gt;The field that is relevant to the problem is the &lt;a href=&quot;http://en.wikipedia.org/wiki/Queueing_model&quot; rel=&quot;nofollow&quot;&gt;Queuing theory&lt;/a&gt;, a particular sub-field is a &lt;a href=&quot;http://en.wikipedia.org/wiki/Birth-death_process&quot; rel=&quot;nofollow&quot;&gt;Birth-death&lt;/a&gt; processes. An article that in my opinion is helpful to your task is R.C. Larson and K.Satsunama (2010) &lt;a href=&quot;http://www.jise.info/issues/volume4no1/01-V4N1-Full.pdf&quot; rel=&quot;nofollow&quot;&gt;&lt;em&gt;Congestion Pricing: A Parking Queue Model&lt;/em&gt;&lt;/a&gt;, following the links in references would give more ideas on where to proceed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note, that recently R package &lt;a href=&quot;http://cran.r-project.org/web/packages/queueing/&quot; rel=&quot;nofollow&quot;&gt;queueing&lt;/a&gt; has been released (with misprint in the title however). Finally, I think, that this link for &lt;a href=&quot;http://web2.uwindsor.ca/math/hlynka/qsoft.html&quot; rel=&quot;nofollow&quot;&gt;queuing software&lt;/a&gt; could be helpful.&lt;/p&gt;&#xA;" OwnerUserId="2645" LastActivityDate="2011-02-14T11:47:39.873" CommentCount="2" />
  <row Id="7196" PostTypeId="1" AcceptedAnswerId="7203" CreationDate="2011-02-14T13:47:16.960" Score="3" ViewCount="72" Body="&lt;p&gt;The data I want to analyze consist of set of votes similar to the voting system here on stackexchange. Votes are binary, i.e, items can receive up- or down-votes. The data have been gathered in an A/B controlled experiment. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to compare the control group to the treatment group according to some gold standard. That is, for some items I know what the &quot;correct&quot; vote (according to the gold standard) should have been. However, the number of votes per item varies and can be different between control and treatment group. My hypothesis is, that the treatment reduces the voting &quot;error&quot;. But I am not sure, which statistical test to use.&lt;/p&gt;&#xA;" OwnerUserId="3191" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-02-14T14:48:20.330" LastActivityDate="2011-02-14T16:51:13.537" Title="Appropriate test to compare online voting data" Tags="&lt;hypothesis-testing&gt;" AnswerCount="1" />
  <row Id="7197" PostTypeId="1" CreationDate="2011-02-14T14:36:39.887" Score="2" ViewCount="308" Body="&lt;p&gt;I'm looking for sample R code, or pointers to sample R code for the following.  (Gentle) critique of the approach would also be appreciated.  I'm not a statistician and I'm pretty new to R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have duration of hospitalization (&quot;length of stay&quot;=LOS) data for 2,000 patients from my hospital and 50,000 patients from a comparison data set.  Each patient has a discharge year (YEAR) and is assigned to a diagnosis group (DG).  My goal is to compare (with confidence intervals) the overall average LOS at my hospital to the expected average LOS if those patients had been part of the comparison data set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that for a specific year and specific DG there may be anywhere from a few to 1,000s of patients, and, of course, the distribution of LOS is not normal.  There could possibly even be a DG with 1 or more patients for a given year in my hospital's data but none in the comparison data set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The approach I was considering was to create a comparison group where a random patient from the comparison data set is chosen for each patient from my hospital.  The random patient would be matched by YEAR and DG.  I would calculate the expected average LOS for this group and then repeat the process 10,000 times to determine the 2.5th to 97.5th percentile.  I would repeat for each year and plot my hospital's average LOS versus the 95% CI for the expected average LOS.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To deal with the issue of there not being a match for a patient for a given DG for a given YEAR I was thinking of loosening the match criteria to pick patients from the previous or next year.  I could keep broadening the match year until there were at least N patients from which to randomly pick.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thoughts?&lt;/p&gt;&#xA;" OwnerUserId="690" LastActivityDate="2011-05-26T15:50:28.417" Title="Calculate confidence interval using R for average hospital length of stay, case-mix adjusted" Tags="&lt;r&gt;&lt;confidence-interval&gt;&lt;bootstrap&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="3" />
  <row Id="7198" PostTypeId="2" ParentId="7185" CreationDate="2011-02-14T15:10:27.500" Score="0" ViewCount="" Body="&lt;p&gt;It appears to me you have multiple measures for each id at each time.  You need to aggregate these for the aov because it unfairly increases power in that analysis.  I'm not saying doing the aggregate will make the outcomes the same but it should make them more similar.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dat.agg &amp;lt;- aggregate(UOP.kg ~ time + treat + id, raw3.42, mean)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then run your aov model as before replacing the data with dat.agg.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, I believe anova(lme) is more what you want to do to compare the outcomes. The direction and magnitude of an effect is not the same as the ratio of model variance to error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(BTW, if you do the lme analysis on the aggregate data, which you shouldn't, and check anova(lme) you'll get almost the same results as aov)&lt;/p&gt;&#xA;" OwnerUserId="601" LastActivityDate="2011-02-14T15:10:27.500" />
  <row Id="7199" PostTypeId="2" ParentId="7185" CreationDate="2011-02-14T15:35:47.063" Score="4" ViewCount="" Body="&lt;p&gt;Based on your description, it appears that you have a repeated-measures model with a single treatment factor. Since I do not have access to the dataset (&lt;code&gt;raw3.42&lt;/code&gt;), I will use the Orthodont data from the &lt;code&gt;nlme&lt;/code&gt; package to illustrate what is going on here. The data structure is the same (repeated measurements for two different groups - in this case, males and females).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you run the following code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(nlme)&#xA;data(Orthodont)&#xA;&#xA;res &amp;lt;- lme(distance ~ age*Sex, random = ~ 1 | Subject, data = Orthodont)&#xA;anova(res)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;you will get the following results:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;            numDF denDF  F-value p-value&#xA;(Intercept)     1    79 4123.156  &amp;lt;.0001&#xA;age             1    79  122.450  &amp;lt;.0001&#xA;Sex             1    25    9.292  0.0054&#xA;age:Sex         1    79    6.303  0.0141&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you run:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;res &amp;lt;- aov(distance ~ age*Sex + Error(Subject), data = Orthodont)&#xA;summary(res)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;you will get:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Error: Subject&#xA;          Df Sum Sq Mean Sq F value   Pr(&amp;gt;F)   &#xA;Sex        1 140.46 140.465  9.2921 0.005375 **&#xA;Residuals 25 377.91  15.117                    &#xA;&#xA;Error: Within&#xA;          Df  Sum Sq Mean Sq  F value  Pr(&amp;gt;F)    &#xA;age        1 235.356 235.356 122.4502 &amp;lt; 2e-16 ***&#xA;age:Sex    1  12.114  12.114   6.3027 0.01410 *  &#xA;Residuals 79 151.842   1.922                     &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Note that the F-tests are exactly identical.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For &lt;code&gt;lme()&lt;/code&gt;, you used &lt;code&gt;list(id=pdDiag(~time))&lt;/code&gt;, which not only adds a random intercept to the model, but also a random slope. Moreover, by using &lt;code&gt;pdDiag&lt;/code&gt;, you are setting the correlation between the random intercept and slope to zero. This is a different model than what you specified via &lt;code&gt;aov()&lt;/code&gt; and hence you get different results.&lt;/p&gt;&#xA;" OwnerUserId="1934" LastActivityDate="2011-02-14T15:35:47.063" CommentCount="2" />
  <row Id="7200" PostTypeId="1" AcceptedAnswerId="7206" CreationDate="2011-02-14T15:43:39.727" Score="7" ViewCount="179" Body="&lt;p&gt;I know that an easy to handle formula for the CDF of a normal distribution is somewhat missing, due to the complicated error function in it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I wonder if there is a a nice formula for $N(c_{-} \leq x &amp;lt; c_{+}| \mu, \sigma^2)$. Or what the &quot;state of the art&quot; approximation for this problem might be.&lt;/p&gt;&#xA;" OwnerUserId="2860" LastActivityDate="2011-05-04T13:33:45.540" Title="Evaluate definite interval of normal distribution" Tags="&lt;normal-distribution&gt;&lt;approximation&gt;" AnswerCount="2" FavoriteCount="2" />
  <row Id="7201" PostTypeId="2" ParentId="7155" CreationDate="2011-02-14T16:36:53.473" Score="2" ViewCount="" Body="&lt;p&gt;I don't think it is possible to define an outlier without assuming a model of the underlying process giving rise to the data.  Without such a model we have no frame of reference to decide whether the data are anomalous or &quot;wrong&quot;.  The definition of an outlier that I have found useful is that an outlier is an observation (or observations) that cannot be reconciled to a model that otherwise performs well.  &lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2011-02-14T16:36:53.473" CommentCount="2" />
  <row Id="7202" PostTypeId="1" CreationDate="2011-02-14T16:40:18.977" Score="1" ViewCount="336" Body="&lt;h3&gt;Context:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;I have &quot;pooled data&quot; with time and cross section dimensions. &#xA;It is unbalanced data without a full range of time observations for each cross section of observations. &#xA;It is kind of like unbalanced panel data? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wish to proceed with the limited number of observations without losing any data.   &lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Questions:&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;How can I get regression results (generalized least squares?) in SPSS with this kind of data? &lt;/li&gt;&#xA;&lt;li&gt;What should I pay attention to when evaluating the results?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="3192" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-02-15T11:20:15.173" LastActivityDate="2011-02-18T22:55:09.213" Title="Regression with Pooled data in SPSS" Tags="&lt;regression&gt;&lt;spss&gt;&lt;panel-data&gt;" AnswerCount="2" />
  <row Id="7203" PostTypeId="2" ParentId="7196" CreationDate="2011-02-14T16:51:13.537" Score="0" ViewCount="" Body="&lt;p&gt;If you know what the correct vote should be, you could start with a simple chi-square or KS test, testing for the number of votes misclassified for each group (treatment and control) as compared to the gold standard.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-Ralph Winters&lt;/p&gt;&#xA;" OwnerUserId="3489" LastActivityDate="2011-02-14T16:51:13.537" />
  <row Id="7205" PostTypeId="2" ParentId="7202" CreationDate="2011-02-14T16:58:27.340" Score="1" ViewCount="" Body="&lt;p&gt;You should be able to weight by duration of observation, perhaps by simple division, in essence turning data into an annualized rates. There will be issues relating to whether persons (or other units of analysis) have systematically higher or lower rates in the early period of observation or enrollment.&lt;/p&gt;&#xA;" OwnerUserId="2129" LastActivityDate="2011-02-14T16:58:27.340" />
  <row Id="7206" PostTypeId="2" ParentId="7200" CreationDate="2011-02-14T17:01:06.277" Score="9" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;It depends on exactly what you are looking for&lt;/strong&gt;. Below are some brief details and references.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Much of the literature for approximations centers around the function&#xA;$$&#xD;&#xA;Q(x) = \int_x^\infty \frac{1}{\sqrt{2\pi}} e^{-\frac{u^2}{2}} \, \mathrm{d}u&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;for $x &amp;gt; 0$. This is because the function you provided can be decomposed as a simple difference of the function above (possibly adjusted by a constant). This function is referred to by many names, including &quot;upper-tail of the normal distribution&quot;, &quot;right normal integral&quot;, and &quot;Gaussian $Q$-function&quot;, to name a few. You'll also see approximations to &lt;strong&gt;Mills' ratio&lt;/strong&gt;, which is&#xA;$$&#xD;&#xA;R(x) = \frac{Q(x)}{\varphi(x)}&#xD;&#xA;$$&#xA;where $\varphi(x) = (2\pi)^{-1/2} e^{-x^2 / 2}$ is the Gaussian pdf.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here I list some references for various purposes that you might be interested in.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Computational&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The de-facto standard for computing the $Q$-function or the related complementary error function is&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;W. J. Cody, &lt;a href=&quot;http://www.ams.org/journals/mcom/1969-23-107/S0025-5718-1969-0247736-4/S0025-5718-1969-0247736-4.pdf&quot; rel=&quot;nofollow&quot;&gt;Rational Chebyshev Approximations for the Error Function&lt;/a&gt;, &lt;em&gt;Math. Comp.&lt;/em&gt;, 1969, pp. 631--637.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;&lt;strong&gt;Every&lt;/em&gt;&lt;/strong&gt; (self-respecting) implementation uses this paper. (MATLAB, R, etc.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&quot;Simple&quot; Approximations&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://people.math.sfu.ca/~cbm/aands/page_299.htm&quot; rel=&quot;nofollow&quot;&gt;Abramowitz and Stegun&lt;/a&gt; have one based on a polynomial expansion of a transformation of the input. Some people use it as a &quot;high-precision&quot; approximation. I don't like it for that purpose since it behaves badly around zero. For example, their approximation does &lt;strong&gt;not&lt;/strong&gt; yield $\hat{Q}(0) = 1/2$, which I think is a big no-no. Sometimes &lt;a href=&quot;http://www.codeplanet.eu/files/download/accuratecumnorm.pdf#page=5&quot; rel=&quot;nofollow&quot;&gt;bad things&lt;/a&gt; happen because of this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Borjesson and Sundberg give a simple approximation which works pretty well for most applications where one only requires a few digits of precision. The &lt;em&gt;absolute relative error&lt;/em&gt; is never worse than 1%, which is quite good considering its simplicity. The basic approximation is&#xA;$$&#xD;&#xA;\hat{Q}(x) = \frac{1}{(1-a) x + a \sqrt{x^2 + b}} \varphi(x)&#xD;&#xA;$$&#xA;and their preferred choices of the constants are $a = 0.339$ and $b = 5.51$. That reference is&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;P. O. Borjesson and C. E. Sundberg. Simple approximations of the error function Q(x) for communications&#xA;  applications. &lt;em&gt;IEEE Trans. Commun.&lt;/em&gt;, COM-27(3):639–643, March 1979.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Here is a plot of its absolute relative error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/KtzSn.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The electrical-engineering literature is awash with various such approximations and seem to take an overly intense interest in them. Many of them are poor though or expand to very strange and convoluted expressions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might also look at&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;W. Bryc. &lt;a href=&quot;http://math.uc.edu/~brycw/preprint/z-tail/z-tail.pdf&quot; rel=&quot;nofollow&quot;&gt;A uniform approximation to the right normal integral&lt;/a&gt;. &lt;em&gt;Applied Mathematics and Computation&lt;/em&gt;,&#xA;  127(2-3):365–374, April 2002.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Laplace's continued fraction&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Laplace has a beautiful continued fraction which yields successive upper and lower bounds for every value of $x &amp;gt; 0$. It is, in terms of Mills' ratio,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;R(x) = \frac{1}{x+}\frac{1}{x+}\frac{2}{x+}\frac{3}{x+}\cdots ,&#xD;&#xA;$$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;where the notation I've used is fairly standard for a continued fraction, i.e., $1/(x+1/(x+2/(x+3/x+\cdots)))$. This expression doesn't converge very fast for small $x$, though, and it diverges at $x = 0$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This continued fraction actually yields many of the &quot;simple&quot; bounds on $Q(x)$ that were &quot;rediscovered&quot; in the mid-to-late 1900s. It's easy to see that for a continued fraction in &quot;standard&quot; form (i.e., composed of positive integer coefficients), truncating the fraction at odd (even) terms gives an upper (lower) bound.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hence, Laplace tells us immediately that &#xA;$$&#xD;&#xA;\frac{x}{x^2 + 1} &amp;lt; R(x) &amp;lt; \frac{1}{x} \&amp;gt;,&#xD;&#xA;$$&#xA;both of which are bounds that were &quot;rediscovered&quot; in the mid-1900's. In terms of the $Q$-function, this is equivalent to&#xA;$$&#xD;&#xA;\frac{x}{x^2 + 1} \varphi(x) &amp;lt; Q(x) &amp;lt; \frac{1}{x} \varphi(x) .&#xD;&#xA;$$&#xA;An alternative proof of this using simple integration by parts can be found in S. Resnick, &lt;em&gt;Adventures in Stochastic Processes&lt;/em&gt;, Birkhauser, 1992, in Chapter 6 (Brownian motion).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Notice, in particular, that this immediately implies that $Q(x) \sim \varphi(x)/x$. This fact can be established using L'Hopital's rule as well. This also helps explain the choice of the functional form of the Borjesson-Sundberg approximation. Any choice of $a \in [0,1]$ maintains the asymptotic equivalence as $x \to \infty$. The parameter $b$ serves as a &quot;continuity correction&quot; near zero.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a plot of the $Q$-function and the two Laplace bounds.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/GwqjZ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;C-I. C. Lee has a paper from the early 1990's that does a &quot;correction&quot; for small values of $x$. See&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;C-I. C. Lee. &lt;a href=&quot;http://www.springerlink.com/content/r23221165q7h4344/&quot; rel=&quot;nofollow&quot;&gt;On Laplace continued fraction for the normal integral&lt;/a&gt;. Ann. Inst. Statist. Math., 44(1):107–120,&#xA;  March 1992.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Durrett's &lt;em&gt;Probability: Theory and Examples&lt;/em&gt; provides the classical upper and lower bounds on $Q(x)$ on pages 6&amp;ndash;7 of the 3rd edition. They're meant for larger values of $x$ (say, $x &gt; 3$) and are asymptotically tight.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hopefully this will get you started. If you have a more specific interest, I might be able to point you somewhere.&lt;/p&gt;&#xA;" OwnerUserId="2970" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-04-15T01:49:07.410" LastActivityDate="2011-04-15T01:49:07.410" CommentCount="2" />
  <row Id="7207" PostTypeId="1" AcceptedAnswerId="7210" CreationDate="2011-02-14T17:10:17.143" Score="5" ViewCount="294" Body="&lt;p&gt;I understand the formal differences between them, what I want to know is when it is more relevant to use one vs. the other. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Do they always provide complementary insight about the performance of a given classification/detection system? &lt;/li&gt;&#xA;&lt;li&gt;Is it reasonable to provide them both, say, in a paper? (when does it make sense to provide both of them instead of just one of them?).&lt;/li&gt;&#xA;&lt;li&gt;Are there any alternative (maybe more modern) descriptors that capture the relevant aspects of both ROC and precision recall for a classification system?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I am interested in arguments for both binary and multi-class (e.g. as one-vs-all) cases.&lt;/p&gt;&#xA;" OwnerUserId="2798" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-02-14T23:41:07.917" LastActivityDate="2011-02-14T23:41:07.917" Title="ROC vs precision-and-recall curves" Tags="&lt;machine-learning&gt;&lt;roc&gt;&lt;precision-recall&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="2" />
  <row Id="7208" PostTypeId="1" AcceptedAnswerId="7338" CreationDate="2011-02-14T17:16:52.580" Score="5" ViewCount="181" Body="&lt;p&gt;I am using &lt;a href=&quot;http://en.wikipedia.org/wiki/Cohen%27s_kappa&quot; rel=&quot;nofollow&quot;&gt;Cohen's Kappa&lt;/a&gt; to calculate the inter-agreement between two judges.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is calculated as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ \frac{P(A) - P(E)}{1 - P(E)}  $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $P(A)$ is the proportion of agreement and $P(E)$ the probability of agreement by chance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now for the following dataset, I get the expected results:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;User A judgements: &#xA;  - 1, true&#xA;  - 2, false&#xA;User B judgements: &#xA;  - 1, false&#xA;  - 2, false&#xA;Proportion agreed: 0.5&#xA;Agreement by chance: 0.625&#xA;Kappa for User A and B: -0.3333333333333333&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;We can see that both judges have not agreed very well. However in the following case where both judges evaluate one criteria, kappa evaluates to zero:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;User A judgements: &#xA;  - 1, false&#xA;User B judgements: &#xA;  - 1, false&#xA;Proportion agreed: 1.0&#xA;Agreement by chance: 1.0&#xA;Kappa for User A and B: 0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now I can see that the agreement by chance is obviously 1, which leads to kappa being zero, but does this count as a reliable result? The problem is that I normally don't have more than two judgements per criteria, so these will all never evaluate to any kappa greater than 0, which I think is not very representative.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Am I right with my calculations? Can I use a different method to calculate inter-agreement?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here we can see that kappa works fine for multiple judgements:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;User A judgements: &#xA;  - 1, false&#xA;  - 2, true&#xA;  - 3, false&#xA;  - 4, false&#xA;  - 5, true&#xA;User A judgements: &#xA;  - 1, true&#xA;  - 2, true&#xA;  - 3, false&#xA;  - 4, true&#xA;  - 5, false&#xA;Proportion agreed: 0.4&#xA;Agreement by chance: 0.5&#xA;Kappa for User A and B: -0.19999999999999996&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1205" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-02-17T22:01:30.213" LastActivityDate="2011-02-18T01:26:44.263" Title="Can one use Cohen's Kappa for two judgements only?" Tags="&lt;reliability&gt;&lt;information-retrieval&gt;" AnswerCount="2" CommentCount="6" FavoriteCount="3" />
  <row Id="7209" PostTypeId="1" CreationDate="2011-02-14T17:40:19.597" Score="4" ViewCount="56" Body="&lt;p&gt;My aim is to predict quarterly customer-default probabilities: I have data on ~ 2 million individuals, who default on average with a probability of ~ 0.3 percent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore I am thinking about undersampling the majority class (non-defaults) to save computing-time (kernel methods can be quite costly, I know about correcting the probabability predictions) -- the other option being just taking a sub-sample of the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What do you think would be a good ratio of defaults to non-defaults in my learning sample?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for your help!&lt;/p&gt;&#xA;" OwnerUserId="2549" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-02-14T18:40:11.043" LastActivityDate="2011-02-14T23:36:54.893" Title="How much undersampling should be done?" Tags="&lt;sampling&gt;" AnswerCount="1" />
  <row Id="7210" PostTypeId="2" ParentId="7207" CreationDate="2011-02-14T18:11:52.737" Score="5" ViewCount="" Body="&lt;p&gt;The key difference is that ROC curves will be the same no matter what the baseline probability is, but PR curves may be more useful in practice for needle-in-haystack type problems or problems where the &quot;positive&quot; class is more interesting than the negative class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To show this, first let's start with a very nice way to define precision, recall and specificity.  Assume you have a &quot;positive&quot; class called 1 and a &quot;negative&quot; class called 0.  $\hat{Y}$ is your estimate of the true class label $Y$.  Then:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Precision = P(Y = 1 | \hat{Y} = 1)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Recall = Sensitivity = P(\hat{Y} = 1 | Y = 1)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Specificity = P(\hat{Y} = 0 | Y = 0)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The key thing to note is that sensitivity/recall and specificity, which make up the ROC curve, are probabilities &lt;em&gt;conditioned on the true class label&lt;/em&gt;.  Therefore, they will be the same regardless of what $P(Y = 1)$ is.  Precision is a probability conditioned on &lt;em&gt;your estimate of the class label&lt;/em&gt; and will thus vary if you try your classifier in different populations with different baseline $P(Y = 1)$.  However, it may be more useful in practice is you only care about one population with known background probability and the &quot;positive&quot; class is much more interesting than the &quot;negative&quot; class.  (IIRC precision is popular in the document retrieval field, where this is the case.)  This is because it directly answers the question, &quot;What is the probability that this is a real hit given my classifier says it is?&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Interestingly, by Bayes' theorem you can work out cases where sensitivity can be very high and precision very low simultaneously.  All you have to do is assume $P(Y = 1)$ is very close to zero.  In practice I've developed several classifiers with this performance characteristic when searching for needles in DNA sequence haystacks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;IMHO when writing a paper you should provide whichever curve answers the question you want answered (or whichever one is more favorable to your method, if you're cynical).  If your question is, &quot;How meaningful is a positive result from my classifier &lt;em&gt;given the baseline probabilities of my problem&lt;/em&gt;?&quot;, use a PR curve.  If your question is, &quot;How well can this classifier be expected to perform &lt;em&gt;in general, at a variety of different baseline probabilities&lt;/em&gt;?&quot;, go with a ROC curve.&lt;/p&gt;&#xA;" OwnerUserId="1347" LastEditorUserId="1347" LastEditorDisplayName="" LastEditDate="2011-02-14T18:17:00.953" LastActivityDate="2011-02-14T18:17:00.953" CommentCount="1" />
  <row Id="7211" PostTypeId="1" CreationDate="2011-02-14T19:29:14.993" Score="5" ViewCount="312" Body="&lt;p&gt;Can you recommend a text mining package in R that can be used against large volumes of data?&lt;br&gt;&#xA;Secondly, is there a GUI available for any of the text mining packages in R?&lt;br&gt;&#xA;Thirdly, is there another open source text mining program that is easy and intuitive to use?&lt;/p&gt;&#xA;" OwnerUserId="2715" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-02-14T23:38:31.837" LastActivityDate="2011-02-21T19:26:03.743" Title="What are the text-mining packages for R and are there other open source text-mining programs?" Tags="&lt;r&gt;&lt;text-mining&gt;" AnswerCount="6" CommentCount="1" FavoriteCount="5" />
  <row Id="7212" PostTypeId="2" ParentId="7211" CreationDate="2011-02-14T19:35:54.927" Score="2" ViewCount="" Body="&lt;p&gt;See the &lt;a href=&quot;http://tm.r-forge.r-project.org/&quot; rel=&quot;nofollow&quot;&gt;tm package&lt;/a&gt; and &lt;a href=&quot;http://www.rinfinance.com/agenda/2010/Theussl+Feinerer+Hornik.pdf&quot; rel=&quot;nofollow&quot;&gt;this presentation&lt;/a&gt; by &lt;a href=&quot;http://statmath.wu-wien.ac.at/~theussl/&quot; rel=&quot;nofollow&quot;&gt;Stefan Theussl&lt;/a&gt; given at the 2010 &lt;a href=&quot;http://www.rinfinance.com&quot; rel=&quot;nofollow&quot;&gt;R/Finance conference&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1657" LastActivityDate="2011-02-14T19:35:54.927" />
  <row Id="7213" PostTypeId="2" ParentId="7165" CreationDate="2011-02-14T20:08:32.530" Score="2" ViewCount="" Body="&lt;p&gt;I think Harvey Motulsky's Intuitive Biostatistics is pretty good for non-mathematical &quot;intuitive&quot; explanations to basic statistical methods most commonly employed in the biological and medical sciences. &lt;/p&gt;&#xA;" OwnerUserId="3183" LastActivityDate="2011-02-14T20:08:32.530" />
  <row Id="7214" PostTypeId="2" ParentId="7211" CreationDate="2011-02-14T20:50:04.587" Score="7" ViewCount="" Body="&lt;p&gt;Please see the &lt;a href=&quot;http://cran.r-project.org/web/views/NaturalLanguageProcessing.html&quot; rel=&quot;nofollow&quot;&gt;CRAN Task View on Natural Language Processing&lt;/a&gt; which includes, among others, the &lt;a href=&quot;http://cran.r-project.org/package=tm&quot; rel=&quot;nofollow&quot;&gt;tm&lt;/a&gt; package already mentioned by Josh.&lt;/p&gt;&#xA;" OwnerUserId="334" LastActivityDate="2011-02-14T20:50:04.587" />
  <row Id="7215" PostTypeId="6" CreationDate="2011-02-14T21:19:12.923" Score="0" ViewCount="" Body="&lt;p&gt;I have been one of the pro tem moderators and I would like to continue my mission as a regular mod.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, if you like what I was doing here (compulsive edits, protecting wide scope, journal club), vote for me!&lt;/p&gt;&#xA;" OwnerUserId="88" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-02-14T21:19:12.923" LastActivityDate="2011-02-14T21:19:12.923" CommunityOwnedDate="2011-02-14T21:19:12.923" CommentCount="2" />
  <row Id="7216" PostTypeId="2" ParentId="7211" CreationDate="2011-02-14T21:24:14.247" Score="2" ViewCount="" Body="&lt;p&gt;Sure, RapidMiner with the text mining extension.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are many videos that show how it is done.&lt;/p&gt;&#xA;" OwnerUserId="3196" LastActivityDate="2011-02-14T21:24:14.247" CommentCount="1" />
  <row Id="7217" PostTypeId="2" ParentId="7211" CreationDate="2011-02-14T21:44:54.347" Score="1" ViewCount="" Body="&lt;p&gt;GATE is very comprehensive.  It also allows you to work with different languages and has an ontology editor.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-Ralph Winters&lt;/p&gt;&#xA;" OwnerUserId="3489" LastActivityDate="2011-02-14T21:44:54.347" />
  <row Id="7218" PostTypeId="1" AcceptedAnswerId="7221" CreationDate="2011-02-14T22:05:32.307" Score="6" ViewCount="173" Body="&lt;p&gt;Suppose I have a large population of data points $(x,y)$ and that the Pearson's correlation is &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\textrm{corr}(X,Y) = \rho$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What can I reasonably say about the correlation I expect to observe in a sample of size $n$?  If the sample correlation is $\rho_s$, roughly what is the spread is $\rho_s$?  Is $\rho_s$ biased?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we make some assumptions like normality, can we compute the exact likelihood function of $\rho_s$ as a function of $\rho$?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Ultimately, I am wondering about the problem of whether an observed high correlation is a fluke or not, and all I have is the sample size and the correlation.)&lt;/p&gt;&#xA;" OwnerUserId="2665" LastEditorUserId="2665" LastEditorDisplayName="" LastEditDate="2011-02-16T10:29:54.563" LastActivityDate="2011-02-16T10:29:54.563" Title="Distribution of sample correlation" Tags="&lt;correlation&gt;" AnswerCount="1" CommentCount="5" />
  <row Id="7219" PostTypeId="2" ParentId="6" CreationDate="2011-02-14T22:09:41.657" Score="6" ViewCount="" Body="&lt;p&gt;The biggest difference I see between the communities is that statistics emphasizes inference, whereas machine learning emphasized prediction.  When you do statistics, you want to infer &lt;em&gt;the process by which data you have was generated.&lt;/em&gt;  When you do machine learning, you want to know how you can &lt;em&gt;predict what future data will look like w.r.t. some variable.&lt;/em&gt;  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course the two overlap.  Knowing how the data was generated will give you some hints about what a good predictor would be, for example.  However, one example of the difference is that machine learning has dealt with the p &gt;&gt; n problem (more features/variables than training samples) since its infancy, whereas statistics is just starting to get serious about this problem.  Why?  Because you can still make good predictions when p &gt;&gt; n, but you can't make very good inferences about what variables are actually important and why.&lt;/p&gt;&#xA;" OwnerUserId="1347" LastActivityDate="2011-02-14T22:09:41.657" CommunityOwnedDate="2011-02-14T22:09:41.657" CommentCount="5" />
  <row Id="7220" PostTypeId="2" ParentId="770" CreationDate="2011-02-14T22:29:44.980" Score="2" ViewCount="" Body="&lt;p&gt;Depends on what you mean.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've gotten deeper and deeper into statistics based on exposure to machine learning. (I had previously been more of a general AI guy, and hadn't had good experience with statistics, but gained greater understanding and appreciation of statistics as time has gone on.) So it's certainly a useful gateway.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've been mostly self-taught in statistics, however, and that leaves a lot of holes. I understand and use some fairly advanced techniques, but I wouldn't have to get too far off of the path to get stymied, while someone with a firmer statistical foundation would not. And I think that would apply in your scenario of using Machine Learning as the way to learn statistics: your knowledge would be fragile.&lt;/p&gt;&#xA;" OwnerUserId="1764" LastEditorUserId="1764" LastEditorDisplayName="" LastEditDate="2011-02-14T23:04:02.267" LastActivityDate="2011-02-14T23:04:02.267" />
  <row Id="7221" PostTypeId="2" ParentId="7218" CreationDate="2011-02-14T22:34:14.100" Score="6" ViewCount="" Body="&lt;p&gt;To quote the &lt;a href=&quot;http://en.wikipedia.org/wiki/Fisher_transformation&quot; rel=&quot;nofollow&quot;&gt;Wikipedia article on the Fisher transformation&lt;/a&gt; :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If $(X, Y)$ has a bivariate normal distribution, and if the $(X_i, Y_i)$ pairs used to form the sample correlation coefficient $r$ are independent for $i = 1, \ldots, n,$ then $$z = {1 \over 2}\ln{1+r \over 1-r} = \operatorname{arctanh}(r)$$ is approximately normally distributed with mean&#xA;${1 \over 2}\ln{{1+\rho} \over {1-\rho}},$&#xA;and standard error&#xA;${1 \over \sqrt{N-3}},$&#xA;where $N$ is the sample size.&lt;/p&gt;&#xA;" OwnerUserId="449" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-02-15T08:55:41.067" LastActivityDate="2011-02-15T08:55:41.067" CommentCount="3" />
  <row Id="7222" PostTypeId="2" ParentId="7209" CreationDate="2011-02-14T23:36:54.893" Score="2" ViewCount="" Body="&lt;p&gt;In a first approximation, 1:1 is a good proportion, but:  &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Some methods are more vulnerable to unequal classes, some are less -- plain decision tree will almost always vote for a much larger class, 1-NN will be not affected at all. It is a good idea to check this out (in literature or by asking here) in context of your problem. &lt;/li&gt;&#xA;&lt;li&gt;You should beware of possible inhomogeneities (let's say hidden &quot;subclasses&quot;) -- subsampling may change the proportions between subclasses and thus lead to strange effects. To this end, it is good to at least try few subsampling realizations to have a change to spot possible problems. (Well, I called it a problem, but this may be also a quite enlightening insight in data)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="88" LastActivityDate="2011-02-14T23:36:54.893" CommentCount="1" />
  <row Id="7223" PostTypeId="1" CreationDate="2011-02-14T23:51:15.593" Score="5" ViewCount="202" Body="&lt;p&gt;I would like to pose this question in two parts. Both deal with a generalized linear model, but the first deals with model selection and the other deals with regularization.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;b&gt;Background:&lt;/b&gt; I utilize GLMs (linear, logistic, gamma regression) models for both prediction and for description. When I refer to the &quot;&lt;i&gt;normal things one does with a regression&lt;/i&gt;&quot; I largely mean description with (i) confidence intervals around coefficients, (ii) confidence intervals around predictions and (iii) hypothesis tests concerning linear combinations of the coefficients such as &quot;is there a difference between treatment A and treatment B?&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you legitimately lose the ability to do these things using the normal theory under each of the following? And if so, are these things really only good for models used for pure prediction?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;b&gt;I.&lt;/b&gt; When a GLM has been fit via some model selection process (for concreteness say its a stepwise procedure based on AIC).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;b&gt;II.&lt;/b&gt; When a GLM has been fit via a regularization method (say using glmnet in R).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My sense is that for I. the answer is technically that you should use a bootstrap for the &quot;&lt;i&gt;normal things one does with a regression&lt;/i&gt;&quot; but no one really abides by that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;b&gt;Add:&lt;/b&gt;&lt;br/&gt;&#xA;After getting a few responses and reading elsewhere, here is my take on this (for anyone else benefit as well as to receive correction).&lt;br/&gt;&lt;br/&gt;&#xA;&lt;b&gt;I.&lt;/b&gt; &lt;br/&gt;&#xA;&lt;b&gt;A) RE: Error Generalize.&lt;/b&gt; In order to generalize error rates on new data, when there is no hold out set, cross validation can work but you need to repeat the process completely for each fold - using nested loops - thus any feature selection, parameter tuning, etc. must be done independently each time. This idea should hold for any modeling effort (including penalized methods).&lt;br/&gt;&lt;br/&gt;&#xA;&lt;b&gt;B) RE: Hypothesis testing and confidence intervals of GLM.&lt;/b&gt; When using model selection (feature selection, parameter tuning, variable selection) for a generalized linear model and a hold out set exists, it is permissible to train the model on a partition and then fit the model on the remaining data or the full data set and use that model/data to perform hypothesis tests etc. If a hold out set does not exist, a bootstrap can be used, as long as the full process is repeated for each bootstrap sample. This limits the hypothesis tests that can be done though since perhaps a variable will not always be selected for example.&lt;br/&gt;&lt;br/&gt;&#xA;&lt;b&gt;C) RE: Not carrying about prediction on future data sets&lt;/b&gt;, then fit a purposeful model guided by theory and a few hypothesis tests and even consider leaving all  variables in the model (significant or not) (along the lines of Hosmer and Lemeshow). This is small variable set classical type of regression modeling and then allows the use of CI's and hypothesis test.&lt;br/&gt;&lt;br/&gt;&#xA;&lt;b&gt;D) RE: Penalized regression.&lt;/b&gt; No advice, perhaps consider this suitable for prediction only (or as a type of feature selection to then apply to another data set as in B above) as the bias introduced makes CI's and hypothesis tests unwise - even with the bootstrap.&lt;/p&gt;&#xA;" OwnerUserId="2040" LastEditorUserId="2040" LastEditorDisplayName="" LastEditDate="2011-02-20T17:04:06.420" LastActivityDate="2011-02-20T17:04:06.420" Title="GLM after model selection or regularization" Tags="&lt;regression&gt;&lt;model-selection&gt;&lt;regularization&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="7224" PostTypeId="1" CreationDate="2011-02-14T22:41:09.187" Score="64" ViewCount="11638" Body="&lt;p&gt;I'm working on a little project involving the faces of twitter users via their profile pictures.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A problem I've encountered is that after I filter out all but the images that are clear portrait photos, a small but significant percentage of twitter users use a picture of Justin Bieber as their profile picture.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to filter them out, how can I tell programmatically whether a picture is that of Justin Bieber?&lt;/p&gt;&#xA;" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-02-15T15:22:25.870" LastActivityDate="2011-05-11T12:53:49.673" Title="Detecting a given face in a database of facial images" Tags="&lt;machine-learning&gt;&lt;clustering&gt;&lt;image-processing&gt;" AnswerCount="7" CommentCount="10" FavoriteCount="12" />
  <row Id="7225" PostTypeId="1" CreationDate="2011-02-15T00:38:52.977" Score="6" ViewCount="348" Body="&lt;p&gt;I am using the R package &lt;em&gt;penalized&lt;/em&gt; to obtain shrunken estimates of coefficients for a dataset where I have lots of predictors and little knowledge of which ones are important. After I've picked tuning parameters L1 and L2 and I'm satisfied with my coefficients, is there a statistically sound way to summarize the model fit with something like R-squared? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furthermore, I'm interested in testing the overall significance of the model (i.e. does R²=0, or do all the =0). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've read through the answers on a similar question asked &lt;a href=&quot;http://stats.stackexchange.com/questions/2121/how-can-i-estimate-coefficient-standard-errors-when-using-ridge-regression&quot;&gt;here&lt;/a&gt;, but it didn't quite answer my question. There's an excellent tutorial on the R package that I'm using &lt;a href=&quot;http://cran.r-project.org/web/packages/penalized/vignettes/penalized.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;, and the author Jelle Goeman had the following note at the end of the tutorial regarding confidence intervals from penalized regression models:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;It is a very natural question to ask for standard errors of regression coefficients or other estimated quantities. In principle such standard errors can easily be calculated, e.g. using the bootstrap.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Still, this package deliberately does not provide them. The reason for this is that standard errors are not very meaningful for strongly biased estimates such as arise from penalized estimation methods. Penalized estimation is a procedure that reduces the variance of estimators by introducing substantial bias. The bias of each estimator is therefore a major component of its mean squared error, whereas its variance may contribute only a small part.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Unfortunately, in most applications of penalized regression it is impossible to obtain a sufficiently precise estimate of the bias. Any bootstrap-based cal- culations can only give an assessment of the variance of the estimates. Reliable estimates of the bias are only available if reliable unbiased estimates are available, which is typically not the case in situations in which penalized estimates are used.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Reporting a standard error of a penalized estimate therefore tells only part of the story. It can give a mistaken impression of great precision, completely ignoring the inaccuracy caused by the bias. It is certainly a mistake to make confidence statements that are only based on an assessment of the variance of the estimates, such as bootstrap-based confidence intervals do.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="36" LastActivityDate="2011-05-17T14:47:06.857" Title="Estimating R-squared and statistical significance from penalized regression model" Tags="&lt;regression&gt;&lt;lasso&gt;&lt;stepwise-regression&gt;&lt;ridge-regression&gt;" AnswerCount="1" CommentCount="7" />
  <row Id="7226" PostTypeId="2" ParentId="7197" CreationDate="2011-02-15T00:58:32.400" Score="0" ViewCount="" Body="&lt;p&gt;@Jim,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given that you're comparing n groups (n &gt; 2) and that your count data are, as you say, likely non-normally distributed, I wonder if perhaps applying Kruskal–Wallis analysis of variance to your data might be an option.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;--Jeff&lt;/p&gt;&#xA;" OwnerUserId="2730" LastActivityDate="2011-02-15T00:58:32.400" />
  <row Id="7227" PostTypeId="2" ParentId="7224" CreationDate="2011-02-15T01:03:52.237" Score="22" ViewCount="" Body="&lt;p&gt;A better idea might be to trash all images that appear in the feed of more than one user - no recognition needed.&lt;/p&gt;&#xA;" OwnerUserId="3203" LastActivityDate="2011-02-15T01:03:52.237" CommentCount="4" />
  <row Id="7228" PostTypeId="2" ParentId="7128" CreationDate="2011-02-15T01:14:20.777" Score="0" ViewCount="" Body="&lt;p&gt;A few thoughts&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;you might want to rearrange your data frame either in wide (rows are cases and columns are ratings for stimuli pairs) or long format (rows are case by stimuli pair combinations). &#xA;Then you could run mixed models &lt;code&gt;lme4&lt;/code&gt; or perhaps &lt;code&gt;aov&lt;/code&gt; (see here for &lt;a href=&quot;http://cran.r-project.org/doc/contrib/Baron-rpsych.pdf&quot; rel=&quot;nofollow&quot;&gt;info on different ANOVAs&lt;/a&gt;).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;You might find useful the tutorial on &lt;a href=&quot;http://www.statmethods.net/advstats/mds.html&quot; rel=&quot;nofollow&quot;&gt;multidimensional scaling on the Quick-R website&lt;/a&gt;.&#xA;It shows how to prepare the data and produce a plot.&#xA;However, from what I understand of your data, you may have to transform your ratings. At the moment, higher scores seem to indicate greater similarity, whereas MDS will expect that higher scores mean greater dissimilarity.&#xA;At a basic level you could use (9 minus rating).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-02-16T01:26:19.933" LastActivityDate="2011-02-16T01:26:19.933" CommentCount="1" />
  <row Id="7229" PostTypeId="2" ParentId="7224" CreationDate="2011-02-15T01:19:42.080" Score="10" ViewCount="" Body="&lt;p&gt;I have a feeling that &lt;a href=&quot;http://www.tineye.com/commercial_api&quot; rel=&quot;nofollow&quot;&gt;http://www.tineye.com/commercial_api&lt;/a&gt; may be the solution here.&#xA;Simply throw the Twitter profile image to Tineye, see if it returns images (and associated URLs) that can clearly be identified (or automatically scored using simple word-count logic) as being related to (or of) that little sack of &lt;em&gt;*&lt;/em&gt;*.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Simples!&lt;/p&gt;&#xA;" OwnerUserId="3205" LastActivityDate="2011-02-15T01:19:42.080" />
  <row Id="7231" PostTypeId="2" ParentId="7224" CreationDate="2011-02-15T02:28:19.450" Score="1" ViewCount="" Body="&lt;p&gt;You could try &lt;a href=&quot;http://en.wikipedia.org/wiki/Locality_sensitive_hashing&quot; rel=&quot;nofollow&quot;&gt;locality sensitive hashing&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="900" LastActivityDate="2011-02-15T02:28:19.450" CommentCount="1" />
  <row Id="7232" PostTypeId="2" ParentId="7224" CreationDate="2011-02-15T02:52:31.463" Score="7" ViewCount="" Body="&lt;p&gt;Since you are able to filter to only those that are clear portrait photos, I'm assuming you have some method of feature generation to transform the raw images into features that are useful for machine learning purposes.  If that's true, you could try to train a classification algorithm (there are lots of them: neural networks, etc.) by feeding the algorithm a bunch of known Bieber photos as well as a bunch of known non-Biebers.  Once you have trained the model, it could be used to predict whether a new image is Bieber or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This sort of supervised learning technique does require you to have data where you know the correct answer (Bieber or not), but those could probably be found from a Google image search.  It also requires that you have the right sorts of features, and I don't know enough about image processing or your algorithm to know if that is a major drawback.&lt;/p&gt;&#xA;" OwnerUserId="2485" LastActivityDate="2011-02-15T02:52:31.463" CommentCount="4" />
  <row Id="7233" PostTypeId="1" AcceptedAnswerId="7243" CreationDate="2011-02-15T03:05:32.217" Score="0" ViewCount="180" Body="&lt;p&gt;How to interpret residuals in random and fixed effects models? I am a medical doctor. Kindly explain in simple words.&#xA;&lt;strong&gt;Updated&lt;/strong&gt; Here is an example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Weight (Fixed)      Weight (Random)    Residual (Fixed)      Residual (Random)&#xA;Relative weight    Relative weight     Std Residual         Std Residual&#xA;&#xA;32.27                21.56               -4.61                -1.62&#xA;25.14                21.02               -0.41                -0.46&#xA;18.52                20.21               2.16                  0.52&#xA;14.39                19.39               2.10                  0.62&#xA; 9.68                17.81               2.56                  1.04&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2956" LastEditorUserId="2956" LastEditorDisplayName="" LastEditDate="2011-02-15T04:40:52.997" LastActivityDate="2011-02-15T14:29:12.967" Title="Interpret residuals in random and fixed effects models" Tags="&lt;meta-analysis&gt;&lt;random-effects-model&gt;&lt;fixed-effects-model&gt;&lt;residuals&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="7234" PostTypeId="2" ParentId="7224" CreationDate="2011-02-15T03:30:32.663" Score="3" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://face.com/&quot; rel=&quot;nofollow&quot;&gt;http://face.com/&lt;/a&gt; has free a facial recognition API&lt;/p&gt;&#xA;" OwnerUserId="2973" LastActivityDate="2011-02-15T03:30:32.663" />
  <row Id="7235" PostTypeId="2" ParentId="7223" CreationDate="2011-02-15T03:40:52.463" Score="3" ViewCount="" Body="&lt;p&gt;You might check out David Freedman's paper, &quot;&lt;a href=&quot;http://www.psych.umn.edu/faculty/waller/classes/mult11/readings/freedman83.pdf&quot; rel=&quot;nofollow&quot;&gt;A Note on Screening Regression Equations.&lt;/a&gt;&quot; (ungated) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using completely uncorrelated data in a simulation, he shows that, if there are many predictors relative to the number of observations, then a standard screening procedure will produce a final regression that contains many (more than by chance) significant predictors and a highly significant F statistic. The final model suggests that it is effective at predicting the outcome, but this success is spurious. He also illustrates these results using asymptotic calculations. Suggested solutions include screening on a sample and assessing the model on the full data set and using at least an order of magnitude more observations than predictors.&lt;/p&gt;&#xA;" OwnerUserId="401" LastActivityDate="2011-02-15T03:40:52.463" CommentCount="6" />
  <row Id="7236" PostTypeId="1" CreationDate="2011-02-15T04:22:10.493" Score="6" ViewCount="165" Body="&lt;p&gt;I've got a dataset similar to this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pat_id   epis Care Type&#xA;1       1722650 Acute Care&#xA;1       1723120 Rehabilitation care&#xA;2       1584309 Acute Care&#xA;2       1585705 Rehabilitation care&#xA;3       1726487 GEM&#xA;3       1664031 Acute Care&#xA;3       1726488 Acute Care&#xA;3       1726489 Rehabilitation care&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Each patient has multiple &quot;episodes/care types&quot;.  I want to sample 50 patients but also approximately proportional to the &quot;Care Type&quot; from the population (say, 50% acute care, 30% rehabilitation care, 20% GEM).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One way I thought of doing is to split the data set into say &quot;acute&quot;, &quot;rehab&quot; and &quot;gem&quot; then sample 25 from &quot;acute&quot;, 15 from &quot;rehab&quot; and 10 from &quot;gem&quot;.  But there would be an overlap using this approach &quot;e.g. Patient 1375 would be in gem, acute and rehab.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a R package that would handle this sort of sampling easily?&lt;/p&gt;&#xA;" OwnerUserId="3227" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2011-02-15T18:21:57.053" LastActivityDate="2011-05-18T13:47:38.917" Title="Multistage Sampling in R" Tags="&lt;r&gt;&lt;sampling&gt;" AnswerCount="3" FavoriteCount="1" />
  <row Id="7238" PostTypeId="1" CreationDate="2011-02-15T05:53:31.987" Score="2" ViewCount="240" Body="&lt;p&gt;I have two vectors of matching lengths. They are readings from two different sensors (one is from a smartphone and the other is from a wiimote) of the same hand movement. I am trying to find the time offset between them to synchronise the readings for further processing. The readings I get are of the format (Time(ms) Value) for accelerations in the X,Y and Z direction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the synchronization, I plotted the cross-correlation function &lt;code&gt;xcorr2()&lt;/code&gt; between the two sets. I am getting the same graph (a weird triangle peak and a straight line at the bottom) for Accelerations along the x, y and z directions (which I guess is good) but I don't know how to interpret it. What do the axes in the graph represent?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone explain to me what &lt;code&gt;xcorr2()&lt;/code&gt; means in a qualitative sense. From the correlation function, how do I determine the offset (i.e. how many seconds is sensor1 behind sensor2)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Imelza&lt;/p&gt;&#xA;" OwnerUserId="3233" LastActivityDate="2011-02-15T05:53:31.987" Title="MATLAB interpretation of Xcorr2 - Cross Correlation function" Tags="&lt;data-visualization&gt;&lt;correlation&gt;&lt;matlab&gt;&lt;interpretation&gt;&lt;cross-correlation&gt;" CommentCount="1" />
  <row Id="7239" PostTypeId="1" AcceptedAnswerId="7264" CreationDate="2011-02-15T08:40:40.117" Score="1" ViewCount="158" Body="&lt;p&gt;I am fitting a simple linear regression model with R &amp;amp; JAGS:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model {&#xA;    for (i in 1:length(ri)){&#xA;        ri[i] ~ dnorm(mu[i],tau)&#xA;        mu[i] &amp;lt;- alpha + b.vr*vr[i] + b.ir*ir[i]&#xA;    }&#xA;&#xA;    #posterior predictive distribution&#xA;    for (j in 1:length(pvr)){&#xA;        pmu[j] &amp;lt;- alpha + b.vr*pvr[j] + b.ir*pir[j]&#xA;        pri[j] ~ dnorm(pmu[j],tau)&#xA;     }  &#xA;&#xA;    #priors for regression&#xA;    alpha ~ dnorm(0,0.01)&#xA;    b.vr ~ dunif(-10,0)&#xA;    b.ir ~ dunif(0,10)&#xA;&#xA;    #prior for precision&#xA;    tau ~ dgamma(0.001,0.001)&#xA;    sigma &amp;lt;- 1/sqrt(tau)&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am trying to calculate the posterior predictive distribution with new data (&lt;code&gt;pvr&lt;/code&gt; and &lt;code&gt;pir&lt;/code&gt; in the model). But somehow the reported results (&lt;code&gt;pri&lt;/code&gt;) do not make sense (the means of &lt;code&gt;pri&lt;/code&gt; are smaller than expected). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could someone explain me, is something wrong with the model specification?&lt;/p&gt;&#xA;" OwnerUserId="1443" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2011-02-15T18:23:17.730" LastActivityDate="2011-02-18T21:58:49.337" Title="JAGS: posterior predictive distribution" Tags="&lt;mcmc&gt;&lt;bugs&gt;&lt;gibbs&gt;&lt;jags&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="7240" PostTypeId="1" CreationDate="2011-02-15T09:11:10.857" Score="2" ViewCount="209" Body="&lt;p&gt;I do not know if this has been asked before, but I do not found anything about it. My question is if anyone can provide a good reference to learn how to obtain the proportion of variance explained by each one of the fixed and random factors in a mixed-effects model.&lt;/p&gt;&#xA;" OwnerUserId="221" LastActivityDate="2011-02-15T11:12:47.363" Title="Proportion of explained variance in a mixed-effects model" Tags="&lt;mixed-model&gt;&lt;variance&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="1" />
  <row Id="7241" PostTypeId="2" ParentId="7240" CreationDate="2011-02-15T10:27:16.157" Score="3" ViewCount="" Body="&lt;p&gt;I can provide some references:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Xu, R. (2003). Measuring explained variation in linear mixed effects models. &lt;em&gt;Statistics in Medicine&lt;/em&gt;, &lt;strong&gt;22&lt;/strong&gt;, 3527-3541. &lt;a href=&quot;http://dx.doi.org/10.1002/sim.1572&quot; rel=&quot;nofollow&quot;&gt;DOI:10.1002/sim.1572&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edwards, L. J., Muller, K. E., Wolfinger, R. D., Qaqish, B. F., &amp;amp; Schabenberger, O. (2008). An $R^2$ statistic for fixed effects in the linear mixed model. &lt;em&gt;Statistics in Medicine&lt;/em&gt;, &lt;strong&gt;27&lt;/strong&gt;, 6137-6157. &lt;a href=&quot;http://dx.doi.org/10.1002/sim.3429&quot; rel=&quot;nofollow&quot;&gt;DOI:10.1002/sim.3429&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hössjer, O. (2008). On the coefficient of determination for mixed regression models. &lt;em&gt;Journal of Statistical Planning and Inference&lt;/em&gt;, &lt;strong&gt;138&lt;/strong&gt;, 3022-3038. &lt;a href=&quot;http://dx.doi.org/10.1016/j.jspi.2007.11.010&quot; rel=&quot;nofollow&quot;&gt;DOI:10.1016/j.jspi.2007.11.010&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Happy reading!&lt;/p&gt;&#xA;" OwnerUserId="1934" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-02-15T11:12:47.363" LastActivityDate="2011-02-15T11:12:47.363" CommentCount="1" />
  <row Id="7243" PostTypeId="2" ParentId="7233" CreationDate="2011-02-15T13:01:56.817" Score="3" ViewCount="" Body="&lt;p&gt;As @suncooolsu has already pointed out, we need more information. Residuals are often used when it comes to identifying outliers and to gain a better understanding of the appropriateness of a certain model (here: FEM vs REM). Given your &quot;Std Residual&quot;, I would say that a REM seems more appropriate. However, you definitely might want to check other heterogeneity tests (e.g., Q, I^2). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Recently Viechtbauer/Cheung (2010) have published a really nice article on &quot;&lt;a href=&quot;http://onlinelibrary.wiley.com/doi/10.1002/jrsm.11/abstract&quot; rel=&quot;nofollow&quot;&gt;Outlier and influence diagnostics for meta-analysis&lt;/a&gt;&quot; (see esp. section 3).&lt;/p&gt;&#xA;" OwnerUserId="307" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-02-15T14:29:12.967" LastActivityDate="2011-02-15T14:29:12.967" CommentCount="4" />
  <row Id="7244" PostTypeId="1" AcceptedAnswerId="7245" CreationDate="2011-02-15T13:48:07.757" Score="6" ViewCount="187" Body="&lt;p&gt;Lets say I have a website which gets 100 hits per day (mu = 100). Yesterday my website got 130 hits (x = 130). If I assume a Poisson distribution, then the probability of getting 130 hits is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; dpois(130, 100)&#xA;[1] 0.0005752527 # about 0.06%&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So this tells me that getting 130 hits is quite unusual for my website due to the low probability.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My understanding of statistical significance is that it is used to determine whether the outcome of an experiment is due either to chance or some kind of deterministic relationship. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;How would I apply that in this situation?&lt;/li&gt;&#xA;&lt;li&gt;What test should one use? (and is it in R?)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Many thanks in advance for your time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: I saw someone at a business talk asked something very similar to this and I had no idea what they meant by it, and so now I'm just trying to educate myself. I'm new to R, but that seems like the software most used for these kind of questions, hence my request.&lt;/p&gt;&#xA;" OwnerUserId="3253" LastActivityDate="2011-02-15T15:35:20.957" Title="Poisson distribution and statistical significance" Tags="&lt;r&gt;&lt;distributions&gt;&lt;statistical-significance&gt;&lt;poisson&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="7245" PostTypeId="2" ParentId="7244" CreationDate="2011-02-15T14:15:43.147" Score="5" ViewCount="" Body="&lt;p&gt;There are two points to make:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;It is not the specific value of 130 that is unusual, but that it is much larger than 100. If you got more than 130 hits, that would have been even more surprising. So we usually look at the P(X&gt;=130), not just P(X=130). By your logic even 100 hits would be unusual, because &lt;code&gt;dpois(100,100)=0.04&lt;/code&gt;. So a more correct calculation is to look at &lt;code&gt;ppois(129, 100, lower=F)=0.00228&lt;/code&gt;. This is still small, but not as extreme as your value. And this does not even take into account, that an unusually low number of hits might also surprise you. We often multiply the probability of exceeding the observed count by 2 to account for this.&lt;/li&gt;&#xA;&lt;li&gt;If you keep checking your hits every day, sooner or later even rare events will occur. For example P(X&gt;=130) happens to be close to 1/365, so such an event would be expected to occur once a year. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="279" LastActivityDate="2011-02-15T14:15:43.147" CommentCount="1" />
  <row Id="7246" PostTypeId="1" CreationDate="2011-02-15T14:22:27.763" Score="1" ViewCount="49" Body="&lt;p&gt;It seems non-convexity of loss function is not such a problem for &lt;a href=&quot;http://scholar.google.fr/scholar?cluster=11408949777228074694&quot; rel=&quot;nofollow&quot;&gt;boosting with a normalized sigmoid loss function&lt;/a&gt;. Do you know any further work showing better results with this kind of boosting than &lt;a href=&quot;http://en.wikipedia.org/wiki/AdaBoost&quot; rel=&quot;nofollow&quot;&gt;Adaboost&lt;/a&gt; (exponential loss function)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are the graphs of different loss functions adapted from the graphs shown in &lt;a href=&quot;http://scholar.google.fr/scholar?cluster=6494015189632121933&quot; rel=&quot;nofollow&quot;&gt;&lt;em&gt;A note on margin-based loss functions in classification&lt;/em&gt;&lt;/a&gt; by Yi Lin:&#xA;&lt;img src=&quot;http://i.imgur.com/ATixh.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="1351" LastEditorUserId="1351" LastEditorDisplayName="" LastEditDate="2011-02-15T19:44:29.367" LastActivityDate="2011-02-15T19:44:29.367" Title="Normalized sigmoid loss function for boosting?" Tags="&lt;boosting&gt;&lt;loss-functions&gt;" />
  <row Id="7247" PostTypeId="2" ParentId="7244" CreationDate="2011-02-15T14:24:58.077" Score="2" ViewCount="" Body="&lt;p&gt;First, note that &lt;code&gt;dpois(130, 100)&lt;/code&gt; will give you the probability of &lt;em&gt;exactly&lt;/em&gt; 130 hits if you are assuming that the true rate is 100. That probability is indeed very low. However, in the usual hypothesis testing framework, what we calculate is the probability of the observed outcome or an even more extreme outcome. You can obtain this for the Poisson distribution with:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; ppois(129, lambda=100, lower.tail=FALSE)&#xA;[1] 0.002282093&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So, there is a ~.2% probability of observing the 130 hits or even more hits if you are assuming a true rate of 100. By convention, if this value is below .025 (which it is), we would consider this finding &quot;statistically significant&quot; at $\alpha = .05$ (two-sided). What this means is that you are willing to take a 5% risk that your decision (calling the deviation statistically significant and rejecting the hypothesis that the true rate is 100 for that observation) is wrong. That is, if the true rate is indeed 100 for that day, then in 2.5% of the cases, the observed rate will in fact be 120 or larger (&lt;code&gt;qpois(.975, lambda=100)&lt;/code&gt;) and in 2.5% of the cases, the observed rate will be 81 or lower (&lt;code&gt;qpois(.025, lambda=100)&lt;/code&gt;). So, if you are using $\alpha = .05$, then in 5% of the cases, your decision will be false.&lt;/p&gt;&#xA;" OwnerUserId="1934" LastEditorUserId="1934" LastEditorDisplayName="" LastEditDate="2011-02-15T15:35:20.957" LastActivityDate="2011-02-15T15:35:20.957" CommentCount="2" />
  <row Id="7248" PostTypeId="1" CreationDate="2011-02-15T15:14:21.463" Score="1" ViewCount="151" Body="&lt;p&gt;&lt;a href=&quot;http://mifrullaperlatesta.blogspot.com/2010/11/histogram-with-gnuplot-data-elaboration.html&quot; rel=&quot;nofollow&quot;&gt;http://mifrullaperlatesta.blogspot.com/2010/11/histogram-with-gnuplot-data-elaboration.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Referring to the the question in the above link, what are the available options for the boxes, for example, &quot;filled solid box&quot; or a fitted curve that touch the &quot;mid-head&quot; of the box?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks!&lt;/p&gt;&#xA;" OwnerUserId="3172" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-02-15T15:43:29.877" LastActivityDate="2011-02-15T17:25:51.367" ClosedDate="2011-02-16T17:21:39.143" Title="Altering Gnuplot histogram" Tags="&lt;data-visualization&gt;&lt;gnuplot&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="1" />
  <row Id="7249" PostTypeId="1" AcceptedAnswerId="11361" CreationDate="2011-02-15T15:52:09.717" Score="4" ViewCount="157" Body="&lt;p&gt;I was wondering if there is a free tool to build a decision tree in interactive fashion like in SAS Enterprise Mining. I'm used to work with Weka. But nothing fits to my needs. I would like that before splitting every node, the program asks to user which attribute (maybe from a list of the &quot;best&quot; attributes) to choose. I saw that in SAS it is implemented. Should I write some code to get what I want?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="2719" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-02-15T17:21:45.207" LastActivityDate="2011-05-31T19:42:30.857" Title="Interactive decision trees" Tags="&lt;sas&gt;&lt;decision-trees&gt;&lt;weka&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="7250" PostTypeId="1" CreationDate="2011-02-15T16:02:30.703" Score="3" ViewCount="324" Body="&lt;p&gt;I'm having difficulty understanding one or two aspects of the cluster package. I'm following the example from &lt;a href=&quot;http://www.statmethods.net/advstats/cluster.html&quot; rel=&quot;nofollow&quot;&gt;Quick-R&lt;/a&gt; closely, but don't understand one or two aspects of the analysis. I've included the code that I am using for this particular example.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;## Libraries&#xA;library(stats)&#xA;library(fpc) &#xA;&#xA;## Data&#xA;mydata = structure(list(a = c(461.4210925, 1549.524107, 936.42856, 0, &#xA;0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 131.4349206, 0, 762.6110846, &#xA;3837.850406), b = c(19578.64174, 2233.308842, 4714.514274, 0, &#xA;2760.510002, 1225.392118, 3706.428246, 2693.353714, 2674.126613, &#xA;592.7384164, 1820.976961, 1318.654162, 1075.854792, 1211.248996, &#xA;1851.363623, 3245.540062, 1711.817955, 2127.285272, 2186.671242&#xA;), c = c(1101.899095, 3.166506463, 0, 0, 0, 1130.890295, 0, 654.5054857, &#xA;100.9491289, 0, 0, 0, 0, 0, 789.091922, 0, 0, 0, 0), d = c(33184.53871, &#xA;11777.47447, 15961.71874, 10951.32402, 12840.14983, 13305.26424, &#xA;12193.16597, 14873.26461, 11129.10269, 11642.93146, 9684.238583, &#xA;15946.48195, 11025.08607, 11686.32213, 10608.82649, 8635.844964, &#xA;10837.96219, 10772.53223, 14844.76478), e = c(13252.50358, 2509.5037, &#xA;1418.364947, 2217.952853, 166.92007, 3585.488983, 1776.410835, &#xA;3445.14319, 1675.722506, 1902.396338, 945.5376228, 1205.456943, &#xA;2048.880329, 2883.497101, 1253.020175, 1507.442736, 0, 1686.548559, &#xA;5662.704559), f = c(44.24828759, 0, 485.9617601, 372.108855, &#xA;0, 509.4916263, 0, 0, 0, 212.9541122, 80.62920455, 0, 0, 30.16525587, &#xA;135.0501384, 68.38023073, 0, 21.9317122, 65.09052886), g = c(415.8909649, &#xA;0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 637.2629479, 0, 0, &#xA;0), h = c(583.2213618, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, &#xA;0, 0, 0, 0, 0), i = c(68206.47387, 18072.97762, 23516.98828, &#xA;13541.38572, 15767.5799, 19756.52726, 17676.00505, 21666.267, &#xA;15579.90094, 14351.02033, 12531.38237, 18470.59306, 14149.82119, &#xA;15811.23348, 14637.35235, 13588.64291, 12549.78014, 15370.90886, &#xA;26597.08152)), .Names = c(&quot;a&quot;, &quot;b&quot;, &quot;c&quot;, &quot;d&quot;, &quot;e&quot;, &quot;f&quot;, &quot;g&quot;, &#xA;&quot;h&quot;, &quot;i&quot;), row.names = c(NA, -19L), class = &quot;data.frame&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then I standardize the variables:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# standardize variables&#xA;mydata &amp;lt;- scale(mydata) &#xA;&#xA;## K-means Clustering &#xA;&#xA;# Determine number of clusters&#xA;wss &amp;lt;- (nrow(mydata)-1)*sum(apply(mydata,2,var))&#xA;for (i in 2:15) wss[i] &amp;lt;- sum(kmeans(mydata, centers=i)$withinss)&#xA;# Q1&#xA;plot(1:15, wss, type=&quot;b&quot;, xlab=&quot;Number of Clusters&quot;,  ylab=&quot;Within groups sum of squares&quot;) &#xA;&#xA;# K-Means Cluster Analysis&#xA;fit &amp;lt;- kmeans(mydata, 3) # number of values in cluster solution&#xA;&#xA;# get cluster means &#xA;aggregate(mydata,by=list(fit$cluster),FUN=mean)&#xA;&#xA;# append cluster assignment&#xA;mydata &amp;lt;- data.frame(mydata, cluster = fit$cluster)&#xA;&#xA;# Cluster Plot against 1st 2 principal components - vary parameters for most readable graph&#xA;clusplot(mydata, fit$cluster, color=TRUE, shade=TRUE, labels=0, lines=0) # Q2&#xA;&#xA;# Centroid Plot against 1st 2 discriminant functions&#xA;plotcluster(mydata, fit$cluster)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My question is, how can the plot which shows the number of clusters (marked &lt;code&gt;Q1&lt;/code&gt; in my code) be related to the actual values (cluster number and variable name) ? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Update: I now understand that the &lt;code&gt;clusplot()&lt;/code&gt; function is a bivariate plot, with PCA1 and PCA2. However, I don't understand the link between the PCA components and the cluster groups. What is the relationship between the PCA values and the clustering groups? I've read elsewhere about the link between kmeans and PCA, but I still don't understand how they can be displayed on the same bivariate graph. &lt;/p&gt;&#xA;" OwnerUserId="2635" LastEditorUserId="2635" LastEditorDisplayName="" LastEditDate="2011-02-22T19:05:52.250" LastActivityDate="2011-02-22T19:05:52.250" Title="Using the stats package in R for kmeans clustering" Tags="&lt;r&gt;&lt;clustering&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="0" />
  <row Id="7251" PostTypeId="1" CreationDate="2011-02-15T16:09:10.773" Score="5" ViewCount="86" Body="&lt;p&gt;&lt;strong&gt;Framework.&lt;/strong&gt; Fix $\alpha\in ]0,1[$. Imagine you have $n$ $\alpha$-quantile forecast methodologies that give you, at time $t$ for look ahead time $t+h$, an estimation of the quantile of wind power. Formally, for $i=1,\dots,n$, you know how to produce $\hat{q}_{t+h|t}^{(i)}$ at time $t$ for look ahead time $t+h$ an estimation. Each methodology is based on a different modeling+estimation and can have performance that depend, for example, on the weather situation.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question.&lt;/strong&gt;  How do you construct a weighting scheme to combine quantile estimation (say with a linear combination) that can adapt along time $t$?  Formally, how to best construct weights $\lambda_1(t,h),\dots,\lambda_n(t,h)$ such that &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\hat{q}_{t+h|t}=\sum_{i=1}^n \lambda_i(t,h) \hat{q}_{t+h|t}^{(i)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;is a very good quantile forecast. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Side Note.&lt;/strong&gt; For Msc students interested in proposing and elaborating their ideas with the real data, I propose an internship on that subject for summer 2011 (see &lt;a href=&quot;http://www-cep.cma.fr/Public/recrutement/proposition_de_stage/stage_prevision_eol/&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;, it's in french but I can translate to those interested).  &lt;/p&gt;&#xA;" OwnerUserId="223" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-02-17T22:08:28.940" LastActivityDate="2011-02-22T15:26:12.237" Title="How to make a combination (aggregation) of quantile forecast?" Tags="&lt;time-series&gt;&lt;forecasting&gt;&lt;quantiles&gt;&lt;aggregation&gt;&lt;combination&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="7252" PostTypeId="2" ParentId="7236" CreationDate="2011-02-15T16:12:24.807" Score="0" ViewCount="" Body="&lt;p&gt;I think no extra package is needed for the task, just use the basic &lt;code&gt;sample&lt;/code&gt; function, e.g.:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Get sample from the first group:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sample &amp;lt;- sample(data[data$&quot;Care Type&quot; == &quot;Acute Care&quot;,], size = 25)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Get the choosen IDs out of the orig. dataset (making a backup could be a good idea before that):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data &amp;lt;- data[setdiff(data$pat_id, sample_pat_id),]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Get sample from second group in the rest of the dataset and concatenate to sample:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sample &amp;lt;- rbind(sample, sample(data[(data$&quot;Care Type&quot; == &quot;Acute Care&quot;),], size = 25)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Repeat for each segment:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data &amp;lt;- data[setdiff(data$pat_id, sample_pat_id),]&#xA;sample &amp;lt;- rbind(sample, sample(data[(data$&quot;Care Type&quot; == &quot;?&quot;),], size = ?)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Sorry, not tested, but I think the point can be seen. And also: I am sure the above code could be improved and minified.&lt;/p&gt;&#xA;" OwnerUserId="2714" LastActivityDate="2011-02-15T16:12:24.807" />
  <row Id="7253" PostTypeId="2" ParentId="7249" CreationDate="2011-02-15T16:41:24.487" Score="6" ViewCount="" Body="&lt;p&gt;Try the examples under dendrogram.   You can make it as interactive as you want.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(graphics); require(utils)&#xA;&#xA;hc &amp;lt;- hclust(dist(USArrests), &quot;ave&quot;)&#xA;(dend1 &amp;lt;- as.dendrogram(hc)) # &quot;print()&quot; method&#xA;str(dend1)          # &quot;str()&quot; method&#xA;str(dend1, max = 2) # only the first two sub-levels&#xA;&#xA;op &amp;lt;- par(mfrow= c(2,2), mar = c(5,2,1,4))&#xA;plot(dend1)&#xA;## &quot;triangle&quot; type and show inner nodes:&#xA;plot(dend1, nodePar=list(pch = c(1,NA), cex=0.8, lab.cex = 0.8),&#xA;      type = &quot;t&quot;, center=TRUE)&#xA;plot(dend1, edgePar=list(col = 1:2, lty = 2:3),&#xA;      dLeaf=1, edge.root = TRUE)&#xA;plot(dend1, nodePar=list(pch = 2:1,cex=.4*2:1, col = 2:3), &#xA;      horiz=TRUE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Edit 1 ====================================&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The interactivity depends on what you want to do.   It all comes down to the structure of the data that goes to &lt;code&gt;plot&lt;/code&gt;.    To make it easier to see what's going on, I'll only use the first 3 lines of data from the above example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#Use only the first 3 lines from USArrests&#xA;(df &amp;lt;- USArrests[1:3,])&#xA;&#xA;#Perform the hc analysis&#xA;(hcdf &amp;lt;- hclust(dist(df), &quot;ave&quot;))&#xA;&#xA;#Plot the results&#xA;plot(hcdf)&#xA;&#xA;#Look at the names of hcdf&#xA;names(hcdf)&#xA;&#xA;#Look at the structure of hcdf&#xA;dput(hcdf)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The next segment is the output of the above &lt;code&gt;dput&lt;/code&gt; statement.   This structure tells &lt;code&gt;plot&lt;/code&gt; how to draw the tree.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;structure(list(merge = structure(c(-1L, -3L, -2L, 1L), .Dim = c(2L,&#xA;2L)), height = c(37.1770090243957, 54.8004107236398), order = c(3L,&#xA;1L, 2L), labels = c(&quot;Alabama&quot;, &quot;Alaska&quot;, &quot;Arizona&quot;), method = &quot;average&quot;,&#xA;   call = hclust(d = dist(df), method = &quot;ave&quot;), dist.method = &quot;euclidean&quot;),&#xA;.Names = c(&quot;merge&quot;, &quot;height&quot;, &quot;order&quot;, &quot;labels&quot;, &quot;method&quot;, &quot;call&quot;, &quot;dist.method&quot;),&#xA;class = &quot;hclust&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.stack.imgur.com/UgagD.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can easily change the data and see what &lt;code&gt;plot&lt;/code&gt; does.  Just copy/paste the &lt;code&gt;structure&lt;/code&gt; statement from your screen and assign it to a new variable, make your changes, and plot it.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;newvar &amp;lt;- structure(list(merge = structure(c(-1L, -3L, -2L, 1L), .Dim = c(2L, 2L)), height = c(37.1770090243957, 54.8004107236398), order = c(3L, 1L, 2L), labels = c(&quot;Alabama&quot;, &quot;Alaska&quot;, &quot;Arizona&quot;), method = &quot;average&quot;,     call = hclust(d = dist(df), method = &quot;ave&quot;), dist.method = &quot;euclidean&quot;), .Names = c(&quot;merge&quot;, &quot;height&quot;, &quot;order&quot;, &quot;labels&quot;, &quot;method&quot;, &quot;call&quot;, &quot;dist.method&quot;), class = &quot;hclust&quot;)&#xA;&#xA;plot(newvar)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As far as making the clustering more interactive, you'll have to explore the different methods and determine what you want to do.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://cran.cnr.berkeley.edu/web/views/Cluster.html&quot; rel=&quot;nofollow&quot;&gt;http://cran.cnr.berkeley.edu/web/views/Cluster.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://wiki.math.yorku.ca/index.php/R:_Cluster_analysis&quot; rel=&quot;nofollow&quot;&gt;http://wiki.math.yorku.ca/index.php/R:_Cluster_analysis&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.statmethods.net/advstats/cluster.html&quot; rel=&quot;nofollow&quot;&gt;http://www.statmethods.net/advstats/cluster.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.statmethods.net/advstats/cart.html&quot; rel=&quot;nofollow&quot;&gt;http://www.statmethods.net/advstats/cart.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2775" LastEditorUserId="2775" LastEditorDisplayName="" LastEditDate="2011-05-31T19:42:30.857" LastActivityDate="2011-05-31T19:42:30.857" CommentCount="2" />
  <row Id="7254" PostTypeId="2" ParentId="7248" CreationDate="2011-02-15T17:25:51.367" Score="0" ViewCount="" Body="&lt;pre&gt;&lt;code&gt;set style fill solid 1.0 border -1   &#xA;binwidth=5&#xA;bin(x,width)=width*floor(x/width) + binwidth/2.0&#xA;plot 'datafile' using (bin($1,binwidth)):(1.0) smooth freq with boxes&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="3172" LastActivityDate="2011-02-15T17:25:51.367" CommentCount="1" />
  <row Id="7255" PostTypeId="2" ParentId="4364" CreationDate="2011-02-15T17:43:50.737" Score="1" ViewCount="" Body="&lt;p&gt;This is not a characterization but a conjecture, which dates back from 1917 and is due to Cantelli: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;If $f$ is a positive function on $\mathbb{R}$ and $X$ and $Y$ are $N(0,1)$ independent random variables such that $X+f(X)Y$ is normal, then $f$ is a constant almost everywhere.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Mentioned by Gérard Letac &lt;a href=&quot;http://mathoverflow.net/questions/37151/what-are-the-big-problems-in-probability-theory/39999#39999&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2592" LastEditorUserId="2592" LastEditorDisplayName="" LastEditDate="2011-02-15T18:07:59.233" LastActivityDate="2011-02-15T18:07:59.233" CommunityOwnedDate="2011-02-15T17:43:50.737" CommentCount="2" />
  <row Id="7256" PostTypeId="1" AcceptedAnswerId="7283" CreationDate="2011-02-15T18:20:56.520" Score="5" ViewCount="139" Body="&lt;p&gt;I am working on binary classification problem. Data set is very large and highly imbalanced.&lt;br&gt;&#xA;Data dimensionality is also very high.&#xA;Now I want to balance data by under-sampling the majority class, and I also want to reduce data dimensionality by applying PCA, etc... &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my question is that which one should be applied first: data sampling or dimensionality reduction? &#xA;Please also give argument in favor of your answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance &lt;/p&gt;&#xA;" OwnerUserId="2534" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-02-15T19:03:27.827" LastActivityDate="2011-02-16T20:14:40.750" Title="Which one should be applied first: data sampling or dimensionality reduction?" Tags="&lt;classification&gt;&lt;sampling&gt;&lt;dataset&gt;" AnswerCount="4" CommentCount="4" FavoriteCount="2" />
  <row Id="7257" PostTypeId="1" CreationDate="2011-02-15T18:39:29.390" Score="4" ViewCount="66" Body="&lt;p&gt;I'm trying to understand how Boltzmann machines work, but I'm not quite sure how weights are learned, and haven't been able to find a clear description. Is the following correct? (Also, pointers to any good Boltzmann machine explanations would also be great.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We have a set of visible units (e.g., corresponding to black/white pixels in an image) and a set of hidden units. Weights are initialized somehow (e.g., uniformly from [-0.5, 0.5]), and then we alternate between the following two phases until some stopping rule is reached:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Clamped phase - In this phase, all the values of the visible units are fixed, so we only update the states of the hidden units (according to the Boltzmann stochastic activation rule). We update until the network has reached equilibrium. Once we reach equilibrium, we continue updating $N$ more times (for some predefined $N$), keeping track of the average of $x_i x_j$ (where $x_i, x_j$ are the states of nodes $i$ and $j$). After those $N$ equilibrium updates, we update $w_ij = w_ij + \frac{1}{C} Average(x_i  x_j)$, where $C$ is some learning rate. (Or, instead of doing a batch update at the end, do we update after we equilibrium step?)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Free phase - In this phase, the states of all units are updated. Once we reach equilibrium, we similarly continue updating N' more times, but instead of adding correlations at the end, we subtract: $w_{ij} = w_{ij} - \frac{1}{C} Average(x_i  x_j)$.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;So my main questions are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Whenever we're in the clamped phase, do we reset the visible units to one of the patterns we want to learn (with some frequency that represents the importance of that pattern), or do we leave the visible units in the state they were in at the end of the free phase?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Do we do a batch update of the weights at the end of each phase, or update the weights at each equilibrium step in the phase? (Or, is either one fine?)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="1106" LastEditorUserId="1106" LastEditorDisplayName="" LastEditDate="2011-02-15T20:35:20.667" LastActivityDate="2011-02-15T20:35:20.667" Title="Learning weights in a Boltzmann machine" Tags="&lt;neural-networks&gt;" />
  <row Id="7258" PostTypeId="1" CreationDate="2011-02-15T18:50:27.707" Score="4" ViewCount="46" Body="&lt;p&gt;As I have written in my question &quot;&lt;a href=&quot;http://stats.stackexchange.com/questions/7209/how-much-undersampling-should-be-done&quot;&gt;How much undersampling should be done?&lt;/a&gt;&quot;, I want to predict defaults, where a default is per se really unlikely (average ~ 0.3 percent). My models are not affected by the unequal distribution: It's all about saving computing time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Undersampling the majority class to a ratio [defaulting/non-defaulting examples] of 1:1 is the same as expressing the believe that I think examples are equally important in increasing the prediction quality.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone know a reason why/when equal importance could &lt;strong&gt;not&lt;/strong&gt; be the case? Is there literature on this specific topic (I could not find sampling-literature that modeling/computation-oriented)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks a lot for your help!&lt;/p&gt;&#xA;" OwnerUserId="2549" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-10T09:26:59.513" LastActivityDate="2011-03-10T09:26:59.513" Title="Information content of examples and undersampling" Tags="&lt;sampling&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="7259" PostTypeId="1" AcceptedAnswerId="7309" CreationDate="2011-02-15T18:54:10.333" Score="4" ViewCount="462" Body="&lt;p&gt;I have a data-table that has about 26000 rows and about 35 columns.  The columns are paired, so the values in columns 6 and 7 (for example) are related to each other, so are 8 and 9 and so on.  There are 23 different types of annotations in the table, which I have read in as &quot;factor&quot;.  The ratio of these pairs of columns gives me a meaningful number, that I have to plot for each of the annotation.  I was wondering if there is any way to have a lattice plot that will have say 15 boxplots in each panel, and 23 panels one for each annotation?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;UPDATE: Sample table.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;structure(list(chromosome = structure(c(1L, 1L, 1L, 1L, 1L, 1L, &#xA;1L, 1L, 1L, 1L), .Label = c(&quot;chr1&quot;, &quot;chr2&quot;, &quot;chr3&quot;), class = &quot;factor&quot;), &#xA;    start = c(1, 1, 1, 5663, 5726, 6360, 7548, 7619, 11027, 12158&#xA;    ), end = c(5662, 7265, 5579133, 7265, 6331, 6755, 12710, &#xA;    9274, 11556, 12994), strand = structure(c(1L, 1L, 3L, 1L, &#xA;    1L, 1L, 3L, 3L, 1L, 3L), .Label = c(&quot;-&quot;, &quot;.&quot;, &quot;+&quot;), class = &quot;factor&quot;), &#xA;    annotation = structure(c(4L, 13L, 8L, 2L, 13L, 18L, 18L, &#xA;    13L, 12L, 13L), .Label = c(&quot;3'-UTR&quot;, &quot;5'-UTR&quot;, &quot;BLASTN_HIT&quot;, &#xA;    &quot;CDS&quot;, &quot;CDS_motif&quot;, &quot;CDS_parts&quot;, &quot;conflict&quot;, &quot;Contig&quot;, &quot;intron&quot;, &#xA;    &quot;LTR&quot;, &quot;misc_feature&quot;, &quot;misc_RNA&quot;, &quot;mRNA&quot;, &quot;polyA_site&quot;, &#xA;    &quot;promoter&quot;, &quot;real_mRNA&quot;, &quot;rep_origin&quot;, &quot;repeat_region&quot;, &quot;repeat_unit&quot;, &#xA;    &quot;rRNA&quot;, &quot;snoRNA&quot;, &quot;snRNA&quot;, &quot;tRNA&quot;), class = &quot;factor&quot;), Abp1D.sense = c(274.043090077, &#xA;    222.027002967, 273.083037487, 38.3559401569, 80.7384755736, &#xA;    15.9496926371, 54.9087080745, 127.744117176, 11.7165833969, &#xA;    96.1925577965), Abp1D.antisense = c(125.681512904, 151.232091139, &#xA;    254.813202986, 241.034453038, 84.3769908653, 199.467664241, &#xA;    54.1912835565, 94.2017362521, 66.5142677515, 63.28607875), &#xA;    Iki3D.sense = c(1214.1686727, 969.99693773, 261.416187303, &#xA;    107.770848316, 151.518863438, 55.9449713698, 66.0800496533, &#xA;    144.470307921, 21.9708783825, 52.6163190329), Iki3D.antisense = c(786.364743311, &#xA;    728.647444388, 248.288893165, 523.636519401, 263.419180997, &#xA;    351.558399018, 73.754086788, 130.973198864, 93.7873464478, &#xA;    30.858803946), Iki3D.Rrp6D.sense = c(3068.90441567, 2486.4012139, &#xA;    278.274812147, 428.928792511, 639.682546716, 134.968168726, &#xA;    223.376134645, 491.4747595, 72.255001742, 201.429779476), &#xA;    Iki3D.Rrp6D.antisense = c(1928.37423684, 1764.06364622, 271.050084744, &#xA;    1181.76403142, 1276.54960008, 990.571280057, 196.88970278, &#xA;    398.206798139, 62.7937319455, 111.92795268), Rdp1D.sense = c(197.403527744, &#xA;    168.849473212, 399.588620598, 68.0531849874, 128.833494553, &#xA;    30.8082175235, 59.9086910765, 134.404417978, 24.2425410143, &#xA;    85.4825519212), Rdp1D.antisense = c(86.097230688, 254.128565899, &#xA;    388.725581635, 846.769716459, 82.1986385122, 281.872704472, &#xA;    49.97022677, 77.2892621321, 44.6799202033, 1.60870068737), &#xA;    Wt.sense = c(150.835381912, 132.061554165, 607.58955888, &#xA;    65.8027665102, 89.3919476073, 83.4968237124, 7.90112304898, &#xA;    10.714546021, 5e-04, 5e-04), Wt.antisense = c(150.374084859, &#xA;    131.8668254, 659.887826114, 65.7197527173, 45.4289405873, &#xA;    40.4019469576, 7.40733410843, 8.83958796731, 43.5756796108, &#xA;    12.3289419357), Rdp1D.Rrp6D.sense = c(278.940777843, 227.050371919, &#xA;    266.352999304, 43.8265653895, 86.2348572529, 5.1007112686, &#xA;    63.5315969071, 138.590379851, 17.1377883364, 47.2571674648&#xA;    ), Rdp1D.Rrp6D.antisense = c(122.812370852, 165.478532861, &#xA;    262.217884557, 315.685821866, 196.899101029, 181.217276367, &#xA;    64.9492021228, 111.77461648, 62.2771817975, 20.3596716974&#xA;    ), Dcr1D.sense = c(5e-04, 120.491414743, 1325.93762159, 546.346320658, &#xA;    5e-04, 5e-04, 66.3486618734, 5e-04, 5e-04, 5e-04), Dcr1D.antisense = c(5e-04, &#xA;    8346.5035927, 1479.42139464, 37845.8172699, 5e-04, 28845.1503745, &#xA;    1194.26663745, 5e-04, 647.428121154, 5e-04), Er1D.sense = c(387.657094655, &#xA;    332.176880363, 570.413411676, 136.333361806, 228.023187499, &#xA;    5e-04, 24.0778502632, 62.6341480521, 32.1717485621, 5e-04&#xA;    ), Er1D.antisense = c(382.664804454, 343.714717963, 618.13806355, &#xA;    205.325286003, 162.81296098, 145.575708252, 15.3360737154, &#xA;    30.5382985528, 5e-04, 13.8803856753), Rrp6D.sense = c(716.001844534, &#xA;    605.02996247, 444.912126049, 213.265421331, 398.7252034, &#xA;    73.8307932225, 90.5802807096, 172.093792998, 5e-04, 135.365316918&#xA;    ), Rrp6D.antisense = c(690.534019176, 592.944889017, 409.413915909, &#xA;    247.869927895, 160.655498164, 371.504850116, 56.7600331059, &#xA;    119.421944835, 16.7787329876, 20.0208426702), Mlo3D.Ago1D.sense = c(119.466474712, &#xA;    329.741829677, 993.941348153, 1072.99933641, 5e-04, 377.539482989, &#xA;    113.878508361, 50.428609435, 5e-04, 5e-04), Mlo3D.Ago1D.antisense = c(120.543892198, &#xA;    2711.8968975, 1257.1652648, 11870.674213, 125.725150183, &#xA;    8902.64920707, 206.72008398, 37.8215820763, 5e-04, 5e-04), &#xA;    Ago1D.Clr3D.sense = c(184.712264891, 179.831117561, 444.487152139, &#xA;    162.69482267, 202.293495599, 5.61159966339, 63.6233691066, &#xA;    90.544306737, 5e-04, 170.284591079), Ago1D.Clr3D.antisense = c(57.5740294693, &#xA;    67.5638155026, 386.644572497, 102.906975334, 79.4664091704, &#xA;    2.1204925561, 14.4184581702, 35.3125846275, 5e-04, 5e-04), &#xA;    Dcr1D.Rrp6D.sense = c(45.8846113251, 63.7325750806, 360.192351832, &#xA;    126.841847799, 277.614908589, 54.2822292313, 33.9452752392, &#xA;    83.1313557186, 5e-04, 12.8242338794), Dcr1D.Rrp6D.antisense = c(19.3160147626, &#xA;    55.5834301591, 363.594792664, 183.776577157, 18.3768674716, &#xA;    322.564097746, 17.907465048, 33.1088927537, 5e-04, 5e-04), &#xA;    Ago1D.sense = c(29.0628360487, 31.9691923002, 387.82120669, &#xA;    42.2593617334, 64.0004397647, 68.0567121551, 65.0088334947, &#xA;    189.345502766, 5e-04, 26.5639424914), Ago1D.antisense = c(10.918535798, &#xA;    84.6095118936, 373.635073395, 345.064708329, 40.1150042497, &#xA;    266.756186351, 4.38085691952, 5e-04, 5e-04, 5e-04), Mlo3D.sense = c(2798.34040679, &#xA;    2353.07409522, 330.364494647, 781.101862885, 1312.81871554, &#xA;    376.811874795, 124.564566466, 353.76677093, 5e-04, 31.5118039429&#xA;    ), Mlo3D.antisense = c(2532.2553647, 2248.78653802, 292.881120203, &#xA;    1246.84984213, 1981.14439149, 564.070923014, 164.753382721, &#xA;    449.669663275, 5e-04, 5e-04), Ago1D.Rrp6D.sense = c(86.379996345, &#xA;    90.4014346003, 468.105009795, 104.668452639, 203.155350014, &#xA;    62.3955638527, 44.5603393841, 84.3076975857, 16.0419716595, &#xA;    42.5345756816), Ago1D.Rrp6D.antisense = c(45.0506816078, &#xA;    80.7182081997, 481.700138654, 206.646370214, 67.1332741403, &#xA;    129.669542952, 23.7209335341, 26.0270063646, 28.9823086155, &#xA;    16.4901597751)), .Names = c(&quot;chromosome&quot;, &quot;start&quot;, &quot;end&quot;, &#xA;&quot;strand&quot;, &quot;annotation&quot;, &quot;Abp1D.sense&quot;, &quot;Abp1D.antisense&quot;, &quot;Iki3D.sense&quot;, &#xA;&quot;Iki3D.antisense&quot;, &quot;Iki3D.Rrp6D.sense&quot;, &quot;Iki3D.Rrp6D.antisense&quot;, &#xA;&quot;Rdp1D.sense&quot;, &quot;Rdp1D.antisense&quot;, &quot;Wt.sense&quot;, &quot;Wt.antisense&quot;, &#xA;&quot;Rdp1D.Rrp6D.sense&quot;, &quot;Rdp1D.Rrp6D.antisense&quot;, &quot;Dcr1D.sense&quot;, &#xA;&quot;Dcr1D.antisense&quot;, &quot;Er1D.sense&quot;, &quot;Er1D.antisense&quot;, &quot;Rrp6D.sense&quot;, &#xA;&quot;Rrp6D.antisense&quot;, &quot;Mlo3D.Ago1D.sense&quot;, &quot;Mlo3D.Ago1D.antisense&quot;, &#xA;&quot;Ago1D.Clr3D.sense&quot;, &quot;Ago1D.Clr3D.antisense&quot;, &quot;Dcr1D.Rrp6D.sense&quot;, &#xA;&quot;Dcr1D.Rrp6D.antisense&quot;, &quot;Ago1D.sense&quot;, &quot;Ago1D.antisense&quot;, &quot;Mlo3D.sense&quot;, &#xA;&quot;Mlo3D.antisense&quot;, &quot;Ago1D.Rrp6D.sense&quot;, &quot;Ago1D.Rrp6D.antisense&quot;&#xA;), row.names = c(NA, 10L), class = &quot;data.frame&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The question asked above is when you have a &lt;code&gt;data.frame&lt;/code&gt; with all the data.  What if I now want to create a &lt;code&gt;list&lt;/code&gt; so that each entry in the list is actually a &lt;code&gt;data.frame&lt;/code&gt; with a structure similar to one given above.  How do I combine the boxplots in the lattice?  Does the &lt;code&gt;ggplot2&lt;/code&gt; have a solution for this? Can someone guide me to such a solution?&lt;/p&gt;&#xA;" OwnerUserId="3263" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-02-22T07:57:34.973" LastActivityDate="2011-02-22T07:57:34.973" Title="Combine multiple boxplots in a lattice" Tags="&lt;r&gt;&lt;data-visualization&gt;&lt;boxplot&gt;" AnswerCount="2" CommentCount="3" FavoriteCount="1" />
  <row Id="7260" PostTypeId="2" ParentId="7256" CreationDate="2011-02-15T18:55:09.617" Score="2" ViewCount="" Body="&lt;p&gt;Do the dimensionality reduction first: Your error in estimating the principal components will be smaller due to the larger sample (your Corr/Cov-matrix used in PCA has to be estimated!).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The other way around only makes sense for computational reasons.&lt;/p&gt;&#xA;" OwnerUserId="2549" LastActivityDate="2011-02-15T18:55:09.617" />
  <row Id="7261" PostTypeId="1" AcceptedAnswerId="7262" CreationDate="2011-02-15T19:28:22.333" Score="6" ViewCount="260" Body="&lt;p&gt;Hello everyone! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is my first post, so please take it easy on me if I am not following some standards! I did a search for my question and nothing came up.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question relates mostly around the practical differences between&#xA;general linear modeling (GLM) and generalized linear modelling (GZLM). In my&#xA;case it would be a few continuous variables as covariates and a few&#xA;factors in an ANCOVA, versus GZLM. I want to examine the main effects&#xA;of each variable, as well as one three-way interaction that I will outline in&#xA;the model.  I can see this hypothesis being tested in an ANCOVA, or&#xA;using GZLM. To some extent I understand the math processes and&#xA;reasoning behind running a general linear model like an ANCOVA, and I&#xA;somewhat understand that GZLMs allow for a link function connecting&#xA;the linear model and the dependent variable (ok, I lied, maybe I don't really understand the math). What I realy don't understand are the practical differences or reasons for running one analysis and not the other when the probability distribution used in the GZLM is&#xA;normal (i.e., identity link function?). I get very different results when I run one over the other. Could I run either? My&#xA;data is somewhat non-normal, but works to some extent both in the ANCOVA and the GZLM. In both cases my hypothesis is supported, but in the GZLM the p value is &quot;better&quot;.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My thought was that an ANCOVA is a linear model with a normally&#xA;distributed dependent variable using an identity link function, which&#xA;is exactly what I can input in a GZLM, but these are still different.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please shed some light on these questions for me, if you can! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Mike&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Based on the first answer I have the additional question:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If they are identical except for the significance test that it utilized (i.e., F test vs. Wald Chi Square), which would be most appropriate to use? ANCOVA is the &quot;go-to method&quot;, but I am unsure why the F test would be preferable. Can someone shed some light on this question for me? Thanks!  &lt;/p&gt;&#xA;" OwnerUserId="3262" LastEditorUserId="3262" LastEditorDisplayName="" LastEditDate="2011-02-15T21:40:20.300" LastActivityDate="2011-02-18T10:23:28.347" Title="General Linear Model vs. Generalized Linear Model (with an identity link function?)" Tags="&lt;modeling&gt;&lt;linear-model&gt;" AnswerCount="2" FavoriteCount="2" />
  <row Id="7262" PostTypeId="2" ParentId="7261" CreationDate="2011-02-15T20:10:32.780" Score="9" ViewCount="" Body="&lt;p&gt;A generalized linear model specifying an identity link function and a normal family distribution is exactly equivalent to a (general) linear model. If you're getting noticeably different results from each, you're doing something wrong.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that specifying an identity link is &lt;em&gt;not&lt;/em&gt; the same thing as specifying a normal distribution. The distribution and the link function are two different components of the generalized linear model, and each can be chosen independently of the other (although certain links work better with certain distributions, so most software packages specify the choice of links allowed for each distribution).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some software packages may report noticeably different $p$-values when the residual degrees of freedom are small if it calculates these using the asymptotic normal and chi-square distributions for all generalized linear models. All software will report $p$-values based on Student's $t$- and Fisher's $F$-distributions for general linear models, as these are more accurate for small residual degrees of freedom as they do not rely on asymptotics. Student's $t$- and Fisher's $F$-distributions are strictly valid for the normal family only, although some &lt;em&gt;other&lt;/em&gt; software for generalized linear models may also use these as approximations when fitting other families with a scale parameter that is estimated from the data.&lt;/p&gt;&#xA;" OwnerUserId="449" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-02-18T10:23:28.347" LastActivityDate="2011-02-18T10:23:28.347" CommentCount="3" />
  <row Id="7263" PostTypeId="1" AcceptedAnswerId="7269" CreationDate="2011-02-15T20:21:36.447" Score="5" ViewCount="111" Body="&lt;p&gt;I am writing about using a 'joint probability distribution' for an audience that would be more likely to understand 'multivariate distribution' so I am considering using the later. However, I do not want to loose meaning while doing this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Joint_probability_distribution&quot; rel=&quot;nofollow&quot;&gt;Wikipedia&lt;/a&gt; seems to indicate that these are synonyms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are they? If not, why not? &lt;/p&gt;&#xA;" OwnerUserId="1381" LastActivityDate="2011-02-15T21:43:16.040" Title="Difference between the terms 'joint distribution' and 'multivariate distribution'?" Tags="&lt;probability&gt;&lt;definition&gt;&lt;terminology&gt;" AnswerCount="4" />
  <row Id="7264" PostTypeId="2" ParentId="7239" CreationDate="2011-02-15T20:30:35.940" Score="1" ViewCount="" Body="&lt;p&gt;&lt;em&gt;disclaimer&lt;/em&gt; I still don't fully understand your model; but without at least a reproducible example, this is the best I can offer. It is not clear exactly what you are doing here. For example, how are &lt;code&gt;pvr&lt;/code&gt; and &lt;code&gt;pir&lt;/code&gt; calculated? Would it make sense to calculate them inside the same model?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Answer&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am assuming that your data includes observations for &lt;code&gt;mu[]&lt;/code&gt; but not &lt;code&gt;pmu[]&lt;/code&gt; and you want to estimate &lt;code&gt;pmu[j]&lt;/code&gt; given &lt;code&gt;j&lt;/code&gt; values of &lt;code&gt;pvr&lt;/code&gt; and &lt;code&gt;pir&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Append the &lt;code&gt;pir&lt;/code&gt; and &lt;code&gt;pvr&lt;/code&gt; to the &lt;code&gt;ir&lt;/code&gt; and &lt;code&gt;vr&lt;/code&gt; columns, get rid of the second for loop, and then consider the values of &lt;code&gt;mu[]&lt;/code&gt; estimated using &lt;code&gt;pir&lt;/code&gt; and &lt;code&gt;pvr&lt;/code&gt; to be the posterior predictive estimates of &lt;code&gt;mu&lt;/code&gt;. Then replace the two &lt;code&gt;for&lt;/code&gt; loops with this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for (i in 1:length(ri)+length(pri)){&#xA;    ri[i] ~ dnorm(mu[i],tau)&#xA;    mu[i] &amp;lt;- alpha + b.vr*vr[i] + b.ir*ir[i]&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have done something similar, but without predicted regressors, similar to the example given by &lt;a href=&quot;http://www.stat.columbia.edu/~gelman/bugsR/software.pdf&quot; rel=&quot;nofollow&quot;&gt;Gelman et al in 'Bayesian Data Analysis' (pp 598-599&lt;/a&gt; starting under &lt;em&gt;posterior predictive simulations&lt;/em&gt;).&lt;/p&gt;&#xA;" OwnerUserId="1381" LastEditorUserId="1381" LastEditorDisplayName="" LastEditDate="2011-02-18T21:58:49.337" LastActivityDate="2011-02-18T21:58:49.337" CommentCount="8" />
  <row Id="7265" PostTypeId="2" ParentId="7263" CreationDate="2011-02-15T20:39:58.100" Score="0" ViewCount="" Body="&lt;p&gt;I'd be inclined to say that &quot;multivariate&quot; describes the random variable, i.e., it is a vector, and that the components of a multivariate random variable have a joint distribution. &quot;Multivariate random variable&quot; sounds a bit strange, though; I'd call it a random vector.&lt;/p&gt;&#xA;" OwnerUserId="401" LastActivityDate="2011-02-15T20:39:58.100" />
  <row Id="7266" PostTypeId="2" ParentId="7263" CreationDate="2011-02-15T20:45:55.713" Score="0" ViewCount="" Body="&lt;p&gt;I think they are mostly synonyms, and that if there is any difference, it lies in details that are likely irrelevant to your audience.&lt;/p&gt;&#xA;" OwnerUserId="2044" LastActivityDate="2011-02-15T20:45:55.713" />
  <row Id="7267" PostTypeId="2" ParentId="7263" CreationDate="2011-02-15T20:56:41.460" Score="1" ViewCount="" Body="&lt;p&gt;The &lt;a href=&quot;http://www.google.com/search?hl=en&amp;amp;tbs=bks%3A1&amp;amp;q=inauthor%3Ajohnson+inauthor%3Akotz+intitle%3Adistributions&quot; rel=&quot;nofollow&quot;&gt;canonical textbooks describing properties of the various probability distributions by Johnson &amp;amp; Kotz&lt;/a&gt; and later co-authors are entitled &lt;em&gt;Univariate Discrete Distributions&lt;/em&gt;, &lt;em&gt;Continuous Univariate Distributions&lt;/em&gt;, &lt;em&gt;Continuous Multivariate Distributions&lt;/em&gt; and &lt;em&gt;Discrete Multivariate Distributions&lt;/em&gt;. So I think you're on safe ground describing a distribution as 'multivariate' rather than 'joint'.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;sub&gt;Conflict of interest statement: The author is a member of &lt;a href=&quot;http://en.wikipedia.org/wiki/Wikipedia%3aWikiProject_Statistics&quot; rel=&quot;nofollow&quot;&gt;Wikipedia:WikiProject Statistics&lt;/a&gt;.&lt;/sub&gt;&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-02-15T20:56:41.460" />
  <row Id="7268" PostTypeId="1" AcceptedAnswerId="7306" CreationDate="2011-02-15T21:00:59.150" Score="2" ViewCount="319" Body="&lt;p&gt;How would you get hourly means for multiple data columns, for a daily period, and show results for twelve &quot;Hosts&quot; in the same graph?  That is, I'd like to graph what a 24 hour period looks like, for a weeks worth of data.  The eventual goal would be to compare two sets of this data, before and after samplings.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;            dates         Host CPUIOWait CPUUser CPUSys&#xA;1 2011-02-11 23:55:12     db       0      14      8&#xA;2 2011-02-11 23:55:10     app1     0       6      1&#xA;3 2011-02-11 23:55:09     app2     0       4      1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I've been able to run xyplot(CPUUser ~ dates | Host) with good effect.  However, rather than showing each date in the week, I'd like the X axis to be the hours of the day.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Trying to get this data into an xts object results in errors such as &lt;em&gt;&quot;order.by requires an appropriate time-based object&quot;&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a str() of the data frame:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;'data.frame':   19720 obs. of  5 variables:&#xA;$ dates    : POSIXct, format: &amp;quot;2011-02-11 23:55:12&amp;quot; &amp;quot;2011-02-11 23:55:10&amp;quot; ...&#xD;&#xA;$ Host     : Factor w/ 14 levels &quot;app1&quot;,&quot;app2&quot;,..: 9 7 5 4 3 10 6 8 2 1 ...  &#xA;$ CPUIOWait: int  0 0 0 0 0 0 0 0 0 0 ...&#xD;&#xA;$ CPUUser  : int  14 6 4 4 3 10 4 3 4 4 ...&#xA;$ CPUSys   : int  8 1 1 1 1 3 1 1 1 1 ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;UPDATE: Just for future reference, I decided to go with a boxplot, to show both the median, and the 'outliers'.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Essentially:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Data$hour &amp;lt;- as.POSIXlt(dates)$hour  # extract hour of the day&#xA;boxplot(Data$CPUUser ~ Data$hour)    # for a subset with one host or for all hosts&#xA;xyplot(Data$CPUUser ~ Data$hour | Data$Host, panel=panel.bwplot, horizontal=FALSE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="2770" LastEditorUserId="2770" LastEditorDisplayName="" LastEditDate="2011-02-24T18:59:41.103" LastActivityDate="2011-02-24T18:59:41.103" Title="How to aggregate by minute data for a week into hourly means?" Tags="&lt;r&gt;&lt;time-series&gt;&lt;aggregate&gt;" AnswerCount="3" CommentCount="4" FavoriteCount="1" />
  <row Id="7269" PostTypeId="2" ParentId="7263" CreationDate="2011-02-15T21:05:16.413" Score="5" ViewCount="" Body="&lt;p&gt;The terms are basically synonyms, but the usages are slightly different.  Think about the univariate case: you may talk about &quot;distributions&quot; in general, you might more specifically refer to &quot;univariate distributions&quot;, and you refer to &quot;the distribution of $X$&quot;. You &lt;em&gt;don't&lt;/em&gt; normally say &quot;the univariate distribution of $X$&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Similarly, in the multivariate case you may talk about &quot;distributions&quot; in general, you might more specifically refer to &quot;multivariate distribution&quot;, and you refer to &quot;the distribution of $(X,Y)$&quot; or &quot;the joint distribution of $X$ and $Y$&quot;. Thus the joint distribution of $X$ and $Y$ &lt;em&gt;is&lt;/em&gt; a multivariate distribution, but you &lt;em&gt;don't&lt;/em&gt; normally say &quot;the multivariate distribution of $(X,Y)$&quot; or &quot;the multivariate distribution of $X$ and $Y$&quot;.  &lt;/p&gt;&#xA;" OwnerUserId="89" LastEditorUserId="89" LastEditorDisplayName="" LastEditDate="2011-02-15T21:43:16.040" LastActivityDate="2011-02-15T21:43:16.040" CommentCount="1" />
  <row Id="7270" PostTypeId="1" CreationDate="2011-02-15T22:11:35.660" Score="6" ViewCount="200" Body="&lt;p&gt;I have a CSV file with 4 million edges of a directed network representing people communicating with each other (e.g. John sends a message to Mary, Mary sends a message to Ann, John sends &lt;em&gt;another&lt;/em&gt; message to Mary, etc.). I would like to do two things:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Find degree, betweeness and (maybe) eigenvector centrality measures for each person.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Get a visualization of the network.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I would like to do this on the command-line on a Linux server since my laptop does not have much power. I have R installed on that server and the statnet library. I found &lt;a href=&quot;http://www.cybaea.net/Blogs/Data/SNA-with-R-Loading-your-network-data.html&quot; rel=&quot;nofollow&quot;&gt;this 2009 post&lt;/a&gt; of someone more competent than me trying to do the same thing and having problems with it. So I was wondering if anyone else has any pointers on how to do this, preferably taking me step by step since I only know how to load the CSV file and nothing else. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just to give you an idea, this is how my CSV file looks like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;$ head comments.csv&#xD;&#xA;    &amp;quot;src&amp;quot;,&amp;quot;dest&amp;quot;&#xD;&#xA;    &amp;quot;6493&amp;quot;,&amp;quot;139&amp;quot;&#xD;&#xA;    &amp;quot;406705&amp;quot;,&amp;quot;369798&amp;quot;&#xD;&#xA;$ wc -l comments.csv &#xA;4210369 comments.csv&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1762" LastEditorUserId="1762" LastEditorDisplayName="" LastEditDate="2011-02-16T16:25:07.737" LastActivityDate="2011-02-17T16:06:24.620" Title="How to calculate centrality measures in a 4 million edge network using R?" Tags="&lt;r&gt;&lt;data-visualization&gt;&lt;networks&gt;" AnswerCount="4" CommentCount="3" FavoriteCount="2" />
  <row Id="7271" PostTypeId="1" AcceptedAnswerId="7284" CreationDate="2011-02-15T22:30:47.803" Score="4" ViewCount="108" Body="&lt;p&gt;Hellwig's method is a method of selection of variables in linear model. It is widely used in Poland, probably only in Poland because it is really hard to find it in any scientific paper written in English.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Description of method:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$m_{k}$ - set of variables in k'th combination (there are $2^{p}-1$ combinations, where p is number of variables)&lt;br&gt;&#xA;$r_{j}$ - correlation between $Y$ and $X_{j}$&lt;br&gt;&#xA;$r_{ij}$ - correlation between $X_{i}$ and $X_{j}$&lt;br&gt;&#xA;$H_{k}=\sum\limits_{j \in m_{k}}\frac{r_{j}^2}{\sum\limits_{i \in m_{k}}|r_{ij}|}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Choose the combination of variables with the highest $H_{k}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question&lt;/strong&gt;&lt;br&gt;&#xA;Is this method used anywhere outside the Poland?&lt;br&gt;&#xA;Does it have any scientific background? It seems that it based only on intuition that variables in a model should by highly correlated with $Y$ and poorly correlated with eachoter.&lt;/p&gt;&#xA;" OwnerUserId="1643" LastActivityDate="2011-02-16T08:49:24.693" Title="Hellwig's method of selection of variables" Tags="&lt;model-selection&gt;" AnswerCount="1" />
  <row Id="7272" PostTypeId="2" ParentId="7256" CreationDate="2011-02-15T23:43:27.807" Score="1" ViewCount="" Body="&lt;p&gt;Devil's advocate:  I could imagine the principal components differing depending on who's sampled.  I'd think this validity issue would take precedence over the precision issue Richard points out.&lt;/p&gt;&#xA;" OwnerUserId="2669" LastActivityDate="2011-02-15T23:43:27.807" />
  <row Id="7273" PostTypeId="2" ParentId="7250" CreationDate="2011-02-15T23:48:05.417" Score="5" ViewCount="" Body="&lt;p&gt;I did not grasp question 1 completely, but I'll attempt an answer. The plot of Q1 shows how the within sum of squares (wss) changes as cluster number changes. In this kind of plots you must look for the kinks in the graph, a kink at 5 indicates that it is a good idea to use 5 clusters. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;WSS has a relationship with your variables in the following sense, the formula for WSS is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\sum_{j} \sum_{x_i \in C_j} ||x_i - \mu_j||^2$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\mu_j$ is the mean point for cluster $j$ and $x_i$ is the $i$-th observation. We denote cluster j as $C_j$. WSS is sometimes interpreted as &quot;how similar are the points inside of each cluster&quot;. This similarity refers to the variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The answer to question 2 is this. What you are actually watching in the  &lt;code&gt;clusplot()&lt;/code&gt; is the plot of your observations in the principal plane. What this function is doing is calculating the principal component score for each of your observations, plotting those scores and coloring by cluster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Principal component analysis (PCA) is a dimension reduction technique; it &quot;summarizes&quot; the information of all variables into a couple of &quot;new&quot; variables called components. Each component is responsible of explaining certain percentage of the total variability. In the example you read &quot;This two components explain 73.95% of the total variability&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The function &lt;code&gt;clusplot()&lt;/code&gt; is used to identify the effectiveness of clustering. In case you have a successful clustering you will see that clusters are clearly separated in the principal plane. On the other hand, you will see the clusters merged in the principal plane when clustering is unsuccessful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For further reference on principal component analysis you may read wiki. if you want a book I suggest Modern Multivariate Techniques by Izenmann, there you will find PCA and k-means.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps :)&lt;/p&gt;&#xA;" OwnerUserId="2902" LastActivityDate="2011-02-15T23:48:05.417" CommentCount="11" />
  <row Id="7274" PostTypeId="2" ParentId="7270" CreationDate="2011-02-16T00:40:01.187" Score="3" ViewCount="" Body="&lt;p&gt;What you have is an edge list, which can be converted to a network object using the network library. Here is an example using fictitious data.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(network)&#xA;&#xA;src &amp;lt;- c(&quot;A&quot;, &quot;B&quot;, &quot;C&quot;, &quot;D&quot;, &quot;E&quot;, &quot;B&quot;, &quot;A&quot;, &quot;F&quot;)&#xA;dst &amp;lt;- c(&quot;B&quot;, &quot;E&quot;, &quot;A&quot;, &quot;B&quot;, &quot;B&quot;, &quot;A&quot;, &quot;F&quot;, &quot;A&quot;)&#xA;&#xA;edges &amp;lt;- cbind(src, dst)&#xA;Net &amp;lt;- as.network(edges, matrix.type = &quot;edgelist&quot;)&#xA;&#xA;summary(Net)&#xA;plot(Net)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, a warning is in order: you have a very large network and I am not sure a plot will be all that informative. It will probably look like a big ball of yarn. I am also not sure how well these libraries deal with such large datasets. I suggest you take a look at the documentation for the network, statnet, and ergm libraries. The &lt;em&gt;Journal of Statistical Software&lt;/em&gt; (v24/3) offers several articles covering these libraries. The issue can be found here:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.jstatsoft.org/v24&quot; rel=&quot;nofollow&quot;&gt;http://www.jstatsoft.org/v24&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="3265" LastActivityDate="2011-02-16T00:40:01.187" CommentCount="3" />
  <row Id="7275" PostTypeId="2" ParentId="4999" CreationDate="2011-02-16T01:41:06.600" Score="0" ViewCount="" Body="&lt;p&gt;I would guess it might be related to the problems that steepest descent has that the conjugate gradient method corrects. &lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2011-02-16T01:41:06.600" />
  <row Id="7276" PostTypeId="2" ParentId="4999" CreationDate="2011-02-16T02:07:31.860" Score="1" ViewCount="" Body="&lt;p&gt;I believe that in the specific case of L2 loss (ordinary linear regression), the convergence rate of coordinate descent will depend on the correlation structure of the predictors ($X_i$’s).  Consider the case where they are uncorrelated.  Then cyclic coordinate descent converges after one cycle.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another heuristic that has had more empirical evidence in its favor is the idea of active set convergence.  Rather than cycling through all coordinates, only cycle through the ones that are active ($i$’s where $\beta_i$ is non-zero) until convergence, then sweep through the all coordinates to update the active set.  Convergence occurs when the active set does not change.&lt;/p&gt;&#xA;" OwnerUserId="1670" LastActivityDate="2011-02-16T02:07:31.860" />
  <row Id="7277" PostTypeId="2" ParentId="7261" CreationDate="2011-02-16T03:22:22.480" Score="1" ViewCount="" Body="&lt;p&gt;I would like to include my experience in this discussion. I have seen that a generalized linear model (specifying an identity link function and a normal family distribution) is identical to a general linear model only when you use the maximum likelihood estimate as scale parameter method. Otherwise if &quot;fixed value = 1&quot; is chosen as scale parameter method you get very different p values. My experience suggest that usually &quot;fixed value = 1&quot; should be avoided. I'm curious to know if someone knows when it is appropriate to choose fixed value = 1 as scale parameter method.&#xA;Thanks in advance.&#xA;Mark &lt;/p&gt;&#xA;" OwnerUserId="3267" LastActivityDate="2011-02-16T03:22:22.480" CommentCount="1" />
  <row Id="7278" PostTypeId="1" AcceptedAnswerId="7280" CreationDate="2011-02-16T04:02:25.177" Score="2" ViewCount="171" Body="&lt;p&gt;Hi, I'm tasked with finding the MGF of a $\chi^2$ random variable.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think the way to do is is by using the fact that $\Sigma_{j=1}^{m} Z^2_j$ is a $\chi^2$ R.V. and that MGF of a sum is the product of the MGFs of the individual terms.  Although that may not be right and it may be $E(e^{tX})$ way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help would be appreciated.  I don't need it solved really just need to get down the track a little further.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="2387" LastEditorUserId="795" LastEditorDisplayName="" LastEditDate="2011-02-16T04:13:24.880" LastActivityDate="2011-02-16T16:58:31.657" Title="Finding the Moment Generating Function of Chi-Squared Dist" Tags="&lt;chi-squared&gt;&lt;moments&gt;" AnswerCount="1" CommentCount="4" />
  <row Id="7279" PostTypeId="1" CreationDate="2011-02-16T04:34:46.397" Score="4" ViewCount="195" Body="&lt;p&gt;What is paired t-test, and under which circumstances should I use paired t-test?&#xA;Is there any difference between paired t-test and pairwise t-test? Thanks.&lt;/p&gt;&#xA;" OwnerUserId="3269" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-02-16T18:46:55.840" LastActivityDate="2011-02-16T18:46:55.840" Title="Paired t-test and pairwise t-test" Tags="&lt;hypothesis-testing&gt;&lt;anova&gt;&lt;t-test&gt;" AnswerCount="1" />
  <row Id="7280" PostTypeId="2" ParentId="7278" CreationDate="2011-02-16T05:46:50.920" Score="1" ViewCount="" Body="&lt;p&gt;Yes, since $\chi^2$ is a sum of $Z_i^2$ the MGF is a product of individual summands. But then you need the MGF of $Z_i^2$ which is $\chi^2$ with 1 degree of freedom. The obvious way of calculating the MGF of $\chi^2$ is by integrating. It is not that hard:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Ee^{tX}=\frac{1}{2^{k/2}\Gamma(k/2)}\int_0^\infty x^{k/2-1}e^{-x(1/2-t)}dx$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now do the change of variables $y=x(1/2-t)$, then note that you get &lt;a href=&quot;http://en.wikipedia.org/wiki/Gamma_function&quot; rel=&quot;nofollow&quot;&gt;Gamma&lt;/a&gt; function and the result is yours. If you want deeper insights (if there are any) try asking at &lt;a href=&quot;http://math.stackexchange.com&quot;&gt;http://math.stackexchange.com&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="279" LastEditorDisplayName="" LastEditDate="2011-02-16T16:58:31.657" LastActivityDate="2011-02-16T16:58:31.657" CommentCount="1" />
  <row Id="7281" PostTypeId="2" ParentId="7279" CreationDate="2011-02-16T05:57:30.550" Score="3" ViewCount="" Body="&lt;p&gt;Roughly, &lt;a href=&quot;http://en.wikipedia.org/wiki/Paired_difference_test&quot; rel=&quot;nofollow&quot;&gt;paired t-test&lt;/a&gt; is a t-test in which each subject is compared with itself or, in &lt;a href=&quot;http://mathworld.wolfram.com/Pairedt-Test.html&quot; rel=&quot;nofollow&quot;&gt;other words&lt;/a&gt;, determines whether they differ from each other in a significant way under the assumptions that the paired differences are independent and identically normally distributed. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Pairwise t-test, on the other hand is a function in R which performs all possible pairwise comparisons. See &lt;a href=&quot;https://stat.ethz.ch/pipermail/r-help/2004-August/056136.html&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; discussion for more information&lt;/p&gt;&#xA;" OwnerUserId="1496" LastActivityDate="2011-02-16T05:57:30.550" />
  <row Id="7282" PostTypeId="2" ParentId="7270" CreationDate="2011-02-16T06:11:07.573" Score="2" ViewCount="" Body="&lt;p&gt;I don't think that R is a first choice here (maybe I'm wrong). You will need huge arrays here to index and prepare your networks files in the appropriate data format. First of all, I will try to use Jure's (Rob mention him in the post above) &lt;a href=&quot;http://goo.gl/L4jSO&quot; rel=&quot;nofollow&quot;&gt;SNAP&lt;/a&gt; library; it's written in C++ and works very well on large networks.&lt;/p&gt;&#xA;" OwnerUserId="609" LastActivityDate="2011-02-16T06:11:07.573" CommentCount="3" />
  <row Id="7283" PostTypeId="2" ParentId="7256" CreationDate="2011-02-16T06:21:29.233" Score="2" ViewCount="" Body="&lt;p&gt;Generally, you want your training and validation data sets be separate as much as possible. Ideally, the validation set data would have been obtained only after the model has been trained. If you perform dimensionality reduction before splitting your data to separate sets, you break this isolation between the training and the validation and you won't be sure whether the dimensionality reduction process was over-fitted until your model is tested in real life.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having said that, there are cases, where efficient separation to training, testing and validation sets is not feasible and other sampling techniques, such as cross validation, leave k out etc are used. In these cases reducing the dimensionality before the sampling might be the right approach.&lt;/p&gt;&#xA;" OwnerUserId="1496" LastActivityDate="2011-02-16T06:21:29.233" />
  <row Id="7284" PostTypeId="2" ParentId="7271" CreationDate="2011-02-16T08:49:24.693" Score="3" ViewCount="" Body="&lt;p&gt;After spending too long on web research, I'm pretty sure the source of 'Hellwig's method' is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hellwig, Zdzisław. &lt;em&gt;&lt;a href=&quot;http://www.worldcat.org/title/on-the-optimal-choice-of-predictors/oclc/217223232&quot; rel=&quot;nofollow&quot;&gt;On the optimal choice of predictors.&lt;/a&gt;&lt;/em&gt; Study VI in Z. Gostkowski (ed.): &lt;em&gt;Toward a system of quantitative indicators of components of human resources development&lt;/em&gt;; Paris: UNESCO, 1968; 23 pages. [&lt;a href=&quot;http://unesdoc.unesco.org/images/0015/001585/158559eb.pdf&quot; rel=&quot;nofollow&quot;&gt;pdf&lt;/a&gt;]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scholar.google.com/scholar?cites=16684078648545295641&amp;amp;as_sdt=2005&amp;amp;sciodt=0,5&amp;amp;hl=en&quot; rel=&quot;nofollow&quot;&gt;Google Scholar finds 3 papers that have cited it&lt;/a&gt;. None of them appear particularly noteworthy. So I think the answer to your first question is 'No'. As for your second question, I'll leave you to study the paper as i've spent far too long on this already. But from a skim, it appears the &lt;em&gt;motivation&lt;/em&gt; behind his method was to avoid calculations that were very tedious without an electronic computer:&lt;br&gt;&#xA;&quot;&lt;em&gt;... generally speaking one has to compute $2^n-1$ times the inverse matrices, which is of course an extremely dull perspective. The method we are going to present in this paper does not require finding inverse matrices.&lt;/em&gt;&quot; (p3-4).&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Biographical note:&lt;br&gt;&#xA;A little further googling reveals &lt;strong&gt;Zdzisław Hellwig&lt;/strong&gt; was born on 26 May 1925 in &lt;a href=&quot;http://en.wikipedia.org/wiki/Dobrzyca&quot; rel=&quot;nofollow&quot;&gt;Dobrzyca&lt;/a&gt;, Poland, and was for many years professor of statistics at the &lt;a href=&quot;http://www.ue.wroc.pl/english/&quot; rel=&quot;nofollow&quot;&gt;Wrocław University of Economics&lt;/a&gt;. There was &lt;a href=&quot;https://secure.imstat.org/meetings/2010.htm&quot; rel=&quot;nofollow&quot;&gt;a scientific meeting to honor his 85th birthday in November 2010&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-02-16T08:49:24.693" CommentCount="1" />
  <row Id="7285" PostTypeId="1" AcceptedAnswerId="7303" CreationDate="2011-02-16T08:54:55.157" Score="7" ViewCount="115" Body="&lt;h3&gt;My Aim:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;I'd like to a have a function that takes an email address and outputs a quasi-random number of 1, 2, 3, or 4.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;A little detail:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;By quasi-random number I mean that given a typical population of email addresses, the probabilities of getting a value of 1, 2, 3, or 4 are roughly equal, and that obvious systematic properties of the email address such as the domain name do not affect the probability of getting a value of 1, 2, 3, or 4.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;A little background:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;I have an online experiment written in &lt;a href=&quot;http://www.millisecond.com/support/docs/v1/index.htm&quot; rel=&quot;nofollow&quot;&gt;inquisit&lt;/a&gt; where participants log in on two occasions.&#xA;I want to randomly assign participants to one of four groups.&#xA;While this is easy to do for one session (I can just use a random number generator), I need some way of remembering the allocation across sessions. &#xA;Thus, I thought that I could extract a quasi-random group allocation from the participant email.&#xA;I'm also limited in the set of functions that I have at my disposal (&lt;a href=&quot;http://www.millisecond.com/support/docs/v3/html/language/expressions/functions.htm&quot; rel=&quot;nofollow&quot;&gt;see here for full list&lt;/a&gt;).&#xA;The string functions are: tolower toupper capitalize concat&#xA;   search replaceall contains startswith&#xA;   endswith substring trim trimright&#xA;   trimleft length format evaluate&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Initial Thoughts:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;I thought about trying to extract a set of features of the email address that returned a value of 1, 2, 3, or 4 with roughly equal probabilities.&#xA;Then, I could sum these properties and get the mod 4 plus 1 of that.&#xA;Thus, assuming something like the central limit theorem, I might get close.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Possible features that came to my mind:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;length of string&lt;/li&gt;&#xA;&lt;li&gt;position of first &quot;a&quot;, &quot;b&quot;, etc.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastEditorUserId="1381" LastEditorDisplayName="" LastEditDate="2011-02-16T17:21:01.200" LastActivityDate="2011-02-16T17:31:19.580" Title="From an email address to a quasi-random number" Tags="&lt;random-generation&gt;" AnswerCount="3" CommentCount="6" FavoriteCount="1" />
  <row Id="7286" PostTypeId="1" AcceptedAnswerId="7294" CreationDate="2011-02-16T10:20:43.147" Score="6" ViewCount="168" Body="&lt;p&gt;Hello everyone!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Random walk Metropolis-Hasitings with symmetric proposal&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$q(x|y)= g(|y-x|)$  has the property that the acceptance probability &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(accept\ y) = \min\{1, f(y)/f(x)\}$$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;does not depend on proposal $g(\cdot)$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does that mean that I can change the $g(\cdot)$ as a function of previous performance of the chain, without affecting the markovianity of the chain?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of particular interest to me is the adjustment of the scaling of Normal proposal as a function of acceptance rate. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would also greatly  appreciate if someone can point out to the adaptation algorithms used in practice for this type of problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[edit: Starting with the references given by robertsy and wok I found the following references on MH adaptive algorithms:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Andrieu, Christophe, and Éric Moulines. 2006.&lt;br&gt;&#xA;On the Ergodicity Properties of Some Adaptive MCMC Algorithms. The Annals of Applied Probability 16, no. 3: 1462-1505. &lt;a href=&quot;http://www.jstor.org/stable/25442804&quot; rel=&quot;nofollow&quot;&gt;http://www.jstor.org/stable/25442804&lt;/a&gt;.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Andrieu, Christophe, and Johannes Thoms.&lt;br&gt;&#xA;2008. A tutorial on adaptive MCMC. Statistics and Computing 18, no. 4 (12): 343-373. doi:10.1007/s11222-008-9110-y. &lt;a href=&quot;http://www.springerlink.com/content/979087678366r78v/&quot; rel=&quot;nofollow&quot;&gt;http://www.springerlink.com/content/979087678366r78v/&lt;/a&gt;.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Atchadé, Y., G. Fort, E. Moulines, and P. Priouret. 2009.&lt;br&gt;&#xA;Adaptive Markov Chain Monte Carlo: Theory and Methods. Preprint.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Atchadé, Yves. 2010.&lt;br&gt;&#xA;Limit theorems for some adaptive MCMC algorithms with subgeometric kernels. Bernoulli 16, no. 1 (February): 116-154. doi:10.3150/09-BEJ199.   &lt;a href=&quot;http://projecteuclid.org/DPubS?verb=Display&amp;amp;version=1.0&amp;amp;service=UI&amp;amp;handle=euclid.bj/1265984706&amp;amp;page=record&quot; rel=&quot;nofollow&quot;&gt;http://projecteuclid.org/DPubS?verb=Display&amp;amp;version=1.0&amp;amp;service=UI&amp;amp;handle=euclid.bj/1265984706&amp;amp;page=record&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cappé, O., S. J Godsill, and E. Moulines. 2007.&lt;br&gt;&#xA;An overview of existing methods and recent advances in sequential Monte Carlo. Proceedings of the IEEE 95, no. 5: 899-924.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Giordani, Paolo. 2010.&lt;br&gt;&#xA;Adaptive Independent Metropolis–Hastings by Fast Estimation of Mixtures of Normals. Journal of Computational and Graphical Statistics 19, no. 2 (6): 243-259. doi:10.1198/jcgs.2009.07174. &lt;a href=&quot;http://pubs.amstat.org/doi/abs/10.1198/jcgs.2009.07174&quot; rel=&quot;nofollow&quot;&gt;http://pubs.amstat.org/doi/abs/10.1198/jcgs.2009.07174&lt;/a&gt;.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Latuszynski, Krzysztof, Gareth O Roberts, and Jeffrey S Rosenthal. 2011.&lt;br&gt;&#xA;Adaptive Gibbs samplers and related MCMC methods. 1101.5838 (January 30). &lt;a href=&quot;http://arxiv.org/abs/1101.5838&quot; rel=&quot;nofollow&quot;&gt;http://arxiv.org/abs/1101.5838&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Pasarica, C., and A. Gelman. 2009.&lt;br&gt;&#xA;Adaptively scaling the Metropolis algorithm using expected squared jumped distance. Statistica Sinica.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Roberts, Gareth O. 2009.&lt;br&gt;&#xA;Examples of Adaptive MCMC. Journal of Computational and Graphical Statistics 18, no. 2 (6): 349-367. doi:10.1198/jcgs.2009.06134. &lt;a href=&quot;http://pubs.amstat.org/doi/abs/10.1198/jcgs.2009.06134&quot; rel=&quot;nofollow&quot;&gt;http://pubs.amstat.org/doi/abs/10.1198/jcgs.2009.06134&lt;/a&gt;.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;]&lt;/p&gt;&#xA;" OwnerUserId="1542" LastEditorUserId="1542" LastEditorDisplayName="" LastEditDate="2011-02-17T16:34:46.450" LastActivityDate="2011-02-17T16:34:46.450" Title="Can I change the proposal distribution in random-walk MH MCMC without affecting Markovianity?" Tags="&lt;mcmc&gt;&lt;metropolis-hastings&gt;" AnswerCount="2" CommentCount="4" FavoriteCount="1" />
  <row Id="7287" PostTypeId="2" ParentId="7285" CreationDate="2011-02-16T10:59:12.323" Score="3" ViewCount="" Body="&lt;p&gt;Why not just have a look-up table of numbers for each possible character in an email. Then concatenate the numbers to form a seed. For example, &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;A 1&#xA;B 2&#xA;C 3&#xA;....&#xA;@ 27&#xA;....&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So abc@ccc, would be converted to 12327333. This would give you a unique seed for each person. You would then use this to generate the 1, 2, 3, 4.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;From your question, it looks like you don't mind a &quot;quick and dirty solution&quot;. One problem with my solution is that email addresses aren't random - for example you will probably get very few email addresses that contain the letter &quot;z&quot;, but all email addresses contain &quot;@&quot;.&lt;/p&gt;&#xA;" OwnerUserId="8" LastActivityDate="2011-02-16T10:59:12.323" CommentCount="2" />
  <row Id="7288" PostTypeId="2" ParentId="7141" CreationDate="2011-02-16T12:31:45.973" Score="0" ViewCount="" Body="&lt;p&gt;Ok, I've thought of two possible ways to answer to this problem using Bayesian analysis.  I will assume $\sigma$ to be known throughout this answer.  First start with a &quot;baby&quot; case, where $n=2$ (or alternatively, only using the last two observations as a first approximation).  You would usually start this by assuming a &quot;flat&quot; prior for $\mu$, just proportion to 1.  But you have additional information, so we just restrict the prior to conform to this.  So the prior is:&#xA;$$f(\mu_1,\mu_2|\gamma) \propto I_{|\mu_1-\mu_2|\leq\gamma}$$&#xA;(the improper prior should be fine, because you are dealing with normal RVs, and you aren't dividing them)&#xA;Combining this prior with the likelihood, and integrating out $\mu_1$ gives (Writing $\phi(x)$ as standard normal pdf and $\Phi(x)$ as standard normal cdf):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$f(\mu_2 | X_1,X_2,\sigma,\gamma) \propto \phi\big(\frac{\mu_2-X_2}{\sigma}\big) \Bigg[\Phi\Big(\frac{\mu_2-X_1+\gamma}{\sigma}\Big)-\Phi\Big(\frac{\mu_2-X_1-\gamma}{\sigma}\Big)\Bigg]$$&#xA;So in order to calculate the &quot;p-value&quot; for the hypothesis, we need to take $Pr(\mu_2 &amp;gt; 0 |X_1,X_2,\sigma,\gamma)=P$.  This is given by the ratio of two integrals of the posterior:&#xA;$$P=\frac{\int_{-\frac{X_2}{\sigma}}^{\infty}\phi\big(y\big) \Bigg[\Phi\Big(y+\frac{X_2-X_1+\gamma}{\sigma}\Big)-\Phi\Big(y+\frac{X_2-X_1-\gamma}{\sigma}\Big)\Bigg]dy}{\int_{-\infty}^{\infty}\phi\big(z\big) \Bigg[\Phi\Big(z+\frac{X_2-X_1+\gamma}{\sigma}\Big)-\Phi\Big(z+\frac{X_2-X_1-\gamma}{\sigma}\Big)\Bigg]dz}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is beyond my abilities to do either of these integrals exactly, and even if it was possible, you probably would learn anything intuitive about the problem (except that the integral was friggin hard! you'd think it could be derived using something to do with convolutions, but I couldn't work it out).  So I would just numerically evaluate these two integrals.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the whole data set, you will almost surely need some kind of numerical technique, or analytic approximation.  This is a rather quick numerical technique.  Okay, so it basically goes like this: if you knew $\mu_1$, then you could generate a sample of the remaining $\mu_i$ values sequentially, using the uniform distribution $(\mu_{i}|\mu_{i-1}) \sim U(\mu_{i-1}-\gamma,\mu_{i-1}+\gamma)$.  An obvious way to sample $\mu_1$ is from a gaussian with large variance $\mu_1 \sim N(0,\delta^2)$ (&quot;large&quot; meaning relative to your data, say $\delta\approx 10\sigma$).  Use the notation $\mu_{i}^{(b)}$ for the $b$th sample of means $b=1,\dots,B$.  Now you calculate the total likelihood for each iteration.  This will be used as a weight:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$w^{(b)}=\prod_{i=1}^{n} \phi \Big(\frac{\mu_{i}^{(b)}-X_i}{\sigma}\Big)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then you take a &quot;weighted probability&quot; of the alternative hypothesis:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\hat{P}=\frac{\sum_{b=1}^{B}w^{(b)} I(\mu_{n}^{(b)}&amp;gt;0)}{\sum_{b=1}^{B}w^{(b)}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If $P$ is too big (in either case), then you reject the null hypothesis.  A standard value would be $P&amp;gt;0.95$.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastEditorUserId="2392" LastEditorDisplayName="" LastEditDate="2011-03-15T07:16:37.997" LastActivityDate="2011-03-15T07:16:37.997" />
  <row Id="7289" PostTypeId="2" ParentId="7223" CreationDate="2011-02-16T13:18:20.643" Score="1" ViewCount="" Body="&lt;p&gt;Regarding 1) Yes, you do lose this.  See e.g. Harrell Regression Modeling Strategies, a book published by Wiley or a paper I presented with David Cassell called &quot;Stopping Stepwise&quot; available e.g. www.nesug.org/proceedings/nesug07/sa/sa07.pdf&lt;/p&gt;&#xA;" OwnerUserId="686" LastActivityDate="2011-02-16T13:18:20.643" CommentCount="1" />
  <row Id="7290" PostTypeId="2" ParentId="7225" CreationDate="2011-02-16T14:10:31.810" Score="0" ViewCount="" Body="&lt;p&gt;My first reaction to Jelle's comments given is &quot;bias-schmias&quot;.  You have to be careful about what you mean by &quot;large amount of predictors&quot;.  This could be &quot;large&quot; with respect to:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The number of data points (&quot;big p small n&quot;)&lt;/li&gt;&#xA;&lt;li&gt;The amount of time you have to investigate the variables&lt;/li&gt;&#xA;&lt;li&gt;The computational cost of inverting a giant matrix&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;My reaction was based on &quot;large&quot; with respect to point 1.  This is because in this case it is usually worth the trade-off in bias for the reduction in variance that you get.  Bias is only important &quot;in-the-long-run&quot;.  So if you have a small sample, then who care's about &quot;the-long-run&quot;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having said all that above, $R^2$ is probably not a particularly good quantity to calculate, especially when you have lots of variables (because that's pretty much all $R^2$ tells you: you have lots of variables).  I would calculate something more like a &quot;prediction error&quot; using cross validation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ideally this &quot;prediction error&quot; should be based on the context of your modeling situation.  You basically want to answer the question &quot;How well does my model reproduce the data?&quot;.  The context of your situation should be able to tell you what &quot;how well&quot; means in the real world.  You then need to translate this into some sort of mathematical equation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I have no obvious context to go off from the question.  So a &quot;default&quot; would be something like PRESS:&#xA;$$PRESS=\sum_{i=1}^{N} (Y_{i}-\hat{Y}_{i,-i})^2$$&#xA;Where $\hat{Y}_{i,-i}$ is the predicted value for $Y_{i}$ for a model fitted without the ith data point ($Y_i$ doesn't influence the model parameters).  The terms in the summation are also known as &quot;deletion residuals&quot;.  If this is too computationally expensive to do $N$ model fits (although most programs usually gives you something like this with the standard output), then I would suggest grouping the data.  So you set the amount of time you are prepared to wait for $T$ (preferably not 0 ^_^), and then divide this by the time it takes to fit your model $M$.  This will give a total of $G=\frac{T}{M}$ re-fits, with a sample size of $N_{g}=\frac{N\times M}{T}$.&#xA;$$PRESS=\sum_{g=1}^{G}\sum_{i=1}^{N_{g}} (Y_{ig}-\hat{Y}_{ig,-g})^2$$&#xA;A way you can get an idea of how important each variable is, is to re-fit an ordinary regression (variables in the same order).  Then check proportionately how much each estimator has been shrunk towards zero $\frac{\beta_{LASSO}}{\beta_{UNCONSTRAINED}}$.  Lasso, and other constrained regression can be seen as &quot;smooth variable selection&quot;, because rather than adopt a binary &quot;in-or-out&quot; approach, each estimate is brought closer to zero, depending on how important it is for the model (as measured by the errors).&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-02-16T14:10:31.810" CommentCount="1" />
  <row Id="7291" PostTypeId="2" ParentId="7285" CreationDate="2011-02-16T14:25:46.130" Score="1" ViewCount="" Body="&lt;p&gt;You could try converting each character to an ascii number, multiplying them all together to force overflow, and then performing a modulus operation on the least significant digits.  If this is not pseudo-random enough, you can perform a bit-shift the numbers a bit...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-Ralph Winters&lt;/p&gt;&#xA;" OwnerUserId="3489" LastActivityDate="2011-02-16T14:25:46.130" CommentCount="2" />
  <row Id="7292" PostTypeId="1" CreationDate="2011-02-16T14:47:53.653" Score="7" ViewCount="122" Body="&lt;p&gt;If you can measure a time series of observations at any level of precision in time, and your goal of the study is to identify a relationship between X and Y, is there any empirical justification for choosing a specific level of aggregation over another, or should the choice be simply taken based on theory and/or practical limitations?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have three sub-questions to this main one:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Is any non-random variation in X or&#xA;Y within a larger level sufficient&#xA;reasoning to choose a smaller level&#xA;of aggregation (where non-random is&#xA;any temporal pattern of the&#xA;observations)?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Is any variation in the relationship between X and Y at a smaller level of aggregation sufficient reasoning to justify the smaller unit of analysis? If some variation is acceptable how does one decide how much variation is too much?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Can people cite arguments they feel are compelling/well defined for one unit of analysis over another, either for empirical reasons or for theoretical reasons?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I am well aware of the &lt;a href=&quot;http://en.wikipedia.org/wiki/Modifiable_areal_unit_problem&quot; rel=&quot;nofollow&quot;&gt;modifiable area unit problem&lt;/a&gt; in spatial analysis (&lt;a href=&quot;http://qmrg.org.uk/files/2008/11/38-maup-openshaw.pdf&quot; rel=&quot;nofollow&quot;&gt;Openshaw 1984&lt;/a&gt;). I don't claim to be expert on the material, but all I am to think so far of it is that a smaller unit of analysis is always better, as one is less likely to commit an ecological fallacy (&lt;a href=&quot;http://dx.doi.org/10.2307/2087176&quot; rel=&quot;nofollow&quot;&gt;Robinson 1950&lt;/a&gt;). If one has a directly pertinent reference or answer concerning aggregation geographical units I would appreciate that answer as well.&lt;/p&gt;&#xA;" OwnerUserId="1036" LastActivityDate="2011-02-17T11:23:36.643" Title="How do you choose a unit of analysis (level of aggregation) in a time series?" Tags="&lt;time-series&gt;&lt;aggregation&gt;" AnswerCount="1" FavoriteCount="4" />
  <row Id="7293" PostTypeId="1" AcceptedAnswerId="7296" CreationDate="2011-02-16T14:57:58.397" Score="4" ViewCount="56" Body="&lt;p&gt;The following problem comes from a max likelihood calculation for gaussian families, but is of independent interest.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possible to find a closed-form approximation for small values of $x$ for&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\text{det}(B + xI)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where I is the identity matrix and B is hermitian rank-deficient positive semidefinite?&lt;/p&gt;&#xA;" OwnerUserId="30" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-02-16T18:07:22.083" LastActivityDate="2011-02-16T18:07:22.083" Title="Determinant perturbation approximation" Tags="&lt;maximum-likelihood&gt;&lt;matrix&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="7294" PostTypeId="2" ParentId="7286" CreationDate="2011-02-16T15:15:42.040" Score="2" ViewCount="" Body="&lt;p&gt;I think that this &lt;a href=&quot;http://projecteuclid.org/DPubS?service=UI&amp;amp;version=1.0&amp;amp;verb=Display&amp;amp;handle=euclid.bj/1080222083&quot; rel=&quot;nofollow&quot;&gt;paper&lt;/a&gt; from Heikki Haario et al. will give you the answer you need. The markovianity of the chain is affected by the adaptation of the proposal density, because then a new proposed value depends not only of the previous one but on the whole chain. But it seems that the sequence has still the good properties if great care is taken. &lt;/p&gt;&#xA;" OwnerUserId="3108" LastEditorUserId="3108" LastEditorDisplayName="" LastEditDate="2011-02-16T15:35:20.447" LastActivityDate="2011-02-16T15:35:20.447" CommentCount="1" />
  <row Id="7295" PostTypeId="1" AcceptedAnswerId="7299" CreationDate="2011-02-16T15:20:21.510" Score="4" ViewCount="110" Body="&lt;p&gt;Currently i am using RF toolbox on MATLAB for a binary classification Problem&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data Set: 50000 samples  and more than 250 features&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So what should be the number of trees and randomly selected feature on each split to grow the trees?&#xA;can any other parameter greatly affect the results?&lt;/p&gt;&#xA;" OwnerUserId="2534" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-02-16T17:39:26.287" LastActivityDate="2011-02-16T17:39:26.287" Title="What should be the optimal parameters for Random Forest classifier?" Tags="&lt;machine-learning&gt;&lt;classification&gt;&lt;random-forest&gt;" AnswerCount="2" />
  <row Id="7296" PostTypeId="2" ParentId="7293" CreationDate="2011-02-16T15:23:49.660" Score="5" ViewCount="" Body="&lt;p&gt;I'll assume that you already know the eigenvalues of $B$. Since $B$ is symmetric positive semidefinite, it can be decomposed as&#xA;$$&#xD;&#xA;B = U D U^T&#xD;&#xA;$$&#xA;where $U$ is an orthogonal matrix and $D$ is the diagonal of nonnegative eigenvalues (some of which may be exactly zero).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now&#xA;$$&#xD;&#xA;B+xI = U D U^T + x U U^T = U (D + x I) U^T&#xD;&#xA;$$&#xA;and since the determinant of a matrix is the product of its eigenvalues and the determinant is distributive over matrix products, then&#xA;$$&#xD;&#xA;|B+xI| = |D+xI| = \prod_n (d_n + x)&#xD;&#xA;$$&#xA;where $d_n$ is the $n$th diagonal entry of $D$. &lt;/p&gt;&#xA;" OwnerUserId="2970" LastActivityDate="2011-02-16T15:23:49.660" CommentCount="2" />
  <row Id="7297" PostTypeId="1" CreationDate="2011-02-16T15:58:10.347" Score="4" ViewCount="116" Body="&lt;p&gt;I have a smooth but rather complex curve, sampled with a good frequency. I apply the discrete Fourier transform to it using the fast Fourier transform (FFT) algorithm and get its Fourier image. I need to find peaks on the resulting Fourier transform curve, but the image I get contains a substantial amount of noise, which is a real problem because peaks can't be clearly seen. So, my question is as follows: What are the sources of noise in the Fourier transform, and how it can be reduced?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Typical plots look like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Initial function&#xA;&lt;img src=&quot;http://i.imgur.com/nCb7B.png&quot; alt=&quot;Initial function&quot;&gt;&#xA;Initial function, narrow range&#xA;&lt;img src=&quot;http://i.imgur.com/vk3ld.png&quot; alt=&quot;Initial function, narrow range&quot;&gt;&#xA;Fourier transform&#xA;&lt;img src=&quot;http://i.imgur.com/3HpdW.png&quot; alt=&quot;Fourier transform&quot;&gt;&#xA;Fourier transform, narrow range&#xA;&lt;img src=&quot;http://i.imgur.com/DuwNL.png&quot; alt=&quot;Fourier transform, narrow range&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Before transformation a constant was subtracted from the the function so that it goes to zero.&lt;/p&gt;&#xA;" OwnerUserId="3272" LastEditorUserId="3272" LastEditorDisplayName="" LastEditDate="2011-02-16T22:57:59.120" LastActivityDate="2011-02-17T09:11:52.597" Title="Noise in the Fourier transform" Tags="&lt;data-transformation&gt;&lt;fourier-transform&gt;" AnswerCount="1" CommentCount="9" />
  <row Id="7298" PostTypeId="2" ParentId="7286" CreationDate="2011-02-16T16:18:35.263" Score="1" ViewCount="" Body="&lt;p&gt;You can improve the acceptance rate using &lt;strong&gt;&lt;em&gt;delayed rejection&lt;/em&gt;&lt;/strong&gt; as described in &lt;a href=&quot;http://scholar.google.fr/scholar?cluster=8248856686379090018&quot; rel=&quot;nofollow&quot;&gt;Tierney, Mira (1999)&lt;/a&gt;. It is based on a second proposal function and &lt;strong&gt;a second acceptance probability&lt;/strong&gt;, which guarantees the Markov chain is still reversible with the same invariant distribution: you have to be cautious since &quot;&lt;em&gt;it is easy to construct adaptive methods that might seem to work but in fact sample from the wrong distribution&lt;/em&gt;&quot;.&lt;/p&gt;&#xA;" OwnerUserId="1351" LastActivityDate="2011-02-16T16:18:35.263" />
  <row Id="7299" PostTypeId="2" ParentId="7295" CreationDate="2011-02-16T16:26:18.673" Score="2" ViewCount="" Body="&lt;p&gt;Pick a large number of trees, say 100. From what I have read on the Internet, pick $\sqrt{250}$ randomly selected features. However, in &lt;a href=&quot;http://scholar.google.fr/scholar?cluster=5141860777814015165&quot; rel=&quot;nofollow&quot;&gt;the original paper&lt;/a&gt;, Breiman used about the closest integer to $\frac{\log{M}}{\log{2}}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would say cross-validation is usually the key to finding optimal parameters, but I do not know enough about random forests.&lt;/p&gt;&#xA;" OwnerUserId="1351" LastEditorUserId="1351" LastEditorDisplayName="" LastEditDate="2011-02-16T16:34:31.833" LastActivityDate="2011-02-16T16:34:31.833" CommentCount="1" />
  <row Id="7300" PostTypeId="2" ParentId="7270" CreationDate="2011-02-16T16:52:02.890" Score="1" ViewCount="" Body="&lt;p&gt;Gephi ( &lt;a href=&quot;http://gephi.org/&quot; rel=&quot;nofollow&quot;&gt;http://gephi.org/&lt;/a&gt; ) might be an easy way to explore the data. You can almost certainly visualize it, and perform some calculations (though I have not used it for some time so I can't remember all the functions).&lt;/p&gt;&#xA;" OwnerUserId="2635" LastActivityDate="2011-02-16T16:52:02.890" />
  <row Id="7301" PostTypeId="2" ParentId="7259" CreationDate="2011-02-16T16:55:04.833" Score="0" ViewCount="" Body="&lt;p&gt;Are you familiar with &lt;a href=&quot;http://had.co.nz/ggplot2&quot; rel=&quot;nofollow&quot;&gt;ggplot2&lt;/a&gt;? I'm not certain I understand the question, but you can look at histograms colored by a certain paramater (&lt;a href=&quot;http://had.co.nz/ggplot2/geom_histogram.html&quot; rel=&quot;nofollow&quot;&gt;http://had.co.nz/ggplot2/geom_histogram.html&lt;/a&gt;), and also it has a useful facetting function (&lt;a href=&quot;http://had.co.nz/ggplot2/facet_grid.html&quot; rel=&quot;nofollow&quot;&gt;http://had.co.nz/ggplot2/facet_grid.html&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a very useful GUI developed called Deducer (&lt;a href=&quot;http://www.deducer.org/pmwiki/pmwiki.php?n=Main.DeducerManual&quot; rel=&quot;nofollow&quot;&gt;http://www.deducer.org/pmwiki/pmwiki.php?n=Main.DeducerManual&lt;/a&gt;) that is useful for exploring data using ggplot2. Once familiar with the structure of ggplot2 the GUI is no longer required but it can be great for exploring data.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Here is an example of the code, assuming your data frame is called 'dataframename':&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;p0 &amp;lt;- ggplot(dataframename, aes(factor(Iki3D.Rrp6D.sense), Iki3D.Rrp6D.antisense)) + facet_grid(.~Ago1D.antisense)&#xA;p0 &amp;lt;- p0 + geom_boxplot() + xlab('x axis') + ylab('y axis')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2635" LastEditorUserId="2635" LastEditorDisplayName="" LastEditDate="2011-02-16T19:50:51.380" LastActivityDate="2011-02-16T19:50:51.380" CommentCount="4" />
  <row Id="7302" PostTypeId="1" CreationDate="2011-02-16T17:29:10.427" Score="3" ViewCount="30" Body="&lt;p&gt;Hello,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;suppose you might buy some software to be used in a ML/NLP/DM laboratory. What software package would you ask for? Let's say: MATLAB (with some toolboxes), SPSS, and what else?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know that there is a lot of free software one can use like R, Weka, Rapidminer, python packages, and so on. However, the above question targets commercial software.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="976" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-02-16T18:00:10.717" LastActivityDate="2011-02-16T18:00:10.717" CommunityOwnedDate="2011-02-16T18:00:20.950" Title="Software for ML/NLP/DM laboratory" Tags="&lt;machine-learning&gt;&lt;data-mining&gt;&lt;software&gt;" />
  <row Id="7303" PostTypeId="2" ParentId="7285" CreationDate="2011-02-16T17:31:19.580" Score="3" ViewCount="" Body="&lt;p&gt;Look up hash functions, for example at &lt;a href=&quot;http://en.wikipedia.org/wiki/Hash_function&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Hash_function&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="247" LastActivityDate="2011-02-16T17:31:19.580" />
  <row Id="7304" PostTypeId="2" ParentId="7295" CreationDate="2011-02-16T17:39:07.300" Score="4" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Number of trees&lt;/strong&gt; the bigger, the better. You almost can't overshoot with this parameter, but of course the upper limit depends on the computational time you want to spend on RF.&lt;br&gt;&#xA;The good idea is to make a long forest first and then see (I hope it is available in MATLAB implementation) when the OOB accuracy converges.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Number of tried attributes&lt;/strong&gt; the default is square root of the whole number of attributes, yet usually the forest is not very sensitive about the value of this parameter -- in fact it is rarely optimized, especially because stochastic aspect of RF may introduce larger variations.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2011-02-16T17:39:07.300" CommentCount="1" />
  <row Id="7305" PostTypeId="2" ParentId="7293" CreationDate="2011-02-16T18:01:53.177" Score="3" ViewCount="" Body="&lt;p&gt;I second @cardinal's answer, but provide a simple trick: If $p(z)$ is a polynomial (with integer powers), and $\mathbf{v}, \lambda$ are eigenvector and corresponding eigenvalue of matrix $M$, then $\mathbf{v}, p(\lambda)$ are eigenvector and corresponding eigenvalue of $p(M)$. The proof is a simple exercise. The polynomial $p$ may contain negative powers of $z$ and a constant term, which in the case of $p(M)$ corresponds to adding the constant times the identity matrix.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since the determinant is the product of the eigenvalues, the determinant of $A = p(B)$, where $p(z) = z^1 + x$ is the product $\prod_i \left(\lambda_i + x\right)$, where $\lambda_i$ are the eigenvalues of $B$. You can also use this trick to find the trace of $p(B)$, of course, but it is overkill!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This polynomial trick is a classic in numerical analysis, used, for example, to prove convergence of the Gauss-Seidel method. See Cheney &amp;amp; Kincaid, or &lt;a href=&quot;http://stats.stackexchange.com/questions/2615/analytical-solutions-to-limits-of-correlation-stress-testing/2814#2814&quot;&gt;my answer&lt;/a&gt; to another question involving this trick.&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2011-02-16T18:01:53.177" CommentCount="1" />
  <row Id="7306" PostTypeId="2" ParentId="7268" CreationDate="2011-02-16T19:21:35.107" Score="4" ViewCount="" Body="&lt;p&gt;Here is one approach using cut() to create the appropriate hourly factors and ddply() from the plyr library for calculating the means.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(lattice)&#xA;library(plyr)&#xA;&#xA;## Create a record and some random data for every 5 seconds &#xA;## over two days for two hosts.&#xA;dates &amp;lt;- seq(as.POSIXct(&quot;2011-01-01 00:00:00&quot;, tz = &quot;GMT&quot;),&#xA;             as.POSIXct(&quot;2011-01-02 23:59:55&quot;, tz = &quot;GMT&quot;),&#xA;             by = 5)&#xA;hosts &amp;lt;- c(rep(&quot;host1&quot;, length(dates)), rep(&quot;host2&quot;, &#xA;           length(dates)))&#xA;x1    &amp;lt;- sample(0:20, 2*length(dates), replace = TRUE)&#xA;x2    &amp;lt;- rpois(2*length(dates), 2)&#xA;Data  &amp;lt;- data.frame(dates = dates, hosts = hosts, x1 = x1, &#xA;                    x2 = x2)&#xA;&#xA;## Calculate the mean for every hour using cut() to define &#xA;## the factors and ddply() to calculate the means. &#xA;## getmeans() is applied for each unique combination of the&#xA;## hosts and hour factors.&#xA;getmeans  &amp;lt;- function(Df) c(x1 = mean(Df$x1), &#xA;                            x2 = mean(Df$x2))&#xA;Data$hour &amp;lt;- cut(Data$dates, breaks = &quot;hour&quot;)&#xA;Means &amp;lt;- ddply(Data, .(hosts, hour), getmeans)&#xA;Means$hour &amp;lt;- as.POSIXct(Means$hour, tz = &quot;GMT&quot;)&#xA;&#xA;## A plot for each host.&#xA;xyplot(x1 ~ hour | hosts, data = Means, type = &quot;o&quot;,&#xA;       scales = list(x = list(relation = &quot;free&quot;, rot = 90)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="3265" LastActivityDate="2011-02-16T19:21:35.107" CommentCount="3" />
  <row Id="7307" PostTypeId="1" CreationDate="2011-02-16T19:33:34.640" Score="4" ViewCount="224" Body="&lt;p&gt;Can somebody explain me clear the mathematical logic that would link two statements (a) and (b) together? Let us have a set of values (some distribution). Now,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;a) Median does not depend on every value [it just depends on one or two middle values];&#xA;b) Median is the locus of minimal sum-of-absolute-deviations from it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And likewise, and in contrast,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;a) (Arithmetic) mean depends on every value;&#xA;b) Mean is the locus of minimal sum-of-squared-deviations from it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My grasp of it is intuitive so far. Thank you very much.&lt;/p&gt;&#xA;" OwnerUserId="3277" LastActivityDate="2011-02-17T00:59:26.753" Title="Mean and Median properties" Tags="&lt;mean&gt;&lt;median&gt;" AnswerCount="3" CommentCount="2" />
  <row Id="7308" PostTypeId="1" AcceptedAnswerId="7629" CreationDate="2011-02-16T19:52:33.287" Score="7" ViewCount="250" Body="&lt;p&gt;Jeffrey Wooldridge in his &lt;em&gt;Econometric Analysis of Cross Section and Panel Data&lt;/em&gt; (page 357) says that the empirical Hessian &quot;is not guaranteed to be positive definite, or even positive semidefinite, for the particular sample we are working with.&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This seems wrong to me as (numerical problems apart) the Hessian must be positive semidefinite as a result of the definition of the M-estimator as the value of the parameter which minimizes the objective function for the given sample and the well-known fact that at a (local) minimum the Hessian is positive semidefinite.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is my argument right?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;[EDIT: The statement has been removed in the 2nd ed. of the book. See comment.]&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;BACKGROUND&#xA;Suppose that $\widehat \theta_N$ is an estimator obtained by minimizing&#xA;$${1 \over N}\sum_{i=1}^N q(w_i,\theta),$$&#xA;where $w_i$ denotes the $i$-th observation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's denote the Hessian of $q$ by $H$,&#xA;$$H(q,\theta)_{ij}=\frac{\partial^2 q}{\partial \theta_i \partial \theta_j}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The asymptotic covariance of $\widehat \theta_n$ involves $E[H(q,\theta_0)]$ where $\theta_0$ is the true parameter value. One way to estimate it is to use the empirical Hesssian&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\widehat H=\frac{1}{N}\sum_{i=1}^N H(w_i,\widehat \theta_n)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is the definiteness of $\widehat H$ which is in question.&lt;/p&gt;&#xA;" OwnerUserId="1393" LastEditorUserId="1393" LastEditorDisplayName="" LastEditDate="2011-02-28T04:09:22.650" LastActivityDate="2011-02-28T04:09:22.650" Title="Can the empirical Hessian of an M-estimator be indefinite?" Tags="&lt;estimation&gt;&lt;maximum-likelihood&gt;&lt;econometrics&gt;&lt;asymptotics&gt;" AnswerCount="3" CommentCount="9" FavoriteCount="2" />
  <row Id="7309" PostTypeId="2" ParentId="7259" CreationDate="2011-02-16T20:01:30.047" Score="4" ViewCount="" Body="&lt;p&gt;Sam,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think I understood what you are after, so let me know if I've misinterpreted anything:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;You want a separate &lt;code&gt;box_plot&lt;/code&gt; for the ratio of each pairs of columns. There are 15 ratios we are interested in...(column 6 / column 7, column 8 / column 9, etc.)&lt;/li&gt;&#xA;&lt;li&gt;This plot should have a separate &quot;window&quot; or facet for each annotation, for which there are 23 different annotations.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Assuming both of those are right, I think this will give you what you are after. First, we will make the 15 new ratio columns with a for-loop and some indexing. After we make these 15 new columns, we will &lt;code&gt;melt&lt;/code&gt; the data into long format for easy plotting with &lt;code&gt;ggplot2&lt;/code&gt;. Since we are only interested in the columns &lt;code&gt;annotation&lt;/code&gt; and the new ratio columns, we'll specify those in the call to &lt;code&gt;melt&lt;/code&gt;. Then it is a relatively straight forward call to &lt;code&gt;ggplot&lt;/code&gt; to specify the axes and faceting variable. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;These plots don't make much sense with 10 rows of data, but I think it will look better with your full dataset.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(ggplot2)&#xA;&#xA;#EDIT: this removes the call to cbind which should improve performance.&#xA;for (i in seq(6, ncol(df), by = 2)) {&#xA;    df[,  paste(i, i+1, sep = &quot;_&quot;, collapse = &quot;&quot;)] &amp;lt;- df[, i ] / df[, i + 1 ]&#xA;&#xA;}&#xA;&#xA;df.m &amp;lt;- melt(df, id.vars = &quot;annotation&quot;, measure.vars = 36:ncol(df))&#xA;#Note that we use the column name for the id.vars and the column order for&#xA;#the measure.vars. In the case of the latter, this is simply to save on &#xA;#typing.&#xA;&#xA;ggplot(data = df.m, aes(x = variable, y = value)) + &#xA;    geom_boxplot() + &#xA;    facet_wrap(~ annotation) +&#xA;    coord_flip()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="696" LastEditorUserId="696" LastEditorDisplayName="" LastEditDate="2011-02-16T22:11:00.050" LastActivityDate="2011-02-16T22:11:00.050" CommentCount="1" />
  <row Id="7310" PostTypeId="2" ParentId="7307" CreationDate="2011-02-16T20:04:50.077" Score="0" ViewCount="" Body="&lt;p&gt;It is worthwhile to review an older version of the same question:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://stats.stackexchange.com/questions/2547/why-is-median-age-a-better-statistic-than-mean-age/2550#2550&quot;&gt;Why is median age a better statistic than mean age?&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And, a description of Robust Statistics:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Robust_statistics&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Robust_statistics&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2775" LastActivityDate="2011-02-16T20:04:50.077" />
  <row Id="7311" PostTypeId="2" ParentId="7256" CreationDate="2011-02-16T20:14:40.750" Score="0" ViewCount="" Body="&lt;p&gt;You should perform sampling and dimensionality reduction in combination.&#xA;The best way to do this is undersample the majority class, and run a decision tree.  It is the best variable selector you can imagine.&#xA;Perform this a number of times (each time another sample).  The result will be a number of list of candidate predictors.&#xA;And ... yes : combination of your decision trees is already a great model.&#xA;Find out why decision trees is the best data mining algorithm at &lt;a href=&quot;http://bit.ly/a2qDWJ&quot; rel=&quot;nofollow&quot;&gt;http://bit.ly/a2qDWJ&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="3278" LastActivityDate="2011-02-16T20:14:40.750" CommentCount="1" />
  <row Id="7312" PostTypeId="2" ParentId="7307" CreationDate="2011-02-16T20:19:38.270" Score="2" ViewCount="" Body="&lt;p&gt;Hi !&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Roughly speaking, the median is the &quot;middle value&quot;. Now, if you change the highest  value (which is supposed to be positive here) from $x_{(n)}$ to $2 * x_{(n)}$, say, it does not change the median. But it does change the arithmetic mean. This shows, in simple terms, that the median does not depend on every value while the mean does. Actually, the median only depends on the ranks. The mathematical logic behind this simply arises from the mathematical definitions of the median and the mean.&lt;/li&gt;&#xA;&lt;li&gt;Now, it can be shown that, for any $ a \in \mathbb{R}$ &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;$\sum_{i=1}^{n} |x_{i} - median| \leq \sum_{i=1}^{n} |x_{i} - a|$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\sum_{i=1}^{n} (x_{i} - mean)^{2} \leq \sum_{i=1}^{n} (x_{i} - a)^{2}$&lt;/p&gt;&#xA;" OwnerUserId="3019" LastEditorUserId="3019" LastEditorDisplayName="" LastEditDate="2011-02-16T20:26:15.590" LastActivityDate="2011-02-16T20:26:15.590" CommentCount="4" />
  <row Id="7313" PostTypeId="1" CreationDate="2011-02-16T21:18:23.097" Score="2" ViewCount="50" Body="&lt;p&gt;Per my &lt;a href=&quot;http://stats.stackexchange.com/q/7115/1026&quot;&gt;earlier question&lt;/a&gt; I'm trying to find a reasonable metric for the semantic distance between two short text strings.  One metric mentioned in the answers of that question was to use shortest hypernym path to create a metric for phrases.  So for instance, if I was to find the semantic distance between pig and dog, I could ask &lt;a href=&quot;http://wordnet.princeton.edu/&quot; rel=&quot;nofollow&quot;&gt;WordNet&lt;/a&gt; for all of their hypernyms:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;pig&lt;/strong&gt;=&gt; swine=&gt; even-toed ungulate=&gt; hoofed mammal=&gt; placental mammal=&gt; mammal=&gt; vertebrate=&gt; chordate=&gt; animal=&gt; organism=&gt; living thing=&gt; object=&gt; physical entity=&gt; entity&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;dog&lt;/strong&gt;=&gt; canine=&gt; carnivore=&gt; placental mammal=&gt; mammal=&gt; vertebrate=&gt; chordate=&gt; animal=&gt; organism=&gt; living thing=&gt; object=&gt; physical entity=&gt; entity&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and I would find that the shortest path between pig and dog is 8 jumps - so semantic distance = 8.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I wanted to extend this  concept to entire phrases, then perhaps I could (naively) find the average distance between all word pairs in the phrases.  (Obviously, one should be able to find something much better than this.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My question:&lt;/strong&gt; I'm sure someone has thought of this before.  Where should I look in literature to find more information.  And what are the hidden gotchas when using such an approach.&lt;/p&gt;&#xA;" OwnerUserId="1026" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-02-16T21:51:55.563" LastActivityDate="2011-02-16T21:51:55.563" Title="Closest distance in hypernym tree as measure of semantic distance between phrases" Tags="&lt;text-mining&gt;&lt;distance-functions&gt;" />
  <row Id="7314" PostTypeId="1" CreationDate="2011-02-16T22:30:17.503" Score="2" ViewCount="221" Body="&lt;p&gt;Greetings, &#xA;Winbugs seem to support either stochastic or deterministic relationship between variables. &#xA;However, many Bayesian Networks represent relationships between variables using conditional probability tables. The &quot;visit to Asia&quot;, &quot;burglar alarm&quot;, &quot;smoking &amp;amp; cancer&quot; examples are classic introductory material. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, conditional probability tables used in this approach does not correspond to stochastic or deterministic relationships in Winbugs. Google searches bring examples of Gibbs sampling on Bayesian Networks, but these are mostly algorithms that I'd have to implement in either R or another programming language (for example see: &lt;a href=&quot;http://www-users.cselabs.umn.edu/classes/Spring-2010/csci5512/notes/gibbs.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www-users.cselabs.umn.edu/classes/Spring-2010/csci5512/notes/gibbs.pdf&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a way of using Winbugs for complex Bayesian Network inference? I need to express causal relationships between different variables (a continuous variable and a discrete one, a categorical one etc), so I need to perform MCMC based inference on hybrid Bayesian networks. I am not so sure if I can use Winbugs for this&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, would it make sense to try to transform the conditional probability table based, or causal relationships into a stochastic relationships? If I turn CPT based relationships between variables into regression for example, would that be an acceptable way of performing inference on Bayesian Networks using Winbugs?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Best Regards&#xA;Seref&lt;/p&gt;&#xA;" OwnerUserId="3280" LastActivityDate="2011-04-08T11:45:51.657" ClosedDate="2011-04-08T15:18:21.083" Title="How can I represent Conditional Probability Table based Bayesian Networks in Winbugs?" Tags="&lt;bayesian&gt;&lt;causal-inference&gt;&lt;bugs&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="7315" PostTypeId="2" ParentId="7307" CreationDate="2011-02-16T23:13:35.653" Score="3" ViewCount="" Body="&lt;p&gt;For the computation of the median, let $x_1,x_2,\ldots,x_n$ be the data. Assume, for simplicity, that $n$ is even, and the points are distinct! Let $y$ be some number. Let $f(y)$ be the 'sum-of-absolute deviations' of $y$ to the points $x_i$. This means that $f(y) = |x_1 - y| + |x_2 - y| + \ldots + |x_n - y|$. Your goal is to find the $y$ that minimizes $f(y)$. Let $l$ be the number of the $x_i$ that are less than or exactly equal to $y$ at a given point in time, and let $r = n - l$ be the number that are strictly greater than $y$. Pretend you are 'moving $y$ to the right', that is, increase $y$ slightly. What happens to $f(y)$? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose you add an amount of $\Delta y$ to $y$. For those $x_i$ which are less than or equal to $y$, we have $|x_i - y|$ increases by $\Delta y$. And for those greater than $y$, we have $|x_i - y|$ decreases by $\Delta y$. (This assumes $\Delta y$ is so small that $y$ does not cross any of the points). Thus the change in $f(y)$ is $l\Delta y - r \Delta y = (l-r)\Delta y$. Note that this change in $f(y)$ does not depend on the values of the $x_i$ but only on the number to the left and right of $y$. By definition, $y$ is a median value when moving it to the left or right does not increase or decrease $f(y)$.  This would mean that $l-r = 0$, and thus the number of $x_i$ to the left of $y$ is equal to the number to the right of $y$. And thus the median does not depend on the values of $x_i$, just their locations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;edit&lt;/strong&gt; For the mean: the function $f(y)$ becomes $f(y) = (x_1 - y)^2 + \ldots + (x_n - y)^2$. Clearly the change in $f(y)$ for a small change in $y$ now depends on the magnitudes of the $x_i$, not just the number to the left and right of $y$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that this business about the 'small change' is just covert talk for the derivative of $f(y)$...&lt;/p&gt;&#xA;" OwnerUserId="795" LastEditorUserId="795" LastEditorDisplayName="" LastEditDate="2011-02-17T00:59:26.753" LastActivityDate="2011-02-17T00:59:26.753" CommentCount="4" />
  <row Id="7316" PostTypeId="1" AcceptedAnswerId="7317" CreationDate="2011-02-17T03:01:41.173" Score="3" ViewCount="261" Body="&lt;p&gt;I have data with many correlated features, and I want to start by reducing the features with a smooth basis function, before running an LDA.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm trying to use natural cubic splines in the &lt;code&gt;splines&lt;/code&gt; package with the &lt;code&gt;ns&lt;/code&gt; function.  How do I go about assigning the knots?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's the basic R code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(splines)&#xA;lda.pred &amp;lt;- lda(y ~ ns(x, knots=5))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But I have no idea about how to chose the knots in &lt;code&gt;ns&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="988" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-02-17T22:06:07.180" LastActivityDate="2011-02-17T22:06:07.180" Title="Setting knots in natural cubic splines in R" Tags="&lt;r&gt;&lt;splines&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="1" />
  <row Id="7317" PostTypeId="2" ParentId="7316" CreationDate="2011-02-17T04:00:30.850" Score="7" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;How to specify the knots in R&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;code&gt;ns&lt;/code&gt; function generates a &lt;em&gt;natural regression spline&lt;/em&gt; basis given an input vector. The knots can be specified either via a degrees-of-freedom argument &lt;code&gt;df&lt;/code&gt; which takes an integer argument or via a knots argument &lt;code&gt;knots&lt;/code&gt; which takes a &lt;strong&gt;&lt;em&gt;vector&lt;/em&gt;&lt;/strong&gt; giving the desired placement of the knots. Note that in the code you've written&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(splines)&#xA;lda.pred &amp;lt;- lda(y ~ ns(x, knots=5))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;you have not requested five knots, but rather have requested a &lt;em&gt;single&lt;/em&gt; (interior) knot at &lt;em&gt;location&lt;/em&gt; 5.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you use the &lt;code&gt;df&lt;/code&gt; argument, then the interior knots will be selected based on quantiles of the vector &lt;code&gt;x&lt;/code&gt;. For example, if you make the call&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ns(x, df=5)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then the basis will include two boundary knots and &lt;strong&gt;4&lt;/strong&gt; internal knots, places at the 20th, 40th, 60th, and 80th quantiles of &lt;code&gt;x&lt;/code&gt;. The boundary knots, by default, are placed at the min and max of &lt;code&gt;x&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an example to specify the locations of the knots&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x &amp;lt;- 0:100&#xA;ns(x, knots=c(20,35,50)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you were to instead call &lt;code&gt;ns(x, df=4)&lt;/code&gt;, you would end up with 3 internal knots at locations 25, 50, and 75, respectively.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also specify whether you want an intercept term. Normally this &lt;strong&gt;&lt;em&gt;isn't&lt;/em&gt;&lt;/strong&gt; specified since &lt;code&gt;ns&lt;/code&gt; is most often used in conjunction with &lt;code&gt;lm&lt;/code&gt;, which includes an intercept implicitly (unless forced not to). If you use &lt;code&gt;intercept=TRUE&lt;/code&gt; in your call to &lt;code&gt;ns&lt;/code&gt;, make sure you know &lt;strong&gt;why&lt;/strong&gt; you're doing so, since if you do this and then call &lt;code&gt;lm&lt;/code&gt; naively, the design matrix will end up being rank deficient.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Strategies for placing knots&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Knots are most commonly placed at quantiles, like the default behavior of &lt;code&gt;ns&lt;/code&gt;. The intuition is that if you have lots of data clustered close together, then you might want more knots there to model any potential nonlinearities in that region. But, that doesn't mean this is either (a) the only choice or (b) the best choice.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other choices can obviously be made and are domain-specific. Looking at histograms and density estimates of your predictors may provide clues as to where knots are needed, unless there is some &quot;canonical&quot; choice given your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In terms of interpreting regressions, I would note that, while you can certainly &quot;play around&quot; with knot placement, you should realize that you incur a model-selection penalty for this that you should be careful to evaluate and adjust any inferences as a result.&lt;/p&gt;&#xA;" OwnerUserId="2970" LastActivityDate="2011-02-17T04:00:30.850" />
  <row Id="7318" PostTypeId="1" CreationDate="2011-02-17T04:34:41.540" Score="4" ViewCount="311" Body="&lt;p&gt;I have an experiment producing results (dependent variables) that don't pass tests of normality, thus I am testing hypotheses using non-parametric tests.&#xA;My DVs are continuous, while my factors (independent variables) are ordinal or nominal.&#xA;I've been using the Kruskal-Wallis test and Friedman test (using Matlab).&#xA;Most of the time I am only interested I testing 2 IVs for significant effects, though sometimes I test 3.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to know whether there are any significant interaction effects on the DV between my IVs.  Normally I'd use a 2-way ANOVA to do this, however that's not appropriate given the non-normal distributions.  I don't wish to use transformation of my IVs, nor go ahead with ANOVA despite non-normality.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I find which interaction effects are significant?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What non-parametric test could I use?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope someone can help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Nick&lt;/p&gt;&#xA;" OwnerUserId="3285" LastActivityDate="2011-03-23T22:47:02.427" Title="Which non-parametric test can I use to identify significant interactions of independent variables?" Tags="&lt;hypothesis-testing&gt;&lt;anova&gt;&lt;statistical-significance&gt;&lt;nonparametric&gt;&lt;interaction&gt;" AnswerCount="1" CommentCount="5" FavoriteCount="0" />
  <row Id="7319" PostTypeId="1" AcceptedAnswerId="7331" CreationDate="2011-02-17T04:50:17.777" Score="5" ViewCount="209" Body="&lt;p&gt;I am new to statistics, so pardon any mistakes in my question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have two time series $X_i$ and $Y_i$. Assuming that they're stationary AR(1) processes with possibly different means, how do I test for difference of means?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found this link (but tests only one sample) which discusses using gls: &lt;a href=&quot;https://stat.ethz.ch/pipermail/r-help/2006-May/105495.html&quot; rel=&quot;nofollow&quot;&gt;https://stat.ethz.ch/pipermail/r-help/2006-May/105495.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I am wondering how to extend this to multiple samples (use a dummy variable?) or if there are any other methods (preferably in R).&lt;/p&gt;&#xA;" OwnerUserId="3286" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-02-17T08:49:27.607" LastActivityDate="2011-02-20T12:25:06.973" Title="Two sample t-test for data (maybe time series) with autocorrelation?" Tags="&lt;hypothesis-testing&gt;&lt;mean&gt;&lt;autocorrelation&gt;" AnswerCount="1" />
  <row Id="7320" PostTypeId="2" ParentId="5430" CreationDate="2011-02-17T04:51:40.427" Score="0" ViewCount="" Body="&lt;p&gt;You could simply smooth the data and find the peaks. &#xA;Since there are presumably several pertinent, distinct (larger) objects amongst many irrelevant, indistinct (smaller) objects providing the noisy distance environment, you could probably assume that the distance distribution of pertinent objects is likely to be uniform, no?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you can't rely on any particular distance distribution for pertinent objects, then fitting distribution functions won't help at all.&#xA;Thus you're left with identifying real peaks amongst false peaks.&#xA;A low-pass filter can help with that - even as simple as a moving average filter.  You could tune the filter using the likely range of distances of each pertinent object (e.g. a non-uniform object of about 2 metres in size might give peaks that vary within a 2m range).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There may also be further machine learning approaches(?)&lt;/p&gt;&#xA;" OwnerUserId="3285" LastActivityDate="2011-02-17T04:51:40.427" />
  <row Id="7321" PostTypeId="1" CreationDate="2011-02-17T05:29:19.383" Score="5" ViewCount="78" Body="&lt;p&gt;Hi, sorry for the confusing title. This one might take a bit of explaining as I'm not very expert in the relevant statistical terminology.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider a single x axis representing points along a line in space. I've got a set of data (in this case the receptive fields of hippocampal place cells recorded as a rat runs along a linear track), which are themselves spatially extensive, ie. if displayed graphically against the axis they would look like a series of horizontal bars, distributed at different points along the axis.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  ____   &#xA; ____&#xA;           ____ fields&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;&lt;strong&gt;&lt;/em&gt;__&lt;em&gt;_&lt;/em&gt;__&lt;em&gt;_&lt;/em&gt;__&lt;em&gt;_&lt;/em&gt;__&lt;em&gt;_&lt;/em&gt;__&lt;em&gt;_&lt;/em&gt;____&lt;/strong&gt;_axis&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I want to do is to define a point of interest on the line, and determine whether there are significantly more fields overlapping this point than elsewhere. I've achieved a simpler version of this by finding the centre of the fields and then measuring distances from this point and another arbitrarily defined point, but this approach loses all of the information that comes from the fact that they are spatially extensive.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone point me in the right general direction? Even knowing the broad class of statistical test I should use for this kind of situation would be a big help!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="3288" LastActivityDate="2011-05-26T00:50:19.937" Title="I need a statistical test to determine whether a group of spatially extensive data is centred around one point in space..." Tags="&lt;data-visualization&gt;&lt;spatial&gt;&lt;continuous-data&gt;" AnswerCount="1" CommentCount="3" FavoriteCount="1" />
  <row Id="7322" PostTypeId="1" AcceptedAnswerId="7329" CreationDate="2011-02-17T06:46:02.780" Score="4" ViewCount="229" Body="&lt;p&gt;I am working with the dataset of some heights and weights at different ages. My professor wants me to plot the residuals from regression of &lt;code&gt;soma.WT9&lt;/code&gt; against the residuals from  regression of &lt;code&gt;HT9.WT9&lt;/code&gt; (don't mind the notation it's just two columns where soma is being regressed on &lt;code&gt;WT9&lt;/code&gt; and &lt;code&gt;HT9&lt;/code&gt; being regressed on &lt;code&gt;WT9&lt;/code&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the purpose of this plot.&lt;/p&gt;&#xA;" OwnerUserId="3008" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-29T00:51:09.930" LastActivityDate="2011-04-29T00:51:09.930" Title="What does plotting residuals from one regression against the residuals from another regression give us?" Tags="&lt;regression&gt;&lt;residuals&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="7323" PostTypeId="2" ParentId="7322" CreationDate="2011-02-17T07:00:28.667" Score="6" ViewCount="" Body="&lt;p&gt;Hello GKED:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Judging by the details and variable names, soma.WT9 and HT9.WT9, you are obtaining the residuals by first regressing, soma on WT9 and HT9 on WT9 (right?). If I understood you correctly, the scatter plot between soma.WT9 and HT9.WT9 will tell you -- if after removing the effects of WT9 (possibly linear effects in your case) from HT9 and soma, is there a relationship between HT9 and soma. This is beneficial in the case when WT9 explains all the source of variation in soma, then the scatter plot between soma.WT9 and HT9.WT9 will not show any particular (recognizable/standard) pattern. It may be also called partial residual plots.&lt;/p&gt;&#xA;" OwnerUserId="1307" LastActivityDate="2011-02-17T07:00:28.667" CommentCount="4" />
  <row Id="7324" PostTypeId="2" ParentId="7297" CreationDate="2011-02-17T09:11:52.597" Score="4" ViewCount="" Body="&lt;p&gt;This seems like $e^{-ax}\sin(bx)$ function -- FT of such are two Dirac deltas, so it is not surprising at all that they appear as a noisy peaks after DFT (this is a variation of ultraviolet crisis). So, well, don't worry -- you can do nothing wise about it, at least smooth the transform (for instance with moving mean) to find peak locations easier (but better do not report smoothed curve).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, if you are interested in the later signal rather than this initial &quot;bang&quot;, it is better just to cut it off -- this will clear those major peaks and show the more subtle details.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2011-02-17T09:11:52.597" />
  <row Id="7325" PostTypeId="2" ParentId="4451" CreationDate="2011-02-17T10:55:37.620" Score="0" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://arxiv.org/abs/1102.2166&quot; rel=&quot;nofollow&quot;&gt;This paper&lt;/a&gt; uses a &lt;a href=&quot;http://people.maths.ox.ac.uk/~porterm/data/facebook100.zip&quot; rel=&quot;nofollow&quot;&gt;facebook dataset&lt;/a&gt; that is available here. Here is the description from the authors:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The data includes the complete set of nodes and links (and some&#xA;  demographic information) from &lt;strong&gt;100&lt;/strong&gt; US colleges and universities from a&#xA;  single-time snapshot in September 2005.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="3291" LastActivityDate="2011-02-17T10:55:37.620" CommunityOwnedDate="2011-02-17T10:55:37.620" />
  <row Id="7326" PostTypeId="1" AcceptedAnswerId="7363" CreationDate="2011-02-17T11:12:27.553" Score="3" ViewCount="75" Body="&lt;p&gt;Hi everyone,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am new here, so I might have missed a similar question. &#xA;I am trying to do things good in my research and using proper statistical approaches, but in computer science, we had quite a weak training on statistical methods applied to our research.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, what I am asking is probably trivial. Here is the problem:&#xA;We have a dataset of del.icio.us (the social bookmarking website) and want to work on a subset of this dataset for a manual study of the usage of tags. In this dataset, we have urls, users and tags that are linked by triplets  defining an association of a tag by a user to a url. We have  401 970 328 such triplets in our dataset, so we can't look at all of them and want to choose a sample of it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Naively, at the beginning, we started with a purely random sampling to choose 500 user-bookmark pairs and all their associated tags. Because there are some more popular tags on del.icio.us, our sample has a long-tail distribution of tag usage, similar to the one from the whole dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I would like to check is that the actual distribution of each tag is similar in the sample as in the original dataset. That is, if &quot;java&quot; appears 10 times more often than &quot;indonesia&quot; in the dataset, it should be a similar distribution in the sample. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How would I go at doing this? one of the issue is that in the sample, some classes (tag) present in the full dataset will never appear.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second question is also: how to decide of a good size for the sample?&lt;/p&gt;&#xA;" OwnerUserId="3291" LastActivityDate="2011-02-18T06:07:14.057" Title="Checking the distribution of classes in a sample of a dataset" Tags="&lt;distributions&gt;&lt;sampling&gt;&lt;dataset&gt;" AnswerCount="2" CommentCount="6" />
  <row Id="7327" PostTypeId="2" ParentId="7292" CreationDate="2011-02-17T11:23:36.643" Score="3" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Introduction&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My interest in the topic is now about 7 years and resulted in PhD thesis &lt;a href=&quot;http://uosis.mif.vu.lt/~celov/DC/dcelov%20thesis%2009_05.pdf&quot; rel=&quot;nofollow&quot;&gt;Time series: aggregation, disaggregation and long memory&lt;/a&gt;, where attention was paid to a specific question of cross-sectional disaggregation problem for AR(1) scheme. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Data&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Working with different approaches to aggregation the first question you need to clarify is what type of data you deal with (my guess is spatial, the most thrilling one). In practice you may consider temporal aggregation (see &lt;a href=&quot;http://ideas.repec.org/p/bdi/wptemi/td_685_08.html&quot; rel=&quot;nofollow&quot;&gt;Silvestrini, A. and Veridas, D. (2008)&lt;/a&gt;), cross-sectional (I loved the article by &lt;a href=&quot;http://ideas.repec.org/p/fip/fedmem/1.html&quot; rel=&quot;nofollow&quot;&gt;Granger, C. W. J. (1990)&lt;/a&gt;) or both time and space (spatial aggregation is nicely surveyed in &lt;a href=&quot;http://www.homepages.ucl.ac.uk/~uctprgi/Files/GiacominiGranger04.pdf&quot; rel=&quot;nofollow&quot;&gt;Giacomini, R. and Granger, C. W. J. (2004)&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Answers (lengthy)&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, answering your questions, I put some rough intuition first. Since the problems I meet in practice are often based on inexact data (Andy's assumption &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;you can measure a time series of observations at any level of precision in time&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;seems too strong for macro-econometrics, but good for financial and micro-econometrics or any experimental fields, were you do control the precision quite well) I do have to bear in mind that my monthly time series are less precise than when I work with yearly data. Besides more frequent time series at least in macroeconomics do have &lt;em&gt;seasonal patterns&lt;/em&gt;, that may lead to &lt;em&gt;spurious&lt;/em&gt; results (seasonal parts do correlate not the series), so you need to seasonally adjust your data - another source of smaller precision for higher frequency data. Working with cross-sectional data revealed that high level of disaggregation brings more problems with probably, lots of zeroes to deal with. For instance, a particular household in the panel of data may purchase a car once per 5-10 years, but aggregated demand for new (used) cars is much smoother (even for a small town or region).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;em&gt;weakest point&lt;/em&gt; aggregation always results in the &lt;em&gt;loss of information&lt;/em&gt;, you may have the GDP produced by the cross-section of EU countries during the whole decade (say period of 2001-2010), but you will loose all the dynamic features that may be present in your analysis considering detailed panel data set. Large scale cross-sectional aggregation may turn to be even more interesting: you, roughly, take simple things (short memory AR(1)) average them over the quite large population and get &quot;representative&quot; long memory agent that resembles none of the micro units (one more stone to the representative agent's concept). So aggregation ~ loss of information ~ different properties of the objects and you would like to take control over the level of this loss and/or new properties. In my opinion, it is better to have precise micro level data at as high frequency as possible, but... there is a usual measurement trade-off, you can't be everywhere perfect and precise :) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Technically producing any regression analysis you do need more room (degrees of freedom) to be more or less confident that (at least) statistically your results are not junk, though they still may be a-theoretical and junk :) So I do put equal weights to question 1 and 2 (usually choose quarterly data for the macro-analysis). Answering the 3rd sub-question, all you decide in practical applications what is more important to you: more precise data or degrees of freedom. If you take the mentioned assumption into account the more detailed (or higher frequency) data is preferable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Probably the answer will be edited latter after some sort of discussion if any.&lt;/p&gt;&#xA;" OwnerUserId="2645" LastActivityDate="2011-02-17T11:23:36.643" CommentCount="1" />
  <row Id="7329" PostTypeId="2" ParentId="7322" CreationDate="2011-02-17T13:03:10.287" Score="4" ViewCount="" Body="&lt;p&gt;This sounds like what I call an &quot;added variable&quot; plot.  The idea behind these is to provide a visual way of whether adding a variable to a model (ht9 in your case) is likely to add anything to the model (soma on wt9 in your case).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It was explained to me like this.  When you fit a linear regression, the order of the variables matters.  It's kind of like imagining the variance in the soma variable as an &quot;island&quot;. The first variables &quot;claims&quot; a portion of the variance on the island, and the second variable &quot;claims&quot; what it can from what is left over.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So basically this plot will show you if &quot;what is left to explain&quot; in soma's variation (residuals from soma.wt9) can be explained by &quot;the capacity of ht9 to explain anything over and above wt9&quot; (residuals from ht9.wt9).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also show mathematically what is going on.  Residuals from soma.wt9 are calculate as:&#xA;$$e_{i}=soma-\beta_{0}-\beta_{1}wt9$$&#xA;residuals from ht9.wt9 are:&#xA;$$f_{i}=ht9-\alpha_{0}-\alpha_{1}wt9$$&#xA;Regression of $e_i$ on $f_i$ through the origin (because $\overline{e}=\overline{f}=0$, so line will pass through origin) gives&#xA;$$e_{i}=\delta f_{i}$$&#xA;Substituting the residual equations into this one gives:&#xA;$$soma-\beta_{0}-\beta_{1}wt9=\delta (ht9-\alpha_{0}-\alpha_{1}wt9)$$&#xA;Re-arranging terms gives:&#xA;$$soma=(\beta_{0}-\delta\alpha_{0})+(\beta_{1}-\delta\alpha_{1})wt9+\delta ht9$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hence, the estimated slope (using OLS regression) will be the same in the model with $soma = \beta_0+\beta_{wt9}wt9 + \beta_{ht9}ht9$ as in the model $resid.soma=\beta_{ht9} resid.ht9$&#xA;This also shows explicitly why having correlated regressor variables ($\alpha_{1}$ is a rescaled correlation) will make the estimated slopes change, and possibly be the &quot;opposite sign&quot; to what is expected.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think this method was actually how Multiple regression was carried out before computers were able to invert large matrices.  It may be quicker to invert lots of $2\times 2$ matrices than it is to invert one huge matrix.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-02-17T13:03:10.287" />
  <row Id="7330" PostTypeId="1" AcceptedAnswerId="7332" CreationDate="2011-02-17T13:55:51.813" Score="3" ViewCount="99" Body="&lt;p&gt;I have a question about a rotation matrix, which can be represented in 2 dimensions as:&#xA;$$R_{2}(\theta)=\begin{bmatrix}  \cos\theta &amp;amp; \sin\theta \\ -\sin\theta &amp;amp; \cos\theta \end{bmatrix}$$&#xA;For some arbitrary angle $\theta$.  This can be extended to an arbitrary number of dimensions by adding an identity matrix:&#xA;$$R_{n}(\theta)=\begin{bmatrix} R_{2}(\theta) &amp;amp; 0 \\ 0 &amp;amp; I_{n-2}\end{bmatrix}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have found some &quot;invariance&quot; properties of a n-dimensional prior distribution when rotated in 2 arbitrary dimensions.  My question is: can any rotation in arbitrary dimensions be represented by a sequence of 2-D rotations?  It doesn't matter if the sequence is unique or not for my purposes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or perhaps a better question is: if a prior distribution is invariant when rotated about 2 arbitrary dimensions, is it invariant when rotated about an arbitrary number of dimensions?&lt;/p&gt;&#xA;" OwnerUserId="2392" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-02-17T19:33:00.870" LastActivityDate="2011-02-17T22:00:09.717" Title="Rotation matrices and prior invariance for arbitrary dimensions" Tags="&lt;prior&gt;&lt;rotation&gt;&lt;invariance&gt;" AnswerCount="2" CommentCount="4" />
  <row Id="7331" PostTypeId="2" ParentId="7319" CreationDate="2011-02-17T14:33:45.927" Score="4" ViewCount="" Body="&lt;p&gt;Since $X_i$ and $Y_i$ are AR(1) processes, we can treat $(X_i,Y_i)$ as a VAR(1) process. Then testing the difference of means means testing the restrictions on VAR coefficients. For this simple case there is a &lt;a href=&quot;http://books.google.com/books?id=B8_1UBmqVUoC&amp;amp;lpg=PP1&amp;amp;dq=hamilton%20time%20series%20analysis&amp;amp;hl=fr&amp;amp;pg=PA300#v=onepage&amp;amp;q&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;formula&lt;/a&gt; derived in Hamilton's book &quot;Time series analysis&quot;. The formula in the link is the one before the start of new section. Unfortunately it is not implemented in R, as far as I know. I checked only &lt;strong&gt;vars&lt;/strong&gt; package, but I can say with some confidence that if it is not implemented there, it is not implemented in other packages.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is some code I've cobbled to calculate this statistic for simulated AR(1) processes:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;n &amp;lt;- 1000&#xA;&#xA;e1 &amp;lt;- rnorm(n,2,sd=1)&#xA;e2 &amp;lt;- rnorm(n,2.5,sd=1)&#xA;&#xA;xm &amp;lt;- arima.sim(n,model=list(ar=0.5),innov=e1)&#xA;ym &amp;lt;- arima.sim(n,model=list(ar=0.6),innov=e2)&#xA;&#xA;eq1&amp;lt;-dynlm(xm~L(xm)+L(ym))&#xA;eq2&amp;lt;-dynlm(ym~L(xm)+L(ym))&#xA;&#xA;res&amp;lt;-rbind(residuals(eq1),residuals(eq2))&#xA;&#xA;omega &amp;lt;- 1/n*tcrossprod(res,res)&#xA;&#xA;xx &amp;lt;- rbind(1,xm[-1000],ym[-1000])&#xA;&#xA;xxinv &amp;lt;- solve(tcrossprod(xx,xx))&#xA;&#xA;c1 &amp;lt;- coef(eq1)[1]&#xA;c2 &amp;lt;- coef(eq2)[1]&#xA;&#xA;chi &amp;lt;- (c1-c2)^2/(xxinv[1,1]*(omega[1,1]-2*omega[1,2]+omega[2,2]))&#xA;&#xA;1-pchisq(chi,1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;After playing with different values of means, it seems that statistic is working more or less. Of course the results should be tested for different values of variance of errors and different AR specifications and different sample sizes. On the other hand from theoretical point of view statistic is quite nice, since it allows unequal variances.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt; Out of curiosity I did some MC simulations. For this I wrote a bit more efficient version of test:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ham.test.lm2 &amp;lt;- function(xm,ym) {&#xA;    n &amp;lt;- length(xm)&#xA;&#xA;    df &amp;lt;- cbind(x=xm[-1],y=ym[-1],lx=xm[-n],ly=ym[-n])&#xA;    eq1&amp;lt;-lsfit(df[,3:4],df[,1])&#xA;    eq2&amp;lt;-lsfit(df[,3:4],df[,2])&#xA;&#xA;    res&amp;lt;-rbind(residuals(eq1),residuals(eq2))&#xA;&#xA;    omega &amp;lt;- 1/n*tcrossprod(res,res)&#xA;&#xA;    xx &amp;lt;- rbind(1,xm[-n],ym[-n])&#xA;&#xA;    ch &amp;lt;- chol(tcrossprod(xx,xx))&#xA;    xxinv &amp;lt;- chol2inv(ch)&#xA;&#xA;    c1 &amp;lt;- coef(eq1)[1]&#xA;    c2 &amp;lt;- coef(eq2)[1]&#xA;&#xA;    chi &amp;lt;- (c1-c2)^2/(xxinv[1,1]*(omega[1,1]-2*omega[1,2]+omega[2,2]))&#xA;&#xA;    res&amp;lt;-1-pchisq(chi,1)&#xA;    names(res) &amp;lt;-NULL&#xA;    res&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This runs about 60 times faster than the original. Then I wrote special function for generating two AR(1) processes given their parameters.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;gen.arima &amp;lt;-function(n,N,mn1,mn2,rho1,rho2,sd1=1,sd2=2,burn.in=50){&#xA;    foreach(i=1:N,.combine=c) %dopar% {&#xA;        e1 &amp;lt;- rnorm(n+burn.in,mn1,sd=sd1)&#xA;        e2 &amp;lt;- rnorm(n+burn.in,mn2,sd=sd2)&#xA;&#xA;        xm &amp;lt;- arima.sim(n+burn.in,model=list(ar=rho1),innov=e1)&#xA;        ym &amp;lt;- arima.sim(n+burn.in,model=list(ar=rho2),innov=e2)&#xA;&#xA;        list(window(cbind(xm,ym),start=burn.in+1))&#xA;    }&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This function generates &lt;code&gt;N&lt;/code&gt; pairs of &lt;code&gt;n&lt;/code&gt; sized sample of two AR(1) processes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then I ran the following simulation.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(&quot;iterators&quot;)&#xA;library(&quot;foreach&quot;)&#xA;library(&quot;multicore&quot;)&#xA;library(&quot;doMC&quot;)&#xA;registerDoMC(16) ##the simulations ran on server with 2 quad-core Xeon processors with HT&#xA;&#xA;rho &amp;lt;- seq(0.1,0.9,by=0.1)&#xA;nr &amp;lt;- length(rho)&#xA;&#xA;&#xA;rr &amp;lt;- numeric()&#xA;for(i in 1:nr) {&#xA;    for(j in (i):nr)&#xA;      rr &amp;lt;- rbind(rr,cbind(rho[i],rho[j]))&#xA;}&#xA;&#xA;mu &amp;lt;- seq(0,2,by=0.1)&#xA;&#xA;rrm &amp;lt;- foreach(m=mu,.combine=rbind) %do% {&#xA;   cbind(rr,m)&#xA;}&#xA;&#xA;nn &amp;lt;- c(10,30,100,500,1000)&#xA;&#xA;rrmn &amp;lt;- foreach(n=nn,.combine=rbind) %do% {&#xA;   cbind(rrm,n)&#xA;}&#xA;&#xA;rrmns &amp;lt;- foreach(s=c(0.1,0.5,1),.combine=rbind) %do% {&#xA;   cbind(rrmn,s)&#xA;}&#xA;&#xA;colnames(rrmns) &amp;lt;- c(&quot;rho1&quot;,&quot;rho2&quot;,&quot;mudiff&quot;,&quot;slen&quot;,&quot;sigma&quot;)&#xA;&#xA;cc &amp;lt;- system.time(res &amp;lt;- foreach(i=1:nrow(rrmns),.combine=rbind) %dopar% {&#xA;    a &amp;lt;- rrmns[i,]&#xA;    aa &amp;lt;- gen.arima(a[4],500,2,2+a[3],a[1],a[2],a[5],a[5])&#xA;    hm&amp;lt;-sapply(aa,function(x)ham.test.lm2(x[,1],x[,2]))&#xA;    tt&amp;lt;-sapply(aa,function(x)t.test(x[,1],x[,2])$p.value)&#xA;    h1&amp;lt;-sum(hm&amp;lt;0.05)/length(hm)&#xA;    t1&amp;lt;-sum(tt&amp;lt;0.05)/length(tt)&#xA;    c(a,h1,t1)&#xA;})&#xA;&#xA;write.csv(file=&quot;hammc.csv&quot;,res)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This simulation calculates the number of times the statistic rejects the null hypothesis that the means are zero for various sample sizes and various combinations of parameters of AR(1) processes. I used the maximum burn in for $\rho=0.9$ when generating AR(1) processes. I also compare the performance of the statistic to simple t-test. The resulting data.frame can be downloaded &lt;a href=&quot;http://mif.vu.lt/~zemlys/download/source/hammc.csv.gz&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a few graphs illustrating the perfomance.&#xA;&lt;img src=&quot;http://i.imgur.com/vweYF.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is for AR(1) processes with $\rho=0.1$, 0.1 and 1 indicate the variance of the errors. &lt;code&gt;mudiff&lt;/code&gt; is the true difference of the means. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the similar graph for $\rho=0.9$:&#xA;&lt;img src=&quot;http://i.imgur.com/4GWFc.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now the same graphs for t.test ($\rho=0.1$):&#xA;&lt;img src=&quot;http://i.imgur.com/rxn25.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And $\rho=0.9$:&#xA;&lt;img src=&quot;http://i.imgur.com/c6DW6.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the code which produced the graphs:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mc &amp;lt;- read.csv(&quot;hammc.csv&quot;)&#xA;colnames(mc)[7] &amp;lt;- &quot;Hamilton&quot;&#xA;colnames(mc)[8] &amp;lt;- &quot;t.test&quot;&#xA;colnames(mc)[5] &amp;lt;- &quot;sample.size&quot;&#xA;qplot(x=sample.size,y=Hamilton,data=mc[mc$sigma!=0.5 &amp;amp; mc$rho1==0.1 &amp;amp; mc$rho2==0.1,],group=mudiff,geom=&quot;line&quot;,color=mudiff)+facet_wrap(~sigma)&#xA;qplot(x=sample.size,y=Hamilton,data=mc[mc$sigma!=0.5 &amp;amp; mc$rho1==0.9 &amp;amp; mc$rho2==0.9,],group=mudiff,geom=&quot;line&quot;,color=mudiff)+facet_wrap(~sigma)&#xA;qplot(x=sample.size,y=t.test,data=mc[mc$sigma!=0.5 &amp;amp; mc$rho1==0.1 &amp;amp; mc$rho2==0.1,],group=mudiff,geom=&quot;line&quot;,color=mudiff)+facet_wrap(~sigma)&#xA;qplot(x=sample.size,y=t.test,data=mc[mc$sigma!=0.5 &amp;amp; mc$rho1==0.9 &amp;amp; mc$rho2==0.9,],group=mudiff,geom=&quot;line&quot;,color=mudiff)+facet_wrap(~sigma)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Conclusions&lt;/strong&gt; Note that the graphs measure the &lt;a href=&quot;http://en.wikipedia.org/wiki/Statistical_power&quot; rel=&quot;nofollow&quot;&gt;power&lt;/a&gt; of the statistic for &lt;code&gt;mudiff&lt;/code&gt; greater than zero and size of the test for zero &lt;code&gt;mudiff&lt;/code&gt;, so ideally the the blue line for &lt;code&gt;mudiff=0&lt;/code&gt; should be straight line at 0.05, and all the other lines should be straight lines at 0.95. Since the statistic is assymptotic this does not happen. What it is clear that t.test is not suitable for higher values of $\rho$, it performs quite nice for $\rho=0.1$, but very bad for $\rho=0.9$. Hamilton's test on the other hand has low power even for larger sample sizes when the true difference in means is not very large. Also the results are the same for error variance 0.1 and 1, so I do not rule out the error in my code.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-02-20T12:25:06.973" LastActivityDate="2011-02-20T12:25:06.973" />
  <row Id="7332" PostTypeId="2" ParentId="7330" CreationDate="2011-02-17T14:41:10.700" Score="5" ViewCount="" Body="&lt;p&gt;The answer, I believe, to your first question is &quot;yes&quot;. This can be accomplished with &lt;a href=&quot;http://en.wikipedia.org/wiki/Givens_rotation&quot; rel=&quot;nofollow&quot;&gt;Givens rotations&lt;/a&gt;, which allow for the annihilation of arbitrary elements of a matrix via a $2\times 2$ rotation matrix. The implication is that if you start with a rotation matrix, then you can reduce it to a diagonal matrix via Givens rotations. But, since orthogonality of a matrix is preserved by multiplication with another orthogonal matrix, this means that the diagonal matrix must be orthogonal, and hence, must contain only 1's and -1's. Additional rotations then reduce this matrix to the identity. The affirmative answer to your first question follows immediately. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus the space of $n\times n$ orthogonal matrices is spanned by Givens rotations with respect to matrix multiplication. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If this doesn't give you enough detail, let me know and I'll fill it in. &lt;/p&gt;&#xA;" OwnerUserId="2970" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-02-17T19:44:10.390" LastActivityDate="2011-02-17T19:44:10.390" CommentCount="5" />
  <row Id="7333" PostTypeId="2" ParentId="7084" CreationDate="2011-02-17T15:47:56.670" Score="3" ViewCount="" Body="&lt;p&gt;The best argument I have heard about using the normal distribution is one of &quot;sufficient statistics&quot; given by Larry Bretthorst in his PhD thesis on spectral estimation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically because the normal distribution has a set of sufficient statistics (mean and covariance matrix), then this is the only thing that matters when fitting your data.  So if the &quot;true&quot; distribution is not normal, but it gives you the same set of sufficient statistics, then you will get exactly the same answer as if it was normal, whenever the parameters are linear in these sufficient statistics (if non-linear, then its a projection onto the error space).  This is in line with &quot;BLUE&quot; type estimators.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So in this sense, testing for normality is &quot;stupid&quot;.  But checking the residuals is certainly a good idea.  This is because it can help improve your model, by looking for systematic patterns in them.  The normality tests are usually designed to pick up these systematic patterns, so in this sense they are good.  The problem with them, is that they don't &quot;give you a clue&quot; about how to fix the &quot;non-normality&quot;.  Looking at the residuals does, though.  e.g. see curvature in residuals: put a quadratic term in, see &quot;harmonic&quot; behaviour: put a sine or cosine function in, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another justification comes from the MaxEnt world.  The normal distribution is the one with the largest entropy (i.e. uncertainty) among all continuous distributions on the reals with given mean and variance.  So this means that the normal &quot;assumes the least&quot;, in a sense for a given mean and variance.  This means that &quot;Nature&quot; has to work incredibly hard to move to a part of the &quot;distribution space&quot; which is non-normal and keep the mean and variance fixed.  The normal can be realised in a massively larger number of ways compared to any other with fixed mean and fixed variance.  So a departure usually means there is some &quot;unspecified&quot; constraint operating on the data - which could be thought as a &quot;covariate missed&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is because the normal distribution is non-robust that it allows you to &quot;learn from your mistakes&quot;.  If you are wrong, the normal distribution will pick this up by producing large residuals.  When you use &quot;robust&quot; or &quot;non-parametric&quot; methods, it becomes more difficult to see where a model can be improved.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And as a final note: it matters not one iota if the normal distribution is an awful fit, &lt;em&gt;unless you have something to replace it with!&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-02-17T15:47:56.670" />
  <row Id="7334" PostTypeId="2" ParentId="4978" CreationDate="2011-02-17T16:03:19.943" Score="1" ViewCount="" Body="&lt;p&gt;It is true that in the past, MaxEnt and Bayes have dealt with different types or forms of information. I would say that Bayes uses &quot;hard&quot; constraints as well though, the likelihood.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In any case, it is not an issue anymore as Bayes Rule (not the product rule) can be obtained from Maximum relative Entropy (MrE), and not in an ambiguous way:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://arxiv.org/PS_cache/arxiv/pdf/0708/0708.1593v2.pdf&quot; rel=&quot;nofollow&quot;&gt;Updating Probabilities with Data and Moments&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://arxiv.org/PS_cache/arxiv/pdf/0901/0901.0401v1.pdf&quot; rel=&quot;nofollow&quot;&gt;From Physics to Economics: An Econometric Example Using Maximum Relative Entropy&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;It's a new world...&lt;/p&gt;&#xA;" OwnerUserId="3297" LastActivityDate="2011-02-17T16:03:19.943" />
  <row Id="7335" PostTypeId="2" ParentId="7270" CreationDate="2011-02-17T16:06:24.620" Score="1" ViewCount="" Body="&lt;p&gt;From past experience with a network of 7 million nodes, I think visualizing your complete network will give you an uninterpretable image. I might suggest different visualizations using subsets of your data such as just using the top 10 nodes with the most inbound or outbound links. I second celenius's suggestion on using gephi.&lt;/p&gt;&#xA;" OwnerUserId="3298" LastActivityDate="2011-02-17T16:06:24.620" CommentCount="1" />
  <row Id="7336" PostTypeId="1" CreationDate="2011-02-17T16:18:29.617" Score="8" ViewCount="70" Body="&lt;p&gt;I'm running a simulation on R and a cluster of computers and have the following problem. On each of X computers I run:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;fxT2 &amp;lt;- function(i) runif(10)&#xA;nessay &amp;lt;- 100&#xA;c(mclapply(1:nessay, fxT2), recursive=TRUE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;There are 32 computers, each with 16 cores. However, around 2% of the random numbers are identical. What strategies would you adopt to avoid this? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've been able to avoid this problem for fxT2 by setting a latency (i.e. delaying by a sec the time at which each job is send to each of the X computers). But it's seems very ad-hoc to fxt2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is that in reality fxT2 is a long task involving pseudo random numbers. At the end of the process, I expect to get X*nessay reproduction of the same statistical experiment, &lt;strong&gt;not&lt;/strong&gt; nessay reproductions. How to make sure that this is indeed the case and is there a way to check this?. &lt;/p&gt;&#xA;" OwnerUserId="603" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2011-02-17T17:52:48.100" LastActivityDate="2011-02-17T18:27:08.447" Title="RNG, R, mclapply and cluster of computers" Tags="&lt;r&gt;&lt;random-generation&gt;&lt;parallel-computing&gt;&lt;multicore&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="7337" PostTypeId="2" ParentId="7336" CreationDate="2011-02-17T16:39:34.187" Score="2" ViewCount="" Body="&lt;p&gt;You need to use a RNG specifically designed for parallel computing.  See the &quot;Parallel computing: Random numbers&quot; section of the &lt;a href=&quot;http://cran.r-project.org/web/views/HighPerformanceComputing.html&quot; rel=&quot;nofollow&quot;&gt;High Performance Computing Task View&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1657" LastActivityDate="2011-02-17T16:39:34.187" CommentCount="1" />
  <row Id="7338" PostTypeId="2" ParentId="7208" CreationDate="2011-02-17T17:14:48.070" Score="2" ViewCount="" Body="&lt;p&gt;The &quot;chance correction&quot; in Cohen's $\kappa$ estimates probabilities with which each rater chooses the existing categories. The estimation comes from the marginal frequencies of the categories. When you only have 1 judgement for each rater, this means that $\kappa$ assumes the category chosen for this single judgement in general has a probability of 1. This obviously makes no sense since the number of judgements (1) is too small to reliably estimate the base rates of all categories.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An alternative might be a simple binomial model: without additional information, we might assume that the probability of agreement between two raters for one judgement is 0.5 since judgements are binary. This means that we implicitly assume that both raters pick each category with probability 0.5 for all criteria. The number of agreements expected by chance over all criteria then follows a binomial distribution with $p=0.5$.&lt;/p&gt;&#xA;" OwnerUserId="1909" LastActivityDate="2011-02-17T17:14:48.070" CommentCount="1" />
  <row Id="7340" PostTypeId="2" ParentId="7326" CreationDate="2011-02-17T18:25:00.827" Score="0" ViewCount="" Body="&lt;p&gt;What you might want to try is to take many bootstrap samples of 500 pairs and then build a confidence interval of this distribution to see if the population mean is included. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However you really first need to go through your tags and spell correct, replace synonyms etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;www.kamalnigam.com/papers/bootstrap-ijcaiws99.pdf&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-Ralph Winters&lt;/p&gt;&#xA;" OwnerUserId="3489" LastActivityDate="2011-02-17T18:25:00.827" CommentCount="2" />
  <row Id="7341" PostTypeId="2" ParentId="7336" CreationDate="2011-02-17T18:27:08.447" Score="4" ViewCount="" Body="&lt;p&gt;The &lt;a href=&quot;http://cran.r-project.org/package=snow&quot; rel=&quot;nofollow&quot;&gt;snow&lt;/a&gt; has explicit support to initialise the given number of RNG streams in a cluster computation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It can employ one of two RNG implementations:  &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://cran.r-project.org/package=rsprng&quot; rel=&quot;nofollow&quot;&gt;rsprng&lt;/a&gt; and &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://cran.r-project.org/package=rlecuyer&quot; rel=&quot;nofollow&quot;&gt;rlecuyer&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Otherwise you have to do the coordination by hand.  &lt;/p&gt;&#xA;" OwnerUserId="334" LastActivityDate="2011-02-17T18:27:08.447" />
  <row Id="7342" PostTypeId="2" ParentId="7268" CreationDate="2011-02-17T20:19:02.183" Score="2" ViewCount="" Body="&lt;p&gt;You might checkout the &lt;code&gt;aggregate.zoo&lt;/code&gt; function from the package &lt;code&gt;zoo&lt;/code&gt;:&#xA;&lt;a href=&quot;http://cran.r-project.org/web/packages/zoo/zoo.pdf&quot; rel=&quot;nofollow&quot;&gt;http://cran.r-project.org/web/packages/zoo/zoo.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Charlie&lt;/p&gt;&#xA;" OwnerUserId="401" LastActivityDate="2011-02-17T20:19:02.183" CommentCount="2" />
  <row Id="7343" PostTypeId="1" AcceptedAnswerId="7345" CreationDate="2011-02-17T20:44:05.473" Score="4" ViewCount="73" Body="&lt;p&gt;Suppose I have a function &lt;code&gt;f&lt;/code&gt;, and I want to sample it at 100 points in the interval &lt;code&gt;[0, 100]&lt;/code&gt;. For some reason (that seemed smart to me at the time), I decided to not sample at equidistant intervals, but rather use the following function to determine the sample points:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;log2(x)*(100/log2(100))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This gives me a sequence of sample points that becomes denser as it approaches 100. The problem is, that now I need to calculate the mean over the values I have sampled, but due to the bad sampling that would be heavily biased. I cannot resample the data, this would take way too long (several days), and I am on a very tight schedule. So, the solution that comes to mind is to calculate a weighted average to correct the error. My question is, how do I determine the weights?&lt;/p&gt;&#xA;" OwnerUserId="977" LastActivityDate="2011-02-17T21:41:38.403" Title="How to correct uneven sampling distribution when calculating the mean?" Tags="&lt;sampling&gt;&lt;error&gt;&lt;averages&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="7344" PostTypeId="1" AcceptedAnswerId="7369" CreationDate="2011-02-17T20:44:25.817" Score="2" ViewCount="319" Body="&lt;p&gt;I'm trying to write a function to graphically display predicted vs. actual relationships in a linear regression.  What I have so far works well for linear models, but I'd like to extend it in a few ways.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Handle glm models&lt;/li&gt;&#xA;&lt;li&gt;Deal with NAs in the predicted values&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Does what I have so far seem like a good solution, or is there an existing package somewhere that's already implemented this?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;DF &amp;lt;- as.data.frame(na.exclude(airquality))&#xA;DF$Month &amp;lt;- as.factor(DF$Month)&#xA;DF$Day &amp;lt;- as.factor(DF$Day)&#xA;&#xA;my_model &amp;lt;- lm(Ozone~Solar.R+Wind+Temp+Month+Day,DF)&#xA;&#xA;PvA&amp;lt;- function(model,varlist=NULL,smooth=.5) { #Plot predicted vs actual for a model&#xA;&#xA;    indvars &amp;lt;- attr(terms(model),&quot;term.labels&quot;)&#xA;&#xA;    if (is.null(varlist)) {&#xA;        varlist &amp;lt;- indvars&#xA;    }&#xA;&#xA;    Y &amp;lt;- as.character(as.list(attr(terms(model),&quot;variables&quot;))[2])&#xA;    P.Y &amp;lt;- paste('P',Y,sep='.')&#xA;&#xA;    DF &amp;lt;- as.data.frame(get(as.character(model$call$data)))&#xA;    DF[,P.Y] &amp;lt;- predict.lm(model)&#xA;&#xA;    par(ask=TRUE)&#xA;    for (X in varlist) {&#xA;        print(X)&#xA;        A &amp;lt;- na.omit(DF[,c(X,Y)])&#xA;        P &amp;lt;- na.omit(DF[,c(X,P.Y)])&#xA;        plot(A)&#xA;        points(P,col=2)&#xA;        lines(lowess(A,f=smooth),col=1)&#xA;        lines(lowess(P,f=smooth),col=2)&#xA;    }&#xA;&#xA;}&#xA;PvA(my_model)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2817" LastEditorUserId="2817" LastEditorDisplayName="" LastEditDate="2011-02-17T23:03:57.857" LastActivityDate="2011-02-18T10:04:53.950" Title="How to graphically compare predicted and actual values from multivariate regression in R?" Tags="&lt;r&gt;&lt;regression&gt;&lt;multivariate-analysis&gt;&lt;multiple-regression&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="1" />
  <row Id="7345" PostTypeId="2" ParentId="7343" CreationDate="2011-02-17T21:41:38.403" Score="7" ViewCount="" Body="&lt;p&gt;The answer depends on the characteristics of $f$.  Regardless, its average (by definition) is $\frac{1}{100}\int_0^{100}{f(x)dx}$, so your problem is one of estimating that integral from values at a discrete set of points.  For a highly discontinuous function you can use any &lt;a href=&quot;http://en.wikipedia.org/wiki/Riemann_sum&quot; rel=&quot;nofollow&quot;&gt;Riemann sum&lt;/a&gt;; for a differentiable function use the &lt;a href=&quot;http://en.wikipedia.org/wiki/Trapezoidal_rule&quot; rel=&quot;nofollow&quot;&gt;Trapezoidal Rule&lt;/a&gt;, for which you can estimate the error in terms of derivatives of $f$; for a thrice differentiable function use &lt;a href=&quot;http://en.wikipedia.org/wiki/Simpson%27s_rule&quot; rel=&quot;nofollow&quot;&gt;Simpson's Rule&lt;/a&gt;, etc.  If necessary, you can combine the error estimates of these rules with statistical estimates of the error in computing each value of $f$ to obtain an estimate of the error in the average.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-02-17T21:41:38.403" CommentCount="1" />
  <row Id="7346" PostTypeId="6" CreationDate="2011-02-17T21:56:24.370" Score="0" ViewCount="" Body="&lt;p&gt;I did not participate to the beta, but I have been very happy to contribute (Q&amp;amp;A, edits, votes, etc.) during the last six months. Despite being also very happy with our actual moderators, I would like to propose myself for fulfilling this task if one of them feel the need to get some time off.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Needless to say, I will vote for them if they nominate themselves for a second round.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-02-17T21:56:24.370" LastActivityDate="2011-02-17T21:56:24.370" CommunityOwnedDate="2011-02-17T21:56:24.370" CommentCount="4" />
  <row Id="7347" PostTypeId="2" ParentId="7330" CreationDate="2011-02-17T22:00:09.717" Score="0" ViewCount="" Body="&lt;p&gt;Yes, arbitrary rotations in an $n$ dimensional space can be written as the compositions of Givens rotations. The other question (and I think what you're aiming at) is&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Or perhaps a better question is: if a&#xA;  prior distribution is invariant when&#xA;  rotated about 2 arbitrary dimensions,&#xA;  is it invariant when rotated about an&#xA;  arbitrary number of dimensions?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The answer is yes, and moreover look no further! since this class of distribution has been completely characterized: it's a subset of the &lt;a href=&quot;http://en.wikipedia.org/wiki/Elliptical_distribution&quot; rel=&quot;nofollow&quot;&gt;elliptical distributions&lt;/a&gt;, when $\Sigma=I$. Elliptical distributions are all and only the rotationally invariant distirbutions, after an affine transformation of its variables. The standard normal multivariate distribution is its best-known example.&lt;/p&gt;&#xA;" OwnerUserId="30" LastActivityDate="2011-02-17T22:00:09.717" />
  <row Id="7348" PostTypeId="1" AcceptedAnswerId="7349" CreationDate="2011-02-17T22:29:30.310" Score="6" ViewCount="256" Body="&lt;p&gt;I find R can take a long time to generate plots when millions of points are present - unsurprising given that points are plotted individually.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many of the points overlap and form a black mass and a lot of time is spent plotting more points into that mass.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any plotting functions that trade off processing time to work out in advance what plotting is redundant?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not looking for a density plot (although those are often useful), I would want the same output as a simple plot call but much faster than millions of overplots if possible.&lt;/p&gt;&#xA;" OwnerUserId="2726" LastActivityDate="2011-02-18T02:02:39.183" Title="More efficient plot functions in R when millions of points are present?" Tags="&lt;r&gt;&lt;data-visualization&gt;" AnswerCount="3" CommentCount="2" FavoriteCount="1" />
  <row Id="7349" PostTypeId="2" ParentId="7348" CreationDate="2011-02-17T23:32:55.593" Score="5" ViewCount="" Body="&lt;p&gt;This is a hard task with no ready solutions (this is of course because density plot is so a tempting fallback than no one really cares). So, what can you do?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If they really overlap (i.e. have exactly the same X &amp;amp; Y coordinates) and you are not using alpha, the best idea would be just to reduce the overlap using &lt;code&gt;unique&lt;/code&gt; (with alpha, it may be summed over such groups).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If not, you may manually round the coordinates to the nearest pixels and use the previous method (yet this is a dirty solution).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, you can make a density plot only to use it to subsample the points in the most dense areas. This on the other hand will not make the exactly same plot and may introduce artifacts if not precisely tuned.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2011-02-17T23:32:55.593" CommentCount="1" />
  <row Id="7350" PostTypeId="2" ParentId="7" CreationDate="2011-02-18T00:06:24.633" Score="0" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://www.ckan.net&quot; rel=&quot;nofollow&quot;&gt;http://www.ckan.net&lt;/a&gt; has a number of datasets too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.biotorrents.net/browse.php&quot; rel=&quot;nofollow&quot;&gt;http://www.biotorrents.net/browse.php&lt;/a&gt; is also starting to have quite a large amount of BIG datasets.&lt;/p&gt;&#xA;" OwnerUserId="3291" LastActivityDate="2011-02-18T00:06:24.633" CommunityOwnedDate="2011-02-18T00:06:24.633" />
  <row Id="7351" PostTypeId="1" AcceptedAnswerId="7352" CreationDate="2011-02-18T00:35:17.267" Score="10" ViewCount="529" Body="&lt;p&gt;I am trying to get upto speed in Bayesian Statistics. I have a little bit of stats background (STAT 101) but not too much - I think I can understand prior, posterior, and likelihood :D. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't want to read a Bayesian textbook just yet.&#xA;I'd prefer to read from a source (website preferred) that will ramp me up quickly. Something like &lt;a href=&quot;http://www.stat.washington.edu/raftery/Research/PDF/bayescourse.pdf&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt;, but that has more details.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any advice?&lt;/p&gt;&#xA;" OwnerUserId="3301" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-02-21T05:58:12.543" LastActivityDate="2011-02-21T05:58:12.543" CommunityOwnedDate="2011-02-18T15:05:48.033" Title="Bayesian statistics tutorial" Tags="&lt;bayesian&gt;" AnswerCount="4" CommentCount="1" FavoriteCount="9" />
  <row Id="7352" PostTypeId="2" ParentId="7351" CreationDate="2011-02-18T01:04:21.207" Score="8" ViewCount="" Body="&lt;p&gt;Here's a place to start:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;ftp://selab.janelia.org/pub/publications/Eddy-ATG3/Eddy-ATG3-reprint.pdf&quot; rel=&quot;nofollow&quot;&gt;ftp://selab.janelia.org/pub/publications/Eddy-ATG3/Eddy-ATG3-reprint.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://blog.oscarbonilla.com/2009/05/visualizing-bayes-theorem/&quot; rel=&quot;nofollow&quot;&gt;http://blog.oscarbonilla.com/2009/05/visualizing-bayes-theorem/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://yudkowsky.net/rational/bayes&quot; rel=&quot;nofollow&quot;&gt;http://yudkowsky.net/rational/bayes&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.math.umass.edu/~lavine/whatisbayes.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.math.umass.edu/~lavine/whatisbayes.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Bayesian_inference&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Bayesian_inference&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Bayesian_probability&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Bayesian_probability&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2775" LastActivityDate="2011-02-18T01:04:21.207" CommentCount="1" />
  <row Id="7353" PostTypeId="2" ParentId="7348" CreationDate="2011-02-18T01:13:27.283" Score="6" ViewCount="" Body="&lt;p&gt;I must admit that I do not fully understand your last paragraph:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&quot;I am not looking for a density plot&#xA;  (although those are often useful), I&#xA;  would want the same output as a simple&#xA;  plot call but much faster than&#xA;  millions of overplots if possible.&quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It is also unclear what type of plot (function) you are looking for. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given that you have metric variables, you might find hexagon binned plots or sunnflower plots usefull. For further references, see&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://rosuda.org/gold/index.html&quot; rel=&quot;nofollow&quot;&gt;Graphics of Large Datasets&lt;/a&gt; by Unwin/Theus/Hofmann   &lt;/li&gt;&#xA;&lt;li&gt;Quick-R on &quot;&lt;a href=&quot;http://www.statmethods.net/graphs/scatterplot.html&quot; rel=&quot;nofollow&quot;&gt;High Density Scatterplots&lt;/a&gt;&quot;&lt;/li&gt;&#xA;&lt;li&gt;ggplot2's &lt;a href=&quot;http://had.co.nz/ggplot2/stat_binhex.html&quot; rel=&quot;nofollow&quot;&gt;stat_hexbin&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="307" LastActivityDate="2011-02-18T01:13:27.283" />
  <row Id="7354" PostTypeId="2" ParentId="7208" CreationDate="2011-02-18T01:26:44.263" Score="1" ViewCount="" Body="&lt;p&gt;I find caracal's answer convincing, but I also believe Cohen's Kappa can only account for part of what constitutes interrater reliability.  The simple % of ratings in agreement accounts for another part, and the correlation between ratings, a third.  It takes all three methods to gain a complete picture.  For details please see &lt;a href=&quot;http://pareonline.net/getvn.asp?v=9&amp;amp;n=4&quot; rel=&quot;nofollow&quot;&gt;http://pareonline.net/getvn.asp?v=9&amp;amp;n=4&lt;/a&gt; :&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&quot;[...] the general practice of&#xA;  describing interrater reliability as a&#xA;  single, unified concept is at best&#xA;  imprecise, and at worst potentially&#xA;  misleading.&quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="2669" LastActivityDate="2011-02-18T01:26:44.263" />
  <row Id="7355" PostTypeId="2" ParentId="7351" CreationDate="2011-02-18T01:33:51.137" Score="2" ViewCount="" Body="&lt;p&gt;Some more depth:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://math.tut.fi/~piche/bayes/notes01.pdf&quot; rel=&quot;nofollow&quot;&gt;http://math.tut.fi/~piche/bayes/notes01.pdf&lt;/a&gt; covers Bayes' theorem&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://ccrma.stanford.edu/~jos/bayes/bayes.pdf&quot; rel=&quot;nofollow&quot;&gt;https://ccrma.stanford.edu/~jos/bayes/bayes.pdf&lt;/a&gt; and &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www-personal.une.edu.au/~jvanderw/Introduction_to_Bayesian_Statistics1.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www-personal.une.edu.au/~jvanderw/Introduction_to_Bayesian_Statistics1.pdf&lt;/a&gt; are more about statistical applications&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-02-18T01:33:51.137" />
  <row Id="7356" PostTypeId="2" ParentId="7348" CreationDate="2011-02-18T02:02:39.183" Score="13" ViewCount="" Body="&lt;p&gt;Look at the &lt;a href=&quot;http://cran.r-project.org/package=hexbin&quot; rel=&quot;nofollow&quot;&gt;hexbin&lt;/a&gt; package which implements paper/method by Dan Carr. The &lt;a href=&quot;http://cran.r-project.org/web/packages/hexbin/vignettes/hexagon_binning.pdf&quot; rel=&quot;nofollow&quot;&gt;pdf vignette&lt;/a&gt; has more details which I quote below:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;h2&gt;1 Overview&lt;/h2&gt;&#xA;  &#xA;  &lt;p&gt;Hexagon binning is a form of bivariate&#xA;  histogram useful for visualizing the&#xA;  struc- ture in datasets with large n.&#xA;  The underlying concept of hexagon&#xA;  binning is extremely simple;&lt;/p&gt;&#xA;  &#xA;  &lt;ol&gt;&#xA;  &lt;li&gt;the xy plane over the set (range(x), range(y)) is tessellated by&#xA;  a regular grid of hexagons.&lt;/li&gt;&#xA;  &lt;li&gt;the number of points falling in each hexagon are counted and stored in&#xA;  a data structure&lt;/li&gt;&#xA;  &lt;li&gt;the hexagons with count &gt; 0 are plotted using a color ramp or varying&#xA;  the radius of the hexagon in&#xA;  proportion to the counts. The&#xA;  underlying algorithm is extremely fast&#xA;  and eective for displaying the&#xA;  structure of datasets with $n \ge 10^6$&lt;/li&gt;&#xA;  &lt;/ol&gt;&#xA;  &#xA;  &lt;p&gt;If the size of the grid and the cuts&#xA;  in the color ramp are chosen in a&#xA;  clever fashion than the structure&#xA;  inherent in the data should emerge in&#xA;  the binned plots. The same caveats&#xA;  apply to hexagon binning as apply to&#xA;  histograms and care should be&#xA;  exercised in choosing the binning&#xA;  parameters&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="334" LastActivityDate="2011-02-18T02:02:39.183" CommentCount="2" />
  <row Id="7357" PostTypeId="1" AcceptedAnswerId="7359" CreationDate="2011-02-18T02:32:48.823" Score="9" ViewCount="281" Body="&lt;p&gt;I know this is a fairly specific R question, but I may be thinking about proportion variance explained, R², incorrectly. Here goes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm trying to use the R package randomForest. I have some training data and testing data. When I fit a random forest model, the randomForest function allows you to input new testing data to test. It then tells you the % variance explained in this new data. When I look at this, I get one number.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I use the predict() function to predict the outcome value of the testing data based on the model fit from the training data, and I take the squared correlation coefficient between these values and the &lt;em&gt;actual&lt;/em&gt; outcome values for the testing data, I get a different number. &lt;em&gt;These values don't match up&lt;/em&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's some R code to demonstrate the problem.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# use the built in iris data&#xA;data(iris)&#xA;&#xA;#load the randomForest library&#xA;library(randomForest)&#xA;&#xA;# split the data into training and testing sets&#xA;index &amp;lt;- 1:nrow(iris)&#xA;trainindex &amp;lt;- sample(index, trunc(length(index)/2))&#xA;trainset &amp;lt;- iris[trainindex, ]&#xA;testset &amp;lt;- iris[-trainindex, ]&#xA;&#xA;# fit a model to the training set (column 1, Sepal.Length, will be the outcome)&#xA;set.seed(42)&#xA;model &amp;lt;- randomForest(x=trainset[ ,-1],y=trainset[ ,1])&#xA;&#xA;# predict values for the testing set (the first column is the outcome, leave it out)&#xA;predicted &amp;lt;- predict(model, testset[ ,-1])&#xA;&#xA;# what's the squared correlation coefficient between predicted and actual values?&#xA;cor(predicted, testset[, 1])^2&#xA;&#xA;# now, refit the model using built-in x.test and y.test&#xA;set.seed(42)&#xA;randomForest(x=trainset[ ,-1], y=trainset[ ,1], xtest=testset[ ,-1], ytest=testset[ ,1])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Thanks for any help you might be willing to lend.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Stephen &lt;/p&gt;&#xA;" OwnerUserId="36" LastEditorUserId="36" LastEditorDisplayName="" LastEditDate="2011-02-18T20:14:48.287" LastActivityDate="2011-02-18T20:14:48.287" Title="Manually calculated R² doesn't match up with randomForest() R² for testing new data" Tags="&lt;r&gt;&lt;correlation&gt;&lt;predictive-models&gt;&lt;random-forest&gt;&lt;r-squared&gt;" AnswerCount="1" FavoriteCount="4" />
  <row Id="7358" PostTypeId="1" AcceptedAnswerId="7377" CreationDate="2011-02-18T02:40:12.390" Score="8" ViewCount="357" Body="&lt;p&gt;I've got a particular MCMC algorithm which I would like to port to C/C++. Much of the expensive computation is in C already via Cython, but I want to have the whole sampler written in a compiled language so that I can just write wrappers for Python/R/Matlab/whatever.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After poking around I'm leaning towards C++. A couple of relevant libraries I know of are Armadillo (http://arma.sourceforge.net/) and Scythe (http://scythe.wustl.edu/). Both try to emulate some aspects of R/Matlab to ease the learning curve, which I like a lot. Scythe squares a little better with what I want to do I think. In particular, its RNG includes a lot of distributions where Armadillo only has uniform/normal, which is inconvenient. Armadillo seems to be under pretty active development while Scythe saw its last release in 2007.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So what I'm wondering is if anyone has experience with these libraries -- or others I have almost surely missed -- and if so, whether there is anything to recommend one over the others for a statistician very familiar with Python/R/Matlab but less so with compiled languages (not completely ignorant, but not exactly proficient...).&lt;/p&gt;&#xA;" OwnerUserId="26" LastActivityDate="2011-02-19T02:40:37.700" Title="C++ libraries for statistical computing" Tags="&lt;software&gt;&lt;computing&gt;&lt;library&gt;&lt;c++&gt;" AnswerCount="4" CommentCount="1" FavoriteCount="4" />
  <row Id="7359" PostTypeId="2" ParentId="7357" CreationDate="2011-02-18T03:31:08.217" Score="10" ViewCount="" Body="&lt;p&gt;The reason that the $R^2$ values are not matching is because &lt;code&gt;randomForest&lt;/code&gt; is reporting &lt;em&gt;variation&lt;/em&gt; explained as opposed to &lt;em&gt;variance&lt;/em&gt; explained. I think this is a common misunderstanding about $R^2$ that is perpetuated in textbooks. I even mentioned this on another thread the other day. If you want an example, see the (otherwise quite good) textbook Seber and Lee, &lt;em&gt;Linear Regression Analysis&lt;/em&gt;, 2nd. ed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A general definition for $R^2$ is&#xA;$$&#xD;&#xA;R^2 = 1 - \frac{\sum_i (y_i - \hat{y}_i)^2}{\sum_i (y_i - \bar{y})^2} .&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That is, we compute the mean-squared error, divide it by the variance of the original observations and then subtract this from one. (Note that if your predictions are really bad, this value can go negative.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, what happens with linear regression (&lt;em&gt;with an intercept term!&lt;/em&gt;) is that the &lt;em&gt;average&lt;/em&gt; value of the $\hat{y}_i$'s matches $\bar{y}$. Furthermore, the residual vector $y - \hat{y}$ is orthogonal to the vector of fitted values $\hat{y}$. When you put these two things together, then the definition reduces to the one that is more commonly encountered, i.e.,&#xA;$$&#xD;&#xA;R^2_{\mathrm{LR}} = \mathrm{Corr}(y,\hat{y})^2 .&#xD;&#xA;$$&#xA;(I've used the subscripts $\mathrm{LR}$ in $R^2_{\mathrm{LR}}$ to indicate &lt;em&gt;linear regression&lt;/em&gt;.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;code&gt;randomForest&lt;/code&gt; call is using the first definition, so if you do&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   &amp;gt; y &amp;lt;- testset[,1]&#xA;   &amp;gt; 1 - sum((y-predicted)^2)/sum((y-mean(y))^2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;you'll see that the answers match.&lt;/p&gt;&#xA;" OwnerUserId="2970" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-02-18T04:21:19.807" LastActivityDate="2011-02-18T04:21:19.807" CommentCount="5" />
  <row Id="7360" PostTypeId="2" ParentId="7358" CreationDate="2011-02-18T04:08:36.080" Score="0" ViewCount="" Body="&lt;p&gt;There are numerous C/C++ libraries out there, most focusing on a particular problem domain of (e.g. PDE solvers). There are two comprehensive libraries I can think of that you may find especially useful because they are written in C but have excellent Python wrappers already written.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) &lt;a href=&quot;http://www.roguewave.com/products/imsl-numerical-libraries/c-library.aspx&quot; rel=&quot;nofollow&quot;&gt;IMSL C&lt;/a&gt; and &lt;a href=&quot;http://www.roguewave.com/products/imsl-numerical-libraries/pyimsl-studio.aspx&quot; rel=&quot;nofollow&quot;&gt;PyIMSL&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) &lt;a href=&quot;http://trilinos.sandia.gov/&quot; rel=&quot;nofollow&quot;&gt;trilinos&lt;/a&gt; and &lt;a href=&quot;http://trilinos.sandia.gov/packages/pytrilinos/index.html&quot; rel=&quot;nofollow&quot;&gt;pytrilinos&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have never used trilinos as the functionality is primarily on numerical analysis methods, but I use PyIMSL a lot for statistical work (and in a previous work life I developed the software too).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With respect to RNGs, here are the ones in C and Python in IMSL&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;DISCRETE&lt;/h2&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;random_binomial: Generates pseudorandom binomial numbers from a binomial distribution.&lt;/li&gt;&#xA;&lt;li&gt;random_geometric: Generates pseudorandom numbers from a geometric distribution.&lt;/li&gt;&#xA;&lt;li&gt;random_hypergeometric: Generates pseudorandom numbers from a hypergeometric distribution.&lt;/li&gt;&#xA;&lt;li&gt;random_logarithmic: Generates pseudorandom numbers from a logarithmic distribution.&lt;/li&gt;&#xA;&lt;li&gt;random_neg_binomial: Generates pseudorandom numbers from a negative binomial distribution.&lt;/li&gt;&#xA;&lt;li&gt;random_poisson: Generates pseudorandom numbers from a Poisson distribution.&lt;/li&gt;&#xA;&lt;li&gt;random_uniform_discrete: Generates pseudorandom numbers from a discrete uniform distribution.&lt;/li&gt;&#xA;&lt;li&gt;random_general_discrete: Generates pseudorandom numbers from a general discrete distribution using an alias method or optionally a table lookup method.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h2&gt;UNIVARIATE CONTINUOUS DISTRIBUTIONS&lt;/h2&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;random_beta: Generates pseudorandom numbers from a beta distribution.&lt;/li&gt;&#xA;&lt;li&gt;random_cauchy: Generates pseudorandom numbers from a Cauchy distribution.&lt;/li&gt;&#xA;&lt;li&gt;random_chi_squared: Generates pseudorandom numbers from a chi-squared distribution.&lt;/li&gt;&#xA;&lt;li&gt;random_exponential: Generates pseudorandom numbers from a standard exponential distribution.&lt;/li&gt;&#xA;&lt;li&gt;random_exponential_mix: Generates pseudorandom mixed numbers from a standard exponential distribution.&lt;/li&gt;&#xA;&lt;li&gt;random_gamma: Generates pseudorandom numbers from a standard gamma distribution.&lt;/li&gt;&#xA;&lt;li&gt;random_lognormal: Generates pseudorandom numbers from a lognormal distribution.&lt;/li&gt;&#xA;&lt;li&gt;random_normal: Generates pseudorandom numbers from a standard normal distribution.&lt;/li&gt;&#xA;&lt;li&gt;random_stable: Sets up a table to generate pseudorandom numbers from a general discrete distribution.&lt;/li&gt;&#xA;&lt;li&gt;random_student_t: Generates pseudorandom numbers from a Student's t distribution.&lt;/li&gt;&#xA;&lt;li&gt;random_triangular: Generates pseudorandom numbers from a triangular distribution.&lt;/li&gt;&#xA;&lt;li&gt;random_uniform: Generates pseudorandom numbers from a uniform (0, 1) distribution.&lt;/li&gt;&#xA;&lt;li&gt;random_von_mises: Generates pseudorandom numbers from a von Mises distribution.&lt;/li&gt;&#xA;&lt;li&gt;random_weibull: Generates pseudorandom numbers from a Weibull distribution.&lt;/li&gt;&#xA;&lt;li&gt;random_general_continuous: Generates pseudorandom numbers from a general continuous distribution.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h2&gt;MULTIVARIATE CONTINUOUS DISTRIBUTIONS&lt;/h2&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;random_normal_multivariate: Generates pseudorandom numbers from a multivariate normal distribution.&lt;/li&gt;&#xA;&lt;li&gt;random_orthogonal_matrix: Generates a pseudorandom orthogonal matrix or a correlation matrix.&lt;/li&gt;&#xA;&lt;li&gt;random_mvar_from_data: Generates pseudorandom numbers from a multivariate distribution determined from a given sample.&lt;/li&gt;&#xA;&lt;li&gt;random_multinomial: Generates pseudorandom numbers from a multinomial distribution.&lt;/li&gt;&#xA;&lt;li&gt;random_sphere: Generates pseudorandom points on a unit circle or K-dimensional sphere.&lt;/li&gt;&#xA;&lt;li&gt;random_table_twoway: Generates a pseudorandom two-way table.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h2&gt;ORDER STATISTICS&lt;/h2&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;random_order_normal: Generates pseudorandom order statistics from a standard normal distribution.&lt;/li&gt;&#xA;&lt;li&gt;random_order_uniform: Generates pseudorandom order statistics from a uniform (0, 1) distribution.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h2&gt;STOCHASTIC PROCESSES&lt;/h2&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;random_arma: Generates pseudorandom ARMA process numbers.&lt;/li&gt;&#xA;&lt;li&gt;random_npp: Generates pseudorandom numbers from a nonhomogeneous Poisson process.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h2&gt;SAMPLES AND PERMUTATIONS&lt;/h2&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;random_permutation: Generates a pseudorandom permutation.&lt;/li&gt;&#xA;&lt;li&gt;random_sample_indices: Generates a simple pseudorandom sample of indices.&lt;/li&gt;&#xA;&lt;li&gt;random_sample: Generates a simple pseudorandom sample from a finite population.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h2&gt;UTILITY FUNCTIONS&lt;/h2&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;random_option: Selects the uniform (0, 1) multiplicative congruential pseudorandom number generator.&lt;/li&gt;&#xA;&lt;li&gt;random_option_get: Retrieves the uniform (0, 1) multiplicative congruential pseudorandom number generator.&lt;/li&gt;&#xA;&lt;li&gt;random_seed_get: Retrieves the current value of the seed used in the IMSL random number generators.&lt;/li&gt;&#xA;&lt;li&gt;random_substream_seed_get: Retrieves a seed for the congruential generators that do not do shuffling that will generate random numbers beginning 100,000 numbers farther along.&lt;/li&gt;&#xA;&lt;li&gt;random_seed_set: Initializes a random seed for use in the IMSL random number generators.&lt;/li&gt;&#xA;&lt;li&gt;random_table_set: Sets the current table used in the shuffled generator.&lt;/li&gt;&#xA;&lt;li&gt;random_table_get: Retrieves the current table used in the shuffled generator.&lt;/li&gt;&#xA;&lt;li&gt;random_GFSR_table_set: Sets the current table used in the GFSR generator.&lt;/li&gt;&#xA;&lt;li&gt;random_GFSR_table_get: Retrieves the current table used in the GFSR generator.&lt;/li&gt;&#xA;&lt;li&gt;random_MT32_init: Initializes the 32-bit Mersenne Twister generator using an array.&lt;/li&gt;&#xA;&lt;li&gt;random_MT32_table_get: Retrieves the current table used in the 32-bit Mersenne Twister generator.&lt;/li&gt;&#xA;&lt;li&gt;random_MT32_table_set: Sets the current table used in the 32-bit Mersenne Twister generator.&lt;/li&gt;&#xA;&lt;li&gt;random_MT64_init: Initializes the 64-bit Mersenne Twister generator using an array.&lt;/li&gt;&#xA;&lt;li&gt;random_MT64_table_get: Retrieves the current table used in the 64-bit Mersenne Twister generator.&lt;/li&gt;&#xA;&lt;li&gt;random_MT64_table_set: Sets the current table used in the 64-bit Mersenne Twister generator.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h2&gt;LOW-DISCREPANCY SEQUENCE&lt;/h2&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;faure_next_point: Computes a shuffled Faure sequence.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="1080" LastEditorUserId="1080" LastEditorDisplayName="" LastEditDate="2011-02-19T02:40:37.700" LastActivityDate="2011-02-19T02:40:37.700" />
  <row Id="7361" PostTypeId="2" ParentId="7358" CreationDate="2011-02-18T04:25:52.087" Score="3" ViewCount="" Body="&lt;p&gt;Boost Random from the Boost C++ libraries could be a good fit for you. In addition to many types of RNGs, it offers a variety of different distributions to draw from, such as&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Uniform (real)&lt;/li&gt;&#xA;&lt;li&gt;Uniform (unit sphere or arbitrary dimension)&lt;/li&gt;&#xA;&lt;li&gt;Bernoulli&lt;/li&gt;&#xA;&lt;li&gt;Binomial&lt;/li&gt;&#xA;&lt;li&gt;Cauchy&lt;/li&gt;&#xA;&lt;li&gt;Gamma&lt;/li&gt;&#xA;&lt;li&gt;Poisson&lt;/li&gt;&#xA;&lt;li&gt;Geometric&lt;/li&gt;&#xA;&lt;li&gt;Triangle&lt;/li&gt;&#xA;&lt;li&gt;Exponential&lt;/li&gt;&#xA;&lt;li&gt;Normal&lt;/li&gt;&#xA;&lt;li&gt;Lognormal&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In addition, &lt;a href=&quot;http://www.boost.org/doc/libs/1_45_0/libs/math/doc/sf_and_dist/html/index.html&quot; rel=&quot;nofollow&quot;&gt;Boost Math&lt;/a&gt; complements the above distributions you can sample from with numerous density functions of many distributions. It also has several neat helper functions; just to give you an idea:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;students_t dist(5);&#xA;&#xA;cout &amp;lt;&amp;lt; &quot;CDF at t = 1 is &quot; &amp;lt;&amp;lt; cdf(dist, 1.0) &amp;lt;&amp;lt; endl;&#xA;cout &amp;lt;&amp;lt; &quot;Complement of CDF at t = 1 is &quot; &amp;lt;&amp;lt; cdf(complement(dist, 1.0)) &amp;lt;&amp;lt; endl;&#xA;&#xA;for(double i = 10; i &amp;lt; 1e10; i *= 10)&#xA;{&#xA;   // Calculate the quantile for a 1 in i chance:&#xA;   double t = quantile(complement(dist, 1/i));&#xA;   // Print it out:&#xA;   cout &amp;lt;&amp;lt; &quot;Quantile of students-t with 5 degrees of freedom\n&quot;&#xA;           &quot;for a 1 in &quot; &amp;lt;&amp;lt; i &amp;lt;&amp;lt; &quot; chance is &quot; &amp;lt;&amp;lt; t &amp;lt;&amp;lt; endl;&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you decided to use Boost, you also get to use its UBLAS library that features a variety of different matrix types and operations.&lt;/p&gt;&#xA;" OwnerUserId="1537" LastActivityDate="2011-02-18T04:25:52.087" CommentCount="1" />
  <row Id="7362" PostTypeId="1" AcceptedAnswerId="7368" CreationDate="2011-02-18T05:58:42.340" Score="2" ViewCount="154" Body="&lt;p&gt;In Orwin's fail safe N test how to decide the values of criterion for a trivial log odd's ratio and mean log odds ratio in missing studies. I am a medical doctor. Please tell me in simple english.&#xA;The data are&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1. Classic fail-safe N &#xA;Z-value for observed studies                                    27.97543&#xA;P-value for observed studies                                    0.00000&#xA;Alpha                                                          0.05000&#xA;Tails                                                           2.00000 &#xA;Z for alpha                                                    1.95996&#xA;Number of observed studies                                    5.00000&#xA;Number of missing studies that wouldbring p-value to &amp;gt;alpha  1014.0000&#xA;&#xA;2. Orwin's fail-safe N&#xA;Odds ration in observed studies                              5.7339 &#xA;Criterian for a ‘trivial’ odds ratio                           ?&#xA;Mean odds ratio in missing studies                           ?&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2956" LastEditorUserId="307" LastEditorDisplayName="" LastEditDate="2011-02-18T09:45:21.120" LastActivityDate="2011-02-18T10:01:57.817" Title="Orwin's fail safe N test" Tags="&lt;meta-analysis&gt;&lt;publication-bias&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="7363" PostTypeId="2" ParentId="7326" CreationDate="2011-02-18T06:07:14.057" Score="0" ViewCount="" Body="&lt;p&gt;I believe a chi-squared test is what you are looking for. Because your dataset has a long tail, many tags will not be sampled well or will not end up in your sample at all. You may want to look into Yates' chi-square test, which attempts to correct for this by loosening the standards of what is significant for rare tags.    &lt;/p&gt;&#xA;" OwnerUserId="2965" LastActivityDate="2011-02-18T06:07:14.057" />
  <row Id="7364" PostTypeId="1" CreationDate="2011-02-18T06:10:01.733" Score="4" ViewCount="66" Body="&lt;p&gt;The standard factor model formulation is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$y=W x+\epsilon$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $x \sim \mathcal{N}(0, I)$, $\epsilon \sim\mathcal{N}(0, \Sigma)$. $W$ and $\Sigma$ are typically estimated from MLE. The solution can be obtained numerically; in general there are no analytical solutions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Question: assume that $\Sigma$ belongs to some class of psd matrices, such as matrices of bounded norm, or with bounded trace. Is there an analytical solution to factor models in which the noise is vanishing, i.e.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$y=W x+ n^{-1}\epsilon$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;with $n\rightarrow \infty$? By this, I mean than the solutions $(W_n, \Sigma_n)$ converge and that possibly the solution can be expressed in closed form. I know the answer is in the affirmative when it is known a priori that $\Sigma = I$, but I would be happy to see that it holds more generally.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Pointers to literature welcome.&lt;/p&gt;&#xA;" OwnerUserId="30" LastActivityDate="2011-02-18T06:10:01.733" Title="Factor models with small noises" Tags="&lt;factor-analysis&gt;&lt;maximum-likelihood&gt;&lt;asymptotics&gt;" CommentCount="6" />
  <row Id="7365" PostTypeId="2" ParentId="7308" CreationDate="2011-02-18T08:56:34.040" Score="6" ViewCount="" Body="&lt;p&gt;The quotation in full &lt;a href=&quot;http://books.google.com/books?id=cdBPOJUP4VsC&amp;amp;lpg=PP1&amp;amp;dq=wooldridge%20econometrics&amp;amp;hl=fr&amp;amp;pg=PA357#v=onepage&amp;amp;q=wooldridge%20econometrics&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;can be found here&lt;/a&gt;. The estimate $\hat{\theta}_N$ is the solution of minimization problem (&lt;a href=&quot;http://books.google.com/books?id=cdBPOJUP4VsC&amp;amp;lpg=PP1&amp;amp;dq=wooldridge%20econometrics&amp;amp;hl=fr&amp;amp;pg=PA357#v=onepage&amp;amp;q=wooldridge%20econometrics&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;page 344&lt;/a&gt;):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align}&#xA;\min_{\theta\in \Theta}N^{-1}\sum_{i=1}^Nq(w_i,\theta)&#xA;\end{align}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the solution $\hat{\theta}_N$ is interior point of $\Theta$, objective function is twice differentiable and gradient of the objective function is zero, then Hessian of the objective function (which is $\hat{H}$) is positive semi-definite. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now what Wooldridge is saying that for given sample the empirical Hessian is not guaranteed to be positive definite or even positive semidefinite. This is true, since Wooldridge does not require that objective function $N^{-1}\sum_{i=1}^Nq(w_i,\theta)$ has nice properties, he requires that there exists a unique solution $\theta_0$ for&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\min_{\theta\in\Theta}Eq(w,\theta).$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So for given sample objective function $N^{-1}\sum_{i=1}^Nq(w_i,\theta)$ may be minimized on the boundary point of $\Theta$ in which Hessian of objective function needs not to be positive definite.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Further in his book Wooldridge gives an examples of estimates of Hessian which are guaranteed to be numerically positive definite. In practice non-positive definiteness of Hessian should indicate that solution is either on the boundary point or the algorithm failed to find the solution. Which usually is a further indication that the model fitted may be inappropriate for a given data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the numerical example. I generate non-linear least squares problem:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$y_i=c_1x_i^{c_2}+\varepsilon_i$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I take $X$ uniformly distributed in interval $[1,2]$ and $\varepsilon$ normal with zero mean and variance $\sigma^2$. I generated a sample of size 10, in R 2.11.1 using &lt;code&gt;set.seed(3)&lt;/code&gt;. Here is the &lt;a href=&quot;http://mif.vu.lt/~zemlys/download/source/badhessian.csv&quot; rel=&quot;nofollow&quot;&gt;link to the values&lt;/a&gt; of $x_i$ and $y_i$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I chose the objective function square of usual non-linear least squares objective function:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$q(w,\theta)=(y-c_1x_i^{c_2})^4$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the code in R for optimising function, its gradient and hessian. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;##First set-up the epxressions for optimising function, its gradient and hessian.&#xA;##I use symbolic derivation of R to guard against human error    &#xA;mt &amp;lt;- expression((y-c1*x^c2)^4)&#xA;&#xA;gradmt &amp;lt;- c(D(mt,&quot;c1&quot;),D(mt,&quot;c2&quot;))&#xA;&#xA;hessmt &amp;lt;- lapply(gradmt,function(l)c(D(l,&quot;c1&quot;),D(l,&quot;c2&quot;)))&#xA;&#xA;##Evaluate the expressions on data to get the empirical values. &#xA;##Note there was a bug in previous version of the answer res should not be squared.&#xA;optf &amp;lt;- function(p) {&#xA;    res &amp;lt;- eval(mt,list(y=y,x=x,c1=p[1],c2=p[2]))&#xA;    mean(res)&#xA;}&#xA;&#xA;gf &amp;lt;- function(p) {&#xA;    evl &amp;lt;- list(y=y,x=x,c1=p[1],c2=p[2]) &#xA;    res &amp;lt;- sapply(gradmt,function(l)eval(l,evl))&#xA;    apply(res,2,mean)&#xA;}&#xA;&#xA;hesf &amp;lt;- function(p) {&#xA;    evl &amp;lt;- list(y=y,x=x,c1=p[1],c2=p[2]) &#xA;    res1 &amp;lt;- lapply(hessmt,function(l)sapply(l,function(ll)eval(ll,evl)))&#xA;    res &amp;lt;- sapply(res1,function(l)apply(l,2,mean))&#xA;    res&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;First test that gradient and hessian works as advertised. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(3)&#xA;x &amp;lt;- runif(10,1,2)&#xA;y &amp;lt;- 0.3*x^0.2&#xA;&#xA;&amp;gt; optf(c(0.3,0.2))&#xA;[1] 0&#xA;&amp;gt; gf(c(0.3,0.2))&#xA;[1] 0 0&#xA;&amp;gt; hesf(c(0.3,0.2))&#xA;     [,1] [,2]&#xA;[1,]    0    0&#xA;[2,]    0    0&#xA;&amp;gt; eigen(hesf(c(0.3,0.2)))$values&#xA;[1] 0 0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The hessian is zero, so it is positive semi-definite. Now for the values of $x$ and $y$ given in the link we get&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; df &amp;lt;- read.csv(&quot;badhessian.csv&quot;)&#xA;&amp;gt; df&#xA;          x          y&#xA;1  1.168042  0.3998378&#xA;2  1.807516  0.5939584&#xA;3  1.384942  3.6700205&#xA;4  1.327734 -3.3390724&#xA;5  1.602101  4.1317608&#xA;6  1.604394 -1.9045958&#xA;7  1.124633 -3.0865249&#xA;8  1.294601 -1.8331763&#xA;9  1.577610  1.0865977&#xA;10 1.630979  0.7869717&#xA;&amp;gt; x &amp;lt;- df$x&#xA;&amp;gt; y &amp;lt;- df$y&#xA;&amp;gt; opt &amp;lt;- optim(c(1,1),optf,gr=gf,method=&quot;BFGS&quot;)  &#xA;&amp;gt; opt$par&#xA;[1] -114.91316  -32.54386&#xA;&amp;gt; gf(opt$par)&#xA;[1] -0.0005795979 -0.0002399711&#xA;&amp;gt; hesf(opt$par)&#xA;              [,1]         [,2]&#xA;[1,]  0.0002514806 -0.003670634&#xA;[2,] -0.0036706345  0.050998404&#xA;&amp;gt; eigen(hesf(opt$par))$values&#xA;[1]  5.126253e-02 -1.264959e-05&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Gradient is zero, but the hessian is non positive. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Note:&lt;/em&gt; This is my third attempt to give an answer. I hope I finally managed to give precise mathematical statements, which eluded me in the previous versions. &lt;/p&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-02-25T11:23:38.733" LastActivityDate="2011-02-25T11:23:38.733" CommentCount="9" />
  <row Id="7366" PostTypeId="1" CreationDate="2011-02-18T09:09:36.343" Score="2" ViewCount="145" Body="&lt;p&gt;Hi everybody, I am a beginner in statistics, therefore I hope I can state my problem in a correct manner. I have a some instances or samples and I can collect below statistical parameters for &lt;strong&gt;classification and regression&lt;/strong&gt; problem:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Sample Size&lt;/li&gt;&#xA;&lt;li&gt;Minimum value&lt;/li&gt;&#xA;&lt;li&gt;Maximum value&lt;/li&gt;&#xA;&lt;li&gt;Standard deviation&lt;/li&gt;&#xA;&lt;li&gt;Variance&lt;/li&gt;&#xA;&lt;li&gt;Mean&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;And, I want to use &lt;strong&gt;z-scores&lt;/strong&gt; to compare or classify samples, my question is: does using z-score &lt;strong&gt;make sense&lt;/strong&gt; or what can I use instead of z-score to obtain meaningful classification parameter?&lt;/p&gt;&#xA;" OwnerUserId="2170" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-02-18T10:05:42.170" LastActivityDate="2011-02-18T11:08:11.973" Title="Is z-score meaningful in classification or regression?" Tags="&lt;classification&gt;&lt;z-statistic&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="1" />
  <row Id="7367" PostTypeId="2" ParentId="7358" CreationDate="2011-02-18T09:32:00.917" Score="6" ViewCount="" Body="&lt;p&gt;I would strongly suggest that you have a look at &lt;code&gt;RCpp&lt;/code&gt; and &lt;code&gt;RcppArmadillo&lt;/code&gt; packages for &lt;code&gt;R&lt;/code&gt;. Basically, you would not need to worry about the wrappers as they are already &quot;included&quot;. Furthermore the syntactic sugar is really sweet (pun intended).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a side remark, I would recommend that you have a look at &lt;code&gt;JAGS&lt;/code&gt;, which does MCMC and its source code is in C++.&lt;/p&gt;&#xA;" OwnerUserId="1443" LastActivityDate="2011-02-18T09:32:00.917" CommentCount="2" />
  <row Id="7368" PostTypeId="2" ParentId="7362" CreationDate="2011-02-18T10:01:57.817" Score="6" ViewCount="" Body="&lt;p&gt;The criterion for a 'trivial' effect size (odds ratio in your example) should be decided based on the size of effect that would be considered 'trivial' in the particular scenario, rather than on statistical grounds. If you're looking at an intervention that may be given to a considerable segment of the population with few side-effects and may prevent early death in a few (statins are one example that come to my mind, but you're the medic), then a small reduction in death rates might still be important, so a trivial reduction could perhaps be 1% or less, i.e. an odds ratio of 0.99 or closer to 1. If you're looking at an invasive or costly intervention or one with severe side-effects, or a condition that is an irritation or of short duration, the trivial reduction would be very much larger.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rosenthal's original fail-safe N based on statistical significance assumed the mean effect size in missing studies was the null effect size. Orwin's method allows you to choose this, but the null effect size remains the simplest choice.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having said all that, I don't like either Rosenthal's or Orwin's 'fail-safe N' myself (though I prefer Orwin's to Rosenthal's). As Rosenberg points out in the abstract of the paper below, they &quot;are unweighted and are not based on the framework in which most meta-analyses are performed&quot;. He suggests a general, weighted fail-safe N using either the fixed- or random-effects frameworks that are far more commonly used for meta-analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Michael S. Rosenberg. &lt;a href=&quot;http://dx.doi.org/10.1111/j.0014-3820.2005.tb01004.x&quot; rel=&quot;nofollow&quot;&gt;The file-drawer problem revisited: a general weighted method for calculating fail-safe numbers in meta-analysis.&lt;/a&gt; &lt;em&gt;Evolution&lt;/em&gt; &lt;strong&gt;59&lt;/strong&gt; (2):464-468, 2005.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-02-18T10:01:57.817" CommentCount="1" />
  <row Id="7369" PostTypeId="2" ParentId="7344" CreationDate="2011-02-18T10:04:53.950" Score="3" ViewCount="" Body="&lt;p&gt;In addition to @mpiktas's comment, you can also have a look at the &lt;a href=&quot;http://cran.r-project.org/web/packages/rms/index.html&quot; rel=&quot;nofollow&quot;&gt;rms&lt;/a&gt; package from Frank Harrell. The advantage is that it handles both LM and GLM for model fitting and prediction; see for example the &lt;code&gt;plot.Predict()&lt;/code&gt; function. If you're planning to do serious job in regression modeling, this package and its companion &lt;a href=&quot;http://cran.r-project.org/web/packages/Hmisc/index.html&quot; rel=&quot;nofollow&quot;&gt;Hmisc&lt;/a&gt; are really good.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-02-18T10:04:53.950" />
  <row Id="7370" PostTypeId="1" AcceptedAnswerId="7372" CreationDate="2011-02-18T10:21:17.707" Score="3" ViewCount="82" Body="&lt;p&gt;I would like to check different gradient algorithms. For example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; fr &amp;lt;- function(x) {   ## Rosenbrock Banana function&#xA;         x1 &amp;lt;- x[1]&#xA;         x2 &amp;lt;- x[2]&#xA;        print(c(x1,x2))&#xA;         100 * (x2 - x1 * x1)^2 + (1 - x1)^2&#xA;     }&#xA;optim(c(-1.2,1),fr,method=&quot;BFGS&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;prints to the screen the values at which the RBF has been evaluated. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I store these values in a matrix ? (instead of just printing them to the screen)&lt;/p&gt;&#xA;" OwnerUserId="603" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-02-21T06:56:36.677" LastActivityDate="2011-02-21T06:56:36.677" Title="How to store checks of gradient algorithm in a matrix using R?" Tags="&lt;r&gt;&lt;optimization&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="7372" PostTypeId="2" ParentId="7370" CreationDate="2011-02-18T11:00:21.777" Score="3" ViewCount="" Body="&lt;p&gt;Use the function &lt;strong&gt;capture.output&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cc&amp;lt;-capture.output(vv&amp;lt;-optim(c(-1.2,1),fr,method=&quot;BFGS&quot;))&#xA;t(sapply(strsplit(gsub(&quot; +&quot;,&quot; &quot;,cc),&quot; &quot;),function(l)as.numeric(l[2:3])))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The variable &lt;code&gt;vv&lt;/code&gt; is used so that the result of optim will not be printed only your calls to the function. Each call to the function results in one element of &lt;code&gt;cc&lt;/code&gt;. Then I strip extra spaces, so that I can split the strings with the space. Afterwards you simply select what is needed. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This solution will work if optim produces errors, but I think it is not very hard to adapt the code to guard against that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is of course only the quick hack. For more complicated solutions you will need to delve into the code of optim.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;@onestop suggested using &lt;strong&gt;sink&lt;/strong&gt;, it is more simple:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sink(&quot;bu.txt&quot;)&#xA;vv&amp;lt;-optim(c(-1.2,1),fr,method=&quot;BFGS&quot;)&#xA;sink()&#xA;read.table(&quot;bu.txt&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Note the additional &lt;code&gt;sink&lt;/code&gt;, so that the output is diverted again to the console.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-02-19T06:50:00.900" LastActivityDate="2011-02-19T06:50:00.900" CommentCount="2" />
  <row Id="7373" PostTypeId="2" ParentId="7366" CreationDate="2011-02-18T11:08:11.973" Score="3" ViewCount="" Body="&lt;p&gt;In order to perform linear regression you'd need not only the means and variances of the variables but also all their covariances (or equivalently their correlations). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you can collect the means, variances and covariances/correlations separately in each of the classes you wish to classify, you can do &lt;a href=&quot;http://en.wikipedia.org/wiki/Linear_discriminant_analysis&quot; rel=&quot;nofollow&quot;&gt;linear discriminant analysis&lt;/a&gt;, which is a classification method, albeit a somewhat old-fashioned one with some rather restrictive normality assumptions.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-02-18T11:08:11.973" CommentCount="4" />
  <row Id="7374" PostTypeId="2" ParentId="7351" CreationDate="2011-02-18T12:10:29.613" Score="1" ViewCount="" Body="&lt;p&gt;You could try '&lt;a href=&quot;http://library.mpib-berlin.mpg.de/ft/gg/GG_Teaching_2001.pdf&quot; rel=&quot;nofollow&quot;&gt;Teaching Bayesian Reasoning In Less Than Two Hours&lt;/a&gt;'.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know it's not web resource as you wanted but I found Lynch's &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1441924345&quot; rel=&quot;nofollow&quot;&gt;book&lt;/a&gt; very approachable.  &lt;/p&gt;&#xA;" OwnerUserId="22" LastEditorUserId="2129" LastEditorDisplayName="" LastEditDate="2011-02-18T17:54:53.613" LastActivityDate="2011-02-18T17:54:53.613" />
  <row Id="7375" PostTypeId="2" ParentId="7351" CreationDate="2011-02-18T12:59:46.510" Score="3" ViewCount="" Body="&lt;p&gt;If you'd like to try a few learn by examples, you may be interested in &quot;Bayesian Computation in R&quot; (http://bayes.bgsu.edu/bcwr/) by Jim Albert. It's related R package is called LearnBayes. &lt;/p&gt;&#xA;" OwnerUserId="3306" LastActivityDate="2011-02-18T12:59:46.510" />
  <row Id="7376" PostTypeId="1" AcceptedAnswerId="7378" CreationDate="2011-02-18T13:07:06.643" Score="4" ViewCount="227" Body="&lt;p&gt;Inter-market analysis is a method of modeling market behavior by means of finding relationships between different markets. Often times, a correlation is computed between two markets, say S&amp;amp;P 500 and 30-Year US treasuries. These computations are more often than not based on price data, which is obvious to everyone that it does not fit the definition of stationary time series. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Possible solutions aside (using returns instead), is the computation of correlation whose data is non-stationary even a valid statistical calculation?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would you say that such a correlation calculation is somewhat unreliable, or just plain nonsense?&lt;/p&gt;&#xA;" OwnerUserId="3306" LastActivityDate="2011-02-21T07:05:04.457" Title="Does correlation assume stationarity of data?" Tags="&lt;correlation&gt;&lt;stationarity&gt;" AnswerCount="2" CommentCount="5" FavoriteCount="2" />
  <row Id="7377" PostTypeId="2" ParentId="7358" CreationDate="2011-02-18T15:41:38.457" Score="9" ViewCount="" Body="&lt;p&gt;We have spent some time making the &lt;strong&gt;wrapping&lt;/strong&gt; from C++ into &lt;a href=&quot;http://www.r-project.org&quot; rel=&quot;nofollow&quot;&gt;R&lt;/a&gt; (and back for that matter) a lot easier via our &lt;a href=&quot;http://dirk.eddelbuettel.com/code/rcpp.html&quot; rel=&quot;nofollow&quot;&gt;Rcpp&lt;/a&gt; package.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;And because linear algebra is already such a well-understood and coded-for field, &lt;a href=&quot;http://arma.sf.net&quot; rel=&quot;nofollow&quot;&gt;Armadillo&lt;/a&gt;, a current, modern, plesant, well-documted, small, templated, ... library was a very natural fit for our first extended wrapper: &lt;a href=&quot;http://dirk.eddelbuettel.com/code/rcpp.armadillo.html&quot; rel=&quot;nofollow&quot;&gt;RcppArmadillo&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This has caught the attention of other MCMC users as well. I gave a one-day work at the U of Rochester business school last summer, and have help another researcher in the MidWest with similar explorations.  Give &lt;a href=&quot;http://dirk.eddelbuettel.com/code/rcpp.armadillo.html&quot; rel=&quot;nofollow&quot;&gt;RcppArmadillo&lt;/a&gt; a try -- it works well, is actively maintained (new Armadillo release 1.1.4 today, I will make a new RcppArmadillo later) and supported.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And because I just luuv this example so much, here is a quick &quot;fast&quot; version of &lt;code&gt;lm()&lt;/code&gt; returning coefficient and std.errors:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;extern &quot;C&quot; SEXP fastLm(SEXP ys, SEXP Xs) {&#xA;&#xA;  try {&#xA;    Rcpp::NumericVector yr(ys);                 // creates Rcpp vector &#xA;    Rcpp::NumericMatrix Xr(Xs);                 // creates Rcpp matrix &#xA;    int n = Xr.nrow(), k = Xr.ncol();&#xA;&#xA;    arma::mat X(Xr.begin(), n, k, false);       // avoids extra copy&#xA;    arma::colvec y(yr.begin(), yr.size(), false);&#xA;&#xA;    arma::colvec coef = arma::solve(X, y);      // fit model y ~ X&#xA;    arma::colvec res = y - X*coef;              // residuals&#xA;&#xA;    double s2 = std::inner_product(res.begin(), res.end(), &#xA;                                   res.begin(), double())/(n - k);&#xA;                                            // std.errors of coefficients&#xA;    arma::colvec std_err = &#xA;           arma::sqrt(s2 * arma::diagvec( arma::pinv(arma::trans(X)*X) ));  &#xA;&#xA;    return Rcpp::List::create(Rcpp::Named(&quot;coefficients&quot;) = coef,&#xA;                              Rcpp::Named(&quot;stderr&quot;)       = std_err,&#xA;                              Rcpp::Named(&quot;df&quot;)           = n - k);&#xA;&#xA;  } catch( std::exception &amp;amp;ex ) {&#xA;      forward_exception_to_r( ex );&#xA;  } catch(...) { &#xA;      ::Rf_error( &quot;c++ exception (unknown reason)&quot; ); &#xA;  }&#xA;  return R_NilValue; // -Wall&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Lastly, you also get immediate prototyping via &lt;a href=&quot;http://cran.r-project.org/package=inline&quot; rel=&quot;nofollow&quot;&gt;inline&lt;/a&gt; which may make 'time to code' faster.&lt;/p&gt;&#xA;" OwnerUserId="334" LastEditorUserId="334" LastEditorDisplayName="" LastEditDate="2011-02-18T16:39:18.800" LastActivityDate="2011-02-18T16:39:18.800" CommentCount="4" />
  <row Id="7378" PostTypeId="2" ParentId="7376" CreationDate="2011-02-18T15:46:08.050" Score="9" ViewCount="" Body="&lt;p&gt;The correlation measures linear relationship. In informal context relationship means something stable.  When we calculate the sample correlation for stationary variables and increase the number of available data points this sample correlation tends to true correlation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It can be shown that for prices, which usually are random walks, the sample correlation tends to random variable. This means that no matter how much data we have, the result will always be different.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Note&lt;/em&gt; I tried expressing mathematical intuition without the mathematics. From mathematical point of view the explanation is very clear: Sample moments of stationary processes converge in probability to constants. Sample moments of random walks converge to integrals of brownian motion which are random variables. Since relationship is usually expressed as a number and not a random variable, the reason for not calculating the correlation for non-stationary variables becomes evident.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt; Since we are interested in correlation between two variables assume first that they come from stationary process $Z_t=(X_t,Y_t)$. Stationarity implies that $EZ_t$ and $cov(Z_t,Z_{t-h})$ do not  depend on $t$. So correlation&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$corr(X_t,Y_t)=\frac{cov(X_t,Y_t)}{\sqrt{DX_tDY_t}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;also does not depend on $t$, since all the quantities in the formula come from matrix $cov(Z_t)$, which does not depend on $t$. So calculation of sample correlation &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\hat{\rho}=\frac{\frac{1}{T}\sum_{t=1}^T(X_t-\bar{X})(Y_t-\bar{Y})}{\sqrt{\frac{1}{T^2}\sum_{t=1}^T(X_t-\bar{X})^2\sum_{t=1}^T(Y_t-\bar{Y})}}$$&#xA;makes sense, since we may have reasonable hope that sample correlation will estimate $\rho=corr(X_t,Y_t)$. It turns out that this hope is not unfounded, since for stationary processes satisfying certain conditions we have that $\hat{\rho}\to\rho$, as $T\to\infty$ in probability. Furthermore $\sqrt{T}(\hat{\rho}-\rho)\to N(0,\sigma_{\rho}^2)$ in distribution, so we can test the hypotheses about $\rho$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now suppose that $Z_t$ is not stationary. Then $corr(X_t,Y_t)$ may depend on $t$. So when we observe a sample of size $T$ we potentialy need to estimate $T$ different correlations $\rho_t$. This is of course infeasible, so in best case scenario we only can estimate some functional of $\rho_t$ such as mean or variance. But the result may not have sensible interpretation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now let us examine what happens with correlation of probably most studied non-stationary process random walk. We call process $Z_t=(X_t,Y_t)$ a random walk if $Z_t=\sum_{s=1}^t(U_t,V_t)$, where $C_t=(U_t,V_t)$ is a stationary process. For simplicity assume that $EC_t=0$. Then &lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align}&#xA;corr(X_tY_t)=\frac{EX_tY_t}{\sqrt{DX_tDY_t}}=\frac{E\sum_{s=1}^tU_t\sum_{s=1}^tV_t}{\sqrt{D\sum_{s=1}^tU_tD\sum_{s=1}^tV_t}}&#xA;\end{align}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To simplify matters further, assume that $C_t=(U_t,V_t)$ is a white noise. This means that all correlations $E(C_tC_{t+h})$ are zero for $h&amp;gt;0$. Note that this does not restrict $corr(U_t,V_t)$ to zero. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then &#xA;\begin{align}&#xA;corr(X_t,Y_t)=\frac{tEU_tV_t}{\sqrt{t^2DU_tDV_t}}=corr(U_0,V_0).&#xA;\end{align}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far so good, though the process is not stationary, correlation makes sense, although we had to make same restrictive assumptions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now to see what happens to sample correlation we will need to use the following fact about random walks, called functional central limit theorem:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align}&#xA;\frac{1}{\sqrt{T}}Z_{[Ts]}=\frac{1}{\sqrt{T}}\sum_{t=1}^{[Ts]}C_t\to (cov(C_0))^{-1/2}W_s,&#xA;\end{align}&#xA;in distribution, where $s\in[0,1]$ and $W_s=(W_{1s},W_{2s})$ is bivariate &lt;a href=&quot;http://en.wikipedia.org/wiki/Wiener_process&quot; rel=&quot;nofollow&quot;&gt;Brownian motion&lt;/a&gt; (two-dimensional Wiener process). For convenience introduce definition $M_s=(M_{1s},M_{2s})=(cov(C_0))^{-1/2}W_s$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Again for simplicity let us define sample correlation as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align}&#xA;\hat{\rho}=\frac{\frac{1}{T}\sum_{t=1}^TX_{t}Y_t}{\sqrt{\frac{1}{T}\sum_{t=1}^TX_t^2\frac{1}{T}\sum_{t=1}^TY_t^2}}&#xA;\end{align}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let us start with the variances. We have&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align}&#xA;E\frac{1}{T}\sum_{t=1}^TX_t^2=\frac{1}{T}E\sum_{t=1}^T\left(\sum_{s=1}^tU_t\right)^2=\frac{1}{T}\sum_{t=1}^Tt\sigma_U^2=\sigma_U\frac{T+1}{2}.&#xA;\end{align}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This goes to infinity as $T$ increases, so we hit the first problem, sample variance does not converge. On the other hand  &lt;a href=&quot;http://en.wikipedia.org/wiki/Continuous_mapping_theorem&quot; rel=&quot;nofollow&quot;&gt;continuous mapping theorem&lt;/a&gt; in conjunction with functional central limit theorem gives us&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align}&#xA;\frac{1}{T^2}\sum_{t=1}^TX_t^2=\sum_{t=1}^T\frac{1}{T}\left(\frac{1}{\sqrt{T}}\sum_{s=1}^tU_t\right)^2\to \int_0^1M_{1s}^2ds&#xA;\end{align}&#xA;where convergence is convergence in distribution, as $T\to \infty$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Similarly we get&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align}&#xA;\frac{1}{T^2}\sum_{t=1}^TY_t^2\to \int_0^1M_{2s}^2ds&#xA;\end{align}&#xA;and&#xA;\begin{align}&#xA;\frac{1}{T^2}\sum_{t=1}^TX_tY_t\to \int_0^1M_{1s}M_{2s}ds&#xA;\end{align}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So finally  for sample correlation of our random walk we get&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align}&#xA;\hat{\rho}\to \frac{\int_0^1M_{1s}M_{2s}ds}{\sqrt{\int_0^1M_{1s}^2ds\int_0^1M_{2s}^2ds}}&#xA;\end{align}&#xA;in distribution as $T\to \infty$.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So although correlation is well defined, sample correlation does not converge towards it, as in stationary process case. Instead it converges to a certain random variable. &lt;/p&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-02-21T07:05:04.457" LastActivityDate="2011-02-21T07:05:04.457" CommentCount="5" />
  <row Id="7379" PostTypeId="1" CreationDate="2011-02-18T16:40:42.403" Score="2" ViewCount="68" Body="&lt;p&gt;I'm reading through someone else's code for plotting the results of a psychology experiment, and (according to the code comments) they calculate the accuracy error of their behavioral paradigm as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\textit{accuracy error} = \sqrt{\frac{(\textit{accuracy}) (1-\textit{accuracy})}{\textit{total trials}}}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's output seems to be very similar to the original accuracy. What is this? Is this some sort of multiple comparison correction? Why would they do this?&lt;/p&gt;&#xA;" OwnerUserId="2019" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-02-18T16:54:52.120" LastActivityDate="2011-02-18T16:54:52.120" Title="What is this measure of error?" Tags="&lt;binomial&gt;&lt;error&gt;" CommentCount="6" />
  <row Id="7380" PostTypeId="5" CreationDate="2011-02-18T17:52:05.987" Score="0" ViewCount="" Body="&lt;p&gt;Econometrics is a field of statistics dealing with applications to economics. &lt;/p&gt;&#xA;" OwnerUserId="-1" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-02-18T17:54:26.343" LastActivityDate="2011-02-18T17:54:26.343" />
  <row Id="7381" PostTypeId="4" CreationDate="2011-02-18T17:52:05.987" Score="0" ViewCount="" Body="Econometrics is a field of statistics dealing with applications to economics." OwnerUserId="-1" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-02-18T20:29:10.600" LastActivityDate="2011-02-18T20:29:10.600" />
  <row Id="7382" PostTypeId="2" ParentId="7376" CreationDate="2011-02-18T19:18:50.377" Score="4" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;...is the computation of correlation whose data is non-stationary even a valid statistical calculation?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Let $W$ be a discrete random walk.  Pick a positive number $h$.  Define the processes $P$ and $V$ by $P(0) = 1$, $P(t+1) = -P(t)$ if $V(t) &amp;gt;  h$, and otherwise $P(t+1) = P(t)$; and $V(t) = P(t)W(t)$.  In other words, $V$ starts out identical to $W$ but every time $V$ rises above $h$, it switches signs (otherwise emulating $W$ in all respects).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/lKGlL.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(In this figure (for $h=5$) $W$ is blue and $V$ is red.  There are four switches in sign.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In effect, over short periods of time $V$ tends to be either perfectly correlated with $W$ or perfectly anticorrelated with it; however, using a correlation function to describe the relationship between $V$ and $W$ wouldn't be &lt;em&gt;useful&lt;/em&gt; (a word that perhaps more aptly captures the problem than &quot;unreliable&quot; or &quot;nonsense&quot;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Mathematica code to produce the figure:&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;With[{h=5},&#xA;pv[{p_, v_}, w_] := With[{q=If[v &amp;gt; h, -p, p]}, {q, q w}];&#xA;w = Accumulate[RandomInteger[{-1,1}, 25 h^2]];&#xA;{p,v} = FoldList[pv, {1,0}, w] // Transpose;&#xA;ListPlot[{w,v}, Joined-&amp;gt;True]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-02-18T19:18:50.377" CommentCount="4" />
  <row Id="7383" PostTypeId="2" ParentId="4762" CreationDate="2011-02-18T20:40:13.133" Score="1" ViewCount="" Body="&lt;p&gt;To use SPSS for the Lack of fit test go to: Analyze&gt;&gt;Compare Means&gt;&gt;Means.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then in the dialogue box that appears assign your Independent and Dependent Variables. Select Options and a new dialogue box will appear. Check the option at the bottom of the screen that says &quot;Test for Linearity&quot;.&lt;/p&gt;&#xA;" OwnerUserId="3308" LastActivityDate="2011-02-18T20:40:13.133" />
  <row Id="7384" PostTypeId="2" ParentId="7268" CreationDate="2011-02-18T20:53:58.767" Score="1" ViewCount="" Body="&lt;p&gt;Aggregation also works without using &lt;code&gt;zoo&lt;/code&gt; (with random data from 2 variables for 3 days and 4 hosts like from JWM). I assume that you have data from all hosts for each hour.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;nHosts &amp;lt;- 4  # number of hosts&#xA;dates  &amp;lt;- seq(as.POSIXct(&quot;2011-01-01 00:00:00&quot;),&#xA;              as.POSIXct(&quot;2011-01-03 23:59:30&quot;), by=30)&#xA;hosts  &amp;lt;- factor(sample(1:nHosts, length(dates), replace=TRUE),&#xA;                 labels=paste(&quot;host&quot;, 1:nHosts, sep=&quot;&quot;))&#xA;x1     &amp;lt;- sample(0:20, length(dates), replace=TRUE)  # data from 1st variable&#xA;x2     &amp;lt;- rpois(length(dates), 2)                    # data from 2nd variable&#xA;Data   &amp;lt;- data.frame(dates=dates, hosts=hosts, x1=x1, x2=x2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm not entirely sure if you want to average just within each hour, or within each hour over all days. I'll do both.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Data$hFac &amp;lt;- droplevels(cut(Data$dates, breaks=&quot;hour&quot;))&#xA;Data$hour &amp;lt;- as.POSIXlt(dates)$hour  # extract hour of the day&#xA;&#xA;# average both variables over days within each hour and host&#xA;# formula notation was introduced in R 2.12.0 I think&#xA;res1 &amp;lt;- aggregate(cbind(x1, x2) ~ hour + hosts, data=Data, FUN=mean)&#xA;# only average both variables within each hour and host&#xA;res2 &amp;lt;- aggregate(cbind(x1, x2) ~ hFac + hosts, data=Data, FUN=mean)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The result looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; head(res1)&#xA;  hour hosts        x1       x2&#xA;1    0 host1  9.578431 2.049020&#xA;2    1 host1 10.200000 2.200000&#xA;3    2 host1 10.423077 2.153846&#xA;4    3 host1 10.241758 1.879121&#xA;5    4 host1  8.574713 2.011494&#xA;6    5 host1  9.670588 2.070588&#xA;&#xA;&amp;gt; head(res2)&#xA;                 hFac hosts        x1       x2&#xA;1 2011-01-01 00:00:00 host1  9.192308 2.307692&#xA;2 2011-01-01 01:00:00 host1 10.677419 2.064516&#xA;3 2011-01-01 02:00:00 host1 11.041667 1.875000&#xA;4 2011-01-01 03:00:00 host1 10.448276 1.965517&#xA;5 2011-01-01 04:00:00 host1  8.555556 2.074074&#xA;6 2011-01-01 05:00:00 host1  8.809524 2.095238&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm also not entirely sure about the type of graph you want. Here's the bare-bones version of a graph for just the first variable with separate data lines for each host.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# using the data that is averaged over days as well&#xA;res1L &amp;lt;- split(subset(res1, select=&quot;x1&quot;), res1$hosts)&#xA;mat1  &amp;lt;- do.call(cbind, res1L)&#xA;colnames(mat1) &amp;lt;- levels(hosts)&#xA;rownames(mat1) &amp;lt;- 0:23&#xA;matplot(mat1, main=&quot;x1 per hour, avg. over days&quot;, xaxt=&quot;n&quot;, type=&quot;o&quot;, pch=16, lty=1)&#xA;axis(side=1, at=seq(0, 23, by=2))&#xA;legend(x=&quot;topleft&quot;, legend=colnames(mat1), col=1:nHosts, lty=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The same graph for the data that is only averaged within each hour.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;res2L &amp;lt;- split(subset(res2, select=&quot;x1&quot;), res2$hosts)&#xA;mat2  &amp;lt;- do.call(cbind, res2L)&#xA;colnames(mat2) &amp;lt;- levels(hosts)&#xA;rownames(mat2) &amp;lt;- levels(Data$hFac)&#xA;matplot(mat2, main=&quot;x1 per hour&quot;, type=&quot;o&quot;, pch=16, lty=1)&#xA;legend(x=&quot;topleft&quot;, legend=colnames(mat2), col=1:nHosts, lty=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1909" LastEditorUserId="1909" LastEditorDisplayName="" LastEditDate="2011-02-18T21:03:34.373" LastActivityDate="2011-02-18T21:03:34.373" CommentCount="1" />
  <row Id="7385" PostTypeId="1" AcceptedAnswerId="7418" CreationDate="2011-02-18T21:20:51.227" Score="5" ViewCount="333" Body="&lt;p&gt;Hi everyone, this is my first post. I'm truly grateful for this community.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am trying to analyze longitudinal count data that is zero-truncated (probability that response variable = 0 is 0), and the mean != variance, so a negative binomial distribution was chosen over a poisson.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Functions/commands I've ruled out:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;R&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;gee() function in R does not account for zero-truncation nor the negative binomial distribution (not even with the MASS package loaded)&lt;/li&gt;&#xA;&lt;li&gt;glm.nb() in R doesn't allow for different correlation structures&lt;/li&gt;&#xA;&lt;li&gt;vglm() from the VGAM package can make use of the posnegbinomial family, but it has the same problem as Stata's ztnb command (see below) in that I can't refit the models using a non-independent correlation structure.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Stata&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;If the data wasn't longitudinal, I could just use the Stata packages ztnb to run my analysis, BUT that command assumes that my observations are independent.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I've also ruled out GLMM for various methodological/philosophical reasons.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For now, I've settled on Stata's xtgee command (yes, I know that xtnbreg also does the same thing) that takes into account both the nonindependent correlation structures and the neg binomial family, but not the zero-truncation. The added benefit of using xtgee is that I can also calculate qic values (using the qic command) to determine the best fitting correlation structures for my response variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If there is a package/command in R or Stata that can take 1) nbinomial family, 2) GEE and 3) zero-truncation into account, I'd be dying to know.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd greatly appreciate any ideas you may have. Thank you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-Casey&lt;/p&gt;&#xA;" OwnerUserId="3309" LastEditorUserId="3309" LastEditorDisplayName="" LastEditDate="2011-02-19T03:52:51.513" LastActivityDate="2011-05-20T14:15:43.860" Title="R/Stata package for zero-truncated negative binomial GEE?" Tags="&lt;r&gt;&lt;stata&gt;&lt;count-data&gt;&lt;longitudinal&gt;&lt;truncation&gt;" AnswerCount="3" FavoriteCount="1" />
  <row Id="7386" PostTypeId="6" CreationDate="2011-02-18T22:29:42.007" Score="0" ViewCount="" Body="&lt;p&gt;I am reluctant to make this nomination because I have been happy with the moderators. I would be delighted to see them continue in their roles.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, to date only two people have entered nominations.  (Is everyone else waiting until just before the deadline?)  As you might guess from my activity here, I value this forum and hope to see it attract many more participants. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As part of this self-nomination process, we're supposed to say a little about our qualifications.  The statistics about my participation here are clear enough; there's no need to dwell on that.  I have successfully nurtured technical online communities (via listservers--remember them?--and a Web magazine) and greatly enjoyed how they fostered collegial, productive interchanges.   I have long believed strongly in contributing original content to the Web (rather than just copying bits and pieces of other stuff) and in the power of communities of collaborators.  This site combines both those tenets, in a good way.  Let's all keep contributing as much as we can to keep it growing and successful.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-02-18T22:29:42.007" LastActivityDate="2011-02-18T22:29:42.007" CommunityOwnedDate="2011-02-18T22:29:42.007" CommentCount="3" />
  <row Id="7387" PostTypeId="2" ParentId="7202" CreationDate="2011-02-18T22:55:09.213" Score="1" ViewCount="" Body="&lt;p&gt;Mixed models are usually used to take account of the correlation structure likely with a model like this.  Look up Analyze&gt;Mixed Models (MIXED) or the newer Mixed Models&gt;Generalized Linear if you have the latest version.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;HTH,&#xA;Jon Peck&lt;/p&gt;&#xA;" OwnerUserId="3312" LastActivityDate="2011-02-18T22:55:09.213" />
  <row Id="7388" PostTypeId="2" ParentId="6955" CreationDate="2011-02-18T22:56:20.907" Score="0" ViewCount="" Body="&lt;p&gt;Please advise if a solution to this is known. I have the same problem even though I am admin on my computer, and it only recently started happening.  I did a spring cleaning with various anti-spyware/malware/AV/Registry repair software is the culprit, but removing them is not helpful. I need a specific solution, please advise. &#xA;matthew.markert@gmail.com&lt;/p&gt;&#xA;" OwnerUserId="3313" LastActivityDate="2011-02-18T22:56:20.907" CommentCount="1" />
  <row Id="7389" PostTypeId="1" CreationDate="2011-02-18T23:34:23.113" Score="4" ViewCount="233" Body="&lt;p&gt;My question deals with how to be able to assert that an &quot;improved&quot;&#xA;evolutionary algorithm is indeed improved (at least from a statistic&#xA;point of view) and not just random luck (a concern given the&#xA;stochastic nature of these algorithms).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's assume I am dealing with a standard GA (before) and an &quot;improved&quot;&#xA;GA (after). And I have a suite of 8 test problems.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I run both both of these algorithms repeatedly, for instance 10 times(?)&#xA;through each of the 8 test problems and and record how many&#xA;generations it took to come up with the solution. I would start out&#xA;with the same initial random population (using same seed).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would I use a paired t-test for means to verify that any difference&#xA;(hopefully an improvement) between the averages for each test question&#xA;would be statistically significant? Should I run these algorithms more&#xA;than 10 times for each test/pair?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any pitfalls I should be aware of? I assume I could use this approach&#xA;for any (evolutionary) algorithm comparison.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or am I really on the wrong track here? I am basically looking for a &#xA;way to compare two implementations of an evolutionary algorithm and&#xA;report on how well one might work compared to the other.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="3311" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-02-20T23:52:35.640" LastActivityDate="2011-03-03T08:31:27.563" Title="How to check if modified genetic algorithm is significantly better than the original?" Tags="&lt;t-test&gt;&lt;genetic-algorithms&gt;&lt;pairwise-comparison&gt;" AnswerCount="3" CommentCount="6" FavoriteCount="1" />
  <row Id="7391" PostTypeId="1" CreationDate="2011-02-19T01:57:52.283" Score="3" ViewCount="94" Body="&lt;p&gt;I have seen asserted that the problem of computing the null distribution of Kolmogorov's $D_n^+$ statistic for a finite sample size maps onto the problem of computing the number of lattice paths that stay below the diagonal, and thus can be solved by the &lt;a href=&quot;http://en.wikipedia.org/wiki/Bertrand%27s_ballot_theorem&quot; rel=&quot;nofollow&quot;&gt;ballot theorem&lt;/a&gt;. I am familiar with lattice paths and the ballot problem. I am also familiar the expression of the distribution of $D_n^+$ as a series of integrals. But I don't see how one problem maps onto the other. Can someone explain or point me to an ariticle or book that does?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also see the claim the the null distribution of the Kolmogorov-Smirnov $D_n = \max(D_n^+,D_n^-)$ maps onto another lattice path problem that could be solved by a &quot;two-sided ballot theorem&quot;. I don't know what a &quot;two-sided&quot; version of the ballot problem would be. Again, can someone explain or point me to an explanation?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, is there a general framework around all of this? Can the Kuiper statistic be mapped to yet another lattice path problem? The two-sample KS test? The AD statistic?&lt;/p&gt;&#xA;" OwnerUserId="3315" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-02-19T11:37:22.803" LastActivityDate="2011-02-19T11:37:22.803" Title="Kolmogorov-Smirnov and lattice paths" Tags="&lt;kolmogorov-smirnov&gt;&lt;lattice-paths&gt;" CommentCount="1" />
  <row Id="7392" PostTypeId="2" ParentId="7385" CreationDate="2011-02-19T10:01:05.533" Score="5" ViewCount="" Body="&lt;p&gt;Hmm, good first question! I don't know of a package that meets your precise requirements. I think Stata's &lt;a href=&quot;http://www.stata.com/help.cgi?xtgee&quot; rel=&quot;nofollow&quot;&gt;xtgee&lt;/a&gt; is a good choice if you also specify the &lt;code&gt;vce(robust)&lt;/code&gt; option to give Huber-White standard errors, or &lt;code&gt;vce(bootstrap)&lt;/code&gt; if that's practical. Either of these options will ensure the standard errors are consistently estimated despite the model misspecification that you'll have by ignoring the zero truncation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That leaves the question of what effect ignoring the zero truncation will have on the point estimate(s) of interest to you. It's worth a quick search to see if there is relevant literature on this in general, i.e. not necessarily in a GEE context -- i would have thought you can pretty safely assume any such results will be relevant in the GEE case too. If you can't find anything, you could always simulate data with zero-truncation and known effect estimates and assess the bias by simulation.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-02-19T10:01:05.533" CommentCount="3" />
  <row Id="7393" PostTypeId="2" ParentId="7389" CreationDate="2011-02-19T10:03:01.937" Score="3" ViewCount="" Body="&lt;p&gt;It might not be what you want to hear, but from what I've seen the new algorithm is just compared to the old one on benchmark functions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;E.g. as done here:  &lt;a href=&quot;http://www.idsia.ch/~tom/publications/enes.pdf&quot; rel=&quot;nofollow&quot;&gt;Efﬁcient Natural Evolution Strategies, (Schaul, Sun Yi, Wierstra, Schmidhuber)&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2860" LastEditorUserId="2860" LastEditorDisplayName="" LastEditDate="2011-02-19T13:02:11.847" LastActivityDate="2011-02-19T13:02:11.847" CommentCount="1" />
  <row Id="7394" PostTypeId="1" AcceptedAnswerId="7395" CreationDate="2011-02-19T18:41:52.773" Score="4" ViewCount="275" Body="&lt;p&gt;I have searched a lot, and I can only find tables that show critical values up to n=30. Can someone provide, or point me to, a simple method of estimating this value for different $\alpha$?&lt;/p&gt;&#xA;" OwnerUserId="977" LastActivityDate="2011-02-19T19:16:55.457" Title="How to estimate a critical value of Spearman's correlation for n=100?" Tags="&lt;hypothesis-testing&gt;&lt;correlation&gt;&lt;spearman&gt;" AnswerCount="2" />
  <row Id="7395" PostTypeId="2" ParentId="7394" CreationDate="2011-02-19T19:08:00.543" Score="2" ViewCount="" Body="&lt;p&gt;For values over thirty the approximation (for a two-tailed test) is &#xA;$$\frac{\Phi^{-1}\left(1-\tfrac{\alpha}{2}\right)}{\sqrt{n-1}}$$&#xA;so for example with $\alpha = 0.05$ and $n=100$ the numerator is about 1.96 and the denominator about 9.95, giving a critical value of about 0.197.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This comes from $\rho$ having approximately a normal distribution for large $n$, with mean $0$ and variance $1/(n − 1)$, assuming independence of the observations.&lt;/p&gt;&#xA;" OwnerUserId="2958" LastEditorUserId="2958" LastEditorDisplayName="" LastEditDate="2011-02-19T19:16:17.797" LastActivityDate="2011-02-19T19:16:17.797" CommentCount="2" />
  <row Id="7396" PostTypeId="2" ParentId="7394" CreationDate="2011-02-19T19:16:55.457" Score="6" ViewCount="" Body="&lt;p&gt;See Wikipedia: &lt;a href=&quot;http://en.wikipedia.org/wiki/Spearman%27s_rank_correlation_coefficient#Determining_significance&quot; rel=&quot;nofollow&quot;&gt;Spearman's rank correlation coefficient#Determining significance&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;One can test for significance using&#xA;$$t = r \sqrt{\frac{n-2}{1-r^2}},$$&#xA;which is distributed approximately as Student's $t$ distribution with $n − 2$ degrees of freedom under the null hypothesis.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here $r$ is the sample estimate of Spearman's rank correlation coefficient. The reason critical values often aren't tabulated for $n &amp;gt; 30$ is that this approximation gets better as $n$ gets larger, and is very good for $n &amp;gt; 30$. The Stata statistical software package uses this formula to calculate $p$-values for &lt;em&gt;all&lt;/em&gt; values of $n$.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-02-19T19:16:55.457" />
  <row Id="7397" PostTypeId="2" ParentId="363" CreationDate="2011-02-19T19:25:19.687" Score="0" ViewCount="" Body="&lt;ol&gt;&#xA;&lt;li&gt;Michael Oakes' &lt;em&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0471104434&quot; rel=&quot;nofollow&quot;&gt;Statistical Inference:  A Commentary for the Social and Behavioral Sciences&lt;/a&gt;&lt;/em&gt;.&lt;/li&gt;&#xA;&lt;li&gt;Elazar Pedhazur's &lt;em&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0030728312&quot; rel=&quot;nofollow&quot;&gt;Multiple Regression in Behavioral Research&lt;/a&gt;&lt;/em&gt;.  If you can stand the immense detail and the self-important tone.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In case you're interested, I've reviewed both on Amazon and at &lt;a href=&quot;http://www.integrativestatistics.com/favorites.htm&quot; rel=&quot;nofollow&quot;&gt;http://www.integrativestatistics.com/favorites.htm&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2669" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2011-02-20T02:37:07.083" LastActivityDate="2011-02-20T02:37:07.083" CommunityOwnedDate="2011-02-19T19:25:19.687" />
  <row Id="7398" PostTypeId="2" ParentId="363" CreationDate="2011-02-19T19:47:16.733" Score="1" ViewCount="" Body="&lt;p&gt;Rice: &lt;a href=&quot;http://goo.gl/wKbcW&quot; rel=&quot;nofollow&quot;&gt;Mathematical Statistics and Data Analysis&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="609" LastActivityDate="2011-02-19T19:47:16.733" CommunityOwnedDate="2011-02-19T19:47:16.733" />
  <row Id="7399" PostTypeId="6" CreationDate="2011-02-19T19:49:46.760" Score="0" ViewCount="" Body="&lt;p&gt;I am nominating myself in part because of friendly pressure and because election is really election when there are more candidates than places to be filled.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I came to this site nearly three months ago and became instantly hooked. Moderating would not take a lot out of me, since  I am already visiting the site daily, reading all the questions, trying to get clarifications, fixing formatting  and of course answering the questions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think that current moderators do wonderful job and if I will be their replacement I intend to continue in the same spirit. &lt;/p&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-02-19T19:49:46.760" LastActivityDate="2011-02-19T19:49:46.760" CommunityOwnedDate="2011-02-19T19:49:46.760" CommentCount="1" />
  <row Id="7400" PostTypeId="1" AcceptedAnswerId="7405" CreationDate="2011-02-19T18:52:26.557" Score="3" ViewCount="307" Body="&lt;p&gt;Given two histograms, how do we assess whether they are similar or not?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it sufficient to simply look at the two histograms?&#xA;The simple one to one mapping has the problem that if a histogram is slightly different and slightly shifted then we'll not get the desired result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestions?&lt;/p&gt;&#xA;" OwnerUserId="3325" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-02-21T06:40:35.937" LastActivityDate="2011-02-21T06:40:35.937" Title="How to assess the similarity of two histograms?" Tags="&lt;histogram&gt;&lt;image-processing&gt;" AnswerCount="3" CommentCount="1" FavoriteCount="1" />
  <row Id="7401" PostTypeId="2" ParentId="7400" CreationDate="2011-02-19T19:22:04.820" Score="5" ViewCount="" Body="&lt;p&gt;You're looking for the &lt;a href=&quot;http://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test&quot; rel=&quot;nofollow&quot;&gt;Kolmogorov-Smirnov test&lt;/a&gt;. Don't forget to divide the bar heights by the sum of all observations of each histogram.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that the KS-test is also reporting a difference if e.g. the means of the distributions are shifted relative to one another. If translation of the histogram along the x-axis is not meaningful in your application, you may want to subtract the mean from each histogram first. &lt;/p&gt;&#xA;" OwnerUserId="198" LastActivityDate="2011-02-19T20:09:43.253" CommentCount="1" />
  <row Id="7402" PostTypeId="1" AcceptedAnswerId="7404" CreationDate="2011-02-19T20:56:08.153" Score="3" ViewCount="342" Body="&lt;p&gt;I know that a Type II error is where H1 is true, but H0 is not rejected. &lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Question&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;How do I calculate the probability of a Type II error involving a normal distribution, where the standard deviation is known?&lt;/p&gt;&#xA;" OwnerUserId="3062" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-02-21T05:55:26.353" LastActivityDate="2011-02-21T10:53:26.320" Title="How do I find the probability of a type II error?" Tags="&lt;probability&gt;&lt;power-analysis&gt;&lt;type-ii-errors&gt;" AnswerCount="3" CommentCount="2" FavoriteCount="3" />
  <row Id="7403" PostTypeId="2" ParentId="7402" CreationDate="2011-02-19T21:06:53.507" Score="0" ViewCount="" Body="&lt;p&gt;Try this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Type_I_and_type_II_errors&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Type_I_and_type_II_errors&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2775" LastActivityDate="2011-02-19T21:06:53.507" />
  <row Id="7404" PostTypeId="2" ParentId="7402" CreationDate="2011-02-19T21:13:06.140" Score="14" ViewCount="" Body="&lt;p&gt;In addition to specifying $\alpha$ (probability of a type I error), you need a fully specified hypothesis pair, i.e., $\mu_{0}$, $\mu_{1}$ and $\sigma$ need to be known. $\beta$ (probability of type II error) is $1 - \textrm{power}$. I assume a one-sided $H_{1}: \mu_{1} &amp;gt; \mu_{0}$. In R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; sigma &amp;lt;- 15    # theoretical standard deviation&#xA;&amp;gt; mu0   &amp;lt;- 100   # expected value under H0&#xA;&amp;gt; mu1   &amp;lt;- 130   # expected value under H1&#xA;&amp;gt; alpha &amp;lt;- 0.05  # probability of type I error&#xA;&#xA;# critical value for a level alpha test&#xA;&amp;gt; crit &amp;lt;- qnorm(1-alpha, mu0, sigma)&#xA;&#xA;# power: probability for values &amp;gt; critical value under H1&#xA;&amp;gt; (pow &amp;lt;- pnorm(crit, mu1, sigma))&#xA;[1] 0.36124&#xA;&#xA;# probability for type II error: 1 - power&#xA;&amp;gt; (beta &amp;lt;- 1-pow)&#xA;[1] 0.63876&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Edit: visualization&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/rxoDS.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;xLims &amp;lt;- c(50, 180)&#xA;left  &amp;lt;- seq(xLims[1],   crit, length.out=100)&#xA;right &amp;lt;- seq(crit, xLims[2],   length.out=100)&#xA;yH0r  &amp;lt;- dnorm(right, mu0, sigma)&#xA;yH1l  &amp;lt;- dnorm(left,  mu1, sigma)&#xA;yH1r  &amp;lt;- dnorm(right, mu1, sigma)&#xA;&#xA;curve(dnorm(x, mu0, sigma), xlim=xLims, lwd=2, col=&quot;red&quot;, xlab=&quot;x&quot;, ylab=&quot;density&quot;,&#xA;      main=&quot;Normal distribution under H0 and H1&quot;, ylim=c(0, 0.03), xaxs=&quot;i&quot;)&#xA;curve(dnorm(x, mu1, sigma), lwd=2, col=&quot;blue&quot;, add=TRUE)&#xA;polygon(c(right, rev(right)), c(yH0r, numeric(length(right))), border=NA,&#xA;        col=rgb(1, 0.3, 0.3, 0.6))&#xA;polygon(c(left,  rev(left)),  c(yH1l, numeric(length(left))),  border=NA,&#xA;        col=rgb(0.3, 0.3, 1, 0.6))&#xA;polygon(c(right, rev(right)), c(yH1r, numeric(length(right))), border=NA,&#xA;        density=5, lty=2, lwd=2, angle=45, col=&quot;darkgray&quot;)&#xA;abline(v=crit, lty=1, lwd=3, col=&quot;red&quot;)&#xA;text(crit+1,  0.03,  adj=0, label=&quot;critical value&quot;)&#xA;text(mu0-10,  0.025, adj=1, label=&quot;distribution under H0&quot;)&#xA;text(mu1+10,  0.025, adj=0, label=&quot;distribution under H1&quot;)&#xA;text(crit+8,  0.01,  adj=0, label=&quot;power&quot;, cex=1.3)&#xA;text(crit-12, 0.004,  expression(beta),  cex=1.3)&#xA;text(crit+5,  0.0015, expression(alpha), cex=1.3)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1909" LastEditorUserId="1909" LastEditorDisplayName="" LastEditDate="2011-02-21T10:53:26.320" LastActivityDate="2011-02-21T10:53:26.320" />
  <row Id="7405" PostTypeId="2" ParentId="7400" CreationDate="2011-02-19T22:11:05.970" Score="1" ViewCount="" Body="&lt;p&gt;A recent paper that may be worth reading is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://dx.doi.org/10.1016/j.jcp.2005.06.012&quot; rel=&quot;nofollow&quot;&gt;Cao, Y. Petzold, L.&lt;/a&gt; Accuracy limitations and the measurement of errors in the stochastic simulation of chemically reacting systems, 2006.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Although this paper's focus is on comparing stochastic simulation algorithms, essentially the main idea is how to compare two histogram. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can access the &lt;a href=&quot;http://engineering.ucsb.edu/~cse/Files/distributiondistance042.pdf&quot; rel=&quot;nofollow&quot;&gt;pdf&lt;/a&gt; from the author's webpage.&lt;/p&gt;&#xA;" OwnerUserId="8" LastActivityDate="2011-02-19T22:11:05.970" CommentCount="1" />
  <row Id="7406" PostTypeId="2" ParentId="7211" CreationDate="2011-02-19T22:26:45.590" Score="0" ViewCount="" Body="&lt;p&gt;Andrew McCallum (UMass) has a few NLP related software projects available on his &lt;a href=&quot;http://www.cs.umass.edu/~mccallum/code.html&quot; rel=&quot;nofollow&quot;&gt;webpage&lt;/a&gt;. These are all in Java (I think) with source code available.&lt;/p&gt;&#xA;" OwnerUserId="1913" LastActivityDate="2011-02-19T22:26:45.590" />
  <row Id="7407" PostTypeId="1" CreationDate="2011-02-19T22:41:23.000" Score="2" ViewCount="77" Body="&lt;p&gt;Hi, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I recently stumbled upon the concept of &lt;a href=&quot;http://www.google.com/search?q=%22sample%20complexity%22&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;sample complexity&lt;/strong&gt;&lt;/a&gt;, and was wondering if there are any texts, papers or tutorials that provide:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;A good introduction to the concept&lt;/li&gt;&#xA;&lt;li&gt;An analysis of the sample complexity of different classification methods or kernel methods.&lt;/li&gt;&#xA;&lt;li&gt;Advice or information on how to measure it in practice.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Any help with the topic would be greatly appreciated.&lt;/p&gt;&#xA;" OwnerUserId="2798" LastEditorUserId="2798" LastEditorDisplayName="" LastEditDate="2011-02-20T19:06:47.563" LastActivityDate="2011-02-20T19:06:47.563" Title="Measuring and analyzing sample complexity" Tags="&lt;machine-learning&gt;" FavoriteCount="1" />
  <row Id="7408" PostTypeId="1" AcceptedAnswerId="7409" CreationDate="2011-02-19T23:46:53.130" Score="3" ViewCount="105" Body="&lt;p&gt;Here's a real basic question. I'm trying to teach myself a bit of stats with Verzani's Using R for Introductory Statistics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In question 5.13 he asks: A sample of 100 people is drawn from a population of 600,000. If it is known that 40% of the population has a specific attribute, what is the probability that 35 or fewer in the sample have that attribute.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, I guess you're supposed to reason that the population is sufficiently large that assuming independent Bernoulli trials is close enough. Then, you get your answer like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; pbinom(35,100,0.4)&#xA;&#xA;[1] 0.1794694&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My question is this. How would you go about answering a question like that without assuming independence, say if the population was smaller.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm sure it'll become obvious after I read more. Just trying to make sure I'm not missing something. Sorry for the intro level question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="3317" LastActivityDate="2011-02-20T00:44:16.023" Title="Sampling from a fixed population" Tags="&lt;homework&gt;&lt;sampling&gt;" AnswerCount="1" />
  <row Id="7409" PostTypeId="2" ParentId="7408" CreationDate="2011-02-20T00:05:12.063" Score="5" ViewCount="" Body="&lt;p&gt;When sampling without replacement, the distribution is a hypergeometric one. The problem is usually presented as follows: in an urn with $n$ (600.000) marbles, $m$ (40% = 240.000) are red, $n-m$ (60% = 360.000) are black. What is the probability of picking $r$ (35) red marbles in a sample of $k$ (100) marbles? The error by assuming sampling with replacement is really small when $n$ is very large, such as in your case (thanks Henry!).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\begin{array}{r|ll|l}&#xD;&#xA;~     &amp;amp; y_{1} &amp;amp; y_{2} &amp;amp; \Sigma \\\hline&#xD;&#xA;x_{1} &amp;amp; r     &amp;amp; m-r   &amp;amp; m   \\&#xD;&#xA;x_{2} &amp;amp; k-r   &amp;amp; ~     &amp;amp; n-m \\\hline&#xD;&#xA;\Sigma   &amp;amp; k     &amp;amp; n-k   &amp;amp; n&#xD;&#xA;\end{array}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In R: &lt;code&gt;dhyper(r, m, n-m, k)&lt;/code&gt;. For the total probability of $0, \ldots, r$ marbles: &lt;code&gt;phyper(r, m, n-m, k)&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; phyper(35, 240000, 360000, 100)&#xA;[1] 0.1794489&#xA;&#xA;# check&#xA;&amp;gt; sum(dhyper(0:35, 240000, 360000, 100))&#xA;[1] 0.1794489&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Google &quot;finite population correction&quot; for correcting the error when computing sample mean and variance with small populations.&lt;/p&gt;&#xA;" OwnerUserId="1909" LastEditorUserId="1909" LastEditorDisplayName="" LastEditDate="2011-02-20T00:44:16.023" LastActivityDate="2011-02-20T00:44:16.023" CommentCount="3" />
  <row Id="7410" PostTypeId="2" ParentId="7400" CreationDate="2011-02-20T06:29:59.583" Score="6" ViewCount="" Body="&lt;p&gt;The standard answer to this question is the &lt;a href=&quot;http://www.itl.nist.gov/div898/handbook/eda/section3/eda35f.htm&quot; rel=&quot;nofollow&quot;&gt;chi-squared test&lt;/a&gt;. The KS test is for unbinned data, not binned data. (If you have the unbinned data, then by all means use a KS-style test, but if you only have the histogram, the KS test is not appropriate.)&lt;/p&gt;&#xA;" OwnerUserId="3315" LastActivityDate="2011-02-20T06:29:59.583" CommentCount="1" />
  <row Id="7411" PostTypeId="2" ParentId="3" CreationDate="2011-02-20T07:03:34.513" Score="1" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://www.meta-numerics.net&quot; rel=&quot;nofollow&quot;&gt;Meta.Numerics&lt;/a&gt; is a .NET library with good support for statistical analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unlike R (an S clone) and Octave (a Matlab clone), it does not have a &quot;front end&quot;. It is more like GSL, in that it is a library that you link to when you are writing your own application that needs to do statistical analysis. C# and Visual Basic are more common programming languages than C/C++ for line-of-business apps, and Meta.Numerics has more extensive support for statistical constructs and tests than GSL.&lt;/p&gt;&#xA;" OwnerUserId="3315" LastActivityDate="2011-02-20T07:03:34.513" CommunityOwnedDate="2011-02-20T07:03:34.513" />
  <row Id="7412" PostTypeId="1" CreationDate="2011-02-20T08:03:25.657" Score="1" ViewCount="62" Body="&lt;p&gt;I want to model how traffic will flow on real networks (not just the internet, also, say, Intel's internal LAN).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a place I can get real network topologies data I can use?&lt;/p&gt;&#xA;" OwnerUserId="3328" LastActivityDate="2011-02-20T08:03:25.657" Title="Where can I get real data of big network topology?" Tags="&lt;networks&gt;&lt;topologies&gt;" CommentCount="2" FavoriteCount="1" />
  <row Id="7413" PostTypeId="2" ParentId="7211" CreationDate="2011-02-20T09:20:16.843" Score="2" ViewCount="" Body="&lt;p&gt;Here are two further integrated projects:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Python &lt;a href=&quot;http://www.nltk.org/&quot; rel=&quot;nofollow&quot;&gt;Natural Language Toolkit&lt;/a&gt; (easy installation, good documentation)&lt;/li&gt;&#xA;&lt;li&gt;Java &lt;a href=&quot;http://mallet.cs.umass.edu/&quot; rel=&quot;nofollow&quot;&gt;MALLET&lt;/a&gt; (no experience with it, but looks promising; included in the link given by @Nick)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Both are open-source software.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-02-20T09:20:16.843" />
  <row Id="7414" PostTypeId="2" ParentId="6234" CreationDate="2011-02-20T09:40:05.920" Score="5" ViewCount="" Body="&lt;p&gt;This won't compete with @Shane's answer because circular displays are really well suited for displaying complex relationships with high-dimensional datasets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For Venn diagrams, I've been using the &lt;a href=&quot;http://cran.r-project.org/web/packages/venneuler/index.html&quot; rel=&quot;nofollow&quot;&gt;venneuler&lt;/a&gt; R package. It has a simple yet intuitive interface and produce nifty diagrams with transparency, compared to the basic &lt;code&gt;venn()&lt;/code&gt; function &lt;a href=&quot;http://www.jstatsoft.org/v11/c01/paper&quot; rel=&quot;nofollow&quot;&gt;described&lt;/a&gt; in the &lt;em&gt;Journal of Statistical Software&lt;/em&gt;. It does not handle more than 3 categories, though. Another project is &lt;a href=&quot;http://cran.r-project.org/web/packages/eVenn/index.html&quot; rel=&quot;nofollow&quot;&gt;eVenn&lt;/a&gt; and it deals with $K=4$ sets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More recently, I came across a new package that deal with higher-order relation sets, and probably allow to reproduce some of the Venn diagrams shown on Wikipedia or on this webpage, &lt;a href=&quot;http://www.combinatorics.org/Surveys/ds5/VennWhatEJC.html&quot; rel=&quot;nofollow&quot;&gt;What is a Venn Diagram?&lt;/a&gt;, but it is also limited to $K=4$ sets. It is called VennDiagram, but see the reference paper: &lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pubmed/21269502&quot; rel=&quot;nofollow&quot;&gt;VennDiagram: a package for the generation of highly-customizable Venn and Euler diagrams in R&lt;/a&gt; (Chen and Boutros, &lt;em&gt;BMC Bioinformatics&lt;/em&gt; 2011, 12:35).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For further reference, you might be interested in&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Kestler et al., &lt;a href=&quot;http://bioinformatics.oxfordjournals.org/content/21/8/1592.full&quot; rel=&quot;nofollow&quot;&gt;Generalized Venn&#xA;  diagrams: a new method of visualizing&#xA;  complex genetic set relations&lt;/a&gt;,&#xA;  Bioinformatics, 21(8), 1592-1595&#xA;  (2004).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Venn diagrams have their limitations, though. In this respect, I like the approach taken by Robert Kosara in &lt;a href=&quot;http://eagereyes.org/blog/2008/sightings-a-vennerable-challenge.html&quot; rel=&quot;nofollow&quot;&gt;Sightings: A Vennerable Challenge&lt;/a&gt;, or with &lt;a href=&quot;http://kosara.net/papers/2010/Kosara_BeautifulVis_2010.pdf&quot; rel=&quot;nofollow&quot;&gt;Parallel Sets&lt;/a&gt; (but see also &lt;a href=&quot;http://www.stat.columbia.edu/~cook/movabletype/archives/2007/10/venn_diagram_ch.html&quot; rel=&quot;nofollow&quot;&gt;this discussion&lt;/a&gt; on Andrew Gelman weblog).&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-02-20T09:48:31.850" LastActivityDate="2011-02-20T09:48:31.850" CommentCount="1" />
  <row Id="7415" PostTypeId="1" AcceptedAnswerId="7424" CreationDate="2011-02-20T09:54:18.693" Score="3" ViewCount="222" Body="&lt;p&gt;What are the statistical techniques to create a sample set, which is representative of the entire population (with a known confidence level)? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also,&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;How to validate, if the sample fits the overall dataset?&lt;/li&gt;&#xA;&lt;li&gt;Is it possible, without parsing the entire dataset (which could be billions of records)?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="3292" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-02-20T09:56:10.747" LastActivityDate="2011-02-20T19:28:51.147" Title="How to make representative sample set from a large overall dataset?" Tags="&lt;sampling&gt;&lt;sample-size&gt;&lt;validation&gt;" AnswerCount="2" FavoriteCount="2" />
  <row Id="7416" PostTypeId="2" ParentId="7048" CreationDate="2011-02-20T10:46:02.080" Score="3" ViewCount="" Body="&lt;p&gt;In addition to linking quantitative or qualitative data to spatial patterns, as illustrated by @whuber, I would like to mention the use of EDA, with brushing and the various of linking plots together, for &lt;em&gt;longitudinal&lt;/em&gt; and &lt;em&gt;high-dimensional&lt;/em&gt; data analysis. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Both are discussed in the excellent book, &lt;a href=&quot;http://www.ggobi.org/book/&quot; rel=&quot;nofollow&quot;&gt;&lt;em&gt;Interactive and Dynamic Graphics for Data Analysis With R and GGobi&lt;/em&gt;&lt;/a&gt;, by Dianne Cook and Deborah F. Swayne (Springer UseR!, 2007), that you surely know. The authors have a nice discussion on EDA in Chapter 1, justifying the need for EDA to &quot;force the unexpected upon us&quot;, quoting John Tukey (p. 13): The use of interactive and dynamic displays is neither &lt;a href=&quot;http://en.wikipedia.org/wiki/Data_dredging&quot; rel=&quot;nofollow&quot;&gt;data snooping&lt;/a&gt;, nor preliminary data inspection (e.g., purely graphical summaries of the data), but it is merely seen as an interactive investigation of the data which might precede or complement pure hypothesis-based statistical modeling. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using GGobi together with its R interface (&lt;a href=&quot;http://cran.r-project.org/web/packages/rggobi/index.html&quot; rel=&quot;nofollow&quot;&gt;rggobi&lt;/a&gt;) also solves the problem of how to generate static graphics for intermediate report or final publication, even with &lt;a href=&quot;http://en.wikipedia.org/wiki/Projection_pursuit&quot; rel=&quot;nofollow&quot;&gt;Projection Pursuit&lt;/a&gt; (pp. 26-34), thanks to the &lt;a href=&quot;http://cran.r-project.org/web/packages/DescribeDisplay/index.html&quot; rel=&quot;nofollow&quot;&gt;DescribeDisplay&lt;/a&gt; or &lt;a href=&quot;http://had.co.nz/ggplot2/&quot; rel=&quot;nofollow&quot;&gt;ggplot2&lt;/a&gt; packages.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the same line, &lt;a href=&quot;http://www.datavis.ca/&quot; rel=&quot;nofollow&quot;&gt;Michael Friendly&lt;/a&gt; has long advocated the use of data visualization in Categorical Data Analysis, which has been largely exemplified in the vcd package, but also in the more recent &lt;a href=&quot;http://cran.r-project.org/web/packages/vcdExtra/index.html&quot; rel=&quot;nofollow&quot;&gt;vcdExtra&lt;/a&gt; package (including dynamic viz. through the &lt;a href=&quot;http://cran.r-project.org/web/packages/rgl/index.html&quot; rel=&quot;nofollow&quot;&gt;rgl&lt;/a&gt; package), which acts as a glue between the &lt;a href=&quot;http://cran.r-project.org/web/packages/vcd/index.html&quot; rel=&quot;nofollow&quot;&gt;vcd&lt;/a&gt; and &lt;a href=&quot;http://cran.r-project.org/web/packages/gnm/index.html&quot; rel=&quot;nofollow&quot;&gt;gnm&lt;/a&gt; packages for extending log-linear models. He recently gave a nice summary of that work during the &lt;a href=&quot;http://carme2011.agrocampus-ouest.fr/&quot; rel=&quot;nofollow&quot;&gt;6th CARME&lt;/a&gt; conference, &lt;a href=&quot;http://www.datavis.ca/papers/adv-vcd-4up.pdf&quot; rel=&quot;nofollow&quot;&gt;Advances in Visualizing Categorical Data Using the vcd, gnm and vcdExtra Packages in R&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hence, EDA can also be thought of as providing a visual explanation of data (in the sense that it may account for unexpected patterns in the observed data), prior to a purely statistical modeling approach, &lt;em&gt;or&lt;/em&gt; in parallel to it. That is, EDA not only provides useful ways for studying the internal structure of the data at hand, but it may also help to refine and/or summarize statistical models applied on it. It is in essence what &lt;a href=&quot;http://en.wikipedia.org/wiki/Biplot&quot; rel=&quot;nofollow&quot;&gt;biplots&lt;/a&gt; allow to do, for example. Although they are not multidimensional analysis techniques &lt;em&gt;per se&lt;/em&gt;, they are tools for visualizing results from multidimensional analysis (by giving an &lt;em&gt;approximation&lt;/em&gt; of the relationships when considering all individuals together, or all variables together, or both). Factor scores can be used in subsequent modeling in place of the original metric to either reduce the dimensionality or to provide intermediate levels of representation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Sidenote&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At risk of being old-fashionned, I'm still using &lt;code&gt;xlispstat&lt;/code&gt; (&lt;a href=&quot;http://www.stat.uiowa.edu/~luke/&quot; rel=&quot;nofollow&quot;&gt;Luke Tierney&lt;/a&gt;) from time to time. It has simple yet effective functionalities for interactive displays, currently not available in base R graphics. I'm not aware of similar capabilities in Clojure+Incanter (+Processing).&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-02-20T10:46:02.080" />
  <row Id="7417" PostTypeId="2" ParentId="7389" CreationDate="2011-02-20T11:18:24.227" Score="4" ViewCount="" Body="&lt;p&gt;I used paired t-test to compare my algorithm to GA, although I had about 200 test cases. You can use a non-parametric alternative such as Wilcoxon Ranks Test.  Regardless of what you use to test the statistical significance, bare in mind the &quot;real-life&quot; significance. If the performance improvement that your algorithm provides is below measurement limits, or below any practical interest, then event if it statistically significant (i.e. &quot;good&quot; p-value), it doesn't  matter.&lt;/p&gt;&#xA;" OwnerUserId="1496" LastActivityDate="2011-02-20T11:18:24.227" CommentCount="2" />
  <row Id="7418" PostTypeId="2" ParentId="7385" CreationDate="2011-02-20T11:51:29.197" Score="3" ViewCount="" Body="&lt;p&gt;For R two options spring to mind, both of which I am only vaguely familiar at best.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first is the &lt;code&gt;pscl&lt;/code&gt; package, which can fit zero &lt;strike&gt;truncated&lt;/strike&gt; inflated and hurdle models in a very nice, flexible manner. The &lt;code&gt;pscl&lt;/code&gt; package suggests the use of the &lt;code&gt;sandwich&lt;/code&gt; package which provides &quot;Model-robust standard error estimators for cross-sectional, time series and longitudinal data&quot;. So you could fit your count model and then use the &lt;code&gt;sandwich&lt;/code&gt; package to estimate an appropriate covariance matrix for the residuals taking into account the longitudinal nature of the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second option might be to look the &lt;code&gt;geepack&lt;/code&gt; package which looks like it can do what you want but only for a negative binomial model with known theta, as it will fit any type of GLM that R's &lt;code&gt;glm()&lt;/code&gt; function can (so use the family function from MASS).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A third option has raised it's head: &lt;code&gt;gamlss&lt;/code&gt; and it's add-on package &lt;code&gt;gamlss.tr&lt;/code&gt;. The latter includes a function &lt;code&gt;gen.trun()&lt;/code&gt; that can turn any of the distributions supported by &lt;code&gt;gamlss()&lt;/code&gt; into a truncated distribution in a flexible way - you can specify left truncated at 0 negative binomial distribution for example. &lt;code&gt;gamlss()&lt;/code&gt; itself includes support for random effects which should take care of the longitudinal nature of the data. It isn't immediately clear however if you have to use at least one smooth function of a covariate in the model or can just model everything as linear functions like in a GLM.&lt;/p&gt;&#xA;" OwnerUserId="1390" LastEditorUserId="1390" LastEditorDisplayName="" LastEditDate="2011-02-21T10:26:55.850" LastActivityDate="2011-02-21T10:26:55.850" CommentCount="5" />
  <row Id="7419" PostTypeId="1" AcceptedAnswerId="7674" CreationDate="2011-02-20T12:13:24.220" Score="6" ViewCount="197" Body="&lt;p&gt;I have been doing some casual internet research on biclusters.  (I have read the Wiki article several times.)  So far, it seems as if there are few definitions or standard terminology.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;I was wondering if there were any standard papers or books that anybody who is interested in algorithms for finding biclusters should read.  &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Is it possible to say what is the state of the art in the field?  I was intrigued by the notion of finding biclusters using genetic algorithms, so I would appreciate comments on that approach in particular in the context of other approaches.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Usually in clustering, the goal is to partition the data-set into groups where each element is in some group.  Do bicluster algorithms also seek to put all elements in a particular group?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="847" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-02-21T06:59:37.047" LastActivityDate="2011-02-27T11:51:49.200" Title="Getting started with biclustering" Tags="&lt;clustering&gt;" AnswerCount="1" />
  <row Id="7420" PostTypeId="2" ParentId="7415" CreationDate="2011-02-20T16:16:58.580" Score="1" ViewCount="" Body="&lt;p&gt;On your second question first, you might ask, &quot;how was the data entered?&quot; If you think that the data was entered in a relatively arbitrary fashion (i.e., independent of any observable or unobservable characteristics of your observations that might influence your ultimate analysis using the data), then you might consider the first 5 million, say, or however many you're comfortable working with, as representative of the full sample and select randomly from this group to create a sample that you can work with.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To compare two empirical distributions, you can use qq-plots and the two-sample Kolmogorov–Smirnov non-parametric test for differences in distributions (see, e.g., here: &lt;a href=&quot;http://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test&lt;/a&gt;). In this case, you would test the distribution of each variable in your sample against the distribution of that variable in your &quot;full&quot; data set (again, it could be just 5 million observations from your full sample). The KS test can suffer from low power (i.e., it's hard to reject the null hypothesis of no difference between the groups), but, with that many observations, you should be okay.&lt;/p&gt;&#xA;" OwnerUserId="401" LastEditorUserId="401" LastEditorDisplayName="" LastEditDate="2011-02-20T16:24:28.540" LastActivityDate="2011-02-20T16:24:28.540" />
  <row Id="7422" PostTypeId="1" CreationDate="2011-02-20T16:48:57.387" Score="2" ViewCount="154" Body="&lt;p&gt;I have network traffic data in the following for for each hour of a ten day period as follows in a R dataset. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   Day   Hour         Volume          Category&#xA;    0    00            100            P2P&#xA;    0    00            50             email&#xA;    0    00            200            gaming&#xA;    0    00            200            video&#xA;    0    00            150            web&#xA;    0    00            120            P2P&#xA;    0    00            180            web&#xA;    0    00            80             email&#xA;    ....&#xA;    0    01            150            P2P&#xA;    0    01            200            P2P&#xA;    0    01             50            Web&#xA;    ...&#xA;    ...&#xA;    10   23            100            web&#xA;    10   23            200            email&#xA;    10   23            300            gaming&#xA;    10   23            300            gaming&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As seen there are repetitions of Category within a single hour also. I need to calculate the volatility and the peak hour to average hour ratios of these different application categories.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Volatility&lt;/strong&gt;: Standard deviation of hourly volumes divided by hourly average.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Peak hour to avg. hour ratio&lt;/strong&gt;: Ratio of volume of the maximum hour to the volume of the average hour for that application.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So how do I aggregate and calculate these two statistics for each category? I am new to R and don't have much knowledge of how to aggregate and get the averages as mentioned.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, the final result would look something like this&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Category    Volatility      Peak to Avg. Ratio&#xA;Web            0.55            1.5&#xA;P2P            0.30            2.1&#xA;email          0.6             1.7&#xA;gaming         0.4             2.9&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2101" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-02-21T14:28:56.023" LastActivityDate="2011-02-21T14:28:56.023" Title="Calculating hourly volatility and peak-to-average ratio in R" Tags="&lt;r&gt;&lt;aggregation&gt;" AnswerCount="1" CommentCount="3" FavoriteCount="1" />
  <row Id="7423" PostTypeId="2" ParentId="7422" CreationDate="2011-02-20T18:24:48.317" Score="2" ViewCount="" Body="&lt;p&gt;Check out the &lt;a href=&quot;http://cran.r-project.org/web/packages/plyr/index.html&quot; rel=&quot;nofollow&quot;&gt;plyr package&lt;/a&gt;, which has &lt;a href=&quot;http://had.co.nz/plyr/&quot; rel=&quot;nofollow&quot;&gt;great documentation&lt;/a&gt;. While you could solve your problem with &lt;code&gt;aggregate()&lt;/code&gt; function, I'd argue that learning the plyr family of functions will be worth it in the end.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For your specific problem, this would obtain what you want:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;stats = ddply(&#xA;    .data = my_data&#xA;    , .variables = .( Hour , Category)&#xA;    , .fun = function(x){&#xA;        to_return = data.frame(&#xA;            volatility = sd(x$Volume)/mean(x$Volume)&#xA;            , pa_ratio = max(x$Volume)/mean(x$Volume)&#xA;        )&#xA;        return( to_return )&#xA;    }&#xA;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="364" LastActivityDate="2011-02-20T18:24:48.317" CommentCount="3" />
  <row Id="7424" PostTypeId="2" ParentId="7415" CreationDate="2011-02-20T18:49:31.990" Score="3" ViewCount="" Body="&lt;p&gt;If you don't wish to parse the entire data set then you probably can't use &lt;a href=&quot;http://en.wikipedia.org/wiki/Stratified_sampling&quot; rel=&quot;nofollow&quot;&gt;stratified sampling&lt;/a&gt;, so I'd suggest taking a large &lt;a href=&quot;http://en.wikipedia.org/wiki/Simple_random_sample&quot; rel=&quot;nofollow&quot;&gt;simple random sample&lt;/a&gt;. By taking a &lt;em&gt;random&lt;/em&gt; sample, you ensure that the sample will, on average, be representative of the entire dataset, and standard statistical measures of precision such as standard errors and confidence intervals will tell you how far off the population values your sample estimates are likely to be, so there's no real need to validate that a sample is representative of the population unless you have some concerns that is was truly sampled at random.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How large a simple random sample? Well, the larger the sample, the more precise your estimates will be. As you already have the data, conventional sample size calculations aren't really applicable -- you may as well use as much of your dataset as is practical for computing. Unless you're planning to do some complex analyses that will make computation time an issue, a simple approach would be to make the simple random sample as large as can be analysed on your PC without leading to &lt;a href=&quot;http://en.wikipedia.org/wiki/Paging&quot; rel=&quot;nofollow&quot;&gt;paging&lt;/a&gt; or other memory issues. One rule of thumb to limit the size of your dataset to no more than half your computer's RAM so as to have space to manipulate it and leave space for the OS and maybe a couple of other smaller applications (such as an editor and a web browser). Another limitation is that 32-bit Windows operating systems won't allow the address space for any single application to be larger than $2^{31}$ bytes = 2.1GB, so if you're using 32-bit Windows, 1GB may be a reasonable limit on the size of a dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's then a matter of some simple arithmetic to calculate how many observations you can sample given how many variables you have for each observation and how many bytes each variable takes up.&lt;/p&gt;&#xA;" OwnerUserId="449" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-02-20T19:28:51.147" LastActivityDate="2011-02-20T19:28:51.147" CommentCount="1" />
  <row Id="7425" PostTypeId="2" ParentId="7175" CreationDate="2011-02-20T19:52:06.927" Score="3" ViewCount="" Body="&lt;p&gt;I think the best quality measure for clustering is the cluster assumption, as given by Seeger in &lt;a href=&quot;http://webcache.googleusercontent.com/search?q=cache%3ahttp://people.kyb.tuebingen.mpg.de/seeger/papers/review.pdf&quot; rel=&quot;nofollow&quot;&gt;Learning with labeled and unlabeled data&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;For example, assume X = Rd andthe validity of the “cluster assumption”, namely that two points x, x shouldhave the same label t if there is a path between them in X which passes onlythrough regions of relatively high P(x). &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Yes, this brings the whole idea of centroids and centers down. After all, this are rather arbitrary concepts if you think about the fact that your data might lie within a non-linear submanifold of the space you are actually operating in. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can easily construct a synthetic dataset where mixture models break down. E.g. this one: &lt;img src=&quot;http://i.imgur.com/4BNwC.png&quot; alt=&quot;a circle within a cloud&quot;&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Long story short: I'd measure the quality of a clustering algorithm in a minimax way. The best clustering algorithm is the one which minimizes the maximal distance of a point to its nearest neighbor of the same cluster while it maximizes the minimal distance of a point to its nearest neighbor from a different cluster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might also be interested in &lt;a href=&quot;http://www.icml2010.org/papers/168.pdf&quot; rel=&quot;nofollow&quot;&gt;A Nonparametric Information Theoretic Clustering Algorithm&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2860" LastActivityDate="2011-02-20T19:52:06.927" CommentCount="5" />
  <row Id="7426" PostTypeId="1" AcceptedAnswerId="7428" CreationDate="2011-02-20T21:27:02.557" Score="2" ViewCount="309" Body="&lt;p&gt;How can I conduct an Egger's test using SPSS17? For each study included in the meta-analysis I know effect size and sample size of patients and controls groups.&lt;/p&gt;&#xA;" OwnerUserId="3333" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-02-20T23:42:57.017" LastActivityDate="2011-02-23T04:50:37.073" Title="Egger's test in SPSS" Tags="&lt;spss&gt;&lt;meta-analysis&gt;&lt;funnel-plot&gt;&lt;publication-bias&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="7427" PostTypeId="2" ParentId="7426" CreationDate="2011-02-20T21:53:59.553" Score="2" ViewCount="" Body="&lt;p&gt;I don't use PASW anymore, but implementation of the Egger's test for asymmetry is quite simple. First please look at the Egger's &lt;a href=&quot;http://goo.gl/6gnEj&quot; rel=&quot;nofollow&quot;&gt;paper&lt;/a&gt; where he propose &quot;theory&quot; behind the test.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically you have two variables: (i) normalized effect estimate (your estimate divided by its standard error), and (ii) precision (reciprocal of the standard error of the estimate). Then you should conduct simple linear regression and test for intercept $\beta_0 = 0$.&lt;/p&gt;&#xA;" OwnerUserId="609" LastEditorUserId="609" LastEditorDisplayName="" LastEditDate="2011-02-23T04:50:37.073" LastActivityDate="2011-02-23T04:50:37.073" CommentCount="1" />
  <row Id="7428" PostTypeId="2" ParentId="7426" CreationDate="2011-02-20T22:12:41.893" Score="5" ViewCount="" Body="&lt;p&gt;In order to conduct Egger's regression test you will also need the standard errors ($SE_i$) of your effect sizes ($ES_i$). Then generate the so called standard normal deviate (SND) which is defined as effect size divided by its standard error ($ES_i / SE_i$). Next, generate the precision which is $\frac{1}{SE_i}$. The regression model is: $SND = a + b \cdot precision$ (I know the error term is missing but let's keep it simple). Finally, estimate this regression model (unweighted) in SPSS/PASW (see &lt;a href=&quot;http://www.bmj.com/content/315/7109/629.full&quot; rel=&quot;nofollow&quot;&gt;Egger et al 1997&lt;/a&gt;: &quot;Methods: Measures of funnel plot asymmetry&quot;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The logic of Egger's regression test in explained in another CrossValidated thread: &quot;&lt;a href=&quot;http://stats.stackexchange.com/questions/7040/eggers-linear-regression-method-intercept-in-meta-analysis&quot;&gt;Egger’s linear regression method intercept in meta analysis&lt;/a&gt;&quot;.&lt;/p&gt;&#xA;" OwnerUserId="307" LastActivityDate="2011-02-20T22:12:41.893" CommentCount="18" />
  <row Id="7429" PostTypeId="1" CreationDate="2011-02-21T00:53:30.507" Score="0" ViewCount="123" Body="&lt;p&gt;What should I do if the expected value in a Chi-square goodness-of-fit test is zero?&#xA;I know there's Fisher's test but I have a very large table!&lt;/p&gt;&#xA;" OwnerUserId="3338" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-02-21T08:08:14.360" LastActivityDate="2011-02-21T08:14:45.830" Title="Computing chi-square for large tables with some expected cell counts equal to zero" Tags="&lt;chi-squared&gt;&lt;goodness-of-fit&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="7430" PostTypeId="1" AcceptedAnswerId="7434" CreationDate="2011-02-21T02:44:28.760" Score="3" ViewCount="108" Body="&lt;p&gt;A uniform prior for a scale parameter (like the variance) is uniform on the logarithmic scale. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What functional form does this prior have on the linear scale? And why so?&lt;/p&gt;&#xA;" OwnerUserId="1098" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-02-21T05:59:30.047" LastActivityDate="2011-03-27T09:09:00.470" Title="Creating a uniform prior on the logarithmic scale" Tags="&lt;bayesian&gt;&lt;prior&gt;" AnswerCount="2" />
  <row Id="7431" PostTypeId="2" ParentId="7429" CreationDate="2011-02-21T02:54:03.923" Score="2" ViewCount="" Body="&lt;p&gt;If the expected value of a cell is zero in a goodness of fit test (I'm assuming you really mean goodness of fit, where the fit is to a theoretical distribution, not another observed distribution) then there are two possibilities:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;You also observed this value zero times.  Just discard the zero expected value and try again.  Zero observations is the only value you should get with zero expected value.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;You observed this value more than zero times.  In this case your null hypothesis is obviously wrong, because this value has a nonzero probability.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="1347" LastActivityDate="2011-02-21T02:54:03.923" />
  <row Id="7432" PostTypeId="1" CreationDate="2011-02-21T03:21:45.580" Score="1" ViewCount="216" Body="&lt;p&gt;In SPSS Version 19 there seems to be a new feature called Automatic Linear Modelling.  It creates a 'Model' (which is new to me) and the function seems to combine a number of the functions that is typically required for prediction model development.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The functionality seems incomplete with only a subset of prediction selection techniques and most notable it's missing Backwards step wise.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;QUESTIONS&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Do people see this as good or evil?  &lt;/li&gt;&#xA;&lt;li&gt;And if 'good' then are there ways to decompose what it is doing? &lt;/li&gt;&#xA;&lt;li&gt;Specifically how do I find the regression equation co-efficients when bagging or boosting? &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;To me it seems to hides a lot of steps and I'm not exactly sure how it's creating what it presents.  So any pointers to tutorials or the like (as the SPSS documentation isn't great) is appreciated.&lt;/p&gt;&#xA;" OwnerUserId="3189" LastEditorUserId="3189" LastEditorDisplayName="" LastEditDate="2011-02-21T03:45:19.933" LastActivityDate="2011-02-21T05:38:03.153" Title="Is Automatic Linear Modelling in SPSS a good or bad thing?" Tags="&lt;regression&gt;&lt;modeling&gt;&lt;spss&gt;" AnswerCount="1" />
  <row Id="7433" PostTypeId="2" ParentId="7432" CreationDate="2011-02-21T03:36:42.370" Score="3" ViewCount="" Body="&lt;p&gt;I had a quick look at the &lt;a href=&quot;ftp://public.dhe.ibm.com/common/ssi/ecm/en/ytd03023usen/YTD03023USEN.PDF&quot; rel=&quot;nofollow&quot;&gt;IBM SPSS advertising material&lt;/a&gt;.&#xA;It sounds like it is part of a general move on the part of IBM/SPSS to get involved with predictive analytics.&#xA;Terms like automatic data preparation, boosting, bagging, and automated model selection are popular in data mining and predictive analytics communities.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In that sense you may see similarities with open source tools like &lt;a href=&quot;http://rattle.togaware.com/&quot; rel=&quot;nofollow&quot;&gt;Rattle&lt;/a&gt; and &lt;a href=&quot;http://www.cs.waikato.ac.nz/ml/weka/&quot; rel=&quot;nofollow&quot;&gt;Weka&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might find useful &lt;a href=&quot;http://maths.anu.edu.au/~johnm/dm/dmpaper.html&quot; rel=&quot;nofollow&quot;&gt;this article by John Maindonald introducing data mining&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In summary, if you have some combination of the following factors, then such tools may interest you:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;interested in building predictive models (as opposed to testing apriori hypotheses)&lt;/li&gt;&#xA;&lt;li&gt;you have lots of data&lt;/li&gt;&#xA;&lt;li&gt;you want some hand holding on the steps of data analysis&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-02-21T05:38:03.153" LastActivityDate="2011-02-21T05:38:03.153" CommentCount="2" />
  <row Id="7434" PostTypeId="2" ParentId="7430" CreationDate="2011-02-21T05:27:07.727" Score="7" ViewCount="" Body="&lt;p&gt;It's just a standard change of variables; the (monotone &amp;amp; 1-1) transformation is $y = \exp(x)$ with inverse $x=\log(y)$ and Jacobian $\frac{dx}{dy} = \frac{1}{y}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With a uniform prior $p_y(y) \propto 1$ on $\mathbb{R}$ we get $p_x(x) = p_y(x(y)) |\frac{dx}{dy}| \propto \frac{1}{y}$ on $(0, \infty)$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: Wikipedia has a bit on transformations of random variables: &lt;a href=&quot;http://en.wikipedia.org/wiki/Probability_density_function#Dependent_variables_and_change_of_variables&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Probability_density_function#Dependent_variables_and_change_of_variables&lt;/a&gt;. Similar material will be in any intro probability book. Jim Pitman's &quot;Probability&quot; presents the material in a pretty distinctive way as well IIRC.&lt;/p&gt;&#xA;" OwnerUserId="26" LastEditorUserId="26" LastEditorDisplayName="" LastEditDate="2011-02-21T07:00:52.987" LastActivityDate="2011-02-21T07:00:52.987" CommentCount="1" />
  <row Id="7435" PostTypeId="2" ParentId="7402" CreationDate="2011-02-21T06:37:28.657" Score="2" ViewCount="" Body="&lt;p&gt;To supplement caracal's answer, if you are looking for a user-friendly GUI option for calculating Type II error rates or power for many common designs including the ones implied by your question, you may wish to check out the free software, &lt;a href=&quot;http://www.psycho.uni-duesseldorf.de/abteilungen/aap/gpower3/&quot; rel=&quot;nofollow&quot;&gt;G Power 3&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2011-02-21T06:37:28.657" />
  <row Id="7436" PostTypeId="2" ParentId="7429" CreationDate="2011-02-21T08:14:45.830" Score="1" ViewCount="" Body="&lt;p&gt;If you are using &lt;a href=&quot;http://en.wikipedia.org/wiki/Pearson%27s_chi-square_test#Test_of_independence&quot; rel=&quot;nofollow&quot;&gt;Pearson's chi-square test as a test independence&lt;/a&gt; of two variables in a two-way &lt;a href=&quot;http://en.wikipedia.org/wiki/Contingency_table&quot; rel=&quot;nofollow&quot;&gt;contingency table&lt;/a&gt;, you'll get a zero &lt;em&gt;expected&lt;/em&gt; value only if you have a whole row of zeroes or a whole column of zeroes. You can simply remove that row or column.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-02-21T08:14:45.830" />
  <row Id="7437" PostTypeId="1" CreationDate="2011-02-21T09:14:44.763" Score="0" ViewCount="61" Body="&lt;p&gt;Please help me with the following questions.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt; &lt;li&gt;If x is normally distributed with population mean 100 and standard deviation 8, find&#xA;&lt;ol&gt;&#xA;&lt;li&gt;p(x &amp;lt 107)&lt;/li&gt;&#xA;&#xA;&lt;li&gt;p(x &amp;lt 100)&lt;/li&gt;&#xA;&#xA;&lt;li&gt;p(103 &amp;lt x &amp;lt 114)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;/li&gt;&#xA;&#xA;&lt;li&gt; Find the following probabilities for Standard Normal Distribution, Z&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;p(z &amp;gt 1.5)&lt;/li&gt;&#xA;&#xA;&lt;li&gt; p(z &amp;lt -0.68)&lt;/li&gt;&#xA;&#xA;&lt;li&gt; p(0 &amp;lt z &amp;lt 1.5)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;/li&gt;&#xA;&#xA;&lt;li&gt; The distribution of marks in a certain test has mean 582 and standard deviation 75.&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;If a student's mark is 696, what is the standardized mark?&lt;/li&gt;&#xA;&#xA;&lt;li&gt; If the standardized mark is -8, what is the mark?&lt;/li&gt;&#xA;&#xA;&lt;li&gt;Find the interval of standardized marks corresponding to marks of 300 to 560.&lt;/li&gt;&#xA;&#xA;&lt;li&gt;Find the interval marks corresponding to the interval of standardized marks of -1.2 to 1.2.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="3340" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-02-21T09:34:33.040" LastActivityDate="2011-02-21T09:34:33.040" ClosedDate="2011-02-21T09:59:15.753" Title="Help with homework with Normal distribution" Tags="&lt;distributions&gt;&lt;homework&gt;" CommentCount="4" />
  <row Id="7438" PostTypeId="1" CreationDate="2011-02-21T09:57:19.063" Score="1" ViewCount="199" Body="&lt;p&gt;As a basic question in Regression Analysis, I wanted to ask how can I calculate Margin of Error when I fit a straight line to a set of data.&#xA;Assume that I have variation of parameter $A$ as a function of parameter $B$, then $A=mB \pm e$, where $m$ is the tangent of the fitted straight line and $e$ is what i'm looking for!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Part of my data:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;{{39.7678, 2320.3}, {30.8438, 1614.21}, {125.846, 3078.81}, {55.2345, \&#xA;1947.98}, {22.0671, 972.995}, {30.1827, 701.99}, {29.5734, 837.784}, \&#xA;{24.6913, 1134.23}, {27.2493, 918.887}, {62.7684, 4535.07}, {101.449, \&#xA;5499.83}, {125.248, 6513.04}, {187.409, 6257.72}, {174.138, 5243.63}, \&#xA;{120.747, 3768.02}, {84.178, 3453.12}, {60.2404, 3075.15}, {63.8622, \&#xA;3517.73}, {101.9, 7240.11}, {90.6265, 5706.74}, {100.897, 7353.84}, \&#xA;{159.316, 9867.36}, {109.798, 11471.2}, {104.311, 6924.54}, {82.7057, \&#xA;6339.06}, {140.205, 6555.52}, {173.469, 8644.27}, {138.432, 9655.86}, \&#xA;{95.2955, 5643.33}, {64.563, 3848.77}, {50.7936, 4733.24}, {34.776, &#xA; 0. - 2707.89 I}, {25.3775, 6158.}}&lt;/p&gt;&#xA;" OwnerUserId="3341" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-02-21T10:07:55.150" LastActivityDate="2011-02-21T19:18:51.447" Title="How to calculate margin of error in linear regression?" Tags="&lt;regression&gt;" AnswerCount="2" />
  <row Id="7439" PostTypeId="1" AcceptedAnswerId="7444" CreationDate="2011-02-21T10:27:05.680" Score="6" ViewCount="217" Body="&lt;p&gt;You can have data in wide format or in long format.&#xA;This is quite an important thing, as the useable methods are different, depending on the format.&#xA;I know you have to work with &lt;code&gt;melt()&lt;/code&gt; and &lt;code&gt;cast()&lt;/code&gt; from the reshape package, but there seems some things that I don't get.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone give me a short overview how you do this?&lt;/p&gt;&#xA;" OwnerUserId="3140" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2011-02-21T16:30:39.317" LastActivityDate="2011-02-21T16:30:39.317" Title="How to change data between wide and long formats in R?" Tags="&lt;r&gt;&lt;data-transformation&gt;&lt;dataframe&gt;" AnswerCount="3" CommentCount="4" FavoriteCount="2" />
  <row Id="7440" PostTypeId="1" AcceptedAnswerId="7449" CreationDate="2011-02-21T10:30:18.527" Score="1" ViewCount="256" Body="&lt;p&gt;I need to determine the KL-divergence between two Gaussians. I am comparing my results to &lt;a href=&quot;http://allisons.org/ll/MML/KL/Normal/&quot; rel=&quot;nofollow&quot;&gt;these&lt;/a&gt;, but I can't reproduce their result. My result is obviously wrong, because the KL is not 0 for KL(p, p).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wonder where I am doing a mistake and ask if anyone can spot it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let $p(x) = N(\mu_1, \sigma_1)$ and $q(x) = N(\mu_2, \sigma_2)$. From Bishop's&#xA;PRML I know that&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$KL(p, q) = - \int p(x) \log q(x) dx + \int p(x) \log p(x) dx$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where integration is done over all real line, and that&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\int p(x) \log p(x) dx = \frac{1}{2} (1 + \log 2 \pi \sigma_1^2),$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;so I restrict myself to $\int p(x) \log q(x) dx$, which I can write out as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$-\int p(x) \log \frac{1}{(2 \pi \sigma_2^2)^{(1/2)}} e^{-\frac{(x-\mu_2)^2}{2 \sigma_2^2}} dx,$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;which I can separate into&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{1}{2} \log (2 \pi \sigma_2^2) - \int p(x) \log e^{-\frac{(x-\mu_2)^2}{2 \sigma_2^2}} dx.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Taking the log I get&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{1}{2} \log (2 \pi \sigma_2^2) - \int p(x) -\frac{(x-\mu_2)^2}{2 \sigma_2^2} dx,$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where I separate the sums and get $\sigma_2^2$ out of the integral.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{1}{2} \log (2 \pi \sigma_2^2) + \frac{\int p(x) x^2 dx - \int p(x) 2x\mu dx + \int p(x) \mu^2 dx}{2 \sigma_2^2}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Letting $\langle \rangle$ denote the expectation operator under $p$, I can rewrite this as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{1}{2} \log (2 \pi \sigma_2^2) + \frac{\langle x^2 \rangle - 2 \langle x \rangle \mu_2 + \mu_2^2}{2 \sigma_2^2}.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We know that $var(x) = \langle x^2 \rangle - \langle x \rangle ^2$. Thus &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\langle x^2 \rangle = \sigma_1^2 + \mu_1^2$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and therefore&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{1}{2} \log (2 \pi \sigma^2) + \frac{\sigma_1^2 + \mu_1^2 - 2 \mu_1 \mu_2 + \mu_2^2}{2 \sigma_2^2},$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;which I can put as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{1}{2} \log (2 \pi \sigma_2^2) + \frac{\sigma_1^2 + (\mu_1 - \mu_2)^2}{2 \sigma_2^2}.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Putting everything together, I get to &lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align*}&#xA;KL(p, q) &amp;amp;= - \int p(x) \log q(x) dx + \int p(x) \log p(x) dx\\&#xA;&amp;amp;= \frac{1}{2} \log (2 \pi \sigma_2^2) + \frac{\sigma_1^2 + (\mu_1 - \mu_2)^2}{2 \sigma_2^2} - \frac{1}{2} (1 + \log 2 \pi \sigma_1^2)\\&#xA;&amp;amp;= \log \frac{\sigma_2}{\sigma_1} + \frac{\sigma_1^2 + (\mu_1 - \mu_2)^2}{2 \sigma_2^2}.&#xA;\end{align*}&#xA;Which is wrong since it equals $1$ for two identical Gaussians.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone spot my error?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks to mpiktas for clearing things up. The correct answer is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$KL(p, q) = \log \frac{\sigma_2}{\sigma_1} + \frac{\sigma_1^2 + (\mu_1 - \mu_2)^2}{2 \sigma_2^2} - \frac{1}{2}$&lt;/p&gt;&#xA;" OwnerUserId="2860" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-02-21T19:23:38.103" LastActivityDate="2011-02-21T19:23:38.103" Title="KL divergence between two univariate Gaussians" Tags="&lt;normal-distribution&gt;&lt;kullback-leibler&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="7441" PostTypeId="2" ParentId="7438" CreationDate="2011-02-21T10:30:24.187" Score="2" ViewCount="" Body="&lt;p&gt;It seems that you want the residuals of linear regression without the intercept term where dependent variable is $A$ and the independent variable is $B$. This can be done with various statistical packages. Here is the implementation in R.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;aa &amp;lt;-&quot;(39.7678, 2320.3}, {30.8438, 1614.21}, {125.846, 3078.81}, {55.2345,  1947.98}, {22.0671, 972.995}, {30.1827, 701.99}, {29.5734, 837.784},  {24.6913, 1134.23}, {27.2493, 918.887}, {62.7684, 4535.07}, {101.449,  5499.83}, {125.248, 6513.04}, {187.409, 6257.72}, {174.138, 5243.63},  {120.747, 3768.02}, {84.178, 3453.12}, {60.2404, 3075.15}, {63.8622,  3517.73}, {101.9, 7240.11}, {90.6265, 5706.74}, {100.897, 7353.84},  {159.316, 9867.36}, {109.798, 11471.2}, {104.311, 6924.54}, {82.7057,  6339.06}, {140.205, 6555.52}, {173.469, 8644.27}, {138.432, 9655.86},  {95.2955, 5643.33}, {64.563, 3848.77}, {50.7936, 4733.24}, {34.776, 2707.89 }, {25.3775, 6158)&quot;&#xA;&#xA;aa &amp;lt;- gsub(&quot; *&quot;,&quot;&quot;,aa)&#xA;&#xA;aa &amp;lt;- gsub(&quot;[}],[{]&quot;,&quot;);(&quot;,aa)&#xA;&#xA;df&amp;lt;-t(sapply(strsplit(aa,&quot;;&quot;)[[1]],function(l)eval(parse(text=paste(&quot;c&quot;,l,sep=&quot;&quot;)))))&#xA;&#xA;rownames(df) &amp;lt;-NULL&#xA;&#xA;colnames(df)&amp;lt;-c(&quot;A&quot;,&quot;B&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;All the code above is to read in your data into R. It would helped a lot if you simply provided the link to txt or csv file. Note that I fixed some errors in the last two sets of data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are the residuals:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; residuals(lsfit(y=df[,&quot;A&quot;],x=df[,&quot;B&quot;],intercept=FALSE))&#xA;&#xA; [1]   1.4348447   4.1759382  74.9819367  23.0525277   5.9925519&#xA; [6]  18.5853413  15.7326309   5.9530386  12.0686534 -12.1540823&#xA;[11]  10.5880176  17.6480900  84.0271524  87.5096181  58.4967124&#xA;[16]  27.1300775   9.4368024   5.7468757 -17.7116076  -3.6527783&#xA;[21] -20.5935056  -3.6995884 -79.7141307 -10.0871737 -22.0199405&#xA;[26]  31.9032911  30.6597014 -21.0894627   2.0637986   0.9786668&#xA;[31] -27.4027873  -9.9602093 -76.3569045&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Note that these residuals have non-zero mean.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;res &amp;lt;- residuals(lsfit(y=df[,&quot;A&quot;],x=df[,&quot;B&quot;],intercept=FALSE))&#xA;mean(res)&#xA;&#xA;[1] 6.779518&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So clearly you need to include intercept, since otherwise you cannot write $\pm e$. As @whuber pointed out, you probably need root mean square error. For the model with the intercept you can calculate is as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; res &amp;lt;- residuals(lsfit(y=df[,&quot;A&quot;],x=df[,&quot;B&quot;],intercept=TRUE))&#xA;&amp;gt; res&#xA; [1] -15.716076 -15.933997  61.009708   4.341324 -16.804527  -5.347439&#xA; [7]  -7.631077 -16.168352 -10.955176 -20.023554   6.761571  18.067708&#xA;[13]  83.376799  82.609513  47.412760  14.726472  -4.550764  -6.385968&#xA;[19] -14.245054  -6.612126 -16.650343  10.776996 -58.516325  -7.943081&#xA;[25] -22.329422  32.500930  40.010676  -7.499212  -1.161282  -9.766886&#xA;[31] -34.441787 -25.486854 -77.425155&#xA;&#xA;&amp;gt; sqrt(sum(res^2)/(length(res)-2))&#xA;[1] 34.70359&#xA;&amp;gt;  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Note that you can use functions &lt;code&gt;lm&lt;/code&gt; and &lt;code&gt;summary&lt;/code&gt; to get all the proper statistics of linear regression:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; summary(lm(A~B,data=data.frame(df)))&#xA;&#xA;Call:&#xA;lm(formula = A ~ B, data = data.frame(df))&#xA;&#xA;Residuals:&#xA;    Min      1Q  Median      3Q     Max &#xA;-77.425 -16.168  -7.499  10.777  83.377 &#xA;&#xA;Coefficients:&#xA;             Estimate Std. Error t value Pr(&amp;gt;|t|)    &#xA;(Intercept) 26.874614  12.027882   2.234   0.0328 *  &#xA;B            0.012330   0.002169   5.685 3.02e-06 ***&#xA;---&#xA;Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 &#xA;&#xA;Residual standard error: 34.7 on 31 degrees of freedom&#xA;Multiple R-squared: 0.5104, Adjusted R-squared: 0.4946 &#xA;F-statistic: 32.32 on 1 and 31 DF,  p-value: 3.018e-06 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-02-21T19:18:51.447" LastActivityDate="2011-02-21T19:18:51.447" />
  <row Id="7443" PostTypeId="2" ParentId="7440" CreationDate="2011-02-21T10:58:18.427" Score="2" ViewCount="" Body="&lt;p&gt;Hi !&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I did not have a look at your calculation but here is mine with a lot of details. &#xA;Suppose $p$ is the density of a normal random variable with mean $\mu_1$ and variance $\sigma^2_1$, and that $q$ is the density of a normal random variable with mean $\mu_2$ and variance $\sigma^2_2$. &#xA;The Kullback-Leibler distance from $q$ to $p$ is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\int \left[\log( p(x)) - log( q(x)) \right] p(x) dx$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$=\int \left[ -\frac{1}{2} \log(2\pi) - \log(\sigma_1) - \frac{1}{2} \left(\frac{x-\mu_1}{\sigma_1}\right)^2 + \frac{1}{2}\log(2\pi) + \log(\sigma_2) + \frac{1}{2} \left(\frac{x-\mu_2}{\sigma_2}\right)^2  \right]$&#xA;$\times \frac{1}{\sqrt{2\pi}\sigma_1} \exp\left[-\frac{1}{2}\left(\frac{x-\mu_1}{\sigma_1}\right)^2\right] dx$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$=\int \left\{\log\left(\frac{\sigma_2}{\sigma_1}\right) + \frac{1}{2} \left[ \left(\frac{x-\mu_2}{\sigma_2}\right)^2 + \left(\frac{x-\mu_1}{\sigma_1}\right)^2 \right] \right\}$&#xA;$\times \frac{1}{\sqrt{2\pi}\sigma_1} \exp\left[-\frac{1}{2}\left(\frac{x-\mu_1}{\sigma_1}\right)^2\right] dx$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$=E_{1} \left\{\log\left(\frac{\sigma_2}{\sigma_1}\right) + \frac{1}{2} \left[ \left(\frac{x-\mu_2}{\sigma_2}\right)^2 + \left(\frac{x-\mu_1}{\sigma_1}\right)^2 \right]\right\}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$=\log\left(\frac{\sigma_2}{\sigma_1}\right) + \frac{1}{2\sigma_2^2} E_1 \left\{(X-\mu_2)^2\right\} - \frac{1}{2\sigma_1^2} E_1 \left\{(X-\mu_1)^2\right\}$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$=-\frac{1}{2} \log\left(\frac{\sigma_1^2}{\sigma_2^2}\right) + \frac{1}{2\sigma_2^2} E_1 \left\{(X-\mu_2)^2\right\} - \frac{1}{2}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Now note that $(X - \mu_2)^2 = (X-\mu_1+\mu_1-\mu_2)^2 = (X-\mu_1)^2 + 2(X-\mu_1)(X-\mu_2) + (\mu_1-\mu_2)^2$)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$=-\frac{1}{2} \log\left(\frac{\sigma_1^2}{\sigma_2^2}\right) + \frac{1}{2\sigma_2^2}&#xD;&#xA;\left[E_1\left\{(X-\mu_1)^2\right\} + 2(\mu_1-\mu_2)E_1\left\{X-\mu_1\right\} + (\mu_1-\mu_2)^2\right] - \frac{1}{2}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$=-\frac{1}{2} \log\left(\frac{\sigma_1^2}{\sigma_2^2}\right) + \frac{1}{2}\frac{\sigma_1^2}{\sigma_2^2} + \frac{1}{2\sigma_2^2}(\mu_1-\mu_2)^2 - \frac{1}{2}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$=\frac{(\mu_1-\mu_2)^2}{2\sigma^2_2} + \frac{1}{2}\left(\frac{\sigma_1^2}{\sigma_2^2} - 1 - \log \frac{\sigma_1^2}{\sigma_2^2}\right)$&lt;/p&gt;&#xA;" OwnerUserId="3019" LastEditorUserId="3019" LastEditorDisplayName="" LastEditDate="2011-02-21T11:08:31.370" LastActivityDate="2011-02-21T11:08:31.370" />
  <row Id="7444" PostTypeId="2" ParentId="7439" CreationDate="2011-02-21T11:09:33.423" Score="7" ViewCount="" Body="&lt;p&gt;There are several resources on Hadley Wickham's &lt;a href=&quot;http://had.co.nz/reshape/&quot; rel=&quot;nofollow&quot;&gt;website&lt;/a&gt; for the package (now called &lt;code&gt;reshape2&lt;/code&gt;), including a link to a &lt;a href=&quot;http://www.jstatsoft.org/v21/i12&quot; rel=&quot;nofollow&quot;&gt;paper&lt;/a&gt; on the package in the Journal of Statistical Software.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a brief example from the paper:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; require(reshape2)&#xA;Loading required package: reshape2&#xA;&amp;gt; data(smiths)&#xA;&amp;gt; smiths&#xA;     subject time age weight height&#xA;1 John Smith    1  33     90   1.87&#xA;2 Mary Smith    1  NA     NA   1.54&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;We note that the data are in the wide form. To go to the long form, we make the &lt;code&gt;smiths&lt;/code&gt; data frame &lt;em&gt;molten&lt;/em&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; melt(smiths)&#xA;Using subject as id variables&#xA;     subject variable value&#xA;1 John Smith     time  1.00&#xA;2 Mary Smith     time  1.00&#xA;3 John Smith      age 33.00&#xA;4 Mary Smith      age    NA&#xA;5 John Smith   weight 90.00&#xA;6 Mary Smith   weight    NA&#xA;7 John Smith   height  1.87&#xA;8 Mary Smith   height  1.54&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Notice how &lt;code&gt;melt()&lt;/code&gt; chose one of the variables as the id, but we can state explicitly which to use via argument &lt;code&gt;'id'&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; melt(smiths, id = &quot;subject&quot;)&#xA;     subject variable value&#xA;1 John Smith     time  1.00&#xA;2 Mary Smith     time  1.00&#xA;3 John Smith      age 33.00&#xA;4 Mary Smith      age    NA&#xA;5 John Smith   weight 90.00&#xA;6 Mary Smith   weight    NA&#xA;7 John Smith   height  1.87&#xA;8 Mary Smith   height  1.54&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here is another example from &lt;code&gt;?cast&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#Air quality example&#xA;names(airquality) &amp;lt;- tolower(names(airquality))&#xA;aqm &amp;lt;- melt(airquality, id=c(&quot;month&quot;, &quot;day&quot;), na.rm=TRUE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If we store the molten data frame, we can &lt;em&gt;cast&lt;/em&gt; into other forms. In the new version of &lt;code&gt;reshape&lt;/code&gt; (called &lt;code&gt;reshape2&lt;/code&gt;) there are functions &lt;code&gt;acast()&lt;/code&gt; and &lt;code&gt;dcast()&lt;/code&gt; returning an array-like (array, matrix, vector) result or a data frame respectively. These functions also take an aggregating function (eg &lt;code&gt;mean()&lt;/code&gt;) to provide summaries of data in molten form. For example, following on from the Air Quality example above, we can generate, in wide form, monthly mean values for the variables in the data set:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; dcast(aqm, month ~ variable, mean)&#xA;  month    ozone  solar.r      wind     temp&#xA;1     5 23.61538 181.2963 11.622581 65.54839&#xA;2     6 29.44444 190.1667 10.266667 79.10000&#xA;3     7 59.11538 216.4839  8.941935 83.90323&#xA;4     8 59.96154 171.8571  8.793548 83.96774&#xA;5     9 31.44828 167.4333 10.180000 76.90000&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;There are really only two main functions in &lt;code&gt;reshape2&lt;/code&gt;: &lt;code&gt;melt()&lt;/code&gt; and the &lt;code&gt;acast()&lt;/code&gt; and &lt;code&gt;dcast()&lt;/code&gt; pairing. Look at the examples in the help pages for these two functions, see Hadley's website (link above) and look at the paper I mentioned. That should get you started.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might also look into Hadley's &lt;a href=&quot;http://had.co.nz/plyr/&quot; rel=&quot;nofollow&quot;&gt;&lt;code&gt;plyr&lt;/code&gt; package&lt;/a&gt; which does similar things to &lt;code&gt;reshape2&lt;/code&gt; but is designed to do a whole lot more besides.&lt;/p&gt;&#xA;" OwnerUserId="1390" LastActivityDate="2011-02-21T11:09:33.423" />
  <row Id="7446" PostTypeId="2" ParentId="7439" CreationDate="2011-02-21T11:14:36.693" Score="6" ViewCount="" Body="&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://www.statmethods.net/management/reshape.html&quot; rel=&quot;nofollow&quot;&gt;Quick-R&lt;/a&gt; has simple example of using reshape package&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;See also &lt;code&gt;?reshape&lt;/code&gt; (&lt;a href=&quot;http://stat.ethz.ch/R-manual/R-patched/library/stats/html/reshape.html&quot; rel=&quot;nofollow&quot;&gt;LINK&lt;/a&gt;) for the Base R way of moving between wide and long format.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastActivityDate="2011-02-21T11:14:36.693" />
  <row Id="7447" PostTypeId="1" CreationDate="2011-02-21T11:40:25.130" Score="4" ViewCount="148" Body="&lt;p&gt;Both &lt;a href=&quot;http://en.wikipedia.org/wiki/Root_mean_square&quot; rel=&quot;nofollow&quot;&gt;Root Mean Square&lt;/a&gt; and &lt;a href=&quot;http://en.wikipedia.org/wiki/Absolute_deviation#Average_absolute_deviation&quot; rel=&quot;nofollow&quot;&gt;Average absolute deviation&lt;/a&gt; seem like the measures of the magnitude of variability (especially when the variates are both +ve and -ve). What are the rules of thumb to choose one of them over the other?&lt;/p&gt;&#xA;" OwnerUserId="3292" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-02-21T11:50:01.563" LastActivityDate="2011-02-21T11:52:42.560" Title="Root mean square vs average absolute deviation?" Tags="&lt;regression&gt;&lt;standard-deviation&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="7448" PostTypeId="2" ParentId="7447" CreationDate="2011-02-21T11:52:42.560" Score="7" ViewCount="" Body="&lt;p&gt;In theory, this should be determined by how important different sized errors are to you, or in other words, your &lt;em&gt;loss function&lt;/em&gt;.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the real world, people put ease of use first.  So RMS deviations (or the related variances) are easier to combine, and easier to calculate in a single pass, while average absolute deviations are more robust to outliers and exist for more distributions. Basic linear regression and many of its offshoots are based on minimsing RMS errors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another point is that the mean will minimise RMS deviations while the median will minimise absolute deviations, and you may prefer one of these.      &lt;/p&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-02-21T11:52:42.560" CommentCount="2" />
  <row Id="7449" PostTypeId="2" ParentId="7440" CreationDate="2011-02-21T11:55:19.103" Score="3" ViewCount="" Body="&lt;p&gt;OK, my bad. The error is in the last equation:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align}&#xA;KL(p, q) &amp;amp;= - \int p(x) \log q(x) dx + \int p(x) \log p(x) dx\\&#xA;&amp;amp;=\frac{1}{2} \log (2 \pi \sigma_2^2) + \frac{\sigma_1^2 + (\mu_1 - \mu_2)^2}{2 \sigma_2^2} - \frac{1}{2} (1 + \log 2 \pi \sigma_1^2)\\&#xA;&amp;amp;= \log \frac{\sigma_2}{\sigma_1} + \frac{\sigma_1^2 + (\mu_1 - \mu_2)^2}{2 \sigma_2^2}&#xA;\end{align}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$-\frac{1}{2}$ is missing in the last line. With it the last line becomes zero when $\mu_1=\mu_2$ and $\sigma_1=\sigma_2$.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-02-21T17:58:30.720" LastActivityDate="2011-02-21T17:58:30.720" CommentCount="3" />
  <row Id="7450" PostTypeId="1" CreationDate="2011-02-21T13:51:50.393" Score="6" ViewCount="380" Body="&lt;p&gt;I have a problem with the estimation parameter for Zipf. My situation is the following: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a sample set (measured from an experiment that generates calls that should follow a Zipf distribution). I have to demonstrate that this generator really generates calls with zipf distribution. &#xA;I already read this Q&amp;amp;A &lt;a href=&quot;http://stats.stackexchange.com/questions/6780/how-to-calculate-zipfs-law-coefficient-from-a-set-of-top-frequencies&quot;&gt;How to calculate Zipf's law coefficient from a set of top frequencies?&lt;/a&gt; but I reach bad results because i use a truncated distribution. For example if I set the &quot;s&quot; value to &quot;0.9&quot; for the generation process, if I try to estimate &quot;s&quot; value as wrote in the reported Q&amp;amp;A I obtain &quot;s&quot; equal to 0.2 ca. I think this is due to the fact that i use a TRUNCATED distribution (i have to limit the zipf with a truncation point, it is right-truncated).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can i estimate parameters with a truncated zipf distribution?&lt;/p&gt;&#xA;" OwnerUserId="3342" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2011-03-15T19:04:25.767" LastActivityDate="2011-04-08T01:51:15.640" Title="How to estimate parameters for Zipf truncated distribution from a data sample?" Tags="&lt;distributions&gt;&lt;estimation&gt;&lt;pareto-distribution&gt;&lt;zipf&gt;" AnswerCount="3" CommentCount="4" FavoriteCount="2" />
  <row Id="7452" PostTypeId="2" ParentId="7450" CreationDate="2011-02-21T14:36:44.550" Score="3" ViewCount="" Body="&lt;p&gt;The paper&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Clauset, A &lt;em&gt;et al&lt;/em&gt;, &lt;a href=&quot;http://arxiv.org/abs/0706.1062&quot; rel=&quot;nofollow&quot;&gt;Power-law Distributions in Empirical Data&lt;/a&gt;. 2009&lt;/p&gt;&#xA;&#xA;&lt;p&gt;contains a very good description of how to go about fitting power law models. The associated &lt;a href=&quot;http://tuvalu.santafe.edu/~aaronc/powerlaws/&quot; rel=&quot;nofollow&quot;&gt;web-page&lt;/a&gt; has code samples. Unfortunately, it doesn't give code for truncated distributions, but it may give you a pointer.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;As an aside, the paper discusses the fact that many &quot;power-law datasets&quot; can be modelled equally well (and in some cases better) with the Log normal or exponential distributions!&lt;/p&gt;&#xA;" OwnerUserId="8" LastActivityDate="2011-02-21T14:36:44.550" CommentCount="2" />
  <row Id="7454" PostTypeId="2" ParentId="7439" CreationDate="2011-02-21T15:25:54.473" Score="3" ViewCount="" Body="&lt;p&gt;You don't have to use melt() and cast().  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Reshaping data can be done lots of ways.  In your particular example on your site using recast with aggregate was redundant.  aggregate() does the task fine all on it's own.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;aggregate(cbind(LPMVTUZ, LPMVTVC, LPMVTXC) ~ year, dtm, sum)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I do like how, in your blog post, you explain what melt() is doing.  Very few people understand that and once you see it then it gets easier to see how cast() works and how you might write your own functions if you want.&lt;/p&gt;&#xA;" OwnerUserId="601" LastActivityDate="2011-02-21T15:25:54.473" />
  <row Id="7455" PostTypeId="1" AcceptedAnswerId="7473" CreationDate="2011-02-21T16:34:03.293" Score="3" ViewCount="86" Body="&lt;p&gt;Can someone refer me to a good reference that explains the connection between Bayesian statistics and generative modeling techniques? Why do we usually use generative models with Bayesian techniques?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why it is especially appealing to use Bayesian statistics in the absence of complete data, if at all?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that I come from a more machine learning oriented view, and I am interested in reading more about it from the statistics community.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any good reference that discusses these points would be greatly appreciated.&#xA;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="3347" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-02-22T07:53:42.247" LastActivityDate="2011-02-22T07:53:42.247" Title="The connection between Bayesian statistics and generative modeling" Tags="&lt;bayesian&gt;&lt;generative-models&gt;" AnswerCount="1" />
  <row Id="7456" PostTypeId="2" ParentId="7438" CreationDate="2011-02-21T16:44:01.187" Score="2" ViewCount="" Body="&lt;p&gt;You are asking to &lt;em&gt;estimate&lt;/em&gt; the typical size of the error.  This is usually done by estimating the variance of the random variable $e$ and taking its square root.  It is often called the &quot;root mean square error&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These data look like they are formatted for &lt;em&gt;Mathematica&lt;/em&gt;.  A &lt;em&gt;Mathematica&lt;/em&gt; 8 solution is&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;{a, b} = {{39.7678, 2320.3},...&amp;lt;31 values omitted&amp;gt;...{25.3775, 6158.}} // Transpose;&#xA;model = LinearModelFit[{{b} // Transpose, a}];&#xA;model[&quot;EstimatedVariance&quot;] // Sqrt&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This assumes the data are given in the sequence (a,b) and not (b,a)!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is important to note that this model omits the usual constant term.  If you intended that to be there, include it by inserting a column of 1's in the &quot;design matrix&quot;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;m = {ConstantArray[1, Length[a]], b} // Transpose;&#xA;model = LinearModelFit[{m, a}];&#xA;model[&quot;EstimatedVariance&quot;] // Sqrt&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In either case the output is a single &lt;em&gt;number&lt;/em&gt;: your &quot;margin of error.&quot;&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-02-21T16:44:01.187" CommentCount="5" />
  <row Id="7457" PostTypeId="1" AcceptedAnswerId="7458" CreationDate="2011-02-21T16:54:06.813" Score="1" ViewCount="133" Body="&lt;p&gt;I'm trying to normalize a set of columns of data in an excel spreadsheet.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to get the values so that the highest value in a column is = 1 and lowest is = to 0, so I've come up with the formula:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;=(A1-MIN(A1:A30))/(MAX(A1:A30)-MIN(A1:A30))&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This seems to work fine, but when I drag down the formula to populate the cells below it, now only does &lt;code&gt;A1&lt;/code&gt; increase, but &lt;code&gt;A1:A30&lt;/code&gt; does too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a way to lock the range while updating just the number I'm interested in?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've tried putting the Max and min in a different cell and referencing that but it just references the cell under the one that the Max and min are in and I get divide by zero errors because there is nothing there.&lt;/p&gt;&#xA;" OwnerUserId="3348" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-02-21T19:04:33.937" LastActivityDate="2011-02-21T19:04:33.937" Title="How to stop excel from changing a range when you drag a formula down?" Tags="&lt;excel&gt;" AnswerCount="1" />
  <row Id="7458" PostTypeId="2" ParentId="7457" CreationDate="2011-02-21T17:02:21.187" Score="10" ViewCount="" Body="&lt;p&gt;A '$' will lock down the reference to an absolute one versus a relative one. You can lock down the column, row or both. Here is a locked down absolute reference for your example.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(A1-MIN($A$1:$A$30))/(MAX($A$1:$A$30)-MIN($A$1:$A$30))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2040" LastActivityDate="2011-02-21T17:02:21.187" CommentCount="4" />
  <row Id="7459" PostTypeId="2" ParentId="7450" CreationDate="2011-02-21T17:31:02.337" Score="7" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt;: 7 Apr 2011&#xA;This answer is getting quite long and covers multiple aspects of the problem at hand. However, I've resisted, so far, breaking it into separate answers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've added at the very bottom a discussion of the performance of Pearson's $\chi^2$ for this example.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Bruce M. Hill authored, perhaps, the &quot;seminal&quot; paper on estimation in a Zipf-like context. He wrote several papers in the mid-1970's on the topic. However, the &quot;Hill estimator&quot; (as it's now called) essentially relies on the maximal order statistics of the sample and so, depending on the type of truncation present, that could get you in some trouble.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The main paper is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;B. M. Hill, &lt;a href=&quot;http://projecteuclid.org/euclid.aos/1176343247&quot; rel=&quot;nofollow&quot;&gt;A simple general approach to inference about the tail of a distribution&lt;/a&gt;, &lt;em&gt;Ann. Stat.&lt;/em&gt;, 1975.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your data truly are initially Zipf and are then truncated, then a nice correspondence between the &lt;em&gt;degree distribution&lt;/em&gt; and the &lt;em&gt;Zipf plot&lt;/em&gt; can be harnessed to your advantage.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Specifically, the &lt;em&gt;degree distribution&lt;/em&gt; is simply the empirical distribution of the number of times that each integer response is seen,&#xA;$$&#xD;&#xA;d_i = \frac{\#\{j: X_j = i\}}{n} .&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we plot this against $i$ on a log-log plot, we'll get a linear trend with a slope corresponding to the scaling coefficient.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, if we plot the &lt;em&gt;Zipf plot&lt;/em&gt;, where we sort the sample from largest to smallest and then plot the values against their ranks, we get &lt;em&gt;a different&lt;/em&gt; linear trend with a &lt;em&gt;different&lt;/em&gt; slope. However the slopes are related.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If $\alpha$ is the scaling-law coefficient for the Zipf distribution, then the slope in the first plot is $-\alpha$ and the slope in the second plot is $-1/(\alpha-1)$. Below is an example plot for $\alpha = 2$ and $n = 10^6$. The left-hand pane is the degree distribution and the slope of the red line is $-2$. The right-hand side is the Zipf plot, with the superimposed red line having a slope of $-1/(2-1) = -1$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/uNriV.png&quot; alt=&quot;Degree distribution (left) and Zipf (right) plots for an i.i.d. sample from a Zipf distribution.&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, if your data have been truncated so that you see no values larger than some threshold $\tau$, but the data are otherwise Zipf-distributed and $\tau$ is reasonably large, then you can estimate $\alpha$ from the &lt;em&gt;degree distribution&lt;/em&gt;. A very simple approach is to fit a line to the log-log plot and use the corresponding coefficient.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your data are truncated so that you don't see &lt;em&gt;small&lt;/em&gt; values (e.g., the way much filtering is done for large web data sets), then you can use the Zipf plot to estimate the slope on a log-log scale and then &quot;back out&quot; the scaling exponent. Say your estimate of the slope from the Zipf plot is $\hat{\beta}$. Then, one simple estimate of the scaling-law coefficient is&#xA;$$&#xD;&#xA;\hat{\alpha} = 1 - \frac{1}{\hat{\beta}} .&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;@csgillespie gave one recent paper co-authored by Mark Newman at Michigan regarding this topic. He seems to publish a lot of similar articles on this. Below is another along with a couple other references that might be of interest. Newman sometimes doesn't do the most sensible thing statistically, so be cautious.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;MEJ Newman, &lt;a href=&quot;http://arxiv.org/abs/cond-mat/0412004&quot; rel=&quot;nofollow&quot;&gt;Power laws, Pareto distributions and Zipf's law&lt;/a&gt;, &lt;em&gt;Contemporary Physics&lt;/em&gt; 46, 2005, pp. 323-351.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;M. Mitzenmacher, &lt;a href=&quot;http://projecteuclid.org/euclid.im/1089229510&quot; rel=&quot;nofollow&quot;&gt;A Brief History of Generative Models for Power Law and Lognormal Distributions&lt;/a&gt;, &lt;em&gt;Internet Math.&lt;/em&gt;, vol. 1, no. 2, 2003, pp. 226-251.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;K. Knight, &lt;a href=&quot;http://www.utstat.utoronto.ca/keith/papers/robusthill.pdf&quot; rel=&quot;nofollow&quot;&gt;A simple modification of the Hill estimator with applications to robustness and bias reduction&lt;/a&gt;, 2010.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Addendum&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a simple simulation in $R$ to demonstrate what you might expect if you took a sample of size $10^5$ from your distribution (as described in your comment below your original question).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; x &amp;lt;- (1:500)^(-0.9)&#xA;&amp;gt; p &amp;lt;- x / sum(x)&#xA;&amp;gt; y &amp;lt;- sample(length(p), size=100000, repl=TRUE, prob=p)&#xA;&amp;gt; tab &amp;lt;- table(y)&#xA;&amp;gt; plot( 1:500, tab/sum(tab), log=&quot;xy&quot;, pch=20, &#xA;        main=&quot;'Truncated' Zipf simulation (truncated at i=500)&quot;,&#xA;        xlab=&quot;Response&quot;, ylab=&quot;Probability&quot; )&#xA;&amp;gt; lines(p, col=&quot;red&quot;, lwd=2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The resulting plot is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/tTLNr.png&quot; alt=&quot;&amp;quot;Truncated&amp;quot; Zipf plot (truncated at i=500)&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the plot, we can see that the relative error of the degree distribution for $i \leq 30$ (or so) is &lt;em&gt;very&lt;/em&gt; good. You could do a formal chi-square test, but this does &lt;strong&gt;not&lt;/strong&gt; strictly tell you that the data follow the prespecified distribution. It only tells you that you have no evidence to conclude that they &lt;em&gt;don't&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Still, from a practical standpoint, such a plot should be relatively compelling.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Addendum 2&lt;/strong&gt;: Let's consider the example that Maurizio uses in his comments below. We'll assume that $\alpha = 2$ and $n = 300\,000$, with a truncated Zipf distribution having maximum value $x_{\mathrm{max}} = 500$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We'll calculate Pearson's $\chi^2$ statistic in two ways. The standard way is via the statistic&#xA;$$&#xD;&#xA;X^2 = \sum_{i=1}^{500} \frac{(O_i - E_i)^2}{E_i}&#xD;&#xA;$$&#xA;where $O_i$ is the observed counts of the value $i$ in the sample and $E_i = n p_i = n i^{-\alpha} / \sum_{j=1}^{500} j^{-\alpha}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We'll also calculate a second statistic formed by first binning the counts in bins of size 40, as shown in Maurizio's spreadsheet (the last bin only contains the sum of twenty separate outcome values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's draw 5000 separate samples of size $n$ from this distribution and calculate the $p$-values using these two different statistics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The histograms of the $p$-values are below and are seen to be quite uniform. The empirical Type I error rates are 0.0716 (standard, unbinned method) and 0.0502 (binned method), respectively and neither are statistically significantly different from the target 0.05 value for the sample size of 5000 that we've chosen.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/9vHTa.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the $R$ code.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Chi-square testing of the truncated Zipf.&#xA;&#xA;a &amp;lt;- 2&#xA;n &amp;lt;- 300000&#xA;xmax &amp;lt;- 500&#xA;&#xA;nreps &amp;lt;- 5000&#xA;&#xA;zipf.chisq.test &amp;lt;- function(n, a=0.9, xmax=500, bin.size = 40)&#xA;{&#xA;  # Make the probability vector&#xA;  x &amp;lt;- (1:xmax)^(-a)&#xA;  p &amp;lt;- x / sum(x)&#xA;&#xA;  # Do the sampling&#xA;  y &amp;lt;- sample(length(p), size=n, repl=TRUE, prob=p)&#xA;&#xA;  # Use tabulate, NOT table!&#xA;  tab &amp;lt;- tabulate(y,xmax)&#xA;&#xA;  # unbinned chi-square stat and p-value&#xA;  discrepancy &amp;lt;- (tab-n*p)^2/(n*p)&#xA;  chi.stat &amp;lt;- sum(discrepancy)&#xA;  p.val    &amp;lt;- pchisq(chi.stat, df=xmax-1, lower.tail = FALSE)&#xA;&#xA;  # binned chi-square stat and p-value&#xA;  bins &amp;lt;- seq(bin.size,xmax,by=bin.size)&#xA;  if( bins[length(bins)] != xmax )&#xA;    bins &amp;lt;- c(bins, xmax)&#xA;&#xA;  tab.bin  &amp;lt;- cumsum(tab)[bins]&#xA;  tab.bin &amp;lt;- c(tab.bin[1], diff(tab.bin))&#xA;&#xA;  prob.bin &amp;lt;- cumsum(p)[bins] &#xA;  prob.bin &amp;lt;- c(prob.bin[1], diff(prob.bin))&#xA;&#xA;  disc.bin &amp;lt;- (tab.bin - n*prob.bin)^2/(n * prob.bin)&#xA;  chi.stat.bin &amp;lt;- sum(disc.bin)&#xA;  p.val.bin &amp;lt;- pchisq(chi.stat.bin, df=length(tab.bin)-1, lower.tail = FALSE)&#xA;&#xA;  # Return the binned and unbineed p-values&#xA;  c(p.val, p.val.bin, chi.stat, chi.stat.bin)&#xA;}&#xA;&#xA;set.seed( .Random.seed[2] )&#xA;&#xA;all &amp;lt;- replicate(nreps, zipf.chisq.test(n, a, xmax))&#xA;&#xA;par(mfrow=c(2,1))&#xA;hist( all[1,], breaks=20, col=&quot;darkgrey&quot;, border=&quot;white&quot;,&#xA;      main=&quot;Histogram of unbinned chi-square p-values&quot;, xlab=&quot;p-value&quot;)&#xA;hist( all[2,], breaks=20, col=&quot;darkgrey&quot;, border=&quot;white&quot;,&#xA;      main=&quot;Histogram of binned chi-square p-values&quot;, xlab=&quot;p-value&quot; )&#xA;&#xA;type.one.error &amp;lt;- rowMeans( all[1:2,] &amp;lt; 0.05 )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2970" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-04-08T01:51:15.640" LastActivityDate="2011-04-08T01:51:15.640" CommentCount="11" />
  <row Id="7460" PostTypeId="2" ParentId="7224" CreationDate="2011-02-21T18:23:19.980" Score="0" ViewCount="" Body="&lt;p&gt;If you want to do it yourself, I would recommend using Intel's free and open source OpenCV (CV for computer vision) project. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://opencv.willowgarage.com/&quot; rel=&quot;nofollow&quot;&gt;http://opencv.willowgarage.com/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://oreilly.com/catalog/9780596516130&quot; rel=&quot;nofollow&quot;&gt;http://oreilly.com/catalog/9780596516130&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="74" LastActivityDate="2011-02-21T18:23:19.980" />
  <row Id="7461" PostTypeId="2" ParentId="6870" CreationDate="2011-02-21T18:53:41.467" Score="0" ViewCount="" Body="&lt;p&gt;Thanks for the tips David.  I posted this question on the JAGS support forum and got a useful answer.  The key was to use a two dimensional array for the 'true' values.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for (j in 1:n){ &#xA;  x_obs[j] ~ dnorm(xy_true[j,1], prec_x)T(xy_true[j,1],) &#xA;  y_obs[j] ~ dnorm(xy_true[j,2], prec_y)&#xA;  xy_true[j, ] ~ dmnorm(mu[ z [j],1:2], tau[z[j],1:2,1:2]) &#xA;  z[j]~dcat(prob[ ]) &#xA;}&#xA;&#xA; #priors for measurement error &#xA; e_x~dunif(.1,.9)&#xA; prec_x&amp;lt;-1/pow(e_x,2)&#xA; e_y~dunif(2,4)&#xA; prec_y&amp;lt;-1/pow(e_y,2) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2310" LastActivityDate="2011-02-21T18:53:41.467" />
  <row Id="7462" PostTypeId="2" ParentId="6653" CreationDate="2011-02-21T19:22:09.767" Score="0" ViewCount="" Body="&lt;p&gt;Check out the stats nerds at &lt;a href=&quot;http://www.footballoutsiders.com/info/methods&quot; rel=&quot;nofollow&quot;&gt;Football Outsiders&lt;/a&gt; as well as the book &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/069113913X&quot; rel=&quot;nofollow&quot;&gt;Mathletics&lt;/a&gt; for some inspiration.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Football Outsiders guys make game predictions based on &lt;em&gt;every play&lt;/em&gt; in a football game. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Winston in Mathletics uses some techniques such as dynamic programming as well. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also consider other algorithms such as SVM. &lt;/p&gt;&#xA;" OwnerUserId="74" LastActivityDate="2011-02-21T19:22:09.767" />
  <row Id="7465" PostTypeId="2" ParentId="6538" CreationDate="2011-02-21T19:51:03.823" Score="1" ViewCount="" Body="&lt;p&gt;I would go to the curriculum websites of the top stats schools, write down the books they use in their undergrad courses, see which ones are highly rated on Amazon, and order them at your public/university library.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some schools to consider:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://statistics.mit.edu/classes.php&quot; rel=&quot;nofollow&quot;&gt;MIT&lt;/a&gt; - technically, cross-taught with Harvard. &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://catalog.caltech.edu/pdf/catalog_10_11.pdf&quot; rel=&quot;nofollow&quot;&gt;Caltech&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.stat.cmu.edu/programs/undergraduate/the-major-in-statistics&quot; rel=&quot;nofollow&quot;&gt;Carnegie Mellon&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www-stat.stanford.edu/academics/undergrad.html&quot; rel=&quot;nofollow&quot;&gt;Stanford&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Supplement the texts with the various lecture video sites such as MIT OCW and videolectures.net.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Caltech doesn't have an undergrad degree in statistics, but you won't go wrong by following the curriculum of their undergrad stats courses. &lt;/p&gt;&#xA;" OwnerUserId="74" LastEditorUserId="74" LastEditorDisplayName="" LastEditDate="2011-02-22T00:26:09.117" LastActivityDate="2011-02-22T00:26:09.117" CommunityOwnedDate="2011-02-21T19:51:03.823" CommentCount="4" />
  <row Id="7466" PostTypeId="1" AcceptedAnswerId="7468" CreationDate="2011-02-21T20:09:05.493" Score="5" ViewCount="72" Body="&lt;p&gt;I want to cluster elements in array. The crucial difference from a normal clustering algorithm is that the order of elements is significant. For instance if we look at a simple sequence of numbers like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1.1, 1.2, 1.0, 3.3, 3.3, 2.9, 1.0, 1.1, 3.0, 2.8, 3.2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It is obvious that there are two clusters in there (1.1, 1.2, 1.0, 1.0, 1.1) and (3.3, 3.3, 2.9, 3.0, 2.8, 3.2). What I want is to find sequential groups of similar elements &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(1.1, 1.2, 1.0), (3.3, 3.3, 2.9), (1.0, 1.1), (3.0, 2.8, 3.2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;4 in this case. Of course I can run some variant of a normal clustering algorithm and then split clusters according elements' indices, but there's probably a simpler way to do this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any algorithm that I can use for this?&lt;/p&gt;&#xA;" OwnerUserId="255" LastActivityDate="2011-02-21T20:28:50.977" Title="Sequential clustering algorithm" Tags="&lt;clustering&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="1" />
  <row Id="7467" PostTypeId="1" AcceptedAnswerId="7472" CreationDate="2011-02-21T20:24:52.310" Score="7" ViewCount="127" Body="&lt;h2&gt;Background&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I am overseeing the input of data from primary literature into a &lt;a href=&quot;http://ebi-forecast.igb.uiuc.edu/documentation.html&quot; rel=&quot;nofollow&quot;&gt;database&lt;/a&gt;. The data entry process is error prone, particularly because users must interpret experimental design, extract data from graphics and tables, and transform results to standardized units.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data are input into a MySQL database through a web interface. Over 10k data points from &gt; 20 variables, &gt; 100 species, and &gt; 500 citations have been included so far. I need to run checks of the quality of not only the variable data, but also the data contained in lookup tables, such as the species associated with each data point, the location of the study, etc.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data entry is ongoing, so QA/QC will need to be run intermittently. The data have not yet been publicly released, but we are planning to release them in the next few months.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently, my QA/QC involves three steps: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;a second user checks each data point,&lt;/li&gt;&#xA;&lt;li&gt;visually inspect histogram each variable for outliers &lt;/li&gt;&#xA;&lt;li&gt;users report questionable data after spurious results are obtained&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;h2&gt;Questions&lt;/h2&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Are there guidelines that I can use for developing a robust QA/QC procedure for this database?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The first step is the most time consuming; is there anything that I can do to make this more efficient?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="1381" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-02-24T16:43:12.607" LastActivityDate="2011-02-24T16:43:12.607" Title="Quality assurance and quality control (QA/QC) guidelines for a database" Tags="&lt;meta-analysis&gt;&lt;dataset&gt;&lt;best-practices&gt;&lt;quality-control&gt;" AnswerCount="1" FavoriteCount="3" />
  <row Id="7468" PostTypeId="2" ParentId="7466" CreationDate="2011-02-21T20:28:50.977" Score="0" ViewCount="" Body="&lt;p&gt;Constrained clustering maintains data order.  There is a package in R called 'rioja' that implements this in the function 'chclust'.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The procedure isn't too complex though:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Calculate inter-point distance&lt;/li&gt;&#xA;&lt;li&gt;Find the smallest distance between adjacent points&lt;/li&gt;&#xA;&lt;li&gt;Average the value of the two points to generate a single value&lt;/li&gt;&#xA;&lt;li&gt;Spit the list out again and start from one until you have a single point.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;You need to maintain some sort of tree structure, but with some elementary programming experience you should be able to do it.&lt;/p&gt;&#xA;" OwnerUserId="3351" LastActivityDate="2011-02-21T20:28:50.977" CommentCount="2" />
  <row Id="7469" PostTypeId="2" ParentId="4805" CreationDate="2011-02-21T20:55:54.027" Score="2" ViewCount="" Body="&lt;p&gt;The classical transformations include the log, sqrt, and inverse (1/Y) transformations. More sophisticated transformations include the power transformation, from which the Box-Cox optimization chooses a particular transformation which optimized a log-likelihood. Which transformation to use is almost becoming a lost art form, but there is an excellent book by A. C. Atkinson (1985) called &lt;em&gt;Plots, Transformations, and Regression&lt;/em&gt; that talks about how to analyze your data and decide how to transform it. For example, the book discusses special transformations for data that are proportions. &lt;/p&gt;&#xA;" OwnerUserId="2773" LastActivityDate="2011-02-21T20:55:54.027" />
  <row Id="7470" PostTypeId="2" ParentId="7152" CreationDate="2011-02-21T21:01:10.717" Score="2" ViewCount="" Body="&lt;p&gt;A very nice discussion of structural zeros in contingency tables is provided by &#xA;West, L. and Hankin, R. (2008), “Exact Tests for Two-Way Contingency Tables with Structural Zeros,” Journal of Statistical Software, 28(11), 1–19.&#xA;URL &lt;a href=&quot;http://www.jstatsoft.org/v28/i11&quot; rel=&quot;nofollow&quot;&gt;http://www.jstatsoft.org/v28/i11&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As the title implies, they implement Fisher’s exact test for two-way contingency tables&#xA;in the case where some of the table entries are constrained to be zero.&lt;/p&gt;&#xA;" OwnerUserId="2773" LastActivityDate="2011-02-21T21:01:10.717" CommentCount="2" />
  <row Id="7471" PostTypeId="1" CreationDate="2011-02-21T22:39:49.407" Score="4" ViewCount="261" Body="&lt;p&gt;Can the standard deviation be calculated for the harmonic mean? I understand that the standard deviation can be calculated for arithmetic mean, but if you have harmonic mean, how do you calculate the standard deviation or CV?&lt;/p&gt;&#xA;" OwnerUserId="3353" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2011-02-22T14:37:00.030" LastActivityDate="2011-02-23T09:28:01.863" Title="Can the standard deviation be calculated for harmonic mean?" Tags="&lt;standard-deviation&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="7472" PostTypeId="2" ParentId="7467" CreationDate="2011-02-21T23:27:05.723" Score="11" ViewCount="" Body="&lt;p&gt;This response focuses on the second question, but in the process a partial answer to the first question (guidelines for a QA/QC procedure) will emerge.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By far the best thing you can do is check data quality at the time entry is attempted.  The user checks and reports are labor-intensive and so should be reserved for later in the process, as late as is practicable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are some principles, guidelines, and suggestions, derived from extensive experience (with the design and creation of many databases comparable to and much larger than yours).  They are not rules; you do not have to follow them to be successful and efficient; but they are all here for excellent reasons and you should think hard about deviating from them.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Separate data entry from all intellectually demanding activities&lt;/strong&gt;.  Do not ask data entry operators simultaneously to check anything, count anything, etc.  Restrict their work to creating a computer-readable facsimile of the data, nothing more.  In particular, this principle implies the data-entry forms should reflect the format in which you originally obtain the data, not the format in which you plan to store the data.  It is relatively easy to transform one format to another later, but it's an error-prone process to attempt the transformation on the fly while entering data.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Create a data audit trail&lt;/strong&gt;: whenever anything is done to the data, starting at the data entry stage, document this and record the procedure in a way that makes it easy to go back and check what went wrong (because things will go wrong).  Consider filling out fields for time stamps, identifiers of data entry operators, identifiers of sources for the original data (such as reports and their page numbers), etc.  Storage is cheap, but the time to track down an error is expensive.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Automate &lt;em&gt;everything.&lt;/em&gt;&lt;/strong&gt;  Assume any step will have to be redone (at the worst possible time, according to Murphy's Law), and plan accordingly.  Don't try to save time now by doing a few &quot;simple steps&quot; by hand.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;In particular, &lt;strong&gt;create support for data entry&lt;/strong&gt;: make a front end for each table (even a spreadsheet can do nicely) that provides a clear, simple, uniform way to get data in.  At the same time the front end should enforce your &quot;business rules:&quot; that is, it should perform as many simple validity checks as it can.  (E.g., pH must be between 0 and 14; counts must be positive.)  Ideally, use a DBMS to enforce relational integrity checks (e.g., every species associated with a measurement really exists in the database).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Constantly count things&lt;/strong&gt; and check that counts &lt;em&gt;exactly&lt;/em&gt; agree.  E.g., if a study is supposed to measure attributes of 10 species, make sure (as soon as data entry is complete) that 10 species really are reported.  Although checking counts is simple and uninformative, it's great at detecting duplicated and omitted data. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If the data are valuable and important, &lt;strong&gt;consider independently double-entering the entire dataset&lt;/strong&gt;.  This means that each item will be entered at separate times by two different non-interacting people.  This is a great way to catch typos, missing data, and so on.  The cross-checking can be completely automated.  This is faster, better at catching errors, and more efficient than 100% manual double checking.  (The data entry &quot;people&quot; can include devices such as scanners with OCR.)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Use a DBMS&lt;/strong&gt; to store and manage the data.  Spreadsheets are great for supporting data entry, but get your data out of the spreadsheets or text files and into a real database as soon as possible.  This prevents all kinds of insidious errors while adding lots of support for automatic data integrity checks.  If you must, use your statistical software for data storage and management, but seriously consider using a dedicated DBMS: it will do a better job.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;After all data are entered and automatically checked, &lt;strong&gt;draw pictures&lt;/strong&gt;: make sorted tables, histograms, scatterplots, etc., and look at them all.  These are easily automated with any full-fledged statistical package.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Do not ask people to do repetitive tasks that the computer can do&lt;/strong&gt;.  The computer is much faster and more reliable at these.  Get into the habit of writing (and documenting) little scripts and small programs to do any task that cannot be completed immediately.  These will become part of your audit trail and they will enable work to be redone easily.  Use whatever platform you're comfortable with and that is suitable to the task.  (Over the years, depending on what was available, I have used a wide range of such platforms and all have been effective in their way, ranging from C and Fortran programs through AWK and SED scripts, VBA scripts for Excel and Word, and custom programs written for relational database systems, GIS, and statistical analysis platforms like R and Stata.)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;If you follow most of these guidelines, approximately 50%-80% of the work in getting data into the database will be database design and writing the supporting scripts.  It is not unusual to get 90% through such a project and be less than 50% complete, yet still finish on time: once everything is set up and has been tested, the data entry and checking can be amazingly efficient.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-02-21T23:27:05.723" CommentCount="4" />
  <row Id="7473" PostTypeId="2" ParentId="7455" CreationDate="2011-02-21T23:50:01.567" Score="2" ViewCount="" Body="&lt;p&gt;In machine learning a full probability model p(x,y) is called generative because it can be used to generate the data whereas a conditional model p(y|x) is called discriminative because it does not specify a probability model for p(x) and can only generate y given x.  Both can be estimated in Bayesian fashion. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bayesian estimation is inherently about specifying a full probability model and performing inference conditional on the model and data.  That makes many Bayesian models have a generative feel.  However to a Bayesian the important distinction is not so much about how to generate the data, but more about what is needed to obtain the posterior distribution of the unknown parameters of interest.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The discriminative model p(y|x) is part of bigger model where p(y, x) = p(y|x)p(x). In many instances, p(x) is irrelevant to the posterior distribution of the parameters in the model p(y|x).  Specifically, if the parameters of p(x) are distinct from p(y|x) and the priors are independent, then the model p(x) contains no information about the unknown parameters of the conditional model p(y|x), so a Bayesian does not need to model it.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;At a more intuitive level, there is a clear link between &quot;generating data&quot; and &quot;computing the posterior distribution.&quot; Rubin (1984) gives the following excellent description of this link:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/hhYZ8.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Bayesian statistics is useful given missing data primarily because it provides a unified way to eliminate nuisance parameters -- integration.  Missing data can be thought of as (many) nuisance parameters.  Alternative proposals such as plugging in the expected value typically will perform poorly because we can rarely estimate missing data cells with high levels of accuracy. Here, integration is better than maximization.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Discriminative models like p(y|x) also become problematic if x includes missing data because we only have data to estimate p(y|x_obs) but most sensible models are written with respect to the complete data p(y|x).  If you have a fully probability model p(y,x) and are Bayesian,  then you're fine because you can just integrate over the missing data like you would any other unknown quantity.&lt;/p&gt;&#xA;" OwnerUserId="493" LastEditorUserId="493" LastEditorDisplayName="" LastEditDate="2011-02-22T00:06:57.613" LastActivityDate="2011-02-22T00:06:57.613" />
  <row Id="7474" PostTypeId="2" ParentId="7471" CreationDate="2011-02-21T23:51:06.993" Score="1" ViewCount="" Body="&lt;p&gt;Here is an example for Exponential r.v's.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The harmonic mean for $n$ data points is defined as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$S = \frac{1}{\frac{1}{n} \sum_{i=1}^n X_i}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose you have n iid samples of an Exponential random variable,  $X_i \sim {\rm Exp}(\lambda)$. The sum of $n$ Exponential variables follows a Gamma distribution &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\sum_{i=1}^n X_i \sim {\rm Gamma}(n, \theta)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\theta = \frac{1}{\lambda}$. We also know that&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{1}{n} {\rm Gamma}(n, \theta) \sim {\rm Gamma}(n, \frac{\theta}{n})$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The distribution of $S$ is therefore&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$S \sim {\rm InvGamma}(n, \frac{n}{\theta})$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The variance (and standard deviation) of this r.v. are well known, see, for example &lt;a href=&quot;http://en.wikipedia.org/wiki/Inverse-gamma_distribution&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="530" LastEditorUserId="530" LastEditorDisplayName="" LastEditDate="2011-02-22T05:48:10.370" LastActivityDate="2011-02-22T05:48:10.370" CommentCount="2" />
  <row Id="7475" PostTypeId="1" AcceptedAnswerId="7479" CreationDate="2011-02-22T01:09:31.170" Score="1" ViewCount="177" Body="&lt;p&gt;I am trying to do a multiple logistic regression for 2 similar groups. I have a few questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;In doing a univariate analysis, do I enter each independent variable, one at a time, first into the binary regression, before going on to do the multivariate analysis? Or is the significance values from Chi-square or t-test enough to go on with?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I have a test group and a control group and I want to determine the effect of independent variables (e.g HIV status, maternal weight etc) on a particular dependent variable (low birth weight).  Should I perform the regression on a dataset with both the test and the control cases or should I split the file? In this case I want to see the effect of HIV on birthweight and I am having a hard time knowing how to move on.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="3355" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-02-22T06:57:06.070" LastActivityDate="2011-02-22T06:57:06.070" Title="Entering variables in multivariate logistic regression and running regression across two groups" Tags="&lt;logistic-regression&gt;" AnswerCount="1" CommentCount="0" />
  <row Id="7476" PostTypeId="1" AcceptedAnswerId="7493" CreationDate="2011-02-22T01:12:00.607" Score="5" ViewCount="148" Body="&lt;p&gt;Before submission of my meta-analysis I want to make a funnel plot to test for heterogeneity and publication bias. I have the pooled effect size and the effect sizes from each study, that take values from -1 to +1. I have the sample sizes n1, n2 for patients and controls from each study. As I cannot calculate the standard error (SE), I cannot perform Egger's regression. I cannot use SE or precision=1/SE on the vertical axis. &lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Questions&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Can I still make a funnel plot with effect size on the horizontal axon and total sample size n (n=n1+n2) on the vertical axis? &lt;/li&gt;&#xA;&lt;li&gt;How should such a funnel plot be interpreted?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Some published papers presented such funnel plot with total sample size on the vertical axis (Pubmed PMIDs: 10990474, 10456970). Also, wikipedia funnel plot wiki agree on this. But, most importantly, Mathhias Egger's paper on BMJ 1999 (PubMed PMID: 9451274) shows such a funnel plot, with no SE but only sample size on the vertical axis. &lt;/p&gt;&#xA;&#xA;&lt;h3&gt;More Questions&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Is such a plot acceptable when the standard error is not known? &lt;/li&gt;&#xA;&lt;li&gt;Is it the same as the classical funnel plot with SE or presicion=1/SE on the vertical axon? &lt;/li&gt;&#xA;&lt;li&gt;Is its interpretation different? &lt;/li&gt;&#xA;&lt;li&gt;How should I  set the lines to make the equilateral triangle? &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="3333" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2011-02-22T12:02:14.783" LastActivityDate="2011-02-22T12:08:20.327" Title="Alternative funnel plot, without using standard error (SE)" Tags="&lt;meta-analysis&gt;&lt;sample-size&gt;&lt;standard-error&gt;&lt;funnel-plot&gt;&lt;publication-bias&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="1" />
  <row Id="7477" PostTypeId="2" ParentId="6538" CreationDate="2011-02-22T02:08:30.283" Score="11" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;(Very) short story&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Long story short, in some sense, statistics is like any other technical field: &lt;strong&gt;&lt;a href=&quot;http://norvig.com/21-days.html&quot; rel=&quot;nofollow&quot;&gt;There is no fast track&lt;/a&gt;&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Long story&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bachelor's degree programs in statistics are relatively rare in the U.S. One reason I believe this is true is that it is quite hard to pack all that is necessary to learn statistics well into an undergraduate curriculum. This holds particularly true at universities that have significant general-education requirements.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Developing the necessary skills: mathematical, computational, and intuitive takes a lot of effort and time. Statistics can begin to be understood at a fairly decent &quot;operational&quot; level once the student has mastered calculus and a decent amount of linear and matrix algebra. However, any applied statistician knows that it is quite easy to find oneself in territory that doesn't conform to a cookie-cutter or recipe-based approach to statistics. To really understand what is going on beneath the surface requires as a &lt;em&gt;prerequisite&lt;/em&gt; mathematical and, in today's world, computational maturity that are only really attainable in the later years of undergraduate training. This is one reason that true statistical training mostly starts at the M.S. level in the U.S. (India, with their dedicated ISI is a little different story. A similar argument might be made for some Canadian-based education. I'm not familiar enough with European-based or Russian-based undergraduate statistics education to have an informed opinion.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Nearly any (interesting) job would require an M.S. level education and the &lt;em&gt;really&lt;/em&gt; interesting (in my opinion) jobs essentially require a doctorate-level education. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Seeing as you have a doctorate in mathematics, though we don't know in what area, here are my suggestions for something closer to an M.S.-level education. I include some parenthetical remarks to explain the choices.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;D. Huff, &lt;em&gt;How to Lie with Statistics&lt;/em&gt;. (Very quick, easy read. Shows many of the conceptual ideas and pitfalls, in particular, in presenting statistics to the layman.)&lt;/li&gt;&#xA;&lt;li&gt;Mood, Graybill, and Boes, &lt;em&gt;Introduction to the Theory of Statistics&lt;/em&gt;, 3rd ed., 1974. (M.S.-level intro to theoretical statistics. You'll learn about sampling distributions, point estimation and hypothesis testing in a classical, frequentist framework. My opinion is that this is generally better, and a bit more advanced, than modern counterparts such as Casella &amp;amp; Berger or Rice.)&lt;/li&gt;&#xA;&lt;li&gt;Seber &amp;amp; Lee, &lt;em&gt;Linear Regression Analysis&lt;/em&gt;, 2nd ed. (Lays out the theory behind point estimation and hypothesis testing for linear models, which is probably the most important topic to understand in applied statistics. Since you probably have a good linear algebra background, you should immediately be able to understand what is going on geometrically, which provides a lot of intuition. Also has good information related to assessment issues in model selection, departures from assumptions, prediction, and robust versions of linear models.)&lt;/li&gt;&#xA;&lt;li&gt;Hastie, Tibshirani, and Friedman, &lt;em&gt;Elements of Statistical Learning&lt;/em&gt;, 2nd ed., 2009. (This book has a much more applied feeling than the last and broadly covers lots of modern machine-learning topics. The major contribution here is in providing statistical interpretations of many machine-learning ideas, which pays off particularly in quantifying uncertainty in such models. This is something that tends to go un(der)addressed in typical machine-learning books. Legally available for free &lt;strong&gt;&lt;a href=&quot;http://www-stat.stanford.edu/~tibs/ElemStatLearn/&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;&lt;/strong&gt;.)&lt;/li&gt;&#xA;&lt;li&gt;A. Agresti, &lt;em&gt;Categorical Data Analysis&lt;/em&gt;, 2nd ed. (Good presentation of how to deal with discrete data in a statistical framework. Good theory and good practical examples. Perhaps on the traditional side in some respects.)&lt;/li&gt;&#xA;&lt;li&gt;Boyd &amp;amp; Vandenberghe, &lt;em&gt;Convex Optimization&lt;/em&gt;. (Many of the most popular modern statistical estimation and hypothesis-testing problems can be formulated as convex optimization problems. This also goes for numerous machine-learning techniques, e.g., SVMs. Having a broader understanding and the ability to recognize such problems as convex programs is quite valuable, I think. Legally available for free &lt;strong&gt;&lt;a href=&quot;http://www.stanford.edu/~boyd/cvxbook/&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;&lt;/strong&gt;.)&lt;/li&gt;&#xA;&lt;li&gt;Efron &amp;amp; Tibshirani, &lt;em&gt;An Introduction to the Bootstrap&lt;/em&gt;. (You ought to at least be familiar with the bootstrap and related techniques. For a textbook, it's a quick and easy read.)&lt;/li&gt;&#xA;&lt;li&gt;J. Liu, &lt;em&gt;Monte Carlo Strategies in Scientific Computing&lt;/em&gt; or P. Glasserman, &lt;em&gt;Monte Carlo Methods in Financial Engineering&lt;/em&gt;. (The latter sounds very directed to a particular application area, but I think it'll give a good overview and practical examples of all the most important techniques. Financial engineering applications have driven a fair amount of Monte Carlo research over the last decade or so.)&lt;/li&gt;&#xA;&lt;li&gt;E. Tufte, &lt;em&gt;The Visual Display of Quantitative Information&lt;/em&gt;. (Good visualization and presentation of data is [highly] underrated, even by statisticians.)&lt;/li&gt;&#xA;&lt;li&gt;J. Tukey, &lt;em&gt;Exploratory Data Analysis&lt;/em&gt;. (Standard. Oldie, but goodie. Some might say outdated, but still worth having a look at.)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Complements&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are some other books, mostly of a little more advanced, theoretical and/or auxiliary nature, that are helpful.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;F. A. Graybill, &lt;em&gt;Theory and Application of the Linear Model&lt;/em&gt;. (Old fashioned, terrible typesetting, but covers all the same ground of Seber &amp;amp; Lee, and more. I say old-fashioned because more modern treatments would probably tend to use the SVD to unify and simplify a lot of the techniques and proofs.)&lt;/li&gt;&#xA;&lt;li&gt;F. A. Graybill, &lt;em&gt;Matrices with Applications in Statistics&lt;/em&gt;. (Companion text to the above. A wealth of good matrix algebra results useful to statistics here. Great desk reference.)&lt;/li&gt;&#xA;&lt;li&gt;Devroye, Gyorfi, and Lugosi, &lt;em&gt;A Probabilistic Theory of Pattern Recognition&lt;/em&gt;. (Rigorous and theoretical text on quantifying performance in classification problems.)&lt;/li&gt;&#xA;&lt;li&gt;Brockwell &amp;amp; Davis, &lt;em&gt;Time Series: Theory and Methods&lt;/em&gt;. (Classical time-series analysis. Theoretical treatment. For more applied ones, Box, Jenkins &amp;amp; Reinsel or Ruey Tsay's texts are decent.)&lt;/li&gt;&#xA;&lt;li&gt;Motwani and Raghavan, &lt;em&gt;Randomized Algorithms&lt;/em&gt;. (Probabilistic methods and analysis for computational algorithms.)&lt;/li&gt;&#xA;&lt;li&gt;D. Williams, &lt;em&gt;Probability and Martingales&lt;/em&gt; and/or R. Durrett, &lt;em&gt;Probability: Theory and Examples&lt;/em&gt;. (In case you've seen measure theory, say, at the level of D. L. Cohn, but maybe not probability theory. Both are good for getting quickly up to speed if you already know measure theory.)&lt;/li&gt;&#xA;&lt;li&gt;F. Harrell, &lt;em&gt;Regression Modeling Strategies&lt;/em&gt;. (Not as good as &lt;em&gt;Elements of Statistical Learning&lt;/em&gt; [ESL], but has a different, and interesting, take on things. Covers more &quot;traditional&quot; applied statistics topics than does ESL and so worth knowing about, for sure.)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;More Advanced (Doctorate-Level) Texts&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Lehman and Casella, &lt;em&gt;Theory of Point Estimation&lt;/em&gt;. (PhD-level treatment of point estimation. Part of the challenge of this book is reading it and figuring out what is a typo and what is not. When you see yourself recognizing them quickly, you'll know you understand. There's plenty of practice of this type in there, especially if you dive into the problems.)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Lehmann and Romano, &lt;em&gt;Testing Statistical Hypotheses&lt;/em&gt;. (PhD-level treatment of hypothesis testing. Not as many typos as TPE above.)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;A. van der Vaart, &lt;em&gt;Asymptotic Statistics&lt;/em&gt;. (A beautiful book on the asymptotic theory of statistics with good hints on application areas. Not an applied book though. My only quibble is that some rather bizarre notation is used and details are at times brushed under the rug.)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="2970" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-02-26T17:19:06.130" LastActivityDate="2011-02-26T17:19:06.130" CommunityOwnedDate="2011-02-22T02:08:30.283" CommentCount="18" />
  <row Id="7478" PostTypeId="1" CreationDate="2011-02-22T02:32:12.893" Score="3" ViewCount="81" Body="&lt;p&gt;I've performed a study which yielded (?) the following results:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;-                   no bike box      bike box     % change&#xA;correct procedure      173              55           -27%&#xA;incorrect procedure    68               50           69%&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Since a result could only be one of the two - correct and incorrect procedure, do I need both quantities in my data, or should I just interpret one of them? If so, which one should I use, or does it depend what i'm trying to demonstrate?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sorry for being so vague, but I hope this is enough to answer my question.&lt;/p&gt;&#xA;" OwnerUserId="3357" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-02-22T07:47:37.857" LastActivityDate="2011-02-22T23:27:11.683" Title="How to compare outcomes from single variable experiment?" Tags="&lt;basic-concepts&gt;" AnswerCount="3" />
  <row Id="7479" PostTypeId="2" ParentId="7475" CreationDate="2011-02-22T02:34:30.143" Score="4" ViewCount="" Body="&lt;p&gt;I would start with estimating a (simple) bivariate correlation matrix which includes your outcome variable as well as all predictors. This will give you very first insights into the dependency structure of all your variables. Especially correlation coefficients of $|r| &amp;gt; 0.4$ (between your predictor variables) can indicate later multicollinearity problems. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Next, I would continue with a series of bivariate regressions. That is, for each predictor ('independent variable') run one logistic regression. One of these regressions will focus on your control-/treatment-group variable. This will inform you about the 'gross' effects, that is the unadjusted effects. Please, do not think that statistically non-significant effects could be excluded from later analysis. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you assume an interaction effect between HIV and your treatment-variable (1=treatment, 0=control)? Then you could run two separate models with HIV as predictor variable, i.e. one model for control- and one model for treatment-group. Given that you observe different coefficients for HIV, you will need to run another model which includes an interaction effect $HIV \times treatment$ (and the two main effects). This second model will allow you to test for statistically significant differences between the groups. You also might include your other predictor variables. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please be aware that interaction effects in logistic regression models are more complicated than in the case of a simple linear regression model. Edward Norton has published &lt;a href=&quot;http://www.unc.edu/~enorton/&quot; rel=&quot;nofollow&quot;&gt;several papers&lt;/a&gt; which discuss that topic. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You also did not tell us what software you are using to estimate the models. However, most software packages are able to test for multicollinearity (VIF or tolerance).&lt;/p&gt;&#xA;" OwnerUserId="307" LastActivityDate="2011-02-22T02:34:30.143" CommentCount="4" />
  <row Id="7480" PostTypeId="2" ParentId="7478" CreationDate="2011-02-22T03:03:53.333" Score="1" ViewCount="" Body="&lt;p&gt;You can report percent correct along with sample size $n$, and reporting percent correct instead would be sufficient in most cases, even if you focus more on the percent incorrect in your interpretation.&lt;/p&gt;&#xA;" OwnerUserId="1381" LastActivityDate="2011-02-22T03:03:53.333" />
  <row Id="7481" PostTypeId="1" CreationDate="2011-02-22T03:24:55.477" Score="5" ViewCount="209" Body="&lt;h3&gt;Context&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;I have a survey that asks 11 questions about self-efficacy.&#xA;Each question has 3 response options (disagree, agree, strongly agree).&#xA;Nine questions ask about self-esteem. &#xA;I have used a factor analysis of the 11 self-efficacy items and extracted two factors. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$x_1$ to $x_{11}$ denote the 11 self-efficacy questions in the survey, and $f_1$ ($x_1$ to $x_6$) , $f_2$ ($x_7$ to $x_{11}$) denote the two factors I got from the factor analysis. &#xA;$y$ is a Dependent variable. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then I created two new variables: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   f1=mean(x1 to x6); &#xA;   f2=mean(x7-x11). &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So the logistic regression would looks like this: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   y=a+bf1+cf2+....&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h3&gt;My question:&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Can i use these two factors as predictor variables in my multivariate logistic regression model? &lt;/li&gt;&#xA;&lt;li&gt;Should I calculate the mean of each items in each factor and use this mean as a continuous variable in my logistic regression model?&lt;/li&gt;&#xA;&lt;li&gt;Is this an appropriate use of factor analysis?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="3358" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-02-22T07:46:48.870" LastActivityDate="2011-02-22T18:06:08.293" Title="How to use variables derived from factor analysis as predictors in logistic regression?" Tags="&lt;logistic-regression&gt;&lt;factor-analysis&gt;" AnswerCount="4" CommentCount="1" />
  <row Id="7482" PostTypeId="1" CreationDate="2011-02-22T06:56:36.897" Score="3" ViewCount="91" Body="&lt;p&gt;If kNN doesn't perform well for classification on a dataset, is there any hope for parametric methods to perform better? Kernel-based methods, SVM, random forests, and neural networks. Could any of these outperform kNN method?&lt;/p&gt;&#xA;" OwnerUserId="3359" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-02-23T21:39:34.423" LastActivityDate="2011-02-23T21:39:34.423" Title="Accuracy of advanced parametric methods compared to kNN method" Tags="&lt;machine-learning&gt;&lt;k-nearest-neighbour&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="7483" PostTypeId="2" ParentId="7482" CreationDate="2011-02-22T07:02:04.770" Score="4" ViewCount="" Body="&lt;p&gt;Hastie et al give a nice overview in &lt;a href=&quot;http://www-stat.stanford.edu/~tibs/ElemStatLearn/&quot; rel=&quot;nofollow&quot;&gt;their book&lt;/a&gt;, look into 2nd chapter. The short answer is yes. Otherwise why do you think these methods were developed and are still widely used?&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-02-22T07:02:04.770" CommentCount="6" />
  <row Id="7484" PostTypeId="2" ParentId="7471" CreationDate="2011-02-22T07:43:52.837" Score="4" ViewCount="" Body="&lt;p&gt;The harmonic mean $H$ of random variables $X_1,...,X_n$ is defined as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$H=\frac{1}{\frac{1}{n}\sum_{i=1}^n\frac{1}{X_i}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Taking moments of fractions is a messy business, so instead I would prefer working with the $1/H$. Now&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{1}{H}=\frac{1}{n}\sum_{i=1}^n\frac{1}{X_i}$$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Usin central limit theorem we immediately get that&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\sqrt{n}\left(H^{-1}-EX_1^{-1}\right)\to N(0,VarX_1^{-1})$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;if of course $VarX_1^{-1}&amp;lt;\infty$ and $X_i$ are iid, since we simple work with arithmetic mean of variables $Y_i=X_i^{-1}$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now using delta method for function $g(x)=x^{-1}$ we get that&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\sqrt{n}(H-(EX_1^{-1})^{-1})\to N\left(0, \frac{VarX_1^{-1}}{(EX_1^{-1})^4}\right)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This result is asymptotic, but for simple applications it might suffice.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt; As @whuber rightfully points out, simple applications is a misnomer. The central limit theorem holds only if $VarX_1^{-1} exists, which is quite a restrictive assumption. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update 2&lt;/strong&gt; If you have a sample, then to calculate the standard deviation, simply plug  sample moments into the formula. So for sample $X_1,...,X_n$, the estimate of harmonic mean is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align}&#xA;\hat{H}=\frac{1}{\frac{1}{n}\sum_{i=1}^n\frac{1}{X_i}}&#xA;\end{align}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;the sample moments $EX_1^{-1}$ and $Var(X_1^{-1})$ respectively are:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align}&#xA;\hat{\mu}&lt;em&gt;{R}&amp;amp;=\frac{1}{n}\sum&lt;/em&gt;{i=1}^n\frac{1}{X_i}\\&#xA;\hat{\sigma}&lt;em&gt;{R}^2&amp;amp;=\frac{1}{n}\sum&lt;/em&gt;{i=1}^n\left(\frac{1}{X_i}-\mu_R\right)^2&#xA;\end{align}&#xA;here $R$ stands for reciprocal. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally the approximate formula for standard deviation of $\hat{H}$ is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align*}&#xA;sd(\hat{H})=\sqrt{\frac{\hat{\sigma}_R^2}{n\hat{\mu}_R^4}}&#xA;\end{align*}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I ran some Monte-Carlo simulations for random variables uniformly distributed in interval $[2,3]$. Here is the code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;hm &amp;lt;- function(x)1/mean(1/x)&#xA;sdhm &amp;lt;- function(x)sqrt((mean(1/x))^(-4)*var(1/x)/length(x))&#xA;&#xA;n&amp;lt;-1000&#xA;&#xA;nn &amp;lt;- c(10,30,50,100,500,1000,5000,10000)&#xA;&#xA;N&amp;lt;-1000&#xA;&#xA;mc&amp;lt;-foreach(n=nn,.combine=rbind) %do% {&#xA;&#xA;    rr &amp;lt;- matrix(runif(n*N,min=2,max=3),nrow=N)&#xA;&#xA;    c(n,mean(apply(rr,1,sdhm)),sd(apply(rr,1,sdhm)),sd(apply(rr,1,hm)))&#xA;&#xA;}&#xA;colnames(mc) &amp;lt;- c(&quot;n&quot;,&quot;DeltaSD&quot;,&quot;sdDeltaSD&quot;,&quot;trueSD&quot;)&#xA;&#xA;&amp;gt; mc&#xA;             n     DeltaSD    sdDeltaSD      trueSD&#xA;result.1    10 0.089879211 1.528423e-02 0.091677622&#xA;result.2    30 0.052870477 4.629262e-03 0.051738941&#xA;result.3    50 0.040915607 2.705137e-03 0.040257673&#xA;result.4   100 0.029017031 1.407511e-03 0.028284458&#xA;result.5   500 0.012959582 2.750145e-04 0.013200580&#xA;result.6  1000 0.009139193 1.357630e-04 0.009115592&#xA;result.7  5000 0.004094048 2.685633e-05 0.004070593&#xA;result.8 10000 0.002894254 1.339128e-05 0.002964259&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I simulated &lt;code&gt;N&lt;/code&gt; samples of &lt;code&gt;n&lt;/code&gt; sized sample. For  each &lt;code&gt;n&lt;/code&gt; sized sample I calculated estimate of standard estimation (function &lt;code&gt;sdhm&lt;/code&gt;). Then I compare the mean and standard deviation of these estimates with the sample standard deviation of harmonic mean estimated for each sample, which supposably should be the true standard deviation of harmonic mean. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you can see the results are quite good even for moderate sample sizes. Of course uniform distribution is a very well behaved one, so it is not surprising that results are good. I'll leave for someone else to investigate the behaviour for other distributions, the code is very easy to adapt.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Note:&lt;/em&gt; In previous version of this answer there was an error in the result of delta method, incorrect variance. &lt;/p&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-02-23T09:28:01.863" LastActivityDate="2011-02-23T09:28:01.863" CommentCount="8" />
  <row Id="7485" PostTypeId="1" CreationDate="2011-02-22T08:53:18.667" Score="0" ViewCount="65" Body="&lt;p&gt;Hello !&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can you help me to complete the following argument about the identifiability of the random effect $v$ in an additive model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In an additive model such as $X\beta + v$ the location $v$ is unidentifiable since&#xA;$X\beta + v = (X\beta + a) + (v-a)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt; (following mpiktas's comment)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does the following make sense to you?&#xA;Let us assume $v\sim N(\mu, \sigma^2)$. In the previous model, $\mu$ cannot be identified because we can always add some $a$ to the intercept and subtract it to $\mu$, and this does not change the likelihood. In other words, one needs to impose some constraint, e.g., $E(v)=\mu=0$.&lt;/p&gt;&#xA;" OwnerUserId="3019" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-02-22T09:24:27.597" LastActivityDate="2011-02-22T09:24:27.597" Title="Identifiability of a random effect in an additive model" Tags="&lt;random-effects-model&gt;" CommentCount="8" />
  <row Id="7487" PostTypeId="1" CreationDate="2011-02-22T09:12:10.840" Score="1" ViewCount="267" Body="&lt;p&gt;How can I test the statistical significance of regression coefficients in multivariate multiple regression?&lt;/p&gt;&#xA;" OwnerUserId="3361" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-02-22T09:23:48.610" LastActivityDate="2011-02-22T14:37:56.237" Title="How to test for statistical significance of regression coefficients in multivariate multiple regression?" Tags="&lt;regression&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="7488" PostTypeId="2" ParentId="7478" CreationDate="2011-02-22T10:35:29.137" Score="1" ViewCount="" Body="&lt;p&gt;If I understand correctly, your IV is &quot;bike box&quot; vs. &quot;no bike box&quot;, and your DV is &quot;correct&quot; vs. &quot;incorrrect&quot;. The resulting $2 \times 2$ classification table can be summarized with the Odds Ratio: &quot;given the bike-box condition, what are the odds of getting a correct response?&quot; compared to &quot;given the no-bike-box condition, what are the odds of getting a correct response?&quot; If the odds are identical, OR is 1. Yule's Q standardizes OR to the $[-1, 1]$ interval. In R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; IV    &amp;lt;- factor(rep(c(&quot;no bbox&quot;, &quot;bbox&quot;), c(241, 105)))&#xA;&amp;gt; DVnbb &amp;lt;- rep(c(&quot;correct&quot;, &quot;incorrect&quot;),   c(173,  68))&#xA;&amp;gt; DVbb  &amp;lt;- rep(c(&quot;correct&quot;, &quot;incorrect&quot;),   c( 55,  50))&#xA;&amp;gt; DV    &amp;lt;- factor(c(DVnbb, DVbb))&#xA;&amp;gt; cTab  &amp;lt;- table(IV, DV)&#xA;&amp;gt; addmargins(cTab)&#xA;         DV&#xA;IV      correct incorrect Sum&#xA;bbox         55        50 105&#xA;no bbox     173        68 241&#xA;Sum         228       118 346&#xA;&#xA;&amp;gt; library(vcd)                   # for oddsratio()&#xA;&amp;gt; (OR &amp;lt;- oddsratio(cTab, log=FALSE))&#xA;[1] 0.4323699&#xA;&#xA;&amp;gt; (55/50) / (173/68)             # check: ratio of odds&#xA;[1] 0.4323699&#xA;&#xA;&amp;gt; (Q &amp;lt;- (OR-1) / (OR+1))         # Yule's Q&#xA;[1] -0.3962873&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;A corresponding test for equal distributions of your DV within IV groups is Fisher's test.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; fisher.test(cTab)&#xA;Fisher's Exact Test for Count Data&#xA;data:  cTab &#xA;p-value = 0.0008111&#xA;alternative hypothesis: true odds ratio is not equal to 1 &#xA;95 percent confidence interval:&#xA;0.2619504 0.7158848 &#xA;sample estimates:&#xA;odds ratio &#xA;0.4334897&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Note that &lt;code&gt;fisher.test()&lt;/code&gt; does not report the empirical OR, but a maximum-likelihood estimation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: Reading your answer, another measure that might capture some relevant information is relative risk: its definition is very similar to OR but calculates the &quot;risk&quot; of getting a correct response given one of the two conditions (and not the odds), i.e., the conditional relative frequency of a correct response.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# risk of getting correct (1st column) response in the two conditions&#xA;# calculated as conditional frequency: (cell count) / (sum of row counts)&#xA;&amp;gt; (risk &amp;lt;- prop.table(cTab, margin=1))&#xA;              DV&#xA;IV        correct incorrect&#xA;bbox    0.5238095 0.4761905&#xA;no bbox 0.7178423 0.2821577&#xA;&#xA;# compare risk in experimental condition to risk in control condition&#xA;&amp;gt; (relRisk &amp;lt;- risk[1, 1] / risk[2, 1])&#xA;0.7297&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1909" LastEditorUserId="1909" LastEditorDisplayName="" LastEditDate="2011-02-22T23:27:11.683" LastActivityDate="2011-02-22T23:27:11.683" />
  <row Id="7489" PostTypeId="2" ParentId="7487" CreationDate="2011-02-22T11:15:15.733" Score="4" ViewCount="" Body="&lt;p&gt;For a start, have a look at this pdf: &lt;a href=&quot;http://www.psych.yorku.ca/lab/psy6140/lectures/MultivariateRegression2x2.pdf&quot; rel=&quot;nofollow&quot;&gt;multivariate multiple regression&lt;/a&gt;. An example in R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;N  &amp;lt;- 50                   # number of participants&#xA;X1 &amp;lt;- rnorm(N, 175,  7)    # predictor 1&#xA;X2 &amp;lt;- rnorm(N,  30,  8)    # predictor 2&#xA;X3 &amp;lt;- rnorm(N,  60, 30)    # predictor 3&#xA;Y1 &amp;lt;-  0.2*X1 - 0.3*X2 - 0.2*X3 + rnorm(N, 0, 50)  # predicted variable 1&#xA;Y2 &amp;lt;- -0.1*X1 + 0.2*X2 + rnorm(N, 50)              # predicted variable 2&#xA;Y  &amp;lt;- cbind(Y1, Y2)        # predicted variables in multivariate form&#xA;&#xA;# fit OLS regression, coefficients are identical to the two separate univariate fits&#xA;(lmFit &amp;lt;- lm(Y ~ X1 + X2 + X3))&#xA;&#xA;# fit MANOVA and do several multivariate tests&#xA;manFit &amp;lt;- manova(lmFit)&#xA;summary(manFit, test=&quot;Hotelling-Lawley&quot;) # Hotelling-Lawley trace&#xA;summary(manFit, test=&quot;Pillai&quot;)           # Pillai-Bartlett trace&#xA;summary(manFit, test=&quot;Roy&quot;)              # Roy's largest root&#xA;summary(manFit, test=&quot;Wilks&quot;)            # Wilks' lambda&#xA;&#xA;# compare to separate univariate regression analyses: different p-values&#xA;summary(lm(Y1 ~ X1 + X2 + X3))&#xA;summary(lm(Y2 ~ X1 + X2 + X3))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1909" LastActivityDate="2011-02-22T11:15:15.733" />
  <row Id="7490" PostTypeId="2" ParentId="7481" CreationDate="2011-02-22T11:23:45.177" Score="6" ViewCount="" Body="&lt;p&gt;If I understand you correctly, you are using FA to extract two subscales from your 11-item questionnaire. They are supposed to reflect some specific dimensions of self-efficacy (for example, self-regulatory vs. self-assertive efficacy).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, you are free to use &lt;em&gt;individual&lt;/em&gt; mean (or sum) scores computed on the two subscales as predictors in a regression model. In others words, instead of considering 11 item scores, you are now working with 2 subscores, computed as described above for each individual. The only assumption that is made is that those scores reflect one's location on an &quot;hypothetical construct&quot; or latent variable, defined as a continuous scale.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As @JMS said, there are other issues that you might further clarify, especially which kind of FA was done. A subtle issue is that measurement error will not be accounted for by a standard regression approach. An alternative is to use &lt;a href=&quot;http://en.wikipedia.org/wiki/Structural_equation_modeling&quot; rel=&quot;nofollow&quot;&gt;Structural Equation Models&lt;/a&gt; or any latent variables model (e.g. those coming from the &lt;a href=&quot;http://en.wikipedia.org/wiki/Item_response_theory&quot; rel=&quot;nofollow&quot;&gt;IRT&lt;/a&gt; literature), but here the regression approach should provide a good approximation. The analysis of ordinal variables (Likert-type item) has been discussed elsewhere on this site. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, in current practice, your approach is what is commonly found when validating a questionnaire or constructing scoring rules: We use weighted or unweighted combination of item scores (hence, they are treated as numeric variables) to report individual location on the latent trait(s) under consideration.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-02-22T11:23:45.177" />
  <row Id="7491" PostTypeId="2" ParentId="7481" CreationDate="2011-02-22T11:24:18.743" Score="4" ViewCount="" Body="&lt;h3&gt;Using factor scores as predictors&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Yes, you can use variables derived from a factor analysis as predictors in subsequent analyses. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other options include running some form of structural equation model where you posit a latent variable with the items or bundles of items as observed variables.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Mean as scale score&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Yes, in your case, the mean would be a typical option for computing a scale score.&#xA;If you have any reversed items, you have to deal with this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could also use factor saved scores instead of taking the mean. Although when all items load reasonably well on each factor and all items are on the same scale and all items are positively worded, there is rarely much difference between the mean and factor saved scores.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could also look at methods that acknowledge the ordinal nature of the scale and therefore do not treat the scale options as equally distant.&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2011-02-22T11:24:18.743" CommentCount="1" />
  <row Id="7492" PostTypeId="2" ParentId="7481" CreationDate="2011-02-22T11:37:13.330" Score="1" ViewCount="" Body="&lt;p&gt;Everything have be said by chl and Jeromy for the theorical part... If you don't have use sum/mean of variables you identify with FA you can use scores of FA.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regarding the syntax you use you're probably using SAS. So to do a correct use of factor analysis you must use the score of observations and not the mean of variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You find below the code to obtain score for 2 factors with an FA. Scores you'll have to use will be call Factor1, Factor2, ... by SAS. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a 2 steps... 1) First FA then 2) call the proc score to compute Scores.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;proc factor &#xA;    data = Data&#xA;    method = ml &#xA;    rotate = promax &#xA;    outstat = FAstats&#xA;    n=3&#xA;    heywood residuals msa score&#xA;    ;&#xA;    var x:;&#xA;run;&#xA;&#xA;proc score data=Data score=FAstats out=MyScores; &#xA;    var x:;&#xA;run; &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The variables to use are Factor1, Factor2, ... in MyScores datasets.&lt;/p&gt;&#xA;" OwnerUserId="1154" LastEditorUserId="1154" LastEditorDisplayName="" LastEditDate="2011-02-22T12:55:24.417" LastActivityDate="2011-02-22T12:55:24.417" />
  <row Id="7493" PostTypeId="2" ParentId="7476" CreationDate="2011-02-22T12:08:20.327" Score="7" ViewCount="" Body="&lt;p&gt;Q: &lt;em&gt;Can I still make a funnel plot with effect size on the horizontal axon and total sample size n (n=n1+n2) on the vertical axis?&lt;/em&gt;&lt;br/&gt;&#xA;A: Yes&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Q: &lt;em&gt;How should such a funnel plot be interpreted?&lt;/em&gt; &lt;br/&gt;&#xA;A: It is still a funnel plot. However, funnel plots should be interpreted with caution. For example, if you have only 5-10 effect sizes, a funnel plot is useless. Furthermore, although funnel plots are a helpful visualization technique, their interpretation can be misleading. The presence of an asymmetry does not proof the existence of publication bias. Egger et al. (1997: 632f.) mention a number of reasons that can result in funnel plot asymmetries, e.g. true heterogeneity, data irregularities like methodologically poorly designed small studies or fraud. So, funnel plots can be helpful in identifying possible publication bias, however, they should always be combined with a statistical test.&lt;br/&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Q: &lt;em&gt;Is such a plot acceptable when the standard error is not known?&lt;/em&gt;&lt;br/&gt;&#xA;A: Yes&lt;br/&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Q: &lt;em&gt;Is it the same as the classical funnel plot with SE or presicion=1/SE on the vertical axon?&lt;/em&gt;&lt;br/&gt;&#xA;A: No, the shape of the 'funnel' can be different.&lt;br/&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Q: &lt;em&gt;Is its interpretation different?&lt;/em&gt;&lt;br/&gt;&#xA;A: Yes, see above&lt;br/&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Q: &lt;em&gt;How should I set the lines to make the equilateral triangle?&lt;/em&gt;&lt;br/&gt;&#xA;A: What do you mean by &quot;lines to make the equilateral triangle&quot;? Do you mean the 95%-CI lines? You will need the standard errors...&lt;br/&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You also might be interested in:  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://jama.ama-assn.org/content/295/6/676.abstract&quot; rel=&quot;nofollow&quot;&gt;Peters, Jaime L., Alex J. Sutton, David R. Jones, Keith R. Abrams, and Lesly Rushton. 2006. Comparison of two methods to detect publication bias in meta-analysis. Journal of the American Medical Association 295, no. 6: 676--80&lt;/a&gt;. (see &quot;An Alternative to Egger’s&#xA;Regression Test&quot;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;They propose a statistical test which focuses on sample size instead of standard errors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By the way, do you know the book &quot;&lt;a href=&quot;http://www.wiley.com/WileyCDA/WileyTitle/productCd-0470870141.html&quot; rel=&quot;nofollow&quot;&gt;Publication Bias in Meta-Analysis: Prevention, Assessment and Adjustments&lt;/a&gt;&quot;? It will answer a lot of your questions.&lt;/p&gt;&#xA;" OwnerUserId="307" LastActivityDate="2011-02-22T12:08:20.327" CommentCount="2" />
  <row Id="7494" PostTypeId="1" AcceptedAnswerId="7507" CreationDate="2011-02-22T14:24:42.300" Score="3" ViewCount="94" Body="&lt;p&gt;There is a catalog of noninformative priors over here:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.stats.org.uk/priors/noninformative/YangBerger1998.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.stats.org.uk/priors/noninformative/YangBerger1998.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;in page 11, they give the noninformative Jeffreys prior for the Dirichlet distribution. They give the Fisher information matrix for the Dirichlet. Can someone tell me exactly what is cell (i,j) there for the matrix?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it all 0s, except for the diagonals and the upper right element and the bottom left element?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="3347" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-02-22T14:48:49.360" LastActivityDate="2011-02-22T16:26:30.153" Title="Fisher information matrix for the Dirichlet distribution" Tags="&lt;distributions&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="1" />
  <row Id="7495" PostTypeId="2" ParentId="7482" CreationDate="2011-02-22T14:35:49.593" Score="4" ViewCount="" Body="&lt;p&gt;The &quot;no free lunch&quot; theorems (Wolpert) suggest there are no a-priori distinctions between classifiers; essentially whether one classifier performs better than another depends on the nature of the dataset.  Note also for kNN a lot depends on what distance metric you use and how you choose a good value for k.  It is not unlikely that a well-tuned kNN classifier will out-perform a badly tuned SVM.  At the end of the day, there is only one way to know for sure if an SVM will out-perform a kNN on a particular dataset, which is to try it.&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2011-02-22T14:35:49.593" CommentCount="1" />
  <row Id="7496" PostTypeId="2" ParentId="7487" CreationDate="2011-02-22T14:37:56.237" Score="1" ViewCount="" Body="&lt;p&gt;The standard approach is to use the &lt;strong&gt;partial F test&lt;/strong&gt;, explained on fine websites all over town.&lt;/p&gt;&#xA;" OwnerUserId="1920" LastActivityDate="2011-02-22T14:37:56.237" CommentCount="1" />
  <row Id="7497" PostTypeId="1" AcceptedAnswerId="7506" CreationDate="2011-02-22T14:49:46.453" Score="7" ViewCount="221" Body="&lt;p&gt;This is a question about terminology. Is a &quot;vague prior&quot; the same as a non-informative prior, or is there some difference between the two?&#xA;My impression is that they are same (from looking up vague and non-informative together), but I can't be certain.&lt;/p&gt;&#xA;" OwnerUserId="3347" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2011-02-22T16:13:36.303" LastActivityDate="2011-02-22T16:47:34.070" Title="Is a vague prior the same as a non-informative prior?" Tags="&lt;bayesian&gt;&lt;prior&gt;&lt;terminology&gt;" AnswerCount="4" />
  <row Id="7498" PostTypeId="2" ParentId="7497" CreationDate="2011-02-22T15:01:50.813" Score="1" ViewCount="" Body="&lt;p&gt;I suspect &quot;vague prior&quot; is used to mean a prior that is known to encode some small, but non-zero amount of knowledge regarding the true value of a parameter, whereas a &quot;non-informative prior&quot; would be used to mean complete ignorance regarding the value of that parameter.  It would perhaps be used to show that the analysis was not &lt;em&gt;completely&lt;/em&gt; objective.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example a very broad Gaussian might be a vague prior for a parameter where a non-informative prior would be uniform.  The Gaussian would be very nearly flat on the scale of interest, but would nevertheless favour one particular value a bit more than any other (but it might make the problem more mathematically tractable).&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2011-02-22T15:01:50.813" CommentCount="2" />
  <row Id="7499" PostTypeId="1" CreationDate="2011-02-22T15:14:04.833" Score="2" ViewCount="131" Body="&lt;p&gt;When presenting statistical information using bar charts, when will you need to use Error Bar charts?&lt;/p&gt;&#xA;" OwnerUserId="546" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-02-24T11:54:32.427" LastActivityDate="2011-02-24T14:56:55.430" Title="Which type of statistical information will need Error Bar charts for presentation?" Tags="&lt;data-visualization&gt;&lt;error&gt;" AnswerCount="2" />
  <row Id="7500" PostTypeId="2" ParentId="7497" CreationDate="2011-02-22T15:21:27.733" Score="3" ViewCount="" Body="&lt;p&gt;Definitely not, although they are frequently used interchangeably. A vague prior (relatively uninformed, not really favoring some values over others) on a parameter $\theta$ can actually induce a very informative prior on some other transformation $f(\theta)$. This is at least part of the motivation for Jeffrey's prior, which was initially constructed to be as non-informative as possible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Vague priors can also do some pretty miserable things to your model. The now-classic example is using $InverseGamma(\epsilon, \epsilon)$ as  $\epsilon\rightarrow 0$ priors on variance components in a hierarchical model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The improper limiting prior gives an improper posterior in this case. A popular alternative was to take $\epsilon$ to be really small, which results in a prior that looks almost uniform on $\mathbb{R}^+$. But it also results in a posterior that is almost improper, and model fitting and inferences suffered. See Gelman's &lt;a href=&quot;http://www.stat.columbia.edu/~gelman/research/published/taumain.pdf&quot; rel=&quot;nofollow&quot;&gt;Prior distributions for variance parameters in hierarchical models&lt;/a&gt; for a complete exposition.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: @csgillespie (rightly!) points out that I haven't completely answered your question. To my mind a non-informative prior is one that is vague in the sense that it doesn't particularly favor one area of the parameter space over another, but in doing so it shouldn't induce informative priors on other parameters. So a non-informative prior is vague but a vague prior isn't necessarily noninformative. One example where this comes into play is Bayesian variable selection; a &quot;vague&quot; prior on variable inclusion probabilities can actually induce a pretty informative prior on the total number of variables included in the model!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems to me that the search for truly noninformative priors is quixotic (though many would disagree); better to use so-called &quot;weakly&quot; informative priors (which, I suppose, are generally vague in some sense). Really, how often do we know &lt;em&gt;nothing&lt;/em&gt; about the parameter in question?&lt;/p&gt;&#xA;" OwnerUserId="26" LastEditorUserId="26" LastEditorDisplayName="" LastEditDate="2011-02-22T16:32:01.113" LastActivityDate="2011-02-22T16:32:01.113" CommentCount="4" />
  <row Id="7501" PostTypeId="2" ParentId="7251" CreationDate="2011-02-22T15:26:12.237" Score="3" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Short answer.&lt;/strong&gt;&#xA;The problem you mention is well studied by Granger C.W.J. with co-authors, and known as the forecasts combination (or pooling) problem. The general idea is to choose the loss function criterion and the parameters (may be time dependent) that minimize the latter. Below I put some references that may be useful (only publicly available, look for the original works in references after the text).&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www2.warwick.ac.uk/fac/soc/economics/staff/academic/wallis/publications/wallis_obes_05.pdf&quot; rel=&quot;nofollow&quot;&gt;K.F.Wallis&lt;/a&gt; Combining Density and Interval Forecasts: A Modest Proposal // &lt;em&gt;Oxford bulletin of economics and statistics, 67, supplement (2005) 0305-9049&lt;/em&gt; (provides a general idea of how to combine interval forecasts, though there is no details on how to choose the weights)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://management.ucsd.edu/faculty/directory/timmermann/docs/forecast-combinations.pdf&quot; rel=&quot;nofollow&quot;&gt;Allan Timmermann&lt;/a&gt; Forecast combinations. (a survey on different aspects of the forecast combinations by one of the co-editors of &lt;a href=&quot;http://books.google.lt/books?id=FSejuLtBaHUC&amp;amp;lpg=PA522&amp;amp;ots=ljqi12N_Vz&amp;amp;dq=Granger%20forecasts%20combining&amp;amp;pg=PP1#v=onepage&amp;amp;q=Granger%20forecasts%20combining&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;Handbook of economic forecasting&lt;/a&gt; that I would like to study myself)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Hoping for the &lt;strong&gt;longer answer&lt;/strong&gt; from the community.  &lt;/p&gt;&#xA;" OwnerUserId="2645" LastActivityDate="2011-02-22T15:26:12.237" CommentCount="1" />
  <row Id="7502" PostTypeId="1" CreationDate="2011-02-22T15:34:46.480" Score="1" ViewCount="61" Body="&lt;p&gt;I am going to do an occupational safety intervention with one group receiving the intervention and one group serving as the control group. I had planned on having 25 participants in each group. Each group would be given a pretest and then after some time will be given a posttest. I was told an ANOVA would provide good analysis to see if the intervention is effective. I don't know what I should put as the power and confidence level or exactly how the ANOVA will be constructed or if my sample size is large enough. Do I need to use composite scores or factors from a principal components analysis? &lt;/p&gt;&#xA;" OwnerUserId="3366" LastActivityDate="2011-02-22T15:34:46.480" Title="How to Construct an ANOVA design" Tags="&lt;anova&gt;" CommentCount="1" />
  <row Id="7503" PostTypeId="2" ParentId="7497" CreationDate="2011-02-22T15:37:55.493" Score="2" ViewCount="" Body="&lt;p&gt;Lambert et al (2005) raise the question &lt;a href=&quot;http://onlinelibrary.wiley.com/doi/10.1002/sim.2112/abstract&quot; rel=&quot;nofollow&quot;&gt;&quot;How Vague is Vague? A simulation study of the impact of the use of vague prior distributions in MCMC using WinBUGS&lt;/a&gt;&quot;. They write: &quot;We do not advocate the use of the term non-informative prior distribution as we consider all priors to contribute some information&quot;. I tend to agree but I am definitely no expert in Bayesian statistics.&lt;/p&gt;&#xA;" OwnerUserId="307" LastActivityDate="2011-02-22T15:37:55.493" CommentCount="3" />
  <row Id="7504" PostTypeId="2" ParentId="7494" CreationDate="2011-02-22T15:45:19.270" Score="3" ViewCount="" Body="&lt;p&gt;I &lt;em&gt;think&lt;/em&gt; it's meant to have constant off-diagonal entries, i.e. it could also be written&#xA;$$I(\alpha_1, \alpha_2, \ldots, \alpha_k) = \operatorname{diag}\left[ PG(1,\alpha_1) , PG(1, \alpha_2), \ldots, PG(1,\alpha_k) \right] - PG(1,\alpha_0) J_k $$&#xA;where $J_k$ is a $k \times k$ &lt;a href=&quot;http://en.wikipedia.org/wiki/Matrix_of_ones&quot; rel=&quot;nofollow&quot;&gt;matrix of ones&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-02-22T15:45:19.270" />
  <row Id="7505" PostTypeId="1" CreationDate="2011-02-22T15:52:50.807" Score="3" ViewCount="76" Body="&lt;p&gt;Hello,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;there is a short introduction to AB Tests in &lt;a href=&quot;http://stats.stackexchange.com/questions/4884/aggregation-level-in-ab-tests&quot;&gt;this question&lt;/a&gt; or &lt;a href=&quot;http://20bits.com/articles/statistical-analysis-and-ab-testing/&quot; rel=&quot;nofollow&quot;&gt;here at 20bits&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We are currently testing different versions of landing pages and are using the conversion rate (e.g. 4% vs. 5%) to track which version performs better. This is working fine so far.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I would like to do is start calculating whether a version performs better using the sales volume. So I could say the control version sold $\$$10.000 with 100 visitors, while the new version sold $\$$12.000 with 110 visitors. Is the difference statistically significant?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would appreciate any points in the right direction. Specifically I am having problems understanding how I can calculate 95% and 99% percentile intervals for the data above.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can share my Excel sheet for calculating &quot;basic&quot; A/B tests, if that helps.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you in advance for any help!&lt;/p&gt;&#xA;" OwnerUserId="3367" LastEditorUserId="264" LastEditorDisplayName="" LastEditDate="2011-04-20T06:17:50.750" LastActivityDate="2011-04-21T15:05:07.710" Title="Moving from conversion rates to sales volume in A/B tests" Tags="&lt;hypothesis-testing&gt;&lt;confidence-interval&gt;&lt;testing&gt;&lt;ab-test&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="7506" PostTypeId="2" ParentId="7497" CreationDate="2011-02-22T16:13:39.453" Score="1" ViewCount="" Body="&lt;p&gt;Gelman et al (2003) say: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;there has long been a desire for prior distributions that can be guaranteed to play a minimal role in the posterior distribution. Such distributions are sometimes called 'reference prior distributions' and the prior density is described as vague, flat, or &lt;em&gt;noninformative&lt;/em&gt;.[emphasis from original text]&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Based on my reading of the discussion of Jeffrey's prior in Gelman et al (2003, pp 62ff., there is no consensus about the existence of a truly non-informative prior, and that sufficiently vague/flat/diffuse priors are sufficient.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some of the points that they make:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Any prior includes information, including priors that state that no information is known. &#xA;&lt;ul&gt;&#xA;&lt;li&gt;For example, if we know that we know nothing about the parameter in question, then we know something about it.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;In most applied context, there is no clear advantage to a truly non-informative prior when sufficiently vague priors suffice, and in many cases there are advantages - like finding a proper prior - to using a vague parameterization of a conjugate prior.&lt;/li&gt;&#xA;&lt;li&gt;Jeffrey's principle can be useful to construct priors that minimize Fisher's information content in univariate models, but there is no analogue for the multivariate case&lt;/li&gt;&#xA;&lt;li&gt;When comparing models, the Jeffrey's prior will vary with the distribution of the likelihood, so priors would also have to change   &lt;/li&gt;&#xA;&lt;li&gt;there has generally been a lot of debate about whether a non-informative prior even exists (because of 1, but also see discussion and references on p 66 in Gelman et al for the history of this debate).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;note&lt;/em&gt; this is community wiki - The underlying theory is at the limits of my understanding, and I would appreciate contributions to this answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.stat.columbia.edu/~gelman/book/&quot; rel=&quot;nofollow&quot;&gt;Gelman et al 2003 Bayesian Data Analysis, Chapman and Hall/CRC&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1381" LastEditorUserId="1381" LastEditorDisplayName="" LastEditDate="2011-02-22T16:47:34.070" LastActivityDate="2011-02-22T16:47:34.070" CommunityOwnedDate="2011-02-22T16:13:39.453" CommentCount="3" />
  <row Id="7507" PostTypeId="2" ParentId="7494" CreationDate="2011-02-22T16:26:30.153" Score="7" ViewCount="" Body="&lt;p&gt;Let's work it out.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The logarithm of the Dirichlet density function is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\lambda(\mathbf{x}|\mathbf{\alpha}) = \log(\Gamma(\alpha_0)) - \sum_{i=1}^{k}{\log(\Gamma(\alpha_i)))} + \sum_{i=1}^{k}{(\alpha_i - 1)\log(x_i)},$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\alpha_0 = \alpha_1 + \alpha_2 + \cdots + \alpha_k$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Taking second partial derivatives with respect to the parameters $\alpha_i$ is particularly simple; all we really need to know (in addition to the most basic properties of derivatives) is that $\partial \alpha_0 / \partial \alpha_i = 1$ and $\partial x_j / \partial \alpha_i = 0$.  Thus&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{\partial \lambda}{\partial \alpha_i} = \psi(\alpha_0) - \psi(\alpha_i) + \log(x_i)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{\partial^2 \lambda}{\partial \alpha_i \partial \alpha_j} &#xD;&#xA;= \psi&amp;#39;(\alpha_0) - \psi&amp;#39;(\alpha_j)\delta_{i j},$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\psi$ (the &lt;a href=&quot;http://mathworld.wolfram.com/GammaFunction.html&quot; rel=&quot;nofollow&quot;&gt;digamma function&lt;/a&gt;) is the derivative of $\log(\Gamma)$ and $\delta_{i j} = 1$ if and only if $i = j$ and is $0$ otherwise: that is, it's the $k$ by $k$ identity matrix.  The &lt;a href=&quot;http://en.wikipedia.org/wiki/Fisher_information&quot; rel=&quot;nofollow&quot;&gt;Fisher Information Matrix&lt;/a&gt; is, by definition, the negative expectation of the matrix of second partial derivatives.  Because its entries are constant with respect to the random variable $\mathbf{x}$, taking expectations is trivial.  We obtain a matrix with the values $\psi&amp;#39;(\alpha_i)$ along the diagonal and $\psi&amp;#39;(\alpha_0)$ is subtracted &lt;em&gt;everywhere,&lt;/em&gt; showing that @onestop's interpretation is correct.  (&quot;$PG(1,\alpha)$&quot; is merely an idiosyncratic notation for the polygamma function $\psi&amp;#39;(\alpha)$.)&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-02-22T16:26:30.153" />
  <row Id="7508" PostTypeId="1" CreationDate="2011-02-22T16:39:14.313" Score="5" ViewCount="115" Body="&lt;p&gt;If I hypothesize that a gene signature will identify subjects at a lower risk of recurrence, that is decrease by 0.5 (hazard ratio of 0.5) the event rate in 20% of the population and I intend to use samples from a retrospective cohort study does the sample size need to be adjusted for unequal numbers in the two hypothesised groups? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example using Collett, D: Modelling Survival Data in Medical Research, Second Edition - 2nd Edition 2003. The required total number of events, d, can be found using,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{equation}&#xA;d = \frac{(Z_{\alpha/2} + Z_{\beta/2})^2}{p_1 p_2 (\theta R)^2}&#xA;\end{equation}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $Z_{\alpha/2}$ and $Z_{\beta/2}$ are the upper $\alpha/2$ and upper $\beta/2$ points, respectively, of the standard normal distribution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the particular values, &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;$p_1 = 0.20$&lt;/li&gt;&#xA;&lt;li&gt;$p_2 = 1 - p_1$&lt;/li&gt;&#xA;&lt;li&gt;$\theta R =  -0.693$&lt;/li&gt;&#xA;&lt;li&gt;$\alpha = 0.05$ and so $Z_{0.025}= 1.96$&lt;/li&gt;&#xA;&lt;li&gt;$\beta = 0.10$ and so $Z_{0.05} = 1.28$, &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;and taking $\theta R = \log \psi R = \log 0.50 = -0.693$, the number of events required (rounded up) to have a 90% chance of detecting a hazard ratio of 0.50 to be significant at the two-sided 5% level is then given by&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{equation}&#xA;d = \frac{(1.96 + 1.28)^2}{0.20 \times 0.80\times (\log 0.5)^2}= 137&#xA;\end{equation}&lt;/p&gt;&#xA;" OwnerUserId="712" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2011-02-22T18:25:40.793" LastActivityDate="2011-02-22T18:25:40.793" Title="Power analysis for survival analysis" Tags="&lt;survival&gt;&lt;power-analysis&gt;" CommentCount="3" />
  <row Id="7509" PostTypeId="2" ParentId="2787" CreationDate="2011-02-22T17:40:39.037" Score="1" ViewCount="" Body="&lt;p&gt;Jakob Nielsen &lt;a href=&quot;http://www.useit.com/alertbox/20000319.html&quot; rel=&quot;nofollow&quot;&gt;recommends testing with five users&lt;/a&gt; for optimal results. This assertion has been challenged a few times, both empirically and theoretically, but generally seems to hold quite well.&lt;/p&gt;&#xA;" OwnerUserId="3367" LastActivityDate="2011-02-22T17:40:39.037" />
  <row Id="7510" PostTypeId="2" ParentId="7481" CreationDate="2011-02-22T18:06:08.293" Score="1" ViewCount="" Body="&lt;p&gt;Continuous latent variables with discrete (polytomous in your case) manifest variables is part of item response analysis. Package 'ltm' in R covers a variety of such models. I refer you to &lt;a href=&quot;http://goo.gl/lJh5s&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; paper, which deals with exactly same problem.&lt;/p&gt;&#xA;" OwnerUserId="609" LastActivityDate="2011-02-22T18:06:08.293" CommentCount="2" />
  <row Id="7511" PostTypeId="1" CreationDate="2011-02-22T18:27:41.433" Score="4" ViewCount="72" Body="&lt;p&gt;We are dealing with a measurement apparatus that is cursed by noise and are trying to find out if a measurement was noise or an actual measurement.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assume we have a beam of light incident on an square array of photo detectors. The &quot;counts&quot; measured in the individual tubes follow Poisson statistics. A typical beam distributes light across 4 photo tubes, most in the tube closest to the point of incident. Noise would be distributed uncorrelated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can measure the count distributions for a location where we know we had an actual beam hitting, and for a location where we know no beam was present. From these measurements we can construct empirical count PDFs for measurements in the individual tubes, $p_{i,\,\rm{true}}$ and $p_{i,\,\rm{noise}}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since for a noise measurement the counts in the individual tubes are uncorrelated I would guess that&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ P_\mathrm{noise} = \prod_i p_{i,\,\mathrm{noise}} $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;would be a good guess for the probability that this measurement was noise. Can I use a similar expression&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ P_\mathrm{true} = \prod_i p_{i,\,\mathrm{true}} $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;even though counts would be correlated for the &quot;true&quot; case?&lt;/p&gt;&#xA;" OwnerUserId="56" LastEditorUserId="56" LastEditorDisplayName="" LastEditDate="2011-02-22T18:58:19.827" LastActivityDate="2011-02-23T16:02:13.747" Title="Construct probability that measurement belongs to one of two sets" Tags="&lt;hypothesis-testing&gt;&lt;correlation&gt;" AnswerCount="1" CommentCount="4" />
  <row Id="7512" PostTypeId="1" CreationDate="2011-02-22T18:43:11.643" Score="2" ViewCount="63" Body="&lt;p&gt;What is the best way to correlate zero-inflated count variables with a small sample size (n=~50 and N=99)?&lt;/p&gt;&#xA;" OwnerUserId="3368" LastActivityDate="2011-02-22T18:43:11.643" Title="Zero inflated correlation" Tags="&lt;correlation&gt;" CommentCount="2" />
  <row Id="7513" PostTypeId="1" AcceptedAnswerId="7530" CreationDate="2011-02-22T19:38:50.313" Score="6" ViewCount="278" Body="&lt;p&gt;Hi all,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could anyone offer some pointers on how to use the &lt;code&gt;weights&lt;/code&gt; argument in R's &lt;code&gt;lm&lt;/code&gt; function?  Say, for instance you were trying to fit a model on traffic data, and you had several hundred rows, each of which represented a city (with a different population).  If you wanted the model to adjust the relative influence of each observation based on population size, could you simply specify &lt;code&gt;weights=[the column containing the city's population]&lt;/code&gt;?  Is that the sort of vector that can go into &lt;code&gt;weights&lt;/code&gt;?  Or would you need to use a different R function/package/approach entirely?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Curious to hear how people tackle this one - didn't see it covered in any of the linear modeling tutorials I saw out there.  Thanks!&lt;/p&gt;&#xA;" OwnerUserId="3320" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-02-23T08:20:59.090" LastActivityDate="2011-02-23T08:20:59.090" Title="How to use weights in function lm in R?" Tags="&lt;r&gt;&lt;regression&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="7514" PostTypeId="2" ParentId="7478" CreationDate="2011-02-22T19:58:23.480" Score="0" ViewCount="" Body="&lt;p&gt;my brother posted the original data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;what is happening here is that i am observing two different conditions. the control is no bike box. this is the baseline. a normal intersection with bike lanes. the experimental condition is an intersection that has bike boxes added. i am attempting to interpret the correct/incorrect use as a function of this change in the environment.  basically, i want to know how best to report the data in a simple and understandable way. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;the problem i find it that stuff gets messed up because boxes/no boxes are independent from the correct/incorrect behaviour. i think.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;if you take this as percentages&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;           correct       incorrect&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;no box (control)   72%          28%&lt;/p&gt;&#xA;&#xA;&lt;p&gt;bbox (exp)         52%          48%&lt;/p&gt;&#xA;&#xA;&lt;p&gt;i basically want to have a descriptive value so i can explain how the bike boxes effect the outcome of correct or incorrect behaviour at a specific intersection.&#xA;the two intersections have different sample sizes which complicate things from my basic psych stats text and knowledge. what i am also thinking is that i need to use the no box data as my baseline in which to compare the experimental condition? that makes sense, no?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;what i was thinking is that i could report a decrease of 20% correct behaviour and an increase of 20% for incorrect responses.  so does that mean there is a total difference of 40%? or should i just focus on the 20% and report for one variable, correct or incorrect as to which i see as more important for the report. but to me this data seems incomplete. it doesnt really tell the whole story.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;i also thought about doing a percent change ie (new val/old val)-1. but again, i used the percentages of the total per outcome (correct incorrect) as reported above. so i then got those values:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-27% change for correct behaviour due to addition of bike boxes &#xA;+69% change for incorrect behaviour due to addition of bike boxes&lt;/p&gt;&#xA;&#xA;&lt;p&gt;does that make any sense to what i am trying to show??&lt;/p&gt;&#xA;&#xA;&lt;p&gt;i am basically confused as to what makes the best sense to report in this case given the comparison i am making and what is the most accurate data that describes these changes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;thank you all. &lt;/p&gt;&#xA;" OwnerUserId="3370" LastActivityDate="2011-02-22T19:58:23.480" />
  <row Id="7515" PostTypeId="1" AcceptedAnswerId="7516" CreationDate="2011-02-22T20:43:25.633" Score="4" ViewCount="152" Body="&lt;p&gt;What are some techniques for sampling two correlated random variables:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;if their probability&#xA;distributions are parameterized&#xA;(e.g., log-normal)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;if they have non-parametric&#xA;distributions.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The data are two time series for which we can compute non-zero correlation coefficients.  We wish to simulate these data in the future, assuming the historical correlation and time series CDF is constant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For case (2), the 1-D analogue would be to construct the CDF and sample from it.  So I guess, I could construct a 2-D CDF and do the same thing.  However, I wonder if there is a way to come close by using the individual 1-D CDFs and somehow linking the picks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="2260" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-02-22T21:35:51.057" LastActivityDate="2011-02-22T21:35:51.057" Title="What are some techniques for sampling two correlated random variables?" Tags="&lt;correlation&gt;&lt;sampling&gt;&lt;monte-carlo&gt;&lt;stochastic-processes&gt;&lt;copula&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="1" />
  <row Id="7516" PostTypeId="2" ParentId="7515" CreationDate="2011-02-22T21:09:42.563" Score="5" ViewCount="" Body="&lt;p&gt;I think what you're looking for is a copula. You've got two marginal distributions (specified by either parametric or empirical cdfs) and now you want to specify the dependence between the two. For the bivariate case there are all kinds of choices, but the basic recipe is the same. I'll use a Gaussian copula for ease of interpretation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To draw from the Gaussian copula with correlation matrix $C$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Draw $(Z=(Z_1, Z_2)\sim N(0, C)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Set $U_i = \Phi(Z_i)$ for $i=1, 2$ (with $\Phi$ the standard normal cdf). Now $U_1, U_2\sim U[0,1]$, but they're dependent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) Set $Y_i = F_i^{-1}(U_i)$ where $F_i^{-1}$ is the (pseudo) inverse of the marginal cdf for variable $i$. This implies that $Y_i$ follow the desired distribution (this step is just inverse transform sampling).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Voila! Try it for some simple cases, and look at marginal histograms and scatterpolots, it's fun. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;No guarantee that this is appropriate for your particular application though (in particular, you might need to replace the Gaussian copula with a t copula) but this should get you started. A good reference on copula modeling is Nelsen (1999), An Introduction to Copulas, but there are some pretty good introductions online too.&lt;/p&gt;&#xA;" OwnerUserId="26" LastActivityDate="2011-02-22T21:09:42.563" CommentCount="2" />
  <row Id="7517" PostTypeId="2" ParentId="7511" CreationDate="2011-02-22T21:18:31.240" Score="3" ViewCount="" Body="&lt;p&gt;According to the comments after the question, this is a hypothesis testing situation.  You have stipulated that you can accurately assess the null distribution of the individual cell counts.  We need a test statistic.  The nature of the problem suggests running a small kernel over the array (essentially to deconvolve the signal).  A simple choice would be a 2 x 2 mean.  We would then look at the maximum of those values.  The Neyman-Pearson lemma says that a good test is based on setting a critical value for this statistic.  The test size depends on the null distribution of the test statistic while its power depends on the intensity of the signal.  You need to choose a critical value to appropriately balance the expected false positive rate with the power for the kinds of signals you are looking for.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All this is routine; the only non-routine aspect is determining the null distribution of the statistic.  It's the maximum of a set of fairly highly correlated linear combinations of Poisson variates.  Specifically, if we index the rows and columns of the array and let $X_{i,j}$ be the Poisson variate for the cell in row $i$, $1 \le i \le m$, and column $j$, $1 \le j \le n$, then the 2 x 2 mean is the array of values $Y_{i,j} = (X_{i,j} + X_{i+1,j} + X_{i,j+1} + X_{i+1,j+1})/4$, $1 \le i \le m-1$, $1 \le j \le n-1$, and the test statistic is $t = \max\{Y_{i,j}\}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If someone could jump in and tell us how to compute the distribution of $t$ that would be nice, but I suspect it's not an easy calculation.  Would you consider a small simulation?  It's easy to set up in &lt;strong&gt;R&lt;/strong&gt; or &lt;em&gt;Mathematica&lt;/em&gt;, for instance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively, use an approximation.  The results of a few simulations with arrays from 9 to 1600 elements and photon intensities from 1 to 4096 per cell are consistent with two obvious approximations: one can treat all $(m-1)(n-1)$ of the $Y_{i,j}$ as independent or one can take every fourth one and treat them as independent, and then calculate the distribution of their maximum.  For small intensities the upper tails of the simulated distributions of $t$ (10,000 iterations per simulation) appear to behave like the latter approximation: that is, $t$ behaves like the largest of $(m-1)(n-1)/4$ independent averages of four Poisson variates.  For large intensities the tails are quite close to the former approximation: that is, $t$ behaves like the largest of $(m-1)(n-1)$ independent averages of Poisson variates.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/pZDCj.png&quot; alt=&quot;Histogram of simulated test statistic&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a histogram of $t$ for a 30 by 40 grid with cell intensity 100 (10,000 simulations).  Because its 99th quantile equals 122.25, you could create a test with at most 1% false positive rate by setting the critical value to 123.  An average photon count of around 135 in a single block of four cells would be readily detectable with this method.  That would represent a total flux of 4*(135 - 100) = 140 photons above background.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the record, here is the &lt;em&gt;Mathematica&lt;/em&gt; code used to generate this histogram.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;simulate[m_Integer, n_Integer, \[Mu]_] /;&#xA;   m &amp;gt;= 2 &amp;amp;&amp;amp; n &amp;gt;= 2 &amp;amp;&amp;amp; \[Mu] &amp;gt; 0 := With[&#xA;   {f = PoissonDistribution[\[Mu]]},&#xA;   y = ListConvolve[{{1, 1}, {1, 1}}/4, RandomInteger[f, {m, n}]]&#xA;   ];&#xA;With[{m = 30, n = 40, \[Mu] = 100, nTrials = 10000},&#xA; null = ParallelTable[Max[Flatten[simulate[m, n, \[Mu]]]], {i, 1, nTrials}];&#xA; Histogram[null, {1/4}, AxesLabel -&amp;gt; {&quot;t&quot;, &quot;Count&quot;}]&#xA; ]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-02-23T16:02:13.747" LastActivityDate="2011-02-23T16:02:13.747" CommentCount="8" />
  <row Id="7518" PostTypeId="1" CreationDate="2011-02-22T21:20:34.683" Score="3" ViewCount="59" Body="&lt;p&gt;If I define a function in IML:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;start func(a);&#xA;  submit a / R;&#xA;   print(&amp;amp;a);&#xA;  endsubmit;&#xA;finish;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and run it:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;run func(&quot;character string&quot;);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I get the error message: object 'character string' does not exist.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So R or IML is evaluating the character string into an object.  I want R to output &quot;character string.&quot;  How do I do this?&lt;/p&gt;&#xA;" OwnerUserId="3371" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-02-22T21:22:59.367" LastActivityDate="2011-02-23T00:49:33.997" Title="How to pass character strings to R from IML Studio" Tags="&lt;r&gt;&lt;sas&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="7519" PostTypeId="1" AcceptedAnswerId="7526" CreationDate="2011-02-22T23:01:36.607" Score="4" ViewCount="127" Body="&lt;p&gt;Consider a Jeffreys prior where $p(\theta) \propto \sqrt{|i(\theta)|}$, where $i$ is the Fisher information.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I keep seeing this prior being mentioned as a uninformative prior, but I never saw an argument why it is uninformative. After all, it is not a constant prior, so there has to be some other argument.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand that it does not depends on reparametrization, which brings me to the next question. Is it that the determinant of the Fisher information does not depend on reparametrization? Because Fisher information definitely depends on the parametrization of the problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="3347" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-02-22T23:49:45.727" LastActivityDate="2011-02-23T02:27:01.233" Title="Why are Jeffreys priors considered noninformative?" Tags="&lt;bayesian&gt;&lt;prior&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="1" />
  <row Id="7520" PostTypeId="2" ParentId="7513" CreationDate="2011-02-22T23:05:22.627" Score="2" ViewCount="" Body="&lt;p&gt;What you suggest should work.  See if this makes sense:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;lm(c(8000, 50000, 116000) ~ c(6, 7, 8))&#xA;lm(c(8000, 50000, 116000) ~ c(6, 7, 8), weight = c(123, 123, 246))&#xA;lm(c(8000, 50000, 116000, 116000) ~ c(6, 7, 8, 8))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The second line produces the same intercept and slope as the third line (distinct from the first line's result), by giving one observation relatively twice the weight of each of the other two observations, similar to the impact of duplicating the third observation.  &lt;/p&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-02-22T23:05:22.627" />
  <row Id="7521" PostTypeId="1" AcceptedAnswerId="7566" CreationDate="2011-02-23T00:30:42.133" Score="2" ViewCount="56" Body="&lt;p&gt;I have quarterly sales data for a variety of stores and would like to estimate the effect of a regulation on sales.  A panel type model would appear to be appropriate in this case, with the regulation as a dummy variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the quantity of sales varies by two-orders of magnitude between stores.  How should I account for this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Apologies for any problems with the question, I am something of a beginner.&lt;/p&gt;&#xA;" OwnerUserId="179" LastEditorUserId="179" LastEditorDisplayName="" LastEditDate="2011-02-24T00:42:07.217" LastActivityDate="2011-02-24T13:20:24.633" Title="Subjects with different order of magnitude values in panel data" Tags="&lt;time-series&gt;&lt;basic-concepts&gt;&lt;longitudinal&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="7522" PostTypeId="2" ParentId="7518" CreationDate="2011-02-23T00:33:24.057" Score="2" ViewCount="" Body="&lt;p&gt;I haven't used the new R/IML interface functionality, but from reading the help it looks like &lt;code&gt;submit foo&lt;/code&gt; does text substitution. That is, if the IML variable &lt;code&gt;foo&lt;/code&gt; contains the value &lt;code&gt;x&lt;/code&gt;, then any occurrence of &lt;code&gt;&amp;amp;foo&lt;/code&gt; in the submit block is replaced with &lt;code&gt;x&lt;/code&gt;. When R is involved, IML generates R code to treat x as a symbol rather plain text. With that in mind, see if&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;start func(a);&#xA;  submit a / R;&#xA;    print(&quot;&amp;amp;a&quot;);&#xA;  endsubmit;&#xA;finish;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;does what you want. Alternatively, you could use the ExportMatrixToR module to send the data to R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;start func(a);&#xA;   run ExportMatrixToR(a, &quot;a&quot;);&#xA;   submit / R;&#xA;     print(a);&#xA;   endsubmit;&#xA;finish;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Caveat: untested code.&lt;/p&gt;&#xA;" OwnerUserId="1569" LastEditorUserId="1569" LastEditorDisplayName="" LastEditDate="2011-02-23T00:49:33.997" LastActivityDate="2011-02-23T00:49:33.997" CommentCount="1" />
  <row Id="7523" PostTypeId="1" CreationDate="2011-02-23T01:01:33.197" Score="3" ViewCount="161" Body="&lt;p&gt;I would appreciate some advice on how best to weight or give more importance to a percentage with a larger denominator.&#xA;Eg&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;A. 1 out of 2 = 50%&lt;/li&gt;&#xA;&lt;li&gt;B. 5 out of 10 = 50%&lt;/li&gt;&#xA;&lt;li&gt;C. 500 out of 1000 = 50%&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Some of the data is sparse and continually emerging so &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;A) could just be a blip and could change, &lt;/li&gt;&#xA;&lt;li&gt;B) is emerging and may vary slightly &lt;/li&gt;&#xA;&lt;li&gt;C) is becoming fairly certain&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Also, I'm trying to minimize the number of parameters so is there a way to weight the relative 50%'s for each scenario instead of creating a separate parameter? &lt;/p&gt;&#xA;" OwnerUserId="3373" LastEditorUserId="3484" LastEditorDisplayName="" LastEditDate="2011-03-01T11:57:42.837" LastActivityDate="2011-03-01T11:57:42.837" Title="How do I introduce features and their confidence values into classifiers" Tags="&lt;proportion&gt;&lt;weighted-mean&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="7525" PostTypeId="2" ParentId="7523" CreationDate="2011-02-23T02:02:42.370" Score="4" ViewCount="" Body="&lt;p&gt;It might be a good idea to rename the question to something like &quot;how do I introduce features and their confidence values into classifiers&quot;. I can think of two ways to do it, but if you phrase the question differently, more people will look at it and will maybe have additional suggestions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first option is to add two features to the classifier, the first feature being the predictor itself (e.g. the success rate), and the second feature being the confidence of this predictor. As mentioned below, this confidence can be just the denominator or width of the confidence interval. This is the more principled way to do it. The problem is that this will not work with linear classifiers. Basically what you want the classifier to learn is that the predictor has high weight if the confidence is high, and low weight if the confidence is low. I.e. you want the weight for one feature to depend on another feature. Linear classifiers cannot do that. So linear NNs (perceptron) or linear SVM will not work. Decision trees are likely to work, and you might have a chance with non-linear NNs and kernel SVM. But these have their own issues.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second option is to have some kind of &quot;prior&quot; on your estimate of the predictor itself. This can be a rigorous prior in the Bayesian sense, or can be just something simple and hacky (since you feed it to a classifier anyway). One simple way to do it is to add some number (e.g. 10) &quot;fabricated&quot; observations to each trainer's data. E.g. you believe that each trainer's prior probability of winning on a particular course is 50%. So you start with 5 wins and 5 loses for every trainer. To these fabricated observations you add the real statistics. For Catterick this will give you 11 wins out of 58 total, so 19%. The more real data you have, the less significant the effect of this prior will be. On the other hand, for the Exeter data this will give you 6 wins out of 11 total, so 55% (rather than your overly confident estimate of 100%). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the second option is to compute the percentage this way, and then feed just this single percentage figure to the classifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Depending on what you expect the classifier to do, you may vary the prior probability as needed. E.g. if you expect the classifier to heed data close to 0% or to 100% and ignore the 50% region, then starting with the 50/50 prior may be OK. If you want to get more accurate estimates of these percentages, you can change the prior according to the trainer. E.g. you have only 1/1 real statistics for the trainer at Exeter, but you know this particular trainer wins 80% of races at other courses. You can account for this by introducing 8 fabricated wins and 2 losses (rather than 5 and 5).&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Weighing by the denominator itself may be reasonable, especially if it represents the number of samples. This way you can easily combine percentages. E.g. in one (small) experiment you get 3 out of 10 = 30%; in another (larger) experiment you get 50 out of 100 = 50%, and the average is (30%*10 + 50%*100) / (10 + 100) = 48%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you just want to know how good your estimate is, then try computing &quot;confidence intervals&quot; -- this gives you an idea of how certain your estimates are. E.g. for a small experiment you might get an interval of [25%, 75%], for a larger one you may get [49%. 51%] interval. How to compute these confidence intervals will depend on the details of your data and the model.&lt;/p&gt;&#xA;" OwnerUserId="3369" LastEditorUserId="3369" LastEditorDisplayName="" LastEditDate="2011-02-24T20:36:45.130" LastActivityDate="2011-02-24T20:36:45.130" CommentCount="1" />
  <row Id="7526" PostTypeId="2" ParentId="7519" CreationDate="2011-02-23T02:27:01.233" Score="2" ViewCount="" Body="&lt;p&gt;It's considered noninformative &lt;em&gt;because&lt;/em&gt; of the parameterization invariance. You seem to have the impression that a uniform (constant) prior is noninformative. Sometimes it is, sometimes it isn't.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What happens with Jeffrey's prior under a transformation is that the Jacobian from the transformation gets sucked into the original Fisher information, which ends up giving you the Fisher information under the new parameterization. No magic (in the mechanics at least), just a little calculus and linear algebra.&lt;/p&gt;&#xA;" OwnerUserId="26" LastActivityDate="2011-02-23T02:27:01.233" />
  <row Id="7527" PostTypeId="1" AcceptedAnswerId="7529" CreationDate="2011-02-23T03:27:05.930" Score="8" ViewCount="232" Body="&lt;p&gt;I'm trying to implement a &quot;change point&quot; analysis, or a multiphase regression using &lt;code&gt;nls()&lt;/code&gt; in R. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://i.imgur.com/27f1S.png&quot; rel=&quot;nofollow&quot;&gt;Here's some fake data I've made&lt;/a&gt;. The formula I want to use to fit the data is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$y = \beta_0 + \beta_1x + \beta_2\max(0,x-\delta)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What this is supposed to do is fit the data up to a certain point with a certain intercept and slope ($\beta_0$ and $\beta_1$), then, after a certain x value ($\delta$), augment the slope by $\beta_2$. That's what the whole max thing is about. Before the $\delta$ point, it'll equal 0, and $\beta_2$ will be zeroed out.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, here's my function to do this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;changePoint &amp;lt;- function(x, b0, slope1, slope2, delta){ &#xA;   fit &amp;lt;- b0 + (x*slope1) + (max(0, x-delta) * slope2)&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And I try to fit the model this way&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;nls(y ~ changePoint(x, b0, slope1, slope2, delta), &#xA;    data = data, &#xA;    start = c(b0 = 50, slope1 = 0, slope2 = 2, delta = 48))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I chose those starting parameters, because I &lt;em&gt;know&lt;/em&gt; those are the starting parameters, because I made the data up.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I get this error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Error in nlsModel(formula, mf, start, wts) : &#xA;  singular gradient matrix at initial parameter estimates&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Have I just made unfortunate data? I tried fitting this on real data first, and was getting the same error, and I just figured that my initial starting parameters weren't good enough. &lt;/p&gt;&#xA;" OwnerUserId="287" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-14T19:28:00.103" LastActivityDate="2011-04-14T19:28:00.103" Title="Change point analysis using R's nls()" Tags="&lt;r&gt;&lt;regression&gt;&lt;change-point&gt;&lt;nls&gt;" AnswerCount="1" FavoriteCount="3" />
  <row Id="7528" PostTypeId="1" AcceptedAnswerId="7557" CreationDate="2011-02-23T03:42:14.227" Score="2" ViewCount="122" Body="&lt;p&gt;A simplified regression equation $ES=\frac{a+b}{n_1+n_2}$ has been suggested as an alternative to Egger's regression equation $\frac{ES}{SE}=\frac{a+b}{SE}$, where ES=Effect Size, $n_1$=sample size of the patients, $n_2$=sample size of the controls, SE=Standard Error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This alternative test, that was presented by Peters et al. in their &lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pubmed/16467236&quot; rel=&quot;nofollow&quot;&gt;2006 paper in JAMA&lt;/a&gt;, is supposed to be better than Egger's test when the ES is the lnOR.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This alternative test could also be valuable in cases Standard Error (SE) cannot be calculated, as SE is not taking part in the equation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could this alternative Egger's test be used with the other types of Effect Size? When the ES is the SMD? When the ES is the RR? When the ES is the Pearson's correlation coefficient?&lt;/p&gt;&#xA;" OwnerUserId="3333" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-02-23T22:19:05.607" LastActivityDate="2011-02-24T00:34:36.170" Title="Alternative Egger's test, without using standard error" Tags="&lt;meta-analysis&gt;&lt;standard-error&gt;&lt;effect-size&gt;&lt;funnel-plot&gt;&lt;publication-bias&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="7529" PostTypeId="2" ParentId="7527" CreationDate="2011-02-23T06:27:52.350" Score="4" ViewCount="" Body="&lt;p&gt;(At first I thought it could be a problem resulting from the fact that &lt;code&gt;max&lt;/code&gt; is not vectorized, but that's not true.  It &lt;em&gt;does&lt;/em&gt; make it a pain to work with changePoint, wherefore the following modification:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;changePoint &amp;lt;- function(x, b0, slope1, slope2, delta){ &#xA;   fit &amp;lt;- b0 + (x*slope1) + (sapply(x-delta, function (t) max(0, t)) * slope2)&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://tolstoy.newcastle.edu.au/R/help/06/06/28419.html&quot; rel=&quot;nofollow&quot;&gt;This R-help mailing list post&lt;/a&gt; describes one way in which this error may result: the rhs of the formula is overparameterized, such that changing two parameters in tandem gives the same fit to the data.  I can't see how that is true of your model, but maybe it is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In any case, you can write your own objective function and minimize it.  The following function gives the squared error for data points (x,y) and a certain value of the parameters (the weird argument structure of the function is to account for how &lt;code&gt;optim&lt;/code&gt; works):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sqerror &amp;lt;- function (par, x, y)&#xA;  sum((y - changePoint(x, par[1], par[2], par[3], par[4]))^2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then we say:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;optim(par = c(50, 0, 2, 48), fn = sqerror, x = x, y = data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And see:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;$par&#xA;[1] 54.53436800 -0.09283594  2.07356459 48.00000006&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Note that for my fake data (&lt;code&gt;x &amp;lt;- 40:60; data &amp;lt;- changePoint(x, 50, 0, 2, 48) + rnorm(21, 0, 0.5)&lt;/code&gt;) there are lots of local maxima depending on the initial parameter values you give.  I suppose if you wanted to take this seriously you'd call the optimizer many times with random initial parameters and examine the distribution of results.&lt;/p&gt;&#xA;" OwnerUserId="2975" LastEditorUserId="2975" LastEditorDisplayName="" LastEditDate="2011-02-23T07:20:46.123" LastActivityDate="2011-02-23T07:20:46.123" CommentCount="2" />
  <row Id="7530" PostTypeId="2" ParentId="7513" CreationDate="2011-02-23T08:15:22.403" Score="3" ViewCount="" Body="&lt;p&gt;I think R help page of &lt;code&gt;lm&lt;/code&gt; answers your question pretty well. The only requirement for weights is that the vector supplied must be the same length as the data. You can even supply only the name of the variable in the data set, R will take care of the rest, NA management, etc. You can also use formulas in the &lt;code&gt;weight&lt;/code&gt; argument. Here is the example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x &amp;lt;-c(rnorm(10),NA)&#xA;df &amp;lt;- data.frame(y=1+2*x+rnorm(11)/2,x=x,wght1=1:11)&#xA;&#xA;##Fancy weights as numeric vector&#xA;summary(lm(y~x,data=df,weights=(df$wght1)^(3/4))) &#xA;&#xA;#Fancy weights as formula on column of the data set&#xA;summary(lm(y~x,data=df,weights=I(wght1^(3/4))))&#xA;&#xA;#Mundane weights as the column of the data set&#xA;summary(lm(y~x,data=df,weights=wght1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Note that weights must be positive, otherwise R will produce an error.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-02-23T08:15:22.403" />
  <row Id="7531" PostTypeId="1" AcceptedAnswerId="7545" CreationDate="2011-02-23T08:23:08.650" Score="3" ViewCount="194" Body="&lt;p&gt;I have the data about process duration (in minutes) and components (procedures) done during it like this (CSV):&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;id,time,p1,p2,p3,p4&#xA;1,30,1,0,0,0&#xA;2,32,1,0,0,0&#xA;3,56,1,1,0,0&#xA;4,78,1,1,0,1&#xA;5,78,1,1,0,1&#xA;6,100,1,1,1,1&#xA;7,98,0,1,1,1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I need to estimate the duration of each component(procedure) &#xA;I want to get something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;component,timeMax,timeMin,timeAverage,timeSD,samples&#xA;p1,...&#xA;p2,....&#xA;p3,...&#xA;p4,....&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note:&lt;/strong&gt; I need the estimated time of procedures not time of processes where procedure was used. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think the solution shold initially group all combinations first and then &#xA;simple procedures (1 process = 1 procedure ) time shuld be evaluated&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$t_1 = 30$$ #id=1&#xA;$$t_1 = 32$$ #id=2&lt;/p&gt;&#xA;&#xA;&lt;p&gt;then more complex actions should be performed: for example, time of procedure 2 (from sample) could be calculated by subtraction: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$t_1 = \sum{(t_1+t_2+t_3+t_4)} - \sum{(t_2+t_3+t_4)} = 100 - 98 = 2$$ # id 6 - 7&#xA;$$t_2 = \sum{(t_1+t_2)} - t_1 = 56 - 30\pm1 = 26\pm1$$ #id 3 - (1,2)&#xA;$$t_3 = \sum{(t_1+t_2+t_3+t_4)} - \sum{(t_1+t_2+t_4)} = 100 - 78 = 22$$ # id 6 - 5&#xA;$$t_4 = \sum{(t_1+t_2+t_4)} - \sum{(t_1+t_2)} = 78 - 56 = 22$$ #id 4 - 3&lt;/p&gt;&#xA;&#xA;&lt;p&gt;then average, SD, Min,Max for all $t_i$ is calculated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If there are several procedures are always combined time for each is calculated by dividing combination time by combination size. I think this procedure should be performed only before result output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;May be also some kind of correction for procedures that are performed during this sequence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;May be there should be iteration limit, or stop condition then last iteration brings no or &amp;lt;1% result change in comparison with previous one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second part is to compare procedure times when it is done separately and in combination with others. And to estimate most effective (reducing total time) and ineffective (increasing total time) procedure combinations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The question is:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;How to achieve this?&lt;/li&gt;&#xA;&lt;li&gt;What methods should/could be used?&lt;/li&gt;&#xA;&lt;li&gt;What statistical software could be used for this task?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="3376" LastEditorUserId="3376" LastEditorDisplayName="" LastEditDate="2011-02-24T10:48:25.453" LastActivityDate="2011-02-25T13:23:21.070" Title="Multiple regression with binary predictors. Component value analysis." Tags="&lt;methodology&gt;&lt;multiple-regression&gt;" AnswerCount="2" CommentCount="3" />
  <row Id="7532" PostTypeId="1" AcceptedAnswerId="7540" CreationDate="2011-02-23T09:50:38.280" Score="5" ViewCount="346" Body="&lt;p&gt;Dear researchers &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to ask you, what is the correct number of Lags in ARCH LM Test?  I am referring to ArchTest in FinTS package, but other ArchTest (such as the one in Eviews) provide same results. In many time series, when I choose Lags between 1:5 the p.value is usually higher than 0.05, but with increasing of Lags, p.value becomes smaller. So how to do the correct decision if for lag=1, the time series looks homoscedastic(NO ARCH Effects), but for lags=5 and lags=12 result is heteroscedastic (presence of ACH) or reverse? &#xA;Thank you &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sincerely Jan &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#Example code in R&#xA;library(quantmod) &#xA;library(FinTS) &#xA;getSymbols(&quot;XPT/USD&quot;,src=&quot;oanda&quot;) &#xA;ret_xptusd&amp;lt;-as.numeric(diff(log(XPTUSD))) &#xA;ones&amp;lt;-rep(1,500) &#xA;ols&amp;lt;-lm(ts(ret_xptusd)~ones);ols &#xA;residuals&amp;lt;-ols$residuals &#xA;&#xA;ArchTest(residuals,lags=1)   # p-value = 0.008499 &#xA;ArchTest(residuals,lags=5)   # p-value = 0.08166 &#xA;ArchTest(residuals,lags=12)  #p-value = 0.2317 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="3378" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-02-23T11:39:21.630" LastActivityDate="2011-02-23T15:21:49.200" Title="How to choose number of lags in ARCH models using ARCH LM test?" Tags="&lt;time-series&gt;&lt;garch&gt;" AnswerCount="1" />
  <row Id="7533" PostTypeId="2" ParentId="7531" CreationDate="2011-02-23T09:52:39.107" Score="1" ViewCount="" Body="&lt;p&gt;Software:&#xA;I advice to use R because it is free and designed for data analysis.&#xA;&lt;a href=&quot;http://cran.r-project.org/&quot; rel=&quot;nofollow&quot;&gt;http://cran.r-project.org/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;About your first question, the R code to answer it, is the following (I suppose that your data are in the file &quot;data.csv&quot;)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# load the file &quot;data.csv&quot;&#xA;d &amp;lt;- read.table(&quot;data.csv&quot;,header=T,sep=&quot;,&quot;)&#xA;&#xA;# create a data frame for the results&#xA;res &amp;lt;- data.frame(component=paste(&quot;op&quot;,1:(ncol(d)-2),sep=&quot;&quot;),timeMax=rep(0,ncol(d)-2),timeMin=rep(0,ncol(d)-2),timeAverage=rep(0,ncol(d)-2),timeSD=rep(0,ncol(d)-2),samples=rep(0,ncol(d)-2))&#xA;&#xA;# loop over the operations&#xA;for (ind in 3:ncol(d))&#xA;    {&#xA;    res[ind-2,2:6]&amp;lt;-c(max(d$time*d[,ind]),min(d$time*d[,ind]),mean(d$time*d[,ind]),sd(d$time*d[,ind]),sum(d[,ind]))&#xA;    }&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For the question about the combinations, you could use the same trick by replacing&#xA;&lt;code&gt;d$time*d[,ind]&lt;/code&gt;&#xA;by&#xA;&lt;code&gt;d$time*d[,ind1]*d[,ind2]&lt;/code&gt;&#xA;if you want to obtain statistics about the combination of op1 and op2. But if you have many op, this is maybe not suited to your case as the number of combinations is equal to 2^N... Is N large? Do you want statistics about all the possible combinations or only about some of them?&lt;/p&gt;&#xA;" OwnerUserId="3377" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-02-23T13:46:38.320" LastActivityDate="2011-02-23T13:46:38.320" CommentCount="3" />
  <row Id="7534" PostTypeId="1" CreationDate="2011-02-23T10:12:20.310" Score="1" ViewCount="78" Body="&lt;p&gt;hi! i have a contigency table yet it is not fix. it varies depending on on the data. i want to know how significant is a particular cell compared to other cells. Is there a method of doing this?&lt;/p&gt;&#xA;" OwnerUserId="3379" LastActivityDate="2011-05-27T04:50:39.487" Title="How do you determine if a particular cell of an ixj contingency table is statistically significant among all other cells?" Tags="&lt;contingency-tables&gt;" AnswerCount="2" />
  <row Id="7535" PostTypeId="1" AcceptedAnswerId="7537" CreationDate="2011-02-23T11:05:14.260" Score="10" ViewCount="239" Body="&lt;p&gt;What is the appropriate strategy for deciding which model to use with count data? &#xA;I have count data that i need to model as a multilevel model and it was recommended to me (on this site) that the best way to do so this is through bugs or MCMCglmm. However i am still trying to learn about bayesian statistics, and i thought i should first try to fit my data as generalized linear models and ignore the nested structure of the data (just so i can get a vague idea of what to expect). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;About 70% of the data are 0 and the ratio of variance to the mean is 33. So the data is quite over-dispersed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After trying a number of different options (including poisson, negative binomial, quassi and zero inflated model)  i see very little consistency in the results (varying from everything is significant to nothing is significant). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can i go about making an informed decision about which type of model to choose based on the 0 inflation and over-dispersion? &#xA;For instance, how can i infer that quassi-poisson is more appropriate than negative binomial (or vise versa) and how can i know that using either has dealt adequately (or not) with the excess zeros? &#xA;Similarly, how do i evaluate that there is no more over-dispersion if a zero-inflated model is used? or how should i decide between a zero inflated poisson and a zero inflated negative binomial? &lt;/p&gt;&#xA;" OwnerUserId="1871" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-02-23T21:29:37.407" LastActivityDate="2011-05-06T19:48:18.143" Title="Strategy for deciding appropriate model for count data" Tags="&lt;poisson&gt;&lt;glm&gt;&lt;negative-binomial&gt;&lt;overdispersion&gt;" AnswerCount="4" FavoriteCount="4" />
  <row Id="7536" PostTypeId="2" ParentId="6330" CreationDate="2011-02-23T13:29:56.240" Score="2" ViewCount="" Body="&lt;p&gt;By Their Fruits Ye Shall Know Them&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The assumption (to be tested) is that the errors from the model have constant variance. Note this does not mean the errors from an assumed model. When you use a simple graphical analysis you are essentially assuming a linear model in time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus if you have an inadequate model such as might be suggested by a casual plot of the data against time you may incorrectly conclude about the need for a power transform. Box and Jenkins did do in their Airline Data example. They did not not account for 3 unusual values in the most recent data thus they incorrectly concluded that there was higher variation in the residuals at the highest level of the series.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For more on this subject please see &lt;a href=&quot;http://www.autobox.com/pdfs/vegas_ibf_09a.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.autobox.com/pdfs/vegas_ibf_09a.pdf&lt;/a&gt;    &lt;/p&gt;&#xA;" OwnerUserId="3382" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2011-02-23T20:20:26.387" LastActivityDate="2011-02-23T20:20:26.387" />
  <row Id="7537" PostTypeId="2" ParentId="7535" CreationDate="2011-02-23T14:24:38.070" Score="6" ViewCount="" Body="&lt;p&gt;You can always compare count models by looking at their predictions (preferrably on a hold out set). J. Scott Long discusses this graphically (plotting the predicted values against actuals). His text book &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0803973748&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; describes in details but you can also look at &lt;a href=&quot;http://www.stata-journal.com/sjpdf.html?articlenum=st0002&quot; rel=&quot;nofollow&quot;&gt;6.4 on this document&lt;/a&gt;. &lt;br/&gt;&lt;br/&gt;You can compare models using AIC or BIC and there is also a test called Voung test that I am not terribly familiar with but can compare zero inflated to non nested models. Here is a Sas paper describing it briefly on page &lt;a href=&quot;http://www2.sas.com/proceedings/forum2008/371-2008.pdf&quot; rel=&quot;nofollow&quot;&gt;10&lt;/a&gt; to get you started. It also is implmented in R &lt;a href=&quot;http://tolstoy.newcastle.edu.au/R/e2/help/06/09/1538.html&quot; rel=&quot;nofollow&quot;&gt;posting&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="2040" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-02-24T07:37:31.597" LastActivityDate="2011-02-24T07:37:31.597" CommentCount="1" />
  <row Id="7538" PostTypeId="1" CreationDate="2011-02-23T15:01:40.443" Score="2" ViewCount="149" Body="&lt;p&gt;I have a data file that looks a bit like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;A 0.2  0.5&#xA;B 0.65 0.8&#xA;C 0.4  0.2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;i.e., it contains three columns where the first column contains labels and the other two columns float values. Columns are separated by spaces.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd like to plot this in a way that the labels appear as tics on the x-axis while the colums are plotted as two differently colored barcharts on top of each other.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I achieve this using gnuplot?&lt;/p&gt;&#xA;" OwnerUserId="3383" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-25T19:08:42.357" LastActivityDate="2011-04-25T19:08:42.357" Title="How to display a stacked barchart in Gnuplot?" Tags="&lt;data-visualization&gt;&lt;gnuplot&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="7539" PostTypeId="2" ParentId="7535" CreationDate="2011-02-23T15:20:31.053" Score="3" ViewCount="" Body="&lt;p&gt;A couple things to add to what B_Miner said:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) You wrote that the models varied from &quot;everything significant&quot; to &quot;nothing significant&quot; but this is not a good way to compare models.  Look, instead, at predicted values (as B_miner suggested) and effect sizes.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) If 70% of the data are 0, I can't imagine that a model without 0 inflation is appropriate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) Even if you don't want to go Bayesian, you can use GLMMs in SAS (PROC GLIMMIX or NLMIXED) and in R (various packages).  Ignoring the nested nature may mess everything up.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;4) In general, deciding on which model is best is an art, not a science.  There are statistics to use, but they are a guide to judgment.  Just looking at what you wrote, I would say a ZINB model looks right&lt;/p&gt;&#xA;" OwnerUserId="686" LastActivityDate="2011-02-23T15:20:31.053" CommentCount="1" />
  <row Id="7540" PostTypeId="2" ParentId="7532" CreationDate="2011-02-23T15:21:49.200" Score="4" ViewCount="" Body="&lt;p&gt;Arch LM tests whether coefficients in the regression:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$a_t^2=\alpha_0+\alpha_1 a_{t-1}^2+...+\alpha_p a_{t-p}^2+e_t$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;are zero, where $a_t$ is either observed series which we want to test for ARCH effects. So the null hypothesis is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\alpha_0=\alpha_1=...=\alpha_p=0$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If hypothesis is accepted then we can say that series have no ARCH effects. If it is rejected then one or more coefficients are non zero and we say that there are ARCH effects.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here we have classical regression problem of joint hypotheses versus individual hypothesis. When more regressors are included the regression is jointly insignificant, although a few regressors seem to be significant. All the introductory books about regression usually have chapter dedicated to this. The key motive is that joint hypotheses take into account all the interactions, when individual hypotheses do not. So in this case the statistic with few lags do not take into account the effects of more lags.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When statistical tests give conflicting results, for me it is an indication that data should be reexamined. Statistical tests usually have certain assumptions, which data may violate. In your case if we look at the graph of the series, we see a lot of zeroes. &#xA;&lt;img src=&quot;http://i.imgur.com/IUZro.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So this is not an ordinary time series and I would hesitate to use plain ARCH model.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-02-23T15:21:49.200" CommentCount="1" />
  <row Id="7541" PostTypeId="2" ParentId="7534" CreationDate="2011-02-23T15:24:51.763" Score="0" ViewCount="" Body="&lt;p&gt;I am not sure if I understand the first part of your question - everything varies depending on the data, does it not?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are ways to partition chi-square; I am not near my references, but this article seems apropos &lt;a href=&quot;http://www.jstor.org/pss/2283933&quot; rel=&quot;nofollow&quot;&gt;http://www.jstor.org/pss/2283933&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you know in advance which cell you will want to test, you can simply combine the other rows or columns.  &lt;/p&gt;&#xA;" OwnerUserId="686" LastActivityDate="2011-02-23T15:24:51.763" CommentCount="1" />
  <row Id="7542" PostTypeId="1" CreationDate="2011-02-23T16:03:54.663" Score="3" ViewCount="280" Body="&lt;p&gt;Hi to all,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have to demonstrate that a generator of VoIP calls generates calls uniformly distributed between callers.&#xA;In particular the distribution is the uniform (min, max) one where the volume per caller distribution is uniformly distributed between a minimum and maximum. So by running a test with 10000 users and a min value equal to 30 calls per week and a max value equal to 90 calls/week i obtain that not all the users respect this limits.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The situation is depicted in figure:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The few users that generate &amp;lt;30 or &gt;90 calls spoil the chi-square goodness of fit test and I don't know how can i proceed with a goodness of fit test. Most of the values is within the interval (and from these we obtain low chi-square value), but the few out-of-range values spoil the final chi-square calculation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your opinion what is the best way to operate? What should I do with the &quot;out-of-range&quot;values? Thank you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: the chi-square goodness-of-fit test performed is reported in the following figure:&#xA;&lt;img src=&quot;http://i.imgur.com/ulmcI.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where still I don't know what to do with the out-of-range values. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;after talking with people linked with the project we concluded that the generator does not satisfy the uniform distribution. We have to do a theoretic analysis of what we expect really to be the distribution at the end of the generation based on the inputs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This means that I have to do it!&#xA;More details: &#xA;The generator assigns a &quot;probability&quot; between 0 and 1 to the callers (with a particular method, that probably is the problem). Then it generates a random value from 0 to 1 and it finds the associated user and assigns the call to him.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The generator generates calls for a week with the constant rate equal to 1 call per second, this means that it generates ca. 604800 total calls.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My goal is to distribute the callers between the min and max number of calls in a week. For example if I have 10000 users and the min limit is equal to 30 calls per week and max = 90 calls per week I should obtain something about:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;30 calls : 163 users.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;31 calls : 163 users.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;....&lt;/p&gt;&#xA;&#xA;&lt;p&gt;90 calls : 163 users.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So 163 users generate 30 calls in a week..etc, etc and finally 163 users generate 90 calls in a week. How should I assign the probability to callers in order that the generator distributes the callers uniformly between the range 30-90?&lt;/p&gt;&#xA;" OwnerUserId="3342" LastEditorUserId="3342" LastEditorDisplayName="" LastEditDate="2011-03-18T09:55:27.057" LastActivityDate="2011-05-17T10:47:06.473" Title="How to perform goodness of fit test and how  to assign probability with uniform distribution (min,max)?" Tags="&lt;distributions&gt;&lt;chi-squared&gt;&lt;goodness-of-fit&gt;&lt;uniform&gt;" AnswerCount="2" CommentCount="5" FavoriteCount="1" />
  <row Id="7543" PostTypeId="2" ParentId="7084" CreationDate="2011-02-23T16:21:47.927" Score="5" ViewCount="" Body="&lt;h3&gt;The question:&lt;/h3&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;How can normality be validated without using visual cues such as QQ plots? (the validation will be a part of larger software)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Can a &quot;goodness of fit&quot; score be calculated?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Although enumerated separately, these parts are (appropriately) one question: you compute an appropriate goodness of fit and use that as a test statistic in a hypothesis test.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Some answers&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;There are plenty of such tests; the best among them are the Kolmogorov-Smirnov, Shapiro-Wilks, and Anderson-Darling tests.  Their properties have been extensively studied.  An excellent resource is the work of M. A. Stephens, especially the 1974 article, &lt;a href=&quot;http://www.jstor.org/pss/2286009&quot; rel=&quot;nofollow&quot;&gt;EDF Statistics for Goodness of Fit and Some Comparisons&lt;/a&gt;.  Rather than supply a long list of references, I will leave it to you to Google this title: the trail quickly leads to useful information.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One thing I like about Stephens' work, in addition to the comparisons of the properties of various GoF tests, is that it provides clear descriptions of how to compute the statistics and how to compute, or at least approximate, their null distributions.  This gives you the option to implement your favorite test yourself.  The EDF statistics (empirical distribution function) are easy to compute: they tend to be linear combinations of the order statistics, so all you have to do is sort the data and go.  The complications concern (a) computing the coefficients--this used to be a barrier in applying the S-W test, but good approximations now exist--and (b) computing the null distributions.  Most of those can be computed or have been adequately tabulated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is characteristic about any GoF tests for distributions is that (a) they need a certain amount of data to become powerful (for detecting true deviations) and (b) very quickly thereafter, as you acquire more data, they become &lt;em&gt;so&lt;/em&gt; powerful that deviations that are &lt;em&gt;practically&lt;/em&gt; inconsequential become &lt;em&gt;statistically significant.&lt;/em&gt;  (This is very well known and is easily confirmed with simulation or mathematical analysis.)  In this is the origin of the reluctance to answer the original question without obtaining substantial clarification.  If you have a few hundred values or more, you will find that any of these tests demonstrate your data are not &quot;normal.&quot;  But does this matter for your intended analysis?  We simply cannot say.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-02-23T16:21:47.927" CommentCount="4" />
  <row Id="7544" PostTypeId="2" ParentId="7084" CreationDate="2011-02-23T18:11:56.453" Score="0" ViewCount="" Body="&lt;p&gt;I would maybe (I don't know if it is feasible in your context) suggest another approach.&#xA;You could force your experimental data to follow a standard normal distribution by applying a normal quantile tranformation on it. The principle is to &lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) rank your values from high to low&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) assign the value of r-th rank the (r-0.5)/n th quantile of the standard normal distribution. This ensures that your data is N(0,1)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) perform the analysis on the transformed data&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I feel that if your original data is close to normal this would not change much your inference. Then of course you can simulate data from a standard normal distribution&#xA;and it will fit your experimental data.&lt;/p&gt;&#xA;" OwnerUserId="3386" LastActivityDate="2011-02-23T18:11:56.453" CommentCount="2" />
  <row Id="7545" PostTypeId="2" ParentId="7531" CreationDate="2011-02-23T18:21:58.147" Score="5" ViewCount="" Body="&lt;p&gt;I don't think the problem, as is, is well-defined. You mention the possibility that the duration of each operation may vary if done in combination with others. If that's the case, the &quot;duration of operation&quot; is not defined. E.g. in your example, op1 time is 30-32 min, and you say &quot;therefore op2 time is 24-26 min&quot;, but how do you know that? Maybe op1, when done together with op2, takes only 10 min and op2 takes the remaining 46 min. So you'd need some more assumptions to figure out individual durations from such data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you assume the operations are independent, then it seems that an easy first step would be to build a set of linear equations. In your example, the first three equations would be:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$t_1 = 30$$&#xA;$$t_1 = 32$$&#xA;$$t_1 + t_2 = 56$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then solve it (in the least squares sense). The way to solve it is probably using linear regression, with operation durations as independent variables and the process times as (known) dependent variables. Standard regression solvers will give you all kinds of estimates of variance. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Solution for R: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;d &amp;lt;- read.table(&quot;data.csv&quot;,header=T,sep=&quot;,&quot;) # read data&#xA;r &amp;lt;- lm(time ~ 0+p1+p2+p3+p4, data=d) # multiple linear regression &#xA;summary(r) #result output&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="3369" LastEditorUserId="3376" LastEditorDisplayName="" LastEditDate="2011-02-25T13:23:21.070" LastActivityDate="2011-02-25T13:23:21.070" CommentCount="14" />
  <row Id="7546" PostTypeId="1" CreationDate="2011-02-23T18:29:52.990" Score="5" ViewCount="149" Body="&lt;p&gt;For R, I understand that the package lme4 and the function glmer roughly corresponds to glimmix in SAS.  What is the default covariance structure when fit and can it be changed?  If so how?&lt;/p&gt;&#xA;" OwnerUserId="1364" LastActivityDate="2011-02-24T07:51:30.473" Title="What is the default covariance structure in glmer and can I change it?" Tags="&lt;r&gt;&lt;mixed-model&gt;&lt;variance-covariance&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="7547" PostTypeId="2" ParentId="7542" CreationDate="2011-02-23T19:43:14.900" Score="0" ViewCount="" Body="&lt;p&gt;What you are describing resembles a &quot;continuous uniform distribution&quot;,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://mathworld.wolfram.com/UniformDistribution.html&quot; rel=&quot;nofollow&quot;&gt;http://mathworld.wolfram.com/UniformDistribution.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-Ralph Winters&lt;/p&gt;&#xA;" OwnerUserId="3489" LastActivityDate="2011-02-23T19:43:14.900" CommentCount="1" />
  <row Id="7548" PostTypeId="2" ParentId="4200" CreationDate="2011-02-23T19:56:53.833" Score="1" ViewCount="" Body="&lt;p&gt;You should consider the Cross Correlation Function as that is meant to identify the lead/lag relationship.  Dirk had mentioned the Autocorrelation Function, but that is meant for just one single time series and not for multivariate. You should consider looking at the Box-Jenkins textbook Chapter 10 where they introduce the steps do this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You say your data is noisy, but if it has a pattern where the lead/lag response is strong then you will find significance.  &lt;/p&gt;&#xA;" OwnerUserId="3382" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2011-02-23T20:19:48.650" LastActivityDate="2011-02-23T20:19:48.650" />
  <row Id="7549" PostTypeId="1" CreationDate="2011-02-23T20:03:53.400" Score="6" ViewCount="112" Body="&lt;p&gt;As the title suggests, I'm pretty well befuddled about which approach makes the most sense for my data. Let me try to succinctly explain the problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have binary choice data representing whether a specific person for a specific event took the train or bus. I have event level predictors (location of event, duration of event) as well as person-level predictors (income level, education level). There are multiple, but unbalanced, events per person.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's the slightly unusual part: I have a bunch of historic info with all predictor values as well as observed choice. I want to build a regression model from that I can then apply to new data (consisting of everything except education level) to infer with as much confidence as possible that person's education, based on their observed choices.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My thoughts on how to do this:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Build a mixed-effect, multilevel logistic regression model, with transportation choice as my dependent variable, and education_level as one of the predictors. Now solve for education_level using something like inverse logistic regression.&lt;/li&gt;&#xA;&lt;li&gt;Do a regression on counts. Now, education is the dependent variable, and we sum up counts of each subset of predictor variables we've seen (eg, there were 5 nearby events where rich males took the bus, 3 faraway events where...)   &lt;/li&gt;&#xA;&lt;li&gt;Some kind of latent class model?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;What are the tradeoffs between these alternatives? Also, are there still other approaches worth examining (eg, CFA)? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(And please let me know if I need to provide more detail on the problem.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you for your time, &#xA;Ian.&lt;/p&gt;&#xA;" OwnerUserId="3387" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-02-23T21:24:22.357" LastActivityDate="2011-02-24T16:39:05.287" Title="Inverse logistic regression vs. repeated-measures vs. latent class?" Tags="&lt;regression&gt;&lt;latent-class&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="7550" PostTypeId="2" ParentId="7499" CreationDate="2011-02-23T20:06:57.793" Score="4" ViewCount="" Body="&lt;p&gt;Adding error bars to a bar graph is a choice you make as a presenter to communicate more information to your audience.  They are useful because they communicate visually how certain you can be, based on your data, of the specific values you are presenting.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In some cases, there is no uncertainty.  Imagine you are graphing the number of students in each grade in a school district.  These numbers are known so presenting the exact values without error bars makes sense.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If however you are graphing the height of students by gender and you only had time to measure students in one class, you are making statistical inferences about the larger population.  In this case, error bars are helpful to communicate the range of likely true values. If the 15 boys in the class you measured averaged 48 inches you could include error bars to show that you are 95% sure that the average of all boys in the district is between say, 46 and 50.&lt;/p&gt;&#xA;" OwnerUserId="3388" LastActivityDate="2011-02-23T20:06:57.793" />
  <row Id="7551" PostTypeId="1" AcceptedAnswerId="7553" CreationDate="2011-02-23T20:22:00.287" Score="3" ViewCount="194" Body="&lt;p&gt;I am running an ordinal logistic regression in R and running into trouble when I include dummy variables.  My model works great with my first set of predictors.  Next I want to add dummy variables for each of the years represented in my dataset.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I created the dummy variables with &lt;code&gt;car:recode&lt;/code&gt; in this manner (one statement like this for each of the 11 years)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;fsd$admityear2000 &amp;lt;- recode(fsd$ApplicationYear ,&quot;2000=1;else=0&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The lrm model is specified as follows&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(Design)&#xA;ddist&amp;lt;- datadist(fsd)&#xA;options(datadist='ddist')&#xA;m4 &amp;lt;- lrm(Outcome ~ relGPA + mcAvgGPA + Interview_Z + WorkHistory_years + GMAT + UGI_Gourman + admityear1999 + admityear2000 + admityear2001 + admityear2002 + admityear2003 + admityear2004 + admityear2005 + admityear2006 + admityear2007 + admityear2008 + admityear2009, data=fsd)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(sorry for all of the other random variables, but I don't want to introduce confusion by changing my code)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I get the error&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;singular information matrix in lrm.fit (rank= 22 ).  Offending variable(s):&#xA;admityear2009 admityear2000 admityear1999 &#xA;Error in lrm(Outcome ~ relGPA + mcAvgGPA + Interview_Z + WorkHistory_years +  : &#xA;  Unable to fit model using “lrm.fit”&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I understand that including all options of a dummy variable over-defines the model, but I get the error whether I include all 11 years or just 10.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found a suggestion &lt;a href=&quot;http://r.789695.n4.nabble.com/Singular-information-matrix-in-lrm-fit-td869221.html&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; to set the penalty parameter of &lt;code&gt;lrm&lt;/code&gt; to a &lt;em&gt;small positive value&lt;/em&gt;.  Setting it to 1 or 5 changes the error such that it only names one of the variables as offending.  The error doesn't go away even with &lt;code&gt;penalty=100&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm pretty new to R, but loving the freedom so far.  Thanks for any help!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Responses and Lessons&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Factors are awesome and I can't believe I didn't notice them earlier. Man that cleans up my code a lot.  Thanks!&lt;/li&gt;&#xA;&lt;li&gt;My DV, 'Outcome' is indeed ordinal and after making it a factor(), I also made it ordered().&lt;/li&gt;&#xA;&lt;li&gt;The str() command is also awesome and this is what my data now looks like (with some of the non-relevant variables omitted)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;output:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;str(fsd)     &#xA;Outcome      : Ord.factor w/ 3 levels &quot;0&quot;&amp;lt;&quot;1&quot;&amp;lt;&quot;2&quot;&#xA;relGPA       : num  &#xA;mcAvgGPA     : num &#xA;admitschool  : Factor w/ 4 levels &quot;1&quot;,&quot;2&quot;,&quot;3&quot;,&quot;4&quot;&#xA;appyear      : Factor w/ 11 levels &quot;1999&quot;,&quot;2000&quot;,..&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;both lrm() and polr() now run successfully, and they both deal with appyear by dropping some values of the factor.  lrm() drops 1999, 2000, and 2001 while polr() just drops 1999 and 2000.  lrm() gives no warnings while polr() says &quot;design appears to be rank-deficient, so dropping some coefs.&quot;  This is an improvement, but I still don't understand why more than one value needs to be dropped. xtabs shows that there isn't full seperation right?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;output:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;xtabs(~fsd$appyear + fsd$Outcome)&#xA;           fsd$Outcome&#xA;fsd$appyear    0    1    2&#xA;       1999 1207  123  418&#xA;       2000 1833  246  510&#xA;       2001 1805  294  553&#xA;       2002 1167  177  598&#xA;       2003 4070  158 1076&#xA;       2004 2803  106 1138&#xA;       2005 3749  513 2141&#xA;       2006 4429  519 2028&#xA;       2007 6134  670 1947&#xA;       2008 7446  662 1994&#xA;       2009 4411   86 1118&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="3388" LastEditorUserId="3388" LastEditorDisplayName="" LastEditDate="2011-02-24T16:55:33.830" LastActivityDate="2011-02-24T16:55:33.830" Title="Singular information matrix error in lrm.fit in R" Tags="&lt;r&gt;&lt;logistic-regression&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="7552" PostTypeId="2" ParentId="7538" CreationDate="2011-02-23T21:07:29.020" Score="0" ViewCount="" Body="&lt;p&gt;I can't answer this question myself, but you might try asking on the dedicated Gnuplot forum &lt;a href=&quot;http://old.nabble.com/Gnuplot-f1920.html&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;. In the past I have always got an answer to my problems with Gnuplot from this forum.&lt;/p&gt;&#xA;" OwnerUserId="226" LastActivityDate="2011-02-23T21:07:29.020" CommentCount="1" />
  <row Id="7553" PostTypeId="2" ParentId="7551" CreationDate="2011-02-23T22:45:27.753" Score="4" ViewCount="" Body="&lt;p&gt;Creating dummy variables should not be necessary. You should just use factors when modeling in R.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;admityear &amp;lt;- factor(admityear) &#xA;m4 &amp;lt;- lrm(Outcome ~ relGPA + mcAvgGPA + Interview_Z + WorkHistory_years + &#xA;                       GMAT + UGI_Gourman + admityear, data=fsd)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If the singular condition still  persists, then you have multicollinearity and need to try dropping other variables. (I would be suspicious of WorkHistory_years.)  I also don't see anything ordinal about that model. Ordinal logistic regression in the rms package (or the no longer actively supported Design package) is done with polr(). And it would be really helpful to see the results from str(fasd).&lt;/p&gt;&#xA;" OwnerUserId="2129" LastActivityDate="2011-02-23T22:45:27.753" CommentCount="3" />
  <row Id="7554" PostTypeId="1" CreationDate="2011-02-24T00:05:24.610" Score="1" ViewCount="151" Body="&lt;p&gt;I have data about how many unique users do a certain thing for each day of a month. I can average it, and i would like to display the variation in a intuitive format (such as % of  something).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a standard way of doing this? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've found standard error, which is stddev/sqrt(n), which is not particularly intuitive for users of the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If anyone needs clarifications, please ask. I'm not too clear about this myself.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: in response to the answers, it's for building out a analytics dashboard for use by the entire company (so many people probably don't understand standard deviation). In particular, we are doing A/B testing for various metrics over a period of say 1 month. We basically would average all the metric per day to give a number for that period, but there are variations day-to-day and would like to have some good way of expressing that. &lt;/p&gt;&#xA;" OwnerUserId="3392" LastEditorUserId="3392" LastEditorDisplayName="" LastEditDate="2011-02-28T22:36:02.480" LastActivityDate="2011-02-28T23:04:16.287" Title="How to express error as a percentage?" Tags="&lt;variance&gt;" AnswerCount="4" />
  <row Id="7555" PostTypeId="1" AcceptedAnswerId="7573" CreationDate="2011-02-24T00:21:51.900" Score="2" ViewCount="249" Body="&lt;p&gt;I am trying to manage a meta-regression in SPSS17 using the effect size as the dependent variable. I want to explore if my independent variables affects the effect size. Some small practical questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;What is the minimum number of studies necessary for a meta-regression?&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Some people suggest at least 10 studies are required. Why not 20 or 5 studies? &lt;/li&gt;&#xA;&lt;li&gt;Is the total sample size an important consideration? &lt;/li&gt;&#xA;&lt;li&gt;Why would 10 studies with 200 patients be enough, but 5 studies with 400 patients not be enough?&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Can I enter all three regressors at once and report the global model, or do I have to enter one regressor at a time and report 3 models each one separately? &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;How does the correlation between the independent variables affects this choice? &lt;/li&gt;&#xA;&lt;li&gt;How does the number of the studies affect the number of independent variables that I   should enter simultaneously?&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Does the independent variable have to be a scale variable?&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The dependent variable (=effect size) is, of course, scale. The independent variable must be also scale, or could be ordinal or nominal?&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;How can I weight my effect size for sample size?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;What is the preferable level of significance? &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Is p&amp;lt;0.05 still acceptable for clinical research in such an analysis?&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="3333" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-02-24T07:22:39.853" LastActivityDate="2011-02-24T15:18:02.870" Title="How to do meta-regression in SPSS?" Tags="&lt;spss&gt;&lt;meta-analysis&gt;&lt;sample-size&gt;&lt;meta-regression&gt;" AnswerCount="1" CommentCount="4" FavoriteCount="1" />
  <row Id="7556" PostTypeId="2" ParentId="7554" CreationDate="2011-02-24T00:28:50.507" Score="2" ViewCount="" Body="&lt;p&gt;It may be meaningful to just divide by the average. E.g. the average number is 1000 and the std is 200, so in a sense this means the actual number can vary by 20% from the baseline.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, if you could say who the user of the data are and what they are doing with it, it might be useful.&lt;/p&gt;&#xA;" OwnerUserId="3369" LastActivityDate="2011-02-24T00:28:50.507" CommentCount="3" />
  <row Id="7557" PostTypeId="2" ParentId="7528" CreationDate="2011-02-24T00:34:36.170" Score="1" ViewCount="" Body="&lt;p&gt;Yes, I think that this approach can be use with other types of effect sizes as long as they are (approximately) normally distributed (that's why you use $log(OR)$; to be more precise, the errors of the linear regression model need to be $N(0,1)$).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your regression equation is wrong. It is $\overline{ES} = a + b \cdot \frac{1}{N}$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furthermore, it is a &lt;em&gt;weighted&lt;/em&gt; regression. So, unfortunately, you still need the standard errors. &lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pubmed/11223905&quot; rel=&quot;nofollow&quot;&gt;Macascill et al (2001: 644)&lt;/a&gt; write: &quot;The observations are weighted by the inverse variance of the estimate to allow for possible heteroscedasticity (FIV)&quot;. However, since I &lt;a href=&quot;http://stats.stackexchange.com/questions/7426/eggers-test-in-spss/7428#7428&quot;&gt;know&lt;/a&gt; that you can compute the standard errors, this shouldn't be a problem (trust the authors ;-).&lt;/p&gt;&#xA;" OwnerUserId="307" LastActivityDate="2011-02-24T00:34:36.170" CommentCount="5" />
  <row Id="7558" PostTypeId="2" ParentId="7554" CreationDate="2011-02-24T01:02:48.370" Score="1" ViewCount="" Body="&lt;p&gt;The standard error of the mean tells you how precise your estimate of the mean is; that doesn't seem to capture what you're trying to do.  I would use either a) a histogram, if you care mostly about showing variation, or b) a line chart or area chart, if you want to say something about variation while also showing progression over time.&lt;/p&gt;&#xA;" OwnerUserId="2669" LastActivityDate="2011-02-24T01:02:48.370" />
  <row Id="7559" PostTypeId="1" AcceptedAnswerId="7560" CreationDate="2011-02-24T03:01:15.167" Score="4" ViewCount="154" Body="&lt;p&gt;The linear SVM in textbook takes form of maximizing&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$L_D = \sum_i{a_i} - \frac{1}{2}\sum_{i,j}{a_ia_jy_iy_jx_i^Tx_j}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;over $a_i$ where $a_i \geq 0$ and $\sum_i{a_iy_i} = 0$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since $w = \sum_i{a_iy_ix_i}$, the classifier will take the form $\text{Sgn}(wx - b)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, it seems to solve linear SVM, I need to figure out $a_i$ with some gradient based methods. However, recently, I came across a paper which states that they try to minimize the following form:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$L_P = \frac{1}{2}||w||^2+C\sum_i{\text{max}(0, 1-y_if_w(x_i))}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and they claim $C$ is a constant. It seems to me that this form is quite different from primal form of $L_P$ in linear SVM because of the missing $a_i$. As far as the paper goes, it seems to me that they optimized on $w$ directly. I am puzzled here as if I missed something. Can you optimize $w$ directly on linear SVM? Why is that possible?&lt;/p&gt;&#xA;" OwnerUserId="3395" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-02-24T07:24:09.663" LastActivityDate="2011-02-24T07:24:09.663" Title="The difference between linear SVM and other linear classifiers?" Tags="&lt;svm&gt;&lt;linear-model&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="7560" PostTypeId="2" ParentId="7559" CreationDate="2011-02-24T03:11:36.903" Score="6" ViewCount="" Body="&lt;p&gt;There are two things going on here.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Difference between primal and dual problem. The &quot;original&quot; objective function of SVM is to minimize $1/2 ||w||^2$. This is called &quot;primal form&quot;. Turns out that the objective function you wrote (the one involving $L_D$) is the dual form of this problem. So the two lead to equivalent solutions and can be used interchangeably.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The second formulation you describe is called &quot;soft margin SVM&quot;. It is obtained by taking the primal form of (1) above and replacing the constraint $y_i f_w(x_i) \geq 1$ by the penalty term $C \cdot \max(0, 1 - y_i f_w(x_i))$. The effect is that you allow violations of the constraint. This is useful e.g. if your data is not linearly separable. You can obtain a dual formulation of this (similar to your expression with $L_D$) as well.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="3369" LastActivityDate="2011-02-24T03:11:36.903" />
  <row Id="7561" PostTypeId="2" ParentId="7546" CreationDate="2011-02-24T07:51:30.473" Score="1" ViewCount="" Body="&lt;p&gt;I do not know about SAS, but variance in &lt;code&gt;glmer&lt;/code&gt; is controlled by &lt;code&gt;family&lt;/code&gt; argument. If you want to change correlation structure then I suspect you will have to use &lt;code&gt;nlme&lt;/code&gt; from &lt;strong&gt;nlme&lt;/strong&gt; package. &lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-02-24T07:51:30.473" />
  <row Id="7562" PostTypeId="1" AcceptedAnswerId="8390" CreationDate="2011-02-24T10:15:16.720" Score="4" ViewCount="134" Body="&lt;p&gt;I just got my hands on the &lt;a href=&quot;http://www.electionstudies.org/&quot; rel=&quot;nofollow&quot;&gt;ANES (American National Election Studies)&lt;/a&gt; 2008 data set, and would like to do some simple analysis in R. However, I've never worked with this complex of a data set before and I've run into an issue.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The survey uses oversampling and has a variable for post stratification weights. I had only the vaguest idea of what that meant, so I read the &lt;a href=&quot;http://en.wikipedia.org/wiki/Sampling_%28statistics%29#Stratified_sampling&quot; rel=&quot;nofollow&quot;&gt;wikipedia page&lt;/a&gt; on it, which I understand conceptually. Unfortunately, I don't know how to manipulate R such that the post stratification weights are reflected when I do my analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While conceptually, the idea of oversampling didn't confuse me, the following documentation for the R &quot;survey&quot; package is completely unintelligible to me. I'll show what I've found so far, and I would really appreciate either an explanation of what's going on with these methods, or, if anyone knows a simpler way to apply a post-stratification weight to a data frame of variables, I'd love to here that too. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, I found the &quot;survey&quot; package from CRAN, and I have the &lt;a href=&quot;http://cran.r-project.org/web/packages/survey/survey.pdf&quot; rel=&quot;nofollow&quot;&gt;manual&lt;/a&gt;, and, after looking through it, it seems that the most promising method is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;postStratify(design, strata, population, partial = FALSE, ...)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, when I look at the documentation for what needs to be passed for each of these arguments, I'm completely lost. They are as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;design           A survey design with replicate weights&#xA;&#xA;strata           A formula or data frame of post-stratifying variables&#xA;&#xA;population       A table, xtabs or data.frame with population frequencies&#xA;&#xA;partial          if TRUE, ignore population strata not present in the sample&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;None of these make a lot of sense to me, but I'm pretty sure that the design argument is supposed to be of a class also defined in this package:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;svydesign(ids, probs=NULL, strata = NULL, variables = NULL, fpc=NULL, &#xA;    data = NULL, nest = FALSE, &#xA;    check.strata = !nest, weights=NULL,pps=FALSE,...)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you notice, there are a ton of optional arguments here, which all seem to do similar types of things (at least to me, after reading the docs...). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm basically at a loss for why this is so complicated in R. Am I misunderstanding things? Is there a simpler way to do this? Any help would be appreciated.&lt;/p&gt;&#xA;" OwnerUserId="726" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-02-24T11:57:29.023" LastActivityDate="2011-03-17T00:40:10.997" Title="Simple post-stratification weights in R" Tags="&lt;r&gt;&lt;survey&gt;&lt;post-hoc&gt;&lt;stratification&gt;" AnswerCount="1" FavoriteCount="0" />
  <row Id="7563" PostTypeId="1" AcceptedAnswerId="7568" CreationDate="2011-02-24T10:16:34.377" Score="6" ViewCount="120" Body="&lt;p&gt;What techniques/approaches are useful in testing statistical software? I'm particularly interested in programs that do parametric estimation using maximum likelihood.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Comparing results to those from other programs or published sources is not always possible since most of the time when I write a program of my own it is because the computation I need is not already implemented in an existing system.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not insisting on methods which can guarantee correctness. I would be happy with techniques that can catch some fraction of errors.&lt;/p&gt;&#xA;" OwnerUserId="1393" LastActivityDate="2011-02-24T18:12:36.290" Title="Testing statistical software" Tags="&lt;software&gt;&lt;computing&gt;&lt;computational-statistics&gt;" AnswerCount="2" FavoriteCount="2" />
  <row Id="7564" PostTypeId="1" CreationDate="2011-02-24T10:53:47.720" Score="3" ViewCount="58" Body="&lt;p&gt;Hi all,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;as my first post I would like to know if there are SOM implementations (preferably R) available that accept fuzzy input. That is, I have data in which some nominal features are spread out between a number of categories. For example: feature 1 has 5 categories and an observation might have the values (which are actually probabilities) [0, 0.5, 0.25, 0.25, 0].&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks,&#xA;Figaro&lt;/p&gt;&#xA;" OwnerUserId="3401" LastEditorUserId="3401" LastEditorDisplayName="" LastEditDate="2011-02-24T13:15:07.433" LastActivityDate="2011-02-24T13:15:07.433" Title="Self-organizing maps: fuzzy input?" Tags="&lt;machine-learning&gt;&lt;neural-networks&gt;&lt;self-organizing-maps&gt;" />
  <row Id="7566" PostTypeId="2" ParentId="7521" CreationDate="2011-02-24T13:20:24.633" Score="3" ViewCount="" Body="&lt;p&gt;Actually the @mpiktas comment is the answer to your particular question. Sales models are usually multiplicative by the nature (some intuition could be found in Market response  models &lt;a href=&quot;http://books.google.ru/books?id=xZyJamKdpIsC&amp;amp;printsec=frontcover&amp;amp;dq=market+response+models&amp;amp;source=bl&amp;amp;ots=0-G9pzeYtd&amp;amp;sig=Qgt51utD-Hcg1E0kgvgMdC8_2ZI&amp;amp;hl=ru&amp;amp;ei=3llmTeKjJtHJswa0yMHaDA&amp;amp;sa=X&amp;amp;oi=book_result&amp;amp;ct=result&amp;amp;resnum=4&amp;amp;ved=0CDsQ6AEwAw#v=onepage&amp;amp;q&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;book&lt;/a&gt;). There is also a number of reasons for logs discussed for ARIMA models in my earlier &lt;a href=&quot;http://stats.stackexchange.com/questions/6330/when-to-log-transform-a-time-series-before-fitting-an-arima-model/6348#6348&quot;&gt;post&lt;/a&gt;. In your case it is the &lt;strong&gt;scale effect&lt;/strong&gt; that troubles you, therefore log transformation works well here. Another useful trick is to divide by some size variable (plot of the store, number of workers, etc.), so moving to fractions could help also.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In &lt;strong&gt;addition&lt;/strong&gt; to your question. What you have to pay attention to are other important explanatory variables: location variables or density of the population, size, variety of products (categories) and there average prices, number of workers, distances to the rival shops etc. that will matter (omitting them will cause you some estimates with poor properties: probably biased and inconsistent). Regulation can't be put as the solo explanatory variable in this context.&lt;/p&gt;&#xA;" OwnerUserId="2645" LastActivityDate="2011-02-24T13:20:24.633" CommentCount="3" />
  <row Id="7567" PostTypeId="2" ParentId="7450" CreationDate="2011-02-24T13:41:49.787" Score="1" ViewCount="" Body="&lt;p&gt;Following the detailed answer of the user cardinal i performed the chi-square test on my presumable truncated zipf distribution. The results of the chi-square test are reported in the following table:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/q3OPH.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where the &lt;strong&gt;StartInterval&lt;/strong&gt; and &lt;strong&gt;EndInterval&lt;/strong&gt; represent for example the range of calls and the &lt;strong&gt;Observed&lt;/strong&gt; is the number of callers generating from 0 to 19 calls, and so on..&#xA;The chi-square test is good until the last columns are reach, they increase the final calculation, otherwise until that point the &quot;partial&quot; chi-square value was acceptable!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With other tests the result is the same, the last column (or the last 2 columns) always increases the final value and i don't know why and i don't know if (and how) use another validation test.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: for completeness, to calculate the expected values (&lt;strong&gt;Expected&lt;/strong&gt;) i follow cardinal's suggestion in this way:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/e18If.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where &lt;strong&gt;X_i&lt;/strong&gt;'s are used to calculate:&#xA;&lt;code&gt;x &amp;lt;- (1:n)^-S&lt;/code&gt;, the &lt;strong&gt;P_i&lt;/strong&gt;'s to calculate &lt;code&gt;p &amp;lt;- x / sum(x)&lt;/code&gt; and finally the &lt;strong&gt;E_i&lt;/strong&gt; (Expected nr of users for each nr of calls) is obtained by &lt;code&gt;P_i * Total_Caller_Observed&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and with Degree of Freedom=13 the Chi-Square goodness rejects always the Hyphotesis that the sample set follow Zipf Distribution because the Test Statistics (64,14 in this case) is larger than that reported in the chi-square tables, &quot;demerit&quot; for the last column.&#xA;The graphical result is reported here:&#xA;&lt;img src=&quot;http://i.imgur.com/um3Eg.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;although the truncation point is set to 500 the maximum value obtaines is 294. I think that the final &quot;dispersion&quot; is the cause of the failure of the chi-square test.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE!!&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I try to perform the chi-square test on a presumable zipf data sample generated with the R code reported in the answer above.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; x &amp;lt;- (1:500)^(-2)&#xA;&amp;gt; p &amp;lt;- x / sum(x)&#xA;&amp;gt; y &amp;lt;- sample(length(p), size=300000, repl=TRUE, prob=p)&#xA;&amp;gt; tab &amp;lt;- table(y)&#xA;&amp;gt; length(tab)&#xA;[1] 438&#xA;&amp;gt; plot( 1:438, tab/sum(tab), log=&quot;xy&quot;, pch=20, &#xA;        main=&quot;'Truncated' Zipf simulation (truncated at i=500)&quot;,&#xA;        xlab=&quot;Response&quot;, ylab=&quot;Probability&quot; )&#xA;&amp;gt; lines(p, col=&quot;red&quot;, lwd=2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The associated plot is the following:&#xA;&lt;img src=&quot;http://i.imgur.com/kE77i.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The chi-square test results are reported in the following figure:&#xA;&lt;img src=&quot;http://i.imgur.com/dElAN.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and the chi-square test statistic (44,57) is too high for the validation with the chosen Degree of Freedom. Also in this case the final &quot;dispersion&quot; of data is the cause of the high chi-square value. But there is a procedure to validate this zipf distribution (regardless my &quot;wrong&quot; generator, i want to focus on the R data sample) ??? &lt;/p&gt;&#xA;" OwnerUserId="3342" LastEditorUserId="3342" LastEditorDisplayName="" LastEditDate="2011-03-14T13:42:16.817" LastActivityDate="2011-03-14T13:42:16.817" CommentCount="7" />
  <row Id="7568" PostTypeId="2" ParentId="7563" CreationDate="2011-02-24T14:15:01.640" Score="4" ViewCount="" Body="&lt;p&gt;One useful technique is monte carlo testing.  If there are two algorithms that do the same thing, implement both, feed them random data, and check that (to within a small tolerance for numerical fuzz) they produce the same answer.  I've done this several times before:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;I wrote an efficient but hard to implement $O(N\ log\ N)$ implementation of Kendall's Tau B.  To test it I wrote a dead-simple 50-line implementation that ran in $O(N^2)$.  &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I wrote some code to do ridge regression.  The best algorithm for doing this depends on whether you're in the $n &amp;gt; p$ or $p &amp;gt; n$ case, so I needed two algorithms anyhow.  &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In both of these cases I was implementing relatively well-known techniques in the D programming language (for which no implementation existed), so I also checked a few results against R.  Nonetheless, the monte carlo testing caught bugs I never would have caught otherwise.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another good test is &lt;a href=&quot;http://en.wikipedia.org/wiki/Assertion_%28computing%29&quot; rel=&quot;nofollow&quot;&gt;asserts&lt;/a&gt;.  You may not know exactly what the correct results of your computation should be, but that doesn't mean that you can't perform sanity checks at various stages of the computation.  In practice if you have a lot of these in your code and they all pass, then the code is usually right.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit:  A third method is to feed the algorithm data (synthetic or real) where you know at least approximately what the right answer is, even if you don't know exactly, and see by inspection if the answer is reasonable.  For example, you may not know exactly what the estimates of your parameters are, but you may know which ones are supposed to be &quot;big&quot; and which ones are supposed to be &quot;small&quot;.&lt;/p&gt;&#xA;" OwnerUserId="1347" LastEditorUserId="1347" LastEditorDisplayName="" LastEditDate="2011-02-24T14:46:27.493" LastActivityDate="2011-02-24T14:46:27.493" />
  <row Id="7569" PostTypeId="2" ParentId="4753" CreationDate="2011-02-24T14:24:17.157" Score="0" ViewCount="" Body="&lt;p&gt;I suggest that you look at the 2009 paper by Leng and Wang in JCGS:&#xA;&lt;a href=&quot;http://pubs.amstat.org/toc/jcgs/18/1&quot; rel=&quot;nofollow&quot;&gt;http://pubs.amstat.org/toc/jcgs/18/1&lt;/a&gt;&#xA;If this is what you want, the authors supply R code in the supplementary materials.&lt;/p&gt;&#xA;" OwnerUserId="2773" LastActivityDate="2011-02-24T14:24:17.157" />
  <row Id="7570" PostTypeId="2" ParentId="6886" CreationDate="2011-02-24T14:37:31.103" Score="0" ViewCount="" Body="&lt;p&gt;If we are talking about, for example, a neural network that is using back-propagation to learn, then a better way of thinking about this is that you are trying to set a,b,c,d,e using the neural network.  Your fitness function is fine, but that would be what your neural network is optimizing (a single output).  You don't select a,b,c,d,e, you use a method like back-propagation of errors to set these for you.  That's normally the point of using the neural network (not having to solve for the a,b,c,d,e coefficients yourself).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If, on the other hand, you're really convinced you don't want to use the iterative approach, you might look at the method of Dr. Hu of Southern Illinios University (Carbondale), who developed a non-iterative approach that lets you directly solve for the neural network coefficients.  Here's an article on it:&#xA;&lt;a href=&quot;http://books.google.com/books?id=bl9CyjErsusC&amp;amp;pg=PA416&amp;amp;lpg=PA416&amp;amp;dq=hu+carbondale+neural&amp;amp;source=bl&amp;amp;ots=--fEDuu5pk&amp;amp;sig=fr3lTT_BAon9f2ur3wzmxlB2jT8&amp;amp;hl=en&amp;amp;ei=-WtmTbGGGImjtgfP66TmAw&amp;amp;sa=X&amp;amp;oi=book_result&amp;amp;ct=result&amp;amp;resnum=1&amp;amp;ved=0CBQQ6AEwAA&quot; rel=&quot;nofollow&quot;&gt;article in 1996 Proceedings of World Congress on Neural Networks&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Myself, I would use your fitness function as the (single) output you're training the network to optimize, and use back-propagation (supported by many open-source neural network simulations) to find the input coefficients.  But Hu's approach (above) is workable if for some reason you don't want to use the iterative approach to find them. &lt;/p&gt;&#xA;" OwnerUserId="2917" LastActivityDate="2011-02-24T14:37:31.103" />
  <row Id="7571" PostTypeId="1" CreationDate="2011-02-24T14:48:35.657" Score="4" ViewCount="147" Body="&lt;p&gt;I'm trying to estimate the state of a Gaussian random walk with central tendency based on time series measurements with varying uncertainties.  My random variable has the following form:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ \frac{d x}{d t} \equiv F(t) - \alpha x $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where F is a Gaussian random variable.  I've noticed that this problem is analogous to the velocity of a bubble experiencing Brownian motion.  (See for example, F. Reif, &lt;em&gt;Fundamentals of Statistical and Thermal Physics&lt;/em&gt;, p. 565).  As a result of the $ -\alpha x $ term, the position has a central tendency (i.e. the variance does not become infinity as time approaches infinity).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, like any good physicist, I know that I cannot exactly measure the value $x$.  The best I can do is to measure it at time $ t_i $ within some uncertainty, $ \sigma_i $.  Using a Kalman filter, I can estimate the value of $x$ from several measurements.  Let's call that $ \hat x $.  The approach is as follows.  For each measurement, we compute:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;$ \delta t = t_i - t_{i-1} $&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;$ P(t) = P(t_{i-1}) * e^{-\alpha\,  \delta t} + \langle x^2 | \delta t \rangle $&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;$ K = {{P}\over{P + \sigma_i}} $&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Our incoming est estimate of $x_i$:&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;$ \hat x_{i-} = \hat x_{i-1} \, e^{-\alpha\,\delta t} $&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;$ \hat x_i = \hat x_{i-} + K [x_{obs} - \hat x_{i-}] $&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This works great for propagating our estimates forward in time.  My question is: If I have measurements at times that span the time at which I want the best estimate, how do I compute an $\hat x(t) $ where $ t_i &amp;lt; t &amp;lt; t_{i+1} $?&lt;/p&gt;&#xA;" OwnerUserId="3405" LastEditorUserId="3405" LastEditorDisplayName="" LastEditDate="2011-02-27T02:18:19.110" LastActivityDate="2011-02-27T02:18:19.110" Title="How to apply a Kalman filter to use both previous and future measurements of a random variable?" Tags="&lt;regression&gt;&lt;estimation&gt;&lt;kalman-filter&gt;" AnswerCount="1" CommentCount="4" />
  <row Id="7572" PostTypeId="2" ParentId="7499" CreationDate="2011-02-24T14:56:55.430" Score="1" ViewCount="" Body="&lt;p&gt;Here is a heuristic: if the practical importance of the result depends a lot on the value of point estimate, use CIs; if the practical importance of the result depends mainly on the existence/magnitude of effect, consider leaving them out.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is why:  error bars correct one sort of misunderstanding but invite others. The misunderstanding they correct is obvious: people assume too much precision in point estimate. But the ones they invite can be bad too; these include that all values within the interval are &quot;equally likely&quot;; that &quot;big confidence intervals&quot; are &quot;bad&quot;; &amp;amp; that point estimates w/ overlapping CIs are &quot;not significantly different&quot; from each other. If a particular result is has practical meaning b/c of the point estimate (e.g., likelihood a candidate will win an election or candidates' vote share in election), the former misunderstanding is more consequential -- that is, people will too likely make a mistake in relying on the result if they don't see the imprecision of it.  If a particular result, however, is practically significant b/c it discloses an effect that people wouldn't otherwise likely perceive -- consider a small-sample experiment that shows some framing-effect manipulation induces a large change in the valence of subjects' affective reaction to a political candidates' message, where effect is obviously there but scale for measuring it is not that important &amp;amp; CIs are big relative to dimensions of scale b/c of small sample  -- then the CI will often add little information, clutter things up, &amp;amp; invite the sort of &quot;a little knowledge is dangerous&quot; types of confusion I mentioned.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another approach is to try to find some alternative to convey precision of estimate w/o inviting typical misunderstanding of CIs. Some researchers try using graphics w/ multi-color shadings around point estimate, or elongated diamond shapes for pt estimates, to denote probability density of likely &quot;true&quot; values around the point estimate. My understanding is that people who have examined these alternatives conclude that they are confusing too...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;BTW, bar graphs are usually pretty rotten way to convey information. They are class Tufte chartjunk. There are lots of better alternatives. &lt;/p&gt;&#xA;" OwnerUserId="2347" LastActivityDate="2011-02-24T14:56:55.430" />
  <row Id="7573" PostTypeId="2" ParentId="7555" CreationDate="2011-02-24T15:18:02.870" Score="2" ViewCount="" Body="&lt;p&gt;Don't use the built-in routines of SPSS to conduct a meta-regression (wrong standard errors; does not give you correct model indices; no heterogeneity statistics). Have a look at David Wilson's SPSS &lt;a href=&quot;http://mason.gmu.edu/~dwilsonb/ma.html&quot; rel=&quot;nofollow&quot;&gt;&quot;macros for performing meta-analytic analyses&lt;/a&gt;&quot;. One of these macros is called &lt;code&gt;MetaReg&lt;/code&gt; which can perform fixed-effect or mixed-effects meta-regression. I would always use &lt;a href=&quot;http://www.stata.com/support/faqs/stat/meta.html&quot; rel=&quot;nofollow&quot;&gt;Stata&lt;/a&gt; or R. By the way, user &lt;a href=&quot;http://stats.stackexchange.com/users/1934/wolfgang&quot;&gt;Wolfgang&lt;/a&gt; is the author of an R package called &lt;a href=&quot;http://www.metafor-project.org/&quot; rel=&quot;nofollow&quot;&gt;metafor&lt;/a&gt;. This is an excellent piece of software to conduct meta-regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a general (non-technical) intro to meta-regression, I can recommend Thompson/Higgins (2002) &lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pubmed/12111920&quot; rel=&quot;nofollow&quot;&gt;&quot;How should meta-regression analyses be undertaken and interpreted?&lt;/a&gt;&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now to your question:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Q1: What is the minimum number of studies necessary for a meta-regression? Some people suggest at least 10 studies are required. Why not 20 or 5 studies?&lt;/em&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The answer can be found in &lt;a href=&quot;http://www.wiley.com/WileyCDA/WileyTitle/productCd-0470057246.html&quot; rel=&quot;nofollow&quot;&gt;Borenstein et al (2009: 188)&lt;/a&gt;: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&quot;As is true in primary studies, where&#xA;  we need an appropriately large ratio&#xA;  of subjects to covariates in order for&#xA;  the analysis be to meaningful, in&#xA;  meta-analysis we need an appropriately&#xA;  large ratio of studies to covariates.&#xA;  Therefore, the use of metaregression,&#xA;  especially with multiple covariates,&#xA;  is not a recommended option when the&#xA;  number of studies is small. In primary&#xA;  studies some have recommended a ratio&#xA;  of at least ten subjects for each&#xA;  covariate, which would correspond to&#xA;  ten studies for each covariate in&#xA;  meta-regression. In fact, though,&#xA;  there are no hard and fast rules in&#xA;  either case.&quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Q2: Is the total sample size an important consideration?&lt;/em&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is total sample size? The number of studies? Yes, it is important. Or the number of individuals? No, it is not (or less) important.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Q3: Why would 10 studies with 200 patients be enough, but 5 studies with 400 patients not be enough?&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is just a(n ordinary) regression. You wouldn't run a regression with 5 data points, would you? In your comment, you state that you have 20 studies which is enough to run a meta-regression. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Q4: Can I enter all three regressors at once and report the global model, or do I have to enter one regressor at a time and report 3 models each one separately?&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is just a regression. I would start with three simple bivariate models then build more complex models (be aware of multicollinearity, see below).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Q5: How does the correlation between the independent variables affects this choice?&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A high correlation between your predictor variables will have a (negative) impact on your results. You should avoid that. Please consult a textbook for the problem of &lt;a href=&quot;http://en.wikipedia.org/wiki/Multicollinearity&quot; rel=&quot;nofollow&quot;&gt;multicollinearity&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Q6: How does the number of the studies affect the number of independent variables that I should enter simultaneously?&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See the Borenstein et al citation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Q7: Does the independent variable have to be a scale variable? [...] The independent variable must be also scale, or could be ordinal or nominal?&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is a &quot;scale variable&quot;? Do you mean a continuous/metric variable? You predictor variables can have any &lt;a href=&quot;http://en.wikipedia.org/wiki/Level_of_measurement&quot; rel=&quot;nofollow&quot;&gt;level of measurement&lt;/a&gt;. However, if you have a categorical (nominal) predictor variable, you will have to deal with dummy variables (see &lt;a href=&quot;http://www.psychstat.missouristate.edu/multibook/mlt08m.html&quot; rel=&quot;nofollow&quot;&gt;Multiple Regression with Categorical Variables&lt;/a&gt;). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Q8: How can I weight my effect size for sample size?&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as I know, all meta-regression approaches expect the weights to be the inverse study variance, i.e. $\frac{1}{v_i}=\frac{1}{SE_i^2}$. Again, you will need standard errors :-)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Q9: What is the preferable level of significance? Is p&amp;lt;0.05 still acceptable for clinical research in such an analysis?&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I cannot answer your question. That really depends on your research question. In my (non-clinical) research I am happy with p &amp;lt; 0.10. &lt;/p&gt;&#xA;" OwnerUserId="307" LastActivityDate="2011-02-24T15:18:02.870" CommentCount="3" />
  <row Id="7574" PostTypeId="2" ParentId="7308" CreationDate="2011-02-24T15:33:43.887" Score="3" ViewCount="" Body="&lt;p&gt;The hessian is indefinite at a saddle point.  It’s possible that this may be the only stationary point in the interior of the parameter space.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Update: Let me elaborate.  First, let’s assume that the empirical Hessian exists everywhere.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If $\hat{\theta}_n$ is a local (or even global) minimum of $\sum_i q(w_i, \cdot)$ and in the interior of the parameter space (assumed to be an open set) then necessarily the Hessian $(1/N) \sum_i H(w_i, \hat{\theta}_n)$ is positive semidefinite.  If not, then $\hat{\theta}_n$ is not a local minimum.  This follows from second order optimality conditions — locally $\sum_i q(w_i, \cdot)$ must not decrease in any directions away from $\hat{\theta}_n$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One source of the confusion might the &quot;working&quot; definition of an M-estimator.  Although in principle an M-estimator should be defined as $\arg\min_\theta \sum_i q(w_i, \theta)$, it might also be defined as a solution to the equation $$0 = \sum_i \dot{q}(w_i, \theta)\,,$$ where $\dot{q}$ is the gradient of $q(w, \theta)$ with respect to $\theta$.  This is sometimes called the $\Psi$-type.  In the latter case a solution of that equation need not be a local minimum.  It can be a saddle point and in this case the Hessian would be indefinite.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Practically speaking, even a positive definite Hessian that is nearly singular or ill-conditioned would suggest that the estimator is poor and you have more to worry about than estimating its variance.&lt;/p&gt;&#xA;" OwnerUserId="1670" LastEditorUserId="1670" LastEditorDisplayName="" LastEditDate="2011-02-27T20:06:09.503" LastActivityDate="2011-02-27T20:06:09.503" CommentCount="3" />
  <row Id="7575" PostTypeId="2" ParentId="7549" CreationDate="2011-02-24T16:39:05.287" Score="1" ViewCount="" Body="&lt;p&gt;My first thought would be to regress education (using a proportional odds model or whatever is appropriate for your education variable) on person-level variables and a few simple transportation choice aggregates. The main variable that comes to mind is the proportion of train vs. bus rides (%train), but if you only have two event level variables -- distance and duration -- then another option would be %train-near, %train-far, %train-short, %train-long. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If something simple like the above won't work because you have too many event level variables or you're not willing to categorize them, then your first thought of using a logistic regression with random effects for person-level variables (I presume) is the right idea. However, I would modify your suggestion by using a structural equation model (SEM) to regress education on transportation choice, which is in turn regressed on event and person level variables (except for education) and the random effects. Education can additionally be regressed directly on the event and person level variables. All regressions are estimated simultaneously. This can be done in Mplus, but currently is not possible in R, as far as I know, because none of the SEM packages (lavaan, sem, e.g.) allow for mixed effects like those offered by the lme4 package. It can probably be done in SAS with a lot of coding. No idea about other software.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is your second thought of regressing education on combinations of your predictors feasible given the number of combinations and amount of data? How many event and person level variables do you have? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Latent class regression wouldn't make sense for your data because individual response patterns aren't comparable (e.g. person 1 might have chosen 00 for near-short, near-short and person 2 might have chosen 0000 for far-long, far-long, far-long, far long -- you could recode response vectors with a lot of missing values, but there are better approaches).&lt;/p&gt;&#xA;" OwnerUserId="3408" LastActivityDate="2011-02-24T16:39:05.287" CommentCount="4" />
  <row Id="7576" PostTypeId="2" ParentId="7535" CreationDate="2011-02-24T17:06:39.503" Score="2" ViewCount="" Body="&lt;p&gt;My understanding is that zero-inflated distributions should be used when there is a rationale for certain items to produce counts of zeroes versus any other count. In other words, a zero-inflated distribution should be used if the zeroes are produced by a separate process than the one producing the other counts. If you have no rationale for this, given the overdispersion in your sample, I suggest using a negative binomial distribution because it accurately represents the abundance of zeroes  and it represents unobserved heterogeneity by freely estimating this parameter. As mentioned above, Scott Long's book is a great reference.&lt;/p&gt;&#xA;" OwnerUserId="2322" LastActivityDate="2011-02-24T17:06:39.503" CommentCount="1" />
  <row Id="7577" PostTypeId="2" ParentId="6886" CreationDate="2011-02-24T17:37:38.757" Score="0" ViewCount="" Body="&lt;p&gt;Based on your question and the comments you have given to answers, I think there's a fundamental misunderstanding in your logic/problem formulation. In order to optimize something, no matter the method, you need to have something to optimize over. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to formulate a proper solution, you need to have a clear question. I suggest instead of fiddling with implementation of your NN, try to go back to your model (you should have one) and try to define the problem you want to optimize in more clear terms. Once you have the problem defined, then you can use the appropriate means to solve it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's of course possible that you have a function you want to optimize, and I misunderstood you question. I apologize if that's the case, but I believe in that scenario more detail, and a clear definition of the problem would certainly help your chances of getting help.&lt;/p&gt;&#xA;" OwnerUserId="3014" LastActivityDate="2011-02-24T17:37:38.757" />
  <row Id="7578" PostTypeId="2" ParentId="7563" CreationDate="2011-02-24T18:12:36.290" Score="3" ViewCount="" Body="&lt;p&gt;Not sure if this is really an answer to your question, but it is at least tangentially related.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I maintain the &lt;a href=&quot;http://www.maplesoft.com/support/help/Maple/view.aspx?path=Statistics&quot; rel=&quot;nofollow&quot;&gt;Statistics&lt;/a&gt; package in &lt;a href=&quot;http://www.maplesoft.com/products/Maple/index.aspx&quot; rel=&quot;nofollow&quot;&gt;Maple&lt;/a&gt;. An interesting example of difficult to test code is random sample generation according to different distributions; it is easy to test that no errors are generated, but it is trickier to determine whether the samples that are generated conform to the requested distribution &quot;well enough&quot;. Since Maple has both symbolic and numerical features, you can use some of the symbolic features to test the (purely numerical) sample generation:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;We have implemented a few types of statistical hypothesis testing, one of which is the &lt;a href=&quot;http://www.maplesoft.com/support/help/Maple/view.aspx?path=Statistics%2fChiSquareSuitableModelTest&quot; rel=&quot;nofollow&quot;&gt;chi square suitable model test&lt;/a&gt; - a chi square test of the numbers of samples in bins determined from the inverse CDF of the given probability distribution. So for example, to test Cauchy distribution sample generation, I run something like&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;with(Statistics):&#xA;infolevel[Statistics] := 1:&#xA;distribution := CauchyDistribution(2, 3):&#xA;sample := Sample(distribution, 10^6):&#xA;ChiSquareSuitableModelTest(sample, distribution, 'bins' = 100, 'level' = 0.001);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Because I can generate as large a sample as I like, I can make $\alpha$ pretty small.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;For distributions with finite moments, I compute on the one hand a number of sample moments, and  on the other hand, I symbolically compute the corresponding distribution moments and their standard error. So for e.g. the beta distribution:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;with(Statistics):&#xA;distribution := BetaDistribution(2, 3):&#xA;distributionMoments := Moment~(distribution, [seq(1 .. 10)]);&#xA;standardErrors := StandardError[10^6]~(Moment, distribution, [seq(1..10)]);&#xA;evalf(distributionMoments /~ standardErrors);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This shows a decreasing list of numbers, the last of which is 255.1085766. So for even the 10th moment, the value of the moment is more than 250 times the value of the standard error of the sample moment for a sample of size $10^6$. This means I can implement a test that runs more or less as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;with(Statistics):&#xA;sample := Sample(BetaDistribution(2, 3), 10^6):&#xA;sampleMoments := map2(Moment, sample, [seq(1 .. 10)]);&#xA;distributionMoments := [2/5, 1/5, 4/35, 1/14, 1/21, 1/30, 4/165, 1/55, 2/143, 1/91];&#xA;standardErrors := &#xA;  [1/5000, 1/70000*154^(1/2), 1/210000*894^(1/2), 1/770000*7755^(1/2), &#xA;   1/54600*26^(1/2), 1/210000*266^(1/2), 7/5610000*2771^(1/2), &#xA;   1/1567500*7809^(1/2), 3/5005000*6685^(1/2), 1/9209200*157366^(1/2)];&#xA;deviations := abs~(sampleMoments - distributionMoments) /~ standardErrors;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The numbers in &lt;code&gt;distributionMoments&lt;/code&gt; and &lt;code&gt;standardErrors&lt;/code&gt; come from the first run above. Now if the sample generation is correct, the numbers in deviations should be relatively small. I assume they are approximately normally distributed (which they aren't really, but it comes close enough - recall these are scaled versions of sample moments, not the samples themselves) and thus I can, for example, flag a case where a deviation is greater than 4 - corresponding to a sample moment that deviates more than four times the standard error from the distribution moment. This is very unlikely to occur at random if the sample generation is good. On the other hand, if the first 10 sample moments match the distribution moments to within less than half a percent, we have a fairly good approximation of the distribution.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The key to why both of these methods work is that the sample generation code and the symbolic code are almost completely disjoint. If there would be overlap between the two, then an error in that overlap could manifest itself both in the sample generation and in its verification, and thus not be caught.&lt;/p&gt;&#xA;" OwnerUserId="2898" LastActivityDate="2011-02-24T18:12:36.290" CommentCount="1" />
  <row Id="7579" PostTypeId="1" CreationDate="2011-02-24T18:16:42.123" Score="7" ViewCount="201" Body="&lt;p&gt;I am trying to model weekly disease counts in 25 different regions within 1 country over a ten year period as influenced by temperature.  The data is zero inflated and over dispersed.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am most familiar with Stata but I don't think that there is any option amongst the &lt;code&gt;gee&lt;/code&gt;, &lt;code&gt;xtmixed&lt;/code&gt;, &lt;code&gt;xtmepoisson&lt;/code&gt; etc. commands that allows me to account for the zero inflation and over dispersion issues as well as the autocorrelation.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I log transformed the incidence data and used a SARIMA model but the residuals are not quite normal.  I think that there are versions of the ARIMA model for integer data like disease counts but I can't find a program for it.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was also thinking that I could create a hierarchical model with random intercepts for each region and random effects of temperature in each region, while also accounting for the regular seasonal disease cycle.  I believe that I could model this in R using a package like &lt;a href=&quot;http://admb-project.org/examples/r-stuff/glmmadmb&quot; rel=&quot;nofollow&quot;&gt;glmm.admb&lt;/a&gt; but due to my limited statistical and R knowledge I am not entirely sure how to do use it.  I am mainly confused about accounting for the autocorrelation and seasonal cycle part of the data using a program like this.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any advice on how to best do this?&lt;/p&gt;&#xA;" OwnerUserId="3410" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-02-25T09:29:17.020" LastActivityDate="2011-02-25T16:31:35.000" Title="How to model zero inflated, over dispersed poisson time series?" Tags="&lt;time-series&gt;&lt;poisson&gt;&lt;autocorrelation&gt;&lt;overdispersion&gt;" AnswerCount="3" FavoriteCount="1" />
  <row Id="7581" PostTypeId="1" CreationDate="2011-02-24T18:57:13.633" Score="5" ViewCount="154" Body="&lt;p&gt;What is the relation between estimator and estimate?&lt;/p&gt;&#xA;" OwnerUserId="3412" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-02-25T02:56:18.663" LastActivityDate="2011-02-25T07:03:35.063" Title="What is the relation between estimator and estimate?" Tags="&lt;estimation&gt;" AnswerCount="3" CommentCount="2" />
  <row Id="7582" PostTypeId="2" ParentId="7579" CreationDate="2011-02-24T19:21:15.930" Score="4" ViewCount="" Body="&lt;p&gt;You may want to check out &lt;code&gt;hurdle()&lt;/code&gt; from the &lt;code&gt;pscl&lt;/code&gt; package in R. It specifies two-component models, one that handles the zero counts and one that handles the positive counts. Check out the &lt;code&gt;hurdle&lt;/code&gt; help page &lt;a href=&quot;http://rss.acs.unt.edu/Rdoc/library/pscl/html/hurdle.html&quot; rel=&quot;nofollow&quot;&gt;here.&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: I just found &lt;a href=&quot;http://r.789695.n4.nabble.com/Problems-using-gamlss-to-model-zero-inflated-and-overdispersed-count-data-quot-the-global-deviance-i-td2239925.html&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; post in R help that describes the &lt;code&gt;zeroinf()&lt;/code&gt; function in R (also from the pscl package), as well as &lt;code&gt;gamlss&lt;/code&gt; and &lt;code&gt;VGAM&lt;/code&gt; options. However, I don't believe that the &lt;code&gt;VGAM&lt;/code&gt; options will allow you to take into account non-independent correlation structures.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another option is the &lt;code&gt;zinb&lt;/code&gt; command in Stata. Fitting a model using the negative binomial family will account for the overdispersion.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not sure if they allow for seasonality adjustments, however.&lt;/p&gt;&#xA;" OwnerUserId="3309" LastEditorUserId="3309" LastEditorDisplayName="" LastEditDate="2011-02-24T19:27:49.170" LastActivityDate="2011-02-24T19:27:49.170" CommentCount="1" />
  <row Id="7583" PostTypeId="1" CreationDate="2011-02-24T19:44:47.270" Score="2" ViewCount="82" Body="&lt;p&gt;Greetings,&#xA;I need to determine the trend for barometric pressure based on measurements taken at 15-minute intervals over a 3 hour period. A rise or fall trend occurs if the the pressure changes more than 0.1 kPa, and rapid rise or fall if pressure changes more than 0.2 kPa over a three hour period. Measurements are in kPa, ranging roughly from 97.00 to 104.00.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Haven't used my college math in over thirty years - just can't seem to remember this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help in this area would be greatly appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="3407" LastActivityDate="2011-02-24T19:44:47.270" Title="How to calculate barometric pressure trend?" Tags="&lt;time-series&gt;" CommentCount="1" FavoriteCount="0" />
  <row Id="7585" PostTypeId="2" ParentId="7581" CreationDate="2011-02-24T21:10:40.313" Score="0" ViewCount="" Body="&lt;p&gt;Estimator is a random variable, and an estimate is the realization of an estimator (rv).&lt;/p&gt;&#xA;" OwnerUserId="3416" LastActivityDate="2011-02-24T21:10:40.313" CommentCount="1" />
  <row Id="7586" PostTypeId="2" ParentId="7581" CreationDate="2011-02-24T21:18:57.810" Score="3" ViewCount="" Body="&lt;p&gt;It might be helpful to illustrate whuber's answer in the context of a linear regression model. Let's say you have some bivariate data and you use Ordinary Least Squares to come up with the following model:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;code&gt;Y = 6X + 1&lt;/code&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;At this point, you can take any value of X, plug it into the model and predict the outcome, Y. In this sense, you might think of the individual components of the generic form of the model (&lt;em&gt;mX + B&lt;/em&gt;) as &lt;em&gt;estimators&lt;/em&gt;. The sample data (which you presumably plugged into the generic model to calculate the specific values for &lt;em&gt;m&lt;/em&gt; and &lt;em&gt;B&lt;/em&gt; above) provided a basis on which you could come up with &lt;em&gt;estimates&lt;/em&gt; for &lt;em&gt;m&lt;/em&gt; and &lt;em&gt;B&lt;/em&gt; respectively.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consistent with @whuber's points in our thread below, whatever values of &lt;em&gt;Y&lt;/em&gt; a particular set of estimators generate you for are, in the context of linear regression, thought of as predicted values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;(edited -- a few times -- to reflect the comments below)&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="3396" LastEditorUserId="3396" LastEditorDisplayName="" LastEditDate="2011-02-24T22:37:39.223" LastActivityDate="2011-02-24T22:37:39.223" CommentCount="5" />
  <row Id="7587" PostTypeId="2" ParentId="7554" CreationDate="2011-02-24T21:31:54.297" Score="1" ViewCount="" Body="&lt;p&gt;Assuming your distribution remotely resembles a normal curve, you could convert the standard errors into a more intuitive percentage value pretty easily. For example, if the distribution is pretty normal, approximately 95% of your population falls within +/-1.96*SE of the mean. Building from SheldonCooper's sample values, you could say, &quot;The average was 1000 and about 95% of the population was between 600 and 1400.&quot; Likewise, about 70% of the population falls within +/- 1*SE, etc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your sample distribution tends to deviate from normal by a lot, don't despair, but try to provide more details so we can help.&lt;/p&gt;&#xA;" OwnerUserId="3396" LastActivityDate="2011-02-24T21:31:54.297" CommentCount="2" />
  <row Id="7589" PostTypeId="2" ParentId="7579" CreationDate="2011-02-24T22:21:30.493" Score="3" ViewCount="" Body="&lt;p&gt;Another option for negative binomial regression in R is the excellent MASS package's &lt;code&gt;glm.nb()&lt;/code&gt; function. UCLA's statistical consulting group has a &lt;a href=&quot;http://www.ats.ucla.edu/stat/R/dae/nbreg.htm&quot; rel=&quot;nofollow&quot;&gt;pretty clear vignette&lt;/a&gt;, which unfortunately does not seem to provide any obvious insights into your autocorrelation issues, but maybe searching these various nb-regression options on R-seek or elsewhere would help?&lt;/p&gt;&#xA;" OwnerUserId="3396" LastActivityDate="2011-02-24T22:21:30.493" />
  <row Id="7590" PostTypeId="2" ParentId="1099" CreationDate="2011-02-24T22:29:52.487" Score="0" ViewCount="" Body="&lt;p&gt;If you were considering conducting Poisson or related regressions on this data (with your outcome variable as a rate), remember to include an offset term for the patient bed days as it technically becomes the &quot;exposure&quot; to your counts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, in that case, you may also want to consider using just the infection count (not the rate) as your dependent variable, and include the patient bed days as a covariate. I am working on a data set with a similar count vs. rate decision and it seems like converting your dependent variable to a rate leads to a decrease in variability, an increase in skewness and a proportionally larger standard deviation. This makes it more difficult to detect any significant effects.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also watch out if your data is zero-truncated or zero-inflated, and make the appropriate adjustments.&lt;/p&gt;&#xA;" OwnerUserId="3309" LastActivityDate="2011-02-24T22:29:52.487" />
  <row Id="7591" PostTypeId="1" AcceptedAnswerId="7592" CreationDate="2011-02-24T22:54:17.743" Score="5" ViewCount="167" Body="&lt;p&gt;Is there an R package, website, or command that will allow one to search for a specific statistical procedure they desire?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, if I wanted to find a package that had the Box-Cox Transformation, the website/package/command might return &quot;MASS&quot; and refer me to the &lt;code&gt;boxcox()&lt;/code&gt; function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is fairly straightforward with something like Box-Cox, but I was hoping it would allow me to find more difficult procedures or search by what the function does (&quot;Concatenating columns to a data frame&quot; might turn up &lt;code&gt;cbind()&lt;/code&gt;). Does something like this exist?&lt;/p&gt;&#xA;" OwnerUserId="1118" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-02-25T09:25:58.053" LastActivityDate="2011-02-25T09:25:58.053" Title="How to search for a statistical procedure in R?" Tags="&lt;r&gt;" AnswerCount="4" />
  <row Id="7592" PostTypeId="2" ParentId="7591" CreationDate="2011-02-24T23:02:05.017" Score="9" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://rseek.org/&quot; rel=&quot;nofollow&quot;&gt;rseek&lt;/a&gt; is pretty good. More abstract semantic queries along the lines of your second example are hard anywhere.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, see this &lt;a href=&quot;http://stackoverflow.com/questions/102056/how-to-search-for-r-materials&quot;&gt;SO thread&lt;/a&gt; from the &lt;a href=&quot;http://stackoverflow.com/tags/r/faq&quot;&gt;R-faq listing&lt;/a&gt; there.&lt;/p&gt;&#xA;" OwnerUserId="3396" LastEditorUserId="3396" LastEditorDisplayName="" LastEditDate="2011-02-24T23:49:21.960" LastActivityDate="2011-02-24T23:49:21.960" CommentCount="1" />
  <row Id="7593" PostTypeId="2" ParentId="7591" CreationDate="2011-02-24T23:02:57.447" Score="3" ViewCount="" Body="&lt;p&gt;I would try two things.  One is the ?? help search in R, so for Box-Cox I would do&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;??cox&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which should list packages or functions with that text&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The other is to try the &lt;a href=&quot;http://www.rseek.org/&quot; rel=&quot;nofollow&quot;&gt;http://www.rseek.org/&lt;/a&gt; site which is like google just for R.&lt;/p&gt;&#xA;" OwnerUserId="114" LastActivityDate="2011-02-24T23:02:57.447" />
  <row Id="7594" PostTypeId="1" AcceptedAnswerId="7600" CreationDate="2011-02-24T23:03:07.367" Score="1" ViewCount="85" Body="&lt;p&gt;Let's say the number of hotel rooms in a city is X. We know the arrival rate of visitors every day. We don't know the current occupancy. Is it possible to&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;estimate the occupancy rate over all hotel rooms (assume 1 visitor / room)&lt;/li&gt;&#xA;&lt;li&gt;estimate the distribution of stay duration (assume every visitor stays at least 1 day)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;On the flip side, if we knew just the occupancy rate, can we estimate the total number of hotel rooms from the arrival data?&lt;/p&gt;&#xA;" OwnerUserId="3418" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-02-25T06:23:48.497" LastActivityDate="2011-02-25T06:23:48.497" Title="Estimating occupancy rates from arrival rates" Tags="&lt;estimation&gt;&lt;stochastic-processes&gt;" AnswerCount="1" CommentCount="5" />
  <row Id="7595" PostTypeId="1" CreationDate="2011-02-24T21:33:14.667" Score="2" ViewCount="137" Body="&lt;p&gt;I've been using the &lt;code&gt;lm&lt;/code&gt; function in R to do demand modeling (tons of steel to be predicted by various economic indicators).  I used $R^2$  and $F$ to report on the strength of the model.  However, when I use the R function &lt;code&gt;lqs&lt;/code&gt; (&quot;resistant regression&quot;) and then type in &lt;code&gt;summary(model_name)&lt;/code&gt; I do not get any statistics that I can use to report on the strength of the regression model.  Any suggestions? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT:&#xA;Thanks for your quick response. I don't have a problem with lqs(). The problem is that when I type in summary(Model) I do not get any goodness of fit information (e.g., adjusted R squared) as I do when I enter summary(x) where X is a model created using the lm function. I'd like to have something to show the strength of the model. I&quot;m using MASS. See below. Regards, Bill Yarberry&lt;/p&gt;&#xA;&#xA;&lt;p&gt;library(MASS)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;M10 = lqs(agri ~ p12+p1+p11+p5+p8+p6+p25+p50+p35, data = agri_data2) summary(M10) Length Class Mode&#xA;crit 1 -none- numeric&#xA;sing 1 -none- character coefficients 10 -none- numeric&#xA;bestone 10 -none- numeric&#xA;fitted.values 103 -none- numeric&#xA;residuals 103 -none- numeric&#xA;scale 2 -none- numeric&#xA;terms 3 terms call&#xA;call 3 -none- call&#xA;xlevels 0 -none- list&#xA;model 10 data.frame list&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-02-28T20:22:32.760" LastActivityDate="2011-03-16T16:22:45.420" Title="How to get summary statistics from &quot;resistant regression&quot; - lqs - in R?" Tags="&lt;r&gt;&lt;regression&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="7596" PostTypeId="2" ParentId="7591" CreationDate="2011-02-24T23:36:35.363" Score="4" ViewCount="" Body="&lt;p&gt;Sometimes I just go to crantastic and search for keywords&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://crantastic.org/search?q=Box+Cox&quot; rel=&quot;nofollow&quot;&gt;Search for Box Cox on Crantastic&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="569" LastActivityDate="2011-02-24T23:36:35.363" />
  <row Id="7597" PostTypeId="2" ParentId="7591" CreationDate="2011-02-24T23:45:54.360" Score="10" ViewCount="" Body="&lt;p&gt;The &lt;a href=&quot;http://cran.r-project.org/web/packages/sos&quot; rel=&quot;nofollow&quot;&gt;sos package&lt;/a&gt; lets you search the help documentation for all cran packages from within R itself.&lt;/p&gt;&#xA;" OwnerUserId="364" LastActivityDate="2011-02-24T23:45:54.360" CommentCount="1" />
  <row Id="7599" PostTypeId="2" ParentId="7595" CreationDate="2011-02-25T00:02:22.710" Score="3" ViewCount="" Body="&lt;p&gt;Try typing:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model_name&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Based on a quick skim of the &lt;code&gt;lqs()&lt;/code&gt; documentation in the &lt;code&gt;MASS&lt;/code&gt; package this looks like it should work. If it doesn't work and you're not using &lt;code&gt;MASS&lt;/code&gt;, please specify which library you're running &lt;code&gt;lqs()&lt;/code&gt; from (and maybe even point to the documentation if you want to make everybody's life easier).&lt;/p&gt;&#xA;" OwnerUserId="3396" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-02-25T09:20:31.630" LastActivityDate="2011-02-25T09:20:31.630" CommentCount="2" />
  <row Id="7600" PostTypeId="2" ParentId="7594" CreationDate="2011-02-25T00:54:02.860" Score="2" ViewCount="" Body="&lt;p&gt;No.  You need more information. In particular, given the number of rooms and the arrival rate, then the average occupancy rate and the average length of stay are proportional to each other.&lt;/p&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-02-25T00:54:02.860" CommentCount="2" />
  <row Id="7601" PostTypeId="1" CreationDate="2011-02-25T01:05:59.623" Score="4" ViewCount="97" Body="&lt;p&gt;I have a multinomial logistic regression with dependent variable valued in {-1,0,1} (reference category is 0) and a number of continuous and discrete predictors. After running the regression a continuous predictor of interest ('size') has a Type 3 analysis of effects p-value of 0.0683, and the two coefficients (corresponding to outcomes of -1 and 1) have p-values of 0.8786 and 0.0220 respectively.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I read somewhere that one should only look at the significance of the coefficients if the predictor itself is significant at the chosen level. Is this right? My naive sense is that the predictor is borderline (taking alpha=0.05 for argument's sake), and that 'size' has a significant relationship to outcome=1 but not to outcome = -1. I would say that the significance of the relationship to outcome=1 is not terribly strong, but that is ok for the application in mind (or at least, with the indirect data I am forced to use)&lt;/p&gt;&#xA;" OwnerUserId="1144" LastActivityDate="2011-03-27T03:38:16.917" Title="Interpreting significance of predictor vs significance of predictor coeffs in multinomial logistic regression" Tags="&lt;logistic-regression&gt;&lt;statistical-significance&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="7602" PostTypeId="2" ParentId="6524" CreationDate="2011-02-25T02:10:06.680" Score="1" ViewCount="" Body="&lt;p&gt;OK, your question isn't perfectly clear but maybe I can help a little.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A statistic $T(X)$ is sufficient for a parameter $\theta$ if&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(X|T(X), \theta) = P(X|T(X)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In terms of likelihood functions you can verify that this implies&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$f(x;\theta) = h(x)g(T(x); \theta)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;for some $h$ and $g$, which is known by a few different monikers (the factorization theorem/lemme/criteria and sometimes with a name or two attached). This is where @probabilityislogic's comment comes from, although like I said it's just a property of the likelihood function. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are often a lot of different sufficient statistics (in particular, take $h=1$ and $g=f$, where $T(X)=X$ is just the entire dataset). Since the goal is to find a particular way to reduce the data without losing information, this leads into questions of minimal/complete sufficient statistics, etc. It's not clear what you need for your question, so I'll leave off there.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In terms of the MLE, your notation is a little confusing to me so I'll make a couple general comments. What problems can happen finding the MLE? It might not have a closed form, which is less a problem than a complication. It can fail to be unique, or occur at the edge of the parameter space, be infinite, etc. You need to at least define the parameter space, which you haven't done in your problem statement so far as I can tell.&lt;/p&gt;&#xA;" OwnerUserId="26" LastActivityDate="2011-02-25T02:10:06.680" />
  <row Id="7603" PostTypeId="2" ParentId="7601" CreationDate="2011-02-25T02:48:40.657" Score="2" ViewCount="" Body="&lt;p&gt;The &lt;em&gt;p&lt;/em&gt;-value itself cannot tell you how strong the relationship is, because the &lt;em&gt;p&lt;/em&gt;-value is so influenced by sample size, among other things.  But assuming your N is something on the order of 100-150, I'd say there's a reasonably strong effect involving Size whereby as Size increases, the log of the odds of Y being 1 is notably different from the log of the odds of Y being 0.  As you indicate, the same cannot be said of the comparison of Y values of -1 and 0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You are right in viewing all of this as somewhat invalidated by the overall nonsignificance of Size (depending on your &lt;em&gt;alpha&lt;/em&gt;, or criterion for significance).  You wouldn't get too many arguments if you simply declared Size a nonfactor due to its high &lt;em&gt;p&lt;/em&gt;-value.  But then again, if your N is sufficiently small--perhaps below 80 or 100--then your design affords low power for detecting effects, and you might make a case for taking seriously the specific effect that managed to show up anyway.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A way around the problem of relying on &lt;em&gt;p&lt;/em&gt;-values involves two steps.  First, decide what range of odds ratios would constitute an effect worth bothering with, or worth calling substantial.  (The trick there is in being facile enough with odds to recognize what they mean for the more intuitive metric of probability.)  Then construct a confidence interval for the odds ratio associated with each coefficient and consider it in light of your hypothetical  range.  Regardless of statistical significance, does the effect have practical significance?    &lt;/p&gt;&#xA;" OwnerUserId="2669" LastEditorUserId="2669" LastEditorDisplayName="" LastEditDate="2011-02-25T02:58:50.813" LastActivityDate="2011-02-25T02:58:50.813" CommentCount="2" />
  <row Id="7604" PostTypeId="1" CreationDate="2011-02-25T03:04:54.500" Score="1" ViewCount="73" Body="&lt;p&gt;I have a single model (e.g generalized Pareto distribution) to test with a different data set (I have a set of different increasing threshold and fit the same model with a data above these threshold). I want to know which set of data will give me the best model fit. Can i use likelihood ratio test in this case or any other suggestion to get this? Many thanks in advance. &lt;/p&gt;&#xA;" OwnerUserId="3423" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-02-25T09:22:56.200" LastActivityDate="2011-02-25T19:51:48.713" Title="Single model for a different data set" Tags="&lt;model-selection&gt;&lt;likelihood-ratio&gt;" AnswerCount="1" />
  <row Id="7605" PostTypeId="1" AcceptedAnswerId="7614" CreationDate="2011-02-25T03:42:55.343" Score="2" ViewCount="89" Body="&lt;p&gt;I'm running an EFA using orthogonal/varimax rotation, and assigning variables to a factors based on maximum load (so only each variable only gets one factor). I then want to validate the model using SEM... since the rotation I used to determine the variable&amp;lt;-&gt;factor loads was orthogonal, is it &quot;wrong&quot; to let the factors in my model have a covariance with one another? (eg, using RAM: Factor1&amp;lt;-&gt;Factor2,theta,NA)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I ask, as I get a much better model fit if I allow for this to occur.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More explicitly, what does it actual mean for underlying factors to have a correlation between them?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="3424" LastActivityDate="2011-02-25T12:53:09.590" Title="What does it mean if there is a correlation between underlying factors in factor analysis?" Tags="&lt;psychometrics&gt;" AnswerCount="2" CommentCount="3" />
  <row Id="7606" PostTypeId="2" ParentId="7604" CreationDate="2011-02-25T03:43:01.500" Score="2" ViewCount="" Body="&lt;p&gt;You can't use the standard likelihood ratio test. It only works for comparing the likelihoods of different models on the same dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the more general question of how to pick the set to which the model fits best, you need to define what &quot;best&quot; means. If your data is iid, the likelihood is obtained by multiplying a probability value (a number less than one) for each point. Just comparing likelihood will favor small sets (because fewer small numbers are multiplied). This is probably not what you want. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;GP has two parameters, $x_m$ and $\alpha$. I assume you fit both of them on each dataset (because if you hold $x_m$ fixed and change the threshold, then for larger thresholds there won't be any data near $x_m$). In this case, the best fit will be when your threshold is such that there is only one point left. But this fit only looks best because the problem of fitting one data point is very easy. Fitting many points is more difficult, so naturally the fit looks worse. So you need to account for the difficulty of the problem somehow. The problem seems complementary to the usual problem of accounting for the &quot;model complexity&quot;. I'm not sure whether there is a standard way of doing this. This would certainly be interesting to learn about.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Meanwhile, a way to sidestep this would be the following. Do you have a model for what the data &lt;em&gt;below&lt;/em&gt; the threshold should look like? Maybe they come from a different process for which you know the distribution. If yes, what you can do is fit a mixture model to the data both above and below threshold. This mixture would always fit the same dataset (namely, all of your points). So for the mixture you could use the likelihood test or some other model selection method.&lt;/p&gt;&#xA;" OwnerUserId="3369" LastEditorUserId="3369" LastEditorDisplayName="" LastEditDate="2011-02-25T19:51:48.713" LastActivityDate="2011-02-25T19:51:48.713" CommentCount="1" />
  <row Id="7607" PostTypeId="1" AcceptedAnswerId="7954" CreationDate="2011-02-25T06:40:57.203" Score="6" ViewCount="121" Body="&lt;p&gt;&lt;strong&gt;What are the methods to minimize the effect of boundaries in a wavelet decomposition?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I use R and the package &lt;a href=&quot;http://cran.r-project.org/web/packages/waveslim/index.html&quot; rel=&quot;nofollow&quot;&gt;waveslim&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have found for instance the function &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;?brick.wall&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;but&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;I am not too use how to use it.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I am not sure the best solution is to remove some coefficient. I have read somewhere that it exists some wavelets that are not the same everywhere and their shape change at the boudaries.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Any ideas?&lt;/p&gt;&#xA;" OwnerUserId="1709" LastEditorUserId="223" LastEditorDisplayName="" LastEditDate="2011-03-12T10:03:39.110" LastActivityDate="2011-03-12T10:03:39.110" Title="Boundary effect in a wavelet multi resolution analysis" Tags="&lt;r&gt;&lt;signal-processing&gt;&lt;wavelet&gt;&lt;boundary&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="3" />
  <row Id="7608" PostTypeId="2" ParentId="7581" CreationDate="2011-02-25T07:03:35.063" Score="4" ViewCount="" Body="&lt;p&gt;E. L. Lehman, in his classic &lt;em&gt;Theory of Point Estimation&lt;/em&gt;, answers this question on pp 1-2. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The observations are now postulated to be the values taken on by random variables which are assumed to follow a joint probability distribution, $P$, belonging to some known class...&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;...let us now specialize to point estimation...suppose that $g$ is a real-valued function defined [on the stipulated class of distributions] and that we would like to know the value of $g$ [at whatever is the actual distribution in effect, $\theta$].  Unfortunately, $\theta$, and hence $g(\theta)$, is unknown.  However, the data can be used to obtain an estimate of $g(\theta)$, a value that one hopes will be close to $g(\theta)$.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In words: an &lt;em&gt;estimator&lt;/em&gt; is a definite mathematical procedure that comes up with a number (the &lt;em&gt;estimate&lt;/em&gt;) for any possible set of data that a particular problem could produce.  That number is intended to represent some definite numerical property ($g(\theta)$) of the data-generation process; we might call this the &quot;estimand.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The estimator itself is &lt;em&gt;not&lt;/em&gt; a random variable: it's just a mathematical function.  However, the estimate it produces is based on data which themselves are modeled as random variables.  This makes the &lt;em&gt;estimate&lt;/em&gt; (thought of as depending on the data) into a random variable and &lt;em&gt;a particular estimate&lt;/em&gt; for &lt;em&gt;a particular set of data&lt;/em&gt; becomes a realization of that random variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In one (conventional) ordinary least squares formulation, the data consist of ordered pairs $(x_i, y_i)$.  The $x_i$ have been determined by the experimenter (they can be amounts of a drug administered, for example).  Each $y_i$ (a response to the drug, for instance) is assumed to come from a probability distribution that is Normal but with unknown mean $\mu_i$ and &lt;em&gt;common&lt;/em&gt; variance $\sigma^2$.  Furthermore, it is assumed that the means are related to the $x_i$ via a formula $\mu_i = \beta_0 + \beta_1 x_i$.  These three parameters--$\sigma$, $\beta_0$, and $\beta_1$--determine the underlying distribution of $y_i$ for any value of $x_i$.  Therefore &lt;em&gt;any&lt;/em&gt; property of that distribution can be thought of as a function of $(\sigma, \beta_0, \beta_1)$.  Examples of such properties are the intercept $\beta_0$, the slope $\beta_1$, the value of $\cos(\sigma + \beta_0^2 - \beta_1)$, or even the mean at the value $x=2$, which (according to this formulation) must be $\beta_0 + 2 \beta_1$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this OLS context, a &lt;em&gt;non-example&lt;/em&gt; of an estimator would be a procedure to guess at the value of $y$ if $x$ were set equal to 2.  This is &lt;em&gt;not&lt;/em&gt; an estimator because this value of $y$ is &lt;em&gt;random&lt;/em&gt; (in a way completely separate from the randomness of the data): it is not a (definite numerical) property of the distribution, even though it is related to that distribution.  (As we just saw, though, the &lt;em&gt;expectation&lt;/em&gt; of $y$ for $x=2$, equal to $\beta_0 + 2 \beta_1$, can be estimated.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In Lehman's formulation, almost any formula can be an estimator of almost any property.  &lt;em&gt;There is no inherent mathematical link between an estimator and an estimand.&lt;/em&gt;  However, we can assess--in advance--the chance that an estimator will be reasonably close to the quantity it is intended to estimate.  Ways to do this, and how to exploit them, are the subject of estimation theory.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-02-25T07:03:35.063" CommentCount="2" />
  <row Id="7609" PostTypeId="2" ParentId="6026" CreationDate="2011-02-25T07:52:29.567" Score="2" ViewCount="" Body="&lt;p&gt;Hi.&#xA;I am not a statistician, but an MD, trying to sort things out in the world of statistics. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The way you have to interpret this output is by looking at the $\exp(B)$ values. A value of &amp;lt; 1 says that an increase in one unit for that particular variable, will decrease the probability of experiencing an end point throughout the observation period. By inverting (that is $1/\exp(B)$), you will find the &quot;protective effect&quot;, for example if $\exp(B) = 0.407$ (as is the case for your &quot;Gender&quot; value), the interpretation will be that having the value of gender = 1 means that you decrease the probability of experiencing an en point with $1/0.407 = 2.46$, compared to when the Gender value = 0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For $\exp(B) &amp;gt; 1$, the interpretation is even easier, as a value of, say $\exp(B) = 1.259$ (as is the case for your &quot;stenosis&quot; variable), means that scoring &quot;stenosis&quot; = 1 will result in an increased probability (25.9%) of experiencing an end point compared to when &quot;stenosis&quot; = 0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The confidence interval (CI) tells us within which range (of 95% probability) we can expect this value to differ, if we were to repeat this survey for an infinite number of times. If the 95% CI overlaps the value of 1, then the result is not statistically significant (since $\exp(B) = 1$ means that there is no difference between the probability of experiencing an end point if the variable value is either &quot;0&quot; or &quot;1&quot;), and the P value will exceed 0.05. If the 95% CI keeps out of the value 1 (on either side), the $\exp(B)$ is statistically significant. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;From your analysis, it seems as no one of your variables are significant predictors (at a sign level of 5%) of your endpoint, although being a &quot;high risk&quot; patient is of borderline significance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Reading the book &quot;&lt;strong&gt;SPSS survival manual&lt;/strong&gt;&quot;, by Julie Pallant will probably enlighten you further on this (and more) topic(s).&lt;/p&gt;&#xA;" OwnerUserId="3428" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-02-25T09:50:26.273" LastActivityDate="2011-02-25T09:50:26.273" CommentCount="1" />
  <row Id="7610" PostTypeId="1" CreationDate="2011-02-25T09:45:02.317" Score="8" ViewCount="381" Body="&lt;p&gt;Besides obvious classifier characteristics like&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;computational cost,&lt;/li&gt;&#xA;&lt;li&gt;expected data types of features/labels and&lt;/li&gt;&#xA;&lt;li&gt;suitability for certain sizes and dimensions of data sets,&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;what are the top five (or 10, 20?) classifiers to try first on a new data set &lt;em&gt;one does not know much about yet&lt;/em&gt; (e.g. semantics and correlation of individual features)? Usually I try Naive Bayes, Nearest Neighbor, Decision Tree and SVM - though I have no good reason for this selection other than I know them and mostly understand how they work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I guess one should choose classifiers which cover the most important &lt;em&gt;general&lt;/em&gt; classification approaches. Which selection would you recommend, according to that criterion or for any other reason?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance for some suggestions!&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE:&lt;/strong&gt; An alternative formulation for this question could be: &lt;em&gt;&quot;Which general approaches to classification exist and which specific methods cover the most important/popular/promising ones?&quot;&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="2230" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-02-25T13:25:18.103" LastActivityDate="2011-02-25T16:30:28.397" CommunityOwnedDate="2011-02-25T13:25:01.677" Title="Top five classifiers to try first" Tags="&lt;machine-learning&gt;&lt;classification&gt;&lt;methodology&gt;" AnswerCount="3" CommentCount="2" FavoriteCount="5" />
  <row Id="7611" PostTypeId="1" AcceptedAnswerId="7612" CreationDate="2011-02-25T09:49:28.517" Score="3" ViewCount="106" Body="&lt;p&gt;Hi,&#xA;I am constructing Cox models that predict survival in a clinical trials cohort. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;After speaking to our statistician (who is away at the moment, hence this post), I was advised to take a forward likelihood ratio-test approach to building Cox survival models, starting with a base model and adding the term that improved the model, by computing a p value from subtracting the likelihood ratio statistic from the extended model from the likelihood ratio from the base model, as outlined in the R code below.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I realise that Stata is probably a better fit for this sort of analysis, but I i) I don't have easy access to Stata and ii) am familiar with R (also have access to SPSS), so with that caveat, here is the general format of the code I am using:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Conceptually, this makes sense to me if I add binary covariates to the model, but I was wondering whether this approach is appropriate for adding a continuous variable, as outlined below? I'm not sure that a degree of freedom equal to one is correct for this comparison?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;y&amp;lt;-0:1&#xA;data&amp;lt;-data.frame(cbind(sample(y,100,replace=TRUE),runif(100,min=0,max=10),sample(y,100,replace=TRUE),runif(100,min=0,max=1)))&#xA;colnames(data)&amp;lt;-c(&quot;EFS_Status&quot;,&quot;EFS_Time&quot;,&quot;var1&quot;,&quot;Contvar2&quot;)   &#xA;library(survival)  &#xA;base &amp;lt;- coxph(Surv(EFS_Time,EFS_Status) ~ var1, data=data) # Create base model  &#xA;lr1 &amp;lt;- -2*base$loglik[2] # Likelihood ratio of base model&#xA;&#xA;extend &amp;lt;- coxph(Surv(EFS_Time,EFS_Status) ~ var1 + Contvar2, data=data) # Extended model  &#xA;lr2&amp;lt;- -2*extend$loglik[2] # Likelihood ratio of extended model &#xA;&#xA;pchisq(q=lr1-lr2,df=1,lower.tail=FALSE) # 1 df correct for continuous variables?&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Any guidance is most appreciated. Obviously, I could binarize the continuous variable, but I suppose you're losing information by taking that approach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for reading,&#xA;Ed&lt;/p&gt;&#xA;" OwnerUserId="3429" LastActivityDate="2011-02-25T10:04:12.440" Title="Building Cox Model using forward likelihood-ratio testing - Appropriateness of adding continuous variables to model?" Tags="&lt;survival&gt;&lt;continuous-data&gt;&lt;clinical-trials&gt;" AnswerCount="1" />
  <row Id="7612" PostTypeId="2" ParentId="7611" CreationDate="2011-02-25T10:04:12.440" Score="3" ViewCount="" Body="&lt;p&gt;Yes, a continuous variable added as a linear effect to the model formula as you have here adds one degree of freedom to the model, as there is one more model parameter to estimate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By the way, instead of using &lt;code&gt;pchisq&lt;/code&gt; to compute the p-value it's easier and less error-prone to use &lt;code&gt;anova()&lt;/code&gt;. This will automatically calculate the correct degrees of freedom for the test. See &lt;code&gt;?anova.coxph&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-02-25T10:04:12.440" CommentCount="1" />
  <row Id="7613" PostTypeId="1" CreationDate="2011-02-25T10:25:39.287" Score="3" ViewCount="183" Body="&lt;p&gt;Does the magnitudes of principal eigenvectors obtained by PCA have anything to do with correlations of original variables, and can we use PCA for clustering?&#xA;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="3430" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-02-26T06:11:02.380" LastActivityDate="2011-02-26T06:11:02.380" Title="Do correlations relate to PCA eigenvectors and can PCA be used for clustering?" Tags="&lt;pca&gt;" AnswerCount="1" CommentCount="3" FavoriteCount="2" />
  <row Id="7614" PostTypeId="2" ParentId="7605" CreationDate="2011-02-25T11:42:09.717" Score="3" ViewCount="" Body="&lt;p&gt;OK, in my experience cases like this mean that you should have allowed the factors to correlate in the first place. You should probably rerun a factor analysis using either oblimin or promax rotation and test the fit of your uncorrelated model against your correlated model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please do note that SEM loses its utility as a method for testing theories once you start changing the model based on fit indices. &lt;/p&gt;&#xA;" OwnerUserId="656" LastActivityDate="2011-02-25T11:42:09.717" CommentCount="2" />
  <row Id="7615" PostTypeId="2" ParentId="7610" CreationDate="2011-02-25T12:01:31.843" Score="3" ViewCount="" Body="&lt;p&gt;Gaussian process classifier (not using the Laplace approximation), preferably with marginalisation rather than optimisation of the hyper-parameters.  Why?&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;because they give a probabilistic classification&lt;/li&gt;&#xA;&lt;li&gt;you can use a kernel function that allows you to operate directly on non-vectorial data and/or incorporate expert knowledge&lt;/li&gt;&#xA;&lt;li&gt;they deal with the uncertainty in fitting the model properly, and you can propagate that uncertainty through to the decision making process&lt;/li&gt;&#xA;&lt;li&gt;generally very good predictive performance.  &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Downsides &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;slow &lt;/li&gt;&#xA;&lt;li&gt;requires a lot of memory&lt;/li&gt;&#xA;&lt;li&gt;impractical for large scale problems.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;First choice though would be regularised logistic regression or ridge regression [without feature selection] - for most problems, very simple algorithms work rather well and are more difficult to get wrong (in practice the differences in performance between algorithms is smaller than the differences in performance between the operator driving them).&lt;/p&gt;&#xA;" OwnerUserId="887" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-02-25T16:30:28.397" LastActivityDate="2011-02-25T16:30:28.397" CommunityOwnedDate="2011-02-25T13:25:01.677" />
  <row Id="7616" PostTypeId="2" ParentId="7610" CreationDate="2011-02-25T12:16:40.370" Score="-1" ViewCount="" Body="&lt;p&gt;By myself when you are approaching to a new data set you should start to watch to the whole problem. First of all get a distribution for categorical features and mean and standard deviations for each continuous feature. Then:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Delete features with more than X% missing values;&lt;/li&gt;&#xA;&lt;li&gt;Delete categorical features when a particular value gets more then 90-95% of relative frequency;&lt;/li&gt;&#xA;&lt;li&gt;Delete continuous features with CV=std/mean&amp;lt;0.1;&lt;/li&gt;&#xA;&lt;li&gt;Get a parameter ranking, eg ANOVA for continuous and Chi-square for categorical;&lt;/li&gt;&#xA;&lt;li&gt;Get a significant subset of features;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Then I usually split the classification techniques in 2 sets: white box and black box technique. If you need to know 'how the classifier works' you should choose in the first set, eg Decision-Trees or Rules-based classifiers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you need to classify new records without building a model should should take a look to eager learner, eg KNN. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;After that I think is better to have a threshold between accuracy and speed: Neural Network are a bit slower than SVM.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is my top five classification technique:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Decision Tree;&lt;/li&gt;&#xA;&lt;li&gt;Rule-based classifiers;&lt;/li&gt;&#xA;&lt;li&gt;SMO (SVM);&lt;/li&gt;&#xA;&lt;li&gt;Naive Bayes;&lt;/li&gt;&#xA;&lt;li&gt;Neural Networks.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="2719" LastActivityDate="2011-02-25T12:16:40.370" CommunityOwnedDate="2011-02-25T13:25:01.677" CommentCount="2" />
  <row Id="7617" PostTypeId="2" ParentId="6963" CreationDate="2011-02-25T12:25:12.067" Score="1" ViewCount="" Body="&lt;p&gt;Just an initial remark, if you want computational speed you usually have to sacrifice accuracy.  &quot;More accuracy&quot; = &quot;More time&quot; in general.  Anyways here is a second order approximation, should improve on the &quot;crude&quot; approx you suggested in your comment above:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$E\Bigg(\frac{X_{j}}{\sum_{i}X_{i}}\Bigg)\approx&#xD;&#xA;\frac{E[X_{j}]}{E[\sum_{i}X_{i}]} &#xD;&#xA;-\frac{cov[\sum_{i}X_{i},X_{j}]}{E[\sum_{i}X_{i}]^2}&#xD;&#xA;+\frac{E[X_{j}]}{E[\sum_{i}X_{i}]^3} Var[\sum_{i}X_{i}]&#xD;&#xA;$$&#xA;$$= \frac{\alpha_{j}}{\sum_{i} \frac{\beta_{j}}{\beta_{i}}\alpha_{i}}\times\Bigg[1 - \frac{1}{\Bigg(\sum_{i} \frac{\beta_{j}}{\beta_{i}}\alpha_{i}\Bigg)}&#xD;&#xA;+ \frac{1}{\Bigg(\sum_{i} \frac{\alpha_{i}}{\beta_{i}}\Bigg)^2}\Bigg(\sum_{i} \frac{\alpha_{i}}{\beta_{i}^2}\Bigg)\Bigg]&#xD;&#xA;&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt; An explanation for the above expansion was requested.  The short answer is &lt;a href=&quot;http://en.wikipedia.org/wiki/Taylor_expansions_for_the_moments_of_functions_of_random_variables&quot; rel=&quot;nofollow&quot;&gt;wikipedia&lt;/a&gt;.  The long answer is given below.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;write $f(x,y)=\frac{x}{y}$.  Now we need all the &quot;second order&quot; derivatives of $f$.  The first order derivatives will &quot;cancel&quot; because they will all involve multiples $X-E(X)$ and $Y-E(Y)$ which are both zero when taking expectations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{\partial^2 f}{\partial x^2}=0$$&#xA;$$\frac{\partial^2 f}{\partial x \partial y}=-\frac{1}{y^2}$$&#xA;$$\frac{\partial^2 f}{\partial y^2}=2\frac{x}{y^3}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And so the taylor series up to second order is given by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{x}{y} \approx \frac{\mu_x}{\mu_y}+\frac{1}{2}\Bigg(-\frac{1}{\mu_y^2}2(x-\mu_x)(y-\mu_y) + 2\frac{\mu_x}{\mu_y^3}(y-\mu_y)^2 \Bigg)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Taking expectations yields:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$E\Big[\frac{x}{y}\Big] \approx \frac{\mu_x}{\mu_y}-\frac{1}{\mu_y^2}E\Big[(x-\mu_x)(y-\mu_y)\Big] + \frac{\mu_x}{\mu_y^3}E\Big[(y-\mu_y)^2\Big]$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which is the answer I gave.  (although I initially forgot the minus sign in the second term)&lt;/p&gt;&#xA;" OwnerUserId="2392" LastEditorUserId="2392" LastEditorDisplayName="" LastEditDate="2011-03-08T07:45:43.053" LastActivityDate="2011-03-08T07:45:43.053" CommentCount="1" />
  <row Id="7618" PostTypeId="1" AcceptedAnswerId="7634" CreationDate="2011-02-25T12:32:09.693" Score="3" ViewCount="78" Body="&lt;p&gt;Have you ever heard about Child-Pugh cirrhosis score? There are five features, each feature is discretized in three intervals. For each interval you get a point, eg 1 for the first, 2 for the second etc. After computing all points you sum them up and then you get a cirrhosis stages ranked from A to C. Each stage is correlated to a particular probability to survive. &lt;a href=&quot;http://en.wikipedia.org/wiki/Child-Pugh_score&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Child-Pugh_score&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a DB with a many features and a label, the label is binary: ill and good. I have chosen probability estimation trees to get a white box model that estimates probabilities to be 'ill'. The model can show almost clearly the classification process, but I think that a Child-Pugh like score could be clearer than this for a medical doctor. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there something similar in literature? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks, bye.&lt;/p&gt;&#xA;" OwnerUserId="2719" LastActivityDate="2011-02-25T20:57:11.160" Title="White box machine learning probability estimator" Tags="&lt;probability&gt;&lt;machine-learning&gt;" AnswerCount="1" />
  <row Id="7619" PostTypeId="2" ParentId="7605" CreationDate="2011-02-25T12:53:09.590" Score="2" ViewCount="" Body="&lt;ol&gt;&#xA;&lt;li&gt;The first thing to do is see if the structure changes when you use an oblique rotation that allows the factors to correlated.&lt;/li&gt;&#xA;&lt;li&gt;If you truly want to validate your structure then use a confirmatory factor analysis (CFA, which is the measurement part of SEM) as you are already doing but with a &lt;strong&gt;different&lt;/strong&gt; sample.&lt;/li&gt;&#xA;&lt;li&gt;If you just have one sample and you are interested in whether the factors correlate, look into building a second order CFA that posits a higher order factor explaining the common variance among 2 or more of the lower order factors.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="1916" LastActivityDate="2011-02-25T12:53:09.590" CommentCount="1" />
  <row Id="7620" PostTypeId="1" CreationDate="2011-02-25T13:57:09.637" Score="1" ViewCount="91" Body="&lt;p&gt;My problem is this:&#xA;I have one dependent variable and 4 independent ones: one is age and the other three are temperament dimensions. I did 3 sets of two-way ANOVAs. The first independent variable is always the same (age) and the second is always different - one of the tempeament dimensions. In one case I get that age has significant effecet, the temeprament1 does not, and no interaction. in other case I get that age doesn't have the significant effecet, but still no influence of temperament2, and no significant effect of interaction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;one-way ANOVA for age shows there is significant effect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is how do I interpret the data? My plan was to say if there is effect of age and temperament and interaction of age and temeprament dimensions, but now - the effect of age is sometimes there, and sometimes not?!&lt;/p&gt;&#xA;" OwnerUserId="3433" LastActivityDate="2011-02-25T16:43:05.643" Title="Main effect of the first independent variable in two-way ANOVa lost depending on the second independent variable" Tags="&lt;anova&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="7621" PostTypeId="2" ParentId="7613" CreationDate="2011-02-25T14:09:13.833" Score="1" ViewCount="" Body="&lt;p&gt;@ q2: Ifyou use all principal components, then this is only a rotation in the multidimensional space and the euclidean(!) distances between datapoints are not affected.     &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, for instance taxicab-distances are affected by rotation. (This ca be seen if you consider the unit square and the taxicab-distance between the two edges of the diagonal. Unrotated you have two times one border s the distance is 2, but if you rotate it by 45 deg you have the distance sqrt(2))     &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furthermore, once you employ PCA then your goal is to reduce dimensionality, thus usually you discard variance/covariance (according to the ignored less-principal components), and this cannot be reflected by any selection of sets of items in a &quot;canned&quot; cluster-analysis, so the solutions must be different.&lt;br&gt;&#xA;&lt;em&gt;[opinion] Well, that the PCA and the unrotated clusters are usually not perfectly equal need not mean, that the PCA-based clusters are worse/bad/meaningless [/opinion]&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="1818" LastActivityDate="2011-02-25T14:09:13.833" />
  <row Id="7622" PostTypeId="2" ParentId="7610" CreationDate="2011-02-25T15:05:57.503" Score="5" ViewCount="" Body="&lt;p&gt;Random Forest&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Fast, robust, good accuracy, in most cases nothing to tune, requires no normalization, immune to collinearity, generates quite good error approximation and useful importance ranking as a side effect of training, trivially parallel, predicts in a blink of an eye.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Drawbacks: slower than trivial methods like kNN or NB, works best with equal classes, worse accuracy than SVM for problems desperately requiring kernel trick, is a hard black-box, does not make coffee. &lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2011-02-25T15:05:57.503" CommunityOwnedDate="2011-02-25T15:05:57.503" CommentCount="7" />
  <row Id="7623" PostTypeId="2" ParentId="7620" CreationDate="2011-02-25T15:45:06.957" Score="1" ViewCount="" Body="&lt;p&gt;Unfortunately there is no good short answer to your question--not one that is likely to help you understand these findings on more than a superficial level.  What is required is for you to begin exploring the literature on statistical control and on partialling out (adjusting for, controlling for, or holding constant) extraneous variables.  One might spend the better part of a semester on this topic, and there are sources at all levels of sophistication that you might read.  James Davis' &lt;em&gt;The Logic of Causal Order&lt;/em&gt; and Dana Keller's &lt;em&gt;The Tao of Statistics&lt;/em&gt; are two very short, user-friendly, introductory books that come to mind.  My short, very basic piece at &lt;a href=&quot;http://www.integrativestatistics.com/partial.htm&quot; rel=&quot;nofollow&quot;&gt;http://www.integrativestatistics.com/partial.htm&lt;/a&gt; might also be of some use as a way of orienting you before you delve into more detailed treatments.&lt;/p&gt;&#xA;" OwnerUserId="2669" LastActivityDate="2011-02-25T15:45:06.957" />
  <row Id="7624" PostTypeId="2" ParentId="7579" CreationDate="2011-02-25T16:31:35.000" Score="0" ViewCount="" Body="&lt;p&gt;If you have access to SAS 9.2 you could use PROC COUNTREG. It's a fairly new procedure and if you poke around the SAS site you can find out about it in the SAS/ETS(R) 9.2 User's Guide. COUNTREG does count modeling with or without zero inflation, has a &quot;by&quot; clause to split analyses, and allows both categorical and continuous variables. &lt;/p&gt;&#xA;" OwnerUserId="3434" LastActivityDate="2011-02-25T16:31:35.000" />
  <row Id="7625" PostTypeId="1" CreationDate="2011-02-25T16:36:28.977" Score="2" ViewCount="204" Body="&lt;p&gt;I have a zero-inflated negative binomial model for highly skewed, high zero-count data, n=6800. The null model does not reject the zinb model (and Stata countfit also indicated zinb over other count models). Yet, when I run the model with more than the minimum of covariates, the inflation intercept has a coefficient but no error, no t, no p-value. Does this mean the model is incorrect? Or just the zero-inflated assumptions? Any suggestions? &lt;/p&gt;&#xA;" OwnerUserId="3434" LastActivityDate="2011-05-26T17:50:29.717" Title="Help Interpreting Zero-inflated negative binomial model output" Tags="&lt;regression&gt;&lt;nonparametric&gt;&lt;negative-binomial&gt;" AnswerCount="1" />
  <row Id="7626" PostTypeId="2" ParentId="7620" CreationDate="2011-02-25T16:43:05.643" Score="1" ViewCount="" Body="&lt;p&gt;Sounds like age is correlated with one or more of your temperament measures, which means you're violating the assumptions of ANOVA/regression. You might want to instead look at path analysis to ascertain the relationships amongst your variables. &lt;/p&gt;&#xA;" OwnerUserId="364" LastActivityDate="2011-02-25T16:43:05.643" CommentCount="3" />
  <row Id="7627" PostTypeId="2" ParentId="7625" CreationDate="2011-02-25T16:45:40.170" Score="0" ViewCount="" Body="&lt;p&gt;Are you using Stata's zinb command to run your regressions?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When there are no standard error estimates, the things that comes to mind as possible explanations are that the model may be too collinear, or that there may be zero-cells for some variables.&lt;/p&gt;&#xA;" OwnerUserId="3309" LastActivityDate="2011-02-25T16:45:40.170" />
  <row Id="7629" PostTypeId="2" ParentId="7308" CreationDate="2011-02-25T17:33:31.707" Score="7" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;I think you're right.&lt;/strong&gt;  Let's distill your argument to its essence:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;$\widehat \theta_N$ minimizes the function $Q$ defined as $Q(\theta) = {1 \over N}\sum_{i=1}^N q(w_i,\theta).$&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Let $H$ be the Hessian of $Q$, whence $H(\theta) = \frac{\partial^2 Q}{\partial \theta_i \partial \theta_j}$ by definition and this in turn, by linearity of differentiation, equals $\frac{1}{N}\sum_{i=1}^N H(w_i, \theta_n)$.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Assuming $\widehat \theta_N$ lies in the interior of the domain of $Q$, then $H(\widehat \theta_N)$ must be positive semi-definite.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;This is merely a statement about the function $Q$: how it is defined is merely a distraction, except insofar as the assumed second order differentiability of $q$ with respect to its second argument ($\theta$) assures the second order differentiability of $Q$.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Finding M-estimators can be tricky.  Consider these data provided by @mpiktas:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;{1.168042, 0.3998378}, {1.807516, 0.5939584}, {1.384942, 3.6700205}, {1.327734, -3.3390724}, {1.602101, 4.1317608}, {1.604394, -1.9045958}, {1.124633, -3.0865249}, {1.294601, -1.8331763},{1.577610, 1.0865977}, { 1.630979, 0.7869717}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The R procedure to find the M-estimator with $q((x,y),\theta)=(y-c_1x^{c_2})^4$ produced the solution $(c_1, c_2)$ = $(-114.91316, -32.54386)$.  The value of the objective function (the average of the $q$'s) at this point equals 62.3542.  Here is a plot of the fit:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/9NAmz.png&quot; alt=&quot;Fit 1&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a plot of the (log) objective function in a neighborhood of this fit:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/lG5FZ.png&quot; alt=&quot;Objective 1&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Something is fishy here:&lt;/strong&gt; the parameters of the fit are extremely far from the parameters used to simulate the data (near $(0.3, 0.2)$) and we do not seem to be at a minimum: we are in an extremely shallow valley that is sloping towards larger values of both parameters:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/PUWRw.png&quot; alt=&quot;Objective 1, 3D view&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The negative determinant of the Hessian at this point &lt;em&gt;confirms&lt;/em&gt; that this is not a local minimum!&lt;/strong&gt;  Nevertheless, when you look at the z-axis labels, you can see that &lt;em&gt;this function is flat to five-digit precision&lt;/em&gt; within the entire region, because it equals a constant 4.1329 (the logarithm of 62.354).  This probably led the R function minimizer (with its default tolerances) to conclude it was near a minimum.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In fact, the solution is far from this point.  To be sure of finding it, I employed the computationally expensive but highly effective &quot;&lt;a href=&quot;http://reference.wolfram.com/mathematica/tutorial/UnconstrainedOptimizationPrincipalAxisMethod.html&quot; rel=&quot;nofollow&quot;&gt;Principal Axis&lt;/a&gt;&quot; method in &lt;em&gt;Mathematica&lt;/em&gt;, using 50-digit precision (base 10) to avoid possible numerical problems.  It finds a minimum near $(c_1, c_2) = (0.02506, 7.55973)$ where the objective function has the value 58.292655: about 6% smaller than the &quot;minimum&quot; found by R.  This minimum occurs in an extremely flat-looking section, but I can make it look (just barely) like a true minimum, with elliptical contours, by exaggerating the $c_2$ direction in the plot:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/HLnKR.png&quot; alt=&quot;Objective 2&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The contours range from 58.29266 in the middle all the way up to 58.29284 in the corners(!).  Here's the 3D view (again of the log objective):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/Exo3U.png&quot; alt=&quot;Objective 2, 3D view&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here the Hessian is positive-definite: its eigenvalues are 55062.02 and 0.430978.  Thus &lt;strong&gt;this point is a local minimum&lt;/strong&gt; (and likely a global minimum).  Here is the fit it corresponds to:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/Qkfnt.png&quot; alt=&quot;Fit 2&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think it's better than the other one.  The parameter values are certainly more realistic and it's clear we're not going to be able to do much better with this family of curves.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;There are useful lessons we can draw from this example:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Numerical optimization can be difficult, especially with nonlinear fitting and non-quadratic loss functions.  Therefore:&lt;/li&gt;&#xA;&lt;li&gt;Double-check results in as many ways as possible, including:&lt;/li&gt;&#xA;&lt;li&gt;Graph the objective function whenever you can.&lt;/li&gt;&#xA;&lt;li&gt;When numerical results appear to violate mathematical theorems, be extremely suspicious.&lt;/li&gt;&#xA;&lt;li&gt;When statistical results are surprising--such as the surprising parameter values returned by the R code--be extra suspicious.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-02-26T22:16:44.043" LastActivityDate="2011-02-26T22:16:44.043" CommentCount="5" />
  <row Id="7630" PostTypeId="1" AcceptedAnswerId="7671" CreationDate="2011-02-25T18:01:07.703" Score="5" ViewCount="174" Body="&lt;p&gt;I am clustering probability distributions using the &lt;a href=&quot;http://www.psi.toronto.edu/index.php?q=affinity%20propagation&quot; rel=&quot;nofollow&quot;&gt;Affinity Propagation&lt;/a&gt; algorithm, and I plan to use Jensen-Shannon Divergence as my distance metric.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it correct to use JSD itself as the distance, or JSD squared? Why? What differences would result from choosing one or the other?  &lt;/p&gt;&#xA;" OwnerUserId="2759" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-02-27T16:51:14.127" LastActivityDate="2011-02-27T16:51:14.127" Title="Clustering: Should I use the Jensen-Shannon Divergence or its square?" Tags="&lt;machine-learning&gt;&lt;clustering&gt;&lt;entropy&gt;&lt;distance-functions&gt;" AnswerCount="1" />
  <row Id="7631" PostTypeId="1" AcceptedAnswerId="7645" CreationDate="2011-02-25T19:29:22.413" Score="4" ViewCount="206" Body="&lt;p&gt;Suppose we have random variable $X_1$ distributed as $U[0,1]$ and $X_2$ distributed as  $U[0,X_1]$, where $U[a,b]$ means uniform distribution in interval $[a,b]$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was able to compute joint pdf of $(X_1,X_2)$ and marginal pdf of $X_1$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ p(x_1,x_2) = \frac{1}{x_1}, \text{ for }\quad 0\le x_1\le 1, \quad 0\le x_2 \le x_1,$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ p(x_1)= 1, \text{ for } \quad 0\le x_1\le 1.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However while computing marginal pdf of $X_2$ I am encountering limits problem. The resultant of integral through marginal of $X_2$ is $\log(X_1)$ and the limits are from 0 to 1. As $\log(X_1)$ is not defined for $X_1=0$, I am facing a difficulty.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Am I wrong somwhere? Thanks.&lt;/p&gt;&#xA;" OwnerUserId="1102" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-29T01:02:03.633" LastActivityDate="2011-04-29T01:02:03.633" Title="Problem calculating joint and marginal distribution of two uniform distributions" Tags="&lt;pdf&gt;&lt;marginal&gt;" AnswerCount="2" CommentCount="5" />
  <row Id="7632" PostTypeId="2" ParentId="7571" CreationDate="2011-02-25T19:48:31.010" Score="2" ViewCount="" Body="&lt;p&gt;Seems to me like a Kalman smoothing problem, which essentially computes the mean value of the state given past, present and future observations, with a computational effort similar to two Kalman filter passes. You might want to check any of the good books in existence which deal with the Kalman smoother, among them &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0198523548&quot; rel=&quot;nofollow&quot;&gt;Durbin-Koopman&lt;/a&gt; and &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0486439380&quot; rel=&quot;nofollow&quot;&gt;Anderson-Moore&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="892" LastActivityDate="2011-02-25T19:48:31.010" CommentCount="1" />
  <row Id="7633" PostTypeId="2" ParentId="7631" CreationDate="2011-02-25T19:54:44.657" Score="2" ViewCount="" Body="&lt;p&gt;You should not have $X_1$ in the marginal distribution for $X_2$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would expect you to get $P(X_2 \le x_2) = x_2 (1-\log(x_2))$ and so the derivative gives a marginal density of $-\log(x_2)$.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This comes from $P(X_2 \le x_2 |X_1=x_1) = 1$ if $x_1 \le x_2$, and $ P(X_2 \le x_2 |X_1=x_1) = \frac{x_2}{x_1}$ if $x_2 \le x_1$ so the integral is &#xA;$$P(X_2 \le x_2) = \int_{x_1=0}^{x_2} dx_1 + \int_{x_1=x_2}^{1} \frac{x_2}{x_1} dx_1$$ &#xA;$$ = \left[ x_1 \right]_{x_1=0}^{x_1=x_2} + \left[x_2 \log(x_1)\right]_{x_1=x_2}^{x_1=1} $$&#xA;$$ = x_2 - 0 +x_2 \log(1) - x_2 \log(x_2) $$&#xA;$$ = x_2 (1-\log(x_2))$$ &lt;/p&gt;&#xA;" OwnerUserId="2958" LastEditorUserId="2958" LastEditorDisplayName="" LastEditDate="2011-02-25T23:06:14.040" LastActivityDate="2011-02-25T23:06:14.040" CommentCount="6" />
  <row Id="7634" PostTypeId="2" ParentId="7618" CreationDate="2011-02-25T20:57:11.160" Score="1" ViewCount="" Body="&lt;p&gt;One option is to take a single tree and make it small (e.g. 3-5 features). This will show a classification process that is easy to interpret.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want something closer to a &quot;score&quot;, then try using a linear classifier (such as Naive Bayes or SVM). Again, use few (3-5) features and discretize them into levels if you want. They will compute something like&#xA;$$Score = w_1 \cdot f_1 + w_2 \cdot f_2 + ... ,$$&#xA;where $f_i$ are feature values (1, 2, or 3 in your example) and $w_i$ is the weight for that feature (i.e. how many points you get for that feature).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The disadvantage of all this is that you need to use few features and small classifiers for them to be easy to interpret. But they may not work as well as your big classifier.&lt;/p&gt;&#xA;" OwnerUserId="3369" LastActivityDate="2011-02-25T20:57:11.160" />
  <row Id="7635" PostTypeId="2" ParentId="1099" CreationDate="2011-02-25T21:34:57.103" Score="0" ViewCount="" Body="&lt;p&gt;One way of proceeding is to construct various null models each of which assume factors are independent of one another. The independence assumption often makes these easy to construct. Then the predicted joint densities are the products of the marginal densities.  To the degree the actual data are consistent with these, you know factors are independent. If they are greater or lesser than the joint prediction, you may be able to infer they co-vary positively or negatively. Be careful to consider numbers of observations in each case, and you may be able to do that formally by treating populations as extended hypergeometrics. This is all in the spirit of the Fisher Exact Test, but Fisher actually formulated it so more general situations could be modeled.  See, for example, Discrete Multivariate Analysis: Theory and Practice, by Yvonne M. Bishop, Stephen E. Fienberg, Paul W. Holland, R.J. Light, F. Mosteller, and The Analysis of Cross-Classified Categorical Data, by Stephen E. Fienberg.&lt;/p&gt;&#xA;" OwnerUserId="3437" LastActivityDate="2011-02-25T21:34:57.103" />
  <row Id="7636" PostTypeId="2" ParentId="3955" CreationDate="2011-02-25T21:42:09.303" Score="1" ViewCount="" Body="&lt;p&gt;Caution that arrival rates of users at Web sites are nasty series, tend to be overdispersed (from a Poisson standpoint), so consider negative binominal distributions to look at arrivals, and their fitting. Also, you may want to examine the order statistics of the sites on each day rather than their numbers.&lt;/p&gt;&#xA;" OwnerUserId="3437" LastActivityDate="2011-02-25T21:42:09.303" />
  <row Id="7637" PostTypeId="1" AcceptedAnswerId="7641" CreationDate="2011-02-25T22:30:25.270" Score="1" ViewCount="96" Body="&lt;p&gt;Suppose I roll a 4-side dice 25 time to get a final sum between 25 and 100. How I calculated the distribution of probability for each sum between 25 and 100?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you&lt;/p&gt;&#xA;" OwnerUserId="3438" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-02-25T22:54:12.943" LastActivityDate="2011-02-26T01:51:47.000" Title="Multinomial distribution for 4 side dice roll" Tags="&lt;probability&gt;&lt;multinomial&gt;&lt;dice&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="7638" PostTypeId="1" CreationDate="2011-02-25T22:48:52.360" Score="5" ViewCount="196" Body="&lt;h3&gt;Context:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;I am working with an ordinal logistic model and trying to interpret/present the results.  The model has two continuous predictors of interests, and a mix of continuous and categorical controls.  I was hoping to graph the predicted likelihood of the top outcome (being accepted into a school) across multiple levels of my IVs of interest.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using R's predict() function to generate predicted likelihoods.  For my IVs of interest, I chose a range of reasonable values (i.e. mean +- 1 SD).  For the continuous predictors, I can use sensible baseline values (usually 0) because they are mean-centered or standardized.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am trying to work out how to approach the categorical predictors. I've explored my options by plugging in different values, and in most cases the result is just a small shift in the output curve.  For one variable however, the differences are huge, so I need to find a way to present results that are general to the different levels of that variable.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Perhaps an example would help clarify.  In these two graphs, the two IVs of interest are plotted on the x-axis and as the 3 lines.  Each graph shows the output given a single level of my troublesome categorical control, &quot;Admitting School&quot; (which has 4 levels total)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/5rIXW.png&quot; alt=&quot;enter image description here&quot;&gt;&#xA;&lt;img src=&quot;http://i.imgur.com/gu1X8.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://swift.cbdr.cmu.edu/files/fsd/&quot; rel=&quot;nofollow&quot;&gt;Other graphs and R syntax here if you're curious&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Question:&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;How should I represent the model across all levels of the categorical variables in a single graph?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;Initial Thoughts:&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Aggregate predicted values across each level of Admitting School with some sort of weighted average.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://stats.stackexchange.com/questions/3623/plugging-in-mean-values-proportions-to-a-logistic-regression-with-continuous-disc&quot;&gt;This post&lt;/a&gt; suggests using the proportion of cases of each type as the input for each variable.  As in, if 32% of my cases came from School 1, I would use .32*B-school1 in the prediction formula.  I don't know how to do that in R, since those variables are factors, but if it's an appropriate approach, I'm sure I could figure it out.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Sorry for the verbosity and thanks in advance for any help.&lt;/p&gt;&#xA;" OwnerUserId="3388" LastEditorUserId="3388" LastEditorDisplayName="" LastEditDate="2011-02-26T18:54:20.857" LastActivityDate="2011-02-26T18:54:20.857" Title="Calculating predicted values from categorical predictors in logistic regression" Tags="&lt;data-visualization&gt;&lt;logistic-regression&gt;&lt;categorical-data&gt;" AnswerCount="1" />
  <row Id="7639" PostTypeId="1" CreationDate="2011-02-25T22:50:28.983" Score="2" ViewCount="224" Body="&lt;p&gt;I have 4301 lines of data from my science project.  I have to do unorthodox statistical analysis for my results because they are 4-dimensional; I have three independent variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With Excel I can do a lot of analysis, even multivariable linear regression, but what I need is a step up:  Multivariable nonlinear regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have R.  Is there a way I can do it with R?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, is there a way to generate multiple 3-dimensional graphs with slices of the data automatically?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT:  I've searched with no luck.  Is there &lt;em&gt;any&lt;/em&gt; way of doing multivariable nonlinear regression automatically?&lt;/p&gt;&#xA;" OwnerUserId="3439" LastEditorUserId="3439" LastEditorDisplayName="" LastEditDate="2011-02-26T00:16:22.893" LastActivityDate="2011-02-28T15:30:51.777" Title="Doing statistical analysis and charting with large data?" Tags="&lt;r&gt;&lt;multivariate-analysis&gt;&lt;large-data&gt;" AnswerCount="4" CommentCount="2" />
  <row Id="7640" PostTypeId="2" ParentId="7639" CreationDate="2011-02-26T00:23:13.350" Score="4" ViewCount="" Body="&lt;p&gt;It's not real clear what you're asking.  Is the following useful?  I knocked it down from 4301 to 430 lines in case your machine is slow.  You can move the little green box around to see how those points fit in all of the other graphs.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(TeachingDemos)&#xA;&#xA;#Build 3 independent variables (for speed reasons, I knocked it down to 430)&#xA;x &amp;lt;- rnorm(430)&#xA;y &amp;lt;- runif(430)&#xA;z &amp;lt;- rnorm(430)&#xA;&#xA;#Build the dependent variable&#xA;dep &amp;lt;- 2 + (0.5 * x) + (0.9 * y) + (-0.8 * z)&#xA;&#xA;#Put everything in a data frame&#xA;df &amp;lt;- data.frame(x=x, y=y, z=z, dep=dep)&#xA;&#xA;#Plot it and look at chunks of the data&#xA;dra &amp;lt;- tkBrush(df)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/hp5fy.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="2775" LastEditorUserId="2775" LastEditorDisplayName="" LastEditDate="2011-02-26T00:30:37.350" LastActivityDate="2011-02-26T00:30:37.350" CommentCount="1" />
  <row Id="7641" PostTypeId="2" ParentId="7637" CreationDate="2011-02-26T01:51:47.000" Score="1" ViewCount="" Body="&lt;p&gt;There are several approaches:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Use the normal distribution as an approximation as described in whuber's link to the &lt;a href=&quot;http://stats.stackexchange.com/questions/3614/how-to-easily-determine-the-results-distribution-for-multiple-dice&quot;&gt;earlier question&lt;/a&gt;, or use some of the other suggestions there.  The mean will be 62.5 while the variance will be 31.25.&lt;/li&gt;&#xA;&lt;li&gt;Do a recurrence starting with $f(i,0)=0$ except when $i=0$ in which case $f(0,0)=1$, and then use $f(i,j+1)=f(i-4,j)/4+f(i-3,j)/4+f(i-2,j)/4+f(i-1,j)/4$ to find $f(i,25)$&lt;/li&gt;&#xA;&lt;li&gt;Expand $(x/4+x^2/4+x^3/4+x^4/4)^{25}$, for example by putting &lt;code&gt;Expand[(x/4+x^2/4+x^3/4+x^4/4)^25]&lt;/code&gt; into &lt;a href=&quot;http://www.wolframalpha.com&quot; rel=&quot;nofollow&quot;&gt;Wolfram Alpha&lt;/a&gt; and press &quot;=&quot; then &quot;show more terms&quot; several times&lt;/li&gt;&#xA;&lt;li&gt;Hope somebody has already done some recurrence calculations for you, which they have in rows 925-1000 of an &lt;a href=&quot;http://oeis.org/A008287/b008287.txt&quot; rel=&quot;nofollow&quot;&gt;OEIS table&lt;/a&gt; and then divide the results by $4^{25} = 1125899906842624$ &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-02-26T01:51:47.000" CommentCount="1" />
  <row Id="7642" PostTypeId="2" ParentId="7639" CreationDate="2011-02-26T02:06:46.497" Score="1" ViewCount="" Body="&lt;p&gt;Save your table as a csv file, with one header row that contains the names of your 4 variables.  Save the following code as an R script file in the same directory as your data, and then run it.  Lets say your variables are called X1, X2, X3, and Y, where X1, X2, X3 are your independent variables and Y is your dependent variable.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;MyData &amp;lt;- read.csv('path/to/MyData.csv')&#xA;model &amp;lt;- lm(Y~X1+X2+X3,data=MyData)&#xA;predictions &amp;lt;- predict(model)&#xA;&#xA;model&#xA;plot(model)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Building a linear regression model and diagnosing it is relatively simple in R, even with 4000+ lines of data.  What exactly are you trying to do with this dataset?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can build a non-linear model using the &quot;loess&quot; or &quot;glm&quot; commands, but specifying and diagnosing such models is more difficult than simple linear models.  Again, if you can be a bit clearer about what exactly you need to do, we can help you more.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Fitting a linear model with 3 dependent variables and 4000 observations isn't exactly unorthodox, and you can do it quickly and easily in R.&lt;/p&gt;&#xA;" OwnerUserId="2817" LastEditorUserId="2817" LastEditorDisplayName="" LastEditDate="2011-02-28T15:30:51.777" LastActivityDate="2011-02-28T15:30:51.777" CommentCount="5" />
  <row Id="7643" PostTypeId="2" ParentId="7639" CreationDate="2011-02-26T02:11:08.043" Score="2" ViewCount="" Body="&lt;p&gt;What sort of nonlinear model do you want?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4300 lines of data is not very large in today's world.  Data sets containing millions of records exist.  Certainly 4300 will be no problem&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This link to a work by John Fox may be helpful&#xA;&lt;a href=&quot;http://cran.r-project.org/doc/contrib/Fox-Companion/appendix-nonlinear-regression.pdf&quot; rel=&quot;nofollow&quot;&gt;http://cran.r-project.org/doc/contrib/Fox-Companion/appendix-nonlinear-regression.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;it highlights use of the nls function in R, which is part of the nls library&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps&lt;/p&gt;&#xA;" OwnerUserId="686" LastActivityDate="2011-02-26T02:11:08.043" CommentCount="1" />
  <row Id="7644" PostTypeId="1" CreationDate="2011-02-26T03:19:55.543" Score="6" ViewCount="169" Body="&lt;p&gt;I am looking for methods which can be used to estimate the &quot;OLS&quot; measurement error model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$y_{i}=Y_{i}+e_{y,i}$$&#xA;$$x_{i}=X_{i}+e_{x,i}$$&#xA;$$Y_{i}=\alpha + \beta X_{i}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where the errors are independent normal with unknown variances $\sigma_{y}^{2}$ and $\sigma_{x}^{2}$.  &quot;Standard&quot; OLS won't work in this case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Measurement_error_model&quot; rel=&quot;nofollow&quot;&gt;Wikipedia&lt;/a&gt; has some unappealing solutions - the two given force you to assume that either the &quot;variance ratio&quot; $\delta=\frac{\sigma_{y}^{2}}{\sigma_{x}^{2}}$ or the &quot;reliability ratio&quot; $\lambda=\frac{\sigma_{X}^{2}}{\sigma_{x}^{2}+\sigma_{X}^{2}}$ is known, where $\sigma_{X}^2$ is the variance of the true regressor $X_i$.  I am not satisfied by this, because how can someone who doesn't know the variances know their ratio?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyways, are there any other solutions besides these two which don't require me to &quot;know&quot; anything about the parameters?       &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Solutions for just the intercept and slope are fine.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-02-26T17:50:46.793" LastActivityDate="2011-03-29T17:38:35.623" Title="Methods for fitting a &quot;simple&quot; measurement error model" Tags="&lt;regression&gt;&lt;estimation&gt;&lt;errors-in-variables&gt;" AnswerCount="1" CommentCount="4" FavoriteCount="2" />
  <row Id="7645" PostTypeId="2" ParentId="7631" CreationDate="2011-02-26T03:46:46.097" Score="3" ViewCount="" Body="&lt;p&gt;In the &quot;marginalisation&quot; integral, the lower limit for $x_1$ is not $0$ but $x_2$ (because of the $0&amp;lt;x_2&amp;lt;x_1$ condition).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the integral should be:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p(x_2)=\int p(x_1,x_2) dx_1=\int \frac{I(0\leq x_2\leq x_1\leq 1)}{x_1} dx_1=\int_{x_2}^{1} \frac{dx_1}{x_1}=log\big(\frac{1}{x_2}\big)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You have stumbled across, what I think is one of the hardest parts of statistical integrals - determining the limits of integration.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;NOTE: This is consistent with Henry's answer, mine is the PDF, and his is the CDF.  Differentiating his answer gives you mine, which shows we are both right.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastEditorUserId="2392" LastEditorDisplayName="" LastEditDate="2011-02-26T03:55:16.637" LastActivityDate="2011-02-26T03:55:16.637" CommentCount="2" />
  <row Id="7646" PostTypeId="2" ParentId="7534" CreationDate="2011-02-26T04:23:53.270" Score="0" ViewCount="" Body="&lt;p&gt;The term &quot;statistically significant&quot; is a bit arbitrary, because it is defined relative to a null hypothesis.  The basic way to test this is to use your &quot;null hypothesis&quot; to define a set of Expected values for each cell.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The standard way to measure the error is $r_i^2=\frac{(O_i-E_i)^2}{E_i}$ (O=observed count, E=expected count under the null hypothesis), which is the &quot;partition&quot; of the chi-square.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another alternative is to use $d_i=O_i log(\frac{O_i}{E_i})$ (&quot;entropy&quot; statistic).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Both are approximately equal in large tables, with big expected values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &quot;significant cells&quot; have large values of these two values&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-02-26T04:23:53.270" />
  <row Id="7647" PostTypeId="1" CreationDate="2011-02-26T07:22:39.430" Score="1" ViewCount="52" Body="&lt;p&gt;I have been looking for an alternative to Google Docs as a web based spreadsheet application as my employer's corporate firewall now blocks this application.  I kicked around a few alternatives such as Zoho.com.  I found the latter practically unusable.  All its applications took a long time to load and really did not work well if at all most of the time. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I came across EditGrid, I was delighted.  It feels pretty much like Excel on a lot of counts.  However, there is no apparent support besides a Help Center that is inactive and invariably does not address the problems I have.  For instance, I have no idea how to include titles and legends within charts.  I also don't see how one can create folders.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I thought maybe someone would be familiar with EditGrid and either being able to help me out directly or point me in a direction of a support group somewhere.  Thanks in advance for any info. &lt;/p&gt;&#xA;" OwnerUserId="1329" LastEditorUserId="1329" LastEditorDisplayName="" LastEditDate="2011-02-26T18:09:53.463" LastActivityDate="2011-02-26T18:09:53.463" ClosedDate="2011-03-02T06:16:43.527" Title="Does anyone use the web based spreadsheet EditGrid.com?" Tags="&lt;application&gt;&lt;web&gt;" CommentCount="3" />
  <row Id="7648" PostTypeId="1" CreationDate="2011-02-26T08:32:17.770" Score="4" ViewCount="230" Body="&lt;p&gt;I'm looking at doing text classification/spam filtering using naive Bayesian classifiers with the e1071 or klaR package on R. Is there a good tutorial out there to describe this? I'm kind of stuck because I'm not sure what to use as the data to input into the NaiveBayes function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some help very much appreciated, thanks!&lt;/p&gt;&#xA;" OwnerUserId="3442" LastActivityDate="2011-02-28T21:16:27.870" Title="Spam filtering using naive Bayesian classifiers with the e1071/klaR package on R" Tags="&lt;r&gt;&lt;classification&gt;&lt;naive-bayes&gt;&lt;e1071&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="7649" PostTypeId="2" ParentId="7389" CreationDate="2011-02-26T08:56:37.837" Score="1" ViewCount="" Body="&lt;p&gt;You would not use a paired sample t-test. The reason for this is that a particular random seed cannot be assumed to bias the outcome of both algorithms in the same way, even if that random seed is only used to generate the population and not for later operations such as mutation and selection. In other words, its logically possible that, under one algorithm, a given population will evolve better than the average for that algorithm, but will perform in the opposite way under another. If you have reason to believe that there is a similar connection between seed and performance for both algorithms, you can test this using a Pearson correlation coefficient to compare each seed's performance on both tests. By default, however, I would assume that there is no connection, especially if you have reasonably large populations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as running more than 10 times, of course more samples are always better, though your computational resources obviously may be a limiting factor. It could be a good idea to generate a power curve, which will show you the relationship between the size of difference needed for statistical significance at you're alpha level, and the SD and n. In other words, at a given n and SD, how big does the difference have to be? &lt;a href=&quot;http://moon.ouhsc.edu/dthompso/CDM/power/hypoth.htm&quot; rel=&quot;nofollow&quot;&gt;http://moon.ouhsc.edu/dthompso/CDM/power/hypoth.htm&lt;/a&gt; &amp;lt;-- see bottom of page for power curve info.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, if you are running a genetic algorithm that actually has a defined stopping point, as yours does, you can just do a plain unpaired t-test on the number of generations needed to find the solution. Otherwise, quantifying algorithm performance tends to get a bit trickier &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as pitfalls, and generalizability of algorithm efficiency to other problems, you really cannot take effectiveness of your algorithm for granted when porting it to other problems. In my experience, genetic algorithms usually have to be tweaked quite a bit for each new problem that you apply them to. Having said that, depending on how diverse your set of 8 tests is, they may give you some indication of how generalizable your results are, and within which scope of applications they are generalizable.&lt;/p&gt;&#xA;" OwnerUserId="3443" LastEditorUserId="3443" LastEditorDisplayName="" LastEditDate="2011-03-03T08:31:27.563" LastActivityDate="2011-03-03T08:31:27.563" CommentCount="3" />
  <row Id="7650" PostTypeId="2" ParentId="7638" CreationDate="2011-02-26T09:08:28.737" Score="5" ViewCount="" Body="&lt;p&gt;My initial thought would have been to display the probability of of acceptance as a function of relative GPA for each of your four schools, using some kind of &lt;a href=&quot;http://cm.bell-labs.com/cm/ms/departments/sia/project/trellis/&quot; rel=&quot;nofollow&quot;&gt;trellis displays&lt;/a&gt;. In this case, facetting should do the job well as the number of schools is not so large. This is very easy to do with &lt;a href=&quot;http://cran.r-project.org/web/packages/lattice/index.html&quot; rel=&quot;nofollow&quot;&gt;lattice&lt;/a&gt; (&lt;code&gt;y ~ gpa | school&lt;/code&gt;) or &lt;a href=&quot;http://had.co.nz/ggplot2/&quot; rel=&quot;nofollow&quot;&gt;ggplot2&lt;/a&gt; (&lt;code&gt;facet_grid(. ~ school)&lt;/code&gt;). In fact, you can choose the conditioning variable you want: this can be school, but also situation at undergrad institution. In the latter case, you'll have 4 curves for each plot, and three three plot of &lt;code&gt;Prob(admitting) ~ GPA&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, if you are looking for effective displays of effects in GLM, I would recommend the &lt;a href=&quot;http://cran.r-project.org/web/packages/effects/index.html&quot; rel=&quot;nofollow&quot;&gt;effects&lt;/a&gt; package, from John Fox. Currently, it works with binomial and multinomial link, and ordinal logistic model. Marginalizing over other covariates is handled internally, so you don't have to bother with that. There are a lot of illustrations in the on-line help, see &lt;code&gt;help(effect)&lt;/code&gt;. But, for a more thorough overview of effects displays in GLM, please refer to&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Fox (2003). &lt;a href=&quot;http://www.jstatsoft.org/v08/i15/paper&quot; rel=&quot;nofollow&quot;&gt;Effect Displays in R for Generalised Linear Models&lt;/a&gt;. JSS 8(15).&lt;/li&gt;&#xA;&lt;li&gt;Fox and Andersen (2004). &lt;a href=&quot;http://socserv.mcmaster.ca/jfox/Misc/polytomous-effect-displays/polytomous-effect-displays.pdf&quot; rel=&quot;nofollow&quot;&gt;Effect displays for multinomial and proportional-odds logit models&lt;/a&gt;. ASA Methodology Conference -- Here is the corresponding &lt;a href=&quot;http://www.jstatsoft.org/v32/i01/paper&quot; rel=&quot;nofollow&quot;&gt;JSS paper&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-02-26T09:08:28.737" CommentCount="3" />
  <row Id="7651" PostTypeId="1" AcceptedAnswerId="7654" CreationDate="2011-02-26T12:14:50.623" Score="1" ViewCount="69" Body="&lt;p&gt;I have collected a bunch of statistical results in the form of &quot;YES&quot; and &quot;NO&quot; strings.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I would like to have a summary cell displaying &quot;YES&quot; if all these cells equal &quot;YES&quot; (or are empty).&lt;/p&gt;&#xA;" OwnerUserId="3446" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-02-27T10:49:04.070" LastActivityDate="2011-02-27T10:49:04.070" Title="How to assess whether a set of cells all have a specified value in Excel" Tags="&lt;excel&gt;" AnswerCount="1" />
  <row Id="7653" PostTypeId="1" AcceptedAnswerId="7658" CreationDate="2011-02-26T13:52:56.903" Score="4" ViewCount="110" Body="&lt;p&gt;I read &lt;em&gt;&lt;a href=&quot;http://eu.wiley.com/WileyCDA/WileyTitle/productCd-0471727814.html&quot; rel=&quot;nofollow&quot;&gt;Clinical Trials, a Methodologic Perspective&lt;/a&gt;&lt;/em&gt; (S. Piantadosi) as I was suggested by one of you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;According to the author:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;A trialist must understand two&#xA;  different modes of thinking that&#xA;  support the science-clinical and&#xA;  statistical. They both underlie the&#xA;  re-emergence of therapeutics as a&#xA;  modern science. Each method of&#xA;  reasoning arose independently and must&#xA;  be combined skillfully if they are to&#xA;  serve therapeutic questions&#xA;  effectively.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I cannot figure out what the author means by &quot;clinical reasoning&quot;. Can you help me to understand that notion?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you in advance&lt;/p&gt;&#xA;" OwnerUserId="3019" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-02-27T10:47:08.083" LastActivityDate="2011-02-27T10:47:08.083" Title="In relation to clinical trials, what is clinical reasoning in contrast to statistical reasoning?" Tags="&lt;clinical-trials&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="7654" PostTypeId="2" ParentId="7651" CreationDate="2011-02-26T14:31:51.987" Score="2" ViewCount="" Body="&lt;p&gt;Would you settle for a &quot;TRUE&quot; instead of a &quot;YES&quot; in case there are no &quot;NO&quot; cells?  If so, in a nearby cell type &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;=countif([first cell in range]:[last cell in range],&quot;NO&quot;)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then in an adjoining cell type&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;=and([previous cell]=&quot;0&quot;)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an example, if you had YES or NO strings in A1 through A20, then in A22 you would type&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;=countif(a1:a20,&quot;NO&quot;)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and in A21 you would type&#xA;&lt;code&gt;=and(a22=&quot;0&quot;)&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="2669" LastEditorUserId="582" LastEditorDisplayName="" LastEditDate="2011-02-26T16:05:11.483" LastActivityDate="2011-02-26T16:05:11.483" CommentCount="2" />
  <row Id="7655" PostTypeId="2" ParentId="7653" CreationDate="2011-02-26T14:46:08.097" Score="4" ViewCount="" Body="&lt;p&gt;I have not read the book, but my best guess would be that the author wants to points out that sometimes a critical reasoning has to be made when applying statistics to biological and medical issues.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The sole fact that, for instance, a treatment does not have a &quot;statistically significant&quot; effect does not imply that the treatment does not have a biological effect and viceversa. Statistics can tell you if a certain event is likely or unlikely to be happening, but does not give you any hint as to whether something is biologically plausible.&lt;/p&gt;&#xA;" OwnerUserId="582" LastActivityDate="2011-02-26T14:46:08.097" CommentCount="3" />
  <row Id="7656" PostTypeId="2" ParentId="3955" CreationDate="2011-02-26T14:46:45.880" Score="2" ViewCount="" Body="&lt;p&gt;There are definitely more and less complex ways to address this kind of problem. From the sound of things, you started out with a fairly simple solution (the formula you found on SO). With that kind of simplicity in mind, I thought I would revisit a few key points you make in (the current version of) your post.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far, you've said you want your measurement of &quot;site activity&quot; to capture:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Slope changes in visits/day over &quot;the past few&#xA;days&quot;&lt;/li&gt;&#xA;&lt;li&gt;Magnitude changes in visits/day over &quot;the past few days&quot;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;As @jan-galkowski points out, you also seem to be (at least tacitly) interested in the rank of the sites relative to each other along these dimensions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If that description is accurate, I would propose exploring the simplest possible solution that incorporates those three measures (change, magnitude, rank) as separate components. For example, you could grab:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The results of your SO solution to capture slope variation (although I would incorporate 3 or 4 days of data)&lt;/li&gt;&#xA;&lt;li&gt;Magnitude of each site's most recent visits/day value &lt;code&gt;(y2)&lt;/code&gt; divided by the mean visits/day for that site (&lt;code&gt;Y&lt;/code&gt;):&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;y2 / mean(Y)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For W0, W1, and W2 respectively, that yields 0.16, 1.45, and 2.35. (For the sake of interpretation, consider that a site whose most recent visits-per-day value was equal to it's mean visits-per-day would generate a result of 1). Note that you could also adjust this measure to capture the most recent 2 (or more) days:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;y2 + y1 / 2 * mean(Y)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That yields: 0.12, 1.33, 1.91 for your three sample sites.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you do, in fact, use the mean of each site's visit/day distribution for this kind of measure, I would also look at the distribution's standard deviation to get a sense of its relative volatility. The standard deviation for each site's visit/day distribution is: 12.69, 12.12, and 17.62. Thinking about the &lt;code&gt;y2/mean(Y)&lt;/code&gt; measure relative to the standard deviation is helpful because it allows you to keep the recent magnitude of activity on site W2 in perspective (bigger standard deviation = less stable/consistent overall).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, if you're interested in ranks, you can extend these approaches in that direction too. For example, I would think that knowing a site's rank in terms of the most recent visits per day values as well as the rank of each site's mean visits per day (the rank of &lt;code&gt;mean (Y)&lt;/code&gt; for each &lt;code&gt;W&lt;/code&gt; in &lt;code&gt;Wn&lt;/code&gt;) could be useful. Again, you can tailor to suit your needs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could present the results of all these calculations as a table, or create a regularly-updated visualization to track them on a daily basis.&lt;/p&gt;&#xA;" OwnerUserId="3396" LastActivityDate="2011-02-26T14:46:45.880" />
  <row Id="7658" PostTypeId="2" ParentId="7653" CreationDate="2011-02-26T18:29:26.887" Score="8" ViewCount="" Body="&lt;p&gt;I like @nico's response because it makes clear that statistical and pragmatic thinking shall come hand in hand; this also has the merit to bring out issues like statistical vs. clinical significance. But about your specific question, I would say this is clearly detailed in the two sections that directly follow your quote (p. 10).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rereading Piantadosi's textbook, it appears that the author means that &lt;strong&gt;clinical thinking&lt;/strong&gt; applies to the situation where a physician has to interpret the results of RCTs or other studies in order to decide of the best treatment to apply to a &lt;em&gt;new patient&lt;/em&gt;. This has to do with the extent to which (population-based) conclusions drawn from previous RCT might generalize to new, unobserved, samples. In a certain sense, such decision or judgment call for some form of clinical experience, which is not necessarily of the resort of a consistent statistical framework. Then, the author said that &quot;the solution offered by &lt;strong&gt;statistical reasoning&lt;/strong&gt; is to control the signal-to-noise ratio by design.&quot; In other words, this is a way to reduce uncertainty, and &quot;the chance of drawing incorrect conclusions from either good or bad data.&quot; In sum, both lines of reasoning are required in order to draw valid conclusions from previous (and 'localized') studies, and choose the right treatment to administer to a new individual, given his history, his current medication, etc. -- treatment efficacy follows from a good balance between statistical facts and clinical experience. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I like to think of a statistician as someone who is able to mark off the extent to which we can draw firm inferences from the observed data, whereas the clinician is the one that will have a more profound insight onto the implications or consequences of the results at the individual or population level.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-02-26T18:29:26.887" CommentCount="4" />
  <row Id="7659" PostTypeId="1" CreationDate="2011-02-26T19:28:19.357" Score="3" ViewCount="184" Body="&lt;p&gt;Hi there, I am a 3rd year undergraduate psychology student and I have used an established empathy questionnaire in my dissertation which has 33 items. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have reduced the number of responses from a 9 point Likert scale to a 4. This was done as I wanted a forced answer and to reduce ambiguity. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, do I need to carry out tests on the reduced responses, or does it not matter as the means are taken? Any help or guidance would be great as my supervisor is away for 2 weeks and person ment to be taken over has not replied to any of my emails.&lt;/p&gt;&#xA;" OwnerUserId="3447" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-03-30T18:40:59.383" LastActivityDate="2011-03-30T18:40:59.383" Title="Psychology empathy scale with rescaled Likert response" Tags="&lt;scales&gt;&lt;likert&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="7660" PostTypeId="2" ParentId="6538" CreationDate="2011-02-26T19:58:58.230" Score="1" ViewCount="" Body="&lt;p&gt;I have seen Statistical Inference, by Silvey, used by mathematicians who needed some workaday grasp of statistics. It's a small book, and should by rights be cheap. Looking at &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0412138204&quot; rel=&quot;nofollow&quot;&gt;http://www.amazon.com/Statistical-Inference-Monographs-Statistics-Probability/dp/0412138204/ref=sr_1_1?ie=UTF8&amp;amp;s=books&amp;amp;qid=1298750064&amp;amp;sr=1-1&lt;/a&gt;, it seems to be cheap second hand.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's old and concentrates on classical statistics. While it's not highly abstract, it is intended for a reasonably mathematical audience - many of the exercises are from the Cambridge (UK) Diploma in Mathematical Statistics, which is basically an MSc.&lt;/p&gt;&#xA;" OwnerUserId="1789" LastActivityDate="2011-02-26T19:58:58.230" CommunityOwnedDate="2011-02-26T19:58:58.230" />
  <row Id="7662" PostTypeId="2" ParentId="7639" CreationDate="2011-02-26T20:49:23.040" Score="0" ViewCount="" Body="&lt;p&gt;From your comment to my previous answer &quot;....I also want to automatically generate 3-dimensional graphs of each slice of the three IVs (as in leaving one IV constant while using the other two as x and y, and the DV as z).....&quot;  Try the following (using the same data as above):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(lattice)&#xA;&#xA;#Build 3 independent variables&#xA;x &amp;lt;- rnorm(4300)&#xA;y &amp;lt;- runif(4300)&#xA;z &amp;lt;- rnorm(4300)&#xA;&#xA;#Build the dependent variable&#xA;dep &amp;lt;- 2 + (0.5 * x) + (0.9 * y) + (-0.8 * z)&#xA;&#xA;#Put everything in a data frame&#xA;df &amp;lt;- data.frame(x=x, y=y, z=z, dep=dep)&#xA;&#xA;#Break up the x data into 4 chunks of equal counts&#xA;df$x4 &amp;lt;- equal.count(df$x, 4)&#xA;cloud(dep ~ y + z | x4, data = df,&#xA;      zlim = rev(range(df$dep)),&#xA;      screen = list(z = 130, x = -65), panel.aspect = 0.75,&#xA;      xlab = &quot;y&quot;, ylab = &quot;z&quot;, zlab = &quot;dep&quot;)&#xA;&#xA;#Break up the z data into 2 chunks of equal counts&#xA;df$z2 &amp;lt;- equal.count(df$z, 2)&#xA;cloud(dep ~ x + y | z2, data = df,&#xA;      zlim = rev(range(df$dep)),&#xA;      screen = list(z = 160, x = -80), panel.aspect = 0.75,&#xA;      xlab = &quot;x&quot;, ylab = &quot;y&quot;, zlab = &quot;dep&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The above generates two plots.   Below is the first plot.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/QWOmX.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="2775" LastEditorUserId="2775" LastEditorDisplayName="" LastEditDate="2011-02-26T21:08:01.213" LastActivityDate="2011-02-26T21:08:01.213" />
  <row Id="7664" PostTypeId="1" AcceptedAnswerId="7667" CreationDate="2011-02-26T21:28:48.913" Score="1" ViewCount="55" Body="&lt;p&gt;In many cases, we do not actually have to explicitly compute the normalization constant of a certain distribution, if we know its kernel and it corresponds to a known family.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a name for this &quot;trick&quot; that I can look up and read more about? I don't think there is something very deep about it, but I figured it must be carrying some name.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="3347" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-02-26T21:45:46.313" LastActivityDate="2011-02-26T22:07:32.897" Title="Is there a name for the &quot;kernel principle&quot;?" Tags="&lt;distributions&gt;&lt;normalization&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="7665" PostTypeId="2" ParentId="7659" CreationDate="2011-02-26T21:31:05.807" Score="4" ViewCount="" Body="&lt;p&gt;Whether you need to retest your instrument's validity and reliability now that you have changed the number of response options is a matter for you and your dissertation committee to decide; there is no standard statistical knowledge that can definitively answer the question for you.  You and/or your colleagues may find that, as you suggest, going from 9 to 4 reduces the arbitrariness with which respondents choose an option.  On the other hand, when you call the instrument &quot;established&quot; you are implying that it already had good psychometric properties the way it was, and so changing the method of measurement entails some risk.  Perhaps there are so few options now that the choice between, say, the first and the second has become more arbitrary.  Alternatively, perhaps the lack of a neutral option would seem unreasonable to some; the topic might require it.  If so, this would reduce reliability, and with it validity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The bottom line is, if there is good evidence for reliability and validity at 9 options, the burden could easily be placed on you to show that these properties have improved, or at least not worsened, at 4 options.&lt;/p&gt;&#xA;" OwnerUserId="2669" LastEditorUserId="2669" LastEditorDisplayName="" LastEditDate="2011-02-26T21:36:06.303" LastActivityDate="2011-02-26T21:36:06.303" CommentCount="2" />
  <row Id="7666" PostTypeId="2" ParentId="7644" CreationDate="2011-02-26T21:48:43.443" Score="2" ViewCount="" Body="&lt;p&gt;There are a range of possibilities described by J.W. Gillard in &lt;a href=&quot;http://www.cardiff.ac.uk/maths/resources/Gillard_Tech_Report.pdf&quot; rel=&quot;nofollow&quot;&gt;An Historical Overview&#xA;of Linear Regression with Errors in both Variables&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are not interested in details or reasons for choosing one method over another, just go with the simplest, which is to draw the line through the centroid $(\bar{x},\bar{y})$ with slope $\hat{\beta}=s_y/s_x$, i.e. the ratio of the observed standard deviations (making the sign of the slope the same as the sign of the covariance of $x$ and $y$); as you can probably work out, this gives an intercept on the $y$-axis of $\hat{\alpha}=\bar{y}-\hat{\beta}\bar{x}.$  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The merits of this particular approach are &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;it gives the same line comparing $x$ against $y$ as $y$ against $x$, &lt;/li&gt;&#xA;&lt;li&gt;it is scale-invariant so you do not need to worry about units,&lt;/li&gt;&#xA;&lt;li&gt;it lies between the two ordinary linear regression lines &lt;/li&gt;&#xA;&lt;li&gt;it crosses them where they cross each other at the centroid of the observations, and &lt;/li&gt;&#xA;&lt;li&gt;it is very easy to calculate.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The slope is the geometric mean of the slopes of the two ordinary linear regression slopes. It is also what you would get if you standardised the $x$ and $y$ observations, drew a line at 45&amp;deg; (or 135&amp;deg; if there is negative correlation) and then de-standardised the line.  It could also be seen as equivalent to making an implicit assumption that the variances of the two sets of errors are proportional to the variances of the two sets of observations; as far as I can tell, you claim not to know which way this is wrong. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is some R code to illustrate: the red line in the chart is OLS regression of $Y$ on $X$, the blue line is OLS regression of $X$ on $Y$, and the green line is this simple method. Note that the slope should be about 5.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X0 &amp;lt;- 1600:3600&#xA;Y0 &amp;lt;- 5*X0 + 700&#xA;X1 &amp;lt;- X0 + 400*rnorm(2001)&#xA;Y1 &amp;lt;- Y0 + 2000*rnorm(2001)&#xA;slopeOLSXY  &amp;lt;- lm(Y1 ~ X1)$coefficients[2]     #OLS slope of Y on X&#xA;slopeOLSYX  &amp;lt;- 1/lm(X1 ~ Y1)$coefficients[2]   #Inverse of OLS slope of X on Y&#xA;slopesimple &amp;lt;- sd(Y1)/sd(X1) *sign(cov(X1,Y1)) #Simple slope&#xA;c(slopeOLSXY, slopeOLSYX, slopesimple)         #Show the three slopes&#xA;plot(Y1~X1)&#xA;abline(mean(Y1) - slopeOLSXY  * mean(X1), slopeOLSXY,  col=&quot;red&quot;)&#xA;abline(mean(Y1) - slopeOLSYX  * mean(X1), slopeOLSYX,  col=&quot;blue&quot;)&#xA;abline(mean(Y1) - slopesimple * mean(X1), slopesimple, col=&quot;green&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2958" LastEditorUserId="2958" LastEditorDisplayName="" LastEditDate="2011-02-27T17:23:09.987" LastActivityDate="2011-02-27T17:23:09.987" CommentCount="22" />
  <row Id="7667" PostTypeId="2" ParentId="7664" CreationDate="2011-02-26T22:07:32.897" Score="1" ViewCount="" Body="&lt;p&gt;I was taught the stock phrase &quot;recognising the density of a &lt;em&gt;whatever&lt;/em&gt; distribution,&quot; to use as part of a mathematical proof. I agree this 'trick' is useful, but AFAIK it doesn't have a name.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-02-26T22:07:32.897" CommentCount="1" />
  <row Id="7668" PostTypeId="1" AcceptedAnswerId="7669" CreationDate="2011-02-26T22:43:50.250" Score="3" ViewCount="76" Body="&lt;p&gt;Suppose I have data with p explanatory variables, and I want to use a LARS algorithm to build a model. Do I&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Run LARS until all p variables have been added to my model, and the correlation between each variable and the residual is 0?&lt;/li&gt;&#xA;&lt;li&gt;Or do I terminate LARS earlier, say a) by stopping LARS once k &amp;lt; p variables have been added to my model, or b) by stopping when the correlation between each variable and the residual crosses below some threshold t? (Where k or t could be determined by cross-validation.)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In reading the original paper by Efron et al. again, it sounds like I do version 1 (build the full model). But then I'm confused -- how does this solution differ from the full maximum-likelihood OLS estimate? (The OLS estimate also produces zero correlation between each variable and the residual, and I thought there was only a single way to get this zero correlation, i.e., by orthogonal projection. Or am I mistaken?)&lt;/p&gt;&#xA;" OwnerUserId="1106" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-02-26T23:20:04.150" LastActivityDate="2011-02-27T05:44:07.347" Title="Stopping condition for least-angle regression" Tags="&lt;regression&gt;&lt;regularization&gt;" AnswerCount="1" CommentCount="3" FavoriteCount="1" />
  <row Id="7669" PostTypeId="2" ParentId="7668" CreationDate="2011-02-26T23:25:15.310" Score="5" ViewCount="" Body="&lt;p&gt;Certainly, if $p \leq n$ and you run LARS until you've included all $p$ variables in the model and the correlations are zero, then the solution will be exactly the OLS solution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can view LARS as just another &quot;regularized&quot; least-squares estimate. Of course, it has a very close connection to both forward-stagewise regression and the lasso. My suspicion is that most people use the LARS algorithm primarily to compute the lasso solutions and that LARS itself has gotten fairly little use as an estimation method in its own right.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cross-validation to choose $k$ and $t$ should be feasible if your interest is on prediction. Since the maximal correlation decreases monotonically, then $k$ is completely determined by $t$ in LARS (see, e.g., the right pane of Fig. 3 of the paper). Hence, you only need optimize over the threshold $t$ in the cross-validation. This is cleaner in a couple of ways: (1) There is only one parameter to optimize over and (2) any sensible objective function you choose should be pretty much continuous in $t$, whereas $k$ is discrete. Discrete-valued parameters can often be less pleasant to deal with in CV.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For more details regarding some of the properties of least angle regression, you can also see the solution to &lt;strong&gt;&lt;a href=&quot;http://stats.stackexchange.com/questions/6795/least-angle-regression-keeps-the-correlations-monotonically-decreasing-and-tied&quot;&gt;this question&lt;/a&gt;&lt;/strong&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2970" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-02-27T05:44:07.347" LastActivityDate="2011-02-27T05:44:07.347" />
  <row Id="7670" PostTypeId="1" CreationDate="2011-02-27T00:40:09.317" Score="7" ViewCount="95" Body="&lt;p&gt;I am designing a Hybrid Monte Carlo sampling algorithm for &lt;a href=&quot;http://code.google.com/p/pymc/&quot; rel=&quot;nofollow&quot;&gt;PyMC&lt;/a&gt;, and I am trying to make it as fuss free and general as possible, so I am looking for good advice on designing an HMC algorithm. I have read &lt;a href=&quot;http://www.cs.utoronto.ca/~radford/ham-mcmc.abstract.html&quot; rel=&quot;nofollow&quot;&gt;Radford's survey chapter&lt;/a&gt; and &lt;a href=&quot;http://arxiv.org/abs/1001.4460&quot; rel=&quot;nofollow&quot;&gt;Beskos et. al.'s recent paper&lt;/a&gt; on optimal (step size) tuning of HMC and I gathered the following tips:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Momentum variables should be distributed with covariance $C^{-1}$, where $C$ is generally something like the covariance matrix of the distribution (for simple distributions), but could conceivably be different (for funny shaped distributions). By default I am using the hessian at the mode.&lt;/li&gt;&#xA;&lt;li&gt;Trajectories should be calculated with the leapfrog method (other integrators don't seem to be worth it)&lt;/li&gt;&#xA;&lt;li&gt;Optimal acceptance rate is .651 for really large problems and otherwise higher.&lt;/li&gt;&#xA;&lt;li&gt;Step size should be scaled like $L\times d^{(1/4)}$, where $L$ is a free variable and $d$ is the number of dimensions.&lt;/li&gt;&#xA;&lt;li&gt;Step size should be smaller when there are light tails or otherwise regions with odd stability characteristics. Step size randomization can help with this.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Are there other ideas that I should adopt or at least consider? Other papers I should read? For example, are there adaptive step size algorithms that are worth it? Is there good advice on trajectory length? Are there in fact better integrators?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Someone please make this a community-wiki.&lt;/p&gt;&#xA;" OwnerUserId="1146" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-02-27T11:56:09.117" LastActivityDate="2011-03-29T12:38:35.117" CommunityOwnedDate="2011-02-27T16:45:14.290" Title="What should I know about designing a good Hybrid/Hamiltonian Monte Carlo algorithm?" Tags="&lt;bayesian&gt;&lt;monte-carlo&gt;&lt;hmc&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="7671" PostTypeId="2" ParentId="7630" CreationDate="2011-02-27T03:32:38.977" Score="4" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;I think it depends on how it is to be used.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just for reference for other readers, if $P$ and $Q$ are probability measures, then the Jensen-Shannon Divergence is&#xA;$$&#xD;&#xA;J(P,Q) = \frac{1}{2} \big( D(P \mid\mid R) + D(Q\mid\mid R) \big)&#xD;&#xA;$$&#xA;where $R = \frac{1}{2} (P + Q)$ is the mid-point measure and $D(\cdot\mid\mid\cdot)$ is the Kullback-Leibler divergence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, I would be tempted to use the &lt;strong&gt;&lt;em&gt;square root&lt;/em&gt;&lt;/strong&gt; of the Jensen-Shannon Divergence since it is a &lt;strong&gt;metric&lt;/strong&gt;, i.e. it satisfies all the &quot;intuitive&quot; properties of a distance measure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For more details on this, see&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Endres and Schindelin, &lt;a href=&quot;http://research-repository.st-andrews.ac.uk/bitstream/10023/1591/1/Endres2003-IEEETransInfTheory49-NewMetric.pdf&quot; rel=&quot;nofollow&quot;&gt;A new metric for probability distributions&lt;/a&gt;, &lt;em&gt;IEEE Trans. on Info. Thy.&lt;/em&gt;, vol. 49, no. 3, Jul. 2003, pp. 1858-1860.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Of course, in some sense, it depends on what you need it for. If all you are using it for is to evaluate some pairwise measure, then any monotonic transformation of JSD would work. If you're looking for something that's closest to a &quot;squared-distance&quot;, then the JSD itself is the analogous quantity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Incidentally, you might also be interested in &lt;a href=&quot;http://stats.stackexchange.com/questions/6907/an-adaptation-of-the-kullback-leibler-distance&quot;&gt;this previous question&lt;/a&gt; and the associated answers and discussions.&lt;/p&gt;&#xA;" OwnerUserId="2970" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-02-27T05:49:03.880" LastActivityDate="2011-02-27T05:49:03.880" CommentCount="2" />
  <row Id="7672" PostTypeId="2" ParentId="7648" CreationDate="2011-02-27T10:13:09.253" Score="4" ViewCount="" Body="&lt;p&gt;The &lt;code&gt;NaiveBayes()&lt;/code&gt; function in the &lt;a href=&quot;http://cran.r-project.org/web/packages/klaR/index.html&quot; rel=&quot;nofollow&quot;&gt;klaR&lt;/a&gt; package obeys the classical &lt;code&gt;formula&lt;/code&gt; R interface whereby you express your outcome as a function of its predictors, e.g. &lt;code&gt;spam ~ x1+x2+x3&lt;/code&gt;. If your data are stored in a &lt;code&gt;data.frame&lt;/code&gt;, you can input all predictors in the rhs of the formula using dot notation: &lt;code&gt;spam ~ ., data=df&lt;/code&gt; means &quot;&lt;code&gt;spam&lt;/code&gt; as a function of all other variables present in the &lt;code&gt;data.frame&lt;/code&gt; called &lt;code&gt;df&lt;/code&gt;.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a toy example, using the &lt;code&gt;spam&lt;/code&gt; dataset discussed in the &lt;a href=&quot;http://www-stat.stanford.edu/~tibs/ElemStatLearn/&quot; rel=&quot;nofollow&quot;&gt;Elements of Statistical Learning&lt;/a&gt; (Hastie et al., Springer 2009, 2nd ed.), available on-line. This really is to get you started with the use of the R function, not the methodological aspects for using NB classifier.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data(spam, package=&quot;ElemStatLearn&quot;)&#xA;library(klaR)&#xA;&#xA;# set up a training sample&#xA;train.ind &amp;lt;- sample(1:nrow(spam), ceiling(nrow(spam)*2/3), replace=FALSE)&#xA;&#xA;# apply NB classifier&#xA;nb.res &amp;lt;- NaiveBayes(spam ~ ., data=spam[train.ind,])&#xA;&#xA;# show the results&#xA;opar &amp;lt;- par(mfrow=c(2,4))&#xA;plot(nb.res)&#xA;par(opar)&#xA;&#xA;# predict on holdout units&#xA;nb.pred &amp;lt;- predict(nb.res, spam[-train.ind,])&#xA;&#xA;# raw accuracy&#xA;confusion.mat &amp;lt;- table(nb.pred$class, spam[-train.ind,&quot;spam&quot;])&#xA;sum(diag(confusion.mat))/sum(confusion.mat)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;A recommended add-on package for such ML task is the &lt;a href=&quot;http://caret.r-forge.r-project.org/Classification_and_Regression_Training.html&quot; rel=&quot;nofollow&quot;&gt;caret&lt;/a&gt; package. It offers a lot of useful tools for preprocessing data, handling training/test samples, running different classifiers on the same data, and summarizing the results. It is available from CRAN and has a lot of vignettes that describe common tasks. &lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-02-28T21:16:27.870" LastActivityDate="2011-02-28T21:16:27.870" CommentCount="1" />
  <row Id="7673" PostTypeId="2" ParentId="7670" CreationDate="2011-02-27T10:33:26.677" Score="1" ViewCount="" Body="&lt;p&gt;This paper is very interesting (although I haven't yet fully got my head around it),&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Girolami M. Calderhead B. (2011) Riemann manifold Langevin and Hamiltonian Monte Carlo methods. J. R. Statist. Soc. B. (with discussion). 73, Part 2. pp 1-37. &lt;/p&gt;&#xA;" OwnerUserId="887" LastEditorUserId="887" LastEditorDisplayName="" LastEditDate="2011-02-27T10:57:13.037" LastActivityDate="2011-02-27T10:57:13.037" />
  <row Id="7674" PostTypeId="2" ParentId="7419" CreationDate="2011-02-27T11:51:49.200" Score="6" ViewCount="" Body="&lt;p&gt;I never used it directly, so I can only share some papers I had and general thoughts about that technique (which mainly address your questions 1 and 3).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My general understanding of biclustering mainly comes from genetic studies (2-6) where we seek to account for clusters of genes and grouping of individuals: in short, we are looking to groups samples sharing similar profile of gene expression together (this might be related to disease state, for instance) &lt;em&gt;and&lt;/em&gt; genes that contribute to this pattern of gene profiling. A survey of the state of the art for biological &quot;massive&quot; datasets is available in Pardalos's slides, &lt;a href=&quot;http://www.ise.ufl.edu/cao/DMinAgriculture/Lecture6.biclustering.pdf&quot; rel=&quot;nofollow&quot;&gt;Biclustering&lt;/a&gt;. Note that there is an R package, &lt;a href=&quot;http://cran.r-project.org/web/packages/biclust/index.html&quot; rel=&quot;nofollow&quot;&gt;biclust&lt;/a&gt;, with applications to microarray data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In fact, my initial idea was to apply this methodology to clinical diagnosis, because it allows to put features or variables in more than one cluster, which is interesting from a semeiological perpective because symptoms that cluster together allow to define &lt;em&gt;syndrome&lt;/em&gt;, but some symptoms can overlap in different diseases. A good discussion may be found in Cramer et al., &lt;a href=&quot;http://sites.google.com/site/borsboomdenny/CramerEtAl2010.pdf?attredirects=0&quot; rel=&quot;nofollow&quot;&gt;Comorbidity: A network perspective&lt;/a&gt; (Behavioral and Brain Sciences 2010, 33, 137-193). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A somewhat related technique is &lt;em&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Collaborative_filtering&quot; rel=&quot;nofollow&quot;&gt;collaborative filtering&lt;/a&gt;&lt;/em&gt;. A good review was made available by Su and Khoshgoftaar (&lt;em&gt;Advances in Artificial Intelligence&lt;/em&gt;, 2009): &lt;a href=&quot;http://www.hindawi.com/journals/aai/2009/421425.html&quot; rel=&quot;nofollow&quot;&gt;A Survey of Collaborative Filtering Techniques&lt;/a&gt;. Other references are listed at the end. Maybe analysis of &lt;a href=&quot;http://www.albionresearch.com/data_mining/market_basket.php&quot; rel=&quot;nofollow&quot;&gt;frequent itemset&lt;/a&gt;, as exemplified in the &lt;a href=&quot;http://www.albionresearch.com/data_mining/market_basket.php&quot; rel=&quot;nofollow&quot;&gt;market-basket problem&lt;/a&gt;, is also linked to it, but I never investigated this. Another example of co-clustering is when we want to simultaneously cluster words and documents, as in text mining, e.g. Dhillon (2001). &lt;a href=&quot;http://www.cs.utexas.edu/users/inderjit/public_papers/kdd_bipartite.pdf&quot; rel=&quot;nofollow&quot;&gt;Co-clustering documents and words using bipartite spectral graph partitioning&lt;/a&gt;. &lt;em&gt;Proc. KDD&lt;/em&gt;, pp. 269–274.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;About some general references, here is a not very exhaustive list that I hope you may find useful:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Jain, A.K. (2010). &lt;a href=&quot;http://biometrics.cse.msu.edu/Publications/Clustering/JainClustering_PRL10.pdf&quot; rel=&quot;nofollow&quot;&gt;Data clustering: 50 years beyond K-means&lt;/a&gt;. &lt;em&gt;Pattern Recognition Letters&lt;/em&gt;, &lt;em&gt;31&lt;/em&gt;, 651–666&lt;/li&gt;&#xA;&lt;li&gt;Carmona-Saez et al. (2006). &lt;a href=&quot;http://www.biomedcentral.com/1471-2105/7/78&quot; rel=&quot;nofollow&quot;&gt;Biclustering of gene expression data by non-smooth non-negative matrix factorization&lt;/a&gt;. &lt;em&gt;BMC Bioinformatics&lt;/em&gt;, &lt;em&gt;7&lt;/em&gt;, 78.&lt;/li&gt;&#xA;&lt;li&gt;Prelic et al. (2006). &lt;a href=&quot;http://bioinformatics.oxfordjournals.org/content/22/9/1122.abstract&quot; rel=&quot;nofollow&quot;&gt;A systematic comparison and evaluation of biclustering methods for gene expression data&lt;/a&gt;. &lt;em&gt;Bioinformatics&lt;/em&gt;, &lt;em&gt;22(9)&lt;/em&gt;, 1122-1129. &lt;a href=&quot;http://www.tik.ee.ethz.ch/sop/bimax&quot; rel=&quot;nofollow&quot;&gt;www.tik.ee.ethz.ch/sop/bimax&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;DiMaggio et al. (2008). &lt;a href=&quot;http://www.biomedcentral.com/1471-2105/9/458&quot; rel=&quot;nofollow&quot;&gt;Biclustering via optimal re-ordering of data matrices in systems biology: rigorous methods and comparative studies&lt;/a&gt;. &lt;em&gt;BMC Bioinformatics&lt;/em&gt;, &lt;em&gt;9&lt;/em&gt;, 458.&lt;/li&gt;&#xA;&lt;li&gt;Santamaria et al. (2008). &lt;a href=&quot;http://bioinformatics.oxfordjournals.org/content/24/9/1212.abstract&quot; rel=&quot;nofollow&quot;&gt;BicOverlapper: A tool for bicluster visualization&lt;/a&gt;. &lt;em&gt;Bioinformatics&lt;/em&gt;, &lt;em&gt;24(9)&lt;/em&gt;, 1212-1213.&lt;/li&gt;&#xA;&lt;li&gt;Madeira, S.C. and Oliveira, A.L. (2004) &lt;a href=&quot;http://www.inesc-id.pt/ficheiros/publicacoes/1281.pdf&quot; rel=&quot;nofollow&quot;&gt;Bicluster algorithms for biological data analysis: a survey&lt;/a&gt;. &lt;em&gt;IEEE Trans. Comput. Biol. Bioinform.&lt;/em&gt;, &lt;em&gt;1&lt;/em&gt;, 24–45.&lt;/li&gt;&#xA;&lt;li&gt;Badea, L. (2009). &lt;a href=&quot;http://ijcai.org/papers09/Papers/IJCAI09-232.pdf&quot; rel=&quot;nofollow&quot;&gt;Generalized Clustergrams for Overlapping Biclusters&lt;/a&gt;. IJCAI&lt;/li&gt;&#xA;&lt;li&gt;Symeonidis, P. (2006). &lt;a href=&quot;http://webmining.spd.louisville.edu/webkdd06/papers/paper-3-Nearest%20Bi-Cluster%20CF_SymeonidiswebKDD06.pdf&quot; rel=&quot;nofollow&quot;&gt;Nearest-Biclusters Collaborative Filtering&lt;/a&gt;. WEBKDD&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-02-27T11:51:49.200" CommentCount="2" />
  <row Id="7675" PostTypeId="1" AcceptedAnswerId="7677" CreationDate="2011-02-27T14:09:22.073" Score="5" ViewCount="166" Body="&lt;p&gt;An experiment I conducted recently used a 2 (between participants) x 3 (within participants) design. That is, participants were randomly allocated to one of two conditions, and then completed three similar tasks each (in a counterbalanced order). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In each of these tasks, participants made binary choices (2AFC) in a number of trials, each of which had a normatively correct answer. &lt;br /&gt;&#xA;In every trial, participants were presented a distractor, which was assumed to bias responses towards one of the alternatives. The tasks differed only in presence and magnitude of this distractor (i.e. no distractor vs. distractor of small and large magnitude).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to examine the error rates (deviations from the normatively correct answer) across these conditions. I hypothesize that the error rate will increase when a distractor is present, but will not increase further when the magnitude of the distractor is increased. Also, I expect this increase to differ between the between-subjects conditions. The latter interaction is the central interest.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From &lt;a href=&quot;http://stats.stackexchange.com/questions/3874/unbalanced-mixed-effect-anova-for-repeated-measures&quot;&gt;discussions here&lt;/a&gt; and from the literature (Dixon, 2008; Jaeger, 2008), I gather that logit mixed-models are the appropriate analysis method, and that, in R, the &lt;a href=&quot;http://lme4.r-forge.r-project.org/&quot; rel=&quot;nofollow&quot;&gt;lme4 package&lt;/a&gt; is the tool of choice. &lt;br /&gt;&#xA;While I could compute some basic analyses (e.g. random intercept model, random effects ANCOVA) with lme4, I am stuck as to how to apply the models to the design in question -- I have the feeling that I am very much thinking in terms of HLMs, and have not yet quite understood the entirety of mixed effects models. Therefore, I would be very grateful for your help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have two basic questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;In a first analysis, I would like to consider the error rates in only those trials in which participants were biased towards the &lt;em&gt;wrong&lt;/em&gt; answer. The first model would therefore look only at trials in which the bias would point away from the correct answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If my observations were independent, i would probably use a model like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;correct ~ condition + distractor + condition:distractor&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;... but obviously, they aren't: Observations are grouped within a task (with a constant presence of a distractor) and within participants. My question, then, is this: How do I add the random effects to reflect this?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;(If I haven't lost you already :-) ) Would it be possible to include all trials (those where the bias would be into the direction of the wrong and of the correct answer), and include this difference (i.e. direction of the bias) as a trial-level predictor?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my imagination of HLMs, such a predictor (at the level of the trial) would depend on on the magnitude of the distractor present (at the level of the block), which again would depend on the condition of the participant (plus, possibly, a unique factor for each participant).&lt;br /&gt;The interactions would then emerge ›automatically‹ as cross-level interactions. How would such a model be specified in the ›flat‹ lme4 syntax? (Would such a model make sense at all?)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Ok, I hope all of this makes sense -- I will gladly elaborate otherwise. Again, I would be most grateful for any ideas and comments regarding this analysis, and would like to thank you for taking the time and trouble to respond.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;References&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Dixon, P. (2008). Models of accuracy in repeated-measures designs. &lt;em&gt;Journal of Memory and Language, 59&lt;/em&gt;(4), 447-456. doi: 10.1016/j.jml.2007.11.004&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Jaeger T. F. (2008). Categorical data analysis: Away from ANOVAs (transformation or not) and towards logit mixed models. &lt;em&gt;Journal of Memory and Language, 59&lt;/em&gt;(4), 434-446. doi: 10.1016/j.jml.2007.11.007&lt;/p&gt;&#xA;" OwnerUserId="3451" LastActivityDate="2011-04-18T10:22:26.577" Title="Analyzing a 2x3 repeated measures design using a logit mixed model" Tags="&lt;r&gt;&lt;logistic-regression&gt;&lt;mixed-model&gt;&lt;repeated-measures&gt;" AnswerCount="2" />
  <row Id="7676" PostTypeId="1" CreationDate="2011-02-27T15:01:27.410" Score="5" ViewCount="111" Body="&lt;p&gt;Hi folks, I have data from 40 people, each measured once in 2 conditions. For reasons tangential to my question I'm not comfortable using parametric analyses to compare the two conditions, so I'm employing bootstrapping. However, when I run a bootstrap analysis (resampling people with replacement, computing condition means, then repeating 1000 times), the condition means I obtain appear to be correlated across iterations. I noticed this because when I plot the 95% confidence interval in each condition the intervals capture each others' means, but when I compute a difference score between the conditions means within each iteration, the 95% confidence interval of this difference score excludes zero. Going back to the results of the bootstrapping, I find that the correlation of the conditions across the bootstrap iterations is .7! Should this affect my confidence in the interval generated for the difference score? Alternatively, does this tell me something (possibly about individual differences) about my data otherwise?&lt;/p&gt;&#xA;" OwnerUserId="364" LastEditorUserId="364" LastEditorDisplayName="" LastEditDate="2011-02-28T18:21:04.023" LastActivityDate="2011-05-10T05:44:57.843" Title="Should I be concerned if the cells of values obtained from bootstrapping are correlated?" Tags="&lt;confidence-interval&gt;&lt;bootstrap&gt;" AnswerCount="1" CommentCount="3" FavoriteCount="1" />
  <row Id="7677" PostTypeId="2" ParentId="7675" CreationDate="2011-02-27T15:09:35.677" Score="2" ViewCount="" Body="&lt;p&gt;I would definitely use all your data and add direction of intended bias induction as a variable. Since you already have variables in the model describing the difference between the tasks, I don't believe that adding task as a random effect is necessary. The model would be:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;my_model = lmer(&#xA;    correct ~ (1|subject) + condition*distractor*direction&#xA;    , family = 'binomial'&#xA;    , data = my_data&#xA;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Check out the &lt;code&gt;ezMixed&lt;/code&gt; function from the &lt;a href=&quot;http://cran.r-project.org/web/packages/ez/index.html&quot; rel=&quot;nofollow&quot;&gt;ez package&lt;/a&gt; for an automated way of evaluating evidence for each effect in the model.&lt;/p&gt;&#xA;" OwnerUserId="364" LastActivityDate="2011-02-27T15:09:35.677" CommentCount="1" />
  <row Id="7678" PostTypeId="2" ParentId="1239" CreationDate="2011-02-27T15:30:18.387" Score="3" ViewCount="" Body="&lt;p&gt;There are 3 ways that I know of to access R from Ruby:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;RinRuby (https://sites.google.com/a/ddahl.org/rinruby-users/)&lt;/li&gt;&#xA;&lt;li&gt;RSRuby (https://github.com/alexgutteridge/rsruby)&lt;/li&gt;&#xA;&lt;li&gt;Rserve through the Rserve-Ruby-Client (https://github.com/clbustos/Rserve-Ruby-client)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;RinRuby is the slowest, RSRuby is the fastest and Rserve is closer to RSRuby in performance. RSRuby however is quite platform specific and you need to compile R with the sharelib option. Rserve-Ruby-Client is easier in that aspect since Rserve provides a TCP socket server for you to send commands to R interpreter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;AFAIK all 3 are unfortunately not as well documented for beginners. Rserve-Ruby-Client is pretty active, the other 2 are less so.&lt;/p&gt;&#xA;" OwnerUserId="3442" LastEditorUserId="3442" LastEditorDisplayName="" LastEditDate="2011-02-28T06:34:16.960" LastActivityDate="2011-02-28T06:34:16.960" />
  <row Id="7679" PostTypeId="1" AcceptedAnswerId="7682" CreationDate="2011-02-27T16:16:32.887" Score="5" ViewCount="190" Body="&lt;p&gt;I've scoured Google and this site and I am still confused about the lmer function in the lme4 library.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have some data collected from different psychiatric wards, which have a multilevel structure. To simplify, I'll pick two level 2 and two level 1 variables, although I actually have a few more.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Level 2- WardSize [this is the number of people on the ward] &amp;amp; WAS [this is a measure of how &quot;nice&quot; the ward is]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The grouping variable that tells R who's in which ward is called &quot;Ward&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Level one- Gender [this is gender, obviously] &amp;amp; BSITotal [this is a measure of symptom severity]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Outcome is Selfreject, which again is what it sounds like.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have this formula:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;help=lmer(formula=Selfreject~WardSize+WAS+Gender+BSITotal+(1|Ward))&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm hoping this means &quot;each individual has a score related to their own Gender and symptom severity, and also a ward-level effect relating to the size of the ward and how &quot;nice&quot; it is&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this correct? The thing that's confusing me is that I can't see how R can tell which are level 1 and which level 2 variables, except for the ward level intercept given at the end.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If anyone could explain the notation so an idiot like me can understand that would be even better.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks!&lt;/p&gt;&#xA;" OwnerUserId="199" LastActivityDate="2011-02-27T19:38:14.600" Title="Am I specifying my lmer model correctly?" Tags="&lt;r&gt;&lt;mixed-model&gt;&lt;lmer&gt;" AnswerCount="1" />
  <row Id="7680" PostTypeId="2" ParentId="2213" CreationDate="2011-02-27T16:29:58.647" Score="3" ViewCount="" Body="&lt;p&gt;What gd047 writes is correct, however the use of RNNs in practice today is restricted to a simpler class of problems: time series/sequential tasks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While feedforward networks are used to learn datasets like $(i, t)$ where $i$ and $t$ are vectors (eg $i \in \mathcal{R}^n$, for recurrent networks $i$ will always be a sequence, e.g. $i \in (\mathcal{R}^n)^*$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;RNNs have been shown to be able to represent any measureable sequence to sequence mapping by Hammer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, RNNs are being used nowadays for  all kinds of sequential tasks: time series prediction, sequence labeling, sequence classification etc. A good overview can be found on &lt;a href=&quot;http://www.idsia.ch/~juergen/rnn.html&quot; rel=&quot;nofollow&quot;&gt;Schmidhuber's page on RNNs&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2860" LastActivityDate="2011-02-27T16:29:58.647" />
  <row Id="7681" PostTypeId="1" CreationDate="2011-02-27T19:05:14.440" Score="5" ViewCount="136" Body="&lt;p&gt;Say we have the following 5 cities, each with the same population &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;CityA with 20% each of 5 ethnicities&lt;/li&gt;&#xA;&lt;li&gt;CityB with 99% of one ethnicity, but 100 different ethnicities in the remaining 1%&lt;/li&gt;&#xA;&lt;li&gt;CityC with 40% of one ethnicity and the remaining 60% distributed evenly over 10 different ethnicities&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;How can one measure their relative diversities?&lt;/p&gt;&#xA;" OwnerUserId="276" LastEditorUserId="276" LastEditorDisplayName="" LastEditDate="2011-02-28T05:49:46.323" LastActivityDate="2011-02-28T14:03:20.077" Title="Is there a way to compute diversity in a population?" Tags="&lt;distributions&gt;&lt;population&gt;" AnswerCount="4" CommentCount="3" FavoriteCount="1" />
  <row Id="7682" PostTypeId="2" ParentId="7679" CreationDate="2011-02-27T19:38:14.600" Score="6" ViewCount="" Body="&lt;p&gt;Your model specification is fine. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The varying intercept for Ward, specified in lmer as you've done with (1 | Ward), is saying that subjects within each ward might be more similar to each other on Selfreject for reasons other than WardSize or Gender, so  you are controlling for between-ward heterogeneity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can think of the &quot;1&quot; as a column of 1s (i.e., a constant) in the data to which an intercept is fit. Usually the &quot;1&quot; is implied automatically in lm, for instance&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;lm(Y ~ X1 + X2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;actually specifies&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;lm(Y ~ 1 + X1 + X2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now that you have your basic model, you can start asking further questions like &quot;Does the relationship between BSItotal and Selfreject differ &lt;strong&gt;between&lt;/strong&gt; wards?&quot;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;lmer(formula=Selfreject ~ WardSize + WAS + Gender + BSITotal + (1 + BSITotal | Ward))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;That is, both the intercept and the slope of BSITotal can differ between wards.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you haven't picked it up yet, Gelman &amp;amp; Hill's &lt;a href=&quot;http://www.stat.columbia.edu/~gelman/arm/&quot; rel=&quot;nofollow&quot;&gt;Data Analysis Using Regression and Multilevel Model/Hierarchical Models&lt;/a&gt; is a great book that explains fitting models like this with lmer.&lt;/p&gt;&#xA;" OwnerUserId="1916" LastActivityDate="2011-02-27T19:38:14.600" />
  <row Id="7683" PostTypeId="1" CreationDate="2011-02-27T21:02:23.573" Score="7" ViewCount="163" Body="&lt;p&gt;Hi all,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm struggling to find a method for reducing the number of categories in nominal or ordinal data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, let's say that I want to build a regression model on a dataset that has a number of nominal and ordinal factors. While I have no problems with this step, I often run into situations where a nominal feature is without observations in the training set, but subsequently exists in the validation dataset. This naturally leads to and error when the model is presented with (so far) unseen cases. &#xA;Another situation where I would like to combine categories is simply when there are too many categories with few observations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my questions are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;While I realize it might be best to combine many nominal (and ordinal) categories based on the prior real-world background information they represent, are there &lt;em&gt;systematic&lt;/em&gt; methods (&lt;code&gt;R&lt;/code&gt; packages preferably) available?&lt;/li&gt;&#xA;&lt;li&gt;What guidelines and suggestions would you make regarding, cut-off thresholds and so on?&lt;/li&gt;&#xA;&lt;li&gt;What are the most popular solutions in literature?&lt;/li&gt;&#xA;&lt;li&gt;Are there other strategies than combining small nominal categories to a new, &quot;OTHERS&quot; category?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Please feel free to chime in if you have other suggestions also.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks, Figaro&lt;/p&gt;&#xA;" OwnerUserId="3401" LastActivityDate="2011-03-01T03:12:29.883" Title="Methods for merging/reducing categories in ordinal or nominal data?" Tags="&lt;r&gt;&lt;categorical-data&gt;&lt;dimensionality-reduction&gt;" AnswerCount="2" />
  <row Id="7684" PostTypeId="2" ParentId="213" CreationDate="2011-02-27T21:22:37.633" Score="2" ViewCount="" Body="&lt;p&gt;I novel approach I saw was by &lt;em&gt;IT Jolliffe Principle Components Analysis&lt;/em&gt;.  You run a PCA on your data (Note: PCA can be quite a useful data exploration tool in its own right), but instead of looking at the first few Principal Components (PCs), you plot the last few PCs.  These PCs are the linear relationships between your variables with the smallest variance possible.  Thus they detect &quot;exact&quot; or close to exact multivariate relationships in your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A plot of the PC scores for the last PC will show outliers not easily detectable by looking individually at each variable.  One example is for height and weight - some who has &quot;above average&quot; height and &quot;below average&quot; weight would be detected by the last PC of height and weight (assuming these are positively correlated), even if their height and weight were not &quot;extreme&quot; individually (e.g. someone who was 180cm and 60kg).&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-02-27T21:22:37.633" />
  <row Id="7685" PostTypeId="2" ParentId="7683" CreationDate="2011-02-27T21:52:51.347" Score="6" ViewCount="" Body="&lt;p&gt;This is a response to your second question. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suspect the &lt;em&gt;correct&lt;/em&gt; approach to these kinds of decisions will be determined largely by disciplinary norms and the expectations of the intended audience of your work. As a social scientist, I often work with survey (or survey-like) data and I always try to balance substantive and data-driven logics when I collapse ordinal scales or categorical variables. In other words, I'll do my best to consider what combinations of items &quot;hang together&quot; in terms of their substance as well as the distribution of responses before I collapse the items. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's a recent example of a specific (ordinal) survey question that involved a five-point frequency scale:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;How often do you attend the meetings of a club or organization in your community? &lt;/p&gt;&#xA;  &#xA;  &lt;ul&gt;&#xA;  &lt;li&gt;Never&lt;/li&gt;&#xA;  &lt;li&gt;A few times a year &lt;/li&gt;&#xA;  &lt;li&gt;Once a month&lt;/li&gt;&#xA;  &lt;li&gt;A few times a month&lt;/li&gt;&#xA;  &lt;li&gt;Once a week or more&lt;/li&gt;&#xA;  &lt;/ul&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I don't have the data available to me at the moment, but the results were strongly skewed towards the &quot;never&quot; end of the scale. As a result, my co-author and I chose to pool responses into two groups: &quot;Once a month or more&quot; and &quot;Less than once a month.&quot; The resulting (binary) variable was more evenly distributed &lt;em&gt;and&lt;/em&gt; reflected a meaningful distinction in practical terms: since many clubs and organizations don't meet more than once a month, there are good reasons to believe that people who attend  meetings at least that often are &quot;active&quot; members of such groups whereas those who attend less frequently (or never) are &quot;inactive.&quot; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So in my experience, these decisions are at least as much art as science. That said, I also usually try to do this before fitting any models, since I work in a discipline where anything else is viewed (negatively) as data mining and highly un-scientific (fun times!) .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With that in mind, it might help if you could say a little bit more about what sort of audience you have in mind for this work. It would also be in your best interests to review a few prominent methodology textbooks in your field as they can often clarify what passes for &quot;normal&quot; behavior among a given research community. &lt;/p&gt;&#xA;" OwnerUserId="3396" LastEditorUserId="3396" LastEditorDisplayName="" LastEditDate="2011-03-01T03:12:29.883" LastActivityDate="2011-03-01T03:12:29.883" />
  <row Id="7686" PostTypeId="1" AcceptedAnswerId="7696" CreationDate="2011-02-27T22:04:16.123" Score="1" ViewCount="138" Body="&lt;p&gt;Hi,&#xA;if range of gaussian pdf is not a probability then how come it is used in Bayes rule in the same way as pmf?&lt;/p&gt;&#xA;" OwnerUserId="3456" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-02-27T22:18:27.367" LastActivityDate="2011-02-28T16:23:11.257" Title="Bayes rule and gaussian PDF" Tags="&lt;bayesian&gt;&lt;normal-distribution&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="7687" PostTypeId="2" ParentId="7681" CreationDate="2011-02-28T00:42:13.393" Score="6" ViewCount="" Body="&lt;p&gt;How about the &lt;a href=&quot;http://en.wikipedia.org/wiki/Shannon_index&quot; rel=&quot;nofollow&quot;&gt;Shannon index&lt;/a&gt;?&lt;/p&gt;&#xA;" OwnerUserId="1050" LastActivityDate="2011-02-28T00:42:13.393" />
  <row Id="7688" PostTypeId="1" CreationDate="2011-02-28T01:24:26.660" Score="1" ViewCount="104" Body="&lt;p&gt;I searched on internet for books on statistics (particularly 4shared.com :P) and I found that mostly all the books do not cover multivariate statistics in detail. Are there any good books which cover these topics in detail with sufficient examples? Thanks.&lt;/p&gt;&#xA;" OwnerUserId="1102" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-02-28T10:39:48.440" LastActivityDate="2011-02-28T10:39:48.440" Title="Books with good coverage on joint distributions, multivariate statistics, etc?" Tags="&lt;books&gt;&lt;multivariate-analysis&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="7689" PostTypeId="2" ParentId="7681" CreationDate="2011-02-28T03:15:48.423" Score="2" ViewCount="" Body="&lt;p&gt;You may be interested in &lt;a href=&quot;http://arxiv.org/abs/1101.5305&quot; rel=&quot;nofollow&quot;&gt;this paper&lt;/a&gt;: &quot;A new axiomatic approach to diversity&quot; from Chris Dowden.&lt;/p&gt;&#xA;" OwnerUserId="3459" LastActivityDate="2011-02-28T03:15:48.423" CommentCount="3" />
  <row Id="7690" PostTypeId="2" ParentId="7686" CreationDate="2011-02-28T03:36:47.303" Score="1" ViewCount="" Body="&lt;p&gt;The posterior distribution derived using continuous distributions in Bayes Theorem can always be integrated (although maybe not be hand) to give a probability. If you want to convince yourself &quot;caveman style,&quot; run the desired probabilities through Bayes Theorem using a Gaussian CDF, then take the derivative to get the posterior PDF.  Sounds like a great homework problem to torture undergrads with.  Thanks for the idea.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're not masochistic, and are amenable to a &quot;handwaving argument,&quot; just think of $f(t) dt$ as an infinitesimal probability and plug it in to the theorem.&lt;/p&gt;&#xA;" OwnerUserId="1920" LastActivityDate="2011-02-28T03:36:47.303" CommentCount="2" />
  <row Id="7691" PostTypeId="2" ParentId="6155" CreationDate="2011-02-28T06:14:53.413" Score="2" ViewCount="" Body="&lt;p&gt;Aside from what has been said, for the vusualization task alone (and outside from R), you might be interested in checking &lt;a href=&quot;http://gephi.org&quot; rel=&quot;nofollow&quot;&gt;Gephi&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="892" LastActivityDate="2011-02-28T06:14:53.413" />
  <row Id="7692" PostTypeId="2" ParentId="7688" CreationDate="2011-02-28T07:39:54.400" Score="1" ViewCount="" Body="&lt;p&gt;Despite @whuber's sound comment--covering all advances in MV analysis for the last 30 years is also outside the scope of e.g. the famous &lt;a href=&quot;http://www.elsevier.com/locate/inca/BS_HS&quot; rel=&quot;nofollow&quot;&gt;Handbook of Statistics&lt;/a&gt; series--, I would like to recommend &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Izenman, &lt;a href=&quot;http://www.springer.com/statistics/statistical+theory+and+methods/book/978-0-387-78188-4&quot; rel=&quot;nofollow&quot;&gt;&lt;em&gt;Modern Multivariate&#xA;  Statistical Techniques&lt;/em&gt;&lt;/a&gt;, Springer&#xA;  2008.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Although it has pretty much the same coverage than the &lt;a href=&quot;http://www-stat.stanford.edu/~tibs/ElemStatLearn/&quot; rel=&quot;nofollow&quot;&gt;&lt;em&gt;Elements of Statistical Learning&lt;/em&gt;&lt;/a&gt;, from Hastie and coll., it has some different applications and covers extra topic, like Correspondence Analysis. There is a &lt;a href=&quot;http://www.jstatsoft.org/v29/b11/paper&quot; rel=&quot;nofollow&quot;&gt;short review&lt;/a&gt; by John Maindonald in the JSS.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-02-28T07:39:54.400" CommunityOwnedDate="2011-02-28T07:39:54.400" />
  <row Id="7693" PostTypeId="2" ParentId="7688" CreationDate="2011-02-28T07:57:23.627" Score="2" ViewCount="" Body="&lt;p&gt;Last year, I spent every lunchtime for a week going to the Waterstones University bookshop in London looking for a good book on multivariate statistics (sad I know!). I also endorse Izenman, Modern Multivariate Statistical Techniques, Springer 2008, as it really was the stand-out book. It starts every chapter with an easy to understand outline and gradually progresses through the very detailed theory with lots of real world examples, data sources and visualisations. It's actually quite a good read for the basic principles and you come back later when the going gets heavy.&lt;/p&gt;&#xA;" OwnerUserId="3461" LastActivityDate="2011-02-28T07:57:23.627" />
  <row Id="7694" PostTypeId="1" CreationDate="2011-02-28T08:38:16.633" Score="5" ViewCount="59" Body="&lt;p&gt;I am doing time series data analysis by state space methods. With my data stochastic local level model totally outperformed the deterministic one. But deterministic level and slope model gives better results than stochastic level and stochastic/deterministic slope. Is this something usual? &#xA;All methods in R require initial values, and i read somewhere that fitting ARIMA model first and taking values from there as initial values for state space analysis is one way; possible? or any other proposition?&#xA;I should confess here that i'm totally new to state space anlaysis.&lt;/p&gt;&#xA;" OwnerUserId="3462" LastEditorUserId="3462" LastEditorDisplayName="" LastEditDate="2011-02-28T08:54:40.270" LastActivityDate="2011-02-28T08:54:40.270" Title="How to check which model is better in state space time series analysis?" Tags="&lt;time-series&gt;&lt;state-space-models&gt;" CommentCount="2" />
  <row Id="7695" PostTypeId="1" CreationDate="2011-02-28T08:55:17.277" Score="2" ViewCount="90" Body="&lt;p&gt;The Pareto distribution can be used to give a pdf for the wealth of a person chosen randomly from a population.  (In fact, this was its origin.  See, for instance, &lt;a href=&quot;http://en.wikipedia.org/wiki/Pareto_principle&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Pareto_principle&lt;/a&gt; ).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to explore the reciprocal question: Given the total amount of wealth in a population, what is the pdf of the portion that a randomly chosen person has. I conjecture that this is simply a constant times the Pareto distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More interestingly: What is the shape of the distribution curve, if the richest person would be at the 0 point on the x axis, the next richest person to the right, and so on - we would see a monotonically decreasing curve. But what is its shape? What is its derivative?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's quite likely that I'm not phrasing that question properly. Let me ask a more basic question: What is the appropriate terminology to explore the question? Give a probability distribution applied many times over, what is the shape of the resultant allocation curve?&lt;/p&gt;&#xA;" OwnerUserId="3463" LastEditorUserId="3463" LastEditorDisplayName="" LastEditDate="2011-02-28T09:26:52.597" LastActivityDate="2011-02-28T09:26:52.597" Title="Understanding the Pareto distribution as applied to wealth" Tags="&lt;distributions&gt;&lt;modeling&gt;&lt;predictive-models&gt;&lt;application&gt;&lt;pareto-distribution&gt;" CommentCount="8" />
  <row Id="7696" PostTypeId="2" ParentId="7686" CreationDate="2011-02-28T09:26:48.403" Score="2" ViewCount="" Body="&lt;p&gt;@Ahmed - you are definitely correct in thinking that something is not quite right here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Conditioning on &quot;point values&quot; which have probability/measure 0 can be &quot;dangerous&quot; and can lead to what is called a &lt;a href=&quot;http://en.wikipedia.org/wiki/Borel%E2%80%93Kolmogorov_paradox&quot; rel=&quot;nofollow&quot;&gt;Borel and Kolmogorov Paradox&lt;/a&gt;.  The lesson from this is that in order to be certain that we are defining a conditional probability unambiguously, it is necessary to:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Specify a definite limiting process towards the null point.&lt;/li&gt;&#xA;&lt;li&gt;Ensure that the limit exists and is &quot;well behaved.&quot;&lt;/li&gt;&#xA;&lt;li&gt;Don't change limits halfway through your calculations!&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;This is a rather tedious and often unnecessary process, but if find yourself getting weird answers, this is the &quot;insurance policy&quot; so to speak.  You can pretty much never get absurdities if you follow the above rules.  But like all insurance, there is a price - the painful process of working out a limit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I like Edwin Jaynes in this case:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;there is no right choice or wrong choice because it is for us to say which limit we want to take; i.e., which problem we want to solve.&#xA;This is Sermon #1 on mathematical limits; although it was given long ago by Kolmogorov, many who try to do probability calculations still fail to heed it and get themselves into trouble. The moral is that, unless they are defined in the statement of a problem, probabilities conditional on point values of a parameter, have no meaning until the specific limiting process is stated. More generally, probabilities conditional on any propositions of probability zero, are undefined.&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="2392" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-02-28T16:23:11.257" LastActivityDate="2011-02-28T16:23:11.257" />
  <row Id="7697" PostTypeId="2" ParentId="672" CreationDate="2011-02-28T09:55:19.267" Score="2" ViewCount="" Body="&lt;p&gt;Bayes theorem in its most obvious form is simply a re-statement of two things:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;the joint probability is symmetric in its arguments $P(HD|I)=P(DH|I)$&lt;/li&gt;&#xA;&lt;li&gt;the product rule $P(HD|I)=P(H|I)P(D|HI)$&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;So by using the symmetry:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(HD|I)=P(H|I)P(D|HI)=P(D|I)P(H|DI)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now if $P(D|I) \neq 0$ you can divide both sides by $P(D|I)$ to get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(H|DI)=P(H|I)\frac{P(D|HI)}{P(D|I)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So this is it?  How can something so simple be so awesome?  As with most things &quot;its the journey that's more important than the destination&quot;.  Bayes theorem rocks because of the arguments that lead to it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is missing from this is that the product rule and sum rule $P(H|I)=1-P(\overline{H}|I)$, can be derived using deductive logic based on axioms of consistent reasoning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now the &quot;rule&quot; in deductive logic is that if you have a relationship &quot;A implies B&quot; then you also have &quot;Not B implies Not A&quot;.  So we have &quot;consistent reasoning implies Bayes theorem&quot;.  This means &quot;Not Bayes theorem implies Not consistent reasoning&quot;.  i.e. if your result isn't equivalent to a Bayesian result for some prior and likelihood then you are reasoning inconsistently.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This result is called Cox's theorem and was proved in &quot;Algebra of Probable inference&quot; in the 1940's.  A more recent derivation is given in Proability theory: The logic of science.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-02-28T09:55:19.267" />
  <row Id="7698" PostTypeId="1" CreationDate="2011-02-28T10:53:14.850" Score="4" ViewCount="134" Body="&lt;p&gt;I want to approximate a non-linear function with a limited value range by an artificial neural network (feed forward, back propagation).&#xA;Most tools and literature availabe suggest linear functions for the output neurons when doing regressions. However, I know a priori that my goal function is of limited range, therefore is it reasonable to use a function for the output neurons with limited value range, too? To be more explicit: My target function's values are in the range between  0 and 1, but the neural net does predict    occasionally values that exceed this range (e.g. -1.3). Can i prevent the net from doing so, and is it reasonable?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you very much,&#xA;Julian&lt;/p&gt;&#xA;" OwnerUserId="3465" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-03-01T05:37:39.587" LastActivityDate="2011-03-31T14:38:49.423" Title="Output layer of artificial neural networks when learning non-linear functions with limited value range" Tags="&lt;neural-networks&gt;" AnswerCount="5" />
  <row Id="7699" PostTypeId="1" AcceptedAnswerId="7700" CreationDate="2011-02-28T11:36:50.210" Score="1" ViewCount="72" Body="&lt;p&gt;I have just done a Chow test on a regression in order to see whether there is a structural break. I am a bit stumped however as my Chow test returns a negative number. Now what do I do?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More specifically, this expression (from the Wikipedia &lt;a href=&quot;http://en.wikipedia.org/wiki/Chow_test&quot; rel=&quot;nofollow&quot;&gt;entry&lt;/a&gt; for the Chow test):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\frac{(S_c-(S_1+S_2))/(k)}{(S_1+S_2)/(N_1+N_2-2k)}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;turns out negative for me.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: It turns out it was a simple programming error.&lt;/p&gt;&#xA;" OwnerUserId="3086" LastEditorUserId="3086" LastEditorDisplayName="" LastEditDate="2011-02-28T14:44:58.283" LastActivityDate="2011-02-28T14:44:58.283" Title="What to do with negative chow test?" Tags="&lt;change-point&gt;&lt;discontinuity&gt;&lt;chow-test&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="7700" PostTypeId="2" ParentId="7699" CreationDate="2011-02-28T12:09:40.833" Score="4" ViewCount="" Body="&lt;p&gt;You've made a mistake somewhere in your calculations. It's not possible for the sum of the squared residuals from a single regression using the combined data to be less than the sum of the sums of squared residuals from the regressions using the two separate sets of data.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-02-28T12:09:40.833" />
  <row Id="7701" PostTypeId="1" CreationDate="2011-02-28T12:10:44.647" Score="1" ViewCount="238" Body="&lt;p&gt;I have data from human participants in a study. There are more females in the study (60%) and males are older. I have a binary categorical variable $O$. If those who are $True$ for $O$ are older, do I need to correct for sex and/or age? Maybe those $True$ for $O$ contain more men. What concepts or methods should I use to determine this? I'm using R. &lt;/p&gt;&#xA;" OwnerUserId="2824" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-02-28T14:12:06.410" LastActivityDate="2011-02-28T14:33:46.443" Title="Adjusting for Confounding Variables" Tags="&lt;r&gt;&lt;regression&gt;&lt;categorical-data&gt;" AnswerCount="1" CommentCount="5" />
  <row Id="7702" PostTypeId="2" ParentId="7681" CreationDate="2011-02-28T12:32:27.213" Score="2" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://www.worldagroforestry.org/units/library/books/PDFs/Kindt%20b2005.pdf&quot; rel=&quot;nofollow&quot;&gt;Tree diversity analysis&lt;/a&gt; book will get you up to speed with common diversity indices, along with some useful packages in R and their usage. While the book talks about trees, it can be used with marine fauna (which I did for my thesis) or even people.&lt;/p&gt;&#xA;" OwnerUserId="144" LastActivityDate="2011-02-28T12:32:27.213" CommentCount="1" />
  <row Id="7703" PostTypeId="2" ParentId="7698" CreationDate="2011-02-28T13:01:10.830" Score="1" ViewCount="" Body="&lt;p&gt;If you use a logistic activation function in the output layer it will restrict the output to the range 0-1 as you require.  However if you have a regression problem with a restricted output range the sum-of-squares error metric may not be ideal and maybe a beta noise model might be more appropriate (c.f. beta regression, which IIRC is implemented in an R package but I have never used it myself)&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2011-02-28T13:01:10.830" CommentCount="5" />
  <row Id="7704" PostTypeId="2" ParentId="7681" CreationDate="2011-02-28T13:07:11.443" Score="2" ViewCount="" Body="&lt;p&gt;This paper by &lt;a href=&quot;http://dx.doi.org/10.2307/2579183&quot; rel=&quot;nofollow&quot;&gt;Massey and Denton 1988&lt;/a&gt; is a fairly prolific overview of commonly used indices in Sociology/Demography. It would also be useful for some other key terms used for searching articles. Frequently in Sociology the indices are labelled with names such as &quot;hetereogeinity&quot; and &quot;segregation&quot; as well as &quot;diversity&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Part of the reason no absolute right answer exists to your question is that people frequently only use epistemic logic to reason why one index is a preferred measurement. Infrequently are those arguments so strong that one should entirely discount other suggested measures. The work of Massey and Denton is useful to highlight what many of these indices theoretically measure and when they differ to a substantively noticeable extent (in large cities in the US).&lt;/p&gt;&#xA;" OwnerUserId="1036" LastEditorUserId="1036" LastEditorDisplayName="" LastEditDate="2011-02-28T14:03:20.077" LastActivityDate="2011-02-28T14:03:20.077" />
  <row Id="7705" PostTypeId="2" ParentId="7698" CreationDate="2011-02-28T13:45:26.013" Score="0" ViewCount="" Body="&lt;p&gt;If you know an absolute range for the output, but there is no reason to expect it to have the non-linear characteristic of the typical logistic activation function (i.e. a value in the middle is just as likely as a value near 0 or 1), then you can just transform the output by dividing by the absolute maximum.  If the minimum were not 0, you could subtract the absolute minimum before dividing by the value (maximum - minimum).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So basically don't try to train the neural network to the raw value, train it to the percentile value (0 for minimum, 1 for maximum).&lt;/p&gt;&#xA;" OwnerUserId="2917" LastActivityDate="2011-02-28T13:45:26.013" CommentCount="2" />
  <row Id="7706" PostTypeId="1" CreationDate="2011-02-28T14:03:40.473" Score="1" ViewCount="74" Body="&lt;p&gt;If my dataset comprises few censored variables (&amp;lt;1%) and I fit the OLS regression using a heteroscedastic resistant estimator (the residuals are not terribly heteroscedastic to begin with)- are the results valid? &lt;/p&gt;&#xA;" OwnerUserId="1291" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-02-28T14:41:16.510" LastActivityDate="2011-02-28T19:50:18.567" Title="What is the magnitude of bias in censored regression when OLS is applied?" Tags="&lt;survival&gt;&lt;least-squares&gt;&lt;censoring&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="7707" PostTypeId="2" ParentId="7701" CreationDate="2011-02-28T14:05:58.000" Score="1" ViewCount="" Body="&lt;p&gt;&lt;em&gt;Edited several times to reflect comments&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I realized I should give an example of what I meant by &quot;what your model looks like now.&quot; From what you've said, I'm assuming that $Variant$ or $O$ is your dependent or outcome variable and that you're starting with something like the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Variant = \beta_0 + \beta_1(Female) + \beta_2(Age) + \beta_3(Treatment) + \varepsilon$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where I've inserted &quot;Treatment&quot; you can also think of any other sort of indicator that may be relevant to you (and in fact, if you're really dealing with experimental data, I would recommend considering something other than regression!)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If that's the case (and assuming you're confident in your sampling methods, data collection procedures, decision to use regression, etc.), then the answers to your questions are in the results of your regression model -- post the results and maybe we can help you interpret them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If $Variant$ is on the other side of the equation (i.e. if it's a covariate or an independent variable), then you might want to think about &lt;a href=&quot;http://en.wikipedia.org/wiki/Interaction_%28statistics%29&quot; rel=&quot;nofollow&quot;&gt;interaction terms&lt;/a&gt;. They can help you test for associations in your data that are not simply additive. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;From your response to my comment above (and below), it sounds like the model might look more like this (I've now replaced all the occurrences of $O$ with $Variant$ just to distinguish $O$ from $\theta$):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Y = \beta_0 + \beta_1(Female) + \beta_2(Age) + \beta_3(Variant) + \varepsilon$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In that case, I would definitely consider introducing interaction terms between $Age$ and $Variant$ as well as between $Gender$ and $Variant$.&lt;/p&gt;&#xA;" OwnerUserId="3396" LastEditorUserId="3396" LastEditorDisplayName="" LastEditDate="2011-02-28T14:33:46.443" LastActivityDate="2011-02-28T14:33:46.443" CommentCount="5" />
  <row Id="7708" PostTypeId="2" ParentId="7698" CreationDate="2011-02-28T14:07:31.857" Score="0" ViewCount="" Body="&lt;p&gt;Would it work to use the linear function and simply cut all values below 0 to 0, and values above 1 to 1? &lt;/p&gt;&#xA;" OwnerUserId="3465" LastActivityDate="2011-02-28T14:07:31.857" />
  <row Id="7709" PostTypeId="2" ParentId="6809" CreationDate="2011-02-28T14:09:51.923" Score="2" ViewCount="" Body="&lt;p&gt;We've been working on this for a bit now (starting with Cubist then working on C5.0).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you'd like to contribute:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://r-forge.r-project.org/projects/rulebasedmodels/&quot; rel=&quot;nofollow&quot;&gt;https://r-forge.r-project.org/projects/rulebasedmodels/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;was created recently and we should be checking the initial code in.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We've had access to the Cubist sources for a while now (but there was an explicit agreement not to link it to other sw) and been debating the different options for incorporating the code, but I thin &lt;/p&gt;&#xA;" OwnerUserId="3468" LastActivityDate="2011-02-28T14:09:51.923" />
  <row Id="7710" PostTypeId="2" ParentId="7706" CreationDate="2011-02-28T14:38:32.627" Score="4" ViewCount="" Body="&lt;p&gt;Suppose you observe $(y_i,x_i)$, which come frome censored regression model:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align}&#xA;y^*_i&amp;amp;=x_i\beta+u_i \\&#xA;y_i&amp;amp;= \max(y_i^*,0)&#xA;\end{align}&#xA;with $u_i|x_i\sim N(0,\sigma^2)$, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then this model is equivalent to:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align}&#xA;y_i=x_i\beta+\sigma\lambda(x_i\beta/\sigma)+e_i,&#xA;\end{align}&#xA;where $E(e_i|x_i,y_i&amp;gt;0)=0$ and $\lambda$ is &lt;a href=&quot;http://en.wikipedia.org/wiki/Mills_ratio&quot; rel=&quot;nofollow&quot;&gt;inverse Mills ratio&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you apply OLS ignoring truncation then you omit the term with Mills ratio, so it does not matter that you account for heteroscedasticity, your coefficients will still be biased. If as you say you have only few censored observations, then the bias might be small. But it is better to come to this conclusion by comparing both models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've lifted most of the material from &lt;a href=&quot;http://books.google.com/books?id=cdBPOJUP4VsC&amp;amp;lpg=PP1&amp;amp;dq=wooldridge%20econometrics&amp;amp;hl=fr&amp;amp;pg=PP1#v=onepage&amp;amp;q=wooldridge%20econometrics&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;Wooldridge's book&lt;/a&gt;, page 524, so you can read more about it there. You should also check whether you observe truncation or censoring, since these are usually confused and different results apply.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-02-28T19:50:18.567" LastActivityDate="2011-02-28T19:50:18.567" />
  <row Id="7712" PostTypeId="1" AcceptedAnswerId="7713" CreationDate="2011-02-28T16:30:13.000" Score="1" ViewCount="68" Body="&lt;p&gt;I am not sure if this is an instance of vectorizing the operations in R, but this is where I am stuck:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to get: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dpois(1, 0.1)&#xA;dpois(2, 0.2)&#xA;dpois(3, 0.3)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and I tried:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dpois(1:3, 0.1:0.3)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;do.call(dpois, list(x = 1:3, lambda = 0.1:0.3))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;both do not work. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It there a R-ish way of doing this? &lt;/p&gt;&#xA;" OwnerUserId="1307" LastActivityDate="2011-02-28T17:16:06.103" Title="How do I &quot;vectorize&quot; calls to dpois?" Tags="&lt;r&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="7713" PostTypeId="2" ParentId="7712" CreationDate="2011-02-28T16:48:15.783" Score="6" ViewCount="" Body="&lt;p&gt;From &lt;code&gt;help(dpois)&lt;/code&gt; it looks like you need &lt;code&gt;x&lt;/code&gt; and &lt;code&gt;lambda&lt;/code&gt; to be vectors (read more about object classes in the R Intro or any other R documentation to understand what this means). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The following works:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;dpois(1:3, c(seq(0.1, 0.3, .1)))&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your first attempt fails because you are not concatenating (see: &lt;code&gt;help(c)&lt;/code&gt;) the values for &lt;code&gt;0.1:0.3&lt;/code&gt; into a vector and you are not providing any way for R to know what you want it to do with &lt;code&gt;0.1:0.3&lt;/code&gt;. Calling &lt;code&gt;seq()&lt;/code&gt; in the manner above tells it to get a sequence from 0.1 to 0.3 by 0.1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your second attempt is pretty far into the weeds. There's no way you need a power-tool like &lt;code&gt;do.call&lt;/code&gt; for this kind of thing.&lt;/p&gt;&#xA;" OwnerUserId="3396" LastEditorUserId="3396" LastEditorDisplayName="" LastEditDate="2011-02-28T17:16:06.103" LastActivityDate="2011-02-28T17:16:06.103" CommentCount="5" />
  <row Id="7714" PostTypeId="1" CreationDate="2011-02-28T17:00:48.593" Score="2" ViewCount="80" Body="&lt;p&gt;What inter-rate reliability test is best for continuous data? I am doing a study with one variable with continuous data, now the measurement involves measurements done by two people. I would wish to do inter-rater reliability test for the data, so far I have collected a few samples and  a sample data I have given below, what test would I use?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rater A 23.1  22.0  21.8  24.1  20.2  22.1  23.8&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rater B 23.0  21.5  22.0  23.9  19.8  21.9  22.9 &lt;/p&gt;&#xA;" OwnerUserId="3472" LastEditorUserId="3472" LastEditorDisplayName="" LastEditDate="2011-02-28T18:59:08.360" LastActivityDate="2011-02-28T19:08:43.810" Title="Interater reliability" Tags="&lt;reliability&gt;&lt;inter-rater&gt;" AnswerCount="1" CommentCount="4" />
  <row Id="7715" PostTypeId="1" CreationDate="2011-02-28T17:03:32.650" Score="0" ViewCount="138" Body="&lt;p&gt;I'm a bit stuck on this - any suggestions? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have the following:&#xA;R-squared for Y on X1,2,3 is 0.25,&#xA;R-squared for Y on X1,2 is 0.15, &#xA;variance of Y is 100,&#xA;and there are 50 cases, &#xA;what would the incremental explained sum of squares for X3 net of X12 equal?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas on how to get started? Much appreciated, thanks!&lt;/p&gt;&#xA;" OwnerUserId="3473" LastActivityDate="2011-02-28T17:03:32.650" Title="R-squared and explained sum of squares" Tags="&lt;regression&gt;" CommentCount="4" />
  <row Id="7716" PostTypeId="2" ParentId="2957" CreationDate="2011-02-28T18:06:45.623" Score="8" ViewCount="" Body="&lt;p&gt;Unbiased estimates are typical in introductory statistics courses because they are: 1) classic, 2) easy to analyze mathematically.  The Cramer-Rao lower bound is one of the main tools for 2).  Away from unbiased estimates there is possible improvement.  The bias-variance trade off is an important concept in statistics for understanding how biased estimates can be better than unbiased estimates.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unfortunately, biased estimators are typically harder to analyze.  In regression, much of the  research in the past 40 years has been about biased estimation.  This began with ridge regression (Hoerl and Kennard, 1970).  See &lt;a href=&quot;http://www.jstor.org/stable/1269656&quot; rel=&quot;nofollow&quot;&gt;Frank and Friedman (1996)&lt;/a&gt; and &lt;a href=&quot;http://www.jstor.org/stable/25471022&quot; rel=&quot;nofollow&quot;&gt;Burr and Fry (2005)&lt;/a&gt; for some review and insights.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The bias-variance tradeoff becomes more important in high-dimensions, where the number of variables is large.  Charles Stein surprised everyone when he proved that in the Normal means problem the sample mean is no longer admissible if $p \geq 3$ (see Stein, 1956).  The James-Stein estimator (James and Stein 1961) was the first example of an estimator that dominates the sample mean.  However, it is also  inadmissible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An important part of the bias-variance problem is determining how bias should be traded off.  &lt;strong&gt;There is no single “best” estimator&lt;/strong&gt;.  Sparsity has been an important part of research in the past decade.  See &lt;a href=&quot;http://arxiv.org/pdf/0802.0964&quot; rel=&quot;nofollow&quot;&gt;Hesterberg et al. (2008)&lt;/a&gt; for a partial review.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most of the estimators referenced above are non-linear in $Y$.  Even ridge regression is non-linear once the data is used to determine the ridge parameter.&lt;/p&gt;&#xA;" OwnerUserId="1670" LastEditorUserId="1670" LastEditorDisplayName="" LastEditDate="2011-03-01T01:07:08.683" LastActivityDate="2011-03-01T01:07:08.683" CommentCount="4" />
  <row Id="7717" PostTypeId="2" ParentId="7714" CreationDate="2011-02-28T19:08:43.810" Score="1" ViewCount="" Body="&lt;p&gt;I'd suggest you &lt;a href=&quot;http://en.wikipedia.org/wiki/Bland-Altman_plot&quot; rel=&quot;nofollow&quot;&gt;plot the difference against the mean &lt;/a&gt; then quantify things using the mean difference and the standard deviation of the difference. Seven samples is rather few though.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-02-28T19:08:43.810" CommentCount="4" />
  <row Id="7718" PostTypeId="1" AcceptedAnswerId="7725" CreationDate="2011-02-28T19:56:26.907" Score="4" ViewCount="78" Body="&lt;p&gt;Is there any standard method to determine an &quot;optimal&quot; operation point on a &lt;a href=&quot;http://en.wikipedia.org/wiki/Precision_and_recall&quot; rel=&quot;nofollow&quot;&gt;precision recall&lt;/a&gt; curve? (i.e., determining the point on the curve that offers a good trade-off between precision and recall)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="2798" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-02-28T21:22:53.433" LastActivityDate="2011-03-01T23:35:48.853" Title="How to choose a good operation point from precision recall curves?" Tags="&lt;machine-learning&gt;&lt;precision-recall&gt;" AnswerCount="2" FavoriteCount="0" />
  <row Id="7719" PostTypeId="1" CreationDate="2011-02-28T20:21:29.907" Score="1" ViewCount="106" Body="&lt;p&gt;The Data:&#xA;The observed probability (proportions) of three mutually exclusive events for five species.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the best way to plot these data in R along with their standard errors? I'd like to avoid a &quot;beside&quot; bar plot with error bars (3 bars for each species). I was hoping to use a stacked bar plot, but I'm unsure how to incorporate standard error!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="3474" LastActivityDate="2011-02-28T20:21:29.907" Title="Plotting Multiple Proportions With Standard Error" Tags="&lt;r&gt;&lt;data-visualization&gt;&lt;proportion&gt;" CommentCount="1" />
  <row Id="7720" PostTypeId="1" CreationDate="2011-02-28T20:51:28.700" Score="5" ViewCount="407" Body="&lt;p&gt;I am new to R, ordered logistic regression, and &lt;code&gt;polr&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &quot;Examples&quot; section at the bottom of the help page for &lt;a href=&quot;http://stat.ethz.ch/R-manual/R-patched/library/MASS/html/polr.html&quot; rel=&quot;nofollow&quot;&gt;polr&lt;/a&gt; (that fits a logistic or probit regression model to an ordered factor response) shows&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;options(contrasts = c(&quot;contr.treatment&quot;, &quot;contr.poly&quot;))&#xA;house.plr &amp;lt;- polr(Sat ~ Infl + Type + Cont, weights = Freq, data = housing)&#xA;pr &amp;lt;- profile(house.plr)&#xA;plot(pr)&#xA;pairs(pr)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;What information does &lt;code&gt;pr&lt;/code&gt; contain?  The help page on &lt;a href=&quot;http://stat.ethz.ch/R-manual/R-patched/library/stats/html/profile.html&quot; rel=&quot;nofollow&quot;&gt;profile&lt;/a&gt; is&#xA;generic, and gives no guidance for polr.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;What is &lt;code&gt;plot(pr)&lt;/code&gt; showing?  I see six graphs. Each has an X axis that is&#xA;numeric, although the label is an indicator variable (looks like an input variable that is an indicator for an ordinal value).  Then the Y axis&#xA;is &quot;tau&quot; which is completely unexplained.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;What is &lt;code&gt;pairs(pr)&lt;/code&gt; showing?  It looks like a plot for each pair of input&#xA;variables, but again I see no explanation of the X or Y axes.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;How can one understand if the model gave a good fit?&#xA;&lt;code&gt;summary(house.plr)&lt;/code&gt; shows Residual Deviance 3479.149 and AIC (Akaike&#xA;Information Criterion?) of 3495.149.  Is that good?  In the case those&#xA;are only useful as relative measures (i.e. to compare to another model&#xA;fit), what is a good absolute measure?  Is the residual deviance approximately chi-squared distributed?  Can one use &quot;% correctly predicted&quot; on the original data or some cross-validation?  What is the easiest way to do that?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;How does one apply and interpret &lt;code&gt;anova&lt;/code&gt; on this model?  The docs say &quot;There are methods for the standard model-fitting functions, including predict, summary, vcov, anova.&quot;  However, running &lt;code&gt;anova(house.plr)&lt;/code&gt; results in &lt;code&gt;anova is not implemented for a single &quot;polr&quot; object&lt;/code&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;How does one interpret the t values for each coefficient?  Unlike some&#xA;model fits, there are no P values here.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I realize this is a lot of questions, but it makes sense to me to ask as one bundle (&quot;how do I use this thing?&quot;) rather than 7 different questions.  Any information appreciated.&lt;/p&gt;&#xA;" OwnerUserId="2849" LastEditorUserId="2849" LastEditorDisplayName="" LastEditDate="2011-03-01T21:22:49.080" LastActivityDate="2011-03-17T18:20:48.850" Title="How to understand output from R's polr function (ordered logistic regression)?" Tags="&lt;r&gt;&lt;logistic-regression&gt;" AnswerCount="2" CommentCount="6" FavoriteCount="2" />
  <row Id="7721" PostTypeId="1" AcceptedAnswerId="7770" CreationDate="2011-02-28T21:24:23.323" Score="5" ViewCount="132" Body="&lt;p&gt;Greetings. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to do high dimensional biological data analysis, where my data can easily exist in hundreds of thousands of dimensions. I am looking for an implementation of multinomial logistic regression that will scale well to data of this size. Ideally, it should allow me to also do Ridge and Lasso regressions also. Which software should I be using? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Andy&lt;/p&gt;&#xA;" OwnerUserId="3301" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-02T22:48:44.927" LastActivityDate="2011-03-02T22:48:44.927" Title="Scalable multinomial regression implementation" Tags="&lt;regression&gt;&lt;multinomial&gt;" AnswerCount="1" CommentCount="11" FavoriteCount="1" />
  <row Id="7722" PostTypeId="1" CreationDate="2011-02-28T21:42:36.043" Score="1" ViewCount="64" Body="&lt;p&gt;Our experimental design is as follows.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We will have 4*N subjects.&#xA;Half with diagnosis 1, half with diagnosis 2. (Diagnosis = fixed factor 1).&#xA;Each diagnosis group is also equally and randomly divided between two different repeated measures arms of the study.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first two measurement sessions are identical for each arm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Session 1 We take baseline measure of 3 independent variables (V1, V2, V3).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Session 2 We perform a stress test, which gives independent variable V4, then we repeat measurement of V1, V2, and V3, from which we obtain their change with respect to baseline due to the stress test ΔV1, ΔV2, and ΔV3.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All subsequent measurement sessions are repeats of session 2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then the study arms diverge after session 2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Arm 1:&#xA;12 weeks of THERAPY then repeat all measures (session 3). End arm 1&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Arm 2:&#xA;12 weeks NO THERAPY then repeat all measures (session 3)&#xA;then &#xA;12 weeks of THERAPY, then repeat all measures (session 4) End arm 2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;End of study.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Both arms allow measurement of change in V4 with respect to session 2.&#xA;Session 3 allows an independent between groups comparison of the effect of therapy vs no therapy.&#xA;Session 4 (arm 2 only) can be seen as folding the no therapy group into the therapy group and &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The key concerns in this study are&#xA;1. Does therapy result in a change in any variable?&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;To what degree do the diagnosis factor, the baseline variables V1-V3, or their change in response the initial stress test ΔV1, ΔV2, and ΔV3, predict ΔV4, the change in outcome variable V4 (stress test performance) with respect to session 2?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Do the respective abilities of baseline V1, V2, V3, ΔV1, ΔV2, and ΔV3 to predict ΔV4 vary between diagnostic groups.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The fly in the ointment is the investigators desire to combine the between groups and within groups approaches to assess the effect of therapy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is, is there a way to set up a single mixed model analysis that addresses these questions and accounts for the unbalanced nature of the design and its hybrid blend of between groups and within groups treatment of the effect of therapy.  If so CAN IT BE DONE IN SPSS, and How?&lt;/p&gt;&#xA;" OwnerUserId="3477" LastActivityDate="2011-02-28T21:42:36.043" Title="How to deal with repeated measures design with two arms of different length (Can SPSS handle it)" Tags="&lt;mixed-model&gt;" />
  <row Id="7723" PostTypeId="1" CreationDate="2011-02-28T21:58:04.160" Score="0" ViewCount="270" Body="&lt;p&gt;We are learning pivot functions, test statistics, and hypothesis testing at university but it makes no sense. I've tried reading my text book/notes, going through examples, etc., but the concepts seem like a random guess and I'm clueless about how to even start guessing what the answer could be. &lt;/p&gt;&#xA;&#xA;&lt;h3&gt;1st part&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Can you please explain how to calculate a pivot function? E.g $X_{1},\dots,X_{n} \sim N(\mu, \sigma^2)$. Pivot function for $\sigma^2$ when $\mu$ is known and when $\mu$ is unknown. Why does $\mu$ being un/known matter?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also how would you calculate the pivot function for the ratio of two variances ($\sigma_{x}^2$ and $\sigma_{y}^2$)? Is it an F distribution? Assume $\mu_x$ and $\mu_y$ are known and $X_1,\dots,X_n \sim N(\mu_x,\sigma_x^2)$ and $Y_1,\dots,Y_n \sim N(\mu_y, \sigma_y^2)$.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;2nd part&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Can you please explain how to calculate a test statistic (I get how to show it's a test statistic but don't know how to form one from scratch). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Last, I have a few questions about hypothesis testing. I don't really understand how to calculate the power of a test or even what it means, to be honest. There is a whole bunch of theory and many definitions out there but they are rather abstract so I don't get it at all... I don't understand the notation or how to calculate the size/power of a test (generic form - not just with numbers). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example: $X \sim N(\mu, \sigma^2)$. $H_0: \mu &amp;gt;= \mu_0$ and $H_1: \mu &amp;lt; \mu_0$. $\mu$ and $\sigma$ unknown. Calculate the power and size of this test. How do I even start? I'm so confused. :(&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm really stuck with all of this and I hope you can help me! :) if there's a better resource out there to help please do let me know.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT1: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for your reply.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I did ask my lecturer to clarify… but ended up even more confused. He agrees the notes are unclear but will not rectify them because everyone else seems to get them! :( I also went to my class teacher and read through the Statistical Inference chapter several times – I get the basics but still don’t really understand most of it. I have googled around – and read Wikipedia – but it’s just more and more theory with no step by step examples explaining what to do. Everything seems randomly chosen and guesswork and hence my massive confusion. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One thing though – I can’t read this:&#xA;$$T_{X}=\sum_{i=1}^{N}\Big(\frac{X_{i}-\mu_{X}}{\sigma{X}}\Big)^{2} \sim \chi^{2}(N)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it supposed to be in mathematical notation? How do I view it properly? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Yes- you are correct – the pivot function is used to calculate the confidence interval. The thing is – once I have the pivot it’s quite straightforward to calculate the CI. But it’s the pivot that’s causing the problems. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I still don’t get the following:&#xA;Pivot function for sigma^2 when mu is known and when mu is unknown. Why does mu being un/known matter?&#xA;How would you calculate the pivot function for the ratio of 2 variances (sigmax^2 and sigmay^2)? Is it an F distribution? Assume mu x and mu y are known and X1....Xn - N(mu x,sigmax^2) and Y1...Yn -N(mu y, sigmay^2). Is it 1/Fn-1, m-1 = Fm-1,n-1? &#xA;And the hypothesis testing questions above please…. Can you shed some light on this please?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT2:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I did ask my lecturer to clarify… but ended up even more confused. He agrees the notes are unclear but will not rectify them because everyone else seems to get them! :( I also went to my class teacher and read through the Statistical Inference chapter several times – I get the basics but still don’t really understand most of it. I have googled around – and read Wikipedia – but it’s just more and more theory with no step by step examples explaining what to do. Everything seems randomly chosen and guesswork and hence my massive confusion. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One thing though – I can’t read this:&#xA;$$T_{X}=\sum_{i=1}^{N}\Big(\frac{X_{i}-\mu_{X}}{\sigma{X}}\Big)^{2} \sim \chi^{2}(N)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it supposed to be in mathematical notation? How do I view it properly? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Yes- you are correct – the pivot function is used to calculate the confidence interval. The thing is – once I have the pivot it’s quite straightforward to calculate the CI. But it’s the pivot that’s causing the problems. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I still don’t get the following:&#xA;Pivot function for sigma^2 when mu is known and when mu is unknown. Why does mu being un/known matter?&#xA;How would you calculate the pivot function for the ratio of 2 variances (sigmax^2 and sigmay^2)? Is it an F distribution? Assume mu x and mu y are known and X1....Xn - N(mu x,sigmax^2) and Y1...Yn -N(mu y, sigmay^2). Is it 1/Fn-1, m-1 = Fm-1,n-1? &#xA;And the hypothesis testing questions above please…. Can you shed some light on this please?&lt;/p&gt;&#xA;" OwnerUserId="3483" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-08T20:04:13.470" LastActivityDate="2011-03-08T20:04:13.470" Title="Pivotal quantities, test statistics and hypothesis tests" Tags="&lt;hypothesis-testing&gt;&lt;pivot&gt;" AnswerCount="1" CommentCount="8" FavoriteCount="0" />
  <row Id="7725" PostTypeId="2" ParentId="7718" CreationDate="2011-02-28T22:11:04.873" Score="6" ViewCount="" Body="&lt;p&gt;The definition of &quot;optimal&quot; will of course depend on your specific goals, but here are a few relatively &quot;standard&quot; methods:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Equal error rate (EER) point: the point where precision equals recall. This feels to some people like a &quot;natural&quot; operating point.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;A refined and more principled version of the above is to specify cost of the different kind of errors and optimize that cost. Say misclassifying an item (an error in precision) is twice as expensive as missing an item completely (error in recall). Then the best operating point is that where (1 - recall) = 2*(1 - precision).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;In some problems people have a natural minimal acceptable rate of either precision or recall. Say you know that if more than 20% of retrieved data is incorrect, the users will stop using your application. Then it is natural to set precision to 20% (or a bit less) and accept whatever recall you have at that point.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="3369" LastActivityDate="2011-02-28T22:11:04.873" />
  <row Id="7726" PostTypeId="2" ParentId="7554" CreationDate="2011-02-28T23:04:16.287" Score="0" ViewCount="" Body="&lt;p&gt;As @rolando2 mentioned, the histogram might be a tool for displaying the variations; and also, as @ashaw stated, you might need to find where 95% percentage of number go to, then, you can probably just use box plot to generate the basic features of your dataset, and put the data to the dashboard that is shown in the company without the box plot. For box plot, you might check here:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;http://en.wikipedia.org/wiki/Box_plot&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Hope this helps, at least for inspiration...&lt;/p&gt;&#xA;" OwnerUserId="3296" LastActivityDate="2011-02-28T23:04:16.287" />
  <row Id="7727" PostTypeId="1" CreationDate="2011-03-01T01:13:29.410" Score="1" ViewCount="245" Body="&lt;p&gt;I asked &lt;a href=&quot;http://stackoverflow.com/questions/5130808/how-to-correlate-two-time-series-with-gaps-and-different-time-bases&quot;&gt;&lt;strong&gt;this question&lt;/strong&gt;&lt;/a&gt; over on StackOverflow, and was recommended to ask it here.  Rather than duplicate the entire question here, I hope I can start new comment/solution chains here.  If I should repost the question, please let me know.&lt;/p&gt;&#xA;" OwnerUserId="3479" LastActivityDate="2011-03-07T04:29:43.617" Title="How to correlate two time series with gaps and different time bases?" Tags="&lt;time-series&gt;&lt;correlation&gt;" AnswerCount="1" CommentCount="9" FavoriteCount="0" />
  <row Id="7728" PostTypeId="2" ParentId="7683" CreationDate="2011-03-01T01:29:23.287" Score="2" ViewCount="" Body="&lt;p&gt;The kinds of approaches ashaw discusses can lead to a relatively more systematic methodology.  But I also think that by systematic you mean algorithmic.  Here data mining tools may fill a gap.  For one, there's the chi-squared automated interaction detection (CHAID) procedure built into SPSS's Decision Tree module; it can, according to rules set by the user, collapse ordinal or nominal categories of predictor variables when they show similar values on the outcome variable (whether it's continuous or nominal). These rules might depend on the size of the groups being collapsed or being created by collapsing, or on the &lt;em&gt;p&lt;/em&gt;-values of related statistical tests.  I believe some classification and regression tree (CART) programs can do the same things.  Other respondents should be able to speak about similar functions performed by neural network or other applications provided through various data mining packages.&lt;/p&gt;&#xA;" OwnerUserId="2669" LastActivityDate="2011-03-01T01:29:23.287" CommentCount="2" />
  <row Id="7730" PostTypeId="1" CreationDate="2011-03-01T03:09:39.890" Score="4" ViewCount="237" Body="&lt;p&gt;If I have a dependent variable and $N$ predictor variables and wanted my stats software to examine all the possible models, there would be $2^N$ possible resulting equations. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am curious to find out what the limitations are with regard to $N$ for major/popular statistic software since as $N$ gets large there is a combinatorial explosion.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've poked around the various web pages for packages but not been able to find this information. I would suspect a value of 10 - 20 for $N$?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If anyone knows (and has links) I would be grateful for this information.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Aside from R, Minitab, I can think of these packages SAS, SPPS, Stata, Matlab, Excel(?), any other packages I should consider?&lt;/p&gt;&#xA;" OwnerUserId="3311" LastEditorUserId="3311" LastEditorDisplayName="" LastEditDate="2011-03-01T12:53:58.627" LastActivityDate="2011-03-02T14:06:57.093" Title="What are the software limitations in all possible subsets selection in regression? " Tags="&lt;regression&gt;&lt;model-selection&gt;&lt;multivariable&gt;" AnswerCount="4" CommentCount="7" FavoriteCount="0" />
  <row Id="7731" PostTypeId="2" ParentId="7730" CreationDate="2011-03-01T03:19:54.343" Score="7" ViewCount="" Body="&lt;p&gt;I suspect 30--60 is about the best you'll get. The standard approach is the &lt;em&gt;leaps-and-bounds&lt;/em&gt; algorithm which doesn't require fitting every possible model. In $R$, the &lt;strong&gt;&lt;a href=&quot;http://cran.r-project.org/web/packages/leaps/index.html&quot; rel=&quot;nofollow&quot;&gt;leaps&lt;/a&gt;&lt;/strong&gt; package is one implementation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The documentation for the &lt;code&gt;regsubsets&lt;/code&gt; function in the &lt;strong&gt;leaps&lt;/strong&gt; package states that it will handle up to 50 variables without complaining. It can be &quot;forced&quot; to do more than 50 by setting the appropriate boolean flag.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might do a bit better with some parallelization technique, but the number of total models you can consider will (almost undoubtedly) only scale linearly with the number of CPU cores available to you. So, if 50 variables is the upper limit for a single core, and you have 1000 cores at your disposal, you could bump that to about 60 variables.&lt;/p&gt;&#xA;" OwnerUserId="2970" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-03-01T04:09:02.487" LastActivityDate="2011-03-01T04:09:02.487" CommentCount="6" />
  <row Id="7732" PostTypeId="1" CreationDate="2011-03-01T03:21:16.320" Score="1" ViewCount="270" Body="&lt;p&gt;I have 2 acceleration vectors, each represented by a matrix with its first column corresponding to the magnitude of acceleration and second column corresponding to the time (in ms) They both represent the same data, but one sensor is started a little later than the other, so I'm trying to remove the time lag using correlation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I find the correlation between these 2 matrices? I tried to do &lt;code&gt;xcorr2&lt;/code&gt; but it doesn't seem right and doing &lt;code&gt;xcorr&lt;/code&gt; would probably calculate correlation with only the values into consideration, but I also want to account for the time. The idea is that I should finally get a correlation vector wrt time and then when I calculate the peak of the correlation vector, its index would correspond to the time lag.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm really confused here, will an FFT help? How would you suggest I go about it?&lt;/p&gt;&#xA;" OwnerUserId="3233" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-03-01T04:27:36.000" LastActivityDate="2011-03-01T04:27:36.000" Title="Calculate Cross Correlation of two matrices of the 'Values Vs. Time' representation" Tags="&lt;correlation&gt;&lt;matlab&gt;&lt;autocorrelation&gt;&lt;cross-correlation&gt;&lt;fourier-transform&gt;" CommentCount="7" FavoriteCount="1" />
  <row Id="7733" PostTypeId="1" CreationDate="2011-03-01T05:59:05.950" Score="0" ViewCount="206" Body="&lt;p&gt;Let's say I have a time series of stocks (50 stocks).  What's the best way to compare such a time series and then derive/sort the results based on the most valuable stocks (ie stocks that have been going up/stayed the same relative to others). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Obviously one can use just average the whole time series, but I've been thinking about using a weighted average (putting weight on the most recent numbers) to account for trends in stock price.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I basically want a sorted list of the most valuable stocks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would this make sense?  or is there another method?&lt;/p&gt;&#xA;" OwnerUserId="2664" LastActivityDate="2011-05-21T00:48:26.250" Title="Compare time series of data" Tags="&lt;time-series&gt;&lt;weighted-mean&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="7734" PostTypeId="1" CreationDate="2011-03-01T06:45:55.910" Score="14" ViewCount="250" Body="&lt;h3&gt;Context:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;In an effort to structure the center pieces that I have came across in probability theory and statics, I created a reference document focussing on the mathematical essentials (available &lt;a href=&quot;http://cs.berkeley.edu/~mavam/probstat.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By sharing this document, I hope to give statistics students a comprehensive summary of the core material taught in graduate courses about these topics. While mainly intended as a teaching resource, folks might also find it helpful as a personal reference, e.g., to look up distribution relationships or illustrations of common PDFs. I also maintain a &#xA;&lt;a href=&quot;http://matthias.vallentin.net/2010/10/probability-theory-and-statistics.html&quot; rel=&quot;nofollow&quot;&gt;page&lt;/a&gt; with updates and fixes. Feedback is always much appreciated.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Question:&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What are your favorite statistics cheat sheets, references, or cookbooks that I could use for inspiration?&lt;/li&gt;&#xA;&lt;li&gt;What helped you to structure your knowledge in this domain? &lt;/li&gt;&#xA;&lt;li&gt;In the long term, my plan is to enrich this document (or create a separate one) with R examples  to bridge the gap between theory and practice. Would you deem this a valuable extension?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="1537" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-03-01T10:36:33.360" LastActivityDate="2011-03-01T10:36:33.360" CommunityOwnedDate="2011-03-01T07:17:43.573" Title="Suggestions for improving a probability and statistics cheat sheet" Tags="&lt;teaching&gt;" AnswerCount="2" CommentCount="0" FavoriteCount="7" />
  <row Id="7735" PostTypeId="2" ParentId="7734" CreationDate="2011-03-01T07:40:34.397" Score="3" ViewCount="" Body="&lt;p&gt;My favorite is the &lt;a href=&quot;http://www.burns-stat.com/pages/Tutor/R_inferno.pdf&quot; rel=&quot;nofollow&quot;&gt;R Inferno&lt;/a&gt; by Patrick Burns.&lt;/p&gt;&#xA;" OwnerUserId="3309" LastActivityDate="2011-03-01T07:40:34.397" CommunityOwnedDate="2011-03-01T07:40:34.397" CommentCount="1" />
  <row Id="7736" PostTypeId="2" ParentId="7734" CreationDate="2011-03-01T08:01:54.057" Score="4" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://cran.r-project.org/doc/contrib/Short-refcard.pdf&quot; rel=&quot;nofollow&quot;&gt;Tom Short's R Reference Card&lt;/a&gt; is excellent.&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2011-03-01T08:01:54.057" CommunityOwnedDate="2011-03-01T08:01:54.057" CommentCount="1" />
  <row Id="7737" PostTypeId="2" ParentId="7723" CreationDate="2011-03-01T09:34:17.290" Score="1" ViewCount="" Body="&lt;p&gt;The first thing you should do is &lt;em&gt;challenge&lt;/em&gt; your lecturer to explain these things clearly.  If &lt;em&gt;anything whatsoever&lt;/em&gt; seems counter-intuitive or backwards, them demand that he/she explains why it is intuitive.  Statistics always makes sense if you think about it in the &quot;right&quot; way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Calculating pivotal quantities is a very tricky business - I completely understand your bewilderment in &quot;where should I start?&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For normal variance parameters, The &quot;pivotal quantity&quot; is the sum of squares divided by the variance parameters:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$T_{X}=\sum_{i=1}^{N}\Big(\frac{X_{i}-\mu_{X}}{\sigma{X}}\Big)^{2} \sim \chi^{2}(N)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And a similar expression for $T_{Y}$.  Note that the distribution only depends on $N$, which is known (if $\mu_{X}$ is unknown, replace by $\overline{X}$ and you lose one degree of freedom in the chi-square distribution).  Thus $\frac{T_{X}}{T_{Y}}$ is a pivotal quantity, which has a value of:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{\sum_{i=1}^{N}\Big(\frac{X_{i}-\mu_{X}}{\sigma{X}}\Big)^{2}}{\sum_{i=1}^{N}\Big(\frac{Y_{i}-\mu_{Y}}{\sigma{Y}}\Big)^{2}}&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that because it is a pivotal quantity, we can create an exact confidence interval using the pivot as a starting point, and then substituting in our statistic.  Now because the degrees of freedom are the same for each chi-square, we do indeed have an F distribution.  So you can write:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$1-\alpha=Pr(L &amp;lt; F &amp;lt; U)=Pr(L &amp;lt; \frac{T_{X}}{T_{Y}} &amp;lt; U)$$&#xA;$$1-\alpha=Pr(L &amp;lt; \frac{\sum_{i=1}^{N}\Big(\frac{X_{i}-\mu_{X}}{\sigma{X}}\Big)^{2}}{\sum_{i=1}^{N}\Big(\frac{Y_{i}-\mu_{Y}}{\sigma{Y}}\Big)^{2}} &amp;lt; U)$$&#xA;$$1-\alpha=Pr(L &amp;lt; \frac{\sigma_{Y}^{2}}{\sigma_{X}^{2}}\frac{\sum_{i=1}^{N}(X_{i}-\mu_{X})^{2}}{\sum_{i=1}^{N}(Y_{i}-\mu_{Y})^{2}} &amp;lt; U)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Writing the observed ratio of the sum of squares as $R$ we get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$1-\alpha=Pr(L &amp;lt; \frac{\sigma_{Y}^{2}}{\sigma_{X}^{2}} R &amp;lt; U)$$&#xA;$$1-\alpha=Pr(\frac{L}{R} &amp;lt; \frac{\sigma_{Y}^{2}}{\sigma_{X}^{2}} &amp;lt; \frac{U}{R})$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for how this solution comes about, I have absolutely no idea.  What &quot;principles&quot; were followed (apart from being good at re-arranging statistical expression)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One thing that I can think of is that you need to find some way to &quot;standardise&quot; your sampling distribution.  So for example, normals you subtract mean and divide by standard deviation.  For gamma you multiply by the scale parameter.  I don't know many pivotal quantities that exist outside of the normal and gamma families.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think this is one reason why ordinary &quot;sampling statistics&quot; is an art more than a sciecne, because you have to use your intuition about what statistics to try.  And then you have to try and figure out if you can standardise your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am almost certain your lecturer will bring up the subject of confidence intervals - be sure to ask him/her what you should do when you only have one sample, or when you have 2  or more nuisance parameters. :)&lt;/p&gt;&#xA;" OwnerUserId="2392" LastEditorUserId="2392" LastEditorDisplayName="" LastEditDate="2011-03-06T12:13:16.900" LastActivityDate="2011-03-06T12:13:16.900" CommentCount="2" />
  <row Id="7738" PostTypeId="1" CreationDate="2011-03-01T10:03:07.877" Score="0" ViewCount="98" Body="&lt;p&gt;Hi Everyone&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone give a insight about how to compute p-value.&#xA;I tried a lot and found that, look into the table and you will find the range for p-value, but instead of that I want to compute the exact p-value, perhaps I want to compute something like this:-&#xA;&lt;a href=&quot;http://www.graphpad.com/quickcalcs/pvalue1.cfm?CFID=11145312&amp;amp;CFTOKEN=81e59ca508e5ac24-68552A6D-1A4B-0BFB-435F96639648BF7A&quot; rel=&quot;nofollow&quot;&gt;http://www.graphpad.com/quickcalcs/pvalue1.cfm?CFID=11145312&amp;amp;CFTOKEN=81e59ca508e5ac24-68552A6D-1A4B-0BFB-435F96639648BF7A&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm interested to know, if there is any formula, which can directly compute this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks a lot, in very-advance....for putting any valuable thoughts.&lt;/p&gt;&#xA;" OwnerUserId="3481" LastActivityDate="2011-03-01T10:03:07.877" ClosedDate="2011-03-01T13:52:35.050" Title="Compute P-Value" Tags="&lt;p-value&gt;" CommentCount="2" />
  <row Id="7739" PostTypeId="2" ParentId="7730" CreationDate="2011-03-01T10:17:03.143" Score="2" ViewCount="" Body="&lt;p&gt;As $N$ gets big, your ability to use maths becomes absolutely crucial.  &quot;inefficient&quot; mathematics will cost you at the PC.  The upper limit depends on what equation you are solving.  Avoiding matrix inverse or determinant calculations is a big advantage.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One way to help with increasing the limit is to use theorems for decomposing a large matrix inverse from into smaller matrix inverses.  This can often means the difference between feasible and not feasible.  But this involves some hard work, and often quite complicated mathematical manipulations!  But it is usually worth the time.  Do the maths or do the time!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bayesian methods might be able to give an alternative way to get your result - might be quicker, which means your &quot;upper limit&quot; will increase (if only because it gives you two alternative ways of calculating the same answer - the smaller of two, will always be smaller than one of them!).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you can calculate a regression coefficient without inverting a matrix, then you will probably save a lot of time.  This may be particularly useful in the Bayesian case, because &quot;inside&quot; a normal marginalisation integral, the $X^{T}X$ matrix does not need to be inverted, you just calculate a sum of squares.  Further, the determinant matrix will form part of the normalising constant.  So &quot;in theory&quot; you could use sampling techniques to numerically evaluate the integral (even though it has an analytic expression) which will be eons faster than trying to evaluate the &quot;combinatorical explosion&quot; of matrix inverses and determinants.  (it will still be a &quot;combinatorical explosion&quot; of numerical integrations, but this may be quicker).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This suggestion above is a bit of a &quot;thought bubble&quot; of mine.  I want to actually test it out, see if it's any good.  I think it would be (5,000 simulations + calculate exp(sum of squares) + calculate weighted average beta should be faster than matrix inversion for a big enough matrix.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The cost is approximate rather than exact estimates.  There is nothing to stop you from using the same set of pseudo random numbers to numerically evaluate the integral, which will again, save you a great deal of time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is also nothing stopping you from using a combination of either technique.  Use exact when the matrices are small, use simulation when they are big.  This is because in this part of the analysis.  It is just different numerical techniques - just pick the technique which is quickest!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course this is all just a bit of &quot;hand wavy&quot; arguments, I don't exactly know the best software packages to use - and worse, trying to figure out which algorithms they actually use.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-03-01T10:17:03.143" CommentCount="14" />
  <row Id="7741" PostTypeId="2" ParentId="7698" CreationDate="2011-03-01T10:48:20.840" Score="0" ViewCount="" Body="&lt;p&gt;&quot;Would it work to use the linear function and simply cut all values below 0 to 0, and values above 1 to 1?&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I believe in many cases the cut-off value should be the percentage split of the training data. Eg if your training data has 13% - 0's and 87%  - 1's, then the cut-off would be 0.13; For example anything 0.13 and below on the output is 0 and anything 0.14 and above is 1. Obviously there is more uncertainty the closer to the cut-off the output provides. It may also help adjusting the cut-off limits especially where the cost of a mis-classification is high. This link may help a little &lt;a href=&quot;http://timmanns.blogspot.com/2009/11/building-neural-networks-on-unbalanced.html&quot; rel=&quot;nofollow&quot;&gt;http://timmanns.blogspot.com/2009/11/building-neural-networks-on-unbalanced.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="3484" LastActivityDate="2011-03-01T10:48:20.840" />
  <row Id="7742" PostTypeId="1" CreationDate="2011-03-01T12:01:31.540" Score="5" ViewCount="192" Body="&lt;p&gt;Suppose that the quantity which we want to infer is a probability distribution. All we know is that the distribution comes from a set $E$ determined, say, by some of its moments and we have a prior $Q$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The maximum entropy principle(MEP) says that the $P^{\star}\in E$ which has least relative entropy from $Q$ (i.e., $P^{\star}=\displaystyle \text{argmin}_{P\in E}D(P\|Q)$) is the best one to select.  Whereas the Bayesian rule of selection has a process of selecting the posterior given the prior which is supported by Bayes' theorem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is whether there is any connection between these two inference methods (i. e., whether the two methods apply to the same problem and have something in common)? Or whether in Bayesian inference the setting is completely different from the above mentioned setting? Or am I not making sense?!&lt;/p&gt;&#xA;" OwnerUserId="3485" LastEditorUserId="3485" LastEditorDisplayName="" LastEditDate="2011-03-09T14:09:38.940" LastActivityDate="2011-03-09T14:09:38.940" Title="Bayesian vs Maximum entropy" Tags="&lt;bayesian&gt;&lt;estimation&gt;&lt;maxent&gt;" CommentCount="9" FavoriteCount="2" />
  <row Id="7743" PostTypeId="2" ParentId="7730" CreationDate="2011-03-01T13:01:35.607" Score="6" ViewCount="" Body="&lt;p&gt;Just a caveat, but feature selection is a risky business, and the more features you have, the more degrees of freedom you have with which to optimise the feature selection criterion, and hence the greater the risk of over-fitting the feature selection criterion and in doing so obtain a model with poor generalisation ability.  It is possible that with an efficient algorithm and careful coding you can perform all subsets selection with a large number of features, that doesn't mean that it is a good idea to do it, especially if you have relatively few observations.  If you do use all subsets selection, it is vital to properly cross-validate the whole model fitting procedure (so that all-subset selection is performed independently in each fold of the cross-validation).  In practice, ridge regression with no feature selection often out-performs linear regression with feature selection (that advice is given in Millar's monograph on feature selection).&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2011-03-01T13:01:35.607" CommentCount="6" />
  <row Id="7744" PostTypeId="1" CreationDate="2011-03-01T13:42:42.827" Score="2" ViewCount="80" Body="&lt;p&gt;Can anyone give some advice on how to start proving this algebraically?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Define the residual from a regression (one independent variable) algebraically and show that:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;the mean of the residuals is zero&lt;/li&gt;&#xA;&lt;li&gt;the correlation of the residuals and the independent variable is zero&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="3487" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2011-03-01T13:51:30.480" LastActivityDate="2011-03-01T14:35:34.517" Title="Algebraic definition of a residual from a regression" Tags="&lt;regression&gt;&lt;homework&gt;&lt;residuals&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="7745" PostTypeId="1" AcceptedAnswerId="7752" CreationDate="2011-03-01T14:04:45.603" Score="2" ViewCount="72" Body="&lt;p&gt;Hello,&#xA;hopefully you can help me with the meaning of the following, I don't really understand the terminology:&#xA;&quot;regression of a vector of ones on the matrix $W$&quot;,&#xA;where $W$ is something like $(W_t)&amp;#39; = (w_{1t},w_{2t},w_{3t}, w_{4t})$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't understand, which regression I actually have to compute. If it is of help for you, I'm trying to understand the Godfrey/Wickens test according to this article: Ghosh, S., Gilbert, C. L., &amp;amp; Hallett, A. J. H. (1983). &lt;em&gt;Empirical Economics&lt;/em&gt;, 8, 63–69. &lt;a href=&quot;http://dx.doi.org/10.1007/BF01973190&quot; rel=&quot;nofollow&quot;&gt;DOI: 10.1007/BF01973190&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Greeting, Julius&lt;/p&gt;&#xA;" OwnerUserId="3104" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-03-01T14:34:47.363" LastActivityDate="2011-03-01T15:05:06.350" Title="Terminology question concerning regression" Tags="&lt;regression&gt;&lt;hypothesis-testing&gt;&lt;terminology&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="7746" PostTypeId="2" ParentId="7744" CreationDate="2011-03-01T14:10:29.967" Score="1" ViewCount="" Body="&lt;p&gt;Suppose you have the following regression model:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;y_i=\alpha+\beta x_i+\varepsilon_i&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Least squares problem looks for  $\alpha$ and $\beta$ which minimize the following function:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$g(\alpha,\beta)=\sum_{i=1}^n(y_i-\alpha-\beta x_i)^2$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Solution for this problem will satisfy &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{\partial g}{\partial \alpha}=0, \quad \frac{\partial g}{\partial \beta}=0.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Try differencing and look hard at the resulting expressions. You will see that this answers your problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that @cardinal is right, if you do not include $\alpha$, your first statement is false.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt; This is might be considered non-algebraic solution, so please state more clearly what do you mean by algebraic. If this is not helpful, I will retract my answer, which really is a long comment.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-03-01T14:35:34.517" LastActivityDate="2011-03-01T14:35:34.517" CommentCount="4" />
  <row Id="7747" PostTypeId="2" ParentId="5292" CreationDate="2011-03-01T14:19:56.700" Score="18" ViewCount="" Body="&lt;p&gt;You can also try the brand-new &lt;a href=&quot;http://www.rstudio.org/&quot; rel=&quot;nofollow&quot;&gt;R-Studio&lt;/a&gt;.  Reasonably full-featured IDE with easy set-up.  I played with it yesterday and it seems nice.&lt;/p&gt;&#xA;" OwnerUserId="3488" LastActivityDate="2011-03-01T14:19:56.700" CommunityOwnedDate="2011-03-01T14:19:56.700" CommentCount="3" />
  <row Id="7748" PostTypeId="2" ParentId="7698" CreationDate="2011-03-01T14:31:43.597" Score="0" ViewCount="" Body="&lt;p&gt;I am opposed to cutting values of, since this will lead to an undifferentiable transfer function and your gradient based training algorithm might screw up.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The sigmoid function at the output layer is fine: $\sigma(x) = \frac{1}{1 + e^{-x}}$. It will squash any output to lie within $(0, 1)$. So you can get arbitrarily close to the targets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, if you use the squared error you will lose the property of a &quot;matching loss function&quot;. When using linear outputs for a squared error, the derivatives of the error reduce to $y - t$ where $y$ is the output and $t$ the corresponding target value. So you have to check your gradients.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have personally had good results with sigmoids as outputs when I have targets in that range and using sum of squares error anyway.&lt;/p&gt;&#xA;" OwnerUserId="2860" LastActivityDate="2011-03-01T14:31:43.597" />
  <row Id="7749" PostTypeId="1" CreationDate="2011-03-01T14:36:24.950" Score="2" ViewCount="92" Body="&lt;p&gt;&quot;Under what condition (or conditions if you think it necessary) would one observe no change in the regression coefficient (e.g., b-hat Y on X1) for some variable when another variable is added to the regression equation?&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think the answer is when the exogenous variables are perfectly uncorrelated - is that correct?&lt;/p&gt;&#xA;" OwnerUserId="3487" LastActivityDate="2011-03-01T17:15:36.960" Title="Changes in the regression coefficient" Tags="&lt;regression&gt;" AnswerCount="1" />
  <row Id="7750" PostTypeId="2" ParentId="7749" CreationDate="2011-03-01T14:55:17.340" Score="5" ViewCount="" Body="&lt;p&gt;Basically yes. This follows from the &lt;a href=&quot;http://en.wikipedia.org/wiki/Omitted_variable_bias&quot; rel=&quot;nofollow&quot;&gt;omitted variable bias&lt;/a&gt; problem. As you can see the bias depends on crossproduct of the variables in regression (in this case the intercept and your variable of interest) and the omitted variable. If the sample correlation of the variables is zero and the sample mean of the omitted variable is zero, then  crossproduct is zero, hence there will be no change in the coefficient value. If true correlation and expectation of the omitted variable is zero then the expectation of this crossproduct is zero and bias is zero.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-03-01T17:15:36.960" LastActivityDate="2011-03-01T17:15:36.960" CommentCount="4" />
  <row Id="7751" PostTypeId="2" ParentId="7730" CreationDate="2011-03-01T14:56:19.553" Score="2" ViewCount="" Body="&lt;p&gt;I was able to generate all possible subsets using 50 variables in SAS.  I do not believe there is any hard limitation other than memory and CPU speed.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Edit&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;I generated the 2 best models for N=1 to 50 variables for 5000 observations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;@levon9 - No, this ran in under 10 seconds. I generated 50 random variables from (0,1)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-Ralph Winters&lt;/p&gt;&#xA;" OwnerUserId="3489" LastEditorUserId="3489" LastEditorDisplayName="" LastEditDate="2011-03-02T14:06:57.093" LastActivityDate="2011-03-02T14:06:57.093" CommentCount="4" />
  <row Id="7752" PostTypeId="2" ParentId="7745" CreationDate="2011-03-01T15:05:06.350" Score="4" ViewCount="" Body="&lt;p&gt;Ordinary least squares regression of $y$ on $X$ involves solving the normal equations&#xA;$$X&amp;#39;X\hat{\beta} = X&amp;#39;y$$&#xA;for $\hat{\beta}$, so I'd assume OLS regression of a vector of ones on $W$ implies solving&#xA;$$W&amp;#39;W\hat{\beta} = W&amp;#39;\bf{1},$$&#xA;where $\bf{1}$ is a vector of ones. If the matrix $X$ itself contained a column of ones, i.e. if the RHS of the regression included a constant, then the solution would be trivial, so I assume none of the $w_{it}$'s are constant in your case.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-03-01T15:05:06.350" CommentCount="3" />
  <row Id="7753" PostTypeId="2" ParentId="5292" CreationDate="2011-03-01T16:16:25.957" Score="1" ViewCount="" Body="&lt;p&gt;Despite all of the good recommendations, I've not found anything radically better than the default Mac GUI. R-Studio shows promise, but it's not currently that much more customizable or featureful than R and, say, BBEdit to edit.&lt;/p&gt;&#xA;" OwnerUserId="1764" LastActivityDate="2011-03-01T16:16:25.957" CommunityOwnedDate="2011-03-01T16:16:25.957" />
  <row Id="7754" PostTypeId="1" AcceptedAnswerId="7755" CreationDate="2011-03-01T16:37:19.447" Score="3" ViewCount="158" Body="&lt;h2&gt;Background&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I have two estimates of variance and their associated standard errors calculated from sample sizes of $n=500$ and $n=10,000$ the results are $\hat{\sigma^2} (sd_{\hat{\sigma^2}})$:&#xA;$$\hat{\sigma^2}_{n=500}=69 (6.4)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\hat{\sigma^2}_{n=10,000}=72 (1.5)$$&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Question&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;If I say that variance increased by 3, what is the standard deviation around this estimate?&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Notes&lt;/h2&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;$SD$ of var calculated using $SD_{\hat{\sigma^2}}=\sqrt{s^4(2/(n-1) + k/n)}$&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I suspect that the fact that I am estimating the sd of a variance is not relevant to the calculation, but may help in the interpretation of what I am doing.  &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="1381" LastEditorUserId="1381" LastEditorDisplayName="" LastEditDate="2011-03-01T19:10:57.737" LastActivityDate="2011-03-01T19:10:57.737" Title="How to calculate the difference of two standard deviations?" Tags="&lt;standard-deviation&gt;&lt;variance&gt;" AnswerCount="1" CommentCount="6" />
  <row Id="7755" PostTypeId="2" ParentId="7754" CreationDate="2011-03-01T17:00:15.870" Score="5" ViewCount="" Body="&lt;p&gt;The standard deviation of the difference between two independent random variables is the square root of the sum of the squares of their individual standard deviations (easier to express as variances) so in this case&#xA;$$\sqrt{6.4^2 + 1.5^2} \approx 6.6$$&lt;/p&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-03-01T17:00:15.870" CommentCount="5" />
  <row Id="7757" PostTypeId="1" AcceptedAnswerId="7759" CreationDate="2011-03-01T18:53:04.537" Score="3" ViewCount="224" Body="&lt;p&gt;I am trying to predict the outcome of a complex system using neural networks (ANN's). The outcome (dependent) values range between 0 and 10,000. The different input variables have different ranges. All the variables have roughly normal distributions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I consider different options to scale the data before training. One option is to scale the input (independent) and output (dependent) variables to [0, 1] by &lt;a href=&quot;http://en.wikipedia.org/wiki/Cumulative_distribution_function&quot; rel=&quot;nofollow&quot;&gt;computing cumulative distribution function&lt;/a&gt; using the mean and standard deviation values of each variable, independently. The problem with this method is that if I use the sigmoid activation function at the output, I will very likely miss extreme data, especially those not seen in the training set&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another option is to use a z-score. In that case I don't have the extreme data problem; however, I'm limited to a linear activation function at the output. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are other accepted normalization techniques that are in use with ANN's? I tried to look for reviews on this topic, but failed to find anything useful.&lt;/p&gt;&#xA;" OwnerUserId="1496" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-02T05:07:11.097" LastActivityDate="2011-03-02T05:07:11.097" Title="Data normalization and standardization in neural networks" Tags="&lt;machine-learning&gt;&lt;neural-networks&gt;&lt;multidimensional-scaling&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="1" />
  <row Id="7758" PostTypeId="1" AcceptedAnswerId="7767" CreationDate="2011-03-01T19:27:34.413" Score="2" ViewCount="106" Body="&lt;p&gt;I play a lot with &lt;a href=&quot;http://pybrain.org&quot; rel=&quot;nofollow&quot;&gt;PyBrain&lt;/a&gt; -- Artificial Neural Network implementation in Python. I have noticed that in all the models that I receive the weights of the connections are roughly normally distributed around zero with a pretty low standard deviation (~0.3), which means that they are effectively limited within the [-1, 1] range. What does this mean? Is it a requirement of ANN? An outcome of backpropagaion learning? A sign for network health? Or just a random observation?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/5q8YS.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="1496" LastActivityDate="2011-05-01T01:42:31.060" Title="On connection weights in an Artificial Neural Network" Tags="&lt;neural-networks&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="7759" PostTypeId="2" ParentId="7757" CreationDate="2011-03-01T20:27:31.693" Score="1" ViewCount="" Body="&lt;p&gt;A standard approach is to scale the inputs to have mean 0 and a variance of 1. Also linear decorrelation/whitening/pca helps a lot.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are interested in the tricks of the trade, I can recommend &lt;a href=&quot;http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf&quot; rel=&quot;nofollow&quot;&gt;LeCun's efficient backprop paper.&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2860" LastActivityDate="2011-03-01T20:27:31.693" />
  <row Id="7760" PostTypeId="1" CreationDate="2011-03-01T20:47:52.863" Score="1" ViewCount="24" Body="&lt;p&gt;I just want to confirm that I'm on the right track.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to derive an expression for b^Y1.2 that refers solely to variances and covariances (that is, only to bi-variate and mono-variate terms).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have come up with the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;b-hatY1.2 = (CovY1)/(Var1) - (Cov1,2/V1)*(b-hatY2.1)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That should be correct, but I just wanted some feedback.&lt;/p&gt;&#xA;" OwnerUserId="3495" LastActivityDate="2011-03-01T20:47:52.863" Title="Residualized Coefficient Using Solely Variances and Covariances" Tags="&lt;regression&gt;" CommentCount="1" />
  <row Id="7763" PostTypeId="1" AcceptedAnswerId="7764" CreationDate="2011-03-01T21:10:20.670" Score="3" ViewCount="66" Body="&lt;p&gt;I'm examining correlations in a data set with a large number of variables but small sample sizes. To get a feel for how these quantities behave, I generated some random data and looked at the distribution of correlations:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;n = 4&#xA;y = matrix(rnorm(1000 * n), 1000, n)&#xA;x = matrix(rnorm(1000 * n), 1000, n)&#xA;p = as.numeric(cor(t(x),t(y)))&#xA;hist(p)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To my surprise, the distribution is almost perfectly uniform:&#xA;&lt;img src=&quot;http://i.imgur.com/XbDeb.png&quot; alt=&quot;histogram&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone have an explanation for this phenomenon? It makes some sense in that for n=2 we have either p=1 or p=-1, and as n-&gt;infinity the distribution becomes normal, so this distribution falls somewhere inbetween. But why uniform? I'm stumped.&lt;/p&gt;&#xA;" OwnerUserId="2111" LastActivityDate="2011-03-01T21:37:34.823" Title="Curious Sample Correlation Property" Tags="&lt;distributions&gt;&lt;correlation&gt;" AnswerCount="1" />
  <row Id="7764" PostTypeId="2" ParentId="7763" CreationDate="2011-03-01T21:37:34.823" Score="11" ViewCount="" Body="&lt;p&gt;For independent Normal variates, the distribution of the correlation coefficient $r$ is proportional to $(1 - r^2)^{{1\over2} (n-4)}dr$.  When $n=4$, that's uniform.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Reference&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;R. A. Fisher, &lt;a href=&quot;http://www.stat.duke.edu/courses/Spring05/sta215/lec/Fish1915.pdf&quot; rel=&quot;nofollow&quot;&gt;Frequency-distribution of the values of the correlation coefficient in samples from an indefinitely large population&lt;/a&gt;.  &lt;em&gt;Biometrika&lt;/em&gt;, &lt;strong&gt;10&lt;/strong&gt;, 507.  See Section 3.  (Quoted in &lt;em&gt;Kendall's Advanced Theory of Statistics&lt;/em&gt;, 5th Ed., section 16.24.)&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-03-01T21:37:34.823" CommentCount="1" />
  <row Id="7765" PostTypeId="2" ParentId="7733" CreationDate="2011-03-01T21:50:02.553" Score="0" ViewCount="" Body="&lt;p&gt;It sounds as if you would be interested in computing Relative Strength&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.investopedia.com/terms/r/relativestrength.asp&quot; rel=&quot;nofollow&quot;&gt;http://www.investopedia.com/terms/r/relativestrength.asp&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could either measure it against a benchmark index such as the Dow 30,  or compute your own index from your 50 stocks and measure each individual stock against the index.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-Ralph Winters&lt;/p&gt;&#xA;" OwnerUserId="3489" LastActivityDate="2011-03-01T21:50:02.553" />
  <row Id="7766" PostTypeId="1" AcceptedAnswerId="7773" CreationDate="2011-03-01T22:38:49.620" Score="7" ViewCount="192" Body="&lt;p&gt;How must I syntax the &lt;code&gt;rma&lt;/code&gt; function from &lt;a href=&quot;http://cran.r-project.org/web/packages/metafor/index.html&quot; rel=&quot;nofollow&quot;&gt;metafor&lt;/a&gt; package in order to get results in the following real-life example of a small meta-analysis? &#xA;(random-effect, summary statistic SMD)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;study,    mean1,  sd1,    n1, mean2,   sd2,   n2&#xA;&#xA;Foo2000,    0.78,   0.05,   20, 0.82,    0.07,  25&#xA;&#xA;Sun2003,    0.74,   0.08,   30, 0.72,    0.05,  19&#xA;&#xA;Pric2005,   0.75,   0.12,   20, 0.74,   0.09,  29&#xA;&#xA;Rota2008,   0.62,  0.05,   24, 0.66,    0.03,  24&#xA;&#xA;Pete2008,   0.68,  0.03,   10, 0.68,    0.02,  10&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I tried metafor's tutorials and R function's examples with a poor outcome for the time being :(&lt;/p&gt;&#xA;" OwnerUserId="3333" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-03-02T08:07:11.010" LastActivityDate="2011-03-02T08:07:11.010" Title="Meta-analysis in R, using metafor package" Tags="&lt;r&gt;&lt;meta-analysis&gt;&lt;summary-statistics&gt;&lt;random-effects-model&gt;&lt;metafor&gt;" AnswerCount="1" FavoriteCount="3" />
  <row Id="7767" PostTypeId="2" ParentId="7758" CreationDate="2011-03-01T23:01:28.537" Score="2" ViewCount="" Body="&lt;p&gt;I just took a look at some of my neural networks; the weights in those look normally distributed. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One possible argument is that each weight is the sum of IID delta values during backpropagation, so they will be Gaussian (due to the central limit theorem). This argument involves making some simplifications; for example the summed deltas are probably not independent of each other during backpropagation. &lt;/p&gt;&#xA;" OwnerUserId="2965" LastEditorUserId="2965" LastEditorDisplayName="" LastEditDate="2011-03-02T01:00:12.740" LastActivityDate="2011-03-02T01:00:12.740" />
  <row Id="7768" PostTypeId="1" AcceptedAnswerId="7787" CreationDate="2011-03-01T23:04:01.467" Score="5" ViewCount="126" Body="&lt;p&gt;I've never really found any good text or examples on how to handle 'non-existent' data for inputs to any sort of classifier. I've read a lot on missing data but what can be done about data that cannot or doesn't exist in relation to multivariate inputs. I understand this is a very complex question and will vary depending on training methods used...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Eg if trying to predict laptime for several runners with good accurate data. Amongst many inputs, possible variables amongst many are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Input Variable - First time runner (Y/N)&lt;/li&gt;&#xA;&lt;li&gt;Input Variable - Previous laptime ( 0 - 500 seconds)&lt;/li&gt;&#xA;&lt;li&gt;Input Variable - Age&lt;/li&gt;&#xA;&lt;li&gt;Input Variable - Height&#xA;.&#xA;.&#xA;. many more Input variables etc&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&amp;amp; Output Predictor - Predicted Laptime (0 - 500 seconds)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A 'missing variable' for '2.Previous laptime' could be computed several ways but '1. First time runner' would always equal N . But for 'NON EXISTENT DATA' for a first time runner (where '1. First time runner' = Y) what value/treatment should I give for '2. Previous laptime'?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example assigning  '2. Previous laptime' as -99 or 0 can skew the distribution dramatically and make it look like a new runner has performed well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My current training methods have been using Logistic regression, SVM, NN &amp;amp; Decision trees&lt;/p&gt;&#xA;" OwnerUserId="3484" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-03-02T11:37:02.397" LastActivityDate="2011-03-02T15:01:25.117" Title="How to handle non existent (not missing) data?" Tags="&lt;missing-data&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="1" />
  <row Id="7769" PostTypeId="2" ParentId="7718" CreationDate="2011-03-01T23:35:48.853" Score="1" ViewCount="" Body="&lt;p&gt;Following up on SheldonCooper's second and third bullet points: The ideal choice is to have &lt;em&gt;somebody else&lt;/em&gt; make the choice, either in the form of a threshold (point 3) or a cost benefit tradeoff (point 2).  And perhaps the nicest way to offer them the choice is with an &lt;a href=&quot;http://en.wikipedia.org/wiki/Receiver_operating_characteristic&quot; rel=&quot;nofollow&quot;&gt;ROC curve&lt;/a&gt;.  &lt;/p&gt;&#xA;" OwnerUserId="1739" LastActivityDate="2011-03-01T23:35:48.853" />
  <row Id="7770" PostTypeId="2" ParentId="7721" CreationDate="2011-03-01T23:49:23.417" Score="3" ViewCount="" Body="&lt;p&gt;I've had good experiences with Madigan's and Lewis's &lt;a href=&quot;http://www.bayesianregression.org&quot; rel=&quot;nofollow&quot;&gt;BMR and BBR&lt;/a&gt; packages for multiple category dependent variables, lasso or ridge priors on parameters, and high dimensional input data.  Not quite as high as yours, but it might still be worth a look.  Instructions are here: &lt;a href=&quot;http://bayesianregression.com/bmr.html&quot; rel=&quot;nofollow&quot;&gt;http://bayesianregression.com/bmr.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1739" LastEditorUserId="1739" LastEditorDisplayName="" LastEditDate="2011-03-02T20:44:34.913" LastActivityDate="2011-03-02T20:44:34.913" CommentCount="3" />
  <row Id="7771" PostTypeId="1" AcceptedAnswerId="7849" CreationDate="2011-03-02T00:54:11.730" Score="5" ViewCount="210" Body="&lt;p&gt;Hi,&#xA;is there a way of calculating an effect size for the Kolmogorov-Smirnov Z statistic (in SPSS or by hand)? Or should I stick to the Mann-Whitney test, even though my group sizes are less than n=25?&lt;/p&gt;&#xA;" OwnerUserId="2025" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-04-28T20:23:56.830" LastActivityDate="2011-04-28T20:23:56.830" Title="How do I calculate the effect size for the Kolmogorov-Smirnov Z statistic?" Tags="&lt;effect-size&gt;&lt;kolmogorov-smirnov&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="7772" PostTypeId="1" AcceptedAnswerId="7819" CreationDate="2011-03-02T03:05:21.403" Score="4" ViewCount="136" Body="&lt;p&gt;I'm trying to fit a multivariate multiple regression model where the independent variable X is latent but I don't know where to start (I have prior information about the coefficient matrix so I can use some iterative method).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The dependent variable Y is a NxM matrix denoting N observations each from M variables. The latent variable X is a NxP matrix about which we know nothing except the dimensions. In addition to these, we have an initial estimate of the coefficient matrix beta based on prior knowledge. My goal is to find the estimates of latent X and coefficient matrix beta by both using the data matrix Y and the initial coefficient matrix. I thought of constructing an EM algorithm but because of the complexity of multivariate data and latent variable concept, I am totally confused.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you.&lt;/p&gt;&#xA;" OwnerUserId="3499" LastEditorUserId="3499" LastEditorDisplayName="" LastEditDate="2011-03-02T20:20:02.473" LastActivityDate="2011-03-02T20:42:51.933" Title="Is it possible to fit a multivariate regression model where the independent variable is latent?" Tags="&lt;regression&gt;&lt;multivariate-analysis&gt;&lt;latent-variable&gt;" AnswerCount="1" CommentCount="6" />
  <row Id="7773" PostTypeId="2" ParentId="7766" CreationDate="2011-03-02T03:34:49.143" Score="7" ViewCount="" Body="&lt;p&gt;Create a proper &lt;code&gt;data.frame&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df &amp;lt;- structure(list(study = structure(c(1L, 5L, 3L, 4L, 2L), .Label = c(&quot;Foo2000&quot;, &#xA;&quot;Pete2008&quot;, &quot;Pric2005&quot;, &quot;Rota2008&quot;, &quot;Sun2003&quot;), class = &quot;factor&quot;), &#xA;    mean1 = c(0.78, 0.74, 0.75, 0.62, 0.68), sd1 = c(0.05, 0.08, &#xA;    0.12, 0.05, 0.03), n1 = c(20L, 30L, 20L, 24L, 10L), mean2 = c(0.82, &#xA;    0.72, 0.74, 0.66, 0.68), sd2 = c(0.07, 0.05, 0.09, 0.03, &#xA;    0.02), n2 = c(25L, 19L, 29L, 24L, 10L)), .Names = c(&quot;study&quot;, &#xA;&quot;mean1&quot;, &quot;sd1&quot;, &quot;n1&quot;, &quot;mean2&quot;, &quot;sd2&quot;, &quot;n2&quot;), class = &quot;data.frame&quot;, row.names = c(NA, &#xA;-5L))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Run the &lt;code&gt;rma&lt;/code&gt;-function:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(metafor)&#xA;rma(measure = &quot;SMD&quot;, m1i = mean1, m2i = mean2, &#xA;    sd1i = sd1, sd2i = sd2, n1i = n1, n2i = n2, &#xA;    method = &quot;REML&quot;, data = df)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Please be aware that &lt;code&gt;rma&lt;/code&gt; assumes &lt;code&gt;(m1i-m2i)&lt;/code&gt;. This results in the following univariate random effects model meta-analysis: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; rma(measure = &quot;SMD&quot;, m1i = mean1, m2i = mean2, &#xA;+     sd1i = sd1, sd2i = sd2, n1i = n1, n2i = n2, &#xA;+     method = &quot;REML&quot;, data = df)&#xA;&#xA;Random-Effects Model (k = 5; tau^2 estimator: REML)&#xA;&#xA;tau^2 (estimate of total amount of heterogeneity): 0.1951 (SE = 0.2127)&#xA;tau (sqrt of the estimate of total heterogeneity): 0.4416&#xA;I^2 (% of total variability due to heterogeneity): 65.61%&#xA;H^2 (total variability / within-study variance):   2.91&#xA;&#xA;Test for Heterogeneity: &#xA;Q(df = 4) = 11.8763, p-val = 0.0183&#xA;&#xA;Model Results:&#xA;&#xA;estimate       se     zval     pval    ci.lb    ci.ub          &#xA; -0.2513   0.2456  -1.0233   0.3061  -0.7326   0.2300          &#xA;&#xA;---&#xA;Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You might want to change the estimation &lt;code&gt;method&lt;/code&gt;, e.g. &lt;code&gt;method = &quot;DL&quot;&lt;/code&gt; (but I would stick with &lt;code&gt;REML&lt;/code&gt;).&lt;/p&gt;&#xA;" OwnerUserId="307" LastEditorUserId="307" LastEditorDisplayName="" LastEditDate="2011-03-02T04:38:14.870" LastActivityDate="2011-03-02T04:38:14.870" CommentCount="1" />
  <row Id="7774" PostTypeId="1" AcceptedAnswerId="7917" CreationDate="2011-03-02T03:58:17.613" Score="9" ViewCount="214" Body="&lt;p&gt;In the classic &lt;a href=&quot;http://en.wikipedia.org/wiki/Coupon_collector%27s_problem&quot; rel=&quot;nofollow&quot;&gt;Coupon Collector's problem&lt;/a&gt;, it is well known that the time $T$ necessary to complete a set of $n$ randomly-picked coupons satisfies $E[T] \sim n \ln n $,$Var(T) \sim n^2$, and $\Pr(T &amp;gt; n \ln n + cn) &amp;lt; e^{-c}$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This upper bound is better than the one given by the Chebyshev inequality, which would be roughly  $1/c^2$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is: is there a corresponding better-than-Chebyshev &lt;em&gt;lower bound&lt;/em&gt; for $T$? (e.g., something like $\Pr(T &amp;lt; n \ln n - cn) &amp;lt; e^{-c}$ ) ?&lt;/p&gt;&#xA;" OwnerUserId="3500" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-03-02T08:10:56.640" LastActivityDate="2011-03-08T01:58:42.140" Title="What is a tight lower bound on the coupon collector time?" Tags="&lt;probability&gt;" AnswerCount="3" CommentCount="1" FavoriteCount="5" />
  <row Id="7775" PostTypeId="1" CreationDate="2011-03-02T04:13:28.443" Score="7" ViewCount="219" Body="&lt;p&gt;Does anyone have suggestions or packages that will calculate the coefficient of partial determination?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The coefficient of partial determination can be defined as the percent of variation that cannot be explained in a reduced model, but can be explained by the predictors specified in a full(er) model. This coefficient is used to provide insight into whether or not one or more additional predictors may be useful in a more fully specified regression model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The calculation for the partial r^2 is relatively straight forward after estimating your two models and generating the ANOVA tables for them. The calculation for the partial r^2 is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(SSEreduced - SSEfull) / SSEreduced&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've written this relatively simple function that will calculate this for a multiple linear regression model. I'm unfamiliar with other model structures in R where this function may not perform as well:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;partialR2 &amp;lt;- function(model.full, model.reduced){&#xA;    anova.full &amp;lt;- anova(model.full)&#xA;    anova.reduced &amp;lt;- anova(model.reduced)&#xA;&#xA;    sse.full &amp;lt;- tail(anova.full$&quot;Sum Sq&quot;, 1)&#xA;    sse.reduced &amp;lt;- tail(anova.reduced$&quot;Sum Sq&quot;, 1)&#xA;&#xA;    pR2 &amp;lt;- (sse.reduced - sse.full) / sse.reduced&#xA;    return(pR2)&#xA;&#xA;    }&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Any suggestions or tips on more robust functions to accomplish this task and/or more efficient implementations of the above code would be much appreciated.&lt;/p&gt;&#xA;" OwnerUserId="696" LastActivityDate="2011-05-18T20:47:47.030" Title="R implementation of coefficient of partial determination" Tags="&lt;r&gt;&lt;regression&gt;&lt;anova&gt;" AnswerCount="1" CommentCount="5" />
  <row Id="7776" PostTypeId="2" ParentId="5292" CreationDate="2011-03-02T05:24:33.013" Score="1" ViewCount="" Body="&lt;p&gt;At least on linux, &lt;a href=&quot;http://rkward.sourceforge.net/&quot; rel=&quot;nofollow&quot;&gt;RKWard&lt;/a&gt; offers the best functionality. The new &lt;a href=&quot;http://www.rstudio.org/&quot; rel=&quot;nofollow&quot;&gt;RStudio&lt;/a&gt; appears quite promising as well.&lt;/p&gt;&#xA;" OwnerUserId="3502" LastActivityDate="2011-03-02T05:24:33.013" CommunityOwnedDate="2011-03-02T05:24:33.013" />
  <row Id="7777" PostTypeId="2" ParentId="7720" CreationDate="2011-03-02T05:54:08.080" Score="3" ViewCount="" Body="&lt;p&gt;Hi dfrankow:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would suggest you look at books on categorical data analysis (cf. Alan Agresti's Categorical Data Analysis, 2002) for better explanation and understanding of &lt;strong&gt;ordered logistic regression&lt;/strong&gt;. All the questions that you ask are basically answered by a few chapters in such books. If you are only interested in &lt;code&gt;R&lt;/code&gt; related examples, &lt;em&gt;Extending Linear Models in R&lt;/em&gt; by &lt;em&gt;Julian Faraway&lt;/em&gt; (CRC Press, 2008) is a great reference.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Before I answer your questions, &lt;em&gt;ordered logistic regression&lt;/em&gt; is a case of &lt;em&gt;multinomial logit models&lt;/em&gt; in which the categories are ordered. Suppose we have $J$ ordered categories and that for individual $i$, with ordinal response $Y_i$,&#xA;$p_{ij}=P(Yi=j)$ for $j=1,..., J$. With an ordered response, it is often easier to work with the cumulative probabilities, $\gamma_{ij}=P(Y_i \le j)$. The cumulative probabilities are increasing and invariant to combining adjacent categories. Furthermore, $\gamma_{iJ}=1$, so we need only model $J–1$ probabilities.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now we want to link $\gamma_{ij}$s to covariates $x$. In your case, &lt;code&gt;Sat&lt;/code&gt; has 3 ordered levels: &lt;code&gt;low&lt;/code&gt;, &lt;code&gt;medium&lt;/code&gt;, &lt;code&gt;high&lt;/code&gt;. It makes more sense to treat them as ordered rather than unordered. The remaining variables are your covariates. The specific model that you are considering is the &lt;strong&gt;proportional odds model&lt;/strong&gt; and is mathematically equivalent to:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\mbox{logit } \gamma_j(x_i) = \theta_j - \beta^T x_i, j = 1 \ldots J-1$$&#xA;$$\mbox{where }\gamma_j(x_i)=P(Y_i \le j | x_i)$$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is so called because the relative odds for $Y \le j$ comparing $x_1$ and $x_2$ are:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$(\frac {\gamma_j(x_1)}{1-\gamma_j(x_1)}) / (\frac {\gamma_j(x_2)}{1-\gamma_j(x_2)})=\exp(-\beta^T (x_1-x_2))$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Notice, the above expression does not depend on $j$. Of course, the assumption of proportional odds does need to be checked for a given dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, I will answer some (1, 2, 4) questions.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;How can one understand if the model&#xA;  gave a good fit? summary(house.plr)&#xA;  shows Residual Deviance 3479.149 and&#xA;  AIC (Akaike Information Criterion?) of&#xA;  3495.149. Is that good? In the case those are only useful as relative&#xA;  measures (i.e. to compare to another&#xA;  model fit), what is a good absolute&#xA;  measure? Is the residual deviance&#xA;  approximately chi-squared distributed?&#xA;  Can one use &quot;% correctly predicted&quot; on&#xA;  the original data or some&#xA;  cross-validation? What is the easiest&#xA;  way to do that?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;A model fit by &lt;code&gt;polr&lt;/code&gt; is a special &lt;code&gt;glm&lt;/code&gt;, so all the assumptions that hold for a traditional &lt;code&gt;glm&lt;/code&gt; hold here. If you take care of the parameters properly, you can figure out the distribution. Specifically, to test the &lt;strong&gt;if the model is good or not&lt;/strong&gt; you may want to do a &lt;strong&gt;goodness of fit test&lt;/strong&gt;, which test the following null (notice this is subtle, mostly you want to reject the null, but here you don't want to reject it to get a good fit):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$H_o: \mbox{ current model is good enough }$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You would use the &lt;strong&gt;chi-square test&lt;/strong&gt; for this. The p-value is obtained as:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1-pchisq(deviance(house.plr),df.residual(house.plr))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Most of the time you'd hope to obtain a p-value greater than 0.05 so that you don't reject the null to conclude that the model is good fit (philosophical correctness is ignored here).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;AIC should be high for a good fit at the same time you don't want to have a large number of parameters. &lt;code&gt;stepAIC&lt;/code&gt; is a good way to check this. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Yes, you can definitely use cross validation to see if the predictions hold. See &lt;code&gt;predict&lt;/code&gt; function (option: &lt;code&gt;type = &quot;probs&quot;&lt;/code&gt;) in &lt;code&gt;?polr&lt;/code&gt;. All you need to take care of is the covariates.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;What information does pr contain? The&#xA;  help page on profile is generic, and&#xA;  gives no guidance for polr&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;As pointed by @chl and others, &lt;code&gt;pr&lt;/code&gt; contains all the information needed for obtaining CIs and other likelihood related information of the &lt;code&gt;polr fit&lt;/code&gt;. All &lt;code&gt;glm&lt;/code&gt;s are fit using iteratively weighted least square estimation method for the log likelihood. In this optimization you obtain a lot of information (please see the references) which will be needed for calculating Variance Covariance Matrix, CI, t-value etc. It includes all of it.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;How does one interpret the t values for each coefficient? Unlike some model &gt;fits, there are no P values here.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Unlike normal linear model (special &lt;code&gt;glm&lt;/code&gt;) other &lt;code&gt;glm&lt;/code&gt;s are don't have the nice t-distribution for the regression coefficients. Therefore all you can get is the parameter estimates and their asymptotic variance covariance matrix using the max-likelihood theory. Therefore:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Variance(\hat \beta) = (X^T W X)^{-1}\hat \phi$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Estimate divided by its standard error is what BDR and WV call t-value (I am assuming &lt;code&gt;MASS&lt;/code&gt; convention here). It is equivalent to t-value from normal linear regression but does not follow a t-distribution. Using CLT, it is asymptotically normally distributed. But they prefer not to use this approx (I guess), hence no p-values. (I hope I am not wrong, and if I am, I hope BDR is not on this forum. I further hope, someone will correct me if I am wrong.)&lt;/p&gt;&#xA;" OwnerUserId="1307" LastEditorUserId="1307" LastEditorDisplayName="" LastEditDate="2011-03-02T06:24:14.897" LastActivityDate="2011-03-02T06:24:14.897" CommentCount="4" />
  <row Id="7778" PostTypeId="1" CreationDate="2011-03-02T07:01:14.617" Score="-2" ViewCount="92" Body="&lt;p&gt;For example, how do they come to  the conclusion that there are ??? people visited a particular web site?&lt;/p&gt;&#xA;" OwnerUserId="3503" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-02T07:57:16.683" LastActivityDate="2011-03-02T07:57:16.683" ClosedDate="2011-03-02T10:03:59.793" Title="How does website know how many people have visited?" Tags="&lt;data-collection&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="7779" PostTypeId="2" ParentId="7778" CreationDate="2011-03-02T07:13:26.197" Score="0" ViewCount="" Body="&lt;p&gt;Every time someone connects they update a counter in their database. They may use IP and/or cookie filtering (usually reset on a daily basis) to avoid counting the same person more then once.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I don't really see why would you tag this question as &quot;statistical-significance&quot;... maybe you want to know something else? &lt;/p&gt;&#xA;" OwnerUserId="582" LastActivityDate="2011-03-02T07:13:26.197" CommentCount="3" />
  <row Id="7780" PostTypeId="1" AcceptedAnswerId="7781" CreationDate="2011-03-02T07:31:29.113" Score="5" ViewCount="87" Body="&lt;p&gt;Hello !&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a question about &lt;strong&gt;group sequential methods&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;According to Wikipedia:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In a randomized trial with two treatment groups, classical group sequential testing is used in the following manner: If n subjects in each group are available, an interim analysis is conducted on the 2n subjects. The statistical analysis is performed to compare the two groups, and if the alternative hypothesis is accepted, the trial is terminated. Otherwise, the trial continues for another 2n subjects, with n subjects per group. The statistical analysis is performed again on the 4n subjects. If the alternative is accepted, then the trial is terminated. Otherwise, it continues with periodic evaluations until N sets of 2n subjects are available. At this point, the last statistical test is conducted, and the trial is discontinued&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;But by repeatedly testing accumulating data in this fashion, the type I error level is inflated...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the samples were independent of one another, the overall type I error, $\alpha^{\star}$, would be&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;$\alpha^{\star} = 1 - (1 - \alpha)^k$&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;where $\alpha$ is the level of each test, and $k$ is the number of interim looks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But the samples are not independent since they overlap. Assuming interim analyses are performed at equal information increments, &lt;a href=&quot;http://www.biostat.uzh.ch/teaching/master/methods2009/biostat6_group_sequential.pdf&quot; rel=&quot;nofollow&quot;&gt;it can be found that (slide 6)&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/iFT4L.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can you explain me how this table is obtained?&lt;/p&gt;&#xA;" OwnerUserId="3019" LastActivityDate="2011-03-02T08:36:21.063" Title="Overall type I error when repeatedly testing accumulating data" Tags="&lt;multiple-comparisons&gt;&lt;clinical-trials&gt;&lt;type-i-errors&gt;" AnswerCount="1" />
  <row Id="7781" PostTypeId="2" ParentId="7780" CreationDate="2011-03-02T08:23:13.177" Score="6" ViewCount="" Body="&lt;p&gt;The following slides, through 14, explain the idea.  The point, as you note, is that the sequence of statistics is correlated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The context is a z-test with known standard deviation.  The first test statistic $z_1$, suitably standardized, has a Normal(0,1) distribution with cdf $\Phi$.  So does the second statistic $z_2$, but--because the first uses a subset of the data used for the second--the two statistics are correlated with correlation coefficient $\sqrt{1/2}$.  Therefore $(z_1, z_2)$ has a binormal distribution.  The probability of a type I error (under the null hypothesis) equals the probability that either (a) a type I error occurs in the first test or (b) a type I error does not occur in the first test but does occur in the second test.  Let $c = \Phi^{-1}(1 - 0.05/2)$ be the critical value (for a two-sided test with nominal size $\alpha$ = 0.05).  Then the chance of a type I error after two analyses equals the chance that $|z_1| &amp;gt; c$ or $|z_1| \le c$ and $|z_2| &amp;gt; c$.  Numeric integration gives the value 0.0831178 for this probability, agreeing with the table.  Subsequent values in the table are obtained with similar reasoning (and more complicated integrations).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This graphic depicts the binormal pdf and the region of integration (solid surface).&#xA;&lt;img src=&quot;http://i.imgur.com/3bulv.png&quot; alt=&quot;Binormal PDF, 3D surface plot&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-02T08:36:21.063" LastActivityDate="2011-03-02T08:36:21.063" CommentCount="3" />
  <row Id="7782" PostTypeId="1" CreationDate="2011-03-02T10:01:07.933" Score="7" ViewCount="96" Body="&lt;h3&gt;Context&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;I have got some problems with my doctoral dissertation. My thesis is Investigating Secondary Primary School Teachers' Organizational citizenship behaviours through their perceptions about organizational culture and their organizational trust levels. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a sample of 871 teachers. &#xA;I have three instruments but they were developed by other researchers and they were used in some other studies. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have been trying to analyse my data using structural equation modelling.&#xA;However while trying to do confirmatory factor analysis, only one instrument (organizational citizenship) was ok.&#xA;The other two intruments did not give the first order confirmatory analysis.&#xA;The RMSEA values were around 0.100.&#xA;Chi-square was too high, and chi-square divided by degrees of freedom was also too high.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Question&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What should I do with my instruments and CFAs? &lt;/li&gt;&#xA;&lt;li&gt;Would item parcelling or removing items from these instruments be a good idea? &lt;/li&gt;&#xA;&lt;li&gt;Or should I skip structural equation modelling and go on with regression? &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="3505" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-03-02T11:08:14.763" LastActivityDate="2011-03-02T14:30:59.527" Title="What to do following poor fit statistics for a confirmatory factor analysis?" Tags="&lt;factor-analysis&gt;&lt;sem&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="7783" PostTypeId="2" ParentId="7782" CreationDate="2011-03-02T10:59:44.943" Score="1" ViewCount="" Body="&lt;p&gt;Instead of looking for statistical solutions that directly solve this problem, I would look for solutions that improve the diagnosis.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, I'd compare the different samples used in the different studies.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, if you have the data, I'd look at the correlation patterns among the variables in the different samples.  (You may be able to get these from other authors).&lt;/p&gt;&#xA;" OwnerUserId="686" LastActivityDate="2011-03-02T10:59:44.943" />
  <row Id="7784" PostTypeId="1" CreationDate="2011-03-02T11:04:04.407" Score="6" ViewCount="59" Body="&lt;p&gt;I am training an artificial neural network (backpropagation, feed-forward) with non-normal distributed data. Beside the root mean squared error, literature does often suggest the Pearson correlation coefficient for evaluating the quality of the trained net. But, is the Pearson correlation coefficient reasonable, if the training data is not normally distributed? Would it not be more reasonable, to use a rank-based correlation measure, e.g. Spearman rho?&lt;/p&gt;&#xA;" OwnerUserId="3465" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-03-02T11:39:29.713" LastActivityDate="2011-03-03T16:45:19.690" Title="Measuring correlation of trained neural networks" Tags="&lt;correlation&gt;&lt;neural-networks&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="7785" PostTypeId="1" AcceptedAnswerId="7789" CreationDate="2011-03-02T11:06:55.250" Score="5" ViewCount="55" Body="&lt;p&gt;Hello, I am looking for good references on using directional data (measure of direction in degrees) as an independent variable in regression; ideally, it would also be useful for hierarchical nonlinear models (the data are nested).  I am also interested in directional data more generally.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have found a text by Mardia, which I am going to get, but wondered if there were good articles.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am more interested in practical articles about how to deal with this type of data than in theorems and proofs, or formal statements of distributions and such.  Thanks&lt;/p&gt;&#xA;&#xA;&lt;p&gt;UPDATE  I have got the Mardia text, which is quite comprehensive.  After some more reading, I may be back with more questions.&lt;/p&gt;&#xA;" OwnerUserId="686" LastEditorUserId="686" LastEditorDisplayName="" LastEditDate="2011-03-05T11:29:56.993" LastActivityDate="2011-03-05T11:29:56.993" Title="Logistic regression with directional data as IV" Tags="&lt;directional-statistics&gt;" AnswerCount="1" />
  <row Id="7786" PostTypeId="1" CreationDate="2011-03-02T11:19:11.983" Score="2" ViewCount="60" Body="&lt;p&gt;I am looking for some suggestions about assessing the representativeness of a particular dataset I am analyzing. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this dataset I am looking at the relationship between two variables (e.g., X and Y) in a population that is split into five distinct blocks. The main problem is that the data is based upon reports from the public, so some blocks have much more data than others.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The goal is to assess whether the relationship between X and Y differs between the blocks, but also to determine how reliable such estimates are given that we do not have a truly random sample of the overall population.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestions appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks &lt;/p&gt;&#xA;" OwnerUserId="3136" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-03-02T11:41:04.033" LastActivityDate="2011-03-02T11:41:04.033" Title="Assessing the representativeness of population sampling" Tags="&lt;sampling&gt;&lt;survey&gt;&lt;dataset&gt;&lt;resampling&gt;" CommentCount="5" />
  <row Id="7787" PostTypeId="2" ParentId="7768" CreationDate="2011-03-02T11:48:13.157" Score="4" ViewCount="" Body="&lt;p&gt;For a logistic regression fitted by maximum likelihood, as long as you have both (1) and (2) in the model, then no matter what &quot;default&quot; value that you give new runners for (2), the estimate for (1) will adjust accordingly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, let $X_1$ be the indicator variable for &quot;is a new runner&quot;, and $X_2$ be the variable &quot;previous laptime in seconds&quot;. Then the linear predictor is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\eta = \alpha + \beta_1 X_1 + \beta_2 X_2 + \ldots$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the default for $X_2$ is zero, then the linear predictor for a new runner is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\eta = \alpha + \beta_1 + \ldots$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;whereas for an existing runner, it will be:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\eta = \alpha + \beta_2 X_2 + \ldots$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now suppose that you change the default for $X_2$ from 0 to -99. Then the linear predictor for a new runner is now:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\eta = \alpha + \beta&amp;#39;_1 - 99 \beta_2 + \ldots$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;but for an existing runner, it will remain the same. So all you've done is reparameterise the model, such that $\beta&amp;#39;_1 - 99 \beta_2 = \beta_1$, and since maximum likelihood is paremeterisation invariant, the estimates will adjust accordingly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, if you're not using maximum likelihood (i.e. you're using some sort of penalisation or prior on the parameters), then you're going to get different values unless you adjust the penalisation/prior accordingly. And if the model is non-linear (e.g. SVM, NN &amp;amp; Decision trees), then this argument doesn't work at all.&lt;/p&gt;&#xA;" OwnerUserId="495" LastActivityDate="2011-03-02T11:48:13.157" />
  <row Id="7788" PostTypeId="1" AcceptedAnswerId="7793" CreationDate="2011-03-02T11:54:29.660" Score="4" ViewCount="68" Body="&lt;p&gt;Some edits made...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a dataset which other researchers have used mixed effects modelling with to come up with a nice set of associations. I also have a much smaller dataset which is the same variables but from a different country. The first dataset is plenty powerful enough (350 individuals from 30 locations) but the second dataset is not very powerful at all (70 individuals from 15 locations).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there some way to take what I know about the first data set and use it to boost the power of the second dataset? I've pooled all the data and analysed it and replicated the results, but I fear that is because the first dataset is just swamping the second one. So far I've come up with:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Analyse first dataset, then pool, analyse again, and compare AIC to see if it gets worse (seems a bit crude).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Analyse both datasets using a dummy variable for first/ second dataset (they're from different countries). This seems like a good idea but I'm not convinced it would eliminate the whole problem of the first swamping the second.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) It sounds a bit like Bayesian statistics, but I don't know the first thing about Bayesian statistics, so if it is this then I'd better hit the library.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any and all help as ever gratefully received!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Clarifications:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) dependent variable same for both countries&#xA;2) oh dear, not sure I understand this question. All the variables are the same, so I have a list of their data, with a country code, and then the same variables again, with a different country code. All observations are individuals clustered in wards clustered in countries. Does this answer?&#xA;3) Really I'm just trying to replicate the significant linear associations which they found (or, I guess, highlight where they are different). So e.g. they found that size of ward improved outcomes, we would like to replicate that, along with some other associations.&lt;/p&gt;&#xA;" OwnerUserId="199" LastEditorUserId="199" LastEditorDisplayName="" LastEditDate="2011-03-03T09:46:22.250" LastActivityDate="2011-03-03T09:46:22.250" Title="How to replicate large well powered mixed effects model with a smaller sample?" Tags="&lt;bayesian&gt;&lt;mixed-model&gt;&lt;power&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="7789" PostTypeId="2" ParentId="7785" CreationDate="2011-03-02T11:55:26.133" Score="5" ViewCount="" Body="&lt;p&gt;I would suggest applying a transform which deals with periodicity. i.e. $\lim_{x \to 360} f(x) = f(0)$. An easy option is to take the sin and cos, and put them both as covariates in the model. &lt;/p&gt;&#xA;" OwnerUserId="495" LastActivityDate="2011-03-02T11:55:26.133" CommentCount="1" />
  <row Id="7790" PostTypeId="1" AcceptedAnswerId="7807" CreationDate="2011-03-02T12:03:36.580" Score="2" ViewCount="152" Body="&lt;p&gt;I have two exclusive groups of people and a counter of how many events happened for each group.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lets say group 1 has 7000 people and group 2 has 3000 people. &#xA;group 1 had 50 events and group 2 had 40 events.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm calculating the event percentage for each group for example for group1 its 50/7000. for group 2 its 40/3000.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to calculate how much statistical validity these results have (in other words is the groups large enough or I need to collect more data). Probably in percentage (where &gt;95% means its valid statistically)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone point me to how to do it. I need to implement it in PHP code. I have little statistics knowledge. I think it involves square chi function but I'm not sure how to use the data with PHP chi square function &lt;a href=&quot;http://www.php.net/manual/en/function.stats-cdf-chisquare.php&quot; rel=&quot;nofollow&quot;&gt;http://www.php.net/manual/en/function.stats-cdf-chisquare.php&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ADDITIONAL INFO: &#xA;We're talking about visitors to a web store. I divide them randomly to 30% group B (test group) and 70% group A. I expose gruop A to a certain message.&#xA;I compare the conversion rates of the groups (% of visitors who buy something). And I want to know when the samples are large enough to be statistically significant.&lt;/p&gt;&#xA;" OwnerUserId="3506" LastEditorUserId="3506" LastEditorDisplayName="" LastEditDate="2011-03-02T13:32:53.977" LastActivityDate="2011-03-02T17:14:47.650" Title="How to determine statistical validity of results" Tags="&lt;statistical-significance&gt;&lt;chi-squared&gt;" AnswerCount="1" CommentCount="4" FavoriteCount="1" />
  <row Id="7791" PostTypeId="1" AcceptedAnswerId="7808" CreationDate="2011-03-02T12:04:56.220" Score="8" ViewCount="111" Body="&lt;p&gt;Some edits made...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This question is just for fun, so if it isn't fun then please feel free to ignore it. I already get a lot of help from this site so I don't want to bite the hand that feeds me. It's based on a real life example and it's just something I've wondered about a lot.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I visit my local dojo to train on an essentially random basis Monday-Friday. Let's assume I visit twice a week. This means I visit exactly twice, every week, with only the two days varying. There is one individual who is nearly always there whenever I am there. If he visits on the same day as me then I will see him. Let's assume he's there 90% of the time when I'm there. I want to know two things:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) how often he trains&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) whether he comes on a random basis or on set days of the week.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm guessing perhaps we have to assume one to guess the other? I've really got nowhere with this at all. I just think about it in the warm-up every week and am baffled anew. Even if somebody gave me a way in to think about the problem I would be most grateful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cheers!&lt;/p&gt;&#xA;" OwnerUserId="199" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-02T17:09:01.147" LastActivityDate="2011-03-02T17:09:01.147" Title="Can I estimate the frequency of an event based on random samplings of its occurrence?" Tags="&lt;probability&gt;&lt;estimation&gt;&lt;sampling&gt;" AnswerCount="1" CommentCount="6" FavoriteCount="0" />
  <row Id="7792" PostTypeId="2" ParentId="168" CreationDate="2011-03-02T12:11:19.977" Score="0" ViewCount="" Body="&lt;p&gt;For univariate kernel density estimation, the bandwidth can be estimated by Normal reference rule or Cross Validation method or plug-in approach. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For multivariate kernel density estimation, a Bayesian bandwidth selection method may be utilized, see Zhang, X., M.L. King and R.J. Hyndman (2006), A Bayesian approach to bandwidth selection for multivariate kernel density estimation, Computational Statistics and Data Analysis, 50, 3009-3031&lt;/p&gt;&#xA;" OwnerUserId="3507" LastActivityDate="2011-03-02T12:11:19.977" />
  <row Id="7793" PostTypeId="2" ParentId="7788" CreationDate="2011-03-02T12:20:21.893" Score="2" ViewCount="" Body="&lt;p&gt;Sounds like what you want is a hierarchical model: you would have a country-level effect, and nested within that would be a location-level effect. I've only ever skimmed through it, but the Gelman and Hill book seems to be quite highly regarded in this area:&#xA;&lt;a href=&quot;http://www.stat.columbia.edu/~gelman/arm/&quot; rel=&quot;nofollow&quot;&gt;http://www.stat.columbia.edu/~gelman/arm/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for whether this sounds Bayesian: this is one area where the distinction between Bayesian and non-Bayesian techniques becomes very blurred. Personally, I find it easier to think of these models in Bayesian terms, because then you don't need to get bogged down in making the distinction between fixed and random effects.&lt;/p&gt;&#xA;" OwnerUserId="495" LastActivityDate="2011-03-02T12:20:21.893" />
  <row Id="7795" PostTypeId="1" CreationDate="2011-03-02T13:16:19.200" Score="2" ViewCount="166" Body="&lt;p&gt;I would like to find out the values &lt;code&gt;(x,y)&lt;/code&gt; used in plotting &lt;code&gt;plot(b,seWithMean = TRUE)&lt;/code&gt; in &lt;strong&gt;mgcv&lt;/strong&gt; package.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone know how I can extract or compute these values?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an example &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(mgcv) &#xA;set.seed(0)&#xA;dat&amp;lt;-gamSim(1,n=400,dist=&quot;normal&quot;,scale=2) &#xA;b&amp;lt;-gam(y~s(x0),data=dat) &#xA;plot(b,seWithMean = TRUE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Thank you for your help.&lt;/p&gt;&#xA;" OwnerUserId="2264" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-02T22:48:27.533" LastActivityDate="2011-03-02T22:48:27.533" Title="How to obtain the values used in plot.gam in mgcv?" Tags="&lt;r&gt;&lt;time-series&gt;&lt;smoothing&gt;&lt;mgcv&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="7796" PostTypeId="2" ParentId="7774" CreationDate="2011-03-02T13:50:29.307" Score="1" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Important Note&lt;/strong&gt;: I've decided to remove the proof I gave originally in this answer. It was longer, more computational, used bigger hammers, and proved a weaker result as compared to the other proof I've given. All around, an inferior approach (in my view). If you're &lt;em&gt;really&lt;/em&gt; interested, I suppose you can look at the edits.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The asymptotic results that I originally quoted and which are still found below in this answer do show that as $n \to \infty$ we can do a bit better than the bound proved in the other answer, which holds for &lt;em&gt;all&lt;/em&gt; $n$.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;The following &lt;em&gt;asymptotic&lt;/em&gt; results hold&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;\mathbb{P}(T &amp;gt; n \log n + c n ) \to 1 - e^{-e^{-c}}&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;\mathbb{P}(T \leq n \log n - c n ) \to e^{-e^c} \&amp;gt;.&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The constant $c \in \mathbb{R}$ and the limits are taken as $n \to \infty$. Note that, though they're separated into two results, they're pretty much the same result since $c$ is not constrained to be nonnegative in either case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See, e.g., Motwani and Raghavan, &lt;em&gt;Randomized Algorithms&lt;/em&gt;, pp. 60--63 for a proof.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Also&lt;/strong&gt;: David kindly provides a proof for his stated upper bound in the comments to this answer.&lt;/p&gt;&#xA;" OwnerUserId="2970" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-03-07T13:01:34.917" LastActivityDate="2011-03-07T13:01:34.917" CommentCount="5" />
  <row Id="7797" PostTypeId="1" AcceptedAnswerId="7798" CreationDate="2011-03-02T13:58:38.210" Score="6" ViewCount="38" Body="&lt;p&gt;This question might be way off base as I am just getting to know vector autoregressive models, but I've tried searching through the usual channels and if this actually is a valid question it might be of help to others that it is here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should I think in the same way about sample size for a VAR-model as for any other regression method? That is, calculate the degrees of freedom that will be used and work from there to establish what a suitable number of observations would be, or is there something else to take into account?&lt;/p&gt;&#xA;" OwnerUserId="2105" LastActivityDate="2011-03-02T14:15:22.457" Title="Is there a method to find what is a good sample size for a VAR-model?" Tags="&lt;time-series&gt;&lt;multivariate-analysis&gt;&lt;sample-size&gt;&lt;econometrics&gt;" AnswerCount="1" />
  <row Id="7798" PostTypeId="2" ParentId="7797" CreationDate="2011-03-02T14:15:22.457" Score="4" ViewCount="" Body="&lt;p&gt;The usual way of estimating VAR(p) model for $n$-dimensional vector $X_t=(X_{1t},...,X_{nt})$ is using OLS for each equation:&#xA;\begin{align}&#xA;X_{it}=\theta_{0i}+\sum_{s=1}^p\sum_{j=1}^n\theta_{sj}X_{j,t-s}+\varepsilon_{it}&#xA;\end{align}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the answer is yes. For each equation you will need to estimate $np+1$ parameters, so the sample size is chosen as in usual regression with $np+1$ independent variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Things you will need to take into account is whether additional exogenous variables are included and what type of covariance matrix for $\varepsilon_t=(\varepsilon_{t1},...,\varepsilon_{tn})$ you intend to postulate. This will increase the number of parameters to be estimated. &lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-03-02T14:15:22.457" />
  <row Id="7799" PostTypeId="1" AcceptedAnswerId="7803" CreationDate="2011-03-02T14:18:10.247" Score="3" ViewCount="320" Body="&lt;p&gt;I am a programmer who, in a past life, used the products RS/1, RS/Explore, and RS/Discover in a manufacturing engineer career.  Now, years later, I would like to do some multivariate regression on some real-world data (sales data from my wife's store).  The point would be to highlight what sales days are truly exceptional, not just part of the normal distribution, after taking into account month, day of week, weather, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I still had access to RS/1 and associated products, I'd know how to do this, but that is not the case, so I'm thinking I'd want to use R.  However, most R tutorials I have found just cover the very basics, and don't get to the point of multivariate regression.  Can you recommend an R tutorial that takes one past the basics of plotting a histogram, etc. into in-depth analysis of real-world ad-hoc data, presumably using multi-variate regression?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="2917" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-03-02T14:20:51.750" LastActivityDate="2011-03-02T20:06:24.910" Title="Tutorial for using R to do multivariate regression?" Tags="&lt;r&gt;&lt;data-visualization&gt;&lt;multivariate-analysis&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="7800" PostTypeId="2" ParentId="7782" CreationDate="2011-03-02T14:30:59.527" Score="3" ViewCount="" Body="&lt;p&gt;I would probably do the following:&#xA;1) Split the data into two roughly equal segments.&#xA;2) Perform exploratory analyses on one of these and derive a new model&#xA;3) Test the model on the other half of the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This at least will be something that's not done all that often, which will make it a better fit for publication (should you want to do so), and will give you an independent test of your model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could also fit both models (the prior one and the one you develop) to your test data, and compare the fit of both.  &lt;/p&gt;&#xA;" OwnerUserId="656" LastActivityDate="2011-03-02T14:30:59.527" />
  <row Id="7801" PostTypeId="2" ParentId="7795" CreationDate="2011-03-02T14:45:37.597" Score="0" ViewCount="" Body="&lt;p&gt;This will not be a complete answer. All the plotting for &lt;code&gt;gam&lt;/code&gt; objects is being done with function &lt;code&gt;plot.gam&lt;/code&gt;. You can look at its code by simply typing&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; plot.gam&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;in R console. As you will see the code is huge. What I've gleaned from it, that all the plotting is done by gathering relevant information in &lt;code&gt;pd&lt;/code&gt; object which is a list. So one of the possible solution would be to edit &lt;code&gt;plot.gam&lt;/code&gt;, using &lt;code&gt;edit&lt;/code&gt; for example, so that it returns that object. Adding &lt;code&gt;pd&lt;/code&gt; before last &lt;code&gt;}&lt;/code&gt; will be enough. I would advise adding &lt;code&gt;invisible(pd)&lt;/code&gt;, so that this object is returned only if you ask for it:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; pd &amp;lt;- plot(b,seWithMean = TRUE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then inspect this object and search in the code of &lt;code&gt;plot.gam&lt;/code&gt; for the lines with &lt;code&gt;plot&lt;/code&gt; and &lt;code&gt;lines&lt;/code&gt;. Then you will see which of the relevant &lt;code&gt;x&lt;/code&gt; and &lt;code&gt;y&lt;/code&gt; values appear in the plot.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-03-02T14:45:37.597" CommentCount="2" />
  <row Id="7803" PostTypeId="2" ParentId="7799" CreationDate="2011-03-02T14:49:04.513" Score="6" ViewCount="" Body="&lt;p&gt;This is my favorite one: &lt;a href=&quot;http://www.statmethods.net/stats/regression.html&quot; rel=&quot;nofollow&quot;&gt;Quick-R&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1499" LastActivityDate="2011-03-02T14:49:04.513" CommentCount="2" />
  <row Id="7804" PostTypeId="2" ParentId="7768" CreationDate="2011-03-02T15:01:25.117" Score="3" ViewCount="" Body="&lt;p&gt;Instead of assigning special value for non-existent first time runner previous lap time, simply use interaction term for previous lap time with the inverse of first time runner dummy:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Y_i=\beta_0+\beta_1 FTR_i+\beta_2 (NFTR_i)\times PLT_i+...$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;here &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;$Y_i$ is your input variable,  &lt;/li&gt;&#xA;&lt;li&gt;$...$ is your other variables,   &lt;/li&gt;&#xA;&lt;li&gt;$FTR_i$ is&#xA;dummy for the first time runner, &lt;/li&gt;&#xA;&lt;li&gt;$PLT_i$ is the previous lap time and &lt;/li&gt;&#xA;&lt;li&gt;$NFTR_i$ is dummy for non first time&#xA;runner equaling 1, when $FTR_i=0$ and&#xA;0 otherwise.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Then the model for first time runners will be:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Y_i=(\beta_0+\beta_1) + ...$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and for non first time runners:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Y_i=\beta_0+ \beta_2 PLT_i + ...$$&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-03-02T15:01:25.117" />
  <row Id="7805" PostTypeId="1" AcceptedAnswerId="7812" CreationDate="2011-03-02T15:03:56.520" Score="5" ViewCount="59" Body="&lt;p&gt;Let's say you have a jointly gaussian vector random variable $\mathbf{x}$, with mean $\mathbf{M}$ and covariance $\mathbf{S}$. I now transform each scalar element of  $\mathbf{x}$ , say $x_j$, with a sigmoid:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$y_j = 1/(1+\exp(-x_j))$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am interested in the expectation between two variables $y_j$ and $y_j&amp;#39;$ of the resulting distribution, that is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$E\{y_j y_j&amp;#39;\}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $E\{\}$ is the expectation operator. Note:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;The whole PDF of $y$ can be computed just by applying a change of variable. Unfortunately the integrals leading to the expectations are intractable (to the best of my knowledge).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I'm looking for closed-form approximations, no Markov Chain-Monte Carlo, no variational stuff. That is, approximations to the variable change, to the expectation integrals, to the sigmoid, to the resulting PDF, that allow computing $E\{y_j y_j&amp;#39;\}$. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Some dead ends: Taylor, often used in papers on the topic, is inaccurate by the mile. &lt;a href=&quot;http://www.mathtable.com/gr/&quot; rel=&quot;nofollow&quot;&gt;Gradshteyn and Ryzhik&lt;/a&gt; does not seem to contain the integrals.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="3509" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-02T16:00:10.463" LastActivityDate="2011-03-02T19:39:10.920" Title="Correlation between two nodes of a single layer MLP for joint-Gaussian input" Tags="&lt;machine-learning&gt;&lt;bayesian&gt;&lt;normal-distribution&gt;&lt;data-transformation&gt;&lt;neural-networks&gt;" AnswerCount="1" CommentCount="7" FavoriteCount="1" />
  <row Id="7806" PostTypeId="2" ParentId="7795" CreationDate="2011-03-02T15:15:49.453" Score="3" ViewCount="" Body="&lt;p&gt;I've repeatedly cursed the fact that the plot functions for &lt;code&gt;mgcv&lt;/code&gt; don't return the stuff they are plotting -- what follows is ugly but it works:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(mgcv) &#xA;set.seed(0)&#xA;dat&amp;lt;-gamSim(1,n=400,dist=&quot;normal&quot;,scale=2) &#xA;b&amp;lt;-gam(y~s(x0),data=dat) &#xA;plot(b,seWithMean = TRUE)&#xA;&#xA;&#xA;plotData &amp;lt;- list()&#xA;trace(mgcv:::plot.gam, at=list(c(24, 3,4,3)), &#xA;        #this gets you to the location where plot.gam calls plot.mgcv.smooth (see ?trace)&#xA;        #plot.mgcv.smooth is the function that does the actual plotting and&#xA;        #we simply assign its main argument into the global workspace&#xA;        #so we can work with it later.....&#xA;        quote({&#xA;                    #browser()&#xA;                    plotData &amp;lt;&amp;lt;- c(plotData, pd[[i]])&#xA;                }))&#xA;&#xA;#test: &#xA;mgcv::plot.gam(b,seWithMean = TRUE)&#xA;x11()&#xA;plot(plotData$x, plotData$fit, type=&quot;l&quot;, xlim=plotData$xlim, &#xA;        ylim=range(plotData$fit+ plotData$se, plotData$fit - plotData$se))&#xA;matlines(plotData$x, &#xA;        cbind(plotData$fit+ plotData$se, plotData$fit - plotData$se), lty=2, col=1)&#xA;rug(plotData$raw)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1979" LastActivityDate="2011-03-02T15:15:49.453" CommentCount="3" />
  <row Id="7807" PostTypeId="2" ParentId="7790" CreationDate="2011-03-02T15:20:12.543" Score="3" ViewCount="" Body="&lt;p&gt;Hello,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You are calculating the mean of a variable that is 0 if no event and 1 if there is an event. The sum of $N$ such (independent) binomial random variables has a variance $N\times p(1-p)$. The mean has a variance $p(1-p)/N$. We can use a two-sample difference in means test to see whether the difference in proportions between the groups is significant. Calculate:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\begin{equation*}\frac{p - q}{\sqrt{p(1-p)/N + q(1-q)/M}}\end{equation*}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $p$ is the proportion from group 1, which has $N$ observations, and $q$ is the proportion from group 2 with $M$ observations. If this number is large in absolute value (bigger than 1.96 is a typical norm, giving a hypothesis test with a significance level of 5%), then you can reject the claim that the two groups have the same proportion of events. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This assumes that each person in group 1 has the same probability of having an event and each person in group 2 has the same probability of event, but these probabilities can differ across groups. Since you are randomly assigning people to the groups (e.g., they aren't self-selecting into them), this is a reasonably good assumption.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unfortunately, I can't help with you PHP coding, but I hope that this gets you started.&lt;/p&gt;&#xA;" OwnerUserId="401" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-02T17:14:47.650" LastActivityDate="2011-03-02T17:14:47.650" />
  <row Id="7808" PostTypeId="2" ParentId="7791" CreationDate="2011-03-02T17:06:53.537" Score="5" ViewCount="" Body="&lt;p&gt;Your data will give partial answers by means of the Hansen-Hurwitz or &lt;a href=&quot;http://www.math.umt.edu/patterson/549/Horvitz-Thompson.pdf&quot; rel=&quot;nofollow&quot;&gt;Horvitz-Thompson&lt;/a&gt; estimators.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The model is this: represent this individual's attendance as a sequence of indicator (0/1) variables $(q_i)$, $i=1, 2, \ldots$.  You randomly observe a two-element subset out of each weekly block $(q_{5k+1}, q_{5k+2}, \ldots, q_{5k+5})$.  (This is a form of systematic sampling.)&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;How often does he train&lt;/strong&gt;?  You want to estimate the weekly mean of the $q_i$.  The statistics you gather tell you the mean observation is 0.9.  Let's suppose this was collected over $w$ weeks.  Then the Horvitz-Thompson estimator of the total number of the individual's visits is $\sum{\frac{q_i}{\pi_i}}$ = ${5\over2} \sum{q_i}$ = ${5\over2} (2 w) 0.9$ = $4.5 w$ (where $\pi_i$ is the chance of observing $q_i$ and the sum is over your actual observations.)  That is, &lt;strong&gt;you should estimate he trains 4.5 days per week.&lt;/strong&gt;  See the reference for how to compute the standard error of this estimate.  As an extremely good approximation you can use the usual (Binomial) formulas.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Does he train randomly&lt;/strong&gt;?  There is no way to tell.  You would need to maintain totals by day of week.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-03-02T17:06:53.537" />
  <row Id="7809" PostTypeId="1" CreationDate="2011-03-02T17:30:26.333" Score="4" ViewCount="61" Body="&lt;p&gt;I am trying to find a simple similarity metric that will compare vectors of indeterminate length. These vectors will be populated with values between 1 and 5.  In this situation a 1 is closer to a 2 then it is to a 5 etc etc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am new to this type of math. I naively considered using cosine similarity. However, when I looked into this further I realized that this probably wasn't the right metric for this sort of calculation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance and I apologize for the newbie question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: I will be programming this in PHP.&lt;/p&gt;&#xA;" OwnerUserId="1514" LastActivityDate="2011-03-02T19:01:44.663" Title="Simple similarity metric" Tags="&lt;correlation&gt;" AnswerCount="1" />
  <row Id="7810" PostTypeId="1" AcceptedAnswerId="7814" CreationDate="2011-03-02T17:47:10.903" Score="6" ViewCount="86" Body="&lt;p&gt;What is the median of the &lt;a href=&quot;http://en.wikipedia.org/wiki/Noncentral_t-distribution&quot; rel=&quot;nofollow&quot;&gt;non-central t distribution&lt;/a&gt; with non-centrality parameter $\delta \ne 0$? This may be a hopeless question because the CDF appears to be expressed as an infinite sum, and I cannot find any information about the inverse CDF function.&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2011-03-02T19:32:53.217" Title="What is the median of a non-central t distribution?" Tags="&lt;distributions&gt;&lt;median&gt;&lt;non-central&gt;&lt;t-distribution&gt;" AnswerCount="1" />
  <row Id="7811" PostTypeId="2" ParentId="7809" CreationDate="2011-03-02T17:52:27.593" Score="5" ViewCount="" Body="&lt;p&gt;You could just use a norm: given vectors $\mathbf{x}$ and $\mathbf{y}$, we define the &quot;2-norm&quot; by&#xA;$$&#xD;&#xA;||\mathbf{x} - \mathbf{y}||_2 = \sqrt{\sum_i (x_i - y_i)^2}&#xD;&#xA;$$&#xA;Similarly we define the&quot;p-norm&quot; by&#xA;$$&#xD;&#xA;||\mathbf{x} - \mathbf{y}||_p = \left(\sum_i |x_i - y_i|^p\right)^{1/p}&#xD;&#xA;$$&#xA;In the limit, as $p \to \infty$, we get the &quot;infinity norm&quot;&#xA;$$&#xD;&#xA;||\mathbf{x} - \mathbf{y}||_\infty = \max_i |x_i - y_i|&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note the two vectors are assumed to be the same length! If you want your &quot;similarity metric&quot; to be sensibly defined when, say, comparing two vectors of length 3 or two vectors of length 300, you probably have to normalize by the size. That is, you should choose a similarity metric of the form&#xA;$$d(\mathbf{x},\mathbf{y}) = \frac{||\mathbf{x} - \mathbf{y}||_p}{n^{1/p}}$$&#xA;where the vectors are of length $n$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;edit&lt;/strong&gt;: (I changed $d$ above slightly) to turn this distance metric into a similarity metric, I would abuse the fact that the vectors are known to range between $1$ and $5$. This tells me the maximum value that $d(\mathbf{x},\mathbf{y})$ can take occurs when one is a vector of all $1$s and the other is a vector of all $5$s. For the definition of $d$ above, the maximum value it takes is $4$. The proposed similarity metric is then:&#xA;$$f(\mathbf{x},\mathbf{y}) = 4 - \frac{||\mathbf{x} - \mathbf{y}||_p}{n^{1/p}}$$&lt;/p&gt;&#xA;" OwnerUserId="795" LastEditorUserId="795" LastEditorDisplayName="" LastEditDate="2011-03-02T19:01:44.663" LastActivityDate="2011-03-02T19:01:44.663" CommentCount="2" />
  <row Id="7812" PostTypeId="2" ParentId="7805" CreationDate="2011-03-02T17:59:22.200" Score="2" ViewCount="" Body="&lt;p&gt;The question really concerns pairs of normal variates.  Let's call them $x_1$ and $x_2$ with means $\mu_i$, standard deviations $\sigma_i$, and correlation $\rho$.  Whence their joint pdf is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{1}{2 \pi \sqrt{1 - \rho^2} \sigma_1 \sigma_2}&#xD;&#xA;e^{-\frac{1}{1-\rho^2} \left(\frac{(x_1 - \mu_1)^2}{2 \sigma_1^2} + \frac{(x_2 - \mu_2)^2}{2 \sigma_2^2} - \frac{\rho (x_1 - \mu_1)(x_2 - \mu_2)}{\sigma_1 \sigma_2}\right)} dx_1 dx_2\text{.}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let $f(x_1,x_2)$ be the product of this with the $y_i$ (as functions of the $x_i$).  The first component of the gradient of $\log(f)$ is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{\partial \log(f)}{\partial x_1} &#xD;&#xA;= \frac{1}{1 + e^{x_1}} + \frac{\rho(\mu_2 - x_2) \sigma_1 + (x_1 - \mu_1)\sigma_2}{(\rho^2-1)\sigma_1^2 \sigma_2},$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;with a similar expression for the second component (via the symmetry achieved by exchanging the subscripts 1 and 2).  There will be a unique global maximum, which we can detect by setting the gradient to zero.  This pair of nonlinear equations has no closed form solution.  It is rapidly found by a few Newton-Raphson iterations.  Alternatively, we can linearize these equations.  Indeed, through second order, the first component equals&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{1}{2} + x_1\left(\frac{-1}{4} + \frac{1}{(\rho^2-1)\sigma_1^2}\right) + \frac{-\rho x_2 \sigma_1 + \rho \mu_2 \sigma_1 - \mu_1 \sigma_2}{(\rho^2 -1)\sigma_1^2 \sigma_2}.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This gives a pair of &lt;em&gt;linear&lt;/em&gt; equations in $(x_1, x_2)$, which therefore do have a closed form solution, say $\hat{x}_i(\mu_1, \mu_2, \sigma_1, \sigma_2, \rho)$, which obviously are rational polynomials.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Jacobian at this critical point has 1,1 coefficient&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{e^\hat{x_1}\left(2 - (\rho^2-1)\sigma_1^2 + 2\cosh(\hat{x_1})\right)}{(1+e^\hat{x_1})^2(\rho^2-1)\sigma_1^2},$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1,2 and 2,1 coefficients&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{\rho}{\sigma_1 \sigma_2(1 - \rho^2)},$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and 2,2 coefficient obtained from the 1,1 coefficient by symmetry.  Because this is a critical point (at least approximately), we can substitute&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$e^\hat{x_1} = \frac{(\rho^2-1)\sigma_1^2 \sigma_2}{(\mu_2 - \hat{x_2})\rho \sigma_1 + (\hat{x_1} - \mu_1)\sigma_2} - 1$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and use that also to compute $\cosh(\hat{x_1}) = \frac{e^\hat{x_1} - e^{-\hat{x_1}}}{2}$, with a similar manipulation for $e^\hat{x_2}$ and $\cosh(\hat{x_2})$.  This enables evaluation of the Hessian (the determinant of the Jacobian) as a rational function of the parameters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The rest is routine: the Hessian tells us how to approximate the integral as a binormal integral (a &lt;a href=&quot;http://www.questia.com/googleScholar.qst?docId=5001894379&quot; rel=&quot;nofollow&quot;&gt;saddlepoint approximation&lt;/a&gt;).  The answer equals $\frac{1}{2\pi}$ times a rational function of the five parameters: that's your closed form (for what it's worth!).&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-02T19:39:10.920" LastActivityDate="2011-03-02T19:39:10.920" CommentCount="5" />
  <row Id="7813" PostTypeId="1" AcceptedAnswerId="7877" CreationDate="2011-03-02T18:06:19.327" Score="2" ViewCount="149" Body="&lt;p&gt;I am trying to read up about AdaBoost from &lt;a href=&quot;http://www.stanford.edu/~hastie/local.ftp/Springer/ESLII_print5.pdf&quot; rel=&quot;nofollow&quot;&gt;Tibshirani&lt;/a&gt; (page 337 onwards), and would appreciate some help in understanding it better.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The book says that &quot;For each successive iteration m = 2, 3, . . . , M the observation weights are individually modiﬁed and the classiﬁcation algorithm is reapplied to the weighted observations. At step m, those observations that were misclassiﬁed by the classiﬁer Gm−1 (x) induced at the previous step have their weights increased, whereas the weights are decreased for those that were classiﬁed correctly. Thus as iterations proceed, observations that are diﬃcult to classify correctly receive ever-increasing inﬂuence. Each successive classiﬁer is thereby forced to concentrate on those training observations that are missed by previous ones in the sequence.&quot; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not able to understand what it means to &quot;reapply the algorithm to the weighted observations&quot;. Say for example if I am doing 2-class text classification, what am I doing to my observations (document vectors)? How does this &quot;force&quot; the classifier to concentrate on some samples vs others?&lt;/p&gt;&#xA;" OwnerUserId="3301" LastEditorUserId="-1" LastEditorDisplayName="" LastEditDate="2011-03-03T06:12:11.953" LastActivityDate="2011-03-08T07:41:14.860" Title="Adjusting sample weights in AdaBoost" Tags="&lt;machine-learning&gt;&lt;boosting&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="7814" PostTypeId="2" ParentId="7810" CreationDate="2011-03-02T19:00:15.790" Score="7" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;You can approximate it.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, I made the following nonlinear fits for $\nu$ (degrees of freedom) from 1 through 20 and $\delta$ (noncentrality parameter) from 0 through 5 (in steps of 1/2).  Let&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$a(\nu) = 0.963158 + \frac{0.051726}{\nu-0.705428} + 0.0112409\log(\nu),$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$b(\nu) = -0.0214885+\frac{0.406419}{0.659586 +\nu}+0.00531844 \log(\nu),$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$g(\nu, \delta) = \delta + a(\nu) \exp(b(\nu) \delta) - 1.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then $g$ estimates the median to within 0.15 for $\nu=1$, 0.03 for $\nu=2$, .015 for $\nu=3$, and .007 for $\nu = 4, 5, \ldots, 20$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The estimation was done by computing the values of $a$ and $b$ for each value of $\nu$ from 1 through 20 and then separately fitting $a$ and $b$ to $\nu$.  I examined plots of $a$ and $b$ to determine an appropriate functional form for these fits.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can do better by focusing on the intervals of these parameters of interest to you.  In particular, if you're not interested in really small values of $\nu$ you could easily improve these estimates, likely to within 0.005 consistently.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are plots of the median &lt;em&gt;versus&lt;/em&gt; $\delta$ for $\nu=1$, the hardest case, and the negative residuals (true median minus approximate value) &lt;em&gt;versus&lt;/em&gt; $\delta$:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/zlbFe.png&quot; alt=&quot;Non-central t median, delta from 0 to 5, nu=1&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/V7TDH.png&quot; alt=&quot;Non-central t median residuals, delta from 0 to 5, nu=1&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The residuals are truly small compared to the medians.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;BTW, for all but the smallest degrees of freedom the median is close to the noncentrality parameter.  Here's a graph of the median, for $\delta$ from 0 to 5 and $\nu$ (treated as a real parameter) from 1 to 20. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/ILKpP.png&quot; alt=&quot;Non-central t median versus nu and delta (in pseudo 3D)&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For many purposes using $\delta$ to estimate the median might be good enough.  Here is a plot of the error (relative to $\delta$) made by assuming the median equals $\delta$ (for $\nu$ from 2 through 20).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/1pm6e.png&quot; alt=&quot;(Median - delta)/delta versus delta and nu&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-02T19:32:53.217" LastActivityDate="2011-03-02T19:32:53.217" CommentCount="2" />
  <row Id="7815" PostTypeId="1" CreationDate="2011-03-02T19:05:46.350" Score="22" ViewCount="451" Body="&lt;p&gt;Many statistical jobs ask for experience with large scale data. What are the sorts of statistical and computational skills that would be need for working with large data sets. For example, how about building regression models given a data set with 10 million samples?&lt;/p&gt;&#xA;" OwnerUserId="3026" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-03-06T21:39:42.707" LastActivityDate="2011-03-07T00:07:36.283" Title="What skills are required to perform large scale statistical analyses?" Tags="&lt;regression&gt;&lt;machine-learning&gt;&lt;multivariate-analysis&gt;&lt;large-data&gt;" AnswerCount="6" CommentCount="1" FavoriteCount="14" />
  <row Id="7816" PostTypeId="2" ParentId="7815" CreationDate="2011-03-02T19:27:50.943" Score="10" ViewCount="" Body="&lt;p&gt;Your question should yield some good answers.  Here are some starting points.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;An ability to work with the tradeoffs between precision and the demands placed on computing power.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Facility with data mining techniques that can be used as preliminary screening tools before conducting regression.  E.g., chaid, cart, or neural networks.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;A deep understanding of the relationship between statistical significance and practical significance.  A wide repertoire of methods for variable selection.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The instinct to crossvalidate.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="2669" LastActivityDate="2011-03-02T19:27:50.943" CommentCount="4" />
  <row Id="7817" PostTypeId="1" CreationDate="2011-03-02T19:32:52.250" Score="5" ViewCount="89" Body="&lt;p&gt;For a paper, I need an example of a dataset $(x_i,y_i)$ where the residuals are $iid$ (the $x$ do not represent time) with a discontinuity on the effect of $x$ on $y$. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;I have already found a dataset in &lt;a href=&quot;http://www.stat.columbia.edu/~gelman/stuff_for_blog/devinpope.pdf&quot; rel=&quot;nofollow&quot;&gt;Berger and Pope&lt;/a&gt; 2010 but the jump there is not clearly visible&lt;/li&gt;&#xA;&lt;li&gt;The 'LifeCycleSavings' dataset has been suggested, but it has only 50 observations (ideally, I would be looking at 100).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Does anyone know of other datasets?&lt;/p&gt;&#xA;" OwnerUserId="603" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2011-03-02T20:27:52.733" LastActivityDate="2011-03-08T09:15:01.260" Title="Example of discontinous effect of x on y dataset (for paper)" Tags="&lt;dataset&gt;" AnswerCount="1" CommentCount="4" />
  <row Id="7818" PostTypeId="2" ParentId="7799" CreationDate="2011-03-02T20:06:24.910" Score="3" ViewCount="" Body="&lt;p&gt;+1 for Quick-R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another great resource that I (re)turn to regularly is the website of the &lt;a href=&quot;http://www.ats.ucla.edu/stat/R/&quot; rel=&quot;nofollow&quot;&gt;UCLA Statistical Consulting Group&lt;/a&gt;. In particular, it sounds like you might find their &lt;a href=&quot;http://www.ats.ucla.edu/stat/R/dae/default.htm&quot; rel=&quot;nofollow&quot;&gt;data analysis examples&lt;/a&gt; useful. Many of the cases walk through the logic of inquiry and model design steps in addition to providing sample code and datasets. They also have a separate section of &lt;a href=&quot;http://www.ats.ucla.edu/stat/examples/default.htm&quot; rel=&quot;nofollow&quot;&gt;textbook examples and code&lt;/a&gt;, which I have found useful for self-teaching purposes.&lt;/p&gt;&#xA;" OwnerUserId="3396" LastActivityDate="2011-03-02T20:06:24.910" CommentCount="2" />
  <row Id="7819" PostTypeId="2" ParentId="7772" CreationDate="2011-03-02T20:42:51.933" Score="4" ViewCount="" Body="&lt;p&gt;Might we reformulate the question as: 'I have N M-variate observations which I assume to be generated by N corresponding P-variate latent variables i.e. for each case/row M observed numbers are generated by P unobserved numbers.  I have an idea that this mapping is linear with an M x P matrix of coefficients and I want to know what the latent matrix values should be.'?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If that's accurate then you have a multivariate version of the &lt;em&gt;regression calibration problem&lt;/em&gt;.  Normally one knows X and Y and estimates beta, whereas here one knows Y and beta and estimates / 'backs-out' X.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is what motivates suncoolsu's question about control - the question is about what distribution assumptions can be made about the marginal distribution of X (if any).  Your EM idea will make sense if you are happy to make distributional assumptions about P(Y | X; beta) and P(X) to apply Bayes theorem (although you won't need to iterate.) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or maybe that's not the problem you're facing and I just don't understand your description.&lt;/p&gt;&#xA;" OwnerUserId="1739" LastActivityDate="2011-03-02T20:42:51.933" CommentCount="1" />
  <row Id="7820" PostTypeId="2" ParentId="6455" CreationDate="2011-03-02T23:55:45.330" Score="0" ViewCount="" Body="&lt;p&gt;... h should be as small as possible to preserve whatever power the LB test may have under the circumstances. As h increases the power drops.  The LB test is a dreadfully weak test; you must have a lot of samples; n must be ~&gt; 100 to be meaningful. Unfortunately I have never&#xA;seen a better test.  But perhaps one exists.  Anyone know of one ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Paul3nt &lt;/p&gt;&#xA;" OwnerUserId="3516" LastActivityDate="2011-03-02T23:55:45.330" />
  <row Id="7821" PostTypeId="2" ParentId="7815" CreationDate="2011-03-03T02:00:03.717" Score="2" ViewCount="" Body="&lt;ol&gt;&#xA;&lt;li&gt;Framing the problem in the Map-reduce framework.&lt;/li&gt;&#xA;&lt;li&gt;The &lt;em&gt;Engineering&lt;/em&gt; side of the problem, eg., how much does it hurt to use lower precision for the parameters, or model selection based not only on generalization but storage and computation costs as well.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="2728" LastActivityDate="2011-03-03T02:00:03.717" />
  <row Id="7824" PostTypeId="2" ParentId="7542" CreationDate="2011-03-03T05:39:05.227" Score="0" ViewCount="" Body="&lt;p&gt;Perhaps you should randomly permute the callers, then, per your example, claim that the first 163 made 30 calls, the next 163 made 31 calls, &lt;em&gt;etc&lt;/em&gt;. By 'permutation', I mean sort them in a random order. The simple way to do this is to use your random number generator to assign each caller a number, then sort them by that number. &lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2011-03-03T05:39:05.227" />
  <row Id="7825" PostTypeId="1" CreationDate="2011-03-03T09:41:24.050" Score="6" ViewCount="313" Body="&lt;p&gt;I'm looking at the effect defeat and entrapment inducing conditions have on subjective ratings of defeat and entrapment at three different time points (among other things). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However the subjective ratings are not normally distributed. I've done several transformations and the squareroot transformation seems to work best. However  there are still some aspects of the data that have not normalized. This non-normality manifests itself in negative skewness in High entrapment high defeat conditions at the time point I expected there to be the highest defeat and entrapment ratings. Consequently I think it could be argued that this skew is due to the experimental manipulation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would it be acceptable to run ANOVAs on this data despite the lack of normality, given the manipulations? Or would non-parametric tests be more appropriate? If so is there a non parametric equivalent of a 4x3 mixed ANOVA? &lt;/p&gt;&#xA;" OwnerUserId="3520" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-03-03T10:34:07.277" LastActivityDate="2011-03-03T12:27:03.903" Title="How to do ANOVA on data which is still not normal after transformations?" Tags="&lt;nonparametric&gt;&lt;data-transformation&gt;" AnswerCount="3" FavoriteCount="1" />
  <row Id="7826" PostTypeId="1" CreationDate="2011-03-03T09:52:03.557" Score="4" ViewCount="123" Body="&lt;p&gt;I have a dataset about a population in hospital, and what type of infections patients have.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let say the number of patients is 100, 10 of them have pneumonia &lt;code&gt;(group A)&lt;/code&gt;, 20 of them have urinary tract infection &lt;code&gt;(group B)&lt;/code&gt;; keep in mind that group A and B can be overlapping, that is a patient suffering from pneumonia can also have urinary tract infection.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to estimate the prevalence of different infection type in this population (i.e., prevalence of pneumonia, prevalence of urinary tract infection). I'm not sure whether assuming binomial distribution, like the one below from &lt;a href=&quot;http://www.promesa.co.nz/Help/EP_est_simple_random_sample.htm&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;, is appropriate:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\text{SE}=\sqrt{\frac{p\times (1-p)}{n}}\times\sqrt{1-f}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using this formula, I will compute multiple &quot;binomial&quot; estimates (i.e., one for each type of infection). I would feel comfortable to use this if only I need to describe prevalence of one type of infection, but in this case, I need to describe several ones from the same population. I am not sure if using the formula is appropriate or not in this case. Can anyone here enlighten me? Thanks!&lt;/p&gt;&#xA;" OwnerUserId="588" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-03-03T10:08:24.523" LastActivityDate="2011-03-05T13:02:13.523" Title="Computing confidence intervals for prevalence for several types of infection" Tags="&lt;estimation&gt;&lt;confidence-interval&gt;&lt;epidemiology&gt;&lt;prevalence&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="7827" PostTypeId="2" ParentId="7825" CreationDate="2011-03-03T10:47:45.707" Score="3" ViewCount="" Body="&lt;p&gt;I believe with negatively skewed data, you may have to reflect the data to become positively skewed before applying another data transformation (e.g. log or square root). However, this tends to make interpretation of your results difficult.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is your sample size? Depending on how large it is exactly, parametric tests may give fairly good estimates.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Otherwise, for a non-parametric alternative, maybe you can try the &lt;a href=&quot;http://en.wikipedia.org/wiki/Friedman_test&quot; rel=&quot;nofollow&quot;&gt;Friedman test&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In addition, you may try conducting a MANOVA for repeated measures, with an explicit time variable included, as an alternative to a 4x3 Mixed ANOVA. A major difference is that the assumption of sphericity is relaxed (or rather, it is estimated for you), and that all time-points of your outcome variable are fitted at once.&lt;/p&gt;&#xA;" OwnerUserId="3309" LastActivityDate="2011-03-03T10:47:45.707" CommentCount="1" />
  <row Id="7828" PostTypeId="2" ParentId="7825" CreationDate="2011-03-03T11:20:05.713" Score="12" ViewCount="" Body="&lt;p&gt;It's the residuals that should be normally distributed, not the marginal distribution of your response variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would try using transformations, do the ANOVA, and check the residuals. If they look noticeably non-normal regardless of what transformation you use, I would switch to a non-parametric test such as the Friedman test.&lt;/p&gt;&#xA;" OwnerUserId="159" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2011-03-03T11:40:26.200" LastActivityDate="2011-03-03T11:40:26.200" CommentCount="1" />
  <row Id="7829" PostTypeId="1" CreationDate="2011-03-03T11:26:58.357" Score="4" ViewCount="97" Body="&lt;p&gt;I used &lt;a href=&quot;http://en.wikipedia.org/wiki/BIRCH_%28data_clustering%29&quot; rel=&quot;nofollow&quot;&gt;BIRCH&lt;/a&gt; and &lt;a href=&quot;http://en.wikipedia.org/wiki/Cluster_analysis#Agglomerative_hierarchical_clustering&quot; rel=&quot;nofollow&quot;&gt;HAC&lt;/a&gt; to do clustering on my data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to now what type of information I can include in reports that my users can generate to get more insights on the clusters. I would have to dumb down the statistical terms and represent them as much as possible visually in these reports for my users.&lt;/p&gt;&#xA;" OwnerUserId="3523" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-03-03T12:29:01.593" LastActivityDate="2011-03-03T18:01:32.537" Title="How to generate user-friendly summaries of cluster analysis?" Tags="&lt;data-visualization&gt;&lt;clustering&gt;&lt;teaching&gt;" AnswerCount="3" CommentCount="1" />
  <row Id="7830" PostTypeId="2" ParentId="7825" CreationDate="2011-03-03T12:27:03.903" Score="2" ViewCount="" Body="&lt;p&gt;A boxcox transformation (there's one in the MASS package) works as well on negatively as positively skewed data.  FYI, you need to enter a formula in that function like y~1 and make sure all of y is positive first (if it's not just add a constant like abs(min(y))).  You may have to adjust the lambda range in the function to find the peak of the curve.  It will give you the best lambda value to choose and then you just apply this transform:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;b &amp;lt;- boxcox(y~1)&#xA;lambda &amp;lt;- b$x[b$y == max(b$y)]&#xA;yt &amp;lt;- (y^lambda-1)/lambda&#xA;#you can transform back with&#xA;ytb &amp;lt;- (t*lambda+1)^(1/lambda)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;See if your data are normal then.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#you can transform back with&#xA;ytb &amp;lt;- (t*lambda+1)^(1/lambda)&#xA;#maybe put back the min&#xA;ytb &amp;lt;- ytb - abs(min(y))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="601" LastActivityDate="2011-03-03T12:27:03.903" />
  <row Id="7831" PostTypeId="1" CreationDate="2011-03-03T12:43:42.567" Score="5" ViewCount="78" Body="&lt;p&gt;I am working with the two step cluster process in SPSS Modeler (Clementine) and trying to get a sense for the distance function used. It is a log-likelihood function (as stated in docs) but I am unsure for even the continuous variables (the function handles continuous and nominal variable) how this is a log likelihood (it is missing most of the elements of a Gaussian). Below is a screen shot of the documentation describing the distance formula.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Has anyone seen the derivation of this distance function?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/LlSix.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="2040" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-03T12:48:53.830" LastActivityDate="2011-03-03T12:48:53.830" Title="Derivation of distance in two-step clustering" Tags="&lt;clustering&gt;&lt;spss&gt;&lt;distance-functions&gt;" CommentCount="0" />
  <row Id="7832" PostTypeId="2" ParentId="7829" CreationDate="2011-03-03T12:58:25.523" Score="1" ViewCount="" Body="&lt;p&gt;The best method I have found for a non-technical audience is to present a table or plots of the centroids of each cluster along with a description of that cluster. It helps in the business world (not sure your domain) to give a name to each cluster describing it's principle characteristics. Example when clustering customers would be: &quot;Long time loyals&quot; for that cluster that is generally comprised of long tenured customers.&lt;/p&gt;&#xA;" OwnerUserId="2040" LastActivityDate="2011-03-03T12:58:25.523" />
  <row Id="7833" PostTypeId="2" ParentId="7829" CreationDate="2011-03-03T13:43:30.573" Score="2" ViewCount="" Body="&lt;p&gt;Bubble Charts are a good visual device that you can use to represent your cluster.  Pick your 4 most important variables and plot each cluster using the x and y axis, size and color of bubble to represent the 4 factors.  If you have many variables you can perform a principal components analysis first to reduce them to 4 factors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.google.com/images?um=1&amp;amp;hl=en&amp;amp;rlz=1I7GGLD_en&amp;amp;tbs=isch:1&amp;amp;aq=f&amp;amp;aqi=g6&amp;amp;oq=&amp;amp;q=bubble%20chart&quot; rel=&quot;nofollow&quot;&gt;http://www.google.com/images?um=1&amp;amp;hl=en&amp;amp;rlz=1I7GGLD_en&amp;amp;tbs=isch:1&amp;amp;aq=f&amp;amp;aqi=g6&amp;amp;oq=&amp;amp;q=bubble%20chart&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-Ralph Winters&lt;/p&gt;&#xA;" OwnerUserId="3489" LastActivityDate="2011-03-03T13:43:30.573" />
  <row Id="7834" PostTypeId="2" ParentId="6835" CreationDate="2011-03-03T15:26:59.480" Score="1" ViewCount="" Body="&lt;p&gt;I recommend using ARIMA with checks for level shift, trends, and interventions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can't just try and model the data with the only intent of identifying a trend.  It is more complex than just a single focus.  Let me explain....&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to determine if there is a trend, you need to be careful as the trend may be just a change in the intercept or also known as a &quot;level shift&quot;. You also need to be aware that you can't assume the trend started at the beginning of the time series as we are taught in Economics classes so this is an iterative process to determine the beginning and end of the trend. You will also need to determine your threshold of how many periods before you can &quot;call&quot; a trend. You will also need to be adjusting for interventions as they can skew the t-test when trying to identify a trend.&lt;/p&gt;&#xA;" OwnerUserId="3411" LastEditorUserId="3411" LastEditorDisplayName="" LastEditDate="2011-03-03T15:32:02.253" LastActivityDate="2011-03-03T15:32:02.253" />
  <row Id="7835" PostTypeId="1" CreationDate="2011-03-03T15:28:29.620" Score="3" ViewCount="67" Body="&lt;p&gt;I have a bunch of points that belong to one of population P1, P2, ... Pn AND to class A or B.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Within each population I'll be doing classification between A and B, and I want to select features that discriminates the best between A and B. Now, my features are also correlated with population membership, but I don't care about that, I only want to know how well the measures discriminate within a population.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, is there a &quot;proper&quot; way to measure that? Within a population, I can just check how much the metric is correlated with class membership (biserial correlation), but if I get the correlation on the whole population, Simpson's paradox gets in the way and I may end up measuring the wrong thing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I could just calculate correlations for each population, and average those or something, but that sounds ugly to me (this is for proramming real-world applications, not for publishing in a paper, so I don't mind if my method isn't &lt;em&gt;that&lt;/em&gt; rigorous as long as the result is OK).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, what's the proper way to control for population membership?&lt;/p&gt;&#xA;" OwnerUserId="1737" LastActivityDate="2011-03-04T10:29:42.123" Title="Feature selection for classification, controlling for sub-population" Tags="&lt;classification&gt;&lt;feature-selection&gt;&lt;simpsons-paradox&gt;&lt;confounding&gt;" AnswerCount="4" CommentCount="3" />
  <row Id="7836" PostTypeId="1" AcceptedAnswerId="7883" CreationDate="2011-03-03T15:29:29.913" Score="5" ViewCount="116" Body="&lt;p&gt;In regression analysis what's the difference between 'data-generation process' and 'model'? &lt;/p&gt;&#xA;" OwnerUserId="3525" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-03T19:40:41.407" LastActivityDate="2011-03-04T16:48:38.367" Title="In regression analysis what's the difference between data-generation process and model?" Tags="&lt;econometrics&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="7837" PostTypeId="2" ParentId="7836" CreationDate="2011-03-03T15:36:16.930" Score="2" ViewCount="" Body="&lt;p&gt;The DGP is the true model.  The model is what we have tried to, using our best skills, to represent the true state of nature.  The DGP is influenced by &quot;noise&quot;.  Noise can be of many kinds:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;One time interventions&lt;/li&gt;&#xA;&lt;li&gt;Level shifts&lt;/li&gt;&#xA;&lt;li&gt;Trends&lt;/li&gt;&#xA;&lt;li&gt;Changes in Seasonality&lt;/li&gt;&#xA;&lt;li&gt;Changes in Model Parameters&lt;/li&gt;&#xA;&lt;li&gt;Changes in Variance&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;If you don't control for these 6 items than your ability to identify the true DGP is reduced.  &lt;/p&gt;&#xA;" OwnerUserId="3411" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-03T19:40:02.140" LastActivityDate="2011-03-03T19:40:02.140" />
  <row Id="7838" PostTypeId="2" ParentId="7835" CreationDate="2011-03-03T15:40:09.830" Score="0" ViewCount="" Body="&lt;p&gt;A bit more research on Wikipedia indicates that &lt;a href=&quot;http://en.wikipedia.org/wiki/Analysis_of_covariance&quot; rel=&quot;nofollow&quot;&gt;ANCOVA&lt;/a&gt; may be what I need.&lt;/p&gt;&#xA;" OwnerUserId="1737" LastActivityDate="2011-03-03T15:40:09.830" />
  <row Id="7839" PostTypeId="1" CreationDate="2011-03-03T15:43:53.747" Score="1" ViewCount="120" Body="&lt;ol&gt;&#xA;&lt;li&gt;Using either percent error or percent difference, I want to compare one of my measured values from a set to the mean of the set. By reading the Wikipedia article on &lt;a href=&quot;http://en.wikipedia.org/wiki/Percent_difference&quot; rel=&quot;nofollow&quot;&gt;percent difference&lt;/a&gt;, it's still not quite clear which I should choose.&lt;/li&gt;&#xA;&lt;li&gt;What if I wanted to compare an instantaneous slope to the slope of a linear regression, do I use percent error or percent difference?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;If it isn't clear yet, my misunderstanding lies in the comparison of one value from a set to some computed aggregated value from that set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-JP&lt;/p&gt;&#xA;" OwnerUserId="3527" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-03T19:45:16.987" LastActivityDate="2011-03-03T22:12:41.800" Title="Percent error or percent difference? " Tags="&lt;regression&gt;&lt;sampling&gt;&lt;average&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="7840" PostTypeId="2" ParentId="7835" CreationDate="2011-03-03T15:48:13.713" Score="0" ViewCount="" Body="&lt;p&gt;I think that linear discriminant analysis is a better option:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Linear_discriminant_analysis&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Linear_discriminant_analysis&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-Ralph Winters&lt;/p&gt;&#xA;" OwnerUserId="3489" LastActivityDate="2011-03-03T15:48:13.713" />
  <row Id="7842" PostTypeId="2" ParentId="3194" CreationDate="2011-03-03T15:58:17.137" Score="1" ViewCount="" Body="&lt;p&gt;Hello,&#xA;I have thought about a test for a d6 (six sided die). This included finding the number of rolls necessary to test. It is very basic but nevertheless takes long to compute. Have a look at &lt;a href=&quot;http://localtrainbeplac.bplaced.net/die.php&quot; rel=&quot;nofollow&quot;&gt;http://localtrainbeplac.bplaced.net/die.php&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="3529" LastActivityDate="2011-03-03T15:58:17.137" CommentCount="1" />
  <row Id="7843" PostTypeId="2" ParentId="7815" CreationDate="2011-03-03T16:03:30.053" Score="1" ViewCount="" Body="&lt;p&gt;Some good pointers &lt;a href=&quot;http://www.quora.com/How-do-I-become-a-data-scientist&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="22" LastActivityDate="2011-03-03T16:03:30.053" CommentCount="1" />
  <row Id="7844" PostTypeId="2" ParentId="7829" CreationDate="2011-03-03T16:28:00.933" Score="4" ViewCount="" Body="&lt;p&gt;I like a 2D plot that shows the clusters and the actual data points so readers can get an idea of the quality of the clustering. If there are more than two factors, you can put the principle components on the axes, as in my example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/I4Zt9.png&quot; alt=&quot;JMP K-Means 2D Cluster Report&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The equivalent 3D plots are only good if the viewer can interact with it get a sense of depth and obscured pieces. Here's a 3D example with the same data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/Puh45.png&quot; alt=&quot;JMP K-Means 3D Cluster Report&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="1191" LastEditorUserId="1191" LastEditorDisplayName="" LastEditDate="2011-03-03T18:01:32.537" LastActivityDate="2011-03-03T18:01:32.537" CommentCount="2" />
  <row Id="7845" PostTypeId="1" AcceptedAnswerId="7847" CreationDate="2011-03-03T16:28:11.247" Score="9" ViewCount="301" Body="&lt;p&gt;Let's say I have two distributions I want to compare in detail, i.e. in a way that makes shape, scale and shift easily visible.  One good way to do this is to plot a histogram for each distribution, put them on the same X scale, and stack one underneath the other.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When doing this, how should binning be done?  Should both histograms use the same bin boundaries even if one distribution is much more dispersed than the other, as in Image 1 below?  Should binning be done independently for each histogram before zooming, as in Image 2 below?  Is there even a good rule of thumb on this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/Kqzdr.png&quot; alt=&quot;Image 1&quot;&gt;&#xA;&lt;img src=&quot;http://i.imgur.com/BpPwP.png&quot; alt=&quot;Image 2&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="1347" LastActivityDate="2011-03-08T14:43:57.757" Title="Best way to put two histograms on same scale?" Tags="&lt;data-visualization&gt;&lt;histogram&gt;&lt;density-function&gt;&lt;binning&gt;&lt;quantization&gt;" AnswerCount="3" CommentCount="6" />
  <row Id="7846" PostTypeId="2" ParentId="7784" CreationDate="2011-03-03T16:45:19.690" Score="2" ViewCount="" Body="&lt;p&gt;The Pearson correlation coefficient measures linear association.  Being based on empirical second central moments, it is influenced by extreme values.  Therefore:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Evidence of nonlinearity in a scatterplot of actual-vs-predicted values would suggest using an alternative such as the rank correlation (Spearman) coefficient;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;If the relationship looks monotonic on average (as in the upper row of the illustration), a rank correlation coefficient will be effective;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Otherwise, the relationship is curvilinear (as in some examples from the lower row of the illustration, such as the leftmost or the middle u-shaped one) and likely any measure of correlation will be an inadequate description; using a rank correlation coefficient won't fix this.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The presence of outlying data in the scatterplot indicates the Pearson correlation coefficient may be overstating the strength of the linear relationship.  It might or might not be correct; use it with due caution.  The rank correlation coefficient might or might not be better, depending on how trustworthy the outlying values are.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/AcC3d.png&quot; alt=&quot;Examples of scatterplots and their Pearson correlations&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Image copied from the Wikipedia article on &lt;a href=&quot;http://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient&quot; rel=&quot;nofollow&quot;&gt;Pearson product-moment correlation coefficient&lt;/a&gt;.)&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-03-03T16:45:19.690" />
  <row Id="7847" PostTypeId="2" ParentId="7845" CreationDate="2011-03-03T16:55:22.073" Score="6" ViewCount="" Body="&lt;p&gt;I think you need to use the same bins.  Otherwise the mind plays tricks on you. Normal(0,2) looks more dispersed relative to Normal(0,1) in Image #2 than it does in Image #1.  Nothing to do with statistics.  It just looks like Normal(0,1) went on a &quot;diet&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-Ralph Winters&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Midpoint and histogram end points can also alter perception of the dispersion.&#xA;Notice that in this applet a maximum bin selection implies a range of &gt;1.5 - ~5 while a minimum bin selection implies a range of &amp;lt;1 - &gt; 5.5&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.stat.sc.edu/~west/javahtml/Histogram.html&quot; rel=&quot;nofollow&quot;&gt;http://www.stat.sc.edu/~west/javahtml/Histogram.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="3489" LastEditorUserId="3489" LastEditorDisplayName="" LastEditDate="2011-03-08T14:43:57.757" LastActivityDate="2011-03-08T14:43:57.757" CommentCount="8" />
  <row Id="7848" PostTypeId="2" ParentId="2077" CreationDate="2011-03-03T17:46:47.633" Score="4" ViewCount="" Body="&lt;p&gt;Logs and reciprocals and other power transformations often yield unexpected results.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for detrending residuals(ie Tukey), this may have some application in some cases but could be dangerous.  On the other hand, detecting level shifts and trend changes are systematically available to researchers employing intervention detection methods.  Since a level shift is the difference of a time trend just as a pulse is the difference of a level shift the methods employed by Ruey Tsay are easily covered by this problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If a series exhibits level shifts (ie change in intercept) the appropriate remedy to make the series stationary is to &quot;demean&quot; the series.  Box-Jenkins errored critically by assuming that the remedy for non-stationarity was a differencing operators. So, sometimes differencing is appropriate and other times adjusting for the mean shift&quot;s&quot; is appropriate.  In either case, the autocorrelation function can exhibit non-stationarity.  This is a symptom of the state of the series(ie stationary or non-stationary).  In the case of evidented non-stationarity the causes can be different.  For example, the series has truly a continuous varying mean or the series has had a temporary change in mean.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The suggested approach was first proposed Tsay in 1982 and has been added to some software.  Researchers should refer to Tsay's Journal of Forecasting article titled &quot;Outliers,  Level Shifts,  and Variance Changes in Time Series &quot;  , Journal  of Forecasting,  Vol. 7, I-20  (1988).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As usual, textbooks are slow to incorporate leading edge technology, but this material can be referenced in the Wei book (ie Time Series Analysis), Delurgio and Makradakis cover the incorporating interventions, but not how to detect as Wei's text does.&lt;/p&gt;&#xA;" OwnerUserId="3411" LastEditorUserId="3411" LastEditorDisplayName="" LastEditDate="2011-03-03T19:41:27.587" LastActivityDate="2011-03-03T19:41:27.587" CommentCount="1" />
  <row Id="7849" PostTypeId="2" ParentId="7771" CreationDate="2011-03-03T18:32:32.497" Score="1" ViewCount="" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Yes. $D = Z/\sqrt{n}$ for the one-sample test. $D = Z/\sqrt{\frac{n_1 n_2}{n_1 + n_2}}$ for the two-sample test. $D$ should also be the &quot;Most Extreme Differences - Absolute&quot; entry in the output graphic (double-click the table shown in the SPSS output viewer). $Z$ might be labeled &quot;Test Statistic,&quot; &quot;Kolmogorov-Smirnov Z,&quot; or something else depending on which test and version of SPSS you're using. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;It depends. Mann-Whitney tests for a difference in the central tendencies by comparing average ranks; K-S tests for a difference in distributions by comparing the maximum difference in empirical cumulative distribution functions. If you expect strong shape differences, such as only low and high values in one group but middle values for the other group (this would be atypical for most data), K-S is a better choice. If you expect just a location shift, Mann-Whitney is more powerful. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="3408" LastActivityDate="2011-03-03T18:32:32.497" CommentCount="2" />
  <row Id="7850" PostTypeId="1" AcceptedAnswerId="7854" CreationDate="2011-03-03T18:48:35.460" Score="4" ViewCount="159" Body="&lt;p&gt;Hello everyone,&#xA;I working on this data set with marital status and age. I want to plot the percentage of never married man versus each age. Could you please help me to figure out the way how to do it in R? So far I have created two separate arrays with men never marries and ever married. I know how many case of each I have. What i need to do is to count number of people that were never married at each age and divide it by the total number of never married people to get a percentage. I hope I was clear. &#xA;Thank you&lt;/p&gt;&#xA;" OwnerUserId="3008" LastEditorUserId="71" LastEditorDisplayName="" LastEditDate="2011-03-03T19:36:01.193" LastActivityDate="2011-03-03T22:27:22.243" Title="Calculating proportions by age in R" Tags="&lt;r&gt;" AnswerCount="3" CommentCount="1" FavoriteCount="1" />
  <row Id="7851" PostTypeId="2" ParentId="7839" CreationDate="2011-03-03T19:07:46.923" Score="0" ViewCount="" Body="&lt;p&gt;I do not necessary agree with the Wikipedia article, however, of the two,  Percent Error would be a better statistic since you would are essentially treating your average as the  &quot;theoretical&quot; value.  The article seems to state (though not clearly) that &quot;Percent Difference&quot; involves two experimental values x1 and x2, which you do not have.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-Ralph Winters&lt;/p&gt;&#xA;" OwnerUserId="3489" LastActivityDate="2011-03-03T19:07:46.923" CommentCount="2" />
  <row Id="7852" PostTypeId="2" ParentId="7850" CreationDate="2011-03-03T19:29:42.647" Score="2" ViewCount="" Body="&lt;p&gt;Probably what you need is &lt;code&gt;table&lt;/code&gt; or &lt;code&gt;aggregate&lt;/code&gt;. If you add more details I can give you a more in-depth explanation.&lt;/p&gt;&#xA;" OwnerUserId="582" LastActivityDate="2011-03-03T19:29:42.647" />
  <row Id="7853" PostTypeId="1" AcceptedAnswerId="7856" CreationDate="2011-03-03T19:35:51.870" Score="12" ViewCount="210" Body="&lt;p&gt;I understand two-tailed hypothesis testing. You have $H_0 : \theta = \theta_0$ (vs. $H_1 = \neg H_0 : \theta \ne \theta_0$). The p-value is the probability that $\theta$ generates data at least as extreme as what was observed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't understand one-tailed hypothesis testing. Here, $H_0 : \theta\le\theta_0$ (vs. $H_1 = \neg H_0 : \theta &gt; \theta_0$). The definition of p-value shouldn't have changed from above: it should still be the probability that $\theta$ generates data at least as extreme as what was observed. But we don't &lt;em&gt;know&lt;/em&gt; $\theta$, only that it's upper-bounded by $\theta_0$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So instead, I see texts telling us to assume that $\theta = \theta_0$ (not $\theta \le \theta_0$ as per $H_0$) and calculate the probability that this generates data at least as extreme as what was observed, but only on one end. This seems to have nothing to do with the hypotheses, technically.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, I understand that this is frequentist hypothesis testing, and that frequentists place no priors on their $\theta$s. But shouldn't that just mean the hypotheses are then impossible to accept or reject, rather than shoehorning the above calculation into the picture?&lt;/p&gt;&#xA;" OwnerUserId="1720" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-03T20:14:14.630" LastActivityDate="2011-03-04T00:34:19.993" Title="Justification of one-tailed hypothesis testing" Tags="&lt;hypothesis-testing&gt;" AnswerCount="3" CommentCount="1" />
  <row Id="7854" PostTypeId="2" ParentId="7850" CreationDate="2011-03-03T19:44:17.863" Score="8" ViewCount="" Body="&lt;p&gt;Your approach seems way too complicated to me. Let's start with some data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;## make up some data&#xA;status &amp;lt;- factor(rbinom(1000, 1, 0.3), labels = c(&quot;single&quot;, &quot;married&quot;))&#xA;age &amp;lt;- sample(20:50, 1000, replace = TRUE)&#xA;df &amp;lt;- data.frame(status, age)&#xA;head(df)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Print the first six cases:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; head(df)&#xA;   status age&#xA;1 married  21&#xA;2  single  50&#xA;3  single  43&#xA;4  single  28&#xA;5 married  28&#xA;6  single  40&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Next, we need to calculate row wise percentages; even if I doubt that this makes sense (it refers to your statement: &quot;What i need to do is to count number of people that were never married at each age and divide it by the total number of never married people to get a percentage.&quot;).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;## calculate row wise percentages (is that what you are looking for?)&#xA;(tab &amp;lt;- prop.table(table(df), 1)*100)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The resulting table looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; (tab &amp;lt;- prop.table(table(df), 1)*100)&#xA;         age&#xA;status          20       21       22       23       24       25       26&#xA;  single  1.857143 3.142857 3.428571 2.285714 2.142857 2.857143 3.428571&#xA;  married 2.333333 2.333333 5.666667 1.333333 3.333333 5.333333 2.000000&#xA;         age&#xA;status          27       28       29       30       31       32       33&#xA;  single  2.857143 3.142857 3.428571 3.285714 2.714286 3.714286 3.571429&#xA;  married 5.000000 4.333333 2.666667 4.000000 1.666667 4.666667 3.000000&#xA;         age&#xA;status          34       35       36       37       38       39       40&#xA;  single  3.000000 2.857143 5.000000 3.571429 2.857143 3.571429 3.000000&#xA;  married 3.333333 4.000000 4.000000 2.333333 2.000000 2.000000 2.000000&#xA;         age&#xA;status          41       42       43       44       45       46       47&#xA;  single  4.285714 3.000000 3.714286 3.857143 2.857143 3.714286 1.714286&#xA;  married 2.333333 3.333333 2.000000 4.333333 3.666667 5.333333 2.666667&#xA;         age&#xA;status          48       49       50&#xA;  single  2.857143 3.428571 4.857143&#xA;  married 2.333333 3.000000 3.666667&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;That is, if you sum up row wise, it gives 100%&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; sum(tab[1,])&#xA;[1] 100&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Finally, plot it. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;## plot it&#xA;plot(as.numeric(dimnames(tab)$age), tab[1,], &#xA;     xlab = &quot;Age&quot;, ylab = &quot;Single [%]&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/wlZEV.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="307" LastActivityDate="2011-03-03T19:44:17.863" CommentCount="1" />
  <row Id="7855" PostTypeId="2" ParentId="7850" CreationDate="2011-03-03T19:57:08.010" Score="4" ViewCount="" Body="&lt;p&gt;I did something similar recently.  There are quite a few ways to aggregate data like this in R, but the &lt;code&gt;ddply&lt;/code&gt; function from the package &lt;code&gt;plyr&lt;/code&gt; is my security blanket, and I turn to it for things like this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm assuming that you have individual records for each person in your dataset, with age, sex, and marital status.  There's no need to split up the data into multiple tables for this approach - if you have women in the original table, just leave them in and add sex as a grouping variable.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(plyr)&#xA;results.by.age &amp;lt;- ddply(.data = yourdata, .var = c(&quot;sex&quot;, &quot;age&quot;), .fun = function(x) {&#xA;    data.frame(n = nrow(x),&#xA;               ever.married.n = nrow(subset(x, marital.status %in%&#xA;                                            c(&quot;Married&quot;, &quot;Divorced&quot;))),&#xA;               ever.married.prop = nrow(subset(x, marital.status %in%&#xA;                                            c(&quot;Married&quot;, &quot;Divorced&quot;))) / nrow(x)&#xA;               )&#xA;    }&#xA;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This splits the data.frame &lt;code&gt;yourdata&lt;/code&gt; by unique combinations of the variables &lt;code&gt;sex&lt;/code&gt; and &lt;code&gt;age&lt;/code&gt;.  Then, for each of those chunks (referred to as &lt;code&gt;x&lt;/code&gt;), it calculates the number of people who belong to that group (&lt;code&gt;n&lt;/code&gt;), how many of them are married (&lt;code&gt;ever.married.n&lt;/code&gt;), and what proportion of them are married (&lt;code&gt;ever.married.prop&lt;/code&gt;).  It will then return a data.frame called &lt;code&gt;results.by.age&lt;/code&gt; with rows like&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sex       age    n       ever.married.n     ever.married.prop&#xA;&quot;Male&quot;    25     264     167                0.633&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is perhaps not the most elegant or efficient way to do this, but this general pattern has been very helpful for me.  One advantage of this is that you can easily and transparently collect whatever statistics you want from the subset, which can be helpful if you want to, say, add a regression line to the plot (weight by &lt;code&gt;n&lt;/code&gt;) or have both male and female proportions on the same plot and color the points by sex.&lt;/p&gt;&#xA;" OwnerUserId="71" LastEditorUserId="71" LastEditorDisplayName="" LastEditDate="2011-03-03T22:27:22.243" LastActivityDate="2011-03-03T22:27:22.243" />
  <row Id="7856" PostTypeId="2" ParentId="7853" CreationDate="2011-03-03T20:05:43.230" Score="12" ViewCount="" Body="&lt;p&gt;That's a thoughtful question.  Many texts (perhaps for pedagogical reasons) paper over this issue.  What's really going on is that $H_0$ is a &lt;em&gt;composite&lt;/em&gt; &quot;hypothesis&quot; in your one-sided situation: it's actually a set of hypotheses, not a single one.  It is necessary &lt;em&gt;that for every possible hypothesis in&lt;/em&gt; $H_0$, the chance of the test statistic falling in the critical region must be less than or equal to the test size.  Moreover, if the test is actually to achieve its nominal size (which is a good thing for achieving high power), then the supremum of these chances (taken over all the null hypotheses) should equal the nominal size.  In practice, for simple one-parameter tests of location involving certain &quot;nice&quot; families of distributions, this supremum is attained for the hypothesis with parameter $\theta_0$.  Thus, as a practical matter, &lt;em&gt;all computation focuses on this one distribution.&lt;/em&gt;  But we mustn't forget about the rest of the set $H_0$: that is a crucial distinction between two-sided and one-sided tests (and between &quot;simple&quot; and &quot;composite&quot; tests in general).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This subtly influences the interpretation of results of one-sided tests.  When the null is rejected, we can say the evidence points against the true state of nature being any of the distributions in $H_0$.  When the null is not rejected, we can only say &lt;em&gt;there exists&lt;/em&gt; a distribution in $H_0$ which is &quot;consistent&quot; with the observed data.  We are &lt;em&gt;not&lt;/em&gt; saying that &lt;em&gt;all&lt;/em&gt; distributions in $H_0$ are consistent with the data: far from it!  Many of them may yield extremely low likelihoods.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-03-03T20:05:43.230" />
  <row Id="7857" PostTypeId="2" ParentId="7845" CreationDate="2011-03-03T20:23:45.953" Score="0" ViewCount="" Body="&lt;p&gt;So it's a question of maintaining the same bin size or maintaining the same number of bins? I can see arguments for both sides. A work-around would be to &lt;a href=&quot;http://en.wikipedia.org/wiki/Standard_score&quot; rel=&quot;nofollow&quot;&gt;standardize&lt;/a&gt; the values first. Then you could maintain both.&lt;/p&gt;&#xA;" OwnerUserId="1191" LastActivityDate="2011-03-03T20:23:45.953" CommentCount="3" />
  <row Id="7858" PostTypeId="2" ParentId="7839" CreationDate="2011-03-03T20:26:17.933" Score="2" ViewCount="" Body="&lt;p&gt;Well, neither of them. And you are trying to bite it from a wrong side.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Namely, you need some hypothesis -- to what end you are comparing individual measurements to the mean? This is important, since this determines what method you should use; for instance:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;you may have a million normally distributed numbers and 3 outliers and you want to find them -- this way a Z-score may be a good idea;&lt;/li&gt;&#xA;&lt;li&gt;you may have some numbers form unknown distribution and want to find some outliers -- then you should think about some IQR-based Z alternative or some other nonparametric methods;&lt;/li&gt;&#xA;&lt;li&gt;you may want to check if your sample is from a certain distribution -- making a qqplot is a some way to go.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Without it, you will just get 15.3%, 93.4%, 7.532% or some any other number that will be equally useless regardless of being percent error or percent difference.&lt;/p&gt;&#xA;" OwnerUserId="88" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-03-03T22:12:41.800" LastActivityDate="2011-03-03T22:12:41.800" />
  <row Id="7859" PostTypeId="2" ParentId="7835" CreationDate="2011-03-03T21:06:23.557" Score="0" ViewCount="" Body="&lt;p&gt;If you're only classifying within a population, don't worry about it. Just make sure you present your results in separate tables (or charts or whatever), one for each class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're trying to classify observations who's class you do not know, then you need to be more careful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, try a randomForest if you know how to use R, because I love random forests.&lt;/p&gt;&#xA;" OwnerUserId="2817" LastActivityDate="2011-03-03T21:06:23.557" />
  <row Id="7860" PostTypeId="1" AcceptedAnswerId="7862" CreationDate="2011-03-03T22:59:23.757" Score="10" ViewCount="247" Body="&lt;p&gt;Is it possible to visualize the output of Principal Component Analysis in ways that give more insight than just summary tables? Is it possible to do it when the number of observations is large, say ~1e4? And is it possible to do it in R [other environments welcome]?&lt;/p&gt;&#xA;" OwnerUserId="30" LastActivityDate="2011-03-04T15:10:04.247" Title="Visualizing a million, PCA edition" Tags="&lt;r&gt;&lt;data-visualization&gt;&lt;pca&gt;" AnswerCount="3" CommentCount="1" FavoriteCount="5" />
  <row Id="7861" PostTypeId="2" ParentId="7853" CreationDate="2011-03-04T00:15:13.317" Score="1" ViewCount="" Body="&lt;p&gt;I see the p-value as the &lt;em&gt;maximum&lt;/em&gt; probability of a type I error. If $\theta \ll \theta_0$, the probability of a type I error rate may be effectively zero, but so be it. When looking at the test from a minimax perspective, an adversary would never draw from deep in the 'interior' of the null hypothesis anyway, and the power should not be affected. For simple situations (the t-test, for example) it is possible to construct a test with a guaranteed maximum type I rate, allowing such one sided null hypotheses. &lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2011-03-04T00:15:13.317" />
  <row Id="7862" PostTypeId="2" ParentId="7860" CreationDate="2011-03-04T00:28:50.527" Score="9" ViewCount="" Body="&lt;p&gt;The &lt;a href=&quot;http://en.wikipedia.org/wiki/Biplot&quot; rel=&quot;nofollow&quot;&gt;biplot&lt;/a&gt; is a useful tool for visualizing the results of PCA. It allows you to visualize the principal component scores and directions simultaneously.  With 10,000 observations you’ll probably run into a problem with over-plotting.  Alpha blending could help there.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a PC biplot of the &lt;a href=&quot;http://archive.ics.uci.edu/ml/datasets/Wine&quot; rel=&quot;nofollow&quot;&gt;wine data from the UCI ML repository&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/PDYH3.png&quot; alt=&quot;PC Biplot of Wine Data from  UCI ML Repository&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The points correspond to the PC1 and PC2 scores of each observation.&#xA;The arrows represent the correlation of the variables with PC1 and PC2.  The white circle indicates the theoretical maximum extent of the arrows.  The ellipses are 68% data ellipses for each of the 3 wine varieties in the data.&lt;/p&gt;&#xA;" OwnerUserId="1670" LastEditorUserId="1670" LastEditorDisplayName="" LastEditDate="2011-03-04T15:10:04.247" LastActivityDate="2011-03-04T15:10:04.247" CommentCount="4" />
  <row Id="7863" PostTypeId="2" ParentId="7853" CreationDate="2011-03-04T00:34:19.993" Score="1" ViewCount="" Body="&lt;p&gt;You would use a one-sided hypothesis test if only results in one direction are supportive of the conclusion you are trying to reach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Think of this in terms of the question you are asking. Suppose, for example, you want to see whether obesity leads to increased risk of heart attack. You gather your data, which might consist of 10 obese people and 10 non-obese people. Now let's say that, due to unrecorded confounding factors, poor experimental design, or just plain bad luck, you observe that only 2 of the 10 obese people have heart attacks, compared to 8 of the non-obese people.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now if you were to conduct a 2-sided hypothesis test on this data, you would conclude that there was a statistically significant association (p ~ 0.02)between obesity and heart attack risk. However, the association would be in the &lt;em&gt;opposite&lt;/em&gt; direction to that which you were actually expecting to see, hence the test result would be misleading.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(In real life, an experiment that produced such a counterintuitive result could lead to further questions that are interesting in themselves: for example, the data collection process might need to be improved, or there might be previously-unknown risk factors at work, or maybe conventional wisdom is simply mistaken. But these issues aren't really related to the narrow question of what sort of hypothesis test to use.)&lt;/p&gt;&#xA;" OwnerUserId="1569" LastActivityDate="2011-03-04T00:34:19.993" />
  <row Id="7864" PostTypeId="1" CreationDate="2011-03-03T20:37:42.777" Score="0" ViewCount="32" Body="&lt;p&gt;When working with image histograms, I often find my thought stumbling against the lack of exact word to name &quot;intensity element&quot;. When someone needs the analogous term for time domain element, he uses &quot;pixel&quot;, or &quot;sample&quot;. However, algorithms working with histograms usually operate on specific intensity values, like &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;count all values/intensity domain elements/THEWORD of all the pixels in the window&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;What is the word? Does it ever exist?&lt;/p&gt;&#xA;" OwnerUserId="4885" LastActivityDate="2011-03-04T01:15:57.373" ClosedDate="2011-03-04T21:40:11.433" Title="How to call intensity domain elements?" Tags="&lt;signal-processing&gt;&lt;image-processing&gt;" CommentCount="2" />
  <row Id="7865" PostTypeId="1" CreationDate="2011-03-04T02:44:35.870" Score="2" ViewCount="117" Body="&lt;p&gt;How large sample size should be for Ljung-Box statistic to achieve a power $\ge 0.5$ when number of lags tested is 1?  How does the power fall ( assuming an AR(1) process), with increasing number of lags? The LB test is more than 40 years old, surely someone must have answered these questions by now.&lt;/p&gt;&#xA;" OwnerUserId="3533" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-03-04T07:28:34.923" LastActivityDate="2011-03-04T07:43:57.670" Title="What is the power of the Ljung-Box Test for auto-correlation?" Tags="&lt;time-series&gt;&lt;autocorrelation&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="7867" PostTypeId="2" ParentId="7817" CreationDate="2011-03-04T04:29:51.180" Score="2" ViewCount="" Body="&lt;p&gt;Hello,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In economics, this is called &quot;regression discontinuity.&quot; For one example, check out&lt;/p&gt;&#xA;&#xA;&lt;p&gt;David Card &amp;amp; Carlos Dobkin &amp;amp; Nicole Maestas, 2008. &quot;The Impact of Nearly Universal Insurance Coverage on Health Care Utilization: Evidence from Medicare,&quot; American Economic Review, American Economic Association, vol. 98(5), pages 2242-58. [&lt;a href=&quot;http://www.e-aer.org/data/dec08/20051296_data.zip&quot; rel=&quot;nofollow&quot;&gt;data&lt;/a&gt;]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's a summary piece: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lee, David S., and Thomas Lemieux. 2010. &quot;Regression Discontinuity Designs in Economics.&quot; Journal of Economic Literature, 48(2): 281–355. [&lt;a href=&quot;http://www.aeaweb.org/articles.php?doi=10.1257/jel.48.2.281&quot; rel=&quot;nofollow&quot;&gt;article page&lt;/a&gt;]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope that helps.&lt;/p&gt;&#xA;" OwnerUserId="401" LastActivityDate="2011-03-04T04:29:51.180" />
  <row Id="7868" PostTypeId="1" CreationDate="2011-03-04T06:28:00.747" Score="2" ViewCount="102" Body="&lt;p&gt;I have a database of some 10000 cell-numbers with mapping of geographical areas they are associated to. I believe the first 5-6 characters in the numbers can point to the geo. area the cell number belongs to.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to build the rules for such mapping so that geographical area for a new number can be calculated. Can someone point how should I approach the problem?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt;: I am not sure how many digits are specific to geo. location. It is only my guess that 5-6 digits may be involved.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, more than one set of digits can be associated to a particular geo-area. Every operator will have their own digit-set to refer to area and my database has cell-numbers from several operators.&lt;/p&gt;&#xA;" OwnerUserId="1261" LastEditorUserId="1261" LastEditorDisplayName="" LastEditDate="2011-03-07T06:01:22.580" LastActivityDate="2011-03-07T06:01:22.580" Title="Building a classification rule for geographical mapping of cell phone number" Tags="&lt;classification&gt;" AnswerCount="2" CommentCount="5" FavoriteCount="1" />
  <row Id="7869" PostTypeId="1" CreationDate="2011-03-04T06:54:53.380" Score="3" ViewCount="236" Body="&lt;p&gt;&lt;strong&gt;Background:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have the following data (an example):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;headings = { &#xA;         :heading1 =&amp;gt; { :weight =&amp;gt; 25, :views =&amp;gt; 0, :conversions =&amp;gt; 0}&#xA;         :heading2 =&amp;gt; { :weight =&amp;gt; 25, :views =&amp;gt; 0, :conversions =&amp;gt; 0}&#xA;         :heading3 =&amp;gt; { :weight =&amp;gt; 25, :views =&amp;gt; 0, :conversions =&amp;gt; 0}&#xA;         :heading4 =&amp;gt; { :weight =&amp;gt; 25, :views =&amp;gt; 0, :conversions =&amp;gt; 0}&#xA;       }&#xA;total_views = 0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I got to serve these headings based on their weights. Every time a heading is served its &lt;code&gt;views&lt;/code&gt; is incremented by one and &lt;code&gt;total_views&lt;/code&gt; also incremented. And whenever a user clicks on a served heading its &lt;code&gt;conversions&lt;/code&gt; is incremented by one. I've written a program (in Ruby) which is performing this well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to &lt;strong&gt;&lt;code&gt;Auto Optimize&lt;/code&gt;&lt;/strong&gt; best converting heading. Consider the following views and conversions for all headings:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;heading1: views =&amp;gt; 50, conversions =&amp;gt; 30&#xA;heading2: views =&amp;gt; 50, conversions =&amp;gt; 10&#xA;heading3: views =&amp;gt; 50, conversions =&amp;gt; 15&#xA;heading4: views =&amp;gt; 50, conversions =&amp;gt; 5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I need to automatically increase the weights of heading(s) which is/are converting more and vice versa. The sum of weights will always be 100.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any standard algorithm/formula/technique to do this? There might be some other parameters that need to predefined before making these calculations. But I am not getting it through. &lt;/p&gt;&#xA;" OwnerUserId="3535" LastEditorUserId="264" LastEditorDisplayName="" LastEditDate="2011-03-04T21:35:45.670" LastActivityDate="2011-05-17T19:33:04.497" Title="What statistical technique would be appropriate for optimising the weights?" Tags="&lt;optimization&gt;&lt;reinforcement-learning&gt;" AnswerCount="2" CommentCount="6" FavoriteCount="1" />
  <row Id="7870" PostTypeId="2" ParentId="7860" CreationDate="2011-03-04T06:56:34.357" Score="2" ViewCount="" Body="&lt;p&gt;A Wachter plot can help you visualize the eigenvalues of your PCA. It is essentially a Q-Q plot of the eigenvalues against the Marchenko-Pastur distribution. I have an example here:&lt;img src=&quot;http://i.imgur.com/1gkaA.jpg&quot; alt=&quot;Wachter plot showing a single dominant eigenvalue&quot;&gt; There is one dominant eigenvalue which falls outside the Marchenko-Pastur distribution. The usefulness of this kind of plot depends on your application. &lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2011-03-04T06:56:34.357" />
  <row Id="7871" PostTypeId="2" ParentId="7865" CreationDate="2011-03-04T07:43:57.670" Score="3" ViewCount="" Body="&lt;p&gt;A little bit of googling would have answered your question. I am posting this as an answer because &lt;a href=&quot;http://lib.stat.cmu.edu/S/Spoetry/Working/ljungbox.pdf&quot; rel=&quot;nofollow&quot;&gt;the document&lt;/a&gt; answering the question is number 2 in the google search &quot;power of Ljung-Box test&quot; and this question is number 3. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The answer to your question is in graphs Exhibit 14 and Exibit 15 in page 12 of said document. The power depends on the parameter of AR(1) process, the smaller it is, the lower the power, which is perfectly natural.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can perfectly replicate these graphs yourself, here is an &lt;a href=&quot;http://stats.stackexchange.com/questions/7319/two-sample-t-test-for-data-maybe-time-series-with-autocorrelation/7331#7331&quot;&gt;example of power analysis&lt;/a&gt; for the AR(1) processes of another test, I think it is not that hard to adapt it for Ljung-Box test. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for the literature the usual time series books do not mention or mention the power of Ljung-Box test briefly. This means that purely theoretical results are probably not available.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-03-04T07:43:57.670" />
  <row Id="7872" PostTypeId="2" ParentId="7860" CreationDate="2011-03-04T08:37:56.817" Score="0" ViewCount="" Body="&lt;p&gt;You could also use the psych package.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This contains a plot.factor method, which will plot the different components against one another in the style of a scatterplot matrix.  &lt;/p&gt;&#xA;" OwnerUserId="656" LastActivityDate="2011-03-04T08:37:56.817" />
  <row Id="7873" PostTypeId="1" CreationDate="2011-03-04T10:09:32.567" Score="7" ViewCount="149" Body="&lt;p&gt;Hi everyone,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Today I have got a question about binomial/ logistic regression, its based on an analysis that a group in my department have done and were seeking comments upon. I made up the example below to protect their anonymity, but they were keen to see the responses. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Firstly, the analysis began with a simple 1 or 0 binomial response (e.g. survival from one breeding season to the next) and the goal was to model this response as a function of some co-variates. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, multiple measurements of some co-variates were available for some individuals, but not for others. For example, imagine variable x is a measure of metabolic rate during labour and individuals vary in the number of offspring that they have (e.g. variable x was measured 3 times for individual A, but only once for individual B). This imbalance is not due to the sampling strategy of the researchers per se, but reflects the characteristics of the population they were sampling from; some individuals have more offspring than others. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I should also point out that measuring the binomial 0\1 response between labour events was not possible because the interval between these events was quite short. Again, imagine the species in questions has a short breeding season, but can give birth to more than one offspring during the season.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The researchers chose to run a model in which they used the mean of variable x as one covariate and the number of offspring an individual gave birth to as another covariate. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, I was not keen on this approach for a number of reasons&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1)Taking the average of x means losing information in the within-individual variability of x. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) The mean is itself a statistic, so by putting  it in the model we end up doing statistics on statistics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) The number of offspring an individual had is in the model, but it is also used to calculate the mean of variable x, which I think could cause trouble.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, my question is how would people go about modeling this type of data? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;At the moment, I would probably run separate models for individuals that had one offspring, then for individuals that had two offspring etc. Also, I would not use the mean of variable x and just use the raw data for each birth, but I am not convinced this is much better either. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you for your time&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(PS: I apologize that its quite a long question, and I hope that the example is clear)&lt;/p&gt;&#xA;" OwnerUserId="3136" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-29T01:03:49.467" LastActivityDate="2011-04-29T01:03:49.467" Title="Discussing binomial regression and modeling strategies" Tags="&lt;logistic-regression&gt;&lt;modeling&gt;&lt;binomial&gt;&lt;model-selection&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="1" />
  <row Id="7874" PostTypeId="2" ParentId="7835" CreationDate="2011-03-04T10:29:42.123" Score="1" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Regarding &quot;Calculating one measure across all populations&quot;:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to find &lt;strong&gt;one&lt;/strong&gt; set of features so that the discriminative measure is good &lt;strong&gt;for each&lt;/strong&gt; population, then calculating an aggregation of the measures (e.g. avg) is certainly a way to go. Depending on what you want to achieve, you can vary how this aggregation is calculated. Examples:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;All populations are weighted equal =&gt; avg&lt;/li&gt;&#xA;&lt;li&gt;The quality shall be as good as possible for every population, but is not allowed to drop beneath a certain threshold =&gt; min &lt;/li&gt;&#xA;&lt;li&gt;The population with greater size shall be weighted more =&gt; weighted avg with weight = size of population&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Regarding &quot;General Model-Building&quot;:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As the hands-on machine learner that I am, I recommend to &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;merge the populations and calculate a good feature subset (this is what you want to do, if I understood you correctly)&lt;/li&gt;&#xA;&lt;li&gt;merge the populations + add a feature &quot;population&quot;, which contains something like the &quot;population id&quot; and than calculate a good feature subset&lt;/li&gt;&#xA;&lt;li&gt;calculate a feature subset for each population separately&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;This will show you whether the division into subpopulations is really an issue or can be skipped anyways. Sometimes correlations between predictors and population do not influence the correlation between predictors and response.&lt;/p&gt;&#xA;" OwnerUserId="264" LastActivityDate="2011-03-04T10:29:42.123" />
  <row Id="7875" PostTypeId="2" ParentId="7873" CreationDate="2011-03-04T10:55:33.093" Score="2" ViewCount="" Body="&lt;p&gt;I think you could explore a nonlinear mixed model; this should allow you to use the data you have effectively.  But if relatively few subjects have multiple measures, it won't matter much and may not work well (I think there could be convergence problems).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are using SAS you could use PROC GLIMMIX; if using R I think lme4 should be useful.&lt;/p&gt;&#xA;" OwnerUserId="686" LastActivityDate="2011-03-04T10:55:33.093" />
  <row Id="7876" PostTypeId="1" AcceptedAnswerId="7880" CreationDate="2011-03-04T11:01:08.517" Score="4" ViewCount="149" Body="&lt;p&gt;I have trouble figuring out what the range bars in &lt;code&gt;plot.stl&lt;/code&gt; exactly mean. I found Gavin's post on this question and read the documentation as well, I understand that they tell the relative magnitude of the decomposed components, but still I am not entirely sure how they work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;E.g.:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;data: tiny bar, no scale seasonal: full bar, with scale ranging from -0.6 to 0.2 trend: another tiny bar (seems to be equal to data), no scale remainder: medium sized bar with scale from -1.5 to 0.5&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I do not understand what's the basis to relation and why trend has no scale. I tried &lt;code&gt;stl&lt;/code&gt; and &lt;code&gt;decompose&lt;/code&gt; with identical results for multiplicative and additive methods.&lt;/p&gt;&#xA;" OwnerUserId="704" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-03-04T13:31:52.003" LastActivityDate="2011-03-04T13:31:52.003" Title="Interpreting range bars in R's plot.stl?" Tags="&lt;r&gt;&lt;time-series&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="7877" PostTypeId="2" ParentId="7813" CreationDate="2011-03-04T11:46:01.650" Score="3" ViewCount="" Body="&lt;p&gt;mpq has already explained &lt;a href=&quot;http://stats.stackexchange.com/questions/256/how-boosting-works&quot;&gt;boosting in plain english&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A picture may substitute a thousand words ... (stolen from &lt;a href=&quot;http://www.boosting.org/papers/MeiRae03.ps.gz&quot; rel=&quot;nofollow&quot;&gt;R. Meir and G. Rätsch. An introduction to boosting and leveraging&lt;/a&gt;)&#xA;&lt;img src=&quot;http://i.imgur.com/0QPUi.png&quot; alt=&quot;example adaboost&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Image remark:&lt;/strong&gt;  In the 1st Iteration the classifier based on all datapoints classifies all points correctly except those in x&amp;lt;0.2/y&gt;0.8 and the point around 0.4/0.55 (see the circles in the second picture). In the second iteration exactly those points gain a higher weight so that the classifier based on that weighted sample classifies them correctly (2nd Iteration, added dashed line). The combined classifiers (i.e. &quot;the combination of the dashed lines&quot;) result in the classifier represent by the green line. Now the second classifier produces another missclassifications  (x in [0.5,0.6] / y in [0.3,0.4]), which gain more focus in the third iteration and so on and so on. In every step, the combined classifier gets closer and closer to the best shape (although not continuously). The final classifier (i.e. the combination of all single classifiers) in the 100th Iteration classifies all points correctly. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now it should be more clear how boosting works. Two questions remain about the algorithmic details.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;1. How to estimate missclassifications ?&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In every iteration, only a sample of the training data available in this iteration is used for training the classifier, the rest is used for estimating the error / the missclassifications. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;2. How to apply the weights ?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can do this in the following ways:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;You sample the data using an appropriate sample algorithm which can handle weights (e.g. weighted random sampling or rejection sampling) and build classification model on that sample. The resulting sample contains missclassified examples with a higher probability than correctly classified ones, hence the model learned on that sample is forced to concentrate on the missclassified part of the data space. &lt;/li&gt;&#xA;&lt;li&gt;You use a classification model which is capable of handling such weights implicitly like e.g. Decision Trees. DT simply count. So instead of using 1 as counter/increment if an example with a certain predictor and class values is presented, it uses the specified weight w. If w = 0, the example is practically ignored. As a result, the missclassified examples have &lt;em&gt;more influence&lt;/em&gt; on the class probability estimated by the model.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Regarding your document example: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Imagine a certain word separates the classes perfectly, but it only appears in a certain part of the data space (i.e. near the decision boundary). Then the word has no power to separate all documents (hence it's expressiveness for the whole dataset is low), but only those near the boundary (where the expressiveness is high). Hence the documents containing this word will be misclassified in the first iteration(s), and hence gain more focus in later applications. Restricting the dataspace to the boundary (by document weighting), your classifier will/should detect the expressiveness of the word and classify the examples in that subspace correctly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(God help me, I can't think of a more precise example. If the muse later decides to spend some time with me, I will edit my answer).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that boosting assumes weak classifiers. E.g. Boosting applied together with NaiveBayes will have no significant effect (at least regarding my experience).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;edit:&lt;/strong&gt; Added some details on the algorithm and explanation for the picture.&lt;/p&gt;&#xA;" OwnerUserId="264" LastEditorUserId="264" LastEditorDisplayName="" LastEditDate="2011-03-08T07:41:14.860" LastActivityDate="2011-03-08T07:41:14.860" CommentCount="3" />
  <row Id="7878" PostTypeId="2" ParentId="7869" CreationDate="2011-03-04T12:05:12.543" Score="2" ViewCount="" Body="&lt;p&gt;So, if I understand correctly, you are trying to arrive at a weighting scheme that maximizes the amount of total conversions. I am assuming that you are going to reset the conversion values for each heading after each weight change.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A simple solution then could be something like this: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(I'm not familiar with your syntax so hopefully you understand this pseudocode)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;heading1: {ConvPView = conversions / views}   &#xA;heading2: {ConvPView = conversions / views} &#xA;heading3: {ConvPView = conversions / views}&#xA;heading4: {ConvPView = conversions / views}&#xA;&#xA;AvgeConvPView = (h1.cpv + h2.cpv +h3.cpv +h4.cpv) / 4&#xA;&#xA;heading1: {weight = weight + ((ConvPView - AvgeConvPView) * weightChangeConstant)}&#xA;heading2: {etc..}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So, basically, you take a heading's conversions-per-view (conversions / views) and then subtract the average conversions-per-view for all headings. Then you multiply that by a weight-change constant and add the result to the previous weight. Or, in brief:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Weight = weight + (((conversions / views) - (mean of (conversions / views)) * constant)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This will give a negative number if bellow avg and a positive number if above. A larger constant will give you faster modification of weights but will also be more volatile and less able to settle on an optimal weighting distribution.&lt;/p&gt;&#xA;" OwnerUserId="3443" LastEditorUserId="3443" LastEditorDisplayName="" LastEditDate="2011-03-04T13:51:43.177" LastActivityDate="2011-03-04T13:51:43.177" CommentCount="9" />
  <row Id="7879" PostTypeId="2" ParentId="7869" CreationDate="2011-03-04T12:05:25.883" Score="7" ViewCount="" Body="&lt;p&gt;Your problem is a standard problem in the area of Reinforcement Learning and can be reformulated as &lt;a href=&quot;http://en.wikipedia.org/wiki/N_armed_bandith&quot; rel=&quot;nofollow&quot;&gt;n-armed bandit problem&lt;/a&gt;, where you have to find the best arms to pull to optimize the overall gain. In this case one arm = one header and gain is equivalent to 1 (if conversion), else 0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I really recommend to read the book of &lt;a href=&quot;http://webdocs.cs.ualberta.ca/~sutton/book/ebook/the-book.html&quot; rel=&quot;nofollow&quot;&gt;Sutton and Barto&lt;/a&gt;, especially &lt;a href=&quot;http://webdocs.cs.ualberta.ca/~sutton/book/ebook/node14.html&quot; rel=&quot;nofollow&quot;&gt;chapter 2&lt;/a&gt;, where the basic technique to solve the n-armed-bandit problem is explained in detail (how to select, how to increase weights, how to cope with gains changing over time etc.). It is truly a great (and not unnecessarily complicated) read. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt; Here are some detailed explanations as an outline how RL works in the case of the OP. Some parts are rather similar to Matt's answer, but not all.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In Reinforcement Learning (RL) we differentiate between &quot;exploration&quot; and &quot;exploitation&quot;. During exploration we search the space of available action (aka headings to show) to find a better or the best one meanwhile in exploitation we just use the actions/headings for which we already know that they are good. To measure how good an action is, we calculate the reward an action gained when used and hence use this  value to estimate the reward of further usage of this action. In our case the expected mean reward of an action/heading is simply the conversion-rate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some definitions:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$h_i$= Heading i&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Q(h_i)$ = Expected reward function of heading i = $conversion_i / views_i$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How to select an action or heading ?&lt;/strong&gt;&#xA;One option is to select greedily the action with the highest reward / conversionrate estimate. However, we are not able to find new or better solutions this way (no exploration at all). What we actually want is a balance between exploration and exploitation. So we use a procedure called softmax-selection&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$weight(h_i)=\frac{exp(Q(h_i)/\tau)}{\sum_j exp(Q(h_j)/\tau)}$&#xA;(see &lt;a href=&quot;http://webdocs.cs.ualberta.ca/~sutton/book/ebook/node17.html&quot; rel=&quot;nofollow&quot;&gt;softmax-selection in the book of sutton&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Calculate this weights and then select an action randomly with respect to these weights (see e.g. the function sample() in R)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By setting the parameter tau, one can control the balance between exploration and exploitation. If $\tau$ = 0, then we are back to pure greedy action selection, if $\tau$ reaches infinity (or is big enough), all weights become equal and we are restrained to pure random sampling (no exploitation).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How to update the rewards ?&lt;/strong&gt;&#xA;One can use this formula ...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Q_{k+1}(h)=Q_k(h) + \alpha*(r_{k+1}-Q_k(h))$&#xA;(see &lt;a href=&quot;http://webdocs.cs.ualberta.ca/~sutton/book/ebook/node20.html&quot; rel=&quot;nofollow&quot;&gt;see the formula in the book of Sutton&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where k - denotes the k-th time the heading h has been shown and $r_k$= 1 (if during the k-th time the header h was shown a conversion happened) or $r_k$= 0 (else)&#xA;For step size $\alpha$ you can e.g. choose:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;$1/k$ which in your case should lead to convergence (more recent rewards / conversions are weighted less)&lt;/li&gt;&#xA;&lt;li&gt;a constant, which will result in no convergence (all the rewards are weighted equal), but allow the system to adapt to changes. E.g. imagine that the concept what the best header is changes over time.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Final remark&lt;/strong&gt;&#xA;How to set the parameters $\tau$ and $\alpha$ is problem dependent. I suggest to run some simulations to see which are the best ones for you. For this and in general, I can only recommend to read chapter 2 of the linked book. It is worth it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another remark from practical experience with this formula (if the headers to show are changing constantly) is to not always use the softmax-selection formula. I rather suggest to choose the best header p% of the time a header shall be shown and select a another header in (100-p)% of the time to find another possibly better one. On the other hand, if your goal is to find the best header among a fixed number of headers, it is better to always use softmax selection and set $\alpha=1/k$ and $\tau$ close to zero.&lt;/p&gt;&#xA;" OwnerUserId="264" LastEditorUserId="264" LastEditorDisplayName="" LastEditDate="2011-03-10T07:31:01.523" LastActivityDate="2011-03-10T07:31:01.523" CommentCount="11" />
  <row Id="7880" PostTypeId="2" ParentId="7876" CreationDate="2011-03-04T13:03:27.320" Score="7" ViewCount="" Body="&lt;p&gt;Here is an example to discuss specifics against:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; plot(stl(nottem, &quot;per&quot;))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/hyPtT.png&quot; alt=&quot;Nottingham temperature STL&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So on the upper panel, we might consider the bar as 1 unit of variation. The bar on the seasonal panel is only slightly larger than that on the data panel, indicating that the seasonal signal is large relative to the variation in the data. In other words, if we shrunk the seasonal panel such that the box became the same size as that in the data panel, the range of variation on the shrunk seasonal panel would be similar to but slightly smaller than that on the data panel.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now consider the trend panel; the grey box is now much larger than either of the ones on the data or seasonal panel, indicating the variation attributed to the trend is much smaller than the seasonal component and consequently only a small part of the variation in the data series. The variation attributed to the trend is considerably smaller than the stochastic component (the remainders). As such, we can deduce that these data do not exhibit a trend.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now look at another example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; plot(stl(co2, &quot;per&quot;))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which gives&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/ElB91.png&quot; alt=&quot;Mauna Loa CO2 data&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we look at the relative sizes of the bars on this plot, we note that the trend dominates the data series and consequently the grey bars are of similar size. Of next greatest importance is variation at the seasonal scale, although variation at this scale is a much smaller component of the variation exhibited in the original data. The residuals (remainder) represent only small stochastic fluctuations as the grey bar is very large relative to the other panels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the general idea is that if you scaled all the panels such that the grey bars were all the same size, you would be able to determine the relative magnitude of the variations in each of the components and how much of the variation in the original data they contained. But because the plot draws each component on it's own scale, we need the bars to give us a relative scale for comparison.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does this help any?&lt;/p&gt;&#xA;" OwnerUserId="1390" LastActivityDate="2011-03-04T13:03:27.320" CommentCount="8" />
  <row Id="7881" PostTypeId="2" ParentId="7845" CreationDate="2011-03-04T13:32:57.760" Score="1" ViewCount="" Body="&lt;p&gt;Another approach would be to plot the different distributions on the same plot and use something like the &lt;code&gt;alpha&lt;/code&gt; parameter in &lt;code&gt;ggplot2&lt;/code&gt; to address the overplotting issues. The utility of this method will be dependent on the differences or similarities in your distribution as they will be plotted with the same bins. Another alternative would be to display smoothed density curves for each distribution. Here's an example of these options and the other options discussed in the thread:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(ggplot2)&#xA;&#xA;df &amp;lt;- melt(&#xA;    data.frame( &#xA;        x = rnorm(1000)&#xA;        , y = rnorm(1000, 0, 2)&#xA;    )&#xA;)&#xA;&#xA;&#xA;ggplot(data = df) + &#xA;#   geom_bar(aes(x = value, fill = variable), alpha = 1/2)&#xA;#   geom_bar(aes(x = value)) + facet_grid(variable ~ .)&#xA;#   geom_density(aes(x = value, colour = variable))&#xA;#   stat_qq(aes(sample = value, colour = variable))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="696" LastActivityDate="2011-03-04T13:32:57.760" CommentCount="3" />
  <row Id="7882" PostTypeId="1" CreationDate="2011-03-04T16:32:05.223" Score="4" ViewCount="69" Body="&lt;p&gt;Does anyone know how to construct a confidence interval for predicting a new test value given a trained Relevance Vector Machine (&lt;code&gt;rvm&lt;/code&gt;) and/or Gaussian Process Regression (&lt;code&gt;gausspr&lt;/code&gt;) using the &lt;a href=&quot;http://cran.r-project.org/web/packages/kernlab/index.html&quot; rel=&quot;nofollow&quot;&gt;kernlab&lt;/a&gt; R package?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More specifically, how do I get:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;The standard error/deviation (variance) of a new test point;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The parameters estimates of posterior distribution of the parameters?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I would appreciate if anyone could point to a document that discuss how to obtain/calculate the above, from the output of function call (&lt;code&gt;rvm&lt;/code&gt; or &lt;code&gt;gausspr&lt;/code&gt;).&lt;/p&gt;&#xA;" OwnerUserId="3544" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-03-04T20:02:12.327" LastActivityDate="2011-03-04T20:02:12.327" Title="How to compute prediction error from Relevance Vector Machine and Gaussian Process Regression? " Tags="&lt;r&gt;&lt;machine-learning&gt;&lt;bayesian&gt;" />
  <row Id="7883" PostTypeId="2" ParentId="7836" CreationDate="2011-03-04T16:48:38.367" Score="3" ViewCount="" Body="&lt;p&gt;We all have a good sense of what &quot;model&quot; might mean, although its technical definition will vary among disciplines.  To compare this to DGP, I began by looking at the top five hits (counting two hits with the same author as one) in Googling &quot;data generation process&quot;.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;A &lt;a href=&quot;http://www.disam.dsca.mil/pubs/Vol%2018_2/Alexander.pdf&quot; rel=&quot;nofollow&quot;&gt;paper&lt;/a&gt; on how the US Air Force actually &lt;em&gt;creates&lt;/em&gt; data in logistics support.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://www.envplan.com/abstract.cgi?id=a4147&quot; rel=&quot;nofollow&quot;&gt;Abstract of a paper&lt;/a&gt; published in Environment and Planning A concerning how &quot;synthetic micropopulations&quot; are created via computer &quot;simulation models.&quot;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;A &lt;a href=&quot;http://kdl.cs.umass.edu/software/documentation/tutorial/ch06s09.html&quot; rel=&quot;nofollow&quot;&gt;Web page&lt;/a&gt; on &quot;synthetic data generation&quot;; that is, simulation &quot;to explore the effects of certain data characteristics on ... models.&quot; &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=4389656&quot; rel=&quot;nofollow&quot;&gt;Abstract of a conference paper&lt;/a&gt; in data mining, asserting that &quot;data in databases is the outcome of an underlying data generation process (dgp).&quot;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;A &lt;a href=&quot;http://weber.ucsd.edu/~hwhite/pub_files/gwchapt2.pdf&quot; rel=&quot;nofollow&quot;&gt;book chapter&lt;/a&gt; that characterizes the data of interest as &quot;arising from some transformation $W_t$ of an underlying [stochastic] process $V_t$ ... some or all [of which] may be unobserved...&quot;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;These links exhibit three slightly different but closely related uses of the term &quot;data generation process.&quot;  The commonest is in a context of statistical simulation.  The others refer to the actual means by which data are created in an ongoing situation (logistics) and to a probability model for an ongoing data creation procedure, &lt;em&gt;intended not to be analyzed directly.&lt;/em&gt;  In the last case the text is differentiating an &lt;em&gt;unobservable stochastic process,&lt;/em&gt; which nevertheless is &lt;em&gt;modeled&lt;/em&gt; mathematically, from the &lt;em&gt;actual numbers that will be analyzed.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These suggest two slightly different answers are tenable: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;In the context of simulation or creating &quot;synthetic&quot; data for analysis, the &quot;data generation process&quot; is a way to make data for subsequent study, usually by means of a computer's pseudo random number generator.  The analysis will implicitly adopt some model that describes the mathematical properties of this DGP.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;In the context of statistical analysis, we may want to distinguish a real-world phenomenon (the DGP) from the observations that will be analyzed.  We have &lt;em&gt;models&lt;/em&gt; for both the phenomenon and the observations as well as a model for how the two are connected.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In regression, then, the &lt;em&gt;DGP&lt;/em&gt; would normally describe how a set of data $(\mathbf{X}, Y)_i$ = $(X_{1i}, X_{2i}, \dots, X_{pi}, Y_{i})$, $i=1, 2, \ldots, n$ is assumed to be produced.  &lt;em&gt;E.g.&lt;/em&gt;, the $X_{ji}$ could be set by the experimenter or they could be observed in some way and then be presumed to &lt;em&gt;cause&lt;/em&gt; or &lt;em&gt;be related to&lt;/em&gt; the values of the $Y_i$.  The &lt;em&gt;model&lt;/em&gt; would describe the possible ways in which these data could be mathematically related; &lt;em&gt;e.g.&lt;/em&gt;, we might say that each $Y_{i}$ is a random variable with expectation $\mathbf{X} \mathbf{\beta}$ and variance $\sigma^2$ for unknown parameters $\beta$ and $\sigma$.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-03-04T16:48:38.367" />
  <row Id="7884" PostTypeId="1" AcceptedAnswerId="7889" CreationDate="2011-03-04T17:17:28.997" Score="5" ViewCount="330" Body="&lt;p&gt;Sometimes I need to get only the first row of a data set grouped by an identifier, as when retrieving age and gender when there are multiple observations per individual. What's a fast (or the fastest) way to do this in R? I used aggregate() below and suspect there are better ways. Before posting this question I searched a bit on google, found and tried ddply, and was surprised that it was extremely slow and gave me memory errors on my dataset (400,000 rows x 16 cols, 7,000 unique IDs), whereas the aggregate() version was reasonably fast.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(dx &amp;lt;- data.frame(ID = factor(c(1,1,2,2,3,3)), AGE = c(30,30,40,40,35,35), FEM = factor(c(1,1,0,0,1,1))))&#xA;# ID AGE FEM&#xA;#  1  30   1&#xA;#  1  30   1&#xA;#  2  40   0&#xA;#  2  40   0&#xA;#  3  35   1&#xA;#  3  35   1&#xA;ag &amp;lt;- data.frame(ID=levels(dx$ID))&#xA;ag &amp;lt;- merge(ag, aggregate(AGE ~ ID, data=dx, function(x) x[1]), &quot;ID&quot;)&#xA;ag &amp;lt;- merge(ag, aggregate(FEM ~ ID, data=dx, function(x) x[1]), &quot;ID&quot;)&#xA;ag&#xA;# ID AGE FEM&#xA;#  1  30   1&#xA;#  2  40   0&#xA;#  3  35   1&#xA;#same result:&#xA;library(plyr)&#xA;ddply(.data = dx, .var = c(&quot;ID&quot;), .fun = function(x) x[1,])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE:&lt;/strong&gt; See Chase's answer and Matt Parker's comment for what I consider to be the most elegant approach. See @Matthew Dowle's answer for the fastest solution which uses the &lt;code&gt;data.table&lt;/code&gt; package.&lt;/p&gt;&#xA;" OwnerUserId="3432" LastEditorUserId="3432" LastEditorDisplayName="" LastEditDate="2011-03-14T15:33:12.630" LastActivityDate="2011-03-14T15:33:12.630" Title="Fast ways in R to get the first row of a data frame grouped by an identifier" Tags="&lt;r&gt;&lt;dataset&gt;&lt;aggregate&gt;&lt;plyr&gt;" AnswerCount="6" CommentCount="3" FavoriteCount="1" />
  <row Id="7885" PostTypeId="2" ParentId="7884" CreationDate="2011-03-04T17:26:41.523" Score="2" ViewCount="" Body="&lt;p&gt;You could try&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;agg &amp;lt;- by(dx, dx$ID, FUN = function(x) x[1, ])&#xA;# Which returns a list that you can then convert into a data.frame thusly:&#xA;do.call(rbind, agg)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have no idea if this will be any faster than &lt;code&gt;plyr&lt;/code&gt;, though.&lt;/p&gt;&#xA;" OwnerUserId="71" LastActivityDate="2011-03-04T17:26:41.523" />
  <row Id="7886" PostTypeId="2" ParentId="7884" CreationDate="2011-03-04T17:33:51.057" Score="7" ViewCount="" Body="&lt;p&gt;You can try to use the &lt;a href=&quot;http://cran.r-project.org/web/packages/data.table/index.html&quot; rel=&quot;nofollow&quot;&gt;data.table&lt;/a&gt; package.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For your particular case, the upside is that it's (insanely) fast. The first time I was introduced to it, I was working on data.frame objects with hundreds of thousands of rows. &quot;Normal&quot; &lt;code&gt;aggregate&lt;/code&gt; or &lt;code&gt;ddply&lt;/code&gt; methods were taken ~ 1-2 mins to complete (this was before Hadley introduced the &lt;code&gt;idata.frame&lt;/code&gt; mojo into &lt;code&gt;ddply&lt;/code&gt;). Using &lt;code&gt;data.table&lt;/code&gt;, the operation was literally done in a matter of seconds.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The downside is that its so fast because it will resort your data.table (it's just like a data.frame) by &quot;key columns&quot; and use a smart searching strategy to find subsets of your data. This will result in a reordering of your data before you collect stats over it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given that you will just want the first row of each group -- maybe the reordering will mess up which row is first, which is why it might not be appropriate in your situation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyway, you'll have to judge whether or not &lt;code&gt;data.table&lt;/code&gt; is appropriate here, but this is how you would use it with the data you've presented:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;install.packages('data.table') ## if yo udon't have it already&#xA;library(data.table)&#xA;dxt &amp;lt;- data.table(dx, key='ID')&#xA;dxt[, .SD[1,], by=ID]&#xA;     ID AGE FEM&#xA;[1,]  1  30   1&#xA;[2,]  2  40   0&#xA;[3,]  3  35   1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update:&lt;/strong&gt; Matthew Dowle (the main developer of the data.table package) has provided a better/smarter/(extremely) more efficient way to use data.table to solve this problem as one of the answers here ... definitely check that out.&lt;/p&gt;&#xA;" OwnerUserId="227" LastEditorUserId="227" LastEditorDisplayName="" LastEditDate="2011-03-10T04:01:35.727" LastActivityDate="2011-03-10T04:01:35.727" />
  <row Id="7887" PostTypeId="2" ParentId="7884" CreationDate="2011-03-04T17:42:50.517" Score="3" ViewCount="" Body="&lt;p&gt;Try reshape2&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(reshape2)&#xA;dx &amp;lt;- melt(dx,id=c('ID','FEM'))&#xA;dcast(dx,ID+FEM~variable,fun.aggregate=mean)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2817" LastActivityDate="2011-03-04T17:42:50.517" />
  <row Id="7888" PostTypeId="2" ParentId="7884" CreationDate="2011-03-04T17:43:21.490" Score="5" ViewCount="" Body="&lt;p&gt;You don't need multiple &lt;code&gt;merge()&lt;/code&gt; steps, just &lt;code&gt;aggregate()&lt;/code&gt; both variables of interest:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; aggregate(dx[, -1], by = list(ID = dx$ID), head, 1)&#xA;  ID AGE FEM&#xA;1  1  30   1&#xA;2  2  40   0&#xA;3  3  35   1&#xA;&#xA;&amp;gt; system.time(replicate(1000, aggregate(dx[, -1], by = list(ID = dx$ID), &#xA;+                                       head, 1)))&#xA;   user  system elapsed &#xA;  2.531   0.007   2.547 &#xA;&amp;gt; system.time(replicate(1000, {ag &amp;lt;- data.frame(ID=levels(dx$ID))&#xA;+ ag &amp;lt;- merge(ag, aggregate(AGE ~ ID, data=dx, function(x) x[1]), &quot;ID&quot;)&#xA;+ ag &amp;lt;- merge(ag, aggregate(FEM ~ ID, data=dx, function(x) x[1]), &quot;ID&quot;)&#xA;+ }))&#xA;   user  system elapsed &#xA;  9.264   0.009   9.301&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Comparison timings:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Matt's solution:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; system.time(replicate(1000, {&#xA;+ agg &amp;lt;- by(dx, dx$ID, FUN = function(x) x[1, ])&#xA;+ # Which returns a list that you can then convert into a data.frame thusly:&#xA;+ do.call(rbind, agg)&#xA;+ }))&#xA;   user  system elapsed &#xA;  3.759   0.007   3.785&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;2) Zach's reshape2 solution:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; system.time(replicate(1000, {&#xA;+ dx &amp;lt;- melt(dx,id=c('ID','FEM'))&#xA;+ dcast(dx,ID+FEM~variable,fun.aggregate=mean)&#xA;+ }))&#xA;   user  system elapsed &#xA; 12.804   0.032  13.019&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;3) Steve's data.table solution:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; system.time(replicate(1000, {&#xA;+ dxt &amp;lt;- data.table(dx, key='ID')&#xA;+ dxt[, .SD[1,], by=ID]&#xA;+ }))&#xA;   user  system elapsed &#xA;  5.484   0.020   5.608 &#xA;&amp;gt; dxt &amp;lt;- data.table(dx, key='ID') ## one time step&#xA;&amp;gt; system.time(replicate(1000, {&#xA;+ dxt[, .SD[1,], by=ID] ## try this one line on own&#xA;+ }))&#xA;   user  system elapsed &#xA;  3.743   0.006   3.784&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;4) Chase's fast solution using numeric, not factor, &lt;code&gt;ID&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; dx2 &amp;lt;- within(dx, ID &amp;lt;- as.numeric(ID))&#xA;&amp;gt; system.time(replicate(1000, {&#xA;+ dy &amp;lt;- dx[order(dx$ID),]&#xA;+ dy[ diff(c(0,dy$ID)) != 0, ]&#xA;+ }))&#xA;   user  system elapsed &#xA;  0.663   0.000   0.663&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and 5) Matt Parker's alternative to Chase's solution, for character or factor &lt;code&gt;ID&lt;/code&gt;, which is slightly faster than Chase's numeric &lt;code&gt;ID&lt;/code&gt; one:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; system.time(replicate(1000, {&#xA;+ dx[c(TRUE, dx$ID[-1] != dx$ID[-length(dx$ID)]), ]&#xA;+ }))&#xA;   user  system elapsed &#xA;  0.513   0.000   0.516&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1390" LastEditorUserId="1390" LastEditorDisplayName="" LastEditDate="2011-03-04T20:16:34.900" LastActivityDate="2011-03-04T20:16:34.900" CommentCount="8" />
  <row Id="7889" PostTypeId="2" ParentId="7884" CreationDate="2011-03-04T17:56:24.290" Score="6" ViewCount="" Body="&lt;p&gt;Is your ID column really a factor? If it is in fact numeric, I think you can use the &lt;code&gt;diff&lt;/code&gt; function to your advantage. You could also coerce it to numeric with &lt;code&gt;as.numeric()&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dx &amp;lt;- data.frame(&#xA;    ID = sort(sample(1:7000, 400000, TRUE))&#xA;    , AGE = sample(18:65, 400000, TRUE)&#xA;    , FEM = sample(0:1, 400000, TRUE)&#xA;)&#xA;&#xA;dx[ diff(c(0,dx$ID)) != 0, ]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="696" LastEditorUserId="696" LastEditorDisplayName="" LastEditDate="2011-03-05T14:12:49.310" LastActivityDate="2011-03-05T14:12:49.310" CommentCount="2" />
  <row Id="7890" PostTypeId="1" CreationDate="2011-03-04T18:16:51.187" Score="3" ViewCount="103" Body="&lt;p&gt;I have data relative to image intensity, which range from 0 to 255. For some models I need to run, I must transform this intensity data to a different scale, which must have a certain mean (for example, 1.5) and a certain maximum (for this example, 2). I have found information on how to convert data to a scale based on a min and a max, and also to a scale which has a given mean and variance, but these don't apply to my case - I just want to specify the mean and the maximum, deriving other properties from the data distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks!&lt;/p&gt;&#xA;" OwnerUserId="3374" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-04T19:43:24.580" LastActivityDate="2011-03-04T21:53:32.937" Title="How to transform data to have a new mean and maximum value?" Tags="&lt;data-transformation&gt;" AnswerCount="2" />
  <row Id="7891" PostTypeId="1" CreationDate="2011-03-04T18:24:38.790" Score="3" ViewCount="42" Body="&lt;p&gt;Given a dataset $D$ and a distance measure, I want to split the dataset into two disjoint subsets $X, Y$ of a specified size (say 80% and 20% of the original size), so that the minimum distance of all pairs $(x, y)$ with $x \in X$ and $y \in Y$ is maximized. I found references to max-margin clustering without the size constraint, however my feeling is that the constraint should make the problem easier - does it? Is there some straightforward way to solve this problem?&lt;/p&gt;&#xA;" OwnerUserId="3336" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-05T00:03:34.177" LastActivityDate="2011-03-05T00:03:34.177" Title="Max-margin clustering with size constraint" Tags="&lt;clustering&gt;&lt;max-margin&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="7892" PostTypeId="2" ParentId="5733" CreationDate="2011-03-04T19:06:54.170" Score="0" ViewCount="" Body="&lt;p&gt;(1) It's the CDF you'll need to generate your simulated time-series.  To build it, first histogram your price changes/returns.  Take a cumulative sum of bin population starting with your left-most populated bin.  Normalize your new function by dividing by the total bin population.  What you are left with is a CDF.  Here is some numpy code that does the trick:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Make a histogram of price changes&#xA;&#xA;counts,bin_edges = np.histogram(deltas,numbins,normed=False) # numpy histogram&#xA;&#xA;# Make a CDF of the price changes&#xA;&#xA;n_counts,bin_edges2 = np.histogram(deltas,numbins,normed=True)&#xA;cdf = np.cumsum(n_counts)  # cdf not normalized, despite above&#xA;scale = 1.0/cdf[-1]&#xA;ncdf = scale * cdf&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(2) To generate correlated picks, use a copula.  See &lt;a href=&quot;http://stats.stackexchange.com/questions/7515/what-are-some-techniques-for-sampling-two-correlated-random-variables&quot;&gt;this answer&lt;/a&gt; to my previous question on generating correlated time series.&lt;/p&gt;&#xA;" OwnerUserId="2260" LastActivityDate="2011-03-04T19:06:54.170" />
  <row Id="7893" PostTypeId="2" ParentId="7890" CreationDate="2011-03-04T19:09:05.393" Score="9" ViewCount="" Body="&lt;p&gt;Let the mean of your sample be $\bar{x}$ and the desired mean be $\bar{y}$. Also, $x_{(n)}$ is your sample maximum and $y_{(n)}$ is the desired maximum. Then, create a transformation of your data $y=ax + b$ such that $\bar{x}$ corresponds to $\bar{y}$ and $x_{(n)}$ corresponds to $y_{(n)}$:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\begin{align*}&#xD;&#xA;  a &amp;amp;= \frac{y_{(n)}-\bar{y}}{x_{(n)}-\bar{x}} \quad \text{and} \\&#xD;&#xA;  b&amp;amp;=\bar{y}-a\bar{x}.&#xD;&#xA;\end{align*}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that this is just a line through the points $(\bar{x},\bar{y})$ and $(x_{(n)},y_{(n)})$. It requires that $x_{(n)} \neq \bar{x}$, essentially implying that $x$ must have positive variance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since this is a linear transformation, deriving properties of the distribution of $y$ using the distribution of $x$ (other than the mean and maximum) should be easy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope that helps.&lt;/p&gt;&#xA;" OwnerUserId="401" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-04T21:53:32.937" LastActivityDate="2011-03-04T21:53:32.937" CommentCount="4" />
  <row Id="7894" PostTypeId="1" AcceptedAnswerId="7904" CreationDate="2011-03-04T19:10:07.560" Score="5" ViewCount="134" Body="&lt;p&gt;If my response variable has 2 levels, 0 and 1, I can use a &lt;a href=&quot;http://rocr.bioinf.mpi-sb.mpg.de/&quot; rel=&quot;nofollow&quot;&gt;ROC curve&lt;/a&gt; to assess the accuracy of my predictive model.  But what if my response variable has 3 levels, -1, 0, and 1?  Is there a way to plot &quot;&lt;a href=&quot;http://users.dsic.upv.es/grupos/elp/cferri/vus-ecml03-camera-ready4.pdf&quot; rel=&quot;nofollow&quot;&gt;ROC surfaces&lt;/a&gt;&quot; in R?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I started trying to code this up myself, but I couldn't figure out how to calculate and plot the surface.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(rpart)&#xA;library(randomForest)&#xA;&#xA;y &amp;lt;- as.factor(c(-1,-1,-1,0,0,0,1,1,1,1,1,0,0,0,1,1,1,-1,0,1,0,1,0,-1,0,-1,0,-1,1,0))&#xA;x1 &amp;lt;- rnorm(1:30)&#xA;x2 &amp;lt;- rnorm(1:30)&#xA;x3 &amp;lt;- rnorm(1:30)&#xA;&#xA;model &amp;lt;- randomForest(y~x1+x2+x3)&#xA;model&#xA;y1 &amp;lt;- predict(model,type='prob')&#xA;&#xA;DF &amp;lt;- data.frame(y=as.numeric(as.character(y)),'negone'=y1[,'-1'],'zero'=y1[,'0'],'one'=y1[,'1'])&#xA;&#xA;DF$FPrOne &amp;lt;- &#xA;DF$FPrNegOne &amp;lt;- &#xA;DF$TPR &amp;lt;- &#xA;&#xA;library(rgl)&#xA;plot3d(DF$FPrOne,DF$FPrNegOne,DF$TPR,type=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2817" LastEditorUserId="2817" LastEditorDisplayName="" LastEditDate="2011-03-04T19:17:36.493" LastActivityDate="2011-03-04T23:27:51.500" Title="ROC surfaces in R" Tags="&lt;r&gt;&lt;data-visualization&gt;&lt;roc&gt;" AnswerCount="1" CommentCount="3" FavoriteCount="1" />
  <row Id="7895" PostTypeId="1" CreationDate="2011-03-04T20:14:05.883" Score="0" ViewCount="104" Body="&lt;p&gt;I start with two independent random variables  $X_1$, $X_2$. They are each normally distributed. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$X_1^2+X_2^2=X_?^2$$ &lt;/p&gt;&#xA;" OwnerUserId="3549" LastEditorUserId="3549" LastEditorDisplayName="" LastEditDate="2011-03-05T07:40:35.747" LastActivityDate="2011-03-05T07:40:35.747" ClosedDate="2011-03-04T21:19:55.073" Title="Chi-squared distribution " Tags="&lt;homework&gt;&lt;chi-squared&gt;" AnswerCount="0" CommentCount="4" />
  <row Id="7896" PostTypeId="2" ParentId="4155" CreationDate="2011-03-04T20:38:15.360" Score="3" ViewCount="" Body="&lt;p&gt;Perhaps your ARIMA model needs to be tempered with identifiable deterministic structure such as &quot;changes in intercept&quot; or changes in trend. These models would then be classified as robust ARIMA models or Transfer FunctionModels. If there is a True Limiting Value then the data might suggest that as it grows towards that limit. Good analysis might lead to a forecast that reached its zenith at that limit.It might help the list if you actually posted one of these troublesome series.&lt;/p&gt;&#xA;" OwnerUserId="3382" LastActivityDate="2011-03-04T20:38:15.360" />
  <row Id="7897" PostTypeId="1" CreationDate="2011-03-04T21:17:23.477" Score="9" ViewCount="163" Body="&lt;p&gt;A newbie question here. I am currently performing a nonparametric regression using the np package in R. I have 7 features and using a brute force approach I identified the best 3. But, soon I will have many more than 7 features!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is what are the current best methods for feature selection for nonparametric regression. And which if any packages implement the methods. Thank you.&lt;/p&gt;&#xA;" OwnerUserId="3550" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-04T21:25:50.763" LastActivityDate="2011-03-23T18:01:20.987" Title="Best methods of feature selection for nonparametric regression" Tags="&lt;r&gt;&lt;machine-learning&gt;&lt;nonparametric&gt;&lt;feature-selection&gt;" AnswerCount="2" CommentCount="3" />
  <row Id="7898" PostTypeId="2" ParentId="7890" CreationDate="2011-03-04T21:22:36.553" Score="1" ViewCount="" Body="&lt;p&gt;Transforming based on maximum/minimums is dangerous because outliers can drastically affect the scaling. For example, if you have an image mostly in the 10-50 range, with a couple of noisy points in the 250 range, you will be adjusting the image based on a questionable maximum, hurting the dynamic range. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think it would be safer to adjust your image based on mean/median and variance, and then cap the maximum (clipping). For example, you can set the variance so the vast majority of data points in your typical image would fall in the range you desire, and cap anything that goes above it.  &lt;/p&gt;&#xA;" OwnerUserId="2965" LastActivityDate="2011-03-04T21:22:36.553" />
  <row Id="7899" PostTypeId="1" AcceptedAnswerId="7911" CreationDate="2011-03-04T21:25:00.717" Score="5" ViewCount="342" Body="&lt;p&gt;I need to draw a complex graphics for visual data analysis.&#xA;I have 2 variables and a big number of cases (&gt;1000). For example (number is 100 if to make dispersion less &quot;normal&quot;):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x &amp;lt;- rnorm(100,mean=95,sd=50)&#xA;y &amp;lt;- rnorm(100,mean=35,sd=20)&#xA;d &amp;lt;- data.frame(x=x,y=y)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;1) I need to plot raw data with point size, corresponding the relative frequency of coincidences, so &lt;code&gt;plot(x,y)&lt;/code&gt; is not an option - I need point sizes. What should be done to achieve this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) On the same plot I need to plot 95% confidence interval ellipse and line representing change of correlation (do not know how to name it correctly) - something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(corrgram)&#xA;corrgram(d, order=TRUE, lower.panel=panel.ellipse, upper.panel=panel.pts)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/R561Q.png&quot; alt=&quot;correlogramm&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;but with both graphs at one plot.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) Finally, I need to draw a resulting linar regression model on top of this all:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;r&amp;lt;-lm(y~x, data=d)&#xA;abline(r,col=2,lwd=2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;but with error range... something like on QQ-plot:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/vgvRr.png&quot; alt=&quot;QQ-plot&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;but for fitting errors, if it is possible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the question is: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How to achieve all of this at one graph?&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="3376" LastEditorUserId="3376" LastEditorDisplayName="" LastEditDate="2011-03-04T22:51:29.580" LastActivityDate="2011-03-06T15:50:56.043" Title="Complex regression plot in R" Tags="&lt;r&gt;&lt;data-visualization&gt;&lt;regression&gt;" AnswerCount="2" FavoriteCount="6" />
  <row Id="7900" PostTypeId="1" CreationDate="2011-03-04T22:29:50.427" Score="4" ViewCount="122" Body="&lt;p&gt;I'm trying to generate random textual data based on regular expressions. I'd like to be able to do this in R, as I know that R does have regex capabilities. Any leads?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This question has come up before in forums (&lt;a href=&quot;http://stackoverflow.com/questions/274011/random-text-generator-based-on-regex&quot;&gt;StackOverflow Post 1&lt;/a&gt;, &lt;a href=&quot;http://stackoverflow.com/questions/2905229/reverse-regular-expressions-to-generate-data&quot;&gt;StackOverflow Post 2&lt;/a&gt;, etc.), but they always mention solutions based on other programming platforms (perl, .NET, ...), not R.&lt;/p&gt;&#xA;" OwnerUserId="3551" LastEditorUserId="3551" LastEditorDisplayName="" LastEditDate="2011-03-05T04:42:48.977" LastActivityDate="2011-03-05T04:42:48.977" Title="Generate random strings based on regular expressions in R" Tags="&lt;r&gt;&lt;data-mining&gt;&lt;random-generation&gt;&lt;text-mining&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="7901" PostTypeId="2" ParentId="7899" CreationDate="2011-03-04T22:35:44.150" Score="2" ViewCount="" Body="&lt;p&gt;For point 1 just use the &lt;code&gt;cex&lt;/code&gt; parameter on plot to set the point size.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x = rnorm(100)&#xA;plot(x, pch=20, cex=abs(x))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To have multiple graphs in one plot use &lt;code&gt;par(mfrow=c(numrows, numcols))&lt;/code&gt; to have an evenly spaced layout or &lt;code&gt;layout&lt;/code&gt; to make more complex ones.&lt;/p&gt;&#xA;" OwnerUserId="582" LastActivityDate="2011-03-04T22:35:44.150" CommentCount="2" />
  <row Id="7902" PostTypeId="1" CreationDate="2011-03-04T22:58:38.217" Score="3" ViewCount="136" Body="&lt;p&gt;I will simplify our problem in this way. Say there are 100,000 cases in total to examine. Due to the time limitation, we randomly selected 2,000 of them. Then we found 1,000 of them are invalid, so we have only 1,000 valid cases left. Finally we categorize these 1,000 valid cases into 2 categories, A and B; they have 300 and 700 cases respectively.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We want to calculate the confidence interval, in order to check the statistical significance of the results. In other words, if the result shows that there are 30% of the cases in category A, how trustworthy this percentage is when talking about the whole population. We used this website, &lt;a href=&quot;http://www.surveysystem.com/sscalc.htm&quot; rel=&quot;nofollow&quot;&gt;http://www.surveysystem.com/sscalc.htm&lt;/a&gt;, to calculate the confidence interval, so the percentages will be like $30\pm 3.7\%$ and $70\pm 3.7\%$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So there are two ways of deciding the population and the sample size.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(1) Population is 100,000 (real population) and sample size is 2,000;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(2) Population is 50,000 (estimated) and sample size is 1,000.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think we should use option (2), because we actually found &lt;em&gt;roughly&lt;/em&gt; only 50% of the original cases are valid cases. But this ratio is actually &lt;em&gt;estimated&lt;/em&gt; by the 2,000 cases we sampled. How does this estimation affect the confidence of the result?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would somebody recommend other ways to check the statistical significance of the result in our case?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for your help!&lt;/p&gt;&#xA;" OwnerUserId="3545" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-04T23:03:13.063" LastActivityDate="2011-03-05T01:53:21.813" Title="How to calculate confidence interval when only a part of the samples are valid?" Tags="&lt;confidence-interval&gt;&lt;sampling&gt;&lt;sample-size&gt;&lt;non-response&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="7903" PostTypeId="1" AcceptedAnswerId="7908" CreationDate="2011-03-04T23:19:54.100" Score="17" ViewCount="384" Body="&lt;p&gt;QUESTION:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have binary data on exam questions (correct/incorrect). Some individuals might have had prior access to a subset of questions and their correct answers. I don’t know who, how many, or which. If there were no cheating, suppose I would model the probability of a correct response for item $i$ as $logit((p_i = 1 | z)) = \beta_i + z$, where $\beta_i$ represents question difficulty and $z$ is the individual’s latent ability. This is a very simple item response model that can be estimated with functions like ltm’s rasch() in R. In addition to the estimates $\hat{z}_j$ (where $j$ indexes individuals) of the latent variable, I have access to separate estimates $\hat{q}_j$ of the same latent variable which were derived from another dataset in which cheating was not possible. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The goal is to identify individuals who likely cheated and the items they cheated on. What are some approaches you might take? In addition to the raw data, $\hat{\beta}_i$, $\hat{z}_j$, and $\hat{q}_j$ are all available, although the first two will have some bias due to cheating. Ideally, the solution would come in the form of probabilistic clustering/classification, although this is not necessary. Practical ideas are highly welcomed as are formal approaches. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far, I have compared the correlation of question scores for pairs of individuals with higher vs. lower $\hat{q}_j -\hat{z}_j $ scores (where $\hat{q}_j - \hat{z}_j $ is a rough index of the probability that they cheated). For example, I sorted individuals by $\hat{q}_j - \hat{z}_j $ and then plotted the correlation of successive pairs of individuals’ question scores. I also tried plotting the mean correlation of scores for individuals whose $\hat{q}_j - \hat{z}_j $ values were greater than the $n^{th}$ quantile of $\hat{q}_j - \hat{z}_j $, as a function of $n$. No obvious patterns for either approach.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;UPDATE:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I ended up combining ideas from @SheldonCooper and the helpful &lt;a href=&quot;http://pricetheory.uchicago.edu/levitt/Papers/JacobLevitt2003.pdf&quot; rel=&quot;nofollow&quot; title=&quot;Freakonomics paper&quot;&gt;Freakonomics paper&lt;/a&gt; that @whuber pointed me toward. &lt;em&gt;Other ideas/comments/criticisms welcome.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let $X_{ij}$ be person $j$’s binary score on question $i$. Estimate the item response model $$logit(Pr(X_{ij} = 1 | z_j) = \beta_i + z_j,$$ where $\beta_i$ is the item’s easiness parameter and $z_j$ is a latent ability variable. (A more complicated model can be substituted; I’m using a 2PL in my application). As I mentioned in my original post, I have estimates $\hat{q_j } $ of the ability variable from a separate dataset $\{y_{ij}\}$ (different items, same persons) on which cheating was not possible. Specifically, $\hat{q_j} $ are empirical Bayes estimates from the same item response model as above. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The probability of the observed score $x_{ij}$, conditional on item easiness and person ability, can be written $$p_{ij} = Pr(X_{ij} = x_{ij} | \hat{\beta_i }, \hat{q_j }) = P_{ij}(\hat{\beta_i }, \hat{q_j })^{x_{ij}} (1 - P_{ij}(\hat{\beta_i }, \hat{q_j }))^{1-x_{ij}},$$ where $P_{ij}(\hat{\beta_i }, \hat{q_j }) = ilogit(\hat{\beta_i} + \hat{q_j})$ is the predicted probability of a correct response, and $ilogit$ is the inverse logit. Then, conditional on item and person characteristics, the joint probability that person $j$ has the observations $x_j$ is $$p_j = \prod_i p_{ij},$$ and similarly, the joint probability that item $i$ has the observations $x_i$ is $$p_i = \prod_j p_{ij}.$$ Persons with the lowest $p_j$ values are those whose observed scores are conditionally least likely -- they are possibly cheaters. Items with the lowest $p_j$ values are those which are conditionally least likely -- they are the possible leaked/shared items. This approach relies on the assumptions that the models are correct and that person $j$’s scores are uncorrelated conditional on person and item characteristics. A violation of the second assumption isn’t problematic though, as long as the degree of correlation does not vary across persons, and the model for $p_{ij}$ could easily be improved (e.g., by adding additional person or item characteristics).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An additional step I tried is to take r% of the least likely persons (i.e. persons with the lowest r% of sorted p_j values), compute the mean distance between their observed scores x_j (which should be correlated for persons with low r, who are possible cheaters), and plot it for r = 0.001, 0.002, ..., 1.000. The mean distance increases for r = 0.001 to r = 0.025, reaches a maximum, and then declines slowly to a minimum at r = 1. Not exactly what I was hoping for. &lt;/p&gt;&#xA;" OwnerUserId="3432" LastEditorUserId="3432" LastEditorDisplayName="" LastEditDate="2011-03-15T15:26:35.357" LastActivityDate="2011-03-15T15:26:35.357" Title="Detecting patterns of cheating on a multi-question exam" Tags="&lt;r&gt;&lt;clustering&gt;&lt;classification&gt;&lt;psychometrics&gt;" AnswerCount="2" CommentCount="7" FavoriteCount="4" />
  <row Id="7904" PostTypeId="2" ParentId="7894" CreationDate="2011-03-04T23:22:09.147" Score="6" ViewCount="" Body="&lt;p&gt;Some idea to stuff it into 2D in 3D is a surface defined by true positive rate of each class (&lt;a href=&quot;http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=5306177&quot; rel=&quot;nofollow&quot;&gt;He, Xin, B.D. Gallas, and E.C. Frey. 2009.&lt;/a&gt;), so I'll go this way. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say the &lt;code&gt;g&lt;/code&gt; variable holds the vote proportions:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;g&amp;lt;-model$votes;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;First you need two thresholds, one for each of some two classes (let's say for 1 and -1) -- I'll call them &lt;code&gt;qP&lt;/code&gt; and &lt;code&gt;qN&lt;/code&gt;, which will create the surface:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;qP&amp;lt;-qN&amp;lt;-seq(0,1,30)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, let's calculate a true positive rate for positive class (and negative, which is mostly the same):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;tpP&amp;lt;-function(qP,qN,g) sum(g[,&quot;1&quot;]&amp;gt;qP &amp;amp; y==&quot;1&quot;)/sum(y==&quot;1&quot;)&#xA;tpN&amp;lt;-function(qP,qN,g) sum(g[,&quot;-1&quot;]&amp;gt;qN &amp;amp; y==&quot;-1&quot;)/sum(y==&quot;-1&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The zero class gets the rest of votes:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;tpZ&amp;lt;-function(qP,qN,g) sum(!(q[,&quot;-1&quot;]&amp;gt;qN) &amp;amp; !(q[,&quot;1&quot;]&amp;gt;qP) &amp;amp; y==&quot;0&quot;)/sum(y==&quot;0&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now we just have to apply it over the whole qPxqN, easiest way is to use &lt;code&gt;outer&lt;/code&gt; on &lt;code&gt;Vectorize&lt;/code&gt;d functions (note that this is not the effective method, though I don't see any obvious optimizations which will not degenerate readability):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X&amp;lt;-outer(qP,qN,Vectorize(tpP,vec=c(&quot;qP&quot;,&quot;qN&quot;)),g)&#xA;   outer(qP,qN,Vectorize(tpN,vec=c(&quot;qP&quot;,&quot;qN&quot;)),g)-&amp;gt;Y&#xA;Z&amp;lt;-outer(qP,qN,Vectorize(tpZ,vec=c(&quot;qP&quot;,&quot;qN&quot;)),g)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now all points (&lt;code&gt;X[[i]]&lt;/code&gt;,&lt;code&gt;Y[[i]]&lt;/code&gt;,&lt;code&gt;Z[[i]]&lt;/code&gt;) should lay on the surface; the matrix format is suitable to run &lt;code&gt;rgl.surface&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;rgl.surface(X,Z,Y) //Yup, the order is weired&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/LE2yN.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Definitely does not seem too good, but this is what one can expect on a small random data ;-) &lt;/p&gt;&#xA;" OwnerUserId="88" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-04T23:27:51.500" LastActivityDate="2011-03-04T23:27:51.500" />
  <row Id="7905" PostTypeId="2" ParentId="7891" CreationDate="2011-03-05T00:02:10.823" Score="1" ViewCount="" Body="&lt;p&gt;A bit vague answer, but I'll give it a chance. I always felt that the size constraint is the central idea behind this method -- without is seems just to converge to other approaches, effectively to 2-means and ideologically to unsupervised SVM. The previous rather invalidates this idea, the latter way is more intriguing while you may hope to save some pain using SVM optimization framework and kernel tricks.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2011-03-05T00:02:10.823" CommentCount="3" />
  <row Id="7906" PostTypeId="2" ParentId="7900" CreationDate="2011-03-05T00:36:49.843" Score="3" ViewCount="" Body="&lt;p&gt;While generating random data from regular expressions would be a convenient interface, it is not directly supported in R. You could try one level of indirection though: generate random numbers and convert them into strings. For example, to convert a number into a character, you could use the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; rawToChar(as.raw(65))&#xA;[1] &quot;A&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;By carefully selecting the range of the random number to draw you can restrict your self to a desired set of ASCII characters that might correspond to a regular expression, e.g., to the character class &lt;code&gt;[a-zA-Z]&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Clearly, this is neither an elegant nor efficient solution, but it is at least native and could give you the desired effect with some boilerplate.&lt;/p&gt;&#xA;" OwnerUserId="1537" LastActivityDate="2011-03-05T00:36:49.843" CommentCount="1" />
  <row Id="7907" PostTypeId="2" ParentId="7902" CreationDate="2011-03-05T01:53:21.813" Score="4" ViewCount="" Body="&lt;p&gt;From a response to comment, we can adopt an urn model.  The urn contains 100,000 balls representing all cases.  An unknown number of these are black (&quot;invalid&quot;); they are of no interest.  We are interested &lt;em&gt;solely in the non-black balls in the urn.&lt;/em&gt;  Of those, some are of color &quot;A&quot; and others of color &quot;B&quot;.  &lt;strong&gt;The main research question appears to be &quot;what proportion of the balls of interest are A's?&quot;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;This urn model says option (2) is the one to use.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A simple random sample (without replacement) of 2,000 balls from this urn yielded 1000 black balls, 300 A's, and 700 B's, for n = 1000 A's &amp;amp; B's.  The rest is routine.  In particular, the distribution of A's (conditional on a non-black ball being drawn) is Binomial(p, 1000).  A standard estimate of p is #A's / (Total A's &amp;amp; B's) = 30%.  The estimated variance of the total is p(1-p), whence the variance of the estimated proportion of A's equals p(1-p)/n = 0.00021.  Its square root, 1.45%, is the standard error of estimate of p.  Because the numbers of A's and B's are large, yet are small compared to the expected number of non-black balls (about 50,000), it is appropriate to use normal-theory confidence intervals and to ignore the correction for sampling without replacement.  (The correction shrinks the confidence interval to 0.99 times its width.)  A 99% two-sided confidence interval therefore extends 2.58 * 1.45% = 3.73% to either side of the estimated proportions &lt;em&gt;E.g.&lt;/em&gt;, a confidence interval for the proportion of A's (out of all the A's and B's in the urn) extends from 26.27% to 33.73%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;If you are uncomfortable using conditional probabilities&lt;/strong&gt; (which is at the root of this analysis), you can estimate the contents of the urn (i.e., total numbers of black balls, A's, and B's) using the &lt;a href=&quot;http://en.wikipedia.org/wiki/Multinomial_distribution&quot; rel=&quot;nofollow&quot;&gt;multinomial distribution&lt;/a&gt;.  You will get exactly the same results, because in the end you care only about the proportion of A's relative to the numbers of A's and B's, so all estimates involving the number of black balls never enter the calculation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Another way to get some intuition&lt;/strong&gt; is to recognize that (except for the tiny correction term being neglected here) the size of the confidence interval depends &lt;em&gt;only on the observed numbers of A's and B's&lt;/em&gt; and not on the number of balls in the urn.  That's why there's no concern here about whether the &quot;population&quot; is 50,000 or 100,000.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;An auxiliary research question&lt;/strong&gt; seems to be to estimate the total number of A's and B's in the urn.  For this purpose the urn contains only two kinds of balls, black ones and non-black ones, and we want to estimate the number of non-black balls.  This is a standard binomial sampling situation.  Without more ado, the estimated number of non-black balls equals 100,000 * (1000/2000) = 50,000 and the estimated proportion is 1/2, with standard error $\sqrt{(1/2)(1 - 1/2)/2000}$ = 1.1%.  Therefore the estimate of 50,000 has a 99% two-sided confidence interval from 48,560 to 51,440.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-03-05T01:53:21.813" CommentCount="1" />
  <row Id="7908" PostTypeId="2" ParentId="7903" CreationDate="2011-03-05T03:21:10.143" Score="3" ViewCount="" Body="&lt;h2&gt;Ad hoc approach&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I'd assume that $\beta_i$ is reasonably reliable because it was estimated on many students, most of who did not cheat on question $i$. For each student $j$, sort the questions in order of increasing difficulty, compute $\beta_i + q_j$ (note that $q_j$ is just a constant offset) and threshold it at some reasonable place (e.g. p(correct) &amp;lt; 0.6). This gives a set of questions which the student is unlikely to answer correctly. You can now use hypothesis testing to see whether this is violated, in which case the student probably cheated (assuming of course your model is correct). One caveat is that if there are few such questions, you might not have enough data for the test to be reliable. Also, I don't think it's possible to determine which question he cheated on, because he always has a 50% chance of guessing. But if you assume in addition that many students got access to (and cheated on) the same set of questions, you can compare these across students and see which questions got answered more often than chance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can do a similar trick with questions. I.e. for each question, sort students by $q_j$, add $\beta_i$ (this is now a constant offset) and threshold at probability 0.6. This gives you a list of students who shouldn't be able to answer this question correctly. So they have a 60% chance to guess. Again, do hypothesis testing and see whether this is violated. This only works if most students cheated on the same set of questions (e.g. if a subset of questions 'leaked' before the exam).&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Principled approach&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;For each student, there is a binary variable $c_j$ with a Bernoulli prior with some suitable probability, indicating whether the student is a cheater. For each question there is a binary variable $l_i$, again with some suitable Bernoulli prior, indicating whether the question was leaked. Then there is a set of binary variables $a_{ij}$, indicating whether student $j$ answered question $i$ correctly. If $c_j = 1$ and $l_i = 1$, then the distribution of $a_{ij}$ is Bernoulli with probability 0.99. Otherwise the distribution is $logit(\beta_i + q_j)$. These $a_{ij}$ are the observed variables. $c_j$ and $l_i$ are hidden and must be inferred. You probably can do it by Gibbs sampling. But other approaches might also be feasible, maybe something related to biclustering.&lt;/p&gt;&#xA;" OwnerUserId="3369" LastEditorUserId="3369" LastEditorDisplayName="" LastEditDate="2011-03-05T07:35:00.167" LastActivityDate="2011-03-05T07:35:00.167" CommentCount="1" />
  <row Id="7909" PostTypeId="2" ParentId="7868" CreationDate="2011-03-05T05:12:14.717" Score="1" ViewCount="" Body="&lt;p&gt;This can be done using relational database. R has a nice implementation of this (see this post on &lt;a href=&quot;http://stackoverflow.com/questions/1169551/sql-like-functionality-in-r&quot;&gt;sqldf&lt;/a&gt;). MS Access (or even Excel) will work just as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The idea here is you want to create a table that maps a number (as you say, of 5/6 digits) to a geographical region (75 or however many you have). Then, you join your table of 10000 records onto your reference table.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let table &lt;code&gt;mydata&lt;/code&gt; contain your 10000 records and holds at least 1 column:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;ID - contains your 'cell number'&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Let table &lt;code&gt;myreftable&lt;/code&gt; contain your reference table, which should have exactly 1 row for each geographic region, and holds 2 columns:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;ID - contains the relevent 5/6 digits of your 'cell number'&lt;/li&gt;&#xA;&lt;li&gt;Geo - contains the description of the geographic region&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The table you'd want would be generate by the following SQL:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;select&#xA;   m.ID as cell_number&#xA;   ,r.Geo as geo_region&#xA;from mydata m&#xA;inner join myreftable r on left(m.ID,6)=r.ID&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;... where 'left()' is any function that takes the first 'n' characters of a string. Each database has different text/string functions you can use for this purpose. &lt;/p&gt;&#xA;" OwnerUserId="3551" LastActivityDate="2011-03-05T05:12:14.717" />
  <row Id="7911" PostTypeId="2" ParentId="7899" CreationDate="2011-03-05T09:57:01.973" Score="15" ViewCount="" Body="&lt;p&gt;Does the picture below look like what you want to achieve?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/NDRc2.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's the &lt;strong&gt;updated&lt;/strong&gt; R code, following your comments:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;do.it &amp;lt;- function(df, type=&quot;confidence&quot;, ...) {&#xA;  require(ellipse)&#xA;  lm0 &amp;lt;- lm(y ~ x, data=df)&#xA;  xc &amp;lt;- with(df, xyTable(x, y))&#xA;  df.new &amp;lt;- data.frame(x=seq(min(df$x), max(df$x), 0.1))&#xA;  pred.ulb &amp;lt;- predict(lm0, df.new, interval=type)&#xA;  pred.lo &amp;lt;- predict(loess(y ~ x, data=df), df.new)&#xA;  plot(xc$x, xc$y, cex=xc$number*2/3, xlab=&quot;x&quot;, ylab=&quot;y&quot;, ...)&#xA;  abline(lm0, col=&quot;red&quot;)&#xA;  lines(df.new$x, pred.lo, col=&quot;green&quot;, lwd=1.5)&#xA;  lines(df.new$x, pred.ulb[,&quot;lwr&quot;], lty=2, col=&quot;red&quot;)&#xA;  lines(df.new$x, pred.ulb[,&quot;upr&quot;], lty=2, col=&quot;red&quot;)    &#xA;  lines(ellipse(cor(df$x, df$y), scale=c(sd(df$x),sd(df$y)), &#xA;        centre=c(mean(df$x),mean(df$y))), lwd=1.5, col=&quot;green&quot;)&#xA;  invisible(lm0)&#xA;}&#xA;&#xA;set.seed(101)&#xA;n &amp;lt;- 1000&#xA;x &amp;lt;- rnorm(n, mean=2)&#xA;y &amp;lt;- 1.5 + 0.4*x + rnorm(n)&#xA;df &amp;lt;- data.frame(x=x, y=y)&#xA;&#xA;# take a bootstrap sample&#xA;df &amp;lt;- df[sample(nrow(df), nrow(df), rep=TRUE),]&#xA;&#xA;do.it(df, pch=19, col=rgb(0,0,.7,.5))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And here is the &lt;em&gt;ggplotized&lt;/em&gt; version&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/MjLli.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;produced with the following piece of code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;xc &amp;lt;- with(df, xyTable(x, y))&#xA;df2 &amp;lt;- cbind.data.frame(x=xc$x, y=xc$y, n=xc$number)&#xA;df.ell &amp;lt;- as.data.frame(with(df, ellipse(cor(x, y), &#xA;                                         scale=c(sd(x),sd(y)), &#xA;                                         centre=c(mean(x),mean(y)))))&#xA;library(ggplot2)&#xA;&#xA;ggplot(data=df2, aes(x=x, y=y)) + &#xA;  geom_point(aes(size=n), alpha=.6) + &#xA;  stat_smooth(data=df, method=&quot;loess&quot;, se=FALSE, color=&quot;green&quot;) + &#xA;  stat_smooth(data=df, method=&quot;lm&quot;) +&#xA;  geom_path(data=df.ell, colour=&quot;green&quot;, size=1.2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It could be customized a little bit more by adding model fit indices, like Cook's distance, with a color shading effect.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-03-06T15:50:56.043" LastActivityDate="2011-03-06T15:50:56.043" CommentCount="13" />
  <row Id="7912" PostTypeId="1" AcceptedAnswerId="7921" CreationDate="2011-03-05T10:48:05.390" Score="6" ViewCount="411" Body="&lt;p&gt;The question is pretty much contained in the title. What is the Mahalanobis distance for two distributions of different covariance matrices? What I have found till now assumes the same covariance for both distributions, i.e., something of this sort:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\Delta^T \Sigma^{-1} \Delta$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What if I have two different $\Sigma$s?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note:- The problem is this: there are two bivariate distributions that have the same dimensions but that are rotated and translated with respect to each other (sorry I come from a pure mathematical background, not a statistics one). I need to measure their degree of overlap/distance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;*&lt;em&gt;Update: *&lt;/em&gt; What might or might not be implicit in what I'm asking is that I need a distance between the means of the two distributions. I &lt;em&gt;know&lt;/em&gt; where the means are, but since the two distributions are rotated with respect to one another, I need to assign different weights to different orientations and therefore a simple Euclidean distance between the means does not work. Now, as I have understood it, the Mahalanobis distance cannot be used to measure this information if the distributions are differently shaped (apparently it works with two multivariate normal distributions of identical covariances, but not in the general case). Is there a good measure that encodes this wish to encode orientations with different weights?&lt;/p&gt;&#xA;" OwnerUserId="3586" LastEditorUserId="223" LastEditorDisplayName="" LastEditDate="2011-03-07T14:24:08.353" LastActivityDate="2011-03-11T12:33:23.773" Title="Mahalanobis distance between two bivariate distributions with different covariances" Tags="&lt;normal-distribution&gt;&lt;multivariate-analysis&gt;&lt;distance-functions&gt;" AnswerCount="2" CommentCount="6" />
  <row Id="7913" PostTypeId="2" ParentId="7903" CreationDate="2011-03-05T11:33:10.813" Score="2" ViewCount="" Body="&lt;p&gt;If you want to get into some more complex approaches, you might look at item response theory models.  You could then model the difficulty of each question.  Students who got difficult items correct while missing easier ones would, I think, be more likely to be cheating than those who did the reverse.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's been more than a decade since I did this sort of thing, but I think it could be promising.  For more detail, check out psychometrics books&lt;/p&gt;&#xA;" OwnerUserId="686" LastActivityDate="2011-03-05T11:33:10.813" CommentCount="3" />
  <row Id="7914" PostTypeId="2" ParentId="7826" CreationDate="2011-03-05T12:46:54.847" Score="1" ViewCount="" Body="&lt;p&gt;So you have a population each of whom can have zero or more conditions.  To answer the question: How many hospital patients have A?  It seems to me that the best you can do is take your favourite proportion estimator and offer it up with your favourite confidence interval.  There are lots of choices, which will make a difference for very high or very low proportions.  If you have such a situation, the estimator above may not be optimal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are interested in the population of just your hospital then you can, as SheldonCooper points out, dispense with the statistics altogether.  I suspect however that you are interested in hospital patients more generally, so your standard errors and intervals might be interpreted relative to this population.  In your suggested estimator the identity of the population will determine what 1-F is.  Certainly hospital patients don't look like non-hospital patients with respect to the conditions you're counting, but that need not matter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Following Sheldon's second observation, it is probable that the conditions correlate.  But as far as I can see this is only useful information if you are asking conditional questions, e.g. the prevalence of A among B sufferers.  In probabilistic terms your question is about estimating marginals, and correlation information only tells you about estimating conditionals.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you &lt;em&gt;were&lt;/em&gt; interested in these sorts of subgroups, you'd certainly want to model this information.  You'd also want it if there were differential measurement errors or sample selection issues, etc. e.g. only getting tested for A if you have a B diagnosis...  That might also make certain sample marginals problematic as estimates of population marginals.  Thankfully, I don't know much about hospital populations, but I'd be willing to bet that there are some of these issues around.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, about reporting: If you in fact want to report confidence &lt;em&gt;regions&lt;/em&gt; rather than condition-wise &lt;em&gt;intervals&lt;/em&gt;, then again the correlation structure matters, and things get considerably trickier.  I seem to remember that Agresti had a paper on simultaneous confidence intervals for multivariate Binomial proportions, which might be helpful for this approach.  &lt;/p&gt;&#xA;" OwnerUserId="1739" LastEditorUserId="1739" LastEditorDisplayName="" LastEditDate="2011-03-05T13:02:13.523" LastActivityDate="2011-03-05T13:02:13.523" CommentCount="2" />
  <row Id="7915" PostTypeId="1" AcceptedAnswerId="7922" CreationDate="2011-03-05T13:09:10.763" Score="3" ViewCount="195" Body="&lt;p&gt;I'm doing a binary classification using SVM classfier, libsvm,  where roughly 95% belongs to one class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The parameters C and gamma are to be set before the actual training takes place.&#xA;I followed &lt;a href=&quot;http://www.csie.ntu.edu.tw/~cjlin/libsvm/&quot; rel=&quot;nofollow&quot;&gt;the tutorial&lt;/a&gt; but still can't get any good results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a script that comes with the library that is supposed to help with choosing the right values for parameters but what this script is doing is basically maximizing the accuracy metric (TP+TN)/ALL, so in my case it chooses the parameters to label all data with prevailing class label.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to choose parameters with recall and precision based metrics. How could I approach this problem. Accuracy is a meaningless metric for what I'm doing. Also I'm keen on changing the library libsvm to any other one that can help me with this problem as long as it takes data in the same format.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1 1:0.3 2:0.4 ...&#xA;-1 1.0.4 2:0.23 and so on&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anybody help?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;UPDATE:&#xA;yes I did try both grid.py and easy.py but even though grid search uses logarithmic scale it is extremely slow.I mean even if I run it on just small chunk of my data it takes tens of hours to finish. Is this the most efficient way to use SVM?? Have also tried svmlight but it does exactly the same it labels all data with one label.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;UPDATE2:&#xA;I reformed my question the better reflect what sort of issues I am facing&lt;/p&gt;&#xA;" OwnerUserId="1371" LastEditorUserId="1371" LastEditorDisplayName="" LastEditDate="2011-03-05T15:49:16.910" LastActivityDate="2011-03-05T21:56:07.913" Title="Efficient way to classify with SVM" Tags="&lt;machine-learning&gt;&lt;svm&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="7916" PostTypeId="2" ParentId="7915" CreationDate="2011-03-05T14:19:04.687" Score="1" ViewCount="" Body="&lt;p&gt;By Default libSVM find the optimal hyper-parameters, for the SVM model using cross validation methods and by using Accuracy (for classification), or Mean Square Error (for regression) as a measure for evaluation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Weka has several other evaluation metric to find the optimal parameters (using the gridSearch)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the metric you are interested is not there, the simplest solution that comes to mind is to write a little program that would perform cross validation, and optimize the parameters of the model based on your measure of choice.&lt;/p&gt;&#xA;" OwnerUserId="3544" LastActivityDate="2011-03-05T14:19:04.687" CommentCount="3" />
  <row Id="7917" PostTypeId="2" ParentId="7774" CreationDate="2011-03-05T14:28:38.093" Score="6" ViewCount="" Body="&lt;p&gt;I'm providing this as a second answer since the analysis is completely elementary and provides exactly the desired result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Proposition&lt;/strong&gt; For $c &amp;gt; 0$ and $n \geq 1$,&#xA;$$&#xD;&#xA;\mathbb{P}(T &amp;lt; n \log n - c n ) &amp;lt; e^{-c} \&amp;gt;.&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The idea behind the proof is simple:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Represent the time until all coupons are collected as $T = \sum_{i=1}^n T_i$, where $T_i$ is the time that the $i$th (heretofore) &lt;strong&gt;&lt;em&gt;unique&lt;/em&gt;&lt;/strong&gt; coupon is collected. The $T_i$ are geometric random variables with mean times of $\frac{n}{n-i+1}$.&lt;/li&gt;&#xA;&lt;li&gt;Apply a version of the Chernoff bound and simplify.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Proof&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For any $t$ and &lt;em&gt;any&lt;/em&gt; $s &amp;gt; 0$, we have that&#xA;$$&#xD;&#xA;\mathbb{P}(T &amp;lt; t) = \mathbb{P}( e^{-s T} &amp;gt; e^{-s t} ) \leq e^{s t} \mathbb{E} e^{-s T} \&amp;gt; . &#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since $T = \sum_i T_i$ and the $T_i$ are independent, we can write&#xA;$$&#xD;&#xA;\mathbb{E} e^{-s T} = \prod_{i=1}^n \mathbb{E} e^{- s T_i}&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now since $T_i$ is geometric, let's say with probability of success $p_i$, then a simple calculation shows&#xA;$$&#xD;&#xA;\mathbb{E} e^{-s T_i} = \frac{p_i}{e^s - 1 + p_i} .&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The $p_i$ for our problem are $p_1 = 1$, $p_2 = 1 - 1/n$, $p_3 = 1 - 2/n$, etc. Hence,&#xA;$$&#xD;&#xA;\prod_{i=1}^n \mathbb{E} e^{-s T_i} = \prod_{i=1}^n \frac{i/n}{e^s - 1 + i/n}.&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's &lt;em&gt;choose&lt;/em&gt; $s = 1/n$ and $t = n \log n - c n$ for some $c &amp;gt; 0$. Then&#xA;$$&#xD;&#xA;e^{s t} = n e^{-c}&#xD;&#xA;$$&#xA;and $e^s = e^{1/n} \geq 1 + 1/n$, yielding&#xA;$$&#xD;&#xA;\prod_{i=1}^n \frac{i/n}{e^s - 1 + i/n} \leq \prod_{i=1}^n \frac{i}{i+1} = \frac{1}{n+1} \&amp;gt; .&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Putting this together, we get that&#xA;$$&#xD;&#xA;P(T &amp;lt; n \log n - c n) \leq \frac{n}{n+1} e^{-c} &amp;lt; e^{-c}&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;as desired.&lt;/p&gt;&#xA;" OwnerUserId="2970" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-03-05T14:34:54.470" LastActivityDate="2011-03-05T14:34:54.470" CommentCount="5" />
  <row Id="7919" PostTypeId="1" AcceptedAnswerId="7924" CreationDate="2011-03-05T18:30:43.537" Score="3" ViewCount="144" Body="&lt;p&gt;I am working on a linear regression with R and there are many 0 values in my predictor variables. How are these handled in R's &lt;code&gt;lm()&lt;/code&gt; function? Should I remove this data for more accurate analysis? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any advice is appreciated. Thanks. &lt;/p&gt;&#xA;" OwnerUserId="1422" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-03-05T19:07:45.807" LastActivityDate="2011-03-06T09:56:37.253" Title="How are zero values handled in lm()?" Tags="&lt;r&gt;&lt;regression&gt;" AnswerCount="2" CommentCount="5" FavoriteCount="1" />
  <row Id="7920" PostTypeId="2" ParentId="7919" CreationDate="2011-03-05T18:49:20.003" Score="0" ViewCount="" Body="&lt;p&gt;What % of the predictor is 0, and what other values does it take on?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The concern is whether a predictor with such little variation (vast majority being the value of 0) would be useful in a regression model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To approach this, you can first stratify and do one analysis with the subset of the data where predictor is 0, and another analysis where the predictor is != 0. Once you get a sense of the structure of the data, you can decide whether to proceed with analysis using the entire dataset, and whether the predictor variable should stay in the model. &lt;/p&gt;&#xA;" OwnerUserId="812" LastActivityDate="2011-03-05T18:49:20.003" />
  <row Id="7921" PostTypeId="2" ParentId="7912" CreationDate="2011-03-05T20:38:55.867" Score="4" ViewCount="" Body="&lt;p&gt;There are many notions of distance between probability distributions.  Which one to use depends on your goals.  &lt;a href=&quot;http://en.wikipedia.org/wiki/Total_variation_distance&quot; rel=&quot;nofollow&quot;&gt;Total variation distance&lt;/a&gt; is a natural way of measuring overlap between distributions.&#xA;If you are working with multivariate Normals, the &lt;a href=&quot;http://en.wikipedia.org/wiki/Multivariate_normal_distribution#Kullback.E2.80.93Leibler_divergence&quot; rel=&quot;nofollow&quot;&gt;Kullback-Leibler Divergence&lt;/a&gt; is mathematically convenient.  Though it is not actually a distance (as it fails to be symmetric and fails to obey the triangle inequality), it upper bounds the total variation distance — see &lt;a href=&quot;http://en.wikipedia.org/wiki/Pinsker%27s_inequality&quot; rel=&quot;nofollow&quot;&gt;Pinsker’s Inequality&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1670" LastActivityDate="2011-03-05T20:38:55.867" CommentCount="1" />
  <row Id="7922" PostTypeId="2" ParentId="7915" CreationDate="2011-03-05T21:56:07.913" Score="3" ViewCount="" Body="&lt;p&gt;I would do two things. First, to address your issue with accuracy due to imbalanced data, you need to set the cost of misclassifying positive and negative examples separately. A reasonable rule of thumb in your case would be to set the cost to 5 for the larger class and to 95 for the smaller class. This way misclassifying 10% of the smaller class will have the same cost as misclassifying 10% of the larger class even though the latter 10% is a much larger number of points. If you use the command line, the command is something like -w0 5 -w1 95. I feel this needs to be done anyway (even though you are using F score for now) because this is what SVM optimizes, so unless you do it, all your F scores will be poor.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second, to address the issue of speed, I would try pre-computing the kernel. For 26k points this is borderline infeasible, but if you are willing to subsample, you can precompute the kernel once for each gamma and reuse it across all C's.&lt;/p&gt;&#xA;" OwnerUserId="3369" LastActivityDate="2011-03-05T21:56:07.913" CommentCount="1" />
  <row Id="7923" PostTypeId="1" CreationDate="2011-03-06T08:21:51.870" Score="2" ViewCount="229" Body="&lt;p&gt;In SPSS, I want to &lt;strong&gt;compare&lt;/strong&gt; two clusters of management sciences department faculty members in two universities.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Which test should I use?&lt;/li&gt;&#xA;&lt;li&gt;Can you explain how to do it in SPSS?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="3570" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-03-30T18:33:50.403" LastActivityDate="2011-03-30T18:33:50.403" Title="How can I compare Likert scale data of two clusters in SPSS?" Tags="&lt;clustering&gt;&lt;spss&gt;&lt;scales&gt;&lt;likert&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="7924" PostTypeId="2" ParentId="7919" CreationDate="2011-03-06T09:56:37.253" Score="2" ViewCount="" Body="&lt;p&gt;The problem you described here is known as &lt;strong&gt;limited dependent variable&lt;/strong&gt; problem usually represented by truncated or censored data (the former could be seen as a special case of the later). In this case application of &lt;code&gt;lm()&lt;/code&gt; function would not be the best choice, since it in general will produce biased and inconsistent estimates of the true regression line. However, truncation (dropping zeroes from the sample, as you suggested in the comment) will make this bias even larger.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Likely the problem is well known and there are usually two common options to solve it either to use a &lt;a href=&quot;http://en.wikipedia.org/wiki/Tobit_model&quot; rel=&quot;nofollow&quot;&gt;Tobit model&lt;/a&gt; or a &lt;a href=&quot;http://en.wikipedia.org/wiki/Heckman_correction&quot; rel=&quot;nofollow&quot;&gt;Heckman&lt;/a&gt;'s two step approach, it would be useful to study any common econometric textbook on the topic (this Cross Validated &lt;a href=&quot;http://stats.stackexchange.com/questions/4612/good-econometrics-textbooks&quot;&gt;link&lt;/a&gt; will be useful). The difference in two models is that Heckman's method allows for either explanatory variables or parameter estimates to differ across the estimated parts that influence the zeros and the magnitude of the observed non zero values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To implement the Tobit and Heckman models in R you will need &lt;code&gt;sampleSelection&lt;/code&gt; or &lt;code&gt;censReg&lt;/code&gt; packages. There are also nice Vignettes corresponding to these packages, so read them first. &lt;/p&gt;&#xA;" OwnerUserId="2645" LastActivityDate="2011-03-06T09:56:37.253" />
  <row Id="7925" PostTypeId="1" AcceptedAnswerId="7927" CreationDate="2011-03-06T10:43:38.317" Score="10" ViewCount="134" Body="&lt;p&gt;I am currently working on a project where I generate random values using &lt;a href=&quot;http://www.puc-rio.br/marco.ind/quasi_mc.html&quot; rel=&quot;nofollow&quot;&gt;low discrepancy / quasi-random point sets&lt;/a&gt;, such as Halton and Sobol point sets. These are essentially $d$-dimensional vectors that mimic a $d$-dimensional uniform(0,1) variables, but have a better spread. In theory, they are supposed to help reduce the variance of my estimates in another part of the project.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unfortunately, I've been running into issues working with them and much of the literature on them is dense. I was therefore hoping to get some insight from someone who has experience with them, or at least figure out a way to empirically assess what is going on:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;If you have worked with them:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;What exactly is scrambling? And what effect does it have on the stream of points that are generated? In particular, is there an effect when the dimension of the points that are generated increases?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Why is it that if I generate two streams of Sobol points with MatousekAffineOwen scrambling, I get two different streams of points. Why is this not the case when I use reverse-radix scrambling with Halton points? Are there other scrambling methods that exist for these point sets - and if so, is there a MATLAB implementation of them? &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;If you have not worked with them:&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Say I have $n$ sequences $S_1, S_2, \ldots,S_n$ of supposedly random numbers, what type of statistics should I use to show that they are not correlated to each other? And what number $n$ would I need to prove that my result is statistically significant? Also, how could I do the same thing if I had $n$ sequences $S_1, S_2, \ldots,S_n$ of $d$-dimensional random $[0,1]$ vectors?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Follow-Up Questions on Cardinal's Answer&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Theoretically speaking, can we pair any scrambling method with any low discrepany sequence? MATLAB only allows me to apply reverse-radix scrambling on Halton sequences, and am wondering whether that's simply an implementation issue or a compability issue.  &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I am looking for a way that will allow me to generate two (t,m,s) nets that are uncorrelated with each another. Will MatouseAffineOwen allow me to do this? How about if I used a deterministic scrambling algorithm and simply decided to choose every 'kth' value where k was a prime?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="3572" LastEditorUserId="3572" LastEditorDisplayName="" LastEditDate="2011-03-07T02:48:43.523" LastActivityDate="2011-03-07T02:48:43.523" Title="Scrambling and correlation in low discrepancy sequences (Halton/Sobol)" Tags="&lt;hypothesis-testing&gt;&lt;monte-carlo&gt;&lt;random-generation&gt;&lt;randomness&gt;" AnswerCount="1" CommentCount="3" FavoriteCount="1" />
  <row Id="7926" PostTypeId="2" ParentId="7873" CreationDate="2011-03-06T11:31:31.633" Score="3" ViewCount="" Body="&lt;p&gt;It does sound like you are in a bit of a quandary because you only have 1 response variable for each individual measurement.  I was initially going to recommend a multi-level approach.  But in order for that to work you need to observe the response at the lowest level - which you do not - you observe your response at the individual level (which would be level 2 in a MLM)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;1)Taking the average of x means losing information in the within-individual variability of x.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You are losing variability of the covariate x, but this only matters if the other information contained in X is related to the response.  There is nothing from stopping you from putting the variance of X in as a covariate either.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;2) The mean is itself a statistic, so by putting it in the model we end up doing statistics on statistics.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A statistic is a function of the observed data.  So any covariate is a &quot;statistic&quot;.  So you are already doing &quot;statistics on statistics&quot; whether you like it or not.  However, it does make a difference to how you should &lt;em&gt;interpret&lt;/em&gt; the slope coefficient - as an average value, and not a value in the individual birth.  If you don't care about the individual births, then this matters little.  If you do, then this approach can be misleading.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;3) The number of offspring an individual had is in the model, but it is also used to calculate the mean of variable x, which I think could cause trouble.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It would only matter if the mean of X was functionally/deterministically related to number of offspring.  One way this can happen is if the value of X is the same for each individual who had the same number of births.  Usually this isn't the case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could specify a model which includes each value of X as a covariate.  But this would probably involve some new methodological research on your part I would imagine.  Your likelihood function would be different for different individuals, due to the different number of measurements within individuals.  I don't think multi-level modeling applies in this case &lt;em&gt;conceptually&lt;/em&gt;.  This is simply because the births are not a subset or sample within individuals.  Although the maths may be the same.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One way you could incorporate this structure is to create a model like:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$(Y_{ij}|x_{ij}) \sim Bin(Y_{ij}|n_{ij},p_{ij})$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $Y_{ij}$ is the binomial response for individual $i$ and $j$ denotes the number of births, $x_{ij}$ is the covariates, and $n_{ij}$ is the number of individuals with the same covariate values, and also had the same number of births.  $p_{ij}$ is the probability, which you normally model as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$g(p_{ij}) = x_{ij}^{T}\beta$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For some monotonic/invertible function $g(.)$.  The &quot;tricky&quot; part comes in because the dimension of $x_{ij}$ varies with $j$.   The log-likelihood in this case is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$L=L(\beta)=\sum_{j\in B}\Bigg[\sum_{i=1}^{N_{j}} log[Bin(Y_{ij}|n_{ij},g^{-1}(x_{ij}^{T}\beta))]\Bigg]$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $B$ is just the set of the number of births which you have available in your data set.  To maximise it is likely to be a nontrivial task, and you probably won't get the usual IRLS equations from doing a taylor series expansions about the current estimate.  Taylor series is the way I would go from here - I just don't have the energy to run through the process at this time.  I would suggest you try to re-arrange your answer so that it looks like an &quot;ordinary&quot; binomial GLM.  This will allow you to take advantage of the standard software available.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I can tell you is that when you differentiate with respect to a beta which depends on $j$ (e.g. the coefficient for the metabolic rate for the third birth), some of terms in this summation will drop out.  This is basically the likelihood &quot;telling you&quot; that certain observations contribute nothing to estimating certain parameters (e.g. individuals who give birth to two or less offspring contribute nothing to the estimated slope for the metabolic rate for the third birth).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So in summary, your intuition is spot on when you suggest that something is being lost.  However, the price for &quot;purity&quot; could be high - especially if you need to write your own algorithm to get your estimates.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-03-06T11:31:31.633" />
  <row Id="7927" PostTypeId="2" ParentId="7925" CreationDate="2011-03-06T16:07:27.103" Score="6" ViewCount="" Body="&lt;p&gt;Scrambling is usually an operation applied to a $(t,m,s)$ digital net which uses some base $b$. Sobol' nets use $b = 2$, for example, while Faure nets use a prime number for $b$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The purpose of scrambling is to (hopefully) get an even more uniform distribution, especially if you can only use a small number of points. A good example to see why this works is to look at the Halton sequence in $d = 2$ and choose two &quot;largish&quot; primes, like 29 and 31. The square gets filled in very slowly using the standard Halton sequence. But, with scrambling, it is filled in more uniformly much more quickly. Here is a plot for the first couple hundred points using a deterministic scrambling approach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/TDvSO.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The most basic forms of scrambling essentially permute the base $b$ digit expansions of the original $n$ points among themselves. For more details, here is a &lt;a href=&quot;http://www-stat.stanford.edu/~owen/reports/altscram.pdf&quot; rel=&quot;nofollow&quot;&gt;clear exposition&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The nice thing about scrambling is that if you start with a $(t,m,s)$ net and scramble it, you get a $(t,m,s)$ net back out. So, there is a &lt;strong&gt;closure&lt;/strong&gt; property involved. Since you want to use the theoretical benefits of a $(t,m,s)$ net in the first place, this is very desirable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regarding types of scrambling, the reverse-radix scramble is a &lt;em&gt;deterministic&lt;/em&gt; scrambling. The Matousek scrambling algorithm is a &lt;strong&gt;&lt;em&gt;random&lt;/em&gt;&lt;/strong&gt; scramble, done, again, in a way to maintain the closure property. If you set the random seed before you make the call to the scrambling function, you should always get the same net back.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might also be interested in the &lt;a href=&quot;http://mint.sbg.ac.at/index.php&quot; rel=&quot;nofollow&quot;&gt;MinT project&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2970" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-03-06T20:52:24.660" LastActivityDate="2011-03-06T20:52:24.660" CommentCount="1" />
  <row Id="7928" PostTypeId="2" ParentId="7868" CreationDate="2011-03-06T16:39:48.037" Score="3" ViewCount="" Body="&lt;p&gt;Ah!  &quot;Cell&quot; apparently means &quot;cell &lt;em&gt;phone&lt;/em&gt;&quot; (rather than a generic cell such as a square on a map grid).  Thus, for each prefix, you would like to identify a geographic region in which that prefix is found.  These regions are not predefined; rather, you would like to estimate their extents from the data you have.  (This is why it's not a simple database tabulation problem.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If this is right, then you probably would like to apply a &quot;&lt;a href=&quot;http://gis.stackexchange.com/search?q=concave+hull&quot;&gt;concave hull&lt;/a&gt;&quot; algorithm with a GIS.  Each prefix gives a set of points in your database which translates to a polygonal region (the concave hull).  By applying this algorithm separately for each prefix, you would obtain a set of possibly overlapping geographic regions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Cluster_analysis&quot; rel=&quot;nofollow&quot;&gt;Statistical clustering&lt;/a&gt; algorithms look promising in this context, but they don't seem appropriate for several reasons.  First, you already have the clusters: the cell phone number prefix gives the cluster identifiers.  Second, the clusters might not be spatially separated, because sets of distinct ids can be geographically overlaid.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to pursue the concave hull idea, we can migrate this question to the GIS site.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-03-06T16:39:48.037" CommentCount="2" />
  <row Id="7929" PostTypeId="1" AcceptedAnswerId="7930" CreationDate="2011-03-06T17:18:02.327" Score="5" ViewCount="316" Body="&lt;p&gt;I played around with some unit root testing in R and I am not entirely sure what to make of the k lag parameter. I used the augmented &lt;em&gt;Dickey Fuller test&lt;/em&gt; and the &lt;em&gt;Philipps Perron test&lt;/em&gt; from the &lt;a href=&quot;http://cran.r-project.org/web/packages/tseries/index.html&quot; rel=&quot;nofollow&quot;&gt;tseries&lt;/a&gt; package. Obviously the default $k$ parameter (for the &lt;code&gt;adf.test&lt;/code&gt;) depends only on the length of the series. If I choose different $k$-values I get pretty different results wrt. rejecting the null: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Dickey-Fuller = -3.9828, Lag order = 4, p-value = 0.01272&#xA;alternative hypothesis: stationary &#xA;# 103^(1/3)=k=4 &#xA;&#xA;&#xA;Dickey-Fuller = -2.7776, Lag order = 0, p-value = 0.2543&#xA;alternative hypothesis: stationary&#xA;# k=0&#xA;&#xA;Dickey-Fuller = -2.5365, Lag order = 6, p-value = 0.3542&#xA;alternative hypothesis: stationary&#xA;# k=6&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;plus the PP test result:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Dickey-Fuller Z(alpha) = -18.1799, Truncation lag parameter = 4, p-value = 0.08954&#xA;alternative hypothesis: stationary &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Looking at the data, I would think the underlying data is non-stationary, but still I do not consider these results a strong backup, in particular since I do not understand the role of the $k$ parameter. If I look at decompose / stl I see that the trend has strong impact as opposed to only small contribution from remainder or seasonal variation. My series is of quarterly frequency.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any hints?&lt;/p&gt;&#xA;" OwnerUserId="704" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-03-06T18:47:18.750" LastActivityDate="2011-03-06T19:40:52.517" Title="Understanding the k lag in R's augmented Dickey Fuller test" Tags="&lt;r&gt;&lt;time-series&gt;&lt;trend&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="7930" PostTypeId="2" ParentId="7929" CreationDate="2011-03-06T19:07:39.157" Score="2" ViewCount="" Body="&lt;p&gt;It's been a while since I looked at ADF tests, however I do remember at least two versions of the adf test.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.stat.ucl.ac.be/ISdidactique/Rhelp/library/tseries/html/adf.test.html&quot; rel=&quot;nofollow&quot;&gt;http://www.stat.ucl.ac.be/ISdidactique/Rhelp/library/tseries/html/adf.test.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://cran.r-project.org/web/packages/fUnitRoots/&quot; rel=&quot;nofollow&quot;&gt;http://cran.r-project.org/web/packages/fUnitRoots/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The fUnitRoots package has a function called adfTest().   I think the &quot;trend&quot; issue is handled differently in those packages.    &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit ------ From page 14 of the following link, there were 4 versions (uroot discontinued) of the adf test:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://math.uncc.edu/~zcai/FinTS.pdf&quot; rel=&quot;nofollow&quot;&gt;http://math.uncc.edu/~zcai/FinTS.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One more link.  Read section 6.3 in the following link. It does a far btter job than I could do in explaining the lag term:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.yats.com/doc/cointegration-en.html&quot; rel=&quot;nofollow&quot;&gt;http://www.yats.com/doc/cointegration-en.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, I would be careful with any seasonal model.   Unless you're sure there's some seasonality present, I would avoid using seasonal terms.   Why?   Anything can be broken down into seasonal terms, even if it's not.   Here are two examples:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#First example: White noise&#xA;x &amp;lt;- rnorm(200)&#xA;&#xA;#Use stl() to separate the trend and seasonal term&#xA;x.ts &amp;lt;- ts(x, freq=4) &#xA;x.stl &amp;lt;- stl(x.ts, s.window = &quot;periodic&quot;)&#xA;plot(x.stl)&#xA;&#xA;#Use decompose() to separate the trend and seasonal term&#xA;x.dec &amp;lt;- decompose(x.ts)&#xA;plot(x.dec)&#xA;&#xA;#===========================================&#xA;&#xA;#Second example, MA process&#xA;x1 &amp;lt;- cumsum(x)&#xA;&#xA;#Use stl() to separate the trend and seasonal term&#xA;x1.ts &amp;lt;- ts(x1, freq=4)&#xA;x1.stl &amp;lt;- stl(x1.ts, s.window = &quot;periodic&quot;)&#xA;plot(x1.stl)&#xA;&#xA;#Use decompose() to separate the trend and seasonal term&#xA;x1.dec &amp;lt;- decompose(x1.ts)&#xA;plot(x1.dec)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The graph below is from the above plot(x.stl) statement.   stl() found a small seasonal term in white noise.   You might say that term is so small that it's really not an issue.   The problem is, in real data, you don't know if that term is a problem or not.  In the example below, notice that the trend data series has segments where it looks like a filtered version of the raw data, and other segments where it might be considered significantly different than the raw data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/wtW2G.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="2775" LastEditorUserId="2775" LastEditorDisplayName="" LastEditDate="2011-03-06T19:40:52.517" LastActivityDate="2011-03-06T19:40:52.517" />
  <row Id="7931" PostTypeId="2" ParentId="7815" CreationDate="2011-03-06T20:31:50.597" Score="2" ViewCount="" Body="&lt;p&gt;I would also add that the large scale data also introduces the problem of potential &quot;Bad data&quot;. Not only missing data, but data errors and inconsistent definitions introduced by every piece of a system which ever touched the data.  So, in additional to statistical skills, you need to become an expert data cleaner, unless someone else is doing it for you.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;-Ralph Winters&lt;/p&gt;&#xA;" OwnerUserId="3489" LastActivityDate="2011-03-06T20:31:50.597" CommentCount="1" />
  <row Id="7932" PostTypeId="2" ParentId="7815" CreationDate="2011-03-06T21:38:24.733" Score="4" ViewCount="" Body="&lt;p&gt;Good programming skills are a must.  You need to be able to write efficient code that can deal with huge amounts of data without choking, and maybe be able to parallelize said code to get it to run in a reasonable amount of time.&lt;/p&gt;&#xA;" OwnerUserId="1347" LastActivityDate="2011-03-06T21:38:24.733" CommentCount="3" />
  <row Id="7933" PostTypeId="2" ParentId="7815" CreationDate="2011-03-06T22:16:42.373" Score="18" ViewCount="" Body="&lt;p&gt;Good answers have already appeared.  I will therefore just share some thoughts based on personal experience: adapt the relevant ones to your own situation as needed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For &lt;strong&gt;background and context&lt;/strong&gt;--so you can account for any personal biases that might creep in to this message--much of my work has been in helping people make important decisions based on relatively small datasets.  They are small because the data can be expensive to collect (10K dollars for the first sample of a groundwater monitoring well, for instance, or several thousand dollars for analyses of unusual chemicals).  I'm used to getting as much as possible out of any data that are available, to exploring them to death, and to inventing new methods to analyze them if necessary.  However, in the last few years I have been engaged to work on some fairly large databases, such as one of socioeconomic and engineering data covering the entire US at the Census block level (8.5 million records, 300 fields) and various large GIS databases (which nowadays can run from gigabytes to hundreds of gigabytes in size).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;With very large datasets one's entire approach and mindset change&lt;/strong&gt;.  There are now &lt;em&gt;too much&lt;/em&gt; data to analyze.  Some of the immediate (and, in retrospect) obvious implications (with emphasis on regression modeling) include&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Any analysis you think about doing can take a lot of time and computation.  You will need to &lt;strong&gt;develop methods of subsampling&lt;/strong&gt; and working on partial datasets so you can plan your workflow when computing with the entire dataset.  (Subsampling can be complicated, because you need a representative subset of the data that is as rich as the entire dataset.  And don't forget about &lt;strong&gt;cross-validating&lt;/strong&gt; your models with the held-out data.)&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Because of this, you will &lt;strong&gt;spend more time documenting&lt;/strong&gt; what you do and scripting everything (so that it can be repeated).  &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;As @dsimcha has just noted, &lt;strong&gt;good programming skills&lt;/strong&gt; are useful.  Actually, you don't need much in the way of experience with programming environments, but you need a willingness to program, the ability to recognize when programming will help (at just about every step, really) and a good understanding of basic elements of computer science, such as design of appropriate data structures and how to analyze computational complexity of algorithms.  That's useful for knowing &lt;em&gt;in advance&lt;/em&gt; whether code you plan to write will scale up to the full dataset.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Some datasets are large because they have many variables (thousands or tens of thousands, all of them different).  &lt;strong&gt;Expect to spend a great deal of time just summarizing and understanding the data&lt;/strong&gt;.  A &lt;a href=&quot;http://nces.ed.gov/nationsreportcard/tdw/database/codebooks.asp&quot; rel=&quot;nofollow&quot;&gt;codebook&lt;/a&gt; or &lt;a href=&quot;http://en.wikipedia.org/wiki/Data_dictionary&quot; rel=&quot;nofollow&quot;&gt;data dictionary&lt;/a&gt;, and other forms of &lt;a href=&quot;http://en.wikipedia.org/wiki/Metadata&quot; rel=&quot;nofollow&quot;&gt;metadata&lt;/a&gt;, become essential.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Much of your time is spent simply moving data around and reformatting them.  You need &lt;strong&gt;skills with processing large databases&lt;/strong&gt; and skills with summarizing and graphing large amounts of data.  (&lt;a href=&quot;http://en.wikipedia.org/wiki/Small_multiple&quot; rel=&quot;nofollow&quot;&gt;Tufte's Small Multiple&lt;/a&gt; comes to the fore here.)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Some of your favorite software tools will fail.  Forget spreadsheets, for instance.  A lot of open source and academic software will just not be up to handling large datasets: the processing will take forever or the software will crash.  Expect this and &lt;strong&gt;make sure you have multiple ways to accomplish your key tasks.&lt;/strong&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Almost any statistical test you run will be so powerful that it's almost sure to identify a &quot;significant&quot; effect.  &lt;strong&gt;You have to focus much more on statistical &lt;em&gt;importance&lt;/em&gt;&lt;/strong&gt;, such as effect size, rather than significance.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Similarly, model selection is troublesome because almost any variable and any interaction you might contemplate is going to look significant.  &lt;strong&gt;You have to focus more on the &lt;em&gt;meaningfulness&lt;/em&gt; of the variables&lt;/strong&gt; you choose to analyze.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;There will be more than enough information to &lt;strong&gt;identify appropriate nonlinear transformations&lt;/strong&gt; of the variables.  Know how to do this.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;You will have enough data to &lt;strong&gt;detect nonlinear relationships, changes in trends, nonstationarity, heteroscedasticity&lt;/strong&gt;, etc.  &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;You will never be finished&lt;/strong&gt;.  There are so much data you could study them forever.  It's important, therefore, to establish your analytical objectives at the outset and constantly keep them in mind.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I'll end with a &lt;strong&gt;short anecdote&lt;/strong&gt; which illustrates one unexpected difference between regression modeling with a large dataset compared to a smaller one.  At the end of that project with the Census data, a regression model I had developed needed to be implemented in the client's computing system, which meant writing SQL code in a relational database.  This is a routine step but the code generated by the database programmers involved thousands of lines of SQL.  This made it almost impossible to guarantee it was bug free--although we could detect the bugs (it gave different results on test data), finding them was another matter.  (All you need is one typographical error in a coefficient...)  Part of the solution was to &lt;em&gt;write a program that generated the SQL commands directly from the model estimates&lt;/em&gt;.  This assured that what came out of the statistics package was exactly what went into the RDBMS.  As a bonus, a few hours spent on writing this script replaced possibly several weeks of SQL coding and testing.  This is a small part of what it means for the statistician to be able to communicate their results.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-07T00:07:36.283" LastActivityDate="2011-03-07T00:07:36.283" CommentCount="4" />
  <row Id="7934" PostTypeId="2" ParentId="5007" CreationDate="2011-03-06T23:17:04.847" Score="1" ViewCount="" Body="&lt;p&gt;Besides the well known and excellent documentation on Hadley's page, the not-so-known &lt;a href=&quot;https://github.com/hadley/ggplot2/wiki/%2Bopts%28%29-List&quot; rel=&quot;nofollow&quot;&gt;reference on Git Hub&lt;/a&gt; really helps with these kind of issues.&lt;/p&gt;&#xA;" OwnerUserId="704" LastActivityDate="2011-03-06T23:17:04.847" />
  <row Id="7935" PostTypeId="1" AcceptedAnswerId="7938" CreationDate="2011-03-06T23:21:24.703" Score="7" ViewCount="198" Body="&lt;p&gt;From what I know, using lasso for variable selection handles the problem of correlated inputs. Also, since it is equivalent to Least Angle Regression, it is not slow computationally. However, many people (for example people I know doing bio-statistics) still seem to favour stepwise or stagewise variable selection. Are there any practical disadvantages of using the lasso that makes it unfavourable?&lt;/p&gt;&#xA;" OwnerUserId="2973" LastActivityDate="2011-03-07T00:58:47.170" Title="What are disadvantages of using the lasso for variable selection for regression?" Tags="&lt;regression&gt;&lt;lasso&gt;" AnswerCount="3" FavoriteCount="1" />
  <row Id="7936" PostTypeId="2" ParentId="7935" CreationDate="2011-03-07T00:23:32.770" Score="2" ViewCount="" Body="&lt;p&gt;One practical disadvantage of lasso and other regularization techniques is finding the optimal regularization coefficient, lambda. Using cross validation to find this value can be just as expensive as stepwise selection techniques. &lt;/p&gt;&#xA;" OwnerUserId="2965" LastActivityDate="2011-03-07T00:23:32.770" />
  <row Id="7937" PostTypeId="2" ParentId="7935" CreationDate="2011-03-07T00:38:38.003" Score="0" ViewCount="" Body="&lt;p&gt;One big one is the difficulty of doing hypothesis testing.  You can't easily figure out which variables are statistically significant with Lasso.  With stepwise regression, you can do hypothesis testing to some degree, if you're careful about your treatment of multiple testing.&lt;/p&gt;&#xA;" OwnerUserId="1347" LastActivityDate="2011-03-07T00:38:38.003" CommentCount="3" />
  <row Id="7938" PostTypeId="2" ParentId="7935" CreationDate="2011-03-07T00:58:47.170" Score="3" ViewCount="" Body="&lt;p&gt;There is NO reason to do stepwise selection.  It's just wrong.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;LASSO/LAR are the best automatic methods.  But they are automatic methods.  They let the analyst not think.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In many analyses, some variables should be in the model REGARDLESS of any measure of significance.  Sometimes they are necessary control variables.  Other times, finding a small effect can be substantively important.&lt;/p&gt;&#xA;" OwnerUserId="686" LastActivityDate="2011-03-07T00:58:47.170" />
  <row Id="7939" PostTypeId="1" CreationDate="2011-03-07T01:06:05.880" Score="4" ViewCount="245" Body="&lt;p&gt;I'm working on a problem from &lt;em&gt;&quot;The Elements of Statistical Learning&quot;&lt;/em&gt; (prob. 6.8):&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Suppose that for continuous response&#xA;  $Y$ and predictor $X$, we model the&#xA;  joint density of $X, Y$ using a&#xA;  multivariate Gaussian kernel&#xA;  estimator. Note that the kernel in&#xA;  this case would be the product kernel&#xA;  $\phi_{\lambda}(X) \phi_{\lambda}(Y)$.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;(a) Show that the conditional mean $E(Y|X)$&#xA;  derived from this estimate is a&#xA;  Nadaraya-Watson estimator. &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;(b) Extend this&#xA;  result to classification by providing&#xA;  a suitable kernel for the estimation&#xA;  of the joint distribution of a&#xA;  continuous $X$ and discrete $Y$.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I know that the Nadaraya-Watson estimator is just the weighted average (equation 2.41 and 6.2 in ESL):&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;$$\hat f (x_0) = \frac{\sum_{i=0}^N K_{\lambda}(x_0, x_i) y_i}{\sum_{i=0}^N K_{\lambda}(x_0, x_i)}$$&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Where $K$ in this case would be the multivariate Gaussian kernel function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can think about how to extend this to a classification problem, but am not sure how to approach the first part of this question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any pointers would be greatly appreciated!&lt;/p&gt;&#xA;" OwnerUserId="988" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-29T01:04:13.537" LastActivityDate="2011-04-29T01:04:13.537" Title="Gaussian kernel estimator as Nadaraya-Watson estimator?" Tags="&lt;classification&gt;&lt;homework&gt;&lt;kernel&gt;" AnswerCount="1" />
  <row Id="7940" PostTypeId="2" ParentId="6225" CreationDate="2011-03-07T01:52:33.090" Score="0" ViewCount="" Body="&lt;p&gt;Yes, it is possible to prove the null--in exactly the same sense that it is possible to prove any alternative to the null. In a Bayesian analysis, it is perfectly possible for the odds in favor of the null versus any of the proposed alternatives to it to become arbitrarily large. Moreover, it is false to assert, as some of the above answers assert, that one can only prove the null if the alternatives to it are disjoint (do not overlap with the null). In a Bayesian analysis every hypothesis has a prior probability distribution. This distribution spreads a unit mass of prior probability out over the proposed alternatives. The null hypothesis puts all of the prior probability on a single alternative. In principle, alternatives to the null may put all of the prior probability on some non-null alternative (on another &quot;point&quot;), but this is rare. In general, alternatives hedge, that is, they spread the same mass of prior probability out over other alternatives--either to the exclusion of the null alternative, or, more commonly, including the null alternative. The question then becomes which hypothesis puts the most prior probability where the experimental data actually fall. If the data fall tightly around where the null says they should fall, then it will be the odds-on favority (among the proposed hypotheses) EVEN THOUGH IT IS INCLUDED IN (NESTED IN, NOT MUTUALLY EXCLUSIVE WITH) THE ALTERNATIVES TO IT. The believe that it is not possible for a nested alternative to be more likely than the set in which it is nested reflects the failure to distinguish between probability and likelihood. While it is impossible for a component of a set to be less probable than the entire set, it is perfectly possible for the posterior likelihood of a component of a set of hypotheses to be greater than the posterior likelihood of the set as a whole. The posterior likelihood of an hypothesis is the product of the likelihood function and the prior probability distribution that the hypothesis posits. If an hypothesis puts all of the prior probability in the right place (e.g., on the null), then it will have a higher posterior likelihood than an hypothesis that puts some of the prior probability in the wrong place (not on the null).&lt;/p&gt;&#xA;" OwnerUserId="3581" LastActivityDate="2011-03-07T01:52:33.090" />
  <row Id="7941" PostTypeId="1" AcceptedAnswerId="7942" CreationDate="2011-03-07T02:49:18.230" Score="3" ViewCount="107" Body="&lt;p&gt;I am not in statistics field. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have seen the word &quot;tied data&quot; while reading about Rank Correlation Coefficients.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What is tied data?&lt;/li&gt;&#xA;&lt;li&gt;What is an example of tied data?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="3584" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-03-07T04:55:58.390" LastActivityDate="2011-03-07T04:55:58.390" Title="What is tied data in the context of a rank correlation coefficient?" Tags="&lt;ranking&gt;" AnswerCount="3" CommentCount="1" />
  <row Id="7942" PostTypeId="2" ParentId="7941" CreationDate="2011-03-07T02:57:35.370" Score="1" ViewCount="" Body="&lt;p&gt;It means data that have the same value; for instance if you have 1,2,3,3,4 as the dataset then the two 3's are tied data. If you have 1,2,3,4,5,5,5,6,7,7 as the dataset then the 5's and the 7's are tied data.&lt;/p&gt;&#xA;" OwnerUserId="3585" LastActivityDate="2011-03-07T02:57:35.370" />
  <row Id="7943" PostTypeId="2" ParentId="7941" CreationDate="2011-03-07T02:58:09.177" Score="2" ViewCount="" Body="&lt;p&gt;It's simply two identical data values, such as observing 7 twice in the same data set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This comes up in the context of statistical methods that assume data has a continuous and so identical measurements are impossible (or technically, the probability identical values is zero). Practical complications arise when these methods are applied to data that are rounded or clipped so that identical measurements are not only possible but fairly common.&lt;/p&gt;&#xA;" OwnerUserId="319" LastActivityDate="2011-03-07T02:58:09.177" />
  <row Id="7944" PostTypeId="2" ParentId="7941" CreationDate="2011-03-07T03:02:06.560" Score="7" ViewCount="" Body="&lt;p&gt;&quot;Tied data&quot; comes up in the context of rank-based non-parametric statistical tests.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Non-parametric tests&lt;/strong&gt;: testing that does not assume a particular probability distribution, eg it does not assume a bell-shaped curve. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;rank-based&lt;/strong&gt;: a large class of non-parametric tests start by converting the numbers (eg &quot;3 days&quot;, &quot;5 days&quot;, and &quot;4 days&quot;) into ranks (eg &quot;shortest duration (3rd)&quot;, &quot;longest duration (1st)&quot;, &quot;second longest duration (2nd)&quot;). A traditional parametric testing method is then applied to these ranks. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Tied data&lt;/strong&gt; is an issue since numbers that are identical now need to be converted into rank. Sometimes ranks are randomly assigned, sometimes an average rank is used. Most importantly, a protocol for breaking tied ranks needs to be described for reproducibility of the result. &lt;/p&gt;&#xA;" OwnerUserId="812" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-07T03:49:19.673" LastActivityDate="2011-03-07T03:49:19.673" CommentCount="1" />
  <row Id="7945" PostTypeId="2" ParentId="7727" CreationDate="2011-03-07T04:29:43.617" Score="2" ViewCount="" Body="&lt;p&gt;The question concerns &lt;strong&gt;calculating the correlation between two irregularly sampled time series&lt;/strong&gt; (one-dimensional stochastic processes) and using that to find the time offset where they are maximally correlated (their &quot;phase difference&quot;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This problem is not usually addressed in time series analysis, because time series data are presumed to be collected systematically (at regular intervals of time).  It is rather the province of &lt;a href=&quot;http://en.wikipedia.org/wiki/Geostatistics&quot; rel=&quot;nofollow&quot;&gt;geostatistics&lt;/a&gt;, which concerns the multidimensional generalizations of time series.  The archetypal geostatistical dataset consists of measurements of geological samples at irregularly spaced locations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With irregular spacing, the distances among pairs of locations vary: no two distances may be the same.  Geostatistics overcomes this with the &lt;a href=&quot;http://en.wikipedia.org/wiki/Variogram#Empirical_variogram&quot; rel=&quot;nofollow&quot;&gt;empirical variogram&lt;/a&gt;.  This computes a &quot;typical&quot; (often the mean or median) value of $(z(p) - z(q))^2 / 2$--the &quot;semivariance&quot;--where $z(p)$ denotes a measured value at point $p$ and the distance between $p$ and $q$ is constrained to lie within an interval called a &quot;lag&quot;.  If we assume the process $Z$ is stationary and has a covariance, then the expectation of the semivariance equals the maximum covariance (equal to $Var(Z(p))$ for any $p$) minus the covariance between $Z(p)$ and $Z(q)$.  This binning into lags copes with the irregular spacing problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When an ordered pair of measurements $(z(p), w(p))$ is made at each point, one can similarly compute the empirical cross-variogram between the $z$'s and the $w$'s and thereby estimate the covariance at any lag.  &lt;strong&gt;You want the one-dimensional version of the cross-variogram.&lt;/strong&gt;  The &lt;strong&gt;R&lt;/strong&gt; packages &lt;a href=&quot;http://www.gstat.org/&quot; rel=&quot;nofollow&quot;&gt;gstat&lt;/a&gt; and &lt;a href=&quot;http://www.stat.ucl.ac.be/ISdidactique/Rhelp/library/sgeostat/html/est.variogram.html&quot; rel=&quot;nofollow&quot;&gt;sgeostat&lt;/a&gt;, among others, will estimate cross-variograms.  Don't worry that your data are one-dimensional; if the software won't work with them directly, just introduce a constant second coordinate: that will make them appear two-dimensional.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With two million points you should be able to detect small deviations from stationarity.  It's possible the phase difference between the two time series could vary over time, too.  Cope with this by computing the cross-variogram separately for different windows spaced throughout the time period.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;@cardinal has already brought up most of these points in comments.  The main contribution of this reply is to point towards the use of spatial statistics packages to do your work for you and to use techniques of geostatistics to analyze these data.  As far as computational efficiency goes, note that the full convolution (cross-variogram) is not needed: you only need its values near the phase difference.  This makes the effort $O(nk)$, not $O(n^2)$, where $k$ is the number of lags to compute, so it might be feasible even with out-of-the-box software.  If not, the direct convolution algorithm is easy to implement.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-03-07T04:29:43.617" CommentCount="9" />
  <row Id="7946" PostTypeId="1" AcceptedAnswerId="8208" CreationDate="2011-03-07T05:39:06.227" Score="3" ViewCount="242" Body="&lt;p&gt;I'm not in Statistics field. I conducted the case study and collected the data as shown below&#xA;I have data as shown in the table below:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/mH1tC.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/ExRl1.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to find correlation coefficient from this two table data(between &lt;strong&gt;NOA and HVOC&lt;/strong&gt;, and between &lt;strong&gt;NOA and HVOL&lt;/strong&gt;). I conducted the case study with the source code. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I measured software metrics named &quot;NOA&quot; and &quot;HVOL&quot; for all the method/function before I modified this source code. And then, after I modified the code, I again measureed the same metrics for all the method. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;NOA Diff&lt;/strong&gt; field in the table is calculated from NOA (after modifying the code) minus NOA (before modifying the code). That is &quot;NOA Diff = NOA(after)-NOA(before)&quot;. The same way was applied to HVOC metric; &lt;strong&gt;HVOC Diff&lt;/strong&gt; = HVOC(after)-HVOC(before)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My questions are &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What type of &lt;strong&gt;correlation coefficient&lt;/strong&gt; should I use?&lt;/li&gt;&#xA;&lt;li&gt;What &lt;strong&gt;kind of graph&lt;/strong&gt; should I create to illustrate my data?&lt;/li&gt;&#xA;&lt;li&gt;The table above is all data, i mean it's &lt;strong&gt;population&lt;/strong&gt; not a sample, can I use the method that is used with a &lt;strong&gt;sample&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;li&gt;Is Spearman is for non normally distributed data?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="3584" LastEditorUserId="3584" LastEditorDisplayName="" LastEditDate="2011-03-19T04:38:14.770" LastActivityDate="2011-03-21T15:39:29.850" Title="What correlation coefficient and graph is appropriate with this data?" Tags="&lt;data-visualization&gt;&lt;spearman&gt;&lt;ranks&gt;" AnswerCount="2" CommentCount="6" FavoriteCount="1" />
  <row Id="7947" PostTypeId="2" ParentId="7923" CreationDate="2011-03-07T06:58:33.697" Score="1" ViewCount="" Body="&lt;p&gt;As chl says, the general issue of what statistical test to use when the dependent variable is a scale based on likert items has been discussed &lt;a href=&quot;http://stats.stackexchange.com/questions/203/group-differences-on-a-five-point-likert-item&quot;&gt;elsewhere on this site&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the pragmatic task of running such analyses in SPSS, typing &quot;spss t-test&quot; or &quot;spss Mann-Whitney&quot; into Google will give you many SPSS tutorial options.&#xA;Check out for example, some of the following tutorial sites:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www2.gsu.edu/~eslnxj/spss/spssmean.html&quot; rel=&quot;nofollow&quot;&gt;gsu&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://academic.uofs.edu/department/psych/methods/cannon99/level2b.html&quot; rel=&quot;nofollow&quot;&gt;uofs&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastActivityDate="2011-03-07T06:58:33.697" />
  <row Id="7948" PostTypeId="1" AcceptedAnswerId="7950" CreationDate="2011-03-07T09:14:00.487" Score="11" ViewCount="248" Body="&lt;p&gt;I am running linear regression models and wondering what the conditions are for removing the intercept term of &lt;code&gt;lm()&lt;/code&gt;? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In comparing results from two different lm functions where one has the intercept and the other does not, I notice that the $R^2$ of the function without the intercept is much higher. Are there certain conditions or assumptions I should be following to make sure the removal of the intercept term is valid? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any advice is appreciated. &lt;/p&gt;&#xA;" OwnerUserId="1422" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-03-07T09:23:14.337" LastActivityDate="2011-03-07T10:16:46.850" Title="When is it ok to remove the intercept in lm()?" Tags="&lt;r&gt;&lt;regression&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="3" />
  <row Id="7949" PostTypeId="2" ParentId="7912" CreationDate="2011-03-07T09:19:14.240" Score="2" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Intro&lt;/strong&gt; As @vqv mentionned Total variation and Kullback Leibler are two interesting distance. The first one is meaningfull because it can be directly related to first and second type errors in hypothesis testing. The problem with the Total variation distance is that it can be difficult to compute. The Kullback Leibler distance is easier to compute and I will come to that later. It is not symetric but can be made symetric (somehow a little bit artificially).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Answer&lt;/strong&gt;  Something I mention &lt;a href=&quot;http://mathoverflow.net/questions/29054/l1-distance-between-gaussian-measures&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; is that if $\mathcal{L}$ is the log likelihood ratio between your two gaussian measures $P_0,P_1$ (say that for $i=0,1$ $P_i$ has mean $\mu_i$ and covariance $C_i$) error measure that is also interseting (in the gaussian case I found it quite central actually) is &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ \|\mathcal{L}\|^2_{L_2(P_{1/2})} $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;for a well chosen $P_{1/2}$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;In simple words&lt;/strong&gt;: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;there might be different interesting &quot;directions&quot; rotations, that are obtained using your formula with one of the &quot;interpolated&quot; covariance matrices $\Sigma=C_{i,1/2}$ ($i=1,2,3,4$ or $5$) defined at the end of this post (the number $5$ is the one you propose in your comment to your question).  &lt;/li&gt;&#xA;&lt;li&gt;since your two distributions have different covariances, it is &lt;strong&gt;not sufficiant to compare the means&lt;/strong&gt;, you also need to compare the covariances.  &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Let me explain you why this is my feeling, how you can compute this in the case of $C_1\neq C_0$ and how to choose $P_{1/2}$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Linear case&lt;/strong&gt; If $C_1=C_0=\Sigma$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\sigma= \Delta \Sigma^{-1} \Delta=\|2\mathcal{L}\|^2_{L_2(P_{1/2})}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $P_{1/2}$ is the &quot;interpolate&quot; between $P_1$ and $P_0$ (gaussian with covariance $\Sigma$ and mean $(\mu_1+\mu_0)/2$). Note that in this case, the Hellinger distance, the total variation distance can all be written using $\sigma$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How to compute $\mathcal{L}$ in the general case&lt;/strong&gt; A natural question that arises from your question (and &lt;a href=&quot;http://mathoverflow.net/questions/29054/l1-distance-between-gaussian-measures&quot; rel=&quot;nofollow&quot;&gt;mine&lt;/a&gt;) is what is a natural &quot;interpolate&quot; between $P_1$ and $P_0$ when $C_1\neq C_0$. Here the word natural may be user specific but for example it may be related to the best interpolation to have a tight upper bound with another distance (e.g. $L_1$ distance &lt;a href=&quot;http://mathoverflow.net/questions/29054/l1-distance-between-gaussian-measures&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Writting &#xA;$$ \mathcal{L}= \phi (C^{-1/2}_i(x-\mu_i))-\phi (C^{-1/2}_j(x-\mu_j))-\frac{1}{2}\log \left ( C_iC_j^{-}\right )&#xD;&#xA; $$&#xA;($i=0,j=1$)&#xA;may help to see where is the interpolation task, but : &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\mathcal{L}(x)=-\frac{1}{2}\langle&#xD;&#xA;A_{ij}(x-s_{ij}),x-s_{ij}\rangle_{\mathbb{R}^p}+\langle&#xD;&#xA;G_{ij},x-s_{ij}\rangle_{\mathbb{R}^p}-c_{ij}, \;[1]$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;with &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$A_{ij}=C_i^{-}-C_j^{-},\;\; G_{ij}=S_{ij}m_{ij},\;\; S_{ij}=\frac{C_i^{-}+C_j^{-}}{2}, $$&#xA;$$  c_{ij}=\frac{1}{8}\langle A_{ij}&#xD;&#xA;m_{ij},m_{ij}\rangle_{\mathbb{R}^p}+\frac{1}{2}\log|\det(C_j^{-}C_i)| $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ m_{ij}=\mu_i-\mu_j \;\; and\;\; s_{ij}=\frac{\mu_i+\mu_j}{2}$$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;is more relevant for computational purpose. &#xA;For any gaussian $P_{1/2}$ with mean $s_{01}$ and covariance $C$ the calculation of $\|\mathcal{L}\|^2_{L_2(P_{1/2})}$ from Equation $1$ is a bit technical but faisible. You might also use it to compute the Kulback leibler distance. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What interpolation should we choose (i.e. how to choose $P_{1/2}$)&lt;/strong&gt;&#xA;It is clearly understood from Equation $1$ that there are many different candidates for $P_{1/2}$ (interpolate) in the &quot;quadratic&quot; case. The two candidates I found &quot;most natural&quot; (subjective:) ) arise from defining for $t\in [0,1]$ a gaussian distribution $P_t$ with mean $t\mu_1+(1-t)\mu_0$:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;$P^1_t$ as the distribution of $$ \xi_t=t\xi_1+(1-t)\xi_0$$  (where $\xi_i$ is drawn from $P_i$ $i=0,1$) which has covariance $C_{t,1}=(tC_1^{1/2}+(1-t)C_0^{1/2})^2$). &lt;/li&gt;&#xA;&lt;li&gt;$P^2_t$ with inverse covariance $C_{t,2}^{-1}=tC_{1}^{-1}+(1-t)C_0^{-1}$ &lt;/li&gt;&#xA;&lt;li&gt;$P^3_t$ with covariance $C_{t,3}=tC_1+(1-t)C_0$&lt;/li&gt;&#xA;&lt;li&gt;$P^4_t$ with inverse covariance $C_{t,4}^{-1}=(tC^{-1/2}_1+(1-t)C^{-1/2}_0)^{2}$&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;EDIT: The one you propose in a comment to your question could be $C_{t,5}=C_1^{t}C_0^{1-t}$, why not ... &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have my favorite choice which is not the first one :) don't have much time to discuss that here. Maybe I'll edit this answer later... &lt;/p&gt;&#xA;" OwnerUserId="223" LastEditorUserId="223" LastEditorDisplayName="" LastEditDate="2011-03-11T12:33:23.773" LastActivityDate="2011-03-11T12:33:23.773" />
  <row Id="7950" PostTypeId="2" ParentId="7948" CreationDate="2011-03-07T10:16:46.850" Score="13" ViewCount="" Body="&lt;p&gt;The &lt;strong&gt;shortest&lt;/strong&gt; answer: &lt;strong&gt;never&lt;/strong&gt;, unless you are &lt;em&gt;sure&lt;/em&gt; that your linear approximation of the data generating process (linear regression model) either by some theoretical or any other reasons &lt;em&gt;is forced to go through the origin&lt;/em&gt;. If not the other regression parameters will be biased even if intercept is statistically insignificant (strange but it is so, consult &lt;a href=&quot;http://www.cambridge.org/features/economics/brooks/&quot; rel=&quot;nofollow&quot;&gt;Brooks&lt;/a&gt; &lt;em&gt;Introductory Econometrics&lt;/em&gt; for instance). Finally, as I do often explain to my students, by leaving the intercept term you insure that the residual term is zero-mean.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For your two models case we need more context. It may happen that linear model is not suitable here. For example, you need to log transform first if the model is multiplicative. Having exponentially growing processes it may occasionally happen that $R^2$ for the model without the intercept is &quot;much&quot; higher. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Screen the data, test the model with RESET test or any other linear specification test, this may help to see if my guess is true. And, building the models highest $R^2$ is one of the last statistical properties I do really concern about, but it is nice to present to the people who are not so well familiar with econometrics (there are many dirty tricks to make determination close to 1 :)).&lt;/p&gt;&#xA;" OwnerUserId="2645" LastActivityDate="2011-03-07T10:16:46.850" CommentCount="1" />
  <row Id="7951" PostTypeId="1" CreationDate="2011-03-07T10:22:05.520" Score="1" ViewCount="120" Body="&lt;p&gt;I would like to find the correlation between two data set. I was suggested that two set of data should be independent, otherwise it is no meaning in term of Statistics to find a correlation. For example, one set of data is x and another is y which won't be calculated from x, e.g., y=ab/c+x. a, b, and c are some constant value that are same for all y&#xA;Question 1 what I understand is right?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Question 2 the data set that y = P+Q+x. P and Q depends on each x, and the same x value can have different value of P and Q. Is it meaningful to find a correlation of this data set.&lt;/p&gt;&#xA;" OwnerUserId="3584" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-07T15:32:55.757" LastActivityDate="2011-03-07T15:32:55.757" Title="Meaningful data set for finding a correlation" Tags="&lt;correlation&gt;&lt;dataset&gt;&lt;independence&gt;&lt;non-independent&gt;" CommentCount="3" />
  <row Id="7952" PostTypeId="1" AcceptedAnswerId="7953" CreationDate="2011-03-07T11:15:38.937" Score="5" ViewCount="209" Body="&lt;p&gt;I am using Sweave and &lt;strong&gt;xtable&lt;/strong&gt; to generate a report. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to add some coloring on a table. But I have not managed to find any way to generate colored tables with xtable. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any other option?&lt;/p&gt;&#xA;" OwnerUserId="1709" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-03-07T11:35:29.127" LastActivityDate="2011-03-08T10:30:23.020" Title="How to create coloured tables with Sweave and xtable?" Tags="&lt;r&gt;&lt;reproducible-research&gt;&lt;sweave&gt;" AnswerCount="1" CommentCount="5" />
  <row Id="7953" PostTypeId="2" ParentId="7952" CreationDate="2011-03-07T11:59:48.103" Score="6" ViewCount="" Body="&lt;p&gt;Although I didn't try this explicitly from with R (I usually post-process the Tables in Latex directly with &lt;code&gt;\rowcolor&lt;/code&gt;, &lt;code&gt;\rowcolors&lt;/code&gt;, or the &lt;a href=&quot;http://ctan.org/pkg/colortbl&quot; rel=&quot;nofollow&quot;&gt;colortbl&lt;/a&gt; package), I think it would be easy to do this by playing with the &lt;code&gt;add.to.row&lt;/code&gt; arguments in &lt;code&gt;print.xtable()&lt;/code&gt;. It basically expect two components (passed as &lt;code&gt;list&lt;/code&gt;): (1) row number, and (2) $\LaTeX$ command. Please note that command are added at the end of the specified row(s).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems to work, with the &lt;code&gt;colortbl&lt;/code&gt; package. So, something like this&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;lt;&amp;lt;result=tex&amp;gt;&amp;gt;&#xA;library(xtable)&#xA;m &amp;lt;- matrix(sample(1:10,10), nr=2)&#xA;print(xtable(m), add.to.row=list(list(1),&quot;\\rowcolor[gray]{.8} &quot;))&#xA;@&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;gives me&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/BkcUT.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(This is a customized Beamer template, but this should work with a standard document. With Beamer, you'll probably want to add the &lt;code&gt;table&lt;/code&gt; option when loading the package.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Following @Conjugate's suggestion, you can also rely on &lt;a href=&quot;http://cran.r-project.org/web/packages/Hmisc/index.html&quot; rel=&quot;nofollow&quot;&gt;Hmisc&lt;/a&gt; facilities for handling $\TeX$ output, see the many options of the &lt;code&gt;latex()&lt;/code&gt; function. Here is an example of use:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(Hmisc)&#xA;## print the second row in bold (including row label)&#xA;form.mat &amp;lt;- matrix(c(rep(&quot;&quot;, 5), rep(&quot;bfseries&quot;, 5)), nr=2, byrow=TRUE)&#xA;w1 &amp;lt;- latex(m, rownamesTexCmd=c(&quot;&quot;,&quot;bfseries&quot;), cellTexCmds=form.mat, &#xA;            numeric.dollar=FALSE, file='/tmp/out1.tex')&#xA;w1  # call latex on /tmp/out1.tex&#xA;## highlight the second row in gray (as above)&#xA;w2 &amp;lt;- latex(m, rownamesTexCmd=c(&quot;&quot;,&quot;rowcolor[gray]{.8}&quot;), &#xA;            numeric.dollar=FALSE, file='/tmp/out2.tex')&#xA;w2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-03-08T10:30:23.020" LastActivityDate="2011-03-08T10:30:23.020" CommentCount="2" />
  <row Id="7954" PostTypeId="2" ParentId="7607" CreationDate="2011-03-07T13:17:28.140" Score="2" ViewCount="" Body="&lt;p&gt;I think this is a good question and I don't kown much about implementations. Since wavelet is 'mutli-resolution' you have two types of solutions (which are somehow connected): &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Modify your signal&lt;/strong&gt; for example extend you signal over the actual boundary to have meaningfull coefficients.&#xA;Exemples of that are : &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;periodic wavelet on the interval&lt;/li&gt;&#xA;&lt;li&gt;Zero padding (extend the signal by zero outside ist domain&lt;/li&gt;&#xA;&lt;li&gt;finer prodecure are extensions of zero padding with smoothness condition at the boundary. &lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Modify the wavelet&lt;/strong&gt; (somehow equivalent to threshold or lower wavelet coefficient that are near the boundary). More generally, there are procedures I know there have been many work  since that of A Cohen I Daubechies et P Vial   1993. For example, in (Monasse and Perrier, 1995), wavelet that forms a basis adapted to conditions such as Dirichlet or Neumann are constructed. I guess some are implemented ? If you found implementations, I am interested. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;References:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Monasse and Perrier : 1995 CRAS Ondelettes sur lintervalle pour la prise en compte de conditions aux limites&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A Cohen  I Daubechies et P Vial  Wavelets on the interval and fast wavelet transforms    Appl Comp Harmonic Analysis  (1993)&lt;/p&gt;&#xA;" OwnerUserId="223" LastActivityDate="2011-03-07T13:17:28.140" CommentCount="2" />
  <row Id="7955" PostTypeId="1" CreationDate="2011-03-07T14:10:05.547" Score="1" ViewCount="63" Body="&lt;p&gt;Would you please give an intuitive illustration of Newton's Method, when we deal with nonlinear regression?&lt;br&gt;&#xA; Basically I understand that if we can use Taylor's theorem to expand the RSS function of parameter beta, we can change it into quadratic form, and minimize RSS w.r.t parameter. Please give me a multivariate example by using gradient and Hessian matrix.&lt;br&gt;&#xA;Sorry I intend to input equation and function here, but I have no idea how to use LaTeX code here. &lt;/p&gt;&#xA;" OwnerUserId="3525" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-07T14:55:08.810" LastActivityDate="2011-03-07T14:55:08.810" Title="The usage of Newton's method in nonlinear regression" Tags="&lt;econometrics&gt;&lt;nonlinear-regression&gt;" CommentCount="2" />
  <row Id="7956" PostTypeId="1" CreationDate="2011-03-07T14:24:12.427" Score="3" ViewCount="147" Body="&lt;p&gt;Starting out with arima models in R, I do not understand why fitted.values (of an AR(2) process for example) are not part of the output like they are in regressions. Did I miss them when running &lt;code&gt;str(result)&lt;/code&gt; or did I get something completely wrong? &lt;/p&gt;&#xA;" OwnerUserId="704" LastActivityDate="2011-03-08T01:26:36.423" Title="Why are fitted.values not part the R object returned from arima?" Tags="&lt;r&gt;&lt;time-series&gt;&lt;arima&gt;" AnswerCount="1" />
  <row Id="7957" PostTypeId="2" ParentId="7956" CreationDate="2011-03-07T15:10:22.120" Score="6" ViewCount="" Body="&lt;p&gt;Use &lt;code&gt;fitted()&lt;/code&gt; function from the &lt;code&gt;forecast&lt;/code&gt; package. Since arima object saves residuals it is easy to compute fitted values from it.&lt;/p&gt;&#xA;" OwnerUserId="2645" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2011-03-08T01:26:36.423" LastActivityDate="2011-03-08T01:26:36.423" CommentCount="4" />
  <row Id="7959" PostTypeId="1" CreationDate="2011-03-07T15:53:27.493" Score="4" ViewCount="107" Body="&lt;p&gt;Hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to estimate the quantile of some data. The data are so huge that they can not be accommodated in the memory. And data are not static, new data keep coming. Does anyone know any algorithm to monitor the quantiles of the data observed so far with very limited memory and computation? I find &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=742445C867F4FC0A1658981316B79CC3?doi=10.1.1.74.2672&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;P2 algorithm&lt;/a&gt; useful, but it does not work very well for my data, which are extremely heavy-tailed distributed.&lt;/p&gt;&#xA;" OwnerUserId="3594" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-07T16:56:57.453" LastActivityDate="2011-04-20T19:09:57.173" Title="Algorithm to dynamically monitor quantiles" Tags="&lt;quantiles&gt;" AnswerCount="4" CommentCount="2" FavoriteCount="2" />
  <row Id="7960" PostTypeId="1" AcceptedAnswerId="7967" CreationDate="2011-03-07T16:07:26.140" Score="13" ViewCount="232" Body="&lt;p&gt;The New York Times has a long comment on the 'value-added' teacher evaluation system being used to give feedback to New York City educators.  The lede is the equation used to calculate the scores - presented without context.  The rhetorical strategy appears to be intimidation via math:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://graphics8.nytimes.com/images/2011/03/07/education/07winerip_graphic/07winerip_graphic-articleLarge-v2.jpg&quot; alt=&quot;alt text&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The full text of the article is available at: &lt;a href=&quot;http://www.nytimes.com/2011/03/07/education/07winerip.html&quot; rel=&quot;nofollow&quot;&gt;http://www.nytimes.com/2011/03/07/education/07winerip.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The author, Michael Winerip, argues that the meaning of the equation is beyond the capacity of anyone other than, um, Matt Damon to understand, much less an average teacher:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&quot;The calculation for Ms. Isaacson’s 3.69 predicted score is even more daunting. It is based on 32 variables — including whether a student was “retained in grade before pretest year” and whether a student is “new to city in pretest or post-test year.”&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Those 32 variables are plugged into a statistical model that looks like one of those equations that in “Good Will Hunting” only Matt Damon was capable of solving.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;The process appears transparent, but it is clear as mud, even for smart lay people like teachers, principals and — I hesitate to say this — journalists.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Ms. Isaacson may have two Ivy League degrees, but she is lost. “I find this impossible to understand,” she said.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;In plain English, Ms. Isaacson’s best guess about what the department is trying to tell her is: Even though 65 of her 66 students scored proficient on the state test, more of her 3s should have been 4s.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;But that is only a guess.&quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;How would you explain the model to a layperson?  FYI, the full technical report is at:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://schools.nyc.gov/NR/rdonlyres/A62750A4-B5F5-43C7-B9A3-F2B55CDF8949/87046/TDINYCTechnicalReportFinal072010.pdf&quot; rel=&quot;nofollow&quot;&gt;http://schools.nyc.gov/NR/rdonlyres/A62750A4-B5F5-43C7-B9A3-F2B55CDF8949/87046/TDINYCTechnicalReportFinal072010.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Update: Andrew Gelman offers his thoughts here: &#xA;&lt;a href=&quot;http://www.stat.columbia.edu/~cook/movabletype/archives/2011/03/its_no_fun_bein.html&quot; rel=&quot;nofollow&quot;&gt;http://www.stat.columbia.edu/~cook/movabletype/archives/2011/03/its_no_fun_bein.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="3320" LastEditorUserId="3320" LastEditorDisplayName="" LastEditDate="2011-03-10T15:08:07.000" LastActivityDate="2011-03-10T15:08:07.000" Title="Equations in the news: Translating a multi-level model to a general audience" Tags="&lt;regression&gt;&lt;multilevel-analysis&gt;" AnswerCount="3" FavoriteCount="3" />
  <row Id="7961" PostTypeId="2" ParentId="6896" CreationDate="2011-03-07T17:08:51.720" Score="1" ViewCount="" Body="&lt;p&gt;What are you trying to do with the nearest neighbor information?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would answer that question, and then compare the different distance measures in light of that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, say you are trying to classify poses based on the joint configuration, and would like joint vectors from the same pose to be close together.  A straightforward way to evaluate the suitability of different distance metrics is to use each of them in a KNN classifier, and compare the out-of-sample accuracies of each of the resulting models.&lt;/p&gt;&#xA;" OwnerUserId="3595" LastActivityDate="2011-03-07T17:08:51.720" />
  <row Id="7962" PostTypeId="2" ParentId="7959" CreationDate="2011-03-07T17:16:58.733" Score="2" ViewCount="" Body="&lt;p&gt;This can be adapted from algorithms that determine the median of a dataset online.  For more information, see this stackoverflow post - &lt;a href=&quot;http://stackoverflow.com/questions/1387497/find-median-value-from-a-growing-set&quot;&gt;http://stackoverflow.com/questions/1387497/find-median-value-from-a-growing-set&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="3595" LastActivityDate="2011-03-07T17:16:58.733" />
  <row Id="7963" PostTypeId="2" ParentId="7959" CreationDate="2011-03-07T17:29:15.047" Score="5" ViewCount="" Body="&lt;p&gt;The P2 algorithm is a nice find.  It works by making several estimates of the quantile, updating them periodically, and using &lt;em&gt;quadratic&lt;/em&gt; (not linear, not cubic) interpolation to estimate the quantile.  The authors claim quadratic interpolation works better in the tails than linear interpolation and cubic would get too fussy and difficult.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You do not state exactly how this approach fails for your &quot;heavy-tailed&quot; data, but it's easy to guess: estimates of extreme quantiles for heavy-tailed distributions will be unstable until a large amount of data are collected.  But this is going to be a problem (to a lesser extent) even if you were to store all the data, so don't expect miracles!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At any rate, why not set auxiliary markers--let's call them $x_0$ and $x_6$--within which you are highly certain the quantile will lie, and store all data that lie between $x_0$ and $x_6$?  When your buffer fills you will have to update these markers, always keeping $x_0 \le x_6$.  A simple algorithm to do this can be devised from a combination of (a) the current P2 estimate of the quantile and (b) stored counts of the number of data less than $x_0$ and the number of data greater than $x_6$.  In this fashion you can, with high certainty, estimate the quantile just as well as if you had the entire dataset always available, but you only need a relatively small buffer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Specifically, I am proposing a data structure $(k, \mathbf{y}, n)$ to maintain partial information about a sequence of $n$ data values $x_1, x_2, \ldots, x_n$.  Here, $\mathbf{y}$ is a linked list&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\mathbf{y} = (x^{(n)}_{[k+1]} \le x^{(n)}_{[k+2]} \le \cdots \le x^{(n)}_{[k+m]}).$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this notation $x^{(n)}_{[i]}$ denotes the $i^\text{th}$ smallest of the $n$ $x$ values read so far.  $m$ is a constant, the size of the buffer $\mathbf{y}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The algorithm begins by filling $\mathbf{y}$ with the first $m$ data values encountered and placing them in sorted order, smallest to largest.  Let $q$ be the quantile to be estimated; e.g., $q$ = 0.99.  Upon reading $x_{n+1}$ there are three possible actions:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;If $x_{n+1} \lt x^{(n)}_{[k+1]}$, increment $k$.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If $x_{n+1} \gt x^{(n)}_{[k+m]}$, do nothing.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Otherwise, &lt;strong&gt;insert&lt;/strong&gt; $x_{n+1}$ into $\mathbf{y}$.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In any event, increment $n$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;strong&gt;insert&lt;/strong&gt; procedure puts $x_{n+1}$ into $\mathbf{y}$ in sorted order and then eliminates one of the extreme values in $\mathbf{y}$:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;If $k + m/2 \lt n q$, then remove $x^{(n)}_{[k+1]}$ from $\mathbf{y}$ and increment $k$;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Otherwise, remove $x^{(n)}_{[k+m]}$ from $\mathbf{y}$.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Provided $m$ is sufficiently large, this procedure will bracket the true quantile of the distribution with high probability.  At any stage $n$ it can be estimated in the usual way in terms of $x^{(n)}_{[\lfloor{q n}\rfloor]}$ and  $x^{(n)}_{[\lceil{q n}\rceil]}$, which will likely lie in $\mathbf{y}$.  (I believe $m$ only has to scale like the square root of the maximum amount of data ($N$), but I have not carried out a rigorous analysis to prove that.)  At any rate, the algorithm will detect whether it has succeeded (by comparing $k/n$ and $(k+m)/n$ to $q$).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Testing with up to 100,000 values, using $m = 2\sqrt{N}$ and $q=.5$ (the most difficult case) indicates this algorithm has a 99.5% success rate in obtaining the correct value of  $x^{(n)}_{[\lfloor{q n}\rfloor]}$.  For a stream of $N=10^{12}$ values, that would require a buffer of only two million (but three or four million would be a better choice).  Using a sorted doubly linked list for the buffer requires $O(\log(\sqrt{N}))$ = $O(\log(N))$ effort while identifying and deleting the max or min are $O(1)$ operations.  The relatively expensive insertion typically needs to be done only $O(\sqrt{N})$ times.  Thus the computational costs of this algorithm are $O(N + \sqrt{N} \log(N)) = O(N)$ in time and $O(\sqrt{N})$ in storage.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-03-07T17:29:15.047" CommentCount="8" />
  <row Id="7964" PostTypeId="2" ParentId="7774" CreationDate="2011-03-07T17:56:05.703" Score="4" ViewCount="" Body="&lt;p&gt;Although @cardinal has already given an answer that gives precisely the bound I was looking for, I have found a similar Chernoff-style argument that can give a stronger bound:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Proposition&lt;/strong&gt;:&#xA;$$&#xD;&#xA;Pr (T \leq n \log n - c n) \leq \exp(- \frac{3c^2}{\pi^2}  ) \&amp;gt; .&#xD;&#xA;$$&#xA;(this is stronger for $c &amp;gt; \frac{\pi^2}{3}$ )&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Proof&lt;/em&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As in @cardinal's answer, we can use the fact that $T$ is a sum of independent geometric random variables $T_i$ with success probabilities $p_i = 1 - i/n$. It follows that $E[T_i] = 1/p_i$ and $E[T] = \sum_{i=1}^{n} E[T_i] = n \sum_{i=1}^n \frac{1}{i}\geq n \log n$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Define now new variables $S_i  : = T_i  - E[T_i]$, and $S : = \sum_i S_i$. We can then write&#xA;$$&#xD;&#xA;\Pr (T \leq n \log n - c n)  \leq \Pr (T \leq E[T] - c n) = \Pr (S \leq - c n)&#xD;&#xA;$$&#xA;$$&#xD;&#xA;= \Pr\left(\exp(-s S ) \geq \exp( s cn) \right) \leq e^{-s c n} E\left[ e^{-s S} \right]&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Computing the averages, we have&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;E[e^{-s S}] = \prod_i E[e^{-s S_i}] =  \prod_i \frac{e^{s / p_i} } {1 + \frac{1}{p_i} (e^s -1)} &#xD;&#xA;    \leq e^{\frac{1}{2}s^2\sum_i p_i^{-2}}&#xD;&#xA;$$&#xA;where the inequality follows from the facts that $e^s - 1\geq s$ and also $\frac{e^z}{1+z}\leq e^{\frac{1}{2}z^2}$ for $z\geq 0$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, since $\sum_i p_i ^{-2} = n^2 \sum_{i=1}^{n-1} \frac{1}{i^2} \leq n^2 \pi^2/6$,  we can write&#xA;\begin{align*}&#xA;\Pr( T \leq n \log n - c n ) \leq e^{\frac{1}{12} (n \pi s)^2 - s c n}.&#xA;\end{align*}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Minimizing over $s&amp;gt;0$, we finally obtain&#xA;$$&#xD;&#xA;\Pr( T \leq n\log n  -cn ) \leq e^{-\frac{3 c^2 }{\pi^2}}&#xD;&#xA;$$&lt;/p&gt;&#xA;" OwnerUserId="3500" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-03-08T01:58:42.140" LastActivityDate="2011-03-08T01:58:42.140" CommentCount="3" />
  <row Id="7965" PostTypeId="2" ParentId="7960" CreationDate="2011-03-07T18:09:46.317" Score="2" ViewCount="" Body="&lt;p&gt;There is just nothing to understand here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Well, ok, it is just a standard linear regression model. It assumes that the score of a student can be described as a linear function of several factors, including school and teacher efficiency coefficients -- thus it shares all the standard problems of linear models, mainly the fact that it is a great approximation of a nonlinear world and may as well work perfectly or embarrassingly bad depending on a situation and on how far one would try to extrapolate with it. (However one should expect the authors of the tech rep checked it and found out that it's ok ;-) ).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;But the real problem&lt;/strong&gt; is that this is an analytical tool and such shouldn't be used to assess people achievements -- this way (totally regardless if the marks are fair or not) every evaluee trying to understand her/his mark (probably in hope of optimizing it) will only meet hopeless confusion, as in this case.&lt;/p&gt;&#xA;" OwnerUserId="88" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-07T19:35:16.397" LastActivityDate="2011-03-07T19:35:16.397" CommentCount="3" />
  <row Id="7966" PostTypeId="2" ParentId="7959" CreationDate="2011-03-07T18:20:10.767" Score="4" ViewCount="" Body="&lt;p&gt;I think &lt;a href=&quot;http://stats.stackexchange.com/questions/7959/algorithm-to-dynamically-monitor-quantiles/7963#7963&quot;&gt;whuber's suggestion&lt;/a&gt; is great and I would try that first. However, if you find you really can't accomodate the $O(\sqrt N)$ storage or it doesn't work out for some other reason, here is an idea for a different generalization of P2. It's not as detailed as what whuber suggests - more like a research idea instead of as a solution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead of tracking the quantiles at $0$, $p/2$, $p$, $(1+p)/2$, and $1$, as the original P2 algorithm suggests, you could simply keep track of more quantiles (but still a constant number). It looks like the algorithm allows for that in a very straightforward manner; all you need to do is compute the correct &quot;bucket&quot; for incoming points, and the right way to update the quantiles (quadratically using adjacent numbers).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say you keep track of $25$ points. You could try tracking the quantile at $0$, $p/12$, $\dotsc$, $p \cdot 11/12$, $p$, $p + (1-p)/12$, $\dotsc$, $p + 11\cdot(1-p)/12$, $1$ (picking the points equidistantly in between $0$ and $p$, and between $p$ and $1$), or even using $22$ &lt;a href=&quot;http://en.wikipedia.org/wiki/Chebyshev_nodes&quot; rel=&quot;nofollow&quot;&gt;Chebyshev nodes&lt;/a&gt; of the form $p/2 \cdot (1 + \cos \frac{(2 i - 1)\pi}{22})$ and $p + (1 - p)/2 \cdot (1 + \cos \frac{(2i-1)\pi}{22})$. If $p$ is close to $0$ or $1$, you could try putting fewer points on the side where there is less probability mass and more on the other side.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you decide to pursue this, I (and possibly others on this site) would be interested in knowing if it works...&lt;/p&gt;&#xA;" OwnerUserId="2898" LastActivityDate="2011-03-07T18:20:10.767" CommentCount="1" />
  <row Id="7967" PostTypeId="2" ParentId="7960" CreationDate="2011-03-07T19:26:04.770" Score="9" ViewCount="" Body="&lt;p&gt;Here's one possibility.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assessing teacher performance has traditionally been difficult. One part of this difficulty is that different students have different levels of interest in a given subject. If a given student gets an A, this doesn't necessarily mean that teaching was excellent -- rather, it may mean that a very gifted and interested student did his best to succeed even despite poor teaching quality. Conversely, a student getting a D doesn't necessarily mean that the teaching was poor -- rather, it may mean that a disinterested student coasted despite the teacher's best efforts to educate and inspire.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The difficulty is aggravated by the fact that student selection (and therefore the students' level of interest) is far from random. It is common for schools to emphasize one subject (or a group of subjects) over others. For example, a school may emphasize technical subjects over humanities. Students in such schools are probably so interested in technical areas that they will receive a passing grade even with the worst possible teacher. Thus the fraction of students passing math is not a good measure of teaching -- we expect good teachers to do much better than that with students who are so eager to learn. In contrast, those same students may not be interested at all in arts. It would be difficult to expect even from the best teacher to ensure all students get A's. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another difficulty is that not all success in a given class is attributable to that class's teacher directly. Rather, the success may be due to the school (or entire district) creating motivation and framework for achievement.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To take into account all of these difficulties, researchers have created a model that evaluates teacher's 'added value'. In essence, the model takes into account the intrinsic characteristics of each student (overall level of interest and success in learning), as well as the school and district's contributions to student success, and predicts the student grades that would be expected with 'average' teaching in that environment. The model then compares the actual grades to the predicted ones and based on it decides whether teaching was adequate given all the other considerations, better than adequate, or worse. Although the model may seem complex to a non-mathematician, it is actually pretty simple and standard. Mathematicians have been using similar (and even more complex) models for decades.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To summarize, Ms. Isaacson's guess is correct. Even though 65 of her 66 students scored proficient on the state test, they would have scored just the same even if a dog were their teacher. An actual good teacher would enable these students to achieve not merely 'proficient', but actually 'good' scores on the same test.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;At this point I could mention some of my concerns with the model. For example, the model developers claim it addresses some of the difficulties with evaluating teaching quality. Do I have enough reasons to believe them? Neighborhoods with lower-income population will have lower expected 'district' and 'school' scores. Say a neighborhood will have an expected score of 2.5. A teacher that will achieve an average of 3 will get a good evaluation. This may prompt teachers to aim for the score of 3, rather than for a score of, say, 4 or 5. In other words, teachers will aim for mediocrity rather than perfection. Do we want this to happen? Finally, even though the model is simple mathematically, it works in a way very different from how human intuition works. As a result, we have no obvious way to validate or dispute the model's decision. Ms. Isaacson's unfortunate example illustrates what this may lead to. Do we want to depend blindly on the computer in something so important? &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Note that this is an explanation to a layperson. I sidestepped several potentially controversial issues here. For example, I didn't want to say that school districts with low income demographics are expected to perform poorer, because this wouldn't sound good to a layperson.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, I have assumed that the goal is actually to give a reasonably fair description of the model. But I'm pretty sure that this wasn't NYT's goal here. So at least part of the reason their explanation is poor is intentional FUD, in my opinion.&lt;/p&gt;&#xA;" OwnerUserId="3369" LastEditorUserId="3369" LastEditorDisplayName="" LastEditDate="2011-03-07T23:43:58.997" LastActivityDate="2011-03-07T23:43:58.997" CommentCount="3" />
  <row Id="7968" PostTypeId="2" ParentId="7960" CreationDate="2011-03-07T19:31:19.560" Score="9" ViewCount="" Body="&lt;p&gt;&quot;Your teaching score depends on how well your students did compared to a prediction made based on&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;What they knew beforehand, as measured by a pretest,&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;How well we think the students can learn based on what we know about them individually (their &quot;characteristics&quot;), &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;And how well students do on average in your district, school, and classroom (if there are other teachers in your classroom).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&quot;In other words, we are evaluating you based on the &lt;em&gt;amount of learning&lt;/em&gt; that was measured, after factoring in the preparation and characteristics of your students and the typical performances of all students in settings like yours with the resources that were available to you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;In this way your score reflects what &lt;em&gt;you&lt;/em&gt; contributed to the student performances, insofar as we can determine that.  Of course we cannot know everything: we know you had unique and special students and that the situation you faced could never be duplicated.  Therefore we know this score is only an estimate that imperfectly reflects how well you taught, but it is a fairer and more accurate estimate than one based solely on post testing or on raw test gains made by your class.&quot;&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-07T19:48:59.913" LastActivityDate="2011-03-07T19:48:59.913" CommentCount="3" />
  <row Id="7969" PostTypeId="1" AcceptedAnswerId="7971" CreationDate="2011-03-07T19:44:05.827" Score="7" ViewCount="103" Body="&lt;p&gt;I have a simple sampling problem, where my inner loop looks like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;v = sample_gamma(k, a)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where &lt;code&gt;sample_gamma&lt;/code&gt; samples from the Gamma distribution to form a Dirichlet sample.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It works well, but for some values of k/a, some of the downstream computation underflows.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I adapted it to use log space variables:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;v = log(sample_gamma(k, a))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;After adapting all of the rest of the programme, it works correctly (at least it gives me the same exact results on test cases). However, it is slower than before.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a way to directly sample $X, \exp(X) \sim \text{Gamma}$ without using slow functions like $\log()$? I tried googling for this, but I do not even know if this distribution has a common name (log-gamma?).&lt;/p&gt;&#xA;" OwnerUserId="2067" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-03-08T08:19:12.243" LastActivityDate="2011-03-16T21:15:33.023" Title="How to quickly sample X if exp(X) ~ Gamma?" Tags="&lt;sampling&gt;&lt;gamma-distribution&gt;" AnswerCount="2" CommentCount="5" />
  <row Id="7970" PostTypeId="2" ParentId="5115" CreationDate="2011-03-07T20:52:49.007" Score="0" ViewCount="" Body="&lt;p&gt;Bill James for his work in statistics that evaluate MLB player performance. His work spawned the term Sabermetics.  He has created numerous statistics that can be found throughout the baseball world.  His ideas stem from how to capture a player's overall impact on a game through run production (offense) and runs saved (defense).  His work has led to less emphaisis on statistics that have low correlation to run production (batting average) and more on OPS (on-base + slugging).  He works as an advisor to the Boston Red Sox and is credited to the World Series Championships in 2004 and 2007.  His work has influenced the book and upcoming feature film Moneyball.&lt;/p&gt;&#xA;" OwnerUserId="3599" LastActivityDate="2011-03-07T20:52:49.007" CommunityOwnedDate="2011-03-07T20:52:49.007" />
  <row Id="7971" PostTypeId="2" ParentId="7969" CreationDate="2011-03-07T21:14:37.513" Score="6" ViewCount="" Body="&lt;p&gt;Consider a small shape parameter $\alpha$ near 0, such as $\alpha = 1/100$.  In the range between 0 and $\alpha$, $e^{-\alpha}$ is approximately $1$, so the Gamma pdf is approximately $x^{\alpha-1}dx / \Gamma(\alpha)$.  This can be integrated to an approximate CDF, $F_\alpha(x) = \frac{x^\alpha}{\alpha \Gamma(\alpha)}$.  Inverting it, we see a $1/\alpha$ power: a huge exponent.  For $\alpha = 1/100$ this causes some chance of underflow (a double precision value less than $10^{-300}$, more or less).  Here is a plot of the chance of getting underflow as a function of the base-ten logarithm of $\alpha$:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/AB5ML.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One solution is to exploit this approximation for generating log(Gamma) variates: in effect, try to generate a Gamma variate and if it's too small, generate its logarithm from this approximate power distribution (as shown below).  (Do this repeatedly until the log is within the underflow range, so that it is a valid substitute for the original underflowing variate.)  For the Dirichlet calculation, subtract the maximum of all the logarithms from each of the log values: this implicitly rescales all the Gamma variates so it won't affect the Dirichlet values.  Treat any resulting log that is too small (say, less than -100) as being the log of a true zero.  Exponentiate the other logs.  Now you can proceed without underflow.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is going to take even longer than before, but at least it will work!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;To generate an approximate log Gamma variate&lt;/strong&gt; with shape parameter $\alpha$, precompute $C = \log(\Gamma(\alpha)) + \log(\alpha)$.  This is easy, because there are algorithms to &lt;a href=&quot;http://en.wikipedia.org/wiki/Factorial#Computation&quot; rel=&quot;nofollow&quot;&gt;compute values of log Gamma directly&lt;/a&gt;.  Generate a uniform random float between 0 and 1, take its logarithm, divide by $\alpha$, and add $C$ to it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because the scale parameter merely rescales the variate, there is no problem accommodating it in these procedures.  You don't even need it if all scale parameters are the same.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Edit&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;In another reply the OP describes a method in which the $1/\alpha$ power of a uniform variate (a $B(\alpha)$ variate) is multiplied by a $\Gamma(\alpha+1)$ variate.  This works because the pdf of the joint distribution of these two variates equals $\left(\alpha x^{\alpha-1}\right) \left(y^{\alpha}e^{-y}dy/\Gamma(\alpha+1)\right)$.  To find the pdf of $z = xy$ we substitute $y \to z/x$, divide by the Jacobean $x$, and integrate out $x$.  The integral must range from $z$ to $\infty$ because $0 \le y \le 1$, whence&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\text{pdf}(z)=\frac{\alpha}{\Gamma(\alpha+1)}\int_z^{\infty}{\left(x^\alpha / x\right) e^{-x} (z/x)^{\alpha-1} dx} dz = \frac{1}{\Gamma(\alpha)}z^{\alpha-1}e^{-z}dz,$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;which is the pdf of a $\Gamma(\alpha)$ distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The whole point is that when $0 \lt \alpha \lt 1$, a value drawn from $\Gamma(\alpha+1)$ is unlikely to underflow and by summing its log and $1/\alpha$ times the log of an independent uniform variate we will have the log of a $\Gamma(\alpha)$ variate.  The log is likely to be very negative, but we will have bypassed the construction of its antilog, which will underflow in a floating point representation.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-16T21:15:33.023" LastActivityDate="2011-03-16T21:15:33.023" />
  <row Id="7972" PostTypeId="1" AcceptedAnswerId="7978" CreationDate="2011-03-07T21:32:50.493" Score="4" ViewCount="75" Body="&lt;p&gt;I have a number of rasters of environmental data (~10) which may be important predictors for modelling species presence and abundance at ~10 different locations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to know which of the rasters are important in explaining the variance in observed results. Is it appropriate to look for principal components of the rasters? Is this done across the entire extent, or only with respect to my observed sites? Any reference on this would be appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: It turns out that PCs are not useful in this case. Even though two rasters may have very large correlation across the landscape, if they are different at the narrow range of sites occupied by the modelled species, this is what really matters.&lt;/p&gt;&#xA;" OwnerUserId="2993" LastEditorUserId="2993" LastEditorDisplayName="" LastEditDate="2011-04-12T08:55:39.260" LastActivityDate="2011-04-12T08:55:39.260" Title="Principal components of spatial variables" Tags="&lt;pca&gt;&lt;spatial&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="2" />
  <row Id="7973" PostTypeId="1" CreationDate="2011-03-07T21:52:23.480" Score="3" ViewCount="67" Body="&lt;p&gt;Imagine you have a set of four elements (A-D) with some numeric values of a measured property (several observations for each element):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;A: 26 25 29 21&#xA;B: 24 17 16&#xA;C: 32 34 29 19 25 27 28&#xA;D: 23 29 26 20 14&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have to detect if there are significant differences on the average levels. So I run a one way ANOVA to determine if differences are found. It works fine but, when I get a new sample, I need to execute the ANOVA again for just one new sample.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any way to do an &quot;incremental one-way ANOVA&quot; that works &lt;strong&gt;without redoing the whole computations&lt;/strong&gt; ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a lot of data and the process is expensive in terms of time and memory consumption.&lt;/p&gt;&#xA;" OwnerUserId="3576" LastEditorUserId="3576" LastEditorDisplayName="" LastEditDate="2011-03-08T16:45:09.477" LastActivityDate="2011-03-09T18:50:07.717" Title="Incremental one-way ANOVA" Tags="&lt;anova&gt;" AnswerCount="1" />
  <row Id="7974" PostTypeId="2" ParentId="5115" CreationDate="2011-03-07T22:00:16.830" Score="2" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Samuel_S._Wilks&quot; rel=&quot;nofollow&quot;&gt;Samuel S. Wilks&lt;/a&gt; was a leader in the development of mathematical statistics. He developed the &lt;a href=&quot;http://en.wikipedia.org/wiki/Likelihood-ratio_test#Distribution%3a_Wilks.27_theorem&quot; rel=&quot;nofollow&quot;&gt;theorem on the distribution of the likelihood ratio&lt;/a&gt;, a fundamental result that is used in a wide variety of situations.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;He also helped found the Princeton statistics department, where he was Fred Mosteller's advisor, among others, and has a prestigious &lt;a href=&quot;http://www.amstat.org/careers/samuelwilksaward.cfm&quot; rel=&quot;nofollow&quot;&gt;ASA award&lt;/a&gt; named after him.&lt;/p&gt;&#xA;" OwnerUserId="3601" LastActivityDate="2011-03-07T22:00:16.830" CommunityOwnedDate="2011-03-07T22:00:16.830" />
  <row Id="7975" PostTypeId="1" AcceptedAnswerId="8037" CreationDate="2011-03-07T22:42:12.067" Score="4" ViewCount="135" Body="&lt;p&gt;Having worked mostly with cross sectional data so far and very very recently browsing, scanning stumbling through a bunch of introductory time series literature I wonder what which role explanatory variables are playing in time series analysis. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to &lt;em&gt;explain a trend&lt;/em&gt; instead of de-trending.&#xA;Most of what I read as an introduction assumes that the series is stemming from some stochastic process. I read about AR(p) and MA processes as well as ARIMA modelling. Wanting to deal with more information than only autoregressive processes I found VAR / VECM and ran some examples, but still I wonder if there is some case that is related closer to what explanatories do in cross sections. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The motivation behind this is that decomposition of my series shows that the trend is the major contributor while remainder and seasonal effect hardly play a role. I would like to explain this trend.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can / should I regress my series on multiple different series? Intuitively I would use gls because of serial correlation (I am not so sure about the cor structure). I heard about spurious regression and understand that this is a pitfall, nevertheless I am looking for a way to explain a trend. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this completely wrong or uncommon? Or have I just missed the right chapter so far?&lt;/p&gt;&#xA;" OwnerUserId="704" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-03-08T09:12:10.913" LastActivityDate="2011-03-08T22:02:19.000" Title="What to make of explanatories in time series?" Tags="&lt;r&gt;&lt;time-series&gt;&lt;multivariate-analysis&gt;" AnswerCount="3" FavoriteCount="2" />
  <row Id="7976" PostTypeId="1" CreationDate="2011-03-07T22:50:42.937" Score="1" ViewCount="69" Body="&lt;p&gt;Can we do that? If yes, then what are the conditions which should be met? Thanks.&lt;/p&gt;&#xA;" OwnerUserId="1102" LastActivityDate="2011-03-07T23:44:19.147" Title="Can we compute bivariate from marginal distributions?" Tags="&lt;distributions&gt;&lt;multivariate-analysis&gt;&lt;marginal&gt;" AnswerCount="1" FavoriteCount="0" />
  <row Id="7977" PostTypeId="1" AcceptedAnswerId="7984" CreationDate="2011-03-07T22:57:20.690" Score="10" ViewCount="243" Body="&lt;p&gt;I am wondering how to generate uniformly distributed points on the surface of the 3-d unit sphere? Also after generating those points, what is the best way to visualize and check whether they are truly uniform on the surface $x^2+y^2+z^2=1$?&lt;/p&gt;&#xA;" OwnerUserId="3552" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-03-07T23:42:01.520" LastActivityDate="2011-03-08T01:19:47.993" Title="How to generate uniformly distributed points on the surface of the 3-d unit sphere?" Tags="&lt;random-generation&gt;" AnswerCount="4" FavoriteCount="2" />
  <row Id="7978" PostTypeId="2" ParentId="7972" CreationDate="2011-03-07T23:13:09.110" Score="3" ViewCount="" Body="&lt;p&gt;Your idea about the &quot;rasters&quot; is not very clearly stated, but you might have a look at the paper by Borcard and Legendre (1994) and &lt;a href=&quot;http://www.bio.umontreal.ca/legendre/reprints/index.html&quot; rel=&quot;nofollow&quot;&gt;their later works&lt;/a&gt; on spatial eigenvector-based analyses to see if one of the approaches will fit to your problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.bio.umontreal.ca/legendre/reprints/B&amp;amp;L_EES1994.pdf&quot; rel=&quot;nofollow&quot;&gt;Borcard, D., Legendre, P., (1994) Environmental control and spatial structure in ecological communities: an example using oribatid mites (Acari, Oribatei).  Environmental and Ecological Statistics 1, 37–61.&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="3467" LastEditorUserId="3467" LastEditorDisplayName="" LastEditDate="2011-03-07T23:28:39.240" LastActivityDate="2011-03-07T23:28:39.240" />
  <row Id="7979" PostTypeId="1" CreationDate="2011-03-07T23:15:04.000" Score="8" ViewCount="113" Body="&lt;p&gt;Suppose $X \sim \mathcal{N}(\mu_x, \sigma^2_x)$ and $Y \sim \mathcal{N}(\mu_y, \sigma^2_y)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am interested in $z = \min(\mu_x, \mu_y)$.  Is there an unbiased estimator for $z$?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The simple estimator of $\min(\bar{x}, \bar{y})$ where $\bar{x}$ and $\bar{y}$ are sample means of $X$ and $Y$, for example, is biased (though consistent).  It tends to undershoot $z$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can't think of an unbiased estimator for $z$.  Does one exist?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for any help.&lt;/p&gt;&#xA;" OwnerUserId="3602" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-08T06:24:48.590" LastActivityDate="2011-03-11T20:26:09.263" Title="Unbiased estimator for the smaller of two random variables" Tags="&lt;random-variable&gt;&lt;unbiased-estimator&gt;&lt;minimum&gt;" AnswerCount="3" />
  <row Id="7980" PostTypeId="2" ParentId="7977" CreationDate="2011-03-07T23:32:00.477" Score="0" ViewCount="" Body="&lt;p&gt;My best guess would be to first generate a set of uniformly distributed points in 2 dimensional space and to then project those points onto the surface of a sphere using some sort of projection. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You will probably have to mix and match the way you generate the points with the way that you map them. In terms of the 2D point generationgeneration, I think that scrambled low-discrepancy sequences would be a good place to start (i.e. a scrambled Sobol sequence) since it usually produces points that are not &quot;clumped together&quot;. I'm not as sure about which type of mapping to use, but Woflram popped up the &lt;a href=&quot;http://mathworld.wolfram.com/GnomonicProjection.html&quot; rel=&quot;nofollow&quot;&gt;Gnonomic projection&lt;/a&gt;... so maybe that could work? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;MATLAB has a decent implementation of low discrepancy sequences which you can generate using &lt;code&gt;q = sobolset(2)&lt;/code&gt; and scramble using &lt;code&gt;q = scramble(q)&lt;/code&gt;. There is also a mapping toolbox in MATLAB with a bunch of different projection functions you could use in case you did not want to code the mapping and graphics yourself. &lt;/p&gt;&#xA;" OwnerUserId="3572" LastActivityDate="2011-03-07T23:32:00.477" CommentCount="3" />
  <row Id="7981" PostTypeId="2" ParentId="7976" CreationDate="2011-03-07T23:44:19.147" Score="3" ViewCount="" Body="&lt;p&gt;Just knowing the marginal distributions of two variables isn't sufficient to specify their bivariate distribution. You need more information about their joint relationship. Simple example: two random normals can have any particular correlation with each other, but still have the same marginal distributions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It sounds like you may want to consider &lt;a href=&quot;http://en.wikipedia.org/wiki/Copula&quot; rel=&quot;nofollow&quot; title=&quot;copulas&quot;&gt;copulas&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1569" LastActivityDate="2011-03-07T23:44:19.147" CommentCount="6" />
  <row Id="7982" PostTypeId="1" AcceptedAnswerId="8139" CreationDate="2011-03-08T00:05:58.443" Score="4" ViewCount="106" Body="&lt;p&gt;I'm working on a binary classification problem, with about 1000 binary features in total. The problem is that for each datapoint, I only know the values of a small subset of the features (around 10-50), and the features in this subset are pretty much random.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What's a good way to deal with the problem of the missing features? Is there a particular classification algorithm that handles missing features well? (Naive Bayes should work, but is there anything else?) I'm guessing I don't want to do some kind of variable imputation, since I have so many missing features.&lt;/p&gt;&#xA;" OwnerUserId="1106" LastActivityDate="2011-03-11T07:37:13.680" Title="Binary classification when many binary features are missing" Tags="&lt;classification&gt;&lt;missing-data&gt;" AnswerCount="2" CommentCount="4" />
  <row Id="7983" PostTypeId="2" ParentId="7977" CreationDate="2011-03-08T00:22:36.827" Score="2" ViewCount="" Body="&lt;p&gt;I had a similar problem (n-sphere) during my PhD and one of the local 'experts' suggested rejection sampling from a n-cube! This, of course, would have taken the age of the universe as I was looking at n in the order of hunderds.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The algorithm I ended up using is very simple and published in:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;W.P. Petersen and A. Bernasconic&#xA;Uniform sampling from an n-sphere: Isotropic method&#xA;Technical Report, TR-97-06, Swiss Centre for Scientific Computing&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also have this paper in my bibliography that I havent looked at. You may find it useful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Harman, R. &amp;amp; Lacko, V. On decompositional algorithms for uniform sampling from $n$-spheres and $n$-balls Journal of Multivariate Analysis, 2010&lt;/p&gt;&#xA;" OwnerUserId="530" LastActivityDate="2011-03-08T00:22:36.827" CommentCount="2" />
  <row Id="7984" PostTypeId="2" ParentId="7977" CreationDate="2011-03-08T00:30:43.540" Score="9" ViewCount="" Body="&lt;p&gt;A standard method is to generate three standard normals and construct a unit vector from them.  That is, when $X_i \sim N(0,1)$ and $\lambda^2 = X_1^2 + X_2^2 + X_3^2$, then $(X_1/\lambda, X_2/\lambda, X_3/\lambda)$ is uniformly distributed on the sphere.  This method works well for $d$-dimensional spheres, too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In 3D you can use rejection sampling: draw $X_i$ from a uniform$[-1,1]$ distribution until the length of $(X_1, X_2, X_3)$ is less than or equal to 1, then--just as with the preceding method--normalize the vector to unit length.  The expected number of trials per spherical point equals $2^3/(4 \pi / 3)$ = 1.91.  In higher dimensions the expected number of trials gets so large this rapidly becomes impracticable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;There are many ways to check uniformity&lt;/strong&gt;.  A neat way, although somewhat computationally intensive, is with &lt;a href=&quot;http://www.public.iastate.edu/~pcaragea/S40608/Notes/Dixon_Ripley_K.pdf&quot; rel=&quot;nofollow&quot;&gt;Ripley's K function&lt;/a&gt;.  The expected number of points within (3D Euclidean) distance $\rho$ of any location on the sphere is proportional to the area of the sphere within distance $\rho$, which equals $2\pi\rho^2$.  By computing all interpoint distances you can compare the data to this ideal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;General principles of constructing statistical graphics suggest a good way to make the comparison is to plot variance-stabilized residuals $(d_{[i]} - i/m)/m$ against $i = 1, 2, \ldots, n(n-1)/2$ where $d_{[i]}$ is the $i^\text{th}$ smallest of the mutual distances.  The plot should be close to zero.  (This approach is unconventional.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a picture of 100 independent draws from a uniform spherical distribution obtained with the first method:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/EZEcr.png&quot; alt=&quot;100 uniform spherical points&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the diagnostic plot of the distances:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/8y1H9.png&quot; alt=&quot;Diagnostic plot&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The y scale suggests these values are all close to zero.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the accumulation of 100 such plots to suggest what size deviations might actually be significant indicators of non-uniformity:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/h4kec.png&quot; alt=&quot;Simulated values&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(These plots look an awful lot like &lt;a href=&quot;http://en.wikipedia.org/wiki/Brownian_bridge&quot; rel=&quot;nofollow&quot;&gt;Brownian bridges&lt;/a&gt;...there may be some interesting theoretical discoveries lurking here.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, here is the diagnostic plot for a set of 100 uniform random points plus another 41 points uniformly distributed in the upper hemisphere only:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/d5z4u.png&quot; alt=&quot;Simulated non-uniform values&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Relative to the uniform distribution, it shows a significant &lt;em&gt;decrease&lt;/em&gt; in average interpoint distances out to a range of one hemisphere.  That in itself is meaningless, but the useful information here is that something is non-uniform on the scale of one hemisphere.  In effect, this plot readily detects that one hemisphere has a different density than the other.  (A simpler chi-square test would do this with more power &lt;em&gt;if you knew in advance which hemisphere to test&lt;/em&gt; out of the infinitely many possible ones.)&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-03-08T00:30:43.540" CommentCount="7" />
  <row Id="7985" PostTypeId="2" ParentId="7884" CreationDate="2011-03-08T00:53:05.203" Score="4" ViewCount="" Body="&lt;p&gt;Following up to Steve's reply, there is a much faster way in data.table :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; # Preamble&#xA;&amp;gt; dx &amp;lt;- data.frame(&#xA;+     ID = sort(sample(1:7000, 400000, TRUE))&#xA;+     , AGE = sample(18:65, 400000, TRUE)&#xA;+     , FEM = sample(0:1, 400000, TRUE)&#xA;+ )&#xA;&amp;gt; dxt &amp;lt;- data.table(dx, key='ID')&#xA;&#xA;&amp;gt; # fast self join&#xA;&amp;gt; system.time(ans2&amp;lt;-dxt[J(unique(ID)),mult=&quot;first&quot;])&#xA; user  system elapsed &#xA;0.048   0.016   0.064&#xA;&#xA;&amp;gt; # slower using .SD&#xA;&amp;gt; system.time(ans1&amp;lt;-dxt[, .SD[1], by=ID])&#xA;  user  system elapsed &#xA;14.209   0.012  14.281 &#xA;&#xA;&amp;gt; mapply(identical,ans1,ans2)  # ans1 is keyed but ans2 isn't, otherwise identical&#xA;  ID  AGE  FEM &#xA;TRUE TRUE TRUE &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you merely need the first row of each group, it's much faster to join to that row directly. Why create the .SD object each time, only to use the first row of it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Compare the 0.064 of data.table to &quot;Matt Parker's alternative to Chase's solution&quot; (which seemed to be the fastest so far) :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; system.time(ans3&amp;lt;-dxt[c(TRUE, dxt$ID[-1] != dxt$ID[-length(dxt$ID)]), ])&#xA; user  system elapsed &#xA;0.284   0.028   0.310 &#xA;&amp;gt; identical(ans1,ans3)&#xA;[1] TRUE &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So ~5 times faster, but it's a tiny table at under 1 million rows. As size increases, so does the difference.&lt;/p&gt;&#xA;" OwnerUserId="3589" LastActivityDate="2011-03-08T00:53:05.203" CommentCount="2" />
  <row Id="7986" PostTypeId="1" CreationDate="2011-03-08T01:01:16.077" Score="2" ViewCount="33" Body="&lt;p&gt;This problem is basically the classic asset selling problem but with imperfect state information.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the classical problem, we have an asset that we wish to sell, we receive offers w(0) to w(N-1).  If we accept the offer at a given period, we can invest that money with a certain rate of interest r &gt; 0.  We suppose that at period N-1, we will accept the offer if we haven't accepted any already.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the problem I am trying to solve, the offers w(k) are independent and identically distributed.  However the common distribution of the w(k) is unknown.  Instead it is known that this distribution is one out of two given distributions F1 and F2, and that the a priori probability that F1 is the correct distribution is a given scalar q, with 0 &amp;lt; q &amp;lt; 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;a)Formulate this as an imperfect state information problem and identify the state, control, system disturbance, observation, and observation disturbance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;b)Show that (x(k), q(k)), where &lt;/p&gt;&#xA;&#xA;&lt;p&gt;q(k) = P(distribution is F1 | w(0), ... , w(k-1))&lt;/p&gt;&#xA;&#xA;&lt;p&gt;is a suitable sufficient statistic, write a corresponding DP algorithm, and derive the form of the optimal selling policy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The state would represent whether or not we have already sold our asset and if we haven't, the offer we have at this period.  The control would be to either accept or refuse the offer at this period.  The system disturbance would be the offer.  As for the rest of the question, I'm not too sure.&lt;/p&gt;&#xA;" OwnerUserId="3607" LastActivityDate="2011-03-08T01:01:16.077" Title="Optimal stopping under partially observable state" Tags="&lt;optimal-stopping&gt;" />
  <row Id="7987" PostTypeId="2" ParentId="7979" CreationDate="2011-03-08T01:06:59.107" Score="4" ViewCount="" Body="&lt;p&gt;This is just a couple of comments not an answer (don't have enough rep. point).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(1). There is an explicit formula for the bias of the simple estimator $min(\bar{x},\bar{y})$ here: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Clark, C. E. 1961, Mar-Apr. The greatest of a ﬁnite set of random variables.&#xA;Operations Research 9 (2): 145–162.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Not sure how this helps though&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(2). This is just intuition, but I think such an estimator doesn't exist. If there is such an estimator, it should also be unbiased when $\mu_x=\mu_y=\mu$. Thus any 'downgrading' which makes the estimator less than say the weighted average of the two sample means make the estimator biased for this case.&lt;/p&gt;&#xA;" OwnerUserId="3036" LastActivityDate="2011-03-08T01:06:59.107" CommentCount="4" />
  <row Id="7988" PostTypeId="2" ParentId="7977" CreationDate="2011-03-08T01:19:47.993" Score="6" ViewCount="" Body="&lt;p&gt;Here is some rather simple R code &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;n     &amp;lt;- 100000                  # large enough for meaningful tests&#xA;z     &amp;lt;- 2*runif(n) - 1          # uniform on [-1, 1]&#xA;theta &amp;lt;- 2*pi*runif(n) - pi      # uniform on [-pi, pi]&#xA;x     &amp;lt;- sin(theta)*sqrt(1-z^2)  # based on angle&#xA;y     &amp;lt;- cos(theta)*sqrt(1-z^2)     &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It is very simple to see from the construction that $x^2+y^2 = 1- z^2$ and so $x^2+y^2+z^2=1$ but if it needs to be tested then &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mean(x^2+y^2+z^2)  # should be 1&#xA;var(x^2+y^2+z^2)   # should be 0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and easy to test that each of $x$ and $y$ are uniformly distributed on $[-1,1]$ ($z$ obviously is) with&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plot.ecdf(x)  # should be uniform on [-1, 1]&#xA;plot.ecdf(y)&#xA;plot.ecdf(z)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Clearly, given a value of $z$, $x$ and $y$ are uniformly distributed around a circle of radius $\sqrt{1-z^2}$ and this can be tested by looking at the distribution of the arctangent of their ratio.   But since $z$ has the same marginal distribution as $x$ and as $y$, a similar statement is true for any pair, and this too can be tested.  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plot.ecdf(atan2(x,y)) # should be uniform on [-pi, pi]&#xA;plot.ecdf(atan2(y,z))&#xA;plot.ecdf(atan2(z,x))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If still unconvinced, the next steps would be to look at some arbitrary 3-D rotation or how many points fell within a given solid angle, but that starts to get more complicated, and I think is unnecessary.&lt;/p&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-03-08T01:19:47.993" CommentCount="4" />
  <row Id="7989" PostTypeId="1" CreationDate="2011-03-08T01:48:03.257" Score="1" ViewCount="79" Body="&lt;p&gt;I have a general question. What kind of noise is additive, and what about multiplicative noise? How to determine the nature of noise?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks a lot for your help.&lt;/p&gt;&#xA;" OwnerUserId="3552" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-29T01:04:54.580" LastActivityDate="2011-04-29T01:04:54.580" Title="Determining the nature of noise" Tags="&lt;regression&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="7990" PostTypeId="1" AcceptedAnswerId="8051" CreationDate="2011-03-08T01:51:53.857" Score="1" ViewCount="67" Body="&lt;p&gt;Suppose we have the following random process.  We start with two vectors $a_1=(0)$ and $b_1=(0)$.  In going from $i$ to $i+1$, we will a perturbation to $a_i$ and $b_i$.  With probability $p$, we perform Case 1, otherwise perform Case 2.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Case 1: We pick an element of $a_i$, say at coordinate $x$ (chosen uniformly at random from all coordinates $1,2,\ldots,|a_i|$, where $|a_i|$ is the length of $a_i$), and append it to the end of $a_i$ to form $a_{i+1}$.  Similarly, to form $b_{i+1}$ append the element at coordinate $x$ from $b_i$ to the end of $b_i$.  The remainder of $a_{i+1}$ and $b_{i+1}$ is the same as $a_i$ and $b_i$, respectively.  [The copied element is at coordinate $x$ in both $a_i$ and $b_i$.]&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Case 2: Choose $x$ and $y$ to be two coordinates chosen uniformly at random from $1,2,\ldots,|a_i|$.  If $x \neq y$ then set $a_{i+1}[x]=a_i[x]+1$ and $b_{i+1}[y]=b_i[y]+1$ (if $x=y$ then don't do anything [it's not clear at this point whether or not I want to enforce $x \neq y$]).  Again, the remainder of $a_{i+1}$ and $b_{i+1}$ is the same as $a_i$ and $b_i$, respectively.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;Question&lt;/strong&gt;: What is the expected dot product of $a_i$ and $b_i$?  That is, what is $\mathrm{E}(\sum_x a_i[x] \cdot b_i[x])$?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So here's an example of what these vectors could look like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;i a_i b_i&#xA;1 (0) (0)&#xA;2 (0,0) (0,0) [case 1 x=1]&#xA;3 (0,0,0) (0,0,0) [case 1 x=1]&#xA;4 (1,0,0) (0,1,0) [case 2 x=1 y=2]&#xA;5 (1,0,0,1) (0,1,0,0) [case 1 x=1]&#xA;6 (1,0,0,1) (0,1,0,0) [case 2 x=4 y=4]&#xA;7 (1,0,0,2) (0,1,1,0) [case 2 x=4 y=3]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm looking at a related problem studying evolving random graphs.  In the random graph problem, the two vectors represent the in-degrees and out-degrees of an evolving network over time.  In the graph, we can duplicate vertices or add or delete an edge (which is related to cases 1 and 2 above).  In this case, the dot product therefore counts the number of directed paths of length 3.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the above problem differs from the one I'm considering since (a) it's not necessarily the case that $\sum_x a_i[x]=\sum_x b_i[x]$ and (b) there's nothing to stop parallel edges arising here.  Although, I'm hoping that this simplified question will be more answerable and the techniques can be re-used.&lt;/p&gt;&#xA;" OwnerUserId="386" LastEditorUserId="386" LastEditorDisplayName="" LastEditDate="2011-03-08T03:55:47.993" LastActivityDate="2011-03-09T10:28:37.683" Title="What is the expected dot product of two evolving vectors?" Tags="&lt;stochastic-processes&gt;&lt;expected-value&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="7991" PostTypeId="2" ParentId="7155" CreationDate="2011-03-08T01:55:40.563" Score="0" ViewCount="" Body="&lt;p&gt;There are MANY rigorous definitions of outlier.  But the choice among those can seem arbitrary.  But I think this is part of the misconception that statistics is a subject in which each problem has one correct answer.&lt;/p&gt;&#xA;" OwnerUserId="686" LastActivityDate="2011-03-08T01:55:40.563" />
  <row Id="7994" PostTypeId="1" CreationDate="2011-03-08T04:08:37.843" Score="1" ViewCount="123" Body="&lt;p&gt;This is problem 12.10 in &lt;a href=&quot;http://www-stat.stanford.edu/~tibs/ElemStatLearn/&quot; rel=&quot;nofollow&quot;&gt;&quot;The Elements of Statistical Learning&quot;&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Suppose you wish to carry&#xA;  out a linear discriminant analysis&#xA;  (two classes) using a vector of&#xA;  transformations of the input variables&#xA;  $h(x)$. Since $h(x)$ is&#xA;  high-dimensional, you will use a&#xA;  regularized within-class covariance&#xA;  matrix $W_h + \gamma I$. Show that the&#xA;  model can be estimated using only the&#xA;  inner products  $K(x_i, x_{i&amp;#39;}) = \left &amp;lt; h(x_i), h(x_{i&amp;#39;}) \right &amp;gt;$.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;How can I go about showing that regularized linear discriminant analysis can be estimated using only inner products, as in the &quot;kernel trick&quot; that is often used with SVM's?&lt;/p&gt;&#xA;" OwnerUserId="988" LastEditorUserId="988" LastEditorDisplayName="" LastEditDate="2011-03-09T02:41:08.560" LastActivityDate="2011-03-09T11:16:18.363" Title="Linear discriminant analysis and the &quot;kernel trick&quot;?" Tags="&lt;classification&gt;&lt;homework&gt;&lt;kernel&gt;" AnswerCount="1" />
  <row Id="7995" PostTypeId="2" ParentId="7989" CreationDate="2011-03-08T04:09:54.563" Score="4" ViewCount="" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Do you know where does the noise comes from? Before doing any statistical test, you think about the origin of the noise you want to remove. Additive noise is independent from the level of the signal, whereas multiplicative noise is proportional to the level of the signal.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If you are not able to know, or if you want to check, try for instance to plot the sd of the noise as a function of the level of the signal.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="1709" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-03-08T13:51:49.943" LastActivityDate="2011-03-08T13:51:49.943" />
  <row Id="7996" PostTypeId="1" AcceptedAnswerId="8042" CreationDate="2011-03-08T05:24:45.063" Score="3" ViewCount="97" Body="&lt;p&gt;I am evaluating a scenario's output parameter's dependence on three parameters: A, B and C. For this, I am conducting the following experiments:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Fix A+B, Vary C - Total four sets of (A+B) each having 4 variations of C&lt;/li&gt;&#xA;&lt;li&gt;Fix B+C, Vary A - Total four sets of (B+C) each having 3 variations of C&lt;/li&gt;&#xA;&lt;li&gt;Fix C+A, Vary B - Total four sets of (C+A) each having 6 variations of C&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The output of any simulation is the value of a variable over time. For instance, A could be the area, B could be the velocity and C could be the number of vehicles. The output variable I am observing is the number of car crashes over time. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am trying to determine which parameter(s) dominate the outcome of the experiment. By dominate, I mean that sometimes, the outcomes just does not change when one of the parameters change but when some other parameter is changed even by a small amount, a large change in the output is observed. I need to capture this effect and output some analysis from which I can understand the dependence of the output on the input parameters. A friend suggested Sensitivity Analysis but am not sure if there are simpler ways of doing it. Can someone please help me with a good (possibly easy because I don't have a Stats background) technique? It would be great if all this can be done in R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update:&lt;/strong&gt; &#xA;I used linear regression to obtain the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;lm(formula = T ~ A + S + V)&#xA;&#xA;Residuals:&#xA;     Min       1Q   Median       3Q      Max &#xA;-0.35928 -0.06842 -0.00698  0.05591  0.42844 &#xA;&#xA;Coefficients:&#xA;            Estimate Std. Error t value Pr(&amp;gt;|t|)    &#xA;(Intercept) -0.01606    0.16437  -0.098 0.923391    &#xA;A            0.80199    0.15792   5.078 0.000112 ***&#xA;S           -0.27440    0.13160  -2.085 0.053441 .  &#xA;V           -0.31898    0.14889  -2.142 0.047892 *  &#xA;---&#xA;Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 &#xA;&#xA;Residual standard error: 0.1665 on 16 degrees of freedom&#xA;Multiple R-squared: 0.6563, Adjusted R-squared: 0.5919 &#xA;F-statistic: 10.18 on 3 and 16 DF,  p-value: 0.0005416 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Does this mean that the output depends mostly on A and less on V?&lt;/p&gt;&#xA;" OwnerUserId="2164" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2011-03-09T12:46:51.747" LastActivityDate="2011-03-09T12:46:51.747" Title="What is a good way of estimating the dependence of an output variable on the input parameters?" Tags="&lt;r&gt;&lt;regression&gt;&lt;experiment-design&gt;" AnswerCount="2" CommentCount="3" FavoriteCount="2" />
  <row Id="7997" PostTypeId="2" ParentId="7155" CreationDate="2011-03-08T05:30:48.113" Score="0" ViewCount="" Body="&lt;p&gt;Outliers are important only in the frequentist realm. If a single datapoint adds bias to your model which is defined by an underlying distribution predeterimined by your theory, then it is an outlier for that model. The subjectivity lies in the fact that if your theory posits a different model, then you can have a different set of points as outliers.&lt;/p&gt;&#xA;" OwnerUserId="3611" LastActivityDate="2011-03-08T05:30:48.113" CommentCount="1" />
  <row Id="7998" PostTypeId="2" ParentId="7979" CreationDate="2011-03-08T05:36:05.240" Score="0" ViewCount="" Body="&lt;p&gt;I'd be fairly sure an unbiased estimator does not exist.  But unbiased estimators don't exist for most quantities, and unbiasedness is not a particularly desirable property in the first place.  Why do you want one here?&lt;/p&gt;&#xA;" OwnerUserId="3612" LastActivityDate="2011-03-08T05:36:05.240" CommentCount="1" />
  <row Id="7999" PostTypeId="1" AcceptedAnswerId="8002" CreationDate="2011-03-08T06:32:46.820" Score="4" ViewCount="193" Body="&lt;p&gt;Let's say I have a function &lt;code&gt;funToRepeat&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;funToRepeat &amp;lt;- function(df){&#xA;    rows &amp;lt;- nrow(df)&#xA;    cols &amp;lt;- ncol(df)&#xA;    err &amp;lt;- runif( rows * cols )&#xA;    dfNew &amp;lt;- df + as.data.frame( matrix(err, nrow=rows, ncol=cols) )&#xA;    dfNew&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to operate &lt;code&gt;funToRepeat&lt;/code&gt; on &lt;code&gt;dfTest&lt;/code&gt; (a toy data set), multiple times, say &lt;code&gt;n&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dfTest &amp;lt;-&#xA;  structure(list(V1 = 1:2, V2 = 3:4), .Names = c(&quot;V1&quot;, &quot;V2&quot;), &#xA;  row.names = c(NA,-2L), class = &quot;data.frame&quot;)                                  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How would I do this and average the &lt;code&gt;n&lt;/code&gt; outputs of &lt;code&gt;funToRepeat&lt;/code&gt; in &lt;code&gt;R&lt;/code&gt;, efficiently?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A very bad example with &lt;code&gt;n=10&lt;/code&gt; may be:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dfAvg &amp;lt;- as.data.frame(list(V1=c(0, 0), V2=c(0, 0)))&#xA;for(i in 1:10){&#xA;    dfAvg &amp;lt;- dfAvg + funToRepeat(dfTest)&#xA;}&#xA;dfAvg &amp;lt;- dfAvg/10&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Please notice, I am not trying to bootstrap. I want to operate &lt;code&gt;funToRepeat&lt;/code&gt; on my data set and take the average of the &lt;code&gt;n&lt;/code&gt; outputs. I want to do this as I am adding &lt;code&gt;err&lt;/code&gt;, a random term, to my data.frame and want to provide a reasonable answer. (I am not doing exactly this in real life, but something very similar)&lt;/p&gt;&#xA;" OwnerUserId="1307" LastActivityDate="2011-03-08T17:04:35.610" Title="How to efficiently repeat a function on a data set in R?" Tags="&lt;r&gt;" AnswerCount="1" CommentCount="3" FavoriteCount="3" />
  <row Id="8000" PostTypeId="1" AcceptedAnswerId="8014" CreationDate="2011-03-08T07:16:01.813" Score="7" ViewCount="152" Body="&lt;p&gt;Recurrent neural networks differ from &quot;regular&quot; ones by the fact that they have a &quot;memory&quot; layer. Due to this layer, recurrent NN's are supposed to be useful in time series modelling. However, I'm not sure I understand correctly how to use them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say I have the following time series (from left to right): &lt;code&gt;[0, 1, 2, 3, 4, 5, 6, 7]&lt;/code&gt;, my goal is to predict &lt;code&gt;i&lt;/code&gt;-th point using points &lt;code&gt;i-1&lt;/code&gt; and &lt;code&gt;i-2&lt;/code&gt; as an input (for each &lt;code&gt;i&amp;gt;2&lt;/code&gt;). In a &quot;regular&quot;, non-recurring ANN I would do process the data as follows:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;&lt;pre&gt;&lt;code&gt; target| input&#xA;      2| 1 0&#xA;      3| 2 1&#xA;      4| 3 2&#xA;      5| 4 3&#xA;      6| 5 4&#xA;      7| 6 5 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I would then create a net with two input and one output node and train it with the data above.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How does one need to alter this process (if at all) in the case of recurrent networks? &lt;/p&gt;&#xA;" OwnerUserId="1496" LastActivityDate="2011-03-09T13:57:54.917" Title="Proper way of using recurrent neural network for time series analysis" Tags="&lt;time-series&gt;&lt;machine-learning&gt;&lt;neural-networks&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="8001" PostTypeId="1" CreationDate="2011-03-08T07:37:42.527" Score="2" ViewCount="87" Body="&lt;p&gt;I want to discover which genes are expressed in only one of five treatments. This is my pipeline:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;ANOVA between the five treatments &lt;/li&gt;&#xA;&lt;li&gt;Holm multiple testing correction &lt;/li&gt;&#xA;&lt;li&gt;Tukey for significant genes discovered in step 2&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;My question is: Should I also correct Tukey p-values for example multiplying the p-value by the number of significant ANOVA p-values (Bonferroni correction on the number of tests performed), or should I only correct at the level of ANOVA?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks a lot in advance,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rossella&lt;/p&gt;&#xA;" OwnerUserId="3614" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-03-08T07:44:41.863" LastActivityDate="2011-03-08T07:44:41.863" Title="ANOVA and multiple testing correction in gene screening" Tags="&lt;anova&gt;&lt;multiple-comparisons&gt;&lt;genetics&gt;" CommentCount="4" FavoriteCount="2" />
  <row Id="8002" PostTypeId="2" ParentId="7999" CreationDate="2011-03-08T08:37:39.897" Score="6" ViewCount="" Body="&lt;p&gt;I for one would put everything to the list and then use &lt;code&gt;Reduce&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;funToRepeat &amp;lt;- function(dims) {&#xA;    err &amp;lt;- runif(prod(dims))&#xA;    matrix(err,nrow=dims[1],ncol=dims[2]) &#xA;}&#xA;&#xA;ll &amp;lt;- alply(cbind(rep(10,10),rep(20,10)),1,funToRepeat)&#xA;&#xA;sum.ll &amp;lt;- Reduce(&quot;+&quot;,ll)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then convert to data.frame you need after dividing by length of the list to get the average. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that your function does not need whole of a &lt;code&gt;data.frame&lt;/code&gt; to operate, only its dimensions. The conversion to data.frame is unnecessary cost here, so it is better not to use it.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-03-08T17:04:35.610" LastActivityDate="2011-03-08T17:04:35.610" CommentCount="3" />
  <row Id="8004" PostTypeId="2" ParentId="7975" CreationDate="2011-03-08T09:35:09.840" Score="3" ViewCount="" Body="&lt;p&gt;The same intuition as in cross-section regression can be used in time-series regression. It is perfectly valid to try to explain the trend using other variables. The main difference is that it is implicitly assumed that the regressors are random variables. So in regression model:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Y_t=\beta_0+X_{t1}\beta_1+...+X_{tk}\beta_k+\varepsilon_t$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;we require $E(\varepsilon_t|X_{t1},...,X_{tk})=0$ instead of $E\varepsilon_t=0$ and &#xA;$E(\varepsilon_t^2|X_{t1},...,X_{tk})=\sigma^2$ instead of $E\varepsilon_t^2=\sigma^2$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The practical part of regression stays the same, all the usual statistics and methods apply. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The hard part is to show for which types of random variables, or in this cases stochastic processes $X_{tk}$ we can use classical methods. The usual central limit theorem cannot be applied, since it involves independent random variables. Time series processes are usually not independent. This is where importance of stationarity comes into play. It is shown that for large part of stationary processes the central limit theorem can be applied, so classical regression analysis can be applied. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The main caveat of time-series regression is that it can massively fail when the regressors are not stationary. Then usual regression methods can show that the trend is explained, when in fact it is not. So if you want to explain trend you must check for non-stationarity before proceeding. Otherwise you might arrive at false conclusions.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-03-08T09:35:09.840" CommentCount="8" />
  <row Id="8005" PostTypeId="1" CreationDate="2011-03-08T10:06:32.450" Score="3" ViewCount="63" Body="&lt;p&gt;I have a hypothetical experiment where I am comparing scores on some measure at 1 year to scores on the measure at 4 years. I use a non-paired t-test to see if there is a significant difference between the two. Then, I do:  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(4 year mean) - (1 year mean)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, I want to take that value, a function of two different distributions that have different n and SD, and compare it to another value that was obtained in the same way from two similar distributions (1st &amp;amp; 4th year but under a different independent variable). How can I do this? And is it valid do 4yrMean - 1yrMean to obtain the value, or should I do p-value4  -  p-value1 or something?&lt;/p&gt;&#xA;" OwnerUserId="3443" LastEditorUserId="3443" LastEditorDisplayName="" LastEditDate="2011-03-09T02:41:24.117" LastActivityDate="2011-03-09T02:52:53.240" Title="Hypothesis testing on values that are functions of multiple distributions" Tags="&lt;hypothesis-testing&gt;" AnswerCount="1" />
  <row Id="8006" PostTypeId="1" CreationDate="2011-03-08T10:48:41.757" Score="2" ViewCount="125" Body="&lt;p&gt;How much data is needed to properly fit a GARCH(1,1) model?&lt;/p&gt;&#xA;" OwnerUserId="3588" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-03-08T11:34:08.070" LastActivityDate="2011-03-08T12:08:20.493" Title="Fitting a GARCH(1,1) model" Tags="&lt;time-series&gt;&lt;statistical-significance&gt;&lt;model-selection&gt;&lt;garch&gt;&lt;volatility-forecasting&gt;" AnswerCount="1" />
  <row Id="8007" PostTypeId="1" CreationDate="2011-03-08T10:50:34.163" Score="5" ViewCount="88" Body="&lt;p&gt;Workers in a factory are assembling items built from several parts. For each item they fetch the parts from the warehouse and then assemble the items.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suppose they need a certain time per item for assembly plus a certain time per part for fetching them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$D_{order} = n_{items} * D_{fetch} + n_{parts} * D_{assemble}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a list of past orders which tells me&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;$n_{items}$: how many items there are in that order&lt;/li&gt;&#xA;&lt;li&gt;$n_{parts}$: how many parts there are in that order and&lt;/li&gt;&#xA;&lt;li&gt;$D_{order}$: how long it took the worker to complete the order.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I want to estimate, based on the number of items and parts, how long a future order will take. So I need to find out $D_{fetch}$ and $D_{assemble}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which is the best way to plot the data to estimate those two factors?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is some sample data (order completion time on the Y-axis):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This chart shows the time to complete an order vs. the overall number of parts in it:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/CA5wM.jpg&quot; alt=&quot;time_vs_parts&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This chart shows the time to complete an order vs. the number of assembled items in it:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/3qPbG.jpg&quot; alt=&quot;time_vs_items&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I didn't tell before: Usually several of the items are the same product, so maybe there is a overhead per product for getting/reading the appropriate blueprints.&#xA;So this chart shows the time to complete an order vs. the number of unique products in it:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/cgQFf.jpg&quot; alt=&quot;time_vs_unique_products&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="3615" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-14T19:27:35.580" LastActivityDate="2011-05-14T21:46:15.897" Title="How to estimate time-per-product in a factory?" Tags="&lt;data-visualization&gt;&lt;modeling&gt;" AnswerCount="1" CommentCount="7" FavoriteCount="3" />
  <row Id="8008" PostTypeId="2" ParentId="5926" CreationDate="2011-03-08T11:07:54.290" Score="3" ViewCount="" Body="&lt;p&gt;Why not use Cohen's (1988, 1992) guidelines for effect size values? He defines a &lt;strong&gt;&quot;small&quot;&lt;/strong&gt; $(0.1 \leq r \leq 0.23)$, &lt;strong&gt;&quot;medium&quot;&lt;/strong&gt; $(0.24 \leq r \leq 0.36)$ and &lt;strong&gt;&quot;large&quot;&lt;/strong&gt; $(r \geq 0.37)$ effect. This would suggest to use MANOVA with variables whose $r$ is below $0.37$.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;References&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Cohen, J. (1988) Statistical Power Analysis for the Behavioral Sciences. 2nd Ed. Routledge Academic, 567 pp.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cohen, J (1992). A power primer. Psychological Bulletin 112, 155–159.&lt;/p&gt;&#xA;" OwnerUserId="3467" LastEditorUserId="3467" LastEditorDisplayName="" LastEditDate="2011-03-16T08:44:56.973" LastActivityDate="2011-03-16T08:44:56.973" />
  <row Id="8009" PostTypeId="1" CreationDate="2011-03-08T11:16:35.207" Score="1" ViewCount="73" Body="&lt;p&gt;We are measuring conversion rates (% of visitors who bought) on an e-commerce site. &#xA;The test apply to a segment of visitors who meet specific criteria (for example people from a certain country). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The people from the segment are divided into 2 groups. Part of them see a banner and the other don't (control group). Usually the control group is 30% of visitors. The test begin after the banner is shown to all users in the segment for a while so the data of exposed people extends much longer than the data of the control group.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So at a given time we have for example X people exposed and Xb of them converted; likewise for Y people who were not exposed, Yb of them were converted. Y and Yb are much smaller than X and Xb.&#xA;The conversion Rate for X is Xb/X and for Y is Yb/Y.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My &lt;strong&gt;first question&lt;/strong&gt; is how to determine statistical validity. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We used Chi-square test to do it, and got results similar to those from this &lt;a href=&quot;http://www.usereffect.com/split-test-calculator&quot; rel=&quot;nofollow&quot;&gt;on-line calculator&lt;/a&gt; (implemented similar to &lt;a href=&quot;http://math.hws.edu/javamath/ryan/ChiSquare.html&quot; rel=&quot;nofollow&quot;&gt;table1 here&lt;/a&gt;).&#xA;However, sometimes it looks like the number of purchases is extremely small yet the chi-square test says its valid (&gt;95%).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a real life example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X=189 Xb=1 Y=93 Yb=3&#xA;Conversion X= 0.5% Conversion Y=3.2% &#xA;Statistical confidence (chi-square based) 92.8%&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Altough the confidence is below 95%, it seems too close for determining its confidence while there was only 1 conversion for X.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My &lt;strong&gt;second question&lt;/strong&gt; is then: Should there also be a requirement for a minimal number of conversions for the confidence to be valid? If so, how do we calculate it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="3506" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-03-08T11:31:35.020" LastActivityDate="2011-03-08T11:31:35.020" Title="Minimal number of samples/conversions for statistical validity" Tags="&lt;confidence-interval&gt;&lt;chi-squared&gt;&lt;cross-validation&gt;&lt;validation&gt;" CommentCount="2" FavoriteCount="1" />
  <row Id="8010" PostTypeId="1" AcceptedAnswerId="8016" CreationDate="2011-03-08T12:01:10.703" Score="3" ViewCount="117" Body="&lt;p&gt;Here is the dual problem for L2 support vector machine:&#xA;$$\max_{\alpha\in\mathbb{R}^{n}} 2\alpha^{T}y-\alpha^{T}\left(K+n\lambda Id_{\mathbb{R}^{n}}\right)\alpha$$&#xA;$$\forall i\in\left\{ 1,\ldots,n\right\} ,\,\alpha_{i}y_{i} \geq0$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, using the Lagrangian formulation, I get the following. Is there an argument missing?&#xA;$$\max_{\mu\in\mathbb{R}^{n},\nu\in\mathbb{R}^{n}} \sum_{i=1}^{n}\mu_{i}-\frac{1}{4\lambda}\sum_{i=1}^{n}\sum_{j=1}^{n}\mu_{i}y_{i}\mu_{j}y_{j}K\left(x_{i},x_{j}\right)-\frac{n}{4}\sum_{i=1}^{n}\left(\mu_{i}+\nu_{i}\right)^{2}$$&#xA;$$\forall i\in\left\{ 1,\ldots,n\right\} ,\,\mu_i\geq0$$&#xA;$$\forall i\in\left\{ 1,\ldots,n\right\} ,\,\nu_i\geq0$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We want the max, which explains $\nu=0$.&lt;/p&gt;&#xA;" OwnerUserId="1351" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-03-09T10:09:17.420" LastActivityDate="2011-03-09T18:19:41.447" Title="Dual problem for L2 support vector machine" Tags="&lt;homework&gt;&lt;svm&gt;&lt;loss-functions&gt;&lt;proof&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="0" />
  <row Id="8011" PostTypeId="2" ParentId="8006" CreationDate="2011-03-08T12:08:20.493" Score="3" ViewCount="" Body="&lt;p&gt;Depends on the coefficients. Simple Monte-Carlo analysis suggests that a lot, about 1000, which is quite surprising. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;N &amp;lt;- 1000&#xA;n &amp;lt;- 1000+N&#xA;a &amp;lt;- c(0.2, 0.3, 0.4)  # GARCH(1,1) coefficients&#xA;e &amp;lt;- rnorm(n)  &#xA;x &amp;lt;- double(n)&#xA;s &amp;lt;-double(n)&#xA;x[1] &amp;lt;- rnorm(1) &#xA;s[1] &amp;lt;- 0&#xA;for(i in 2:n)  # Generate GARCH(1,1) process&#xA;{&#xA;  s[i] &amp;lt;- a[1]+a[3]*s[i-1]+a[2]*x[i-1]^2    &#xA;  x[i] &amp;lt;- e[i]*sqrt(s[i])&#xA;}&#xA;x &amp;lt;- ts(x[1000+1:N])&#xA;x.garch &amp;lt;- garchFit(data=x)  # Fit GARCH(1,1) &#xA;summary(x.garch)     &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I modified example code from &lt;code&gt;garch&lt;/code&gt; from &lt;strong&gt;tseries&lt;/strong&gt; package, but I used &lt;code&gt;garchFit&lt;/code&gt; from &lt;strong&gt;fGarch&lt;/strong&gt; package, since it seemed that it gave better results.  I used 1000 values for  burn-in. &lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-03-08T12:08:20.493" />
  <row Id="8012" PostTypeId="1" CreationDate="2011-03-08T12:13:11.033" Score="3" ViewCount="49" Body="&lt;p&gt;I start with a presumably non-stationary time series. By some criterium $k$ I extract $n_k$ pairs of times $(t_{k,i,2},t_{k,i,1})$.&#xA;Now I compute the following quantity: $\bar{\tau} = \frac{1}{K} \sum_k \frac{1}{n_k} \sum_i (t_{k,i,2}-t_{k,i,1})$&#xA;What I'm looking for is the error of that quantity. I first began by summing up the variances of the &quot;inner&quot; mean, but I guess thats far off because of the high correlations.&#xA;So I guess, somehow the autocorrelation function has to be incorporated. But as I'm not too familiar with time series, I don't even really know what to look for.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Updated with info from comments&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the example of how the pairs of times are chosen:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/nH0Dq.jpg&quot; alt=&quot;enter image description here&quot;&gt;&#xA;$A(t)$ is some observed quantity. $A_k$ is an upper threshold and $A_0$ some fixed lower one $(0\notin k)$. In physics we call the $y_{ki}=t_{k,i,2}-t_{k,i,1}$ first passage times.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've been thinking on this for a while and I've got some ideas, but I'm not sure. First I came up with this (acv is the autocovariance function): $\sigma^2(\bar\tau) = \frac{1}{K^2}\sum_{\forall t_{k_1,i_1,j_1}, \forall t_{k_2,i_2,j_2}} a(j_1,j_2) \frac{1}{n_{k_1} n_{k_2}} acv(|t_{k_1,i_1,j_1}-t_{k_2,i_2,j_2}|)$. $a(j_1, j_2)$ is supposed to be some function that is -1 if either of the arguments is 1 (meaning that the corresponding $t$ is subtracted in the formula in my original post).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I don't see that this reduces to $\sigma^2(\bar\tau) = \frac{1}{K^2}\sum_k\frac{1}{n_k^2}\sigma_k^2$, which was my original guess assuming no correlation. $\sigma_k^2$ denotes the variance of the $k$th &quot;inner mean&quot;. So finally I thought about something like this: $\sigma^2(\bar\tau) = \frac{1}{K^2}\sum_k\frac{1}{n_k^2}\sigma_k^2 + \frac{1}{K^2}\sum_{\forall t_{k_1,i_1,j_1}, \forall t_{k_2,i_2,j_2}, t_{k_1,i_1,j_1}\neq t_{k_2,i_2,j_2}} a(j_1, j_2) \frac{1}{n_{k_1} n_{k_2}} acv(|t_{k_1,i_1,j_1}-t_{k_2,i_2,j_2}|)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can you help me? Am I thinking at least in the right direction? Thanks, Julius&lt;/p&gt;&#xA;" OwnerUserId="3104" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-10T15:51:47.273" LastActivityDate="2011-03-10T15:51:47.273" Title="Error of a quantity computed from a time series" Tags="&lt;time-series&gt;&lt;correlation&gt;&lt;error&gt;&lt;error-propagation&gt;" AnswerCount="1" CommentCount="5" />
  <row Id="8013" PostTypeId="1" CreationDate="2011-03-08T12:51:04.077" Score="7" ViewCount="210" Body="&lt;p&gt;I have data that is equivalent to:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;shopper_1 = ['beer', 'eggs', 'water',...]&#xA;shopper_2 = ['diapers', 'beer',...]&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I would like to do some analysis on this data set to get a correlation matrix that would have an implication similar to: if you bought x, you are likely to buy y.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using python (or perhaps anything but MATLAB), how can I go about that? Some basic guidelines, or pointers to where I should look would help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit - What I have learned: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;These kinds of problems are known as association rule discovery. Wikipedia has a &lt;a href=&quot;http://en.wikipedia.org/wiki/Association_rule_learning&quot; rel=&quot;nofollow&quot;&gt;good article&lt;/a&gt; covering some of the common algorithms to do so. The classic algorithm to do so seems to be Apriori, due Agrawal et. al.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;That lead me to &lt;a href=&quot;http://orange.biolab.si/&quot; rel=&quot;nofollow&quot;&gt;orange&lt;/a&gt;, a python interfaced data mining package. For Linux, the best way to install it seems to be from source using the supplied setup.py&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Orange by default reads input from files, formatted in one of several supported ways.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Finally, a simple Apriori association rule learning is &lt;a href=&quot;http://orange.biolab.si/doc/reference/associationRules.htm&quot; rel=&quot;nofollow&quot;&gt;simple&lt;/a&gt; in orange.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="3618" LastEditorUserId="3618" LastEditorDisplayName="" LastEditDate="2011-03-09T05:54:19.903" LastActivityDate="2011-04-11T16:13:02.430" Title="How to do a 'beer and diapers' correlation analysis" Tags="&lt;correlation&gt;&lt;econometrics&gt;&lt;python&gt;&lt;cross-correlation&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="1" />
  <row Id="8014" PostTypeId="2" ParentId="8000" CreationDate="2011-03-08T12:51:29.233" Score="5" ViewCount="" Body="&lt;p&gt;What you describe is in fact a &quot;sliding time window&quot; approach and is different to recurrent networks. You can use this technique with any regression algorithm. There is a huge limitation to this approach: events in the inputs can only be correlatd with other inputs/outputs which lie at most t timesteps apart, where t is the size of the window.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;E.g. you can think of a Markov chain of order t. RNNs don't suffer from this in theory, however in practice learning is difficult.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is best to illustrate an RNN in contrast to a feedfoward network. Consider the (very) simple feedforward network $y = Wx$ where $y$ is the output, $W$ is the weight matrix, and $x$ is the input.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, we use a recurrent network. Now we have a sequence of inputs, so we will denote the inputs by $x^{i}$ for the ith input. The corresponding ith output is then calculated via $y^{i} = Wx^i + W_ry^{i-1}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, we have another weight matrix $W_r$ which incorporates the output at the previous step linearly into the current output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is of course a simple architecture. Most common is an architecture where you have a hidden layer which is recurrently connected to itself. Let $h^i$ denote the hidden layer at timestep i. The formulas are then:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$h^0 = 0$$&#xA;$$h^i = \sigma(W_1x^i + W_rh^{i-1})$$&#xA;$$y^i = W_2h^i$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $\sigma$ is a suitable non-linearity/transfer function like the sigmoid. $W_1$ and $W_2$ are the connecting weights between the input and the hidden and the hidden and the output layer. $W_r$ represents the recurrent weights.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a diagram of the structure: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/xHmPm.png&quot; alt=&quot;schematic&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="2860" LastEditorUserId="2860" LastEditorDisplayName="" LastEditDate="2011-03-08T15:43:55.373" LastActivityDate="2011-03-08T15:43:55.373" />
  <row Id="8015" PostTypeId="1" CreationDate="2011-03-08T13:16:00.873" Score="6" ViewCount="120" Body="&lt;p&gt;I was wondering if anyone could help me with information about Kurtosis (i.e. is there any way to transform your data to reduce it?)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a questionnaire dataset with a large number of cases and variables. For a few of my variables, the data shows pretty high kurtosis values (i.e. a leptokurtic distribution) which is derived from the fact that many of the participants gave the exact same score for the variable. I do have a particularly large sample size, so according to central limit theorem, violations of normality should still be fine. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem however, is that fact that the particularly high levels of Kurtosis are producing a number of univariate outliers in my dataset. As such, even if I transform the data, or remove/adjust the outliers, the high levels of kurtosis means that the next most extreme scores automatically become outliers. I aim to use (Discriminant function analysis). DFA is said to be robust to departures from normality provided that the violation is caused by skewness and not outliers. Furthermore, DFA is also said to be particularly influenced by outliers in the data (Tabachnick &amp;amp; Fidel). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas of how to get around this? (My initial thought was some way of controlling the Kurtosis, but isn’t it kind of a good thing if most of my sample are giving similar ratings?)&lt;/p&gt;&#xA;" OwnerUserId="3619" LastActivityDate="2011-03-08T14:11:17.500" Title="Treatment of outliers produced by Kurtosis" Tags="&lt;distributions&gt;&lt;assumptions&gt;&lt;kurtosis&gt;&lt;discriminant-analysis&gt;&lt;parametric&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="8016" PostTypeId="2" ParentId="8010" CreationDate="2011-03-08T13:19:37.123" Score="2" ViewCount="" Body="&lt;p&gt;The initial problem is:&#xA;$$\min_{f\in H}\frac{1}{n}\sum_{i=1}^{n}\phi\left(y_{i}f\left(x_{i}\right)\right)+\lambda\left\Vert f\right\Vert _{H}^{2}$$&#xA;$$\lambda\geq0$$&#xA;$$\phi\left(u\right)=\max\left(1-u,\,0\right)^{2}$$&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Since, $f=\sum_{i=1}^{n}\alpha_i K_{x_i}$ and we are considering a RKHS, the primal problem is:&#xA;$$\min_{\alpha\in\mathbb{R}^{n},\zeta\in\mathbb{R}^{n}}\frac{1}{n}\sum_{i=1}^{n}\zeta_{i}^{2}+\lambda\alpha^{T}K\alpha$$&#xA;$$\forall i\in\left\{ 1,\ldots,n\right\} ,\,\zeta_{i}\geq0$$&#xA;$$\forall i\in\left\{ 1,\ldots,n\right\} ,\,\zeta_{i}-1+y_{i}\left(K\alpha\right)_{i}\geq0$$&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Using Lagrangian multipliers, we get the result mentioned in the question. The computations are right: $\nu=0$ gives the result we want to achieve since there is a special link between $\alpha$ and $\mu$, thanks to the Lagrangian formulation:&#xA;$$\forall i\in\left\{ 1,\ldots,n\right\} ,\,\alpha_{i}^{*}=\frac{\mu_{i}y_{i}}{2\lambda}$$&#xA;where:&#xA;$$\forall i\in\left\{ 1,\ldots,n\right\} ,\, y_i\in\left\{ -1,1\right\}$$&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Why is $\nu=0$?&#xA;Of course, $\nu=0$ since:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;there is only one term depending on $\nu$&lt;/li&gt;&#xA;&lt;li&gt;we want the max&lt;/li&gt;&#xA;&lt;li&gt;$\mu$ and $\nu$ are positive.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="1351" LastEditorUserId="1351" LastEditorDisplayName="" LastEditDate="2011-03-09T18:19:41.447" LastActivityDate="2011-03-09T18:19:41.447" CommentCount="1" />
  <row Id="8017" PostTypeId="2" ParentId="7989" CreationDate="2011-03-08T13:35:37.283" Score="4" ViewCount="" Body="&lt;p&gt;One thing that you must recognise is that the term &quot;noise&quot; is &lt;strong&gt;relative&lt;/strong&gt; not absolute.  Thinking about an obvious example (not anything to do with statistics per se), imagine you are at a night club.  What is &quot;noise&quot; in here?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you trying to have a conversation with someone, then the music is &quot;noise&quot;, and so are the other conversations going on inside the night club.  The &quot;signal&quot; is the dialogue of the converstion.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are dancing, then the music is no longer &quot;noise&quot;, but it becomes the &quot;signal&quot; to which you react to.  The conversation has changed from &quot;signal&quot; into &quot;noise&quot; merely by a change in your state of mind!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Statistics works in &lt;strong&gt;exactly the same way&lt;/strong&gt; (you could in theory develop a statistical model which describes both these &quot;noise&quot; processes).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a regression setting, take the simple linear case with 1 covariate, X, and 1 dependent variable Y.  What you are effectively saying here is that you want to extract the &lt;em&gt;linear&lt;/em&gt; component of X that is related to Y.  The general conditions for &quot;additive noise&quot;, so that you have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Y_{i}=a+b X_{i} + n_{i}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is &quot;small noise&quot;, or more precisely in a mathematical sense, the conditions for Taylor Series linearisation are good enough for your purposes.  To show this in the multiplicative case, suppose the actual distribution is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Y_{i}=(a+b X_{i})n_{i}^{(T)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which we can consider as a function of $n_{i}^{(T)}$, a taylor series expansion about the value 1 gives:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$(a+b X_{i})n_{i}^{(T)}=a+b X_{i}+(a+b X_{i})(n_{i}^{(T)}-1)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the noise is &quot;small&quot; then it should not differ much from 1, and so the second term will be much small than the first, when the noise is &quot;small&quot; compared to the &quot;signal&quot;, which is given by the regression line.  So we can make the approximation&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$(a+b X_{i})n_{i}^{(T)} \approx a+b X_{i}+n_{i}^{(A)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where the approximate noise ignores the dependence on the actual regression line.  This dependence will only matter when the noise is large, compared to the slope.  If the slope does not vary appreciably over the range of the model, then the &quot;fanning&quot; of the true noise will be indistinguishable from independent noise.  This also applies for the general case, for any function &lt;em&gt;g&lt;/em&gt; satisfying $g(1)=1$:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$(a+bX)g(n_{i}^{(T)}) \approx a+b X_{i}+(g(n_{i}^{(T)})-1)(a+bX)g^{(1)}(n_{i}^{(T)}) $$&#xA;$$\approx a+b X_{i}+n_{i}^{(A)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$g^{(1)}(x)=\frac{\partial g(x)}{\partial x}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But note that this approximation will only apply in the case of &quot;small noise&quot;, or that $g(n_{i}^{(T)}) \approx 1$.  This &quot;smallness&quot; make all the details of the particular function &lt;em&gt;g&lt;/em&gt; irrelevant for all practical purpose.  Going through the laborious calculations using &lt;em&gt;g&lt;/em&gt; directly will only matter in the decimal places (estimate is 1.0189, using true &lt;em&gt;g&lt;/em&gt; it is 1.0233).  The more the function &lt;em&gt;g&lt;/em&gt; departs from 1, the further up the decimal values will be affected.  This is why &quot;small noise&quot; is required&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-03-08T13:35:37.283" CommentCount="3" />
  <row Id="8018" PostTypeId="2" ParentId="8015" CreationDate="2011-03-08T14:11:17.500" Score="4" ViewCount="" Body="&lt;p&gt;The obvious &quot;common sense&quot; way to resolving your problem is to&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Get the &lt;strong&gt;conclusion&lt;/strong&gt; using the full data set.  i.e. what results will you declare ignoring intermediate calculations?&lt;/li&gt;&#xA;&lt;li&gt;Get the &lt;strong&gt;conclusion&lt;/strong&gt; using the data set with said &quot;outliers&quot; removed.  i.e. what results will you declare ignoring intermediate calculations?&lt;/li&gt;&#xA;&lt;li&gt;Compare step 2 with step 1&lt;/li&gt;&#xA;&lt;li&gt;If there is no difference, forget you even had a problem.  Outliers are irrelevant to &lt;strong&gt;your conclusion&lt;/strong&gt;.  The outliers may influence some other conclusion that may have been drawn using these data, but this is irrelevant to your work.  It is somebody else's problem.&lt;/li&gt;&#xA;&lt;li&gt;If there is a difference, then you have basically a question of &quot;trust&quot;.  Are these &quot;outliers&quot; real in the sense that they genuinely represent something about your analysis?  Or are the &quot;outliers&quot; bad in that they come from some &quot;contaminated source&quot;?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In situation 5 you basically have a case of what-ever &quot;model&quot; you have used to describe the &quot;population&quot; is incomplete - there are details which have been left unspecified, but which matter to the conclusions.  There are two ways to resolve this, corresponding to the two &quot;trust&quot; scenarios:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Add some additional structure to your model so that is describes the &quot;outliers&quot;.  So instead of $P(D|\theta)$, consider $P(D|\theta)=\int P(\lambda|\theta)P(D|\theta,\lambda) d\lambda$.&lt;/li&gt;&#xA;&lt;li&gt;Create a &quot;model-model&quot;, one for the &quot;good&quot; observations, and one for the &quot;bad&quot; observations.  So instead of $P(D|\theta)$ you would use $P(D|\theta)=G(D|\theta)u+B(D|\theta)(1-u)$, were &lt;em&gt;u&lt;/em&gt; is the probability of obtaining a &quot;good&quot; observation in your sample, and &lt;em&gt;G&lt;/em&gt; and &lt;em&gt;B&lt;/em&gt; represent the models for the &quot;good&quot; and &quot;bad&quot; data.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Most of the &quot;standard&quot; procedures can be shown to be approximations to these kind of models.  The most obvious one is by considering case 1, where the variance has been assumed constant across observations.  By relaxing this assumption into a distribution you get a mixture distribution.  This is the connection between &quot;normal&quot; and &quot;t&quot; distributions.  The normal has fixed variance, whereas the &quot;t&quot; mixes over different variances,  the amount of &quot;mixing&quot; depends on the degrees of freedom.  High DF means low mixing (outliers are unlikely), low DF means high mixing (outliers are likely).  In fact you could take case 2 as a special case of case 1, where the &quot;good&quot; observations are normal, and the &quot;bad&quot; observations are Cauchy (t with 1 DF).&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-03-08T14:11:17.500" CommentCount="2" />
  <row Id="8019" PostTypeId="1" AcceptedAnswerId="8023" CreationDate="2011-03-08T15:06:44.510" Score="5" ViewCount="83" Body="&lt;p&gt;Let's say I test how variable &lt;code&gt;Y&lt;/code&gt; depends on variable &lt;code&gt;X&lt;/code&gt; under different experimental conditions and obtain the following graph:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/hpihy.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The dash lines in the graph above represent linear regression for each data series (experimental setup) and the numbers in the legend denote the Pearson correlation of each data series.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to calculate the &quot;average correlation&quot; (or &quot;mean correlation&quot;) between &lt;code&gt;X&lt;/code&gt; and &lt;code&gt;Y&lt;/code&gt;. May I simply average the &lt;code&gt;r&lt;/code&gt; values? What about the &quot;average determination criterion&quot;, $R^2$? Should I calculate the average &lt;code&gt;r&lt;/code&gt; and than take the square of that value or should I compute the average of individual $R^2$'s?&lt;/p&gt;&#xA;" OwnerUserId="1496" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-08T15:57:47.940" LastActivityDate="2011-03-08T15:57:47.940" Title="Averaging correlation values" Tags="&lt;regression&gt;&lt;correlation&gt;&lt;mean&gt;&lt;averages&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="8021" PostTypeId="1" CreationDate="2011-03-08T15:34:34.147" Score="3" ViewCount="117" Body="&lt;p&gt;Hi all,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have posted a &lt;a href=&quot;http://stats.stackexchange.com/questions/7977/how-to-generate-uniformly-distributed-points-on-the-surface-of-the-3-d-unit-spher&quot;&gt;previous question&lt;/a&gt;, this is related but I think it is better to start another thread. This time, I am wondering how to generate uniformly distributed points inside the 3-d unit sphere and how to check the distribution visually and statistically too? I don't see the strategies posted there directly transferable to this situation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please help. Many thanks.&lt;/p&gt;&#xA;" OwnerUserId="3552" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-08T16:05:24.577" LastActivityDate="2011-03-08T16:05:24.577" Title="How to generate uniformly distributed points in the 3-d unit ball?" Tags="&lt;random-generation&gt;" AnswerCount="1" CommentCount="7" />
  <row Id="8022" PostTypeId="2" ParentId="8021" CreationDate="2011-03-08T15:52:11.650" Score="6" ViewCount="" Body="&lt;p&gt;The easiest way is to sample points uniformly in the corresponding hypercube and discard those that do not lie within the sphere. In 3D, this should not happen that often, about 50% of the time. (Volume of the hypercube is 1, volume of the sphere is $\frac{4}{3}\pi r^3 = 0.523...$.)&lt;/p&gt;&#xA;" OwnerUserId="2860" LastActivityDate="2011-03-08T15:52:11.650" CommentCount="1" />
  <row Id="8023" PostTypeId="2" ParentId="8019" CreationDate="2011-03-08T15:57:22.620" Score="3" ViewCount="" Body="&lt;p&gt;The simple way is to add a categorical variable $z$ to identify the different experimental conditions and include it in your model along with an &quot;interaction&quot; with $x$; that is, $y \sim z + x\#z$.  This conducts all five regressions at once.  Its $R^2$ is what you want.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-03-08T15:57:22.620" CommentCount="3" />
  <row Id="8025" PostTypeId="1" AcceptedAnswerId="8026" CreationDate="2011-03-08T16:31:51.660" Score="2" ViewCount="81" Body="&lt;p&gt;Precision is defined as:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;p = true positives / (true positives + false positives)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;What is the value of precision if (true positives + false positives) = 0? Is it just undefined?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Same question for recall:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;r = true positives / (true positives + false negatives)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In this case, what is the value of recall if (true positives + false negatives) = 0?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S. This question is very similar to the question &lt;a href=&quot;http://stats.stackexchange.com/questions/1773/what-are-correct-values-for-precision-and-recall-in-edge-cases&quot;&gt;What are correct values for precision and recall in edge cases?&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="3604" LastActivityDate="2011-03-08T17:02:35.940" Title="What are correct values for precision and recall when the denominators equal 0?" Tags="&lt;precision-recall&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="8026" PostTypeId="2" ParentId="8025" CreationDate="2011-03-08T17:02:35.940" Score="2" ViewCount="" Body="&lt;p&gt;The answers to the linked earlier question apply here too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If (true positives + false negatives) = 0 then no positive cases in the input data, so any analysis of this case has no information, and so no conclusion about how positive cases are handled.  You want N/A or something similar as the ratio result, avoiding a &lt;em&gt;division by zero&lt;/em&gt; error&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If (true positives + false positives) = 0 then all cases have been predicted to be negative: this is one end of the ROC curve. Again, you want to recognise and report this possibility while avoiding a &lt;em&gt;division by zero&lt;/em&gt; error.  &lt;/p&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-03-08T17:02:35.940" CommentCount="2" />
  <row Id="8027" PostTypeId="1" CreationDate="2011-03-08T18:11:56.820" Score="2" ViewCount="78" Body="&lt;p&gt;I have the following problem:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Suppose we have a sample $Y_1,...,Y_n \sim N(\mu, \sigma^2)$. Find the&#xA;  pivotal function for $\sigma^2$ when&#xA;  $\mu$ is known and when $\mu$ is&#xA;  unknown.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I know that you need to find a statistic where its distribution doesn't depend on the parameter. For $\mu$ unknown I'm assuming you use $(n-1)/S^2$ which is a chi-squared distribution ($n-1$ degrees of freedom). For $\mu$ known, what do you use? And why? In my notes they say $(\bar Y  - \mu) / (\sigma\sqrt{n}) \sim N(0,1)$ but then it goes on to say something about summing the $Y_i$'s. Why?&lt;/p&gt;&#xA;" OwnerUserId="3483" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-03-08T21:30:03.127" LastActivityDate="2011-04-10T16:39:56.567" ClosedDate="2011-04-10T17:11:07.427" Title="Pivot functions" Tags="&lt;hypothesis-testing&gt;&lt;homework&gt;&lt;pivot&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="8028" PostTypeId="1" CreationDate="2011-03-08T18:35:12.617" Score="0" ViewCount="60" Body="&lt;p&gt;My second question is about Test Statistics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The questions and answers are on this PDF: &lt;a href=&quot;http://www.mediafire.com/?b74e633lxdb49rb&quot; rel=&quot;nofollow&quot;&gt;http://www.mediafire.com/?b74e633lxdb49rb&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand that they work out the join density and I know how to calculate this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I really don’t understand is how they figure out the test statistic. They randomly multiply by 1 in some cases and in some cases they just pick a sum. I don’t understand WHY that’s the test statistic - is it a guess?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, how do you check a test statistic you have calculated is correct?&lt;/p&gt;&#xA;" OwnerUserId="3483" LastActivityDate="2011-03-08T19:20:35.300" ClosedDate="2011-03-08T19:45:33.623" Title="Test Statistics" Tags="&lt;summary-statistics&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="8029" PostTypeId="1" AcceptedAnswerId="8030" CreationDate="2011-03-08T18:17:59.787" Score="2" ViewCount="126" Body="&lt;p&gt;Hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am writing a program in C# that requires me to use the Ttest formula.  I have to effectively interpret the Excel formula:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;=TTEST(range1,range2,1,3)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am using the formula given from &lt;a href=&quot;http://www.monarchlab.umn.edu/lab/research/stats/2SampleT.aspx&quot; rel=&quot;nofollow&quot;&gt;http://www.monarchlab.umn.edu/lab/research/stats/2SampleT.aspx&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and have interpreted into code as such:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; double TStatistic = (mean1 - mean2) / Math.Sqrt((Math.Pow(variance1, 2) / count1) + (Math.Pow(variance2, 2) / count2));&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, I don't fully understand t-test and the values I am getting are completely different than those calculated within Excel.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have been using the following ranges:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;R1:&#xA;91.17462277,&#xA;118.3936425,&#xA;96.6746393,&#xA;102.488785,&#xA;91.26831043&#xA;&#xA;R2:&#xA;17.20546254,&#xA;19.56969811,&#xA;19.2831241,&#xA;13.03360631,&#xA;13.86577314&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The value I am getting using my attempt is 1.8248, however that from Excel is 1.74463E-05.  Could somebody please point me in the right direction?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="3624" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-08T19:48:02.840" LastActivityDate="2011-03-08T19:48:02.840" Title="Implementing t-test" Tags="&lt;t-test&gt;" AnswerCount="1" CommentCount="6" />
  <row Id="8030" PostTypeId="2" ParentId="8029" CreationDate="2011-03-08T19:04:14.603" Score="3" ViewCount="" Body="&lt;p&gt;There are at least two problems with what you have done.  &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;You have misinterpreted the formula &#xA;$$t = \frac{\bar{x}_1-\bar{x}_2}{\sqrt{s_1^2 / n_1 + s_2^2 / n_2}}$$ since $s^2$ is already a variance (square of standard deviation) and does not need to be squared again. &lt;/li&gt;&#xA;&lt;li&gt;You are comparing eggs and omelettes: you need compare your &quot;calculated $t$-value, with $k$ degrees of freedom ... to the $t$ distribution table&quot;.  Excel has already done this with &lt;code&gt;TTEST()&lt;/code&gt;.     &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;There are other possible issues such as using a population variance or sample variance formula.&lt;/p&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-03-08T19:04:14.603" CommentCount="9" />
  <row Id="8032" PostTypeId="2" ParentId="8028" CreationDate="2011-03-08T19:20:35.300" Score="1" ViewCount="" Body="&lt;p&gt;They don't randomly multiply by 1.  What they do is split the joint density into the product of two functions: $g(\text{sufficient statistic},\text{parameter})$ and $h(\text{data})$.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The advantage of $h()$ is that it removes sometimes complicated parts of the density function which provide no useful information about estimating the parameter. At other times this is unnecessary, in which case $h()$ can be set to 1 and ignored. In either case you can concentrate on the &lt;a href=&quot;http://en.wikipedia.org/wiki/Sufficient_statistic&quot; rel=&quot;nofollow&quot;&gt;sufficient statistic&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-03-08T19:20:35.300" />
  <row Id="8033" PostTypeId="1" CreationDate="2011-03-08T20:00:17.407" Score="3" ViewCount="134" Body="&lt;p&gt;A lecturer wishes to &quot;grade on the curve&quot;. The students' marks seem to be normally distributed with mean 70 and standard deviation 8. If the lecturer wants to give 20% A's, what should be the &lt;strong&gt;threshold&lt;/strong&gt; between an A grade and a B grade?&lt;/p&gt;&#xA;" OwnerUserId="3623" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-08T20:23:45.947" LastActivityDate="2011-03-18T21:12:59.237" Title="Normal distribution homework" Tags="&lt;homework&gt;&lt;normal-distribution&gt;" AnswerCount="1" CommentCount="6" />
  <row Id="8034" PostTypeId="2" ParentId="8013" CreationDate="2011-03-08T20:59:01.840" Score="6" ViewCount="" Body="&lt;p&gt;In addition to the links that were given in comments, here are some further pointers:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://infolab.stanford.edu/~ullman/mining/assocrules.pdf&quot; rel=&quot;nofollow&quot;&gt;Association rules and frequent itemsets&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.93.791&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;Survey on Frequent Pattern Mining&lt;/a&gt; -- look around Table 1, p. 4&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;About Python, I guess now you have an idea of what you should be looking for, but the &lt;a href=&quot;http://orange.biolab.si/&quot; rel=&quot;nofollow&quot;&gt;Orange&lt;/a&gt; data mining package features a package on &lt;a href=&quot;http://orange.biolab.si/doc/reference/associationRules.htm&quot; rel=&quot;nofollow&quot;&gt;Association rules&lt;/a&gt; and Itemsets (although for the latter I cannot found any reference on the website).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I recently came across &lt;a href=&quot;http://code.google.com/p/pysuggest/&quot; rel=&quot;nofollow&quot;&gt;pysuggest&lt;/a&gt; which is &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;a Top-N recommendation engine that&#xA;  implements a variety of recommendation&#xA;  algorithms. Top-N recommender systems,&#xA;  a personalized information filtering&#xA;  technology, are used to identify a set&#xA;  of N items that will be of interest to&#xA;  a certain user. In recent years, top-N&#xA;  recommender systems have been used in&#xA;  a number of different applications&#xA;  such to recommend products a customer&#xA;  will most likely buy; recommend&#xA;  movies, TV programs, or music a user&#xA;  will find enjoyable; identify&#xA;  web-pages that will be of interest; or&#xA;  even suggest alternate ways of&#xA;  searching for information.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-11T16:13:02.430" LastActivityDate="2011-04-11T16:13:02.430" CommentCount="1" />
  <row Id="8035" PostTypeId="2" ParentId="7975" CreationDate="2011-03-08T21:06:10.590" Score="2" ViewCount="" Body="&lt;p&gt;When you have supporting/causal/helping/right-hand side/exogenous/predictor series, the approach that is preferred is to construct a single equation, multiple-input Transfer Function. One needs to examine possible model residuals for both unspecified/omitted deterministic inputs i.e. do Intervention Detection ala Ruey Tsay 1988 Journal of Forecasting and unspecified stochastic inputs via an ARIMA component. Thus you can explicitly include not only the user-suggested causals (and any needed lags !) but two kinds of omitted structures ( dummies and ARIMA ).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Care should be taken to ensure that the parameters of the final model do not change significantly over time otherwise data segmentation might be in order and that the residuals from the final model can not be proven to have heterogeneous variance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The trend in the original series may be due to trends in the predictor series or due to Autoregressive dynamics in the series of interest or potentially due to an omitted deterministic series proxied by a steady state constant or even one or more local time trends.  &lt;/p&gt;&#xA;" OwnerUserId="3382" LastActivityDate="2011-03-08T21:06:10.590" />
  <row Id="8036" PostTypeId="1" CreationDate="2011-03-08T21:08:10.473" Score="3" ViewCount="106" Body="&lt;p&gt;I have gazillion documents (your tax return) that I need to check for correctness, but I don't the manpower nor the will power to read through all of it.  Even if I do, I can not guarantee the quality and consistency of the proof reading process.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The only thing I can do is to pick a sample collection of document to proof read and assign &lt;code&gt;accept&lt;/code&gt; or &lt;code&gt;reject&lt;/code&gt; to it.  From that I want to determine the confidence level of certain confidence interval...I am clueless on what I should do next or I am using the right approach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have no experience with this problem domain, perhaps someone with more QA experience can point me in the right direction.  Like what question to ask ...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you for reading :)&lt;/p&gt;&#xA;" OwnerUserId="3625" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-31T07:36:08.540" LastActivityDate="2011-03-31T12:47:51.767" Title="Proofreading lots of documents based on small sample" Tags="&lt;confidence-interval&gt;&lt;quality-control&gt;" AnswerCount="1" CommentCount="9" FavoriteCount="1" />
  <row Id="8037" PostTypeId="2" ParentId="7975" CreationDate="2011-03-08T22:02:19.000" Score="5" ViewCount="" Body="&lt;p&gt;Based upon the comments that you've offered to the responses, you need to be aware of &lt;strong&gt;spurious causation&lt;/strong&gt;. Any variable with a time trend is going to be correlated with another variable that also has a time trend. For example, my weight from birth to age 27 is going to be highly correlated with your weight from birth to age 27. Obviously, my weight isn't &lt;strong&gt;caused&lt;/strong&gt; by your weight. If it was, I'd ask that you go to the gym more frequently, please.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you are familiar with cross-section data, I'll give you an omitted variables explanation. Let my weight be $x_t$ and your weight be $y_t$, where&#xA;$$\begin{align*}x_t &amp;amp;= \alpha_0 + \alpha_1 t + \epsilon_t \text{ and} \\ y_t &amp;amp;= \beta_0 + \beta_1 t + \eta_t.\end{align*}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then the regression&#xA;$$\begin{equation*}y_t = \gamma_0 + \gamma_1 x_t + \nu_t\end{equation*}$$&#xA;has an omitted variable---the time trend---that is correlated with the included variable, $x_t$. Hence, the coefficient $\gamma_1$ will be biased (in this case, it will be positive, as our weights grow over time).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you are performing time series analysis, you need to be sure that your variables are stationary or you'll get these spurious causation results. An exception would be integrated series, but I'd refer you to time series texts to hear more about that.&lt;/p&gt;&#xA;" OwnerUserId="401" LastActivityDate="2011-03-08T22:02:19.000" CommentCount="2" />
  <row Id="8040" PostTypeId="1" AcceptedAnswerId="8048" CreationDate="2011-03-08T22:32:45.597" Score="5" ViewCount="166" Body="&lt;p&gt;Is there an easy way to apply the trend line formula from a chart to any given X value in Excel?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, I want to get the Y value for a given X = $2,006.00. I've already taken the formula and retyped it out be:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;=-0.000000000008*X^3 - 0.00000001*X^2 + 0.0003*X - 0.0029&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am continually making adjustments to the trend line by adding more data, and don't want to retype out the formula every time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/tgq8c.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="3626" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-11T00:04:11.817" LastActivityDate="2011-03-11T00:04:11.817" Title="Use a trendline formula to get values for any given X with Excel" Tags="&lt;regression&gt;&lt;excel&gt;" AnswerCount="3" CommentCount="3" FavoriteCount="1" />
  <row Id="8041" PostTypeId="1" CreationDate="2011-03-08T22:40:06.737" Score="2" ViewCount="45" Body="&lt;p&gt;I have a bunch of experiments in which I am calculating precision and recall. I want to present a mean precision and recall for these experiments. Should these values be weighted by anything?&lt;/p&gt;&#xA;" OwnerUserId="3604" LastActivityDate="2011-03-08T22:40:06.737" Title="Should the mean precision and recall be weighted?" Tags="&lt;precision-recall&gt;" CommentCount="3" />
  <row Id="8042" PostTypeId="2" ParentId="7996" CreationDate="2011-03-08T22:49:23.450" Score="2" ViewCount="" Body="&lt;p&gt;EDIT: After some reflection, I modified my answer substantially.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The best thing to do would be to try to find a reasonable model for your data (for example, by using multiple linear regression).  If you cannot get enough data to do this, I would try the following &quot;non-parametric&quot; approach.  Suppose that in your data set, the covariate $A$ takes on the values $A=a_1, ..., a_{n_A}$, and likewise for $B$, $C$, etc.  Then what you can do is perform a linear regression on your dependent variables against the indicator variables $I(A= a_1), I(A=a_2), ..., I(A = a_{n_A}), I(B = b_1),...$ etc.  If you have enough data you can also include interaction terms such as $I(A=a_1, B=b_1)$.  Then you can use model selection techniques to eliminate the covariates that have the least effect.&lt;/p&gt;&#xA;" OwnerUserId="3567" LastEditorUserId="3567" LastEditorDisplayName="" LastEditDate="2011-03-08T23:40:46.820" LastActivityDate="2011-03-08T23:40:46.820" CommentCount="9" />
  <row Id="8043" PostTypeId="2" ParentId="7996" CreationDate="2011-03-08T23:15:32.683" Score="4" ViewCount="" Body="&lt;p&gt;A few comments: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Why did you go with your particular experimental design set-up? For example, fix A+B and vary C. What would you fix A + B at? If you are interesting in determining the effect of A and B, it seems a bit strange that you can fix them at &quot;optimal values&quot;. There are standard statistical techniques for sampling from multi-dimension space. For example, &lt;a href=&quot;http://en.wikipedia.org/wiki/Latin_hypercube_sampling&quot; rel=&quot;nofollow&quot;&gt;latin hypercubes&lt;/a&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Once you have your data, why not start with something simple, say multiple linear regression. You have 3 inputs A, B, C and one response variable. I suspect from your description, you may have to include interaction terms for the covariates.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A few comments on your regression:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Does the data fit your model? You need to check the residuals. Try googling &quot;R and regression&quot;.&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Just because one of your covariates has a smaller p-value, it doesn't mean that it has the strongest effect. For that, look at the estimates of the $\beta_i$ terms: 0.8, -0.23, -0.31.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So a one unit change in $A$ results in $T$ increasing by 0.8, whereas a  one unit change in $S$ results in $T$ decreasing by 0.23. However, are the units of the covariates comparable? For example, is it may be physically impossible for $A$ to change by 1 unit. Only you can make that decision.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;BTW, try not to update your question so that it changes your original meaning. If you have a new question, then just ask a new question.&lt;/p&gt;&#xA;" OwnerUserId="8" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2011-03-09T12:46:36.880" LastActivityDate="2011-03-09T12:46:36.880" CommentCount="2" />
  <row Id="8044" PostTypeId="1" CreationDate="2011-03-08T23:27:13.440" Score="10" ViewCount="207" Body="&lt;p&gt;I use R. Every day. I think in terms of data.frames, the apply() family of functions, object-oriented programming, vectorization, and ggplot2 geoms/aesthetics. I just started working for an organization that primarily uses SAS. I know there's a book about &lt;a href=&quot;http://www.amazon.com/o/ASIN/B001Q3LXNI/ref=nosim/gettgenedone-20&quot; rel=&quot;nofollow&quot;&gt;learning R for SAS users&lt;/a&gt;, but what are some good resources for R users who've never used SAS?&lt;/p&gt;&#xA;" OwnerUserId="36" LastActivityDate="2011-03-09T14:29:35.183" Title="Resources for an R user who must learn SAS" Tags="&lt;r&gt;&lt;sas&gt;" AnswerCount="3" CommentCount="7" FavoriteCount="1" />
  <row Id="8045" PostTypeId="5" CreationDate="2011-03-08T23:41:16.543" Score="0" ViewCount="" Body="&lt;p&gt;A &lt;em&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Distribution&quot; rel=&quot;nofollow&quot;&gt;distribution&lt;/a&gt;&lt;/em&gt; is a mathematical description of &lt;em&gt;probabilities&lt;/em&gt; or &lt;em&gt;frequencies.&lt;/em&gt;  It can be applied to &lt;em&gt;observed&lt;/em&gt; frequencies, &lt;em&gt;estimated&lt;/em&gt; probabilities or frequencies, and theoretically &lt;em&gt;hypothesized&lt;/em&gt; probabilities or frequencies.  Distributions can be &lt;em&gt;univariate,&lt;/em&gt; describing outcomes written with a single number, or &lt;em&gt;multivariate,&lt;/em&gt; describing outcomes requiring ordered tuples of numbers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Two devices are in common use to present univariate distributions.  The &lt;em&gt;cumulative&lt;/em&gt; form, or &quot;cumulative distribution function&quot; (CDF), gives--for every real number $x$--the chance (or frequency) of a value less than or equal to $x$.  The &quot;density&quot; form, or &quot;probability density function&quot; (PDF), is the derivative (rate of change) of the CDF.  The PDF might not exist (in this restricted sense), but a CDF always will exist.  The CDF for a set of observations is called the &quot;empirical density function&quot; (EDF).  Thus, its value at any number $x$ is the proportion of observations in the dataset less than or equal to $x$.&lt;/p&gt;&#xA;" OwnerUserId="-1" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-08T23:41:16.543" LastActivityDate="2011-03-08T23:41:16.543" />
  <row Id="8046" PostTypeId="4" CreationDate="2011-03-08T23:41:16.543" Score="0" ViewCount="" Body="A distribution is a mathematical description of *probabilities* or *frequencies.*" OwnerUserId="-1" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-08T23:41:16.543" LastActivityDate="2011-03-08T23:41:16.543" />
  <row Id="8048" PostTypeId="2" ParentId="8040" CreationDate="2011-03-09T00:12:52.290" Score="10" ViewCount="" Body="&lt;p&gt;Use &lt;code&gt;LINEST&lt;/code&gt;, as shown:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/D7buI.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The method is to create new columns (C:E here) containing the variables in the fit.  Perversely, &lt;code&gt;LINEST&lt;/code&gt; returns the coefficients in the &lt;em&gt;reverse&lt;/em&gt; order, as you can see from the calculation in the &quot;Fit&quot; column.  An example prediction is shown outlined in blue: all the formulas are exactly the same as for the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that &lt;code&gt;LINEST&lt;/code&gt; is an array formula: the answer will occupy p+1 cells in a row, where p is the number of variables (the last one is for the constant term).  Such formulas are created by selecting all the output cells, pasting (or typing) the formula in the formula textbox, and pressing Ctrl-Shift-Enter (instead of the usual Enter).&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-03-09T00:12:52.290" CommentCount="2" />
  <row Id="8049" PostTypeId="2" ParentId="8040" CreationDate="2011-03-09T00:46:15.910" Score="6" ViewCount="" Body="&lt;p&gt;Try trend(known_y's, known_x's, new_x's, const).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Column A below is X.  Column B is X^2 (the cell to the left squared).   Column C is X^3 (two cells to the left cubed).  The trend() formula is in Cell E24 where the cell references are shown in red.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &quot;known_y's&quot; are in E3:E22&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &quot;known_x's&quot; are in A3:C22&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &quot;new_x's&quot; are in A24:C24&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &quot;const&quot; is left blank.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cell A24 contains the new X, and is the cell to change to update the formula in E24&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cell B24 contains the X^2 formula (A24*A24) for the new X&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cell C24 contains the X^3 formula (A24*A24*A24) for the new X&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you change the values in E3:E22, the trend() function will update Cell E24 for your new input at Cell A24. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/g4LIc.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit ====================================&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Kirk,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There's not much to the spreadsheet.  I posted a &quot;formula view&quot; below.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &quot;known_x&quot; values are in green in A3:C22&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &quot;known_y&quot; values are in green in E3:E22&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &quot;new_x&quot; values are in cells A24:C24, where B24 and C24 are the formulas as shown.  And, cell E24 has the trend() formula.   You enter your new &quot;X&quot; in cell A24.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That's all there is to it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/39miz.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="2775" LastEditorUserId="2775" LastEditorDisplayName="" LastEditDate="2011-03-09T01:42:27.743" LastActivityDate="2011-03-09T01:42:27.743" CommentCount="2" />
  <row Id="8050" PostTypeId="2" ParentId="8044" CreationDate="2011-03-09T00:47:43.870" Score="12" ViewCount="" Body="&lt;p&gt;15 months ago, I started my current job as someone who had been using R exclusively for about 3 years; I had used SAS in my first-ever stats class, loathed it, and never touched it again until I started here.  Here's what has been helpful for me, and what hasn't:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Helpful:&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Colleagues' code&lt;/strong&gt;.  This is the single most useful source, for me.  Some of it was very good code, some of it was very bad code, but all of it showed me how to think in SAS.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;SUGI&lt;/strong&gt;.  Though they are often almost unbearably corny, there is a vast wealth of these little how-to papers all over the Internet.  You don't need to look for them; just Google, and they'll present themselves to you.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;The O'Reilly SQL Pocket Guide&lt;/strong&gt;, by Gennick.  I dodge a lot of SAS coding by using PROC SQL for data manipulation and summarization.  This is cheating, and I don't care.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;&lt;a href=&quot;http://svn.opendatafoundation.org/ddidext/org.opendatafoundation.data/references/sas/SAS_Informats_and_formats.pdf&quot; rel=&quot;nofollow&quot;&gt;This paper explaining formats and informats (PDF)&lt;/a&gt;&lt;/strong&gt;.  This is without a doubt the least-intuitive part of SAS for me.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;&lt;a href=&quot;http://www.ats.ucla.edu/stat/default.htm&quot; rel=&quot;nofollow&quot;&gt;UCLA's Academic Technology Services' Statistical Computing site&lt;/a&gt;&lt;/strong&gt;.  UCLA has heaps of great introductory material here, and there's a lot of parallel material between its R and SAS sections (&lt;a href=&quot;http://www.ats.ucla.edu/stat/sas/dae/&quot; rel=&quot;nofollow&quot;&gt;like these analysis examples&lt;/a&gt;).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Not helpful:&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Anything I've ever read that is intended for people transitioning between R and SAS.  I have the &quot;R and SAS&quot; book from Kleinman and Horton, which I've opened twice only to not find the answers I needed.  I've read a few other guides here and there.  Maybe it's just my learning style, but none of this stuff has ever stuck with me, and I inevitably end up googling for it once I actually need it.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You'll be okay, though.  Just read your colleagues' code, ask questions here and on StackOverflow, and - whatever you do - don't try to plot anything.&lt;/p&gt;&#xA;" OwnerUserId="71" LastEditorUserId="71" LastEditorDisplayName="" LastEditDate="2011-03-09T01:07:51.007" LastActivityDate="2011-03-09T01:07:51.007" CommentCount="7" />
  <row Id="8051" PostTypeId="2" ParentId="7990" CreationDate="2011-03-09T00:50:01.100" Score="2" ViewCount="" Body="&lt;p&gt;EDIT 2: We have to simplify the problem by removing the restriction that $x \ne y$ always in case two.  Otherwise correlation issues vastly complicate the answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let $|v|$, the 1-norm, denote the sum of the coordinates of the vector.&#xA;Let $v * w$ denote the coordinate-wise product of the vectors $v$ and $w$.&#xA;(So the dot-product of $v$ and $w$ is $|v * w|$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that $|a_i| = |b_i|$ always.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In case one: &#xA;$$E[|a_{i+1}|] = E[|a_{i}| + a_{i+1}] = E[|a_{i}|] + E[a_{i+1}]$$&#xA;$$= E[|a_{i+1}|] + \frac{1}{i}E[|a_{i+1}|]= \frac{i+1}{i} E[|a_i|],$$&#xA;$$E[|a_{i+1}|^2] = E[(|a_{i}| + a_{i+1})^2] = E[|a_i|^2 + 2|a_i|a_{i+1} + a_{i+1}^2]$$&#xA;$$= E[|a_i|^2] + 2E[|a_i|a_{i+1}] + E[a_{i+1}^2]$$&#xA;$$= E[|a_i|^2] + 2E[E[|a_i|a_{i+1}]:|a_i|=k] + E[a_{i+1}^2]$$&#xA;$$= E[|a_i|^2] + 2E[kE[a_{i+1}]:|a_i|=k] + E[a_{i+1}^2]$$&#xA;$$= E[|a_i|^2] + 2E[k\frac{1}{i} k:|a_i|=k] + E[a_{i+1}^2]$$&#xA;$$= E[|a_i|^2] + \frac{2}{i}E[|a_i|^2] + E[a_{i+1}^2]$$&#xA;$$= E[|a_i|^2] + \frac{2}{i}E[|a_i|^2] + E[E[a_{i+1}^2] : |a_i| = k]$$&#xA;$$= E[|a_i|^2] + \frac{2}{i}E[|a_i|^2] + E[E[\frac{k}{i}^2] : |a_i| = k]$$&#xA;$$= E[|a_i|^2] + \frac{2}{i}E[|a_i|^2] + E[\frac{1}{i^2}|a_i|^2]$$&#xA;$$= \frac{(i+1)^2}{i^2} E[|a_i|^2],$$&#xA;$$E[|a_{i+1}*b_{i+1}|] = \frac{i+1}{i} E[|a_i * b_i|].$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In case two,&#xA;$$E[|a_{i+1}|] = \frac{i+1}{i} E[|a_i|] + 1,$$&#xA;$$E[|a_{i+1}|^2] = \frac{(i+1)^2}{i^2} E[|a_i|^2] + 2E[|a_i|] + 1,$$&#xA;$$E[|a_{i+1}*b_{i+1}|]= E|a_i * b_i| + E[a_{i+1} b_{i+1}]$$&#xA;$$=E|a_i * b_i| + E[kE(a_{i+1}): b_{i+1} = k]$$&#xA;$$=E|a_i * b_i| + E[k(\frac{1}{i}|a_i| + 1): b_{i+1} = k]$$&#xA;$$=E|a_i * b_i| + E[(\frac{1}{i}k|a_i| + k): b_{i+1} = k]$$&#xA;$$=E|a_i * b_i| + \frac{1}{i} E[(\frac{1}{i}|b_i|+1)|a_i|] + E[b_{i+1}]$$&#xA;$$=E|a_i * b_i| + \frac{1}{i^2} E[|a_i||b_i|]+ \frac{1}{i} E[|a_i|] + \frac{1}{i}E[|b_i|] + 1$$&#xA;$$=E|a_i * b_i| + \frac{1}{i^2} E[|a_i|^2]+ \frac{2}{i} E[|a_i|] + 1.$$&#xA;&lt;em&gt;[The last equation would be much more complicated if we require $x \ne y$ in case 2.]&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So overall,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$E[|a_{i+1}|] = \frac{i+1}{i} E[|a_i|] + (1-p),$$&#xA;$$E[|b_{i+1}|] = \frac{i+1}{i} E[|b_i|] + (1-p),$$&#xA;$$E[|a_{i+1}|^2|]= \frac{(i+1)^2}{i^2} E[|a_i|^2] + (1-p) [E[|a_i|^2] + 2E[|a_i|] + 1]$$&#xA;$$E[|a_{i+1}*b_{i+1}|] = \frac{p+i}{i} E[|a_i * b_i|] + (1-p)[\frac{1}{i^2} E[|a_i|^2]+ \frac{1}{i} [2E[|a_i|] + 1].$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which allows us to efficiently tabulate the answer for $i = 1, 2, 3, ...$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;DISCLAIMER: I have yet to numerically verify this answer.&lt;/p&gt;&#xA;" OwnerUserId="3567" LastEditorUserId="3567" LastEditorDisplayName="" LastEditDate="2011-03-09T10:28:37.683" LastActivityDate="2011-03-09T10:28:37.683" CommunityOwnedDate="2011-03-09T10:22:58.640" />
  <row Id="8052" PostTypeId="1" AcceptedAnswerId="8053" CreationDate="2011-03-09T00:54:14.237" Score="2" ViewCount="226" Body="&lt;p&gt;Suppose I'm doing binary classification, and I want to test whether using feature X is significant or not. (For example, I could be building a decision tree, and I want to see whether I should prune feature X or not.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I believe the standard method is to use a chi-square test on the 2x2 table&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;             X = 0    X = 1&#xA;Outcome = 0      A        B&#xA;Outcome = 1      C        D&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;A &quot;simpler&quot; (IMO) test, though, would be to calculate a statistic on the probability that X gives the correct outcome: take p = [(x = 0 and Outcome = 0) + (x = 1 and Outcome = 1)] / [Total number of observations], and calculate the significance that p is far from 0.5 (say, by using a normal approximation or a Wilson score).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are the disadvantages/advantages of this approach, compared to the chi-square method? Is it totally misguided? Are they equivalent?&lt;/p&gt;&#xA;" OwnerUserId="1106" LastActivityDate="2011-03-11T20:05:12.323" Title="2x2 chi-square test vs. binomial proportion statistic" Tags="&lt;statistical-significance&gt;&lt;chi-squared&gt;" AnswerCount="3" FavoriteCount="1" />
  <row Id="8053" PostTypeId="2" ParentId="8052" CreationDate="2011-03-09T01:14:39.337" Score="2" ViewCount="" Body="&lt;p&gt;Suppose you have&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Case 1:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;A=200, B=100&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;C=100, D=200&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;versus&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Case 2:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;A=200, B=0&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;C=200, D=200&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The B=0 in case 2 means that case 2 provides much stronger evidence than case 1 of a relationship between X and Outcome; but in your test, both cases would be scored the same.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Chi-Square test, informally speaking, not only takes into account the &quot;X XOR Outcome&quot; relationship (which is what you test) but also &quot;X implies Outcome&quot;, &quot;not X implies Outcome&quot; and so on.&lt;/p&gt;&#xA;" OwnerUserId="3567" LastEditorUserId="3567" LastEditorDisplayName="" LastEditDate="2011-03-09T02:04:47.480" LastActivityDate="2011-03-09T02:04:47.480" />
  <row Id="8054" PostTypeId="2" ParentId="8005" CreationDate="2011-03-09T02:52:53.240" Score="1" ViewCount="" Body="&lt;p&gt;I'm not sure I understand your question.  But if you are working with nontraditional distributions you might want to look at nonparametric methods--in particular, you can use the nonparametric bootstrap to get confidence intervals for &quot;4yrMean - 1yrMean&quot; or whatever statistic you need.&lt;/p&gt;&#xA;" OwnerUserId="3567" LastActivityDate="2011-03-09T02:52:53.240" CommentCount="1" />
  <row Id="8055" PostTypeId="1" CreationDate="2011-03-08T18:50:07.840" Score="1" ViewCount="141" Body="&lt;p&gt;Could someone walk me through an example on how to use DLM Kalman filtering in R on a time series. Say I have a these values (quarterly values with yearly seasonality); how would you use DLM to predict the next values? And BTW, do I have enough historical data (what is the minimum)?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;89  2009Q1  &#xA;82  2009Q2  &#xA;89  2009Q3  &#xA;131 2009Q4  &#xA;97  2010Q1  &#xA;94  2010Q2  &#xA;101 2010Q3  &#xA;151 2010Q4  &#xA;100 2011Q1  &#xA;?   2011Q2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm looking for a R code cookbook-style how-to step-by-step type of answer. Accuracy of the prediction is not my main goal, I just want to learn the sequence of code that gives me a number for 2011Q2, even if I don't have enough data. Thanks!  &lt;/p&gt;&#xA;" OwnerUserId="3645" LastActivityDate="2011-03-09T05:56:56.280" Title="How to use DLM with Kalman filtering for forecasting" Tags="&lt;r&gt;&lt;time-series&gt;&lt;forecasting&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="1" />
  <row Id="8056" PostTypeId="2" ParentId="8055" CreationDate="2011-03-08T21:47:06.360" Score="5" ViewCount="" Body="&lt;p&gt;The paper at &lt;a href=&quot;http://www.jstatsoft.org/v39/i02&quot; rel=&quot;nofollow&quot;&gt;JSS 39-02&lt;/a&gt; compares 5 different Kalman filtering R packages and gives sample code.&lt;/p&gt;&#xA;" OwnerUserId="4704" LastActivityDate="2011-03-08T21:47:06.360" />
  <row Id="8057" PostTypeId="2" ParentId="8055" CreationDate="2011-03-09T05:56:56.280" Score="1" ViewCount="" Body="&lt;p&gt;I suggest you read the dlm vignette &lt;a href=&quot;http://cran.r-project.org/web/packages/dlm/vignettes/dlm.pdf&quot; rel=&quot;nofollow&quot;&gt;http://cran.r-project.org/web/packages/dlm/vignettes/dlm.pdf&lt;/a&gt; especially the chapter 3.3&lt;/p&gt;&#xA;" OwnerUserId="1709" LastActivityDate="2011-03-09T05:56:56.280" />
  <row Id="8058" PostTypeId="1" AcceptedAnswerId="8060" CreationDate="2011-03-09T06:04:31.097" Score="5" ViewCount="154" Body="&lt;p&gt;I'm building a logistic regression, and two of my variables are categorical with three levels each. (Say one variable is male, female, or unknown, and the other is single, married, or unknown.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How many dummy variables am I supposed to make? Do I make 4 in total (2 for each of the categorical variables, e.g., a male variable, a female variable, a single variable, and a married variable) or 5 in total (2 for one of the categorical variables, 3 for the other)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know most textbooks say that when you're dummy encoding a categorical variable with k levels, you should only make k-1 dummy variables, since otherwise you'll get a collinearity with the constant. But what do you do when you're dummy encoding several categorical variables? By the collinearity argument, it sounds like I'd only make k-1 dummy variables for one of the categorical variables, and for the rest of the categorical variables I'd build all k dummy variables.&lt;/p&gt;&#xA;" OwnerUserId="3632" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-03-09T07:56:21.490" LastActivityDate="2011-03-09T08:14:12.080" Title="How to choose number of dummy variables when encoding several categorical variables?" Tags="&lt;logistic-regression&gt;&lt;categorical-data&gt;" AnswerCount="1" />
  <row Id="8059" PostTypeId="1" AcceptedAnswerId="8076" CreationDate="2011-03-09T06:14:27.973" Score="4" ViewCount="108" Body="&lt;p&gt;I'm doing some work on modelling transition matrices, and for this I need a measure of discrepancy or lack of fit: that is, if I have a matrix $T$ and a target matrix $T_0$, I want to be able to calculate how far $T$ is from $T_0$. Would anyone be able to provide pointers on what measure I should be using?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've seen some references to using an elementwise squared-error measure, ie sum up the squared differences of the elements of $T$ and $T_0$, but this seems rather ad-hoc.&lt;/p&gt;&#xA;" OwnerUserId="1569" LastEditorUserId="223" LastEditorDisplayName="" LastEditDate="2011-03-09T16:06:28.627" LastActivityDate="2011-03-09T16:06:28.627" Title="Discrepancy measures for transition matrices" Tags="&lt;estimation&gt;&lt;markov-process&gt;&lt;distance-functions&gt;" AnswerCount="2" />
  <row Id="8060" PostTypeId="2" ParentId="8058" CreationDate="2011-03-09T06:43:14.660" Score="6" ViewCount="" Body="&lt;p&gt;You would make k-1 dummy variables for each of your categorical variables. The textbook argument holds; if you were to make k dummies for any of your variables, you would have a collinearity. You can think of the k-1 dummies as being contrasts between the effects of their corresponding levels, and the level whose dummy is left out.&lt;/p&gt;&#xA;" OwnerUserId="1569" LastActivityDate="2011-03-09T06:43:14.660" CommentCount="1" />
  <row Id="8063" PostTypeId="1" CreationDate="2011-03-09T09:43:14.043" Score="1" ViewCount="135" Body="&lt;p&gt;How can I measure linear correlation of non-normally distributed variables?&#xA;Pearson coefficient is not valid for non-normally distributed data, and Spearman's rho does not capture linear correlation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you&lt;/p&gt;&#xA;" OwnerUserId="3465" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-03-09T09:59:04.997" LastActivityDate="2011-03-10T06:07:40.783" Title="Measuring linear correlation of non-normally distributed variables" Tags="&lt;correlation&gt;&lt;measurement&gt;" AnswerCount="3" CommentCount="2" FavoriteCount="3" />
  <row Id="8064" PostTypeId="2" ParentId="8063" CreationDate="2011-03-09T09:49:24.237" Score="2" ViewCount="" Body="&lt;p&gt;Why do you require normality for computing a correlation? How about a simple scatterplot?&#xA;As long as the data are continuous, ordinary (Pearson) correlation should be fine. All that it is measuring is the strength of the linear relationship between two variables (if indeed there is such a relationship).&lt;/p&gt;&#xA;" OwnerUserId="1945" LastActivityDate="2011-03-09T09:49:24.237" />
  <row Id="8065" PostTypeId="1" AcceptedAnswerId="8069" CreationDate="2011-03-09T10:11:34.210" Score="0" ViewCount="80" Body="&lt;p&gt;I have the following data of 2 diseases from 5 areas. I want to see if there is any relationship between the 2 diseases.&#xA;Incidence Rates of 2 diseases (cases per million per year)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    Areas   Disease 1       Disease 2&#xA;    1       4.653           0.751&#xA;    2       6.910           1.121&#xA;    3       4.957           0.745&#xA;    4       2.870           0.848&#xA;    5       2.819           1.166&#xA;&#xA;Actual number of cases are&#xA;Areas   Disease 1       Disease 2&#xA;1       1152            186&#xA;2       2601            422&#xA;3       1051            158&#xA;4        403            119&#xA;5        290            120&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am a beginner in Biostatistics. Kindly advise me in simple terms&lt;/p&gt;&#xA;" OwnerUserId="2956" LastEditorUserId="2956" LastEditorDisplayName="" LastEditDate="2011-03-09T12:36:20.820" LastActivityDate="2011-03-09T12:36:20.820" Title="Establishing relationship between 2 diseases" Tags="&lt;correlation&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="8066" PostTypeId="2" ParentId="8063" CreationDate="2011-03-09T10:11:52.887" Score="0" ViewCount="" Body="&lt;p&gt;What @galit said is absolutely right. You can find the linear correlation between any two continuously distributed variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But perhaps you are thinking of the &lt;em&gt;meaning&lt;/em&gt; of such a correlation? Indeed, &lt;a href=&quot;http://en.wikipedia.org/wiki/Anscombe%27s_quartet&quot; rel=&quot;nofollow&quot;&gt;Anscombe's quartet&lt;/a&gt; shows that while the correlation is &lt;em&gt;defined&lt;/em&gt; for any pair of continuous variables, and its mathematical and statistical meaning is the same, its substantive meaningfulness may vary.&lt;/p&gt;&#xA;" OwnerUserId="686" LastActivityDate="2011-03-09T10:11:52.887" />
  <row Id="8067" PostTypeId="2" ParentId="8044" CreationDate="2011-03-09T10:18:53.657" Score="5" ViewCount="" Body="&lt;p&gt;A couple things to add to what @matt said:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In addition to SUGI (which is now renamed SAS Global Forum, and will be held this year in Las Vegas) there are numerous local and regional SAS user groups.  These are smaller, more intimate, and (usually) a lot cheaper.  Some local groups are even free. See &lt;a href=&quot;http://support.sas.com/usergroups/&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;SAS-L.  This is a mailing list for SAS questions.  It is quite friendly, and some of the participants are among the best SAS programmers there are.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The book &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1420070576&quot; rel=&quot;nofollow&quot;&gt;SAS and R: Data Management, Statistical Analysis and Graphics&lt;/a&gt; by Kleinman and Horton.  Look up what you want to do in the R index, and you'll find how to do it in SAS as well.  Sort of like a inter-language dictionary.&lt;/p&gt;&#xA;" OwnerUserId="686" LastActivityDate="2011-03-09T10:18:53.657" CommentCount="1" />
  <row Id="8068" PostTypeId="2" ParentId="8059" CreationDate="2011-03-09T10:34:57.833" Score="4" ViewCount="" Body="&lt;p&gt;Why does one want the measure of discrepancy to be a true metric? There is a huge literature on axiomatic characterizations of I-divergence as measure of distance. It is neither symmetric nor satisfies triangle inequality.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope by 'transition matrix' you mean 'probability transition matrix'. Never mind, as long as the entries are non negative, I-divergence is considered to be the &quot;best&quot; measure of discrimination. See for example &lt;a href=&quot;http://www.mdpi.com/1099-4300/10/3/261/&quot; rel=&quot;nofollow&quot;&gt;http://www.mdpi.com/1099-4300/10/3/261/&lt;/a&gt;. In fact certain axioms which any one would feel desirable lead to measures which are nonsymmetric in general.&lt;/p&gt;&#xA;" OwnerUserId="3485" LastActivityDate="2011-03-09T10:34:57.833" CommentCount="8" />
  <row Id="8069" PostTypeId="2" ParentId="8065" CreationDate="2011-03-09T10:51:36.343" Score="2" ViewCount="" Body="&lt;p&gt;There is no simple answer for how to &quot;establish a relationship between two variables;&quot; indeed, your question is one of the central issues in statistics and research is still going on on how to do this.  But some basics:  first you will want to plot your data, and then you will want to carry out a linear regression to test some specific type of relationship between variables in your data.  You will need to obtain the &quot;p-score&quot; of the regression to get an idea of how well your purported relationship is supported by the data.  Generally if you can get a very low p-score (e.g. p &amp;lt; 0.01), then it will be safe to say that there is a relationship between variables.&lt;/p&gt;&#xA;" OwnerUserId="3567" LastEditorUserId="3567" LastEditorDisplayName="" LastEditDate="2011-03-09T10:56:46.197" LastActivityDate="2011-03-09T10:56:46.197" CommentCount="5" />
  <row Id="8070" PostTypeId="2" ParentId="7994" CreationDate="2011-03-09T11:16:18.363" Score="1" ViewCount="" Body="&lt;p&gt;See the papers by Sebastian Mika and co-authors (I think this was the subject of Mika's PhD thesis).  The original paper is &lt;a href=&quot;http://dx.doi.org/10.1109/NNSP.1999.788121&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; and for free &lt;a href=&quot;http://ml.cs.tu-berlin.de/publications/MikRaeWesSchMue99.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2011-03-09T11:16:18.363" />
  <row Id="8071" PostTypeId="1" CreationDate="2011-03-09T11:28:52.587" Score="5" ViewCount="814" Body="&lt;p&gt;How do I know when to choose between Spearman's $\rho$ and Pearson's $r$? My variable includes satisfaction and the scores were interpreted using the sum of the scores. However, these scores could also be ranked. &lt;/p&gt;&#xA;" OwnerUserId="3636" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-03-09T11:35:38.550" LastActivityDate="2011-03-09T13:16:03.530" Title="How to choose between Pearson and Spearman correlation?" Tags="&lt;correlation&gt;" AnswerCount="3" CommentCount="1" FavoriteCount="1" />
  <row Id="8072" PostTypeId="1" AcceptedAnswerId="8102" CreationDate="2011-03-09T11:34:14.167" Score="3" ViewCount="150" Body="&lt;p&gt;If I want to achieve a margin-of-error of &amp;lt;= 5 % for a representative population sample, how large a sample do I need when: The interviewees are picked from &lt;em&gt;X&lt;/em&gt; regions and from &lt;em&gt;Y&lt;/em&gt; age groups? That is, how many samples from each age group &amp;amp; region?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know how many people live in each region and how they are distributed in the &lt;em&gt;Y&lt;/em&gt; age groups in each region.&lt;/p&gt;&#xA;" OwnerUserId="3401" LastEditorUserId="3401" LastEditorDisplayName="" LastEditDate="2011-03-09T11:52:29.903" LastActivityDate="2011-03-09T22:35:21.137" Title="Choosing sample size to achieve pre-specified margin-of-error " Tags="&lt;statistical-significance&gt;&lt;sampling&gt;" AnswerCount="2" CommentCount="3" FavoriteCount="1" />
  <row Id="8073" PostTypeId="2" ParentId="8071" CreationDate="2011-03-09T11:34:52.310" Score="6" ViewCount="" Body="&lt;p&gt;This happens often in statistics: there are a variety of methods which could be applied in your situation, and you don't know which one to choose.  You should base your decision the pros and cons of the methods under consideration and the specifics of your problem, but even then the decision is usually subjective with no agreed-upon &quot;correct&quot; answer.  Usually it is a good idea to try out as many methods as seem reasonable and that your patience will allow and see which ones give you the best results in the end.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The difference between the Pearson correlation and the Spearman correlation is that the Pearson is most appropriate for measurements taken from an &lt;em&gt;interval&lt;/em&gt; scale, while the Spearman is more appropriate for measurements taken from &lt;em&gt;ordinal&lt;/em&gt; scales.  Examples of interval scales include &quot;temperature in Farenheit&quot; and &quot;length in inches&quot;, in which the individual units (1 deg F, 1 in) are meaningful.  Things like &quot;satisfaction scores&quot; tend to of the ordinal type since while it is clear that &quot;5 happiness&quot; is happier than &quot;3 happiness&quot;, it is not clear whether you could give a meaningful interpretation of &quot;1 unit of happiness&quot;.  But when you &lt;em&gt;add up&lt;/em&gt; many measurements of the ordinal type, which is what you have in your case, you end up with a measurement which is really neither ordinal nor interval, and is difficult to interpret.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would recommend that you convert your satisfaction scores to &lt;em&gt;quantile&lt;/em&gt; scores and then work with the sums of those, as this will give you data which is a little more amenable to interpretation.  But even in this case it is not clear whether Pearson or Spearman would be more appropriate.&lt;/p&gt;&#xA;" OwnerUserId="3567" LastEditorUserId="3567" LastEditorDisplayName="" LastEditDate="2011-03-09T11:53:50.617" LastActivityDate="2011-03-09T11:53:50.617" CommentCount="1" />
  <row Id="8074" PostTypeId="2" ParentId="8071" CreationDate="2011-03-09T11:54:45.967" Score="2" ViewCount="" Body="&lt;p&gt;While agreeing with charles answer, i would suggest (on a strictly practical level) that you compute both of the coefficients and look at the differences. In many cases, they will be exactly the same, so you don't need to worry.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If however, they are different then you need to look at whether or not you met the assumptions of Pearsons (constant variance and linearity) and if these are not met, you are probably better off using Spearmans.&lt;/p&gt;&#xA;" OwnerUserId="656" LastActivityDate="2011-03-09T11:54:45.967" CommentCount="5" />
  <row Id="8075" PostTypeId="2" ParentId="8072" CreationDate="2011-03-09T12:13:57.863" Score="3" ViewCount="" Body="&lt;p&gt;If you weight your measurements (proportion of subpopulation/proportion of subpopulation in sample), your estimates will be &lt;em&gt;unbiased&lt;/em&gt;.  I assume this is what you meant by &quot;poll results being skewed&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I interpret your question correctly, your goal is the simultaneous estimation of multiple population proportions, where your proportions are &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;P_1 = proportion of population voting yes on poll question 1&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;P_2 = proportion of population voting yes on poll question 2&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;etc.  (Let's work with one region at a time for now.) These can be represented in a proportion vector $P= (P_1, P_2, ...)$.  We will denote a &lt;em&gt;point estimate&lt;/em&gt; of $P$ by $\hat{P}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By what you want is probably not a point estimate, but a &lt;em&gt;95% confidence interval&lt;/em&gt;.  This is an interval $(P_1 \pm t, P_2 \pm t, ...)$ where $t$ is your tolerance.  (What 95% confidence means is a tricky issue which is hard to explain and easy to misunderstand, so I'll skip it for now.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The thing is, it is &lt;em&gt;always&lt;/em&gt; possible to construct a 95% confidence set no matter how small your sample size is.  For your problem to be properly defined you need to specify the $t$, which is how accurate you require your estimates to be.  The more accuracy you require the more samples you will need.  In the problem as I have set it up it is possible to find the minimum number of samples given $t$, but there you can get better results if you can estimate the variability of your respective subpopulations ahead of time (which does not seem to be the case in your problem.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please give further clarification to your problem, though.&lt;/p&gt;&#xA;" OwnerUserId="3567" LastEditorUserId="3567" LastEditorDisplayName="" LastEditDate="2011-03-09T12:28:18.687" LastActivityDate="2011-03-09T12:28:18.687" CommentCount="2" />
  <row Id="8076" PostTypeId="2" ParentId="8059" CreationDate="2011-03-09T13:09:16.753" Score="4" ViewCount="" Body="&lt;p&gt;As long as your matrix represent conditional probability I think that using a general matrix norm is a bit artificial. Using some sort of geodesic distance on the set of transition matrix might be more relevant but I clearly prefer to come back to probabilities. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I assume you want to compare $Q=(Q_{ij})$ and $P=(P_{ij})$ with $P_{ij}=P(X^P_{t}=j|X^P_{t-1}=j)$ and that for $P$ (resp. $Q$) there exists a unique stationnary measure $\pi_{P}$ (resp. $\pi_{Q}$).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Under these assumptions, I guess it is meaningfull to compare $\pi_{P}$ and $\pi_{Q}$ for example with a $L_{1}$ distance: $\sum_{j}|\pi_{P}[j]-\pi_{Q}[j]|$ or hellinger distance: $\sum_{j}|\pi^{1/2}_{P}[j]-\pi^{1/2}_{Q}[j]|^2$ or Kullback divergence: $\sum_{j}\pi_{P}[j] \log(\frac{\pi_{P}[j]}{\pi_{Q}[j]})$. &lt;/p&gt;&#xA;" OwnerUserId="223" LastActivityDate="2011-03-09T13:09:16.753" CommentCount="1" />
  <row Id="8077" PostTypeId="2" ParentId="8071" CreationDate="2011-03-09T13:16:03.530" Score="7" ViewCount="" Body="&lt;p&gt;Shortest and mostly correct answer is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Pearson benchmarks &lt;em&gt;linear&lt;/em&gt; relationship, Spearman benchmarks &lt;em&gt;monotonic&lt;/em&gt; relationship (few infinities more general case, but for some power tradeoff).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So if you assume/think that the relation is linear (or, as a special case, that those are a two measures of the same thing, so the relation is $y=1\cdot x+0$) and the situation is not too weired (check other answers for details), go with Pearson. Otherwise use Spearman. &lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2011-03-09T13:16:03.530" />
  <row Id="8078" PostTypeId="2" ParentId="8000" CreationDate="2011-03-09T13:57:54.917" Score="1" ViewCount="" Body="&lt;p&gt;You may also consider simply using a number of transforms of time series for the input data.  Just for one example, the inputs could be:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;the most recent interval value&#xA;(7) &lt;/li&gt;&#xA;&lt;li&gt;the next most recent interval&#xA;    value (6)&lt;/li&gt;&#xA;&lt;li&gt;the delta between most&#xA;    recent and next most recent (7-6=1)&lt;/li&gt;&#xA;&lt;li&gt;the third most recent interval&#xA;    value (5)&lt;/li&gt;&#xA;&lt;li&gt;the delta between the&#xA;    second and third most recent (6-5=1)&lt;/li&gt;&#xA;&lt;li&gt;the average of the last three&#xA;    intervals ((7+6+5)/3=6)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;So, if your inputs to a conventional neural network were these six pieces of transformed data, it would not be a difficult task for an ordinary backpropagation algorithm to learn the pattern.  You would have to code for the transforms that take the raw data and turn it into the above 6 inputs to your neural network, however.&lt;/p&gt;&#xA;" OwnerUserId="2917" LastActivityDate="2011-03-09T13:57:54.917" CommentCount="4" />
  <row Id="8079" PostTypeId="1" CreationDate="2011-03-09T14:15:07.337" Score="0" ViewCount="54" Body="&lt;p&gt;Hi.&#xA;Let say we have only one sample in which we are interested only on one statistics c of this sample. This is usually true in practice when we usually have only one data set. So, most probably the usual estimating method like MLE cant be applied.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And let say we want to estimate 2 parameters (i.e. a &amp;amp; b) of a certain dist which depend on c. Is bootstrap technique will work for this? Is there any theoretical proof for such practice?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help will be appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;tsukiko&lt;/p&gt;&#xA;" OwnerUserId="3638" LastActivityDate="2011-03-09T14:15:07.337" Title="Bootstrap for estimating parameters with only one sample" Tags="&lt;estimation&gt;&lt;bootstrap&gt;" AnswerCount="0" CommentCount="0" FavoriteCount="0" />
  <row Id="8080" PostTypeId="2" ParentId="8044" CreationDate="2011-03-09T14:29:35.183" Score="3" ViewCount="" Body="&lt;p&gt;In addition to Matt Parkers excellent advice (particularly about reading colleagues code), the actual SAS documentation can be surprisingly helpful (once you've figured out the name of what you want):&#xA;&lt;a href=&quot;http://support.sas.com/documentation/&quot; rel=&quot;nofollow&quot;&gt;http://support.sas.com/documentation/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And the Global Forum/SUGI proceedings are available here:&#xA;&lt;a href=&quot;http://support.sas.com/events/sasglobalforum/previous/online.html&quot; rel=&quot;nofollow&quot;&gt;http://support.sas.com/events/sasglobalforum/previous/online.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="495" LastActivityDate="2011-03-09T14:29:35.183" CommentCount="1" />
  <row Id="8081" PostTypeId="1" CreationDate="2011-03-09T15:06:03.670" Score="2" ViewCount="65" Body="&lt;p&gt;Is it true that horizontal and vertical bar charts are in different length for human eye?&lt;/p&gt;&#xA;" OwnerUserId="290" LastActivityDate="2011-03-09T15:06:03.670" ClosedDate="2011-03-11T04:09:55.230" Title="Is it true that horizontal and vertical bar charts are in different length for human eye?" Tags="&lt;histogram&gt;&lt;barplot&gt;" CommentCount="1" FavoriteCount="2" />
  <row Id="8082" PostTypeId="1" AcceptedAnswerId="8084" CreationDate="2011-03-09T15:10:25.447" Score="2" ViewCount="125" Body="&lt;p&gt;I am running &lt;code&gt;ivprobit&lt;/code&gt; in Stata to look at the determinants of enrolment in health insurance (cbhi). I have several exogenous regressors and one endogenous regressor (consumption). I am using wealthindex as an intrumental variable for consumption. However, when I run the &lt;code&gt;ivprobit&lt;/code&gt; model all my exogenous regressors appear in the &quot;instruments&quot; list. Could someone please tell me how to prevent this from happening? I am copying the stata code and results below. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you in advance for your help.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;. # delimit;&#xA;delimiter now ;&#xA;. ivprobit cbhi age_hhead age2 edu_hh2 edu_hh3 edu_hh4 edu_hh5 mar_hh hhmem_cat2 hhmem_cat3&#xA;&amp;gt; hhhealth_poor hh_chronic hh_diffic risk pregnancy wra any_oldmem j_hh2 j_hh3 j_hh4&#xA;&amp;gt; hlthseek_mod campaign qual_percep urban door_to_door chief_mem&#xA;&amp;gt; pharm_in_vil drugsel_in_vil hlthcent_in_vil privclin_in_vil young edu_chief num_camp_vill home_collection&#xA;&amp;gt; qual2 qual3 size_ln name_hosp1 name_hosp3 name_hosp4 name_hosp5 name_hosp6 (cons_pcm=wealthindex) ;&#xA;&#xA;Fitting exogenous probit model&#xA;&#xA;Iteration 0:   log likelihood = -1909.5425&#xA;Iteration 1:   log likelihood =  -1687.946&#xA;Iteration 2:   log likelihood = -1681.6413&#xA;Iteration 3:   log likelihood = -1681.6142&#xA;Iteration 4:   log likelihood = -1681.6142&#xA;&#xA;Fitting full model&#xA;&#xA;Iteration 0:   log likelihood = -42769.776 &#xA;Iteration 1:   log likelihood = -42766.676 &#xA;Iteration 2:   log likelihood = -42736.028 &#xA;Iteration 3:   log likelihood = -42734.629 &#xA;Iteration 4:   log likelihood = -42734.269 &#xA;Iteration 5:   log likelihood = -42734.266 &#xA;Iteration 6:   log likelihood = -42734.266 &#xA;&#xA;Probit model with endogenous regressors           Number of obs   =       3000&#xA;                                                  Wald chi2(42)   =    1113.82&#xA;Log likelihood = -42734.266                       Prob &amp;gt; chi2     =     0.0000&#xA;&#xA;------------------------------------------------------------------------------&#xA;             |      Coef.   Std. Err.      z    P&amp;gt;|z|     [95% Conf. Interval]&#xA;-------------+----------------------------------------------------------------&#xA;    cons_pcm |   3.59e-06   2.15e-07    16.68   0.000     3.17e-06    4.01e-06&#xA;   age_hhead |   .0320042   .0129973     2.46   0.014       .00653    .0574784&#xA;        age2 |  -.0001774   .0001228    -1.45   0.148    -.0004181    .0000632&#xA;     edu_hh2 |    .051426   .0879334     0.58   0.559    -.1209203    .2237723&#xA;     edu_hh3 |  -.0075595   .1023748    -0.07   0.941    -.2082106    .1930915&#xA;     edu_hh4 |   -.000683   .1285689    -0.01   0.996    -.2526735    .2513075&#xA;     edu_hh5 |  -.2986249   .1713749    -1.74   0.081    -.6345136    .0372638&#xA;      mar_hh |  -.0539035    .067056    -0.80   0.421    -.1853308    .0775237&#xA;  hhmem_cat2 |   .4222092   .0675468     6.25   0.000     .2898198    .5545985&#xA;  hhmem_cat3 |   .7099401   .0822271     8.63   0.000     .5487778    .8711023&#xA;hhhealth_p~r |   .1550617   .0699122     2.22   0.027     .0180362    .2920872&#xA;  hh_chronic |    .159681   .0718297     2.22   0.026     .0188973    .3004647&#xA;   hh_diffic |   .2613713   .0741209     3.53   0.000      .116097    .4066456&#xA;        risk |  -.0482252   .0467953    -1.03   0.303    -.1399423    .0434919&#xA;   pregnancy |   .1200344   .1214774     0.99   0.323    -.1180569    .3581257&#xA;         wra |   .1033809   .0289192     3.57   0.000     .0467004    .1600614&#xA;  any_oldmem |   .0792428   .0680778     1.16   0.244    -.0541872    .2126729&#xA;       j_hh2 |   .0512404   .0815692     0.63   0.530    -.1086323     .211113&#xA;       j_hh3 |   -.349126   .0806268    -4.33   0.000    -.5071516   -.1911004&#xA;       j_hh4 |   .1928496   .0843208     2.29   0.022     .0275838    .3581154&#xA;hlthseek_mod |  -.0359996   .0969011    -0.37   0.710    -.2259223     .153923&#xA;    campaign |   .5068671   .0781293     6.49   0.000     .3537365    .6599977&#xA; qual_percep |   .1840314   .0514758     3.58   0.000     .0831408    .2849221&#xA;       urban |  -.0999875   .0590944    -1.69   0.091    -.2158103    .0158353&#xA;door_to_door |   .0564977   .0719369     0.79   0.432     -.084496    .1974914&#xA;   chief_mem |   .0811248   .0599964     1.35   0.176     -.036466    .1987156&#xA;pharm_in_vil |   .0269211   .0582651     0.46   0.644    -.0872763    .1411185&#xA;drugsel_in~l |   .0693485   .0586169     1.18   0.237    -.0455385    .1842355&#xA;hlthcent_i~l |   .0089279   .0823509     0.11   0.914    -.1524769    .1703327&#xA;privclin_i~l |  -.0547559   .0857815    -0.64   0.523    -.2228846    .1133728&#xA;       young |   .0899074   .0715818     1.26   0.209    -.0503903    .2302051&#xA;   edu_chief |  -.0888131   .0528856    -1.68   0.093    -.1924671    .0148408&#xA;num_camp_v~l |   .0655667   .0548026     1.20   0.232    -.0418444    .1729779&#xA;home_colle~n |   .1944581   .0606159     3.21   0.001     .0756531    .3132631&#xA;       qual2 |  -.1778623   .0896638    -1.98   0.047    -.3536001   -.0021245&#xA;       qual3 |  -.2049839   .1081036    -1.90   0.058     -.416863    .0068952&#xA;     size_ln |  -.0059746    .042326    -0.14   0.888    -.0889321    .0769829&#xA;  name_hosp1 |   .1476783   .1168295     1.26   0.206    -.0813032    .3766598&#xA;  name_hosp3 |    .536909   .1350044     3.98   0.000     .2723053    .8015128&#xA;  name_hosp4 |   .4347936   .1069197     4.07   0.000     .2252348    .6443523&#xA;  name_hosp5 |    .228802   .1274495     1.80   0.073    -.0209945    .4785985&#xA;  name_hosp6 |   .9189969   .1301651     7.06   0.000     .6638779    1.174116&#xA;       _cons |   -4.82205   .4438201   -10.86   0.000    -5.691922   -3.952179&#xA;-------------+----------------------------------------------------------------&#xA;     /athrho |  -.8556093   .0998439    -8.57   0.000      -1.0513   -.6599188&#xA;    /lnsigma |   12.27712     .01291   950.98   0.000     12.25181    12.30242&#xA;-------------+----------------------------------------------------------------&#xA;         rho |  -.6939885   .0517571                     -.7823111   -.5783094&#xA;       sigma |   214725.5   2772.097                      209360.5    220228.1&#xA;------------------------------------------------------------------------------&#xA;Instrumented:  cons_pcm&#xA;Instruments:   age_hhead age2 edu_hh2 edu_hh3 edu_hh4 edu_hh5 mar_hh&#xA;               hhmem_cat2 hhmem_cat3 hhhealth_poor hh_chronic hh_diffic risk&#xA;               pregnancy wra any_oldmem j_hh2 j_hh3 j_hh4 hlthseek_mod&#xA;               campaign qual_percep urban door_to_door chief_mem&#xA;               pharm_in_vil drugsel_in_vil hlthcent_in_vil privclin_in_vil&#xA;               young edu_chief num_camp_vill home_collection qual2 qual3&#xA;               size_ln name_hosp1 name_hosp3 name_hosp4 name_hosp5&#xA;               name_hosp6 wealthindex&#xA;------------------------------------------------------------------------------&#xA;Wald test of exogeneity (/athrho = 0): chi2(1) =    73.44 Prob &amp;gt; chi2 = 0.0000&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="834" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-03-09T16:32:43.060" LastActivityDate="2011-03-09T16:32:43.060" Title="Why is Stata automatically converting regressors to instrumental variables in ivprobit model?" Tags="&lt;stata&gt;&lt;instrumental-variables&gt;" AnswerCount="1" />
  <row Id="8083" PostTypeId="1" CreationDate="2011-03-09T15:12:20.300" Score="0" ViewCount="159" Body="&lt;p&gt;&lt;em&gt;Revised Version&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Situation:&#xA;We have a sample of size $r$. Then, we count how many out of these $r$ to be &quot;success&quot;.&#xA;Let $X=$ the no. of the &quot;successes&quot; and say $X$ follow Poisson dist.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Goal: To test whether $X$  is really Poisson using Dispersion Test with test statistics given by&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$D=\frac{\sum_{i=1}^{n}(X_{i}-\overline{X})^2}{\overline{X}}$ --- As state in &lt;a href=&quot;http://www.stats.uwo.ca/faculty/aim/2004/04-259/notes/DispersionTests.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Problem: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;I have only one sample, so my n=1. So, D=0 if i used $\overline{X}=X_1$. To overcome this, i use bootstrap to generate &quot;n&quot; bootstrap samples and obtain $\overline{X}$. Is this valid?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Should i choose the generated bootstrap samples used for hypothesis testing in such a way the resulting $\overline{X}=Var(X)$? -- I think this rule should be used if i want to estimate the Poisson parameter. I'm confused in using bootstrap for estimation and bootstrap for hypothesis testing.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;3.Illustration: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;a. The data:  0.42172863  0.28830514  0.66452743  0.01578868  0.02810549. So,$ r=5$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;b.Define “success”: Observation which larger than 0.4. Here, $X=2$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;c.Obtain let say 3 bootstrap samples (from data in Step a):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;i) 0.28830514 0.01578868 0.66452743 0.02810549 0.02810549. So, $X=1$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ii) 0.42172863 0.66452743 0.02810549 0.66452743 0.66452743. So, $X=4$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;iii) 0.02810549 0.66452743 0.01578868 0.66452743 0.42172863. So, $X=3$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;d.So, $\overline{X}= 2.666667$ and $Var(X)= 2.333333$. Note that here, I didn’t control on how the 3 bootstrap samples (in Step c) are generated. That’s why $\overline{X}&amp;gt;Var(X)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, my questions:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1.Is bootstrap a valid technique to obtain $\overline{X}$ in this case?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2.Should I control on how the bootstrap samples generated so that the resulting $\overline{X}=Var(X)$ to be used for the hypothesis testing?&lt;/p&gt;&#xA;" OwnerUserId="3638" LastEditorUserId="3638" LastEditorDisplayName="" LastEditDate="2011-03-10T03:59:32.407" LastActivityDate="2011-03-10T13:31:19.273" Title="One-sample-dispersion-test for Poisson parameter" Tags="&lt;hypothesis-testing&gt;&lt;poisson&gt;&lt;bootstrap&gt;" AnswerCount="2" CommentCount="5" FavoriteCount="1" />
  <row Id="8084" PostTypeId="2" ParentId="8082" CreationDate="2011-03-09T15:27:12.967" Score="5" ViewCount="" Body="&lt;p&gt;In general all exogenous variables are always included as instruments. Usually instruments are picked for variables which are endogenous, but we can think (it follows from the mathematical derivation of instrumental variable estimation) that we need to choose the instruments for all the variables. Instruments for exogenous variables then are naturally themselves.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-03-09T15:27:12.967" />
  <row Id="8086" PostTypeId="2" ParentId="8083" CreationDate="2011-03-09T16:40:43.410" Score="3" ViewCount="" Body="&lt;p&gt;First of all, I don't think you can estimate (over)dispersion from a sample of size 1.  Overdispersion means that the sample variance is more than predicted based on other properties of the sample (mean) and properties of the assumed model (for Poisson models, variance = mean).  But for a sample of size one, the population variance estimate is not finite (it involves dividing by $n-1 = 0$).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In any case, your ideas about the bootstrap procedure are not headed in the right direction.  When you bootstrap, you are sampling data values &lt;em&gt;from the sample&lt;/em&gt;, which in this case won't help.  (i.e., say that $X_1 = 1$.  Then the possible bootstrap samples are $[1,1], [1,1,1] ...$)  Certainly, drawing extra data from a process that constrains the mean to be equal to the variance will invalidate your estimates of dispersion, as that is exactly the hypothesis under test.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, I'm not familiar with the test formula you are using.  Perhaps you are trying for the &lt;a href=&quot;https://secure.wikimedia.org/wikipedia/en/wiki/Variance-to-mean_ratio&quot; rel=&quot;nofollow&quot;&gt;variance to mean ratio&lt;/a&gt; aka Index of dispersion, in which case the formula you give for $D$ is not correct?&lt;/p&gt;&#xA;" OwnerUserId="2975" LastActivityDate="2011-03-09T16:40:43.410" CommentCount="3" />
  <row Id="8087" PostTypeId="1" CreationDate="2011-03-09T16:46:25.777" Score="3" ViewCount="54" Body="&lt;p&gt;I am trying to study hidden conditional random fields but I still have some fundamental questions about those methods. I would be immensely grateful if someone could provide some clarification over the notation used on most papers about the topic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In several papers the most common form of the the HCRF model is given as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$p(w|o;\theta) = \frac{1}{z(o; \theta)} \sum_{s} \exp{ \Psi(w, s, o; \theta) } $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In which $\theta$ is the parameter vector, $w$ is the class label, $o$ is the observation sequence, $s$ is the hidden state sequence and $\Psi$ is the potential function. However, I still could't figure out what $s$ means. Is it just a sequence of integer numbers, or is it actually a sequence of nodes in a graph? How one actually computes this summation?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://people.csail.mit.edu/sybor/cvpr06_wang.pdf&quot; rel=&quot;nofollow&quot;&gt;Most papers&lt;/a&gt; I &lt;a href=&quot;http://www.stanford.edu/~jurafsky/asru09.pdf&quot; rel=&quot;nofollow&quot;&gt;have read&lt;/a&gt; mention only that each $s_i \in S$ captures certain underlying structure of each class ($S$ being the set of hidden states in the model). But I still couldn't figure out what this actually means.&lt;/p&gt;&#xA;" OwnerUserId="1538" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-09T16:53:24.623" LastActivityDate="2011-03-09T16:53:24.623" Title="Hidden states in hidden conditional random fields" Tags="&lt;machine-learning&gt;&lt;classification&gt;&lt;image-processing&gt;&lt;hcrf&gt;" CommentCount="1" FavoriteCount="1" />
  <row Id="8088" PostTypeId="1" CreationDate="2011-03-09T17:14:51.320" Score="4" ViewCount="136" Body="&lt;p&gt;Hi folks, I'm working on a project where we observe behaviour on a task (eg. response time) and model this behaviour as a function of several experimentally manipulated variables as well as several observed variable (participant sex, participant IQ, responses on a follow-up questionnaire). I don't have concerns about multicollinearity amongst the experimental variables because they were specifically manipulated to be independent, but I am concerned about the observed variables. However, I'm unsure how to assess independence amongst the observed variables, partially because I seem to get somewhat different results depending on how I set up the assessent, and also because I'm not very familiar with correlation in the context where one or both variables are dichotomous.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, here are two different approaches to determining if sex is independent of IQ. I'm not a fan of null hypothesis significance testing, so in both approaches I build two models, one with a relationship and one without, then compute and AIC-corrected log likelihood ratio:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;m1 = lm(IQ ~ 1)&#xA;m2 = lm(IQ ~ sex)&#xA;LLR1 = AIC(m1)-AIC(m2)&#xA;&#xA;m3 = glm(sex~1,family='binomial')&#xA;m4 = glm(sex~IQ,family='binomial')&#xA;LLR2 = AIC(m3)-AIC(m4)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, these approaches yield somewhat different answers; LLR1 is about 7, suggesting strong evidence in favor of a relationship, while LLR2 is about 0.3, suggesting very weak evidence in favor of a relationship.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furthermore, if I attempt to assess the independence between sex and another dichotomous observed variable, &quot;yn&quot;, the resultant LLR similarly depends on whether I set up the models to predict sex from yn, or to predict yn from sex.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestions on why these differences are arising and how to most reasonably proceed?&lt;/p&gt;&#xA;" OwnerUserId="364" LastEditorUserId="364" LastEditorDisplayName="" LastEditDate="2011-03-09T18:54:53.200" LastActivityDate="2011-03-11T17:47:46.867" Title="Assessing multicollinearity of dichotomous predictor variables" Tags="&lt;regression&gt;&lt;binomial&gt;&lt;independence&gt;&lt;multicollinearity&gt;&lt;likelihood-ratio&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="1" />
  <row Id="8089" PostTypeId="1" AcceptedAnswerId="8129" CreationDate="2011-03-09T17:28:58.713" Score="3" ViewCount="61" Body="&lt;p&gt;Why doesn't recall take into account true negatives? In experiments where true negatives are just as important as true positives, is their a comparable metric that does take it into account?&lt;/p&gt;&#xA;" OwnerUserId="3604" LastActivityDate="2011-03-10T14:51:01.483" Title="Why doesn't recall take into account true negatives?" Tags="&lt;precision-recall&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="8090" PostTypeId="1" AcceptedAnswerId="8092" CreationDate="2011-03-09T17:49:53.070" Score="7" ViewCount="268" Body="&lt;p&gt;I have a dataset that's nominally 16-dimensional.  I have about 100 samples in one case and about 20,000 in another.  Based on various exploratory analyses I've conducted using PCA and heat maps, I'm convinced that the true dimensionality (i.e. the number of dimensions needed to capture most of the &quot;signal&quot;) is around 4.  I want to create a slide to that effect for a presentation.  The &quot;conventional wisdom&quot; about this data, which I'm looking to disprove, is that the true dimensionality is one or two.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What's a good, simple visualization for showing the true dimensionality of a dataset?  Preferably it should be understandable to people who have some background in statistics but are not &quot;real&quot; statisticians.&lt;/p&gt;&#xA;" OwnerUserId="1347" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-09T17:57:03.650" LastActivityDate="2011-03-10T04:47:44.177" Title="How to visualize the true dimensionality of the data?" Tags="&lt;data-visualization&gt;&lt;pca&gt;&lt;dimensionality-reduction&gt;" AnswerCount="3" CommentCount="5" FavoriteCount="3" />
  <row Id="8091" PostTypeId="2" ParentId="8090" CreationDate="2011-03-09T18:03:12.767" Score="4" ViewCount="" Body="&lt;p&gt;One way to visualize this would be as follows: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Perform a PCA on the data.&lt;/li&gt;&#xA;&lt;li&gt;Let $V$ be the vector space spanned by the first two principal component vectors, and let $V^\top$ be the complement.&lt;/li&gt;&#xA;&lt;li&gt;Decompose each vector $x_i$ in your data set as the sum of an element in $V$ plus a remainder term (which is in $V^\top$). Write this as $x_i = v_i + c_i$. (this should be easy using the results of the PCA.)&lt;/li&gt;&#xA;&lt;li&gt;Create a scatter plot of $||c_i||$ versus $||v_i||$. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;If the data is truly $\le 2$ dimensional, the plot should look like a flat line.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In Matlab (ducking from all the shoes being thrown): &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;lat_d = 2;   %the latent dimension of the generating process&#xA;vis_d = 16;  %manifest dimension&#xA;n = 10000;   %number of samples&#xA;x = randn(n,lat_d) * randn(lat_d,vis_d) + 0.1 * randn(n,vis_d); %add some noise&#xA;xmu = mean(x,1);&#xA;xc = bsxfun(@minus,x,xmu);    %Matlab syntax for element recycling: ugly, weird.&#xA;[U,S,V] = svd(xc);  %this will be slow;&#xA;prev = U(:,1:2) * S(1:2,1:2);&#xA;prec = U(:,3:end) * S(3:end,3:end);&#xA;normv = sqrt(sum(prev .^2,2));&#xA;normc = sqrt(sum(prec .^2,2));&#xA;scatter(normv,normc);&#xA;axis equal;  %to illlustrate the differences in scaling, make axis 'square'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This generates the following scatter plot: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/ZYOwT.png&quot; alt=&quot;scatter plot made by the code&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you change &lt;code&gt;lat_d&lt;/code&gt; to 4, the line is less flat. &lt;/p&gt;&#xA;" OwnerUserId="795" LastEditorUserId="795" LastEditorDisplayName="" LastEditDate="2011-03-09T18:19:03.830" LastActivityDate="2011-03-09T18:19:03.830" CommentCount="10" />
  <row Id="8092" PostTypeId="2" ParentId="8090" CreationDate="2011-03-09T18:11:31.550" Score="6" ViewCount="" Body="&lt;p&gt;A standard approach would be to do PCA and then show a scree plot, which you ought to be able to get that out of any software you might choose. A little tinkering and you could make it more interpretable for your particular audience if necessary. Sometimes they can be convincing, but often they're ambiguous and there'a always room to quibble about how to read them so a scree plot may (edit: not!) be ideal. Worth a look though.&lt;/p&gt;&#xA;" OwnerUserId="26" LastEditorUserId="26" LastEditorDisplayName="" LastEditDate="2011-03-10T04:47:44.177" LastActivityDate="2011-03-10T04:47:44.177" CommentCount="8" />
  <row Id="8093" PostTypeId="2" ParentId="8040" CreationDate="2011-03-09T18:34:35.850" Score="1" ViewCount="" Body="&lt;p&gt;Here is a link to a fuller list of Excel charting possibilites&#xA;&lt;a href=&quot;http://spreadsheetpage.com/index.php/tip/chart_trendline_formulas/&quot; rel=&quot;nofollow&quot;&gt;http://spreadsheetpage.com/index.php/tip/chart_trendline_formulas/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="3484" LastActivityDate="2011-03-09T18:34:35.850" />
  <row Id="8094" PostTypeId="2" ParentId="7973" CreationDate="2011-03-09T18:50:07.717" Score="1" ViewCount="" Body="&lt;p&gt;You just asked for a mathematical shortcut. But I want to take a step back and point out that your approach could give you misleading results. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are you planning this approach? Add additional observations only if you aren't happy with the initial results. Once the results make you happy, presumably because the P value is low enough, then you'll stop adding more samples. If the null hypothesis were true (all population means equal), this approach is far more likely than 5% to give you a P value less than 0.05. If you go long enough, you are certain to get a P value less than 0.05. But that may take longer than the age of the universe! With time constraints, you aren't certain to get a P value less than 0.05, but the chance is way more than 5% (depending, of course, on how often you add more data and retest). You really need to choose your sample size in advance (or use specialized techniques that account for sequential testing.)&lt;/p&gt;&#xA;" OwnerUserId="25" LastActivityDate="2011-03-09T18:50:07.717" CommentCount="2" />
  <row Id="8096" PostTypeId="1" CreationDate="2011-03-09T19:54:06.577" Score="0" ViewCount="161" Body="&lt;p&gt;How do I establish the +/- in terms of a set of mean ages. My mean age is 27.2. The ages are 20 23 24 43 22 26 18 32 18 41 22 20 26 46 21 27 19 19 39 40 19 39 18 38 24 24 23 30.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you. &lt;/p&gt;&#xA;" OwnerUserId="3646" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-03-10T03:29:13.523" LastActivityDate="2011-03-10T15:50:11.723" Title="How to calculate confidence interval of the mean age of a sample?" Tags="&lt;mean&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="8097" PostTypeId="2" ParentId="8090" CreationDate="2011-03-09T20:11:09.467" Score="0" ViewCount="" Body="&lt;p&gt;I've done similar using PROC Varclus in SAS.  The basic idea is to generate a 4 cluster solution, pick the highest correlated variable with each cluster, and then to demonstrate that this 4 cluster solution explains more of the variation than the two cluster solution.  For the 2 cluster solution you could use either Varclus or the first 2 Principal Components, but I like Varclus since everything is explained via variables and not the components.  There is a varclus in R, but I'm not sure if it does the same thing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-Ralph Winters&lt;/p&gt;&#xA;" OwnerUserId="3489" LastEditorUserId="3489" LastEditorDisplayName="" LastEditDate="2011-03-09T20:18:42.687" LastActivityDate="2011-03-09T20:18:42.687" />
  <row Id="8099" PostTypeId="1" AcceptedAnswerId="8101" CreationDate="2011-03-09T20:39:48.563" Score="4" ViewCount="83" Body="&lt;p&gt;I have a linear string of unit length, and I randomly sample two locations a and b from Uniform(0, 1). Then I cut the string at these two locations to get a sub-string. What is the distribution for the size of the sub-string (i.e. |a-b|)? Otherwise, I want to at least know the mean and variance.&lt;/p&gt;&#xA;" OwnerUserId="578" LastActivityDate="2011-03-10T01:29:29.180" Title="Distribution of a random segment on a string" Tags="&lt;distributions&gt;&lt;variance&gt;&lt;mean&gt;" AnswerCount="2" />
  <row Id="8100" PostTypeId="2" ParentId="8099" CreationDate="2011-03-09T21:12:21.203" Score="6" ViewCount="" Body="&lt;p&gt;Here's a snippet of Maple code that shows that the PDF of $\lvert a - b \rvert$ is $2 - 2 t$:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;with(Statistics):&#xA;a := RandomVariable(Uniform(0,1));&#xA;b := RandomVariable(Uniform(0,1));&#xA;PDF(abs(X - Y), t) assuming 0 &amp;lt; t, t &amp;lt; 1;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;returns $2 - 2 t$. You can see that it's true by computing the probability that $\lvert a - b \rvert &amp;lt; t$ by hand: it's the area of the subset of the unit square that has horizontal or vertical distance to the diagonal of at most $t$; which means that it's one minus the area of two right triangles with legs of length $1 - t$; which means that it's $1 - 2 \cdot \frac{(1 - t)^2}{2} = 1 - (1 - t)^2$. The derivative of that is $2 - 2t$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Disclaimer: I maintain Maple's Statistics package.)&lt;/p&gt;&#xA;" OwnerUserId="2898" LastActivityDate="2011-03-09T21:12:21.203" CommentCount="2" />
  <row Id="8101" PostTypeId="2" ParentId="8099" CreationDate="2011-03-09T21:13:30.397" Score="8" ViewCount="" Body="&lt;p&gt;$(a,b)$ constitute one draw from the uniform distribution on the unit square. The region where $|a - b| \gt x$ is a pair of equilateral right triangles, one at $(0,1)$ and the other at $(1,0)$ with side lengths $1-x$; they fit together into a square with area $(1-x)^2$.  Thus the CDF of $x = |a-b|$ equals $1 - (1-x)^2$, $0 \le x \le 1$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/SFV4m.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A similar analysis appears in the solution to problem 43 of Fred Mosteller's &lt;em&gt;Fifty Challenging Problems in Probability&lt;/em&gt; (1965).  It asks for the expected values of the smallest, middle, and largest pieces.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-10T01:29:29.180" LastActivityDate="2011-03-10T01:29:29.180" />
  <row Id="8102" PostTypeId="2" ParentId="8072" CreationDate="2011-03-09T22:35:21.137" Score="4" ViewCount="" Body="&lt;p&gt;Your problem can be solved using the info in the &lt;a href=&quot;http://en.wikipedia.org/wiki/Margin_of_error#Different_confidence_levels&quot; rel=&quot;nofollow&quot;&gt;Wikipedia article for 'Margin of error'&lt;/a&gt;. As it says, the margin of error is largest when the proportion is 0.5. The maximum margin of error at 95% confidence is $m = 0.98 / \sqrt{n}.$ Rearranging gives $n = (0.98/m)^2.$ So if you want a margin of error of 5% = 0.05, $n = (0.98/0.05)^2 = 384$. So you need around 400 subjects for each group in which you want to estimate the proportion of 'yes' responses.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-03-09T22:35:21.137" CommentCount="1" />
  <row Id="8103" PostTypeId="1" CreationDate="2011-03-10T01:12:57.133" Score="1" ViewCount="67" Body="&lt;p&gt;Asking this for a colleague:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;They have two groups of people with various characteristics and to address the issue  of comparability of outcomes they would like to reference a paper or book that discusses the concept and problems of common support.&lt;/p&gt;&#xA;" OwnerUserId="1144" LastActivityDate="2011-05-09T10:44:38.510" Title="What is a good reference that discusses the problem of common support?" Tags="&lt;distributions&gt;&lt;propensity-scores&gt;" AnswerCount="1" CommentCount="6" />
  <row Id="8104" PostTypeId="1" AcceptedAnswerId="8108" CreationDate="2011-03-10T03:00:48.210" Score="5" ViewCount="127" Body="&lt;h2&gt;Background&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;One of the most commonly used weak prior on variance is the inverse-gamma with parameters $\alpha =0.001, \beta=0.001$ &lt;a href=&quot;http://www.stat.columbia.edu/~gelman/research/published/taumain.pdf&quot; rel=&quot;nofollow&quot;&gt;(Gelman 2006)&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, this distribution has a 90%CI of approximately $[3\times10^{19},\infty]$.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(pscl)&#xA;sapply(c(0.05, 0.95), function(x) qigamma(x, 0.001, 0.001))&#xA;&#xA;[1] 3.362941e+19          Inf&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;From this, I interpret that the $IG(0.001, 0.001)$ gives a low probability that variance will be very high, and the very low probability  that variance will be less than 1 $P(\sigma&amp;lt;1|\alpha=0.001, \beta=0.001)=0.006$.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pigamma(1, 0.001, 0.001)&#xA;[1] 0.006312353&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h2&gt;Question&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Am I missing something or is this actually an informative prior?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;update&lt;/em&gt; to clarify, the reason that I was considering this 'informative' is because it claims very strongly that the variance is enormous and well beyond the scale of almost any variance ever measured. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;follow-up&lt;/em&gt; would a meta-analysis of a large number of variance estimates provide a more reasonable prior?&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;h2&gt;Reference&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Gelman 2006. &lt;a href=&quot;http://www.stat.columbia.edu/~gelman/research/published/taumain.pdf&quot; rel=&quot;nofollow&quot;&gt;Prior distributions for variance parameters in&#xA;hierarchical models&lt;/a&gt;. Bayesian Analysis 1(3):515–533&lt;/p&gt;&#xA;" OwnerUserId="1381" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-29T00:40:19.050" LastActivityDate="2011-04-29T00:40:19.050" Title="Why is a $p(\sigma^2)\sim\text{IG(0.001, 0.001)}$ prior on variance considered weak?" Tags="&lt;bayesian&gt;&lt;multilevel-analysis&gt;&lt;prior&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="8105" PostTypeId="2" ParentId="8104" CreationDate="2011-03-10T03:15:51.573" Score="4" ViewCount="" Body="&lt;p&gt;It's pretty close to flat.  Its median is 1.9 E298, almost the largest number one can represent in double precision floating arithmetic.  As you point out, the probability it assigns to any interval that isn't really huge is really small.  It's hard to get less informative than that!&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-03-10T03:15:51.573" CommentCount="2" />
  <row Id="8106" PostTypeId="1" CreationDate="2011-03-10T03:45:41.767" Score="12" ViewCount="513" Body="&lt;p&gt;I am currently reading a &lt;a href=&quot;http://www.sciencedirect.com/science?_ob=ArticleURL&amp;amp;_udi=B6VG2-4Y648C8-1&amp;amp;_user=10&amp;amp;_coverDate=11/30/2009&amp;amp;_rdoc=1&amp;amp;_fmt=high&amp;amp;_orig=gateway&amp;amp;_origin=gateway&amp;amp;_sort=d&amp;amp;_docanchor=&amp;amp;view=c&amp;amp;_searchStrId=1672491428&amp;amp;_rerunOrigin=google&amp;amp;_acct=C000050221&amp;amp;_version=1&amp;amp;_urlVersion=0&amp;amp;_userid=10&amp;amp;md5=da95476f2c8535361fb031914a475b32&amp;amp;searchtype=a&quot; rel=&quot;nofollow&quot;&gt;paper&lt;/a&gt; concerning voting location and voting preference in the 2000 and 2004 election. In it, there is a chart which displays logistic regression coefficients.  From courses years back and a little &lt;a href=&quot;http://en.wikipedia.org/wiki/Logistic_regression&quot; rel=&quot;nofollow&quot;&gt;reading up&lt;/a&gt;, I understand logistic regression to be a way of describing the relationship between multiple independent variables and a binary response variable. What I'm confused about is, given the table below, because the South has a logistic regression coefficient of .903, does that mean that 90.3% of Southerners vote republican? Because of the logistical nature of the metric, that this direct correlation does not exist. Instead, I assume that you can only say that the south, with .903, votes Republican more than the Mountains/plains, with the regression of .506.  Given the latter to be the case, how do I know what is significant and what is not and is it possible to extrapolate a percentage of republican votes given this logistic regression coefficient.&#xA;&lt;img src=&quot;http://i.imgur.com/xsiyn.png&quot; alt=&quot;Table showing logistic regression coefficients&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a side note, please edit my post if anything is stated incorrectly&lt;/p&gt;&#xA;" OwnerUserId="3652" LastEditorUserId="3652" LastEditorDisplayName="" LastEditDate="2011-03-10T05:57:02.523" LastActivityDate="2011-03-10T19:22:19.677" Title="What is the significance of logistic regression coefficients?" Tags="&lt;logistic-regression&gt;" AnswerCount="5" FavoriteCount="2" />
  <row Id="8107" PostTypeId="1" CreationDate="2011-03-10T04:09:10.763" Score="2" ViewCount="154" Body="&lt;p&gt;In &lt;code&gt;R&lt;/code&gt; I can perform a Breusch–Pagan test for heteroscedasticity using the &lt;code&gt;ncvTest&lt;/code&gt; function of the &lt;code&gt;car&lt;/code&gt; package.  A Breusch–Pagan test is a type of chi squared test.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I interpret these results:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; require(car)&#xA;&amp;gt; set.seed(100)&#xA;&amp;gt; x1 = runif(100, -1, 1)&#xA;&amp;gt; x2 = runif(100, -1, 1)&#xA;&amp;gt; ncvTest(lm(x1 ~ x2))&#xA;Non-constant Variance Score Test &#xA;Variance formula: ~ fitted.values &#xA;Chisquare = 0.2343406    Df = 1     p = 0.6283239 &#xA;&amp;gt; y1 = cumsum(runif(100, -1, 1))&#xA;&amp;gt; y2 = runif(100, -1, 1)&#xA;&amp;gt; ncvTest(lm(y1 ~ y2))&#xA;Non-constant Variance Score Test &#xA;Variance formula: ~ fitted.values &#xA;Chisquare = 1.191635    Df = 1     p = 0.2750001 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="179" LastEditorUserId="179" LastEditorDisplayName="" LastEditDate="2011-03-10T04:25:41.763" LastActivityDate="2011-03-11T17:42:32.797" Title="How do I interpret the results of a Breusch–Pagan test?" Tags="&lt;basic-concepts&gt;&lt;interpretation&gt;&lt;heteroscedasticity&gt;" AnswerCount="2" />
  <row Id="8108" PostTypeId="2" ParentId="8104" CreationDate="2011-03-10T04:27:49.310" Score="10" ViewCount="" Body="&lt;p&gt;Using the inverse gamma distribution, we get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p(\sigma^2|\alpha,\beta) \propto (\sigma^2)^{-\alpha-1} exp(-\frac{\beta}{\sigma^2})$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can see easily that if $\beta \rightarrow 0$ and $\alpha \rightarrow 0$ then the inverse gamma will approach the Jeffrey's prior.  This distribution is called &quot;uninformative&quot; because it is a proper approximation to the Jeffrey's prior&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p(\sigma^2) \propto \frac{1}{\sigma^2}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which is uninformative for scale parameters &lt;a href=&quot;http://bayes.wustl.edu/etj/articles/prior.pdf&quot; rel=&quot;nofollow&quot;&gt;see page 18 here for example&lt;/a&gt;, because this prior is the only one which remains invariant under a change of scale  (note that the approximation is not invariant).  This has a indefinite integral of $log(\sigma^2)$ which shows that it is improper if the range of $\sigma^2$ includes either $0$ or $\infty$.  But these cases are only problems in the maths - not in the real world.  Never actually observe infinite value for variance, and if the observed variance is zero, you have perfect data!.  For you can set a lower limit equal to $L&amp;gt;0$ and upper limit equal $U&amp;lt;\infty$, and your distribution is proper.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While it may seem strange that this is &quot;uninformative&quot; in that it prefers small variance to large ones, but this is only on one scale.  You can show that $log(\sigma^2)$ has an improper uniform distribution.  So this prior does not favor any one scale over any other&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Although not directly related to your question, I would suggest a &quot;better&quot; non-informative distribution by choosing the upper and lower limits $L$ and $U$ in the jeffreys prior rather than $\alpha$ and $\beta$.  Usually the limits can be set fairly easily with a bit of thought to what $\sigma^2$ actually means in the real world.  If it was the error in some kind of physical quantity - $L$ cannot be smaller than the size of an atom, or the smallest size you can observe in your experiment.  Further $U$ could not be bigger than the earth (or the sun if you wanted to be really conservative).  This way you keep your invariance properties, and its an easier prior to sample from: take $q_{(b)} \sim Uniform(log(L),log(U))$, and then the simulated value as $\sigma^{2}_{(b)}=exp(q_{(b)})$.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastEditorUserId="2392" LastEditorDisplayName="" LastEditDate="2011-03-10T04:38:30.580" LastActivityDate="2011-03-10T04:38:30.580" CommentCount="9" />
  <row Id="8109" PostTypeId="2" ParentId="8106" CreationDate="2011-03-10T04:30:00.697" Score="6" ViewCount="" Body="&lt;p&gt;The idea here is that in logistic regression, we predict not the actual probability that, say, a southerner votes Republican, but a transformed version of it, the &quot;log odds&quot;. Instead of the probability $p$, we deal with $\log p/(1-p)$ and find linear regression coefficients for the log odds.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So for example, let's assume that an urban Northeasterner has probability 0.3 of voting for a Republican. (This would of course be part of the regression; I don't see it reported in this table, although I assume it's  in the original paper.) Now, $x = 1/(1+e^{-z})$ gives $z = \log {x \over 1-x}$; that is, $f^{-1}(x) = \log {x \over 1-x}$, the &quot;log odds&quot; corresponding to $x$. These &quot;log odds&quot; are what behaves linearly; the log odds corresponding to $0.3$ are $\log 0.3/0.7 \approx -0.85$. So the log odds for an urban Southerner voting Republican are this (what Wikipedia calls the intercept, $\beta_0$) plus  the logistic regression coefficient for the South, $0.903$ -- that is, $-0.85 + 0.904 = 0.05$. But you want an actual probability, so we need to invert the function $p \to \log p/(1-p)$. That gives $f(0.05) \approx 1/(1+e^{-0.05}) \approx 0.51$. The actual odds have gone from $0.43$ to $1$, to $1.05$ to $1$; the ratio $1.05/0.43$ is $e^{0.903}$, the exponential of the logistic regression coefficient.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furthermore, the effects for, say, region of the country and urban/suburban/rural don't interact. So the log odds of a rural Midwesterner voting Republican, say, are $-0.85 + 0.37 + 0.68 = +0.20$ according to this model; the probability is $f(0.20) = 1/(1+e^{-0.20}) = 0.55$.&lt;/p&gt;&#xA;" OwnerUserId="98" LastEditorUserId="98" LastEditorDisplayName="" LastEditDate="2011-03-10T05:05:51.337" LastActivityDate="2011-03-10T05:05:51.337" CommentCount="1" />
  <row Id="8110" PostTypeId="2" ParentId="8106" CreationDate="2011-03-10T05:42:36.100" Score="2" ViewCount="" Body="&lt;p&gt;The coefficients in the logistic regression represent the tendency for a given region/demographic to vote Republican, compared to a reference category. A positive coefficent means that region is more likely to vote Republican, and vice-versa for a negative coefficient; a larger absolute value means a stronger tendency than a smaller value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reference categories are &quot;Northeast&quot; and &quot;urban voter&quot;, so all the coefficients represent contrasts with this particular voter type.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, there's also no restriction on the coefficients in a logistic regression to be in [0, 1], even in absolute value. Notice that the Wikipedia article itself has an example of a logistic regression with coefficients of -5 and 2.&lt;/p&gt;&#xA;" OwnerUserId="1569" LastActivityDate="2011-03-10T05:42:36.100" />
  <row Id="8111" PostTypeId="2" ParentId="8088" CreationDate="2011-03-10T05:44:26.280" Score="1" ViewCount="" Body="&lt;p&gt;I think you are trying to interpret P(A|B) and P(B|A) as if they should be the same thing.  There is no reason for them to be equal, because of the product rule:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(AB)=P(A|B)P(B)=P(B|A)P(A)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;unless $P(B)=P(A)$ then $P(A|B) \neq P(B|A)$ in general.  This explains the difference in the &quot;yn&quot; case.  Unless you have a &quot;balanced&quot; table (row totals equal to column totals), the conditional probabilities (row and column) will not be equal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A test for &quot;logical/statistical independence&quot; (but not causal independence) between categorical variables can be given as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$T=\sum_{ij} O_{ij} log\Big(\frac{O_{ij}}{E_{ij}}\Big)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $ij$ indexes the cells of the table (so in your example, $ij=11,12,21,22$).  $O_{ij}$ is the observed value in the table, and $E_{ij}$ is what is &quot;expected&quot; under independence, which is simply the product of the marginals &#xA;$$E_{ij}=O_{\bullet \bullet}\frac{O_{i \bullet}}{O_{\bullet \bullet}}\frac{O_{\bullet j}}{O_{\bullet \bullet}}&#xD;&#xA;=\frac{O_{i \bullet}O_{\bullet j}}{O_{\bullet \bullet}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where a &quot;$\bullet$&quot; indicates that you sum over that index.  You can show that if you had a prior log-odds value for independence of $L_{I}$ then the posterior log-odds is $L_{I}-T$.  The alternative hypothesis is $E_{ij}=O_{ij}$ (i.e. no simplification, no independence), for which $T=0$.  Thus T says &quot;how strongly&quot; the data support non-independence, within the class of multinomial distributions.  The good thing about this test is that it works for all $E_{ij}&amp;gt;0$, so you don't have to worry about a &quot;sparse&quot; table.  This test will still give sensible results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the regressions, this is telling you that the average IQ value is different between the two values of sex, although I don't know the scale of the AIC difference (is this &quot;big&quot;?).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not sure how appropriate the AIC is to a binomial GLM.  It may be a better idea to look at the ANOVA and deviance tables for the LM and GLM respectively.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, have you plotted the data? always plot the data!!!  this will be able to tell you things that the test does not.  How different do the IQs look when plotted by sex? how different do the sexes look when plotted by IQ?&lt;/p&gt;&#xA;" OwnerUserId="2392" LastEditorUserId="2392" LastEditorDisplayName="" LastEditDate="2011-03-10T05:51:25.883" LastActivityDate="2011-03-10T05:51:25.883" />
  <row Id="8112" PostTypeId="2" ParentId="8063" CreationDate="2011-03-10T06:01:42.227" Score="1" ViewCount="" Body="&lt;p&gt;In addition to Anscombe's quartet as mentioned by Peter Flom, &lt;a href=&quot;http://ferrari.dmat.fct.unl.pt/personal/mle/GestaoRisco/Embrechts/risk.pdf&quot; rel=&quot;nofollow&quot; title=&quot;Correlation: pitfalls and alternatives&quot;&gt;here&lt;/a&gt; is a very nice paper in the risk-management context illustrating the problems of using linear correlation with non-normally distributed variables. In a nutshell, much of our intuition about how correlation behaves -- all values of $\rho \in [-1, 1]$ are possible; an exact monotonic relationship implies $|\rho | = 1$; $\rho = 0$ implies independence; etc, doesn't necessarily apply in the case of non-normality.&lt;/p&gt;&#xA;" OwnerUserId="1569" LastEditorUserId="1569" LastEditorDisplayName="" LastEditDate="2011-03-10T06:07:40.783" LastActivityDate="2011-03-10T06:07:40.783" CommentCount="1" />
  <row Id="8113" PostTypeId="2" ParentId="8107" CreationDate="2011-03-10T07:53:02.863" Score="2" ViewCount="" Body="&lt;p&gt;First application of &lt;code&gt;ncvTest&lt;/code&gt; reports that there is no heteroscedasticity, as it should. The second is not meaningful, since your dependent random variable is random walk. &lt;a href=&quot;http://en.wikipedia.org/wiki/Breusch%E2%80%93Pagan_test&quot; rel=&quot;nofollow&quot;&gt;Breusch-Pagan test&lt;/a&gt; is assymptotic, so I suspect that it cannot be readily applied for random walk. I do not think that there are tests for heteroscedasticity for random walks, due to the fact that non-stationarity poses much more problems than the heteroscedasticity, hence testing for the latter in the presence of the former is not practical.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-03-10T07:53:02.863" />
  <row Id="8114" PostTypeId="1" AcceptedAnswerId="8120" CreationDate="2011-03-10T08:07:51.637" Score="5" ViewCount="110" Body="&lt;p&gt;According to &lt;a href=&quot;http://www.stat.psu.edu/online/courses/stat509/09_interim/09_interim_print.htm&quot; rel=&quot;nofollow&quot;&gt;this document&lt;/a&gt;: &lt;em&gt;The estimated logarithm of the hazard ratio is approximately normally distributed with variance (1/d1) + (1/d2), where d1 and d2 are the numbers of events in the two treatment groups.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you have a reference for this statement? Or at least can you say me which estimator is used?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you in advance, Marco&lt;/p&gt;&#xA;" OwnerUserId="3019" LastActivityDate="2011-03-10T10:44:15.927" Title="The estimated logarithm of the hazard ratio is approximately normally distributed" Tags="&lt;survival&gt;&lt;hazard&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="8115" PostTypeId="2" ParentId="7258" CreationDate="2011-03-10T08:41:57.213" Score="1" ViewCount="" Body="&lt;p&gt;If I understand, you have a classification problem where you observe $(X_1,Y_1),\dots,(X_n,Y_n)$ ($Y_i=1$ correspond to default and $Y_i=0$ no default), and you have much more $Y_i=0$ than $Y_i=1$. You want to take a subsample $S$ of the $(X_i,Y_i)$ that have $Y_i=0$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not sure I understand all the points in your question but I will address the question of the effect of subsampling, and the other question of assigning equal/non equal importance to classes in classification. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Forecasting probability of default.&lt;/strong&gt; &#xA;For predictive purpose, if you use the bayes principle, you migh construct, for $i=0,1$ an estimate $\hat{f}_i(x)$ of the density $f_i$ of $(X|Y=i)$,  and estimate the probability that $Y=i$ by $\hat{\pi}_i$. This will permit you to estimate the probability of default given $X=x$:&#xA;$$\hat{P}(x)=\frac{\hat{\pi}_0\hat{f}_0(x)}{\hat{\pi}_1\hat{f}_1(x)+\hat{\pi}_0\hat{f}_0(x)}$$&#xA;If you have a correct subsample and if you know the proportions, you only need to ajust $\hat{\pi}_0$ (resp. $\pi_{1}$) of no default (res. of default). The only consequence of subsampling will then be that $\hat{f}_0$ will not be as good as it should. This will affect the most the probability $\hat{P}$  when $\hat{\pi}_1\hat{f}_1(x)&amp;gt;&amp;gt;\hat{\pi}_0\hat{f}_0(x)$, that is when you are certain that there is no default. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Forecasting default&lt;/strong&gt; &#xA;If you just want a classification rule (i.e. forecast default/ or no default) then can set up a threshold on the preceding probability which will not necessarily be $1/2$. If you think false default alarm costs more than missed default alarm, then you will take a threshold larger than $1/2$, and if it is the contrary you will use a threshold lower than $1/2$. In this case the effect of subsampling can be deduced from the effect in the case of probability forecast. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Hypothesis testing approach&lt;/strong&gt;&#xA; I think it is clear that there are cases when you don't want these errors to be treated with equal importance and setting the threshold in the preceding paragraph my be subjective. I found interesting to use a hypothesis testing approach (i.e. not bayesian) in which you want to construct a decision rule $\psi(x)$ that minimizes&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ \alpha P_1(\psi(X)= 0) + (1-\alpha) P_0(\psi(X)=1) \;\; [1]$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\alpha$ is a parameter you need to choose to set relative importance of your errors. For example if you don't want to give more importance to one error than to the other you can take $\alpha=1/2$. The rule that minimizes equation [1] above is given by &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ \psi^*(x)= \left \{ \begin{array}{ccc} 0 &amp;amp; if  &amp;amp; \alpha f_0(x)/(\alpha f_0(x)+(1-\alpha)f_1(x))&amp;gt;1/2 \\ 1 &amp;amp; else &amp;amp; \end{array} \right .$$&#xA;If you replace, for $i=0,1$ $f_i$ by $\hat{f}_i$ then you get the bayesian classification rule but with the weight $\hat{\pi}_0$ (frequency of default) replace by the real importance $\alpha$ you give to a default.  &lt;/p&gt;&#xA;" OwnerUserId="223" LastActivityDate="2011-03-10T08:41:57.213" />
  <row Id="8116" PostTypeId="1" CreationDate="2011-03-10T08:59:15.300" Score="0" ViewCount="156" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;Possible Duplicate:&lt;/strong&gt;&lt;br&gt;&#xA;  &lt;a href=&quot;http://stats.stackexchange.com/questions/7727/how-to-correlate-two-time-series-with-gaps-and-different-time-bases&quot;&gt;How to correlate two time series with gaps and different time bases?&lt;/a&gt;  &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&#xA;&#xA;&lt;p&gt;Hi, I have values from two sensors stored in two vectors A and B. They both represent values of the sensors at times TA and TB which is stored in two other vectors(since it is not uniform sampling) Both A and B represent the same data but A is shifted a bit to the right because of the delay in starting the sensors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is, how do I calculate this delay and more importantly, how do I shift A to match B or vice versa, such that I can do a one-to-one correspondence of the data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Right now, I am finding the peak of the cross-correlation function of A and B to find the offset (in number of samples, not time) and padding the arrays with zero upto that number, but I am 100% sure that is wrong. Problem is, I don't know how else.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PLEASE help!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="3233" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-10T09:26:35.410" LastActivityDate="2011-03-10T09:26:35.410" ClosedDate="2011-04-11T01:53:12.223" Title="Cross correlation in MATLAB" Tags="&lt;time-series&gt;&lt;matlab&gt;&lt;cross-correlation&gt;" CommentCount="1" />
  <row Id="8117" PostTypeId="1" AcceptedAnswerId="8118" CreationDate="2011-03-10T09:17:00.893" Score="6" ViewCount="138" Body="&lt;p&gt;Our small team was having a discussion and got stuck. Does anyone know whether Cox regression has an underlying Poisson distribution. We had a debate that maybe Cox regression with constant time at risk will have similarities with Poisson regression with a robust variance. Any ideas?&lt;/p&gt;&#xA;" OwnerUserId="2961" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2011-03-10T12:11:19.693" LastActivityDate="2011-03-10T12:11:19.693" Title="Does Cox Regression have an underlying Poisson distribution?" Tags="&lt;regression&gt;&lt;poisson&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="8118" PostTypeId="2" ParentId="8117" CreationDate="2011-03-10T09:34:42.957" Score="9" ViewCount="" Body="&lt;p&gt;Yes, there is a link between these two regression models. Here is an illustration:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose the baseline hazard is constant over time: $h_{0}(t) = \lambda$. In that case, the survival function is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$S(t) = \exp\left(-\int_{0}^{t} \lambda du\right) = \exp(-\lambda t)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and the density function is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$f(t) = h(t) S(t) = \lambda \exp(-\lambda t)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is the pdf of an exponential random variable with expectation $\lambda^{-1}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Such a configuration yields the following parametric Cox model (with obvious notations):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$h_{i}(t) = \lambda \exp(x&amp;#39;_{i} \beta)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the parametric setting the parameters are estimated using the classical likelihood method. The log-likelihood is given by&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$l = \sum_{i} \left\{ d_{i}\log(h_{i}(t_{i})) - t_{i} h_{i}(t_{i}) \right\}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $d_{i}$ is the event indicator.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Up to an additive constant, this is nothing but the same expression as the log-likelihood of the $d_{i}$'s seen as realizations of a Poisson variable with mean $\mu_{i} = t_{i}h_{i}(t)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a consequence, one can obtain estimates using the following Poisson model:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\log(\mu_{i}) = \log(t_{i}) + \beta_0 + x_{i}&amp;#39;\beta$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\beta_0 = \log(\lambda)$.&lt;/p&gt;&#xA;" OwnerUserId="3019" LastActivityDate="2011-03-10T09:34:42.957" CommentCount="4" />
  <row Id="8119" PostTypeId="1" CreationDate="2011-03-10T10:09:33.283" Score="2" ViewCount="72" Body="&lt;p&gt;I am trying to sample from a Gamma distribution in JAGS&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;gd[i] ~ dgamma(k,r)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where &lt;strong&gt;k&lt;/strong&gt; and &lt;strong&gt;r&lt;/strong&gt; are have priors such that they are positive. However I get the following error when I compile the model:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  Error in node gd[9]&#xA;Unobserved node inconsistent with unobserved parents at initialization&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Any ideas?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT:  BACKGROUND&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I am trying to do is to estimate the parameters of the CIR process with Bayesian techniques. In order to assess the estimator, I have simulated some data with given parameters on a quarterly basis (timeStep=90):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;jdata &amp;lt;- list(timeSerie=c(35000, 47123.3999346341, 55373.904244638, 71755.7487795017, 91246.3370619453, 76969.2358694695, 83390.030308266, 80657.1222156083, 76784.466470129, 83962.4642428193, 85335.5796326587, 87018.8279086056, 74059.9912730579, 69683.5404626939, 90356.0122717258, 87240.7793078735, 74276.2931903815, 39899.5948334073, 37911.738710607, 73811.7536157058, 69334.5390010963, 51326.297331755, 42711.7358571988, 32409.9939465308, 16663.7353817099, 22421.1721803643, 38604.0551895535, 50181.3269231991, 44346.3517857709, 52127.2156791002, 54740.1181745155, 41869.456839522, 50665.5626313423, 51121.4339441224, 39059.432945341, 44350.0205383802, 68629.3291332604, 49933.2126408239, 64358.5131164748, 102975.634845413, 84658.5787221384, 52243.061086781, 56988.7184089767, 43548.6558054764, 38536.145535317, 43115.4155553003, 50328.6208374212, 47191.4284861612, 42603.6038389972, 43749.2977769786, 43583.8022320709, 36672.3782472362, 26377.9033658583, 41746.6472558272, 26086.7868276075, 47228.5390765096, 75765.4757503756, 70939.107521347, 63949.8857556708, 55153.6701237129),timeStep=90)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then I wrote a model with JAGS. The problem is that the CIR process has scaled noncentral chi square distribution and this is not directly supported in JAGS. So inspired by &lt;code&gt;R&lt;/code&gt; implementation, I came up with &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model {&#xA;    for (i in 2:length(timeSerie)){&#xA;        ncp[i] &amp;lt;- 2*c*et*timeSerie[i-1]&#xA;        pd[i] ~ dpois(ncp[i]/2)&#xA;        cd[i] ~ dgamma(pd[i]+1E-10,c)&#xA;        gd[i] ~ dgamma(df/2,4*c)&#xA;        timeSerie[i] ~ dsum(cd[i],gd[i])&#xA;    }&#xA;&#xA;    #parameters&#xA;    et &amp;lt;- exp(-kappa*timeStep)&#xA;    c &amp;lt;- 2*kappa/(sigma^2*(1-et))&#xA;    df &amp;lt;- 4*kappa*mu/sigma^2&#xA;&#xA;    #CIR parameter priors &#xA;    lkappa ~ dunif(-10,-2)&#xA;    kappa &amp;lt;- exp(lkappa)&#xA;    mu ~ dunif(30000,100000)&#xA;    lsigma ~ dunif(0,5) &#xA;    sigma &amp;lt;- exp(lsigma)&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Trying to run the model (&lt;code&gt;jags.model&lt;/code&gt; in &lt;code&gt;R&lt;/code&gt;), I get the following error &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  Error in node gd[18]&#xA;Unobserved node inconsistent with unobserved parents at initialization&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(Now it is 18 instead of 8, because I use other simulated data).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried to debug JAGS (gdb and valgrind) to no avail: I don't have enough knowledge to do it...&lt;/p&gt;&#xA;" OwnerUserId="1443" LastEditorUserId="1443" LastEditorDisplayName="" LastEditDate="2011-03-11T08:03:16.257" LastActivityDate="2011-03-11T08:03:16.257" Title="Jags error with dgamma" Tags="&lt;bayesian&gt;&lt;mcmc&gt;&lt;jags&gt;" CommentCount="2" />
  <row Id="8120" PostTypeId="2" ParentId="8114" CreationDate="2011-03-10T10:38:25.433" Score="7" ViewCount="" Body="&lt;p&gt;The fact that this is approximately normally distributed relies on the central limit theorem (CLT), so will be a better approximation in large samples. The CLT works better for the log of any ratio (risk ratio, odds ratio, hazard ratio..) than for the ratio itself. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In suitably large samples, I think this is a good approximation to the variance in two situations:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The hazard in each group is constant over time (regardless of the hazard ratio)&lt;/li&gt;&#xA;&lt;li&gt;The proportional hazards assumption holds and the hazard ratio is close to 1&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I think it may become a fairly crude assumption in situations far from these, i.e. if the hazards vary considerably over time and the hazard ratio is far from 1. Whether you can do better depends on what information is available. If you have access to the full data you can fit a proportional hazards model and get the variance of the log hazard ratio from that. If you only have the info in a published paper, various other approximations have been developed by meta-analysts. These two references are taken from the &lt;a href=&quot;http://cochrane-handbook.org&quot; rel=&quot;nofollow&quot;&gt;Cochrane Handbook&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;M. K. B. Parmar, V. Torri, and L. Stewart (1998). &lt;a href=&quot;http://dx.doi.org/10.1002/%28SICI%291097-0258%2819981230%2917:24%3C2815%3a%3aAID-SIM110%3E3.0.CO;2-8&quot; rel=&quot;nofollow&quot;&gt;&quot;Extracting summary statistics to perform meta-analyses of the published literature for survival endpoints.&quot;&lt;/a&gt; &lt;em&gt;Statistics in Medicine&lt;/em&gt; &lt;strong&gt;17&lt;/strong&gt; (24):2815-2834. &lt;/li&gt;&#xA;&lt;li&gt;Paula R. Williamson, Catrin Tudur Smith, Jane L. Hutton, and Anthony G. Marson. &lt;a href=&quot;http://dx.doi.org/10.1002/sim.1303&quot; rel=&quot;nofollow&quot;&gt;&quot;Aggregate data meta-analysis with time-to-event outcomes&quot;&lt;/a&gt;. &lt;em&gt;Statistics in Medicine&lt;/em&gt; &lt;strong&gt;21&lt;/strong&gt; (22):3337-3351, 2002.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In Parmar et al, the expression you give would follow from using observed numbers in place of expected in their equation (5), or combining equations (6) and (12). Equations (5) and (6) are based on &lt;a href=&quot;http://en.wikipedia.org/wiki/Logrank_test&quot; rel=&quot;nofollow&quot;&gt;logrank&lt;/a&gt; methods. They reference &lt;a href=&quot;http://books.google.com/books?id=pivvAAAAMAAJ&quot; rel=&quot;nofollow&quot;&gt;Kalbfleisch &amp;amp; Prentice&lt;/a&gt; for equation (12) but I don't have that to hand, so maybe someone who does might like to check it and add to this.&lt;/p&gt;&#xA;" OwnerUserId="449" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-03-10T10:44:15.927" LastActivityDate="2011-03-10T10:44:15.927" CommentCount="1" />
  <row Id="8121" PostTypeId="2" ParentId="8103" CreationDate="2011-03-10T10:39:31.160" Score="0" ViewCount="" Body="&lt;p&gt;If I understand, you have a classification problem where you observe $(X_1,Y_1),\dots,(X_n,Y_n)$. Denoting by $\mathcal{X}$ the space where $X$ takes values, you assume the existance of $\mathcal{X}_{10}$ the smallest closed subset of $\mathcal{X}$  such that: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\mathcal{X}\setminus \mathcal{X}_{10}$  can be partinioned into $\mathcal{X}_{1\setminus 0}\cup \mathcal{X}_{0\setminus 1}$ with $P_{0}(X\in \mathcal{X}_{1\setminus 0})=0$, $P_{1}(X\in \mathcal{X}_{0\setminus 1})=0$ (for $i=0,1$ $P_i$ is the distribution of $(X|Y=i)$)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think there are two problems that may arise from your question: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;How do you justify a decomposition of the distribution to be able to solve the problem only on $\mathcal{X}_{10}$ (Theoretical problem)&lt;/li&gt;&#xA;&lt;li&gt;How do you estimate $\mathcal{X}_{1\setminus 0}$, $\mathcal{X}_{0\setminus 1}$ and$\mathcal{X}_{0 1}$ from data (practical estimation problem) &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Theoretical problem&lt;/strong&gt;&#xA;In this case (and under suitables conditions) you can decompose $P_{1}$ and $P_0$ using the &lt;strong&gt;Radon-Nikodym theorem&lt;/strong&gt; (This &lt;a href=&quot;http://fr.wikipedia.org/wiki/Th%C3%A9or%C3%A8me_de_Radon-Nikodym-Lebesgue#Th.C3.A9or.C3.A8me_de_Radon-Nikodym&quot; rel=&quot;nofollow&quot;&gt;version&lt;/a&gt;) into:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P_{1}=P_{1,\mathcal{X}_{1\setminus 0}}+P_{1,\mathcal{X}_{0 1}}$$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P_{0}=P_{0,\mathcal{X}_{0\setminus 1}}+P_{0,\mathcal{X}_{0 1}}$$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;with $P_{0,\mathcal{X}_{0 1}}\sim P_{1,\mathcal{X}_{0 1}}$ (mutually absolutly continuous) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P_{1,\mathcal{X}_{0 1}} \bot P_{1,\mathcal{X}_{1\setminus 0}} \bot P_{0,\mathcal{X}_{0\setminus 1}}$ (mutually singular)&#xA;Note that distributions in the decomposition are not probabilities, and you need to renormalize (i.e. compute $P_{i}(X\in A)$ when $A$ is one of the three above mentionned sets). This renormalization is also required from a practical point of view. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you have done the renormalization you can construct a classification rule on $\mathcal{X}_{0 1}$ using renormalized version of $P_1$ and $P_0$ 's restrictions. You can extend this classification rule in the obvious way to the whole $\mathcal{X}$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Estimation problem&lt;/strong&gt;&#xA;If you want to &quot;learn&quot; the above mentionned rule from the data, you need to estimate the corresponding sets I guess there is a large litterature in &quot;support&quot; estimation but in view of the classification problem at the end, it is not a big deal since the places where you might be wrong in terms of support estimation are does where you don't have much &quot;mass&quot; and hence does that are not really important for the classification errors at the end... at least from a bayesian perspective of which error is important... (see my answer &lt;a href=&quot;http://stats.stackexchange.com/questions/7258/information-content-of-examples-and-undersampling/8115#8115&quot;&gt;here&lt;/a&gt;)&lt;/p&gt;&#xA;" OwnerUserId="223" LastActivityDate="2011-03-10T10:39:31.160" />
  <row Id="8123" PostTypeId="2" ParentId="8106" CreationDate="2011-03-10T13:08:08.867" Score="3" ViewCount="" Body="&lt;p&gt;You also asked &quot;how do I know what is significant and what is not.&quot;  (I assume you mean statistically significant, since practical or substantive significance is another matter.)   The asterisks in the table refer to the footnote:  some effects are noted as having small &lt;em&gt;p&lt;/em&gt;-values. These are obtained using a Wald test of the significance of each coefficient. Assuming random sampling, &lt;em&gt;p&lt;/em&gt;&amp;lt;.05 means that, if there were no such effect in the larger population, the probability of seeing a connection as strong as the one observed, or stronger, in a sample of this size would be less than .05.  You'll see many threads on this site discussing the subtle but important related point that &lt;em&gt;p&lt;/em&gt;&amp;lt;.05 does &lt;em&gt;not&lt;/em&gt; mean that there is a .05 probability of there being no connection in the larger population.&lt;/p&gt;&#xA;" OwnerUserId="2669" LastActivityDate="2011-03-10T13:08:08.867" />
  <row Id="8124" PostTypeId="2" ParentId="8083" CreationDate="2011-03-10T13:31:19.273" Score="2" ViewCount="" Body="&lt;p&gt;Your question contains a &quot;contradiction&quot;.  The way you have described it, the variable defined as $X$ &lt;em&gt;cannot have a Poisson distribution&lt;/em&gt;.  I'll explain why.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, you choose a cut-off point, $y_{0}$, and define the original variables $Y_{i}\:(i=1,\dotsc,r)$.  You then take a new variable $X_{i}$ and define it as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;  X_{i} \equiv \Bigg(&#xD;&#xA;  \begin{matrix}&#xD;&#xA;  1 &amp;amp; Y_{i} &amp;gt; y_{0} \\&#xD;&#xA;  0 &amp;amp; Y_{i} \leq y_{0}&#xD;&#xA; \end{matrix}&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then $X=\sum_{i} X_{i} \sim Poisson(\lambda)$.  &lt;a href=&quot;http://en.wikipedia.org/wiki/Raikov%27s_theorem&quot; rel=&quot;nofollow&quot;&gt;Raikov's theorem&lt;/a&gt;  Says that this implies the individual terms are also poisson $X_{i}\sim Poisson(\frac{\lambda}{r})$.  But this is impossible, for as $X_{i}$ has been defined, it can only have two values $(0,1)$, a Poisson has support $(0,1,2,\dots)$.  Therefore $X$ cannot be poisson.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the way it has been defined in the question $X$ is binomially distributed &#xA;$$X \sim Bin(r,\theta)$$&#xA; where &#xA;$$\theta \equiv Pr(Y_{i}&amp;gt;y_{0})=1-F_{Y}(y_{0})$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What your bootstrap procedure is doing is attempting to estimate $E(X)=r\theta$, but the estimate of $\theta$ is just given by $\hat{\theta}=\frac{X_{obs}}{r}$ and has variance of $\frac{\hat{\theta}(1-\hat{\theta})}{r+I_{B}}$ where $I_{B}$ is what I call the &quot;Bayesian indicator&quot;.  If you are Bayesian, with haldene prior, $I_{B}=1$, if you are frequentist then $I_{B}=0$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could do a bootstrap if you wanted, by because the maths is there, I'm sure you would get the same answers for big enough bootstrap samples (the simulation will only demonstrate the maths).  Also note that the binomial is &quot;under-dispersed&quot; with the mean being greater than the variance, as your bootstrap shows.  Also your 3 iterations are &quot;approximately&quot; in line with theoretical mean $r\hat{\theta}=2$ and theoretical variance $r\hat{\theta}(1-\hat{\theta})=1.2$.  From looking at the bootstrap samples, this is to be expected (your average/variance being below the theoretical average/variance) as you sampled the &quot;high&quot; number disproportionately in 2 of the simulations.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-03-10T13:31:19.273" CommentCount="8" />
  <row Id="8125" PostTypeId="2" ParentId="8012" CreationDate="2011-03-10T13:55:34.027" Score="1" ViewCount="" Body="&lt;p&gt;First let us introduce notation $y_{ki}=t_{k,i,2}-t_{k,i,1}$. Then $\bar\tau=\frac{1}{K}\sum_k\frac{1}{n_k}\sum_iy_{ki}$. The variance then is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align}&#xA;Var(\bar\tau)&amp;amp;=E(\bar\tau-E\bar\tau)^2=E\left(\frac{1}{K}\sum_k\frac{1}{n_k}\sum_i(y_{ki}-Ey_{ki})\right)^2\\&#xA;&amp;amp;=E\frac{1}{K^2}\sum_k\sum_l\frac{1}{n_kn_l}\sum_i\sum_jcov(y_{ki},y_{lj})&#xA;\end{align}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now the question is what can you say about $cov(y_{ki},y_{lj})$ which converted to original notation is $cov(t_{k,i,2}-t_{k,i,1},t_{l,j,2}-t_{l,j,1})$. If they are not correlated when $k\neq l$ or $i\neq j$ then the variance is simply&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align}&#xA;Var(\bar\tau)=\frac{1}{K^2}\sum_k\frac{1}{n_k^2}\sum_i\sigma_{ki}^2.&#xA;\end{align}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would be careful to employ stationarity. Your process has 2 indexes, when stationarity is usually defined for single indexed processes. When you have single index there is natural order of the observations $t_1&amp;lt;t_2&amp;lt;...$ and stationarity means that if you fix times $t_1&amp;lt;...&amp;lt;t_k$ then vectors $(X_{t_1},...,X_{t_k})$ and $(X_{t_1+h},...,X_{t_k+h})$ behave similarly for every shift $h$. If the index is two-dimensional there is no natural order, also shifting is now two-dimensional so there is no one clear way how to define stationarity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So what process exactly you observe and how $t_{k,i}$ is chosen from it?&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-03-10T13:55:34.027" CommentCount="1" />
  <row Id="8126" PostTypeId="1" AcceptedAnswerId="8128" CreationDate="2011-03-10T14:04:22.007" Score="4" ViewCount="204" Body="&lt;p&gt;I want to calculate the hat matrix directly in R for a logit model. According to Long (1997) the hat matrix for logit models is defined as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$H = VX(X&amp;#39;VX)^{-1} X&amp;#39;V$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;X is the vector of independent variables, and V is a diagonal matrix with $\sqrt{\pi(1-\pi)}$ on the diagonal. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I use the &lt;code&gt;optim&lt;/code&gt; function to maximize the likelihood and derive the hessian. So I guess my question is: how do i calculuate $V$  in R?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Note:&lt;/em&gt; My likelihood function looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;loglik &amp;lt;-  function(theta,x,y){&#xA;y &amp;lt;- y&#xA;x &amp;lt;- as.matrix(x)&#xA;beta &amp;lt;- theta[1:ncol(x)]&#xA;loglik &amp;lt;- sum(-y*log(1 + exp(-(x%*%beta))) - (1-y)*log(1 + exp(x%*%beta)))&#xA;return(-loglik)&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And i feed this to the optim function as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;logit &amp;lt;- optim(c(1,1),loglik, y = y, x = x, hessian = T)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Where x is a matrix of independent variables, and y is a vector with the dependent variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Note:&lt;/em&gt; I know that there are canned procedures for doing this, but I need to do it from scratch&lt;/p&gt;&#xA;" OwnerUserId="2704" LastEditorUserId="2704" LastEditorDisplayName="" LastEditDate="2011-03-10T14:55:03.557" LastActivityDate="2011-03-10T14:59:47.350" Title="How to calculate the hat matrix for logistic regression in R?" Tags="&lt;r&gt;&lt;logistic-regression&gt;&lt;deviance&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="8127" PostTypeId="1" CreationDate="2011-03-10T14:12:47.033" Score="0" ViewCount="82" Body="&lt;p&gt;I have two variables and 1000 cases. How can I statistically find representative cases from total of 1000, based on statistical properties of both variables and correlation between them. Perhaps something based on T-test and 95% (or 99%) interval but for both variables? I would like to know which statistical method can find cases that have both values (simultaneously) statistically the most significant. I know that this deals with sample distribution and estimating the proportions.&lt;/p&gt;&#xA;" OwnerUserId="3656" LastEditorUserId="3656" LastEditorDisplayName="" LastEditDate="2011-03-10T14:36:47.307" LastActivityDate="2011-03-10T14:36:47.307" Title="How to make representative sample in dataset with two variables" Tags="&lt;distributions&gt;&lt;sample&gt;" CommentCount="4" />
  <row Id="8128" PostTypeId="2" ParentId="8126" CreationDate="2011-03-10T14:20:46.650" Score="9" ViewCount="" Body="&lt;p&gt;For logistic regression $\pi$ is calculated using formula&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\pi=\frac{1}{1+\exp(-X\beta)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So diagonal values of $V$ can be calculated in the following manner:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pi &amp;lt;- 1/(1+exp(-X%*%beta))&#xA;v &amp;lt;- sqrt(pi*(1-pi))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now multiplying by diagonal matrix from left means that each row is multiplied by corresponding element from diagonal. Which in R can be achieved using simple multiplication:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;VX &amp;lt;- X*v &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then &lt;code&gt;H&lt;/code&gt; can be calculated in the following way:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;H &amp;lt;- VX%*%solve(crossprod(VX,VX),t(VX))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note&lt;/strong&gt; Since $V$ contains standard deviations I suspect that the correct formula for $H$ is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$H=VX(X&amp;#39;V^2X)^{-1}X&amp;#39;V$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The example code works for this formula.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-03-10T14:59:47.350" LastActivityDate="2011-03-10T14:59:47.350" CommentCount="6" />
  <row Id="8129" PostTypeId="2" ParentId="8089" CreationDate="2011-03-10T14:38:23.600" Score="2" ViewCount="" Body="&lt;p&gt;Recall (in combination with precision) is generally used in areas where one is primarily interested in finding the Positives. An example for such an area is e.g. Performance Marketing or (as already suggested by ch'ls link) the area of Information Retrieval.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are primarily interested in finding the negatives, &quot;True Negative Rate&quot; (as already suggested by chl) is the way to go. But don't forget to look at a &quot;precision for focus on negatives&quot;-metric (i.e. $\frac{TN}{TN + FN}$, because otherwise the &quot;True Negative Rate&quot; can be optimized by setting the prediction to &quot;Negative&quot; for all data points).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are interested in optimizing recall for both negatives AND positives, you should look at &quot;Accuracy&quot; (see again chl's link). But beware of class skew (i.e. you have many many more positives than negatives or vice versa ... in this case one can &quot;optimize&quot; accuracy by setting the prediction to the major class for all data points).&lt;/p&gt;&#xA;" OwnerUserId="264" LastEditorUserId="264" LastEditorDisplayName="" LastEditDate="2011-03-10T14:51:01.483" LastActivityDate="2011-03-10T14:51:01.483" CommentCount="3" />
  <row Id="8130" PostTypeId="1" CreationDate="2011-03-10T14:52:20.337" Score="10" ViewCount="290" Body="&lt;p&gt;Hi guys,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am a software designer by trade and I am working on a project for a client, and I would like to make sure that my analysis is statistically sound.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider the following:  &lt;strong&gt;We have &lt;em&gt;n&lt;/em&gt; advertisements (n &amp;lt; 10), and we simply want to know which ad performs the best.&lt;/strong&gt;  Our ad server will randomly serve one of these ads.  Success is if the a user clicks on the ad -- our server keeps track of that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given: Confidence Interval: 95%&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Question:  What is the estimated sample size? (How many total ads must we serve), Why? (remember i am a dummy)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="3658" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-03-11T08:06:25.987" LastActivityDate="2011-03-15T02:41:01.220" Title="Sample size required to determine which of a set of advertisements has the highest click through rate" Tags="&lt;anova&gt;&lt;sample-size&gt;&lt;t-test&gt;&lt;rule-of-thumb&gt;" AnswerCount="1" CommentCount="4" FavoriteCount="5" />
  <row Id="8131" PostTypeId="2" ParentId="8106" CreationDate="2011-03-10T15:03:04.673" Score="7" ViewCount="" Body="&lt;p&gt;That the author has forced someone as thoughtful as you to have ask a question like this is compelling illustration of why the practice -- still &lt;em&gt;way&lt;/em&gt; too common -- of confining reporting of regression model results to a table like this is so unacceptable. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;You can, as pointed out, try to transform the logit coefficient into some meaningful indication of the effect being estimated for the predictor in question but that's cumbersome and doesn't convey information about the &lt;em&gt;precision&lt;/em&gt; of the prediction, which is usually pretty important in a logistic regression model (on voting in particular). &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Also, the use of multiple asterisks to report &quot;levels&quot; of significance reinforces the misconception that p-values are some meaningful index of effect size (&quot;wow--that one has &lt;em&gt;3&lt;/em&gt; asterisks!!&quot;); for crying out loud, w/ N's of 10,000 to 20,000, completely trivial differences will be &quot;significant&quot; at p &amp;lt; .001 blah blah.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;There is absolutely no need to mystify in this way. The logistic regression model is an equation that can be used (through determinate calculation or better still simulation) to predict probability of an outcome conditional on specified values for predictors, subject to measurement error. So the researcher should &lt;em&gt;report&lt;/em&gt; what the impact of predictors of interest are on the probability of the outcome variable of interest, &amp;amp; associated CI, as measured in units the practical importance of which can readily be grasped.  To assure ready grasping, the results should be graphically displayed. Here, for example, the researcher could report that being a rural as opposed to an urban voter increases the likelihood of voting Republican, all else equal, by X pct points (I'm guessing around 17 in 2000; &quot;divide by 4&quot; is a reasonable heuristic) +/- x% at 0.95 level of confidence-- if that's something that is useful to know. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The reporting of pseudo R^2 is also a sign that the modeler is engaged in statistical ritual rather than any attempt to illuminate. There are scores of ways to compute &quot;pseudo R^2&quot;; one might complain that the one used here is not specified, but why bother? All are  next to meaningless.  The only reason anyone uses pseudo R^2 is that they or the reviewer who is torturing them learned (likely 25 or more yrs ago) that OLS linear regression is the holy  grail of statistics &amp;amp; thinks the only thing one is ever trying to figure out is &quot;variance explained.&quot; There are plenty of defensible ways to assess the adequacy of overall model fit for logistic analysis, and likelihood ratio conveys meaningful information for comparing models that reflect alternative hypotheses. King, G. How Not to Lie with Statistics. Am. J. Pol. Sci. 30, 666-687 (1986).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If you read a paper in which reporting is more or less confined to a table like this don't be confused, don't be intimidated, &amp;amp; definitely don't be impressed; instead be &lt;em&gt;angry&lt;/em&gt; &amp;amp; tell the researcher he or she is doing a lousy job (particularly if he or she is polluting your local intellectual environment w/ mysticism &amp;amp; awe--amazing how many completely mediocre thinkers trick smart people into thinking they know something just b/c they can produce a table that the latter can't understand). For smart, &amp;amp; temperate, expositions of these ideas, see King, G., Tomz, M. &amp;amp; Wittenberg., J. Making the Most of Statistical Analyses: Improving Interpretation and Presentation. Am. J. Pol. Sci. 44, 347-361 (2000); and Gelman, A., Pasarica, C. &amp;amp; Dodhia, R. Let's Practice What We Preach: Turning Tables into Graphs. Am. Stat. 56, 121-130 (2002).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="2347" LastEditorUserId="2347" LastEditorDisplayName="" LastEditDate="2011-03-10T19:22:19.677" LastActivityDate="2011-03-10T19:22:19.677" CommentCount="1" />
  <row Id="8132" PostTypeId="2" ParentId="8106" CreationDate="2011-03-10T15:34:15.527" Score="4" ViewCount="" Body="&lt;p&gt;Let me just stress the importance of what rolando2 and dmk38 both noted: significance is commonly misread, and there is a high risk of that happening with that tabular presentation of results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Paul Schrodt &lt;a href=&quot;http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1661045&quot; rel=&quot;nofollow&quot;&gt;recently&lt;/a&gt; offered a nice description of the issue:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Researchers find it nearly impossible to adhere to the correct interpretation of the significance test. The p-value tells you only the likelihood that you would get a result under the [usually] completely unrealistic conditions of the null hypothesis. Which is not what you want to know—you usually want to know the magnitude of the effect of an independent variable, given the data. That’s a Bayesian question, not a frequentist question. Instead we see—constantly—the p-value interpreted as if it gave the strength of association: this is the ubiquitous Mystical Cult of the Stars and P-Values which permeates our journals.(fn) This is not what the p-value says, nor will it ever.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;In my experience, this mistake is almost impossible to avoid: even very careful analysts who are fully aware of the problem will often switch modes when verbally discussing their results, even if they’ve avoided the problem in a written exposition. And let’s not even speculate on the thousands of hours and gallons of ink we’ve expended correcting this in graduate papers.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;(fn) The footnote also informs on another issue, mentioned by dmk38: “[the ubiquitous Mystical Cult of the Stars and P-Values] supplanted the earlier—and equally pervasive—Cult of the Highest R2, demolished… by &lt;a href=&quot;http://gking.harvard.edu/files/mist.pdf&quot; rel=&quot;nofollow&quot;&gt;King (1986)&lt;/a&gt;.”&lt;/p&gt;&#xA;" OwnerUserId="3582" LastActivityDate="2011-03-10T15:34:15.527" CommentCount="1" />
  <row Id="8133" PostTypeId="1" CreationDate="2011-03-10T15:41:13.017" Score="2" ViewCount="120" Body="&lt;p&gt;I read from a textbook that Gauss-Newton regression is also called 'artificial regression'.  Please give me an example, how does it work? And what's the relation with Newton's method? Thank you.&lt;/p&gt;&#xA;" OwnerUserId="3525" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-11T04:05:28.043" LastActivityDate="2011-03-11T04:05:28.043" Title="Intuitive explanation of Gauss-Newton regression" Tags="&lt;econometrics&gt;&lt;nonlinear-regression&gt;" CommentCount="5" FavoriteCount="1" />
  <row Id="8134" PostTypeId="2" ParentId="8096" CreationDate="2011-03-10T15:50:11.723" Score="1" ViewCount="" Body="&lt;ul&gt;&#xA;&lt;li&gt;Obtain the five-number summary for your sample.&lt;/li&gt;&#xA;&lt;li&gt;Calculate the standard error for your sample.&lt;/li&gt;&#xA;&lt;li&gt;Select a level of confidence, based on the z-score.&lt;/li&gt;&#xA;&lt;li&gt;Calculate the lower and upper bound of the interval.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If you are doing this by hand, use the formulae provided by any statistics handbook, such as Agresti and Finlay's &lt;em&gt;Statistics for the Social Sciences&lt;/em&gt;, to move through each stage. If you are using a computer solution such as R or Stata:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Obs = 28, Mean = 27.17, Std. Dev. = 8.82 (&lt;code&gt;su age&lt;/code&gt; in Stata)&lt;/li&gt;&#xA;&lt;li&gt;Std. Err. = 1.66 (&lt;code&gt;ci age&lt;/code&gt; in Stata)&lt;/li&gt;&#xA;&lt;li&gt;z = 1.96 at 95% confidence, z = 2.58 at 99% confidence (&lt;code&gt;level(95)&lt;/code&gt; or &lt;code&gt;level(99)&lt;/code&gt; in Stata) &lt;/li&gt;&#xA;&lt;li&gt;95% CI = [23.75, 30.60] and 99% CI = [22.55, 31.80] (truncated to two digits)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;To me, the key element of learning that you should get from this exercise is that the standard error will decrease with the square root of the sample size. The only way to avoid having to trade off precision against efficiency is to maximise sample size.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Agresti and Finlay cover this in detail in Chapter 5 of their handbook, starting at page 126. Previous chapters provide the formulae for calculating the mean, variance, square root of variance (standard deviation) and standard error of the mean (SEM).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you were dealing with proportions, you would be calculating the standard error slightly differently, but the underlying logic would remain: √N, the square root of the sample size, would stay critical in minimising the standard error.&lt;/p&gt;&#xA;" OwnerUserId="3582" LastActivityDate="2011-03-10T15:50:11.723" />
  <row Id="8135" PostTypeId="1" CreationDate="2011-03-10T16:33:50.490" Score="4" ViewCount="240" Body="&lt;p&gt;I am running a large OLS regression where all the independent variables (around 400) are dummy variables. If all are included, there is perfect multicollinearity (the dummy variable trap), so I have to omit one of the variables before running the regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My first question is, which variable should be omitted? I have read that it is better to omit a variable that is present in many of the observations rather than one that is present in only a few (e.g. if almost all observations are &quot;male&quot; or &quot;female&quot; and just a few are &quot;unknown&quot;, omit either &quot;male&quot; or &quot;female&quot;). Is this justified?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After running the regression with a variable omitted, I am able to estimate the coefficient value of the omitted variable because I know that the overall mean of all my independent variables should be 0. So I use this fact to shift the coefficient values for all the included variables, and get an estimate for the omitted variable. My next question is whether there is some similar technique that can be used to estimate the standard error for the coefficient value of the omitted variable. As it is I have to re-run the regression omitting a different variable (and including the variable I had omitted in the first regression) in order to acquire a standard error estimate for the coefficient of the originally omitted variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, I notice that the coefficient estimates I get (after re-centering around zero) vary slightly depending on which variable is omitted. In theory, would it be better to run several regressions, each omitting a different variable, and then average the coefficient estimates from all the regressions?&lt;/p&gt;&#xA;" OwnerUserId="1090" LastActivityDate="2011-03-11T18:02:32.733" Title="Dummy variable trap issues" Tags="&lt;categorical-data&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="8136" PostTypeId="2" ParentId="8052" CreationDate="2011-03-10T17:37:32.670" Score="3" ViewCount="" Body="&lt;p&gt;Shouldn't you favour &lt;a href=&quot;http://en.wikipedia.org/wiki/Fisher%27s_exact_test&quot; rel=&quot;nofollow&quot;&gt;Fisher's exact test&lt;/a&gt; on a 2x2 contingency table? The advantages of the Chi-squared test would be preserved, with the additional advantages of an exact test. Based on a few handbook readings, I believe that Fisher's exact test is recommended chiefly with low cell counts, but it is also often also recommended for 2x2 tables.&lt;/p&gt;&#xA;" OwnerUserId="3582" LastActivityDate="2011-03-10T17:37:32.670" CommentCount="1" />
  <row Id="8137" PostTypeId="1" AcceptedAnswerId="8235" CreationDate="2011-03-10T17:52:10.160" Score="2" ViewCount="383" Body="&lt;p&gt;I have a boxplot output in R using ggplot2:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;p &amp;lt;- ggplot(data, aes(y = age, x = group))&#xA;p &amp;lt;- p + geom_boxplot()&#xA;p &amp;lt;- p + scale_x_discrete(name= &quot;Group&quot;,)&#xA;p &amp;lt;- p + scale_y_continuous(name= &quot;Age&quot;)&#xA;p&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/fxI8B.png&quot; alt=&quot;ggplot2 boxplots&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to add horisontal lines like on common boxplot (and to change vertical line style if possible):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;boxplot(age~group,data=data,names=c('1','2'),ylab=&quot;Age&quot;, xlab=&quot;Group&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/hvP44.png&quot; alt=&quot;common boxplots&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How I could do this using ggplot2?&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="3376" LastActivityDate="2011-03-13T19:22:24.783" Title="How to add horizontal lines to ggplot2 boxplot?" Tags="&lt;r&gt;&lt;boxplot&gt;&lt;ggplot2&gt;" AnswerCount="1" CommentCount="7" FavoriteCount="1" />
  <row Id="8139" PostTypeId="2" ParentId="7982" CreationDate="2011-03-10T21:29:22.377" Score="2" ViewCount="" Body="&lt;p&gt;Assuming data are considered missing completely at random (cf. @whuber's comment), using an ensemble learning technique as described in the following paper might be interesting to try:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Polikar, R. et al. (2010).&#xA;  &lt;a href=&quot;http://www.cs.man.ac.uk/~gbrown/research/polikar10missing.pdf&quot; rel=&quot;nofollow&quot;&gt;Learn++.MF: A random subspace&#xA;  approach for the missing feature&#xA;  problem&lt;/a&gt;. &lt;em&gt;Pattern Recognition&lt;/em&gt;,&#xA;  &lt;em&gt;43(11)&lt;/em&gt;, 3817-3832.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The general idea is to train multiple classifiers on a subset of the variables that compose your dataset (like in Random Forests), but to use only the classifiers trained with the non-missing features for building the classification rule. Be sure to check what the authors call the &quot;distributed redundancy&quot; assumption (p. 3 in the preprint linked above), that is there must be some equally balanced redundancy in your features set.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-03-11T07:37:13.680" LastActivityDate="2011-03-11T07:37:13.680" />
  <row Id="8140" PostTypeId="2" ParentId="7982" CreationDate="2011-03-10T21:50:11.123" Score="1" ViewCount="" Body="&lt;p&gt;If the features in the subset are random you can still impute values. However, if you have that much missing data, I would think twice about whether or not you really have enough valid data to do any kind of analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The multiple imputation FAQ page ----&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.stat.psu.edu/~jls/mifaq.html&quot; rel=&quot;nofollow&quot;&gt;http://www.stat.psu.edu/~jls/mifaq.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="3489" LastActivityDate="2011-03-10T21:50:11.123" CommentCount="3" />
  <row Id="8141" PostTypeId="2" ParentId="8135" CreationDate="2011-03-10T22:14:10.927" Score="2" ViewCount="" Body="&lt;p&gt;James, first of all why regression analysis, but not &lt;strong&gt;ANOVA&lt;/strong&gt; (there are many specialists in this kind of analysis that could help you)? The &lt;em&gt;pros&lt;/em&gt; for ANOVA is that all you actually interested in are differences in the means of different groups described by combinations of dummy variables (unique categories, or profiles). Well, if you do study impacts of each of categorical variable you include, you may run regression as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think the type of the data you do have here is described in the sense of &lt;em&gt;conjoint analysis&lt;/em&gt;: many attributes of the object (gender, age, education, etc.) each having several categories, thus you omit the whole largest profile, not just one dummy variable. A common practise is to code the categories within the attribute as follows (this &lt;a href=&quot;http://dissertations.ub.rug.nl/FILES/faculties/eco/1999/m.e.haaijer/c2.pdf&quot; rel=&quot;nofollow&quot;&gt;link&lt;/a&gt; may be useful, you probably do not do conjoint analysis here, but coding is similar): suppose you have $n$ categories (three, as you suggested, male, female, unknown) then, first two are coded as usual you do include two dummies (male, female), giving $(1, 0)$ if male, $(0, 1)$ if female, and $(-1, -1)$ if unknown. In this way the results indeed will be placed around intercept term. You may code in a different way, however, but will lose the mentioned interpretation advantage. To sum up, you drop one category from   &lt;strong&gt;each category&lt;/strong&gt;, and code your observations in the described way. You do include intercept term also. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Well to omit the largest profile's categories seems good for me, though not so important, at least it is not empty I think. Since you code the variables in specific manner, joint statistical significance of included dummy variables (both male female, could be tested by F test) imply the significance of the omitted one. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It may happen that the results slightly different, but may be it is the wrong coding that influence this? &lt;/p&gt;&#xA;" OwnerUserId="2645" LastActivityDate="2011-03-10T22:14:10.927" CommentCount="1" />
  <row Id="8142" PostTypeId="1" CreationDate="2011-03-11T00:45:27.277" Score="1" ViewCount="40" Body="&lt;p&gt;Is there an analytic form for the Hellinger distance between von Mises distributions?&lt;/p&gt;&#xA;" OwnerUserId="3595" LastActivityDate="2011-03-11T00:45:27.277" Title="Is there an analytic form for the Hellinger distance between von Mises distributions?" Tags="&lt;distributions&gt;&lt;distance-functions&gt;" CommentCount="1" />
  <row Id="8143" PostTypeId="1" CreationDate="2011-03-11T03:01:15.340" Score="0" ViewCount="97" Body="&lt;p&gt;Hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If, for example, I have &lt;code&gt;0011&lt;/code&gt; as a set of known bits $x$, how do I determine the probability that a sequence of randomly generated bits is equal to $x$?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for the help! I'm sure this is a dumb question, but probability has always been my weakness (which my algorithms class is highlighting!).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;EDIT:&lt;/em&gt; This is what I should've asked:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do you find the probability of randomly getting &lt;code&gt;0011&lt;/code&gt; on the left-side of an 8-bit sequence?&lt;/p&gt;&#xA;" OwnerUserId="3666" LastEditorUserId="3666" LastEditorDisplayName="" LastEditDate="2011-03-11T16:27:33.847" LastActivityDate="2011-03-11T16:27:33.847" Title="How to calculate the probability of a random sequence of bits being a specific sequence?" Tags="&lt;probability&gt;" CommentCount="6" />
  <row Id="8144" PostTypeId="2" ParentId="7676" CreationDate="2011-03-11T05:25:46.947" Score="1" ViewCount="" Body="&lt;p&gt;Hey Mike,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While you didn't say anything about the actual condtions (experimental treatment), think of the following example: you measure the weight of 40 people at some point during the day, then set them the task to drink (as much as possible of) 5 quarts of water (experimental manipulation), and them weigh them again (hinting at &quot;yes&quot; being the answer to your final question...).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, the values for the means of each group will, pre and post treatment, very likely overlap (given that, unless you select people with the same pre-treatment weight, the general population shows a great variability). The mean of the difference between post and pre (post - pre), on the other hand, is likely to be significantly different from (i.e. greater than) 0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words, without knowing a little more about your conditions (e.g. how does the metric change between conditions? do you expect it to show signs of stability?) and also about the variability on that measure across the population (if you think of the measure as a random variable, each subject has his/her own probability distribution of getting a certain score under a certain condition; and the measurement error might be very small compared to the reliable difference of each participants measure compared to the population mean).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does any of this help?&lt;/p&gt;&#xA;" OwnerUserId="3667" LastActivityDate="2011-03-11T05:25:46.947" />
  <row Id="8145" PostTypeId="1" AcceptedAnswerId="8149" CreationDate="2011-03-11T09:46:09.730" Score="10" ViewCount="123" Body="&lt;p&gt;I solve Rubik's cubes as a hobby.  I record the time it took me to solve the cube using some software, and so now I have data from thousands of solves.  The data is basically a long list of numbers representing the time each sequential solve took (e.g. 22.11, 20.66, 21.00, 18.74, ...)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The time it takes me to solve the cube naturally varies somewhat from solve to solve, so there are good solves and bad solves.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to know whether I &quot;get hot&quot; - whether the good solves come in streaks.  For example, if I've just had a few consecutive good solves, is it more likely that my next solve will be good?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What sort of analysis would be appropriate?  I can think of a few specific things to do, for example treating the solves as a Markov process and seeing how well one solve predicts the next and comparing to random data, seeing how long the longest streaks of consecutive solves below the median for the last 100 are and comparing to what would be expected in random data, etc.  I am not sure how insightful these tests would be, and wonder whether there are some well-developed approaches to this sort of problem.&lt;/p&gt;&#xA;" OwnerUserId="2665" LastActivityDate="2011-03-12T05:40:11.007" Title="How do you tell whether good performances come in streaks?" Tags="&lt;probability&gt;" AnswerCount="3" />
  <row Id="8146" PostTypeId="1" CreationDate="2011-03-11T10:25:28.287" Score="1" ViewCount="66" Body="&lt;p&gt;Hello,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does any body know of a Java implementation of McNemar's Test?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regards,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Diptendu Dutta&lt;/p&gt;&#xA;" OwnerUserId="3669" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-11T13:27:02.053" LastActivityDate="2011-03-11T23:19:14.227" Title="McNemar's test implementation in Java" Tags="&lt;java&gt;" AnswerCount="2" />
  <row Id="8147" PostTypeId="1" CreationDate="2011-03-11T10:29:28.877" Score="1" ViewCount="71" Body="&lt;p&gt;I have almost the same question as:&#xA;&lt;a href=&quot;http://stats.stackexchange.com/questions/5347/how-can-i-efficiently-model-the-sum-of-bernoulli-random-variables&quot;&gt;How can I efficiently model the sum of Bernoulli random variables?&lt;/a&gt;&#xA;But:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(1) The number of random variables for summation is ~ N=20 (case 1) or N=90 (case 2).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(2) $p_i$ ~ 0.13 (case 1)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(3) The precision of the model based on Poisson law is not enough.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(4) We need that our approx would be the good enough to model partial sums like these as well: $\sum_{i=k,N}{X_i}$, ( $k=1,N$ )&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(5) We have empirical data for every $X_i$. The diagram shows that there is almost linear dependence for $Pr(X_i=1)$ for i=1,6 and then almost constant function or some small linear dependence (for i=7,20 ).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Really I'm not sure in (3) since the Le Cam's inequality looks like very general...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What class of model could we try?&lt;/p&gt;&#xA;" OwnerUserId="3670" LastActivityDate="2011-04-18T21:04:11.073" Title="How can I efficiently approximate the sum of Bernoulli random variables for any number of summands in partial sum?" Tags="&lt;distributions&gt;&lt;modeling&gt;" AnswerCount="1" />
  <row Id="8148" PostTypeId="1" CreationDate="2011-03-11T10:34:36.383" Score="11" ViewCount="268" Body="&lt;p&gt;Apologies for the rudimentary question, I am new to this form of analysis and have a very limited understanding of the principles so far.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was just wondering if many of the parametric assumptions for multivariate/univariate tests apply for Cluster analysis? Many of the sources of information I have read regarding cluster analysis fail to specify any assumptions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am particularly interested in the assumption of independence of observations. My understanding is that violation of this assumption (in ANOVA and MAVOVA for example) is serious because it influences estimates of error. From my reading so far, it seems that cluster analysis is largely a descriptive technique (that only involves statistical inference in certain specified cases). Accordingly, are assumptions such as independence and normally distributed data required?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any recommendations of texts that discuss this issue would be greatly appreciated.&#xA;Many Thanks.  &lt;/p&gt;&#xA;" OwnerUserId="3619" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-11T13:26:42.053" LastActivityDate="2011-06-02T10:51:50.320" Title="Assumptions of cluster analysis" Tags="&lt;clustering&gt;&lt;assumptions&gt;" AnswerCount="4" FavoriteCount="4" />
  <row Id="8149" PostTypeId="2" ParentId="8145" CreationDate="2011-03-11T10:41:21.237" Score="7" ViewCount="" Body="&lt;p&gt;The &lt;a href=&quot;http://en.wikipedia.org/wiki/Wald%E2%80%93Wolfowitz_runs_test&quot; rel=&quot;nofollow&quot;&gt;Wald-Wolfowitz Runs Test&lt;/a&gt; seems to be a possible candidate, where a &quot;run&quot; is what you called a &quot;streak&quot;. It requires dichotomous data, so you'd have to label each solve as &quot;bad&quot; vs. &quot;good&quot; according to some threshold - like the median time as you suggested. The null hypothesis is that &quot;good&quot; and &quot;bad&quot; solves alternate randomly. A one-sided alternative hypothesis corresponding to your intuition is that &quot;good&quot; solves clump together in long streaks, implying that there are fewer runs than expected with random data. Test statistic is the number of runs. In R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; N      &amp;lt;- 200                          # number of solves&#xA;&amp;gt; DV     &amp;lt;- round(runif(N, 15, 30), 1)   # simulate some uniform data&#xA;&amp;gt; thresh &amp;lt;- median(DV)                   # threshold for binary classification&#xA;&#xA;# do the binary classification&#xA;&amp;gt; DVfac &amp;lt;- cut(DV, breaks=c(-Inf, thresh, Inf), labels=c(&quot;good&quot;, &quot;bad&quot;))&#xA;&amp;gt; Nj    &amp;lt;- table(DVfac)                  # number of &quot;good&quot; and &quot;bad&quot; solves&#xA;&amp;gt; n1    &amp;lt;- Nj[1]                         # number of &quot;good&quot; solves&#xA;&amp;gt; n2    &amp;lt;- Nj[2]                         # number of &quot;bad&quot; solves&#xA;&amp;gt; (runs &amp;lt;- rle(as.character(DVfac)))     # analysis of runs&#xA;Run Length Encoding&#xA;lengths: int [1:92] 2 1 2 4 1 4 3 4 2 5 ...&#xA;values : chr [1:92] &quot;bad&quot; &quot;good&quot; &quot;bad&quot; &quot;good&quot; &quot;bad&quot; &quot;good&quot; &quot;bad&quot; ...&#xA;&#xA;&amp;gt; (nRuns &amp;lt;- length(runs$lengths))        # test statistic: observed number of runs&#xA;[1] 92&#xA;&#xA;# theoretical maximum of runs for given n1, n2&#xA;&amp;gt; (rMax &amp;lt;- ifelse(n1 == n2, N, 2*min(n1, n2) + 1))&#xA;199 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When you only have a few observations, you can calculate the exact probabilities for each number of runs under the null hypothesis. Otherwise, the distribution of &quot;number of runs&quot; can be approximated by a standard normal distribution.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; (muR  &amp;lt;- 1 + ((2*n1*n2) / N))                     # expected value&#xA;100.99 &#xA;&#xA;&amp;gt; varR  &amp;lt;- (2*n1*n2*(2*n1*n2 - N)) / (N^2 * (N-1))  # theoretical variance&#xA;&amp;gt; rZ    &amp;lt;- (nRuns-muR) / sqrt(varR)                 # z-score&#xA;&amp;gt; (pVal &amp;lt;- pnorm(rZ, mean=0, sd=1))                 # one-sided p-value&#xA;0.1012055&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The p-value is for the one-sided alternative hypothesis that &quot;good&quot; solves come in streaks.&lt;/p&gt;&#xA;" OwnerUserId="1909" LastEditorUserId="1909" LastEditorDisplayName="" LastEditDate="2011-03-11T14:42:16.717" LastActivityDate="2011-03-11T14:42:16.717" CommentCount="1" />
  <row Id="8150" PostTypeId="1" CreationDate="2011-03-11T10:44:36.757" Score="2" ViewCount="89" Body="&lt;p&gt;I have a prospective study  with no data about estimated results, that could be used to get required sample size. Data looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;caseID;groupID;value,result&#xA;1;1;12.3;0&#xA;2;1;15.6;1&#xA;3;2;11.3;0&#xA;4;2;13.4;1&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is it possible to determine how much observations should be made to complete the study after adding new portion of data? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need both descriptive statistics ( % of cases have sign X, &lt;em&gt;result&lt;/em&gt; in example ) and comparative (2-4 groups compared by continuous variable,&lt;em&gt;value&lt;/em&gt; in example). Probability of type I  error ($\alpha$) is 10%    and type II ($\beta$) - 5% (&lt;em&gt;for example&lt;/em&gt;) Statistical significance and power - 0,9 and 0,95. The best solution is to have a grid like this for making desision to stop or continue the study:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;a;b;n&#xA;5;5;30&#xA;10;5;28&#xA;5;10;29&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Where a is $\alpha$ , b is $\beta$ and n is the size of sample required for study with this $\alpha$ and $\beta$ (Type I and type 2 error probability).&#xA;My data is imported to R. Is it possible to perform some calculations to get?:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Number observations required to complete study&lt;/li&gt;&#xA;&lt;li&gt;Required number of observations for each group&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;So I need to estimate sample size based on actual data to know it is time to finish the study.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The question is: &#xA;&lt;strong&gt;How to do this in R, if it is possible?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestions are welcomed.&lt;/p&gt;&#xA;" OwnerUserId="3376" LastEditorUserId="3376" LastEditorDisplayName="" LastEditDate="2011-03-15T19:46:01.623" LastActivityDate="2011-03-15T19:46:01.623" Title="Dynamic estimation of required sample size in R" Tags="&lt;r&gt;&lt;estimation&gt;&lt;sample-size&gt;" CommentCount="3" FavoriteCount="1" />
  <row Id="8151" PostTypeId="1" CreationDate="2011-03-11T11:42:59.117" Score="2" ViewCount="119" Body="&lt;p&gt;Is there a way to easily create factors based on quantiles of selected variables in a dataframe? Say, in a datatset D, I have variables V1 to V10, which are all numeric. I would like to create dummies for V7 to V10 based on their respective quantiles.&lt;/p&gt;&#xA;" OwnerUserId="3671" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-11T13:25:51.697" LastActivityDate="2011-03-11T13:38:19.330" Title="Quantile transformation in R" Tags="&lt;r&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="8152" PostTypeId="1" AcceptedAnswerId="8153" CreationDate="2011-03-11T12:06:54.577" Score="2" ViewCount="108" Body="&lt;p&gt;I have a few questions regarding multiple imputation for nested data. &#xA;Context: I have repeated measures (4 times) from a survey and these are clustered in workplaces (205 workplaces). There are about 180 items on this survey.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;q1. Is it possible to take both the repeated measures and the workplace clustering into consideration or do i have to decide for one of the two?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;q2. If i can only take into consideration one of the two clusterings (repeated measures vs workplace) which one would you recommend&lt;/p&gt;&#xA;&#xA;&lt;p&gt;q3. I have about 10000 observations and about 400 of them have missing values for the workplace. What would you recommend to do in this case? (also i should mention that the 205 workplaces are nested in 17 Organizations - For the moment i use general categories based on the organization: e.g. Organization1-Unclassified). Is there a meaningful way to actually impute these categories? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;q4. Would you recommend to use all 180 items for imputation or the items that i intend to use in each of my models? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I use R for analysis and it would be greatly appreciated if you can recommend any packages for multiple imputation for clustered data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance&lt;/p&gt;&#xA;" OwnerUserId="1871" LastEditorUserId="1871" LastEditorDisplayName="" LastEditDate="2011-03-11T12:40:58.963" LastActivityDate="2011-03-11T13:50:53.047" Title="Multiple imputation for clustered data" Tags="&lt;r&gt;&lt;multilevel-analysis&gt;&lt;longitudinal&gt;&lt;data-imputation&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="8153" PostTypeId="2" ParentId="8152" CreationDate="2011-03-11T13:24:39.780" Score="2" ViewCount="" Body="&lt;p&gt;Take a look at the &lt;a href=&quot;http://gking.harvard.edu/amelia/&quot; rel=&quot;nofollow&quot;&gt;Amelia II&lt;/a&gt; package, by Honaker, King and Blackwell.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Amelia II &quot;multiply imputes&quot; missing data in a single cross-section (such as a survey), from a time series (like variables collected for each year in a country), or from a time-series-cross-sectional data set (such as collected by years for each of several countries).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;q1. Yes, it is possible with the above package.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;q3. I guess it would be possible to impute workplace using a more general imputation method. (&lt;a href=&quot;http://web.inter.nl.net/users/S.van.Buuren/mi/hmtl/mice.htm&quot; rel=&quot;nofollow&quot;&gt;MICE&lt;/a&gt; could help).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;q4. As a general rule, do not throw information out. The imputation model, at a minimun, should include all covariates in all your models. But if extra information help predicting the missing data, include it! &lt;/p&gt;&#xA;" OwnerUserId="375" LastEditorUserId="375" LastEditorDisplayName="" LastEditDate="2011-03-11T13:50:53.047" LastActivityDate="2011-03-11T13:50:53.047" />
  <row Id="8154" PostTypeId="2" ParentId="8151" CreationDate="2011-03-11T13:38:19.330" Score="1" ViewCount="" Body="&lt;p&gt;To get the factors, use:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cut(dataset, quantile(dataset))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;From the help:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;## Default S3 method:&#xA;cut(x, breaks)&#xA;&#xA;x          a numeric vector which is to be converted to a factor by cutting.&#xA;breaks   either a numeric vector of two or more cut points or a single number (greater than or equal to 2) giving the number of intervals into which x is to be cut.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1351" LastActivityDate="2011-03-11T13:38:19.330" CommentCount="2" />
  <row Id="8155" PostTypeId="2" ParentId="8148" CreationDate="2011-03-11T14:17:04.913" Score="1" ViewCount="" Body="&lt;p&gt;Cluster analysis does not involve hypothesis testing per se, but is really just a collection of different similarity algorithms for exploratory analysis. You can force hypothesis testing somewhat but the results are often inconsistent, since cluster changes are very sensitive to changes in parameters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://support.sas.com/documentation/cdl/en/statug/63033/HTML/default/viewer.htm#statug_introclus_sect010.htm&quot; rel=&quot;nofollow&quot;&gt;http://support.sas.com/documentation/cdl/en/statug/63033/HTML/default/viewer.htm#statug_introclus_sect010.htm&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="3489" LastActivityDate="2011-03-11T14:17:04.913" />
  <row Id="8156" PostTypeId="1" CreationDate="2011-03-11T15:08:43.463" Score="1" ViewCount="102" Body="&lt;p&gt;I work with a lot of bar charts. In particular, these bar charts are of basecalls along segments on the human genome.  Each point along the x-axis is one of the four nitrogenous bases(A,C,T,G) that compose DNA and the y-axis is essentially how many times a base was able to be &quot;called&quot; (or recognized by a sequencer machine, so as to sequence the genome, which is simply determining the identity of each base along the genome).&lt;br&gt;&#xA;Many of these bar charts display roughly linear dropoffs (when the machines aren't able to get sufficient read depth) that fall to 0 or (almost-0) from plateau-like regions. Is there a straightforward algorithm for assigning these plots a score that reflects this tendency?  Is stdev a good place to start? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm a programmer but not much of a statistician.&lt;/p&gt;&#xA;" OwnerUserId="3672" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-12T10:45:54.777" LastActivityDate="2011-03-12T10:45:54.777" Title="Assessing DNA sequencing quality" Tags="&lt;histogram&gt;&lt;bioinformatics&gt;" CommentCount="17" FavoriteCount="1" />
  <row Id="8157" PostTypeId="1" CreationDate="2011-03-11T15:32:55.813" Score="6" ViewCount="103" Body="&lt;p&gt;I need to create random vectors of real numbers a_i satisfying the following constraints:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;abs(a_i) &amp;lt; c_i;      &#xA;sum(a_i)&amp;lt; A;        # sum of elements smaller than A&#xA;sum(b_i * a_i) &amp;lt; B; # weighted sum is smaller than B &#xA;aT*A*a &amp;lt; D          # quadratic multiplication with A smaller than D&#xA;&#xA;where c_i, b_i, A, B, D are constants.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What would be the typical algorithm to generate efficiently this kind of vector?&lt;/p&gt;&#xA;" OwnerUserId="3673" LastEditorUserId="3673" LastEditorDisplayName="" LastEditDate="2011-03-15T18:07:51.207" LastActivityDate="2011-03-17T02:56:04.293" Title="Generating random vectors with constraints" Tags="&lt;random-generation&gt;" AnswerCount="1" CommentCount="8" FavoriteCount="1" />
  <row Id="8158" PostTypeId="2" ParentId="8027" CreationDate="2011-03-11T15:55:06.620" Score="0" ViewCount="" Body="&lt;p&gt;Thanks mqb - I had no idea I was using so many accounts... :(&#xA;Why was my Test Statistic page closed. the reason i uploaded the PDF was to provide examples.... i just needed one explained but giving the reader a choice is better right? and test statistics are very popular compared to pivot functions so i dont get why it was closed... I was told it was a sensible question and that I should post it here....&lt;/p&gt;&#xA;&#xA;&lt;p&gt;mpiktas - i dont get what was wrong? Whats there not to understand above? The question is in the first line.... My chain of thinking is paragraph 2&lt;/p&gt;&#xA;" OwnerUserId="3676" LastActivityDate="2011-03-11T15:55:06.620" CommentCount="1" />
  <row Id="8159" PostTypeId="1" CreationDate="2011-03-11T16:57:52.407" Score="1" ViewCount="105" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;Possible Duplicate:&lt;/strong&gt;&lt;br&gt;&#xA;  &lt;a href=&quot;http://stats.stackexchange.com/questions/5026/what-is-the-difference-between-data-mining-statistics-machine-learning-and-ai&quot;&gt;What is the difference between data mining, statistics, machine learning and AI?&lt;/a&gt;  &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&#xA;&#xA;&lt;p&gt;How to compare &#xA;machine learning vs. data mining?&#xA;data mining vs. statistical analysis?&#xA;machine learning vs. statistical analysis?&lt;/p&gt;&#xA;" OwnerUserId="3026" LastActivityDate="2011-03-11T16:57:52.407" ClosedDate="2011-03-11T22:18:47.230" Title="The relationship between machine learning, data mining and statistical analysis?" Tags="&lt;machine-learning&gt;&lt;multivariate-analysis&gt;&lt;data-mining&gt;" CommentCount="2" />
  <row Id="8160" PostTypeId="1" AcceptedAnswerId="8173" CreationDate="2011-03-11T17:23:08.010" Score="4" ViewCount="146" Body="&lt;p&gt;I have 3,000 observations (administrative communities) characterized by five variables. Four of them work in the direction 'the more, the worse' and one goes in the opposite.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd like to create one score or an ordered list of these observations that will best take into account all of those five variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have tried clustering using MCLUST package in R, and it gives some meaningful results but it's hard to decide about the ordering of observations on the basis of cluster membership.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My second attempt was to run PCA and extract the first component, which is closer to what I'd like to get.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What other solutions (R- or Stata-based preferably) could I use to deal with this problem?&lt;/p&gt;&#xA;" OwnerUserId="22" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-03-12T06:15:11.397" LastActivityDate="2011-03-30T18:54:57.467" Title="How to create one score from a mixed set of positive and negative variables?" Tags="&lt;clustering&gt;&lt;pca&gt;&lt;composite&gt;" AnswerCount="4" CommentCount="2" />
  <row Id="8161" PostTypeId="1" AcceptedAnswerId="8200" CreationDate="2011-03-11T17:25:48.310" Score="1" ViewCount="81" Body="&lt;p&gt;Suppose I have two I(1) time series X and Y, and I want to know whether X and Y are &quot;related&quot; (for some definition of &quot;related&quot;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The standard cointegration approach defines relationship as cointegration, and says that X and Y are cointegrated if some linear combination of X and Y is stationary. To test whether X and Y are cointegrated, you perform a regression on X and Y, and test for stationarity of the residual errors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems to me like another approach might be to difference the I(1) time series X and Y, to get new I(0) time series X' and Y', and to use a standard linear regression relationship test on X' and Y' (i.e., perform a regression to get Y' = aX' + b, and use a t-test to see whether a is significantly non-zero). You could then define X and Y to be related if X' and Y' pass this test.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this second approach valid, or do you get spurious relationships? What's the difference between this approach and the cointegration approach, or what are the advantages of the cointegration definition? &lt;/p&gt;&#xA;" OwnerUserId="1106" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-12T10:53:24.833" LastActivityDate="2011-03-12T12:25:42.250" Title="Testing two I(1) vectors for a relationship" Tags="&lt;time-series&gt;&lt;correlation&gt;&lt;cointegration&gt;&lt;stationarity&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="8162" PostTypeId="2" ParentId="8107" CreationDate="2011-03-11T17:42:32.797" Score="0" ViewCount="" Body="&lt;p&gt;Are you asking about these results in particular or the Breusch-Pagan test more generally? For these particular tests, see @mpiktas's answer. Broadly, the BP test asks whether the squared residuals from a regression can be predicted using some set of predictors. These predictors may be the same as those from the original regression. The White test version of the BP test includes all the predictors from the original regression, plus their squares and interactions in a regression against the squared residuals. If the squared residuals are predictable using some set of covariates, then the estimated squared residuals and thus the variances of the residuals (which follows because the mean of the residuals is 0) appear to vary across units, which is the definition of heteroskedasticity or non-constant variance, the phenomenon that the BP test considers.&lt;/p&gt;&#xA;" OwnerUserId="401" LastActivityDate="2011-03-11T17:42:32.797" />
  <row Id="8163" PostTypeId="2" ParentId="8088" CreationDate="2011-03-11T17:47:46.867" Score="1" ViewCount="" Body="&lt;p&gt;Why are you worried about multicolinearity? The only reason that we need this assumption in regression is to ensure that we get unique estimates. Multicolinearity only matters for estimation when it is perfect---when one variable is an exact linear combination of the others.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your experimentally-manipulated variables were randomly assigned, then their correlations with the observed predictors as well as unobserved factors should be (roughly) 0; it is this assumption that helps you get unbiased estimates.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That said, non-perfect multicolinearity can make your standard errors larger, but only on those variables that experience the multicolinearity issue. In your context, the standard errors of the coefficients on your experimental variables should not be impacted.&lt;/p&gt;&#xA;" OwnerUserId="401" LastActivityDate="2011-03-11T17:47:46.867" />
  <row Id="8164" PostTypeId="1" AcceptedAnswerId="8175" CreationDate="2011-03-11T17:50:19.530" Score="4" ViewCount="148" Body="&lt;p&gt;In R, made a &lt;a href=&quot;http://biocluster.ucr.edu/~alevchuk/cs235-proj/fig-5.png&quot; rel=&quot;nofollow&quot;&gt;plot with a 3M points&lt;/a&gt; and saved it as PNG. It took a few hours and I would like to avoid re-drawing all the points.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I generate a new plot that has this PNG as a background?&lt;/p&gt;&#xA;" OwnerUserId="1794" LastActivityDate="2011-03-11T20:07:36.363" Title="In R, how to plot with a png as background?" Tags="&lt;r&gt;&lt;large-data&gt;&lt;figures&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="1" />
  <row Id="8165" PostTypeId="2" ParentId="8135" CreationDate="2011-03-11T18:02:32.733" Score="1" ViewCount="" Body="&lt;p&gt;You should get the &quot;same&quot; estimates no matter which variable you omit; the &lt;em&gt;coefficients&lt;/em&gt; may be different, but the estimates of particular quantities or &lt;em&gt;expectations&lt;/em&gt; should be the same across all the models. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a simple case, let $x_i=1$ for men and 0 for women. Then, we have the model:&#xA;$$\begin{align*} &#xD;&#xA;E[y_i \mid x_i] &amp;amp;= x_iE[y_i \mid x_i = 1] + (1 - x_i)E[y_i \mid x_i = 0] + \epsilon_i \\&#xD;&#xA;&amp;amp;= E[y_i \mid x_i=0] + \left[E[y_i \mid x_i= 1] - E[y_i \mid x_i=0]\right]x_i + \epsilon_i \\&#xD;&#xA;&amp;amp;= \beta_0 + \beta_1 x_i + \epsilon_i.&#xD;&#xA;\end{align*}$$&#xA;Now, let $z_i=1$ for women. Then&#xA;$$\begin{align*} &#xD;&#xA;E[y_i \mid z_i] &amp;amp;= z_iE[y_i \mid z_i = 1] + (1 - z_i)E[y_i \mid z_i = 0] + \epsilon_i \\&#xD;&#xA;&amp;amp;= E[y_i \mid z_i=0] + \left[E[y_i \mid z_i= 1] - E[y_i \mid z_i=0]\right]z_i + \epsilon_i \\&#xD;&#xA;&amp;amp;= \gamma_0 + \gamma_1 z_i + \epsilon_i.&#xD;&#xA;\end{align*}$$&#xA;The expected value of $y$ for women is $\beta_0$ and also $\gamma_0 + \gamma_1$. For men, it is $\beta_0 + \beta_1$ and $\gamma_0$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;These results show how the coefficients from the two models are related. For example, $\beta_1 = -\gamma_1$. A similar exercise using your data should show that the &quot;different&quot; coefficients that you get are just sums and differences of one another.&lt;/p&gt;&#xA;" OwnerUserId="401" LastActivityDate="2011-03-11T18:02:32.733" />
  <row Id="8166" PostTypeId="1" CreationDate="2011-03-11T18:17:02.197" Score="5" ViewCount="66" Body="&lt;p&gt;The UN has a convention against corruption (UNCAC) that has been signed by some 140 countries and ratified by most (&lt;a href=&quot;http://www.unodc.org/unodc/en/treaties/CAC/signatories.html&quot; rel=&quot;nofollow&quot;&gt;link&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Transparency International publishes an annual report on corruption. They have data for most countries for the period 2000-2010, each country is given a score between zero and ten where ten is best (lowest level of corruption). There is usually a slight trend in the corruption score, indicating that there is some inertia in the level of corruption (&lt;a href=&quot;http://www.transparency.org/&quot; rel=&quot;nofollow&quot;&gt;link&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;I want to test whether ratifying the convention has any positive effect on the level of corruption in the country.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My initial idea is to calculate the mean level of corruption 3 years before and after signing the treaty and using a students paired t-test to see if latter mean is significantly larger than the former.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another approach, that I would know less about, is to use some kind of event model where dummy variables are used for the years after the convention is ratified.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are your thoughts on this? Would the first approach work, is there anything I have overlooked? All feedback is appreciated.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Notes&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;I should note that this is a minor part of a course term paper. We weren't  supposed to do anything quantitative, just a write-up on a topic of our choice. However I want to take a quick look at the statistics since data is readily available and I prefer to look at numbers if possible. Hence, I don't need the ideal PhD-level statistical model, just something quick and simple.&lt;/p&gt;&#xA;" OwnerUserId="3182" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-12T10:38:19.107" LastActivityDate="2011-03-12T10:38:19.107" Title="Test for the effect of UN ratification on corruption in a set of countries" Tags="&lt;time-series&gt;&lt;statistical-significance&gt;&lt;mean&gt;&lt;panel-data&gt;&lt;students-t&gt;" AnswerCount="2" />
  <row Id="8167" PostTypeId="1" AcceptedAnswerId="8174" CreationDate="2011-03-11T18:27:08.630" Score="7" ViewCount="100" Body="&lt;p&gt;I have a set of around 500 responses to a online survey that offered an incentive to complete. While most of the data appears to be valid, it's clear that some people were able to get around the (inadequate) browser cookie-based duplicate survey protection. Some respondents clearly randomly clicked through the survey to recieve the incentive and then repeated the process via a couple methods. My question is what is the best way to try and filter out the invalid responses?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The information that I have is limited to:  &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The amount of time it took to complete the survey (time started and ended)&lt;/li&gt;&#xA;&lt;li&gt;The IP address of each respondent&lt;/li&gt;&#xA;&lt;li&gt;The User Agent (browser identifier) of each respondent&lt;/li&gt;&#xA;&lt;li&gt;The survey answers of each respondent (over 100 questions in the survey)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The most obvious sign of an invalid response is when (sorted by time started) there will be a group all from the same IP address or similar IP (sharing the same first three octets for example, 255.255.255.*) which were all completed in a much shorter amount of time than the total average in quick succession.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With this information there must be a thoughtful way to weed out the people who were exploiting the survey for the incentive from the rest of the survey population. I know that someone from the community here would have an interesting idea about how to approach this. I'm willing to accept false positives as long as I can be confident that I've gotten rid of most of the invalid responses. Thanks for your advice!&lt;/p&gt;&#xA;" OwnerUserId="1220" LastEditorUserId="1220" LastEditorDisplayName="" LastEditDate="2011-03-11T18:45:41.050" LastActivityDate="2011-03-11T19:48:07.653" Title="How to identify invalid online survey responses?" Tags="&lt;classification&gt;&lt;survey&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="1" />
  <row Id="8168" PostTypeId="1" CreationDate="2011-03-11T18:50:29.743" Score="4" ViewCount="71" Body="&lt;p&gt;Building on the post &lt;a href=&quot;http://stats.stackexchange.com/questions/2910/how-to-efficiently-manage-a-statistical-analysis-project&quot;&gt;How to efficiently manage a statistical analysis project&lt;/a&gt; and the  &lt;a href=&quot;http://www.johnmyleswhite.com/notebook/2010/08/26/projecttemplate/&quot; rel=&quot;nofollow&quot;&gt;&lt;code&gt;ProjectTemplate&lt;/code&gt; package&lt;/a&gt; in R...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Q: How do you build your statistical project directory structure when multiple languages feature heavily (e.g, R AND Splus)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most of the discussions on this topic have been limited to projects which primarily use one language. I'm concerned with how to minimize sloppiness, confusion, and breakage, when using multiple languages. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've included below my current project structure and methods for doing things. An alternative might be to separate code so that I have &lt;code&gt;./R&lt;/code&gt; and &lt;code&gt;./Splus&lt;/code&gt; directories---each containing their own &lt;code&gt;/lib&lt;/code&gt;, &lt;code&gt;/src&lt;/code&gt;, &lt;code&gt;/util&lt;/code&gt;, &lt;code&gt;/tests&lt;/code&gt;, and &lt;code&gt;/munge&lt;/code&gt; directories. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Q: Which approach would be closest to &quot;best practices&quot; (if any exist)?&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;/data&lt;/strong&gt; - data shared across projects &lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;/libraries&lt;/strong&gt; - scripts shared across projects &lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;/projects/myproject&lt;/strong&gt; - my working directory. Currently, if I use multiple languages they share this location as their working directory.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;./data/&lt;/strong&gt; - data specific to &lt;code&gt;/myproject&lt;/code&gt; and symlinks to data in &lt;code&gt;/data&lt;/code&gt; &lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;./cache/&lt;/strong&gt; - cached workspaces (e.g., &lt;code&gt;.RData&lt;/code&gt; files saved using &lt;code&gt;save.image()&lt;/code&gt; in R or &lt;code&gt;.sdd&lt;/code&gt; files saved using &lt;code&gt;data.dump()&lt;/code&gt; in Splus) &lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;./lib/&lt;/strong&gt; - main project files. Same across all projects. An R project will be run via &lt;code&gt;source(&quot;./lib/main.R&quot;)&lt;/code&gt; which in turn runs &lt;code&gt;load.R&lt;/code&gt;, &lt;code&gt;clean.R&lt;/code&gt;, &lt;code&gt;test.R&lt;/code&gt;, &lt;code&gt;analyze.R&lt;/code&gt;, &lt;code&gt;.report.R&lt;/code&gt;. Currently, if multiple languages are being used, say, Splus in addition to R, I'll throw &lt;code&gt;main.ssc&lt;/code&gt;, &lt;code&gt;clean.ssc&lt;/code&gt;, etc. into this directory as well. Not sure I like this though.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;./src/&lt;/strong&gt; - project-specific functions. Collected one function per file.  &lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;./util/&lt;/strong&gt; - general functions eventually to be packaged. Collected one function per file.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;./tests/&lt;/strong&gt; - files for running test cases. Used by &lt;code&gt;./lib/test.R&lt;/code&gt; &lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;./munge/&lt;/strong&gt; - files for cleaning data. Used by &lt;code&gt;./lib/clean.R&lt;/code&gt; &lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;./figures/&lt;/strong&gt; - tables and figure output from &lt;code&gt;./lib/report.R&lt;/code&gt; to be used in the final report &lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;./report/&lt;/strong&gt; - &lt;code&gt;.tex&lt;/code&gt; files and symlinks to files in &lt;code&gt;./figures&lt;/code&gt; &lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;./presentation/&lt;/strong&gt; - &lt;code&gt;.tex&lt;/code&gt; files for presentations (usually the &lt;code&gt;Beamer&lt;/code&gt; class) &lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;./temp/&lt;/strong&gt; - location for temporary scripts &lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;./README&lt;/strong&gt; &lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;./TODO&lt;/strong&gt; &lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;./.RData&lt;/strong&gt; - for storing R project workspaces &lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;./.Data/&lt;/strong&gt; - for storing S project workspaces&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="3577" LastEditorUserId="3577" LastEditorDisplayName="" LastEditDate="2011-03-11T19:31:20.497" LastActivityDate="2011-03-11T19:31:20.497" Title="Statistical project directory structure with multiple languages (e.g., R and Splus)?" Tags="&lt;r&gt;&lt;project-management&gt;&lt;splus&gt;" CommentCount="1" FavoriteCount="2" />
  <row Id="8169" PostTypeId="1" AcceptedAnswerId="8193" CreationDate="2011-03-11T19:29:25.397" Score="1" ViewCount="121" Body="&lt;p&gt;Given the prior probability of 2 distributions, $N(x,y)$ and $N(a,b)$, where $N(\mu,\sigma^2)$:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do you make a decision rule to minimize the probability of error, if the prior probabilities are equal? Can you give an example?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What if the prior probabilities are different, such as&lt;br&gt;&#xA;$P(\text{Distribution 1}) = 0.70$&lt;br&gt;&#xA;$P(\text{Distribution 2}) = 0.30$?&lt;/p&gt;&#xA;" OwnerUserId="3681" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-12T10:41:39.367" LastActivityDate="2011-03-12T15:22:41.277" Title="Bayes classifier" Tags="&lt;machine-learning&gt;&lt;naive-bayes&gt;" AnswerCount="2" CommentCount="4" FavoriteCount="2" />
  <row Id="8170" PostTypeId="2" ParentId="8052" CreationDate="2011-03-11T19:33:09.387" Score="1" ViewCount="" Body="&lt;p&gt;Fisher's exact was once only recommended for low cell counts because back in &quot;the dark times&quot; it was computationally infeasible to use it for large counts.  Indeed, with some approximations doing small counts properly demands a correction be applied. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;No matter, the hypergeometric tests involved are good and powerful.  They can be generalized to NxM tables by deposing these into 2x2 tables, where a count of significance for row and column is kept in row 1 and in column 1, and their complements summed to form row 2 and column 2, respectively. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;See &lt;a href=&quot;http://www.stat.psu.edu/online/courses/stat504/03_2way/30_2way_exact.htm&quot; rel=&quot;nofollow&quot;&gt;http://www.stat.psu.edu/online/courses/stat504/03_2way/30_2way_exact.htm&lt;/a&gt; for a nice overview and the text(s) by Bishop and Fienberg as well as Agresti for a detailed presentation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Realizing the connection with the hypergeometric lets you pick two aspects of whether an intersection represents independence or not, letting both false alarms and effect size be specified. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I like the Bishop and Fienberg, and Fienberg books:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0387728058&quot; rel=&quot;nofollow&quot;&gt;http://www.amazon.com/Discrete-Multivariate-Analysis-Theory-Practice/dp/0387728058/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0387728244&quot; rel=&quot;nofollow&quot;&gt;http://www.amazon.com/Analysis-Cross-Classified-Categorical-Data/dp/0387728244/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;as well as the related one by Zelterman:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0198567014&quot; rel=&quot;nofollow&quot;&gt;http://www.amazon.com/Models-Discrete-Data-Daniel-Zelterman/dp/0198567014/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="3437" LastEditorUserId="3437" LastEditorDisplayName="" LastEditDate="2011-03-11T20:05:12.323" LastActivityDate="2011-03-11T20:05:12.323" />
  <row Id="8171" PostTypeId="5" CreationDate="2011-03-11T19:34:18.293" Score="0" ViewCount="" Body="&lt;p&gt;&quot;Regression&quot; is a general term for a wide variety of techniques to analyze the relationship between one (or more) &lt;em&gt;dependent&lt;/em&gt; variables and &lt;em&gt;independent&lt;/em&gt; variables.  Typically the dependent variables are modeled with probability distributions whose parameters are assumed to vary (deterministically) with the independent variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ordinary least squares (OLS) regression affords a simple example in which the expectation of one dependent variable is assumed to depend linearly on the independent variables.  The unknown coefficients in the assumed linear function are estimated by choosing values for them that minimize the sum of squared differences between the values of the dependent variable and the corresponding fitted values.&lt;/p&gt;&#xA;" OwnerUserId="-1" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-11T19:34:18.293" LastActivityDate="2011-03-11T19:34:18.293" />
  <row Id="8172" PostTypeId="4" CreationDate="2011-03-11T19:34:18.293" Score="0" ViewCount="" Body="Techniques for analyzing the relationship between one (or more) &quot;dependent&quot; variables and &quot;independent&quot; variables." OwnerUserId="-1" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-11T19:34:18.293" LastActivityDate="2011-03-11T19:34:18.293" />
  <row Id="8173" PostTypeId="2" ParentId="8160" CreationDate="2011-03-11T19:47:59.110" Score="6" ViewCount="" Body="&lt;p&gt;You might consider u-scores as defined in [1] Wittkowski, K. M., Lee, E., Nussbaum, R., Chamian, F. N. and Krueger, J. G. (2004), Combining several ordinal measures in clinical studies. Statistics in Medicine, 23: 1579–1592. (&lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=46258B9F5D853340BDE8FAF578CFE5C2?doi=10.1.1.60.9819&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;PDF&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The basic idea is that for each observation you count how many observations there are compared to which it is definitely better (four variables lower, one higher), and how many are definitely worse, and then create a combined score.&lt;/p&gt;&#xA;" OwnerUserId="279" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-03-13T01:08:30.113" LastActivityDate="2011-03-13T01:08:30.113" CommentCount="3" />
  <row Id="8174" PostTypeId="2" ParentId="8167" CreationDate="2011-03-11T19:48:07.653" Score="5" ViewCount="" Body="&lt;p&gt;1) Flag all responses with duplicate IP addresses. Create a new variable for this purpose -- say FLAG1, which takes on values of 1 or 0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Choose a threshold for an impossibly fast response time based on common sense (e.g., less than 1 second per question) and the aid of a histogram of response times -- flag people faster than this threshold again using another variable, FLAG2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) &quot;Some respondents clearly randomly clicked through...&quot; -- Apparently you can manually identify some respondents who cheated. Sort the data by response time and look at the fastest 5% or 10% (25 or 50 respondents for your data). Manually examine these respondents and flag any &quot;clearly random&quot; ones using FLAG3. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;4) Apply Sheldon's suggestion by creating an inconsistency score -- 1 point for each inconsistency. You can do this by creating a new variable that identifies inconsistencies for each pair of redundant items, and then adding across these variables. You could keep this variable as is, as higher inconsistency scores obviously correspond to higher probabilities of cheating. But a reasonable approach is to flag people who fall above a cut-off chosen by inspecting a histogram -- call this FLAG4. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyone who is flagged on each of FLAG1-4 is highly likely to have cheated, but you can set aside flagged people for a separate analysis based on any weighting scheme of FLAG1-4 you want. Given your tolerance for false positives, I would eliminate anyone flagged on FLAG1, FLAG2, or FLAG4.&lt;/p&gt;&#xA;" OwnerUserId="3432" LastActivityDate="2011-03-11T19:48:07.653" CommentCount="1" />
  <row Id="8175" PostTypeId="2" ParentId="8164" CreationDate="2011-03-11T20:07:36.363" Score="9" ViewCount="" Body="&lt;p&gt;Try this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(png)&#xA;&#xA;#Replace the directory and file information with your info&#xA;ima &amp;lt;- readPNG(&quot;C:\\Documents and Settings\\Bill\\Data\\R\\Data\\Images\\sun.png&quot;)&#xA;&#xA;#Set up the plot area&#xA;plot(1:2, type='n', main=&quot;Plotting Over an Image&quot;, xlab=&quot;x&quot;, ylab=&quot;y&quot;)&#xA;&#xA;#Get the plot information so the image will fill the plot box, and draw it&#xA;lim &amp;lt;- par()&#xA;rasterImage(ima, lim$usr[1], lim$usr[3], lim$usr[2], lim$usr[4])&#xA;grid()&#xA;lines(c(1, 1.2, 1.4, 1.6, 1.8, 2.0), c(1, 1.3, 1.7, 1.6, 1.7, 1.0), type=&quot;b&quot;, lwd=5, col=&quot;white&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Below is the plot.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/Hnhir.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="2775" LastActivityDate="2011-03-11T20:07:36.363" />
  <row Id="8176" PostTypeId="2" ParentId="7979" CreationDate="2011-03-11T20:26:09.263" Score="1" ViewCount="" Body="&lt;p&gt;There is an estimator for the minimum (or the maximum) of a set of numbers given a sample. See Laurens de Haan, &quot;Estimation of the minimum of a function using order statistics,&quot; JASM, 76(374), June 1981, 467-469.&lt;/p&gt;&#xA;" OwnerUserId="3437" LastActivityDate="2011-03-11T20:26:09.263" CommentCount="1" />
  <row Id="8177" PostTypeId="2" ParentId="8166" CreationDate="2011-03-11T20:45:47.983" Score="5" ViewCount="" Body="&lt;p&gt;It's probably not a good idea without digging deeper into the source surveys. TI themselves note that changes in a country's index can come from either a change in corruption or just a change in methodology of the sources that they use. The sources themselves change over time as well, so comparing the index from year to year is actually fairly complicated to do properly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See the references at &lt;a href=&quot;http://en.wikipedia.org/wiki/Corruption_Perceptions_Index&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Corruption_Perceptions_Index&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="26" LastActivityDate="2011-03-11T20:45:47.983" CommentCount="2" />
  <row Id="8178" PostTypeId="2" ParentId="8169" CreationDate="2011-03-11T21:13:46.550" Score="0" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Linear_discriminant_analysis&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Linear_discriminant_analysis&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2077" LastActivityDate="2011-03-11T21:13:46.550" CommentCount="1" />
  <row Id="8179" PostTypeId="2" ParentId="8146" CreationDate="2011-03-11T21:15:49.980" Score="1" ViewCount="" Body="&lt;p&gt;I do not know such a library, but the statistics part of the &lt;a href=&quot;http://commons.apache.org/math/userguide/stat.html&quot; rel=&quot;nofollow&quot;&gt;apache commons math library&lt;/a&gt; (written in java) provides a series of distributions, including chi2. Since the McNemar's Test does not seem to be thaaat complicated, you may figure an implementation out on your own.&lt;/p&gt;&#xA;" OwnerUserId="264" LastActivityDate="2011-03-11T21:15:49.980" />
  <row Id="8180" PostTypeId="1" CreationDate="2011-03-11T21:24:19.273" Score="3" ViewCount="202" Body="&lt;p&gt;Does any body know how to run a post hoc comparison in a 2X2 ANOVA with covariate in R. multcomp package seems very nice but I could not find clear answer or example to my question with this package. Thanks so much in advance&lt;/p&gt;&#xA;" OwnerUserId="3682" LastActivityDate="2011-03-12T19:35:02.983" Title="Post hoc comparison in two way ANOVA with covariate using R" Tags="&lt;r&gt;&lt;multiple-comparisons&gt;&lt;contrasts&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="2" />
  <row Id="8181" PostTypeId="2" ParentId="8180" CreationDate="2011-03-11T22:06:14.867" Score="3" ViewCount="" Body="&lt;p&gt;This is data from Maxwell &amp;amp; Delaney (2004), artificially extended to include a second between-subjects IV, yielding a 3x2 design. Using &lt;code&gt;multcomp&lt;/code&gt;'s &lt;code&gt;glht()&lt;/code&gt; function is easier once you switch to the associated one-factorial design by combining your two IVs into one with &lt;code&gt;interaction()&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;DV is depression scores pre-treatment and post-treatment. Treatment is one of SSRI, Placebo or Waiting List. Pre-treatment score is the covariate. I included a second IV.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;P        &amp;lt;- 3     # number of groups in IV1&#xA;Q        &amp;lt;- 2     # number of groups in IV2&#xA;Njk      &amp;lt;- 5     # cell size&#xA;SSRIpre  &amp;lt;- c(18, 16, 16, 15, 14, 20, 14, 21, 25, 11)&#xA;SSRIpost &amp;lt;- c(12,  0, 10,  9,  0, 11,  2,  4, 15, 10)&#xA;PlacPre  &amp;lt;- c(18, 16, 15, 14, 20, 25, 11, 25, 11, 22)&#xA;PlacPost &amp;lt;- c(11,  4, 19, 15,  3, 14, 10, 16, 10, 20)&#xA;WLpre    &amp;lt;- c(15, 19, 10, 29, 24, 15,  9, 18, 22, 13)&#xA;WLpost   &amp;lt;- c(17, 25, 10, 22, 23, 10,  2, 10, 14,  7)&#xA;IV1      &amp;lt;- factor(rep(1:3,  each=Njk*Q), labels=c(&quot;SSRI&quot;, &quot;Placebo&quot;, &quot;WL&quot;))&#xA;IV2      &amp;lt;- factor(rep(1:2, times=Njk*P), labels=c(&quot;A&quot;, &quot;B&quot;))&#xA;&#xA;# combine both IVs into 1 to get the associated one-factorial design&#xA;IVi      &amp;lt;- interaction(IV1, IV2)&#xA;DVpre    &amp;lt;- c(SSRIpre,  PlacPre,  WLpre)&#xA;DVpost   &amp;lt;- c(SSRIpost, PlacPost, WLpost)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now do the ANCOVA with &lt;code&gt;IVi&lt;/code&gt; as between-subjects factor and &lt;code&gt;DVpre&lt;/code&gt; as covariate. Using the associated one-factorial design is possible since it has the same Error-MS as the two-factorial design.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; aovAncova1 &amp;lt;- aov(DVpost ~ IVi     + DVpre, data=dfAncova)  # one-factorial design&#xA;&amp;gt; aovAncova2 &amp;lt;- aov(DVpost ~ IV1*IV2 + DVpre, data=dfAncova)  # two-factorial design&#xA;&amp;gt; summary(aovAncova1)[[1]][[&quot;Mean Sq&quot;]][3]  # Error MS one-factorial design&#xA;[1] 32.84399&#xA;&#xA;&amp;gt; summary(aovAncova2)[[1]][[&quot;Mean Sq&quot;]][5]  # Error MS two-factorial design&#xA;[1] 32.84399&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Next comes the matrix defining 3 cell comparisons with sum-to-zero coefficients. The coefficients follow the order of 2*3 levels of &lt;code&gt;IVi&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; levels(IVi)&#xA;[1] &quot;SSRI.A&quot; &quot;Placebo.A&quot; &quot;WL.A&quot; &quot;SSRI.B&quot; &quot;Placebo.B&quot; &quot;WL.B&quot;&#xA;&#xA;&amp;gt; cntrMat &amp;lt;- rbind(&quot;SSRI-Placebo&quot;  = c(-1, 1, 0, -1,  1,  0),&#xA;+                  &quot;SSRI-0.5(P+WL)&quot;= c(-2, 1, 1, -2,  1,  1),&#xA;+                  &quot;A-B&quot;           = c( 1, 1, 1, -1, -1, -1))&#xA;&#xA;&amp;gt; library(multcomp)&#xA;&amp;gt; summary(glht(aovAncova1, linfct=mcp(IVi=cntrMat), alternative=&quot;greater&quot;),&#xA;+         test=adjusted(&quot;none&quot;))&#xA;&#xA;Simultaneous Tests for General Linear Hypotheses&#xA;Multiple Comparisons of Means: User-defined Contrasts&#xA;Fit: aov(formula = DVpost ~ IVi + DVpre, data = dfAncova)&#xA;&#xA;Linear Hypotheses:&#xA;                    Estimate Std. Error t value  Pr(&amp;gt;t)   &#xA;SSRI-Placebo &amp;lt;= 0      8.887      5.135   1.731 0.04846 * &#xA;SSRI-WL &amp;lt;= 0          12.878      5.129   2.511 0.00976 **&#xA;A-B &amp;lt;= 0               1.024      6.492   0.158 0.4380&#xA;---&#xA;Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 &#xA;(Adjusted p values reported -- none method)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1909" LastEditorUserId="1909" LastEditorDisplayName="" LastEditDate="2011-03-12T19:35:02.983" LastActivityDate="2011-03-12T19:35:02.983" CommentCount="2" />
  <row Id="8182" PostTypeId="1" CreationDate="2011-03-11T22:58:23.873" Score="2" ViewCount="130" Body="&lt;p&gt;Hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;is it possible to use kernel PCA for Latent Semantic Indexing (LSI) in the same way as PCA is used?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I perform LSI in R using the &lt;strong&gt;prcomp&lt;/strong&gt; PCA function and extract the features with highest loadings from the first k components. By that i get the features describing the component best.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried to use the &lt;strong&gt;kpca&lt;/strong&gt; function (kernlib package) but cannot see how to access the weights of the features to a principal component. Is this possible overall when using kernel methods? &lt;/p&gt;&#xA;" OwnerUserId="3683" LastActivityDate="2011-03-13T23:02:22.433" Title="Kernel PCA for LSI using R" Tags="&lt;r&gt;&lt;pca&gt;&lt;kernel&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="8183" PostTypeId="2" ParentId="8146" CreationDate="2011-03-11T23:19:14.227" Score="1" ViewCount="" Body="&lt;p&gt;I found this on &lt;a href=&quot;http://www.google.com/search?sourceid=chrome&amp;amp;ie=UTF-8&amp;amp;q=Mcnemar+test+java&quot; rel=&quot;nofollow&quot;&gt;Google&lt;/a&gt;:&lt;br&gt;&#xA;&lt;a href=&quot;http://www.jsc.nildram.co.uk/api/jsc/contingencytables/McNemarTest.html&quot; rel=&quot;nofollow&quot;&gt;http://www.jsc.nildram.co.uk/api/jsc/contingencytables/McNemarTest.html&lt;/a&gt;&lt;br&gt;&#xA;Does it do anything for you? McNemar's Test is implemented in the Java Statistical Classes Library, specifically in &lt;a href=&quot;http://www.jsc.nildram.co.uk/api/jsc/contingencytables/package-summary.html&quot; rel=&quot;nofollow&quot;&gt;jsc.contingencytables&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1118" LastActivityDate="2011-03-11T23:19:14.227" CommentCount="1" />
  <row Id="8184" PostTypeId="1" CreationDate="2011-03-11T23:36:27.563" Score="4" ViewCount="98" Body="&lt;p&gt;It's well established that both Anderson-Darling and Shapiro-Wilk have a much higher power to detect departures from normality than a KS-Test.&lt;br&gt;&#xA;I have been told that Shapiro-Wilk is usually the best test to use if you want to test if a distribution is normal because it has one of the highest powers to detect lack of normality, but my limited experience, it seems that Shapiro-Wilk gives me the same result as Anderson-Darling every time.&lt;br&gt;&#xA;I thus have two questions:  &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;When does the Shapiro-Wilk test out-perform Anderson-Darling?&lt;/li&gt;&#xA;&lt;li&gt;Is there a uniformly most powerful lack of normality test, or, barring that possibility, a normality test that out-performs nearly all other normality tests, or is Shapiro-Wilk the best bet?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="1118" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-12T10:43:08.117" LastActivityDate="2011-03-12T16:15:55.317" Title="Most powerful GoF test for normality" Tags="&lt;goodness-of-fit&gt;" AnswerCount="0" CommentCount="14" />
  <row Id="8185" PostTypeId="1" CreationDate="2011-03-11T23:54:24.433" Score="4" ViewCount="152" Body="&lt;p&gt;From basic statistics and hearing &quot;correlation is not causation&quot; all the time, I tend to think it's fine to say that &quot;X and Y are correlated&quot; even if X and Y aren't in a causal relationship. For example, I'd normally think it's perfectly okay to say that ice cream sales and swimsuit sales are correlated, since high swimsuit sales probably means high ice cream sales (even though increases in swimsuit sales don't cause an increase an ice cream sales).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, when studying time analysis, I get a little confused about this terminology. It seems like a time series analyst would &lt;em&gt;not&lt;/em&gt; say that ice cream sales are correlated with swimsuit sales, but rather that ice cream sales are &lt;em&gt;spuriously&lt;/em&gt; correlated with swimsuit sales. An unmodified &quot;X is correlated with Y&quot; seems to be reserved for the case where X actually &lt;em&gt;causes&lt;/em&gt; Y, so it's fine to say &lt;em&gt;temperature&lt;/em&gt; (but not ice cream) is correlated with swimsuit sales.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this correct? My problem is that there seem to be two meanings to spurious correlation:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Regress two independent random walks against each other, and ordinary statistical tests will say that they're correlated, even though the two random walks are obviously unrelated in any fashion.  (I'm fine with this meaning of spurious correlation, since there really is no relationship.)&lt;/li&gt;&#xA;&lt;li&gt;Regress ice cream sales against swimsuit sales. It confuses me that &lt;em&gt;this&lt;/em&gt; correlation is called spurious, since there really &lt;em&gt;is&lt;/em&gt; a relationship between ice cream sales and swimsuit sales, even though this relationship isn't causal.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;So I guess my question is: do time series analysts reserve the term &quot;(non-spurious) correlation&quot; for causal relationships -- so that for time series analysts, correlation &lt;em&gt;is&lt;/em&gt; meant to suggest causation! -- while statisticians in general are fine with using &quot;correlation&quot; to indicate any kind of (possibly non-causal) relationship?&lt;/p&gt;&#xA;" OwnerUserId="3684" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-12T10:39:22.473" LastActivityDate="2011-03-12T16:37:12.907" Title="&quot;Correlation&quot; terminology in time series analysis" Tags="&lt;time-series&gt;&lt;correlation&gt;" AnswerCount="3" CommentCount="1" />
  <row Id="8186" PostTypeId="2" ParentId="8185" CreationDate="2011-03-12T00:04:38.953" Score="2" ViewCount="" Body="&lt;p&gt;In order to avoid the spurious correlation problem, you should regress two &lt;em&gt;stationary&lt;/em&gt; time series against one another. This can (potentially) provide a causal story. It is non-stationary series that lead to spurious correlation. See the reasoning given by my answer to &lt;a href=&quot;http://stats.stackexchange.com/questions/7975/what-to-make-of-explanatories-in-time-series/8037#8037&quot;&gt;this question&lt;/a&gt; (As a footnote, you may not need stationary series if they are integrated series, but I'd point you to any of the applied time series books to learn more about that.)&lt;/p&gt;&#xA;" OwnerUserId="401" LastActivityDate="2011-03-12T00:04:38.953" CommentCount="3" />
  <row Id="8187" PostTypeId="1" AcceptedAnswerId="8279" CreationDate="2011-03-12T01:05:02.263" Score="0" ViewCount="173" Body="&lt;p&gt;I am a beginner in statistics. I have these unpublished data (cases and deaths) of a disease for 7 years (2004-2010) from 2 neighbouring states. The study was started in 2004. State 1 and State 2 received different treatments. The death rate is high in one State. I want to prove that treatment given in State 1 is superior compared to that given in State 2. Time series was not accepted because of many possible confounding variables.&#xA;I have SPSS and Comprehensive Meta-analysis software. Please guide me.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;What is the best design? Prospective cohort study or Case-control study?&lt;/li&gt;&#xA;&lt;li&gt;What is the best effect measure (Odds ratio, risk ratio, any other)?&lt;/li&gt;&#xA;&lt;li&gt;What is the best statistical test (?Chi square etc)&lt;/li&gt;&#xA;&lt;li&gt;Can I use meta-analysis in this case?&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Any other information you think that would be of use.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;State 1         State 2 &#xA;Cases   Deaths  Cases   Deaths&#xA;2004    1125    5   2024    254&#xA;2005    1213    5   1978    209&#xA;2006    1003    4   2294    217&#xA;2007    1425    6   2312    249&#xA;2008    1172    4   1528    197&#xA;2009    1092    3   1683    204&#xA;2010    1316    4   2024    218&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="2956" LastEditorUserId="2956" LastEditorDisplayName="" LastEditDate="2011-03-13T04:16:15.557" LastActivityDate="2011-03-14T18:12:36.827" Title="How to assess effect of intervention in one state versus another using annual death rate data?" Tags="&lt;meta-analysis&gt;&lt;experiment-design&gt;&lt;effect-size&gt;&lt;odds-ratio&gt;&lt;relative-risk&gt;" AnswerCount="2" CommentCount="11" />
  <row Id="8188" PostTypeId="2" ParentId="8145" CreationDate="2011-03-12T03:35:21.030" Score="2" ViewCount="" Body="&lt;p&gt;Calculate &lt;a href=&quot;http://en.wikipedia.org/wiki/Correlogram&quot; rel=&quot;nofollow&quot;&gt;correlogram&lt;/a&gt; for your process. If your process is gaussian (by the looks of your sample it is) you can establish lower/upper bounds (B) and check if the correlations at given lag are significant. Positive autocorrelation at lag 1 would indicate existence of &quot;streaks of luck&quot;.&lt;/p&gt;&#xA;" OwnerUserId="3686" LastActivityDate="2011-03-12T03:35:21.030" CommentCount="1" />
  <row Id="8189" PostTypeId="2" ParentId="8185" CreationDate="2011-03-12T03:46:28.557" Score="2" ViewCount="" Body="&lt;p&gt;As to your main question, my answer is No.  If you've seen such a distinction between the way the terms are used in time series contexts vs. cross-sectional contexts, it must be due to one or two idiosyncratic authors you've read.  Rigorous authors would never be correct in using &quot;correlation&quot; to mean &quot;causation.&quot;  You've gotten ahold of a spurious terminology distinction there.  Interesting question, though.&lt;/p&gt;&#xA;" OwnerUserId="2669" LastActivityDate="2011-03-12T03:46:28.557" CommentCount="2" />
  <row Id="8190" PostTypeId="2" ParentId="8145" CreationDate="2011-03-12T05:40:11.007" Score="4" ViewCount="" Body="&lt;p&gt;A few thoughts:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Plot the distribution of times.&#xA;My guess is that they will be positively skewed, such that some solution times are really slow. In that case you might want to consider a log or some other transformation of solution times.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Create a scatter plot of trial on the x axis and solution time (or log solution time on the y-axis).&#xA;This should give you an intuitive understanding of the data.&#xA;It may also reveal other kinds of trends besides the &quot;hot streak&quot;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Consider whether there is a learning effect over time.&#xA;With most puzzles, you get quicker with practice.&#xA;The plot should help to reveal whether this is the case.&#xA;Such an effect is different to a &quot;hot streak&quot; effect.&#xA;It will lead to correlation between trials because when you are first learning, slow trials will co-occur with other slow trials, and as you get more experienced, faster trials will co-occur with faster trials.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Consider your conceptual definition of &quot;hot streaks&quot;.&#xA;For example, does it only apply to trials that are proximate in time or is about proximity of order. Say you solved the cube quickly on Tuesday, and then had a break and on the next Friday you solved it quickly. Is this a hot streak, or does it only count if you do it on the same day?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Are there other effects that might be distinct from a hot streak effect? &#xA;E.g., time of day that you solve the puzzle (e.g., fatigue), degree to which you are actually trying hard? etc.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Once the alternative systematic effects have been understood, you could develop a model that includes as many of them as possible.&#xA;You could plot the residual on the y axis and trial on the x-axis.&#xA;Then you could see whether there are auto-correlations in the residuals in the model.&#xA;This auto-correlation would provide some evidence of hot streaks.&#xA;However, an alternative interpretation is that there is some other systematic effect that you have not excluded.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastActivityDate="2011-03-12T05:40:11.007" CommentCount="2" />
  <row Id="8191" PostTypeId="2" ParentId="8160" CreationDate="2011-03-12T06:02:41.313" Score="6" ViewCount="" Body="&lt;h3&gt;Data or Theory Driven?&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;The first issue is whether you want the composite to be data driven or theory driven?&#xA;If you are wishing to form a composite variable, it is likely that you think that each component variable is important in measuring some overall domain. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this case, you are likely going to prefer a theoretical set of weights. If, alternatively, you are interested in whatever is shared or common amongst the component variables, at the risk of not including one of the variables because it measures something that is  orthogonal or less related to the remaining set, then you might want to explore data driven approaches.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This question maps on to the discussion in the structural equation modelling literature between reflective and formative measures&#xA;( e.g., see &lt;a href=&quot;http://smib.vuw.ac.nz:8081/WWW/ANZMAC2004/CDsite/papers/Bucic1.PDF&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Whatever you do it is important to align your measurement with your actual research question.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Theory Driven&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;If the composite is theoretically driven then you will want to form a weighted composite of the component variables where the weight assigned aligns with your theoretical weighting of the component.&#xA;If the variables are ordinal, then you'll have to think about how to scale the variable.&#xA;After scaling each component variable, you'll have to think about theoretical relative weighting and issues related to differential standard deviations of the variable.&#xA;One simple strategy is to convert all component variables into z-scores, and sum the z-scores.&#xA;If you have component variables, where some are positive and others are negative, then you'll need to reverse either just the negative or just the positive component variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wrote a &lt;a href=&quot;http://jeromyanglim.blogspot.com/2009/03/calculating-composite-scores-of-ability.html&quot; rel=&quot;nofollow&quot;&gt;post on forming composites&lt;/a&gt; which addresses several scenarios for forming composites.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Theoretical driven approaches can be implemented easily in any statistical packages.&#xA;&lt;code&gt;score.items&lt;/code&gt; in the &lt;code&gt;psych&lt;/code&gt; package is one function that makes it a little easier, but it is limited.&#xA;You might just write your own equation using simple arithmetic, and perhaps the &lt;code&gt;scale&lt;/code&gt; function.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Data Driven&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;If you are more interested in being data driven, then there are many possible approaches.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Taking the first principal component sounds like a reasonable idea.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have ordinal variables you might think about categorical PCA which would allow the component variables to be reweighted. This could automatically handle the quantification given the constraints you provide.&lt;/p&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-03-13T01:06:48.167" LastActivityDate="2011-03-13T01:06:48.167" CommentCount="2" />
  <row Id="8192" PostTypeId="1" CreationDate="2011-03-12T06:59:04.090" Score="2" ViewCount="164" Body="&lt;p&gt;I want to test whether a correlation coefficient is greater than 0.5.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;What should the null and alternative hypothesis be?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Should I do a two tailed test or one tailed test?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="3688" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-03-12T07:18:23.997" LastActivityDate="2011-03-13T03:40:01.790" Title="Hypothesis test for whether correlation coefficient is greater than specified value?" Tags="&lt;hypothesis-testing&gt;&lt;correlation&gt;" AnswerCount="2" />
  <row Id="8193" PostTypeId="2" ParentId="8169" CreationDate="2011-03-12T08:08:58.610" Score="2" ViewCount="" Body="&lt;p&gt;Let's say we have to classes $C$, i.e. $c_1$ and $c_2$ and further that the points of the class $c_1$ are drawn from N(a,b) and the points of $c_2$ from N(c,d) respectively (note the changes in notation). The point variable shall be X, a value of this variable x. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the Bayes-Theorem:&#xA;$p(C|X)=\frac{p(X|C)*p(C)}{p(X)}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The bayes decision rule for binary classification states: Given x, one decides...&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;in favor of $c_1$ If $p(c_1|x) &amp;gt; p(c_2|x)$ &lt;/li&gt;&#xA;&lt;li&gt;in favor of $c_2$ If $p(c_2|x) &amp;gt; p(c_1|x)$ &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The cases that both posterior probabilities is equal has to be treated separately. Normally one favors one class more than the other and hence decides in favor of that class (if a decision is forced). Let's say, that if the posteriors are equal, we decide in favor of class $c_2$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The above decision rule can be reformulated to:&#xA;If $\frac{p(c_1|x)}{p(c_2|x)} &amp;gt; 0$, decide in favor of $c_1$, else $c_2$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Applying Bayes Theorem ...&#xA;$\frac{p(c_1|x)}{p(c_2|x)}$ = $\frac{p(x|c_1)*p(c_1)}{p(x|c_2)*p(c_2)}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the priors are equal, the decision rule resolves to $\frac{p(x|c_1)}{p(x|c_2)}$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If p(c_1)=0.7 and p(c_2)=0.3, the decision rule resolves to $\frac{0.7}{0.3}*\frac{p(x|c_1)}{p(x|c_2)}\approx 2.33*\frac{p(x|c_1)}{p(x|c_2)}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The conditional densities p(X|C) are defined by the corresponding normal distributions. The density of a normal distribution for (not at) a point x can be calculated as $pdf(x|N(a,b))=\frac{1}{\sqrt{2\pi*b}} * e^{-\frac{(x-a)^2}{2b}}$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Inserting this into our formula (for case of equal priors) one gets &#xA;$\frac{p(x|c_1)}{p(x|c_2)}=\frac{pdf(x|N(a,b))}{pdf(x|N(c,d))}$. Starting from here you can &quot;simplify&quot; the formula even further.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt;&#xA;Here is some R - code to visualize the priors * conditional probabilities (ignoring the normalization factor p(X)):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(lattice)&#xA;#&#xA;set.seed(42)&#xA;&#xA;# parameters for distribution of class 1&#xA;a &amp;lt;- 0&#xA;b &amp;lt;- 1&#xA;# parameters for distribution of class 2&#xA;c &amp;lt;- 3&#xA;d &amp;lt;- 1&#xA;&#xA;x &amp;lt;- c(sort(rnorm(1000,mean=a,sd=b)),sort(c(rnorm(1000,mean=c,sd=d))))&#xA;y &amp;lt;- c(dnorm(x[1:1000],mean=a,sd=b),dnorm(x[1001:2000],mean=c,sd=d))&#xA;labels &amp;lt;- factor(rep(c(&quot;class 1&quot;,&quot;class 2&quot;),each=1000))&#xA;&#xA;dat &amp;lt;- data.frame(&quot;x&quot;=x,&quot;density&quot;=y,&quot;groups&quot;=labels)&#xA;xyplot(density~x,data=dat,groups=labels,type=&quot;b&quot;,auto.key=T)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which results in this plot, where you can see the decision &quot;rule&quot; (line). Starting from here, I suggest to play around a little bit with the priors to get a good feeling &quot;what's going on&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/m1RJJ.png&quot; alt=&quot;class conditionals&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="264" LastEditorUserId="264" LastEditorDisplayName="" LastEditDate="2011-03-12T15:22:41.277" LastActivityDate="2011-03-12T15:22:41.277" />
  <row Id="8194" PostTypeId="2" ParentId="8192" CreationDate="2011-03-12T08:18:32.070" Score="1" ViewCount="" Body="&lt;p&gt;I came up with the following by taking a look at some video lectures. Are they correct?&lt;br/&gt;&#xA;The null hypothesis : correlation = 0.5&lt;br/&gt;&#xA;The alternative hypothesis : correlation &gt; 0.5&lt;br/&gt;&lt;br/&gt;&#xA;Please confirm if they are correct.&lt;/p&gt;&#xA;" OwnerUserId="3688" LastEditorUserId="3688" LastEditorDisplayName="" LastEditDate="2011-03-12T09:21:54.810" LastActivityDate="2011-03-12T09:21:54.810" CommentCount="3" />
  <row Id="8195" PostTypeId="2" ParentId="8185" CreationDate="2011-03-12T08:51:13.900" Score="3" ViewCount="" Body="&lt;p&gt;There is a good definition of spurious relationship in &lt;a href=&quot;http://en.wikipedia.org/wiki/Spurious_relationship&quot; rel=&quot;nofollow&quot;&gt;wikipedia&lt;/a&gt;. Spurious means that there is some hidden variable or feature which causes both of the variables. In both time-series and in usual regression then terminology means the same, the relationship between two variables is spurious when something else causes both variables. In time-series context this something else is inherent property of random walks, in usual regression analysis some other variable. &lt;/p&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-03-12T11:44:43.593" LastActivityDate="2011-03-12T11:44:43.593" CommentCount="5" />
  <row Id="8196" PostTypeId="1" CreationDate="2011-03-12T09:27:27.410" Score="8" ViewCount="329" Body="&lt;p&gt;Hi all,&lt;br/&gt;&#xA;This is somewhat related to &lt;a href=&quot;http://stats.stackexchange.com/questions/8192/hypothesis-test-for-whether-correlation-coefficient-is-greater-than-specified-val&quot;&gt;another question&lt;/a&gt; that I asked. The question I have is, when doing hypothesis testing, when the alternative hypothesis is a range, the null hypothesis is still a point value.&lt;br/&gt; As an example, when  testing whether a correlation coefficient is greater than 0.5, the null hypothesis is &quot;correlation = 0.5&quot; instead of &quot;correlation &amp;lt;= 0.5&quot;. Why is this the case? (or have I got it wrong?)&lt;/p&gt;&#xA;" OwnerUserId="3688" LastActivityDate="2011-03-12T13:29:49.183" Title="Why is the null hypothesis always a point value rather than a range in hypothesis testing?" Tags="&lt;hypothesis-testing&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="8197" PostTypeId="2" ParentId="8166" CreationDate="2011-03-12T09:34:31.413" Score="3" ViewCount="" Body="&lt;p&gt;One problem with your pre-post paired &lt;em&gt;t&lt;/em&gt;-test idea is that it chould give a small &lt;em&gt;p&lt;/em&gt;-value if there's a general upward (or downward) trend in the corruption score over time regardless of ratification. You need some sort of comparison group in which the ratification status did not change between the two time points.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In principle, one approach to forming a comparison group could be to find a matched 'control' country that didn't change its ratification status during the same time period for each country (of a sample of those) that did. The 'control' country should have similar baseline corruption score, and ideally be as similar as possible in other respects that might affect the rate of change of corruption over time (that's what I mean by matching). I haven't looked at the data so I've no idea if this is feasible. You'd want to consider removing the country that changed status from your analysis if you can't find a 'control' country that matches closely enough (but what's 'close enough'?). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In any case it certainly wouldn't be simple, but drawing valid and defensible conclusions about causal effects from observational data seldom is. I agree with @JMS that it's probably not worth attempting in your situation.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-03-12T09:34:31.413" CommentCount="1" />
  <row Id="8198" PostTypeId="2" ParentId="8196" CreationDate="2011-03-12T10:13:40.070" Score="8" ViewCount="" Body="&lt;p&gt;First, it is not always the case. There might be &lt;strong&gt;composite null&lt;/strong&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Most standard test have simple null because&lt;/strong&gt; in the framework of Neymann and Pearson the aim is to provide a decision rule that permits you to control the error of rejecting the null when it is true. &lt;strong&gt;To control this error you need to specify one distribution for the null.&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;When you have composite&lt;/strong&gt; hypothesis there are many possibilities. In this case, there are two natural types of strategies, either a &lt;strong&gt;bayesian&lt;/strong&gt; one (i.e. put weigths on the different null distribution) or a &lt;strong&gt;minimax&lt;/strong&gt; one (where you want to construct a test that has a controled error in the worst case. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the bayesian setting, using the posterior, your are rapidly back to the case of a simple null. In the minimax setting, if the null is something like corre&amp;lt;= 0.5 it might be that the problem is equivalent to using the simple null corre=0.5. Hence to avoid talking about minimax people directly take the simple null that is the 'extrem point' of the composite setting. In the general case it is often possible to transform the composite minimax null in a simple null... hence treating rigorously the case of composite null is to my knowledge mostly done by going back somhow to the simple null. &lt;/p&gt;&#xA;" OwnerUserId="223" LastEditorUserId="223" LastEditorDisplayName="" LastEditDate="2011-03-12T13:29:49.183" LastActivityDate="2011-03-12T13:29:49.183" CommentCount="5" />
  <row Id="8199" PostTypeId="2" ParentId="8196" CreationDate="2011-03-12T10:24:55.427" Score="1" ViewCount="" Body="&lt;p&gt;I don't think that the null hypothesis should always be something like correlation=0.5. At least in the problems which I have come across that wasn't the case. For example in information theoretic statistics the following problem is considered. Suppose that $X_1, X_2, \cdots, X_n$ are coming from an unknown distribution $Q$. In the simplest case we want to test between two distributions $P_1$ and $P_2$. So the hypotheses are $H_1:Q=P_1$ and $H_2:Q=P_2$.&lt;/p&gt;&#xA;" OwnerUserId="3485" LastEditorUserId="3485" LastEditorDisplayName="" LastEditDate="2011-03-12T10:31:47.807" LastActivityDate="2011-03-12T10:31:47.807" CommentCount="1" />
  <row Id="8200" PostTypeId="2" ParentId="8161" CreationDate="2011-03-12T12:25:42.250" Score="3" ViewCount="" Body="&lt;p&gt;If time series are cointegrated they admit VECM representation according to Granger Representation theorem. This is scantily explained in &lt;a href=&quot;http://en.wikipedia.org/wiki/Johansen_test&quot; rel=&quot;nofollow&quot;&gt;this wikipedia page&lt;/a&gt;. So if we have I(1) process:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$X_t=\mu+\Phi D_t+\Pi_1 X_{t-1}+...+\Pi_p X_{t-p}+\varepsilon_t$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;it admits VECM representation&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\Delta X_t=\mu+\Phi D_t+\Pi X_{t-p}+ \Gamma_1 \Delta X_{t-1}+...+\Gamma_p \Delta X_{t-p+1}+\varepsilon_t$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What this means is that if you difference time series and do a linear regression as per your second approach you are not including the cointegration term $\Pi X_{t-p}$. So your regression suffers from &lt;a href=&quot;http://en.wikipedia.org/wiki/Omitted-variable_bias&quot; rel=&quot;nofollow&quot;&gt;omitted variable problem&lt;/a&gt;, which in turn makes the test you are trying to use not viable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note:&lt;/strong&gt; This a classical explanation. If on the other hand you search for &quot;Granger representation theorem&quot; quite a few links pop-up with examples when this theorem does not hold. Here &lt;a href=&quot;http://www.ecosoc.org.au/files/File/TAS/ACE07/presentations%20%28pdf%29/Wilson.pdf&quot; rel=&quot;nofollow&quot;&gt;is one for example&lt;/a&gt;. I would be wary though as with all claims which goes counter the widely accepted truths. They may be right, but they certainly attract certain kind of people which contribute nothing interesting to discussion, albeit they loudly claim they do. It is a general advice, I may be very wrong on this one, it was a bit of surprise for me to find these links.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-03-12T12:25:42.250" />
  <row Id="8201" PostTypeId="1" AcceptedAnswerId="8205" CreationDate="2011-03-12T15:58:35.533" Score="4" ViewCount="56" Body="&lt;p&gt;EDIT: I reduced &lt;em&gt;my&lt;/em&gt; problem to a more specific question:&#xA;&lt;a href=&quot;http://math.stackexchange.com/questions/26573/&quot;&gt;http://math.stackexchange.com/questions/26573/&lt;/a&gt;&#xA;But I am still interested in other ideas.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say our data is generated by&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Y_i = f(X_i) + \epsilon_i$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $X_i$ are observed vectors, and $f$ is an unknown function.  We know that $f$ is invariant with respect to &lt;em&gt;permutation&lt;/em&gt; of the elements of $X$.  For example, if $X_i=[x_{i1},x_{i2},x_{i3}]$, then we have&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;f([x_{i1},x_{i2},x_{i3}]) = f([x_{i1},x_{i3},x_{i2}]) = f([x_{i2},x_{i1},x_{i3}])=\cdots &#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there modified versions of linear regression, support vector machines, forests, etc. which can be used to estimate $f$?  I'm specifically interested in the case when $X_i$ are eigenvalues of matrices (so they have complex-valued entries).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: A desperation move would be to make replicates of each data point with all permutations of each $X_i$ vectors and then apply standard methods, but this is clearly computationally impractical.&lt;/p&gt;&#xA;" OwnerUserId="3567" LastEditorUserId="3567" LastEditorDisplayName="" LastEditDate="2011-03-12T17:19:17.993" LastActivityDate="2011-03-12T18:08:32.600" Title="Learning from unordered tuples?" Tags="&lt;regression&gt;" AnswerCount="2" CommentCount="7" />
  <row Id="8204" PostTypeId="1" CreationDate="2011-03-12T16:38:34.327" Score="2" ViewCount="140" Body="&lt;p&gt;I have used factor analysis with regression method. I was trying to develop an index using this regression equation by using the output of the component score coefficient matrix (SPSS) generated for each factor. Should I include the residuals found under the &quot;Reproduced Correlations Table&quot; (SPSS) in the regression equation?&lt;/p&gt;&#xA;" OwnerUserId="3690" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-12T19:26:13.213" LastActivityDate="2011-03-12T19:26:13.213" Title="Factor analysis: regression equation and residuals" Tags="&lt;spss&gt;&lt;factor-analysis&gt;" FavoriteCount="1" />
  <row Id="8205" PostTypeId="2" ParentId="8201" CreationDate="2011-03-12T17:21:16.623" Score="2" ViewCount="" Body="&lt;p&gt;For regression models, one way would be to generate derived variables that are invariant to permutation of the labelling of the $x_i$s.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;E.g. in your three-variable example, considering only polynomials of total order up to 3, such combinations would be:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;$w_1 = x_1 + x_2 + x_3$&lt;/li&gt;&#xA;&lt;li&gt;$w_2 = x_1x_2 + x_1x_3 + x_2x_3$&lt;/li&gt;&#xA;&lt;li&gt;$w_3 = x_1^2 + x_2^2 + x_3^2$&lt;/li&gt;&#xA;&lt;li&gt;$w_4 = x_1x_2x_3$&lt;/li&gt;&#xA;&lt;li&gt;$w_5 = x_1^2x_2 + x_1^2x_3 + x_2^2x_1 + x_2^2x_3 + x_3^2x_1 + x_3^2x_2$&lt;/li&gt;&#xA;&lt;li&gt;$w_6 = x_1^3 + x_2^3 + x_3^3$&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You could then use any any form of regression that includes some function $f(a_1w_1 + a_2w_2 + \cdots + a_8w_8)$ and find values for the $a_i$s by non-linear least squares, generalized linear modelling or other methods. The combination $a_1w_1 + a_2w_2 + \cdots + a_8w_8$ fits a response surface that's a polynomial of order 3 that is symmetric in permutation of $x_1, x_2, x_3$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Clearly there would be many more possibilities if you wished to allow functions other than polynomials such as logs, fractional powers...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(EDIT I finished this post before I saw your edit to the answer with the link to the more specific question on mathoverflow. I was started to think there must be some mathematical framework for listing all such polynomials of a given total order, but it sounds you already know more than me about the relevant area of maths!)&lt;/p&gt;&#xA;" OwnerUserId="449" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-03-12T17:33:03.023" LastActivityDate="2011-03-12T17:33:03.023" CommentCount="1" />
  <row Id="8206" PostTypeId="1" CreationDate="2011-03-12T17:31:58.277" Score="5" ViewCount="392" Body="&lt;p&gt;I need to build a boxplot without any axes and add it to the current plot (ROC curve), but I need to add more text information to the boxplot: the labels for min and max. Current line of code is below (current graph also).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks a lot for assistance.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;boxplot(data, horizontal = TRUE, range = 0, axes=FALSE, col = &quot;grey&quot;, add = TRUE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The other solution is add the line from 0 to 1 (instead of x-axis), but I want it to go through the central line...for example like this graphic &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/goNiC.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="3345" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-12T22:36:01.140" LastActivityDate="2011-03-19T12:26:25.473" Title="Labeling boxplots in R" Tags="&lt;r&gt;&lt;boxplot&gt;" AnswerCount="4" FavoriteCount="1" />
  <row Id="8207" PostTypeId="2" ParentId="8201" CreationDate="2011-03-12T18:08:32.600" Score="2" ViewCount="" Body="&lt;p&gt;To add to onestop's response, it was confirmed on math.SE that the polynomials&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$w_1 = x_1 + \cdots + x_n$$&#xA;$$w_2 = x_1^2 + \cdots + x_n^2$$&#xA;$$\cdots$$&#xA;$$w_n = x_1^n + \cdots + x_n ^n$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;give you all the information needed to determine the original $X=(x_1, \cdots, x_n)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a neat result because it also applies to moments of a discrete distribution with uniform probabilities.&lt;/p&gt;&#xA;" OwnerUserId="3567" LastActivityDate="2011-03-12T18:08:32.600" />
  <row Id="8208" PostTypeId="2" ParentId="7946" CreationDate="2011-03-12T18:10:30.497" Score="1" ViewCount="" Body="&lt;p&gt;To echo everyone else: MORE DETAILS ABOUT YOUR DATA. Please give a qualitative description of what your independent and dependent variable(s) is/are.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;b&gt;EDIT: Yes this is confusing; hopefully it's cleared up now.&lt;/b&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, you probably want to avoid using sample statistics to estimate population parameters if you have the population data. This is because sample statistics are &lt;b&gt;estimates&lt;/b&gt; of population parameters, thus the methods used to compute sample statistics always have less power than those same methods in their population parameter version(s). Of course, most of the time you have to use sample statistics because you don't have complete population data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case either way you slice it inferring anything about a population from a case study is dubious because case studies are, by definition, case by case. You could make an inference about the case on which you collected data, but how useful is that? Maybe in your case it is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Either way, forget about whether or not you can/should use a sample method when you have the population data. You don't have population data if it's a case study. Also, sample vs. population has to do with making inferences. You do not need to worry about sample vs. population methods if all you want is a correlation coefficient, because it is a purely descriptive statistic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your fourth bullet point is completely unintelligible. Please clear that up if you would like people to help you with it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;@mpiktas A Spearman rank correlation is NOT the proper correlation coefficient to use here. To use that test all data must be ranked and discrete (unless &gt;= 2 values compete for a rank), i.e., they must be ordinal data. Maybe the HVOC table could be analyzed via Spearman's $\rho$, however more information must be provided by the poster to make that conclusion.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;@whuber Yes all data are discrete when represented on a computer, however in this case it seems like what BB01 was referring to was the scale of measurement, not the electronic representation of numbers.&lt;/p&gt;&#xA;" OwnerUserId="2660" LastEditorUserId="2660" LastEditorDisplayName="" LastEditDate="2011-03-21T03:33:02.500" LastActivityDate="2011-03-21T03:33:02.500" CommentCount="2" />
  <row Id="8209" PostTypeId="2" ParentId="8148" CreationDate="2011-03-12T18:11:59.990" Score="3" ViewCount="" Body="&lt;p&gt;There is a very wide variety of clustering methods, which are exploratory by nature, and I do not think that any of them, whether hierarchical or partition-based, relies on the kind of assumptions that one has to meet for analysing variance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having a look at the [MV] documentation in Stata to answer your question, I found this amusing quote at page 85:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Although some have said that there are as many cluster-analysis methods as there are people performing cluster analysis. This is a gross understatement! There exist infinitely more ways to perform a cluster analysis than people who perform them.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In that context, I doubt that there are any assumptions applying across clustering method. The rest of the text just sets out as a general rule that you need some form of &quot;dissimilarity measure&quot;, which need not even be a metric distance, to create clusters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is one exception, though, which is when you are clustering observations as part of a post-estimation analysis. In Stata, the &lt;code&gt;vce&lt;/code&gt; command comes with the following warning, at page 86 of the same source:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;If you are familiar with Stata’s large array of estimation commands, be careful to distinguish between cluster analysis (the cluster command) and the vce(cluster clustvar) option allowed with many estimation commands. Cluster analysis finds groups in data. The vce(cluster clustvar) option allowed with various estimation commands indicates that the observations are independent across the groups defined by the option but are not necessarily independent within those groups. A grouping variable produced by the cluster command will seldom satisfy the assumption behind the use of the vce(cluster clustvar) option.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Based on that, I would assume that independent observations are not required outside of that particular case. Intuitively, I would add that cluster analysis might even be used to the precise purpose of exploring the extent to which the observations are independent or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'll finish by mentioning that, at &lt;a href=&quot;http://books.google.fr/books?id=jVTEmrenxTQC&amp;amp;lpg=PA337&amp;amp;ots=J0HpEFmuN3&amp;amp;hl=en&amp;amp;pg=PA356&quot; rel=&quot;nofollow&quot;&gt;page 356&lt;/a&gt; of &lt;em&gt;Statistics with Stata&lt;/em&gt;, Lawrence Hamilton mentions standardized variables as an &quot;essential&quot; aspect of cluster analysis, although he does not go into more depth on the issue.&lt;/p&gt;&#xA;" OwnerUserId="3582" LastEditorUserId="3582" LastEditorDisplayName="" LastEditDate="2011-03-12T18:24:45.247" LastActivityDate="2011-03-12T18:24:45.247" />
  <row Id="8210" PostTypeId="2" ParentId="8187" CreationDate="2011-03-12T18:45:14.493" Score="2" ViewCount="" Body="&lt;p&gt;I might have wrongly understood the data (see my comment above), but what would be so wrong in simply t-testing the difference in means of the deaths-to-cases ratios between the two series, assuming that the states are clearly independent from each other?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A t-test shows that State 2 performs far better than State 1 in preventing deaths in its cases of the disease:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;year    state   ratio&#xA;2004    2   7.968504&#xA;2004    1   225&#xA;2005    2   9.464115&#xA;2005    1   242.6&#xA;2006    1   250.75&#xA;2006    2   10.57143&#xA;2007    1   237.5&#xA;2007    2   9.285141&#xA;2008    2   7.756345&#xA;2008    1   293&#xA;2009    1   364&#xA;2009    2   8.25&#xA;2010    2   9.284404&#xA;2010    1   329&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Given the huge difference between the two series, and given that you cannot design a proper experiment, it seems to me that no superior level of &quot;proof&quot; is really needed to proclaim one state's approach superior to the other.&lt;/p&gt;&#xA;" OwnerUserId="3582" LastEditorUserId="3582" LastEditorDisplayName="" LastEditDate="2011-03-13T02:49:01.330" LastActivityDate="2011-03-13T02:49:01.330" CommentCount="6" />
  <row Id="8211" PostTypeId="2" ParentId="8206" CreationDate="2011-03-12T18:55:43.080" Score="5" ViewCount="" Body="&lt;p&gt;Try something like this for a standalone version:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;bxp &amp;lt;- boxplot(rnorm(100), horizontal=TRUE, axes=FALSE)&#xA;mtext(c(&quot;Min&quot;,&quot;Max&quot;), side=3, at=bxp$stats[c(1,5)], line=-3)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Note that you can get some information when calling &lt;code&gt;boxplot&lt;/code&gt;, in particular the &quot;five numbers&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want it to be superimposed onto another graphic, use &lt;code&gt;add=T&lt;/code&gt; but replace &lt;code&gt;mtext&lt;/code&gt; by &lt;code&gt;text&lt;/code&gt;; you will need to set a $y$ value (which depend on the way you plot the other graphic).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A more complete example was given by &lt;a href=&quot;http://maths.anu.edu.au/~johnm/&quot; rel=&quot;nofollow&quot;&gt;John Maindonald&lt;/a&gt; (code should be on his website):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/nOPof.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-03-12T18:55:43.080" />
  <row Id="8212" PostTypeId="2" ParentId="8206" CreationDate="2011-03-12T19:06:21.527" Score="5" ViewCount="" Body="&lt;p&gt;I think you will find this produces something like your hand-drawn diagram.  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data    &amp;lt;- c(0.4, 0.7, 0.75, 0.82, 0.9)&#xA;endaxis &amp;lt;- c(0, 1)  # endpoints of axis&#xA;datamm  &amp;lt;- c(min(data), max(data))&#xA;boxplot(data, horizontal = TRUE, range = 0, ylim = endaxis,&#xA;                    axes = FALSE, col = &quot;grey&quot;, add = FALSE)&#xA;arrows(endaxis, 1,  datamm, 1,  code = 1, angle = 90, length = 0.1)&#xA;valuelabels &amp;lt;- c(endaxis[1], round(fivenum(data)[2], digits = 2) ,&#xA;                 round(fivenum(data)[4], digits = 2), endaxis[2]  ) &#xA;text(x = valuelabels, y = c(1.05, 1.25, 1.25, 1.05), labels = valuelabels)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/COYFG.png&quot; alt=&quot;R boxplot with value labels&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are probably better ways of doing it.&#xA;You may need to adapt it to fit your ROC plot, including changing &lt;code&gt;add = FALSE&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="2958" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-03-13T09:09:30.617" LastActivityDate="2011-03-13T09:09:30.617" CommentCount="1" />
  <row Id="8213" PostTypeId="1" AcceptedAnswerId="8214" CreationDate="2011-03-12T19:50:46.497" Score="6" ViewCount="143" Body="&lt;p&gt;For some volume reconstruction algorithm I'm working on, I need to detect an arbitrary number of circular patterns in 3d point data (coming from a LIDAR device). The patterns can be arbitrarily oriented in space, and be assumed to lie (although not perfectly) in thin 2d planes. Here is an example with two circles in the same plane (although remember this is a 3d space):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/FZxAU.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried many approaches.. the simplest (but the one working best so far) is clustering based on disjoint sets of the nearest neighbor graph. This works reasonably well when the patterns are far apart, but less so with circles like the ones in the example, really close to each other.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried K-means, but it doesn't do well: I suspect the circular point arrangement might not be well suited for it. Plus I have the additional problem of not knowing in advance the value of K.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried more complicated approaches, based on the detection of cycles in the nearest neighbor graph, but what I got was either too fragile or computationally expensive.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also read about a lot of related topics (Hough transform, etc) but nothing seems to apply perfectly in this specific context. Any idea or inspiration would be appreciated.&lt;/p&gt;&#xA;" OwnerUserId="3693" LastEditorUserId="3693" LastEditorDisplayName="" LastEditDate="2011-03-12T20:39:07.317" LastActivityDate="2011-03-12T22:32:41.720" Title="Detect circular patterns in point cloud data" Tags="&lt;clustering&gt;&lt;image-processing&gt;" AnswerCount="1" CommentCount="8" />
  <row Id="8214" PostTypeId="2" ParentId="8213" CreationDate="2011-03-12T22:32:41.720" Score="5" ViewCount="" Body="&lt;p&gt;A generalized &lt;a href=&quot;http://en.wikipedia.org/wiki/Hough_transform&quot; rel=&quot;nofollow&quot;&gt;Hough transform&lt;/a&gt; is exactly what you want.  The difficulty is to do it efficiently, because the space of circles in 3D has six dimensions (three for the center, two to orient the plane, one for the radius).  This seems to rule out a direct calculation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One possibility is to sneak up on the result through a sequence of simpler Hough transforms.  For instance, you could start with the (usual) Hough transform to detect planar subsets: those require only a 3D grid for the computation.  For each planar subset detected, slice the original points along that plane and perform a generalized Hough transform for circle detection.  This should work well provided the original image does not have a lot of coplanar points (other than the ones formed by the circles) that could drown out the signal generated by the circles.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the circle sizes have a predetermined upper bound you can potentially save a lot of computation: rather than looking at all pairs or triples of points in the original image, you can focus on pairs or triples within a bounded neighborhood of each point.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-03-12T22:32:41.720" CommentCount="4" />
  <row Id="8215" PostTypeId="2" ParentId="125" CreationDate="2011-03-12T23:55:47.807" Score="1" ViewCount="" Body="&lt;p&gt;My favourite first undergraduate text for bayesian statistics is by Bolstad, &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0470141158&quot; rel=&quot;nofollow&quot;&gt;Introduction to Bayesian Statistics&lt;/a&gt;.  If you're looking for something graduate level, this will be too elementary, but for someone who is new to statistics this is ideal.&lt;/p&gt;&#xA;" OwnerUserId="3694" LastActivityDate="2011-03-12T23:55:47.807" CommunityOwnedDate="2011-03-12T23:55:47.807" />
  <row Id="8216" PostTypeId="2" ParentId="125" CreationDate="2011-03-13T00:58:39.420" Score="0" ViewCount="" Body="&lt;p&gt;I've at least glanced at most of these on this list and none are as good as the new &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1439803544&quot; rel=&quot;nofollow&quot;&gt;Bayesian Ideas and Data Analysis&lt;/a&gt; in my opinion.&lt;/p&gt;&#xA;" OwnerUserId="2310" LastActivityDate="2011-03-13T00:58:39.420" CommunityOwnedDate="2011-03-13T00:58:39.420" />
  <row Id="8217" PostTypeId="2" ParentId="8160" CreationDate="2011-03-13T03:12:26.717" Score="2" ViewCount="" Body="&lt;p&gt;For a non-ordinal measure, you could try MDS (multi-dimensional scaling). This can be done easily in R. This will try to arrange the points on a line (1d in your case) in such a way that distances between points will be preserved.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some general comments: as you probably realize, the question is pretty vague, and not much can be said without knowing more about the data. For example, normalizing the variables (to zero mean and unit variance) may or may not be appropriate; weighing all variable equally may or may not be appropriate; etc. If this is not an exploratory analysis and you do have some 'correct' score in mind, then it may be appropriate to learn a set of weights either on a different dataset, or on a subset of your current dataset, and using these weights instead.&lt;/p&gt;&#xA;" OwnerUserId="3369" LastActivityDate="2011-03-13T03:12:26.717" />
  <row Id="8218" PostTypeId="2" ParentId="8192" CreationDate="2011-03-13T03:40:01.790" Score="0" ViewCount="" Body="&lt;p&gt;One thing which may be difficult to determine using sampling theory is $P(data|\rho&amp;gt;0.5)$.  I don't know of a sampling theory based decision problem which evaluates this directly (that is not to say it is impossible, I may just be ignorant).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You usually have to take a &quot;minimax&quot; approach.  That is, find the &quot;most favorable&quot; $\rho_1$ within the range $\rho&amp;gt;0.5$, and do a &lt;em&gt;specific&lt;/em&gt; test of $H_0:\rho=0.5$ vs $H_1:\rho=\rho_1$.  Usually the &quot;most favourable&quot; value is the MLE of $\rho$, which is the sample correlation, so you would test against $H_1:\rho=r$.  I believe this is called a &quot;severe test&quot;  because if you reject $H_1$ in favor of $H_0$, then you will also reject all the other possibilities in favor of $H_0$ (because $H_1$ was the &quot;best&quot; of the alternatives).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But note that this hypothesis test is all under the assumption that the model/structure is true.  If the correlation were to enter the model in a different functional form, then the test may be different.  It is very easy to think that the test is &quot;absolute&quot; because usually you don't explicitly specify a class of alternatives.  But the class of alternatives is always there, whether you specify it implicitly or explicitly.  So if you were being pedantic about it, you should really include the full model specification and any additional assumptions in both $H_0$ and $H_1$.  This will help you understand exactly what the test has and has not achieved.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-03-13T03:40:01.790" />
  <row Id="8220" PostTypeId="1" CreationDate="2011-03-13T07:09:44.603" Score="1" ViewCount="42" Body="&lt;p&gt;Hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd like to perform semi-supervised LDA (Latent Dirichlet Allocation) in the following sense:&#xA;I have several topics that I'd like to use, and have seed documents that relate to these topics. I'd like to run LDA to classify other documents, (and discover other topics). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would guess there is work done on that, as the problem is natural, and the LDA framework seem to suggest it, nevertheless, I'm not an expert and do not know about such work. &#xA;Can you guide me to papers / tools ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;thanks,&#xA;    yoad  &lt;/p&gt;&#xA;" OwnerUserId="3696" LastActivityDate="2011-03-13T07:09:44.603" Title="Can you point me to a semi-supervised LDA" Tags="&lt;unsupervised-learning&gt;" CommentCount="1" />
  <row Id="8221" PostTypeId="1" CreationDate="2011-03-13T09:50:31.517" Score="0" ViewCount="86" Body="&lt;p&gt;Dear Group,&#xA;I am trying to implement one shallow parser and one deep parser and trying to implement them in the four models namely, Naive Bayes, SVM, HMM and CRF. I am trying to implement in each one and compare the performances. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As per as I read the problems and their resolutions I am generally confident that I am doing fine. But, I was thinking back if I can discuss my approaches with some expert people.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have coded them and implemented in Python. I wanted to show my code too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If anyone knows all the formalisms and their Python implementations that would be great. &#xA;But if you want to see it part by part, some one some formalisms and some one else Python, &#xA;then also it would fit. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;May I discuss them? And may I send the code snippet?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Best Regards,&#xA;Subhabrata Banerjee. &lt;/p&gt;&#xA;" OwnerUserId="2329" LastActivityDate="2011-03-13T09:50:31.517" ClosedDate="2011-04-11T01:50:48.213" Title="Question On Naive Bayes, SVM, HMM and CRF" Tags="&lt;r&gt;&lt;regression&gt;&lt;machine-learning&gt;&lt;bayesian&gt;" CommentCount="2" />
  <row Id="8222" PostTypeId="1" CreationDate="2011-03-13T09:58:02.480" Score="4" ViewCount="125" Body="&lt;p&gt;I have been looking for computer game datasets, but so far I've only been able to find the 'Avatar History' dataset for WoW.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any other interesting datasets out there, possibly for other genres?&lt;/p&gt;&#xA;" OwnerUserId="37" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-13T11:54:19.637" LastActivityDate="2011-03-13T13:13:15.433" CommunityOwnedDate="2011-03-13T11:54:36.830" Title="Computer game datasets" Tags="&lt;data-mining&gt;&lt;dataset&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="1" />
  <row Id="8223" PostTypeId="2" ParentId="4099" CreationDate="2011-03-13T10:44:00.427" Score="2" ViewCount="" Body="&lt;p&gt;&lt;em&gt;Multicollinearity&lt;/em&gt; problem is well studied in actually most econometric textbooks. Moreover there is a good article in &lt;a href=&quot;http://en.wikipedia.org/wiki/Multicollinearity&quot; rel=&quot;nofollow&quot;&gt;wikipedia&lt;/a&gt; which actually summarizes most of the key issues.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In practice one starts to bear in mind the multicollinearity problem if it causes some visual signs of parameter instability (most of them are implied by non (poor) invertability of $X^TX$ matrix): &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;large changes in parameter estimates while performing rolling regressions or estimates on smaller sub-samples of the data&lt;/li&gt;&#xA;&lt;li&gt;averaging of parameter estimates, the latter may fall to be insignificant (by $t$ tests) even though junk-regression $F$ test shows high joint significance of the results&lt;/li&gt;&#xA;&lt;li&gt;VIF statistic (average value of auxiliary regressions) merely depends on your requirements to tolerance level, most practical suggestions put an acceptable tolerance to be lower than 0.2 or 0.1 meaning that corresponding averages of auxiliary regressions $R^2$ should be higher than 0.9 or 0.8 to detect the problem. Thus VIF should be larger than rule-of-thumb's 10 and 5 values. In small samples (less than 50 points) 5 is preferable, in larger you can go to larger values.  &lt;/li&gt;&#xA;&lt;li&gt;Condition index is an alternative to VIF in your case neither VIF nor CI show the problem is left, so you may be satisfied &lt;em&gt;statistically&lt;/em&gt; on this result, but...&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;probably not theoretically, since it may happen (and usually is the case) that you need all variables to be present in the model. Excluding relevant variables (omitted variable problem) will make biased and inconsistent parameter estimates anyway. On the other hand you may be forced to include all focus variables simply because your analysis is based on it. In data-mining approach though you are more technical in searching for the best fit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So keep in mind the alternatives (that I would use myself):&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;obtain more data points (recall that VIF requirements are smaller for larger data set and the explanatory variables if they are slowly varying, may change for some crucial points in time or cross-section)&lt;/li&gt;&#xA;&lt;li&gt;search for lattent factors through &lt;em&gt;principal components&lt;/em&gt; (the latter are orthogonal combinations so not multi-collinear by the construction, more over involve all explanatory variables)&lt;/li&gt;&#xA;&lt;li&gt;ridge-regression (it introduces small bias in parameter estimates, but makes them highly stable)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Some other tricks are in the wiki article noted above. &lt;/p&gt;&#xA;" OwnerUserId="2645" LastActivityDate="2011-03-13T10:44:00.427" />
  <row Id="8224" PostTypeId="2" ParentId="8222" CreationDate="2011-03-13T11:20:58.870" Score="3" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Starcraft I&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://eis.ucsc.edu/StarCraft_Data_Mining&quot; rel=&quot;nofollow&quot;&gt;Starcraft Data Mining Project&lt;/a&gt;, providing some game data.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://eis.ucsc.edu/StarCraftAICompetition&quot; rel=&quot;nofollow&quot;&gt;Starcraft AI Competition&lt;/a&gt;, does not directly provide data, but allows you to connect a program written by you with the game. Although I did not try it, I expect that data collection is possible this way ;).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If you are generally interested in data mining + gaming, you may find the project &lt;a href=&quot;http://robocode.sourceforge.net/&quot; rel=&quot;nofollow&quot;&gt;Robocode aka Java Robot Wars&lt;/a&gt; interesting, where you can program a bot for a simpler environment (simpler than Starcraft) and let it battle against other bots.&lt;/p&gt;&#xA;" OwnerUserId="264" LastActivityDate="2011-03-13T11:20:58.870" CommunityOwnedDate="2011-03-13T11:54:36.830" />
  <row Id="8225" PostTypeId="1" AcceptedAnswerId="8230" CreationDate="2011-03-13T12:02:38.533" Score="4" ViewCount="431" Body="&lt;p&gt;I have R data frame like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;        age group&#xA;1   23.0883     1&#xA;2   25.8344     1&#xA;3   29.4648     1&#xA;4   32.7858     2&#xA;5   33.6372     1&#xA;6   34.9350     1&#xA;7   35.2115     2&#xA;8   35.2115     2&#xA;9   35.2115     2&#xA;10  36.7803     1&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I need to get data frame in the following form:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;group mean     sd&#xA;1     34.5     5.6&#xA;2     32.3     4.2&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Group number may vary, but their names and quantity could be obtained by calling &lt;code&gt;levels(factor(data$group))&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What manipulations should be done with the data to get the result?&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="3376" LastEditorUserId="3376" LastEditorDisplayName="" LastEditDate="2011-03-13T15:20:27.067" LastActivityDate="2011-03-15T10:17:52.233" Title="How to summarize data by group in R?" Tags="&lt;r&gt;&lt;data-transformation&gt;" AnswerCount="4" CommentCount="5" FavoriteCount="2" />
  <row Id="8226" PostTypeId="2" ParentId="8225" CreationDate="2011-03-13T12:12:54.217" Score="5" ViewCount="" Body="&lt;p&gt;Hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One possibility is to use the &lt;a href=&quot;http://stat.ethz.ch/R-manual/R-patched/library/stats/html/aggregate.html&quot; rel=&quot;nofollow&quot;&gt;aggregate function&lt;/a&gt;.&#xA;For instance,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;aggregate(data$age, by=list(data$group), FUN=mean)[2]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;gives you the second column of the desired result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Hyperlink edited (tkank you chl!)&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="3019" LastEditorUserId="3019" LastEditorDisplayName="" LastEditDate="2011-03-13T12:30:10.263" LastActivityDate="2011-03-13T12:30:10.263" CommentCount="4" />
  <row Id="8227" PostTypeId="2" ParentId="8225" CreationDate="2011-03-13T12:15:29.533" Score="3" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Edited:&lt;/strong&gt; According to chl's suggestions&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The function you are looking for is called &quot;tapply&quot; which applies a function per group specified by a factor.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# create some artificial data&#xA;set.seed(42)&#xA;groups &amp;lt;- 5&#xA;&#xA;agedat &amp;lt;- c()&#xA;groupdat &amp;lt;- c()&#xA;&#xA;for(group in 1:groups){&#xA;    agedat &amp;lt;- c(agedat,rnorm(100,mean=0 + group,1/group))&#xA;    groupdat &amp;lt;- c(groupdat,rep(group,100))&#xA;}&#xA;dat &amp;lt;- data.frame(&quot;age&quot;=agedat,&quot;group&quot;=factor(groupdat))&#xA;&#xA;# calculate mean and stdev age per group&#xA;res &amp;lt;- rbind.data.frame(group=1:5, with(dat, tapply(age, group, function(x) c(mean(x), sd(x)))))&#xA;names(res) &amp;lt;- paste(&quot;group&quot;,1:5)&#xA;row.names(res)[2:3] &amp;lt;- c(&quot;mean&quot;,&quot;sd&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I really suggest to work through a basic R tutorial explaining all commonly used datastructures and methods. Otherwise you will get stuck every inch during programming. See &lt;a href=&quot;http://stats.stackexchange.com/questions/138/resources-for-learning-r&quot;&gt;this question&lt;/a&gt; for a collection of free available resources.&lt;/p&gt;&#xA;" OwnerUserId="264" LastEditorUserId="264" LastEditorDisplayName="" LastEditDate="2011-03-13T18:21:19.523" LastActivityDate="2011-03-13T18:21:19.523" CommentCount="9" />
  <row Id="8228" PostTypeId="2" ParentId="1352" CreationDate="2011-03-13T12:26:51.003" Score="1" ViewCount="" Body="&lt;p&gt;Answering with an aphorism, I believe that your study design will be successful as soon as it actually exists in its full-fledged form. The game of reviewing as it is played in academia is primarily a game of academics showing to each other that they have &lt;em&gt;not&lt;/em&gt; completed that step in its full depth, e.g. by violating assumptions or omitting biases where they should be expected. If study design is a skill, it's the skill of making your research bulletproof to these critics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your question is very interesting but I am afraid that there is no short answer. To the best of my knowledge, the only way to learn thoroughly about research designs, whether experimental or observational, is to read the literature in your field of specialisation, and then to go the extra mile by connecting with academics in order to learn even more on how they work, in order to, eventually, write up your own research design.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my field (European political science), we generically offer &quot;research design&quot; courses that span over all types of studies, but even then we miss important trends and also lack a deep understanding of our methods. After taking at least three of these courses, I have become convinced that no academic resource can replace learning from other academics, before confronting real-world settings directly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I guess that your field also has these 'methods journals' that can be as painfully boring and complex to the outsider than they are helpful and interesting to actual 'study designers' -- and so would recommend that you start digging this literature first, eventually tracking down the recurring bibliographic items that might help you most with study design in biology/ecology. Google Scholar definitely flags a few books with the words 'ecology research methods'.&lt;/p&gt;&#xA;" OwnerUserId="3582" LastActivityDate="2011-03-13T12:26:51.003" CommunityOwnedDate="2011-03-13T12:26:51.003" />
  <row Id="8229" PostTypeId="2" ParentId="8225" CreationDate="2011-03-13T12:38:57.817" Score="2" ViewCount="" Body="&lt;p&gt;In addition to existing suggestions, you might want to check out the &lt;code&gt;describe.by&lt;/code&gt; function in the &lt;code&gt;psych&lt;/code&gt; package.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It provides a number of descriptive statistics including the mean and standard deviation based on a grouping variable.&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2011-03-13T12:38:57.817" CommentCount="1" />
  <row Id="8230" PostTypeId="2" ParentId="8225" CreationDate="2011-03-13T12:44:13.913" Score="10" ViewCount="" Body="&lt;p&gt;Here is the &lt;strong&gt;plyr&lt;/strong&gt; one line variant using &lt;strong&gt;ddply&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dt &amp;lt;- data.frame(age=rchisq(20,10),group=sample(1:2,20,rep=T))&#xA;ddply(dt,~group,summarise,mean=mean(age),sd=sd(age))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here is another one line variant using new package &lt;strong&gt;data.table&lt;/strong&gt;. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dtf &amp;lt;- data.frame(age=rchisq(100000,10),group=factor(sample(1:10,100000,rep=T)))&#xA;dt &amp;lt;- data.table(dt)&#xA;dt[,list(mean=mean(age),sd=sd(age)),by=group]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This one is faster, though this is noticeable only on table with 100k rows. Timings on my Macbook Pro with 2.53 Ghz Core 2 Duo processor and R 2.11.1:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; system.time(aa &amp;lt;- ddply(dtf,~group,summarise,mean=mean(age),sd=sd(age)))&#xA;utilisateur     système      écoulé &#xA;      0.513       0.180       0.692 &#xA;&amp;gt; system.time(aa &amp;lt;- dt[,list(mean=mean(age),sd=sd(age)),by=group])&#xA;utilisateur     système      écoulé &#xA;      0.087       0.018       0.103 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Further savings are possible if we use &lt;code&gt;setkey&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; setkey(dt,group)&#xA;&amp;gt; system.time(dt[,list(mean=mean(age),sd=sd(age)),by=group])&#xA;utilisateur     système      écoulé &#xA;      0.040       0.007       0.048 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-03-15T10:17:52.233" LastActivityDate="2011-03-15T10:17:52.233" CommentCount="3" />
  <row Id="8231" PostTypeId="2" ParentId="8222" CreationDate="2011-03-13T12:45:03.647" Score="1" ViewCount="" Body="&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;John Myles White has a &lt;a href=&quot;http://www.johnmyleswhite.com/notebook/2009/11/15/the-top-scores-for-canabalt-take-2/&quot; rel=&quot;nofollow&quot;&gt;dataset and analysis of Canabalt scores as posted on Twitter&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Stats at Berkeley has a dataset for a &lt;a href=&quot;http://www.stat.berkeley.edu/users/statlabs/labs.html#video&quot; rel=&quot;nofollow&quot;&gt;Video Games Survey&lt;/a&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-03-13T13:13:15.433" LastActivityDate="2011-03-13T13:13:15.433" CommunityOwnedDate="2011-03-13T12:45:03.647" />
  <row Id="8232" PostTypeId="1" CreationDate="2011-03-13T13:17:04.273" Score="2" ViewCount="168" Body="&lt;p&gt;Hi all,&lt;br&gt;&#xA;I'm trying to figure out the usage of the $\chi^2$ distribution. In other words, in what kind of situations it occurs and how is it useful in that situations.&lt;br&gt;&#xA;I read the wikipedia definition of $\chi^2$. Can someone give an example of a $\chi^2$ distribution so that I can better understand it?&lt;/p&gt;&#xA;" OwnerUserId="3688" LastEditorUserId="3688" LastEditorDisplayName="" LastEditDate="2011-03-14T06:54:07.003" LastActivityDate="2011-03-14T06:54:07.003" Title="What is an example of a chi-squared distribution?" Tags="&lt;chi-squared&gt;" AnswerCount="1" CommentCount="6" />
  <row Id="8233" PostTypeId="2" ParentId="8232" CreationDate="2011-03-13T14:09:18.563" Score="1" ViewCount="" Body="&lt;p&gt;I found &lt;a href=&quot;http://www.youtube.com/watch?v=dXB3cUGnaxQ&quot; rel=&quot;nofollow&quot;&gt;this video&lt;/a&gt; useful in finding the solution to my question.&lt;/p&gt;&#xA;" OwnerUserId="3688" LastActivityDate="2011-03-13T14:09:18.563" CommentCount="1" />
  <row Id="8234" PostTypeId="1" AcceptedAnswerId="8244" CreationDate="2011-03-13T14:46:36.113" Score="3" ViewCount="73" Body="&lt;p&gt;Some datasets come with full-lowercase value labels, and I end up with graphs and tables showing results for &quot;egypt&quot;, &quot;jordan&quot; and &quot;saudi arabia&quot; instead of the capitalized country names.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I guess that the &lt;code&gt;proper()&lt;/code&gt; string function can do something for me, but I am not finding the right way to write the code for Stata 11 that will capitalize all value labels for a given variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I basically need to run the &lt;code&gt;proper()&lt;/code&gt; function on all value labels on the variable, and then assign them to the variable. Is that possible using a &lt;code&gt;foreach&lt;/code&gt; loop and macros in Stata?&lt;/p&gt;&#xA;" OwnerUserId="3582" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-13T14:49:27.283" LastActivityDate="2011-03-13T18:55:16.907" Title="Capitalizing value labels in Stata" Tags="&lt;stata&gt;" AnswerCount="1" FavoriteCount="2" />
  <row Id="8235" PostTypeId="2" ParentId="8137" CreationDate="2011-03-13T14:51:25.300" Score="1" ViewCount="" Body="&lt;p&gt;Found solution myself. Maybe someone could use it:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#step 1: preparing data&#xA;ageMetaData &amp;lt;- ddply(data,~group,summarise,&#xA;            mean=mean(age),&#xA;            sd=sd(age),&#xA;            min=min(age),&#xA;            max=max(age),&#xA;            median=median(age),&#xA;            Q1=summary(age)['1st Qu.'],&#xA;            Q3=summary(age)['3rd Qu.']&#xA;            )&#xA;#step 2: correction for outliers&#xA;out &amp;lt;- data.frame() #initialising storage for outliers&#xA;for(group in 1:length((levels(factor(data$group))))){&#xA;    bps &amp;lt;- boxplot.stats(data$age[data$group == group],coef=1.5) &#xA;    ageMetaData[ageMetaData$group == group,]$min &amp;lt;- bps$stats[1] #lower wisker&#xA;    ageMetaData[ageMetaData$group == group,]$max &amp;lt;- bps$stats[5] #upper wisker&#xA;    if(length(bps$out) &amp;gt; 0){ #adding outliers&#xA;        for(y in 1:length(bps$out)){&#xA;            pt &amp;lt;-data.frame(x=group,y=bps$out[y]) &#xA;            out&amp;lt;-rbind(out,pt) &#xA;        }&#xA;    }&#xA;}&#xA;#step 3: drawing&#xA;p &amp;lt;- ggplot(ageMetaData, aes(x = group,y=mean)) &#xA;p &amp;lt;- p + geom_errorbar(aes(ymin=min,ymax=max),linetype = 1,width = 0.5) #main range&#xA;p &amp;lt;- p + geom_crossbar(aes(y=median,ymin=Q1,ymax=Q3),linetype = 1,fill='white') #box&#xA;# drawning outliers if any&#xA;if(length(out) &amp;gt;0) p &amp;lt;- p + geom_point(data=out,aes(x=x,y=y),shape=4) &#xA;p &amp;lt;- p + scale_x_discrete(name= &quot;Group&quot;)&#xA;p &amp;lt;- p + scale_y_continuous(name= &quot;Age&quot;)&#xA;p&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The quantile data resulution is ugly, but works. Maybe there is another way.&#xA;The result looks like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/n7bAW.png&quot; alt=&quot;ggplot2 boxplot with limit lines&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also improved boxplot a little:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;added second smaller dotted errorbar to reflect sd range. &lt;/li&gt;&#xA;&lt;li&gt;added point to reflect mean&lt;/li&gt;&#xA;&lt;li&gt;removed background&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;maybe this also could be useful to someone:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;p &amp;lt;- ggplot(ageMetaData, aes(x = group,y=mean)) &#xA;p &amp;lt;- p + geom_errorbar(aes(ymin=min,ymax=max),linetype = 1,width = 0.5) #main range&#xA;p &amp;lt;- p + geom_crossbar(aes(y=median,ymin=Q1,ymax=Q3),linetype = 1,fill='white') #box&#xA;p &amp;lt;- p + geom_errorbar(aes(ymin=mean-sd,ymax=mean+sd),linetype = 3,width = 0.25) #sd range&#xA;p &amp;lt;- p + geom_point() # mean&#xA;# drawning outliers if any&#xA;if(length(out) &amp;gt;0) p &amp;lt;- p + geom_point(data=out,aes(x=x,y=y),shape=4) &#xA;p &amp;lt;- p + scale_x_discrete(name= &quot;Group&quot;)&#xA;p &amp;lt;- p + scale_y_continuous(name= &quot;Age&quot;)&#xA;p + opts(panel.background = theme_rect(fill = &quot;white&quot;,colour = NA))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The result is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/uiWn4.png&quot; alt=&quot;advanced boxplot with ggplot2&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and the same data with smaller range (boxplot &lt;code&gt;coef = 0.5&lt;/code&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/jgwgn.png&quot; alt=&quot;boxplot with outliers&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="3376" LastEditorUserId="3376" LastEditorDisplayName="" LastEditDate="2011-03-13T19:22:24.783" LastActivityDate="2011-03-13T19:22:24.783" CommentCount="3" />
  <row Id="8236" PostTypeId="1" AcceptedAnswerId="8275" CreationDate="2011-03-13T14:52:13.300" Score="8" ViewCount="130" Body="&lt;p&gt;I have a dataset of events that happened during the same period of time. Each event has a type (there are few different types, less then ten) and a location, represented as a 2D point.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to check if there is any correlation between types of events, or between the type and the location. For example, maybe events of type A usually don't occur where events of type B do. Or maybe in some area, there are mostly events of type C.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What kind of tools could I use to perform this ? Being a novice in statistical analysis, my first idea was to use some kind of PCA (Principal Component Analysis) on this dataset to see if each type of event had its own component, or maybe some shared the same (ie were correlated) ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have to mention that my dataset is of the order of 500'000 points $(x, y, type)$, thus making things a bit harder to deal with.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: As noted in the answers below and the comments, the way to go is to model this as a marked point process, and then use R to do all the heavy-lifting, as explained in details in this workshop report : &lt;a href=&quot;http://www.csiro.edu.au/resources/Spatial-Point-Patterns-in-R.html&quot; rel=&quot;nofollow&quot;&gt;http://www.csiro.edu.au/resources/Spatial-Point-Patterns-in-R.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="3699" LastEditorUserId="3699" LastEditorDisplayName="" LastEditDate="2011-03-15T16:27:00.380" LastActivityDate="2011-03-15T16:27:00.380" Title="How to find relationships between different types of events (defined by their 2D location)?" Tags="&lt;correlation&gt;&lt;pca&gt;&lt;multivariate-analysis&gt;&lt;point-process&gt;" AnswerCount="2" CommentCount="6" />
  <row Id="8237" PostTypeId="1" CreationDate="2011-03-13T15:15:05.713" Score="5" ViewCount="322" Body="&lt;p&gt;Hi All,&lt;br&gt;&#xA;I'm trying to understand the logic behind the ANOVA F-test in Simple Linear Regression Analysis. The question I have is like follows. When the F value, i.e. &#xA;&lt;code&gt;MSR/MSE&lt;/code&gt; is large we accept the model as significant. What is the logic behind this?&lt;/p&gt;&#xA;" OwnerUserId="3688" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-13T20:22:49.233" LastActivityDate="2011-03-13T21:38:19.687" Title="Logic behind the ANOVA F-test in simple linear regression" Tags="&lt;regression&gt;&lt;anova&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="2" />
  <row Id="8238" PostTypeId="2" ParentId="290" CreationDate="2011-03-13T15:23:49.450" Score="1" ViewCount="" Body="&lt;p&gt;I have listed some resources on my &lt;a href=&quot;http://f.briatte.org/teaching/quanti/stata.html&quot; rel=&quot;nofollow&quot;&gt;Stata page&lt;/a&gt;. It includes links to the UCLA and Princeton tutorials, as well as a few more resources in several formats. The &quot;&lt;a href=&quot;http://f.briatte.org/teaching/quanti/stata.html#download&quot; rel=&quot;nofollow&quot;&gt;Stata Guide&lt;/a&gt;&quot; document, which is very much a work in progress at that stage, is my personal contribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of the fantastic things about Stata is the wealth of online and offline documentation, but never forget that the first resource to learn Stata is the very good set of documentation pages that you can access right away from Stata through the &lt;code&gt;help&lt;/code&gt; command (&lt;a href=&quot;http://www.stata.com/help.cgi?help&quot; rel=&quot;nofollow&quot;&gt;also available online&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you prefer using books, Alan Acock's &lt;em&gt;A Gentle Introduction to Stata&lt;/em&gt; combined with Lawrence Hamilton's &lt;em&gt;Statistics with Stata&lt;/em&gt; will get you at a fairly high level of proficiency. You might then add a third book (like &lt;a href=&quot;http://www.stata.com/bookstore/regmodcdvs.html&quot; rel=&quot;nofollow&quot;&gt;Long and Freese&lt;/a&gt;'s excellent handbook) focused on a special research interest.&lt;/p&gt;&#xA;" OwnerUserId="3582" LastActivityDate="2011-03-13T15:23:49.450" />
  <row Id="8239" PostTypeId="1" CreationDate="2011-03-13T15:25:27.093" Score="3" ViewCount="111" Body="&lt;p&gt;I have just discovered, through &lt;a href=&quot;http://stats.stackexchange.com/questions/8232/understanding-chi-squared-distribution/8233#8233&quot;&gt;this answer&lt;/a&gt;, a &lt;a href=&quot;http://www.youtube.com/user/khanacademy#grid/user/1328115D3D8A2566&quot; rel=&quot;nofollow&quot;&gt;nice video resource&lt;/a&gt; to share with students in introductory statistics courses. The lectures, produced by the &lt;a href=&quot;http://www.khanacademy.org/#Statistics&quot; rel=&quot;nofollow&quot;&gt;Khan Academy&lt;/a&gt;, are fairly didactic and well illustrated. I could not find the author of the videos.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any other set of video courses which I could recommend to my students, who are social scientists taking an introductory course in statistics? The aforementioned set is software-independent and free to watch, two conditions that I need to maintain.&lt;/p&gt;&#xA;" OwnerUserId="3582" LastActivityDate="2011-03-13T15:25:27.093" Title="Introductory statistics video courses" Tags="&lt;video&gt;" CommentCount="5" FavoriteCount="3" />
  <row Id="8240" PostTypeId="1" CreationDate="2011-03-13T16:10:01.597" Score="6" ViewCount="94" Body="&lt;p&gt;I have N uniform-random points $p_j$ in a box in $E^d$,&#xA;$a_i \le x_i \le b_i$,&#xA;and want to estimate the expected distance of the point nearest the origin in $L_q$:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\quad$ nearest( points $p_j$, box $a_i .. b_i$, $q$ ) $\;\equiv\;$&#xA;$\min_{j=1..N}$ $\sum_{i=1..d}$  |$p_{ji}|^q$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The box might straddle 0 in some dimensions, $a_i &amp;lt; 0 &amp;lt; b_i$.&lt;br&gt;&#xA;Also I'd like the estimator to work for $0 &amp;lt; q &amp;lt; 1$ too, the &quot;fractional metric&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(16 March) &#xA;Let's try to do a simpler case: $L_1$, unit cube 0 $\le x_i \le$ 1.&lt;br&gt;&#xA;Geometrically, we want (correct me) the diagonal slice&#xA;of the unit cube with volume $\frac{1}{N+1}$.&lt;br&gt;&#xA;(Does anyone have a picture or applet&#xA;of a 3d cube sliced into equal-volume slices ?)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If d is big enough for a central limit theorem to hold,&lt;br&gt;&#xA;$\quad \sum_{i=1..d} uniform_i&#xD;&#xA;\ \sim\ \mathcal{N}( \frac{d}{2}, \frac{d}{12} )$;&lt;br&gt;&#xA;so&#xA;$\mathcal{N}^{-1}( \frac{1}{N+1} )$ gives approximately the cut,&#xA;and the expected nearest distance, that I want.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In Python with scipy.stats, this is&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def cutcube( dim, vol ):&#xA;    &quot;&quot;&quot; cut a slice of the unit cube in E^dim with volume vol&#xA;        normal approximation: cutcube( 3, 1/6 ) = .339 ~ 1/3&#xA;        vol 1/(N+1) -&amp;gt; E nearest of N random points in the cube ?&#xA;    &quot;&quot;&quot;&#xA;    return norm.ppf( vol, loc=dim/2, scale=np.sqrt( dim/12 )) / dim&#xA;&#xA;cutcube( dim=2, vol=1/10 ) = 0.24&#xA;cutcube( dim=4, vol=1/10 ) = 0.32&#xA;cutcube( dim=8, vol=1/10 ) = 0.37&#xA;cutcube( dim=16, vol=1/10 ) = 0.41&#xA;cutcube( dim=32, vol=1/10 ) = 0.43&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="557" LastEditorUserId="557" LastEditorDisplayName="" LastEditDate="2011-03-18T13:20:32.560" LastActivityDate="2011-03-18T13:20:32.560" Title="Estimate the nearest of N random points in a box in E^d?" Tags="&lt;central-limit-theorem&gt;&lt;extreme-value&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="8241" PostTypeId="2" ParentId="8240" CreationDate="2011-03-13T16:51:04.557" Score="5" ViewCount="" Body="&lt;p&gt;First, here are some commonly known facts which will be useful.  Suppose i.i.d $X_1, \cdots, X_n$ have the cumulative distribution function $F(X) = P[X \leq x]$, then cumulative distribution function of $\min X_i$ is $G(X) = 1-(1-F(X))^n.$  The expected value of a nonnegative random variable in terms of its cdf is $E[X] = \int_0^\infty G(x) dx$. The median is $G^{-1}(1/2)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this problem, the cdf of the distance from the center is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$F(r) = P[|X| \leq r] = \frac{vol(B_q(r) \cap E^d)}{vol(E^d)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $B_q(r) = \{x: |x| \leq q\}$ and $vol(S)$ denotes the volume (Lesbesgue measure) of $S$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The tricky part is evaluating $vol(B_q(r) \cap E^d)$.&#xA;We will deal with the special case that $a_i \geq 0$ for $1 \leq i \leq d$.&#xA;The general case follows from subdividing $E^d$ by quadrant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since we are working with the $q$-norm, it will be invaluable to do a change of coordinates, letting $y_i = x_i^q$.  (Remember that we are assuming that the $x_i$ are nonnegative.)  Then we have $\frac{dy_i}{dx_i} = qx_i^{q-1}$, so &#xA;$$dx_i = q^{-1} x_i^{1-q}dy_i = (1/q) y_i^{(1-q)/q} dy_i.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For notational clarity, we illustrate the calculation for $d=2$, and for general $d$ our methods should extend in the obvious way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming that $|(a_1, a_2)| \leq r$, we have&#xA;$$&#xD;&#xA;vol(B_q(r) \cap E^d) = \int_{a_1}^{\min(b_1, r^q)} \int_{a_2}^{\min(b_2, r^q - y_1)} (1/q) y_2^{(1-q)/q} dy_2 (1/q) y_1^{(1-q)/q} dy_1&#xD;&#xA;$$&#xA;$$= (1/q)^2 \int_{a_1}^{\min(b_1, r^q)} \int_{a_2}^{\min(b_2, r^q - y_1)} y_2^{(1-q)/q} dy_2 y_1^{(1-q)/q} dy_1$$&#xA;$$=(1/q)^2 \int_{a_1}^{\min(b_1, r^q)} q [\min(b_2, r^q - y_1)^{1/q} - a_2^{1/q}]y_1^{(1-q)/q} dy_1$$&#xA;$$=(1/q) \int_{a_1}^{\min(b_1, r^q)} [\min(b_2, r^q - y_1)^{1/q} - a_2^{1/q}]y_1^{(1-q)/q} dy_1$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We will want to split the preceding integral into the cases when $y_1 &amp;gt; r^q - b_2$ and $y_1 \leq r^q - b_2$.  The integral in the first case is&#xA;$$(1/q) \int_{a_1}^{r^q - b_2} [(r^q - y_1)^{1/q} - a_2^{1/q}]y_1^{(1-q)/q} dy_1$$&#xA;and the integral in the second case is&#xA;$$&#xD;&#xA;(1/q) \int_{r^q - b_2}^{\min(b_1, r^q)} [b_2^{1/q} - a_2^{1/q}]y_1^{(1-q)/q} dy_1.&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reader is strongly advised to find a Computer Algebra System at this point.&lt;/p&gt;&#xA;" OwnerUserId="3567" LastEditorUserId="3567" LastEditorDisplayName="" LastEditDate="2011-03-14T02:20:23.793" LastActivityDate="2011-03-14T02:20:23.793" CommentCount="1" />
  <row Id="8242" PostTypeId="1" AcceptedAnswerId="8245" CreationDate="2011-03-13T17:21:18.183" Score="8" ViewCount="204" Body="&lt;p&gt;I would like to use R to plot out something like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/oY7UO.png&quot; alt=&quot;Something like this&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It would seem possible but highly complex to keep track of the coordinates, width, height, etc. Intuitively it would seem best to treat each cell as a new plot and transform the coordinates for each cell. Is there a way to do this in R? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;thanks!&lt;/p&gt;&#xA;" OwnerUserId="812" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-13T20:20:12.920" LastActivityDate="2011-03-13T20:20:12.920" Title="Plotting sparklines in R" Tags="&lt;r&gt;&lt;data-visualization&gt;&lt;tables&gt;&lt;sparkline&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="1" />
  <row Id="8243" PostTypeId="1" CreationDate="2011-03-13T17:25:26.993" Score="1" ViewCount="181" Body="&lt;p&gt;I need to find partial derivatives for:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$L=\frac{1}{2}\sum_{i=1}^{n} w_{i}^{2}\sigma_{i}^{2} -\lambda \left( \sum_{i=1}^{n} w_{i} \bar{r_{i}} - \bar{r} \right) -\mu \left( \sum_{i=1}^{n} w_{i}-1 \right)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;with 5 variables, I get 5 partial derivatives $\frac{\partial L}{\partial w_{1}}$, $\frac{\partial L}{\partial w_{2}}$, $\frac{\partial L}{\partial w_{3}}$, $\frac{\partial L}{\partial \lambda}$ and $\frac{\partial L}{\partial \mu}$ with which I need to solve for $w_{1}$, $w_{2}$ and $w_{3}$. For all $i$, $\sigma_{i}$ is known. I don't want to manually differentiate; it could be more cumbersome with more restrictions. Later, I need to find the point where the minimum variance is located; i.e. to find out the optimal values for $\lambda$, $\mu$ and $w_{i}$ for all $i$ where $i\in\{1,2,3\}$. I need to vary the 5 variables to find the optimal solutions (a bit like in Excel but no such thing currently available) and then show the frontier graphically. So my question is which program would you use to solve such thing?&lt;/p&gt;&#xA;" OwnerUserId="2914" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-13T20:10:10.260" LastActivityDate="2011-03-14T09:10:13.773" Title="Program to compute partial derivatives" Tags="&lt;optimization&gt;" AnswerCount="2" CommentCount="5" FavoriteCount="1" />
  <row Id="8244" PostTypeId="2" ParentId="8234" CreationDate="2011-03-13T18:55:16.907" Score="4" ViewCount="" Body="&lt;p&gt;Yes.  First let's create some sample data with labels for testing:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;clear&#xA;drawnorm x, n(10)&#xA;gen byte v = int(4+x)&#xA;drop x&#xA;label define types 0 &quot;zero&quot; 1 &quot;one&quot; 2 &quot;two&quot; 3 &quot;three&quot; 4 &quot;four&quot; 5 &quot;five&quot; 6 &quot;six&quot;&#xA;label list types&#xA;label values v types&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here's a macro to capitalize the values associated with the variable &quot;v&quot;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;local varname v&#xA;local sLabelName: value label `varname'&#xA;di &quot;`sLabelName'&quot;&#xA;&#xA;levelsof `varname', local(xValues)&#xA;foreach x of local xValues {&#xA;    local sLabel: label (`varname') `x', strict&#xA;    local sLabelNew =proper(&quot;`sLabel'&quot;)&#xA;    noi di &quot;`x': `sLabel' ==&amp;gt; `sLabelNew'&quot;&#xA;    label define `sLabelName' `x' &quot;`sLabelNew'&quot;, modify&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;After running it, check the results:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;label list types&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-03-13T18:55:16.907" CommentCount="1" />
  <row Id="8245" PostTypeId="2" ParentId="8242" CreationDate="2011-03-13T19:59:54.457" Score="8" ViewCount="" Body="&lt;p&gt;I initially managed to produce something approaching your original picture with some quick and dirty R code (see this &lt;a href=&quot;https://gist.github.com/868376&quot; rel=&quot;nofollow&quot;&gt;gist&lt;/a&gt;), until I discovered that the &lt;a href=&quot;http://cran.r-project.org/web/packages/sparkTable/index.html&quot; rel=&quot;nofollow&quot;&gt;sparkTable&lt;/a&gt; package should do this very much better, provided you are willing to use $\LaTeX$. (In the meantime, it has also been pointed out by @Bernd!)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an example, from &lt;code&gt;help(sparkEPS)&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/C463D.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It should not be too difficult to arrange this the way you want.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-03-13T19:59:54.457" CommentCount="1" />
  <row Id="8246" PostTypeId="1" CreationDate="2011-03-13T20:45:44.760" Score="4" ViewCount="49" Body="&lt;p&gt;I am evaluating a Quadratic Discriminant Analysis (QDA) classifier on a high-dimensionality feature set.  The features come from highly non-Gaussian distributions.  However, when I transform the features to each have a Gaussian distribution in two different ways, the resulting classifiers performs worse than QDA applied on the raw features with the following three metrics:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Accuracy (classes are balanced)&lt;/li&gt;&#xA;&lt;li&gt;Area under the ROC&lt;/li&gt;&#xA;&lt;li&gt;A probabilistic metric&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The first way of transforming each feature to a Gaussian distribution disregarded the class.  For each feature, it found the parameters of the the corresponding distribution, used the distribution's CDF function or approximation for each data point, and then used the Gaussian's inverse CDF.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second way did the same thing, but included class labels and treated data from each class independently.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any idea why this occurs?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have confirmed that it is not due to ...&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;A bug in the code&lt;/li&gt;&#xA;&lt;li&gt;Distributions changing from train set to test set&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="3595" LastActivityDate="2011-03-13T20:45:44.760" Title="Using QDA for Non-Gaussian distributions" Tags="&lt;distributions&gt;&lt;classification&gt;&lt;discriminant-analysis&gt;" CommentCount="6" FavoriteCount="0" />
  <row Id="8247" PostTypeId="2" ParentId="8237" CreationDate="2011-03-13T21:04:34.900" Score="11" ViewCount="" Body="&lt;p&gt;In the simplest case, when you have only one predictor (simple regression), say $X_1$, the $F$-test tells you whether including $X_1$ does explain a larger part of the variance observed in $Y$ compared to the null model (intercept only). The idea is then to test if the added explained variance (total variance, TSS, minus residual variance, RSS) is large enough to be considered as a &quot;significant quantity&quot;. We are here comparing a model with one predictor, or explanatory variable, to a baseline which is just &quot;noise&quot; (nothing except the grand mean).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Likewise, you can compute an $F$ statistic in a multiple regression setting: In this case, it amounts to a test of &lt;em&gt;all predictors&lt;/em&gt; included in the model, which under the HT framework means that we wonder whether any of them is useful in predicting the response variable. This is the reason why you may encounter situations where the $F$-test for the whole model is significant whereas some of the $t$ or $z$-tests associated to each regression coefficient are not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The $F$ statistic looks like&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ F = \frac{(\text{TSS}-\text{RSS})/(p-1)}{\text{RSS}/(n-p)},$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $p$ is the number of model parameters and $n$ the number of observations. This quantity should be referred to an $F_{p-1,n-p}$ distribution for a critical or $p$-value. It applies for the simple regression model as well, and obviously bears some analogy with the classical ANOVA framework. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Sidenote.&lt;/strong&gt;&#xA;When you have more than one predictor, then you may wonder whether considering only a subset of those predictors &quot;reduces&quot; the quality of model fit. This corresponds to a situation where we consider &lt;em&gt;nested models&lt;/em&gt;. This is exactly the same situation as the above ones, where we compare a given regression model with a &lt;em&gt;null&lt;/em&gt; model (no predictors included). In order to assess the reduction in explained variance, we can compare the residual sum of squares (RSS) from both model (that is, what is left unexplained once you account for the effect of predictors present in the model). Let $\mathcal{M}_0$ and $\mathcal{M}_1$ denote the base model (with $p$ parameters) and a model with an additional predictor ($q=p+1$ parameters), then if $\text{RSS}_{\mathcal{M}_1}-\text{RSS}_{\mathcal{M}_0}$ is small, we would consider that the smaller model performs as good as the larger one. A good statistic to use would the ratio of such SS, $(\text{RSS}_{\mathcal{M}_1}-\text{RSS}_{\mathcal{M}_0})/\text{RSS}_{\mathcal{M}_0}$, weighted by their degrees of freedom ($p-q$ for the numerator, and $n-p$ for the denominator). As already said, it can be shown that this quantity follows an $F$ (or Fisher-Snedecor) distribution with $p-q$ and $n-p$ degrees of freedom. If the observed $F$ is larger than the corresponding $F$ quantile at a given $\alpha$ (typically, $\alpha=0.05$), then we would conclude that the larger model makes a &quot;better job&quot;. (This by no means implies that the model is correct, from a practical point of view!) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A generalization of the above idea is the &lt;a href=&quot;http://en.wikipedia.org/wiki/Likelihood-ratio_test&quot; rel=&quot;nofollow&quot;&gt;likelihood ratio test&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are using R, you can play with the above concepts like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df &amp;lt;- transform(X &amp;lt;- as.data.frame(replicate(2, rnorm(100))), &#xA;                                   y = V1+V2+rnorm(100))&#xA;## simple regression&#xA;anova(lm(y ~ V1, df))         # &quot;ANOVA view&quot;&#xA;summary(lm(y ~ V1, df))       # &quot;Regression view&quot;&#xA;## multiple regression&#xA;summary(lm0 &amp;lt;- lm(y ~ ., df))&#xA;lm1 &amp;lt;- update(lm0, . ~ . -V2) # reduced model&#xA;anova(lm1, lm0)               # test of V2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-03-13T21:38:19.687" LastActivityDate="2011-03-13T21:38:19.687" CommentCount="7" />
  <row Id="8248" PostTypeId="2" ParentId="8236" CreationDate="2011-03-13T21:29:15.393" Score="1" ViewCount="" Body="&lt;p&gt;First, the size of the dataset.  I recommend taking small, tractable samples of the dataset (either by randomly choosing N datapoints, or by randomly choosing several relatively small rectangles in the X-Y plane and taking all points that fall within that plane) and then honing your analysis techniques on this subset.  Once you have an idea of the form of analysis that works, you can apply it to larger portions of the dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PCA is primarily used as a dimensionality reduction technique; your dataset is only three dimensions (one of which is categorical), so I doubt it would apply here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Try working with Matlab or R to visualize the points you are analyzing in the X-Y plane (or their relative density if working with the entire data set), both for individual types and all types the combined, and seeing what patterns emerge visually.  That can help guide a more rigorous analysis.&lt;/p&gt;&#xA;" OwnerUserId="3595" LastActivityDate="2011-03-13T21:29:15.393" CommentCount="2" />
  <row Id="8249" PostTypeId="2" ParentId="8182" CreationDate="2011-03-13T23:02:22.433" Score="1" ViewCount="" Body="&lt;p&gt;The following example (taken from kernlab reference manual), show u how to access the various component of the (kernel) pca&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data(iris)&#xA;test &amp;lt;- sample(1:50,20)&#xA;kpc &amp;lt;- kpca(~.,data=iris[-test,-5],kernel=&quot;rbfdot&quot;,kpar=list(sigma=0.2),features=2)&#xA;&#xA;&#xA;pcv(kpc)        # returns the principal component vectors&#xA;eig(kpc)        # returns the eigenvalues&#xA;rotated(kpc)    # returns the data projected in the (kernel) pca space&#xA;kernelf(kpc)    # returns the kernel used when kpca was performed&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Does this answer your question?&lt;/p&gt;&#xA;" OwnerUserId="3544" LastActivityDate="2011-03-13T23:02:22.433" CommentCount="3" />
  <row Id="8250" PostTypeId="2" ParentId="8243" CreationDate="2011-03-14T03:05:50.457" Score="5" ViewCount="" Body="&lt;p&gt;Just a remark: is there an inequality constraint $\sum_{i=1}^{n} w_{i} \leq 1$ in your system? I merely ask because $\mu$ is usually the KKT multiplier for inequality constraints. If indeed there is an inequality constraint, you will need to satisfy more conditions than just $\nabla L = 0$ to attain optimality (i.e. dual feasibility, complementarity slackness conditions etc.) In which case, you'd be better off using a proper optimization solver. I don't know if $\bar r_{i}, \bar r$ are constants and if the Hessian is positive semidefinite, but if so, this looks like a quadratic program (QP), and you can get solvers for this type of problem (e.g. &lt;code&gt;quadprog&lt;/code&gt; in MATLAB). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But, if you know what you're doing.... here are some ways to get derivatives.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Finite differencing&lt;/strong&gt;. You did not mention if you wanted numerical or exact derivatives. If accuracy isn't too big a deal, a finite difference perturbation is the easiest option. Choose a small enough $h$ for your application and you're all set. &lt;a href=&quot;http://en.wikipedia.org/wiki/Numerical_differentiation#Finite_difference_formulae&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Numerical_differentiation#Finite_difference_formulae&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Complex methods&lt;/strong&gt;. If you want a more accurate derivative, you can also calculate a complex derivative. &lt;a href=&quot;http://en.wikipedia.org/wiki/Numerical_differentiation#Complex_variable_methods&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Numerical_differentiation#Complex_variable_methods&lt;/a&gt;. Essentially, the idea is this:&#xA;$$F&amp;#39;(x) \approx \frac{\mathrm{Im}(F(x + ih))}{h}&#xD;&#xA;$$&#xA;Any programming language that implements a complex number type (e.g. Fortran) can be used to implement this. The derivative you get will be a real number. This is far more accurate than finite differencing, and for a sufficiently small $h$, you'll get a derivative that's pretty close to the exact derivative (to the limit of your machine precision). Note that you can only get 1st order derivatives using this method; it cannot be chained. Some people do interpolation to calculate 2nd order derivatives, but all the nice properties of the complex method are lost in that approach.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Automatic differentiation (AD)&lt;/strong&gt;. This is the fastest and most accurate technique for obtaining numerical derivatives of any order (they can be made accurate to machine precision), but it's also the most complicated from a software point of view. Most optimization modeling languages (like AMPL or GAMS) provide AD facilities to solvers. This is a whole topic unto itself, but in short, if you're using MATLAB, you can use the INTLAB toolbox to quickly and easily calculate the derivatives you need. See this page for options: &lt;a href=&quot;http://www.autodiff.org/?module=Tools&quot; rel=&quot;nofollow&quot;&gt;http://www.autodiff.org/?module=Tools&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;But for a system as small and as simple as yours, I'd just do it by hand. Or use a symbolic tool like Maxima (free) or Sage (also free, front end to Maxima). &lt;/p&gt;&#xA;" OwnerUserId="2833" LastActivityDate="2011-03-14T03:05:50.457" CommentCount="1" />
  <row Id="8251" PostTypeId="1" AcceptedAnswerId="8259" CreationDate="2011-03-14T07:13:51.857" Score="7" ViewCount="175" Body="&lt;p&gt;In a paper I was reading recently I came across the following bit in their data analysis section: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The data table was then split into tissues and cell lines, and the two subtables were separately median polished (the rows and columns were iteratively adjusted to have median 0) before being rejoined into a single table. We finally then selected for the subset of genes whose expression varied by at least 4-fold from the median in this sample set in at least three of the samples tested&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I have to say I don't really follow the reasoning here. I was wondering if you could help me answer the following two questions: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Why is it desirably/helpful to adjust the median in the datasets? Why should it be done separately for different type of samples? &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;How is this not modifying the experimental data? Is this a known way of picking a number of genes/variables from a large set of data, or is it rather adhoc? &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Thanks,&lt;/p&gt;&#xA;" OwnerUserId="3014" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-03-14T07:34:02.277" LastActivityDate="2011-03-14T13:06:06.607" Title="The use of median polish for feature selection" Tags="&lt;feature-selection&gt;&lt;median&gt;&lt;genetics&gt;" AnswerCount="3" CommentCount="4" />
  <row Id="8252" PostTypeId="2" ParentId="8251" CreationDate="2011-03-14T08:02:07.447" Score="4" ViewCount="" Body="&lt;p&gt;You may find some clues in pages 4 and 5 of &lt;a href=&quot;http://www.stats.ox.ac.uk/pub/MASS4/VR4stat.pdf&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is a method of calculating residuals for the model &#xA;$$y_{i,j} = m + a_i + b_j + e_{i,j}$$&#xA;by calculating values for $m$, $a_i$ and $b_j$ so that if the $e_{i,j}$ are tabulated, the median of each row and of each column is 0.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The more conventional approach amounts to calculating values for $m$, $a_i$ and $b_j$ so that the mean (or sum) of each row and each column of residuals is 0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The advantage of using the median is robustness to a small number of outliers; the disadvantage is that you are throwing away potentially useful information if there are no outliers.  &lt;/p&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-03-14T08:02:07.447" CommentCount="7" />
  <row Id="8253" PostTypeId="2" ParentId="8243" CreationDate="2011-03-14T09:10:13.773" Score="4" ViewCount="" Body="&lt;p&gt;Here is the example of implementation with R. R is flexible in regards of combining symbolic differentiation and numerical optimisation. This means that you can get from symbolic expression to function used in numerical optimisation quite fast. The cost is of course speed, since hand-written functions will certainly work faster. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So first create the expression:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;n &amp;lt;- 3&#xA;&#xA;wi &amp;lt;- paste(&quot;w&quot;,1:n,sep=&quot;&quot;)&#xA;&#xA;sigma&amp;lt;-paste(&quot;sigma&quot;,1:n,sep=&quot;&quot;)&#xA;&#xA;lfun &amp;lt;- paste(paste(wi,sigma,sep=&quot;^2*&quot;),collapse=&quot;+&quot;)&#xA;&#xA;rr &amp;lt;- paste(&quot;r&quot;,1:n,sep=&quot;&quot;)&#xA;&#xA;res1 &amp;lt;- paste(paste(wi,rr,sep=&quot;*&quot;),collapse=&quot;+&quot;)&#xA;&#xA;res2 &amp;lt;- paste(wi,collapse=&quot;+&quot;)&#xA;&#xA;optfun &amp;lt;- paste(&quot;1/2*(&quot;,lfun,&quot;)-lambda*(&quot;,res1,&quot;-barr)-mu*(&quot;,res2,&quot;-1)&quot;,collapse=&quot;&quot;)&#xA;&#xA;vars &amp;lt;- c(wi,&quot;lambda&quot;,&quot;mu&quot;)&#xA;&#xA;optexpr &amp;lt;- parse(text=optfun)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Note that I only did string manipulation which is pretty readable. The result is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; optexpr&#xA;expression(1/2*( w1^2*sigma1+w2^2*sigma2+w3^2*sigma3 )-lambda*( w1*r1+w2*r2+w3*r3 -barr)-mu*( w1+w2+w3 -1))&#xA;attr(,&quot;srcfile&quot;)&#xA;&amp;lt;text&amp;gt; &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now use symbolic differentiation to get the equations which we need to solve:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; gradexpr &amp;lt;- lapply(vars,function(l)D(optexpr,name=l))&#xA;&amp;gt; gradexpr&#xA;[[1]]&#xA;1/2 * (2 * w1 * sigma1) - lambda * r1 - mu&#xA;&#xA;[[2]]&#xA;1/2 * (2 * w2 * sigma2) - lambda * r2 - mu&#xA;&#xA;[[3]]&#xA;1/2 * (2 * w3 * sigma3) - lambda * r3 - mu&#xA;&#xA;[[4]]&#xA;-(w1 * r1 + w2 * r2 + w3 * r3 - barr)&#xA;&#xA;[[5]]&#xA;-(w1 + w2 + w3 - 1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now each of the variables is separate variable. For numerical optimisation it makes sense to combine indexed &lt;code&gt;w&lt;/code&gt;, &lt;code&gt;r&lt;/code&gt;, and &lt;code&gt;sigma&lt;/code&gt; into vectors. This involves a little black magic, but it is not hard. We need the following function:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;subvars &amp;lt;- function(expr,tb) {&#xA;    if(length(expr)==1) {&#xA;        nm &amp;lt;- deparse(expr)&#xA;        if(nm %in% tb[,1]) {&#xA;            e &amp;lt;- tb[tb[,1]==nm,2]&#xA;            return(parse(text=e)[[1]])&#xA;        }&#xA;        else&#xA;          return(expr)&#xA;    }&#xA;    else {&#xA;        for (i in 2:length(expr)) {&#xA;            expr[[i]] &amp;lt;- subvars(expr[[i]],tb)&#xA;        }          &#xA;    }       &#xA;    return(expr)&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This function substitutes the expressions according to substitution table. Here is the end result.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; subtable &amp;lt;- rbind(cbind(wi,paste(&quot;w[&quot;,1:n,&quot;]&quot;,sep=&quot;&quot;)),&#xA;                  cbind(sigma,paste(&quot;sigma[&quot;,1:n,&quot;]&quot;,sep=&quot;&quot;)),&#xA;                  cbind(rr,paste(&quot;r[&quot;,1:n,&quot;]&quot;,sep=&quot;&quot;))  &#xA;                  )&#xA;&amp;gt; eqs &amp;lt;- lapply(gradexpr,function(l)subvars(l,subtable))&#xA;&#xA;&amp;gt; eqs&#xA;[[1]]&#xA;1/2 * (2 * w[1] * sigma[1]) - lambda * r[1] - mu&#xA;&#xA;[[2]]&#xA;1/2 * (2 * w[2] * sigma[2]) - lambda * r[2] - mu&#xA;&#xA;[[3]]&#xA;1/2 * (2 * w[3] * sigma[3]) - lambda * r[3] - mu&#xA;&#xA;[[4]]&#xA;-(w[1] * r[1] + w[2] * r[2] + w[3] * r[3] - barr)&#xA;&#xA;[[5]]&#xA;-(w[1] + w[2] + w[3] - 1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now write a function which evaluates the equations.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;eqs.fun &amp;lt;- function(w,sigma,r,barr,lambda,mu) {&#xA;    cl &amp;lt;- match.call()&#xA;    args &amp;lt;- as.list(cl)&#xA;    args &amp;lt;- args[-1]&#xA;    env &amp;lt;- parent.frame()&#xA;    args &amp;lt;- lapply(args,eval,envir=env)  &#xA;    sapply(eqs,function(l)eval(l,env=args))&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is a human readable format, where we supply all the neccessary data to the equations. For optimisation we need a variant of this function where all the variables and the data is separated:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;nl.eqs.fun &amp;lt;- function(p,sigma,r,barr) {&#xA;    eqs.fun(w=p[1:3],sigma=sigma,r=r,barr=barr,lambda=p[4],mu=p[5])&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And now we can solve the equations. For this use function &lt;code&gt;nleqslv&lt;/code&gt; from the package &lt;strong&gt;nleqslv&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;nleqslv(c(0,0,0,0,0),nl.eqs.fun,nl.jac.fun,sigma=c(1,1,1),r=c(1,0.5,0.5),barr=1)&#xA;$x&#xA;[1]  1.000000e+00 -1.254481e-16  2.403703e-16  2.000000e+00 -1.000000e+00&#xA;&#xA;$fvec&#xA;[1] -3.330669e-16 -5.551115e-16 -1.110223e-16  0.000000e+00 -2.220446e-16&#xA;&#xA;$termcd&#xA;[1] 1&#xA;&#xA;$message&#xA;[1] &quot;Function criterion near zero&quot;&#xA;&#xA;$scalex&#xA;[1] 1 1 1 1 1&#xA;&#xA;$nfcnt&#xA;[1] 1&#xA;&#xA;$njcnt&#xA;[1] 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;We can confirm that the solution is correct with &lt;code&gt;solve.QP&lt;/code&gt; from the &lt;code&gt;quadprog&lt;/code&gt; package, since our problem is quadratic optimisation problem.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;solve.QP(Dmat=diag(c(1,1,1)),dvec=rep(0,3),Amat=cbind(c(1,0.5,0.5),c(1,1,1)),bvec=c(1,1),meq=2)&#xA;$solution&#xA;[1]  1.000000e+00  0.000000e+00 -1.110223e-16&#xA;&#xA;$value&#xA;[1] 0.5&#xA;&#xA;$unconstrained.solution&#xA;[1] 0 0 0&#xA;&#xA;$iterations&#xA;[1] 3 0&#xA;&#xA;$Lagrangian&#xA;[1] 2 1&#xA;&#xA;$iact&#xA;[1] 1 2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In this case approach described is an overkill, since there is readily available package to solve the problem. But it is not that hard to extend it to any optimisation function. Also you can calculate the jacobian of the equations very easy. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The main advantage is that this code is much easier to debug, and if you want to change your optimisation function, you only need to change the expressions, conversion to numerical functions is automatic (if the data remains the same).&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-03-14T09:10:13.773" CommentCount="9" />
  <row Id="8254" PostTypeId="1" AcceptedAnswerId="8255" CreationDate="2011-03-14T09:16:32.003" Score="5" ViewCount="163" Body="&lt;p&gt;Lets say I am regressing Y on X1 and X2, where X1 is a numeric variable and X2 is a factor with four levels (A:D). Is there any way to write the linear regression function &lt;code&gt;lm(Y ~ X1 + as.factor(X2))&lt;/code&gt; so that I can choose a particular level of X2 -- say, B -- as the baseline? &lt;/p&gt;&#xA;" OwnerUserId="3671" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-18T16:17:07.760" LastActivityDate="2011-03-18T16:17:07.760" Title="Choose factor level as dummy base in lm() in R" Tags="&lt;r&gt;" AnswerCount="1" />
  <row Id="8255" PostTypeId="2" ParentId="8254" CreationDate="2011-03-14T09:21:08.567" Score="8" ViewCount="" Body="&lt;p&gt;You can use &lt;code&gt;relevel()&lt;/code&gt; to change the baseline level of your factor. For instance, &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; g &amp;lt;- gl(3, 2, labels=letters[1:3])&#xA;&amp;gt; g&#xA;[1] a a b b c c&#xA;Levels: a b c&#xA;&amp;gt; relevel(g, &quot;b&quot;)&#xA;[1] a a b b c c&#xA;Levels: b a c&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-03-14T09:21:08.567" CommentCount="1" />
  <row Id="8257" PostTypeId="2" ParentId="8251" CreationDate="2011-03-14T10:18:31.373" Score="3" ViewCount="" Body="&lt;p&gt;Looks like you are reading a paper that has some gene differential expression analysis. Having done some research involving microarray chips, I can share what little knowledge (hopefully correct) I have about using median polish.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using median polish during the summarization step of microarray preprocessing is somewhat of a standard way to rid data of outliers with perfect match probe only chips (at least for RMA).   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Median polish for microarray data is where you have the chip effect and probe effect as your rows and columns:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;for each probe set (composed of n number of the same probe) on x chips:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;         chip1    chip2    chip3   ...  chipx&#xA;probe1      iv       iv       iv   ...     iv&#xA;probe2      iv       iv       iv   ...     iv &#xA;probe3      iv       iv       iv   ...     iv&#xA;...&#xA;proben      iv       iv       iv   ...     iv&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where iv are intensity values&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because of the variability of the probe intensities, almost all analysis of microarray data is preprocessed using some sort of background correction and normalization before summarization. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;here are some links to the bioC mailing list threads that talk about using median polish vs other methods:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stat.ethz.ch/pipermail/bioconductor/2004-May/004752.html&quot; rel=&quot;nofollow&quot;&gt;&lt;code&gt;https://stat.ethz.ch/pipermail/bioconductor/2004-May/004752.html&lt;/code&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stat.ethz.ch/pipermail/bioconductor/2004-May/004734.html&quot; rel=&quot;nofollow&quot;&gt;&lt;code&gt;https://stat.ethz.ch/pipermail/bioconductor/2004-May/004734.html&lt;/code&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data from tissues and cell lines are usually analysed separately because when cells are cultured their expression profiles change dramatically from collected tissue samples. Without having more of the paper it is difficult to say whether or not processing the samples separately was appropriate. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Normalization, background correction, and summarization steps in the analysis pipeline are all modifications of experimental data, but in it's unprocessed state, the chip effects, batch effects, processing effects would overshadow any signal for analysis. These microarray experiments generate lists of genes that are candidates for follow up experiments (qPCR, etc) to confirm the results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as being ad hoc, ask 5 people what fold difference is required for a gene to be considered differentially expressed and you will come up with at least 3 different answers.&lt;/p&gt;&#xA;" OwnerUserId="3704" LastEditorUserId="3704" LastEditorDisplayName="" LastEditDate="2011-03-14T10:56:54.570" LastActivityDate="2011-03-14T10:56:54.570" CommentCount="2" />
  <row Id="8258" PostTypeId="2" ParentId="4099" CreationDate="2011-03-14T10:57:21.963" Score="1" ViewCount="" Body="&lt;p&gt;I believe Belsely said that CI over 10 is indicative of a possible moderate problem, while over 30 is more severe.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In addition, though, you should look at the variance shared by sets of variables in the high condition indices.  There is debate (or was, last time I read this literature) on whether collinearity that involved one variable and the intercept was problematic or not, and whether centering the offending variable got rid of the problem, or simply moved it elsewhere.  &lt;/p&gt;&#xA;" OwnerUserId="686" LastActivityDate="2011-03-14T10:57:21.963" />
  <row Id="8259" PostTypeId="2" ParentId="8251" CreationDate="2011-03-14T11:43:23.427" Score="5" ViewCount="" Body="&lt;p&gt;Hello Posdef:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tukey Median Polish, algorithm is used in the &lt;a href=&quot;http://bmbolstad.com/misc/ComputeRMAFAQ/ComputeRMAFAQ.html&quot; rel=&quot;nofollow&quot;&gt;RMA&lt;/a&gt; normalization of microarrays. As you may be aware, microarray data is quite noisy, therefore they need a more robust way of estimating the probe intensities taking into account of observations for all the probes and microarrays. This is a typical model used for normalizing intensities of probes across arrays.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Y_{ij} = \mu_{i} + \alpha_{j} + \epsilon_{ij}$$&#xA;$$i=1,\ldots,I \qquad j=1,\ldots, J$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $Y_{ij}$ is the $log$ transformed PM intensity for the $i^{th}$probe on the $j^{th}$ array. $\epsilon_{ij}$ are background noise and they can be assumed to correspond to noise in normal linear regression. However, a distributive assumption on $\epsilon$ may be restrictive, therefore we use Tukey Median Polish to get the estimates for $\hat{\mu_i}$ and $\hat{\alpha_j}$. This is a robust way of normalizing across arrays, as we want to separate signal, the intensity due to probe, from the array effect, $\alpha$. We can obtain the signal by normalizing for the array effect $\hat{\alpha_j}$ for all the arrays. Thus, we are only left with the probe effects plus some random noise. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The link that I have quoted before uses Tukey median polish to estimate the differentially expressed genes or &quot;interesting&quot; genes by ranking by the probe effect. However, the paper is pretty old, and probably at that time people were still trying to figure out how to analyze microarray data. Efron's non-parametric empirical Bayesian methods paper came in 2001, but probably may not have been widely used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, now we understand a lot about microarrays (statistically) and are pretty sure about their statistical analysis. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Microarray data is pretty noisy and RMA (which uses Median Polish) is one of the most popular normalization methods, may be because of its simplicity. Other popular and sophisticated methods are: GCRMA, VSN. It is important to normalize as the interest is probe effect and &lt;em&gt;not&lt;/em&gt; array effect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you expect, the analysis could have benefited by some methods which take advantage of information borrowing across genes. These may include, Bayesian or empirical Bayesian methods. May be the paper that you are reading is old and these techniques weren't out until then. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regarding your second point, yes they are probably modifying the experimental data. But, I think, this modification is for a better cause, hence justifiable. The reason being &lt;/p&gt;&#xA;&#xA;&lt;p&gt;a) Microarray data are pretty noisy. When the interest is probe effect, normalizing data by RMA, GCRMA, VSN, etc. is necessary and may be taking advantage of any special structure in the data is good. But I would avoid doing the second part. This is mainly because if we don't know the structure in advance, it is better not impose a lot of assumptions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;b) Most of the microarray experiments are exploratory in their nature, that is, the researchers are trying to narrow down to a few set of &quot;interesting&quot; genes for further analysis or experiments. If these genes have a strong signal, modifications like normalizations should not (substantially) effect the final results. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, the modifications may be justified. But I must remark, overdoing the normalizations may lead to wrong results. &lt;/p&gt;&#xA;" OwnerUserId="1307" LastEditorUserId="1307" LastEditorDisplayName="" LastEditDate="2011-03-14T13:06:06.607" LastActivityDate="2011-03-14T13:06:06.607" CommentCount="4" />
  <row Id="8260" PostTypeId="1" CreationDate="2011-03-14T12:12:27.620" Score="2" ViewCount="74" Body="&lt;p&gt;I've known that, in orthogonal rotation, if the rotation matrix has determinant of -1 then reflection is present. Otherwise the determinant is +1 and we have pure rotation. May I extend this &quot;sign-of-determinant&quot; rule for non-orthogonal rotations? Such as orthogonal-into-oblique axes or oblique-into-orthogonal axes rotations? For example, this matrix&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; .9427   .2544   .1665   .1377&#xA;-.0451  -.0902  -.9940  -.0421&#xA; .3325   .3900   .1600   .8437&#xA; .4052   .8702   .2269   .1644&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;is an oblique-to-orthogonal rotation (I think, because sums of squares in rows, not in columns, are 1). Its determinant is -0.524. May I state that the rotation contains a reflection? Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="3277" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-14T15:38:45.710" LastActivityDate="2011-03-14T15:38:45.710" Title="Detecting reflection in non-orthogonal rotation" Tags="&lt;rotation&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="8261" PostTypeId="1" CreationDate="2011-03-14T12:19:15.427" Score="3" ViewCount="96" Body="&lt;p&gt;If $X$ is a $m &amp;#215; n$ matrix, where $m$ is the number of measurement types (variables) and $n$ is the number of samples, would it be correct to perform a PCA on a matrix that has $m \geq n$ ? If not, please provide some arguments why would this be a problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I remember having heard that doing such an analysis would be invalid, but the &lt;a href=&quot;http://en.wikipedia.org/wiki/Principal_component_analysis&quot; rel=&quot;nofollow&quot;&gt;Wikipedia page for PCA&lt;/a&gt; doesn't mention a low $n/m$ ratio as being a potential limitation for using the method.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please note that I am a biologist and aim at a more practical answer (if possible).&lt;/p&gt;&#xA;" OwnerUserId="3467" LastActivityDate="2011-03-14T12:52:57.630" Title="Is the matrix dimension important for performing a valid PCA?" Tags="&lt;pca&gt;" AnswerCount="2" />
  <row Id="8263" PostTypeId="2" ParentId="8261" CreationDate="2011-03-14T12:37:29.147" Score="2" ViewCount="" Body="&lt;p&gt;I do not think that you would get any useful information from such an analysis, as lore in my subject area (psychology) suggests a 10;1 ratio in favour of n as a precondition. In some circumstances (where communalities are high) you can get away with 5 or 3 to 1, but a ratio of less than 1 is probably a recipe for disaster.&lt;/p&gt;&#xA;" OwnerUserId="656" LastActivityDate="2011-03-14T12:37:29.147" CommentCount="3" />
  <row Id="8264" PostTypeId="2" ParentId="8261" CreationDate="2011-03-14T12:52:57.630" Score="2" ViewCount="" Body="&lt;p&gt;PCA of variables. Number of observations n is low relative to number of variables.&#xA;1) Mathematical aspect. Whenever n&amp;lt;=m correlation matrix is singular which means some of last m principle components are zero-variance, that is, they are not existant. This is not a problem to PCA, generally speaking, since you could just ignore those. However, many software (mostly those uniting PCA and Factor Analysis in one command or procedure) will not allow you to have singular correlation matrix.&#xA;2) Statistical aspect. To have your results reliable you must have correlations reliable; that requires considerable sample size which always should be larger than number of variables. They say, if you have m=20 you ought to have n=100 or so. But if you have m=100 you should have n=300 or so. As m grows, minimal recommended n/m proportion diminishes.&lt;/p&gt;&#xA;" OwnerUserId="3277" LastActivityDate="2011-03-14T12:52:57.630" CommentCount="2" />
  <row Id="8265" PostTypeId="2" ParentId="7103" CreationDate="2011-03-14T12:55:11.880" Score="1" ViewCount="" Body="&lt;p&gt;You could use any probabilistic time series model in combination with &lt;a href=&quot;http://en.wikipedia.org/wiki/Arithmetic_coding&quot; rel=&quot;nofollow&quot;&gt;arithmetic coding&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You'd have to quantize the data, though. Idea: the more likely an &quot;event&quot; is to occur, the more bits for that event are reserved. E.g if $p(x_t = 1| x_{1:t-1}) = 0.5$ with $x_{1:t-1}$ being the history of events seen so far, then coding that event will cost you 1 bit, while all others have to use more bits.&lt;/p&gt;&#xA;" OwnerUserId="2860" LastActivityDate="2011-03-14T12:55:11.880" CommentCount="2" />
  <row Id="8266" PostTypeId="1" AcceptedAnswerId="8284" CreationDate="2011-03-14T13:43:20.683" Score="3" ViewCount="133" Body="&lt;p&gt;I have a lot of data on previous race history and I'm trying to predict a percentage chance of winning the next race using Regression, kNN, and SVM learning algorithms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say a race has 5 runners, and each runner has a previous best course time of, say $T_i$ (seconds).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've also introduced an additional input for RANK of previous best course time of the 5 runners with value 0 to $1 - \frac{T_i-T_{min}}{T_{max}-T_{min}}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is: does introducing both the absolute best course time and rank best course time cause any problems?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand that these inputs are likely to correlate but if someone runs a world record time they are more likely to win easily but this will get lost using the rank input only which would assign them a rank of 1.&lt;/p&gt;&#xA;" OwnerUserId="3484" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-14T15:15:59.677" LastActivityDate="2011-03-14T20:02:55.153" Title="Advice on classifier input correlation" Tags="&lt;machine-learning&gt;&lt;correlation&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="8267" PostTypeId="1" AcceptedAnswerId="8269" CreationDate="2011-03-14T14:30:10.763" Score="0" ViewCount="285" Body="&lt;p&gt;In R, I am trying to write a function to subset and exclude observations in a data frame based on three variables. My data looks something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data.frame':   43 obs. of  8 variables:&#xA; $ V1: chr  &amp;quot;ENSG00000008438&amp;quot; &amp;quot;ENSG00000048462&amp;quot; &amp;quot;ENSG00000006075&amp;quot; &amp;quot;ENSG00000049130&amp;quot; ...&#xD;&#xA;     $ V2: chr  &quot;ENST00000008938&quot; &quot;ENST00000053243&quot; &quot;ENST00000225245&quot; &quot;ENST00000228280&quot; ...&#xA; $ V3: chr  &amp;quot;ENSP00000008938&amp;quot; &amp;quot;ENSP00000053243&amp;quot; &amp;quot;ENSP00000225245&amp;quot; &amp;quot;ENSP00000228280&amp;quot; ...&#xD;&#xA;     $ V4: chr  &quot;PGLYRP1&quot; &quot;TNFRSF17&quot; &quot;CCL3&quot; &quot;KITLG&quot; ...&#xA; $ V5: chr  &amp;quot;19&amp;quot; &amp;quot;16&amp;quot; &amp;quot;17&amp;quot; &amp;quot;12&amp;quot; ...&#xD;&#xA;     $ V6: chr  &quot;q13.32&quot; &quot;p13.13&quot; &quot;q12&quot; &quot;q21.32&quot; ...&#xA; $ V7: int  46522415 12058964 34415602 88886566 8276226 150285143 29138654 76424442 136871919 6664568 ...&#xD;&#xA;     $ V8: int  46526304 12061925 34417515 88974238 8291203 150294844 29190208 76448092 136875735 6670599 ...&#xA;&amp;gt;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What I am trying to do is this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;input = function(x) {&#xA;        pruned1 &amp;lt;- subset(x[-which(x$V5 == 1 &amp;amp; x$V7 &amp;gt; 113818477 &amp;amp; x$V8 &amp;lt; 114658477), ])&#xA;        pruned2 &amp;lt;- subset(pruned1[-which(pruned1$V5 == 1 &amp;amp; pruned1$V7 &amp;gt; 192461456 &amp;amp; pruned1$V8 &amp;lt; 192549912), ])&#xA;        # and so on&#xA;        }&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So I want to take my original data frame, exlude observations that fulfil all three of the subsetting conditions, make a new object of the remaining observations (pruned1) and exclude any observations that fulfil a new set of condidtions and so on.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem I am having seems to be the third condition in the first subsetting, as this returns:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;'data.frame':   1 obs. of  8 variables:&#xA; $ V1: chr NA&#xD;&#xA;     $ V2: chr NA&#xA; $ V3: chr NA&#xD;&#xA;     $ V4: chr NA&#xA; $ V5: chr NA&#xD;&#xA;     $ V6: chr NA&#xA; $ V7: int NA&#xD;&#xA;     $ V8: int NA&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;While only using the first two conditions returns:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;'data.frame':   38 obs. of  8 variables:&#xA; $ V1: chr  &amp;quot;ENSG00000008438&amp;quot; &amp;quot;ENSG00000048462&amp;quot; &amp;quot;ENSG00000006075&amp;quot; &amp;quot;ENSG00000049130&amp;quot; ...&#xD;&#xA;     $ V2: chr  &quot;ENST00000008938&quot; &quot;ENST00000053243&quot; &quot;ENST00000225245&quot; &quot;ENST00000228280&quot; ...&#xA; $ V3: chr  &amp;quot;ENSP00000008938&amp;quot; &amp;quot;ENSP00000053243&amp;quot; &amp;quot;ENSP00000225245&amp;quot; &amp;quot;ENSP00000228280&amp;quot; ...&#xD;&#xA;     $ V4: chr  &quot;PGLYRP1&quot; &quot;TNFRSF17&quot; &quot;CCL3&quot; &quot;KITLG&quot; ...&#xA; $ V5: chr  &amp;quot;19&amp;quot; &amp;quot;16&amp;quot; &amp;quot;17&amp;quot; &amp;quot;12&amp;quot; ...&#xD;&#xA;     $ V6: chr  &quot;q13.32&quot; &quot;p13.13&quot; &quot;q12&quot; &quot;q21.32&quot; ...&#xA; $ V7: int  46522415 12058964 34415602 88886566 8276226 150285143 29138654 76424442 136871919 6664568 ...&#xD;&#xA;     $ V8: int  46526304 12061925 34417515 88974238 8291203 150294844 29190208 76448092 136875735 6670599 ...&#xA;&amp;gt;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Doing this manually is not an options, since I have a large number of input files that I would like so prune in the same manner. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance for any help! &lt;/p&gt;&#xA;" OwnerUserId="3705" LastEditorUserId="3705" LastEditorDisplayName="" LastEditDate="2011-03-14T14:41:49.677" LastActivityDate="2011-03-15T07:55:15.623" Title="Help on subsetting data frames using multiple logical operators in R" Tags="&lt;r&gt;" AnswerCount="3" CommentCount="0" />
  <row Id="8268" PostTypeId="2" ParentId="8266" CreationDate="2011-03-14T14:42:57.020" Score="3" ViewCount="" Body="&lt;p&gt;I think it depends on if the purpose of your model is descriptive (e.g. considering variable importance or hypothesis tests in the regression) or purley predictive. If it is the former, then certainly input features that are strongly correlated will create difficulties in making inference about how variables impact the output and relate to each other. For example, can you say variable 1 is the most important if it shares most of its variance with variable 2? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even in regression, multicollinearity will not impact the coefficients, only the standard error (the estimates will still be those that minimize the squared error) so the predictions are ok. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tend to consider colinearity between inputs not that big an issue when building a predictive model. The best and only way to know for certain is to build a model with both variables and then with only the one with the strongest relationship to the target variable and see which produces the best predictions on new data.....&lt;/p&gt;&#xA;" OwnerUserId="2040" LastActivityDate="2011-03-14T14:42:57.020" CommentCount="3" />
  <row Id="8269" PostTypeId="2" ParentId="8267" CreationDate="2011-03-14T15:07:22.570" Score="1" ViewCount="" Body="&lt;p&gt;You are using subset wrongly. The syntax of subset is &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;&lt;pre&gt;&lt;code&gt;    pruned1 &amp;lt;- subset(x, !( V5 == 1 &amp;amp; V7 &amp;gt; 113818477 &amp;amp; V8 &amp;lt; 114658477) )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;(&quot;!&quot; is the negation operator)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and not &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;&lt;pre&gt;&lt;code&gt;    pruned1 &amp;lt;- subset(x[-which(x$V5 == 1 &amp;amp; x$V7 &amp;gt; 113818477 &amp;amp; x$V8 &amp;lt; 114658477)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Hope that helps&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS You might want to look into whether using '&amp;amp;' or '&amp;amp;&amp;amp;' fits your situation best, I can't remember if that will cause a problem.&lt;/p&gt;&#xA;" OwnerUserId="2807" LastEditorUserId="2807" LastEditorDisplayName="" LastEditDate="2011-03-14T17:11:35.753" LastActivityDate="2011-03-14T17:11:35.753" CommentCount="3" />
  <row Id="8270" PostTypeId="2" ParentId="8260" CreationDate="2011-03-14T15:16:08.273" Score="1" ViewCount="" Body="&lt;p&gt;When the determinant is negative, composing with &lt;em&gt;any&lt;/em&gt; reflection will give a positive determinant.  In that sense you are correct.  However, in another sense the question does not appear to be meaningful, because the matrix you give, although it is &lt;em&gt;row normalized,&lt;/em&gt; is not orthogonal (it is not a &quot;rotation,&quot; nor--unlike rotation matrices--can it be written as a finite product of reflections).  Whether or not it &quot;contains&quot; a reflection depends on what group you consider the matrix to be part of and what subgroup you want to relate it to, neither of which has been specified.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-03-14T15:16:08.273" CommentCount="4" />
  <row Id="8271" PostTypeId="1" CreationDate="2011-03-14T15:36:51.410" Score="1" ViewCount="375" Body="&lt;p&gt;I wanted to carry out a two-way ANCOVA for my data. However, SPSS isn't liking that I have only one IV. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;One IV: Group (3 levels, i.e. 2 experimental groups and a control group),&lt;/li&gt;&#xA;&lt;li&gt;One DV: outcome measure (2 levels, i.e. time 1 and time 2),&lt;/li&gt;&#xA;&lt;li&gt;One covariate (I have checked and this demographic continuous variable moderately correlates with the DV).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Please could someone help as to whether I am missing something in terms of putting this into SPSS?&lt;/p&gt;&#xA;" OwnerUserId="3706" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-15T04:13:26.403" LastActivityDate="2011-04-15T04:13:26.403" Title="How to set up a two-way mixed ANOVA with a covariate  in SPSS?" Tags="&lt;spss&gt;&lt;repeated-measures&gt;&lt;ancova&gt;" AnswerCount="2" CommentCount="3" FavoriteCount="1" />
  <row Id="8272" PostTypeId="2" ParentId="8267" CreationDate="2011-03-14T16:00:19.347" Score="0" ViewCount="" Body="&lt;p&gt;Another way to do this is&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;y &amp;lt;- &#xA;    x[&#xA;        !(x$V5 == 1 &amp;amp; x$V7 &amp;gt; 113818477 &amp;amp; x$V8 &amp;lt; 114658477) &amp;amp;&#xA;        !(x$V5 == 1 &amp;amp; x$V7 &amp;gt; 192461456 &amp;amp; x$V8 &amp;lt; 192549912),&#xA;    ]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Also, V5 is a character variable, so &lt;code&gt;x$V5 == “1”&lt;/code&gt; might be preferable -- e.g., if you have any codes that are actually formatted as &quot; 1&quot;, you will need to take care to specify &lt;code&gt;x$V5 == “ 1”&lt;/code&gt;. @ulvund &lt;code&gt;&amp;amp;&lt;/code&gt; is necessary for this application because it returns element-wise logical AND, whereas &lt;code&gt;&amp;amp;&amp;amp;&lt;/code&gt; “evaluates left to right examining only the first element of each vector. Evaluation proceeds only until the result is determined” (?Logic).&lt;/p&gt;&#xA;" OwnerUserId="3432" LastActivityDate="2011-03-14T16:00:19.347" CommentCount="2" />
  <row Id="8273" PostTypeId="1" CreationDate="2011-03-14T16:19:27.127" Score="5" ViewCount="189" Body="&lt;p&gt;I got asked something similar to this in interview today.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The interviewer wanted to know what is the probability that an at-the-money option will end up in-the-money when volatility tends to infinity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I said 0% because the normal distributions that underly the Black-Scholes model and the random walk hypothesis will have infinite variance. And so I figured the probability of all values will be zero.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My interviewer said the right answer is 50% because the normal distribution will still be symmetric and almost uniform. So when you integrate from mean to +infinity you get 50%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am still not convinced with his reasoning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Who is right?&lt;/p&gt;&#xA;" OwnerUserId="2283" LastActivityDate="2011-03-14T18:34:18.040" Title="What is the probability that a normal distribution with infinite variance has a value greater than its mean?" Tags="&lt;normal-distribution&gt;&lt;variance&gt;" AnswerCount="2" FavoriteCount="2" />
  <row Id="8274" PostTypeId="2" ParentId="8273" CreationDate="2011-03-14T17:02:26.200" Score="8" ViewCount="" Body="&lt;p&gt;Neither form of reasoning is mathematically rigorous--there's no such thing as a normal distribution with infinite variance, nor is there a limiting distribution as the variance grows large--so let's be a little careful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the Black-Scholes model, the log price of the underlying asset is assumed to be undergoing a random walk.  The problem is equivalent to asking &quot;what is the chance that the asset's (log) value at the expiration date will exceed its current (log) value?&quot;  Letting the volatility increase without limit is equivalent to letting the expiration date increase without limit.  Thus, the answer should be the same as asking &quot;what is the limit, as $t \to \infty$, that the value of a random walk at time $t$ is greater than its value at time $0$?&quot;  By symmetry (exchanging upticks and downticks), (and noting that in the continuous model the chance of being at the money is $0$) those probabilities equal $1/2$ for any $t \gt 0$, whence their limit indeed exists and equals $1/2$.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-14T18:34:18.040" LastActivityDate="2011-03-14T18:34:18.040" CommentCount="1" />
  <row Id="8275" PostTypeId="2" ParentId="8236" CreationDate="2011-03-14T17:17:15.127" Score="2" ViewCount="" Body="&lt;p&gt;The type of data you describe is ususally called &quot;marked point patterns&quot;, R has a task view for spatial statistics that offers many good packages for this type of analysis, most of which are probably not able to deal with the kind of humongous data you have :( &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;For example, maybe events of type A usually don't occur where events of type B do. Or maybe in some area, there are mostly events of type C.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;These are two fairly different type of questions:&#xA;The second asks about the positioning of one type of mark/event. Buzzwords to look for in this context are f.e. intensity estimation or K-function estimation if you are interested in discovering patterns of clustering (events  of a kind tend to group together) or repulsion (events of a kind tend to be separated).&#xA;The first asks about the correlation between different types of events. This is usually measured with mark correlation functions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think subsampling the data to get a more tractable data size is dangerous (see comment to @hamner's reply), but maybe you could aggregate your data: Divide the observation window into a managable number of cells of equal size and tabulate the event counts in each. Each cell is then described by the location of its centre and a 10-vector of counts for your 10 mark types. You should be able to use the standard methods for marked point processes on this aggregated process.  &lt;/p&gt;&#xA;" OwnerUserId="1979" LastActivityDate="2011-03-14T17:17:15.127" CommentCount="2" />
  <row Id="8276" PostTypeId="1" AcceptedAnswerId="8326" CreationDate="2011-03-14T17:26:32.603" Score="3" ViewCount="65" Body="&lt;p&gt;The two main packages I use at work are Palisade Risk and SPSS Clementine - they are both quite old versions and I've been supplementing the ability to analyze properly at work with more modern abilities available at home. E .g.at home I've been testing RapidMiner 5, which I really like in principle but have experienced a few issues with large data sets whereby the computational time progresses for hours then crashes for yet unknown reasons.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I like about Palisade &amp;amp; Clementine is they seem quite robust in ability to handle large amounts of input data and more importantly the ability to easily deploy models, i.e. once we have a model(s) that works we just feed it raw data and the outputs are predicted. However, Palisade &amp;amp; SPSS are both very limited in modern stats methods and machine learning algorithms that are hugely behind methods available in R/RapidMiner (Eg cross validation to name one). I would really like to convince my company to invest in R/Rapidminer but the ease of deployment of models is really the crux of the problem. May I ask how easy or difficult it is to achieve model deployment in the following and ask any advice from fellow professionals?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;i) RapidMiner&#xA;ii) R&#xA;iii) other recommendations?&lt;/p&gt;&#xA;" OwnerUserId="3484" LastActivityDate="2011-03-15T17:14:46.560" Title="Help with deploying a model" Tags="&lt;machine-learning&gt;&lt;software&gt;" AnswerCount="1" />
  <row Id="8277" PostTypeId="2" ParentId="8130" CreationDate="2011-03-14T17:32:09.420" Score="1" ViewCount="" Body="&lt;p&gt;The test you probably want is &lt;a href=&quot;http://en.wikipedia.org/wiki/Fisher%27s_exact_test&quot; rel=&quot;nofollow&quot;&gt;Fisher's exact test&lt;/a&gt;. Unfortunately, given the likely very low click-through rate and small expected effect size, you will need enormous N to achieve the confidence interval you want. Lets say that the 'true' click-through rate of your best ad is .11, and your second best .1. Further, let's say you want the probability that you improperly fail to reject the null hypothesis (that there is no difference between the two ads), to be less than .20. If this is so, you will need an N on the order of 10,000.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; library(statmod)   &#xA;&amp;gt; power.fisher.test(.1,.11,20000,20000,.05)&#xA;[1] 0.84&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As a commenter suggested, you likely should not care about a ten percent difference in ad performance. For grosser differences, the necessary size of the samples decreases quickly.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; power.fisher.test(.1,.2,200,200,.05)&#xA;[1] 0.785&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="82" LastEditorUserId="82" LastEditorDisplayName="" LastEditDate="2011-03-15T02:41:01.220" LastActivityDate="2011-03-15T02:41:01.220" />
  <row Id="8278" PostTypeId="1" AcceptedAnswerId="8325" CreationDate="2011-03-14T17:53:57.350" Score="2" ViewCount="93" Body="&lt;p&gt;I am planning a study to collect data of patients dying after brain infection versus number of cases for the next 5 years in 2 cities where they manage the disease with different drugs.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Will a &lt;strong&gt;&lt;em&gt;year wise t-test be better or year wise Relative Risk of death and meta analysis&lt;/em&gt;&lt;/strong&gt; be better to compare the results of the study?&lt;/li&gt;&#xA;&lt;li&gt;I am a beginner in statistics. A word of explanation would benefit me.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The data would look like&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Year     City 1 Deaths    City 1 cases  City 2 Deaths   City 2 cases&#xA;2011     237                1124             10            1226&#xA;2012     228                1030             26           1181&#xA;2013     1500              6061             10           1122&#xA;2014     528                2320             32           1173&#xA;2015     645                3024             11           1232&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2956" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-03-15T07:13:19.640" LastActivityDate="2011-03-15T17:10:20.340" Title="How to assess drug effects using annual death rate data?" Tags="&lt;meta-analysis&gt;&lt;experiment-design&gt;&lt;t-test&gt;&lt;relative-risk&gt;" AnswerCount="1" CommentCount="12" />
  <row Id="8279" PostTypeId="2" ParentId="8187" CreationDate="2011-03-14T18:12:36.827" Score="1" ViewCount="" Body="&lt;p&gt;I'm not sure why time series is not being accepted as a solution to your problem since this is equally spaced chronological data .  Confounding variables although unknown in nature can be proxied by both ARIMA  structure and/or Detectable Interventions.  I'm not sure who is not accepting that time series analysis is appropriate so we disagree with that advice. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Time series methods are not just pure autoregressive in form but easily extend to Polynomial Distributed Lags or ADL in user-specified supporting series such as the number of reported cases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my opinion this is an example of a pooled cross-sectional time series problem. Gregory Chow developed a test for the constancy of parameters across groups in 1960 ;   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Chow_test&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Chow_test&lt;/a&gt;&lt;br&gt;&#xA;&lt;a href=&quot;http://en.wikipedia.org/wiki/Test_for_structural_change&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Test_for_structural_change&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this case, the test needs to be front-ended with Outlier Detection to ensure a Gaussian set of errors  i.e. no proven anomalous data or in other words the error process can't be proven to be Non-Gaussian within each state.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;X1 is the number of cases and Y is the number of deaths.  X2 is the empirically identified point of anomaly ;(2009 .. period 6  for State1 and  2004 .. period 1 for State2  . Outlier Detection  led to identifying  one anomalous data point for each state reflecting some unknown background variable thus yielding a more robust estimate of the mortality rates.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Analysis of State1&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;State1    Y(T) = -.65649&lt;br&gt;&#xA;                     +[X1(T)][.0046)]      CASES&#xA;                     +[X2(T)][-1.3608)]    PULSE6 I~P00006STATE1&#xA;                     +       [A(T)]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suggesting an unusually low mortality rate for 2009&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Analysis of State2&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;State2   Y(T) =  123.55&lt;br&gt;&#xA;                     +[X1(T)][(+  .0468)]  CASES&#xA;                     +[X2(T)][(+ 35.7590)] PULSE1 I~P00001STATE2&#xA;                     +       [A(T)]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suggesting an unusually high mortality rate for 2004&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This leads to estimating two cleansed data points&lt;/p&gt;&#xA;&#xA;&lt;p&gt;STATE  YEAR            Y OBSERVED   Y ADJUSTED&lt;/p&gt;&#xA;&#xA;&lt;p&gt;STATE1 2009             3             4.36  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;STATE2 2004             254         218.24&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Replacing these two observed possibly errant values possibly due to some unspecified concomitant factor (“lurking Variable”) one computes a rate of.0046 for STATE1 and .0468 for STATE2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Chow Test for constancy of parameters across groups easily yields a rejection of the null hypothesis of equal coefficients, thus the states can be said to have significantly different mortality rates. Note that even though the “experiment was not a controlled one” it is possible to identify deterministic effects reflecting some uncontrolled input which untreated can distort the analysis.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S. I am fully aware that we can’t have 4.36 or 218.24 mortalities in real life !&lt;/p&gt;&#xA;" OwnerUserId="3382" LastActivityDate="2011-03-14T18:12:36.827" CommentCount="7" />
  <row Id="8280" PostTypeId="2" ParentId="8273" CreationDate="2011-03-14T18:17:05.443" Score="0" ViewCount="" Body="&lt;p&gt;You should be doing your analysis based on a log normal distribution, not a normal one. You interviewer is wrong when he states that the distribution is symmetrical. It would never be, regardless of the variance.  You also need to distinguish between volatility and what you are calling infinite variance. A stocks price, for example,  has no upper limit, thus it has &quot;infinite variance&quot;.&lt;/p&gt;&#xA;" OwnerUserId="3489" LastActivityDate="2011-03-14T18:17:05.443" CommentCount="3" />
  <row Id="8281" PostTypeId="1" CreationDate="2011-03-14T18:19:51.103" Score="2" ViewCount="68" Body="&lt;p&gt;The two-sample Kolmogorov-Smirnov statistic is normally defined as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$D = \max_x |A(x) - B(x)|.$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to compute a variant that retains the sign of the difference between the distribution functions:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$D^\prime = A(x) - B(x) \quad\text{where}\quad x = \arg\max_x |A(x) - B(x)|.$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the right name of $D^\prime$?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that $D^\prime$ is different than&#xA;$D^+ = \max_x [A(x) - B(x)]$&#xA;and&#xA;$D^- = \max_x [B(x) - A(x)],$&#xA;which &lt;a href=&quot;http://dx.doi.org/10.1007/BF02479749&quot; rel=&quot;nofollow&quot;&gt;Coberly and Lewis&lt;/a&gt; describe as “one-sided” KS-statistics. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For context, the $D^\prime$ as defined above is used by &lt;a href=&quot;http://dx.doi.org/10.1126/science.1100709&quot; rel=&quot;nofollow&quot;&gt;Perlman et al.&lt;/a&gt; to profile drug-treated cultured cells. (See page 3–4 of the supplement.) They compute $D^\prime$ independently for each image feature of the sample, then standardize the $D^\prime$-values (using the mean and standard deviation of the same measurement on mock-treated samples). Correlated profiles (of standardized $D^\prime$-values) indicate drugs that have similar biological effects.&lt;/p&gt;&#xA;" OwnerUserId="220" LastEditorUserId="220" LastEditorDisplayName="" LastEditDate="2011-03-14T18:45:55.043" LastActivityDate="2011-05-13T19:46:00.740" Title="What is the right name for the variant of the Kolmogorov-Smirnov statistic that retains the sign of the difference?" Tags="&lt;kolmogorov-smirnov&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="1" />
  <row Id="8282" PostTypeId="2" ParentId="8281" CreationDate="2011-03-14T18:33:40.373" Score="0" ViewCount="" Body="&lt;p&gt;It looks like a variant of &lt;a href=&quot;http://en.wikipedia.org/wiki/Kuiper%27s_test&quot; rel=&quot;nofollow&quot;&gt;Kuiper's test&lt;/a&gt; to me, although Kuiper's &lt;em&gt;V&lt;/em&gt; = &lt;em&gt;D&lt;sup&gt;+&lt;/sup&gt;&lt;/em&gt; + &lt;em&gt;D&lt;sup&gt;−&lt;/sup&gt;&lt;/em&gt; ≠ &lt;em&gt;D'&lt;/em&gt;.&lt;/p&gt;&#xA;" OwnerUserId="3582" LastActivityDate="2011-03-14T18:33:40.373" />
  <row Id="8283" PostTypeId="1" CreationDate="2011-03-14T17:42:26.557" Score="2" ViewCount="126" Body="&lt;p&gt;Hello, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know that Mclust does the fit on its own but I am trying to implement an optimization with the aim to generate a mixture of 2 gaussians with the combine moments as closed as possible to the moment of my returns' distribution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The objective is to &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;Min Abs((Mean Ret - MeanFit)/Mean Fit) + Abs((Std Ret -Stdev Fit)/Stdev) + Abs((Sk Ret-Sk fit)/Sk Fit) + Abs((Kurt Ret- Kurt Fit))&lt;/code&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Taking into account that I fix the weight between the two gaussians at (0.2;0.8) I implement the below code in R: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;distance &amp;lt;-function(parameter,x) { &#xA; u=mean(x) &#xA; s=sd(x) &#xA; sk=skewness(x) &#xA; kurt=kurtosis(x) &#xA; d1=dnorm(x,parameter[1],parameter[2]) &#xA; d2=dnorm(x,parameter[3],parameter[4]) &#xA; dfit=0.2d1+0.8d2 &#xA; ufit=mean(dfit) &#xA; sdfit=sd(dfit) &#xA; skfit=skewness(fit) &#xA; kurtfit=kurtosis(fit) &#xA;&#xA;&#xA; abs((u-ufit)/ufit)+abs(s-sdfit)/sdfit)+abs((sk-skfit)/skfit)+abs((kurt-kurtfit)/kurtfit)) &#xA;} &#xA;Parameter&amp;lt;-c(0,0.01,0,0.01)  # starting point of the optimization &#xA;opp&amp;lt;-optim(parameter,distance,x=conv) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;could anybody tell me whether it is the right approach ? &lt;/li&gt;&#xA;&lt;li&gt;should I add some constraint like &#xA;&lt;code&gt;ufit=0.2*mean(d1)+0.8*mean(d2)...&lt;/code&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;thank you very much in advance for your time and help. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sam&lt;/p&gt;&#xA;" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-27T16:03:13.370" LastActivityDate="2011-03-27T16:03:13.370" Title="Fitting 4-moment distribution with mixture gaussian" Tags="&lt;r&gt;&lt;distributions&gt;&lt;optimization&gt;&lt;mixture&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="1" />
  <row Id="8284" PostTypeId="2" ParentId="8266" CreationDate="2011-03-14T20:02:55.153" Score="2" ViewCount="" Body="&lt;p&gt;It depends on the classifier. Some classifiers (such as Naive Bayes) explicitly assume feature independence, so they might behave in unexpected ways. Other classifiers (such as SVM) care about it much less. In image analysis it is routine to throw thousands of highly correlated features at SVM, and SVM seems to perform decently. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For kNN, adding more features will artificially inflate their importance. Suppose you have two features, best course time and coach experience. They will influence the distance equally. Now you add another feature 'course time multiplied by two'. This is essentially a replica of the first feature, but kNN doesn't know about it. So this feature now will influence the distance computation more significantly. Whether you want this or not will depend on the task. You probably don't want features to influence the distance more just because you thought of more &quot;synonyms&quot; for them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A compromise might be to perform feature selection first and then use kNN. This way two &quot;synonyms&quot; of the same feature will be retained only if both are important.&lt;/p&gt;&#xA;" OwnerUserId="3369" LastActivityDate="2011-03-14T20:02:55.153" CommentCount="2" />
  <row Id="8285" PostTypeId="1" CreationDate="2011-03-14T20:48:28.090" Score="2" ViewCount="360" Body="&lt;p&gt;I am a novice when it comes to stats, so I apologize beforehand for the simplicity of my question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I trying to figure out the best way to normalize (this may be the wrong term) my data in so that the maximum value is 1 and the minimum value is 0. What are my options and what methods would you suggest as the best?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My data will be non-negative and has no bound (though it may be bound in some situations, how would this make it different?).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am measuring content across many different dimensions and I want to be able to make comparisons in terms of how relevant a given piece of content is. Additionally, I want to display values across these dimensions that is explicable and easily understood.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you in advance for helping a newbie :)&lt;/p&gt;&#xA;" OwnerUserId="1514" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-15T12:05:06.223" LastActivityDate="2011-03-16T00:27:24.787" Title="Normalizing data between 0 and 1" Tags="&lt;normal-distribution&gt;&lt;normalization&gt;" AnswerCount="0" CommentCount="0" FavoriteCount="0" />
  <row Id="8286" PostTypeId="1" CreationDate="2011-03-14T21:04:46.997" Score="4" ViewCount="66" Body="&lt;p&gt;Hi,&#xA;I have this dataset with size (rows) and shape (columns) &#xA;and in each cell  #disease/#in cell .&#xA;How would I answer the following questions?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is size an indicator of disease?&lt;br&gt;&#xA;Is shape an indicator of disease?&lt;br&gt;&#xA;Is there a size*shape effect?           &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Size          shapea    shapeb     shapec    %a    %b    %c&#xA;&amp;lt;4 mm       5 / 5771    0 / 26    1 / 522  0.1%  0.0%  0.2%&#xA;4-6 mm      9 /  739    1 / 11    2 / 149  1.2%  9.1%  1.3%&#xA;6-8 mm      3 /  496    0 /  7    1 / 126  0.6%  0.0%  0.8%&#xA;&amp;gt;=8 mm     33 /  276   18 / 26    8 / 146 12.0% 69.2%  5.5%&#xA;&#xA;Total      50 / 7282   19 / 70   12 / 943  0.7% 27.1%  1.3%&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Thanks,&#xA;Luther88&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edits&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;@suncoolsu &#xA;in detail these are nodule data.  size is the size of the nodule (the cutoffs are what have been published and accepted as standards).  shape is the shape of the nodule. the numerator is the number of malignant nodules and the denominator is the number of nodules of given size and shape combinations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;@onestop&#xA;i do not have the raw data to do logistic regression.  just this table.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;any further suggestions?&lt;/p&gt;&#xA;" OwnerUserId="3710" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-03-16T09:38:20.743" LastActivityDate="2011-03-16T09:38:20.743" Title="Indicators of disease" Tags="&lt;proportion&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="8287" PostTypeId="2" ParentId="1112" CreationDate="2011-03-14T21:20:55.353" Score="1" ViewCount="" Body="&lt;p&gt;My earlier post has a method to rank between 0 and 1. &lt;a href=&quot;http://stats.stackexchange.com/questions/8266/advice-on-classifier-input-correlation&quot;&gt;Advice on classifier input correlation&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the ranking I have used, Tmin/Tmax uses the sample min/max but you may find the population min/max more appropriate. Also look up z scores&lt;/p&gt;&#xA;" OwnerUserId="3484" LastActivityDate="2011-03-14T21:20:55.353" />
  <row Id="8289" PostTypeId="2" ParentId="8286" CreationDate="2011-03-14T21:42:36.733" Score="4" ViewCount="" Body="&lt;p&gt;You can use &lt;a href=&quot;http://en.wikipedia.org/wiki/Logistic_regression&quot; rel=&quot;nofollow&quot;&gt;logistic regression&lt;/a&gt; to answer these questions. Disease is the outcome. Fitting a model with size and shape as covariates will allow you to test for a size effect and a shape effect, and adding an interaction between size and shape will allow you to test for a size*shape interaction. You can get &lt;em&gt;p&lt;/em&gt;-value for each from &lt;a href=&quot;http://en.wikipedia.org/wiki/Likelihood-ratio_test&quot; rel=&quot;nofollow&quot;&gt;likelihood ratio tests&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It probably makes sense to treat shape as categorical, i.e. use indicator variables, but fit a linear effect of size, at least to start off with. To do that you could represent size by the midpoint of the two inner bounded intervals, i.e. 5 and 7, and for the outer two intervals either use the mean sizes if you know them or pick some reasonable values (you know more than me how small and large the smallest and largest values are so what are reasonable values).&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-03-14T21:42:36.733" />
  <row Id="8290" PostTypeId="1" AcceptedAnswerId="8295" CreationDate="2011-03-14T22:23:47.850" Score="3" ViewCount="125" Body="&lt;p&gt;I have data in this format:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;Words,Source,Result&#xA;pestology Gomel, cheap, 0&#xA;cocreating, cheap, 1&#xA;munitioner impersonating nonextinct, cheap, 1&#xA;Kolomna, expensive, 1&#xA;Enyo's snakemouth, expensive, 0&#xA;blueberries backare farriers, cheap, 0&#xA;markets rafales, cheap, 0&#xA;...&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://bit.ly/h2ynoG&quot; rel=&quot;nofollow&quot;&gt;http://bit.ly/h2ynoG&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How should I determine whether the phrases in column one can explain the Result value?  What if I want to determine which individual words in the phrase are correlated with successful results?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, how should I determine whether the data in which Source is cheap is representative of all of the data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="3712" LastEditorUserId="3489" LastEditorDisplayName="" LastEditDate="2011-03-15T19:11:31.293" LastActivityDate="2011-03-15T19:11:31.293" Title="How should I measure the relationship between a variable containing text and a binomial variable?" Tags="&lt;correlation&gt;&lt;binomial&gt;&lt;text-mining&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="3" />
  <row Id="8291" PostTypeId="1" CreationDate="2011-03-14T22:28:24.027" Score="3" ViewCount="163" Body="&lt;p&gt;Stata allows estimating clustered standard errors in models with fixed effects but not in models random effects?  Why is this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By clustered standard errors, I mean clustering as done by stata's cluster command (and as advocated in Bertrand, Duflo and Mullainathan).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By fixed effects and random effects, I mean varying-intercept.  I have not considered varying slope.&lt;/p&gt;&#xA;" OwnerUserId="3700" LastActivityDate="2011-03-15T01:01:45.957" Title="Clustered standard errors and multi-level models" Tags="&lt;clustering&gt;&lt;multilevel-analysis&gt;&lt;random-effects-model&gt;&lt;fixed-effects-model&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="8292" PostTypeId="1" CreationDate="2011-03-14T23:04:00.050" Score="0" ViewCount="53" Body="&lt;p&gt;Hi there, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have got a strange problem with using functions which are supposed to be in the libraries lmtest &amp;amp; language R (downloaded from cran.r-project recently), namely lrtest() and pvals.fnc() - however, they are not available. I have upgraded my R-version to 2.12.2 but it has not solved the problem.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(lmtest)&#xA;lrtest()&#xA;returns: Error: could not find function &quot;lrtest&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Any help would be greatly appreciated.&#xA;T&lt;/p&gt;&#xA;" OwnerUserId="3715" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-15T00:10:00.890" LastActivityDate="2011-03-15T00:10:00.890" Title="No lrtest() function available in lmtest" Tags="&lt;r&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="8293" PostTypeId="2" ParentId="8292" CreationDate="2011-03-14T23:12:21.900" Score="3" ViewCount="" Body="&lt;p&gt;post your sessionInfo(). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;lmtest:::lrtest(x) should call a generic function which then determines the appropriate code to run on 'x'.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have not updated your packages, you may perhaps need to do so, or simply re-run install.packages('lmtest') for your updated R install.&lt;/p&gt;&#xA;" OwnerUserId="317" LastActivityDate="2011-03-14T23:12:21.900" />
  <row Id="8294" PostTypeId="2" ParentId="8291" CreationDate="2011-03-15T01:01:45.957" Score="5" ViewCount="" Body="&lt;p&gt;When you cluster on some observed attribute, you are making a statistical correction to the standard errors to account for some presumed similarity in the distribution of observations within clusters. When you estimate a multi-level model with random effects, you are explicitly modeling that variation, not treating it simply as a nuisance, thus clustering is not needed.&lt;/p&gt;&#xA;" OwnerUserId="3265" LastActivityDate="2011-03-15T01:01:45.957" CommentCount="2" />
  <row Id="8295" PostTypeId="2" ParentId="8290" CreationDate="2011-03-15T02:13:29.387" Score="3" ViewCount="" Body="&lt;p&gt;I would start by transforming the phrases into numbers via a document term matrix, with a 1 denoting the presence of a word and 0 being the absence of a word. Then you can perform correlation analysis. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Document-term_matrix&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Document-term_matrix&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;R Code&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/1Vrqm.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="3489" LastEditorUserId="3489" LastEditorDisplayName="" LastEditDate="2011-03-15T16:55:53.313" LastActivityDate="2011-03-15T16:55:53.313" CommentCount="8" />
  <row Id="8297" PostTypeId="1" CreationDate="2011-03-15T04:06:00.550" Score="4" ViewCount="130" Body="&lt;p&gt;Hello all,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am looking for a self-study site on the web that will allow me to verify my understanding of some basic probability and stats concepts and operations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I would like is a site with data and problems (along with the solutions) that I could use to practice my skills (such as regression methods/analysis, pairwise t-tests, computation and interpretation of confidence intervals, etc.). I've seen workbooks like this, but the data is not available for me to import into R or Excel.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Surely there must be a site like this on the internet, arranged by topics for study, but somehow I have not been able to find it. Also, if anyone knows of a good workbook that comes with data (on CD or via web) I'd be interested in it too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Re tools, I'll be using R and Excel - so pointers re these would be great too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="3311" LastActivityDate="2011-03-15T11:37:46.567" CommunityOwnedDate="2011-03-15T12:03:42.653" Title="Looking for stats/probability practice problems with data and solutions" Tags="&lt;teaching&gt;&lt;learning&gt;&lt;resources&gt;" AnswerCount="3" CommentCount="3" FavoriteCount="1" />
  <row Id="8298" PostTypeId="2" ParentId="8297" CreationDate="2011-03-15T04:09:32.143" Score="5" ViewCount="" Body="&lt;p&gt;If you are interested in Statistical Machine Learning, which seems to be THE thing these days, &lt;a href=&quot;http://www-stat.stanford.edu/~tibs/ElemStatLearn/&quot; rel=&quot;nofollow&quot;&gt;Tibshirani, Hastie, and Friedman's book&lt;/a&gt; is an invaluable resource. It is the latest edition and has a self contained website devoted to it.&lt;/p&gt;&#xA;" OwnerUserId="1307" LastActivityDate="2011-03-15T04:09:32.143" CommunityOwnedDate="2011-03-15T12:03:42.653" />
  <row Id="8299" PostTypeId="1" AcceptedAnswerId="8302" CreationDate="2011-03-15T05:44:31.573" Score="4" ViewCount="48" Body="&lt;p&gt;While trying to make sense of MDL and stochastic complexity, I found this previous question: &lt;a href=&quot;http://stats.stackexchange.com/questions/2828&quot;&gt;Measures of model complexity&lt;/a&gt;, in which Yaroslav Bulatov defines model complexity as &quot;how hard it is to learn from limited data.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is not clear to me how Minimal Description Length (MDL) measures this.  What I am looking for is some sort of probability inequality (analagous to the VC upper bound) which relates the &quot;code length&quot; of a model with its worst case behavior on fitting data generated by itself.  If such a concrete result cannot be found in the literature, even an empirical example would be enlightening.&lt;/p&gt;&#xA;" OwnerUserId="3567" LastEditorUserId="2860" LastEditorDisplayName="" LastEditDate="2011-03-16T20:23:20.790" LastActivityDate="2011-03-16T20:23:20.790" Title="Relationship between MDL and &quot;difficulty of learning from data&quot;" Tags="&lt;model-selection&gt;&lt;mdl&gt;" AnswerCount="2" FavoriteCount="2" />
  <row Id="8300" PostTypeId="1" CreationDate="2011-03-15T06:59:28.983" Score="4" ViewCount="182" Body="&lt;p&gt;I am investigating the effect of a drug on EEG and cognition in epilepsy patients.&#xA;We have done EEG and neuropsychological (NP) tests twice, before and after medical treatment.&#xA;Some of parameters in EEG and NP tests were significantly altered after drug treatment (done by Wilcoxon rank sum test, as the distribution was not normal).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, I would like to calculate correlation between change in EEG parameters and change in NP tests.&#xA;There are 7 parameters in EEG parameters and 27 in NP tests. I would like to run correlation analysis on every pair and see if any pair is significantly correlated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My questions are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Should I use nonparametric correlation, say Spearman, to calculate rho? (as values of EEG and NP paramaters are not normal)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Since there are 7 parameters in EEG and 27 parameters in NP tests, there should be 7*27 number of testing, which requires correction of p value for multiple comparisons. Which method should I use? &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I am looking for not-so-conservative method. I have seen from another posting that I can use permutation test for multiple comparisons. Could someone explain how I can do this?&lt;/p&gt;&#xA;" OwnerUserId="3720" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-03-15T07:17:34.970" LastActivityDate="2011-03-15T15:33:39.937" Title="Correlation analysis and correcting p values for multiple testing" Tags="&lt;correlation&gt;&lt;multiple-comparisons&gt;&lt;permutation&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="8301" PostTypeId="2" ParentId="8300" CreationDate="2011-03-15T07:31:45.437" Score="3" ViewCount="" Body="&lt;p&gt;In my opinion you should test if your variables are distributed normally and chose a suitable test accordingly. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Concerning the correction for alpha inflation: What you are doing is data mining. You have experimental data and now you are digging in it to find ... anything. Do that. But also know that anything you might find is just an observation and as such not reliable. Perform that exploratory thing, pick some promising pairs of variables and conduct new experiments to test these pairs for correlation. &lt;/p&gt;&#xA;" OwnerUserId="1048" LastActivityDate="2011-03-15T07:31:45.437" CommentCount="1" />
  <row Id="8302" PostTypeId="2" ParentId="8299" CreationDate="2011-03-15T07:36:50.213" Score="3" ViewCount="" Body="&lt;p&gt;The worst case of code length of a code induced by a probability density is just the negative log of the most unlikely event, since $L_p(x) = -\log p(x)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am currently working through Peter Grünwald's book 'The MDL principle'. He defines the complexity of a model class as the log of the sum of probabilities it can assign to data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, the more different datasets a model can fit, the more complex it is. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$x^n$ here is a complete data set. $X^n$ is the set of all possible data sets of length $n$. This might seem confusing, but it is done this way wo allow non-iid data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The formal definition goes as follows: $$COMP(\mathcal{M}) := \log \sum_{x^n \in X^n} p_{x^n}(x^n)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $p_{x^n}$ is the optimal $p \in \mathcal{M}$ with regard to $x^n$. &lt;/p&gt;&#xA;" OwnerUserId="2860" LastEditorUserId="2860" LastEditorDisplayName="" LastEditDate="2011-03-15T07:42:07.103" LastActivityDate="2011-03-15T07:42:07.103" />
  <row Id="8303" PostTypeId="1" CreationDate="2011-03-15T07:45:52.233" Score="10" ViewCount="298" Body="&lt;p&gt;Hi, I am fitting a binomial family glm in R, and I have a whole troupe of explanatory variables, and I need to find the best (R-squared as a measure is fine). Short of writing a script to loop through random different combinations of the explanatory variables and then recording which performs the best, I really dont know what to do. And the &lt;code&gt;leaps&lt;/code&gt; function from package &lt;strong&gt;&lt;a href=&quot;http://cran.r-project.org/web/packages/leaps/index.html&quot; rel=&quot;nofollow&quot;&gt;leaps&lt;/a&gt;&lt;/strong&gt; does not seem to do logistic regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help or suggestions would be greatly appreciated&#xA;Leendert   &lt;/p&gt;&#xA;" OwnerUserId="3721" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-03-15T08:01:02.250" LastActivityDate="2011-03-16T16:04:09.750" Title="How to do logistic regression subset selection?" Tags="&lt;r&gt;&lt;logistic-regression&gt;" AnswerCount="3" CommentCount="4" FavoriteCount="4" />
  <row Id="8304" PostTypeId="2" ParentId="8267" CreationDate="2011-03-15T07:55:15.623" Score="2" ViewCount="" Body="&lt;p&gt;If you are using &lt;code&gt;which&lt;/code&gt;, then &lt;code&gt;subset&lt;/code&gt; is unnecessary. If we have a logical condition these are equivalent in R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data[cond,]&#xA;data[which(cond),]&#xA;subset(data,cond)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The benefit of the subset is that you do not need to use &lt;code&gt;$&lt;/code&gt; to get to the variables you are subsetting on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furthermore if you do successive subsetings it makes more sense to concatenate all the conditions and then do subseting. So if you have the condition 1 and condition 2, which themselves may be complicated conditions then instead of &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pruned1 &amp;lt;- data[cond1, ]&#xA;pruned2 &amp;lt;- pruned1[cond2, ]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Do &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cond &amp;lt;- cond1 &amp;amp; cond2&#xA;pruned2 &amp;lt;- data[cond,]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The main benefit of the second approach that it is faster and cleaner, since you will not be creating new objects. &lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-03-15T07:55:15.623" CommentCount="2" />
  <row Id="8305" PostTypeId="2" ParentId="8303" CreationDate="2011-03-15T08:54:26.503" Score="6" ViewCount="" Body="&lt;p&gt;First of all $R^2$ is not an appropriate goodness-of-fit measure for logistic regression, take an information criterion $AIC$ or $BIC$, for example, as a good alternative.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Logistic regression is estimated by maximum likelihood method, so &lt;code&gt;leaps&lt;/code&gt; is not used directly here. An extension of &lt;code&gt;leaps&lt;/code&gt; to &lt;code&gt;glm()&lt;/code&gt; functions is the &lt;a href=&quot;http://cran.r-project.org/web/packages/bestglm/index.html&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;bestglm&lt;/strong&gt;&lt;/a&gt; package (as usually recommendation follows, consult vignettes there).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may be also interested in the article by David W. Hosmer, Borko Jovanovic and Stanley Lemeshow &lt;strong&gt;&lt;a href=&quot;http://www.jstor.org/pss/2531779&quot; rel=&quot;nofollow&quot;&gt;Best Subsets Logistic Regression&lt;/a&gt;&lt;/strong&gt; // Biometrics Vol. 45, No. 4 (Dec., 1989), pp. 1265-1270 (usually accessible through the university networks).&lt;/p&gt;&#xA;" OwnerUserId="2645" LastActivityDate="2011-03-15T08:54:26.503" CommentCount="1" />
  <row Id="8306" PostTypeId="1" AcceptedAnswerId="8308" CreationDate="2011-03-15T10:02:23.260" Score="5" ViewCount="160" Body="&lt;p&gt;As title, is there anything like this? I know how to calculate CI for arithmetic mean, but how about geometric mean? Thanks.&lt;/p&gt;&#xA;" OwnerUserId="588" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-15T12:03:24.493" LastActivityDate="2011-03-15T15:51:17.957" Title="Confidence interval for geometric mean" Tags="&lt;confidence-interval&gt;&lt;mean&gt;" AnswerCount="1" />
  <row Id="8307" PostTypeId="2" ParentId="1112" CreationDate="2011-03-15T10:04:23.910" Score="10" ViewCount="" Body="&lt;p&gt;As often, my first question was going to be &quot;&lt;em&gt;why&lt;/em&gt; do you want to do this&quot;, then I saw you've already answered this in the comments to the question: &quot;&lt;em&gt;I am measuring content across many different dimensions and I want to be able to make comparisons in terms of how relevant a given piece of content is. Additionally, I want to display values across these dimensions that is explicable and easily understood.&lt;/em&gt;&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is no reason to normalize the data so that the max is 1 and the min is zero in order to achieve this, and my opinion is that this would be a &lt;strong&gt;bad idea in general&lt;/strong&gt;. The max or min values could very easily be &lt;a href=&quot;http://en.wikipedia.org/wiki/Outlier&quot; rel=&quot;nofollow&quot;&gt;outliers&lt;/a&gt; that are unrepresentative of the population distribution. @osknows parting remark about &lt;strong&gt;using &lt;a href=&quot;http://en.wikipedia.org/wiki/Z-score&quot; rel=&quot;nofollow&quot;&gt;$z$-scores&lt;/a&gt; is a much better idea&lt;/strong&gt;. $z$-scores (aka standard scores) normalize each variable using its standard deviation rather than its range. The standard deviation is less influenced by outliers. In order to use $z$-scores, it's preferable that each variable has a roughly normal distribution, or at least has a roughly symmetric distribution (i.e. isn't severely skew) but if necessary you can apply some appropriate &lt;a href=&quot;http://en.wikipedia.org/wiki/Data_transformation_%28statistics%29&quot; rel=&quot;nofollow&quot;&gt;data transformation&lt;/a&gt; first in order to achieve this; which transformation to use could be determined by finding the best fitting &lt;a href=&quot;http://en.wikipedia.org/wiki/Box-Cox_transformation&quot; rel=&quot;nofollow&quot;&gt;Box–Cox transformation&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="449" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-03-15T10:49:20.283" LastActivityDate="2011-03-15T10:49:20.283" CommentCount="1" />
  <row Id="8308" PostTypeId="2" ParentId="8306" CreationDate="2011-03-15T10:17:39.707" Score="11" ViewCount="" Body="&lt;p&gt;The geometric mean $(\prod_{i=1}^n X_i)^{1/n}$ is an arithmetic mean after taking logs $1/n \sum_{i=1}^n \log X_i$, so if you do know the CI for the arithmetic mean do the same for the logarithms of your data points and take exponents of the upper and lower bounds.&lt;/p&gt;&#xA;" OwnerUserId="2645" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-15T15:51:17.957" LastActivityDate="2011-03-15T15:51:17.957" CommentCount="2" />
  <row Id="8309" PostTypeId="1" CreationDate="2011-03-15T10:34:15.403" Score="1" ViewCount="179" Body="&lt;p&gt;Let A be $n \times p$ matrix of independent variables and B be the corresponding $n \times 1$ matrix of the dependent  values. In ridge regression, we define a parameter $\lambda$ so that: $\beta=(A^\mathrm{T}A+\lambda I)^{-1}A^\mathrm{T}B$ .&#xA;Now let [u s v]=svd(A) and $d_{i}=i^{th}$ diagonal entry of 's'. we define degrees of freedom (df)= $\sum_{i=1}^{n} \frac{(d_{i})^2}{(d_{i})^2+\lambda}$ . Ridge regression shrinks the coefficients of low-variance components and hence the parameter $\lambda$ controls the degrees of freedom.So for $\lambda=0$, which is the case of normal regression, df=n, and hence all the independent variables will be considered. The problem I am facing is to find the value of $\lambda$ given 'df' and the matrix 's'. I have tried to re-arrange the above equation but was not getting a closed form solution. Please provide any helpful pointers.&lt;/p&gt;&#xA;" OwnerUserId="3722" LastEditorUserId="3722" LastEditorDisplayName="" LastEditDate="2011-03-15T10:47:46.733" LastActivityDate="2011-03-15T14:48:46.887" Title="How to calculate regularization parameter in ridge regression given degrees of freedom and input matrix?" Tags="&lt;ridge-regression&gt;" AnswerCount="2" CommentCount="3" FavoriteCount="2" />
  <row Id="8310" PostTypeId="2" ParentId="7939" CreationDate="2011-03-15T10:42:34.337" Score="3" ViewCount="" Body="&lt;p&gt;The conditional mean is defined by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$E(Y|X)\equiv\int y f(y|x) dy$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $f(Y|X)$ is the conditional density.  Using the product rule, you can show:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$f(y|x)=\frac{f(y,x)}{f(x)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Substituting this back into the integral you get&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$E(Y|X)\equiv\frac{\int y f(y,x) dy}{f(x)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which is of the form you seek, if you use the kernel density estimator.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-03-15T10:42:34.337" CommentCount="3" />
  <row Id="8312" PostTypeId="2" ParentId="8303" CreationDate="2011-03-15T11:20:17.760" Score="10" ViewCount="" Body="&lt;p&gt;Stepwise and &quot;all subsets&quot; methods are generally bad.  See Stopping Stepwise: &lt;a href=&quot;http://www.nesug.org/proceedings/nesug07/sa/sa07.pdf&quot; rel=&quot;nofollow&quot;&gt;Why Stepwise Methods are Bad and what you Should Use&lt;/a&gt; by David Cassell and myself (we used SAS, but the lesson applies) or Frank Harrell Regression Modeling Strategies.  If you need an automatic method, I recommend LASSO or LAR.  A LASSO package for logistic regression is available &lt;a href=&quot;http://www4.stat.ncsu.edu/~boos/var.select/lasso.adaptive.html&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;, another interesting article is on the &lt;a href=&quot;http://www.stat.uiowa.edu/techrep/tr392.pdf&quot; rel=&quot;nofollow&quot;&gt;iterated LASSO for logistic&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="686" LastActivityDate="2011-03-15T11:20:17.760" CommentCount="4" />
  <row Id="8313" PostTypeId="2" ParentId="8297" CreationDate="2011-03-15T11:26:34.293" Score="4" ViewCount="" Body="&lt;p&gt;The &lt;strong&gt;&lt;a href=&quot;http://en.wikiversity.org/wiki/Statistics&quot; rel=&quot;nofollow&quot;&gt;Statistics topic area on Wikiversity&lt;/a&gt;&lt;/strong&gt; is worth a &lt;em&gt;look&lt;/em&gt;. It's got a long way to go before it's a comprehensive stand-alone syllabus, to be honest, but &lt;em&gt;some&lt;/em&gt; of the Courses are more advanced than others, and when there's not much material as yet there are often links to free online resources.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-03-15T11:26:34.293" CommunityOwnedDate="2011-03-15T11:28:53.603" />
  <row Id="8314" PostTypeId="2" ParentId="8297" CreationDate="2011-03-15T11:37:46.567" Score="3" ViewCount="" Body="&lt;p&gt;I realise that this may not be what you are looking for, but R core and all packages come with data sets on which to practice the functionalities in each package. Many of these data sets are quite famous, and often a link is given to the paper in which the data are described. You could use these datasets in R and then after you finish your analysis look at what the authors of the paper did with the same data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That being said, its rare for there to be a right answer in any real data analysis problem, mostly one learns by realising that your techniques were not appropriate and re-iterating until one reaches some level of statisfaction. Obviosuly this is a moving target though, as your skills increase an older dataset may yield new insights. &lt;/p&gt;&#xA;" OwnerUserId="656" LastActivityDate="2011-03-15T11:37:46.567" CommunityOwnedDate="2011-03-15T12:03:42.653" CommentCount="2" />
  <row Id="8315" PostTypeId="2" ParentId="8309" CreationDate="2011-03-15T12:11:19.050" Score="3" ViewCount="" Body="&lt;p&gt;A newton-rhapson/fisher-scoring/taylor-series algorithm would be suited to this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You have the equation to solve for $\lambda$&#xA;$$h(\lambda)=\sum_{i=1}^{p}\frac{d_{i}^{2}}{d_{i}^{2}+\lambda}-df=0$$&#xA;with derivative&#xA;$$\frac{\partial h}{\partial \lambda}=-\sum_{i=1}^{p}\frac{d_{i}^{2}}{(d_{i}^{2}+\lambda)^{2}}$$&#xA;You then get:&#xA;$$h(\lambda)\approx h(\lambda^{(0)})+(\lambda-\lambda^{(0)})\frac{\partial h}{\partial \lambda}|_{\lambda=\lambda^{(0)}}=0$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;re-arranging for $\lambda$ you get:&#xA;$$\lambda=\lambda^{(0)}-\left[\frac{\partial h}{\partial \lambda}|_{\lambda=\lambda^{(0)}}\right]^{-1}h(\lambda^{(0)})$$&#xA;This sets up the iterative search.  For initial starting values, assume $d^{2}_{i}=1$ in the summation, then you get $\lambda^{(0)}=\frac{p-df}{df}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\lambda^{(j+1)}=\lambda^{(j)}+\left[\sum_{i=1}^{p}\frac{d_{i}^{2}}{(d_{i}^{2}+\lambda^{(j)})^{2}}\right]^{-1}\left[\sum_{i=1}^{p}\frac{d_{i}^{2}}{d_{i}^{2}+\lambda^{(j)}}-df\right]$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This &quot;goes&quot; in the right direction (increase $\lambda$ when summation is too big, decrease it when too small), and typically only takes a few iterations to solve.  Further the function is monotonic (an increase/decrease in $\lambda$ will always decrease/increase the summation), so that it will converge uniquely (no local maxima).&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-03-15T12:11:19.050" CommentCount="4" />
  <row Id="8316" PostTypeId="1" AcceptedAnswerId="8322" CreationDate="2011-03-15T13:11:49.413" Score="3" ViewCount="59" Body="&lt;p&gt;I assume a simple model $\log(y_i) \sim \mathcal{N}(\mu_i,\sigma)$ with $\mu_i=\alpha + \beta x_i$ . Now if I have the estimates $\hat{\alpha}$ and $\hat{\beta}$, how do I calculate the fitted values $\hat{y}$? (Using $\hat{y}= \exp(\hat{\alpha} + \hat{\beta} x_i)$ seems to work; this would be the median of the log-normal distribution.) &lt;/p&gt;&#xA;" OwnerUserId="1443" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-03-15T13:16:59.297" LastActivityDate="2011-03-15T15:17:13.680" Title="Fitted values for a log-normal model" Tags="&lt;modeling&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="8317" PostTypeId="2" ParentId="8283" CreationDate="2011-03-15T14:04:23.587" Score="2" ViewCount="" Body="&lt;p&gt;Hi Sam R,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not exactly sure what your code is trying to do, but it seems like you should be using $rnorm()$ (with a large $n$) instead of $dnorm()$, since the functions $mean()$, $var()$, etc. are designed to be used on samples, not densities.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In any case, using the method-of-moments is easy to do algebraically.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say your sample is $X=[x_1, \cdots, x_n]$.  Your sample moments are:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$m_1(X) = (1/n) \sum_{i} x_i$$&#xA;$$m_2(X) = (1/n) \sum_{i} x_i^2$$&#xA;$$m_3(X) = (1/n) \sum_{i} x_i^3$$&#xA;$$m_4(X) = (1/n) \sum_{i} x_i^4$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The moments of $Z \sim N(\mu, \sigma^2)$ are:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$m_1(Z) = \mu$$&#xA;$$m_2(Z) = \mu^2 + \sigma^2$$&#xA;$$m_3(Z) = \mu^3 + 3\mu\sigma^2$$&#xA;$$m_4(Z) = \mu^3 + 6\mu^2\sigma^2 + 3(\sigma^2)^2$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furthermore, if your random variable $Y$ is a mixture of $Z_1, Z_2, \cdots, Z_m$, with $P[X=Z_i]=p_i$, then&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$m_1(Y) = p_1 m_1(Z_1) + p_2 m_1 (Z_2) + \cdots + p_m m_1 (Z_m)$$&#xA;$$m_2(Y) = p_1 m_2(Z_1) + p_2 m_2 (Z_2) + \cdots + p_m m_1 (Z_m)$$&#xA;$$\cdots$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you use enough gaussians you should be able to match a finite number of moments exactly.  Under constraint you may need to define a distance metric on your moments vector $(m_1, m_2, m_3, m_4)$.  The euclidean distance should work fine but you will want to differentially weight the moments, since the higher-order moments could be very large or very small depending on what distribution you are working with.  Also, the first and second moments are generally more important.&lt;/p&gt;&#xA;" OwnerUserId="3567" LastEditorUserId="3567" LastEditorDisplayName="" LastEditDate="2011-03-15T14:27:33.083" LastActivityDate="2011-03-15T14:27:33.083" />
  <row Id="8318" PostTypeId="1" CreationDate="2011-03-15T14:27:14.323" Score="4" ViewCount="160" Body="&lt;p&gt;One of the predictors in my logistic model has been log transformed. How do you interpret the estimated coefficient of the log transformed predictor and how do you calculate the impact of that predictor on the odds ratio?&lt;/p&gt;&#xA;" OwnerUserId="3724" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-16T13:06:23.200" LastActivityDate="2011-03-16T13:06:23.200" Title="Interpretation of log transformed predictors in logistic regression" Tags="&lt;logistic-regression&gt;" AnswerCount="1" />
  <row Id="8319" PostTypeId="2" ParentId="8309" CreationDate="2011-03-15T14:34:44.480" Score="2" ViewCount="" Body="&lt;p&gt;Here is the small Matlab code based on the formula proved by probabilityislogic:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    function [lamda] = calculate_labda(Xnormalised,df)&#xA;        [n,p] = size(Xnormalised);   &#xA;&#xA;        %Finding SVD of data&#xA;        [u s v]=svd(Xnormalised);&#xA;        Di=diag(s);&#xA;        Dsq=Di.^2;&#xA;&#xA;        %Newton-rapson method to solve for lamda&#xA;        lamdaPrev=(p-df)/df;&#xA;        lamdaCur=Inf;%random large value&#xA;        diff=lamdaCur-lamdaPrev;   &#xA;        threshold=eps(class(XstdArray));    &#xA;        while (diff&amp;gt;threshold)          &#xA;            numerator=(sum(Dsq ./ (Dsq+lamdaPrev))-df);        &#xA;            denominator=sum(Dsq./((Dsq+lamdaPrev).^2));        &#xA;            lamdaCur=lamdaPrev+(numerator/denominator);        &#xA;            diff=lamdaCur-lamdaPrev;        &#xA;            lamdaPrev=lamdaCur;        &#xA;        end&#xA;        lamda=lamdaCur;&#xA;    end&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="3722" LastEditorUserId="3722" LastEditorDisplayName="" LastEditDate="2011-03-15T14:48:46.887" LastActivityDate="2011-03-15T14:48:46.887" CommentCount="1" />
  <row Id="8320" PostTypeId="2" ParentId="8318" CreationDate="2011-03-15T14:40:15.637" Score="6" ViewCount="" Body="&lt;p&gt;If you exponentiate the estimated coefficient, you'll get an &lt;strong&gt;odds ratio associated with a $b$-fold increase&lt;/strong&gt; in the predictor, where $b$ is the base of the logarithm you used when log-transforming the predictor. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I usually choose to take logarithms to base 2 in this situation, so I can interpet the exponentiated coefficient as an odds ratio associated with a &lt;em&gt;doubling&lt;/em&gt; of the predictor. &lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-03-15T14:40:15.637" CommentCount="3" />
  <row Id="8321" PostTypeId="1" CreationDate="2011-03-15T15:02:17.730" Score="3" ViewCount="255" Body="&lt;p&gt;Hi all,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In multiple linear regression, let us say that the F test shows that the model is significant. But the t-tests for beta values does not say that the beta values are non-zero. What can we conclude in such a situation?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does the fact that the tests for the beta values failed affect the fact that the model is significant?&lt;/p&gt;&#xA;" OwnerUserId="3725" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-16T13:05:34.637" LastActivityDate="2011-03-16T13:05:34.637" Title="Significant multiple linear regression model with non-significant betas?" Tags="&lt;multiple-regression&gt;" AnswerCount="2" CommentCount="3" />
  <row Id="8322" PostTypeId="2" ParentId="8316" CreationDate="2011-03-15T15:17:13.680" Score="2" ViewCount="" Body="&lt;p&gt;The &lt;em&gt;conditional expectation&lt;/em&gt; of $y_i$ at the value $x_i$ can be estimated as $\exp(\hat{\alpha} + \hat{\beta}x_i + s^2/2)$ where $s^2$ is the mean squared error in your regression (estimating $\sigma^2$).  Don't trust this until you have closely looked for evidence of heteroscedasticity (&lt;em&gt;i.e.&lt;/em&gt;, changes in the true value of $\sigma$ for different values of $x$) and found nothing of importance.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-03-15T15:17:13.680" />
  <row Id="8323" PostTypeId="2" ParentId="8300" CreationDate="2011-03-15T15:33:39.937" Score="2" ViewCount="" Body="&lt;p&gt;You want to perform a &lt;strong&gt;canonical correlation analysis&lt;/strong&gt;.  This will provide information about correlations among &lt;em&gt;linear combinations&lt;/em&gt; of your sets of parameters, potentially uncovering stronger information of the type you seek.  The &lt;a href=&quot;http://en.wikipedia.org/wiki/Canonical_correlation&quot; rel=&quot;nofollow&quot;&gt;Wikipedia article&lt;/a&gt; explains the theory, provides the equations, and presents the appropriate hypothesis test.  It requires solving a 7 by 7 eigenvalue problem determined by the correlation and cross-correlation matrices of the variables, which is fast and straightforward.  The &lt;code&gt;R&lt;/code&gt; package &lt;a href=&quot;http://cc.oulu.fi/~jarioksa/softhelp/vegan/html/CCorA.html&quot; rel=&quot;nofollow&quot;&gt;CCorA&lt;/a&gt; will do it, for instance.  (I haven't tested this package.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It might help first to re-express your original variables so they are each approximately symmetrically distributed.  There are many ways to do this using, for example, &lt;a href=&quot;http://www.itl.nist.gov/div898/handbook/pmc/section5/pmc52.htm&quot; rel=&quot;nofollow&quot;&gt;Box-Cox transformations&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-03-15T15:33:39.937" CommentCount="4" />
  <row Id="8324" PostTypeId="2" ParentId="8321" CreationDate="2011-03-15T17:02:20.803" Score="1" ViewCount="" Body="&lt;p&gt;Read the section on tests of joint significance here: &lt;a href=&quot;http://www.people.fas.harvard.edu/~blackwel/ftests.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.people.fas.harvard.edu/~blackwel/ftests.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your F-test is of a different hypothesis than the t test, that all beta = 0, rather than some beta = 0. &lt;/p&gt;&#xA;" OwnerUserId="1893" LastActivityDate="2011-03-15T17:02:20.803" />
  <row Id="8325" PostTypeId="2" ParentId="8278" CreationDate="2011-03-15T17:10:20.340" Score="1" ViewCount="" Body="&lt;p&gt;Since your data are categorical, a t test is likely inappropriate for any parameter I can think you might want to do inference on. Here is a brief intro to a very similar problem as yours:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://walkerbioscience.com/word-files/Categorical%20data.doc&quot; rel=&quot;nofollow&quot;&gt;http://walkerbioscience.com/word-files/Categorical%20data.doc&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1893" LastActivityDate="2011-03-15T17:10:20.340" CommentCount="2" />
  <row Id="8326" PostTypeId="2" ParentId="8276" CreationDate="2011-03-15T17:14:46.560" Score="1" ViewCount="" Body="&lt;p&gt;R/Rapidminer are free. Deployment can get tricky if you are concerned about multiple GB datasets, but other than that, everything is a breeze. RStudio on a linux server has been getting great reviews, and Revolution Analytics sells an enterprise version of R at a very reasonable cost with a focus on enhanced deployability, big data, fancy UI, and data export to make R more accessible to the non-statistician.&lt;/p&gt;&#xA;" OwnerUserId="1893" LastActivityDate="2011-03-15T17:14:46.560" />
  <row Id="8327" PostTypeId="2" ParentId="8271" CreationDate="2011-03-15T18:43:25.903" Score="0" ViewCount="" Body="&lt;p&gt;Set it up as a regular regression, they are mathematically equivalent, just make sure your non-continuous variables are defined appropriately (as categorical or whatever) to SPSS.&lt;/p&gt;&#xA;" OwnerUserId="1893" LastActivityDate="2011-03-15T18:43:25.903" />
  <row Id="8328" PostTypeId="1" CreationDate="2011-03-15T19:27:04.463" Score="9" ViewCount="271" Body="&lt;p&gt;I'm trying to implement an EM algorithm for the following factor analysis model;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$W_j = \mu+B a_j+e_j \quad\text{for}\quad j=1,\ldots,n$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $W_j$ is p-dimensional random vector, $a_j$ is a q-dimensional vector of latent variables and $B$ is a pxq matrix of parameters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a result of other assumptions used for the model, I know that $W_j\sim N(\mu, BB&amp;#39;+D)$ where $D$ is the variance covariance matrix of error terms $e_j$, $D$ = diag($\sigma_1^2$,$\sigma_2^2$,...,$\sigma_p^2$).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the EM algorithm to work, I'm doing dome iterations involving estimation of $B$ and $D$ matrices and during these iterations I'm computing the inverse of $BB&amp;#39;+D$ at each iteration using new estimates of $B$ and $D$. Unfortunately during the course of iterations, $BB&amp;#39;+D$ loses its positive definiteness (but it shouldn't because it is a variance-covariance matrix) and this situation ruins the convergence of the algorithm. My questions are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Does this situation show that there is something wrong with my algorithm since the likelihood should increase at every step of EM?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;What are the practical ways to make a matrix positive definite?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Edit: I'm computing the inverse by using a matrix inversion lemma which states that:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$(BB&amp;#39;+D)^{-1}=D^{-1}-D^{-1}B (I_q+B&amp;#39;D^{-1}B)^{-1} B&amp;#39;D^{-1}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where the right side involves only the inverses of $q\times q$ matrices.&lt;/p&gt;&#xA;" OwnerUserId="3499" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-17T20:09:34.710" LastActivityDate="2011-03-17T20:09:34.710" Title="How to make a matrix positive definite?" Tags="&lt;factor-analysis&gt;&lt;em-algorithm&gt;" AnswerCount="1" CommentCount="7" FavoriteCount="0" />
  <row Id="8329" PostTypeId="2" ParentId="8321" CreationDate="2011-03-15T19:59:57.180" Score="3" ViewCount="" Body="&lt;p&gt;One way that this could happen is to put two highly correlated predictors into your model. Because they are highly correlated, each of their coefficients will have relatively wide standard errors. As a result, these coefficients may not be statistically significant &lt;em&gt;separately&lt;/em&gt;. Taken together, they may be a strong predictor of the outcome. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The covariance between the coefficient estimates would be pretty negative (increasing the estimate of one of the coefficients leads to a strong decrease in the estimate of the other coefficient because the two are highly linearly related), cancelling out the high variances for the estimators separately. As a result, your model has predictive power and your regressors are important, but their colinearity obscures this fact by creating large (though accurate) standard errors.&lt;/p&gt;&#xA;" OwnerUserId="401" LastActivityDate="2011-03-15T19:59:57.180" />
  <row Id="8330" PostTypeId="1" CreationDate="2011-03-15T20:31:46.950" Score="2" ViewCount="172" Body="&lt;p&gt;Hi, I have a multiple response (categories) question (Q11, ..., Q15). My problem is that the data-entry clerck repeats the answers in the same case. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;       Q11 Q12 Q13 Q14 Q15&#xA;Case 1 001 003 015 001 022 (001 was type twice)&#xA;Case 2 089 032 089 089 014 (089 was typed three times)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I need to find a way to identified the cases with these errors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks,&lt;/p&gt;&#xA;" OwnerUserId="3729" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-16T13:04:04.497" LastActivityDate="2011-03-16T13:04:04.497" Title="Multiple response question with duplicated answers" Tags="&lt;spss&gt;" CommentCount="2" />
  <row Id="8332" PostTypeId="2" ParentId="8328" CreationDate="2011-03-15T21:56:51.387" Score="2" ViewCount="" Body="&lt;p&gt;OK, since you're doing FA I'm assuming that $B$ is of full column rank $q$ and $q&amp;lt;p$. We need a few more details though. This may be a numerical problem; it may also be a problem with your data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How are you computing the inverse? Do you need the inverse explicitly, or can re-express the calculation as the solution to a linear system? (ie to get $A^{-1}b$ solve $Ax=b$ for x, which is typically faster and more stable)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is happening to $D$? Are the estimates really small/0/negative? In some sense it is the critical link, because $BB&amp;#39;$ is of course rank deficient and defines a singular covariance matrix before adding $D$, so you can't invert it. Adding the positive diagonal matrix $D$ technically makes it full rank but $BB&amp;#39;+D$ could still be horribly ill conditioned if $D$ is small.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Oftentimes the  estimate for the idiosyncratic variances (your $\sigma^2_i$, the diagonal elements of $D$) is near zero or even negative; these are called Heywood cases. See eg &lt;a href=&quot;http://www.technion.ac.il/docs/sas/stat/chap26/sect21.htm&quot; rel=&quot;nofollow&quot;&gt;http://www.technion.ac.il/docs/sas/stat/chap26/sect21.htm&lt;/a&gt; (any FA text should discuss this as well, it's a very old and well-known problem). This can result from model misspecification, outliers, bad luck, solar flares... the MLE is particularly prone to this problem, so if your EM algorithm is designed to get the MLE look out.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your EM algorithm is approaching a mode with such estimates it's possible for $BB&amp;#39;+D$ to lose its positive definiteness, I think. There are various solutions; personally I'd prefer a Bayesian approach but even then you need to be careful with your priors (improper priors or even proper priors with too much mass near 0 can have the same problem for basically the same reason)&lt;/p&gt;&#xA;" OwnerUserId="26" LastEditorUserId="26" LastEditorDisplayName="" LastEditDate="2011-03-17T18:21:42.010" LastActivityDate="2011-03-17T18:21:42.010" CommentCount="3" />
  <row Id="8333" PostTypeId="2" ParentId="7969" CreationDate="2011-03-15T22:29:01.410" Score="2" ViewCount="" Body="&lt;p&gt;I'm answering my own question, but I found a pretty good solution, even if I do not fully understand it. Looking at the code from the GNU Scientific Library, here is how it samples gamma variables (&lt;code&gt;r&lt;/code&gt; is the random number generator, &lt;code&gt;a&lt;/code&gt; is $\alpha$ and &lt;code&gt;b&lt;/code&gt; is $\beta$):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  if (a &amp;lt; 1)&#xA;    {&#xA;      double u = gsl_rng_uniform_pos (r);&#xA;      return gsl_ran_gamma (r, 1.0 + a, b) * pow (u, 1.0 / a);&#xA;   }&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;gsl_ran_gamma&lt;/code&gt; is the function which returns a gamma random sample (so the above is a recursive call), while &lt;code&gt;gsl_rng_uniform_pos&lt;/code&gt; returns a uniformly distributed number in $(0,1)$ (the &lt;code&gt;_pos&lt;/code&gt; is for strictly positive as it is guaranteed to not return 0.0).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, I can take the log of the last expression and use&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;return log(gsl_ran_gamma(r, 1.0 + a, b)) + log(u)/a;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To get what I wanted. I now have two &lt;code&gt;log()&lt;/code&gt; calls (but one less &lt;code&gt;pow()&lt;/code&gt;), but the result is probably better. Before, as whuber pointed out, I had something raised to the power of $1/a$, potentially a huge number. Now, in logspace, I'm multiplying by $1/a$. So, it is less likely to underflow.&lt;/p&gt;&#xA;" OwnerUserId="2067" LastEditorUserId="2067" LastEditorDisplayName="" LastEditDate="2011-03-16T18:15:03.767" LastActivityDate="2011-03-16T18:15:03.767" CommentCount="5" />
  <row Id="8334" PostTypeId="1" AcceptedAnswerId="8357" CreationDate="2011-03-15T22:59:36.823" Score="5" ViewCount="129" Body="&lt;p&gt;I am trying to set up an automatic screen to detect structural breaks in large numbers of time series.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The time series are weekly and represent behaviour of customers. I have set up a Chow test. I use the most recent 4 weeks and compare it to the preceding 22. I want to know if their recent behaviour has significantly diverged from the preceding behaviour. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is this: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Is the Chow Test the most appropriate test for this question?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;If this is not the most appropriate test how can I determine what test is the most appropriate one?&lt;/p&gt;&#xA;" OwnerUserId="2997" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-16T13:02:46.637" LastActivityDate="2011-03-16T13:50:24.793" Title="Chow test or not?" Tags="&lt;time-series&gt;&lt;hypothesis-testing&gt;&lt;chow-test&gt;&lt;change-point&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="2" />
  <row Id="8335" PostTypeId="1" AcceptedAnswerId="8353" CreationDate="2011-03-15T23:30:08.777" Score="2" ViewCount="130" Body="&lt;p&gt;I am a beginner in statistics and poor in mathematics. I am trying to to assess effect of intervention in one state versus another using annual data.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;My data are&#xA;State 1         State 2 &#xA;Cases   Deaths  Cases   Deaths&#xA;2004    1125    5   2024    254&#xA;2005    1213    5   1978    209&#xA;2006    1003    4   2294    217&#xA;2007    1425    6   2312    249&#xA;2008    1172    4   1528    197&#xA;2009    1092    3   1683    204&#xA;2010    1316    4   2024    218&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When I was stuck for the correct statistical procedure, one senior member &quot;irishstat&quot; advised me the following very convincing analysis for which I am ever grateful:&#xA;X1 is the number of cases and Y is the number of deaths. X2 is the empirically identified point of anomaly; (2009 .. period 6 for State1 and 2004 .. period 1 for State2. Outlier detection led to identifying one anomalous data point for each state reflecting some unknown background variable thus yielding a more robust estimate of the mortality rates.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Analysis of State1&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;State1 Y(T) = -.65649&#xA;+[X1(T)][.0046)] CASES +[X2(T)][-1.3608)] PULSE6 I~P00006STATE1 + [A(T)]&#xA;Suggesting an unusually low mortality rate for 2009&#xA;Analysis of State2&#xA;State2 Y(T) = 123.55&#xA;+[X1(T)][(+ .0468)] CASES +[X2(T)][(+ 35.7590)] PULSE1 I~P00001STATE2 + [A(T)]&#xA;Suggesting an unusually high mortality rate for 2004&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This leads to estimating two cleansed data points&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;STATE    YEAR   Y OBSERVED   Y ADJUSTED&#xA;STATE1   2009   3            4.36&#xA;STATE2   2004   254          218.24&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Replacing these two observed possibly errant values possibly due to some unspecified concomitant factor (“lurking Variable”) one computes a rate of.0046 for STATE1 and .0468 for STATE2.&#xA;My problem now is how to do The Chow Test for constancy of parameters across groups to check for  rejection of the null hypothesis of equal coefficients. I have SPSS v19. Kindly advise me step by step. &lt;/p&gt;&#xA;" OwnerUserId="2956" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-16T13:01:26.090" LastActivityDate="2011-03-17T11:37:41.620" Title="How to do Chow test for constancy of parameters across 2 groups" Tags="&lt;statistical-significance&gt;&lt;spss&gt;&lt;chow-test&gt;" AnswerCount="1" />
  <row Id="8336" PostTypeId="2" ParentId="8206" CreationDate="2011-03-15T23:45:40.963" Score="1" ViewCount="" Body="&lt;p&gt;Fully customizable &lt;a href=&quot;http://had.co.nz/ggplot2/&quot; rel=&quot;nofollow&quot;&gt;ggplot2&lt;/a&gt; boxplot...&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#bootstrap&#xA;data &amp;lt;- data.frame(value=rnorm(100,mean = 0.5, sd = 0.2),group=0)&#xA;#processing&#xA;metaData &amp;lt;- ddply(data,~group,summarise,&#xA;            mean=mean(data$value),&#xA;            sd=sd(data$value),&#xA;            min=min(data$value),&#xA;            max=max(data$value),&#xA;            median=median(data$value),&#xA;            Q1=0,Q3=0&#xA;            )&#xA;bps &amp;lt;- boxplot.stats(data$value,coef=1.5) &#xA;metaData$min &amp;lt;- bps$stats[1] #lower wisker&#xA;metaData$max &amp;lt;- bps$stats[5] #upper wisker&#xA;metaData$Q1 &amp;lt;- bps$stats[2] # 1st Quartile&#xA;metaData$Q3 &amp;lt;- bps$stats[4] # 3rd Quartile&#xA;&#xA;#adding outliers&#xA;out &amp;lt;- data.frame() #initialising storage for outliers&#xA;if(length(bps$out) &amp;gt; 0){&#xA;    for(n in 1:length(bps$out)){&#xA;        pt &amp;lt;-data.frame(value=bps$out[n],group=0) &#xA;        out&amp;lt;-rbind(out,pt) &#xA;    }&#xA;}&#xA;#adding labels&#xA;labels &amp;lt;-data.frame(value=metaData$max, label=&quot;Upper bound&quot;)&#xA;labels &amp;lt;-rbind(labels,data.frame(value=metaData$min, label=&quot;Lower bound&quot;))&#xA;labels &amp;lt;-rbind(labels,data.frame(value=metaData$median, label=&quot;Median&quot;))&#xA;labels &amp;lt;-rbind(labels,data.frame(value=metaData$Q1, label=&quot;First quartile&quot;))&#xA;labels &amp;lt;-rbind(labels,data.frame(value=metaData$Q3, label=&quot;Third quartile&quot;))&#xA;&#xA;#drawing&#xA;library(ggplot2)&#xA;p &amp;lt;- ggplot(metaData,aes(x=group,y=mean))&#xA;p &amp;lt;- p + geom_segment(aes(x=c(0.1,0,0.1),y=c(0,0,1),xend=c(0,0,-0.1),yend=c(0,1,1)))&#xA;p &amp;lt;- p + geom_text(aes(y=c(0,1),label=c(0,1),x=0.2))&#xA;p &amp;lt;- p + geom_errorbar(aes(ymin=min,ymax=max),linetype = 1,width = 0.5) #main range&#xA;p &amp;lt;- p + geom_linerange(aes(ymin=min,ymax=max),linetype = 1,width = 0, color=&quot;white&quot;)# white line range&#xA;p &amp;lt;- p + geom_linerange(aes(ymin=min,ymax=max),linetype = 2)    #main range dotted&#xA;p &amp;lt;- p + geom_crossbar(aes(y=median,,ymin=Q1,ymax=Q3),linetype = 1,fill='white') #box&#xA;if(length(out) &amp;gt;0) p &amp;lt;- p + geom_point(data=out,aes(x=group,y=value),shape=4) # drawning outliers if any&#xA;p &amp;lt;- p + scale_x_discrete(breaks=c(0))&#xA;p &amp;lt;- p + scale_y_continuous(name= &quot;Value&quot;)&#xA;p &amp;lt;- p + geom_text(data=labels,aes(x=0.5,y=value,label=round(value,2)),colour=&quot;black&quot;,angle=0,hjust=0.5, vjust=0.5,size=3)&#xA;&#xA;p &amp;lt;- p + opts(panel.background = theme_rect(fill = &quot;white&quot;,colour = NA)) &#xA;p &amp;lt;- p + opts(panel.grid.minor = theme_blank(), panel.grid.major = theme_blank())&#xA;p &amp;lt;- p + opts(axis.title.x=theme_blank())&#xA;p &amp;lt;- p + opts(axis.text.x = theme_blank())&#xA;p &amp;lt;- p + opts(axis.title.y=theme_blank())&#xA;p &amp;lt;- p + opts(axis.text.y = theme_blank())&#xA;&#xA;p + coord_flip()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Result:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/HMkMM.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;...code maybe a bit ugly but works the right way.&lt;/p&gt;&#xA;" OwnerUserId="3376" LastEditorUserId="3376" LastEditorDisplayName="" LastEditDate="2011-03-16T00:07:39.330" LastActivityDate="2011-03-16T00:07:39.330" CommentCount="0" />
  <row Id="8337" PostTypeId="2" ParentId="1112" CreationDate="2011-03-16T00:27:24.787" Score="0" ViewCount="" Body="&lt;p&gt;A very simple option is dividing each number in your data by the largest number in your data.  If you have many small numbers and a few very large ones, this might not convey the information well.  But it's relatively easy; if you think meaningful information is lost when you graph the data like this, you could try one of the more sophisticated techniques that others have suggested.&lt;/p&gt;&#xA;" OwnerUserId="3700" LastActivityDate="2011-03-16T00:27:24.787" />
  <row Id="8338" PostTypeId="1" AcceptedAnswerId="8339" CreationDate="2011-03-16T01:37:12.077" Score="2" ViewCount="58" Body="&lt;p&gt;I'm looking at binomial data where I believe that the probability of the outcome is the product of two independent factors. If you think of it as a two step decision, &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;At the first step, there is a certain probability $p$ that you will have a success, and $1-p$ that you will have a failure. If you have a failure at this step, there is no further opportunity for success. &lt;/li&gt;&#xA;&lt;li&gt;Then, at the second step, there is a certain probability $q$ of success.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;So, the overall probability of success is $p\times q$. I'm assuming, for now, that $p$ and $q$ vary independently across individuals. Both $p$ and $q$ have some variance across individuals, but not necessarily the same one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Due to the nature of the data, I can estimate the variance of $q$, and the variance of $p\times q$. From this, I would like to estimate the variance of $p$. I just can't wrap my mind around how.&lt;/p&gt;&#xA;" OwnerUserId="287" LastActivityDate="2011-03-17T12:41:40.457" Title="Estimating variability of unseen factor" Tags="&lt;probability&gt;&lt;estimation&gt;&lt;binomial&gt;&lt;variance&gt;" AnswerCount="1" />
  <row Id="8339" PostTypeId="2" ParentId="8338" CreationDate="2011-03-16T01:46:27.583" Score="3" ViewCount="" Body="&lt;p&gt;Supposing for independent $X$, $Y$, we know $E[X]$, $E[XY]$, $var(X)$, $var(XY)$.  Thus we know $E[Y] = E[XY]/E[X]$, $E[X^2]=var(X) - E[X]^2$ and $E[(XY)^2]=var[XY]-E[XY]^2$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since $E[(XY)^2] = E[X^2 Y^2] = E[X^2] E[Y^2]$ for independent $X$, $Y$, we have $E[Y^2] = E[(XY)^2]/E[X^2]$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To summarize,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$var(Y) = \frac{var(XY) + E[XY]^2}{var(X) + E[X]^2}- \left(\frac{E[XY]}{E[X]}\right)^2.$$&lt;/p&gt;&#xA;" OwnerUserId="3567" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-03-17T12:41:40.457" LastActivityDate="2011-03-17T12:41:40.457" />
  <row Id="8340" PostTypeId="1" CreationDate="2011-03-16T03:51:14.743" Score="1" ViewCount="181" Body="&lt;p&gt;&lt;strong&gt;Update: I wanted to clarify that this is a simulation. Sorry if I confused everyone. I have also used meaningful names for my variables.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not a statistician so please correct me if I make a blunder in explaining what I want. In regard to my &lt;a href=&quot;http://stats.stackexchange.com/questions/7996/what-is-a-good-way-of-estimating-the-dependence-of-an-output-variable-on-the-inpu&quot;&gt;previous question&lt;/a&gt;, I have reproduced parts of my question here for reference.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;I am evaluating a scenario's output&#xA;  dependence on three&#xA;  variables: Area, Speed and NumOfVehicles. For this, I am&#xA;  conducting the following experiments:&lt;/p&gt;&#xA;  &#xA;  &lt;ul&gt;&#xA;  &lt;li&gt;Fix Area+Speed, Vary NumOfVehicles - Total four sets of (Area+Speed) each having 4 variations of NumOfVehicles&lt;/li&gt;&#xA;  &lt;li&gt;Fix Speed+NumOfVehicles, Vary Area - Total four sets of (Speed+NumOfVehicles) each having 3 variations of Area&lt;/li&gt;&#xA;  &lt;li&gt;Fix NumOfVehicles+Area, Vary Speed - Total four sets of (NumOfVehicles+Area) each having 6 variations of Speed&lt;/li&gt;&#xA;  &lt;/ul&gt;&#xA;  &#xA;  &lt;p&gt;The output of any simulation is the&#xA;  value of a variable over time. The output&#xA;  variable I am observing is the time at which 80% of the cars crashe.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;I am trying to determine which&#xA;  parameter(s) dominate the outcome of&#xA;  the experiment. By dominate, I mean&#xA;  that sometimes, the outcomes just does&#xA;  not change when one of the parameters&#xA;  change but when some other parameter&#xA;  is changed even by a small amount, a&#xA;  large change in the output is&#xA;  observed. I need to capture this&#xA;  effect and output some analysis from&#xA;  which I can understand the dependence&#xA;  of the output on the input parameters.&#xA;  A friend suggested Sensitivity&#xA;  Analysis but am not sure if there are&#xA;  simpler ways of doing it. Can someone&#xA;  please help me with a good (possibly&#xA;  easy because I don't have a Stats&#xA;  background) technique? It would be&#xA;  great if all this can be done in R.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;My previous result was not very satisfactory looking at the regression results. So what I did was that I went ahead and repeated all my experiments 20 times each with different variations of each variable (so for instance, instead of 4 variations of Area, I now have 8 and so on). Following is the summary I obtained out of R after using linear regression:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Call:&#xA;lm(formula = T ~ Area + Speed + NumOfVehicles)&#xA;&#xA;Residuals:&#xA;     Min       1Q   Median       3Q      Max &#xA;-0.13315 -0.06332 -0.01346  0.04484  0.29676 &#xA;&#xA;Coefficients:&#xA;                 Estimate Std. Error t value Pr(&amp;gt;|t|)    &#xA;(Intercept)      0.04285    0.02953   1.451    0.148    &#xA;Area             0.70285    0.02390  29.406  &amp;lt; 2e-16 ***&#xA;Speed           -0.15560    0.02080  -7.479 2.12e-12 ***&#xA;NumOfVehicles   -0.27447    0.02927  -9.376  &amp;lt; 2e-16 ***&#xA;---&#xA;Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 &#xA;&#xA;Residual standard error: 0.08659 on 206 degrees of freedom&#xA;Multiple R-squared: 0.8304, Adjusted R-squared: 0.8279 &#xA;F-statistic: 336.2 on 3 and 206 DF,  p-value: &amp;lt; 2.2e-16 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;as opposed to my previous result:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;lm(formula = T ~ Area + Speed + NumOfVehicles)&#xA;&#xA;Residuals:&#xA;     Min       1Q   Median       3Q      Max &#xA;-0.35928 -0.06842 -0.00698  0.05591  0.42844 &#xA;&#xA;Coefficients:&#xA;               Estimate Std. Error t value Pr(&amp;gt;|t|)    &#xA;(Intercept)   -0.01606    0.16437  -0.098 0.923391    &#xA;Area           0.80199    0.15792   5.078 0.000112 ***&#xA;Speed         -0.27440    0.13160  -2.085 0.053441 .  &#xA;NumOfVehicles -0.31898    0.14889  -2.142 0.047892 *  &#xA;---&#xA;Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 &#xA;&#xA;Residual standard error: 0.1665 on 16 degrees of freedom&#xA;Multiple R-squared: 0.6563, Adjusted R-squared: 0.5919 &#xA;F-statistic: 10.18 on 3 and 16 DF,  p-value: 0.0005416 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;From my understanding, my current results have a lower standard error so that is good. In addition the Pr value also seems quite low which tells me that this result is better than my previous result. So can I go ahead and say that A has the maximum effect on the output and then come S and V in that order? Can I make any other deductions from this result?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, I was suggested that I look into adding additional variates like $A^2$ etc. but if $A$ is the area, what does saying &quot;time&quot; depends on $A^2$ actually mean? &lt;/p&gt;&#xA;" OwnerUserId="2164" LastEditorUserId="2164" LastEditorDisplayName="" LastEditDate="2011-03-16T16:07:19.607" LastActivityDate="2011-03-16T16:07:19.607" Title="Comparing and understanding my linear regression result with a previous attempt." Tags="&lt;r&gt;&lt;regression&gt;&lt;experiment-design&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="8342" PostTypeId="1" AcceptedAnswerId="8378" CreationDate="2011-03-16T04:49:19.827" Score="6" ViewCount="118" Body="&lt;p&gt;Hi all, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a three level contingency table, with count data for several species, the host plant from which they were collected and whether that collection happened on a rainy day (this actually matters!). Using R, fake data might be something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;count &amp;lt;- rpois(8, 10)&#xA;species &amp;lt;- rep(c(&quot;a&quot;, &quot;b&quot;), 4)&#xA;host &amp;lt;- rep(c(&quot;c&quot;,&quot;c&quot;, &quot;d&quot;, &quot;d&quot;), 2)&#xA;rain &amp;lt;- c(rep(0,4), rep(1,4))&#xA;my.table &amp;lt;- xtabs(count ~ host + species + rain)&#xA;&#xA;&#xA;, , rain = 0&#xA;&#xA;    species&#xA;host  a  b&#xA;   c 12 15&#xA;   d 10 13&#xA;&#xA;, , rain = 1&#xA;&#xA;    species&#xA;host  a  b&#xA;   c 11 12&#xA;   d 12  7&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, I want to know two things: Are species associated with host plants? Does &quot;rain or not&quot; effect this association. I used loglm() from MASS for this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#Are species independent to host plants, given the effect of rain?&#xA;loglm(~species + host + rain + species*rain + host*rain, data=my.table)&#xA;&#xA;#Given any realationship between host plants and species, does rain change it?&#xA;loglm(~species+host+rain+species*host)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is a little outside my comfort level, and wanted to check that I'd set models up right and that this was the best way to approach these questions. &lt;/p&gt;&#xA;" OwnerUserId="3732" LastEditorUserId="3467" LastEditorDisplayName="" LastEditDate="2011-03-16T16:18:58.100" LastActivityDate="2011-03-17T08:06:04.890" Title="Appropriate way to deal with a 3-level contingency table" Tags="&lt;r&gt;&lt;categorical-data&gt;&lt;log-linear&gt;" AnswerCount="3" FavoriteCount="1" />
  <row Id="8343" PostTypeId="2" ParentId="8340" CreationDate="2011-03-16T05:06:26.793" Score="1" ViewCount="" Body="&lt;p&gt;This result looks great. One thing that I would recommend doing is looking at the residuals for each of the two models to see if there are any trends in the first model that are not present in the second. If the model has residuals with an approximately normal distribution then you won't need to add more terms like A^2. The command for this is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;my_model &amp;lt;- (formula = T ~ A + S + V)&#xA;resid(my_model&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="3727" LastActivityDate="2011-03-16T05:06:26.793" CommentCount="1" />
  <row Id="8344" PostTypeId="1" CreationDate="2011-03-16T06:50:47.653" Score="4" ViewCount="110" Body="&lt;p&gt;I am trying to understand how influence functions work. Could someone explain in the context of a simple OLS regression&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{equation}&#xA;y_i = \alpha + \beta \cdot x_i + \varepsilon_i&#xA;\end{equation}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where I want the &lt;a href=&quot;http://en.wikipedia.org/wiki/Robust_statistics#Empirical_influence_function&quot; rel=&quot;nofollow&quot;&gt;influence function&lt;/a&gt; for $\beta$.&lt;/p&gt;&#xA;" OwnerUserId="2693" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-29T00:57:46.037" LastActivityDate="2011-04-29T00:57:46.037" Title="Influence functions and OLS" Tags="&lt;regression&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="8345" PostTypeId="2" ParentId="8299" CreationDate="2011-03-16T07:16:45.877" Score="1" ViewCount="" Body="&lt;p&gt;MDL for model selection seems to mean &quot;choose model with lowest parametric complexity&quot;. Since parametric complexity is equal to worst case excess code length when you don't know the distribution, I bet it also gives you the number of observations needed to get close to true member of &lt;code&gt;M&lt;/code&gt; in KL-divergence. Parametric complexity is a function of number of observations &lt;code&gt;n&lt;/code&gt; and is given as &lt;code&gt;COMP&lt;/code&gt; in bayer's answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Chapter 14 and chapter 7 of Grunwald's book give more details&lt;/p&gt;&#xA;" OwnerUserId="511" LastActivityDate="2011-03-16T07:16:45.877" />
  <row Id="8346" PostTypeId="2" ParentId="8340" CreationDate="2011-03-16T07:17:02.403" Score="5" ViewCount="" Body="&lt;p&gt;First, to begin with, I think it was ok to rephrase the question in your previous post, now I just remember I have read it, but all the discussion is lost. I beg pardons if the ideas here are repeated (was lazy to search for the original post).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are some doubts regarding your experimental scheme and conclusions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Is the data you are working with: 3 explanatory variables A, V and S (not A, B, C as you wrote and it is nor correct to call them &lt;em&gt;parameters&lt;/em&gt;, parameters are the coefficients you estimate by linear regression) and the dependent T, true empirical data you observe in real life or you just simulate the data based on some likely values that you &lt;em&gt;believe&lt;/em&gt; are true? However even in &lt;em&gt;believable&lt;/em&gt; profile of the attributes (T, A, V, S) what is the joint distribution then? The conclusion that the data is not from real observations: goes from here&#xA;&lt;em&gt;So what I did was that I went ahead and repeated all my experiments 20 times each with different variations of each parameters (so for instance, instead of 4 variations of A, I now have 8 and so on).&lt;/em&gt;&#xA;Thus there is (I guess) little to do with &lt;a href=&quot;http://en.wikipedia.org/wiki/Bootstrapping_%28statistics%29&quot; rel=&quot;nofollow&quot;&gt;bootstrapping&lt;/a&gt; or any other re-sampling techniques or Monte Carlo simulations. Therefore by manually setting the experimental design you do the conclusions that have little empirical background. I wish to be wrong in this place though.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Impacts of the variables are not measured by the value of parameters NOR by the smaller standard errors, p values! If you wish to compare the impacts in terms of marginal effects you need to implement the &lt;a href=&quot;http://www.nd.edu/~rwilliam/stats1/x92.pdf&quot; rel=&quot;nofollow&quot;&gt;standardized regression&lt;/a&gt; coefficients also known as beta coefficients. You can calculate them manually (coef()*sd()/sd()) or redo the regressions scaling all 4 variables first with &lt;code&gt;scale()&lt;/code&gt; function. You may compare then the marginal effects that are represented by beta parameters. Standardized data is dimensionless so you may compare the impacts of different explanatory variables. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;However if the variables are slowly varying and strictly positive you may be interested not in marginal impacts, but in percent changes that are given by &lt;a href=&quot;http://en.wikipedia.org/wiki/Elasticity_of_a_function&quot; rel=&quot;nofollow&quot;&gt;elasticities&lt;/a&gt; of different variables. Take the original model and estimated parameters coef() and multiply it by mean()/mean(), in additive model you get mean elasticity then. Note that you can't do the same in standardized (zero-mean) model, and since the fractions are in general different so will be betas and elasticities. The interpretation of elasticity is straightforward if the explanatory variable changes by 1 per cent what is the per cent change of dependent variable.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;All other suggestion like take transformations, add some other variables ($A^2$ for example) comes from the further statistical inspection of the residuals for regression's traditional weak spots: is linear model a good approximation (RESET), no multicollinearity (VIF, CI), no heteroscedasticity (White or any other), no autocorrelation (DW, Ljung-Box, etc.)  and so on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S. the current result is neither good nor bad, it is just looks suspicious to me.&lt;/p&gt;&#xA;" OwnerUserId="2645" LastActivityDate="2011-03-16T07:17:02.403" CommentCount="2" />
  <row Id="8347" PostTypeId="1" CreationDate="2011-03-11T19:47:33.830" Score="13" ViewCount="331" Body="&lt;p&gt;While attending conferences, there has been a bit of a push by advocates of Bayesian statistics for assessing the results of experiments.  It is vaunted as both more sensitive, appropriate, and selective towards genuine findings (fewer false positives) than frequentist statistics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have explored the topic somewhat, and I am left unconvinced so far of the benefits to using Bayesian statistics.  Bayesian analyses were used to refute &lt;a href=&quot;http://dbem.ws/&quot; rel=&quot;nofollow&quot;&gt;Daryl Bem&lt;/a&gt;'s research supporting precognition, however, so I remain cautiously curious about how Bayesian analyses might benefit even my own research.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I am curious about the following:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Power in a Bayesian analysis vs. a frequentist analysis&lt;/li&gt;&#xA;&lt;li&gt;Susceptibility to Type 1 error in each type of analysis&lt;/li&gt;&#xA;&lt;li&gt;The trade-off in complexity of the analysis (Bayesian seems more complicated) vs. the benefits gained.  Traditional statistical analyses are straightforward, with well-established guidelines for drawing conclusions.  The simplicity could be viewed as a benefit.  Is that worth giving up?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Thanks for any insight!&lt;/p&gt;&#xA;" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-03-16T07:50:42.073" LastActivityDate="2011-03-18T11:36:47.960" Title="Is Bayesian statistics genuinely an improvement over traditional (frequentist) statistics for behavioral research?" Tags="&lt;bayesian&gt;&lt;power&gt;&lt;frequentist&gt;" AnswerCount="3" CommentCount="16" FavoriteCount="4" />
  <row Id="8348" PostTypeId="2" ParentId="8347" CreationDate="2011-03-13T04:16:06.883" Score="2" ViewCount="" Body="&lt;p&gt;I am not familiar with Bayesian Statistics myself but I do know that Skeptics Guide to the Universe Episode 294 has and interview with Eric-Jan Wagenmakers where they discuss Bayesian Statistics. Here is a link to the podcast:&#xA;&lt;a href=&quot;http://www.theskepticsguide.org/archive/podcastinfo.aspx?mid=1&amp;amp;pid=294&quot; rel=&quot;nofollow&quot;&gt;http://www.theskepticsguide.org/archive/podcastinfo.aspx?mid=1&amp;amp;pid=294&lt;/a&gt;    &lt;/p&gt;&#xA;" LastActivityDate="2011-03-13T04:16:06.883" />
  <row Id="8349" PostTypeId="1" CreationDate="2011-03-16T09:06:15.457" Score="1" ViewCount="51" Body="&lt;p&gt;Hi all,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm a complete noob when it comes to statistics, so please excuse my lack of standard terminology. I think my question might be related to normalization, but please re-tag if I'm wrong. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm a programmer by trade, and I've been tasked with creating a chart which shows how the number of errors in a data set change from month to month. Unfortunately we don't have a statistician in-house to help me figure it out...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data is as follows: Each month, a number of items are entered into a database. These items are checked for errors, and the errors are also entered into the database. Each item can have multiple errors of different types, and the number of items added each month can vary a lot. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I show how the error frequency changes from month to month? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It doesn't &quot;feel&quot; right to show a percentage of errors/number of items since there can be multiple errors for each item. The chart also needs to be easy to understand even by people with no background in statistics, so something as close as possible to the real values or percentages would be good.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An example of the data with total number of errors per month:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;Month  Errors   Items &#xA;=====================&#xA;March     208    2027&#xA;April     276    1304&#xA;May       609    1721&#xA;June      167    1561&#xA;July      268     513&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The errors for March could be broken down like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;Type        Errors&#xA;==================&#xA;Type 1           2&#xA;Type 2          43&#xA;Type 3          93&#xA;Type 4           0&#xA;Type 5           1&#xA;Type 6           0&#xA;Type 7           4&#xA;Type 8           0&#xA;Type 9          22&#xA;Type 10          0&#xA;Type 11         43&#xA;&lt;/pre&gt;&#xA;" OwnerUserId="3735" LastEditorUserId="3735" LastEditorDisplayName="" LastEditDate="2011-03-16T11:55:43.973" LastActivityDate="2011-03-19T10:40:15.893" Title="Charting errors based on number of items per month" Tags="&lt;data-visualization&gt;&lt;summary-statistics&gt;" AnswerCount="1" CommentCount="6" />
  <row Id="8350" PostTypeId="2" ParentId="8347" CreationDate="2011-03-16T09:23:16.897" Score="4" ViewCount="" Body="&lt;p&gt;Bayesian statistics can be derived from a few Logical principles.  Try Searching &quot;probability as extended logic&quot; and you will find more in depth analysis of the fundamentals.  But basically, Bayesian statistics rests on three basic &quot;desiderata&quot; or normative principles:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The plausability of a proposition is to be represented by a single real number&lt;/li&gt;&#xA;&lt;li&gt;The plausability if a proposition is to have qualitative correspondance with &quot;common sense&quot;.  If given initial plausibility $p(A|C^{(0)})$, then change from $C^{(0)}\rightarrow C^{(1)}$ such that $p(A|C^{(1)})&amp;gt;p(A|C^{(0)})$ (A becomes more plausible) and also $p(B|A C^{(0)})=p(B|AC^{(1)})$ (given A, B remains just as plausible) then we must have $p(AB| C^{(0)})\leq p(AB|C^{(1)})$ (A and B must be at least as plausible) and $p(\overline{A}|C^{(1)})&amp;lt;p(\overline{A}|C^{(0)})$ (not A must become less plausible).&lt;/li&gt;&#xA;&lt;li&gt;The plausability of a proposition is to be calculated &lt;em&gt;consistently&lt;/em&gt;.  This means a) if a plausability can be reasoned in more than 1 way, all answers must be equal; b) In two problems where we are presented with the same information, we must assign the same plausabilities; and c) we must take account of all the information that is available.  We must not add information that isn't there, and we must not ignore information which we do have.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;These three desiderata (along with the rules of logic and set theory) uniquely determine the sum and product rules of probability theory.  Thus, if you would like to reason according to the above three desiderata, they you &lt;strong&gt;must&lt;/strong&gt; adopt a Bayesian approach.  You do not have to adopt the &quot;Bayesian Philosophy&quot; but you must adopt the numerical results.&#xA;&lt;a href=&quot;http://bayes.wustl.edu/etj/prob/book.pdf&quot; rel=&quot;nofollow&quot;&gt;The first three chapters of this book&lt;/a&gt; describe these in more detail, and provide the proof.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And last but not least, the &quot;Bayesian machinery&quot; is the most powerful data processing tool you have.  This is mainly because of the desiderata 3c) using all the information you have (this also explains why Bayes can be more complicated than non-Bayes).  It can be quite difficult to decide &quot;what is relevant&quot; using your intuition.  Bayes theorem does this for you (and it does it without adding in arbitrary assumptions, also due to 3c).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: to address the question more directly (as suggested in the comment), suppose you have two hypothesis $H_0$ and $H_1$.  You have a &quot;false negative&quot; loss $L_1$ (Reject $H_0$ when it is true: type 1 error) and &quot;false positive&quot; loss $L_2$ (Accept $H_0$ when it is false: type 2 error).  probability theory says you should:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Calculate $P(H_0|E_1,E_2,\dots)$, where $E_i$ is all the pieces of evidence related to the test: data, prior information, whatever &lt;em&gt;you want the calculation to incorporate into the analysis&lt;/em&gt;&lt;/li&gt;&#xA;&lt;li&gt;Calculate $P(H_1|E_1,E_2,\dots)$&lt;/li&gt;&#xA;&lt;li&gt;Calculate the odds $O=\frac{P(H_0|E_1,E_2,\dots)}{P(H_1|E_1,E_2,\dots)}$&lt;/li&gt;&#xA;&lt;li&gt;Accept $H_0$ if $O &amp;gt; \frac{L_2}{L_1}$&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Although you don't really need to introduce the losses.  If you just look at the odds, you will get one of three results: i) definitely $H_0$, $O&amp;gt;&amp;gt;1$, ii) definitely $H_1$, $O&amp;lt;&amp;lt;1$, or iii) &quot;inconclusive&quot; $O\approx 1$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now if the calculation becomes &quot;too hard&quot;, then you must either approximate the numbers, or ignore some information.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a actual example with worked out numbers &lt;a href=&quot;http://stats.stackexchange.com/questions/8419/bayesian-classifier-with-multivariate-normal-densities/8434#8434&quot;&gt;see my answer to this question&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2392" LastEditorUserId="2392" LastEditorDisplayName="" LastEditDate="2011-03-18T11:36:47.960" LastActivityDate="2011-03-18T11:36:47.960" CommentCount="12" />
  <row Id="8351" PostTypeId="1" AcceptedAnswerId="8359" CreationDate="2011-03-16T09:29:27.813" Score="3" ViewCount="232" Body="&lt;p&gt;I have a quarterly time series and test for stationarity with an augmented Dickey-Fuller test using R.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;adf.test(myseries)&#xA;# returns&#xA;# Dickey-Fuller = -3.9828, Lag order = 4, p-value = 0.01272&#xA;# alternative hypothesis: stationary &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;so the H0 is rejected. I tried to validate this intuitively and regressed the same series on a linear trend.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x&amp;lt;- 104:1&#xA;fit.1&amp;lt;-lm(myseries~x)&#xA;summary(fit.1)&#xA;#returns&#xA;# x      0.024  1.31e-05 ***&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Even though a simple linear model is not so appropriate here and the intercept is large (around 80), there seems to be a slight downwards trend over time, which is in line with my thoughts after looking at the initial data. So do I get the adf.test wrong or is the trend just to small to be discovered? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Besides I used&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plot(stl(myseries,&quot;per&quot;))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and ended up with a graph which sidebars suggested that trend and remainder were the main components driving the data, while seasonal influence was negligible. I saw that &lt;code&gt;stl()&lt;/code&gt; uses Local Polynomial Regression Filtering and got a rough idea how that works (still I wonder why smoothed trends of Hadley's ggplot2 package looked that different even though it uses the same method by default).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So summing up I got:&#xA;- adf finding no evidence for a trend&#xA;- a slight downwards trends &quot;detected&quot; by eyeballing and the naive approach&#xA;- loess decomposition stating that the trend has strong influence (by the relation of its bars in the plot)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So what can I learned from this? Probably I do have a terminology problem here, because the former two seem to address time trends while the latter address some other trend I cannot fully grasp yet. Maybe my question is just: Can you help me to understand the trend extracted by loess? And how is it related to smoothed / filtered stuff like HP-Filter or Kalman Smoothing (if there is a relationship and similarity does not only occur in my case)?&lt;/p&gt;&#xA;" OwnerUserId="704" LastEditorUserId="704" LastEditorDisplayName="" LastEditDate="2011-03-16T15:06:42.183" LastActivityDate="2011-03-16T22:55:24.480" Title="Trend or no trend?" Tags="&lt;r&gt;&lt;time-series&gt;&lt;loess&gt;" AnswerCount="3" CommentCount="3" />
  <row Id="8352" PostTypeId="1" CreationDate="2011-03-16T10:00:26.227" Score="1" ViewCount="64" Body="&lt;p&gt;Hello,&#xA;I would like to analyse the following type of experiment usig two-level fully nested ANOVA. We have mice of two genotypes (Group factor) and for each of the genotypes we take a certain number of mice (sub-groups) and measure several identical samples per each mouse. Thus it looks like a classic two-level nested design with Genotype and mice as fixed factors. I am interested in the main group effect of the genotype on the measurement outcome. I used Statistica where F value is calculated by the ratio of MSgroup/MSresidual. However I found several notions on the web that it is better to calculate F value from the &#xA;ratio of MSgroup/MSsub-group. &#xA;For example, on &lt;a href=&quot;http://udel.edu/~mcdonald/statnested.html&quot; rel=&quot;nofollow&quot;&gt;John McDonald's web-page&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In a two-level nested anova, there are two F statistics, one for subgroups (Fsubgroup) and one for groups (Fgroup). The subgroup F-statistic is found by dividing the among-subgroup mean square, MSsubgroup (the average variance of subgroup means within each group) by the within-subgroup mean square, MSwithin (the average variation among individual measurements within each subgroup). The group F-statistic is found by dividing the among-group mean square, MSgroup (the variation among group means) by MSsubgroup. The P-value is then calculated for the F-statistic at each level...&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;On the same web-page, he comments that in Rweb: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Fgroup is calculated as MSgroup/MSwithin, which is not a good idea if Fsubgroup is significant.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In my analysis, if there was a significant Group(Genotype) effect calculated as MSgroup/MSresidual, I never see a significant effect of sub-groups (mice). Does it mean that I can use MSgroup/MSresidual ratio for calculating F?&lt;/p&gt;&#xA;" OwnerUserId="1727" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-03-16T12:49:14.007" LastActivityDate="2011-03-16T15:13:35.767" Title="What is the proper ratio of mean squares for the two-level nested ANOVA?" Tags="&lt;anova&gt;" AnswerCount="1" />
  <row Id="8353" PostTypeId="2" ParentId="8335" CreationDate="2011-03-16T10:25:22.210" Score="2" ViewCount="" Body="&lt;p&gt;Using the suggested corrected data we have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The model to be tested is : Y(T)=B0 + B1*X(T) + A(T)&#xA;The null hypothesis is that the set B0 and B1 are the same over the two states&lt;/p&gt;&#xA;&#xA;&lt;p&gt;step 1 : Estimate this for STATE1 obtaining an error sum of squares SOS1 =.789&lt;/p&gt;&#xA;&#xA;&lt;p&gt;step 2 : Estimate this for STATE2 obtaining an error sum of squares SOS2 = 548.272&lt;/p&gt;&#xA;&#xA;&lt;p&gt;step 3 : Estimate this for all of the data (12 pairs) obtaining an error sum of squares SOS3 = 23920.4&lt;/p&gt;&#xA;&#xA;&lt;p&gt;step 4 : Compute NUM= [SOS3-(SOS2+SOS3)]/2 = 11685&lt;/p&gt;&#xA;&#xA;&lt;p&gt;step 5 : Compute MSE for composite analysis =23920.4/10 = 2392&lt;/p&gt;&#xA;&#xA;&lt;p&gt;step 5 : F value = NUM / MSE = 11685/2392 = 4.9&lt;/p&gt;&#xA;&#xA;&lt;p&gt;STEP 6 : An F OF 4.9 with 2 and 10 degrees of freedom is .03 Thus the hypothesis of                 equality is rejected at alpha &amp;lt; .03 ; accepted otherwise&lt;/p&gt;&#xA;" OwnerUserId="3382" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-03-17T11:37:41.620" LastActivityDate="2011-03-17T11:37:41.620" CommentCount="4" />
  <row Id="8354" PostTypeId="2" ParentId="8344" CreationDate="2011-03-16T11:27:17.790" Score="5" ViewCount="" Body="&lt;p&gt;Influence functions are basically an analytical tool that can be used to assess the effect (or &quot;influence&quot;) of removing an observation on the value of a statistic &lt;em&gt;without having to re-calculate that statistic&lt;/em&gt;.  They can also be used to create asymptotic variance estimates.  If influence equals $I$ then asymptotic variance is $\frac{I^2}{n}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The way I understand influence functions is as follows.  You have some sort of theoretical CDF, denoted by $F_{i}(y)=Pr(Y_{i}&amp;lt;y_{i})$.  For simple OLS, you have&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Pr(Y_{i}&amp;lt;y_{i})=Pr(\alpha+\beta x_{i} + \epsilon_{i} &amp;lt; y_{i})=\Phi\left(\frac{y_{i}-(\alpha+\beta x_{i})}{\sigma}\right)$$&#xA;Where $\Phi(z)$ is the standard normal CDF, and $\sigma^2$ is the error variance.  Now you can show that any statistic will be a function of this CDF, hence the notation $S(F)$ (i.e. some function of $F$).  Now suppose we change the function $F$ by a &quot;little bit&quot;, to $F_{(i)}(z)=(1+\zeta)F(z)-\zeta \delta_{(i)}(z)$ Where $\delta_{i}(z)=I(y_{i}&amp;lt;z)$, and $\zeta=\frac{1}{n-1}$.  Thus $F_{(i)}$ represents the CDF of the data with the &quot;ith&quot; data point removed.  We can do a taylor series of $F_{(i)}(z)$ about $\zeta=0$.  This gives:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$S[F_{(i)}(z,\zeta)] \approx S[F_{(i)}(z,0)]+\zeta\left[\frac{\partial S[F_{(i)}(z,\zeta)]}{\partial \zeta}|_{\zeta=0}\right]$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that $F_{(i)}(z,0)=F(z)$ so we get:&#xA;$$S[F_{(i)}(z,\zeta)] \approx S[F(z)]+\zeta\left[\frac{\partial S[F_{(i)}(z,\zeta)]}{\partial \zeta}|_{\zeta=0}\right]$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The partial derivative here is called the influence function.  So this represents an approximate &quot;first order&quot; correction to be made to a statistic due to deleting the &quot;ith&quot; observation.  Note that in regression the remainder does not go to zero asymtotically, so that this is an approximation to the changes you may actually get.  Now write $\beta$ as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\beta=\frac{\frac{1}{n}\sum_{j=1}^{n}(y_{j}-\overline{y})(x_{j}-\overline{x})}{\frac{1}{n}\sum_{j=1}^{n}(x_{j}-\overline{x})^2}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus beta is a function of two statistics: the variance of X and covariance between X and Y.  These two statistics have representations in terms of the CDF as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$cov(X,Y)=\int(X-\mu_x(F))(Y-\mu_y(F))dF$$&#xA;and&#xA;$$var(X)=\int(X-\mu_x(F))^{2}dF$$&#xA;where&#xA;$$\mu_x=\int xdF$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To remove the ith observation we replace $F\rightarrow F_{(i)}=(1+\zeta)F-\zeta \delta_{(i)}$ in both integrals to give:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\mu_{x(i)}=\int xd[(1+\zeta)F-\zeta \delta_{(i)}]=\mu_x-\zeta(x_{i}-\mu_x)$$&#xA;$$Var(X)_{(i)}=\int(X-\mu_{x(i)})^{2}dF_{(i)}=\int(X-\mu_x+\zeta(x_{i}-\mu_x))^{2}d[(1+\zeta)F-\zeta \delta_{(i)}]$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ignoring terms of $\zeta^{2}$ and simplifying we get:&#xA;$$Var(X)_{(i)}\approx Var(X)-\zeta\left[(x_{i}-\mu_x)^2-Var(X)\right]$$&#xA;Similarly for the covariance&#xA;$$Cov(X,Y)_{(i)}\approx Cov(X,Y)-\zeta\left[(x_{i}-\mu_x)(y_{i}-\mu_y)-Cov(X,Y)\right]$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So we can now express $\beta_{(i)}$ as a function of $\zeta$.  This is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\beta_{(i)}(\zeta)\approx \frac{Cov(X,Y)-\zeta\left[(x_{i}-\mu_x)(y_{i}-\mu_y)-Cov(X,Y)\right]}{Var(X)-\zeta\left[(x_{i}-\mu_x)^2-Var(X)\right]}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can now use the Taylor series:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\beta_{(i)}(\zeta)\approx \beta_{(i)}(0)+\zeta\left[\frac{\partial \beta_{(i)}(\zeta)}{\partial \zeta}\right]_{\zeta=0}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Simplifying this gives:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\beta_{(i)}(\zeta)\approx \beta-\zeta\left[\frac{(x_{i}-\mu_x)(y_{i}-\mu_y)}{Var(X)}-\beta\frac{(x_{i}-\mu_x)^2}{Var(X)}\right]$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And plugging in the values of the statistics $\mu_y$, $\mu_x$, $var(X)$, and $\zeta=\frac{1}{n-1}$ we get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\beta_{(i)}\approx \beta-\frac{x_{i}-\overline{x}}{n-1}\left[\frac{y_{i}-\overline{y}}{\frac{1}{n}\sum_{j=1}^{n}(x_{j}-\overline{x})^2}-\beta\frac{x_{i}-\overline{x}}{\frac{1}{n}\sum_{j=1}^{n}(x_{j}-\overline{x})^2}\right]&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And you can see how the effect of removing a single observation can be approximated without having to re-fit the model.  You can also see how an x equal to the average has &lt;em&gt;no influence on the slope of the line&lt;/em&gt;.  Think about this and you will see how it makes sense.  You can also write this more succinctly in terms of the standardised values $\tilde{x}=\frac{x-\overline{x}}{s_{x}}$ (similarly for y):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\beta_{(i)}\approx \beta-\frac{\tilde{x_{i}}}{n-1}\left[\tilde{y_{i}}\frac{s_y}{s_x}-\tilde{x_{i}}\beta\right]&#xD;&#xA;$$&lt;/p&gt;&#xA;" OwnerUserId="2392" LastEditorUserId="2392" LastEditorDisplayName="" LastEditDate="2011-03-16T11:35:25.490" LastActivityDate="2011-03-16T11:35:25.490" CommentCount="2" />
  <row Id="8355" PostTypeId="2" ParentId="8351" CreationDate="2011-03-16T11:46:52.670" Score="3" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://bayes.wustl.edu/glb/book.pdf&quot; rel=&quot;nofollow&quot;&gt;Larry Bretthorst's extended phd&lt;/a&gt; will greatly help you I think.  You should take the discrete fourier transform of the data.  This will give you a look at your series in the frequency domain.  Trend is represented by low frequency.  Ultimate modeling book.  It's 200 pages, but well worth it - includes computer code to implement the methods&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-03-16T11:46:52.670" CommentCount="3" />
  <row Id="8356" PostTypeId="1" CreationDate="2011-03-16T12:17:32.300" Score="2" ViewCount="117" Body="&lt;p&gt;I have a log-log scale plot by gnuplot. With following ranges,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set xrange [1:1000]&#xA;set yrange [1:1000]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How do I draw slope 5/3 slope, ie, a slope that intersects y-axis at 5 and x-axis at 3?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks!&lt;/p&gt;&#xA;" OwnerUserId="3172" LastActivityDate="2011-04-15T13:40:26.583" Title="How to draw a slope on a log-log scale in gnuplot" Tags="&lt;gnuplot&gt;" AnswerCount="0" CommentCount="1" />
  <row Id="8357" PostTypeId="2" ParentId="8334" CreationDate="2011-03-16T12:27:38.923" Score="4" ViewCount="" Body="&lt;p&gt;Your question is most interesting to me and it's solution has been my primary research for a number of years.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are a number of ways that &quot;a structural break&quot; may occur. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If there is a change in the Intercept or a change in Trend in &quot;the latter portion of the time series&quot; then one would be better suited to perform Intervention Detection (N.B. this is the empirical identification of the significant impact of an unspecified Deterministic Variable such as a Level Shift or a Change in Trend or the onset of a Seasonal Pulse ). Intervention Detection then is a pre-cursor to Intervention Modelling where a suggested variable is included in the model. You can find information on the web by googling &quot;AUTOMATIC INTERVENTION DETECTION&quot; . Some authors use the term &quot;OUTLIER DETECTION&quot; but like a lot of statistical language this can be confusing/imprecise . Detected Interventions can be any of the following (detecting a significant change in the mean of the residuals );&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;a 1 period change in Level ( i.e. a Pulse )&lt;/li&gt;&#xA;&lt;li&gt;a multi-period contiguous change in Level ( i.e. a change in Intercept )&lt;/li&gt;&#xA;&lt;li&gt;a systematic Pulse ( i.e. a Seasonal Pulse )&lt;/li&gt;&#xA;&lt;li&gt;a trend change  (i.e. 1,2,3,4,5,7,9,11,13,15 ..... )&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;These procedures are easily programmed IN R/SAS/Matlab and routinely available in a number of commercially available time series packages however there are many pitfalls that you need to be wary of such as whether to detect the stochastic structure first or do Intervention detection on the original series.  This is like the chicken and egg problem. Early work in this area was limited to type 1's and as such will probably be insufficient for your needs .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If no such phenomenon is detected then one might consider the CHOW TEST which normally requires the user to pre-specify the point of hypothesized change. I have been researching and implementing procedures to DETECT the point of change by evaluating alternative hypothetical points in time to determine the most likely break point.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In closing one might also be sensitive to the possibility that there might have been a structural change in the error variance thus that might mask the CHOW TEST leading to a false acceptance of the null hypothesis of no significant break points in parameters.&lt;/p&gt;&#xA;" OwnerUserId="3382" LastEditorUserId="3382" LastEditorDisplayName="" LastEditDate="2011-03-16T13:50:24.793" LastActivityDate="2011-03-16T13:50:24.793" CommentCount="1" />
  <row Id="8358" PostTypeId="1" AcceptedAnswerId="8543" CreationDate="2011-03-16T12:31:39.013" Score="2" ViewCount="165" Body="&lt;p&gt;I am a beginner in statistics with just basic knowledge. I have these data: cases, deaths and CFR (Case Fatality Rate-deaths per 100 cases) of a disease for 17 years (1994-2010) from 2 neighbouring states where people can walk across the states freely. This is a population based cohort study.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data are available from 1994. The treatment protocol  was started in 1996. State 1 and state 2 implemented the same treatment but state 2 could not implement the treatment perfectly due to local administrative problems. The death rate fell in 1 state and continued to be high in the other state. Because it would be unethical to subject patients to do a case-control study, I want to analyse the following available data to see if there is a significant difference in the death rates and risk ratios of these 2 states because of poor implementation of the treatment guidelines in state 2 from 1996 to 2010.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;St. refers to State&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Year   St.1 Cases  St.1 Deaths St.1 CFR St.2 Cases St.2 Deaths  State2 CFR  Risk Ratio&#xA;1994    1836        383         20.86     583        121         20.75        0.99&#xA;1995    1246        257         20.63    1126        227         20.16        0.98&#xA;1996    1450        263         18.14     896        179         19.98        1.10&#xA;1997    2953        407         13.78     351         76         21.65        1.57&#xA;1998    1161        149         12.83     1061       195         18.55        1.43&#xA;1999    2924        434         14.84     1371       275         20.06        1.35&#xA;2000    1729        169          9.77     1170       253         21.62        2.21&#xA;2001   1888      275         14.57     1005      199          19.80        1.36&#xA;2002    919     178          19.37      604      133         22.02         1.14&#xA;2003    865        142         16.42      1124      237         21.09         1.28&#xA;2004    1543       131          8.49      1030      228         22.14         2.61&#xA;2005    2887       336         11.64      6061     1500         24.75         2.13&#xA;2006    1484       108         7.28       2320     528          22.76         3.13&#xA;2007    1592       75          4.71       3024     645          21.33         4.53&#xA;2008    1920       53          2.76       3012     537          17.83         6.46&#xA;2009    1477       40          2.71       3073     556          18.09         6.68&#xA;2010    1534      26           1.69       3540     494          13.95         8.23&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Kindly advise me what is the best way to go.&lt;br&gt;&#xA;I have basic knowledge of using SPSS V19 and Comprehensive Meta-analysis V2.&lt;/p&gt;&#xA;" OwnerUserId="2956" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-16T13:00:03.317" LastActivityDate="2011-03-20T20:31:57.940" Title="How to assess effect of intervention in one state versus another using annual case fatality rate?" Tags="&lt;time-series&gt;&lt;statistical-significance&gt;&lt;spss&gt;&lt;relative-risk&gt;" AnswerCount="2" CommentCount="7" />
  <row Id="8359" PostTypeId="2" ParentId="8351" CreationDate="2011-03-16T12:44:21.670" Score="4" ViewCount="" Body="&lt;p&gt;The answer to your first question is no. If the null hypothesis of unit root is rejected, the alternative in its most general form is stationary series with &lt;a href=&quot;http://en.wikipedia.org/wiki/Augmented_Dickey%E2%80%93Fuller_test&quot; rel=&quot;nofollow&quot;&gt;time trend&lt;/a&gt;. Here is the example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; rr &amp;lt;- 1+0.01*(1:100)+rnorm(100)&#xA;&amp;gt; plot(rr)&#xA;&amp;gt; adf.test(rr)&#xA;&#xA;    Augmented Dickey-Fuller Test&#xA;&#xA;data:  rr &#xA;Dickey-Fuller = -4.1521, Lag order = 4, p-value = 0.01&#xA;alternative hypothesis: stationary &#xA;&#xA;Message d'avis :&#xA;In adf.test(rr) : p-value smaller than printed p-value&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So your findings are consistent with ADF test: there is no unit root, but there is a time trend.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-03-16T12:44:21.670" CommentCount="6" />
  <row Id="8361" PostTypeId="5" CreationDate="2011-03-16T14:34:23.620" Score="0" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://had.co.nz/ggplot2/&quot; rel=&quot;nofollow&quot;&gt;ggplot2&lt;/a&gt; is a plotting system for &lt;code&gt;R&lt;/code&gt; based on Leland Wilkinson's &quot;Grammar of graphics.&quot;   Building on the base and lattice graphics in &lt;code&gt;R&lt;/code&gt;, it automates some details (like drawing legends) and eases the production of complex multi-layered graphics.&lt;/p&gt;&#xA;" OwnerUserId="-1" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-16T16:24:02.493" LastActivityDate="2011-03-16T16:24:02.493" />
  <row Id="8363" PostTypeId="2" ParentId="8342" CreationDate="2011-03-16T14:47:55.757" Score="1" ViewCount="" Body="&lt;p&gt;Initially I suggested to try one of the constrained ordination techniques from the &lt;code&gt;vegan&lt;/code&gt; package, but on a second thought I doubt that this would be useful, as you actually have 2 contingency tables. I hope that the second part of &lt;a href=&quot;http://is.gd/CSS9QV&quot; rel=&quot;nofollow&quot;&gt;this example&lt;/a&gt; [PDF: R Demonstration – Categorical Analysis] will be helpful.&lt;/p&gt;&#xA;" OwnerUserId="3467" LastEditorUserId="3467" LastEditorDisplayName="" LastEditDate="2011-03-17T08:06:04.890" LastActivityDate="2011-03-17T08:06:04.890" CommentCount="2" />
  <row Id="8364" PostTypeId="2" ParentId="8347" CreationDate="2011-03-16T14:54:27.780" Score="9" ViewCount="" Body="&lt;p&gt;A quick response to the bulleted content:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Power / Type 1 error in a Bayesian analysis vs. a frequentist analysis&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Asking about Type 1 and power (i.e. one minus the probability of Type 2 error) implies that you can put your inference problem into a repeated sampling framework.  Can you?  If you can't then there isn't much choice but to move away from frequentist inference tools.  If you can, and if the behavior of your estimator over many such samples is of relevance, and if you are not particularly interested in making probability statements about particular events, then I there's no strong reason to move.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The argument here is not that such situations never arise - certainly they do - but that they typically don't arise in the fields where the methods are applied.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) The trade-off in complexity of the analysis (Bayesian seems more complicated) vs. the benefits gained. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is important to ask where the complexity goes.  In frequentist procedures the &lt;em&gt;implementation&lt;/em&gt; may be very simple, e.g. minimize the sum of squares, but the &lt;em&gt;principles&lt;/em&gt; may be arbitrarily complex, typically revolving around which estimator(s) to choose, how to find the right test(s), what to think when they disagree.  For an example. see the still lively discussion, picked up in this forum, of different confidence intervals for a proportion!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In Bayesian procedures the &lt;em&gt;implementation&lt;/em&gt; can be arbitrarily complex even in models that look like they 'ought' to be simple, usually because of difficult integrals but the &lt;em&gt;principles&lt;/em&gt; are extremely simple.  It rather depends where you'd like the messiness to be.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) Traditional statistical analyses are straightforward, with well-established guidelines for drawing conclusions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Personally I can no longer remember, but certainly my students &lt;em&gt;never&lt;/em&gt; found these straightforward, mostly due to the principle proliferation described above.  But the question is not really whether a procedure is straightforward, but whether is closer to being right given the structure of the problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, I strongly disagree that there are &quot;well-established guidelines for drawing conclusions&quot; in either paradigm.  And I think that's a &lt;em&gt;good&lt;/em&gt; thing.  Sure, &quot;find p&amp;lt;.05&quot; is a clear guideline, but for what model, with what corrections, etc.? And what do I do when my tests disagree?  Scientific or engineering judgement is needed here, as it is elsewhere.&lt;/p&gt;&#xA;" OwnerUserId="1739" LastActivityDate="2011-03-16T14:54:27.780" CommentCount="12" />
  <row Id="8366" PostTypeId="2" ParentId="8352" CreationDate="2011-03-16T15:13:35.767" Score="2" ViewCount="" Body="&lt;p&gt;I have not heard of nested ANOVA before.  However, from the link to John McDonald's website, it appears that nested ANOVA is used &lt;em&gt;specifically&lt;/em&gt; for the purpose of testing multiple hypotheses: &quot;A nested anova has one null hypothesis for each level. In a two-level nested anova, one null hypothesis would be that the subgroups within each group have the same means; the second null hypothesis would be that the groups have the same means.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, in your case, you are only interested in the first hypothesis.  The test statistic for this hypothesis is MSgroup/MSresidual.  To use any other statistic would be to test a different hypothesis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, if you randomized your subgroups properly in your experimental design, there is no need to worry about significant subgroup differences.&lt;/p&gt;&#xA;" OwnerUserId="3567" LastActivityDate="2011-03-16T15:13:35.767" />
  <row Id="8367" PostTypeId="2" ParentId="8303" CreationDate="2011-03-16T16:04:09.750" Score="1" ViewCount="" Body="&lt;p&gt;One idea would be to use a random forest and then use the variable importance measures it outputs to choose your best 8 variables.  Another idea would be to use the &quot;boruta&quot; package to repeat this process a few hundred times to find the 8 variables that are consistently most important to the model.&lt;/p&gt;&#xA;" OwnerUserId="2817" LastActivityDate="2011-03-16T16:04:09.750" CommentCount="8" />
  <row Id="8369" PostTypeId="2" ParentId="8342" CreationDate="2011-03-16T16:38:09.173" Score="1" ViewCount="" Body="&lt;p&gt;Logistic regression seems appropriate for your problem.  The variable you are trying to predict is the probability that an observation (which is either species A or species B) is species A.  The covariates are $host$, $rain$ and optionally $host*rain$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The R command would be:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;glm(formula = species ~ host + rain, family = binomial(logit), weights=counts)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;and you will be interested in the $p$-values of the slopes.  Keep in mind that you are testing multiple hypotheses, though.&lt;/p&gt;&#xA;" OwnerUserId="3567" LastEditorUserId="3567" LastEditorDisplayName="" LastEditDate="2011-03-16T18:38:27.713" LastActivityDate="2011-03-16T18:38:27.713" CommentCount="1" />
  <row Id="8370" PostTypeId="1" AcceptedAnswerId="8372" CreationDate="2011-03-16T17:04:50.353" Score="4" ViewCount="175" Body="&lt;p&gt;I have a matrix of positive real numbers between 0 and 1; the rows represent genes and columns represent samples. Number of rows is greater than the number of columns by a magnitude of $10^4$. I am wondering how to visualize this in &lt;code&gt;R&lt;/code&gt;. I know heatmap is one of the ways to do this, but are there other ideas. Here are a few points which I want to emphasize in the visualization:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Data:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The rows and columns have no order to them (as you may have already realized); specifically rows and columns are exchangeable.&lt;/li&gt;&#xA;&lt;li&gt;The entries of the matrix are positive real numbers between 0 and 1.&lt;/li&gt;&#xA;&lt;li&gt;A small fraction of the data (10% of the rows or genes, around 1000) are actually &quot;interesting&quot;.&lt;/li&gt;&#xA;&lt;li&gt;The matrix represents the estimated probability of genes being more active in a sample.    &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Aim:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;I want show: which genes are more active and in which sample. The matrix has a lot of rows in which the probabilities are very similar across columns.&lt;/li&gt;&#xA;&lt;li&gt;I am ok with ordering the rows (genes) to make the pattern clearer.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My thoughts:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At the moment I can determine active genes in a sample by choosing a cutoff (say $\ge 95\%$) and arrange the genes in such a way that first set of rows are active genes in sample 1, second set of rows are active genes in sample 2, ...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was also thinking about visualizing a subset of the data, may be by sampling rows. But I did not have any success. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know these ideas may not be very elegant but rearranges my data in a way which makes the pattern more recognizable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know similar questions have been asked before, but I thought my query was a bit more specific, so hopefully I can get better inputs from the members of this forum. &lt;/p&gt;&#xA;" OwnerUserId="1307" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-03-16T17:44:54.310" LastActivityDate="2011-03-16T20:14:48.487" Title="How to visualize/summarize a matrix with number of rows $\gg$ number of columns?" Tags="&lt;r&gt;&lt;data-visualization&gt;&lt;genetics&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="2" />
  <row Id="8371" PostTypeId="1" AcceptedAnswerId="9657" CreationDate="2011-03-16T17:38:34.997" Score="5" ViewCount="117" Body="&lt;p&gt;French polling institutes are currently facing a major crisis after they recently published what can only be called &lt;a href=&quot;http://www.bbc.co.uk/news/world-europe-12660329&quot; rel=&quot;nofollow&quot;&gt;the most ridiculous poll so far&lt;/a&gt; on the 2012 presidential election horse race. The French Senate is now considering to legislate on the issue by forcing polling institutes to publish, among other things, the confidence intervals for their results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, some pollsters are &lt;a href=&quot;http://www.lemonde.fr/idees/article/2011/03/08/il-est-inutile-de-legiferer-sur-la-question-des-sondages_1490046_3232.html&quot; rel=&quot;nofollow&quot;&gt;opposing&lt;/a&gt; the measure, claiming that confidence intervals do not apply to &lt;a href=&quot;http://en.wikipedia.org/wiki/Quota_sampling&quot; rel=&quot;nofollow&quot;&gt;quota sampling&lt;/a&gt;, which is the method used by polling institutes in France. Since quota sampling is formally non-probabilistic sampling, there is some truth to the claim. But since quota sampling is fundamentally &lt;a href=&quot;http://en.wikipedia.org/wiki/Stratified_sampling&quot; rel=&quot;nofollow&quot;&gt;stratified sampling&lt;/a&gt;, confidence intervals should apply, right?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;May I ask for experiences about this issue outside of France, in countries where pollsters also use quota sampling?&lt;/p&gt;&#xA;" OwnerUserId="3582" LastActivityDate="2011-04-18T16:21:07.000" Title="Do confidence intervals apply to quota sampling?" Tags="&lt;sampling&gt;&lt;polling&gt;" AnswerCount="2" CommentCount="6" FavoriteCount="1" />
  <row Id="8372" PostTypeId="2" ParentId="8370" CreationDate="2011-03-16T18:29:31.027" Score="4" ViewCount="" Body="&lt;p&gt;Find one-dimensional &lt;a href=&quot;http://en.wikipedia.org/wiki/Multidimensional_scaling&quot; rel=&quot;nofollow&quot;&gt;multidimensional scaling solutions&lt;/a&gt; for the rows and for the columns (separately), using whatever similarity measures you like (such as correlation).  Sort the rows and columns according to their MDS positions.  This will bring similar genes together and similar samples together.  The whole thing can then easily be visualized as an array plot (&lt;em&gt;e.g.&lt;/em&gt;, normalize the values to the range 0..255 and display it as a grayscale image).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A 50 by 6 array of standard normal variates was processed in this way (using Euclidean distances as the proximity measures):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/HijSa.png&quot; alt=&quot;Array plot&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There's not much to see--after all, these data are iid--but look at the correlation matrices of the reordered columns and rows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/9swI8.png&quot; alt=&quot;Column correlations&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/XrPvr.png&quot; alt=&quot;Row correlations&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(red = positive, blue = negative).  The concentrations of positive correlations along the diagonals and negative correlation off the diagonals demonstrate the method has worked as advertised.  (With the original data, the correlation matrices are random, too, causing the red and blue cells to be more evenly interspersed throughout.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my experience, when there are even subtle nonzero correlations among rows and/or columns, this method does an excellent job of bringing them out in the original array plot (grayscale, above) and providing a visual display of clustering along both dimensions.  Larger blocks along the diagonals of the corresponding correlation matrix plots help identify strongly clustered groups of rows or columns.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-03-16T18:29:31.027" CommentCount="8" />
  <row Id="8373" PostTypeId="1" CreationDate="2011-03-16T18:40:35.840" Score="4" ViewCount="153" Body="&lt;p&gt;I am seeing a regression model which is regressing Year-on-Year stock index returns on lagged (12 months) Year-on-Year returns of the same stock index, credit spread (difference between monthly mean of risk-free bonds and corporate bond yields), YoY Inflation rate and YoY index of industrial production.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It looks thus (though you'd subsitute the data specific to India in this case):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;SP500YOY(T) = a + b1*SP500YOY(T-12) + b2*CREDITSPREAD(T) +    &#xA;b4*INDUSTRIALPRODUCTION(T+2) + b3*INFLATION(T+2) + b4*INFLATIONASYMM(T+2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;SP500YOY is the year-on-year returns for SP500 index To compute this, monthly average of SP500 values is computed and then converted to year-on-year returns for each month (i.e. Jan'10-Jan'11, Feb'10-Feb'11, Mar'10-Mar'11, . . ). On the explanatory variables side, a 12-month lagged value of the SP500YOY is used along with the CREDITSPREAD at time T and INFLATION and INDUSTRIALPRODUCTION two period AHEAD. The INFLATIONASYMM is a dummy for whether the Inflation is above a threshold value of 5.0%. The index in the parenthesis shows the time index for each variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is estimated by standard OLS linear regression. To use this model for forecasting the 1,2 and 3-months ahead YOY returns of SP500, one has to generate 3,4 and 5-month ahead forecasts for Inflation and the Index of Industrial Production. These forecasts are done after fitting an ARIMA model to each of the two individually. The CreditSpread forecasts for 1,2 and 3 month ahead are just thrown in as mental estimates.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd like to know whether this OLS linear regression is correct/incorrect, efficient/inefficient or generally valid statistical practice.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first problem I see is that of using overlapping data. i.e. the daily values of the stock index are averaged each month, and then used to compute yearly returns which are rolled over monthly. This should make the error term autocorrelated. I would think that one would have to use some 'correction' on the lines of one of the following:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;White's heteroscedasticity consistent covariance estimator&lt;/li&gt;&#xA;&lt;li&gt;Newey &amp;amp; West heteroscedasticity and autocorrelation consistent (HAC) estimator&lt;/li&gt;&#xA;&lt;li&gt;heteroscedasticity-consistent version of Hansen &amp;amp; Hodrick&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Does it really make sense to apply standard OLS linear regression (without any corrections) to such overlapping data, and more so, use 3-period ahead ARIMA forecasts for explanatory variables to use in the original OLS linear regression for forecasting SP500YOY? I have not seen such a form before, and hence cannot really judge it, without the exception of correcting for the use of overlapping observations.&lt;/p&gt;&#xA;" OwnerUserId="1349" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-03T17:20:37.413" LastActivityDate="2011-05-03T17:20:37.413" Title="Time series regression with overlapping data" Tags="&lt;regression&gt;&lt;time-series&gt;&lt;autocorrelation&gt;" AnswerCount="1" CommentCount="3" FavoriteCount="3" />
  <row Id="8374" PostTypeId="1" CreationDate="2011-03-16T18:52:37.840" Score="6" ViewCount="68" Body="&lt;p&gt;I am trying to correlate age (6-90 yrs) with loudness of voice (in dB). However, my data do not contain any data points in the range of 20-50 yrs. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What correlation measure is most appropriate with such a considerable gap, and why? I have been using Kendall Tau so far.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that we are not dealing with bimodally distributed data here, but with a substantial missing data gap in the age range.&lt;/p&gt;&#xA;" OwnerUserId="3742" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-16T22:44:11.883" LastActivityDate="2011-03-16T22:59:14.990" Title="Which correlation measure should be used with a large gap (missing data)?" Tags="&lt;distributions&gt;&lt;correlation&gt;&lt;missing-data&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="8375" PostTypeId="1" AcceptedAnswerId="8806" CreationDate="2011-03-16T19:58:45.120" Score="5" ViewCount="221" Body="&lt;p&gt;All,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am looking to build a predictive model for predicting churn and looking to use a discrete time survival model fitted to a person-period training dataset (one row for each customer and discrete period they were at risk, with an indicator for event – equaling 1 if the churn happened in that period, else 0). &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;I am fitting the model using ordinary&#xA;logistic regression using the&#xA;technique from &lt;a href=&quot;http://www.ats.ucla.edu/stat/sas/examples/alda/&quot; rel=&quot;nofollow&quot;&gt;Singer and&#xA;Willet.&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;The churn of a customer can happen&#xA;anywhere during a month, but it is&#xA;only at the end of the month that we&#xA;know about it (i.e. sometime during&#xA;that month they left). 24 months is&#xA;being used for training.&lt;/li&gt;&#xA;&lt;li&gt;The time variable being used is the&#xA;origin time of the sample - all&#xA;customers active as of 12/31/2008 - they all receive t=0 as of Jan 2009 (not the classical way to do it, but I believe the way when building a predictive model versus a traditional statistical one). A&#xA;covariate used is the tenure of the&#xA;customer at that point in time.&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;There are a series of covariates that&#xA;were constructed – some that do not&#xA;change across the rows of the dataset&#xA;(for a given customer) and some that&#xA;do.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;These time variant covariates are the&#xA;issue and what is causing to me&#xA;question a survival model for churn&#xA;prediction (compared to a regular&#xA;classifier that predicts churn in the&#xA;next x months based on current&#xA;snapshot data). The time-invariant ones describe activity the month prior and are expected to be important triggers.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The implementation of this predictive model, at least based on my current thinking, is to score the customer base at the end of each month, calculating the probability / risk of churn sometime during the next month. Then again for the next 1,2 or 3 months. Then for the next 1,2,3,4,5,6 months. For the 3 and 6 month churn probability, I would be using the estimated survival curve.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;b&gt;The problem:&lt;/b&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When it comes to thinking about scoring, how can I incorporate time-varying predictors? It seems like I can only score with time-invariant  predictors or to include those that are time invariant, you have to make them time invariant – set to the value “right now”. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone have experience or thoughts on this use of a survival model?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;b&gt; Update based on @JVM comment:&lt;/b&gt;&lt;br/&gt;&lt;br/&gt;&#xA;The issue is not with estimating the model, interpreting coefficients, plotting the hazard/survival plots of interesting covariate values using the training data etc. The issue is in using the model to forecast risk for a given customer. Say at the end of this month, I want to score everyone who is still an active customer with this model. I want to forecast that risk estimate out x periods (risk of closing the account at the end of next month. risk of closing the account at the end of two months from now, etc.). If there are time varying covariates, their values are unknown out any future periods, so how to utilize the model?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;b&gt; Final Update:&lt;/b&gt;&lt;br/&gt;&lt;br/&gt;&#xA;A person period data set will have an entry for each person and each time period they are at risk. Say there are J time periods (maybe J =1...24 for 24 months) Lets say I construct a discrete time survival model, where for simplicity we just treat time T as linear and have two covariates X and Z where X is time-invariant, meaning it is constant in every period for the ith person and Z is time varying, meaning that each record for the ith person can take on a different value. For example, X may be the customers gender and Z might be how much they were worth to the company in the prior month. The model for the logit of the hazard for the ith person in the jth time period is :&lt;br/&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$logit(h(t_{ij}))=\alpha_{0}+\alpha_{1}T_{j}+\beta_{1}X_{i}+\beta_{2}Z_{ij}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the issue is, when using time varying covariates, and forecasting (into the yet unseen future) with new data, the $Z_{j}$ are unknown.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The only solutions I can think are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Don't use time varying covariates like Z. This would greatly weaken the model to predict the event of churning though since, for example, seeing a decrease in Z would tell us the customer is disengaging and perhaps preparing to leave.&lt;/li&gt;&#xA;&lt;li&gt;Use time varying covariates but lag them (like Z was above) which allows us to forecast out however many periods we have lagged the variable (again, thinking of the model scoring new current data).&lt;/li&gt;&#xA;&lt;li&gt;Use time varying covariates but keep them as constants in the forecast (so the model was fitted for varying data but for prediction we leave them constant and simulate how changes in these values, if later actually observed, will impact risk of churning.&lt;/li&gt;&#xA;&lt;li&gt;Use time varying covariates but impute their future values based on a forecast from known data. E.g. Forecast the $Z_{j}$ for each customer.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="2040" LastEditorUserId="2040" LastEditorDisplayName="" LastEditDate="2011-03-27T16:02:32.887" LastActivityDate="2011-03-27T16:02:32.887" Title="Survival Model for Predicting Churn - Time-varying predictors?" Tags="&lt;survival&gt;&lt;predictive-models&gt;&lt;churn&gt;" AnswerCount="2" CommentCount="4" />
  <row Id="8376" PostTypeId="2" ParentId="8370" CreationDate="2011-03-16T20:14:48.487" Score="4" ViewCount="" Body="&lt;p&gt;I was about to suggest something along @whuber's answer (I used this reordering technique but in a context of feature selection, so I was mainly concerned with the &quot;variables view&quot;). So let me suggest two other directions (but the first one is close to the already proposed one).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as &lt;strong&gt;heatmaps&lt;/strong&gt; are concerned, you can display them after a slight rearrangement of rows (samples) and/or columns (genes) through hierarchical clustering (yet another aggregation method based on a (dis)similarity measure). There're a lot of R packages that can do this, for example the &lt;code&gt;cim()&lt;/code&gt; function in &lt;a href=&quot;http://cran.r-project.org/web/packages/mixOmics/index.html&quot; rel=&quot;nofollow&quot;&gt;mixOmics&lt;/a&gt;. Another package that might be of interest is &lt;a href=&quot;http://www.bioinf.ucd.ie/people/aedin/R/&quot; rel=&quot;nofollow&quot;&gt;MADE4&lt;/a&gt;; it relies on the very good &lt;a href=&quot;http://cran.r-project.org/web/packages/ade4/index.html&quot; rel=&quot;nofollow&quot;&gt;ade4&lt;/a&gt; package for multivariate data analysis and visualization.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are concerned with the rather large number of variables, you might also consider some reduction method for &lt;strong&gt;genes clustering&lt;/strong&gt;. One that I've heard about is &lt;em&gt;PCA-gene shaving&lt;/em&gt; (Hastie et al., 2000), that is largely described in Izenman (2008). In essence, this is a two-stage iterative procedure where (a) for &lt;em&gt;feature selection&lt;/em&gt;, we single out genes whose correlation with the first principal component is below a distribution-based threshold (say, the 10% of genes having the lowest correlation at each step), and (b) for &lt;em&gt;clustering&lt;/em&gt;, we seek to maximize an $R^2$ measure (between-cluster/within-cluster variances) for $j$ successive clusters of size $k_j$, where the optimal $k_j$ is obtained by a permutation technique and the use of the &lt;em&gt;gap statistic&lt;/em&gt; (after effects of the former cluster has been removed by residualization). More detailed informations can be found in the paper referenced below, but the general idea is to cluster genes into &lt;em&gt;small and potentially overlapping subsets of correlated genes that vary as much as possible across individuals&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;References&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Hastie, T., Tibshirani, R., Eisen, M.B., Alzadeh, A., Levy, R., Staudt, L., Chan, W.C., Botstein, D., and Brown, P.O. (2000). &lt;a href=&quot;http://genomebiology.com/2000/1/2/research/0003/&quot; rel=&quot;nofollow&quot;&gt;'Gene shaving' as a method for identifying distinct sets of genes with similar expression patterns&lt;/a&gt;. &lt;em&gt;Genome Biology&lt;/em&gt;, 1(2).&lt;/li&gt;&#xA;&lt;li&gt;Izenman, A.J. (2008). &lt;em&gt;Modern Multivariate Statistical Techniques&lt;/em&gt;. Springer.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-03-16T20:14:48.487" CommentCount="2" />
  <row Id="8377" PostTypeId="1" AcceptedAnswerId="8383" CreationDate="2011-03-16T20:55:08.437" Score="5" ViewCount="272" Body="&lt;p&gt;I am a first year psychology student. I am doing some research work with a prof, unfortunately the material that I need to use right now is covered only in my second year. But I need to already know it now. So I am burning through any resources I can find to quickly come up to speed. I need help to understand this particular situation here. Involves SAS, Regression Analysis. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I ran a regression in SAS ( proc reg ) using two variables say a and b. I got this. I understand this as saying that both these variables (a&amp;amp;b) do not significantly predict my target variable. Here is the SAS output. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;                                     Analysis of Variance&#xA;&#xA;                                            Sum of           Mean&#xA;        Source                   DF        Squares         Square    F Value    Pr &amp;gt; F&#xA;&#xA;        Model                     2        3.32392        1.66196       1.00    0.3774&#xA;        Error                    46       76.80649        1.66971&#xA;        Corrected Total          48       80.13041&#xA;&#xA;&#xA;                     Root MSE              1.29217    R-Square     0.0415&#xA;                     Dependent Mean       -0.23698    Adj R-Sq    -0.0002&#xA;                     Coeff Var          -545.26074&#xA;&#xA;&#xA;                                     Parameter Estimates&#xA;&#xA;                            Parameter       Standard                           Standardized&#xA;   Variable         DF       Estimate          Error    t Value    Pr &amp;gt; |t|        Estimate&#xA;&#xA;   Intercept         1       -0.25713        0.18515      -1.39      0.1716               0&#xA;   a                 1       -0.35394        0.28797      -1.23      0.2253        -0.19510&#xA;   b                 1       -0.04706        0.39586      -0.12      0.9059        -0.01887&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now I tried to include the interaction of a and b into the picture. Lets call it aXb, now the out put indicates that a and aXb significantly predict my target variable.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;                                     Analysis of Variance&#xA;&#xA;                                            Sum of           Mean&#xA;        Source                   DF        Squares         Square    F Value    Pr &amp;gt; F&#xA;&#xA;        Model                     3       16.64439        5.54813       3.93    0.0142&#xA;        Error                    45       63.48602        1.41080&#xA;        Corrected Total          48       80.13041&#xA;&#xA;&#xA;                     Root MSE              1.18777    R-Square     0.2077&#xA;                     Dependent Mean       -0.23698    Adj R-Sq     0.1549&#xA;                     Coeff Var          -501.20683&#xA;&#xA;&#xA;                                     Parameter Estimates&#xA;&#xA;                            Parameter       Standard                           Standardized&#xA;   Variable         DF       Estimate          Error    t Value    Pr &amp;gt; |t|        Estimate&#xA;&#xA;   Intercept         1       -0.06807        0.18098      -0.38      0.7086               0&#xA;   a                 1        3.01517        1.12795       2.67      0.0104         1.66201&#xA;   b                 1       -0.00994        0.36407      -0.03      0.9783        -0.00399&#xA;   aXb               1       -1.13782        0.37029      -3.07      0.0036        -1.90743&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Here are my questions:&lt;/strong&gt; I am not sure what to make out of this situation. Taken together what does this indicate to me? Also while you are answering the question, could you supplement it with some resources, goog keywords etc for me to learn more surrounding these topics. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you so much for your help. &lt;/p&gt;&#xA;" OwnerUserId="3745" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-17T16:58:37.463" LastActivityDate="2011-03-17T16:58:37.463" Title="Understanding multiple regression output" Tags="&lt;regression&gt;&lt;anova&gt;&lt;sas&gt;" AnswerCount="4" CommentCount="6" />
  <row Id="8378" PostTypeId="2" ParentId="8342" CreationDate="2011-03-16T21:16:26.233" Score="4" ViewCount="" Body="&lt;p&gt;There are two ways to interpret your first question, which are reflected in the two ways you asked it: “Are species associated with host plants?” and, “Are species independent to host plants, given the effect of rain?” &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first interpretation corresponds to a model of &lt;em&gt;joint independence&lt;/em&gt;, which states that species and hosts are dependent, but jointly independent of whether it rained:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\quad p_{shr} = p_{sh} p_r$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $p_{shr}$ is the probability that an observation falls into the $(s,h,r)$ cell where $s$ indexes species, $h$ host type, and $r$ rain value, $p_{sh}$ is the marginal probability of the $(s,h,\cdot)$ cell where we collapse over the rain variable, and $p_r$ is the marginal probability of rain. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second interpretation corresponds to a model of &lt;em&gt;conditional independence&lt;/em&gt;, which states that species and hosts are independent given whether it rained:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\quad p_{sh|r} = p_{s|r}p_{h|r}$ or $p_{shr} = p_{sr}p_{hr} / p_r$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $p_{sh|r}$ is the conditional probability of the $(s,h,r)$ cell, given a value of $r$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can test these models in R (&lt;code&gt;loglin&lt;/code&gt; would work fine too but I’m more familiar with &lt;code&gt;glm&lt;/code&gt;):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;count &amp;lt;- c(12,15,10,13,11,12,12,7)&#xA;species &amp;lt;- rep(c(&quot;a&quot;, &quot;b&quot;), 4)&#xA;host &amp;lt;- rep(c(&quot;c&quot;,&quot;c&quot;, &quot;d&quot;, &quot;d&quot;), 2)&#xA;rain &amp;lt;- c(rep(0,4), rep(1,4))&#xA;my.table &amp;lt;- xtabs(count ~ host + species + rain)&#xA;my.data &amp;lt;- as.data.frame.table(my.table)&#xA;mod0 &amp;lt;- glm(Freq ~ species + host + rain, data=my.data, family=poisson())&#xA;mod1 &amp;lt;- glm(Freq ~ species * host + rain, data=my.data, family=poisson())&#xA;mod2 &amp;lt;- glm(Freq ~ (species + host) * rain, data=my.data, family=poisson())&#xA;anova(mod0, mod1, test=&quot;Chi&quot;) #Test of joint independence&#xA;anova(mod0, mod2, test=&quot;Chi&quot;) #Test of conditional independence&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Above, &lt;code&gt;mod1&lt;/code&gt; corresponds to joint independence and &lt;code&gt;mod2&lt;/code&gt; corresponds to conditional independence, whereas &lt;code&gt;mod0&lt;/code&gt; corresponds to a &lt;em&gt;mutual independence&lt;/em&gt; model $p_{shr} = p_s p_h p_r$. You can see the parameter estimates using &lt;code&gt;summary(mod2)&lt;/code&gt;, etc. As usual, you should check to see if model assumptions are met. In the data you provided, the null model actually fits adequately. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A different way of approaching your first question would be to perform Fischer’s exact test (&lt;code&gt;fisher.test(xtabs(count ~ host + species))&lt;/code&gt;) on the collapsed 2x2 table (first interpretation) or the Mantel-Haenszel test (&lt;code&gt;mantelhaen.test(xtabs(count ~ host + species + rain))&lt;/code&gt;) for 2-stratified 2x2 tables or to write a permutation test that respects the stratification (second interpretation). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To paraphrase your second question, &lt;em&gt;Does the relationship between species and host depend on whether it rained?&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mod3 &amp;lt;- glm(Freq ~ species*host*rain - species:host:rain, data=my.data, family=poisson())&#xA;mod4 &amp;lt;- glm(Freq ~ species*host*rain, data=my.data, family=poisson())&#xA;anova(mod3, mod4, test=”Chi”)&#xA;pchisq(deviance(mod3), df.residual(mod3), lower=F)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The full model &lt;code&gt;mod4&lt;/code&gt; is saturated, but you can test the effect in question by looking at the deviance of &lt;code&gt;mod3&lt;/code&gt; as I've done above.&lt;/p&gt;&#xA;" OwnerUserId="3432" LastEditorUserId="3432" LastEditorDisplayName="" LastEditDate="2011-03-16T21:21:45.497" LastActivityDate="2011-03-16T21:21:45.497" CommentCount="1" />
  <row Id="8379" PostTypeId="2" ParentId="8374" CreationDate="2011-03-16T22:15:34.437" Score="6" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Create a scatterplot&lt;/strong&gt; to check whether it makes any sense to suppose that a single correlation coefficient is an adequate description of the association between the variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, in these (simulated) data the correlation for ages 6-20 is 90%, for ages 50+ it's -70%, and overall it's 15%.  In such a situation reporting a single correlation coefficient would be as deceptive as reporting that the average number of legs among household pets is four when half of the pets are fish and the other half are spiders...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/4Wzfy.png&quot; alt=&quot;Scatterplot of loudness vs. age for 150 simulated people&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The choice of &lt;em&gt;how&lt;/em&gt; to express correlation is of secondary concern and rests on other aspects of the dataset.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-03-16T22:15:34.437" CommentCount="2" />
  <row Id="8380" PostTypeId="1" CreationDate="2011-03-16T22:15:53.453" Score="4" ViewCount="126" Body="&lt;p&gt;I want to build a linear model to predict a scalar output from a vector of noisy scalar variable measurements.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have two separate training data sets.  One has output data and corresponding exact variable measurements.  The other has exact variable measurements and corresponding noisy variable measurements.  The noisy measurements of some variables are noisier (higher error variance) than others, and the noisy measurements of some variables are biased.  I do &lt;em&gt;not&lt;/em&gt; have a single data set with output data and corresponding noisy variable measurements.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How should I build my linear model?  Should I use the exact variable measurements and ignore the fact that when the model is applied/used, noisy measurements of the variables will be input to the model?  Or is there some way I can make use of what I can figure out about the noisy measurements of each variable when building the model? Can R help me with this problem?&lt;/p&gt;&#xA;" OwnerUserId="3744" LastEditorUserId="3744" LastEditorDisplayName="" LastEditDate="2011-03-21T16:06:20.593" LastActivityDate="2011-03-25T16:48:20.410" Title="Building linear model from exact variable measurements for use with noisy variable measurements" Tags="&lt;r&gt;&lt;regression&gt;&lt;predictive-models&gt;&lt;errors-in-variables&gt;" AnswerCount="2" CommentCount="9" />
  <row Id="8381" PostTypeId="2" ParentId="652" CreationDate="2011-03-16T22:34:19.733" Score="0" ViewCount="" Body="&lt;p&gt;Agresti &amp;amp; Finlay's Statistical Methods for the Social Sciences is quite good, though I'd like to believe there is a good open source alternative.  Is it wrong to use an amazon affiliate link here?&lt;/p&gt;&#xA;" OwnerUserId="3748" LastActivityDate="2011-03-16T22:34:19.733" CommunityOwnedDate="2011-03-16T22:34:19.733" />
  <row Id="8382" PostTypeId="1" AcceptedAnswerId="8438" CreationDate="2011-03-16T22:52:36.403" Score="6" ViewCount="91" Body="&lt;p&gt;So I have data that has been quantized by an A-&gt;D converter. (continuous data has been turned into discrete data and the values range from 0 to the saturation value , which is 127 in this case).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This particular instrument I used to gather the data is quite noisy, let's say there is added gaussian noise to the real value. Luckily, when taking single measurements, I have enough time to take multiple measurements and average them to reduce the noise. Note that sampling rate is not an issue here since the thing that I'm taking measurements of is completely stable. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Obviously , taking the simple mean will produce a biased result because values cannot exceed 0 or 127 (for example, if you attempt to use a plain old averaging on something with a &quot;real&quot; value of 126, you will get an estimated value that is less than 126. This is because the added gaussian noise will not give you any value higher than 127 because of the truncation).  So how do I take the average so the result will give me an unbiased estimator of the real value? &lt;/p&gt;&#xA;" OwnerUserId="3749" LastEditorUserId="3749" LastEditorDisplayName="" LastEditDate="2011-03-18T18:13:08.663" LastActivityDate="2011-03-18T18:13:08.663" Title="How to average quantized and truncated data?" Tags="&lt;averages&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="8383" PostTypeId="2" ParentId="8377" CreationDate="2011-03-16T22:54:57.350" Score="5" ViewCount="" Body="&lt;p&gt;It seems like you need an introduction to regression.  People made book recommendations &lt;a href=&quot;http://stats.stackexchange.com/questions/652/best-books-for-an-introduction-to-statistical-data-analysis&quot;&gt;here&lt;/a&gt;.  Free book recommendations &lt;a href=&quot;http://stats.stackexchange.com/questions/170/free-statistical-textbooks&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's hard to make sure you're doing the analysis right when we don't know what the variables are or what the goal is.  But based on the output, I can tell you that your second regression specification looks better than your first.  I say that because you have two highly significant coefficients, and the adjusted R^2 value took a big jump.  Note though, although I consider these important clues, it is not true that models with more significant coefficients or higher adjusted R^2 are consistently better.  There are lots of other issues to consider.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your regression models are predicting Y, using a and b.  In your second model, the estimated regression equation is -0.06807 + (3.01517 * a) - (0.00994 * b) - (1.13782 *a*b)&#xA;In other words, plug in a and b, and you get the models prediction for Y.  I could say a lot more, but I'll leave you there and suggest you pick up a textbook.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I strongly recommend you try plotting your data.  Y with a on the x-axis, Y with b on the x-axis, and a by b as well.&lt;/p&gt;&#xA;" OwnerUserId="3748" LastEditorUserId="3748" LastEditorDisplayName="" LastEditDate="2011-03-17T00:22:58.507" LastActivityDate="2011-03-17T00:22:58.507" CommentCount="3" />
  <row Id="8384" PostTypeId="2" ParentId="8351" CreationDate="2011-03-16T22:55:24.480" Score="1" ViewCount="" Body="&lt;p&gt;The ADF test has weak power and, as Dmitrij Celov mentioned, you should probably also check the results of PP and KPSS tests. If you find that your results are on the margin of detecting a unit root, it's possible your series is fractionally integrated. I would also check ACF and PACF plots of the series, looking for slow decay patterns. Generally, if you find that ADF test and Phillips-Perron reject the null of a unit root, but that the KPSS and ACF/PACF plots demonstrate some statistically-significant persistence through several lags, this may be strong evidence for fractional integration.&lt;/p&gt;&#xA;" OwnerUserId="3265" LastActivityDate="2011-03-16T22:55:24.480" />
  <row Id="8386" PostTypeId="2" ParentId="290" CreationDate="2011-03-16T23:45:24.433" Score="1" ViewCount="" Body="&lt;p&gt;Alex Tabarrok posted a great list of resources here on &lt;a href=&quot;http://marginalrevolution.com/marginalrevolution/2011/01/stata-resources.html?utm_source=feedburner&amp;amp;utm_medium=feed&amp;amp;utm_campaign=Feed%3a+marginalrevolution/hCQh+%28Marginal+Revolution%29&quot; rel=&quot;nofollow&quot;&gt;marginalrevolution.com&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Gabriel Rossman shares a good introductory guide &lt;a href=&quot;http://gabrielr.bol.ucla.edu/stataprogramming.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="3748" LastEditorUserId="3748" LastEditorDisplayName="" LastEditDate="2011-03-17T00:16:14.973" LastActivityDate="2011-03-17T00:16:14.973" />
  <row Id="8387" PostTypeId="2" ParentId="103" CreationDate="2011-03-16T23:59:43.900" Score="2" ViewCount="" Body="&lt;p&gt;Andrew Gelman doesn't limit himself to visualization, but he comments on it frequently.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.stat.columbia.edu/~gelman/blog/&quot; rel=&quot;nofollow&quot;&gt;http://www.stat.columbia.edu/~gelman/blog/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="3748" LastActivityDate="2011-03-16T23:59:43.900" CommunityOwnedDate="2011-03-16T23:59:43.900" />
  <row Id="8388" PostTypeId="2" ParentId="363" CreationDate="2011-03-17T00:09:23.670" Score="1" ViewCount="" Body="&lt;p&gt;Andrew Gelman's interesting book recommendations are here:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://thebrowser.com/interviews/andrew-gelman-on-statistics&quot; rel=&quot;nofollow&quot;&gt;http://thebrowser.com/interviews/andrew-gelman-on-statistics&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="3748" LastActivityDate="2011-03-17T00:09:23.670" CommunityOwnedDate="2011-03-17T00:09:23.670" />
  <row Id="8389" PostTypeId="1" CreationDate="2011-03-17T00:30:01.570" Score="4" ViewCount="93" Body="&lt;p&gt;I need a closed-form expression for the probability that 2 values drawn at random from a normal distribution are separated by at most T, as a function of T and the variance of the distribution. I can formulate the problem as an integration, but can't solve it. I'd be happy with a well-motivated approximation. I'd be amenable to making some simplifying assumptions about the distribution, if that would help.&lt;/p&gt;&#xA;" OwnerUserId="3751" LastActivityDate="2011-03-17T02:07:17.637" Title="Probability that two values drawn at random from a normal distribution are separated by at most T" Tags="&lt;normal-distribution&gt;&lt;sample&gt;" AnswerCount="1" />
  <row Id="8390" PostTypeId="2" ParentId="7562" CreationDate="2011-03-17T00:40:10.997" Score="2" ViewCount="" Body="&lt;p&gt;Looking at the example for &lt;code&gt;postStratify&lt;/code&gt; in the &lt;a href=&quot;http://cran.r-project.org/web/packages/survey/survey.pdf&quot; rel=&quot;nofollow&quot;&gt;manual&lt;/a&gt;, you are correct: you seem to be required to give a &lt;code&gt;svydesign&lt;/code&gt; object (though you can if needed use &lt;code&gt;svrepdesign&lt;/code&gt; to specify it instead).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;code&gt;svydesign&lt;/code&gt; object must have &lt;code&gt;ids&lt;/code&gt;; all the others are optional, though you will almost certainly want &lt;code&gt;data&lt;/code&gt; to have something to work with, and you will probably want some of the others.  At this stage I would suggest you ignore all those appearing after &lt;code&gt;data&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;postStratify&lt;/code&gt; also needs &lt;code&gt;strata&lt;/code&gt;, the variable to post-stratify on: the example uses &lt;code&gt;apiclus1$stype&lt;/code&gt; which simply specifies the school type (E, M or H).  It also needs &lt;code&gt;population&lt;/code&gt; which you can either specify yourself or take from some other source: the example gives &lt;code&gt;data.frame(stype=c(&quot;E&quot;,&quot;H&quot;,&quot;M&quot;), Freq=c(4421,755,1018))&lt;/code&gt; though, as you say, &lt;code&gt;table&lt;/code&gt; or  &lt;code&gt;xtabs&lt;/code&gt; can be used instead.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Again, you can then ignore all the other options unless you know you need them, so you can end up with something as simple as the example's &lt;code&gt;dclus1p&amp;lt;-postStratify(dclus1, ~stype, pop.types)&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-03-17T00:40:10.997" />
  <row Id="8391" PostTypeId="2" ParentId="8389" CreationDate="2011-03-17T00:47:18.960" Score="9" ViewCount="" Body="&lt;p&gt;The distribution of the difference between two single independent samples from normal distributions has a mean which is the difference between the means of the original distributions and a variance which is the sum of the variances of the original distributions.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So in your case, if the normal distribution of $X$ has mean $\mu$ and variance $\sigma^2$ then $X_1-X_2$ has a normal distribution with mean $0$ and variance $2\sigma^2$; if you prefer,$\frac{X_1-X_2}{\sqrt{2} \sigma}$ has a standard normal distribution.  So long as you do a two-tailed calculation, that should be enough for you to find your answer.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To spell it out, your answer should be &#xA;$$2 \Phi \left(\frac{T}{\sqrt{2} \sigma} \right) - 1$$&#xA;where $\Phi$ is the cumulative distribution function of a standard normal distribution.&lt;/p&gt;&#xA;" OwnerUserId="2958" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-17T02:07:17.637" LastActivityDate="2011-03-17T02:07:17.637" CommentCount="5" />
  <row Id="8392" PostTypeId="1" AcceptedAnswerId="8431" CreationDate="2011-03-17T01:02:41.093" Score="5" ViewCount="119" Body="&lt;p&gt;I have logs of which users run which programs on our systems at a university. The data shows username, process time, and the time stamp for the sample. I also know which classes each user is in. Given this data, is it possible to reliably tell which professors/courses use a particular piece of software? I figure this method might be more reliable than asking the professors, who never respond to our queries! What statistical method, if any, would appropriate here? Pointers to the appropriate R function would be appreciated. Would principal component analysis be appropriate here?&lt;/p&gt;&#xA;" OwnerUserId="2814" LastEditorUserId="3369" LastEditorDisplayName="" LastEditDate="2011-03-17T21:59:53.503" LastActivityDate="2011-03-19T09:46:11.220" Title="How to determine course-based usage of software?" Tags="&lt;r&gt;&lt;modeling&gt;" AnswerCount="2" CommentCount="5" FavoriteCount="0" />
  <row Id="8393" PostTypeId="1" AcceptedAnswerId="8403" CreationDate="2011-03-17T01:17:24.617" Score="2" ViewCount="59" Body="&lt;p&gt;Hello, I have three questions.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;I want to conduct a planned comparison in ANCOVA which compares three groups (n=2, n=7 and n=5) with two groups (n=11 and n=25). Is it ok to include the group with n=2 if I am only conducting a planned comparison as described above?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Is it ok to conduct an ANCOVA when I am only comparing two groups? E.g. if I wanted to compare left and right hemisphere groups, controlling for another variable.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If the DV in an ANCOVA has been transformed (because it was non-normal) does the covariate have to undergo the same transformation? E.g. If the DV is (log transformed) heart rate during an experimental condition, and the covariate is heart rate measured at baseline (which is normally distributed), does the baseline HR need to be transformed as well?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="2025" LastActivityDate="2011-03-17T10:39:06.427" Title="Questions about number of groups and group size in planned comparisons in ANCOVA" Tags="&lt;data-transformation&gt;&lt;ancova&gt;&lt;small-sample&gt;" AnswerCount="1" />
  <row Id="8394" PostTypeId="2" ParentId="2917" CreationDate="2011-03-17T01:53:03.130" Score="0" ViewCount="" Body="&lt;p&gt;You may consider trying the R package &lt;a href=&quot;http://cran.r-project.org/web/packages/compute.es/index.html&quot; rel=&quot;nofollow&quot;&gt;compute.es&lt;/a&gt;. There are several functions for deriving effect size estimates and the variance of the effect size.  &lt;/p&gt;&#xA;" OwnerUserId="3752" LastEditorUserId="1381" LastEditorDisplayName="" LastEditDate="2011-03-17T02:47:49.370" LastActivityDate="2011-03-17T02:47:49.370" CommunityOwnedDate="2011-03-17T01:53:03.130" CommentCount="1" />
  <row Id="8395" PostTypeId="2" ParentId="8157" CreationDate="2011-03-17T02:56:04.293" Score="1" ViewCount="" Body="&lt;p&gt;If I understand you correctly, only points in some small volume of n-dimensional space meet your constraints.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your first constraint constrains it to the interior of a hypersphere,&#xA;which reminds me of the comp.graphics.algorithms FAQ &lt;a href=&quot;http://www.cgafaq.info/wiki/Uniform_random_points_on_sphere&quot; rel=&quot;nofollow&quot;&gt;&quot;Uniform random points on sphere&quot;&lt;/a&gt; and &lt;a href=&quot;http://stats.stackexchange.com/questions/8021/how-to-generate-uniformly-distributed-points-in-the-3-d-unit-ball&quot;&gt;How to generate uniformly distributed points in the 3-d unit ball?&lt;/a&gt;&#xA;The second constraint slices a bit out of the hypersphere,&#xA;and the other constraints further whittle away at the volume that meets your constraints.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think the simplest thing to do is one of the approaches suggested by the FAQ:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;choose some arbitrary &lt;a href=&quot;http://en.wikipedia.org/wiki/Bounding_box&quot; rel=&quot;nofollow&quot;&gt;Axis-aligned bounding box&lt;/a&gt; that we are sure contains the entire volume.&#xA;In this case, -c &amp;lt; a_1 &amp;lt; c, -c &amp;lt; a_2 &amp;lt; c, ... -c &amp;lt; a_n &amp;lt; c&#xA;contains the entire constrained volume, since it contains the hypersphere described by the first constraint, and the other constraints keep whittling away at that volume.&lt;/li&gt;&#xA;&lt;li&gt;The algorithm uniformly picks points throughout that bounding box.&#xA;In this case, the algorithm independently sets each coordinate of a candidate vector to some independent uniformly distributed random number from -c to +c.&#xA;(I am assuming you want points distributed with equal density throughout this volume. I suppose you could make the algorithm select some or all coordinates with a Poisson distribution or some other non-uniform distribution, if you had some reason to do that).&lt;/li&gt;&#xA;&lt;li&gt;Once you have a candidate vector, check each constraint. If it fails any of them, go back and pick another point.&lt;/li&gt;&#xA;&lt;li&gt;Once you have a candidate vector, store it somewhere for later use.&lt;/li&gt;&#xA;&lt;li&gt;If you don't have enough stored vectors, go back and try to generate another one.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;With sufficiently high-quality random number generator, this gives you a set of stored coordinates that meet your criteria with (expected) uniform density.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alas, if your have a relatively high dimensionality n (i.e., if you construct each vectors out of a relatively long list of coordinates), the inscribed sphere (much less your whittled-down volume) has a surprisingly small part of the total volume of the total bounding box, so it might need to execute many iterations, most of them generating rejected points outside your constrained area, before finding a point inside your constrained area.&#xA;Since computers these days are pretty fast, will that be fast enough?&lt;/p&gt;&#xA;" OwnerUserId="3754" LastActivityDate="2011-03-17T02:56:04.293" />
  <row Id="8396" PostTypeId="1" CreationDate="2011-03-17T03:27:07.263" Score="2" ViewCount="120" Body="&lt;p&gt;I do not know why package forecast 2.16 in R does not produce Theil's U?&#xA;I really appreciate your efforts.&lt;/p&gt;&#xA;" OwnerUserId="3755" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-03-17T07:22:09.587" LastActivityDate="2011-03-17T09:28:55.800" Title="How to produce Theil's U with package forecast 2.16 in R? " Tags="&lt;r&gt;&lt;forecasting&gt;&lt;goodness-of-fit&gt;" AnswerCount="1" />
  <row Id="8397" PostTypeId="2" ParentId="8377" CreationDate="2011-03-17T04:08:33.860" Score="1" ViewCount="" Body="&lt;p&gt;You may be interested by &lt;a href=&quot;http://joyeuserrance.wordpress.com/2010/11/25/the-linear-model-2-simple-linear-regression/&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; &lt;strong&gt;introduction to the linear model&lt;/strong&gt; (basis of almost any statistical analyses), and linear regression in particular:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;it thoroughly explains lots of the &lt;strong&gt;mathematical aspects of linear regression&lt;/strong&gt;, by detailing all important equations (which is usually left for exercise anywhere else on the Internet);&lt;/li&gt;&#xA;&lt;li&gt;it uses a simple, yet informative enough, &lt;strong&gt;data set as an example&lt;/strong&gt;;&lt;/li&gt;&#xA;&lt;li&gt;and it gives all the &lt;strong&gt;R commands&lt;/strong&gt; required to do the computations step by step, as well as plot the results.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="3459" LastActivityDate="2011-03-17T04:08:33.860" />
  <row Id="8398" PostTypeId="2" ParentId="8396" CreationDate="2011-03-17T05:01:07.543" Score="8" ViewCount="" Body="&lt;p&gt;It does. Use the &lt;code&gt;accuracy()&lt;/code&gt; command.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update:&lt;/strong&gt; here is an example.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(forecast)&#xA;x &amp;lt;- EuStockMarkets[1:200,1]&#xA;f &amp;lt;- EuStockMarkets[201:300,1]&#xA;fit1 &amp;lt;- ses(x,h=100)&#xA;accuracy(fit1,f)&#xA;        ME       RMSE        MAE        MPE       MAPE       MASE       ACF1  Theil's U &#xA; 0.8065983 78.1801986 63.2728352 -0.1725009  3.7876802  7.0619776  0.9586859  6.6120277 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you want the &lt;em&gt;in-sample&lt;/em&gt; value of U (which is of limited value), the following will work:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;fpe &amp;lt;- fit1$fitted[2:200]/x[1:199] - 1&#xA;ape &amp;lt;- x[2:200]/x[1:199] - 1&#xA;U &amp;lt;- sqrt(sum((fpe - ape)^2)/sum(ape^2))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="159" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2011-03-17T09:28:55.800" LastActivityDate="2011-03-17T09:28:55.800" CommentCount="2" />
  <row Id="8399" PostTypeId="1" CreationDate="2011-03-17T05:09:45.780" Score="2" ViewCount="134" Body="&lt;p&gt;Hi, I posted &lt;a href=&quot;http://electronics.stackexchange.com/q/10632/2939&quot;&gt;this&lt;/a&gt; question on Electronics.Stackexchange and someone told me I'll be better off posting it here. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Its an implementation of the Particle Filter using MATLAB but the results never follow the observations. I changed the weighting function to be Gaussian but still no avail. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;NOTE: I haven't taken into account that the process/measurement noise is time-correlated. Will this significantly change the accuracy of my program? If so, how do I correct for it? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;MY OLD QUESTION&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm working on a particle filter experiment for multi-sensor fusion and I just programmed it in MATLAB. However, I get very low accuracies for my final values. Plus, I read a lot of literature where they talk about pdf of state and observations etc. but my practical knowledge is still extremely shaky, since I've had no formal training in filtering/Bayesian estimates etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have devised my algorithm like this:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Initialize particles = I'm doing it as a Gaussian distribution - 10 particles&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Move the 10 particles forward using the state transition equation: &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;\begin{equation}&#xA;X_{t+1} = A \times X_t + 0.1 \times rand()&#xA;\end{equation}&#xA;I'm only injecting Gaussian noise so far.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Using the observation, calculate the weights for the particles. I do this like a root mean square of the difference between predicted state and observation. For example, if my azimuth(a) = 40, pitch(p) = 3, roll(r)=4 in my state and in my observation it is a = 39, p = 3, r = 3, then I do rms = sqrt((40-39)^2 + (3-3)^2 + (4-3)^2). Then my weight is assigned as 1/rms in order for it to be inversely proportional to the 'distance' between the prediction and observation&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Then I normalize these weights to get norm_weight = weight/norm(weight) so that their sum is equal to one.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Then I continue forward for all the observations. I have not included resampling yet because when I run this experiment, I do not experience any degeneracy, which is also very puzzling.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Where am I going wrong? I realized that I haven't 'computed' a lot of the Bayesian equations given in the literature i.e. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{equation}&#xA;p(x|z_t) = p(z_t|x)\times p(x)/p(z) &#xA;\end{equation}&#xA;and I don't know where it fits in here either. Can somebody please help me?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My Matlab code looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;function resultx = particlefilter(resultx_1, observationx, A, noiseP)&#xA;  for j = 1:length(observationx)&#xA;    for i = 1:length(resultx_1)&#xA;      apriori_state{i} = A*resultx_1{i} + noiseP;&#xA;      rms(i) = sqrt((observationx{j}(1) - apriori_state{i}(1))^2 +&#xA;                     (observationx{j}(2) - apriori_state{i}(2))^2);&#xA;      weight(i) = 1/rms(i);&#xA;    end;&#xA;    norm_weight = weight/norm(weight);&#xA;    for i = 1:length(apriori_state)&#xA;      plot(apriori_state{i});&#xA;    end&#xA;    disp(rms);&#xA;    disp(norm_weight);&#xA;  end&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="3233" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2011-03-17T13:32:09.867" LastActivityDate="2011-03-17T13:32:09.867" Title="Particle filter in Matlab - what is going wrong?" Tags="&lt;correlation&gt;&lt;matlab&gt;&lt;monte-carlo&gt;&lt;particle-filter&gt;" CommentCount="1" FavoriteCount="1" />
  <row Id="8401" PostTypeId="1" CreationDate="2011-03-17T06:41:46.460" Score="5" ViewCount="125" Body="&lt;p&gt;I am working on a research project that is related to optimization and recently had an idea to use MCMC in this setting. Unfortunately, I am fairly new to MCMC methods so I had several questions. I'll start by describing the problem and then asking my questions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Our problem boils down to estimating the expected value of a cost function $c(\omega)$ where $\omega = (\omega_1,\omega_2,...\omega_h)$ is an $h$-dimentional random variable with a density $f(\omega)$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In our case, a closed form version of $c(\omega)$ does not exist. This means that we have to use Monte Carlo methods to approximate the expected value. Unfortunately, it turns out that estimates of $E[c(\omega)]$ that are generated using MC or QMC methods have too much variance to be useful within in a practical setting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One idea that we had to use an importance sampling distribution to generate sample points that will produce a low variance estimate of $E[c(\omega)]$. In our case, the ideal importance sampling distribution, $g(\omega)$, has to be roughly proportional to $c(\omega)f(\omega)$. Seeing how $g(\omega)$ is known up to constant, I am wondering whether I can use MCMC along with the proposal distribution $c(\omega)f(\omega)$ to eventually generate samples from $g(\omega)$ . &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My questions here are: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Can MCMC be used within this setting? If so, what MCMC method would be appropriate? I am working in MATLAB, so I have a preference to anything that already has a MATLAB implementation.  &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Are there any techniques that I can use to speed up the burn-in period for MCMC.  And how can I tell that the stationary distribution has been reached? In this case, it actually takes a fair bit of time to calculate $c(\omega)$ for a given $\omega$.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="3572" LastEditorUserId="3572" LastEditorDisplayName="" LastEditDate="2011-03-17T08:51:51.883" LastActivityDate="2011-03-19T01:38:59.533" Title="Using MCMC to evaluate the expected value of a high-dimensional function" Tags="&lt;sampling&gt;&lt;mcmc&gt;&lt;matlab&gt;&lt;expected-value&gt;" AnswerCount="2" CommentCount="9" />
  <row Id="8402" PostTypeId="2" ParentId="8377" CreationDate="2011-03-17T10:33:44.997" Score="2" ViewCount="" Body="&lt;p&gt;If you want a book specifically on this sort of regression - as opposed to data analysis in general - I recommend &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0471746967&quot; rel=&quot;nofollow&quot;&gt;Regression Analysis by Example by Chatterjee and Price&lt;/a&gt;.  Good, not technical, but it doesn't oversimplify.  &lt;/p&gt;&#xA;" OwnerUserId="686" LastActivityDate="2011-03-17T10:33:44.997" />
  <row Id="8403" PostTypeId="2" ParentId="8393" CreationDate="2011-03-17T10:39:06.427" Score="4" ViewCount="" Body="&lt;p&gt;First off, with such small N, I think anything that comes close to statistical significance is going to pass the inter-ocular trauma test - it will hit you between the eyes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not sure what you mean by compare 3 groups with 2 groups.  Do you mean you want to combine the three groups into one, and the 2 groups into another and then compare?  Whether this is reasonable or not depends on whether it makes substantive sense to combine these groups.  But then the group sizes are a bit larger, which is good.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ANCOVA is fine with only 2 groups.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;No, what you do to the DV does NOT have to be done to the IVs.  The topic of transformations is a complex one, but this is an easy question to answer.  Of course, whether it is right to transform your DV in this way is a different matter.&lt;/p&gt;&#xA;" OwnerUserId="686" LastActivityDate="2011-03-17T10:39:06.427" CommentCount="3" />
  <row Id="8404" PostTypeId="2" ParentId="726" CreationDate="2011-03-17T10:53:40.767" Score="3" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;The plural of anecdote is not data.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;-- Roger Brinner&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(in the context of &lt;a href=&quot;http://en.wikipedia.org/wiki/Anecdotal_evidence&quot; rel=&quot;nofollow&quot;&gt;Anecdotal_evidence&lt;/a&gt;)&lt;/p&gt;&#xA;" OwnerUserId="264" LastActivityDate="2011-03-17T10:53:40.767" CommunityOwnedDate="2011-03-17T10:53:40.767" />
  <row Id="8405" PostTypeId="1" AcceptedAnswerId="8410" CreationDate="2011-03-17T12:03:05.233" Score="1" ViewCount="135" Body="&lt;p&gt;I have continuous data $A$ and categorical data $O$. I need counts of $A$ in bins by group $O$. I'm working in R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know how to bin data (using &lt;code&gt;cut2&lt;/code&gt;) and how to get the counts of $O$ (using &lt;code&gt;aggregate&lt;/code&gt; or &lt;code&gt;by&lt;/code&gt; or &lt;code&gt;describe&lt;/code&gt; or &lt;code&gt;summaryBy&lt;/code&gt;). I could also get what I need by running one of these functions with a subset (bin) of the data one by one, but is there a way to do this automatically in one command?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update:&lt;/strong&gt;&lt;br&gt;&#xA;The output I want is just basic descriptive statistics (counts in this case, but also means and/or medians for example):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Age (A)................Outcome (O)&#xA;........................Yes....No&#xA;18-27 ...................10....12&#xA;28-37 ....................2....11&#xA;38-47 ....................9.....7&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;by(subset(X, X$Age&amp;gt;47)$Age, subset(X, X$Age&amp;gt;47)$Outcome, length)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;gives me the number of data points for ages &gt; 47 where &lt;code&gt;Outcome&lt;/code&gt; is true and where &lt;code&gt;O&lt;/code&gt; is false. I want to do this for all bins automatically, without having to run separate commands with long &gt; and &amp;lt; conditions for each bin.&lt;/p&gt;&#xA;" OwnerUserId="2824" LastEditorUserId="2824" LastEditorDisplayName="" LastEditDate="2011-03-22T09:04:58.307" LastActivityDate="2011-03-28T06:07:52.767" Title="Counts of binned data by group" Tags="&lt;r&gt;&lt;descriptive-statistics&gt;&lt;binning&gt;" AnswerCount="3" CommentCount="2" />
  <row Id="8406" PostTypeId="2" ParentId="726" CreationDate="2011-03-17T12:05:58.503" Score="2" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;The researcher armed with a confidence&#xA;  interval, but deprived of the false&#xA;  respectability of statistical&#xA;  significance, must work harder to&#xA;  convince himself and others of the&#xA;  importance of his findings. This can&#xA;  only be good.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Michael Oakes, &lt;em&gt;Statistical inference: A commentary for the social and behavioural sciences&lt;/em&gt;  (NY: Wiley, 1986)&lt;/p&gt;&#xA;" OwnerUserId="2669" LastActivityDate="2011-03-17T12:05:58.503" CommunityOwnedDate="2011-03-17T12:05:58.503" />
  <row Id="8407" PostTypeId="1" AcceptedAnswerId="8417" CreationDate="2011-03-17T12:11:10.293" Score="9" ViewCount="293" Body="&lt;p&gt;I would like to import the &quot;Last Trade&quot; stock price from Yahoo finance into R. The intention is to work with (almost) real time data. Are there any solutions?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance for any helpful comment. &lt;/p&gt;&#xA;" OwnerUserId="3757" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-18T16:15:41.810" LastActivityDate="2011-04-05T03:09:33.447" Title="Import stock price from Yahoo Finance into R?" Tags="&lt;r&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="8408" PostTypeId="2" ParentId="8407" CreationDate="2011-03-17T12:29:03.407" Score="7" ViewCount="" Body="&lt;p&gt;That is pretty easy given that &lt;a href=&quot;http://www.r-project.org&quot; rel=&quot;nofollow&quot;&gt;R&lt;/a&gt; can read directly off a given URL. The key is simply to know how to form the URL.  Here is a quick and dirty example based on code Dj Padzensky wrote in the late 1990s and which I have been maintaining in the &lt;a href=&quot;http://www.perl.org&quot; rel=&quot;nofollow&quot;&gt;Perl&lt;/a&gt; module &lt;a href=&quot;http://dirk.eddelbuettel.com/code/yahooquote.html&quot; rel=&quot;nofollow&quot;&gt;Yahoo-FinanceQuote&lt;/a&gt; (which is of course also on &lt;a href=&quot;http://search.cpan.org/~edd/Finance-YahooQuote/YahooQuote.pm&quot; rel=&quot;nofollow&quot;&gt;CPAN here&lt;/a&gt;) for almost as long.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you know a little R, the code should be self-explanatory. Getting documentation for the format string is a little trickier but e.g. the Perl module has some.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;R&amp;gt; syms &amp;lt;- c(&quot;^GSPC&quot;, &quot;^IXIC&quot;)&#xA;R&amp;gt; baseURL &amp;lt;- &quot;http://download.finance.yahoo.com/d/quotes.csvr?e=.csv&amp;amp;f=&quot;&#xA;R&amp;gt; formatURL &amp;lt;- &quot;snl1d1t1c1p2va2bapomwerr1dyj1x&quot;&#xA;R&amp;gt; endURL &amp;lt;- &quot;&amp;amp;s=&quot;&#xA;R&amp;gt; url &amp;lt;- paste(baseURL, formatURL, endURL, paste(syms, collapse=&quot;+&quot;), sep=&quot;&quot;)&#xA;R&amp;gt; read.csv(url, header=FALSE)&#xA;     V1                V2      V3        V4     V5 V6    V7&#xA;1 ^GSPC S&amp;amp;P 500 INDEX,RTH 1256.88 3/16/2011 4:04pm  0 0.00%&#xA;2 ^IXIC  NASDAQ Composite 2616.82 3/16/2011 5:30pm  0 0.00%&#xA;          V8 V9 V10 V11     V12     V13               V14&#xA;1 4282084608  0 N/A N/A 1256.88 1279.46 1249.05 - 1280.91&#xA;2          0  0 N/A N/A 2616.82    0.00       0.00 - 0.00&#xA;                V15 V16 V17 V18 V19 V20 V21      V22&#xA;1 1010.91 - 1344.07 N/A N/A N/A N/A N/A N/A      SNP&#xA;2 2061.14 - 2840.51 N/A N/A N/A N/A N/A N/A NasdaqSC&#xA;R&amp;gt; &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Column three is your last trade. During open market hours you will get fewer NAs and more data variability. But note though that most prices are 15 or 20 minute delayed---but some indices are real-time.  Real-time data is a big business and major revenue for exchanges so they tend not to give it away.  Also, and if I remember correctly, the newer and more real-time displays on the Finance pages at Google and Yahoo use something more AJAXy that is harder to milk from the outside. &lt;/p&gt;&#xA;" OwnerUserId="334" LastActivityDate="2011-03-17T12:29:03.407" CommentCount="1" />
  <row Id="8409" PostTypeId="2" ParentId="8401" CreationDate="2011-03-17T12:40:34.437" Score="2" ViewCount="" Body="&lt;p&gt;I would always remember, that MCMC is just a numerical integration tool (and a rather inefficient one at that).  It is not some magic/mystical thing.  It is very useful because it is reasonably easy to apply.  It does not require much thinking compared to some other numerical integration techniques.  For instance, you do not have to do any derivatives.  You only have to generate &quot;random numbers&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, like any numerical integration method, it is not a universal catch all tool.  There are conditions when it is useful, and conditions when it isn't.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It may be wiser to set up another technique.  Depending on how big $h$ is, and how fast your computer is, and how much time you are prepared to wait for results.  A uniform grid may do the job (although this requires small $h$ or a long amount of waiting).  The &quot;job&quot; is to evaluate the integral - the equation does not care what meaning you or I attach to the result (and hence it does not care whether we obtained the result randomly or not).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Additionally, if your estimates of $\omega$ are quite accurate, the $f(\omega)$ will be sharply peaked and closely resemble a delta function, so the integral is effectively substituting $\omega\rightarrow\omega_{max}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another numerical integration technique is using a taylor series under the integral.  $f(\omega)\approx f(\omega_{max})+(\omega-\omega_{max})f&amp;#39;(\omega_{max})+\frac{1}{2}(\omega-\omega_{max})^{2}f&amp;#39;&amp;#39;(\omega_{max})+\dots$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a useful strategy when the moments of $\omega$ are easily obtained.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edwin Jaynes has a nice quote on this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;whenever there is a randomised way of doing something, there is a non-randomised way which yields better results, but requires more thinking&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One &quot;more thinking&quot; way is to use &quot;stratified MCMC&quot; to do the integral.  So rather than &quot;randomly&quot; pick a spot on the whole parameter space: divide it up into &quot;strata&quot;.  These &quot;strata&quot; should be picked so that you get a good range of the high part of the integral.  Then randomly sample within each strata.  But this will require you to write your own code I would imagine (i.e. more thinking).&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-03-17T12:40:34.437" CommentCount="2" />
  <row Id="8410" PostTypeId="2" ParentId="8405" CreationDate="2011-03-17T13:04:36.523" Score="4" ViewCount="" Body="&lt;p&gt;I really like using the &lt;code&gt;summary.formula&lt;/code&gt; function in the &lt;code&gt;Hmisc&lt;/code&gt; package for these tasks. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some artificial data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;A &amp;lt;- rnorm(100, mean=50, sd=20)&#xA;G &amp;lt;- gl(2, 50)&#xA;B &amp;lt;- unclass(G)*10 + rnorm(100, sd=3)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Descriptives by &lt;code&gt;G&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;summary( G ~ cut2(A,g=4) + B, method=&quot;reverse&quot;)&#xA;&#xA;Descriptive Statistics by G    &#xA;+------------------------------+-----------------------------+-----------------------------+&#xA;|                              |1                            |2                            |&#xA;|                              |(N=50)                       |(N=50)                       |&#xA;+------------------------------+-----------------------------+-----------------------------+&#xA;|cut2(A, g = 4) : [ 5.19, 43.0)|           22% (11)          |           28% (14)          |&#xA;+------------------------------+-----------------------------+-----------------------------+&#xA;|    [42.98, 49.7)             |           18% ( 9)          |           32% (16)          |&#xA;+------------------------------+-----------------------------+-----------------------------+&#xA;|    [49.68, 60.3)             |           26% (13)          |           24% (12)          |&#xA;+------------------------------+-----------------------------+-----------------------------+&#xA;|    [60.27,107.5]             |           34% (17)          |           16% ( 8)          |&#xA;+------------------------------+-----------------------------+-----------------------------+&#xA;|B                             | 7.849173/10.153238/11.737293|18.766779/20.155744/22.139419|&#xA;+------------------------------+-----------------------------+-----------------------------+&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Summarizing &lt;code&gt;B&lt;/code&gt; by &lt;code&gt;G&lt;/code&gt; and ranges of &lt;code&gt;A&lt;/code&gt; with a custom function:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;summary(B ~ cut2(A, g=4) + stratify(G), fun=function(x)c(mean(x),sd(x)))&#xA;B by g    N=100&#xA;&#xA;+--------------+-------------+--+---------+--------+--+--------+--------+&#xA;|              |             |N |         |B       |N |        |B       |&#xA;+--------------+-------------+--+---------+--------+--+--------+--------+&#xA;|cut2(A, g = 4)|[ 5.19, 43.0)|11|10.351295|3.069181|14|20.55056|3.659311|&#xA;|              |[42.98, 49.7)| 9|10.269449|2.460152|16|19.18630|1.628746|&#xA;|              |[49.68, 60.3)|13| 9.432093|2.928439|12|21.07909|3.064897|&#xA;|              |[60.27,107.5]|17| 9.683748|2.800625| 8|20.65612|2.685183|&#xA;+--------------+-------------+--+---------+--------+--+--------+--------+&#xA;|Overall       |             |50| 9.870604|2.777788|50|20.25773|2.844987|&#xA;+--------------+-------------+--+---------+--------+--+--------+--------+&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The documentation describes lots of options to control what is calculated and how the output looks like.&lt;/p&gt;&#xA;" OwnerUserId="279" LastEditorUserId="279" LastEditorDisplayName="" LastEditDate="2011-03-17T15:52:42.263" LastActivityDate="2011-03-17T15:52:42.263" CommentCount="10" />
  <row Id="8413" PostTypeId="2" ParentId="8377" CreationDate="2011-03-17T14:42:00.950" Score="3" ViewCount="" Body="&lt;p&gt;The two together don't tell you anything more than the second one would alone! The main effects are uninteresting and misleading when there is interaction present. The second model tells you all you need to know.  Here are a couple of plots, with R code, to help you understand what that second model looks like...&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(lattice)&#xA;&#xA;a &amp;lt;- rep(seq(-1.37, 2.12, (2.12--1.37)/9),4) &#xA;b &amp;lt;- sort(rep(quantile(seq(-1.03, 1.30, .01),c(.2,.4,.6,.8)),10) )&#xA;y &amp;lt;- -0.06807 + (3.01517 * a) + (-0.00994 * b) + (-1.13782 *a*b)&#xA;&#xA;xyplot(y~a|factor(b))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/cL3dM.png&quot; alt=&quot;Effect of a on y by Levels of b&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This one shows the estimated effect of &lt;strong&gt;&lt;em&gt;a&lt;/em&gt;&lt;/strong&gt; on &lt;strong&gt;&lt;em&gt;y&lt;/em&gt;&lt;/strong&gt; by levels of &lt;strong&gt;&lt;em&gt;b&lt;/em&gt;&lt;/strong&gt;.  At each level of &lt;strong&gt;&lt;em&gt;b&lt;/em&gt;&lt;/strong&gt;, the relationship is positive.  This is your significant positive slope for main effect of &lt;strong&gt;&lt;em&gt;a&lt;/em&gt;&lt;/strong&gt; in the presence of the interaction &lt;strong&gt;&lt;em&gt;a:b&lt;/em&gt;&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;a &amp;lt;- sort(rep(quantile(seq(-1.37, 2.12, .01),c(.2,.4,.6,.8)),10) )&#xA;b &amp;lt;- rep(seq(-1.03, 1.30, (1.30--1.03)/9),4) &#xA;y &amp;lt;- -0.06807 + (3.01517 * a) + (-0.00994 * b) + (-1.13782 *a*b)&#xA;&#xA;xyplot(y~b|factor(a))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/awmJk.png&quot; alt=&quot;Effect of b on y by Levels of a&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is image shows the estimated effects of &lt;strong&gt;&lt;em&gt;b&lt;/em&gt;&lt;/strong&gt; on &lt;strong&gt;&lt;em&gt;y&lt;/em&gt;&lt;/strong&gt; within levels of &lt;strong&gt;&lt;em&gt;a&lt;/em&gt;&lt;/strong&gt;.  You can see why you have no significant main effect for &lt;strong&gt;&lt;em&gt;b&lt;/em&gt;&lt;/strong&gt;.  The direction of the &lt;strong&gt;&lt;em&gt;y~b&lt;/em&gt;&lt;/strong&gt; relationship depends on level of &lt;strong&gt;&lt;em&gt;a&lt;/em&gt;&lt;/strong&gt;.  Thus, no independent relationship (imagine averaging those lines) but a significant interaction (clear pattern when you take into account the level of &lt;strong&gt;&lt;em&gt;a&lt;/em&gt;&lt;/strong&gt;) &lt;/p&gt;&#xA;" OwnerUserId="485" LastActivityDate="2011-03-17T14:42:00.950" />
  <row Id="8415" PostTypeId="1" AcceptedAnswerId="8416" CreationDate="2011-03-17T16:09:46.423" Score="3" ViewCount="83" Body="&lt;p&gt;I have a time series, let's call it $y(t)$. Augmented Dickey-Fuller and Phillips-Perron tests indicate no unit root but a deterministic trend. The series is strongly autocorrelated. I estimated the following short-run dynamic model: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$y(t) = \alpha + \beta\cdot y(t-1) + \gamma\cdot t + \varepsilon_t$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\alpha$, $\beta$ and $\gamma$ are estimated coefficients. The estimated model has no statistical anomalies and the coefficients are significantly different from zero. The estimated value of $\gamma$ measures a trend. However, a long-run trend can be estimated by calculating $\gamma/(1-\beta)$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can estimate directly the following static, long-run model: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$y(t) = \alpha + \gamma\cdot t + \varepsilon_t.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question&lt;/strong&gt;: What is the &lt;em&gt;best&lt;/em&gt;, &lt;em&gt;consistent&lt;/em&gt; model to apply in an analysis of a deterministic trend?&lt;/p&gt;&#xA;" OwnerUserId="3762" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-17T16:30:57.613" LastActivityDate="2011-03-17T16:46:02.370" Title="Short and long-run trend" Tags="&lt;time-series&gt;&lt;model-selection&gt;&lt;trend&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="8416" PostTypeId="2" ParentId="8415" CreationDate="2011-03-17T16:46:02.370" Score="2" ViewCount="" Body="&lt;p&gt;There is no quick solution to the problem, since the deterministic trend is just a function of $t$. We may denote this trend by $f(t)$ and it is not evident that the trend is linear.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So some quick tips what to do in this case:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Plot the original data, though the noise ratio could be huge therefore...&lt;/li&gt;&#xA;&lt;li&gt;Make technical decomposition of your time series into &lt;em&gt;Trend&lt;/em&gt; + &lt;em&gt;Cycle&lt;/em&gt; + &lt;em&gt;Seasonal&lt;/em&gt; + &lt;em&gt;Residual&lt;/em&gt; components and try to plot them (implementation of this step depends on the software you do use), usual approach is to apply Hodrick-Prescott filter for example. &lt;/li&gt;&#xA;&lt;li&gt;After you decide on the form of trend you may conclude that some polynomial $\sum_{i=0}^n \alpha_i t^i$ is suitable, to check this you may apply $n$ times difference operator, the reminder term should be close to white noise then...&lt;/li&gt;&#xA;&lt;li&gt;If it is not search in addition for the best fit from the $(S)ARMA(p, q)$ class (since there is no unit root in your case, in general you will work with SARIMA, or some advanced users would go for SARFIMA-GARCH may be :)).&lt;/li&gt;&#xA;&lt;li&gt;Decide on the efficient estimate of the final model on the basis of any information criteria you think is relevant ($BIC$ for example).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Anyway, the time series models are needed for short term forecasting needs (without clear structure and theoretical restrictions long-term forecasts are useless), short and long term betas therefore are just a matter of curiosity. Since you do not have a structural model here you &lt;strong&gt;do not need to care&lt;/strong&gt; about &lt;em&gt;consistency&lt;/em&gt;. For forecasting needs any parsimonious (the simpler the better) model with &lt;em&gt;lowest information criteria&lt;/em&gt; (&lt;strong&gt;the best&lt;/strong&gt;) will do nicely in the short run, even if the true data generating process is more complicated.&lt;/p&gt;&#xA;" OwnerUserId="2645" LastActivityDate="2011-03-17T16:46:02.370" CommentCount="3" />
  <row Id="8417" PostTypeId="2" ParentId="8407" CreationDate="2011-03-17T16:48:37.487" Score="7" ViewCount="" Body="&lt;p&gt;This really isn't a statistics question (perhaps this could be moved to SO?), but there's a nice function in &lt;a href=&quot;http://www.quantmod.com&quot; rel=&quot;nofollow&quot;&gt;quantmod&lt;/a&gt; that does what Dirk has done by hand.  See &lt;code&gt;getQuote()&lt;/code&gt; and &lt;code&gt;yahooQF()&lt;/code&gt;.  Typing &lt;code&gt;yahooQF()&lt;/code&gt; will bring up a menu of all the possible quote formats you can use.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; getQuote(&quot;QQQQ;SPY&quot;, what=yahooQF(&quot;Last Trade (Price Only)&quot;))&#xA;              Trade Time   Last&#xA;QQQQ 2011-03-17 12:33:00  55.14&#xA;SPY  2011-03-17 12:33:00 128.17&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1657" LastActivityDate="2011-03-17T16:48:37.487" CommentCount="2" />
  <row Id="8418" PostTypeId="1" CreationDate="2011-03-17T16:52:45.347" Score="1" ViewCount="100" Body="&lt;p&gt;I have a dataset with repeated measures at different speeds. I've binned the speed ranges into 0-0.5, 0.5-1.5, and 1.5-infinity. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mydata`$`fact&amp;lt;−cut(mydata$section, breaks=c(0,0.5,1.5,Inf))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which gives the levels (0,0.5]  (0.5,1.5]  (1.5,Inf] &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I first used the lme to test all speeds together to see if x and y are speed-dependent. Now I want to isolate speed bins, but I want to do it in the model instead of re-defining the factor. I can't seem to get it to work. &quot;Fact&quot; is what the speed bins correspond to.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;lme(y ~ x + factor(repeatedmeasures) + fact, random = ~1 | z, data=mydata, na.action=na.omit)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Thanks for your help!&lt;/p&gt;&#xA;" OwnerUserId="3761" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-17T16:56:34.597" LastActivityDate="2011-03-17T18:10:24.983" Title="How can I specify a level of a factor while in an lme?" Tags="&lt;r&gt;&lt;mixed-model&gt;" AnswerCount="1" CommentCount="4" />
  <row Id="8419" PostTypeId="1" AcceptedAnswerId="8434" CreationDate="2011-03-17T17:18:25.883" Score="2" ViewCount="158" Body="&lt;p&gt;Supposing a Bayesian classifier with multivariate normal densities, how do I find the error rate of the classifier when we have two classes?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When dimension $d = 1$:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(x | \mu , \sigma^2) = N(x, \mu, \sigma^2) = \frac{1}{\sqrt{2\pi \sigma^2}} e^{-\frac{(x-\mu)^2}{2\sigma^2}} $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In $d$ dimensions:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(x | \mu, \Sigma) = \frac{1}{(2\pi)^{d/2} |\Sigma|^{1/2}} \cdot e^{-\frac{1}{2}{(x-μ)^T\Sigma^{-1}(x-μ)}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\mu$ is in $\mathbb{R}^d$ and $\Sigma$ is a $d \times d$ variance-covariance matrix.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What would be the algebra to get the error rate, or could you give an example? I would like to do this in &lt;code&gt;Matlab&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lets say I have 2 classes and 2 atributes with 20 examples:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;features:&#xA;    1.8756    1.4236&#xA;    2.0677    1.3759&#xA;    0.8540    0.7782&#xA;    0.5651   -0.3511&#xA;    0.6103   -0.6901&#xA;   -0.1945    1.6438&#xA;   -0.2620    0.8022&#xA;    1.5326    0.0188&#xA;    0.3334    1.0578&#xA;    0.8535   -1.8545&#xA;    0.3066    1.9716&#xA;   -1.4424    0.4216&#xA;    0.3275   -0.2844&#xA;   -0.0079    2.8506&#xA;    0.0114    1.4001&#xA;   -0.4049   -0.3981&#xA;   -0.0913    2.2094&#xA;    0.3376   -1.0467&#xA;    0.3455    2.4960&#xA;    0.3232   -0.5614&#xA;&#xA;&#xA;and targets&#xA;&#xA;     2&#xA;     2&#xA;     1&#xA;     1&#xA;     1&#xA;     2&#xA;     1&#xA;     1&#xA;     2&#xA;     1&#xA;     2&#xA;     1&#xA;     1&#xA;     2&#xA;     2&#xA;     1&#xA;     2&#xA;     1&#xA;     2&#xA;     1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What is the process to classify them and get the error rate of this examples, so we divide the set in 15 examples for training and the other 5 for testing?&lt;/p&gt;&#xA;" OwnerUserId="3681" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-29T00:57:16.927" LastActivityDate="2011-04-29T00:57:16.927" Title="Bayesian classifier with multivariate normal densities" Tags="&lt;machine-learning&gt;&lt;naive-bayes&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="8420" PostTypeId="1" AcceptedAnswerId="8450" CreationDate="2011-03-17T17:33:20.797" Score="3" ViewCount="71" Body="&lt;p&gt;Hello all,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other then gapminder.org, could you please direct me to any good examples (interactive or static) of visualizations that compare between countries?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="253" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-18T15:59:39.973" LastActivityDate="2011-03-18T15:59:39.973" Title="Examples of visualization featuring comparisons between countries?" Tags="&lt;data-visualization&gt;" AnswerCount="3" CommentCount="1" />
  <row Id="8421" PostTypeId="2" ParentId="7720" CreationDate="2011-03-17T17:52:24.270" Score="0" ViewCount="" Body="&lt;p&gt;dfrankow,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To 'test' (i.e., evaluate) the proportional odds assumption in R, you can use residuals.lrm() in Frank Harrell Jr.'s Design package. If you type ?residuals.lrm , there is a quick-to-replicate example of how Frank Harrell recommends evaluating the proportional odds assumption (i.e., visually, rather than by a push-button test). Design estimates ordered logistic regressions using lrm(), which you can substitute for polr() from MASS. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a more formal example of how to visually test the proportional odds assumption in R, see:&#xA;Paper: Ordinal Response Regression Models in Ecology &#xA;Author(s): Antoine Guisan and Frank E. Harrell &#xA;Source: Journal of Vegetation Science, Vol. 11, No. 5 (Oct., 2000), pp. 617-626&lt;/p&gt;&#xA;&#xA;&lt;p&gt;HTH,&#xA;Mark&lt;/p&gt;&#xA;" OwnerUserId="3763" LastEditorUserId="3763" LastEditorDisplayName="" LastEditDate="2011-03-17T18:20:48.850" LastActivityDate="2011-03-17T18:20:48.850" CommentCount="1" />
  <row Id="8422" PostTypeId="2" ParentId="8206" CreationDate="2011-03-17T17:58:12.653" Score="1" ViewCount="" Body="&lt;p&gt;Here is mine implementation of your solutions. I've decided not to map the mean value, there is not to much space left. Also the line from 0 to 1 seems odd.&#xA;Thank's a lot everyone.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data &amp;lt;- read.table(&quot;roc_average.txt&quot;)&#xA;bxp &amp;lt;- boxplot(data, horizontal = TRUE, range = 0, axes = FALSE, col = &quot;grey&quot;, add = TRUE, at = 0.2, varwidth=FALSE, boxwex=0.3)&#xA;valuelabels &amp;lt;- c(round(fivenum(data)[2], digits = 2), round(fivenum(data)[4], digits = 2))&#xA;text(x = valuelabels, y = c(0.35, 0.35), labels = valuelabels, font = 2)&#xA;mtext(c(min(round(data, digits = 2)),max(round(data, digits = 2))), side=1, at=bxp$stats[c(1,5)], line=-3, font = 2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/GlIEA.png&quot; alt=&quot;roc curve with boxplot&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="3345" LastEditorUserId="3345" LastEditorDisplayName="" LastEditDate="2011-03-19T12:26:25.473" LastActivityDate="2011-03-19T12:26:25.473" CommentCount="5" />
  <row Id="8423" PostTypeId="2" ParentId="8420" CreationDate="2011-03-17T18:07:05.030" Score="1" ViewCount="" Body="&lt;p&gt;Tableau Public also has some (non map) visualizations which compare data across countries.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.tableausoftware.com/public/gallery&quot; rel=&quot;nofollow&quot;&gt;http://www.tableausoftware.com/public/gallery&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="3489" LastActivityDate="2011-03-17T18:07:05.030" />
  <row Id="8424" PostTypeId="2" ParentId="8418" CreationDate="2011-03-17T18:10:24.983" Score="1" ViewCount="" Body="&lt;p&gt;If you just want to fit separate models for each level of your factor, then probably the easiest way is to use the &lt;code&gt;subset=&lt;/code&gt; argument to &lt;code&gt;lme&lt;/code&gt; (or any other GLM in R, btw). For example,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;lme(y ~ x + factor(repeatedmeasures) + fact, &#xA;    random = ~1 | z, data=mydata, subset=as.numeric(fact) == 1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;should subset on the first level of your factor. I used &lt;code&gt;as.numeric()&lt;/code&gt; because you didn't provide labels to your newly created factor. To do so, you can fill the &lt;code&gt;labels=&lt;/code&gt; argument when calling &lt;code&gt;cut()&lt;/code&gt;, such that the above code could read&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;lme(..., subset=fact == &quot;low&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;for example (assuming the first level of &lt;code&gt;fact&lt;/code&gt; is named &lt;code&gt;low&lt;/code&gt;). In both cases, you can easy set up a loop (using something like &lt;code&gt;for i in 1:nlevels(fact)&lt;/code&gt; or &lt;code&gt;for i in seq_along(levels(fact))&lt;/code&gt;) to iterate over your different models.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-03-17T18:10:24.983" CommentCount="5" />
  <row Id="8425" PostTypeId="1" CreationDate="2010-08-27T18:14:14.280" Score="4" ViewCount="105" Body="&lt;p&gt;I hope this is an appropriate forum for this question...if not, any pointers on a place to ask would be great.  If my questions is not clear, please just let me know and I'll try to add information/explanation where I can.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say I have a set of data points that I run through a &lt;a href=&quot;http://en.wikipedia.org/wiki/FLAME_clustering&quot; rel=&quot;nofollow&quot;&gt;FLAME clustering algorithm&lt;/a&gt;, and I get my set of cluster supporting objects, and the fuzzy memberships among them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Next, let's say I get an additional set of data, for argument's sake, let's say it doubles the amount of data.  If I want to add that to my clustering, without having to reprocess the old data (i.e. just using the cluster supporting objects, and adding these new observations to the cluster), will I be introducing some kind of bias?  To get the most &quot;accurate&quot; picture, would it be better to reprocess ALL the data, re-establish CSO's, re-compute distances, etc.?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To add to the complexity, let's say I'm going to be periodically adding similar quantities of data points.  Does the answer to the above change?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And I guess a final question - is there a clustering algorithm (or family of algorithms) that is not subject to extreme bias in the beginning, such that added data points can be added relatively pain-free?  I'm sort of thinking that FLAME clustering, with its nice fuzziness properties is a reasonable way of going about this, but any more suggestions would be helpful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="338" LastEditorUserId="3369" LastEditorDisplayName="" LastEditDate="2011-03-17T21:59:03.863" LastActivityDate="2011-05-17T08:47:06.717" Title="Bias from increased information in FLAME clustering" Tags="&lt;clustering&gt;&lt;online&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="8426" PostTypeId="2" ParentId="8425" CreationDate="2010-12-17T07:46:14.033" Score="0" ViewCount="" Body="&lt;p&gt;I think it is possible to introduce the new set of data periodically by assigning its initial value with the interpolated value of its neighbor while the possible set of neighbor for each new data point could be the points in the triangle that data point resides in. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think what is more important is the order of assigning the initial values for this set of data point since the data point that is closer to CSO will be updated first. To determine the closeness of data point to CSO, we can use Euclidean distance, weighted link or both. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: something to think about&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Data point that lie outside the contour of current data set.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The number of update recurring. The data point tends to converge to CSO as the number of update increases. This can cause saturation problem. To solve this problem, subsidizing current data points or levitating new data points is necessary. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-18T08:01:26.373" LastActivityDate="2011-03-18T08:01:26.373" />
  <row Id="8428" PostTypeId="1" CreationDate="2011-03-17T19:28:38.270" Score="3" ViewCount="166" Body="&lt;p&gt;I am finishing up some analysis on a large set of data. I would like to take the linear model used in the first part of the work and re-fit it using an LME. The LME would be very similar with the exception that one of the variables used in the model would be used in the random effects. This data comes from many observations (&gt;1000) in a small group of subjects (~10) and I know that modeling the effect of subject is better done as a random effect (this is a variable that I want to shift). The R code would look like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;my_modelB &amp;lt;- lm(formula = A ~ B + C + D)    &#xA;lme_model &amp;lt;- lme(fixed=A ~ B + C, random=~1|D, data=my_data, method='REML')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Everything runs fine and the results are vastly similar. It would be nice if I could use something like RLRsim or an AIC/BIC to compare these two models and decide which is the most appropriate. My colleagues don't want to report the LME because there isn't an easily accessible way of choosing which is &quot;better&quot;, even though I think the LME is the more appropriate model. Any suggestions?&lt;/p&gt;&#xA;" OwnerUserId="3727" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-17T19:40:50.973" LastActivityDate="2011-03-17T20:50:30.897" Title="Comparing an LME to a LM" Tags="&lt;r&gt;&lt;regression&gt;&lt;lmer&gt;" AnswerCount="2" />
  <row Id="8429" PostTypeId="2" ParentId="8428" CreationDate="2011-03-17T20:08:02.357" Score="1" ViewCount="" Body="&lt;p&gt;I am not totally sure to figure out what model is fitted when you use the lme function. (I guess the random effect is supposed to follow a normal distribution with zero mean?). However, the linear model is a special case of the mixed model when the variance of the random effect is zero. Although some technical difficulties exist (because $0$ is in the boundary of the parameter space for the variance) it should be possible to test $H_0:variance = 0$ vs $H_1: variance &amp;gt; 0$... &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to avoid confusion: The test mentioned above is sometimes used to decide whether or not the random effect is significant... but not to decide whether or not it should be transformed into a fixed effect.&lt;/p&gt;&#xA;" OwnerUserId="3019" LastEditorUserId="3019" LastEditorDisplayName="" LastEditDate="2011-03-17T20:34:47.713" LastActivityDate="2011-03-17T20:34:47.713" CommentCount="4" />
  <row Id="8430" PostTypeId="2" ParentId="8428" CreationDate="2011-03-17T20:43:54.430" Score="1" ViewCount="" Body="&lt;p&gt;This is to add to @ocram's answer because it is too long to post as a comment. I would treat &lt;code&gt;A ~ B + C&lt;/code&gt; as your null model so you can assess the statistical significance of a &lt;code&gt;D&lt;/code&gt;-level random intercept in a nested model setup. As ocram pointed out, regularity conditions are violated when $H_0: \sigma^2 = 0$, and the likelihood ratio test statistic (LRT) will not necessarily be asymptotically distributed $\chi^2$. The solution was I taught was to bootstrap the LRT (whose bootstrap distribution will likely not be $\chi^2$) parametrically and compute a bootstrap p-value like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(lme4)&#xA;my_modelB &amp;lt;- lm(formula = A ~ B + C)&#xA;lme_model &amp;lt;- lmer(y ~ B + C + (1|D), data=my_data, REML=F)&#xA;lrt.observed &amp;lt;- as.numeric(2*(logLik(lme_model) - logLik(my_modelB)))&#xA;nsim &amp;lt;- 999&#xA;lrt.sim &amp;lt;- numeric(nsim)&#xA;for (i in 1:nsim) {&#xA;    y &amp;lt;- unlist(simulate(mymodlB))&#xA;    nullmod &amp;lt;- lm(y ~ B + C)&#xA;    altmod &amp;lt;- lmer(y ~ B + C + (1|D), data=my_data, REML=F)&#xA;    lrt.sim[i] &amp;lt;- as.numeric(2*(logLik(altmod) - logLik(nullmod)))&#xA;}&#xA;mean(lrt.sim &amp;gt; lrt.observed) #pvalue&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The proportion of bootstrapped LRTs more extreme that the observed LRT is the p-value. &lt;/p&gt;&#xA;" OwnerUserId="3432" LastEditorUserId="3432" LastEditorDisplayName="" LastEditDate="2011-03-17T20:50:30.897" LastActivityDate="2011-03-17T20:50:30.897" CommentCount="5" />
  <row Id="8431" PostTypeId="2" ParentId="8392" CreationDate="2011-03-17T21:11:16.093" Score="1" ViewCount="" Body="&lt;p&gt;Purely from a machine learning perspective, I think it is possible to do the inference you want from this data. How reliable it will be will depend on how much data you have (the more the better) and on usage patterns (do students mostly do course-related or course-unrelated stuff on these computers). However, this is not a trivial task. In particular, I don't think there is a single R function which you can just run and it will give you the results. So it is worth exploring other options (particularly social engineering) to get the exact data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you do go for a statistical analysis, here are a few ideas to try.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One option is to use regression. Suppose you are interested in just one program and which courses it is used in. Define the following variables: $s_i$ is 1 if in a given term student $i$ used the program, 0 otherwise (you can potentially replace this with how often they used it); $x_{ij}$ is 1 if student $i$ is enrolled in class $j$ and 0 otherwise; $\beta_j$ is 1 if class $j$ uses the program and 0 otherwise. You can build a logistic regression $$logit(s_i) = \sum_j \beta_j x_{ij}.$$ The variables $s_i$ and $x_{ij}$ are known, and the regression will give you the $\beta$'s. If $\beta_j$ is high, it is likely the course uses the program.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another option is to apply the author-topic model with some modifications. If your dataset is large enough, it can work well. The advantage, compared to logistic regression, is that it is more versatile, so it's easy to incorporate additional information that you have (such as that all students will use standard software like firefox, and there will be a small number of programs that only 1-2 students use). The disadvantage is that it is harder to implement. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't think PCA or LDA would work well here. Your clusters would be the sets of programs students in a particular class use. The problem is that two classes may overlap in what they use, in which case that component will not be picked up by PCA/LDA.&lt;/p&gt;&#xA;" OwnerUserId="3369" LastActivityDate="2011-03-17T21:11:16.093" />
  <row Id="8432" PostTypeId="2" ParentId="8392" CreationDate="2011-03-17T22:52:13.613" Score="0" ViewCount="" Body="&lt;p&gt;Considering you observe co-occurences of discrete variables, I would do the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;num_programs = 3&#xA;num_students = 10&#xA;num_courses = 5&#xA;&#xA;# How long a given student uses a given program&#xA;random_exponential = rexp(num_programs*num_students)&#xA;P = matrix(random_exponential, nrow = num_programs)&#xA;# Given a student, probability he/she uses the program&#xA;P = t(t(P)/apply(P, 2, sum))&#xA;&#xA;# Which student attends which course&#xA;random_bernoulli = sample(c(0,1), num_courses*num_students, replace = TRUE)&#xA;C = matrix(random_bernoulli, nrow = num_courses)&#xA;# Given a course, probability a student attends this course&#xA;C = C/apply(C, 1, sum)&#xA;&#xA;# Given a course, probability a program is used&#xA;solution = P%*%t(C)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This yields:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; random_exponential&#xA; [1] 1.19398141 0.67134008 0.09263323 0.38315782 1.54479115 0.32107116 1.33303520 1.33895914&#xA; [9] 0.66164138 0.48749235 0.44726465 1.17094483 1.08530260 3.17384173 0.81148181 0.95276800&#xA;[17] 1.09086805 0.04151037 0.03336502 1.14701888 0.37769900 0.56155261 0.49287836 1.10332629&#xA;[25] 2.54373887 2.32393698 0.44819034 3.46339510 1.24639024 1.36241789&#xA;&amp;gt; P&#xA;           [,1]      [,2]      [,3]      [,4]      [,5]       [,6]       [,7]      [,8]&#xA;[1,] 0.60981053 0.1703666 0.3998743 0.2315106 0.2140372 0.45693098 0.02141415 0.2602483&#xA;[2,] 0.34287825 0.6868730 0.4016513 0.2124064 0.6259270 0.52316137 0.73617320 0.2284216&#xA;[3,] 0.04731122 0.1427605 0.1984744 0.5560829 0.1600358 0.01990765 0.24241265 0.5113301&#xA;           [,9]     [,10]&#xA;[1,] 0.47851823 0.5703688&#xA;[2,] 0.43716996 0.2052616&#xA;[3,] 0.08431182 0.2243696&#xA;&amp;gt; random_bernoulli&#xA; [1] 1 0 0 1 1 0 0 0 0 0 1 0 0 0 1 0 1 1 1 1 1 0 0 0 0 0 1 1 0 0 0 1 0 0 1 1 0 0 0 1 1 1 0 0&#xA;[45] 1 0 0 0 0 0&#xA;&amp;gt; C &#xA;          [,1] [,2]      [,3]      [,4] [,5] [,6]      [,7]      [,8]      [,9] [,10]&#xA;[1,] 0.2000000    0 0.2000000 0.0000000  0.2 0.00 0.0000000 0.2000000 0.2000000     0&#xA;[2,] 0.0000000    0 0.0000000 0.2500000  0.0 0.25 0.2500000 0.0000000 0.2500000     0&#xA;[3,] 0.0000000    0 0.0000000 0.5000000  0.0 0.50 0.0000000 0.0000000 0.0000000     0&#xA;[4,] 0.5000000    0 0.0000000 0.5000000  0.0 0.00 0.0000000 0.0000000 0.0000000     0&#xA;[5,] 0.1666667    0 0.1666667 0.1666667  0.0 0.00 0.1666667 0.1666667 0.1666667     0&#xA;&amp;gt; solution&#xA;          [,1]      [,2]      [,3]      [,4]      [,5]&#xA;[1,] 0.3924977 0.2970935 0.3442208 0.4206606 0.3335627&#xA;[2,] 0.4072096 0.4772277 0.3677839 0.2776423 0.3931168&#xA;[3,] 0.2002927 0.2256788 0.2879953 0.3016971 0.2733205&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1351" LastEditorUserId="1351" LastEditorDisplayName="" LastEditDate="2011-03-19T09:46:11.220" LastActivityDate="2011-03-19T09:46:11.220" />
  <row Id="8433" PostTypeId="1" CreationDate="2011-03-17T23:33:22.827" Score="4" ViewCount="86" Body="&lt;p&gt;I have data on 70,000 students, nested in 120 schools.  I'm starting with fixed effects for the schools, but at some point I might start letting intercepts and slopes vary.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some key predictors (e.g. gpa, test scores) have non-linear relationships with some outcomes.  I definitely want to interact these predictors with categorical variables, and maybe with continuous ones as well.  Natural cubic splines might be the right approach, but I haven't seen any examples with interactions or nested data, so I could use some pointers.  These non-linear relationships aren't the sole focus of my research, so I don't want to focus on them to the neglect of other aspects of the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd appreciate suggestions of R packages, examples of related work, and general advice on how to approach this.  Thanks!&lt;/p&gt;&#xA;" OwnerUserId="3748" LastActivityDate="2011-03-18T02:30:58.790" Title="Interactions between non-linear predictors" Tags="&lt;r&gt;&lt;mixed-model&gt;&lt;interaction&gt;&lt;splines&gt;&lt;lmer&gt;" AnswerCount="1" />
  <row Id="8434" PostTypeId="2" ParentId="8419" CreationDate="2011-03-17T23:34:51.363" Score="5" ViewCount="" Body="&lt;p&gt;I would have thought the &quot;best&quot; (most intuitive) estimate of error would be the probability that the classification is incorrect.  Or alternatively/equivalently the odds against the class.  Using the &lt;a href=&quot;http://en.wikipedia.org/wiki/Naive_Bayes_classifier&quot; rel=&quot;nofollow&quot;&gt;wikipedia page&lt;/a&gt;, you classify as Male or female.  I would have thought the estimate of accuracy should be:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$O(Male|evidence)=\frac{P(Male|evidence)}{P(Female|evidence)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So you would report the data in terms of &quot;the evidence gives odds O:1 in favour of this classification&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a problem with more than 2 classes, you should report the &quot;worst&quot; odds ratio.  That is given classes $C_{i}\;\;(i=1,\dots,R+1)$, one (and only one) of which is assumed to be true.  Suppose you classify an observation as $C_{R+1}$, then you would report odds of:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$O(C_{R+1}|evidence)=\frac{P(C_{R+1}|evidence)}{Max_{i\in[1,\dots,R]}P(C_{i}|evidence)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So you would report the data in terms of &quot;the evidence gives odds O:1 in favour of this classification against the next best alternative&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT/UPDATE&lt;/strong&gt;: in response to the new part of the question, to put some numbers into the calculation.  Using the first 15 observations to &quot;train&quot; the classifier.  Because you are dealing with normal distribution, you only need the sufficient statistics, which in this case are:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ \begin{pmatrix}&#xD;&#xA;  \hat{\mu}_{1} &amp;amp; \hat{\mu}_{2} \\&#xD;&#xA;  \hat{\nu}_{1} &amp;amp; \hat{\nu}_{2} \\&#xD;&#xA;  \hat{\sigma}_{1} &amp;amp; \hat{\sigma}_{2}  \\&#xD;&#xA;  \hat{\tau}_{1} &amp;amp; \hat{\tau}_{2}&#xD;&#xA; \end{pmatrix}&#xD;&#xA;=&#xD;&#xA; \begin{pmatrix}&#xD;&#xA;  0.3798 &amp;amp; 0.6275 \\&#xD;&#xA;  -0.1449 &amp;amp; 1.6748 \\&#xD;&#xA;  0.8367  &amp;amp; 0.8685  \\&#xD;&#xA;  0.8200 &amp;amp; 0.5451&#xD;&#xA; \end{pmatrix}&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where the subscript denotes the class.  $\mu$ and $\sigma$ denote the mean and standard deviation of the first attribute.  $\nu$ and $\tau$ denote the mean and standard deviation of the second attribute.  You could include correlation, but it is &quot;safer&quot; not to unless you know correlations actually exist.  As these are just numbers to me, I have no reason to suppose them to be dependent.  So I will not constrain them by forcing a dependence assumption.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now you need a &lt;em&gt;decision rule&lt;/em&gt; in order to classify an observation into 1 class or the other.  The one I was suggesting was to use the odds ratio.  So we need to calculate the probability of belonging to class 1, given the training data (denoted by D), the prior information (denoted by I), and the sample to test (denoted by y):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p(C_{1}|y,D,I)=\frac{p(C_{1}|D,I)p(y|C_{1},D,I)}{p(y|D,I)}&#xD;&#xA;\rightarrow O(C_{1}|y,D,I)=\frac{p(C_{1}|D,I)p(y|C_{1},D,I)}{p(C_{2}|D,I)p(y|C_{2},D,I)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $p(C_{1}|D,I) = \frac{8}{15}$ assuming complete initial ignorance (because I am ignorant prior to seeing the data).  If you knew it was possible for both categories to occur prior to observing the training data, then the probability would be given by the rule of succession $\frac{9}{17}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Pr(y|C_{1},D,I)$ is the posterior predictive distribution for class 1, given by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p(y|C_{1},D,I)=\int p(y|\mu_{1},\nu_{1},\sigma_{1},\tau_{1},I) p(\mu_{1},\nu_{1},\sigma_{1},\tau_{1}|D,I)d\mu_{1}d\nu_{1}d\sigma_{1}d\tau_{1}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I will just give the posterior assuming complete ignorance, it is not hard to derive, you use a prior $p(\mu_{1},\nu_{1},\sigma_{1},\tau_{1}|I)\propto\frac{1}{\sigma_{1}\tau_{1}}$ and do the necessary integrals.  It is a product of two student t densities (denoted by $St(x|\mu,\sigma,df)$)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p(y|C_{1},D,I)=St(y_{1}|\hat{\mu}_{1},\hat{\sigma}_{1}\sqrt{\frac{8+1}{8-1}},8-1)St(y_{2}|\hat{\nu}_{1},\hat{\tau}_{1}\sqrt{\frac{8+1}{8-1}},8-1)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $y_j$ is the attribute j value of the new data point.  This should make it fairly obvious how it would generalise to more than two attributes.  Similarly, we have $p(C_{2}|D,I) = \frac{7}{15}$ assuming ignorance or $\frac{8}{17}$ using the rule of succession.  The posterior predictive is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p(y|C_{2},D,I)=St(y_{1}|\hat{\mu}_{2},\hat{\sigma}_{2}\sqrt{\frac{7+1}{7-1}},7-1)St(y_{2}|\hat{\nu}_{2},\hat{\tau}_{2}\sqrt{\frac{7+1}{7-1}},7-1)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And so the final odds ratio is given by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$O(C_{1}|y,D,I)=\frac{8}{7}&#xD;&#xA;\times&#xD;&#xA;\frac{St(y_{1}|\hat{\mu}_{1},\hat{\sigma}_{1}\sqrt{\frac{9}{7}},7)}{St(y_{1}|\hat{\mu}_{2},\hat{\sigma}_{2}\sqrt{\frac{8}{6}},6)}&#xD;&#xA;\times&#xD;&#xA;\frac{St(y_{2}|\hat{\nu}_{1},\hat{\tau}_{1}\sqrt{\frac{9}{7}},7)}{St(y_{2}|\hat{\nu}_{2},\hat{\tau}_{2}\sqrt{\frac{8}{6}},6)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think you will agree that this number is sensible by any criteria.  It &quot;goes in all the right directions&quot;, and it appropriately accounts for the uncertainty in estimating the parameters of the model.  inserting in these densities gives:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{8}{7}&#xD;&#xA;\times\frac{\hat{\sigma}_{2}\hat{\tau}_{2}}{\hat{\sigma}_{1}\hat{\tau}_{1}}&#xD;&#xA;\times\left[\frac{\frac{\Gamma(4)}{\Gamma(\frac{7}{2})\sqrt{7}}}{\frac{\Gamma(\frac{7}{2})}{\Gamma(3)\sqrt{6}}}\right]^2&#xD;&#xA;&#xD;&#xA;\frac{&#xD;&#xA;\left[1+\frac{1}{8}\left(\frac{y_{1}-\hat{\mu}_{2}}{\hat{\sigma}_{2}}\right)^2&#xD;&#xA;\right]^{\frac{7}{2}}&#xD;&#xA;\left[1+\frac{1}{8}\left(\frac{y_{2}-\hat{\nu}_{2}}{\hat{\tau}_{2}}\right)^2&#xD;&#xA;\right]^{\frac{7}{2}}&#xD;&#xA;}{&#xD;&#xA;\left[1+\frac{1}{9}\left(\frac{y_{1}-\hat{\mu}_{1}}{\hat{\sigma}_{1}}\right)^2&#xD;&#xA;\right]^{\frac{8}{2}}&#xD;&#xA;\left[1+\frac{1}{9}\left(\frac{y_{2}-\hat{\nu}_{1}}{\hat{\tau}_{1}}\right)^2&#xD;&#xA;\right]^{\frac{8}{2}}&#xD;&#xA;}&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now in order to make a decision, you need to think about the consequences of making a wrong classification.  is it worse to classify a $1$ as a $2$ compared to classifying a $2$ as a $1$?  If not then the cut-off is simply $O(C_{1}|y,D,I)&amp;gt;1$ are classed as $1$, otherwise as $2$.  the cut-off will slide up or down depending on &quot;which is more important to get right&quot;.  Although for this particular example, the classifier is so good, you hardly need to bother about this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The table below shows the odds for each of the testing observations.  Interestingly, attribute 2 is what is driving most of the changes in odds - attribute 1 does not appear to be as useful.  In fact you can see that attribute 1 actually introduces more uncertainty with identifying group 2.  This is obvious when you note that the mean and variance of attribute 1 are basically the same for each group (but mean and variance of attribute 2 is quite different between two groups):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;\begin{array}{c|c}&#xD;&#xA;  y_{1} &amp;amp; y_{2} &amp;amp; O(C_{1}|y_{1},y_{2},D,I) &amp;amp; O(C_{1}|y_{2},D,I) &amp;amp; \text{True Class} \\&#xD;&#xA;  \hline&#xD;&#xA;&#xD;&#xA;-0.4049	&amp;amp; -0.3981 &amp;amp; 47.0 &amp;amp;37.0 &amp;amp; 1 \\&#xD;&#xA;-0.0913	&amp;amp; 2.2094 &amp;amp; 0.109 &amp;amp; 0.089 &amp;amp; 2 \\&#xD;&#xA;0.3376	&amp;amp; -1.0467 &amp;amp; 102.4 &amp;amp; 93.5 &amp;amp; 1 \\&#xD;&#xA;0.3455	&amp;amp; 2.496 &amp;amp; 0.104 &amp;amp; 0.095 &amp;amp; 2 \\&#xD;&#xA;0.3232	&amp;amp; -0.5614 &amp;amp; 54.6 &amp;amp; 49.7 &amp;amp; 1&#xD;&#xA;&#xD;&#xA; \end{array}&#xD;&#xA;$$&lt;/p&gt;&#xA;" OwnerUserId="2392" LastEditorUserId="2392" LastEditorDisplayName="" LastEditDate="2011-03-19T01:49:46.053" LastActivityDate="2011-03-19T01:49:46.053" CommentCount="3" />
  <row Id="8435" PostTypeId="1" CreationDate="2011-03-18T01:21:08.357" Score="0" ViewCount="131" Body="&lt;p&gt;When to use multivariate logistic regression versus generalized linear mixed-effects models? What is the difference between the two?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit (in response to comments):&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was hoping to see multivariate as more than one covariate. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My problemset is analysis for a longitudinal prospective cohort study where im measuring vitamin K intake in relation to incidence of calcification. vitamin K intake is primarily bein assessed by food questionnaires but im also looking at plasma vitamin K levels. the exposure variable and the outcome variable are ordinal/categorical. on the other hand, outcome also has more than two categories. &#xA;i will be using a multivariate mixed model. considering i know my covariates, it should be suffice to use a mixed regression model?&#xA;Thank you .&lt;/p&gt;&#xA;" OwnerUserId="3771" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-03-29T06:32:50.853" LastActivityDate="2011-03-29T06:32:50.853" Title="Generalized Linear Mixed Effects" Tags="&lt;regression&gt;&lt;mixed-model&gt;&lt;longitudinal&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="8436" PostTypeId="1" AcceptedAnswerId="8440" CreationDate="2011-03-18T02:30:48.143" Score="6" ViewCount="122" Body="&lt;p&gt;I have some data; it's a proportion $y$ of some stuff  relative to everything, so it's bounded between 0 and 1 by definition. The proportion changes over time. Besides fairly high variance there is a step-like change about the middle of the time period; the step isn't very large, but it's there, and it happens pretty fast relative to the whole time period. So I have an S-curve, and I want to (and was told to) do logistic regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But: (1) this is not monotone, not even close, because of the high variance. (2) it does not go from 0 to 1. Rather, it goes from about 0.2 to about 0.8, if you look at the means. So it would seem that the right thing to do will be to fit something like $a\phi(x)+b$, where $\phi$ is the usual logistic S-curve, so we have 4 parameters altogether. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What bothers me is that I've never seen an example of logistic regression used like that (I'll admit to not seeing too many). It's not that I'm not sure how to  implement this -- I'm pretty confident I can figure that out, although specific pointers will be appreciated -- but I am afraid that the assumption that the data is bounded by the limits of the S-curve is important for all the estimates afterward, like goodness of fit, significance of the step, confidence intervals, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Are my fears justified?&lt;/li&gt;&#xA;&lt;li&gt;Can you point me to a relevant example in the literature? Not necessarily statistical literature, if some biologists (or whoever) use it, that's fine. Or even just someone mentioning this possibility.&lt;/li&gt;&#xA;&lt;li&gt;If there are indeed problems with this approach, what are the alternatives?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update:&lt;/strong&gt; Well, I was so ignorant that I could not even ask the questions properly. I was looking for multinomial (aka polytomous) regression.&lt;/p&gt;&#xA;" OwnerUserId="2658" LastEditorUserId="2658" LastEditorDisplayName="" LastEditDate="2011-03-19T17:35:35.033" LastActivityDate="2011-03-19T17:35:35.033" Title="Logistic regression for bounds different from 0 and 1" Tags="&lt;logistic-regression&gt;" AnswerCount="3" CommentCount="2" FavoriteCount="1" />
  <row Id="8437" PostTypeId="2" ParentId="8433" CreationDate="2011-03-18T02:30:58.790" Score="5" ViewCount="" Body="&lt;p&gt;You could try generalized additive mixed models, handily implemented in the &lt;a href=&quot;http://cran.r-project.org/web/packages/gamm4/index.html&quot; rel=&quot;nofollow&quot;&gt;gamm4&lt;/a&gt; package. The way I've used them, you can do something like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;fit1 = gamm4(&#xA;    formula = V1 ~ V2 + s(V3)&#xA;    , random = ~ (1|V4)&#xA;)&#xA;fit2 = gamm4(&#xA;    formula = V1 ~ V2 + s(V3,by=V2)&#xA;    , random = ~ (1|V4)&#xA;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;fit1&lt;/code&gt; seeks to predict V1 using V4 as a random effect and V2 and V3 as fixed effects, but where V3 is spline-smoothed. &lt;code&gt;fit2&lt;/code&gt; seeks the same, except with the addition of permitting the smooth of V3 to vary within levels of V2, thus implementing an interaction. Comparison of &lt;code&gt;fit1&lt;/code&gt; to &lt;code&gt;fit2&lt;/code&gt; evaluates the necessity of permitting the interaction.&lt;/p&gt;&#xA;" OwnerUserId="364" LastActivityDate="2011-03-18T02:30:58.790" CommentCount="2" />
  <row Id="8438" PostTypeId="2" ParentId="8382" CreationDate="2011-03-18T02:54:43.513" Score="0" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Truncation_%28statistics%29&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Truncation_%28statistics%29&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is not much help, but at least it gives the correct buzzword (truncated, not quantized; quantization is not your problem) and one pointer to a paper. This should do as a starting point for further search.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Oh, and Winsorized mean is the exact opposite from what you want. &lt;/p&gt;&#xA;" OwnerUserId="2658" LastActivityDate="2011-03-18T02:54:43.513" CommentCount="3" />
  <row Id="8439" PostTypeId="1" CreationDate="2011-03-18T03:00:31.760" Score="2" ViewCount="82" Body="&lt;p&gt;I would like to validate a commonly used questionnaire for internet-based administration.  What statistical tests should I be using?&lt;/p&gt;&#xA;" OwnerUserId="3772" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-03-18T08:16:00.613" LastActivityDate="2011-03-18T11:38:41.063" Title="Validating a paper questionnaire in a web-based format" Tags="&lt;validation&gt;&lt;questionnaires&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="8440" PostTypeId="2" ParentId="8436" CreationDate="2011-03-18T05:48:45.150" Score="2" ViewCount="" Body="&lt;p&gt;To begin with, I think we have to distinguish between &lt;a href=&quot;http://en.wikipedia.org/wiki/Logistic_regression&quot; rel=&quot;nofollow&quot;&gt;logistic regression&lt;/a&gt; and (generalized) &lt;a href=&quot;http://en.wikipedia.org/wiki/Generalised_logistic_curve&quot; rel=&quot;nofollow&quot;&gt;logistic function&lt;/a&gt;. Though the latter may be viewed as a separate case of a the former taking the time as the only explanatory variable. It is then straightforward to see that fitted process will go by $S$ shaped path that goes to its upper (or, probably, lower) limit when $t \rightarrow \infty$. Therefore moving along the $S$ curve is the influence of over covariates (not time) that are changing up and down with time (in consumption structures these are income, tastes, prices, etc.). So there could be jumps or whatever, because nobody restricts the linear regression to go only up or only down to the $S$ curve's bounds.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since you are working with structures $0$ and $1$ are natural limits. You can never be sure that any other bounds won't be hit higher or lower in the future, when your conclusions are based only on historical data analysis, and arguing that the process never did so is not an appropriate reasoning. Therefore your fears are not justified, logistic regression (but not fitting the logistic curve! that comes as a solution of &lt;em&gt;deterministic&lt;/em&gt; differential equation!) will work just fine here. Pay attention to the fact that there could be several categories that some up to 1, so you need multinomial logit model to fit the structure in this case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Among the alternatives there could be any model that could be applied to &lt;a href=&quot;http://en.wikipedia.org/wiki/Discrete_choice&quot; rel=&quot;nofollow&quot;&gt;discrete choice&lt;/a&gt;. Commonly used candidates are probit and logit models. Even if you think that &lt;em&gt;there is no decision in my model&lt;/em&gt; actually all structures in the world are the result of decision processes solved either by humans, nature or the aliens ^_^. &lt;/p&gt;&#xA;" OwnerUserId="2645" LastEditorUserId="2645" LastEditorDisplayName="" LastEditDate="2011-03-18T09:15:16.640" LastActivityDate="2011-03-18T09:15:16.640" CommentCount="1" />
  <row Id="8441" PostTypeId="2" ParentId="8439" CreationDate="2011-03-18T08:13:31.180" Score="5" ViewCount="" Body="&lt;p&gt;The following are two complementary strategies that you could adopt.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;1. Conduct your own empirical validation:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;A simple strategy would be to run some form of experiment.&#xA;Randomly assign half of your participants to web based completion and the other half to paper based completion.&#xA;Then, examine whether statistical properties of the questionnaire are the same across administration formats; e.g., item mean, item standard deviation, intercorrelations between items, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may find structural equation modelling approaches useful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You'll also need a sufficiently large sample size because you are effectively trying to prove the null hypothesis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other designs would also be possible. For example, you could get participants to complete both versions and counterbalance for order. This mixed design would have more power for a given sample size but would suffer the potential confound of carry-over effects.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;2. Rely on existing research literature&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;There's been a lot of research looking at consistency of web based and paper based formats.&#xA;Over the while I've read research on online-paper consistency for personality tests, ability tests, questionnaires, and more.&#xA;I don't have specific references on hand, but you should do a literature review to gather  relevant evidence from analogous existing studies.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, you'd want to think about the reasons why you would expect differences. For example, I've seen cases where data collected online is more noisy than face-to-face data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You also need to consider what you mean by &quot;questionnaire&quot;. A questionnaire can be mail, face-to-face, self-administered but with an investigator in the room, and so on.&lt;/p&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-03-18T11:38:41.063" LastActivityDate="2011-03-18T11:38:41.063" />
  <row Id="8442" PostTypeId="1" CreationDate="2011-03-18T08:46:16.270" Score="3" ViewCount="59" Body="&lt;p&gt;Hello all,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My goal is to create CI for the CART prediction of new_x&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider the following code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(rpart)    &#xA;set.seed(147830)&#xA;n &amp;lt;- 100&#xA;x1 &amp;lt;- runif(n)&#xA;x2 &amp;lt;- rnorm(n)&#xA;y &amp;lt;- x1 + 2*x2 + rnorm(n, 0, .5)&#xA;DAT &amp;lt;- data.frame(y,x1,x2)&#xA;fit &amp;lt;- rpart(y ~ x1 + x2, data = DAT)&#xA;new_x &amp;lt;- data.frame(x1 = .5 , x2 = .25)&#xA;predict(fit, newdata = new_x) # 1.353142 # &quot;should&quot; be ~1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The only way I can think of at the moment is to bootstrap on x1/x2 from DAT, build many prediction models with them, for each predict new_x, and then use the lower/upper percentiles.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there some other way to do it?  What are the advantages/disadvantages of it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="253" LastActivityDate="2011-03-18T09:58:04.173" Title="How to produce a CI for a value predicted in CART?" Tags="&lt;r&gt;&lt;predictive-models&gt;&lt;decision-trees&gt;&lt;cart&gt;&lt;rpart&gt;" AnswerCount="1" />
  <row Id="8443" PostTypeId="2" ParentId="7236" CreationDate="2011-03-18T09:21:26.243" Score="0" ViewCount="" Body="&lt;p&gt;What I would do is provide &lt;code&gt;prob&lt;/code&gt; argument with weights for each data point based on number of levels in your variable. Example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df &amp;lt;- data.frame(oks = sample(100),&#xA;        grp = c(rep(&quot;trt1&quot;, times = 30), rep(&quot;trt2&quot;, times = 70)))&#xA;&#xA;&amp;gt; head(df)&#xA;  oks  grp&#xA;1  40 trt1&#xA;2  29 trt1&#xA;3  12 trt1&#xA;4  25 trt1&#xA;5  19 trt1&#xA;6  45 trt1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Obviously:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; (df.prob &amp;lt;- table(df$grp))&#xA;&#xA;trt1 trt2 &#xA;  30   70 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You pass a vector of probabilities to &lt;code&gt;sample&lt;/code&gt;. You can sort your &lt;code&gt;data.frame&lt;/code&gt; by your desired variable (and use the adaptation of solution provided here), or you could assign weights to individual rows based on the level of the treatment (not presented here, but shouldn't be too hard to recode).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df[sample(x = df$oks, size = 30, prob = rep(df.prob/nrow(df), df.prob)), ] # / by nrow(df) to get appropriate weight per treatment&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is the approximate ratio you're looking for, right?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; table(df[sample(x = df$oks, size = 30, prob = rep(df.prob/nrow(df), df.prob)), ]$grp)&#xA;&#xA;trt1 trt2 &#xA;  12   18 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="144" LastActivityDate="2011-03-18T09:21:26.243" />
  <row Id="8444" PostTypeId="2" ParentId="8442" CreationDate="2011-03-18T09:58:04.173" Score="1" ViewCount="" Body="&lt;p&gt;Couldn't you use the distribution of the observed &lt;code&gt;y&lt;/code&gt; in the leafs/terminal nodes to get an idea about the precision of the estimated mean for new observations falling into that leaf? For CIs with large coverage you'd probably need a lot of data / a small tree so that each terminal node contains a lot of observations, though.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;i.e. do s.th like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;quantile(DAT$y[predict(fit) ==  predict(fit, newdata = new_x)], &#xA;           probs=c(.025, .05, .1, .9, .95, .975))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;maybe with a more elegant method than what I used to find out which of the original &lt;code&gt;y&lt;/code&gt; fall into the same leaf as the observation you are forecasting. &lt;/p&gt;&#xA;" OwnerUserId="1979" LastActivityDate="2011-03-18T09:58:04.173" />
  <row Id="8445" PostTypeId="2" ParentId="8436" CreationDate="2011-03-18T09:58:07.093" Score="1" ViewCount="" Body="&lt;p&gt;Logistic regression is something that is typically used for classification and not for regression. (Despite of the name). So if you do not actually tackle a classification problem it strikes me as odd to use it - mostly because of the use of the cross entropy error function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively, I would use a linear regression with basis functions which have the desired S shape and sum of squares error.&lt;/p&gt;&#xA;" OwnerUserId="2860" LastActivityDate="2011-03-18T09:58:07.093" />
  <row Id="8446" PostTypeId="1" CreationDate="2011-03-18T10:03:48.483" Score="3" ViewCount="168" Body="&lt;p&gt;Hi to all,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My supervisor asked me to find out which distribution represents a particular situation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a VoIP generator that generates calls &quot;uniformly&quot; distributed between callers. This means that the volume per caller distribution is &quot;almost&quot; uniformly distributed between a minimum and maximum. So by running a test with 10000 users and a min value equal to 30 calls per week and a max value equal to 90 calls/week i obtain that not all the users respect this limits: we have some users that generate &amp;lt;30 calls and some other users that generate &gt;90 calls. It is clear that the obtained distribution is not uniform.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The situation is this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/0KX7A.png&quot; width=&quot;500&quot; height=&quot;450&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;He said that i have to perform a sort of numerical process in order to find some formulas that could define this distribution. Initially, as wrote before,we wanted to obtain a uniform (min,max) distribution (the green area in the figure) but this is not the case as proved with chi-square test. &#xA;Moreover the curve in figure is not symmetric, the probability of call generation below 30 call/week and greater that 90 call/week are not identical (it is high for 90calls/week). The variability of the number of generated call increases with the increasing call generation rates. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Actually implementation of this distribution is nothing but assigning different call rates in a range for users in domain which indicates implementation of several delta functions. As the call rates increases the variability of the generated calls also increases with the average call rate and this leads to the asymmetric behavior of the curve.&quot; [cit. from the Voipgenerator documentation]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Someone can help me?I think that now i cannot use Q-Q plot because i don't know which theoretical distribution i have to use in order to compare it with my empirical data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sorry if I have stressed with a similar problem a few weeks ago, but initially we thought we could change the implementation, but now we cannot. Hence i have to discover the type of the distribution i obtained and i don't know how can i do this.&lt;/p&gt;&#xA;" OwnerUserId="3342" LastEditorUserId="3342" LastEditorDisplayName="" LastEditDate="2011-03-19T11:32:24.970" LastActivityDate="2011-03-19T11:32:24.970" Title="What kind of distribution is this?" Tags="&lt;distributions&gt;&lt;test&gt;&lt;numerics&gt;" AnswerCount="2" CommentCount="9" FavoriteCount="0" />
  <row Id="8447" PostTypeId="1" CreationDate="2011-03-18T10:55:33.273" Score="0" ViewCount="67" Body="&lt;p&gt;I want to graphically show the change in relative risk along with confidence limits over 17 years. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can I use a forest plot without meta-analysis because it will have the added advantage to tabulate Z-values and p-values also? Here I want to show the progressive change in the relative risk and NOT meta-analysis. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or is there a better alternative for the graphic presentation? &lt;/p&gt;&#xA;" OwnerUserId="2956" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-18T16:01:58.567" LastActivityDate="2011-03-18T16:01:58.567" Title="Visualizing change in risk ratio along with confidence limits" Tags="&lt;confidence-interval&gt;&lt;relative-risk&gt;&lt;graphical-model&gt;" CommentCount="5" />
  <row Id="8448" PostTypeId="2" ParentId="8405" CreationDate="2011-03-18T11:12:23.097" Score="2" ViewCount="" Body="&lt;p&gt;@SabreWolfy: I've edited my answer to add an additional factor by binning data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(Hmisc)&#xA;library(doBy)&#xA;set.seed(123)&#xA;&#xA;A=rnorm(100, mean=50, sd=20)&#xA;&#xA;bin.data&amp;lt;-data.frame(bins.A=cut2(A, g=4),G=gl(2,50), A)&#xA;&#xA;summaryBy(A~bins.A+G,data=bin.data,FUN=each(length,mean,median))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;will give output for age:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;        bins.A G A.length   A.mean A.median&#xA;1 [ 3.82,40.2) 1       14 29.44193 29.06172&#xA;2 [ 3.82,40.2) 2       11 29.41250 29.62849&#xA;3 [40.18,51.4) 1       12 44.04858 43.98966&#xA;4 [40.18,51.4) 2       13 45.93984 45.48458&#xA;5 [51.41,64.0) 1       11 57.00405 58.01543&#xA;6 [51.41,64.0) 2       14 57.24880 57.11421&#xA;7 [64.03,93.7] 1       13 74.35300 74.15924&#xA;8 [64.03,93.7] 2       12 77.01414 75.09460&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;for each combination of the bin factor and the grouping factor.&lt;/p&gt;&#xA;" OwnerUserId="3774" LastEditorUserId="3774" LastEditorDisplayName="" LastEditDate="2011-03-18T18:19:36.817" LastActivityDate="2011-03-18T18:19:36.817" CommentCount="7" />
  <row Id="8449" PostTypeId="2" ParentId="8446" CreationDate="2011-03-18T11:24:22.680" Score="1" ViewCount="" Body="&lt;p&gt;Try Q-Q plots (or P-P plots, only they are somewhat less widely used) of your empirical distribution against each of hypothetical type of distribution. The parameters for the latter are usually deduced by software from your empirical distribution although you could input parameter values you wish to check.&lt;/p&gt;&#xA;" OwnerUserId="3277" LastActivityDate="2011-03-18T11:24:22.680" CommentCount="1" />
  <row Id="8450" PostTypeId="2" ParentId="8420" CreationDate="2011-03-18T11:48:32.543" Score="1" ViewCount="" Body="&lt;p&gt;Although I haven't seen international examples, I think &lt;a href=&quot;http://www.instantatlas.com/&quot; rel=&quot;nofollow&quot;&gt;InstantAtlas&lt;/a&gt; offers quite a lot of nice features that could be useful in your case:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/C5SA5.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.sacmeq.org/statplanet/&quot; rel=&quot;nofollow&quot;&gt;StatPlanet&lt;/a&gt; might be another (free) option:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/zlRkJ.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="22" LastActivityDate="2011-03-18T11:48:32.543" />
  <row Id="8451" PostTypeId="2" ParentId="8436" CreationDate="2011-03-18T13:04:44.530" Score="3" ViewCount="" Body="&lt;p&gt;You are looking for the wrong keywords. Logistic regression is for 0-1 outcomes, where the &lt;em&gt;probability&lt;/em&gt; of being 1 is modeled with an S-shaped (logistic) function, not the actual data points themselves. Looking for nonlinear regression, specifically the four-parameter logistic model: $y = A + (B-A)/(1+\exp(-(a+bx))) + \epsilon$ (it can show up with other parametrizations). You fill find lots of ELISA related stuff if you search for this model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is specialized software for fitting this, and of course the major statistical packages also can handle it. Unfortunately, if there is a lot of variability, estimation might be difficult, and the fitting process might &quot;not converge&quot;. &lt;/p&gt;&#xA;" OwnerUserId="279" LastActivityDate="2011-03-18T13:04:44.530" />
  <row Id="8452" PostTypeId="2" ParentId="8435" CreationDate="2011-03-18T13:08:03.853" Score="2" ViewCount="" Body="&lt;p&gt;This depends on what you mean by &quot;multivariate&quot;.  Doing a quick google search, the term seems to refer to &quot;more than 1 co variate&quot;.  So the model equation is for &quot;ordinary logistic regression&quot; you have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$log(\frac{\pi}{1-\pi})=\beta_0 + \beta_1 X_1$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For &quot;multivariate logistic regression&quot; you have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$logit(\frac{\pi}{1-\pi})=\beta_0 + \beta_1 X_1 + \beta_2 X_2 +\dots+ \beta_p X_p=X\beta$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $X\beta$ is the &quot;matrix version&quot; - just for more compact notation.  But as @onestop points out, the term &quot;multivariate&quot; could also refer to more than 1 probability.  The model equations are then:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$logit(\frac{\pi_1}{\pi_m})=X^{(1)}\beta^{(1)}$$&#xA;$$logit(\frac{\pi_2}{\pi_m})=X^{(2)}\beta^{(2)}$$&#xA;$$\dots$$&#xA;$$logit(\frac{\pi_{m-1}}{\pi_m})=X^{(m-1)}\beta^{(m-1)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where the brackets $(1),(2),..,(m-1)$ just indicate that the co variates and betas could be different in each model.  The &quot;multinomial&quot; simplifies back to the &quot;multivariate&quot; when $m=2$ (only 2 categories), and the &quot;multivariate&quot; simplifies back to the &quot;ordinary logistic regression&quot; when you have only 1 co variate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now these are all &quot;independence&quot; based models, in that given the covariates and betas, any one individual's probability tells you nothing about any others.  When you move from this framework, which is called &quot;generalised linear modelling&quot; or GLM in the literature, to the &quot;extra M&quot; of GLMM, the observations are now assumed to somehow depend on each other.  The easiest way I can see a reason for this happening is that you know a particular attribute will increase or decrease the probability of a certain group of observations, but you have not measured that covariate in your data (e.g. you are analyzing the incidence of cancer among people, but you do not know which people in your data set are smokers.  However have variables which could predict smoking status.).  The usual GLM model goes:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$DATA=MODEL+NOISE^{(1)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(NOTE: although the likelihood isn't linear for GLMs, the usual fisher-scoring IRLS or newton-rhapson algorithm means this is the effective model one is actually estimating - because it is based on a 1 term taylor series expansion &quot;the delta method&quot;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Whereas the GLMM goes:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$DATA=MODEL+MODEL(UNOBSERVED)+NOISE^{(2)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The unobserved model is sometimes written as $Z\gamma$.  So your model equation becomes:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$logit(\frac{\pi}{1-\pi})=X\beta+Z\gamma$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The vector $\gamma$ is often called a &quot;random effect&quot; (and is usually assumed to have a normal distribution) and the vector $\beta$ is called a &quot;fixed effect&quot; (although I don't particularly like this terminology).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And the idea is to &quot;decompose&quot; the noise into the &quot;correlated part&quot; and the &quot;independent part&quot;.  You fit the &quot;fixed effect&quot; part of the model - this is like the &quot;global model&quot;.  The GLMM basically allows you to recognise that there can be &quot;local effects&quot; in your data - systematic deviations from the &quot;global model&quot; for small groups, in a very similar way to ANOVA and ANCOVA.  In fact, all of the GLMMs I have fit, my $Z$ matrix has been a classical design matrix for ANOVA.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So basically what the GLMM does is make a compromise between the global model and the local model.  The compromise depends on the estimated variance for the &quot;random effect&quot; and on the estimate variance of the noise.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-03-18T13:08:03.853" />
  <row Id="8453" PostTypeId="1" CreationDate="2011-03-18T13:17:29.973" Score="0" ViewCount="90" Body="&lt;p&gt;Does ((R^2) * Slope) provide an estimation of cause and effect?  Eg, if R^2 of miles driven and coffee consumed is x% and the slope of the coffee-and-miles dataset is y (miles/coffee), can you say &quot;z of each cup of coffee drunk is related to the number of miles driven&quot;?&lt;/p&gt;&#xA;" OwnerUserId="3776" LastActivityDate="2011-03-18T14:17:10.117" Title="Can you use r^2 and regression to estimate cause and effect?" Tags="&lt;regression&gt;&lt;correlation&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="8454" PostTypeId="1" CreationDate="2011-03-18T13:53:10.767" Score="0" ViewCount="47" Body="&lt;pre&gt;&lt;code&gt;I just read the statement&quot;Confounding variables are unknown in nature&quot;. Is that correct? &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If it is correct, Why confounding variables are unknown in Nature?&lt;/p&gt;&#xA;" OwnerUserId="2956" LastActivityDate="2011-03-18T13:53:10.767" ClosedDate="2011-03-18T15:16:57.877" Title="Are Confounding variables unknown in nature?" Tags="&lt;confounding&gt;" CommentCount="5" />
  <row Id="8455" PostTypeId="1" AcceptedAnswerId="8531" CreationDate="2011-03-18T13:59:43.307" Score="9" ViewCount="331" Body="&lt;p&gt;The recent events in Japan have made me think about the following.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Nuclear Plants are usually designed to limit risk of serious accidents&#xA;to a 'design basis probability' for example, say, 10E-6/year.&#xA;This is the criteria for a single plant.&#xA;However, when there is a population of hundreds of reactors, how do we combine the individual probabilities of a serious accident ?&#xA;I know I could probably research this myself but having found this site I a sure&#xA;there is someone that will be able to answer this question quite easily.&#xA;Thanks&lt;/p&gt;&#xA;" OwnerUserId="3777" LastActivityDate="2011-03-21T17:36:23.963" Title="Combining probabilities of nuclear accidents" Tags="&lt;probability&gt;" AnswerCount="5" CommentCount="15" />
  <row Id="8456" PostTypeId="1" CreationDate="2011-03-18T14:03:39.683" Score="1" ViewCount="141" Body="&lt;p&gt;I am a senior doing a science fair project on the efficiency of fuels. For each fuel I tested, I have a time series of temperature.&#xA;I want to plot these into an excel xy chart, how do I plot them if the constants (time) are not the same for each set of data?&lt;/p&gt;&#xA;" OwnerUserId="3778" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-18T16:07:45.020" LastActivityDate="2011-03-20T21:41:48.500" Title="Excel xy chart with unequal x values in series" Tags="&lt;data-visualization&gt;&lt;excel&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="8457" PostTypeId="2" ParentId="8455" CreationDate="2011-03-18T14:09:25.707" Score="0" ViewCount="" Body="&lt;p&gt;As commentators pointed out, this has the very strong independency assumption.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let the probability that a plant blows up be $p$. Then the probability that a plant does not blow up is $1-p$. Then the probability that $n$ plants do not blow up is $(1-p)^n$. The expected number of plants blown up per year is $np$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In case you're interested: &lt;a href=&quot;http://en.wikipedia.org/wiki/Binomial_distribution&quot; rel=&quot;nofollow&quot;&gt;binomial distribution&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2860" LastEditorUserId="2860" LastEditorDisplayName="" LastEditDate="2011-03-18T16:04:05.963" LastActivityDate="2011-03-18T16:04:05.963" CommentCount="13" />
  <row Id="8458" PostTypeId="2" ParentId="8453" CreationDate="2011-03-18T14:17:10.117" Score="2" ViewCount="" Body="&lt;p&gt;Regression gives you, essentially, a correlation. Correlation is not causality -- or else global warming would be caused by a decline in global number of pirates.&lt;/p&gt;&#xA;" OwnerUserId="2044" LastActivityDate="2011-03-18T14:17:10.117" CommentCount="3" />
  <row Id="8459" PostTypeId="1" CreationDate="2011-03-18T15:26:07.260" Score="2" ViewCount="86" Body="&lt;p&gt;Hi folks, I recently was asked to help characterize some surfaces (raised ink dots on paper) that were scanned with a profilometer, yielding height data across a grid of x &amp;amp; y coordinates. The critical question was to characterize the average dot height. The data were rather noisy as the paper was not perfectly flat, so I played around with various approaches to removing this noise and came upon an approach that appears to do very well (to the eye at least). See below for pseudocode and R code. The logic is that every row and column definitely has an area with no dots on it, so ideally the minimum for each row and column should be around zero. However, one obtains different surfaces depending on whether row minimums are shifted to zero or whether column minimums are shifted to zero. To solve this ambiguity, I simply do both and average the results. I then noticed that repeating this process many times appeared to converge on a very nicely de-noised final result. Is there a name for this approach or something similar?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#Pseudocode (&quot;data&quot; is a RxC matrix of height values)&#xA;repeat the following 1000 times:&#xA;    create a copy of data, call it &quot;copy1&quot;&#xA;    for every row of height data in copy1:&#xA;        subtract the minimum of that row from each value in the row (i.e. shift the row's data so that the minimum is zero)&#xA;    create a second copy of data, call it &quot;copy2&quot;&#xA;    for every column of height data in copy2:&#xA;        subtract the minimum of that column from each value in the column (i.e. shift the column's data so that the minimum is zero)&#xA;    replace data with the cell-wise average of copy1 and copy2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;_&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#R-code (data is in &quot;a&quot;)&#xA;for(i in 1:1e3){&#xA;    c1 = a&#xA;    for(i in 1:ncol(c1)){&#xA;        c1[,i] = c1[,i]-min(c1[,i])&#xA;    }&#xA;    c2 = a&#xA;    for(i in 1:nrow(c2)){&#xA;        c2[i,] = c2[i,]-min(c2[i,])&#xA;    }&#xA;    a = (c1+c2)/2&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="364" LastActivityDate="2011-03-19T15:56:45.143" Title="Does this de-noising algorithm have a name?" Tags="&lt;smoothing&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="8460" PostTypeId="2" ParentId="8420" CreationDate="2011-03-18T15:49:52.650" Score="0" ViewCount="" Body="&lt;p&gt;I like Visualizing data patterns with micromaps by daniel carr and linda pickle, specially the concept of linked micromaps, which is influenced by the idea of small multiples (Tufte) and the idea of dot - charts (Cleveland). I strong recommend to read it. &lt;/p&gt;&#xA;" OwnerUserId="2348" LastActivityDate="2011-03-18T15:49:52.650" />
  <row Id="8461" PostTypeId="2" ParentId="8456" CreationDate="2011-03-18T18:13:25.927" Score="2" ViewCount="" Body="&lt;p&gt;Use a Scatter Plot where the horizontal axis is time.   Below is a screenshot of an Excel sheet where two temperatures are shown.  Temp 1 starts at 9:00:00 AM and increments by 1 minute and 26 seconds.    Temp 2 starts at 9:30:00 AM and increments by 2 minutes and 53 seconds.   They are both plotted in the graph.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As shown in the plot, the blue Temp 1 has been &quot;selected&quot; in the graph and the corresponding graph formula is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;=SERIES(Sheet1!$B$1,Sheet1!$A$2:$A$27,Sheet1!$B$2:$B$27,1)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I select Temp 2, the corresponding graph formula is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;=SERIES(Sheet1!$E$1,Sheet1!$D$2:$D$27,Sheet1!$E$2:$E$27,2)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Both Temp 1 and Temp 2 uses the same x-axis and y-axis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/JW6E5.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a second way to do this where the blue series has its own x-axis and y-axis, and the red series has a different x-axis and y-axis.   However, I wouldn't recommend this scheme because it is typically too confusing for others to interpret the results.  It requires that you assign one (or more) of the lines to a secondary y-axis and then a secondary x-axis.   Here are the steps:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Right click on the series that you want to assign to the secondary axes.  Select &quot;Format Data Series&quot;.   Under &quot;Series Options&quot; select &quot;Secondary Axis&quot;.    That will assign the selected series to a secondary y-axis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) With the series still selected, go to the &quot;Chart Tools&quot; &quot;Layout&quot; &quot;Axes&quot; selection on the ribbon.  Click &quot;Secondary Horizontal Axis&quot; and pick whatever options you want.   That will assign the selected series to a secondary x-axis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Below is an example.   I purposely changed the color of the text of the lower x-axis and the left y-axis to blue to signify that these are the axes to use for the blue series.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I changed the color of the text of the upper x-axis and the right y-axis to red to signify their use for the red series.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Notice that the two y-axes have different scales.  The blue y-axis runs from 0 to 90, while the red y-axis runs from 0 to 100.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Notice also that I set both x-axis to start, stop, and increment at the same values.   If you set the increment of the two x-axes to two different values, this graph can be even more confusing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/HTVTv.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="2775" LastActivityDate="2011-03-18T18:13:25.927" />
  <row Id="8462" PostTypeId="1" AcceptedAnswerId="8484" CreationDate="2011-03-18T18:19:39.250" Score="9" ViewCount="174" Body="&lt;p&gt;We're creating a chart showing traffic by time of day over a given period. So the y-axis is traffic, the x-axis is midnight, 1am, 2am, etc. It could also be days of the week. What's the generic name for this type of chart? I've come up with &quot;cycle chart&quot;. Is that the standard? Is there one?&#xA;&lt;img src=&quot;http://i.imgur.com/IDu3k.png&quot; alt=&quot;example&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Update:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just to add a bit more clarity, what's being shown in the top chart is &lt;em&gt;not&lt;/em&gt; one day, it's an aggregation of many days. E.g. over the last month, 6am has on average been lower than noon. Similarly, in the bottom chart, over the last year, traffic dips on Saturdays.&lt;/p&gt;&#xA;" OwnerUserId="1531" LastEditorUserId="1531" LastEditorDisplayName="" LastEditDate="2011-03-18T21:56:26.593" LastActivityDate="2011-03-19T15:02:19.393" Title="What's a good, generic name for chart of things by time of day?" Tags="&lt;time-series&gt;&lt;data-visualization&gt;" AnswerCount="6" CommentCount="6" />
  <row Id="8463" PostTypeId="2" ParentId="8455" CreationDate="2011-03-18T18:47:04.663" Score="4" ViewCount="" Body="&lt;p&gt;Before you set up your analysis, keep in mind the reality of what the current situation involves.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This meltdown was not directly caused by the earthquake or the tsunami.   It was because of a lack of back-up power.   If they had enough back-up power, regardless of the earthquake/tsunami, they could have kept the cooling water running, and none of the meltdowns would have happened.  The plant would probably be back up and running by now.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Japan, for whatever reason, has two electrical frequencies (50 Hz and 60 Hz).  And, you can't run a 50 Hz motor at 60 Hz or vice versa.   So, whatever frequency the plant was using/providing is the frequency they need to power up.   &quot;U.S. type&quot; equipment runs at 60 Hz and &quot;European type&quot; equipment runs at 50 Hz, so in providing an alternative power source, keep that in mind.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Next, that plant is in a fairly remote mountainous area.   To supply external power requires a LONG power line from another area (requiring days/weeks to build) or large gasoline/diesel driven generators.  Those generators are heavy enough that flying them in with a helicopter is not an option.   Trucking them in may also be a problem due to the roads being blocked from the earthquake/tsunami.   Bringing them in by ship is an option, but it also takes days/weeks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The bottom line is, the risk analysis for this plant comes down to a lack of SEVERAL (not just one or two) layers of back-ups.   And, because this reactor is an &quot;active design&quot;, which means it requires power to stay safe, those layers are not a luxury, they're required.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is an old plant.  A new plant would not be designed this way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit (03/19/2011) ==============================================&lt;/p&gt;&#xA;&#xA;&lt;p&gt;J Presley: To answer your question requires a short explanation of terms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As I said in my comment, to me, this is a matter of &quot;when&quot;, not &quot;if&quot;, and as a crude model, I suggested the Poisson Distribution/Process.   The Poisson Process is a series of events that happen at an average rate over time (or space, or some other measure).  These events are independent of each other and random (no patterns).  The events happen one at a time (2 or more events don't happen at the exact same time).   It is basically a binomial situation (&quot;event&quot; or &quot;no event&quot;) where the probability that the event will happen is relatively small.   Here are some links:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Poisson_process&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Poisson_process&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Poisson_distribution&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Poisson_distribution&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Next, the data.    Here's a list of nuclear accidents since 1952 with the INES Level:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Nuclear_and_radiation_accidents&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Nuclear_and_radiation_accidents&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I count 19 accidents, 9 state an INES Level.    For those without an INES level, all I can do is assume the level is below Level 1, so I'll assign them Level 0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, one way to quantify this is 19 accidents in 59 years (59 = 2011 -1952).    That's 19/59 = 0.322 acc/yr.   In terms of a century, that's 32.2 accidents per 100 years.  Assuming a Poisson Process gives the following graphs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/wOKlY.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Originally, I suggested a Lognormal, Gamma, or Exponential Distribution for the severity of the accidents.   However, since the INES Levels are given as discrete values, the distribution would need to be discrete.   I would suggest either the Geometric or Negative Binomial Distribution.  Here are their descriptions:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Negative_binomial_distribution&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Negative_binomial_distribution&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Geometric_distribution&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Geometric_distribution&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;They both fit the data about the same, which is not very well (lots of Level 0's, one Level 1, zero Level 2's, etc).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; Fit for Negative Binomial Distribution&#xA;&#xA; Fitting of the distribution ' nbinom ' by maximum likelihood &#xA; Parameters : &#xA;      estimate Std. Error&#xA; size 0.460949  0.2583457&#xA; mu   1.894553  0.7137625&#xA; Loglikelihood:  -34.57827   AIC:  73.15655   BIC:  75.04543 &#xA; Correlation matrix:&#xA;              size           mu&#xA; size 1.0000000000 0.0001159958 &#xA; mu   0.0001159958 1.0000000000&#xA;&#xA; #====================&#xA; Fit for Geometric Distribution&#xA;&#xA; Fitting of the distribution ' geom ' by maximum likelihood &#xA; Parameters : &#xA;       estimate Std. Error&#xA; prob 0.3454545  0.0641182&#xA; Loglikelihood:  -35.4523   AIC:  72.9046   BIC:  73.84904 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The Geometric Distribution is a simple one parameter function while the Negative Binomial Distribution is a more flexible two parameter function.   I would go for the flexibility, plus the underlying assumptions of how the Negative Binomial Distribution was derived.    Below is a graph of the fitted Negative Binomial Distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/UwlD7.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Below is the code for all this stuff.   If anyone finds a problem with my assumptions or coding, don't be afraid to point it out.   I checked through the results, but I didn't have enough time to really chew on this.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; library(fitdistrplus)&#xA;&#xA; #Generate the data for the Poisson plots&#xA; x &amp;lt;- dpois(0:60, 32.2)&#xA; y &amp;lt;- ppois(0:60, 32.2, lower.tail = FALSE)&#xA;&#xA; #Cram the Poisson Graphs into one plot&#xA; par(pty=&quot;m&quot;, plt=c(0.1, 1, 0, 1), omd=c(0.1,0.9,0.1,0.9))&#xA; par(mfrow = c(2, 1))&#xA;&#xA; #Plot the Probability Graph&#xA; plot(x, type=&quot;n&quot;, main=&quot;&quot;, xlab=&quot;&quot;, ylab=&quot;&quot;, xaxt=&quot;n&quot;, yaxt=&quot;n&quot;)&#xA; mtext(side=3, line=1, &quot;Poisson Distribution Averaging 32.2 Nuclear Accidents Per Century&quot;, cex=1.1, font=2)&#xA; xaxisdat &amp;lt;- seq(0, 60, 10)&#xA; pardat &amp;lt;- par()&#xA; yaxisdat &amp;lt;- seq(pardat$yaxp[1], pardat$yaxp[2], (pardat$yaxp[2]-pardat$yaxp[1])/pardat$yaxp[3])&#xA; axis(2, at=yaxisdat, labels=paste(100*yaxisdat, &quot;%&quot;, sep=&quot;&quot;), las=2, padj=0.5, cex.axis=0.7, hadj=0.5, tcl=-0.3)&#xA; mtext(&quot;Probability&quot;, 2, line=2.3)&#xA; abline(h=yaxisdat, col=&quot;lightgray&quot;)&#xA; abline(v=xaxisdat, col=&quot;lightgray&quot;)&#xA; lines(x, type=&quot;h&quot;, lwd=3, col=&quot;blue&quot;)&#xA;&#xA; #Plot the Cumulative Probability Graph&#xA; plot(y, type=&quot;n&quot;, main=&quot;&quot;, xlab=&quot;&quot;, ylab=&quot;&quot;, xaxt=&quot;n&quot;, yaxt=&quot;n&quot;)&#xA; pardat &amp;lt;- par()&#xA; yaxisdat &amp;lt;- seq(pardat$yaxp[1], pardat$yaxp[2], (pardat$yaxp[2]-pardat$yaxp[1])/pardat$yaxp[3])&#xA; axis(2, at=yaxisdat, labels=paste(100*yaxisdat, &quot;%&quot;, sep=&quot;&quot;), las=2, padj=0.5, cex.axis=0.7, hadj=0.5, tcl=-0.3)&#xA; mtext(&quot;Cumulative Probability&quot;, 2, line=2.3)&#xA; abline(h=yaxisdat, col=&quot;lightgray&quot;)&#xA; abline(v=xaxisdat, col=&quot;lightgray&quot;)&#xA; lines(y, type=&quot;h&quot;, lwd=3, col=&quot;blue&quot;)&#xA;&#xA; axis(1, at=xaxisdat, padj=-2, cex.axis=0.7, hadj=0.5, tcl=-0.3)&#xA; mtext(&quot;Number of Nuclear Accidents Per Century&quot;, 1, line=1)&#xA; legend(&quot;topright&quot;, legend=c(&quot;99% Probability - 20 Accidents or More&quot;, &quot; 1% Probability - 46 Accidents or More&quot;), bg=&quot;white&quot;, cex=0.8)&#xA;&#xA; #Calculate the 1% and 99% values&#xA; qpois(0.01, 32.2, lower.tail = FALSE)&#xA; qpois(0.99, 32.2, lower.tail = FALSE)&#xA;&#xA; #Fit the Severity Data&#xA; z &amp;lt;- c(rep(0,10), 1, rep(3,2), rep(4,3), rep(5,2), 7)&#xA; zdis &amp;lt;- fitdist(z, &quot;nbinom&quot;)&#xA; plot(zdis, lwd=3, col=&quot;blue&quot;)&#xA; summary(zdis)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Edit (03/20/2011) ======================================================&lt;/p&gt;&#xA;&#xA;&lt;p&gt;J Presley:  I'm sorry I couldn't finish this up yesterday.   You know how it is on weekends, lots of duties.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The last step in this process is to assemble a simulation using the Poisson Distribution to determine when an event happens, and then the Negative Binomial Distribution to determine the severity of the event.  You might run 1000 sets of &quot;century chunks&quot; to generate the 8 probability distributions for Level 0 through Level 7 events.   If I get the time, I might run the simulation, but for now, the description will have to do.  Maybe someone reading this stuff will run it.    After that is done, you'll have a &quot;base case&quot; where all of the events are assumed to be INDEPENDENT.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Obviously, the next step is to relax one or more of the above assumptions.    An easy place to start is with the Poisson Distribution.  It assumes that all events are 100% independent.   You can change that in all sorts of ways.   Here are some links to Non-homogeneous Poisson Distributions:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.math.wm.edu/~leemis/icrsa03.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.math.wm.edu/~leemis/icrsa03.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://filebox.vt.edu/users/pasupath/papers/nonhompoisson_streams.pdf&quot; rel=&quot;nofollow&quot;&gt;http://filebox.vt.edu/users/pasupath/papers/nonhompoisson_streams.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The same idea goes for the Negative Binomial Distribution.   This combination will lead you down all sorts of paths.  Here are some examples:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://surveillance.r-forge.r-project.org/&quot; rel=&quot;nofollow&quot;&gt;http://surveillance.r-forge.r-project.org/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.m-hikari.com/ijcms-2010/45-48-2010/buligaIJCMS45-48-2010.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.m-hikari.com/ijcms-2010/45-48-2010/buligaIJCMS45-48-2010.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.michaeltanphd.com/evtrm.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.michaeltanphd.com/evtrm.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The bottom line is, you asked a question where the answer depends on how far you want to take it.    My guess is, someone, somewhere will be commissioned to generate &quot;an answer&quot; and will be surprised at how long it takes to do the work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit (03/21/2011) ====================================================&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I had a chance to slap together the above mentioned simulation.  The results are shown below.   From the original Poisson Distribution, the simulation provides eight Poisson Distributions, one for each INES Level.   As the severity level rises (INES Level Number rises), the number of expected events per century drops.   This may be a crude model, but it's a reasonable place to start.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/6QjPg.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="2775" LastEditorUserId="2775" LastEditorDisplayName="" LastEditDate="2011-03-21T17:36:23.963" LastActivityDate="2011-03-21T17:36:23.963" CommentCount="8" />
  <row Id="8464" PostTypeId="2" ParentId="8462" CreationDate="2011-03-18T19:05:28.163" Score="0" ViewCount="" Body="&lt;p&gt;The type of chart you've drawn is known as a Histogram &lt;a href=&quot;http://en.wikipedia.org/wiki/Histogram&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Histogram&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="3484" LastActivityDate="2011-03-18T19:05:28.163" CommentCount="7" />
  <row Id="8465" PostTypeId="2" ParentId="8462" CreationDate="2011-03-18T19:05:58.343" Score="3" ViewCount="" Body="&lt;p&gt;What you've illustrated is a time series column (or bar) graph. The two graphs are of differing time resolution or differing time aggregation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There may be industry specific terms for these types of charts. In finance, for example, the &lt;a href=&quot;http://en.wikipedia.org/wiki/Open-high-low-close_chart&quot; rel=&quot;nofollow&quot;&gt;open-high-low-close chart&lt;/a&gt; is a very common time series plot:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/2eiG2.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When the x axis is time, as in your example, it's often common to illustrate the points as a line graph, instead of bars/columns. The reason for this is to put the visual emphasis on the change from one period to the next. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might also consider graphing period-over-period. For example a &lt;a href=&quot;http://spotfire.tibco.com/community/blogs/stn/archive/2009/01/28/create-a-year-over-year-comparison-chart.aspx&quot; rel=&quot;nofollow&quot;&gt;year-over-year&lt;/a&gt; would show how the numbers for a given month (typically, although could be month or day) compare to the numbers of the prior year for the same month.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/nrlo1.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I realize your question was about naming, not all the other cool graphs you can do ;) &lt;/p&gt;&#xA;" OwnerUserId="29" LastActivityDate="2011-03-18T19:05:58.343" CommentCount="3" />
  <row Id="8466" PostTypeId="1" AcceptedAnswerId="8482" CreationDate="2011-03-18T19:07:53.047" Score="1" ViewCount="169" Body="&lt;p&gt;We know that density for a student-t distribution is given as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{\Gamma(\frac{\nu + 1}{2})}{\Gamma(\frac{\nu}{2})} \left(\frac{\lambda}{\pi\nu}\right)^{\frac{1}{2}} \left[1+\frac{\lambda(x-\mu)^2}{\nu}\right]^{-\frac{\nu+1}{2}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;with &#xA;$\text{E}(X) = \mu$, &#xA;$\text{var}(X) = \frac{1}{\lambda}\frac{\nu}{\nu-2}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where the three parameters are shape factor $\nu$, location $\mu$ and dispersion $\lambda$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now if I standardize X to Z as $\frac{X-\text{E}(X)}{\text{Std}(X)}$, what is the distribution of $Z$. If I standardize my input data to $Z$ as above, what is the density function for $f_Z(z)$?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I came across this question in Stephen Taylor's Asset Dynamics book. I am a bit confused after reading this question. As I understand, $\frac{X-\mu}{\sigma}$ where $\sigma=1/\sqrt(\lambda)$ will give the standardized t distribution with center 0 and disperson 1. But not sure what $Z$ above will look like.&lt;/p&gt;&#xA;" OwnerUserId="862" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-19T04:11:17.990" LastActivityDate="2011-03-19T04:11:17.990" Title="Standardized Student's-t distribution" Tags="&lt;likelihood-function&gt;&lt;t-distribution&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="8467" PostTypeId="2" ParentId="8462" CreationDate="2011-03-18T19:58:55.227" Score="0" ViewCount="" Body="&lt;p&gt;&lt;strike&gt;Short, simple, descriptive: time series plot.&lt;/strike&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: In light of the discussion, I'd vote for histogram as well. At least, thats the generic name for this kind of chart, where the hours of the day are a natural division of stacks.&lt;/p&gt;&#xA;" OwnerUserId="1766" LastEditorUserId="1766" LastEditorDisplayName="" LastEditDate="2011-03-18T23:20:04.863" LastActivityDate="2011-03-18T23:20:04.863" CommentCount="3" />
  <row Id="8468" PostTypeId="1" CreationDate="2011-03-18T21:04:55.237" Score="2" ViewCount="69" Body="&lt;p&gt;Let's say I'm performing regularized regression and I want to validate the results using holdout.  (I'm choosing holdout instead of cross-validation because my dataset is fairly large, so computational power is an issue and the difference between holdout and cross-validation estimates will likely be negligible.)  I want to put a confidence interval on the $R^2$ score for the holdout set.  Is the following procedure valid?&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Compute the regression coefficients on the training data.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Compute the predictions on the held out data.  Treat each prediction as if it were an observed random variable.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Compute the Pearson correlation of the predictions with the actual $Y$ values for the held-out data.  Compute a confidence interval for this correlation using Fisher's Z-Transformation.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If the confidence interval of this correlation is some set of real numbers, then the confidence interval of $R^2$ is just the set of the squares of these numbers.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I can't think of any reason why this procedure would be wrong, but I've never heard of it being done before, so I wanted to run it by some more experienced statisticians.&lt;/p&gt;&#xA;" OwnerUserId="1347" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2011-03-19T00:01:10.343" LastActivityDate="2011-03-19T00:01:10.343" Title="Confidence Intervals for Holdout R^2?" Tags="&lt;regression&gt;&lt;correlation&gt;&lt;confidence-interval&gt;&lt;forecasting&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="8469" PostTypeId="2" ParentId="8033" CreationDate="2011-03-18T21:12:59.237" Score="1" ViewCount="" Body="&lt;p&gt;Ten days later this is probably worth an answer:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A normal distribution has about 20% of its distribution more than 0.842 standard deviations above the mean; using the cumulative distribution of standard normal $\Phi$,&#xA;$$\Phi(0.842) \approx 0.8$$&#xA;so the threshold should be about $70 + 8\times 0.842 \approx 76.7$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I do wonder slightly why the lecturer would do this instead of just giving an A to the top 20% of students. &lt;/p&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-03-18T21:12:59.237" CommentCount="3" />
  <row Id="8470" PostTypeId="1" CreationDate="2011-03-18T21:27:39.560" Score="2" ViewCount="150" Body="&lt;p&gt;I'm working on a naive Bayes classifier that calculates probabilities using a normal Gaussian distribution. This works very well when I am classifying something into two mutually exclusive buckets (e.g. spam vs. not-spam), but when I am working with a factor that is not easily classified that way (when the classifications are not mutually exclusive) I would like to express the result as a percentage.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I combine the probability density of several factors (by multiplying them together) I tend to get a very small number and I would like to adjust that so I can express it in a 0-100 percent range, so it will be more easily understood. Is there another factor I can use to adjust the probability density into a percentage range?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example: in the Wikipedia article for &lt;a href=&quot;http://en.wikipedia.org/wiki/Naive_Bayes_classifier&quot; rel=&quot;nofollow&quot;&gt;naive Bayes classifiers&lt;/a&gt;, there is an example of using height, weight and shoe size variables to classify a person as male or female. After computing the numbers, the posterior result for the female classification is 5.3778E-4 (or .00053778). Out of context that seems minuscule, but if you compare that number to the result for the male classification, the percentage would be 99.99% female. What factors (if any) could I apply to the posterior result to get that percentage, without the benefit of the male result to compare it to?&lt;/p&gt;&#xA;" OwnerUserId="3784" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-28T22:54:36.987" LastActivityDate="2011-04-28T22:54:36.987" Title="Can you express a probability density as a percentage?" Tags="&lt;probability&gt;" AnswerCount="1" CommentCount="7" />
  <row Id="8471" PostTypeId="2" ParentId="8462" CreationDate="2011-03-18T22:12:17.783" Score="4" ViewCount="" Body="&lt;p&gt;I'd suggest &quot;diurnal&quot; or &quot;circadian&quot; rhythm chart. For weekly, the latter would be &quot;circaseptan&quot;, &quot;circamensual&quot; for &quot;monthly&quot;, and &quot;circannual&quot; for &quot;yearly&quot;.&lt;/p&gt;&#xA;" OwnerUserId="3369" LastEditorUserId="3369" LastEditorDisplayName="" LastEditDate="2011-03-18T22:20:42.863" LastActivityDate="2011-03-18T22:20:42.863" CommentCount="1" />
  <row Id="8472" PostTypeId="1" AcceptedAnswerId="8495" CreationDate="2011-03-18T23:33:45.757" Score="8" ViewCount="433" Body="&lt;p&gt;In network motif algorithms, it seems quite common to return both a &lt;a href=&quot;http://en.wikipedia.org/wiki/P-value&quot; rel=&quot;nofollow&quot;&gt;p-value&lt;/a&gt; and a &lt;a href=&quot;http://en.wikipedia.org/wiki/Standard_score&quot; rel=&quot;nofollow&quot;&gt;Z-score&lt;/a&gt; for a statistic: &quot;Input network contains X copies of subgraph G&quot;.  A subgraph is deemed a motif if it satisfies&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;p-value &amp;lt; A,&lt;/li&gt;&#xA;&lt;li&gt;Z-score &gt; B and&lt;/li&gt;&#xA;&lt;li&gt;X &gt; C,&#xA;for some user-defined (or community-defined) A, B and C.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;This motivates the question:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;Question&lt;/strong&gt;: What are the differences between p-value and Z-score?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;And the subquestion:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;Question&lt;/strong&gt;: Are there situations where the p-value and Z-score of the same statistic might suggest opposite hypotheses?  Are the first and second conditions listed above essentially the same?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="386" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-29T00:42:18.500" LastActivityDate="2011-04-29T00:42:18.500" Title="What is the difference between Z-scores and p-values?" Tags="&lt;hypothesis-testing&gt;&lt;p-value&gt;&lt;z-statistic&gt;" AnswerCount="3" FavoriteCount="2" />
  <row Id="8473" PostTypeId="2" ParentId="8472" CreationDate="2011-03-18T23:40:41.547" Score="5" ViewCount="" Body="&lt;p&gt;$p$-value indicates how unlikely the statistic is. $z$-score indicates how far away from the mean it is. There may be a difference between them, depending on the sample size. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For large samples, even small deviations from the mean become unlikely. I.e. the $p$-value may be very small even for a low $z$-score. Conversely, for small samples even large deviations are not unlikely. I.e. a large $z$-score will not necessarily mean a small $p$-value.&lt;/p&gt;&#xA;" OwnerUserId="3369" LastActivityDate="2011-03-18T23:40:41.547" CommentCount="3" />
  <row Id="8474" PostTypeId="2" ParentId="8468" CreationDate="2011-03-19T00:00:53.150" Score="2" ViewCount="" Body="&lt;p&gt;That should work ok provided the lower limit of your confidence interval is positive. Otherwise the square transformation is not monotonically increasing and then the probability coverage is not preserved.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've never seen it done before either.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2011-03-19T00:00:53.150" CommentCount="4" />
  <row Id="8475" PostTypeId="2" ParentId="8470" CreationDate="2011-03-19T00:23:09.483" Score="2" ViewCount="" Body="&lt;p&gt;You can use any &quot;monotonic&quot; transformation of the probabilities as you choose (at least as far as I know).  As long as your transformation preserves the ordering of probabilities, you will not be lead astray in your decision making.  Personally, I prefer to use odds ratios.  They seem to make intuitive sense to me, and I know what decision to make almost straight away (and the &quot;region of uncertainty&quot; is easily identified, perhaps odds between 0.1 to 10?).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://stats.stackexchange.com/questions/8419/bayesian-classifier-with-multivariate-normal-densities/8434#8434&quot;&gt;my answer to this question&lt;/a&gt; goes through a worked example of how you would use the odds in naive Bayes classifier.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-03-19T00:23:09.483" />
  <row Id="8476" PostTypeId="2" ParentId="8472" CreationDate="2011-03-19T00:51:10.900" Score="6" ViewCount="" Body="&lt;p&gt;A $Z$-score describes your deviation from the mean in units of standard deviation. It is not explicit as to whether you accept or reject your null hypothesis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A $p$-value is the probability that under the null hypothesis we could observe a point that is as extreme as your statistic. This explicitly tells you whether you reject or accept your null hypothesis given a test size $\alpha$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider an example where $X\sim \mathcal{N}(\mu,1)$ and the null hypothesis is $\mu=0$. Then you observe $x_1=5$. Your $Z$-score is 5 (which only tells you how far you deviate from your null hypothesis in terms of $\sigma$) and your $p$-value is 5.733e-7. For 95% confidence, you will have a test size $\alpha=0.05$ and since $p&amp;lt;\alpha$ then you reject the null hypothesis. But for any given statistic, there should be some equivalent $A$ and $B$ such that the tests are the same.&lt;/p&gt;&#xA;" OwnerUserId="3786" LastActivityDate="2011-03-19T00:51:10.900" CommentCount="6" />
  <row Id="8477" PostTypeId="1" AcceptedAnswerId="8489" CreationDate="2011-03-19T01:03:17.700" Score="1" ViewCount="43" Body="&lt;p&gt;I am reading Stephen Taylor's Asset Dynamics book and came across something I didn't fully understand.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For an ARCH process, the return series is modeled as&#xA;$r_t = \mu_t + h_t^{1/2}z_t$ where is $z_t$ is $D(0,1)$ and may be normal/non-normal and $h_t$ is conditional variance (some function of subset of $\theta$).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then he says the conditional density of $r_t$ is $f(r_t|\theta) = \frac{f(z_t(\theta)|\theta)}{\sqrt{h_t(\theta)}}$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't understand how does the conditional variance term occur in the denominator of density relation and why doesn't $\mu_t$ occur in the relation. Can someone help me how to derive this?&lt;/p&gt;&#xA;" OwnerUserId="862" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-19T11:03:50.653" LastActivityDate="2011-03-19T11:10:07.747" Title="Expression for conditional density for ARCH processes" Tags="&lt;likelihood-function&gt;&lt;conditional-probability&gt;&lt;garch&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="8478" PostTypeId="2" ParentId="8466" CreationDate="2011-03-19T01:03:58.207" Score="0" ViewCount="" Body="&lt;p&gt;If you're referring to the density function it's the exact same thing. You have simply shifted the distribution and normalized by the standard deviation. &lt;/p&gt;&#xA;" OwnerUserId="3786" LastActivityDate="2011-03-19T01:03:58.207" />
  <row Id="8479" PostTypeId="2" ParentId="8446" CreationDate="2011-03-19T01:10:58.090" Score="2" ViewCount="" Body="&lt;p&gt;This looks a lot like a Beta distribution from its shape and seemingly bounded domain. You can use maximum likelihood estimation to estimate the parameters $\alpha$ and $\beta$.&lt;/p&gt;&#xA;" OwnerUserId="3786" LastActivityDate="2011-03-19T01:10:58.090" CommentCount="1" />
  <row Id="8480" PostTypeId="2" ParentId="866" CreationDate="2011-03-19T01:21:05.993" Score="0" ViewCount="" Body="&lt;p&gt;Generally, when you have many small/medium sized effects you should go with ridge. If you have only a few variables with a medium/large effect, go with lasso.&#xA;&lt;a href=&quot;http://www-stat.stanford.edu/~tibs/ElemStatLearn/&quot; rel=&quot;nofollow&quot;&gt;Hastie, Tibshirani, Friedman&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="3786" LastActivityDate="2011-03-19T01:21:05.993" />
  <row Id="8481" PostTypeId="2" ParentId="8401" CreationDate="2011-03-19T01:38:59.533" Score="1" ViewCount="" Body="&lt;p&gt;There isn't any indication that your variables here are correlated so I dont know why you would use MCMC as opposed to regular Monte Carlo. There are many different sampling methods including the mentioned stratified sampling (Latin hypercube) and QMC. Sparse quadrature methods are very good if the dimension of the problem is not too high (not more than 10) since sparse quadrature grids grow geometrically (curse of dimensionality). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But it sounds like you are on the right track with respect to importance sampling. The key here is to choose a biased distribution that has large probability concentrated near your region of interest and that it has thicker tails than the nominal distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd like to add that this is an open research problem so if you can come up with something good it would be of great interest to the community!&lt;/p&gt;&#xA;" OwnerUserId="3786" LastActivityDate="2011-03-19T01:38:59.533" CommentCount="1" />
  <row Id="8482" PostTypeId="2" ParentId="8466" CreationDate="2011-03-19T04:10:02.780" Score="3" ViewCount="" Body="&lt;p&gt;Assume $\nu \gt 2$ so that this distribution actually has a mean and standard deviation (otherwise you cannot standardize it).  By direct calculation, its mean equals $\mu$ and its variance equals $\nu / (\lambda (\nu-2))$.  Standardizing it, by construction, &lt;em&gt;creates a distribution of the same shape but zero mean and unit standard deviation&lt;/em&gt;.  Thus, for the standardized distribution, the formula is the same but we must have $\mu = 0$ and $\nu / (\lambda (\nu-2)) = 1$; that is, $\lambda = \nu/(\nu-2)$.  Whence, plugging these values into the formula,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$f_Z(z) = \frac{\Gamma(\frac{\nu + 1}{2})}{\Gamma(\frac{\nu}{2})} \frac{1}{\sqrt{\pi(\nu-2)}} \left[1+\frac{z^2}{\nu-2}\right]^{-\frac{\nu+1}{2}}.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This assumes $\mathbb{E}[X] = \mu$ and $\mathbb{E}[(X-\mu)^2]$ are known in advance, not estimated from data.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-03-19T04:10:02.780" />
  <row Id="8483" PostTypeId="1" CreationDate="2011-03-19T07:48:59.207" Score="3" ViewCount="91" Body="&lt;p&gt;I am researching age at first sexual debut and HIV prevalence in Lesotho.  I want to analyse the data using logistic regression with SPSS.  My variables are age, sex, social status, education level, and environment.  How am I to allocate my variables?&lt;/p&gt;&#xA;" OwnerUserId="3787" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-19T14:07:17.233" LastActivityDate="2011-03-19T14:07:17.233" Title="Analysis plan using logistic regression" Tags="&lt;logistic-regression&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="8484" PostTypeId="2" ParentId="8462" CreationDate="2011-03-19T08:32:20.230" Score="6" ViewCount="" Body="&lt;p&gt;Nick Cox &lt;a href=&quot;http://stata-journal.com/sjpdf.html?articlenum=gr0025&quot; rel=&quot;nofollow&quot;&gt;(&lt;em&gt;Stata Journal&lt;/em&gt; 2006, p403)&lt;/a&gt; calls this sort of plot a &lt;strong&gt;'cycle plot'&lt;/strong&gt;, but notes that:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Cycle plots have been discussed under other names in the literature, including cycle-subseries plot, month plot, seasonal-by-month plot, and seasonal subseries plot.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;(followed by a load of refs to textbooks and papers)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many of these are clearly specific to seasonality, i.e. periods of one year. I still like the suggestion of 'periodic plot/chart' that I made in a comment to the question, but it appears the questioner's original suggestion of 'cycle plot/chart' is in fact the more standard generic term.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-03-19T08:32:20.230" CommentCount="1" />
  <row Id="8485" PostTypeId="1" CreationDate="2011-03-19T09:07:52.767" Score="7" ViewCount="218" Body="&lt;p&gt;I want to learn how Gibbs Sampling works and I am looking for a good basic to intermediate paper. I have a computer science background and basic statistic knowledge. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyone has read good material around? where did you learn it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="3788" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-19T11:01:12.700" LastActivityDate="2011-03-19T21:18:56.360" Title="A good Gibbs sampling tutorials and references" Tags="&lt;books&gt;&lt;gibbs&gt;" AnswerCount="2" CommentCount="3" FavoriteCount="3" />
  <row Id="8486" PostTypeId="1" CreationDate="2011-03-19T09:26:43.973" Score="2" ViewCount="104" Body="&lt;p&gt;I want to compare the slopes of progression of a variable that is measured in percentage versus a variable that is measured in decibels. What is a good method to compare? &lt;/p&gt;&#xA;" OwnerUserId="3194" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-03-19T11:17:48.263" LastActivityDate="2011-03-19T11:17:48.263" Title="How to compare slopes between variables with different scales?" Tags="&lt;pairwise-comparison&gt;" CommentCount="2" />
  <row Id="8487" PostTypeId="1" CreationDate="2011-03-19T09:37:12.297" Score="4" ViewCount="97" Body="&lt;p&gt;I've calculated Cohen's d for regression coefficients (from the t statistic), odds ratios and means differences, hoping to pool the results in a meta-analysis and see how it works. However, in STATA, it doesn't seem you're able to pool these results without confidence intervals for Cohen's d, so my question is how do I get around this? Is there a way of calculating it, or is there a way of pooling the results in STATA without this information? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know there are several negative sides to this sort of meta-analysis, but am intrigued to see how this works in comparison to several smaller analyses of specific effect sizes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks.&lt;/p&gt;&#xA;" OwnerUserId="3789" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-19T11:01:40.547" LastActivityDate="2011-03-19T20:05:45.363" Title="How do you calculate confidence intervals for Cohen's d?" Tags="&lt;cohens-d&gt;" AnswerCount="1" />
  <row Id="8488" PostTypeId="2" ParentId="8349" CreationDate="2011-03-19T10:40:15.893" Score="1" ViewCount="" Body="&lt;p&gt;If you only have two charts I would show the following pair.  You are right to want to avoid percentages if some items have several errors, but you can show average errors per item which amounts to the same thing without the misleading impression, and could go over 1 without any need for an explanation of how a percentage goes over 100%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You want to emphasize that: errors peaked in May; the error rate peaked in July; March and June were &quot;good months&quot; with high items and low errors and July was a &quot;bad month&quot;; that type 3 errors are most common; and (if true) the types of error vary between months.  I think these do that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/0RWPo.png&quot; alt=&quot;enter image description here&quot;&gt;  &lt;/p&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-03-19T10:40:15.893" CommentCount="1" />
  <row Id="8489" PostTypeId="2" ParentId="8477" CreationDate="2011-03-19T11:10:07.747" Score="1" ViewCount="" Body="&lt;p&gt;If the formula in the question is &lt;em&gt;exactly&lt;/em&gt; how it is written in the book, then this is a bit of slack notation, with the ambiguous looking $f(.)$.  The subscripts, while a bit ugly, are one way to make it more clear what the function represents (for $f(.)$ is essentially defined as two different things, which in &quot;nit-picking&quot; world is a contradiction unless $f(.)=0$ or $f(.)=\pm\infty$).  In my notation,a capital letter stands for the &quot;random variable&quot; and a small letter for a potential value of that random variable.  And the subscripts represent what the distribution is for (e.g. $f_{X|Y}(x|y)$ is the pdf for $X$, given $Y$ evaluated at values $x$ and $y$).  I also used a capital $F$ for CDFs and small $f$ for PDFs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I could have just written down &quot;use the jacobian&quot; as an answer, but I can never quite remember exactly what jacobian to use (is it J, or 1 over inverse J, or inverse J?).  Going through the tedious process below (of essentially re-deriving the change-of-variables formula for PDFs) is the only way that I feel sure I've got the right answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;AS far as I can tell, what has occurred here is that you define a &quot;random variable&quot; or &quot;source of uncertainty&quot;, and call it $Z_t$.  This has an unspecified PDF, denoted by $f_{Z_t}(z_{t})$.  I am going to assume that by $D(0,1)$ you mean that the pdf is to have mean 0 and variance 1, but is otherwise ambiguous.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So we write the cumulative distribution function (CDF) for $R_t$ as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$F_{R_t|\theta}(r_t|\theta)=Pr(R_t&amp;lt;r_t|\theta)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now $R_t$, given $\theta$ is a 1-to-1 function of $Z_t$.  Now we substitute this relation $R_t=\mu_t(\theta)+\sqrt{h_t(\theta)}Z_t$ and re-arrange the expression to get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$F_{R_t|\theta}(r_t|\theta)=Pr(\mu_t(\theta)+\sqrt{h_t(\theta)}Z_t&amp;lt;r_t|\theta)=Pr(Z_t&amp;lt;\frac{r_t-\mu_t(\theta)}{\sqrt{h_t(\theta)}}|\theta)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now you can see that this last expression is just the CDF of $Z_t$ evaluated at $\frac{r_t-\mu_t(\theta)}{\sqrt{h_t(\theta)}}$.  I assume this is what is meant by $z_t(\theta)$.  Now to get from the CDF to the PDF, you need to differentiate with respect to $r_t$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$f_{R_t|\theta}(r_t|\theta)=\frac{\partial}{\partial r_t}\left[F_{R_t|\theta}(r_t|\theta)\right]=\frac{\partial}{\partial r_t}\left[F_{Z_t|\theta}(z_t(\theta)|\theta)\right]$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now you use the chain rule for differentiation to get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$f_{R_t|\theta}(r_t|\theta)=\frac{\partial z_t(\theta)}{\partial r_t}\frac{\partial}{\partial z_t(\theta)}\left[F_{Z_t|\theta}(z_t(\theta)|\theta)\right]=\frac{\partial z_t(\theta)}{\partial r_t}f_{Z_t|\theta}(z_t(\theta)|\theta)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now the derivative here can be calculated as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{\partial z_t(\theta)}{\partial r_t}=\frac{\partial \left[\frac{r_t-\mu_t(\theta)}{\sqrt{h_t(\theta)}}\right]}{\partial r_t}=\frac{1}{\sqrt{h_t(\theta)}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Substituting this back into the answer gives the density required:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$f_{R_t|\theta}(r_t|\theta)=\frac{f_{Z_t|\theta}(z_t(\theta)|\theta)}{\sqrt{h_t(\theta)}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that this actually holds regardless of the distribution of the $Z_t$.  For nowhere did I make use of the $D(0,1)$ assumption.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-03-19T11:10:07.747" CommentCount="1" />
  <row Id="8490" PostTypeId="1" CreationDate="2011-03-19T11:38:14.290" Score="3" ViewCount="111" Body="&lt;p&gt;I'm trying to make sense of some data and get statistical results on them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I have is the following: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Subject    TimeOfDay    Test1    Test2    Test3&#xA;A          10:00        valA1    valA2    valA3&#xA;B          10:00        valB1    valB2    valB3&#xA;C          15:00        valC1    valC2    valC3&#xA;D          15:00        valD1    valD2    valD3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm trying to see if there's a significant difference between test notes (which have a normal distribution) at 10:00 and 15:00.&#xA;My problem is that I only have a few subjects for each time of day (3 or 4 in reality).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The good thing is here that &lt;strong&gt;I expect test notes to be correlated&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My questions are: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;How should I test for these correlations? I guess that I can use the whole population for that, and I was thinking of a standard pearson correlation coefficient and test&lt;/li&gt;&#xA;&lt;li&gt;Since I expect these values correlated (and all drawn from a normal distribution), can I just consider each of them as a different subject, same test: &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;For example: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Subject    TimeOfDay    Test&#xA;A1         10:00        valA1&#xA;A2         10:00        valA2&#xA;A3         10:00        valA3&#xA;B1         10:00        valB1&#xA;B2         10:00        valB2&#xA;B3         10:00        valB3&#xA;C1         15:00        valC1&#xA;C2         15:00        valC2&#xA;C3         15:00        valC3&#xA;D1         15:00        valD1&#xA;D2         15:00        valD2&#xA;D3         15:00        valD3  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If yes, what kind of test should I do to make sure that what I have done is valid? Or do you see a better way to get the same result (more significant performances results between these times) with another method than this one I've  thought of?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A lot for your thoughts!&lt;/p&gt;&#xA;" OwnerUserId="3791" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-19T15:32:26.560" LastActivityDate="2011-03-20T18:42:55.053" Title="What kind of statistical analysis should I do to aggregate these values?" Tags="&lt;correlation&gt;&lt;multivariate-analysis&gt;&lt;factor-analysis&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="8491" PostTypeId="2" ParentId="3542" CreationDate="2011-03-19T12:09:23.263" Score="2" ViewCount="" Body="&lt;p&gt;I cover table design in the seminars I offer. My sources are primarily Chapter 8 of Few’s &lt;em&gt;Show Me the Numbers&lt;/em&gt; and a paper by Martin Koschat:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Koschat, Martin. 2005. “A Case for Simple Tables,” The &lt;em&gt;American Statistician&lt;/em&gt; 59:1, 31-40.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, Howard Wainer discusses table design in &lt;em&gt;Visual Revelations&lt;/em&gt;.&lt;/p&gt;&#xA;" OwnerUserId="3792" LastActivityDate="2011-03-19T12:09:23.263" />
  <row Id="8492" PostTypeId="2" ParentId="8483" CreationDate="2011-03-19T12:43:19.813" Score="2" ViewCount="" Body="&lt;p&gt;You will find help on allocating categorical variables with &lt;a href=&quot;http://www.ats.ucla.edu/stat/spss/dae/logit.htm&quot; rel=&quot;nofollow&quot;&gt;UCLA's tutorials&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;logistic regression HIV with age sex&#xA;   /categorical = sex.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You might also find help &lt;a href=&quot;http://www.childrens-mercy.org/stats/weblog2004/categorical.asp&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1351" LastActivityDate="2011-03-19T12:43:19.813" />
  <row Id="8495" PostTypeId="2" ParentId="8472" CreationDate="2011-03-19T13:49:39.730" Score="8" ViewCount="" Body="&lt;p&gt;I would say, based on your question, that there is no difference between the three tests.  This is in the sense that you can always choose A, B, and C such that the same decision is arrived at regardless of what criterion you are using.  Although you need to have the p-value be based on the same statistic (i.e. the Z-score)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To Use the Z-score, both the mean $\mu$ and variance $\sigma^2$ are assumed to be known, and the distribution is assumed normal (or asymptotically/approximately normal).  Suppose the p-value criterion is the usual 5%.  Then we have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p=Pr(Z&amp;gt;z)&amp;lt;0.05\rightarrow Z&amp;gt;1.645\rightarrow \frac{X-\mu}{\sigma}&amp;gt;1.645\rightarrow X &amp;gt; \mu+1.645\sigma$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So we have the triple $(0.05, 1.645, \mu+1.645\sigma)$ which all represent the same cut-offs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that the same correspondence will apply to the t-test, although the numbers will be different.  The two tails test will also have a similar correspondence, but with different numbers.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-03-19T13:49:39.730" CommentCount="1" />
  <row Id="8496" PostTypeId="5" CreationDate="2011-03-19T13:54:41.287" Score="0" ViewCount="" Body="&lt;p&gt;These guidelines are for &lt;strong&gt;those who are asking&lt;/strong&gt; and &lt;strong&gt;those who would answer&lt;/strong&gt; homework-related questions.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;They are rooted in two principles:&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;It is okay to ask about homework&lt;/strong&gt;.  This site exists to help people learn and provide a standard repository for questions in statistics and machine learning, both simple and complex, and this includes helping students.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Providing an answer that doesn't help a student learn is not in the student's own best interest. Therefore you might choose to &lt;strong&gt;treat homework questions differently&lt;/strong&gt; than other questions.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;Asking about homework&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Make a &lt;strong&gt;good faith attempt&lt;/strong&gt; to solve the problem yourself first. If we can't see enough work on your part your question will likely be booed off the stage; it will be voted down and closed.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Ask about &lt;strong&gt;specific problems&lt;/strong&gt; you have encountered in your initial efforts. If you can't do that yet, try some more of your own work first or searching for more general help.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Admit that the question is homework.&lt;/strong&gt;  Trying to hide it will just get the question closed faster. Do this by mentioning that it is homework in the question text and by using the homework tag.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Be aware of school policy&lt;/strong&gt; (if relevant). If your school has a policy regarding outside help on homework, make sure you are aware of it before you ask for/receive help here. If there are specific restrictions (for example, you can receive help, but not full solutions), include them in the question so that those providing assistance can keep you out of trouble.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Never use a solution you don't understand&lt;/strong&gt;. It definitely won't help you later (after school, in later assignments, on tests, etc.) and it could be, at best, very embarrassing if you are asked to explain what you turned in.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;Answering homework questions&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Try to provide explanations that will &lt;strong&gt;lead the asker&lt;/strong&gt; in the correct direction. Genuine understanding is the real goal for students, but trying to provide that is seldom unappreciated for any question.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;It's usually better &lt;strong&gt;not to provide a complete solution&lt;/strong&gt; (or code sample) if you believe it would not help the student, using your best judgment. You can use pseudo-code and general descriptions first.  In the spirit of creating a resource, you may come back after a suitable amount of time and edit your response to include more details, if the question seems like such information will have lasting value. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Don't downvote others&lt;/strong&gt; who answer homework questions in good faith, even if they break these guidelines. It's not always obvious at first glance that a question is homework, especially when you're not expecting to see it here. It is a good idea to suggest editing the response in a comment.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Don't ridicule&lt;/strong&gt; a student because they haven't yet learned something obvious or developed the good habits you'd expect from an expert. Do add a respectful comment or answer that points them towards best practices and better style.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Don't downvote a homework question&lt;/strong&gt; that follows the guidelines and was asked in &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Don't edit a question to add the homework tag&lt;/strong&gt;. If there's any room for doubt at all, it's best to leave it as is. Instead, add a comment first requesting that the asker clarify the situation.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;(Adapted from an &lt;a href=&quot;http://meta.stackoverflow.com/questions/10811/how-to-ask-and-answer-homework-questions/10812#10812&quot;&gt;SO post by Joe Coehoorn&lt;/a&gt; as suggested in a &lt;a href=&quot;http://meta.stats.stackexchange.com/q/12/919&quot;&gt;meta discussion&lt;/a&gt;.)&lt;/p&gt;&#xA;" OwnerUserId="-1" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-19T13:54:41.287" LastActivityDate="2011-03-19T13:54:41.287" />
  <row Id="8497" PostTypeId="4" CreationDate="2011-03-19T13:54:41.287" Score="0" ViewCount="" Body="A routine question from a textbook, course, or test used for a class or self-study.  This community's policy is to &quot;provide helpful hints.&quot;" OwnerUserId="-1" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-19T13:54:41.287" LastActivityDate="2011-03-19T13:54:41.287" />
  <row Id="8498" PostTypeId="2" ParentId="8462" CreationDate="2011-03-19T15:02:19.393" Score="0" ViewCount="" Body="&lt;p&gt;Your charts are a diurnal hourly-average bar chart, and a one-week daily-average bar chart, respectively.&lt;/p&gt;&#xA;" OwnerUserId="3794" LastActivityDate="2011-03-19T15:02:19.393" />
  <row Id="8499" PostTypeId="2" ParentId="8455" CreationDate="2011-03-19T15:10:37.453" Score="2" ViewCount="" Body="&lt;p&gt;The underlying difficulty behind the question is that situations that have been anticipated, have generally been planned for, with mitigation measures in place. Which means that the situation should not even turn into a serious accident.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The serious accidents stem from &lt;strong&gt;unanticipated&lt;/strong&gt; situations. Which means that you cannot assess probabilities for them - they are your Rumsfeldian unknown unknowns.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The assumption of independence is clearly invalid - Fukushima Daiichi shows that. Nuclear plants can have common-mode failures. (i.e. more than one reactor becoming unavailable at once, due to a common cause).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Although probabilities cannot be quantitatively calculated, we can make some qualitative assertions about common-mode failures.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example: if the plants are all built to the same design, then they are more likely to have common-mode failures (for example the known problem with pressurizer cracks in EPRs / PWRs)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the plant sites share geographic commonalities, they are more likely to have common-mode failures: for example, if they all lie on the same earthquake fault line; or if they all rely on similar rivers within a single climatic zone for cooling (when a very dry summer can cause all such plants to be taken offline).&lt;/p&gt;&#xA;" OwnerUserId="3794" LastActivityDate="2011-03-19T15:10:37.453" CommentCount="2" />
  <row Id="8500" PostTypeId="2" ParentId="8459" CreationDate="2011-03-19T15:40:13.427" Score="1" ViewCount="" Body="&lt;p&gt;I think that there might be a conceptual problem with this approach. If your piece of paper is not flat it is possible that a &quot;kink&quot; in the paper at a point with no ink dot might be higher than surrounding areas with ink dots. The proposed algorithm might inadvertently average away the very points of interest. Also &quot;I then noticed that repeating this process many times&quot; might introduce spuriousness due to the Slutsky Yule Effect. Might I suggest that you use more specialised approaches e.g. consider your data to be an image and use the relevant tools from a package such as &lt;a href=&quot;http://octave.sourceforge.net/image/index.html&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt;?   &lt;/p&gt;&#xA;" OwnerUserId="226" LastEditorUserId="226" LastEditorDisplayName="" LastEditDate="2011-03-19T15:56:45.143" LastActivityDate="2011-03-19T15:56:45.143" />
  <row Id="8501" PostTypeId="1" CreationDate="2011-03-19T15:56:42.623" Score="2" ViewCount="171" Body="&lt;p&gt;Hello&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm using the following function to calculate Edwards R^2 (formula 19 in Edwards et al. 2008) of a mixed effects model (I hope the implementation is correct):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;r2lmer &amp;lt;- function(model) {&#xA;   require(aod) # need the aod package for wald.test function&#xA;   if (class(model) != &quot;mer&quot;) stop(&quot;mer object expected&quot;)&#xA;   n &amp;lt;- model@dims[['n']] # the number of observations&#xA;   p &amp;lt;- model@dims[['p']] # number of parameters&#xA;   df.mod &amp;lt;- n - p&#xA;&#xA;   wald.model &amp;lt;- wald.test(b = fixef(model), Sigma = vcov(model), df = df.mod, Terms = 2:p)&#xA;   wald.F &amp;lt;- as.numeric(wald.model$result$Ftest[1])&#xA;   ( (p - 1) * 1 / df.mod * wald.F ) / (1 + (p - 1 ) * 1 / df.mod * wald.F )&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In order to internally valdiate a mixed effects model, I would like to estimate the variance explained when the model is applied to a new data set, that is, when only the data changes, but not the parameter estimates. I'm not sure if it's even theoretically possible. Any help is highly appreciated.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Edit&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;If it's not a good idea to use an R^2 statistic for LMM, what other performance measures could one use to internally validate LMM? I have a LMM with just one random effects factor (varying intercept due to repeated measures) and several fixed effects factors. I'am primarily interested in the fixed effects and it is my understanding that Edward's R^2 is a good measure for the variance explained by the fixed effects. Edward's R^2 is also recommended for cross-validation in this paper:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cheng et al. Real longitudinal data analysis for real people: Building a good enough mixed model. Statistics in Medicine. 2010, 29 504-520&lt;/p&gt;&#xA;" OwnerUserId="3795" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-04-05T15:50:11.470" LastActivityDate="2011-04-18T10:01:20.853" Title="Variance explained of a mixed effects model in a new data set" Tags="&lt;r&gt;&lt;mixed-model&gt;&lt;variance&gt;&lt;validation&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="8502" PostTypeId="1" AcceptedAnswerId="8684" CreationDate="2011-03-19T16:32:07.477" Score="4" ViewCount="183" Body="&lt;p&gt;I have two sets of coefficients from similar data taken at different times.  What I want to do is combine the two sets of coefficients giving greater weight to the more most recent set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The goal is building a predictive model.  So say I have dataset A from 2009, and dataset B from 2010.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My coefficients for A are:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;param1: 0.33&#xA;param2: 1.224&#xA;param3: -0.119&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My coefficients for B are:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;param1: 0.42&#xA;param2: 1.309&#xA;param3: -0.011&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If I wanted B to be considered twice as important, would it be sound to use a formula like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(2*B + A) / 3 = New Coeeficient&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And do that for each parameter?  Or am I suggesting something that is fundamentally flawed?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general could one combine coefficients effectively using the basic forumula:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(Weight * DatasetACoeffcient + DatasetBCoeffient) / (Weight + 1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit&lt;/strong&gt;&#xA;This is a multivariate linear regression problem where the datasets may not be available when someone decides something like this needs to be done.&lt;/p&gt;&#xA;" OwnerUserId="3491" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-29T01:05:55.050" LastActivityDate="2011-04-29T01:05:55.050" Title="Combining 2 sets of coefficients, weighting one of the sets" Tags="&lt;time-series&gt;&lt;multivariate-analysis&gt;&lt;predictive-models&gt;" AnswerCount="4" CommentCount="3" FavoriteCount="1" />
  <row Id="8503" PostTypeId="2" ParentId="8502" CreationDate="2011-03-19T16:40:18.300" Score="1" ViewCount="" Body="&lt;p&gt;There is no reason accounting for the use of convex linear combinations of coefficients in order to &quot;average&quot; two models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At best, you could consider the three coefficients for each dataset are realizations of the same three random variables, and you would be interested in the distribution of each random variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I would do would be to fit the model again with a new dataset (of size $n$) consisting of a random sample of size $\lambda\times n$ taken from the B dataset, and $\left(1-\lambda\right)\times n$ of the A dataset. You could use $\lambda=\frac{2}{3}$ for instance.&lt;/p&gt;&#xA;" OwnerUserId="1351" LastEditorUserId="1351" LastEditorDisplayName="" LastEditDate="2011-03-19T16:46:40.243" LastActivityDate="2011-03-19T16:46:40.243" CommentCount="2" />
  <row Id="8504" PostTypeId="1" CreationDate="2011-03-19T17:49:13.440" Score="4" ViewCount="171" Body="&lt;p&gt;car packages's &lt;a href=&quot;http://finzi.psych.upenn.edu/R/library/car/html/Ellipses.html&quot; rel=&quot;nofollow&quot;&gt;ellipse function&lt;/a&gt; asks for a &lt;code&gt;radius&lt;/code&gt; parameter. In help says that is the &quot;radius of circle generating the ellipse&quot;. Could you please tell me which circle is this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you very much&lt;/p&gt;&#xA;" OwnerUserId="339" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-19T18:55:07.057" LastActivityDate="2011-03-19T19:40:46.057" Title="Help in drawing confidence ellipse" Tags="&lt;r&gt;&lt;confidence-interval&gt;&lt;multivariate-analysis&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="1" />
  <row Id="8505" PostTypeId="1" CreationDate="2011-03-19T17:58:42.400" Score="6" ViewCount="149" Body="&lt;p&gt;A Poission regression is a &lt;a href=&quot;http://en.wikipedia.org/wiki/Generalized_linear_model&quot; rel=&quot;nofollow&quot;&gt;GLM&lt;/a&gt; with a log-link function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An alternative way to model non-normally distributed count data is to preprocess by taking the log (or rather, log(1+count) to handle 0's).  If you do a least-squares regression on log-count responses, is that related to a Poisson regression?  Can it handle similar phenomena?&lt;/p&gt;&#xA;" OwnerUserId="3799" LastEditorUserId="3799" LastEditorDisplayName="" LastEditDate="2011-03-21T00:45:59.103" LastActivityDate="2011-03-21T00:45:59.103" Title="Poisson regression vs. log-count least-squares regression?" Tags="&lt;regression&gt;&lt;poisson&gt;&lt;glm&gt;" AnswerCount="2" CommentCount="3" />
  <row Id="8506" PostTypeId="2" ParentId="8505" CreationDate="2011-03-19T18:27:09.653" Score="5" ViewCount="" Body="&lt;p&gt;Hi !&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the one hand, in a Poisson regression, the left-hand side of the model equation is the logarithm of the expected count: $\log(E[Y|x])$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, in a &quot;standard&quot; linear model, the left-hand side is the expected value of the normal response variable: $E[Y|x]$. In particular, the link function is the identity function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, let us say $Y$ is a Poisson variable and that you intend to normalise it by taking the log: $Y&amp;#39; = \log(Y)$. Because $Y&amp;#39;$ is supposed to be normal you plan to fit the standard linear model for which the left-hand side is $E[Y&amp;#39;|x] = E[\log(Y)|x]$. But, in general, $E[\log(Y) | x] \neq \log(E[Y|x])$. As a consequence, these two modelling approaches are different. &lt;/p&gt;&#xA;" OwnerUserId="3019" LastActivityDate="2011-03-19T18:27:09.653" CommentCount="2" />
  <row Id="8507" PostTypeId="2" ParentId="8504" CreationDate="2011-03-19T18:49:26.673" Score="3" ViewCount="" Body="&lt;p&gt;An ellipse can be parametrized as the affine image of any given circle. If we consider the unit circle:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$x=a \cos (t)$$&#xA;$$y=b \sin (t)$$&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ellipse(center, shape, radius, log=&quot;&quot;, center.pch=19, center.cex=1.5, &#xA;  segments=51, add=TRUE, xlab=&quot;&quot;, ylab=&quot;&quot;, &#xA;   col=palette()[2], lwd=2, fill=FALSE, fill.alpha=0.3, grid=TRUE, ...)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can notice the &lt;code&gt;ellipse&lt;/code&gt; function asks for the center and the radius of the circle, as well as the covariance matrix, which is equivalent to giving the parameters of the affine transformation.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;center  2-element vector with coordinates of center of ellipse.&#xA;shape   2 * 2 shape (or covariance) matrix.&#xA;radius  radius of circle generating the ellipse.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Let us have a look at the &lt;code&gt;car&lt;/code&gt; package function:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ellipse &amp;lt;- t(center + radius * t(unit.circle %*% chol(shape)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The &lt;code&gt;radius&lt;/code&gt; parameter can be set to 1 if you want to use the covariance matrix directly for the &lt;code&gt;shape&lt;/code&gt; parameter. I believe it was introduced to help people use normalized matrices instead if they prefer so.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Edit: As mentioned in whuber's comment, the two ellipses below are the same.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; library(car)&#xA;&amp;gt; s=matrix(c(1,0,0,1), nrow=2, ncol=2)&#xA;&amp;gt; plot(0, 0, xlim=c(-5,5), ylim=c(-5,5))&#xA;&amp;gt; ellipse(c(0,0), 4*s, 1)&#xA;&amp;gt; ellipse(c(0,0), s, 2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1351" LastEditorUserId="1351" LastEditorDisplayName="" LastEditDate="2011-03-19T19:40:46.057" LastActivityDate="2011-03-19T19:40:46.057" CommentCount="1" />
  <row Id="8508" PostTypeId="2" ParentId="8487" CreationDate="2011-03-19T19:59:33.950" Score="2" ViewCount="" Body="&lt;p&gt;According to &lt;a href=&quot;http://books.google.com/books?id=cQxN792ttyEC&amp;amp;pg=PA238&quot; rel=&quot;nofollow&quot;&gt;p238&lt;/a&gt; of standard text on meta-analysis in social science &lt;a href=&quot;http://books.google.com/books?id=cQxN792ttyEC&quot; rel=&quot;nofollow&quot;&gt;&lt;em&gt;The Handbook of Research Synthesis&lt;/em&gt;&lt;/a&gt;, the variance of Cohen's $d$ is&#xA;$$\left( \frac{n_1 + n_2}{n_1 n_2} + \frac{d^2}{2(n_1+n_2-2)}\right) \left(\frac{n_1 + n_2}{n_1+n_2-2} \right), $$&#xA;where $n_1$ and $n_2$ are the sample sizes of the two groups being compared and $d$ is Cohen's $d$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Taking the square-root of this variance gives the standard error of $d$, needed as input by several of the user-written &lt;a href=&quot;http://www.stata.com/support/faqs/stat/meta.html&quot; rel=&quot;nofollow&quot;&gt;meta-analysis packages for Stata&lt;/a&gt;. (Some of them also accept confidence intervals as input, but they simply convert them to standard errors internally anyway.)&lt;/p&gt;&#xA;" OwnerUserId="449" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-03-19T20:05:45.363" LastActivityDate="2011-03-19T20:05:45.363" />
  <row Id="8509" PostTypeId="2" ParentId="8485" CreationDate="2011-03-19T20:19:38.393" Score="5" ViewCount="" Body="&lt;p&gt;I'd start with:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Casella, George; George, Edward I. (1992). &quot;&lt;a href=&quot;http://www.jstor.org/stable/2685208&quot; rel=&quot;nofollow&quot;&gt;Explaining the Gibbs sampler&lt;/a&gt;&quot;. &lt;em&gt;The American Statistician&lt;/em&gt; &lt;strong&gt;46&lt;/strong&gt; (3): 167–174.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;Abstract&lt;/strong&gt;: Computer-intensive algorithms, such as the Gibbs sampler, have become increasingly popular statistical tools, both in applied and theoretical work. The properties of such algorithms, however, may sometimes not be obvious. Here we give a simple explanation of how and why the Gibbs sampler works. We analytically establish its properties in a simple case and provide insight for more complicated cases. There are also a number of examples.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;The American Statistician&lt;/em&gt; is often a good source for short(ish) introductory articles that don't assume any prior knowledge of the topic, though they do assume you have the background in probability and statistics that could reasonably be expected of a member of the &lt;a href=&quot;http://www.amstat.org&quot; rel=&quot;nofollow&quot;&gt;American Statistical Association&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-03-19T20:19:38.393" />
  <row Id="8510" PostTypeId="2" ParentId="8485" CreationDate="2011-03-19T21:18:56.360" Score="3" ViewCount="" Body="&lt;p&gt;One online article that really helped me understand Gibbs Sampling is &lt;a href=&quot;http://www.arbylon.net/publications/text-est.pdf&quot; rel=&quot;nofollow&quot;&gt;Parameter estimation for text analysis&lt;/a&gt; by Gregor Heinrich. It's not a general Gibbs sampling tutorial but it discusses it in terms of latent dirichlet allocation, a fairly popular Bayesian model for document modeling. It goes into the math in fair detail.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One that goes into even more exhaustive mathematical detail is &lt;a href=&quot;http://drum.lib.umd.edu/handle/1903/10058&quot; rel=&quot;nofollow&quot;&gt;Gibbs Sampling for the Uninitiated&lt;/a&gt;. And I mean exhaustive in that it assumes you know some multivariate calculus and then lays out every step from that point. So while there's a lot of math, none of it is advanced.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I assume these will be more useful to you then something that derives more advanced results, such as those that prove why Gibbs sampling converges to the correct distribution. The references I point out don't prove this.&lt;/p&gt;&#xA;" OwnerUserId="3167" LastActivityDate="2011-03-19T21:18:56.360" CommentCount="1" />
  <row Id="8511" PostTypeId="1" CreationDate="2011-03-19T22:44:06.767" Score="4" ViewCount="236" Body="&lt;p&gt;Christopher Manning's &lt;a href=&quot;http://nlp.stanford.edu/~manning/courses/ling289/logistic.pdf&quot; rel=&quot;nofollow&quot;&gt;writeup on logistic regression in R&lt;/a&gt; shows a logistic regression in R as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ced.logr &amp;lt;- glm(ced.del ~ cat + follows + factor(class), &#xA;  family=binomial)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Some output:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; summary(ced.logr)&#xA;Call:&#xA;glm(formula = ced.del ~ cat + follows + factor(class),&#xA;    family = binomial(&quot;logit&quot;))&#xA;Deviance Residuals:&#xA;Min            1Q    Median       3Q      Max&#xA;-3.24384 -1.34325   0.04954  1.01488  6.40094&#xA;&#xA;Coefficients:&#xA;              Estimate Std. Error z value Pr(&amp;gt;|z|)&#xA;(Intercept)   -1.31827    0.12221 -10.787 &amp;lt; 2e-16&#xA;catd          -0.16931    0.10032  -1.688 0.091459&#xA;catm           0.17858    0.08952   1.995 0.046053&#xA;catn           0.66672    0.09651   6.908 4.91e-12&#xA;catv          -0.76754    0.21844  -3.514 0.000442&#xA;followsP       0.95255    0.07400  12.872 &amp;lt; 2e-16&#xA;followsV       0.53408    0.05660   9.436 &amp;lt; 2e-16&#xA;factor(class)2 1.27045    0.10320  12.310 &amp;lt; 2e-16&#xA;factor(class)3 1.04805    0.10355  10.122 &amp;lt; 2e-16&#xA;factor(class)4 1.37425    0.10155  13.532 &amp;lt; 2e-16&#xA;(Dispersion parameter for binomial family taken to be 1)&#xA;Null deviance: 958.66 on 51 degrees of freedom&#xA;Residual deviance: 198.63 on 42 degrees of freedom&#xA;AIC: 446.10&#xA;Number of Fisher Scoring iterations: 4&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;He then goes into some detail about how to interpret coefficients, compare different models, and so on.  Quite useful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, how much variance does the model account for?  A &lt;a href=&quot;http://www.ats.ucla.edu/stat/stata/output/old/lognoframe.htm&quot; rel=&quot;nofollow&quot;&gt;Stata page on logistic regression&lt;/a&gt; says:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Technically, R2 cannot be computed the same way in logistic regression as it is in OLS regression. The pseudo-R2, in logistic regression, is defined as (1 - L1)/L0, where L0 represents the log likelihood for the &quot;constant-only&quot; model and L1 is the log likelihood for the full model with constant and predictors. &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I understand that at the high level. The constant-only model would be without any of the parameters (only the intercept term).  Log likelihood is a measure of how closely the parameters fit the data.  In fact, Manning sort of hints that the deviance might be -2 log L. Perhaps null deviance is constant-only and residual deviance is -2 log L of the model?  However, I'm not crystal clear on it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone verify how one actually computes the pseudo-R^2 in R using this example?&lt;/p&gt;&#xA;" OwnerUserId="2849" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-20T23:28:30.353" LastActivityDate="2011-04-20T23:28:30.353" Title="How to calculate pseudo-$R^2$ from R's logistic regression?" Tags="&lt;r&gt;&lt;logistic-regression&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="8512" PostTypeId="1" CreationDate="2011-03-20T04:19:28.777" Score="3" ViewCount="28" Body="&lt;p&gt;Suppose I have some stochastic process $X_t$. At each time $t$, I receive an estimated probability distribution for $x_t$, followed by an observation $x_t$. After receiving a set of observations ${x_1, \ldots, x_n}$, I want to go back and re-estimate the probability distribution for each $x_t$, $1 \le t \le n$. What are some ways of doing this? Let's make the assumption that the estimated probability distributions are &quot;pretty good&quot;, so that we do want to use the information they contain; what other assumptions do I need in order to make this an interesting/tractable problem?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, some (potentially totally misguided) thoughts I had were:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Suppose $X_t$ is a &quot;first-order continuous&quot; model: $x_t$ depends only on $x_{t-1}$, and if $x_{t-1}, x_{s-1}$ are fairly close, then the probability distributions for $x_t, x_s$ should also be fairly close. Then to revise my estimate of the probability distribution for $x_t$, I could use a kind of kernel density bootstrap sample and mix it with the original estimate: take all points $x_s$ such that $x_{s-1}$ is close to $x_{t-1}$, use these points $x_s$ in some weighted fashion to form a kernel density estimate $K$, and average $K$ with the original estimated probability distribution to get a new estimated probability distribution for $x_t$.&lt;/li&gt;&#xA;&lt;li&gt;Suppose $X_t$ is a linear Gaussian model (so that it satisfies the assumptions of a standard Kalman filter). I could try forming a different kind of bootstrap sample: for each $x_t$, use a window around its estimated probability distribution to sample a new observation $x&amp;#39;_t$, and then use Kalman smoothing on the observations ${x&amp;#39;_t}$ to get a set of smoothed observations $\tilde{x_t}$. Use these bootstrap samples to generate new estimated probability distributions.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;This is a mostly random question a friend asked me a while ago that I just thought of again, so I don't have a particular application (and hence why I don't have a concrete set of assumptions on the process).&lt;/p&gt;&#xA;" OwnerUserId="1106" LastActivityDate="2011-03-20T04:19:28.777" Title="Updating a set of estimated forecasts" Tags="&lt;estimation&gt;&lt;forecasting&gt;&lt;stochastic-processes&gt;&lt;smoothing&gt;" CommentCount="1" />
  <row Id="8513" PostTypeId="1" CreationDate="2011-03-20T06:47:01.650" Score="3" ViewCount="107" Body="&lt;p&gt;Let's say $y$ is a linear function of $x$ and a dummy $d$. My hypothesis is that $d$ itself is like a hedonistic index of a vector of other variables, $Z$. I have support for this in a $MANOVA$ of $Z$ (i.e. $z_1$, $z_2$, ..., $z_n$) on $d$. Is there any way to test the &lt;em&gt;equivalence&lt;/em&gt; of these two models:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Model 1: $y = b_0 + b_1 \cdot x + b_2\cdot d + e_1$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Model 2: $y = g_0 + Z\cdot G + e_2$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $G$ is the column vector of parameters.&lt;/p&gt;&#xA;" OwnerUserId="3671" LastEditorUserId="2645" LastEditorDisplayName="" LastEditDate="2011-03-20T09:53:58.310" LastActivityDate="2011-03-20T09:53:58.310" Title="Test equivalence of non-nested models" Tags="&lt;r&gt;&lt;hypothesis-testing&gt;&lt;regression&gt;&lt;model-selection&gt;" AnswerCount="1" />
  <row Id="8514" PostTypeId="1" CreationDate="2011-03-20T07:38:30.500" Score="11" ViewCount="232" Body="&lt;p&gt;I am by no means good in statistics, but I think I have come to the right place.&#xA;My question is simple:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My problem consists of comparing the population of several states in a small country, but some states have a population of 3000,000 and some a population of 2,000.&lt;br&gt;&#xA;I am painting it on a map, and the &quot;intensity&quot; of the color depends on how the population of every state compares to the population of the whole country.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is that the states with a lot of population are shown with really intense colors and the small states barely have any color.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there an easy way to &quot;normalize&quot; or make the data comparable?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I dont know if I am explaining myself properly but I hope some one can help me.&#xA;Please comment if my question is not clear and I will clarify.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you for your help!&lt;/p&gt;&#xA;" OwnerUserId="3803" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-20T10:11:32.580" LastActivityDate="2011-05-13T21:01:47.547" Title="How to make a good color intensity scale?" Tags="&lt;data-visualization&gt;" AnswerCount="5" CommentCount="5" FavoriteCount="2" />
  <row Id="8515" PostTypeId="1" AcceptedAnswerId="8534" CreationDate="2011-03-20T07:49:43.677" Score="4" ViewCount="124" Body="&lt;p&gt;Positive stable distributions are described by four parameters: the skewness parameter $\beta\in[-1,1]$, the scale parameter $\sigma&amp;gt;0$, the location parameter $\mu\in(-\infty,\infty)$, and the so-called index parameter $\alpha\in(0,2]$. When $\beta$ is zero the distribution is symmetric around $\mu$, when it is positive (resp. negative) the distribution is skewed to the right (resp. to the left). Stable distributions allow fat tails when $\alpha$ decreases. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When $\alpha$ is strictly less than one and $\beta=1$ the support of the distribution restricts to $(\mu,\infty)$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The density function only has a closed-form expression for some particular combinations of values for the parameters. When $\mu=0$, $\alpha&amp;lt;1$, $\beta=1$, and $\sigma=\alpha$ it is (see formula (4.4) &lt;a href=&quot;http://kb.unipune.ac.in/bitstream/123456789/200/1/9.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$f(y) = -\frac{1}{\pi y} \sum_{k=1}^{\infty} \frac{\Gamma(k\alpha+1)}{k!} (-y^{-\alpha})^k \sin(\alpha k \pi)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It has infinite mean and variance. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to use that density in R. I use&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; alpha &amp;lt;- ...&#xA;&amp;gt; dstable(y, alpha=alpha, beta=1, gamma=alpha, delta=0, pm=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where the &lt;a href=&quot;http://math.furman.edu/~dcs/courses/math47/R/library/fBasics/html/013A-StableDistribution.html&quot; rel=&quot;nofollow&quot;&gt;dstable&lt;/a&gt; function comes with the fBasics package.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can you confirm this is the right way to compute that density in R?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you in advance!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One reason why I am suspicious is that, in the output, the value of &lt;em&gt;delta&lt;/em&gt; is different from that in the input. Example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; library(fBasics)&#xA;&amp;gt; alpha &amp;lt;- 0.4&#xA;&amp;gt; dstable(4, alpha=alpha, beta=1, gamma=alpha, delta=0, pm=1)&#xA;[1] 0.02700602&#xA;attr(,&quot;control&quot;)&#xA;   dist alpha beta gamma    delta pm&#xA;stable   0.4    1   0.4 0.290617  1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="3019" LastEditorUserId="3019" LastEditorDisplayName="" LastEditDate="2011-03-20T11:05:15.790" LastActivityDate="2011-03-20T20:50:31.327" Title="The positive stable distribution in R" Tags="&lt;r&gt;&lt;positive-stable&gt;" AnswerCount="2" />
  <row Id="8516" PostTypeId="2" ParentId="8514" CreationDate="2011-03-20T08:08:21.577" Score="3" ViewCount="" Body="&lt;p&gt;You could divide by the total population. This would ensure that everything lies between 0 and 1. If the scales are still too disparate, consider a log scale.&lt;/p&gt;&#xA;" OwnerUserId="3786" LastActivityDate="2011-03-20T08:08:21.577" />
  <row Id="8517" PostTypeId="2" ParentId="8514" CreationDate="2011-03-20T08:10:04.853" Score="5" ViewCount="" Body="&lt;p&gt;Good question, One solution is to rescale the colors to have them more uniformly distributed, or to a distribution with lower tails... but then your legend has to be clear enough because deforming the scale, somehow, is unfair... &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, in R, rescaling a normal to a uniform . (what you have maybe goes more the other way since you have large tails and you want them smaller, but the principle is the same) &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X=array(rnorm(10000),c(100,100))&#xA;ramp=colorRamp(c(&quot;blue&quot;,&quot;cyan&quot;,&quot;white&quot;,&quot;yellow&quot;,&quot;red&quot;),space =&quot;rgb&quot;)&#xA;kleur &amp;lt;- rgb( ramp(seq(0,1,length=200)),max = 255)&#xA;par(mfrow=c(1,2))&#xA;image(X,col=kleur)### image without rescaling&#xA;Fn=ecdf(X)&#xA;ScaledX=array(Fn(X),c(100,100))&#xA;image(ScaledX,col=kleur)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="223" LastEditorUserId="223" LastEditorDisplayName="" LastEditDate="2011-03-20T08:15:42.257" LastActivityDate="2011-03-20T08:15:42.257" CommentCount="1" />
  <row Id="8518" PostTypeId="2" ParentId="8358" CreationDate="2011-03-20T09:03:55.500" Score="1" ViewCount="" Body="&lt;p&gt;What time-lag might you expect between cases recorded, and fatality? What time-lag between start of treatment and impact on fatality rates?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If either of those numbers is much greater than one year, then there may be a case for aggregating all your data from first year of treatment impact (i.e. 1996+time to impact) to 2010, and just test to see if mean CFR rates vary. If possible, ask whoever rejected the time-series approach whether this would satisfactorily deal with the confounding factors that concern them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do look at specifics of the confounding factors: for example, if palliative care for terminal patients was better in State 2, then those patients originally in State 1 for whom treatment had failed, might move to State 2 for their final days/weeks/months of life; in that case, their deaths would be registered in state 2, and the numbers you have won't provide you much useful information, unless you could get the numbers of terminal migrations between states.&lt;/p&gt;&#xA;" OwnerUserId="3794" LastActivityDate="2011-03-20T09:03:55.500" CommentCount="5" />
  <row Id="8519" PostTypeId="2" ParentId="8513" CreationDate="2011-03-20T09:09:12.033" Score="2" ViewCount="" Body="&lt;p&gt;To begin with you have to define the &lt;strong&gt;equivalence&lt;/strong&gt; concept. One may think that two models are equivalent when they do produce almost the same &lt;em&gt;forecasting accuracy&lt;/em&gt; (this one would be relevant for time series and panel data), another one could be interested in if &lt;em&gt;the fits from the model are close&lt;/em&gt;. The former is the object for different cross-validation (jack-knife usually or some out-of-sample tests, Rob's &lt;code&gt;accuracy()&lt;/code&gt; do this nicely), the latter goes for the minimization of some information criterion. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In microeconometrics the choice is &lt;a href=&quot;http://en.wikipedia.org/wiki/Bayesian_information_criterion&quot; rel=&quot;nofollow&quot;&gt;$BIC$&lt;/a&gt;, though you may also consider &lt;a href=&quot;http://en.wikipedia.org/wiki/Akaike_information_criterion&quot; rel=&quot;nofollow&quot;&gt;$AIC$&lt;/a&gt; if you are working with small sample sizes. Note, that the choice based on minimization of information criterion is also relevant for nested models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A nice discussion is given in must-have-it &lt;a href=&quot;http://books.google.ru/books?id=Zf0gCwxC9ocC&amp;amp;pg=PA278&amp;amp;lpg=PA278&amp;amp;dq=information+criteria+non-nested+models&amp;amp;source=bl&amp;amp;ots=CY24qI6KtY&amp;amp;sig=_kPXnMgkG3g-X3wQaF2oGJ955X8&amp;amp;hl=ru&amp;amp;ei=qbKFTf23DJDAswby69GQAw&amp;amp;sa=X&amp;amp;oi=book_result&amp;amp;ct=result&amp;amp;resnum=10&amp;amp;ved=0CGAQ6AEwCQ#v=onepage&amp;amp;q&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;book&lt;/a&gt; by Cameron and Trivedi (Chapter 8.5 provides excellent review of the methods), more specific theoretical details are found in Hong and Preston &lt;a href=&quot;http://www.stanford.edu/~doubleh/eco273B/nonnestmsc.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Roughly speaking, choosing from two models the more parsimonious (having less parameters to estimate, therefore more degrees of freedom) will be suggested as preferable. An information criterion introduces a special penalty function that restricts the inclusion of additional explanatory variables into linear model conceptually similar to the restrictions introduced by adjusted $R^2$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However you may not be just interested in choosing the model that minimizes selected information criterion. Equivalence concept implies that some &lt;em&gt;test statistic&lt;/em&gt; should be formulated. Therefore you may go for likelihood ratio tests either Cox or Voung $LR$ tests, Davidson-MacKinnon $J$ test.    &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, according to the tags, you may be just interested in $R$ functions:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(lmtest)&#xA;coxtest(fit1, fit2)&#xA;jtest(fit1, fit2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Where &lt;code&gt;fit1&lt;/code&gt; and &lt;code&gt;fit2&lt;/code&gt; are two non-nested fitted linear regression models, &lt;code&gt;coxtest&lt;/code&gt; is the Cox $LR$ test, and &lt;code&gt;jtest&lt;/code&gt; Davidson-MacKinnon $J$ test.&lt;/p&gt;&#xA;" OwnerUserId="2645" LastActivityDate="2011-03-20T09:09:12.033" CommentCount="2" />
  <row Id="8520" PostTypeId="2" ParentId="665" CreationDate="2011-03-20T09:27:37.320" Score="0" ViewCount="" Body="&lt;p&gt;In probability, the distribution is known and knowable in advance - you start with a known probability distribution function (or similar), and sample from it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In statistics, the distribution is unknown in advance. It may even be unknowable. Assumptions are hypothesised about the probability distribution behind observed data, in order to be able to apply probability theory to that data in order to know whether a null hypothesis about that data can be rejected or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a philosophical discussion about whether there is such a thing as probability in the real world, or whether it is an ideal figment of our mathematical imaginations, and all our observations can only be statistical.&lt;/p&gt;&#xA;" OwnerUserId="3794" LastActivityDate="2011-03-20T09:27:37.320" />
  <row Id="8521" PostTypeId="1" CreationDate="2011-03-20T09:36:05.143" Score="4" ViewCount="59" Body="&lt;p&gt;I have a set of systems where uncertainties accumulate within it. These aren't always purely additive - sometimes they are, sometimes they aren't. I've had some success in using fan-charts, bar charts with confidence-intervals, and box plots for communicating single items.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But how can I show how uncertainties accumulate and combine - while also showing the data points around which the uncertainties lie?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyone? Anyone? Bueller?&lt;/p&gt;&#xA;" OwnerUserId="3794" LastEditorUserId="3794" LastEditorDisplayName="" LastEditDate="2011-03-24T12:27:33.990" LastActivityDate="2011-03-24T12:27:33.990" Title="What graphical methods are useful for visualising how uncertainties aggregate?" Tags="&lt;data-visualization&gt;&lt;confidence-interval&gt;&lt;uncertainty&gt;" CommentCount="1" FavoriteCount="1" />
  <row Id="8522" PostTypeId="1" CreationDate="2011-03-20T12:30:45.973" Score="3" ViewCount="68" Body="&lt;p&gt;We're thinking of adding an interactive near real-time analytics functionality (a-la &quot;Google Analytics&quot;) to a product Movie Recommender Engine.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We need to let the user interactively create analyses deciding on a case by case basis the analysis dimensions (e.g. by Genre, by Actor, by Publisher), metrics (e.g. Views, Purchases, Ratings) and the time window of the analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We're considering several options like:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;charting libraries + custom built&lt;/li&gt;&#xA;&lt;li&gt;reporting engines (e.g. BIRT)&lt;/li&gt;&#xA;&lt;li&gt;OEM interactive analysis tools (e.g.&#xA;Tableau)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Our solution is Oracle and java-based. The front-end is built using the Liferay Portal framework&lt;/p&gt;&#xA;" OwnerUserId="3804" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-03-20T15:53:55.847" LastActivityDate="2011-05-24T14:18:05.413" Title="Suggestions for embedded interactive analytical functionalities?" Tags="&lt;data-visualization&gt;&lt;interactive-visualization&gt;&lt;recommender-system&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="1" />
  <row Id="8523" PostTypeId="2" ParentId="2181" CreationDate="2011-03-20T12:49:47.597" Score="1" ViewCount="" Body="&lt;p&gt;If you look at &lt;a href=&quot;http://www.plymouth.ac.uk/staff/phewson&quot; rel=&quot;nofollow&quot;&gt;Paul Hewison's webpage&lt;/a&gt;, you can find his free book on &lt;em&gt;Multivariate Statistics and R&lt;/em&gt;. Another free book is by Wolfgang Hardle and Leopold Simar. I have been&#xA;working my way through Johnson and Wichern, a book that has been used in the US for&#xA;over twenty years; you will have to buy this book. &lt;/p&gt;&#xA;" OwnerUserId="3805" LastEditorUserId="3582" LastEditorDisplayName="" LastEditDate="2011-03-21T22:38:56.370" LastActivityDate="2011-03-21T22:38:56.370" CommunityOwnedDate="2011-03-20T12:49:47.597" />
  <row Id="8524" PostTypeId="1" CreationDate="2011-03-20T13:06:51.123" Score="3" ViewCount="64" Body="&lt;p&gt;Here's what I know:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have read the chapter (p347ff) in Agresti, 1990, regarding dependent two-way tables, and I believe I understand the basics. My problem is that Agresti's model-based approaches seem to rely on large-sample theory. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have questions from 24 students where they rate something on a scale from 1-5. If I collapse 1-2=Agreement, 3=Neutrality, 4-5=Disagreement I still have relatively sparse data. The relevant question is the strength of evidence that the change in opinions between before and after is not due to random variation in response. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently I am using mh_test in the coin package in R. Here are some specific questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;How can I see what the mh_test is actually doing? When I type &lt;code&gt;print(mh_test)&lt;/code&gt; it will not show me the function, even though I can &lt;em&gt;use&lt;/em&gt; the function after loading the package.&lt;/li&gt;&#xA;&lt;li&gt;Does the distribution=&quot;approximate&quot; use a bootstrap method to obtain a p-value, and is that a way to do with the sparseness of data?&lt;/li&gt;&#xA;&lt;li&gt;Does anyone know of an exact version of the test of marginal homogeneity in this situation, and ideally how to implement such a test in R/S?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Thanks for reading. -DB&lt;/p&gt;&#xA;" OwnerUserId="3806" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-20T20:52:23.870" LastActivityDate="2011-03-20T20:52:23.870" Title="Is there an exact version of marginal homogeneity test?" Tags="&lt;r&gt;&lt;homogeneity&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="8525" PostTypeId="2" ParentId="8514" CreationDate="2011-03-20T13:25:00.360" Score="2" ViewCount="" Body="&lt;p&gt;I feel awkward asking it, but are you really committed to using &lt;em&gt;colour&lt;/em&gt; to portray a quantitative amount? Is there no way to put a bar in each state, whose height represents the quantity?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another way might be to show the map with areas representing the geographic areas, together with a map where each state's area is proportional to the population size - similar to how the &lt;a href=&quot;http://tumbledore.tumblr.com/post/249784948/sensory-homunculus-i-miss-this-high-school&quot; rel=&quot;nofollow&quot;&gt;sensory homunculus&lt;/a&gt; does. But that would be a painful amount of drawing - I don't know of any way to automate that (though it may exist)&lt;/p&gt;&#xA;" OwnerUserId="3794" LastActivityDate="2011-03-20T13:25:00.360" CommentCount="2" />
  <row Id="8526" PostTypeId="1" AcceptedAnswerId="8529" CreationDate="2011-03-20T13:33:59.587" Score="4" ViewCount="111" Body="&lt;p&gt;I have a rather basic question about &lt;a href=&quot;http://research.microsoft.com/en-us/um/people/cmbishop/downloads/Bishop-PPCA-JRSS.pdf&quot; rel=&quot;nofollow&quot;&gt;Probabilistic Principal Component Analysis&lt;/a&gt;, which I am now trying to apply to a real-world problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In PPCA, the crucial assumption is that the generating process of the observations in $R^n$ is $t=Wx +\sigma^2\epsilon$, where x are iid standard gaussians in $R^q$ (with $q \le n$) and $\epsilon$ are iid standard gaussian vectors in $R^n$. The authors find that the MLE solution is $\hat W=U_q (\Lambda_q - \sigma^2 I)^{1/2}$, where $\Lambda_q$ is the diagonal matrix of the $q$ largest eigenvalues of the empirical covariance matrix, and $U_q$ is the corresponding eigenvectors submatrix, and $\sigma^2$ is the average of the remaining smaller eigenvalues  (see sec. 3.2 of the linked paper).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is simple. The covariance matrix is $C=WW&amp;#39;+\sigma^2 I$. Directly plugging in the above result, we obtain $\hat C=U_q\Lambda_q U_q$ (the $\sigma^2$ terms cancel out). Can this be correct? The covariance matrix would be rank-deficient. Am I missing something?&lt;/p&gt;&#xA;" OwnerUserId="30" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-21T15:11:04.680" LastActivityDate="2011-03-21T15:11:04.680" Title="Question about probabilistic principal component analysis" Tags="&lt;pca&gt;&lt;dimensionality-reduction&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="8527" PostTypeId="2" ParentId="8455" CreationDate="2011-03-20T13:53:14.633" Score="0" ViewCount="" Body="&lt;p&gt;**Thanks to everyone who has responded to my question. The discussion has made me realize the original question was unclear. It should have read                                           &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Given a set of &lt;em&gt;n&lt;/em&gt; elements, each having independent probabilities of failure, &lt;em&gt;p&lt;/em&gt;, what is the total probability of at least one element of the set failing.&lt;/strong&gt;&lt;br&gt;&#xA; I was considering my set to be all of the reactor 'sites' (not each reactor) in the world, and each of their 'calculated, theoretical' probabilities of failure. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this case Bayer's answer does indeed apply.  We can debate just how well it corresponds to the world in reality, but this is the answer to the question I originally had in mind.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Again, thanks to all posters.**&lt;/p&gt;&#xA;" OwnerUserId="3777" LastActivityDate="2011-03-20T13:53:14.633" CommentCount="4" />
  <row Id="8528" PostTypeId="1" CreationDate="2011-03-20T14:07:14.917" Score="3" ViewCount="104" Body="&lt;p&gt;First of all I'd like to apologize for the vague title, I couldn't really formulate a better one just now, please feel free to change, or advice me to change, the title to make it better fit the core of the question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now about the question itself, I have been working on a software in which I have come across the idea of using an empirical distribution for sampling, however now that it's implemented I am not sure how to interpret it all. Allow me to describe what I have done, and why: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a bunch of calculations for a set of objects, yielding a final score. The score as it is however is very ad-hoc. So in order to make some sense out of the score of a particular object, what I do is to do a large number of (N = 1000) calculations of scores with mock/randomly generated values, yielding 1000 mock scores. Estimating an empirical &quot;score distribution&quot; for that particular object is then achieved by these 1000 mock score values. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have implemented this in Java (as the rest of the software is also written in Java environment) using &lt;a href=&quot;http://commons.apache.org/math/&quot; rel=&quot;nofollow&quot;&gt;Apache Commons Math library&lt;/a&gt;, in particular the &lt;a href=&quot;http://commons.apache.org/math/api-2.1/org/apache/commons/math/random/EmpiricalDistributionImpl.html&quot; rel=&quot;nofollow&quot;&gt;&lt;code&gt;EmpiricalDistImpl&lt;/code&gt; class&lt;/a&gt;. According to the documentation this class uses: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;what amounts to the Variable Kernel&#xA;  Method with Gaussian smoothing: &#xA;  Digesting the input file&lt;/p&gt;&#xA;  &#xA;  &lt;ol&gt;&#xA;  &lt;li&gt;Pass the file once to compute min and max.&lt;/li&gt;&#xA;  &lt;li&gt;Divide the range from min-max into binCount &quot;bins.&quot;&lt;/li&gt;&#xA;  &lt;li&gt;Pass the data file again, computing bin counts and univariate&#xA;  statistics (mean, std dev.) for each&#xA;  of the bins&lt;/li&gt;&#xA;  &lt;li&gt;Divide the interval (0,1) into subintervals associated with the bins,&#xA;  with the length of a bin's subinterval&#xA;  proportional to its count.&lt;/li&gt;&#xA;  &lt;/ol&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Now my question is, does it make sense to sample from this distribution in order to calculate some sort of an expected value? In other words how much could I trust/rely on this distribution? Could I for instance draw conclusion about significance of observing a score $S$ by checking the distribution? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I realize that this is perhaps an unorthodox way looking at a problem like this but I think it would be interesting to get a better grip over the concept of empirical distributions, and how they can/cant be used in analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance,&lt;/p&gt;&#xA;" OwnerUserId="3014" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-20T20:49:46.447" LastActivityDate="2011-03-21T08:29:02.110" Title="How to use/interpret empirical distribution?" Tags="&lt;distributions&gt;&lt;sampling&gt;&lt;java&gt;&lt;empirical&gt;" AnswerCount="1" CommentCount="5" />
  <row Id="8529" PostTypeId="2" ParentId="8526" CreationDate="2011-03-20T14:19:44.563" Score="6" ViewCount="" Body="&lt;p&gt;Unless I'm missing something, I think $U_q U_q&amp;#39; \neq I$ here (almsot surely, at least). The columns of $U_q$ are orthogonal, not the rows since the last $n-q$ columns are removed.&lt;/p&gt;&#xA;" OwnerUserId="26" LastActivityDate="2011-03-20T14:19:44.563" CommentCount="1" />
  <row Id="8530" PostTypeId="2" ParentId="8524" CreationDate="2011-03-20T14:40:12.293" Score="4" ViewCount="" Body="&lt;p&gt;1: &lt;code&gt;mh_test()&lt;/code&gt; is an &lt;code&gt;S3&lt;/code&gt; generic function, you can check what methods it has using &lt;code&gt;methods(&quot;mh_test&quot;)&lt;/code&gt;. To show the source of a non-visible method, you can use &lt;code&gt;getAnywhere()&lt;/code&gt; or &lt;code&gt;getS3method()&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(coin)                     # for mh_test()&#xA;methods(&quot;mh_test&quot;)                # available methods for mh_test(), all non-visible ...&#xA;getS3method(&quot;mh_test&quot;, &quot;table&quot;)   # get appropriate method -&amp;gt; uses SymmetryProblem ...&#xA;getS3method(&quot;mh_test&quot;, &quot;SymmetryProblem&quot;)    # get relevant method ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The code probably won't help you very much without reading the theory behind the &lt;code&gt;coin&lt;/code&gt; package which is explained in &lt;code&gt;vignette(&quot;coin_implementation&quot;)&lt;/code&gt;. To check what &lt;code&gt;mh_test()&lt;/code&gt; does in the asymptotic $\chi^{2}$ case, just compare it with the manual calculation:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; one  &amp;lt;- sample(LETTERS[1:3], 24, replace=TRUE)  # observations condition 1&#xA;&amp;gt; two  &amp;lt;- sample(LETTERS[1:3], 24, replace=TRUE)  # observations condition 2&#xA;&amp;gt; cTab &amp;lt;- table(one, two)  # cross tabulation&#xA;&amp;gt; addmargins(cTab)         # marginal frequencies&#xA;          two&#xA;one    A  B  C Sum&#xA;  A    3  3  0   6&#xA;  B    2  1  3   6&#xA;  C    5  3  4  12&#xA;  Sum 10  7  7  24&#xA;&#xA;&amp;gt; mh_test(cTab)            # test for marginal homogeneity&#xA;Asymptotic Marginal-Homogeneity Test&#xA;data:  response by groups (one, two) stratified by block&#xA;chi-squared = 2.6588, df = 2, p-value = 0.2646&#xA;&#xA;# manual calculation following textbook formulas: S will be the estimated&#xA;# covariance matrix for the differences in marginal frequencies&#xA;&amp;gt; S         &amp;lt;- -(cTab + t(cTab))&#xA;&amp;gt; diag(S)   &amp;lt;- rowSums(cTab) + colSums(cTab) - 2*diag(cTab)     # change diagonal&#xA;&amp;gt; keep      &amp;lt;- 1:(nrow(cTab)-1)              # last category is pre-determined&#xA;&amp;gt; d         &amp;lt;- rowSums(cTab) - colSums(cTab) # differences in marginal frequencies&#xA;&amp;gt; (chisqVal &amp;lt;- t(d[keep]) %*% solve(S[keep, keep]) %*% d[keep]) # test statistic&#xA;2.658824&#xA;&#xA;&amp;gt; (smmhDf &amp;lt;- nrow(cTab)-1)                   # degrees of freedom&#xA;[1] 2&#xA;&#xA;&amp;gt; (pVal &amp;lt;- 1-pchisq(chisqVal, smmhDf))       # p-value from chi-square distribution&#xA;0.2646329&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;2: &lt;code&gt;coin&lt;/code&gt; implements a permutation-test framework, not a bootstrapping framework. &lt;code&gt;distribution=approximate(B=9999)&lt;/code&gt; means that instead of using all possible permutations of the data for generating the distribution of the test statistic, it only uses a random sample of size &lt;code&gt;B&lt;/code&gt; of these permutations. The value of the test-statistic will be the same, but the p-value will differ from the $\chi^{2}$ approximation. IMHO, it's a good idea to compare the p-values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3: An exact permuation test might be done using functions in package &lt;code&gt;vegan&lt;/code&gt;, but I haven't tried that myself.&lt;/p&gt;&#xA;" OwnerUserId="1909" LastEditorUserId="1909" LastEditorDisplayName="" LastEditDate="2011-03-20T16:35:29.667" LastActivityDate="2011-03-20T16:35:29.667" />
  <row Id="8531" PostTypeId="2" ParentId="8455" CreationDate="2011-03-20T15:18:34.720" Score="0" ViewCount="" Body="&lt;p&gt;To answer the pure probabilistic question that J Presley presented, using bayer's notation (p=probability of an item failing), the the probability of &lt;strong&gt;at least&lt;/strong&gt; one element failing is 1-P(none fail)= 1-(1-p)^n. This type of calculation is common in system reliability where a bunch of components are linked in parallel, so that the system continues to function if at least one component is functioning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can still use this formula even if each plant item has a different failure probability (p_i). The formula would then be 1- (1-p_1)(1-p_2)...(1-p_n).&lt;/p&gt;&#xA;" OwnerUserId="1945" LastActivityDate="2011-03-20T15:18:34.720" CommentCount="2" />
  <row Id="8532" PostTypeId="2" ParentId="8522" CreationDate="2011-03-20T15:26:14.557" Score="1" ViewCount="" Body="&lt;p&gt;I love interactive visualization software like Spotfire and Tableau because it is easy to use and very insightful. My MBA students also become addicted.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am more familiar with Spotfire, so I can say that they have a nice solution (Spotfire Silver) that allows you to create a dashboard of visualizations and post it to the Web so that users can play with a structured visualization or answer particular queries of interest.&lt;/p&gt;&#xA;" OwnerUserId="1945" LastActivityDate="2011-03-20T15:26:14.557" CommentCount="3" />
  <row Id="8533" PostTypeId="2" ParentId="8505" CreationDate="2011-03-20T15:33:58.773" Score="0" ViewCount="" Body="&lt;p&gt;One obvious difference is that Poisson regression will yield integers as point predictions whereas log-count linear regression can yield non-integers.&lt;/p&gt;&#xA;" OwnerUserId="1945" LastActivityDate="2011-03-20T15:33:58.773" CommentCount="1" />
  <row Id="8534" PostTypeId="2" ParentId="8515" CreationDate="2011-03-20T15:45:20.570" Score="4" ViewCount="" Body="&lt;p&gt;What I think is happening is that in the output &lt;code&gt;delta&lt;/code&gt; may be reporting an internal location value, while in the input &lt;code&gt;delta&lt;/code&gt; is describing the shift. [There seems to be a similar issue with &lt;code&gt;gamma&lt;/code&gt; when &lt;code&gt;pm=2&lt;/code&gt;.]  So if you try increasing the shift to 2&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; dstable(4, alpha=0.4, beta=1, gamma=0.4, delta=2, pm=1)&#xA;[1] 0.06569375&#xA;attr(,&quot;control&quot;)&#xA;   dist alpha beta gamma    delta pm&#xA; stable   0.4    1   0.4 2.290617  1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;then you add 2 to the location value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With &lt;code&gt;beta=1&lt;/code&gt; and &lt;code&gt;pm=1&lt;/code&gt; you have a positive random variable with a distribution lower bound at 0.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; min(rstable(100000, alpha=0.4, beta=1, gamma=0.4, delta=0, pm=1))&#xA;[1] 0.002666507&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Shift by 2 and the lower bound rises by the same amount&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; min(rstable(100000, alpha=0.4, beta=1, gamma=0.4, delta=2, pm=1))&#xA;[1] 2.003286&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But if you want the &lt;code&gt;delta&lt;/code&gt; input to be the internal location value rather than the shift or lower bound, then you need to use a different specification for the parameters. For example if you try the following (with &lt;code&gt;pm=3&lt;/code&gt; and trying &lt;code&gt;delta=0&lt;/code&gt; and the &lt;code&gt;delta=0.290617&lt;/code&gt; you found earlier), you seem to get the same &lt;code&gt;delta&lt;/code&gt; in and out. With &lt;code&gt;pm=3&lt;/code&gt; and &lt;code&gt;delta=0.290617&lt;/code&gt; you get the same density of 0.02700602 you found earlier and a lower bound at 0. With &lt;code&gt;pm=3&lt;/code&gt; and &lt;code&gt;delta=0&lt;/code&gt; you get a negative lower bound (in fact -0.290617).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; dstable(4, alpha=0.4, beta=1, gamma=0.4, delta=0, pm=3)&#xA;[1] 0.02464434&#xA;attr(,&quot;control&quot;)&#xA;   dist alpha beta gamma delta pm&#xA; stable   0.4    1   0.4     0  3&#xA;&amp;gt; dstable(4, alpha=0.4, beta=1, gamma=0.4, delta=0.290617, pm=3)&#xA;[1] 0.02700602&#xA;attr(,&quot;control&quot;)&#xA;   dist alpha beta gamma    delta pm&#xA; stable   0.4    1   0.4 0.290617  3&#xA;&amp;gt; min(rstable(100000, alpha=0.4, beta=1, gamma=0.4, delta=0, pm=3))&#xA;[1] -0.2876658&#xA;&amp;gt; min(rstable(100000, alpha=0.4, beta=1, gamma=0.4, delta=0.290617, pm=3))&#xA;[1] 0.004303485&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You may find it easier simply to ignore &lt;code&gt;delta&lt;/code&gt; in the output, and so long as you keep &lt;code&gt;beta=1&lt;/code&gt; then using &lt;code&gt;pm=1&lt;/code&gt; means &lt;code&gt;delta&lt;/code&gt; in the input is the distribution lower bound, which it seems you want to be 0.&lt;/p&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-03-20T15:45:20.570" CommentCount="2" />
  <row Id="8536" PostTypeId="2" ParentId="665" CreationDate="2011-03-20T16:02:25.653" Score="2" ViewCount="" Body="&lt;p&gt;It's misleading to simply say that statistics is simply the inverse of probability.  Yes, statistical questions &lt;em&gt;are&lt;/em&gt; questions of inverse probability, but they are &lt;em&gt;ill-posed inverse problems&lt;/em&gt;, and this makes a big difference in terms of how they are addressed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Probability is a branch of pure mathematics--probability questions can be posed and solved using axiomatic reasoning, and therefore there is one correct answer to any probability question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Statistical questions can be &lt;em&gt;converted&lt;/em&gt; to probability questions by the use of &lt;em&gt;probability models&lt;/em&gt;.  Once we make certain assumptions about the mechanism generating the data, we can answer statistical questions using probability theory.  HOWEVER, the proper formulation and checking of these probability models is just as important, or even more important, than the subsequent analysis of the problem using these models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One could say that statistics comprises of two part.  The first part is the question of how to formulate and evaluate probabilistic models for the problem, and this endeavor lies within the domain of &quot;philosophy of science.&quot;  The second part is the question of obtaining answers after a certain model has been assumed.  This part of statistics is indeed a matter of applied probability theory, and in practice, a fair deal of numerical analysis as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See:&#xA;&lt;a href=&quot;http://cscs.umich.edu/~crshalizi/reviews/error/&quot; rel=&quot;nofollow&quot;&gt;http://cscs.umich.edu/~crshalizi/reviews/error/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="3567" LastActivityDate="2011-03-20T16:02:25.653" />
  <row Id="8537" PostTypeId="2" ParentId="2181" CreationDate="2011-03-20T16:20:26.003" Score="1" ViewCount="" Body="&lt;p&gt;Hands down best basic text on multivariate regression is (still) Cohen, J., Cohen, P., West, S.G. &amp;amp; Aiken, L.S. Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences, (L. Erlbaum Associates, Mahwah, N.J., 2003).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cohen made his name in statistics yet was a psychologist; still if you want social psychology-focused treatment of multivariate, one not limited to multivariate regression (although it definitely favors it over ANOVA &amp;amp; MANOVA, which ought to be banned by some sort of Intellectual Human Rights Commission), then your best bet is Judd, C.M., McClelland, G.H. &amp;amp; Ryan, C.S. Data analysis : a model comparison approach, (Routledge/Taylor and Francis, New York, NY, 2008).  Judd also has a very very good chapter on multivariate regression in Judd, C.M. Everyday Data Analysis in Social Psychology: Comparisons of Linear Models. in Handbook of research methods in social and personality psychology (eds. Reis, H.T. &amp;amp; Judd, C.M.) 370-392 (Cambridge University Press, New York, 2000).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I agree that Gelman, A. &amp;amp; Hill, J. Data Analysis Using Regression and Multilevel/Hierarchical Models, (Cambridge University Press, Cambridge ; New York, 2007), is amazing, but it is really more geared to someone already comfortable w/ basics of multivariate regression--it's primarily about multilevel modeling. Also is focused on observational study methodology--not experimental (Judd is best for that; Cohen okay too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want something on interactions in multivariate -- which you likely will if you are using experimental methods -- then best two texts are Aiken, L.S., West, S.G. &amp;amp; Reno, R.R. Multiple Regression: Testing and Interpreting Interactions, (Sage Publications, Newbury Park, Calif., 1991) &amp;amp; Jaccard, J. &amp;amp; Turrisi, R. Interaction Effects in Multiple Regression, (Sage Publications, Thousand Oaks, Calif., 2003). (Both Cohen &amp;amp; Cohen &amp;amp; Judd do treat this topic, though.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On &quot;free&quot; side, you probably know about &lt;a href=&quot;http://faculty.chass.ncsu.edu/garson/PA765/statnote.htm&quot; rel=&quot;nofollow&quot;&gt;http://faculty.chass.ncsu.edu/garson/PA765/statnote.htm&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Last bit of advice:  Never ever split your continuous variables!!! It's amazing how many social psychologists, used to ANOVA, still do this even as they make use of multivariate techniques such as regression analysis!&lt;/p&gt;&#xA;" OwnerUserId="2347" LastEditorUserId="2347" LastEditorDisplayName="" LastEditDate="2011-03-20T16:33:12.413" LastActivityDate="2011-03-20T16:33:12.413" CommunityOwnedDate="2011-03-20T16:20:26.003" />
  <row Id="8538" PostTypeId="2" ParentId="8501" CreationDate="2011-03-20T18:11:16.920" Score="2" ViewCount="" Body="&lt;p&gt;I tend to be more and more convinced that this just generally isn't a good idea because the meaning the R^2 isn't really exactly the same as in a conventional linear regression.  As such, one always runs into interpretation issues and it often distracts from the meat of the story. Time spent writing a good description of the fit of model is time better spent... but I could be wrong.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[A good answer to a question on how to write up the fit of an LMM model would be really handy (which a cursory search didn't show me).]&lt;/p&gt;&#xA;" OwnerUserId="601" LastActivityDate="2011-03-20T18:11:16.920" CommentCount="1" />
  <row Id="8539" PostTypeId="2" ParentId="8514" CreationDate="2011-03-20T18:38:25.820" Score="3" ViewCount="" Body="&lt;p&gt;I'm sorry, but to me it sounds like you are trying to fix what isn't broken.  In fact, you might even be trying to break what isn't broken.  When you have a quantitative variable (here, population) that spans a wide range, then whatever metric you use to represent it should also span a wide range.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But for all things related to color (and esp. maps), the key source is, I think &lt;a href=&quot;http://colorbrewer2.org/&quot; rel=&quot;nofollow&quot;&gt;ColorBrewer&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="686" LastActivityDate="2011-03-20T18:38:25.820" CommentCount="1" />
  <row Id="8540" PostTypeId="2" ParentId="8490" CreationDate="2011-03-20T18:42:55.053" Score="2" ViewCount="" Body="&lt;p&gt;If you only have three or four people, then the right test is IOTT - the inter-ocular trauma test.  That is, it hits you between the eyes.  To allow the data to hit you properly, I would recommend graphics.  In particular, I'd put time on the x-axis, score on the y-axis, and put lines for each person.&lt;/p&gt;&#xA;" OwnerUserId="686" LastActivityDate="2011-03-20T18:42:55.053" CommentCount="1" />
  <row Id="8541" PostTypeId="1" AcceptedAnswerId="8607" CreationDate="2011-03-20T19:09:39.007" Score="1" ViewCount="59" Body="&lt;p&gt;Greetings, &#xA;I've been trying to get my hands on a substantial resource for using Gibbs sampling in hybrid Bayesian networks, that is, networks with both continuous and discrete variables. &#xA;So far I can't say I have succeeded. I'm interested in hybrid networks where there are no constraints regarding discrete children having continuous parents.&#xA;Gibbs sampling is a very widely employed method in approximate inference in Bayesian methods, and yet, I can't seem to find detailed resources that focus on hybrid networks &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help would be much appreciated!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Best Regards&#xA;Seref&lt;/p&gt;&#xA;" OwnerUserId="3280" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-20T20:47:15.967" LastActivityDate="2011-03-22T04:50:36.987" Title="Resources about Gibbs sampling in hybrid Bayesian networks" Tags="&lt;machine-learning&gt;&lt;bayesian&gt;&lt;books&gt;&lt;gibbs&gt;" AnswerCount="2" />
  <row Id="8542" PostTypeId="2" ParentId="866" CreationDate="2011-03-20T20:13:05.083" Score="0" ViewCount="" Body="&lt;p&gt;A similar question has just been asked on metaoptimize (keeping in mind that l1 = LASSO and l2 = ridge):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://metaoptimize.com/qa/questions/5205/when-to-use-l1-regularization-and-when-l2&quot; rel=&quot;nofollow&quot;&gt;http://metaoptimize.com/qa/questions/5205/when-to-use-l1-regularization-and-when-l2&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1265" LastActivityDate="2011-03-20T20:13:05.083" CommunityOwnedDate="2011-03-20T20:13:05.083" />
  <row Id="8543" PostTypeId="2" ParentId="8358" CreationDate="2011-03-20T20:31:57.940" Score="3" ViewCount="" Body="&lt;p&gt;I don't know why I took the time to answer this. Is it because I can or maybe it's because DrWho seems to think it is very important. In either case ....&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Though well intentioned&lt;/p&gt;&#xA;&#xA;&lt;p&gt;“Time series expert modeler of IBM SPSS Forecast v19 was used. Both exponential smoothening models and ARIMA models were examined.Outliers were detected and prevented from influencing parameter estimates”&lt;/p&gt;&#xA;&#xA;&lt;p&gt;may have suffered from an inability to detect level shifts i.e. “Intercept Changes” which are a sequence of pulses with the same value and sign. Note below a reasonable model for STATE1 using all 17 values suggests a Level Shift at 2004 ( period 11). This model [AR(1)]  was used to  cleanse STATE1 of unspecified background factors that may have been present to cause significant changes in Y given X.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Y(T) = -87.899&lt;br&gt;&#xA;       +[X1(T)][(+  .158)]                  M_CASES&#xA;       +[X2(T)][(- 77.4369)]                :PULSE         7 I~P00007STATE1&#xA;       +[X3(T)][(- 65.2775)]                :PULSE        15 I~P00015STATE1&#xA;       +[X4(T)][(-  112.77    )]            :LEVEL SHIFT  11 I~L00011STATE1&#xA;       +[X5(T)][(+ 43.0999)]                :PULSE         9 I~P00009STATE1&#xA;       +[X6(T)][(- 58.4117)]                :PULSE         4 I~P00004STATE1&#xA;      +     [(1-  .840B** 1)]**-1  [A(T)]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Leading to a cleansed set of values FOR STATE1&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1994               383.0000000000&lt;br&gt;&#xA;1995               257.0000000000&lt;br&gt;&#xA;1996               263.0000000000&lt;br&gt;&#xA;1997               465.4116693551&lt;br&gt;&#xA;1998               149.0000000000&lt;br&gt;&#xA;1999               434.0000000000&lt;br&gt;&#xA;2000               246.4369202361&lt;br&gt;&#xA;2001               275.0000000000&lt;br&gt;&#xA;2002               134.9000542626&lt;br&gt;&#xA;2003               142.0000000000&lt;br&gt;&#xA;2004               131.0000000000&lt;br&gt;&#xA;2005               336.0000000000&lt;br&gt;&#xA;2006               108.0000000000&lt;br&gt;&#xA;2007                75.0000000000&lt;br&gt;&#xA;2008               118.2775018144&lt;br&gt;&#xA;2009                40.0000000000&lt;br&gt;&#xA;2010                                26.0000000000        &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Notice that a simple line graph between Y and X visually support the change in the relationship between Y and on or about period 11 (2004) such that the Y values are clearly lower than expected for the period 2004-2010 ( 11-20) as compared to period 1994-2003 (1-10). This is a classic case of an outside factor impacting either Y or X ( but not both !) starting at time period 11. Normal statistical commentary would refer to this level shift as a “lurking variable” confounding simple analysis if untreated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For STATE2&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Y(T) = -6.4072&lt;br&gt;&#xA;       +[X1(T)][(+  .213)]         M_CASES&#xA;       +[X2(T)][(-  254.25    )]   :PULSE 17                  I~P00017STATE2&#xA;       +[X3(T)][(+  214.32    )]   :PULSE 12                  I~P00012STATE2&#xA;       +[X4(T)][(- 92.6947)]       :PULSE 16                  I~P00016STATE2&#xA;       +[X5(T)][(- 98.6907)]       :PULSE 15                  I~P00015STATE2&#xA;       +[X6(T)][(+ 39.8298)]       :PULSE 13                  I~P00013STATE2&#xA;      +       [A(T)]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1994               121.0000000000&lt;br&gt;&#xA;1995               227.0000000000&lt;br&gt;&#xA;1996               179.0000000000&lt;br&gt;&#xA;1997                76.0000000000&lt;br&gt;&#xA;1998               195.0000000000&lt;br&gt;&#xA;1999               275.0000000000&lt;br&gt;&#xA;2000               253.0000000000&lt;br&gt;&#xA;2001               199.0000000000&lt;br&gt;&#xA;2002               133.0000000000&lt;br&gt;&#xA;2003               237.0000000000&lt;br&gt;&#xA;2004               228.0000000000&lt;br&gt;&#xA;2005              1285.6763010529&lt;br&gt;&#xA;2006               488.1702389682&lt;br&gt;&#xA;2007               645.0000000000&lt;br&gt;&#xA;2008               635.6907401611&lt;br&gt;&#xA;2009               648.6947149772&lt;br&gt;&#xA;2010               748.2497352909   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that the Pulses were identified GIVEN the number of cases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now proceeding with the CHOW TEST to test the significant difference between the two sets of regression coefficients:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We get the following OLS Model for the combined 34 (cleansed values used )&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With Error Sum of Squares   =     363716.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For STATE1 we get&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Y(T) = -49.293&lt;br&gt;&#xA;       +[X1(T)][(+  .150)]   M_CASES&lt;br&gt;&#xA;       +       [A(T)]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sum of Squares    142759.    &lt;/p&gt;&#xA;&#xA;&lt;p&gt;And for STATE2&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Y(T) = -6.4072&lt;br&gt;&#xA;       +[X1(T)][(+  .213)]   M_CASES&#xA;       +       [A(T)]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sum of Squares       1445.64     &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As before we have the following F test &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Numerator  =   [363716 – (142759+1445)] /2     =  109,776&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Denominator = 363716/30                        =   12,123&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Computed F value of 9.0 with 2 and 30 degrees of freedom is significant at alpha less than .001 . Thus one could conclude that there is a statistically significant difference between the two states at about a 99.9% Level of Confidence given that there was a significant effect in STATE1 at or about 2004.&lt;/p&gt;&#xA;" OwnerUserId="3382" LastActivityDate="2011-03-20T20:31:57.940" CommentCount="7" />
  <row Id="8544" PostTypeId="2" ParentId="8515" CreationDate="2011-03-20T20:50:31.327" Score="2" ViewCount="" Body="&lt;p&gt;Also of note: Martin Maechler just refactored the code for the stable distributed and added some improvements. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;His new package &lt;a href=&quot;http://cran.r-project.org/package=stabledist&quot; rel=&quot;nofollow&quot;&gt;stabledist&lt;/a&gt; will be used by fBasics as well, so you may want to give this a look as well.&lt;/p&gt;&#xA;" OwnerUserId="334" LastActivityDate="2011-03-20T20:50:31.327" CommentCount="1" />
  <row Id="8545" PostTypeId="1" CreationDate="2011-03-20T20:54:24.877" Score="3" ViewCount="176" Body="&lt;p&gt;I have some problems in using (and finding) the Chow test for structural breaks in a regression analysis using R. I want to find out if there are some structural changes including another variable (represents 3 spatial subregions).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Namely, is the regression with the subregions better than the overall model. Therefore I need some statistical validation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope my problem is clear, isn't it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Kind regards&lt;br&gt;&#xA;marco&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Toy example in R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(mlbench) # dataset&#xA;data(&quot;BostonHousing&quot;)&#xA;&#xA;# data preparation&#xA;BostonHousing$region &amp;lt;- ifelse(BostonHousing$medv &amp;lt;= &#xA;                               quantile(BostonHousing$medv)[2], 1, &#xA;                        ifelse(BostonHousing$medv &amp;lt;= &#xA;                               quantile(BostonHousing$medv)[3], 2,&#xA;                        ifelse(BostonHousing$medv &amp;gt; &#xA;                               quantile(BostonHousing$medv)[4], 3, 1)))&#xA;&#xA;BostonHousing$region &amp;lt;- as.factor(BostonHousing$region)&#xA;&#xA;# regression without any subregion &#xA;reg1&amp;lt;- lm(medv ~ crim + indus + rm, data=BostonHousing)&#xA;&#xA;summary(reg1)&#xA;&#xA;# are there structural breaks using the factor &quot;region&quot; which&#xA;# indicates 3 spatial subregions&#xA;reg2&amp;lt;- lm(medv ~ crim + indus + rm + region, data=BostonHousing)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;------- subsequent entry&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I struggled with your suggested package &quot;strucchange&quot;, not knowing how to use the &quot;from&quot; and &quot;to&quot; arguments correctly with my factor &quot;region&quot;. Nevertheless, I found one hint to calculate it by hand (https://stat.ethz.ch/pipermail/r-help/2007-June/133540.html). This results in the following output, but now I am not sure if my interpetation is valid. The results from the example above below.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does this mean that region 3 is significant different from region 1? Contrary, region 2 is not? Further, each parameter (eg region1:crim) represents the beta for each regime and the model for this region respectively? Finally, the ANOVA states that there is a signif. difference between these models and that the consideration of regimes leads to a better model?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you for your advices!&#xA;Best Marco&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;fm0 &amp;lt;- lm(medv ~ crim + indus + rm, data=BostonHousing)&#xA;summary(fm0)&#xA;fm1 &amp;lt;- lm(medv  ~ region / (crim + indus + rm), data=BostonHousing)&#xA;summary(fm1)&#xA;anova(fm0, fm1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Results:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Call:&#xA;lm(formula = medv ~ region/(crim + indus + rm), data = BostonHousing)&#xA;&#xA;Residuals:&#xA;       Min         1Q     Median         3Q        Max &#xA;-21.079383  -1.899551   0.005642   1.745593  23.588334 &#xA;&#xA;Coefficients:&#xA;               Estimate Std. Error t value Pr(&amp;gt;|t|)    &#xA;(Intercept)    12.40774    3.07656   4.033 6.38e-05 ***&#xA;region2         6.01111    7.25917   0.828 0.408030    &#xA;region3       -34.65903    4.95836  -6.990 8.95e-12 ***&#xA;region1:crim   -0.19758    0.02415  -8.182 2.39e-15 ***&#xA;region2:crim   -0.03883    0.11787  -0.329 0.741954    &#xA;region3:crim    0.78882    0.22454   3.513 0.000484 ***&#xA;region1:indus  -0.34420    0.04314  -7.978 1.04e-14 ***&#xA;region2:indus  -0.02127    0.06172  -0.345 0.730550    &#xA;region3:indus   0.33876    0.09244   3.665 0.000275 ***&#xA;region1:rm      1.85877    0.47409   3.921 0.000101 ***&#xA;region2:rm      0.20768    1.10873   0.187 0.851491    &#xA;region3:rm      7.78018    0.53402  14.569  &amp;lt; 2e-16 ***&#xA;---&#xA;Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 &#xA;&#xA;Residual standard error: 4.008 on 494 degrees of freedom&#xA;Multiple R-squared: 0.8142,     Adjusted R-squared: 0.8101 &#xA;F-statistic: 196.8 on 11 and 494 DF,  p-value: &amp;lt; 2.2e-16&#xA;&#xA;&amp;gt; anova(fm0, fm1)&#xA;Analysis of Variance Table&#xA;&#xA;Model 1: medv ~ crim + indus + rm&#xA;Model 2: medv ~ region/(crim + indus + rm)&#xA;  Res.Df     RSS Df Sum of Sq     F    Pr(&amp;gt;F)    &#xA;1    502 18559.4                                 &#xA;2    494  7936.6  8     10623 82.65 &amp;lt; 2.2e-16 ***&#xA;---&#xA;Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="3814" LastEditorUserId="1390" LastEditorDisplayName="" LastEditDate="2011-03-21T12:39:22.767" LastActivityDate="2011-03-21T12:39:22.767" Title="Identifying structural breaks in regression with Chow test" Tags="&lt;r&gt;&lt;chow-test&gt;&lt;structural-change&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="8546" PostTypeId="2" ParentId="8456" CreationDate="2011-03-20T21:41:48.500" Score="1" ViewCount="" Body="&lt;p&gt;Simple answer:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Select one set of X and Y values, and create your XY chart.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Copy the second set of X and Y values, select the chart, and use paste special to add the data as a new series.&lt;/p&gt;&#xA;" OwnerUserId="3815" LastActivityDate="2011-03-20T21:41:48.500" />
  <row Id="8547" PostTypeId="2" ParentId="8545" CreationDate="2011-03-20T21:51:03.190" Score="5" ViewCount="" Body="&lt;p&gt;The &lt;strong&gt;&lt;a href=&quot;http://cran.r-project.org/web/packages/strucchange/index.html&quot; rel=&quot;nofollow&quot;&gt;strucchange&lt;/a&gt;&lt;/strong&gt; package contains Chow and F tests for structural changes in regression models. The package comes with a vignette which shows how to use the package.&lt;/p&gt;&#xA;" OwnerUserId="1390" LastActivityDate="2011-03-20T21:51:03.190" CommentCount="3" />
  <row Id="8548" PostTypeId="2" ParentId="8541" CreationDate="2011-03-20T22:23:16.703" Score="0" ViewCount="" Body="&lt;p&gt;I think this is still an open research question and there has been little consensus on the best way to do this. &lt;/p&gt;&#xA;" OwnerUserId="3816" LastActivityDate="2011-03-20T22:23:16.703" />
  <row Id="8549" PostTypeId="2" ParentId="6033" CreationDate="2011-03-21T00:08:22.847" Score="3" ViewCount="" Body="&lt;p&gt;There are a number of ways that &quot;a structural break&quot; may occur.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If there is a change in the Intercept or a change in Trend in &quot;the latter portion of the time series&quot; then one would be better suited to perform Intervention Detection (N.B. this is the empirical identification of the significant impact of an unspecified Deterministic Variable such as a Level Shift or a Change in Trend or the onset of a Seasonal Pulse ). Intervention Detection then is a pre-cursor to Intervention Modelling where a suggested variable is included in the model. You can find information on the web by googling &quot;AUTOMATIC INTERVENTION DETECTION&quot; . Some authors use the term &quot;OUTLIER DETECTION&quot; but like a lot of statistical language this can be confusing/imprecise . Detected Interventions can be any of the following (detecting a significant change in the mean of the residuals );&lt;/p&gt;&#xA;&#xA;&lt;p&gt;a 1 period change in Level ( i.e. a Pulse )&#xA;a multi-period contiguous change in Level ( i.e. a change in Intercept )&#xA;a systematic Pulse ( i.e. a Seasonal Pulse )&#xA;a trend change (i.e. 1,2,3,4,5,7,9,11,13,15 ..... )&#xA;These procedures are easily programmed IN R/SAS/Matlab and routinely available in a number of commercially available time series packages however there are many pitfalls that you need to be wary of such as whether to detect the stochastic structure first or do Intervention detection on the original series. This is like the chicken and egg problem. Early work in this area was limited to type 1's and as such will probably be insufficient for your needs as your examples illustrate LEVEL SHIFTS.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a lot of material on the web and even a free program at &lt;a href=&quot;http://www.autobox.com/30day.exe&quot; rel=&quot;nofollow&quot;&gt;http://www.autobox.com/30day.exe&lt;/a&gt; that even allows you to use your own data for 30days. You might learn lot &quot;by simply watching&quot; as Yogi once said and replicate their results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The web references for the exact equations for you to use can be found starting at page 134 in &#xA;&lt;a href=&quot;http://www.autobox.com/pdfs/autoboxusersguide.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.autobox.com/pdfs/autoboxusersguide.pdf&lt;/a&gt; . I am one of the authors of AUTOBOX.&lt;/p&gt;&#xA;" OwnerUserId="3382" LastEditorUserId="3382" LastEditorDisplayName="" LastEditDate="2011-04-01T09:08:51.430" LastActivityDate="2011-04-01T09:08:51.430" CommentCount="1" />
  <row Id="8550" PostTypeId="2" ParentId="8528" CreationDate="2011-03-21T02:15:26.610" Score="1" ViewCount="" Body="&lt;p&gt;Empirical distributions are used all the time for inference so you're definitely on the right track! One of the most common use of empirical distributions is for bootstrapping. In fact, you don't even have to use any of the machinery you've described above. In an nutshell, you make many draws (with replacement) from the original samples in a uniform fashion and the results can be used to calculate the confidence intervals on your previously calculated statistical quantities. Furthermore, these samples have well developed theoretical convergence properties. Check out the wikipedia article on the topic &lt;a href=&quot;http://en.wikipedia.org/wiki/Bootstrapping_%28statistics%29&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="3786" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-03-21T08:29:02.110" LastActivityDate="2011-03-21T08:29:02.110" CommentCount="4" />
  <row Id="8551" PostTypeId="1" CreationDate="2011-03-21T03:31:20.583" Score="2" ViewCount="93" Body="&lt;p&gt;What do you call a curve that is just the first half of a bell curve.  For example, let's say in a typical bell curve of letter grades, a few students get F grades most get C grades and just a few get A grades.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd like a curve that is the first half of the bell curve so that a few students get F grades and the most common grade is an A grade.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(I'm not actually doing this for grading. It's just the example that came to mind.)&lt;/p&gt;&#xA;" OwnerUserId="3820" LastActivityDate="2011-03-22T05:20:15.463" Title="What do you call just the first half of a bell curve?" Tags="&lt;distributions&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="8552" PostTypeId="2" ParentId="8551" CreationDate="2011-03-21T03:43:56.787" Score="6" ViewCount="" Body="&lt;p&gt;A &quot;bell curve&quot; in the non-technical sense could refer to one of a family of statistical distributions which are bell-shaped.  In the context of grading I've only ever seen the normal distribution (and it is by far the most common in general), but others include the logistic, t, etc.  The &lt;a href=&quot;https://secure.wikimedia.org/wikipedia/en/wiki/Half-normal_distribution&quot; rel=&quot;nofollow&quot;&gt;half-normal distribution&lt;/a&gt; is generated by taking the absolute value of a (zero-mean) normal distribution.  That is to say, it represents the case where most students get an F and few get A's.  By making a suitable transformation of this distribution (taking the negative to get the mirror image then shifting so that the maximum is 100 points), you can get the distribution you're after.&lt;/p&gt;&#xA;" OwnerUserId="2975" LastEditorUserId="2975" LastEditorDisplayName="" LastEditDate="2011-03-22T05:20:15.463" LastActivityDate="2011-03-22T05:20:15.463" />
  <row Id="8553" PostTypeId="2" ParentId="1556" CreationDate="2011-03-21T04:16:34.817" Score="3" ViewCount="" Body="&lt;p&gt;Why stop at t-tests?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can think of two variables being independent (also called uncorrelated sometimes) as two orthogonal vectors, exactly like the $x$ and $y$ axes in a two dimensional Cartesian coordinate system.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When either of two vectors, let's say $\mathbf{x}$ and $\mathbf{y}$ is correlated with the other, there will be a certain part of x that can be projected onto y and vice versa. With that in mind, it's fairly easy to see that since,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;\begin{align*}&#xD;&#xA;\left&amp;lt;\mathbf{x},\mathbf{y}\right&amp;gt;&amp;amp;=\|x\|\|y\|\cos\left(\theta\right)\\&#xD;&#xA;\frac{\left&amp;lt;\mathbf{x},\mathbf{y}\right&amp;gt;}{\|x\|\|y\|}&amp;amp;=\cos\left(\theta\right)=r&#xD;&#xA;\end{align*}&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $r$ is Pearson's correlation coefficient and $\left&amp;lt;\cdot,\cdot\right&amp;gt;$ is the inner product of the arguments. When I learned this I was totally blown away by how geometrically simple the idea of correlation is. And this is definitely not the only way to measure the correlation between two (or more) variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Significance testing is a different ball game. Often we want to know &lt;strong&gt;by how much&lt;/strong&gt; two (or more) groups differ on some outcome variable as a result of some manipulation that was performed (or not performed) on said groups. Like Brian said, you want to know if the two groups come from the same distribution, thus you compute the probability of sampling the mean difference (scaled by the standard error of the mean) that you obtained from your experiment, given that the null hypothesis (there's no significant difference in the means) is true. In behavioral research (and often elsewhere) if this probability is less 0.05, you can conclude that the difference in the two (or more) means is likely due to your manipulation.&lt;/p&gt;&#xA;" OwnerUserId="2660" LastActivityDate="2011-03-21T04:16:34.817" />
  <row Id="8555" PostTypeId="5" CreationDate="2011-03-21T04:30:19.377" Score="0" ViewCount="" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditorDisplayName="" LastEditDate="2011-03-21T04:30:19.377" LastActivityDate="2011-03-21T04:30:19.377" />
  <row Id="8556" PostTypeId="4" CreationDate="2011-03-21T04:30:19.377" Score="0" ViewCount="" Body="Problems dealing with the relationship between two or more variables." OwnerUserId="-1" LastEditorUserId="2660" LastEditorDisplayName="" LastEditDate="2011-03-21T07:45:16.517" LastActivityDate="2011-03-21T07:45:16.517" />
  <row Id="8557" PostTypeId="1" CreationDate="2011-03-21T04:57:51.157" Score="5" ViewCount="199" Body="&lt;p&gt;The whole point of AIC or any other information criterion is that less is better. So if I have two models M1: y = a0 + XA + e and M2: y = b0 + ZB + u, and if the AIC of the first (A1) is less than that of the second (A2), then M1 has a better fit from the information theory standpoint. But is there any cutoff benchmark for the difference A1-A2? How much less is actually less? In other words, is there a test for (A1-A2) other than just eyeballing?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: Peter/Dmitrij... Thanks for responding. Actually, this is a case where my substantive expertise is conflicting with my statistical expertise. Essentially, the problem is NOT choosing between two models, but in checking if two variables which I know to be largely equivalent  add equivalent amounts of information (Actually, one variable in the first model and a vector in the second. Think about the case of a bunch of variables as against an index of them.). As Dmitrij pointed out, the best bet seems to be the Cox Test. But is there a way of actually testing the difference between the information contents of the two models?&lt;/p&gt;&#xA;" OwnerUserId="3671" LastEditorUserId="3671" LastEditorDisplayName="" LastEditDate="2011-03-21T11:28:01.733" LastActivityDate="2011-03-21T11:28:01.733" Title="Testing the difference in AIC of two non-nested models" Tags="&lt;regression&gt;&lt;aic&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="8558" PostTypeId="2" ParentId="8557" CreationDate="2011-03-21T05:56:28.850" Score="5" ViewCount="" Body="&lt;p&gt;Is is the question of curiosity, i.e. you are not satisfied by my answer &lt;a href=&quot;http://stats.stackexchange.com/questions/8513/test-equivalence-of-non-nested-models/8519#8519&quot;&gt; here &lt;/a&gt;? If not...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The further investigation of this tricky question showed that there do exist a commonly used rule-of-thumb, that states two models are indistinguishable by $AIC$ criterion if the difference  $|AIC_1 - AIC_2| &amp;lt; 2$. The same you actually will read in wikipedia's article on &lt;a href=&quot;http://en.wikipedia.org/wiki/Akaike_information_criterion#How_to_apply_AIC_in_practice&quot; rel=&quot;nofollow&quot;&gt;$AIC$&lt;/a&gt; (note the link is clickable!). Just for those who do not click the links:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;$AIC$ estimates relative support for a model. To apply this in practice, we start with a set of candidate models, and then find the models' corresponding $AIC$ values. Next, identify the minimum $AIC$ value. The selection of a model can then be made as follows.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;As a rough rule of thumb, models having their $AIC$ within $1–2$ of the minimum have substantial support and should receive consideration in making inferences. Models having their $AIC$ within about $4–7$ of the minimum have considerably less support, while models with their $AIC &amp;gt; 10$ above the minimum have either essentially no support and might be omitted from further consideration or at least fail to explain some substantial structural variation in the data.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;A more general approach is as follows...&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Denote the $AIC$ values of the candidate models by $AIC1$, $AIC2$, $AIC3$, …, $AICR$. Let $AICmin$ denotes the minimum of those values. Then $e^{(AICmin−AICi)/2}$ can be interpreted as the relative probability that the $i$-th model minimizes the (expected estimated) information loss.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;As an example, suppose that there were three models in the candidate set, with $AIC$ values $100$, $102$, and $110$. Then the second model is $e^{(100−102)/2} = 0.368$ times as probable as the first model to minimize the information loss, and the third model is $e^{(100−110)/2} = 0.007$ times as probable as the first model to minimize the information loss. In this case, we might omit the third model from further consideration and take a weighted average of the first two models, with weights $1$ and $0.368$, respectively. Statistical inference would then be based on the weighted multimodel.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Nice explanation and useful suggestions, in my opinion. Just don't be afraid of reading what is clickable! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In &lt;strong&gt;addition&lt;/strong&gt;, note once more, $AIC$ is less preferable for large-scale data sets. In addition to $BIC$ you may find useful to apply bias-corrected version of $AIC$ criterion $AICc$ (you may use this $R$ &lt;a href=&quot;http://www.awblocker.com/AICc.R&quot; rel=&quot;nofollow&quot;&gt;code&lt;/a&gt; or use the formula $AICc = AIC + \frac{2p(p+1)}{n-p-1}$, where $p$ is the number of estimated parameters). Rule-of-thumb will be the same though.  &lt;/p&gt;&#xA;" OwnerUserId="2645" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-03-21T08:00:32.837" LastActivityDate="2011-03-21T08:00:32.837" CommentCount="2" />
  <row Id="8559" PostTypeId="1" CreationDate="2011-03-21T07:48:30.303" Score="3" ViewCount="60" Body="&lt;p&gt;Related to &lt;a href=&quot;http://stats.stackexchange.com/questions/8236/how-to-find-relationships-between-different-types-of-events-defined-by-their-2d&quot;&gt;my previous question&lt;/a&gt;, I have a dataset of 2D points with an associated label (this label can take 6 different values). As suggested in the answers to my other question, this can be modeled as a marked point process (or 6 different point processes), allowing to apply standard tools to study this dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to take the approach that I first suggested in my first question, and try to apply PCA on this dataset, to see if the different types of points are correlated or not (i.e. are some types always happening together?). Here's how I want to do it:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Split my 2D space in a grid&lt;/li&gt;&#xA;&lt;li&gt;For each cell of that grid, count the number of points of each type. For one cell, this gives me a point in $\mathbb{R}^6 : x_i = (N_1(A_i), N_2(A_i), N_3(A_i), N_4(A_i), N_5(A_i), N_6(A_i))$, where $N_k(A_i)$ is the number of points of the $k^{th}$ point process (corresponding to points of type $k$) in the cell $A_i$&lt;/li&gt;&#xA;&lt;li&gt;Combine all the $x_i$ into a matrix $X \in \mathbb{R}^{6 \times M}$ and apply PCA to this matrix.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;My question is the following: how do I build the grid? In other words, how do I rediscretize this dataset?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Indeed, the intensities of each process are not equal: some types appear more than others. If I just use a regular grid (all cells have the same area), the resulting points will have one or two components that dominates the others.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was thinking of building my grid such that each cell has at most $N$ points, thus bounding the norm of the data points, but I don't think this will solve my &quot;balance&quot; problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestion, or pointer to litterature, are appreciated.&lt;/p&gt;&#xA;" OwnerUserId="3699" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-21T08:57:00.910" LastActivityDate="2011-03-21T08:57:00.910" Title="How can I rediscretize my data?" Tags="&lt;pca&gt;&lt;multivariate-analysis&gt;&lt;normalization&gt;" CommentCount="2" />
  <row Id="8560" PostTypeId="2" ParentId="8557" CreationDate="2011-03-21T10:13:20.787" Score="3" ViewCount="" Body="&lt;p&gt;I think this may be an attempt to get what you don't really want.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Model selection is not a science.  Except in rare circumstances, there is no one perfect model, or even one &quot;true&quot; model; there is rarely even one &quot;best&quot; model.  Discussions of AIC vs. AICc vs BIC vs. SBC vs. whatever leave me somewhat nonplussed.  I think the idea is to get some GOOD models.  You then choose among them based on a combination of substantive expertise and statistical ideas.  If you have no substantive expertise (rarely the case; much more rarely than most people suppose) then choose the lowest AIC (or AICc or whatever).  But you usually DO have some expertise - else why are you investigating these particular variables?&lt;/p&gt;&#xA;" OwnerUserId="686" LastActivityDate="2011-03-21T10:13:20.787" CommentCount="1" />
  <row Id="8561" PostTypeId="1" CreationDate="2011-03-21T11:28:58.193" Score="3" ViewCount="104" Body="&lt;p&gt;I wonder if it possible to include a mediation effect in multinomial logistic regression. I have a categorical (3 categories) outcome variable and four predictors (all continuous). I expect one of the predictors (X1) to mediate the relationship between the outcome variable and another predictor (X2). I also expect direct effects of X1 and X2 on the outcome.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Apparently, I could not see an option of running the regression in two steps (or blocks). Is there a way of including mediation in this analysis?  I would be glad if you could inform me how it is possible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;thanks,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: I am using PASW 18 for Mac.    &lt;/p&gt;&#xA;" OwnerUserId="3821" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-03-21T11:37:40.173" LastActivityDate="2011-03-21T11:37:40.173" Title="How to assess mediation effect in multinomial logistic regression?" Tags="&lt;logistic-regression&gt;&lt;spss&gt;&lt;multinomial&gt;&lt;mediation&gt;" CommentCount="1" FavoriteCount="1" />
  <row Id="8562" PostTypeId="1" CreationDate="2011-03-21T11:49:35.680" Score="3" ViewCount="86" Body="&lt;p&gt;I have a completely within-subjects design with 3 independent variables:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Trial type (3 levels)&lt;/li&gt;&#xA;&lt;li&gt;Task order (2 levels)&lt;/li&gt;&#xA;&lt;li&gt;Modality (3 levels)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;However, for one of my levels of Trial type, the Task order and Modality levels are redundant (because it is essentially a baseline measurement).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ideally, I'd like to run a 3 (trial type: 1, 2, or 3) x 2 (task order: a or b) x 3 (modality: vm, av, hm) repeated measures ANOVA.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you know of a way around this? If I were to duplicate the Task 3 data across 'modalities' and 'task orders' I would have the correct number of levels for each within-subjects factor (i.e. six identical columns of data to represent imaginary modalities and task orders for Task 3). I'm assuming this violates an assumption but I'd be worried about looking at a similar analysis with a huge number of paired sample t-tests. Is lots of t-tests the way to go?&#xA;Thanks for your help!&lt;/p&gt;&#xA;" OwnerUserId="3822" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-03-21T12:28:42.143" LastActivityDate="2011-04-13T14:22:25.767" Title="How to deal with a specific case of unbalanced within-subjects design?" Tags="&lt;anova&gt;&lt;repeated-measures&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="1" />
  <row Id="8563" PostTypeId="1" CreationDate="2011-03-21T11:57:44.363" Score="0" ViewCount="67" Body="&lt;p&gt;Is precision a &quot;quality&quot; of a measurement?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a better (accepted by the literature) word?&lt;/p&gt;&#xA;" OwnerUserId="3823" LastActivityDate="2011-03-21T19:17:16.670" ClosedDate="2011-03-21T21:27:29.203" Title="What is precision for a measurement?" Tags="&lt;terminology&gt;" AnswerCount="2" CommentCount="11" />
  <row Id="8564" PostTypeId="2" ParentId="8563" CreationDate="2011-03-21T12:07:30.953" Score="0" ViewCount="" Body="&lt;p&gt;I believe it's only one aspect of a quality of measurement, an other is known as accuracy. Precision and accuracy are not the same thing. I always think about it as a bell curve, the position of the mean is accuracy, the spread from the mean is precision&lt;/p&gt;&#xA;" OwnerUserId="3484" LastActivityDate="2011-03-21T12:07:30.953" />
  <row Id="8566" PostTypeId="1" AcceptedAnswerId="8616" CreationDate="2011-03-21T13:07:22.040" Score="4" ViewCount="94" Body="&lt;p&gt;Hi everybody,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Actually this question may be simple for you, but I need to learn the correct answer.&lt;br&gt;&#xA;If I remove misclassified instances from data set with Naive Bayes (it gives minimum FP rate) and then train logistic classifier on this filtered data set, will it overfit or not?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="2170" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-21T13:53:24.840" LastActivityDate="2011-03-22T07:39:48.263" Title="Overfit by removing misclassified objects?" Tags="&lt;machine-learning&gt;&lt;naive-bayes&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="1" />
  <row Id="8567" PostTypeId="1" CreationDate="2011-03-21T13:29:38.997" Score="2" ViewCount="132" Body="&lt;p&gt;I am trying to use DLM to model a time series. Candiate model includes local level, local trend and local trend with seasonal part. I do not know how to do model selection. Can AIC be calculated? I found no function in the R package &lt;a href=&quot;http://cran.r-project.org/web/packages/dlm/index.html&quot; rel=&quot;nofollow&quot;&gt;&lt;code&gt;dlm&lt;/code&gt;&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="3824" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-21T13:49:35.403" LastActivityDate="2011-04-03T04:34:24.763" Title="How to do model selection in dynamic linear model?" Tags="&lt;r&gt;&lt;time-series&gt;&lt;model-selection&gt;&lt;dlm&gt;" AnswerCount="3" CommentCount="1" FavoriteCount="1" />
  <row Id="8568" PostTypeId="1" CreationDate="2011-03-21T15:08:27.397" Score="4" ViewCount="53" Body="&lt;p&gt;Hi, my stats question is as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The research group I work for have developed a theoretical growth model for a particular species of fish. The idea is that if you provide some initial starting values for the model you then generate an expected growth curve along with 95% confidence bands. To extend the model we would like to be able to update the model and recalculate the curve when real data becomes available. For example, imagine that at age 6 the model predicts an average weight of 34g, but in a random sample of fish we find that the mean age is, say, 24g, we would like to be able to use this new data to 'tweak' our previously estimated curve. At the moment I am not sure quite how to address this problem so any suggestions would be much appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="3136" LastActivityDate="2011-04-26T14:41:39.463" Title="Updating/ adjusting theoretical growth curves when raw data becomes available" Tags="&lt;time-series&gt;&lt;regression&gt;&lt;forecasting&gt;&lt;updating&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="1" />
  <row Id="8569" PostTypeId="1" CreationDate="2011-03-21T15:10:02.380" Score="2" ViewCount="46" Body="&lt;p&gt;I am trying to test importance sampling for a simple a Wiener process $W_t$ in R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(123)&#xA;Z &amp;lt;- matrix(rnorm(12*1000),12,1000)&#xA;W &amp;lt;- apply(Z,2,cumsum) #Wiener process simulated 1000 times for 12 periods&#xA;B &amp;lt;- W-(1:12) #Brownian motion with drift -1&#xA;w &amp;lt;- exp(-W[12,]) #Radon-Nikodym derivative (actually w &amp;lt;- exp(-W[12,]-12/2))&#xA;mean(B[12,]) #-11.973&#xA;sum(W[12,]*(w/sum(w))) #-7.881 to compare with -12!&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I think that the estimate with the derivative is not really &quot;precise&quot;! Any suggestions to improve the accuracy, besides increasing the number of simulations?&lt;/p&gt;&#xA;" OwnerUserId="1443" LastEditorUserId="1443" LastEditorDisplayName="" LastEditDate="2011-03-21T15:46:03.440" LastActivityDate="2011-03-21T15:46:03.440" Title="Change of measures with Wiener process" Tags="&lt;r&gt;&lt;monte-carlo&gt;" FavoriteCount="1" />
  <row Id="8570" PostTypeId="1" AcceptedAnswerId="8574" CreationDate="2011-03-21T15:24:18.457" Score="7" ViewCount="126" Body="&lt;p&gt;I'm looking for a solid reference (or references) on numerical optimization techniques aimed at statisticians, that is, it would apply these methods to some standard inferential problems (eg MAP/MLE in common models). Things like gradient descent (straight and stochastic), EM and its spinoffs/generalizations, simulated annealing, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm hoping it would have some practical notes on implementation (so often lacking in papers). It doesn't have to be completely explicit but should at least provide a solid bibliography. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some cursory searching turned up a couple of texts: Numerical Analysis for Statisticians by Ken Lange and Numerical Methods of Statistics by John Monahan. Reviews of each seem mixed (and sparse). Of the two a perusal of the table of contents suggests the 2nd edition of Lange's book is closest to what I'm after. &lt;/p&gt;&#xA;" OwnerUserId="26" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-30T12:04:29.620" LastActivityDate="2011-04-30T12:04:29.620" Title="References on numerical optimization for statisticians" Tags="&lt;estimation&gt;&lt;books&gt;&lt;optimization&gt;" AnswerCount="3" FavoriteCount="3" />
  <row Id="8571" PostTypeId="2" ParentId="7946" CreationDate="2011-03-21T15:39:29.850" Score="1" ViewCount="" Body="&lt;p&gt;&lt;em&gt;What kind of graph should I create to illustrate my data?&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would use a scatterplot - it would give you an idea about the type of relationship between the data. It is important to identify if the relationship is linear or not, before calculating correlation between the measurements.&lt;/p&gt;&#xA;" OwnerUserId="2635" LastActivityDate="2011-03-21T15:39:29.850" />
  <row Id="8572" PostTypeId="1" AcceptedAnswerId="8876" CreationDate="2011-03-21T15:51:51.340" Score="10" ViewCount="207" Body="&lt;p&gt;I know a fair amount about fitting continuous parameters particularly gradient-based methods, but not much about fitting discrete parameters. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are commonly used MCMC algorithms/techniques for fitting discrete parameters? Are there algorithms which are both fairly general and fairly powerful? Are there algorithms which deal with the curse of dimensionality well? For example, I would say Hamiltonian MCMC is general, powerful and scales well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sampling from an arbitrary discrete distribution seems more difficult than sampling from a continuous distribution, but I am curious what the state of the art is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit&lt;/strong&gt;: JMS asked me to elaborate. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't have specific applications in mind, but here are some kinds of models I am imagining: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Model selection between several kinds of continuous regression models. You have a discrete single 'model' parameter  &lt;/li&gt;&#xA;&lt;li&gt;A continuous model where each observation has a possibility of being an 'outlier' and drawn from a much more dispersed distribution. I suppose this is a mixture model.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I would expect many models to include both continuous and discrete parameters.&lt;/p&gt;&#xA;" OwnerUserId="1146" LastEditorUserId="1146" LastEditorDisplayName="" LastEditDate="2011-03-31T14:59:59.527" LastActivityDate="2011-03-31T16:45:44.557" Title="What MCMC algorithms/techniques are used for discrete parameters?" Tags="&lt;bayesian&gt;&lt;mcmc&gt;" AnswerCount="1" CommentCount="0" FavoriteCount="7" />
  <row Id="8573" PostTypeId="1" AcceptedAnswerId="8580" CreationDate="2011-03-21T16:06:07.347" Score="2" ViewCount="168" Body="&lt;p&gt;Probability distribution of two classes is given by  $N(5,1)$ and $N(6,1)$ where &#xA;$N(\mu,\sigma^2)$:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$f(x) = \frac{1}{\sqrt{2\pi \sigma^2}} e^{-\frac{(x-\mu)^2}{2\sigma^2}} $$&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;How to classify them, and see error rate?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I am doing this in MATLAB&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Taking 500 samples of each distribution and tagging them depending where they came from.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sample1 =    &#xA;    0.8864   -1.0000&#xA;    0.1560   -1.0000&#xA;    0.8502   -1.0000&#xA;   -0.4059   -1.0000&#xA;    0.9298   -1.0000&#xA;sample2 =&#xA;   -0.0671    1.0000&#xA;    0.7057    1.0000&#xA;    0.3310    1.0000&#xA;   -0.7314    1.0000&#xA;   -0.4524    1.0000&#xA;data =&#xA;    0.8864   -1.0000&#xA;    0.1560   -1.0000&#xA;    0.8502   -1.0000&#xA;   -0.4059   -1.0000&#xA;    0.9298   -1.0000&#xA;   -0.0671    1.0000&#xA;    0.7057    1.0000&#xA;    0.3310    1.0000&#xA;   -0.7314    1.0000&#xA;   -0.4524    1.0000&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now I would like to classify them using Bayes, I am using &lt;code&gt;normpdf&lt;/code&gt;, to make this easier I am taking prior probabilities equal so they are not important in creating the rule, but I do not know how to code this in MATLAB, any idea?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;n=500;&#xA;sample1=[randn(n,1) -1*ones(n,1)];&#xA;sample2=[randn(n,1) ones(n,1)];&#xA;data=[sample1; sample2];&#xA;mu1    =5;&#xA;sigma1 =1;&#xA;mu2    =6;&#xA;sigma2 =1;&#xA;x1=linspace(mu1-1*sigma1,mu1+1*sigma1,500);&#xA;p1=normpdf(x1,mu1,sigma1);  &#xA;x2=linspace(mu2-1*sigma2,mu2+1*sigma2,500);&#xA;p2=normpdf(x2,mu2,sigma2);&#xA;plot(x1,p1,x2,p2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Also Is it correct to label with &lt;code&gt;-1 and 1&lt;/code&gt;?&#xA;If I compute mean and variance of &lt;code&gt;sample1, and sample2&lt;/code&gt;?&#xA; mean_S1=mean(sample1);&#xA; mean_S2=mean(sample2);&#xA; var_S1 = var(sample1);&#xA; var_S2 = var(sample2);&#xA; - What is next step?&#xA; - for error rate Im planning to do a comparison between the original class vector (-1,1) and the result of classifier like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;errorRate = mean(OriginalClasses ~= ResultOfClassifier);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;&lt;strong&gt;&lt;/em&gt;**UPDATED*&lt;em&gt;*&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;clear;&#xA;clc;&#xA;n = 500;&#xA;mu1    =5;&#xA;sigma1 =1;&#xA;mu2    =6;&#xA;sigma2 =1;&#xA;mu = [mu1,mu2];sigma = [sigma1,sigma2];  %group them&#xA;%suppose you get your test data from somewhere. &#xA;%for kicks, I put random data in:&#xA;%xtest = randn(2*n,1);     %OP example code has the labels in the data var; ack&#xA;sample1=[randn(n,1) -1*ones(n,1)];&#xA;sample2=[randn(n,1) ones(n,1)];&#xA;data=[sample1; sample2];&#xA;deviance = bsxfun(@minus,data,mu);  %tbc&#xA;deviance = bsxfun(@rdivide,deviance,sigma); %tbc&#xA;deviance = deviance .^ 2; %tbc&#xA;deviance = bsxfun(@plus,deviance,2*log(abs(sigma))); %tbc&#xA;deviance = deviance + log(2*pi);  %not necessary here, actually;&#xA;[dummy,mini] = min(deviance,[],2);  %find which class;&#xA; ResultOfClassifier = 2 * mini - 3;  %is now a -1/1&#xA; errorRate = mean(data(:,2) ~= ResultOfClassifier)&#xA;&#xA;errorRate =&#xA;&#xA;    0.2680&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="3681" LastEditorUserId="3681" LastEditorDisplayName="" LastEditDate="2011-03-21T18:32:43.573" LastActivityDate="2011-03-21T18:32:43.573" Title="Bayes classifier of two normal distributions in MATLAB" Tags="&lt;bayesian&gt;&lt;classification&gt;&lt;matlab&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="8574" PostTypeId="2" ParentId="8570" CreationDate="2011-03-21T17:29:40.023" Score="3" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;James Gentle's Computational Statistics (2009).&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;James Gentle's Matrix algebra: theory, computations, and applications in statistics‎ (2007)&lt;/strong&gt;, more so towards the end of the book, the beginning is great too but it's not exactly what you're looking for.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Christopher M. Bishop's Pattern Recognition (2006).&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Hastie et al.'s The elements of statistical learning: data mining, inference, and prediction‎ (2009).&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are you looking for something as low-level as a text that will answer a question such as: &quot;Why is it more efficient to store matrices and higher dimensional arrays as a 1-D array, and how can I index them in the usual M(0, 1, 3, ...) way?&quot; or something like &quot;What are some common techniques used to optimize standard algorithms such as gradient descent, EM, etc.?&quot;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most texts on machine learning will provide in-depth discussions of the topic(s) you're looking for.&lt;/p&gt;&#xA;" OwnerUserId="2660" LastActivityDate="2011-03-21T17:29:40.023" CommentCount="2" />
  <row Id="8576" PostTypeId="5" CreationDate="2011-03-21T17:40:28.580" Score="0" ViewCount="" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditorDisplayName="" LastEditDate="2011-03-21T17:40:28.580" LastActivityDate="2011-03-21T17:40:28.580" />
  <row Id="8577" PostTypeId="4" CreationDate="2011-03-21T17:40:28.580" Score="0" ViewCount="" Body="Problems focusing on (but not necessarily limited to) using Python for statistical computation." OwnerUserId="-1" LastEditorUserId="2660" LastEditorDisplayName="" LastEditDate="2011-03-21T17:53:02.750" LastActivityDate="2011-03-21T17:53:02.750" />
  <row Id="8578" PostTypeId="5" CreationDate="2011-03-21T17:46:43.223" Score="0" ViewCount="" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditorDisplayName="" LastEditDate="2011-03-21T17:46:43.223" LastActivityDate="2011-03-21T17:46:43.223" />
  <row Id="8579" PostTypeId="4" CreationDate="2011-03-21T17:46:43.223" Score="0" ViewCount="" Body="Principal component analysis is a technique to decompose an array of numerical data into a set of orthogonal vectors (uncorrelated linear combinations of the variables)." OwnerUserId="-1" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-21T17:52:53.733" LastActivityDate="2011-03-21T17:52:53.733" />
  <row Id="8580" PostTypeId="2" ParentId="8573" CreationDate="2011-03-21T18:19:28.450" Score="2" ViewCount="" Body="&lt;p&gt;Here's how I have done this in matlab:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mu = [mu1,mu2];sigma = [sigma1,sigma2];  %group them&#xA;%suppose you get your test data from somewhere. &#xA;%for kicks, I put random data in:&#xA;xtest = randn(2*n,1);     %OP example code has the labels in the data var; ack&#xA;deviance = bsxfun(@minus,xtest,mu);  %tbc&#xA;deviance = bsxfun(@rdivide,deviance,sigma); %tbc&#xA;deviance = deviance .^ 2; %tbc&#xA;deviance = bsxfun(@plus,deviance,2*log(abs(sigma))); %tbc&#xA;deviance = deviance + log(2*pi);  %not necessary here, actually;&#xA;[dummy,mini] = min(deviance,[],2);  %find which class;&#xA;ResultOfClassifier = 2 * mini - 3;  %is now a -1/1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="795" LastActivityDate="2011-03-21T18:19:28.450" CommentCount="4" />
  <row Id="8581" PostTypeId="1" AcceptedAnswerId="8613" CreationDate="2011-03-21T18:33:17.083" Score="8" ViewCount="189" Body="&lt;h3&gt;Context:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;I'm a Psychology PhD student. As with many psychology PhD students, I know how to perform various statistical analyses using statistical software, up to techniques such as PCA, classification trees, and cluster analysis.&#xA;But it's not really satisfying because though I can explain why I did an analysis and what the indicators mean, I can't explain how the technique works.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The real problem is that mastering statiscal software is easy, but it is limited.&#xA;To learn new techniques in articles requires that I understand how to read mathematical equations. At present I couldn't calculate eigenvalues or K-means. Equations are like a foreign language to me.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Question:&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Is there a comprehensive guide that helps with understanding equations in journal articles?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;h3&gt;EDIT&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Thanks for your comments.&#xA; I thought the question would be more self explanatory: above a certain complexity, statistical notation becomes gibberish for me; let's say I would like to code my own functions in R or C++ to understand a technique but there's a barrier. I can't transform an equation into a program.&#xA;And really: I don't know the situation in US doctoral schools, but in mine (France), the only courses I can follow is about some 16th century litterary movement...&lt;/p&gt;&#xA;" OwnerUserId="3827" LastEditorUserId="3827" LastEditorDisplayName="" LastEditDate="2011-03-22T09:45:41.400" LastActivityDate="2011-03-22T13:58:45.933" Title="How to transition from using statistical software to understanding mathematical equations in journal articles?" Tags="&lt;books&gt;&lt;software&gt;&lt;learning&gt;" AnswerCount="3" CommentCount="7" FavoriteCount="2" />
  <row Id="8582" PostTypeId="2" ParentId="8563" CreationDate="2011-03-21T19:17:16.670" Score="0" ViewCount="" Body="&lt;p&gt;In the first statistics class I ever took, a darts analogy was used. If you have a bunch of darts around the bullseye but fairly spread out you have an accurate but imprecise landing position, whereas if you have a bunch of darts tightly clustered around some other position be it the wall or somewhere on the dartboard you have a precise but inaccurate landing position.&lt;/p&gt;&#xA;" OwnerUserId="2660" LastActivityDate="2011-03-21T19:17:16.670" />
  <row Id="8583" PostTypeId="1" AcceptedAnswerId="8584" CreationDate="2011-03-21T19:51:20.863" Score="8" ViewCount="145" Body="&lt;p&gt;I have conducted an analysis in which I have modeled different variance components. When reporting the results in a table, it is much more concise to report standard deviations instead of variances.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, this brings me to the question - is there ever a reason to report variance instead of standard deviation? Is it ever more appropriate to report one over the other?&lt;/p&gt;&#xA;" OwnerUserId="2750" LastActivityDate="2011-03-21T19:57:27.323" Title="When would it be appropriate to report variance instead of standard deviation?" Tags="&lt;standard-deviation&gt;&lt;variance&gt;&lt;tables&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="8584" PostTypeId="2" ParentId="8583" CreationDate="2011-03-21T19:57:14.420" Score="12" ViewCount="" Body="&lt;p&gt;If you report the mean, then it is more appropriate to report the standard deviation as it is expressed in the same unity. Think about dimensional homogeneity in physics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Moreover, it is easier for the reader to consider confidence intervals (for large n, in order to use the Central Limit Theorem and consider a normal distribution) if the standard deviation is provided rather than the variance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, you may consider reporting the variance if you are interested in comparing variance and bias, or giving &quot;different variance components&quot;, since the total variance is the sum of the intra and inter variances, while the standard deviations do not sum up.&lt;/p&gt;&#xA;" OwnerUserId="1351" LastActivityDate="2011-03-21T19:57:14.420" />
  <row Id="8585" PostTypeId="2" ParentId="8583" CreationDate="2011-03-21T19:57:27.323" Score="7" ViewCount="" Body="&lt;p&gt;This is equivalent. Nonetheless, standard deviation is expressed in the same units as the variable whereas the units of the variance are those of the variable to the power two. This makes standard deviation easier to interpret.&lt;/p&gt;&#xA;" OwnerUserId="3019" LastActivityDate="2011-03-21T19:57:27.323" />
  <row Id="8586" PostTypeId="1" AcceptedAnswerId="8589" CreationDate="2011-03-21T20:22:14.803" Score="6" ViewCount="197" Body="&lt;p&gt;I have a few hundred estimates of a parameter calculated from two different models and I would like to know if these parameters have different variances.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is a straightforward test for comparing the variances of these parameters? (straightforward meaning, least assumptions).&lt;/p&gt;&#xA;" OwnerUserId="2750" LastEditorUserId="2750" LastEditorDisplayName="" LastEditDate="2011-03-21T21:34:42.577" LastActivityDate="2011-04-03T06:51:57.360" Title="How can I test $H_0:\sigma^2_1=\sigma^2_2$?" Tags="&lt;hypothesis-testing&gt;&lt;variance&gt;&lt;mean&gt;" AnswerCount="3" CommentCount="7" FavoriteCount="3" />
  <row Id="8588" PostTypeId="2" ParentId="8586" CreationDate="2011-03-21T20:53:34.403" Score="0" ViewCount="" Body="&lt;p&gt;What about a permutation approach?&lt;/p&gt;&#xA;" OwnerUserId="1050" LastActivityDate="2011-03-21T20:53:34.403" CommentCount="4" />
  <row Id="8589" PostTypeId="2" ParentId="8586" CreationDate="2011-03-21T21:36:33.820" Score="5" ViewCount="" Body="&lt;p&gt;For comparing &lt;em&gt;variances&lt;/em&gt;, Wilcox suggests a percentile bootstrap method. See &lt;a href=&quot;http://books.google.com/books?id=_tAJr4ooOM8C&amp;amp;lpg=PR1&amp;amp;dq=Introduction%20to%20Robust%20Estimation%20and%20Hypothesis%20Testing&amp;amp;pg=PA170#v=onepage&amp;amp;q&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;chapter 5.5.1 of 'Introduction to Robust Estimation and Hypothesis Testing'&lt;/a&gt;. This is available as &lt;code&gt;comvar2&lt;/code&gt; from &lt;a href=&quot;https://r-forge.r-project.org/R/?group_id=468&quot; rel=&quot;nofollow&quot;&gt;the wrs package&lt;/a&gt; in R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;edit&lt;/strong&gt;: to find the amount of bootstrap differences to trim from each side for different values of $\alpha$, one would perform a Monte Carlo study, as suggested by Wilcox. I have a quick and dirty one here in Matlab (duck from thrown shoes):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;randn('state',0);           %to make the results replicable.&#xA;alphas = [0.001,0.005,0.01,0.025,0.05,0.10,0.15,0.20,0.25,0.333];&#xA;nreps  = 4096;&#xA;nsizes = round(2.^ (4:0.5:9));&#xA;nboots = 599;&#xA;cutls  = nan(numel(nsizes),numel(alphas));&#xA;&#xA;for ii=1:numel(nsizes)&#xA;    n = nsizes(ii);&#xA;    imbalance = nan(nreps,1);&#xA;    for jj=1:nreps&#xA;        x1 = randn(n,1);x2 = randn(n,1);&#xA;        %make bootstrap samples;&#xA;        x1b = x1(ceil(n * rand(n,nboots)));&#xA;        x2b = x2(ceil(n * rand(n,nboots)));&#xA;        %compute stdevs&#xA;        sig1 = std(x1b,1);sig2 = std(x2b,1);&#xA;        %compute difference in stdevs&#xA;        Dvar = (sig1.^2 - sig2.^2);&#xA;        %compute the minimum of {the # &amp;lt; 0} and {the # &amp;gt; 0}&#xA;        %in (1-alpha) of the cases you want this minimum to match&#xA;        %your l number; then let u = 599 - l + 1&#xA;        imbalance(jj,1) = min(sum(Dvar &amp;lt; 0),sum(Dvar &amp;gt; 0));&#xA;    end&#xA;    imbalance = sort(imbalance);&#xA;    cutls(ii,:) = interp1(linspace(0,1,numel(imbalance)),imbalance(:)',alphas,'nearest');&#xA;end&#xA;%plot them;&#xA;lh = loglog(nsizes(:),cutls + 1);&#xA;legend(lh,arrayfun(@(x)(sprintf('alpha = %g',x)),alphas,'UniformOutput',false))&#xA;ylabel('l + 1');&#xA;xlabel('sample size, n_m');&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I get the rather unhelpful plot:&#xA;&lt;img src=&quot;http://i.imgur.com/CPXKc.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A little bit of hackery indicates that a model of the form $l + 0.5 = \exp{5.18} \alpha^{0.94} n^{0.067}$ fits my Monte Carlo simulations fairly well, but they do not give the same results that Wilcox quotes in his book. You might be better served running these experiments yourself at your preferred $\alpha$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;edit&lt;/strong&gt; I ran this experiment again, using many more replicates ($2^{18}$) per sample size. Here's a table of the empirical values of $l$. The first row is a NaN, then the alpha (type I rate). Following that, the first column is the size of the samples, $n$, then the empirical values of $l$. (I would expect that as $n \to \infty$ we would have $l \to 599 \alpha /2$)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;NaN,0.001,0.005,0.01,0.025,0.05,0.1,0.15,0.2,0.25,0.333&#xA;16,0,0,1,4,9,22,35,49,64,88&#xA;23,0,0,1,4,10,23,37,51,66,91&#xA;32,0,0,1,4,10,24,38,52,67,92&#xA;45,0,0,1,5,11,25,39,54,69,94&#xA;64,0,0,2,5,12,26,41,55,70,95&#xA;91,0,1,2,6,13,27,42,56,71,96&#xA;128,0,1,2,6,13,28,42,58,72,97&#xA;181,0,1,2,6,13,28,43,58,73,98&#xA;256,0,1,2,6,14,28,43,58,73,98&#xA;362,0,1,2,7,14,29,44,59,74,99&#xA;512,0,1,2,7,14,29,44,59,74,99&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="795" LastEditorUserId="795" LastEditorDisplayName="" LastEditDate="2011-03-24T17:17:31.463" LastActivityDate="2011-03-24T17:17:31.463" CommentCount="9" />
  <row Id="8590" PostTypeId="1" CreationDate="2011-03-21T21:39:32.833" Score="6" ViewCount="243" Body="&lt;p&gt;I come from the social sciences, where p &amp;lt; 0.05 is pretty much the norm, with p &amp;lt; 0.1 and p &amp;lt; 0.01 also showing up, but I was wondering: what fields of study, if any, use lower p-values as a common standard?&lt;/p&gt;&#xA;" OwnerUserId="3582" LastActivityDate="2011-03-21T23:26:06.033" Title="Examples of studies using p &lt; 0.001, p &lt; 0.0001 or even lower p-values?" Tags="&lt;statistical-significance&gt;&lt;p-value&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="8591" PostTypeId="1" CreationDate="2011-03-21T16:57:30.150" Score="4" ViewCount="91" Body="&lt;p&gt;How do probability distributions of continuous random variables transform under functions? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I.e. I have a random variable, X, drawn from a normal distribution with mean 0 and variance 1. What is the probability distribution associated with sin(X)?&#xA;&lt;img src=&quot;http://i.imgur.com/KY5Wc.png&quot; alt=&quot;Histograms mimicking probability density functions of X and sin(X)&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More generally, what are the rules for transforming continuous random variables? If we know the PDF and CDF of two random variables, X,Y what is the PDF/CDF of Z=X*Y ? How about Z=X^Y ? How about Z = sin(X+Y)+3?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there Computer Algebra Systems which can compute this symbolically? Is this possible generally? If not, for what class of probability distributions and functions is it possible?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: excuse the plot. These are obviously noisy histograms and obviously not to scale (areas  under the curves do not match and so could not both sum to one). Hopefully the plot does get the point across though. Given the blue distribution describing X, I want the red distribution describing sin(X). &lt;/p&gt;&#xA;" OwnerUserId="3830" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-04-08T20:45:09.330" LastActivityDate="2011-04-08T20:45:09.330" Title="Operations on a probability distributions of continuous random variables" Tags="&lt;distributions&gt;&lt;probability&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="8592" PostTypeId="2" ParentId="8591" CreationDate="2011-03-21T17:14:00.880" Score="3" ViewCount="" Body="&lt;p&gt;See this &lt;a href=&quot;http://en.wikipedia.org/wiki/Random_variable#Functions_of_random_variables&quot; rel=&quot;nofollow&quot;&gt;Wikipedia entry&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="319" LastActivityDate="2011-03-21T17:14:00.880" />
  <row Id="8593" PostTypeId="2" ParentId="8590" CreationDate="2011-03-21T22:14:36.940" Score="5" ViewCount="" Body="&lt;p&gt;My opinion is that it does (and should) not depend on the field of study. For example, you may well work at a lower significance level than $p&amp;lt;0.001$ if, for example, you are trying to replicate a study with historical or well-established results (I can think of several studies on the &lt;a href=&quot;http://en.wikipedia.org/wiki/Stroop_effect&quot; rel=&quot;nofollow&quot;&gt;Stroop effect&lt;/a&gt;, which had led to some controversies in the past few years). That amounts to consider a lower &quot;threshold&quot; within the classical Neyman-Pearson framework for testing hypothesis. However, statistical and practical (or substantive) significance is another matter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Sidenote&lt;/strong&gt;.&#xA;The &quot;star system&quot; seems to have dominated scientific inquiries as early as the 70's, but see The Earth Is Round (p &amp;lt; .05), by J. Cohen (&lt;em&gt;American Psychologist&lt;/em&gt;, 1994, 49(12), 997-1003), despite the fact that what we often want to know is given the data I have observed, what is the probability that $H_0$ is true? Anyway, there's also a nice discussion on &quot;&lt;a href=&quot;http://www.jerrydallal.com/LHSP/p05.htm&quot; rel=&quot;nofollow&quot;&gt;Why P=0.05?&lt;/a&gt;&quot;, by Jerry Dallal.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-03-21T22:14:36.940" CommentCount="4" />
  <row Id="8595" PostTypeId="2" ParentId="4086" CreationDate="2011-03-21T23:19:35.030" Score="0" ViewCount="" Body="&lt;p&gt;you have 5 years of data and 40 observations per year. Why don't you post them on the web and allow us to actually answer this at ground zero rather than philosophizing at 500 miles high.&#xA;I look forward to the numbers. WE have seen data like this for example the number of customers who trade in their time sharing week ON a weekly basis. The series each year starts at zero and accumulates to a limiting value.&lt;/p&gt;&#xA;" OwnerUserId="3382" LastActivityDate="2011-03-21T23:19:35.030" CommentCount="4" />
  <row Id="8596" PostTypeId="2" ParentId="8590" CreationDate="2011-03-21T23:26:06.033" Score="3" ViewCount="" Body="&lt;p&gt;It might be rare for anyone to use a pre-specified alpha level lower than, say, 0.01, but it is not nearly as rare that people claim an implied alpha of less than 0.01 in the mistaken belief that an observed P value of less than 0.01 is the same as a Neyman-Pearson alpha of less than 0.01.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Fisher's P values are not the same as, or interchangeable with, Neyman-Pearson error rates. P = 0.0023 does not mean alpha = 0.0023 unless one has decided to use 0.0023 as the critical level for significance when the experiment is designed. If you would have taken P = 0.05 as significant then P = 0.0023 means that there is an 0.05 probability of a false positive claim.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Have a look at Hubbard et al. Confusion over Measures of Evidence (p's) versus Errors (α's) in Classical Statistical Testing. The American Statistician (2003) vol. 57 (3) (http://www.jstor.org/stable/30037265)&lt;/p&gt;&#xA;" OwnerUserId="1679" LastActivityDate="2011-03-21T23:26:06.033" CommentCount="5" />
  <row Id="8597" PostTypeId="2" ParentId="7733" CreationDate="2011-03-21T23:41:19.940" Score="0" ViewCount="" Body="&lt;p&gt;you might want to rank stocks on the basis of standard dev of a forecast divided by the forecast. In this way the &quot;tighter&quot; the value the more predictable the stock.&lt;/p&gt;&#xA;" OwnerUserId="3382" LastActivityDate="2011-03-21T23:41:19.940" CommentCount="3" />
  <row Id="8598" PostTypeId="1" AcceptedAnswerId="8602" CreationDate="2011-03-21T23:43:26.520" Score="2" ViewCount="155" Body="&lt;p&gt;Hello.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm working on a problem as follows for a course that I'm auditing:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Suppose a 95% symmetric t-interval is applied to estimate a mean, but the&#xA;  sample data are non-normal. Then the probability that the confidence interval covers&#xA;  the mean is not necessarily equal to 0.95.&#xA;  Use a Monte Carlo experiment to&#xA;  estimate the coverage probability of&#xA;  the t-interval for random samples of&#xA;  $\chi^2(2)$ data with sample size $n = 20$.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Here is the current state of my R code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;alpha = 0.05;&#xA;n = 20;&#xA;m = 1000;&#xA;&#xA;UCL = numeric(m);&#xA;LCL = numeric(m);&#xA;&#xA;for(i in 1:m)&#xA;{&#xA;    x = rchisq(n, 2);&#xA;    LCL[i] = mean(x) - qt(alpha / 2, lower.tail = FALSE) * sd(x);&#xA;    UCL[i] = mean(x) + qt(alpha / 2, lower.tail = FALSE) * sd(x);&#xA;}&#xA;&#xA;# This line below is wrong...&#xA;mean(LCL &amp;gt; 0 &amp;amp; UCL &amp;lt; 0);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The problem is that the result is $0$. Am I approaching this question incorrectly? What exactly does coverage probability mean...?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you :)&lt;/p&gt;&#xA;" OwnerUserId="3832" LastEditorUserId="3832" LastEditorDisplayName="" LastEditDate="2011-03-22T01:32:21.843" LastActivityDate="2011-03-22T07:20:49.733" Title="Monte Carlo experiment to estimate coverage probability" Tags="&lt;r&gt;&lt;homework&gt;&lt;monte-carlo&gt;&lt;simulation&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="8599" PostTypeId="2" ParentId="6498" CreationDate="2011-03-21T23:53:18.980" Score="-1" ViewCount="" Body="&lt;p&gt;An ARIMA model is simply a weighted average. It answers the double question;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;How many period (k )should I use to compute a weighted average&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;and&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Precisely what are the  k weights &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;It answers the maiden's prayer to determine how to adjust to previous values ( and previous values ALONE ) in order to project the series ( which is really being caused by unspecified causal variables ) Thus an ARIMA model is a poor man's causal model .&lt;/p&gt;&#xA;" OwnerUserId="3382" LastActivityDate="2011-03-21T23:53:18.980" CommentCount="6" />
  <row Id="8601" PostTypeId="2" ParentId="8598" CreationDate="2011-03-22T00:36:10.603" Score="-1" ViewCount="" Body="&lt;p&gt;You have several issues with your code:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Your &lt;code&gt;mean(UCL &amp;lt; 0 &amp;amp; LCL &amp;gt; 0)&lt;/code&gt; is decidedly strange, and in particular is failing because UCL is coming out positive so you are taking the mean of an empty set.  A $\chi^2$ distribution takes only positive values.&lt;/li&gt;&#xA;&lt;li&gt;(since solved) You have UCL less than LCL, which is a slightly odd use of &lt;em&gt;upper&lt;/em&gt; and &lt;em&gt;lower&lt;/em&gt;&lt;/li&gt;&#xA;&lt;li&gt;You do not need semicolons in R unless you want more than one instruction on the same line &lt;/li&gt;&#xA;&lt;li&gt;(Wrong - as pointed out by mark999) You are dividing by &lt;code&gt;sqrt(n)&lt;/code&gt;.  This wrongly narrows your confidence intervals: it is for finding the standard error of the mean, but you care about the original distribution.&lt;/li&gt;&#xA;&lt;li&gt;The question tells you to use &quot;t-interval&quot; but you are using a normal distribution. You might try typing &lt;code&gt;?qt&lt;/code&gt; into R&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Try this&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;alpha = 0.05&#xA;n = 20&#xA;m = 1000&#xA;&#xA;UCL = numeric(m)&#xA;LCL = numeric(m)&#xA;&#xA;for(i in 1:m)&#xA;{&#xA;    x = rchisq(n, 2) # compare with x = rnorm(n) + 2&#xA;    LCL[i] = mean(x) - qt(alpha / 2, df=n-1, lower.tail = FALSE)*sd(x)/sqrt(n)&#xA;    UCL[i] = mean(x) + qt(alpha / 2, df=n-1, lower.tail = FALSE)*sd(x)/sqrt(n)&#xA;}&#xA;&#xA;mean(LCL &amp;lt; 2 &amp;amp; UCL &amp;gt; 2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2958" LastEditorUserId="2958" LastEditorDisplayName="" LastEditDate="2011-03-22T07:20:49.733" LastActivityDate="2011-03-22T07:20:49.733" CommunityOwnedDate="2011-03-22T00:36:10.603" CommentCount="3" />
  <row Id="8602" PostTypeId="2" ParentId="8598" CreationDate="2011-03-22T02:49:38.337" Score="4" ViewCount="" Body="&lt;p&gt;I disagree with Henry - I think you should be dividing by sqrt(n), because it's a confidence interval for the mean. You also have to add a &lt;code&gt;df = n-1&lt;/code&gt; argument to your qt calls.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And the last line should be &lt;code&gt;mean(LCL &amp;lt; 2 &amp;amp; UCL &amp;gt; 2)&lt;/code&gt;. This is because 2 is the true mean, and you're interested in the condition that 2 is in the confidence interval.&lt;/p&gt;&#xA;" OwnerUserId="3835" LastActivityDate="2011-03-22T02:49:38.337" CommentCount="1" />
  <row Id="8603" PostTypeId="2" ParentId="8566" CreationDate="2011-03-22T02:53:50.613" Score="0" ViewCount="" Body="&lt;p&gt;Naive Bayes and Logistic Regression (Classification) are both linear classifiers. If you remove all misclassified instances, then you will allow an infinite number of separators to have 0 training error. In the case of the logistic regression, this translate to your information matrix being singular (The information matrix must be inverted at each iteration of GLM). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't know if that's what you mean by overfit.&lt;/p&gt;&#xA;" OwnerUserId="3834" LastActivityDate="2011-03-22T02:53:50.613" />
  <row Id="8604" PostTypeId="1" CreationDate="2011-03-22T03:41:20.293" Score="4" ViewCount="179" Body="&lt;p&gt;I admit I'm relatively new to propensity scores and causal analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One thing that's not obvious to me as a newcomer is how the &quot;balancing&quot; using propensity scores is mathematically different from what happens when we add covariates in a regression? What's different about the operation, and why is it (or is it) better than adding subpopulation covariates in a regression?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've seen some studies that do an empirical comparison of the methods, but I haven't seen a good discussion relating the mathematical properties of the two methods and why PSM lends itself to causal interpretations while including regression covariates does not. There also seems to be a lot of confusion and controversy in this field, which makes things even more difficult to pick up.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any thoughts on this or any pointers to good resources/papers to better understand the distinction? (I'm slowly making my way through Judea Pearl's causality book, so no need to point me to that)&lt;/p&gt;&#xA;" OwnerUserId="3836" LastActivityDate="2011-03-23T06:18:03.407" Title="How are propensity scores different from adding covariates in a regression, and when are they preferred to the latter?" Tags="&lt;regression&gt;&lt;multivariate-analysis&gt;&lt;causal-inference&gt;&lt;propensity-scores&gt;" AnswerCount="3" FavoriteCount="1" />
  <row Id="8605" PostTypeId="1" AcceptedAnswerId="8612" CreationDate="2011-03-22T04:17:39.163" Score="2" ViewCount="207" Body="&lt;p&gt;I would like to perform column-wise normalization of a matrix in R. Given a matrix &lt;code&gt;m&lt;/code&gt;, I want to normalize each column by dividing each element by the sum of the column. One (hackish) way to do this is as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;m / t(replicate(nrow(m), colSums(m)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is there a more succinct/elegant/efficient way to achieve the same task?&lt;/p&gt;&#xA;" OwnerUserId="1537" LastEditorUserId="1537" LastEditorDisplayName="" LastEditDate="2011-03-23T20:16:41.383" LastActivityDate="2011-03-23T20:16:41.383" Title="Column-wise matrix normalization in R" Tags="&lt;r&gt;&lt;data-transformation&gt;&lt;normalization&gt;&lt;matrix&gt;" AnswerCount="1" FavoriteCount="2" />
  <row Id="8606" PostTypeId="1" CreationDate="2011-03-22T04:28:52.787" Score="1" ViewCount="76" Body="&lt;p&gt;I am working with an optimization algorithm that produces upper- and lowerbounds for the objective value of an optimization problem. I am trying to devise a good &quot;stopping rule&quot; for this algorithm. Theoretically speaking, the optimization algorithm should stop when the upperbound is equal to the lowerbound.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my case, statistics comes into play because I have to use a sample average estimate of the upperbound, where &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\hat{UB} = \sum_{i=0}^n UB_i$)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I should also add that I have to estimate the standard deviation of the upperbound from the data &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\hat{\sigma}_{UB} = \sum_{i=0}^{i=n} (UB_i - \hat{UB})^2$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In contrast to the upperbound, the lowerbound ($LB$) is a deterministic value. This setup may seem weird to some of you, but it has to do with the fact that my algorithm is used for stochastic optimization problems. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The stopping rule that I have in mind is to stop when $\frac{\hat{UB}}{LB} = 1$. More formally, the test boils down to:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$H_0: \frac{\hat{UB}}{LB} = 1$  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$H_A:  \frac{\hat{UB}}{LB} &amp;gt; 1$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The formulation is a little twisted, but I would stop when I fail to reject the null hypothesis and yet also ensure that $P(\frac{\hat{UB}}{LB} &amp;gt; 1 \text{ } |  \text{ fail to reject }H_0) = \alpha$. I am thinking of doing this by running a one-sided t-test that has a significance level of $1 - \alpha$. Would this be right?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As usual I appreciate any input, either on my specific question or the problem in general!&lt;/p&gt;&#xA;" OwnerUserId="3572" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-22T22:57:25.897" LastActivityDate="2011-03-23T06:01:07.603" Title="Stopping rule that would accept $H_0$ with a given probability" Tags="&lt;hypothesis-testing&gt;&lt;optimization&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="8607" PostTypeId="2" ParentId="8541" CreationDate="2011-03-22T04:41:00.800" Score="1" ViewCount="" Body="&lt;p&gt;Short answer: Gibbs or Metropolis-Hastings-within-Gibbs (MCMC) should work just fine on joint distributions and full conditional distributions that are mixed products of pmfs and pdfs. If you're doing MCMC, just make sure that sampling from the candidate distributions gives you values in the right domain.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Long answer:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The most comprehensive way we know of to account for probability is to use measure theory. In measure-theoretic probability theory, there is little difference between pmfs and pdfs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In measure-theoretic probability, pmfs and pdfs are both called &quot;densities.&quot; More precisely, they are both Radon-Nikodym (pronounced &quot;RahDOHN-NickohDEEM&quot;) derivatives. The only difference is that a pmf is a derivative with respect to a counting measure, and a pdf is a derivative with respect to Lebesgue (&quot;LehBAYG&quot;) measure (i.e. n-dimensional volume). When you integrate under a pmf, you integrate with respect to counting measure: you sum over part of its domain. When you integrate under a pdf, you integrate with respect to Lebesgue measure. In every case that Bayesians tend to care about, the latter is equivalent to regular old Riemann integration.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because pmfs and pdfs are the same kind of thing, they obey the same laws, such as Bayes' law and the product rule. Thus, you can use both pmfs and pdfs in a model and get a meaningful joint density and meaningful conditional densities. But these densities won't necessarily be pdfs or pmfs. In general, they will be Radon-Nikodym derivatives with respect to products of counting and Lebesgue measures.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Riemann integration can't handle Radon-Nikodym derivatives with respect to mixed spaces like that, but Lebesgue integration can. Sampling methods approximately carry out Lebesgue integration, so they just work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might be wondering, then, what &lt;i&gt;doesn't&lt;/i&gt; just work. Well, the only thing I've mentioned so far are products of densities and integrating under those products. A random variable with a distribution that is the sum of a distribution with a pmf and a distribution with a pdf can cause problems.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any easy way to get a sum like that is to use an &quot;if.&quot; For example, say you have this model:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$X \sim \mathrm{Normal}(0,1)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Y \sim \mathrm{Geometric}(1/2)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$B \sim \mathrm{Bernoulli}(1/2)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and you define a new random variable $Z$ by $Z = X$ if $B = 0$, otherwise $Z = Y$. The distribution of $Z$ can't be represented by a pmf or a pdf: its cdf has a discontinuity at each positive integer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can still approximate $Z$'s distribution using samples. You can even use $Z$ as a parameter for another random variable's distribution, and answer conditional queries with a Gibbs sampler. But conditioning on $Z$ in a query will &lt;i&gt;not&lt;/i&gt; work without measure-theoretic tools. Also, you might find your current tools, like the density version of Bayes' law, inadequate. (I haven't looked into it enough to say when.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another easy way to get a non-pmf/non-pdf distribution is to truncate a random variable; something like $W = X$ if $X \le 0$, otherwise $W = 0$. (Note that truncating $X$'s distribution won't give you $W$'s, because $P[W=0] = 1/2$. Truncated distributions and truncated random variables are very different things.) $W$ would give you the same problems as $Z$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other things off the cuff:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might have a hard time getting good n-dimensional KDEs that have discrete axes, if you want them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're doing Metropolis-Hastings, you might have a hard time coming up with candidate pmfs for categorical random variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;--&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want another way to think about it, consider what would happen if you transformed your model. Convert your pmfs into pdfs and floor the corresponding discrete random variables before using them as parameters in other random variables' distributions. The joint distribution that includes the un-floored random variables (but not the floored ones) obviously has a pdf.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The easiest way to get an equivalent model is to define each new pdf $d$ so that, if its corresponding pmf is $m$, $d(x) = m(\lfloor x \rfloor)$. Now suppose you created an MCMC sampler from the transformed model. It shouldn't take too long to convince yourself that the fractional part of a candidate sample for any originally discrete random variable contributes nothing to the accept/reject decision.&lt;/p&gt;&#xA;" OwnerUserId="3831" LastEditorUserId="3831" LastEditorDisplayName="" LastEditDate="2011-03-22T04:50:36.987" LastActivityDate="2011-03-22T04:50:36.987" CommentCount="1" />
  <row Id="8608" PostTypeId="1" AcceptedAnswerId="8647" CreationDate="2011-03-22T04:58:01.640" Score="2" ViewCount="76" Body="&lt;p&gt;I am reading this example, but could you explain a little more.  I don't get the part where it says &quot;then we Normalize&quot;... I know&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;P(sun) * P(F=bad|sun)  = 0.7*0.2 = 0.14&#xA;P(rain)* P(F=bad|rain) = 0.3*0.9 = 0.27&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But where do they get&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;W    P(W | F=bad)&#xA;-----------------&#xA;sun     0.34&#xA;rain    0.66&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/9ZWgQ.png&quot; alt=&quot;enter image description here&quot;&gt;&#xA;&lt;img src=&quot;http://i.imgur.com/fkFsh.png&quot; alt=&quot;enter image description here&quot;&gt;&#xA;&lt;img src=&quot;http://i.imgur.com/74J9o.png&quot; alt=&quot;enter image description here&quot;&gt;&#xA;Example &lt;a href=&quot;http://inst.eecs.berkeley.edu/~cs188/fa10/slides/FA10%20cs188%20lecture%2018%20--%20decision%20diagrams%20%286PP%29.pdf&quot; rel=&quot;nofollow&quot;&gt;from&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="3681" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-29T00:58:38.457" LastActivityDate="2011-04-29T00:58:38.457" Title="Decision network example" Tags="&lt;probability&gt;&lt;bayesian&gt;&lt;conditional-probability&gt;" AnswerCount="2" />
  <row Id="8610" PostTypeId="2" ParentId="8604" CreationDate="2011-03-22T05:19:25.640" Score="3" ViewCount="" Body="&lt;p&gt;The short answer is that propensity scores are not any better than the equivalent ANCOVA model, particularly with regard to causal interpretation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Propensity scores are best understood as a data reduction method.  They are an effective means to reduce many covariates into a single score that can be used to adjust an effect of interest for a set of variables.  In doing so, you save degrees of freedom by adjusting for a single propensity score rather than multiple covariates.  This presents a statistical advantage, certainly, but nothing more.&lt;/p&gt;&#xA;" OwnerUserId="485" LastActivityDate="2011-03-22T05:19:25.640" CommentCount="5" />
  <row Id="8611" PostTypeId="1" CreationDate="2011-03-22T05:21:30.313" Score="3" ViewCount="148" Body="&lt;p&gt;I have two different columns of data which are recorded in different configurations and &lt;strong&gt;I want to show the users that these two records varie&lt;/strong&gt; (data is time in seconds). The dataset is not of the same size as shown below. The end users are all experienced people in stats and math.  My question is how can I plot a graph (which kind of(?)) easily.&#xA;First dataset:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;0.000223&#xA;0.000206&#xA;0.000223&#xA;0.000193&#xA;0.001321&#xA;0.000223&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Second dataset:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;0.076975003&#xA;0.076724999&#xA;0.076600999&#xA;0.0766&#xA;0.050742&#xA;0.000397&#xA;0.000642&#xA;0.000522&#xA;0.000772&#xA;0.000522&#xA;0.076725997&#xA;0.158800006&#xA;0.159801006&#xA;0.159426004&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Some of the graphs I came across: boxplot, meantime.&lt;/p&gt;&#xA;" OwnerUserId="3270" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-22T16:38:59.353" LastActivityDate="2011-03-22T17:53:05.827" Title="How to show differences between two univariate datasets graphically?" Tags="&lt;data-visualization&gt;&lt;matlab&gt;&lt;gnuplot&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="8612" PostTypeId="2" ParentId="8605" CreationDate="2011-03-22T06:07:21.883" Score="5" ViewCount="" Body="&lt;p&gt;This is what sweep and scale are for.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sweep(m, 2, colSums(m), FUN=&quot;/&quot;)&#xA;scale(m, center=FALSE, scale=colSums(m))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Alternatively, you could use recycling, but you have to transpose it twice.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;t(t(m)/colSums(m))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Or you could construct the full matrix you want to divide by, like you did in your question.  Here's another way you might do that.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;m/colSums(m)[col(m)]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="3601" LastActivityDate="2011-03-22T06:07:21.883" CommentCount="1" />
  <row Id="8613" PostTypeId="2" ParentId="8581" CreationDate="2011-03-22T06:18:34.707" Score="6" ViewCount="" Body="&lt;h3&gt;Overview:&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;My impression is that your experience is common to a lot of students in the social sciences.&lt;/li&gt;&#xA;&lt;li&gt;The starting point is a motivation to learn.&lt;/li&gt;&#xA;&lt;li&gt;You can either go down the &lt;strong&gt;self-taught&lt;/strong&gt; or &lt;strong&gt;formal instruction&lt;/strong&gt; routes.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;Formal instruction:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;There are many options in this regard.&#xA;You might consider a masters in statistics or just taking a few subjects in a statistics department.&#xA;However, you'd probably want to check that you have the necessary mathematical background. Depending on the course, you may find that you need to revisit pre-calculus mathematics, and perhaps some material such as calculus and linear algebra before tackling university-level mathematically rigorous statistics subjects.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Self-taught&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Alternatively, you could go down the self-taught route.&#xA;There are heaps of good resources on the internet.&#xA;In particular, reading and doing exercises in mathematics text books is important, but probably not sufficient. It's important to listen to instructors talking about mathematics and watch them solve problems.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's also important to think about your mathematical goals and the mathematical prerequisites required to achieve those goals. If equations are like a foreign language to you, then you may find that you need to first study elementary mathematics. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've prepared a few resources aimed at assisting people who are making the transition from using statistical software to understanding the underlying mathematics.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Videos&lt;/strong&gt;: &lt;a href=&quot;http://jeromyanglim.blogspot.com/2009/05/online-mathematics-video-courses-for.html&quot; rel=&quot;nofollow&quot;&gt;List of Free Online Mathematics Videos&lt;/a&gt; - This post also provides some guidance regarding what would be an appropriate mathematical sequence starting from pre-calculus and working through calculus, linear algebra, probability, and mathematical statistics.&#xA;Also see this question on &lt;a href=&quot;http://stats.stackexchange.com/questions/485/mathematical-statistics-videos&quot;&gt;mathematical statistics videos&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Reading and pronunciation&lt;/strong&gt; - One of the first challenges is learning how to pronounce and read mathematical equations. I wrote two posts, one on &lt;a href=&quot;http://jeromyanglim.blogspot.com/2009/05/pronunciation-guides-for-mathematical.html&quot; rel=&quot;nofollow&quot;&gt;pronunciation&lt;/a&gt; and another on &lt;a href=&quot;http://jeromyanglim.blogspot.com/2010/01/tips-on-reading-mathematics-for-non.html&quot; rel=&quot;nofollow&quot;&gt;tips for reading mathematics for the non-mathematician&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Writing&lt;/strong&gt; - Learning to write mathematics can help with reading mathematics. Try learning LaTeX and check out some of the guides on &lt;a href=&quot;http://en.wikibooks.org/wiki/LaTeX/Mathematics&quot; rel=&quot;nofollow&quot;&gt;mathematics in LaTeX&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Books&lt;/strong&gt;: When it comes to learning mathematics, I think it is worth investing in a few good textbooks. However, there are a lot of &lt;a href=&quot;http://jeromyanglim.blogspot.com/2010/01/free-online-mathematics-books.html&quot; rel=&quot;nofollow&quot;&gt;free online options&lt;/a&gt; these days&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-03-22T10:51:37.847" LastActivityDate="2011-03-22T10:51:37.847" CommentCount="2" />
  <row Id="8614" PostTypeId="1" AcceptedAnswerId="8622" CreationDate="2011-03-22T06:54:59.047" Score="1" ViewCount="162" Body="&lt;p&gt;I am trying to determine if a given noise from a compass sensor is time-correlated (it is supposed to be!) and for that I tried to compute the cross correlation between the noise signal and the time of sampling using Matlab xcorr() function. However, I am getting a random value indicating that it is not time-correlated, whereas in reality it has to be that way. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Am I doing something wrong? I am not able to find references to determine if a signal is time-correlated, so any ideas would be greatly appreciated!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Imelza&lt;/p&gt;&#xA;" OwnerUserId="3233" LastActivityDate="2011-03-31T22:39:28.480" Title="How to determine if a given signal is time-correlated?" Tags="&lt;time-series&gt;&lt;matlab&gt;&lt;cross-correlation&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="1" />
  <row Id="8615" PostTypeId="2" ParentId="8606" CreationDate="2011-03-22T07:17:13.800" Score="2" ViewCount="" Body="&lt;p&gt;You should take the difference between upper and lower, then t-test migh be more suitable than with ratios... &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyway, if the probability to reject the null wrongly is $1-\alpha$ the probability to accept it wrongly won't be $\alpha$, if you want to control the error of accepting the null wrongly you have to specify one alternative or at least to assume it is separated from the null and use the &quot;extreme&quot; point of the alternative to compute the threshold at the level  you want.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I really like this quote from Tukey: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;All we know about the world teaches us that the effects of A and B&#xA;  are always different---in some decimal&#xA;  place---for any A and B. Thus asking&#xA;  &quot;are the effects different?&quot; is&#xA;  foolish.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;note that you should work on differentiating what are the parameters (deterministic) and what are the random variables in your question. In addition you should explain a bit more how you obtain these upper and lower bound, might help to get a better answer than mine. &lt;/p&gt;&#xA;" OwnerUserId="223" LastEditorUserId="223" LastEditorDisplayName="" LastEditDate="2011-03-22T07:37:55.537" LastActivityDate="2011-03-22T07:37:55.537" />
  <row Id="8616" PostTypeId="2" ParentId="8566" CreationDate="2011-03-22T07:39:48.263" Score="4" ViewCount="" Body="&lt;p&gt;&lt;em&gt;The following is not restricted to NB + LogRes&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Overfitting = Loss of generalization.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you train a model on dataset you generally assume that the data you use for training has a similar structure than the data the model is applied to later (the general assumption of predicting the future from the past). So if you remove parts of the data (namely the misclassified instances) and train a model on this reduced dataset, you effectively change the structure of the data in comparison to the test dataset (and hence violate this assumption). In this case the following can happen (when testing this model on an unreduced test-dataset):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;In the best case&lt;/strong&gt; nothing happens, e.g. of the following reasons:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The missclassified instances represented only a tiny subspace of the dataspace (corresponds to a high accuracy achieved by the first model)&lt;/li&gt;&#xA;&lt;li&gt;The model classifies one part of the dataspace better and another one worse so that they even out.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;In the worst case&lt;/strong&gt; the quality decreases rapidly, because of overfitting / loss of generalization power. The model focuses too hard on the part of the dataspace of in first step correctly classified instances and hence is not able anymore to make even an approximate statement for the rest of the dataspace. &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;I think what you are actually looking for is called &lt;a href=&quot;http://en.wikipedia.org/wiki/Boosting&quot; rel=&quot;nofollow&quot;&gt;Boosting&lt;/a&gt;, where one restricts the dataspace to the missclassified instances (i.e. doing the opposite of your strategy) to refine the model. The procedure tries to avoid overfitting by combining the different (subspace-)models afterwards, but nevertheless it is still an issue.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an &lt;a href=&quot;http://stats.stackexchange.com/questions/7813/adjusting-sample-weights-in-adaboost/7877#7877&quot;&gt;plain-text explanation of boosting&lt;/a&gt; with a illustrative graphic you might find helpful.&lt;/p&gt;&#xA;" OwnerUserId="264" LastActivityDate="2011-03-22T07:39:48.263" />
  <row Id="8617" PostTypeId="1" AcceptedAnswerId="8618" CreationDate="2011-03-22T07:56:30.263" Score="10" ViewCount="258" Body="&lt;p&gt;An increase in the number of cases and deaths occurs during epidemics (sudden increase in numbers) due to a virus circulation (like West Nile Virus in USA in 2002) or decreasing resistance of people or contamination of food or water or increase in the number of mosquitoes. &#xA;These epidemics will present as outliers which can occur every 1 to 5 years. &#xA;By removing these outliers we are removing evidence of epidemics which form an important part of forecasting and disease understanding. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is data cleaning necessary while dealing with outliers caused by epidemics?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it going to improve the results or worsen the results of statistical analysis?&lt;/p&gt;&#xA;" OwnerUserId="2956" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-22T22:55:17.803" LastActivityDate="2011-03-24T12:21:41.397" Title="Can data cleaning worsen the results of statistical analysis?" Tags="&lt;time-series&gt;&lt;forecasting&gt;&lt;epidemiology&gt;&lt;outliers&gt;" AnswerCount="4" CommentCount="1" FavoriteCount="4" />
  <row Id="8618" PostTypeId="2" ParentId="8617" CreationDate="2011-03-22T08:51:11.383" Score="7" ViewCount="" Body="&lt;p&gt;It actually depends on the purpose of your research. In my opinion, there could be several:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;You want to understand what are the typical factors that causes cases and deaths and that are not affected by epidemic periods and factors that causes epidemics (so you are interested in typical not force major probabilities) - in this case you obviously need to remove the epidemic periods from the data, as they are by the purpose of research the &lt;em&gt;outliers&lt;/em&gt; to what you would like to conclude&lt;/li&gt;&#xA;&lt;li&gt;You may want to include epidemic changes into your models (regime-switching models, for instance, any good links and model suggestions from the community are welcome here), because you want to know the probability of epidemic period to occur (and also how long it will last), to test the stability and to &lt;em&gt;forecast&lt;/em&gt; - in this case you do not exclude epidemic periods, but search for more complicated models rather than go for hammer-econometric-tool $OLS$ or something similar &lt;/li&gt;&#xA;&lt;li&gt;Your primarily goal IS to detect epidemic periods and monitor for them in real-time - it's a special field in econometrics a number of my colleagues are working with at Vilnius University (definitely, you would like to have a lot of epidemic observations to deal with) &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;So if your primarily goal is something like 2, clearing the data will cause wrong conclusions about the future forecasts, i.e. inaccurate forecasting performance. It is also true that the 2nd case does not necessarily provide better forecasts, but you at least could make conclusions about the probabilities of epidemic periods and their length. This IS vitally important for actuarial mathematicians, so may be you are the one?&lt;/p&gt;&#xA;" OwnerUserId="2645" LastActivityDate="2011-03-22T08:51:11.383" CommentCount="1" />
  <row Id="8619" PostTypeId="2" ParentId="8617" CreationDate="2011-03-22T09:24:20.850" Score="11" ViewCount="" Body="&lt;p&gt;I personally wouldn't call this &quot;data cleaning&quot;.  I think of data cleaning more in the sense of data editing - cleaning up inconsistencies in the data set (e.g. a record has reported age of 1000, or a person aged 4 is a single parent, etc.).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The presence of a real effect in your data does not make it &quot;messy&quot; (to the contrary, the presence of real effects would make it rich) - although it can make your mathematical task more involved.  I would suggest that the data be &quot;cleaned&quot; in this way if it is the only feasible way to get a prediction.  If there is a feasible way which doesn't throw away information, then use that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It sounds like you may benefit from some sort of cyclical analysis, given that you say this effect comes around periodically (kind of like a &quot;business cycle&quot;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From my point of view, if you are looking at forecasting something, then removing a genuine effect from that source can only make your predictions worse.  This is because you have effectively &quot;thrown away&quot; the very information that you wish to predict!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The other point is that it may be difficult to determine how much of a set of deaths were due to the epidemic, and how much was caused by the ordinary fluctuations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In statistical terminology, the epidemic sounds like that, from your point of view, it is a &quot;nuisance&quot; to what you actually want to analyse.  So you aren't particularly interested in it, but you need to somehow account for it in your analysis.  One &quot;quick and dirty&quot; way to do this in a regression setting is to include an indicator for the epidemic years/periods as a regressor variable.  This will give you an average estimate of the effect of epidemics (and implicitly assumes the affect is the same for each epidemic).  However, this approach only works for describing the effect, because in forecasting, your regression variable is unknown (you don't know which periods in the future will be epidemic ones).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another way to account for the epidemic is to use a mixture model with two components: one model for the epidemic part and one model for the &quot;ordinary&quot; part.  The model then proceeds in two steps: 1) classify an period as epidemic or normal, then 2) apply the model to which it was classified.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-03-22T09:24:20.850" CommentCount="2" />
  <row Id="8620" PostTypeId="2" ParentId="8608" CreationDate="2011-03-22T09:45:20.753" Score="4" ViewCount="" Body="&lt;p&gt;This is a straight forward analysis of Bayes Theorem.  Now Bayes Theorem reads:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(H|F,I)=\frac{P(H|I)P(F|H,I)}{P(F|I)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So in your example you have two &quot;hypothesis&quot; SUN and RAIN.  $F$ stands for the the forecast, and $I$ for the prior information (or assumptions).  The &quot;I&quot; has not been included explicitly in your question, but it is always there.  These create 2 equations,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(SUN|F,I)=\frac{P(SUN|I)P(F|SUN,I)}{P(F|I)}$$&#xA;$$P(RAIN|F,I)=\frac{P(RAIN|I)P(F|RAIN,I)}{P(F|I)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now you have been given all probabilities except for the $P(F|I)$.  You can explicitly calculate this, but a quicker way is to get rid of it by taking the odds of SUN to RAIN.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{P(SUN|F,I)}{P(RAIN|F,I)}=\frac{\frac{P(SUN|I)P(F|SUN,I)}{P(F|I)}}{\frac{P(RAIN|I)P(F|RAIN,I)}{P(F|I)}}=\frac{P(SUN|I)P(F|SUN,I)}{P(RAIN|I)P(F|RAIN,I)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And this odds ratio is given by $\frac{0.14}{0.27}$.  And then you can use the exhaustiveness of the two hypothesis - because $$P(SUN|I)+P(RAIN|I)=1\rightarrow P(SUN|F,I)+P(RAIN|F,I)=1$$&#xA;$$\rightarrow P(RAIN|F,I)=1-P(SUN|F,I)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;.  And you get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{P(SUN|F,I)}{1-P(SUN|F,I)}=\frac{0.14}{0.27}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And re-arranging for $P(SUN|F,I)$ gives:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(SUN|F,I)=\frac{\frac{0.14}{0.27}}{1+\frac{0.14}{0.27}}=\frac{0.14}{0.41}=0.34$$&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-03-22T09:45:20.753" CommentCount="0" />
  <row Id="8621" PostTypeId="2" ParentId="8611" CreationDate="2011-03-22T10:10:29.543" Score="2" ViewCount="" Body="&lt;p&gt;Rather than a boxplot, I'd use a strip chart, since both data sets are small and it will be easy to see each data point.&lt;/p&gt;&#xA;" OwnerUserId="686" LastActivityDate="2011-03-22T10:10:29.543" CommentCount="5" />
  <row Id="8622" PostTypeId="2" ParentId="8614" CreationDate="2011-03-22T11:00:05.527" Score="3" ViewCount="" Body="&lt;p&gt;Despite the fact that there were a lot of discussion on the relative topics (though no any answer provided), I would like to add some ideas from my own teaching and model-building experience. It would be also very useful for you to study/read any (good) textbook in econometrics or time-series analysis first (just to speak the same language). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Sidenote&lt;/strong&gt;: it would be also great to switch from master-of-all-trades matlab to either &lt;a href=&quot;http://www.r-project.org/&quot; rel=&quot;nofollow&quot;&gt;$R$&lt;/a&gt; or &lt;a href=&quot;http://gretl.sourceforge.net/&quot; rel=&quot;nofollow&quot;&gt;gretl&lt;/a&gt;, both open-source products that are just great to do time-series (regarding $R$ then not only) things.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;As it follows from your description of the problem, you have a &lt;em&gt;time-series process&lt;/em&gt;, denote it by $y(t)$, that could be very noisy (low signal/noise ratio, with high noisy part) for a researcher to see any good time-dependence by simply screening the data (though a plot of a typical sensor's output would be very appreciated to improve this answer). Note, that &lt;strong&gt;time&lt;/strong&gt; is already present in the definition of the object. From here on you have a number of options of what to do next:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;You may go &lt;em&gt;solo&lt;/em&gt; , first of all, trying to &lt;strong&gt;decompose&lt;/strong&gt; the time-series object into trend + cycle + seasonal part (for magnetic fields both cycle and seasonal parts are very likely to present) + noise. At this stage it could happen that you do have something there close to &lt;a href=&quot;http://en.wikipedia.org/wiki/White_noise&quot; rel=&quot;nofollow&quot;&gt;white noise&lt;/a&gt; sequence. Note also that your trend could be either deterministic (fully predictable) or stochastic. The form of the trend could be tested by a number of &lt;a href=&quot;http://en.wikipedia.org/wiki/Unit_root&quot; rel=&quot;nofollow&quot;&gt;unit-root&lt;/a&gt; tests. Plotting the data again could be useful to detect the deterministic behaviour.&lt;/li&gt;&#xA;&lt;li&gt;Alternatively you may be interested in &lt;a href=&quot;http://en.wikipedia.org/wiki/Box%E2%80%93Jenkins&quot; rel=&quot;nofollow&quot;&gt;Box-Jenkins&lt;/a&gt; approach and try to build best parsimonious &lt;strong&gt;(S)ARIMA model&lt;/strong&gt; for $y(t)$. As a side-step in this approach is the analysis of correlogram (acf and pacf plots), anyway it is a good thing to begin from plotting this one as a first thing you do with your time series object.&lt;/li&gt;&#xA;&lt;li&gt;You may also find useful to move from &lt;strong&gt;time&lt;/strong&gt; to &lt;strong&gt;frequency&lt;/strong&gt; domain: produce &lt;a href=&quot;http://en.wikipedia.org/wiki/Periodogram&quot; rel=&quot;nofollow&quot;&gt;periodogram&lt;/a&gt;, search for deterministic sinusoidal behaviour or trends (again if the latter is flat you are probably indeed dealing with white noise process)&lt;/li&gt;&#xA;&lt;li&gt;You may search for meaningful covariates like average temperature, solar activity, earthquakes, etc. In this case you could build the theory based &lt;strong&gt;regression&lt;/strong&gt; model as opposed to a-theoretical decomposition and (S)ARIMA models.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Concluding remark&lt;/strong&gt;: as you may see from the response time-dependence could be far more complicated than just a linear time trend. There is no short answer to your question (well the &lt;strong&gt;shortest&lt;/strong&gt; could be: produce a correlogram, apply &lt;a href=&quot;http://en.wikipedia.org/wiki/Ljung%E2%80%93Box_test&quot; rel=&quot;nofollow&quot;&gt;Ljung-Box&lt;/a&gt; test, if it is close to white noise characteristics, conclude that there is no time-dependence, else some kind of it IS present, but are you happy with that one?). Good luck.&lt;/p&gt;&#xA;" OwnerUserId="2645" LastActivityDate="2011-03-22T11:00:05.527" />
  <row Id="8623" PostTypeId="2" ParentId="8617" CreationDate="2011-03-22T11:18:37.447" Score="5" ViewCount="" Body="&lt;p&gt;The role of &quot;data cleansing&quot; is to identify when &quot;our laws (model) do not work&quot;. Adjusting for Outliers or abnormal data points serve to allow us to get &quot;robust estimates&quot; of the parameters in the current model that we are entertaining. These &quot;outliers&quot; if untreated permit an unwanted distortion in the model parameters as estimation is &quot;driven to explain these data points&quot; that are &quot;not behaving according to our hypothesized model&quot;. In other words there is a lot of payback in terms of explained Sum of Squares by focusing on the &quot;baddies&quot;. The empirically identified points that require cleansing should be carefully scrutinized in order to potentially develop/suggest cause factors which are not in the current model. The identified Level Shift in STATE1 for the data you presented in the question below is an example of &quot;knowledge waiting to be discovered&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://stats.stackexchange.com/questions/8358/how-to-assess-effect-of-intervention-in-one-state-versus-another-using-annual-cas&quot;&gt;How to assess effect of intervention in one state versus another using annual case fatality rate?&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To do science is to search for repeated  patterns.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To detect anomalies is to identify values that do not follow repeated patterns. &#xA;How else would you know that a point violated that model? In fact, the process of growing, &#xA;understanding, finding, and examining outliers must be iterative. This isn't a new thought.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sir Frances Bacon, writing in Novum Organum about 400 years ago said: “Errors of Nature, Sports and Monsters correct the understanding in regard to ordinary things, and reveal general forms. For whoever knows the ways of Nature will more easily notice her deviations; and, on the other hand, whoever knows herdeviations will more accurately describe her ways.” &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We change our rules by observing when the current rules fail.      &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If indeed the identified outliers are all pulses and have similar effects (size) then we suggest the following ( quoted from another poster )&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;One &quot;quick and dirty&quot; way to do this in a regression setting is to include an indicator for the epidemic years/periods as a regressor variable. This will give you an average estimate of the effect of epidemics (and implicitly assumes the affect is the same for each epidemic). However, this approach only works for describing the effect, because in forecasting, your regression variable is unknown (you don't know which periods in the future will be epidemic ones).&quot;    &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This if course requires that the individual anomalies(pulse years) have similar effects. If they differ then a portmanteau variable described above would be incorrect.&lt;/p&gt;&#xA;" OwnerUserId="3382" LastEditorUserId="3382" LastEditorDisplayName="" LastEditDate="2011-03-24T12:21:41.397" LastActivityDate="2011-03-24T12:21:41.397" CommentCount="6" />
  <row Id="8624" PostTypeId="2" ParentId="4604" CreationDate="2011-03-22T11:32:23.003" Score="1" ViewCount="" Body="&lt;p&gt;To me, it sounds more like you want a &lt;em&gt;conditional frequency&lt;/em&gt;, as a conditional probability has no &quot;error&quot; so to speak.  The only error from a probability is from either from a mathematical approximation, or a mathematical error in the calculation.  Once you make this conceptual distinction, I think finding the exact measure of error is easy.  @robin's answer is a very conservative number - and represents the worst case error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyways, I assume that your data is a $K\times K$ frequency table.  The first way (row) is the letter previous $p_j\;(j=1,..,K)$, and the second way is the current letter $c_i\;(i=1,..,K)$ (column).  Let the counts in each cell be denoted by $n_{ij}$.  So $n_{11}$ is the number of times the character $p_1$ preceded the character $c_1$.  Expressing your problem this way also allows a straight-forward generalisation to a more general model - just add extra ways into the frequency table to give your process more &quot;memory&quot; (although the notation becomes a bit cumbersome, for you have quantities such as $f_{i|(j_1,j_2,...,j_L)}$ for a Lth order auto-regressive process).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now we need to write down the model.  Because you have a markov model, this means that the frequencies are given by $0\leq f_{i|j}\leq 1$.  This means that the frequency is a function of $p_j$ (i.e. we are modeling each row of the frequency table separately).  They are constrained to sum to 1 over the rows $\sum_{i=1}^{K}f_{i|j}=1\;(j=1,\dots,K)$&#xA;The simplest model is to assume that each frequency is unconnected with any other.  In this case you have a N-dimensional multinomial distribution for each row&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p(n_{1j},n_{2j},\dots,n_{Kj}|f_{1|j},f_{2|j},\dots,f_{K|j},I)\propto&#xD;&#xA;f_{1|j}^{n_{1j}},f_{2|j}^{n_{2j}},\dots,f_{K|j}^{n_{Kj}}&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now you need a prior distribution for the $f_{i|j}$ values, which expresses what you know about them, prior to seeing the data.  If you know that each letter is possible prior to seeing the data, then the uniform prior is the appropriate one.  This is the one I will use.  This gives a &lt;a href=&quot;http://en.wikipedia.org/wiki/Dirichlet_distribution&quot; rel=&quot;nofollow&quot;&gt;dirichlet posterior distribution&lt;/a&gt; for the $f_{i|j}$ (same form as above, because of the uniform prior).  This has a mode (most likely value) where $f_{i|j}=n_{ij}$ as you would expect.  Now, this means that each of the marginal posterior distributions have a Beta distribution&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$(f_{i|j}|D,I)\sim Beta(n_{ij}+1,\sum_{m\neq i}n_{mj}+K-1)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With a simple form for its mean and variance:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$E(f_{i|j}|D,I)=\frac{n_{ij}+1}{\sum_{m}n_{mj}+K}$$&#xA;$$Var(f_{i|j}|D,I)=\frac{E(f_{i|j}|D)\left[1-E(f_{i|j}|D)\right]}{\sum_{m}n_{mj}+K+1}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, you can use the dirichlet distribution to get the correlation between two of the frequencies:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Corr(f_{i|j},f_{l|j}|D)=-\sqrt{\frac{E(f_{i|j}|D)}&#xD;&#xA;{1-E(f_{i|j}|D)}&#xD;&#xA;\frac{E(f_{l|j}|D)}&#xD;&#xA;{1-E(f_{l|j}|D)}&#xD;&#xA;}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think this approach gives you a far better idea of the accuracy in your data, which shows the value in making a distinction between probability and frequency.  probability can never be observed, only calculated or assigned.  and frequencies can never be calculated or assigned, it can only be observed and predicted.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-03-22T11:32:23.003" />
  <row Id="8625" PostTypeId="1" AcceptedAnswerId="8671" CreationDate="2011-03-22T12:43:24.383" Score="3" ViewCount="88" Body="&lt;p&gt;I was fiddling with PCA and LDA methods and I am stuck at a point, I have a feeling that it is so simple that I can't see it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Within-class ($S_W$) and between-class ($S_B$) scatter matrices are defined as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;S_W = \sum_{i=1}^C\sum_{t=1}^N(x_t^i - \mu_i)(x_t^i - \mu_i)^T&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;S_B = \sum_{i=1}^CN(\mu_i-\mu)(\mu_i-\mu)^T&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Total scatter matrix $S_T$ is given as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;S_T = \sum_{i=1}^C\sum_{t=1}^N(x_t^i - \mu)(x_t^i - \mu)^T = S_W + S_B&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where C is number of classes and N is number of samples $x$ are samples, $\mu_i$ is ith class mean, $\mu$ is overall mean.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While trying to derive $S_T$ I came up to a point where I had:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;(x-\mu_i)(\mu_i-\mu)^T + (\mu_i-\mu)(x-\mu_i)^T&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;as a term. This needs to be zero, but why?&lt;/p&gt;&#xA;" OwnerUserId="760" LastEditorUserId="760" LastEditorDisplayName="" LastEditDate="2011-03-23T08:46:12.890" LastActivityDate="2011-03-23T14:12:28.437" Title="Deriving total (within class + between class) scatter matrix" Tags="&lt;pca&gt;&lt;discriminant-analysis&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="8626" PostTypeId="2" ParentId="8581" CreationDate="2011-03-22T13:11:06.113" Score="2" ViewCount="" Body="&lt;p&gt;I understand your difficulty as I have a similar problem when I try to do something new in statistics (I'm also a grad student, but in a different field). I have found examining the R code quite useful to get an idea how something is calculated. For example, I have been recently learning how to use &lt;code&gt;kmeans&lt;/code&gt; clustering and have many basic questions, both conceptual and how it is implemented. Using an &lt;code&gt;R&lt;/code&gt; installation (I recommend &lt;code&gt;R Studio&lt;/code&gt;, &lt;a href=&quot;http://www.rstudio.org/&quot; rel=&quot;nofollow&quot;&gt;http://www.rstudio.org/&lt;/a&gt;, but any installation works), just type &lt;code&gt;kmeans&lt;/code&gt; in to the command line. Here is an example of &lt;em&gt;part&lt;/em&gt; of the output:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x &amp;lt;- as.matrix(x)&#xA;    m &amp;lt;- nrow(x)&#xA;    if (missing(centers)) &#xA;        stop(&quot;'centers' must be a number or a matrix&quot;)&#xA;    nmeth &amp;lt;- switch(match.arg(algorithm), `Hartigan-Wong` = 1, &#xA;        Lloyd = 2, Forgy = 2, MacQueen = 3)&#xA;    if (length(centers) == 1L) {&#xA;        if (centers == 1) &#xA;            nmeth &amp;lt;- 3&#xA;        k &amp;lt;- centers&#xA;        if (nstart == 1) &#xA;            centers &amp;lt;- x[sample.int(m, k), , drop = FALSE]&#xA;        if (nstart &amp;gt;= 2 || any(duplicated(centers))) {&#xA;            cn &amp;lt;- unique(x)&#xA;            mm &amp;lt;- nrow(cn)&#xA;            if (mm &amp;lt; k) &#xA;                stop(&quot;more cluster centers than distinct data points.&quot;)&#xA;            centers &amp;lt;- cn[sample.int(mm, k), , drop = FALSE]&#xA;        }&#xA;    } &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm not sure how practical it is to examine the source every time, but it really helps me get an idea what is going on, assuming you have some familiarity with the syntax. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A previous question I asked on &lt;a href=&quot;http://stackoverflow.com/questions/5297889/how-do-i-find-the-parameters-used-in-kmeans-to-create-clusters&quot;&gt;stackoverflow&lt;/a&gt; pointed me in this direction, but also helpfully told me that the comments  about the code are sometimes included &lt;a href=&quot;http://svn.r-project.org/R/trunk/src/library/stats/R/kmeans.R&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;More generally, the &lt;a href=&quot;http://www.jstatsoft.org/&quot; rel=&quot;nofollow&quot;&gt;Journal of Statistical Software&lt;/a&gt; illustrates this link between theory and implementation, but it is frequently about advanced topics (that I personally have difficulty understanding), but is useful as an example.&lt;/p&gt;&#xA;" OwnerUserId="2635" LastEditorUserId="2635" LastEditorDisplayName="" LastEditDate="2011-03-22T13:30:20.690" LastActivityDate="2011-03-22T13:30:20.690" />
  <row Id="8627" PostTypeId="2" ParentId="8604" CreationDate="2011-03-22T13:14:03.273" Score="2" ViewCount="" Body="&lt;p&gt;A likely obtuse reference, but if you by chance have access to it I would recommend reading this book chapter (&lt;a href=&quot;http://dx.doi.org/10.1007/978-0-387-77650-7_26&quot; rel=&quot;nofollow&quot;&gt;Apel and Sweeten, 2010&lt;/a&gt;). It is aimed at social scientists and so perhaps not as mathematically rigorous as you seem to want, but it should go into enough depth to be more than a satisfactory answer to your question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are a few different ways people treat propensity scores that can result in different conclusions from simply including covariates in a regression model. When one matches scores one does not necessarily have common support for all observations (i.e. one has some observations that appear to never have the chance to be in the treatment group, and some that are always in the treatment group). Also one can weight observations in various ways that can result in different conclusions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In addition to the answers here I would also suggest you check out the answers to the &lt;a href=&quot;http://stats.stackexchange.com/questions/3400/from-a-statistical-perspective-can-one-infer-causality-using-propensity-scores-w/3443#3443&quot;&gt;question&lt;/a&gt; chl cited. There is more substance behind propensity scores than simply a statistical trick to achieve covariate balance. It you read and understand the highly cited articles by Rosenbaum and Rubin it will be more clear why the approach is different than simply adding in covariates in a regression model. I think a more satisfactory answer to your question is not necessarily in the math behind propensity scores but in their logic.&lt;/p&gt;&#xA;" OwnerUserId="1036" LastActivityDate="2011-03-22T13:14:03.273" CommentCount="1" />
  <row Id="8628" PostTypeId="2" ParentId="8617" CreationDate="2011-03-22T13:25:28.850" Score="5" ViewCount="" Body="&lt;p&gt;To give you a general answer to your question, let me parapharse one of my old general managers: the opportunities of research are found in the outliers of the model you are fitting. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The situation is similar to the experiment performed my Robert Millikan in determining the charge of an electron. Decades after winning the Nobel prize for his experiment, his notes were examined and it was found that he threw out a large number of data points because they disagreed with the results he was looking for. Is that bad science?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you find a few outliers, then maybe they are due to &quot;statistical abberations&quot;. However,&#xA;if you find more than a few outliers, you need to explore your data more closely. If you cannot attribute a cause for the abberations, then you do not understand the process and a statistical model will not solve your problem. The purpose of a model is to summarize a process, the model will not magically summarize a process the experimenter does not understand.&lt;/p&gt;&#xA;" OwnerUserId="3805" LastActivityDate="2011-03-22T13:25:28.850" CommentCount="1" />
  <row Id="8629" PostTypeId="2" ParentId="8581" CreationDate="2011-03-22T13:58:45.933" Score="2" ViewCount="" Body="&lt;p&gt;I get the impression that you think that you can get insight into a statistical equation&#xA;by programming it into either R or C++; you can't. To understand a statistical equation,&#xA;find an &quot;undergraduate&quot; textbook with lots of homework problems at the end of each chapter that contains the equation, and then do the homework at the end of the chapter containing the equation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, to understand PCA you do need a good understanding of linear algebra and in particular singular value decomposition. While learning quantum computing through Michael Nielsen's book, it became apparent to me that I needed to review linear algebra. I came across Gilbert Strang's videos, they were extremely helpful in establishing a foundational understanding of concepts. However, the nuance of the material did not get through until&#xA;I found a linear algebra book containing alot of homework problems, and then I needed to do them.&lt;/p&gt;&#xA;" OwnerUserId="3805" LastActivityDate="2011-03-22T13:58:45.933" CommentCount="1" />
  <row Id="8630" PostTypeId="1" CreationDate="2011-03-22T14:00:23.313" Score="4" ViewCount="172" Body="&lt;p&gt;I have carried out a principal components analysis of six variables $A$, $B$, $C$, $D$, $E$ and $F$. If I understand correctly -- unrotated PC1 tells me what linear combination of these variables describes/explains the most variance in the data and PC2 tells me what linear combination of these variables describes the next most variance in the data and so on, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm just curious -- is there any way of doing this backwards? Lets say I choose some linear combination of these variables -- e.g. $A+2B+5C$, could I work out how much variance in the data this describes?&lt;/p&gt;&#xA;" OwnerUserId="3845" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-03-23T14:18:01.630" LastActivityDate="2011-03-24T12:23:14.710" Title="Principal component analysis -- backwards?" Tags="&lt;pca&gt;&lt;factor-analysis&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="1" />
  <row Id="8631" PostTypeId="1" CreationDate="2011-03-22T14:12:22.203" Score="4" ViewCount="44" Body="&lt;p&gt;I want to understand how to make calculations on the prevalence of a disease in a country population and the impact that the element of average life expectancy (of those suffering with the disease at time of diagnosis) has on this calculation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any 'best practice' papers on making epidemiology calculations? &lt;/p&gt;&#xA;" OwnerUserId="3844" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-03-22T14:52:42.580" LastActivityDate="2011-03-22T14:52:42.580" Title="How would life expectancy impact the calculation of disease prevalence?" Tags="&lt;homework&gt;&lt;books&gt;&lt;epidemiology&gt;" CommentCount="2" FavoriteCount="2" />
  <row Id="8632" PostTypeId="1" CreationDate="2011-03-22T14:28:02.757" Score="1" ViewCount="64" Body="&lt;p&gt;The &lt;a href=&quot;http://www.iso.org/sites/JCGM/VIM/JCGM_200e.html&quot; rel=&quot;nofollow&quot;&gt;ISO VIM&lt;/a&gt; defines them as:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;measurement method:&lt;/strong&gt;&#xA;  generic description of a logical organization of operations used in a&#xA;  measurement.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;&lt;strong&gt;measurement procedure:&lt;/strong&gt; detailed description of a measurement according to one or more measurement principles and to a given measurement method, based on a measurement model and including any calculation to obtain a measurement result.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;But I'm still not sure what their difference is. Can anyone explain this to me, maybe with some examples?&lt;/p&gt;&#xA;" OwnerUserId="3823" LastEditorUserId="2645" LastEditorDisplayName="" LastEditDate="2011-03-23T08:23:14.400" LastActivityDate="2011-03-23T08:55:35.553" Title="What's the difference between &quot;measurement method&quot; and &quot;measurement procedure&quot;?" Tags="&lt;teaching&gt;&lt;terminology&gt;&lt;measurement&gt;&lt;methodology&gt;" AnswerCount="2" />
  <row Id="8633" PostTypeId="1" CreationDate="2011-03-22T14:57:53.010" Score="2" ViewCount="103" Body="&lt;p&gt;If I find that my covariate (reaction time) alters over the length of my experiment (e.g. due to fatigue), can I somehow build that into my model?&lt;br&gt;&#xA;So what I am saying is that the effect of my covariate is not constant (between subjects and within subjects).&lt;/p&gt;&#xA;" OwnerUserId="3822" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-22T22:51:11.193" LastActivityDate="2011-05-27T00:50:33.740" Title="ANCOVA with multiple instances of the between-subject covariate" Tags="&lt;repeated-measures&gt;&lt;ancova&gt;" AnswerCount="2" />
  <row Id="8634" PostTypeId="1" AcceptedAnswerId="8648" CreationDate="2011-03-22T16:15:30.263" Score="4" ViewCount="129" Body="&lt;p&gt;Given two bivariate normal distributions $P \equiv \mathcal{N}(\mu_p, \Sigma_p)$ and $Q \equiv \mathcal{N}(\mu_q, \Sigma_q)$, I am trying to calculate the Jensen-Shannon divergence between them, defined (for the discrete case) as:&#xA;    $JSD(P\|Q) = \frac{1}{2} (KLD(P\|M)+ KLD(Q\|M))$ &#xA;where $KLD$ is the Kullback-Leibler divergence, and $M=\frac{1}{2}(P+Q)$&lt;br&gt;&#xA;I've found the way to calculate &lt;a href=&quot;http://en.wikipedia.org/wiki/Multivariate_normal_distribution#Kullback.E2.80.93Leibler_divergence&quot; rel=&quot;nofollow&quot;&gt;$KLD$&lt;/a&gt; in terms of the distributions' parameters, and thus $JSD$.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My doubts are:  &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;To calculate $M$, I just did $M \equiv \mathcal{N}(\frac{1}{2}(\mu_p + \mu_q), \frac{1}{2}(\Sigma_p + \Sigma_q))$. Is this right? &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I've read in [&lt;a href=&quot;http://research-repository.st-andrews.ac.uk/bitstream/10023/1591/1/Endres2003-IEEETransInfTheory49-NewMetric.pdf&quot; rel=&quot;nofollow&quot;&gt;1&lt;/a&gt;] that the $JSD$ is bounded, but that doesn't appear to be true when I calculate it as described above for normal distributions. Does it mean I am calculating it wrong, violating an assumption, or something else I don't understand? &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="3843" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-03-23T01:53:13.153" LastActivityDate="2011-03-26T14:30:10.350" Title="Jensen-Shannon divergence for bivariate normal distributions" Tags="&lt;normal-distribution&gt;&lt;distance-functions&gt;&lt;information-theory&gt;" AnswerCount="1" />
  <row Id="8635" PostTypeId="1" CreationDate="2011-03-22T16:56:05.970" Score="1" ViewCount="55" Body="&lt;p&gt;Hi All,  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was wondering if anyone would be able to offer advice on how best to analyse/ interpret my results for one of my studies. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've measured parental attachment over time in 4 case studies using 3 measures of attachment (investigations were 6 months apart). At each time I also measured 5 aspects of their peer behaviour. All of these variables are rated on scales (i.e not rated as secure/insecure, they're rated on a security scale).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to test whether:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;attachment is stable across that time period  &lt;/li&gt;&#xA;&lt;li&gt;there is an association between change in attachment and change in behaviour  &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Any help would be great, I've looked at the Wilcoxon signed-rank test but I'm not sure its really right for what I'm doing.&lt;/p&gt;&#xA;" OwnerUserId="3849" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-22T22:50:11.397" LastActivityDate="2011-03-22T22:50:11.397" Title="Testing stability and correlation in repeated measures experiment" Tags="&lt;nonparametric&gt;&lt;repeated-measures&gt;&lt;small-sample&gt;" CommentCount="1" />
  <row Id="8636" PostTypeId="1" CreationDate="2011-03-22T17:18:14.077" Score="1" ViewCount="189" Body="&lt;p&gt;Hello, &#xA;I have design and analysis problems. I have 2 groups (experimental vs. control), 4 time points and 6 outcome measures (not strongly correlated). I need to conduct a MANOVA first, then multiple mixed-design ANOVAs (aka split plot anova) for each of the 6 measures. I know this is possible in SPSS but for my MSc coursework I need to write a Matlab script. I'm a beginner with Matlab, got the Psychtoolbox and correct function but I don't understand the outputs and therefore interpretations (after working through the mathsworks example as well). The internet hasn't been forthcoming, can anyone recommend instructions to perform a MANOVA and mixed-design ANOVA in matlab with less jargon? website/ book/ person - anything!?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help would be greatly appreciated!&lt;/p&gt;&#xA;" OwnerUserId="3851" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-03-26T03:29:23.420" LastActivityDate="2011-03-26T03:29:23.420" Title="How to perform MANOVA with mix of between and within subjects variables in Matlab?" Tags="&lt;anova&gt;&lt;homework&gt;&lt;matlab&gt;&lt;manova&gt;" />
  <row Id="8637" PostTypeId="2" ParentId="2181" CreationDate="2011-03-22T17:21:14.290" Score="4" ViewCount="" Body="&lt;p&gt;Almost the same question was asked recently on the &lt;a href=&quot;http://www.lawrence.edu/fast/jordanj/isostat.html&quot; rel=&quot;nofollow&quot;&gt;ISOSTAT&lt;/a&gt; listserver (frequented by college professors):&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;If you had a strong undergraduate student who was interested in learning about various multivariate methods (e.g. PCA, MANOVA, discriminant analysis, ...) is there a good, accessible book you might recommend she/he purchase?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Here are the responses:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Perhaps &quot;&lt;em&gt;Applied Multivariate Data Analysis&lt;/em&gt;&quot;, 2nd edition, by &lt;strong&gt;Everitt, B. and Dunn, G&lt;/strong&gt;. (2001), published by Arnold.  [Roger Johnson]&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Rencher&lt;/strong&gt;'s &lt;em&gt;Methods of Multivariate Analysis&lt;/em&gt; is a great resource.  I think a strong undergraduate student could grasp the material. [Philip Yates].  I'm fond of Rencher's approach. He offers good intuition and examples.  But the matrix algebra can get pretty thick; I'm not sure &quot;accessible&quot; is an adjective I'd use. Nevertheless, I've taught undergrads successfully with his book. His second edition is a good improvement over the first. [Paul Velleman]&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;em&gt;Applied Multivariate Statistics&lt;/em&gt; by &lt;strong&gt;Johnson and Wichern&lt;/strong&gt;.  [Brad Hartlaub]&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I haven't done much with it, but I do like the idea of using modern techniques and modern data sets: &lt;em&gt;Modern Multivariate Statistical Techniques&lt;/em&gt; by &lt;strong&gt;Alan Julian Izenman&lt;/strong&gt;. (I own the book, it has the topics you are looking for, and the text seems accessible.)  [Johanna Hardin]&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-03-22T17:21:14.290" CommunityOwnedDate="2011-03-22T17:21:14.290" CommentCount="3" />
  <row Id="8638" PostTypeId="2" ParentId="8611" CreationDate="2011-03-22T17:21:44.560" Score="3" ViewCount="" Body="&lt;p&gt;I would try a q-q plot if you have enough data;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;%make fake data;&#xA;x1 = randn(1000,1) .^ 2;x2 = (1.3 * randn(2000,1)).^2;&#xA;%which quantiles?&#xA;alphas = linspace(0,1,100);alphas = alphas(2:end-1);&#xA;q1 = interp1(linspace(0,1,numel(x1)),sort(x1),alphas,'linear');&#xA;q2 = interp1(linspace(0,1,numel(x2)),sort(x2),alphas,'linear');&#xA;scatter(q1,q2)&#xA;set(gca(),'XScale','log');set(gca(),'YScale','log');&#xA;xyline = [min(min(q1),min(q2)),max(max(q1),max(q2))];&#xA;hold on;&#xA;plot(xyline,xyline,'r-');&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/QGNhR.png&quot; alt=&quot;with x=y line added&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Adding a linear fit with stock matlab tools is not as simple as a one-liner, regrettably...&lt;/p&gt;&#xA;" OwnerUserId="795" LastEditorUserId="795" LastEditorDisplayName="" LastEditDate="2011-03-22T17:53:05.827" LastActivityDate="2011-03-22T17:53:05.827" CommentCount="2" />
  <row Id="8639" PostTypeId="1" AcceptedAnswerId="8641" CreationDate="2011-03-22T17:23:12.153" Score="2" ViewCount="35" Body="&lt;p&gt;Greetings, &#xA;Is it possible to use evidence in a Winbug model? For example, a random variable in a model has been observed, and I'd like to update the other variables in the model, pretty much the same update perfomed in tools like Smile, or other inference software. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Gibbs sampling is supposed to use observed values in full conditional distribution when there observations/evidence are entered into the model, but I am not sure if Winbugs allows this. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regards&lt;/p&gt;&#xA;&#xA;&lt;p&gt;edit to clarify:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The winbug documentation says a stochastic node is a data node if it has been observed, but a stochastic node is described via a distribution as in some_var ~ dbin(theta,n) &#xA;if it has been observed, then I'd like to tell this to winbugs without losing the semantics of the stochastic node, expressing something like &quot;some_var  has this distribution, and it has been observed to have this particular value&quot;. &#xA;so how do I do that? by declaring some_var as I've done above and than setting a value to it as in some_var = 5 ? Would that express what I want to express?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this case, in each observation of a node in a bayesian network, I'd need to redefine the winbugs model, (quite likely) replacing initial values of the unobserved nodes with the outcomes of the previous simulation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In short, I'm trying to understand how to perform updates on a Bayesian network similar to message passing in exact inference, but using Gibbs sampling instead, via Winbugs.&lt;/p&gt;&#xA;" OwnerUserId="3280" LastEditorUserId="3280" LastEditorDisplayName="" LastEditDate="2011-03-23T10:13:02.737" LastActivityDate="2011-03-23T10:13:02.737" Title="Can I insert an observation (evidence) to a Winbugs model?" Tags="&lt;bugs&gt;&lt;inference&gt;&lt;updating&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="8640" PostTypeId="2" ParentId="8630" CreationDate="2011-03-22T17:46:39.170" Score="2" ViewCount="" Body="&lt;p&gt;This is the very question that &lt;em&gt;linear regression&lt;/em&gt; is designed for.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let $Y$ be your data, and $X$ be a linear combination of the covariates.  If you carry out a linear regression on $Y$ vs $X$, the &quot;proportion of variance&quot; which $X$ accounts for is simply the square of the Pearson correlation, $R^2.$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This can be explained in the following way.  Suppose your fitted values are $\hat{Y} = aX+ b$.  It is a regression result that $cov(Y,Y-\hat{Y})=0$, therefore&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$var(Y) = var(\hat{Y} + Y-\hat{Y}) = var(\hat{Y}) + var(Y-\hat{Y}).$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &quot;proportion of variance&quot; explained by $X$ is then&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{var(\hat{Y})}{var(Y)}.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: Thanks Cardinal&lt;/p&gt;&#xA;" OwnerUserId="3567" LastEditorUserId="3567" LastEditorDisplayName="" LastEditDate="2011-03-24T12:23:14.710" LastActivityDate="2011-03-24T12:23:14.710" CommentCount="1" />
  <row Id="8641" PostTypeId="2" ParentId="8639" CreationDate="2011-03-22T18:58:10.133" Score="2" ViewCount="" Body="&lt;p&gt;Of course it's possible to use evidence from observations in WinBUGS! Try working through any of the examples in the documentation that comes with the program to see how.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-03-22T18:58:10.133" />
  <row Id="8642" PostTypeId="1" AcceptedAnswerId="8643" CreationDate="2011-03-22T18:58:38.307" Score="4" ViewCount="211" Body="&lt;p&gt;I want to perform a single-tail test on a single sample of real numbers (N~100) against an expected value. The population is known to be not normally distributed. So from what I've read about stats, I can do my testing using (i) Wilcoxon test, or (ii) bootstrap my sample data and then do a regular t-test. Is that correct?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which method is preferred for minimizing type I error, and if possible, why please?&lt;/p&gt;&#xA;" OwnerUserId="3847" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-22T22:45:00.280" LastActivityDate="2011-04-27T01:51:25.523" Title="What's the difference between bootstrapping t-test and just using a nonparametric test?" Tags="&lt;hypothesis-testing&gt;&lt;nonparametric&gt;&lt;bootstrap&gt;&lt;wilcoxon&gt;&lt;resampling&gt;" AnswerCount="4" CommentCount="1" />
  <row Id="8643" PostTypeId="2" ParentId="8642" CreationDate="2011-03-22T19:52:01.620" Score="1" ViewCount="" Body="&lt;p&gt;You just described the difference.  No one can know in advance outcome differences because it greatly depends on the nature of your data.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you know the non-normal distribution you're working with?  If so, you could simulate some results and see what the typical error rates for the different tests were and how they differed.&lt;/p&gt;&#xA;" OwnerUserId="601" LastActivityDate="2011-03-22T19:52:01.620" />
  <row Id="8644" PostTypeId="2" ParentId="6772" CreationDate="2011-03-22T20:22:25.160" Score="4" ViewCount="" Body="&lt;p&gt;Following whuber's &lt;a href=&quot;http://en.wikipedia.org/wiki/Fieller%27s_theorem#Case_1&quot; rel=&quot;nofollow&quot;&gt;link to Wikipedia&lt;/a&gt; you have &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Assume that $a$ and $b$ are jointly&#xA;  normally distributed, and that $b$ is&#xA;  not too near zero (i.e. more&#xA;  specifically, that the standard error&#xA;  of $b$ is small compared to $b$)&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;$$\operatorname{Var} \left( \frac{a}{b} \right) = \left(&#xD;&#xA; \frac{a}{b} \right)^{2} \left( \frac{\operatorname{Var}(a)}{a^2} +&#xD;&#xA; \frac{\operatorname{Var}(b)}{b^2}\right).$$&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;though in fact you want $\operatorname{Var} \left( \frac{B}{A} \right)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your 95% CI is $\pm 0.002$ then your variances for $A$ and $B$ are $(0.002/1.96)^2 \approx 0.00000104$, so $\operatorname{Var} \left( \frac{B}{A} \right) \approx 0.00846$. Taking the square root and multiplying by 1.96 you get $$\frac{B}{A} \approx 1.5 \pm 0.18$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you must turn this into percentages (I think it confuses more than it enlightens) then it becomes&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;B's proportion is 50% higher than A's, plus or minus 18%, i.e. between 32% higher and 68% higher.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In R you could simulate this by something like&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; n &amp;lt;- 1000000&#xA;&amp;gt; A &amp;lt;- 0.02 + (0.002 / qnorm(0.975)) * rnorm(n)&#xA;&amp;gt; B &amp;lt;- 0.03 + (0.002 / qnorm(0.975)) * rnorm(n)&#xA;&amp;gt; C &amp;lt;- B / A&#xA;&amp;gt; quantile(C,  probs = c(0.025, 0.5, 0.975))&#xA;    2.5%      50%    97.5% &#xA;1.333514 1.499955 1.697418 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which is reasonably close.&lt;/p&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-03-22T20:22:25.160" />
  <row Id="8645" PostTypeId="2" ParentId="8614" CreationDate="2011-03-22T21:48:37.590" Score="0" ViewCount="" Body="&lt;p&gt;Following DC's excellent summary of available approaches let me add:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The question &quot;if a given noise from a compass sensor is time-correlated &quot; raises suggestions as how to analyse it in order to make a conclusion. In the absence of user-specified possible support/explanatory series one is left with approaches that entertain Deterministic Structure , ARIMA Structure and hybrids. When faced with an unknown underlying frequency we often scan for hidden stochastic seasonalities by a search process. In the absence of user-specified suggestions like these readings are taken every 24 hours. One must reach out and via trial and error to deduce the strong (if any) hidden periodicity. Outliers , Pulses, Level Shifts m Local Time Trends can also play havoc with system identification. We use very aggressive model identification strategies ( no neural nets here ) to extract the signal be it stocahastic or deterministic or both.&lt;/p&gt;&#xA;" OwnerUserId="3382" LastEditorUserId="3382" LastEditorDisplayName="" LastEditDate="2011-03-31T22:39:28.480" LastActivityDate="2011-03-31T22:39:28.480" CommentCount="6" />
  <row Id="8646" PostTypeId="2" ParentId="8568" CreationDate="2011-03-22T21:54:37.287" Score="0" ViewCount="" Body="&lt;p&gt;Post your trial (preconcevied model ) ; post your actual data and see what the list can suggest as possible modifications to either the guessed parameters or the guessed structure.&lt;/p&gt;&#xA;" OwnerUserId="3382" LastActivityDate="2011-03-22T21:54:37.287" CommentCount="1" />
  <row Id="8647" PostTypeId="2" ParentId="8608" CreationDate="2011-03-22T22:45:45.143" Score="5" ViewCount="" Body="&lt;p&gt;Research has shown that people have difficulty reasoning in terms of probabilities but can do so accurately when presented with the same questions in terms of frequencies.  So, let's consider a closely related setting where the probabilities are expressed as numbers of occurrences:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;In 100 similar situations, it rained 30 times and was sunny 70 times.  This matches P(W=Sun) = 0.7 = 70/100 and P(W=Rain) = 0.3 = 30/100.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;From P(F=good|Sun) = 0.8 we compute that 0.8 * 70 = 56 times F will be &quot;good&quot; when W is &quot;sun&quot;.  Likewise, from P(F=bad|Sun) = 0.2 we compute that 0.2 * 70 = 14 times F will be &quot;bad&quot; when W is &quot;sun&quot;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;From P(F=good|Rain) = 0.1 we compute that 0.1 * 30 = 3 times F will be &quot;good&quot; when W is &quot;rain&quot; and from P(F=bad|Rain) = 0.9 we compute that 0.9 * 30 = 27 times F will be &quot;bad&quot; when W is &quot;rain&quot;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If F is &quot;bad&quot;, what can we say?  Well, this situation happened 14 + 27 = 41 times.  In 14/41 = 0.34 of those times W was &quot;sun&quot;; therefore, we expect P(W=Sun|F=Bad) = 0.34.  In the other 27/41 = 0.66 of those times W was &quot;rain&quot;; therefore, P(W=Rain|F=Bad) = 0.66.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, &quot;normalization&quot; means &lt;em&gt;we focus only on those situations where the prior condition holds&lt;/em&gt; (F=Bad in the example) &lt;em&gt;and rescale the probabilities to sum to unity&lt;/em&gt; (as they must).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is an archetypal example of &lt;a href=&quot;http://en.wikipedia.org/wiki/Bayes%27_theorem&quot; rel=&quot;nofollow&quot;&gt;Bayes' Theorem&lt;/a&gt; which in mathematical terms says that to compute conditional probabilities, &lt;strong&gt;focus&lt;/strong&gt; and &lt;strong&gt;rescale&lt;/strong&gt;.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-03-22T22:45:45.143" />
  <row Id="8648" PostTypeId="2" ParentId="8634" CreationDate="2011-03-22T23:34:52.187" Score="3" ViewCount="" Body="&lt;p&gt;The midpoint measure $\newcommand{\bx}{\mathbf{x}} \newcommand{\KL}{\mathrm{KL}}M$ is a &lt;em&gt;mixture distribution&lt;/em&gt; of the two multivariate normals, so it does not have the form that you give in the original post. Let $\varphi_p(\bx)$ be the probability density function of a $\mathcal{N}(\mu_p, \Sigma_p)$ random vector and $\varphi_q(\bx)$ be the pdf of $\mathcal{N}(\mu_q, \Sigma_q)$. Then the pdf of the midpoint measure is&#xA;$$&#xD;&#xA;\varphi_m(\bx) = \frac{1}{2} \varphi_p(\bx) + \frac{1}{2} \varphi_q(\bx) \&amp;gt; .&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Jensen-Shannon divergence is&#xA;$$&#xD;&#xA;\mathrm{JSD} = \frac{1}{2} (\KL(P\,\|M)+ \KL(Q\|M)) = h(M) - \frac{1}{2} (h(P) + h(Q)) \&amp;gt;,&#xD;&#xA;$$&#xA;where $h(P)$ denotes the (differential) entropy corresponding to the measure $P$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, your calculation reduces to calculating differential entropies. For the multivariate normal $\mathcal{N}(\mu, \Sigma)$, the answer is well-known to be&#xA;$$&#xD;&#xA;\frac{1}{2} \log_2\big((2\pi e)^n |\Sigma|\big)&#xD;&#xA;$$ &#xA;and the proof can be found in any number of sources, e.g., Cover and Thomas (1991), pp. 230-231. It is worth pointing out that the entropy of a multivariate normal is invariant with respect to the mean, as the expression above shows. However, this almost assuredly does not carry over to the case of a mixture of normals. (Think about picking one broad normal centered at zero and another concentrated normal where the latter is pushed out far away from the origin.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the midpoint measure, things appear to be more complicated. That I know of, there is no closed-form expression for the differential entropy $h(M)$. &lt;a href=&quot;http://www.google.com/search?q=entropy+of+Gaussian+mixtures&quot; rel=&quot;nofollow&quot;&gt;Searching on Google&lt;/a&gt; yields a couple potential hits, but the top ones don't appear to give closed forms in the general case. You may be stuck with approximating this quantity in some way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note also that the paper you reference does not restrict the treatment to only discrete distributions. They treat a case general enough that your problem falls within their framework. See the middle of column two on page 1859. Here is where it is also shown that the divergence is bounded. This holds for the case of two general measures and is not restricted to the case of two discrete distributions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Jensen-Shannon Divergence has come up a couple of times recently in other questions on this site. See &lt;a href=&quot;http://stats.stackexchange.com/questions/7630/clustering-should-i-use-the-jensen-shannon-divergence-or-its-square&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;http://stats.stackexchange.com/questions/6907/an-adaptation-of-the-kullback-leibler-distance&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Addendum&lt;/strong&gt;: Note that a mixture of normals is not the same as a linear combination of normals. The simplest way to see this is to consider the one-dimensional case. Let $X_1 \sim \mathcal{N}(-\mu, 1)$ and $X_2 \sim \mathcal{N}(\mu, 1)$ and let them be &lt;em&gt;independent&lt;/em&gt; of one another. Then a mixture of the two normals using weights $(\alpha, 1-\alpha)$ for $\alpha \in (0,1)$ has the distribution&#xA;$$&#xD;&#xA;\varphi_m(x) = \alpha \cdot \frac{1}{\sqrt{2\pi}} e^{-\frac{(x+\mu)^2}{2}} + (1-\alpha) \cdot&#xD;&#xA;\frac{1}{\sqrt{2\pi}} e^{-\frac{(x-\mu)^2}{2}} \&amp;gt; .&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The distribution of a &lt;em&gt;linear combination&lt;/em&gt; of $X_1$ and $X_2$ using the same weights as before is, via the &lt;a href=&quot;http://en.wikipedia.org/wiki/Stable_distribution&quot; rel=&quot;nofollow&quot;&gt;stable&lt;/a&gt; property of the normal distribution is&#xA;$$&#xD;&#xA;\varphi_{\ell}(x) = \frac{1}{\sqrt{2\pi\sigma^2}} e^{-\frac{(x-(1-2\alpha)\mu)^2}{2\sigma^2}} \&amp;gt;,&#xD;&#xA;$$&#xA;where $\sigma^2 = \alpha^2 + (1-\alpha)^2$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These two distributions are very different, though they have the same mean. This is not an accident and follows from linearity of expectation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To understand the mixture distribution, imagine that you had to go to a statistical consultant so that she could produce values from this distribution for you. She holds one realization of $X_1$ in one palm and one realization of $X_2$ in the other palm (though you don't know which of the two palms each is in). Now, her assistant flips a biased coin with probability $\alpha$ out of sight of you and then comes and whispers the result into the statistician's ear. She opens one of her palms and shows you the realization, but doesn't tell you the outcome of the coin flip. This process produces the mixture distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, the linear combination can be understood in the same context. The statistical consultant merely takes both realizations, multiplies the first by $\alpha$ and the second by $(1-\alpha)$, adds the result up and shows it to you.&lt;/p&gt;&#xA;" OwnerUserId="2970" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-03-26T14:30:10.350" LastActivityDate="2011-03-26T14:30:10.350" CommentCount="3" />
  <row Id="8649" PostTypeId="1" CreationDate="2011-03-23T00:50:44.210" Score="3" ViewCount="124" Body="&lt;p&gt;I'm using mean values to show of my data along with the standard errors, from a tutorial I found. But I'm having a problem discussing the results. My plot is as shown below: some of the standard errors (shown as a error bar) vary much and some of them are very close to zero.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/3sUze.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="3270" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-23T03:05:29.017" LastActivityDate="2011-03-23T10:54:10.860" Title="What is standard error used for?" Tags="&lt;data-visualization&gt;&lt;standard-error&gt;" AnswerCount="3" FavoriteCount="1" />
  <row Id="8650" PostTypeId="2" ParentId="8649" CreationDate="2011-03-23T01:06:09.497" Score="4" ViewCount="" Body="&lt;p&gt;Error bars in general are to convince the plot reader that the differences she/he sees on the plot are statistically significant. In an approximation, you may imagine a small gaussian which $\pm1\sigma$ range is shown as this error bar -- &quot;visual integration&quot; of a product of two such gaussians is more-less a chance that the two values are really equal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this particular case, one can see that both the difference between red and violet bar as well as gray and green ones are not too significant.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2011-03-23T01:06:09.497" CommentCount="1" />
  <row Id="8651" PostTypeId="1" CreationDate="2011-03-23T02:48:08.760" Score="0" ViewCount="60" Body="&lt;p&gt;Is there a way to find the spearman correlation between two weibull distributions? I need it as a parameter in a copula function for the joint weibull distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I learnt that using a pearson correlation which i can easily obtain from the variances and cross variances of the given spectra is not reliable with copulas especially when the data is not normally distributed.&lt;/p&gt;&#xA;" OwnerUserId="3854" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-03-23T14:15:36.093" LastActivityDate="2011-03-23T14:15:36.093" Title="Spearman Correlation of two Weibull distributions" Tags="&lt;correlation&gt;&lt;spearman&gt;" CommentCount="2" />
  <row Id="8652" PostTypeId="2" ParentId="8649" CreationDate="2011-03-23T04:01:23.240" Score="2" ViewCount="" Body="&lt;p&gt;As mbq says, error bars are a way of letting your readers to get a feel if the differences between two groups are significant - i.e. if the variation &lt;em&gt;within&lt;/em&gt; each of your groups is small enough to believe that the difference you've found for the mean &lt;em&gt;between&lt;/em&gt; your groups. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;All else being equal, larger error bars mean more within-group difference, but it looks like the y-axis of your plot is log-transformed, so the lower groups aren't quite on the same scale as the higher ones.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should be aware, &lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pubmed/16392994&quot; rel=&quot;nofollow&quot;&gt;many of your readers won't understand what error bars represent, even if you explicitly explain it!&lt;/a&gt; Often you can achieve the same goal with a &lt;a href=&quot;http://had.co.nz/ggplot2/geom_jitter.html&quot; rel=&quot;nofollow&quot;&gt;jittered dot-plot&lt;/a&gt; or a boxplot (or both together) to achieve the same effect.&lt;/p&gt;&#xA;" OwnerUserId="3732" LastActivityDate="2011-03-23T04:01:23.240" CommentCount="1" />
  <row Id="8653" PostTypeId="1" CreationDate="2011-03-23T05:45:42.340" Score="4" ViewCount="31" Body="&lt;p&gt;Hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am interested in estimating how many subjects should be included in a brain imaging study. Although the design is a fairly straight forward cross-sectional comparison, there are a number of tweakable image processing steps between the raw image and the processed image in which we carry out pixel-wise comparisons.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've used the power.t.test function in R to estimate the number of subjects required to reach statistical significance, using hypothetical delta values and sd estimates from a control population. I've run these analyses with varying image processing settings, resulting in a large number of &quot;number of subjects per group&quot; vs &quot;some image processing parameter&quot; plots. So many that it looks kind of messy to provide hundreds of plots.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I would like is an empirical equation that allows users to estimate the number of subjects per group as a function of effect size, alpha, and a few other image processing specific parameters. That way I only need to report the coefficients for the equation rather than supplying hundreds of graphs. Is this kind of thing possible?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far if I model N ~ k1/effect.size^2 that looks OK for some values of alpha (for example) but doesn't work so good for others. A log-log plot doesn't model the relationship well either. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can't seem to find an explicit formula that relates the number of subjects per group with the other factors in a power analysis. Does this exist?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="3855" LastActivityDate="2011-05-26T21:50:31.813" Title="How to model the relationship between number of subjects per group, derived using standard power analysis methods, and study specific parameters" Tags="&lt;modeling&gt;&lt;model-selection&gt;&lt;power-analysis&gt;&lt;cross-section&gt;" AnswerCount="1" FavoriteCount="2" />
  <row Id="8655" PostTypeId="2" ParentId="8604" CreationDate="2011-03-23T06:18:03.407" Score="2" ViewCount="" Body="&lt;p&gt;One big difference is that regression &quot;controls for&quot; those characteristics in a linear fashion. Matching by propensity scores eliminates the linearity assumption, but, as some observations may not be matched, you may not be able to say anything about certain groups.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if you are studying a worker training program, you may have all the enrollees be men, but the control, non-participant population be composed of men and women. Using regression, you could regress, income, say, on a participation indicator variable and a male indicator. You would use all your data and could estimate the income of a female had she participated in the program.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you were doing matching, you could only match men to men. As a result, you wouldn't be using any women in your analysis and your results wouldn't pertain to them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regression can extrapolate using the linearity assumption, but matching cannot. All the other assumptions are essentially the same between regression and matching. The benefit of matching over regression is that it is non-parametric (except you do have to assume that you have the right propensity score, if that is how you are doing your matching).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For more discussion, see &lt;a href=&quot;http://cgibbons.berkeley.edu/Courses/ps236.html&quot; rel=&quot;nofollow&quot;&gt;my page here&lt;/a&gt; for a course that was heavily focused on matching methods. See especially &lt;a href=&quot;http://cgibbons.berkeley.edu/Courses/PS236_F08/StrategyComparisons.pdf&quot; rel=&quot;nofollow&quot;&gt;Causal Effects Estimation Strategy Assumptions&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, be sure to check out the &lt;a href=&quot;http://homes.stat.unipd.it/erich/teaching/eval/papers/rosenbaum_rubin83.pdf&quot; rel=&quot;nofollow&quot;&gt;Rosenbaum and Rubin (1983)&lt;/a&gt; article that outlines propensity score matching.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lastly, matching has come a long way since 1983. Check out &lt;a href=&quot;http://sekhon.berkeley.edu&quot; rel=&quot;nofollow&quot;&gt;Jas Sekhon's webpage&lt;/a&gt; to learn about his genetic matching algorithm.&lt;/p&gt;&#xA;" OwnerUserId="401" LastActivityDate="2011-03-23T06:18:03.407" />
  <row Id="8656" PostTypeId="2" ParentId="8632" CreationDate="2011-03-23T07:35:19.457" Score="1" ViewCount="" Body="&lt;p&gt;Actually if drop &lt;strong&gt;measurement&lt;/strong&gt; and leave only or narrowed question: What's the difference between &lt;strong&gt;method&lt;/strong&gt; and &lt;strong&gt;procedure&lt;/strong&gt;? A nice answer could be found &lt;a href=&quot;http://wiki.answers.com/Q/What_is_the_difference_between_a_method_and_a_procedure&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;. From this answer we may conclude that method is a wider concept (more &lt;strong&gt;abstract&lt;/strong&gt;) as opposed to a certain (more &lt;strong&gt;concrete&lt;/strong&gt;) set of actions that the procedure is. You may think about the procedure as being a subset of method. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Measurement&lt;/em&gt; in this context is just a field where some particular examples live. For me it is easier to think of many examples in estimation methods, so any help form the community is welcome.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, in econometrics you often need to obtain data through the set of Monte Carlo simulations (that is a method). When I solve a particular simulation problem I first describe the data generating process, set the parameters of the experiment and run the simulations, that are than generalized into some averaged results. All this sequence is the procedure.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Most of the data collection methods are design based. So you have to work wit one of the &lt;a href=&quot;http://en.wikipedia.org/wiki/Sampling_%28statistics%29#Sampling_methods&quot; rel=&quot;nofollow&quot;&gt;sampling&lt;/a&gt; methods. Take the first, simple random sampling (method) and apply it to a specific population sampling problem, the way you apply the method, a certain list of actions in this case will be a procedure of collecting your data.&lt;/p&gt;&#xA;" OwnerUserId="2645" LastEditorUserId="2645" LastEditorDisplayName="" LastEditDate="2011-03-23T08:55:35.553" LastActivityDate="2011-03-23T08:55:35.553" />
  <row Id="8657" PostTypeId="1" CreationDate="2011-03-23T08:37:03.867" Score="0" ViewCount="54" Body="&lt;p&gt;hi there&lt;/p&gt;&#xA;&#xA;&lt;p&gt;which is the % of population with vote rights(above 18yo for example). Can u point me to some papers which talks about voting and ages?&lt;/p&gt;&#xA;" OwnerUserId="3856" LastActivityDate="2011-03-23T18:45:20.533" ClosedDate="2011-03-23T17:06:37.100" Title="vote population" Tags="&lt;population&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="8658" PostTypeId="2" ParentId="8632" CreationDate="2011-03-23T08:41:36.297" Score="1" ViewCount="" Body="&lt;p&gt;If you are familiar with programming, you could perhaps think of it as a method (not to be confused with a logical part of some code) being a brief description of an algorithm in pseudo-code, whereas a procedure is a specific implementation with exact syntax.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Admitted that this is not a perfect metaphor but I think it portrays the difference. &lt;/p&gt;&#xA;" OwnerUserId="3014" LastActivityDate="2011-03-23T08:41:36.297" />
  <row Id="8659" PostTypeId="2" ParentId="8642" CreationDate="2011-03-23T09:18:53.263" Score="2" ViewCount="" Body="&lt;p&gt;This answer may be helpful, and/or it may be annoying.  Your welcome and my apologies at the same time :)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One thing to remember when using a normal distribution, is that it has a set of sufficient statistics, namely the mean and variance.  What this indicates is that only the mean and variance matter in the inference.  Any property of your sample besides the mean and variance will be thrown away when you use a normal distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The statement that the &quot;population is not normally distributed&quot; is a bit of a misnomer - the population is not &quot;distributed&quot; at all - there is one and only one population (imaginary data sets and alternate worlds aside).  It sounds like what you are actually saying is that &lt;em&gt;your knowledge of the population consists of something other than the average and variance&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So presumably, the only thing to do is to state what this extra/different knowledge is.  Perhaps you know the skewness (or you know the skewness is important/relevant for the analysis, and not &quot;noise&quot;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would suggest that you simply calculate the probability that your hypothesis is true, conditional on the information you have.  This would include the data, and whatever &quot;structure&quot; you claim to know about the population that makes it non-normal (something other than the mean and variance of the population).  So call your one sided test $T$, then you simply calculate:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(T|D,I)=\frac{P(T|I)P(D|T,I)}{P(D|I)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(T|I)$ is the prior probability for the test being &quot;true&quot; or &quot;successful&quot; (what did know about the test prior to seeing the data?).  $P(D|T,I)$ is the &quot;model&quot; or &quot;likelihood&quot; and is similar to a p-value (How likely is the data you observed, given the test is true?).  And $P(D|I)$ is often  called the &quot;evidence&quot; (how well do any of the hypothesis predict the observed data?) - this quantity does not need to be explicitly assigned, as it can be derived from the requirement that the probability must add to 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The good thing about this method, is that probability theory will &quot;construct the optimal test for you&quot;.  You just need describe your prior information, and then simply do the mathematics.  Now you may find that a bootstrap may be necessary in order to evaluate some mathematical formula - you may find that you should do the wilcoxon test - or probability theory will construct a test which is better than either of them (in terms of that type 1 and type 2 error you speak of).&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-03-23T09:18:53.263" CommentCount="1" />
  <row Id="8660" PostTypeId="2" ParentId="8502" CreationDate="2011-03-23T09:47:40.163" Score="3" ViewCount="" Body="&lt;p&gt;This is &quot;taylor made&quot; almost for a Bayesian regression.  First of all, there is nothing &quot;fundamentally wrong&quot; with what you suggest.  You result may not be optimal by some mathematical standard, but it will almost certainly be optimal time wise.  Most other methods will involve much more time than a straight multiplication and division.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would use a normal likelihood $(y_i|\beta,\sigma,x_i,I)\sim N(x_i^T\beta,\sigma^2)$ and the &quot;jeffreys prior&quot; $p(\beta,\sigma|x_i,I) \propto \frac{1}{\sigma}$.  This gives a posterior for $\beta$ as a multivariate t-distribution, with scale matrix $s^2(X^TX)^{-1}$ and mean vector $\beta_{ols}$ with the standard $n-p$ degrees of freedom.  Now you simply use this posterior based on the &quot;A&quot; data set as the prior for the &quot;B&quot; data set.  Now because you have a &quot;t&quot; prior and a &quot;normal&quot; likelihood, the posterior for beta will favour the normal likelihood, because the t has fatter tails - hence less &quot;pulling power&quot;.  This regression will balance the A and B regression between how accurately A was estimated, and how well the B estimate fits the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An &quot;add-hoc&quot; way that you could add more weight to &quot;B&quot; is by setting the degrees of freedom to 1 in the &quot;A&quot; posterior.  But then you may as well save some time and do the multiply the B estimate by two.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't think there is a simple analytic expression for this posterior, so will likely need to simulate.  But you only require the estimate from the &quot;A&quot; data set, and the covariance matrix from the &quot;A&quot; data set, and the number of observations in the &quot;A&quot; data set.  Once you have these quantities, you don't require the original data set.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-03-23T09:47:40.163" CommentCount="5" />
  <row Id="8661" PostTypeId="1" CreationDate="2011-03-23T09:59:21.777" Score="2" ViewCount="312" Body="&lt;p&gt;I'm trying to undertake a logistic regression analysis in &lt;code&gt;R&lt;/code&gt;. I have attended courses covering this material using STATA. I am finding it very difficult to replicate functionality in &lt;code&gt;R&lt;/code&gt;. Is it mature in this area? There seems to be little documentation or guidance available. Producing odds ratio output seems to require installing &lt;code&gt;epicalc&lt;/code&gt; and/or &lt;code&gt;epitools&lt;/code&gt; and/or others, none of which I can get to work, are outdated or lack documentation. I've used &lt;code&gt;glm&lt;/code&gt; to do the logistic regression. Any suggestions would be welcome.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd better make this a real question. How do I run a logistic regression and produce odds rations in &lt;code&gt;R&lt;/code&gt;?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's what I've done for a univariate analysis:  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;x = glm(Outcome ~ Age, family=binomial(link=&quot;logit&quot;))&lt;/code&gt;  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;And for multivariate:  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;y = glm(Outcome ~ Age + B + C, family=binomial(link=&quot;logit&quot;))&lt;/code&gt;  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've then looked at &lt;code&gt;x&lt;/code&gt;, &lt;code&gt;y&lt;/code&gt;, &lt;code&gt;summary(x)&lt;/code&gt; and &lt;code&gt;summary(y)&lt;/code&gt;.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is &lt;code&gt;x$coefficients&lt;/code&gt; of any value?&lt;/p&gt;&#xA;" OwnerUserId="2824" LastEditorUserId="2824" LastEditorDisplayName="" LastEditDate="2011-03-23T10:18:41.703" LastActivityDate="2011-03-23T14:39:55.110" Title="Logistic Regression in R (Odds Ratio)" Tags="&lt;r&gt;&lt;logistic-regression&gt;&lt;odds-ratio&gt;" AnswerCount="3" FavoriteCount="1" />
  <row Id="8662" PostTypeId="1" AcceptedAnswerId="8674" CreationDate="2011-03-23T10:20:57.830" Score="2" ViewCount="183" Body="&lt;p&gt;I have the sample population of a certain signal's registered amplitude maxima. Population is about 15 million samples. I produced a histogram of the population, but cannot guess the distribution with such a histogram.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT1: File with raw sample values is here: &lt;a href=&quot;http://hotfile.com/dl/111583549/5c73384/TDETQ_Z2M_ALL_TIM50_N.txt.zip.html&quot; rel=&quot;nofollow&quot;&gt;raw data&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone help estimate the distribution with the following histogram:&#xA;&lt;img src=&quot;http://i.imgur.com/h1n2y.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="2820" LastEditorUserId="2820" LastEditorDisplayName="" LastEditDate="2011-03-24T04:54:13.743" LastActivityDate="2011-03-24T04:54:13.743" Title="Need help identifying a distribution by its histogram" Tags="&lt;distributions&gt;&lt;histogram&gt;" AnswerCount="2" CommentCount="9" FavoriteCount="1" />
  <row Id="8663" PostTypeId="1" CreationDate="2011-03-23T10:21:41.670" Score="1" ViewCount="136" Body="&lt;p&gt;I used &lt;code&gt;summary.formula&lt;/code&gt; from &lt;code&gt;Hmisc&lt;/code&gt; with continuous &lt;code&gt;Age&lt;/code&gt; and binary outcome &lt;code&gt;O&lt;/code&gt; with &lt;code&gt;test=TRUE&lt;/code&gt;. This returned a p-value for &lt;code&gt;Age&lt;/code&gt; predicting &lt;code&gt;O&lt;/code&gt; (if I understand this correctly).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I then ran a &lt;code&gt;glm&lt;/code&gt; using &lt;code&gt;Age&lt;/code&gt; and &lt;code&gt;O&lt;/code&gt; (univariate logistic regression), which returned a different p-value. I thought that the p-value should be the same?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt;  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;library(Hmisc)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;summary.formula(Outcome ~ cut2(Age, seq(15,75,10)), method=&quot;reverse&quot;, test=TRUE)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;p-value = 0.8&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;x = glm(Outcome ~ Age, family=binomial(link=&quot;logit&quot;))&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;p-value = 0.4&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(I'm getting quite confused: the example above which gives the 0.8 p-value is binning the ages, but I was told that the p-value it returns is for all the ages. However, if I run the summary without the binning, the p-value is 0.5.)&lt;/p&gt;&#xA;" OwnerUserId="2824" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-24T16:12:05.140" LastActivityDate="2011-03-24T16:12:05.140" Title="Chi-squared versus logistic regression" Tags="&lt;logistic-regression&gt;&lt;chi-squared&gt;&lt;p-value&gt;" AnswerCount="2" CommentCount="4" FavoriteCount="2" />
  <row Id="8664" PostTypeId="1" CreationDate="2011-03-23T10:35:06.573" Score="2" ViewCount="152" Body="&lt;p&gt;I want to carry out a power analysis on a one group repeated measures experiment using G*power.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a group of subjects who tested a set of products. Each subject test one time the products. Hence, I have one observation per cell.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To test the product effect, I used a two anova model with subjects as random effect with proc glm in sas. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to calculate the sample size for a given study having the same design. Given an estimation of effect size, alpha, Beta and an estimation of  &quot;a correlation among the repeated measures&quot; could I use the ratio variance of subjets/ (variance of subjects + variance of error) derived from the two way anova model as an estimation of this correlation?&lt;/p&gt;&#xA;" OwnerUserId="3858" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-03-23T13:49:22.730" LastActivityDate="2011-05-26T20:50:30.690" Title="How to estimate correlation among repeated measures?" Tags="&lt;correlation&gt;&lt;repeated-measures&gt;" AnswerCount="1" />
  <row Id="8665" PostTypeId="2" ParentId="8649" CreationDate="2011-03-23T10:54:10.860" Score="0" ViewCount="" Body="&lt;p&gt;Plenty of researchers have trouble interpreting these graphs. See &lt;a href=&quot;http://scienceblogs.com/cognitivedaily/2008/07/most_researchers_dont_understa_1.php&quot; rel=&quot;nofollow&quot;&gt;http://scienceblogs.com/cognitivedaily/2008/07/most_researchers_dont_understa_1.php&lt;/a&gt; for a more detailed elaboration.&lt;/p&gt;&#xA;" OwnerUserId="1048" LastActivityDate="2011-03-23T10:54:10.860" />
  <row Id="8666" PostTypeId="2" ParentId="8661" CreationDate="2011-03-23T11:27:58.007" Score="7" ViewCount="" Body="&lt;p&gt;You are right that R's output usually contains only essential information, and more needs to be calculated separately.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;N  &amp;lt;- 100               # generate some data&#xA;X1 &amp;lt;- rnorm(N, 175, 7)&#xA;X2 &amp;lt;- rnorm(N,  30, 8)&#xA;X3 &amp;lt;- abs(rnorm(N, 60, 30))&#xA;Y  &amp;lt;- 0.5*X1 - 0.3*X2 - 0.4*X3 + 10 + rnorm(N, 0, 12)&#xA;&#xA;# dichotomize Y and do logistic regression&#xA;Yfac   &amp;lt;- cut(Y, breaks=c(-Inf, median(Y), Inf), labels=c(&quot;lo&quot;, &quot;hi&quot;))&#xA;glmFit &amp;lt;- glm(Yfac ~ X1 + X2 + X3, family=binomial(link=&quot;logit&quot;))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;coefficients()&lt;/code&gt; gives you the estimated regression parameters $b_{j}$. It's easier to interpret $exp(b_{j})$ though (except for the intercept).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; exp(coefficients(glmFit))&#xA; (Intercept)           X1           X2           X3 &#xA;5.811655e-06 1.098665e+00 9.511785e-01 9.528930e-01&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To get the odds ratio, we need the classification cross-table of the original dichotomous DV and the predicted classification according to some probability threshold that needs to be chosen first. You can also see function &lt;code&gt;ClassLog()&lt;/code&gt; in package &lt;code&gt;QuantPsyc&lt;/code&gt; (as chl mentioned in &lt;a href=&quot;http://stats.stackexchange.com/questions/4832/logistic-regression-classification-tables-a-la-spss-in-r/4835#4835&quot;&gt;a related question&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# predicted probabilities or: predict(glmFit, type=&quot;response&quot;)&#xA;&amp;gt; Yhat    &amp;lt;- fitted(glmFit)&#xA;&amp;gt; thresh  &amp;lt;- 0.5  # threshold for dichotomizing according to predicted probability&#xA;&amp;gt; YhatFac &amp;lt;- cut(Yhat, breaks=c(-Inf, thresh, Inf), labels=c(&quot;lo&quot;, &quot;hi&quot;))&#xA;&amp;gt; cTab    &amp;lt;- table(Yfac, YhatFac)    # contingency table&#xA;&amp;gt; addmargins(cTab)                   # marginal sums&#xA;     YhatFac&#xA;Yfac   lo  hi Sum&#xA;  lo   41   9  50&#xA;  hi   14  36  50&#xA;  Sum  55  45 100&#xA;&#xA;&amp;gt; sum(diag(cTab)) / sum(cTab)        # percentage correct for training data&#xA;[1] 0.77&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For the odds ratio, you can either use package &lt;code&gt;vcd&lt;/code&gt; or do the calculation manually.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; library(vcd)                       # for oddsratio()&#xA;&amp;gt; (OR &amp;lt;- oddsratio(cTab, log=FALSE)) # odds ratio&#xA;[1] 11.71429&#xA;&#xA;&amp;gt; (cTab[1, 1] / cTab[1, 2]) / (cTab[2, 1] / cTab[2, 2])&#xA;[1] 11.71429&#xA;&#xA;&amp;gt; summary(glmFit)  # test for regression parameters ...&#xA;&#xA;# test for the full model against the 0-model&#xA;&amp;gt; glm0 &amp;lt;- glm(Yfac ~ 1, family=binomial(link=&quot;logit&quot;))&#xA;&amp;gt; anova(glm0, glmFit, test=&quot;Chisq&quot;)&#xA;Analysis of Deviance Table&#xA;Model 1: Yfac ~ 1&#xA;Model 2: Yfac ~ X1 + X2 + X3&#xA;  Resid. Df Resid. Dev Df Deviance P(&amp;gt;|Chi|)    &#xA;1        99     138.63                          &#xA;2        96     110.58  3   28.045 3.554e-06 ***&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1909" LastActivityDate="2011-03-23T11:27:58.007" CommentCount="2" />
  <row Id="8667" PostTypeId="2" ParentId="8661" CreationDate="2011-03-23T11:28:45.930" Score="8" ViewCount="" Body="&lt;p&gt;if you want to interpret the estimated effects as relative odds ratios, just do &lt;code&gt;exp(coef(x))&lt;/code&gt; (gives you $e^\beta$, the multiplicative change in the odds ratio for $y=1$ if the covariate associated with $\beta$ increases by 1). For profile likelihood intervals for this quantity, you can do&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(MASS)&#xA;exp(cbind(coef(x), confint(x)))  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;EDIT: @caracal was quicker...&lt;/p&gt;&#xA;" OwnerUserId="1979" LastActivityDate="2011-03-23T11:28:45.930" CommentCount="1" />
  <row Id="8668" PostTypeId="2" ParentId="8662" CreationDate="2011-03-23T11:43:00.033" Score="1" ViewCount="" Body="&lt;p&gt;I am not sure why you would want to classify a sample to a specific distribution with such a large sample size; parsimony, comparing it to another sample, looking for physical interpretation of the paramters?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most statistical packages(R, SAS, Minitab) allow one to plot data on a graph that yields a straight line if the data come from a particular distribution. I have seen graphs that yield a straight line if the data is normal(log normal-after a log transformation), Weibull, and chi-squared come to mine immediately. This technique will allow you to see outliers and give you the possiblity to assign reasons for why data points are outliers.&#xA;In R, the normal probability plot is called qqnorm.  &lt;/p&gt;&#xA;" OwnerUserId="3805" LastActivityDate="2011-03-23T11:43:00.033" CommentCount="3" />
  <row Id="8669" PostTypeId="1" AcceptedAnswerId="9376" CreationDate="2011-03-23T12:02:12.467" Score="2" ViewCount="59" Body="&lt;p&gt;Hi everyone, I currently have two sets of input variables say, $X$ and $Y$ with one output variable $Z$. That is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Z = a_0 + a_1X_1 + a_2X_2... + a_{11}X_{11} = b_0 + b_1Y_1 + b_2Y_2 + b_3Y_3 + b_4Y_4$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have the independent $X$ and $Y$ values but don't have the dependent variable $Z$ values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there anyway that I can estimate coefficients $a$ and $b$ and also the value of R squared?&lt;/p&gt;&#xA;" OwnerUserId="3859" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-03-23T13:47:27.210" LastActivityDate="2011-04-09T18:43:50.403" Title="Two sets of input variables for the same unknown dependent variable" Tags="&lt;regression&gt;" AnswerCount="3" CommentCount="3" />
  <row Id="8670" PostTypeId="2" ParentId="8669" CreationDate="2011-03-23T12:21:50.093" Score="0" ViewCount="" Body="&lt;p&gt;This sounds to me like problem where a canonical correlation study might help. In canonical correlation, we are given a random vector W that is partitioned into two sub-random vectors X and Y; and the issue is to find linear combinations of the two subvectors that have maximal correlation and are orthogonal to one another.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The end result is the discovery of correlates that will point to relationships that have some physical meaning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are comfortable with matrices, then Google Wolfgang Hardle and Leopold Simar's online book and read Chapter 14 on Canonical Correlation. If you use R, then read Chapter 10 of Paul Hewson's online book &quot;Multivariate Statistics with R&quot;. &lt;/p&gt;&#xA;" OwnerUserId="3805" LastActivityDate="2011-03-23T12:21:50.093" CommentCount="1" />
  <row Id="8671" PostTypeId="2" ParentId="8625" CreationDate="2011-03-23T14:12:28.437" Score="3" ViewCount="" Body="&lt;p&gt;If you assume &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{1}{N}\sum_{t=1}^Nx_t^{i}=\mu_i$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\sum_{i=1}^C\sum_{t=1}^N(x_t^i-\mu_i)(\mu_i-\mu)^T=\sum_{i=1}^C\left(\sum_{t=1}^N(x_t^i-\mu_i)\right)(\mu_i-\mu)^T=0$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and formula holds. You deal with the second term in the similar way.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-03-23T14:12:28.437" CommentCount="2" />
  <row Id="8672" PostTypeId="2" ParentId="8661" CreationDate="2011-03-23T14:39:55.110" Score="3" ViewCount="" Body="&lt;p&gt;The UCLA stats page has &lt;a href=&quot;http://www.ats.ucla.edu/stat/r/dae/logit.htm&quot; rel=&quot;nofollow&quot;&gt;a nice walk-through&lt;/a&gt; of performing logistic regression in R. It includes a brief section on calculating odds ratios.&lt;/p&gt;&#xA;" OwnerUserId="124" LastActivityDate="2011-03-23T14:39:55.110" />
  <row Id="8673" PostTypeId="2" ParentId="5346" CreationDate="2011-03-23T14:55:49.300" Score="1" ViewCount="" Body="&lt;p&gt;I am so nervous to answer this as I am a total noob, but I found this article by Algina &amp;amp; Olejnik, 1984: &lt;a href=&quot;http://epm.sagepub.com/content/44/1/39.short&quot; rel=&quot;nofollow&quot;&gt;http://epm.sagepub.com/content/44/1/39.short&lt;/a&gt; The abstract: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The Welch-James procedure may be used&#xA;  to test hypotheses on means, when&#xA;  independent samples from populations&#xA;  with heterogeneous variances are&#xA;  available. Until recently the&#xA;  complexity of the available&#xA;  presentations of this procedure&#xA;  limited the application of this&#xA;  procedure. To resolve this state of&#xA;  affairs, summation formulas for the&#xA;  Welch-James procedure are presented&#xA;  for the 2 x 2 design. In addition,&#xA;  matrix formulas that permit routine&#xA;  application of the procedure to&#xA;  crossed factorial designs are&#xA;  presented.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It frankly looks a little hairy, but I thought it might be a start. &lt;/p&gt;&#xA;" OwnerUserId="3861" LastActivityDate="2011-03-23T14:55:49.300" CommentCount="1" />
  <row Id="8674" PostTypeId="2" ParentId="8662" CreationDate="2011-03-23T15:04:06.113" Score="10" ViewCount="" Body="&lt;p&gt;Use fitdistrplus:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://cran.r-project.org/web/packages/fitdistrplus/index.html&quot; rel=&quot;nofollow&quot;&gt;http://cran.r-project.org/web/packages/fitdistrplus/index.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The vignette does a good job at explaining how to use the package.   You can look at how various distributions fit in a short period of time.   It also produces a Cullen/Frey Diagram.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/tC5qK.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/XORdX.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="2775" LastEditorUserId="2775" LastEditorDisplayName="" LastEditDate="2011-03-23T15:10:21.820" LastActivityDate="2011-03-23T15:10:21.820" CommentCount="1" />
  <row Id="8675" PostTypeId="2" ParentId="8642" CreationDate="2011-03-23T15:33:25.353" Score="1" ViewCount="" Body="&lt;p&gt;The inferences generated by Wilcoxon vs bootstrapping cannot be compared as they pertain to different data. Wilcoxon is a rank test, thus generates inferences that pertain to ranks. Bootstrapping applies to the raw data, and thus generates inferences that pertain to the raw data. If you dislike bootstrapping but want inferences that pertain to the raw data, then you may want to try a permutation test (sometimes referred to as a randomization test).&lt;/p&gt;&#xA;" OwnerUserId="364" LastActivityDate="2011-03-23T15:33:25.353" CommentCount="1" />
  <row Id="8676" PostTypeId="2" ParentId="8630" CreationDate="2011-03-23T15:34:35.570" Score="0" ViewCount="" Body="&lt;p&gt;Let the total variance in a data set of vectors be the sum of squared errors (SSE) between the vectors in the data set and the mean vector of the data set.  I.e. the sum of (x_i - x_bar) dot (x_i - x_bar), where x_bar is the mean vector of the data set, x_i is the ith vector in the data set, and dot is the dot product of two vectors.  Said another way, the total variance is the sum of squared errors (SSE) between each x_i and its predicted value, where the predicted value is x_bar.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now let the predictor of x_i, f(x_i), be the projection of vector x_i onto a unit vector c, which is (c dot x_i)*c.  Then the SSE is the sum over all x_i of (x_i - (c dot x_i)*c) dot (x_i - (c dot x_i)*c)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think that if you choose c to minimize the SSE above, then c is the first principal component.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If instead you choose c to be the normalized version of the vector (1, 2, 5, ...), then you can calculate the SSE.  Subtracting this SSE from the total variance gives the variance in the data described by using c as a predictor.&lt;/p&gt;&#xA;" OwnerUserId="3864" LastActivityDate="2011-03-23T15:34:35.570" />
  <row Id="8677" PostTypeId="1" AcceptedAnswerId="8687" CreationDate="2011-03-23T15:47:58.887" Score="3" ViewCount="97" Body="&lt;p&gt;Hi all,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I just came by &lt;a href=&quot;http://www.investuotojas.eu/?p=464&quot; rel=&quot;nofollow&quot;&gt;a post talking about&lt;/a&gt; networks for displaying correlations:&#xA;&lt;img src=&quot;http://i.imgur.com/BhyaZ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this a known method?  Can someone shed some insights into it? (I'm wondering about how useful it might be, and when.)&lt;/p&gt;&#xA;" OwnerUserId="253" LastEditorUserId="26" LastEditorDisplayName="" LastEditDate="2011-03-23T19:55:45.100" LastActivityDate="2011-03-23T19:55:45.100" Title="References for using networks to display correlations?" Tags="&lt;data-visualization&gt;&lt;correlation&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="8678" PostTypeId="2" ParentId="8677" CreationDate="2011-03-23T16:36:48.973" Score="3" ViewCount="" Body="&lt;p&gt;Surprisingly, as a &lt;a href=&quot;http://www.google.com/images?q=multiple+correlation&quot; rel=&quot;nofollow&quot;&gt;search of Google Images&lt;/a&gt; indicates, such graphs do not appear to be in common use to study or explain multiple correlations.  That's a pity, because I'm sure much of this theory can be reduced to simple operations on graphs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Nevertheless, this graphical method to display correlations (or their mathematical equivalents, cosines of angles) has been in use a long time (&lt;a href=&quot;http://en.wikipedia.org/wiki/Dynkin_diagram#History&quot; rel=&quot;nofollow&quot;&gt;at least 75 years&lt;/a&gt;) in the form of a &lt;a href=&quot;http://en.wikipedia.org/wiki/Coxeter%E2%80%93Dynkin_diagram&quot; rel=&quot;nofollow&quot;&gt;Coxeter-Dynkin diagram &lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, the A3 diagram 0--0--0 represents three variables &lt;strong&gt;X&lt;/strong&gt;, &lt;strong&gt;Y&lt;/strong&gt;, and &lt;strong&gt;Z&lt;/strong&gt; where &lt;strong&gt;X&lt;/strong&gt; and &lt;strong&gt;Z&lt;/strong&gt; (the outer nodes) are uncorrelated and the correlations between &lt;strong&gt;X&lt;/strong&gt; and &lt;strong&gt;Y&lt;/strong&gt; and &lt;strong&gt;Z&lt;/strong&gt; and &lt;strong&gt;Y&lt;/strong&gt; are both -0.5.  In the usual applications of these diagrams, certain special &quot;correlations&quot; (angles) are important, so a special method of labeling the edges with the correlations is used, but this functions no differently than using other forms of labeling such as colors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you use a &quot;distance&quot; metric monotonically related to correlation, then any 2D &lt;a href=&quot;http://en.wikipedia.org/wiki/Multidimensional_scaling&quot; rel=&quot;nofollow&quot;&gt;MDS calculation&lt;/a&gt; can be (and usually is) thought of as embedding this graph in the plane so that Euclidean distances reflect the correlations.  This illustrates the intimate connection between correlation-based clustering methods and graphs of correlation.  As another example in this vein, a &lt;a href=&quot;http://www.mathworks.com/help/toolbox/stats/dendrogram.html&quot; rel=&quot;nofollow&quot;&gt;dendrogram&lt;/a&gt;, when it is derived from a correlation-based similarity matrix, is another network-based way of displaying correlations.  (However, it uses vertical position in an essential way to display similarity, and so is not a purely network-based method.)&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-03-23T16:36:48.973" CommentCount="1" />
  <row Id="8679" PostTypeId="2" ParentId="8657" CreationDate="2011-03-23T16:42:38.700" Score="3" ViewCount="" Body="&lt;p&gt;You can find minimum voting ages in &lt;a href=&quot;http://en.wikipedia.org/wiki/Voting_age&quot; rel=&quot;nofollow&quot;&gt;Wikipedia&lt;/a&gt;.  Most large countries use 18, except for Brazil and Indonesia.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can find country population by age in the &lt;a href=&quot;http://www.census.gov/ipc/www/idb/&quot; rel=&quot;nofollow&quot;&gt;U.S Census Bureau International Data Base&lt;/a&gt;. It does not seem to use 18 as a break point, but you should be able to divide the 15-19 age group safely. This suggests slightly less than 69% of the world population is 18+ in 2011.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Voter turnout by age is harder, though you may not be interested. I would expect there to be national figures in scattered sources but not a central collection.  The &lt;a href=&quot;http://www.idea.int/vt/by_age.cfm&quot; rel=&quot;nofollow&quot;&gt;International Institute for Democracy and Electoral Assistance&lt;/a&gt; tried but only managed to list three countries.   &lt;/p&gt;&#xA;" OwnerUserId="2958" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-03-23T18:45:20.533" LastActivityDate="2011-03-23T18:45:20.533" CommentCount="1" />
  <row Id="8680" PostTypeId="2" ParentId="8502" CreationDate="2011-03-23T16:55:09.020" Score="1" ViewCount="" Body="&lt;p&gt;Maybe you should look into &quot;stacking&quot;. Or even &quot;feature-weighed stacking&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The former is using a cross validation method to determine the weights you should use to linearly stack them. The latter is using &quot;meta-parameters&quot; to give even more insight on how to weight the parameters depending on what is being predicted. This is a method that the #2 Netflix competition team developed. &lt;a href=&quot;http://arxiv.org/abs/0911.0460&quot; rel=&quot;nofollow&quot;&gt;http://arxiv.org/abs/0911.0460&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="3834" LastActivityDate="2011-03-23T16:55:09.020" />
  <row Id="8681" PostTypeId="2" ParentId="6920" CreationDate="2011-03-23T17:29:07.060" Score="0" ViewCount="" Body="&lt;p&gt;The problem is more easily solved when you rewrite things a little bit:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Y = y&lt;/p&gt;&#xA;&#xA;&lt;p&gt;X = [x, 1 ]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;then&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Y = A*X&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A one time-solution is found by calculating &lt;/p&gt;&#xA;&#xA;&lt;p&gt;V = X' * X&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and&lt;/p&gt;&#xA;&#xA;&lt;p&gt;C = X' * Y&lt;/p&gt;&#xA;&#xA;&lt;p&gt;note the V should have size N-by-N and C a size of N-by-M.&#xA;The parameters you're looking for are then given by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A = inv(V) * C&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since both V and C are calculated by summing over your data, you can calculate&#xA;A at every new sample. This has a time complexity of O(N^3), however.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since V is square and semi-definite positive, a LU decomposition does exists, that&#xA;makes inverting V numerically more stable.&#xA;There are algorithms to perform rank-1 updates to the inverse of a matrix. Find those and you'll have the efficient implementation you're looking for.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The rank-1 update algorithms can be found in &quot;Matrix computations&quot; by Golub and van Loan. It's tough material, but it does have a comprehensive overview of such algorithms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note:&#xA;The method above gives a least-square estimate at each step. You can easily adding weights to the updates to X and Y. When the values of X and Y grow too large, they can be scaled by a single scalar, without affecting the result.&lt;/p&gt;&#xA;" OwnerUserId="3867" LastActivityDate="2011-03-23T17:29:07.060" />
  <row Id="8682" PostTypeId="2" ParentId="8663" CreationDate="2011-03-23T17:41:08.740" Score="1" ViewCount="" Body="&lt;p&gt;First, you need to know what a p-value is.  A p-value is the probability that you would observe results as extreme, or more extreme, than the ones you have, if the null hypothesis was in fact true.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reason you aren't getting the same p-value in your two tests, is that you aren't examining the same null hypotheses under the same assumptions.  Pick up a stats textbook.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If a relationship between two variables is approximately linear, binning will reduce statistical power, which probably explains why you get a lower p-value without binning.  Think of it this way, binning treats different values within a bin as identical.  The information about the differences within a bin may be valuable, but this information is ignored by statistical tests.  &lt;/p&gt;&#xA;" OwnerUserId="3748" LastActivityDate="2011-03-23T17:41:08.740" CommentCount="1" />
  <row Id="8683" PostTypeId="2" ParentId="7897" CreationDate="2011-03-23T17:46:42.810" Score="0" ViewCount="" Body="&lt;p&gt;Lasso is indeed a good one. Simple things like starting with none, and adding them one by one sorted on 'usefullness' (via cross-validation) do also work quite well in practice. &#xA;This is sometimes called stagewise feedforward selection.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that the subset selection problem is fairly independent on the type of classification / regression. It's just that nonparametric methods can be slow and therefore require more intelligent methods of selection.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The book 'The elements of statistical learning' from T. Hastie gives a nice overview.&lt;/p&gt;&#xA;" OwnerUserId="3867" LastActivityDate="2011-03-23T17:46:42.810" />
  <row Id="8684" PostTypeId="2" ParentId="8502" CreationDate="2011-03-23T17:56:30.697" Score="6" ViewCount="" Body="&lt;p&gt;You are retaining $p$ (=3 in this case) values for each regression: the estimated coefficients.  If you are willing to retain $p(p+1)$ (=12) values per regression, you can weight your results in a way that is equivalent to having &lt;em&gt;all&lt;/em&gt; the data and performing a weighted least squares regression with them &lt;em&gt;en masse&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The analysis is simple: let $X_1$ be the design matrix (&lt;em&gt;i.e.&lt;/em&gt;, an $n_1$ by $p$ matrix of independent variable values) for the first year and $y_1$ be the $n_1$-vector of dependent values for that year.  The estimated coefficients are&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\hat{\beta}_1 = \left( X_1&amp;#39; X_1 \right)^{-1} X_1&amp;#39; y_1.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let the subscript $2$ designate the same quantities for the second year.  Suppose you would like to uniformly weight all observations with (positive) values $w_1^2$ and $w_2^2$ in those two years.  The design matrix $X$ is the vertical concatenation of $X_1$ and $X_2$, an $n_1+n_2$ by $p$ matrix, and similarly the vector of dependent values $y$ is the vertical concatenation of $y_1$ and $y_2$.  Let $W$ be the diagonal matrix with values $w_1$ along the first $n_1$ places and $w_2$ along the last $n_2$ places.  The &lt;a href=&quot;http://en.wikipedia.org/wiki/Least_squares#Weighted_least_squares&quot; rel=&quot;nofollow&quot;&gt;weighted least squares&lt;/a&gt; solution is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\hat{\beta} = \left( (W X)&amp;#39; (W X) \right)^{-1} (W X)&amp;#39; W y.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, $(W X)&amp;#39; (W X)$ = $X&amp;#39; W&amp;#39;W X$ is the vertical concatenation of $X_1 W_1 W_1&amp;#39; X_1$ and $X_2 W_2 W_2&amp;#39; X_1$.  Because both $W_1 W_1&amp;#39;$ and $W_2 W_2&amp;#39;$ are multiples of identity matrices, they factor through, giving&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\hat{\beta} = \left( w_1^2 X_1&amp;#39; X_1 + w_2^2 X_2&amp;#39; X_2 \right)^{-1} \left(w_1 X_1 y_1 + w_2 X_2 y_2\right).$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Notice that $X_1&amp;#39; X_1$ and $X_2&amp;#39; X_2$ are just $p$ by $p$ matrices and that $X_1 y_1$ and $X_2 y_2$ are just $p$-vectors.  Therefore you can obtain $\hat{\beta}$ just from the two $p$ by $p$ matrices, the two $p$-vectors, and the two numbers $w_1$ and $w_2$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This approach generalizes in an obvious way when more than two regressions are involved.  It shows, incidentally, that the weighted combination $w_1^2 \hat{\beta_1} + w_2^2 \hat{\beta_2}$ as suggested in the question will not in general equal the weighted least-squares solution.  Therefore, if you are using least squares for any of its optimality properties, you should &lt;em&gt;not&lt;/em&gt; want to use this seductively simple solution, because it will be suboptimal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;In conclusion&lt;/strong&gt;, if you would store the 12 numbers $X_i&amp;#39; X_i$ and $X_i&amp;#39; y_i$ each year, then &lt;em&gt;retrospectively&lt;/em&gt; (without needing the original data) you can fit &lt;em&gt;any&lt;/em&gt; regression on &lt;em&gt;all the data&lt;/em&gt; for &lt;em&gt;any set of positive weights&lt;/em&gt; without any loss of information.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would recommend saving some additional values such as the estimated error variances: these will help you detect changes in variability over time (heteroscedasticity).&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-03-23T17:56:30.697" CommentCount="2" />
  <row Id="8685" PostTypeId="2" ParentId="7897" CreationDate="2011-03-23T18:01:20.987" Score="2" ViewCount="" Body="&lt;p&gt;Unless identification of the most relevant variables is a key aim of the analysis, it is often better not to do any feature selection at all and use regularisation to prevent over-fitting.  Feature selection is a tricky procedure and it is all too easy to over-fit the feature selection criterion as there are many degrees of freedom.  LASSO and elastic net are a good compromise, the achieve sparsity via regularisation rather than via direct feature selection, so they are less prone to that particular form of over-fitting.&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2011-03-23T18:01:20.987" />
  <row Id="8687" PostTypeId="2" ParentId="8677" CreationDate="2011-03-23T18:32:52.837" Score="4" ViewCount="" Body="&lt;p&gt;Do you know the &lt;a href=&quot;http://sites.google.com/site/qgraphproject/&quot; rel=&quot;nofollow&quot;&gt;qgraph&lt;/a&gt; project (and the related &lt;a href=&quot;http://cran.r-project.org/web/packages/qgraph/index.html&quot; rel=&quot;nofollow&quot;&gt;R package&lt;/a&gt;)? It aims at providing various displays for psychometric models, especially those relying on correlations. I discovered this approach for displaying correlation measures when I was reading a very nice and revolutionary article on diagnostic medicine by Denny Borsboom and coll.: &lt;a href=&quot;http://sites.google.com/site/borsboomdenny/CramerEtAl2010.pdf&quot; rel=&quot;nofollow&quot;&gt;Comorbidity: A network perspective&lt;/a&gt;, BBS (2010) 33: 137-193. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;An oversimplified summary of their &lt;em&gt;network approach&lt;/em&gt; of comorbidity is that it is “hypothesized to arise from direct relations between symptoms of multiple disorders”, contrary to the more classical view where these are comorbid disorders themselves that causes their associated symptoms to correlate (as reflected in a latent variable model, like factor or item response models, where a given symptom would allow to measure a particular disorder). In fact, symptoms are part of disorder, but they don’t measure it (and this is a mereological relationship). Their figure 5 describes such a &quot;comorbidity network&quot; and is particularly interesting as it embeds the frequency of symptoms and magnitude of their bivariate association in the same picture. They were using &lt;a href=&quot;http://www.cytoscape.org/&quot; rel=&quot;nofollow&quot;&gt;Cytoscape&lt;/a&gt; at that time, but the qgraph project has now reached a mature state. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are some examples from the on-line R help; basically, these are (1) an association graph with circular or (2) spring layout, (3) a concentration graph with spring layout, and (4) a factorial graph with spring layout (but see &lt;code&gt;help(qgraph.panel)&lt;/code&gt;):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/hYT3r.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(See also &lt;code&gt;help(qgraph.pca)&lt;/code&gt; for nice circular displays of an observed correlation matrix for the NEO-FFI, which is a 60-item personality inventory.)&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-03-23T18:32:52.837" CommentCount="8" />
  <row Id="8688" PostTypeId="2" ParentId="1133" CreationDate="2011-03-23T19:27:19.840" Score="0" ViewCount="" Body="&lt;p&gt;I don't know exactly what your goals are, or why they are what they are.  But rather than hypothesis testing, I usually recommend focusing attention on predictions and confidence intervals.&lt;/p&gt;&#xA;" OwnerUserId="3748" LastActivityDate="2011-03-23T19:27:19.840" />
  <row Id="8689" PostTypeId="1" AcceptedAnswerId="8713" CreationDate="2011-03-23T19:48:25.340" Score="1" ViewCount="272" Body="&lt;p&gt;In R, if I write&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;lm(a ~ b + c + b*c) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;would this still be a linear regression? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How to do other kinds of regression in R? I would appreciate any recommendation for textbooks or tutorials?&lt;/p&gt;&#xA;" OwnerUserId="3870" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-03-23T20:09:01.350" LastActivityDate="2011-03-24T04:03:07.070" Title="What does linear stand for in linear regression?" Tags="&lt;r&gt;&lt;regression&gt;" AnswerCount="3" CommentCount="3" />
  <row Id="8690" PostTypeId="1" AcceptedAnswerId="8760" CreationDate="2011-03-23T20:11:04.610" Score="6" ViewCount="148" Body="&lt;p&gt;Suppose I wanted to fit a model of the form&#xA;$$y_i = \beta_0 + \sum_{1 \le j \le k} \beta_j X_{i,j} + \gamma_i Z_i + \epsilon_i,$$&#xA;to some data, where the regressors $X$ and $Z$, and the regressand $y$ are observed, and where $\gamma_i$ is a Bernoulli random variable that equals one with (unknown) probability $p$ and is zero otherwise. We can assume all kinds of 'regularity': the errors $\epsilon_i$ are independent of the regressors, and are independent of $\gamma_i$, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;What is the name for this, if there is one?&lt;/li&gt;&#xA;&lt;li&gt;If I just throw the $Z$ data in with the $X$ data and perform an ordinary multiple least squares, will the least squares coefficient corresponding to the $Z$ term converge to $p$ as I add more observations?&lt;/li&gt;&#xA;&lt;li&gt;If least squares regression is advisable for this model, what is the distribution of the least squares coefficient corresponding to the $Z$ term under the null hypothesis $p = p_0$? (For a 'deterministic' regression, the coefficient has a certain t-distribution, with parameters depending on $p_0, n, k$ and the design matrix; I am looking for the analogue for the random coefficient.)&lt;/li&gt;&#xA;&lt;li&gt;If least squares regression is advisable, how will the presence of the random coefficient affect the distribution of the other (sample) regression coefficients?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="795" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-25T10:59:58.783" LastActivityDate="2011-03-28T12:08:49.257" Title="Linear model with random coefficient" Tags="&lt;regression&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="8691" PostTypeId="1" CreationDate="2011-03-23T20:14:56.090" Score="7" ViewCount="77" Body="&lt;p&gt;I'm comparing scores for two small groups of individuals that competed in a tournament, and I'm being told that the comparison calls for a Mann-Whitney U test. It feels wrong to me, though: my two sets of scores are fundamentally interdependent because the two groups competed against one another.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Briefly: I have two groups, a control group A (n=10 men) and an experimental group B (n=12 men). Group B received a treatment, and then the members of group A and group B were pitted against each other in a tournament. I'm interested in the degree to which A's vs. B's were successful at beating the opposite-group men they competed against. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In each day of tournament play, two A's and two B's competed. Each game was every man for himself, competing for points (the task is irrelevant, I think). On a single day, there were always 4 competitors - 2 A's and 2 B's. But if any of those individuals reached criterion that day (scored a certain number of points, essentially), he got pulled and replaced the next day with a new player from the same home group (A or B). An individual could reach criterion in up to 7 days (if you went 7 days without reaching criterion you were considered to have lost, and pulled from play). This meant that an individual (say, an A) that played and won in a single day faced only three other competitors - 1 A and 2 B's. But an A that did poorly and stayed for 7 days could face many more players - a bunch of A's and B's - as better competitors cycled through.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I've given each of the men a rank score that reflects the percentage of opposite-group men that he beat. Let's say an A named Joe competed for two days and faced 1 other A and 3 B's, and he came in second, behind the one other A in the group but above the three B's. His score would be 1.0. If Joe had had a harder time and had taken more days to reach criterion, he would probably have faced more competitors overall, but his score would still have been 1.0 if he had outcompeted all of the B's that he met. The score attempts to measure players' effectiveness at beating opposite-group men, and to allow comparison  across men that faced different numbers of competitors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the rank scores for the two groups look like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A: 1, 1, 1, 0.833333, 0.75, 0.833333, 0.5, 0.333333, 0.333333, 0.5, 0.333333, 0.125&lt;/p&gt;&#xA;&#xA;&lt;p&gt;B: 1, 0.5, 0.666667, 0.5, 0.333333, 0.5, 0.5, 0.2, 0.166667, 0&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And my question is: is there a more valid way to see if there's a difference between the groups than a Mann-Whitney?&lt;/p&gt;&#xA;" OwnerUserId="3871" LastActivityDate="2011-03-24T10:34:54.860" Title="Statistical analysis of competition data." Tags="&lt;hypothesis-testing&gt;&lt;nonparametric&gt;" AnswerCount="2" />
  <row Id="8692" PostTypeId="1" AcceptedAnswerId="8697" CreationDate="2011-03-23T20:16:06.923" Score="1" ViewCount="133" Body="&lt;p&gt;I want to transpose a data frame in R with &lt;code&gt;unstack&lt;/code&gt;. Consider the two data frames, &lt;code&gt;a&lt;/code&gt; and &lt;code&gt;b&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; a&#xA;   count state&#xA;1 199665  RSTO&#xA;2   4147  RSTR&#xA;3  31274    S1&#xA;4      1    S2&#xA;5   2522    S3&#xA;6 118009    SF&#xA;&amp;gt; b&#xA;   count state&#xA;1  31956  RSTO&#xA;2  11689  RSTR&#xA;3   6702    S1&#xA;4   2838    S2&#xA;5   6268    S3&#xA;6 672561    SF&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My problem is that unstacking a single one does not work:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; formula(a)&#xA;count ~ state&#xA;&amp;gt; unstack(a)&#xA;        res&#xA;RSTO 199665&#xA;RSTR   4147&#xA;S1    31274&#xA;S2        1&#xA;S3     2522&#xA;SF   118009&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, if I concatenate &lt;code&gt;a&lt;/code&gt; and &lt;code&gt;b&lt;/code&gt;, &lt;code&gt;unstack&lt;/code&gt; works as expected.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; unstack(rbind(a,b))&#xA;    RSTO  RSTR    S1   S2   S3     SF&#xA;1 199665  4147 31274    1 2522 118009&#xA;2  31956 11689  6702 2838 6268 672561&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Why is this happening? Does the groups (i.e., the RHS of the formula) need to repeat for &lt;code&gt;unstack&lt;/code&gt; to work properly? How can I make &lt;code&gt;unstack&lt;/code&gt; work with a single data frame?&lt;/p&gt;&#xA;" OwnerUserId="1537" LastActivityDate="2011-03-25T09:15:16.203" Title="Transposing data frames in R via unstack" Tags="&lt;r&gt;&lt;data-transformation&gt;" AnswerCount="3" />
  <row Id="8693" PostTypeId="2" ParentId="8689" CreationDate="2011-03-23T20:23:54.697" Score="2" ViewCount="" Body="&lt;p&gt;I would be careful in asking this as an &quot;R linear regression&quot; question versus a &quot;linear regression&quot; question.   Formulas in R have rules that you may or may not be aware of.  For example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://wiener.math.csi.cuny.edu/st/stRmanual/ModelFormula.html&quot; rel=&quot;nofollow&quot;&gt;http://wiener.math.csi.cuny.edu/st/stRmanual/ModelFormula.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming you're asking if the following equation is linear:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;a = coeff0 + (coeff1 * b) + (coeff2 * c) + (coeff3 * (b*c))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The answer is yes, if you assemble a new independent variable such as:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;newv = b * c&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Substituting the above newv equation into the original equation probably looks like what you're expecting for a linear equation:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;a = coeff0 + (coeff1 * b) + (coeff2 * c) + (coeff3 * newv)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As far as references go, Google &quot;r regression&quot;, or whatever you think might work for you. &lt;/p&gt;&#xA;" OwnerUserId="2775" LastEditorUserId="2775" LastEditorDisplayName="" LastEditDate="2011-03-23T21:48:53.517" LastActivityDate="2011-03-23T21:48:53.517" CommentCount="4" />
  <row Id="8694" PostTypeId="2" ParentId="8691" CreationDate="2011-03-23T20:54:36.260" Score="6" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Yes&lt;/strong&gt;.  This situation is so complicated and the results are so interdependent that the applicability of almost any &lt;em&gt;standard&lt;/em&gt; test has to be called into question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why not conduct a &lt;a href=&quot;http://en.wikipedia.org/wiki/Resampling_%28statistics%29#Permutation_tests&quot; rel=&quot;nofollow&quot;&gt;permutation test&lt;/a&gt;?  This is a natural situation for it: the null hypothesis of no treatment effect basically says the labels are meaningless.  So, keep all the results but permute the labels, always maintaining a group of 10 &quot;controls&quot; and a group of 12 &quot;treatment&quot; subjects.  Compute any ranking or relative score between the groups you deem meaningful for every one of the $\binom{22}{10}$ = 646,646 permutations.  (You could randomly sample the permutations to save time, but their number is small enough that this brute-force calculation is easily carried out.)  That's the permutation distribution for your statistic under the null hypothesis.  To determine the p-value, see where the observed value of the statistic falls on the cumulative distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;BTW, if the men were not chosen to compete at random (formally--not arbitarily--using a random number generator), then one could validly suspect any apparent difference might be due to the sequence in which the men competed.  No statistical test can overcome such a deficiency if it is present.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-23T21:19:11.903" LastActivityDate="2011-03-23T21:19:11.903" CommentCount="1" />
  <row Id="8695" PostTypeId="1" CreationDate="2011-03-23T20:57:27.787" Score="2" ViewCount="474" Body="&lt;p&gt;I got some issues with an explorative factor analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anbody please tell me how to calculate the Average Variance&#xA;Extracted (AVE) and the Composite Reliability from two factors each with three items using SPSS? If not with SPSS Stata might help too...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thx in advance for your help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Raoul&lt;/p&gt;&#xA;" OwnerUserId="3872" LastActivityDate="2011-03-26T17:51:56.140" Title="AVE &amp; Composite Reliability with SPSS" Tags="&lt;spss&gt;&lt;factor-analysis&gt;&lt;reliability&gt;&lt;composite&gt;" AnswerCount="2" />
  <row Id="8696" PostTypeId="1" AcceptedAnswerId="9956" CreationDate="2011-03-23T21:04:15.040" Score="4" ViewCount="74" Body="&lt;p&gt;I have the &lt;a href=&quot;http://caret.r-forge.r-project.org/Classification_and_Regression_Training.html&quot; rel=&quot;nofollow&quot;&gt;caret&lt;/a&gt; package for R, and I'm interesting in using the &lt;code&gt;train&lt;/code&gt; function to cross-validate my models.  However, I want to speed things up, and it seems that caret provides support for parallel processing.  What is the best way to access this feature on a Windows machine?  I have the &lt;a href=&quot;http://cran.r-project.org/web/packages/doSMP/doSMP.pdf&quot; rel=&quot;nofollow&quot;&gt;doSMP&lt;/a&gt; package, but I can't figure out how to translate the &lt;code&gt;foreach&lt;/code&gt; function into an &lt;code&gt;lapply&lt;/code&gt; function, so I can pass it to the &lt;code&gt;train&lt;/code&gt; function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an example of what I want to do, from the &lt;code&gt;train&lt;/code&gt; documentation:  This is exactly what I want to do, but using the &lt;code&gt;doSMP&lt;/code&gt; package, rather than the &lt;code&gt;doMPI&lt;/code&gt; package.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;## A function to emulate lapply in parallel&#xA;mpiCalcs &amp;lt;- function(X, FUN, ...)&#xA;}&#xA;    theDots &amp;lt;- list(...)&#xA;    parLapply(theDots$cl, X, FUN)&#xA;{&#xA;&#xA;library(snow)&#xA;cl &amp;lt;- makeCluster(5, &quot;MPI&quot;)&#xA;&#xA;## 50 bootstrap models distributed across 5 workers&#xA;mpiControl &amp;lt;- trainControl(workers = 5,&#xA;    number = 50,&#xA;    computeFunction = mpiCalcs,&#xA;    computeArgs = list(cl = cl))&#xA;&#xA;set.seed(1)&#xA;usingMPI &amp;lt;- train(medv ~ .,&#xA;    data = BostonHousing,&#xA;    &quot;glmboost&quot;,&#xA;    trControl = mpiControl)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2817" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-25T14:52:31.177" LastActivityDate="2011-04-25T15:34:00.080" Title="Parallelizing the caret package using doSMP" Tags="&lt;r&gt;&lt;parallel-computing&gt;" AnswerCount="1" FavoriteCount="2" />
  <row Id="8697" PostTypeId="2" ParentId="8692" CreationDate="2011-03-23T21:15:39.913" Score="3" ViewCount="" Body="&lt;p&gt;Note that &lt;code&gt;rbind(a, b)&lt;/code&gt; creates a single data frame, so that's not it. The unexpected behavior of &lt;code&gt;unstack(a)&lt;/code&gt; results from the fact that you only have one observation (&lt;code&gt;count&lt;/code&gt;) per factor level (&lt;code&gt;state&lt;/code&gt;). To see what's going on, you have to look at the &lt;code&gt;unstack()&lt;/code&gt; function.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# list source code for unstack()'s method for a data frame&#xA;&amp;gt; getS3method(&quot;unstack&quot;, &quot;data.frame&quot;)&#xA;function (x, form, ...) &#xA;{&#xA;    form &amp;lt;- if (missing(form)) &#xA;        stats::formula(x)&#xA;    else stats::as.formula(form)&#xA;    if (length(form) &amp;lt; 3) &#xA;        stop(&quot;'form' must be a two-sided formula&quot;)&#xA;    res &amp;lt;- c(tapply(eval(form[[2L]], x), eval(form[[3L]], x), &#xA;        as.vector))&#xA;    if (length(res) &amp;lt; 2L || any(diff(unlist(lapply(res, length))) != &#xA;        0L)) &#xA;        return(res)&#xA;    data.frame(res)&#xA;}&#xA;&amp;lt;environment: namespace:utils&amp;gt;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The relevant bits are &lt;code&gt;c(tapply(eval(form[[2L]], x), eval(form[[3L]], x), as.vector))&lt;/code&gt; and &lt;code&gt;data.frame(res)&lt;/code&gt;. The first expression here is equivalent to&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; c(tapply(a$count, a$state, as.vector))&#xA;  RSTO   RSTR     S1     S2     S3     SF &#xA;199665   4147  31274      1   2522 118009&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The output of &lt;code&gt;tapply()&lt;/code&gt; is a 1-dimensional array if the provided function only returns one value when applied to the data in each level of the factor. &lt;code&gt;c()&lt;/code&gt; strips the class, thus returning a numeric vector. &lt;code&gt;data.frame(&amp;lt;vector&amp;gt;)&lt;/code&gt; creates a data frame with one variable equal to that vector. Compare this to what happens with two observations per condition:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; ab &amp;lt;- rbind(a, b)&#xA;&amp;gt; c(tapply(ab$count, ab$state, as.vector))&#xA;$RSTO&#xA;[1] 199665  31956&#xA;&#xA;$RSTR&#xA;[1]  4147 11689&#xA;&#xA;$S1&#xA;[1] 31274  6702&#xA;&#xA;$S2&#xA;[1]    1 2838&#xA;&#xA;$S3&#xA;[1] 2522 6268&#xA;&#xA;$SF&#xA;[1] 118009 672561&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here, the function provided for &lt;code&gt;tapply()&lt;/code&gt; returns a vector with two elements for each level of the factor. The result then is a list with as many components as you have factor levels, each component containing the result of the supplied function for that level. All components have equal length, hence &lt;code&gt;data.frame(&amp;lt;list&amp;gt;)&lt;/code&gt; returns a data frame where each component becomes a variable (data frames are special lists with components of equal length).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, another question would be if the behavior of &lt;code&gt;unstack()&lt;/code&gt; is sensible in a case like yours. If this is of practical importance, you can add an id-variable and use &lt;code&gt;reshape()&lt;/code&gt; instead.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; a$id &amp;lt;- 1&#xA;&amp;gt; reshape(a, direction=&quot;wide&quot;, v.names=&quot;count&quot;, timevar=&quot;state&quot;, idvar=&quot;id&quot;)&#xA;  id count.RSTO count.RSTR count.S1 count.S2 count.S3 count.SF&#xA;1  1     199665       4147    31274        1     2522   118009&#xA;&#xA;&amp;gt; ab$id &amp;lt;- rep(1:2, each=nrow(a))&#xA;&amp;gt; reshape(ab, direction=&quot;wide&quot;, v.names=&quot;count&quot;, timevar=&quot;state&quot;, idvar=&quot;id&quot;)&#xA;   id count.RSTO count.RSTR count.S1 count.S2 count.S3 count.SF&#xA;1   1     199665       4147    31274        1     2522   118009&#xA;11  2      31956      11689     6702     2838     6268   672561&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1909" LastEditorUserId="1909" LastEditorDisplayName="" LastEditDate="2011-03-23T21:44:51.900" LastActivityDate="2011-03-23T21:44:51.900" CommentCount="1" />
  <row Id="8698" PostTypeId="2" ParentId="8692" CreationDate="2011-03-23T21:18:52.463" Score="0" ViewCount="" Body="&lt;p&gt;Caracal has answered the question. Here are some additional pieces of information.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Try this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(stats)&#xA;formula(PlantGrowth)&#xA;x = PlantGrowth[c(1,11,21),]&#xA;y = PlantGrowth[c(1,2,11,12,21,22),]&#xA;x&#xA;unstack(x)&#xA;y&#xA;unstack(y)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The output:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;     x&#xA;   weight group&#xA;1    4.17  ctrl&#xA;11   4.81  trt1&#xA;21   6.31  trt2&#xA;&amp;gt;     unstack(x)&#xA;      res&#xA;ctrl 4.17&#xA;trt1 4.81&#xA;trt2 6.31&#xA;&amp;gt;     y&#xA;   weight group&#xA;1    4.17  ctrl&#xA;2    5.58  ctrl&#xA;11   4.81  trt1&#xA;12   4.17  trt1&#xA;21   6.31  trt2&#xA;22   5.12  trt2&#xA;&amp;gt;     unstack(y)&#xA;  ctrl trt1 trt2&#xA;1 4.17 4.81 6.31&#xA;2 5.58 4.17 5.12&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;We just have to define a new unstack function:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;NewUnstack &amp;lt;- function(x)&#xA;{&#xA; df = unstack(x)&#xA; if(dim(df)[2]==1)&#xA; {&#xA;  df = t(df)&#xA; }&#xA; return(df)&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here we go:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; NewUnstack(x)&#xA;    ctrl trt1 trt2&#xA;res 4.17 4.81 6.31&#xA;&amp;gt; NewUnstack(y)&#xA;  ctrl trt1 trt2&#xA;1 4.17 4.81 6.31&#xA;2 5.58 4.17 5.12&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1351" LastEditorUserId="1351" LastEditorDisplayName="" LastEditDate="2011-03-23T21:30:36.963" LastActivityDate="2011-03-23T21:30:36.963" />
  <row Id="8699" PostTypeId="2" ParentId="8669" CreationDate="2011-03-23T21:45:18.903" Score="0" ViewCount="" Body="&lt;p&gt;Are you sure there's a Z variable?    Since you have an equation of X's versus Y's, isn't one of the X's or Y's the dependent variable?  For example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$X_1 = - newa_2X_2... - newa_{11}X_{11} + newb_0 + newb_1Y_1 + newb_2Y_2 + new b_3Y_3 + newb_4Y_4$$&lt;/p&gt;&#xA;" OwnerUserId="2775" LastActivityDate="2011-03-23T21:45:18.903" CommentCount="4" />
  <row Id="8700" PostTypeId="2" ParentId="8690" CreationDate="2011-03-23T22:14:01.047" Score="1" ViewCount="" Body="&lt;p&gt;Maybe I'm missing something here, but couldn't you rearrange the equation and do a  logit or probit analysis on gamma?&lt;/p&gt;&#xA;" OwnerUserId="2775" LastEditorUserId="2775" LastEditorDisplayName="" LastEditDate="2011-03-23T22:28:41.923" LastActivityDate="2011-03-23T22:28:41.923" />
  <row Id="8701" PostTypeId="2" ParentId="7318" CreationDate="2011-03-23T22:47:02.427" Score="1" ViewCount="" Body="&lt;p&gt;Non-parametric tests are likely to be less powerful than parametric tests and thus require a larger sample size. This is annoying because if you had a large sample size, sample means would be approximately normally distributed by the central limit theorem, and you thus wouldn't need non-parametric tests.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Look at generalized linear models, of which least squares and Poisson are special cases. I've never found a text that explains this particularly well; try talking to someone about it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Look at non-parametric methods if you feel like it, but I have a hunch that they won't help you much in this case unless you're using ordinal data or a large set of very bizarrely distributed data.&lt;/p&gt;&#xA;" OwnerUserId="3874" LastActivityDate="2011-03-23T22:47:02.427" CommentCount="2" />
  <row Id="8702" PostTypeId="1" CreationDate="2011-03-23T23:00:11.000" Score="1" ViewCount="69" Body="&lt;p&gt;I am examining the difference between a physical feature of different species of animals. Due to the nature of my experiments I'm using a nonlinear mixed model with the following setup:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;lme(log10(feature) ~ log10(Body.mass) + factor(Trial.Number), random = ~1 | IndividualID, data=animals, subset=Frfactor==&quot;low&quot;, na.action=na.omit )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where &lt;code&gt;subset=Frfactor==&quot;low&quot;&lt;/code&gt; refers to a specific speed range that I'm interested in.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I get great results which I'm happy about. But now I want to see how species affects my feature. Since the same conditions apply (tons of repeated effects) I've kept the lme and changed the structure to: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;lme(log10(feature) ~ specfactor + factor(Trial.Number), random = ~1 | IndividualID, data=animals, subset=Frfactor==&quot;low&quot;, na.action=na.omit )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where specfactor lists the names of the species. Looking at the p values it looks like these species are not significantly different from the intercept (which is specfactorserval). However when I create a boxplot, it certainly looks like there are some big interspecies differences!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I guess because the lme is a test for regressions, it doesn't really make sense to use when comparing the feature against categorical variables. But I still need to account for repeated effects. My question is if there's a better way to test for significance between species using the boxplot? I need the usual- p-values, confidence intervals. The &quot;list&quot; command seems to fall short of such comparisons. I don't know if a t-test would cut it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS I originally posted an image of my test results and of the boxplot, but I'm too new of a user to be allowed....&lt;/p&gt;&#xA;" OwnerUserId="3761" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-03-24T07:07:50.403" LastActivityDate="2011-03-24T07:07:50.403" Title="Boxplot with glme" Tags="&lt;r&gt;&lt;repeated-measures&gt;&lt;boxplot&gt;" CommentCount="2" />
  <row Id="8703" PostTypeId="2" ParentId="8562" CreationDate="2011-03-23T23:01:21.810" Score="2" ViewCount="" Body="&lt;p&gt;You might use a nested design (and a hierarchical/multilevel model). The top level would be Baseline v Non-Baseline, and the Non-Baseline would include your 2^3 factorial design.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I unfortunately can't post my pretty picture of the nesting structure because I don't have 10 reputation yet.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd have to read a bit to remember exactly how to do this, but I'm pretty sure that this is a reasonable way of doing it.&lt;/p&gt;&#xA;" OwnerUserId="3874" LastActivityDate="2011-03-23T23:01:21.810" CommentCount="1" />
  <row Id="8704" PostTypeId="1" CreationDate="2011-03-24T00:05:13.620" Score="4" ViewCount="65" Body="&lt;p&gt;The title defines the question. May be the concept would do...like how to go about it? Thanks.&lt;/p&gt;&#xA;" OwnerUserId="1102" LastEditorUserId="2645" LastEditorDisplayName="" LastEditDate="2011-03-24T08:52:26.383" LastActivityDate="2011-03-24T14:24:39.583" Title="Given pdf of $I$ and $R$ (both $I$ and $R$ are independent RV's), how to find pdf of $W =I^2\cdot R$?" Tags="&lt;homework&gt;&lt;independence&gt;&lt;pdf&gt;" AnswerCount="1" CommentCount="4" />
  <row Id="8706" PostTypeId="2" ParentId="8689" CreationDate="2011-03-24T01:32:10.797" Score="6" ViewCount="" Body="&lt;p&gt;Linear refers to the relationship between the parameters that you are estimating (e.g., $\beta$) and the outcome (e.g., $y_i$). Hence, $y=e^x\beta+\epsilon$ is linear, but $y=e^\beta x + \epsilon$ is not. A linear model means that your estimate of your parameter vector can be written $\hat{\beta} = \sum_i{w_iy_i}$, where the $\{w_i\}$ are weights determined by your estimation procedure. Linear models can be solved algebraically in closed form, while many non-linear models need to be solved by numerical maximization using a computer.&lt;/p&gt;&#xA;" OwnerUserId="401" LastActivityDate="2011-03-24T01:32:10.797" CommentCount="3" />
  <row Id="8708" PostTypeId="1" CreationDate="2011-03-24T01:53:05.040" Score="2" ViewCount="75" Body="&lt;p&gt;Eg: In a list of three items (&quot;1&quot;, &quot;2&quot;, &quot;3&quot;) the user is asked which item is higher, 1 or 2. They select 1. They are then asked which is higher, 2 or 3. They select 3. The system then  reasons: &quot;1 is higher than 2. If 3 is lower than 2 it must logically be lower than 1 also.&quot; etc.&#xA;(So that you can sort a list of items with the fewest theoretical comparisons possible.)&#xA;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="3877" LastActivityDate="2011-03-24T04:16:50.900" ClosedDate="2011-03-24T21:11:34.533" Title="Is there a program or app that can sort items using manual choices and deductive logic?" Tags="&lt;algorithms&gt;" AnswerCount="1" CommentCount="4" />
  <row Id="8709" PostTypeId="2" ParentId="8633" CreationDate="2011-03-24T02:07:04.620" Score="0" ViewCount="" Body="&lt;p&gt;If you're interested in parsing a measure of fatigue from your RT data for use as a covariate, then I'd suggest computing the slope of RT as a function of time. An additional measure of &quot;noisiness&quot; might be the variance of RT once the effect of time has been removed.&lt;/p&gt;&#xA;" OwnerUserId="364" LastActivityDate="2011-03-24T02:07:04.620" />
  <row Id="8710" PostTypeId="1" AcceptedAnswerId="8730" CreationDate="2011-03-24T02:28:44.653" Score="3" ViewCount="62" Body="&lt;p&gt;I have 2 alternative methods to solve a problem, and I was just wondering what people who know the math better than I think, and if there is a better method to use for this type of problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem: I have a list of lat/lon positions and a value for the time interval between position updates and wish to find the SOG (Speed Over Ground). However there is some uncertainty in the exact value for each time interval - as the data is retrieved over the internet, even though in a program it would (for example) be set to request an update every 60 seconds. The data comes from an ocean racing yacht simulator, so SOG will also change with varying conditions and also with changes in course (relative to wind).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One approach is to weight new updates proportional to the length of time that interval and multiply into our current best SOG estimate. Something like this (T in seconds): &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\text{speed} = \text{distance} / \Delta T$$ &#xA;$$d = 1 - e^{-\Delta T}$$&#xA;$$\text{sog} = d * \text{speed} + (1 - d) * \text{sog}$$  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;And this works ok, especially if the speed is relatively consistent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Prior to seeing this method, the one I had come up with to estimate SOG is this:&lt;br&gt;&#xA;Maintain a list of the $N$ most previous {timestamp, lat, lon} tuples.&lt;br&gt;&#xA;On each update, compute the harmonic mean of all possible sub intervals from $0, N-1$:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ S = N(N+1)/2 $$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ \text{sog estimate} = S / \sum_{i=0,j=i+1}^{N-1} \text{time}_{ij} / \text{distance}_{ij}  $$  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then keep a list of the $M$ most previous estimates and again take the harmonic mean. $N$ &amp;amp; $M$ should not be too large. And obviously if there are not yet $N$ tuples to sum sub-intervals over, just use however many are available. Since the updates occur each 60 seconds, a value of 10-15 seems appropriate for both $N$ &amp;amp; $M$. One possible improvement I can think of is to weight each sub interval similar to method 1. &#xA;This second method seems to be more accurate than the first, although can sometimes take longer to converge on a reasonably accurate estimate - usually approximately N updates are required before it becomes accurate for any practical use.&lt;br&gt;&#xA;It also seems to handle changes in 'real' SOG better, ie slight accelerations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given that SOG is rarely consistent over say more than 15 or 20 minutes, ie from an increase in wind strength, or change of angle to one with higher/lower boatspeed etc, and also that direction can change (again, with a change in speed), what would be the best algorithm to compute the SOG from uncertain time intervals and change in position. Also, is there a possible way to estimate the error in estimation? Even possibly correct for this error term once computed?&#xA;I should mention all the data I have available: heading, lattitude and longitude, and I keep a timestamp from the beginning of each update request.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to be able to compute this speed as accurately as possible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance for any assistance, and forgive my poor use of MathJaX.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: I get Math processing errors from MathJaX that works fine on math.stackexchange? Notice a lot of them in either questions to?? So reposted plain text math.&lt;/p&gt;&#xA;" OwnerUserId="3878" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-24T21:42:40.373" LastActivityDate="2011-03-24T21:42:40.373" Title="Estimating speed from position updates with uncertain time intervals" Tags="&lt;regression&gt;&lt;estimation&gt;&lt;functional-data-analysis&gt;&lt;measurement-error&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="8712" PostTypeId="2" ParentId="8708" CreationDate="2011-03-24T03:15:51.210" Score="-1" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://www.wolframalpha.com/input/?i=max%281%2C2%2C3%29&quot; rel=&quot;nofollow&quot;&gt;http://www.wolframalpha.com/input/?i=max%281%2C2%2C3%29&lt;/a&gt; or &lt;a href=&quot;http://www.wolframalpha.com/input/?i=max%28max%281%2C2%29%2C3%29&quot; rel=&quot;nofollow&quot;&gt;http://www.wolframalpha.com/input/?i=max%28max%281%2C2%29%2C3%29&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="3484" LastEditorUserId="3484" LastEditorDisplayName="" LastEditDate="2011-03-24T04:16:50.900" LastActivityDate="2011-03-24T04:16:50.900" CommentCount="1" />
  <row Id="8713" PostTypeId="2" ParentId="8689" CreationDate="2011-03-24T04:03:07.070" Score="-2" ViewCount="" Body="&lt;p&gt;I think the simplest way to put this is to say that a linear regression constructs a best fit line to your data, something of the form $y = bx + c$, as apposed to something polynomial, ie $y = ax^2 + bx + c$ or higher order (including higher powers of $x$). $\{a, b, c: Constant\}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another way to think of it is as a polynomial with all terms of higher order than 1 having coefficients of 0. ie you can see the 2 equations above are the same if $a = 0$.&lt;/p&gt;&#xA;" OwnerUserId="3878" LastActivityDate="2011-03-24T04:03:07.070" CommentCount="6" />
  <row Id="8714" PostTypeId="1" CreationDate="2011-03-24T05:30:59.240" Score="2" ViewCount="142" Body="&lt;p&gt;Hi. Is it possible to perform logarithmic regression on multiple variables with Excel? If I just have a single independent variable than it's very easy to do this using the best-fit line option (it lets me switch from linear to logarithmic). But this feature does not work for multiple variable regression and the regression feature under the Data Analysis plugin only seems to support linear multiple regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I have a table that has 3 columns containing 3 independent variables and 1 column with the corresponding dependent variable (outcome). I'm pretty sure there's a logarithmic relationship, but I'm not sure how to use Excel to get the coefficients. Thanks!&lt;/p&gt;&#xA;" OwnerUserId="3881" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-24T16:10:25.313" LastActivityDate="2011-03-24T17:34:04.710" Title="Performing logarithmic multiple regression with Excel?" Tags="&lt;regression&gt;&lt;excel&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="8715" PostTypeId="2" ParentId="8714" CreationDate="2011-03-24T06:26:38.550" Score="6" ViewCount="" Body="&lt;p&gt;If by logarithmic regression you mean the model &lt;code&gt;log(y) = m1.x1 + m2.x2 + ... + b + (Error)&lt;/code&gt;, you can use &lt;code&gt;LOGEST&lt;/code&gt; and &lt;code&gt;GROWTH&lt;/code&gt; with multiple independent variables. Note that if you want the estimated coefficients &lt;code&gt;m1, m2, ..., b&lt;/code&gt; from &lt;code&gt;LOGEST&lt;/code&gt;, you'll have to enter the formula into multiple cells as an array. See Excel's online help for the steps required.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively, you can log-transform your dependent variable and use &lt;code&gt;LINEST&lt;/code&gt;/&lt;code&gt;TREND&lt;/code&gt; which does the same thing under the hood. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;ObWarning: Excel isn't the best regression package in the world. See, for example, McCullough &amp;amp; Heiser (2008), On the accuracy of statistical procedures in Microsoft Excel 2007, &lt;em&gt;Comp Stats &amp;amp; Data Analysis&lt;/em&gt; 52(10) pp.4570-4578.&lt;/p&gt;&#xA;" OwnerUserId="1569" LastEditorUserId="1569" LastEditorDisplayName="" LastEditDate="2011-03-24T06:37:46.187" LastActivityDate="2011-03-24T06:37:46.187" CommentCount="1" />
  <row Id="8716" PostTypeId="1" CreationDate="2011-03-24T08:39:25.643" Score="3" ViewCount="70" Body="&lt;p&gt;I have a basket of time series (stock prices). I want to find the N (fixed or not) time series that will best replicate the basket in the sense that combination of them will be best cointegrated with the basket.&lt;br&gt;&#xA;Beside using the N series that have the best cointegration scores (ADF test) and regressing these variables against the basket. Do you know of other/better methodologies for doing that?&lt;/p&gt;&#xA;" OwnerUserId="3362" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-03-24T18:21:05.500" LastActivityDate="2011-04-23T20:26:08.810" Title="Cointegration-based feature selection" Tags="&lt;cointegration&gt;" AnswerCount="0" FavoriteCount="1" />
  <row Id="8717" PostTypeId="1" CreationDate="2011-03-24T08:44:11.823" Score="2" ViewCount="68" Body="&lt;p&gt;Let $X_1, X_2, ..., X_n$ be a random sample from a distribution with p.d.f.,&#xA;$$f(x;\theta)=\theta^2xe^{-x\theta} ; 0&amp;lt;x&amp;lt;\infty, \theta&amp;gt;0$$ Obtain minimum variance unbiased estimator of $\theta$ and examine whether it is attained?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;MY WORK:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using MLE i have found the estimator for $\theta=\frac{2}{\bar{x}}$&#xA;Or as $$X\sim Gamma(2, \theta)$$So &#xA;$E(X)=2\theta$&#xA;$E(\frac{x}{2})=\theta$&#xA;so can i take $\frac {x}{2}$ as unbiased estimator of $\theta$.&#xA;I'm stuck and confused need some help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank u.&lt;/p&gt;&#xA;" OwnerUserId="3846" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-24T16:11:25.250" LastActivityDate="2011-03-24T16:11:25.250" Title="What will be minimum variance unbiased estimator?" Tags="&lt;probability&gt;&lt;estimation&gt;" CommentCount="7" FavoriteCount="1" />
  <row Id="8718" PostTypeId="1" CreationDate="2011-03-24T09:06:46.867" Score="2" ViewCount="270" Body="&lt;p&gt;I've developed a logit model to be applied to six different sets of cross-sectional data. What I'm trying to uncover is whether there are changes in the substantive effect of a given independent variable (IV) on the dependent variable (DV) controlling for other explanations at different times and across time. My question is: how do I assess increased/decreased size in the association between the IV and DV? Can I simply look at the different magnitudes (sizes) of the coefficients across the models or do I need to go through some other process? If I need to do something else, what is it and can it be done/how do I do it in SPSS?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, within a single model, can I compare the relative size of independent variables based on unstandardised scores if all are coded 0-1 or do I need to convert them to standardised scores? Are there problems involved with standardised scores?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="3883" LastActivityDate="2011-03-25T16:37:41.953" Title="Comparing logistic regression coefficients across models?" Tags="&lt;logistic-regression&gt;" AnswerCount="3" CommentCount="1" />
  <row Id="8719" PostTypeId="2" ParentId="8718" CreationDate="2011-03-24T10:20:54.480" Score="3" ViewCount="" Body="&lt;p&gt;Are there changes across data sets? I can answer that without seeing the data!  Yes.  There are.  How big are they?  That's key.  For me, the way to see is by looking.  You will have odds ratios for each independent variable for each data set - are they different in ways people would find interesting?  Now, it's true each will have a standard error and so on, and there are probably ways to see if they are statistically significantly different from each other, but is that really an interesting question?  If it is, then one way to test it easily with software would be to combine all the studies, and include &quot;study&quot; as another independent variable.  You could then even test interactions, if you wanted.  Whether you want to do this depends on your substantive questions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As to comparing variables within a model, the main problem with standardized scores is that they are standardized on your particular sample.  So, the parameter estimates and so on are then in terms of standard deviations of the variables in your particular sample.  Even if  your sample is truly a random sample from some population, it will have (slightly) different standard deviations from other random samples.  This makes things confusing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The other problem is what the question of &quot;relative size&quot; even means.  If your IVs are things that are well-understood, you can compare the ORs across ranges that mean something.  &lt;/p&gt;&#xA;" OwnerUserId="686" LastActivityDate="2011-03-24T10:20:54.480" />
  <row Id="8720" PostTypeId="2" ParentId="8691" CreationDate="2011-03-24T10:34:54.860" Score="0" ViewCount="" Body="&lt;p&gt;I like what @whuber said, but one additional note - if your scores are percentages, they aren't really ranks but ratings.  Ranks have integer values, except for ties (which are handled in various ways).&lt;/p&gt;&#xA;" OwnerUserId="686" LastActivityDate="2011-03-24T10:34:54.860" />
  <row Id="8721" PostTypeId="2" ParentId="8704" CreationDate="2011-03-24T10:46:20.527" Score="3" ViewCount="" Body="&lt;p&gt;Maybe you know that if $X$ and $Y$ are independent then the pdf of $X+Y$ is given by a convolution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can generalize the idea of a &lt;strong&gt;convolution to another group&lt;/strong&gt; than $(+,\mathbb{R})$ and the idea that pdf of $X\cdot Y$ (where $\cdot$ is the operation of the group) is given by the convolution is still valid. For the convolution on group see &lt;a href=&quot;http://en.wikipedia.org/wiki/Convolution#Convolutions_on_groups&quot; rel=&quot;nofollow&quot;&gt;wikipedia&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the group of multiplication $(\cdot,\mathbb{R}_+)$, the (left)-haar measure is $dt/t$ (see &lt;a href=&quot;http://en.wikipedia.org/wiki/Haar_measure#Examples&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;), and this gives (if $f_X$ and $f_Y$ are densities of $X$ and $Y$):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$f_{X\cdot Y}(z)=\int_{\mathbb{R}_+} f_X(t)f_Y(z/t)dt/t$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;the density of the product $X\cdot Y$. It is straigthforward to derive the density of $I^2\cdot R$ from that.  &lt;/p&gt;&#xA;" OwnerUserId="223" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-03-24T14:24:39.583" LastActivityDate="2011-03-24T14:24:39.583" CommentCount="5" />
  <row Id="8722" PostTypeId="1" CreationDate="2011-03-24T10:57:34.487" Score="0" ViewCount="64" Body="&lt;p&gt;Given $Y = \beta_0 + \beta_1 \cdot X + \varepsilon$, show that:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\beta_1 = Cov (X,Y) / Var (X) = Corr (X,Y)/ \sqrt{Var (Y)/Var(X)}$&lt;/p&gt;&#xA;" OwnerUserId="3062" LastEditorUserId="2645" LastEditorDisplayName="" LastEditDate="2011-03-24T12:44:51.057" LastActivityDate="2011-03-24T12:44:51.057" ClosedDate="2011-03-24T14:03:01.217" Title="Let $X$ and $\varepsilon$ be two independent random variables, where $E(\varepsilon) = 0$, Show the following..." Tags="&lt;regression&gt;&lt;homework&gt;" CommentCount="3" />
  <row Id="8723" PostTypeId="2" ParentId="6920" CreationDate="2011-03-24T10:57:37.647" Score="0" ViewCount="" Body="&lt;p&gt;You can always just perform gradient descent on the sum of squares cost $E$ wrt  the parameters of your model $W$. Just take the gradient of it but don't go for the closed form solution but only for the search direction instead. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let $E(i; W)$ be the cost of the i'th training sample given the parameters $W$. Your update for the j'th parameter is then &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$W_{j} \leftarrow W_j + \alpha \frac{\partial{E(i; W)}}{\partial{W_j}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\alpha$ is a step rate, which you should pick via cross validation or good measure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is very efficient and the way neural networks are typically trained. You can process even lots of samples in parallel (say, a 100 or so) efficiently.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course more sophisticated optimization algorithms (momentum, conjugate gradient, ...) can be applied.&lt;/p&gt;&#xA;" OwnerUserId="2860" LastActivityDate="2011-03-24T10:57:37.647" />
  <row Id="8724" PostTypeId="1" AcceptedAnswerId="8727" CreationDate="2011-03-24T11:04:28.350" Score="2" ViewCount="64" Body="&lt;p&gt;I have a database with many attributes. I would like to know which attributes has the minimum variation in the data. Is there some standard technique? It should be like clustering without split records in clusters. I would like to know what the records in particular cluster have in common. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was going to compute the mean ($\bar{x}$) and st.d. ($s$) for each continuous attribute $x$. After computing the coefficient of variation $CV=\frac{s}{\bar{x}}$ I would say that attributes with $CV\leq0.1$ are the similar ones. For categorical ones I would choose attributes with more than $90\%$ relative frequency for the mode.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there some standard technique?&lt;/p&gt;&#xA;" OwnerUserId="2719" LastActivityDate="2011-03-24T12:42:28.827" Title="Exploring data attributes" Tags="&lt;clustering&gt;" AnswerCount="1" />
  <row Id="8725" PostTypeId="1" CreationDate="2011-03-24T11:50:53.710" Score="2" ViewCount="98" Body="&lt;p&gt;I tried to use the Kernel Density plot method from Racine's np package for my own data, but somehow ended up with different type of plots and I have no idea what the difference is between my data and the example data provided by the package. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I used the Italy GDP example that is provided in the np package and described in &lt;a href=&quot;http://socserv.mcmaster.ca/racine/ECO0301.pdf&quot; rel=&quot;nofollow&quot;&gt;Racine's primer&lt;/a&gt;. The example itself worked well and I ended up with a nice 3D plot. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I used it with my own data I just got two 2D plots instead of one 3D plot. Why is that?&#xA;Do I miss something content wise (that's what I think and it's the reason being why I guess this is rather a CV than a SO question) or is it a syntax problem?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# get a bandwith object&#xA;  bw2 &amp;lt;- npcdensbw(formula=mydf$values~ordered(mydf$datefield), tol=.1, ftol=.1) &#xA;  str(mydf)&#xA;# returns&#xA;  data.frame':  780 obs. of  2 variables:&#xA;  $ datefield: Ord.factor w/ 104 levels &amp;quot;1984-04-01&amp;quot;&amp;lt;&amp;quot;1984-07-01&amp;quot;&amp;lt;..: 1 1 1 1 1 1 1 1 1     1 ...&#xD;&#xA;      $ values : num  50.7 58.5 56.1 55.5 62.7 ...&#xA;tables(mydf$datefield)&#xA;# shows that I have 13 entries per quarter.&#xA;&#xA;quartz()&#xA;plot(bw2)&#xA;npplot(bw2)&#xA;# same 2D plot for both :(&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;all of these properties are exactly in line with the example data. Still I get only 2D plots instead of a 3D plot. I do have three dimensions: quarters,values and conditional density. What's wrong?&lt;/p&gt;&#xA;" OwnerUserId="704" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-24T16:11:44.437" LastActivityDate="2011-03-24T16:11:44.437" Title="Conditional kernel density plot with R's np package" Tags="&lt;r&gt;&lt;nonparametric&gt;&lt;conditional-probability&gt;" />
  <row Id="8726" PostTypeId="2" ParentId="8663" CreationDate="2011-03-24T12:20:57.903" Score="-1" ViewCount="" Body="&lt;p&gt;$p-value=Pr(E||H)$ where E is the &quot;data at least as extreme as what was observed&quot; event, and $H$ is the hypothesis, usually of the form &quot;some set of the parameters are zero&quot;.  I have used the double lines $||$ to indicate that it is not a conditional probability per se, rather a probability based on the the assumption that the null hypothesis is true.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You have not explicitly said with the null hypothesis was is both examples.  If you take the time to carefully state exactly what the null hypothesis is in each case, you may find that it makes perfect sense for them to give different p-values.  You may also find that the event $E$ is not the same in the two cases either.  If the &quot;extreme event&quot; is different, then you would in general expect to see different p-values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;p-values are usually used to &quot;reject&quot; models rather than compare models.  At least that's how I use them.  A small p-value usually means (at least to me anyways) &quot;there is a better explanation for the observed data than the null hypothesis&quot;.  I usually take the size of this p-value to be a rough measure of &quot;how much easier&quot; such an explanation would be to find.  For me, p-values of the size $0.4$ and $0.8$ just indicate that the data is fairly consistent with either model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not so sure that it says that the &quot;binned&quot; model is better than the &quot;line&quot; model.  This is because there is no explicit comparison or reference to the other model made when either of these two p-values are calculated.  The reason I say this is that, generally there is an implicit class of alternative hypothesis to which a p-value refers.  If both models are in this implicit class when calculating the p-value based on the other, then it is does indicate that the binned model is better.  But determining this implicit class is not always straight forward, unless the p-value is based on a sufficient statistic for the null hypothesis.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-03-24T12:20:57.903" CommentCount="1" />
  <row Id="8727" PostTypeId="2" ParentId="8724" CreationDate="2011-03-24T12:42:28.827" Score="2" ViewCount="" Body="&lt;p&gt;It reminds me of what is implemented in the &lt;a href=&quot;http://caret.r-forge.r-project.org/&quot; rel=&quot;nofollow&quot;&gt;caret&lt;/a&gt; package for data pre-processing. It is fully described in one of the accompanying vignette, namely &lt;a href=&quot;http://cran.r-project.org/web/packages/caret/vignettes/caretMisc.pdf&quot; rel=&quot;nofollow&quot;&gt;Data Sets and Miscellaneous Functions in the caret Package&lt;/a&gt;. What is actually done is to identify predictors that have low variance in the full dataset, as you described, whether it be a continuous or a categorical feature. They compute:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;the frequency of the most prevalent value over the second most frequent value (termed &quot;frequency ratio&quot;),&lt;/li&gt;&#xA;&lt;li&gt;the proportion of unique values (subject-wise),&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;considering that&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;If the frequency ratio is less than a&#xA;  &lt;strong&gt;pre–specified threshold&lt;/strong&gt; and the unique&#xA;  value percentage is less than a&#xA;  threshold, we might consider a&#xA;  predictor to be near zero–variance. (p. 5, emphasis is mine)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The rationale is that near-zero variance predictors may have exact zero variance when using cross-validation, or induce model instability. They also address the problem of collinearity, but then this really is a matter of statistical modeling (some models, like classical regression models, don't accommodate well correlated predictors because it will inflates standard error of regression coefficients; others don't care about that).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Besides screening those low informative predictors, you can also use a hierarchical clustering method (by variables, not by individuals) to see how it goes. This is often used for studying missing data patterns (i.e., where we are interested in examining which variables are consistently showing an increased number of missing responses across all samples, or a particular subgroup).&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-03-24T12:42:28.827" CommentCount="6" />
  <row Id="8728" PostTypeId="2" ParentId="8695" CreationDate="2011-03-24T13:52:21.373" Score="2" ViewCount="" Body="&lt;p&gt;The following is shamelessly extracted from the following link. &#xA;&lt;a href=&quot;http://findarticles.com/p/articles/mi_qa3713/is_199704/ai_n8770931/pg_10/&quot; rel=&quot;nofollow&quot;&gt;http://findarticles.com/p/articles/mi_qa3713/is_199704/ai_n8770931/pg_10/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{shamelesscopyandpaste}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, the &quot;average variance extracted&quot; measures the amount of variance that is captured by the construct in relation to the amount of variance due to measurement error and can be calculated using the following formula: (summation of squared factor loadings)/(summation of squared factor loadings) (summation of error variances) (Fornell &amp;amp; Larcker). If the average variance extracted is less than .50, then the variance due to measurement error is greater than the variance due to the construct. In this case, the convergent validity of the construct is questionable. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;\end{shamelesscopyandpaste}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I haven't used SPSS in some time, and i don't remember seeing an option to perform these calculations, but you can certainly do it using the syntax. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.ats.ucla.edu/stat/spss/seminars/spss_syntax08/default08.htm&quot; rel=&quot;nofollow&quot;&gt;http://www.ats.ucla.edu/stat/spss/seminars/spss_syntax08/default08.htm&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.ats.ucla.edu/stat/spss/seminars/spss_syntax08/default08_part2.htm&quot; rel=&quot;nofollow&quot;&gt;http://www.ats.ucla.edu/stat/spss/seminars/spss_syntax08/default08_part2.htm&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The two links above give you an introduction to spss syntax.&#xA;What i would do from here is examine the syntax spss gives you when you perform an FA, and use the variables named to compute the average variance explained.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sorry for the lack of a definitive answer, but i felt that some response was definitely better than none. &lt;/p&gt;&#xA;" OwnerUserId="656" LastActivityDate="2011-03-24T13:52:21.373" CommentCount="1" />
  <row Id="8729" PostTypeId="1" CreationDate="2011-03-24T14:51:27.800" Score="2" ViewCount="83" Body="&lt;p&gt;I'm trying to gain a better understanding of kmeans clustering and am still unclear about colinearity and scaling of data. To explore colinearity, I made a plot of all five variables that I am considering shown in the figure below, along with a correlation calculation.&#xA;&lt;img src=&quot;http://i.imgur.com/W5MZJ.jpg&quot; alt=&quot;colinearity&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I started off with a larger number of parameters, and excluded any that had a correlation higher than 0.6 (an assumption I made). The five I choose to include are shown in this diagram.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, I scaled the date using the &lt;code&gt;R&lt;/code&gt; function &lt;code&gt;scale(x)&lt;/code&gt; before applying the &lt;code&gt;kmeans()&lt;/code&gt; function. However, I'm not sure whether &lt;code&gt;center = TRUE&lt;/code&gt; and &lt;code&gt;scale = TRUE&lt;/code&gt; should also be included as I don't understand the differences that these arguments make. (The &lt;code&gt;scale()&lt;/code&gt; description is given as &lt;code&gt;scale(x, center = TRUE, scale = TRUE)&lt;/code&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is the process that I describe an appropriate way of identifying clusters?&lt;/p&gt;&#xA;" OwnerUserId="2635" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-24T16:12:33.673" LastActivityDate="2011-03-24T16:12:33.673" Title="Colinearity and scaling when using k-means" Tags="&lt;r&gt;&lt;clustering&gt;" CommentCount="7" FavoriteCount="2" />
  <row Id="8730" PostTypeId="2" ParentId="8710" CreationDate="2011-03-24T15:33:25.113" Score="3" ViewCount="" Body="&lt;p&gt;Because you trust the GPS positions (and therefore the distances computed from them) but the times have errors, &lt;em&gt;regress the times against the cumulative distances.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To account for acceleration and deceleration, consider a model of the form&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\text{Time} = t = \beta_0 + \beta_1 X + \beta_2 X^2 + \varepsilon$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $X$ is distance and $\varepsilon$ represents the time errors.  After fitting this you can estimate $\widehat{dt/dX} = \hat{\beta}_1 + 2 \hat{\beta}_2 X$, whence&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\text{Speed estimate} = \widehat{dX/dt} = 1/\widehat{dt/dX} = \frac{1}{\hat{\beta}_1 + 2 \hat{\beta}_2 X}.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This will likely work best for times near the middle of your dataset.  Thus, for instance, you could maintain a circular buffer of $k$ observations which at any moment would span times $t_1, t_2, \ldots, t_k$ and use them to estimate speeds near the mean position $\frac{1}{k}\sum_{i=1}^{k}X_k$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an example I generated distances according to the formula $X(t) = 15 + i - i^2/25$ for times $t=1, 2, \ldots, 10$.  This represents a period of uniform deceleration from a unit speed.  Note that it is &lt;em&gt;not&lt;/em&gt; the same as the postulated model: the variation in time is given by the quadratic formula.  I then varied those times by independent standard normal variates.  Compared to the nominal sampling interval this is a &lt;em&gt;huge&lt;/em&gt; error: you can't even be sure that two successive times are actually in the right order.  This is a fairly severe test of the method.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a plot of a typical realization, with the red dots showing the true values, the blue ones showing the observed values (&lt;em&gt;i.e.&lt;/em&gt;, the true values with time jittering), and the ordinary least squares fit of the time.  I used $k=10$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/wBAls.png&quot; alt=&quot;Plot 1&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that the time measurements are so awful, the boat seems to reverse its course several times, especially near times 2 and 4.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unlike many of the realizations I looked at, this fit doesn't look terribly good in the center: the line departs from the red dots.  Be that as it may, let's look at how the estimated and actual speeds varied during this period:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/x7qPn.png&quot; alt=&quot;Speed vs. position&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Note that you can estimate a speed at &lt;em&gt;any&lt;/em&gt; time, not just a measured time, because the fitted coefficients give a formula for the speed.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this plot of speed versus &lt;em&gt;position&lt;/em&gt; (not time), the blue curve is the estimate and the red curve is the actual speed.  Obviously they are not very close near the ends, &lt;em&gt;but for the middle times the estimate is excellent.&lt;/em&gt;  (The middle times are around positions 19-20, as the first plot indicates.)  Note that you would run into severe difficulties estimating speeds directly from the sequence of positions, because (due to the errors in the times) the boat seems to be jumping around forwards and backwards.  If you treat the backwards jumps as true reversals you would grossly overestimate the speeds, but if you treat them as &lt;em&gt;negative&lt;/em&gt; speeds you would run into severe problems when averaging.  &lt;strong&gt;The moral here&lt;/strong&gt; is that it's best to fit a model of position versus time and only then attempt to estimate speeds; don't attempt to estimate speeds directly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The larger $k$ is, the more accurate you can expect to be.  Use the machinery of OLS to estimate the &lt;a href=&quot;http://www.sas.com/resources/whitepaper/wp_4430.pdf&quot; rel=&quot;nofollow&quot;&gt;prediction error&lt;/a&gt; of the time for any desired position; from that you can &lt;a href=&quot;http://stats.stackexchange.com/questions/tagged/error-propagation&quot;&gt;propagate the error&lt;/a&gt; to the speed estimate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The speed estimates might jump around a little due to the addition of new values and dropping of old values as you go along.  You could get fancier and use weighted regression, decreasing the weights smoothly to zero near the extreme positions.  In doing so, a plot of estimated speeds versus position would be a little smoother.  (This technique is akin to the recently popularized &lt;a href=&quot;http://ncg.nuim.ie/ncg/GWR/&quot; rel=&quot;nofollow&quot;&gt;&quot;Geographically Weighted Regression&quot;&lt;/a&gt; of Fotheringham, Charlton, and Brunsdon.)&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-03-24T15:33:25.113" CommentCount="1" />
  <row Id="8732" PostTypeId="1" CreationDate="2011-03-24T18:26:43.530" Score="7" ViewCount="173" Body="&lt;p&gt;I am new to time series analysis, and would appreciate any suggestions on how best to approach the following time-series regression problem: I have hourly temperature measurements at approximately 20 locations across one site over three years, along with static ancillary information (slope, elevation, aspect, canopy cover). The site is several hectares in size, and the temperature recording devices are spread across the site along a couple of transects, at ~20-50 m intervals. About 1 km away, I have hourly data from a weather station, which also provides measurements of wind speed, wind direction, humidity, solar illumination, etc.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to be able to predict the temperature (min,max,mean) at the site (in general) using only the data from the weatherstation; it is in place semi-permanently, whereas the temperature recorders at the site were only in place for 3 years. So in essence I have multiple independent variables (temperature, humidity, wind, etc) at one location (the weatherstation), but a single dependent variable (temperature) at multiple locations, each of which also has several time-invariant attributes: slope, elevation, aspect, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am most interested in predicting the daily lows and highs at the site in general, rather than hourly temperatures at each temperature recording location in the site. Although, those hourly predictions would certainly be of value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My initial approach has been to compute daily average, minimum, and maximums from the temperatures at the site, and use these as dependent variables in simple linear regressions, using the measurements available at the weatherstation as independent variables. This works reasonably well (R2 &gt; 0.50 with 2 predictors), but seems rather too simplistic for many reasons, and I imagine there must be more sophisticated (and powerful) ways to do this. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For one, I'm not doing anything explicit about the time-series nature of the daily values in the regression, and although the min or average temp from one day to the next may not be as correlated as it is from one hour to the next, I wonder about issues with the independence of these daily data (or certainly hourly, if I were trying to predict hourly temperatures).  Second, due to concerns with having multiple somewhat-correlated temperature measurements across the site (they are much more similar among themselves than any are to the weather station data), I am simply using the mean or min or max of all measurements across the site, versus including the data from each individual measurement location directly.  But this also prevents me from using the time-invariant ancillary information from each temperature measurement location (slope, elevation, aspect, canopy cover), which presumably will explain a good part of the differences in temperatures between locations at the site.  Third, due to concerns with the regression being dominated by the very strong diurnal cycle in temperatures, I'm only looking at daily values instead of hourly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestions on better ways to go about this (especially in R), or where to start looking, would be most appreciated! I realize there are alot of R packages that deal with time-series, but I'm having trouble finding the best place to start with this type of problem as none of the examples I've seen really seem to reflect the situation I'm trying to model here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Update: thinking about this a bit more, it is not clear to me whether time-series models are really appropriate here because I am not interested in predicting what will happen at some future specific point in time. Rather, I'm simply interested in how temperatures at the site are related to temperatures (and other environmental variables) at the weatherstation. I thought that perhaps time-series analysis would be of value because I was concerned that subsequent temperature measurements might not be sufficiently independent. Certainly, one hour's temperature depends a great deal on the previous hour, but the dependence is weaker for daily data. In either case, is the time-correlation/non-independence of time-series data a valid concern that should be addressed if one is not interested in a time-series prediction?&lt;/p&gt;&#xA;" OwnerUserId="3889" LastEditorUserId="3889" LastEditorDisplayName="" LastEditDate="2011-03-25T15:08:06.370" LastActivityDate="2011-03-30T23:37:04.383" Title="How to model time-series temperature data at multiple sites as a function of data at one site?" Tags="&lt;time-series&gt;&lt;regression&gt;&lt;multivariate-analysis&gt;&lt;spatio-temporal&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="4" />
  <row Id="8733" PostTypeId="1" CreationDate="2011-03-24T19:02:00.630" Score="2" ViewCount="78" Body="&lt;p&gt;Hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most repeated measures ANOVAs have time as the repeated measure; I was wondering about using a repeated measure that is not time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say we fed two groups of animals different diets. At the end of the experiment, we sample the tissues, and measure ~30 different compounds (e.g. different fatty acids [FA]).  Animals are sampled but once. Each FA is not independent of each other, as some of these compounds are converted from one form to another. As such, they are frequently moderately correlated with each other.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would it be fair to treat each compound as a repeated measure, with diet as a between subjects factor? Thus, the interaction between Diet X FA would tell me if the FA content differed among diets?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I note that in many papers, researchers would perform 25-30 separate ANOVAs; one on each compound. Yet, these compounds are not independent of each other, as they are each measured simultaneously on the same animals.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for any pointers.&lt;/p&gt;&#xA;" OwnerUserId="3886" LastEditorUserId="485" LastEditorDisplayName="" LastEditDate="2011-03-24T21:27:29.747" LastActivityDate="2011-03-24T21:27:29.747" Title="Repeated measures with correlated measures (not time)" Tags="&lt;pca&gt;&lt;repeated-measures&gt;&lt;manova&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="8734" PostTypeId="1" AcceptedAnswerId="8740" CreationDate="2011-03-24T19:15:34.247" Score="7" ViewCount="122" Body="&lt;p&gt;Is there a well founded rule for the number of significant figures to publish? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are some specific examples / questions:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Is there any way to relate the number of significant figures to the coefficient of variation? For example, if the estimate is 12.3 and the CV is 50%, does that mean that the information represented by '.3' approaches zero?  &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If a confidence interval has a range of orders of magnitude, should they still have the same number of significant figures, e.g.:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;12.3 (1.2, 123.4) vs 12 (1.2, 120) &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Should the number of significant figures in an error estimate be the same or less than the number of significant figures in a mean?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="1381" LastEditorUserId="1381" LastEditorDisplayName="" LastEditDate="2011-03-26T23:21:47.630" LastActivityDate="2011-03-26T23:21:47.630" Title="Number of significant figures to put in a table?" Tags="&lt;tables&gt;" AnswerCount="2" CommentCount="9" FavoriteCount="2" />
  <row Id="8736" PostTypeId="2" ParentId="8734" CreationDate="2011-03-24T19:47:02.827" Score="0" ViewCount="" Body="&lt;p&gt;I'd suggest 12 (1.2, 123.4). Omit the .3 since it's nearly meaningless, but many people when they see (1.2, 120) will assume that the last '0' in 120 is significant.&lt;/p&gt;&#xA;" OwnerUserId="2658" LastActivityDate="2011-03-24T19:47:02.827" CommentCount="3" />
  <row Id="8737" PostTypeId="2" ParentId="8733" CreationDate="2011-03-24T20:08:10.450" Score="0" ViewCount="" Body="&lt;p&gt;It's interesting... how do you compare the compounds with each other?  Repeated measures don't have to be necessarily over time, but they need to be measures of the same thing or things that can be treated as if they are the same.  You could certainly measure the same compound in different areas of the body and that would be a repeated measure.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I guess I'm asking the first question out of ignorance of your field.  If you can compare the compounds one to another then they are repeated measures grouped by individual animal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suppose one could even consider multi-level modelling beyond simple repeated measures because you could take the comparable body part as one level and the animal itself as another.&lt;/p&gt;&#xA;" OwnerUserId="601" LastActivityDate="2011-03-24T20:08:10.450" CommentCount="3" />
  <row Id="8738" PostTypeId="1" AcceptedAnswerId="8741" CreationDate="2011-03-24T20:25:16.957" Score="5" ViewCount="122" Body="&lt;p&gt;It's pretty tough to search the Web for info on something when you don't know what words are commonly used to describe it. In this case, I'm wondering what it's called when you include another predictor in a time series.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an example, say I'm modeling a variable $X$ using AR(3):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ X_t = \varphi_1 X_{t-1} + \varphi_2 X_{t-2} + \varphi_3 X_{t-3} + \varepsilon_t $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want my model to include the effects of another variable—say, $Y$—so my model is now described as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ X_t = \varphi_1 X_{t-1} + \varphi_2 X_{t-2} + \varphi_3 X_{t-3} + \beta_1 Y_{t-1} + \beta_2 Y_{t-2} + \beta_3 Y_{t-3} + \varepsilon_t $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What would the term (or terms) be that distinguishes the former model from the latter?&lt;/p&gt;&#xA;" OwnerUserId="1583" LastActivityDate="2011-04-07T21:01:14.247" Title="What is the term for a time series regression having more than one predictor?" Tags="&lt;time-series&gt;&lt;terminology&gt;" AnswerCount="3" CommentCount="1" FavoriteCount="1" />
  <row Id="8739" PostTypeId="2" ParentId="8733" CreationDate="2011-03-24T20:29:46.690" Score="1" ViewCount="" Body="&lt;p&gt;Have a look in a Multivariate text for MANOVA--multivariate ANOVA.  Here is a website...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://faculty.chass.ncsu.edu/garson/PA765/manova.htm&quot; rel=&quot;nofollow&quot;&gt;http://faculty.chass.ncsu.edu/garson/PA765/manova.htm&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Though, that's a lot of dependent variables and it could be hard to interpret.  It might be simpler to do some sort of data reduction first, like PCA among your set of fatty acids.  I suppose it depends on how much overlap there is and how many PCs you'd need to extract to account for the 30 vars.&lt;/p&gt;&#xA;" OwnerUserId="485" LastEditorUserId="485" LastEditorDisplayName="" LastEditDate="2011-03-24T21:05:57.457" LastActivityDate="2011-03-24T21:05:57.457" CommentCount="5" />
  <row Id="8740" PostTypeId="2" ParentId="8734" CreationDate="2011-03-24T20:37:17.667" Score="7" ViewCount="" Body="&lt;p&gt;I doubt there's a universal rule so I'm not going to make any up.  I can share these thoughts and the reasons behind them:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;When summaries reflect the data themselves--max, min, order statistics, etc.--&lt;strong&gt;use the same number of significant figures&lt;/strong&gt; used to record the data in the first place.  This provides a consistent representation throughout the document concerning the precision of the data.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;When summaries have higher precision than the data, &lt;strong&gt;write the values in a way that reflects that extra precision&lt;/strong&gt;.  For instance, a mean of $n$ values has $\sqrt{n}$ times the precision of the individual values: roughly, include one extra significant figure for $3 \le n \le 30$, two for $30 \lt n \le 300$, etc.  (This is rounding on a log-10 scale, obviously.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-Note that the CV does &lt;em&gt;not&lt;/em&gt; provide useful information in this regard.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-Some estimates can be obtained with great precision.  They don't have to be rounded to match something else.  For instance, the mean of 1,000,000 integers might be 10.977 with a standard error of 0.00301.  My decision to write the mean to three decimal places (and 4-5 sig figs) was based on the order of magnitude of the SE, which indicates the last digit is partially reliable.  The decision to write the SE to three sig figs (five decimal places) is more arbitrary: two sig figs would work; one probably would not; four sig figs would also work and be consistent with the 4-5 sig figs in the mean; more than four sig figs would be overkill.  (One could estimate the standard error of the SE itself in terms of the fourth moment of the data, and use that to determine an appropriate amount of rounding, but most of us don't go to such trouble...)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Signal the reader when you are doing substantial rounding&lt;/strong&gt;.  &lt;em&gt;Be especially careful when the report is discussing the statistical test itself&lt;/em&gt;.  The reason is that people may use your work to check their own calculations.  Sometimes even a slight difference can reveal an error.  You don't want to cause trouble because you rounded 123 to 120 and someone else, checking the work, obtains 123 and suspects one of you has erred.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Be consistent&lt;/strong&gt;.  You might lose some readers if you list a value as 123 at one point and later reference it as 120.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Don't be ridiculous&lt;/strong&gt;.  (I automatically suspect incompetence when I encounter reports that give statistical results to 15 sig figs when the data have only two sig figs, for instance.)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-03-24T20:37:17.667" CommentCount="1" />
  <row Id="8741" PostTypeId="2" ParentId="8738" CreationDate="2011-03-24T20:41:18.543" Score="5" ViewCount="" Body="&lt;p&gt;ARIMAX (Box-Tiao) is what it is called when you add covariates to Arima models, is is basically arima + X. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.r-bloggers.com/the-arimax-model-muddle/&quot; rel=&quot;nofollow&quot;&gt;http://www.r-bloggers.com/the-arimax-model-muddle/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also search for Panel data or TSCS: 'Time-series–cross-section (TSCS) data consist of comparable time series data observed on a variety of units'&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See:&#xA;&lt;a href=&quot;http://as.nyu.edu/docs/IO/2576/beck.pdf&quot; rel=&quot;nofollow&quot;&gt;http://as.nyu.edu/docs/IO/2576/beck.pdf&lt;/a&gt;&#xA;or &lt;a href=&quot;https://stat.ethz.ch/pipermail/r-sig-mixed-models/2010q4/004530.html&quot; rel=&quot;nofollow&quot;&gt;https://stat.ethz.ch/pipermail/r-sig-mixed-models/2010q4/004530.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1893" LastEditorUserId="1893" LastEditorDisplayName="" LastEditDate="2011-03-24T20:54:54.370" LastActivityDate="2011-03-24T20:54:54.370" />
  <row Id="8742" PostTypeId="1" AcceptedAnswerId="8745" CreationDate="2011-03-24T21:12:58.117" Score="1" ViewCount="149" Body="&lt;p&gt;There is a bayesian network Asia:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/sCh2D.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am computing based on&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;A (visit to Asia)&#xA;S (smoker)&#xA;T (tuberculosis)&#xA;L (lung cancer)&#xA;B (bronchitis)&#xA;E (tuberculosis versus lung cancer/bronchitis)&#xA;D (dyspnoea)&#xA;X (chest X-ray)&#xA;&#xA;P(A)=0.01 &#xA;P(S)=0.50 &#xA;P(T)=0.0104 &#xA;P(L)=0.055 &#xA;P(B)=0.45&#xA;P(E)=0.064828 &#xA;P(D)=0.4393105 &#xA;P(X)=0.11029004&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How would you compute probabilities when you &lt;strong&gt;assign truth values to ceratin observable variables like&lt;/strong&gt;?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;p(X=yes|A=no,  S=yes)&#xA;p(D=yes|L=no,  B=yes)&#xA;p(E=yes|L=yes, T=no)&#xA;p(D=yes|B=yes, T=yes)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="3681" LastEditorUserId="3681" LastEditorDisplayName="" LastEditDate="2011-03-27T01:30:17.867" LastActivityDate="2011-03-27T01:30:17.867" Title="Bayes Network computing conditional probabilities" Tags="&lt;bayesian&gt;&lt;conditional-probability&gt;" AnswerCount="3" CommentCount="2" FavoriteCount="3" />
  <row Id="8743" PostTypeId="2" ParentId="8742" CreationDate="2011-03-24T22:58:02.023" Score="1" ViewCount="" Body="&lt;p&gt;You have to compute &lt;a href=&quot;http://en.wikipedia.org/wiki/Joint_probability&quot; rel=&quot;nofollow&quot;&gt;joint probabilities&lt;/a&gt; first, and then the &lt;a href=&quot;http://en.wikipedia.org/wiki/Marginal_distribution&quot; rel=&quot;nofollow&quot;&gt;marginal probabilities&lt;/a&gt; you are interested in, thanks to &lt;a href=&quot;http://en.wikipedia.org/wiki/Belief_propagation&quot; rel=&quot;nofollow&quot;&gt;sum-product&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1351" LastEditorUserId="1351" LastEditorDisplayName="" LastEditDate="2011-03-25T15:18:57.543" LastActivityDate="2011-03-25T15:18:57.543" CommentCount="0" />
  <row Id="8744" PostTypeId="1" CreationDate="2011-03-24T23:07:21.220" Score="4" ViewCount="134" Body="&lt;p&gt;I have some points $X=\{x_1,...,x_n\}$ in $R^p$, and I want to cluster the points so that:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Each cluster contains an equal number of elements of $X$. (Assume that the number of clusters divides $n$.)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Each cluster is &quot;spatially cohesive&quot; in some sense, like the clusters from $k$-means.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;It's easy to think of a lot of clustering procedures that satisfy one or the other of these, but does anyone know of a way to get both at once?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance for any pointers!&lt;/p&gt;&#xA;" OwnerUserId="3891" LastEditorUserId="3891" LastEditorDisplayName="" LastEditDate="2011-03-25T02:17:13.157" LastActivityDate="2011-03-28T09:21:38.480" Title="Clustering procedure where each cluster has an equal number of points?" Tags="&lt;clustering&gt;" AnswerCount="2" CommentCount="7" />
  <row Id="8745" PostTypeId="2" ParentId="8742" CreationDate="2011-03-24T23:54:20.790" Score="3" ViewCount="" Body="&lt;p&gt;For your four example questions, it looks rather easy, if I am reading this correctly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first asks for the probability the individual has tuberculosis or cancer given not having tuberculosis and not having cancer.  That should be 0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second, third and fourth ask for the probability the individual has tuberculosis or cancer given some combination of having them.  That should be 1 each time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suspect that these are not quite what you are trying to ask.  For example the proportion of individuals who have both tuberculosis and cancer might be $0.0104 + 0.055 - 0.064828 =  0.000572$.  This is also suggested if the lack of an arrow between tuberculosis and cancer means they are independent and $0.0104 \times 0.055 = 0.000572$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But there is no such clarity for example with the relationship between a visit to Asia and tuberculosis.  This is not independent as there is an arrow, but more individuals have tuberculosis than visited Asia.  Are we supposed to assume that everyone who visited Asia now has tuberculosis? It does not say that.&lt;/p&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-03-24T23:54:20.790" CommentCount="6" />
  <row Id="8746" PostTypeId="2" ParentId="8738" CreationDate="2011-03-24T23:59:50.460" Score="3" ViewCount="" Body="&lt;p&gt;This is called a Transfer Function Model. It has also been referred to as a Dynamic Regression Model.&lt;/p&gt;&#xA;" OwnerUserId="3382" LastActivityDate="2011-03-24T23:59:50.460" CommentCount="1" />
  <row Id="8747" PostTypeId="1" CreationDate="2011-03-25T01:15:45.307" Score="4" ViewCount="122" Body="&lt;p&gt;I am working on something to test advertisements.  I have 3 independent variables I want to test (with mixed numbers of variations of each variable), and I would like to find the best combination of the three by looking at their effect on a single dependent variable (the percentage who purchase).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;TITLE     IMAGE  DESCRIPTION  (Percentage who purchased)&#xA;Title1    Image1 Decription1  .05%&#xA;Title2    Image2 Description1 .02%&#xA;Title3    Image? Description1 .08%&#xA;Title4    Image? Description1 .02%&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;There can be any number of variations of each variable, sometimes there will be 10 images, other times there will only be one, etc.&#xA;My goal is to find out what Titles/Images/Descriptions affect the purchase percentage.  &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;How can I test and analyze something like this?  &lt;/li&gt;&#xA;&lt;li&gt;Is it possible to not have to test every single variation and still find the best combination? &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;(I've read about chi-square tests but I don't know if they are the proper fit)&lt;/p&gt;&#xA;" OwnerUserId="3892" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-12T04:34:57.197" LastActivityDate="2011-04-12T04:34:57.197" Title="A test for assessing advertisement efficiency" Tags="&lt;chi-squared&gt;&lt;conjoint&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="8748" PostTypeId="1" AcceptedAnswerId="8758" CreationDate="2011-03-25T02:55:34.180" Score="7" ViewCount="104" Body="&lt;p&gt;I am working in hospital processing infection data, and start to read more and more articles on regressions and statistics, having realized that my mathematics background is not sufficient for me to handle all the maths inside the article. I plan to do some self-study.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have seen from &lt;a href=&quot;http://www.biostat.jhsph.edu/academics/courses/651_FAQ.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; that calculus and linear algebra is needed for going further in biostatistics, I am thinking of finding some textbook for that, preferably free textbook online.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And one more question, which one should I start first? Calculus or algebra?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know this may not be relevant to be asked here, but can anyone here kindly give me some suggestions that I can start my reading? Thanks.&lt;/p&gt;&#xA;" OwnerUserId="588" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-25T09:29:26.643" LastActivityDate="2011-03-25T12:14:55.213" Title="Introduction to maths for a junior in epidemiology" Tags="&lt;books&gt;" AnswerCount="2" FavoriteCount="4" />
  <row Id="8749" PostTypeId="1" AcceptedAnswerId="8928" CreationDate="2011-03-25T03:04:24.757" Score="10" ViewCount="198" Body="&lt;p&gt;I need to implement a program that will classify records into 2 categories (true/false) based on some training data, and I was wondering at which algorithm/methodology I should be looking at. There seem to be a lot of them to choose from -- Artificial Neural Network, Genetic Algorithm, Machine Learning, Bayesian Optimization etc. etc., and I wasn't sure where to start. So, my questions is: &#xA;&lt;strong&gt;How should I choose a learning algorithm I should use for my problem?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If this helps, here is the problem I need to solve.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;&lt;strong&gt;The training data:&lt;/em&gt;&lt;/strong&gt;&lt;br&gt;&#xA;The training data consists of many rows like this:  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Precursor1, Precursor2, Boolean (true/false)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;&lt;strong&gt;The run&lt;/em&gt;&lt;/strong&gt;&lt;br&gt;&#xA;I will be given a bunch of precursors.&lt;br&gt;&#xA;Then,  &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;I choose an algorithm A from different algorithms (or dynamically generate an algorithm), and apply it on every possible combinations of these precursors and collect the &quot;record&quot;s &#xA;that are emitted. The &quot;record&quot; consists of several key-value pairs*.&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I apply some awesome algorithm and classify these records in to 2 categories&#xA;(true/false).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I will generate a table that has the same format as the train data:&lt;br&gt;&#xA;&lt;code&gt;Precursor1, Precursor2, Boolean&lt;/code&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;And the whole program is scored based on how many true/false I got right.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;*:&quot;Record&quot;s will look like this (hope this makes sense)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Record         [1...*] Score&#xA;-Precursor1             -Key&#xA;-Precursor2             -Value&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;There are only a finite number of possible Keys. Records contain different subset of these keys (some records have key1, key2, key3... other records have key3, key4... etc.).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I actually need 2 learning. One is for step 1. I need to have a module that looks at the Precursor pairs etc. and decide what algorithm to apply in order to emit a record for the comparison. Another is for step 2. I need a module that analyzes the collection of records and categorize them into the 2 categories (true/false).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you in advance!  &lt;/p&gt;&#xA;" OwnerUserId="800" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-25T13:48:01.460" LastActivityDate="2011-03-29T15:52:01.003" Title="How to choose between learning algorithms" Tags="&lt;machine-learning&gt;&lt;bayesian&gt;&lt;optimization&gt;&lt;genetic-algorithms&gt;" AnswerCount="2" FavoriteCount="4" />
  <row Id="8750" PostTypeId="1" AcceptedAnswerId="8751" CreationDate="2011-03-25T03:29:12.967" Score="2" ViewCount="97" Body="&lt;p&gt;If I have an arima object like &lt;code&gt;a&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(100)&#xA;x1 &amp;lt;- cumsum(runif(100))&#xA;x2 &amp;lt;- c(rnorm(25, 20), rep(0, 75))&#xA;x3 &amp;lt;- x1 + x2&#xA;&#xA;dummy = c(rep(1, 25), rep(0, 75))&#xA;&#xA;a &amp;lt;- arima(x3, order=c(0, 1, 0), xreg=dummy)&#xA;print(a)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Series: x3 &#xA;ARIMA(0,1,0)                    &#xA;&#xA;Call: arima(x = x3, order = c(0, 1, 0), xreg = dummy) &#xA;&#xA;Coefficients:&#xA;        dummy&#xA;      17.7665&#xA;s.e.   1.1434&#xA;&#xA;sigma^2 estimated as 1.307:  log likelihood = -153.74&#xA;AIC = 311.48   AICc = 311.6   BIC = 316.67&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How do calculate the R squared of this regression?&lt;/p&gt;&#xA;" OwnerUserId="179" LastEditorUserId="179" LastEditorDisplayName="" LastEditDate="2011-03-25T04:42:25.623" LastActivityDate="2011-03-25T05:19:58.357" Title="How can I calculate the R-squared of a regression with arima errors using R?" Tags="&lt;r&gt;&lt;time-series&gt;&lt;regression&gt;&lt;basic-concepts&gt;&lt;arima&gt;" AnswerCount="1" />
  <row Id="8751" PostTypeId="2" ParentId="8750" CreationDate="2011-03-25T05:19:58.357" Score="6" ViewCount="" Body="&lt;p&gt;Once you have ARMA errors, it is not a simple linear regression any more. So you would have to define what you mean by $R^2$. Perhaps the squared correlation of fitted to actuals? In that case:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cor(fitted(a),x3)^2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The &lt;code&gt;fitted()&lt;/code&gt; function will only work if you have loaded the &lt;code&gt;forecast&lt;/code&gt; package, but it looks like you have already done that judging from the output in your question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case, you don't have ARMA errors, but you do have differencing. So it is equivalent to the linear model&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;b &amp;lt;- lm(diff(x3) ~ diff(dummy) - 1)&#xA;summary(b)&#xA;&#xA;Coefficients:&#xA;            Estimate Std. Error t value Pr(&amp;gt;|t|)    &#xA;diff(dummy)   17.766      1.149   15.46   &amp;lt;2e-16 ***&#xA;---&#xA;Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 &#xA;&#xA;Residual standard error: 1.149 on 98 degrees of freedom&#xA;Multiple R-squared: 0.7092,     Adjusted R-squared: 0.7062 &#xA;F-statistic:   239 on 1 and 98 DF,  p-value: &amp;lt; 2.2e-16 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Of course, that is a very different value of $R^2$ than just using the correlations as above because it is now being computed on the differences. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You will need define what you mean by $R^2$, and what you want to use it for. Once you move away from the usual regression set up with an intercept and iid errors, $R^2$ ceases to be uniquely defined and is not particularly useful.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2011-03-25T05:19:58.357" CommentCount="1" />
  <row Id="8752" PostTypeId="1" CreationDate="2011-03-25T06:00:09.103" Score="1" ViewCount="30" Body="&lt;p&gt;I want to perform quadrat count analysis on several point processes (or one marked point process), to then apply some dimensionality reduction techniques.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The marks are not identically distributed, i.e. some marks are appearing quite often, and some are pretty rare. Thus, I cannot simply divide my 2D space in a regular grid, because the more frequent marks will &quot;overwhelm&quot; the lesser frequent ones, masking their appearance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, I tried to build my grid such that each cell has at most N points in it (to do so, I simply divide each cell in four smaller (and equally sized) cells, recursively, until no cell has more than N points in it). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What do you think of this &quot;normalization&quot; technique ? Is there a standard way to do such things ?&lt;/p&gt;&#xA;" OwnerUserId="3699" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-25T09:24:01.730" LastActivityDate="2011-03-25T09:24:01.730" Title="When doing quadrat counting, how do you construct the quadrats?" Tags="&lt;multivariate-analysis&gt;&lt;normalization&gt;&lt;point-process&gt;" />
  <row Id="8753" PostTypeId="2" ParentId="8747" CreationDate="2011-03-25T07:09:32.060" Score="4" ViewCount="" Body="&lt;p&gt;In marketing-oriented statistics the analysis you need is called conjoint analysis. You construct a number of product &quot;scenarios&quot; which are various &quot;mixes&quot; of attributes such your 3 attributes, each being allowed to vary across some &quot;levels&quot;. The Conjoint will tell you what's the best scenario and what are &quot;utility&quot; coefficients for each of your attributes - these allow to predict popularity of scenarios which you didn't test. Algorithmically conjoint analysis could be based on regressions or multidimensional scaling or other technique. Look for software performing conjoint analysis.&lt;/p&gt;&#xA;" OwnerUserId="3277" LastActivityDate="2011-03-25T07:09:32.060" CommentCount="2" />
  <row Id="8754" PostTypeId="1" AcceptedAnswerId="8768" CreationDate="2011-03-25T07:45:24.537" Score="10" ViewCount="171" Body="&lt;p&gt;An &lt;strong&gt;interim analysis&lt;/strong&gt; is an analysis of the data at one or more time points prior the official close of the study with the intention of, e.g., possibly terminating the study early.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;According to Piantadosi, S. (&lt;a href=&quot;http://eu.wiley.com/WileyCDA/WileyTitle/productCd-0471727814.html&quot;&gt;Clinical trials - a methodologic perspective&lt;/a&gt;):&#xA;&quot;&lt;em&gt;The estimate of a treatment effect will be biased when a trial is terminated at an early stage. The earlier the decision, the larger the bias.&lt;/em&gt;&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can you explain me this claim. I can easily understand that the accuracy is going to be affected, but the claim about the bias is not obvious to me...&lt;/p&gt;&#xA;" OwnerUserId="3019" LastActivityDate="2011-03-25T16:18:10.147" Title="Why is bias affected when a clinical trial is terminated at an early stage?" Tags="&lt;clinical-trials&gt;&lt;interim-analysis&gt;&lt;bias&gt;" AnswerCount="5" CommentCount="1" FavoriteCount="1" />
  <row Id="8755" PostTypeId="1" AcceptedAnswerId="8773" CreationDate="2011-03-25T08:22:34.893" Score="4" ViewCount="103" Body="&lt;p&gt;I have some data to analyze where $y$ is dependent of $x$ - a linear regression was used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's a question from an exam, so I think it should be solvable. The regression was used to estimate the mean &lt;em&gt;miles per gallon&lt;/em&gt; (response) from the &lt;em&gt;amount of miles driven&lt;/em&gt; (predictor).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have the following statistics available:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Correlation coefficient (0.117)&lt;/li&gt;&#xA;&lt;li&gt;Standard deviation (0.482)&lt;/li&gt;&#xA;&lt;li&gt;Number of observations (101)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;An ANOVA of this regression yields (Regression and residuals, respectively):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;df: 1, 99&lt;/li&gt;&#xA;&lt;li&gt;SS: 0.319, 22.96&lt;/li&gt;&#xA;&lt;li&gt;MS: 0.319, 0.232&lt;/li&gt;&#xA;&lt;li&gt;F-value: 1.374, critical F-value: 0.244&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The regression itself (Intercept and Slope, respectively):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Coefficients: 6.51, -0.00024&lt;/li&gt;&#xA;&lt;li&gt;Standard deviations: 0.186, 0.0002&lt;/li&gt;&#xA;&lt;li&gt;t-Values: 34.90, -1.17&lt;/li&gt;&#xA;&lt;li&gt;p-Values: 1.93E-57, 0.2439&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Also, the &quot;upper and lower 95% and 99%&quot; are given for the above regression (although I'm not sure what that means).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, I am asked to calculate the mean $y$ for several values $x$, that's relatively easy, I just use the coefficients. So for example, I can calculate the mean miles per gallon for 500 miles driven.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Part where I'm stuck: &lt;strong&gt;I need to calculate the 99% confidence interval for the mean of $y$.&lt;/strong&gt;. Obviously, this is what the example is all about - the introduction states that the mileage of a car should be estimated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question: How can I find out the mean of $y$ using the data provided above? (And, subsequently, the 99% confidence interval, although I seem to have the standard deviation, so that shouldn't be the problem)&lt;/p&gt;&#xA;" OwnerUserId="1205" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-29T00:56:50.313" LastActivityDate="2011-04-29T00:56:50.313" Title="Calculating the mean using regression data" Tags="&lt;regression&gt;&lt;homework&gt;&lt;mean&gt;" AnswerCount="1" CommentCount="6" />
  <row Id="8756" PostTypeId="2" ParentId="8718" CreationDate="2011-03-25T09:14:34.840" Score="0" ViewCount="" Body="&lt;p&gt;That useful, thanks Peter. The reason I asked the first question is because I've seen exactly that - comparisons of coefficients across models within a sample and comparisons across models from different samples - in refereed papers. I didn't feel it was the right approach, and, evidently, I'm correct.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regarding the technical details, can't I simply estimate a model on all six samples and an interaction term between the key predictor that I want to compare and a variable specifying each sample (representing a different time period)? Is that what you were saying? Do I need a variable for each sample?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, I've read that interactions can be difficult to interpret, can anyone shed any light on that?&lt;/p&gt;&#xA;" OwnerUserId="3883" LastActivityDate="2011-03-25T09:14:34.840" CommentCount="3" />
  <row Id="8757" PostTypeId="2" ParentId="8692" CreationDate="2011-03-25T09:15:16.203" Score="2" ViewCount="" Body="&lt;p&gt;To get unstacking of &lt;code&gt;a&lt;/code&gt; similar to &lt;code&gt;rbind(a,b)&lt;/code&gt; you can simply transpose the result:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; t(unstack(a))&#xA;      RSTO RSTR    S1 S2   S3     SF&#xA;res 199665 4147 31274  1 2522 118009&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You will get a &lt;code&gt;matrix&lt;/code&gt; instead of &lt;code&gt;data.frame&lt;/code&gt; though. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is also possible to use the &lt;code&gt;cast&lt;/code&gt; function from package &lt;strong&gt;reshape&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; cast(~state,data=a,value=&quot;count&quot;)&#xA;  value   RSTO RSTR    S1 S2   S3     SF&#xA;1 (all) 199665 4147 31274  1 2522 118009 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For the &lt;code&gt;rbind(a,b)&lt;/code&gt; with added &lt;code&gt;id&lt;/code&gt; column the cast will give you the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; ab &amp;lt;- rbind(a,b)&#xA;&amp;gt; ab$id=rep(1:2,each=nrow(a))&#xA;&amp;gt; cast(id~state,data=ab,value=&quot;count&quot;)&#xA;  id   RSTO  RSTR    S1   S2   S3     SF&#xA;1  1 199665  4147 31274    1 2522 118009&#xA;2  2  31956 11689  6702 2838 6268 672561&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-03-25T09:15:16.203" CommentCount="2" />
  <row Id="8758" PostTypeId="2" ParentId="8748" CreationDate="2011-03-25T10:13:50.523" Score="4" ViewCount="" Body="&lt;p&gt;Jeff Gill has a good book, on Essential Mathematics for Social and Political Research: &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/052168403X&quot; rel=&quot;nofollow&quot;&gt;http://www.amazon.com/Essential-Mathematics-Political-Research-Analytical/dp/052168403X/ref=sr_1_2?ie=UTF8&amp;amp;s=books&amp;amp;qid=1301047912&amp;amp;sr=8-2&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found it quite useful for getting a good overview of linear algebra and calculus. He only assumes knowledge of basic algebra (i.e. x+y=2 etc).&#xA;Despite the name, its a good read for anyone interested in bringing their maths up to the level required for reading journal articles and multivariate textbooks. &lt;/p&gt;&#xA;" OwnerUserId="656" LastActivityDate="2011-03-25T10:13:50.523" />
  <row Id="8759" PostTypeId="2" ParentId="8754" CreationDate="2011-03-25T10:18:25.167" Score="1" ViewCount="" Body="&lt;p&gt;Well, my knowledge on this comes from the Harveian oration in 2008 &lt;a href=&quot;http://bookshop.rcplondon.ac.uk/details.aspx?e=262&quot; rel=&quot;nofollow&quot;&gt;http://bookshop.rcplondon.ac.uk/details.aspx?e=262&lt;/a&gt;&#xA;Essentially, to the best of my recollection the results will be biased as 1) stopping early usually means that either the treatment was more or less effective than one hoped, and if this is positive, then you may be capitalising on chance. &#xA;I believe that p values are calculated on the basis of the planned sample size (but i could be wrong on this), and also if you are constantly checking your results to see if any effects have been shown, you need to correct for multiple comparisons in order to insure that you are not merely finding a chance effect.&#xA;For example, if you check 20 times for p values below .05 then statistically speaking, you are almost certain to find one significant result.&lt;/p&gt;&#xA;" OwnerUserId="656" LastActivityDate="2011-03-25T10:18:25.167" CommentCount="2" />
  <row Id="8760" PostTypeId="2" ParentId="8690" CreationDate="2011-03-25T10:26:13.193" Score="5" ViewCount="" Body="&lt;p&gt;I will try to answer the questions 2 to 4.  Suppose that we observe sample $(y_i,\mathbf{x}_i,z_i,\gamma_i,\varepsilon_i)$. Suppose that our model is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$y_i=\mathbf{x}_i\beta+\gamma_iz_i+\varepsilon_i$$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;and&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$E(\varepsilon_i|\mathbf{x}_i,z_i,\gamma_i)=0.$$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The least squares estimate of the regression will be&#xA;\begin{align}&#xA;(\hat{\beta},\hat{\gamma})'=\left(\sum_{i=1}^n&#xA;\begin{bmatrix} \mathbf{x}_i'\\&#xA; z_i&#xA;\end{bmatrix}&#xA;[\mathbf{x}_i,z_i]\right)^{-1}\sum_{i=1}^n\begin{bmatrix} \mathbf{x}_i'\\&#xA; z_i&#xA;\end{bmatrix}y_i&#xA;\end{align}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now since we assumed sample due to law of large numbers we get that&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align}&#xA;\frac{1}{n}\sum_{i=1}^n&#xA;\begin{bmatrix} \mathbf{x}_i'\\&#xA; z_i&#xA;\end{bmatrix}&#xA;[\mathbf{x}_i,z_i]\to &#xA;\begin{bmatrix}&#xA;E\mathbf{x}_1'\mathbf{x}_1 &amp;amp; E\mathbf{x_1}'z_1\\&#xA;E\mathbf{x}_1z_1 &amp;amp; Ez_1^2&#xA;\end{bmatrix}&#xA;\end{align}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now &#xA;\begin{align}&#xA;\sum_{i=1}^n\begin{bmatrix} \mathbf{x}_i'\\&#xA; z_i&#xA;\end{bmatrix}y_i=\sum_{i=1}^n\begin{bmatrix} \mathbf{x}_i'\\&#xA; z_i&#xA;\end{bmatrix}\varepsilon_i+\sum_{i=1}^n&#xA;\begin{bmatrix}&#xA; \mathbf{x}_i'\mathbf{x}_i\beta +\mathbf{x}_i'z_i\gamma_i \\&#xA;\mathbf{x}_iz_i\beta + z_i^2\gamma_i&#xA;\end{bmatrix}&#xA;\end{align}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Due to law of large numbers and our conditional expectation condition we get that&#xA;\begin{align}&#xA;\frac{1}{n}\sum_{i=1}^n\begin{bmatrix} \mathbf{x}_i'\\&#xA; z_i&#xA;\end{bmatrix}\varepsilon_i\to 0.&#xA;\end{align}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now comes the part where we need more assumptions. Assume that $(\mathbf{x}_i,z_i)$ is independent of the $\gamma_i$. Then due to law of large numbers&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align}&#xA;\frac{1}{n}\sum_{i=1}^n\mathbf{x}_iz_i\gamma_i\to E\mathbf{x}_1z_1\gamma_1=pE\mathbf{x}_1z_1&#xA;\end{align}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $p=P(\gamma_i=1)$. Similarly&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align}&#xA;\frac{1}{n}\sum_{i=1}^nz_i^2\gamma_i\to Ez_1^2\gamma_1=pEz_1^2&#xA;\end{align}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Gathering all the results we get &lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align}&#xA;(\hat{\beta},\hat{\gamma})'\to \left(\begin{bmatrix}&#xA;E\mathbf{x}_1'\mathbf{x}_1 &amp;amp; E\mathbf{x_1}'z_1\\&#xA;E\mathbf{x}_1z_1 &amp;amp; Ez_1^2&#xA;\end{bmatrix}\right)^{-1}\begin{bmatrix}&#xA;E\mathbf{x}_1'\mathbf{x}_1\beta + pE\mathbf{x_1}'z_1\\&#xA;E\mathbf{x}_1z_1\beta + pEz_1^2&#xA;\end{bmatrix}=(\beta,p)'&#xA;\end{align}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the answer to second question is yes. Simple experiment in R confirms this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; g&amp;lt;-sample(0:1,1000,prob=c(1/3,1-1/3),replace=TRUE)&#xA;&amp;gt; z &amp;lt;- rnorm(1000)&#xA;&amp;gt; y&amp;lt;-1+g*z+rnorm(1000)/3&#xA;&amp;gt; dt &amp;lt;- data.frame(y=y,z=z)&#xA;&amp;gt; lm(y~z,data=dt)&#xA;&#xA;Call:&#xA;lm(formula = y ~ z, data = dt)&#xA;&#xA;Coefficients:&#xA;(Intercept)            z  &#xA;     1.0050       0.6595  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now let us proceed to question 3. Introduce notation $X_i=(\mathbf{x}_i,z_i)$. Then&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align}&#xA;\sqrt{n}\begin{bmatrix}&#xA;\hat{\beta}-\beta\\&#xA;\hat{\gamma}-p &#xA;\end{bmatrix}=\left(\frac{1}{n}\sum_{i=1}^nX_i'X_i\right)^{-1}\frac{1}{\sqrt{n}}\left(\sum_{i=1}^nX_i'\varepsilon_i+\sum_{i=1}^n&#xA;\begin{bmatrix}&#xA;\mathbf{x}_i'z_i\\\&#xA;z_i^2&#xA;\end{bmatrix}(\gamma_i-p)\right)&#xA;\end{align}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Introduce notation $Z_i=(\mathbf{x}_iz_i,z_i^2)$ and $C_i=(X_i\varepsilon_i,Z_i(\gamma_i-p))$. We have that $EC_i=0$ and since we have sample we can apply multivariate central limit theorem for $C_i$:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align}&#xA;\frac{1}{\sqrt{n}}\sum_{i=1}^nC_i'\to N(0,\Sigma_C)&#xA;\end{align}&#xA;where&#xA;\begin{align}&#xA;\Sigma_C=EC_i'C_i&amp;amp;=&#xA;\begin{bmatrix}&#xA;EX_1'X_1\varepsilon_1^2 &amp;amp; EX_1'Z_1\varepsilon_1(\gamma_1-p)\\&#xA;EZ_1'X_1\varepsilon_1(\gamma_1-p) &amp;amp; EZ_1'Z_1(\gamma_1-p)^2&#xA;\end{bmatrix}\\&#xA;&amp;amp;=\begin{bmatrix}&#xA;EX_1'X_1\varepsilon_1^2 &amp;amp; 0\\&#xA;0 &amp;amp; EZ_1'Z_1(\gamma_1-p)^2&#xA;\end{bmatrix}&#xA;\end{align}&#xA;due to condition $E(\varepsilon_i|\mathbf{x}_i,z_i,\gamma_i)=0$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now assume further that $E(\varepsilon_i^2|\mathbf{x}_i,z_i,\gamma_i)=\sigma^2$. Denote &#xA;\begin{align}&#xA;\mathbf{A}=EX_1'X_1, \quad \mathbf{B}=EZ_1'Z_1&#xA;\end{align}&#xA;Then we get &#xA;\begin{align}&#xA;\left(\frac{1}{n}\sum_{i=1}^nX_i'X_i\right)^{-1}\frac{1}{\sqrt{n}}\left(\sum_{i=1}^nX_i'\varepsilon_i+\sum_{i=1}^n&#xA;\begin{bmatrix}&#xA;\mathbf{x}_i'z_i\\\&#xA;z_i^2&#xA;\end{bmatrix}(\gamma_i-p)\right)\to N(0,\sigma^2A+p(1-p)B)&#xA;\end{align}&#xA;so&#xA;\begin{align}&#xA;\sqrt{n}\begin{bmatrix}&#xA;\hat{\beta}-\beta\\&#xA;\hat{\gamma}-p &#xA;\end{bmatrix}\to N(0,\sigma^2A^{-1}+p(1-p)A^{-1}BA^{-1})&#xA;\end{align}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This should give an idea how to construct the test for testing $H_0:p=p_0$. I am not happy about the presence of $p$ in the variance matrix, this could pose some problems. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now given the above the answer to the fourth question is that random coefficient influences only the covariance matrix of the other coefficients. If there is bias, it vanishes asymptotically. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I should note that the way I derived these results is pretty straightforward application of LLN and CLT. If there is some elegant way to avoid this I would really like to know.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-03-28T12:08:49.257" LastActivityDate="2011-03-28T12:08:49.257" />
  <row Id="8762" PostTypeId="2" ParentId="8744" CreationDate="2011-03-25T11:07:50.053" Score="2" ViewCount="" Body="&lt;p&gt;I suggest the recent paper &lt;a href=&quot;http://las.ethz.ch/files/gomes10discriminative.pdf&quot; rel=&quot;nofollow&quot;&gt;Discriminative Clustering by Regularized Information Maximization&lt;/a&gt; (and references therein). Specifically, Section 2 talks about class balance and cluster assumption.&lt;/p&gt;&#xA;" OwnerUserId="3785" LastEditorUserId="3785" LastEditorDisplayName="" LastEditDate="2011-03-26T09:01:37.207" LastActivityDate="2011-03-26T09:01:37.207" />
  <row Id="8763" PostTypeId="2" ParentId="8754" CreationDate="2011-03-25T11:15:05.500" Score="1" ViewCount="" Body="&lt;p&gt;I would disagree with that claim, unless by &quot;bias&quot; Piantadosi means that part of the accuracy which is commonly called bias.  The inference won't be &quot;biased&quot; because you chose to stop per se: it will be &quot;biased&quot; because you have less data.  The so called &quot;likelihood principle&quot; states that inference should only depend on data that was observed, and not on data that might have been observed, but was not.  The LP says&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(H|D,S,I)=P(H|D,I)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $H$ stands for the hypothesis you are testing (in the form of a proposition, such as &quot;the treatment was effective&quot;), $D$ stands for the data you actually observed, and $S$ stands for the proposition &quot;the experiment was stopped early&quot;, and $I$ stands for the prior information (such as a model). Now suppose your stopping rule depends on the data $D$ and on the prior information $I$, so you can write $S=g(D,I)$.  Now an elementary rule of logic is $AA=A$ - saying that A is true twice is the same thing as saying it once.&#xA;This means that because $S=g(D,I)$ will be true whenever $D$ and $I$ are also true.  So in &quot;boolean algebra&quot; we have $D,S,I = D,g(D,I),I = D,I$.  This proves the above equation of the likelihood principle.  It is only if your stopping rule depends on something other than the data $D$ or the prior information $I$ that it matters.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-03-25T11:15:05.500" CommentCount="8" />
  <row Id="8764" PostTypeId="2" ParentId="8748" CreationDate="2011-03-25T12:14:55.213" Score="3" ViewCount="" Body="&lt;p&gt;As far as learning the information on slide 6 in the slideshow you linked to, I would suggest &lt;a href=&quot;http://www.sagepub.com/books/Book232153&quot; rel=&quot;nofollow&quot;&gt;A Mathematical Primer for Social Statistics&lt;/a&gt; by John Fox (not free but cheap, &lt;a href=&quot;http://books.google.com/books?id=S4_VhrdIKS4C&quot; rel=&quot;nofollow&quot;&gt;Google book link&lt;/a&gt;). All of those sage green books are aimed at individuals with only a very brief statistics background.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are interested in taking that specific class I would also suggest you ask the professor for a syllabus and maybe some example problems. Although the professor did not state any preferred reference mathematical book I would imagine if pressed they could give some recommendations.&lt;/p&gt;&#xA;" OwnerUserId="1036" LastActivityDate="2011-03-25T12:14:55.213" />
  <row Id="8765" PostTypeId="1" CreationDate="2011-03-25T02:38:17.140" Score="0" ViewCount="49" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;Possible Duplicate:&lt;/strong&gt;&lt;br&gt;&#xA;  &lt;a href=&quot;http://stats.stackexchange.com/questions/8749/how-to-choose-between-learning-algorithms&quot;&gt;How to choose between learning algorithms&lt;/a&gt;  &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&#xA;&#xA;&lt;p&gt;Hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to implement a program that will classify records into 2 categories (true/false) based on some training data, and I was wondering at which algorithm/methodology I should be looking at. There seem to be a lot of them to choose from -- Artificial Neural Network, Genetic Algorithm, Machine Learning, Bayesian Optimization etc. etc., and I wasn't sure where to start. So, my questions is: &#xA;&lt;strong&gt;How should I choose a learning algorithm I should use for my problem?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If this helps, here is the problem I need to solve.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;&lt;strong&gt;The training data:&lt;/em&gt;&lt;/strong&gt;&lt;br&gt;&#xA;The training data consists of many rows like this:  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Precursor1, Precursor2, Boolean (true/false)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;&lt;strong&gt;The run&lt;/em&gt;&lt;/strong&gt;&lt;br&gt;&#xA;I will be given a bunch of precursors.&lt;br&gt;&#xA;Then,  &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;I choose an algorithm A from different algorithms (or dynamically generate an algorithm), and apply it on every possible combinations of these precursors and collect the &quot;record&quot;s &#xA;that are emitted. The &quot;record&quot; consists of several key-value pairs*.&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I apply some awesome algorithm and classify these records in to 2 categories&#xA;(true/false).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I will generate a table that has the same format as the train data:&lt;br&gt;&#xA;&lt;code&gt;Precursor1, Precursor2, Boolean&lt;/code&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;And the whole program is scored based on how many true/false I got right.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;*:&quot;Record&quot;s will look like this (hope this makes sense)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Record         [1...*] Score&#xA;-Precursor1             -Key&#xA;-Precursor2             -Value&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;There are only a finite number of possible Keys. Records contain different subset of these keys (some records have key1, key2, key3... other records have key3, key4... etc.).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I actually need 2 learning. One is for step 1. I need to have a module that looks at the Precursor pairs etc. and decide what algorithm to apply in order to emit a record for the comparison. Another is for step 2. I need a module that analyzes the collection of records and categorize them into the 2 categories (true/false).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you in advance!&lt;/p&gt;&#xA;" OwnerUserId="800" LastActivityDate="2011-03-25T12:55:30.820" ClosedDate="2011-03-25T13:47:26.547" Title="How to choose between learning algorithms" Tags="&lt;optimization&gt;&lt;machine-learning&gt;" CommentCount="1" />
  <row Id="8766" PostTypeId="2" ParentId="8380" CreationDate="2011-03-25T13:18:27.827" Score="1" ViewCount="" Body="&lt;p&gt;This sounds like something very similar to a method I have seen Jerry Reiter using multiple imputation for missing data.  However I can't quite remember the name of the paper.  But these terms will probably be able to get you in the right(er) direction (pardon the pun).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So basically you have three variables $X$, $Y$, and $Z$.  the variable $Z$ is your &quot;gold standard&quot; variable, and the variable $X$ is the &quot;bronze standard&quot; variable.  You would prefer to have observed $Y$ and $Z$ together.  But unfortunately, you only observe $X$ and $Y$ together, and $X$ and $Z$ together.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;you can set up the model as follows.  If you knew $Z$, then $X$ would be irrelevant to making inference about $Y$ (why use bronze when you've got gold?).  So you have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(Y|X,Z)=P(Y|Z)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But you also have a model relating $X$ and $Z$, $P(Z|X)$.  Now use the law of total probability, the product rule, and the above equation to expand $P(Y|X)$:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(Y|X)=\int P(Y,Z|X)dZ=\int P(Y|Z,X)P(Z|X) dZ=\int P(Y|Z)P(Z|X)dZ$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And so you have a &quot;weighted average&quot; of the &quot;gold model&quot; $P(Y|Z)$, where the weights depend on the &quot;error model&quot; $P(Z|X)$ (i.e. how well the bronze standard predicts the gold standard).  If the $Z$ can be quite well predicted, then $P(Z|X)$ will resemble a &quot;delta function&quot; and the model will essentially be a &quot;plug in&quot; model:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(Y|Z)\approx P(Y|Z)_{Z=X}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If $Z$ is poorly estimated from $X$, then $P(Z|X)$ will be quite &quot;flat&quot; and this procedure will &quot;spread out its bets&quot; over many different models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Jerry Reiter's method should give some more details about the actual implementation of this.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-03-25T13:18:27.827" CommentCount="1" />
  <row Id="8767" PostTypeId="2" ParentId="8754" CreationDate="2011-03-25T13:53:13.750" Score="0" ViewCount="" Body="&lt;p&gt;there &lt;em&gt;will&lt;/em&gt; be bias (in &quot;statistical sense&quot;) if termination of studies is not random. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a set of experiments run to conclusion, the &quot;early on&quot; results of (a) some experiments that ultimately find &quot;no effect&quot; will show some effect (as a result of chance) and (b) some experiments that ultimately do find an effect will show &quot;no effect&quot; (likely as a result of lack of power). In a world in which you terminate trials, if you stop (a) more often than (b), you'll end up across run of studies with bias in favor of finding an effect. (Same logic applies for effect &lt;em&gt;sizes&lt;/em&gt;; terminating studies that show &quot;bigger than expected&quot; effect early on more often than ones that show &quot;as expected or lower&quot; will inflate count of findings of &quot;big effect.&quot;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If in fact medical trials are terminated when early results show a positive effect -- in order to make treatment available to subjects in placebo or others -- but not when early results are inconclusive, then there will be more type 1 error in such testing than there would be if all experiments were run to conclusion. But that doesn't meant the practice is wrong; the cost of type 1 error, morally speaking, might be lower than denying treatment as quickly as one otherwise would for treatments that really would be shown to work at end of full trial. &lt;/p&gt;&#xA;" OwnerUserId="2347" LastEditorUserId="2347" LastEditorDisplayName="" LastEditDate="2011-03-25T13:58:46.677" LastActivityDate="2011-03-25T13:58:46.677" CommentCount="8" />
  <row Id="8768" PostTypeId="2" ParentId="8754" CreationDate="2011-03-25T14:10:28.553" Score="6" ViewCount="" Body="&lt;p&gt;First of all, you have to note the context: this only applies when the trial was stopped early due to interim monitoring showing efficacy/futility, not for some random outside reason. In that case the &lt;em&gt;estimate&lt;/em&gt; of the effect size will be biased in a completely statististical sense. If you stopped for efficacy, the estimated effect will be too high (assuming it is positive), if you stopped for futility, it will be too low. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Piantodosi does give an intuitive explanantion as well (Sec 10.5.4 in my edition). Suppose the true difference in two means is 1 unit. When you run a lot of trials, and look at them at your interim analysis time, some of them will have observed effect sizes much above 1, some much below one, and most around 1 - the distribution will be wide, but symmetric. The estimated effect size at this point would not be very accurate, but would be unbiased. However you only stop and report an effect size if the difference is significant (adjusted for multiple testing), that is the estimate is on the high side. In all other cases you keep going and don't report an estimate. That means that &lt;em&gt;conditional on having stopped early&lt;/em&gt;, the distribution of the effect size is not symmetric, and its expected value is above the true value of the estimate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The fact that this effect is more severe early on comes from the larger hurdle for stopping the trial, thus a larger part of the distribution being thrown away during the conditioning.&lt;/p&gt;&#xA;" OwnerUserId="279" LastEditorUserId="279" LastEditorDisplayName="" LastEditDate="2011-03-25T14:56:53.367" LastActivityDate="2011-03-25T14:56:53.367" CommentCount="10" />
  <row Id="8769" PostTypeId="1" CreationDate="2011-03-25T14:54:47.357" Score="2" ViewCount="43" Body="&lt;p&gt;I am taking a course in data mining. I am not sure how a non linear SVM when transformed to high dimensional space becomes a linear classification problem. It would be good if someone can provide me an intuition on this.&lt;/p&gt;&#xA;" OwnerUserId="3897" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-25T14:58:14.440" LastActivityDate="2011-03-25T15:09:20.313" Title="Linear behaviour of nonlinear SVM in higher dimensional space" Tags="&lt;svm&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="1" />
  <row Id="8770" PostTypeId="2" ParentId="8769" CreationDate="2011-03-25T15:09:20.313" Score="2" ViewCount="" Body="&lt;p&gt;You should look at the problem the other way arround. SVM algorithms solve Linear classification problems in feature space. Depending on the kernel you use, the boundaries in the original space might not be linear. &lt;/p&gt;&#xA;" OwnerUserId="3362" LastActivityDate="2011-03-25T15:09:20.313" />
  <row Id="8771" PostTypeId="1" CreationDate="2011-03-25T15:10:02.870" Score="0" ViewCount="84" Body="&lt;p&gt;Given a factor with a number of levels, say for example, versions of a banner advertisement on a web page, where the measurement of interest is the click through rate (# clicks / # of times banner add was viewed), is there a principled way to determine the best performing ad, controlling for multiple testing? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hsu's &lt;a href=&quot;https://onlinecourses.science.psu.edu/stat502/node/34&quot; rel=&quot;nofollow&quot;&gt;Multiple Comparisons with the Best&lt;/a&gt; appears like the perfect solution (but it works for normally distributed continuous data). Does anyone know of an analog for binary response data? Preferably in R and preferably in a glm context where covariates could be controlled for. This last piece is not vital however as the experiment will be randomized and a one-way situation should be adequate.&lt;/p&gt;&#xA;" OwnerUserId="2040" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-26T17:18:46.097" LastActivityDate="2011-03-26T17:18:46.097" Title="Multiple comparisons with binary data: Hsu's MCB method" Tags="&lt;r&gt;&lt;multiple-comparisons&gt;&lt;binary-data&gt;" CommentCount="5" />
  <row Id="8772" PostTypeId="2" ParentId="8742" CreationDate="2011-03-25T15:13:51.963" Score="0" ViewCount="" Body="&lt;p&gt;@darkcminor: I wonder if the following short tutorial would help you (look especially at the chain rule and the section on inference). I have not looked at these for a long time, but I believe with a few principles you can figure out the values of any query. Some of them will just be onerous done by hand.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.cs.ubc.ca/~murphyk/Bayes/bnintro.html&quot; rel=&quot;nofollow&quot;&gt;http://www.cs.ubc.ca/~murphyk/Bayes/bnintro.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2040" LastActivityDate="2011-03-25T15:13:51.963" />
  <row Id="8773" PostTypeId="2" ParentId="8755" CreationDate="2011-03-25T15:28:23.537" Score="5" ViewCount="" Body="&lt;p&gt;Contrary to @whuber's claim, the mean of x and y are contained in the information given.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Okay, so you have the line equation&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$y_i=\alpha +x_i\beta + e_i$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;estimates $\hat{\beta}=r\frac{s_y}{s_x}$ and $\hat{\alpha}=\overline{y}-\hat{\beta}\overline{x}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $r$ is the correlation.  The question doesn't state whether the standard deviation (0.482) is for $s_y$ or $s_x$ (the MLE standard deviation, with divisor $n$).  Either way, you can work out the either from the info given.  for their ratio must satisfy:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{\hat{\beta}}{r}=\frac{s_y}{s_x}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The slope can't be negative if the correlation is positive, so I have assumed that you have done something incorrectly (for you have correlation of 0.117, and slope of -0.00024; this is impossible).  This will affect the numbers, but not the general method.  So I will assume the standard deviations are both known, but not write in the specific values.  The same goes for the rest of the actual numbers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now the variance of $\hat{\beta}$ is given by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$var(\hat{\beta})=s_e^2(X^TX)^{-1}_{22}=\frac{s_e^2 (X^TX)_{11}}{|X^TX|}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that $(X^TX)_{11}=n$ and $s_e^2$ is the &quot;mean square error&quot;.  The variance of $\alpha$ is given by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$var(\hat{\alpha})=s_e^2(X^TX)^{-1}_{11}=\frac{s_e^2 (X^TX)_{22}}{|X^TX|}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now $(X^TX)_{22}=\sum_i x_i^2 = n(s_x^2+n\overline{x}^2)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And dividing these two variances gives:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{var(\hat{\alpha})}{var(\hat{\beta})}=\frac{(X^TX)_{22}}{(X^TX)_{11}}=\frac{n(s_x^2+n\overline{x}^2)}{n}=s_x^2+n\overline{x}^2$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now all quantities in the equation are known, except for the mean $\overline{x}$.  So we can re-arrange this equation and solve for the mean:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\overline{x}=\pm\sqrt{\frac{\frac{var(\hat{\alpha})}{var(\hat{\beta})}-s_x^2}{n}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But we know from the start that $x_i&amp;gt;0$ - you can't drive &quot;negative miles&quot;.  So only the positive square root is to be taken.  The rest is straight-forward CI stuff.  The estimate of the mean $\hat{\overline{y}}$ is given by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\hat{\overline{y}}=\hat{\alpha}+\hat{\beta}\overline{x}=\hat{\alpha}+\hat{\beta}\sqrt{\frac{\frac{var(\hat{\alpha})}{var(\hat{\beta})}-s_x^2}{n}}=\overline{y}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And the variance is given by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$var(\hat{\overline{y}})=var(\hat{\alpha})+\overline{x}^2 var(\hat{\beta})+2\overline{x}cov(\hat{\alpha},\hat{\beta})$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now the covariance is equal to:&#xA;$$cov(\hat{\alpha},\hat{\beta})=s_e^2(X^TX)^{-1}_{21}=-\frac{s_e^2 (X^TX)_{21}}{|X^TX|}=-\frac{s_e^2 n\overline{x}}{ns_x^2}=-\frac{s_e^2 \overline{x}}{s_x^2}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And so the variance is given by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$var(\hat{\overline{y}})=var(\hat{\alpha})+\overline{x}^2 var(\hat{\beta})-2\frac{s_e^2 \overline{x}^2}{s_x^2}=var(\hat{\alpha})+\frac{\frac{var(\hat{\alpha})}{var(\hat{\beta})}-s_x^2}{n}\left(var(\hat{\beta})-2\frac{s_e^2}{s_x^2}\right)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So you construct your $100(1-P)$% confidence interval by choosing $T_{1-P/2}^{(n-2)}$ as the $P/2$ quantile of standard T distribution with $n-1$ degrees of freedom (which effectively equal to the standard normal, as $n-1=100$), and you have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$CI=\overline{y}\pm T_{1-P/2}^{(n-2)}\sqrt{var(\hat{\overline{y}})}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And all quantities are calculable, given the information.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-03-25T15:28:23.537" CommentCount="3" />
  <row Id="8774" PostTypeId="1" CreationDate="2011-03-25T16:15:37.153" Score="1" ViewCount="107" Body="&lt;p&gt;What is the difference between &lt;code&gt;independence.test&lt;/code&gt; in R and COTT (Cochrane and Armitage) tests?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How these tests are calculated?&#xA;Where do we and how do we define x=0.0 0.5 1.0 (genetic studies) for both of the tests?&lt;/p&gt;&#xA;" OwnerUserId="3870" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-03-30T19:07:47.877" LastActivityDate="2011-05-04T16:55:49.133" Title="What is the difference between independence.test in R and COTT?" Tags="&lt;genetics&gt;&lt;association-measure&gt;" AnswerCount="1" CommentCount="3" FavoriteCount="1" />
  <row Id="8775" PostTypeId="2" ParentId="8754" CreationDate="2011-03-25T16:18:10.147" Score="2" ViewCount="" Body="&lt;p&gt;Here is an illustration of how bias might arise in conclusions, and why it may not be the full story.  Suppose you have a sequential trial of a drug which is expected to have a positive (+1) effect but may have a negative effect (-1).  Five guinea pigs are tested one after the other.  The unknown probability of a positive outcome in a single case is in fact $\frac{3}{4}$ and a negative outcome $\frac{1}{4}$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So after five trials the probabilities of the different outcomes are &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; Outcome     Probability&#xA;+5-0 = +5    243/1024&#xA;+4-1 = +3    405/1024&#xA;+3-2 = +1    270/1024&#xA;+2-3 = -1     90/1024&#xA;+1-4 = -3     15/1024&#xA;+0-5 = -5      1/1024&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;so the probability of a positive outcome overall is 918/1024 = 0.896, and the mean outcome is +2.5. Dividing by the 5 trials, this is an average of a +0.5 outcome per trial. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is the unbiased figure, as it is also $+1\times\frac{3}{4}-1\times\frac{1}{4}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose that in order to protect guinea pigs, the study will be terminated if at any stage the cumulative outcome is negative. Then the probabilities become &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; Outcome     Probability&#xA;+5-0 = +5    243/1024&#xA;+4-1 = +3    324/1024&#xA;+3-2 = +1    135/1024&#xA;+2-3 = -1     18/1024&#xA;+1-2 = -1     48/1024&#xA;+0-1 = -1    256/1024&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;so the probability of a positive outcome overall is 702/1024 = 0.6855, and the mean outcome is +1.953.  If we looked the mean value of outcome per trial in the previous calculation, i.e. using $\frac{+5}{5}$, $\frac{+3}{5}$, $\frac{+1}{5}$, $\frac{-1}{5}$, $\frac{-1}{3}$ and $\frac{-1}{1}$ then we would get +0.184. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;These are the senses in which there is bias by stopping early in the second scheme, and the bias is in the predicted direction.  But it is not the full story.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why do whuber and probabilityislogic think stopping early should produce unbiased results?  We know the expected outcome of the trials in the second scheme is +1.953.  The expected number of trials turns out to be 3.906.  So dividing one by the other we get +0.5, exactly as before and what was described as unbiased.&lt;/p&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-03-25T16:18:10.147" CommentCount="2" />
  <row Id="8776" PostTypeId="2" ParentId="8718" CreationDate="2011-03-25T16:37:41.953" Score="0" ViewCount="" Body="&lt;p&gt;Another tool that may be useful is the standardarized regression coefficient, or at least a rough-and-ready pseudo-version.  You can obtain one such version by multiplying your obtained coefficient by the standard deviation of the predictor.  (There are other versions and some debate about the best one, e.g. see Menard 2002, &lt;a href=&quot;http://books.google.com/books?id=EAI1QmUUsbUC&amp;amp;printsec=frontcover&amp;amp;dq=menard+logistic&amp;amp;source=bl&amp;amp;ots=4SBKH0mVJS&amp;amp;sig=br7joGc43N7NEDIAIU7ItVCTPpU&amp;amp;hl=en&amp;amp;ei=rcOMTb6ZBI6y0QHD24W3Cw&amp;amp;sa=X&amp;amp;oi=book_result&amp;amp;ct=result&amp;amp;resnum=3&amp;amp;ved=0CC0Q6AEwAg&quot; rel=&quot;nofollow&quot;&gt;http://books.google.com/books?id=EAI1QmUUsbUC&amp;amp;printsec=frontcover&amp;amp;dq=menard+logistic&amp;amp;source=bl&amp;amp;ots=4SBKH0mVJS&amp;amp;sig=br7joGc43N7NEDIAIU7ItVCTPpU&amp;amp;hl=en&amp;amp;ei=rcOMTb6ZBI6y0QHD24W3Cw&amp;amp;sa=X&amp;amp;oi=book_result&amp;amp;ct=result&amp;amp;resnum=3&amp;amp;ved=0CC0Q6AEwAg&lt;/a&gt;  ). This will give you a way to assess the strength of the effect across studies.&lt;/p&gt;&#xA;" OwnerUserId="2669" LastActivityDate="2011-03-25T16:37:41.953" />
  <row Id="8777" PostTypeId="1" CreationDate="2011-03-25T16:39:10.923" Score="4" ViewCount="159" Body="&lt;p&gt;In genome wide association studies:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;What are the Principal Components&lt;/li&gt;&#xA;&lt;li&gt;Why are they used? &lt;/li&gt;&#xA;&lt;li&gt;How are they calculated?&lt;/li&gt;&#xA;&lt;li&gt;Can genome wide association studies be done without Principal Components?&lt;/li&gt;&#xA;&lt;li&gt;Are there any freely available real or simulated datasets relevant to performing genome wide association studies?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="3870" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-04T12:47:25.180" LastActivityDate="2011-04-04T12:47:25.180" Title="In genome wide association studies, what are principal components?" Tags="&lt;pca&gt;&lt;genetics&gt;" AnswerCount="1" CommentCount="6" FavoriteCount="2" />
  <row Id="8778" PostTypeId="2" ParentId="8380" CreationDate="2011-03-25T16:48:20.410" Score="1" ViewCount="" Body="&lt;p&gt;One approach would be to use the second dataset to make a secondary model that predicts the exact variables as a function of the noisy variables and use this in operation to provide the inputs for the primary model trained on the first dataset (which predicts the target given the exact variables).  However, to do this properly you would want to propogate the uncertainty of the secondary model through the primary model when making predictions.  Say you used a multivariate regression model (as the uncertainty in the conditional estimates of the exact variables are unlikely to be uncorrellated) you would have a multi-variate normal distribution for the plausible values of the exact variables conditioned on the noisy ones.  This could then be sampled a few thousand times and the output of the primary model averaged over that sample to get a more robust estimate for the target.  I suspect if you kept everything linear and normal there would be an analytic solution so you didn't have to sample.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The regression may at least help with the bias of the noise, if nothing else.&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2011-03-25T16:48:20.410" />
  <row Id="8779" PostTypeId="1" CreationDate="2011-03-25T18:35:44.733" Score="8" ViewCount="134" Body="&lt;p&gt;This problem is actually about fire detection, but it is strongly analogous to some radioactive decay detection problems. The phenomena being observed is both sporadic and highly variable; thus, a time series will consist of long strings of zeroes interrupted by variable values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The objective is not merely capturing events (breaks in the zeroes), but quantitative characterization of the events themselves. However, the sensors are limited, and thus will sometimes record zero even if the &quot;reality&quot; is non-zero. For this reason, zeroes must be included when comparing sensors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sensor B might be more sensitive than Sensor A, and I would like to be able to describe that statistically. For this analysis, I do not have &quot;truth,&quot; but I do have a Sensor C, which is independent of Sensors A&amp;amp;B. Thus my expectation is that better agreement between A/B and C indicates better agreement with &quot;truth.&quot; (This may seem shaky, but you'll have to trust me-- I'm on solid ground here, based on what is known from other studies about the sensors).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem, then, is how to quantify &quot;better agreement of time series.&quot; Correlation is the obvious choice, but will be affected by all those zeroes (which cannot be left out), and of course disproportionately affected by the maximum values. RMSE could also be calculated, but would be strongly weighted toward the behavior of the sensors in the near-zero case. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Q1: What is the best way to apply a logarithmic scaling to non-zero values that will then be combined with zeroes in a time-series analysis?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Q2: What &quot;best practices&quot; can you recommend for a time-series analysis of this type, where behavior at non-zero values is the focus, but zero values dominate and cannot be excluded?&lt;/p&gt;&#xA;" OwnerUserId="3898" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-26T00:43:20.430" LastActivityDate="2011-04-28T10:40:21.957" Title="Analysis of time series with many zero values" Tags="&lt;time-series&gt;&lt;correlation&gt;" AnswerCount="1" FavoriteCount="2" />
  <row Id="8780" PostTypeId="2" ParentId="8777" CreationDate="2011-03-25T19:32:21.563" Score="5" ViewCount="" Body="&lt;p&gt;In this particular context, PCA is mainly used to account for population-specific variations in alleles distribution on the SNPs (or other DNA markers, although I'm only familiar with the SNP case) under investigation. Such &quot;population substructure&quot; mainly arises as a consequence of varying frequencies of minor alleles in genetically distant ancestries (e.g. japanese and black-african or european-american). The general idea is well explained in &lt;a href=&quot;http://www.genome.duke.edu/education/seminars/journal-club/documents/PopStructure.pdf&quot; rel=&quot;nofollow&quot;&gt;Population Structure and Eigenanalysis&lt;/a&gt;, by Patterson et al. (&lt;em&gt;PLoS Genetics&lt;/em&gt; 2006, 2(12)), or the &lt;em&gt;Lancet&lt;/em&gt;'s special issue on genetic epidemiology (2005, 366; most articles can be found on the web, start with Cordell &amp;amp; Clayton, &lt;a href=&quot;http://www.molmed.nl/uploads/abstracts/127/Lancet%20genetic%20epi%203.pdf&quot; rel=&quot;nofollow&quot;&gt;Genetic Association Studies&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The construction of principal axes follows from the classical approach to PCA, which is applied to the scaled matrix (individuals by SNPs) of observed genotypes (AA, AB, BB; say B is the minor allele in all cases), to the exception that an additional normalization to account for population drift might be applied. It all assumes that the frequency of the minor allele (taking value in {0,1,2}) can be considered as numeric, that is we work under an &lt;em&gt;additive model&lt;/em&gt; (also called allelic dosage) or any equivalent one that would make sense. As the successive orthogonal PCs will account for the maximum variance, this provides a way to highlight groups of individuals differing at the level of minor allele frequency. The software used for this is known as &lt;a href=&quot;http://genepath.med.harvard.edu/~reich/Software.htm&quot; rel=&quot;nofollow&quot;&gt;Eigenstrat&lt;/a&gt;. It is also available in the &lt;code&gt;egscore()&lt;/code&gt; function from the &lt;a href=&quot;http://cran.r-project.org/web/packages/GenABEL/&quot; rel=&quot;nofollow&quot;&gt;GenABEL&lt;/a&gt; R package (see also &lt;a href=&quot;http://www.genabel.org/&quot; rel=&quot;nofollow&quot;&gt;GenABEL.org&lt;/a&gt;). It is worth to note that other methods to detect population substructure were proposed, in particular model-based cluster reconstruction (see references at the end). More information can be found by browsing the &lt;a href=&quot;http://hapmap.ncbi.nlm.nih.gov/&quot; rel=&quot;nofollow&quot;&gt;Hapmap&lt;/a&gt; project, and available tutorial coming from the &lt;a href=&quot;http://www.bioconductor.org&quot; rel=&quot;nofollow&quot;&gt;Bioconductor&lt;/a&gt; project. (Search for Vince J Carey or David Clayton's nice tutorials on Google). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Apart from clustering subpopulations, this approach can also be used for detecting outliers which might arise in two cases (AFAIK): (a) genotyping errors, and (b) when working with an homogeneous population (or assumed so, given self-reported ethnicity), individuals exhibiting unexpected genotype. What is usually done in this case is to apply PCA in an iterative manner, and remove individuals whose scores are below $\pm 6$ SD on at least one of the first 20 principal axes; this amounts to &quot;whiten&quot; the sample, in some sense. Note that any such measure of genotype distance (this also holds when using Multidimensional Scaling in place of PCA) will allow to spot relatives or siblings. The &lt;a href=&quot;http://pngu.mgh.harvard.edu/~purcell/plink/&quot; rel=&quot;nofollow&quot;&gt;plink&lt;/a&gt; software provides additional methods, see the section on &lt;a href=&quot;http://pngu.mgh.harvard.edu/~purcell/plink/strat.shtml&quot; rel=&quot;nofollow&quot;&gt;Population stratification&lt;/a&gt; in the on-line help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Considering that eigenanalysis allows to uncover some structure at the level of the individuals, we can use this information when trying to explain observed variations in a given phenotype (or any distribution that might be defined according to a binary criterion, e.g. disease or case-control situation). Specifically, we can adjust our analysis with those PCs (i.e., the factor scores of individuals), as illustrated in &lt;a href=&quot;http://www.biostat.jhsph.edu/~iruczins/teaching/misc/gwas/papers/price2006.pdf&quot; rel=&quot;nofollow&quot;&gt;Principal components analysis corrects for stratification in genome-wide association studies&lt;/a&gt;, by Price et al. (&lt;em&gt;Nature Genetics&lt;/em&gt; 2006, 38(8)), and later work (there was a nice picture showing axes of genetic variation in Europe, but I can't find it actually). Note also that another solution is to carry out a stratified analysis (by including ethnicity in an GLM)--this is readily available in the &lt;a href=&quot;http://www.bioconductor.org/packages/2.3/bioc/html/snpMatrix.html&quot; rel=&quot;nofollow&quot;&gt;snpMatrix&lt;/a&gt; package, for example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;References&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Daniel Falush, Matthew Stephens, and Jonathan K Pritchard (2003). &lt;a href=&quot;http://pritch.bsd.uchicago.edu/software/FalushEtAl03.pdf&quot; rel=&quot;nofollow&quot;&gt;Inference of population structure using multilocus genotype data: linked loci and correlated allele frequencies&lt;/a&gt;. &lt;em&gt;Genetics&lt;/em&gt;, 164(4): 1567–1587.&lt;/li&gt;&#xA;&lt;li&gt;B Devlin and K Roeder (1999). &lt;a href=&quot;http://www.biostat.jhsph.edu/~iruczins/teaching/misc/gwas/papers/devlin1999.pdf&quot; rel=&quot;nofollow&quot;&gt;Genomic control for association studies&lt;/a&gt;. &lt;em&gt;Biometrics&lt;/em&gt;, 55(4): 997–1004.&lt;/li&gt;&#xA;&lt;li&gt;JK Pritchard, M Stephens, and P Donnelly (2000). &lt;a href=&quot;http://pritch.bsd.uchicago.edu/publications/structure.pdf&quot; rel=&quot;nofollow&quot;&gt;Inference of population structure using multilocus genotype data&lt;/a&gt;. &lt;em&gt;Genetics&lt;/em&gt;, 155(2): 945–959.&lt;/li&gt;&#xA;&lt;li&gt;Gang Zheng, Boris Freidlin, Zhaohai Li, and Joseph L Gastwirth (2005). &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.167.8448&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;Genomic control for association studies under various genetic models&lt;/a&gt;.   &lt;em&gt;Biometrics&lt;/em&gt;, 61(1): 186–92.    &lt;/li&gt;&#xA;&lt;li&gt;Chao Tian, Peter K. Gregersen, and Michael F. Seldin1 (2008). &lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pubmed/18852203&quot; rel=&quot;nofollow&quot;&gt;Accounting for ancestry: population substructure and genome-wide association studies&lt;/a&gt;. &lt;em&gt;Human Molecular Genetics&lt;/em&gt;, 17(R2): R143-R150.&lt;/li&gt;&#xA;&lt;li&gt;Kai Yu, &lt;a href=&quot;http://biowulf.nih.gov/symposium/slides/Kai_Yu.pdf&quot; rel=&quot;nofollow&quot;&gt;Population Substructure and Control Selection in Genome-wide Association Studies&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;li&gt;Alkes L. Price, Noah A. Zaitlen, David Reich and Nick Patterson (2010). &lt;a href=&quot;http://genetics.med.harvard.edu/reich/Reich_Lab/Publications_files/2010_Price_stratificationreview.pdf&quot; rel=&quot;nofollow&quot;&gt;New approaches to population stratification in genome-wide association studies&lt;/a&gt;, &lt;em&gt;Nature Reviews Genetics&lt;/em&gt;&lt;/li&gt;&#xA;&lt;li&gt;Chao Tian, et al. (2009). &lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2730349/&quot; rel=&quot;nofollow&quot;&gt;European Population Genetic Substructure: Further Definition of Ancestry Informative Markers for Distinguishing among Diverse European Ethnic Groups&lt;/a&gt;, Molecular Medicine, 15(11-12): 371–383.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-04T09:12:52.413" LastActivityDate="2011-04-04T09:12:52.413" CommentCount="4" />
  <row Id="8781" PostTypeId="2" ParentId="4276" CreationDate="2011-03-25T20:20:58.920" Score="1" ViewCount="" Body="&lt;p&gt;Since there is quite some variation in how the steps look,&#xA;you could try a statistical approach. Which could, for example, be done&#xA;in the following steps:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Generate the feature vector.&#xA;Filter the signal by a number of filters, each having a different&#xA;frequency response. A set foo (haar)-wavelets might be a reasonable&#xA;starting points. if your original signal has N samples, and you have K filters,&#xA;this filtering should result in an N-by-K matrix. take the element-wise square to determine the energy in each of the signals.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Generate ground truth.&#xA;Write down the sample numbers which mark the start of each step, store them in vector S.&#xA;Use this to make ground truth output data: Y = zeros(N,1); Y(S) = 1.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Train your classifier.&#xA;Now you can apply a generic classification algorithm (e.g. LDA or logistic regression) to the results of step 1 and 2. Matlab implementations should not be hard to find.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Apply you classifier on new data.&#xA;For new data, repeat step 1. this can then be used to as input for the classifier resulting from step 3. It might be necessary to post-process this data, for example by low-pass filtering it. Setting a threshold with some hysteresis should then give you the start of each step&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="3867" LastActivityDate="2011-03-25T20:20:58.920" />
  <row Id="8782" PostTypeId="2" ParentId="8779" CreationDate="2011-03-25T23:55:45.633" Score="3" ViewCount="" Body="&lt;p&gt;To restate your question “ How does the analyst deal with long periods of no demand that follow no specific pattern?”&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The answer to your question is Intermittent Demand Analysis or Sparse Data Analysis. This arises normally when you have &quot;lots of zeros&quot; relative to the number of non-zeros.The issue is that there are two random variables ; the time between events and the expected size of the event. As you said the autocorrelation (acf) of the complete set of readings is meaningless due to the sequence of zeroes falsely enhancing the acf. You can pursue threads like &quot;Croston's method”  which is a model-based procedure rather than a data-based procedure. Croston's method is vulnerable to outliers and changes/trends/level shifts in the rate of demand i.e. the demand divided by the number of periods since the last demand. A much more rigorous approach might be to pursue &quot;Sparse Data - Unequally Spaced Data&quot; or searches like that. A rather ingenious solution was suggested to me by Prof. Ramesh Sharda of OSU and I have been using it for a number of years in my consulting practice. &#xA; If a series has time points where sales arise and long periods of time where no sales arise it is possible to convert sales to sales per period by dividing the observed sales by the number of periods of no sales thus obtaining a rate. It is then possible to identify a model between rate and the interval between sales culminating in a forecasted rate and a forecasted interval. You can find out more about this at autobox.com and google &quot;intermittent demand&quot;&lt;/p&gt;&#xA;" OwnerUserId="3382" LastActivityDate="2011-03-25T23:55:45.633" CommentCount="3" />
  <row Id="8783" PostTypeId="1" CreationDate="2011-03-26T00:33:57.720" Score="0" ViewCount="111" Body="&lt;p&gt;I am wondering that how one can calculate &lt;a href=&quot;http://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence&quot; rel=&quot;nofollow&quot;&gt;KL-divergence&lt;/a&gt; on two probability distributions. For example, if we have&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;t1 = 0.4, 0.2, 0.3, 0.05, 0.05&#xA;t2 = 0.23, 0, 0.14, 0.17&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The formula is bit complicated for me :( &lt;/p&gt;&#xA;" OwnerUserId="3900" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-04-08T20:44:20.170" LastActivityDate="2011-04-08T20:44:20.170" Title="KL divergence calculation" Tags="&lt;distributions&gt;&lt;machine-learning&gt;&lt;distance-functions&gt;&lt;information-retrieval&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="8784" PostTypeId="1" CreationDate="2011-03-26T01:07:46.037" Score="9" ViewCount="393" Body="&lt;p&gt;This is a follow up question from &lt;a href=&quot;http://stats.stackexchange.com/questions/8718/comparing-logistic-regression-coefficients-across-models/&quot;&gt;the one I asked a couple of days ago&lt;/a&gt;. I feel it puts a different slant on the issue, so listed a new question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The question is: can I compare the magnitude of coefficients across models with different dependent variables? For example, on a single sample say I want to know whether the economy is a stronger predictor of votes in the House of Representatives or for President. In this case, my two dependent variables would be the vote in the House (coded 1 for Democrat and 0 for Republican) and vote for President (1 for Democrat and 0 for Republican) and my independent variable is the economy. I'd expect a statistically significant result in both offices, but how do I assess whether it has a 'bigger' effect in one more than the other? This might not be a particularly interesting example, but i'm curious about whether there is a way to compare. I know one can't just look at the 'size' of the coefficient. So, is comparing coefficients on models with different dependent variables possible? And, if so, how can it be done?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If any of this doesn't make sense, let me know. All advice and comments are appreciated.&lt;/p&gt;&#xA;" OwnerUserId="3883" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-03-26T09:38:23.173" LastActivityDate="2011-04-28T00:05:39.210" Title="Comparing logistic coefficients on models with different dependent variables?" Tags="&lt;regression&gt;&lt;logistic-regression&gt;" AnswerCount="4" CommentCount="4" FavoriteCount="2" />
  <row Id="8785" PostTypeId="2" ParentId="8783" CreationDate="2011-03-26T01:09:49.170" Score="1" ViewCount="" Body="&lt;p&gt;Using brute force and the first formula &lt;a href=&quot;http://en.wikipedia.org/wiki/Jensen%E2%80%93Shannon_divergence&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; based on the first formula for the &lt;a href=&quot;http://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence&quot; rel=&quot;nofollow&quot;&gt;Kullback-Leibler divergence&lt;/a&gt;, you are starting from two multisets each with 5 values, 3 of which are shared between them.  So the combination of them is the multiset &#xA;$$M={0, 0.05, 0.05, 0.1, 0.2, 0.2, 0.3, 0.3, 0.4, 0.4}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;so using $D_{\mathrm{KL}}(P\|Q) = \sum_i P(i) \log \frac{P(i)}{Q(i)}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$JSD(t_1 \parallel t_2)= \frac{1}{2}D_{\mathrm{KL}}(t_1 \parallel M)+\frac{1}{2}D_{\mathrm{KL}}(t_2 \parallel M)$$&#xA;$$=\frac{1}{2}\left(1\cdot\frac{2}{5} \log\left(\frac{2/5}{2/10}\right) +3\cdot\frac{1}{5} \log\left(\frac{1/5}{2/10}\right)\right)  $$&#xA;$$+\frac{1}{2}\left(2\cdot\frac{1}{5} \log\left(\frac{1/5}{1/10}\right) +3\cdot\frac{1}{5} \log\left(\frac{1/5}{2/10}\right)\right)  $$&#xA;$$= \dfrac{2}{5}\log(2) \approx 0.277$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;though you may want to check this. Other calculations, such as using Shannon entropy should produce the same result. &lt;/p&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-03-26T01:09:49.170" CommentCount="3" />
  <row Id="8786" PostTypeId="2" ParentId="8749" CreationDate="2011-03-26T05:43:34.727" Score="2" ViewCount="" Body="&lt;p&gt;I would use probability theory to start with, and then pick whichever algorithm best calculates what probability theory tells you to do.  So you have training data $T$, and some new precursors $X$, and an object to classify $Y$, as well as your prior information $I$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So you want to know about $Y$.  Then probability theory says, just calculate its probability, conditional on all the information you have available to you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(Y|T,X,I)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now we can use any of the rules of probability theory to manipulate this into things that we do know how to calculate.  So using Bayes theorem, you get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(Y|T,X,I)=\frac{P(Y|T,I)P(X|Y,T,I)}{P(X|T,I)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now $P(Y|T,I)$ is usually easy - unless you prior information can tell you something about $Y$ beyond the training data (e.g. correlations), then it is given by the rule of succession - or basically the observed fraction of times $Y$ was true in the training data set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the second term $P(X|Y,T,I)$ - this is your model, and where most of your work will go, and where different algorithms will do different things.  $P(X|T,I)$ is a bit of a vicious beast to calculate, so we do the following trick to avoid having to do this: take the odds of $Y$ against $\overline{Y}$ (i.e. not $Y$). And we get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$O(Y|T,X,I)=\frac{P(Y|T,X,I)}{P(\overline{Y}|T,X,I)}=\frac{P(Y|T,I)}{P(\overline{Y}|T,I)}\frac{P(X|Y,T,I)}{P(X|\overline{Y},T,I)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now you basically need a decision rule - when the odds/probability is above a certain threshold, you will classify $Y$ as &quot;true&quot;, otherwise you will classify it as &quot;false&quot;.  Now nobody can really help you with this - it is a decision which depends on the consequences of making right and wrong decisions.  This is a subjective exercise, and only the proper context can answer this.  Of course the &quot;subjectivity&quot; will only matter if there is high uncertainty (i.e. you have a &quot;crap&quot; model/data which can't distinguish the two very well).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second quantity - the model $P(X|Y,T,I)$ is a &quot;predictive&quot; model.  Suppose the prior information indicates a single model which depends on parameter $\theta_{Y}$.  Then the quantity is given by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(X|Y,T,I)=\int P(X,\theta_{Y}|Y,T,I) d\theta = \int P(X|\theta_{Y},Y,T,I)P(\theta_{Y}|Y,T,I) d\theta_{Y}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now if your model is of the &quot;iid&quot; variety, then $P(X|\theta_{Y},Y,T,I)=P(X|\theta_{Y},Y,I)$.  But if you have a dependent model, such as an autoregressive one, then $T$ may still matter.  And $P(\theta_{Y}|Y,T,I)$ is the posterior distribution for the parameters in the model - this is the part that the training data would determine.  And this is probably where most of the work will go.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But what if the model is not known with certainty?  well it just becomes another nuisance parameter to integrate out, just as was done for $\theta_{Y}$.  Call the ith model $M_i$ and its set of parameters $\theta^{(i)}_{Y}$, and the equation becomes:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(X|Y,T,I)= \sum_{i}P(M_{i}|Y,T,I)\int P(X|\theta_{Y}^{(i)},M_{i},Y,T,I)P(\theta_{Y}^{(i)}|M_{i},Y,T,I) d\theta_{Y}^{(i)}$$&#xA;Where&#xA;$$P(M_{i}|Y,T,I)=P(M_{i}|Y,I)\int P(\theta_{Y}^{(i)}|M_{i},Y,I)P(T|\theta_{Y}^{(i)},M_{i},Y,I) d\theta_{Y}^{(i)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(NOTE: $M_i$ is a proposition of the form &quot;the ith model is the best in the set that is being considered&quot;. and no improper priors allowed if you are integrating over models - the infinities do not cancel out in this case, and you will be left with non-sense)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, up to this point, all results are exact and optimal (this is the option 2 - apply some awesome algorithm to the data).  But this a daunting task to undertake.  In the real world, the mathematics required may be not feasible to do in practice - so you will have to compromise.  you should always &quot;have a go&quot; at doing the exact equations, for any maths that you can simplify will save you time at the PC.  However, this first step is important, because this sets &quot;the target&quot;, and it makes it clear what is to be done.  Otherwise you are left (as you seem to be) with a whole host of potential options with nothing to choose between them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now at this stage, we are still in &quot;symbolic logic&quot; world, where nothing really makes sense.  So you need to link these to your specific problem:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;$P(M_{i}|Y,I)$ is the prior probability for the ith model - generally will be equal for all i.&lt;/li&gt;&#xA;&lt;li&gt;$P(\theta_{Y}^{(i)}|M_{i},Y,I)$ is the prior for the parameters in the ith model (must be proper!)&lt;/li&gt;&#xA;&lt;li&gt;$P(T|\theta_{Y}^{(i)},M_{i},Y,I)$ is the likelihood function for the training data, given the ith model&lt;/li&gt;&#xA;&lt;li&gt;$P(\theta_{Y}^{(i)}|T,M_{i},Y,I)$ is the posterior for the parameters in the ith model, conditional on the training data.&lt;/li&gt;&#xA;&lt;li&gt;$P(M_{i}|Y,T,I)$ is the posterior for the ith model conditional on the training data&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;There will be another set of equations for $\overline{Y}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that the equations will simplify enormously if a) one model is a clear winner, so that $P(M_{j}|Y,T,I)\approx 1$ and b) within this model, its parameters are very accurate, so the integrand resembles a delta function (and integration is very close to substitution or plug-in estimates).  If both these conditions are met you have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(X|Y,T,I)\approx P(X|\theta_{Y}^{(j)},M_{j},Y,T,I)_{\theta_{Y}^{(j)}=\hat{\theta}_{Y}^{(j)}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which is the &quot;standard&quot; approach to this kind of problem.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastEditorUserId="2392" LastEditorDisplayName="" LastEditDate="2011-03-27T00:34:48.983" LastActivityDate="2011-03-27T00:34:48.983" />
  <row Id="8787" PostTypeId="1" CreationDate="2011-03-26T05:58:44.600" Score="4" ViewCount="75" Body="&lt;p&gt;I'm looking for any reference about Generalized Linear Latent and Mixed Model (GLLAMM) for Crossed Factors involving both Measurement Model and Structured Model of GLLAMM (See the problem below). Any help in this regard will be highly appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Problem:&lt;/strong&gt; A researcher observed four responses Y1, Y2, Y3, and Y4 along with three covariates X1, X2, and X3 from an experiment involving ab treatment combinations from a fixed factor A with a levels and a random factor B with b levels. Based on past experience, it is assumed four responses are correlated and Y1 is also influenced by the other three (Y2, Y3, and Y4). This data set can be analyzed by &lt;a href=&quot;http://www.gllamm.org/&quot; rel=&quot;nofollow&quot;&gt;GLLAMM&lt;/a&gt; in Stata.&lt;/p&gt;&#xA;" OwnerUserId="3903" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-26T17:17:30.030" LastActivityDate="2011-03-26T17:17:30.030" Title="Generalized linear latent and mixed model (GLLAMM) for crossed factors" Tags="&lt;mixed-model&gt;&lt;stata&gt;" CommentCount="2" FavoriteCount="1" />
  <row Id="8788" PostTypeId="1" CreationDate="2011-03-25T11:57:14.800" Score="2" ViewCount="217" Body="&lt;p&gt;Hey guys!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a problem when I run the Komogorov-Smirnov test.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have to samples of daily prices distributions estimated with density(). Now I would like to compare these two distributions with each other.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;data.1:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Date           price&#xA;01.01.2010     1.2&#xA;02.01.2010     1.5&#xA;etc.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;data.2:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Date           price&#xA;01.01.2009     0.1&#xA;02.01.2009     0.05&#xA;etc.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For the probability density, I calculated&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;density.1 &amp;lt;- density(data.1$price)&#xA;density.2 &amp;lt;- density(data.2$price)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now I wanted to run the KS-test:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ks &amp;lt;- ks.test(density.1$x, density.2$x)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and got the results that p=1, hence the two distributions are the same. However, it is already observable from eye that they differ quite heavily from each other.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where is my mistake?&#xA;Thank you, Dani&lt;/p&gt;&#xA;" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-04-08T20:44:41.933" LastActivityDate="2011-04-08T20:44:41.933" Title="Goodness-of-fit test using Kolmogorov-Smirnov" Tags="&lt;r&gt;&lt;distributions&gt;" AnswerCount="2" CommentCount="0" />
  <row Id="8789" PostTypeId="2" ParentId="8788" CreationDate="2011-03-25T12:23:49.427" Score="4" ViewCount="" Body="&lt;p&gt;ks.test receives &lt;strong&gt;values&lt;/strong&gt;, not densities. So you don't need to call density().&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Probably what you should do is simply:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ks.test(data.1$return, data.2$return)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and the reason why you get p=1 is that you passed &lt;code&gt;return.density.1$x&lt;/code&gt; instead of &lt;code&gt;return.density.1$y&lt;/code&gt;.&#xA;&lt;code&gt;density(foo)$x&lt;/code&gt; is &lt;em&gt;the n coordinates of the points where the density is estimated.&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="2280" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-26T14:54:51.527" LastActivityDate="2011-03-26T14:54:51.527" />
  <row Id="8790" PostTypeId="2" ParentId="8788" CreationDate="2011-03-25T12:29:53.250" Score="7" ViewCount="" Body="&lt;p&gt;First of all, you don't calculate the ks on an estimated density, as the ks test works on the empiric cumulative distribution function (ecdf). So you add the raw data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ks.test(data.1$return, data.2$return)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Second, the &lt;code&gt;$x&lt;/code&gt; is not the density, but the &lt;strong&gt;uniformly distributed&lt;/strong&gt; grid constructed by the density function. So off course they are rather alike if the means are alike.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x &amp;lt;- rnorm(100,3)&#xA;y &amp;lt;- runif(100,min(x),max(x))&#xA;xx &amp;lt;- density(x)$x&#xA;yy &amp;lt;- density(y)$x&#xA;ks.test(xx,yy)&#xA;qqplot(xx,yy)&#xA;ks.test(x,y)&#xA;qqplot(x,y)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Last, please read in on a test if you use it before using it. Many mistakes in statistics are made by people that have no clue what they're actually doing. I don't say this to be rude, I just see things like this happen on an almost daily basis...&lt;/p&gt;&#xA;" OwnerUserId="1124" LastActivityDate="2011-03-25T12:29:53.250" />
  <row Id="8791" PostTypeId="1" CreationDate="2011-03-26T01:54:14.677" Score="6" ViewCount="48" Body="&lt;p&gt;The best way I can think to describe this question is by example: Imagine there is a ship sailing around the pacific ocean on an unknown path (possibly random.) Other ships passing by sometimes see this ship and radio in its location to me. Some of these scout ships have better instruments or a more trustworthy crew than others, so I assign an accuracy weight to each of them. If the ship were static, it would be a simple problem to collect all of the reports and calculate an area where the ship is located (with high probability.) How can I adapt this for when the ship is constantly moving (assuming a fixed speed.) Obviously newer reports need to be given more weight and older reports need to be &quot;faded out&quot; so the ship's calculated location changes over time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I started trying to design something like this but I think I'm making it too complicated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a name for this sort of problem? Any suggestions for a good method to solve it? Thanks!&lt;/p&gt;&#xA;" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-30T19:16:40.193" LastActivityDate="2011-03-30T19:16:40.193" Title="Weighted discrete measurements of a value changing over time" Tags="&lt;time-series&gt;&lt;predictive-models&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="1" />
  <row Id="8792" PostTypeId="2" ParentId="8791" CreationDate="2011-03-26T01:59:01.030" Score="2" ViewCount="" Body="&lt;p&gt;Sounds like you might want to look at &lt;a href=&quot;http://en.wikipedia.org/wiki/Moving_average&quot; rel=&quot;nofollow&quot;&gt;(Weighted) Moving Average&lt;/a&gt;.&lt;/p&gt;&#xA;" LastActivityDate="2011-03-26T01:59:01.030" CommentCount="5" />
  <row Id="8793" PostTypeId="2" ParentId="8784" CreationDate="2011-03-26T10:34:21.970" Score="1" ViewCount="" Body="&lt;p&gt;Let us say the interest lies in comparing two groups of people: those with $X_{1} = 1$ and those with $X_{1} = 0$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The exponential of $\beta_{1}$, the corresponding coefficient, is interpreted as the ratio of the odds of success for those with $X_{1} = 1$ over the odds of success for those with $X_{1} = 0$, &lt;strong&gt;conditional on&lt;/strong&gt; the other variables in the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, if you have two models with different dependend variables then the interpretation of $\beta_{1}$ changes since it is not conditioned upon the same set of variables. As a consequence, the comparison is not direct...&lt;/p&gt;&#xA;" OwnerUserId="3019" LastActivityDate="2011-03-26T10:34:21.970" CommentCount="2" />
  <row Id="8794" PostTypeId="2" ParentId="8784" CreationDate="2011-03-26T12:21:03.737" Score="1" ViewCount="" Body="&lt;p&gt;I assume that by &quot;my independent variable is the economy&quot; you're using shorthand for some specific predictor.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At one level, I see nothing wrong with making a statement such as &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;X predicts Y1 with an odds ratio of _ and a 95% confidence interval of [ _ , _ ]&#xA;   while &#xA;  X predicts Y2 with an odds ratio of _ and a 95% confidence interval of [ _ , _ ].&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;@dmk38's recent suggestions look very helpful in this regard. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might also want to standardize the coefficients to facilitate comparison.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At another level, beware of taking inferential statistics (standard errors, &lt;em&gt;p&lt;/em&gt;-values, CIs) literally when your sample constitutes a nonrandom sample of the population of years to which you might want to generalize.&lt;/p&gt;&#xA;" OwnerUserId="2669" LastEditorUserId="2669" LastEditorDisplayName="" LastEditDate="2011-04-23T15:38:48.887" LastActivityDate="2011-04-23T15:38:48.887" CommentCount="2" />
  <row Id="8795" PostTypeId="1" AcceptedAnswerId="8809" CreationDate="2011-03-26T13:29:21.417" Score="5" ViewCount="185" Body="&lt;p&gt;If you flip a coin and get 268 heads and 98 tails, you can calculate the probability that coin is fair several ways. A simple, heuristic observation would have most likely conclude that such a coin is unfair. I've calculated the p-value in R with:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; coin &amp;lt;- pbinom(98, 366, 0.5)&#xA;&amp;gt; coin*2&#xA;[1] 2.214369e-19&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This value is smaller than .05, ergo we reject the hypothesis that it's a fair coin.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But what if you where told that the same coin landed on its side 676 times during the trial. Heuristically you'll likely come to the same conclusion, but would the typical fair coin tests still be valid?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a graph to illustrate the problem:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/Lz6Ix.png&quot; width=&quot;500&quot; height=&quot;500&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are valid methods to test the hypothesis that there is equal probability that an event occurs in the shaded areas?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;NOTE: there are 629 positive moves (413 negative) in the graph illustration. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;R code that generates the data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(&quot;quantmod&quot;)&#xA;&#xA;ticker &amp;lt;- getSymbols(&quot;SLV&quot;)[,6]&#xA;&#xA;change &amp;lt;- (ticker - lag(ticker, 24)) / lag(ticker, 24)  &#xA;change &amp;lt;- na.locf(change, na.rm=TRUE)   &#xA;&#xA;# some other calculations&#xA;&#xA;dens &amp;lt;- density(change)&#xA;plot(dens)&#xA;&#xA;# some formatting stuff&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="3306" LastEditorUserId="3306" LastEditorDisplayName="" LastEditDate="2011-03-27T23:31:20.117" LastActivityDate="2011-03-27T23:31:20.117" Title="Can a fair coin test be applied to a coin that often lands on its edge?" Tags="&lt;probability-theory&gt;" AnswerCount="2" CommentCount="10" FavoriteCount="1" />
  <row Id="8796" PostTypeId="2" ParentId="5997" CreationDate="2011-03-26T14:30:20.593" Score="1" ViewCount="" Body="&lt;p&gt;You can try playing with spam filtering, that's quite a common use of Naive Bayesian Classifiers.&lt;/p&gt;&#xA;" OwnerUserId="3442" LastActivityDate="2011-03-26T14:30:20.593" />
  <row Id="8797" PostTypeId="1" CreationDate="2011-03-26T15:37:23.837" Score="1" ViewCount="157" Body="&lt;p&gt;I've made a little questionnaire where participants can rate an answer between 1 and 5. I calculated the mean value, the average value and the standard deviation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I was asking myself if it is possible to calculated a confidence interval for these results and if yes, if this would tell me anything. So I just tested it and used excel to calculate a 95% confidence interval.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are the values:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Arithmetic average: 4.60&#xA;Median: 5.00&#xA;Max: 5.00&#xA;Min: 3.00&#xA;Standard deviation: 0.63&#xA;95% Confidence interval: 0.32&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But what is this value telling me? I can be sure by 32% that the values aren't random values? Or is a confidence interval for those kinds of questions useless?&lt;/p&gt;&#xA;" OwnerUserId="3908" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-26T17:16:41.450" LastActivityDate="2011-03-28T13:54:58.437" Title="Understanding confidence interval" Tags="&lt;confidence-interval&gt;" AnswerCount="2" CommentCount="3" FavoriteCount="2" />
  <row Id="8798" PostTypeId="1" CreationDate="2011-03-26T15:55:40.037" Score="2" ViewCount="64" Body="&lt;p&gt;In the absence of good a priori guesses about the number of components to request in Independent Components Analysis, I'm looking to automate a selection process.  I think that a reasonable criterion might be the number that minimizes the global evidence for correlation amongst the computed components. Here's pseudocode of this approach: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for each candidate number of components, n:&#xA;    run ICA specifying n as requested number of components&#xA;    for each pair (c1,c2) of resulting components:&#xA;        compute a model, m1: lm(c1 ~ 1)&#xA;        compute a model, m2: lm(c1 ~ c2)&#xA;        compute log likelihood ratio ( AIC(m2)-AIC(m1) ) representing the relative likelihood of a correlation between c1 &amp;amp; c2&#xA;    compute mean log likelihood ratio across pairs&#xA;Choose the final number of components as that which minimizes the mean log likelihood of component relatedness&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I figure this should automatically penalize candidates larger than the &quot;true&quot; number of components because ICAs resulting from such candidates should be forced to distribute information from single true components across multiple estimated components, increasing the average evidence of correlation across pairs of components.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does this make sense? If so, is there a faster way of achieving an aggregate metric of relatedness across estimated components than the mean log likelihood approach suggested above (which can be rather slow computationally)? If this approach doesn't make sense, what might a good alternative procedure look like?&lt;/p&gt;&#xA;" OwnerUserId="364" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-26T17:16:12.450" LastActivityDate="2011-03-26T17:16:12.450" Title="How do I select the number of components for independent components analysis?" Tags="&lt;ica&gt;" />
  <row Id="8799" PostTypeId="1" CreationDate="2011-03-26T15:57:41.113" Score="1" ViewCount="184" Body="&lt;p&gt;I have made a very simple questionnaire that asks questions that are independent of each other. Every question can be answered with a rating between 1 and 5 where 1 means I strongly disagree and 5 I strongly agree.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I was wondering with statistical methods I could use to evaluate the results. I know maybe I should have thought about that before performing the evaluation but now its too late and I have to get the most out of it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently i just calculate the following values for each question:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;arithmetic mean&lt;/li&gt;&#xA;&lt;li&gt;median&lt;/li&gt;&#xA;&lt;li&gt;max value&lt;/li&gt;&#xA;&lt;li&gt;min value&lt;/li&gt;&#xA;&lt;li&gt;standard deviation&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;But are there any other good indicators I could use to analyze the answers?&lt;/p&gt;&#xA;" OwnerUserId="3908" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-03-26T16:46:53.343" LastActivityDate="2011-03-26T19:04:16.593" Title="How to evaluate a simple questionnaire with statistical methods?" Tags="&lt;questionnaires&gt;" AnswerCount="1" CommentCount="4" />
  <row Id="8800" PostTypeId="1" CreationDate="2011-03-26T16:37:30.610" Score="2" ViewCount="138" Body="&lt;p&gt;i'm making experiments using app. 5000 labeled dataset.i'm trying different supervised ML algorithm to evaluate the results.The vector size is 13 with the labels (totally 12 features+1 label) and i have 15 vector of labeled &quot;flower&quot; class. experiments consist of all data set using 10k cross validation. All features are continuous.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1 experiments using the &quot;pure&quot; features of all dataset.&#xA;2 experiments using only &quot;one&quot; feature (out of 12) change of the flower class.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;i applied naive bayes, C4.5 but all results of 1 and 2 is same, however logistic regression gives different results and lasted longer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1- To your best experiences, what causes the difference between naive bayes, c.45 and logistic regression, how should i evaluate the results to make the audiences satisfied?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2- if performance is an important metric, and classifier is used for IDS systems, which ML algorithms do you offer?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit: More explanation to make the question clear:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;we have 8 different class labels. flower + other 7 labels. In experiment 2 we change the only one attribute of the flower class out of 12 attributes (15 flower labeled class stays same but only one attribute is changed.). all other dataset stays same. so we make experiments using logistic regression, naive bayes and c4.5 seperately with two different dataset. (1- with 5000 dataset, 2- other dataset has difference of change in one attribute of flower class, all other classes stays same).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;comparision: we have results of situation number 1 and 2, in C4.5 and naive bayes, nothing changes, FP and FN. but logistic regression gives interesting results.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;12 0.4 0.4 0.5 2.333 434 12.2 10 2 10 12 12 flower &#xA;........................................... flower&#xA;...........................................&#xA;........................................... flower  (total 15 flower class.)&#xA;&#xA;// one feature change 2nd feature.&#xA;&#xA;12 0.8 0.4 0.5 2.333 434 12.2 10 2 10 12 12 flower &#xA;........................................... flower&#xA;...........................................&#xA;........................................... flower  (total 15 flower class,2nd feature all changed.)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;For example can i make i comment like&#xA;  that: because C4.5 uses the maximum&#xA;  number of class in leaf nodes, the&#xA;  change of one feature in flower class&#xA;  will not affect/change the leaf node classes&#xA;  However, logistic regression uses ... so we observe this kind of differences.???&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="3270" LastEditorUserId="3270" LastEditorDisplayName="" LastEditDate="2011-03-31T17:16:29.933" LastActivityDate="2011-04-01T11:26:39.950" Title="Interpretation of   &quot;one&quot; feature change in a supervised classifier " Tags="&lt;machine-learning&gt;&lt;classification&gt;&lt;feature-selection&gt;&lt;dataset&gt;" AnswerCount="1" CommentCount="4" FavoriteCount="2" />
  <row Id="8802" PostTypeId="2" ParentId="8695" CreationDate="2011-03-26T17:51:56.140" Score="1" ViewCount="" Body="&lt;p&gt;In case richiemorrisroe's response doesn't give you quite enough, I suggest...&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;right-clicking your SPSS factor&#xA;analysis output and choosing Results&#xA;Coach to clarify the contents of the Variance&#xA;Explained table &lt;/li&gt;&#xA;&lt;li&gt;searching the Help    files or&#xA;Tutorial for Reliability    Analysis.&#xA;I'm thinking that by    &quot;composite&#xA;reliability&quot; you mean    internal&#xA;consistency reliability&lt;br&gt;&#xA;(Cronbach's alpha).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="2669" LastActivityDate="2011-03-26T17:51:56.140" CommentCount="4" />
  <row Id="8803" PostTypeId="2" ParentId="8797" CreationDate="2011-03-26T18:49:30.687" Score="3" ViewCount="" Body="&lt;p&gt;IME, the confidence interval is useful as it gives a sense of how uncertain your estimates are. Its a good way to check how variable your results might be, and to give others a sense for how likely the results are to be within a particular range.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That being said, the typical interpretation of one is that 95 (for a 95% interval) times of 100 if this experiment were repeated, the true value of the mean (or whatever you've calculated the interval for) would lie in this range. So it does not tell you that it is 95% certain that the true value lies within that range. On the other hand, a bayesian credible interval will tell you this, but these are not as widely used. &lt;/p&gt;&#xA;" OwnerUserId="656" LastActivityDate="2011-03-26T18:49:30.687" CommentCount="4" />
  <row Id="8804" PostTypeId="2" ParentId="8375" CreationDate="2011-03-26T19:03:42.100" Score="0" ViewCount="" Body="&lt;p&gt;As I see it there are two survival analysis paradigms that could be used. The Cox regression framework allows time varying covariates and would produce an estimate for the  risk of cancellation conditioned on any particular set of covariates relative to the mean level of cancellation. The glm framework with Poisson errors is also a proportional hazards model and is particularly suited to discrete intervals. JVM has pointed out that there is potential error in using incomplete data in the current month, but the sense I get is that you want an estimate that is conditional on the latest value of a co-variate or set of covariates. Better description of the data situation could yield better worked examples....&lt;/p&gt;&#xA;" OwnerUserId="2129" LastActivityDate="2011-03-26T19:03:42.100" />
  <row Id="8805" PostTypeId="2" ParentId="8799" CreationDate="2011-03-26T19:04:16.593" Score="2" ViewCount="" Body="&lt;p&gt;There are a number of ways you can approach this problem (as chl has noted) and you should definitely read the links he gives to other questions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That being said, here is some advice which you may find useful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The psych package is quite good for simple analysis of questionnaries.&#xA;Download this using install.packages(&quot;psych&quot;) from a local mirror.&#xA;There is a useful pairs.panels() function which will show you the correlations between your variables, their distributions and plot regression lines through the points for you. Its a great graphic, but not to be used if you have over 10 variables.&#xA;Your next step should probably be to run a factor analysis.&#xA;This can be done with either the factanal function in base R, or with the fa function in psych. &#xA;Note that this is likely to produce misleading results if you have a small sample size. You can test how many factors to extract using parallel analysis (fa.parallel in the psych package) or Minimum average partial (VSS in the psych package). This could give you some good ideas of how many factors to retain.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your question suggests that you have no prior hypotheses about the structure of the instrument which may suggest factoring the questions a number of times and finding the solution that makes the most sense.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also assess cronbach's alpha which is calculated as the mean of all possible split half reliabilites. The reason i suggested doing the factor analysis first is that cronbach's alpha tends to give weird results if applied to a questionnaire which has multiple factors. The alpha function in the psych package could be used for this computation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you wish to formally test which model is best, then you could look into Confirmatory Factor Analysis, but that might be overkill right now. If you are interested, the sem lavaan and OpenMx packages for R can all carry out this kind of analysis.&lt;/p&gt;&#xA;" OwnerUserId="656" LastActivityDate="2011-03-26T19:04:16.593" CommentCount="2" />
  <row Id="8806" PostTypeId="2" ParentId="8375" CreationDate="2011-03-26T20:40:48.077" Score="1" ViewCount="" Body="&lt;p&gt;Thank you for the clarification, B_Miner. I don't do a lot of forecasting myself, so take what follows with a pinch of salt. Here is what I would do as at least a first cut at the data.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;First, formulate and estimate a model that explains your TVCs. Do all of the cross-validation, error checking, etc., to make sure you have a decent model for the data.&lt;/li&gt;&#xA;&lt;li&gt;Second, formulate and estimate a survival model (of whatever flavor). Do all of the cross-validation, error checking, to make sure this model is reasonable as well.&lt;/li&gt;&#xA;&lt;li&gt;Third, settle on a method of using the forecasts from the TVCs model as the basis of forecasting risks of churn and whatever else you want. Once again, verify that the predictions are reasonable using your sample.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Once you have a model that you think is reasonable, I would suggest bootstrapping the data as a way to incorporate the error in the first TVC model into the second model. Basically, apply steps 1-3 N times, each time taking a bootstrap sample from the data and producing a set of forecasts. When you have a reasonable number of forecasts, summarize them in any way you think is appropriate for your task; e.g., provide mean risk of churn for each individual or covariate profile of interest as well as 95% confidence intervals.&lt;/p&gt;&#xA;" OwnerUserId="3265" LastActivityDate="2011-03-26T20:40:48.077" CommentCount="1" />
  <row Id="8807" PostTypeId="1" AcceptedAnswerId="8832" CreationDate="2011-03-26T20:50:33.563" Score="7" ViewCount="238" Body="&lt;p&gt;I've been using the &lt;a href=&quot;http://cran.r-project.org/web/packages/caret/index.html&quot; rel=&quot;nofollow&quot;&gt;caret package&lt;/a&gt; in R to build predictive models for classification and regression.  Caret provides a unified interface to tune model hyper-parameters by cross validation or boot strapping.  For example, if you are building a simple 'nearest neighbors' model for classification, how many neighbors should you use?  2? 10? 100? Caret helps you answer this question by re-sampling your data, trying different parameters, and then aggregating the results to decide which yield the best predictive accuracy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I like this approach because it is provides a robust methodology for choosing model hyper-parameters, and once you've chosen the final hyper-parameters it provides a cross-validated estimate of how 'good' the model is, using accuracy for classification models and RMSE for regression models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I now have some time-series data that I want to build a regression model for, probably using a random forest. What is a good technique to assess the predictive accuracy of my model, given the nature of the data? If random forests don't really apply to time series data, what's the best way to build an accurate ensemble model for time series analysis?&lt;/p&gt;&#xA;" OwnerUserId="2817" LastEditorUserId="2817" LastEditorDisplayName="" LastEditDate="2011-03-27T17:59:44.317" LastActivityDate="2011-05-08T05:51:21.903" Title="Cross-validating time-series analysis" Tags="&lt;r&gt;&lt;time-series&gt;&lt;cross-validation&gt;" AnswerCount="4" CommentCount="1" FavoriteCount="2" />
  <row Id="8808" PostTypeId="1" CreationDate="2011-03-26T20:51:44.430" Score="1" ViewCount="138" Body="&lt;p&gt;How do you derive the expression for the $100(1-\alpha)$% bayesian confidence interval when working with the uniform distribution in the interval $[-\theta,\theta]$?&lt;/p&gt;&#xA;" OwnerUserId="3910" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-29T01:06:26.093" LastActivityDate="2011-04-29T01:06:26.093" Title="Derivation of bayesian confidence interval" Tags="&lt;bayesian&gt;&lt;homework&gt;" AnswerCount="2" CommentCount="5" FavoriteCount="1" />
  <row Id="8809" PostTypeId="2" ParentId="8795" CreationDate="2011-03-26T22:13:15.587" Score="5" ViewCount="" Body="&lt;p&gt;I'm pretty sure the answer is &lt;strong&gt;yes&lt;/strong&gt;, the standard binomial 'fair coin' test &lt;em&gt;is&lt;/em&gt; still valid: if you wish to test whether two of the three probabilities of a &lt;a href=&quot;http://en.wikipedia.org/wiki/Multinomial_distribution&quot; rel=&quot;nofollow&quot; rel=&quot;nofollow&quot;&gt;multinomial distribution&lt;/a&gt; are the same but you're not interested in any hypotheses about the third probability, you can analyse the numbers of the corresponding two outcomes as if they were drawn from a &lt;a href=&quot;http://en.wikipedia.org/wiki/Binomial_distribution&quot; rel=&quot;nofollow&quot; rel=&quot;nofollow&quot;&gt;binomial distribution&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In fact this seems to make quite a nice exercise about sufficient statistics and conditional likelihood:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can think of this as a multinomial distribution with three possible outcomes and hence two estimable parameters (as the three probabilities must sum to 1). But you're not interested in the probability of the 'middle' outcome, so you can take this to be the &lt;a href=&quot;http://en.wikipedia.org/wiki/Nuisance_parameter&quot; rel=&quot;nofollow&quot; rel=&quot;nofollow&quot;&gt;nuisance parameter&lt;/a&gt;, and the difference between the number of 'top' and 'bottom' outcomes to be the parameter of interest. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's straightforward to show (using the &lt;a href=&quot;http://en.wikipedia.org/wiki/Sufficient_statistic#Fisher.E2.80.93Neyman_factorization_theorem&quot; rel=&quot;nofollow&quot; rel=&quot;nofollow&quot;&gt;Fisher–Neyman factorization theorem&lt;/a&gt;) that the numbers of 'top' and 'bottom' outcomes together form a (two-dimensional) &lt;a href=&quot;http://en.wikipedia.org/wiki/Sufficient_statistic&quot; rel=&quot;nofollow&quot; rel=&quot;nofollow&quot;&gt;sufficient statistic&lt;/a&gt; for the parameter of interest, i.e. the number of 'middle' outcomes doesn't provide any additional information about the value of the parameter of interest. The number of 'middle' outcomes is clearly a sufficient statistic for the nuisance prameter. If we condition on the latter, I think (haven't checked properly) that the resulting &lt;a href=&quot;http://en.wikipedia.org/wiki/Conditional_likelihood#Conditional_likelihood&quot; rel=&quot;nofollow&quot; rel=&quot;nofollow&quot;&gt;conditional likelihood&lt;/a&gt; will end up the same as the likelihood for the binomial distribution, i.e. the coin-tossing problem.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-03-26T22:13:15.587" CommentCount="5" />
  <row Id="8810" PostTypeId="1" AcceptedAnswerId="8815" CreationDate="2011-03-26T23:13:27.653" Score="2" ViewCount="150" Body="&lt;p&gt;I have loaded a log onto a dataframe &lt;code&gt;v&lt;/code&gt;. You can see the output of &lt;code&gt;head(v)&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  user_id    page_id           timestamp&#xA;1     139    1612783 2011-02-22 06:24:40&#xA;2     139    1612783 2011-02-22 06:28:40&#xA;3     139    1612783 2011-02-22 06:41:01&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can I &lt;code&gt;qplot&lt;/code&gt; the number of &lt;code&gt;page_id&lt;/code&gt;'s per day?&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;On the x-axis it would be the day (e.g. &lt;code&gt;2011-02-22&lt;/code&gt;).&lt;/li&gt;&#xA;&lt;li&gt;On the y-axis it would be the number of &lt;code&gt;page_id&lt;/code&gt;'s for that particular date.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="1762" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-03-27T19:45:06.057" LastActivityDate="2011-03-28T15:08:14.560" Title="How to plot page views over time with qplot?" Tags="&lt;r&gt;&lt;data-visualization&gt;&lt;ggplot2&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="1" />
  <row Id="8811" PostTypeId="2" ParentId="7321" CreationDate="2011-03-26T23:42:45.120" Score="0" ViewCount="" Body="&lt;p&gt;A &lt;em&gt;null hypothesis&lt;/em&gt; of your significance test could be that the number of fields overlapping an $x$ point of the axis ($fields(x)$) comes from the same $k$ distribution no matter which point of the axis we take, i.e. $fields(x) \sim K$, where $K$ may take the values $0, 1, 2, ... \infty$. The &lt;em&gt;alternative hypothesis&lt;/em&gt; could be that the number of fields at the point of interest ($fields(x_A)$) comes from an other distribution. You have already defined the &lt;em&gt;test statistic&lt;/em&gt;: the number of fields overlapping ($fields(x)$), and the wording “significantly more” suggests you are thinking in terms of a one tailed test.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you say “elsewhere” does it mean anywhere along this continuous axis? Or does it mean a number of points along the axis? I will assume the second, and that you have an $(x_1, fields(x_1)), (x_2, fields(x_2)), ... (x_N, fields(x_N))$ sample. I will also assume that the $x_1, x_2, ... X_N$ points are “not too close to each other” - your above figure suggests that the width of the fields is not negligible, thus $fields(x_1) = fields(x_2)$ if $x_1$ and $x_2$ are close to each other.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The significance value of the test is the probability of observing a number of fields &lt;em&gt;equal to or more than&lt;/em&gt; $fields(x_A)$ under the null hypothesis. Under the above assumptions this is $p \approx 1 - ECDF_K(fields(x_A)-1))$, where $ECDF_K$ is the empirical cumulative distribution function of $K$ estimated using the sample of $N$ points (not including $x_A$).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you use R all you have to do is use the &lt;code&gt;ecdf()&lt;/code&gt; function to perform this calculation.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-03-26T23:55:53.190" LastActivityDate="2011-03-26T23:55:53.190" />
  <row Id="8812" PostTypeId="1" AcceptedAnswerId="8826" CreationDate="2011-03-27T01:37:04.537" Score="1" ViewCount="79" Body="&lt;p&gt;What is the Jeffreys prior for the geometric distribution?&lt;/p&gt;&#xA;" OwnerUserId="3911" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-29T00:56:16.210" LastActivityDate="2011-04-29T00:56:16.210" Title="Jeffreys prior for geometric distribution?" Tags="&lt;distributions&gt;&lt;bayesian&gt;&lt;prior&gt;" AnswerCount="1" />
  <row Id="8813" PostTypeId="2" ParentId="8807" CreationDate="2011-03-27T01:49:23.647" Score="3" ViewCount="" Body="&lt;p&gt;If you have time series data then you might have a &quot;degrees of freedom problem&quot; . For example if you have 4 observations taken at hourly intervals and then decide to use 241 observations at 1minute intervals, you have 241 observations but they are not necessarily independent. When you submit these 241 values/measurements to an analytical package, the package might expect that these are 241 independent values as it proceeds to perform it's particular magic. If you have time series data you might have to upgrade your analytics. I don't know the program you refer to but it is a reasonable guess on my part ( I could be wrong ! ) that it's tests ( F tests / T tests ...etc ) probably don't apply to your problem set.&lt;/p&gt;&#xA;" OwnerUserId="3382" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-03-27T18:40:07.790" LastActivityDate="2011-03-27T18:40:07.790" />
  <row Id="8814" PostTypeId="2" ParentId="8791" CreationDate="2011-03-27T02:11:59.317" Score="1" ViewCount="" Body="&lt;p&gt;There might be a bit of confusion here with some imprecise statistical jargon. If you have data points that have been measured/reported with different precision/reliability/variability then one turns naturally to Generalized Least Squares where one transforms/weights the data by adjusting for the relative variability . Search for Weighted Least Squares for example. Now given that one has weighted/transformed observed data one might we faced with another weighting issue. When you have correlated observations over space and/or time (taken at fixed intervals either time or space) one is advised to form an adaptive/autoregressive/auto-projected model called an ARIMA Model. Please review my answer to &lt;a href=&quot;http://stats.stackexchange.com/questions/6498/seeking-certain-type-of-arima-explanation/8599#8599&quot;&gt;Seeking certain type of ARIMA explanation&lt;/a&gt; which suggests that an ARIMA is simply a weighted average of previous values. For example y(t)=.5*y(t-1)+.25*y(t-2)+.125*y(t-3) +.... or y(t)=.5*y(t-1)+.5y(t-12)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These are two totally different &quot;weighting solutions&quot; . From your very vivid example it might be that you might have both opportunities to investigate. For more on time series you might review some of my postings and review what others might have said. &lt;/p&gt;&#xA;" OwnerUserId="3382" LastEditorUserId="3382" LastEditorDisplayName="" LastEditDate="2011-03-27T11:15:02.570" LastActivityDate="2011-03-27T11:15:02.570" CommentCount="1" />
  <row Id="8815" PostTypeId="2" ParentId="8810" CreationDate="2011-03-27T03:05:28.227" Score="2" ViewCount="" Body="&lt;p&gt;This will work:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;v &amp;lt;- data.frame(&#xA;          timestamp = as.Date(c(&quot;2011-02-22&quot;, &quot;2011-02-22&quot;, &quot;2011-02-23&quot;)), &#xA;          page_id = c(1,2,1))&#xA;newdata &amp;lt;- data.frame(time=names(new), count=new)&#xA;qplot(time, page, data = newdata)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It is worth reading the ggplot &lt;a href=&quot;http://had.co.nz/ggplot2/book/qplot.pdf&quot; rel=&quot;nofollow&quot;&gt;manual&lt;/a&gt;, which has plenty of &lt;code&gt;qplot&lt;/code&gt; examples, including times series.&lt;/p&gt;&#xA;" OwnerUserId="1381" LastEditorUserId="1381" LastEditorDisplayName="" LastEditDate="2011-03-28T14:35:28.253" LastActivityDate="2011-03-28T14:35:28.253" />
  <row Id="8816" PostTypeId="2" ParentId="8795" CreationDate="2011-03-27T03:21:38.070" Score="3" ViewCount="" Body="&lt;p&gt;If you frame this as a binomial problem (p, 1-p), not a multinomial problem, you'll only be able to describe the past.  You won't be able to say anything about the future.  Why?  Your removal of the middle &quot;edge flips&quot; is implied in your regrouping of the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words, your &quot;data described&quot; probability &quot;p&quot; of a positive result and probability &quot;1-p&quot; of a negative result will not apply on the next &quot;binomial flip of the coin&quot;, because in the future you really have probabilities &quot;x&quot;, &quot;y&quot;, and &quot;(1-x-y)&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit (03/27/2011) ===============================&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I added the following diagram to help explain my comments below.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/Nbwz4.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="2775" LastEditorUserId="2775" LastEditorDisplayName="" LastEditDate="2011-03-27T22:54:06.670" LastActivityDate="2011-03-27T22:54:06.670" CommentCount="5" />
  <row Id="8817" PostTypeId="1" AcceptedAnswerId="8833" CreationDate="2011-03-27T04:00:59.400" Score="10" ViewCount="457" Body="&lt;p&gt;Hello all, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am considering using Python libraries for doing my Machine Learning experiments. Thus far, I had been relying on WEKA but have been pretty dissatisfied on the whole. This is primarily because I have found WEKA to be not so well supported (very few examples, documentation is sparse and community support is less than desirable in my experience), and have found myself in sticky situations with no help forthcoming. Another reason I am contemplating this move is because I am really liking Python (I am new to Python), and don't want to go back to coding in Java. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my question is, what are the more (1) comprehensive (2) &lt;strong&gt;scalable&lt;/strong&gt; (100k features, 10k examples) and (3) well supported libraries for doing ML in Python out there? I am particularly interested in doing text classification, and so would like to use a library that has a good collection of classifiers, feature selection methods (Information Gain, Chi-Sqaured etc.), and text pre-processing capabilities (stemming, stopword removal, tf-idf etc.).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Based on the past e-mail threads here and elsewhere, I have been looking at PyML, scikits-learn and Orange so far. How have people's experiences been with respect to the above 3 metrics that I mention?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any other suggestions?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance.&#xA;Andy&lt;/p&gt;&#xA;" OwnerUserId="3301" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-27T08:46:49.903" LastActivityDate="2011-03-27T15:16:27.763" Title="Machine Learning using Python" Tags="&lt;machine-learning&gt;&lt;python&gt;" AnswerCount="7" CommentCount="3" FavoriteCount="6" />
  <row Id="8818" PostTypeId="1" AcceptedAnswerId="8825" CreationDate="2011-03-27T04:17:46.557" Score="4" ViewCount="84" Body="&lt;p&gt;What is the relation between a dimension and a component in a Gaussian Mixture Model? And what are the meanings of dimension and component? Thank you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please correct me if Im wrong: my understanding is the observed data have many dimensions. Each dimension represents a feature/aspect of the collected data and has its own Gaussian distribution. I don't know where &quot;component&quot; fits into this picture and what it means.&lt;/p&gt;&#xA;" OwnerUserId="2729" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-04-08T20:43:53.850" LastActivityDate="2011-04-08T20:43:53.850" Title="What's a component in gaussian mixture model?" Tags="&lt;multivariate-analysis&gt;&lt;normal-distribution&gt;&lt;mixture&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="1" />
  <row Id="8819" PostTypeId="2" ParentId="8817" CreationDate="2011-03-27T04:41:29.610" Score="6" ViewCount="" Body="&lt;p&gt;In terms of working with text, have a look at NLTK. Very, very well supported &amp;amp; documented (there's even a book online, or in paper if you prefer) and will do the preprocesing you require. You might find Gensim useful as well; the emphasis is on vector space modeling and it's got scalable implementations of LSI and LDA (pLSI too I think) if those are of interest. It will also do selection by tf-idf - I'm not sure that NLTK does. I've used pieces of these on corpora of ~50k without much difficulty.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;NLTK:&#xA;&lt;a href=&quot;http://www.nltk.org/&quot; rel=&quot;nofollow&quot;&gt;http://www.nltk.org/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Gensim:&#xA;&lt;a href=&quot;http://nlp.fi.muni.cz/projekty/gensim/&quot; rel=&quot;nofollow&quot;&gt;http://nlp.fi.muni.cz/projekty/gensim/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unfortunately, as to the main thrust of your question I'm not familiar with the specific libraries you reference (although I've used bits of scikits-learn before).&lt;/p&gt;&#xA;" OwnerUserId="26" LastActivityDate="2011-03-27T04:41:29.610" CommentCount="2" />
  <row Id="8820" PostTypeId="1" AcceptedAnswerId="8831" CreationDate="2011-03-27T04:42:00.160" Score="2" ViewCount="66" Body="&lt;p&gt;I have a data sample (in this case an EEG data sample, but my question refers to any type of data samples of prior unknown distributions).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to do a nonparametric estimate of the expected value for my sample. I did some research, from what I understood I can do this using bootstrap sampling. I found a pdf &lt;a href=&quot;http://www.scss.tcd.ie/Rozenn.Dahyot/453Bootstrap/2007Bootstrap02.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; giving a formula for bootstrap expected value, hopefully it's correct.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In case it's not, can someone please let me know how to do it once I have generated the samples by bootstrapping?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another possibility seems to be MCMC, but I would need to know the distribution from what I understood. I could do a kernel density estimation probably, but I think using bootstrapping might be less complex?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can use python, Matlab or R, in case you do this kind of thing often and have code at hand to share, I'd really appreciate it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any other methods/suggestions are more than welcome.&lt;/p&gt;&#xA;" OwnerUserId="3782" LastActivityDate="2011-03-27T10:59:28.920" Title="Nonparametric expected value estimation of sample from unknown distribution" Tags="&lt;estimation&gt;&lt;sampling&gt;&lt;nonparametric&gt;&lt;mcmc&gt;&lt;bootstrap&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="8821" PostTypeId="2" ParentId="8817" CreationDate="2011-03-27T05:23:07.833" Score="0" ViewCount="" Body="&lt;p&gt;Check out &lt;a href=&quot;http://www.csie.ntu.edu.tw/~cjlin/libsvm/&quot; rel=&quot;nofollow&quot;&gt;libsvm&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="364" LastActivityDate="2011-03-27T05:23:07.833" CommentCount="2" />
  <row Id="8822" PostTypeId="2" ParentId="2547" CreationDate="2011-03-27T05:39:30.573" Score="0" ViewCount="" Body="&lt;p&gt;I don't there is a good descriptive reason for choosing median over mean for age distributions.  There is one of practicality when comparing reported data.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many countries report their population in 5-year age intervals with the top band open-ended.  This causes some difficulties calculating the mean from the intervals, especially for the youngest interval (affected by infant mortality rates), the top &quot;interval&quot; (what is the mean of an 80+ &quot;interval&quot;?), and the near top intervals (the mean of each interval is usually lower than the middle).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is far easier estimate the median by interpolating within the median interval, often approximating by assuming a flat or trapezium age distribution in that interval (death rates in many countries are relatively low around the median age, making this a more reasonable approximation than it is for the young or old). &lt;/p&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-03-27T05:39:30.573" />
  <row Id="8823" PostTypeId="1" AcceptedAnswerId="9187" CreationDate="2011-03-27T08:36:41.173" Score="6" ViewCount="191" Body="&lt;p&gt;This an exercise given in &lt;strong&gt;Probability Theory: The Logic of Science&lt;/strong&gt; by Edwin Jaynes, 2003.  There is a partial solution &lt;a href=&quot;http://ksvanhorn.com/bayes/Papers/sanders.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.  I have worked out a more general partial solution, and was wondering if anyone else has solved it.  I will wait a bit before posting my answer, to give others a go.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Okay, so suppose we have $n$ mutually exclusive and exhaustive hypothesis, denoted by $H_i \;\;(i=1,\dots,n)$.  Further suppose we have $m$ data sets, denoted by $D_j \;\;(j=1,\dots,m)$.  The likelihood ratio for the ith hypothesis is given by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$LR(H_{i})=\frac{P(D_{1}D_{2}\dots,D_{m}|H_{i})}{P(D_{1}D_{2}\dots,D_{m}|\overline{H}_{i})}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that these are conditional probabilities.  Now suppose that given the ith hypothesis $H_{i}$ the $m$ data sets are independent, so we have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(D_{1}D_{2}\dots,D_{m}|H_{i})=\prod_{j=1}^{m}P(D_{j}|H_{i}) \;\;\;\; (i=1,\dots,n)\;\;\;\text{Condition 1}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now it would be quite convenient if the denominator also factored in this situation, so that we have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(D_{1}D_{2}\dots,D_{m}|\overline{H}_{i})=\prod_{j=1}^{m}P(D_{j}|\overline{H}_{i}) \;\;\;\; (i=1,\dots,n)\;\;\;\text{Condition 2}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For in this case the likelihood ratio will split into a product of smaller factors for each data set, so that we have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$LR(H_i)=\prod_{j=1}^{m}\frac{P(D_{j}|H_{i})}{P(D_{j}|\overline{H}_{i})}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So in this case, each data set will &quot;vote for $H_i$&quot; or &quot;vote against $H_i$&quot; independently of any other data set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The exercise is to prove that if $n&amp;gt;2$ (more than two hypothesis), there is no such non-trivial way in which this factoring can occur.  That is, if you assume that condition 1 and condition 2 hold, then at most one of the factors:&#xA;$$\frac{P(D_{1}|H_{i})}{P(D_{1}|\overline{H}_{i})}\frac{P(D_{2}|H_{i})}{P(D_{2}|\overline{H}_{i})}\dots\frac{P(D_{m}|H_{i})}{P(D_{m}|\overline{H}_{i})}$$&#xA;is different from 1, and thus only 1 data set will contribute to the likelihood ratio.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I personally found this result quite fascinating, because it basically shows that multiple hypothesis testing is nothing but a series of binary hypothesis tests.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-04-05T07:46:04.800" Title="Has anyone solved PTLOS exercise 4.1?" Tags="&lt;hypothesis-testing&gt;&lt;independence&gt;&lt;likelihood-ratio&gt;&lt;hypothesis&gt;&lt;multiple-hypothesis&gt;" AnswerCount="1" CommentCount="4" FavoriteCount="2" />
  <row Id="8824" PostTypeId="2" ParentId="8817" CreationDate="2011-03-27T08:55:32.053" Score="1" ViewCount="" Body="&lt;p&gt;Python has a wide range of ML libraries (check out mloss.org as well). However, I always have the feeling that it's more of use for ml researchers than for ml practitioners.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scipy.org&quot; rel=&quot;nofollow&quot;&gt;Numpy/SciPy&lt;/a&gt; and &lt;a href=&quot;http://matplotlib.sf.net&quot; rel=&quot;nofollow&quot;&gt;matplotlib&lt;/a&gt; are excellent tools for scientific work with Python. If you are not afraid to hack in most of the math formulas yourself, you will not be disappointed. Also, it is very easy to use the GPU with &lt;a href=&quot;http://code.google.com/p/cudamat/&quot; rel=&quot;nofollow&quot;&gt;cudamat&lt;/a&gt; or &lt;a href=&quot;http://www.cs.toronto.edu/~tijmen/gnumpy.html&quot; rel=&quot;nofollow&quot;&gt;gnumpy&lt;/a&gt; - experiments that took days before are now completed in hours or even minutes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The latest kid on the block is probably &lt;a href=&quot;http://www.google.de/search?sourceid=chrome&amp;amp;ie=UTF-8&amp;amp;q=theano%20python&quot; rel=&quot;nofollow&quot;&gt;Theano&lt;/a&gt;. It is a symbolic language for mathematical expressions that comes with opmitimzations, GPU implementations and the über-feature automatic differentiation which is nothing short of awesome for gradient based methods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, as far as I know the NLTK mentioned by JMS is basically the number one open source natural language library out there.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Python is the right tool for machine learning.&lt;/p&gt;&#xA;" OwnerUserId="2860" LastActivityDate="2011-03-27T08:55:32.053" />
  <row Id="8825" PostTypeId="2" ParentId="8818" CreationDate="2011-03-27T08:59:37.747" Score="4" ViewCount="" Body="&lt;p&gt;A mixture of Gaussians is defined as a linear combination of multiple Gaussian distributions. Thus it has multiple modes. The dimension refers to the data (e.g. the color, length, width, height and material of a shoe) while the number of components refers to the model. Each Gaussian in your mixture is one component. Thus each component will correspond to one mode, in most of the cases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suggest you read up on &lt;a href=&quot;http://en.wikipedia.org/wiki/Mixture_of_gaussians&quot; rel=&quot;nofollow&quot;&gt;mixture models on wikipedia&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2860" LastActivityDate="2011-03-27T08:59:37.747" />
  <row Id="8826" PostTypeId="2" ParentId="8812" CreationDate="2011-03-27T08:59:51.443" Score="5" ViewCount="" Body="&lt;p&gt;The geometric distribution is given by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p(X|\theta)=(1-\theta)^{X-1}\theta \;\;\; X=1,2,3,\dots$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The log likelihood is thus given by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$log[p(X|\theta)]=L=(X-1)log(1-\theta)+log(\theta)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Differentiate once:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{\partial L}{\partial \theta}=\frac{1}{\theta}-\frac{X-1}{1-\theta}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And again:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{\partial^{2} L}{\partial \theta^{2}}=-\frac{1}{\theta^{2}}-\frac{X-1}{(1-\theta)^{2}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Take the negative expectation of this conditional on $\theta$ (called Fisher information), note that $E(X|\theta)=\frac{1}{\theta}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And so we have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$I(\theta)=\frac{1}{\theta^{2}}+\frac{\theta^{-1}-1}{(1-\theta)^{2}}=\theta^{-2}\left(1+\frac{\theta}{1-\theta}\right)=\theta^{-2}(1-\theta)^{-1}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The jeffrey's prior is given by the square root of this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p(\theta|I) \propto \sqrt{I(\theta)}=\theta^{-1}(1-\theta)^{-\frac{1}{2}}$$&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-03-27T08:59:51.443" CommentCount="2" />
  <row Id="8827" PostTypeId="2" ParentId="7430" CreationDate="2011-03-27T09:09:00.470" Score="2" ViewCount="" Body="&lt;p&gt;@JMS answer is adequate for the nuts and bolts of changing variables.  However, &lt;a href=&quot;http://stats.stackexchange.com/questions/8104/why-is-a-p-sigma2-sim-textig0-001-0-001-prior-on-variance-considered-we&quot;&gt;This question&lt;/a&gt; may help you a bit with why it is uniform on that scale.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My answer to &lt;a href=&quot;http://stats.stackexchange.com/questions/8477/expression-for-conditional-density-for-arch-processes/8489#8489&quot;&gt;this question&lt;/a&gt; goes through a slightly longer derivation of the &quot;jacobian rule&quot; result given in @JMS's answer.  It may help with understanding why the rule applies.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-03-27T09:09:00.470" CommentCount="3" />
  <row Id="8828" PostTypeId="2" ParentId="8817" CreationDate="2011-03-27T09:10:04.543" Score="1" ViewCount="" Body="&lt;p&gt;Not sure if this is particularly useful, but there's a guide for programmers to learn statistics in Python available online. &lt;a href=&quot;http://www.greenteapress.com/thinkstats/&quot; rel=&quot;nofollow&quot;&gt;http://www.greenteapress.com/thinkstats/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems pretty good from my brief scan, and it appears to talk about some machine learning methods, so it might be a good place to start. &lt;/p&gt;&#xA;" OwnerUserId="656" LastActivityDate="2011-03-27T09:10:04.543" />
  <row Id="8829" PostTypeId="2" ParentId="8808" CreationDate="2011-03-27T09:43:27.663" Score="3" ViewCount="" Body="&lt;p&gt;So you have a density of:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p(X_i|\theta)=\frac{1}{2\theta}\;\;\;\; X_i\in[-\theta,\theta]$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now this is what is called a scale density, and $\theta$ is a scale parameter, just like the standard deviation in a normal distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now to do a Bayesian CI you require a prior distribution for $\theta$.  Because $\theta$ is a scale parameter, the prior describing complete initial ignorance is given by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p(\theta|\theta_L,\theta_U) = \frac{1}{\theta [log(\frac{\theta_U}{\theta_L})]} \;\;\; \theta\in[\theta_L,\theta_U]$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your comment, you state that &quot;the improper prior is to be used&quot; so this means that you take the upper limit $\theta_U\rightarrow \infty$ and the lower limit $\theta_L\rightarrow 0$.  But this is in principle, to be done at the end of the calculation, and not at the start.  This is to ensure that you don't have an &quot;infinity&quot; floating around in your results, making them arbitrary.  If the limit does not exist, then &quot;probability theory&quot; is &quot;telling you&quot; that the actual bounds are important to your conclusion.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I assume you want to work this out for yourself, so the remainder just goes:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;calculate the posterior (Note: I have used $s$ as a dummy variable to indicate that the denominator is independent of $\theta$). $$p(\theta|D,\theta_L,\theta_U)=\frac{p(\theta|\theta_L,\theta_U)\prod_{i=1}^{n}p(X_i|\theta)}{\int_{\theta_L}^{\theta_U}p(s|\theta_L,\theta_U)\prod_{i=1}^{n}p(X_i|s)ds}$$&lt;/li&gt;&#xA;&lt;li&gt;Calculate calculate the lower bound $C_L$ and upper bound $C_U$ such that there is a $100(1-\alpha)$% probability that $C_L&amp;lt;\theta&amp;lt;C_U$.  You can do this by solving for general limits&#xA;$$\int_{C_L}^{C_U}p(\theta|D,L,U)d\theta=1-\alpha$$&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;This should give you enough of the &quot;machinery&quot; to go and solve the problem.  However, if you need more details, I can post them.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-03-27T09:43:27.663" />
  <row Id="8830" PostTypeId="2" ParentId="8817" CreationDate="2011-03-27T10:03:22.980" Score="2" ViewCount="" Body="&lt;p&gt;Let me suggest &lt;a href=&quot;http://orange.biolab.si/&quot; rel=&quot;nofollow&quot;&gt;Orange&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;comprehensive&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Yes&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;scalable (100k features, 10k examples) &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Yes&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;well supported libraries for doing ML in Python out there? &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Yes&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;library that has a good collection of classifiers, feature selection methods (Information Gain, Chi-Sqaured etc.), &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;All of these work out of box in Orange&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;and text pre-processing capabilities (stemming, stopword removal, tf-idf etc.).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I have never used Orange for text processing, though&lt;/p&gt;&#xA;" OwnerUserId="1496" LastActivityDate="2011-03-27T10:03:22.980" />
  <row Id="8831" PostTypeId="2" ParentId="8820" CreationDate="2011-03-27T10:51:10.953" Score="2" ViewCount="" Body="&lt;p&gt;It is quite simple; you make a subsample by sampling with replacement:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sample(x,replace=T)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;calculate the statistic you want on it:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mean(sample(x,replace=T))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;finally average it over many repetitions:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mean(replicate(1000,mean(sample(x,replace=T)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="88" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-27T10:59:28.920" LastActivityDate="2011-03-27T10:59:28.920" CommentCount="1" />
  <row Id="8832" PostTypeId="2" ParentId="8807" CreationDate="2011-03-27T11:19:52.957" Score="2" ViewCount="" Body="&lt;p&gt;The &quot;classical&quot; k-times cross-validation technique is based on the fact that each sample in the available data set is used (k-1)-times to train a model and 1 time to test it. Since it is very important to validate time series models on &quot;future&quot; data, this approach will not contribute to the stability of the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One important property of many (most?) time series is the correlation between the adjacent values. As pointed out by IrishStat, if you use previous readings as the independent variables of your model candidate, this correlation (or lack of independence) plays a significant role and is another reason why k-times cross validation isn't a good idea. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One way to overcome over this problem is to &quot;oversample&quot; the data and decorrelate it. If the decorrelation process is successful, then using cross validation on time series becomes less problematic. It will not, however, solve the issue of validating the model using future data&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Clarifications&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;by validating model on future data I mean constructing the model, waiting for new data that wasn't available during model construction, testing, fine-tuning etc and validating it on that &lt;strong&gt;new&lt;/strong&gt; data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;by oversampling the data I mean collecting time series data at frequency much higher than practically needed. For example: sampling stock prices every 5 seconds, when you are really interested in hourly alterations. Here, when I say &quot;sampling&quot; I don't mean &quot;interpolating&quot;, &quot;estimating&quot; etc. If the data cannot be measured at higher frequency, this technique is meaningless&lt;/p&gt;&#xA;" OwnerUserId="1496" LastEditorUserId="1496" LastEditorDisplayName="" LastEditDate="2011-03-28T14:49:33.693" LastActivityDate="2011-03-28T14:49:33.693" CommentCount="1" />
  <row Id="8833" PostTypeId="2" ParentId="8817" CreationDate="2011-03-27T11:20:59.597" Score="7" ViewCount="" Body="&lt;p&gt;About the scikit-learn option: 100k (sparse) features and 10k samples is reasonably small enough to fit in memory hence perfectly doable with scikit-learn (same size as the 20 newsgroups dataset).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a tutorial I gave at PyCon 2011 with a chapter on text classification with exercises and solutions:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.github.com/scikit-learn-tutorial/&quot; rel=&quot;nofollow&quot;&gt;http://scikit-learn.github.com/scikit-learn-tutorial/&lt;/a&gt; (online HTML version)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://github.com/downloads/scikit-learn/scikit-learn-tutorial/scikit_learn_tutorial.pdf&quot; rel=&quot;nofollow&quot;&gt;https://github.com/downloads/scikit-learn/scikit-learn-tutorial/scikit_learn_tutorial.pdf&lt;/a&gt; (PDF version)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://github.com/scikit-learn/scikit-learn-tutorial&quot; rel=&quot;nofollow&quot;&gt;https://github.com/scikit-learn/scikit-learn-tutorial&lt;/a&gt; (source code + exercises)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I also gave a talk on the topic which is an updated version of the version I gave at PyCon FR. Here are the slides (and the embedded video in the comments):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.slideshare.net/ogrisel/statistical-machine-learning-for-text-classification-with-scikitlearn-and-nltk&quot; rel=&quot;nofollow&quot;&gt;http://www.slideshare.net/ogrisel/statistical-machine-learning-for-text-classification-with-scikitlearn-and-nltk&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;As for feature selection, have a look at this answer on quora where all the examples are based on the scikit-learn documentation:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.quora.com/What-are-some-feature-selection-methods/answer/Olivier-Grisel&quot; rel=&quot;nofollow&quot;&gt;http://www.quora.com/What-are-some-feature-selection-methods/answer/Olivier-Grisel&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;We don't have collocation feature extraction in scikit-learn yet. Use nltk and nltk-trainer to do this in the mean time:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/japerk/nltk-trainer&quot; rel=&quot;nofollow&quot;&gt;https://github.com/japerk/nltk-trainer&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="2150" LastEditorUserId="2150" LastEditorDisplayName="" LastEditDate="2011-03-27T11:35:10.863" LastActivityDate="2011-03-27T11:35:10.863" CommentCount="1" />
  <row Id="8834" PostTypeId="2" ParentId="8732" CreationDate="2011-03-27T13:06:33.083" Score="1" ViewCount="" Body="&lt;p&gt;Whether or not you wish to forecast or not has nothing whatsoever to do with correct time series analysis. Time series methods can develop a robust model which can be used simply to characterize the relationship between a dependent series and a set of user-suggested inputs (a.k.a. user-specified predictor series) and empirically identified omitted variables be they deterministic or stochastic.Users at their option can then extend the &quot;signal&quot; into the future i.e. forecast with uncertainties based upon the uncertainty in the coefficients and the uncertainty in the future values of the predictor . Now these two kinds of empirically identified &quot;omitted series&quot; can be classified as 1) deterministic and 2) stochastic. The first type are simply Pulses, Level Shifts , Seasonal Pulses and Local Time Trends whereas the second type is represented by the ARIMA portion of your final model. When one omits one or more stochastic series from the list of possible predictors, the omission is characterized by the ARIMA component in your final model. Time series modelers refer to ARIMA models as a &quot;Poor Man's Regression Model&quot; because the past of the series is being used as a proxy for omitted stochastic input series.&lt;/p&gt;&#xA;" OwnerUserId="3382" LastActivityDate="2011-03-27T13:06:33.083" />
  <row Id="8836" PostTypeId="1" CreationDate="2011-03-27T13:33:06.930" Score="1" ViewCount="59" Body="&lt;p&gt;Dear Group,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was trying to read through Maximum Likelihood Estimation(MLE) and Expectation and Maximization(EM) algorithm. But while reading them, I got two interpretations. I am trying to post my questions, readings and confusions here as under. If any one of the learned members of the group can kindly absolve my queries I would be grateful enough.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;                              MY READINGS/QUESTIONS&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;I. MLE AND EM VERSION I&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We have calculated the likelihood which is, L(f;p)=Πp(x)f(x), where x€X and : f is a non empty and finite corpus on a countable set of X of types. The maximum of Likelihood function is, argmax(L(f;p)). After taking the maximum likelihood, we are trying to take the Laplace smoothing and take Expectation and Maximization algorithm,&#xA;which aims at finding maximum likelihood estimates for settings where this appears to be difficult if not impossible. The trick of the EM algorithm is to map the given data to complete data on which it is well-known how to perform maximum-likelihood estimation. Instead of a complete data corpus, the input of the EM algorithm is an incomplete data corpus, together with the so-called symbolic analyzer. Symbolic Analyzer is a device assigning to each incomplete data type a set of analysis, where each analysis is a complete data type. Thus, the missing complete data part can be partly compensated. The application of the symbolic analyzer to the incomplete-data corpus leads to an ambiguous complete-data corpus. The ambiguity arises as a consequence of the inherent analytical ambiguity of the symbolic analyzer: The expectation-maximization algorithm performs a sequence of runs over the resulting ambiguous complete-data corpus. Each of these runs consists of an expectation step followed by a maximization step. In the E step, the expectation-maximization algorithm combines the symbolic analyzer with an instance of the probability model. The result of this combination is a statistical analyzer which is able to resolve the analytical ambiguity introduced by the symbolic analyzer.  In the M step, the expectation-maximization algorithm calculates an ordinary maximum-likelihood estimate on the resolved complete-data corpus. Let X and Y be non empty and countable set. A function, A: Y-&gt;2X, is called a symbolic analyzer, if the possible empty set of analysis, A(y) ⊆ X are pair wise disjoint, and the union of all sets of all sets of analysis, A(y) is complete &#xA;                        X=ΣA(y), y€Y.                                                                     (1)&#xA; In this case, Y is called the set of incomplete data types, whereas X is called the set of complete data types. So, in other words, the analysis of A(y) of the incomplete data types y form a partition to the complete data type X. Therefore, for each x€X exists a unique y€Y, the so-called yield of x, such that x is an analysis of y, &#xA;Y=yield(x) if and only if x€A(y).  A pair  consisting of a symbolic analyzer A and a probability distribution p on the complete-data types X is called a statistical analyzer. We use a statistical analyzer to induce probabilities for the incomplete-data types y€Y,&#xA;p(y):=Σp(x), x€A(Y). X x2A(y) p(x) Even more important, we use a statistical analyzer to resolve the analytical ambiguity of an incomplete-data type y€Y by looking at the conditional probabilities of the analyzes x€A(y) p(x|y) := p(x) p(y) where y = yield(x). &#xA;Input, Procedure and Output of an EM algorithm: &#xA; (i) A symbolic analyzer, i.e., a function A which assigns a set of analysis, A(y) ⊆X, to each incomplete data types, y€Y, such that all sets of analyses form a partition of the set X of complete data types, X=ΣA(Y) y€Y. &#xA;(ii) a non-empty and finite incomplete-data corpus, i.e., a frequency distribution f on&#xA;the set of incomplete-data types,&#xA; f:Y-&gt;R. such that f(y)&gt;=0, for all y€Y, and 0&amp;lt;|f|&amp;lt;α. &#xA;(iii) a complete-data model M( M(X), i.e., each instance p€ M is a probability&#xA;distribution on the set of complete-data types, &#xA;                       p:X-&gt;[0,1] and Σp(x)=1, where x€X                                    (2) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(iv) Implicit Input : an incomplete-data model M ⊆ M(Y) induced by the symbolic&#xA;analyzer and the complete-data model. Together with a given instance of the complete-data model, the symbolic analyzer constitutes a statistical analyzer which, in turn, induces the following instance of the incomplete-data model. &#xA;                  p:Y[0,1] and p(Y)=Σp(x), x€A(y).                                            (3) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(v) a (randomly generated) starting instance p0 of the complete-data model M.&#xA;(Note: If permitted by M, then p0 should not assign to any x€X a probability of zero.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Procedure of EM algorithm is, &#xA;      (i) for each i=1,2,3,…do, &#xA;      (ii) q:=pi-1&#xA;        (iii) E step: Compute the whole corpus fq: X-&gt;R, expected by q, fq(x)=f(y)*q(x|y), where &#xA;            y=yield(x). &#xA;      (iv) M step: compute a MLE p’ of M on fq, &#xA;             L(fq;p’)=maxL(fq, p)&#xA;       (v) pi=p’&#xA;       (vi) end//for each i.&#xA;        (vii) print p0,p1,p2,p3. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;II. MLE AND EM VERSION 2&#xA;To calculate the Maximum Likelihood, The category distribution is estimated by: &#xA;p'(c) = freq(c) / Σc' freq(c')                                                                            (4)&#xA;where freq(c) is the number of times the category c showed up in the training data, with the denominator being the total number of training instances (each instance has a unique category).&#xA;The maximum likelihood estimates for words in a category are computed similarly:&#xA;p'(w|c) = freq(w,c) / Σw' freq(w',c)                                                                (5)&#xA;where freq(w,c) is the number of times the word w appeared in a document labeled with category c. &#xA;Smoothing:&#xA;Maximum likelihood estimates provide estimates of zero probability for words that were not seen in a category during training. To overcome this gross underestimation, it is common to smoothe the estimated distribution. A typical way of doing this corresponds to assigning a Dirichlet prior to the multinomial parameters (this is a typical Bayesian prior), then computing the maximum a posteriori (MAP) estimate instead of the maximum likelihood estimate.&#xA;In practice, this works out to a technique introduced by Laplace, and known as additive smoothing. The Dirichlet prior is parameterized by a prior number of counts per outcome. For instance, we can take 0.5 as the prior number of counts for a category, then estimate:&#xA;p&quot;(c) = (freq(c) + 0.5) / Σc' (freq(c') + 0.5)                                                              (6)&#xA;and similarly for the word in category estimates, which we might give a 0.01 prior count:&#xA;p&quot;(w|c) = (freq(w,c) + 0.01) / Σw' (freq(w',c) + 0.01)                                             (7) &#xA;Given estimates of p(c) and p(w|c), we can classify new texts consisting of a sequence of words ws as follows, using Bayes's rule (which is where the technique gets its name):&#xA;p(c|ws) = p(ws|c) * p(c) / p(ws)                                                                              (8) &#xA;We expand out the probability of words assuming each word is generated independently (this is the naive assumption from which the technique gets its name), and hence the probabilty of all the words is the product of the probability of each word:&#xA;p(ws|c) = Πi p(ws[i]|c)                                                                                            (9)&#xA;In statistical terms, the naive assumption is that the distribution of words is a multinomial.&#xA;We can compute the marginal p(ws) by summation over its probability in each category weighted by the probability of that category:&#xA;p(ws) = Σc' p(ws|c') * p(c')                                                                                      (10)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Best Regards,&#xA;Subhabrata&lt;/p&gt;&#xA;" OwnerUserId="2329" LastActivityDate="2011-03-27T13:33:06.930" ClosedDate="2011-03-29T21:45:14.350" Title="Confusion in MLE and EM" Tags="&lt;regression&gt;&lt;machine-learning&gt;&lt;bayesian&gt;&lt;estimation&gt;&lt;algorithms&gt;" CommentCount="3" />
  <row Id="8837" PostTypeId="2" ParentId="8744" CreationDate="2011-03-27T14:00:14.600" Score="1" ViewCount="" Body="&lt;p&gt;I suggest a two-step approach: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;get a good initial estimates of the cluster centers, e.g. using hard or fuzzy K-means. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Use Global Nearest Neighbor assignment to associate points with cluster centers: Calculate a distance matrix between each point and each cluster center (you can make the problem a bit smaller by only calculating reasonable distances), replicate each cluster center X times, and solve the &lt;a href=&quot;http://en.wikipedia.org/wiki/Assignment_problem&quot; rel=&quot;nofollow&quot;&gt;linear assignment problem&lt;/a&gt;. You'll get, for each cluster center, exactly X matches to data points, so that, globally, the distance between data points and cluster centers is minimized. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Note that you can update cluster centers after step 2 and repeat step 2 to basically run K-means with fixed number of points per cluster. Still, it will be a good idea to get a good initial guess first.&lt;/p&gt;&#xA;" OwnerUserId="198" LastEditorUserId="198" LastEditorDisplayName="" LastEditDate="2011-03-27T14:28:15.977" LastActivityDate="2011-03-27T14:28:15.977" />
  <row Id="8838" PostTypeId="2" ParentId="8568" CreationDate="2011-03-27T14:26:36.253" Score="1" ViewCount="" Body="&lt;p&gt;You could rewrite your model in a Bayesian software (OpenBUGS, PyMC).&#xA;When any new information is available add them to the model  and re-estimate the posterior.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastActivityDate="2011-03-27T14:26:36.253" />
  <row Id="8839" PostTypeId="2" ParentId="7197" CreationDate="2011-03-27T14:57:41.187" Score="0" ViewCount="" Body="&lt;p&gt;You could set up a model that predicts LOS using YEAR, DG and other variables available (hospital datasets usually include age, gender and many other potential predictors).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One way of comparing your hospital to the comparison set is joining the two datasets and adding a hospital column (either 'my hospital' or 'comparison hospital'). If the conditions (linearity, independence, normality, homoscedasticity) of a linear model are met, you could use a simple ANCOVA. It looks reasonable to assume that LOS differences between your hospital and the comparison data may depend on DG, so you should probably include the DG hospital interaction term. Depending on the details more sophisticated models may be needed.R code sample:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = lm(LOS ~ YEAR + DG * hospital, data=theJoinedTable)&#xA;summary(model)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="3911" LastActivityDate="2011-03-27T14:57:41.187" />
  <row Id="8840" PostTypeId="2" ParentId="8817" CreationDate="2011-03-27T15:16:27.763" Score="0" ViewCount="" Body="&lt;p&gt;SHOGUN (&lt;a href=&quot;http://www.shogun-toolbox.org/&quot; rel=&quot;nofollow&quot;&gt;将軍&lt;/a&gt;) is a large scale machine learning toolbox, which seems promising.&lt;/p&gt;&#xA;" OwnerUserId="1351" LastActivityDate="2011-03-27T15:16:27.763" />
  <row Id="8841" PostTypeId="5" CreationDate="2011-03-27T16:20:20.390" Score="0" ViewCount="" Body="&lt;p&gt;Mixture models arise in attempts to characterize complicated probability distributions, especially those with two or more modes, in terms of distributions with mathematically simple descriptions.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Disambiguation&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Do not confuse a &quot;mixture model&quot; with a &quot;&lt;a href=&quot;http://en.wikipedia.org/wiki/Mixed_model&quot; rel=&quot;nofollow&quot;&gt;mixed model&lt;/a&gt;&quot;!  The former concerns distributions, typically multi-modal, that will be analyzed as positive linear combinations of other distributions.  The latter occurs in a regression setting where some of the independent variables are viewed as fixed and others are viewed as realizations of random variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that although the density of a mixture is, by definition, a linear combination of densities, &lt;em&gt;it is not in general the same as the density of a linear combination of random variables.&lt;/em&gt;  For example, the average of two normal random variables is normal (and therefore has a single mode), but a 50:50 mixture of two different normal densities often has two modes and is never normal.&lt;/p&gt;&#xA;" OwnerUserId="-1" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-27T16:20:20.390" LastActivityDate="2011-03-27T16:20:20.390" />
  <row Id="8842" PostTypeId="4" CreationDate="2011-03-27T16:20:20.390" Score="0" ViewCount="" Body="A mixture distribution is one that is written as a positive linear combination of other distributions." OwnerUserId="-1" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-27T16:20:20.390" LastActivityDate="2011-03-27T16:20:20.390" />
  <row Id="8844" PostTypeId="1" CreationDate="2011-03-27T17:31:46.470" Score="3" ViewCount="68" Body="&lt;p&gt;How to calculate discrete interval coverage?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I know how to do: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I had a continuous model, I could define a 95% confidence interval for each of my predicted values, and then see how often the actual values were within the confidence interval.  I might find that only 88% of the time did my 95% confidence interval cover the actual values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I don't know how to do:  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I do this for a discrete model, such as poisson or gamma-poisson? What I have for this model is as follows, taking a single observation (out of over 100,000 I plan to generate:)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Observation #: (arbitrary)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Predicted value: 1.5&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Predicted probability of 0: .223&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Predicted probability of 1: .335&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Predicted probability of 2: .251&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Predicted probability of 3: .126&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Predicted probability of 4: .048&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Predicted probability of 5: .014   [and 5 or more is .019]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;...(etc)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Predicted probability of 100 (or to some otherwise unrealistic figure): .000&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Actual value (an integer such as &quot;4&quot;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that while I've given poisson values above, in the actual model a predicted value of 1.5 may have different predicted probabilities of 0,1,...100 across observations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm confused by the discreteness of the values.  A &quot;5&quot; is obviously outside the 95% interval, since there's only .019 at 5 and above, which is less than .025.  But there will be a lot of 4's -- individually they are within, but how do I jointly evaluate the number of 4's more appropriately?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why do I care? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The models I'm looking at have been criticized for being accurate at the aggregate level but giving poor individual predictions. I want to see how much worse the poor individual predictions are than the inherently wide confidence intervals predicted by the model. I'm expecting the empirical coverage to be worse (e.g. I might find 88% of the values lie within the 95% confidence interval), but I hope only a bit worse.&lt;/p&gt;&#xA;" OwnerUserId="3919" LastActivityDate="2011-03-27T22:45:57.677" Title="Discrete functions: Confidence interval coverage?" Tags="&lt;confidence-interval&gt;&lt;discrete-data&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="8845" PostTypeId="1" AcceptedAnswerId="8861" CreationDate="2011-03-27T17:46:18.863" Score="4" ViewCount="107" Body="&lt;p&gt;&lt;a href=&quot;http://www.jstor.org/stable/2285344&quot; rel=&quot;nofollow&quot;&gt;Samiuddin, (1976)&lt;/a&gt; states:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.stack.imgur.com/jvBNh.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;or, typset with $\LaTeX$ as originally posted&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;We start with the usual noninformative&#xA;  prior distribution of $\mu_i$ and&#xA;  $\sigma_i (i = 1,2,\ldots, k)$&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;$$\pi(\mu_1, \mu_2, \ldots, \mu_k;&#xD;&#xA; \sigma_1, \sigma_2, \ldots, \sigma_k) &#xD;&#xA; \propto d\mu_1d\mu_2\ldots d\mu_k&#xD;&#xA; \frac{d\sigma_1d\sigma_2\ldots d\sigma_k}&#xD;&#xA; {\sigma_1\sigma_2\ldots \sigma_k}$$&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;What does this notation mean?&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Samiuddin, M. 1976. Bayesian Test of Homogeneity of Variance. Journal of the American Statistical Assoc. Vol. 71, No. 354  &lt;/p&gt;&#xA;" OwnerUserId="1381" LastEditorUserId="1381" LastEditorDisplayName="" LastEditDate="2011-05-18T13:45:14.577" LastActivityDate="2011-05-18T13:45:14.577" Title="What does $d$ mean in this notation of the &quot;usual noninformative prior of $\mu_i$ and $\sigma_i$?&quot;" Tags="&lt;probability&gt;&lt;bayesian&gt;&lt;prior&gt;&lt;notation&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="8846" PostTypeId="1" AcceptedAnswerId="8863" CreationDate="2011-03-27T19:12:32.223" Score="3" ViewCount="223" Body="&lt;p&gt;I'm about to apply Kruskal-Wallis test (non-parametric ANOVA) on three groups of unequal length. I was taught/advised to apply Krukal-Wallis only if:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;dependent variable is at least at ordinal level of measurement&lt;/li&gt;&#xA;&lt;li&gt;group's $ n &amp;gt; 5 $ (otherwise H statistic is not $ \chi ^2 $ distributed, so exact p-value cannot be calculated, etc.)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Kruskal-Wallis_test&quot; rel=&quot;nofollow&quot;&gt;Wikipedia page&lt;/a&gt; says that it requires &lt;em&gt;an identically-shaped and scaled distribution for each group&lt;/em&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Apart from that, what are general considerations when applying Kruskal-Wallis test? Should I check homoscedascity with Fligner-Killeen's test, and what should I do in case of small $ n $, or unequal group sizes?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are &lt;code&gt;exactRankTests::wilcox.exact&lt;/code&gt; and &lt;code&gt;PASWR::wilcoxE.test&lt;/code&gt; functions in R, but I can't seem to find an analogous one for Kruskal-Wallis...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Advice, anyone?&lt;/p&gt;&#xA;" OwnerUserId="1356" LastEditorUserId="1356" LastEditorDisplayName="" LastEditDate="2011-03-27T21:11:14.717" LastActivityDate="2011-03-28T02:25:16.240" Title="Kruskal-Wallis test data considerations" Tags="&lt;r&gt;&lt;nonparametric&gt;&lt;kruskal-wallis&gt;" AnswerCount="2" CommentCount="4" FavoriteCount="1" />
  <row Id="8847" PostTypeId="2" ParentId="8807" CreationDate="2011-03-27T19:20:57.207" Score="3" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://robjhyndman.com/researchtips/crossvalidation/&quot; rel=&quot;nofollow&quot;&gt;http://robjhyndman.com/researchtips/crossvalidation/&lt;/a&gt; contains a quick tip for cross validation of time series. Regarding using random forest for time series data....not sure although it seems like an odd choice given that the model is fitted using bootstrap samples. There are classic time series methods of course (e.g. ARIMA) that can be used, as can ML techniques like Neural Nets (example &lt;a href=&quot;http://www.neural-forecasting.com/Downloads/EVIC05_tutorial%20EVIC%2705%20Slides%20-%20Forecasting%20with%20Neural%20Networks%20Tutorial%20SFCrone.pdf&quot; rel=&quot;nofollow&quot;&gt;example pdf&lt;/a&gt;). Perhaps some of the time series experts can comment on how well ML techniques work compared to time series specific algorithms.&lt;/p&gt;&#xA;" OwnerUserId="2040" LastActivityDate="2011-03-27T19:20:57.207" CommentCount="1" />
  <row Id="8848" PostTypeId="2" ParentId="8810" CreationDate="2011-03-27T19:27:21.377" Score="0" ViewCount="" Body="&lt;pre&gt;&lt;code&gt;t1 = with(v, table(as.Date(timestamp), page_id))&#xA;t2 = apply(t1 &amp;gt; 0, 1, sum)&#xA;dates = as.Date(names(t2))&#xA;plot(t2 ~ dates)                             # using plot&#xA;qplot(dates, t2, data=data.frame(t2, dates)) # using qplot&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can write this in a more compact form, but this way you can check the values of &lt;code&gt;t1&lt;/code&gt; and &lt;code&gt;t2&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-03-28T15:08:14.560" LastActivityDate="2011-03-28T15:08:14.560" CommentCount="2" />
  <row Id="8849" PostTypeId="2" ParentId="8846" CreationDate="2011-03-27T19:34:23.447" Score="0" ViewCount="" Body="&lt;ul&gt;&#xA;&lt;li&gt;&lt;s&gt;You need not check homoscedasticity.&lt;/s&gt; Kruskal and Wallis stated in their original paper that the “test may be fairly insensitive to differences in variability”.&lt;/li&gt;&#xA;&lt;li&gt;If there is no exact test available, you can use bootstrap.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="3911" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-03-27T23:12:57.357" LastActivityDate="2011-03-27T23:12:57.357" CommentCount="7" />
  <row Id="8851" PostTypeId="2" ParentId="8664" CreationDate="2011-03-27T20:16:46.323" Score="1" ViewCount="" Body="&lt;p&gt;When you include subject as a random effect in ANOVA you assume that the subject effect and the product effect are additive. Have you thought about negative covariances? Maybe the more one likes red socks the less they like green ones...&lt;/p&gt;&#xA;" OwnerUserId="3911" LastActivityDate="2011-03-27T20:16:46.323" />
  <row Id="8852" PostTypeId="2" ParentId="4150" CreationDate="2011-03-27T20:36:29.907" Score="0" ViewCount="" Body="&lt;p&gt;Write down the complete likelihood, take the derivative and do a gradient based optimization.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can do this online very easily (that is, process one point after the other) and this might result in far faster convergence than EM if the redundancy in your data is high.&lt;/p&gt;&#xA;" OwnerUserId="2860" LastActivityDate="2011-03-27T20:36:29.907" CommentCount="1" />
  <row Id="8853" PostTypeId="2" ParentId="8653" CreationDate="2011-03-27T20:51:45.013" Score="1" ViewCount="" Body="&lt;p&gt;I understand you have&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;many brain imaging datasets&lt;/li&gt;&#xA;&lt;li&gt;classified into 2 groups, study and control&lt;/li&gt;&#xA;&lt;li&gt;image processing methods&lt;/li&gt;&#xA;&lt;li&gt;parameters of the image processing to tune&lt;/li&gt;&#xA;&lt;li&gt;a collection of processed images with various parameter settings&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;and that you will&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;run a new study recruiting similar subjects and controls&lt;/li&gt;&#xA;&lt;li&gt;pick a single pixel of each dataset&lt;/li&gt;&#xA;&lt;li&gt;compare pixels of study subjects with controls&lt;/li&gt;&#xA;&lt;li&gt;use a two sample T test&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;and you want&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;a sample size formula that includes the parameters of image processing.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I believe you need to explore and understand how the difference between groups and the standard deviation depends on the parameters of image processing. In a second step you can understand how the requires sample size depends on the parameters. (You mentioned log-log plot: the relationship may only be linear after double log transformation if very special conditions are fulfilled, however a linear approximation may be satisfactory in the parameter range you find practical.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;I suggest to perform visualization of the dependence of effect size and SD on the parameters of image processing, and other explorative statistics.&lt;/strong&gt; After these you can set up a model that predicts the inputs of the sample size formula using the image processing parameters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may find that even your hundreds of parameter settings already evaluated do not give sufficient insight (especially if there are many parameters), in which case you may need to evaluate further parameter combinations. Most image processing methods may be automated, automation saves a lot of time when tweaking the parameters.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastActivityDate="2011-03-27T20:51:45.013" />
  <row Id="8854" PostTypeId="1" AcceptedAnswerId="8857" CreationDate="2011-03-27T21:15:26.893" Score="0" ViewCount="107" Body="&lt;p&gt;I'm interested in determining whether two or more groups of data share the same mean, and it seems like the ANOVA framework is a good way to approach this. However, ANOVA assumes residuals are normally distributed, while each of my data is a number between 0 and 100 (a percentage). Because normal distributions have support on all the real line, my data cannot adhere to the normality assumption.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question: Is there an ANOVA-like test which is suited to this domain (in which variables are bounded)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="3921" LastActivityDate="2011-03-27T21:56:14.757" Title="ANOVA-like test for bounded variables (percentages)" Tags="&lt;hypothesis-testing&gt;&lt;anova&gt;" AnswerCount="2" />
  <row Id="8855" PostTypeId="1" CreationDate="2011-03-27T21:20:10.870" Score="0" ViewCount="64" Body="&lt;p&gt;Hi:&#xA;My first question here..&#xA;I use a linear model to estimate daily fish passage through a diversion Dam. After I collect say 2 weeks of data, in order to get weekly means and estimate confidence Intervals, do I use the standard error of the model or use the normal formula:&#xA;standard deviation of FishCaught dividede by the square root of the length of FishCaught?&#xA;I would post the data for the model and the dataset upon request. Thanks&lt;/p&gt;&#xA;" OwnerUserId="3922" LastActivityDate="2011-03-27T21:20:10.870" ClosedDate="2011-04-06T15:07:32.500" Title="Standard error of linear model or standard error of Data?" Tags="&lt;regression&gt;" CommentCount="3" />
  <row Id="8856" PostTypeId="2" ParentId="8633" CreationDate="2011-03-27T21:28:27.603" Score="0" ViewCount="" Body="&lt;p&gt;Your covariate is not only different across subjects, but also gets different during the multiple measurements on the same subject. This has implications on the study design and the analysis method as well.&#xA;(I'm not sure if you really meant the &lt;em&gt;effect&lt;/em&gt; of the covariate in your second sentence.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Study design&lt;/em&gt;: if your multiple measurements on the same subject are of various natures it's important that you run the measurements in various orders so as to be able to separate the effect of increasing fatigue from the effect of the natures of measurement.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Analysis&lt;/em&gt;: some statistical software require the dataset of repeated measure ANCOVA be formatted in the wide format, some other software require the long format (&lt;a href=&quot;http://www.ats.ucla.edu/stat/Spss/modules/reshapel115.htm&quot; rel=&quot;nofollow&quot;&gt;example for wide/long&lt;/a&gt;, &lt;a href=&quot;http://www.webcitation.org/5xVQrvOJ9&quot; rel=&quot;nofollow&quot;&gt;permalink&lt;/a&gt;). Only in the long format will you be able to specify multiple RTs per subject. You will need a statistical software that supports the long format for repeated measures ANCOVA.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastActivityDate="2011-03-27T21:28:27.603" />
  <row Id="8857" PostTypeId="2" ParentId="8854" CreationDate="2011-03-27T21:36:20.393" Score="2" ViewCount="" Body="&lt;p&gt;Is the percentage the most raw data you have, or did you compute the percentage from some sort of binomial count data? If the latter, then you should submit the raw 1s and 0s to a logistic regression. In R, check out glm:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;glm(&#xA;    response ~ group&#xA;    , family = binomial&#xA;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If each individual of each group contributes multiple 1s an 0s, then use lmer from the lme4 package to model the individuals as random effects:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;lmer(&#xA;    response ~ (1|individual) + group&#xA;    , family = binomial&#xA;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If all you have are the percentages, then maybe consider either bootstrapping confidence interval on the group means and differences, or employ a permutation test for differences.&lt;/p&gt;&#xA;" OwnerUserId="364" LastActivityDate="2011-03-27T21:36:20.393" />
  <row Id="8858" PostTypeId="2" ParentId="8854" CreationDate="2011-03-27T21:56:14.757" Score="0" ViewCount="" Body="&lt;p&gt;Percentage values may have normal distributions, e.g. cholesterol levels across humans is approximately normally distributed, and it remains normal even if expressed as percentage of the maximal cholesterol level seen. In such cases you need not worry about the fact that your data cover only a narrow interval.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, there may be a very different mechanism which leads to your percentage. Often percentages summarize many binary measurements on the same subject (e.g. proportion of malignant cells in the tissue sample from the subject). In such cases it's best to use a model that takes the appropriate distribution into consideration (e.g. binomial distribution). To get more specific advice on such a model, tell more about your percentage values!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other generic methods include &lt;a href=&quot;http://en.wikipedia.org/wiki/Data_transformation_%28statistics%29&quot; rel=&quot;nofollow&quot;&gt;transforming&lt;/a&gt; the percentage variable, or using a non-parametric test (e.g. &lt;a href=&quot;http://en.wikipedia.org/wiki/Kruskal%E2%80%93Wallis_one-way_analysis_of_variance&quot; rel=&quot;nofollow&quot;&gt;Kruskal-Wallis&lt;/a&gt;).&lt;/p&gt;&#xA;" OwnerUserId="3911" LastActivityDate="2011-03-27T21:56:14.757" />
  <row Id="8859" PostTypeId="1" CreationDate="2011-03-27T22:13:20.827" Score="0" ViewCount="101" Body="&lt;p&gt;I am currently implementing a text classification program with Naive Bayes. I produce two multinominal models in my training function: p(w|nonSPAM) and p(w|SPAM)) as well as a prior probability P(S).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my testing function I go through each test document, and for each test document, I go through all the terms and compute logP(nonSPAM|D) and logP(SPAM|D). Then I make a classification decision by comparing these two quantities (SPAM = 0 or nonSPAM =1). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My problem is: I want to return a score (e.g. 0.52121) rather then a strict probability (0 or 1), so I can use different thresholds in my program. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possible to calculate a score only by using logP(nonSPAM|D) and logP(SPAM|D) and prior probability P(S)? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I asked a similar question &lt;a href=&quot;http://stackoverflow.com/questions/5451004/log-likelihood-to-implement-naive-bayes-for-text-classification?tab=votes#tab-top&quot;&gt;here&lt;/a&gt; but my current question is more related with statistics.&lt;/p&gt;&#xA;" OwnerUserId="3924" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-04-08T20:43:40.633" LastActivityDate="2011-04-08T20:43:40.633" Title="How to convert log likelihoods into scores in Naive Bayes?" Tags="&lt;machine-learning&gt;&lt;maximum-likelihood&gt;&lt;naive-bayes&gt;" CommentCount="3" />
  <row Id="8860" PostTypeId="2" ParentId="8844" CreationDate="2011-03-27T22:45:57.677" Score="1" ViewCount="" Body="&lt;p&gt;Neyman's confidence intervals make no attempt to provide coverage of the parameter in the case of any particular interval. Instead they provide coverage over all possible parameter values in the long run. In a sense they attempt to be globally accurate at the expense of local accuracy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Confidence intervals for binomial proportions offer a clear illustration of this issue. Neymanian assessment of intervals yields the irregular coverage plots like this, which is for 95% Clopper-Pearson intervals for n=10 Binomial trials:&#xA;&lt;img src=&quot;http://i.imgur.com/Xg9e0.png&quot; alt=&quot;Clopper-Pearson coverage plot&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is an alternative way to do coverage, one that I personally think is much more intuitively approachable and (thus) useful. The coverage by intervals can be specified conditional on the observed result. That coverage would be local coverage. Here is a plot showing local coverage for three different methods of calculation of confindence intervals for binomial proportions: Clopper-Pearson, Wilson's scores, and a conditional exact method that yield intervals identical to Bayesian intervals with a uniform prior:&#xA;&lt;img src=&quot;http://i.imgur.com/ryXNu.png&quot; alt=&quot;Conditional coverage for three types of interval&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Notice that the 95% Clopper-Pearson method gives over 98% local coverage but the exact conditional intervals are, well, exact.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A way to think of the difference between the global and local intervals is to consider the global to be inversions of Neyman-Pearson hypothesis tests where the outcome is a decision that is made on the basis of consideration of long-term error rates for the current experiment as a member of the global set of all experiments that might be run. The local intervals are more akin to inversion of Fisherian significance tests which yield a P value which represents evidence against the null in from &lt;em&gt;this particular&lt;/em&gt; experiment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(As far as I know, the distinction between global and local statistics was first made in an unpublished Master’s thesis by Claire F Leslie (1998) Lack of confidence : a study of the suppression of certain counter-examples to the Neyman-Pearson theory of statistical inference with particular reference to the theory of confidence intervals. That thesis is held by the Baillieu library at The University of Melbourne.)&lt;/p&gt;&#xA;" OwnerUserId="1679" LastActivityDate="2011-03-27T22:45:57.677" />
  <row Id="8861" PostTypeId="2" ParentId="8845" CreationDate="2011-03-27T22:58:10.103" Score="5" ViewCount="" Body="&lt;p&gt;This is shorthand notation for a &quot;differential&quot; of the mean and variance parameters.  The longhand version goes:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p(\mu\in[\mu_1,\mu_1+d\mu_1)|I)\propto d\mu_1$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This indicates a uniform probability with respect to $\mu$.  A more familiar notation is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p(\mu|I)\propto 1$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It comes from the &quot;proper&quot; derivation of a PDF from a CDF.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$lim_{dy\rightarrow 0}P(Y\in[y,y+dy))=f(y)dy$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt;: I initially wrote this answer in a hasty fashion, and so had a bit of unclear notation myself.  In my example, I only had a 1-dimension variable $\mu_1$, and all the above relate to a 1-dimensional random variable.  I think the statistical physics literature (&quot;maxent&quot; people) uses this notation (but not entirely sure) - Edwin Jaynes, Larry Bretthorst, Stephen Gull, and others.  I've never seen it explained in any more detail than what I have given.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And second is that $I$ stands for &quot;prior information&quot;, not an identity matrix.  This is just a good habit to express $I$ explicitly as part of your assumptions - so that you don't forget that 1) they are there, and 2) you answer depends on the prior information just as much it depends on the data.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastEditorUserId="2392" LastEditorDisplayName="" LastEditDate="2011-03-28T09:27:14.167" LastActivityDate="2011-03-28T09:27:14.167" CommentCount="4" />
  <row Id="8862" PostTypeId="2" ParentId="8818" CreationDate="2011-03-28T02:16:21.630" Score="1" ViewCount="" Body="&lt;p&gt;A mixture of Gaussians algorithm is a probabilistic generalization of the $k$-means algorithm. Each mean vector in $k$-means is component. The number of elements in each of the $k$ vectors is the dimension of the model. Thus, if you have $n$ dimensions, you have a $k\times n$ matrix of mean vectors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is no different in a mixture of Gaussians except that now you have to deal with covariance matrices in your model.&lt;/p&gt;&#xA;" OwnerUserId="2660" LastEditorUserId="2660" LastEditorDisplayName="" LastEditDate="2011-03-28T19:37:08.190" LastActivityDate="2011-03-28T19:37:08.190" CommentCount="1" />
  <row Id="8863" PostTypeId="2" ParentId="8846" CreationDate="2011-03-28T02:17:44.203" Score="4" ViewCount="" Body="&lt;p&gt;With small, and possibly unequal group sizes, I'd go with chl's and onestop's suggestion and do a Monte-Carlo permutation test. For the permutation test to be valid, you need exchangeability under $H_{0}$. If all distributions have the same shape (and are therefore identical under $H_{0}$), this is true.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's a first try at looking at the case of 3 groups and no ties. First, let's compare the asymptotic $\chi^{2}$ distribution function against a MC-permutation one for given group sizes (this implementation will break for larger group sizes).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;P  &amp;lt;- 3                     # number of groups&#xA;Nj &amp;lt;- c(4, 8, 6)            # group sizes&#xA;N  &amp;lt;- sum(Nj)               # total number of subjects&#xA;IV &amp;lt;- factor(rep(1:P, Nj))  # grouping factor&#xA;alpha &amp;lt;- 0.05               # alpha-level&#xA;&#xA;# there are N! permutations of ranks within the total sample, but we only want 5000&#xA;nPerms &amp;lt;- min(factorial(N), 5000)&#xA;&#xA;# random sample of all N! permutations&#xA;# sample(1:factorial(N), nPerms) doesn't work for N! &amp;gt;= .Machine$integer.max&#xA;permIdx &amp;lt;- unique(round(runif(nPerms) * (factorial(N)-1)))&#xA;nPerms  &amp;lt;- length(permIdx)&#xA;H       &amp;lt;- numeric(nPerms)  # vector to later contain the test statistics&#xA;&#xA;# function to calculate test statistic from a given rank permutation&#xA;getH &amp;lt;- function(ranks) {&#xA;    Rj &amp;lt;- tapply(ranks, IV, sum)&#xA;    (12 / (N*(N+1))) * sum((1/Nj) * (Rj-(Nj*(N+1) / 2))^2)&#xA;}&#xA;&#xA;# all test statistics for the random sample of rank permutations (breaks for larger N)&#xA;# numperm() internally orders all N! permutations and returns the one with a desired index&#xA;library(sna)                # for numperm()&#xA;for(i in seq(along=permIdx)) { H[i] &amp;lt;- getH(numperm(N, permIdx[i]-1)) }&#xA;&#xA;# cumulative relative frequencies of test statistic from random permutations&#xA;pKWH   &amp;lt;- cumsum(table(round(H, 4)) / nPerms)&#xA;qPerm  &amp;lt;- quantile(H, probs=1-alpha)  # critical value for level alpha from permutations&#xA;qAsymp &amp;lt;- qchisq(1-alpha, P-1)        # critical value for level alpha from chi^2&#xA;&#xA;# illustration of cumRelFreq vs. chi^2 distribution function and resp. critical values&#xA;plot(names(pKWH), pKWH, main=&quot;Kruskal-Wallis: permutation vs. asymptotic&quot;,&#xA;     type=&quot;n&quot;, xlab=&quot;h&quot;, ylab=&quot;P(H &amp;lt;= h)&quot;, cex.lab=1.4)&#xA;points(names(pKWH), pKWH, pch=16, col=&quot;red&quot;)&#xA;curve(pchisq(x, P-1), lwd=2, n=200, add=TRUE)&#xA;abline(h=0.95, col=&quot;blue&quot;)                         # level alpha&#xA;abline(v=c(qPerm, qAsymp), col=c(&quot;red&quot;, &quot;black&quot;))  # critical values&#xA;legend(x=&quot;bottomright&quot;, legend=c(&quot;permutation&quot;, &quot;asymptotic&quot;),&#xA;       pch=c(16, NA), col=c(&quot;red&quot;, &quot;black&quot;), lty=c(NA, 1), lwd=c(NA, 2))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/P3rUu.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now for an actual MC-permutation test. This compares the asymptotic $\chi^{2}$-derived p-value with the result from &lt;code&gt;coin&lt;/code&gt;'s &lt;code&gt;oneway_test()&lt;/code&gt; and the cumulative relative frequency distribution from the MC-permutation sample above.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; DV1 &amp;lt;- round(rnorm(Nj[1], 100, 15), 2)  # data group 1&#xA;&amp;gt; DV2 &amp;lt;- round(rnorm(Nj[2], 110, 15), 2)  # data group 2&#xA;&amp;gt; DV3 &amp;lt;- round(rnorm(Nj[3], 120, 15), 2)  # data group 3&#xA;&amp;gt; DV  &amp;lt;- c(DV1, DV2, DV3)                 # all data&#xA;&amp;gt; kruskal.test(DV ~ IV)                   # asymptotic p-value&#xA;Kruskal-Wallis rank sum test&#xA;data:  DV by IV &#xA;Kruskal-Wallis chi-squared = 7.6506, df = 2, p-value = 0.02181&#xA;&#xA;&amp;gt; library(coin)                           # for oneway_test()&#xA;&amp;gt; oneway_test(DV ~ IV, distribution=approximate(B=9999))&#xA;Approximative K-Sample Permutation Test&#xA;data:  DV by IV (1, 2, 3) &#xA;maxT = 2.5463, p-value = 0.0191&#xA;&#xA;&amp;gt; Hobs &amp;lt;- getH(rank(DV))                  # observed test statistic&#xA;&#xA;# proportion of test statistics at least as extreme as observed one (+1)&#xA;&amp;gt; (pPerm &amp;lt;- (sum(H &amp;gt;= Hobs) + 1) / (length(H) + 1))&#xA;[1] 0.0139972&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1909" LastEditorUserId="1909" LastEditorDisplayName="" LastEditDate="2011-03-28T02:25:16.240" LastActivityDate="2011-03-28T02:25:16.240" CommentCount="1" />
  <row Id="8864" PostTypeId="1" CreationDate="2011-03-28T02:44:24.290" Score="4" ViewCount="141" Body="&lt;p&gt;Hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm new to predictive models and I have a problem at hand that I need some advice with. Basically for a clinical application we want to predict the outcome of a rating scale with a model built on top of outcomes of our new measurement device. My dependent variable, a clinical rating scale, is an integer between 0 and 10 (inclusive). Unfortunately I don't have a large sample ($n \approx 100$) and I have a lot features to select from ($p \approx 120$). Also many of these features are correlated. Nearly all of the features are continuous variables. I have a separate sample for validation ($ n \approx 40$). There are several issues I'd like have your advice about: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Should I go for regression or tree based methods? &lt;/li&gt;&#xA;&lt;li&gt;Should I try ensemble learning methods or I'd better stick with a single model? Which methods should I try and why?&lt;/li&gt;&#xA;&lt;li&gt;If it's better to go for a single model, how should I handle the model selection problem? Should I e.g. limit the number of predictors and go for methods like LEAPS with AIC or should I go for methods like LASSO?&lt;/li&gt;&#xA;&lt;li&gt;If ensemble methods are suggested, which methods can handle cases with small $n$ and large $p$ better?&lt;/li&gt;&#xA;&lt;li&gt;Discussing selected/influential features is important for me. Depending on the answers to previous questions, how should I go about it? &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I have some understanding of regression modeling and model selection problems. I have used the bestglm package in the past. Currently I'm looking at the Caret package as it brings a large number of methods under the same interface. References get technical about the details of the models but so far I didn't find a good one to go over practical issues for problems with small n and big p. I appreciate your suggestions and help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks,&#xA;AlefSin&lt;/p&gt;&#xA;" OwnerUserId="2020" LastActivityDate="2011-03-29T19:28:12.003" Title="Looking for ideas to build a predictive model" Tags="&lt;r&gt;&lt;regression&gt;&lt;predictive-models&gt;" AnswerCount="1" FavoriteCount="2" />
  <row Id="8866" PostTypeId="2" ParentId="8405" CreationDate="2011-03-28T06:07:52.767" Score="0" ViewCount="" Body="&lt;p&gt;As an alternative to the &lt;code&gt;Hmisc&lt;/code&gt; package, you can use &lt;code&gt;table&lt;/code&gt; or &lt;code&gt;xtabs&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;table(cut(X$Age, c(0, 27, 37, 47, 999), X$Outcome)&#xA;xtabs(~ cut(Age, c(0, 27, 37, 47, 999) + Outcome, data=X)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1569" LastActivityDate="2011-03-28T06:07:52.767" CommentCount="1" />
  <row Id="8867" PostTypeId="1" CreationDate="2011-03-28T06:33:55.567" Score="4" ViewCount="42" Body="&lt;p&gt;Given the following hierarchical model&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$X \sim {\rm N}(\mu,1)$, and&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\mu \sim {\rm Laplace}(0, c)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $N(\cdot,\cdot)$ is a normal distribution. Is there any way to get an exact expression for the Fisher information of the marginal distribution of $X$ given $c$; i.e., what is the Fisher information of&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\begin{equation}&#xD;&#xA;p(x | c) = \int p(x|\mu) p(\mu|c) d\mu&#xD;&#xA;\end{equation}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can get an expression for the marginal distribution of X given c, but differentiating w.r.t. $c$ and then taking expectations seems very difficult. Am I missing something obvious? Any help would be appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="530" LastActivityDate="2011-03-28T06:33:55.567" Title="Fisher information in a hierarchical model" Tags="&lt;multilevel-analysis&gt;&lt;fisher&gt;&lt;information&gt;" CommentCount="4" FavoriteCount="1" />
  <row Id="8868" PostTypeId="1" AcceptedAnswerId="8870" CreationDate="2011-03-28T09:19:01.760" Score="3" ViewCount="207" Body="&lt;p&gt;When doing time series  research in R, I found that &lt;code&gt;arima&lt;/code&gt;  provides only the coefficient values and their standard errors of fitted model. However, I also want to get the p-value of the coefficients.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I did not find any function that provides the significance of coef.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I wish to calculate it by myself, but I don't know the degree of freedom in the t or chisq distribution of the coefficients. So my question is how to get the p-values for the coefficients of fitted arima model in R?&lt;/p&gt;&#xA;" OwnerUserId="3926" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-03-28T11:45:30.353" LastActivityDate="2011-03-28T11:45:30.353" Title="How to calculate the p-value of parameters for ARIMA model in R?" Tags="&lt;r&gt;&lt;time-series&gt;&lt;chi-squared&gt;&lt;arima&gt;&lt;parametric&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="8869" PostTypeId="1" AcceptedAnswerId="8871" CreationDate="2011-03-28T09:21:13.490" Score="5" ViewCount="143" Body="&lt;p&gt;After knowing how LSA works, I went on continue reading on pLSA but couldn't really make sense of the mathematical formula. This is what I get from &lt;a href=&quot;http://en.wikipedia.org/wiki/Probabilistic_latent_semantic_analysis&quot; rel=&quot;nofollow&quot;&gt;wikipedia&lt;/a&gt; (other academic papers/tutorial show similar form)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align*} &#xA;P(w,d) = \sum_{c} P(c) P(d|c) P(w|c)\\&#xA;&amp;amp;= P(d) \sum_{c} P(c|d) P(w|c)\\&#xA;\end{align*}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I gave up trying to derive it, and found &lt;a href=&quot;http://www.hongliangjie.com/2010/01/04/notes-on-probabilistic-latent-semantic-analysis-plsa/&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; instead&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align*}&#xA;P(c|d) = \frac{P(d|c)P(c)}{P(d)}\\&#xA;P(c|d)P(d) = P(d|c)P(c)\\&#xA;P(w|c)P(c|d)P(d) = P(w|c)P(d|c)P(c)\\&#xA;P(d) \sum_{c} P(w|c)P(c|d) = \sum_{c} P(w|c)P(d|c)P(c)&#xA;\end{align*}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How does the summation appear at the last line? I am currently reading through some tutorial on &lt;a href=&quot;http://en.wikipedia.org/wiki/Bayesian_inference&quot; rel=&quot;nofollow&quot;&gt;Bayesian Inferencing&lt;/a&gt; (learnt basic probability rules and Bayesian theorem before but can't really see them being useful enough here).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;p/s: not sure how to write multiline equation in MathJax way, will fix when I figure out the right way...&lt;/p&gt;&#xA;" OwnerUserId="3837" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-12T00:13:38.720" LastActivityDate="2011-04-12T00:13:38.720" Title="Deriving mathematical model of pLSA" Tags="&lt;machine-learning&gt;&lt;probability&gt;&lt;bayesian&gt;&lt;multilevel-analysis&gt;&lt;plsa&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="8870" PostTypeId="2" ParentId="8868" CreationDate="2011-03-28T09:36:22.263" Score="1" ViewCount="" Body="&lt;p&gt;The &quot;t value&quot; is the ratio of the coefficient to the standard error. The degrees of freedom (ndf) would be the number of observations minus the max order of difference in the model minus the number of estimated coefficients. The &quot;F value &quot; would be the square of the &quot;t value&quot; In order to exactly compute probability you would have to call a non-central chi-square function and pass in the F value and the degrees of freedom (1,ndf) or perhaps simply call an F function lookup.&lt;/p&gt;&#xA;" OwnerUserId="3382" LastActivityDate="2011-03-28T09:36:22.263" CommentCount="2" />
  <row Id="8871" PostTypeId="2" ParentId="8869" CreationDate="2011-03-28T11:24:36.273" Score="5" ViewCount="" Body="&lt;p&gt;Hi Jeff:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am assuming you want to derive:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align*} &#xA;P(w,d) = \sum_{c} P(c) P(d|c) P(w|c)&#xA;&amp;amp;= P(d) \sum_{c} P(c|d) P(w|c)&#xA;\end{align*}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Further, this is similar to Probabilistic latent semantic indexing (cf. Blei, Jordan, and Ng (2003) Latent Dirichlet Allocation. JMLR section 4.3). PLSI posits that a document label $d$ and a word $w$ are &lt;em&gt;conditionally independent&lt;/em&gt; given an unobserved topic $z$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If this is true, your formula is a simple consequence of &lt;a href=&quot;http://en.wikipedia.org/wiki/Bayes%27_theorem&quot; rel=&quot;nofollow&quot;&gt;Bayes theorem&lt;/a&gt;. Here are the steps:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align*}&#xA;P(w, d) &amp;amp;= \displaystyle \sum_z P(w, z, d)\\&#xA; &amp;amp; = \displaystyle \sum_z P(w, d | z) p(z)\\&#xA; &amp;amp;= \displaystyle \sum_z P(w | z) p(d|z) p(z),&#xA;\end{align*}&#xA;where factorization into products is because of conditional independence. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now use Bayes theorem again to get&#xA;\begin{align}&#xA;\displaystyle \sum_z P(w | z) p(d|z) p(z) &amp;amp;= \displaystyle \sum_z P(w | z) p(z,d)\\&#xA;&amp;amp;= \displaystyle \sum_z P(w | z) p(z|d)p(d)\\&#xA;&amp;amp;= p(d)\displaystyle \sum_z P(w | z) p(z|d)&#xA;\end{align}&lt;/p&gt;&#xA;" OwnerUserId="1307" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-03-28T11:53:06.290" LastActivityDate="2011-03-28T11:53:06.290" CommentCount="8" />
  <row Id="8872" PostTypeId="2" ParentId="8868" CreationDate="2011-03-28T11:41:48.280" Score="5" ViewCount="" Body="&lt;p&gt;Since &lt;code&gt;arima&lt;/code&gt; uses maximum likelihood for estimation, the coefficients are assymptoticaly normal. Hence divide coefficients by their standard errors to get the z-statistics and then calculate p-values. Here is the example with in R with the first example from &lt;code&gt;arima&lt;/code&gt;  help page:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; aa &amp;lt;- arima(lh, order = c(1,0,0))&#xA;&amp;gt; aa&#xA;&#xA;Call:&#xA;arima(x = lh, order = c(1, 0, 0))&#xA;&#xA;Coefficients:&#xA;         ar1  intercept&#xA;      0.5739     2.4133&#xA;s.e.  0.1161     0.1466&#xA;&#xA;sigma^2 estimated as 0.1975:  log likelihood = -29.38,  aic = 64.76&#xA;&amp;gt; (1-pnorm(abs(aa$coef)/sqrt(diag(aa$var.coef))))/2&#xA;         ar1    intercept &#xA;1.935776e-07 0.000000e+00 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The last line gives the p-values.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-03-28T11:41:48.280" />
  <row Id="8873" PostTypeId="1" CreationDate="2011-03-28T13:24:08.807" Score="2" ViewCount="69" Body="&lt;p&gt;How to calculate sample size needed for GWAS for a given MAF, power, $p$-value and frequency of the disease ?&lt;/p&gt;&#xA;" OwnerUserId="3870" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-03-28T14:17:07.833" LastActivityDate="2011-03-28T20:05:23.343" Title="Sample size in genome-wide studies" Tags="&lt;genetics&gt;&lt;power-analysis&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="1" />
  <row Id="8874" PostTypeId="2" ParentId="8797" CreationDate="2011-03-28T13:54:58.437" Score="2" ViewCount="" Body="&lt;p&gt;Based on my calculations, it seems that you had about 16 or 17 participants. The typical methods for calculating confidence intervals of means assume that sample means are normally distributed. In the case of very skewed distributions, that assumption is only valid for large samples, which rules of thumb define as at least 20 or 30 (depending on whose thumb you talk to).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, your data are ordinal data but not necessarily interval data; the difference between a 3 and a 4 is not necessarily the same as the difference between a 1 and a 2. This also makes the typical methods less valid.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to develop some quantitative measure of variability, I suggest that you use a binomial test to estimate a confidence interval of the median. But I probably wouldn't even do that; the exact numbers aren't particularly meaningful unless you have a random sample and you've tested the questionnaire for validity and reliability or if you're comparing it to a similar question or by some blocking factor.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Considering all of this, I don't trust statistics to be particularly meaningful on individual questions from questionnaires like this. When I run questionnaires like this, I generally just plot histograms. I think they tell you more than the numbers.&lt;/p&gt;&#xA;" OwnerUserId="3874" LastActivityDate="2011-03-28T13:54:58.437" />
  <row Id="8875" PostTypeId="2" ParentId="8869" CreationDate="2011-03-28T13:56:59.857" Score="2" ViewCount="" Body="&lt;p&gt;The line $P(c|d)P(c) = P(d|c)P(c)$ (your eq 2) should be $P(c|d)P(d) = P(d|c)P(c)$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not sure why you don't think Bayes theorem and basic probability rules are useful:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Eq 1 is Bayes theorem (ie recognizing that $P(d|c)P(c) = P(c,d)$ and plugging in to the definition of conditional probability)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Eq 2 follows immediately from eq 1&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Eq 3 is just eq 2 multiplied through by $P(w|c)$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since eq 3 holds for all $c$ the sums are equal. Then since $w$ is independent of $d$ given $c$ (an assumption from the model), $P(w|c)P(c|d) = P(w|c, d)P(c|d) = P(w,c|d)$ and so $\sum_c\ P(w,c|d) = P(w|d)$ by the law of total probability, giving you $P(w|d)P(d)$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, $P(w|d)P(d)=P(w,d)$ from the definition of conditional probability.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So basic probability is in fact both necessary and sufficient for the derivation!&lt;/p&gt;&#xA;" OwnerUserId="26" LastActivityDate="2011-03-28T13:56:59.857" CommentCount="2" />
  <row Id="8876" PostTypeId="2" ParentId="8572" CreationDate="2011-03-28T14:24:43.273" Score="5" ViewCount="" Body="&lt;p&gt;So the simple answer is yes: Metropolis-Hastings and its special case Gibbs sampling :) General and powerful; whether or not it scales depends on the problem at hand. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not sure why you think sampling an arbitrary discrete distribution is more difficult than an arbitrary continuous distribution. If you can calculate the discrete distribution and the sample space isn't huge then it's much, much easier (unless the continuous distribution is standard, perhaps). Calculate the likelihood $f(k)$ for each category, then normalise to get the probabilities $P(\tilde k = k) = f(k)/\sum f(k)$ and use inverse transform sampling (imposing an arbitrary order on $k$).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Have you got a particular model in mind? There are all sorts of MCMC approaches to fitting mixture models, for example, where the latent component assignments are discrete parameters. These range from very simple (Gibbs) to quite complex.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How big is the parameter space? Is it potentially enormous (eg in the mixture model case, it's N by the number of mixture components)? You might not need anything more than a Gibbs sampler, since conjugacy is no longer an issue (you can get the normalizing constant directly so you can compute the full conditionals). In fact griddy Gibbs used to be popular for these cases, where a continuous prior is discretized to ease computation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't think there is a particular &quot;best&quot; for all problems having a discrete parameter space any more than there is for the continuous case. But if you tell us more about the models you're interested in perhaps we can make some recommendations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: OK, I can give a little more information in re: your examples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your first example has pretty long history, as you might imagine. A recent-ish review is in [1], see also [2]. I'll try to give some details here: A relevant example is stochastic search variable selection. The initial formulation was to use absolutely continuous priors like $p(\beta)\sim \pi N(\beta; 0, \tau) + (1-\pi) N(\beta, 0, 1000\tau)$. That actually turns out to work poorly compared to priors like $p(\beta)\sim \pi \delta_0 (\beta) + (1-\pi) N(\beta, 0, \tau)$ where $\delta_0$ is a point mass at 0. Note that both fit into your original formulation; an MCMC approach would usually proceed by augmenting $\beta$ with a (discrete) model indicator (say $Z$). This is equivalent to a model index; if you have $Z_1\dots, Z_p$ then obviously you can remap the $2^p$ possible configurations to numbers in $1:2^p$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So how can you improve the MCMC? In a lot of these models you can sample from $p(Z, \beta|y)$ by composition, ie using that $p(Z, \beta|y) = p(\beta | Y, Z)p(Z|Y)$. Block updates like this can tremendously improve mixing since the correlation between $Z$ and $\beta$ is now irrelevant to the sampler&lt;/p&gt;&#xA;&#xA;&lt;p&gt;SSVS embeds the whole model space in one big model. Often this is easy to implement but gives works poorly. Reversible jump MCMC is a different kind of approach which lets the dimension of the parameter space vary explicitly; see [3] for a review and some practical notes. You can find more detailed notes on implementation in different models in the literature, I'm sure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Oftentimes a complete MCMC approach is infeasible; say you have a linear regression with $p=1000$ variables and you're using an approach like SSVS. You can't hope for your sampler to converge; there's not enough time or computing power to visit all those model configurations, and you're especially hosed if some of your variables are even moderately correlated. You should be especially skeptical of people trying to estimate things like variable inclusion probabilities in this way. Various stochastic search algorithms used in conjunction with MCMC have been proposed for such cases. One example is BAS [4], another is in [5] (Sylvia Richardson has other relevant work too); most of the others I'm aware of are geared toward a particular model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A different approach which is gaining in popularity is to use absolutely continuous shrinkage priors that mimic model averaged results. Typically these are formulated as scale mixtures of normals. The Bayesian lasso is one example, which is a special case of normal-gamma priors and a limiting case of normal-exponential-gamma priors. Other choices include the horseshoe and the general class of normal distributions with inverted beta priors on their variance. For more on these, I'd suggest starting with [6] and walking back through the references (too many for me to replicate here :) )&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'll add more about outlier models later if I get a chance; the classic reference is [7]. They're very similar in spirit to shrinkage priors. Usually they're pretty easy to do with Gibbs sampling. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Perhaps not as practical as you were hoping for; model selection in particular is a hard problem and the more elaborate the model the worse it gets. Block update wherever possible is the only piece of general advice I have. Sampling from a mixture of distributions you will often have the problem that membership indicators and component parameters are highly correlated. I also haven't touched on label switching issues (or lack of label switching); there is quite a bit of literature there but it's a little out of my wheelhouse.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyway, I think it's useful to start with some of the references here, to get a feeling for the different ways that others are approaching similar problems.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[1] Merlise Clyde and E. I. George. Model Uncertainty Statistical Science 19 (2004): 81--94.&#xA;&lt;a href=&quot;http://www.isds.duke.edu/~clyde/papers/statsci.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.isds.duke.edu/~clyde/papers/statsci.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[2]http://www-personal.umich.edu/~bnyhan/montgomery-nyhan-bma.pdf&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[3] Green &amp;amp; Hastie Reversible jump MCMC (2009)&#xA;&lt;a href=&quot;http://www.stats.bris.ac.uk/~mapjg/papers/rjmcmc_20090613.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.stats.bris.ac.uk/~mapjg/papers/rjmcmc_20090613.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[4] &lt;a href=&quot;http://www.stat.duke.edu/~clyde/BAS/&quot; rel=&quot;nofollow&quot;&gt;http://www.stat.duke.edu/~clyde/BAS/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[5] &lt;a href=&quot;http://ba.stat.cmu.edu/journal/2010/vol05/issue03/bottolo.pdf&quot; rel=&quot;nofollow&quot;&gt;http://ba.stat.cmu.edu/journal/2010/vol05/issue03/bottolo.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[6] &lt;a href=&quot;http://www.uv.es/bernardo/Polson.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.uv.es/bernardo/Polson.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[7] Mike West Outlier models and prior distributions in Bayesian linear regression (1984) JRSS-B&lt;/p&gt;&#xA;" OwnerUserId="26" LastEditorUserId="26" LastEditorDisplayName="" LastEditDate="2011-03-31T16:45:44.557" LastActivityDate="2011-03-31T16:45:44.557" CommentCount="1" />
  <row Id="8877" PostTypeId="1" CreationDate="2011-03-28T15:37:13.107" Score="4" ViewCount="129" Body="&lt;p&gt;My &lt;a href=&quot;http://stats.stackexchange.com/questions/8846/kruskal-wallis-test-data-considerations&quot;&gt;struggle&lt;/a&gt; with non-parametric methods continues... I'd like to apply a median polish instead of two-way ANOVA (normality and homoscedascity assumptions are violated, and $ n_{ij} $ are small, so I can't use CLT as an excuse). I've never used median polish so far, and our course in statistics taught us to worship ANOVA and forget about robust methods if basic assumptions are not met. I saw &lt;a href=&quot;http://stats.stackexchange.com/questions/3634/multi-way-nonparametric-anova&quot;&gt;this post&lt;/a&gt; and it seems that median polish can be applied for two-way factorial design. Which technique do you find appropriate in case of violation of ANOVA assumptions?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, what are the basic data considerations for median polish (or any other technique you find appropriate in this case)? Same shape, homoscedascity? Any resource (link/reference) is appreciated. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;hr /&gt;&#xA;P.S.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that I'm aware of &lt;code&gt;medpolish&lt;/code&gt; function in R. &lt;/p&gt;&#xA;" OwnerUserId="1356" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-28T22:54:48.707" LastActivityDate="2011-03-28T22:54:48.707" Title="Two-way robust ANOVA" Tags="&lt;r&gt;&lt;anova&gt;&lt;nonparametric&gt;&lt;median&gt;&lt;robust&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="1" />
  <row Id="8878" PostTypeId="2" ParentId="8877" CreationDate="2011-03-28T16:13:24.807" Score="1" ViewCount="" Body="&lt;p&gt;How is normality violated?  Medians are more sensitive to skew than means as n gets low.  Be careful of that.  It would be very problematic if small n's varied in a systematic way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How much is homoscedascity violated?  If the n's are about equal it won't matter much for quite large differences.&lt;/p&gt;&#xA;" OwnerUserId="601" LastActivityDate="2011-03-28T16:13:24.807" CommentCount="8" />
  <row Id="8879" PostTypeId="1" CreationDate="2011-03-28T18:14:57.557" Score="0" ViewCount="39" Body="&lt;p&gt;Let $X$ be a $p \times q$ random matrix in which each entry is distributed &#xA;according to the standard normal distribution $N(0,1)$. And let $D$ be a &#xA;$q\times q$ diagonal matrix. My questions are the following: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) What is the expected Hilbert-Schmidt norm of $M=XX^{\dagger}$ ? Is the norm of $M$ concentrated around the mean? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) What is the expected Hilbert-Schmidt norm of $X D X^{\dagger}$?  Again, is it concentrated around the mean? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hilbert-Schmidt norm : The usual euclidean norm if we treat the matrix as a vector.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Concentrated around the mean: $\|M\|$ is concentrated around the mean if $Pr[|\|M\|-E[\|M\|] | &amp;gt; \epsilon]$ decays exponentially with $\epsilon$.&lt;/p&gt;&#xA;" OwnerUserId="3930" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-28T22:16:12.780" LastActivityDate="2011-03-28T22:16:12.780" Title="Conjugation by random Gaussian matrix" Tags="&lt;homework&gt;&lt;normal-distribution&gt;" CommentCount="1" />
  <row Id="8880" PostTypeId="2" ParentId="8808" CreationDate="2011-03-28T18:36:09.553" Score="3" ViewCount="" Body="&lt;p&gt;The question is not totally clear, but I am going to assume that that you have an improper prior for $\theta$ proportional to $1/\theta$ and $n$ observed points $\{X_i\}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A sufficient statistic is $Y=\max_i |X_i|$, and you will have  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Pr(Y \le y|\theta) = (y/\theta)^n \textrm{ if } 0 \le y \le \theta $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;so with density $p(y|\theta) = ny^{n-1}/\theta^n$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So using Bayes' theorem you get &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p(\theta|y) = \frac{ny^{n-1}/\theta^{n+1}}{\int_{\phi=y}^{\infty}ny^{n-1}/\phi^{n+1} \; d\phi} =\frac{n y^{n}}{\theta^{n+1}} \textrm{ if } \theta \ge y.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a decreasing function of $\theta$ and has an integral $Pr(\theta \le \theta_0|Y=y) = 1-(y/\theta_0)^{n}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So there are several credible intervals for $\theta$ given $Y=\max_i |X_i|$.  The narrowest (highest density) $1-\alpha$ interval is $$\left[Y, \frac{Y}{\sqrt[n]{\alpha}}\right]$$ while the quantile centred interval is $$\left[\frac{Y}{\sqrt[n]{1-\frac{\alpha}{2}}}, \frac{Y}{\sqrt[n]{\frac{\alpha}{2}}}\right] .$$&lt;/p&gt;&#xA;" OwnerUserId="2958" LastEditorUserId="2958" LastEditorDisplayName="" LastEditDate="2011-03-29T21:02:58.523" LastActivityDate="2011-03-29T21:02:58.523" CommentCount="2" />
  <row Id="8881" PostTypeId="1" CreationDate="2011-03-28T18:53:05.850" Score="1" ViewCount="150" Body="&lt;p&gt;I have a dataset with the following types of predictors:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;binary (e.g., gender),&lt;/li&gt;&#xA;&lt;li&gt;nominal with 3 categories,&lt;/li&gt;&#xA;&lt;li&gt;ordinal, and&lt;/li&gt;&#xA;&lt;li&gt;continuous&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;Question:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;What is the best way to set up a regression model that includes these different types of variable?&lt;/p&gt;&#xA;" OwnerUserId="3472" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-05T13:28:12.420" LastActivityDate="2011-04-06T09:30:46.060" Title="How to perform a regression model with a mix of binary, nominal, ordinal, and continuous predictors?" Tags="&lt;regression&gt;" AnswerCount="2" CommentCount="3" FavoriteCount="1" />
  <row Id="8882" PostTypeId="1" AcceptedAnswerId="8888" CreationDate="2011-03-28T19:45:31.427" Score="3" ViewCount="133" Body="&lt;p&gt;If one has dataset with a single outlier such as the following graph taken from Vanni-Mercer et al. (2009), is there a statistical test that one can use that accounts for the single outlier rather than having to throw it out or declare significance because of a single data point?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;RT is reaction time. Trial rank is essentially the trial number.&#xA;&lt;img src=&quot;http://i.imgur.com/b7s2a.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="2660" LastActivityDate="2011-03-29T07:25:09.323" Title="Alternative to Tukey's HSD" Tags="&lt;post-hoc&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="8883" PostTypeId="1" AcceptedAnswerId="8921" CreationDate="2011-03-28T19:53:17.083" Score="3" ViewCount="92" Body="&lt;p&gt;I have a process which consists of a number of events and what is known is the timings between the events. What I'm trying to determine is a distribution that allows me to determine a likelyhood that a new sample fits the distibution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The issue is mainly that if you have lots of samples you can approximate the result using a standard gaussian and use the mean and standard deviation. But if you only have a handful of samples, the gaussian does not accurately represent the situation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From what I've read it is common to model waiting times using the gamma distribution. Looking at how the process evolves it looks like it matches well. The unknown is the scale parameter, since the shape parameter I think should be the number of samples. What I've worked out so far is that given the timings $X_1 ... X_N$ you can say:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ \sum_{n=1}^N X_n \sim \Gamma(N,\theta) $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;($N$ is known and fixed)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, $\theta$ is unknown, but the maximum likelihood parameter is the average of the $X_i$ (according to wikipedia anyway). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is, can I use this to estimate a distribution for $X_i$, that is, since the $X_i$ are independent:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ N X_i | \sum_n X_n \sim \Gamma(N, \tfrac{1}{N}\sum_n X_n) $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Something else I've wondered about. Suppose I do have information about $\theta$, say a distribution. How can I incorporate this into the model?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: Clarified that N is fixed.&lt;/p&gt;&#xA;" OwnerUserId="3932" LastEditorUserId="3932" LastEditorDisplayName="" LastEditDate="2011-03-29T16:18:23.670" LastActivityDate="2011-03-30T04:22:56.787" Title="Conditional expection of gamma distribution on sum" Tags="&lt;conditional-probability&gt;&lt;gamma-distribution&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="8884" PostTypeId="1" AcceptedAnswerId="8887" CreationDate="2011-03-28T19:59:07.047" Score="3" ViewCount="122" Body="&lt;p&gt;The central limit theorem as I am familiar with it applies to the limiting (rescaled) distribution of $n$ convolutions of a single probability distribution as $n$ goes to infinity, or equivalently, to distribution one gets from taking a sum of $n$ random variables each with a single fixed distribution. That is, it is a theorem about the (limiting as $n\to \infty$) probability distribution of&#xA;$A_1 + A_2 + ... + A_n$ where each term has a fixed distribution $P$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am asking about a theorem about the limiting probability distribution of&#xA;$A_1 + A_2 + ... + A_n$&#xA;where $A_1$ has probability distribution $P_1$, $A_2$ has probability distribution $P_2$, $A_3$ has probability distribution $P_3$, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, is there a theorem for the case where each distribution isn't fixed, but is selected at random with probability determined by a measure $\mu$?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there such a general theorem, where the limit isn't necessarily gaussian, the limit can be reconstructed from $\mu$, and the convergence is pretty strong?&lt;/p&gt;&#xA;" OwnerUserId="2912" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-03-28T20:24:39.990" LastActivityDate="2011-03-30T03:34:38.853" Title="Central limit theorem for sum from varied distributions" Tags="&lt;central-limit-theorem&gt;" AnswerCount="2" CommentCount="5" />
  <row Id="8885" PostTypeId="1" CreationDate="2011-03-28T20:01:44.653" Score="2" ViewCount="43" Body="&lt;p&gt;How to order a set of vectors $W$ if we are given a training set $V$ consisting of $k$ $n$-dimensional vectors and partial order of them? It is not the total order, so some vectors might not be comparable with some other. The answer will depend on assumptions, so feel free to make any reasonable assumptions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Example&lt;/strong&gt;&lt;br&gt;&#xA;Let: $k=4$ and $n=2$&lt;br&gt;&#xA;$v_{1}=(1,2)$&lt;br&gt;&#xA;$v_{2}=(5,8)$&lt;br&gt;&#xA;$v_{3}=(4,3)$&lt;br&gt;&#xA;$v_{4}=(9,6)$&lt;br&gt;&#xA;We know that $v_{1}&amp;lt;v_{3}$, $v_{2}&amp;lt;v_{4}$ and $v_{3}&amp;lt;v_{4}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Vectors that we want to order are following:&lt;br&gt;&#xA;$w_{1} = (2,6)$&lt;br&gt;&#xA;$w_{2} = (7,4)$&lt;br&gt;&#xA;$w_{3} = (5,5)$  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The most intuitive order is $w_{1}&amp;lt;w_{3}&amp;lt;w_{2}$, because it seems that the first attribute is the most important.&lt;/p&gt;&#xA;" OwnerUserId="1643" LastActivityDate="2011-05-27T23:50:55.233" Title="Prediction of an order of vectors using partially ordered set" Tags="&lt;ordered-variables&gt;" AnswerCount="1" CommentCount="5" />
  <row Id="8886" PostTypeId="2" ParentId="8873" CreationDate="2011-03-28T20:05:23.343" Score="1" ViewCount="" Body="&lt;p&gt;Haven't tried it myself, but you might like to try the &lt;a href=&quot;http://www.stats.ox.ac.uk/~marchini/software.html#GWApower&quot; rel=&quot;nofollow&quot;&gt;GWApower&lt;/a&gt; R package. See &lt;a href=&quot;http://dx.doi.org/10.1371/journal.pgen.1000477&quot; rel=&quot;nofollow&quot;&gt;Spencer &lt;em&gt;et al.&lt;/em&gt; 2009&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-03-28T20:05:23.343" CommentCount="1" />
  <row Id="8887" PostTypeId="2" ParentId="8884" CreationDate="2011-03-28T20:18:16.797" Score="6" ViewCount="" Body="&lt;p&gt;The &lt;a href=&quot;http://books.google.com/books?id=4LkdSaI4xXMC&amp;amp;lpg=PP1&amp;amp;dq=inauthor%3a%22Valentin%20Vladimirovich%20Petrov%22&amp;amp;hl=fr&amp;amp;pg=PA91#v=onepage&amp;amp;q&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;theorem 3.1&lt;/a&gt; in this &lt;a href=&quot;http://books.google.com/books?id=4LkdSaI4xXMC&amp;amp;lpg=PP1&amp;amp;dq=inauthor%3A%22Valentin%20Vladimirovich%20Petrov%22&amp;amp;hl=fr&amp;amp;pg=PP1#v=onepage&amp;amp;q&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;book&lt;/a&gt; answers your first question. The key restriction in central limit theorem is not different distributions but the independence. The result is a very nice one, since it says that for interesting sums of  independent random variables the limiting distribution has to have certain property, namely &lt;a href=&quot;http://en.wikipedia.org/wiki/Infinite_divisibility_%28probability%29&quot; rel=&quot;nofollow&quot;&gt;infinite divisibility&lt;/a&gt;. The classical central limit theorem (with iid variables with finite variances) is then only a very special case of this theorem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that this is a very general answer to very general question. Given the nature of your distributions more precise answer can be given. For example if the distributions  satisfy &lt;a href=&quot;http://en.wikipedia.org/wiki/Lindeberg%27s_condition&quot; rel=&quot;nofollow&quot;&gt;Lindeberg's condition&lt;/a&gt; then the limiting distribution is necessary normal (if we exclude let us say non-interesting cases). &lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-03-28T20:18:16.797" CommentCount="11" />
  <row Id="8888" PostTypeId="2" ParentId="8882" CreationDate="2011-03-28T20:20:14.667" Score="4" ViewCount="" Body="&lt;p&gt;Typically in RT studies there's good reason to believe that the first trials are different qualitatively from the rest and the long RT is merely an indicator of that.  Why would you want to bother keeping them?&lt;/p&gt;&#xA;" OwnerUserId="601" LastActivityDate="2011-03-28T20:20:14.667" CommentCount="3" />
  <row Id="8890" PostTypeId="2" ParentId="8864" CreationDate="2011-03-28T21:43:57.283" Score="4" ViewCount="" Body="&lt;p&gt;If you use a regression model you may start with ordinal logistic regression since your dependent variable has an ordinal scale of 11 levels. Then you may want to look at the threshold values as you may find that they are equidistant (after some transformation), in which case you may go for linear regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tree based methods are able to capture some non-linearities, interactions and they are very good at finding thresholds in the explanatory variables. You may be able to explain some of these by adding transformed versions of the explanatory variables to the feature set of the regression analysis. Playing with &lt;a href=&quot;http://cran.r-project.org/web/packages/acepack/acepack.pdf&quot; rel=&quot;nofollow&quot;&gt;ACE or AVAS&lt;/a&gt; may help finding suitable transformations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As it is important for you to discuss the influential features I recommend you to do extensive exploration with trees, regression models and graphs to understand the biology behind the data and models. I would start with your last question, understand the biology first, and then formulate a conformable model.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-03-29T19:28:12.003" LastActivityDate="2011-03-29T19:28:12.003" CommentCount="1" />
  <row Id="8891" PostTypeId="1" CreationDate="2011-03-28T21:53:25.267" Score="5" ViewCount="181" Body="&lt;p&gt;Hi!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am trying to predict real estate sales prices. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;In my dataset there are independent variables that are both nominal and numeric (square meters, prices etc.) &lt;/li&gt;&#xA;&lt;li&gt;Before feeding the data to any regression algorithm I'd like to preprocess it correctly (binning, normalizing mean / std deviation, discretization etc.)&lt;/li&gt;&#xA;&lt;li&gt;I am overwhelmed by the many methods listed in various textbooks and try to find out what works well in practice&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Although the most reasonable answer to this question is probably 'it depends', could you maybe give me some rules of thumb / war stories / general advice?&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;How do you usually preprocess data for regression? &lt;/li&gt;&#xA;&lt;li&gt;What methods do you usually apply?&lt;/li&gt;&#xA;&lt;li&gt;Which regression algorithms need a special treatment?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;As my tools I am using weka and R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many Thanks!&lt;/p&gt;&#xA;" OwnerUserId="3933" LastActivityDate="2011-03-29T14:11:32.650" Title="Data preparation for regression" Tags="&lt;r&gt;&lt;regression&gt;&lt;predictive-models&gt;&lt;standarization&gt;" AnswerCount="4" FavoriteCount="1" />
  <row Id="8893" PostTypeId="2" ParentId="8882" CreationDate="2011-03-28T23:11:36.710" Score="2" ViewCount="" Body="&lt;p&gt;You might consider checking out the &lt;a href=&quot;http://cran.r-project.org/web/packages/gamm4/index.html&quot; rel=&quot;nofollow&quot;&gt;gamm4&lt;/a&gt; package in R, which basically finds a non-linear function that fits the data while auto-penalizing complexity. I recently used it to fit a similar data set, then obtained the residuals and used these to bootstrap pretty confidence ribbons for the fit.&lt;/p&gt;&#xA;" OwnerUserId="364" LastActivityDate="2011-03-28T23:11:36.710" CommentCount="1" />
  <row Id="8894" PostTypeId="2" ParentId="8885" CreationDate="2011-03-28T23:39:05.167" Score="0" ViewCount="" Body="&lt;p&gt;In the example if we order the vectors according to the first attribute the three pairwise comparisons will be satisfied. This is the same solution as what you suggested in your last sentence. Why aren't you satisfied with this solution? Do you have any further information on the problem? &lt;/p&gt;&#xA;" OwnerUserId="3911" LastActivityDate="2011-03-28T23:39:05.167" CommentCount="1" />
  <row Id="8895" PostTypeId="2" ParentId="8891" CreationDate="2011-03-28T23:57:31.773" Score="1" ViewCount="" Body="&lt;p&gt;The real estate prices that you are tying to predict , are they consecutive/chronological values i.e. time series data or are they prices for different classes e.g. this years prices for  different classes for the same time frame. You might want to read something I wrote on these two kinds of problems as it warns that if you are dealing with longitudinal data ( time series) then the tools of ordinary cross-sectional regression will not ordinarily apply. It is entitled &quot;Regression vs Box-Jenkins&quot; &lt;a href=&quot;http://www.autobox.com/pdfs/regvsbox.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.autobox.com/pdfs/regvsbox.pdf&lt;/a&gt; .&lt;/p&gt;&#xA;" OwnerUserId="3382" LastActivityDate="2011-03-28T23:57:31.773" CommentCount="6" />
  <row Id="8897" PostTypeId="2" ParentId="8891" CreationDate="2011-03-29T06:38:54.180" Score="1" ViewCount="" Body="&lt;p&gt;Binning your data is usually a bad idea because it will cause you to lose information, which will likely result in loss of power. Also, I would rarely standardise variables before doing regression, although some people may like to.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A really good book to read, if you can get it, is &quot;Regression Modeling Strategies&quot; by Frank Harrell.&lt;/p&gt;&#xA;" OwnerUserId="3835" LastActivityDate="2011-03-29T06:38:54.180" CommentCount="1" />
  <row Id="8898" PostTypeId="1" CreationDate="2011-03-29T06:56:06.277" Score="6" ViewCount="137" Body="&lt;p&gt;Hello, &#xA;A pathologist friend came to me for help with the following question for a research project.  The goal is to compare the effectiveness of three different diagnostic techniques.  The data set is as follows: there are 50 different specimens, each specimen was evaluated by 4 pathologists, and 3 different instruments (ie 600 total diagnoses).  Each case has a possible diagnosis of positive or negative, and the true results are known, as they have been independently determined.  The success rate depends on both the quality of the instrument and the skill of the pathologist, and we cannot assume that the four pathologists have the same proficiency.  Finally, even though each person measured the same specimen 3 times, they can be treated as independent measurements.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are the appropriate tests for comparing effectiveness among the instruments?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ADDED:&#xA;Lots of good info in the answers, thanks to both.  Any thoughts on how ROC and randomized block compare/contrast? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not sure I've digested it enough to know which method is &quot;better&quot;. Since the results need to be communicated to a certain audience, it probably depends on which is more widely used among that audience.&lt;/p&gt;&#xA;" OwnerUserId="3939" LastEditorUserId="3939" LastEditorDisplayName="" LastEditDate="2011-04-14T02:01:51.063" LastActivityDate="2011-04-24T17:14:47.580" Title="How to compare the effectiveness of medical diagnostic techniques?" Tags="&lt;hypothesis-testing&gt;&lt;statistical-significance&gt;" AnswerCount="2" CommentCount="3" FavoriteCount="1" />
  <row Id="8899" PostTypeId="1" AcceptedAnswerId="8906" CreationDate="2011-03-29T07:06:51.563" Score="0" ViewCount="109" Body="&lt;p&gt;Suppose I have 2 variables&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$A$:&lt;br&gt;&#xA;$P(A) =$ 0.01&lt;br&gt;&#xA;$P( \lnot A) =$ 0.99&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And $B$ that depends on $A$:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(B|A) =$ 0.05&lt;br&gt;&#xA;$P( \lnot B|A) =$ 0.95&lt;br&gt;&#xA;$P(B| \lnot A) =$ 0.01&lt;br&gt;&#xA;$P( \lnot B| \lnot A) =$ 0.99  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Applying: &#xA;$$P(B)=\sum_{A}^{ }  P(B|A)P(A)$$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;we get &#xA;$P(B)=(0.01)(0.05)+(0.99)(0.01)=0.0104$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ok, my question is the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I set the probability of $P(B)=1$&lt;br&gt;&#xA;How do I get the values of $P(A)$?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As $B$ depends on $A$, how are all the probabilities affected?&lt;br&gt;&#xA;How to compute&lt;br&gt;&#xA;$P(A)$ ?&lt;br&gt;&#xA;$P(B|A)$?&lt;/p&gt;&#xA;" OwnerUserId="3681" LastEditorUserId="2958" LastEditorDisplayName="" LastEditDate="2011-03-29T14:41:30.803" LastActivityDate="2011-03-29T14:41:30.803" Title="Several questions about conditional probability" Tags="&lt;conditional-probability&gt;" AnswerCount="2" CommentCount="3" FavoriteCount="3" />
  <row Id="8901" PostTypeId="2" ParentId="1564" CreationDate="2011-03-29T08:44:46.397" Score="0" ViewCount="" Body="&lt;p&gt;check this &lt;a href=&quot;http://en.wikipedia.org/wiki/Bayes%27_theorem&quot; rel=&quot;nofollow&quot;&gt;wikipedia&lt;/a&gt; page under the sub-section named extensions, they do show how to derive conditional probability involving more than 2 events.&lt;/p&gt;&#xA;" OwnerUserId="3837" LastActivityDate="2011-03-29T08:44:46.397" />
  <row Id="8903" PostTypeId="1" CreationDate="2011-03-29T09:06:24.640" Score="2" ViewCount="134" Body="&lt;p&gt;I'm computing cosine similarities between 2 vectors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These vectors are information retrieval query and document representations respectively.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;They have been computed using &lt;a href=&quot;http://en.wikipedia.org/wiki/Tf%E2%80%93idf&quot; rel=&quot;nofollow&quot;&gt;tf-idf&lt;/a&gt; weights.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since my documents have different length, tf-idf weights are theoretically unbounded.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The question is: is cosine similarity still a valid measure? Can I compare several cosine similarities for each doc?&lt;/p&gt;&#xA;" OwnerUserId="3941" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-03-29T11:55:26.907" LastActivityDate="2011-05-27T06:31:45.607" Title="Comparing cosine similarities for tf-idf vectors for documents with different length" Tags="&lt;text-mining&gt;&lt;information-retrieval&gt;" AnswerCount="1" />
  <row Id="8904" PostTypeId="1" CreationDate="2011-03-29T09:09:00.380" Score="0" ViewCount="145" Body="&lt;p&gt;I need to run Newey West t statistics in SAS 9.2. I already run regression, White's test, Breusch Godfrey test and Jarque-Bera normality test.&#xA;Regression is simple. Number of observations = 522 &#xA;Depended variable name: rtest 1&#xA;Independent variable name: rtest 2&#xA;Data are time series data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found somewhere that i should start with:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;PROC model;&#xA;PARMS B0 B1;&#xA;rtest1=B0+B1*rtest2;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I do not understand this part:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;FIT rtest1/GMM Kernel=(BART,1,0.605);&#xA;RUN;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How to get numbers in parenthesis. What is meaning of those numbers?&#xA;What does command &lt;code&gt;FIT&lt;/code&gt; do to data?&#xA;Do I need to add more commands?&lt;/p&gt;&#xA;" OwnerUserId="3942" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-03-29T11:27:01.840" LastActivityDate="2011-03-29T11:27:01.840" Title="How to calculate Newey West t-statistic in SAS 9.2?" Tags="&lt;sas&gt;" CommentCount="4" />
  <row Id="8905" PostTypeId="2" ParentId="8899" CreationDate="2011-03-29T09:23:21.043" Score="0" ViewCount="" Body="&lt;p&gt;Using subjective probabilities adding the “information that $P(B)=1$ to the model as an equation” is no different from adding the “data that $B$ is observed to be true”. So you can use the Bayes theorem:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P_{prior} = \begin{smallmatrix} 0.05 \cdot 0.01 &amp;amp; 0.01 \cdot 0.99 &#xD;&#xA;                              \\ 0.95 \cdot 0.01 &amp;amp; 0.99 \cdot 0.99 &#xD;&#xA;\end{smallmatrix}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P_{posterior} = \begin{smallmatrix} 0.05 \cdot 0.01 &amp;amp; 0.01 \cdot 0.99 &#xD;&#xA;                                  \\ 0 &amp;amp; 0 &#xD;&#xA;\end{smallmatrix} \cdot constant = \begin{smallmatrix}&#xD;&#xA;0.0480769230769 &amp;amp; 0.9519230769231&#xD;&#xA;\\ 0 &amp;amp; 0 &#xD;&#xA;\end{smallmatrix}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P_{posterior}(A)={0.05 \cdot 0.01} / ({0.05 \cdot 0.01 + 0.01 \cdot 0.99}) = 0.0480769230769$&lt;/p&gt;&#xA;" OwnerUserId="3911" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-03-29T09:37:12.443" LastActivityDate="2011-03-29T09:37:12.443" CommentCount="2" />
  <row Id="8906" PostTypeId="2" ParentId="8899" CreationDate="2011-03-29T09:39:14.657" Score="1" ViewCount="" Body="&lt;p&gt;When you set $Pr(B)=1$ other things will change, though some can remain the same.  So you have to decide what is remaining the same.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, in the first part, you could have worked out $Pr(A|B)$, $Pr( \lnot A|B)$, $Pr(A| \lnot B) $ and  $Pr( \lnot A| \lnot B)$.  So $Pr(A|B) = \frac{Pr(B|A)Pr(A)}{Pr(B)} = \frac{0.0005}{0.0104} \approx 0.0480769\ldots$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you assume $Pr(A|B)$ stays the same into the second part of the question then $Pr(B)=1$ would give $Pr(A)=\approx 0.0480769\ldots$&lt;/p&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-03-29T09:39:14.657" CommentCount="4" />
  <row Id="8907" PostTypeId="1" AcceptedAnswerId="8910" CreationDate="2011-03-29T09:47:18.770" Score="0" ViewCount="66" Body="&lt;p&gt;I have downloaded the Gaussian Processes for Machine Learning (GPML) package (gpml-matlab-v3.1-2010-09-27.zip) from the website,&#xA;and I can run the regression example (&lt;a href=&quot;http://www.gaussianprocess.org/gpml/code/matlab/doc/index.html&quot; rel=&quot;nofollow&quot;&gt;demoRegression&lt;/a&gt;) in &lt;a href=&quot;http://www.gnu.org/software/octave/&quot; rel=&quot;nofollow&quot;&gt;Octave&lt;/a&gt;. It works just fine.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I have my own data for regression where the x (input) matrix is a 54x10 matrix (54 samples, 10 input vars), and the y (target) vector is 54x1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is that I do not understand hov to calculate the meanfunc and the covfunc, the code provided for the regression example does not work for multiple input datasets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I do not understand enough Octave to decode the specific code used to calculate this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there anybody who has tried this, and maybe can show an example?&lt;/p&gt;&#xA;" OwnerUserId="3943" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-29T12:24:44.713" LastActivityDate="2011-03-29T12:24:44.713" Title="How do I use the  GPML package for multi dimensional input?" Tags="&lt;regression&gt;&lt;machine-learning&gt;&lt;matlab&gt;" AnswerCount="1" />
  <row Id="8908" PostTypeId="1" CreationDate="2011-03-29T10:09:49.400" Score="3" ViewCount="70" Body="&lt;p&gt;I'm working up a taxonomy showing different methods used in pattern recognition and I'd be curious to hear about how it could be improved. The Mind Map groups different methods based on the discipline which influenced their development.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/gdlHc.png&quot; alt=&quot;taxonomy&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="2624" LastEditorUserId="2624" LastEditorDisplayName="" LastEditDate="2011-03-29T12:03:18.673" LastActivityDate="2011-03-29T15:21:19.173" Title="What is this taxonomy missing?" Tags="&lt;algorithms&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="8909" PostTypeId="1" AcceptedAnswerId="8989" CreationDate="2011-03-29T10:26:51.437" Score="4" ViewCount="112" Body="&lt;p&gt;I am designing a data capture method for a client for inplay sporting events and he wants to record the odds movements for later analysis in Excel once every half second. I want to get this right so that it's easy to use the data down the line for analysis in other packages.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A bit more background and assumptions.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Each event can have between 4 - 40&#xA;contenders (c)&lt;/li&gt;&#xA;&lt;li&gt;Each event has 10 variables that&#xA;apply equally to all contenders (e)&lt;/li&gt;&#xA;&lt;li&gt;Each contender has 20 variables of same heading/type with values unique to contender (i)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In essence I need to choose between&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;1. Having 1 timeframe on 1 row, so each&#xA;timeframe capture has&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Columns required = e+max(c)i = 810&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rows required = 1&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good: Easy to manipulate, data on one row, 1 row describes all contenders in event per row.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bad: Huge number of columns, lots of blank column data if c is less than max(c), hard to search names across multiple columns&lt;/p&gt;&#xA;&#xA;&lt;p&gt;or&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;2. Having 1 timeframe on multiple rows,&#xA;so each timeframe has&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Columns required = e+i =30&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rows required = c&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good: Less columns, easy to search/filter as all names in the same column&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bad: Timeframes in different rows for different contenders&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does it matter? Is it easy for packages to handle data in both forms? My client doesn't know the answer but wants the best solution! I'm tending towards 2. as it's much easier to manage and search in database terms but not sure about preparation for time series analysis? Can anyone one with experience offer some advice?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&#xA;Os&lt;/p&gt;&#xA;" OwnerUserId="3484" LastActivityDate="2011-03-31T00:26:24.340" Title="Data collection and storage for time series analysis" Tags="&lt;time-series&gt;&lt;dataset&gt;" AnswerCount="2" />
  <row Id="8910" PostTypeId="2" ParentId="8907" CreationDate="2011-03-29T10:28:10.890" Score="1" ViewCount="" Body="&lt;p&gt;Here is a more minimal example of a 2-d regression problem (I haven't got octave, only matlab, but hopefully the difference won't matter).  meanfunc and covfunc should be happy with any number of inputs, provided that the covariance function doesn't have a hyper-parameter per inpit feature (as e.g. covSEiso does).  Hope this helps&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[X1,X2] = meshgrid(-pi:pi/16:+pi, -pi:pi/16:+pi);&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Y = sin(X1).*sin(X2) + 0.1*randn(size(X1));&lt;/p&gt;&#xA;&#xA;&lt;p&gt;imagesc(Y); drawnow;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;x = [X1(:) X2(:)];&#xA;y = Y(:);&lt;/p&gt;&#xA;&#xA;&lt;p&gt;covfunc = @covSEiso; &#xA;  likfunc = @likGauss; sn = 0.1; hyp.lik = log(sn);&lt;/p&gt;&#xA;&#xA;&lt;p&gt;hyp2.cov = [0 ; 0];&lt;br&gt;&#xA;hyp2.lik = log(0.1);&#xA;hyp2 = minimize(hyp2, @gp, -100, @infExact, [], covfunc, likfunc, x, y);&#xA;exp(hyp2.lik)&#xA;nlml2 = gp(hyp2, @infExact, [], covfunc, likfunc, x, y)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[m s2] = gp(hyp2, @infExact, [], covfunc, likfunc, x, y, x);&lt;/p&gt;&#xA;&#xA;&lt;p&gt;m = reshape(m, size(Y));&lt;/p&gt;&#xA;&#xA;&lt;p&gt;figure(2); imagesc(m);&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2011-03-29T10:28:10.890" CommentCount="2" />
  <row Id="8911" PostTypeId="1" CreationDate="2011-03-29T12:00:38.800" Score="3" ViewCount="198" Body="&lt;p&gt;What free tool can I use to do simple Monte Carlo simulations on OS X?&lt;/p&gt;&#xA;" OwnerUserId="1901" LastActivityDate="2011-04-03T21:31:49.947" CommunityOwnedDate="2011-03-29T13:26:09.840" Title="What free tool can I use to do simple Monte Carlo simulations on OS X?" Tags="&lt;monte-carlo&gt;" AnswerCount="6" CommentCount="3" FavoriteCount="2" />
  <row Id="8912" PostTypeId="2" ParentId="8783" CreationDate="2011-03-29T12:08:09.563" Score="0" ViewCount="" Body="&lt;p&gt;Im not sure how you handled the larger multiset?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You didn't take samples did you?&#xA;I think you just used the mean. Right?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But this wouldn't fit in your formular from wiki.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could you please explain, what you did there?&lt;/p&gt;&#xA;" OwnerUserId="3944" LastActivityDate="2011-03-29T12:08:09.563" />
  <row Id="8913" PostTypeId="2" ParentId="8908" CreationDate="2011-03-29T12:45:45.907" Score="1" ViewCount="" Body="&lt;p&gt;Maximum entropy Markov models could go next to hidden Markov models.&lt;/p&gt;&#xA;" OwnerUserId="3874" LastActivityDate="2011-03-29T12:45:45.907" />
  <row Id="8914" PostTypeId="2" ParentId="8911" CreationDate="2011-03-29T12:46:58.267" Score="8" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://www.r-project.org/&quot; rel=&quot;nofollow&quot;&gt;&lt;img src=&quot;http://i.imgur.com/HZ1At.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is a probability that a sum of a 3 highest results from 5 throws of a dice is divisible by seven?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; mean(replicate(1e5,sum(sort(sample(1:6,5,replace=T))[3:5])%%7==0))&#xA;[1] 0.16068&#xA;&amp;gt; mean(replicate(1e5,sum(sort(sample(1:6,5,replace=T))[3:5])%%7==0))&#xA;[1] 0.16032&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Circa 16%.&lt;/p&gt;&#xA;" OwnerUserId="88" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-29T12:53:10.303" LastActivityDate="2011-03-29T12:53:10.303" CommunityOwnedDate="2011-03-29T13:26:09.840" />
  <row Id="8915" PostTypeId="2" ParentId="8911" CreationDate="2011-03-29T12:56:26.060" Score="3" ViewCount="" Body="&lt;p&gt;My favourite platforms are&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://code.google.com/p/pymc/&quot; rel=&quot;nofollow&quot;&gt;PyMC&lt;/a&gt; and&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.openbugs.info/&quot; rel=&quot;nofollow&quot;&gt;OpenBUGS&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;PyMC runs on OS X out of the box, OpenBUGS is originally for windows, but according to &lt;a href=&quot;http://www.openbugs.info/w/Downloads&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; it can be run using Wine.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastActivityDate="2011-03-29T12:56:26.060" CommunityOwnedDate="2011-03-29T13:26:09.840" CommentCount="1" />
  <row Id="8916" PostTypeId="1" CreationDate="2011-03-29T12:58:21.373" Score="2" ViewCount="19" Body="&lt;p&gt;Greetings, &#xA;I am working on joint and conditional density trees for approximating clique potentials in Bayesian Belief Networks. A brief introduction to topic is available from this paper: &lt;a href=&quot;http://www.autonlab.org/autonweb/14653.html&quot; rel=&quot;nofollow&quot;&gt;http://www.autonlab.org/autonweb/14653.html&lt;/a&gt; in case you'd like to get a better description what I'm talking about.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am looking for an implementation that supports both discrete and continuous variables in a joint probability distribution. The leaf nodes in the learned tree would provide probability distributions for both discrete and continuous variables. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd be very glad to find an open source implementation, but any implementation in any form would work for me, at least as a test for my own implementation (which I'd have to do anyway)&#xA;I've seen this approach described in various papers in different contexts, but I could not find any implementations. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Kind regards&#xA;Seref&lt;/p&gt;&#xA;" OwnerUserId="3280" LastActivityDate="2011-03-29T12:58:21.373" Title="Are there any available implementations of density or conditional density tree learning?" Tags="&lt;bayesian&gt;&lt;multivariate-analysis&gt;&lt;approximation&gt;&lt;decision-trees&gt;" />
  <row Id="8917" PostTypeId="2" ParentId="8883" CreationDate="2011-03-29T13:19:54.503" Score="0" ViewCount="" Body="&lt;p&gt;The reason that wikipedia says $\sum_{i=1}^{N}X_{i}\sim\Gamma(N,\theta)$ is actually because we assume a poisson process. That is, the waiting time for the next event is exponentially distributed $X_{i}\sim EXP(\theta)$. You can prove with the moment generating function that given $X_{i}\sim EXP(\theta)$ for $i=1...N$ IID sample, then $\sum_{i=1}^{N}X_{i}\sim\Gamma(N,\theta)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Briefly put, if you assumed the $\sum_{i=1}^{N}X_{i}\sim\Gamma(N,\theta)$ model for your data, then you can also assume $X_{i}\sim EXP(\theta)$. (Meaning that's what they assumed originally...)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One last thing, when you write $NX_{i}|\sum_{i=1}^{N}X_{i}\sim\Gamma(N,\frac{1}{N}\sum_{i=1}^{N}X_{i})$, that is not correct. Instead, $NX_{i}|\sum_{i=1}^{N}X_{i}\sim UNIFORM(0,\sum_{i=1}^{N}X_{i})$. To more or less convince yourself, think that given the value of $\sum_{i=1}^{N}X_{i}=M$, then $NX_{i}$ cannot possibly be greater than $M$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope this helps!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: I'm wrong about the uniform. See comments.&lt;/p&gt;&#xA;" OwnerUserId="3834" LastEditorUserId="3834" LastEditorDisplayName="" LastEditDate="2011-03-30T04:22:56.787" LastActivityDate="2011-03-30T04:22:56.787" CommentCount="9" />
  <row Id="8918" PostTypeId="1" CreationDate="2011-03-29T13:22:43.783" Score="3" ViewCount="135" Body="&lt;p&gt;Hello,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am curious, is there a way to optimize a regression according to a specific statistic? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say I am interested in a model with the best possible AIC statistic (or MSE or whatever measurement I am interested in) - could I somehow direct the regression to give me the top X models that would do this? (Of course I would not ignore the other measures, but would it be possible to ask for this?) What software supports this, or would you write your own code (in say R)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also in general, when multiple regression results are displayed (let's say an all-possible or best-subset regression is done), is there a ranking and if so, according to what measure/criteria?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not saying that this is the best way to evaluate models, but perhaps it would be a way to explore candidate models? (This is really not my main question though.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks .. learning a lot reading this site.&lt;/p&gt;&#xA;" OwnerUserId="3311" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-29T16:04:10.763" LastActivityDate="2011-04-22T02:56:08.773" Title="Is there a way to optimize regression according to a specific criterion?" Tags="&lt;regression&gt;&lt;model-selection&gt;&lt;summary-statistics&gt;" AnswerCount="3" CommentCount="5" FavoriteCount="1" />
  <row Id="8919" PostTypeId="2" ParentId="8911" CreationDate="2011-03-29T13:28:06.453" Score="2" ViewCount="" Body="&lt;p&gt;To some extent (less strictly statistical) &lt;a href=&quot;http://ccl.northwestern.edu/netlogo/&quot; rel=&quot;nofollow&quot;&gt;NetLogo&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2011-03-29T13:28:06.453" CommunityOwnedDate="2011-03-29T13:28:06.453" />
  <row Id="8920" PostTypeId="2" ParentId="8891" CreationDate="2011-03-29T13:38:45.607" Score="5" ViewCount="" Body="&lt;p&gt;(You may start from the after line section, for a shorter answer) To begin with, you are absolutely right saying that &lt;em&gt;it&lt;/em&gt; firstly &lt;em&gt;depends&lt;/em&gt; on the purposes of your analysis: forecasting of average price (at macro level) or a particular price (at micro level), causal analysis of consumer preferences (district, size, age, number of bedrooms, gas, travelling to the job, level of noise, etc.). This verbal specialization secondly will guide you to an appropriate choice of a &lt;em&gt;model&lt;/em&gt; and, finally, requirements for your &lt;em&gt;data&lt;/em&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;From what you have written, I assume, that you deal with the real estate &lt;em&gt;pricing models&lt;/em&gt;. After quick googling showed there are many ways to specify a model. Quite good starting reference could be Simon P. Leblond's &lt;a href=&quot;https://papyrus.bib.umontreal.ca:8443/jspui/bitstream/1866/257/1/a1.1g979.pdf&quot; rel=&quot;nofollow&quot;&gt;article&lt;/a&gt; &lt;em&gt;Comparing predictive accuracy of real estate pricing models: an applied study for the city of Montreal&lt;/em&gt;. From practical point of view you have to choose between additive or multiplicative regression models. The latter has several advantages as opposed to additive models: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;parameter estimates (but intercept term, &lt;em&gt;junk&lt;/em&gt; regression parameter anyway) are not affected by the changes in scale&lt;/li&gt;&#xA;&lt;li&gt;parameters for log-transformed variables have a nice elasticity interpretation, that ...&lt;/li&gt;&#xA;&lt;li&gt;naturally allows for  &lt;a href=&quot;http://en.wikipedia.org/wiki/Diminishing_returns#Returns_to_scale&quot; rel=&quot;nofollow&quot;&gt;diminishing returns to scale&lt;/a&gt; restrictions (in real estate this one could be crucial restrictions)&lt;/li&gt;&#xA;&lt;li&gt;if one studies average prices, more robust averaging is weighted geometric mean than average (this will not be relevant too much at micro level though)&lt;/li&gt;&#xA;&lt;li&gt;you may set price to zero, if, for instance, your apartment has no bedrooms at all (it is hard to do so with additive models)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;One more important thing before you proceed is to think about each of your observation as a unique data point that was jointly set on the market by a decision maker on the basis of utility maximizing behaviour. &lt;em&gt;Jointly&lt;/em&gt; meaning here that you can't separate the variables from each other (for instance, the value of apartments without a bedroom is zero for most of the consumers), but a consumer may or may not like all bundle of the attributes together, after that his or her budget (money in the pocket) is all that matters. Therefore standardization is useful for analysis of relative importance of explanatory variables, but be careful judging what variables are not significant (all factors may be important). Heterogeneity of preferences and budgets (buyers are different households) in each case of your observation also shows why regression at micro level (not averaging or so) could also be misleading. Finally, you have cross-sectional (static) data. Trying to predict prices for different years (than the year of your observations), static pictures work poorly for different periods of time (say you build model based on 2009 year's data, it will be not very useful retrospectively predicting prices for say 2007, or for 2011). At least try to correct the outcomes on the basis of change in average price for a particular year in this case.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Regarding your particular questions (what I personally do for my projects, or at least pretend to do):&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;List all the variables you have and their measurement units&lt;/li&gt;&#xA;&lt;li&gt;Check and re-check the data for imputation errors&lt;/li&gt;&#xA;&lt;li&gt;Make additional imputation for the points with missing values (you may also simply exclude the observations if you have large dataset with not so many missing values)&lt;/li&gt;&#xA;&lt;li&gt;Make all measurement units the same across similar variables (sq. meters, currency units, etc.)&lt;/li&gt;&#xA;&lt;li&gt;Think of a simple data frame structure at once (you need to communicate with $R$ conveniently)&lt;/li&gt;&#xA;&lt;li&gt;Bring only raw data to $R$, make all log, differences, fractions transformations in $R$ directly (logarithms are important for multiplicative models, some pros for one are in the prelude above; fractions are also nice for you may want to eliminate the scale (size) effect at once, and emphasize the differences caused by other factors)&lt;/li&gt;&#xA;&lt;li&gt;Leave dummies as they are but always leave one level of qualitative attribute for intercept term (if not this would be a source for pure multicollinearity problem in your model)&lt;/li&gt;&#xA;&lt;li&gt;For your purposes you may apply ordinary least squares (OLS), though in pricing models I would also consider tobit or Heckman models, that do need a special treatment  (one of my early may-be-not-so-successful &lt;a href=&quot;http://stats.stackexchange.com/questions/6195/how-to-model-prices/6196#6196&quot;&gt;post's&lt;/a&gt; on pricing was about this)&lt;/li&gt;&#xA;&lt;li&gt;OLS is straightforward and usual residual analysis (found in textbooks on econometrics) is done. Violating some of the assumptions you may go for generalized methods, instrumental variables, ridged regression, cures for autoregressive residuals, but...  What you really need to know: are the parameter estimates theoretically reasonable (values, signs, etc.)?&lt;/li&gt;&#xA;&lt;li&gt;Just a nice number... any additions from the community are welcome.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="2645" LastActivityDate="2011-03-29T13:38:45.607" CommentCount="1" />
  <row Id="8921" PostTypeId="2" ParentId="8883" CreationDate="2011-03-29T14:01:17.720" Score="2" ViewCount="" Body="&lt;p&gt;If you have a Poisson process, then each $X_i$ has an exponential distribution (and their sum a Gamma distribution).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can then use Bayesian methods to look at possible values of $\theta$.  For example, the &lt;a href=&quot;http://en.wikipedia.org/wiki/Exponential_distribution#Bayesian_inference&quot; rel=&quot;nofollow&quot;&gt;conjugate prior for an exponential distribution&lt;/a&gt; is also a Gamma distribution, which you may find helpful incorporating your prior information and observed data to get a credible interval for $\theta$. &lt;/p&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-03-29T14:01:17.720" CommentCount="1" />
  <row Id="8922" PostTypeId="2" ParentId="8891" CreationDate="2011-03-29T14:11:32.650" Score="1" ViewCount="" Body="&lt;p&gt;For preprocessing I always like to include outlier detection, and removing bad data.  If your data is of different scales, normalizing the data is a good idea (standardization).  As far as technique goes, it always pays to graph and plot all of your variables with each other, as well as with the predicted variable.  That will tell you a lot about which assumptions you can make about the data such as linearity, equality of variances, normality and can better help you choose a technique.&lt;/p&gt;&#xA;" OwnerUserId="3489" LastActivityDate="2011-03-29T14:11:32.650" />
  <row Id="8923" PostTypeId="1" AcceptedAnswerId="8929" CreationDate="2011-03-29T14:47:42.687" Score="4" ViewCount="54" Body="&lt;p&gt;I have meteorological records for a point, these include temperature and solar irradiation. I want to plot them against another variable, that we shall call Rating, and see how the data are distributed. This is what I'm doing:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;d1 &amp;lt;- ggplot(data = mydata, aes(Temperature, Rating, fill = ..density..)) + stat_binhex(na.rm = TRUE) + opts(aspect.ratio = 1)#, legend.position = &quot;none&quot;) &#xA;d2 &amp;lt;- ggplot(data = mydata, aes(Solar.Irrad, Rating, fill = ..density..)) + stat_binhex(na.rm = TRUE) + opts(aspect.ratio = 1)#, legend.position = &quot;none&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I get both things on the same window by using grid.arrange from package gridExtra:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;grid.arrange(d1,d2, nrow=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/dPHEk.png&quot; alt=&quot;Two graphs together, different z scale&quot;&gt;&#xA;This produces the image shown. Now, my problem is that I would really like that both graphs shared their z scale: the legend should be the same and the color scheme should be homogeneous through different graphs. Is this possible? I'm totally lost here, do anyone know of a way to do this?&lt;/p&gt;&#xA;" OwnerUserId="1504" LastEditorUserId="696" LastEditorDisplayName="" LastEditDate="2011-03-29T15:57:12.923" LastActivityDate="2011-03-29T15:57:12.923" Title="Plotting multiple binhex with the same z levels" Tags="&lt;r&gt;&lt;ggplot2&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="8924" PostTypeId="1" AcceptedAnswerId="8931" CreationDate="2011-03-29T15:06:06.213" Score="4" ViewCount="152" Body="&lt;p&gt;Hi!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would greatly appreciate your advice on the following problem:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've got a large continuous dataset with lots of zeros (~95%) and I need to find the best way to test whether certain subsets of it are &quot;interesting&quot;, ie don't seem to be drawn from the same distribution as the rest. Zero inflation comes from the fact that each data point is based a count measurement with both true and sampling zeros, but the result is continuous as it takes into account some other parameters weighted by the count (and so if the count is zero, the result is also zero).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What would be the best way to do this? I have a feeling that Wilcoxon and even brute-force permutation tests are inadequate as they get skewed by these zeros. Focussing on non-zero measurements also removes true zeros that are extremely important. Zero-inflated models for count data are well-developed, but unsuitable for my case. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I considered fitting a Tweedie distribution to the data and then fitting a glm on response=f(subset_label). Theoretically, this seems feasible, but I'm wondering whether (a) this is overkill and (b) would still implicitly assume that all zeros are sample zeros, ie would be biased in the same way (at best) as a permutation?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Intuitively, it sounds like have some kind of hierarchical design that combines a binomial statistic based on the proportion of zeros and, say, a Wilcoxon statistic computed on non-zero values (or, better still, non-zero values supplemented with a fraction of zeros based on some prior). Sounds like a Bayesian network... &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hopefully I'm not the first one having this problem, so would be very grateful if you could point me to suitable existing techniques...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks!&lt;/p&gt;&#xA;" OwnerUserId="3945" LastEditorUserId="3945" LastEditorDisplayName="" LastEditDate="2011-03-29T15:41:14.403" LastActivityDate="2011-03-30T03:01:12.940" Title="Hypothesis testing on zero-inflated continuous data" Tags="&lt;hypothesis-testing&gt;" AnswerCount="3" CommentCount="2" />
  <row Id="8925" PostTypeId="2" ParentId="8911" CreationDate="2011-03-29T15:14:59.303" Score="1" ViewCount="" Body="&lt;p&gt;You can even do Monte Carlo Simulation in Excel.  It's not a perfect tool, but you probably already have it and know how to use it.  Depending on the scope of your problem, it might be easier to use Excel than to learn something new.  If you are going to learn something new, R is a great choice.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are you trying to simulate?&lt;/p&gt;&#xA;" OwnerUserId="2817" LastActivityDate="2011-03-29T15:14:59.303" CommunityOwnedDate="2011-03-29T15:14:59.303" CommentCount="2" />
  <row Id="8926" PostTypeId="2" ParentId="8908" CreationDate="2011-03-29T15:21:19.173" Score="1" ViewCount="" Body="&lt;p&gt;I would add random forests next to trees, or perhaps under a new category of ensemble methods.&lt;/p&gt;&#xA;" OwnerUserId="2817" LastActivityDate="2011-03-29T15:21:19.173" />
  <row Id="8927" PostTypeId="2" ParentId="8918" CreationDate="2011-03-29T15:32:45.283" Score="4" ViewCount="" Body="&lt;p&gt;Statistical appropriateness aside, R provides some nice functions that allow for this type of analysis. You can take a look at the &lt;code&gt;leaps()&lt;/code&gt; function within the &lt;a href=&quot;http://cran.r-project.org/web/packages/leaps/index.html&quot; rel=&quot;nofollow&quot;&gt;&lt;code&gt;leaps&lt;/code&gt; package&lt;/a&gt;. The &lt;code&gt;leaps()&lt;/code&gt; function returns the top &lt;code&gt;n&lt;/code&gt; models and some fit statistics for a given set of parameters. &lt;code&gt;stepAIC()&lt;/code&gt; within &lt;code&gt;MASS&lt;/code&gt; is another handy function for this type of analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There's a decent tutorial on the &lt;a href=&quot;http://www.statmethods.net/stats/regression.html&quot; rel=&quot;nofollow&quot;&gt;statmethods&lt;/a&gt; site describing these and some other techniques. &lt;/p&gt;&#xA;" OwnerUserId="696" LastActivityDate="2011-03-29T15:32:45.283" CommentCount="2" />
  <row Id="8928" PostTypeId="2" ParentId="8749" CreationDate="2011-03-29T15:52:01.003" Score="4" ViewCount="" Body="&lt;p&gt;There is a package for &quot;&lt;a href=&quot;http://www.r-project.org/&quot; rel=&quot;nofollow&quot;&gt;R&lt;/a&gt;&quot; called &quot;&lt;a href=&quot;http://cran.r-project.org/web/packages/caret/index.html&quot; rel=&quot;nofollow&quot;&gt;caret&lt;/a&gt;,&quot; which stands for &quot;classification and regression testing.&quot;  I think it would be a good place for you to start, as it will easily allow you to apply a dozen or so different learning algorithms to your data, and then cross-validate them to estimate how accurate they each are.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an example that you can modify with your own data/other methods:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;install.packages('caret',dependencies = c('Depends','Suggests'))&#xA;library(caret)&#xA;&#xA;set.seed(999)&#xA;Precursor1 &amp;lt;- runif(25)&#xA;Precursor2 &amp;lt;- runif(25)&#xA;Target &amp;lt;- sample(c('T','F'),25,replace=TRUE)&#xA;MyData &amp;lt;- data.frame(Precursor1,Precursor2,Target)&#xA;str(MyData)&#xA;&#xA;#Try Logistic regression&#xA;model_Logistic &amp;lt;- train(Target~Precursor1+Precursor2,data=MyData,method='glm')&#xA;&#xA;#Try Neural Network&#xA;model_NN &amp;lt;- train(Target~Precursor1+Precursor2,data=MyData,method='nnet',trace=FALSE)&#xA;&#xA;#Try Naive Bayes&#xA;model_NB &amp;lt;- train(Target~Precursor1+Precursor2,data=MyData,method='nb')&#xA;&#xA;#Try Random Forest&#xA;model_RF &amp;lt;- train(Target~Precursor1+Precursor2,data=MyData,method='rf')&#xA;&#xA;#Try Support Vector Machine&#xA;model_SVM&amp;lt;- train(Target~Precursor1+Precursor2,data=MyData,method='svmLinear')&#xA;&#xA;#Try Nearest Neighbors&#xA;model_KNN&amp;lt;- train(Target~Precursor1+Precursor2,data=MyData,method='knn')&#xA;&#xA;#Compare the accuracy of each model&#xA;cat('Logistic:',max(model_Logistic$results$Accuracy))&#xA;cat('Neural:',max(model_NN$results$Accuracy))&#xA;cat('Bayes:',max(model_NB$results$Accuracy))&#xA;cat('Random Forest:',max(model_RF$results$Accuracy))&#xA;cat('Support Vector Machine:',max(model_SVM$results$Accuracy))&#xA;cat('Nearest Neighbors:',max(model_KNN$results$Accuracy))&#xA;&#xA;#Look at other available methods&#xA;?train&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Another idea would be to break your data into a training set and a test set, and then compare how each model performs on the test set.  If you like, I can show you how to do that.&lt;/p&gt;&#xA;" OwnerUserId="2817" LastActivityDate="2011-03-29T15:52:01.003" />
  <row Id="8929" PostTypeId="2" ParentId="8923" CreationDate="2011-03-29T15:54:50.117" Score="4" ViewCount="" Body="&lt;p&gt;Have you thought about using the faceting capabilities within ggplot2 directly? You can allow for free scales as a parameter in the call to &lt;code&gt;facet_wrap()&lt;/code&gt;. Here's an example for you:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(ggplot2)&#xA;library(hexbin)&#xA;#Sample data.frame.&#xA;dat &amp;lt;- data.frame(&#xA;  rating = sample(500:2500, 1000, TRUE)&#xA;  , Solar.Radiation = sample(0:1200, 1000, TRUE)&#xA;  , Ambient.Temperature = sample(-10:25, 1000, TRUE)&#xA;  )&#xA;&#xA;#Melt data.frame for plotting&#xA;dat.m &amp;lt;- melt(dat, id.vars = &quot;rating&quot;)&#xA;&#xA;#Plotting&#xA;ggplot(dat.m, aes(value, rating, fill = ..density..)) +&#xA;  stat_binhex(na.rm = TRUE) +  opts(aspect.ratio = 1) +&#xA;  facet_wrap(facets = ~  variable, scales = &quot;free_x&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="696" LastActivityDate="2011-03-29T15:54:50.117" CommentCount="2" />
  <row Id="8930" PostTypeId="1" CreationDate="2011-03-29T16:14:24.407" Score="4" ViewCount="108" Body="&lt;p&gt;Hi,&#xA;As I've heard of the AdaBoost classifier repeatedly mentioned at work, I wanted to get a better feel for how it works and when one might want to use it. I've gone ahead and read a number of papers and tutorials on it which I found on Google, but there are aspects of the classifier which I'm still having trouble understanding:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Most tutorials I've seen speak of AdaBoost as finding the best weighted combination of many classifiers. This makes sense to me. What does not make sense are implementations (i.e. MALLET) where AdaBoost seems to only accept one weak learner. How does this make any sense? If there's only one classifier provided to AdaBoost, shouldn't it just return back that same classifier with a weight of 1? How does it produce new classifiers from the first classifier?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;When would one actually want to use AdaBoost? I've read that it's supposed to be one of the best out-of-the-box classifiers, but when I try boosting a MaxEnt classifier I was getting f-scores of 70%+ with, AdaBoost murders it and gives me f-scores of something like 15% with very high recall and very low precision instead. So now I'm confused. When would I ever want to use AdaBoost? I'm looking for more of an intuitive rather than strictly statistical answer, if possible. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="3948" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-29T20:19:29.457" LastActivityDate="2011-03-29T20:19:29.457" Title="When would one want to use AdaBoost?" Tags="&lt;machine-learning&gt;&lt;boosting&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="8931" PostTypeId="2" ParentId="8924" CreationDate="2011-03-29T17:17:43.460" Score="2" ViewCount="" Body="&lt;p&gt;@msp, I think you are looking at a two stage model in that attachment (I did not have time to read it), but zero inflated continuous data is the type I work with a lot. To fit a parametric model to this data (to allow hypothesis tests) you can fit a two stage but then you have two models (Y is the target and X are covariates): P(Y=0 |X) and P(Y|X;Y&gt;0). You have to use simulation to &quot;bring&quot; these together. Gelmans &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/052168689X&quot; rel=&quot;nofollow&quot;&gt;book&lt;/a&gt; (and the arm package in R) shows this process for this exact model (using logistic regression and ordinary linear regression with a log link). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The other option I have seen and like better is to fit a zero inflated gamma regression, which is the same as above (but gamma as the error instead of guassian) and you can bring them together for hypothesis tests on P(Y|X). I dont know how to do this in R, but you can in SAS NLMIXED. See this &lt;a href=&quot;http://listserv.uga.edu/cgi-bin/wa?A2=ind0805A&amp;amp;L=sas-l&amp;amp;P=R20779&quot; rel=&quot;nofollow&quot;&gt;post&lt;/a&gt;, it works well.  &lt;/p&gt;&#xA;" OwnerUserId="2040" LastActivityDate="2011-03-29T17:17:43.460" CommentCount="10" />
  <row Id="8932" PostTypeId="2" ParentId="8918" CreationDate="2011-03-29T17:43:06.107" Score="1" ViewCount="" Body="&lt;p&gt;Try wle.cp from package wle:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://cran.r-project.org/web/packages/wle/index.html&quot; rel=&quot;nofollow&quot;&gt;http://cran.r-project.org/web/packages/wle/index.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's based on Mallow's Cp:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Mallows%27_Cp&quot; rel=&quot;nofollow&quot;&gt;Mallow's Cp&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's the example given in the reference manual.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(wle)&#xA;x.data &amp;lt;- c(runif(60,20,80),runif(5,73,78))&#xA;e.data &amp;lt;- rnorm(65,0,0.6)&#xA;y.data &amp;lt;- 8*log(x.data+1)+e.data&#xA;y.data[61:65] &amp;lt;- y.data[61:65]-4&#xA;z.data &amp;lt;- c(rep(0,60),rep(1,5))&#xA;plot(x.data,y.data,xlab=&quot;X&quot;,ylab=&quot;Y&quot;)&#xA;&#xA;xx.data &amp;lt;- cbind(x.data,x.data^2,x.data^3,log(x.data+1))&#xA;colnames(xx.data) &amp;lt;- c(&quot;X&quot;,&quot;X^2&quot;,&quot;X^3&quot;,&quot;log(X+1)&quot;)&#xA;result &amp;lt;- wle.cp(y.data~xx.data,boot=10,group=10,num.sol=2)&#xA;summary(result)&#xA;plot(result,num.max=15)&#xA;&#xA;result &amp;lt;- wle.cp(y.data~xx.data+z.data,boot=10,group=10,num.sol=2)&#xA;summary(result)&#xA;plot(result,num.max=15)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The output from the last summary(result) statement is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Call:&#xA;wle.cp(formula = y.data ~ xx.data + z.data, boot = 10, group = 10, &#xA;    num.sol = 2)&#xA;&#xA;&#xA;Weighted Mallows Cp:&#xA;      (Intercept) xx.dataX xx.dataX^2 xx.dataX^3 xx.datalog(X+1) z.data   wcp&#xA; [1,]           0        0          0          0               1      1 1.570&#xA; [2,]           1        0          0          0               1      1 2.372&#xA; [3,]           0        1          0          0               1      1 2.510&#xA; [4,]           0        0          1          0               1      1 2.564&#xA; [5,]           0        0          0          1               1      1 2.570&#xA; [6,]           1        1          1          1               0      1 4.088&#xA; [7,]           0        1          1          1               1      1 4.289&#xA; [8,]           1        0          1          1               1      1 4.530&#xA; [9,]           1        1          0          1               1      1 4.710&#xA;[10,]           1        1          1          0               1      1 4.888 &#xA;[11,]           1        1          1          1               1      1 6.000&#xA;&#xA;Printed the first  11  best models &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The top model (row [1,] above), which uses log(X+1) from xx.data (see the above 1 under xx.datalog(X+1)) and z.data (see the above 1 under z.data) has the lowest Mallow's Cp value (wcp = 1.57).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The final plot(result,num.max=15) statement provides the following graph where any green model under the black line follows Mallow's criteria.  The blue model in the lower left area is the &quot;best Mallow's Cp model&quot; (see the above list).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/M1jRI.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="2775" LastEditorUserId="2775" LastEditorDisplayName="" LastEditDate="2011-04-22T02:56:08.773" LastActivityDate="2011-04-22T02:56:08.773" CommentCount="1" />
  <row Id="8933" PostTypeId="1" AcceptedAnswerId="8940" CreationDate="2011-03-29T18:10:41.033" Score="3" ViewCount="120" Body="&lt;p&gt;I have some samples from eight people who all gave the same answer to a question. Now, obviously the sample's mean is the answer all people gave, and the standard dev is 0. Excel throws a &lt;code&gt;#NUM!&lt;/code&gt; error when I call the function&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;CONFIDENCE.T(0.05, K33, COUNTA(B33:I33))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where &lt;code&gt;K33&lt;/code&gt; is the standard dev (0).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What would be the correct interpretation of this? Can I even calculate a confidence interval?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For clarification: People are asked to give their opinion on a scale of (1, 2, 3, 4, 5), which is ordinal. Nevertheless, one always calculates the arithmetic mean of all judgements (according to &lt;a href=&quot;http://www.itu.int/rec/dologin_pub.asp?lang=e&amp;amp;id=T-REC-P.800-199608-I!!PDF-E&amp;amp;type=items&quot; rel=&quot;nofollow&quot;&gt;ITU-T P.800&lt;/a&gt;, see also: &lt;a href=&quot;http://en.wikipedia.org/wiki/Mean_opinion_score&quot; rel=&quot;nofollow&quot;&gt;Wikipedia&lt;/a&gt;), so that's why I also want to get a confidence interval.&lt;/p&gt;&#xA;" OwnerUserId="1205" LastEditorUserId="1205" LastEditorDisplayName="" LastEditDate="2011-03-29T18:31:48.777" LastActivityDate="2011-03-30T03:25:30.340" Title="Excel's confidence interval function throws #NUM! when standard deviation is 0" Tags="&lt;confidence-interval&gt;&lt;standard-deviation&gt;&lt;excel&gt;" AnswerCount="4" FavoriteCount="1" />
  <row Id="8934" PostTypeId="2" ParentId="8933" CreationDate="2011-03-29T18:17:12.657" Score="1" ViewCount="" Body="&lt;p&gt;If eight samples from a distribution are exactly the same it is probably not a normal distribution or you use rounding at a higher order of magnitude than of the standard deviation. Or are you calculating means on a numerically coded ordinal scale?&lt;/p&gt;&#xA;" OwnerUserId="3911" LastActivityDate="2011-03-29T18:17:12.657" CommentCount="2" />
  <row Id="8936" PostTypeId="2" ParentId="8924" CreationDate="2011-03-29T18:40:05.443" Score="0" ViewCount="" Body="&lt;p&gt;You could treat the exact number of zeros unknown, but constrained between 0 and the observed number of zeros. This can surely be handled using a Bayesian formulation of the model. Maybe a multiple imputation method can also be tweaked to appropriately vary the weights (between 0 and 1) of the zero observations…&lt;/p&gt;&#xA;" OwnerUserId="3911" LastActivityDate="2011-03-29T18:40:05.443" />
  <row Id="8937" PostTypeId="2" ParentId="3634" CreationDate="2011-03-29T19:00:57.517" Score="0" ViewCount="" Body="&lt;p&gt;Pierre Legendre has some Code on his homepage:&#xA;&lt;a href=&quot;http://www.bio.umontreal.ca/legendre/indexEn.html&quot; rel=&quot;nofollow&quot; title=&quot;nest.anova.perm.R &amp;#40;D. Borcard and P. Legendre&amp;#41;: Nested anova with permutation tests &amp;#40;main factor and one nested factor, balanced design&amp;#41;.&quot;&gt;nest.anova.perm.R (D. Borcard and P. Legendre): Nested anova with permutation tests (main factor and one nested factor, balanced design).&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1050" LastActivityDate="2011-03-29T19:00:57.517" />
  <row Id="8938" PostTypeId="2" ParentId="8930" CreationDate="2011-03-29T19:06:49.200" Score="3" ViewCount="" Body="&lt;p&gt;Adaboost can use multiple instances of the same classifier with different parameters. Thus, a previously linear classifier can be combined into nonlinear classifiers. Or, as the AdaBoost people like to put it, multiple weak learners can make one strong learner. A nice picture can be found &lt;a href=&quot;http://doc.prsdstudio.com/2.1/kb/16.html&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;, on the bottom.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically, it goes as with any other learning algorithm: on some datasets it works, on some it doesn't. There sure are datasets out there, where it excels. And maybe you haven't chosen the right weak learner yet. Did you try logistic regression? Did you visualize how the decision boundaries evolve during adding of learners? Maybe you can tell what is going wrong.&lt;/p&gt;&#xA;" OwnerUserId="2860" LastActivityDate="2011-03-29T19:06:49.200" CommentCount="3" />
  <row Id="8939" PostTypeId="2" ParentId="8933" CreationDate="2011-03-29T19:15:00.720" Score="2" ViewCount="" Body="&lt;p&gt;Let's assume all your 8 subjects chose to answer 3 on the (1, 2, 3, 4, 5) scale. Let's assume that their opinions were continuous in their minds, and they rounded it to the closest values of the scale.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This means that the original opinion of each subject was in the range $[2.5, 3.5)$.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; mean(replicate(1e5, diff(range(rnorm(8)))))&#xA;[1] 2.841661&#xA;&amp;gt; mean(replicate(1e5, diff(range(rnorm(8)))))&#xA;[1] 2.847447&#xA;&amp;gt; 1 / 2.845&#xA;[1] 0.3514938&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The above simulation shows that if you take 8 samples from a normal distribution of sd 0.35 they will cover an interval of the approximate width of 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus in your population the sd is likely to be 0.35 or less. Rounding to one of 1, 2, 3, 4, 5 is not precise enough to measure the sd in this case.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-03-29T19:20:45.847" LastActivityDate="2011-03-29T19:20:45.847" CommentCount="3" />
  <row Id="8940" PostTypeId="2" ParentId="8933" CreationDate="2011-03-29T19:15:03.043" Score="5" ViewCount="" Body="&lt;p&gt;This behavior is questionable but documented.  The help for &quot;confidence&quot; states:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;If standard_dev ≤ 0, CONFIDENCE returns the #NUM! error value. &#xA;  ...&#xA;  If we assume alpha equals 0.05, we need to calculate the area under the standard normal&#xA;  curve that equals (1 - alpha), or 95 percent. This value is ± 1.96. The confidence &#xA;  interval is therefore: &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;$$\bar{x} \pm 1.96\left(\frac{\sigma}{\sqrt{n}}\right).$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Yes, this is badly phrased, but that's a direct quote.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To overcome these (somewhat artificial) limitations, compute the confidence limits yourself (according to this formula) as&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;=AVERAGE(X) + NORMSINV(1-0.05/2) * STDEV(X)/SQRT(COUNT(X))&#xA;=AVERAGE(X) - NORMSINV(1-0.05/2) * STDEV(X)/SQRT(COUNT(X))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where 'X' names a range containing your data (such as B33:I33) and '0.05' is $\alpha$ (the complement of the desired confidence), just as before.  In your case, because STDEV(X) is 0, both limits will equal the mean.  This is legitimate, although it has its own problems (because it almost surely fails to cover the true mean).&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-03-29T19:15:03.043" CommentCount="2" />
  <row Id="8942" PostTypeId="1" CreationDate="2011-03-29T20:36:50.900" Score="2" ViewCount="75" Body="&lt;p&gt;How do I estimate the noise correlation matrix (psi function) in the augmented state approach of Kalman filter?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can I do something like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\text{noise}_2 = \psi \cdot \text{noise}_1\text{;}\quad\text{noise}_3 = \psi \cdot \text{noise}_2 = \psi^2 \cdot \text{noise}_1\text{;}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;so &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\text{noise}_n = \psi^{n-1} \cdot \text{noise}_1 \quad \text{and}\quad \psi = \sqrt[n]{\frac{\text{noise}_n}{\text{noise}_1}}.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please help!&lt;/p&gt;&#xA;" OwnerUserId="3952" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-30T07:59:43.663" LastActivityDate="2011-03-30T07:59:43.663" Title="Estimating noise correlation in augmented state vector Kalman filter" Tags="&lt;autocorrelation&gt;&lt;kalman-filter&gt;" CommentCount="1" />
  <row Id="8943" PostTypeId="5" CreationDate="2011-03-29T20:52:24.643" Score="0" ViewCount="" Body="" OwnerUserId="-1" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-29T20:52:24.643" LastActivityDate="2011-03-29T20:52:24.643" />
  <row Id="8944" PostTypeId="4" CreationDate="2011-03-29T20:52:24.643" Score="0" ViewCount="" Body="The Binomial distribution gives the frequencies of &quot;successes&quot; in a fixed number of independent &quot;trials.&quot;  Use this tag for questions about data that might be Binomially distributed or for questions about the theory of this distribution." OwnerUserId="-1" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-29T20:52:24.643" LastActivityDate="2011-03-29T20:52:24.643" />
  <row Id="8945" PostTypeId="5" CreationDate="2011-03-29T21:04:21.467" Score="0" ViewCount="" Body="" OwnerUserId="-1" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-29T21:04:21.467" LastActivityDate="2011-03-29T21:04:21.467" />
  <row Id="8946" PostTypeId="4" CreationDate="2011-03-29T21:04:21.467" Score="0" ViewCount="" Body="Use this tag for regression models (*q.v.*) that are nonlinear functions of the *parameters* (not the data!)." OwnerUserId="-1" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-29T21:04:21.467" LastActivityDate="2011-03-29T21:04:21.467" />
  <row Id="8947" PostTypeId="1" CreationDate="2011-03-29T22:44:34.297" Score="0" ViewCount="70" Body="&lt;p&gt;Let's say I have a matrix where the underlying data is from the entire population. Let's say I also have a matrix where the underlying data has been sampled from population.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How would I analyze how well the the sample represents the population?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My gut instinct is to convert it into a row-col listing:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;row col popvalue samplevalue&#xA;1   1   34.5     33.2&#xA;[...]&#xA;i   j   54.4     51.2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and then run ANOVA between the two. Does this sound right or is there a better way to measure this?&lt;/p&gt;&#xA;" OwnerUserId="1965" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-29T22:46:50.257" LastActivityDate="2011-03-30T01:32:16.820" Title="How do I compare a matrix of population values to a matrix of sample values?" Tags="&lt;sampling&gt;&lt;matrix&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="8948" PostTypeId="2" ParentId="8947" CreationDate="2011-03-30T01:19:31.500" Score="1" ViewCount="" Body="&lt;p&gt;This is difficult to answer because the within-cell variance is unknown. The variances of the popvalue and samplevalue will give you only between-cell variance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With two-way ANOVA, you could model the average difference between popvalue and samplevalue by row and by column. This doesn't tell you that much. As @whuber mentioned, you may be interested in a different parameter. Moreover, you may be interested at cell-level variation rather than just row- or column- level variation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another thing to consider: If you convert it to that sort of row-column listing and treat each cell as independent, you'll lose information about spatial correlations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If these are aggregated data, I think you'll want to run some sort of chi-square test on the un-aggregated data. But more information about your situation would help.&lt;/p&gt;&#xA;" OwnerUserId="3874" LastActivityDate="2011-03-30T01:19:31.500" />
  <row Id="8949" PostTypeId="2" ParentId="8947" CreationDate="2011-03-30T01:32:16.820" Score="1" ViewCount="" Body="&lt;p&gt;There's many different kinds of matrices that can be obtained from data. Off the top of my head:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;a covariance matrix: you have several variables measured on each data point, and you want to see how they are correlated&lt;/li&gt;&#xA;&lt;li&gt;a misclassification matrix: you have a categorical output or response variable, and another variable that attempts to estimate or predict the response&lt;/li&gt;&#xA;&lt;li&gt;a grid of spatial measurements: adjacent rows/columns are in some sense closer to each other than to non-adjacent rows/columns&lt;/li&gt;&#xA;&lt;li&gt;a contingency table of counts or cell averages&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The methods you'd use to analyse these matrices would differ. I'd guess that your matrix is the last one judging from the ANOVA mention, but it would be good to have more information.&lt;/p&gt;&#xA;" OwnerUserId="1569" LastActivityDate="2011-03-30T01:32:16.820" />
  <row Id="8950" PostTypeId="2" ParentId="8924" CreationDate="2011-03-30T03:01:12.940" Score="0" ViewCount="" Body="&lt;p&gt;A similar approach to the Fletcher paper is used in marketing testing, where we can arbitrarily separate the effects of interventions (such as advertising) into (a) a change in the number buying the brand (i.e. proportion of zeroes) and (b) a change in the frequency of buying the band (sales given sales occur at all).  This is a solid approach and conceptually meaningful in the marketing context and in the ecological context Fletcher discusses. In fact, this can be extended to (c) a change in the size of each purchase.&lt;/p&gt;&#xA;" OwnerUserId="3919" LastActivityDate="2011-03-30T03:01:12.940" CommentCount="1" />
  <row Id="8951" PostTypeId="2" ParentId="8918" CreationDate="2011-03-30T03:10:44.310" Score="1" ViewCount="" Body="&lt;p&gt;At the simple end of the spectrum: Minitab will do &quot;best subsets regression&quot;, which will find the best one predictor model, the best two predictor model, the best 3 predictor model, the best 4 predictor model, etc. The criterion is r-squared.&lt;/p&gt;&#xA;" OwnerUserId="3919" LastActivityDate="2011-03-30T03:10:44.310" CommentCount="1" />
  <row Id="8952" PostTypeId="2" ParentId="8933" CreationDate="2011-03-30T03:25:30.340" Score="0" ViewCount="" Body="&lt;p&gt;Let's suppose that you have a number of instances in which the average rating is 3.  Each of these will have a variance -- if the raters all answered &quot;3&quot; then that variance will be zero.  In such cases, why not use the average of the variances in which the average rating is 3 (including your 0 value)?  This will give you a real number and a reasonable confidence interval.  I would use median rather than mean to &quot;average&quot; the variances, since it is less subject to extremes (although extremes would be unlikely on a fixed 5 point scale).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, you might decide that any average rating in some range (such as 2.5 to 3.499) counts as &quot;3&quot; in order to give you more values to average.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This procedure is simple and intuitive.  I like whuber's approach as well, but then somebody is going to ask you &quot;why 95%? why not some other %&quot;. You are less likely to get this question if you take a simple average.&lt;/p&gt;&#xA;" OwnerUserId="3919" LastActivityDate="2011-03-30T03:25:30.340" CommentCount="1" />
  <row Id="8953" PostTypeId="2" ParentId="8884" CreationDate="2011-03-30T03:34:38.853" Score="0" ViewCount="" Body="&lt;p&gt;mpiktas gave a very good technical answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There's a nice simulation of this here:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://onlinestatbook.com/stat_sim/sampling_dist/index.html&quot; rel=&quot;nofollow&quot;&gt;http://onlinestatbook.com/stat_sim/sampling_dist/index.html&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can manipulate the distribution at the top of this demo to show a combination of different distributions (such as bimodal), and the distribution of sample means will still be normal.&lt;/p&gt;&#xA;" OwnerUserId="3919" LastActivityDate="2011-03-30T03:34:38.853" />
  <row Id="8954" PostTypeId="2" ParentId="258" CreationDate="2011-03-30T04:20:40.470" Score="1" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/K-means_clustering&quot; rel=&quot;nofollow&quot;&gt;K-means&lt;/a&gt; clustering for unsupervised learning.&lt;/p&gt;&#xA;" OwnerUserId="3270" LastActivityDate="2011-03-30T04:20:40.470" CommunityOwnedDate="2011-03-30T04:20:40.470" />
  <row Id="8955" PostTypeId="1" CreationDate="2011-03-30T09:12:59.377" Score="6" ViewCount="148" Body="&lt;p&gt;Hi everyone, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am looking for any help, advice or tips in how to explain heterogeneity/ heteroscedasticity to biologists in my department. In particular I want to explain why its important to look for it and deal with it if it exists, I was looking for opinions on the following questions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Does heterogeneity influence the reliability of random effect estimates? I am pretty sure it does, but I couldn't find a paper. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) How serious a problem is heterogeneity? I have found conflicting views on this, while some say that it makes model standard errors etc. will be unreliable I have also read that it is only a problem if the heterogeneity is severe. How severe is severe?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) Advice on modelling heterogeneity. Currently, I focus largely on the nlme package in R and the use of variance covariates, this is pretty simple and most people here use R so providing scripts is useful. I am also using the MCMCglmm package as well, but other suggestions are welcome, particularly for non-normal data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4) Any other suggestions are welcome.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="3136" LastActivityDate="2011-03-30T11:51:54.740" Title="Advice on explaining heterogeneity/ heteroscedasticty" Tags="&lt;regression&gt;&lt;residuals&gt;&lt;heteroscedasticity&gt;" AnswerCount="2" CommentCount="4" FavoriteCount="1" />
  <row Id="8956" PostTypeId="1" CreationDate="2011-03-30T09:28:01.670" Score="4" ViewCount="362" Body="&lt;p&gt;I am wanting to run correlations on a number of measurements where Likert scales were used. Looking at the scatterplots it appears the assumptions of linearity and homoscedasticity may have been violated. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Given that there appears to be some debate around ordinal level rating approximating interval level scaling should I play it safe and use Spearman's Rho rather than Pearson's r? &lt;/li&gt;&#xA;&lt;li&gt;Is there a reference that I can cite if I go with Spearman's Rho?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="3958" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-30T15:31:35.083" LastActivityDate="2011-03-30T15:31:35.083" Title="Spearman's or Pearson's correlation with Likert scales where linearity and homoscedasticity may be violated" Tags="&lt;correlation&gt;&lt;scales&gt;&lt;homogeneity&gt;&lt;likert&gt;" AnswerCount="2" FavoriteCount="2" />
  <row Id="8957" PostTypeId="2" ParentId="8956" CreationDate="2011-03-30T09:56:10.943" Score="2" ViewCount="" Body="&lt;p&gt;You should almost certainly go for Spearman's rho or Kendall's tau. Often, if the data is non-normal but variances are equal, you can go for Pearson's r as it doesnt make a huge amount of difference. If the variances are significantly different, then you need a non parametric method. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could probably cite almost any introductory statistics textbook to support your use of Spearman's rho.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Update: if the assumption of linearity is violated, then you should not be using a correlation coefficient on your data, as they all assume a linear relationship (AFAIK). You could look at log transforming your data as this might deal with the non-linearity. &lt;/p&gt;&#xA;" OwnerUserId="656" LastActivityDate="2011-03-30T09:56:10.943" />
  <row Id="8958" PostTypeId="2" ParentId="8956" CreationDate="2011-03-30T10:41:17.973" Score="5" ViewCount="" Body="&lt;h3&gt;Previous answers on this site:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Related questions have been asked a few times on this site. Check out&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://stats.stackexchange.com/questions/10/under-what-conditions-should-likert-scales-be-used-as-ordinal-or-interval-data&quot;&gt;whether to treat likert scales as ordinal or interval&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://stats.stackexchange.com/questions/8071/how-to-choose-between-pearson-and-spearman-correlation&quot;&gt;choosing between pearson and spearman correlation&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://stats.stackexchange.com/questions/3730/pearsons-or-spearmans-correlation-with-non-normal-data&quot;&gt;Spearman or Pearson with non-normal data&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;Scales versus items:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;From my experience, there is a difference between running analyses on a likert item as opposed to a likert scale. &#xA;A likert scale is the sum of multiple items. &#xA;After summing multiple items, likert scales obtain more possible values, the resulting scale is less lumpy. Such scales often have a sufficient number of points that many researchers are prepared to treat them as continuous. Of course, some would argue that this is a bit cavalier, and much has been written in psychometrics about how best to measure psychological and related constructs.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Standard practice in social sciences:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;From my casual observations from reading journal articles in psychology, the majority of bivariate relationships between multiple-item likert scales are analysed using Pearson's correlation coefficient. Here, I'm thinking about scales like personality, intelligence, attitudes, well-being, and so forth. If you have scales like this, it is worth considering that your results will be compared to previous results where Pearson may have been the dominant choice.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Compare methods:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;It is an interesting exercise to compare Pearson's with Spearman's (and perhaps even Kendall's tau).&#xA;However, you are still left with the decision of which statistic to use, and this ultimately depends on what definition you have of bivariate association.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Heteroscedasticity&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;A correlation coefficient is an accurate summary of the linear relationship between two variables even in the absence of Homoscedasticity (or perhaps we should say bivariate normality given that neither variable is a dependent variable).&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Nonlinearity&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;If there is a non-linear relationship between your two variables, this is interesting. However, both variables could still be treated as  continuous variables, and thus, you could still use Pearson's. For example, age often has an inverted-U relationship with other variables such as income, yet age is still a continuous variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suggest that you produce a scatter plot and fit some smoothed fits (such as a spline or LOESS) to explore any non-linear relationships. If the relationship is truly non-linear then linear correlation is not the best choice for describing such a relationship. You might then want to explore polynomial or nonlinear regression.&lt;/p&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-03-30T13:19:37.973" LastActivityDate="2011-03-30T13:19:37.973" />
  <row Id="8959" PostTypeId="1" CreationDate="2011-03-30T10:43:28.360" Score="1" ViewCount="293" Body="&lt;p&gt;How can I change the labels of the vertical y axis in a boxplot, e.g. from numbers to text?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, I would like to replace {-2, -1, 0, 1, 2} with {0hour, 1hours, 2hours, ...}.&lt;/p&gt;&#xA;" OwnerUserId="3959" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-03-30T12:16:21.123" LastActivityDate="2011-03-30T12:16:21.123" Title="How to customize axis labels in a boxplot?" Tags="&lt;r&gt;&lt;boxplot&gt;" AnswerCount="2" CommentCount="3" />
  <row Id="8960" PostTypeId="1" AcceptedAnswerId="8965" CreationDate="2011-03-30T10:47:17.780" Score="3" ViewCount="219" Body="&lt;p&gt;I have a histogram of wind speed data which is often represented using a weibull distribution. I would like to calculate the weibull shape and scale factors which give the best fit to the histogram.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need a numerical solution (as opposed to &lt;a href=&quot;http://www.weibull.com/LifeDataWeb/probability_plotting.htm&quot; rel=&quot;nofollow&quot;&gt;graphic solutions&lt;/a&gt;) because the goal is to determine the weibull form programmatically.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt;&#xA;Samples are collected every 10 minutes, the wind speed is averaged over the 10 minutes. Samples also include the maximum and minimum wind speed recorded during each interval which are ignored at present but I would like to incorporate later. Bin width is 0.5 m/s&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/C0WCk.png&quot; alt=&quot;Histogram for 1 month of data&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="3579" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-30T15:29:18.080" LastActivityDate="2011-03-30T15:29:18.080" Title="How can I determine weibull parameters from data?" Tags="&lt;distributions&gt;&lt;histogram&gt;&lt;java&gt;" AnswerCount="3" CommentCount="4" FavoriteCount="0" />
  <row Id="8961" PostTypeId="2" ParentId="8960" CreationDate="2011-03-30T11:30:50.177" Score="0" ViewCount="" Body="&lt;p&gt;Couldn't you take a random sample of the data and perform a MLE of the parameters?&lt;/p&gt;&#xA;" OwnerUserId="3805" LastActivityDate="2011-03-30T11:30:50.177" CommentCount="3" />
  <row Id="8962" PostTypeId="2" ParentId="258" CreationDate="2011-03-30T11:35:21.860" Score="1" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://www.gaussianprocess.org/&quot; rel=&quot;nofollow&quot;&gt;Gaussian Process classifier&lt;/a&gt; - it gives probabilistic predictions (which is useful when your operational relative class frequencies differ from those in your training set, or equivalenty your false-positive/false-negative costs are unknown or variable).  It also provides an inidcation of the uncertainty in model predictions due to the uncertainty in &quot;estimating the model&quot; from a finite dataset. The co-variance function is equivalent to the kernel function in an SVM, so it can also operate directly on non-vectorial data (e.g. strings or graphs etc).  The mathematical framework is also neat (but don't use the Laplace approximation).  Automated model selection via maximising marginal likelihood.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Essentially combines good features of logistic regression and SVM.&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2011-03-30T11:35:21.860" CommunityOwnedDate="2011-03-30T11:35:21.860" />
  <row Id="8963" PostTypeId="2" ParentId="8955" CreationDate="2011-03-30T11:43:19.377" Score="5" ViewCount="" Body="&lt;p&gt;One option is to use a simulation.  So set up a model where you specifically specify the heterogeneity suppose as $var(\alpha_i)=\overline{X}_i^2\sigma^2_u$.  Then generate your data from this model, taking random intercepts as a simple example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\alpha_i=\overline{X}_i u_i\;\;\;\;\;\; u_i\sim N(0,\sigma^2_u)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Y_ij=\alpha_{i}+\beta X_{ij} + e_{ij}\;\;\;\;\;\; e_{ij}\sim N(0,\sigma^2_e)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(hope this notation makes sense).  I believe playing around with a set-up such as this will help you answer question 2).  So you would fit this model using a random intercept, when in fact it should be a random slope (which gives you a partial answer to question 3 - random intercepts can account for &quot;fanning&quot; to a degree - this is &quot;level 2 fanning&quot;).  The idea of the above is to try as hard as you can to break your modeling method - try extreme conditions consistent with what you know about the data, and see what happens.  If you are struggling to find these conditions, then don't worry.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I did a quick check on heteroscedasticity for OLS, and it doesn't seem to affect the estimated betas too much.  To me it just seems like heteroscedasticity will in some places by giving an under-estimate of the likely error, and in other places it will give an over-estimate of the likely error (in predictive terms).  See below:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;awaiting plot of data here, user currently frustrated with computers&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And one thing which I always find amusing is this &quot;non-normality of the data&quot; that people worry about.  The data does not need to be normally distributed, but the error term does.  If this were not true, then GLMs would not work - GLMs use a normal approximation to the likelihood function to estimate the parameters, as do GLMMs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I'd say if estimating fixed effect parameters is the main goal then not much to worry about, but you may get better results for prediction by taking heteroscedasticity into account.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-03-30T11:43:19.377" CommentCount="6" />
  <row Id="8964" PostTypeId="2" ParentId="8955" CreationDate="2011-03-30T11:51:54.740" Score="5" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Allometry&quot; rel=&quot;nofollow&quot;&gt;Allometry&lt;/a&gt; would be a good place to start that will be familiar to biologists.  Logaritmic transformations are often used in allometry because the data have a power-law form, but also because the noise process is heteroskedastic (as the variability is proportional to size).  For an example where this has caused a severe problem, see &lt;a href=&quot;http://10.1111/j.1469-7998.2009.00594.x&quot; rel=&quot;nofollow&quot;&gt;&quot;Allometric equations for predicting body mass of dinosaurs&quot;&lt;/a&gt;, where the conclusion that dinosaurs were only half the size previously though was incorrect because an invalid assumtion of homoscedasticity was made (see the correspondance for details).&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2011-03-30T11:51:54.740" CommentCount="1" />
  <row Id="8965" PostTypeId="2" ParentId="8960" CreationDate="2011-03-30T12:08:23.683" Score="6" ViewCount="" Body="&lt;p&gt;Hi Klonq:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Maximum Likelihood Estimation&lt;/em&gt; of Weibull parameters may be a good idea in your case. A form of Weibull distribution looks like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$(\gamma / \theta) (x)^{\gamma-1}\exp(-x^{\gamma}/\theta)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $\theta, \gamma &amp;gt; 0$ are parameters. Given observations $X_1, \ldots, X_n$, the log-likelihood function is &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$L(\theta, \gamma)=\displaystyle \sum_{i=1}^{n}\log f(X_i| \theta, \gamma)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One &quot;programming based&quot; solution would be optimize this function using constrained optimization. Solving for optimum solution:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac {\partial \log L} {\partial \gamma} = \frac{n}{\gamma} + \sum_1^n \log x_i - \frac{1}{\theta}\sum_1^nx_i^{\gamma}\log x_i = 0 $$&#xA;$$\frac {\partial \log L} {\partial \theta} = -\frac{n}{\theta} + \frac{1}{\theta^2}\sum_1^nx_i^{\gamma}=0$$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On eliminating $\theta$ we get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\Bigg[ \frac {\sum_1^n x_i^{\gamma} \log x_i}{\sum_1^n x_i^{\gamma}} - \frac {1}{\gamma}\Bigg]=\frac{1}{n}\sum_1^n \log x_i$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now this can be solved for ML estimate $\hat \gamma$. This can be accomplished with the aid of standard iterative procedures which solve are used to find the solution of equation such as -- Newton-Raphson or other numerical procedures.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now $\theta$ can be found in terms of $\hat \gamma$ as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\hat \theta = \frac {\sum_1^n x_i^{\hat \gamma}}{n}$$&lt;/p&gt;&#xA;" OwnerUserId="1307" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-03-30T13:02:54.993" LastActivityDate="2011-03-30T13:02:54.993" CommentCount="14" />
  <row Id="8966" PostTypeId="2" ParentId="8959" CreationDate="2011-03-30T12:10:39.237" Score="3" ViewCount="" Body="&lt;p&gt;Here's a reproducible example, that you might adapt to fit with what you want to achieve with your data.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;opar &amp;lt;- par(las=1)&#xA;df &amp;lt;- data.frame(y=rnorm(100), x=gl(2, 50, labels=letters[1:2]))&#xA;with(df, plot(y ~ x, axes=FALSE))&#xA;axis(1, at=1:2, labels=levels(df$x))&#xA;axis(2, at=seq(-3, 3, by=1), &#xA;     labels=paste(seq(-3, 3, by=1), &quot;hr&quot;, sep=&quot;&quot;))&#xA;box()&#xA;par(opar)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-03-30T12:10:39.237" />
  <row Id="8967" PostTypeId="2" ParentId="8959" CreationDate="2011-03-30T12:11:34.760" Score="2" ViewCount="" Body="&lt;pre&gt;&lt;code&gt;data(cars)&#xA;with(cars, boxplot(dist ~ speed))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As &lt;code&gt;speed&lt;/code&gt; was numerical the boxplot has numerical values on the horizontal axis.&#xA;Let's create a &lt;code&gt;character&lt;/code&gt; variable:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;hours = paste(cars$speed, &quot;hours&quot;, sep=&quot;&quot;)&#xA;with(cars, boxplot(dist ~ hours))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now the horizontal axis has text labels.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastActivityDate="2011-03-30T12:11:34.760" CommentCount="4" />
  <row Id="8968" PostTypeId="1" CreationDate="2011-03-30T13:32:31.957" Score="1" ViewCount="176" Body="&lt;p&gt;I'm trying to solve the following issue :&#xA;Let's say that I have three normal random variables a,b,c, non correlated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say that I only have two observations of these, M and N where :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;M = a + b&#xA;N = a + c&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And another variable I'm trying to explain :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;R = a + b + c + err&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I can do a simple regression on this data, &lt;code&gt;lm(R ~ M + N + 0)&lt;/code&gt;, I get the same loading for both &lt;code&gt;M&lt;/code&gt; and &lt;code&gt;N&lt;/code&gt;, equal to &lt;code&gt;0.66&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My final model is &lt;code&gt;1.33a + 0.66b + 0.66c&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know that this model is the best linear model here, but what could I use to improve it. Are there methods to properly get &lt;code&gt;a, b, c&lt;/code&gt; from &lt;code&gt;M&lt;/code&gt;and &lt;code&gt;N&lt;/code&gt; and do the regression on them ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've tried PCA but I only get two vectors, not three. Basically, I'd need a method that would project &lt;code&gt;M&lt;/code&gt; and &lt;code&gt;N&lt;/code&gt; into the three interesting components here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I realise I haven't been clear here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$M$, $N$, $R$ are three vectors, with a high number of observations. If they had the relations above with three (unknown) factors $a$, $b$, $c$, then a simple regression would give me :&#xA;$\hat{R} = 0.66 M + 0.66 N = 1.33 a + 0.66 b + 0.66 c$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am wondering if there is an alternative technique for modelling $R$, that would give a closer result to the actual $R$. Something that would somehow reconstruct the common term $a$ from $M$ and $N$ automatically, and take that into account to finally get :&#xA;$\hat{R} = 1 a + 1 * (M - a) + 1 * (N - a)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope I'm a bit more clear in my explanations ...&lt;/p&gt;&#xA;" OwnerUserId="3791" LastEditorUserId="3791" LastEditorDisplayName="" LastEditDate="2011-04-13T10:40:26.083" LastActivityDate="2011-05-13T10:45:54.703" Title="How can I improve a simple regression here?" Tags="&lt;regression&gt;&lt;pca&gt;&lt;factor-analysis&gt;&lt;linear-model&gt;" AnswerCount="0" CommentCount="5" FavoriteCount="1" />
  <row Id="8969" PostTypeId="2" ParentId="8960" CreationDate="2011-03-30T14:09:37.057" Score="2" ViewCount="" Body="&lt;p&gt;Use fitdistrplus:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://stats.stackexchange.com/questions/8662/need-help-identifying-a-distribution-by-its-histogram/8674#8674&quot;&gt;Need help identifying a distribution by its histogram&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's an example of how the Weibull Distribution is fit:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(fitdistrplus)&#xA;&#xA;#Generate fake data&#xA;shape &amp;lt;- 1.9&#xA;x &amp;lt;- rweibull(n=1000, shape=shape, scale=1)&#xA;&#xA;#Fit x data with fitdist&#xA;fit.w &amp;lt;- fitdist(x, &quot;weibull&quot;)&#xA;summary(fit.w)&#xA;plot(fit.w)&#xA;&#xA;&#xA;Fitting of the distribution ' weibull ' by maximum likelihood &#xA;Parameters : &#xA;       estimate Std. Error&#xA;shape 1.8720133 0.04596699&#xA;scale 0.9976703 0.01776794&#xA;Loglikelihood:  -636.1181   AIC:  1276.236   BIC:  1286.052 &#xA;Correlation matrix:&#xA;          shape     scale&#xA;shape 1.0000000 0.3166085&#xA;scale 0.3166085 1.0000000&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/oH6nP.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="2775" LastEditorUserId="2775" LastEditorDisplayName="" LastEditDate="2011-03-30T14:38:19.420" LastActivityDate="2011-03-30T14:38:19.420" CommentCount="2" />
  <row Id="8970" PostTypeId="1" CreationDate="2011-03-30T14:34:22.073" Score="0" ViewCount="44" Body="&lt;p&gt;Anyone know of any reports on number of data requests the US Census Bureau serves or of estimates of the number of census data users?&lt;/p&gt;&#xA;" OwnerUserId="82" LastEditorUserId="82" LastEditorDisplayName="" LastEditDate="2011-03-30T14:42:09.133" LastActivityDate="2011-03-30T14:42:09.133" Title="Access statistics for US Census Data" Tags="&lt;dataset&gt;&lt;census&gt;" />
  <row Id="8971" PostTypeId="1" CreationDate="2011-03-30T15:07:58.333" Score="0" ViewCount="99" Body="&lt;p&gt;I have an assignment were I need to calculate whats deciding the price of a house. One variable is the quality of the neightborhood, ranked from 1 to 4 (nomina. &lt;/p&gt;&#xA;" OwnerUserId="3960" LastActivityDate="2011-03-30T15:07:58.333" ClosedDate="2011-03-30T15:20:54.750" Title="Nominal data and regression analysis" Tags="&lt;regression&gt;" CommentCount="2" />
  <row Id="8972" PostTypeId="1" CreationDate="2011-03-30T15:27:52.997" Score="4" ViewCount="82" Body="&lt;p&gt;Consider an experiment with multiple human participants, each measured multiple times in two conditions. A mixed effects model can be formulated (using &lt;a href=&quot;http://cran.r-project.org/web/packages/lme4/index.html&quot; rel=&quot;nofollow&quot;&gt;lme4&lt;/a&gt; syntax) as:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;fit = lmer(&#xA;    formula = measure ~ (1|participant) + condition&#xA;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, say I want to generate bootstrapped confidence intervals for the predictions of this model. I think I've come up with a simple and computationally efficient method, and I'm sure I'm not the first to think of it, but I'm having trouble finding any prior publications describing this approach. Here it is:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Fit the model (as above), call this the &quot;original model&quot;&lt;/li&gt;&#xA;&lt;li&gt;Obtain predictions from the original model, call these the &quot;original predictions&quot;&lt;/li&gt;&#xA;&lt;li&gt;Obtain residuals from the original model associated with each response from each participant&lt;/li&gt;&#xA;&lt;li&gt;Resample the residuals, sampling &lt;em&gt;participants&lt;/em&gt; with replacement&lt;/li&gt;&#xA;&lt;li&gt;Fit a linear mixed effects model with gaussian error to the &lt;em&gt;residuals&lt;/em&gt;, call this the &quot;interim model&quot;&lt;/li&gt;&#xA;&lt;li&gt;Compute predictions from the interim model for each condition (these predictions will be very close to zero), call these the &quot;interim predictions&quot;&lt;/li&gt;&#xA;&lt;li&gt;Add the interim predictions to the original predictions, call the result the &quot;resample predictions&quot;&lt;/li&gt;&#xA;&lt;li&gt;Repeat steps 4 through 7 many times, generating a distribution of resample predictions for each condition from which once can compute CIs.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I've seen &lt;a href=&quot;http://en.wikipedia.org/wiki/Bootstrapping_%28statistics%29#Resampling_residuals&quot; rel=&quot;nofollow&quot;&gt;&quot;residual bootstrapping&quot;&lt;/a&gt; procedures in the context of simple regression (i.e. not a mixed model) where residuals are sampled as the unit of resampling and then added to the predictions of the original model before fitting a new model on each iteration of the bootstrap, but this seems rather different from the approach I describe where residuals are never resampled, people are, and only &lt;em&gt;after&lt;/em&gt; the interim model is obtained do the original model predictions come into play. This last feature has a really nice side-benefit in that no matter the complexity of the original model, the interim model can always be fit as a gaussian linear mixed model, which can be substantially faster in some cases. For example, I recently had binomial data and 3 predictor variables, one of which I suspected would cause strongly non-linear effects, so I had to employ &lt;a href=&quot;http://cran.r-project.org/web/packages/gamm4/index.html&quot; rel=&quot;nofollow&quot;&gt;Generalized Additive Mixed Modelling&lt;/a&gt; using a binomial link function. Fitting the original model in this case took over an hour, whereas fitting the gaussian LMM on each iteration took mere seconds.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I really don't want to claim priority on this if it's already a known procedure, so I'd be very grateful if anyone can provide information on where this might have been described before. (Also, if there are any glaring problems with this approach, do let me know!)&lt;/p&gt;&#xA;" OwnerUserId="364" LastActivityDate="2011-03-30T15:27:52.997" Title="Is there a name for this type of bootstrapping?" Tags="&lt;mixed-model&gt;&lt;bootstrap&gt;" CommentCount="8" FavoriteCount="2" />
  <row Id="8974" PostTypeId="1" AcceptedAnswerId="8976" CreationDate="2011-03-30T16:24:11.467" Score="6" ViewCount="190" Body="&lt;p&gt;There seems to be in increasing discussion about pie charts. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The main arguments against it seem to be:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Area is perceived with less power than length.&lt;/li&gt;&#xA;&lt;li&gt;Pie charts have very low data-point-to-pixel ratio&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;However, I think they can be somehow useful when portraying proportions. I agree to use a table in most cases but when you are writing a business report and you've just included hundreds of tables why not having a pie chart?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm curious about what the community thinks about this topic. Further references are welcome.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I include a couple of links:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.juiceanalytics.com/writing/the-problem-with-pie-charts/&quot; rel=&quot;nofollow&quot;&gt;http://www.juiceanalytics.com/writing/the-problem-with-pie-charts/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.usf.uni-osnabrueck.de/~breiter/tools/piechart/warning.en.html&quot; rel=&quot;nofollow&quot;&gt;http://www.usf.uni-osnabrueck.de/~breiter/tools/piechart/warning.en.html&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;In order to conclude this question I decided to build an example of pie-chart vs waffle-chart. Thanks all for your answers! :)&#xA;&lt;img src=&quot;http://i.imgur.com/oMPL4.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="2902" LastEditorUserId="2902" LastEditorDisplayName="" LastEditDate="2011-04-01T15:50:49.193" LastActivityDate="2011-04-01T15:50:49.193" Title="Problems with pie charts" Tags="&lt;data-visualization&gt;&lt;pie-chart&gt;" AnswerCount="5" CommentCount="1" FavoriteCount="3" />
  <row Id="8975" PostTypeId="2" ParentId="8974" CreationDate="2011-03-30T16:50:56.217" Score="1" ViewCount="" Body="&lt;p&gt;I think you've answered your own question for the 2nd bullet point. If you want to take up valuable real estate, so be it!  However the first bullet is more important.  With a bar chart the observer needs to estimate relative proportion based upon only 1 axis.  With a pie chart judging along at least 2 axes are involved. And one axis is curved.&#xA;I think that pie charts are used most effectively when you have many categories in the pie, with a legend,  and it is not all that important to judge proportion.&lt;/p&gt;&#xA;" OwnerUserId="3489" LastActivityDate="2011-03-30T16:50:56.217" CommentCount="5" />
  <row Id="8976" PostTypeId="2" ParentId="8974" CreationDate="2011-03-30T18:17:53.490" Score="8" ViewCount="" Body="&lt;p&gt;I wouldn't say there's an increasing interest or debate about the use of pie charts. They are just found everywhere on the web and in so-called &quot;predictive analytic&quot; solutions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I guess you know Tufte's work (he also discussed the use of &lt;a href=&quot;http://www.edwardtufte.com/bboard/q-and-a-fetch-msg?msg_id=00018S&quot; rel=&quot;nofollow&quot;&gt;multiple pie charts&lt;/a&gt;), but more funny is the fact that the second chapter of Wilkinson's &lt;em&gt;Grammar of Graphics&lt;/em&gt; starts with &quot;How to make a pie chart?&quot;.&#xA;You're probably also aware that Cleveland's &lt;a href=&quot;http://www.b-eye-network.com/view/2468&quot; rel=&quot;nofollow&quot;&gt;dotplot&lt;/a&gt;, or even a barchart, will convey much more precise information. The problem seems to really stem from the way our visual system is able to deal with spatial information. It is even quoted in the R software; from the on-line help for &lt;code&gt;pie&lt;/code&gt;, &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Cleveland (1985), page 264: “Data that&#xA;  can be shown by pie charts always can&#xA;  be shown by a dot chart.  This means&#xA;  that judgements of position along a&#xA;  common scale can be made instead of&#xA;  the less accurate angle judgements.”&#xA;  This statement is based on the&#xA;  empirical investigations of Cleveland&#xA;  and McGill as well as investigations&#xA;  by perceptual psychologists.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Cleveland, W. S. (1985) &lt;em&gt;The elements&#xA;  of graphing data&lt;/em&gt;. Wadsworth:&#xA;  Monterey, CA, USA.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;There are variations of pie charts (e.g., donut-like charts) that all raise the same problems: We are not good at evaluating angle and area. Even the ones used in &quot;corrgram&quot;, as described in Friendly, &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.19.1268&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;Corrgrams: Exploratory displays for correlation matrices&lt;/a&gt;, &lt;em&gt;American Statistician&lt;/em&gt; (2002) 56:316, are hard to read, IMHO.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At some point, however, I wondered whether they might still be useful, for example (1) displaying two classes is fine but increasing the number of categories generally worsen the reading (especially with strong imbalance between %), (2) relative judgments are better than absolute ones, that is displaying two pie charts side by side should favor a better appreciation of the results than a simple estimate from, say a pie chart mixing all results (e.g. a two-way cross-classification table). Incidentally, I asked a similar question to Hadley Wickham who kindly pointed me to the following articles:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Spence, I. (2005). &lt;a href=&quot;http://www.psych.utoronto.ca/users/spence/Spence%202005.pdf&quot; rel=&quot;nofollow&quot;&gt;No Humble Pie: The Origins and Usage of a Statistical Chart&lt;/a&gt;. &lt;em&gt;Journal of Educational and Behavioral Statistics&lt;/em&gt;, 30(4), 353–368.&lt;/li&gt;&#xA;&lt;li&gt;Heer, J. and Bostock, M. (2010). &lt;a href=&quot;http://hci.stanford.edu/publications/2010/crowd-perception/heer-chi2010.pdf&quot; rel=&quot;nofollow&quot;&gt;Crowdsourcing Graphical Perception: Using Mechanical Turk to Assess Visualization Design&lt;/a&gt;. &lt;em&gt;CHI 2010&lt;/em&gt;, April 10–15, 2010, Atlanta, Georgia, USA.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In sum, I think they are just good for grossly depicting the distribution of 2 to 3 classes (I use them, from time to time, to show the distribution of males and females in a sample on top of an histogram of ages), but they must be accompanied by relative frequencies or counts for being really informative. A table would still do a better job since you can add margins, and go beyond 2-way classifications.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, there are alternative displays that are built upon the idea of pie chart. I can think of square pie or &lt;a href=&quot;http://eagereyes.org/communication/Engaging-readers-with-square-pie-waffle-charts.html&quot; rel=&quot;nofollow&quot;&gt;waffle chart&lt;/a&gt;, described by Robert Kosara in &lt;a href=&quot;http://eagereyes.org/techniques/pie-charts&quot; rel=&quot;nofollow&quot;&gt;Understanding Pie Charts&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-03-30T18:47:15.337" LastActivityDate="2011-03-30T18:47:15.337" CommentCount="2" />
  <row Id="8977" PostTypeId="2" ParentId="8160" CreationDate="2011-03-30T18:54:57.467" Score="1" ViewCount="" Body="&lt;p&gt;I am sorry, as it may not be straight answer to your question, by if you are using this &quot;total score&quot; as a predictor of something why dont you try regression and evaluate the results with the AUC of ROC ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;or the other way, maybe user Neural networks / Random Forest / Support Vector machines on them to predict given outcome ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regards Luke&lt;/p&gt;&#xA;" OwnerUserId="3870" LastActivityDate="2011-03-30T18:54:57.467" />
  <row Id="8978" PostTypeId="2" ParentId="8974" CreationDate="2011-03-30T19:05:42.137" Score="6" ViewCount="" Body="&lt;p&gt;My personal problem with pie charts is while they may be useful to show difference like this: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/r5mw1.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;way too many people use it to show that:&#xA;&lt;img src=&quot;http://i.imgur.com/dD9sk.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2011-03-30T19:05:42.137" CommunityOwnedDate="2011-03-30T19:05:42.137" />
  <row Id="8979" PostTypeId="2" ParentId="8774" CreationDate="2011-03-30T19:06:57.707" Score="2" ViewCount="" Body="&lt;p&gt;As a follow-up to my comment, if &lt;code&gt;independence.test&lt;/code&gt; refers to &lt;code&gt;coin::independence_test&lt;/code&gt;, then you can reproduce a Cochrane and Armitage trend test, as it is used in GWAS analysis, as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; library(SNPassoc)&#xA;&amp;gt; library(coin)&#xA;&amp;gt; data(SNPs)&#xA;&amp;gt; datSNP &amp;lt;- setupSNP(SNPs,6:40,sep=&quot;&quot;)&#xA;&amp;gt; ( tab &amp;lt;- xtabs(~ casco + snp10001, data=datSNP) )&#xA;     snp10001&#xA;casco T/T C/T C/C&#xA;    0  24  21   2&#xA;    1  68  32  10&#xA;&amp;gt; independence_test(casco~snp10001, data=datSNP, teststat=&quot;quad&quot;,&#xA;                    scores=list(snp10001=c(0,1,2)))&#xA;&#xA;Asymptotic General Independence Test&#xA;&#xA;data:  casco by snp10001 (T/T &amp;lt; C/T &amp;lt; C/C) &#xA;chi-squared = 0.2846, df = 1, p-value = 0.5937&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is a conditional version of the CATT. About scoring of the ordinal variable (here, the frequency of the minor allele denoted by the letter &lt;code&gt;C&lt;/code&gt;), you can play with the &lt;code&gt;scores=&lt;/code&gt; arguments of &lt;code&gt;independence_test()&lt;/code&gt; in order to reflect the model you want to test (the above result is for a log-additive model).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are five different genetic models that are generally considered&#xA;in GWAS, and they reflect how genotypes might be collapsed: codominant&#xA;(T/T (92) C/T (53) C/C (12), yielding the usual $\chi^2(2)$&#xA;association test), dominant (T/T (92) vs. C/T-C/C (65)), recessive&#xA;(T/T-C/T (145) vs. C/C (12)), overdominant (T/T-C/C (104) vs. C/T&#xA;(53)) and log-additive (0 (92) &amp;lt; 1 (53) &amp;lt; 2 (12)). Note that genotype&#xA;recoding is readily available in inheritance functions from the&#xA;&lt;a href=&quot;http://cran.r-project.org/web/packages/SNPassoc/index.html&quot; rel=&quot;nofollow&quot;&gt;SNPassoc&lt;/a&gt;&#xA;package. The &quot;scores&quot; should reflect these collapsing schemes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Following Agresti (&lt;a href=&quot;http://www.stat.ufl.edu/~aa/cda/cda.html&quot; rel=&quot;nofollow&quot;&gt;CDA&lt;/a&gt;, 2002, p. 182), CATT is computed as $n\cdot r^2$,&#xA;where $r$ stands for the linear correlation between the numerical&#xA;scores and the binary outcome (case/control), that is&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;z.catt &amp;lt;- sum(tab)*cor(datSNP$casco, as.numeric(datSNP$snp10001))^2&#xA;1 - pchisq(z.catt, df = 1)  # p=0.5925&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;There also exist various built-in CATT functions in R/Bioconductor&#xA;ecosystem for GWAS, e.g.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;code&gt;CATT()&lt;/code&gt; from&#xA;&lt;a href=&quot;http://cran.r-project.org/web/packages/Rassoc/index.html&quot; rel=&quot;nofollow&quot;&gt;Rassoc&lt;/a&gt;, e.g.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;with(datSNP, CATT(table(casco, snp10001), 0.5)) # p=0.5925 &#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;(additive/multiplicative)&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;in &lt;a href=&quot;http://www.bioconductor.org/packages/2.3/bioc/html/snpMatrix.html&quot; rel=&quot;nofollow&quot;&gt;snpMatrix&lt;/a&gt;, there are headed as 1-df $\chi^2$-test when you call&#xA;&lt;code&gt;single.snp.tests()&lt;/code&gt; (see the&#xA;&lt;a href=&quot;http://www.bioconductor.org/packages/2.6/bioc//vignettes/snpMatrix/inst/doc/snpMatrix-vignette.pdf&quot; rel=&quot;nofollow&quot;&gt;vignette&lt;/a&gt;);&#xA;please note that the default mode of inheritance is the codominant/additive effect.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Finally, here are two references that discuss the choice of scoring scheme&#xA;depending on the genetic model under consideration, and some issues&#xA;with power/robustness&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Zheng, G, Freidlin, B, Li, Z and Gastwirth, JL (2003). Choice of&#xA;scores in trend tests for case-control studies of candidate-gene&#xA;associations. &lt;em&gt;Biometrical Journal&lt;/em&gt;, &lt;strong&gt;45&lt;/strong&gt;: 335-348. &lt;/li&gt;&#xA;&lt;li&gt;Freidlin, B, Zheng, G, Li, Z, and Gastwirth, JL (2002). &lt;a href=&quot;http://wiki.bmi.utah.edu/student/images/f/f5/Trend_Tests_for_Case-Control_Studies_of_Genetic_Markers_%28Power_Sample_Size_and_Robustness%29.pdf&quot; rel=&quot;nofollow&quot;&gt;Trend Tests&#xA;for Case-Control Studies of Genetic Markers: Power, Sample Size and&#xA;Robustness&lt;/a&gt;. &lt;em&gt;Human Heredity&lt;/em&gt;, &lt;strong&gt;53&lt;/strong&gt;: 146-152.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;See also the &lt;a href=&quot;http://www.bioconductor.org/packages/2.3/bioc/html/GeneticsDesign.html&quot; rel=&quot;nofollow&quot;&gt;GeneticsDesign&lt;/a&gt; (bioc) package for power calculation with&#xA;linear trend tests.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-04T16:55:49.133" LastActivityDate="2011-05-04T16:55:49.133" />
  <row Id="8980" PostTypeId="1" CreationDate="2011-03-30T19:09:55.817" Score="0" ViewCount="89" Body="&lt;p&gt;I went to a stats course refresher last week and the instructor talked about data distribution and sampling distribution. I am just practicing with the exercises shown in class.&#xA;Based on my dataset below, &#xA;For the data in Week2, what can I explain about the ‘data distribution’. and the ‘sampling distribution’ of the sample mean?&#xA;I would appreciate any explanation of these two terms. Thanks&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;fishy &amp;lt;- structure(list(week1 = c(2.011, 1.994, 10.332, 7.056, 4.926,&#xA;                                  6.12, 2.039, 5.948, 7.731, 8.41, &#xA;                                  11.055, 7.157, 5.855, 25.243,&#xA;                                  36.553, 76.281, 38.902, 64.689, 42.934, &#xA;                                  80.373, 115.858, 145.981,84.735, &#xA;                                  163.084, 190.472, 295.79, 254.71, &#xA;                                  273.446, 582.495),&#xA;                       week2 = c(535.013, 513.534, 824.442, 1130.764, &#xA;                                 1396.367, 1122.016, 1263.061, 1449.587,   &#xA;                                 1588.527, 680.99, 1861.677, 1432.656, &#xA;                                 2921.025, 3595.931, 2071.98, 1666.726, &#xA;                                 1594.989, 1522.255, 2496.464, 2169.722, &#xA;                                 1870.255, 1039.203, 54.847, 0.266, 60.603,&#xA;                                 601.822, 244.916, 124.749, 74.059)), &#xA;                      .Names = c(&quot;week1&quot;, &quot;week2&quot;), class = &quot;data.frame&quot;, &#xA;                      row.names = c(NA, -29L))&#xA;boxplot(fishy,week1~week2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="3965" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-31T07:37:27.087" LastActivityDate="2011-03-31T07:37:27.087" Title="Difference between sampling distribution and data distribution" Tags="&lt;distributions&gt;&lt;homework&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="8981" PostTypeId="3" CreationDate="2011-03-30T19:23:14.790" Score="0" ViewCount="" Body="&lt;p&gt;&quot;Statistics&quot; can refer variously to the (wide) field of statistical theory and statistical analysis; to constructing functions of data as used in formal procedures; to collections of data; and to summaries of data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because this site is about statistics and statistical analysis, it is rare that tagging a question with &quot;statistics&quot; will be informative.  Use of this tag will signal that your question is extremely general and broad.&lt;/p&gt;&#xA;" OwnerUserId="-1" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-30T19:23:14.790" LastActivityDate="2011-03-30T19:23:14.790" />
  <row Id="8982" PostTypeId="3" CreationDate="2011-03-30T19:23:14.790" Score="0" ViewCount="" Body="This generic tag is only rarely suitable; use it with caution.  Consider selecting more specific, descriptive tags." OwnerUserId="-1" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-30T19:23:14.790" LastActivityDate="2011-03-30T19:23:14.790" />
  <row Id="8983" PostTypeId="2" ParentId="8980" CreationDate="2011-03-30T21:17:39.063" Score="2" ViewCount="" Body="&lt;p&gt;Data distribution is the distribution of the observations in your data (for example: the scores of students taking statistics course). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sampling distribution of the sample mean: Let imagine you sample the data from population n times (randomly, each sample has N observations), for each sample you compute the mean. So you have n means of n samples. Then you have the distribution of the sample mean.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope it's easy to understand.&lt;/p&gt;&#xA;" OwnerUserId="3967" LastActivityDate="2011-03-30T21:17:39.063" CommentCount="1" />
  <row Id="8984" PostTypeId="1" AcceptedAnswerId="8988" CreationDate="2011-03-30T21:43:38.600" Score="7" ViewCount="74" Body="&lt;p&gt;Hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We have a large number of samples whose concentration we measure twice, averaging the two values. Typically, the coefficient of variation (cv) for each sample is &amp;lt; 5%, but for a few samples the cv is high. We assume that in these cases something went wrong with one or both concentration measurements. We can afford one more concentration measurement for the samples with high cv's.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is, how to use the three measurements to achieve the &quot;best&quot; estimate of the true concentration? Average all three measurements? Pick the two with the lowest cv? Or...?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks for any insights or pointers to literature.&lt;/p&gt;&#xA;" OwnerUserId="3968" LastActivityDate="2011-03-31T01:29:33.023" Title="Remeasuring &quot;bad&quot; values" Tags="&lt;measurement-error&gt;" AnswerCount="1" CommentCount="4" FavoriteCount="1" />
  <row Id="8985" PostTypeId="2" ParentId="8909" CreationDate="2011-03-30T23:25:37.247" Score="1" ViewCount="" Body="&lt;p&gt;In my experience, #1 is the better option. If you store the data in any flatfile setup (as you're suggesting) and don't put the rows as your time variable, it becomes that much harder to import into selected programs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, I work primarily in Fortran/C, with secondary applications occasionally done in R or MATLAB. To be compatible with all of these, I use ASCII flatfiles to store most of my data, with fixed-column width, fixed-precision reporting. Any time I have to work with something that isn't set up in this way, it always ends up being a hassle, regardless of how sexy or novel the method for storing the data was. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having empty columns isn't actually a problem, so long as you figure out a method for flagging them properly. Leaving them blank isn't actually the best option, as this can read (i.e. in Fortran) as a 0, which is decidedly different for most applications to an empty value. If you think your client will want to use any sort of programming language-based analysis, you'll want to try to come up with a consistent way to store / flag missing values. For example, if all of your data samples are positive real numbers, then storing a -99.99 is a good way to flag an entry as missing. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Conclusion: figure out what your client is likely to require. If you really don't know, then go with #1, because it is the most general and the easiest to read in for multiple programs and programming languages. Remember to store the dimension information at the top of the file if you're using ASCII flatfiles, or in a defined data block if you're using binary files. &lt;/p&gt;&#xA;" OwnerUserId="781" LastActivityDate="2011-03-30T23:25:37.247" />
  <row Id="8986" PostTypeId="2" ParentId="8732" CreationDate="2011-03-30T23:37:04.383" Score="1" ViewCount="" Body="&lt;p&gt;You may want to examine the GAM package in R, as it can be adapted to do some (or all) of what you are looking for. The original paper (&lt;a href=&quot;http://projecteuclid.org/DPubS?service=UI&amp;amp;version=1.0&amp;amp;verb=Display&amp;amp;handle=euclid.ss/1177013604&quot; rel=&quot;nofollow&quot;&gt;Hastie &amp;amp; Tibshirani, 1986&lt;/a&gt;) is available via OpenAccess if you're up for reading it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Essentially, you model a single dependent variable as being an additive combination of 'smooth' predictors. One of the typical uses is to have time series and lags thereof as your predictors, smooth these inputs, then apply GAM. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This method has been used extensively to estimate daily mortality as a function of smoothed environmental time series, especially pollutants. It's not OpenAccess, but (&lt;a href=&quot;http://www.jstor.org/stable/2680517&quot; rel=&quot;nofollow&quot;&gt;Dominici et al., 2000&lt;/a&gt;) is a superb reference, and (&lt;a href=&quot;http://books.google.ca/books?id=QbDBxSSXIjsC&amp;amp;pg=PA145&amp;amp;lpg=PA145&amp;amp;dq=springer%20books%20dominici%20peng&amp;amp;source=bl&amp;amp;ots=rJ158hQsYw&amp;amp;sig=we0w2B8naxLeV_bTyaCJKGdS7BE&amp;amp;hl=en&amp;amp;ei=cb6TTdSJCOjB0QGB1-DMBw&amp;amp;sa=X&amp;amp;oi=book_result&amp;amp;ct=result&amp;amp;resnum=1&amp;amp;ved=0CBUQ6AEwAA#v=onepage&amp;amp;q&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;Statistical Methods for Environmental Epidemiology with R&lt;/a&gt;) is an excellent book on how to use R to do this type of analysis.&lt;/p&gt;&#xA;" OwnerUserId="781" LastActivityDate="2011-03-30T23:37:04.383" />
  <row Id="8987" PostTypeId="1" AcceptedAnswerId="8991" CreationDate="2011-03-30T23:49:42.127" Score="0" ViewCount="68" Body="&lt;p&gt;Following a previous &lt;a href=&quot;http://stats.stackexchange.com/questions/8899/several-questions-about-conditional-probability&quot;&gt;question&lt;/a&gt;, lets say we now have 3 variables:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$L$, $B$, $S$:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;     S&#xA;   /   \&#xA;  L     B&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So &#xA;$L$ depends on $S$&#xA;$B$ depends on $S$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(S) =$ 0.5&#xA;$P( \lnot S) =$ 0.5&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$L$ that depends on $S$:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(L|S) =$ 0.10&lt;br&gt;&#xA;$P( \lnot L|S) =$ 0.90&lt;br&gt;&#xA;$P(L| \lnot S) =$ 0.01&lt;br&gt;&#xA;$P( \lnot L| \lnot S) =$ 0.99  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$B$ that depends on $S$:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(B|S) =$ 0.60&lt;br&gt;&#xA;$P( \lnot B|S) =$ 0.40&lt;br&gt;&#xA;$P(B| \lnot S) =$ 0.30&lt;br&gt;&#xA;$P( \lnot B| \lnot S) =$ 0.70  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have gotten&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(L) =$ 0.055    $P(\lnot L) =$ 0.945    &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(B) =$ 0.45     $P(\lnot B) =$ 0.55  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As the previos question to get &#xA;$P(S)$ after I observe that $P(L)=1$&lt;br&gt;&#xA;$P(S∣L)=  P(L∣S)P(S)  / P(L) =$ (0.10)(0.5)  / (0.055)  = 0.9091&lt;br&gt;&#xA;so&#xA;$P(S)=$ 0.9091  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Similarly when we have $P(B)=$1&lt;br&gt;&#xA;$P(S∣B)=  P(B∣S)P(S)  / P(B) =$ (0.40)(0.5)  / (0.45)  = 0.3636&lt;br&gt;&#xA;so&#xA;$P(S)=$ 0.3636    &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But What do you do when you observe both events:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(L)=$1  and&lt;br&gt;&#xA;$P(B)=$1 &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do you modify the above formula to get $P(S)$?&lt;/p&gt;&#xA;" OwnerUserId="3681" LastEditorUserId="3681" LastEditorDisplayName="" LastEditDate="2011-03-31T02:11:30.083" LastActivityDate="2011-03-31T09:51:45.473" Title="3 variables and conditional probability" Tags="&lt;conditional-probability&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="8988" PostTypeId="2" ParentId="8984" CreationDate="2011-03-31T00:10:04.933" Score="5" ViewCount="" Body="&lt;p&gt;If the probability of a bad measurement is small than the probability of having two bad measurements out of three will be very small, thus neglecting the outlying one among the three will usually leave you with two valid measurements.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would, however, record all the values measured, even other measurements on the same subject/sample. On the strength of a data collection of valid and bad measurements you could study the distributions of bad and valid measurements. You might also see that bad values depend on the true value (and thus carry information) and both bad and valid values may depend on other measurements of the same subject/sample. When you possess the (conditional) distributions of bad and valid measurements, and the proportion of bad measurements then for each specific measurement you will be able to calculate the probability that it is bad (comes from an other distribution), and to calculate best estimates and established confidence intervals.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I believe that your protocol (keep the two if CV is low, otherwise use the lowest CV pair of three) may be good to start with, but I would revise it after collecting enough data to know more about bad measurements. However, whether a protocol is acceptable also depends on the probability of a bad measurement, how bad a bad measurement is, and how critical is a bad best value in its application.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I assume that you are talking about CV because you analysed some of your existing data and you found CV to be stable. This suggests that measurement error is proportional to the value, thus measurement error SD is constant on the logarithmic scale. If so, taking the geometric mean (not the arithmetic mean) may be more accurate.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-03-31T01:29:33.023" LastActivityDate="2011-03-31T01:29:33.023" CommentCount="1" />
  <row Id="8989" PostTypeId="2" ParentId="8909" CreationDate="2011-03-31T00:26:24.340" Score="2" ViewCount="" Body="&lt;p&gt;Option #2 is much more flexible than #1, particularly if you plan on using Excel pivot tables and/or R packages such as Hadley Wickham's excellent &lt;a href=&quot;http://had.co.nz/reshape/&quot; rel=&quot;nofollow&quot;&gt;reshape&lt;/a&gt; package. I would  store the data so that each row contains measured (event-level and contender-level) variables and any variables necessary to uniquely identify an instance of the measured variables (contender ID, event ID, measurement occasion ID [e.g., half-second increments]) for a single measurement occasion within an event. This allows for the most flexible reshaping of data into any other format desired, a process Wickham describes as melting and casting. You can export the data into an comma-separated value (CSV) spreadsheet, which of course can be read into Excel and most other statistical software.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have long-format data in Excel, aggregating, summarizing, and tabulating data is also easy using &lt;a href=&quot;https://office.microsoft.com/en-us/excel-help/design-the-layout-and-format-of-a-pivottable-report-HP010168032.aspx&quot; rel=&quot;nofollow&quot;&gt;Pivot Tables&lt;/a&gt;. This enables you to create different views of the data that might be of interest to your client, such that as the data are updated you can update these useful views as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;IMO, the most robust solution for very large amounts of structured data is one you didn't mention: store them in a relational database (using, e.g., MS Access or open-source databases such as PostgreSQL) and use Structured Query Language (SQL) to perform the above operations. Here, your data would be broken up into separate tables containing information that is unique to events (e.g., event ID, event type, etc.), contenders (e.g., contender ID, contender name, etc.), unique event-contender combinations (since a single contender might participate in more than one event, and each event certainly has more than one contender), and the measured data in half-second intervals. This avoids storing redundant data and allows you to enforce the integrity of the data that you articulated in your question as data are added, deleted, or updated. There are methods for calling SQL queries from Excel, R, Matlab, and other statistical programs to extract just the information your client wants. A useful introductory text on relational database theory and application is &quot;Inside Relational Databases&quot; by Whitehorn and Marklyn.&lt;/p&gt;&#xA;" OwnerUserId="3964" LastActivityDate="2011-03-31T00:26:24.340" CommentCount="1" />
  <row Id="8991" PostTypeId="2" ParentId="8987" CreationDate="2011-03-31T01:19:43.640" Score="1" ViewCount="" Body="&lt;p&gt;At first we don't know the outcomes of $L$ and $B$:&#xA;$$P_{prior} = \begin{matrix}&#xD;&#xA;P_{prior}(S \land       L \land       B) &amp;amp; P_{prior}(\lnot S \land       L \land       B) \\&#xD;&#xA;P_{prior}(S \land       L \land \lnot B) &amp;amp; P_{prior}(\lnot S \land       L \land \lnot B) \\&#xD;&#xA;P_{prior}(S \land \lnot L \land       B) &amp;amp; P_{prior}(\lnot S \land \lnot L \land       B) \\&#xD;&#xA;P_{prior}(S \land \lnot L \land \lnot B) &amp;amp; P_{prior}(\lnot S \land \lnot L \land \lnot B)&#xD;&#xA;\end{matrix}$$&#xA;$$ = \begin{matrix}&#xD;&#xA;0.5 \cdot 0.1 \cdot 0.6 &amp;amp; 0.5 \cdot 0.01 \cdot 0.3 \\&#xD;&#xA;0.5 \cdot 0.1 \cdot 0.4 &amp;amp; 0.5 \cdot 0.01 \cdot 0.7 \\&#xD;&#xA;0.5 \cdot 0.9 \cdot 0.6 &amp;amp; 0.5 \cdot 0.99 \cdot 0.3 \\&#xD;&#xA;0.5 \cdot 0.9 \cdot 0.4 &amp;amp; 0.5 \cdot 0.99 \cdot 0.7 \\&#xD;&#xA;\end{matrix}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We then observe L and B.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P_{posterior} = \begin{matrix}&#xD;&#xA;0.5 \cdot 0.1 \cdot 0.6 &amp;amp; 0.5 \cdot 0.01 \cdot 0.3 \\&#xD;&#xA;0 &amp;amp; 0 \\&#xD;&#xA;0 &amp;amp; 0 \\&#xD;&#xA;0 &amp;amp; 0 \\&#xD;&#xA;\end{matrix} \cdot \frac{1}{0.5 \cdot 0.1 \cdot 0.6 + 0.5 \cdot 0.01 \cdot 0.3}$$&#xA;$$ = \begin{matrix}&#xD;&#xA;20/21 &amp;amp; 1/21 \\&#xD;&#xA;0 &amp;amp; 0 \\&#xD;&#xA;0 &amp;amp; 0 \\&#xD;&#xA;0 &amp;amp; 0 \\&#xD;&#xA;\end{matrix}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P_{posterior}(S) = \frac{P_{prior}(S \land       L \land       B)}{P_{prior}(S \land       L \land       B) + P_{prior}(\lnot S \land       L \land       B)} = 20/21 \approx 0.95$$&lt;/p&gt;&#xA;" OwnerUserId="3911" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-03-31T09:51:45.473" LastActivityDate="2011-03-31T09:51:45.473" />
  <row Id="8992" PostTypeId="2" ParentId="8974" CreationDate="2011-03-31T01:40:58.007" Score="1" ViewCount="" Body="&lt;p&gt;I can think of almost no case in which a pie chart is better than a bar chart or stacked bar if you want to convey information.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I do have a theory or two on how pie charts got to be so popular.  My first thought is related to PC commercials. Early PCs had text screens (24 x 80 characters), often green like old mainframe CRTs. To show off the new graphics screens that had a Red-Green-Blue pixel basis, a pie chart was ideal. A text screen could do a bar chart after a fashion, but couldn't do a remotely credible pie chart.  Pie charts looked a lot more serious than showing a Mario Brothers screen, regardless of how the PC would actually be used. Thus, it seemed like every PC commercial in the late 1980s and early 1990s showed a pie chart on the monitor.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A second theory is that a bar chart or stacked bar is better if you want to convey information. But what if you don't? Then a pie chart works -- and charts with 3-D effects work even better.&lt;/p&gt;&#xA;" OwnerUserId="3919" LastActivityDate="2011-03-31T01:40:58.007" />
  <row Id="8995" PostTypeId="2" ParentId="97" CreationDate="2011-03-31T04:27:05.490" Score="2" ViewCount="" Body="&lt;p&gt;I'm going to argue from an applied perspective that the mean is often the best choice for summarising the central tendency of a Likert item.&#xA;Specifically, I'm thinking of contexts such as student satisfaction surveys, market research scales, employee opinion surveys, personality test items, and many social science survey items. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In such contexts, consumers of research often want answers to questions like:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Which statements have more or less agreement relative to others?&lt;/li&gt;&#xA;&lt;li&gt;Which groups agreed more or less with a given statement?&lt;/li&gt;&#xA;&lt;li&gt;Over time, has agreement gone up or down?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;For these purposes, the mean has several benefits:&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;1. Mean is easy to calculate:&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;It is easy to see the relationship between the raw data and the mean.&lt;/li&gt;&#xA;&lt;li&gt;It is pragmatically easy to calculate. Thus, the mean can be easily embedded into reporting systems.&lt;/li&gt;&#xA;&lt;li&gt;It also facilitates comparability across contexts, and settings.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;2. Mean is relatively well understood and intuitive:&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The mean is often used to report central tendency of Likert items. Thus, consumers of research are more likely to understand the mean (and thus trust it, and act on it).&lt;/li&gt;&#xA;&lt;li&gt;Some researchers prefer the, arguably, even more intuitive option of reporting the percentage of the sample answering 4 or 5. I.e., it has the relatively intuitive interpretation of &quot;percentage agreement&quot;. &#xA;In essence, this is just an alternative form of the mean, with &lt;code&gt;0, 0, 0, 1, 1&lt;/code&gt; coding.&lt;/li&gt;&#xA;&lt;li&gt;Also, over time, consumers of research build up frames of reference. For example, when you're comparing your teaching performance from year to year, or across subjects,&#xA;you build up a nuanced sense of what a mean of 3.7, 3.9, or 4.1 indicates.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;3. The mean is a single number:&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;A single number is particularly valuable, when you want to make claims like &quot;students were more satisfied with Subject X than Subject Y.&quot;&lt;/li&gt;&#xA;&lt;li&gt;I also find, empirically, that a single number is actually the main information of interest in a Likert item.&#xA;The standard deviation tends to be related to the extent to which the mean is close to the central score (e.g., 3.0). Of course, empirically, this may not apply in your context. For example, I read somewhere that when You Tube ratings had the star system, there were a large number of either the lowest or the highest rating. For this reason, it is important to inspect category frequencies.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;4. It doesn't make much difference&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Although I have not formally tested it, I would hypothesise that for the purpose of comparing central tendency ratings across items, or groups of participants, or over time, any reasonable choice of scaling for generating the mean would yield similar conclusions.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastActivityDate="2011-03-31T04:27:05.490" />
  <row Id="8996" PostTypeId="1" CreationDate="2011-03-31T05:02:07.447" Score="0" ViewCount="52" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;Possible Duplicate:&lt;/strong&gt;&lt;br&gt;&#xA;  &lt;a href=&quot;http://stats.stackexchange.com/questions/6498/seeking-certain-type-of-arima-explanation&quot;&gt;Seeking certain type of ARIMA explanation&lt;/a&gt;  &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&#xA;&#xA;&lt;p&gt;I am tying to apply ARIMA - Time series modeling methodology on a test data(first time). Please provide some good resources, books or dumbed down explanation here for ARIMA models(for a novice). I have goggled the topic and found many concise explanations that are not of much help. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you...&lt;/p&gt;&#xA;" OwnerUserId="3972" LastActivityDate="2011-03-31T05:02:07.447" ClosedDate="2011-03-31T07:29:53.550" Title="ARIMA Model help..in need of resources" Tags="&lt;time-series&gt;&lt;arima&gt;" CommentCount="1" />
  <row Id="8997" PostTypeId="1" AcceptedAnswerId="8998" CreationDate="2011-03-31T05:23:01.543" Score="3" ViewCount="130" Body="&lt;p&gt;This is in reference to the &lt;a href=&quot;http://en.wikipedia.org/wiki/Girsanov_theorem&quot; rel=&quot;nofollow&quot;&gt;Girsanov theorem&lt;/a&gt; however question is general. If $X$ is a standard normal variable $N(0,1)$, why is expectation of $e^{-\mu X - \mu^2/2}$ equal to 1?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Shouldn't it be $e^{-\mu^2/2}$?&lt;/p&gt;&#xA;" OwnerUserId="862" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-03-31T08:31:28.927" LastActivityDate="2011-03-31T08:31:28.927" Title="Expected value of certain exponential transformation of standard normal variable " Tags="&lt;probability&gt;&lt;expected-value&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="8998" PostTypeId="2" ParentId="8997" CreationDate="2011-03-31T05:38:17.697" Score="8" ViewCount="" Body="&lt;p&gt;If we let $Y = -\mu X$, then $Y$ is distributed as $N(0, \mu^2)$, and $e^Y$ is a lognormal random variable with parameters $0, \mu^2$. The expected value of a lognormal with parameters $a, b^2$ is $e^{a + b^2/2}$, so $E(e^{-\mu X}) = E(e^Y) = e^{\mu^2/2}$. The given result follows immediately.&lt;/p&gt;&#xA;" OwnerUserId="1569" LastEditorUserId="1569" LastEditorDisplayName="" LastEditDate="2011-03-31T05:43:18.937" LastActivityDate="2011-03-31T05:43:18.937" CommentCount="3" />
  <row Id="8999" PostTypeId="2" ParentId="8997" CreationDate="2011-03-31T06:06:51.270" Score="0" ViewCount="" Body="&lt;p&gt;The answer is right, but to make it even more fairly obvious. Imaginae Y = -muX -0.5mu^2, then it follows Y~N(-0.5mu^2,mu^2) as above, and thus expectation of lognormal is e^(-0.5mu^2+0.5mu^2) thus = e^0 = 1&lt;/p&gt;&#xA;" OwnerUserId="3973" LastActivityDate="2011-03-31T06:06:51.270" />
  <row Id="9000" PostTypeId="2" ParentId="8036" CreationDate="2011-03-31T09:48:42.233" Score="1" ViewCount="" Body="&lt;p&gt;As a first step you could take a sample from the documents. This can be random sampling, but if you know that certain characteristics of the documents are particularly relevant you could use stratified sampling.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second step can be feature extraction. Define characteristics of the documents that may help predicting the &lt;code&gt;accept&lt;/code&gt; or &lt;code&gt;reject&lt;/code&gt; labels. Give clear definitions; you can use numerical scale, ordinal and nominal (including binary) variables. Determine features and labels in the sample.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Third step, developing a prediction model. To predict the binary response there are many kinds of methods available. If you have some experience how some features affect acceptance you may want to build it into a model and use regression techniques e.g. logistic regression, probit regression, and model selection (e.g. stepwise variable selection). If you can't or don't want to build in pre-existing knowledge into the automatic assessment you can use machine learning techniques like random forests, logistic regression tree, support vector machines.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Fourth step is validating your model on a second sample. The way to do this can be randomly partitioning your original sample into a modelling and a testing subset before modelling.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you know how well your automatic classification system performs you will be able to judge which documents need human review.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-03-31T12:47:51.767" LastActivityDate="2011-03-31T12:47:51.767" />
  <row Id="9001" PostTypeId="1" AcceptedAnswerId="9007" CreationDate="2011-03-31T10:14:39.883" Score="7" ViewCount="156" Body="&lt;p&gt;Are there well known formulas for the order statistics of certain random distributions? Particularly the first and last order statistics of a normal &#xA;random variable, but a more general answer would also be appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt; To clarify, I am looking for approximating formulas that can be more-or-less explicitly evaluated, not the exact integral expression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, I have seen the following two approximations for the first order statistic (ie the minimum) of a normal rv:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$e_{1:n} \geq \mu - \frac{n-1}{\sqrt{2n-1}}\sigma$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$e_{1:n} \approx \mu + \Phi^{-1} \left( \frac{1}{n+1} \right)\sigma$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first of these, for $n=200$, gives approximately $e_{1:200} \geq \mu - 10\sigma$ which seems like a wildly loose bound.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second gives $e_{1:200} \approx \mu - 2.58\sigma$ whereas a quick Monte Carlo  gives $e_{1:200} \approx \mu - 2.75\sigma$, so it's not a bad approximation but not great either, and more importantly I don't have any intuition about where it comes from.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help?&lt;/p&gt;&#xA;" OwnerUserId="2425" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-29T00:59:10.810" LastActivityDate="2011-04-29T00:59:10.810" Title="Approximate order statistics for normal random variables" Tags="&lt;normal-distribution&gt;&lt;approximation&gt;&lt;order-statistics&gt;" AnswerCount="3" CommentCount="2" FavoriteCount="2" />
  <row Id="9003" PostTypeId="1" CreationDate="2011-03-31T10:31:29.800" Score="3" ViewCount="87" Body="&lt;p&gt;I have about 10 variables about products. I need to rank order the products by quality. Now, the quality is very subjective thing, so I got all 300 products ranked by domain experts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I have 10 variables, and ranking assigned by an expert. About 1% of products should be excelent, about 10% should be very good, 30% should be good, 30% should be average and the rest is below average.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What sort of modeling technique would you use to make sue of this data and create system which will automatically be able to classigy product into those categories and mimic &quot;expert judgement&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="333" LastActivityDate="2011-03-31T12:39:27.733" Title="Rank ordering and/or classification problem" Tags="&lt;classification&gt;&lt;ranking&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="3" />
  <row Id="9004" PostTypeId="2" ParentId="9003" CreationDate="2011-03-31T10:46:33.353" Score="2" ViewCount="" Body="&lt;p&gt;The dependent variable is ordinal (excellent &gt; very good &gt; good &gt; average &gt; below average). You could try ordinal logistic regression, with some variable selection method. After choosing the model you could tweak the thresholds to influence proportions falling into various categories.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastActivityDate="2011-03-31T10:46:33.353" CommentCount="2" />
  <row Id="9005" PostTypeId="2" ParentId="8974" CreationDate="2011-03-31T10:50:11.990" Score="5" ViewCount="" Body="&lt;p&gt;Pie charts, like pie, may be delicious but they are not nutritious.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In addition to points made already, one is that rotating a pie chart changes perception of the size of the angles, as does changing the color.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If a pie chart has only a few categories, make a table.  If it has a LOT of categories, then the slices will be too thin to see (much less to label accurately).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wrote about this on &lt;a href=&quot;http://www.statisticalanalysisconsulting.com/graphics-for-univariate-data-pie-is-delicious-but-not-nutritious/&quot; rel=&quot;nofollow&quot;&gt;my blog&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="686" LastActivityDate="2011-03-31T10:50:11.990" CommentCount="2" />
  <row Id="9006" PostTypeId="2" ParentId="9003" CreationDate="2011-03-31T12:39:27.733" Score="0" ViewCount="" Body="&lt;p&gt;An other approach would be taking the 1..300 ranks assigned to the products by the experts (temporarily neglecting the excellent/very good/... categorisation). You could try &lt;a href=&quot;http://cran.r-project.org/web/packages/acepack/acepack.pdf&quot; rel=&quot;nofollow&quot;&gt;ACE&lt;/a&gt; (linear model with transformations of dependent variable and predictors), you may find that simple transformations of the predictors are satisfactory. Then you can set thresholds for the classification.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastActivityDate="2011-03-31T12:39:27.733" CommentCount="1" />
  <row Id="9007" PostTypeId="2" ParentId="9001" CreationDate="2011-03-31T12:52:27.970" Score="6" ViewCount="" Body="&lt;p&gt;The classic reference is Royston (1982)[1] which has algorithms going beyond explicit formulas. It also quotes a well-known formula by Blom (1958):&#xA;$E(r:n) \approx \mu + \Phi^{-1}(\frac{r-\alpha}{n-2\alpha+1})\sigma$ with $\alpha=0.375$. This formula gives a multiplier of -2.73 for $n=200, r=1$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[1]: &lt;a href=&quot;http://www.jstor.org/stable/2347982&quot; rel=&quot;nofollow&quot;&gt;Algorithm AS 177: Expected Normal Order Statistics (Exact and Approximate)&lt;/a&gt; J. P. Royston. Journal of the Royal Statistical Society. Series C (Applied Statistics) Vol. 31, No. 2 (1982), pp. 161-165&lt;/p&gt;&#xA;" OwnerUserId="279" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-03-31T17:54:50.003" LastActivityDate="2011-03-31T17:54:50.003" CommentCount="1" />
  <row Id="9008" PostTypeId="2" ParentId="9001" CreationDate="2011-03-31T13:01:19.303" Score="1" ViewCount="" Body="&lt;p&gt;Depending on what you want to do, this answer may or may not help - I got the following exact formula from &lt;a href=&quot;http://www.maplesoft.com/support/help/Maple/view.aspx?path=Statistics&quot; rel=&quot;nofollow&quot;&gt;Maple's Statistics package&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;with(Statistics):&#xA;X := OrderStatistic(Normal(0, 1), 1, n):&#xA;m := Mean(X):&#xA;m;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;$$\int _{-\infty }^{\infty }\!1/2\,{\frac {{\it \_t0}\,n!\,\sqrt {2}{&#xD;&#xA;{\rm e}^{-1/2\,{{\it \_t0}}^{2}}} \left( 1/2-1/2\,&#xD;&#xA;{{\rm erf}\left(1/2\,{\it \_t0}\,\sqrt {2}\right)} \right) ^{-1+n}}{&#xD;&#xA; \left( -1+n \right) !\,\sqrt {\pi }}}{d{\it \_t0}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By itself this isn't very useful (and it could probably be derived fairly easily by hand, since it's the minimum of $n$ random variables), but it does allow for quick and very accurate approximation for given values of $n $ - much more accurate than Monte Carlo:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;evalf(eval(m, n = 200));&#xA;evalf[25](eval(m, n = 200));&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;gives -2.746042447 and -2.746042447451154492412344, respectively.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Full disclosure - I maintain this package.)&lt;/p&gt;&#xA;" OwnerUserId="2898" LastActivityDate="2011-03-31T13:01:19.303" CommentCount="1" />
  <row Id="9009" PostTypeId="1" CreationDate="2011-03-31T13:53:50.733" Score="5" ViewCount="93" Body="&lt;p&gt;I'm using the &lt;a href=&quot;http://cran.r-project.org/web/packages/quantreg/index.html&quot; rel=&quot;nofollow&quot;&gt;quantreg&lt;/a&gt; package to make a regression model using the 99th percentile of my values in a data set. Based on advice from a previous stackoverflow &lt;a href=&quot;http://stackoverflow.com/questions/4594370/advice-on-calculating-a-function-to-describe-upper-bound-of-data&quot;&gt;question&lt;/a&gt; I asked, I used the following code structure.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mod &amp;lt;- rq(y ~ log(x), data=df, tau=.99)    &#xA;pDF &amp;lt;- data.frame(x = seq(1,10000, length=1000) ) &#xA;pDF &amp;lt;- within(pDF, y &amp;lt;- predict(mod, newdata = pDF) )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which I show plotted on top of my data. I've plotted this using ggplot2, with an alpha value for the points. I think that the tail of my distribution is not being considered sufficiently in my analysis. Perhaps this is due to the fact that there are individual points, that are being ignored by the percentile type measurement.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of the comments suggested that&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The package vignette includes sections on nonlinear quantile regression and also models with smoothing splines etc. &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Based on my previous question I assumed a logarithmic relationship, but I'm not sure if that is correct. I thought I could extract all the points at the 99th percentile interval and then examine them separately, but I'm not sure how to do that, or if that is a good approach. I would appreciate any advice on how to improve identifying this relationship.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/Jwtlh.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="2635" LastEditorUserId="2635" LastEditorDisplayName="" LastEditDate="2011-03-31T13:59:05.663" LastActivityDate="2011-04-04T02:49:33.990" Title="Advice on identifying curve shape using quantreg" Tags="&lt;regression&gt;&lt;logarithm&gt;" CommentCount="1" FavoriteCount="3" />
  <row Id="9010" PostTypeId="2" ParentId="9001" CreationDate="2011-03-31T14:22:39.473" Score="4" ViewCount="" Body="&lt;p&gt;The distribution of the ith order statistic of any continuous random variable with a PDF is given by the &quot;beta-F&quot; compound distribution.  The intuitive way to think about this distribution, is to consider the ith order statistic in a sample of $N$.  Now in order for the value of the ith order statistic of a random variable $X$ to be equal to $x$ we need 3 conditions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;$i-1$ values below $x$, this has probability $F_{X}(x)$ for each observation, where $F_X(x)=Pr(X&amp;lt;x)$ is the CDF of the random variable X.&lt;/li&gt;&#xA;&lt;li&gt;$N-i$ values above $x$, this has probability $1-F_{X}(x)$&lt;/li&gt;&#xA;&lt;li&gt;1 value inside a infinitesimal interval containing $x$, this has probability $f_{X}(x)dx$ where $f_{X}(x)dx=dF_{X}(x)=Pr(x&amp;lt;X&amp;lt;x+dx)$ is the PDF of the random variable $X$&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;There are ${N \choose 1}{N-1 \choose i-1}$ ways to make this choice, so we have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$f_{i}(x_{i})=\frac{N!}{(i-1)!(N-i)!}f_{X}(x_{i})\left[1-F_{X}(x_{i})\right]^{N-i}\left[F_{X}(x_{i})\right]^{i-1}dx$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt; &lt;em&gt;in my original post, I made a very poor attempt at going further from this point, and the comments below reflect this.  I have sought to rectify this below&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we take the mean value of this pdf we get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$E(X_{i})=\int_{-\infty}^{\infty} x_{i}f_{i}(x_{i})dx_{i}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And in this integral, we make to change of variables to $p_{i}=F_{X}(x_{i})$ (taking @henry's hint), and the integral becomes:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$E(X_{i})=\int_{0}^{1} F_{X}^{-1}(p_{i})Beta(p_{i}|i,N-i+1)dp_{i}=E_{Beta(p_{i}|i,N-i+1)}\left[F_{X}^{-1}(p_{i})\right]$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So this is the expected value of the inverse CDF, which can be well approximated using the delta method to give:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$E_{Beta(p_{i}|i,N-i+1)}\left[F_{X}^{-1}(p_{i})\right]\approx F_{X}^{-1}\left[E_{Beta(p_{i}|i,N-i+1)}\right]=F_{X}^{-1}\left[\frac{i}{N+1}\right]$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To make a better approximation, we can expand to 2nd order (prime denoting differentiation), and noting that the second derivative of an inverse is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{\partial^{2}}{\partial a^{2}}F_{X}^{-1}(a)=-\frac{F_{X}^{&amp;#39;&amp;#39;}(F_{X}^{-1}(a))}{\left[F_{X}^{&amp;#39;}(F_{X}^{-1}(a))\right]^{3}}=-\frac{f_{X}^{&amp;#39;}(F_{X}^{-1}(a))}{\left[f_{X}(F_{X}^{-1}(a))\right]^{3}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let $\nu_{i}=F_{X}^{-1}\left[\frac{i}{N+1}\right]$.  Then We have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$E_{Beta(p_{i}|i,N-i+1)}\left[F_{X}^{-1}(p_{i})\right]\approx F_{X}^{-1}\left[\nu_{i}\right]-\frac{Var_{Beta(p_{i}|i,N-i+1)}\left[p_{i}\right]}{2}\frac{f_{X}^{&amp;#39;}(\nu_{i})}{\left[f_{X}(\nu_{i})\right]^{3}}$$&#xA;$$=\nu_{i}-\frac{\left(\frac{i}{N+1}\right)\left(1-\frac{i}{N+1}\right)}{2(N+2)}\frac{f_{X}^{&amp;#39;}(\nu_{i})}{\left[f_{X}(\nu_{i})\right]^{3}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, specialising to normal case we have&#xA;$$f_{X}(x)=\frac{1}{\sigma}\phi(\frac{x-\mu}{\sigma})\rightarrow f_{X}^{&amp;#39;}(x)=-\frac{x-\mu}{\sigma^{3}}\phi(\frac{x-\mu}{\sigma})=-\frac{x-\mu}{\sigma^{2}}f_{X}(x)$$&#xA;$$F_{X}(x)=\Phi(\frac{x-\mu}{\sigma})\implies F_{X}^{-1}(x)=\mu+\sigma\Phi^{-1}(x)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that $f_{X}(\nu_{i})=\frac{1}{\sigma}\phi\left[\Phi^{-1}\left(\frac{i}{N+1}\right)\right]$ And the expectation approximately becomes:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$E[x_{i}]\approx \mu+\sigma\Phi^{-1}\left(\frac{i}{N+1}\right)+\frac{\left(\frac{i}{N+1}\right)\left(1-\frac{i}{N+1}\right)}{2(N+2)}\frac{\sigma\Phi^{-1}\left(\frac{i}{N+1}\right)}{\left[\phi\left[\Phi^{-1}\left(\frac{i}{N+1}\right)\right]\right]^{2}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And finally:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$E[x_{i}]\approx \mu+\sigma\Phi^{-1}\left(\frac{i}{N+1}\right)\left[1+\frac{\left(\frac{i}{N+1}\right)\left(1-\frac{i}{N+1}\right)}{2(N+2)\left[\phi\left[\Phi^{-1}\left(\frac{i}{N+1}\right)\right]\right]^{2}}\right]$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Although as @whuber has noted, this will not be accurate in the tails.  In fact I think it may be worse, because of the skewness of a beta with different parameters&lt;/p&gt;&#xA;" OwnerUserId="2392" LastEditorUserId="2392" LastEditorDisplayName="" LastEditDate="2011-04-01T23:02:51.050" LastActivityDate="2011-04-01T23:02:51.050" CommentCount="11" />
  <row Id="9012" PostTypeId="2" ParentId="6791" CreationDate="2011-03-31T17:12:31.247" Score="0" ViewCount="" Body="&lt;p&gt;Consider transforming the responses from both data sets into z-scores. There is going to be an ad hoc quality to any sort of rescaling but at least this way you avoid mechanically treating any particular set of intervals on one item as equivalent to any particular set on the other. I'd definitely go this route if I were using the items as predictors or outcome variables in any sort of analysis of variance. If you were doing anything w/ composite scales -- ones that aggregate likert measures  -- you'd likely do essentially what I've prpposed: either you'd convert the item responses to z-scores before summing or taking their mean to form the composite scale; or you'd form a scale with factor analysis or another technique that uses the covariance matrix of the items to determine the affinity of the responses to them.&lt;/p&gt;&#xA;" OwnerUserId="2347" LastActivityDate="2011-03-31T17:12:31.247" CommentCount="3" />
  <row Id="9013" PostTypeId="1" AcceptedAnswerId="9015" CreationDate="2011-03-31T17:37:12.297" Score="3" ViewCount="112" Body="&lt;p&gt;Let's say I have a file A1.txt and a file A2.txt. I have written the statements&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;filename in1 'A1.txt';&#xA;filename in2 'A2.txt';&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, I want to re-do this using B1 and B2 (and eventually C1, C2, D1, D2), and just rename the variable in one place. So, I want a statement like&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;%let prefix = 'B';&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and then I want to put that prefix in the filename statements. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;filename in1 '&amp;amp;prefix1.txt';&#xA;filename in2 '&amp;amp;prefix2.txt';&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm not doing this in a data statement. I can almost get it. filename in1 &amp;amp;prefix'.txt' tries to read B'.txt but that extra quote is pesky. There's something vague in my memory banks about using an _ but I can't pull it out. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks. &lt;/p&gt;&#xA;" OwnerUserId="62" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-03-31T17:43:47.707" LastActivityDate="2011-03-31T18:12:50.597" Title="String replace in SAS?" Tags="&lt;sas&gt;" AnswerCount="1" CommentCount="4" FavoriteCount="1" />
  <row Id="9014" PostTypeId="1" CreationDate="2011-03-31T18:00:00.810" Score="4" ViewCount="106" Body="&lt;p&gt;In &lt;a href=&quot;http://en.wikipedia.org/wiki/Girsanov_theorem&quot; rel=&quot;nofollow&quot;&gt;Girsanov&lt;/a&gt; theorem, the change of probability measure variable $Z_t = \frac{dQ}{dP}|_{\mathcal{F}_t}$, why does it need to be a martingale with respect to measure $P$ for the change of measure $\frac{dQ}{dP}$ to exist?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am having trouble understanding this. Anyone familiar with this?&lt;/p&gt;&#xA;" OwnerUserId="862" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-04-01T08:19:03.897" LastActivityDate="2011-04-12T11:45:39.793" Title="Why local martingale property is important in Girsanov theorem?" Tags="&lt;probability&gt;&lt;stochastic-processes&gt;" AnswerCount="1" />
  <row Id="9015" PostTypeId="2" ParentId="9013" CreationDate="2011-03-31T18:12:50.597" Score="4" ViewCount="" Body="&lt;p&gt;You are looking for a period:  &lt;code&gt;&amp;amp;prefix.1.txt&lt;/code&gt;, where the period after &lt;code&gt;&amp;amp;prefix&lt;/code&gt; tells SAS that the name of the macro variable is finished. If you don't have the &quot;1&quot;, then you need two periods: &lt;code&gt;&amp;amp;prefix..txt&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="279" LastActivityDate="2011-03-31T18:12:50.597" CommentCount="1" />
  <row Id="9016" PostTypeId="1" CreationDate="2011-03-31T18:29:46.377" Score="4" ViewCount="172" Body="&lt;p&gt;Is there are way to determine the optimal cluster number ? or Just i should try different values and check the error rate to decide it? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="3270" LastActivityDate="2011-03-31T19:09:04.953" Title="How to define number of clusters in K-means clustering?" Tags="&lt;clustering&gt;&lt;unsupervised-learning&gt;" AnswerCount="1" CommentCount="6" FavoriteCount="3" />
  <row Id="9017" PostTypeId="2" ParentId="6498" CreationDate="2011-03-31T19:03:44.280" Score="5" ViewCount="" Body="&lt;p&gt;I will try and respond to the gentle urging of whuber to simply “respond to the question” and stay on topic. We are given 144 monthly readings of a series called “The Airline Series” . Box and Jenkins were widely criticized for providing a forecast that was wildly on the high side due to the “explosive nature” of a reverse logged transformation. &lt;img src=&quot;http://i.imgur.com/S8hrL.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Visually we get the impression that the variance of the original series increases with the level of the series suggesting a need for a transformation.  However we know that one the requirements for a useful  model is that the variance of the “model errors” needs to be homogenous. No assumptions are necessary about  the variance of the original series. They are identical if the model is simply a constant i.e. y(t)=u . As &lt;a href=&quot;http://stats.stackexchange.com/users/2392/probabilityislogic&quot;&gt;http://stats.stackexchange.com/users/2392/probabilityislogic&lt;/a&gt; stated so clearly in his response to   &lt;a href=&quot;http://stats.stackexchange.com/questions/8955/advice-on-explaining-heterogeneity-heteroscedasticty&quot;&gt;Advice on explaining heterogeneity/ heteroscedasticty&lt;/a&gt;  “one thing which I always find amusing is this &quot;non-normality of the data&quot; that people worry about. The data does not need to be normally distributed, but the error term does”&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Early work in time series often erroneously jumped to conclusions about unwarranted transformations. We will discover here that the remedial transformation for this data is to simply add three indicator dummy series to the ARIMA model reflecting an adjustment for three unusual data points. &#xA;Following is the plot of the autocorrelation function suggesting a strong  autocorrelation at lag 12 (.76) and at lag 1 (.948). Autocorrelations are simply regression coefficients in a model where y is the dependent variable being predicted by a lag of y.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/IrN2D.jpg&quot; alt=&quot;enter image description here&quot;&gt;! &lt;img src=&quot;http://i.imgur.com/sz3PB.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The analysis above suggests that one model the first differences of the series and study that “residual series”  which is identical to the first differences  first for it’s properties.&#xA;&lt;img src=&quot;http://i.imgur.com/4L07U.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This analysis reconfirms the idea that a strong seasonal pattern exists in the data that could be remedied or modeled by a model that contained two differencing operators .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/TFse2.jpg&quot; alt=&quot;enter image description here&quot;&gt;&#xA;&lt;img src=&quot;http://i.imgur.com/E1s8T.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This simple double differencing  yields a set of residual a.k.a an adjusted series  or loosely speaking a transformed series that evidences non-constant variance but the reason for the non-constant variance is the non-constant mean of the residuals.Here is a plot of the doubly differenced series , suggesting three anomalies at the end of the series. The Autocorrelation of this series falsely indicates that “all is well” and there might be a need for any Ma(1) adjustment.  Care should be taken as there is a suggestion of anomalies in the data thus the acf is biased downwards. This is known as the “Alice in Wonderland Effect”  i.e. accepting the null hypothesis of no evidented structure when that structure is being masked by a violation of one of the assumptions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/DFIro.jpg&quot; alt=&quot;enter image description here&quot;&gt;&#xA;&lt;img src=&quot;http://i.imgur.com/30Tbp.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We visually detect three unusual points ( 117,135,136)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/kGZXe.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This step of detecting the outliers is called Intervention Detection and can be easily , or not so easily, programmed following the following the work of Tsay.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/0ON8N.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;img src=&quot;http://i.imgur.com/6EQck.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we add three indicators to the model, we get &#xA;&lt;img src=&quot;http://i.imgur.com/qKwHK.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can then estimate &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/keOHB.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And receive a plot of the residuals and the acf &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/lEdHy.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/vaCmR.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This acf suggests that we add potentially two moving average coefficients to the model . Thus the next estimated model might be.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/5mEC7.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Yielding&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/A5tDw.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/YCBes.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/GB70k.jpg&quot; alt=&quot;enter image description here&quot;&gt;&#xA; &lt;img src=&quot;http://i.imgur.com/7RT3E.jpg&quot; alt=&quot;enter image description here&quot;&gt;&#xA;&lt;img src=&quot;http://i.imgur.com/Qt7X3.jpg&quot; alt=&quot;enter image description here&quot;&gt;&#xA;One could then delete the non-significant constant and get a refined model :&#xA;&lt;img src=&quot;http://i.imgur.com/u0hrp.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We note that no power transformations were needed whatsoever to obtain a set of residuals that constant variance.  Note that the forecasts are non-explosive.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/xvO67.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/0tKpt.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In terms of a simple weighted sum , we have: 13 weights ; 3 non-zero and equal to (1.0.1,0.,-1.0)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/rMynA.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/LFLMO.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This material was presented in a way that was non-automatic and consequentially required  user interaction in terms of making modeling decisions.  &lt;/p&gt;&#xA;" OwnerUserId="3382" LastEditorUserId="3382" LastEditorDisplayName="" LastEditDate="2011-03-31T22:03:58.057" LastActivityDate="2011-03-31T22:03:58.057" />
  <row Id="9019" PostTypeId="2" ParentId="9016" CreationDate="2011-03-31T19:09:04.953" Score="2" ViewCount="" Body="&lt;p&gt;The method I use is to use CCC (Cubic Clustering Criteria).  I look for CCC to increase to a maximum as I increment the number of clusters by 1, and then observe when the CCC starts to decrease.  At that point I take the number of clusters at the (local) maximum. This would be similar to using a scree plot to picking the number of principal components.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Cubic Clustering Criterion CCC&#xA; SAS Technical Report #108&lt;/p&gt;&#xA;&#xA;&lt;p&gt;n = number of observations&#xA;  nk= number in cluster k&#xA;  p = number of variables&#xA;  q = number of clusters&#xA;  X = nxp data matrix&#xA;  M = qxp matrix of cluster means&#xA;  Z = cluster indicator (zik=1 if obs. i in cluster k)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assume each variable has mean 0.&#xA;Z’Z = diag(n1, ..., nq),  M = (Z’Z)-1Z’X&lt;/p&gt;&#xA;&#xA;&lt;p&gt;SS(total) matrix = T= X’X&#xA;SS(between clusters) matrix = B = M’ Z’Z M&#xA;SS(within clusters) matrix  = W = T-B &lt;/p&gt;&#xA;&#xA;&lt;p&gt;R2 = 1 – trace(W)/trace(T) &#xA;(trace = sum of diagonal elements) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Stack columns of X into one long column. &#xA;Regress on Kronecker product of Z with pxp identity matrix&#xA;Compute R2 for this regression – same R2&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The CCC idea is to compare the R2 you get for a given set of clusters with the R2 you would get by clustering a unfoirmly distributed set of points in p dimensional space. &lt;/p&gt;&#xA;" OwnerUserId="3489" LastActivityDate="2011-03-31T19:09:04.953" />
  <row Id="9020" PostTypeId="1" AcceptedAnswerId="9042" CreationDate="2011-03-31T21:05:49.463" Score="2" ViewCount="91" Body="&lt;p&gt;In SPSS, if I use the hierarchical clustering procedure, I have the ability to cluster both variables and cases using a variety of methods and distance measures.  For this task, I would like to use R to cluster my variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For context, my data come from a survey and the respondents were able to select multiple items from a block of options. In my datafile the data are coded as 0,1.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am trying to learn R so any help you can provide will be greatly appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Best regards,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Brock&lt;/p&gt;&#xA;" OwnerUserId="569" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-01T11:47:09.310" LastActivityDate="2011-04-01T12:48:36.933" Title="Best option to cluster variables (not cases) in R" Tags="&lt;r&gt;&lt;clustering&gt;" AnswerCount="1" CommentCount="3" FavoriteCount="1" />
  <row Id="9021" PostTypeId="2" ParentId="4101" CreationDate="2011-03-31T21:15:33.277" Score="1" ViewCount="" Body="&lt;p&gt;Each time series should be evaluated separately with the ultimate idea of collecting i.e. grouping similar series into groups or sections as having similar/common structure. Since time series data can be intervened with by unknown deterministic structure at unspecified poins in time,one is advised to do Intervention Detection to find where the intervention actually had an effect. If you know a law went into effect at a particular point of (de jure) this may in fact (de facto) not the date when the intervention actually happened. Systems can respond in advance of a known effect date or even after the date due to non-compliance or non-response. Specifying the date of the intervention can lead to Model Specification Bias. I suggest that you google &quot;Intervention Detection&quot; or &quot;Outlier Detection&quot;. A good book on this would be by Prof. Wei of Temple University published by Addison-Wessley. I believe the title is &quot;Time Series Analysis&quot;. One further comment an Intervention Variable might appear as a Pulse or Level/Step Shift or a Seasonal Pulse or a Local Time Trend.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In response to expanding the discussion about Local Time Trends:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have a series that exhibits 1,2,3,4,5,7,9,11,13,15,16,17,18,19... there has been a change in trend at period 5 and at 10. For me a main question in time series is the detection of level shifts e.g. 1,2,3,4,5,8,9,10,..or another example of a level shift 1,1,1,1,2,2,2,2, AND/OR or the detection of time trend breaks. Just as a Pulse is a difference of a Step, a Step is a difference of a Trend. We have extended the theory of Intervention Detection to the 4th dimension i,e, Trend Point Change. In terms of openess, I have been able to implement such Intervention Detection schemes in conjuction with both ARIMA and Transfer Function Models. I am one of the senior time-series statisticians who have collaborated in the development of AUTOBOX which incorporates these features. I am unaware of anyone else who has programmed this exciting innovation.  Perhaps someone else can comment on an R package that might do that but I don’t think so.&lt;/p&gt;&#xA;" OwnerUserId="3382" LastEditorUserId="3382" LastEditorDisplayName="" LastEditDate="2011-04-01T01:07:15.667" LastActivityDate="2011-04-01T01:07:15.667" CommentCount="3" />
  <row Id="9022" PostTypeId="1" CreationDate="2011-03-31T21:48:27.543" Score="4" ViewCount="46" Body="&lt;p&gt;What steps could be taken to check for bivariate Gaussianity without using regression based check? Can we somehow employ the use of definition of &lt;a href=&quot;http://en.wikipedia.org/wiki/Variogram&quot; rel=&quot;nofollow&quot;&gt;variogram&lt;/a&gt; measure for assessing spatial variability?&lt;/p&gt;&#xA;" OwnerUserId="1102" LastEditorUserId="1102" LastEditorDisplayName="" LastEditDate="2011-03-31T21:58:28.240" LastActivityDate="2011-03-31T22:15:50.260" Title="How to check for bivariate Gaussianity without the use of regression?" Tags="&lt;multivariate-analysis&gt;&lt;normal-distribution&gt;&lt;variance&gt;&lt;spatial&gt;" AnswerCount="1" CommentCount="7" FavoriteCount="1" />
  <row Id="9023" PostTypeId="1" AcceptedAnswerId="9025" CreationDate="2011-03-31T21:48:29.263" Score="0" ViewCount="46" Body="&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/GTB1c.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a Minitab printout. I want to find the value of A5, or S. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think S is supposed to be the sample standard deviation, but I don't know how to calculate it. Any tips on how I should go about calculating it?&lt;/p&gt;&#xA;" OwnerUserId="3062" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-03-31T22:35:29.073" LastActivityDate="2011-03-31T22:38:48.313" Title="How do I deduce the SD from regression and ANOVA tables?" Tags="&lt;estimation&gt;&lt;homework&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="9024" PostTypeId="2" ParentId="9022" CreationDate="2011-03-31T22:05:58.560" Score="2" ViewCount="" Body="&lt;p&gt;I have recently come across this method that was displayed in Johnson and Wichern. Let the data points that you want to test for bivariate normality be designated as $\{ x_{i} \}$. Next, compute the sample covariance matrix and deisgnate it as $S$. For each observed point calculate $d_{j}^{2} = (x_{j} - \bar{x})^{T} S^{-1} (x_{j} - \bar{x})$. Order the values  of the $d_{j}^{2}$ from low to high. The last mathematical step is to plot the pair $(q_{c,p}((j- \frac{1}{2}), d_{j}^{2})$, where $q_{c,p}((j- \frac{1}{2})/n)$ is the $100(j- \frac{1}{2})$ quantile of the chi-squared distribution. The plot should be a straight line if the data has a bivariate normal distribution.&lt;/p&gt;&#xA;" OwnerUserId="3805" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-03-31T22:15:50.260" LastActivityDate="2011-03-31T22:15:50.260" CommentCount="1" />
  <row Id="9025" PostTypeId="2" ParentId="9023" CreationDate="2011-03-31T22:38:48.313" Score="0" ViewCount="" Body="&lt;p&gt;I got it! It's the sqrt of residual SS / (n-2). Cheers!&lt;/p&gt;&#xA;" OwnerUserId="3062" LastActivityDate="2011-03-31T22:38:48.313" CommentCount="2" />
  <row Id="9027" PostTypeId="1" AcceptedAnswerId="9028" CreationDate="2011-04-01T00:05:27.990" Score="5" ViewCount="94" Body="&lt;p&gt;I have two logistic regression models in R made with &lt;code&gt;glm()&lt;/code&gt;.  They both use the same variables, but were made using different subsets of a matrix.  Is there an easy way to get an average model which gives the means of the coefficients and then use this with the predict() function?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[ sorry if this type of question should be posted on a programming site let me know and I'll post it there ]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="1991" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-01T10:38:48.073" LastActivityDate="2011-04-01T10:38:48.073" Title="Is there an easy way to combine two glm models in R?" Tags="&lt;r&gt;&lt;glm&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="9028" PostTypeId="2" ParentId="9027" CreationDate="2011-04-01T00:44:23.557" Score="1" ViewCount="" Body="&lt;p&gt;Do you want to take the average of the predicted probabilities, or the average of the coefficients? They will give different results, because a logistic regression involves a nonlinear transform of the linear predictor.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A function to do either would be something like this. Set &lt;code&gt;avg&lt;/code&gt; to &lt;code&gt;&quot;prob&quot;&lt;/code&gt; to get the former, or something else for the latter.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pred_comb &amp;lt;- function(mod1, mod2, dat, avg=&quot;prob&quot;, ...)&#xA;{&#xA;    xb1 &amp;lt;- predict(mod1, dat, type=&quot;link&quot;, ...)&#xA;    xb2 &amp;lt;- predict(mod2, dat, type=&quot;link&quot;, ...)&#xA;    if(avg == &quot;prob&quot;)&#xA;        (plogis(xb1) + plogis(xb2))/2&#xA;    else plogis((xb1 + xb2)/2)&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1569" LastEditorUserId="1569" LastEditorDisplayName="" LastEditDate="2011-04-01T00:50:06.837" LastActivityDate="2011-04-01T00:50:06.837" CommentCount="6" />
  <row Id="9029" PostTypeId="1" AcceptedAnswerId="9057" CreationDate="2011-04-01T05:06:28.050" Score="4" ViewCount="98" Body="&lt;p&gt;Hi, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm working with a data set with 2-3 response variables and 7 predictor variables. All the variables are categorical. If there were just one response variable, I think a multinomial logit would be the right model, but there are 2 or 3. So my question is - is there a multivariate version of the multinomial logit?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've looked at several books on categorical data, but haven't seen anything like this (mainly using Agresti 2002).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have about 2000 observations, though I'll probably need to split it up into 2 or 3 data subsets to really see what's going on. One thing I was thinking about is converting it to counts and use a model for count data. I could also combine the 2-3 response vars into 1 categorical with a lot of categories, but I think that will lower the chances of anything showing up for any of the categories. I could also do 2-3 separate models, one for each variable, which is obviously not as good.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I might also be able to get rid of some of the predictors (I think 3 of the 7 have the most explanatory power). I'm not opposed to using machine learning methods, I've found some interesting stuff already with decision trees.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;thanks,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-paul&lt;/p&gt;&#xA;" OwnerUserId="3984" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-04-01T06:02:59.600" LastActivityDate="2011-04-01T16:57:26.190" Title="Is there a version of multivariate multinomial logit?" Tags="&lt;multivariate-analysis&gt;&lt;categorical-data&gt;&lt;multinomial&gt;" AnswerCount="1" CommentCount="4" FavoriteCount="2" />
  <row Id="9030" PostTypeId="1" CreationDate="2011-04-01T07:38:22.440" Score="6" ViewCount="44" Body="&lt;p&gt;I am looking for a document or research articles classifying physical or chemical measurements (or perhaps better means of measurement) according to the reference statistical distribution and properties they have.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example of questions studied would be&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;What kind of weight balance equipment, technology and procedure have the most gaussian behavior? In what order scale range?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;What is the typical distribution of error in a spectroscopic ray analyzer?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;What is the typical distribution of an amperemeter, a voltmeter, etc.?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;What are the noise distributions of the various diode technologies, etc.?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I am particularly interested by hints of &quot;exotical&quot; distributions in some common apparatus.&lt;/p&gt;&#xA;" OwnerUserId="3840" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-01T10:44:32.360" LastActivityDate="2011-04-01T20:39:18.377" Title="Is there a classification of physical measurements according to their statistical distribution?" Tags="&lt;distributions&gt;&lt;repeated-measures&gt;&lt;measurement&gt;&lt;white-noise&gt;" AnswerCount="1" FavoriteCount="2" />
  <row Id="9032" PostTypeId="2" ParentId="9014" CreationDate="2011-04-01T09:02:56.170" Score="1" ViewCount="" Body="&lt;p&gt;After reading up about Girsanov theorem and martingale theory, I can come up with the following observations. First if we have a filtration $\mathcal{F}_t$ and two probability measures $P$ and $Q$ for which Radon-Nikodym derivative $\frac{dQ}{dP}$ exist, then  for each $\mathcal{F}_t$ there exists a Radon-Nikodym derivative $D_t$ with respect to $\mathcal{F}_t$ and $D_t$ is a uniformly integrable martingale with respect to $\mathcal{F}_t$ and $P$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now if we have measure $P$ and a martingale $Z_t$ with filtration $\mathcal{F}_t$ we can define set function $Q=Z_t\cdot P$ defined on $\cup \mathcal{F}_t$. It will define a unique probability measure $Q$ on $\sigma(\cup \mathcal{F}_t)$ if $Z_t$ has additional properties, $EZ_t\equiv 1$ being one of them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Going into more details requires reposting some book on this topic, which is not feasible. I read &lt;a href=&quot;http://books.google.com/books?id=1ml95FLM5koC&amp;amp;printsec=frontcover&amp;amp;dq=revuz%20yor&amp;amp;hl=fr&amp;amp;ei=bJOVTY-GCcmbOqW8-acH&amp;amp;sa=X&amp;amp;oi=book_result&amp;amp;ct=result&amp;amp;resnum=1&amp;amp;ved=0CC0Q6AEwAA#v=onepage&amp;amp;q&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;this one&lt;/a&gt;. Chapter VIII is a good read for clarifying things up. Naturally other books can be found. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that this is really a comment not an answer, I suggest trying to ask at math.SE, with details what exactly you did not understand, the question in the current format can be answered in many different ways, since there is a lot of things going on.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-04-12T11:45:39.793" LastActivityDate="2011-04-12T11:45:39.793" CommentCount="9" />
  <row Id="9033" PostTypeId="1" CreationDate="2011-04-01T09:19:56.027" Score="1" ViewCount="107" Body="&lt;p&gt;Going through some old notes of mine, I found a formula whose purpose seemed statistical, but whose provenance and use I have dumbly forgotten to write. I am asking if anybody might remember this one:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align*}&#xA;-N-\frac1{N}\sum_{j=1}^N (2j-1)(A_j+B_j),&#xA;\end{align*}&#xA;where&#xA;\begin{align*}&#xA;A_j=\ln\left(\frac{1}{2}+\int_{\bar{x}}^{x_j} \exp\left(-\frac{(t-\bar{x})^2}{2s^2}\right)\mathrm{dt}\right)&#xA;\end{align*}&#xA;and&#xA;\begin{align*}&#xA;B_j=\ln\left(\frac{1}{2}-\int_{\bar{x}}^{x_{N-j+1}} \exp\left(-\frac{(t-\bar{x})^2}{2s^2}\right)\mathrm{dt}\right)&#xA;\end{align*}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and where $x_j,\quad j=1\dots N$  is given data, $\bar{x}$  is the mean, and $s$ is  the standard deviation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anybody help?&lt;/p&gt;&#xA;" OwnerUserId="3987" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-01T11:46:09.357" LastActivityDate="2011-04-01T18:11:38.237" Title="Identifying a statistical formula" Tags="&lt;normal-distribution&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="9034" PostTypeId="2" ParentId="7166" CreationDate="2011-04-01T10:57:40.617" Score="3" ViewCount="" Body="&lt;p&gt;Predicting hourly data has become my main interest. This problem arises normally in Call Center Forecasting. One needs to be concerned with hourly patterns within the day , different daily patterns across the week and seasonal patterns across the year ( Monthly indicators/Weekly Indicators. In addition there can be and I have seen interaction between hourly patterns and daily patterns. Transfer Function ( a generalization/super-set of Regression for time series data ) can easily accommodate the mentioned structures. Additionally events during the year (Xmas, Easter etc) need to be possibly included using lead, contemporaneous and/or lag structures. In time series analysis we need to validate via Intervention Detection schemes that there are no Pulses, Level/Step Shifts , Seasonal Pulses and/or Local Time  Trends remaining in the error process suggesting an augmentation to the model. If the residual series suggests autotregressive structure then one simply adds a suitable ARIMA structure. Care should be taken when selecting a resource to deal with this problem. I recently analyzed and developed forecasts for a similar problem: the number of passengers in the Paris Subway System by hour and by day. IMHO this is a problem of constructing a useful equation from the data which can then be used to simulate possible scenarios which can then be used to evaluate queue length etc. &lt;/p&gt;&#xA;" OwnerUserId="3382" LastActivityDate="2011-04-01T10:57:40.617" CommentCount="1" />
  <row Id="9035" PostTypeId="2" ParentId="8800" CreationDate="2011-04-01T11:26:39.950" Score="2" ViewCount="" Body="&lt;p&gt;Ok, as far as I see your central question can be summarized in the following way:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;I transform one feature F in my&#xA;  dataset, but only for those rows where&#xA;  the label is &quot;flower&quot;, the rest of the&#xA;  rows remain unchanged. I evaluate&#xA;  NB,C4.5 and Logistic Regression before&#xA;  and after the transformation. The results obtained&#xA;  via NB and C4.5 did not change, but&#xA;  the result via LogisticRegression. Why&#xA;  ?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In general, the transformation of values of one feature (and additionally, restricted only to a subset of the data), may cause nearly anything. Imagine for example that one replaces the old value by a constant or values mainly occurring in another class. In this cases the predictive power of the feature may decrease rapidly. On the other hand, if the values do not overlap with the values of F of the other classes before AND after the transformation, the predictive power is not afflicted at all.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So in general it is hard to explain your results without knowing the whole dataset + algorithm parameterization/implementation. I wonder on the other hand whether the results differ significantly, given that the flower class only occurs 15 times in 5000. Did you analyse the confusion matrix ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I can provide some hints:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;NB in my experience is rather robust. So it may that F has less impact because other features have more predictive power and have already &lt;em&gt;sealed&lt;/em&gt; the deal ;). You can e.g. calculate &lt;a href=&quot;http://en.wikipedia.org/wiki/Information_gain_ratio&quot; rel=&quot;nofollow&quot;&gt;InformationGainRatio&lt;/a&gt; to estimate the predictive power before and after the transformation.&lt;/li&gt;&#xA;&lt;li&gt;C4.5: Do the trees differ ? If F is not selected as split node or removed during pruning, the results won't be affected at all. If F has been selected it would be interesting to see whether the new split-point can be inverse transformed to get the old splitpoint. Maybe the overlapping of classes at this point (i.e. given the restriction of the dataspace when F is selected as split-node) has not changed.&lt;/li&gt;&#xA;&lt;li&gt;Logistic Regression: My knowledge is not thaaat solid here, in the implementations I used the coefficients are evaluated via an evolutionary approach. Maybe the results differ because it ran into a local minima ? In this case on can rerun the learner multiple times (before and after the transformation) to see if/how the results change. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Regarding the second question: Since the two questions differ extremely, I recommend to open a new question for IDS. Meanwhile you may some of &lt;a href=&quot;http://www.google.de/#hl=de&amp;amp;source=hp&amp;amp;q=intrusion+detection+survey&amp;amp;aq=f&amp;amp;aqi=&amp;amp;aql=&amp;amp;oq=&amp;amp;fp=2cdc0fce7b07738a&quot; rel=&quot;nofollow&quot;&gt;these links&lt;/a&gt; interesting.&lt;/p&gt;&#xA;" OwnerUserId="264" LastActivityDate="2011-04-01T11:26:39.950" />
  <row Id="9036" PostTypeId="1" AcceptedAnswerId="9037" CreationDate="2011-04-01T11:31:34.533" Score="1" ViewCount="104" Body="&lt;p&gt;Using R, I noticed that I get different results if I use fisher.test() with raw data (as factors) versus a contingency table.  The documentation states that fisher.test(x,y) will compute a contingency table from x,y by treating them as factors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My example is a baseline statistics exercise: comparing the gender split in a placebo group to the experimental group.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's my example for running fisher.test on factors:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; expData &amp;lt;- data.frame(Placebo=factor(c(&quot;Y&quot;,&quot;Y&quot;,&quot;Y&quot;,&quot;Y&quot;,&quot;N&quot;,&quot;N&quot;,&quot;N&quot;,&quot;N&quot;)),&#xA;    Gender=factor(c(&quot;F&quot;,&quot;F&quot;,&quot;F&quot;,&quot;M&quot;,&quot;M&quot;,&quot;M&quot;,&quot;M&quot;,&quot;F&quot;)))&#xA;&amp;gt; expData&#xA;  Placebo Gender&#xA;1       Y      F&#xA;2       Y      F&#xA;3       Y      F&#xA;4       Y      M&#xA;5       N      M&#xA;6       N      M&#xA;7       N      M&#xA;8       N      F&#xA;&amp;gt; fisher.test(expData[expData$Placebo==&quot;Y&quot;,&quot;Gender&quot;],&#xA;    expData[expData$Placebo==&quot;N&quot;,&quot;Gender&quot;])&#xA;&#xA;        Fisher's Exact Test for Count Data&#xA;&#xA;data:  expData[expData$Placebo == &quot;Y&quot;, &quot;Gender&quot;] and &#xA;    expData[expData$Placebo == &quot;N&quot;, &quot;Gender&quot;] &#xA;p-value = 0.25&#xA;alternative hypothesis: true odds ratio is not equal to 1 &#xA;95 percent confidence interval:&#xA;  0.00000 13.00002 &#xA;sample estimates:&#xA;odds ratio &#xA;         0 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here's my example for building a contingency table representing the above data and running fisher.test on that:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; contingency &amp;lt;- matrix(c(3,1,1,3),nrow=2,&#xA;     dimnames = list(tx=c(&quot;Placebo&quot;,&quot;Exper&quot;), gender=c(&quot;F&quot;,&quot;M&quot;)))&#xA;&amp;gt; contingency&#xA;         gender&#xA;tx        F M&#xA;  Placebo 3 1&#xA;  Exper   1 3&#xA;&#xA;&amp;gt; fisher.test(contingency)&#xA;&#xA;        Fisher's Exact Test for Count Data&#xA;&#xA;data:  contingency &#xA;p-value = 0.4857&#xA;alternative hypothesis: true odds ratio is not equal to 1 &#xA;95 percent confidence interval:&#xA;   0.2117329 621.9337505 &#xA;sample estimates:&#xA;odds ratio &#xA;  6.408309 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Am I using fisher.test in the wrong way?  Or maybe building my contingency table wrong?&lt;/p&gt;&#xA;" OwnerUserId="1138" LastActivityDate="2011-04-01T11:39:40.613" Title="In R, fisher.test returns different results if I use vectors vs contingency table" Tags="&lt;r&gt;&lt;hypothesis-testing&gt;&lt;p-value&gt;&lt;contingency-tables&gt;&lt;fisher&gt;" AnswerCount="1" />
  <row Id="9037" PostTypeId="2" ParentId="9036" CreationDate="2011-04-01T11:39:40.613" Score="2" ViewCount="" Body="&lt;p&gt;The first test should read&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; fisher.test(expData[,1], expData[,2])&#xA;&#xA;    Fisher's Exact Test for Count Data&#xA;&#xA;data:  expData[, 1] and expData[, 2] &#xA;p-value = 0.4857&#xA;alternative hypothesis: true odds ratio is not equal to 1 &#xA;95 percent confidence interval:&#xA; 0.001607888 4.722931239 &#xA;sample estimates:&#xA;odds ratio &#xA;  0.156047 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;as per the doc: &lt;code&gt;x&lt;/code&gt; is the outcome and &lt;code&gt;y&lt;/code&gt; is the factor (or vice-versa).&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-04-01T11:39:40.613" CommentCount="1" />
  <row Id="9038" PostTypeId="1" AcceptedAnswerId="9051" CreationDate="2011-04-01T11:40:34.837" Score="2" ViewCount="38" Body="&lt;p&gt;I'm using &lt;a href=&quot;http://www.itl.nist.gov/div898/strd/nls/data/LINKS/DATA/ENSO.dat&quot; rel=&quot;nofollow&quot;&gt;ENSO dataset&lt;/a&gt; from the NIST Statistical Reference Datasets as a test for nonlinear regression code. &#xA;The data are monthly averaged atmospheric pressure differences between Easter Island and Darwin, called &lt;a href=&quot;http://en.wikipedia.org/wiki/El_Ni%C3%B1o-Southern_Oscillation&quot; rel=&quot;nofollow&quot;&gt;Southern Oscillation&lt;/a&gt; Index (SOI). In the test, the data are fitted with three sine cycles (&lt;a href=&quot;http://fityk.nieto.pl/_images/fityk-1.0.1-osx-so.png&quot; rel=&quot;nofollow&quot;&gt;like this&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I got curious about the Southern Oscillations and I started looking for more SOI data. I learned that there are a few different methods of how to calculate the SOI, but none of the other sources that I found (e.g. &lt;a href=&quot;http://www.bom.gov.au/climate/current/soihtm1.shtml&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;) gives the same numbers are in the NIST StRD.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a reference in the dataset, &lt;em&gt;Numerical Methods and Software&lt;/em&gt; by Kahaner, Moler and Nash, where I found only that the data are for years 1962-1975.&#xA;So my question is not really about statistics: how the ENSO dataset is related to the SOI data from other sources.&lt;/p&gt;&#xA;" OwnerUserId="3981" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-01T11:44:21.793" LastActivityDate="2011-04-01T14:50:47.687" Title="The source of ENSO data from NIST" Tags="&lt;dataset&gt;&lt;nonlinear-regression&gt;" AnswerCount="1" />
  <row Id="9039" PostTypeId="2" ParentId="8567" CreationDate="2011-04-01T11:45:03.477" Score="0" ViewCount="" Body="&lt;p&gt;From wiki &quot;Given a data set, several candidate models may be ranked according to their AIC values. From the AIC values one may also infer that e.g. the top two models are roughly in a tie and the rest are far worse. Thus, AIC provides a means for comparison among models—a tool for model selection. AIC does not provide a test of a model in the usual sense of testing a null hypothesis; i.e. AIC can tell nothing about how well a model fits the data in an absolute sense. Ergo, if all the candidate models fit poorly, AIC will not give any warning of that.&quot; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus the AIC should bot be used to do model selection. After reading up on DLM it appears to me that this approach is MODEL-BASED rather than DATA-BASED. One is assuming a model when using DLM. A DLM model uses a mixture of high order polynomials in time and fixed trigonometric structures which might be useful for certain physical time series. This approach does not lend itself to generating a set of residuals free of structure which meet the Gaussian requirements. A useful alternative is to form the model empirically detecting as needed any auto-projective structure, any deterministic structure (Pulses, level Shifts, Seasonal Pulses and.or Local Time Trends via INTERVENTION DETECTION also referred to as OUTLIER DETECTION.&lt;/p&gt;&#xA;" OwnerUserId="3382" LastActivityDate="2011-04-01T11:45:03.477" />
  <row Id="9040" PostTypeId="1" AcceptedAnswerId="9048" CreationDate="2011-04-01T11:54:19.400" Score="7" ViewCount="190" Body="&lt;p&gt;Hi All,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First off, I'm a programmer but my experience with true statistics ended at A-Level so I'm looking to all of you for help with a little side project I've been tinkering with.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At home I use Plex Media Center to display all of my movies. I built an export tool for this to generate a HTML file containing information on your library so that others can view it online. After I made this tool I realised I now had access to a wealth of data about films and the actors in them. And this is where you guys (and gals) hopefully come in.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to visualize the relationships between actors and movies somehow. Initially I just used a node graph library to map all actors who have been in more than one movie to all their movies and ended up with this: &lt;a href=&quot;http://www.flickr.com/photos/dachande663/5574979625/&quot; rel=&quot;nofollow&quot;&gt;http://www.flickr.com/photos/dachande663/5574979625/&lt;/a&gt; [section of a 5000x2500px image]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is, with anything more than 250 movies it just turns into a mess of spaghetti that's impossible to follow. I've looked into arc diagrams but think it would just be even more confusing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question therefore is: how do I visualize this? Size isn't too much of an issue as I'd love to print this out on a large canvas and actually hang it up. Also, I'll eventually replace the text with images of the respective movies and actors. What I'm trying to avoid is having a million lines snaking everywhere. I've tried to find the most important movies and place them more centrally but at the moment that's more guess work than actual logic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there libraries that can do a better job of this, or even a better way of displaying the data (dropping actors as nodes and adding them as edge labels)? I'm currently using Dracula graph, which provides an okay-starting point but can change as needed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any input will be much appreciated. Cheers.&lt;/p&gt;&#xA;" OwnerUserId="3988" LastActivityDate="2011-05-04T13:50:39.353" Title="Visualize movie/actor relationships" Tags="&lt;data-visualization&gt;" AnswerCount="4" CommentCount="3" FavoriteCount="2" />
  <row Id="9041" PostTypeId="2" ParentId="9040" CreationDate="2011-04-01T12:10:43.893" Score="2" ViewCount="" Body="&lt;p&gt;Graphviz can optimise the layout, see something similar &lt;a href=&quot;http://www.graphviz.org/content/siblings&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastActivityDate="2011-04-01T12:10:43.893" CommentCount="2" />
  <row Id="9042" PostTypeId="2" ParentId="9020" CreationDate="2011-04-01T12:48:36.933" Score="1" ViewCount="" Body="&lt;p&gt;Hi&#xA;Take a look at the &lt;strong&gt;varclus&lt;/strong&gt; function in the Frank Harrel's &lt;strong&gt;Hmisc&lt;/strong&gt; package. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(Hmisc)&#xA;? varclus&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2028" LastActivityDate="2011-04-01T12:48:36.933" />
  <row Id="9043" PostTypeId="1" AcceptedAnswerId="9044" CreationDate="2011-04-01T08:53:26.107" Score="1" ViewCount="93" Body="&lt;p&gt;I'm helping my father to translate a text from my own native Latvian language to English. Even though I'm a fair English speaker and have a bit of mathematics in my past, I never did have a firm grasp of statistics. :(&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The term I'm looking for has something to do with error calculation. A GPS device is measuring coordinates and then trying to estimate the error. In wikipedia I've come across two different likely terms - &lt;a href=&quot;http://en.wikipedia.org/wiki/Mean_squared_error&quot; rel=&quot;nofollow&quot;&gt;Mean squared error&lt;/a&gt; and &lt;a href=&quot;http://en.wikipedia.org/wiki/Root_mean_square_deviation&quot; rel=&quot;nofollow&quot;&gt;Root mean square error&lt;/a&gt;. From what I can understand, they are not one and the same thing, but what each of them means is beyond me.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A direct translation from Latvian language would be &quot;average (mean?) squared error estimate&quot;. No idea what that means either, I'm afraid.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Perhaps someone here has a better idea?&lt;/p&gt;&#xA;" OwnerUserId="2590" LastActivityDate="2011-04-01T13:10:24.590" Title="What is the English name for a statistics term that I'm looking for?" Tags="&lt;terminology&gt;" AnswerCount="2" CommentCount="6" />
  <row Id="9044" PostTypeId="2" ParentId="9043" CreationDate="2011-04-01T09:51:37.740" Score="2" ViewCount="" Body="&lt;p&gt;Google translate gives &lt;a href=&quot;http://translate.google.co.uk/?hl=en&amp;amp;tab=wT#lv%7Cen%7Cvid%C4%93jo%20kvadr%C4%81tisko%20k%C4%BC%C5%ABdu%20nov%C4%93rt%C4%93jums&quot; rel=&quot;nofollow&quot;&gt;&quot;rms error of assessment&quot;&lt;/a&gt; which (as &lt;em&gt;root mean square error&lt;/em&gt;) seems to be the intended meaning.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The difference between &lt;em&gt;mean square error&lt;/em&gt; and &lt;em&gt;root &#xA;mean square error&lt;/em&gt; is similar to the differrence between &lt;em&gt;variance&lt;/em&gt; and &lt;em&gt;standard deviation&lt;/em&gt;: if you want the average error to be reported in the same units as the measurement (metres or degrees or whatever) then you need to take the square root.&lt;/p&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-04-01T09:51:37.740" CommentCount="3" />
  <row Id="9045" PostTypeId="2" ParentId="9043" CreationDate="2011-04-01T12:21:40.590" Score="0" ViewCount="" Body="&lt;p&gt;I would guess, it is a Least Mean Squared Error (LMSE) estimator. &lt;a href=&quot;http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=5090171&quot; rel=&quot;nofollow&quot;&gt;publication1&lt;/a&gt; &lt;a href=&quot;http://web.cecs.pdx.edu/~ssp/Reports/2006/Monaghan.pdf&quot; rel=&quot;nofollow&quot;&gt;publication2&lt;/a&gt; &lt;a href=&quot;http://en.wikipedia.org/wiki/Mean_squared_error&quot; rel=&quot;nofollow&quot;&gt;wikipedia&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;the GPS is trying to estimate coordinates, it does so by minimizing a square error function, so reporting the coordinate that is best in terms of MSE.&lt;/p&gt;&#xA;" OwnerUserId="4060" LastActivityDate="2011-04-01T12:29:20.880" />
  <row Id="9048" PostTypeId="2" ParentId="9040" CreationDate="2011-04-01T13:58:06.120" Score="5" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;N.B.&lt;/strong&gt;: &lt;em&gt;This was previously a (long) comment that I've converted to an answer. Hopefully I'll be able to post an example of what I describe below within a day or two.&lt;/em&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why not try something like a heatmap? Have movies as rows and actors as columns. Maybe sort each of them in terms of the number of actors in the movie and number of movies each actor has been in. Then color each cell where there is a match. This is basically a visualization of the &lt;em&gt;adjacency matrix&lt;/em&gt;. The proposed sorting should make some interesting patterns and the right use of color could make it both artistic and more informative. Maybe color by movie type or Netflix rating or proportion of male to female actors (or viewers!), etc.&lt;/p&gt;&#xA;" OwnerUserId="2970" LastActivityDate="2011-04-01T13:58:06.120" CommentCount="4" />
  <row Id="9049" PostTypeId="2" ParentId="9040" CreationDate="2011-04-01T14:30:18.650" Score="1" ViewCount="" Body="&lt;p&gt;I wouldn't know how you'd go about constructing this but I liked the method using hyperbolic geometry&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.newscientist.com/data/images/ns/cms/dn19420/dn19420-1_800.jpg&quot; rel=&quot;nofollow&quot;&gt;http://www.newscientist.com/data/images/ns/cms/dn19420/dn19420-1_800.jpg&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.newscientist.com/article/dn19420-escherlike-internet-map-could-speed-online-traffic.html&quot; rel=&quot;nofollow&quot;&gt;http://www.newscientist.com/article/dn19420-escherlike-internet-map-could-speed-online-traffic.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="3484" LastActivityDate="2011-04-01T14:30:18.650" CommentCount="2" />
  <row Id="9050" PostTypeId="1" CreationDate="2011-04-01T14:40:54.433" Score="6" ViewCount="159" Body="&lt;p&gt;I'd like to obtain a graphic representation of the correlations in articles I have gathered so far to easily explore the relationships between variables. I used to draw a (messy) graph but I have too much data now.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically, I have a table with:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;[0]: name of variable 1&lt;/li&gt;&#xA;&lt;li&gt;[1]: name of variable 2 &lt;/li&gt;&#xA;&lt;li&gt;[2]: correlation value&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The &quot;overall&quot; matrix is incomplete (e.g., I have the correlation of V1*V2, V2*V3, but not V1*V3).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a way to graphically represent this ?&lt;/p&gt;&#xA;" OwnerUserId="3827" LastEditorUserId="3827" LastEditorDisplayName="" LastEditDate="2011-04-01T16:27:54.967" LastActivityDate="2011-04-01T17:02:00.770" Title="How to display a matrix of correlations with missing entries?" Tags="&lt;r&gt;&lt;data-visualization&gt;&lt;correlation&gt;" AnswerCount="3" FavoriteCount="3" />
  <row Id="9051" PostTypeId="2" ParentId="9038" CreationDate="2011-04-01T14:50:47.687" Score="2" ViewCount="" Body="&lt;p&gt;You would not expect your two sets of numbers to be the same: the NIST numbers seem to be have been published to &lt;a href=&quot;http://www.itl.nist.gov/div898/strd/general/bkground.html&quot; rel=&quot;nofollow&quot;&gt;test statistical software&lt;/a&gt; while the Australian BOM numbers are designed to &lt;a href=&quot;http://www.bom.gov.au/climate/glossary/soi.shtml&quot; rel=&quot;nofollow&quot;&gt;identify whether the ENSO cycle is above or below average&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In particular the BOM numbers have been standardised to an index to have monthly seasonal factors removed, and for the remaining numbers to have a mean of around 0 and standard deviation around 10.   They may also have had corrections or improved sources used since the NIST numbers were published.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The NIST have had not had these proceses applied: in fact one of the purposes of the illustrated analysis is identifying the mean (coefficient &lt;code&gt;b1&lt;/code&gt;) and the magnitude of the annual cycle (coefficients &lt;code&gt;b2&lt;/code&gt; and &lt;code&gt;b3&lt;/code&gt;).  It is possible that the NIST numbers are in real physical units (perhaps of pressure).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might have hoped that there was a linear relationship between the two sets of numbers, for example between the January BOM numbers and data points 1, 13, 25, ... , 157 of the NIST numbers.  I did not spot it, though I did not try very hard.&lt;/p&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-04-01T14:50:47.687" CommentCount="1" />
  <row Id="9052" PostTypeId="2" ParentId="9050" CreationDate="2011-04-01T15:10:47.950" Score="5" ViewCount="" Body="&lt;p&gt;Your data may be like&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  name1 name2 correlation&#xA;1    V1    V2         0.2&#xA;2    V2    V3         0.4&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can rearrange your long table into a wide one with the following R code&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;d = structure(list(name1 = c(&quot;V1&quot;, &quot;V2&quot;), name2 = c(&quot;V2&quot;, &quot;V3&quot;), &#xA;    correlation = c(0.2, 0.4)), .Names = c(&quot;name1&quot;, &quot;name2&quot;, &#xA;    &quot;correlation&quot;), row.names = 1:2, class = &quot;data.frame&quot;)&#xA;k = d[, c(2, 1, 3)]&#xA;names(k) = names(d)&#xA;e = rbind(d, k)&#xA;x = with(e, reshape(e[order(name2),], v.names=&quot;correlation&quot;, &#xA;  idvar=&quot;name1&quot;, timevar=&quot;name2&quot;, direction=&quot;wide&quot;))&#xA;x[order(x$name1),]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You get&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  name1 correlation.V1 correlation.V2 correlation.V3&#xA;1    V1             NA            0.2             NA&#xA;3    V2            0.2             NA            0.4&#xA;4    V3             NA            0.4             NA&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now you can use techniques for visualizing correlation matrices (at least ones that can cope with missing values).&lt;/p&gt;&#xA;" OwnerUserId="3911" LastActivityDate="2011-04-01T15:10:47.950" CommentCount="1" />
  <row Id="9053" PostTypeId="1" CreationDate="2011-04-01T16:26:57.173" Score="7" ViewCount="252" Body="&lt;p&gt;Why does a cross-validation procedure overcome the problem of overfitting a model?&lt;/p&gt;&#xA;" OwnerUserId="3269" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-04-01T17:37:38.940" LastActivityDate="2011-04-02T17:40:55.537" Title="How does cross-validation overcome the overfitting problem?" Tags="&lt;regression&gt;&lt;model-selection&gt;&lt;cross-validation&gt;" AnswerCount="3" CommentCount="2" FavoriteCount="2" />
  <row Id="9054" PostTypeId="2" ParentId="9050" CreationDate="2011-04-01T16:38:04.397" Score="7" ViewCount="" Body="&lt;p&gt;Building upon @GaBorgulya's response, I would suggest trying fluctuation or level plot (aka heatmap displays).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, using &lt;a href=&quot;http://had.co.nz/ggplot2/&quot; rel=&quot;nofollow&quot;&gt;ggplot2&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(ggplot2, quietly=TRUE)&#xA;k &amp;lt;- 100&#xA;rvals &amp;lt;- sample(seq(-1,1,by=.001), k, replace=TRUE)&#xA;rvals[sample(1:k, 10)] &amp;lt;- NA&#xA;cc &amp;lt;- matrix(rvals, nr=10)&#xA;ggfluctuation(as.table(cc)) + opts(legend.position=&quot;none&quot;) + &#xA;  labs(x=&quot;&quot;, y=&quot;&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(Here, missing entry are displayed in plain gray, but the default color scheme can be changed, and you can also put &quot;NA&quot; in the legend.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/CiT5m.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;or&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ggfluctuation(as.table(cc), type=&quot;color&quot;) + labs(x=&quot;&quot;, y=&quot;&quot;) +&#xA;  scale_fill_gradient(low = &quot;red&quot;,  high = &quot;blue&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(Here, missing values are simply not displayed. However, you can add a &lt;code&gt;geom_text()&lt;/code&gt; and display something like &quot;NA&quot; in the empty cell.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/SkfOB.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-01T17:02:00.770" LastActivityDate="2011-04-01T17:02:00.770" CommentCount="5" />
  <row Id="9055" PostTypeId="2" ParentId="9053" CreationDate="2011-04-01T16:51:53.760" Score="7" ViewCount="" Body="&lt;p&gt;I can't think of a sufficiently clear explanation just at the moment, so I'll leave that to someone else; however cross-validation does not completely overcome the over-fitting problem in model selection, it just reduces it.  The cross-validation error does not have a negligible variance, especially if the size of the dataset is small; in other words you get a slightly different value depending on the particular sample of data you use.  This means that if you have many degrees of freedom in model selection (e.g. lots of features from which to select a small subset, many hyper-parameters to tune, many models from which to choose) you can over-fit the cross-validation criterion as the model is tuned in ways that exploit this random variation rather than in ways that really do improve performance, and you can end up with a model that performs poorly.  For a discussion of this, see &lt;a href=&quot;http://jmlr.csail.mit.edu/papers/v11/cawley10a.html&quot; rel=&quot;nofollow&quot;&gt;Cawley and Talbot &quot;On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation&quot;, JMLR, vol. 11, pp. 2079−2107, 2010&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sadly cross-validation is most likely to let you down when you have a small dataset, which is exactly when you need cross-validation the most.  Note that k-fold cross-validation is better than leave-one-out as it has a lower variance (but is often more expensive to compute).&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2011-04-01T16:51:53.760" />
  <row Id="9056" PostTypeId="2" ParentId="9050" CreationDate="2011-04-01T16:52:54.727" Score="3" ViewCount="" Body="&lt;p&gt;The &lt;code&gt;corrplot&lt;/code&gt; package is a useful function for visualizing correlation matrices. It accepts a correlation matrix as the input object and has several options for displaying the matrix itself. A nice feature is that it can reorder your variables using hierarchical clustering or PCA methods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See the accepted answer in &lt;a href=&quot;http://stats.stackexchange.com/questions/2086/quickly-evaluate-visually-correlations-between-ordered-categorical-data-in-r&quot;&gt;this thread&lt;/a&gt; for an example visualization.&lt;/p&gt;&#xA;" OwnerUserId="3309" LastActivityDate="2011-04-01T16:52:54.727" />
  <row Id="9057" PostTypeId="2" ParentId="9029" CreationDate="2011-04-01T16:57:26.190" Score="2" ViewCount="" Body="&lt;p&gt;Agresti 2007 discusses them.  They're in chapter 9 and 10. The 2002 edition probably discusses them too, as @suncoolsu mentioned.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Agresti refers to the group of response variables is referred to as a &lt;em&gt;cluster&lt;/em&gt; and discusses their analysis with marginal models, conditional models and generalized estimating equations.&lt;/p&gt;&#xA;" OwnerUserId="3874" LastActivityDate="2011-04-01T16:57:26.190" CommentCount="1" />
  <row Id="9058" PostTypeId="2" ParentId="9033" CreationDate="2011-04-01T18:11:38.237" Score="3" ViewCount="" Body="&lt;p&gt;When correctly rewritten (as indicated in a comment), it will be the &lt;a href=&quot;http://en.wikipedia.org/wiki/Anderson%E2%80%93Darling_test&quot; rel=&quot;nofollow&quot;&gt;Anderson-Darling&lt;/a&gt; statistic (for a normality test).&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-04-01T18:11:38.237" />
  <row Id="9059" PostTypeId="2" ParentId="9053" CreationDate="2011-04-01T18:23:29.910" Score="4" ViewCount="" Body="&lt;p&gt;My answer is more intuitive than rigorous, but maybe it will help...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As I understand it, overfitting is the result of model selection based on training and testing using the same data, where you have a flexible fitting mechanism: you fit your sample of data so closely that you're fitting the noise, outliers, and all the other variance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Splitting the data into a training and testing set keeps you from doing this. But a static split is not using your data efficiently and your split itself could be an issue. Cross-validation keeps the don't-reward-an-exact-fit-to-training-data advantage of the training-testing split, while also using the data that you have as efficiently as possible (i.e. all of your data is used as training and testing data, just not in the same run).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have a flexible fitting mechanism, you need to constrain your model selection so that it doesn't favor &quot;perfect&quot; but complex fits somehow. You can do it with AIC, BIC, or some other penalization method that penalizes fit complexity directly, or you can do it with CV. (Or you can do it by using a fitting method that is not very flexible, which is one reason linear models are nice.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another way of looking at it is that learning is about generalizing, and a fit that's too tight is in some sense not generalizing. By varying what you learn on and what you're tested on, you generalize better than if you only learned the answers to a specific set of questions.&lt;/p&gt;&#xA;" OwnerUserId="1764" LastActivityDate="2011-04-01T18:23:29.910" />
  <row Id="9060" PostTypeId="2" ParentId="9030" CreationDate="2011-04-01T20:39:18.377" Score="2" ViewCount="" Body="&lt;p&gt;Some people have started to look at this issue in the &lt;strong&gt;chemometrics literature&lt;/strong&gt;.  For instance, about 20 years ago Robert Gibbons started to do statistical analyses suggesting instrument responses (for low-level measurement of chemicals) were nonlinear, heteroscedastic, and had non-normal (perhaps lognormal) error distributions.  I found an &lt;a href=&quot;http://www.springerlink.com/content/v2r30g3729752173/&quot; rel=&quot;nofollow&quot;&gt;abstract of one of those papers&lt;/a&gt; on Springer's site.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-04-01T20:39:18.377" CommentCount="1" />
  <row Id="9061" PostTypeId="2" ParentId="8911" CreationDate="2011-04-01T21:35:45.077" Score="0" ViewCount="" Body="&lt;p&gt;R would be my first vote. Another free option would be gretl. If you happen to know BUGS, JAGS makes sense and is free. And I really don't like its syntax, but if you have some knowledge of Matlab, the free alternative Octave runs on MacOS X as well.&lt;/p&gt;&#xA;" OwnerUserId="1764" LastActivityDate="2011-04-01T21:35:45.077" CommunityOwnedDate="2011-04-01T21:35:45.077" />
  <row Id="9062" PostTypeId="1" AcceptedAnswerId="9096" CreationDate="2011-04-01T21:45:15.740" Score="2" ViewCount="143" Body="&lt;p&gt;I have a 2x3 contingency table - the row variable is a factor, the column variable is an ordered factor (ordinal level). I'd like to apply either symmetrical or asymmetrical association technique. What do you recommend me to do? Which technique do you find the most appropriate?&lt;/p&gt;&#xA;" OwnerUserId="1356" LastActivityDate="2011-04-02T16:46:40.420" Title="Measure of association for 2x3 contingency table" Tags="&lt;correlation&gt;&lt;categorical-data&gt;&lt;contingency-tables&gt;&lt;association-measure&gt;" AnswerCount="4" FavoriteCount="1" />
  <row Id="9063" PostTypeId="2" ParentId="9062" CreationDate="2011-04-01T22:39:37.960" Score="2" ViewCount="" Body="&lt;p&gt;On a 2x3 contingency table where the three-level factor is ordered you may use rank correlation (Spearman or Kendall) to assess association between the two variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may also think about the data as an ordered variable observed in two groups. A corresponding significance test could be the Mann-Whitney test (with many ties). This has an associated measure of association, the &lt;a href=&quot;http://www2.sas.com/proceedings/sugi31/209-31.pdf&quot; rel=&quot;nofollow&quot;&gt;WMW odds, related to Agresti’s generalized odds ratio&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Both for rank correlation coefficients and WMW odds confidence intervals can be calculated. I find odds more intuitive, otherwise I believe both kinds of measures are appropriate.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastActivityDate="2011-04-01T22:39:37.960" CommentCount="1" />
  <row Id="9064" PostTypeId="1" AcceptedAnswerId="9084" CreationDate="2011-04-01T23:15:45.397" Score="3" ViewCount="126" Body="&lt;p&gt;I've heard a bit about the 'kernel trick' for support vector machines, and I was wondering:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;How do you identify problems that might benefit from the kernel trick?&lt;/li&gt;&#xA;&lt;li&gt;How to implement it in R?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Thank you&lt;/p&gt;&#xA;" OwnerUserId="2817" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-02T13:22:36.957" LastActivityDate="2011-04-02T13:22:36.957" Title="Implementing the 'kernel trick' for a support vector machine in R" Tags="&lt;r&gt;&lt;machine-learning&gt;&lt;kernel&gt;&lt;svm&gt;" AnswerCount="2" FavoriteCount="4" />
  <row Id="9065" PostTypeId="5" CreationDate="2011-04-01T23:43:56.183" Score="0" ViewCount="" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditorDisplayName="" LastEditDate="2011-04-01T23:43:56.183" LastActivityDate="2011-04-01T23:43:56.183" />
  <row Id="9066" PostTypeId="4" CreationDate="2011-04-01T23:43:56.183" Score="0" ViewCount="" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditorDisplayName="" LastEditDate="2011-04-01T23:43:56.183" LastActivityDate="2011-04-01T23:43:56.183" />
  <row Id="9067" PostTypeId="1" AcceptedAnswerId="9088" CreationDate="2011-04-02T00:07:35.167" Score="0" ViewCount="119" Body="&lt;p&gt;I am taking a class in data mining and I am working on a term project using the &lt;a href=&quot;http://www.cdc.gov/brfss/&quot; rel=&quot;nofollow&quot;&gt;BRFSS&lt;/a&gt; dataset. I have a huge dataset with 405 columns and 12,000 rows. There are many columns which are completely empty. I was trying to remove empty columns using SAS, R or Excel but it doesn't work. Could you suggest a method to remove the empty columns or any tutorial that will help me cleaning up the data?  There are a lot of missing cells too. I am using KNIME to train my data and it doesn't work if there are missing values. How can I handle the missing values? &lt;/p&gt;&#xA;" OwnerUserId="3897" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-02T13:09:16.883" LastActivityDate="2011-04-02T20:36:24.113" Title="Removing empty columns from a dataset" Tags="&lt;excel&gt;" AnswerCount="3" CommentCount="5" />
  <row Id="9068" PostTypeId="1" CreationDate="2011-04-01T14:57:10.287" Score="7" ViewCount="174" Body="&lt;p&gt;I am having a problem computing the pearson correlation coefficient of data sets with possibly zero standard deviation (i.e. all data has the same value).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose that I have the following two data sets:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;float x[] = {2, 2, 2, 3, 2};&#xA;float y[] = {2, 2, 2, 2, 2};&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The correlation coefficient &quot;r&quot;, would be computed using the following equation:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;float r = covariance(x, y) / (std_dev(x) * std_dev(y));&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, because all data in data set &quot;y&quot; has the same value, the standard deviation std_dev(y) would be zero and &quot;r&quot; would be undefined.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any solution for this problem? Or should I use other methods to measure data relationship in this case?&lt;/p&gt;&#xA;" OwnerUserId="3993" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-29T01:08:20.040" LastActivityDate="2011-04-29T01:08:20.040" Title="Pearson correlation of data sets with possibly zero standard deviation?" Tags="&lt;correlation&gt;" AnswerCount="4" CommentCount="11" FavoriteCount="1" />
  <row Id="9069" PostTypeId="2" ParentId="9062" CreationDate="2011-04-02T00:25:13.240" Score="2" ViewCount="" Body="&lt;p&gt;One way to incorporate the ordering of the column factor into your analysis is to use the cumulative frequencies instead of the cell frequencies.  So in your table you have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$f_{ij}=\frac{n_{ij}}{n_{\bullet\bullet}}\;\;\;\; i=1,2\;\;j=1,2,3$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where a &quot;$\bullet$&quot; indicates sum over that index.  So I suggesting modeling instead:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$g_{ij}=\sum_{k=1}^{j}f_{ik}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now you basically have a simple hypothesis for association, that the index $i$ doesn't matter.  So you have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$E(g_{ij}|H_{0})=\sum_{k=1}^{j}\frac{n_{\bullet k}}{n_{\bullet\bullet}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And then use the good old &quot;entropy&quot; test statistic:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$T(H_{0})=n_{\bullet\bullet}\sum_{i,j}g_{ij}log\left(\frac{g_{ij}}{E(g_{ij}|H_{0})}\right)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Plugging in the numbers gives:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$T(H_{0})=\sum_{i,j}\left(\sum_{k=1}^{j}n_{ik}\right)log\left(\frac{\sum_{k=1}^{j}n_{ik}}{\sum_{k=1}^{j}n_{\bullet k}}\right)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And you reject if this number is too big, it should be interpreted as a &quot;log-odds&quot; ratio which will help with choosing cut-offs.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-04-02T00:25:13.240" CommentCount="3" />
  <row Id="9070" PostTypeId="2" ParentId="9062" CreationDate="2011-04-02T01:14:14.767" Score="1" ViewCount="" Body="&lt;p&gt;You could use the Jonckheere Terpstra test.  In SAS, you can get this in PROC FREQ with the /JT option on the tables statement.   I didn't see a function for it in R, but there may be one out there.&lt;/p&gt;&#xA;" OwnerUserId="686" LastActivityDate="2011-04-02T01:14:14.767" CommentCount="2" />
  <row Id="9071" PostTypeId="1" AcceptedAnswerId="9073" CreationDate="2011-04-02T01:28:12.757" Score="5" ViewCount="78" Body="&lt;p&gt;If I have two normally distributed independent random variables $X$ and $Y$ with means $\mu_X$ and $\mu_Y$ and standard deviations $\sigma_X$ and $\sigma_Y$ and I discover that $X+Y=c$, then (assuming I have not made any errors) the conditional distribution of $X$ and $Y$ given $c$ are also normally distributed with means &#xA;$$\mu_{X|c} = \mu_X + (c - \mu_X - \mu_Y)\frac{ \sigma_X^2}{\sigma_X^2+\sigma_Y^2}$$ $$\mu_{Y|c} = \mu_Y + (c - \mu_X - \mu_Y)\frac{ \sigma_Y^2}{\sigma_X^2+\sigma_Y^2}$$&#xA;and standard deviation &#xA;$$\sigma_{X|c} = \sigma_{Y|c} = \sqrt{ \frac{\sigma_X^2 \sigma_Y^2}{\sigma_X^2 + \sigma_Y^2}}.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is no surprise that the conditional standard deviations are the same as, given $c$, if one goes up the other must come down by the same amount.  It is interesting that the conditional standard deviation does not depend on $c$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I cannot get my head round are the conditional means, where they take a share of the excess $(c - \mu_X - \mu_Y)$ proportional to the original variances, not to the original  standard deviations.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if they have zero means, $\mu_X=\mu_Y=0$, and standard deviations $\sigma_X =3$ and $\sigma_Y=1$ then conditioned on $c=4$ we would have $E[X|c=4]=3.6$ and $E[Y|c=4]=0.4$, i.e. in the ratio $9:1$ even though I would have intuitively thought that the ratio $3:1$ would be more natural.  &lt;strong&gt;Can anyone give an intuitive explanation for this?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This was provoked by &lt;a href=&quot;http://math.stackexchange.com/questions/30365/posterior-distribution-after-having-partial-information-on-some-linear-combinatio&quot;&gt;a Math.SE question&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-04-02T03:05:20.290" Title="Intuitive explanation of contribution to sum of two normally distributed random variables" Tags="&lt;normal-distribution&gt;&lt;conditional-probability&gt;" AnswerCount="1" FavoriteCount="3" />
  <row Id="9072" PostTypeId="2" ParentId="9068" CreationDate="2011-04-02T02:07:54.087" Score="0" ViewCount="" Body="&lt;p&gt;The correlation is undefined in that case. If you must define it, I would define it as 0, but consider a simple mean absolute difference instead.&lt;/p&gt;&#xA;" OwnerUserId="2456" LastActivityDate="2011-04-02T02:07:54.087" />
  <row Id="9073" PostTypeId="2" ParentId="9071" CreationDate="2011-04-02T03:05:20.290" Score="5" ViewCount="" Body="&lt;p&gt;The question readily reduces to the case $\mu_X = \mu_Y = 0$ by looking at $X-\mu_X$ and $Y-\mu_Y$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Clearly the conditional distributions are Normal.  Thus, the mean, median, and mode of each are coincident.  The modes will occur at the coordinates of a local maximum of the bivariate PDF of $X$ and $Y$ constrained to the curve $g(x,y) = x+y = c$.  This implies the contour of the bivariate PDF at this location and the constraint curve have parallel tangents.  (This is the theory of Lagrange multipliers.)  Because the equation of any contour is of the form $f(x,y) = x^2/(2\sigma_X^2) + y^2/(2\sigma_Y^2) = \rho$ for some constant $\rho$ (that is, all contours are ellipses), their gradients must be parallel, whence there exists $\lambda$ such that&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\left(\frac{x}{\sigma_X^2}, \frac{y}{\sigma_Y^2}\right) = \nabla f(x,y) = \lambda \nabla g(x,y) = \lambda(1,1).$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/zezHu.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It follows immediately that the &lt;em&gt;modes&lt;/em&gt; of the conditional distributions (and therefore also the means) are determined by the ratio of the variances, not of the SDs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This analysis works for correlated $X$ and $Y$ as well and it applies to any linear constraints, not just the sum.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-04-02T03:05:20.290" CommentCount="2" />
  <row Id="9074" PostTypeId="1" AcceptedAnswerId="9082" CreationDate="2011-04-02T05:39:09.150" Score="3" ViewCount="131" Body="&lt;p&gt;Dear Members,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm wondering how to implement two-way clustering, as explained in &lt;a href=&quot;http://www.statsoft.com/textbook/cluster-analysis/#twotwo&quot; rel=&quot;nofollow&quot;&gt;Statistica documentation&lt;/a&gt; in R. Any help in this regard will be highly appreciated. Thanks&lt;/p&gt;&#xA;" OwnerUserId="3903" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-02T13:07:14.437" LastActivityDate="2011-04-02T13:07:14.437" Title="Two-way clustering in R" Tags="&lt;r&gt;&lt;clustering&gt;&lt;multivariate-analysis&gt;" AnswerCount="1" />
  <row Id="9075" PostTypeId="2" ParentId="9064" CreationDate="2011-04-02T06:08:42.183" Score="3" ViewCount="" Body="&lt;p&gt;you should take a look at kernlab R package. They even have a very nice &lt;a href=&quot;http://cran.r-project.org/web/packages/kernlab/vignettes/kernlab.pdf&quot; rel=&quot;nofollow&quot;&gt;vignette&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="223" LastActivityDate="2011-04-02T06:08:42.183" />
  <row Id="9076" PostTypeId="1" AcceptedAnswerId="9078" CreationDate="2011-04-02T06:29:34.960" Score="2" ViewCount="70" Body="&lt;p&gt;I am trying to programmatically identify an ARIMA model for a series of data and forecast values. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently the problem i am facing is to find a way to evaluate partial autocorrelation. I have been looking for methods to calculate PACF for quite a long time now but in vain. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please provide some online resources which can help me in this matter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you.&lt;/p&gt;&#xA;" OwnerUserId="3972" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-02T10:22:43.337" LastActivityDate="2011-04-04T18:01:02.420" Title="Methods for evaluating partial autocorrelation for identification of ARIMA models" Tags="&lt;autocorrelation&gt;&lt;arima&gt;" AnswerCount="2" />
  <row Id="9077" PostTypeId="2" ParentId="9067" CreationDate="2011-04-02T07:11:25.833" Score="1" ViewCount="" Body="&lt;p&gt;If they really are just completely blank columns then in R...&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;read.table( 'myBigFile', strip.white = TRUE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;might do what you want.  You will have to set other arguments of the read.table() command as needed.  It's best to use this when specifying the specific column delimiter you have.&lt;/p&gt;&#xA;" OwnerUserId="601" LastEditorUserId="601" LastEditorDisplayName="" LastEditDate="2011-04-02T07:24:39.017" LastActivityDate="2011-04-02T07:24:39.017" />
  <row Id="9078" PostTypeId="2" ParentId="9076" CreationDate="2011-04-02T07:12:52.580" Score="3" ViewCount="" Body="&lt;p&gt;Use the Durbin-Levinson algorithm. It will be explained in any good time series book such as &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0387953515&quot; rel=&quot;nofollow&quot;&gt;Brockwell and Davis&lt;/a&gt;. Here are some online explanations:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.stat.ufl.edu/~berg/sta4853/files/sta4853-4.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.stat.ufl.edu/~berg/sta4853/files/sta4853-4.pdf&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://amath.colorado.edu/courses/4540/2008Spr/HandOuts/DurbinLevinson.pdf&quot; rel=&quot;nofollow&quot;&gt;http://amath.colorado.edu/courses/4540/2008Spr/HandOuts/DurbinLevinson.pdf&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="159" LastActivityDate="2011-04-02T07:12:52.580" />
  <row Id="9079" PostTypeId="2" ParentId="9068" CreationDate="2011-04-02T08:31:46.503" Score="3" ViewCount="" Body="&lt;p&gt;I agree with sesqu that the correlation is undefined in this case. Depending on your type of application you could e.g. calculate the Gower Similarity between both vectors, which is:&#xA;$gower(v1,v2)=\frac{\sum_{i=1}^{n}\delta(v1_i,v2_i)}{n}$ where $\delta$ represents the &lt;a href=&quot;http://en.wikipedia.org/wiki/Kronecker_delta&quot; rel=&quot;nofollow&quot;&gt;kronecker-delta&lt;/a&gt;, applied as function on $v1,v2$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So for instance if all values are equal, gower(.,.)=1. If on the other hand they differ only in one dimension, gower(.,.)=0.9. If they differ in every dimension, gower(.,.)=0 and so on. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course this is no measure for correlation, but it allows you to calculate how close the vector with s&gt;0 is to the one with s=0. Of course you can apply other metrics,too, if they serve your purpose better.&lt;/p&gt;&#xA;" OwnerUserId="264" LastActivityDate="2011-04-02T08:31:46.503" CommentCount="2" />
  <row Id="9080" PostTypeId="1" CreationDate="2011-04-02T09:09:33.990" Score="1" ViewCount="111" Body="&lt;p&gt;The objective of this research was to investigate the long-term effects of irrigation with treated waste water on some chemical soil properties. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The investigation was carried out by comparison of soil properties in two different fields: one irrigated with the effluent from Parkan Waste water Treatment Plant over a period of six years, and the other one irrigated with water over the same period of time. Soil samples were taken from different depths of 0-15, 15-30, 30-60, 60-100 and 100-150 cm in both fields, and analyzed for various chemical properties. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For visual summaries, I am going to plot depths of soil and element (with line and bar plots). However, I also need to fit a linear model with one categorical and one continuous predictor. How can this be done in R?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for your answer!&lt;/p&gt;&#xA;" OwnerUserId="3996" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-02T10:36:18.717" LastActivityDate="2011-04-02T12:45:38.977" Title="How to do a linear model in R?" Tags="&lt;r&gt;&lt;regression&gt;" AnswerCount="2" />
  <row Id="9081" PostTypeId="2" ParentId="9080" CreationDate="2011-04-02T09:16:40.097" Score="4" ViewCount="" Body="&lt;p&gt;The syntax is actually absurdly simple:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mymodel &amp;lt;- lm(dependentvariable ~ continuousvariable + categoricalvariable,    &#xA;              data=yourdata).&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can then call &lt;code&gt;summary()&lt;/code&gt; to get the coefficients, and &lt;code&gt;plot()&lt;/code&gt; to examine the residuals. &#xA;HTH.&lt;/p&gt;&#xA;" OwnerUserId="656" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-02T10:37:25.003" LastActivityDate="2011-04-02T10:37:25.003" CommentCount="1" />
  <row Id="9082" PostTypeId="2" ParentId="9074" CreationDate="2011-04-02T10:20:17.737" Score="3" ViewCount="" Body="&lt;p&gt;Generally speaking, you should always find useful pointers by looking at the relevant CRAN TAsk Views, in this case the one that deals with &lt;a href=&quot;http://cran.r-project.org/web/views/Cluster.html&quot; rel=&quot;nofollow&quot;&gt;Cluster&lt;/a&gt; packages, or maybe &lt;a href=&quot;http://www.statmethods.net/&quot; rel=&quot;nofollow&quot;&gt;Quick-R&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's not clear to me whether the link you gave referenced standard clustering techniques for $n$ (individuals) by $k$ (variables) matrix of measures where we impose constraints on the resulting heatmap displays, or two-mode clustering or &lt;a href=&quot;http://en.wikipedia.org/wiki/Biclustering&quot; rel=&quot;nofollow&quot;&gt;biclustering&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;In the first approach&lt;/strong&gt;, we could, for example,&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;compute a measure of (dis)similarity between individuals, or correlation between variables, and show the resulting $n\times n$ or $k\times k$ matrix where rows and columns are rearranged by some kind of partitioning or ordering technique -- this help highlighting possible substructures in the association matrix, and you will find more information in this &lt;a href=&quot;http://stats.stackexchange.com/questions/8370/how-to-visualize-summarize-a-matrix-with-number-of-rows-gg-number-of-columns/8372#8372&quot;&gt;related question&lt;/a&gt;;&lt;/li&gt;&#xA;&lt;li&gt;compute the correlation between two blocks of data observed on the same individuals, and reorder the pattern of correlations following an external ordination technique (e.g., hierarchical clustering) -- it amounts to show a heatmap of the observed statistics reordered by rows &lt;em&gt;and&lt;/em&gt; columns.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;As proposed in an &lt;a href=&quot;http://stats.stackexchange.com/questions/6890/plotting-a-heatmap-given-a-dendrogram-and-a-distance-matrix-in-r/6893#6893&quot;&gt;earlier response&lt;/a&gt;, the latter is readily available in the &lt;code&gt;cim()&lt;/code&gt; function from the &lt;a href=&quot;http://cran.r-project.org/web/packages/mixOmics/index.html&quot; rel=&quot;nofollow&quot;&gt;mixOmics&lt;/a&gt; package. From the on-line help, we can end up with something like that:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/LFj7J.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please, note that this is just a two-step process to conveniently display summary measures of association: clustering of rows (individuals or variables) and columns (individuals or variables) is done separately.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;In the second approach&lt;/strong&gt; (biclustering), that I'm inclined to favour, I only know one R package, &lt;a href=&quot;http://cran.r-project.org/web/packages/biclust/index.html&quot; rel=&quot;nofollow&quot;&gt;biclust&lt;/a&gt;, that is greatly inspired for research in bioinformatics. Some pointers were also given in an &lt;a href=&quot;http://stats.stackexchange.com/questions/7419/getting-started-with-biclustering&quot;&gt;earlier thread&lt;/a&gt;. (But there's even some papers in the psychometrics literature.) In this case, we need to put some constraints during clustering because we want to cluster both individuals &lt;em&gt;and&lt;/em&gt; variables &lt;em&gt;at the same time&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Again, you can display the resulting structure as heatmaps (see &lt;code&gt;help(heatmapBC)&lt;/code&gt;), as shown below&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/fQNgb.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-02T10:27:54.797" LastActivityDate="2011-04-02T10:27:54.797" CommentCount="3" />
  <row Id="9083" PostTypeId="2" ParentId="9080" CreationDate="2011-04-02T11:24:37.767" Score="2" ViewCount="" Body="&lt;p&gt;First you have to enter your data into R, see &lt;a href=&quot;http://www.ats.ucla.edu/stat/r/notes/entering.htm&quot; rel=&quot;nofollow&quot;&gt;this class note&lt;/a&gt;. You can follow the steps of &lt;a href=&quot;http://data.princeton.edu/R/linearModels.html&quot; rel=&quot;nofollow&quot;&gt;this tutorial&lt;/a&gt; in the analysis, section 4.4 has a very similar example. In visualization you could do something similar as the &lt;code&gt;qplot(wt, mpg, data = mtcars, colour = factor(cyl))&lt;/code&gt; example of &lt;a href=&quot;http://had.co.nz/ggplot2/geom_point.html&quot; rel=&quot;nofollow&quot;&gt;this tutorial&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-02T12:45:38.977" LastActivityDate="2011-04-02T12:45:38.977" CommentCount="1" />
  <row Id="9084" PostTypeId="2" ParentId="9064" CreationDate="2011-04-02T13:21:19.140" Score="3" ViewCount="" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Basically anything what is not separable with a line (ok, hyperplane), for instance 2D data like this:&lt;br&gt;&#xA;&lt;img src=&quot;http://i.imgur.com/tmzeQ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;br&gt;&#xA;kernel trick will effectively project this situation into a (higher-dim) space in which linear separation is possible; see this movie for an &lt;a href=&quot;http://www.youtube.com/watch?v=9NrALgHFwTo&quot; rel=&quot;nofollow&quot;&gt;effect of a gaussian kernel on similar data&lt;/a&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Look for a kernel argument in your svm function ;-) Note that using a kernel usually introduces new parameters to the outer optimization.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="88" LastActivityDate="2011-04-02T13:21:19.140" CommentCount="3" />
  <row Id="9085" PostTypeId="1" CreationDate="2011-04-02T13:32:36.553" Score="9" ViewCount="268" Body="&lt;p&gt;In my attempts to fight spreadsheet mayhem, I am often evangelical in pushing for more robust tools such as true statistics software (R, Stata, and the like).  Recently, I was challenged on this view by someone who stated flat out that they simply will not learn to program.  I would like to provide them with data analysis tools that require no programming (but ideally which would extend to programming if they decide to dip a toe into the water later).  What packages are out there for data exploration that I can recommend with a straight face?&lt;/p&gt;&#xA;" OwnerUserId="3488" LastActivityDate="2011-04-04T17:09:56.883" Title="Software for easy-yet-robust data exploration" Tags="&lt;data-visualization&gt;&lt;software&gt;" AnswerCount="9" CommentCount="6" FavoriteCount="1" />
  <row Id="9086" PostTypeId="2" ParentId="9085" CreationDate="2011-04-02T13:35:47.410" Score="2" ViewCount="" Body="&lt;p&gt;A new software system that looks promising for this purpose is &lt;a href=&quot;http://www.r-bloggers.com/r-ready-to-deduce-you/&quot; rel=&quot;nofollow&quot;&gt;Deducer&lt;/a&gt;, built on top of R.  Unfortunately, being new, I suspect it does not yet cover the breadth of questions that people might ask, but it does meet the toe-in-the-water criterion of leading people towards a true package should they so decide later.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've also used JMP in the past, which had a nice interactivity to it.  I am worried that some of the interface might be too complicated for these purposes.  And it's non-free, which makes it harder for potential spreadsheet refugees to try out on a whim.&lt;/p&gt;&#xA;" OwnerUserId="3488" LastActivityDate="2011-04-02T13:35:47.410" />
  <row Id="9087" PostTypeId="1" AcceptedAnswerId="9091" CreationDate="2011-04-02T13:49:55.227" Score="4" ViewCount="84" Body="&lt;p&gt;Let $X = N(0,\frac{1}{\alpha})$, $Y = 2X + 8 + N_{y}$, and $N_{y}$ be a noise $N_{y} = N(0,1)$.  Then, $P(y|x)  = \frac{1}{\sqrt{2\pi}}exp\{ -\frac{1}{2}(y - 2x - 8)^{2} \}$&#xA;and $P(x) = \sqrt{\frac{\alpha}{2\pi}}exp\{-\frac{\alpha x^{2}}{2}\} $.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The mean vector is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\mathbf{\mu} =  \left( \begin{array}{c}&#xD;&#xA;\mu_{x}\\&#xD;&#xA;\mu_{y}\end{array} \right)= \left( \begin{array}{c}&#xD;&#xA;0\\&#xD;&#xA;8\end{array} \right).$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The question is how to calculate the variance of Y.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know that the correct answer is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{4}{\alpha} + 1, $$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;but don't know how to get from &#xA;$$var(Y) = E[(Y-\mu_{y})^{2}] = E[(2X+N_{y})^{2}] $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;to&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{4}{\alpha} + 1. $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anybody help?&#xA;UPDATE:&#xA;Thank you All for answers&lt;/p&gt;&#xA;" OwnerUserId="1371" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-29T00:42:48.390" LastActivityDate="2011-04-29T00:42:48.390" Title="Inference with Gaussian Random Variable" Tags="&lt;normal-distribution&gt;&lt;variance&gt;&lt;random-variable&gt;&lt;inference&gt;" AnswerCount="2" />
  <row Id="9088" PostTypeId="2" ParentId="9067" CreationDate="2011-04-02T14:05:43.993" Score="1" ViewCount="" Body="&lt;p&gt;Empty columns contain &lt;code&gt;NA&lt;/code&gt;s only or &lt;code&gt;&quot;&quot;&lt;/code&gt;s only, they have has no variability. This code removes all columns without variability (which is probably a plus in this case).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;d=data.frame(r=seq(1, 5), a=rep('a', 5), n=rep(NA, 5), n1=c(NA, NA, 3, 3, 3))&#xA;&#xA;homogenous = apply(d, 2, function(var) length(unique(var)) == 1)&#xA;d[, !homogenous]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="3911" LastActivityDate="2011-04-02T14:05:43.993" CommentCount="1" />
  <row Id="9089" PostTypeId="2" ParentId="9087" CreationDate="2011-04-02T14:05:58.993" Score="4" ViewCount="" Body="&lt;p&gt;Solution to this homework is straightforward application of &lt;strong&gt;simple&lt;/strong&gt; algebra and independence of $X$ and $N_y$: $\mathbb{E} (2 X + N_y)^2 = 4 \mathbb{E} X^2 + 4 \mathbb{E} X \mathbb{E} N_y + \mathbb{E} N_y^2 = 4 Var X + 0 + Var N_y = \frac{4}{\alpha} + 1$.&lt;/p&gt;&#xA;" OwnerUserId="2645" LastEditorUserId="2645" LastEditorDisplayName="" LastEditDate="2011-04-02T16:02:38.683" LastActivityDate="2011-04-02T16:02:38.683" CommentCount="1" />
  <row Id="9090" PostTypeId="2" ParentId="9068" CreationDate="2011-04-02T14:06:14.687" Score="4" ViewCount="" Body="&lt;p&gt;The &quot;sampling theory&quot; people will tell you that no such estimate exists.  But you can get one, you just need to be reasonable about your prior information, and do a lot harder mathematical work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you specified a Bayesian method of estimation, and the posterior is the same as the prior, then you can say the data say nothing about the parameter.  Because things may get &quot;singular&quot; on us, then we cannot use infinite parameter spaces.  I am assuming that because you use Pearson correlation, you have a bivariate normal likelihood:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p(D|\mu_x,\mu_y,\sigma_x,\sigma_y,\rho)=\left(\sigma_x\sigma_y\sqrt{2\pi(1-\rho^2)}\right)^{-N}exp\left(-\frac{\sum_{i}Q_i}{2(1-\rho^2)}\right)$$&#xA;where&#xA;$$Q_i=\frac{(x_i-\mu_x)^2}{\sigma_x^2}+\frac{(y_i-\mu_y)^2}{\sigma_y^2}-2\rho\frac{(x_i-\mu_x)(y_i-\mu_y)}{\sigma_x\sigma_y}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now to indicate that one data set may be the same value, write $y_i=y$, and then we get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\sum_{i}Q_i=N\left[\frac{(y-\mu_y)^2}{\sigma_y^2}+\frac{s_x^2 + (\overline{x}-\mu_x)^2}{\sigma_x^2}-2\rho\frac{(\overline{x}-\mu_x)(y-\mu_y)}{\sigma_x\sigma_y}\right]$$&#xA;where&#xA;$$s_x^2=\frac{1}{N}\sum_{i}(x_i-\overline{x})^2$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And so your likelihood depends on four numbers, $s_x^2,y,\overline{x},N$.  So you want an estimate of $\rho$, so you need to multiply by a prior, and integrate out the nuisance parameters $\mu_x,\mu_y,\sigma_x,\sigma_y$.  Now to prepare for integration, we &quot;complete the square&quot;&#xA;$$\frac{\sum_{i}Q_i}{1-\rho^2}=N\left[\frac{\left(\mu_y-\left[y-(\overline{x}-\mu_x)\frac{\rho\sigma_y}{\sigma_x}\right]\right)^2}{\sigma_y^2(1-\rho^{2})}+\frac{s_x^2}{\sigma_{x}^{2}(1-\rho^{2})} + \frac{(\overline{x}-\mu_x)^2}{\sigma_x^2}\right]$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now we should err on the side of caution and ensure a properly normalised probability.  That way we can't get into trouble.  One such option is to use a weakly informative prior, which just places restriction on the range of each.  So we have $L_{\mu}&amp;lt;\mu_x,\mu_y&amp;lt;U_{\mu}$ for the means with flat prior and $L_{\sigma}&amp;lt;\sigma_x,\sigma_y&amp;lt;U_{\sigma}$ for the standard deviations with jeffreys prior.  These limits are easy to set with a bit of &quot;common sense&quot; thinking about the problem.  I will take an unspecified prior for $\rho$, and so we get (uniform should work ok, if not truncate the singularity at $\pm 1$):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p(\rho,\mu_x,\mu_y,\sigma_x,\sigma_y)=\frac{p(\rho)}{A\sigma_x\sigma_y}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $A=2(U_{\mu}-L_{\mu})^{2}[log(U_{\sigma})-log(L_{\sigma})]^{2}$.  This gives a posterior of:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p(\rho|D)=\int p(\rho,\mu_x,\mu_y,\sigma_x,\sigma_y)p(D|\mu_x,\mu_y,\sigma_x,\sigma_y,\rho)d\mu_y d\mu_x d\sigma_x d\sigma_y$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$=\frac{p(\rho)}{A[2\pi(1-\rho^2)]^{\frac{N}{2}}}\int_{L_{\sigma}}^{U_{\sigma}}\int_{L_{\sigma}}^{U_{\sigma}}\left(\sigma_x\sigma_y\right)^{-N-1}exp\left(-\frac{N s_x^2}{2\sigma_{x}^{2}(1-\rho^{2})}\right) \times$$&#xA;$$\int_{L_{\mu}}^{U_{\mu}}exp\left(-\frac{N(\overline{x}-\mu_x)^2}{2\sigma_x^2}\right)\int_{L_{\mu}}^{U_{\mu}}exp\left(-\frac{N\left(\mu_y-\left[y-(\overline{x}-\mu_x)\frac{\rho\sigma_y}{\sigma_x}\right]\right)^2}{2\sigma_y^2(1-\rho^{2})}\right)d\mu_y d\mu_x d\sigma_x d\sigma_y$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now the first integration over $\mu_y$ can be done by making a change of variables $z=\sqrt{N}\frac{\mu_y-\left[y-(\overline{x}-\mu_x)\frac{\rho\sigma_y}{\sigma_x}\right]}{\sigma_y\sqrt{1-\rho^{2}}}\implies dz=\frac{\sqrt{N}}{\sigma_y\sqrt{1-\rho^{2}}}d\mu_y$ and the first integral over $\mu_y$ becomes:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{\sigma_y\sqrt{2\pi(1-\rho^{2})}}{\sqrt{N}}\left[\Phi\left(&#xD;&#xA;\frac{U_{\mu}-\left[y-(\overline{x}-\mu_x)\frac{\rho\sigma_y}{\sigma_x}\right]}{\frac{\sigma_y}{\sqrt{N}}\sqrt{1-\rho^{2}}}&#xD;&#xA;\right)-\Phi\left(&#xD;&#xA;\frac{L_{\mu}-\left[y-(\overline{x}-\mu_x)\frac{\rho\sigma_y}{\sigma_x}\right]}{\frac{\sigma_y}{\sqrt{N}}\sqrt{1-\rho^{2}}}&#xD;&#xA;\right)\right]$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And you can see from here, no analytic solutions are possible.  However, it is also worthwhile to note that the value $\rho$ has not dropped out of the equations.  This means that the data and prior information still have something to say about the true correlation.  If the data said nothing about the correlation, then we would be simply left with $p(\rho)$ as the only function of $\rho$ in these equations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It also shows how that passing to the limit of infinite bounds for $\mu_y$ &quot;throws away&quot; some of the information about $\rho$, which is contained in the complicated looking normal CDF function $\Phi(.)$.  Now if you have a lot of data, then passing to the limit is fine, you don't loose much, but if you have very scarce information, such as in your case - it is important keep every scrap you have.  It means ugly maths, but this example is not too hard to do numerically.  So we can evaluate the integrated likelihood for $\rho$ at values of say $-0.99,-0.98,\dots,0.98,0.99$ fairly easily.  Just replace the integrals by summations over a small enough intervals - so you have a triple summation&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-04-02T14:06:14.687" CommentCount="7" />
  <row Id="9091" PostTypeId="2" ParentId="9087" CreationDate="2011-04-02T14:25:22.247" Score="6" ViewCount="" Body="&lt;p&gt;The law of iterated expectations can help here.  We have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Var[Y]=E(Var[Y|X])+Var[E(Y|X)]$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now conditional on $X$ the expected value of $Y$ is $2X+8$, and its variance is $1$.  So we have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Var[Y]=E(1)+Var[2X+8]=1+4 Var[X]=1+\frac{4}{\alpha}$$&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-04-02T14:25:22.247" CommentCount="1" />
  <row Id="9092" PostTypeId="2" ParentId="9068" CreationDate="2011-04-02T15:04:59.187" Score="0" ViewCount="" Body="&lt;p&gt;This question is coming from programmers, so I'd suggest plugging in zero. There's no evidence of a correlation, and the null hypothesis would be zero (no correlation).  There might be other context knowledge that would provide a &quot;typical&quot; correlation in one context, but the code might be re-used in another context.&lt;/p&gt;&#xA;" OwnerUserId="3919" LastActivityDate="2011-04-02T15:04:59.187" CommentCount="3" />
  <row Id="9093" PostTypeId="2" ParentId="9085" CreationDate="2011-04-02T15:11:16.043" Score="6" ViewCount="" Body="&lt;p&gt;Some people think of programming as simply entering a command line statement.  At that point then perhaps you are a bit lost in encouraging them.  However, if they are using spreadsheets already then they already have to enter formulas.  These are akin to command line statements.  If they really mean they don't want to do any programming in the sense of logical and automated analysis then you can tell them that they can still do the analyses in R or Stata without any programming at all.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If they can do their stats in the spreadsheet... all that they want to do... then all of the statistical analyses they wish to accomplish can be done without 'programming' in R or Stata as well.  They could arrange and organize the data in the spreadsheet and then just export it as text.  Then the analysis is carried out without any programming at all.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That's how I do intro to R sometimes.  No programming is required to do the data analysis you could do in a spreadsheet.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you get them hooked that way then just reel the fish in slowly... :)  In a couple of years compliment them on what a good programmer they've become.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might also want to show &lt;a href=&quot;http://www.burns-stat.com/pages/Tutor/spreadsheet_addiction.html&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; document to your colleagues or at least read it yourself to better make your points.&lt;/p&gt;&#xA;" OwnerUserId="601" LastEditorUserId="601" LastEditorDisplayName="" LastEditDate="2011-04-04T17:09:56.883" LastActivityDate="2011-04-04T17:09:56.883" CommentCount="4" />
  <row Id="9094" PostTypeId="2" ParentId="9085" CreationDate="2011-04-02T15:11:34.827" Score="5" ViewCount="" Body="&lt;p&gt;As far as exploratory (possibly interactive) data analysis is concerned, I would suggest to take a look at:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.cs.waikato.ac.nz/ml/weka/&quot; rel=&quot;nofollow&quot;&gt;Weka&lt;/a&gt;, originally targets data-mining applications, but can be used for data summaries.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://rosuda.org/mondrian/&quot; rel=&quot;nofollow&quot;&gt;Mondrian&lt;/a&gt;, for interactive data visualization.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://knime.org/knime&quot; rel=&quot;nofollow&quot;&gt;KNIME&lt;/a&gt;, which relies on the idea of building data flows and is compatible with Weka and R.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;All three accept data in &lt;code&gt;arff&lt;/code&gt; or &lt;code&gt;csv&lt;/code&gt; format.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my view, Stata does not require so much programming expertise. This is even part of its attractiveness, in fact: Most of basic analysis can be done by point-and-click user actions, with dialog boxes for customizing specific parameters, say, for prediction in a linear model. The same applies, albeit to a lesser extent, to R when you use external GUIs like &lt;a href=&quot;http://socserv.mcmaster.ca/jfox/Misc/Rcmdr/&quot; rel=&quot;nofollow&quot;&gt;Rcmdr&lt;/a&gt;, Deducer, etc. as said by @gsk3.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-04-02T15:11:34.827" />
  <row Id="9096" PostTypeId="2" ParentId="9062" CreationDate="2011-04-02T16:46:40.420" Score="1" ViewCount="" Body="&lt;p&gt;Linear or monotonic trend tests--$M^2$ association measure, WMW test cited by @GaBorgulya, or the Cochran-Armitage trend test--can also be used, and they are well explained in Agresti (&lt;a href=&quot;http://www.stat.ufl.edu/~aa/cda/cda.html&quot; rel=&quot;nofollow&quot;&gt;CDA&lt;/a&gt;, 2002, §3.4.6, p. 90). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The latter is actually equivalent to a score test for testing $H_0:\; \beta = 0$ in a logistic regression model, but it can be computed from the $M^2$ statistic, defined as $(n-1)r^2$ ($\sim\chi^2(1)$ for large sample), where $r$ is the sample correlation coefficient between the two variables (the ordinal measure being recoded as numerical scores), by replacing $n-1$ with $n$ (ibid., p. 182). It is easy to compute in any statistical software, but you can also use the &lt;a href=&quot;http://cran.r-project.org/web/packages/coin/index.html&quot; rel=&quot;nofollow&quot;&gt;coin&lt;/a&gt; package in R (I provided an example of use for a &lt;a href=&quot;http://stats.stackexchange.com/questions/8774/what-is-the-difference-between-independence-test-in-r-and-cott/8979#8979&quot;&gt;related question&lt;/a&gt;). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Sidenote&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are using R, you will find useful resources in either Laura Thompson's &lt;a href=&quot;https://home.comcast.net/~lthompson221/Splusdiscrete2.pdf&quot; rel=&quot;nofollow&quot;&gt;R (and S-PLUS) Manual to Accompany Agresti’s Categorical Data Analysis (2002)&lt;/a&gt;, which shows how to replicate Agresti's results with R, or the &lt;a href=&quot;http://cran.r-project.org/web/packages/gnm/index.html&quot; rel=&quot;nofollow&quot;&gt;gnm&lt;/a&gt; package (and its companion packages, &lt;a href=&quot;http://cran.r-project.org/web/packages/vcd/index.html&quot; rel=&quot;nofollow&quot;&gt;vcd&lt;/a&gt; and &lt;a href=&quot;http://cran.r-project.org/web/packages/vcdExtra/index.html&quot; rel=&quot;nofollow&quot;&gt;vcdExtra&lt;/a&gt;) which allows to fit row-column association models (see the vignette, &lt;a href=&quot;http://cran.r-project.org/web/packages/gnm/vignettes/gnmOverview.pdf&quot; rel=&quot;nofollow&quot;&gt;Generalized nonlinear models in R: An overview of the gnm package&lt;/a&gt;).&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-04-02T16:46:40.420" CommentCount="1" />
  <row Id="9097" PostTypeId="2" ParentId="5690" CreationDate="2011-04-02T16:47:55.793" Score="3" ViewCount="" Body="&lt;p&gt;One thing to keep in mind with the Kaplan-Meier survival curve is that it is basically &lt;em&gt;descriptive&lt;/em&gt; and not &lt;em&gt;inferential&lt;/em&gt;.  It is just a function of the data, with an incredibly flexible model that lies behind it.  This is a strength because this means there is virtually no assumptions that might be broken, but a weakness because it is hard to generalise it, and that it fits &quot;noise&quot; as well as &quot;signal&quot;.  If you want to make an inference, then you basically have to introduce something that is unknown that you wish to know.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now one way to compare the median survival times is to make the following assumptions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;I have an estimate of the median survival time $t_{i}$ for each of the $i$ states, given by the kaplan meier curve.&lt;/li&gt;&#xA;&lt;li&gt;I expect the true median survival time, $T_{i}$ to be equal to this estimate.  $E(T_{i}|t_{i})=t_{i}$&lt;/li&gt;&#xA;&lt;li&gt;I am 100% certain that the true median survival time is positive. $Pr(T_{i}&amp;gt;0)=1$&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Now the &quot;most conservative&quot; way to use these assumptions is the principle of maximum entropy, so you get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p(T_{i}|t_{i})= K exp(-\lambda T_{i})$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $K$ and $\lambda$ are chosen such that the PDF is normalised, and the expected value is $t_{i}$.  Now we have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$1=\int_{0}^{\infty}p(T_{i}|t_{i})dT_{i}&#xD;&#xA;=K \int_{0}^{\infty}exp(-\lambda T_{i})dT_{i}&#xD;&#xA;$$&#xA;$$=K \left[-\frac{exp(-\lambda T_{i})}{\lambda}\right]_{T_{i}=0}^{T_{i}=\infty}=\frac{K}{\lambda}\implies K=\lambda&#xD;&#xA;$$&#xA;and now we have $E(T_{i})=\frac{1}{\lambda}\implies \lambda=t_{i}^{-1}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And so you have a set of probability distributions for each state.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p(T_{i}|t_{i})= \frac{1}{t_{i}} exp\left(-\frac{T_{i}}{t_{i}}\right)\;\;\;\;\;(i=1,\dots,N)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which give a joint probability distribution of:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p(T_{1},T_{2},\dots,T_{N}|t_{1},t_{2},\dots,t_{N})= \prod_{i=1}^{N}\frac{1}{t_{i}} exp\left(-\frac{T_{i}}{t_{i}}\right)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now it sounds like you want to test the hypothesis $H_{0}:T_{1}=T_{2}=\dots=T_{N}=\overline{t}$, where $\overline{t}=\frac{1}{N}\sum_{i=1}^{N}t_{i}$ is the mean median survivial time.  The severe alternative hypothesis to test against is the &quot;every state is a unique and beautiful snowflake&quot; hypothesis $H_{A}:T_{1}=t_{1},\dots,T_{N}=t_{N}$ because this is the most likely alternative, and thus represents the information lost in moving to the simpler hypothesis (a &quot;minimax&quot; test).  The measure of the evidence against the simpler hypothesis is given by the odds ratio:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$O(H_{A}|H_{0})=\frac{p(T_{1}=t_{1},T_{2}=t_{2},\dots,T_{N}=t_{N}|t_{1},t_{2},\dots,t_{N})}{&#xD;&#xA;p(T_{1}=\overline{t},T_{2}=\overline{t},\dots,T_{N}=\overline{t}|t_{1},t_{2},\dots,t_{N})}$$&#xA;$$=\frac{&#xD;&#xA;\left[\prod_{i=1}^{N}\frac{1}{t_{i}}\right] exp\left(-\sum_{i=1}^{N}\frac{t_{i}}{t_{i}}\right)&#xD;&#xA;}{&#xD;&#xA;\left[\prod_{i=1}^{N}\frac{1}{t_{i}}\right] exp\left(-\sum_{i=1}^{N}\frac{\overline{t}}{t_{i}}\right)&#xD;&#xA;}&#xD;&#xA;=exp\left(N\left[\frac{\overline{t}}{t_{harm}}-1\right]\right)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$t_{harm}=\left[\frac{1}{N}\sum_{i=1}^{N}t_{i}^{-1}\right]^{-1}\leq \overline{t}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;is the harmonic mean.  Note that the odds will always favour the perfect fit, but not by much if the median survival times are reasonably close.  Further, this gives you a direct way to state the evidence of this particular hypothesis test: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;assumptions 1-3 give maximum odds of&lt;/em&gt; $O(H_{A}|H_{0}):1$ &lt;em&gt;against equal median survival times across all states&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Combine this with a decision rule, loss function, utility function, etc. which says how advantageous it is to accept the simpler hypothesis, and you've got your conclusion!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is no limit to the amount of hypothesis you can test for, and give similar odds for.  Just change $H_{0}$ to specify a different set of possible &quot;true values&quot;.  You could do &quot;significance testing&quot; by choosing the hypothesis as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$H_{S,i}:T_{i}=t_{i},T_{j}=T=\overline{t}_{(i)}=\frac{1}{N-1}\sum_{j\neq i}t_{j}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So this hypothesis is verbally &quot;state $i$ has different median survival rate, but all other states are the same&quot;.  And then re-do the odds ratio calculation I did above.  Although you should be careful about what the alternative hypothesis is.  For any one of these below is &quot;reasonable&quot; in the sense that they might be questions you are interested in answering (and they will generally have different answers)&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;my $H_{A}$ defined above - how much worse is $H_{S,i}$ compared to the perfect fit?&lt;/li&gt;&#xA;&lt;li&gt;my $H_{0}$ defined above - how much better is $H_{S,i}$ compared to the average fit?&lt;/li&gt;&#xA;&lt;li&gt;a different $H_{S,k}$ - how much is state $k$ &quot;more different&quot; compared to state $i$?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Now one thing which has been over-looked here is &lt;em&gt;correlations&lt;/em&gt; between states - this structure assumes that knowing the median survival rate in one state tells you nothing about the median survival rate in another state.  While this may seem &quot;bad&quot; it is not to difficult to improve on, and the above calculations are good initial results which are easy to calculate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Adding connections between states will change the probability models, and you will effectively see some &quot;pooling&quot; of the median survival times.  One way to incorporate correlations into the analysis is to separate the true survival times into two components, a &quot;common part&quot; or &quot;trend&quot; and an &quot;individual part&quot;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$T_{i}=T+U_{i}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And then constrain the individual part $U_{i}$ to have average zero over all units and unknown variance $\sigma$ to be integrated out using a prior describing what knowledge you have of the individual variability, prior to observing the data (or jeffreys prior if you know nothing, and half cauchy if jeffreys causes problems).&lt;/p&gt;&#xA;" OwnerUserId="2392" LastEditorUserId="2392" LastEditorDisplayName="" LastEditDate="2011-04-02T23:46:01.627" LastActivityDate="2011-04-02T23:46:01.627" CommentCount="4" />
  <row Id="9098" PostTypeId="2" ParentId="5690" CreationDate="2011-04-02T17:07:50.450" Score="2" ViewCount="" Body="&lt;p&gt;First I would visualize the data: calculate confidence intervals and standard errors for the median survivals in each state and show CIs on a forest plot, medians and their SEs using a funnel plot. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The “mean median survival all across the country” is a quantity that is estimated from the data and thus has uncertainty so you can not take it as a sharp reference value during significance testing. An other difficulty with the mean-of-all approach is that when you compare a state median to it you are comparing the median to a quantity that already includes that quantity as a component. So it is easier to compare each state to all &lt;em&gt;other&lt;/em&gt; states combined. This can be done by performing a log rank test (or its alternatives) for each state.&lt;br&gt; &lt;em&gt;(Edit after reading the answer of probabilityislogic: the log rank test does compare survival in two (or more) groups, but it is not strictly the median that it is comparing. If you are sure it is the median that you want to compare, you may rely on his equations or use resampling here, too)&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You labelled your question [multiple comparisons], so I assume you also want to adjust (increase) your p values in a way that if you see at least one adjusted p value less than 5% you could conclude that “median survival across states is not equal” at the 5% significance level. You may use generic and overly conservative methods like Bonferroni, but the optimal correction scheme will take the correlations of the p values into consideration. I assume that you don't want to build any a priori knowledge into the correction scheme, so I will discuss a scheme where the adjustment is multiplying each p value by the same C constant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As I don't know how to derive the formula to obtain the optimal C multiplyer, I would use &lt;a href=&quot;http://en.wikipedia.org/wiki/Resampling_%28statistics%29&quot; rel=&quot;nofollow&quot;&gt;resampling&lt;/a&gt;. Under the null hypothesis that the survival characteristics are the same across all states, so you can permutate the state labels of the cancer cases and recalculate medians. After obtaining many resampled vectors of state p values I would numerically find the C multiplyer below which less than 95% of the vectors include no significant p values and above which more then 95%. While the range looks wide I would repeatedly increase the number of resamples by an order of magnitude.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-03T01:21:43.877" LastActivityDate="2011-04-03T01:21:43.877" CommentCount="3" />
  <row Id="9099" PostTypeId="1" CreationDate="2011-04-02T17:18:59.177" Score="3" ViewCount="59" Body="&lt;p&gt;Diallel Analysis using Griffing and Hayman approach is so common in plant breeding and genetics. I'm wondering if someone can share R worked example on Diallel Analysis. Is there any good referenced book which covered worked examples? Thanks&lt;/p&gt;&#xA;" OwnerUserId="3903" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-02T18:40:14.237" LastActivityDate="2011-04-02T18:40:14.237" Title="How to perform diallel analysis in R?" Tags="&lt;r&gt;&lt;experiment-design&gt;&lt;genetics&gt;" CommentCount="1" FavoriteCount="1" />
  <row Id="9100" PostTypeId="2" ParentId="9053" CreationDate="2011-04-02T17:40:55.537" Score="2" ViewCount="" Body="&lt;p&gt;From a Bayesian perspective, I'm not so sure that cross validation does anything that a &quot;proper&quot; Bayesian analysis doesn't do for comparing models.  But I am not 100% certain that it does.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is because if you are comparing models in a Bayesian way, then you are essentially already doing cross validation.  This is because the posterior odds of model A $M_A$ against model B $M_B$, with data $D$ and prior information $I$ has the following form:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{P(M_A|D,I)}{P(M_B|D,I)}=\frac{P(M_A|I)}{P(M_B|I)}\times\frac{P(D|M_A,I)}{P(D|M_B,I)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And $P(D|M_A,I)$ is given by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(D|M_A,I)=\int P(D,\theta_A|M_A,I)d\theta_A=\int P(\theta_A|M_A,I)P(D|M_A,\theta_A,I)d\theta_A$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which is called the &lt;em&gt;prior predictive distribution&lt;/em&gt;.  It basically says how well the model predicted the data that was actually observed, which is exactly what cross validation does, with the &quot;prior&quot; being replaced by the &quot;training&quot; model fitted, and the &quot;data&quot; being replace by the &quot;testing&quot; data.  So if model B predicted the data better than model A, its posterior probability increases relative to model A.  It seems from this that Bayes theorem will actually do cross validation using all the data, rather than a subset.  However, I am not fully convinced of this - seems like we get something for nothing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another neat feature of this method is that it has an in built &quot;occam's razor&quot;, given by the ratio of normalisation constants of the prior distributions for each model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However cross validation seems valuable for the dreaded old &quot;something else&quot; or what is sometimes called &quot;model mispecification&quot;.  I am constantly torn by whether this &quot;something else&quot; matters or not, for it seems like it should matter - but it leaves you paralyzed with no solution at all when it apparently matters.  Just something to give you a headache, but nothing you can do about it - except for thinking of what that &quot;something else&quot; might be, and trying it out in your model (so that it is no longer part of &quot;something else&quot;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And further, cross validation is a way to actually do a Bayesian analysis when the integrals above are ridiculously hard.  And cross validation &quot;makes sense&quot; to just about anyone - it is &quot;mechanical&quot; rather than &quot;mathematical&quot;.  So it is easy to understand what is going on.  And it also seems to get your head to focus on the important part of models - making good predictions.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-04-02T17:40:55.537" CommentCount="24" />
  <row Id="9101" PostTypeId="2" ParentId="8567" CreationDate="2011-04-02T17:53:53.967" Score="0" ViewCount="" Body="&lt;p&gt;I do not know the answer, so I can only offer some thoughts hoping that someone else can throw light on the issue.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems to me that there is no problem in computing the likelihood. To compute the value of the AIC criterion (which may or may not make sense in this context), what would be required is the number of fitted parameters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is were things become slippery. To take the simplest example, the (univariate) local level model requires fitting two parameters (the variances of the state and of the noise), but those are really best described as metaparameters. If the variance of the state is zero, you are fitting a single mean (1 parameter). If the variance of the state goes to infinity, you are effectively fitting one parameter per observation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One way out is to define the &quot;number of equivalent parameters&quot; as in,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hodges, J. S. and Sargent, D. J. (2001) Counting Degrees of Freedom in &#xA;        Hierarchical and Other Richly-Parameterised Models, &lt;em&gt;Biometrika,&lt;/em&gt; &lt;strong&gt;88&lt;/strong&gt;(2),&#xA;        p. 367-379.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and this is what I have done in my own work (on purely heuristic grounds and with some trepidation!). See for instance,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.et.bs.ehu.es/~etptupaf/nuevo/ficheros/papiros/riw.pdf&quot; rel=&quot;nofollow&quot;&gt;Pérez-Castroviejo, P. and Tusell, F. (2007) Using redundant and incomplete time series for the estimation of cost of living indices, &lt;em&gt;Review of Income and Wealth&lt;/em&gt;, vol. &lt;strong&gt;53&lt;/strong&gt;, p. 673-691.&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are alternative ways of calculating &quot;equivalent parameters&quot;; if you want to follow this route I can give you some references.&lt;/p&gt;&#xA;" OwnerUserId="892" LastActivityDate="2011-04-02T17:53:53.967" />
  <row Id="9102" PostTypeId="2" ParentId="8898" CreationDate="2011-04-02T19:09:55.037" Score="4" ViewCount="" Body="&lt;p&gt;As it is described in the original post, the experiment is a randomized block.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Pathologist (4 levels) is a blocking factor; the experiment is repeated within each pathologist.&lt;/li&gt;&#xA;&lt;li&gt;Instrument (3 levels) and the true result (2 levels) of the test are the two treatments, which I assume were assigned randomly.&lt;/li&gt;&#xA;&lt;li&gt;Consider the different specimens to be replications of each treatment combination.&lt;/li&gt;&#xA;&lt;li&gt;The one response variable is whether the pathologist's diagnosis is correct (2 levels).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Because the result is categorical, the link function will need to be something like logit or probit. Here's some R code that does that. It may need to be extended depending on your friend's hypotheses.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(lme4)&#xA;glmer(correctness ~ instrument*trueresult + (1|pathologist),&#xA;  family = binomial)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The coefficients from a logit model can be interpreted in relation to odds ratios. For a particular combination of predictors, the model estimates an odds ratio. The individual coefficients indicate how the odds ratio changes depending on the predictors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your friend doesn't care about distinguishing between type I and type II error, he or she can drop true result predictor from the model.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(lme4)&#xA;glmer(correctness ~ instrument+trueresult + (1|pathologist),&#xA;  family = binomial)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The measurement in multiple sessions may be an incomplete block design, so your friend should look at those if he or she is concerned about the assumption of independence among measurements.&lt;/p&gt;&#xA;" OwnerUserId="3874" LastEditorUserId="3874" LastEditorDisplayName="" LastEditDate="2011-04-24T17:14:47.580" LastActivityDate="2011-04-24T17:14:47.580" />
  <row Id="9103" PostTypeId="2" ParentId="9067" CreationDate="2011-04-02T20:36:24.113" Score="0" ViewCount="" Body="&lt;p&gt;Hard for me to think of this as a huge dataset, but these things are relative ;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To do this in excel (a version like 2007 or 2010 which has enough columns), you could insert two rows at the top.  In the first row, just have consecutive integers: 1 in column A, 2 in column B, etc.  You can do this with a function [I think it's col()] and then convert it to values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the second row, you need some logical expression which will tell you if the column is empty. This might be as simple as +max(a3:A12002)&amp;lt;&gt;0 if all variables which are non-empty contain a nonzero value. Without knowing the characteristics of your data I can't be sure that will work.  +max(a3:A12002)&amp;lt;&gt;0 should be true if there is data in the column, false if it isn't.  Now, use sort to sort columns based on row 2, true/false.  Next, delete all the false columns. Next, sort on row 1 to put the data back in the original order. Finally, delete rows 1 and 2 and save the file (probably as .CSV).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm writing this from memory; should be right.&lt;/p&gt;&#xA;" OwnerUserId="3919" LastActivityDate="2011-04-02T20:36:24.113" CommentCount="1" />
  <row Id="9104" PostTypeId="1" AcceptedAnswerId="9161" CreationDate="2011-04-02T20:42:02.777" Score="3" ViewCount="211" Body="&lt;p&gt;I have the following question for a course I'm working on:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Conduct a Monte Carlo study to estimate the coverage probabilities of &#xA;  the standard normal bootstrap confidence interval and the basic bootstrap confidence&#xA;  interval. Sample from a normal population and check the empirical coverage rates for the&#xA;  sample mean. &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Coverage probabilities for the standard normal bootstrap CI are easy:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;n = 1000;&#xA;alpha = c(0.025, 0.975);&#xA;x = rnorm(n, 0, 1);&#xA;mu = mean(x);&#xA;sqrt.n = sqrt(n);&#xA;&#xA;LNorm = numeric(B);&#xA;UNorm = numeric(B);&#xA;&#xA;for(j in 1:B)&#xA;{&#xA;    smpl = x[sample(1:n, size = n, replace = TRUE)];&#xA;    xbar = mean(smpl);&#xA;    s = sd(smpl);&#xA;&#xA;    LNorm[j] = xbar + qnorm(alpha[1]) * (s / sqrt.n);&#xA;    UNorm[j] = xbar + qnorm(alpha[2]) * (s / sqrt.n);&#xA;}&#xA;&#xA;mean(LNorm &amp;lt; 0 &amp;amp; UNorm &amp;gt; 0); # Approximates to 0.95&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;From what I've been taught for this course, the basic bootstrap &lt;em&gt;confidence interval&lt;/em&gt; can be calculated like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Using x from previous...&#xA;R = boot(data = x, R=1000, statistic = function(x, i){ mean(x[i]); });&#xA;result = 2 * mu - quantile(R$t, alpha, type=1);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;That makes sense. What I don't understand is how to calculate &lt;em&gt;coverage probabilities&lt;/em&gt; for the basic bootstrap CI. I understand that the coverage probability would represent the number of times that the CI contains the true value (in this case &lt;code&gt;mu&lt;/code&gt;). Do I simply run the &lt;code&gt;boot&lt;/code&gt; function many times?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I approach this question differently?&lt;/p&gt;&#xA;" OwnerUserId="1894" LastEditorUserId="1894" LastEditorDisplayName="" LastEditDate="2011-04-03T21:47:51.833" LastActivityDate="2011-04-04T15:36:29.233" Title="Coverage probabilities of the basic bootstrap confidence Interval" Tags="&lt;r&gt;&lt;confidence-interval&gt;&lt;homework&gt;&lt;bootstrap&gt;&lt;monte-carlo&gt;" AnswerCount="1" CommentCount="3" FavoriteCount="1" />
  <row Id="9105" PostTypeId="1" CreationDate="2011-04-02T21:57:45.523" Score="9" ViewCount="144" Body="&lt;p&gt;for the following 3 values 222,1122,45444&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.wolframalpha.com/input/?i=skew+222%2C1122%2C45444+&quot; rel=&quot;nofollow&quot;&gt;WolframAlpha gives&lt;/a&gt; 0.706&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Excel, using &lt;code&gt;=SKEW(222,1122,45444)&lt;/code&gt; gives 1.729&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What explains the difference?&lt;/p&gt;&#xA;" OwnerUserId="276" LastActivityDate="2011-04-02T22:07:09.333" Title="Why do Excel and WolframAlpha give different values for skewness" Tags="&lt;descriptive-statistics&gt;" AnswerCount="1" />
  <row Id="9106" PostTypeId="2" ParentId="9105" CreationDate="2011-04-02T22:07:09.333" Score="12" ViewCount="" Body="&lt;p&gt;They are using different methods to compute the skew. Searching in the help pages for &lt;code&gt;skewness()&lt;/code&gt; within the R package &lt;code&gt;e1071&lt;/code&gt; yields:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Joanes and Gill (1998) discuss three methods for estimating skewness:&#xA;&#xA;Type 1:&#xA;g_1 = m_3 / m_2^(3/2). This is the typical definition used in many older textbooks.&#xA;Type 2:&#xA;G_1 = g_1 * sqrt(n(n-1)) / (n-2). Used in SAS and SPSS.&#xA;Type 3:&#xA;b_1 = m_3 / s^3 = g_1 ((n-1)/n)^(3/2). Used in MINITAB and BMDP.&#xA;All three skewness measures are unbiased under normality.&#xA;&#xA;#Why are these numbers different?&#xA;&amp;gt; skewness(c(222,1122,45444), type = 2)&#xA;[1] 1.729690&#xA;&amp;gt; skewness(c(222,1122,45444), type = 1)&#xA;[1] 0.7061429&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here's a link to the paper referenced if someone has the credentials to get it for further discussion or education: &lt;a href=&quot;http://onlinelibrary.wiley.com/doi/10.1111/1467-9884.00122/abstract&quot; rel=&quot;nofollow&quot;&gt;http://onlinelibrary.wiley.com/doi/10.1111/1467-9884.00122/abstract&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="696" LastActivityDate="2011-04-02T22:07:09.333" CommentCount="5" />
  <row Id="9107" PostTypeId="1" AcceptedAnswerId="9108" CreationDate="2011-04-02T22:08:57.993" Score="4" ViewCount="78" Body="&lt;p&gt;I would like to plot the attached &lt;a href=&quot;http://i.imgur.com/acNhK.jpg&quot; rel=&quot;nofollow&quot;&gt;income distribution dataset&lt;/a&gt; (rendered as an image) as an area chart.&#xA;As you can see, personal income is divided into 26 intervals of varying width. I also have the average and mean income in the intervals.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To convey a truthful area graphic of this data, I wonder what my options really are?&lt;br&gt;&#xA;Plotting the ordinal categorical data at hand would yield a big hump in the area chart for the 400-499 interval. But this is only because that interval is wider and the user can hence be misguided by the shape. Another issue with the categorical data is that the average of the &quot;1000+ interval&quot; is very far from 1000 (= 1644). An area graphic not taking this into account would do a bad job in showing the actual distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How would you go about and is there any way in which I can use the average/mean to &quot;convert the categorical scale to a continuous scale&quot;?&lt;/p&gt;&#xA;" OwnerUserId="4003" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-03T09:10:18.443" LastActivityDate="2011-04-03T12:15:46.313" Title="Categorical or continuous scale for area chart?" Tags="&lt;distributions&gt;&lt;data-visualization&gt;&lt;data-transformation&gt;&lt;categorical-data&gt;" AnswerCount="2" />
  <row Id="9108" PostTypeId="2" ParentId="9107" CreationDate="2011-04-02T22:43:06.963" Score="4" ViewCount="" Body="&lt;p&gt;This is not exactly what you asked, but still may be helpful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may rely on the &lt;code&gt;plot.histogram&lt;/code&gt; command in R. The usual use is to run a &lt;code&gt;hist&lt;/code&gt; command, which prepares an object of class &lt;code&gt;histogram&lt;/code&gt; and passes it to &lt;code&gt;plot.histogram&lt;/code&gt;. You may prepare a customized &lt;code&gt;histogram&lt;/code&gt; object yourself and plot it with &lt;code&gt;plot.histogram&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The following code prints a &lt;code&gt;histogram&lt;/code&gt; object:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data(cars); dput(hist(cars$dist, plot=FALSE))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can make a similar object, and plot it:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;k = structure(list(breaks = c(0, 5, 10, 15, 25, 50, 75), counts = NULL,&#xA;  intensities = NULL, density = c(0.01, 0.018, 0.011, 0.006, 0.004, &#xA;  0.001), mids = NULL, xname = &quot;dist&quot;, equidist = FALSE), .Names = c(&quot;breaks&quot;, &#xA;  &quot;counts&quot;, &quot;intensities&quot;, &quot;density&quot;, &quot;mids&quot;, &quot;xname&quot;, &quot;equidist&quot;&#xA;  ), class = &quot;histogram&quot;)&#xA;plot(k)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/l3KA4.png&quot; alt=&quot;tweaked histogram&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="3911" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-03T12:15:46.313" LastActivityDate="2011-04-03T12:15:46.313" />
  <row Id="9109" PostTypeId="1" CreationDate="2011-04-02T22:51:25.520" Score="1" ViewCount="87" Body="&lt;p&gt;I have finally been able to wrap my head around the mechanics of how to initialize and train a multivariate Gaussian mixture model using expectation maximization algorithm. So I wonder how difficult this GMM and EM task is in comparison to all other &lt;strong&gt;common&lt;/strong&gt; algorithms and models in machine learning. I appreciate any feedback. Thank you.&lt;/p&gt;&#xA;" OwnerUserId="2729" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-03T09:19:38.667" LastActivityDate="2011-04-04T06:35:27.023" Title="How difficult is it to train a gaussian mixture model compared to other models?" Tags="&lt;mixed-model&gt;&lt;normal-distribution&gt;&lt;maximum-likelihood&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="9110" PostTypeId="2" ParentId="9085" CreationDate="2011-04-02T22:53:03.193" Score="1" ViewCount="" Body="&lt;p&gt;Anyone who answers R, or any of it's &quot;GUIs&quot; didn't read the question. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a program specifically designed for this and it's called JMP. Yes, it's expensive, though it has a free trial, and is incredibly cheap for students or college staff (like $50 cheap). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is also RapidMiner, which is a workflow-based GUI for data mining and statistical analysis. It's free and open source. &lt;/p&gt;&#xA;" OwnerUserId="74" LastActivityDate="2011-04-02T22:53:03.193" CommentCount="8" />
  <row Id="9111" PostTypeId="1" AcceptedAnswerId="9112" CreationDate="2011-04-02T19:10:34.457" Score="1" ViewCount="57" Body="&lt;p&gt;I am assuming R has this built-in. How do I reference it?&lt;/p&gt;&#xA;" OwnerUserId="4008" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-04-08T20:42:13.073" LastActivityDate="2011-04-08T20:42:13.073" Title="In R how do I reference\lookup in the cdf of standard normal distribution table?" Tags="&lt;r&gt;&lt;normal-distribution&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="9112" PostTypeId="2" ParentId="9111" CreationDate="2011-04-02T20:21:03.230" Score="3" ViewCount="" Body="&lt;p&gt;The functions you are looking for are either &lt;code&gt;dnorm&lt;/code&gt;, &lt;code&gt;pnorm&lt;/code&gt; or &lt;code&gt;qnorm&lt;/code&gt;, depending on exactly what you are looking for.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;dnorm(x)&lt;/code&gt; gives the density function at &lt;code&gt;x&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;pnorm(x)&lt;/code&gt; gives the probability that a random value is less than &lt;code&gt;x&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;qnorm(p)&lt;/code&gt; is the inverse of &lt;code&gt;pnorm&lt;/code&gt;, giving the value of &lt;code&gt;x&lt;/code&gt; for which getting a random value less than &lt;code&gt;x&lt;/code&gt; has probability &lt;code&gt;p&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See the help page for these functions to see how to change the parameters and values.&lt;/p&gt;&#xA;" OwnerUserId="72" LastEditorUserId="72" LastEditorDisplayName="" LastEditDate="2011-04-03T01:43:38.397" LastActivityDate="2011-04-03T01:43:38.397" CommentCount="2" />
  <row Id="9114" PostTypeId="2" ParentId="9109" CreationDate="2011-04-03T01:42:41.927" Score="3" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://www.uv.es/~bernardo/Kernel.pdf&quot; rel=&quot;nofollow&quot;&gt;this paper&lt;/a&gt; is a small gem about fitting mixtures of gaussians to the data using Bayesian methods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Big plus: no maximising algorithms required, so should be quick compared to EM algorithm.  Also paper has testing data, so you can test each method against this one if you want.  And it outlines a procedure for choosing the number of mixing components&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The drawback is that it is based on a non-informative prior (so you must throw away info which isn't data), and a common variance (or kernel width).&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-04-03T01:42:41.927" CommentCount="4" />
  <row Id="9115" PostTypeId="2" ParentId="9107" CreationDate="2011-04-03T01:51:49.713" Score="2" ViewCount="" Body="&lt;p&gt;A histogram with a continuous scale as described by GaBorgulya is clearly the way to go. When the blocks are wider, you need to adjust the density appropriately: the block 380-399 with 42246 people should be about 1.6 times the density of the block 400-499 with 132485 people.     &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Except for the extremes of 0 and 1000+, you can just use the blocks you have, with the densities (number of people divided by width of block) as height.  You can get even closer to the distribution by dividing each block at the medians: so for example you have 58700 in the interval &quot;600 to 799 tkr&quot; (i.e to just under 800), for a density of 293.5.  Or you could divide this at the median of 681.3 into two blocks representing 29350 each, to have an interval from 600 to 672.6 of density 404.3 and an interval from 672.6 to 800 of density 230.4.  You could go further and also take into account the means in each interval, but I don't think it is a priority.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The extreme of 1000+ (23143 people, median 1281.0, mean 1644.2) is slightly harder but you can use the median to give you an interval from 1000 to 1281 with density 41.2.  Now it is worth using the mean.  You could for example have the top interval from 1281 to 3014.8&#xA;with density 6.7.  This is not realistic as the maximum income is likely to be higher than 3014.8 and the curve is likely to be decreasing rather than flat, but it does illustrate the issue. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Illustrating the extreme at 0 is even harder.  Depending on how wide you make the block, you can have a spike there as high as you like.  Here is an example from &lt;em&gt;&lt;a href=&quot;http://research.dwp.gov.uk/asd/hbai/hbai_2009/pdf_files/full_hbai10.pdf&quot; rel=&quot;nofollow&quot;&gt;Households Below Average Income&lt;/a&gt;&lt;/em&gt; where they used £10 blocks.  It has other design features, some of which you may find interesting, such as a cutoff at the top end and words describing how many were cut off.&#xA;&lt;img src=&quot;http://i.imgur.com/4zgVm.png&quot; alt=&quot;HBAI 2008/09 figure 2.1&quot;&gt;  &lt;/p&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-04-03T01:51:49.713" CommentCount="2" />
  <row Id="9116" PostTypeId="1" AcceptedAnswerId="9118" CreationDate="2011-04-03T03:04:46.487" Score="0" ViewCount="60" Body="&lt;p&gt;Hi all,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First off, be warned: I am a complete stats novice. I'd like to learn, but at the moment I have an intense business problem to solve that I think (hope?) is straightforward enough that it could be answered easily. I will try to explain as simply as I can so I don't muck it up.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In short, i'm trying to find the right way to display a collection of pie charts. These pie charts are part of a proposal for new business; they measure incumbent capture across multiple contracts from different teaming partners.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Data I'm working with:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Company A: [a1] percentage of capture across [b1] contracts&lt;/li&gt;&#xA;&lt;li&gt;Company B: [a2] percentage of capture across [b2] contracts&lt;/li&gt;&#xA;&lt;li&gt;Company C: [a3] percentage of capture across [b3] contracts&lt;/li&gt;&#xA;&lt;li&gt;Total: [aT] percentage of capture across [bT] contracts&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I have the data for sets 1, 2, and 3 -- the goal is to derive the data for set &quot;T&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Essentially, we'd like to say &quot;business A had 30% capture across 100 contracts, business B had 50% against 70 contracts, and business C had 40% against 500 contracts&quot; the individual pie charts are of course simple enough (even I can do that one ;) ), but I would like them all to feed into a pie chart which shows the average percentage across all contracts with the total of contracts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To try to articulate more bluntly, my question is: how can I ensure that the individual percentages for company performance are weighted according to the number of contracts they are referencing?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you in advance for any help you can give! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;All the best,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;--Sean&lt;/p&gt;&#xA;" OwnerUserId="4004" LastEditorUserId="4004" LastEditorDisplayName="" LastEditDate="2011-04-03T03:18:29.550" LastActivityDate="2011-04-03T03:18:51.280" Title="How to properly weight contributing percentages?" Tags="&lt;pie-chart&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="9117" PostTypeId="2" ParentId="9085" CreationDate="2011-04-03T03:07:13.240" Score="4" ViewCount="" Body="&lt;p&gt;I can recommend Tableau as a good tool for data exploration and visualization, simply because of the different ways that you can explore and view the data, simply by dragging and dropping.  The graphs are fairly sharp and you can easily output to PDF for presentation purposes. If you want you can extend it with some &quot;programming&quot;. I regularly use this tool along with &quot;R&quot; and SAS and they all work together well. &lt;/p&gt;&#xA;" OwnerUserId="3489" LastActivityDate="2011-04-03T03:07:13.240" />
  <row Id="9118" PostTypeId="2" ParentId="9116" CreationDate="2011-04-03T03:18:51.280" Score="1" ViewCount="" Body="&lt;p&gt;It might be easier than I thought -- please let me know if this is correct or on the right path.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's assume the following data:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Company A: 70 percentage of capture across 10 contracts&lt;/li&gt;&#xA;&lt;li&gt;Company B: 60 percentage of capture across 100 contracts&lt;/li&gt;&#xA;&lt;li&gt;Company C: 50 percentage of capture across 40 contracts&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;First, I figured out the percentage of each contractors contribution to the total amount of contracts, so:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Company A: 6.67% of total contracts&lt;/li&gt;&#xA;&lt;li&gt;Company B: 66.67% of total contracts&lt;/li&gt;&#xA;&lt;li&gt;Company C: 26.67% of total contracts&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I then took the company's capture rate and multiplied it by their percentage of total contracts, so:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Company A: 4.67% contributing to total percentage&lt;/li&gt;&#xA;&lt;li&gt;Company B: 40% contributing to total percentage&lt;/li&gt;&#xA;&lt;li&gt;Company C: 13.33% of contributing to total percentage&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Which makes the total:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;58.00% across 150 contracts&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Please let me know if this is correct (or close enough). Thanks!&lt;/p&gt;&#xA;" OwnerUserId="4004" LastActivityDate="2011-04-03T03:18:51.280" CommentCount="3" />
  <row Id="9119" PostTypeId="2" ParentId="8567" CreationDate="2011-04-03T04:34:24.763" Score="1" ViewCount="" Body="&lt;p&gt;The dlmMLE function in the dlm package will compute the likelihood. Then &lt;/p&gt;&#xA;&#xA;&lt;p&gt;AIC = -2 log(likelihood) + 2p &lt;/p&gt;&#xA;&#xA;&lt;p&gt;where p is the number of parameters estimated. You might like to read the &lt;a href=&quot;http://cran.r-project.org/web/packages/dlm/vignettes/dlm.pdf&quot; rel=&quot;nofollow&quot;&gt;vignette for the dlm package&lt;/a&gt; which contains a lot of helpful information and examples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, a much simpler approach is to use the &lt;code&gt;StructTS&lt;/code&gt; function in the &lt;code&gt;stats&lt;/code&gt; package (which is automatically loaded). It will fit the model you want, and returns the loglikelihood.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2011-04-03T04:34:24.763" />
  <row Id="9120" PostTypeId="2" ParentId="2504" CreationDate="2011-04-03T05:03:28.207" Score="0" ViewCount="" Body="&lt;p&gt;In order to test such a vague hypothesis, you need to average out over all densities with finite variance, and all densities with infinite variance.  This is likely to be impossible, you basically need to be more specific.  One more specific version of this and have two hypothesis for a sample $D\equiv Y_{1},Y_{2},\dots,Y_{N}$:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;$H_{0}:Y_{i}\sim Normal(\mu,\sigma)$&lt;/li&gt;&#xA;&lt;li&gt;$H_{A}:Y_{i}\sim Cauchy(\nu,\tau)$&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;One hypothesis has finite variance, one has infinite variance.  Just calculate the odds:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{P(H_{0}|D,I)}{P(H_{A}|D,I)}=\frac{P(H_{0}|I)}{P(H_{A}|I)}\frac{\int P(D,\mu,\sigma|H_{0},I)d\mu d\sigma}{\int P(D,\nu,\tau|H_{A},I)d\nu d\tau}&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $\frac{P(H_{0}|I)}{P(H_{A}|I)}$ is the prior odds (usually 1)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(D,\mu,\sigma|H_{0},I)=P(\mu,\sigma|H_{0},I)P(D|\mu,\sigma,H_{0},I)$$&#xA;And&#xA;$$P(D,\nu,\tau|H_{A},I)=P(\nu,\tau|H_{A},I)P(D|\nu,\tau,H_{A},I)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now you normally wouldn't be able to use improper priors here, but because both densities are of the &quot;location-scale&quot; type, if you specify the standard non-informative prior with the same range $L_{1}&amp;lt;\mu,\tau&amp;lt;U_{1}$ and $L_{2}&amp;lt;\sigma,\tau&amp;lt;U_{2}$, then we get for the numerator integral:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{\left(2\pi\right)^{-\frac{N}{2}}}{(U_1-L_1)log\left(\frac{U_2}{L_2}\right)}\int_{L_2}^{U_2}\sigma^{-(N+1)}\int_{L_1}^{U_1} exp\left(-\frac{N\left[s^{2}-(\overline{Y}-\mu)^2\right]}{2\sigma^{2}}\right)d\mu d\sigma$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $s^2=N^{-1}\sum_{i=1}^{N}(Y_i-\overline{Y})^2$ and $\overline{Y}=N^{-1}\sum_{i=1}^{N}Y_i$.  And for the denominator integral:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{\pi^{-N}}{(U_1-L_1)log\left(\frac{U_2}{L_2}\right)}\int_{L_2}^{U_2}\tau^{-(N+1)}\int_{L_1}^{U_1} \prod_{i=1}^{N}\left(1+\left[\frac{Y_{i}-\nu}{\tau}\right]^{2}\right)^{-1}d\nu d\tau$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And now taking the ratio we find that the important parts of the normalising constants cancel and we get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{P(D|H_{0},I)}{P(D|H_{A},I)}=\left(\frac{\pi}{2}\right)^{\frac{N}{2}}\frac{\int_{L_2}^{U_2}\sigma^{-(N+1)}\int_{L_1}^{U_1} exp\left(-\frac{N\left[s^{2}-(\overline{Y}-\mu)^2\right]}{2\sigma^{2}}\right)d\mu d\sigma}{\int_{L_2}^{U_2}\tau^{-(N+1)}\int_{L_1}^{U_1} \prod_{i=1}^{N}\left(1+\left[\frac{Y_{i}-\nu}{\tau}\right]^{2}\right)^{-1}d\nu d\tau}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And all integrals are still proper in the limit so we can get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{P(D|H_{0},I)}{P(D|H_{A},I)}=\left(\frac{2}{\pi}\right)^{-\frac{N}{2}}\frac{\int_{0}^{\infty}\sigma^{-(N+1)}\int_{-\infty}^{\infty} exp\left(-\frac{N\left[s^{2}-(\overline{Y}-\mu)^2\right]}{2\sigma^{2}}\right)d\mu d\sigma}{\int_{0}^{\infty}\tau^{-(N+1)}\int_{-\infty}^{\infty} \prod_{i=1}^{N}\left(1+\left[\frac{Y_{i}-\nu}{\tau}\right]^{2}\right)^{-1}d\nu d\tau}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The denominator integral cannot be analytically computed, but the numerator can, and we get for the numerator:&#xA;$$\int_{0}^{\infty}\sigma^{-(N+1)}\int_{-\infty}^{\infty} exp\left(-\frac{N\left[s^{2}-(\overline{Y}-\mu)^2\right]}{2\sigma^{2}}\right)d\mu d\sigma=\sqrt{2N\pi}\int_{0}^{\infty}\sigma^{-N} exp\left(-\frac{Ns^{2}}{2\sigma^{2}}\right)d\sigma$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now make change of variables $\lambda=\sigma^{-2}\implies d\sigma = -\frac{1}{2}\lambda^{-\frac{3}{2}}d\lambda$ and you get a gamma integral:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$-\sqrt{2N\pi}\int_{\infty}^{0}\lambda^{\frac{N-1}{2}-1} exp\left(-\lambda\frac{Ns^{2}}{2}\right)d\lambda=\sqrt{2N\pi}\left(\frac{2}{Ns^{2}}\right)^{\frac{N-1}{2}}\Gamma\left(\frac{N-1}{2}\right)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And we get as a final analytic form for the odds for numerical work:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{P(H_{0}|D,I)}{P(H_{A}|D,I)}=\frac{P(H_{0}|I)}{P(H_{A}|I)}\times\frac{\pi^{\frac{N+1}{2}}N^{-\frac{N}{2}}s^{-(N-1)}\Gamma\left(\frac{N-1}{2}\right)}{\int_{0}^{\infty}\tau^{-(N+1)}\int_{-\infty}^{\infty} \prod_{i=1}^{N}\left(1+\left[\frac{Y_{i}-\nu}{\tau}\right]^{2}\right)^{-1}d\nu d\tau}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So this can be thought of as a specific test of finite versus infinite variance.  We could also do a T distribution into this framework to get another test (test the hypothesis that the degrees of freedom is greater than 2).&lt;/p&gt;&#xA;" OwnerUserId="2392" LastEditorUserId="2392" LastEditorDisplayName="" LastEditDate="2011-04-03T22:43:02.550" LastActivityDate="2011-04-03T22:43:02.550" CommentCount="2" />
  <row Id="9121" PostTypeId="1" AcceptedAnswerId="11424" CreationDate="2011-04-03T06:28:11.013" Score="1" ViewCount="45" Body="&lt;p&gt;I've seen couple of articles reporting individuals standard deviations and/or standard errors for groups even after implementing ANOVA. My understanding is that groups SE's should be based on experimental error mean square. Any comment?&lt;/p&gt;&#xA;" OwnerUserId="3903" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-06-01T10:11:22.020" LastActivityDate="2011-06-01T10:11:22.020" Title="Individuals standard deviations and/or standard errors for groups after implementing ANOVA?" Tags="&lt;anova&gt;&lt;standard-deviation&gt;&lt;experiment-design&gt;&lt;standard-error&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="9122" PostTypeId="2" ParentId="8586" CreationDate="2011-04-03T06:51:57.360" Score="0" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://bayes.wustl.edu/glb/diff.pdf&quot; rel=&quot;nofollow&quot;&gt;This paper&lt;/a&gt; gives a method you could use to test if variances are different.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-04-03T06:51:57.360" />
  <row Id="9124" PostTypeId="1" CreationDate="2011-04-03T11:06:21.313" Score="1" ViewCount="116" Body="&lt;p&gt;In S-plus estimates of percentiles for a survival function can be obtained using the &lt;code&gt;qkaplanMeier&lt;/code&gt; function (on the results of a call to kaplanMeier) like that:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;kfit &amp;lt;-kaplanMeier(censor(TIME,STATUS)~1)&#xA;qkaplanMeier(kfit, c(.25, .5, .75))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can I do this in R?. Those functions do not exist anymore. What if I also want the (asymptotic) confidence intervals for the percentiles? How can I get the mean for the survival time?&lt;/p&gt;&#xA;" OwnerUserId="339" LastEditorUserId="339" LastEditorDisplayName="" LastEditDate="2011-04-03T11:18:23.073" LastActivityDate="2011-04-03T12:44:55.260" Title="Estimates and C.I. of percentiles for a survival function" Tags="&lt;r&gt;&lt;confidence-interval&gt;&lt;survival&gt;" AnswerCount="2" />
  <row Id="9125" PostTypeId="2" ParentId="9124" CreationDate="2011-04-03T11:40:56.150" Score="2" ViewCount="" Body="&lt;p&gt;The &lt;a href=&quot;http://cran.r-project.org/web/views/Survival.html&quot; rel=&quot;nofollow&quot;&gt;CRAN task view on survival analysis&lt;/a&gt; says:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Kaplan-Meier:&lt;/strong&gt;&#xA;          The&#xA;    &lt;code&gt;survfit&lt;/code&gt;&#xA;          function from the&#xA;&lt;a href=&quot;http://cran.r-project.org/web/packages/survival/index.html&quot; rel=&quot;nofollow&quot;&gt;survival&lt;/a&gt;&#xA;         package &#xA;    computes the Kaplan-Meier estimator for truncated and/or censored data.&#xA;&lt;a href=&quot;http://cran.r-project.org/web/packages/rms/index.html&quot; rel=&quot;nofollow&quot;&gt;rms&lt;/a&gt;&#xA;          (replacement of the&#xA;&lt;a href=&quot;http://cran.r-project.org/web/packages/Design/index.html&quot; rel=&quot;nofollow&quot;&gt;Design&lt;/a&gt;&#xA;          package) &#xA;    proposes a modified version of the&#xA;&lt;code&gt;survfit&lt;/code&gt;&#xA;          function.&#xA;    The&#xA;&lt;a href=&quot;http://cran.r-project.org/web/packages/prodlim/index.html&quot; rel=&quot;nofollow&quot;&gt;prodlim&lt;/a&gt;&#xA;          package implements a fast algorithm and some features&#xA;    not included in&#xA;&lt;a href=&quot;http://cran.r-project.org/web/packages/survival/index.html&quot; rel=&quot;nofollow&quot;&gt;survival&lt;/a&gt;.&#xA;    Various confidence intervals and confidence bands for the Kaplan-Meier estimator &#xA;    are implemented in the&#xA;&lt;a href=&quot;http://cran.r-project.org/web/packages/km.ci/index.html&quot; rel=&quot;nofollow&quot;&gt;km.ci&lt;/a&gt;&#xA;          package.&#xA;&lt;code&gt;plot.Surv&lt;/code&gt;          of package&#xA;&lt;a href=&quot;http://cran.r-project.org/web/packages/eha/index.html&quot; rel=&quot;nofollow&quot;&gt;eha&lt;/a&gt;&#xA;          plots&#xA;    the Kaplan-Meier estimator.&#xA;&lt;code&gt;svykm&lt;/code&gt;&#xA;          in&#xA;&lt;a href=&quot;http://cran.r-project.org/web/packages/survey/index.html&quot; rel=&quot;nofollow&quot;&gt;survey&lt;/a&gt;&#xA;          provides a weighted Kaplan-Meier &#xA;    estimator.&#xA;&lt;code&gt;nested.km&lt;/code&gt;&#xA;          in&#xA;&lt;a href=&quot;http://cran.r-project.org/web/packages/NestedCohort/index.html&quot; rel=&quot;nofollow&quot;&gt;NestedCohort&lt;/a&gt;&#xA;          estimates the &#xA;    survival curve for each level of categorical variables with missing data.&#xA;    The&#xA;&lt;code&gt;kaplan-meier&lt;/code&gt;&#xA;          function in&#xA;&lt;a href=&quot;http://cran.r-project.org/web/packages/spatstat/index.html&quot; rel=&quot;nofollow&quot;&gt;spatstat&lt;/a&gt;&#xA;          computes the Kaplan-Meier estimator from histogram data.&#xA;    The&#xA;&lt;a href=&quot;http://cran.r-project.org/web/packages/MAMSE/index.html&quot; rel=&quot;nofollow&quot;&gt;MAMSE&lt;/a&gt;&#xA;          package permits to compute a weighted Kaplan-Meier estimate.&#xA;    The&#xA;&lt;code&gt;KM&lt;/code&gt;&#xA;          function in package&#xA;&lt;a href=&quot;http://cran.r-project.org/web/packages/rhosp/index.html&quot; rel=&quot;nofollow&quot;&gt;rhosp&lt;/a&gt;&#xA;          plots the survival &#xA;    function using a variant of the Kaplan-Meier estimator in a hospitalisation &#xA;    risk context.&#xA;    The&#xA;&lt;a href=&quot;http://cran.r-project.org/web/packages/survPresmooth/index.html&quot; rel=&quot;nofollow&quot;&gt;survPresmooth&lt;/a&gt;&#xA;          package computes presmoothed estimates of the main quantities used &#xA;    for right-censored data, i.e., survival, hazard and density functions.&lt;/p&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-04-03T11:40:56.150" />
  <row Id="9126" PostTypeId="2" ParentId="9124" CreationDate="2011-04-03T12:44:55.260" Score="1" ViewCount="" Body="&lt;p&gt;The &lt;code&gt;bootkm()&lt;/code&gt; function in &lt;a href=&quot;http://cran.r-project.org/web/packages/Hmisc/index.html&quot; rel=&quot;nofollow&quot;&gt;Hmisc&lt;/a&gt; provides bootstraped estimate of the probability of survival, as well as the estimate of the quantile of the survival distribution (through either &lt;code&gt;describe&lt;/code&gt; or &lt;code&gt;quantile&lt;/code&gt; applied onto the result of &lt;code&gt;bootkm&lt;/code&gt;).&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-04-03T12:44:55.260" />
  <row Id="9127" PostTypeId="1" CreationDate="2011-04-03T14:27:58.373" Score="2" ViewCount="84" Body="&lt;p&gt;Where can I obtain all hourly weather data available? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Notes: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Ideally, this would include the history of all data currently published at: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://weather.noaa.gov/pub/SL.us008001/DF.an/DC.sflnd/DS.synop/&quot; rel=&quot;nofollow&quot;&gt;http://weather.noaa.gov/pub/SL.us008001/DF.an/DC.sflnd/DS.synop/&lt;/a&gt; &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://weather.noaa.gov/pub/SL.us008001/DF.an/DC.sflnd/DS.metar/&quot; rel=&quot;nofollow&quot;&gt;http://weather.noaa.gov/pub/SL.us008001/DF.an/DC.sflnd/DS.metar/&lt;/a&gt; &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://weather.noaa.gov/pub/SL.us008001/DF.an/DC.sfmar/DS.dbuoy/&quot; rel=&quot;nofollow&quot;&gt;http://weather.noaa.gov/pub/SL.us008001/DF.an/DC.sfmar/DS.dbuoy/&lt;/a&gt; &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://weather.noaa.gov/pub/SL.us008001/DF.an/DC.sfmar/DS.ships/&quot; rel=&quot;nofollow&quot;&gt;http://weather.noaa.gov/pub/SL.us008001/DF.an/DC.sfmar/DS.ships/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I've obtained 10 years of METAR data from wunderground.com like this: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;curl -H 'Cookie: Prefs=|SHOWMETAR:1|;' -o data.txt 'http://www.wunderground.com/history/airport/KABQ/2005/05/02/DailyHistory.html?req_city=NA&amp;amp;req_state=NA&amp;amp;req_statename=NA&amp;amp;theprefset=SHOWMETAR&amp;amp;theprefvalue=1&amp;amp;format=1' &#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;but am sure there's a better way. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;I'd love per-minute/etc data too, though I don't think anyone kept &#xA;measurements better than hourly back then. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="1566" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-03T16:19:31.297" LastActivityDate="2011-04-03T16:19:31.297" Title="How to fetch all historically available hourly weather data?" Tags="&lt;time-series&gt;&lt;dataset&gt;" AnswerCount="1" FavoriteCount="2" />
  <row Id="9128" PostTypeId="2" ParentId="9085" CreationDate="2011-04-03T14:49:47.713" Score="2" ViewCount="" Body="&lt;p&gt;As John said, data exploration doesn't require much programming in R. Here's a list of data exploration commands you can give people. (I just came up with this; you can surely expand it.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Export the data from whatever package it's in. (Exporting numerical data without quotation marks is convenient.) Then read the data in R.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ChickWeight=read.csv('chickweight.csv')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Make a table.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;table(ChickWeight$Diet)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Let R guess what sort of graphic to give you. It sometimes works very nicely.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plot(ChickWeight)&#xA;plot(ChickWeight$weight)&#xA;plot(ChickWeight$weight~ChickWeight$Diet)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;A bunch of specific plotting functions work quite simply on single variables.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;hist(ChickWeight$weight)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Taking subsets&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plot(subset(ChickWeight,Diet=='2'))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;SQL-like syntax in case people are used to that (more &lt;a href=&quot;http://www.r-bloggers.com/make-r-speak-sql-with-sqldf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(sqldf)&#xA;plot(sqldf('select * from ChickWeight where Diet == &quot;2&quot;'))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;PCA (You'd have more than two variables of course.)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;princomp(~ ChickWeight$weight + ChickWeight$Time)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="3874" LastEditorUserId="3874" LastEditorDisplayName="" LastEditDate="2011-04-03T14:55:02.353" LastActivityDate="2011-04-03T14:55:02.353" />
  <row Id="9129" PostTypeId="1" CreationDate="2011-04-03T15:33:54.147" Score="4" ViewCount="96" Body="&lt;p&gt;I am currently looking for some Information Retrieval techniques.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a SQL database table containing strings. It has 1000 records, each being a random sentence I picked from random web sites. I need to get the &lt;em&gt;term frequency&lt;/em&gt; and represent each string into a vector. I also need to cluster the records, e.g. using k-means. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone know what is the best way to do this? Are there any tools I can use? I am new to this and looking for a jump off point.&lt;/p&gt;&#xA;" OwnerUserId="4020" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-03T16:26:29.463" LastActivityDate="2011-06-03T08:54:08.163" Title="How to compute term frequency and find clusters in a dataset composed of strings?" Tags="&lt;clustering&gt;&lt;information-retrieval&gt;" AnswerCount="1" CommentCount="3" FavoriteCount="2" />
  <row Id="9130" PostTypeId="2" ParentId="9127" CreationDate="2011-04-03T15:41:42.047" Score="1" ViewCount="" Body="&lt;p&gt;This may be simplistic, but if you have a consistent directory structure on the NOAA site (they usually are), you can recursive wget the entire thing, then sort through it at your leisure. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;wget -r http://weather.noaa.gov/pub/SL.us008001/DF.an/&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This will grab everything recursively from that URL and deeper. It's what I normally use when I want a huge whack of data from some (typically government site) and they do silly things like store it in 1-file-per-hour and I need 12 years worth. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an aside, I presume you only want historical weather data for the United States? If you're working on this on a slightly longer scale and more global, the Berkeley Earth Surface Temperature Project should be releasing their raw data set in the next few months: see &lt;a href=&quot;http://berkeleyearth.org/dataset&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="781" LastActivityDate="2011-04-03T15:41:42.047" CommentCount="1" />
  <row Id="9131" PostTypeId="1" CreationDate="2011-04-03T18:24:49.593" Score="7" ViewCount="189" Body="&lt;p&gt;Hello all,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's take the following example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(342)&#xA;x1 &amp;lt;- runif(100)&#xA;x2 &amp;lt;- runif(100)&#xA;y &amp;lt;- x1+x2 + 2*x1*x2 + rnorm(100)&#xA;fit &amp;lt;- lm(y~x1*x2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This creates a model of y based on x1 and x2, using a OLS regression.  If we wish to predict y for a given x_vec we could simply use the formula we get from the &lt;code&gt;summary(fit)&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, what if we want to predict the lower and upper predictions of y? (for a given confidence level).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How then would we build the formula?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="253" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-06T14:49:57.180" LastActivityDate="2011-04-09T14:56:49.983" Title="Obtaining a formula for prediction limits in a linear model" Tags="&lt;r&gt;&lt;regression&gt;&lt;predictive-models&gt;" AnswerCount="4" CommentCount="6" FavoriteCount="1" />
  <row Id="9132" PostTypeId="1" AcceptedAnswerId="9133" CreationDate="2011-04-03T20:02:37.747" Score="2" ViewCount="91" Body="&lt;p&gt;We are trying to predict values of variable A having N other variables. What we do, we calculate Pearson correlation between A and each of the other N variables, for last M values, using fixed M. We use variable with largest correlation coefficient as predictor.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This scheme works fine when analyzing N variables during 24 month and decreasing  between month 25-36.&lt;br&gt;&#xA;What could come as improvement of Pearson correlation in this context?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Variables are bi-valued.&lt;/p&gt;&#xA;" OwnerUserId="4010" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-03T20:39:22.490" LastActivityDate="2011-04-03T20:47:44.887" Title="Alternatives to Pearson correlation" Tags="&lt;correlation&gt;" AnswerCount="1" CommentCount="5" />
  <row Id="9133" PostTypeId="2" ParentId="9132" CreationDate="2011-04-03T20:47:44.887" Score="2" ViewCount="" Body="&lt;p&gt;This is generally hard to tell without knowing what the problem exactly is, but I would advise you to try some machine learning methods. For start you may try random forest, which is almost trivial to apply and quite probably will achieve better accuracy than just using one, best-correlated variable.&lt;br&gt;&#xA;Also, it will produce importance measure which will tell you which variables contribute most to the prediction accuracy -- possibly taking into account even quite complex multivariate interactions obviously invisible to Pearson's correlation.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2011-04-03T20:47:44.887" />
  <row Id="9134" PostTypeId="2" ParentId="9131" CreationDate="2011-04-03T21:04:33.583" Score="4" ViewCount="" Body="&lt;p&gt;Are you by chance after the different types of prediction intervals? The &lt;code&gt;predict.lm&lt;/code&gt; manual page has&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; ## S3 method for class 'lm'&#xA; predict(object, newdata, se.fit = FALSE, scale = NULL, df = Inf, &#xA;         interval = c(&quot;none&quot;, &quot;confidence&quot;, &quot;prediction&quot;),&#xA;         level = 0.95, type = c(&quot;response&quot;, &quot;terms&quot;),&#xA;         terms = NULL, na.action = na.pass,&#xA;         pred.var = res.var/weights, weights = 1, ...)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Setting&#xA;  ‘intervals’ specifies computation of confidence or prediction&#xA;  (tolerance) intervals at the specified ‘level’, sometimes referred&#xA;  to as narrow vs. wide intervals.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Is that what you had in mind?&lt;/p&gt;&#xA;" OwnerUserId="334" LastActivityDate="2011-04-03T21:04:33.583" CommentCount="5" />
  <row Id="9135" PostTypeId="1" CreationDate="2011-04-03T21:12:07.560" Score="1" ViewCount="51" Body="&lt;p&gt;Given two point sets, $B$ consisting of blue points, and $R$ of red points, on the plane.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is to formulate a theoretical model to compare the average runtime of the Computational Geometric (CG) Algorithm and the &lt;a href=&quot;http://en.wikipedia.org/wiki/Perceptron#Learning_algorithm&quot; rel=&quot;nofollow&quot;&gt; Perceptron Learning (PL) &lt;/a&gt; for shattering two sets of points by a line. ( Briefly, CG tests if there exists a line, $ax+by = c$ such that the system of equations $ax_r+by_r\leq c; ax_b+by_b \geq c$ for all $(x_b,y_b) \in B$ and $(x_r,y_r) \in R$  is consistent; if so, it finds one such.) AFAIK, only empirical models are known thus far, where we take a large number of such point sets and test out the performance of CG and PL. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;While I can't see any clear solution to this, I am looking for one along the following lines: Choose 2 random integers, $b$ and $r$ in the range $[1,n]$ where $n$ is a large integer, to appear in the answer as a parameter. Choose $b$ blue points and $r$ red points within the unit square. Compute the average runtime performance of CG (essentially determining the complexity of determining consistency of linear equations on a random set of values) and PL (essentially determining the number of weight changes on a random data set) by a multi-fold integration. Further, average across all possible choices of $b$ and $r$ in the range $[1,n]$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can you offer more insight or a better formulation?&lt;/p&gt;&#xA;" OwnerUserId="4011" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-04T07:15:49.620" LastActivityDate="2011-04-04T07:15:49.620" Title="Computational geometry vs perceptron for shattering" Tags="&lt;neural-networks&gt;" />
  <row Id="9136" PostTypeId="2" ParentId="8911" CreationDate="2011-04-03T21:31:49.947" Score="0" ViewCount="" Body="&lt;p&gt;Try &lt;a href=&quot;http://tukhi.com&quot; rel=&quot;nofollow&quot;&gt;http://tukhi.com&lt;/a&gt;. It is not clear whether or not they have a Mac OS X Excel version, but they have contact info on that site. It is pretty amazing. Heh, you could always run Window in a VM.&lt;/p&gt;&#xA;" OwnerUserId="4012" LastActivityDate="2011-04-03T21:31:49.947" CommunityOwnedDate="2011-04-03T21:31:49.947" />
  <row Id="9137" PostTypeId="1" AcceptedAnswerId="9139" CreationDate="2011-04-03T22:17:08.980" Score="3" ViewCount="63" Body="&lt;p&gt;Say there are 3 companies A, B and C. Each company has a quality rating from 0 to 100 and a price in USD.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Company    Quality    Price&#xA;A          80         7.9&#xA;B          70         8.0&#xA;C          75         8.1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How do I determine the best quality-price trade-off? What kind of analysis should I use?&lt;/p&gt;&#xA;" OwnerUserId="4013" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-04-08T20:41:37.837" LastActivityDate="2011-04-08T20:41:37.837" Title="Quality-price trade-off " Tags="&lt;weighted-mean&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="9139" PostTypeId="2" ParentId="9137" CreationDate="2011-04-03T23:04:03.450" Score="4" ViewCount="" Body="&lt;p&gt;The &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0521438837&quot; rel=&quot;nofollow&quot;&gt;Keeney-Raiffa approach to Multi-attribute valuation theory&lt;/a&gt; is well-grounded practically and theoretically, has been successfully applied to many problems, and--when applied to problems with just two attributes--is particularly simple.  It proceeds by systematically exploring the trade-offs you would actually make in hypothetical situations and uses those to deduce two things: (1) an appropriate way to re-express each attribute and (2) a linear combination of the re-expressed attributes that fully reflects an overall value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Be careful when doing Web research on this.  The vast majority of published models of this type appear to ignore (1), which is crucial, and often establish (2) in &lt;em&gt;ad-hoc&lt;/em&gt; or arbitrary ways.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another approach, consistent with (but inferior to) the Keeney-Raiffa theory, establishes an &quot;&lt;strong&gt;efficient frontier&lt;/strong&gt;.&quot;  Plot quality on one axis and price on another for each of the available alternatives.  If you do so with increasing quality to the right and &lt;em&gt;decreasing&lt;/em&gt; price upwards, then points lying at the extreme right or above of all the others are the best candidates to consider.  In effect this method ignores (1) and uses this &quot;frontier&quot; to avoid specifying the coefficients in (2).  It is often used in financial applications where the two attributes are &quot;alpha&quot; (expected rate of return) and &quot;beta&quot; (variance of returns, a surrogate for risk).  &lt;a href=&quot;http://en.wikipedia.org/wiki/Modern_portfolio_theory&quot; rel=&quot;nofollow&quot;&gt;Modern portfolio theory&lt;/a&gt; uses a variant of this approach.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-04-03T23:04:03.450" />
  <row Id="9140" PostTypeId="1" AcceptedAnswerId="9158" CreationDate="2011-04-04T01:09:10.603" Score="3" ViewCount="146" Body="&lt;p&gt;I have heard of survival analysis and life data analysis, but don't quite get the big picture.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was wondering what topics they are covering? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it pure statistics, or just application of statistics on some specific area?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is life date analysis part of survival analysis? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks and regards!&lt;/p&gt;&#xA;" OwnerUserId="1005" LastEditorUserId="1005" LastEditorDisplayName="" LastEditDate="2011-04-04T05:55:45.793" LastActivityDate="2011-04-10T16:02:11.417" Title="Big picture on survival analysis and life data analysis" Tags="&lt;survival&gt;&lt;mathematical-statistics&gt;&lt;life-data&gt;" AnswerCount="3" />
  <row Id="9142" PostTypeId="2" ParentId="9131" CreationDate="2011-04-04T01:15:10.187" Score="3" ViewCount="" Body="&lt;p&gt;@Tal: Might I suggest &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/007310874X&quot; rel=&quot;nofollow&quot;&gt;Kutner et al&lt;/a&gt; as a fabulous source for linear models. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is the distinction between 1) a prediction of Y from an individual new observation X_vec, 2) the expected value of a Y conditioned on X_vec, $E(Y|X_{vec})$ and 3) Y from several instances of x_vec  - all covered in detail in the text. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think you are looking for the formula for the confidence interval around $E(Y|X_{vec})$ and that is $\hat{Y}$ $\pm$ t(1-$\alpha$ /2)s{$\hat{Y}$} where t has n-2 d.f. and s{$\hat{Y}$} is the standard error of $\hat{Y}$, $\frac{\sigma^{2}}{n}$+($X_{vec}-\bar{X})^{2}\frac{\sigma^{2}}{\sum(X_{i}-\bar{X})^{2}}$&lt;/p&gt;&#xA;" OwnerUserId="2040" LastActivityDate="2011-04-04T01:15:10.187" CommentCount="1" />
  <row Id="9143" PostTypeId="2" ParentId="2504" CreationDate="2011-04-04T01:20:36.233" Score="1" ViewCount="" Body="&lt;p&gt;You cannot be certain without knowing the distribution.  But there are certain things you can do, such as looking at what might be called the &quot;partial variance&quot;, i.e. if you have a sample of size $N$, you draw the variance estimated from the first $n$ terms, with $n$ running from 2 to $N$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With a finite population variance, you hope that the partial variance soon settles down close to the population variance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With an infinite population variance, you see jumps up in the partial variance followed by slow declines until the next very large value appears in the sample.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is an illustration with Normal and Cauchy random variables (and a log scale)&#xA;&lt;img src=&quot;http://i.imgur.com/P2iOf.png&quot; alt=&quot;Partial Variance&quot;&gt;  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This may not help if the shape of your distribution is such that a much larger sample size than you have is needed to identify it with sufficient confidence, i.e. where very large values are fairly (but not extremely) rare for a distribution with finite variance, or are extremely rare for a distribution with infinite variance.  For a given distribution there will be sample sizes which are more likely than not to reveal its nature; conversely, for a given sample size, there are distributions which are more likely than not to disguise their natures for that size of sample.  &lt;/p&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-04-04T01:20:36.233" />
  <row Id="9144" PostTypeId="2" ParentId="9131" CreationDate="2011-04-04T01:44:16.487" Score="7" ViewCount="" Body="&lt;p&gt;You will need matrix arithmetic. I'm not sure how Excel will go with that. Anyway, here are the details.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose your regression is written as $\mathbf{y} = \mathbf{X}\mathbf{\beta} + \mathbf{e}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let $\mathbf{X}^*$ be a row vector containing the values of the predictors for the forecasts (in the same format as $\mathbf{X}$). Then the forecast is given by&#xA;$$&#xD;&#xA;\hat{y} = \mathbf{X}^*\hat{\mathbf{\beta}} = \mathbf{X}^*(\mathbf{X}&amp;#39;\mathbf{X})^{-1}\mathbf{X}&amp;#39;\mathbf{Y}&#xD;&#xA;$$&#xA;and its variance by&#xA;$$&#xD;&#xA;\sigma^2 \left[1 + \mathbf{X}^* (\mathbf{X}&amp;#39;\mathbf{X})^{-1} (\mathbf{X}^*)&amp;#39;\right].&#xD;&#xA;$$&#xA;Then a 95% prediction interval can be calculated (assuming normally distributed errors) as&#xA;$$&#xD;&#xA;\hat{y} \pm 1.96 \hat{\sigma} \sqrt{1 + \mathbf{X}^* (\mathbf{X}&amp;#39;\mathbf{X})^{-1} (\mathbf{X}^*)&amp;#39;}.&#xD;&#xA;$$&#xA;This takes account of the uncertainty due to the error term $e$ and the uncertainty in the coefficient estimates. However, it ignores any errors in $\mathbf{X}^*$. So if the future values of the predictors are uncertain, then the prediction interval calculated using this expression will be too narrow.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2011-04-04T01:44:16.487" CommentCount="3" />
  <row Id="9146" PostTypeId="2" ParentId="9085" CreationDate="2011-04-04T02:11:11.327" Score="2" ViewCount="" Body="&lt;p&gt;I program in Python for 95% of my work and the rest in R or MATLAB or IDL/PV-WAVE (and soon SAS). But, I am in an environment where time-to-results is often a huge driver of the analysis chosen and so I often use point-and-click tools as well. In my experience, there is no single, robust, flexible GUI tool for doing analytics, much like there is not a single language. I typically cobble together a collection of the following free and commercial software&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Weka&lt;/li&gt;&#xA;&lt;li&gt;KNIME&lt;/li&gt;&#xA;&lt;li&gt;Excel and its plugins (like Solver)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://alteryx.com/Pages/index.aspx&quot; rel=&quot;nofollow&quot;&gt;Alteryx&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://mvpprograms.com/&quot; rel=&quot;nofollow&quot;&gt;MVP Stats&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I have not used JMP, Stata, Statistica, etc, but would like to.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using these tools involves learning different GUIs and multiple abstractions of modeling, which is a pain at the time but let's me get faster ad hoc results later. I am in the same boat as the OP because while most of the folks I work with are really smart, they do not care to learn a language, nor multiple GUIs and application specific terminology. So, I have resigned myself to accepting that Excel drives 90% of analysis in the business world. Accordingly, I am looking in to using things like &lt;a href=&quot;http://code.google.com/p/pyinex/&quot; rel=&quot;nofollow&quot;&gt;pyinex&lt;/a&gt; to let me provide better analytics to the same Excel presentation layer that the vast majority of my colleagues expect.&lt;/p&gt;&#xA;" OwnerUserId="1080" LastActivityDate="2011-04-04T02:11:11.327" />
  <row Id="9147" PostTypeId="2" ParentId="5926" CreationDate="2011-04-04T02:55:43.847" Score="2" ViewCount="" Body="&lt;p&gt;There's no clear answer.  The idea is that if you have a correlation that approaches 1 then you essentially have one variable and not multiple variables.  So you could test against the hypotheses that r=1.00.  With that said, the idea of MANOVA is to give you something more than a series of ANOVA tests.  It helps you find a relationship with one test because you're able to lower your mean square error when combining dependent variables.  It just won't help if you have highly correlated dependent variables.&lt;/p&gt;&#xA;" OwnerUserId="4018" LastActivityDate="2011-04-04T02:55:43.847" />
  <row Id="9148" PostTypeId="1" CreationDate="2011-04-04T03:33:38.370" Score="2" ViewCount="59" Body="&lt;p&gt;I am looking around for datasets mainly in the form of natural language text corpus, that has been redacted by experts. So far I found only enron dataset that has been redacted ... but it appears only a few specific things(like employee mail-id etc) were redacted, which will not be beneficial for feature extraction. Something like declassified government records, company data would be awesome; medical records would probably be comparatively easier to get, but are quite unrelated to the problem. Any idea if such dataset is available?&lt;/p&gt;&#xA;" OwnerUserId="2192" LastEditorUserId="2192" LastEditorDisplayName="" LastEditDate="2011-04-04T12:33:23.960" LastActivityDate="2011-04-04T12:33:23.960" Title="Looking for redacted text corpus" Tags="&lt;dataset&gt;" CommentCount="3" />
  <row Id="9149" PostTypeId="2" ParentId="9085" CreationDate="2011-04-04T03:54:06.550" Score="1" ViewCount="" Body="&lt;p&gt;Well, this particular tool is popular in my industry (though it is not industry-specific by design):&#xA;&lt;a href=&quot;http://www.umetrics.com/simca&quot; rel=&quot;nofollow&quot;&gt;http://www.umetrics.com/simca&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It allows you to do latent variable type multivariate analysis (PCA and PLS), and it includes all the attendant interpretative plots / calculations and interrogation tools like contribution plots, variable importance plots, Q2 calculations etc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is often used on high-dimensional (and often highly correlated/collinear) industrial datasets where OLS/MLR type methods are unsuitable (e.g. info from a boatload of sensors, log info, etc.).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It operates in a fully GUI environment, and the user does not have to write a single line of code. Unfortunately it is not free, and cannot be extended via programming. &lt;/p&gt;&#xA;" OwnerUserId="2833" LastActivityDate="2011-04-04T03:54:06.550" />
  <row Id="9150" PostTypeId="2" ParentId="9085" CreationDate="2011-04-04T04:11:24.810" Score="0" ViewCount="" Body="&lt;p&gt;In my opinion, if you don't code yourself the test, you are prone to errors and misunderstandings of the results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think that you should recommend them to hire a statistician that has computer skills. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If it is to do always the same thing, then indeed you can use a small tool (blackbox) that will do the stuff. But I am not sure this is still called data exploration.&lt;/p&gt;&#xA;" OwnerUserId="1709" LastActivityDate="2011-04-04T04:11:24.810" CommentCount="1" />
  <row Id="9151" PostTypeId="2" ParentId="9140" CreationDate="2011-04-04T05:53:22.827" Score="6" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;About survival analysis&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In survival analysis, or time-to-event analysis, the variable or interest measures the time from a starting point to a point of interest like death due to some disease. So the response variable is a positive variable which is in most cases skewed. As a consequence the usual assumption of normality fails and, for instance, the classical regression techniques are not applicable. (Though, note that sometimes a transformation of the variable could make the situation better). But the main difference is &lt;strong&gt;censoring&lt;/strong&gt;: a very common feature when dealing with time-to-event data. In its most common form (right censoring), you do not know the exact time for a given individual but you do know that it is larger than some value $t^{\star}$. For example, suppose you follow a patient up to death. At time $t=10$ days, he is alive. At time $t=30$ days he is still alive but then he is lost to follow-up. Then you do not know the exact time of death but you do know that $t &amp;gt; 30$. Ignoring censoring is clearly not the best think to do; instead, you can record $t^{\star} = 30$ as a censored observation. Techniques of survival analysis (e.g., Kaplan-Meier estimator, Cox regression, ...) are specially designed to deal with censoring.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To my point of view &lt;a href=&quot;http://www.crcpress.com/product/isbn/9781584883258&quot; rel=&quot;nofollow&quot;&gt;Modelling Survival Data in Medical Research&lt;/a&gt; is a very good choice as a first book in survival analysis... but there are many others.&lt;/p&gt;&#xA;" OwnerUserId="3019" LastActivityDate="2011-04-04T05:53:22.827" />
  <row Id="9152" PostTypeId="2" ParentId="9109" CreationDate="2011-04-04T06:35:27.023" Score="0" ViewCount="" Body="&lt;p&gt;The log probability of a GMM is non-convex, which makes it converge only locally. Also, EM scales with the number of points in your dataset - you might want to try online EM if you have a big dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Compared to fitting univariate models (like a single Gaussian) performance is of course horrible, since it's an iterative procedure. To speed it up, a good heuristic is to start of with a couple of iterations of K-Means, use the centers as means and go on from there with the GMM modeling.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have had the feeling that GMMs converge globally for very simple and low dimensional datasets. I have never checked this for big datasets though.&lt;/p&gt;&#xA;" OwnerUserId="2860" LastActivityDate="2011-04-04T06:35:27.023" />
  <row Id="9154" PostTypeId="2" ParentId="9129" CreationDate="2011-04-04T08:42:40.867" Score="0" ViewCount="" Body="&lt;p&gt;State of the art is to use &lt;a href=&quot;http://www.cs.toronto.edu/~rsalakhu/papers/semantic_final.pdf&quot; rel=&quot;nofollow&quot;&gt;semantic hashing by Hinton and Salakhutdinov&lt;/a&gt;. If you have a look into the paper, there are some really impressive 2D plots of several benchmark datasets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is a rather advanced algorithm, however. You train a stack of restricted boltzmann machines with contrastive divergence. In the end your representation of a document will be a bit vector. This can be used to do lookups based on the hamming distance. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lots of machine learning knowledge goes is required to sucessfully implement this, and as far as I know there is no out of the box. If you want to do this and you have no prior knowledge in neural networks et al, it will take quite some effort.&lt;/p&gt;&#xA;" OwnerUserId="2860" LastActivityDate="2011-04-04T08:42:40.867" />
  <row Id="9155" PostTypeId="1" AcceptedAnswerId="9196" CreationDate="2011-04-04T10:32:41.897" Score="5" ViewCount="147" Body="&lt;p&gt;Hi&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to clarify how &lt;a href=&quot;http://en.wikipedia.org/wiki/Granger_causality&quot; rel=&quot;nofollow&quot;&gt;the Granger causality&lt;/a&gt; can/should be used in practice, and how to interpret the statistical significance given by the test.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, I would like to fill this table with things like &quot;we don't know&quot; or if we know something, what do we know (It will for sure not be causality, but maybe something else?).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\begin{matrix}&#xD;&#xA; &amp;amp; X granger cause Y significant &amp;amp; X granger cause Y non significant\\ &#xD;&#xA;Y granger cause X significant &amp;amp; ... &amp;amp; ...\\ &#xD;&#xA;Y granger cause X non significant &amp;amp; ... &amp;amp; ...&#xD;&#xA;\end{matrix}&#xD;&#xA;$&lt;/p&gt;&#xA;" OwnerUserId="1709" LastEditorUserId="1709" LastEditorDisplayName="" LastEditDate="2011-04-05T01:48:43.127" LastActivityDate="2011-04-05T13:26:13.777" Title="Interpretation of the Granger causality test" Tags="&lt;test&gt;&lt;granger-causality&gt;" AnswerCount="1" CommentCount="5" />
  <row Id="9156" PostTypeId="1" CreationDate="2011-04-04T11:02:16.770" Score="3" ViewCount="157" Body="&lt;p&gt;I would like to propose a single model (decision tree), that is very variable, and validate it. I have choosen parameters after I had obtained good quality measures with a cross-validation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I could build the model on the whole data set and show cross-validated measures. But I can't get a special graph (called Reliability Plot) specific for that model. I should split my data set in training and test sets to obtain that specific graph. The model builded on the training set is different from the optimezed on the whole dataset. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could I choose my training set (50% of the total) to obtain the same model as the whole data set builded one? There is something unwise or wrong in this method?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="2719" LastEditorUserId="2719" LastEditorDisplayName="" LastEditDate="2011-04-04T20:29:03.047" LastActivityDate="2011-04-04T20:29:03.047" Title="How to choose training and test sets" Tags="&lt;cross-validation&gt;&lt;validation&gt;" AnswerCount="1" CommentCount="0" />
  <row Id="9157" PostTypeId="2" ParentId="9156" CreationDate="2011-04-04T11:28:31.587" Score="2" ViewCount="" Body="&lt;p&gt;If you have tuned the model parameters using cross-validation, then you won't get an unbiased estimnate of performance without using some completely new data.  Even if you re-cross-validate using a different partition of the data, or make a random test/training split using the data you have used already, this will still bias the performance evaluation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note the &quot;cross-validated measures&quot; you already have are a (possibly heavily) biased performance estimate if you have directly optimised it to choose the (hyper-) prarameters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The thing to do would be to used a nested cross-validation, where the outer cross-validation is used for performance estimation, where the model parameters are tuned indpendently in each fold via an &quot;inner&quot; cross-validation.  &lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2011-04-04T11:28:31.587" CommentCount="5" />
  <row Id="9158" PostTypeId="2" ParentId="9140" CreationDate="2011-04-04T12:35:15.843" Score="5" ViewCount="" Body="&lt;p&gt;The concept of censoring is the key to survival analysis and life data analysis. This issue can also enter via industrial statistics. When monitoring the length of time it&#xA;takes for a sample of units to fail, you can have&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Complete data: the exact time a unit fails is known&lt;/li&gt;&#xA;&lt;li&gt;Censored to the right: the time to fail for a unit is beyond the present run time&lt;/li&gt;&#xA;&lt;li&gt;Censored to the left: the known time is after the time a unit failed&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Other issues that enters the data mix are&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Singly censored: all unfailed units have a common run time&lt;/li&gt;&#xA;&lt;li&gt;Multiply censored: the unfailed units have different run times&lt;/li&gt;&#xA;&lt;li&gt;Interval censored: the time to fail is known to be between a particular set of times.&lt;/li&gt;&#xA;&lt;li&gt;Time censored: the censoring time is fixed&lt;/li&gt;&#xA;&lt;li&gt;Failure censored: a test is stopped when a fixed number of units fail&lt;/li&gt;&#xA;&lt;li&gt;Competing failure modes: the sample units fail for different reasons&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Common distributions capable of handling these situations are: lognormal, Weibull, and extreme value. The issues become interesting because there are graphical procedures to handle analysis as well as MLE and Method of Moments methods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Systems reliability is an off-shoot of this topic which gets involved with Bayesian methods, renewal theory, and accelerated life testing. Wayne Nelson and Bill Meeker&#xA;have several good books on the topics.&lt;/p&gt;&#xA;" OwnerUserId="3805" LastActivityDate="2011-04-04T12:35:15.843" CommentCount="1" />
  <row Id="9159" PostTypeId="1" AcceptedAnswerId="9166" CreationDate="2011-04-04T14:25:22.570" Score="6" ViewCount="170" Body="&lt;p&gt;While reading a few papers, I came across the term &quot;gold set&quot; or &quot;gold standard&quot;. What I don't understand is what makes a dataset gold standard? Peer acceptance, citation count and if its the liberty of the researcher and the relevance to problem he is attacking?&lt;/p&gt;&#xA;" OwnerUserId="2192" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-04T14:53:01.333" LastActivityDate="2011-04-05T16:31:03.737" Title="What is the meaning of a gold standard?" Tags="&lt;dataset&gt;&lt;terminology&gt;" AnswerCount="5" CommentCount="2" FavoriteCount="1" />
  <row Id="9160" PostTypeId="2" ParentId="9159" CreationDate="2011-04-04T14:46:40.257" Score="1" ViewCount="" Body="&lt;p&gt;A gold standard is a standard accepted as the most valid one and the most used. You can apply the expression for everything... But you can always accept or critic the standard, especially in the case of a dataset.&lt;/p&gt;&#xA;" OwnerUserId="4024" LastActivityDate="2011-04-04T14:46:40.257" />
  <row Id="9161" PostTypeId="2" ParentId="9104" CreationDate="2011-04-04T15:03:13.660" Score="5" ViewCount="" Body="&lt;p&gt;The terminology is probably not used consistently, so the following is only how I understand the original question. From my understanding, the normal CIs you computed are not what was asked for. Each set of bootstrap replicates gives you one confidence interval, not many. The way to compute different CI-types from the results of a set of bootstrap replicates is as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;B    &amp;lt;- 999                  # number of replicates&#xA;muH0 &amp;lt;- 100                  # for generating data: true mean&#xA;sdH0 &amp;lt;- 40                   # for generating data: true sd&#xA;N    &amp;lt;- 200                  # sample size&#xA;DV   &amp;lt;- rnorm(N, muH0, sdH0) # simulated data: original sample&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Since I want to compare the calculations against the results from package &lt;code&gt;boot&lt;/code&gt;, I first define a function that will be called for each replicate. Its arguments are the original sample, and an index vector specifying the cases for a single replicate. It returns $M^{*}$, the plug-in estimate for $\mu$, as well as $S_{M}^{2*}$, the plug-in estimate for the variance of the mean $\sigma_{M}^{2}$. The latter will be required only for the bootstrap $t$-CI.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; getM &amp;lt;- function(orgDV, idx) {&#xA;+     bsM   &amp;lt;- mean(orgDV[idx])                       # M*&#xA;+     bsS2M &amp;lt;- (((N-1) / N) * var(orgDV[idx])) / N    # S^2*(M)&#xA;+     c(bsM, bsS2M)&#xA;+ }&#xA;&#xA;&amp;gt; library(boot)                                       # for boot(), boot.ci()&#xA;&amp;gt; bOut &amp;lt;- boot(DV, statistic=getM, R=B)&#xA;&amp;gt; boot.ci(bOut, conf=0.95, type=c(&quot;basic&quot;, &quot;perc&quot;, &quot;norm&quot;, &quot;stud&quot;))&#xA;BOOTSTRAP CONFIDENCE INTERVAL CALCULATIONS&#xA;Based on 999 bootstrap replicates&#xA;CALL : &#xA;boot.ci(boot.out = bOut, conf = 0.95, type = c(&quot;basic&quot;, &quot;perc&quot;, &quot;norm&quot;, &quot;stud&quot;))&#xA;&#xA;Intervals : &#xA;Level      Normal            Basic         Studentized        Percentile    &#xA;95%   ( 95.6, 106.0 )   ( 95.7, 106.2 )  ( 95.4, 106.2 )   ( 95.4, 106.0 )  &#xA;Calculations and Intervals on Original Scale&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Without using package &lt;code&gt;boot&lt;/code&gt; you can simply use &lt;code&gt;replicate()&lt;/code&gt; to get a set of bootstrap replicates.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;boots &amp;lt;- t(replicate(B, getM(DV, sample(seq(along=DV), replace=TRUE))))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But let's stick with the results from &lt;code&gt;boot.ci()&lt;/code&gt; to have a reference.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;boots   &amp;lt;- bOut$t                     # estimates from all replicates&#xA;M       &amp;lt;- mean(DV)                   # M from original sample&#xA;S2M     &amp;lt;- (((N-1)/N) * var(DV)) / N  # S^2(M) from original sample&#xA;Mstar   &amp;lt;- boots[ , 1]                # M* for each replicate&#xA;S2Mstar &amp;lt;- boots[ , 2]                # S^2*(M) for each replicate&#xA;biasM   &amp;lt;- mean(Mstar) - M            # bias of estimator M&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The basic, percentile, and $t$-CI rely on the empirical distribution of bootstrap estimates. To get the $\alpha/2$ and $1 - \alpha/2$ quantiles, we find the corresponding indices to the sorted vector of bootstrap estimates (note that &lt;code&gt;boot.ci()&lt;/code&gt; will do a more complicated interpolation to find the empirical quantiles when the indices are not natural numbers).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(idx &amp;lt;- trunc(B * c(0.05/2, 1-(0.05/2))) + 1) # indices for sorted vector of estimates&#xA;[1] 25 975&#xA;&#xA;&amp;gt; (ciBasic &amp;lt;- 2*M - sort(Mstar)[idx])         # basic CI&#xA;[1] 106.21826  95.65911&#xA;&#xA;&amp;gt; (ciPerc &amp;lt;- sort(Mstar)[idx])                # percentile CI&#xA;[1] 95.42188 105.98103&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For the $t$-CI, we need the bootstrap $t^{*}$ estimates to calculate the critical $t$-values. For the standard normal CI, the critical value will just be the $z$-value from the standard normal distribution.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# standard normal CI with bias correction&#xA;&amp;gt; zCrit   &amp;lt;- qnorm(c(0.025, 0.975))   # z-quantiles from std-normal distribution&#xA;&amp;gt; (ciNorm &amp;lt;- M - biasM + zCrit * sqrt(var(Mstar)))&#xA;[1] 95.5566 106.0043&#xA;&#xA;&amp;gt; tStar &amp;lt;- (Mstar-M) / sqrt(S2Mstar)  # t*&#xA;&amp;gt; tCrit &amp;lt;- sort(tStar)[idx]           # t-quantiles from empirical t* distribution&#xA;&amp;gt; (ciT  &amp;lt;- M - tCrit * sqrt(S2M))     # studentized t-CI&#xA;[1] 106.20690  95.44878&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In order to estimate the coverage probabilities of these CI-types, you will have to run this simulation many times. Just wrap the code into a function, return a list with the CI-results and run it with &lt;code&gt;replicate()&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1909" LastEditorUserId="1909" LastEditorDisplayName="" LastEditDate="2011-04-04T15:36:29.233" LastActivityDate="2011-04-04T15:36:29.233" CommentCount="4" />
  <row Id="9162" PostTypeId="2" ParentId="7146" CreationDate="2011-04-04T15:12:40.560" Score="1" ViewCount="" Body="&lt;p&gt;I'm not sure if somebody has already made this point, but pelwei can actually be forced to work as a 2 parameter weibull function by adding in a fixed bound.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Insead of calling &lt;code&gt;moments&amp;lt;-pelwei(wind.moments)&lt;/code&gt; you should simply call &lt;code&gt;moments&amp;lt;-pelwei(wind.moments,bound=0)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;you can always check what the zeta value is. If it's not 0 and you're using dweibull, you need to do something about it.&lt;/p&gt;&#xA;" OwnerUserId="4025" LastActivityDate="2011-04-04T15:12:40.560" />
  <row Id="9163" PostTypeId="2" ParentId="9159" CreationDate="2011-04-04T15:20:46.483" Score="1" ViewCount="" Body="&lt;p&gt;In a challenge, this usually mean the answer to the test set, previously hidden from participants.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2011-04-04T15:20:46.483" />
  <row Id="9164" PostTypeId="2" ParentId="9140" CreationDate="2011-04-04T15:28:25.340" Score="1" ViewCount="" Body="&lt;pre&gt;&lt;code&gt;5, 10, 12+, 14, 17, 18+, 20+&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;A first approximation description of survival analysis: Analysing data where the dependent variable has (1) precise values (the complete observations) and (2) values know to be above a given threshold (the censored observations). The above may be a survival data sample, values without &lt;code&gt;+&lt;/code&gt; are precisely known; values with &lt;code&gt;+&lt;/code&gt; are known to be more, but not how much more. (And there are many extensions.) &lt;/p&gt;&#xA;" OwnerUserId="3911" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-10T16:02:11.417" LastActivityDate="2011-04-10T16:02:11.417" />
  <row Id="9165" PostTypeId="1" CreationDate="2011-04-04T15:29:56.450" Score="5" ViewCount="79" Body="&lt;p&gt;I have a survival cancer clinical trials dataset from which I have generated Cox models using forward likelihood ratio testing within R. These models are based on 'traditional' cancer variables (eg. age, histology, metastasis etc). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to extend the model using high dimensional data (where we have measured many thousands of genes - FWIW, this is DNA methylation data, which can range from zero to one, rather than gene expression). Several approaches have been suggested for investigating survival using high-dimensional data, but I am not aware of any approaches that fit my requirements, i.e. adding high-dimensional data to a base multi-variate model constructed using previously identified survival correlates. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a first step, I am testing for bi-modality and reducing dimensionality by selecting the most bi-modal probes for further analysis. These probes would be the most amenable to testing and verification in the lab. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One approach would simply be to carry on with the forward LR testing, although this would leave me very prone to overfitting. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another (more sensible, in my opinion) approach would be to aggregate collections of genes into (survival-related) metagenes and then trim the metagenes into a handful of testable genes, so that this could be a usable test clinically, although this may also be prone to overfitting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The cancer I work on is rare and test/training cohorts are tricky. To put things into perspective, the clinical trials dataset is 135 cases, with a further 55 age-matched non-clinical-trials cases, which show no difference in survival to the clinical trials dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my question is, what sort of approaches should I be considering and is what I have done so far sensible?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any advice from this rather rambling question is most appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for reading!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ed &lt;/p&gt;&#xA;" OwnerUserId="3429" LastActivityDate="2011-04-05T01:18:24.983" Title="Adding high-dimensional data to mutivariate Cox model" Tags="&lt;survival&gt;&lt;microarray&gt;&lt;large-data&gt;" AnswerCount="2" CommentCount="4" FavoriteCount="2" />
  <row Id="9166" PostTypeId="2" ParentId="9159" CreationDate="2011-04-04T15:34:19.880" Score="5" ViewCount="" Body="&lt;p&gt;Say that you want to measure a certain polluant in drinking water, the golden standard will be the method which detects it with the highest sensitivity and accuracy. Any other method can then be compared to it, knowing that -under certain conditions- the golden standard is the best (e.g.: if you need to measure the polluant on site, the golden standard will not be any method that require huge machinery to be used, as you will not be able to bring it with you).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think the &lt;a href=&quot;http://en.wikipedia.org/wiki/Gold_standard_%28test%29&quot; rel=&quot;nofollow&quot;&gt;Wikipedia article&lt;/a&gt; explains it quite well:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In medicine and statistics, gold standard test refers to a diagnostic, test or benchmark that is &lt;strong&gt;the best available under reasonable conditions&lt;/strong&gt;. It does not have to be necessarily the best possible test for the condition in absolute terms. For example, in medicine, dealing with conditions that require an autopsy to have a perfect diagnosis, the gold standard test is normally less accurate than the autopsy. Anyway, it is obviously preferred by the patients.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="582" LastActivityDate="2011-04-04T15:34:19.880" CommentCount="3" />
  <row Id="9167" PostTypeId="2" ParentId="9159" CreationDate="2011-04-04T15:47:16.553" Score="0" ViewCount="" Body="&lt;p&gt;I have observed the term &quot;gold standard&quot; in quotes more times than not, so I take it to mean something that is highly subjective. Even in the Wikipedia article some paragraphs refer to it in quotes. The OP is also referring to a &quot;gold standard dataset&quot; which I take it to mean a &quot;Gold Standard&quot; for descriminant Analysis, as in Fishers' Iris dataset being the &quot;Gold Standard&quot;.  But I am not 100% sure usage is consistant.&lt;/p&gt;&#xA;" OwnerUserId="3489" LastEditorUserId="3489" LastEditorDisplayName="" LastEditDate="2011-04-04T15:57:54.190" LastActivityDate="2011-04-04T15:57:54.190" />
  <row Id="9168" PostTypeId="2" ParentId="9165" CreationDate="2011-04-04T16:17:49.320" Score="1" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;One approach would simply be to carry on with the forward LR testing, although this would leave me very prone to overfitting. &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;You could penalise model complexity to avoid overfitting. My favourite is the &lt;code&gt;stepAIC&lt;/code&gt; function from the &lt;code&gt;MASS&lt;/code&gt; package that uses AIC (can be configured to use BIC) as a goodness of fit. &lt;/p&gt;&#xA;" OwnerUserId="3911" LastActivityDate="2011-04-04T16:17:49.320" CommentCount="4" />
  <row Id="9169" PostTypeId="2" ParentId="9165" CreationDate="2011-04-04T16:23:56.663" Score="0" ViewCount="" Body="&lt;p&gt;Edit: after the comment below from EdS my original answer was not meaningful any more. @EdS, thanks for the further information!&lt;/p&gt;&#xA;" OwnerUserId="3911" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-05T01:18:24.983" LastActivityDate="2011-04-05T01:18:24.983" CommentCount="4" />
  <row Id="9170" PostTypeId="2" ParentId="9076" CreationDate="2011-04-04T18:01:02.420" Score="1" ViewCount="" Body="&lt;p&gt;You may also be interested in the 'auto.arima' function in the &lt;a href=&quot;http://robjhyndman.com/software/forecast/&quot; rel=&quot;nofollow&quot;&gt;forecast package&lt;/a&gt; for &lt;a href=&quot;http://www.r-project.org/&quot; rel=&quot;nofollow&quot;&gt;r&lt;/a&gt; as an example of a way to programmatically identify ARIMA models.  It probably doesn't find the model in the exact same way you would, but some of the code/ideas might be useful to you.&lt;/p&gt;&#xA;" OwnerUserId="2817" LastActivityDate="2011-04-04T18:01:02.420" />
  <row Id="9171" PostTypeId="1" AcceptedAnswerId="9185" CreationDate="2011-04-04T18:25:56.523" Score="3" ViewCount="304" Body="&lt;p&gt;I'm brand new to this R thing but am unsure which model to select.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;I did a &lt;em&gt;stepwise forward regression&lt;/em&gt; selecting each variable based on the lowest AIC. I came up with 3 models that I'm unsure which is the &quot;best&quot;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Model 1: Var1 (p=0.03) AIC=14.978&#xA;Model 2: Var1 (p=0.09) + Var2 (p=0.199) AIC = 12.543&#xA;Model 3: Var1 (p=0.04) + Var2 (p=0.04) + Var3 (p=0.06) AIC= -17.09&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm inclined to go with Model #3 because it has the lowest AIC (I heard negative is ok) and the p-values are still rather low. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've ran 8 variables as predictors of Hatchling Mass and found that these three variables are the best predictors.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;My next forward stepwise I choose Model 2 because even though the AIC was slightly larger the p values were all smaller. Do you agree this is the best?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Model 1: Var1 (p=0.321) + Var2 (p=0.162) + Var3 (p=0.163) + Var4 (p=0.222)  AIC = 25.63&#xA;Model 2: Var1 (p=0.131) + Var2 (p=0.009) + Var3 (p=0.0056)                  AIC = 26.518&#xA;Model 3: Var1 (p=0.258) + Var2 (p=0.0254)                                   AIC = 36.905&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;thanks!&lt;/p&gt;&#xA;" OwnerUserId="4027" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-04T18:34:21.100" LastActivityDate="2011-04-07T11:23:39.880" Title="AIC or p-value: Which one to choose for model selection?" Tags="&lt;model-selection&gt;&lt;aic&gt;&lt;stepwise-regression&gt;" AnswerCount="2" CommentCount="4" FavoriteCount="2" />
  <row Id="9172" PostTypeId="1" CreationDate="2011-04-04T18:49:54.763" Score="0" ViewCount="99" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;Possible Duplicate:&lt;/strong&gt;&lt;br&gt;&#xA;  &lt;a href=&quot;http://stats.stackexchange.com/questions/9171/aic-or-p-value-which-one-to-choose-for-model-selection&quot;&gt;AIC or p-value: Which one to choose for model selection?&lt;/a&gt;  &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&#xA;&#xA;&lt;p&gt;Hello,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm brand new to this R thing but am unsure which model to select.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;I did a stepwise forward regression selecting each variable based on the lowest AIC. I came up with 3 models that I'm unsure which is the &quot;best&quot;. Model 1: Var1 (p=0.03) AIC=14.978 Model 2: Var1 (p=0.09) + Var2 (p=0.199) AIC = 12.543 Model 3: Var1 (p=0.04) + Var2 (p=0.04) + Var3 (p=0.06) AIC= -17.09&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I'm inclined to go with Model #3 because it has the lowest AIC (I heard negative is ok) and the p values are still rather low.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've ran 8 variables as predictors of Hatchling Mass and found that these three variables are the best predictors.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;My next forward stepwise I choose Model 2 because even though the AIC was slightly larger the p values were all smaller. Do you agree this is the best? Model 1: Var1 (p=0.321) + Var2 (p=0.162) + Var3 (p=0.163) + Var4 (p=0.222) AIC = 25.63 Model 2: Var1 (p=0.131) + Var2 (p=0.009) + Var3 (p=0.0056) AIC = 26.518 Model 3: Var1 (p=0.258) + Var2 (p=0.0254) AIC = 36.905&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="4027" LastActivityDate="2011-04-04T19:53:14.297" ClosedDate="2011-04-04T20:14:39.083" Title="Model Selection-- AIC or p-value??" Tags="&lt;aic&gt;" AnswerCount="0" CommentCount="0" />
  <row Id="9174" PostTypeId="2" ParentId="9171" CreationDate="2011-04-04T18:59:19.727" Score="6" ViewCount="" Body="&lt;p&gt;Looking at individual p-values can be misleading. If you have variables that are colinear (have high correlation), you will get big p-values. This does not mean the variables are useless.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a quick rule of thumb, selecting your model with the AIC criteria is better than looking at p-values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One reason one might not select the model with the lowest AIC is when your variable to datapoint ratio is large.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that model selection and prediction accuracy are somewhat distinct problems. If your goal is to get accurate predictions, I'd suggest cross-validating your model by separating your data in a training and testing set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A paper on variable selection: &lt;a href=&quot;http://arxiv.org/PS_cache/arxiv/pdf/1003/1003.5930v2.pdf&quot; rel=&quot;nofollow&quot;&gt;Stochastic Stepwise Ensembles for Variable Selection&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="3834" LastActivityDate="2011-04-04T18:59:19.727" CommentCount="1" />
  <row Id="9175" PostTypeId="1" CreationDate="2011-04-04T19:08:50.480" Score="1" ViewCount="84" Body="&lt;p&gt;When you make the normal probability plot, the plot may have curved bounds. Then the plot should be roughly linear &lt;strong&gt;and&lt;/strong&gt; the data must lie within the bounds provided by software. In the  examples I have seen the normal was rejected because some data points were not within the bounds. Can you provide an example where the data is between the bounds, but not linear?&lt;/p&gt;&#xA;" OwnerUserId="3454" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-04-05T07:19:51.750" LastActivityDate="2011-04-26T09:44:55.127" Title="Linearity of the normal probability plot" Tags="&lt;normality&gt;" AnswerCount="1" CommentCount="4" FavoriteCount="1" />
  <row Id="9177" PostTypeId="1" CreationDate="2011-04-04T19:10:07.453" Score="0" ViewCount="61" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;Possible Duplicate:&lt;/strong&gt;&lt;br&gt;&#xA;  &lt;a href=&quot;http://stats.stackexchange.com/questions/9171/aic-or-p-value-which-one-to-choose-for-model-selection&quot;&gt;AIC or p-value: Which one to choose for model selection?&lt;/a&gt;  &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&#xA;&#xA;&lt;p&gt;Hello,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm brand new to this R thing but am unsure which model to select.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;I did a stepwise forward regression selecting each variable based on the lowest AIC. I came up with 3 models that I'm unsure which is the &quot;best&quot;. Model 1: Var1 (p=0.03) AIC=14.978 Model 2: Var1 (p=0.09) + Var2 (p=0.199) AIC = 12.543 Model 3: Var1 (p=0.04) + Var2 (p=0.04) + Var3 (p=0.06) AIC= -17.09&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I'm inclined to go with Model #3 because it has the lowest AIC (I heard negative is ok) and the p values are still rather low.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've ran 8 variables as predictors of Hatchling Mass and found that these three variables are the best predictors.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;My next forward stepwise I choose Model 2 because even though the AIC was slightly larger the p values were all smaller. Do you agree this is the best? Model 1: Var1 (p=0.321) + Var2 (p=0.162) + Var3 (p=0.163) + Var4 (p=0.222) AIC = 25.63 Model 2: Var1 (p=0.131) + Var2 (p=0.009) + Var3 (p=0.0056) AIC = 26.518 Model 3: Var1 (p=0.258) + Var2 (p=0.0254) AIC = 36.905&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="4027" LastActivityDate="2011-04-04T19:10:07.453" ClosedDate="2011-04-04T20:14:22.373" Title="Model Selection-- AIC or p-value??" Tags="&lt;aic&gt;" CommentCount="0" />
  <row Id="9180" PostTypeId="2" ParentId="9175" CreationDate="2011-04-04T20:10:31.270" Score="1" ViewCount="" Body="&lt;p&gt;This is a synthetic example. Here the data are close to q1=q2 but stepwise rather than linear. The example draws a sample from a normal distribution (right panel) then rounds the numbers (left panel).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(100)&#xA;x=rnorm(100, sd=3)&#xA;par(mfrow=(c(1, 2)))&#xA;qqnorm(round(x))&#xA;qqnorm(x)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/LxSlg.png&quot; alt=&quot;Normal Q-Q Plot&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="3911" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-26T09:44:55.127" LastActivityDate="2011-04-26T09:44:55.127" />
  <row Id="9182" PostTypeId="1" CreationDate="2011-04-04T23:00:51.540" Score="2" ViewCount="181" Body="&lt;p&gt;Hello,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have multiple variables (here: weight, horizontal diameter, price and dummy) related to different factors (here: Apple, Orange, Banana and Avocado):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Fruit   Weight      HorDiam     Price       Dummy&#xA;Apple   60      60      5       4&#xA;Apple   50      70      8       6&#xA;Orange  80      75      7       2&#xA;Orange  72      70      9       8&#xA;Banana  40      30      3       1&#xA;Banana  45      35      4       2&#xA;Banana  80      50      8       3&#xA;Avocado 100     60      13      8&#xA;Avocado 95      70      14      6&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I need to test if I can group some species together: are apples and oranges significantly different? ANOVA tells me if weight (or horizontal diameter, or price) is significantly different among species. Tukey test gives me if weight of one species is significantly different from weight of another one (pairwise). Clustering seems only able to group individual observations together, not species. I can't find the appropriate test (or algorithm) to tell me if, for a single variable (weight) or for all of them (weight, horDiam and price), apples can be grouped with oranges and/or with bananas. Any suggestion?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I created a &lt;em&gt;R&lt;/em&gt; code for this example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;### CREATE TABLE&#xA;Fruit&amp;lt;-c(&quot;Apple&quot;,&quot;Apple&quot;,&quot;Orange&quot;,&quot;Orange&quot;,&quot;Banana&quot;,&quot;Banana&quot;,&quot;Banana&quot;,&quot;Avocado&quot;,&quot;Avocado&quot;)&#xA;Weight&amp;lt;-c(60,50,80,72,40,45,85,90,95)&#xA;horDiam&amp;lt;-c(60,70,75,70,30,35,50,60,70)&#xA;Price&amp;lt;-c(5,8,7,9,3,4,8,13,14)&#xA;Dummy&amp;lt;-c(4,6,2,8,1,2,3,8,6)&#xA;myData&amp;lt;-data.frame(Fruit=Fruit, Weight=Weight, horDiam=horDiam, Price=Price, Dummy=Dummy)&#xA;rownames(myData)&amp;lt;-c(&quot;Apple1&quot;,&quot;Apple2&quot;,&quot;Orange1&quot;,&quot;Orange2&quot;,&quot;Banana1&quot;,&quot;Banana2&quot;,&quot;Banana3&quot;,&quot;Avocado1&quot;,&quot;Avocado2&quot;)&#xA;&#xA;&#xA;### ANOVA&#xA;fit.aov&amp;lt;-list()&#xA;summaryAOV&amp;lt;-list()&#xA;for (i in 1:3){&#xA;  fit.aov[[i]]&amp;lt;-aov(myData[,i+1]~myData[,1])&#xA;  summaryAOV[[i]]&amp;lt;-summary(fit.aov[[i]])&#xA;}&#xA;&#xA;&#xA;### TUKEY&#xA;par(mfrow=c(1,3))&#xA;testTukey&amp;lt;-list()&#xA;mainTukey&amp;lt;-c(&quot;Weight&quot;, &quot;Horiz. Diameter&quot;, &quot;Price&quot;)&#xA;for (i in 1:3){&#xA;  testTukey[[i]]&amp;lt;-TukeyHSD(fit.aov[[i]], conf.level = 0.95)&#xA;    plot(testTukey[[i]], main=mainTukey[i])&#xA;}&#xA;&#xA;&#xA;### CLUSTERING&#xA;plot( hclust(dist(myData), method=&quot;ward&quot;) )&#xA;&#xA;&#xA;### CLUSTERING WITH P-VALUE&#xA;fit &amp;lt;- pvclust(t(myData[,-1]), method.hclust=&quot;ward&quot;, method.dist=&quot;euclidean&quot;)&#xA;plot(fit)&#xA;pvrect(fit, alpha=0.95)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="4028" LastActivityDate="2011-04-06T11:46:01.810" Title="Multivariate grouping: clustering, anova, tukey" Tags="&lt;r&gt;&lt;hypothesis-testing&gt;&lt;clustering&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="2" />
  <row Id="9183" PostTypeId="2" ParentId="8881" CreationDate="2011-04-05T00:20:39.873" Score="0" ViewCount="" Body="&lt;p&gt;As the comments suggest, it is only by fully understanding and specifying your research design that you will establish what regression method best corresponds to your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the case where your DV is a categorical variable, which seems likely if you are dealing with social data, I would recommend reading extensively from &lt;a href=&quot;http://www.stata.com/bookstore/regmodcdvs.html&quot; rel=&quot;nofollow&quot;&gt;Long and Freese&lt;/a&gt; to make an informed choice. Long and Freese use Stata, but equivalent commands exist in both R and SPSS.&lt;/p&gt;&#xA;" OwnerUserId="3582" LastActivityDate="2011-04-05T00:20:39.873" />
  <row Id="9184" PostTypeId="2" ParentId="9182" CreationDate="2011-04-05T00:39:42.973" Score="2" ViewCount="" Body="&lt;p&gt;Are you looking for &lt;a href=&quot;http://en.wikipedia.org/wiki/Multivariate_analysis_of_variance&quot; rel=&quot;nofollow&quot;&gt;MANOVA&lt;/a&gt;? This is the multivariate generalization of ANOVA where you are testing for differences between mean &lt;em&gt;vectors&lt;/em&gt;. In R the manova function will fit this model.&lt;/p&gt;&#xA;" OwnerUserId="2040" LastActivityDate="2011-04-05T00:39:42.973" CommentCount="2" />
  <row Id="9185" PostTypeId="2" ParentId="9171" CreationDate="2011-04-05T01:20:55.760" Score="3" ViewCount="" Body="&lt;p&gt;AIC is a goodness of fit measure that favours smaller residual error in the model, but penalises for including further predictors and helps avoiding overfitting. In your second set of models model 1 (the one with the lowest AIC) may perform best when used for prediction outside your dataset. A possible explanation why adding Var4 to model 2 results in a lower AIC, but higher p values is that Var4 is somewhat correlated with Var1, 2 and 3. The interpretation of model 2 is thus easier.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastActivityDate="2011-04-05T01:20:55.760" />
  <row Id="9187" PostTypeId="2" ParentId="8823" CreationDate="2011-04-05T07:46:04.800" Score="1" ViewCount="" Body="&lt;p&gt;Okay, so rather than go and re-derive Saunder's equation (5), I will just state it here.  Condition 1 and 2 imply the following equality:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\prod_{j=1}^{m}\left(\sum_{k\neq i}h_{k}d_{jk}\right)=\left(\sum_{k\neq i}h_{k}\right)^{m-1}\left(\sum_{k\neq i}h_{k}\prod_{j=1}^{m}d_{jk}\right)$$&#xA;where&#xA;$$d_{jk}=P(D_{j}|H_{k},I)\;\;\;\;h_{k}=P(H_{k}|I)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now we can specialise to the case $m=2$ (two data sets) by taking $D_{1}^{(1)}\equiv D_{1}$ and relabeling $D_{2}^{(1)}\equiv D_{2}D_{3}\dots D_{m}$.  Note that these two data sets still satisfy conditions 1 and 2, so the result above applies to them as well.  Now expanding in the case $m=2$ we get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\left(\sum_{k\neq i}h_{k}d_{1k}\right)\left(\sum_{l\neq i}h_{l}d_{2l}\right)=\left(\sum_{k\neq i}h_{k}\right)\left(\sum_{l\neq i}h_{l}d_{1l}d_{2l}\right)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\rightarrow\sum_{k\neq i}\sum_{l\neq i}h_{k}h_{l}d_{1k}d_{2l}=\sum_{k\neq i}\sum_{l\neq i}h_{k}h_{l}d_{1l}d_{2l}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\rightarrow\sum_{k\neq i}\sum_{l\neq i}h_{k}h_{l}d_{2l}(d_{1k}-d_{1l})=0\;\;\;\;\;\;\; (i=1,\dots,n)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The term $(d_{1a}-d_{1b})$ occurs twice in the above double summation, once when $k=a$ and $l=b$, and once again when $k=b$ and $l=a$.  This will occur as long as $a,b\neq i$.  The coefficient of each term is given by $d_{2b}$ and $-d_{2a}$.  Now because there are $i$ of these equations, we can actually remove $i$ from these equations.  To illustrate, take $i=1$, now this means we have all conditions except where $a=1,b=2$ and $b=1,a=2$.  Now take $i=3$, and we now can have these two conditions (note this assumes at least three hypothesis).  So the equation can be re-written as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\sum_{l&amp;gt;k}h_{k}h_{l}(d_{2l}-d_{2k})(d_{1k}-d_{1l})=0$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now each of the $h_i$ terms must be greater than zero, for otherwise we are dealing with $n_{1}&amp;lt;n$ hypothesis, and the answer can be reformulated in terms of $n_{1}$.  So these can be removed from the above set of conditions:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\sum_{l&amp;gt;k}(d_{2l}-d_{2k})(d_{1k}-d_{1l})=0$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, there are $\frac{n(n-1)}{2}$ conditions that must be satisfied, and each conditions implies one of two &quot;sub-conditions&quot;: that $d_{jk}=d_{jl}$ for either $j=1$ or $j=2$ (but not necessarily both).  Now we have a set of all of the unique pairs $(k,l)$ for $d_{jk}=d_{jl}$.  If we were to take $n-1$ of these pairs for one of the $j$, then we would have all the numbers $1,\dots,n$ in the set, and $d_{j1}=d_{j2}=\dots=d_{j,n-1}=d_{j,n}$.  This is because the first pair has $2$ elements, and each additional pair brings at least one additional element to the set*&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But note that because there are $\frac{n(n-1)}{2}$ conditions, we must choose at least the smallest integer greater than or equal to $\frac{1}{2}\times\frac{n(n-1)}{2}=\frac{n(n-1)}{4}$ for one of the $j=1$ or $j=2$.  If $n&amp;gt;4$ then the number of terms chosen is greater than $n-1$.  If $n=4$ or $n=3$ then we must choose exactly $n-1$ terms.  This implies that $d_{j1}=d_{j2}=\dots=d_{j,n-1}=d_{j,n}$.  Only with two hypothesis ($n=2$) is where this does not occur.  But from the last equation in Saunder's article this equality condition implies:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(D_{j}|\overline{H}_{i})=\frac{\sum_{k\neq i}d_{jk}h_{k}}{\sum_{k\neq i}h_{k}}=d_{ji}\frac{\sum_{k\neq i}h_{k}}{\sum_{k\neq i}h_{k}}=d_{ji}=P(D_{j}|H_{i})$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, in the likelihood ratio we have:&#xA;$$\frac{P(D_{1}^{(1)}|H_{i})}{P(D_{1}^{(1)}|\overline{H}_{i})}=\frac{P(D_{1}|H_{i})}{P(D_{1}|\overline{H}_{i})}=1 \text{    OR} \frac{P(D_{2}^{(1)}|H_{i})}{P(D_{2}^{(1)}|\overline{H}_{i})}=\frac{P(D_{2}D_{3}\dots,D_{m}|H_{i})}{P(D_{2}D_{3}\dots,D_{m}|\overline{H}_{i})}=1$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To complete the proof, note that if the second condition holds, the result is already proved, and only one ratio can be different from 1.  If the first condition holds, then we can repeat the above analysis by relabeling $D_{1}^{(2)}\equiv D_{2}$ and $D_{2}^{(2)}\equiv D_{3}\dots,D_{m}$.  Then we would have $D_{1},D_{2}$ not contributing, or $D_{2}$ being the only contributor.  We would then have a third relabeling when $D_{1}D_{2}$ not contributing holds, and so on.  Thus, only one data set can contribute to the likelihood ratio when condition 1 and condition 2 hold, and there are more than two hypothesis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;*NOTE: An additional pair might bring no new terms, but this would be offset by a pair which brought 2 new terms. e.g. take $d_{j1}=d_{j2}$ as first[+2], $d_{j1}=d_{j3}$ [+1] and $d_{j2}=d_{j3}$ [+0], but next term must have $d_{jk}=d_{jl}$ for both $k,l\notin (1,2,3)$.  This will add two terms [+2].  If $n=4$ then we don't need to choose any more, but for the &quot;other&quot; $j$ we must choose the 3 pairs which are not $(1,2),(2,3),(1,3)$.  These are $(1,4),(2,4),(3,4)$ and thus the equality holds, because all numbers $(1,2,3,4)$ are in the set.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-04-05T07:46:04.800" />
  <row Id="9190" PostTypeId="1" CreationDate="2011-04-05T11:51:15.763" Score="8" ViewCount="246" Body="&lt;p&gt;I have to build a multi-user webapp which is about traffic measurements, prognoses etc. At this point I know that I'll use bar and pie charts. Unfortunately, those chart types aren't rich in expressing all the data that I collect and compute.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm looking for a collection of graphical charts. It is very ok if I have to buy a book or anything else. I need to find some graphical samples with explanations in order to inspire me. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you know of any such resource? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you have any advice for me?&lt;/p&gt;&#xA;" OwnerUserId="3856" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-04-05T15:22:02.180" LastActivityDate="2011-04-13T15:39:52.810" CommunityOwnedDate="2011-04-05T13:34:02.290" Title="Graphics encyclopedia" Tags="&lt;data-visualization&gt;" AnswerCount="8" CommentCount="2" FavoriteCount="5" />
  <row Id="9191" PostTypeId="2" ParentId="9190" CreationDate="2011-04-05T12:21:39.897" Score="5" ViewCount="" Body="&lt;p&gt;If you fancy R, you can see the &lt;a href=&quot;http://addictedtor.free.fr/graphiques/&quot; rel=&quot;nofollow&quot;&gt;R graph gallery&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="144" LastActivityDate="2011-04-05T12:21:39.897" CommunityOwnedDate="2011-04-05T13:34:02.290" />
  <row Id="9192" PostTypeId="1" CreationDate="2011-04-05T12:24:30.907" Score="5" ViewCount="118" Body="&lt;p&gt;I'm doing some Machine Learning in R using the &lt;code&gt;nnet&lt;/code&gt; package. I want to estimate the generalisation performance of my classifier by using k-fold cross-validation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How should I go about doing this? Are there some built-in functions that do this for me? I've seen &lt;code&gt;tune.nnet()&lt;/code&gt; in the &lt;code&gt;e1071&lt;/code&gt; package, but I'm not sure that does quite what I want.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically I want to do the cross-validation (split into 10 groups, train on 9, test on the other 1, repeat) and then from that obtain some sort of measure of how well my classifier generalises - but I'm not sure what measure that should be. I guess I want to look at the average of the accuracies across the different cross-validation examples, but I'm not sure how to do that with the &lt;code&gt;tune.nnet()&lt;/code&gt; function above.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas?&lt;/p&gt;&#xA;" OwnerUserId="261" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-05T12:41:05.260" LastActivityDate="2011-04-05T14:16:48.083" Title="How to get generalisation performance from nnet in R using k-fold cross-validation?" Tags="&lt;r&gt;&lt;machine-learning&gt;&lt;cross-validation&gt;&lt;neural-networks&gt;" AnswerCount="2" />
  <row Id="9193" PostTypeId="2" ParentId="9190" CreationDate="2011-04-05T12:53:05.117" Score="5" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://www.stat.purdue.edu/~wsc/&quot; rel=&quot;nofollow&quot;&gt;Cleveland, William S&lt;/a&gt;. 1993. &lt;a href=&quot;http://books.google.com/books?id=V-dQAAAAMAAJ&quot; rel=&quot;nofollow&quot;&gt;&lt;em&gt;Visualizing data&lt;/em&gt;.&lt;/a&gt; ISBN 0963488406.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-04-05T12:53:05.117" CommunityOwnedDate="2011-04-05T13:34:02.290" CommentCount="1" />
  <row Id="9194" PostTypeId="2" ParentId="9190" CreationDate="2011-04-05T13:01:16.297" Score="8" ViewCount="" Body="&lt;p&gt;For an online summary, check out A &lt;a href=&quot;http://www.visual-literacy.org/periodic_table/periodic_table.html&quot; rel=&quot;nofollow&quot;&gt;Periodic Table of Visualization Methods&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-04-05T13:01:16.297" CommunityOwnedDate="2011-04-05T13:34:02.290" />
  <row Id="9195" PostTypeId="2" ParentId="9192" CreationDate="2011-04-05T13:11:14.243" Score="4" ViewCount="" Body="&lt;p&gt;If you are planning to tune the network (e.g. select a value for the learning rate) on your training data and determine the error generalization on that same data set, you need to use a nested cross validation, where in each fold, you are tuning the model on that 9/10 of the data set (using 10 fold cv). See this post where I asked a similar question and got very good answers (&lt;a href=&quot;http://stats.stackexchange.com/questions/5918/cross-validation-error-generalization-after-model-selection&quot;&gt;post&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not sure if there is an R function to accomplish this - maybe the &lt;a href=&quot;http://cran.r-project.org/web/packages/ipred/index.html&quot; rel=&quot;nofollow&quot;&gt;ipred&lt;/a&gt; package could be used by passing the tune function with the call - not sure. It is rather trivial to simply write a loop to accomplish this whole process though.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as what metric to examine, that depends on your problem (classification versus regression) and if you are interested in accuracy (classifications rate or kappa) or how well the model ranks (e.g. lift) or the MAE or RMSE for regression.&lt;/p&gt;&#xA;" OwnerUserId="2040" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-05T13:33:28.780" LastActivityDate="2011-04-05T13:33:28.780" />
  <row Id="9196" PostTypeId="2" ParentId="9155" CreationDate="2011-04-05T13:26:13.777" Score="4" ViewCount="" Body="&lt;p&gt;To begin with, the source you added has almost all you need to get acquainted with &lt;em&gt;Granger (non)causality&lt;/em&gt; concept (though I like the &lt;a href=&quot;http://www.scholarpedia.org/article/Granger_causality&quot; rel=&quot;nofollow&quot;&gt;scholarpedia&lt;/a&gt;'s article more). The most crucial is that G-causality in practice looks for the answer: would variable $x$ be useful predicting variable $y$, meaning that information containing in variables up to lag $p$ is &lt;strong&gt;statistically&lt;/strong&gt; significant. Thus G-causality is purely statistical property of the data, that may be though supported by theoretically sound hypothesis. &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Some practical considerations:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;If you have more than two stationary signals, it may happen that they have to be jointly described by a vector autoregressive model (&lt;a href=&quot;http://en.wikipedia.org/wiki/Vector_autoregression&quot; rel=&quot;nofollow&quot;&gt;VAR&lt;/a&gt;). Therefore pairwise G-causality could be misleading, since you ignore the impacts that come from the other variables. &lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Suggestion in&lt;/strong&gt; $R$: try &lt;code&gt;library(vars)&lt;/code&gt; and &lt;code&gt;?causality&lt;/code&gt; for instantaneous and G-causality, when you have more than two variables, and VAR seems meaningful (well it is a separate answer when it really is, some ideas are also related to G-causality concept). &lt;/li&gt;&#xA;&lt;li&gt;Previous suggestion is better in multivariate case, comparing to pairwise case  &lt;code&gt;library(lmtest)&lt;/code&gt; and &lt;code&gt;?grangertest&lt;/code&gt;. On the other hand pairwise case is an option when you do have to work with two variables only. Even in multivariate case you may still perform &lt;code&gt;grangertest&lt;/code&gt; just to mark possible useful covariates or decide on statistical possible endogeneity issues. I usually do so when lacking in time, since identification of subsets of variables and hyper-parameters (lag order) selection for VAR models is a not-so-quick task. So for quick useful-predictive-information-containing variables detection it is alright to go pairwise (but do not stop with this results, they are just auxiliary).&lt;/li&gt;&#xA;&lt;li&gt;Note, that under null hypothesis you do test non-G-causality, thus $p$ values will mark G-causal relationships.&lt;/li&gt;&#xA;&lt;li&gt;Conclusions from G-causality tests would be &quot;We know that if $x$ G-cause $y$ statistically significantly, thus it contains useful information that helps to predict future values of $y$&quot;. However if we conclude the same about $y$ (feedback effect) it would mean that $x$ and $y$ are both endogenous and VAR type of the model is needed. You may also conclude that if none of the variables G-cause another, it is one of the signs that VAR specification is not necessary. And you may go for separate ARMA models (note, that your variables have to be stationary to perform G-causality tests correctly).  &lt;/li&gt;&#xA;&lt;li&gt;Any other suggestions from the community are welcome, @zik you may try &lt;a href=&quot;http://gretl.sourceforge.net/&quot; rel=&quot;nofollow&quot;&gt;gretl&lt;/a&gt; as an alternative to $R$ to implement Granger-causality tests.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="2645" LastActivityDate="2011-04-05T13:26:13.777" />
  <row Id="9197" PostTypeId="2" ParentId="9192" CreationDate="2011-04-05T14:16:48.083" Score="3" ViewCount="" Body="&lt;p&gt;Implementing k-fold CV (with or without nesting) is relatively straightforward in R; and stratified sampling (wrt. class membership or subjects' characteristics, e.g. age or gender) is not that difficult. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;About the way to assess one's classifier performance, you can directly look at the R code for the &lt;code&gt;tune()&lt;/code&gt; function. (Just type &lt;code&gt;tune&lt;/code&gt; at the R prompt.) For a classification problem, this is the class agreement (between predicted and observed class membership) that is computed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, if you are looking for a complete R framework where data preprocessing (feature elimination, scaling, etc.), training/test resampling, and comparative measures of classifiers accuracy are provided in few commands, I would definitely recommend to have a look at the &lt;a href=&quot;http://caret.r-forge.r-project.org/Classification_and_Regression_Training.html&quot; rel=&quot;nofollow&quot;&gt;caret&lt;/a&gt; package, which also includes a lot of useful vignettes (see also the &lt;a href=&quot;http://www.jstatsoft.org/v28/i05/paper&quot; rel=&quot;nofollow&quot;&gt;JSS paper&lt;/a&gt;). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of note, although NNs are part of the methods callable from within &lt;code&gt;caret&lt;/code&gt;, you may probably have to look at other methods that perform as well and most of the times better than NNs (e.g., Random Forests, SVMs, etc.)&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-04-05T14:16:48.083" CommentCount="1" />
  <row Id="9198" PostTypeId="1" CreationDate="2011-04-05T15:02:20.867" Score="10" ViewCount="94" Body="&lt;p&gt;&lt;strong&gt;Seasonal adjustment&lt;/strong&gt; is a crucial step preprocessing the data for further research. Researcher however has a number of options for trend-cycle-seasonal decomposition. The most common (judging by the number of citations in empirical literature) rival seasonal decomposition methods are X-11(12)-ARIMA, Tramo/Seats (both implemented in &lt;a href=&quot;http://circa.europa.eu/irc/dsis/eurosam/info/data/demetra.htm&quot; rel=&quot;nofollow&quot;&gt;Demetra+&lt;/a&gt;) and $R$'s &lt;a href=&quot;http://www.wessa.net/download/stl.pdf&quot; rel=&quot;nofollow&quot;&gt;stl&lt;/a&gt;. Seeking to avoid random choice between mentioned (or other simple methods like seasonal dummy variables) decomposition techniques I would like to know a basic strategy that leads to choosing seasonal decomposition method effectively. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Several important subquestions (links to a discussion are welcome too) could be:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;What are the similiraties and differences, strong and weak points of the methods? Are there any special cases when one method is more preferable than the others?&lt;/li&gt;&#xA;&lt;li&gt;Could you provide general guides to what is inside the black-box of different decomposition methods?&lt;/li&gt;&#xA;&lt;li&gt;Are there special tricks choosing the parameters for the methods (I am not always satisfied with the defaults, &lt;code&gt;stl&lt;/code&gt; for example has many parameters to deal with, sometimes I feel I just don't know how to choose these ones in a right way).&lt;/li&gt;&#xA;&lt;li&gt;Is it possible to suggest some (statistical) criteria that the time series is seasonally adjusted efficiently (correlogram analysis, spectral density? small sample size criteria? robustness?).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="2645" LastActivityDate="2011-05-09T16:23:21.067" Title="Choosing seasonal decomposition method" Tags="&lt;time-series&gt;&lt;data-transformation&gt;&lt;methodology&gt;&lt;seasonality&gt;" AnswerCount="2" FavoriteCount="7" />
  <row Id="9199" PostTypeId="2" ParentId="9190" CreationDate="2011-04-05T15:19:15.227" Score="4" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Systat&lt;/strong&gt; (Lee Wilkinson) was an early leader in statistical graphics software.  It always has had a &lt;a href=&quot;http://www.systat.com/solutions/ImageGallery.aspx&quot; rel=&quot;nofollow&quot;&gt;nice visual gallery&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-04-05T15:19:15.227" CommunityOwnedDate="2011-04-05T15:19:15.227" />
  <row Id="9200" PostTypeId="2" ParentId="9190" CreationDate="2011-04-05T15:21:14.570" Score="3" ViewCount="" Body="&lt;p&gt;A visual &lt;a href=&quot;http://gallery.wolfram.com/&quot; rel=&quot;nofollow&quot;&gt;gallery of really creative graphics&lt;/a&gt; (but without much organization, unfortunately) is available on the Wolfram site (&lt;em&gt;Mathematica&lt;/em&gt;).&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-04-05T15:21:14.570" CommunityOwnedDate="2011-04-05T15:21:14.570" />
  <row Id="9201" PostTypeId="1" CreationDate="2011-04-05T15:38:55.223" Score="7" ViewCount="162" Body="&lt;p&gt;I wrote a script tests the data using the &lt;code&gt;wilcox.test&lt;/code&gt;, but when I got the results, all the p-values where equal to 1.&#xA;I read in some websites that you could use jitter before testing the data (to avoid ties as they said), I did this and now I have an acceptable result. &#xA;Is it wrong to do this?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;test&amp;lt;- function(column,datacol){&#xA;    library(ggplot2)&#xA;t=read.table(&quot;data.txt&quot;, stringsAsFactors=FALSE)&#xA;uni=unique(c(t$V9))&#xA;    for (xp in uni) { &#xA;          for(yp in uni) {&#xA;            testx &amp;lt;- subset(t, V9==xp)&#xA;            testy &amp;lt;- subset(t, V9==yp)&#xA;            zz &amp;lt;- wilcox.test(testx[[datacol]],jitter(testy[[datacol]]))&#xA;            p.value &amp;lt;- zz$p.value&#xA;          }&#xA;        }&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;hr&gt;&#xA;This is the output of &lt;code&gt;dput(head(t))&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;structure(list(V1 = c(0.268912,&#xA;0.314681, 0.347078, 0.286945, &#xA;0.39562, 0.282182), V2 = c(0.158921, 0.210526, 0.262024, 0.322006, &#xA;0.133417, 0.283025), V3 = c(0.214082, 0.166895, 0.132547, 0.147361, &#xA;0.09174, 0.169093), V4 = c(0.358085, 0.307898, 0.258352, 0.243688, &#xA;0.379224, 0.2657), V5= c(-0.142223, 0.010895, 0.14655, &#xA;0.08152, 0.02116, 0.030083), V6 = c(0.096408, -0.091896,&#xA;&#xA;-0.331229, -0.446603, -0.088493, -0.262037), V7` = c(1.680946, &#xA;1.649559, 1.534401, 1.130529, 3.441356, 1.211815), V8 = c(&quot;NC_000834&quot;,  &quot;NC_000844&quot;,&#xA;&quot;NC_000845&quot;, &quot;NC_000846&quot;, &quot;NC_000857&quot;,&#xA;&quot;NC_000860&quot; ), V9 = c(&quot;Chordata&quot;,&#xA;&quot;Arthropoda&quot;, &quot;Chordata&quot;, &quot;Chordata&quot;, &#xA;&quot;Arthropoda&quot;, &quot;Chordata&quot;), V10 =&#xA;c(&quot;???:???&quot;, &quot;Diplostraca&quot;, &#xA;&quot;???:???&quot;, &quot;Rheiformes&quot;, &quot;Diptera&quot;,&#xA;&quot;Salmoniformes&quot;), V11 = c(&quot;???:???&quot;,&#xA;&quot;Branchiopoda&quot;, &quot;Mammalia&quot;, &quot;Aves&quot;,&#xA;&quot;Insecta&quot;, &quot;Actinopterygii&quot; )), .Names&#xA;= c(&quot;V1&quot;, &quot;V2&quot;, &quot;V3&quot;, &quot;V4&quot;, &quot;V5&quot;, &quot;V6&quot;,  &quot;V7&quot;,&#xA;&quot;V8&quot;, &quot;V9&quot;, &quot;V10&quot;,&#xA;&quot;V11&quot;), row.names = c(NA,  6L),&#xA;class = &quot;data.frame&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The data is very large, and that's the thread I started and they told me it might be wrong to do this&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note&lt;/strong&gt; This question comes from tex.SE:&#xA;&lt;a href=&quot;http://tex.stackexchange.com/questions/15013/generating-pdfcontain-r-output-inside-latex-table&quot;&gt;generating PDFcontain R output inside latex table&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="4038" LastEditorUserId="582" LastEditorDisplayName="" LastEditDate="2011-04-06T05:45:53.110" LastActivityDate="2011-04-06T14:41:20.167" Title="Is it wrong to jitter before performing Wilcoxon test?" Tags="&lt;r&gt;" AnswerCount="3" CommentCount="10" FavoriteCount="0" />
  <row Id="9202" PostTypeId="1" AcceptedAnswerId="9211" CreationDate="2011-04-05T15:42:20.177" Score="7" ViewCount="145" Body="&lt;p&gt;I am about to try out a BUGS style environment for estimating Bayesian models.  Are there any important advantages to consider in choosing between OpenBugs or JAGS?  Is one likely to replace the other in the foreseeable future?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I will be using the chosen Gibbs Sampler with R.  I don't have a specific application yet, but rather I am deciding which to intall and learn.&lt;/p&gt;&#xA;" OwnerUserId="3700" LastActivityDate="2011-04-05T19:59:30.963" Title="OpenBugs vs. JAGS" Tags="&lt;software&gt;&lt;bugs&gt;&lt;gibbs&gt;&lt;jags&gt;" AnswerCount="1" FavoriteCount="4" />
  <row Id="9203" PostTypeId="1" AcceptedAnswerId="9204" CreationDate="2011-04-05T15:54:06.677" Score="3" ViewCount="54" Body="&lt;p&gt;I have a distribution $X$. By playing around with random samples from $X$, I've determined that $Var(X^i) &amp;gt; Var(X)$ where $i &amp;gt; 1$. However, I can't seem to find a formula for the expected variance of $X^i$, or why it should be greater than $Var(X)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Moving away from normal distributions, should it generalize that the scale parameter of any $X^i$ will be greater than $X$?&lt;/p&gt;&#xA;" OwnerUserId="287" LastActivityDate="2011-04-06T07:24:02.930" Title="Variance of a distribution's product with itself" Tags="&lt;variance&gt;" AnswerCount="2" />
  <row Id="9204" PostTypeId="2" ParentId="9203" CreationDate="2011-04-05T16:04:03.533" Score="3" ViewCount="" Body="&lt;p&gt;$Var(X^i)$ = $\mathbb{E}[(X^i - \mathbb{E}[X^i])^2]$ = $\mathbb{E}[X^{2i}] - (\mathbb{E}[X^i])^2$ by definition.  This expresses the variance of $X^i$ in terms of moments of $X$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The generalization is false, because $\mathbb{E}[(\lambda X)^{2i}]$ = $|\lambda|^{2i}\mathbb{E}[X^{2i}]$ implies that the scale parameter of $(\lambda X)^i$ will be smaller than the scale parameter of $\lambda X$ when $\lambda$ is sufficiently close to zero and $i \gt 1$.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-04-05T16:04:03.533" />
  <row Id="9205" PostTypeId="2" ParentId="9182" CreationDate="2011-04-05T16:09:22.260" Score="3" ViewCount="" Body="&lt;p&gt;You might have a look at the &lt;code&gt;betadisper()&lt;/code&gt; function in the &lt;code&gt;vegan&lt;/code&gt; package. The function implements the PERMDISP2 procedure (Anderson, 2006) for the analysis of multivariate homogeneity of group dispersions. An example using your data might be the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  require(vegan)&#xA;  distance&amp;lt;-vegdist(myData[,2:5], method=&quot;euclidean&quot;)&#xA;  model&amp;lt;-betadisper(distance, myData[,1])&#xA;  permutest(model, pairwise = TRUE)&#xA;&#xA;Permutation test for homogeneity of multivariate dispersions&#xA;&#xA;No. of permutations: 999  &#xA;Permutation type: free &#xA;Permutations are unstratified&#xA;Mirrored permutations?: No &#xA;Use same permutation within strata?: No &#xA;&#xA;Response: Distances&#xA;          Df  Sum Sq Mean Sq      F N.Perm Pr(&amp;gt;F)&#xA;Groups     3  223.28  74.427 0.3523    999   0.84&#xA;Residuals  5 1056.34 211.268                     &#xA;&#xA;Pairwise comparisons:&#xA;(Observed p-value below diagonal, permuted p-value above diagonal)&#xA;             Apple    Avocado     Banana Orange&#xA;Apple              8.0000e-03 8.5600e-01  0.004&#xA;Avocado 9.2864e-31            7.9800e-01  0.008&#xA;Banana  6.2093e-01 5.6617e-01             0.797&#xA;Orange  1.5060e-31 4.0863e-27 5.6544e-01       &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Below I have inserted a plot of groups and distances to the group centroid [&lt;code&gt;plot(model&lt;/code&gt;)], and a boxplot of the distances to centroid for each group [&lt;code&gt;boxplot(model)&lt;/code&gt;].&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/wwLPy.png&quot; alt=&quot;&quot; title=&quot;Hosted by imgur.com&quot; /&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/WdMFi.png&quot; alt=&quot;&quot; title=&quot;Hosted by imgur.com&quot; /&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;References&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Anderson, M. J. (2006) Distance-based tests for homogeneity of multivariate dispersions. Biometrics 62(1): 245–253. &lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Edit&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;On a second thought I would recommend also a more descriptive approach using linear discriminant analysis (LDA) that could help not only to visualise the spread of objects around their group centroids, but also to find the features that contribute to this configuration. The &lt;code&gt;ade4&lt;/code&gt; package contains the versatile function &lt;code&gt;discrimin()&lt;/code&gt; that does this as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(ade4)&#xA;discr &amp;lt;- discrimin(dudi.pca(myData[,2:5], scan = FALSE), myData[,1], scan = FALSE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Note that the LDA is based on a PCA (function &lt;code&gt;dudi.pca()&lt;/code&gt;) of the data so you will need to consider its properties when applying it to your task.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/YSepG.png&quot; alt=&quot;&quot; title=&quot;Hosted by imgur.com&quot; /&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The top left plot represents the coefficients of the linear discriminant functions on the first two axes of the DA. The &quot;Cos(variates, canonical variates)&quot; plot shows the covariances between the object properties projected on the first two axes. Then, on the bottom left is the eigenvalue screeplot demonstrating the contribution of each axis to the variation. The main plot, &quot;Scores and Classes&quot;, shows the projections of the individuals on the plane defied by the axes of the DA. Groups are displayed by ellipses where the centers are the means and the ellipses show the variance within the objects. All plots are the result of the &lt;code&gt;plot(discr)&lt;/code&gt; command.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A randomisation test (and plot via &lt;code&gt;plot(randtest.discrimin(discr))&lt;/code&gt;) of the eigenvalue significance is also available:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;randtest.discrimin(discr)&#xA;Monte-Carlo test&#xA;Call: randtest.discrimin(xtest = discr)&#xA;&#xA;Observation: 0.5074292 &#xA;&#xA;Based on 999 replicates&#xA;Simulated p-value: 0.052 &#xA;Alternative hypothesis: greater &#xA;&#xA;    Std.Obs Expectation    Variance &#xA;1.549410152 0.376074589 0.007187167 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/fLsgv.png&quot; alt=&quot;&quot; title=&quot;Hosted by imgur.com&quot; /&gt;&lt;/p&gt;&#xA;" OwnerUserId="3467" LastEditorUserId="3467" LastEditorDisplayName="" LastEditDate="2011-04-06T11:46:01.810" LastActivityDate="2011-04-06T11:46:01.810" CommentCount="2" />
  <row Id="9207" PostTypeId="2" ParentId="9159" CreationDate="2011-04-05T16:31:03.737" Score="1" ViewCount="" Body="&lt;p&gt;The term &quot;gold standard&quot; has been used a lot with respect to No Child Left Behind. One important component of the legislation is that it established the need for the education field to move towards interventions and practices that have been demonstrated to be effective in rigorous studies. In the NCLB materials, research designs have been classified into three categories based on the strength of the conclusions they warrant. Randomized control trials (i.e. randomized or true experiments) are the only designs that provide &quot;strong evidence&quot; of effectiveness. Thus they have been dubbed the &quot;gold standard&quot; in educational research. Well-designed true experiments are given higher precedence in the awarding of federal research grants. &lt;/p&gt;&#xA;" OwnerUserId="4041" LastActivityDate="2011-04-05T16:31:03.737" />
  <row Id="9208" PostTypeId="1" CreationDate="2011-04-05T16:40:19.540" Score="3" ViewCount="87" Body="&lt;p&gt;I was wondering how statistics and decision theory are related?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It looks to me all the statistics problems/tasks can be formulated in decision theory. Also problems in decision theory can be formulated in statistics/probability problems, or in deterministic way.  So is statistics just a part of the problems studied in decision theory? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or are they just two theories with overlapping and neither falls completely inside the other?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I have to admit that I don't have a systematic big picture of what topics statistics theory and decision theory are covered respectively, and would like to here some of your point of view.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks and regards!&lt;/p&gt;&#xA;" OwnerUserId="1005" LastEditorUserId="1005" LastEditorDisplayName="" LastEditDate="2011-04-05T16:44:44.153" LastActivityDate="2011-04-05T23:58:15.773" Title="What is the relation between statistics theory and decision theory?" Tags="&lt;mathematical-statistics&gt;&lt;statistics-theory&gt;&lt;decision-theory&gt;" AnswerCount="1" />
  <row Id="9209" PostTypeId="5" CreationDate="2011-04-05T16:46:53.373" Score="0" ViewCount="" Body="" OwnerUserId="-1" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-04-05T16:46:53.373" LastActivityDate="2011-04-05T16:46:53.373" />
  <row Id="9210" PostTypeId="4" CreationDate="2011-04-05T16:46:53.373" Score="0" ViewCount="" Body="Used to tag questions (often Community Wiki) where a collection of replies is requested, such as a list of references, of quotations, etc." OwnerUserId="-1" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-04-05T16:46:53.373" LastActivityDate="2011-04-05T16:46:53.373" />
  <row Id="9211" PostTypeId="2" ParentId="9202" CreationDate="2011-04-05T16:49:23.473" Score="9" ViewCount="" Body="&lt;p&gt;BUGS/OpenBugs has a peculiar build system which made compiling the code difficult to impossible on some systems --- such as Linux (and IIRC OS X) where people had to resort to Windows emulation etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Jags, on the other hand, is a completely new project written with standard GNU tools and hence portable to just about anywhere --- and therefore usable anywhere.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So in short, if your system is Windows then you do a choice, and a potential cost of being stuck to Bugs if you ever move.  If you are not on Windows, then Jags is likely to be the better choice.&lt;/p&gt;&#xA;" OwnerUserId="334" LastActivityDate="2011-04-05T16:49:23.473" CommentCount="2" />
  <row Id="9212" PostTypeId="2" ParentId="9208" CreationDate="2011-04-05T17:01:27.370" Score="3" ViewCount="" Body="&lt;p&gt;Statistical decision theory is a subset of statistical theory. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Exploratory statistics is not decision theory but it is statistics. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A theory about how to make (good) decisions is certainly much wider than statistical decision theory. For example, making a good decision in society may have more relation with psychology or even philosophy than with statistics, don't you think?&lt;/p&gt;&#xA;" OwnerUserId="223" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-04-05T17:57:08.913" LastActivityDate="2011-04-05T17:57:08.913" CommentCount="6" />
  <row Id="9213" PostTypeId="1" AcceptedAnswerId="9219" CreationDate="2011-04-05T17:10:13.750" Score="0" ViewCount="38" Body="&lt;p&gt;Previosuly I have asked how to compute some conditional probabilities, but I am missing this particular case:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;lets say we now have 3 variables:&#xA;$T$, $L$, $E$:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  T     L&#xA;   \   /&#xA;     E&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So &#xA;$E$ depends on $T$ and $E$ depends on $L$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have these probabilities:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(T) =$ 0.0104&lt;br&gt;&#xA;$P(L) =$ 0.055&lt;br&gt;&#xA;$P(E) =$ 0.0648  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(E|L,T) =$ 1.0&lt;br&gt;&#xA;$P(E|L,\lnot T) =$ 1.0&lt;br&gt;&#xA;$P(E|\lnot L,T) =$ 1.0&lt;br&gt;&#xA;$P(E|\lnot L,\lnot T) =$ 0.0  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How is modified the probability of $T$ and $L$ when we observe that $P(E)$ = 1?&lt;/p&gt;&#xA;" OwnerUserId="3681" LastEditorUserId="3681" LastEditorDisplayName="" LastEditDate="2011-04-05T18:59:11.100" LastActivityDate="2011-04-05T19:05:10.680" Title="variable depending on 2 variables conditional probability" Tags="&lt;conditional-probability&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="9214" PostTypeId="1" CreationDate="2011-04-05T17:36:40.303" Score="1" ViewCount="23" Body="&lt;p&gt;Latent Semantic Indexing seems to work well. E.g. it is independent of language, etc. &#xA;However, it appears to use the similarity of frequencies of terms in the corpus to categorize them. If this understanding is correct:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Is there a way to measure the size of the dataset that will give optimal performance?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="2192" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-06T04:11:50.650" LastActivityDate="2011-04-06T04:11:50.650" Title="Estimating sample size required for optimal performance of Latent Semantic Indexing?" Tags="&lt;svd&gt;" />
  <row Id="9215" PostTypeId="1" CreationDate="2011-04-05T18:02:10.560" Score="1" ViewCount="27" Body="&lt;p&gt;&lt;a href=&quot;http://stats.stackexchange.com/q/1521/2192&quot;&gt;Data Mining and Statistical Analysis&lt;/a&gt; has a general discussion on stats vs data mining. If I may, narrow down the question a bit - are there any general demarcations that allows you to decide which approach is more suited for using - LDA or ARM? (other than appreciation for the maths in the former)&lt;/p&gt;&#xA;" OwnerUserId="2192" LastActivityDate="2011-04-05T18:02:10.560" Title="Choosing between Latent dirchlet allocation and Association rule mining" Tags="&lt;data-mining&gt;" FavoriteCount="1" />
  <row Id="9216" PostTypeId="2" ParentId="5926" CreationDate="2011-04-05T18:33:24.237" Score="2" ViewCount="" Body="&lt;p&gt;I would recommend to conduct a MANOVA whenever you are comparing groups on multiple DVs that have been measured on each observation. The data are multivariate, and a MV procedure should be used to model that known data situation. I do not believe in deciding whether to use it on the basis of that correlation. So I would use MANOVA for either of those situations. I would recommend reading the relevant portions of the following conference paper by Bruce Thompson (ERIC ID ED429110). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;p.s. I believe the 'conceptually related' quote comes from the Stevens book. &lt;/p&gt;&#xA;" OwnerUserId="4041" LastActivityDate="2011-04-05T18:33:24.237" />
  <row Id="9217" PostTypeId="1" CreationDate="2011-04-05T18:50:34.460" Score="2" ViewCount="83" Body="&lt;p&gt;I plan to conduct a research study about Interaction Designers. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Prior to starting on sampling methods I am finding it very hard to get my hands on some demographics and statistics, e.g. How many people are registered to be employed as Interaction Designers, User Experience Designers etc. in a particular territory (UK in my case).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help regarding this would be highly appreciated.&lt;/p&gt;&#xA;" OwnerUserId="4043" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-04-08T20:41:08.157" LastActivityDate="2011-04-08T20:41:08.157" Title="Estimating the number of registered interaction designers in a given territory" Tags="&lt;sampling&gt;&lt;survey&gt;&lt;dataset&gt;&lt;job&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="1" />
  <row Id="9218" PostTypeId="1" AcceptedAnswerId="9221" CreationDate="2011-04-05T18:58:01.350" Score="1" ViewCount="38" Body="&lt;pre&gt;&lt;code&gt;str(test)&#xA;'data.frame':   767 obs. of  2 variables:&#xA;$ datefield: Ord.factor w/ 59 levels &amp;quot;1984-04-01&amp;quot;&amp;lt;&amp;quot;1984-07-01&amp;quot;&amp;lt;..: 1 2 3 4 5 6 7 8 9 10  &#xD;&#xA;    $ somevar : num  43.7 55.6 43.5 54.1 42.8 ...&#xA;&#xA;&amp;gt; str(Italy)&#xA;'data.frame':   1008 obs. of  2 variables:&#xA;$ year: Ord.factor w/ 48 levels &amp;quot;1951&amp;quot;&amp;lt;&amp;quot;1952&amp;quot;&amp;lt;..: 1 1 1 1 1 1 1 1 1 1 ...&#xD;&#xA;    $ gdp : num  8.56 12.26 9.59 8.12 5.54 ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I turned the datefield into an ordered factor because I try to reproduce an example with own data. Now I wonder what the difference between is, or put it differently, what does 1 2 3 4 5 6 7 8 9 10  mean opposed to 1 1 1 1 1 1 1 1 1 1 ... ?&lt;/p&gt;&#xA;" OwnerUserId="704" LastActivityDate="2011-04-05T19:16:16.503" Title="What's the difference between these ordered factors?" Tags="&lt;r&gt;&lt;ordered-variables&gt;" AnswerCount="1" />
  <row Id="9219" PostTypeId="2" ParentId="9213" CreationDate="2011-04-05T19:05:10.680" Score="1" ViewCount="" Body="&lt;p&gt;There is a contradiction. Among ($T \land L \land E$, $T \land L \land \lnot E$, $T \land \lnot L \land E$, $T \land \lnot L \land \lnot E$, $\lnot T \land L \land E$, $\lnot T \land L \land \lnot E$, $\lnot T \land \lnot L \land E$, $\lnot T \land \lnot L \land \lnot E$) the $\lnot T \land \lnot L \land \lnot E$ is the only $\lnot E$ case that has a non zero probability. According to $Pr(E)= 0.0648$ this should have a probability of $1-0.0648=0.9452$. You also specified however that $Pr(T)=0.0104$ thus $Pr(\lnot T)=0.8996$. As the previously considered $\lnot T \land \lnot L \land \lnot E$ case is a member of the $\lnot T$ cases this should have a probability less then or equal to $Pr(\lnot T)=0.8996$. Alas $0.9452 \nleq 0.8996$, so your [$P(T) =$ 0.0104, $P(L) =$ 0.055, $P(E) =$ 0.0648, $P(E|L,T) =$ 1.0, $P(E|L,T) =$ 1.0, $P(E|\lnot L,T) =$ 1.0, $P(E|\lnot L,\lnot T) =$ 0.0] statements can not be simultaneously true.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I assume that your question is a formulation of a real problem to solve. So you have presumably not used the correct symbolic way to denote the problem. Maybe you could describe the problem in more detail in words?&#xA;Maybe you could separate which of your formulas correspond to facts, measurements that may be inaccurate or observations.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastActivityDate="2011-04-05T19:05:10.680" CommentCount="2" />
  <row Id="9220" PostTypeId="1" CreationDate="2011-04-05T19:10:30.120" Score="11" ViewCount="232" Body="&lt;p&gt;Assume you are given two objects whose exact locations are unknown, but are distributed according to normal distributions with known parameters (e.g. $a \sim N(m, s)$ and $b \sim N(v, t))$. We can assume these are both bivariate normals, such that the positions are described by a distribution over $(x,y)$ coordinates (i.e. $m$ and $v$ are vectors containing the expected $(x,y)$ coordinates for $a$ and $b$ respectively). We will also assume the objects are independent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone know if the distribution of the squared euclidean distance between these two objects is a known parametric distribution? Or how to derive the pdf/cdf for this function analytically?&lt;/p&gt;&#xA;" OwnerUserId="1913" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-04-06T13:00:13.443" LastActivityDate="2011-05-12T23:43:36.363" Title="What is the distribution of the euclidean distance between two normally distributed random variables" Tags="&lt;normal-distribution&gt;&lt;distance-functions&gt;" AnswerCount="2" CommentCount="7" FavoriteCount="2" />
  <row Id="9221" PostTypeId="2" ParentId="9218" CreationDate="2011-04-05T19:16:16.503" Score="2" ViewCount="" Body="&lt;p&gt;When you convert a variable to a factor variable using the &lt;code&gt;factor&lt;/code&gt; function (without using special arguments) the original values are substituted with the codes 1, 2, 3 ..., and the original values are assigned to these codes as labels. See &lt;code&gt;?factor&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your 1 2 3 4 5 6 7 8 9 10 means that first 10 the values of datefield were the smallest values of the datefield variable, and each of them was unique (only appearing once).&#xA;The 1 1 1 1 1 1 1 1 1 1 means that the first 10 values of year was 1951, and that 1951 was the lowest value of the year variable.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastActivityDate="2011-04-05T19:16:16.503" />
  <row Id="9222" PostTypeId="2" ParentId="9201" CreationDate="2011-04-05T19:24:41.770" Score="5" ViewCount="" Body="&lt;p&gt;There's a thread on the R-help list about this; see for example: &lt;a href=&quot;http://tolstoy.newcastle.edu.au/R/e8/help/09/12/9200.html&quot; rel=&quot;nofollow&quot;&gt;http://tolstoy.newcastle.edu.au/R/e8/help/09/12/9200.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first suggestion there is to repeat the test a large of number of times with different jittering and then combine the p-values to get an overall p-value, either by taking an average or a maximum.  They also suggest that a straightforward permutation test could be used instead (of the two, that's what I'd prefer).  See the question &lt;a href=&quot;http://stats.stackexchange.com/q/6127/3601&quot;&gt;Which permutation test implementation in R to use instead of t-tests (paired and non-paired)?&lt;/a&gt; for some examples of permutation tests.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Elsewhere in that thread, Greg Snow writes:&#xA;Adding random noise to data in order to avoid a warning is like removing the batteries from a smoke detector to silence it rather than investigating the what is causing the alarm to go off. &#xA;(See &lt;a href=&quot;http://tolstoy.newcastle.edu.au/R/e8/help/09/12/9195.html&quot; rel=&quot;nofollow&quot;&gt;http://tolstoy.newcastle.edu.au/R/e8/help/09/12/9195.html&lt;/a&gt; )&lt;/p&gt;&#xA;" OwnerUserId="3601" LastEditorUserId="3601" LastEditorDisplayName="" LastEditDate="2011-04-06T14:41:20.167" LastActivityDate="2011-04-06T14:41:20.167" CommentCount="2" />
  <row Id="9223" PostTypeId="2" ParentId="9217" CreationDate="2011-04-05T19:32:52.573" Score="2" ViewCount="" Body="&lt;p&gt;The recent census (27/March/2011) in the UK had &quot;full and specific job title&quot; as personal question #34. So the Office for National Statistics must have information on this.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastActivityDate="2011-04-05T19:32:52.573" CommentCount="2" />
  <row Id="9224" PostTypeId="2" ParentId="9201" CreationDate="2011-04-05T19:40:42.630" Score="3" ViewCount="" Body="&lt;p&gt;(disclaimer: I didn't check the code, my answer is just based on your description)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have the feeling that what you want to do is a &lt;strong&gt;really bad idea&lt;/strong&gt;. Wilcoxon is a resampling (or randomization) test for ranks. That is, it takes the rank of the values and compares these ranks to all possible permutations of the ranks (see e.g., &lt;a href=&quot;http://www2.statistics.com/resources/glossary/w/wmwutest.php&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, as you realized, ties are pretty bad as you don't get ranks out of them. However, adding random noise (jitter) to your data will transform all ranks, so that they have random ranks! That is, it distorts your data severely.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore: &lt;strong&gt;It is wrong to do so&lt;/strong&gt;.&lt;/p&gt;&#xA;" OwnerUserId="442" LastActivityDate="2011-04-05T19:40:42.630" CommentCount="1" />
  <row Id="9225" PostTypeId="1" CreationDate="2011-04-05T19:49:48.360" Score="9" ViewCount="116" Body="&lt;p&gt;What does it mean for a study to be over-powered? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My impression is that it means that your sample sizes are so large that you have the power to detect  minuscule effect sizes. These effect sizes are perhaps so small that they are more likely to result from slight biases in the sampling process than a (not necessarily direct) causal connection between the variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this the correct intuition? If so, I don't see what the big deal is, as long as the results are interpreted in that light and you manually check and see whether the estimated effect size is large enough to be &quot;meaningful&quot; or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Am I missing something? Is there a better recommendation as to what to do in this scenario?&lt;/p&gt;&#xA;" OwnerUserId="3836" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-06T03:33:40.073" LastActivityDate="2011-04-06T03:33:40.073" Title="What does it mean for a study to be over-powered?" Tags="&lt;statistical-significance&gt;&lt;sample-size&gt;&lt;effect-size&gt;&lt;power-analysis&gt;&lt;power&gt;" AnswerCount="3" CommentCount="1" />
  <row Id="9226" PostTypeId="2" ParentId="9225" CreationDate="2011-04-05T19:56:19.610" Score="0" ViewCount="" Body="&lt;p&gt;Everything you've said makes sense (although I don't know what &quot;big deal&quot; you're referring to), and I esp. like your point about effect sizes as opposed to statistical significance.  One other consideration is that some studies require the allocation of scarce resources to obtain the participation of each case, and so one wouldn't want to overdo it.&lt;/p&gt;&#xA;" OwnerUserId="2669" LastActivityDate="2011-04-05T19:56:19.610" CommentCount="1" />
  <row Id="9228" PostTypeId="1" CreationDate="2011-04-05T20:34:38.033" Score="7" ViewCount="111" Body="&lt;p&gt;In SVM (linear kernel) classification analyses of a data-set of gene expression (~400 variables/genes) for ~25 each of cases and controls, I find that the gene expression-based classifiers have very good performance characteristics. The cases and controls do not differ significantly for a number of categorical and continuous clinical/demographic variables (as per Fisher's exact or t tests), but they do differ significantly for age.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a way to show that the classification analysis results are or are not influenced by age?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am thinking of reducing the gene expression data to principal components, and doing a Spearman correlation analysis of the components against age.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this is a reasonable approach? Alternately, can I check for correlation between age and class-membership probability values obtained in the SVM analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="4045" LastEditorUserId="4045" LastEditorDisplayName="" LastEditDate="2011-04-05T20:56:34.987" LastActivityDate="2011-04-06T22:23:28.577" Title="Correlating continuous clinical variables and gene expression data" Tags="&lt;correlation&gt;&lt;classification&gt;&lt;pca&gt;&lt;continuous-data&gt;" AnswerCount="1" CommentCount="5" FavoriteCount="1" />
  <row Id="9229" PostTypeId="2" ParentId="9217" CreationDate="2011-04-05T20:58:43.643" Score="1" ViewCount="" Body="&lt;p&gt;One of the problems of a national census is that it takes a long time to get all the data together so it may be a while until it appears but if you want census data try&#xA;&lt;a href=&quot;https://www.census.ac.uk&quot; rel=&quot;nofollow&quot;&gt;https://www.census.ac.uk&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="3597" LastActivityDate="2011-04-05T20:58:43.643" CommentCount="1" />
  <row Id="9231" PostTypeId="2" ParentId="9201" CreationDate="2011-04-05T22:15:54.327" Score="2" ViewCount="" Body="&lt;p&gt;You've asked several people what you should do now.  In my view, what you should do now is accept that the proper p-value here is 1.000.  Your groups don't differ.  &lt;/p&gt;&#xA;" OwnerUserId="686" LastActivityDate="2011-04-05T22:15:54.327" CommentCount="4" />
  <row Id="9233" PostTypeId="1" CreationDate="2011-04-05T22:30:48.447" Score="7" ViewCount="226" Body="&lt;p&gt;Here is some context. I am interested in determining how two environmental variables (temperature, nutrient levels) impact the mean value of a response variable over a 11 year period. Within each year, there is data from over 100k locations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The goal is to determine whether, over the 11 year period, the mean value of the response variables has responded to changes in environmental variables (e.g. warmer temperature + more nutrients would = greater response).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unfortunately, since the response is the mean value (without looking at the mean, just regular inter-annual variation will swamp the signal), the regression will be 11 data points (1 mean value per year), with 2 explanatory variables. To me even a linear positive regression will be hard to consider as meaningful given that the dataset is so small (does not even meet the nominal 40 points/variable unless the relationship is super strong).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Am I right to make this assumption? Can anyone offer any other thoughts/perspectives that I may be missing?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: Some caveats: There is no way to get more data without waiting additional years. So the data that is available is what we really have to work with. &lt;/p&gt;&#xA;" OwnerUserId="1451" LastEditorUserId="179" LastEditorDisplayName="" LastEditDate="2011-04-06T06:52:16.380" LastActivityDate="2011-04-24T11:34:08.887" Title="Are short time series worth modelling?" Tags="&lt;time-series&gt;&lt;regression&gt;&lt;sample-size&gt;&lt;small-sample&gt;" AnswerCount="6" CommentCount="2" FavoriteCount="2" />
  <row Id="9234" PostTypeId="2" ParentId="9233" CreationDate="2011-04-05T23:14:11.423" Score="2" ViewCount="" Body="&lt;p&gt;I would say that the validity of the test has less to do with the number of data points and more to do with the validity of the assumption that you have the correct model.  For example, the regression analysis that is used to generate a standard curve may be based on only 3 standards (low, med, and high) but the result is highly valid since there is strong evidence that the response is linear between the points.  On the other hand, even a regression with 1000s of data points will be flawed if the wrong model is applied to the data.  In the first case any variation between the model predictions and the actual data is due to random error.  In the second case some of the variation between the model predictions and the actual data is due to bias from choosing the wrong model. &lt;/p&gt;&#xA;" OwnerUserId="4048" LastActivityDate="2011-04-05T23:14:11.423" />
  <row Id="9236" PostTypeId="2" ParentId="9233" CreationDate="2011-04-05T23:25:14.203" Score="4" ViewCount="" Body="&lt;p&gt;The small number of data points limits what kinds of models you may fit on your data. However it does not necessarily mean that it would make no sense to start modelling. With few data you will only be able to detect associations if the effects are strong and the scatter is weak.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's an other question what kind of model suits your data. You used the word 'regression' in the title. The model should to some extent reflect what you know about the phenomenon. This seems to be an ecological setting, so the previous year may be influential as well. &lt;/p&gt;&#xA;" OwnerUserId="3911" LastActivityDate="2011-04-05T23:25:14.203" />
  <row Id="9237" PostTypeId="1" CreationDate="2011-04-05T23:50:43.807" Score="5" ViewCount="79" Body="&lt;p&gt;I have several dependent variables that are measures of racial disproportionality; I've calculated them as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;% of events caused by racial minority group / % of events caused by racial majority group&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a dependent variable for each racial minority group in my sample. I am running longitudinal Generalized Estimating Equations (GEE) on these models, however I am somewhat stumped as to which family is appropriate for these dependent variables. The probability range for my ratios are truncated at 0, as it's not possible to have negative values in my DVs. This makes me question the validity of using a Gaussian family for my models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The idea behind these variables is that a ratio greater than 1 indicates some level of greater burden of events that a given racial minority is bearing compared to the racial majority, and a ratio less than 1 indicates the opposite.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What would be the most appropriate family to use for my GEE regressions?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;EDIT:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I misspoke about the racial disproportionality measure I was using. The correct formula is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;% events by minority / % of total enrollment that is minority OVER&#xA;% events by non-minority / % of total enrollment that is non-minority&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because they are ratios, the number of observations with value less than 1 is comparable to the number of observations greater than 1, with the lower bound being 0 and the upper bound being non-bounded. Looking at the histograms of my response variables, they definitely seem to fit a negative binomial distribution better than the normal. The QIC (GEE adjustment to AIC) confirms this suspicion. My questions now are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Can I trust this evidence to move forward with the negative binomial family?&lt;/li&gt;&#xA;&lt;li&gt;If so, how do I possibly interpret the exponentiated coefficients from the resulting models? They don't see to be Incidence Rate Ratios, as one would interpret them to be from count variables...&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="3309" LastEditorUserId="3309" LastEditorDisplayName="" LastEditDate="2011-04-27T19:17:16.593" LastActivityDate="2011-04-27T19:17:16.593" Title="Distribution family for a ratio dependent variable in a generalized estimating equation" Tags="&lt;r&gt;&lt;regression&gt;&lt;link-function&gt;&lt;gee&gt;" AnswerCount="1" CommentCount="4" FavoriteCount="1" />
  <row Id="9239" PostTypeId="2" ParentId="9225" CreationDate="2011-04-06T00:22:28.443" Score="2" ViewCount="" Body="&lt;p&gt;In medical research trials may be unethical if they recruit too many patients. For example if the goal is to decide which treatment is better it's not ethical any more to treat patients with the worse treatment after it was established to be inferior. Increasing the sample size would, of course, give you a more accurate estimate of the effect size, but you may have to stop well before the effects of factors like &quot;slight biases in the sampling process&quot; appear.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It may also be unethical to spend public money of sufficiently confirmed research.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-06T03:30:28.593" LastActivityDate="2011-04-06T03:30:28.593" />
  <row Id="9240" PostTypeId="1" CreationDate="2011-04-06T00:45:50.147" Score="5" ViewCount="123" Body="&lt;p&gt;There is a web service where I can request information about a random item. &#xA;For every request each item has an equal chance of being returned.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can keep requesting items and record the number of duplicates and unique. How can I use this data to estimate the total number of items?&lt;/p&gt;&#xA;" OwnerUserId="4049" LastEditorUserId="4049" LastEditorDisplayName="" LastEditDate="2011-04-06T10:45:05.683" LastActivityDate="2011-04-09T14:40:52.987" Title="Estimating population size from the frequency of sampled duplicates and uniques" Tags="&lt;probability&gt;&lt;population&gt;&lt;recapture&gt;" AnswerCount="3" CommentCount="1" FavoriteCount="1" />
  <row Id="9241" PostTypeId="2" ParentId="9240" CreationDate="2011-04-06T00:50:08.733" Score="3" ViewCount="" Body="&lt;p&gt;You can use &lt;a href=&quot;http://en.wikipedia.org/wiki/Capture-recapture&quot; rel=&quot;nofollow&quot;&gt;the capture-recapture method&lt;/a&gt;, also implemented as &lt;a href=&quot;http://cran.r-project.org/web/packages/Rcapture/index.html&quot; rel=&quot;nofollow&quot;&gt;the Rcapture R package&lt;/a&gt;.&lt;hr/&gt;&#xA;Here is an example, coded in R. Let's assume that the web service has N=1000 items. We will make n=300 requests. Generate a random sample where, numbering the elements from 1 to k, where k is how many different items we saw.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;N = 1000; population = 1:N # create a population of the integers from 1 to 1000&#xA;n = 300 # number of requests&#xA;set.seed(20110406)&#xA;observation = as.numeric(factor(sample(population, size=n,&#xA;  replace=TRUE))) # a random sample from the population, renumbered&#xA;table(observation) # a table useful to see, not discussed&#xA;k = length(unique(observation)) # number of unique items seen&#xA;(t = table(table(observation)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The result of the simulation is&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  1   2   3 &#xA;234  27   4 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;thus among the 300 requests there were 4 items seen 3 times, 27 items seen twice, and 234 items seen only once.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now estimate N from this sample:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(Rcapture)&#xA;X = data.frame(t)&#xA;X[,1]=as.numeric(X[,1])&#xA;desc=descriptive(X, dfreq=TRUE, dtype=&quot;nbcap&quot;, t=300)&#xA;desc # useful to see, not discussed&#xA;plot(desc) # useful to see, not discussed&#xA;cp=closedp.0(X, dfreq=TRUE, dtype=&quot;nbcap&quot;, t=300, trace=TRUE)&#xA;cp&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The result:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Number of captured units: 265 &#xA;&#xA;Abundance estimations and model fits:&#xA;                  abundance       stderr      deviance   df           AIC&#xA;M0**                  265.0          0.0  2.297787e+39  298  2.297787e+39&#xA;Mh Chao              1262.7        232.5  7.840000e-01    9  5.984840e+02&#xA;Mh Poisson2**         265.0          0.0  2.977883e+38  297  2.977883e+38&#xA;Mh Darroch**          553.9         37.1  7.299900e+01  297  9.469900e+01&#xA;Mh Gamma3.5**  5644623606.6  375581044.0  5.821861e+05  297  5.822078e+05&#xA;&#xA; ** : The M0 model did not converge&#xA; ** : The Mh Poisson2 model did not converge&#xA; ** : The Mh Darroch model did not converge&#xA; ** : The Mh Gamma3.5 model did not converge&#xA;Note: 9 eta parameters has been set to zero in the Mh Chao model&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Thus only the Mh Chao model converged, it estimated $\hat{N}$=1262.7.&#xA;&lt;hr&gt;&#xA;EDIT: To check the reliability of the above method I ran the above code on 10000 generated samples. The Mh Chao model converged every time. Here is the summary:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; round(quantile(Nhat, c(0, 0.025, 0.25, 0.50, 0.75, 0.975, 1)), 1)&#xA;    0%   2.5%    25%    50%    75%  97.5%   100% &#xA; 657.2  794.6  941.1 1034.0 1144.8 1445.2 2162.0 &#xA;&amp;gt; mean(Nhat)&#xA;[1] 1055.855&#xA;&amp;gt; sd(Nhat)&#xA;[1] 166.8352&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="3911" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-09T14:40:52.987" LastActivityDate="2011-04-09T14:40:52.987" CommentCount="8" />
  <row Id="9242" PostTypeId="1" AcceptedAnswerId="9247" CreationDate="2011-04-06T00:51:24.083" Score="2" ViewCount="252" Body="&lt;p&gt;What does it mean if the F value in one-way ANOVA is less than 1?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Remember the F-ratio is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{\sigma^2+\frac{r\times\sum_{i=1}^t \tau_i^2}{t-1}}{\sigma^2}$$    &lt;/p&gt;&#xA;" OwnerUserId="3903" LastEditorUserId="3903" LastEditorDisplayName="" LastEditDate="2011-04-06T05:57:59.773" LastActivityDate="2011-04-06T15:43:48.640" Title="What is the meaning of an F value less than 1 in one-way ANOVA?" Tags="&lt;anova&gt;&lt;experiment-design&gt;" AnswerCount="4" CommentCount="2" FavoriteCount="3" />
  <row Id="9244" PostTypeId="2" ParentId="9233" CreationDate="2011-04-06T01:47:20.040" Score="1" ViewCount="" Body="&lt;p&gt;I've seen ecological datasets with fewer than 11 points, so I would say if you are very careful, you can draw some limited conclusions with your limited data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could also do a power analysis to determine how small an effect you could detect, given the parameters of your experimental design.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You also might not need to throw out the extra variation per year if you do some careful analysis&lt;/p&gt;&#xA;" OwnerUserId="2817" LastActivityDate="2011-04-06T01:47:20.040" CommentCount="1" />
  <row Id="9245" PostTypeId="2" ParentId="9242" CreationDate="2011-04-06T03:04:46.387" Score="0" ViewCount="" Body="&lt;p&gt;Doesn't it just mean that your treatment was almost completely ineffective?&lt;/p&gt;&#xA;" OwnerUserId="1679" LastActivityDate="2011-04-06T03:04:46.387" CommentCount="4" />
  <row Id="9246" PostTypeId="2" ParentId="9225" CreationDate="2011-04-06T03:19:15.373" Score="4" ViewCount="" Body="&lt;p&gt;I think that your interpretation is incorrect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You say &quot;These effect sizes are perhaps so small as are more likely result from slight biases in the sampling process than a (not necessarily direct) causal connection between the variables&quot; which seems to imply that the P value in an 'over-powered' study is not the same sort of thing as a P value from a 'properly' powered study. That is wrong. In both cases the P value is the probability of obtaining data as extreme as those observed, or more extreme, if the null hypothesis is true.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you prefer the Neyman-Pearson approach, the rate of false positive errors obtained from  the 'over-powered' study is the same as that of a 'properly' powered study if the same alpha value is used for both.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The difference in interpretation that is needed is that there is a different relationship between statistical significance and scientific significance for over-powered studies. In effect, the over-powered study will give a large probability of obtaining significance even though the effect is, as you say, miniscule, and therefore of questionable importance. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As long as results from an 'over-powered' study are appropriately interpreted (and confidence intervals for the effect size help such an interpretation) there is no statistical problem with an 'over-powered' study. In that light, the only criteria by which a study can actually be over-powered  are the ethical and resource allocation issues raised in other answers.&lt;/p&gt;&#xA;" OwnerUserId="1679" LastActivityDate="2011-04-06T03:19:15.373" CommentCount="3" />
  <row Id="9247" PostTypeId="2" ParentId="9242" CreationDate="2011-04-06T03:53:51.367" Score="8" ViewCount="" Body="&lt;p&gt;The F ratio is a statistic.&#xA;When the null hypothesis of no group differences is true, then the expected value of the numerator and denominator of the F ratio will be equal. As a consequence, the expected value of the F ratio when the null hypothesis is true is also close to one (actually it's not exactly one, because of the properties of expected values of ratios).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When the null hypothesis is false and there are group differences between the means, the expected value of the numerator will be larger than the denominator.&#xA;As such the expected value of the F ratio will be larger than under the null hypothesis, and will also more likely be larger than one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the point is that both the numerator and denominator are random variables, and so is the F ratio. The F ratio is drawn from a distribution. If we assume the null hypothesis is true we get one distribution, and if we assume that it is false with various assumptions about effect size, sample size, and so forth we get another distribution. We then do a study and get an F value.&#xA;When the null hypothesis is false, it is still possible to get an F ratio less than one.&#xA;The larger the population effect size is (in combination with sample size), the more the F distribution will move to the right, and the less likely we will be to get a value less than one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The following graphic extracted from the &lt;a href=&quot;http://www.psycho.uni-duesseldorf.de/abteilungen/aap/gpower3/&quot; rel=&quot;nofollow&quot;&gt;G-Power3&lt;/a&gt; demonstrates the idea given various assumptions.&#xA;The red distribution is the distribution of F when H0 is true.&#xA;The blue distribution is the distribution of F when H0 is false given various assumptions.&#xA;Note that the blue distribution does include values less than one, yet they are very unlikely.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/LPeQz.png&quot; title=&quot;Hosted by imgur.com&quot; /&gt;&lt;/p&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-06T06:11:21.993" LastActivityDate="2011-04-06T06:11:21.993" />
  <row Id="9248" PostTypeId="5" CreationDate="2011-04-06T04:31:59.587" Score="0" ViewCount="" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditorDisplayName="" LastEditDate="2011-04-06T04:31:59.587" LastActivityDate="2011-04-06T04:31:59.587" />
  <row Id="9249" PostTypeId="4" CreationDate="2011-04-06T04:31:59.587" Score="0" ViewCount="" Body="Logistic regression is a type of regression where the dependent variable is binary. It can be though of as an analogue of linear regression for binary dependent variables." OwnerUserId="-1" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-04-22T07:25:05.140" LastActivityDate="2011-04-22T07:25:05.140" />
  <row Id="9250" PostTypeId="5" CreationDate="2011-04-06T04:34:48.617" Score="0" ViewCount="" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditorDisplayName="" LastEditDate="2011-04-06T04:34:48.617" LastActivityDate="2011-04-06T04:34:48.617" />
  <row Id="9251" PostTypeId="4" CreationDate="2011-04-06T04:34:48.617" Score="0" ViewCount="" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditorDisplayName="" LastEditDate="2011-04-06T04:34:48.617" LastActivityDate="2011-04-06T04:34:48.617" />
  <row Id="9252" PostTypeId="2" ParentId="9203" CreationDate="2011-04-06T07:24:02.930" Score="0" ViewCount="" Body="&lt;p&gt;Quick note, you may find useful discussion of why the formula for estimating the standard deviation for a sample uses $n-1$ as opposed to $n$.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Paul Savory &lt;a href=&quot;http://digitalcommons.unl.edu/cgi/viewcontent.cgi?article=1008&amp;amp;context=imseteach&amp;amp;sei-redir=1#search=%22why+divide+by+n+-+1+standard+deviation%22&quot; rel=&quot;nofollow&quot;&gt;Why divide by (n-1) for sample standard deviation&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Graphpad &lt;a href=&quot;http://www.graphpad.com/faq/viewfaq.cfm?faq=1382&quot; rel=&quot;nofollow&quot;&gt;Why use n-1 when calculating a standard deviation?&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Andrew Hardwick &lt;a href=&quot;http://duramecho.com/Misc/WhyMinusOneInSd.html&quot; rel=&quot;nofollow&quot;&gt;Why there is a Minus One in Standard Deviations&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastActivityDate="2011-04-06T07:24:02.930" />
  <row Id="9253" PostTypeId="1" AcceptedAnswerId="9256" CreationDate="2011-04-06T07:34:56.343" Score="3" ViewCount="68" Body="&lt;p&gt;For the $\nu$-SVM (for both classification and regression cases) the $\nu \in (0;1)$ should be selected.&#xA;The LIBSVM guide suggests to use grid search for identifying the optimal value of the $C$ parameter for $C$-SVM, it also recommends to try following values $C = 2^{-5}, 2^{-3}, \dots, 2^{15}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the question is, are there any recommendations for values of the $\nu$ parameter in case of $\nu$-SVMs?&lt;/p&gt;&#xA;" OwnerUserId="4051" LastActivityDate="2011-04-06T10:06:32.467" Title="$\nu$-svm parameter selection" Tags="&lt;machine-learning&gt;&lt;svm&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="9254" PostTypeId="2" ParentId="423" CreationDate="2011-04-06T08:35:46.377" Score="12" ViewCount="" Body="&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/QZsY3.png&quot; alt=&quot;XKCD: significant&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;'So, uh, we did the green study again and got no link. It was probably a--' 'RESEARCH CONFLICTED ON GREEN JELLY BEAN/ACNE LINK; MORE STUDY RECOMMENDED!'&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;xkcd: &lt;a href=&quot;http://xkcd.com/882/&quot; rel=&quot;nofollow&quot;&gt;significant&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="442" LastActivityDate="2011-04-06T08:35:46.377" CommunityOwnedDate="2011-04-06T08:35:46.377" />
  <row Id="9255" PostTypeId="2" ParentId="8881" CreationDate="2011-04-06T09:30:46.060" Score="1" ViewCount="" Body="&lt;p&gt;The lm() procedure handles the entire range of linear models, not just multiple regression. All you have to do is make sure your predictors are set up to&#xA;be of the right type. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Binary is the special case of nominal where the number of levels is two. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Nomimal variables must be set to mode factor. They can be coerced to factors from character variables by using factor(). Note that linear models use one of the levels as a baseline, so it effectively disappears. By default this will be the first in your list of levels. If you don't specify the order of the levels they will be put in alphabetic order. You can change the order using relevel().&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For ordinal data you need them to be ordered factors. Use ordered() to coerce characters or factors to ordered factors. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For continuous predictors you want the predictor to be a double. Use double() to enforce this. &lt;/p&gt;&#xA;" OwnerUserId="4052" LastActivityDate="2011-04-06T09:30:46.060" />
  <row Id="9256" PostTypeId="2" ParentId="9253" CreationDate="2011-04-06T10:06:32.467" Score="3" ViewCount="" Body="&lt;p&gt;Rather than use a grid search, you can just optimise the hyper-parameters using standard numeric optimisation techniques (e.g. gradient descent).  If you don't have estimates of the gradients, you can use the &lt;a href=&quot;http://en.wikipedia.org/wiki/Nelder-Mead_simplex_method&quot; rel=&quot;nofollow&quot;&gt;Nelder-Mead simplex method&lt;/a&gt;, which doesn't require gradient information and is vastly more efficient than grid-search methods.  I would use the &lt;a href=&quot;http://en.wikipedia.org/wiki/Logit&quot; rel=&quot;nofollow&quot;&gt;logit&lt;/a&gt; function to map the (0;1) range of $\nu$ onto $(-\infty;+\infty)$ to get an unconstrained optimisation problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you really want to use grid search, then just spacing the evaluation points linearly in the range 0 - 1 should be fine.&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2011-04-06T10:06:32.467" CommentCount="5" />
  <row Id="9257" PostTypeId="2" ParentId="9242" CreationDate="2011-04-06T10:54:17.303" Score="3" ViewCount="" Body="&lt;p&gt;Your question in the title is an interesting question that crossed my mind today too. I just want to add a correction. The F-ratio is :&#xA;$$\frac{MS_{treatment}}{MS_{residual}}=\frac{\frac{SS_{treatment}}{t-1}}{\frac{SS_{residual}}{t(r-1)}}$$&#xA;What you wrote is the &#xA;$$\frac{E(MS_{treatment})}{E(MS_{residual})}$$&#xA;While the first fraction &lt;strong&gt;can&lt;/strong&gt; be less than 1, the second fraction cannot be less than 1. But that's not a problem since it's a quotient of expectations. &lt;/p&gt;&#xA;" OwnerUserId="3454" LastActivityDate="2011-04-06T10:54:17.303" />
  <row Id="9258" PostTypeId="2" ParentId="9190" CreationDate="2011-04-06T10:55:04.980" Score="3" ViewCount="" Body="&lt;p&gt;I've also found good material at The Gallery of Data Visualization: The Best and Worst of Statistical Graphics, at&lt;br&gt;&#xA;&lt;a href=&quot;http://www.datavis.ca/gallery/index.php&quot; rel=&quot;nofollow&quot;&gt;http://www.datavis.ca/gallery/index.php&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2669" LastActivityDate="2011-04-06T10:55:04.980" CommunityOwnedDate="2011-04-06T10:55:04.980" CommentCount="1" />
  <row Id="9259" PostTypeId="1" CreationDate="2011-04-06T10:56:43.190" Score="8" ViewCount="62" Body="&lt;p&gt;I am running a GAM-based regression using the R package &lt;a href=&quot;http://cran.r-project.org/web/packages/gamlss/index.html&quot; rel=&quot;nofollow&quot;&gt;gamlss&lt;/a&gt; and assuming a zero-inflated beta distribution of the data. I have only a single explanatory variable in my model, so it's basically: &lt;code&gt;mymodel = gamlss(response ~ input, family=BEZI)&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The algorithm gives me the coefficient $k$ for the impact of the explanatory variable into the mean ($\mu$) and the associated p-value for $k(\text{input})=0$, something like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Mu link function:  logit                                               &#xA;Mu Coefficients:                                                      &#xA;              Estimate  Std. Error  t value   Pr(&amp;gt;|t|)                  &#xA;(Intercept)  -2.58051     0.03766  -68.521  0.000e+00                  &#xA;input        -0.09134     0.01683   -5.428  6.118e-08&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As you can see in the above example, the hypothesis of $k(\text{input})=0$ is rejected with high confidence. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I then run the null model: &lt;code&gt;null = gamlss(response ~ 1, family=BEZI)&lt;/code&gt; and compare the likelihoods using a likelihood-ratio test:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;p=1-pchisq(-2*(logLik(null)[1]-logLik(mymodel)[1]), df(mymodel)-df(null)).&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In a number of cases, I get $p&amp;gt;0.05$ even when the coefficients at input are reported to be highly significant (as above). I find this quite unusual -- at least it never happened in my experience with linear or logistic regression (in fact, this also never happened when I was using zero-adjusted gamma with gamlss). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is: can I still trust the dependence between response and input when this is the case?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks!  &lt;/p&gt;&#xA;" OwnerUserId="3945" LastEditorUserId="3945" LastEditorDisplayName="" LastEditDate="2011-04-06T12:50:42.720" LastActivityDate="2011-04-06T12:50:42.720" Title="Significance of (GAM) regression coefficients when model likelihood is not significantly higher than null" Tags="&lt;nonlinear-regression&gt;" CommentCount="1" FavoriteCount="1" />
  <row Id="9260" PostTypeId="1" AcceptedAnswerId="9265" CreationDate="2011-04-06T11:13:26.087" Score="1" ViewCount="59" Body="&lt;p&gt;Recent history suggests that one supplier fails to meet this new specification 20% of the time. Assume that the next 15 batches of this alloy are a random sample. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I find  the expected number of shipments that do meet the new specifications,&#xA;and  the standard deviation?&lt;/p&gt;&#xA;" OwnerUserId="4053" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-04-08T20:40:29.503" LastActivityDate="2011-04-08T20:40:29.503" Title="Expected number of shipments and its standard deviation" Tags="&lt;probability&gt;&lt;estimation&gt;&lt;standard-deviation&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="9261" PostTypeId="2" ParentId="9240" CreationDate="2011-04-06T11:14:25.150" Score="4" ViewCount="" Body="&lt;p&gt;This is essentially a variant of the coupon collector's problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If there are $n$ items in total and you have taken a sample size $s$ with replacement then the probability of having identified $u$ unique items is &#xA;$$   Pr(U=u|n,s) =  \frac{S_2(s,u)  n! }{ (n-u)! n^s }$$ &#xA;where $ S_2(s,u)$ gives &lt;a href=&quot;http://en.wikipedia.org/wiki/Stirling_numbers_of_the_second_kind&quot; rel=&quot;nofollow&quot;&gt;Stirling numbers of the second kind&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now all you need is a prior distribution for $Pr(N=n)$, apply Bayes theorem, and get a posterior distribution for $N$. &lt;/p&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-04-06T11:14:25.150" CommentCount="3" />
  <row Id="9262" PostTypeId="2" ParentId="9233" CreationDate="2011-04-06T11:50:08.220" Score="1" ViewCount="" Body="&lt;p&gt;I am not sure I understand this bit: &quot;Unfortunately, since the response is the mean value (without looking at the mean, just regular inter-annual variation will swamp the signal)&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With careful modelling, it seems to me you could gain a lot by modelling this as panel data. Depending on the spatial scope of your data, there may be large differences in the temperatures that your data points were exposed to within any given year. Averaging all these variations seems costly.&lt;/p&gt;&#xA;" OwnerUserId="2044" LastActivityDate="2011-04-06T11:50:08.220" />
  <row Id="9263" PostTypeId="1" CreationDate="2011-04-06T12:20:49.227" Score="4" ViewCount="127" Body="&lt;p&gt;I am using SPSS and having some trouble with a research question which is analogous to the hypothetical question:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Is there a longitudinal Relationship between Happiness and Chocolate Consumption?&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let’s say I take a sample of people and contact them when they are aged 14 and aged 18 and ask them: &#xA;a)  What is your chocolate consumption in grams per day. &#xA;b)  Are you happy? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have my fictitious data in the following wide format: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;ID  HAPPY.14     HAPPY.18   CHOC.14 CHOC.18&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1   YES  YES            100 5&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2   YES  NO         50  30&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3   NO   YES            30  50&#xA;etc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;I would like to know if the mean chocolate consumption per day is higher among happy people than those who are not happy while accounting for the fact that I have taken repeated measures of both chocolate and happiness at the two time points.&lt;/em&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Approach 1&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suppose one way of doing this would be to do an ANCOVA, using time (before/after) as a grouping variable and controlling for happiness status. However, I think this may be inadvisable as correlation between the two time points would be neglected. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Approach 2&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand that one valid approach for this should be a repeated measures ANOVA.   I’m just not sure how to do this correctly in SPSS. I have specified my within-subjects factor as chocolate consumption with age 14 data as level one and age 18 data as level 2. &#xA;What I’m uncertain about is the next step – specifying covariates and between individual factors – I have the option of adding HAPPY.14 and OR HAPPY.18 as a between individual factor.  If I add both the output tells me about the effect of  HAPPY.14 and HAPPY.18 as you’d expect, not about the “effect” of happy (YES/NO)per se. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I realise it’s a basic question. Any feedback would on either of the two approached would be greatly appreciated. &lt;/p&gt;&#xA;" OwnerUserId="4054" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-06T14:27:19.523" LastActivityDate="2011-04-07T01:53:19.273" Title="Longitudinal relationship between chocolate consumption and happiness: repeated measures ANOVA?" Tags="&lt;anova&gt;&lt;spss&gt;&lt;ancova&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="9264" PostTypeId="2" ParentId="9263" CreationDate="2011-04-06T13:03:53.853" Score="2" ViewCount="" Body="&lt;p&gt;You may think about happiness as the dependent variable, and you could use logistic regression with chocolate consumption as a predictor. Some people may be generally happier or less happy independently from chocolate consumption. This can be modelled by including subject id as a random effect categorical predictor. Age might also influence happiness. After these the model would look like this: &lt;code&gt;logit(happy) ~ choc + age + id&lt;/code&gt;, where age is either 14 or 18, and the data are in the long format, a mixed effect logistic regression including a random categorical, a fixed categorical and a continuous predictor. (As an analogue of the repeated measures approach you could use a covariance pattern model, where id is not a predictor, but used in the specification of the covariance.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively chocolate consumption can be regarded as dependent variable. &lt;code&gt;choc ~ happy + age + id&lt;/code&gt; could be the model (long data format), where id is a random effect, mixed effect ANOVA; or &lt;code&gt;choc ~ happy + age&lt;/code&gt;, where repeated measures are considered, repeated measures ANOVA.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have no idea if happiness causes increased/decreased chocolate consumption or vice versa. You are safe asking about a &quot;relationship&quot; between the two.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-06T14:10:11.800" LastActivityDate="2011-04-06T14:10:11.800" />
  <row Id="9265" PostTypeId="2" ParentId="9260" CreationDate="2011-04-06T13:25:03.273" Score="4" ViewCount="" Body="&lt;p&gt;A simple approach would be to assume that each shipment will meet the specification with $p = 1 - 20\% = 80\%$ probability. The number of shipments meeting the specification (k) will then follow a &lt;a href=&quot;http://en.wikipedia.org/wiki/Binomial_distribution&quot; rel=&quot;nofollow&quot;&gt;binomial distribution&lt;/a&gt;: k ~ B(15, 80%) and the expected value will be $n \cdot p = 15 \cdot 80\%=12$. The standard error of this estimate is the standard deviation of the binomial distribution: $\sqrt{n p (1 − p)} = 1.55$, however, k is not normally distributed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A more complicated approach would account for the fact that the 20% fail rate is only an estimate based on &quot;recent history&quot;. So the actual fail rate may be somewhat lower or higher as well, and the above approach underestimates the uncertainty of the expected value. As we don't exactly know where the 20% came from, we can not calculate this.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastActivityDate="2011-04-06T13:25:03.273" />
  <row Id="9266" PostTypeId="1" CreationDate="2011-04-06T13:52:40.207" Score="2" ViewCount="60" Body="&lt;p&gt;Given a collections of sets, which have an inherent but unknown (at runtime) hierarchy, I would like to cluster them based on the sub/super-relationships with respect to their elements. Let me try and illustrate this with a overly simplified example: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Set 1 = {a, b, c, d, e, f}&#xA;Set 2 = {a, b}&#xA;Set 3 = {a, b, c ,d}&#xA;Set 4 = {a, c, d, f, g, h}&#xA;Set 5 = {d, f}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In this example, there would be two main clusters with the following relations; &#xA;cluster 1:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Set 1 $\supset$ Set 3 $\supset$ Set 2;&lt;br&gt;&#xA; Set 1 $\supset$ Set 5&lt;/p&gt;&#xA;&#xA;&lt;p&gt;... and cluster 2:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Set 4 $\supset$ Set 5 &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The way I see it, the complications here from a standard clustering approach are; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) I can not come up with a good measure of correlation between sets that are to be clustered. I was initially thinking of using the number of common elements but then the following scenario (which is essentially rather likely) complicates things: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$_s(Set1 \cap Set2) = 10$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$_s(Set1 \cap Set3) = 10$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$_s(Set3 \cap Set2) = 0$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) In theory there is no reason for a small set to not be sub-set under more than one superset. This effectively makes any Tree-based data structure unusable, or am I mistaken on this point?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I did some googling, checked on both StackOverflow and here briefly but havent really found something that is useful. Before I start implementing something in Java from scratch I was wondering if anyone had ideas or previous experiences on something like this. If there are libraries/functions one can use for this purpose it would be pretty cool, although I doubt there is something like written in Java. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know that most of you use R, but as I said, the rest of the software is written in Java so I'd prefer to keep things there, if at all possible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT:&lt;/strong&gt; Following @whuber's comments I'll try and clarify the question further. I believe a significant portion of reasoning behind the question got lost when I tried to generalize and abstract the concept. &#xA;So here it goes: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The sets mentioned above are gene/protein sets, and the elements are then genes/proteins. As these entities work in connection with one another, one speaks of functional groups/sets. However the databases that hold this data usually have a high degree of redundancy, in the sense that Set A usually has all the elements of Set B, C .. etc. My whole project is based on analyzing these sets, and when I am done with the analysis and present my results I have a long set of these sets with associated scores. However highly scoring sets sometimes cluster, they may or may not be in the same super-sets. Thus the need/desire to cluster these in a structure like a dendogram. Thus one can overlay the scoring data, with the hierarchy data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On a side-note: I was recommended by a colleague of mine to consider spectral clustering, on which I will read more in the coming days to see whether or not the method can be used here or not. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope these notes make things more clear now, I'd do my best to further develop the ideas if necessary. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks again! &lt;/p&gt;&#xA;" OwnerUserId="3014" LastEditorUserId="3014" LastEditorDisplayName="" LastEditDate="2011-04-06T18:37:19.810" LastActivityDate="2011-04-06T20:07:17.577" Title="A smart way of clustering a collection of sets based on an inherent hierarchy" Tags="&lt;clustering&gt;&lt;algorithms&gt;" AnswerCount="1" CommentCount="8" FavoriteCount="1" />
  <row Id="9267" PostTypeId="1" AcceptedAnswerId="9274" CreationDate="2011-04-06T14:16:14.613" Score="5" ViewCount="153" Body="&lt;p&gt;I'm in the process of running a repeated-measures ANOVA on two groups of participants from an experiment. In one group, there are 10 participants, and in the other group, there is only 1 participant. This is because the 1 participant is a patient who is being compared to the 10 other participants.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, my question is: is it fair/allowable to run an ANOVA comparing the patient with the other participants? I wasn't sure, so ran it in SPSS, and it seems to work, with a significant result. Is this naughty?&lt;/p&gt;&#xA;" OwnerUserId="4055" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-06T19:24:02.030" LastActivityDate="2011-04-07T14:33:54.870" Title="What's the minimum number of individuals in a group for repeated-measures ANOVA?" Tags="&lt;anova&gt;&lt;small-sample&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="4" />
  <row Id="9268" PostTypeId="2" ParentId="9240" CreationDate="2011-04-06T14:38:25.303" Score="3" ViewCount="" Body="&lt;p&gt;I have already give a suggestion based on Stirling numbers of the second kind and Bayesian methods.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For those who find Stirling numbers too large or Bayesian methods too difficult, a rougher method might be to use &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$E[U|n,s] = n\left( 1- \left(1-\frac{1}{n}\right)^s\right)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$var[U|n,s] = n\left(1-\frac{1}{n}\right)^s  + n^2 \left(1-\frac{1}{n}\right)\left(1-\frac{2}{n}\right)^s - n^2\left(1-\frac{1}{n}\right)^{2s} $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and back-calculate using numerical methods. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, taking GaBorgulya's example with  $s=300$  and an observed $U = 265$, this might give us an estimate of $\hat{n} \approx 1180$ for the population.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If that had been the population then it would have given us a variance for $U$ of about 25, and an arbitrary two standard deviations either side of 265 would be about 255 and 275 (as I said, this is a rough method).  255 would have given us a estimate for $n$ about 895, while 275 would have given about 1692.  The example's 1000 is comfortably within this interval.     &lt;/p&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-04-06T14:38:25.303" CommentCount="2" />
  <row Id="9269" PostTypeId="1" AcceptedAnswerId="9271" CreationDate="2011-04-06T14:44:19.773" Score="3" ViewCount="164" Body="&lt;p&gt;I can't seem to do a comparison with a sum of numbers and get the right answer (lines alternate between my input and R output):&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;(.6 + .3 + .1) == 1&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;[1] FALSE&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;(.6 + .3 + .1)&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;[1] 1&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;1&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;[1] 1&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I've tried with and without parenthesis, tried comparing to 1.0, tried using as.numeric, but can't get it to work.&lt;/p&gt;&#xA;" OwnerUserId="4056" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-06T15:24:10.977" LastActivityDate="2011-04-06T15:24:10.977" Title="Why is R giving a bad result of addition?" Tags="&lt;r&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="9270" PostTypeId="2" ParentId="9263" CreationDate="2011-04-06T14:47:31.160" Score="4" ViewCount="" Body="&lt;p&gt;You may need to clarify what you mean by &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&quot;accounting for the fact that I have taken repeated measures...&quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;You say that &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&quot;I would like to know if the mean chocolate consumption per day is higher among happy&#xA;  people than those who are not happy...&quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This suggests to me that time is not really relevant to your research question. Thus, you could do one of the following.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;You could correlate mean happiness ([time1 + time2] / 2) with mean chocolate consumption.&lt;/li&gt;&#xA;&lt;li&gt;You could correlate happiness with chocolate consumption at a given time point.&lt;/li&gt;&#xA;&lt;li&gt;You could correlate happiness with chocolate consumption across times (e.g., 1 with 2).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;A variant on the above would involve performing a regression or other predictive model  predicting one variable from the other.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively, you may find that  you can rephrase your research question more clearly to incorporate what you are interested in with regards to the effect of time.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;You could correlate chocolate change scores with happiness change scores.&lt;/li&gt;&#xA;&lt;li&gt;You could predict time 2 chocolate from time 1 chocolate and time 1 happiness to see whether time 1 happiness predicts over and above time 1 chocolate.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;As a side point, while it may be an artificial example, it seems strange to measure happiness as a Yes / No variable. I would measure it as a scale. It is also a little strange talking about the mean of a Yes / No variable. &lt;/p&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-07T01:53:19.273" LastActivityDate="2011-04-07T01:53:19.273" />
  <row Id="9271" PostTypeId="2" ParentId="9269" CreationDate="2011-04-06T14:57:58.367" Score="9" ViewCount="" Body="&lt;p&gt;You've fallen into the &quot;Floating Point Trap.&quot; See the first chapter of the R Inferno:  &lt;a href=&quot;http://www.burns-stat.com/pages/Tutor/R_inferno.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.burns-stat.com/pages/Tutor/R_inferno.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You want to test floating point numbers with &lt;code&gt;all.equal&lt;/code&gt; instead, like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; all.equal(.6+.3+.1, 1)&#xA;[1] TRUE&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="3601" LastActivityDate="2011-04-06T14:57:58.367" CommentCount="1" />
  <row Id="9272" PostTypeId="2" ParentId="9269" CreationDate="2011-04-06T15:04:18.797" Score="9" ViewCount="" Body="&lt;p&gt;It is floating point rounding.  You get &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; round(.6 + .3 + .1, digits=16) == 1&#xA;[1] FALSE&#xA;&amp;gt; round(.6 + .3 + .1, digits=15) == 1&#xA;[1] TRUE&#xA;&amp;gt; (.6 + .3 + .1) - 1&#xA;[1] -1.11022302462516e-16&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-04-06T15:04:18.797" />
  <row Id="9273" PostTypeId="2" ParentId="9242" CreationDate="2011-04-06T15:37:55.743" Score="0" ViewCount="" Body="&lt;p&gt;The issue here is that hypothesis testing involves a null AND an alternative hypotheis, and therefore; the rejection region is determined by both hypotheses. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider a simpler example. If you are investigating a process that possibly has a MEAN of zero, but could not have a mean of less than zero, then you might be interested in performing the following test &lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{equation}&#xA;\begin{array}{c}&#xA;H_{0}: \mu = 0 \&#xA;H_{1}: \mu &gt; 0 \&#xA;\end{array}&#xA;\nonumber&#xA;\end{equation} &lt;/p&gt;&#xA;&#xA;&lt;p&gt;at a level alpha. Your rejection region of the null hypothesis is on the right of zero.&#xA;It is not impossible for you to get a sample mean that is negative, albeit with a small&#xA;probability. If you were to get a negative sample mean in your experiment, you would not question the veracity of the experiment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now consider your question. The reason that the rejection region for the F-statistic is on&#xA;the right is because of the alternative hypothesis in the one-way ANOVA. You are testing the hypothesis that   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{equation}&#xA;\begin{array}{c}&#xA;H_{0}: \sum \tau_{i}^{2} = 0 \&#xA;H_{1}: \sum \tau_{i}^{2} \ne 0 \&#xA;\end{array}&#xA;\nonumber&#xA;\end{equation} &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The null hypothesis dictates that you use the central F distribution, and the alternative hypothesis, forcing the distribution to the right when the alternative hypothesis is true means that all of the Type I error probability must be located on the right. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possible for the test statistic to be less than one? When the null hypothesis is true, it is certainly possible; just as in the previous example where it was possible for the test statistic to be negative even if the MEAN of the data is zero.&lt;/p&gt;&#xA;" OwnerUserId="3805" LastEditorUserId="3805" LastEditorDisplayName="" LastEditDate="2011-04-06T15:43:48.640" LastActivityDate="2011-04-06T15:43:48.640" />
  <row Id="9274" PostTypeId="2" ParentId="9267" CreationDate="2011-04-06T15:56:21.583" Score="9" ViewCount="" Body="&lt;p&gt;Performing the ANOVA assumes that the nature and amount of variation in the hypothetical population represented by the one patient are the same as the variation in the hypothetical population represented by the ten other patients, and that all were obtained randomly and independently from their populations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Under these assumptions a standard approach is to use data from the ten patients to compute a &lt;a href=&quot;http://en.wikipedia.org/wiki/Prediction_interval#Unknown_mean.2C_unknown_variance&quot; rel=&quot;nofollow&quot;&gt;prediction interval&lt;/a&gt; for a single additional patient.  The significance test only has to check whether the lone patient falls within that interval.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The prediction interval is simple to justify and compute.  It depends only on the estimated mean $\bar{m}$ from the ten patients and on their estimated variance $s^2$.  Let $x$ be the value of the lone patient.  Under the null hypothesis (that all patients are drawn independently from a single population), the variance of $x - \bar{m}$ equals $s^2 + s^2/10$.  &lt;em&gt;If you further assume all variation is Normal&lt;/em&gt;--that's critical here because it's hard to check with just ten patients in a group--then $\frac{x - \bar{m}}{\sqrt{s^2 + s^2/10}}$ has a Student's t distribution with 9 degrees of freedom (9 is the df used to estimate $s^2$), allowing you to erect a confidence interval in the standard way for $x$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If this prediction interval test disagrees with SPSS's result, I would not trust the SPSS ANOVA in this case.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-04-06T15:56:21.583" CommentCount="5" />
  <row Id="9275" PostTypeId="1" CreationDate="2011-04-06T16:04:07.350" Score="1" ViewCount="89" Body="&lt;p&gt;I'm concerned about treating my data as gold, especially in areas of low data support, so I would like to apply &lt;a href=&quot;http://en.wikipedia.org/wiki/Additive_smoothing&quot; rel=&quot;nofollow&quot;&gt;additive smoothing&lt;/a&gt;.  I'm then doing several things with this data, and one of them is &lt;a href=&quot;http://en.wikipedia.org/wiki/Pearson%27s_chi-square_test#Test_of_independence&quot; rel=&quot;nofollow&quot;&gt;Pearson's chi-square test for independence&lt;/a&gt;.  Is it still acceptable to first do the smoothing and then do the chi-square test?  The former is very Bayesian and the latter is very frequentest, so at the least it feels somewhat strange.  However, it seems like smoothing should curtail the need to worry much about the data support or the need for something like Fisher's exact test.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My reasoning for wanting to use competing philosophies is because the audience really wants to see p-values, but the Bayesian inside of me is fighting to be let loose.  As such I came up with this compromise.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: To clarify how I envision the smoothing taking place, consider the following contingency table:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    X=0  X=1&#xA;   +-------+&#xA;Y=0|  2|  2|&#xA;   +---+---+&#xA;Y=1|  3|  3|&#xA;   +---+---+&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So without smoothing we have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(X=0)=\frac{5}{10}$, $P(X=1)=\frac{5}{10}$, $P(Y=0)=\frac{4}{10}$, $P(Y=1)=\frac{6}{10}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With additive smoothing and say $\alpha=1$ we get&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(X=0)=\frac{6}{12}$, $P(X=1)=\frac{6}{12}$, $P(Y=0)=\frac{5}{12}$, $P(Y=1)=\frac{7}{12}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I haven't yet decided how to handle the contingency table.  One way would be to smooth each cell:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    X=0  X=1&#xA;   +-------+&#xA;Y=0|  3|  3|&#xA;   +---+---+&#xA;Y=1|  4|  4|&#xA;   +---+---+&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2485" LastEditorUserId="2485" LastEditorDisplayName="" LastEditDate="2011-04-07T18:10:24.933" LastActivityDate="2011-04-07T18:10:24.933" Title="Is it OK to do additive smoothing before applying Pearson's chi-square test for independence?" Tags="&lt;bayesian&gt;&lt;chi-squared&gt;&lt;smoothing&gt;&lt;frequentist&gt;" CommentCount="9" FavoriteCount="2" />
  <row Id="9276" PostTypeId="1" CreationDate="2011-04-06T17:35:50.953" Score="7" ViewCount="230" Body="&lt;p&gt;&lt;strong&gt;Background&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am designing a Monte Carlo simulation that combines the outputs of series of models, and I want to be sure that the simulation will allow me to make reasonable claims about the probability of the simulated outcome and the precision of that probability estimate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The simulation will find the probability that a jury drawn from a specified community will convict a certain defendant. These are the steps of the simulation:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Using existing data, generate a logistic probability model (&lt;em&gt;M&lt;/em&gt;) by regressing “juror first ballot vote” on demographic predictors.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Use Monte Carlo methods to simulate 1,000 versions of &lt;em&gt;M&lt;/em&gt; (i.e., 1000 versions of the coefficients for the model parameters).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Select one of the 1,000 versions of the model (&lt;em&gt;M&lt;sub&gt;i&lt;/sub&gt;&lt;/em&gt;).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Empanel 1,000 juries by randomly selecting 1,000 sets of 12 “jurors” from a “community” (&lt;em&gt;C&lt;/em&gt;) of individuals with specified demographic characteristic distributions.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Deterministically calculate the probability of a first ballot guilty vote for each juror using &lt;em&gt;M&lt;sub&gt;i&lt;/sub&gt;&lt;/em&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Render each &quot;juror’s&quot; probable vote into a determinate vote (based on whether it is greater or less than randomly selected value between 0-1).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Determine each &quot;jury’s&quot; “final vote” by using a model (derived from empirical data) of the probability a jury will convict, conditional on the proportion of jurors voting for conviction on the first ballot.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Store the proportion of guilty verdicts for the 1000 juries (&lt;em&gt;PG&lt;sub&gt;i&lt;/sub&gt;&lt;/em&gt;).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Repeat steps 3-8 for each of the 1,000 simulated versions of &lt;em&gt;M&lt;/em&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Calculate the mean value of &lt;em&gt;PG&lt;/em&gt; and report that as the point estimate of the probability of conviction in &lt;em&gt;C&lt;/em&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Identify the 2.5 &amp;amp; 97.5 percentile values for &lt;em&gt;PG&lt;/em&gt; and report that as 0.95 confidence interval.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I am currently using 1,000 jurors and 1,000 juries on the theory that 1,000 random draws from a probability distribution&amp;mdash;demographic characteristics of &lt;em&gt;C&lt;/em&gt; or versions of &lt;em&gt;M&lt;/em&gt;&amp;mdash;will fill out that distribution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Questions&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Will this allow me to accurately determine the precision of my estimate? If so, how many juries do I need to empanel for each &lt;em&gt;PG&lt;sub&gt;i&lt;/sub&gt;&lt;/em&gt; calculation to cover &lt;em&gt;C&lt;/em&gt;'s probability distribution (so I avoid selection bias); may I use fewer than 1,000?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you so much for any help!&lt;/p&gt;&#xA;" OwnerUserId="4059" LastActivityDate="2011-04-11T16:37:21.353" Title="Finding precision of Monte Carlo simulation estimate" Tags="&lt;confidence-interval&gt;&lt;monte-carlo&gt;&lt;standard-error&gt;&lt;simulation&gt;" AnswerCount="2" CommentCount="10" FavoriteCount="2" />
  <row Id="9277" PostTypeId="1" CreationDate="2011-04-06T18:41:33.737" Score="3" ViewCount="66" Body="&lt;p&gt;I'm trying to use the &lt;code&gt;nnet&lt;/code&gt; library in R, and can't seem to work out how to use the &lt;code&gt;reltol&lt;/code&gt; parameter. It says in the docs:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Stop if the optimizer is unable to reduce the fit criterion by a factor of at least 1 - reltol.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I assume this means that if &lt;code&gt;reltol = 0.5&lt;/code&gt; then it will stop if it can't halve the error rate, and if &lt;code&gt;reltol = 0.1&lt;/code&gt; it will stop if it can't reduce the error rate by 10%. Is that correct? The experiments that I've done don't seem to work like that...&lt;/p&gt;&#xA;" OwnerUserId="261" LastActivityDate="2011-04-06T21:36:21.523" Title="Meaning and use of `reltol` in `nnet` library in R" Tags="&lt;r&gt;&lt;machine-learning&gt;&lt;neural-networks&gt;" AnswerCount="1" />
  <row Id="9278" PostTypeId="1" CreationDate="2011-04-06T19:25:10.097" Score="3" ViewCount="73" Body="&lt;p&gt;Very basic question, I suspect. It's sort of the inverse of &lt;a href=&quot;http://stats.stackexchange.com/questions/5092/representative-sampling&quot;&gt;this one&lt;/a&gt;, and the same disclaimers about my own ignorance apply, but here goes:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a sampling technology. I know the total population, and I know the number of samples I've done. What's the best way to calculate and express how confident I am that my results are correct?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A few more points in case they make a difference:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;My test checks whether an item is in one of three states (and they are the only possible states).&lt;/li&gt;&#xA;&lt;li&gt;I'm assuming that the test is truly a random distribution.&lt;/li&gt;&#xA;&lt;li&gt;In general, my total population is &gt; 1,000,000 and my sample is ~20% of that, but it can vary quite a bit.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;And a wrinkle (I hesitate to even put this in because I want to keep things simple):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;There's a slight error where my reported Total Population value might be slightly smaller than the actual total (by 5-10%). I can get the actual total if need be, but won't bother if it doesn't really make a difference.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Rough and simple / clear is probably better than more accurate if there are different ways of expressing this. I.e., something like &quot;we're 95% confident that this is correct&quot; is probably better than &quot;we're 95% confident that this is correct to within 3%&quot;. (Am I talking about P?) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hmm... I think my &quot;wrinkle&quot; may have introduced confusion. I'm not trying to find out how accurate the total is. I can get the actual total, but I have to use a more convoluted method. So my main question was about saying if I have 10,000 samples out of 1,000,000 items how confident can I be that my sampled distribution among the three states is correct. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My secondary question (which is where the wrinkle comes in) is: given that the actual total is 5-10% larger than the one I'm using, how much of a difference does that make to my confidence? In other words, I'm saying 1,000,000 items, but it might actually be 1,100,000. Should I bother to go through the convoluted process to get the actual total, just to compute the confidence? To me, that seems unlikely to make a significant difference to my confidence level, but I thought I should check.&lt;/p&gt;&#xA;" OwnerUserId="1531" LastEditorUserId="1531" LastEditorDisplayName="" LastEditDate="2011-04-08T19:33:40.597" LastActivityDate="2011-04-16T15:12:30.973" Title="How to check reliability of a given sampling technology?" Tags="&lt;sampling&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="9279" PostTypeId="2" ParentId="9266" CreationDate="2011-04-06T20:01:32.527" Score="1" ViewCount="" Body="&lt;p&gt;If I understood you correctly, you have a list of scores for subsets, and you want to identify the items that contribute most to a high score, but you cannot score arbitrary subsets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This corresponds to a high-dimensional binary regression problem with features $item\in Subset$. You can run a linear or logistic regression on the dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want a multiset or partition instead of an itemwise regression, you'll have to specify the goal and scoring model further.&lt;/p&gt;&#xA;" OwnerUserId="2456" LastEditorUserId="2456" LastEditorDisplayName="" LastEditDate="2011-04-06T20:07:17.577" LastActivityDate="2011-04-06T20:07:17.577" CommentCount="2" />
  <row Id="9280" PostTypeId="1" AcceptedAnswerId="9669" CreationDate="2011-04-06T20:54:39.407" Score="4" ViewCount="79" Body="&lt;p&gt;I have a spatial error model that I have estimated and would like to create a figure with the prediction interval. By spatial error model, I mean that the errors are spatially correlated and thus OLS is unbiased but inefficient (not a spatially auto-regressive model).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I used the geostatistical approach to spatial statistics and estimated a semivariogram for my data and used the semivariogram estimates to create a weighting matrix to account for the spatial autocorrelation in the data. I think the answer to this question though would help anyone using weights in their regression regardless of the exact structure of their weighting matrix.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I see from this &lt;a href=&quot;http://stats.stackexchange.com/questions/9131/obtaining-a-formula-for-prediction-limits-in-a-linear-model&quot;&gt;question&lt;/a&gt; that the formula for a 95% prediction interval for OLS is&lt;br&gt;&#xA;$$&#xD;&#xA;\hat{y} \pm 1.96 \hat{\sigma} \sqrt{1 + \mathbf{X}^* (\mathbf{X}&amp;#39;\mathbf{X})^{-1} (\mathbf{X}^*)&amp;#39;}.&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given my limited intuition, my guess would be that the prediction interval for the case with weights would be this (where $\Omega$ is the weighting matrix):  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;\hat{y} \pm 1.96 \hat{\sigma} \sqrt{1 + \mathbf{X}^* (\mathbf{X}&amp;#39;\mathbf{\Omega}^{-1}\mathbf{X})^{-1} (\mathbf{X}^*)&amp;#39;}&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and was hoping that I could get some feedback from the community here about the correctness or if I am even in the right ballpark. &lt;/p&gt;&#xA;" OwnerUserId="3713" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-04-06T21:03:36.297" LastActivityDate="2011-04-18T13:52:57.690" Title="Formula for prediction interval for spatial regression" Tags="&lt;regression&gt;&lt;spatial&gt;&lt;prediction-limit&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="9281" PostTypeId="1" AcceptedAnswerId="9293" CreationDate="2011-04-06T21:00:00.867" Score="5" ViewCount="82" Body="&lt;p&gt;Hope this newbie question is the right question for this site:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose I would like to compare the composition of ecological communities at two sites A, B. I know all three sites have dogs, cats, cows, and birds, so I sample their abundances at each site (I don't really have an &quot;expected&quot; abundance for each animal at each site).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I count, say, five of each animal at each site, then the A and B are very &quot;similar&quot; (in fact, they're the &quot;same&quot;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But if I find 100 dogs, 5 cats, 2 cows, and 3 birds at site A. 5 dogs, 3 cats, 75 cows, and 2 birds at site B. Then I would say that the sites A and B are &quot;dissimilar&quot;, even though they have the exact same species composition.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(I read up on the Sorensen's and Bray-Curtis indices, but looks like they only consider absence/presence of the dogs, cats, etc., and not their abundances.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a statistical test to determine this? Thanks.&lt;/p&gt;&#xA;" OwnerUserId="2830" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-04-06T21:19:29.840" LastActivityDate="2011-04-07T11:42:05.100" Title="What test to compare community composition?" Tags="&lt;distributions&gt;&lt;correlation&gt;&lt;test&gt;&lt;multinomial&gt;" AnswerCount="3" CommentCount="2" FavoriteCount="1" />
  <row Id="9282" PostTypeId="2" ParentId="9280" CreationDate="2011-04-06T21:13:52.667" Score="3" ViewCount="" Body="&lt;p&gt;Assuming $\Omega$ has been estimated with no appreciable error--which is rarely the case--the correct formula is given by the equation for the &lt;a href=&quot;http://en.wikipedia.org/wiki/Kriging&quot; rel=&quot;nofollow&quot;&gt;kriging prediction error&lt;/a&gt;.  It needs to replace everything under the square root sign.  What is missing from the second formula you give is any explicit accounting of the covariance between the value at the estimation location and the data values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some recent approaches, such as presented in Diggle &amp;amp; Ribeiro's &quot;&lt;a href=&quot;http://www.leg.ufpr.br/mbgbook/&quot; rel=&quot;nofollow&quot;&gt;Model-based Geostatistics&lt;/a&gt;&quot; (implemented as an R package) have the additional merit of incorporating the estimation error of $\Omega$ in the prediction uncertainty.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-04-06T21:13:52.667" CommentCount="1" />
  <row Id="9283" PostTypeId="1" AcceptedAnswerId="9297" CreationDate="2011-04-06T21:16:01.777" Score="4" ViewCount="108" Body="&lt;p&gt;Could you recommend an easy to use or comprehensive conjoint analysis package for R?&lt;/p&gt;&#xA;" OwnerUserId="776" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-07T02:29:22.393" LastActivityDate="2011-04-07T02:29:22.393" Title="Conjoint Packages for R" Tags="&lt;r&gt;&lt;conjoint&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="1" />
  <row Id="9284" PostTypeId="2" ParentId="9277" CreationDate="2011-04-06T21:36:21.523" Score="2" ViewCount="" Body="&lt;p&gt;That is what it is supposed to mean. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A &lt;code&gt;reltol&lt;/code&gt; parameter is common in optimisation functions in R and other mathematical and statistical systems.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In &lt;code&gt;nnet&lt;/code&gt;, it has the default &lt;code&gt;reltol = 1.0e-8&lt;/code&gt;.  In the more often used &lt;code&gt;optim&lt;/code&gt; in the &lt;code&gt;stats&lt;/code&gt; library the statement is: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;code&gt;reltol&lt;/code&gt; &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Relative convergence tolerance.&#xA;  The algorithm stops if it is unable to&#xA;  reduce the value by a factor of &lt;code&gt;reltol * (abs(val) + reltol)&lt;/code&gt; &#xA;  at a step. Defaults to &lt;code&gt;sqrt(.Machine$double.eps)&lt;/code&gt;,&#xA;  typically about &lt;code&gt;1e-8&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;You can see why it has &lt;code&gt;sqrt&lt;/code&gt; since it might be looking at an amount of about &lt;code&gt;reltol^2&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-04-06T21:36:21.523" />
  <row Id="9285" PostTypeId="5" CreationDate="2011-04-06T21:41:34.313" Score="0" ViewCount="" Body="&lt;p&gt;In &lt;a href=&quot;http://en.wikipedia.org/wiki/Statistical_hypothesis_testing&quot; rel=&quot;nofollow&quot;&gt;statistical hypothesis testing&lt;/a&gt;, the &lt;em&gt;size&lt;/em&gt; is the largest chance of rejecting the null when the null is true (a &quot;false positive&quot; error).  The &lt;em&gt;power&lt;/em&gt; is the chance of not rejecting the null; it depends on the &quot;effect size&quot; (a measure of how far reality actually departs from the null).  &lt;em&gt;Caeteris paribus,&lt;/em&gt; power and size are inversely related (one must increase if the other is decreased), so considerations often focus on size, which is simpler to analyze.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When more than one hypothesis test is performed to make a binary decision, the chance of a false positive is usually greater than the size of any of the tests used for that decision.  For example, suppose groups of &quot;control&quot; and &quot;treatment&quot; subjects are randomly selected from the same population and each subject is given a questionnaire comprising 20 yes-no questions.  Let the groups be compared separately for each question using a test of size .05.  If the comparisons are independent, then the chance of at least one of them rejecting the null equals $1 - (1 - 0.05)^{20}$ = 0.64.  Thus a nominal false positive rate of 0.05 in each test is inflated to a decision false positive rate of 0.64.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To avoid unacceptably large chances of reaching mistaken conclusions in such &quot;multiple comparisons&quot; cases, either an overall test of significance is initially conducted or the sizes of the &lt;em&gt;individual&lt;/em&gt; tests leading up to the decision are decreased (that is, the tests are made more stringent).  Examples of the former are the &lt;a href=&quot;http://en.wikipedia.org/wiki/Analysis_of_variance#Follow_up_tests&quot; rel=&quot;nofollow&quot;&gt;F-test in an ANOVA&lt;/a&gt; setting and &lt;a href=&quot;http://en.wikipedia.org/wiki/Tukey%27s_range_test&quot; rel=&quot;nofollow&quot;&gt;Tukey's HSD test&lt;/a&gt;.  Exemplary of the latter approach is the &lt;a href=&quot;http://en.wikipedia.org/wiki/Bonferroni_correction&quot; rel=&quot;nofollow&quot;&gt;Bonferroni correction&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="-1" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-04-06T21:41:34.313" LastActivityDate="2011-04-06T21:41:34.313" />
  <row Id="9286" PostTypeId="4" CreationDate="2011-04-06T21:41:34.313" Score="0" ViewCount="" Body="Signals situations where one is concerned about achieving intended power and size when more than one hypothesis test is performed." OwnerUserId="-1" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-04-06T21:41:34.313" LastActivityDate="2011-04-06T21:41:34.313" />
  <row Id="9287" PostTypeId="2" ParentId="9228" CreationDate="2011-04-06T22:15:08.573" Score="1" ViewCount="" Body="&lt;p&gt;There are at least two possibilities for this data. One possibility is that your microarrays contain no disease markers whatsoever. But, they do contain information about age, and since in your case the sick and control populations are of different age, you get the illusion of good classification performance. Another possibility is that the microarrays do contain disease markers, and, moreover, these markers is exactly what SVM focuses on. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems like the principal components of the data may be correlated with age in both of these possibilities. In the first case it will be because age is what the data expresses. In the second case it will be because disease is what the data expresses, and this disease is itself correlated with age (for your dataset). I don't think there is an easy way to look at the correlation value and conclude which case it is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I could think of several ways to assess the effect differently. One option is to split your training set into groups of equal age. In this case, for 'young' ages the normal class will have more training examples than the disease class, and vice versa for the older ages. But as long as there are enough examples, this should not be a problem. Another option is to do the same with the test sets, i.e. see whether the classifier tends to say 'sick' more often for older patients. Both of these options could be difficult since you don't have that many examples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One more option is to train two classifiers. In the first, the only feature will be the age. It seems this has AUC of 0.82. In the second, there will be age and the microarray data. (It seems that currently you train a different classifier which only uses the microarray data, and it gives you AUC 0.95. Adding the age feature explicitly is likely to improve performance, so AUC will be even higher.) If the second classifier performs better than the first, this indicates that age is not the only thing of interest in this data. Based on your comment, the improvement in AUC is 0.13 or more, which seems fair. &lt;/p&gt;&#xA;" OwnerUserId="3369" LastEditorUserId="3369" LastEditorDisplayName="" LastEditDate="2011-04-06T22:23:28.577" LastActivityDate="2011-04-06T22:23:28.577" CommentCount="10" />
  <row Id="9288" PostTypeId="2" ParentId="9237" CreationDate="2011-04-06T22:38:22.240" Score="1" ViewCount="" Body="&lt;p&gt;maximum entropy is a good way to go here.  With maximum entropy, you specify the &quot;structure&quot; that your model is to depend on, and it does the rest.  It has a very similar form to a generalised estimating equation.  So we have an unknown (or &quot;random&quot;) variable $x$ that is the object of inference (may be a vector).  It can take on $n$ values $x_1,x_2,\dots,x_n$.  We have $m$ &quot;model functions&quot; $g_{k}(x)$  $(k=1,\dots,m)$ and $G_{k}$ &quot;constraints&quot; and the two are related by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\sum_{i=1}^{n}p_i g_{k}(x_i)=G_{k}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $p_i=Pr(x=x_i)$ is the &quot;unknown&quot; (more appropriately &quot;unassigned&quot;) probability distribution.  By choosing the distribution with maximum entropy $-\sum_i p_i log(p_i)$ you get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Pr(x=x_i||G_{1},\dots,G_{m})=p_i=exp\left(-\lambda_0-\sum_{k=1}^{m}\lambda_{k}g_{k}(x_i)\right)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $\lambda_{k}$ are chosen so that the above constraints are satisfied, and that the $p_i$ sum to $1$.  This requires discrete sample space, for continuous case, it is the same form with 2 exceptions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The summation is replaced by an integral with respect to $dx$&lt;/li&gt;&#xA;&lt;li&gt;The probability is multiplied by the invariant measure $m(x)$, which in your case is the probability transform of the improper $Dir(0,\dots,0)$ into &quot;odds&quot; form.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;So you have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p(x|G_{1},\dots,G_{m})=m(x)exp\left(-\lambda_0-\sum_{k=1}^{m}\lambda_{k}g_{k}(x)\right)dx$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now what you will find is that in solving for the lagrange multipliers $\lambda_{k}$ you will get a set of equations which look identical to the GEE equations.  Thus the MaxEnt algorithm basically tells you &quot;which probability model&quot; is the most likely to be consistent with your GEE equations.  You can then use this distribution to make predictions and give an indication of their accuracy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note if the constraints $G_{k}$ are not known, this is solved by multiplying the above $p(x||G_{1},\dots,G_{m})$ by a prior density for $G_{k}$ and then integrating out $G_{k}$ from the probability - this allows for the structure induced by $G_{k}$ to be incorporated into the model, as well as the uncertainty about the true value of $G_{k}$.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-04-06T22:38:22.240" CommentCount="1" />
  <row Id="9289" PostTypeId="2" ParentId="9281" CreationDate="2011-04-06T22:54:23.313" Score="2" ViewCount="" Body="&lt;p&gt;Since you mention Sorensen index, it seems you need a dissimilarity score rather than a dissimilarity test. (A score will give a numerical value indicating how different they are. A test will tell you whether the difference is significant with a given probability.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can represent species abundance at each location by a histogram. This histogram can be normalized if you just care about relative abundance (e.g. that cats are twice as abundant as dogs), or unnormalized if you care about absolute numbers as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are many ways to measure dissimilarity of histograms. Some of the popular ones are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Chi-squared statistic&lt;/li&gt;&#xA;&lt;li&gt;L2 distance&lt;/li&gt;&#xA;&lt;li&gt;Histogram intersection distance&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="3369" LastActivityDate="2011-04-06T22:54:23.313" />
  <row Id="9290" PostTypeId="2" ParentId="9278" CreationDate="2011-04-06T23:24:22.640" Score="1" ViewCount="" Body="&lt;p&gt;This sounds like a simple case of hyper-geometric sampling.  So you have a sampling distribution of:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p(r_1,r_2,r_3|R_1,R_2,R_3,I)=\frac{{R_1 \choose r_1}{R_2 \choose r_2}{R_3 \choose r_3}}{{R_1+R_2+R_3 \choose r_1+r_2+r_3}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Capitals letters denote population totals, and small letters denote sampled numbers.  $I$ is the prior information about the sampling.  You want to &quot;invert&quot; this to get a distribution for $R_{i}$  Just use Bayes theorem:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p(R_1,R_2,R_3|r_1,r_2,r_3,I)=p(R_1,R_2,R_3|I)\frac{p(r_1,r_2,r_3|R_1,R_2,R_3,I)}{p(r_1,r_2,r_3|I)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And you now have a statement about the accuracy of the population totals!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In response to the comment, the values of $R_{j}$ are the unknown population totals for each of the three states - so if you had sampled/tested every item, you would get $R_{j}$ as the numbers in each state.  I assume that these quantities are the target of your inference - this is what you would like to know.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now what do you know? the population total $N=R_{1}+R_{2}+R_{3}\approx 1,000,000$ (we can account for possible error later).  You also know the sampled numbers $r_{1},r_{2},r_{3}$ (the number which tested positive for each state in your sample).  The total total sample size $n=r_{1}+r_{2}+r_{3}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;the quantity $p(R_1,R_2,R_3|I)$ is called the prior, and you assign it based on what is known about the population totals beyond the data from the sample.&#xA;Now you have stated that $N$ is known, which constrains but does not determine the prior.  One way to determine the prior is to break up the three propositions into mutually exclusive and exhaustive pieces, and assign equal probabilities to the ones which add to $N$, and zero to everything else.  A quick counting exercise shows there is $\frac{(N+1)(N+2)}{2}$ combinations of $R_1,R_2,R_3$ which add up to $N$, so the joint prior is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p(R_1,R_2,R_3|N,I)=\frac{2}{(N+1)(N+2)}\delta(N-R_1-R_2-R_3)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $\delta(x)=1$ if $x=0$ and $\delta(x)=0$ if $x\neq 0$.  And you can work out the normalising constant $P(r_1,r_2,r_3|N,I)$ by adding up the prior and the sampling probabilities over the $R_{j}$, so we get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p(r_1,r_2,r_3|N,I)=\sum_{R_1=0}^{N}\sum_{R_2=0}^{N}\sum_{R_3=0}^{N}p(R_1,R_2,R_3|N,I)p(r_1,r_2,r_3|R_1,R_2,R_3,N,I)$$&#xA;$$=\sum_{R_1=0}^{N}\sum_{R_2=0}^{N-R_1}\frac{2}{(N+1)(N+2)}\frac{{R_1 \choose r_1}{R_2 \choose r_2}{N-R_1-R_2 \choose n-r_1-r_2}}{{N \choose n}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now $(N+1)(N+2){N \choose n}=(n+1)(n+2){N+2 \choose n+2}$ and &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\sum_{R_1=0}^{N}\sum_{R_2=0}^{N-R_1}{R_1 \choose r_1}{R_2 \choose r_2}{N-R_1-R_2 \choose n-r_1-r_2}={N+2 \choose n+2}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So we get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p(r_1,r_2,r_3|N,I)=\frac{2}{(n+1)(n+2)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And thus the posterior distribution is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p(R_1,R_2,R_3|r_1,r_2,r_3,N,I)=\frac{2}{(N+1)(N+2)}\frac{\frac{{R_1 \choose r_1}{R_2 \choose r_2}{R_3 \choose r_3}}{{N \choose n}}}{\frac{2}{(n+1)(n+2)}}=\frac{{R_1 \choose r_1}{R_2 \choose r_2}{R_3 \choose r_3}}{{N+2 \choose n+2}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The last form shows very easily how to generalise, for those interested (noting that $2=3-1$).  This posterior has expectation for $R_1$ of:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$E([R_1+1]|r_1,r_2,r_3,N,I)=\sum_{R_1=0}^{N}\sum_{R_2=0}^{N-R_1}\frac{(R_1+1){R_1 \choose r_1}{R_2 \choose r_2}{R_3 \choose r_3}}{{N+2 \choose n+2}}=\frac{(r_1+1)(N+3)}{n+3}$$&#xA;$$\implies E(R_1|r_1,r_2,r_3,N,I)=\frac{(r_1+1)(N-n+n+3)-(n+3)}{n+3}$$&#xA;$$=r_1+(N-n)\hat{p}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\hat{p}=\frac{r_1+1}{n+3}$.  This is the number of observed in category 1 plus an estimate of the number remaining unobserved in category 1.  Now for the accuracy we can take the variance - which, using the same trick calculate&#xA;$$E([R_1+1][R_1+2])=\frac{(r_1+1)(r_1+2)(N+3)(N+4)}{(n+3)(n+4)}=E([R_1+1]^2)+E(R_1+1)$$&#xA;and note that $var(R_1)=var(R_1+1)$ we get &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$var(R_1)=E([R_1+1][R_1+2])-E(R_1+1)-[E(R_1+1)]^2$$&#xA;$$=\frac{(r_1+1)(N+3)}{n+3}\left[\frac{(r_1+2)(N+4)}{n+4}-1-\frac{(r_1+1)(N+3)}{n+3}\right]$$&#xA;which after some tedious manipulations you get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$var(R_1)=\frac{\hat{p}(1-\hat{p})}{n+4}(N-n)(N+3)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could also calculate the mean and variance of the fraction remaining $F=\frac{R_1-r_1}{N-n}$ which are given by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$E(F|r_1,r_2,r_3,N,I)=\hat{p}\;\;\;\;\;var(F|r_1,r_2,r_3,N,I)=\frac{\hat{p}(1-\hat{p})}{n+4}\left(1+\frac{n+3}{N-n}\right)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And then quantities are approximately independent of $N$ - so the accuracy of $N$ is not important for inferring the proportions in each category - but the sampling fraction is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One way to incorporate the uncertainty about $N$ is to use a uniform prior between to bounds $L_N&amp;lt;N&amp;lt;U_N$, and then &quot;average out&quot; the value of $N$ from the posterior:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p(R_1,R_2,R_3|r_1,r_2,r_3,I)=\frac{1}{U_N-L_N}\sum_{N=L_N}^{U_N}p(R_1,R_2,R_3|r_1,r_2,r_3,N,I)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But unless the terms in this summation are appreciably different, the result won't change that much.  It won't in this case as I have shown&lt;/p&gt;&#xA;" OwnerUserId="2392" LastEditorUserId="2392" LastEditorDisplayName="" LastEditDate="2011-04-10T01:58:04.600" LastActivityDate="2011-04-10T01:58:04.600" CommentCount="1" />
  <row Id="9291" PostTypeId="2" ParentId="9281" CreationDate="2011-04-06T23:37:49.453" Score="2" ViewCount="" Body="&lt;p&gt;Bray-Curtis and other similar indices do incorporate differences in species abundance.  In addition to Legendre and Legendre, I would also recommend Charles Krebs' book, Ecological Methology (1999).&lt;/p&gt;&#xA;" OwnerUserId="3774" LastEditorUserId="3774" LastEditorDisplayName="" LastEditDate="2011-04-07T03:14:05.010" LastActivityDate="2011-04-07T03:14:05.010" />
  <row Id="9292" PostTypeId="1" CreationDate="2011-04-06T23:49:38.163" Score="2" ViewCount="42" Body="&lt;p&gt;What is a good book that covers the mixed distributions? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most statistics books either only briefly mention them or do not cover the topic at all. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd like to have a comprehensive resource covering issues of joint distributions with both discrete and continuous variables, conditional and marginal distributions in the mixed case, etc. &lt;/p&gt;&#xA;" OwnerUserId="3280" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-07T01:51:19.417" LastActivityDate="2011-04-07T01:51:19.417" Title="Which books would you recommend for mixed distributions (continuous and discrete)" Tags="&lt;mixed-model&gt;&lt;books&gt;&lt;conditional-probability&gt;" CommentCount="3" FavoriteCount="1" />
  <row Id="9293" PostTypeId="2" ParentId="9281" CreationDate="2011-04-07T00:17:04.250" Score="3" ViewCount="" Body="&lt;p&gt;I will concur with what has been mentioned that Bray-Curtis can handle abundance as well as presence/absence, also to add another good book to the mix: Analysis of Ecological Communities by McCune and Grace.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are a lot of factors to consider as you compare ecological communities and I don't think there is a single test that will do the job.  The appropriateness of the test will depend a lot on the type of question you are asking about the communities and the nature of your dataset.  Common approaches include ordination techniques, which array sites within a multidimensional taxon-space.  However if you truly only have 2 sites then this is not likely to work.  Mantel tests correlate a the distance matrix based on composition (e.g., the pairwise Bray-Curtis distance across all sites) with a distance matrix based on other potential factors of influence.  The simplest case can be just the euclidean distance between the sites in space.  Cluster analysis groups sites with respect to their community composition.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general I would take the approach of using a subset of the many statistical tools described in any of the above books to provide a statistical description of the differences between the communities in question.  There is no single measure of the difference in community composition so the stats are used to synthesize multidimensional data into a more easily interpretable form. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: I also just thought of this paper which lays out many of the different options pretty clearly and thoroughly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anderson, M. J. et al. 2011. Navigating the multiple meanings of Beta diversity: a roadmap for the practicing ecologist.  Ecology Letters 14:19-28&lt;/p&gt;&#xA;" OwnerUserId="4048" LastEditorUserId="4048" LastEditorDisplayName="" LastEditDate="2011-04-07T11:42:05.100" LastActivityDate="2011-04-07T11:42:05.100" CommentCount="1" />
  <row Id="9295" PostTypeId="1" AcceptedAnswerId="9320" CreationDate="2011-04-07T01:50:31.040" Score="4" ViewCount="66" Body="&lt;p&gt;This is a more &quot;statistical&quot; version of my &lt;a href=&quot;http://math.stackexchange.com/questions/31267/mean-chord-length-in-a-convex-polygon&quot;&gt; Math.SE problem &lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;b&gt; Question &lt;/b&gt;  Let $\mathbf{P}$ be a convex polygon of $m$ sides. Pick two of its edges at random, and further pick a random point on each of these edges, and compute the length of the chord we get. If we repeat this process $n$ times,and average the chord lengths we thus obtained, we get an estimate $\hat{\ell}_n$ of the expected chord length $\ell$ of $\mathbf{P}$.  Then,how large should $n$ be to ensure that $\hat{\ell}_n$ differs from $\ell$ by (say) at most 5%,with a confidence level of (say) 95%, and precisely how  should we collect our samples?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that determining the expected chord length analytically leads to intractable integrals as shown in my SE post. Also,note that there are multiple inequivalent ways to define the expected chord length of $\mathbf{P}$; the one we need here is $\ell = \lim_{n\rightarrow \infty} \hat{\ell}_n$.&lt;/p&gt;&#xA;" OwnerUserId="4011" LastEditorUserId="4011" LastEditorDisplayName="" LastEditDate="2011-04-07T02:31:00.333" LastActivityDate="2011-04-07T18:41:45.193" Title="Monte-Carlo estimation of the mean chord length in a polygon" Tags="&lt;monte-carlo&gt;" AnswerCount="1" CommentCount="5" />
  <row Id="9296" PostTypeId="2" ParentId="9233" CreationDate="2011-04-07T02:04:49.847" Score="2" ViewCount="" Body="&lt;p&gt;Modeling the data fundamentally (especially for time series) assumes that you have collected data at a sufficient enough frequency to capture the phenomena of interest.&#xA;Simplest example is for a sine wave - if you are collecting data at a frequency of n*pi&#xA;where n is an integer then you will not see anything but zeros and miss the sinusoidal&#xA;pattern altogether.  There are articles on sampling theory which discuss how often&#xA;should data be collected.&lt;/p&gt;&#xA;" OwnerUserId="4063" LastActivityDate="2011-04-07T02:04:49.847" />
  <row Id="9297" PostTypeId="2" ParentId="9283" CreationDate="2011-04-07T02:21:47.643" Score="4" ViewCount="" Body="&lt;p&gt;I've never used R for conjoint analysis, but here are a couple of things I found when I hunted around.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Aizaki and Nishimura (2008) have an article &quot;Design and Analysis of Choice Experiments Using R: A brief introduction&quot; (&lt;a href=&quot;http://tolstoy.newcastle.edu.au/R/e10/help/att-8876/DCE_with_R.pdf&quot; rel=&quot;nofollow&quot;&gt;Free PDF available here&lt;/a&gt;).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Perhaps check out the following packages:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://cran.r-project.org/web/packages/AlgDesign/index.html&quot; rel=&quot;nofollow&quot;&gt;AlgDesign&lt;/a&gt; for constructing choice sets&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://cran.r-project.org/web/packages/prefmod/prefmod.pdf&quot; rel=&quot;nofollow&quot;&gt;prefmod&lt;/a&gt; for analysing paired comparison data&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://cran.r-project.org/web/packages/conf.design/conf.design.pdf&quot; rel=&quot;nofollow&quot;&gt;conf.design&lt;/a&gt; for constructing factorial designs&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastActivityDate="2011-04-07T02:21:47.643" />
  <row Id="9298" PostTypeId="1" AcceptedAnswerId="9303" CreationDate="2011-04-07T02:42:00.870" Score="4" ViewCount="44" Body="&lt;p&gt;Hello,  a journal article has a method for designing experiments to be fit to a 4-parameter logistic model.  The model used is $y= D + \frac{A - D}{1 + (\frac{x}{C}) ^ B}$&lt;br&gt;&#xA;A = upper asymptote&lt;br&gt;&#xA;B = maximum slope&lt;br&gt;&#xA;C = x value when y = 50% of maximum (i.e. 1/2 of upper asymptote)&lt;br&gt;&#xA;D = lower asymptote&lt;br&gt;&#xA;Using pilot data, further experiments are optimally designed by plugging the preliminary parameter estimates into equations presented in the article.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the nonlinear modeling software that I have access to parameterizes the 4-parameter logistic model differently.  The model used is &#xA;$y = D + \frac{A - D}{1 + e^{B(x-C)}}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once I estimate the parameters from the software, how do I translate these to the exponential parameterization used by the software?  Thank you.&lt;/p&gt;&#xA;" OwnerUserId="2473" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-07T07:02:17.783" LastActivityDate="2011-04-07T07:02:17.783" Title="Reconciling different parameterizations of the same nonlinear model" Tags="&lt;nonlinear-regression&gt;" AnswerCount="1" />
  <row Id="9299" PostTypeId="1" CreationDate="2011-04-07T03:02:36.983" Score="10" ViewCount="135" Body="&lt;p&gt;I'm given an $n\times n$ grid of positive integer values. These numbers represent an intensity that should correspond to the strength of belief of a person occupying that grid location (a higher value indicating a higher belief). A person will in general have an influence over multiple grid cells. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I believe that the pattern of intensities should &quot;look Gaussian&quot; in that there will be a central location of high intensity, and then the intensities taper off radially in all directions. Specifically, I'd like to model the values as coming from a &quot;scaled Gaussian&quot; with a parameter for the variance and another for the scale factor.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are two complicating factors:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;the absence of a person will not correspond to a zero value, because of background noise and other effects, but the values should be smaller. They can be erratic though, and at a first approximation might be difficult to model as simple Gaussian noise. &lt;/li&gt;&#xA;&lt;li&gt;The intensity range can vary. For one instance, the values might range between 1 and 10, and in another, between 1 and 100. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I'm looking for an appropriate parameter estimation strategy, or pointers to relevant literature. Pointers to why I'm approaching this problem the wrong way altogether would also be appreciated :). I've been reading about kriging, and Gaussian processes, but that seems like very heavy machinery for my problem. &lt;/p&gt;&#xA;" OwnerUserId="139" LastActivityDate="2011-05-11T03:57:17.720" Title="Estimating parameters for a spatial process" Tags="&lt;estimation&gt;&lt;normal-distribution&gt;&lt;spatial&gt;" AnswerCount="3" CommentCount="8" FavoriteCount="2" />
  <row Id="9301" PostTypeId="1" AcceptedAnswerId="9302" CreationDate="2011-04-07T04:00:19.483" Score="2" ViewCount="43" Body="&lt;p&gt;REF: &lt;a href=&quot;http://en.wikipedia.org/wiki/Order_of_integration&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Order_of_integration&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the interpretation/intuition behind a time series integrable of order 0? I am reading something on cointegration and this is not yet clear to me.&lt;/p&gt;&#xA;" OwnerUserId="862" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-07T06:59:22.310" LastActivityDate="2011-04-07T06:59:22.310" Title="Interpretation of an integrable time series of an order zero" Tags="&lt;time-series&gt;&lt;cointegration&gt;" AnswerCount="1" />
  <row Id="9302" PostTypeId="2" ParentId="9301" CreationDate="2011-04-07T04:56:57.423" Score="2" ViewCount="" Body="&lt;p&gt;Commonly, a time series is said to be $I(0)$ if the time series itself  is stationary (no need to differentiate it to obtain stationarity).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Wikipedia page you mention says that not all $I(0)$ time series are stationary. I didn't know this and I think that indeed many authors do not make the difference. The paper from Engle and Granger (1987) says that all $I(0)$ are stationary.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Two times series $X^1_t$ and $X^2_t$ are said to be cointegrated if &#xA;$\exists n,d&amp;gt;0$ and $\beta \in \mathbb{R}$ so that   &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;$X^1_t \sim I(n)$ &lt;/li&gt;&#xA;&lt;li&gt;$X^2_t \sim I(n)$ &lt;/li&gt;&#xA;&lt;li&gt;$X^1_t+\beta X^2_t \sim I(n-d)$&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;See Engle, Granger (1987) &lt;a href=&quot;http://www.ntuzov.com/Nik_Site/Niks_files/Research/papers/stat_arb/EG_1987.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.ntuzov.com/Nik_Site/Niks_files/Research/papers/stat_arb/EG_1987.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you reach $n-d=0$ indeed your linear combination of time series is stationary.&lt;/p&gt;&#xA;" OwnerUserId="1709" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-04-07T06:24:38.387" LastActivityDate="2011-04-07T06:24:38.387" CommentCount="1" />
  <row Id="9303" PostTypeId="2" ParentId="9298" CreationDate="2011-04-07T05:03:26.757" Score="5" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;These models are not the same&lt;/strong&gt;, because the first is a rational function of $x$ and the second is an exponential function.  The second one is truly a &quot;logistic&quot; model but the first is not.  Moreover, the claims about $B$ and $C$ in the first model are not true.  The maximum slope depends on $C$, which is a scale parameter, not a location parameter.  Moreover, although the maximum slope does depend on $B$, it does so in a complicated fashion.  For instance, when $B = 2$, the maximum slope equals $\frac{9}{8\sqrt{3}}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you write $x = \log(z)$ and $C = \log(\gamma)$ then the second model is &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$y = D + \frac{A-D}{1 + \exp(B(\log(z)-\log(\gamma))} = D + \frac{A-D}{1 + (z/\gamma)^B},$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;which does have the form of the first model.  In other words, in the first model the &lt;em&gt;logarithm&lt;/em&gt; of $z$ has a logistic form.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-04-07T06:18:15.007" LastActivityDate="2011-04-07T06:18:15.007" CommentCount="1" />
  <row Id="9304" PostTypeId="1" AcceptedAnswerId="9316" CreationDate="2011-04-07T06:20:27.157" Score="5" ViewCount="35" Body="&lt;p&gt;A colleague just asked me this question:&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Context:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;A psychological study had&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;2 &lt;code&gt;group&lt;/code&gt;s  of participants (between subjects)&lt;/li&gt;&#xA;&lt;li&gt;4 &lt;code&gt;context&lt;/code&gt;s (within subjects))&lt;/li&gt;&#xA;&lt;li&gt;each participant provided a &lt;code&gt;response&lt;/code&gt; in each of the four contexts&lt;/li&gt;&#xA;&lt;li&gt;there were three categorically distinct response options (lets call them $A$, $B$, and $C$)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;Question&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What is an appropriate statistical model for analysing the effect of &lt;code&gt;group&lt;/code&gt; and &lt;code&gt;context&lt;/code&gt; on &lt;code&gt;response&lt;/code&gt;?&lt;/li&gt;&#xA;&lt;li&gt;What would you tell someone who analysed this data as a 2 by 4 by 3 log-linear model? (e.g., problems with assuming independence of observations)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastActivityDate="2011-04-07T15:20:29.337" Title="Modelling the effect of a 2 by 4 mixed design on a three-level nominal dependent variable" Tags="&lt;modeling&gt;" AnswerCount="1" FavoriteCount="2" />
  <row Id="9305" PostTypeId="1" CreationDate="2011-04-07T07:38:02.097" Score="3" ViewCount="38" Body="&lt;p&gt;Consider a conjugate prior in the k-parameter canonical exponential family:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p(w|n_0,y_0)=c(n_0,y_0) \exp(n_0(y_0 w&amp;#39;-b(w)))$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $n_0&amp;gt;0, y_0 \in Y$ is the pseudo observation vector, $Y$ is an open subset of $\mathbb{R}^k$. I have to prove that the normalization constant $c(n_0,y_0)$ goes to zero when $y_0$ tends to the boundary of $Y$. Any suggestion? I would like to find a proof that works both when $Y$ is bounded and unbounded (e.g. $Y=\mathbb{R}^k$).&lt;/p&gt;&#xA;" OwnerUserId="4065" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-04-08T13:49:49.117" LastActivityDate="2011-04-08T13:49:49.117" Title="The boundary proprerty of conjugate exponential family" Tags="&lt;homework&gt;&lt;normalization&gt;&lt;exponential&gt;" CommentCount="3" FavoriteCount="2" />
  <row Id="9306" PostTypeId="1" CreationDate="2011-04-07T08:30:58.900" Score="6" ViewCount="166" Body="&lt;p&gt;I am using the &lt;code&gt;evolfft&lt;/code&gt; function in the &lt;code&gt;RSEIS&lt;/code&gt; R package to do a STFT analysis of a signal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The signal is one hour long and it was acquired during 3 different conditions, in particular 0-20' control, 20'-40' stimulus, 40'-60' after stimulus.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Visually, I see a change in the spectrogram during these 3 periods, with higher frequency and increased FFT power during the treatment, but I was wondering whether there was some kind of statistical analysis I could do to &quot;put some numbers&quot; on it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestion?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: as suggested I will add an example of the data I'm dealing with&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/5wPy0.png&quot; alt=&quot;STFT example&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The treatment is between 20' and 40', as you can see it produces an increase in the power of the FFT over a fairly wide range of frequencies.&#xA;I have 50-60 of these STFT for each experiment (for 10 total experiments).&#xA;I can average the spectra for each experiments and still get a similar type of pattern. Now, my problem is how to correctly quantify the data I have and possibly do some stats to compare before, during and after the treatment.&lt;/p&gt;&#xA;" OwnerUserId="582" LastEditorUserId="582" LastEditorDisplayName="" LastEditDate="2011-04-14T13:34:52.933" LastActivityDate="2011-04-14T13:34:52.933" Title="STFT statistical analysis" Tags="&lt;time-series&gt;&lt;fourier-transform&gt;" CommentCount="4" FavoriteCount="1" />
  <row Id="9307" PostTypeId="1" AcceptedAnswerId="9308" CreationDate="2011-04-07T10:56:48.250" Score="1" ViewCount="57" Body="&lt;p&gt;What is a family of functions which goes from 0 to 1 and has one or more parameters that alters the rate at which a value of 1 is approached?&lt;/p&gt;&#xA;" OwnerUserId="333" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-07T14:29:27.167" LastActivityDate="2011-04-07T14:29:27.167" Title="Family of functions ranging from zero to one with parameter influencing rate of approach to one" Tags="&lt;function&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="9308" PostTypeId="2" ParentId="9307" CreationDate="2011-04-07T11:16:04.993" Score="1" ViewCount="" Body="&lt;p&gt;How do you like the &lt;a href=&quot;http://en.wikipedia.org/wiki/Sigmoid_function&quot; rel=&quot;nofollow&quot;&gt;Sigmoid Function&lt;/a&gt; ? By adding a parameter k one gets&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$f_k(x)=\frac{1}{1+exp(-k*x)}$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;By setting k you can control how steep the function is and hence how fast it approaches 1. &lt;/p&gt;&#xA;" OwnerUserId="264" LastActivityDate="2011-04-07T11:16:04.993" CommentCount="2" />
  <row Id="9309" PostTypeId="2" ParentId="9307" CreationDate="2011-04-07T11:16:42.047" Score="1" ViewCount="" Body="&lt;p&gt;There's an incredible amount of those functions... and it really depends on what you want to do with it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general any function $f(x)$ with no infinite bounds can be limited to [0;1] by simply using $\frac{f(x)-min}{max-min}$ (where min and max are the minimum and maximum of the function).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One commonly used function is the &lt;a href=&quot;http://en.wikipedia.org/wiki/Sigmoid_function&quot; rel=&quot;nofollow&quot;&gt;sigmoid&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="582" LastActivityDate="2011-04-07T11:16:42.047" />
  <row Id="9311" PostTypeId="1" AcceptedAnswerId="9338" CreationDate="2011-04-07T11:39:13.870" Score="9" ViewCount="165" Body="&lt;p&gt;I can see that there are a lot of formal differences between Kullback–Leibler vs Kolmogorov-Smirnov distance measures.&#xA;However, both are used to measure the distance between distributions.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Is there a typical situation where one should be used instead of the other? &lt;/li&gt;&#xA;&lt;li&gt;What is the rationale to do so?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="3592" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-07T14:26:13.410" LastActivityDate="2011-04-08T04:07:17.223" Title="Kullback–Leibler vs Kolmogorov-Smirnov distance" Tags="&lt;distributions&gt;&lt;distance-functions&gt;&lt;kolmogorov-smirnov&gt;&lt;kullback-leibler&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="3" />
  <row Id="9312" PostTypeId="1" CreationDate="2011-04-07T12:36:40.230" Score="2" ViewCount="172" Body="&lt;p&gt;SPSS returns lower and upper bounds for Reliability. While calculating the Standard Error of Measurement, should we use the Lower and Upper bounds or continue using the Reliability estimate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using the formula : &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\text{SEM}\% =\left(\text{SD}\times\sqrt{1-R_1} \times 1/\text{mean}\right) &amp;#215; 100$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where SD is the standard deviation, $R_1$ is the intraclass correlation for a single measure (one-way ICC).&lt;/p&gt;&#xA;" OwnerUserId="4066" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-08T01:15:46.230" LastActivityDate="2011-04-08T20:40:25.080" Title="How to compute the standard error of measurement (SEM) from a reliability estimate?" Tags="&lt;spss&gt;&lt;reliability&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="1" />
  <row Id="9313" PostTypeId="1" AcceptedAnswerId="9366" CreationDate="2011-04-07T14:32:47.600" Score="3" ViewCount="55" Body="&lt;p&gt;Can you suggest a good review of case control matching algorithms? Algorithms that can be used to set up the matched &lt;em&gt;pairs of one case and one control&lt;/em&gt;, or &lt;em&gt;matched blocks of a case and multiple controls&lt;/em&gt;.(Preferably a paper, book chapter or website discussing recent developments.)&lt;/p&gt;&#xA;" OwnerUserId="3911" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-08T11:28:14.527" LastActivityDate="2011-04-08T19:16:32.820" Title="Review of case control matching algorithms?" Tags="&lt;experiment-design&gt;&lt;matching&gt;&lt;case-control-study&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="1" />
  <row Id="9315" PostTypeId="1" AcceptedAnswerId="9479" CreationDate="2011-04-07T14:42:24.517" Score="1" ViewCount="198" Body="&lt;p&gt;Hi, I have used LDA on a corpus of documents and found some topics. The output of my code is two matrices containing probabilities; one doc-topic probabilities and the other word-topic probabilities. But I actually don't know how to use these results to predict the topic of a new document. I am using Gibbs sampling. Does anyone know how? thanks&lt;/p&gt;&#xA;" OwnerUserId="2986" LastEditorUserId="3369" LastEditorDisplayName="" LastEditDate="2011-04-11T19:06:17.503" LastActivityDate="2011-04-13T18:39:20.870" Title="Topic prediction using latent Dirichlet allocation" Tags="&lt;text-mining&gt;&lt;topic-models&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="2" />
  <row Id="9316" PostTypeId="2" ParentId="9304" CreationDate="2011-04-07T15:20:29.337" Score="4" ViewCount="" Body="&lt;p&gt;A log-linear model or any model that fails to model the dependence of responses could underestimate (or overestimate) standard errors because they do not take into account potential subject-level association of responses. For example, if some subjects are likely to have responses patterns like (A,A,A,A) and others like (C,C,C,C), treating responses as independent is problematic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An appropriate model would be the multinomial logit with subject-level random intercepts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Depending on your colleague's modeling goals, another approach might be latent class regression, which estimates class probabilities and class-conditional responses probabilities for k latent classes. If you expect strong clustering in subjects' responses, this might be a particularly nice approach because you get regression estimates for each of $k$ fuzzy classes, which might have meaningful psychological labels. Identifiability is an issue here because of the large number of parameters. See &lt;code&gt;poLCA&lt;/code&gt; in R and the PDF write-up  &lt;a href=&quot;http://userwww.service.emory.edu/~dlinzer/poLCA/&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;drm&lt;/code&gt; in R is another &lt;a href=&quot;http://cran.r-project.org/web/packages/drm/index.html&quot; rel=&quot;nofollow&quot;&gt;package&lt;/a&gt; which is supposed to be able to model clustered categorical responses, but I have not tried it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, for very specific applications/hypotheses, you could implement resampling methods by resampling entire &lt;em&gt;vectors&lt;/em&gt; of responses -- e.g., a permutation test on odds ratios across groups by permuting group labels without replacement.&lt;/p&gt;&#xA;" OwnerUserId="3432" LastActivityDate="2011-04-07T15:20:29.337" />
  <row Id="9317" PostTypeId="1" AcceptedAnswerId="9319" CreationDate="2011-04-07T15:45:55.837" Score="2" ViewCount="108" Body="&lt;p&gt;I was reading a Wikipedia article on &lt;a href=&quot;http://en.wikipedia.org/wiki/Birthday_problem&quot; rel=&quot;nofollow&quot;&gt;the birthday paradox&lt;/a&gt; and stumbled upon the following statement:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;...the pairings in a group of 23 people are not statistically equivalent to 253 pairs chosen independently...&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Could you explain what does it mean and why does it matter in this case?&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Here is the quotation in its original context:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Although the pairings in a group of 23 people are not statistically equivalent to 253 pairs chosen independently, the birthday paradox becomes less surprising if a group is thought of in terms of the number of possible pairs, rather than as the number of individuals.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="4068" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-04-07T16:48:57.313" LastActivityDate="2011-04-07T16:48:57.313" Title="Statistical equivalence in the birthday paradox" Tags="&lt;probability&gt;&lt;sampling&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="9318" PostTypeId="1" CreationDate="2011-04-07T15:46:39.943" Score="5" ViewCount="103" Body="&lt;p&gt;I am planning to write a program that performs MDS. Any pointers to where I can access the pseudo-code for MDS?&#xA;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="1417" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-07T21:28:15.390" LastActivityDate="2011-04-07T21:28:15.390" Title="Multidimensional scaling pseudo-code" Tags="&lt;algorithms&gt;&lt;multidimensional-scaling&gt;" AnswerCount="2" CommentCount="3" FavoriteCount="1" />
  <row Id="9319" PostTypeId="2" ParentId="9317" CreationDate="2011-04-07T16:25:28.517" Score="7" ViewCount="" Body="&lt;p&gt;In a group of 23 people, all pairs must involve just those 23 people: the pairs are thus mathematically (and statistically) &lt;em&gt;dependent.&lt;/em&gt;  On the other hand, 253 pairs chosen independently and randomly from the 366*365/2 possible pairs will typically involve around 100 separate people.  This (strong) dependency means we cannot use simple formulas for combining probabilities.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This vague statement is in the Wikipedia article to counter the false intuition some people have that birthday collisions in small groups must be rare.  It is, as the article notes, not at all rigorous.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-04-07T16:25:28.517" CommentCount="8" />
  <row Id="9320" PostTypeId="2" ParentId="9295" CreationDate="2011-04-07T16:46:35.707" Score="3" ViewCount="" Body="&lt;p&gt;This is really a question about numerical integration because it reduces immediately to the problem of finding the mean distance between two line segments in the plane.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Monte-Carlo integration would be wasteful and inefficient compared to other methods that are readily available, including&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Direct analytic integration.  This is messy but can be done.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Numeric quadrature.  Adaptive methods will be accurate.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Analytic integration of the mean distance between a point and a line segment and numeric quadrature of that mean distance.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Discrete approximations, such as Riemann sums, the Trapezoidal Rule, etc.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Because the mean distance function in (3) is differentiable and varies relatively slowly, simple numerical methods such as Simpson's Rule work well.  It's best to do the numeric integration over the shorter of the two line segments.  In experiments I find that using the endpoints and midpoint in Simpson's Rule typically gives better than 1% accuracy and using five equally spaced points is far better than that.  (The toughest cases occur where two sides are equally long, parallel, and close together.)  If you need to, you can do the analysis to estimate the Simpson's Rule error and adaptively decrease the point spacing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the original problem, the approximations sometimes overestimate and sometimes underestimate the mean distance, so we could expect some cancellation of errors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At any rate, you will definitely achieve 5% accuracy using method (3).  For an $n$ sided polygon it requires $n$ computations (each of constant cost) per vertex and $n-1$ computations per midpoint for a cost of $O(n^2)$.  The individual computations of mean point-segment distances require some basic arithmetic, four square roots, and two logarithms, so they're quick and easy.  (&lt;em&gt;Mathematica&lt;/em&gt; computes about 7,500 mean segment-segment distances per core per second.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to avoid all but the simplest arithmetic, you can approximate each segment-segment integral with a double Simpson's Rule pass, one for each segment (method 4).  With three division points (requiring the computation of nine distances) you will achieve 5% accuracy.  With five division points (25 distances) you will be better than 1% accurate (usually better than 0.05%) except for extremely close parallel segments.  In &lt;em&gt;Mathematica,&lt;/em&gt; with its interpreted overhead, this technique actually is only half as fast as method (3).&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/KNPjw.png&quot; alt=&quot;Error plot&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This plot shows (negative) log (base 10) relative errors for 1000 segments randomly distributed near the endpoint of a unit segment (which is the larger of the two).  E.g., a value of 2 is a 1% error; a value of 6 is a 0.0001% error.  This is a severe test, because the worst performances occur when the segments are crossing or close to doing so, situations that are impossible or unlikely in convex polygons.  (Note that the smallest possible mean distance is 1/4 (for an infinitesimally short segment near the middle of the unit segment).)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where the approximation is high, the error is plotted on the right, and where the approximation is low, the error is plotted on the left.  Blue dots are the three-point by three-point Simpson's Rule calculation (method 4) and red dots are the three-point Simpson's Rule calculation (method 3).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Typically method 3 (red) has much better than 2% accuracy and is better than method 4 (blue), which in a few cases has almost 10% error.  Method 3 has a tendency to overestimate slightly (more of its values are at the right).  Both methods almost always obtain better than 0.1% accuracy (a height of 3 or better on the plot), especially at mean distances greater than the longer side.  This implies that if you want to improve the accuracy, focus first on nearby edges of a convex polygon.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/Fjgez.png&quot; alt=&quot;The segments&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this plot of the 1000 segments, the brighter, thicker, redder ones have the greatest relative error for Method 3.  The black segment is the reference (unit) segment.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-04-07T18:41:45.193" LastActivityDate="2011-04-07T18:41:45.193" CommentCount="1" />
  <row Id="9321" PostTypeId="2" ParentId="9318" CreationDate="2011-04-07T17:20:37.147" Score="4" ViewCount="" Body="&lt;p&gt;If you have the Statistics Toolbox in MATLAB you can read the source code of &lt;code&gt;mdscale.m&lt;/code&gt;. While it's not pseudocode, it will definitely help you understand MDS better and gives you one approach to coding it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In MATLAB you can type&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;&#xA;edit mdscale&#xA;&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and that will open up an editor window that shows you the &lt;code&gt;mdscale.m&lt;/code&gt; script that does the work. If you don't have MATLAB, check out &lt;a href=&quot;http://scikits.appspot.com/learn&quot; rel=&quot;nofollow&quot;&gt;Scikits.learn&lt;/a&gt;. It has some code for MDS. A lot of times reading Python code is very similar to reading pseudocode!&lt;/p&gt;&#xA;" OwnerUserId="2660" LastActivityDate="2011-04-07T17:20:37.147" CommentCount="1" />
  <row Id="9322" PostTypeId="1" CreationDate="2011-04-07T18:43:13.910" Score="5" ViewCount="75" Body="&lt;p&gt;From &lt;em&gt;Econometrics&lt;/em&gt;, by Fumio Hyashi (Chpt 1):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unconditional Homoskedasticity:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The second moment of the error terms E(εᵢ²) is constant across the observations&lt;/li&gt;&#xA;&lt;li&gt;The functional form E(εᵢ²|xi) is constant across the observations&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Conditional Homoskedasticity:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The restriction that the second moment of the error terms E(εᵢ²) is constant across the observations is lifted&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Thus the conditional second moment E(εᵢ²|xi) can differ across the observations through possible dependence on xᵢ.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;So then, my question:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How does Conditional Homoskedasticity differ from Heteroskedasticity?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My understanding is that there is heteroskedasticity when the second moment differs across observations (xᵢ). &lt;/p&gt;&#xA;" OwnerUserId="4072" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-07T18:47:01.283" LastActivityDate="2011-04-07T18:47:01.283" Title="Conditional homoskedasticity vs heteroskedasticity" Tags="&lt;regression&gt;&lt;econometrics&gt;&lt;assumptions&gt;" CommentCount="3" FavoriteCount="1" />
  <row Id="9323" PostTypeId="2" ParentId="9299" CreationDate="2011-04-07T19:45:16.923" Score="2" ViewCount="" Body="&lt;p&gt;Your model is a two-dimensional random field $X[i,j]$, and you are trying to estimate the joint distribution of the integer-values random variables $X[i,j]$.  You will want to assume spatial stationarity: that is, the joint distribution of $(X[i_1,j_1],...,X[i_m,j_m])$ is the same as the joint distribution of $(X[i_1+k,j_1+l]...,X[i_m+k,j_m+l])$.  In particular, the marginal distribution is the same for every cell. A simple question to ask is the autocorrelation structure of the field.  That is, what is $corr(X[i_1,j_1],X[i_2,j_2])$ given the distance $d([i_1,j_1],[i_2,j_2])$?  We represent this as a function $\rho(d)$.  A simple model for the autocorrelation structure is $\rho(d)=kd^{-1}$, where $k$ is a constant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A 'gaussian' effect corresponds to a quadratic distance function, but there are many other distance functions you should consider, such as the taxicab norm $d([i_1,j_1],[i_2,j_2]) = |i_1-i_2|+|j_1-j_2|$.  Once you have decided on a distance function and the form of your model for autocorrelation it is simple enough to estimate $\rho(d)$ e.g. via maximum likelihood.  For more ideas, look for &quot;random field&quot;.&lt;/p&gt;&#xA;" OwnerUserId="3567" LastEditorUserId="3567" LastEditorDisplayName="" LastEditDate="2011-04-07T19:59:54.003" LastActivityDate="2011-04-07T19:59:54.003" CommentCount="8" />
  <row Id="9324" PostTypeId="1" AcceptedAnswerId="9356" CreationDate="2011-04-07T20:34:34.710" Score="0" ViewCount="133" Body="&lt;p&gt;I'd love a check if anyone is willing!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am trying to see if there is a statistical difference in female size between sites. Over the years females were repeatedly sampled within sites. I have sampled females opportunistically. Meaning that females were sampled a different number of times between and within sites.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My formula is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; lmerfit1&amp;lt;-lmer(size ~ (1|FEMALE), data=Data)&#xA;&amp;gt; lmerfit2&amp;lt;-lmer(size ~ SITE+(1|FEMALE), data=Data)&#xA;&amp;gt; anova(lmerfit1, lmerfit2)&#xA;Data: Data&#xA;Models:&#xA;lmerfit1: size ~ (1 | FEMALE)&#xA;lmerfit2: size ~ SITE + (1 | FEMALE)&#xA;         Df    AIC    BIC  logLik Chisq Chi Df Pr(&amp;gt;Chisq)&#xA;lmerfit1  3 2167.8 2179.6 -1080.9                        &#xA;lmerfit2  4 2169.8 2185.5 -1080.9     0      1          **1**&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;A p value of &lt;strong&gt;1&lt;/strong&gt; leaves me concerned. The other female traits I ran thru this same formula made sense.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;thanks! &lt;/p&gt;&#xA;" OwnerUserId="4027" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-08T02:47:19.363" LastActivityDate="2011-04-08T14:43:53.077" Title="Check of R command and output of unbalanced repeated measures ANOVA" Tags="&lt;r&gt;&lt;repeated-measures&gt;&lt;lmer&gt;" AnswerCount="2" CommentCount="3" FavoriteCount="0" />
  <row Id="9325" PostTypeId="2" ParentId="8738" CreationDate="2011-04-07T21:01:14.247" Score="2" ViewCount="" Body="&lt;p&gt;Searching the SAS documentation for ARIMAX (based on &lt;a href=&quot;http://stats.stackexchange.com/questions/8738/what-is-the-term-for-a-time-series-regression-having-more-than-one-predictor/8741#8741&quot;&gt;Patrick's answer&lt;/a&gt;), I found (in &quot;&lt;a href=&quot;http://support.sas.com/documentation/cdl/en/etsug/60372/HTML/default/viewer.htm#etsug_arima_sect012.htm&quot; rel=&quot;nofollow&quot;&gt;The ARIMA Procedure: Input Variables and Regression with ARMA Errors&lt;/a&gt;&quot;) that they listed the following terms:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;ARIMAX&lt;/li&gt;&#xA;&lt;li&gt;Box-Tiao&lt;/li&gt;&#xA;&lt;li&gt;Transfer Function Model&lt;/li&gt;&#xA;&lt;li&gt;Dynamic Regression&lt;/li&gt;&#xA;&lt;li&gt;Intervention Model&lt;/li&gt;&#xA;&lt;li&gt;Interrupted Time Series Model&lt;/li&gt;&#xA;&lt;li&gt;Regression Model with ARMA Errors&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The term &lt;em&gt;ARIMAX&lt;/em&gt; was used almost equally as much as the term &lt;em&gt;Transfer Function Model&lt;/em&gt;. &lt;em&gt;Intervention Model&lt;/em&gt; and &lt;em&gt;Interrupted Time Series Model&lt;/em&gt; appear to refer to a different kind of model than the others.&lt;/p&gt;&#xA;" OwnerUserId="1583" LastActivityDate="2011-04-07T21:01:14.247" />
  <row Id="9326" PostTypeId="2" ParentId="9318" CreationDate="2011-04-07T21:27:47.123" Score="7" ViewCount="" Body="&lt;p&gt;There are different kind of MDS (e.g., see this &lt;a href=&quot;http://forrest.psych.unc.edu/teaching/p208a/mds/mds.html&quot; rel=&quot;nofollow&quot;&gt;brief review&lt;/a&gt;). Here are two pointers:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;the &lt;a href=&quot;http://cran.r-project.org/web/packages/smacof/index.html&quot; rel=&quot;nofollow&quot;&gt;smacof&lt;/a&gt; R package, developed by Jan de Leeuw and Patrick Mair has a nice vignette, &lt;a href=&quot;http://cran.r-project.org/web/packages/smacof/vignettes/smacof.pdf&quot; rel=&quot;nofollow&quot;&gt;Multidimensional Scaling Using Majorization: SMACOF in R&lt;/a&gt; (or see, the &lt;em&gt;Journal of Statistical Software&lt;/em&gt; (2009) 31(3)) -- R code is available, of course.&lt;/li&gt;&#xA;&lt;li&gt;there are some handouts on &lt;a href=&quot;http://forrest.psych.unc.edu/teaching/p230/p230.html&quot; rel=&quot;nofollow&quot;&gt;Multidimensional Scaling&lt;/a&gt;, by Forrest Young, where several algorithms are discussed (including INDSCAL (Individual Difference Scaling, or weighted MDS) and ALSCAL, with Fortran &lt;a href=&quot;http://forrest.psych.unc.edu/research/alscal.html&quot; rel=&quot;nofollow&quot;&gt;source code&lt;/a&gt; by the same author) -- this two keywords should help you to find other source code (mostly Fortran, C, or Lisp). &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You can also look for &quot;Manifold learning&quot; which should give you a lot of techniques for dimension reduction (Isomap, PCA, MDS, etc.); the term was coined by the Machine Learning community, among others, and they probably have a different view on MDS compared to psychometricians.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-04-07T21:27:47.123" />
  <row Id="9327" PostTypeId="1" CreationDate="2011-04-07T21:30:43.483" Score="2" ViewCount="225" Body="&lt;p&gt;EDIT: Thanks for the help so far. I have updated the question based on further work. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am interested in finding any difference in response of counts of two competing species to dryness and elevation. Both are caught in the same type of trap, but trap placement (bushes vs buildings) probably biases toward one species or the other, so each trap placement is rated according to indoors/outdoors on a scale of 1-3. A trap does not move once it is placed. Traps are nested in sites, dryness is measured at each site/visit combination, and elevation is of course constant at each site.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My data includes the number of each species caught in 4-12 traps, at 8-12 sites during 6-8 visits to each site. The visits and sites are far enough spaced that I assume they are independent. Most data are zeros, especially for Sp_A which is rare. Below is the structure of the data.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;R&amp;gt; str(mydata)&#xA;'data.frame':   300 obs. of  8 variables:&#xA;     $ count    : num  0 5 1 0 1 1 0 0 0 0 ...&#xD;&#xA;     $ species  : Factor w/ 2 levels &quot;a&quot;,&quot;b&quot;: 1 1 1 1 1 1 1 1 1 1 ...&#xA;     $ elevation: int  1 1 1 1 1 1 1 1 1 1 ...&#xD;&#xA;     $ dryness  : num  0.179 0.179 0.179 0.179 0.179 ...&#xA;     $ site     : Factor w/ 5 levels &amp;quot;a&amp;quot;,&amp;quot;b&amp;quot;,&amp;quot;c&amp;quot;,&amp;quot;d&amp;quot;,..: 1 1 1 1 1 1 1 1 1 1 ...&#xD;&#xA;     $ trap     : Factor w/ 50 levels &quot;MT10a&quot;,&quot;MT10b&quot;,..: 6 11 16 21 26 31 36 41 46 1 ...&#xA;     $ visit    : Factor w/ 3 levels &amp;quot;1&amp;quot;,&amp;quot;2&amp;quot;,&amp;quot;3&amp;quot;: 1 1 1 1 1 1 1 1 1 1 ...&#xD;&#xA;     $ in_out   : num  1 1 1 1 1 1 1 1 1 1 ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As an initial attempt, I had no trouble using R function &lt;code&gt;lmer&lt;/code&gt; to model Sp_A presence/absence with Sp_B as a covariate: &lt;code&gt;(SpA&amp;gt;0) ~ Sp_B + elevation + (1|site/trap) + (1|visit)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However on suggestion from R-sig-mixed-models list, and also &quot;ils&quot; from this site, I have reshaped the data as seen above, and tried to model count with species as a factor in the model. I have used two different packages (lme4 and glmmADMB). The functions handle the data ok in simple models, until I add the factor &quot;species.&quot; Below are the R code and error messages. These same errors also happen when I use simpler dummy data. Any ideas on syntax or other packages?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(lme4)&#xA;glmera1 &amp;lt;- glmer(count~elevation*species*in_out &#xA;                 + (1|visit) + (1|site/trap), &#xA;                 data=mydata, family=&quot;poisson&quot;)&#xA;Error in asMethod(object) : matrix is not symmetric [1,2]&#xA;In addition: Warning messages:&#xA;1: In mer_finalize(ans) :&#xA;  Cholmod warning 'not positive definite' at file:../Cholesky/t_cholmod_rowfac.c, line 432&#xA;2: In mer_finalize(ans) :&#xA;  Cholmod warning 'not positive definite' at file:../Cholesky/t_cholmod_rowfac.c, line 432&#xA;3: In mer_finalize(ans) : singular convergence (7)&#xA;&#xA;require(glmmADMB)&#xA;admba1 &amp;lt;- glmm.admb(count~elevation*species*in_out, random=~visit, &#xA;                    group=&quot;site&quot;, data=mydata, family=&quot;nbinom&quot;)&#xA;Error in glmm.admb(count ~ elevation * species * in_out, random = ~visit,  : &#xA;  The function maximizer failed&#xA;In addition: Warning message:&#xA;running command './nbmm -maxfn 500 ' had status 1   &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2993" LastEditorUserId="2993" LastEditorDisplayName="" LastEditDate="2011-04-20T22:38:40.473" LastActivityDate="2011-04-20T22:38:40.473" Title="Count data with mixed effects" Tags="&lt;r&gt;&lt;mixed-model&gt;&lt;repeated-measures&gt;" AnswerCount="2" CommentCount="7" FavoriteCount="1" />
  <row Id="9328" PostTypeId="2" ParentId="9327" CreationDate="2011-04-07T21:51:47.123" Score="3" ViewCount="" Body="&lt;p&gt;Check out this question: &lt;a href=&quot;http://stats.stackexchange.com/questions/486/negative-values-for-aicc-corrected-akaike-information-criterion&quot;&gt;Negative values for AICc (corrected Akaike Information Criterion)&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The same explanation holds for BIC.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2011-04-07T21:51:47.123" />
  <row Id="9329" PostTypeId="1" CreationDate="2011-04-07T22:21:29.077" Score="7" ViewCount="155" Body="&lt;p&gt;Say I've got a predictive classification model based on a random forest (using the randomForest package in R). I'd like to set it up so that end-users can specify an item to generate a prediction for, and it'll output a classification likelihood. So far, no problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But it would be useful/cool to be able to output something like a variable importance graph, but for the specific item being predicted, not for the training set. Something like:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Item X is predicted to be a Dog (73% likely)&lt;br&gt;&#xA;Because:&lt;br&gt;&#xA;  Legs=4&lt;br&gt;&#xA;  Breath=bad&lt;br&gt;&#xA;  Fur=short&lt;br&gt;&#xA;  Food=nasty  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You get the point. Is there a standard, or at least justifiable, way of extracting this information from a trained random forest? If so, does anyone have code that will do this for the randomForest package?&lt;/p&gt;&#xA;" OwnerUserId="6" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-09T00:36:01.243" LastActivityDate="2011-04-09T00:36:01.243" Title="Is there a way to explain a prediction from a random forest model?" Tags="&lt;machine-learning&gt;&lt;random-forest&gt;" AnswerCount="1" CommentCount="4" FavoriteCount="2" />
  <row Id="9330" PostTypeId="1" CreationDate="2011-04-08T01:35:42.510" Score="6" ViewCount="133" Body="&lt;p&gt;We have a series of experiments where we measure virus transmission to plants when exposed to virus-infected insects for different time periods, so all of the experiments have similar types of independent and dependent variables.  In one experiment, there are 6 time periods (1 to 24 hours) and 25 plants were tested (individually) for each time period.  The response for each plant is yes or no (Plants are scored as virus infected).  For 2 of the time intervals, all of the plants were negative for virus infection (0/25 for each time interval).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using PROC GLIMMIX in SAS for the analyses.  For all of the other experiments, using a binary distribution in the model statement gives reasonable results.  For the experiment where two of the time intervals had 0 positive plants, if I use a binary distribution in the model statement, the standard errors for the two groups with 0 transmissions are huge, thus distorting the results.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I use a negative binomial distribution (based on counts of virus positive plants) the results seem reasonable.  Since the same number of plants were tested for each time interval, using this approach works, but it differs from the other experiments.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a method to adjust/account for the zeros in treatment groups that would allow the binary distribution return reasonable results?    &lt;/p&gt;&#xA;" OwnerUserId="4076" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-08T08:23:11.637" LastActivityDate="2011-04-13T02:40:18.117" Title="Binomial data analysis with all 0 responses for some treatment groups" Tags="&lt;binomial&gt;&lt;sas&gt;" AnswerCount="3" CommentCount="1" FavoriteCount="1" />
  <row Id="9331" PostTypeId="1" AcceptedAnswerId="9339" CreationDate="2011-04-08T01:36:55.427" Score="4" ViewCount="132" Body="&lt;p&gt;I am new to the use of cubic splines for regression purposes and wanted to find out &lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) What is a good source (besides ESL which I read but am still uncertain) to learn about splines for regression? &lt;br/&gt;&#xA;2) How would you calculate the basis of a given natural cubic spline solution on new data? Specifically if one were to do the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data(iris)&#xA;colnames(iris)&#xA;Sepal.Length.ns&amp;lt;-ns(iris$Sepal.Length,df=5)&#xA;Sepal.Length.ns&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How would you take the information in Sepal.Length.ns (knots, boundaries) and compute the values for a new observation? The reason is to code this process outside of R, once fit in R initially (i.e. to put a regression model using cubic splines into a production system).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example I can do this in R, but want to understand the calculation:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#three new observations to predict&#xA;newVector&amp;lt;-c(4.45,3.35,2.2)&#xA;pred.new&amp;lt;-predict(Sepal.Length.ns,newVector)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="2040" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-08T08:22:37.160" LastActivityDate="2011-04-08T20:21:18.013" Title="Calculation of natural cubic splines in R" Tags="&lt;r&gt;&lt;splines&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="1" />
  <row Id="9332" PostTypeId="2" ParentId="9324" CreationDate="2011-04-08T02:21:51.427" Score="1" ViewCount="" Body="&lt;p&gt;How many sites did you have?  The models only differ by one df, so either you only have two sites or you treated site as a continuous variable when it should have been categorical.  If it should have been a factor, use &lt;code&gt;factor(SITE)&lt;/code&gt; instead of &lt;code&gt;SITE&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, try plotting the data (always a good idea!) -- do you see any visual differences?&lt;/p&gt;&#xA;" OwnerUserId="3601" LastActivityDate="2011-04-08T02:21:51.427" CommentCount="1" />
  <row Id="9333" PostTypeId="1" CreationDate="2011-04-08T02:27:00.023" Score="3" ViewCount="31" Body="&lt;h3&gt;Context:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;I have a series of figures for car sales that show me (a) the usual number of car sales for particular models and (b) the number of car sales by a particular car salesperson for each model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say the values are:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Model | Sales by all Salespeople | Sales by Salesperson x&#xA;    A |                      100 |                     20&#xA;    B |                       50 |                     40&#xA;    C |                       50 |                      0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to find out if Salesperson x is significantly more responsible for sales of a particular model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My first thought is to determine the rate of sales per model vs the rate of sales per model for the salesperson, i.e.:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Model | % Total Sales for all Salespeople | % Total Sales for Salesperson x&#xA;    A |                               50% |                             33%&#xA;    B |                               25% |                             66%&#xA;    C |                               25% |                              0%&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, naturally every salesperson has a significant variation in the cars they sell.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Question:&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;How do I determine if salesperson variation from the mean is statistically significant?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;Initial Thoughts:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;I &lt;em&gt;think&lt;/em&gt; the Pearson correlation has some bearing, as perhaps may chi-square distribution, but I really don't have the background yet to understand why, so introductory help is appreciated.&lt;/p&gt;&#xA;" OwnerUserId="4078" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-08T02:51:39.207" LastActivityDate="2011-04-08T02:55:10.523" Title="Determining the relationship between salesperson and products sold?" Tags="&lt;correlation&gt;&lt;variance&gt;&lt;basic-concepts&gt;" AnswerCount="1" CommentCount="3" FavoriteCount="2" />
  <row Id="9334" PostTypeId="1" CreationDate="2011-04-08T02:46:48.813" Score="7" ViewCount="160" Body="&lt;h3&gt;Context:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;From a question on Mathematics Stack Exchange &lt;a href=&quot;http://math.stackexchange.com/questions/31649/can-i-build-a-program-that-will-tell-me-if-a-real-world-data-set-looks-linear-lo/&quot;&gt;(Can I build a program)&lt;/a&gt;, someone has a set of $x-y$ points, and wants to fit a curve to it, linear, exponential or logarithmic. &#xA;The usual method is to start by choosing one of these (which specifies the model), and then do the statistical calculations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But what is really wanted is to find the 'best' curve out of linear, exponential or logarithmic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ostensibly, one could try all three, and choose the best fitted curve of the three according to the best correlation coefficient.&#xA;But somehow I'm feeling this is not quite kosher.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Questions:&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Is it appropriate to choose the best fitting model out of linear, exponential, and logarithmic models, based on a comparison of fit statistics?&lt;/li&gt;&#xA;&lt;li&gt;If so, what is the most appropriate way to do this?&lt;/li&gt;&#xA;&lt;li&gt;If regression helps find parameters (coefficients) in a function, why can't there be a discrete parameter to choose which of three curve families the best would come from?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="3186" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-08T03:26:18.630" LastActivityDate="2011-05-19T08:55:18.997" Title="Determining best fitting curve fitting function out of linear, exponential, and logarithmic functions" Tags="&lt;regression&gt;&lt;predictive-models&gt;&lt;curve-fitting&gt;" AnswerCount="3" FavoriteCount="1" />
  <row Id="9335" PostTypeId="2" ParentId="9333" CreationDate="2011-04-08T02:55:10.523" Score="1" ViewCount="" Body="&lt;p&gt;Chi square test, but I'd set it up as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Model  |  this salesman  |  all other salesmen&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A       |       20      |        80&lt;/p&gt;&#xA;&#xA;&lt;p&gt;B       |       40       |       10&lt;/p&gt;&#xA;&#xA;&lt;p&gt;etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Chi square test of independence, with expected values equal to row total * column total / grand total.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Null hypothesis: this salesman = same as all other.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most stat books will have this test. &lt;/p&gt;&#xA;" OwnerUserId="3919" LastActivityDate="2011-04-08T02:55:10.523" />
  <row Id="9336" PostTypeId="2" ParentId="9334" CreationDate="2011-04-08T03:19:19.230" Score="3" ViewCount="" Body="&lt;p&gt;Since plenty of people routinely explore the fit of various curves to their data, I don't know where your reservations are coming from.  Granted, there is the fact that a quadratic will always fit at least as well as a linear, and a cubic, at least as well as a quadratic, so there are ways to test the statistical significance of adding such a nonlinear term and thus to avoid needless complexity.  But the basic practice of testing many different forms of a relationship is just good practice.  In fact, one might start with a very flexible loess regression to see what is the most plausible kind of curve to fit. &lt;/p&gt;&#xA;" OwnerUserId="2669" LastActivityDate="2011-04-08T03:19:19.230" CommentCount="9" />
  <row Id="9337" PostTypeId="2" ParentId="9334" CreationDate="2011-04-08T03:41:09.710" Score="4" ViewCount="" Body="&lt;ul&gt;&#xA;&lt;li&gt;You might want to check out the free software called &lt;a href=&quot;http://ccsl.mae.cornell.edu/eureqa&quot; rel=&quot;nofollow&quot;&gt;Eureqa&lt;/a&gt;. It has the specific aim of automating the process of finding both the functional form and the parameters of a given functional relationship.&lt;/li&gt;&#xA;&lt;li&gt;If you are comparing models, with different numbers of parameters, you will generally want to use a measure of fit that penalises models with more parameters. There is a rich literature on which fit measure is most appropriate for model comparison, and issues get more complicated when the models are not nested. I'd be interested to hear what others think is the most suitable model comparison index given your scenario (as a side point, there was recently a &lt;a href=&quot;http://jeromyanglim.blogspot.com/2011/02/r-optimisation-tips-using-optim-and.html&quot; rel=&quot;nofollow&quot;&gt;discussion on my blog&lt;/a&gt; about model comparison indices in the context of comparing models for curve fitting).&lt;/li&gt;&#xA;&lt;li&gt;From my experience, non-linear regression models are used for reasons beyond pure statistical fit to the given data: &#xA;&lt;ol&gt;&#xA;&lt;li&gt;Non-linear models make more plausible predictions outside the range of the data&lt;/li&gt;&#xA;&lt;li&gt;Non-linear models require fewer parameters for equivalent fit&lt;/li&gt;&#xA;&lt;li&gt;Non-linear regression models are often applied in domains where there is substantial prior research and theory guiding model selection.&lt;/li&gt;&#xA;&lt;/ol&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastActivityDate="2011-04-08T03:41:09.710" />
  <row Id="9338" PostTypeId="2" ParentId="9311" CreationDate="2011-04-08T04:07:17.223" Score="7" ViewCount="" Body="&lt;p&gt;The KL-divergence is typically used in information-theoretic settings, or even Bayesian settings, to measure the information change between distributions before and after applying some inference, for example. It's not a distance in the typical (metric) sense, because of lack of symmetry and triangle inequality, and so it's used in places where the directionality is meaningful. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The KS-distance is typically used in the context of a non-parametric test. In fact, I've rarely seen it used as a generic &quot;distance between distributions&quot;, where the $\ell_1$ distance, the Jensen-Shannon distance, and other distances are more common. &lt;/p&gt;&#xA;" OwnerUserId="139" LastActivityDate="2011-04-08T04:07:17.223" CommentCount="2" />
  <row Id="9339" PostTypeId="2" ParentId="9331" CreationDate="2011-04-08T05:48:32.833" Score="3" ViewCount="" Body="&lt;p&gt;Wikipedia has a &lt;a href=&quot;http://en.wikipedia.org/wiki/Spline_interpolation&quot; rel=&quot;nofollow&quot;&gt;nice explanation&lt;/a&gt; of spline interpolation&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I posted the code to create cubic &lt;a href=&quot;http://en.wikipedia.org/wiki/B%C3%A9zier_spline&quot; rel=&quot;nofollow&quot;&gt;Bezier splines&lt;/a&gt; on &lt;a href=&quot;http://rosettacode.org/wiki/Bitmap/B%C3%A9zier_curves/Cubic#R&quot; rel=&quot;nofollow&quot;&gt;Rosettacode&lt;/a&gt; a while ago.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, you can have a look at &lt;a href=&quot;http://stackoverflow.com/questions/881143/cubic-spline-extrapolation&quot;&gt;this discussion&lt;/a&gt; on SO about spline extrapolation.&lt;/p&gt;&#xA;" OwnerUserId="582" LastEditorUserId="2040" LastEditorDisplayName="" LastEditDate="2011-04-08T17:55:56.330" LastActivityDate="2011-04-08T17:55:56.330" CommentCount="5" />
  <row Id="9340" PostTypeId="2" ParentId="9330" CreationDate="2011-04-08T07:11:22.210" Score="4" ViewCount="" Body="&lt;p&gt;The problem with zeroes, is that the data do not rule out arbitrarily small proportions.  So your prior information must be assessed more carefully, because in this case &lt;em&gt;it still matters&lt;/em&gt;.  Details which are irrelevant when the prior information is &quot;swamped&quot; by the data can be important.  In this type of problem, the population size $N$ becomes important, but a binomial assumes $N\to\infty$, which gives absurd results, if this limit is applied too soon in the calculations (as your standard errors indicates).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this case, there is relatively straight-forward approximation, you just replace $\frac{0}{25}$ with $\frac{1}{27}$, which is a Bayesian estimate based on a uniform prior for the true fraction of &quot;positive infections&quot;.  Given that you are using GLIMMIX - doing anything more sophisticated will likely wreck your SAS program.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To be consistent, it may be worthwhile to replace all proportions $\frac{r}{n}$ with $\frac{r+1}{n+2}$ - however it shouldn't influence your results too much.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-04-08T07:11:22.210" />
  <row Id="9341" PostTypeId="1" AcceptedAnswerId="9344" CreationDate="2011-04-08T07:18:45.413" Score="4" ViewCount="69" Body="&lt;p&gt;I have a multiple linear regression problem $y=X\beta+\epsilon$. The number of observations $m$ is large, so by the time the data gets to me it's been summarized into:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;$m$&lt;/li&gt;&#xA;&lt;li&gt;$X^TX$&lt;/li&gt;&#xA;&lt;li&gt;$X^Ty$&lt;/li&gt;&#xA;&lt;li&gt;$y^Ty$&lt;/li&gt;&#xA;&lt;li&gt;$\sum_{i=1}^m{y_i}$&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The above list appears sufficient to compute the OLS estimate. But can it be used to compute a regularized estimate of some sort (e.g. ridge regression)? If not, can the list be augmented in the same spirit to enable regularization?&lt;/p&gt;&#xA;" OwnerUserId="439" LastEditorUserId="439" LastEditorDisplayName="" LastEditDate="2011-04-08T11:29:40.227" LastActivityDate="2011-04-08T11:29:40.227" Title="Regularized fit from summarized data" Tags="&lt;regression&gt;&lt;lasso&gt;&lt;regularization&gt;&lt;ridge-regression&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="9342" PostTypeId="1" CreationDate="2011-04-08T07:47:43.787" Score="7" ViewCount="169" Body="&lt;p&gt;I am using hierarchical clustering to analyse time series data. My code is implemented using the &lt;em&gt;Mathematica&lt;/em&gt; function &lt;code&gt;DirectAgglomerate[...]&lt;/code&gt;, which generates hierarchical clusters given the following inputs:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;a distance matrix D&lt;/li&gt;&#xA;&lt;li&gt;the name of the method used to determine inter-cluster linkage.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I have calculated the distance matrix D using Manhattan distance:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$d(x,y) = \sum_i|x_i - y_i|$, where $i = 1,\cdots, n$ and $n \approx 150$ is the number of data points in my timeseries.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is, is it ok to use Ward's inter-cluster linkage with a Manhattan distance matrix?  Some sources suggest that Ward's linkage should only be used with Euclidean distance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that &lt;code&gt;DirectAgglomerate[...]&lt;/code&gt; calculates Ward's linkage using the distance matrix only, not the original observations.  Unfortunately, I am unsure how &lt;em&gt;Mathematica&lt;/em&gt; modifies Ward's original algorithm, which (from my understanding) worked by minimizing the error sum of squares of the observations, calculated with respect to the cluster mean.  For example, for a cluster $c$ consisting of a vector of univariate observations, Ward formulated the error sum of squares as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$(\sum_j||c_j - mean(c)||_2)^2$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Other software tools such as Matlab and R also implement Ward's clustering using just a distance matrix so the question isn't specific to Mathematica).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thankyou very much for your help!&lt;/p&gt;&#xA;" OwnerUserId="4079" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-09T12:57:05.640" LastActivityDate="2011-04-10T02:10:19.477" Title="Is it ok to use Manhattan distance with Ward's inter-cluster linkage in hierarchical clustering?" Tags="&lt;clustering&gt;&lt;distance-functions&gt;" AnswerCount="3" CommentCount="2" FavoriteCount="1" />
  <row Id="9343" PostTypeId="1" CreationDate="2011-04-08T08:14:07.460" Score="4" ViewCount="27" Body="&lt;p&gt;Hi&#xA;I am looking at extremely non linear data for which the ARMA/ARIMA models do not work well. Though, I see some autocorrelation, and I suspect to have better results for non linear autocorrelation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1/ is there an equivalent of the PACF for rank correlation? (in R?)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2/ is there an equivalent of ARMA model for non linear / rank correlation (in R?)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;fRed&lt;/p&gt;&#xA;" OwnerUserId="1709" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-04-08T10:12:34.683" LastActivityDate="2011-04-08T10:12:34.683" Title="Is there an equivalent of ARMA for rank correlation?" Tags="&lt;r&gt;&lt;rank-correlation&gt;&lt;arma&gt;" CommentCount="1" FavoriteCount="1" />
  <row Id="9344" PostTypeId="2" ParentId="9341" CreationDate="2011-04-08T08:23:08.517" Score="4" ViewCount="" Body="&lt;p&gt;It would be nice to have in addition the covariance matrix of the residuals $\hat{\Sigma}$ to draw common inferences about the significance of estimated parameters, or if you are sure it is homoscedastic, then just $\hat{\sigma}^2$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for the regularizations of &lt;em&gt;generalised least squares&lt;/em&gt; type (probably including instrumental variables estimators) the answer will be no. You need the original data matrices (though if you are supplied by $X^T \Omega^{-1}X$ and $X^T\Omega^{-1}y$ you may do GLS, but you loose the control for the choice of $\Omega$ anyway). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For general non-linear &lt;em&gt;&lt;a href=&quot;http://www-stat.stanford.edu/~tibs/lasso/lasso.pdf&quot; rel=&quot;nofollow&quot;&gt;lasso&lt;/a&gt;&lt;/em&gt; regularization it would be even more complicated. Luckily it may be approximated by ridge regression (see p. 273 in the reference) of a special type.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regarding ordinary &lt;em&gt;ridge regression&lt;/em&gt; it is sufficient, since all you need to do in this case is just to add elements to the diagonal $X^TX+\delta I$, where $I$ is an identity matrix. Thus in this particular case it works well. &lt;/p&gt;&#xA;" OwnerUserId="2645" LastEditorUserId="2645" LastEditorDisplayName="" LastEditDate="2011-04-08T08:38:34.733" LastActivityDate="2011-04-08T08:38:34.733" CommentCount="1" />
  <row Id="9345" PostTypeId="1" CreationDate="2011-04-08T11:01:21.077" Score="1" ViewCount="38" Body="&lt;p&gt;When doing multivariate regression, it's often the case that some predictors often have many zero values - dichotomous inputs, dummy coding of polychotomous inputs, interval coding, etc.  The fraction of nonzero observations for a predictor is especially decreased when interacting these variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems obvious that only the observations of a predictor for which it is nonzero (after any transformations) will affect the estimation of its coefficient.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When doing multivariate regression I like to use weakly informative priors (like Gaussian or Cauchy) for regularization.  I've noticed that sometimes, predictors which are usually zero still take on implausible regression coefficients, like ~6 for a logistic regression model.  This seems to be because a variable that is nonzero for only a handful of observations is still overcoming the prior because it's getting full credit for all of its zero observations, and thus getting a huge coefficient, which is in many cases a prior obviously overfitting, and hurts generalization performance for prediction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Multiplying the prior's scale by the fraction of observations of a predictor that are nonzero fixes this and greatly improves generalization performance, makes coefficients plausible, and seems to be common sense.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So - why don't more people do this?  I've tried a cursory search in common textbooks (various machine learning books, applied and theoretical Bayesian statistics), R packages on CRAN, some machine learning packages, and I don't see evidence of anyone else doing this...  isn't this just common sense?  Or is there something wrong with this approach that I'm missing?&lt;/p&gt;&#xA;" OwnerUserId="1119" LastActivityDate="2011-04-08T15:24:16.877" Title="Why doesn't it seem to be standard to multiply prior scale by fraction of non-zero predictor observations?" Tags="&lt;regression&gt;&lt;bayesian&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="9347" PostTypeId="2" ParentId="7314" CreationDate="2011-04-08T11:45:51.657" Score="0" ViewCount="" Body="&lt;p&gt;My bad. I am trying to use too many frameworks at once, and failing to cover documentation properly. The asia example in winbugs does exactly what I've been asking for. &lt;/p&gt;&#xA;" OwnerUserId="3280" LastActivityDate="2011-04-08T11:45:51.657" />
  <row Id="9348" PostTypeId="2" ParentId="9342" CreationDate="2011-04-08T11:46:37.697" Score="2" ViewCount="" Body="&lt;p&gt;I can't think of any reason why Ward should favor any metric. Ward's method is just another option to decide which clusters to fusion next during agglomeration. This is achieved by finding the two clusters whose fusion will minimize a certain error (&lt;a href=&quot;http://www.stat.psu.edu/online/courses/stat505/18_cluster/09_cluster_wards.html&quot; rel=&quot;nofollow&quot;&gt;examplary source for the formula&lt;/a&gt;). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hence it relies on two concepts:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The mean of vectors which (for numerical vectors) is generally calculated by averaging over every dimension separately.&lt;/li&gt;&#xA;&lt;li&gt;The distance metric itself i.e. the concept of similarity expressed by this metric.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;So: As long as the properties of the choosen metric (like e.g. rotation,translation or scale invariance) satisfy your needs (and the metric fits to the way the cluster mean is calculated), I don't see any reason to not use it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suspect that most people suggest the euclidean metric because they&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;want to increase the weight of the differences between a cluster mean and a single observation vector (which is done by quadration)&lt;/li&gt;&#xA;&lt;li&gt;or because it came out as best metric in the validation based on &lt;em&gt;their&lt;/em&gt; data&lt;/li&gt;&#xA;&lt;li&gt;or because it is used in general.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="264" LastActivityDate="2011-04-08T11:46:37.697" CommentCount="2" />
  <row Id="9350" PostTypeId="2" ParentId="9299" CreationDate="2011-04-08T12:39:56.850" Score="3" ViewCount="" Body="&lt;p&gt;Here is a simple idea which might work. As I've said in the comments if you have a grid with intensities why not fit density of bivariate distribution?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the sample graph to illustrate my point:&#xA;&lt;img src=&quot;http://i.imgur.com/YlzSK.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each grid point with is displayed as a square, colored according to intensity. Superimposed on the graph is the contour plot of bivariate normal density plot. As you can see the contour lines expand in the direction of decreasing intensity. The center will be controled by the mean of bivariate normal and the spread of the intensity according to covariance matrix.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To get the estimates of mean and covariance matrix simple numerical optimisation can be used, compare the intensities with values of density function using the mean and the covariance matrix as parameters. Minimize to get the estimates. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is of course strictly speaking not a statistical estimate, but at least it will give you an idea how to proceed further. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the code for reproducing the graph:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(mvtnorm)&#xA;sigma=cbind(c(0.1,0.7*0.1),c(0.7*0.1,0.1))&#xA;&#xA;x&amp;lt;-seq(0,1,by=0.01)&#xA;y&amp;lt;-seq(0,1,by=0.01)&#xA;z&amp;lt;-outer(x,y,function(x,y)dmvnorm(cbind(x,y),mean=mean,sigma=sigma))&#xA;&#xA;mz&amp;lt;-melt(z)&#xA;&#xA;mz$X1&amp;lt;-(mz$X1-1)/100&#xA;mz$X2&amp;lt;-(mz$X2-1)/100&#xA;&#xA;colnames(mz)&amp;lt;-c(&quot;x&quot;,&quot;y&quot;,&quot;z&quot;)&#xA;&#xA;mz$intensity&amp;lt;-round(mz$z*1000)&#xA;&#xA;ggplot(mz, aes(x,y)) + geom_tile(aes(fill = intensity), colour = &quot;white&quot;) + scale_fill_gradient(low = &quot;white&quot;,     high = &quot;steelblue&quot;)+geom_contour(aes(z=z),colour=&quot;black&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-04-08T12:39:56.850" />
  <row Id="9351" PostTypeId="2" ParentId="8784" CreationDate="2011-04-08T12:43:31.697" Score="3" ViewCount="" Body="&lt;p&gt;The short answer is &quot;yes you can&quot; - but you should compare the Maximum Likelihood Estimates (MLEs) of the &quot;big model&quot; with all co variates in either model fitted to both.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a &quot;quasi-formal&quot; way to get probability theory to answer your question&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the example, $Y_{1}$ and $Y_{2}$ are the same type of variables (fractions/percentages) so they are comparable.  I will assume that you fit the same model to both.  So we have two models:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$M_{1}:Y_{1i}\sim Bin(n_{1i},p_{1i})$$&#xA;$$log\left(\frac{p_{1i}}{1-p_{1i}}\right)=\alpha_{1}+\beta_{1}X_{i}$$&#xA;$$M_{2}:Y_{2i}\sim Bin(n_{2i},p_{2i})$$&#xA;$$log\left(\frac{p_{2i}}{1-p_{2i}}\right)=\alpha_{2}+\beta_{2}X_{i}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So you have the hypothesis you want to assess:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$H_{0}:\beta_{1}&amp;gt;\beta_{2}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And you have some data $\{Y_{1i},Y_{2i},X_{i}\}_{i=1}^{n}$, and some prior information (such as the use of logistic model). So you calculate the probability:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P=Pr(H_0|\{Y_{1i},Y_{2i},X_{i}\}_{i=1}^{n},I)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now $H_0$ doesn't depend on the actual value of any of the regression parameters, so they must have be removed by marginalising.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P=\int_{-\infty}^{\infty} \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} Pr(H_0,\alpha_{1},\alpha_{2},\beta_{1},\beta_{2}|\{Y_{1i},Y_{2i},X_{i}\}_{i=1}^{n},I) d\alpha_{1}d\alpha_{2}d\beta_{1}d\beta_{2}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The hypothesis simply restricts the range of integration, so we have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P=\int_{-\infty}^{\infty} \int_{\beta_{2}}^{\infty} \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} Pr(\alpha_{1},\alpha_{2},\beta_{1},\beta_{2}|\{Y_{1i},Y_{2i},X_{i}\}_{i=1}^{n},I) d\alpha_{1}d\alpha_{2}d\beta_{1}d\beta_{2}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because the probability is conditional on the data, it will factor into the two separate posteriors for each model&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Pr(\alpha_{1},\beta_{1}|\{Y_{1i},X_{i},Y_{2i}\}_{i=1}^{n},I)Pr(\alpha_{2},\beta_{2}|\{Y_{2i},X_{i},Y_{1i}\}_{i=1}^{n},I)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now because there is no direct links between $Y_{1i}$ and $\alpha_{2},\beta_{2}$, only indirect links through $X_{i}$, which is known, it will drop out of the conditioning in the second posterior. same for $Y_{2i}$ in the first posterior.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From standard logistic regression theory, and assuming uniform prior probabilities, the posterior for the parameters is approximately bi-variate normal with mean equal to the MLEs, and variance equal to the information matrix, denoted by $V_{1}$ and $V_{2}$ - which do not depend on the parameters, only the MLEs.  so you have straight-forward normal integrals with known variance matrix. $\alpha_{j}$ marginalises out with no contribution (as would any other &quot;common variable&quot;) and we are left with the usual result (I can post the details of the derivation if you want, but its pretty &quot;standard&quot; stuff):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P=\Phi\left(\frac{\hat{\beta}_{2,MLE}-\hat{\beta}_{1,MLE}}{\sqrt{V_{1:\beta,\beta}+V_{2:\beta,\beta}}}\right)&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $\Phi()$ is just the standard normal CDF.  This is the usual comparison of normal means test.  But note that this approach requires the use of the same set of regression variables in each.  In the multivariate case with many predictors, if you have different regression variables, the integrals will become effectively equal to the above test, but from the MLEs of the two betas from the &quot;big model&quot; which includes all covariates from both models.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-04-08T12:43:31.697" />
  <row Id="9352" PostTypeId="1" AcceptedAnswerId="9353" CreationDate="2011-04-08T13:12:09.120" Score="3" ViewCount="56" Body="&lt;p&gt;I'm playing with support vector machines (SVM) using the e1071::svm() function in R, and I encountered a scenario where I asked it for a leave-one-out cross-validated classification of a 2-category response and obtained a total accuracy of 38% (35/90), which, given 90 samples, ends up with a 95% confidence interval that is &lt;strong&gt;below&lt;/strong&gt; chance. Should I consider this a fluke, and if not, how is it possible for an SVM to become anti-predictive?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In case it matters, I used default values for the cost and gamma parameters, and the data predicting the response was a 8192 item vector representing 500 milliseconds of electroencephalogram data collected across 64 electrodes. &lt;/p&gt;&#xA;" OwnerUserId="364" LastActivityDate="2011-04-08T13:17:12.443" Title="Can anyone explain why I have obtained an anti-predictive Support Vector Machine?" Tags="&lt;cross-validation&gt;&lt;svm&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="9353" PostTypeId="2" ParentId="9352" CreationDate="2011-04-08T13:17:12.443" Score="5" ViewCount="" Body="&lt;p&gt;It is very probably the settings for the hyper-parameters that are the issue, leading to severe over-fitting of the data.  Without proper tuning of the hyper-parameters, and SVM can perform arbitrarily badly, especially for high dimensional data (it is the tuning of the regularisation parameter that gives robustness against over-fitting in high dimensional spaces).  I would suggest nested cross-validation, with the outer (leave-oue-out) cross-validation used form performance estimation and the hyper-parameters tuned independently in each fold by minimising a cross-validation based model selection criterion (I use Nelder-Mead simplex method rather than grid search).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The short answer, is never use default hyper-parameter values, always tune them afresh for each new (partition of the) dataset.&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2011-04-08T13:17:12.443" />
  <row Id="9354" PostTypeId="1" AcceptedAnswerId="9355" CreationDate="2011-04-08T13:29:59.237" Score="4" ViewCount="54" Body="&lt;p&gt;The wonderful libsvm package provides a python interface and a file &quot;easy.py&quot; that automatically searches for learning parameters (cost &amp;amp; gamma) that maximize the accuracy of the classifier. Within a given candidate set of learning parameters, accuracy is operationalized by cross-validation, but I feel like this undermines the purpose of cross-validation. That is, insofar as the learning parameters themselves can be chosen an a manner that might cause an over-fit of the data, I feel like a more appropriate approach would be to apply cross validation at the level of the search itself: perform the search on a training data set and then evaluate the ultimate accuracy of SVM resulting from the finally-chosen learning parameters by evaluation within a separate testing data set. Or am I missing something here?&lt;/p&gt;&#xA;" OwnerUserId="364" LastActivityDate="2011-04-08T13:40:43.197" Title="How does one appropriately apply cross-validation in the context of selecting learning parameters for support vector machines?" Tags="&lt;cross-validation&gt;&lt;svm&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="9355" PostTypeId="2" ParentId="9354" CreationDate="2011-04-08T13:40:43.197" Score="5" ViewCount="" Body="&lt;p&gt;If you learn the hyper-parameters in the full training data and then cross-validate, you will get an optimistically biased performance estimate, because the test data in each fold will already have been used in setting the hyper-parameters, so the hyper-parameters selected are selected in part because they suit the data in the test set.  The optimistic bias introduced in this way can be unexpectedly large.  See &lt;a href=&quot;http://jmlr.csail.mit.edu/papers/v11/cawley10a.html&quot; rel=&quot;nofollow&quot;&gt;Cawley and Talbot, &quot;On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation&quot;, JMLR 11(Jul):2079−2107, 2010.&lt;/a&gt; (Particularly section 5.3).  The best thing to do is nested cross-validation.  The basic idea is that you cross-validate the entire method used to generate the model, so treat model selection (choosing the hyper-parameters) as simply part of the model fitting procedure (where the parameters are determined) and you can't go too far wrong. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you use cross-validation on the training set to determine the hyper-parameters and then evaluate the performance of a model trained using those parameters on the whole training set, using a separate test set, that is also fine (provided you have enough data for reliably fitting the model and estimating performance using disjoint partitions).&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2011-04-08T13:40:43.197" />
  <row Id="9356" PostTypeId="2" ParentId="9324" CreationDate="2011-04-08T14:43:53.077" Score="3" ViewCount="" Body="&lt;p&gt;Firstly, you need to set &lt;code&gt;REML=F&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;lmerfit1&amp;lt;-lmer(size ~ (1|FEMALE), data=Data, REML=F)&#xA;lmerfit2&amp;lt;-lmer(size ~ SITE+(1|FEMALE), data=Data, REML=F)&#xA;anova(lmerfit1, lmerfit2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This will use MLE instead of REML, which is necessary because likelihoods from mixed models with different fixed effects are not comparable when REML is used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Secondly, you could do the following quick checks:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;summary(lmerfit2) # To see the size of the SITE coefficient&#xA;summary(lm(size ~ SITE, data=Data)) # To check the fixed effects estimates&#xA;plot(size ~ SITE, data=Data) # Box plot&#xA;dotplot(size ~ SITE, data=Data) # Another visual check&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But given the non-significance of &lt;code&gt;SITE&lt;/code&gt; in your reported test, and the lack of visual difference you reported in your comment, I'm guessing there is no significant main effect of &lt;code&gt;SITE&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="3432" LastActivityDate="2011-04-08T14:43:53.077" />
  <row Id="9357" PostTypeId="1" AcceptedAnswerId="9364" CreationDate="2011-04-08T14:45:04.197" Score="5" ViewCount="145" Body="&lt;p&gt;When you are trying to fit models to a large dataset, the common advice is to partition the data into three parts: the training, validation, and test dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is because the models usually have three &quot;levels&quot; of parameters: the first &quot;parameter&quot; is the model class (e.g. SVM, neural network, random forest), the second set of parameters are the &quot;regularization&quot; parameters or &quot;hyperparameters&quot; (e.g. lasso penalty coefficient, choice of kernel, neural network structure) and the third set are what are usually considered the &quot;parameters&quot; (e.g. coefficients for the covariates.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given a model class and a choice of hyperparameters, one selects the parameters by choosing the parameters which minimize error on the training set.  Given a model class, one tunes the hyperparameters by minimizing error on the validation set.  One selects the model class by performance on the test set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But why not more partitions?  Often one can split the hyperparameters into two groups, and use a &quot;validation 1&quot; to fit the first and &quot;validation 2&quot; to fit the second.  Or one could even treat the size of the training data/validation data split as a hyperparameter to be tuned.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this already a common practice in some applications?  Is there any theoretical work on the optimal partitioning of data?&lt;/p&gt;&#xA;" OwnerUserId="3567" LastActivityDate="2011-04-08T17:58:07.700" Title="Why only three partitions? (training, validation, test)" Tags="&lt;model-selection&gt;&lt;data-mining&gt;" AnswerCount="1" FavoriteCount="2" />
  <row Id="9358" PostTypeId="1" AcceptedAnswerId="9361" CreationDate="2011-04-08T15:01:30.430" Score="5" ViewCount="208" Body="&lt;p&gt;I have four numeric variables. All of them are measures of soil quality. Higher the variable, higher the quality. The range for all of them is different:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Var1 from 1 to 10&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Var2 from 1000 to 2000&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Var3 from 150 to 300&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Var4 from 0 to 5&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to combine four variables into single soil quality score which will successfully rank order.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My idea is very simple. Standardize all four variables, sum them up and whatever you get is the score which should rank-order. Do you see any problem with applying this approach. Is there any other (better) approach that you would recommend?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks guys. A lot of discussion went into &quot;domain expertise&quot;... Agriculture stuff... Whereas I expected more stats-talk. In terms of technique that I will be using... It will probably be simple z-score summation + logistic regression as an experiment. Because vast majority of samples has poor quality 90% I'm going to combine 3 quality categories into one and basically have binary problem (somequality vs no-quality). I kill two birds with one stone. I increase my sample in terms of event rate and I make a use of experts by getting them to clasify my samples. Expert classified samples will then be used to fit log-reg model to maximize level of concordance / discordance with the experts.... How does that sound to you?&lt;/p&gt;&#xA;" OwnerUserId="333" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-12T04:41:21.763" LastActivityDate="2011-04-12T04:41:21.763" Title="Creating an index of quality from multiple variables to enable rank ordering" Tags="&lt;ranking&gt;" AnswerCount="3" FavoriteCount="2" />
  <row Id="9359" PostTypeId="2" ParentId="9345" CreationDate="2011-04-08T15:24:16.877" Score="3" ViewCount="" Body="&lt;p&gt;It is difficult to understand quite what you are saying here and it would be helpful to have an illustration.   Take for example the values $x_1$, $x_2$ and $y$ here&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x_1  x_2   y&#xA;---  ---  ---&#xA; 1  1     8&#xA;-1  1    -1&#xA;-3  0   -15&#xA; 2  0     9&#xA; 0  0    -1&#xA; 2  0    10&#xA;-1  0    -6&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;OLS regression would give $\hat{y} = 4.9 x_1 + 4.1 x_2 - 0.6$.  I don't see how the $+4.1$ can be said to be implausible.  What is true is that it is a more uncertain estimate than the estimate of $+4.9$ since there is less information about the impact of variation of $x_2$ than about the impact of variation of $x_1$.  [When I generated these numbers, I was aiming for $5 x_1 + 3 x_2$.] &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is not the same as saying $x_2$ has too many zeros.  Let $x_3 = 1 - x_2$, so $x_3$ is zero only twice, rather than five times.  Then OLS regression of $y$ on $x_1$ and $x_3$ would give $\hat{y} = 4.9 x_1 - 4.1 x_3 +3.5$, much as you might expect.  But the new coefficient of $-4.1$ is as good (in any sense) as the earlier $+4.1$ even though there are fewer zeros.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On a different issue, if you are using a prior distribution which is not scale-free and you have other prior information that suggests to you that the dispersion of plausible values is going to be narrower or wider than suggested by your distribution then you can (indeed you should) adjust your prior distribution accordingly.    &lt;/p&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-04-08T15:24:16.877" />
  <row Id="9360" PostTypeId="2" ParentId="9330" CreationDate="2011-04-08T15:35:04.183" Score="0" ViewCount="" Body="&lt;p&gt;You could try exact logistic regression with proc logistic, but then you cannot specify random effects somewhere in your model which you probably do now as you're using proc mixed. You'd have to switch to fixed effects but you could keep the binomial error distribution.&lt;/p&gt;&#xA;" OwnerUserId="1573" LastActivityDate="2011-04-08T15:35:04.183" />
  <row Id="9361" PostTypeId="2" ParentId="9358" CreationDate="2011-04-08T15:42:31.763" Score="8" ViewCount="" Body="&lt;p&gt;The proposed approach may give a reasonable result, but only by accident.  At this distance--that is, taking the question at face value, with the meanings of the variables disguised--some problems are apparent:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;It is not even evident that each variable is positively related to &quot;quality.&quot;&lt;/strong&gt;  For example, what if a 10 for 'Var1' means the &quot;quality&quot; is &lt;em&gt;worse&lt;/em&gt; than the quality when Var1 is 1?  Then adding it to the sum is about as wrong a thing as one can do; it needs to be &lt;em&gt;subtracted.&lt;/em&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Standardization implies that &quot;quality&quot; depends on the data set itself.  Thus the definition will change with different data sets or with additions and deletions to these data.  &lt;strong&gt;This can make the &quot;quality&quot; into an arbitrary, transient, non-objective construct&lt;/strong&gt; and preclude comparisons between datasets.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;There is no definition of &quot;quality&quot;.&lt;/strong&gt;  What is it supposed to mean?  Ability to block migration of contaminated water?  Ability to support organic processes?  Ability to promote certain chemical reactions?  Soils good for one of these purposes may be especially poor for others.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;The problem as stated has no purpose:&lt;/strong&gt; &lt;em&gt;why&lt;/em&gt; does &quot;quality&quot; need to be ranked?  What will the ranking be used for--input to more analysis, selecting the &quot;best&quot; soil, deciding a scientific hypothesis, developing a theory, promoting a product?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;The consequences of the ranking are not apparent.&lt;/strong&gt;  If the ranking is incorrect or inferior, what will happen?  Will the world be hungrier, the environment more contaminated, scientists more misled, gardeners more disappointed?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Why should a linear combination of variables be appropriate?&lt;/strong&gt;  Why shouldn't they be multiplied or exponentiated or combined as a &lt;a href=&quot;http://en.wikipedia.org/wiki/Posynomial&quot; rel=&quot;nofollow&quot;&gt;posynomial&lt;/a&gt; or something even more esoteric?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Raw soil quality measures are commonly re-expressed.  For example, log permeability is usually more useful than the permeability itself and log hydrogen ion activity (pH) is much more useful than the activity.  &lt;strong&gt;What are the appropriate re-expressions of the variables for determining &quot;quality&quot;?&lt;/strong&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;One would hope that soils science would answer most of these questions and indicate what the appropriate combination of the variables might be for any objective sense of &quot;quality.&quot;  If not, then you face a &lt;a href=&quot;http://en.wikipedia.org/wiki/Multi-criteria_decision_analysis&quot; rel=&quot;nofollow&quot;&gt;multi-attribute valuation problem&lt;/a&gt;.  The Wikipedia article lists dozens of methods for addressing this.  IMHO, most of them are inappropriate for addressing a scientific question.  One of the few with a solid theory and potential applicability to empirical matters is &lt;a href=&quot;http://books.google.com/books?hl=en&amp;amp;lr=&amp;amp;id=GPE6ZAqGrnoC&amp;amp;oi=fnd&amp;amp;pg=PR11&amp;amp;dq=Keeney%20&amp;amp;%20Raiffa%20multiple%20attribute%20valuation%20theory&amp;amp;ots=EkCGJz7pwx&amp;amp;sig=LVkWUJRADGezaSKENx0-giUehcs#v=onepage&amp;amp;q&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;Keeney &amp;amp; Raiffa's multiple attribute valuation theory&lt;/a&gt; (MVAT).  It requires you to be able to determine, for any two specific combinations of the variables, which of the two should rank higher.  A structured sequence of such comparisons reveals (a) appropriate ways to re-express the values; (b) whether or not a linear combination of the re-expressed values will produce the correct ranking; and (c) if a linear combination is possible, it will let you compute the coefficients.  In short, MVAT provides algorithms for solving your problem provided you already know how to compare specific cases.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-04-08T15:42:31.763" CommentCount="11" />
  <row Id="9362" PostTypeId="2" ParentId="9358" CreationDate="2011-04-08T16:22:39.233" Score="0" ViewCount="" Body="&lt;p&gt;One other thing you did not discuss is the scale of the measurements.  V1 and V5 looks like they are of rank order and the other seem not. So standardization may be skewing the score. So you may be better transforming all of the variables into ranks, and determining a weighting for each variable, since it is highly unlikely that they have the same weight.  Equal weighting is more of a &quot;no nothing&quot; default. You might want to do some correlation or regression analysis to come up with some a priori weights.&lt;/p&gt;&#xA;" OwnerUserId="3489" LastActivityDate="2011-04-08T16:22:39.233" CommentCount="2" />
  <row Id="9363" PostTypeId="2" ParentId="9358" CreationDate="2011-04-08T17:43:33.113" Score="-3" ViewCount="" Body="&lt;p&gt;Following up on Ralph Winters' answer, you might use PCA (principal component analysis) on the matrix of suitably standardized scores. This will give you a &quot;natural&quot; weight vector that you can use to combine future scores. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do this also after all scores have been transformed into ranks. If the results are very similar, you have good reasons to continue with either method. If there are discrepancies, this will lead to interesting questions and a better understanding.  &lt;/p&gt;&#xA;" OwnerUserId="4062" LastActivityDate="2011-04-08T17:43:33.113" CommentCount="5" />
  <row Id="9364" PostTypeId="2" ParentId="9357" CreationDate="2011-04-08T17:58:07.700" Score="2" ViewCount="" Body="&lt;p&gt;First, I think you're mistaken about what the three partitions do. You don't make any choices based on the test data. Your algorithms adjust their parameters based on the training data. You then run them on the validation data to compare your algorithms (and their trained parameters) and decide on a winner. You then run the winner on your test data to give you a forecast of how well it will do in the real world.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You don't validate on the training data because that would overfit your models. You don't stop at the validation step's winner's score because you've iteratively been adjusting things to get a winner in the validation step, and so you need an independent test (that you haven't specifically been adjusting towards) to give you an idea of how well you'll do outside of the current arena.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second, I would think that one limiting factor here is how much data you have. Most of the time, we don't even want to split the data into fixed partitions at all, hence CV.&lt;/p&gt;&#xA;" OwnerUserId="1764" LastActivityDate="2011-04-08T17:58:07.700" CommentCount="2" />
  <row Id="9365" PostTypeId="1" CreationDate="2011-04-08T19:01:11.850" Score="11" ViewCount="326" Body="&lt;p&gt;What are some good papers describing &lt;em&gt;applications&lt;/em&gt; of statistics that would be fun and informative to read? Just to be clear, I'm not really looking for papers describing new statistical methods (e.g., a paper on least angle regression), but rather papers describing how to solve real-world problems.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, one paper that would fit what I'm looking is the climate paper from the &lt;a href=&quot;http://meta.stats.stackexchange.com/questions/685/second-cross-validated-journal-club&quot;&gt;second Cross-Validated Journal Club&lt;/a&gt;. I'm kind of looking for more statistics-ish papers, rather than machine learning papers, but I guess it's kind of a fuzzy distinction (I'd classify the Netflix Prize papers as a bit borderline, and a paper on sentiment analysis as something I'm &lt;em&gt;not&lt;/em&gt; looking for).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm asking because most of the applications of statistics I've seen are either the little snippets you seen in textbooks, or things related to my own work, so I'd like to branch out a bit.&lt;/p&gt;&#xA;" OwnerUserId="1106" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-09T18:54:46.130" LastActivityDate="2011-04-13T06:59:09.850" CommunityOwnedDate="2011-04-08T19:04:21.230" Title="What are some interesting and well-written applied statistics papers?" Tags="&lt;big-list&gt;&lt;application&gt;" AnswerCount="5" CommentCount="2" FavoriteCount="5" />
  <row Id="9366" PostTypeId="2" ParentId="9313" CreationDate="2011-04-08T19:16:32.820" Score="2" ViewCount="" Body="&lt;p&gt;For an overview of some matching algorithms as well as clear examples of applications in everything from education to medical experiments, I would suggest:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Paul R. Rosenbaum (2010). &lt;em&gt;Design of Observational Studies&lt;/em&gt;. Springer.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Rosenbaum's earlier book provides a more technical review, though because matching is such a hot topic at the moment, it may not cover the most current techniques:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Paul R. Rosenbaum (2002). &lt;em&gt;Observational Studies&lt;/em&gt;, Second Edition. Springer.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Even if Rosenbaum doesn't hit on a particular topic of interest, his chapter bibliographies are an excellent resource (particularly those in &lt;em&gt;Design&lt;/em&gt;). He has also done some very valuable work on matching sensitivity analyses, which are covered extensively in these books.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, you would probably also be served by going directly to the source (I haven't read this myself):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Donald B. Rubin (2006). &lt;em&gt;Matched Sampling for Causal Effects&lt;/em&gt;. Cambridge University Press.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;As mentioned above, matching is something of a hot topic. So, generally, I would look through the citations of more recent books and articles. Besides work by Donald Rubin and Paul Rosenbaum, I would look for work by Alberto Abadie and Guido Imbens (both at Harvard) and James Heckman (Chicago), probably in that order. Of course, depending on your particular research interests, others may be equally as important.&lt;/p&gt;&#xA;" OwnerUserId="3265" LastActivityDate="2011-04-08T19:16:32.820" />
  <row Id="9367" PostTypeId="2" ParentId="9331" CreationDate="2011-04-08T20:21:18.013" Score="0" ViewCount="" Body="&lt;p&gt;I learnt about the use of splines in regression from the book &quot;Regression Modeling Strategies&quot; by Frank Harrell. Harrell's R package rms allows you to easily fit regression models in which some predictor variables are represented as splines.&lt;/p&gt;&#xA;" OwnerUserId="3835" LastActivityDate="2011-04-08T20:21:18.013" />
  <row Id="9368" PostTypeId="2" ParentId="9312" CreationDate="2011-04-08T20:40:25.080" Score="1" ViewCount="" Body="&lt;p&gt;You should use the point estimate of the reliability, not the lower bound or whatsoever. I guess by lb/up you mean the 95% CI for the ICC (I don't have SPSS, so I cannot check myself)? It's unfortunate that we also talk of Cronbach's alpha as a &quot;lower bound for reliability&quot; since this might have confused you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It should be noted that this formula is not restricted to the use of an estimate of ICC; in fact, you can plug in any &quot;valid&quot; measure of reliability (most of the times, it is Cronbach's alpha that is being used). Apart from the NCME tutorial that I linked to in my comment, you might be interested in this recent article:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Tighe et al. &lt;a href=&quot;http://www.biomedcentral.com/1472-6920/10/40&quot; rel=&quot;nofollow&quot;&gt;The standard error of&#xA;  measurement is a more appropriate&#xA;  measure of quality for postgraduate&#xA;  medical assessments than is&#xA;  reliability: an analysis of MRCP(UK)&#xA;  examinations&lt;/a&gt;. &lt;em&gt;BMC Medical&#xA;  Education&lt;/em&gt; 2010, 10:40&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Although it might seem to barely address your question at first sight, it has some additional material showing how to compute SEM (here with Cronbach's $\alpha$, but it is straightforward to adapt it with ICC); and, anyway, it's always interesting to look around to see how people use SEM.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-04-08T20:40:25.080" />
  <row Id="9369" PostTypeId="2" ParentId="9365" CreationDate="2011-04-08T21:02:38.377" Score="7" ViewCount="" Body="&lt;p&gt;It's a bit difficult for me to see what paper might be of interest to you, so let me try and suggest the following ones, from the psychometric literature:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Borsboom, D. (2006). &lt;a href=&quot;http://sites.google.com/site/borsboomdenny/BorsboomPM2006.pdf?attredirects=0&quot; rel=&quot;nofollow&quot;&gt;The attack of&#xA;  the psychometricians&lt;/a&gt;.&#xA;  &lt;em&gt;Psychometrika&lt;/em&gt;, &lt;em&gt;71&lt;/em&gt;, 425-440.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;for dressing the scene (Why do we need to use statistical models that better reflect the underlying hypotheses commonly found in psychological research?), and &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Borsboom, D. (2008). &lt;a href=&quot;http://sites.google.com/site/borsboomdenny/PsychometricPerspectivesandDiagnosti.pdf?attredirects=0&quot; rel=&quot;nofollow&quot;&gt;Psychometric&#xA;  perspectives on diagnostic&#xA;  systems&lt;/a&gt;. &lt;em&gt;Journal of Clinical&#xA;  Psychology&lt;/em&gt;, &lt;em&gt;64&lt;/em&gt;, 1089-1108.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;for an applied perspective on diagnostic medicine (transition from yes/no assessment as used in the DSM-IV to the &quot;dimensional&quot; approach intended for the DSM-V). A larger review of latent variable models in biomedical research that I like is:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Rabe-Hesketh, S. and Skrondal, A.&#xA;  (2008). &lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pubmed/17855748&quot; rel=&quot;nofollow&quot;&gt;Classical latent variable&#xA;  models for medical research&lt;/a&gt;.&#xA;  &lt;em&gt;Statistical Methods in Medical Research&lt;/em&gt;, &lt;em&gt;17(1)&lt;/em&gt;, 5-32.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-04-08T21:02:38.377" CommunityOwnedDate="2011-04-08T21:02:38.377" CommentCount="2" />
  <row Id="9370" PostTypeId="2" ParentId="9365" CreationDate="2011-04-08T21:33:23.330" Score="7" ViewCount="" Body="&lt;p&gt;Here are five highly-cited papers from the last 40 years of the &lt;a href=&quot;http://www.blackwellpublishing.com/journal.asp?ref=0035-9254&quot; rel=&quot;nofollow&quot; rel=&quot;nofollow&quot;&gt;&lt;em&gt;Journal of the Royal Statistical Society, Series C: Applied Statistics&lt;/em&gt;&lt;/a&gt; with a clear application in the title that caught my eye while scanning through the Web of Knowledge search results:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Sheila M. Gore, Stuart J. Pocock and Gillian R. Kerr (1984). &lt;a href=&quot;http://www.jstor.org/stable/2347444&quot; rel=&quot;nofollow&quot; rel=&quot;nofollow&quot;&gt;Regression Models and Non-Proportional Hazards in the Analysis of Breast Cancer Survival.&lt;/a&gt; Vol. 33, No. 2, pp. 176-195. (Cited 100 times) (&lt;a href=&quot;http://www.charite.de/biometrie/dwl/Gore.pdf&quot; rel=&quot;nofollow&quot; rel=&quot;nofollow&quot;&gt;Free PDF&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;li&gt;John Haslett and Adrian E. Raftery (1989). &lt;a href=&quot;http://www.jstor.org/stable/2347679&quot; rel=&quot;nofollow&quot; rel=&quot;nofollow&quot;&gt;Space-Time Modelling with Long-Memory Dependence: Assessing Ireland's Wind Power Resource.&lt;/a&gt; Vol. 38, No. 1 pp. 1-50 (Cited 156 times)&lt;/li&gt;&#xA;&lt;li&gt;Stuart G. Coles and Jonathan A. Tawn (1994). &lt;a href=&quot;http://www.jstor.org/stable/2986112&quot; rel=&quot;nofollow&quot; rel=&quot;nofollow&quot;&gt;Statistical Methods for Multivariate Extremes: An Application to Structural Design.&lt;/a&gt; Vol. 43, No. 1, pp. 1-48. (Cited 99 times)&lt;/li&gt;&#xA;&lt;li&gt;Nicholas Lange and Scott L. Zeger (1997). &lt;a href=&quot;http://www.jstor.org/stable/2986204&quot; rel=&quot;nofollow&quot; rel=&quot;nofollow&quot;&gt;Non-linear Fourier time series analysis for human brain mapping by functional magnetic resonance imaging.&lt;/a&gt; Vol. 46, No. 1, pp. 1-29. (Cited 94 times)&lt;/li&gt;&#xA;&lt;li&gt;James P. Hughes, Peter Guttorp and Stephen P. Charles (1999). &lt;a href=&quot;http://www.jstor.org/stable/2680815&quot; rel=&quot;nofollow&quot; rel=&quot;nofollow&quot;&gt;A Non-Homogeneous Hidden Markov Model for Precipitation Occurrence.&lt;/a&gt; Vol. 48, No. 1, pp. 15-30. (Cited 103 times)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="449" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-13T06:57:43.827" LastActivityDate="2011-04-13T06:57:43.827" CommunityOwnedDate="2011-04-08T21:33:23.330" />
  <row Id="9371" PostTypeId="2" ParentId="9329" CreationDate="2011-04-08T23:21:36.193" Score="2" ViewCount="" Body="&lt;p&gt;First idea is just to mimic the knock-out strategy from variable importance and just test how mixing each attribute will degenerate the forest confidence in object classification (on OOB and with some repetitions obviously). This requires some coding, but is certainly achievable. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I feel it is just a bad idea -- the result will be probably variable like hell (without stabilizing impact of averaging over objects), noisy (for not-so-confident objects the nonsense attributes could have big impacts) and hard to interpret (two or more attribute cooperative rules will probably result in random impacts of each contributing attributes). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Not to leave you with negative answer, I would rather try to look at the proximity matrix and the possible archetypes it may reveal -- this seems much more stable and straightforward.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2011-04-08T23:21:36.193" />
  <row Id="9372" PostTypeId="1" CreationDate="2011-04-08T23:28:23.310" Score="2" ViewCount="48" Body="&lt;p&gt;How would I go about finding out the confidence intervals around a set of distinct bianary occurrences where each occurence has a different associated probability and each occurence is weighted?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To be more specific, we typically determine a milestone budget by assigning reasonable probabilities to each milestone.  Each milestone typically usually has a differnt probability of success and a different dollar value.  For example if we need to determine the amount of money we need to hold out for 3 milestones, we would do this:&#xA;$$P(\text{milestone}_1) \text{cost}(\text{milestone}_1)+P(\text{milestone}_2) \text{cost}(\text{milestone}_2)+\dots $$&#xA;So it could look something like this:  Budget = (10% x $\$$1,000,000) + (50% x $\$$5,000,000) + (20% x $\$$20,000,000) = $\$$6,600,000.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each milestone is bianary in that it either happens or it doesn't.  so for the first milestone, we are saying that there is a 10% chance that we will need to pay $\$$1M dollars and a 90% chance that we will pay nothing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Obviously, we won't ever actually be paying the probilized amount of $\$$6.6M.  How would I go about determining the probability range around where the spend is most likely to fall.  In other words, if I wanted to demonstrate where the actual spend might fall with a certainty of 80% how would I go about calculating this?  Note that we have more events in my real world problem...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance for any help.  I am not a statistician at all so please forgive my ignorance and any lack of clarity.&lt;/p&gt;&#xA;" OwnerUserId="4085" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-09T00:27:18.753" LastActivityDate="2011-04-09T00:41:18.617" Title="How do I determine confidence intervals around weighted, probilized events?" Tags="&lt;probability&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="9373" PostTypeId="2" ParentId="9276" CreationDate="2011-04-09T00:04:02.477" Score="1" ViewCount="" Body="&lt;p&gt;There is one general and &quot;in-universe&quot; criterion for goodness of Monte Carlo -- convergence. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Stick to one M and check how the PG behaves with the number of juries -- it should converge, so will show you a number of repetitions for which you will have a reasonable (for your application) number of significant digits. Repeat this benchmark for few other Ms to be sure you wasn't lucky with M selection, then proceed to the whole simulation.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2011-04-09T00:04:02.477" CommentCount="1" />
  <row Id="9374" PostTypeId="2" ParentId="9372" CreationDate="2011-04-09T00:26:56.037" Score="2" ViewCount="" Body="&lt;p&gt;Basically you would need to make a &lt;a href=&quot;http://www.google.com/search?q=probability+tree&quot; rel=&quot;nofollow&quot;&gt;probability tree&lt;/a&gt; with resulting penalty sums in leafs and sum the whole thing up for a criterion of your choice, possibly using some software for more milestones than few (there will be $2^N$ leafs for $N$ milestones). You will be able to build the distribution too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a really huge number of milestones, you'd need a Monte Carlo simulation, so basically just simulate the process huge number of times (using random number generator to decide whether to fire or not certain milestone) and then obtain a histogram of outputs as a distribution approximation.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2011-04-09T00:26:56.037" CommentCount="1" />
  <row Id="9375" PostTypeId="2" ParentId="9372" CreationDate="2011-04-09T00:41:18.617" Score="1" ViewCount="" Body="&lt;p&gt;It rather depends on how many milestones you have.  But if this (call it $n$) is small enough and each one either happens or does not, then you can work out the $2^n$ possibilities, working out the probabilities by multiplying the inidividual probabilies.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So for example the probability of paying out $20,000,000$ is $0.9 \times 0.5 \times 0.2 = 0.09$ or 9%. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You then sort by value and add up the probabilities, to find for example that the probability of paying $20,000,000$ &lt;em&gt;or less&lt;/em&gt; is $0.89$; adding up in reverse you find the probability of paying $20,000,000$ &lt;em&gt;or more&lt;/em&gt; is $0.20$.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your table might look like this  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Prob    CumProb RevCum  Amount&#xA;&#xA;0.36    0.36    1.00            0&#xA;0.04    0.40    0.64    1,000,000&#xA;0.36    0.76    0.60    5,000,000&#xA;0.04    0.80    0.24    6,000,000&#xA;0.09    0.89    0.20   20,000,000&#xA;0.01    0.90    0.11   21,000,000&#xA;0.09    0.99    0.10   25,000,000&#xA;0.01    1.00    0.01   26,000,000&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This will then let you make statements about confidence.&lt;/p&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-04-09T00:41:18.617" CommentCount="1" />
  <row Id="9376" PostTypeId="2" ParentId="8669" CreationDate="2011-04-09T07:49:30.520" Score="0" ViewCount="" Body="&lt;p&gt;Canonical Correlation Analysis was one way to go and it works! Credits to @schenectady for this. Thanks a lot for your help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to write this for future reference and for others who might have a similar query, if you want to perform a regression analysis in such a situation, you should attempt to minimize the squared errors from each of your equations, i.e.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\left\{\begin{array}{l} a_0 + a_1X_1 + a_2X_2 + \dots + a_{11}X_{11}\\ b_0 + b_1Y_1 + b_2Y_2 + b_3Y_3 + b_4Y_4\end{array}\right.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The simplest way is to use the excel solver to arrive at the coefficients with the aim of minimizing the mean of the squared errors from the above two equations.&lt;/p&gt;&#xA;" OwnerUserId="3859" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-09T18:43:50.403" LastActivityDate="2011-04-09T18:43:50.403" />
  <row Id="9377" PostTypeId="1" AcceptedAnswerId="9380" CreationDate="2011-04-09T11:29:27.753" Score="6" ViewCount="93" Body="&lt;p&gt;In the paper &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;M. Avellaneda and J. H. Lee, &lt;a href=&quot;http://www.math.nyu.edu/faculty/avellane/AvellanedaLeeStatArb071108.pdf&quot; rel=&quot;nofollow&quot;&gt;Statistical arbitrage in the U.S. equities market&lt;/a&gt;, July 2008,&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;in the Appendix on page 46, how does he get equilibrium standard deviation as following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\sigma_{eq} = \sqrt{\frac{\text{Variance}(\zeta)}{1 − b^2}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If anyone knows the paper, please explain.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Much appreciated,&lt;/p&gt;&#xA;" OwnerUserId="862" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-04-19T13:04:46.690" LastActivityDate="2011-04-19T13:04:46.690" Title="Origin of strange formula for equilibrium standard deviation" Tags="&lt;regression&gt;&lt;probability&gt;&lt;variance&gt;&lt;stochastic-processes&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="2" />
  <row Id="9378" PostTypeId="1" AcceptedAnswerId="9401" CreationDate="2011-04-09T12:49:36.187" Score="1" ViewCount="140" Body="&lt;p&gt;I'm working with a CSV which contains approximately 220,000 entries. My aim is to predict one of the attributes (ATT1) using the other 3 (ATT2, ATT3, ATT4).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've been able to do this using NaiveBayes, but now I feel unsatisfied with the result. The reason is that ATT1 can be one of 6 values (VAL1-6), but these are not evenly distributed into the dataset. I'm afraid this could lead to an unprecise prediction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I select a given number of entries for each value of ATT1 from within RapidMiner?&lt;/p&gt;&#xA;" OwnerUserId="1522" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-09T13:09:54.470" LastActivityDate="2011-04-10T01:21:50.127" Title="How to choose a data subset in RapidMiner?" Tags="&lt;dataset&gt;&lt;csv-file&gt;&lt;rapidminer&gt;" AnswerCount="1" CommentCount="8" FavoriteCount="1" />
  <row Id="9380" PostTypeId="2" ParentId="9377" CreationDate="2011-04-09T13:32:30.010" Score="11" ViewCount="" Body="&lt;p&gt;The authors are providing a simple means for estimating the parameters of a mean-reverting Orstein-Uhlenbeck process via a regression on returns at discretized points in time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The model they are considering has a representation as a stochastic differential equation of the form [pg. 16, Eq. (12)]&#xA;$$&#xD;&#xA;\newcommand{\rd}{\mathrm{d}}&#xD;&#xA;\rd X(t) = \kappa (m - X(t)) \rd t + \sigma \rd W(t)&#xD;&#xA;$$&#xA;where $W(t)$ is a standard Brownian motion.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The solution to this SDE is well-known and easy to find via Ito's lemma and an analogous technique to integrating constants in ODEs. The solution is [pg. 17, Eq. (13)]&#xA;$$&#xD;&#xA;X(t_0 + \Delta t) = e^{-\kappa \Delta t} X(t_0) + (1-e^{-\kappa \Delta t}) m + \sigma \int_{t_0}^{\,t_0 + \Delta t} e^{-\kappa(t_0+\Delta t - s)} \, \rd W(s) .&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a Gaussian process and so is characterized by its mean and covariance as a function of time. Letting &quot;time go to infinity&quot; (i.e., $\Delta t \to \infty$), we get an equilibrium mean and variance of&#xA;$$&#xD;&#xA;\begin{aligned}&#xD;&#xA;\mathbb{E} X(t) &amp;amp;= m \\&#xD;&#xA;\mathbb{V}\mathrm{ar}(X(t)) &amp;amp;= \frac{\sigma^2}{2 \kappa}&#xD;&#xA;\end{aligned}&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, skipping to the appendix [bottom of page 45], the authors are trying to estimate the parameters by doing a regression using the discrete values of the process and model&#xA;$$&#xD;&#xA;X_{n+1} = a + b X_n + \zeta_{n+1} .&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Matching up the parameters $a$ and $b$ with the portions from above, we get that&#xA;$$&#xD;&#xA;\begin{aligned}&#xD;&#xA;a &amp;amp;= m (1 - e^{-\kappa \Delta t}) \\&#xD;&#xA;b &amp;amp;= e^{-\kappa \Delta t} \\&#xD;&#xA;\mathbb{V}\mathrm{ar}(\zeta) &amp;amp;= \sigma^2 \frac{1-e^{-2\kappa \Delta t}}{2\kappa}&#xD;&#xA;\end{aligned}&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Substituting the second equation into the first and solving for $m$ gives $m = a / (1-b)$. Use the same substitution in the third equation and rearrange to get&#xA;$$&#xD;&#xA;\sigma^2 = \frac{\mathbb{V}\mathrm{ar}(\zeta) \cdot 2 \kappa}{1 - b^2} \&amp;gt;,&#xD;&#xA;$$&#xA;but, recall that the variance of the equilibrium distribution (by looking far into the future) for $X(t)$ is just $\sigma^2 / 2 \kappa$ and so this gives your result.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Addendum&lt;/strong&gt;: If you're wondering how the expression&#xA;$$&#xD;&#xA;\mathbb{V}\mathrm{ar}(\zeta) = \sigma^2 \frac{1-e^{-2\kappa \Delta t}}{2\kappa}&#xD;&#xA;$$&#xA;was obtained, it is via the (remarkable and beautiful!) &lt;a href=&quot;http://en.wikipedia.org/wiki/Ito_isometry&quot; rel=&quot;nofollow&quot; rel=&quot;nofollow&quot;&gt;Ito isometry&lt;/a&gt; and the fact that an Ito integral is a zero-mean martingale; namely, in this instance,&#xA;$$&#xD;&#xA;\mathbb{E}\Big(\sigma \int_{t_0}^{\,t_0 + \Delta t} e^{-\kappa(t_0+\Delta t - s)} \, \rd W(s)\Big)^2 = \sigma^2 \int_{t_0}^{\,t_0 + \Delta t} e^{-2 \kappa(t_0 +\Delta t - s)} \, \rd s&#xD;&#xA;$$&#xA;where we note that the integrand has been squared on the right-hand side and we &quot;get to replace&quot; $\rd W(s)$ with $\rd s$, converting the problem into one of solving a standard Riemann integral.&lt;/p&gt;&#xA;" OwnerUserId="2970" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-04-11T02:17:49.223" LastActivityDate="2011-04-11T02:17:49.223" CommentCount="3" />
  <row Id="9381" PostTypeId="1" CreationDate="2011-04-09T14:00:11.930" Score="1" ViewCount="45" Body="&lt;p&gt;I was wondering from the view of dividing the topics of statistical theory into inference part and non-inference part, what inference topics and non-inference topics statistical theory is covering?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By inference, I mean the task in logic to reach some conclusion from some premises. Probabilistic inference is a way of logical inference using probability.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can we say statistical theory is just providing probabilistic ways to accomplish logic inference? Or statistical theory and logic inference are overlapping, but neither falls into the other? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Relating to &lt;a href=&quot;http://stats.stackexchange.com/questions/9208/what-is-the-relation-between-statistics-theory-and-decision-theory&quot;&gt;my previous post regarding decision theory and statistical theory&lt;/a&gt;, how are logical inference, decision theory and statistical theory are related together and differ from each other?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks and regards!&lt;/p&gt;&#xA;" OwnerUserId="1005" LastEditorUserId="1005" LastEditorDisplayName="" LastEditDate="2011-04-09T14:27:17.710" LastActivityDate="2011-04-09T14:27:17.710" Title="Inference and noninference parts of statistical theory" Tags="&lt;inference&gt;&lt;logic&gt;&lt;statistical-theory&gt;" CommentCount="2" FavoriteCount="2" />
  <row Id="9382" PostTypeId="2" ParentId="9131" CreationDate="2011-04-09T14:56:49.983" Score="0" ViewCount="" Body="&lt;p&gt;See the formula in the &lt;em&gt;Confidence Interval on New Observations&lt;/em&gt; section (prediction interval) of &lt;a href=&quot;http://www.weibull.com/DOEWeb/confidence_intervals_in_multiple_linear_regression.htm&quot; rel=&quot;nofollow&quot;&gt;http://www.weibull.com/DOEWeb/confidence_intervals_in_multiple_linear_regression.htm&lt;/a&gt; &lt;a href=&quot;http://www.webcitation.org/5xonEjGdb&quot; rel=&quot;nofollow&quot;&gt;(permalink)&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastActivityDate="2011-04-09T14:56:49.983" />
  <row Id="9383" PostTypeId="1" CreationDate="2011-04-09T15:24:45.393" Score="1" ViewCount="34" Body="&lt;p&gt;I'm doing 10-fold cross validation on a dataset. But in some folds there are edge cases that the denominator in precision-recall calculation is zero (tp + fp =0).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are the correct values for precision and recall in this case? And what is the correct way of doing cross-validation (should I include these results when reporting average precision-recall over 10 folds)? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: This question is very similar to &lt;a href=&quot;http://stats.stackexchange.com/questions/8025/what-are-correct-values-for-precision-and-recall-when-the-denominators-equal-0&quot;&gt;What are correct values for precision and recall when the denominators equal 0?&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance : -)&lt;/p&gt;&#xA;" OwnerUserId="4091" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-09T15:35:03.190" LastActivityDate="2011-04-09T20:05:43.907" Title="What are the correct edge case values of precision and recall and how to integrate them into cross validation? " Tags="&lt;cross-validation&gt;&lt;precision-recall&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="9384" PostTypeId="2" ParentId="9342" CreationDate="2011-04-09T15:57:28.113" Score="4" ViewCount="" Body="&lt;p&gt;The Ward clustering algorithm is a hierarchical clustering method that minimizes an 'inertia' criteria at each step. This inertia quantifies the sum of squared residuals between the reduced signal and the initial signal: it is a measure of the variance of the error in an l2 (Euclidean) sens. Actually, you even mention it in your question. This is why, I believe, it makes no sens to apply it to a distance matrix that is not a l2 Euclidean distance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, an average linkage or a single linkage hierarchical clustering would be perfectly suitable for other distances.&lt;/p&gt;&#xA;" OwnerUserId="1265" LastActivityDate="2011-04-09T15:57:28.113" CommunityOwnedDate="2011-04-09T15:57:28.113" CommentCount="2" />
  <row Id="9385" PostTypeId="1" CreationDate="2011-04-09T15:59:58.363" Score="4" ViewCount="208" Body="&lt;p&gt;I am using the Holt-Winters' exponential smoothing technique to forecast expenditure data 2 years into the furture. The monthly data has an increasing trend and annual seasonality.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm using MS Excel with the Solver add-in to calculate the optimal values of $\alpha$, $\beta$ and $\gamma$ to give the smallest MSE for the forecasts. The optimal values found for $\alpha$ and $\beta$ lie in (0,1) and $\gamma$ is found to be 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am able to calculate the forecasts for the next year (season) because the seasonals from the previous year exist. However, the forecasts for the second year are calculated to be zero because the seasonals do not exist (because &lt;em&gt;m&lt;/em&gt; is greater than 12). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have discovered that if $\gamma$ is zero, then the seasonals will be periodic, and so could be replicated after the last observed values. Is this the best way to forecast beyond one season after the last observed values? Any advice would be appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example data is below. Forecasts are needed for every month up to December 2011. I cannot see how this is possible unless $\gamma$ is zero.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Numbers of Tourists&lt;br&gt;&#xA;Period  Month   No. Tourists (Yt)&lt;br&gt;&#xA;1   Jan-99  500&lt;br&gt;&#xA;2   Feb-99  543&lt;br&gt;&#xA;3   Mar-99  899&lt;br&gt;&#xA;4   Apr-99  835&lt;br&gt;&#xA;5   May-99  900&lt;br&gt;&#xA;6   Jun-99  881&lt;br&gt;&#xA;7   Jul-99  1154&lt;br&gt;&#xA;8   Aug-99  1586&lt;br&gt;&#xA;9   Sep-99  743&lt;br&gt;&#xA;10  Oct-99  1104&lt;br&gt;&#xA;11  Nov-99  799&lt;br&gt;&#xA;12  Dec-99  560&lt;br&gt;&#xA;13  Jan-00  514&lt;br&gt;&#xA;14  Feb-00  665&lt;br&gt;&#xA;15  Mar-00  949&lt;br&gt;&#xA;16  Apr-00  975&lt;br&gt;&#xA;17  May-00  924&lt;br&gt;&#xA;18  Jun-00  724&lt;br&gt;&#xA;19  Jul-00  1155&lt;br&gt;&#xA;20  Aug-00  1541&lt;br&gt;&#xA;21  Sep-00  746&lt;br&gt;&#xA;22  Oct-00  944&lt;br&gt;&#xA;23  Nov-00  786&lt;br&gt;&#xA;24  Dec-00  652&lt;br&gt;&#xA;25  Jan-01  479.4&lt;br&gt;&#xA;26  Feb-01  644.4&lt;br&gt;&#xA;27  Mar-01  815.8&lt;br&gt;&#xA;28  Apr-01  1035.4&lt;br&gt;&#xA;29  May-01  1000.9&lt;br&gt;&#xA;30  Jun-01  793.8&lt;br&gt;&#xA;31  Jul-01  1347.3&lt;br&gt;&#xA;32  Aug-01  1378&lt;br&gt;&#xA;33  Sep-01  798.1&lt;br&gt;&#xA;34  Oct-01  1070.5&lt;br&gt;&#xA;35  Nov-01  625.3&lt;br&gt;&#xA;36  Dec-01  654&lt;br&gt;&#xA;37  Jan-02  477.5&lt;br&gt;&#xA;38  Feb-02  656.2&lt;br&gt;&#xA;39  Mar-02  888.7&lt;br&gt;&#xA;40  Apr-02  926.6&lt;br&gt;&#xA;41  May-02  1000.1&lt;br&gt;&#xA;42  Jun-02  1030.8&lt;br&gt;&#xA;43  Jul-02  1123&lt;br&gt;&#xA;44  Aug-02  1473.5&lt;br&gt;&#xA;45  Sep-02  717.8&lt;br&gt;&#xA;46  Oct-02  974.7&lt;br&gt;&#xA;47  Nov-02  761.2&lt;br&gt;&#xA;48  Dec-02  641.5&lt;br&gt;&#xA;49  Jan-03  501.6&lt;br&gt;&#xA;50  Feb-03  588.3&lt;br&gt;&#xA;51  Mar-03  917.6&lt;br&gt;&#xA;52  Apr-03  990&lt;br&gt;&#xA;53  May-03  1051&lt;br&gt;&#xA;54  Jun-03  764.4&lt;br&gt;&#xA;55  Jul-03  1014.2&lt;br&gt;&#xA;56  Aug-03  1313.6&lt;br&gt;&#xA;57  Sep-03  736.3&lt;br&gt;&#xA;58  Oct-03  1042.9&lt;br&gt;&#xA;59  Nov-03  685.9&lt;br&gt;&#xA;60  Dec-03  621.5&lt;br&gt;&#xA;61  Jan-04  492.8&lt;br&gt;&#xA;62  Feb-04  722&lt;br&gt;&#xA;63  Mar-04  869.9&lt;br&gt;&#xA;64  Apr-04  927.9&lt;br&gt;&#xA;65  May-04  1028.1&lt;br&gt;&#xA;66  Jun-04  883&lt;br&gt;&#xA;67  Jul-04  1097.4&lt;br&gt;&#xA;68  Aug-04  1398.9&lt;br&gt;&#xA;69  Sep-04  834.4&lt;br&gt;&#xA;70  Oct-04  1072.3&lt;br&gt;&#xA;71  Nov-04  801.9&lt;br&gt;&#xA;72  Dec-04  711.2&lt;br&gt;&#xA;73  Jan-05  616.1&lt;br&gt;&#xA;74  Feb-05  774&lt;br&gt;&#xA;75  Mar-05  1088.5&lt;br&gt;&#xA;76  Apr-05  956.2&lt;br&gt;&#xA;77  May-05  1175.6&lt;br&gt;&#xA;78  Jun-05  949.5&lt;br&gt;&#xA;79  Jul-05  1120.8&lt;br&gt;&#xA;80  Aug-05  1426.2&lt;br&gt;&#xA;81  Sep-05  841.5&lt;br&gt;&#xA;82  Oct-05  996.6&lt;br&gt;&#xA;83  Nov-05  908&lt;br&gt;&#xA;84  Dec-05  696.7&lt;br&gt;&#xA;85  Jan-06  606.4&lt;br&gt;&#xA;86  Feb-06  771.6&lt;br&gt;&#xA;87  Mar-06  967.1&lt;br&gt;&#xA;88  Apr-06  1235&lt;br&gt;&#xA;89  May-06  1216.1&lt;br&gt;&#xA;90  Jun-06  945.1&lt;br&gt;&#xA;91  Jul-06  1194.4&lt;br&gt;&#xA;92  Aug-06  1433.4&lt;br&gt;&#xA;93  Sep-06  830.6&lt;br&gt;&#xA;94  Oct-06  984.7&lt;br&gt;&#xA;95  Nov-06  880.2&lt;br&gt;&#xA;96  Dec-06  668.3&lt;br&gt;&#xA;97  Jan-07  644.9&lt;br&gt;&#xA;98  Feb-07  808&lt;br&gt;&#xA;99  Mar-07  998.2&lt;br&gt;&#xA;100 Apr-07  1283.9&lt;br&gt;&#xA;101 May-07  1080.9&lt;br&gt;&#xA;102 Jun-07  989.9&lt;br&gt;&#xA;103 Jul-07  1167&lt;br&gt;&#xA;104 Aug-07  1568.9&lt;br&gt;&#xA;105 Sep-07  951.7&lt;br&gt;&#xA;106 Oct-07  1121.4&lt;br&gt;&#xA;107 Nov-07  859&lt;br&gt;&#xA;108 Dec-07  660.9&lt;br&gt;&#xA;109 Jan-08  647.9&lt;br&gt;&#xA;110 Feb-08  911.1&lt;br&gt;&#xA;111 Mar-08  1201.2&lt;br&gt;&#xA;112 Apr-08  1258.1&lt;br&gt;&#xA;113 May-08  1177.8&lt;br&gt;&#xA;114 Jun-08  1067.6&lt;br&gt;&#xA;115 Jul-08  1349.4&lt;br&gt;&#xA;116 Aug-08  1702.1&lt;br&gt;&#xA;117 Sep-08  982.8&lt;br&gt;&#xA;118 Oct-08  1116.5&lt;br&gt;&#xA;119 Nov-08  904.7&lt;br&gt;&#xA;120 Dec-08  655.9&lt;br&gt;&#xA;121 Jan-09  733.75&lt;br&gt;&#xA;122 Feb-09  852.67&lt;br&gt;&#xA;123 Mar-09  1049.88&lt;br&gt;&#xA;124 Apr-09  1377.11&lt;br&gt;&#xA;125 May-09  1344.05&lt;br&gt;&#xA;126 Jun-09  1030.95&lt;br&gt;&#xA;127 Jul-09  1242.56&lt;br&gt;&#xA;128 Aug-09  1542.24&lt;br&gt;&#xA;129 Sep-09  1016.42&lt;br&gt;&#xA;130 Oct-09  2301.41&lt;br&gt;&#xA;131 Nov-09  1138.9&lt;br&gt;&#xA;132 Dec-09  1032.87  &lt;/p&gt;&#xA;" OwnerUserId="4092" LastEditorUserId="4092" LastEditorDisplayName="" LastEditDate="2011-04-11T13:12:59.667" LastActivityDate="2011-05-06T15:14:38.527" Title="Forecasting beyond one season using Holt-Winters' exponential smoothing" Tags="&lt;time-series&gt;&lt;forecasting&gt;&lt;excel&gt;&lt;smoothing&gt;&lt;exponential&gt;" AnswerCount="3" CommentCount="6" FavoriteCount="1" />
  <row Id="9387" PostTypeId="2" ParentId="9365" CreationDate="2011-04-09T16:49:38.487" Score="4" ViewCount="" Body="&lt;p&gt;On a wider level I would recommend the [&quot;Statistical Modeling: The Two Cultures&quot;][1] paper by Leo Breiman in 2001 (cited 515) I know it was covered by the journal club recently and I found it to be really interesting. I've c&amp;amp;p'd the abstract.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Abstract. There are two cultures in&#xA;  the use of statistical modeling to&#xA;  reach conclusions from data. One&#xA;  assumes that the data are generated by&#xA;  a given stochastic data model. The&#xA;  other uses algorithmic models and&#xA;  treats the data mechanism as unknown.&#xA;  The statistical community has been&#xA;  committed to the almost exclusive use&#xA;  of data models. This commitment has&#xA;  led to irrelevant theory, questionable&#xA;  conclusions, and has kept&#xA;  statisticians from working on a large&#xA;  range of interesting current problems.&#xA;  Algorithmic modeling, both in theory&#xA;  and practice, has developed rapidlyin&#xA;  fields outside statistics. It can be&#xA;  used both on large complex data sets&#xA;  and as a more accurate and informative&#xA;  alternative to data modeling on&#xA;  smaller data sets. If our goal as a&#xA;  field is to use data to solve&#xA;  problems, then we need to move away&#xA;  from exclusive dependence on data&#xA;  models and adopt a more diverse set of&#xA;  tools.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;[1]:&#xA;  &lt;a href=&quot;http://www.stat.osu.edu/~bli/dmsl/papers/Breiman.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.stat.osu.edu/~bli/dmsl/papers/Breiman.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="3597" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-13T06:59:09.850" LastActivityDate="2011-04-13T06:59:09.850" CommunityOwnedDate="2011-04-09T16:49:38.487" />
  <row Id="9388" PostTypeId="2" ParentId="3898" CreationDate="2011-04-09T18:09:15.317" Score="0" ViewCount="" Body="&lt;p&gt;If the problem at issue consists of testing for the optimal number of factors, Jushan Bai and Serena Ng in several articles provide a test based on AIC/BIC that minimizes, for different options, the variance of the error. They supply to my knowledge the most updated approach to resolve this issue. See also Alexei Onatski who uses a different method based on eigenvalues of the factor covariance matrix.&lt;/p&gt;&#xA;" OwnerUserId="4093" LastActivityDate="2011-04-09T18:09:15.317" CommentCount="1" />
  <row Id="9390" PostTypeId="1" CreationDate="2011-04-09T18:24:23.090" Score="1" ViewCount="124" Body="&lt;p&gt;An English soccer team plays a series of matches against different opponents, of varying ability. A bookmaker offers odds for each match as to whether it will be a home win, away win, or draw. Part-way through the season, the team has played n matches, and has drawn $k$ of them, which is more than might be expected from the odds. What is the probability that the bookmaker is mis-pricing the odds on these matches, rather than just being unlucky? If the bookmaker continues to price the team's remaining matches in a similar way, and I bet $1 that each one will be a draw, what is my expected return?&lt;/p&gt;&#xA;" OwnerUserId="4094" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-14T08:56:10.877" LastActivityDate="2011-04-14T08:56:10.877" Title="What's the probability that a bookmaker is mispricing odds on soccer games?" Tags="&lt;probability&gt;&lt;games&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="2" />
  <row Id="9391" PostTypeId="2" ParentId="9390" CreationDate="2011-04-09T18:49:59.820" Score="0" ViewCount="" Body="&lt;p&gt;Bookmakers use an overround so they don't actually care what the result is because they win whatever. That is why you never meet a poor bookie. If a bookmaker is mispricing draws your ability to make profit would depend on the odds the bookmaker was offering and whether the profits generated would cover the times you lose.&lt;/p&gt;&#xA;" OwnerUserId="3597" LastActivityDate="2011-04-09T18:49:59.820" CommentCount="2" />
  <row Id="9392" PostTypeId="1" CreationDate="2011-04-09T18:54:22.423" Score="0" ViewCount="100" Body="&lt;p&gt;Hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My data looks like this (F=Features)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;        F1     F2     F3     F4     F5     F6     F7     F8....&#xA;ID1     0.67   0.76   0.3    0.54   0.21   0.88   0.97   0.45....&#xA;ID2     0.76   0.68   0.10   0.45   0.12   0.44   0.79   0.54....&#xA;ID3     0.67   0.76   0.3    0.54   0.21   0.88   0.68   0.76....&#xA;ID4     0.67   0.10   0.3    0.45   0.3    0.88   0.97   0.45....&#xA;...&#xA;...&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have about 40 features (I have just put 8 here). If I set a threshold for, say 4 features, what I am looking for is a combination of 4 features which, together, are most relevant and significant in the dataset. I need some kind of score that measures how good a combination of features is. This is what I meant by the confidence score (or whatever score we may call it). So instead of selecting 1 feature I want to select a combination of 4 features. For example.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;F1-F4-F9-F12 = 0.92&#xA;F2-F3-F7-F6 = 0.85&#xA;F5-F3-F4-F8 = 0.667&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here, F1-F4 is not subtraction. I am just attaching the two features together. The scores above, I do not know how to get it. That is my question? So I do not know what kind of a test to use or can be used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I go about it? Thanks&lt;/p&gt;&#xA;" OwnerUserId="3111" LastEditorUserId="3111" LastEditorDisplayName="" LastEditDate="2011-04-11T20:52:30.830" LastActivityDate="2011-04-13T02:59:16.640" Title="How to get scored combination of features" Tags="&lt;feature-selection&gt;&lt;text-mining&gt;" AnswerCount="1" CommentCount="12" FavoriteCount="1" />
  <row Id="9393" PostTypeId="2" ParentId="9383" CreationDate="2011-04-09T20:05:43.907" Score="1" ViewCount="" Body="&lt;p&gt;Instead of using CV to estimate precision and recall, use it to obtain the expected TP, TN, FP, and FN rate.  Then use those values to compute the expected precision and recall and the standard errors.  (Taylor expansions come in handy for the latter.)&lt;/p&gt;&#xA;" OwnerUserId="3567" LastActivityDate="2011-04-09T20:05:43.907" CommentCount="2" />
  <row Id="9394" PostTypeId="2" ParentId="9365" CreationDate="2011-04-09T20:39:39.403" Score="2" ViewCount="" Body="&lt;p&gt;From a genetic epidemiology perspective, I would now recommend the following series of papers about &lt;a href=&quot;http://en.wikipedia.org/wiki/Genome-wide_association_study&quot; rel=&quot;nofollow&quot;&gt;genome-wide association studies&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Cordell, H.J. and Clayton, D.G. (2005). &lt;a href=&quot;http://www.thelancet.com/journals/lancet/article/PIIS0140-6736%2805%2967424-7/abstract&quot; rel=&quot;nofollow&quot;&gt;Genetic association studies&lt;/a&gt;. &lt;em&gt;Lancet&lt;/em&gt; 366, 1121-1131.&lt;/li&gt;&#xA;&lt;li&gt;Cantor, R.M., Lange, K., and Sinsheimer, J.S. (2010). &lt;a href=&quot;http://www.cell.com/AJHG/abstract/S0002-9297%2809%2900532-1&quot; rel=&quot;nofollow&quot;&gt;Prioritizing GWAS results: A review of statistical methods and recommendations for their application&lt;/a&gt;. &lt;em&gt;The American Journal of Human Genetics&lt;/em&gt; 86, 6–22.&lt;/li&gt;&#xA;&lt;li&gt;Ioannidis, J.P.A., Thomas, G., Daly, M.J. (2009). &lt;a href=&quot;http://www.nature.com/nrg/journal/v10/n5/abs/nrg2544.html&quot; rel=&quot;nofollow&quot;&gt;Validating, augmenting and refining genome-wide association signals&lt;/a&gt;. &lt;em&gt;Nature Reviews Genetics&lt;/em&gt; 10, 318-329.&lt;/li&gt;&#xA;&lt;li&gt;Balding, D.J. (2006). &lt;a href=&quot;http://www.nature.com/nrg/journal/v7/n10/full/nrg1916.html&quot; rel=&quot;nofollow&quot;&gt;A tutorial on statistical methods for population association studies&lt;/a&gt;. &lt;em&gt;Nature Reviews Genetics&lt;/em&gt; 7, 781-791.&lt;/li&gt;&#xA;&lt;li&gt;Green, A.E. et al. (2008). &lt;a href=&quot;http://www.nature.com/nrn/journal/v9/n9/abs/nrn2461.html&quot; rel=&quot;nofollow&quot;&gt;Using genetic data in cognitive neuroscience: from growing pains to genuine insights&lt;/a&gt;. &lt;em&gt;Nature Reviews Neuroscience&lt;/em&gt; 9, 710-720.&lt;/li&gt;&#xA;&lt;li&gt;McCarthy, M.I. et al. (2008). &lt;a href=&quot;http://www.nature.com/nrg/journal/v9/n5/abs/nrg2344.html&quot; rel=&quot;nofollow&quot;&gt;Genome-wide association studies for complex traits: consensus, uncertainty and challenges&lt;/a&gt;. &lt;em&gt;Nature Reviews Genetics&lt;/em&gt; 9, 356-369.&lt;/li&gt;&#xA;&lt;li&gt;Psychiatric GWAS Consortium Coordinating Committee (2009). &lt;a href=&quot;http://ajp.psychiatryonline.org/cgi/content/abstract/166/5/540&quot; rel=&quot;nofollow&quot;&gt;Genomewide Association Studies: History, Rationale, and Prospects for Psychiatric Disorders&lt;/a&gt;. &lt;em&gt;American Journal of Psychiatry&lt;/em&gt; 166(5), 540-556.&lt;/li&gt;&#xA;&lt;li&gt;Sebastiani, P. et al. (2009). &lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pubmed/19569043&quot; rel=&quot;nofollow&quot;&gt;Genome-wide association studies and the genetic dissection of complex traits&lt;/a&gt;. &lt;em&gt;American Journal of Hematology&lt;/em&gt; 84(8), 504-15.&lt;/li&gt;&#xA;&lt;li&gt;The Wellcome Trust Case Control Consortium (2007). &lt;a href=&quot;http://www.nature.com/nature/journal/v447/n7145/abs/nature05911.html&quot; rel=&quot;nofollow&quot;&gt;Genome-wide association study of 14,000 cases of seven common diseases and 3,000 shared controls&lt;/a&gt;. &lt;em&gt;Nature&lt;/em&gt; 447, 661-678.&lt;/li&gt;&#xA;&lt;li&gt;The Wellcome Trust Case Control Consortium (2010). &lt;a href=&quot;http://www.nature.com/nature/journal/v464/n7289/full/nature08979.html&quot; rel=&quot;nofollow&quot;&gt;Genome-wide association study of CNVs in 16,000 cases of eight common diseases and 3,000 shared controls&lt;/a&gt;. &lt;em&gt;Nature&lt;/em&gt; 464, 713-720.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-04-09T20:39:39.403" CommunityOwnedDate="2011-04-09T20:39:39.403" />
  <row Id="9395" PostTypeId="1" CreationDate="2011-04-09T23:20:10.947" Score="4" ViewCount="58" Body="&lt;p&gt;I don't know what is the appropriate term for my question. The scenario is described as following.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the analysis there one dependent variable Y and two independent variable X1 and X2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All three variables are continuous.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I converted X1 into a categorical variable which has three levels A, B, and C. It was found that Y and X2 are positively correlated in group A and B, but negatively correlated in group C.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was told that convert a continuous variable into categorical was generally a bad idea and I understand this. My question is, how can I demonstrate the above pattern without break X1 into categories? I was suggested to go with multiple regression, but I still don't know how to demonstrate this kind of relationship in the three variables with multiple regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="400" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-10T13:06:48.453" LastActivityDate="2011-04-10T13:06:48.453" Title="How can I demonstrate non-linearity without categorising a predictor?" Tags="&lt;regression&gt;&lt;categorical-data&gt;&lt;continuous-data&gt;&lt;nonlinear-regression&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="9396" PostTypeId="1" CreationDate="2011-04-09T23:24:44.063" Score="5" ViewCount="58" Body="&lt;p&gt;In the paper &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;M. Avellaneda and J. H. Lee, &lt;a href=&quot;http://www.math.nyu.edu/faculty/avellane/AvellanedaLeeStatArb071108.pdf&quot; rel=&quot;nofollow&quot;&gt;Statistical arbitrage in the U.S. equities market&lt;/a&gt;, July 2008,&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;in the Appendix on page 44, I have some questions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First he runs the regression of stock-return ($R_n^S$) with index/ETF-return ($R_n^I$). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$R_n^S = \beta_0 + \beta R_n^I + \epsilon_n, ~~~~~n=1,2,...,60$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then he defines an auxiliary process $X_n$ as sum of the residuals from regression of $R_n$, and estimates it to be a mean reverting process.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$X_n = \sum_{j=1}^{n} \epsilon_j ~~~~~n=1,2,...,60$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have two related questions?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the significance of taking cumulative sum of residuals as opposed to taking just residuals as a mean reverting process? I have a basic intuition but lack a good understanding.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second, he says that the regression on stock-returns &quot;forces&quot; the residuals to have mean zero. Why is this? How does this imply that the sum of all residuals, $X_{60}=0$?&lt;/p&gt;&#xA;" OwnerUserId="862" LastActivityDate="2011-04-09T23:24:44.063" Title="Why are cumulative residuals from regression on stock and index returns mean reverting" Tags="&lt;regression&gt;&lt;mean&gt;&lt;stochastic-processes&gt;" CommentCount="1" FavoriteCount="3" />
  <row Id="9397" PostTypeId="2" ParentId="9395" CreationDate="2011-04-09T23:49:17.103" Score="2" ViewCount="" Body="&lt;p&gt;Converting a continuous variable into categorical may be a bad idea, but may be a good idea as well, this depends on the problem. When the relationships of the variable can be best described using thresholds, categorisation may be one of the best options.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You wrote that in different categories of X1 the correlation between Y and X2 is very different. This is a clear indication of a non-linear relationship between Y, X1 and X2. Thus multiple linear regression is probably not the best method to use here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In any case I suggest you to visualize your data (maybe using a &lt;a href=&quot;http://addictedtor.free.fr/graphiques/RGraphGallery.php?graph=73&quot; rel=&quot;nofollow&quot;&gt;circles plot&lt;/a&gt;, or &lt;a href=&quot;http://www.cas.manchester.ac.uk/restools/instruments/aerosol/sp2/results/singleplot/index.html&quot; rel=&quot;nofollow&quot;&gt;coloured scatterplot&lt;/a&gt;). You may continue with machine learning, or modelling methods that suit what you know about your data.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastActivityDate="2011-04-09T23:49:17.103" CommentCount="1" />
  <row Id="9398" PostTypeId="1" AcceptedAnswerId="9406" CreationDate="2011-04-09T23:31:25.733" Score="9" ViewCount="162" Body="&lt;p&gt;Suppose you get to observe &quot;matches&quot; between buyers and sellers in a market. You also get to observe characteristics of both buyers and sellers which you would like to use to predict future matches &amp;amp; make recommendations to both sides of the market.    &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For simplicity, assume there are N buyers and N sellers and that each finds a match. There are N matches and (N-1)(N-1) non-matches. The all-inclusive training dataset has N + (N-1)*(N-1) observations, which can be prohibitively large. It would seem that randomly sampling from the (N-1)(N-1) non-matches and training an algorithm on that reduced data could be more efficient. My questions are: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(1) Is sampling from the non-matches to build a training dataset a reasonable way to deal with this problem? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(2) If (1) is true, is there is a rigorous way to decide how big of a chunk of (N-1)(N-1) to include?  &lt;/p&gt;&#xA;" OwnerUserId="4095" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-04-10T17:54:19.620" LastActivityDate="2011-04-10T17:54:19.620" Title="Supervised learning with &quot;rare&quot; events, when rarity is due to the large number of counter-factual events" Tags="&lt;machine-learning&gt;" AnswerCount="2" CommentCount="0" FavoriteCount="4" />
  <row Id="9399" PostTypeId="2" ParentId="9398" CreationDate="2011-04-09T23:47:33.450" Score="1" ViewCount="" Body="&lt;p&gt;Concerning (1). You need to keep positive and negative observations if you want meaningful results.&lt;br&gt;&#xA;(2) There is no wiser method of subsampling than uniform distribution if you don't have any a priori on your data. &lt;/p&gt;&#xA;" OwnerUserId="3896" LastActivityDate="2011-04-10T00:28:15.327" CommentCount="3" />
  <row Id="9400" PostTypeId="1" CreationDate="2011-04-10T00:58:48.477" Score="5" ViewCount="80" Body="&lt;p&gt;My actual project is a bit complicated, but I'll explain by analogy (which I hope facilitates response):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have 3 substances, say water, motor oil, and ethanol. For each substance, I have 5 samples in a beaker (total 15 beakers). I heat all the beakers on a hot-plate up to 70 degrees Celsius, and over the next hour, I measure the temperature of the fluid in each beaker at 5 minute intervals.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Newtonian cooling provides me with a good prediction about these temperature data, namely that the temperature of the fluid in each cup should follow an exponential distribution: y = a + e^(-kt) where a is room temperature.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to estimate the value k for each substance and test the hypothesis that k1 &gt; k2 &gt; k3 (1, 2, 3 corresponding to my three substances). The natural method of estimating k seems to be computing a non-linear regression on each substance's data, or possibly log-transforming all the data and then just computing a simple linear regression. However, there are problems.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Given the obvious autocorrelation in the longitudinal data (confirmed by my (P)ACF plots of course), must I compute an AR term and filter my data prior to computing the regression?&lt;/li&gt;&#xA;&lt;li&gt;Assuming I compute this autoregression term, how do I compute it for five independent sets of data (the five beakers of a give substance)? I could average the five beakers together and then compute the regression, but this screws up my AR term (assuming I need one) and also throws off my estimate of the actual within-beaker variance from the model.&lt;/li&gt;&#xA;&lt;li&gt;What completely wrong-headed assumption(s) have I worked in here...?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="4096" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-10T03:07:18.390" LastActivityDate="2011-04-10T17:25:16.590" Title="How can I compute regression for several longitudinal data sets (thus, with auto-correlated error)?" Tags="&lt;regression&gt;&lt;autocorrelation&gt;&lt;nonlinear-regression&gt;&lt;longitudinal&gt;&lt;exponential&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="1" />
  <row Id="9401" PostTypeId="2" ParentId="9378" CreationDate="2011-04-10T01:21:50.127" Score="1" ViewCount="" Body="&lt;p&gt;Use the Sample operator with the Balance checkbox. You can set the sample size per class that way (to a balanced one)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;@steffen, the mandate for this site covers stats AND stats software. There are tons of R questions on here, so it's fair to ask questions about other software too. &lt;/p&gt;&#xA;" OwnerUserId="74" LastActivityDate="2011-04-10T01:21:50.127" CommentCount="3" />
  <row Id="9402" PostTypeId="2" ParentId="9342" CreationDate="2011-04-10T02:10:19.477" Score="0" ViewCount="" Body="&lt;p&gt;Another way of thinking about this, which might lend itself to an adaptation for $\ell_1$ is that choice of the mean comes from the fact that the mean is the point that minimizes the sum of squared Euclidean distances. If you're using $\ell_1$ to measure the distance between time series, then you should be using a center that minimizes the sum of squared $\ell_1$ distances. &lt;/p&gt;&#xA;" OwnerUserId="139" LastActivityDate="2011-04-10T02:10:19.477" />
  <row Id="9403" PostTypeId="2" ParentId="9400" CreationDate="2011-04-10T02:37:10.260" Score="2" ViewCount="" Body="&lt;p&gt;As we have strong reasons to believe that the cooling will follow the $y(t) = a + e^{-kt}$ function for each beaker I would first check if this model fits the data well indeed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If it does I wouldn't bother with analysing the autocorrelation at all, but focus on the estimation of $k_1$, $k_2$ and $k_3$, and testing the hypothesis about them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To estimate $k_1$, $k_2$ and $k_3$ you need a non-linear model. Your idea of log transformation followed by linear modelling is best when the error (difference between the measured $y$ temperature and the one predicted by the formula) is proportional to the temperature. However, I suspect that the error will be primarily due to temperature measurement and thus normally distributed with the same variance for any temperature (you need to check this). If so, a non-linear model would be more appropriate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A model using the above function will give you estimates for the parameters of the cooling of a single beaker, $a$ and $k$. We may however assume that $a$ should be the same for each beaker, $k$s should be similar for the same substance, and that the standard deviation ($\sigma$) is the same across all temperature measurements. These can be expressed in a model accounting for all the beakers in the same time (second index $j$ is beaker ID):&#xA;$$y_j(t) = a + e^{-(k_i + \alpha_j)t} + \epsilon$$&#xA;where&#xA;$\epsilon$ is normally distributed error of SD $\sigma$, $k_i$ is one of 3 mean $k$ values for substance $i$, $\alpha_j$ is normally distributed random deviation of a specific beaker from the $k_i$ substance mean, with a substance specific SD ($\sigma_{\alpha{}i}$). This is now a non-linear mixed effect model, that can be fitted using various software. After this you have the $k_i$ values and their standard errors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The next question is how to test the hypothesis that $k_1 &amp;gt; k_2 &amp;gt; k_3$. It may be “cleaner” to formulate such a hypothesis in the Bayesian way. However you used the word test, so you probably want a significance test – but in order to do that you have to have a more specific alternative hypothesis (or family of hypotheses).&lt;/p&gt;&#xA;" OwnerUserId="3911" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-10T13:02:01.177" LastActivityDate="2011-04-10T13:02:01.177" CommentCount="2" />
  <row Id="9404" PostTypeId="2" ParentId="9390" CreationDate="2011-04-10T03:24:53.640" Score="2" ViewCount="" Body="&lt;p&gt;The answer to your question depends intricately on what information and assumptions you are going to use.  This is because the result of a game is an extraordinarily complicated process.  It can become arbitrarily complicated depending on what information you have about:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Players in the particular team - perhaps even particular combinations of players may be relevant.&lt;/li&gt;&#xA;&lt;li&gt;Players in other teams&lt;/li&gt;&#xA;&lt;li&gt;Past history of the league&lt;/li&gt;&#xA;&lt;li&gt;How stable the team's players are - do players keep getting selected and dropped, or is it the same 11.&lt;/li&gt;&#xA;&lt;li&gt;The time that you place your bet (during the game? before? how much before? what info is lost from betting before to betting on the day?)&lt;/li&gt;&#xA;&lt;li&gt;some other relevant feature of soccer which I have omitted.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The odds that a book-maker gives are not a reflection of the book-makers odds.  which is impossible if they are probabilities.  A book-maker will adjust the odds down when someone bets on a draw, and adjust them up when someone bets on a non-draw.  Thus, the odds are a reflection of the gamblers (who use that book-maker) odds as a whole.  So it is not the bookmaker who is miss-pricing per se, it is the gambling collective - or the &quot;average gambler&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now if you are willing to assume that whatever &quot;causal mechanism&quot; is resulting in a draw remains constant across the season (reasonable? probably not...), then a simple mathematical problem is obtained (but note there is no reason for this to be &quot;more right&quot; than some other simplifying assumption).  To remind us that this is the assumption being used, an $A$ will be put on the conditioning side of the probabilities.  Under this assumption the binomial distribution applies:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(\text{k Draws in n matches}|\theta,A)={n \choose k}\theta^{k}(1-\theta)^{n-k}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And we want to calculate the following&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(\text{next match is a draw}|\text{k Draws in n matches},A)$$&#xA;$$=\int_{0}^{1}P(\text{next match is a draw}|\theta,A)P(\theta|\text{k Draws in n matches},A)d\theta$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where &#xA;$$P(\theta|\text{k Draws in n matches},A)=P(\theta|A)\frac{P(\text{k Draws in n matches}|\theta,A)}{P(\text{k Draws in n matches}|A)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;is the posterior for $\theta$.  Now in this case, it is fairly obvious that it is possible for a draw to happen, and also possible for it to not happen, so a uniform prior is appropriate (unless there is extra information we wish to include beyond the results of the season) and we set $P(\theta|A)=1$.  The posterior is then given by a beta distribution (where $B(\alpha,\beta)$ is the &lt;a href=&quot;http://en.wikipedia.org/wiki/Beta_function&quot; rel=&quot;nofollow&quot;&gt;beta function&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(\theta|\text{k Draws in n matches},A)=\frac{\theta^{k}(1-\theta)^{n-k}}{B(k+1,n-k+1)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given $\theta$ and $A$ the probability that the next match is a draw is just $\theta$ so the integral becomes:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\int_{0}^{1}\theta \frac{\theta^{k}(1-\theta)^{n-k}}{B(k+1,n-k+1)}d\theta=\frac{B(k+2,n-k+1)}{B(k+1,n-k+1)}=\frac{k+1}{n+2}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and hence the probability is just:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(\text{next match is a draw}|\text{k Draws in n matches},A)=\frac{k+1}{n+2}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But note that it depends on $A$ - the assumptions that were made.  Call the &quot;priced odds&quot; a probability conditional on some other unknown complex information, say $B$.  So if the published odds are different to the the above fraction, then this says that $A$ and $B$ lead to different conclusions, so both can't be right about the &quot;true outcome&quot; (but both &lt;em&gt;can&lt;/em&gt; be right conditional on the assumptions each made).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;THE KILLER BLOW&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This example showed that the answer to your question boiled down to deciding if $A$ was &quot;more accurate&quot; than $B$ in describing the mechanics of the soccer game.  &lt;em&gt;This will happen regardless of what the proposition&lt;/em&gt; $A$ &lt;em&gt;happens to be&lt;/em&gt;.  We will always boil down to the question of asking &quot;whose assumptions are right, the gambling collective's or mine?&quot;  This last question is a basically unanswerable question until you know exactly what the proposition $B$ consists of (or at least some key features of it).  For how can you compare something that is known with something that is not?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE: An actual answer :)&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As @whuber has cheekily pointed out, I haven't actually given an expected value here - so this part simply completes that part of my answer.  If one was to assume that $A$ &lt;em&gt;is true&lt;/em&gt; with priced odds of $Q$, then you would expect, in the next game to receive &#xA;$$Q\times P(\text{next match is a draw}|\text{k Draws in n matches},A)-1$$&#xA;$$=Q\times \frac{k+1}{n+2}-1=\frac{Q(k+1)-n-2}{n+2}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now if you assume that the value of $Q$ is based on the same model as yours then we can predict exactly how $Q$ will change into the future.  Suppose $Q$ was based on a different prior to the uniform one, say $Beta(\alpha_Q,\beta_Q)$, then the corresponding probability is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(\text{next match is a draw}|\text{k Draws in n matches},A_Q)=\frac{k+\alpha_Q}{n+\alpha_Q+\beta_Q}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;with expected return of &#xA;$$\frac{Q(k+\alpha_Q)-n-\alpha_Q-\beta_Q}{n+\alpha_Q+\beta_Q}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now if we make the &quot;prior weight&quot; $\alpha_Q+\beta_Q = \frac{N}{2}$  where $N$ is the length of the season (this will allow the &quot;miss-pricing&quot; to continue into the remainder of the season) and set the expected return to zero we get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\alpha_Q=\frac{2n+N}{2Q}-k$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(NOTE: unless this is the actual model, $\alpha_Q$ will depend on when this calculation was done, as it depends on $n,k,Q$ which will vary over time).  Now we are able to predict how $Q$ will be adjusted into the future, it will add $1$ to the denominator for each match, and $1$ to the numerator if the match was a draw.  So the expected odds after the first match are:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$(1+\frac{n+\beta_Q-k+1}{k+\alpha_Q})\frac{n-k+\beta_Q}{n+\alpha_Q+\beta_Q}+(1+\frac{n+\beta_Q-k}{k+\alpha_Q+1})\frac{k+\alpha_Q}{n+\alpha_Q+\beta_Q}$$&#xA;$$=1+\frac{n+\beta_Q-k}{k+\alpha_Q}\left(1+\frac{2}{(2n+N)(k+\alpha_Q+1)}\right)\approx 1+\frac{n+\beta_Q-k}{k+\alpha_Q}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That is the odds won't change much over the season.  Using this approximation, we get the expected return over the remainder of the season as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$(N-n)\frac{Q(k+1)-n-2}{n+2}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But remember that this is based on the overly simplistic model of a draw (note: this does not necessarily mean that it will be a &quot;crap&quot; predictor).  There can be no unique answer to your question, because there has been no specified model, and no specified prior information (e.g. how many people use this bookie? what is the bookie's turnover? how will my bets influence the odds they price?).  The only thing which has been specified is the data from one season, and that for &quot;some unspecified model&quot; the probabilities are inconsistent with those implied by the odds pricing.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastEditorUserId="2392" LastEditorDisplayName="" LastEditDate="2011-04-14T03:21:44.263" LastActivityDate="2011-04-14T03:21:44.263" />
  <row Id="9405" PostTypeId="1" CreationDate="2011-04-10T05:15:44.633" Score="0" ViewCount="134" Body="&lt;p&gt;I have two random poisson variables $x_1$ and $x_2$ with value 10 and 25 respectively. I am interested to use likelihood ratio test to test the null hypothesis: $\lambda_1=\lambda_2$, versus alernate hypthesis $\lambda_1$ not equal to $\lambda_2$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to use simulation to calculate power and alpha values. I would want to do it in R so any reference to R codes will be appreciated. Thanks in advance&lt;/p&gt;&#xA;" OwnerUserId="4098" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-10T12:18:59.777" LastActivityDate="2011-04-11T17:48:03.737" Title="Simulation of maximum likelihood ratio test to test two poisson random variables" Tags="&lt;r&gt;&lt;maximum-likelihood&gt;&lt;poisson&gt;&lt;power-analysis&gt;&lt;likelihood-ratio&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="1" />
  <row Id="9406" PostTypeId="2" ParentId="9398" CreationDate="2011-04-10T08:29:11.100" Score="5" ViewCount="" Body="&lt;p&gt;If I understand correctly, you have a two class classification problem, where the positive class (matches) is rare.  Many classifiers struggle with such a class imbalance, and it is common practice to sub-sample the majority class in order to obtain better performance, so the answer to the first question is &quot;yes&quot;.  However, if you sub-sample too much, you will end up with a classifier that over-predicts the minority positive class, so the best thing to do is to choose the sub-sampling ration to maximise performance, perhaps by minimising the cross-validation error &lt;i&gt;where the test data has not been sub-sampled&lt;/i&gt; so you get a good indication of operational performance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have a probabilistic classifier, that gives an estimate of the probabiility of class memebership, you can go one better and post-process the output to compensate for the difference between class frequencies in the training set and in operation.  I suspect that for some classifiers, the optimal approach is to optimise both the sub-sampling ratio and the correction to the output by optimising the cross-validation error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rather than sub-sampling, for some classifiers (e.g. SVMs) you can give different weights to positive and negative patterns.  I prefer this to sub-sampling as it means there is no variability in the results due to the particular sub-sample used.  Where this is not possible, use bootstrapping to make a bagged classifier, where a different sub-sample of the majority class is used in each iteration.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The one other thing I would say is that commonly where there is a large class imbalance, false negative errors and false positive error are not equally bad, and it is a good idea to build this into the classifier design (which can be accomplished by sub-sampling or weighting patterns belonging to each class).&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2011-04-10T08:29:11.100" CommentCount="2" />
  <row Id="9407" PostTypeId="1" AcceptedAnswerId="9408" CreationDate="2011-04-10T09:58:24.353" Score="2" ViewCount="277" Body="&lt;p&gt;I'm trying to compute the minimum sample size for a psychometric test based on 7 point Likert scales. I'd like to run ANOVA on each scale to look for differences between groups.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most online survey sample size calculators seem to be designed for polls, e.g. Yes/No, Agree/Disagree. They take as input population size, a confidence interval and a proportion (50% Yes/50% no) and then return the required sample size.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most statistical books suggest using power tests (such as R's power.t.test), which take as input a minimum effect size, alpha, beta and a statistical test and then return the required sample size.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For my purposes power tests seems to make the most sense, but what has me concerned is that none of them take into account the population size, which seems like it ought to have at least some effect on the outcome.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my question is, what is the correct calculation to use in my specific survey situation and more generally what is the connection between power tests and these online survey sample size calculators, does population size matter in some way, perhaps helping to capture the notion of representative sample?&lt;/p&gt;&#xA;" OwnerUserId="4099" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-11T05:55:35.603" LastActivityDate="2011-04-12T06:59:48.757" Title="Statistical power and minimum sample size for ANOVA with likert scale as dependent variable" Tags="&lt;anova&gt;&lt;likert&gt;&lt;power-analysis&gt;&lt;finite-population&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="9408" PostTypeId="2" ParentId="9407" CreationDate="2011-04-10T11:29:49.783" Score="3" ViewCount="" Body="&lt;p&gt;The commonly used statistical methods assume that you take a sample of an infinite or very large population. ANOVA, too, has this assumption. When the subjects of your survey can be viewed as a representative sample of an existing or hypothetical much larger population, you do not need the &lt;em&gt;finite population&lt;/em&gt; methods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second question is if ANOVA is appropriate to analyse the data collected. 7 point Likert scales are strictly speaking ordinal scales, so methods for ordinal dependent variables may be best. However, in psychometry it's usual to assume that the values from a Likert scale will follow a distribution that may be approximated with a normal distribution. In this case ANOVA is an acceptable method; the t-test too, although the latter compares two groups only. (The methods designed for binary (yes/no) outcomes may be used after setting a threshold in your Likert scale and dichotomising your data, however unless this threshold also exists in the psychological mechanism it will lead to loss of detail in your data and loss of power in your test. So not generally recommended.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You need to check or think over if the homoscedasticity assumption of ANOVA is likely to be met. If yes, use a power formula for ANOVA and you need not worry about not having to specify the population size.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastActivityDate="2011-04-10T11:29:49.783" CommentCount="3" />
  <row Id="9409" PostTypeId="2" ParentId="9405" CreationDate="2011-04-10T12:07:16.413" Score="2" ViewCount="" Body="&lt;p&gt;This is a particularly ill-formed question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If by &quot;alpha&quot; you mean Type I error, you need to go back to Square One and get definitions straight.  Type I error is not something inherent in the data, or even in the hypothesis; it's a subjectively and externally applied measure of risk.  And without the Type I error, you have no reference point from which to calculate Type II error, the complement of power.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Worse yet, it's not clear--after Adam's question--whether you have JUST TWO OBSERVATIONS (10 and 25), or two distributions with means of 10 and 25, and you're looking for a suitable sample size for a balanced test comparing the means.  In the first case, all you can do is a likelihood ratio test that gives an approximate p-value; there's no more information to be had in two observations.  In the second case, simulation can give some useful results, but you still need a value for the Type I error to get started.&lt;/p&gt;&#xA;" OwnerUserId="1920" LastActivityDate="2011-04-10T12:07:16.413" CommentCount="3" />
  <row Id="9410" PostTypeId="2" ParentId="9405" CreationDate="2011-04-10T12:09:13.697" Score="1" ViewCount="" Body="&lt;p&gt;For the simulation let's first choose sample sizes N1 and N2 for the two Poisson samples:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(lmtest)&#xA;N1 = 20; N2 = 15&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Generate a random sample and run a likelihood ratio test:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# CODE BLOCK &quot;A&quot;&#xA;x = c(rpois(N1, 10), rpois(N2, 25))&#xA;group = factor(c(rep('a', N1), rep('b', N2)))&#xA;m1 = glm(x ~ 1, family=poisson)&#xA;m2 = glm(x ~ group, family=poisson)&#xA;(t = lrtest(m1, m2))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Result:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Likelihood ratio test&#xA;&#xA;Model 1: x ~ 1&#xA;Model 2: x ~ group&#xA;  #Df  LogLik Df  Chisq Pr(&amp;gt;Chisq)    &#xA;1   1 -158.26                         &#xA;2   2  -93.39  1 129.75  &amp;lt; 2.2e-16 ***&#xA;---&#xA;Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now let's run many simulations to see the power for this particular N1 and N2:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;s = 1000 # 10*1000 simulations&#xA;sigs = NULL&#xA;for (i in 1:10) {&#xA;  sig = 0&#xA;  for (j in i:s) {&#xA;    CODE BLOCK &quot;A&quot; COMES HERE&#xA;    if (t$Pr[2] &amp;lt;= 0.05) sig = sig + 1&#xA;  }&#xA;  sigs = c(sigs, sig / s)&#xA;}&#xA;c(quantile(sigs, c(.025, .5, .975)), mean=mean(sigs), sd=sd(sigs))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Result:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;       2.5%         50%       97.5%        mean          sd &#xA;0.991225000 0.995500000 0.999775000 0.995500000 0.003027650&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Thus the power for N1 = 20; N2 = 15 is 99%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can calculate power for various N1 and N2 values.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-11T17:48:03.737" LastActivityDate="2011-04-11T17:48:03.737" />
  <row Id="9411" PostTypeId="2" ParentId="9400" CreationDate="2011-04-10T13:09:08.573" Score="2" ViewCount="" Body="&lt;p&gt;If I understand your question correctly, you should be able to achieve what you want to do using a non-linear mixed-effects model. If you use R, you can use the &lt;code&gt;nlme&lt;/code&gt; package. Basically as fixed factors you have a covariate (a) and a factor (substance or $i$ in $k_{i}$). You also have a random effect (individual measurements units or unitID). The good thing about &lt;code&gt;nlme&lt;/code&gt; is that it also allows you to model the correlations in the residuals with e.g. an AR covariance structure. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;edit: I always like to use a mixed-model when dealing with repeated measures. Still, if you don't want to include a random factor, you can model it with &lt;code&gt;gnls&lt;/code&gt; in the same package. &lt;code&gt;gnls&lt;/code&gt; still lets you select AR as covariance structure of he residuals. &lt;/p&gt;&#xA;" OwnerUserId="2020" LastEditorUserId="2020" LastEditorDisplayName="" LastEditDate="2011-04-10T17:25:16.590" LastActivityDate="2011-04-10T17:25:16.590" />
  <row Id="9412" PostTypeId="2" ParentId="9330" CreationDate="2011-04-10T13:13:06.367" Score="0" ViewCount="" Body="&lt;p&gt;I believe that this is an experiment where it is safe to assume a monotone relationship: for a longer exposition time the infection probability can not be smaller. So you can run monotone/isotonic regression.&#xA;You can even incorporate into your model that the infection probability at time=0 is 0.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-13T02:40:18.117" LastActivityDate="2011-04-13T02:40:18.117" />
  <row Id="9413" PostTypeId="2" ParentId="2715" CreationDate="2011-04-10T14:26:46.080" Score="3" ViewCount="" Body="&lt;p&gt;Think &lt;em&gt;hard&lt;/em&gt; about the underlying data generating process. If the model you want to use doesn't reflect the DGP, you need to find a new model.&lt;/p&gt;&#xA;" OwnerUserId="3265" LastActivityDate="2011-04-10T14:26:46.080" CommunityOwnedDate="2011-04-10T14:26:46.080" />
  <row Id="9414" PostTypeId="2" ParentId="2" CreationDate="2011-04-10T14:30:50.133" Score="2" ViewCount="" Body="&lt;p&gt;Other answers have covered what is normality and suggested normality test methods. Christian highlighted that in practice perfect normality barely exists.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I highlight that observed deviation from normality does not necessarily mean that methods assuming normality may not be used, and normality test may not be very useful.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Deviation from normality may be caused by outliers that are due to errors in data collection. In many cases checking the data collection logs you can correct these figures and normality often improves.&lt;/li&gt;&#xA;&lt;li&gt;For large samples a normality test will be able to detect a negligible deviation from normality.&lt;/li&gt;&#xA;&lt;li&gt;Methods assuming normality may be robust to non-normality and give results of acceptable accuracy. The t-test is known to be robust in this sense, while the F test is not &lt;sup&gt;&lt;a href=&quot;http://core.ecu.edu/psyc/wuenschk/stathelp/NormalityAssumption.htm&quot; rel=&quot;nofollow&quot;&gt;source&lt;/a&gt;(&lt;a href=&quot;http://www.webcitation.org/5xqHhDpB4&quot; rel=&quot;nofollow&quot;&gt;permalink&lt;/a&gt;)&lt;/sup&gt;. Concerning a specific method it's best to check the literature about robustness.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="3911" LastActivityDate="2011-04-10T14:30:50.133" />
  <row Id="9415" PostTypeId="1" AcceptedAnswerId="9418" CreationDate="2011-04-10T14:46:33.510" Score="4" ViewCount="132" Body="&lt;p&gt;Covariance between two random variables defines a measure of how closely are they linearly related to each other. But what if the joint distribution is circlular? Surely there is structure in the distribution. How is this structure extracted?&lt;/p&gt;&#xA;" OwnerUserId="4101" LastActivityDate="2011-04-11T18:37:15.167" Title="Measuring non-linear dependence" Tags="&lt;variance-covariance&gt;" AnswerCount="2" FavoriteCount="3" />
  <row Id="9416" PostTypeId="1" CreationDate="2011-04-10T14:55:57.757" Score="1" ViewCount="66" Body="&lt;p&gt;I have done an experiment to find the effective n/w bandwidth.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data I got in kbps is &lt;/p&gt;&#xA;&#xA;&lt;p&gt;223, 221, 510, 220, 471, 229, 222, 221, 220, 221 &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I find the effective bandwidth? Averaging gives 275.8. But if I have done only first 4 rounds then the average is 293.5. How can I find out a more reasonable value as the effective bandwidth. Or is averaging the correct way of doing this?&lt;/p&gt;&#xA;" OwnerUserId="4102" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-11T07:38:25.130" LastActivityDate="2011-04-11T07:51:46.153" Title="How to find the effective bandwidth correctly using statistics?" Tags="&lt;estimation&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="9417" PostTypeId="2" ParentId="9415" CreationDate="2011-04-10T16:30:36.543" Score="1" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Mutual_information&quot; rel=&quot;nofollow&quot;&gt;Mutual information&lt;/a&gt; has properties somewhat analogous to covariance. Covariance is a number which is 0 for independent variables and nonzero for variables which are linearly dependent. In particular, if two variables are the same, then the covariance is equal to variance (which is usually a positive number). One issue with covariance is that it may be zero even if two variables are not independent, provided the dependence is nonlinear. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Mutual information (MI) is a non-negative number. It is zero if and only if the two variables are statistically independent. This property is more general than that of covariance and covers any dependencies, including nonlinear ones. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the two variable are the same, MI is equal to the variable's entropy (again, usually a positive number). If the variables are different and not deterministically related, then MI is smaller than the entropy. In this sense, MI of two variables goes between 0 and H (the entropy), with 0 only if independent and H only if deterministically dependent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One difference from covariance is that the &quot;sign&quot; of dependency is ignored. E.g. $Cov(X, -X) = -Cov(X, X) = -Var(X)$, but $MI(X, -X) = MI(X, X) = H(X)$.&lt;/p&gt;&#xA;" OwnerUserId="3369" LastEditorUserId="3369" LastEditorDisplayName="" LastEditDate="2011-04-11T18:37:15.167" LastActivityDate="2011-04-11T18:37:15.167" CommentCount="1" />
  <row Id="9418" PostTypeId="2" ParentId="9415" CreationDate="2011-04-10T16:51:07.647" Score="5" ViewCount="" Body="&lt;p&gt;By &quot;circular&quot; I understand that the distribution is concentrated on a circular region, as in this contour plot of a pdf.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/qGDE6.png&quot; alt=&quot;Contour plot of a circular distribution&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If such a structure exists, even partially, a natural way to identify and measure it is to &lt;strong&gt;average the distribution circularly around its center&lt;/strong&gt;.  (Intuitively, this means that for each possible radius $r$ we should spread the probability of being at distance $r$ from the center equally around in all directions.)  Denoting the variables as $(X,Y)$, the center must be located at the point of first moments $(\mu_X, \mu_Y)$.  To do the averaging it is convenient to define the radial distribution function&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$F(\rho) = \Pr[(X-\mu_X)^2 + (Y-\mu_Y)^2 \le \rho^2], \rho \ge 0;$$&#xA;$$F(\rho) = 0, \rho \lt 0.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This captures the total probability of lying between distance $0$ and $\rho$ of the center.  To spread it out in all directions, let $R$ be a random variable with cdf $F$ and $\Theta$ be a uniform random variable on $[0, 2\pi]$ independent of $R$.  The bivariate random variable $(\Xi, H) = (R\cos(\Theta) + \mu_X, R\sin(\Theta)+\mu_Y)$ is the &lt;em&gt;circular average&lt;/em&gt; of $(X,Y)$.  (This does the job our intuition demands of a &quot;circular average&quot; because (a) it has the correct radial distribution, namely $F$, by construction, and (b) all directions from the center ($\Theta$) are equally probable.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At this point you have many choices: all that remains is to compare the distribution of $(X,Y)$ to that of $(\Xi, H)$.  Possibilities include an &lt;a href=&quot;http://en.wikipedia.org/wiki/Convergence_of_random_variables&quot; rel=&quot;nofollow&quot;&gt;$L^p$ distance&lt;/a&gt; and the &lt;a href=&quot;http://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence&quot; rel=&quot;nofollow&quot;&gt;Kullback-Leibler divergence&lt;/a&gt; (along with myriad related distance measures: symmetrized divergence, Hellinger distance, mutual information, &lt;em&gt;etc.&lt;/em&gt;).  The comparison suggests $(X,Y)$ may have a circular structure when it is &quot;close&quot; to $(\Xi, H)$.  In this case the structure can be &quot;extracted&quot; from properties of $F$.  For instance, a measure of central location of $F$, such as its mean or median, identifies the &quot;radius&quot; of the distribution of $(X,Y)$, and the standard deviation (or other measure of scale) of $F$ expresses how &quot;spread out&quot; $(X,Y)$ are in the radial directions about their central location $(\mu_X, \mu_Y)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When &lt;strong&gt;sampling&lt;/strong&gt; from a distribution, with data $(x_i,y_i), 1 \le i \le n$, a reasonable test of circularity is to estimate the central location as usual (with means or medians) and thence convert each value $(x_i,y_i)$ into polar coordinates $(r_i, \theta_i)$ relative to that estimated center.  Compare the standard deviation (or IQR) of the radii to their mean (or median).  For non-circular distributions the ratio will be large; for circular distributions it should be relatively small.  (If you have a specific model in mind for the underlying distribution, you can work out the sampling distribution of the radial statistic and construct a significance test with it.)  Separately, test the angular coordinate for uniformity in the interval $[0, 2\pi)$.  It will be approximately uniform for circular distributions (and for some other distributions, too); non-uniformity indicates a departure from circularity.  &lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-04-11T14:20:58.970" LastActivityDate="2011-04-11T14:20:58.970" CommentCount="5" />
  <row Id="9420" PostTypeId="1" CreationDate="2011-04-10T17:20:13.737" Score="1" ViewCount="113" Body="&lt;p&gt;Contingency tables tables are typically formatted as tables similar to matrices in mathematics, see &lt;a href=&quot;http://en.wikipedia.org/wiki/Contingency_table#Example&quot; rel=&quot;nofollow&quot;&gt;this example&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is the equation below an accepted notation of expressing the probabilities of the outcomes as a matrix? If not, &lt;strong&gt;what would be the accepted way?&lt;/strong&gt; Are there any published materials using the same notation?&#xA;$$&#xD;&#xA;\widehat{Pr_\text{outcome}} = &#xD;&#xA;&#xD;&#xA;\begin{matrix}&#xD;&#xA; &amp;amp;&#xD;&#xA;\begin{matrix}&#xD;&#xA;\text{RH} &amp;amp; \text{LH}&#xD;&#xA;\end{matrix}&#xD;&#xA;\\&#xD;&#xA;\begin{matrix}&#xD;&#xA;\text{male} \\ \text{female}&#xD;&#xA;\end{matrix}&#xD;&#xA;&amp;amp;&#xD;&#xA;\begin{bmatrix}&#xD;&#xA;43 &amp;amp; 9 \\&#xD;&#xA;44 &amp;amp; 4&#xD;&#xA;\end{bmatrix}&#xD;&#xA;\end{matrix}&#xD;&#xA;\cdot \frac{1}{100}&#xD;&#xA;=&#xD;&#xA;\begin{matrix}&#xD;&#xA; &amp;amp;&#xD;&#xA;\begin{matrix}&#xD;&#xA;\text{RH} &amp;amp; \text{LH}&#xD;&#xA;\end{matrix}&#xD;&#xA;\\&#xD;&#xA;\begin{matrix}&#xD;&#xA;\text{male} \\ \text{female}&#xD;&#xA;\end{matrix}&#xD;&#xA;&amp;amp;&#xD;&#xA;\begin{bmatrix}&#xD;&#xA;0.43 &amp;amp; 0.09 \\&#xD;&#xA;0.44 &amp;amp; 0.04&#xD;&#xA;\end{bmatrix}&#xD;&#xA;\end{matrix}&#xD;&#xA;$$&lt;/p&gt;&#xA;" OwnerUserId="3911" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-13T01:14:23.023" LastActivityDate="2011-04-13T01:14:23.023" Title="Notation of probability matrix corresponding to a contingency table" Tags="&lt;contingency-tables&gt;&lt;matrix&gt;&lt;notation&gt;" AnswerCount="2" CommentCount="3" FavoriteCount="1" />
  <row Id="9421" PostTypeId="2" ParentId="9420" CreationDate="2011-04-10T19:42:39.777" Score="1" ViewCount="" Body="&lt;p&gt;It looks reasonable to me. Though realize that with an N larger than say 1,000, your table of probabilities won't represent exact counts because you'll end up truncating to three or so decimal places.&lt;/p&gt;&#xA;" OwnerUserId="1499" LastActivityDate="2011-04-10T19:42:39.777" CommentCount="1" />
  <row Id="9422" PostTypeId="1" AcceptedAnswerId="9443" CreationDate="2011-04-11T01:00:28.697" Score="5" ViewCount="100" Body="&lt;p&gt;Suppose I have survey responses that look like this: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;N=60000, Population&#xA;n=1000, Total sample&#xA;n=800, Users of Company X&#xA;n=200, Randomly chosen from 800 and asked about their Future Use of Company X &#xA;n=100, Planning to use Company X less in the future&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The reason that only 200 of 800 users were asked about future use was due to them being asked about other companies as well. The survey would be far too long if they were asked about their future use of all companies that they use. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My goal is to understand the flow of future use. For example, of those individuals who are using Company X less in the future, which other companies are they planning on using more. However, with a sample of 100, there are not enough responses to other companies from the same users to other companies to get a usable proportion. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can I, with some level of accuracy, infer the flow of business from a more robust sample of users of Company X? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Update: I think, what I may be referring to is called bootstrapping.&lt;/p&gt;&#xA;" OwnerUserId="776" LastEditorUserId="776" LastEditorDisplayName="" LastEditDate="2011-04-13T04:57:11.190" LastActivityDate="2011-04-13T04:57:11.190" Title="Resampling within a survey to account for missing data" Tags="&lt;sampling&gt;&lt;inference&gt;&lt;resampling&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="9423" PostTypeId="2" ParentId="7019" CreationDate="2011-04-11T01:13:15.487" Score="3" ViewCount="" Body="&lt;p&gt;If you really only have one data point greater than 1000, it would be easiest just to delete that point from the graph. You can make a note in the caption or as a text box that there is an outlier.&lt;/p&gt;&#xA;" OwnerUserId="1569" LastActivityDate="2011-04-11T01:13:15.487" CommentCount="2" />
  <row Id="9424" PostTypeId="2" ParentId="9416" CreationDate="2011-04-11T07:51:46.153" Score="2" ViewCount="" Body="&lt;p&gt;From what I can understand, I think you have the following options: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Sample more! n = 10 is hardly enough for drawing conclusions&lt;/li&gt;&#xA;&lt;li&gt;If you don't/can't do &quot;enough&quot; sampling, you can always try to do some Monte Carlo type study with bootstrapping&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="3014" LastActivityDate="2011-04-11T07:51:46.153" CommentCount="2" />
  <row Id="9425" PostTypeId="1" CreationDate="2011-04-11T12:08:13.933" Score="8" ViewCount="150" Body="&lt;p&gt;Reading Field's Discovering Statistics Using SPSS 3rd Edition I was struck by a bit about post hoc tests in ANOVA. For those wanting to control the Type I error rate he suggests Bonferroni or Tukey and says&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&quot;Bonferroni has more power when the&#xA;  number of comparisons is small,&#xA;  whereas Tukey is more powerful when&#xA;  testing large numbers of means.&quot; P374&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Where should the line be drawn between a small and large number of means?&lt;/p&gt;&#xA;" OwnerUserId="3597" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-11T14:55:36.100" LastActivityDate="2011-05-11T20:10:56.823" Title="Bonferroni or Tukey? When does the number of group means become large?" Tags="&lt;anova&gt;&lt;multiple-comparisons&gt;" AnswerCount="0" CommentCount="3" FavoriteCount="2" />
  <row Id="9426" PostTypeId="2" ParentId="9420" CreationDate="2011-04-11T12:23:03.697" Score="1" ViewCount="" Body="&lt;p&gt;I'm not sure I can justify what I'm about to say, but I would be uneasy about expressing probabilities in a form like this. The structure is too reminiscent of other things that don't properly apply and suggests you should be able to do stuff like matrix multiplication that wouldn't mean anything here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the goal is to express a set of 4 mutually-exclusive outcomes, then your probabilities should be a single vector with 4 entries adding to 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;OTOH, if it is to express a structure within that -- the difference in the handedness distribution by sex -- then it would make more sense if each of the rows added to 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And if what you actually want is a contingency table, use a contingency table. Dividing through by the total count doesn't really gain you anything.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using a convention whereby all entries in a matrix add to 1 seems to me, well, unconventional. But I admit this objection is a bit subjective and hand-wavy.&lt;/p&gt;&#xA;" OwnerUserId="174" LastActivityDate="2011-04-11T12:23:03.697" CommentCount="6" />
  <row Id="9427" PostTypeId="1" AcceptedAnswerId="9428" CreationDate="2011-04-11T12:58:01.893" Score="0" ViewCount="91" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;Possible Duplicate:&lt;/strong&gt;&lt;br&gt;&#xA;  &lt;a href=&quot;http://stats.stackexchange.com/questions/4220/probability-distribution-value-exceeding-1-is-ok&quot;&gt;Probability distribution value exceeding 1 is OK?&lt;/a&gt;  &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&#xA;&#xA;&lt;p&gt;I'm a bit confused how I am getting probabilities greater than 1 when calculating p(x | mu, sigma) when x = mu. For example, if I run: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt; gaussProb(0, 0, 0.1)&#xA;ans =&#xA;    1.2616&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where gaussProb is a matlab function from the PMTK toolbox:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;function p = gaussProb(X, mu, Sigma)&#xA;% Multivariate Gaussian distribution, pdf&#xA;% X(i,:) is i'th case&#xA;% *** In the univariate case, Sigma is the variance, not the standard&#xA;% deviation! ***&#xA;&#xA;% This file is from pmtk3.googlecode.com&#xA;&#xA;&#xA;d = size(Sigma, 2);&#xA;X  = reshape(X, [], d);  % make sure X is n-by-d and not d-by-n&#xA;X = bsxfun(@minus, X, rowvec(mu));&#xA;logp = -0.5*sum((X/(Sigma)).*X, 2); &#xA;logZ = (d/2)*log(2*pi) + 0.5*logdet(Sigma);&#xA;logp = logp - logZ;&#xA;p = exp(logp);        &#xA;&#xA;end&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is this some fundamental property of the Gaussian distribution or an issue with numerical accuracy in the computation? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've come across this issue by trying to weight samples from a Gaussian distribution obtained from a Gaussian process prediction, where I will get massive probabilities. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="4108" LastActivityDate="2011-04-11T13:01:11.077" ClosedDate="2011-04-11T15:04:34.483" Title="Interpreting Gaussian probabilities greater than 1" Tags="&lt;normal-distribution&gt;&lt;matlab&gt;" AnswerCount="1" CommentCount="0" />
  <row Id="9428" PostTypeId="2" ParentId="9427" CreationDate="2011-04-11T13:01:11.077" Score="5" ViewCount="" Body="&lt;p&gt;The code in the question returns the values of &lt;a href=&quot;http://en.wikipedia.org/wiki/Probability_density_function&quot; rel=&quot;nofollow&quot;&gt;probability density function&lt;/a&gt;. The values of probability density function can be greater than one. The actual probability $P(X&amp;lt;x)$ for random variable $X$ with probability density function $p(x)$ is integral $\int_{-\infty}^xp(t)dt$. The values of this integral are of course restricted to interval $[0,1]$.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-04-11T13:01:11.077" CommentCount="1" />
  <row Id="9429" PostTypeId="1" CreationDate="2011-04-11T13:42:47.047" Score="5" ViewCount="101" Body="&lt;p&gt;I am trying to fit a multilevel longitudinal model and i have a question regarding how to specify it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data consist of about 8k observations collected from about 3k individuals at four time points. Individuals are nested in groups and there are about 200 groups. &#xA;I have two different types of fixed effects: (a) repeated measures at the observation level (e.g. pred1.obs ), and (b) group level predictors that also change over time (e.g. pred2.grp). &#xA;Because each group level fixed effect is also longitudinal there are 800 values (4x200 which are repeated for each member of the group at that time) but there are only 200 groups. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is what would be the correct specification for this model and why? &#xA;e.g:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;1: lmer(outcome ~ time + pred1.obs + pred2.grp + (time|id) + (time|grp))&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;2: lmer(outcome ~ time + pred1.obs + pred2.grp + (time|id) + (1|grp:time))&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;3: lmer(outcome ~ time + pred1.obs + pred2.grp + (time|id) + (time|grp) + (1|grp:time))&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Thus, would lme4 correctly estimate the model if i use (time|grp) or do i need to use (1|grp:time) or the combination? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or something else that i haven't thought of? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks, &#xA;George&lt;/p&gt;&#xA;" OwnerUserId="1871" LastActivityDate="2011-05-06T15:27:14.023" Title="Correct specification of longitudinal model in lme4" Tags="&lt;r&gt;&lt;multilevel-analysis&gt;&lt;longitudinal&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="3" />
  <row Id="9430" PostTypeId="2" ParentId="9385" CreationDate="2011-04-11T13:45:02.443" Score="2" ViewCount="" Body="&lt;p&gt;I am not very familiar with Holt-Winters, however I have this &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/3540719164&quot; rel=&quot;nofollow&quot;&gt;excellent book&lt;/a&gt; by @Rob Hyndman. The package &lt;strong&gt;forecast&lt;/strong&gt; (which is based on the book) of statistical package R gives the following result on your data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; hw&amp;lt;-read.table(&quot;~/R/stackoverflow/hw.txt&quot;)&#xA;&amp;gt; tt&amp;lt;-ts(hw[,3],start=c(1999,1),freq=12)&#xA;&#xA;&amp;gt; aa&amp;lt;-forecast(tt)&#xA;&amp;gt; plot(aa)&#xA;&amp;gt; summary(aa)&#xA;&#xA;Forecast method: ETS(M,N,A)&#xA;&#xA;Model Information:&#xA;ETS(M,N,A) &#xA;&#xA;Call:&#xA; ets(y = object) &#xA;&#xA;  Smoothing parameters:&#xA;    alpha = 0.1701 &#xA;    gamma = 1e-04 &#xA;&#xA;  Initial states:&#xA;    l = 870.4847 &#xA;    s = -278.0815 -143.6584 151.959 -135.595 514.2527 236.9216&#xA;           -32.7679 128.8337 115.0829 47.5922 -234.4105 -370.1288&#xA;&#xA;  sigma:  0.1122&#xA;&#xA;     AIC     AICc      BIC &#xA;1892.756 1896.346 1933.115 &#xA;&#xA;In-sample error measures:&#xA;         ME        RMSE         MAE         MPE        MAPE        MASE &#xA; 18.1543007 121.8594668  70.7086492   0.8480306   7.0006920   0.2893504 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here is the graph of the forecast together with the confidence intervals:&#xA;&lt;img src=&quot;http://i.imgur.com/h4AUL.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that the function &lt;strong&gt;forecast&lt;/strong&gt; picks automatically the best exponential smoothing model from 30 models which are classified by the type of trend model, seasonal part model and the additivity or multiplicity of error. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The best model found in your data is with multiplicative error, no trend and additive seasonality, which is less complicated model than you are trying to fit. The way function &lt;strong&gt;forecast&lt;/strong&gt; works is however that the more complicated model was considered and rejected in favor the final model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you provide the exact formulas it would be possible to fit the precise model to see whether the problem you described is really property of the model.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-04-11T13:51:21.390" LastActivityDate="2011-04-11T13:51:21.390" />
  <row Id="9431" PostTypeId="1" AcceptedAnswerId="9528" CreationDate="2011-04-11T14:03:13.367" Score="6" ViewCount="94" Body="&lt;p&gt;I'm running a binary logit regression where I know the dependent variable is miscoded in a small percentage of cases.  So I'm trying to estimate $\beta$ in this model:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$prob(y_i) = 1/(1 + e^{-z_i})$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$z_i = \alpha + X_i\beta$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But instead of the vector $Y$, I have $\tilde{Y}$, which includes some random errors (i.e. $y_i = 1$, but $\tilde{y_i} = 0$, or vice versa, for some $i$).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a (reasonably) simple correction for this problem?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know that logit has some nice properties in case-control studies. It seems likely that something similar applies here, but I haven't been able to find a good solution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A few other constraints: this is a text-mining application, so the dimensions of $X$ are large (in the thousands or tens of thousands). This may rule out some computationally intensive procedures.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, I don't care about correctly estimating $\alpha$, only $\beta$.&lt;/p&gt;&#xA;" OwnerUserId="4110" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-11T14:38:25.397" LastActivityDate="2011-04-13T18:13:24.987" Title="How can I correct for measurement error in the dependent variable in a logit regression?" Tags="&lt;logistic-regression&gt;&lt;measurement-error&gt;&lt;misclassification-error&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="9432" PostTypeId="2" ParentId="1610" CreationDate="2011-04-11T14:31:06.300" Score="0" ViewCount="" Body="&lt;p&gt;This is how I remember the difference between Type I and Type II errors&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Type I is a false POSITIVE&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Type II is a false NEGATIVE&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Type I is so POSITIVE it jumps out of bed first, runs downstairs and finds a significant breakfast while Type II is so NEGATIVE it stays in bed all day so when it eventually crawls out all the food is gone. It can never find anything!&lt;/p&gt;&#xA;" OwnerUserId="3597" LastActivityDate="2011-04-11T14:31:06.300" />
  <row Id="9434" PostTypeId="2" ParentId="9431" CreationDate="2011-04-11T14:41:17.657" Score="2" ViewCount="" Body="&lt;p&gt;This situation is often referred to as misclassification error. &lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pubmed/20552681&quot; rel=&quot;nofollow&quot;&gt;This paper&lt;/a&gt; my help you correctly estimating $\beta$. EDIT: I found relevant-looking papers using &lt;a href=&quot;http://www.google.com/search?q=misclassification+of+dependent+variable+logistic&quot; rel=&quot;nofollow&quot;&gt;http://www.google.com/search?q=misclassification+of+dependent+variable+logistic&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-11T15:34:28.983" LastActivityDate="2011-04-11T15:34:28.983" CommentCount="2" />
  <row Id="9435" PostTypeId="1" CreationDate="2011-04-11T14:46:09.620" Score="4" ViewCount="93" Body="&lt;p&gt;I'm trying to create something similar to this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/QQVtd.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, 3 different Node classes, and a whole bunch of relationships between them. In my case, there should be roughly half of the number of nodes present at most.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I'm looking for is recommendations as to the best way to create a similar type of graph. Spent some time looking at R/GGplot2, but haven't found any solutions so far. I expect it's because I'm not using the correct vocabulary.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The posted image was created using a proprietary app that I unfortunately am not able to leverage, otherwise I'd simply use that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestions/solutions would be fantastic! Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="4112" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-11T14:59:00.107" LastActivityDate="2011-05-04T15:51:07.560" Title="Displaying relationships between nodes" Tags="&lt;data-visualization&gt;" AnswerCount="3" CommentCount="4" />
  <row Id="9436" PostTypeId="2" ParentId="9429" CreationDate="2011-04-11T15:09:01.237" Score="3" ViewCount="" Body="&lt;p&gt;You have a large number of groups, so I speculate that (depending on the setting) you may think about group as a random effect, so your &lt;code&gt;(…|grp)&lt;/code&gt; terms are probably justified. It may also be reasonable to associate random effects with individuals (&lt;code&gt;(…|id)&lt;/code&gt; terms). However you have &lt;code&gt;time&lt;/code&gt; as a covariate in all your models, so I assume you are looking for a linear time effect. In the same time having the &lt;code&gt;(1|grp:time))&lt;/code&gt; term or any kind of &lt;code&gt;(…|time))&lt;/code&gt; terms in the model makes the interpretation of the time covariate difficult.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;code&gt;(1|…)&lt;/code&gt; terms correspond to random intercepts, e.g. including &lt;code&gt;(1|grp)&lt;/code&gt; would estimate a “mean” outcome at time=0 for each group expressed as deviation from the grand intercept; &lt;code&gt;(1|id)&lt;/code&gt; would estimate an individual intercept.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;code&gt;(time|…)&lt;/code&gt; terms correspond to random slopes of the time covariate, e.g. including &lt;code&gt;(time|grp)&lt;/code&gt; would estimate a deviation from the grand slope for each group; &lt;code&gt;(time|id)&lt;/code&gt; would estimate individual slopes. [Edit: Please note the difference between the &lt;code&gt;(time|id)&lt;/code&gt; and &lt;code&gt;(0 + time|id)&lt;/code&gt; expressions, see &lt;code&gt;?lmer&lt;/code&gt;]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your model specifications you are using both random intercepts and random slopes. Whether you need them or which you need depends on the relationships among the variables you study. If you know the relationship, you should specify the corresponding model. Alternatively you can fit multiple models and explain the differences among the results.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-11T17:57:45.803" LastActivityDate="2011-04-11T17:57:45.803" CommentCount="0" />
  <row Id="9437" PostTypeId="1" CreationDate="2011-04-11T15:10:56.120" Score="1" ViewCount="52" Body="&lt;p&gt;I am quite a newbie in this area: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What are the boosting methods for regression systems? I know about &lt;a href=&quot;http://en.wikipedia.org/wiki/Gradient_boosting&quot; rel=&quot;nofollow&quot;&gt;Gradient boosting&lt;/a&gt;; are there any other approaches? &lt;/li&gt;&#xA;&lt;li&gt;Are there textbooks or tutorials devoted to this area?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="976" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-12T04:46:37.933" LastActivityDate="2011-04-12T06:49:35.103" Title="Boosting for regression systems" Tags="&lt;regression&gt;&lt;boosting&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="9438" PostTypeId="2" ParentId="9276" CreationDate="2011-04-11T16:37:21.353" Score="0" ViewCount="" Body="&lt;p&gt;It seems to me that the problem here is whether the model is too complex to look out without using Monte Carlo simulation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the model is all relatively simple then it should be possible to look at it through conventioanl statistics and derive a solution to the question being asked, without re-running the model multiple times. This is a bit of an over simplication, but if all your model did was produce points based on a normal distribution, then you could easily derive the sort of answers you are looking for. Of course, if the model is this simple then you are unlikely to need to do a Monte Carlo simulation find your answers. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the problem is complex and it is not possible to break it down to more elementary, the Monte-Carlo is the right type of model to use, but I don't think there is any way of defining confidence limits without runing the model. Ultimately to get the type of confidence limits described he model would have to be run a number of times, a probability distribution would have to be fit to the outputs and from there the confidnce limits could be defined. One of the challenges with Monte-Carlo simulation is that models give good and regular answers for distributions in the mid range but the tails often give much more variable results, which ultimately means more runs to define the shape of the outputs at the 2.5% and 97.5% percentiles. &lt;/p&gt;&#xA;" OwnerUserId="210" LastActivityDate="2011-04-11T16:37:21.353" />
  <row Id="9439" PostTypeId="2" ParentId="9435" CreationDate="2011-04-11T17:01:03.527" Score="0" ViewCount="" Body="&lt;p&gt;With Graphviz you can easily make graphs like this: &lt;a href=&quot;http://www.graphviz.org/content/unix&quot; rel=&quot;nofollow&quot;&gt;http://www.graphviz.org/content/unix&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="3911" LastActivityDate="2011-04-11T17:01:03.527" CommentCount="2" />
  <row Id="9441" PostTypeId="2" ParentId="8898" CreationDate="2011-04-11T17:33:41.190" Score="2" ViewCount="" Body="&lt;p&gt;The ROC curve (Receiver Operating Characteristics) is one of the techniques available. You can check the questions with the tag roc on this site for further details. The wikipedia article&#xA; &lt;a href=&quot;http://en.wikipedia.org/wiki/Receiver_operating_characteristic&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Receiver_operating_characteristic&lt;/a&gt; and the external links to it may also be useful.&#xA;Some other methods can be found here &#xA;&lt;a href=&quot;http://onbiostatistics.blogspot.com/2011/01/agreement-statistics-and-kappa.html&quot; rel=&quot;nofollow&quot;&gt;http://onbiostatistics.blogspot.com/2011/01/agreement-statistics-and-kappa.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="4116" LastEditorUserId="4116" LastEditorDisplayName="" LastEditDate="2011-04-11T18:21:18.727" LastActivityDate="2011-04-11T18:21:18.727" />
  <row Id="9442" PostTypeId="2" ParentId="9190" CreationDate="2011-04-11T18:45:10.307" Score="1" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://www.edwardtufte.com/tufte/books_visex&quot; rel=&quot;nofollow&quot;&gt;Visual explanations&lt;/a&gt; or anything else by Tufte is inspirational.&lt;/p&gt;&#xA;" OwnerUserId="2817" LastActivityDate="2011-04-11T18:45:10.307" CommunityOwnedDate="2011-04-11T18:45:10.307" />
  <row Id="9443" PostTypeId="2" ParentId="9422" CreationDate="2011-04-11T18:52:07.253" Score="2" ViewCount="" Body="&lt;p&gt;Your question is above my pay grade, as it were, but I can suggest a first look at &lt;a href=&quot;http://faculty.washington.edu/tlumley/survey/&quot; rel=&quot;nofollow&quot;&gt;the R survey package&lt;/a&gt;, which might implement some of the routines that you'd use to answer your questions.&lt;/p&gt;&#xA;" OwnerUserId="1764" LastActivityDate="2011-04-11T18:52:07.253" CommentCount="2" />
  <row Id="9444" PostTypeId="2" ParentId="9220" CreationDate="2011-04-11T19:19:33.157" Score="0" ViewCount="" Body="&lt;p&gt;Why not test it out? &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(347)&#xA;x &amp;lt;- rnorm(10000)&#xA;y &amp;lt;- rnorm(10000)&#xA;x2 &amp;lt;- rnorm(10000)&#xA;y2 &amp;lt;- rnorm(10000)&#xA;&#xA;qdf &amp;lt;- data.frame(x,y,x2,y2)&#xA;qdf &amp;lt;- data.frame(qdf,(x-x2)^2+(y-y2)^2)&#xA;colnames(qdf)[5] &amp;lt;- &quot;euclid&quot; &#xA;&#xA;plot(c(x,y),c(x2,y2))&#xA;plot(qdf$euclid)&#xA;hist(qdf$euclid) &#xA;plot(dentist(qdf$euclid))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/2dVob.jpg&quot; alt=&quot;Plot 1&quot;&gt;&#xA;&lt;img src=&quot;http://i.imgur.com/T3u0X.jpg&quot; alt=&quot;Plot 2&quot;&gt;&#xA;&lt;img src=&quot;http://i.imgur.com/KjZpf.jpg&quot; alt=&quot;Plot 3&quot;&gt;&#xA;&lt;img src=&quot;http://i.imgur.com/XlMoe.jpg&quot; alt=&quot;Plot 4&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="776" LastActivityDate="2011-04-11T19:19:33.157" CommentCount="4" />
  <row Id="9446" PostTypeId="1" CreationDate="2011-04-11T19:37:39.740" Score="1" ViewCount="170" Body="&lt;h3&gt;Context:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;My research aims to assess whether parents who have been involved in a relationship with a social worker are at higher risk of child abuse/neglect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am trying to establish whether there exists a relationship between social workers' attitude toward the parents that reflect an overall outcome of the case (positive or negative). &#xA;I want to assess whether social worker attitudes have an effect on parental responses.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The questionnaire was only given to parents who have a relationship with a social worker. &lt;/li&gt;&#xA;&lt;li&gt;There were 20 participants (16 females and 4 males).&lt;/li&gt;&#xA;&lt;li&gt;I aggregated my 13 Likert items/questions into a single attitude (word) per each. &lt;/li&gt;&#xA;&lt;li&gt;I also reversed the order of the Likert scale when I entered the data, so that a score of 5 stands for &quot;strongly disagree&quot;, whereas a score of 1 reads &quot;strongly agree&quot;. This was done because of the way my Likert items were asked.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;Question:&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Can I run a test between genders given that there are an uneven number of males and females?&lt;/li&gt;&#xA;&lt;li&gt;What analyses can I run to test my research question?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="4119" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-12T04:28:22.427" LastActivityDate="2011-04-12T04:28:22.427" Title="How to carry out a Likert scale analysis?" Tags="&lt;homework&gt;&lt;psychometrics&gt;&lt;scales&gt;&lt;likert&gt;" CommentCount="4" FavoriteCount="1" />
  <row Id="9447" PostTypeId="1" CreationDate="2011-04-11T20:01:06.680" Score="5" ViewCount="65" Body="&lt;p&gt;I would like to predict the average number of days in a year for which two conditions are true:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;daily average temperature is below zero celsius&lt;/li&gt;&#xA;&lt;li&gt;the day was preceded by at least four days with daily average temperature below zero celsius&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I've historical daily average temperature data for the location available for about 10 years. My initial approach was to use the &lt;a href=&quot;http://en.wikipedia.org/wiki/Chebyshev_inequality#Variant%3a_One-sided_Chebyshev_inequality&quot; rel=&quot;nofollow&quot; rel=&quot;nofollow&quot;&gt;one sided Chebyshev inequality&lt;/a&gt;&#xA; which can be used to approximate a probability if the distribution is not known. However in this application I am interested in the probability of a special condition, can I use the Chebyshev inequality for a dummy time series as well? I.e.: 1 if condition is fullfilled, otherwise 0, --&gt; the dataset would therefore look something like 0,0,0,0,0,1,1,0,0,0,0,0,1,1,1,1,1,1,etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How would you approach a problem like that from a different angle, the data clearly has seasonality, is there any distribution which I could use to have a better estimate than Chebyshev?&lt;/p&gt;&#xA;" OwnerUserId="4120" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-12T04:32:44.390" LastActivityDate="2011-04-12T04:32:44.390" Title="Estimating event probability from historical time series with clear seasonality" Tags="&lt;time-series&gt;&lt;predictive-models&gt;&lt;seasonality&gt;" AnswerCount="2" CommentCount="3" FavoriteCount="1" />
  <row Id="9448" PostTypeId="2" ParentId="9447" CreationDate="2011-04-11T20:28:41.743" Score="2" ViewCount="" Body="&lt;p&gt;I think the joint distribution of temperature data on successive days could be reasonably modelled using a multi-variate Gaussian (Gaussian distributions are often used in statistical downscaling of temperature).  What I would try would be to regress the mean and covariance matrix of the temperature time series on sine and cosine components of the day of year (to deal with the seasonality).  The details on how to do that are given in a paper by &lt;a href=&quot;http://dx.doi.org/10.1162/neco.1996.8.4.843&quot; rel=&quot;nofollow&quot;&gt;Peter Williams&lt;/a&gt;, Williams uses a neural network, but I would start off with just a linear model.  This will give you what climatologists would call a &quot;weather generator&quot; (of sorts).  Using this you could generate as many synthetic time series as you want with the appropriate statistical properties, from which you could estimate the probabilities you require directly.  You would need to estimate the window over which temperatures were usefully correllated - which may be quite high in winter due to blocking patterns (for the U.K. anyway).  A bit baroque I suppose, but it would be the thing I would try!&lt;/p&gt;&#xA;" OwnerUserId="887" LastEditorUserId="887" LastEditorDisplayName="" LastEditDate="2011-04-11T20:35:07.073" LastActivityDate="2011-04-11T20:35:07.073" CommentCount="3" />
  <row Id="9449" PostTypeId="1" AcceptedAnswerId="9451" CreationDate="2011-04-11T20:33:14.453" Score="4" ViewCount="102" Body="&lt;p&gt;How do you compute confidence intervals for positive predictive value?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The standard error is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$SE = \sqrt{ \frac{PPV(1-PPV)}{TP+FP}} $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is that right? (here my concern is the denominator)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does that formula work for any similar ratio in a 2x2 table. E.g. for sensitivity, it would be &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$SE = \sqrt{ \frac{SENS(1-SENS)}{FP+TN}} $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is that right? (here my concern is that it is generalizable to other ratios as long as you get the denominator right)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And the for the 95% confidence intervals:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$CI_{PPV} = PPV \pm 1.96*SE$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is that right? (my concern here is how to go from SE to the confidence interval)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(of course with all the cell restrictions like $n\cdot p\cdot (1-p) \ge 5$)&lt;/p&gt;&#xA;" OwnerUserId="3186" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-12T02:45:56.907" LastActivityDate="2011-04-12T02:45:56.907" Title="How do you compute confidence intervals for positive predictive value?" Tags="&lt;confidence-interval&gt;&lt;binomial&gt;&lt;contingency-tables&gt;" AnswerCount="1" CommentCount="3" FavoriteCount="1" />
  <row Id="9450" PostTypeId="2" ParentId="9392" CreationDate="2011-04-11T22:43:54.183" Score="1" ViewCount="" Body="&lt;p&gt;As I understand (see comments to the original question) you want to select a subset of the features by two criteria:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;the subset covers most of the information content of the dataset,&lt;/li&gt;&#xA;&lt;li&gt;the subset includes as few features as possible.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The paper &lt;a href=&quot;http://j.mp/gYJ4sn&quot; rel=&quot;nofollow&quot;&gt;&lt;em&gt;Variable selection in large environmental data sets using principal components analysis&lt;/em&gt; by King and Jackson in Environmetrics, 1999&lt;/a&gt; compares the methods for this problem.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-13T02:59:16.640" LastActivityDate="2011-04-13T02:59:16.640" />
  <row Id="9451" PostTypeId="2" ParentId="9449" CreationDate="2011-04-11T23:22:22.870" Score="4" ViewCount="" Body="&lt;p&gt;Your first SE formula is correct. The second SE formula which concerns sensitivity should have the total number of positive cases in the denominator:&#xA;  $$SE_\text{sensitivity} = \sqrt{ \frac{SENS(1-SENS)}{TP+FN}} $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The logic is that sensitivity = $\frac{TP}{TP+FN}$, and the denominator in the SE formula is the same.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As @onestop pointed out in their comment &lt;a href=&quot;http://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval&quot; rel=&quot;nofollow&quot;&gt;methods of calculating a binomial proportion confidence interval&lt;/a&gt; can be used here. The method you follow is the normal approximation, however unless you have really large counts other methods like the Wilson interval will be more accurate.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastActivityDate="2011-04-11T23:22:22.870" CommentCount="1" />
  <row Id="9452" PostTypeId="2" ParentId="9447" CreationDate="2011-04-11T23:46:32.527" Score="0" ViewCount="" Body="&lt;p&gt;I know little about meteorology, so my following assumptions may be wrong: today's temperature is similar to yesterday's and the day before yesterday's (maybe more days going back), and also similar to temperature a year age, two years ago, three years ago, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If these assumptions got reinforcement I would use an ARMA model using days -1, -2, … and -365, -365*2, -365*3, … as predictors of today's temperature, and maybe a few days looking back in the moving average terms. (You can imagine many variants of this model.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After fitting the model I would make a large number of model based simulations predicting the temperatures for each of the following 365 days, and count the cases satisfying the two conditions.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastActivityDate="2011-04-11T23:46:32.527" />
  <row Id="9454" PostTypeId="1" AcceptedAnswerId="9458" CreationDate="2011-04-12T02:44:24.163" Score="5" ViewCount="130" Body="&lt;h3&gt;Context:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;I have two data sets from the same questionnaire run over two years. Each question is measured using a 5-Likert scale.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Q1: Coding scheme&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;At the moment, I have coded my responses on a [0, 1] interval, with 0 meaning &quot;most negative response&quot;, 1 meaning &quot;most positive response&quot;, and other responses spaced evenly between. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What is the &quot;best&quot; coding scheme to use for the Likert scale? &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I realise that this might be a bit subjective. &lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Q2: Significance across years&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What is the best way to determine whether there is statistically significant change across the two years? &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;That is, looking at the results for question 1 for each year, how do I tell if the difference between the 2011 result and the 2010 result is statistically significant? I've got a vague recollection of the Student's t-test being of use here, but I'm not sure.&lt;/p&gt;&#xA;" OwnerUserId="4123" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-12T04:01:04.667" LastActivityDate="2011-04-13T03:23:53.867" Title="Statistical significance of changes over time on a 5-point Likert item" Tags="&lt;statistical-significance&gt;&lt;likert&gt;" AnswerCount="2" />
  <row Id="9456" PostTypeId="1" CreationDate="2011-04-12T03:14:53.957" Score="3" ViewCount="52" Body="&lt;p&gt;Over the years, I picked up many Statistics Concepts in by a variety of situations and means. I studied some Statistics maybe for a couple semesters almost 10 years back. But I also picked up concepts while doing Machine Learning work. I only understood things in a narrow scope - I always tried to get away knowing only what I need to know for a project etc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consequently I do not have a big picture understanding of Statistics. For eg. I know vaguely what a t-test is and what a chi-square test is. But I can't relate both these concepts solidly in my head. And without relating each of these concepts I feel they are useless and not as powerful. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have come to understand that the only way I understand anything is by working on hard problems chosen by experts. It should be hard so that I take some time to work it, also it should be chosen carefully by experts so that each problem contains a 'moral' which takes me one step close to enlightenment. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So please help me assemble a set of problems to work through to seek out zen through statistics. Scope is all of statistics (regression, ANOVA, t-tests etc, Structural Equations of Latent Variables etc) &lt;/p&gt;&#xA;" OwnerUserId="4126" LastActivityDate="2011-04-12T03:14:53.957" CommunityOwnedDate="2011-04-12T14:20:36.100" Title="What are some good hard exemplary problem sets to work through to solidify my understanding of Statistical concepts?" Tags="&lt;regression&gt;&lt;anova&gt;&lt;t-test&gt;&lt;big-list&gt;" CommentCount="2" FavoriteCount="1" />
  <row Id="9457" PostTypeId="1" CreationDate="2011-04-12T03:25:49.320" Score="2" ViewCount="69" Body="&lt;p&gt;This is a question of definition, does the stats community differentiate these terms?&lt;/p&gt;&#xA;" OwnerUserId="1709" LastActivityDate="2011-04-12T05:41:23.373" Title="Is there a difference between seasonality / cyclicality / periodicity" Tags="&lt;seasonality&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="1" />
  <row Id="9458" PostTypeId="2" ParentId="9454" CreationDate="2011-04-12T04:16:34.013" Score="4" ViewCount="" Body="&lt;h3&gt;1. Coding scheme&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;In terms of assessing statistical significance using a t-test, it is the relative distances between the scale points that matters. Thus, (0, 0.25, 0.5, 0.75, 1) is equivalent to (1, 2, 3, 4, 5).&#xA;From my experience an equal distance coding scheme, such as those mentioned previously are the most common, and seem reasonable for Likert items.&#xA;If you explore optimal scaling, you might be able to  derive an alternative coding scheme.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;2. Statistical test&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;The question of how to assess group differences on a Likert item has already been answered &lt;a href=&quot;http://stats.stackexchange.com/questions/203/group-differences-on-a-five-point-likert-item&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first issue is whether you can link observations across the two time points. It sounds like you had a different sample.&#xA;This leads to a few options:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Independent groups t-test&lt;/strong&gt;: this is a simple option; it also does test for differences in group means; purists will argue that the p-value may be not entirely accurate; however, depending on your purposes, it may be adequate.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Bootstrapped test of differences in group means&lt;/strong&gt;: If you still want to test differences between group means but are uncomfortable with the discrete nature of dependent variable, then you could use a bootstrap to generate confidence intervals from which you could draw inferences about changes in group means.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Mann-Whitney U test&lt;/strong&gt; (among other non-parametric tests): Such a test does not assume normality, but it is also testing a different hypothesis.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-13T03:23:53.867" LastActivityDate="2011-04-13T03:23:53.867" CommentCount="5" />
  <row Id="9459" PostTypeId="2" ParentId="9457" CreationDate="2011-04-12T04:41:37.783" Score="1" ViewCount="" Body="&lt;p&gt;Yes, there is a difference. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A classic time series decomposition model is&#xA;Y = T * S * C * I&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Y = data&#xA;T = trend&#xA;S = seasonal = REGULAR patterns that occur with time, e.g. oatmeal sales higher in winter, or Starbucks coffee sales being highest at 7 a.m. These are usually very predictable.&#xA;C = cyclical = longer term patterns like business cycles. These aren't as regular as seasonality, and may involve some subjectivity in estimation.&#xA;I = irregular (i.e. error left over)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Periodicity refers to seasonal component. Periodicity could be monthly, biweekly, hourly, etc.&lt;/p&gt;&#xA;" OwnerUserId="3919" LastActivityDate="2011-04-12T04:41:37.783" CommentCount="3" />
  <row Id="9460" PostTypeId="2" ParentId="9457" CreationDate="2011-04-12T05:41:23.373" Score="3" ViewCount="" Body="&lt;p&gt;Perhaps.  Though my take could easily be construed as a bit too anal retentive:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tend to use the term seasonality as a metaphor for the 'seasons' of the year: i.e. Spring, Summer, Fall, Winter (or 'Almost Winter', Winter, 'Still Winter', and 'Construction' if you live in Pennsylvania...).  In other words, I would expect a seasonal trend to have a &lt;em&gt;periodicity&lt;/em&gt; of roughly 365 days.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tend to use the term 'cyclicality' to refer to a response, which when decomposed in frequency space has a single dominant peak. Or, a bit more generally, much as one could stare at an engine, 'cyclicality' implies a dominant &lt;em&gt;cycle&lt;/em&gt; -- the piston moves up, and then it moves down, and then it moves up again. Numerically, I would expect low, high, low, high, low, high, etc. So two things: (1) magnitude &amp;amp;/or sign switches from a low to high and (2) these switches occur with a predictable frequency. This rigor naturally evaporates when talking about business cycles -- however, I often find that a dominant frequency remains, e.g. &lt;em&gt;every business quarter&lt;/em&gt;, or &lt;em&gt;every year&lt;/em&gt;, things are slow for the first few weeks and high pressure the last few weeks...  So there is a dominant period, but it could be very different from 'seasonality' which to me implies a year.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lastly, I tend to use 'periodicity' when referring to the frequency of collecting measurements.  Differing from cyclicality, the term 'periodicity' for me implies no expectation for the magnitude or sign of the data collected.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But this is just my $0.02.  And I'm just a stat student -- take from this what you will.&lt;/p&gt;&#xA;" OwnerUserId="1499" LastActivityDate="2011-04-12T05:41:23.373" CommentCount="2" />
  <row Id="9461" PostTypeId="1" CreationDate="2011-04-12T05:42:54.500" Score="5" ViewCount="29" Body="&lt;p&gt;I'm wondering how to test the significance of factor(s) and/or covariate(s) along with modeling the causal relationship among responses.&#xA;Let me explain this with a concrete example.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Example:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Suppose a researcher observed four responses &lt;strong&gt;Y1&lt;/strong&gt;, &lt;strong&gt;Y2&lt;/strong&gt;, &lt;strong&gt;Y3&lt;/strong&gt;, and &lt;strong&gt;Y4&lt;/strong&gt; along with three covariates &lt;strong&gt;X1&lt;/strong&gt;, &lt;strong&gt;X2&lt;/strong&gt;, and &lt;strong&gt;X3&lt;/strong&gt; from an experiment involving &lt;em&gt;ab&lt;/em&gt; treatment combinations from a fixed factor &lt;strong&gt;&lt;em&gt;A&lt;/em&gt;&lt;/strong&gt; with &lt;em&gt;a&lt;/em&gt; levels and a random factor &lt;strong&gt;&lt;em&gt;B&lt;/em&gt;&lt;/strong&gt; with &lt;em&gt;b&lt;/em&gt; levels. Based on past experience, it is assumed four responses are correlated and &lt;strong&gt;Y1&lt;/strong&gt; is also influenced by the other three (&lt;strong&gt;Y2&lt;/strong&gt;, &lt;strong&gt;Y3&lt;/strong&gt;, and &lt;strong&gt;Y4&lt;/strong&gt;). &lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Question:&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;How can I test the significance of Factors (&lt;strong&gt;&lt;em&gt;A&lt;/em&gt;&lt;/strong&gt; and &lt;strong&gt;&lt;em&gt;B&lt;/em&gt;&lt;/strong&gt;) and covariates (&lt;strong&gt;X1&lt;/strong&gt;, &lt;strong&gt;X2&lt;/strong&gt;, and &lt;strong&gt;X3&lt;/strong&gt;) and causality among responses (&lt;strong&gt;Y1&lt;/strong&gt;, &lt;strong&gt;Y2&lt;/strong&gt;, &lt;strong&gt;Y3&lt;/strong&gt;, and &lt;strong&gt;Y4&lt;/strong&gt;) using a single model?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="3903" LastEditorUserId="3903" LastEditorDisplayName="" LastEditDate="2011-04-12T06:48:16.857" LastActivityDate="2011-04-12T06:48:16.857" Title="Testing significance of factors and covariates along with modeling causality among responses" Tags="&lt;multivariate-analysis&gt;&lt;experiment-design&gt;&lt;sem&gt;" CommentCount="1" FavoriteCount="1" />
  <row Id="9462" PostTypeId="2" ParentId="9422" CreationDate="2011-04-12T05:54:54.803" Score="2" ViewCount="" Body="&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Margin_of_error&quot; rel=&quot;nofollow&quot;&gt;Standard formulas for standard errors of a proportion&lt;/a&gt; would be suitable. With regards to your question about which companies the &quot;n=100 sample&quot; plan to use in the future, these standard errors would be based on n = 100. If this yields standard errors that are too large for your liking, then you need to increase your sample size.&lt;/li&gt;&#xA;&lt;li&gt;In some cases you might be able to increase your effective sample size by engaging in more targeted sampling of the subset of the population that interests you (i.e., with company X, but planning to use company X less in the future).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-13T03:30:18.077" LastActivityDate="2011-04-13T03:30:18.077" CommentCount="5" />
  <row Id="9463" PostTypeId="2" ParentId="9437" CreationDate="2011-04-12T06:49:35.103" Score="2" ViewCount="" Body="&lt;p&gt;While I haven't seen anything specifically, I doubt that it would achieve much, at least for linear regression. Each regression equation is just a linear combination of the predictors:&#xA;$$\hat{y}^{(j)} = \sum_i \hat{\beta}_i^{(j)} x_i$$&#xA;Most boosting algorithms in turn combine multiple predictors by taking a weighted average, which is another linear combination:&#xA;$$\hat{y} = \sum_j \hat{w}^{(j)} \hat{y}^{(j)}$$&#xA; So the final output is still a linear combination of the predictors, only with the coefficients obtained by a complicated process.&lt;/p&gt;&#xA;" OwnerUserId="1569" LastActivityDate="2011-04-12T06:49:35.103" />
  <row Id="9464" PostTypeId="1" AcceptedAnswerId="9466" CreationDate="2011-04-12T06:54:42.497" Score="2" ViewCount="92" Body="&lt;p&gt;I'm trying to fully understand the confidence interval formal given on this site:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.iro.umontreal.ca/~lisa/twiki/bin/view.cgi/Public/DeepVsShallowComparisonICML2007#Results&quot; rel=&quot;nofollow&quot;&gt;http://www.iro.umontreal.ca/~lisa/twiki/bin/view.cgi/Public/DeepVsShallowComparisonICML2007#Results&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;so I can reproduce the same type of intervals for my own data.  But I don't quite understand what the parameters such as alpha and Z mean.  I'm guessing they're related to defining a 95% confidence interval if your data were distributed normally.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can explain to me how that formula works or a reference I can find a description of the formula used?&lt;/p&gt;&#xA;" OwnerUserId="4127" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-04-12T12:12:58.863" LastActivityDate="2011-04-12T18:05:04.817" Title="Can someone give me details about a particular confidence interval formula?" Tags="&lt;machine-learning&gt;&lt;confidence-interval&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="1" />
  <row Id="9465" PostTypeId="2" ParentId="9407" CreationDate="2011-04-12T06:59:48.757" Score="1" ViewCount="" Body="&lt;h3&gt;1. Power analysis for one-way ANOVA:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Download &lt;a href=&quot;http://www.psycho.uni-duesseldorf.de/abteilungen/aap/gpower3/download-and-register&quot; rel=&quot;nofollow&quot;&gt;G-Power 3&lt;/a&gt;. It allows you to do for a range of statistical tests including ANOVA&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;a priori&lt;/strong&gt; power analysis (sample required given effect size, desired power, and alpha), and &lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;post hoc&lt;/strong&gt; power analysis (observed power given sample size, effect size, and alpha)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.psycho.uni-duesseldorf.de/abteilungen/aap/gpower3/user-guide-by-distribution/f/anova_fixed_effects_omnibus&quot; rel=&quot;nofollow&quot;&gt;Here's a G Power 3 tutorial for ANOVA&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;2. Power analysis and Likert items&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Assuming that the 7-point Likert item is a continuous variable might be a useful simplifying assumption. This is especially true given that assumptions made about effect size for power analysis are only guesses (if they weren't, then you wouldn't need to do the study).&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;3. Finite populations&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Also, even when there is a finite population, you are often wanting to determine whether any differences reflect a theoretically meaningful difference or just random sampling. For example, assume that you measured the whole population of both groups, and you found that group A had a mean of 6.12, and group B had a mean of 6.11. You could conclude that there is a difference between the groups. Alternatively, you could assume that the participants in each group are drawn from some theoretical data generating process, and that you are interested in drawing inferences about this. Then, you would apply standard t-tests or ANOVAs to test whether the observed difference is statistically significant. If this was your purpose, you would not need to apply corrections for finite populations.&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2011-04-12T06:59:48.757" />
  <row Id="9466" PostTypeId="2" ParentId="9464" CreationDate="2011-04-12T07:45:02.760" Score="1" ViewCount="" Body="&lt;p&gt;If $\hat{\mu}$ is the mean error rate computed averaging $N$ error rates from different $N$ tests, an explanation could be: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let $X$ be the number of errors on $N$ tests, so $X$ is a binomial distributed random variable with mean $N\hat{\mu}$ and variance $N\hat{\mu}(1-\hat{\mu})$ (it is sum of $N$ Bernoulli random variables).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus $X/N\sim Bin\bigg(\hat{\mu},\frac{\hat{\mu}(1-\hat{\mu})}{N}\bigg)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By the central limit theorem it could be approximated to a normal random variable with same mean and variance. Then you can computer the $\alpha$ confidence interval with:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P\bigg(-z_{1-\alpha/2}\leq\frac{\mu-\hat{\mu}}{\sqrt{\hat{\mu}(1-\hat{\mu})/N}}\leq-z_{1-\alpha/2}\bigg) = 1 - \alpha$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bibliography:&#xA;It is similar to estimate a confidence interval for accuracy using a $N$ values test set in a classification problem. You should take a look to P.N. Tan, M. Steinbach, V. Kumar Introduction to Data Mining. Addison Wesley, 2006. &lt;/p&gt;&#xA;" OwnerUserId="2719" LastEditorUserId="2719" LastEditorDisplayName="" LastEditDate="2011-04-12T18:05:04.817" LastActivityDate="2011-04-12T18:05:04.817" CommentCount="2" />
  <row Id="9467" PostTypeId="1" AcceptedAnswerId="9473" CreationDate="2011-04-12T08:21:07.817" Score="6" ViewCount="50" Body="&lt;p&gt;Following on from my &lt;a href=&quot;http://stats.stackexchange.com/questions/9341/regularized-fit-from-summarized-data&quot;&gt;earlier question&lt;/a&gt;, the solution to the normal equations for ridge regression is given by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\hat{\beta}_\lambda = (X^TX+\lambda I)^{-1}X^Ty$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could you offer any guidance for choosing the regularization parameter $\lambda$. Additionally, since the diagonal of $X^TX$ grows with the number of observations $m$, should $\lambda$ also be a function of $m$?&lt;/p&gt;&#xA;" OwnerUserId="439" LastActivityDate="2011-04-12T13:05:16.340" Title="Regularized fit from summarized data: choosing the parameter" Tags="&lt;regression&gt;&lt;regularization&gt;&lt;ridge-regression&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="9468" PostTypeId="1" CreationDate="2011-04-12T10:33:33.660" Score="2" ViewCount="64" Body="&lt;p&gt;Suppose I have a model of stock prices developed using Brownian motion.  I have a second time series derived from the first.  At each price point in the first time series, I take the arithmetic mean up to that point and that is the data point for my second time series.  The volatility of the second time series is lower because the arithmetic mean has a dampening effect.  Are there any statistical tools to model the volatility of the second time series? &lt;/p&gt;&#xA;" OwnerUserId="4128" LastActivityDate="2011-05-12T13:45:34.980" Title="How do I model the volatility of an arithmetic mean?" Tags="&lt;standard-deviation&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="1" />
  <row Id="9469" PostTypeId="2" ParentId="9454" CreationDate="2011-04-12T10:39:25.950" Score="2" ViewCount="" Body="&lt;p&gt;dear jeromy,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Wilcoxon Ranksum Test aka Mann-Whitney is the way to go in the case of ordinal data. The bootstrapping solution is also elegant albeit not the &quot;classic&quot; way to go. The Bootstrapping method might also be valuable in case you aim for other things like factor analysis. In case of regression analysis you might chose ordered probit or ordered logit as a model specification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;BTW: If your scale has a larger range (&gt;10 values per variable) you might use the results as a metric variable, wich makes a t-test a safe choice. Be adviced that this is a little dirty and may be considered devil's work by some. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;stephan&lt;/p&gt;&#xA;" OwnerUserId="4129" LastActivityDate="2011-04-12T10:39:25.950" CommentCount="2" />
  <row Id="9470" PostTypeId="1" CreationDate="2011-04-12T12:12:56.453" Score="1" ViewCount="28" Body="&lt;p&gt;I'm trying to replicate the paper of Blundell et al. (2008) to separate permanent and transitory shocks on income on a panel dataset.&#xA;He solves the non-linear system of equations using Chamberlain's minimum distance estimator (it is shown in the appendix of the paper), but I haven't found a library, nor in R, nor in STATA, that does the trick. Can someone help me?&lt;/p&gt;&#xA;" OwnerUserId="4131" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-04-12T12:23:58.203" LastActivityDate="2011-04-12T12:23:58.203" Title="Minimum distance estimator" Tags="&lt;nonlinear-regression&gt;" CommentCount="3" FavoriteCount="1" />
  <row Id="9471" PostTypeId="2" ParentId="9468" CreationDate="2011-04-12T12:57:20.417" Score="0" ViewCount="" Body="&lt;p&gt;Let's say your stock price series is $S_t$ so that your arithmetic average series is $X_t = \frac{1}{t}\sum_{j=1}^t S_j$. The series of returns that you use to calculate the volatility can be defined in several ways, the most popular being the log-returns series&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$R_t = \frac{1}{\delta t} \log \frac{S_t}{S_{t-1}}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and the simple returns series&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$R_t = \frac{1}{\delta t} \left( \frac{S_t}{S_{t-1}} - 1 \right)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\delta t$ is the time between observations (measured as a fraction of a year). Both of these involve the ratio $S_i / S_{i-1}$, so to look at the volatility of the series $X$ we should calculate this ratio:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\frac{X_t}{X_{t-1}} = \frac{t-1}{t} \left( 1 + \frac{S_t}{S_1+\cdots+S_{t-1}} \right)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Either way you cut it, the returns series for $X$ is not going to be pretty. The most straightforward approach is to look at the simple returns series for $X$, let's call it $Q$:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Q_t = \frac{1}{\delta t} \frac{t-1}{t} \left( \frac{S_t}{S_1+\cdots+S_{t-1}} - \frac{1}{t-1} \right)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The volatility of the series $X$ is the standard deviation of this series. Note that even if the volatility of $S$ is constant, the volatility of $X$ won't be constant, because of the dependence of the series on $t$. In fact, under some mild assumptions on the nature of the series $S$ (eg bounded returns) you should be able to show that the volatility of $X$ decreases to zero as $t\to\infty$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This doesn't fully answer your question, but it points out some of the difficulties involved in what you're trying to do.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Perhaps a better idea (depending on what you're trying to achieve) is to take a moving average of your stock price series, taking the arithmetic average of the previous $T$ observations. At least this way your averaged series won't have an explicit dependence on the time index $t$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might also consider taking the geometric mean rather than the arithmetic mean. Although conceptually less intuitive, it has the advantage that it is equivalent to taking the arithmetic average of the log-returns series, which greatly simplifies the algebra later.&lt;/p&gt;&#xA;" OwnerUserId="2425" LastEditorUserId="2425" LastEditorDisplayName="" LastEditDate="2011-04-12T13:32:31.587" LastActivityDate="2011-04-12T13:32:31.587" />
  <row Id="9472" PostTypeId="2" ParentId="9464" CreationDate="2011-04-12T13:01:17.443" Score="2" ViewCount="" Body="&lt;p&gt;Answering the following part of your question:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;I don't quite understand what the&#xA;  parameters such as alpha and Z mean&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;$\alpha$ is the parameter that defines the confidence level of the interval. Specifically, the confidence level will be $100(1-\alpha)$%, so to get a 95% confidence interval, set $\alpha=0.05$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Z$ is a reference to the normal distribution, and in this case $z_q$ means its $q$-th quantile, that is the value for which $P(Z &amp;lt; z_q) = q$, where $Z$ is the standard normal distribution. This can be looked up in tables or calculated by computers. For example, when $\alpha=0.05$, the formula needs the 0.975-th quantile, that is the value which exceeds 97.5% of the normal distribution. Its value is $z_{0.975}=1.96$.&lt;/p&gt;&#xA;" OwnerUserId="279" LastActivityDate="2011-04-12T13:01:17.443" />
  <row Id="9473" PostTypeId="2" ParentId="9467" CreationDate="2011-04-12T13:05:16.340" Score="4" ViewCount="" Body="&lt;p&gt;My answer will be based on a nice review of the problem by Anders Bjorkstorm &lt;a href=&quot;http://www2.math.su.se/matstat/reports/seriea/2000/rep5/report.pdf&quot; rel=&quot;nofollow&quot;&gt;Ridge regression and inverse problems&lt;/a&gt; (I would recommend to read the whole article).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Part 4 in this review is dedicated to the selection of a parameter $\lambda$ in ridge regression introducing several key approaches:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;em&gt;ridge trace&lt;/em&gt; corresponds to graphical analysis of $\hat{\beta}_{i,\lambda}$ against $\lambda$. A typical plot will depict unstable (for a true ill-posted problem, you have to be sure you need this regularization in any case) behavior of different $\hat{\beta}_{i,\lambda}$ estimates for $\lambda$ close to zero, and almost constant from some point (roughly we have to detect constant behavior intersection region for all of the parameters). However decision regarding where this almost constant behavior starts is somewhat subjective. Good news for this approach is that it does not require to observe $X$ and $y$.&lt;/li&gt;&#xA;&lt;li&gt;&lt;em&gt;$L$-curve&lt;/em&gt; it plots Euclidean norm of the vector of estimated parameters $|\hat{{\beta}}_\lambda|$ against the residual norm $|y - X\hat{\beta}_\lambda|$. The shape is typically close to letter $L$ so there exists a corner that determines where optimal parameter belongs to (one may choose the point in $L$ curve where the latter reaches the maximum &lt;a href=&quot;http://en.wikipedia.org/wiki/Curvature&quot; rel=&quot;nofollow&quot;&gt;curvature&lt;/a&gt;, but it is better to search for &lt;a href=&quot;http://www.jstor.org/pss/2132628&quot; rel=&quot;nofollow&quot;&gt;Hansen&lt;/a&gt;'s article for more details).&lt;/li&gt;&#xA;&lt;li&gt;For &lt;em&gt;cross-validation&lt;/em&gt; actually a simple &quot;leave-one-out&quot; approach is often chosen, seeking for $\lambda$ that maximizes (or minimizes) some forecasting accuracy criterion (you have a wide range of them, RMSE and MAPE are the two to begin with). Difficulties with 2. and 3. are that you have to observe $X$ and $y$ to implement them in practice.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="2645" LastActivityDate="2011-04-12T13:05:16.340" CommentCount="4" />
  <row Id="9474" PostTypeId="1" AcceptedAnswerId="9527" CreationDate="2011-04-12T13:42:54.853" Score="0" ViewCount="51" Body="&lt;p&gt;I'm looking for an implementation of a fast maximum rank correlation (MRC) estimator. This will be applied to large-ish sparse matrices (~100,000 by 10,000) in a text-mining application.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm working in python and R, so it would be nice to find something in those languages. Failing that, I could probably convert code from some other language.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestions?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Quick note: the best algorithm I've seen described is in &lt;a href=&quot;http://www.sciencedirect.com/science?_ob=ArticleURL&amp;amp;_udi=B6V84-3W7XC0R-4&amp;amp;_user=99318&amp;amp;_coverDate=03%2F01%2F1999&amp;amp;_rdoc=1&amp;amp;_fmt=high&amp;amp;_orig=gateway&amp;amp;_origin=gateway&amp;amp;_sort=d&amp;amp;_docanchor=&amp;amp;view=c&amp;amp;_searchStrId=1715406837&amp;amp;_rerunOrigin=scholar.google&amp;amp;_acct=C000007678&amp;amp;_version=1&amp;amp;_urlVersion=0&amp;amp;_userid=99318&amp;amp;md5=2818ab10a7b249b05d8a0d0a24c2ff82&amp;amp;searchtype=a&quot; rel=&quot;nofollow&quot;&gt;Abrevaya (1999)&lt;/a&gt;, which runs in $nlog(n)$ time.  &lt;a href=&quot;http://www.sciencedirect.com/science?_ob=ArticleURL&amp;amp;_udi=B6V8V-4M813VJ-1&amp;amp;_user=99318&amp;amp;_coverDate=03%2F01%2F2007&amp;amp;_rdoc=1&amp;amp;_fmt=high&amp;amp;_orig=gateway&amp;amp;_origin=gateway&amp;amp;_sort=d&amp;amp;_docanchor=&amp;amp;view=c&amp;amp;_searchStrId=1715396891&amp;amp;_rerunOrigin=scholar.google&amp;amp;_acct=C000007678&amp;amp;_version=1&amp;amp;_urlVersion=0&amp;amp;_userid=99318&amp;amp;md5=ed8045c282ad6ba5635207366e765e53&amp;amp;searchtype=a&quot; rel=&quot;nofollow&quot;&gt;Wang (2007)&lt;/a&gt; has an &quot;IMO&quot; algorithm that he claims runs in $n^2log(n)$ -- much worse.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: I've &lt;a href=&quot;http://stackoverflow.com/questions/5636093/is-there-a-library-that-implements-a-fast-maximum-rank-correlation-estimator&quot;&gt;cross-posted&lt;/a&gt; this question on stackoverflow because I'm not sure which community is the right place to ask.&lt;/p&gt;&#xA;" OwnerUserId="4110" LastActivityDate="2011-04-13T18:07:07.070" Title="Is there a library that implements a fast maximum rank correlation estimator?" Tags="&lt;r&gt;&lt;estimation&gt;&lt;algorithms&gt;&lt;python&gt;" AnswerCount="1" />
  <row Id="9475" PostTypeId="1" CreationDate="2011-04-12T14:33:45.357" Score="3" ViewCount="100" Body="&lt;p&gt;I have many  time series in this format 1 column in which I Have date (d/m/yr) format and many columns that represent different time series like here: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;DATE         TS1     TS2     TS3 ...&#xA;24/03/2003   0.00    0.00    ...&#xA;17/04/2003  -0.05    1.46&#xA;11/05/2003   0.46   -3.86&#xA;04/06/2003  -2.21   -1.08&#xA;28/06/2003  -1.18   -2.16&#xA;22/07/2003   0.00    0.23&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;With R, how I can group the time series that show similar trends?&lt;/p&gt;&#xA;" OwnerUserId="4133" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-12T14:37:25.620" LastActivityDate="2011-04-12T14:37:25.620" Title="Time series clustering" Tags="&lt;r&gt;&lt;time-series&gt;&lt;clustering&gt;" CommentCount="4" FavoriteCount="1" />
  <row Id="9476" PostTypeId="2" ParentId="9385" CreationDate="2011-04-12T14:48:00.777" Score="2" ViewCount="" Body="&lt;p&gt;The formulae for Holt-Winters' method include forecasting the seasonal component. You don't need $\gamma=0$. See a forecasting textbook for the details.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2011-04-12T14:48:00.777" />
  <row Id="9477" PostTypeId="1" AcceptedAnswerId="9480" CreationDate="2011-04-12T16:07:15.017" Score="3" ViewCount="260" Body="&lt;p&gt;My attempts:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;I couldn't get confidence intervals in &lt;code&gt;interaction.plot()&lt;/code&gt; &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;and on the other hand &lt;code&gt;plotmeans()&lt;/code&gt; from package 'gplot' wouldn't display two graphs. Furthermore, I couldn't impose two &lt;code&gt;plotmeans()&lt;/code&gt; graphs one on top of the other because by default the axis are different.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I had some success using &lt;code&gt;plotCI()&lt;/code&gt; from package 'gplot'  and superimposing two graphs but still the match of the axis wasn't perfect.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Any advice on how to make an interaction plot with confidence intervals? &#xA;Either by one function, or advice on how to superimpose &lt;code&gt;plotmeans()&lt;/code&gt; or &lt;code&gt;plotCI()&lt;/code&gt; graphs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for reading!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;p.s. I originally posted this at &lt;a href=&quot;http://stackoverflow.com/questions/5524128/r-interaction-plot-with-confidence-intervals&quot;&gt;StackOverflow&lt;/a&gt; but at second thought this question is better for CrossValidated&lt;/p&gt;&#xA;&#xA;&lt;p&gt;code sample&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;br=structure(list(tangle = c(140L, 50L, 40L, 140L, 90L, 70L, 110L, &#xA;150L, 150L, 110L, 110L, 50L, 90L, 140L, 110L, 50L, 60L, 40L, &#xA;40L, 130L, 120L, 140L, 70L, 50L, 140L, 120L, 130L, 50L, 40L, &#xA;80L, 140L, 100L, 60L, 70L, 50L, 60L, 60L, 130L, 40L, 130L, 100L, &#xA;70L, 110L, 80L, 120L, 110L, 40L, 100L, 40L, 60L, 120L, 120L, &#xA;70L, 80L, 130L, 60L, 100L, 100L, 60L, 70L, 90L, 100L, 140L, 70L, &#xA;100L, 90L, 130L, 70L, 130L, 40L, 80L, 130L, 150L, 110L, 120L, &#xA;140L, 90L, 60L, 90L, 80L, 120L, 150L, 90L, 150L, 50L, 50L, 100L, &#xA;150L, 80L, 90L, 110L, 150L, 150L, 120L, 80L, 80L), gtangles = c(141L, &#xA;58L, 44L, 154L, 120L, 90L, 128L, 147L, 147L, 120L, 127L, 66L, &#xA;118L, 141L, 111L, 59L, 72L, 45L, 52L, 144L, 139L, 143L, 73L,  &#xA;59L, 148L, 141L, 135L, 63L, 51L, 88L, 147L, 110L, 68L, 78L, 63L, &#xA;64L, 70L, 133L, 49L, 129L, 100L, 78L, 128L, 91L, 121L, 109L, &#xA;48L, 113L, 50L, 68L, 135L, 120L, 85L, 97L, 136L, 59L, 112L, 103L, &#xA;62L, 87L, 92L, 116L, 141L, 70L, 121L, 92L, 137L, 85L, 117L, 51L, &#xA;84L, 128L, 162L, 102L, 127L, 151L, 115L, 57L, 93L, 92L, 117L, &#xA;140L, 95L, 159L, 57L, 65L, 130L, 152L, 90L, 117L, 116L, 147L, &#xA;140L, 116L, 98L, 95L), up = c(-1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, &#xA;-1L, -1L, 1L, 1L, 1L, 1L, -1L, -1L, -1L, -1L, 1L, 1L, -1L, -1L, &#xA;1L, 1L, -1L, 1L, 1L, -1L, 1L, 1L, 1L, 1L, 1L, -1L, -1L, 1L, 1L, &#xA;1L, 1L, -1L, -1L, 1L, 1L, -1L, -1L, -1L, -1L, -1L, -1L, -1L, &#xA;1L, -1L, -1L, -1L, -1L, -1L, 1L, -1L, 1L, 1L, -1L, -1L, -1L, &#xA;-1L, 1L, -1L, 1L, -1L, -1L, -1L, 1L, -1L, 1L, -1L, 1L, 1L, 1L, &#xA;-1L, -1L, -1L, -1L, -1L, -1L, 1L, -1L, 1L, 1L, -1L, -1L, 1L, &#xA;1L, 1L, -1L, 1L, 1L, 1L)), .Names = c(&quot;tangle&quot;, &quot;gtangles&quot;, &quot;up&quot;&#xA;), class = &quot;data.frame&quot;, row.names = c(NA, -96L))&#xA;&#xA;plotmeans2 &amp;lt;- function(br, alph) {&#xA;dt=br;   tmp   &amp;lt;- split(br$gtangles, br$tangle);   &#xA;means &amp;lt;- sapply(tmp, mean);  stdev &amp;lt;- sqrt(sapply(tmp, var));  &#xA;n &amp;lt;- sapply(tmp,length);  &#xA;ciw   &amp;lt;- qt(alph, n) * stdev / sqrt(n)&#xA;plotCI(x=means, uiw=ciw, col=&quot;black&quot;, barcol=&quot;blue&quot;, lwd=1,ylim=c(40,150),  xlim=c(1,12)); &#xA;par(new=TRUE) dt= subset(br,up==1);   &#xA;tmp   &amp;lt;- split(dt$gtangles, dt$tangle);  &#xA;means &amp;lt;- sapply(tmp, mean);  &#xA;stdev &amp;lt;- sqrt(sapply(tmp, var));  &#xA;n &amp;lt;- sapply(tmp,length); &#xA;ciw  &amp;lt;- qt(0.95, n) * stdev / sqrt(n)&#xA;plotCI(x=means, uiw=ciw, type='l',col=&quot;black&quot;, barcol=&quot;red&quot;, lwd=1,ylim=c(40,150), xlim=c(1,12),pch='+');&#xA;abline(v=6);abline(h=90);abline(30,10); par(new=TRUE);&#xA;dt=subset(br,up==-1);   &#xA;tmp &amp;lt;- split(dt$gtangles, dt$tangle);  &#xA;means &amp;lt;- sapply(tmp, mean);  &#xA;stdev &amp;lt;- sqrt(sapply(tmp, var));  &#xA;n &amp;lt;- sapply(tmp,length); &#xA;ciw &amp;lt;- qt(0.95, n) * stdev / sqrt(n)&#xA;plotCI(x=means, uiw=ciw, type='l', col=&quot;black&quot;, barcol=&quot;blue&quot;,   lwd=1,ylim=c(40,150), xlim=c(1,12),pch='-');abline(v=6);abline(h=90);&#xA;abline(30,10);&#xA;}&#xA;&#xA;plotmeans2(br,.95)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1084" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-12T19:56:54.877" LastActivityDate="2011-04-20T20:02:55.400" Title="R - interaction plot with confidence intervals?" Tags="&lt;r&gt;&lt;data-visualization&gt;&lt;confidence-interval&gt;&lt;interaction&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="2" />
  <row Id="9478" PostTypeId="1" CreationDate="2011-04-12T17:04:06.680" Score="1" ViewCount="86" Body="&lt;p&gt;I have samples with each sample has n features, how to normalize these features to let feature values lie between interval [-1,1], please give a formula.&lt;/p&gt;&#xA;" OwnerUserId="2141" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-12T19:50:34.097" LastActivityDate="2011-04-12T19:50:34.097" Title="How to normalize data to let each feature lie between [-1,1]?" Tags="&lt;normalization&gt;" CommentCount="3" />
  <row Id="9479" PostTypeId="2" ParentId="9315" CreationDate="2011-04-12T17:17:44.110" Score="2" ViewCount="" Body="&lt;p&gt;I'd try 'folding in'. This refers to taking one new document, adding it to the corpus, and then running Gibbs sampling &lt;em&gt;just on the words in that new document&lt;/em&gt;, keeping the topic assignments of the old documents the same. This usually converges fast (maybe 5-10-20 iterations), and you don't need to sample your old corpus, so it also runs fast. At the end you will have the topic assignment for every word in the new document. This will give you the distribution of topics in that document.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your Gibbs sampler, you probably have something similar to the following code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;// This will initialize the matrices of counts, N_tw (topic-word matrix) and N_dt (document-topic matrix)&#xA;for doc = 1 to N_Documents&#xA;    for token = 1 to N_Tokens_In_Document&#xA;       Assign current token to a random topic, updating the count matrices&#xA;    end&#xA;end&#xA;&#xA;// This will do the Gibbs sampling&#xA;for doc = 1 to N_Documents&#xA;    for token = 1 to N_Tokens_In_Document&#xA;       Compute probability of current token being assigned to each topic&#xA;       Sample a topic from this distribution&#xA;       Assign the token to the new topic, updating the count matrices&#xA;    end&#xA;end&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Folding-in is the same, except you start with the existing matrices, add the new document's tokens to them, and do the sampling for only the new tokens. I.e.:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Start with the N_tw and N_dt matrices from the previous step&#xA;&#xA;// This will update the count matrices for folding-in&#xA;for token = 1 to N_Tokens_In_New_Document&#xA;   Assign current token to a random topic, updating the count matrices&#xA;end&#xA;&#xA;// This will do the folding-in by Gibbs sampling&#xA;for token = 1 to N_Tokens_In_New_Document&#xA;   Compute probability of current token being assigned to each topic&#xA;   Sample a topic from this distribution&#xA;   Assign the token to the new topic, updating the count matrices&#xA;end&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;If you do standard LDA, it is unlikely that an entire document was generated by one topic. So I don't know how useful it is to compute the probability of the document under one topic. But if you still wanted to do it, it's easy. From the two matrices you get you can compute $p^i_w$, the probability of word $w$ in topic $i$. Take your new document; suppose the $j$'th word is $w_j$. The words are independent given the topic, so the probability is just $$\prod_j p^i_{w_j}$$ (note that you will probably need to compute it in log space).&lt;/p&gt;&#xA;" OwnerUserId="3369" LastEditorUserId="3369" LastEditorDisplayName="" LastEditDate="2011-04-13T18:39:20.870" LastActivityDate="2011-04-13T18:39:20.870" CommentCount="1" />
  <row Id="9480" PostTypeId="2" ParentId="9477" CreationDate="2011-04-12T18:24:25.537" Score="9" ViewCount="" Body="&lt;p&gt;If you're willing to use &lt;a href=&quot;http://had.co.nz/ggplot2/&quot; rel=&quot;nofollow&quot;&gt;ggplot&lt;/a&gt;, you can try the following code.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;With a continuous predictor&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(ggplot2)&#xA;gp &amp;lt;- ggplot(data=br, aes(x=tangle, y=gtangles)) &#xA;gp + geom_point() + stat_smooth(method=&quot;lm&quot;, fullrange=T) + facet_grid(. ~ up)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;for a facetted interaction plot&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/ihhbW.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a standard interaction plot (like the one produced by &lt;code&gt;interaction.plot()&lt;/code&gt;), you just have to remove the facetting.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;gp &amp;lt;- ggplot(data=br, aes(x=tangle, y=gtangles, colour=factor(up))) &#xA;gp + geom_point() + stat_smooth(method=&quot;lm&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/UUHNu.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;With a discrete predictor&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using the &lt;code&gt;ToothGrowth&lt;/code&gt; dataset (see &lt;code&gt;help(ToothGrowth)&lt;/code&gt;),&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ToothGrowth$dose.cat &amp;lt;- factor(ToothGrowth$dose, labels=paste(&quot;d&quot;, 1:3, sep=&quot;&quot;))&#xA;df &amp;lt;- with(ToothGrowth , aggregate(len, list(supp=supp, dose=dose.cat), mean))&#xA;df$se &amp;lt;- with(ToothGrowth , aggregate(len, list(supp=supp, dose=dose.cat), &#xA;              function(x) sd(x)/sqrt(10)))[,3]&#xA;&#xA;opar &amp;lt;- theme_update(panel.grid.major = theme_blank(),&#xA;                     panel.grid.minor = theme_blank(),&#xA;                     panel.background = theme_rect(colour = &quot;black&quot;))&#xA;gp &amp;lt;- ggplot(df, aes(x=dose, y=x, colour=supp, group=supp))&#xA;gp + geom_line(aes(linetype=supp), size=.6) + &#xA;     geom_point(aes(shape=supp), size=3) + &#xA;     geom_errorbar(aes(ymax=x+se, ymin=x-se), width=.1)&#xA;theme_set(opar)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/ZDfXy.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-20T20:02:55.400" LastActivityDate="2011-04-20T20:02:55.400" CommentCount="2" />
  <row Id="9481" PostTypeId="2" ParentId="7057" CreationDate="2011-04-12T19:43:01.033" Score="2" ViewCount="" Body="&lt;p&gt;In my experience, LARS is faster for small problems, very sparse problems, or very 'wide' problems (much much more features than samples). Indeed, its computational cost is limited by the number of features selected, if you don't compute the full regularization path. On the other hand, for big problems, glmnet (coordinate descent optimization) is faster. Amongst other things, coordinate descent has a good data access pattern (memory-friendly) and it can benefit from redundancy in the data on very large datasets, as it converges with partial fits. In particular, it does not suffer from heavily correlated datasets. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The conclusion that we (the core developers of the &lt;a href=&quot;http://scikit-learn.sourceforge.net/&quot; rel=&quot;nofollow&quot;&gt;scikit-learn&lt;/a&gt;) have come too is that, if you do not have strong a priori knowledge of your data, you should rather use glmnet (or coordinate descent optimization, to talk about an algorithm rather than an implementation).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Interesting benchmarks may be compared in Julien Mairal's thesis:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.di.ens.fr/~mairal/resources/pdf/phd_thesis.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.di.ens.fr/~mairal/resources/pdf/phd_thesis.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Section 1.4, in particular 1.4.5 (page 22)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Julien comes to slightly different conclusions, although his analysis of the problem is similar. I suspect this is because he was very much interested in very wide problems.&lt;/p&gt;&#xA;" OwnerUserId="1265" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-12T20:08:59.603" LastActivityDate="2011-04-12T20:08:59.603" CommunityOwnedDate="2011-04-12T19:43:01.033" CommentCount="4" />
  <row Id="9482" PostTypeId="1" CreationDate="2011-04-12T20:24:51.287" Score="2" ViewCount="52" Body="&lt;p&gt;I am currently working with large datasets that have missing data points. My data ranges from the years 1979-2009. I want to choose, by year, the maximum value from that year. Missing values are replaced with a &quot;.&quot; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am currently using the formula:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;{=MAX((E13:E2863=1979)*(H13:H2863))}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This array returns the max value where the appropriate data is available. However when i use the array over the entire dataset, a #VALUE error occurs within the box. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is how do you overcome this issue? Is there a way to develop a formula that will not take into account for the missing data points i.e. cells with &quot;.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Below i will post a screenshot representing my spreadsheet/issue. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;B       C    D     E     F =MAX((B6:B18=1979)*(E6:E18)) &#xA;1979    0    53    67    38    &#xA;1979    0    57    72    41    &#xA;1979    0    63    .     47    &#xA;1979    0    64    79    49    &#xA;1979    0    65    78    52    &#xA;1979    0    57    71    43    &#xA;1980    0.37 43    52    33    &#xA;1980    0    47    60    34    &#xA;1980    0    55    70    39    &#xA;1980    0    64    79    48    &#xA;1980    0    66    82    50    &#xA;1980    0    69    87    51    &#xA;1981    0    71    85    57&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Any suggestions?&lt;/p&gt;&#xA;" OwnerUserId="4136" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-13T07:57:05.360" LastActivityDate="2011-04-13T07:57:05.360" Title="How do you choose MAX or MIN values from a range of incomplete data?" Tags="&lt;modeling&gt;&lt;excel&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="9483" PostTypeId="1" AcceptedAnswerId="9495" CreationDate="2011-04-12T20:56:40.753" Score="4" ViewCount="252" Body="&lt;p&gt;I have two datasets from genome-wide association studies. The only information available are the odd ratios and their confidence intervals (95%) for each genotyped SNP.&#xA;My want to generate a forest plot comparing these two odds ratios, but I can't find the way to calculate the combined confidence intervals to visualize the summary effects. I used the program &lt;a href=&quot;http://pngu.mgh.harvard.edu/~purcell/plink/&quot; rel=&quot;nofollow&quot;&gt;PLINK&lt;/a&gt; to perform the meta-analysis using fixed effects, but the program did not show these confidence intervals.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;How can I calculate such confidence intervals?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The data available is: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Odd ratios for each study,   &lt;/li&gt;&#xA;&lt;li&gt;95% confidence intervals and  &lt;/li&gt;&#xA;&lt;li&gt;Standard errors.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="4137" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-13T08:25:08.687" LastActivityDate="2011-04-14T12:44:16.130" Title="How to calculate confidence intervals for pooled odd ratios in meta-analysis?" Tags="&lt;confidence-interval&gt;&lt;meta-analysis&gt;&lt;genetics&gt;&lt;odds-ratio&gt;" AnswerCount="3" FavoriteCount="1" />
  <row Id="9484" PostTypeId="2" ParentId="9483" CreationDate="2011-04-12T23:00:57.143" Score="0" ViewCount="" Body="&lt;p&gt;This is a comment (don't have enough rep. points). If you know the sample size (#cases and #controls) in each study, and the odds ratio for a SNP, you can reconstruct the 2x2 table of case/control by a/b (where a and b are the two alleles) for each of the two studies. Then you can just add those counts to get a table for the meta-study, and use this to compute the combined odds-ratio and confidence intervals. &lt;/p&gt;&#xA;" OwnerUserId="3036" LastActivityDate="2011-04-12T23:00:57.143" CommentCount="3" />
  <row Id="9485" PostTypeId="2" ParentId="9365" CreationDate="2011-04-13T01:35:39.070" Score="1" ViewCount="" Body="&lt;p&gt;An article with early impact regarding statistical bioinformatics research:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Jelizarow &lt;em&gt;et al&lt;/em&gt;. &lt;a href=&quot;http://bioinformatics.oxfordjournals.org/content/26/16/1990.abstract&quot; rel=&quot;nofollow&quot;&gt;Over-optimism in bioinformatics: an illustration&lt;/a&gt;. Bioinformatics, 2010&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It makes for an interesting discussion on bias sources, overfitting, and fishing for significance.&lt;/p&gt;&#xA;" OwnerUserId="3770" LastActivityDate="2011-04-13T01:35:39.070" CommunityOwnedDate="2011-04-13T01:35:39.070" />
  <row Id="9486" PostTypeId="2" ParentId="2691" CreationDate="2011-04-13T01:55:06.840" Score="1" ViewCount="" Body="&lt;p&gt;A two pages article explaining PCA for biologists:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ringnér. &lt;a href=&quot;http://www.nature.com/nbt/journal/v26/n3/full/nbt0308-303.html&quot; rel=&quot;nofollow&quot;&gt;What is principal component analysis?&lt;/a&gt;. Nature Biotechnology 26, 303-304 (2008) &lt;/p&gt;&#xA;" OwnerUserId="3770" LastActivityDate="2011-04-13T01:55:06.840" />
  <row Id="9487" PostTypeId="2" ParentId="9482" CreationDate="2011-04-13T02:31:59.070" Score="1" ViewCount="" Body="&lt;p&gt;Using a Pivot Table isn't as quick or dirty, but may give you some more options for further analysis that make this approach worthwhile. I honestly have no idea how to convey in words how to use Excel's pivot tables, but if you highlight all of your data of interest and click on &quot;insert --&gt; pivot table&quot; and then adjust the row labels and values, you can eventually get what you are after. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's a screenshot:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/uJH0l.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="696" LastActivityDate="2011-04-13T02:31:59.070" CommentCount="2" />
  <row Id="9488" PostTypeId="1" AcceptedAnswerId="9489" CreationDate="2011-04-13T03:35:57.670" Score="1" ViewCount="53" Body="&lt;h3&gt;Question:&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Is it possible to get log liklihood values for my stepwise glms?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;Context:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;I am able to get a logliklihood value using lmer with the following model. My study involves unbalanced repeated females, two sites (females don't exchange between sites), 8 predictors, and a response. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    (glmfit1 &amp;lt;- lmer(Response ~ 1 + SITE + (1|SITE:FEMALE) + Variable A + Variable B +&#xA;                        Variable C, data = data))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Removing the repeated females and taking out the Site factor gives me this formula. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(lrfit1 &amp;lt;- glm(Response ~ 1+Variable A + Variable B + Variable C, data = data))&#xA;summary (lrfit1) #just gives me p values for my variables and an AIC but NO LOGLIK.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="4027" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-13T03:49:37.880" LastActivityDate="2011-04-13T03:51:10.630" Title="How to get LogLiklihood value from logistic regression in R" Tags="&lt;r&gt;&lt;likelihood-ratio&gt;" AnswerCount="1" />
  <row Id="9489" PostTypeId="2" ParentId="9488" CreationDate="2011-04-13T03:51:10.630" Score="1" ViewCount="" Body="&lt;p&gt;Have you tried &lt;code&gt;logLik(lrfit1)&lt;/code&gt;?&lt;/p&gt;&#xA;" OwnerUserId="1569" LastActivityDate="2011-04-13T03:51:10.630" CommentCount="1" />
  <row Id="9490" PostTypeId="1" AcceptedAnswerId="9524" CreationDate="2011-04-13T05:49:33.487" Score="2" ViewCount="76" Body="&lt;p&gt;The &lt;em&gt;&lt;strong&gt;Singular Value Decomposition (SVD)&lt;/strong&gt;&lt;/em&gt; of a matrix is&#xA;$$A_{m\times n} = U_{m\times m}\Lambda_{m\times n} V_{n\times n}&amp;#39;$$&#xA;where $U$ and $V$ are orthogonal matrices and $\Lambda$ has (&lt;em&gt;i&lt;/em&gt;, &lt;em&gt;i&lt;/em&gt;) entry $\lambda_i \geq 0$ for $i = 1, 2, \cdots , min(m, n)$ and the other entries are zero. Then the &lt;em&gt;left singular vectors&lt;/em&gt;  $U$  for rows of matrix and &lt;em&gt;right singular vectors&lt;/em&gt; $V$  for columns of matrix can be plotted on the same graph called &lt;em&gt;&lt;strong&gt;bi-plot&lt;/strong&gt;&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm wondering how to do the &lt;em&gt;&lt;strong&gt;SVD&lt;/strong&gt;&lt;/em&gt; of a three dimensional array and plot the singular vectors on the same graph like &lt;em&gt;&lt;strong&gt;bi-plot&lt;/strong&gt;&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="3903" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-13T07:51:09.623" LastActivityDate="2011-04-13T15:51:38.890" Title="Singular value decomposition of a three-dimensional array" Tags="&lt;svd&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="1" />
  <row Id="9491" PostTypeId="1" CreationDate="2011-04-13T06:10:47.197" Score="6" ViewCount="78" Body="&lt;p&gt;I came across the article by &lt;a href=&quot;http://www.utdallas.edu/~herve/Abdi-SVD2007-pretty.pdf&quot; rel=&quot;nofollow&quot;&gt;Hervé Abdi&lt;/a&gt; about generalized SVD. The author mentioned: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The generalized SVD (GSVD) decomposes a rectangular matrix and takes into account constraints imposed on the rows and the columns of the matrix. The GSVD gives a weighted generalized least square estimate of a given matrix by a lower rank matrix and therefore, with an adequate choice of the constraints, the GSVD implements all linear multivariate techniques (e.g., canonical correlation, linear discriminant analysis, correspondence analysis, PLS-regression).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I'm wondering how the GSVD can be related to all linear multivariate techniques (e.g., canonical correlation, linear discriminant analysis, correspondence analysis, PLS-regression).&lt;/p&gt;&#xA;" OwnerUserId="3903" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-13T11:36:57.887" LastActivityDate="2011-05-23T03:29:46.290" Title="Does GSVD implement all linear multivariate techniques?" Tags="&lt;multivariate-analysis&gt;&lt;svd&gt;" AnswerCount="0" FavoriteCount="4" />
  <row Id="9492" PostTypeId="2" ParentId="9477" CreationDate="2011-04-13T08:57:54.630" Score="4" ViewCount="" Body="&lt;p&gt;There's also Fox and Hong's &lt;strong&gt;effects&lt;/strong&gt; package in R.  See the J. Stat. Soft. papers &lt;a href=&quot;http://www.jstatsoft.org/v08/i15/paper&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;http://www.jstatsoft.org/v32/i01/paper&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; for examples with confidence intervals and generating R code.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's not quite as pretty as a ggplot solution, but quite a bit more general, and a lifesaver for moderately complex GLMs.&lt;/p&gt;&#xA;" OwnerUserId="1739" LastActivityDate="2011-04-13T08:57:54.630" CommentCount="1" />
  <row Id="9493" PostTypeId="1" CreationDate="2011-04-13T09:42:14.907" Score="3" ViewCount="21" Body="&lt;p&gt;I have a software application which uses a queue and multiple processors to process those jobs. Jobs get re-run on a daily basis for customers, but we also have new customers signing up regularly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is that customers generally sign up during office hours, and the daily re-running of the jobs simply schedules the job to run at the same time every day. This means that we have built up quite large spikes of work during the day, with very few jobs running overnight.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to even out the spikes in this job queue, but I also want to minimise how far each individual job is moved from it's original time slot. In other words, I would rather move 100 jobs by 5 minutes each than move 1 job by an hour to achieve an even distribution of jobs over the whole 24 hour period.&lt;/p&gt;&#xA;" OwnerUserId="4144" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-04-13T10:32:35.167" LastActivityDate="2011-04-13T10:32:35.167" Title="How can I even out a random distribution while minimising how far each data point is moved?" Tags="&lt;smoothing&gt;&lt;randomness&gt;&lt;queueing&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="9494" PostTypeId="1" CreationDate="2011-04-13T10:01:19.413" Score="2" ViewCount="54" Body="&lt;p&gt;Currently I am working with Text Mining which includes sentiment identification and assigning corresponding business categories using open source tool R. I found these two documents which helped me to some extent:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.jstatsoft.org/v25/i05/&quot; rel=&quot;nofollow&quot;&gt;http://www.jstatsoft.org/v25/i05/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://epub.wu.ac.at/1923/1/document.pdf&quot; rel=&quot;nofollow&quot;&gt;http://epub.wu.ac.at/1923/1/document.pdf&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;My approach is to tokenize the text and then lookup for sentiment and business category. To do this I require positive and negative libraries for sentiment mining and category file which contains word and category. I was able to get positive and negative words but was unable to get a categorization library. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Where can I get a categorization library? &lt;/li&gt;&#xA;&lt;li&gt;Is the above approach appropriate?  Is there a better way to do this?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="4145" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-13T11:34:00.200" LastActivityDate="2011-04-13T11:34:00.200" Title="How to perform text mining, sentiment mining, and business category identification, and where to obtain a categorization library" Tags="&lt;r&gt;&lt;text-mining&gt;" CommentCount="1" FavoriteCount="1" />
  <row Id="9495" PostTypeId="2" ParentId="9483" CreationDate="2011-04-13T10:17:32.607" Score="4" ViewCount="" Body="&lt;p&gt;In most meta-analysis of odds ratios, the standard errors $se_i$ are based on the log odds ratios $log(OR_i)$. So, do you happen to know how your $se_i$ have been estimated (and what metric they reflect? $OR$ or $log(OR)$)? Given that the $se_i$ are based on $log(OR_i)$, then the pooled standard error (under a fixed effect model) can be easily computed. First, let's compute the weights for each effect size: $w_i = \frac{1}{se_i^2}$. Second, the pooled standard error is $se_{FEM} = \sqrt{\frac{1}{\sum w}}$. Furthermore, let $log(OR_{FEM})$ be the common effect (fixed effect model). Then, the (&quot;pooled&quot;) 95% confidence interval is $log(OR_{FEM}) \pm 1.96 \cdot se_{FEM}$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since BIBB kindly provided the data, I am able to run the 'full' meta-analysis in R.  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(meta)&#xA;or &amp;lt;- c(0.75, 0.85)&#xA;se &amp;lt;- c(0.0937, 0.1029)&#xA;logor &amp;lt;- log(or)&#xA;(or.fem &amp;lt;- metagen(logor, se, sm = &quot;OR&quot;))&#xA;&#xA;&amp;gt; (or.fem &amp;lt;- metagen(logor, se, sm = &quot;OR&quot;))&#xA;    OR            95%-CI %W(fixed) %W(random)&#xA;1 0.75  [0.6242; 0.9012]     54.67      54.67&#xA;2 0.85  [0.6948; 1.0399]     45.33      45.33&#xA;&#xA;Number of trials combined: 2 &#xA;&#xA;                         OR           95%-CI       z  p.value&#xA;Fixed effect model   0.7938  [0.693; 0.9092] -3.3335   0.0009&#xA;Random effects model 0.7938  [0.693; 0.9092] -3.3335   0.0009&#xA;&#xA;Quantifying heterogeneity:&#xA;tau^2 &amp;lt; 0.0001; H = 1; I^2 = 0%&#xA;&#xA;Test of heterogeneity:&#xA;    Q d.f.  p.value&#xA; 0.81    1   0.3685&#xA;&#xA;Method: Inverse variance method&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;References&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;See, e.g., &lt;a href=&quot;http://books.google.com/books?id=G-PnRSMxdIoC&amp;amp;lpg=PP1&amp;amp;dq=lipsey%20wilson%20meta&amp;amp;hl=de&amp;amp;pg=PA114#v=onepage&amp;amp;q&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;Lipsey/Wilson (2001: 114)&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="307" LastEditorUserId="307" LastEditorDisplayName="" LastEditDate="2011-04-14T12:44:16.130" LastActivityDate="2011-04-14T12:44:16.130" CommentCount="4" />
  <row Id="9496" PostTypeId="2" ParentId="9493" CreationDate="2011-04-13T10:23:21.593" Score="1" ViewCount="" Body="&lt;p&gt;This is a problem in optimal control.  But you only need a few tools to get a practical solution for your problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) a way to estimate the time distribution of jobs from new customers.  Plotting the hourly, daily, and weekly averages for instances of jobs from new customers will give you a feel for the periodicity of the distribution.  Then you can fit the distribution to a time-dependent Poisson process, with terms for minute/hour/day/week.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) quantification of your &quot;loss function.&quot;  Based on your statement I would imagine that your loss function will include a term penalizing concentrations of jobs in short time periods and a term penalizing time delays on jobs in a nonlinear fashion.  Supposing $t_1,...,t_2$ represents the times at which jobs are executed, and $d_1,...,d_m$ represents the delays for job 1, job 2, etc., your loss function might look like&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;L = k_1 \sum_{i=1}^m (\sum_{j=1}^m I(t_i - t_j &amp;lt; T))^2  + k_2 \sum_{i=1}^m d_i^2&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $k_1$ and $k_2$ are positive constants, and $T$ is a 'critical' time period (like 5 minutes).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) simulation of various control policies (scheduling of recurring jobs) to find their expected loss.  Given a policy of running daily jobs at time $x_1,...,x_l$ you would draw from your distribution of new jobs to get times of new jobs $y_1,...,y_m$ (where $m$ is random).  You can estimate the expected loss by doing the simulation many times and averaging the loss function over the runs.&lt;/p&gt;&#xA;" OwnerUserId="3567" LastActivityDate="2011-04-13T10:23:21.593" />
  <row Id="9497" PostTypeId="2" ParentId="4802" CreationDate="2011-04-13T10:38:55.930" Score="1" ViewCount="" Body="&lt;p&gt;Queue models depend on a few key distributions: the distribution of the &lt;em&gt;time gaps&lt;/em&gt; between incoming jobs, the distribution of service times (how long it takes to process a job).  Some commonly used models for these distributions are the &lt;em&gt;exponential&lt;/em&gt;, &lt;em&gt;gamma&lt;/em&gt;, and &lt;em&gt;Weibull&lt;/em&gt; distributions.  To find out which distribution is appropriate for your situation, you will need to collect data and do some &lt;em&gt;model selection&lt;/em&gt;.  You may also want to model the parameters for the distributions as time-varying, but this makes things much more complicated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: &quot;I found that each query took about 5-15ms with a few taking 1ms and a few taking 600+ms.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Weibull distribution I mentioned before is often used to model &quot;extreme&quot; events.&lt;/p&gt;&#xA;" OwnerUserId="3567" LastActivityDate="2011-04-13T10:38:55.930" />
  <row Id="9499" PostTypeId="1" AcceptedAnswerId="9514" CreationDate="2011-04-13T10:55:07.987" Score="2" ViewCount="57" Body="&lt;p&gt;Hi all,&#xA;I've just started playing with the R forecast package and found I must be doing something wrong because I can't get a decent prediction for a simple sinus.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;weightData &amp;lt;- data.frame(weight = sin(seq(1:100)), week=1:100)&#xA;weight &amp;lt;- as.numeric(weightData$weight)&#xA;predicted &amp;lt;- forecast(weight,h=3,level=95) &#xA;&#xA;# see the predicted values by forecast predicted&#xA;&#xA;myplot &amp;lt;- forecast(weight,h=10,level=95)&#xA;plot(myplot)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And I get a flat prediction. &#xA;I understand the generic forecast methods selects the best method for my data. Isn't that true? Am I missing something?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance!&lt;/p&gt;&#xA;" OwnerUserId="4134" LastActivityDate="2011-04-13T14:14:58.703" Title="Forecast R package producing flat predictions" Tags="&lt;r&gt;&lt;forecasting&gt;" AnswerCount="1" />
  <row Id="9500" PostTypeId="1" AcceptedAnswerId="9522" CreationDate="2011-04-13T10:57:05.660" Score="5" ViewCount="278" Body="&lt;p&gt;You can use the decathlon dataset {FactoMineR} to reproduce this. The question is why the computed eigenvalues differ from those of the covariance matrix&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; library(FactoMineR);data(decathlon)&#xA;&amp;gt; pr &amp;lt;- princomp(decathlon[1:10], cor=F)&#xA;&amp;gt; pr$sd^2&#xA;      Comp.1       Comp.2       Comp.3       Comp.4       Comp.5       Comp.6 &#xA;1.348073e+02 2.293556e+01 9.747263e+00 1.117215e+00 3.477705e-01 1.326819e-01 &#xA;      Comp.7       Comp.8       Comp.9      Comp.10 &#xA;6.208630e-02 4.938498e-02 2.504308e-02 4.908785e-03 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Using another method&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; res&amp;lt;-PCA(decathlon[1:10], scale.unit=FALSE, ncp=5, graph = FALSE)&#xA;&amp;gt; res$eig&#xA;          eigenvalue percentage of variance cumulative percentage of variance&#xA;comp 1  1.348073e+02           79.659589641                          79.65959&#xA;comp 2  2.293556e+01           13.552956464                          93.21255&#xA;comp 3  9.747263e+00            5.759799777                          98.97235&#xA;comp 4  1.117215e+00            0.660178830                          99.63252&#xA;comp 5  3.477705e-01            0.205502637                          99.83803&#xA;comp 6  1.326819e-01            0.078403653                          99.91643&#xA;comp 7  6.208630e-02            0.036687700                          99.95312&#xA;comp 8  4.938498e-02            0.029182305                          99.98230&#xA;comp 9  2.504308e-02            0.014798320                          99.99710&#xA;comp 10 4.908785e-03            0.002900673                         100.00000&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Can you explain to me why the previously computed eigenvalues differ from those ? (the eigenvectors are the same)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; eigen(cov(decathlon[1:10]))$values&#xA; [1] 1.381775e+02 2.350895e+01 9.990945e+00 1.145146e+00 3.564647e-01&#xA; [6] 1.359989e-01 6.363846e-02 5.061961e-02 2.566916e-02 5.031505e-03&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Thank you!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using the alternative &lt;code&gt;prcomp&lt;/code&gt; method..&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; prc &amp;lt;- prcomp(decathlon[1:10])&#xA;&amp;gt; prc$sd^2&#xA; [1] 1.381775e+02 2.350895e+01 9.990945e+00 1.145146e+00 3.564647e-01&#xA; [6] 1.359989e-01 6.363846e-02 5.061961e-02 2.566916e-02 5.031505e-03&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Why do PCA and princomp give different eigenvalues? &lt;/p&gt;&#xA;" OwnerUserId="339" LastEditorUserId="339" LastEditorDisplayName="" LastEditDate="2011-04-13T11:23:12.887" LastActivityDate="2011-04-13T15:39:29.893" Title="Why do the R functions 'PCA' and 'prcomp' give different eigenvalues?" Tags="&lt;r&gt;&lt;pca&gt;" AnswerCount="1" CommentCount="7" FavoriteCount="1" />
  <row Id="9501" PostTypeId="1" AcceptedAnswerId="9576" CreationDate="2011-04-13T11:30:07.083" Score="4" ViewCount="100" Body="&lt;p&gt;Firstly, by analytically integrate, I mean, is there an integration rule to solve this as opposed to numerical analyses (such as trapezoidal, Gauss-Legendre or Simpson's rules)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a function $\newcommand{\rd}{\mathrm{d}}f(x) = x g(x; \mu, \sigma)$ where&#xA;$$&#xD;&#xA;g(x; \mu, \sigma) = \frac{1}{\sigma x \sqrt{2\pi}} e^{-\frac{1}{2\sigma^2}(\log(x) - \mu)^2}&#xD;&#xA;$$&#xA;is the probability density function of a lognormal distribution with parameters $\mu$ and $\sigma$. Below, I'll abbreviate the notation to $g(x)$ and use $G(x)$ for the cumulative distribution function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to calculate the integral&#xA;$$&#xD;&#xA;\int_{a}^{b} f(x) \,\rd x \&amp;gt;. &#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently, I'm doing this with numerical integration using the Gauss-Legendre method. Because I need to run this a large number of times, performance is important. Before I look into optimizing the numerical analyses/other pieces, I would like to know if there are any integration rules to solve this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried applying the integration-by-parts rule, and I got to this, where I'm stuck again,&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;$\int u \,\mathrm{d}v = u v - \int v \mathrm{d}u$.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;$u=x  \implies  \rd u = \rd x$&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;$\rd v = g(x) \rd x  \implies  v = G(x)$&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;$u v - \int v \rd x = x G(x) - \int G(x) \rd x$&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I'm stuck, as I can't evaluate the $\int G(x) \rd x$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is for a software package I'm building.&lt;/p&gt;&#xA;" OwnerUserId="4146" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-04-15T01:07:53.763" LastActivityDate="2011-04-15T02:50:14.760" Title="Is it possible to analytically integrate $x$ multiplied by the lognormal probability density function?" Tags="&lt;distributions&gt;&lt;lognormal&gt;" AnswerCount="1" CommentCount="11" FavoriteCount="1" />
  <row Id="9503" PostTypeId="1" CreationDate="2011-04-13T11:41:57.880" Score="4" ViewCount="33" Body="&lt;p&gt;I am running a regression equation and I want to enter in 12 indepdendent variables then stepwise enter 7 more independent variables and not have an origin.  &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;DV is &lt;code&gt;shfl&lt;/code&gt;. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I  want to enter in the following 12 independent dummy variables&#xA;&lt;code&gt;ajan&#xA;bfeb&#xA;cmar&#xA;dapr&#xA;emay &#xA;fjun&#xA;gjul&#xA;haug&#xA;isep&#xA;joct&#xA;knov&#xA;ldec&lt;/code&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;And then I want to enter in a stepwise fashion &#xA;&lt;code&gt;slag6&#xA;slag7&#xA;slag8&#xA;slag9&#xA;slag10&#xA;slag11&#xA;slag12&lt;/code&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;And finally, I want there to be no origin.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I've done simple regression, but nothing quite like this that enters in the primary variables and step enters several more.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;How can such a model be specified using R?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="4148" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-13T11:53:01.557" LastActivityDate="2011-04-13T13:43:37.057" Title="Multiple regression with no origin and mix of directly entered and stepwise entered variables using R" Tags="&lt;r&gt;&lt;stepwise-regression&gt;" AnswerCount="1" />
  <row Id="9505" PostTypeId="1" AcceptedAnswerId="9543" CreationDate="2011-04-13T12:07:18.300" Score="3" ViewCount="37" Body="&lt;p&gt;I am working on a real-time recommendation engine. At one step, I have a feature that resembles a string-encoded item set, so I took Jaccard on the tokenized string to get a good similarity result. Frankly, Jaccard offers great results, but it takes far too much time on runtime.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My current prototype environment for this is written in Java and I am already using the StringTokenizer to fast transform my string into a token set but the set operations thereafter still take a large amount of time - so my idea was to skip the tokenizing part and work on the string directly, but I am not sure on what metric to count for this puspose.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyone have any ideas or experiences?&lt;/p&gt;&#xA;" OwnerUserId="1158" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-13T12:12:24.520" LastActivityDate="2011-04-14T05:53:49.233" Title="Item set similarity for real-time calculation purposes?" Tags="&lt;java&gt;&lt;recommendations&gt;" AnswerCount="1" />
  <row Id="9506" PostTypeId="1" AcceptedAnswerId="9516" CreationDate="2011-04-13T12:07:56.007" Score="6" ViewCount="140" Body="&lt;p&gt;I am new to R and to time series analysis. I am trying to find the trend of a long (40 years) daily temperature time series and tried to different approximations. First one is just a simple linear regression and second one is Seasonal Decomposition of Time Series by Loess.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the latter it appears that the seasonal component is greater than the trend. But, how do I quantify the trend? I would like just a number telling how strong is that trend.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;     Call:  stl(x = tsdata, s.window = &quot;periodic&quot;)&#xA;     Time.series components:&#xA;        seasonal                trend            remainder               &#xA;Min.   :-8.482470191   Min.   :20.76670   Min.   :-11.863290365      &#xA;1st Qu.:-5.799037090   1st Qu.:22.17939   1st Qu.: -1.661246674 &#xA;Median :-0.756729578   Median :22.56694   Median :  0.026579468      &#xA;Mean   :-0.005442784   Mean   :22.53063   Mean   : -0.003716813 &#xA;3rd Qu.:5.695720249    3rd Qu.:22.91756   3rd Qu.:  1.700826647    &#xA;Max.   :9.919315613    Max.   :24.98834   Max.   : 12.305103891   &#xA;&#xA; IQR:&#xA;         STL.seasonal STL.trend STL.remainder data   &#xA;         11.4948       0.7382    3.3621       10.8051&#xA;       % 106.4          6.8      31.1         100.0  &#xA;     Weights: all == 1&#xA;     Other components: List of 5   &#xA;$ win  : Named num [1:3] 153411 549 365  &#xD;&#xA;    $ deg  : Named int [1:3] 0 1 1   &#xA;$ jump : Named num [1:3] 15342 55 37  &#xD;&#xA;    $ inner: int 2  &#xA;$ outer: int 0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/jwCSr.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="4147" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-13T12:15:51.427" LastActivityDate="2011-04-13T15:43:20.187" Title="STL trend of time series using R" Tags="&lt;r&gt;&lt;time-series&gt;&lt;trend&gt;" AnswerCount="1" FavoriteCount="2" />
  <row Id="9507" PostTypeId="1" AcceptedAnswerId="9564" CreationDate="2011-04-13T12:39:58.633" Score="3" ViewCount="111" Body="&lt;p&gt;As a part of my studies, I’m trying to cluster co-occurrences of URLs and tags in data from Delicious. I found a promising method for this in a paper called “&lt;a href=&quot;http://www.lambda.ee/images/9/93/Semanticsfolksonomies.pdf&quot; rel=&quot;nofollow&quot;&gt;Emergent Semantics from Folksonomies: A Quantitative Study&lt;/a&gt;” (pages 6-13). It used a Separable Mixture Model (SMM, described in the paper “&lt;a href=&quot;http://dspace.mit.edu/bitstream/handle/1721.1/7253/AIM-1625.pdf?sequence=2&quot; rel=&quot;nofollow&quot;&gt;Statistical Models for Co-occurrence Data&lt;/a&gt;” pages 2-4) to model the data and an adapted EM-algorithm to fit the known data to the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I coded the algorithm with Java and ran it with a little piece of real data from Delicious. Unfortunately, the results did not seem correct. The results showed that each tag had equal (although varying from tag to tag) possibility to belong to each concept.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, while this problem could have came from me simply coding the adapted EM-algorithm wrong, I would also like to rule out the possibility of incorrectly initialized variables. This time, since I didn’t know any better way to do it, I simply initialized all the $R_{r\alpha}$ (variables that denote the possibility of co-occurrence $r$ to have raised from concept $\alpha$) to be equal, $1/K$ ($K$ being the number of concepts).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is two-fold.&lt;br&gt;&#xA;Could the flat results come from the flat variable initialization?&lt;br&gt;&#xA;How should I initialize the variables from the EM-algorithm in this case?&lt;/p&gt;&#xA;" OwnerUserId="4141" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-15T08:15:01.263" LastActivityDate="2011-04-20T06:52:07.723" Title="How to initialize EM-algorithm when trying to fit data to a separable mixture model?" Tags="&lt;em-algorithm&gt;" AnswerCount="2" />
  <row Id="9508" PostTypeId="1" CreationDate="2011-04-13T13:09:15.407" Score="3" ViewCount="133" Body="&lt;p&gt;Hello &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm analyzing the results of a hormone manipulation experiment. I measured a number of variables at three times in three groups. The groups are different sizes and not all individuals were measured every time, so I'm using GLMM rather than a repeated-measures ANOVA. I created the model then tested the significance of the terms (time, treatment, and time x treatment) with ANOVA. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm quite new to GLMM, but after doing the tests, further reading suggests that my approach may be inappropriate, particularly with small data sets (I have ~seven animals per group). It seems that there is disagreement about what the degrees of freedom should be. This leads me to three questions:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Is this method acceptable?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) If so, what would be an appropriate method to do post-hoc analyses to determine which groups differ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) Like the fake data, I have a number of negative results. Specifically, I often see significant time effects, but no effect of treatment or treatment X time. If I stick with this method, how can I calculate effects sizes and/or confidence intervals for such tests?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are some fake data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(nlme)&#xA;&#xA;datums&amp;lt;-data.frame(id=rep(1:20,each=3),var1=runif(60,4,6),var2=runif(60,25,30),var3=runif(60,0,1),var4=runif(60,10,15),var.time=rep(1:3,times=20),var.treatment=rep(c('a','b','c'),each=20))&#xA;&#xA;datums$var.time&amp;lt;-as.factor(datums$var.time)&#xA;datums$id&amp;lt;-as.factor(datums$id)&#xA;&#xA;&#xA;#and now the GLMMs on each variable - I'll show just two here&#xA;&#xA;var1.glmm&amp;lt;-lme(var1~var.time + var.treatment + var.time*var.treatment, data=datums, random = ~1| id)&#xA;&#xA;var2.glmm&amp;lt;-lme(var2~var.time + var.treatment + var.time*var.treatment, data=datums, random = ~1|id)&#xA;&#xA;summary(var1.glmm)&#xA;&#xA;anova(var1.glmm)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm aware that the place for me to go is probably the Pinheiro and Bates book, but I don't have access to it at this time. Thanks in advance for any advice. &lt;/p&gt;&#xA;" OwnerUserId="124" LastActivityDate="2011-05-18T10:47:34.463" Title="GLMM: questions of appropriateness, post-hoc tests, effects size, and confidence intervals" Tags="&lt;r&gt;&lt;confidence-interval&gt;&lt;mixed-model&gt;&lt;repeated-measures&gt;&lt;effect-size&gt;" AnswerCount="1" />
  <row Id="9509" PostTypeId="2" ParentId="9483" CreationDate="2011-04-13T13:20:03.047" Score="2" ViewCount="" Body="&lt;p&gt;Actually, you could use software like &lt;a href=&quot;http://bioinformatics.oxfordjournals.org/content/26/17/2190.full&quot; rel=&quot;nofollow&quot;&gt;METAL&lt;/a&gt; which is specifically designed for meta-analyses in GWA context. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's awkward that plink doesn't give the confidence interval. However, you can get the CI because you have the final OR (take $\log(\text{OR})$) and the $p$-value (hence the $z$) for the fixed effect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bernd's method is even more precise.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Beware that I would be more worried about the effect direction as it looks like you only have summary stats for each study but nothing to be sure which is the OR allele. Unless you know it is done on the same allele.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Christian&lt;/p&gt;&#xA;" OwnerUserId="3946" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-13T15:26:01.423" LastActivityDate="2011-04-13T15:26:01.423" />
  <row Id="9510" PostTypeId="1" AcceptedAnswerId="9513" CreationDate="2011-04-13T13:34:06.930" Score="4" ViewCount="127" Body="&lt;p&gt;If I wanted to get the probability of 9 successes in 16 trials with each trial having a probability of 0.6 I could use a binomial distribution. What could I use if each of the 16 trials has a different probability of success?&lt;/p&gt;&#xA;" OwnerUserId="4150" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-14T07:16:48.973" LastActivityDate="2011-04-14T23:19:10.580" Title="Probability distribution for different probabilities" Tags="&lt;distributions&gt;&lt;probability&gt;&lt;binomial&gt;" AnswerCount="2" />
  <row Id="9511" PostTypeId="2" ParentId="9503" CreationDate="2011-04-13T13:43:37.057" Score="3" ViewCount="" Body="&lt;p&gt;I think you can set up your base model, that is the one with your 12 IVs and then use &lt;code&gt;add1()&lt;/code&gt; with the remaining predictors. So, say you have a model &lt;code&gt;mod1&lt;/code&gt; defined like &lt;code&gt;mod1 &amp;lt;- lm(y ~ 0+x1+x2+x3)&lt;/code&gt; (&lt;code&gt;0+&lt;/code&gt; means &lt;em&gt;no intercept&lt;/em&gt;), then&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;add1(mod1, ~ .+x4+x5+x6, test=&quot;F&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;will add and test one predictor after the other on top of the base model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More generally, if you know in advance that a set of variables should be included in the model (this might result from prior knowledge, or whatsoever), you can use &lt;code&gt;step()&lt;/code&gt; or &lt;code&gt;stepAIC()&lt;/code&gt; (in the &lt;code&gt;MASS&lt;/code&gt; package) and look at the &lt;code&gt;scope=&lt;/code&gt; argument.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an illustration, where we specify a priori the functional relationship between the outcome, $y$, and the predictors, $x_1, x_2, \dots, x_{10}$. We want the model to include the first three predictors, but let the selection of other predictors be done by stepwise regression:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(101)&#xA;X &amp;lt;- replicate(10, rnorm(100))&#xA;colnames(X) &amp;lt;- paste(&quot;x&quot;, 1:10, sep=&quot;&quot;)&#xA;y &amp;lt;- 1.1*X[,1] + 0.8*X[,2] - 0.7*X[,5] + 1.4*X[,6] + rnorm(100)&#xA;df &amp;lt;- data.frame(y=y, X)&#xA;&#xA;# say this is one of the base model we think of&#xA;fm0 &amp;lt;- lm(y ~ 0+x1+x2+x3+x4, data=df)&#xA;&#xA;# build a semi-constrained stepwise regression&#xA;fm.step &amp;lt;- step(fm0, scope=list(upper = ~ 0+x1+x2+x3+x4+x5+x6+x7+x8+x9+x10, &#xA;                                lower = ~ 0+x1+x2+x3), trace=FALSE)&#xA;summary(fm.step)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The results are shown below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Coefficients:&#xA;   Estimate Std. Error t value Pr(&amp;gt;|t|)    &#xA;x1   1.0831     0.1095   9.888 2.87e-16 ***&#xA;x2   0.6704     0.1026   6.533 3.17e-09 ***&#xA;x3  -0.1844     0.1183  -1.558    0.123    &#xA;x6   1.6024     0.1142  14.035  &amp;lt; 2e-16 ***&#xA;x5  -0.6528     0.1029  -6.342 7.63e-09 ***&#xA;---&#xA;Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 &#xA;&#xA;Residual standard error: 1.004 on 95 degrees of freedom&#xA;Multiple R-squared: 0.814,  Adjusted R-squared: 0.8042 &#xA;F-statistic: 83.17 on 5 and 95 DF,  p-value: &amp;lt; 2.2e-16 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can see that $x_3$ has been retained in the model, even if it proves to be non-significant (well, the usual caveats with univariate tests in multiple regression setting and model selection apply here -- at least, its relationship with $y$ was not specified).&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-04-13T13:43:37.057" />
  <row Id="9512" PostTypeId="1" CreationDate="2011-04-13T13:48:56.380" Score="2" ViewCount="130" Body="&lt;p&gt;I have a time series data of 30 years and found that ARIMA(0,1,1) has best model among others. I have used the simulate.Arima (forecast package) function to simulate the series into the future.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(forecast)&#xA;&#xA;series &amp;lt;- ts(seq(25,55), start=c(1976,1))&#xA;&#xA;arima_s &amp;lt;- Arima(series, c(0,1,1))&#xA;&#xA;simulate(arima_s, nsim=50, future=TRUE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Later on, i have found the updated value of first forecasted year (i.e. series[31] &amp;lt;- 65). Now i want to simulate the series with this updated value. I am wondering how to do this in R.&lt;/p&gt;&#xA;" OwnerUserId="3084" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-14T11:54:52.443" LastActivityDate="2011-04-14T19:38:27.317" Title="How to update ARIMA forecast in R?" Tags="&lt;r&gt;&lt;time-series&gt;&lt;forecasting&gt;&lt;arima&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="9513" PostTypeId="2" ParentId="9510" CreationDate="2011-04-13T14:03:30.150" Score="4" ViewCount="" Body="&lt;p&gt;This is the sum of 16 (presumably independent) Binomial trials.  The assumption of independence allows us to &lt;em&gt;multiply&lt;/em&gt; probabilities.  Whence, after two trials with probabilities $p_1$ and $p_2$ of success the chance of success on both trials is $p_1 p_2$, the chance of no successes is $(1-p_1)(1-p_2)$, and the chance of one success is $p_1(1-p_2) + (1-p_1)p_2$.  That last expression owes its validity to the fact that the two ways of getting exactly one success are mutually exclusive: at most one of them can actually happen.  That means their probabilities &lt;em&gt;add&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By means of these two rules--independent probabilities multiply and mutually exclusive ones add--you can work out the answers for, say, 16 trials with probabilities $p_1, \ldots, p_{16}$.  To do so, you need to account for all the ways of obtaining each given number of successes (such as 9).  There are $\binom{16}{9} = 11440$ ways to achieve 9 successes.  One of them, for example, occurs when trials 1, 2, 4, 5, 6, 11, 12, 14, and 15 are successes and the others are failures.  The successes had probabilities $p_1, p_2, p_4, p_5, p_6, p_{11}, p_{12}, p_{14},$ and $p_{15}$ and the failures had probabilities $1-p_3, 1-p_7, \ldots, 1-p_{13}, 1-p_{16}$.  Multiplying these 16 numbers gives the chance &lt;em&gt;of this particular sequence of outcomes.&lt;/em&gt;  Summing this number along with the 11,439 remaining such numbers gives the answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course you would use a computer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With many more than 16 trials, there is a need to approximate the distribution.  Provided none of the probabilities $p_i$ and $1-p_i$ get too small, a &lt;a href=&quot;http://en.wikipedia.org/wiki/Binomial_distribution#Normal_approximation&quot; rel=&quot;nofollow&quot;&gt;Normal approximation&lt;/a&gt; tends to work well.  With this method you note that the mean of the $n$ trials is $\mu = (p_1 + p_2 + \cdots + p_n)/n$ and the standard deviation is $\sigma = \sqrt{(p_1(1-p_1) + p_2(1-p_2) + \cdots + p_n(1-p_n)}/n$.  You then pretend the distribution of outcomes is Normal with mean $\mu$ and standard deviation $\sigma$.  The answers tend to be good for computing probabilities corresponding to a proportion of successes that differs from $\mu$ by no more than a few multiples of $\sigma$.  As $n$ grows large this approximation gets ever more accurate and works for even larger multiples of $\sigma$ away from $\mu$.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-04-13T14:03:30.150" CommentCount="5" />
  <row Id="9514" PostTypeId="2" ParentId="9499" CreationDate="2011-04-13T14:14:58.703" Score="3" ViewCount="" Body="&lt;p&gt;No automated forecasting method can produce good forecast for all data. If that was the case, the field of statistics would be for want of a better word dead. The package &lt;strong&gt;forecast&lt;/strong&gt; picks the best exponential smoothing model for the given data. It works best with the certain class of time series data, which is not the case for your data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The precise reason for the failure is evident if you try this (&lt;strong&gt;ets&lt;/strong&gt; is the workhorse behind the &lt;strong&gt;forecast&lt;/strong&gt;):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; ets(weight,&quot;AMM&quot;)&#xA;Erreur dans ets(weight, &quot;AMM&quot;) : Nonseasonal data&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Since you do not supply time series attributes, the forecasting algorithm cannot pickup cyclic component, so the end result is the flat line.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-04-13T14:14:58.703" CommentCount="1" />
  <row Id="9515" PostTypeId="2" ParentId="8562" CreationDate="2011-04-13T14:22:25.767" Score="1" ViewCount="" Body="&lt;p&gt;Riffing off of Thomas' suggestion, I think a multilevel model for this data would look something like this...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;y(hat) = TrialTypeA + TrialTypeB + ModalityA + ModalityB + TaskOrder + TrialTypeA:ModalityA + TrialTypeB:ModalityB + TrialTypeA:TaskOrder + TrialTypeB + TaskOrder&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At a deeper layer in the model each of these variables individual differences can be eliminated in regards to the TrialType variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The As and Bs for TrialType are dummy codes where a value of 0 means the baseline condition and a value of 1 denotes membership in a condition.  E.g. ModalityA might be 0 for VM but be 1 for AV and the ModalityB would be 0 for VM but be 1 for HM.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In R I would specify this model (having loaded the package lme4) as lmer(TrialTypeA + TrialTypeB + ModalityA + ModalityB + TaskOrder + TrialTypeA:ModalityA + TrialTypeB:ModalityB + TrialTypeA:TaskOrder + TrialTypeB + TaskOrder + (1|SubjectID),data=mydata)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good luck.&lt;/p&gt;&#xA;" OwnerUserId="196" LastActivityDate="2011-04-13T14:22:25.767" />
  <row Id="9516" PostTypeId="2" ParentId="9506" CreationDate="2011-04-13T14:26:48.023" Score="4" ViewCount="" Body="&lt;p&gt;I wouldn't bother with &lt;code&gt;stl()&lt;/code&gt; for this - the bandwidth for the lowess smoother used to extract the trend is far, far, to small resulting in the small scale fluctuations you see. I would use an additive model. Here is an example using data and model code from Simon Wood's book on GAMs:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(mgcv)&#xA;require(gamair)&#xA;data(cairo)&#xA;cairo2 &amp;lt;- within(cairo, Date &amp;lt;- as.Date(paste(year, month, day.of.month, &#xA;                                              sep = &quot;-&quot;)))&#xA;plot(temp ~ Date, data = cairo2, type = &quot;l&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/ZxNHU.png&quot; alt=&quot;cairo temperature data&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Fit a model with trend and seasonal components --- &lt;strong&gt;warning&lt;/strong&gt; this is slow:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mod &amp;lt;- gamm(temp ~ s(day.of.year, bs = &quot;cc&quot;) + s(time, bs = &quot;cr&quot;),&#xA;            data = cairo2, method = &quot;REML&quot;,&#xA;            correlation = corAR1(form = ~ 1 | year),&#xA;            knots = list(day.of.year = c(0, 366)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The fitted model looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; summary(mod$gam)&#xA;&#xA;Family: gaussian &#xA;Link function: identity &#xA;&#xA;Formula:&#xA;temp ~ s(day.of.year, bs = &quot;cc&quot;) + s(time, bs = &quot;cr&quot;)&#xA;&#xA;Parametric coefficients:&#xA;            Estimate Std. Error t value Pr(&amp;gt;|t|)    &#xA;(Intercept)  71.6603     0.1523   470.7   &amp;lt;2e-16 ***&#xA;---&#xA;Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 &#xA;&#xA;Approximate significance of smooth terms:&#xA;                 edf Ref.df       F p-value    &#xA;s(day.of.year) 7.092  7.092 555.407 &amp;lt; 2e-16 ***&#xA;s(time)        1.383  1.383   7.035 0.00345 ** &#xA;---&#xA;Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 &#xA;&#xA;R-sq.(adj) =  0.848  Scale est. = 16.572    n = 3780&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and we can visualise the trend and seasonal terms via&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plot(mod$gam, pages = 1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/7B6iy.png&quot; alt=&quot;Cairo fitted trend and seasonal&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and if we want to plot the trend on the observed data we can do that with prediction via:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pred &amp;lt;- predict(mod$gam, newdata = cairo2, type = &quot;terms&quot;)&#xA;ptemp &amp;lt;- attr(pred, &quot;constant&quot;) + pred[,2]&#xA;plot(temp ~ Date, data = cairo2, type = &quot;l&quot;,&#xA;     xlab = &quot;year&quot;,&#xA;     ylab = expression(Temperature ~ (degree*F)))&#xA;lines(ptemp ~ Date, data = cairo2, col = &quot;red&quot;, lwd = 2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/hH4EN.png&quot; alt=&quot;Cairo fitted trend&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or the same for the actual model:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pred2 &amp;lt;- predict(mod$gam, newdata = cairo2)&#xA;plot(temp ~ Date, data = cairo2, type = &quot;l&quot;,&#xA;     xlab = &quot;year&quot;,&#xA;     ylab = expression(Temperature ~ (degree*F)))&#xA;lines(pred2 ~ Date, data = cairo2, col = &quot;red&quot;, lwd = 2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/15BYc.png&quot; alt=&quot;Cairo fitted model&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is just an example, and a more in-depth analysis might have to deal with the fact that there are a few missing data, but the above should be a good starting point.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As to your point about how to quantify the trend - well that is a problem, because the trend is not linear, neither in your &lt;code&gt;stl()&lt;/code&gt; version nor the GAM version I show. If it were, you could give the rate of change (slope). If you want to know by how much has the estimated trend changed over the period of sampling, then we can use the data contained in &lt;code&gt;pred&lt;/code&gt; and compute the difference between the start and the end of the series in the &lt;strong&gt;trend&lt;/strong&gt; component only:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; tail(pred[,2], 1) - head(pred[,2], 1)&#xA;    3794 &#xA;1.756163&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;so temperatures are, on average, 1.76 degrees warmer than at the start of the record.&lt;/p&gt;&#xA;" OwnerUserId="1390" LastEditorUserId="1390" LastEditorDisplayName="" LastEditDate="2011-04-13T15:43:20.187" LastActivityDate="2011-04-13T15:43:20.187" CommentCount="5" />
  <row Id="9517" PostTypeId="1" AcceptedAnswerId="9521" CreationDate="2011-04-13T13:02:01.697" Score="4" ViewCount="72" Body="&lt;p&gt;I work for a commission-based company that gives accounts to employees for about a month and the employees try and resolve the account, if they are successful they get a commission, otherwise the account goes to another employee to try and resolve. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We are running some reports to measure some performance characteristics for the company. We have years of data that we can analyze.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What summary(median or mean) can best describe the central tendency of length of time to resolve an account per employee?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Almost every employee's median is greater than their mean, some are further apart than others. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One thing that we are trying to find out is if we give an employee an account, how long should it take for him or her to resolve it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems to me that median is a better summary.&lt;/p&gt;&#xA;" OwnerUserId="4170" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-13T14:52:39.297" LastActivityDate="2011-04-13T15:19:35.777" Title="Whether to use mean or median to summarise the central tendency of length of time to perform a task" Tags="&lt;average&gt;" AnswerCount="2" CommentCount="4" FavoriteCount="1" />
  <row Id="9518" PostTypeId="2" ParentId="9517" CreationDate="2011-04-13T13:49:45.377" Score="0" ViewCount="" Body="&lt;p&gt;I will advise you to use low and up ends of the confidence interval centered at mean to describe the central tendency of the length. $Median&amp;gt;mean$ seems to be a evidence that there is a peak value smaller then the median and mean is better. You problem is not so mathematical (since you didn't provide a criteria of optimality) - that's why I think that one can only provide his opinion with some arguments rather than a strict solution.&lt;/p&gt;&#xA;" LastActivityDate="2011-04-13T13:49:45.377" CommentCount="1" />
  <row Id="9519" PostTypeId="1" CreationDate="2011-04-13T14:57:38.620" Score="3" ViewCount="48" Body="&lt;p&gt;I am creating a term to describe a design that may not be the proper term (&quot;Staggered within subjects design&quot;).  What I mean by this is one can imagine a within subjects design with 3 levels.  Each participant provides data for two levels, and between participant the order in which the data is collected is counter-balanced.  For example (condition order is in the cells of the table below; sorry for the table formatting, I did my best, but I think you'll get the gist):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Participant Condition A     Condition B     Condition C&#xA;1                     1               2 &#xA;2                     2               1 &#xA;3                                     1               2&#xA;4                                     2               1&#xA;5                     1                               2&#xA;6                     2                               1&#xA;7                     1               2 &#xA;8                     2               1 &#xA;9                                     1               2&#xA;10                                    2               1&#xA;11                    1                               2&#xA;12                    2                               1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Does this sort of design have a name?  What are the consequences of such a design?  How would one analyze it?  I understand that I could probably use a mixed model to analyze this data but I'm curious about the application of ANOVA/t-test style analyses to this sort of data as these statistical approaches are more familiar to many in my field.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The rationale for such a design might be that you have limited time with each participant and so you can't run them through all three conditions however each participant must be rewarded the same whether they participate in 1 condition or 2 conditions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems like, with special consideration, that this data could be run as a between subjects ANOVA (is the same as between subjects t-tests) or a within-subjects ANOVA.  I will assert points, and I would appreciate it if the person to answer this question would accept or reject these points.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;I could conduct a within Subjects anova on this data comparing any two conditions crossing condition with trial order also as a within-subjects factor (I expect that the trial order effects will be non-significant).  &lt;/li&gt;&#xA;&lt;li&gt;If there were main effects of order it would influence the validity of my estimate of the exact amount of effect of being demonstrated by each condition but would not influence subsequent analyses.  &lt;/li&gt;&#xA;&lt;li&gt;If there were interactions between a condition and order I should not conduct any between subjects analyses on data coming from order 2.&lt;/li&gt;&#xA;&lt;li&gt;I could run two between subjects t-tests for each pair of conditions. One for the data from order 1 and the other from order 2 (as using data from both in a single between subjects analysis would violate the assumption of independence).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;If I could run all of these statistics, using the same data I would have 3 statistical tests.  Each of the between subjects tests would be a full replication of the comparison between condition A and B.  The within subjects tests would validate my hypothesis that trial order does not matter and allow the between subjects tests to be applied.  If I fail to find significance in my between subjects analysis my within subjects analysis may provide additional power which will allow me to evaluate whether my hypothesis is sufficiently supported to run an additional study or whether I'm barking up the wrong tree.&lt;/p&gt;&#xA;" OwnerUserId="196" LastActivityDate="2011-04-14T06:50:16.403" Title="Is there literature on staggered within subjects designs? What are the consequences of such a design?" Tags="&lt;anova&gt;&lt;within-subjects&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="9520" PostTypeId="2" ParentId="4473" CreationDate="2011-04-13T15:01:29.147" Score="4" ViewCount="" Body="&lt;p&gt;You use the example &quot;January, February, and March&quot;, and I hope that's more for illustration and it isn't literally all the data you have. For monthly data, you really should have 3+ years of data, and places like the Census Bureau won't touch a monthly series with less than 7 years of data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, you don't mention exactly what data you have for each month, but it really makes a difference. Simply taking a single sales number for each month and fitting a straight line through them (lm) isn't really much of a model. It might give you an idea of the overall trend, but not realistic predictions. If you have monthly data on other factors that are primary drivers of sales, it might work. Then, as chl says, if you want to have an idea if your approach will work, you need to look at the residuals (&lt;code&gt;resid (lm_model)&lt;/code&gt;) with commands like &lt;code&gt;acf&lt;/code&gt; and &lt;code&gt;qqnorm&lt;/code&gt; to see if the errors from your model are basically random -- which is what &lt;code&gt;lm&lt;/code&gt; assumes -- or if they have a pattern to them, in which case your model is missing something important.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Obviously, the ultimate test is how well your model predicts on data that it has not seen, over the long run, but with small amounts of data or the wrong data you can fool yourself with an initial lucky &quot;prediction&quot; and really get embarrassed down the road when the data and approach inevitably fail.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Usual models for univariate time series (such as monthly sales figures and no other data) would involve more complicated models that take some experience to fit, such as &lt;code&gt;arima&lt;/code&gt; or the &lt;code&gt;forecast&lt;/code&gt; package.&lt;/p&gt;&#xA;" OwnerUserId="1764" LastActivityDate="2011-04-13T15:01:29.147" />
  <row Id="9521" PostTypeId="2" ParentId="9517" CreationDate="2011-04-13T15:19:35.777" Score="5" ViewCount="" Body="&lt;p&gt;You should use the median, not the mean. However, you'll need to use methods appropriate for &lt;a href=&quot;http://en.wikipedia.org/wiki/Survival_analysis&quot; rel=&quot;nofollow&quot;&gt;time-to-event (survival)&lt;/a&gt; data that deal appropriately with &lt;a href=&quot;http://en.wikipedia.org/wiki/Censoring_%28statistics%29&quot; rel=&quot;nofollow&quot;&gt;censoring&lt;/a&gt;: if the account was handled to another employee without being resolved you know only that the time &lt;em&gt;this&lt;/em&gt; employee &lt;em&gt;would&lt;/em&gt; have taken to resolve it is greater than or equal to the observed time for which they handled the account, so the observed time is right-censored.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The appropriate method would be to construct the &lt;a href=&quot;http://en.wikipedia.org/wiki/Kaplan-Meier_estimator&quot; rel=&quot;nofollow&quot;&gt;Kaplan–Meier estimator&lt;/a&gt; of the survival function. If you want a single number for each employee, you could use this method to obtain the median 'survival' time, which gives you the median time taken to resolve an account. You can also get confidence intervals for the median if you so wish. It's possible that the median may not be estimable for some employees though if they fail to resolve more than half the accounts before they are re-assigned, in which case you could consider switching to some other percentile that is estimable for all (or at least the great majority of) employees.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Kaplan-Meier estimator isn't difficult -- it's perfectly possible, if somewhat tedious, to construct by hand, and straightforward to program. Confidence intervals are a bit more tricky, but are available in any decent statistical software package.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-04-13T15:19:35.777" CommentCount="3" />
  <row Id="9522" PostTypeId="2" ParentId="9500" CreationDate="2011-04-13T15:31:24.723" Score="3" ViewCount="" Body="&lt;p&gt;As pointed out in the comments, it's because &lt;code&gt;princomp&lt;/code&gt; multiplies the covariance matrix by &lt;code&gt;(1-1/n.obs)&lt;/code&gt;, as you can see in the source (the relevant snippet is below).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;else if (is.null(covmat)) {&#xA;    dn &amp;lt;- dim(z)&#xA;    if (dn[1L] &amp;lt; dn[2L]) &#xA;        stop(&quot;'princomp' can only be used with more units than variables&quot;)&#xA;    covmat &amp;lt;- cov.wt(z)&#xA;    n.obs &amp;lt;- covmat$n.obs&#xA;    cv &amp;lt;- covmat$cov * (1 - 1/n.obs)&#xA;    cen &amp;lt;- covmat$center&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can avoid this multiplication by specifying the &lt;code&gt;covmat&lt;/code&gt; argument instead of the &lt;code&gt;x&lt;/code&gt; argument.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;princomp(covmat=cov(iris[,1:4]))$sd^2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1657" LastEditorUserId="1657" LastEditorDisplayName="" LastEditDate="2011-04-13T15:39:29.893" LastActivityDate="2011-04-13T15:39:29.893" CommentCount="7" />
  <row Id="9523" PostTypeId="2" ParentId="9190" CreationDate="2011-04-13T15:39:52.810" Score="1" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://queue.acm.org/detail.cfm?id=1805128&quot; rel=&quot;nofollow&quot;&gt;A Tour through the Visualization Zoo&lt;/a&gt; (Heer et al., Visualization&#xA; 8(5) 2010) offers a particularly interesting overview of &quot;innovative&quot; and interactive techniques for displaying data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On a related point, a good software for data visualization, including the aforementioned gallery, is &lt;a href=&quot;http://vis.stanford.edu/protovis/&quot; rel=&quot;nofollow&quot;&gt;Protovis&lt;/a&gt;, which comes with a lot of &lt;a href=&quot;http://vis.stanford.edu/protovis/ex/&quot; rel=&quot;nofollow&quot;&gt;examples&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-04-13T15:39:52.810" CommunityOwnedDate="2011-04-13T15:39:52.810" />
  <row Id="9524" PostTypeId="2" ParentId="9490" CreationDate="2011-04-13T15:51:38.890" Score="4" ViewCount="" Body="&lt;p&gt;There are several notions of decomposition of such a tensor. Last year I asked &lt;a href=&quot;http://www.mapleprimes.com/questions/97377-Decompose-A-Third-Order-Tensor-Into-Pure-Tensors&quot; rel=&quot;nofollow&quot;&gt;essentially the same question&lt;/a&gt; on the MaplePrimes site, answered it myself by referring to &lt;a href=&quot;http://en.wikipedia.org/wiki/Higher-order_singular_value_decomposition&quot; rel=&quot;nofollow&quot;&gt;wikipedia&lt;/a&gt;, and provided an implementation for one of those notions (the CANDECOMP/PARAFAC decomposition) in a &lt;a href=&quot;http://www.mapleprimes.com/posts/97861-CANDECOMPPARAFAC-Images-As-Tensors-Of-Order-3&quot; rel=&quot;nofollow&quot;&gt;follow-up post&lt;/a&gt; (applied to decomposing the $3\times m \times n$ tensor given by the R,G,B entries of an image).&lt;/p&gt;&#xA;" OwnerUserId="2898" LastActivityDate="2011-04-13T15:51:38.890" CommentCount="5" />
  <row Id="9525" PostTypeId="2" ParentId="6949" CreationDate="2011-04-13T17:16:56.247" Score="1" ViewCount="" Body="&lt;p&gt;Fraud detection is a rare class problem. &lt;a href=&quot;http://cseweb.ucsd.edu/~elkan/291/&quot; rel=&quot;nofollow&quot;&gt;Chapter Six of Charles Elkan's Notes&lt;/a&gt; for his Graduate Course in Data Mining and Predictive Analytics at UCSD walks you through the prediction of a rare class, and the pitfalls and proper ways to evaluate the success of such a model. The methods he specifically uses are Isotonic and Univariate Logistic Regression. The software he uses in the class is Rapidminer, but I prefer R. If you choose to use R, you can perform both of these functions using the isoreg and glm functions. Many people also like to use SVMs in fraud detection, but part of the model selection criterion should be the speed with which you need to validate the transactions. If, for example, it's the swipe of a credit card, SVMs are wholly unfeasible because it will take far too long to process. This is why, in production environments, variants on regression models are typically used for fraud detection.&lt;/p&gt;&#xA;" OwnerUserId="2166" LastActivityDate="2011-04-13T17:16:56.247" />
  <row Id="9526" PostTypeId="1" CreationDate="2011-04-13T17:52:53.233" Score="2" ViewCount="50" Body="&lt;p&gt;I'm working on a project that will convert $USD to other currencies.  The system will use average monthly exchange rates instead of daily because of pre-existing limitations.  I need to calculate an estimate of the amount of error that will result from using monthly average.  My idea is to&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Download historical daily exchange rates for several years&lt;/li&gt;&#xA;&lt;li&gt;Calculate the monthly average for each month&lt;/li&gt;&#xA;&lt;li&gt;Subtract the monthly average from each daily amount that month&lt;/li&gt;&#xA;&lt;li&gt;Find the standard deviation for the differences&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I would report the standard deviation as the expected error amount.  Is this a valid approach?&lt;/p&gt;&#xA;" OwnerUserId="4154" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-15T08:18:51.293" LastActivityDate="2011-04-15T08:18:51.293" Title="Calculate error for monthly exchange rates" Tags="&lt;estimation&gt;&lt;standard-deviation&gt;" CommentCount="2" />
  <row Id="9527" PostTypeId="2" ParentId="9474" CreationDate="2011-04-13T18:07:07.070" Score="1" ViewCount="" Body="&lt;p&gt;Ha.  It &lt;a href=&quot;http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1308339&quot; rel=&quot;nofollow&quot;&gt;looks like&lt;/a&gt; this is an unsolved problem in applied statistics.  The best MRC estimators execute in $n^2log(n)$ time or worse, so of course no code is available.&lt;/p&gt;&#xA;" OwnerUserId="4110" LastActivityDate="2011-04-13T18:07:07.070" />
  <row Id="9528" PostTypeId="2" ParentId="9431" CreationDate="2011-04-13T18:13:24.987" Score="1" ViewCount="" Body="&lt;p&gt;You can either estimate a parametric model of the error using MLE, or you can use a semi-paramteric approach based on something like the maximal rank correlation (MRC) estimator.  Computationally, MRC is prohibitive for large samples, so it looks like MLE is the right approach for me.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks to GaBorgulya for some good, prompt direction, especially on the term &quot;misclassification error.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are some good sources on the topic:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.sciencedirect.com/science?_ob=ArticleURL&amp;amp;_udi=B6VC0-3V72S8B-B&amp;amp;_user=99318&amp;amp;_coverDate=12%2F31%2F1998&amp;amp;_rdoc=1&amp;amp;_fmt=high&amp;amp;_orig=gateway&amp;amp;_origin=gateway&amp;amp;_sort=d&amp;amp;_docanchor=&amp;amp;view=c&amp;amp;_searchStrId=1714202295&amp;amp;_rerunOrigin=scholar.google&amp;amp;_acct=C000007678&amp;amp;_version=1&amp;amp;_urlVersion=0&amp;amp;_userid=99318&amp;amp;md5=0db67a1d6b1fb454e3ffb9d6c4afb75a&amp;amp;searchtype=a&quot; rel=&quot;nofollow&quot;&gt;The basic model, exactly as described in the original problem&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://faculty.smu.edu/millimet/classes/eco7377/papers/hausman%20et%20al.pdf&quot; rel=&quot;nofollow&quot;&gt;Ungated version of the same&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.jstor.org/stable/20076198&quot; rel=&quot;nofollow&quot;&gt;A more complicated, but more general model&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.jstor.org/stable/2696516&quot; rel=&quot;nofollow&quot;&gt;A nice overview&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="4110" LastActivityDate="2011-04-13T18:13:24.987" />
  <row Id="9532" PostTypeId="1" CreationDate="2011-04-13T21:09:57.260" Score="5" ViewCount="52" Body="&lt;p&gt;This is a statistical version of my &lt;a href=&quot;http://math.stackexchange.com/questions/30731/will-two-convex-hulls-overlap&quot; &gt;Math.SE &lt;/a&gt; post.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given natural numbers $b$ and $r$, uniformly randomly choose $b+r$ points within a unit square. Call the $b$ points the &lt;i&gt;blue points&lt;/i&gt; and the $r$ points &lt;i&gt; red points&lt;/i&gt;. Let $p(b,r)$ be the probability that $H_r$, the convex hull of red points, overlaps with $H_b$, that of the blue points. How do I efficiently estimate $p(b,r)$ via Monte Carlo (or perhaps better) methods?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can think of averaging the probability over a large number of randomly chosen test cases, but how can I sample the search space in a way that gives me error bounds?  &lt;/p&gt;&#xA;" OwnerUserId="4011" LastActivityDate="2011-04-13T21:09:57.260" Title="Monte carlo estimation of convex hull overlap probability" Tags="&lt;monte-carlo&gt;" />
  <row Id="9533" PostTypeId="1" CreationDate="2011-04-13T21:38:03.710" Score="4" ViewCount="103" Body="&lt;p&gt;I have 150 observations, 500 features, and I am interested in novelty detection (outlier detection): given a new observation (let's say 'patient') I want to know if it is different from the previous ones (let's call it 'control'). If I had a lot of data, I would probably be using statistical testing at the univariate parameter level, but, because of multiple testing issues, I end up exploring the tails of the control distribution to achieve significance, and I do not have enough data for non parametric tests for such small p-values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am doing one class SVMs that alleviate this issue by learning a global decision strategy. The limitations of this approach are&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;it is very 'blackboxy'&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;it works poorly if the data is very 'anisotropic', i.e. the marginal distributions of the control are very dissimilar in different directions.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;A trick to work around problem 2 is to center an norm the univariate parameters (this is often called creating 'Z scores'). Ideally, one would like to whiten the data using the control covariance, but there is not enough data to compute it. The values fed in the OC-SVM then can be interpreted as a univariate test statistic (under a normal null distribution for the controls).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my case, I can see from the histograms that the control's distribution is heavy tailed. I would like to learn a univariate transform making it closer to a standard normal. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;By the way, I have no reference about such practices. I have learned them empirically, and from lab discussions. Any pointer would be welcome, even if they don't directly answer my question.&lt;/p&gt;&#xA;" OwnerUserId="1265" LastEditorUserId="1265" LastEditorDisplayName="" LastEditDate="2011-04-14T05:35:23.347" LastActivityDate="2011-04-27T15:10:30.253" Title="Learning a univariate transform (kernel?) for novelty detection" Tags="&lt;kernel&gt;&lt;outliers&gt;&lt;learning&gt;" AnswerCount="2" CommentCount="5" />
  <row Id="9534" PostTypeId="2" ParentId="9512" CreationDate="2011-04-13T21:54:17.267" Score="3" ViewCount="" Body="&lt;p&gt;As it happens, I encountered similar problem recently. Here is the function which takes existing arima model and applies it to new data. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;adjust.amodel &amp;lt;- function(object,extended) {&#xA;   km.mod &amp;lt;- makeARIMA(object$model$phi,object$model$theta,object$model$Delta)&#xA;   km.res &amp;lt;- KalmanRun(extended,km.mod)&#xA;&#xA;   object$x &amp;lt;- extended&#xA;   object$residuals &amp;lt;- ts(km.res$resid,start=start(extended),end=end(extended),frequency=frequency(extended))&#xA;   object$model &amp;lt;- km.mod&#xA;   object&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In your case here is how you should use it:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;series[31] &amp;lt;- 65&#xA;adj_arima_s &amp;lt;- adjust.amodel(arima_s,series)&#xA;simulate(adj_arima_s, nsim=50, future=TRUE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The usual caveats apply though. You need to have good reasons to do that. If the data changes this means that the model should change, so what you are doing is ignoring the new information and sticking to the old model, which might be the wrong one. You can compare this to producing the model fit by plucking the model coefficients out of the blue air. Although the coefficients have statistical validation, it comes from the old data, so interpretation of the results should take this into consideration.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update:&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It turns out that the &lt;strong&gt;Arima&lt;/strong&gt; function has an argument for supplying old model:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;adj_s &amp;lt;- Arima(series,model=arima_s)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The end result might be the same for both approaches, but I would advise using second one, because it clearly is tested more thoroughly.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-04-14T19:38:27.317" LastActivityDate="2011-04-14T19:38:27.317" CommentCount="1" />
  <row Id="9535" PostTypeId="1" AcceptedAnswerId="9538" CreationDate="2011-04-14T00:40:22.830" Score="3" ViewCount="146" Body="&lt;p&gt;I've run across a couple guides suggesting that I use R's nlm for maximum likelihood estimation.  But none of them (including &lt;a href=&quot;http://stat.ethz.ch/R-manual/R-devel/library/stats/html/nlm.html&quot; rel=&quot;nofollow&quot;&gt;R's documentation&lt;/a&gt;) gives much theoretical guidance for when to use or not use the function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as I can tell, nlm is just doing gradient descent along the lines of Newton's method.  Are there principles for when it's reasonable to use this approach? What alternatives are available? Also, are there limits on the size of the arrays, etc. one can pass to nlm?&lt;/p&gt;&#xA;" OwnerUserId="4110" LastActivityDate="2011-04-14T03:00:07.563" Title="When should I *not* use R's nlm function for MLE?" Tags="&lt;r&gt;&lt;maximum-likelihood&gt;" AnswerCount="2" FavoriteCount="3" />
  <row Id="9536" PostTypeId="1" AcceptedAnswerId="9554" CreationDate="2011-04-14T01:33:55.987" Score="3" ViewCount="67" Body="&lt;p&gt;I am sick of using the examples in the book. Is there an easy place to find data for which z-score/percentile/normal distribution stuff would be easy to see?&lt;/p&gt;&#xA;" OwnerUserId="1490" LastActivityDate="2011-04-14T10:55:13.367" Title="Where can I find good publicly available data that I could use to teach z-scores to my college students?" Tags="&lt;dataset&gt;" AnswerCount="2" FavoriteCount="2" />
  <row Id="9537" PostTypeId="2" ParentId="9535" CreationDate="2011-04-14T02:27:08.703" Score="5" ViewCount="" Body="&lt;p&gt;When to use and not to use any particular method of maximization depends to a great extent on the type of data you have. &lt;code&gt;nlm&lt;/code&gt; will work just fine if the likelihood surface isn't particularly &quot;rough&quot; and is everywhere differentiable. &lt;code&gt;nlminb&lt;/code&gt; provides a way to constrain parameter values to particular bounding boxes. &lt;code&gt;optim&lt;/code&gt;, which is probably the most-used optimizer, provides a few different optimization routines; for example, BFGS, L-BFGS-B, and simulated annealing (via the SANN option), the latter of which might be handy if you have a difficult optimizing problem. There are also a number of optimizers available on CRAN. &lt;code&gt;rgenoud&lt;/code&gt;, for instance, provides a genetic algorithm for optimization. &lt;code&gt;DEoptim&lt;/code&gt; uses a different genetic optimization routine. Genetic algorithms can be slow to converge, but are usually guaranteed to converge (in time) even when there are discontinuities in the likelihood. I don't know about &lt;code&gt;DEoptim&lt;/code&gt;, but &lt;code&gt;rgenoud&lt;/code&gt; is set up to use &lt;code&gt;snow&lt;/code&gt; for parallel processing, which helps somewhat.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, a probably somewhat unsatisfactory answer is that you should use &lt;code&gt;nlm&lt;/code&gt; or any other optimizer if it works for the data you have. If you have a well-behaved likelihood, any of the routines provided by &lt;code&gt;optim&lt;/code&gt; or &lt;code&gt;nlm&lt;/code&gt; will give you the same result. Some may be faster than others, which may or may not matter, depending on the size of the dataset, etc. As for the number of parameters these routines can handle, I don't know, though it's probably quite a few. Of course, the more parameters you have, the more likely you are to run into problems with convergence.&lt;/p&gt;&#xA;" OwnerUserId="3265" LastActivityDate="2011-04-14T02:27:08.703" />
  <row Id="9538" PostTypeId="2" ParentId="9535" CreationDate="2011-04-14T03:00:07.563" Score="6" ViewCount="" Body="&lt;p&gt;There are a number of general-purpose optimization routines in base R that I'm aware of: &lt;code&gt;optim&lt;/code&gt;, &lt;code&gt;nlminb&lt;/code&gt;, &lt;code&gt;nlm&lt;/code&gt; and &lt;code&gt;constrOptim&lt;/code&gt; (which handles linear inequality constraints, and calls &lt;code&gt;optim&lt;/code&gt; under the hood). Here are some things that you might want to consider in choosing which one to use.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;code&gt;optim&lt;/code&gt; can use a number of different algorithms including conjugate gradient, Newton, quasi-Newton, Nelder-Mead and simulated annealing. The last two don't need gradient information and so can be useful if gradients aren't available or not feasible to calculate (but are likely to be slower and require more parameter fine-tuning, respectively). It also has an option to return the computed Hessian at the solution, which you would need if you want standard errors along with the solution itself.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;code&gt;nlminb&lt;/code&gt; uses a quasi-Newton algorithm that fills the same niche as the &lt;code&gt;&quot;L-BFGS-B&quot;&lt;/code&gt; method in &lt;code&gt;optim&lt;/code&gt;. In my experience it seems a bit more robust than &lt;code&gt;optim&lt;/code&gt; in that it's more likely to return a solution in marginal cases where &lt;code&gt;optim&lt;/code&gt; will fail to converge, although that's likely problem-dependent. It has the nice feature, if you provide an explicit gradient function, of doing a numerical check of its values at the solution. If these values don't match those obtained from numerical differencing, &lt;code&gt;nlminb&lt;/code&gt; will give a warning; this helps to ensure you haven't made a mistake in specifying the gradient (easy to do with complicated likelihoods).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;code&gt;nlm&lt;/code&gt; only uses a Newton algorithm. This can be faster than other algorithms in the sense of needing fewer iterations to reach convergence, but has its own drawbacks. It's more sensitive to the shape of the likelihood, so if it's strongly non-quadratic, it may be slower or you may get convergence to a false solution. The Newton algorithm also uses the Hessian, and computing that can be slow enough in practice that it more than cancels out any theoretical speedup.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="1569" LastActivityDate="2011-04-14T03:00:07.563" />
  <row Id="9539" PostTypeId="1" CreationDate="2011-04-14T03:20:16.033" Score="1" ViewCount="29" Body="&lt;p&gt;I'm familiar with the diagnostics required for OLS, however I'm in new territory with a model I'm fitting to data in R, using Poisson regression with GLM.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are the standard methods in evaluating a WLS model?&lt;/p&gt;&#xA;" OwnerUserId="1965" LastActivityDate="2011-04-14T03:20:16.033" Title="How do I go about conducting model diagnostics on WLS?" Tags="&lt;r&gt;&lt;modeling&gt;&lt;diagnostic&gt;" CommentCount="2" />
  <row Id="9540" PostTypeId="2" ParentId="7164" CreationDate="2011-04-14T04:00:01.310" Score="1" ViewCount="" Body="&lt;p&gt;The generalization of the change of variable formula to the non-bijective case is generally hard to write out explicitly, check &lt;a href=&quot;http://en.wikipedia.org/wiki/Probability_density_function#Multiple_variables&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Probability_density_function#Multiple_variables&lt;/a&gt;&#xA;which essentially formalizes mpiktas's suggestion&lt;/p&gt;&#xA;" OwnerUserId="4163" LastActivityDate="2011-04-14T04:00:01.310" />
  <row Id="9541" PostTypeId="1" CreationDate="2011-04-14T04:13:36.247" Score="1" ViewCount="93" Body="&lt;p&gt;I am trying to determine model significance using:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1-pchisq(null deviance-residual deviance, null df- residual df)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have 5 models:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Four models were estimated with GLMs, which gave me null and residual DFs in the summary. &lt;/li&gt;&#xA;&lt;li&gt;The fifth model was estimated with the &lt;code&gt;lmer&lt;/code&gt; function because of the nested structure of my data. This does not give me degrees of freedom for null or residuals. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;How do I get my null and residual deviances and degrees of freedoms?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks! &lt;/p&gt;&#xA;" OwnerUserId="4027" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-19T14:30:05.650" LastActivityDate="2011-05-19T14:47:57.887" Title="How can I obtain null and residual deviance/degrees of freedom for assessing model significance?" Tags="&lt;r&gt;&lt;statistical-significance&gt;&lt;degrees-of-freedom&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="9542" PostTypeId="1" CreationDate="2011-04-14T04:51:15.753" Score="7" ViewCount="181" Body="&lt;p&gt;I know that in a regression situation, if you have a set of highly correlated variables this is usually &quot;bad&quot; because of the instability in the estimated coefficients (variance goes toward infinity as determinant goes towards zero).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is whether this &quot;badness&quot; persists in a PCA situation.  Do the coefficients/loadings/weights/eigenvectors for any particular PC become unstable/arbitrary/non-unique as the covariance matrix becomes singular?  I am particularly interested in the case where only the first principal component is retained, and all others are dismissed as &quot;noise&quot; or &quot;something else&quot; or &quot;unimportant&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't think that it does, because you will just be left with a few principal components which have zero, or close to zero variance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Easy to see this isn't the case in the simple extreme case with 2 variables - suppose they are perfectly correlated.  Then the first PC will be the exact linear relationship, and the second PC will be perpindicular to the first PC, with all PC values equal to zero for all observations (i.e. zero variance).  Wondering if its more general.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-14T07:19:10.440" LastActivityDate="2011-04-14T14:57:16.657" Title="Is PCA unstable under multicollinearity?" Tags="&lt;pca&gt;&lt;multicollinearity&gt;" AnswerCount="1" CommentCount="3" FavoriteCount="1" />
  <row Id="9543" PostTypeId="2" ParentId="9505" CreationDate="2011-04-14T05:53:49.233" Score="2" ViewCount="" Body="&lt;p&gt;You can use &lt;a href=&quot;http://en.wikipedia.org/wiki/MinHash&quot; rel=&quot;nofollow&quot;&gt;MinHashing&lt;/a&gt; to get a fast approximate jacard similiarity match for your current item set against a database of existing of item sets. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might use a few min hashes to find quickly find candidate recommendations, and only do the full jacard computation against only the candidates found via min hashing.&lt;/p&gt;&#xA;" OwnerUserId="4164" LastActivityDate="2011-04-14T05:53:49.233" />
  <row Id="9544" PostTypeId="1" AcceptedAnswerId="9563" CreationDate="2011-04-14T05:58:16.167" Score="6" ViewCount="195" Body="&lt;p&gt;Let $A\in\mathbb{R}^{n \times n}$ be a dense symmetric positive-definite matrix (the $X^TX$ from &lt;a href=&quot;http://stats.stackexchange.com/questions/9341/regularized-fit-from-summarized-data&quot;&gt;here&lt;/a&gt;) and $b$ a vector in $\mathbb{R}^n$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to compute $A^{-1}b$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Two questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Could you recommend an efficient and numerically stable algorithm for computing $A^{-1}b$ for $n \approx 1000$?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Let $\tilde{A_i}$ denote the matrix obtained from $A$ by removing its $i$-th row and $i$-th column. Is there an algorithm that, having been allowed to pre-process $A$ in some way, would enable me to quickly compute $\tilde{A_i}^{-1}\tilde b$ for any $i \in \{1,2,\ldots,n\}$ and any $\tilde b \in \mathbb{R}^{n-1}$?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="439" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-14T07:18:21.407" LastActivityDate="2011-05-12T12:01:23.310" Title="Computing $(X^TX)^{-1}X^Ty$ in OLS" Tags="&lt;regression&gt;&lt;least-squares&gt;&lt;matrix-inverse&gt;" AnswerCount="3" FavoriteCount="3" />
  <row Id="9545" PostTypeId="2" ParentId="9510" CreationDate="2011-04-14T06:48:46.820" Score="1" ViewCount="" Body="&lt;p&gt;One alternative to @whuber's normal approximation is to use &quot;mixing&quot; probabilities, or a hierarchical model.  This would apply when the $p_i$ are similar in some way, and you can model this by a probability distribution $p_i\sim Dist(\theta)$ with a density function of $g(p|\theta)$ indexed by some parameter $\theta$.  you get a integral equation:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Pr(s=9|n=16,\theta)={16 \choose 9}\int_{0}^{1} p^{9}(1-p)^{7}g(p|\theta)dp $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The binomial probability comes from setting $g(p|\theta)=\delta(p-\theta)$, the normal approximation comes from (I think) setting $g(p|\theta)=g(p|\mu,\sigma)=\frac{1}{\sigma}\phi\left(\frac{p-\mu}{\sigma}\right)$ (with $\mu$ and $\sigma$ as defined in @whuber's answer) and then noting the &quot;tails&quot; of this PDF fall off sharply around the peak.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could also use a beta distribution, which would lead to a simple analytic form, and which need not suffer from the &quot;small p&quot; problem that the normal approximation does - as beta is quite flexible.  Using a $beta(\alpha,\beta)$ distribution with $\alpha,\beta$ set by the solutions to the following equations (this is the &quot;mimimum KL divergence&quot; estimates):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{\psi(\alpha)-\psi(\alpha+\beta)}{\alpha}=\frac{1}{n}\sum_{i=1}^{n}log[p_{i}]$$&#xA;$$\frac{\psi(\beta)-\psi(\alpha+\beta)}{\beta}=\frac{1}{n}\sum_{i=1}^{n}log[1-p_{i}]$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $\psi(.)$ is the digamma function - closely related to harmonic series.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We get the &quot;beta-binomial&quot; compound distribution:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$${16 \choose 9}\frac{1}{B(\alpha,\beta)}\int_{0}^{1} p^{9+\alpha-1}(1-p)^{7+\beta-1}dp ={16 \choose 9}\frac{B(\alpha+9,\beta+7)}{B(\alpha,\beta)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This distribution converges towards a normal distribution in the case that @whuber points out - but should give reasonable answers for small $n$ and skewed $p_i$ - but not for multimodal $p_i$, as beta distribution only has one peak.  But you can easily fix this, by simply using $M$ beta distributions for the $M$ modes.  You break up the integral from $0&amp;lt;p&amp;lt;1$ into $M$ pieces so that each piece has a unique mode (and enough data to estimate parameters), and fit a beta distribution within each piece. then add up the results, noting that making the change of variables $p=\frac{x-L}{U-L}$ for $L&amp;lt;x&amp;lt;U$ the beta integral transforms to:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$B(\alpha,\beta)=\int_{L}^{U}\frac{(x-L)^{\alpha-1}(U-x)^{\beta-1}}{(U-L)^{\alpha+\beta-1}}dx$$&lt;/p&gt;&#xA;" OwnerUserId="2392" LastEditorUserId="2392" LastEditorDisplayName="" LastEditDate="2011-04-14T23:19:10.580" LastActivityDate="2011-04-14T23:19:10.580" />
  <row Id="9546" PostTypeId="2" ParentId="9519" CreationDate="2011-04-14T06:50:16.403" Score="2" ViewCount="" Body="&lt;p&gt;I'd probably call this a &lt;a href=&quot;http://en.wikipedia.org/wiki/Crossover_design&quot; rel=&quot;nofollow&quot;&gt;cross-over design&lt;/a&gt; with three treatments and two periods, although you could also think of it as a &lt;a href=&quot;http://en.wikipedia.org/wiki/Incomplete_block_design&quot; rel=&quot;nofollow&quot;&gt;balanced incomplete block design&lt;/a&gt; with three treatments and 'participant' as the blocking factor with a block size of 2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Not my area of expertise so I can't accept or reject all your points but I'm sure you can analyse it with ANOVA. One point that comes to mind is, as for any designed experiment, don't forget to randomize, and &lt;a href=&quot;http://en.wikipedia.org/wiki/Allocation_concealment#Allocation_concealment&quot; rel=&quot;nofollow&quot;&gt;conceal allocation&lt;/a&gt;! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Two standard textbooks on crossover trials are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Jones, Byron; Kenward, Michael G. (2003). &lt;a href=&quot;http://books.google.com/books?id=Cc3oTtld0F4C&quot; rel=&quot;nofollow&quot;&gt;&lt;em&gt;Design and Analysis of Cross-Over Trials (2nd ed.).&lt;/em&gt;&lt;/a&gt; Chapman and Hall. &lt;/li&gt;&#xA;&lt;li&gt;Senn, Stephen (2002). &lt;a href=&quot;http://books.google.com/books?id=Mx9FCKnRnVQC&quot; rel=&quot;nofollow&quot;&gt;&lt;em&gt;Cross-over trials in clinical research (2nd ed.).&lt;/em&gt;&lt;/a&gt; Wiley.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-04-14T06:50:16.403" CommentCount="1" />
  <row Id="9547" PostTypeId="1" CreationDate="2011-04-14T07:54:22.887" Score="1" ViewCount="36" Body="&lt;ul&gt;&#xA;&lt;li&gt;When measuring the quantization error of a clustering, should the distance between samples/centroid be squared or not? I found both variants in the literature.&lt;/li&gt;&#xA;&lt;li&gt;Furthermore, is (squared) quantization error not the same as squared sum of errors?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="4166" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-14T07:59:37.813" LastActivityDate="2011-04-14T13:47:33.077" Title="Measuring quantization error for clustering - squared or not?" Tags="&lt;clustering&gt;" AnswerCount="1" />
  <row Id="9548" PostTypeId="2" ParentId="9536" CreationDate="2011-04-14T09:38:51.977" Score="2" ViewCount="" Body="&lt;p&gt;One good website where you can find data for different disciplines.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://robjhyndman.com/TSDL/&quot; rel=&quot;nofollow&quot;&gt;Rob Hyndman's Time Series Data library&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="3084" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-14T10:48:29.207" LastActivityDate="2011-04-14T10:48:29.207" />
  <row Id="9549" PostTypeId="1" CreationDate="2011-04-14T09:43:45.847" Score="5" ViewCount="176" Body="&lt;p&gt;Assume you have a class of approximately 800 students and following a set of assessments each student has a raw grade.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;How should these raw grades be converted into a final grade?&lt;/li&gt;&#xA;&lt;li&gt;Is it a good idea to scale the raw grades to a normal distribution?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="4167" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-14T10:41:27.343" LastActivityDate="2011-04-14T12:54:08.067" Title="Should grades be assigned to students based on a normal distribution?" Tags="&lt;normal-distribution&gt;" AnswerCount="1" CommentCount="8" FavoriteCount="1" />
  <row Id="9550" PostTypeId="1" CreationDate="2011-04-14T10:10:03.263" Score="1" ViewCount="70" Body="&lt;p&gt;I've got more than 20 (10 point likert scale) variables with more than 1000 entries each. What I want to do is compare the means of the answers on the questions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A one-way anova seems suitable for this, but you can only categorize by the values of a variabele. I want to categorize by question, the variable itself.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any way to do this without having to put all answer beneath each other and having another variable saying which question it is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just to be clear, I've got:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;variable: question1 --&gt; 1000 entries ranging from 1 to 10&lt;/p&gt;&#xA;&#xA;&lt;p&gt;variable: question2 --&gt; 1000 entries ranging from 1 to 10&lt;/p&gt;&#xA;&#xA;&lt;p&gt;...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;variable: question20 --&gt; 1000 entries ranging from 1 to 10&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to compare the means of the different questions with one-way anova but I can't choose to factor by question.&lt;/p&gt;&#xA;" OwnerUserId="4168" LastActivityDate="2011-04-14T10:34:50.687" Title="Compare means of different variables" Tags="&lt;anova&gt;&lt;spss&gt;" AnswerCount="1" />
  <row Id="9551" PostTypeId="2" ParentId="9544" CreationDate="2011-04-14T10:32:32.233" Score="5" ViewCount="" Body="&lt;p&gt;The standard answer to your first question is &lt;strong&gt;Cholesky decomposition&lt;/strong&gt;. To quote &lt;a href=&quot;http://en.wikipedia.org/wiki/Cholesky_decomposition#Applications&quot; rel=&quot;nofollow&quot;&gt;the Wikipedia article&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;If $A$ is symmetric and positive definite, then we can solve $Ax = b$ [for $x$] by first computing the Cholesky decomposition $A = LL^\mathrm{T}$, then solving $Ly = b$ for $y$, and finally solving $L^\mathrm{T}  x = y$ for $x$.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I'm not really clear what you're after in your second question. Doesn't a solution to the first question also provide a solution to the second? Surely it's not difficult computationally to remove a row and column from a matrix, and surely that's all the 'pre-proccessing' required??&lt;/p&gt;&#xA;" OwnerUserId="449" LastEditorUserId="449" LastEditorDisplayName="" LastEditDate="2011-04-14T10:41:05.347" LastActivityDate="2011-04-14T10:41:05.347" />
  <row Id="9552" PostTypeId="2" ParentId="9550" CreationDate="2011-04-14T10:34:50.687" Score="4" ViewCount="" Body="&lt;ul&gt;&#xA;&lt;li&gt;If the same 1000 participants answered each question:&#xA;&lt;ul&gt;&#xA;&lt;li&gt;consider using paired samples t-tests and repeated measures ANOVAs&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;If a different 1000 participants answered each question:&#xA;&lt;ul&gt;&#xA;&lt;li&gt;then it sounds like you need to restructure your data file to conform to the expectations of the statistical package you are using.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastActivityDate="2011-04-14T10:34:50.687" CommentCount="1" />
  <row Id="9553" PostTypeId="2" ParentId="9542" CreationDate="2011-04-14T10:44:19.680" Score="2" ViewCount="" Body="&lt;p&gt;PCA is often a means to an ends; leading up to either inputs to a multiple regression or for use in a cluster analysis. I think in your case, you are talking about using the results of a PCA to perform a regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In that case, your objective of performing a PCA is to get rid of mulitcollinearity and get orthogonal inputs to a multiple regression, not surprisingly this is called Principal Components Regression. Here, if all your original inputs were orthogonal then doing a PCA would give you another set of orthogonal inputs. Therefore; if you are doing a PCA, one would assume that your inputs have multicollinearity. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given the above, you would want to do PCA to get a few input variable from a problem that has a number of inputs. To determine how many of those new orthogonal variables you should retain, a scree plot is often used (Johnson &amp;amp; Wichern, 2001, p. 445). If you have a large number of observations, then you could also use the rule of thumb that with $\hat{ \lambda_{i} }$ as the $i^{th}$ largest estimated eigenvalue only use up to and including those values where $\frac{ \hat{ \lambda_{i} } }{p}$ are greater than or equal to one (Johnson &amp;amp; Wichern, 2001, p. 451).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;References&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Johnson &amp;amp; Wichern (2001). &lt;em&gt;Applied Multivariate Statistical Analysis&lt;/em&gt; (6th Edition). Prentice Hall.&lt;/p&gt;&#xA;" OwnerUserId="3805" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-14T10:57:59.350" LastActivityDate="2011-04-14T10:57:59.350" CommentCount="2" />
  <row Id="9554" PostTypeId="2" ParentId="9536" CreationDate="2011-04-14T10:55:13.367" Score="2" ViewCount="" Body="&lt;p&gt;You may wish to read answers to this existing question on &lt;a href=&quot;http://stats.stackexchange.com/questions/7/locating-freely-available-data-samples&quot;&gt;freely available datasets&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, I imagine that you'd want a dataset with some interesting metric variables.&#xA;In psychology research methods classes that I've taught, we've often looked at datasets with intelligence or personality test scores.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want a personality example, I have some &lt;a href=&quot;https://github.com/jeromyanglim/Sweave_Personality_Reports&quot; rel=&quot;nofollow&quot;&gt;personality data and metadata on github&lt;/a&gt; based on the &lt;a href=&quot;http://ipip.ori.org/&quot; rel=&quot;nofollow&quot;&gt;IPIP&lt;/a&gt;, an public domain measure of the Big 5 factors of personality.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/jeromyanglim/Sweave_Personality_Reports&quot; rel=&quot;nofollow&quot;&gt;github repository home&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/jeromyanglim/Sweave_Personality_Reports/blob/master/data/ipip.tsv&quot; rel=&quot;nofollow&quot;&gt;data&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/jeromyanglim/Sweave_Personality_Reports/blob/master/meta/ipipmeta.tsv&quot; rel=&quot;nofollow&quot;&gt;metadata&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://blog.revolutionanalytics.com/2010/12/how-to-create-pdf-reports-with-r.html&quot; rel=&quot;nofollow&quot;&gt;David Smith's summary&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastActivityDate="2011-04-14T10:55:13.367" />
  <row Id="9555" PostTypeId="2" ParentId="9544" CreationDate="2011-04-14T10:57:09.693" Score="6" ViewCount="" Body="&lt;p&gt;To add to @onestop's answer, another efficient way is to use &lt;a href=&quot;http://en.wikipedia.org/wiki/QR_decomposition&quot; rel=&quot;nofollow&quot;&gt;QR decomposition&lt;/a&gt;. The added benefit is that QR decomposition can be applied directly to $X$, and not to $X^TX$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think the QR decomposition can be made to work for your second question, it is definitely straightforward for column removal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However the question is why do you need it? There are &lt;a href=&quot;http://en.wikipedia.org/wiki/Automatically_Tuned_Linear_Algebra_Software&quot; rel=&quot;nofollow&quot;&gt;readily available libraries&lt;/a&gt; which perform these operations very efficiently. Do you want to reimplement them? As far as I understand there is a lot of non-trivial fine-tuning of code even if the algorithm is basically the best for the job, so chances are pretty high that you might not get full benefits by implementing algorithm which is supposedly theoretically superior.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the &lt;a href=&quot;http://books.google.com/books?id=ZecsDBMz5-IC&amp;amp;pg=PA132&quot; rel=&quot;nofollow&quot;&gt;link to the chapter in a book&lt;/a&gt;, which discusses modifications needed for solving the second question in case of QR decomposition. Although it states that the methods apply for least squares problems, it can be applied for system of linear equations. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am pretty sure that this should be a standard problem, so maybe someone will give a more suitable reference.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="830" LastEditorDisplayName="" LastEditDate="2011-05-12T12:01:23.310" LastActivityDate="2011-05-12T12:01:23.310" CommentCount="10" />
  <row Id="9556" PostTypeId="1" CreationDate="2011-04-14T11:20:00.523" Score="1" ViewCount="100" Body="&lt;p&gt;I have 2 genes (tf1 and tf2), which are affecting a third gene (tg). By affecting a gene I mean the changes in the value of tf1 and tf2 changes the value of tg. what we measure in the whole experiment is this value. We want to see if the values of tf1 and tf2 are dependent (which means they are regulating each other) and if (tf1,tg) and (tf2,tg) are dependent. The response of the tf1 and tf1 and the tg differs in different conditions. So my 3 factors are tf1, tf2, and conditions.   &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;First I want to make sure that the design of my 3way-anova is correct. One dimension is the conditions.If we see the second and third factors (tf1 and tf2) as a matrix, I have designed it like this.&lt;br&gt;&#xA;|tg  |tf1          |&lt;br&gt;&#xA;|tf2 |func(tf1,tf2)|  &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;so in the first column, there is no tf1, and in the secong column we always see its effect. In the first row there is no tf2, but in the second row we see its effect. I thought that it would be good idea to fill the cell in the first row, first column with the tg. and the last cell has an effect of both of tf1 and tf2 (I combined the values of tf1 and tf1 with a function like average).&lt;br&gt;&#xA;I appreciate if you tell me what you think about this design.  &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;If a:tf1, b:tf2, and c:condition, which of the SSs shows if there is a dependency between tf1 and tf2? Is it SSab? &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The reason that I am asking is that I don't see much of a difference in the histogram of the SSab for samples in which there is an interaction between a and b with the samples that there is no interaction between a and b.  &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Also, what is the interpretation of SSabc for my particular problem?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="2885" LastEditorUserId="2885" LastEditorDisplayName="" LastEditDate="2011-04-28T17:27:47.293" LastActivityDate="2011-04-28T17:27:47.293" Title="Interpretation of 3-way ANOVA" Tags="&lt;anova&gt;&lt;genetics&gt;&lt;interpretation&gt;&lt;bioinformatics&gt;&lt;biostatistics&gt;" CommentCount="3" />
  <row Id="9557" PostTypeId="1" AcceptedAnswerId="9558" CreationDate="2011-04-14T12:07:36.113" Score="3" ViewCount="108" Body="&lt;p&gt;I was recently exposed to some statistical hypothesis testing methods (e.g. Friedman test) at work, and I would like to increase my knowledge on the topic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can you suggest a good introduction to statistical significance / statistical hypothesis testing for a computer scientist? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am thinking of a PDF book or similar, but any other kind of help is welcome.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: I've already found &lt;a href=&quot;http://www.jerrydallal.com/LHSP/LHSP.HTM&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; website but I was looking preferably for some which is easily printable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you&lt;br /&gt;&#xA;Tunnuz&lt;/p&gt;&#xA;" OwnerUserId="4169" LastEditorUserId="4169" LastEditorDisplayName="" LastEditDate="2011-04-14T12:17:05.553" LastActivityDate="2011-04-14T12:19:03.930" Title="What is a good introduction to  statistical hypothesis testing for computer scientists?" Tags="&lt;hypothesis-testing&gt;&lt;statistical-significance&gt;&lt;p-value&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="9558" PostTypeId="2" ParentId="9557" CreationDate="2011-04-14T12:19:03.930" Score="4" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://greenteapress.com/thinkstats/&quot; rel=&quot;nofollow&quot;&gt;http://greenteapress.com/thinkstats/&lt;/a&gt;&#xA;This seems like it would be useful for you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Full disclosure: I have not read it, but I am working my way through the Think Like a Computer Scientist in Java, and am finding that extremely useful.&lt;/p&gt;&#xA;" OwnerUserId="656" LastActivityDate="2011-04-14T12:19:03.930" CommentCount="3" />
  <row Id="9559" PostTypeId="2" ParentId="9549" CreationDate="2011-04-14T12:54:08.067" Score="2" ViewCount="" Body="&lt;p&gt;Why should grades be normally distributed?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sometimes they are but if the grades are not normally distributed then the bell curve grading system, where the middle say 70% get C's, is probably not a good one to base grades off of.  Although that grading is pretty harsh, few instructors would actually do it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Use distributions to describe the data, don't transform data to fit a particular distribution (although transformations can be helpful at times).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you use the bell curve grading system and, extreme case, everyone aces the class.  How do you decide grades?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is how I would decide final grades:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;90-100%:  A&lt;/p&gt;&#xA;&#xA;&lt;p&gt;80-90%:  B&lt;/p&gt;&#xA;&#xA;&lt;p&gt;...&lt;/p&gt;&#xA;" OwnerUserId="2310" LastActivityDate="2011-04-14T12:54:08.067" />
  <row Id="9560" PostTypeId="2" ParentId="9547" CreationDate="2011-04-14T13:47:33.077" Score="1" ViewCount="" Body="&lt;p&gt;The point of the squared error is that it results out of the underlying assumption, that your data is distributed with a Gaussian random component. (Like noise on your measurements, e.g.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The sum of squares error comes from the log probability. Say your points are distributed according to $p$, then you want to pick your parameters $\theta$ (for K-Means the centroids, e.g.) to maximize the probability of the data:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p(D) = \Pi_i p(x_i|\theta)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, maximization of products is hard, so we maximize the log instead, which makes the product to a sum:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$argmax_{\theta} \sum_i \log p(x_i|\theta)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, the density of the Gaussian distribution is someting like $\frac{1}{\sqrt{2\pi\sigma^2}}\,e^{ -\frac{(x-\mu)^2}{2\sigma^2} }$. If you take the log of this, you are stuck with what you find in the exponent and some constant terms. You ignore the variance in most cases. And voilá, that's where the squared distance comes from.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since the Gaussian punishes outliers very much (because of the square), sometimes it is more robust to assume a distribution that has heavier tails, like student's-t or Laplace.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you only measure the absolute distance, this comes from a Laplace assumption.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, it is a question that the statistician has to answer - it's more part of the model. Of course, you can use a model selection method (like cross validation) that does this part of the job.&lt;/p&gt;&#xA;" OwnerUserId="2860" LastActivityDate="2011-04-14T13:47:33.077" />
  <row Id="9561" PostTypeId="1" AcceptedAnswerId="9569" CreationDate="2011-04-14T14:26:53.683" Score="5" ViewCount="149" Body="&lt;p&gt;This is an elementary question, but I wasn't able to find the answer.  I have two measurements: n1 events in time t1 and n2 events in time t2, both produced (say) by Poisson processes with possibly-different lambda values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is actually from a news article, which essentially claims that since $n_1/t_1\neq n_2/t_2$ that the two are different, but I'm not sure that the claim is valid.  Suppose that the time periods were not chosen maliciously (to maximize the events in one or the other).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can I just do a &lt;em&gt;t&lt;/em&gt;-test, or would that not be appropriate?  The number of events is too small for me to comfortably call the distributions approximately normal.&lt;/p&gt;&#xA;" OwnerUserId="1378" LastActivityDate="2011-04-16T01:47:49.140" Title="Checking if two Poisson samples have the same mean" Tags="&lt;hypothesis-testing&gt;&lt;poisson&gt;" AnswerCount="3" CommentCount="3" FavoriteCount="1" />
  <row Id="9562" PostTypeId="2" ParentId="9533" CreationDate="2011-04-14T14:28:33.387" Score="2" ViewCount="" Body="&lt;p&gt;Your setting is pretty hard. I have no solution, but a couple of points.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Energy based models can give you a scalar corresponding to a &quot;grade of belief&quot; that an input is generated by the distribution of your data. It comes down to chosing a model and a good loss function. Check out &lt;a href=&quot;http://yann.lecun.com/exdb/publis/pdf/lecun-06.pdf&quot; rel=&quot;nofollow&quot;&gt;Yann Lecun's tutorial on energy based models&lt;/a&gt;. Also, there is &lt;a href=&quot;http://yann.lecun.com/exdb/publis/pdf/ranzato-unsup-07.pdf&quot; rel=&quot;nofollow&quot;&gt;Ranzato's energy based unsupervised framework paper&lt;/a&gt; where they use sparse autoencoders. Sparseness is generally desirable given your tiny dataset, I guess.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;A restricted Boltzmann machine might work. You can train your RBM on the data with less than 25 hidden features (which you might anyway because of your lack of data) enabling you to write down the probability of any new input to belong to the data given by your distribution. Actually, an RBM is a energy based model as well.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I have a feeling that SVM with Kernels might be a too complex model for what you are doing. Do you get acceptable scores on a test set?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="2860" LastActivityDate="2011-04-14T14:28:33.387" CommentCount="2" />
  <row Id="9563" PostTypeId="2" ParentId="9544" CreationDate="2011-04-14T14:34:05.890" Score="5" ViewCount="" Body="&lt;p&gt;Regarding your second question, here is a way to do this for $n=i$, for simplicity of notation. Let $\alpha = A_{nn}, \, a = (A_{1n},\dots, A_{n-1,n})^T$ and therefore&#xA;$$ A = \begin{pmatrix} \tilde A &amp;amp; a \\ a^T &amp;amp; \alpha \end{pmatrix}&#xD;&#xA;$$&#xA;Also let $A^{-1}$ be partitioned in the same way,&#xA;$$ A^{-1} = \begin{pmatrix} \tilde C &amp;amp; c \\ c^T &amp;amp; \gamma \end{pmatrix}&#xD;&#xA;$$&#xA;I understand you want $\tilde A^{-1}$ and have already computed $A^{-1}$. Set $d = \tilde A^{-1}a$ and $\delta = c^Ta$. These quantities already appear in $A^{-1}$, since&#xA;$$ A^{-1} = \begin{pmatrix} \tilde A^{-1} + \frac{1}{\alpha - \delta}dd^T &amp;amp; -\frac{1}{\alpha - \delta} d \\ -\frac{1}{\alpha - \delta} d^T &amp;amp; \frac{1}{\alpha - \delta} \end{pmatrix}&#xD;&#xA;$$&#xA;as a direct calculation shows. Therefore,&#xA;$$&#xD;&#xA;\tilde A^{-1}  = \tilde C - \frac{1}{\alpha - \delta} dd^T = \tilde C - \frac{1}{\gamma}cc^T &#xD;&#xA;$$&#xA;which is an $O(n^2)$ update of $A^{-1}$. &lt;/p&gt;&#xA;" OwnerUserId="4062" LastActivityDate="2011-04-14T14:34:05.890" CommentCount="1" />
  <row Id="9564" PostTypeId="2" ParentId="9507" CreationDate="2011-04-14T14:35:38.870" Score="1" ViewCount="" Body="&lt;p&gt;Whenever I implement a new algorithm, I get myself an easy, interpretable dataset on which I can try it out. This has several advantages, for example runtime (use a small dataset) or visualization (make things you want to plot have dimension 2 or 3).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course the behavior you see could result from the dataset. It feels however unlikely to me. However, a standard way to initialize variables is to just randomize them uniformly in an interval like [-1, 1], [-0.1, 0.1] or something like that. I have seen standard normal as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not sure if it works for SMMs, but for Gaussian Mixture Models and PCA mixtures, it makes sense to run a few iterations of K-Means before you go into EM (and use the centers as responsibilities). Maybe you want to try that.&lt;/p&gt;&#xA;" OwnerUserId="2860" LastActivityDate="2011-04-14T14:35:38.870" CommentCount="1" />
  <row Id="9566" PostTypeId="1" CreationDate="2011-04-14T14:48:57.507" Score="8" ViewCount="134" Body="&lt;p&gt;I have the total number of calls received each week and have plotted them on a chart, going back nearly 3 years.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By eye it seems that there was a massive drop over Christmas, that doesn't seem to have recovered, it seems that there has been a step change in requests.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a test I can do that can quantify this difference?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cheers&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ben&lt;/p&gt;&#xA;" OwnerUserId="4171" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-15T03:54:14.787" LastActivityDate="2011-04-16T01:50:04.197" Title="Determining if change in a time series is statistically significant" Tags="&lt;time-series&gt;&lt;statistical-significance&gt;&lt;change-point&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="2" />
  <row Id="9567" PostTypeId="2" ParentId="9561" CreationDate="2011-04-14T14:56:48.807" Score="5" ViewCount="" Body="&lt;p&gt;You're looking for a quick and easy check.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Under the null hypothesis that the rates (lambda values) are equal, say to $\lambda$, then you could view the two measurements as observing a single process for time $t = t_1+t_2$ and counting the events during the interval $[0, t_1]$ ($n_1$ in number) and the events during the interval $[t_1, t_1+t_2]$ ($n_2$ in number).  You would estimate the rate as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\hat{\lambda} = \frac{n_1+n_2}{t_1+t_2}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and from that you can estimate the distribution of the $n_i$: they are Poisson of intensity near $t_i\hat{\lambda}$.  If one or both $n_i$ are situated on tails of this distribution, most likely the claim is valid; if not, the claim may be relying on chance variation.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-04-14T14:56:48.807" CommentCount="1" />
  <row Id="9569" PostTypeId="2" ParentId="9561" CreationDate="2011-04-14T14:59:25.827" Score="6" ViewCount="" Body="&lt;p&gt;To test the Poisson mean, the conditional method was proposed by Przyborowski and Wilenski (1940). The conditional distribution of X1 given X1+X2 follows a binomial distribution&#xA;whose success probability is a function of the ratio two lambda. Therefore,&#xA;hypothesis testing and interval estimation procedures can be readily developed from&#xA;the exact methods for making inferences about the binomial success probability.&#xA;There usually two methods are considered for this purpose,&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;C-test&lt;/li&gt;&#xA;&lt;li&gt;E-test&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;You can find the details about these two tests in this paper.&#xA;&lt;a href=&quot;http://www.ucs.louisiana.edu/~kxk4695/JSPI-04.pdf&quot; rel=&quot;nofollow&quot;&gt;A more powerful test for comparing two Poisson&#xA;means&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="3084" LastActivityDate="2011-04-14T14:59:25.827" CommentCount="1" />
  <row Id="9570" PostTypeId="2" ParentId="9566" CreationDate="2011-04-14T19:29:37.493" Score="8" ViewCount="" Body="&lt;p&gt;A very similar example is used in the tutorial of PyMC. If you assume that the daily amount of requests was constant until some point in time (maybe exactly Christmas) and after that it was constant again, all you need to do is substitute the numbers in the example: &lt;a href=&quot;http://pymc.googlecode.com/svn/doc/tutorial.html&quot; rel=&quot;nofollow&quot; rel=&quot;nofollow&quot;&gt;http://pymc.googlecode.com/svn/doc/tutorial.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As this is the Bayesian approach you won't (easily) get p values. However, the size of the step down and its &lt;a href=&quot;http://en.wikipedia.org/wiki/Credibility_interval&quot; rel=&quot;nofollow&quot; rel=&quot;nofollow&quot;&gt;credible interval&lt;/a&gt; (this is a Bayesian interval, similar to a confidence interval) may be equally useful.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-16T01:50:04.197" LastActivityDate="2011-04-16T01:50:04.197" CommentCount="2" />
  <row Id="9571" PostTypeId="2" ParentId="9561" CreationDate="2011-04-14T20:09:55.573" Score="3" ViewCount="" Body="&lt;p&gt;I would be more interested in a confidence interval than a p value, here is a bootstrap approximation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Calculating the lengths of the intervals first, and a check:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Lrec = as.numeric(as.Date(&quot;2010-07-01&quot;) - as.Date(&quot;2007-12-02&quot;)) # Length of recession&#xA;Lnrec = as.numeric(as.Date(&quot;2007-12-01&quot;) - as.Date(&quot;2001-12-01&quot;)) # L of non rec period&#xA;(43/Lrec)/(50/Lnrec)&#xA;&#xA;[1] 2.000276&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This check gives a slightly different result (100.03% increase) than the one of the publication (101% increase). Go on with the bootstrap (do it twice):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;N = 100000&#xA;k=(rpois(N, 43)/Lrec)/(rpois(N, 50)/Lnrec)&#xA;c(quantile(k, c(0.025, .25, .5, .75, .975)), mean=mean(k), sd=sd(k))&#xA;&#xA;     2.5%       25%       50%       75%     97.5%      mean        sd &#xA;1.3130094 1.7338545 1.9994599 2.2871373 3.0187243 2.0415132 0.4355660 &#xA;&#xA;     2.5%       25%       50%       75%     97.5%      mean        sd &#xA;1.3130094 1.7351970 2.0013578 2.3259023 3.0173868 2.0440240 0.4349706 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The 95% confidence interval of the increase is 31% to 202%.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-16T01:47:49.140" LastActivityDate="2011-04-16T01:47:49.140" />
  <row Id="9572" PostTypeId="2" ParentId="8007" CreationDate="2011-04-14T20:58:55.363" Score="0" ViewCount="" Body="&lt;p&gt;If you are interested in the amount of time it takes to complete an order, it seems that a duration analysis (aka survival or event history analysis) would be most appropriate. See the Wikipedia entry for an overview:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Survival_analysis&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Survival_analysis&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This introduction, which covers issues such as censoring, looks relevant and accessible:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.amstat.org/chapters/northeasternillinois/pastevents/presentations/summer05_Ibrahim_J.pdf&quot; rel=&quot;nofollow&quot;&gt;Survival Analysis Introduction&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And if you are so inclined, R has a task view dedicated to survival analysis:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://cran.r-project.org/web/views/Survival.html&quot; rel=&quot;nofollow&quot;&gt;R Survival Analysis Task View&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since you know pretty well what steps go into the production of each item, and because you seem interested in forecasting, you may begin by estimating a parametric model, such as a Weibull or log-logistic/log-normal. Most software capable of estimating these models will also provide the tools to forecast average time-to-completion for different orders. You should also be able to produce plots of estimated durations.&lt;/p&gt;&#xA;" OwnerUserId="3265" LastActivityDate="2011-04-14T20:58:55.363" />
  <row Id="9573" PostTypeId="1" AcceptedAnswerId="9575" CreationDate="2011-04-14T21:55:43.747" Score="10" ViewCount="309" Body="&lt;p&gt;Long ago I learnt that normal distribution was necessary to use a two sample T-test. Today a colleague told me that she learnt that for N&gt;50 normal distribution was not necessary. Is that true?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If true is that because of the central limit theorem?&lt;/p&gt;&#xA;" OwnerUserId="4176" LastActivityDate="2011-04-24T06:40:04.900" Title="T-test for non normal when N&gt;50?" Tags="&lt;normal-distribution&gt;&lt;t-test&gt;" AnswerCount="5" />
  <row Id="9574" PostTypeId="2" ParentId="4111" CreationDate="2011-04-14T23:19:26.933" Score="1" ViewCount="" Body="&lt;p&gt;Mobile phone providers can count the number of phones in the area. Having an estimate of the mean number of phones/person good approximation can be calculated. This looks simple, so I assume it is in practice.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastActivityDate="2011-04-14T23:19:26.933" />
  <row Id="9575" PostTypeId="2" ParentId="9573" CreationDate="2011-04-15T01:07:12.330" Score="12" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Normality assumption of a t-test&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider a large population from which you could take many different samples of a particular size. (In a particular study, you generally collect just one of these samples.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The t-test assumes that the means of the different samples are normally distributed; it does not assume that the population is normally distributed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By the central limit theorem, means of samples from a population approach a normal distribution regardless of the distribution of the population. Rules of thumb say that the sample means are basically normally distributed as long as the sample size is at least 20 or 30. For a t-test to be valid on a sample of smaller size, the population distribution would have to be approximately normal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The t-test is invalid for small samples from non-normal distributions, but it is valid for large samples from non-normal distributions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Small samples from non-normal distributions&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As Michael notes below, sample size needed for the distribution of means to approximate normality depends on the degree of non-normality of the population. For approximately normal distributions, you won't need as large sample as a very non-normal distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are some simulations you can run in R to get a feel for this. First, here are a couple of population distributions.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;curve(dnorm,xlim=c(-4,4)) #Normal&#xA;curve(dchisq(x,df=1),xlim=c(0,30)) #Chi-square with 1 degree of freedom&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Next are some simulations of samples from the population distributions. In each of these lines, &quot;10&quot; is the sample size, &quot;100&quot; is the number of samples and the function after that specifies the population distribution. They produce histograms of the sample means.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;hist(colMeans(sapply(rep(10,100),rnorm)),xlab='Sample mean',main='')&#xA;hist(colMeans(sapply(rep(10,100),rchisq,df=1)),xlab='Sample mean',main='')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For a t-test to be valid, these histograms should be normal.&lt;/p&gt;&#xA;" OwnerUserId="3874" LastEditorUserId="3874" LastEditorDisplayName="" LastEditDate="2011-04-24T06:40:04.900" LastActivityDate="2011-04-24T06:40:04.900" CommentCount="4" />
  <row Id="9576" PostTypeId="2" ParentId="9501" CreationDate="2011-04-15T01:36:14.320" Score="5" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Short answer&lt;/strong&gt;: No, it is not possible, at least in terms of elementary functions. However, very good (and reasonably fast!) numerical algorithms exist to calculate such a quantity and they should be preferred over any numerical integration technique in this case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Quantity of interest in terms of normal cdf&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The quantity you are interested in is actually closely related to the conditional mean of a lognormal random variable. That is, if $X$ is distributed as a lognormal with parameters $\mu$ and $\sigma$, then, using your notation,&#xA;$$&#xD;&#xA;\newcommand{\e}{\mathbb{E}}\renewcommand{\Pr}{\mathbb{P}}\newcommand{\rd}{\mathrm{d}}&#xD;&#xA;\int_a^b f(x) \rd x = \int_a^b \frac{1}{\sigma \sqrt{2\pi}} e^{-\frac{1}{2\sigma^2}(\log(x) - \mu)^2} \rd x = \Pr(a \leq X \leq b) \e(X \mid a \leq X \leq b) \&amp;gt;.&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To get an expression for this integral, make the substitution $z = (\log(x) - (\mu + \sigma^2))/\sigma$. This may at first appear a bit unmotivated. But, note that using this substitution, $x = e^{\mu + \sigma^2} e^{\sigma z}$ and by a simply change of variables, we get&#xA;$$&#xD;&#xA;\int_a^b f(x) \rd x = e^{\mu + \frac{1}{2}\sigma^2} \int_{\alpha}^{\beta} \frac{1}{\sqrt{2\pi}} e^{-\frac{1}{2} z^2} \rd z \&amp;gt; ,&#xD;&#xA;$$&#xA;where $\alpha = (\log(a) - (\mu + \sigma^2))/\sigma$ and $\beta = (\log(b) - (\mu + \sigma^2))/\sigma$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hence,&#xA;$$&#xD;&#xA;\int_a^b f(x) \rd x = e^{\mu + \frac{1}{2}\sigma^2} \big( \Phi(\beta) - \Phi(\alpha) \big) \&amp;gt;,&#xD;&#xA;$$&#xA;where $\Phi(x) = \int_{-\infty}^x \frac{1}{\sqrt{2\pi}} e^{-z^2/2} \rd z$ is the standard normal cumulative distribution function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Numerical approximation&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is often stated that no known closed form expression for $\Phi(x)$ exists. However, a &lt;a href=&quot;http://en.wikipedia.org/wiki/Liouville%27s_theorem_%28differential_algebra%29&quot; rel=&quot;nofollow&quot;&gt;theorem of Liouville&lt;/a&gt; from the early 1800's asserts something stronger: &lt;em&gt;There is no closed form expression for this function&lt;/em&gt;. (For the proof in this particular case, see &lt;a href=&quot;http://www.claymath.org/programs/outreach/academy/LectureNotes05/Conrad.pdf&quot; rel=&quot;nofollow&quot;&gt;Brian Conrad's writeup&lt;/a&gt;.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, we are left to use a numerical algorithm to approximate the desired quantity. This can be done to within IEEE double-precision floating point via an algorithm of W. J. Cody's. It is &lt;strong&gt;the&lt;/strong&gt; standard algorithm for this problem, and utilizing rational expressions of a fairly low order, it's pretty efficient, too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a reference that discusses the approximation:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;W. J. Cody, &lt;a href=&quot;http://www.ams.org/journals/mcom/1969-23-107/S0025-5718-1969-0247736-4/S0025-5718-1969-0247736-4.pdf&quot; rel=&quot;nofollow&quot;&gt;Rational Chebyshev Approximations for the Error Function&lt;/a&gt;,&#xA;  &lt;em&gt;Math. Comp.&lt;/em&gt;, 1969, pp. 631--637.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It is also the implementation used in both MATLAB and $R$, among others, in case those make it easier to obtain example code.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://stats.stackexchange.com/questions/7200/evaluate-definite-interval-of-normal-distribution&quot;&gt;Here&lt;/a&gt; is a related question, in case you're interested.&lt;/p&gt;&#xA;" OwnerUserId="2970" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-04-15T02:50:14.760" LastActivityDate="2011-04-15T02:50:14.760" CommentCount="1" />
  <row Id="9577" PostTypeId="1" CreationDate="2011-04-15T02:39:43.933" Score="4" ViewCount="68" Body="&lt;p&gt;I have fit two generalized estimating equation (GEE) models to my data:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Model 1: Outcome is longitudinal Yes/No variable (A) (year 1,2,3,4,5) with longitudinal continuous predictor (B) for years 1,2,3,4,5. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Model 2: Outcome is the same longitudinal Yes/No variable (A), but now with my predictor fixed at its year 1 value i.e. forced to be time invariant (B).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Due to missing measurements in my longitudinal predictor at a few time points for different cases, the number of data points in model 2 is higher than in model 1. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to know about what comparisons I can validly make between the odds ratios, p-values and fit of the two models e.g.: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;If the OR for predictor B is bigger in model 1, can I validly say that the association between A and B is stronger in model1?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;How can I assess which is the better model for my data. am I correct in thinking that QIC/AIC pseudo R squareds should not be compared across models if the number of observations is not the same?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Any help would be greatly appreciated.&lt;/p&gt;&#xA;" OwnerUserId="4054" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-15T08:17:00.480" LastActivityDate="2011-04-15T08:17:00.480" Title="How can I assess GEE/logistic model fit when covariates have some missing data?" Tags="&lt;logistic-regression&gt;&lt;gee&gt;" CommentCount="1" />
  <row Id="9578" PostTypeId="2" ParentId="9573" CreationDate="2011-04-15T03:20:49.997" Score="4" ViewCount="" Body="&lt;p&gt;In my experience with just the one-sample t-test, I have found that the &lt;em&gt;skew&lt;/em&gt; of the distributions is more important than the kurtosis, say. For non-skewed but fat-tailed distributions (a t with 5 degrees of freedom, a Tukey h-distribution with $h=0.24999$, etc), I have found that 40 samples has always been sufficient to get an empirical type I rate near the nominal.  When the distribution is very skewed, however, you may need many many more samples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, suppose you were playing the lottery. With probability $p = 10^{-4}$ you will win 100 thousand dollars, and with probability $1-p$ you will lose one dollar. If you perform a t-test for the null that the mean return is &lt;strike&gt;zero&lt;/strike&gt; based on a sample of one thousand draws of this process, I don't think you are going to achieve the nominal type I rate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;edit&lt;/strong&gt;: duh, per @whuber's catch in the comment, the example I gave did not have mean zero, so testing for mean zero has nothing to do with the type I rate. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because the lottery example often has a sample standard deviation of zero, the t-test chokes. So instead, I give a code example using Goerg's &lt;a href=&quot;http://cran.r-project.org/web/packages/LambertW/&quot; rel=&quot;nofollow&quot;&gt;Lambert W x Gaussian&lt;/a&gt; distribution. The distribution I use here has a skew of around 1355.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#hey look! I'm learning R!&#xA;library(LambertW)&#xA;&#xA;Gauss_input = create_LambertW_input(&quot;normal&quot;, beta=c(0,1))&#xA;params = list(delta = c(0), gamma = c(2), alpha = 1)&#xA;LW.Gauss = create_LambertW_output(input = Gauss_input, theta = params)&#xA;#get the moments of this distribution&#xA;moms &amp;lt;- mLambertW(beta=c(0,1),distname=c(&quot;normal&quot;),delta = 0,gamma = 2, alpha = 1)&#xA;&#xA;test_ttest &amp;lt;- function(sampsize) {&#xA;    samp &amp;lt;- LW.Gauss$rY(params)(n=sampsize)&#xA;    tval &amp;lt;- t.test(samp, mu = moms$mean)&#xA;    return(tval$p.value)&#xA;}&#xA;&#xA;#to replicate randomness&#xA;set.seed(1)&#xA;&#xA;pvals &amp;lt;- replicate(1024,test_ttest(50))&#xA;#how many rejects at the 0.05 level?&#xA;print(sum(pvals &amp;lt; 0.05) / length(pvals))&#xA;&#xA;pvals &amp;lt;- replicate(1024,test_ttest(250))&#xA;#how many rejects at the 0.05 level?&#xA;print(sum(pvals &amp;lt; 0.05) / length(pvals))&#xA;&#xA;p    vals &amp;lt;- replicate(1024,test_ttest(1000))&#xA;#how many rejects at the 0.05 level?&#xA;print(sum(pvals &amp;lt; 0.05) / length(pvals))&#xA;&#xA;pvals &amp;lt;- replicate(1024,test_ttest(2000))&#xA;#how many rejects at the 0.05 level?&#xA;print(sum(pvals &amp;lt; 0.05) / length(pvals))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This code gives the empirical reject rate at the nominal 0.05 level for different sample sizes. For sample of size 50, the empirical rate is 0.40 (!); for sample size 250, 0.29; for sample size 1000, 0.21; for sample size 2000, 0.18.  Clearly the one-sample t-test suffers from skew.&lt;/p&gt;&#xA;" OwnerUserId="795" LastEditorUserId="795" LastEditorDisplayName="" LastEditDate="2011-04-15T18:30:54.513" LastActivityDate="2011-04-15T18:30:54.513" CommentCount="1" />
  <row Id="9579" PostTypeId="2" ParentId="8271" CreationDate="2011-04-15T03:40:14.243" Score="1" ViewCount="" Body="&lt;p&gt;I am guessing that you have your data in wide format.&#xA;This seems to be causing you confusion because in wide format it is less obvious that &quot;time&quot; is your second independent variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One option in SPSS for doing what it sounds like you want to do:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;code&gt;Analyze -- GLM -- Repeated Measures&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;Enter time as repeated measures factor with two levels&lt;/li&gt;&#xA;&lt;li&gt;Enter &lt;code&gt;Group&lt;/code&gt; as a between subjects factor&lt;/li&gt;&#xA;&lt;li&gt;Enter the dependent variable (presumably represented as two variables in SPSS as your two times points)&lt;/li&gt;&#xA;&lt;li&gt;Enter your &lt;code&gt;covariate&lt;/code&gt; in the covariate box&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;For more information, see &lt;a href=&quot;http://www.statisticshell.com/repeatedmeasures.pdf&quot; rel=&quot;nofollow&quot;&gt;Andy Field's SPSS Repeated Measures tutorial&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, given that repeated measures ANOVA already controls for individual differences in some respect, you may want to think about the meaning of including a time-invariant covariate in such a model. See here for a &lt;a href=&quot;http://www.psyc.bbk.ac.uk/research/DNL/stats/Repeated_Measures_ANCOVA.html&quot; rel=&quot;nofollow&quot;&gt;discussion of the issues related to covariates in repeated measures ANOVA&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-15T03:50:06.243" LastActivityDate="2011-04-15T03:50:06.243" />
  <row Id="9580" PostTypeId="2" ParentId="9573" CreationDate="2011-04-15T07:23:36.947" Score="6" ViewCount="" Body="&lt;p&gt;See my previous answer to a question on the &lt;a href=&quot;http://stats.stackexchange.com/questions/1386/robust-t-test-for-mean/1391#1391&quot;&gt;robustnest off the t-test&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In particular, I recommend playing around with the &lt;a href=&quot;http://onlinestatbook.com/stat_sim/robustness/index.html&quot; rel=&quot;nofollow&quot;&gt;onlinestatsbook applet&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The image below is based on the following scenario:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Null hypothesis is true&lt;/li&gt;&#xA;&lt;li&gt;fairly severe skewness&lt;/li&gt;&#xA;&lt;li&gt;Same distribution in both groups&lt;/li&gt;&#xA;&lt;li&gt;same variance in both groups&lt;/li&gt;&#xA;&lt;li&gt;sample size per group 5 (i.e., much less than 50 as per your question)&lt;/li&gt;&#xA;&lt;li&gt;I pressed the 10,000 simulations button about 100 times to get up to over one million simulations.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The simulation obtained suggests that instead of getting a 5% Type I errors, I was only getting 4.5% Type I errors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Whether you consider this robust depends on your perspective.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/gOXMq.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2011-04-15T07:23:36.947" CommentCount="1" />
  <row Id="9581" PostTypeId="1" CreationDate="2011-04-15T11:07:26.733" Score="2" ViewCount="226" Body="&lt;p&gt;I have a huge dataset which contains 20 columns and many rows.&#xA;I have done clustering in SAS, Knime and SPSS, but I am new to R.&#xA;I have to do clustering on my dataset. &#xA;I have imported my data into R.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What are some suggestions for getting started with cluster analysis in R?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="4182" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-15T11:45:53.927" LastActivityDate="2011-04-15T14:11:57.157" Title="Getting started with cluster analysis in R" Tags="&lt;r&gt;&lt;clustering&gt;" AnswerCount="2" CommentCount="13" FavoriteCount="1" />
  <row Id="9583" PostTypeId="2" ParentId="9581" CreationDate="2011-04-15T11:41:37.817" Score="4" ViewCount="" Body="&lt;p&gt;A lot of people coming from SAS or SPSS to R find the &lt;a href=&quot;http://www.statmethods.net/&quot; rel=&quot;nofollow&quot;&gt;Quick-R website useful&lt;/a&gt;.&#xA;There is a page on &lt;a href=&quot;http://www.statmethods.net/advstats/cluster.html&quot; rel=&quot;nofollow&quot;&gt;cluster analysis&lt;/a&gt; which you may find useful in getting you started.&lt;/p&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-15T14:11:57.157" LastActivityDate="2011-04-15T14:11:57.157" />
  <row Id="9584" PostTypeId="2" ParentId="9581" CreationDate="2011-04-15T12:10:43.937" Score="12" ViewCount="" Body="&lt;pre&gt;&lt;code&gt;## dummy data&#xA;require(MASS)&#xA;set.seed(1)&#xA;dat &amp;lt;- data.frame(mvrnorm(100, mu = c(2,6,3), &#xA;                          Sigma = matrix(c(10,   2,   4,&#xA;                                            2,   3, 0.5,&#xA;                                            4, 0.5,   2), ncol = 3)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So my data are in object &lt;code&gt;dat&lt;/code&gt;, you have read your data in and called it something. Use that object instead of &lt;code&gt;dat&lt;/code&gt; in this code below:&#xA;&lt;strong&gt;[@sridher - the codes below are the three lines I mentioned!]&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(2) ## *k*-means uses a random start&#xA;klust &amp;lt;- kmeans(scale(dat, center = TRUE, scale = TRUE), centers = 3)&#xA;klust&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The first line (&lt;code&gt;set.seed(2)&lt;/code&gt;) fixes the random number generator at a given starting point so the results are reproducible. We do this because &lt;code&gt;kmeans()&lt;/code&gt;, if not given the starting cluster centres will randomly choose &lt;code&gt;centers&lt;/code&gt; samples from your data as the cluster centres.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second line calls &lt;code&gt;kmeans()&lt;/code&gt; on the standardised data (all the variables in my data set are in different units, so scaling them to zero mean and unit variance would seem appropriate). We ask for 3 groups by specifying &lt;code&gt;centers = 3&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The third line prints the fitted &lt;em&gt;k&lt;/em&gt;-means object to the screen showing the output from the function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is &lt;strong&gt;just&lt;/strong&gt; an example though. Why three groups? I don't even do any subsequent analysis of the clustering solution. Furthermore, you probably want to run the &lt;code&gt;kmeans()&lt;/code&gt; code several times to make sure you get similar clusterings each time, but using different random starts --- set a different seed for each run.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;There is a lot more to clustering than just throwing your data at an algorithm!&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can automate that bit to some extent using the &lt;code&gt;cascadeKM()&lt;/code&gt; function in package &lt;code&gt;vegan&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(vegan)&#xA;fit &amp;lt;- cascadeKM(scale(dat, center = TRUE, scale = TRUE), 1, 4, iter = 100)&#xA;plot(fit, sortg = TRUE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which suggests 2 groups is the best solution for these data:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/yqk8o.png&quot; alt=&quot;cascadeKM output&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;but we know the data generation process had three groups, and as such &lt;em&gt;k&lt;/em&gt;-means and the summary stats of the results have not been able to correctly identify the presence of three groups in this small sample of data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With some real data this time, using the famous Iris data set&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;fit2 &amp;lt;- cascadeKM(iris[,1:4], 1, 4, iter = 100)&#xA;plot(fit2, sortg = TRUE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which clearly favours 3 groups,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/lLbdn.png&quot; alt=&quot;iris cascadeKM output&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;, which is good seeing as there really are three species in the data set&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; with(iris, unique(Species))&#xA;[1] setosa     versicolor virginica &#xA;Levels: setosa versicolor virginica&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1390" LastActivityDate="2011-04-15T12:10:43.937" CommentCount="3" />
  <row Id="9586" PostTypeId="1" CreationDate="2011-04-15T14:47:27.237" Score="5" ViewCount="57" Body="&lt;p&gt;Excuse what may be an obvious question about bootstrapping. I got sucked in the Bayesian world early and never really explored bootstrapping as much as I should have.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I ran across an analysis in which the authors were interested in a survival analysis related to some time to failure data.  They had about 100 points and used regression to fit a Weibull distribution to the data. A result of this they obtained estimates of the scale and shape parameters. A very traditional approach. However, they next used bootstrapping to sample from the original data set and, for each new sample, performed a regression and came up with a new Weibull distribution.  The results of the bootstrapping was then used to construct confidence intervals on the survival distribution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My intuition is a bit conflicted. I'm familiar with bootstrapping confidence intervals on parameters, but not seen it used for constructing distribution confidence intervals. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone point me toward a reference/source that might provide some insight? Thanks in advance.  &lt;/p&gt;&#xA;" OwnerUserId="3591" LastActivityDate="2011-04-15T14:47:27.237" Title="Bootstrap confidence intervals on parameters or on distribution?" Tags="&lt;confidence-interval&gt;&lt;bootstrap&gt;" CommentCount="4" FavoriteCount="1" />
  <row Id="9587" PostTypeId="1" CreationDate="2011-04-15T15:13:03.293" Score="1" ViewCount="110" Body="&lt;p&gt;&lt;em&gt;Adapted from my &lt;a href=&quot;http://stats.stackexchange.com/questions/9508/glmm-questions-of-appropriateness-post-hoc-tests-effects-size-and-confidence&quot;&gt;previous, unanswered question&lt;/a&gt;:&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm analyzing the results of a hormone manipulation experiment. I measured a number of variables at three times in three groups. The groups are different sizes and not all individuals were measured every time, so I'd like to use GLMM rather than a repeated-measures ANOVA. I created the model then tested the significance of the terms (time, treatment, and time x treatment) with ANOVA.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm quite new to GLMM, but after doing the tests, further reading suggests that my approach may be inappropriate, particularly with small data sets (I have ~seven animals per group). One reason given against using this method is that it seems that there is disagreement about what the degrees of freedom should be. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this method of testing the significance of factors in the model appropriate? I'm aware that the place for me to go is probably the Pinheiro and Bates book, but I don't have access to it at this time. Thanks in advance for any advice. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(nlme)&#xA;&#xA;datums&amp;lt;-data.frame(id=rep(1:20,each=3),var1=runif(60,4,6),var2=runif(60,25,30),var3=runif(60,0,1),var4=runif(60,10,15),var.time=rep(1:3,times=20),var.treatment=rep(c('a','b','c'),each=20))&#xA;&#xA;datums$var.time&amp;lt;-as.factor(datums$var.time)&#xA;datums$id&amp;lt;-as.factor(datums$id)&#xA;&#xA;&#xA;#and now the GLMMs on each variable - I'll show just one here&#xA;&#xA;var1.glmm&amp;lt;-lme(var1~var.time + var.treatment + var.time*var.treatment, data=datums, random = ~1| id)&#xA;&#xA;summary(var1.glmm)&#xA;&#xA;anova(var1.glmm)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="124" LastActivityDate="2011-05-18T09:47:33.797" Title="GLMM - test of significance" Tags="&lt;r&gt;&lt;mixed-model&gt;" AnswerCount="1" />
  <row Id="9588" PostTypeId="1" CreationDate="2011-04-15T15:32:24.263" Score="4" ViewCount="96" Body="&lt;p&gt;I'm not really a statistician, but rather in need of statistical guidance, so I hope this is not too much of an off-topic question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm writing a master's thesis (computational linguistics/NLP), and I've got several result sets I'm comparing. Now, I didn't really formulate a null and an alternative hypothesis before I ran the experiments, which I understand means that ideally I shouldn't really go about using the T-test on my datasets. But some of the different results are so close that I've given in to the temptation and T-tested them. How unclean is this? Is it bad enough that I should leave it out of my thesis entirely, or is it less severe?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In case it matters, the data are the error rates of different language models, by 10-fold cross-validation.&lt;/p&gt;&#xA;" OwnerUserId="4185" LastActivityDate="2011-04-16T00:10:23.233" Title="After the fact hypothesis testing" Tags="&lt;hypothesis-testing&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="9589" PostTypeId="2" ParentId="9588" CreationDate="2011-04-15T17:22:47.937" Score="1" ViewCount="" Body="&lt;p&gt;It's not necessarily a problem that you didn't formulate hypotheses before running the study, but you may be doing a post-hoc analysis, which would be relevant. Also consider whether what your tests mean.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The population&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I feel the need to point out that the population is sort of bizarre in this situation. If I understand your study correctly, you have 10 error rates for each model. If you were just interested in the performance for this particular partitioning scheme, you would not need to use statistical tests; these 10 error rates would be the population.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The entire corpus would interest you more. There are (N choose N/10) ways you could partition the data into 90% training set and 10% test set, and you could run the models on a random sample of these partitioning schemes. It seems that this approach is called repeated random sub-sampling validation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Differences between models&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I understand your dataset, the tests that I am about to describe may not really be valid because the 10 error rates are sort of dependent on each other; they are all taken from the same partitioning scheme.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But here we go anyway! I assume that you are trying to see whether any of the models perform significantly differently from than the others. This is a valid hypothesis, but you'll need to use something like ANOVA because you have more than two models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, if you are just trying to tell the difference between two models &lt;em&gt;because they are the two best models&lt;/em&gt;, you have to account for how you decided to compare these two after the fact. Look at post-hoc tests and p-value adjustments.&lt;/p&gt;&#xA;" OwnerUserId="3874" LastActivityDate="2011-04-15T17:22:47.937" />
  <row Id="9590" PostTypeId="1" CreationDate="2011-04-15T17:57:38.110" Score="5" ViewCount="193" Body="&lt;p&gt;I am currently using principal components analysis to select variables to use in modelling.   At the moment, I make measurements A, B and C in my experiments -- What I really want to know is: Can I make fewer measurements and stop recording C and or B to save time and effort?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I find that all 3 variables load heavily onto my first principal component which accounts for 60% of the variance in my data. The component scores tell me that if I add these variables together in a certain ratio (aA+bB+cC). I can get a score on PC1 for each case in my dataset and could use this score as a variable in modelling, but that doesn't allow me to stop measuring B and C.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I square the loadings of A and B and C on PC1, I find that variable A accounts for 65% of the variance in PC1 and variable B accounts for 50% of the the variance in PC1 and variable C also 50%, i.e. some of the variance in PC1 accounted for by each variable A, B and C is shared with another variable, but A comes out on top accounting for slightly more.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it wrong to think that I could just choose variable A or possibly (aA+bB, if necessary) to use in modelling because this variable describes a large proportion of the variance in PC1 and this in turn describes a large proportion of the variance in the data? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which approach have you gone for in the past?&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Single variable which loads heaviest on PC1 even if there are other heavy loaders?    &lt;/li&gt;&#xA;&lt;li&gt;Component score on PC1 using all variables even if they are all heavy loaders?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="4054" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-16T08:04:39.547" LastActivityDate="2011-04-16T22:06:58.427" Title="How to use principal components analysis to select variables for regression?" Tags="&lt;regression&gt;&lt;pca&gt;&lt;model-selection&gt;" AnswerCount="2" FavoriteCount="2" />
  <row Id="9591" PostTypeId="2" ParentId="9590" CreationDate="2011-04-15T19:38:16.313" Score="5" ViewCount="" Body="&lt;p&gt;You haven't specified what &quot;modeling&quot; you plan on, but it sounds like you're asking about how to select &lt;em&gt;independent&lt;/em&gt; variables among $A$, $B$, and $C$ for the purpose of (say) regressing a fourth &lt;em&gt;dependent&lt;/em&gt; variable $W$ on them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To see that this approach &lt;em&gt;can&lt;/em&gt; go wrong, consider three independent Normally distributed variables $X$, $Y$, and $Z$ with unit variance.  For the &lt;em&gt;true, underlying&lt;/em&gt; model choose a small constant $\beta \ll 1$, a really tiny constant $\epsilon \ll \beta$, and let the (dependent variable) $W = Z$ (plus a little bit of error independent of $X$, $Y$, and $Z$).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose the independent variables you have are $A = X + \epsilon Y$, $B = X - \epsilon Y$, and $C = \beta Z$.  Then $W$ and $C$ are strongly correlated (depending on the variance of the error), because each is close to a multiple of $Z$.  However, $W$ is uncorrelated with either of $A$ or $B$.  Because $\beta$ is small, the first principal component for $\{A, B, C\}$ is parallel to $X$ with eigenvalue $2 \gg \beta$.  $A$ and $B$ load heavily on this component and $C$ loads not at all because it is independent of $X$ (and $Y$).  Nevertheless, if you eliminate $C$ from the independent variables, leaving only $A$ and $B$, you will be throwing away &lt;em&gt;all&lt;/em&gt; information about the dependent variable because $W$, $A$, and $B$ are independent!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This example shows that for regression you want to pay attention to how the independent variables are correlated with the dependent one; you can't get away just by analyzing relationships among the independent variables.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-04-15T20:45:56.973" LastActivityDate="2011-04-15T20:45:56.973" CommentCount="2" />
  <row Id="9592" PostTypeId="1" CreationDate="2011-04-15T19:58:13.213" Score="5" ViewCount="113" Body="&lt;p&gt;Here's a problem. I'm a geologist, so if you think you can help me... well, go slowly please.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to predict a continuous variable like porosity from remote sensing data. Let's say I have a measure of the reflectivity of the surface of the earth, densely sampled over an area. And I have rock porosity at some locations. So I'd like to calibrate the former to the latter, and transform reflectivity to porosity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since porosity &lt;em&gt;p&lt;/em&gt; is the unknown, I instinctively plot it on the &lt;em&gt;y&lt;/em&gt;-axis, against reflectivity &lt;em&gt;r&lt;/em&gt; on the &lt;em&gt;x&lt;/em&gt;. Then do a linear regression (let's pretend life is this simple for a mo), and get my equation $p = a r + b$, and go off on my merry way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But this feels wrong. Reflectivity depends on porosity (in my hypothesis, and let's say that we know it does in nature). So in fact I should plot reflectivity on the y-axis, vs porosity on the &lt;em&gt;x&lt;/em&gt;. Then get my linear equation, then rearrange to solve for $p = (r - b)/a$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;These approaches give different answers. I found this a bit surprising. Most people I've asked about this say, 'does it matter?'. It does, and the difference can be quite big (several percent). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Which one is the correct approach? Why?&lt;/strong&gt; (For what it's worth, my answer is the latter approach, because it 'respects' nature, to anthropomorphize a bit... Not sure I can really explain myself any better). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for any insight anyone can offer,&lt;br&gt;&#xA;Matt&lt;/p&gt;&#xA;" OwnerUserId="4187" LastActivityDate="2011-04-17T17:47:38.030" Title="Which parameter on x axis for linear regression?" Tags="&lt;regression&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="9593" PostTypeId="2" ParentId="9592" CreationDate="2011-04-15T20:23:14.450" Score="1" ViewCount="" Body="&lt;p&gt;As far as I can tell, you have a lot of reflectivity measurements and a few porosity measurements, and you want to estimate porosity in areas where you only have reflectivity information.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In that case, you want to regress porosity ($y$) on reflectivity ($x$) to minimise the &lt;em&gt;sum of squares error&lt;/em&gt; in the porosity estimates.  Any physical causal relationship should not affect this.  &lt;/p&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-04-15T20:23:14.450" CommentCount="1" />
  <row Id="9595" PostTypeId="1" CreationDate="2011-04-15T22:57:39.797" Score="6" ViewCount="43" Body="&lt;p&gt;I am using a distributed lag model to analyze a time series data. The duration of study period is 18 years, and the observation is yearly data. When including a 1-year lag effect, the first year of the lag variable becomes missing. Then, a 2-year lag effect makes the first two data of lag variable missing, and so on. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am going to analyze five lag effects in my studies, but five lag variables caused 5 missing data. I assume the multiple imputation may help me overcome the loss of information in these lag variables, but the imputation result is not reasonable. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any better idea to impute the missing data in lag variables? &lt;/p&gt;&#xA;" OwnerUserId="4083" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-16T08:02:51.610" LastActivityDate="2011-05-27T16:43:18.247" Title="How to decrease the information loss from lag variables?" Tags="&lt;time-series&gt;&lt;missing-data&gt;" AnswerCount="0" CommentCount="2" FavoriteCount="1" />
  <row Id="9596" PostTypeId="1" CreationDate="2011-04-15T23:05:00.143" Score="2" ViewCount="68" Body="&lt;p&gt;The health screening for immigration to the US includes a mandatory chest x-ray screening for all adults, with the aim of detecting signs of tuberculosis.  If the x-ray indicates the possibility of TB, the applicant then submits sputum samples for culturing.  That's a lot of x-rays - and many of them don't result in positive cultures, especially among older applicants (whose lungs can be funky for many reasons).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a study, 1,500 applicants (out of 24,000 during the study period) were given a new TB diagnostic in addition to the standard screening process.*  People with chest x-rays indicative of TB were oversampled to ensure a fair number of applicants with culture-positive sputa.  Now, the researchers would like to know (for example): if we had performed this additional diagnostic on all applicants, how many people would have tested positive?  &lt;em&gt;Of those&lt;/em&gt;, how many would have had a TB-indicative x-ray?  &lt;em&gt;Of those&lt;/em&gt;, how many would have had positive cultures?  Demographic, x-ray, and culture results are known for all applicants; new-diagnostic result is only known for study participants.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Two approaches for this study have been proposed.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Build a logistic regression on the sample and use to predict for all applicants.  There are concerns that the standard errors will be too small, as age, x-ray result, and culture result are quite correlated.&lt;/li&gt;&#xA;&lt;li&gt;Pseudo-bootstrap.  Sample 24,000 results from the participants' results (with replacement), assigning a sampling probability to each participant based on the number of all applicants with matching characteristics (demographics, x-ray result, culture result).  From the resample, calculate desired quantities.  Repeat, calculate confidence intervals for the estimates.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Which of these two would you use, if either?  Why?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm happy to clarify any questions.  Thanks for your help!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;*Some study details changed to protect the word count.&lt;/p&gt;&#xA;" OwnerUserId="71" LastActivityDate="2011-04-16T01:03:15.690" Title="Using non-random sample to make population estimates" Tags="&lt;logistic-regression&gt;&lt;proportion&gt;&lt;resampling&gt;&lt;correlated-predictors&gt;&lt;non-random-sampling&gt;" AnswerCount="2" />
  <row Id="9597" PostTypeId="2" ParentId="9592" CreationDate="2011-04-15T23:23:06.797" Score="1" ViewCount="" Body="&lt;p&gt;It's not a good idea to rearrange an OLS fit equation.   If you do, you're changing the assumptions of what variables were assumed to be &quot;independent&quot; versus &quot;dependent&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have to rearrange things, one way around this is to use Principal Component Analysis rather than Ordinary Least Squares.   Here's an article that shows why this makes a difference:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.cerebralmastication.com/2010/09/principal-component-analysis-pca-vs-ordinary-least-squares-ols-a-visual-explination/&quot; rel=&quot;nofollow&quot;&gt;http://www.cerebralmastication.com/2010/09/principal-component-analysis-pca-vs-ordinary-least-squares-ols-a-visual-explination/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit 1 ======================================&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Oops, I forgot that I no longer filter these posts with the [r] tag and the code in that link is in R.   So, I've got some explaining to do.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't know if you're using Excel or some other software to fit your data, but in R there's a lot more freedom.  The dependent variable might be plotted on the horizontal (x) axis or the vertical (y) axis.  It depends on how the code is written.  In the above link, whenever you see:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; y ~ x&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;That is an R formula and it basically means that y is the &quot;dependent&quot; variable and x is the &quot;independent&quot; variable.  When you see:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; x ~ y&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;That means that x is used as the &quot;dependent&quot; variable and y is the &quot;independent&quot; variable.    When you see:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; lm(y ~ x)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;That's where he did a least squares fit with y as the &quot;dependent&quot; variable and x as the &quot;independent&quot; variable.   Obviously, the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; lm(x ~ y)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;That's where he did a least squares fit with x as the &quot;dependent&quot; variable and y as the &quot;independent&quot; variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, if you look closely at his code, you'll see that the RED line in the graph came from his least squares fit lm(y ~ x) where y is the &quot;dependent&quot; variable.  And, the BLUE line came from lm(x ~ y) where x is the &quot;dependent&quot; variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Notice in the first graph that the two lines are NOT the same.   That's because the Ordinary Least Squares (OLS) technique minimizes the error for the DEPENDENT variable.  In his second graph, he shows the error (the distance from the dependent y variable to the RED line) in orange for the lm(y ~ x) case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the third graph, he shows the error (the distance from the dependent x variable to the BLUE line) in orange for the lm(x ~ y) case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What does all of this mean?  It means that if you OLS fit your data with p as the dependent variable, you'll get one set of coefficients.   If you OLS fit your data with r as the dependent variable, you'll get a second set of coefficients.   And, unless the data behaves in a very special way (where the red line is identical to the blue line), those sets of coefficients won't be related by a simple rearrangement of terms.   If you want the ability to rearrange the equation, you have to do a fit that provides the proper coefficients.   That's where the Principal Component Analysis method comes in.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, if you fit your data using Ordinary Least Squares, and your output/calculated variable is porosity p, then you need to fit it with p as the dependent variable.   If you want to &quot;go both ways&quot;, then you'll need to use the above PCA method.&lt;/p&gt;&#xA;" OwnerUserId="2775" LastEditorUserId="2775" LastEditorDisplayName="" LastEditDate="2011-04-17T17:47:38.030" LastActivityDate="2011-04-17T17:47:38.030" CommentCount="2" />
  <row Id="9598" PostTypeId="2" ParentId="9588" CreationDate="2011-04-16T00:01:36.560" Score="1" ViewCount="" Body="&lt;p&gt;I did a Google search and bumped into the following title:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;A study of cross-validation and bootstrap for accuracy estimation and model selection&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would post the link but it's one of those citeseerx.ist.psu.edu links that comes across with an IP number for the URL.   If you do the search, you should be able to find it.    In that article, they give a calculation for a confidence interval.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's a similar article:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.public.asu.edu/~ltang9/papers/ency-cross-validation.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.public.asu.edu/~ltang9/papers/ency-cross-validation.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If those don't help, here are some links on &quot;Multiple Comparison Tests&quot;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.itl.nist.gov/div898/handbook/prc/section4/prc47.htm&quot; rel=&quot;nofollow&quot;&gt;http://www.itl.nist.gov/div898/handbook/prc/section4/prc47.htm&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.uvm.edu/~dhowell/StatPages/More_Stuff/RepMeasMultComp/RepMeasMultComp.html&quot; rel=&quot;nofollow&quot;&gt;http://www.uvm.edu/~dhowell/StatPages/More_Stuff/RepMeasMultComp/RepMeasMultComp.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.utdallas.edu/~herve/Abdi-Bonferroni2007-pretty.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.utdallas.edu/~herve/Abdi-Bonferroni2007-pretty.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.csub.edu/~psmith3/Teaching/310-8.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.csub.edu/~psmith3/Teaching/310-8.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.graphpad.com/faq/viewfaq.cfm?faq=1091&quot; rel=&quot;nofollow&quot;&gt;http://www.graphpad.com/faq/viewfaq.cfm?faq=1091&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Multiple_comparisons&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Multiple_comparisons&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm no expert, but I'm in a similar situation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a recent problem (that's still going on), until I looked at the data, I had no idea what was a reasonable null, much less what was the &quot;right&quot; null.   After some playing around with the data, I noticed some possible relationships.   Am I data snooping (dredging)?   Who knows?  I do know that I can't use an uncorrected t-test, p-value, etc on any of my results/discoveries.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I started looking for was a way to &quot;put a fence around the problem&quot;.  In other words, some way to define how many relationship combinations were available when I found &quot;my&quot; relationship.   Do you have a way to do that?    In other words, can you categorize/count the POSSIBLE solutions that you've tested/allowed-for into XX groups?  If so, that might be a way to approach a Multiple Comparison Test.&lt;/p&gt;&#xA;" OwnerUserId="2775" LastEditorUserId="2775" LastEditorDisplayName="" LastEditDate="2011-04-16T00:10:23.233" LastActivityDate="2011-04-16T00:10:23.233" CommentCount="1" />
  <row Id="9599" PostTypeId="1" CreationDate="2011-04-16T00:10:36.150" Score="3" ViewCount="79" Body="&lt;p&gt;For example, say we have three coins.  Due to several tests, we have 95% confidence intervals for X (the heads percentage) of each coin.  Coin A is 0-5%, coin B is 1-9%, and coin C is 40-70%.    &lt;/p&gt;&#xA;&#xA;&lt;p&gt;[For the curious, these CI's were created using inverse binomial CDF, based a small sample of ~30 flips of the coin.  Not surprisingly, we have no clue about the composition of the coin-bag these coins were drawn from.]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can we say anything about the number of heads we would expect to see if we flipped all three? Or if we chose one coin randomly and flipped it?  Based on the computed CI's alone?  What if we can also use the info that led us to those CI's?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit: let me tell you much more about my problem.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We have 50 coins, all of which we suspect are very low probability heads (well under 5%).  We do 30 trials, where each trial consists of flipping all the coins and recording all the results.  The value we seek is a CI on the average number of heads in a trial.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One approach would be to record the 30 &quot;numbers of heads&quot;, and do a standard normal approximation.  The problem is that many of these 30 numbers are 0 and 1, &#xA;so while we get the correct mean, I fear the resulting CI is way tighter than it has any right to be.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I would like to do is calculate a CI for each coin, using it's 30 binary outcomes.  I have calculated that for 0 heads out of 30, this leads to a 95% CI of 0-11.5% (with a mean of 0, of course).  I did this by substituting values for P in &quot;cumulative distribution function(1) of binomial distribution n=30 p=P&quot; in Wolfram Alpha until the result was around 0.025.  For 1 heads out of 30, I get a CI of 0-17.2%&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once I have done this, can I add up the CI's in any way?&lt;/p&gt;&#xA;" OwnerUserId="4191" LastEditorUserId="4191" LastEditorDisplayName="" LastEditDate="2011-04-16T05:37:24.497" LastActivityDate="2011-04-16T09:58:53.657" Title="Can confidence intervals be added?" Tags="&lt;confidence-interval&gt;&lt;binomial&gt;&lt;frequentist&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="9600" PostTypeId="2" ParentId="9596" CreationDate="2011-04-16T00:26:16.457" Score="1" ViewCount="" Body="&lt;p&gt;The $24,000$ applicant population was divided into two strata&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The $N_1$ applicants who had chest x-rays indicative of TB&lt;/li&gt;&#xA;&lt;li&gt;The $N_2=24,000-N_1$ who had chest x-rays not indicative of TB&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;A stratified random sample was taken. This included $n_1$ participants with x-rays indicative of TB and $n_2=1,500-n_1$ participants with x-rays not indicative of TB. The sampling weights for these two strata are $w_1=\frac{N_1}{n_1}$ and $w_2=\frac{N_2}{n_2}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Within the first of these strata, $x_1$ tested positive with the new method. Within the second of these strata, $x_2$ tested positive. The estimate of how many of the $24,000$ applicants would have tested positive with the new method is thus the weighted sum $T=w_1\times x_1+w_2\times x_2$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Considering the total to be taken from a normal distribution, we can estimate its standard error by adding the estimated variances within each stratum and applying a finite population correction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can additionally stratify by the demographic data. For example, if you stratified by binary sex as well, you would have four strata, each with their own weights.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Men with chest x-rays indicating the possibility of TB&lt;/li&gt;&#xA;&lt;li&gt;Women with chest x-rays indicating the possibility of TB&lt;/li&gt;&#xA;&lt;li&gt;Men with chest x-rays not indicating the possibility of TB&lt;/li&gt;&#xA;&lt;li&gt;Women with chest x-rays not indicating the possibility of TB&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Depending on the sampling process, you may call this post-stratification. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can read more about this in a book on sampling, such as in chapter 6 of Levy &amp;amp; Lemeshaw's &lt;em&gt;Sampling of Populations&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This answer seems too simple, so I suspect that I am missing something. Do tell me if I am.&lt;/p&gt;&#xA;" OwnerUserId="3874" LastEditorUserId="3874" LastEditorDisplayName="" LastEditDate="2011-04-16T00:34:17.477" LastActivityDate="2011-04-16T00:34:17.477" CommentCount="2" />
  <row Id="9601" PostTypeId="2" ParentId="9596" CreationDate="2011-04-16T00:43:15.787" Score="1" ViewCount="" Body="&lt;p&gt;You have measured 3 variables, X-ray (pos or neg), sputum (pos, neg or not tested), new diagnostic (pos, neg or not tested), and you may have 9 outcome groups:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  group X-ray sputum  new&#xA;a     1     +      +    +&#xA;a     2     +      +    -&#xA;a     3     +      +   nt&#xA;b     4     +      -    +&#xA;b     5     +      -    -&#xA;b     6     +      -   nt&#xA;c     7     -     nt    +&#xA;c     8     -     nt    -&#xA;c     9     -     nt   nt&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You told us that the total number of applicants $N=\sum_{g=1}^9{n_g}=24000$, and that $n_1+n_2+n_4+n_5+n_7+n_8=1500$. You also told that the covariate age was available for everyone and correlated with X-ray and sputum.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming that you took independently 3 random samples from (a) the &lt;code&gt;X-ray +, sputum +&lt;/code&gt; groups (1, 2, 3), (b) the &lt;code&gt;X-ray +, sputum -&lt;/code&gt; groups (4, 5, 6) and (c) the &lt;code&gt;X-ray -, sputum nt&lt;/code&gt; groups (7, 8, 9), you can fit logistic models within groups (a), (b) and (c) predicting &lt;code&gt;new&lt;/code&gt; relying on the covariate age. Using the three logistic models you can predict the probability of being &lt;code&gt;new +&lt;/code&gt; for each individual in groups 3, 6 and 9. Adding up these probabilities gives a point estimate of the number of &lt;code&gt;new +&lt;/code&gt; applicants in groups 3, 6 and 9. (This is similar to your method 1.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A method to assess the uncertainty in these estimates may be multiple imputation. Let's see its details on the &lt;code&gt;X-ray +, sputum +&lt;/code&gt; groups (1, 2, 3): Using multiple imputation on the dataset containing groups 1, 2 and 3 (with the logistic model of age as predictor) to impute the missing &lt;code&gt;new&lt;/code&gt; values throughout group 3 should give you realistic &lt;code&gt;new&lt;/code&gt; values in group 3. Taking multiple imputed datasets and counting the &lt;code&gt;new +&lt;/code&gt; cases in each you will both cover the uncertainty stemming from the binomial distribution after logistic regression and the uncertainties (and covariance) of the coefficients estimated in the logistic model. A large number of imputations will minimise the Monte Carlo error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your method 2 neglects that you want to do prediction on the finite population of your specific 24000 applicants.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-16T01:03:15.690" LastActivityDate="2011-04-16T01:03:15.690" CommentCount="2" />
  <row Id="9602" PostTypeId="2" ParentId="9599" CreationDate="2011-04-16T01:35:00.803" Score="2" ViewCount="" Body="&lt;p&gt;There are multiple methods to calculate the &lt;a href=&quot;http://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval&quot; rel=&quot;nofollow&quot;&gt;binomial confidence intervals&lt;/a&gt;, and in case of your small (~30) sample size this may mean notable differences.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Can we say anything about the number of heads we would expect to see if we flipped all three?&lt;/em&gt; Assuming that the normal approximation was used and was appropriate to get the three confidence intervals we can go on with the normal approximation: expected number of heads $E=\frac{0+5}{2} + \frac{1+9}{2} + \frac{40+70}{2}$, $CI = E &amp;#177; &amp;#189; \sqrt{(5-0)^2 + (9-1)^2 + (70-40)^2}$, see &lt;a href=&quot;http://en.wikipedia.org/wiki/Sum_of_normally_distributed_random_variables&quot; rel=&quot;nofollow&quot;&gt;sum of normally distributed random variables&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Or if we chose one coin randomly and flipped it?&lt;/em&gt; If randomly meant choosing one of these 3 coins with equal probabilities than we expect ⅓ times the expected value and a CI of $\frac{1}{\sqrt{3}}$ times the width of the result of the previous question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;What if we can also use the info that led us to those CI's?&lt;/em&gt; I that info could mean all or some of your assumptions before setting up the experiment, the experimental conditions and the observations. In that case you could formulate the full conditional probability, tell a bit more about the coins, tackle the small sample size better, particularly giving more accurate CIs for the previous questions.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-16T01:54:07.583" LastActivityDate="2011-04-16T01:54:07.583" CommentCount="2" />
  <row Id="9603" PostTypeId="2" ParentId="9573" CreationDate="2011-04-16T01:44:37.743" Score="0" ViewCount="" Body="&lt;p&gt;The central limit theorem establishes (under the required conditions) that the numerator of the t-statistic is asymptotically normal. The t-statistic also has a denominator. To have a t-distribution you'd need the denominator to be independent and square-root-of-a-chi-square-on-its-df. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;And we &lt;em&gt;know&lt;/em&gt; it won't be independent (that characterizes the normal!)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Slutsky's theorem would give you that the t-statistic is asymptotically normal (but not necessarily at a very useful rate).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What theorem would establish that the t-statistic is approximately t-distributed when there's non-normality, and how fast it comes in?&lt;/p&gt;&#xA;" OwnerUserId="805" LastActivityDate="2011-04-16T01:44:37.743" CommentCount="1" />
  <row Id="9604" PostTypeId="1" CreationDate="2011-04-16T05:18:41.593" Score="2" ViewCount="62" Body="&lt;p&gt;Is it possible have an SES where the component equations are probabilistic, say, logit or probit? I am evaluating a number of quality metrics of services provided by a number of providers. The metrics are binary (pass/fail quality certification). The obvious approach would appear to be to estimate the logit/probit equations, conditional upon provider characteristics, jointly in something like an SUR, but I can't find anything like that in the discrete choice literature.&lt;/p&gt;&#xA;" OwnerUserId="3671" LastEditorUserId="3671" LastEditorDisplayName="" LastEditDate="2011-04-19T02:33:37.717" LastActivityDate="2011-04-19T02:33:37.717" Title="Simultaneous Equation System for logit/probit?" Tags="&lt;r&gt;&lt;regression&gt;&lt;logit&gt;" CommentCount="1" />
  <row Id="9605" PostTypeId="1" CreationDate="2011-04-16T06:50:43.337" Score="2" ViewCount="40" Body="&lt;p&gt;I am interested in using the MacKinnon product of coefficients test to assess mediation. Is there a quick way to carry out these calculations and determine significance based on the obtained statistic?&lt;/p&gt;&#xA;" OwnerUserId="3958" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-16T08:00:23.233" LastActivityDate="2011-04-16T08:00:23.233" Title="Calculating the MacKinnon empirical distribution test to test mediation" Tags="&lt;mediation&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="9606" PostTypeId="2" ParentId="9605" CreationDate="2011-04-16T07:59:30.117" Score="2" ViewCount="" Body="&lt;p&gt;Tests for full and partial mediation are well explained on the &lt;a href=&quot;http://www.public.asu.edu/~davidpm/ripl/q&amp;amp;a.htm&quot; rel=&quot;nofollow&quot;&gt;Mediation FAQ&lt;/a&gt; webpage by David P. MacKinnon, and they can be implemented in any statistical package offering tools for regression modeling. Bootstraping can be used to derive standard errors and confidence intervals for the estimated coefficients.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are using R, there's even a &lt;a href=&quot;http://cran.r-project.org/web/packages/mediation/index.html&quot; rel=&quot;nofollow&quot;&gt;mediation&lt;/a&gt; package that helps you to estimate those effects, but also to conduct a sensitivity analysis on mediation effect for violations of sequential ignorability assumption. I've been previously using the &lt;a href=&quot;http://cran.r-project.org/web/packages/QuantPsyc/index.html&quot; rel=&quot;nofollow&quot;&gt;QuantPsyc&lt;/a&gt; package, from &lt;a href=&quot;http://www.umsl.edu/~fletchert/quant/&quot; rel=&quot;nofollow&quot;&gt;Thomas Fletcher&lt;/a&gt;, which implements methods proposed by MacKinnon and coll. With Stata, it can be done as described on the UCLA Stata FAQ, &lt;a href=&quot;http://www.ats.ucla.edu/stat/stata/faq/modmed.htm&quot; rel=&quot;nofollow&quot;&gt;How can I do moderated mediation in Stata?&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A good overview is also available in&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Preacher, K.J., Rucker, D.D., and&#xA;  Hayes, A.F. (2007). &lt;a href=&quot;http://bama.ua.edu/~jamie/mediation/Preacher,%20Rucker,%20&amp;amp;%20Hayes%20%28in%20press%29.pdf&quot; rel=&quot;nofollow&quot;&gt;Addressing&#xA;  Moderated Mediation Hypotheses:&#xA;  Theory, Methods, and&#xA;  Prescriptions&lt;/a&gt;. &lt;em&gt;Multivariate&#xA;  Behavioral Research&lt;/em&gt;, &lt;em&gt;42(1)&lt;/em&gt;,&#xA;  185-227.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-04-16T07:59:30.117" />
  <row Id="9607" PostTypeId="1" CreationDate="2011-04-16T09:03:50.340" Score="1" ViewCount="99" Body="&lt;p&gt;&lt;strong&gt;Background:&lt;/strong&gt;&#xA;I've modeled a project effort prediction as a Google Spreadsheet template. Details of the Model: &lt;a href=&quot;http://sites.google.com/site/effortprediction/methodology&quot; rel=&quot;nofollow&quot;&gt;http://sites.google.com/site/effortprediction/methodology&lt;/a&gt;&#xA;.Google Spreadsheet does not implement beta distribution functions.&#xA;In PERT a beta distribution is used to estimate the effort of a task in Ideal Person Days (i.e.: one worker 8h a day without distraction).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Problem:&lt;/strong&gt;&#xA;What consequences has use of normal distributions instead of beta distributions when the distribution are summed up to a single distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Details&lt;/strong&gt;&#xA;A &lt;em&gt;project&lt;/em&gt; consists of a list of &lt;em&gt;tasks&lt;/em&gt;. Each task is &lt;em&gt;beta distributed&lt;/em&gt; and &lt;em&gt;independed&lt;/em&gt;, the sum of the tasks therefore should obey to the &lt;em&gt;central limit theorem&lt;/em&gt;. The &lt;em&gt;project&lt;/em&gt; therefore is normal distributed. How does Expected Value and SD change for the project when I use normal instead of beta distributions for each task?&#xA;I assume that the tasks distribution is skewed more often to the right than to the left.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Questions&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Are the tails fater?&lt;/li&gt;&#xA;&lt;li&gt;Is the Expected Value higher or lower&#xA;with normal distributions? &lt;/li&gt;&#xA;&lt;li&gt;Is the dispersion higher or lower?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="778" LastEditorUserId="1390" LastEditorDisplayName="" LastEditDate="2011-04-16T11:24:25.440" LastActivityDate="2011-05-18T12:47:36.683" Title="Summing normal instead of beta distributions, consequences for the density function of the sum?" Tags="&lt;normal-distribution&gt;&lt;central-limit-theorem&gt;&lt;beta&gt;" AnswerCount="1" CommentCount="17" />
  <row Id="9608" PostTypeId="2" ParentId="9599" CreationDate="2011-04-16T09:58:53.657" Score="1" ViewCount="" Body="&lt;p&gt;This is a problem which can be relatively easily solved using a Bayesian approach.  This is one of the &quot;re-using&quot; the original data answers, rather than one which uses the CIs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each coin has some long run frequency of heads $\theta_i$ $(i=1,\dots,50)$.  Now you have some partial information about these frequencies, that you suspect them to be below 5%.  Not exactly clear what is meant by this: do you want to test this? or do you have information which says that it should be less than 5%?  And do you think its 5% because of the data, or did you think this before seeing the data?  Given that you have a reasonably large amount of data, this kind of information probably won't make much difference - so I will simply ignore it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now there has been no logical connection stated between each coin - so I will not pre-suppose that one exists.  This means in words that knowing the results of one coin doesn't tell you anything about any other coin.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So we start by supposing that the only thing that was known about each coin is that it is possible for each to give heads, and possible for it to give tails (i.e. we assume that if we flipped the coin until we saw both a head and a tail we would not sample forever).  This results in a uniform prior for each:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p(\theta_{i}|I)=1$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(where $I$ simply denotes the &quot;prior information&quot; or assumptions contained in the problem).  Now you will throw each coin $n_{i}=30$ times, and conditional on $\theta_{i}$ the number of heads observed $x_{i}$ will have a binomial sampling distribution&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$(x_{i}|n_{i},\theta_{i},I)\sim Binomial(n_{i},\theta_{i})$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The posterior distribution for each $\theta_{i}$ will have a beta distribution:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$(\theta_{i}|x_{i},n_{i},I)\sim Beta(x_{i}+1,n_{i}-x_{i}+1)\implies p(\theta_{i}|x_{i},n_{i},I)= \frac{\theta_{i}^{x_{i}}(1-\theta_{i})^{n_{i}-x_{i}}}{B(x_{i}+1,n_{i}-x_{i}+1)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $B(a,b)$ is the &lt;a href=&quot;http://en.wikipedia.org/wiki/Beta_function&quot; rel=&quot;nofollow&quot;&gt;beta function&lt;/a&gt;.  Note that each posterior has mode (most likely value) equal to the observed proportion $\frac{x_{i}}{n_{i}}$.  Now this is the posterior for each of the $50$ coins is always proper, well behaved, and &lt;em&gt;exact even when the observed fraction is zero&lt;/em&gt; (no approximations have been made).  Denote the number of future trials as $m_{i}$ and the unknown number of heads in these trials by $y_{i}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So if you knew which coin you picked, then you have a posterior predictive for the next result:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p(y_{i}|m_{i},x_{i},n_{i},\text{ith coin},I)=\int_{0}^{1}p(y_{i}|m_{i},\theta_{i},I)p(\theta_{i}|x_{i},n_{i},I)d\theta_{i}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$={m_{i} \choose y_{i}}\int_{0}^{1}\theta_{i}^{y_{i}}(1-\theta_{i})^{m_{i}-y_{i}}\frac{\theta_{i}^{x_{i}}(1-\theta_{i})^{n_{i}-x_{i}}}{B(x_{i}+1,n_{i}-x_{i}+1)}d\theta_{i}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$={m_{i} \choose y_{i}}\frac{B(x_{i}+y_{i}+1,m_{i}+n_{i}-x_{i}-y_{i}+1)}{B(x_{i}+1,n_{i}-x_{i}+1)}=\frac{{x_{i}+y_{i} \choose y_{i}}{m_{i}+n_{i}-x_{i}-y_{i} \choose m_{i}-y_{i}}}{{m_{i}+n_{i}+1 \choose m_{i}}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which looks like a hypergeometric like distribution, but not quite, as the &quot;random variable&quot; $y_{i}$ appears in different places to what you would see in a hypergeometric.  This is called a &lt;a href=&quot;http://en.wikipedia.org/wiki/Beta-binomial_distribution&quot; rel=&quot;nofollow&quot;&gt;beta-binomial compound distribution&lt;/a&gt;.  Note that it is &quot;parameter free&quot; - it only depends on what is unknown but of interest $y_{i}$ and what is known $m_{i},x_{i},n_{i}$ - no &quot;pluggin in&quot; of any estimate is required - hence this is an exact inference.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What if we chose one coin randomly and flipped it $m$ times?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But if you now don't know which coin is going to be picked, then the answer cannot depend on $i$.  To do this we simply average out, or marginalise out the $i$.  Assuming that you have no reason to suspect any one coin will be preferred in the choice (what is really meant by &quot;randomly&quot; I think), then each is equally likely, and we get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p(y|m,D,I)=\frac{1}{50}\sum_{i=1}^{50}\frac{{x_{i}+y \choose y}{m+n_{i}-x_{i}-y \choose m-y}}{{m+n_{i}+1 \choose m}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $D\equiv x_{1},\dots,x_{50},n_{1},\dots,n_{50}$ (&quot;the data&quot;) and the $i$ subscript has been dropped from $m,y$ to indicate that the result doesn't depend on $i$ (i.e. doesn't depend on which coin is flipped).  Now this is an &lt;em&gt;exact&lt;/em&gt; answer to your problem.  You can calculate a 95% interval for $y$ using this probability distribution, setting $m=30$ (since this is the sample size you used).  This should give you what you are after - in that the frequentist CI is usually phrased in a predictive fashion &quot;if we sampled again what would we likely the true value be covered by the CI 95% of the time&quot; - a CI talks about the future.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-04-16T09:58:53.657" />
  <row Id="9609" PostTypeId="2" ParentId="1040" CreationDate="2011-04-16T10:55:20.060" Score="1" ViewCount="" Body="&lt;p&gt;model misspecification is irrelevant when you are interpreting co-efficients in a model, but any such interpretation is always conditional on the model.  This is because a model is about an association, not about a causal relationship - the OLS coefficients just tells you about the linear association between $X$ and $Y$.  It is impossible for this to be &quot;incorrect&quot; per se, and as long as you interpret your results as estimating linear associations you will not be wrong (but possibly irrelevant, answering the wrong questions).  There just may be more useful non-linear associations between $X$ and $Y$ that are of interest (i.e. more relevant questions)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And further, if you know your model is misspecified, then surely you must also know something about how it is misspecified, i.e. you must have some information about the model function $f(.)$.  So why not use that information and make a better model?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;model misspecification is one of those problems which is real, but mostly irrelevant.  For if the model is misspecified what can you do about it besides choose a different model which is correctly specified?  And if you only have one model, it is either that model or no inference at all (no inference is hardly a good outcome given how much effort and resources it probably took to gather the data).&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-04-16T10:55:20.060" CommentCount="3" />
  <row Id="9610" PostTypeId="1" AcceptedAnswerId="9613" CreationDate="2011-04-16T14:28:54.517" Score="2" ViewCount="89" Body="&lt;p&gt;Is there such a formula? Given a set of data for which the mean, variance, skewness and kurtosis is known, or can be measured, is there a single formula which can be used to calculate the probability density of a value assumed to come from the aforementioned data?&lt;/p&gt;&#xA;" OwnerUserId="226" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-16T15:02:04.343" LastActivityDate="2011-04-16T19:39:11.997" Title="Closed form formula for normal distribution including skewness and kurtosis?" Tags="&lt;normal-distribution&gt;&lt;kurtosis&gt;&lt;density-function&gt;&lt;skew&gt;" AnswerCount="3" CommentCount="3" />
  <row Id="9611" PostTypeId="2" ParentId="9610" CreationDate="2011-04-16T14:46:25.153" Score="0" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/D%27Agostino%27s_K-squared_test&quot; rel=&quot;nofollow&quot;&gt;D’Agostino’s K2 test&lt;/a&gt; will tell you whether a sample distribution came from a normal distribution based on the sample's skewness and kurtosis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to do a test assuming a non-normal distribution (perhaps with high skewness or kurtosis), you'll need to figure out what the distribution is. You can look at the &lt;a href=&quot;http://en.wikipedia.org/wiki/Skew_normal_distribution&quot; rel=&quot;nofollow&quot;&gt;skew normal distribution&lt;/a&gt; and the &lt;a href=&quot;http://en.wikipedia.org/wiki/Generalized_normal_distribution&quot; rel=&quot;nofollow&quot;&gt;generalized normal distribution&lt;/a&gt;. If you do this, you consider other distributions too.&lt;/p&gt;&#xA;" OwnerUserId="3874" LastActivityDate="2011-04-16T14:46:25.153" />
  <row Id="9612" PostTypeId="2" ParentId="9278" CreationDate="2011-04-16T15:12:30.973" Score="0" ViewCount="" Body="&lt;p&gt;You might be looking for a goodness-of-fit test to check one of the following&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;whether the sample distribution are likely to have come from the population distribution or&lt;/li&gt;&#xA;&lt;li&gt;whether multiple sample distributions are likely to have come the same unknown population distribution&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Look at chi-square and Kolmogorov-Smirnov tests.&lt;/p&gt;&#xA;" OwnerUserId="3874" LastActivityDate="2011-04-16T15:12:30.973" />
  <row Id="9613" PostTypeId="2" ParentId="9610" CreationDate="2011-04-16T17:10:33.647" Score="3" ViewCount="" Body="&lt;p&gt;There are many such formulas.  The first successful attempt at solving precisely this problem was made by Karl Pearson in 1895, eventually leading to the system of &lt;a href=&quot;http://en.wikipedia.org/wiki/Pearson_distribution&quot; rel=&quot;nofollow&quot;&gt;Pearson distributions&lt;/a&gt;.  This family can be parameterized by the mean, variance, skewness, and kurtosis.  It includes, as familiar special cases, Normal, Student-t, Chi-square, Inverse Gamma, and F distributions.  &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0340614307&quot; rel=&quot;nofollow&quot;&gt;Kendall &amp;amp; Stuart Vol 1&lt;/a&gt; give details and examples.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-04-16T17:10:33.647" />
  <row Id="9614" PostTypeId="1" AcceptedAnswerId="9624" CreationDate="2011-04-16T17:27:46.990" Score="7" ViewCount="86" Body="&lt;p&gt;I am modeling Diabetes Prediction using Logistic Regression. The dataset used is the  &lt;a href=&quot;http://www.cdc.gov/BRFSS/&quot; rel=&quot;nofollow&quot;&gt;Behavioral Risk Factor Surveillance System (BRFSS)&lt;/a&gt; of the Center for Disease Control (CDC). One of the independent variables is High Blood Pressure. It is categorical with the following levels 'Yes', 'No', 'Don't know/Refused'. Should I remove those rows with 'Don't know/Refused' while building the model? What difference does it make to keep or remove those rows from the model?&lt;/p&gt;&#xA;" OwnerUserId="3897" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-04-16T20:01:35.560" LastActivityDate="2011-04-16T20:11:08.790" Title="Treating 'Don't know/Refused' levels of categorical variables" Tags="&lt;logistic-regression&gt;&lt;missing-data&gt;" AnswerCount="3" CommentCount="0" FavoriteCount="1" />
  <row Id="9615" PostTypeId="1" CreationDate="2011-04-16T17:29:20.930" Score="2" ViewCount="45" Body="&lt;p&gt;I wonder if two identical time series are cointegrated. Can anyone shed some light on this? Thanks&lt;/p&gt;&#xA;" OwnerUserId="4192" LastActivityDate="2011-04-16T17:38:37.700" Title="Are two identical time series cointegrated?" Tags="&lt;time-series&gt;&lt;cointegration&gt;" AnswerCount="1" />
  <row Id="9616" PostTypeId="2" ParentId="9615" CreationDate="2011-04-16T17:38:37.700" Score="2" ViewCount="" Body="&lt;p&gt;If I understand your question correctly, you are asking if you take two identical time series (i.e. a time series and a direct copy of it), are these two cointegrated? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If that is your question, then it really reduces to a simpler form: is a time series stationary? As I understand it, cointegration is the phenomenon whereby non-stationary processes have linear combinations that are stationary. If you have two identical time series, then the set of all linear combinations of those time series is equivalent to the set of all scalar multiples of one of them. In this case, if a time series has a scalar multiple of it that is stationary, then the original time series was itself stationary. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I may be way off base, but this &lt;strong&gt;seems&lt;/strong&gt; like the question you're asking, and it doesn't seem to be much more complicated than this. If you intended something more complicated (i.e. a time series and a lagged version of itself), then there may be some more depth here.&lt;/p&gt;&#xA;" OwnerUserId="781" LastActivityDate="2011-04-16T17:38:37.700" CommentCount="5" />
  <row Id="9617" PostTypeId="1" AcceptedAnswerId="9635" CreationDate="2011-04-16T18:00:25.273" Score="6" ViewCount="135" Body="&lt;p&gt;I'm hoping that someone can explain, in layman's terms, what a characteristic function is and how it is used in practice. I've read that it is the Fourier transform of the pdf, so I guess I know &lt;em&gt;what&lt;/em&gt; it is, but I still don't understand its purpose. If someone could provide an intuitive description of its purpose and perhaps an example of how it is typically used, that would be fantastic!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just one last note: I have see the Wikipedia page, but am apparently too dense to understand what is going on. What I'm looking for is an explanation that someone not immersed in the wonders of probability theory, say a computer scientist, could understand.&lt;/p&gt;&#xA;" OwnerUserId="1913" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-16T18:12:59.480" LastActivityDate="2011-04-18T07:47:28.053" Title="What is a purpose of characteristic functions?" Tags="&lt;probability-theory&gt;" AnswerCount="4" FavoriteCount="3" />
  <row Id="9618" PostTypeId="2" ParentId="9614" CreationDate="2011-04-16T18:57:16.713" Score="0" ViewCount="" Body="&lt;p&gt;Do you have any reason to think that study subjects with diabetes were more likely or less likely to end up with the DK/R response? If not (and I'd be pretty surprised to find out you did), including this predictor in the model w/o excluding these cases will result in noise. That is, you'll end up with less precision in your assessment of how &quot;yes&quot; vs. &quot;no&quot;  influences the estimated probability of diabetes (because you'll be trying to model the influence of either &quot;yes&quot; or &quot;no&quot; vs. random DK/R responses as opposed to just &quot;yes&quot; vs. &quot;no&quot;). The most straightforward option is to exclude the cases with DK/R responses. Assuming that their &quot;yes/no&quot; responses were indeed missing at random, excluding them will not bias your estimate of the influence of &quot;yes&quot; vs. &quot;no.&quot; That approach, however, will reduce your sample size and thus reduce statistical power with regard to the remaining predictors. If you have a lot of DK/R on this variable, you might want to impute &quot;yes&quot;/&quot;no&quot; responses by multiple imputation (arguably the most, maybe only, defensible missing-value imputation strategy).&lt;/p&gt;&#xA;" OwnerUserId="2347" LastActivityDate="2011-04-16T18:57:16.713" />
  <row Id="9619" PostTypeId="2" ParentId="9614" CreationDate="2011-04-16T18:58:14.193" Score="2" ViewCount="" Body="&lt;p&gt;First you have to think over if the missing data are missing completely at random (MCAR), missing at random (MAR) or missing not at random (MNAR) as deletion (in other words complete-case analysis) may lead to biased results. Alternatives are inverse probability weighting, multiple imputation, the full-likelihood method and doubly-robust methods. Multiple imputation with chained equations (MICE) if often the easiest way to go.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastActivityDate="2011-04-16T18:58:14.193" CommentCount="2" />
  <row Id="9620" PostTypeId="2" ParentId="9617" CreationDate="2011-04-16T19:24:55.683" Score="3" ViewCount="" Body="&lt;p&gt;The purpose of characteristic functions is that they can be used to derive  the properties of distributions in probability theory. If you're not interested in such derivations you do not need to learn about characteristic functions.&lt;/p&gt;&#xA;" OwnerUserId="449" LastActivityDate="2011-04-16T19:24:55.683" CommentCount="3" />
  <row Id="9621" PostTypeId="2" ParentId="9617" CreationDate="2011-04-16T19:36:20.890" Score="2" ViewCount="" Body="&lt;p&gt;The characteristic function is the Fourier transform of the density function of the distribution. If you have any intuition regarding Fourier transforms, this fact may be enlightening.  The common story about Fourier transforms is that they describe the function 'in frequency space.' Since a probability density is usually unimodal (at least in the real world, or in the models made about the real world), this doesn't seem terribly interesting.&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2011-04-16T19:36:20.890" />
  <row Id="9622" PostTypeId="2" ParentId="9610" CreationDate="2011-04-16T19:39:11.997" Score="1" ViewCount="" Body="&lt;p&gt;This sounds like a &lt;a href=&quot;http://www.johndcook.com/blog/2010/09/20/skewness-andkurtosis/&quot; rel=&quot;nofollow&quot;&gt;'moment-matching' approach&lt;/a&gt; to fitting a distribution to data. It is generally regarded as not a great idea (the title of John Cook's blog post is 'a statistical dead end').&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2011-04-16T19:39:11.997" />
  <row Id="9623" PostTypeId="1" CreationDate="2011-04-16T19:47:10.053" Score="2" ViewCount="60" Body="&lt;p&gt;I've read the sampling error examples in Wikipedia, but I still don't understand. Could you give me a simple and characteristic example for sampling error?&lt;/p&gt;&#xA;" OwnerUserId="4176" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-20T09:04:49.110" LastActivityDate="2011-05-20T09:04:49.110" Title="Sampling error example" Tags="&lt;sampling&gt;" AnswerCount="2" />
  <row Id="9624" PostTypeId="2" ParentId="9614" CreationDate="2011-04-16T20:11:08.790" Score="5" ViewCount="" Body="&lt;p&gt;I was just wondering about exactly the same question when analyzing the latest &lt;a href=&quot;http://www.cdc.gov/nchs/injury/injury_hospital.htm&quot; rel=&quot;nofollow&quot;&gt;National Hospital Discharge Survey&lt;/a&gt; data.  Several variables have substantial missing values, such as marital status and type of procedure.  This issue came to my attention because these categories showed up with strong (and significant) effects in most logistic regression analyses I was running.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One is inclined to wonder &lt;em&gt;why&lt;/em&gt; a missing code is given.  In the case of marital status, for instance, it is plausible that failure to provide this information could be linked to important factors such as socioeconomic status or type of disease.  In your case of high blood pressure, we should ask why would the value not be known or refused?  This could be related to practices at the institution (perhaps reflecting lax procedures) or even to the individuals (such as religious beliefs).  Those characteristics in turn could be associated with diabetes. Therefore, it seems prudent to continue as you have, rather than to code these values as missing (thereby excluding them from analysis altogether) or attempting to impute the values (which effectively masks the information they provide and could bias the results).  It's really not any more difficult to do: you merely have to make sure this variable is treated as categorical and you'll get one more coefficient in the regression output. Furthermore, I suspect the BRFSS datasets are large enough that you don't have to worry about power.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-04-16T20:11:08.790" CommentCount="1" />
  <row Id="9625" PostTypeId="1" AcceptedAnswerId="9630" CreationDate="2011-04-16T20:11:45.357" Score="6" ViewCount="131" Body="&lt;p&gt;Suppose you wish to test the hypothesis that the mean of a distribution equals the median, given some samples drawn from the distribution. How would this be done? I am guessing that the test statistic would be (the absolute value of) the sample mean minus the sample median, but am not sure about the standard error of that statistic (the sample mean and median are not independent, I believe). Is this a well known test? &lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2011-04-18T03:43:18.880" Title="How to test if the mean equals the median?" Tags="&lt;hypothesis-testing&gt;&lt;mean&gt;&lt;median&gt;" AnswerCount="2" CommentCount="3" />
  <row Id="9626" PostTypeId="1" CreationDate="2011-04-16T20:16:25.510" Score="5" ViewCount="93" Body="&lt;p&gt;I believe that for ANOVA unbalanced means unequal groups. But for randomised controlled trials unbalanced seems to mean something like groups of different characteristics despite randomisation. So in case of RCTs does unbalanced simply mean unlucky randomisation? Nothing to do with group sizes? What's the definition of an unbalanced RCT?&lt;/p&gt;&#xA;" OwnerUserId="4176" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-18T08:36:35.570" LastActivityDate="2011-05-18T22:47:48.143" Title="When is a randomised controlled trial (RCT) balanced?" Tags="&lt;sample-size&gt;&lt;clinical-trials&gt;" AnswerCount="1" />
  <row Id="9627" PostTypeId="1" AcceptedAnswerId="9720" CreationDate="2011-04-16T20:36:49.650" Score="4" ViewCount="153" Body="&lt;p&gt;First of all, I realize multiple regression does not really give actually &quot;causal&quot; inferences about the data. Let me explain my current case:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have four independent variables which I hope (but am not sure) are involved in driving the thing I'm measuring. I wanted to use multiple regression to see how much each of these variables are contributing to my dependent variable, and did so. Supposedly, variable &quot;Number four&quot; is influencing my outcome measure very strongly (beta weight close to 0.7). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I've been told this isn't enough, because some of my &quot;independent&quot; variables may in fact be correlated with each-other. In that case, I could think &quot;Variable four&quot; is driving my dependent variable, when really both three and four could be contributing equally. This seems correct, but since I'm new to this I'm unsure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I systemically avoid this problem in the future? What specific procedures would you recommend when using multiple regression to make sure that your &quot;independent&quot; data does not already contain hidden correlations?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit:&#xA;The data itself is a series of network (graph) models of a particular neurological state. I'm measuring the &quot;clustering coefficient&quot; which describes the topology of each network as a whole (dependent variable here), and then seeing if the individual connectivities of four nodes in the larger 100+ network are driving the global clustering values (four independent variables). However, these nodes are part of a network, so sort of by definition it's possible they're correlated to some extent.&lt;/p&gt;&#xA;" OwnerUserId="4139" LastEditorUserId="4139" LastEditorDisplayName="" LastEditDate="2011-04-20T04:35:36.783" LastActivityDate="2011-04-20T04:35:36.783" Title="What should I be aware of when using multiple regression to find &quot;causal&quot; relationships in my data?" Tags="&lt;multivariate-analysis&gt;&lt;multiple-regression&gt;" AnswerCount="4" CommentCount="5" FavoriteCount="3" />
  <row Id="9628" PostTypeId="2" ParentId="9623" CreationDate="2011-04-16T20:46:32.990" Score="1" ViewCount="" Body="&lt;p&gt;When you take a sample you do so because you want to know what is happening in the wider population but unless you take a census and ask everyone there will always be a difference between the sample and the population. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a good definition below taken from &lt;a href=&quot;http://www.socialresearchmethods.net/kb/sampstat.php&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A standard deviation is the spread of the scores around the average in a single sample. The standard error is the spread of the averages around the average of averages in a sampling distribution&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A classic example is an opinion poll with a 95% confidence interval of + or - 3%&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the headline figure is 45% Democrats and 42% Republicans the reality amongst the real population is anything from 42-48% Democrats and 39-45% Republicans.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another polling company could do the exact same survey and the results could be 44% and 43%. So which is right? They both are because a sample is only an approximation of the population and if you sample a population many times the results you get are going to vary a little bit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The larger the sample the smaller the error. Hope that helps.&lt;/p&gt;&#xA;" OwnerUserId="3597" LastEditorUserId="3597" LastEditorDisplayName="" LastEditDate="2011-04-16T22:24:59.253" LastActivityDate="2011-04-16T22:24:59.253" />
  <row Id="9629" PostTypeId="1" AcceptedAnswerId="9641" CreationDate="2011-04-16T21:08:05.087" Score="2" ViewCount="359" Body="&lt;p&gt;I've read that the chi square test is useful to see if a sample is significantly different from a set of expected values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, here is a table of results of a survey regarding people's favourite colours (n=15+13+10+17=55 total respondents):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;red,blue,green,yellow&#xA;&#xA;15,13,10,17&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;A chi square test can tell me if this sample is significantly different from the null hypothesis of equal probability of people liking each colour.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Question: Can the test be run on the proportions of total respondents who like a certain colour? Like below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;red,blue,green,yellow&#xA;&#xA;0.273,0.236,0.182,0.309&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Where, of course, 0.273+0.236+0.182+0.309=1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the chi square test is not suitable in this case, what test would be? Thanks!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: I tried @Roman Luštrik answer below, and got the following output, why am I not getting a p-value and why does R say &quot;Chi-squared approximation may be incorrect&quot;?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; chisq.test(c(0,0,0,8,6,2,0,0),p = c(0.406197174,0.088746395,0.025193306,0.42041479,0.03192905,0.018328576,0.009190708,0))&#xA;&#xA;    Chi-squared test for given probabilities&#xA;&#xA;data:  c(0, 0, 0, 8, 6, 2, 0, 0) &#xA;X-squared = NaN, df = 7, p-value = NA&#xA;&#xA;Warning message:&#xA;In chisq.test(c(0, 0, 0, 8, 6, 2, 0, 0), p = c(0.406197174, 0.088746395,  :&#xA;  Chi-squared approximation may be incorrect&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2830" LastEditorUserId="2830" LastEditorDisplayName="" LastEditDate="2011-04-18T22:37:37.127" LastActivityDate="2011-04-20T13:12:12.077" Title="Can chi square be used to compare proportions?" Tags="&lt;chi-squared&gt;&lt;test&gt;&lt;proportion&gt;" AnswerCount="5" CommentCount="4" FavoriteCount="2" />
  <row Id="9630" PostTypeId="2" ParentId="9625" CreationDate="2011-04-16T21:10:55.900" Score="2" ViewCount="" Body="&lt;p&gt;This is a bootstrap confidence interval for the (median - mean) difference in R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;z = function() {s = sample(women$weight, replace=TRUE); median(s)-mean(s)}&#xA;k = replicate(10000, z())&#xA;c(quantile(k, c(.025, .5, .975)), mean=mean(k), sd=sd(k), qgte0=mean(k&amp;gt;=0))&#xA;&#xA;     2.5%       50%     97.5%      mean        sd     qgte0 &#xA;-7.933333 -1.333333  5.800000 -1.218007  3.513462  0.362100 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm still pondering if the mean and SD of the &lt;code&gt;k&lt;/code&gt; resample of the difference could be used in a Wald(-like) test, or if the quantile greater than or equal to 0 can be viewed as a one-sided p value under some assumptions — comments on this are welcome.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastActivityDate="2011-04-16T21:10:55.900" CommentCount="3" />
  <row Id="9631" PostTypeId="2" ParentId="9629" CreationDate="2011-04-16T21:58:42.240" Score="3" ViewCount="" Body="&lt;p&gt;Yes, you can test the null hypothesis:&#xA;&quot;H0: prop(red)=prop(blue)=prop(green)=prop(yellow)=1/4&quot;&#xA;using a chi square test that compares the proportions of the survey (0.273, ...) to the expected proportions (1/4, 1/4, 1/4, 1/4)&lt;/p&gt;&#xA;" OwnerUserId="4196" LastActivityDate="2011-04-16T21:58:42.240" CommentCount="3" />
  <row Id="9632" PostTypeId="2" ParentId="9627" CreationDate="2011-04-16T22:01:47.947" Score="3" ViewCount="" Body="&lt;p&gt;If you want to see if the independent variables are correlated, that's easy - just test the correlations e.g. with PROC CORR in SAS, or cor in R, or whatever in whatever package you use.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might, though, want to test collinearity instead, or in addition. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But that's only part of the problem for causation.  More problematic is that some variable that is NOT in your data is involved.  Classic examples:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Students who hire tutors get worse grades than students who do not hire tutors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The amount of damage done by a fire is highly related to the number of firemen who show up. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;and (my favorite)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;if you regress IQ on astrological sign and age among children age 5 - 12, there is a significant interaction and a significant effect of sign on IQ, but only in young children.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Reasons:&#xA;1.  Yes.  Because students who  get really good grades tend not to hire tutors in the first place&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Yes, because bigger fires do more damage and bring more firemen&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The amount of school (in months) a kid has had depends on birth month.  School systems have age cutoffs.  So, one 6 year old may have had 11 months more school than another 6 year old.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;And all that is without getting into philosophy!&lt;/p&gt;&#xA;" OwnerUserId="686" LastActivityDate="2011-04-16T22:01:47.947" CommentCount="1" />
  <row Id="9633" PostTypeId="2" ParentId="9590" CreationDate="2011-04-16T22:06:58.427" Score="0" ViewCount="" Body="&lt;p&gt;If you have only 3 IVs, why do you want to reduce them?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That is, is your sample very small (so that 3 IVs risks overfitting)?  In this case, consider partial least squares&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or are the measurements very expensive (so, in future, you'd like to measure only one IV)?  In this case, I'd consider looking at the different regressions with each IV separately and together.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or did someone in your past over-emphasize the value of parsimony?  In this case, why not include all 3 IVs?&lt;/p&gt;&#xA;" OwnerUserId="686" LastActivityDate="2011-04-16T22:06:58.427" />
  <row Id="9634" PostTypeId="2" ParentId="9623" CreationDate="2011-04-16T22:15:11.883" Score="3" ViewCount="" Body="&lt;p&gt;Imagine that you want to know the average height of men on earth. This average height exists but obviously you will never be able to know it (unless you're able to measure several millions men...).&#xA;What you can do is measure hundreds or thousands of people and calculate the average height of these people. The average height among these people is probably not exactly equal to the average height of men on earth (because they are particular men in the whole population) but, if you did a good job (use a representative sample of the population), it should be close enough.&#xA;The difference between the quantity that you want to know (average height of men on earth) and its estimation through your sample (average height of men in the sample) is the sampling error.&lt;/p&gt;&#xA;" OwnerUserId="3377" LastActivityDate="2011-04-16T22:15:11.883" />
  <row Id="9635" PostTypeId="2" ParentId="9617" CreationDate="2011-04-17T00:40:55.120" Score="9" ViewCount="" Body="&lt;p&gt;Back in the day, people used logarithm tables to multiply numbers faster.  Why is this?  Logarithms convert multiplication to addition, since $log(ab) = log(a) + log(b)$.  So in order to multiply two large numbers $a$ and $b$, you found their logarithms, added the logarithms, $z = log(a) + log(b)$, and then looked up $exp(z)$ on another table.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, characteristic functions do a similar thing for probability distributions.  Suppose $X$ has a distribution $f$ and $Y$ has a distribution $g$, and $X$ and $Y$ are independent.  Then the distribution of $X+Y$ is the &lt;em&gt;convolution&lt;/em&gt; of $f$ and $g$, $f * g$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now the characteristic function is an analogy of the &quot;logarithm table trick&quot; for convolution, since if $\phi_f$ is the characteristic function of f, then the following relation holds:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;\phi_f \phi_g = \phi_{f * g}&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furthermore, also like in the case of logarithms,it is easy to find the inverse of the characteristic function: given $\phi_h$ where $h$ is an unknown density, we can obtain $h$ by the inverse Fourier transform of $\phi_h$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The characteristic function converts &lt;em&gt;convolution&lt;/em&gt; to  &lt;em&gt;multiplication&lt;/em&gt; for density functions the same way that logarithms convert &lt;em&gt;multiplication&lt;/em&gt; into &lt;em&gt;addition&lt;/em&gt; for numbers. Both transformations convert a relatively complicated operation into a relatively simple one.&lt;/p&gt;&#xA;" OwnerUserId="3567" LastActivityDate="2011-04-17T00:40:55.120" CommentCount="3" />
  <row Id="9636" PostTypeId="2" ParentId="9629" CreationDate="2011-04-17T00:57:53.990" Score="2" ViewCount="" Body="&lt;p&gt;The chi-square test is good as long as the expected counts are large, usually above 10 is fine.  below this the $\frac{1}{E(x_{i})}$ part tends to dominate the test.  An exact test statistic is given by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\psi=\sum_{i}x_{i}\log\left(\frac{x_{i}}{np_{i}}\right)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $x_{i}$ is the observed count in category $i$.  $i\in \{\text{red, blue, green, yellow}\}$ in your example. $n$ is your sample size, equal to $55$ in your example. $p_i$ is the hypothesis you wish to test - the most obvious is $p_i=p_j$ (all probabilities are equal).  You can show that the chi-square statistic:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\chi^{2}=\sum_{i}\frac{(x_{i}-np_{i})^{2}}{np_{i}}\approx 2\psi$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In terms of the observed frequencies $f_{i}=\frac{x_{i}}{n}$ we get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\psi=n\sum_{i}f_{i}\log\left(\frac{f_{i}}{p_{i}}\right)$$&#xA;$$\chi^{2}=n\sum_{i}\frac{(f_{i}-p_{i})^{2}}{p_{i}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Note that $\psi$ is the effectively the KL divergence between the hypothesis and the observed values).  You may be able to see intuitively why $\psi$ is better for small $p_{i}$, because it does have a $\frac{1}{p_{i}}$ but it also has a log function which is absent from the chi-square, this &quot;reigns in&quot; the extreme values caused by small expected counts.  Now the &quot;exactness&quot; of this $\psi$ statistic is not as an exact chi-square distribution - it is exact in a probability sense.  The exactness comes about in the following manner, from Jaynes 2003 probability theory: the logic of science.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have two hypothesis $H_{1}$ and $H_{2}$ (i.e. two sets of $p_i$ values) that you wish to test, each with test statistics $\psi_{1}$ and $\psi_{2}$ respectively, then $\exp\left(\psi_{1}-\psi_{2}\right)$ gives you the likelihood ratio for $H_{2}$ over $H_{1}$.  $\exp\left(\frac{1}{2}\chi_{1}^{2}-\frac{1}{2}\chi_{2}^{2}\right)$ gives an approximation to this likelihood ratio.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now if you choose $H_{2}$ to be the &quot;sure thing&quot; or &quot;perfect fit&quot; hypothesis, then we will have $\psi_{2}=\chi^{2}_{2}=0$, and thus the chi-square and psi statistic both tell you &quot;how far&quot; from the perfect fit any single hypothesis is, from one which fit the observed data exactly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Final recommendation:  Use $\chi_{2}^{2}$ statistic when the expected counts are large, mainly because most statistical packages will easily report this value.  If some expected counts are small, say about $np_{i}&amp;lt;10$, then use $\psi$, because the chi-square is a bad approximation in this case, these small cells will dominate the chi-square statistic.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastEditorUserId="2392" LastEditorDisplayName="" LastEditDate="2011-04-17T02:04:47.963" LastActivityDate="2011-04-17T02:04:47.963" CommentCount="31" />
  <row Id="9637" PostTypeId="1" CreationDate="2011-04-17T03:23:52.840" Score="2" ViewCount="85" Body="&lt;p&gt;standard deviation (std) can be used when arithmetic mean is the right statistic. What is the equivalent of std when geometric mean is the right statistic?&lt;/p&gt;&#xA;" OwnerUserId="4197" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-04-17T04:42:11.300" LastActivityDate="2011-04-17T05:38:56.727" Title="Geometric mean and standard deviation" Tags="&lt;hypothesis-testing&gt;&lt;confidence-interval&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="9638" PostTypeId="2" ParentId="9627" CreationDate="2011-04-17T03:46:02.657" Score="1" ViewCount="" Body="&lt;p&gt;The relationship between causation and association is basically in answering the following question:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;What else, besides the hypothesised causal relationship, could have caused $X$ and $Y$ to be related to each other?&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As long as the answer to this question is not &quot;nothing&quot; then you can only talk definitively about association.  There can always be that one proposed &quot;causal&quot; relationship is actually a special case of the &quot;correct&quot; causal relationship - this is what happened between Newton's and Einstein's theory of gravity I think.  Newton's causal relationship was a special case of Einstein's theory.  And his theory will probably be a special case of some other theory.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Additionally, any error at all in your data removes any chance of a definite causal relationship.  This is because the phrase &quot;A causes B&quot; is somewhat of a deductive link between A and B.  All you have to do to disprove this hypothesis is to find 1 case where B is not present but A is present (for then A is true, but this should mean that B is also true - but we observed B false).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a regression setting, it is much more constructive to think of prediction than of interpreting coefficients when looking at causation.  So if you really do have a good reason to think that variable four is the &quot;main cause&quot; of variable $Y$ (your dependent variable), then you should be able to predict $Y$ with near certainty using variable four.  If you cannot do this, then it is inappropriate to concluded that variable four causes $Y$.  But if you can do this prediction to near certainty using all four variables - then this is indicating that particular combinations are &quot;causing&quot; $Y$.  And whenever you propose a causal relationship you will almost certainly have to &quot;prove it&quot; by reproducing your results with new data - you will need to be able to predict what data will be seen, and be correct about it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You also need some kind of physical theory about the &quot;causal mechanism&quot; (when I push that button, the light comes on, when I push this button, the light changes color, etc.).  If all you have is that the &quot;regression coefficient was 0.7&quot; this does little for establishing a causal mechanism which is at work.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-04-17T03:46:02.657" />
  <row Id="9639" PostTypeId="2" ParentId="9637" CreationDate="2011-04-17T05:38:56.727" Score="2" ViewCount="" Body="&lt;p&gt;The geometric mean can be motivated by noting that it is the exponent of the arithmetic mean of the logarithm:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\overline{x}_{geo}=\left(\prod_{i}x_{i}\right)^{\frac{1}{n}}=exp\left[\frac{1}{n}\sum_{i}log(x_{i})\right]=exp\left[\overline{log(x)}\right]$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So it makes sense to take the exponent of the standard deviations of the logs as the equivalent measure:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$s_{x,geo}=exp\left[s_{log(x)}\right]=exp\left[\sqrt{\overline{\{log(x)\}^{2}}-\{\overline{log(x)}\}^{2}}\right]$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where:&#xA;$$\overline{log(x)}=\frac{1}{n}\sum_{i}log(x_{i})$$&#xA;$$\overline{\{log(x)\}^{2}}=\frac{1}{n}\sum_{i}[log(x_{i})]^2$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, in a particular context, better estimates of error may be more appropriate in a specific modeling context, such as the (observed) fisher information.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-04-17T05:38:56.727" />
  <row Id="9640" PostTypeId="2" ParentId="9627" CreationDate="2011-04-17T06:15:50.957" Score="1" ViewCount="" Body="&lt;p&gt;I'm not sure what field your work is in, so this may or may not be of any help- but I'm most familiar with using SPSS with psychological constructs. In my experience, if I have a few variables predicting an outcome variable (or dependent variable) in a regression, and I have one or more independent variables show up as significant predictors, the next step is to see which ones are more incrementally important than others. One way to approach this is with hierarchical regression. This basically answers the question &quot;If I already have 'variable four' to predict my outcome variable, do any of the other variables provide a statistically significant increase in predictive power?&quot; SPSS has a pretty clear way of analyzing this, as I'm sure R and SAS do as well. So, I think hierarchical regression might be your next step in finding out if 'variable four' really is your best bet in predicting your outcome factor. The others who've responded have provided a good discussion of the issues in correlation-causation, so I'll leave that alone... Good luck!&lt;/p&gt;&#xA;" OwnerUserId="4198" LastActivityDate="2011-04-17T06:15:50.957" />
  <row Id="9641" PostTypeId="2" ParentId="9629" CreationDate="2011-04-17T08:16:50.087" Score="4" ViewCount="" Body="&lt;p&gt;Correct me if I'm wrong, but I think this can be done in R using this command&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; chisq.test(c(15,13,10,17))&#xA;&#xA;    Chi-squared test for given probabilities&#xA;&#xA;data:  c(15, 13, 10, 17) &#xA;X-squared = 1.9455, df = 3, p-value = 0.5838&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This assumes proportions of 1/4 each. You can modify expected values via argument &lt;code&gt;p&lt;/code&gt;. For example, you think people may prefer (for whatever reason) one color over the other(s).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; chisq.test(c(15,13,10,17), p = c(0.5, 0.3, 0.1, 0.1))&#xA;&#xA;    Chi-squared test for given probabilities&#xA;&#xA;data:  c(15, 13, 10, 17) &#xA;X-squared = 34.1515, df = 3, p-value = 1.841e-07&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="144" LastActivityDate="2011-04-17T08:16:50.087" CommentCount="8" />
  <row Id="9642" PostTypeId="2" ParentId="9629" CreationDate="2011-04-17T11:21:53.427" Score="2" ViewCount="" Body="&lt;p&gt;The test statistic for Pearson's chi-square test is &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\sum_{i=1}^{n} \frac{(O_i - E_i)^2}{E_i}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you write $o_i = \dfrac{O_i}{n}$ and $e_i = \dfrac{E_i}{n}$ to have proportions, where $n=\sum_{i=1}^{n} O_i$ is the sample size and $\sum_{i=1}^{n} e_i =1$, then the test statistic is is equal to &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$n \sum_{i=1}^{n} \frac{(o_i - e_i)^2}{e_i}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;so a test of the significance of the observed proportions depends on the sample size, much as one would expect. &lt;/p&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-04-17T11:21:53.427" />
  <row Id="9643" PostTypeId="1" CreationDate="2011-04-17T12:42:40.607" Score="1" ViewCount="38" Body="&lt;p&gt;I am using R software (R commander) to cluster my data. I have a smaller subset of my data containing 200 rows and about 800 columns. I am getting the following error when trying kmeans cluster and plot on a graph:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;'princomp' can only be used with more units than variables&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I then created a test doc of 10 row and 10 columns whch plots fine but when I add an extra column I get te error again. Why is this? I need to be able to plot my cluster. When I view my data set after performing kmeans on it I can see the extra results column which shows which clusters they belong to.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there anything I am doing wrong, can I ger rid of this error and plot my larger sample? &lt;/p&gt;&#xA;" OwnerUserId="4020" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-17T12:56:15.330" LastActivityDate="2011-04-17T13:10:59.397" Title="Clustering can be plotted only with more units than variables?" Tags="&lt;r&gt;&lt;clustering&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="9644" PostTypeId="2" ParentId="9643" CreationDate="2011-04-17T13:03:21.863" Score="1" ViewCount="" Body="&lt;p&gt;The clustering itself has no problems with the p&gt;n situation, however the visualization internally uses &lt;code&gt;princomp&lt;/code&gt; (which is incapable of handling p&gt;n) to plot the similarity space projection.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can't fix that, at most try to reproduce similar graph by obtaining similarity space projection with &lt;code&gt;cmdscale(dist(...))&lt;/code&gt; and coloring the points with clusters.&lt;/p&gt;&#xA;" OwnerUserId="88" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-17T13:10:59.397" LastActivityDate="2011-04-17T13:10:59.397" />
  <row Id="9645" PostTypeId="1" AcceptedAnswerId="9646" CreationDate="2011-04-17T14:01:40.170" Score="3" ViewCount="75" Body="&lt;p&gt;Hello,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm kind of new doing data mining, so sorry if my question is not very clear. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm working in a project that is aiming to do data mining over the interactions of the students with a e-learning platform. So, I'm trying to generate decision trees with some data that I collected (number of times that student use a resource, number of activities done, number quizzes taken, etc). The trees that I'm getting are interesting for me and according with the cross-validation tests, it's kind of accurate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my doubt is the following, I generated the tree with the data collected over two months and there were 44 students in the course. Is this enough data to trust in the tree? I have only the 44 instances that gather the two months of interactions... &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advanced for your help.&lt;/p&gt;&#xA;" OwnerUserId="4203" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-17T14:50:36.583" LastActivityDate="2011-05-17T07:51:12.127" Title="Minimum number of instances to create a decision tree" Tags="&lt;decision-trees&gt;&lt;weka&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="9646" PostTypeId="2" ParentId="9645" CreationDate="2011-04-17T14:45:36.953" Score="1" ViewCount="" Body="&lt;p&gt;{I'm guessing that for each person and each variable you have a single value covering the 2 months--i.e., you don't have repeated measures.}  44 sounds like an awfully small number.  Is it a random sample of a larger population?  The answer to that, for one thing, would affect your confidence in the findings. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I won't say it's impossible to achieve useful, replicable results in this situation, but it seems unlikely.  Data mining via decision trees is an opportunistic process and requires crossvalidation, probably even more than other modelling procedures do.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd also like to know, if you're getting statistically significant differences that form the basis for the branchings, what criterion are you using for significance?&lt;/p&gt;&#xA;" OwnerUserId="2669" LastActivityDate="2011-04-17T14:45:36.953" CommentCount="5" />
  <row Id="9647" PostTypeId="1" CreationDate="2011-04-17T17:16:26.730" Score="2" ViewCount="28" Body="&lt;p&gt;Sometimes they ask questions in different orders, or use different prompts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or datasets with instruments (with at least one variable randomized)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to use at least one of them for my causal modelling course (Stat 566), whose syllabus is at &lt;a href=&quot;https://www.stat.washington.edu/tsr/s566/syllabus566.pdf&quot; rel=&quot;nofollow&quot;&gt;https://www.stat.washington.edu/tsr/s566/syllabus566.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="4206" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-04-17T18:11:52.157" LastActivityDate="2011-04-17T18:41:11.000" Title="What are some examples of public datasets that have randomized instruments? " Tags="&lt;dataset&gt;" AnswerCount="1" />
  <row Id="9650" PostTypeId="2" ParentId="9647" CreationDate="2011-04-17T18:41:11.000" Score="2" ViewCount="" Body="&lt;p&gt;Donald Green in Yale's political science department has posted the replication data for many of his experiments. You can find them here:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://vote.research.yale.edu//replication.html&quot; rel=&quot;nofollow&quot;&gt;Donald Green's Replication Data&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Green is also the director for the Yale Institution for Social and Policy Studies. They have a replication data archive here:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://isps.research.yale.edu/research-2/data/&quot; rel=&quot;nofollow&quot;&gt;ISPS Data Archive&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Of course, a lot of data is also available at the ICPSR and Dataverse websites, though you will have to dig through to find experimental data. You can find them at the links below: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.icpsr.umich.edu/icpsrweb/ICPSR/&quot; rel=&quot;nofollow&quot;&gt;ICPSR&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://thedata.org/home&quot; rel=&quot;nofollow&quot;&gt;Dataverse&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Depending on your interests, many of the top journals (in the social sciences, at least) have started to institute policies requiring that researchers make their data available for replication purposes. I suggest reading through several academic journals, find an article that interests you, and then look up the researcher's home page. Many of them make their data readily available, and most &lt;em&gt;should&lt;/em&gt; be happy to share it if it's not already posted.&lt;/p&gt;&#xA;" OwnerUserId="3265" LastActivityDate="2011-04-17T18:41:11.000" />
  <row Id="9653" PostTypeId="1" AcceptedAnswerId="9655" CreationDate="2011-04-17T21:55:55.793" Score="9" ViewCount="161" Body="&lt;p&gt;I've learnt that small sample size may lead to insufficient power and type 2 error. However, I have the feeling that small samples just may be generally unreliable and may lead to any kind of result by chance. Is that true?&lt;/p&gt;&#xA;" OwnerUserId="4176" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-18T07:45:15.140" LastActivityDate="2011-05-05T12:28:58.153" Title="Can a small sample size cause type 1 error?" Tags="&lt;hypothesis-testing&gt;&lt;small-sample&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="3" />
  <row Id="9654" PostTypeId="1" AcceptedAnswerId="9660" CreationDate="2011-04-17T22:31:33.500" Score="8" ViewCount="57" Body="&lt;p&gt;Many randomised controlled trial (RCT) papers report significance tests on baseline parameters just after/before randomisation to show that the groups are indeed similar. This is often part of a &quot;baseline characteristics&quot; table. However, significance tests measure the probability of getting the observed (or a stronger) difference by chance, aren't they? And if the test is significant we conclude that there is a &lt;strong&gt;true difference because a random difference of that extent would be unlikely&lt;/strong&gt;. Does a significance test make any sense after randomisation when we know that any &lt;strong&gt;difference must be due to chance&lt;/strong&gt;?&lt;/p&gt;&#xA;" OwnerUserId="4176" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-18T10:52:12.323" LastActivityDate="2011-04-18T10:52:12.323" Title="Does significance test make sense to compare randomised groups at baseline?" Tags="&lt;statistical-significance&gt;&lt;clinical-trials&gt;" AnswerCount="1" />
  <row Id="9655" PostTypeId="2" ParentId="9653" CreationDate="2011-04-17T22:41:17.137" Score="14" ViewCount="" Body="&lt;p&gt;As a general principle, &lt;strong&gt;small sample size will not increase the Type I error rate&lt;/strong&gt; for the simple reason that the test is arranged to control the Type I rate.  (There are minor technical exceptions associated with discrete outcomes, which can cause the &lt;em&gt;nominal&lt;/em&gt; Type I rate not to be achieved exactly especially with small sample sizes.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is an &lt;strong&gt;important principle&lt;/strong&gt; here: if your test has acceptable size (= nominal Type I rate) and acceptable power for the effect you're looking for, then even if the sample size is small it's ok.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The danger&lt;/strong&gt; is that if we otherwise know little about the situation--maybe these are all the data we have--then we might be concerned about &quot;Type III&quot; errors: that is, model mis-specification.  They can be difficult to check with small sample sets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a &lt;strong&gt;practical example&lt;/strong&gt; of the interplay of ideas, I will share a story.  Long ago I was asked to recommend a sample size to confirm an environmental cleanup.  This was during the &lt;em&gt;pre-cleanup&lt;/em&gt; phase before we had any data.  My plan called for analyzing the 1000 or so samples that &lt;em&gt;would be&lt;/em&gt; obtained during cleanup (to establish that enough soil had been removed at each location) to assess the post-cleanup mean and variance of the contaminant concentration.  Then (to simplify greatly), I said we would use a textbook formula--based on specified power and test size--to determine the number of independent confirmation samples that would be used to prove the cleanup was successful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What made this memorable was that after the cleanup was done, the formula said to use only 3 samples.  Suddenly my recommendation did not look very credible!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reason for needing only 3 samples is that the cleanup was aggressive and worked well.  It reduced average contaminant concentrations to about 100 give or take 100 ppm, consistently below the target of 500 ppm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the end &lt;strong&gt;this approach worked&lt;/strong&gt; because we had obtained the 1000 previous samples (albeit of lower analytical quality: they had greater measurement error) to establish that the statistical assumptions being made were in fact good ones for this site.  &lt;em&gt;That&lt;/em&gt; is how the potential for Type III error was handled.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One more twist for your consideration: knowing the regulatory agency would never approve using just 3 samples, I recommended obtaining 5 measurements.  These were to be made of 25 random samples of the entire site, &lt;a href=&quot;http://www.stat.psu.edu/~gpp/pdfs/TR2001-0202.pdf&quot; rel=&quot;nofollow&quot;&gt;composited&lt;/a&gt; in groups of 5.  Statistically there would be only 5 numbers in the final hypothesis test, but we achieved greater power to detect an isolated &quot;hot spot&quot; by taking 25 physical samples.  This highlights the important relationship between how many &lt;em&gt;numbers&lt;/em&gt; are used in the test and &lt;em&gt;how they were obtained.&lt;/em&gt;  There's more to statistical decision making than just algorithms with numbers!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To my everlasting relief, the five composite values confirmed the cleanup target was met.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-04-17T22:41:17.137" CommentCount="2" />
  <row Id="9656" PostTypeId="1" CreationDate="2011-04-18T00:43:44.917" Score="2" ViewCount="52" Body="&lt;p&gt;Hello everybody,&#xA;am writing a simple R script to test the spectral clustering algorithm but for the eigenvalues I don't get them all positive and lambda0 is different from 0. here is my script&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Dsqrt&amp;lt;-matrix(0,length(V(g)),length(V(g)))  # The D^-0.5 matrix&#xA;   for(i in 1:(length(V(g))-1))&#xA;         Dsqrt[i,i]&amp;lt;-1/sqrt(degree(g,V(g)[i-1]))        &#xA;&#xA;   L&amp;lt;-graph.laplacian(g, normalized=TRUE) #Here L[i,i]=1 and L[i,j]= 1/sqrt(d[i]*d[j])&#xA;   L&amp;lt;-Dsqrt %*% L %*% Dsqrt  &#xA;   Eigns&amp;lt;-eigen(L)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And when I check &lt;code&gt;Eigns$values&lt;/code&gt; sometimes I get some negative values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can any one point me my error?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="2325" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-04-18T07:01:02.667" LastActivityDate="2011-04-18T07:01:02.667" Title="Problem with R code for spectral clustering" Tags="&lt;r&gt;&lt;clustering&gt;" CommentCount="7" />
  <row Id="9657" PostTypeId="2" ParentId="8371" CreationDate="2011-04-18T01:45:03.810" Score="3" ViewCount="" Body="&lt;p&gt;As whuber says, the short answer is that quota samples are the &quot;poster child for outmoded, known-bad sampling methods&quot; and &quot;have long been discredited.&quot;  The longer answer is that there may be conditions under which &quot;quota-like&quot; samples can work reasonably well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Exhibit A here is recent work on reconstructing representative results from opt-in Internet panels. &lt;a href=&quot;http://www.laits.utexas.edu/txp_media/html/poll/files/Rivers_matching.pdf&quot; rel=&quot;nofollow&quot;&gt;This paper&lt;/a&gt; gives the statistical grounding for this approach.  To make a long story short, typical sampling schemes 1) draw a random sample, 2) attempt to recruit subjects, and then 3) add post-stratification weights to compensate for differences in who responds.  In the opt-in approach, you 1) recruit subjects non-randomly, 2) compare responses to a representative baseline, and 3) add weights to compensate for the differences.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In terms of practice, opt-in sampling is similar to quota sampling, but the statistical foundation is more developed. The upside is that you can make claims about representative sampling, confidence intervals, etc. The downside is that your claims are based on difficult-to-verify assumptions about how people self-select into your sample.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A lot of people are skeptical about these methods -- they sound too much like quota sampling. But &lt;a href=&quot;http://sjhill.bol.ucla.edu/HillLoVavreckZaller2007.pdf&quot; rel=&quot;nofollow&quot;&gt;some evidence&lt;/a&gt; suggests that opt-in sampling can work well at least some of the time.  So despite the controversy, &lt;a href=&quot;http://corp.yougov.com/&quot; rel=&quot;nofollow&quot;&gt;Polimetrix/YouGov&lt;/a&gt; (an early adopter of the opt-in sampling model) seems to be doing reasonably well.  Among other things, they've done all the data collection for the &lt;a href=&quot;http://projects.iq.harvard.edu/cces/home&quot; rel=&quot;nofollow&quot;&gt;Cooperative Congressional Election Study&lt;/a&gt;, a series of recent academic U.S. national election studies.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(I'm pretty sure ICPSR carries this data.  If not, Harvard's social science dataverse certainly does. Lots of academics are using data from these samples.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyway, you asked about quota sampling.  As you can see already in the comment thread here, any well-trained pollster will tell you that quota sampling is bunk. The jury is still out on opt-in sampling. For the time being, if you want to draw confidence intervals around quota samples, I'd say these methods are your best bet.&lt;/p&gt;&#xA;" OwnerUserId="4110" LastEditorUserId="3582" LastEditorDisplayName="" LastEditDate="2011-04-18T15:55:58.593" LastActivityDate="2011-04-18T15:55:58.593" CommentCount="3" />
  <row Id="9658" PostTypeId="1" CreationDate="2011-04-18T02:07:59.667" Score="1" ViewCount="45" Body="&lt;p&gt;I am trying to understand the statistics of an X-ray scintillation detector.&#xA;Poisson distributed X-ray photons enter the scintillation crystal and deposit their energy. This energy is given off in the form of scintillation light photons (typically thousands per X-ray photon). These are given off in an exponential decay. A photo detector collects these light photons over a certain time window and produces a signal. The measured signal has a mean and a variance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I understand correctly what I read about compound Poisson processes my measured mean signal would be the product of the mean number of X-rays times the mean number of light photons per X-ray (which makes sense) and the variance of my signal should be the mean number of X-rays times the Expectation value of the squared number of light photons per X-ray.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Am I on the right track here? Can somebody point me to some literature?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you all!&lt;/p&gt;&#xA;" LastActivityDate="2011-04-18T02:26:53.523" Title="Am I looking at a compound Poisson process?" Tags="&lt;stochastic-processes&gt;" CommentCount="1" />
  <row Id="9659" PostTypeId="1" CreationDate="2011-04-18T02:52:39.853" Score="4" ViewCount="96" Body="&lt;p&gt;I wish to infer the posterior distribution on the probability of success $\theta$ in some binomial process, the twist being that I know that $\theta$ lies in the interval [0.5, 1]. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The trouble is that a Beta distribution that is supported on [0.5, 1] is no longer conjugate. In particular, the prior looks like this ($p$ is the pdf):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$p(\theta) \propto {(\theta-0.5)}^{\alpha-1} {(1-\theta)}^{\beta-1}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The likelihood, given $x$ successes in $n$ trials, is the usual binomial likelihood:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$p(n,x | \theta) \propto \theta^x {(1-\theta)}^{n-x}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The posterior, as it stands, looks like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$p( \theta | n, x) \propto \theta^x {(\theta-0.5)}^{\alpha-1} {(1-\theta)}^{n-x+\beta-1}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to be able to make CDF calculations (i.e. compute integrals) of the posterior efficiently. (numerical integration, e.g., may be too slow for my purposes). I am unable to figure out how to do this. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Much appreciate any help the forum can provide. &lt;/p&gt;&#xA;" OwnerUserId="4209" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-18T07:40:42.710" LastActivityDate="2011-05-19T19:02:59.720" Title="Fast integration of a posterior distribution" Tags="&lt;bayesian&gt;&lt;beta-binomial&gt;" AnswerCount="2" CommentCount="3" FavoriteCount="1" />
  <row Id="9660" PostTypeId="2" ParentId="9654" CreationDate="2011-04-18T03:25:38.537" Score="4" ViewCount="" Body="&lt;p&gt;A hypothesis test would be nonsensical, but a significance test may be useful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The hypothesis test would be testing a null hypothesis that is already known to be true, as your question makes clear. It is silly to apply a statistical test to any hypothesis that has a truth value already known via completely reliable information.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A significance test provides a P value that, again as you already say, indicates the probability of getting data as extreme or more extreme given the null hypothesis. However, it seems to me that such a P value can be interpreted in a manner that equates to an answer to the question &quot;How often might I expect to see a difference in baseline values as large as this time, or larger?&quot; The answer might be useful even if it is not clear for what purpose.&lt;/p&gt;&#xA;" OwnerUserId="1679" LastActivityDate="2011-04-18T03:25:38.537" />
  <row Id="9661" PostTypeId="2" ParentId="9625" CreationDate="2011-04-18T03:43:18.880" Score="1" ViewCount="" Body="&lt;p&gt;A permutations test can easily be set up to use the (mean - mode difference) as its test statistic. That would give you an exact P value for the difference.&lt;/p&gt;&#xA;" OwnerUserId="1679" LastActivityDate="2011-04-18T03:43:18.880" CommentCount="5" />
  <row Id="9662" PostTypeId="1" AcceptedAnswerId="9703" CreationDate="2011-04-18T04:37:50.350" Score="4" ViewCount="63" Body="&lt;p&gt;I have 3 different groups: A, B, and C:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;A (has a medical condition) has 30 entries &lt;/li&gt;&#xA;&lt;li&gt;B (another medical condition) has 31 entries &lt;/li&gt;&#xA;&lt;li&gt;C (control group) has 55 entries&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;All participants have information on the same set of variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to assess whether there is a statistically significant difference in risk of a specific disease between:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Group A and C &lt;/li&gt;&#xA;&lt;li&gt;Group B and C &lt;/li&gt;&#xA;&lt;li&gt;Group B and A&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Also, all groups need to be adjusted for other known risk factors.&lt;/p&gt;&#xA;" OwnerUserId="4210" LastEditorUserId="4210" LastEditorDisplayName="" LastEditDate="2011-04-18T12:19:04.447" LastActivityDate="2011-04-18T19:55:39.293" Title="How to assess differential risk of disease across three groups after adjusting for other risk factors?" Tags="&lt;correlation&gt;&lt;statistical-significance&gt;&lt;spss&gt;" AnswerCount="1" />
  <row Id="9663" PostTypeId="1" CreationDate="2011-04-18T05:01:08.790" Score="5" ViewCount="68" Body="&lt;p&gt;If an item follows normal distribution, average also follows normal distribution. What about minimum and maximum?&lt;/p&gt;&#xA;" OwnerUserId="4211" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-18T07:34:28.313" LastActivityDate="2011-04-18T14:37:04.670" Title="Distribution of extremal values" Tags="&lt;normal-distribution&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="2" />
  <row Id="9664" PostTypeId="1" AcceptedAnswerId="9722" CreationDate="2011-04-18T05:44:33.810" Score="17" ViewCount="234" Body="&lt;p&gt;Suppose I have a set of sample data from an unknown or complex distribution, and I want to perform some inference on a statistic $T$ of the data. My default inclination is to just generate a bunch of bootstrap samples with replacement, and calculate my statistic $T$ on each bootstrap sample to create an estimated distribution for $T$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are examples where this is a bad idea?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, one case where naively performing this bootstrap would fail is if I'm trying to use the bootstrap on time series data (say, to test whether I have significant autocorrelation). The naive bootstrap described above (generating the $i$th datapoint of the nth bootstrap sample series by sampling with replacement from my original series) would (I think) be ill-advised, since it ignores the structure in my original time series, and so we get fancier bootstrap techniques like the block bootstrap.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To put it another way, what is there to the bootstrap besides &quot;sampling with replacement&quot;?&lt;/p&gt;&#xA;" OwnerUserId="1106" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-04-19T13:03:21.823" LastActivityDate="2011-04-20T02:08:28.043" Title="What are examples where a &quot;naive bootstrap&quot; fails?" Tags="&lt;hypothesis-testing&gt;&lt;confidence-interval&gt;&lt;bootstrap&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="6" />
  <row Id="9665" PostTypeId="2" ParentId="9663" CreationDate="2011-04-18T05:46:01.393" Score="7" ViewCount="" Body="&lt;p&gt;Hi!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should have a look at the &lt;a href=&quot;http://mathworld.wolfram.com/OrderStatistic.html&quot; rel=&quot;nofollow&quot;&gt;order statistics&lt;/a&gt;. &#xA;Here is a very brief overview.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let $X_{1}, \ldots X_{n}$ be an i.i.d. sample of size $n$ drawn from a population with distribution function $F$ and probability density function $f$. Define $Y_{1}=X_{(1)}, \ldots, Y_{r} = X_{(r)}, \ldots, Y_{n}=X_{(n)}$, where $X_{(r)}$ denotes the $r$th order statistic of the sample $X_{1}, \ldots X_{n}$, i.e., its $r$th smallest value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It can be shown that the joint probability density function of $Y_{1}, \ldots, Y_{n}$ is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$f_{X_{(1)}, \ldots, X_{(n)}}(y_{1}, \ldots, y_{n}) = n! \prod_{i=1}^{n} f(y_{i})$ if $y_{1} &amp;lt; y_{2} &amp;lt; \ldots &amp;lt; y_{n}$ and $0$ otherwise.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By integrating the previous equation we get&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$f_{X_{(r)}}(x) = \frac{n!}{(r - 1)! (n - r)!} f(x) (F(x))^{r-1} (1 - F(x))^{n - r}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In particular, for the minimum and maximum, we respectively have&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$f_{X_{(1)}}(x) = n f(x) (1 - F(x))^{n-1}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$f_{X_{(n)}}(x) = n f(x) (F(x))^{n - 1}$&lt;/p&gt;&#xA;" OwnerUserId="3019" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-04-18T06:59:14.667" LastActivityDate="2011-04-18T06:59:14.667" CommentCount="2" />
  <row Id="9666" PostTypeId="1" CreationDate="2011-04-18T06:10:33.577" Score="4" ViewCount="68" Body="&lt;p&gt;My new computer clock runs at a rate that changes in odd steps over time, even after I tuned it via the Linux adjtimex software.  Here is a plot of the change in the cumulative clock drift for each of about 1700 samples taken every 10000 seconds, with a bunch of missed points and outliers when I was off the network and ntpdate wouldn't work.  E.g. early on, the clock gained about 0.09 seconds every 10000 seconds (9 ppm).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://bcn.boulder.co.us/~neal/media/images/clock-rate.png&quot; alt=&quot;clock-rate.png&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm looking for some clever library functions that can automatically identify what I'll call the various statistical &quot;modes&quot; of this data set - i.e. there is one mode in the middle where y is about -0.02 for a long time, then another one near 0.09 early on and one at 0.202 at the end.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most code for finding the mode of data that I've seen deals with integers and discrete data, but as you see this has pretty messy floating point values.  At any rate, ideally I'd like a summary that automatically finds the modes I identified above, and also gives me a standard deviation for each.  Start/stop points for each mode for extra credit.  Python code preferred.&lt;/p&gt;&#xA;" OwnerUserId="2434" LastActivityDate="2011-04-25T20:13:49.233" Title="Identifying modes in floating point data" Tags="&lt;mode&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="9667" PostTypeId="1" CreationDate="2011-04-18T07:44:31.833" Score="5" ViewCount="124" Body="&lt;p&gt;I am new to time series modelling altogether. But I am aware about regression modelling and some data mining algorithms like decision trees.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to learn time series from scratch. My background is Mathematics. Please suggest me any good book/material/web resource for step by step learning with case studies.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="861" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-18T08:24:47.947" LastActivityDate="2011-04-19T15:57:14.973" Title="Learning material about time series" Tags="&lt;time-series&gt;&lt;books&gt;" AnswerCount="2" CommentCount="4" FavoriteCount="3" />
  <row Id="9668" PostTypeId="2" ParentId="9617" CreationDate="2011-04-18T07:47:28.053" Score="2" ViewCount="" Body="&lt;p&gt;@charles.y.zheng and @cardinal gave very good answers, I will add my two cents. Yes the characteristic function might look like unnecessary complication, but it is a powerful tool which can get you results. If you are trying to prove something with cumulative distribution function it is always advisable to check whether it is not possible to get the result with characteristic function. This sometimes gives very short proofs. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Although at first the characteristic function looks unintuitive way of working with probability distributions, there are some powerful results directly related with it, which imply that you cannot discard this concept as a mere mathematical amusement. For example my favorite result in probability theory is that any &lt;a href=&quot;http://en.wikipedia.org/wiki/Infinite_divisibility_%28probability%29&quot; rel=&quot;nofollow&quot;&gt;infinitely divisible distribution&lt;/a&gt; has the unique &lt;a href=&quot;http://en.wikipedia.org/wiki/Levy-Khintchine_representation&quot; rel=&quot;nofollow&quot;&gt;Lévy–Khintchine representation&lt;/a&gt;. Combined with the fact that the infinitely divisible distributions are the only possible distribution for limits of sums of independent random variables (excluding bizarre cases) this is a deep result using which central limit theorem is derived. &lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-04-18T07:47:28.053" />
  <row Id="9669" PostTypeId="2" ParentId="9280" CreationDate="2011-04-18T09:08:31.047" Score="3" ViewCount="" Body="&lt;p&gt;Note that @whuber adresses all the points raised in the question, I will provide only the mathematical details. The regression with spatial error assumes that model is the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$y=X\beta+u$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $Euu&amp;#39;=\Omega$. The efficient estimate of $\beta$  then is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\hat\beta=(X&amp;#39;\Omega X)^{-1}X&amp;#39;\Omega y$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you have new observation it also follows the model&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$y^*=X^*\beta+u^*$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We predict $y^*$ by $X^*\beta$ so the prediction error is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$y^*-\hat{y^*}=X^*\hat\beta+u^*$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For linear model $u^*$ will be independent of $u$ so &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Var(y^*-\hat{y^*})=Var(X^*\hat\beta+u^*)=Var(X^*\hat\beta)+Var(u^*)=\sigma^2+\sigma^2X^{* } (X&amp;#39;X)^{-1})(X^{* })&amp;#39;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the spatial error model we have&#xA;\begin{align*}&#xA;Var(X^{* }\hat\beta)=&#xA;Var(X^{* } (X'\Omega X)^{-1}X'\Omega u)&#xA;=X^{* } (X'\Omega X)^{-1})(X^{* })'&#xA;\end{align*}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But now $u^*$ correlates with $u$ and we do not have such convenient decomposition. Thus the formula in your question cannot be used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The general approach how to calculate the error is describe in &lt;a href=&quot;http://www.jstor.org/pss/2281645?searchUrl=%2Faction%2FdoBasicSearch%3FQuery%3Dgoldberger%2Bjournal%2Bof%2Bamerican%2Bstatistical%2Bassociation%26acc%3Doff%26wc%3Don&amp;amp;Search=yes&quot; rel=&quot;nofollow&quot;&gt;this article&lt;/a&gt;. It shows how to get the best unbiased linear prediction in linear model when the model errors are correlated. Note that this heavily relies on the fact that you have reliable estimate of $\Omega$, so @whuber answer applies very strongly. Even if you have correct formulas they might give bad results if it is known that $\Omega$ cannot be reliably estimated.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-04-18T13:52:57.690" LastActivityDate="2011-04-18T13:52:57.690" />
  <row Id="9670" PostTypeId="2" ParentId="6989" CreationDate="2011-04-18T09:09:04.673" Score="2" ViewCount="" Body="&lt;p&gt;No need to leave the lme4 package to account for overdispersion; just include a random effect for observation number. The BUGS/JAGS solutions mentioned are probably overkill for you, and if they aren't, you should have the easy to fit lme4 results for comparison.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data$obs_effect&amp;lt;-1:nrow(data)&#xA;overdisp.fit&amp;lt;-lmer(y~1+obs_effect+x+(1|obs_effect)+(1+x|subject_id),data=data,family=poisson)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is discussed here: &lt;a href=&quot;http://article.gmane.org/gmane.comp.lang.r.lme4.devel/4727&quot; rel=&quot;nofollow&quot;&gt;http://article.gmane.org/gmane.comp.lang.r.lme4.devel/4727&lt;/a&gt; informally and here: &lt;a href=&quot;http://www.cefe.cnrs.fr/esp/publis/TB/TBElston_Parasitology2001.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.cefe.cnrs.fr/esp/publis/TB/TBElston_Parasitology2001.pdf&lt;/a&gt; academically.&lt;/p&gt;&#xA;" OwnerUserId="1893" LastActivityDate="2011-04-18T09:09:04.673" />
  <row Id="9671" PostTypeId="1" CreationDate="2011-04-18T09:15:14.207" Score="3" ViewCount="44" Body="&lt;p&gt;I am assessing how good different features are for unsupervised classification of a set of objects. For each different feature I test, I have computed a feature vector that describes the object. I then want to get a metric out for how 'good' this vector is at separating the objects into their respective classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My current method for doing this is to use &lt;a href=&quot;http://en.wikipedia.org/wiki/K-means_clustering&quot; rel=&quot;nofollow&quot;&gt;k-means&lt;/a&gt; to cluster the objects in feature space, and then use the &lt;a href=&quot;http://en.wikipedia.org/wiki/Rand_index#Adjusted_Rand_index&quot; rel=&quot;nofollow&quot;&gt;Adjusted Rand Index&lt;/a&gt; to assess the quality of the clustering.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, is there a better way to assess the 'goodness' of the feature vectors, perhaps using something like &lt;a href=&quot;http://en.wikipedia.org/wiki/Mutual_information&quot; rel=&quot;nofollow&quot;&gt;mutual information&lt;/a&gt;? One drawback with using k-means and the ARI is it provides no indication of the tightness of clustering.&lt;/p&gt;&#xA;" OwnerUserId="4213" LastActivityDate="2011-05-24T15:05:40.330" Title="How can I assess how descriptive feature vectors are?" Tags="&lt;clustering&gt;&lt;classification&gt;&lt;feature-selection&gt;&lt;unsupervised-learning&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="9672" PostTypeId="2" ParentId="9667" CreationDate="2011-04-18T09:17:26.670" Score="4" ViewCount="" Body="&lt;p&gt;I have yet still not found the time series book which I like. Here is the list of books which I found very useful:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://books.google.com/books?id=B8_1UBmqVUoC&amp;amp;printsec=frontcover&amp;amp;dq=hamilton%20time%20series&amp;amp;hl=fr&amp;amp;ei=8f-rTcawKomXOte0qIIK&amp;amp;sa=X&amp;amp;oi=book_result&amp;amp;ct=result&amp;amp;resnum=1&amp;amp;ved=0CDAQ6AEwAA#v=onepage&amp;amp;q&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;Time Series Analysis&lt;/a&gt; by J. D. Hamilton. This is &lt;strong&gt;the&lt;/strong&gt; book which contains practically everything.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://books.google.com/books?id=VXGvQgAACAAJ&amp;amp;dq=enders%20time%20series&amp;amp;hl=fr&amp;amp;ei=VgCsTe_pN4ubOpfo2NYJ&amp;amp;sa=X&amp;amp;oi=book_result&amp;amp;ct=result&amp;amp;resnum=2&amp;amp;ved=0CDQQ6AEwAQ&quot; rel=&quot;nofollow&quot;&gt;Applied econometric time series&lt;/a&gt; by W. Enders. Classical reference&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://books.google.com/books?id=CrPRZexTZLkC&amp;amp;printsec=frontcover&amp;amp;dq=introductory%20Econometrics%20for%20Finance&amp;amp;hl=fr&amp;amp;ei=-ACsTc-ZGovsOYbFxIMK&amp;amp;sa=X&amp;amp;oi=book_result&amp;amp;ct=result&amp;amp;resnum=1&amp;amp;ved=0CDYQ6AEwAA#v=onepage&amp;amp;q&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;Introductory econometrics for finance&lt;/a&gt; by Ch. Brooks. I use this book for teaching time-series to students with little mathematical background. Nevertheless it is a good book for getting the ideas without too much mathematical detail.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-04-18T09:17:26.670" CommentCount="2" />
  <row Id="9673" PostTypeId="2" ParentId="9587" CreationDate="2011-04-18T09:28:42.100" Score="0" ViewCount="" Body="&lt;p&gt;No, that isn't really it. One side point is you should consider looking into multi-modal inference a la Anderson and Burnham. However, there are plenty of resources, many of which also refer to the P+B book; such as this one, with code: &lt;a href=&quot;http://glmm.wikidot.com/basic-glmm-simulation&quot; rel=&quot;nofollow&quot;&gt;http://glmm.wikidot.com/basic-glmm-simulation&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, try investigating these calls:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(arm)&#xA;library(lme4)&#xA;fit&amp;lt;-lmer(y~1+x1*x2+(1+x*x2|id))&#xA;?sim&#xA;fit.sim&amp;lt;-sim(fit, n=1000)&#xA;coef.sim &amp;lt;- coef(fit.sim)&#xA;fixef.sim &amp;lt;- fixef(fit.sim)&#xA;ranef.sim &amp;lt;- ranef(fit.sim)&#xA;sigma.sim &amp;lt;- sigma.hat(fit.sim)&#xA;dotplot(ranef(gm1, postVar=TRUE))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1893" LastActivityDate="2011-04-18T09:28:42.100" />
  <row Id="9674" PostTypeId="1" CreationDate="2011-04-18T09:37:37.580" Score="1" ViewCount="68" Body="&lt;p&gt;I have a bit of a problem with the &lt;a href=&quot;http://cran.r-project.org/web/packages/tm/index.html&quot; rel=&quot;nofollow&quot;&gt;tm&lt;/a&gt; R package for cleaning text documents.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is how my code looks like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(&quot;tm&quot;)&#xA;&#xA;# import text files in corpus text – ok&#xA;c_txt &amp;lt;- Corpus((DirSource(directory = &quot;.&quot;, pattern =&quot;txt&quot;, &#xA;                encoding = &quot;UTF-8&quot;)), readerControl = list(language = &quot;rus&quot;))&#xA;&#xA;# convert to Lower Case – ok&#xA;txt_cl &amp;lt;- tm_map(c_txt, tolower)&#xA;&#xA;# remove Stopwords - not working&#xA;txt_cl &amp;lt;- tm_map(txt_t, removeWords, stopwords(&quot;russian&quot;))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am inspecting the file &lt;code&gt;russian.dat&lt;/code&gt; which is located in the subdirectory &lt;code&gt;tm\stopwords&lt;/code&gt;. There I can notice strange characters -- not Russian stopwords, although it is UTF-8 encoding.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where is my mistake?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="4212" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-18T09:46:25.323" LastActivityDate="2011-04-18T12:04:01.667" Title="How to remove stopwords with Russian documents?" Tags="&lt;r&gt;&lt;text-mining&gt;" AnswerCount="1" CommentCount="5" />
  <row Id="9675" PostTypeId="2" ParentId="9508" CreationDate="2011-04-18T09:38:45.867" Score="0" ViewCount="" Body="&lt;p&gt;A Likelihood ratio test on an lme object is probably fine to test a certain hypothesis; but that is not an appropriate way to build or choose a model. You would do that with REMLoff and: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;anova(fit1, fit2) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;AICc is a well developed criterion for model selection that directly implies and estimates from the data the model weight given the ensemble of models you suggest. For more information, see Burnham and Anderson (2002) or this link: &lt;a href=&quot;http://web.ipac.caltech.edu/staff/fmasci/home/statistics_refs/AIC_in_modelselection.pdf&quot; rel=&quot;nofollow&quot;&gt;http://web.ipac.caltech.edu/staff/fmasci/home/statistics_refs/AIC_in_modelselection.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1893" LastActivityDate="2011-04-18T09:38:45.867" />
  <row Id="9676" PostTypeId="2" ParentId="9327" CreationDate="2011-04-18T09:50:16.247" Score="0" ViewCount="" Body="&lt;p&gt;Try the following packages on cran: Rcapture and mra. The easiest method to use in R for a log linear model, open closed or robust, M0, Mh, Mt, or Mth is well detailed here: &lt;a href=&quot;http://www.jstor.org/pss/3695557&quot; rel=&quot;nofollow&quot;&gt;http://www.jstor.org/pss/3695557&lt;/a&gt; and that is what Rcapture is based on. You forgot to specify in your lmer call that it should be a count response (family = poisson) and your Y should be the abundance associated with a particular capture history. Extracting the estimated abundances, survival probilities, etc, is well detailed in the jstor link.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the premier tool people use for this is MARK (http://www.phidot.org/software/mark/docs/book/), and there is a bridge package not on cran called RMark (http://www.phidot.org/software/mark/rmark/RMarkWorkshopNotes.pdf). RMark is the premier tool for this sort of thing and the one which certainly has the largest community. It has its own support forum at phidot.org.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, google around for Dr. Schofield's BUGS code for this problem; it is online somewhere, I think in the dissertation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Btw, if you are interested in a job in population ecology and R programming, we're hiring: &lt;a href=&quot;https://performancemanager4.successfactors.com/career?company=comScore&amp;amp;career_job_req_id=721&amp;amp;career_ns=job_listing&amp;amp;navBarLevel=JOB_SEARCH&quot; rel=&quot;nofollow&quot;&gt;https://performancemanager4.successfactors.com/career?company=comScore&amp;amp;career_job_req_id=721&amp;amp;career_ns=job_listing&amp;amp;navBarLevel=JOB_SEARCH&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1893" LastActivityDate="2011-04-18T09:50:16.247" />
  <row Id="9677" PostTypeId="2" ParentId="8501" CreationDate="2011-04-18T10:01:20.853" Score="1" ViewCount="" Body="&lt;p&gt;You could always just calculate try to calculate R^2 from the squared covariance of Ypred and Ynew over the product of their variances: &lt;a href=&quot;http://www.stator-afm.com/image-files/r-squared.gif&quot; rel=&quot;nofollow&quot;&gt;http://www.stator-afm.com/image-files/r-squared.gif&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This works for any continuous model, regardless of how the parameters are estimated.&lt;/p&gt;&#xA;" OwnerUserId="1893" LastActivityDate="2011-04-18T10:01:20.853" CommentCount="2" />
  <row Id="9678" PostTypeId="2" ParentId="9664" CreationDate="2011-04-18T10:14:17.253" Score="0" ViewCount="" Body="&lt;p&gt;The naive bootstrap depends on the sample size being large, so that the empirical CDF for the data are a good approximation to the &quot;true&quot; CDF.  This ensures that sampling from the empirical CDF is very much like sampling from the &quot;true&quot; CDF.  The extreme case is when you have only sampled one data point - bootstrapping achieves nothing here.  It will become more and more useless as it approaches this degenerate case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bootstrapping naively will not necessarily fail in times series analysis (although it may be inefficient) - if you model the series using basis functions of continuous time (such a legendre polynomials) for a trend component, and sine and cosine functions of continuous time for cyclical components (plus normal noise error term).  Then you just put in what-ever times you happen to have sampled into the likelihood function.  No disaster for bootstrapping here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any auto-correlation or ARIMA model has a representation in this format above - this model is just easier to use and I think to understand and interpret (easy to understand cycles in sine and cosine functions, hard to understand coefficients of an ARIMA model).  For example the auto-correlation function is the inverse Fourier transform of the power spectrum of a time series.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-04-19T08:24:30.707" LastActivityDate="2011-04-19T08:24:30.707" CommentCount="14" />
  <row Id="9679" PostTypeId="2" ParentId="7675" CreationDate="2011-04-18T10:22:26.577" Score="1" ViewCount="" Body="&lt;p&gt;I disagree with the other responder, you shouldn't just fit varying intercept, fixed slope by default.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;y~1+condition*distractor*direction+(1+condition*distractor*direction|subject) should at least be fit for model testing. Otherwise, you are just assuming the subject effect is only on the intercept and you are assuming there is no correlation of the slope and intercept random effects. Don't skip these crucial investigations.&#xA;See &lt;a href=&quot;http://lme4.r-forge.r-project.org/book/Ch4.pdf&quot; rel=&quot;nofollow&quot;&gt;http://lme4.r-forge.r-project.org/book/Ch4.pdf&lt;/a&gt; and&#xA;&lt;a href=&quot;http://www.stat.columbia.edu/~gelman/research/unpublished/multi.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.stat.columbia.edu/~gelman/research/unpublished/multi.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1893" LastActivityDate="2011-04-18T10:22:26.577" />
  <row Id="9680" PostTypeId="2" ParentId="5517" CreationDate="2011-04-18T10:35:40.950" Score="1" ViewCount="" Body="&lt;p&gt;It looks like your error message isn't about varying slopes, it is about correlated random effects. You can fit the uncorrelated as well: A mixed-eﬀects model with independent random eﬀects&#xA;Linear mixed model fit by REML&#xA;Formula: Reaction ~ Days + (1 | Subject) + (0 + Days | Subject)&#xA;Data: sleepstudy&lt;/p&gt;&#xA;&#xA;&lt;p&gt;from &lt;a href=&quot;http://www.stat.wisc.edu/~bates/IMPS2008/lme4D.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.stat.wisc.edu/~bates/IMPS2008/lme4D.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1893" LastActivityDate="2011-04-18T10:35:40.950" />
  <row Id="9681" PostTypeId="2" ParentId="9659" CreationDate="2011-04-18T11:41:02.280" Score="0" ViewCount="" Body="&lt;p&gt;You can always use Monte Carlo Integration or the Midpoint method. With Monte Carlo, you simply generate a bunch of points in your parameter space and see if they are in the area or volume or hyper-dimensional space you are trying to integrate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From:&#xA;&lt;a href=&quot;http://farside.ph.utexas.edu/teaching/329/lectures/node109.html&quot; rel=&quot;nofollow&quot;&gt;http://farside.ph.utexas.edu/teaching/329/lectures/node109.html&lt;/a&gt;&#xA;&quot;Let us now consider the so-called Monte-Carlo method for evaluating multi-dimensional integrals. Consider, for example, the evaluation of the area, , enclosed by a curve, . Suppose that the curve lies wholly within some simple domain of area , as illustrated in Fig. 97. Let us generate  points which are randomly distributed throughout . Suppose that  of these points lie within curve . Our estimate for the area enclosed by the curve is simply&quot; the ratio of random points in the space times the size of the space. The link has a nice picture and a description of the inferior midpoint method that I would suggest skipping.&lt;/p&gt;&#xA;" OwnerUserId="1893" LastActivityDate="2011-04-18T11:41:02.280" CommentCount="1" />
  <row Id="9682" PostTypeId="2" ParentId="9674" CreationDate="2011-04-18T11:46:17.800" Score="3" ViewCount="" Body="&lt;p&gt;I've just downloaded the Windows binary version of &lt;code&gt;tm&lt;/code&gt; from CRAN: &lt;a href=&quot;http://cran.r-project.org/web/packages/tm/index.html&quot; rel=&quot;nofollow&quot;&gt;http://cran.r-project.org/web/packages/tm/index.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;code&gt;russian.dat&lt;/code&gt; file that's included there isn't UTF-8, but is encoded using &lt;a href=&quot;http://en.wikipedia.org/wiki/KOI8-R&quot; rel=&quot;nofollow&quot;&gt;KOI-8&lt;/a&gt;. I am unfamiliar with the inner workings of &lt;code&gt;tm&lt;/code&gt;, but you state that you think the file is UTF-8. I am wondering if the problem has to do with the different character encodings?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could you try to figure out the encoding of your file using &lt;a href=&quot;http://2cyr.com/decode/&quot; rel=&quot;nofollow&quot;&gt;http://2cyr.com/decode/&lt;/a&gt; and specifying Autodetect as the source encoding.&lt;/p&gt;&#xA;" OwnerUserId="439" LastEditorUserId="439" LastEditorDisplayName="" LastEditDate="2011-04-18T12:04:01.667" LastActivityDate="2011-04-18T12:04:01.667" CommentCount="1" />
  <row Id="9683" PostTypeId="2" ParentId="9607" CreationDate="2011-04-18T11:47:34.240" Score="0" ViewCount="" Body="&lt;p&gt;Just go ahead and calculate a random uniform, then use the inverse cdf method (http://en.wikipedia.org/wiki/Inverse_transform_sampling) to get a random Beta. Here is the formula for the inverse beta cdf: &lt;a href=&quot;http://www.mathworks.com/help/toolbox/stats/betainv.html&quot; rel=&quot;nofollow&quot;&gt;http://www.mathworks.com/help/toolbox/stats/betainv.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you truly want to just not calculate the beta random number, you can be comfortable that beta converges to normal asymptotically, however the folks at PERT feel very strongly that the normal is not appropriate and list several reasons at &lt;a href=&quot;http://laserlightnetworks.com/Documents/Modeling%20Schedule%20Uncertainty%20without%20Monte%20Carlo%20Methods.pdf&quot; rel=&quot;nofollow&quot;&gt;http://laserlightnetworks.com/Documents/Modeling%20Schedule%20Uncertainty%20without%20Monte%20Carlo%20Methods.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;They suggest the triangle distribution is a better approximation because of steeper decent in cdf and defined support, whereas normal has tails that go to infinity, the triangle and beta distributions have fixed range.&lt;/p&gt;&#xA;" OwnerUserId="1893" LastEditorUserId="1893" LastEditorDisplayName="" LastEditDate="2011-04-18T11:54:48.510" LastActivityDate="2011-04-18T11:54:48.510" CommentCount="1" />
  <row Id="9684" PostTypeId="2" ParentId="7236" CreationDate="2011-04-18T12:02:13.153" Score="0" ViewCount="" Body="&lt;p&gt;Yeah, the sampling package handles this, you can do cluster sampling or stratified or a few others:  &lt;a href=&quot;http://cran.r-project.org/web/packages/sampling/sampling.pdf&quot; rel=&quot;nofollow&quot;&gt;http://cran.r-project.org/web/packages/sampling/sampling.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It can then also handle a lot of the special variance estimation techniques you'll have to do for any metric you calculate from the complex design. However, I prefer Lumley's survey package for that.&lt;/p&gt;&#xA;" OwnerUserId="1893" LastActivityDate="2011-04-18T12:02:13.153" />
  <row Id="9685" PostTypeId="1" CreationDate="2011-04-18T13:50:07.170" Score="6" ViewCount="204" Body="&lt;p&gt;My data set is comprised of either total mortality or survival of an organism at three site types, inshore, midchannel and offshore. The numbers in the table below represent the number of sites. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;              100% Mortality            100% Survival&#xA;Inshore             30                       31 &#xA;Midchannel          10                       20 &#xA;Offshore             1                       10&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I would like to know if the # of sites where 100% mortality occurred is significant based on site type.&#xA;If I run a 2 x 3 chi-square, I get a significant result. Is there a post-hoc pairwise comparison that I can run or should I actually be using a logistical ANOVA or regression with binomial distribution? &#xA;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="4216" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-18T19:26:00.267" LastActivityDate="2011-05-15T01:12:35.080" Title="How to carry out multiple post-hoc chi-square tests on a 2 X 3 table?" Tags="&lt;logistic-regression&gt;&lt;multiple-comparisons&gt;&lt;chi-squared&gt;" AnswerCount="4" CommentCount="0" FavoriteCount="4" />
  <row Id="9686" PostTypeId="2" ParentId="9685" CreationDate="2011-04-18T14:20:42.363" Score="5" ViewCount="" Body="&lt;p&gt;A contingency table should contain all the mutually exclusive categories on both axes. Inshore/Midchannel/Offshore look fine, however unless &quot;less than 100% mortality&quot; means &quot;100% survival&quot; in this biological setting you may need to construct tables that account for all the cases observed or explain why you restrict your analysis to the extreme ends of the sample.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As 100% survival means 0% mortality, you could have a table with columns 100%=mortality / 100%&gt;mortality&gt;0% / mortality=0%. In this case you wouldn't any more compare percentages, but compare ordinal mortality measures across three site type categories. (What about using the original percentage values instead of categories?) A version of Kruskal-Wallis test may be appropriate here that takes ties appropriately into consideration (maybe a permutation test).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are established post hoc tests for the Kruskal-Wallis test: &lt;a href=&quot;http://www.gardenersown.co.uk/Education/Lectures/R/nonparam.htm#kw_posthoc&quot; rel=&quot;nofollow&quot;&gt;1&lt;/a&gt;, &lt;a href=&quot;http://www.brightstat.com/index.php?option=com_content&amp;amp;task=view&amp;amp;id=41&amp;amp;Itemid=1&amp;amp;limit=1&amp;amp;limitstart=1&quot; rel=&quot;nofollow&quot;&gt;2, 3&lt;/a&gt;. (A resampling approach may help tackling with ties.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Logistic regression and binomial regression may be even better as they not only give you p values, but also useful estimates and confidence intervals of the effect sizes. However to set up those models more details would be needed concerning the 100%&gt;mortality&gt;0% sites.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-19T15:42:39.223" LastActivityDate="2011-04-19T15:42:39.223" />
  <row Id="9687" PostTypeId="2" ParentId="9663" CreationDate="2011-04-18T14:37:04.670" Score="2" ViewCount="" Body="&lt;p&gt;You might also want to read up on the &lt;a href=&quot;http://en.wikipedia.org/wiki/Generalized_extreme_value_distribution&quot; rel=&quot;nofollow&quot;&gt;generalized extreme value (GEV) distribution&lt;/a&gt;. It turns out that as $n\rightarrow\infty$, the (shifted and scaled) distribution of the maximal value of the sample converges to one of the three special cases of the GEV distribution.&lt;/p&gt;&#xA;" OwnerUserId="279" LastActivityDate="2011-04-18T14:37:04.670" />
  <row Id="9688" PostTypeId="1" AcceptedAnswerId="9689" CreationDate="2011-04-18T14:56:54.450" Score="1" ViewCount="61" Body="&lt;p&gt;Even simple models in probability theory can be quite confusing (e.g. drawing something with or without returning it, conditional probabilities, one and only one outcome, at least one outcome asoasf...)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other areas of mathematical modelling (e.g. differential equations) you often have powerful and easy to use modelling tools - I have never seen this for stochastic models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you know of any&lt;br&gt;&#xA;(a) generator with which you could in a (e.g.) point-and-click style setup certain probability models and the generator gives you the respective formulas.&lt;br&gt;&#xA;(b) an interpreter which gives you possible interpretations of probability formulas in basic (e.g.) urn or coin-toss-terms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you!&lt;/p&gt;&#xA;" OwnerUserId="230" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-18T18:44:21.447" LastActivityDate="2011-05-18T18:44:21.447" Title="Generator and/or interpreter of probability models" Tags="&lt;probability&gt;&lt;modeling&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="2" />
  <row Id="9689" PostTypeId="2" ParentId="9688" CreationDate="2011-04-18T15:42:57.210" Score="0" ViewCount="" Body="&lt;p&gt;I would recommend downloading R and then following a good R book on Statistics&lt;/p&gt;&#xA;&#xA;&lt;p&gt;R download:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://cran.r-project.org/bin/windows/base/&quot; rel=&quot;nofollow&quot;&gt;http://cran.r-project.org/bin/windows/base/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's a reasonable R/Stats book:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0387954759&quot; rel=&quot;nofollow&quot;&gt;http://www.amazon.com/Introductory-Statistics-R-Peter-Dalgaard/dp/0387954759&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm sure that other people can supply recommendations on different books.   I assume there's also some &quot;community wiki&quot; on this subject that someone will point out.&lt;/p&gt;&#xA;" OwnerUserId="2775" LastActivityDate="2011-04-18T15:42:57.210" CommentCount="3" />
  <row Id="9690" PostTypeId="2" ParentId="8371" CreationDate="2011-04-18T16:21:07.000" Score="0" ViewCount="" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;In most non-compulsory survey contexts, there is a substantial problem with nonresponse. This from 2002: &quot;the recently reported estimate of survey cooperation rates from CMOR, the Council for Market and Opinion Research [USA], averaged only 14.7 percent.&quot; and from Paul Gerhold, &quot;I believe that it is still possible to draw random samples. I just don't believe that it is possible to execute them.&quot; In this context, the fact that the SAMPLE is random isn't very relevant, because the resulting data isn't.  &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;This makes bias adjustment the major issue in valid estimation, and field method design is an important component. The ways in which one might want to do this, and the resulting confidence estimates, are well beyond what can be discussed here.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="3919" LastActivityDate="2011-04-18T16:21:07.000" />
  <row Id="9691" PostTypeId="1" CreationDate="2011-04-18T16:56:18.330" Score="1" ViewCount="56" Body="&lt;p&gt;As a follow up to my &lt;a href=&quot;http://stats.stackexchange.com/questions/9629/can-chi-square-be-used-to-compare-proportions&quot;&gt;previous question&lt;/a&gt;, I learned that the chi square test can be used to compare expected and observed frequency distributions. But...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose I have the following data table in CSV, which surveys people on their fruit preferences in three cities (total people surveyed is different in each city):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;apples,oranges,grapes,bananas&#xA;&#xA;Acme Acres,5,15,5,15&#xA;&#xA;Redmond,2,3,16,0&#xA;&#xA;Cupertino,9,13,8,4&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(1) I can run a chi-square test on each city, to see if my observed frequencies are significantly different from expected (in this case, equally distributed). But is there a test that I can run on &lt;em&gt;ALL THREE&lt;/em&gt; cities at once to see if they are different &lt;strong&gt;from each other&lt;/strong&gt;, &lt;em&gt;AND&lt;/em&gt; also if they are individually different from the expected equal distribution?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(2) If I can run a test that says there is a difference between cities, and/or if each one is different from the expected distribution, is there an equivalent of the Tukey test where I can see which specific fruits are the ones that are different?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="2830" LastActivityDate="2011-04-18T19:00:36.163" Title="Equivalent of Tukey test for chi-square?" Tags="&lt;distributions&gt;&lt;chi-squared&gt;&lt;tukeyhsd&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="1" />
  <row Id="9692" PostTypeId="1" AcceptedAnswerId="9698" CreationDate="2011-04-18T17:01:07.550" Score="3" ViewCount="76" Body="&lt;p&gt;In a (one or multi) way anova model, once a new individual is assigned to a treatment, the predicted value for him is calculated using the coefficients of the ANOVA model (simply assigning the treatment mean value to the individual). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How should I construct a confidence (or prediction) interval for that predicted value? Can the &lt;code&gt;predict&lt;/code&gt; function be used in R for an &lt;code&gt;aov&lt;/code&gt; model? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There must be something that I'm missing here, as I can't find anything like this in my books. Thank you.&lt;/p&gt;&#xA;" OwnerUserId="632" LastEditorUserId="3582" LastEditorDisplayName="" LastEditDate="2011-04-19T03:30:54.897" LastActivityDate="2011-04-19T03:30:54.897" Title="Prediction in simple and multiple ANOVA" Tags="&lt;r&gt;&lt;anova&gt;&lt;confidence-interval&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="1" />
  <row Id="9693" PostTypeId="1" CreationDate="2011-04-18T17:20:16.697" Score="4" ViewCount="195" Body="&lt;p&gt;What are some hot topics that mathematical statistics researchers are studying now?&lt;/p&gt;&#xA;" OwnerUserId="4111" LastActivityDate="2011-05-18T23:47:49.377" CommunityOwnedDate="2011-04-18T18:56:29.803" Title="Hot topics in mathematical statistics" Tags="&lt;mathematical-statistics&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="9695" PostTypeId="1" CreationDate="2011-04-18T18:24:55.707" Score="10" ViewCount="218" Body="&lt;p&gt;In &lt;a href=&quot;http://projecteuclid.org/DPubS?verb=Display&amp;amp;version=1.0&amp;amp;service=UI&amp;amp;handle=euclid.ss/1009213726&amp;amp;page=record&quot; rel=&quot;nofollow&quot;&gt;Statistical Modeling: The Two Cultures&lt;/a&gt; Leo Breiman writes &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Current applied practice is to check&#xA;  the data model fit using&#xA;  goodness-of-fit tests and residual&#xA;  analysis. At one point, some years&#xA;  ago, I set up a simulated regression&#xA;  problem in seven dimensions with a&#xA;  controlled amount of nonlinearity.&#xA;  Standard tests of goodness-of-fit did&#xA;  not reject linearity until the&#xA;  nonlinearity was extreme.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Breiman doesn't give the details of his simulation. He references a paper that he says gives theoretical justification for his observation, but the paper is unpublished.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Has anyone seen a published simulation result or theoretical paper to support Brieman's claim?&lt;/p&gt;&#xA;" OwnerUserId="319" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-18T18:55:49.273" LastActivityDate="2011-05-18T21:47:46.860" Title="Difficulty of testing linearity in regression" Tags="&lt;regression&gt;&lt;goodness-of-fit&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="1" />
  <row Id="9696" PostTypeId="2" ParentId="9691" CreationDate="2011-04-18T19:00:36.163" Score="2" ViewCount="" Body="&lt;p&gt;With this many questions to ask of the data, you are approaching the border between a &quot;test&quot; and a &quot;model&quot; (not that such a sharp distinction exists except in the conceptualization of the problem).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A &lt;a href=&quot;https://secure.wikimedia.org/wikipedia/en/wiki/Poisson_regression&quot; rel=&quot;nofollow&quot;&gt;Poisson regression model&lt;/a&gt;, aka log-linear model, can answer these questions for you.  The book &lt;em&gt;The analysis of crossclassified data&lt;/em&gt; by Stephen Fienberg explains this family of models.&lt;/p&gt;&#xA;" OwnerUserId="2975" LastActivityDate="2011-04-18T19:00:36.163" />
  <row Id="9697" PostTypeId="2" ParentId="9685" CreationDate="2011-04-18T19:18:55.227" Score="1" ViewCount="" Body="&lt;p&gt;Here is the code to do the chi square tests as well as generate a variety of test statistics. However, statistical tests of association of the table margins are useless here; the answer is obvious. No one does a statistical test to see if summer is hotter than winter. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Chompy&amp;lt;-matrix(c(30,10,1,31,20,10), 3, 2)&#xA;Chompy&#xA;chisq.test(Chompy)&#xA;chisq.test(Chompy, simulate.p.value = TRUE, B = 10000)&#xA;chompy2&amp;lt;-data.frame(matrix(c(30,10,1,31,20,10,1,2,1,2,1,2,1,2,3,1,2,3), 6,3))&#xA;chompy2&#xA;chompy2$X2&amp;lt;-factor(chompy2$X2) &#xA;chompy2$X3&amp;lt;-factor(chompy2$X3)&#xA;summary(fit1&amp;lt;-glm(X1~X2+X3, data=chompy2, family=poisson))&#xA;summary(fit2&amp;lt;-glm(X1~X2*X3, data=chompy2, family=poisson)) #oversaturated&#xA;summary(fit3&amp;lt;-glm(X1~1, data=chompy2, family=poisson)) #null&#xA;anova(fit3,fit1)&#xA;library(lmtest)&#xA;waldtest(fit1)&#xA;waldtest(fit2) #oversaturated&#xA;kruskal.test(X1~X2+X3, data=chompy2)&#xA;kruskal.test(X1~X2*X3, data=chompy2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1893" LastActivityDate="2011-04-18T19:18:55.227" CommentCount="3" />
  <row Id="9698" PostTypeId="2" ParentId="9692" CreationDate="2011-04-18T19:19:11.087" Score="4" ViewCount="" Body="&lt;p&gt;You can use &lt;code&gt;lm()&lt;/code&gt; instead of &lt;code&gt;aov()&lt;/code&gt; in this case (the latter is a wrapper of the former).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an illustration:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;n &amp;lt;- 100&#xA;A &amp;lt;- gl(2, n/2, n, labels=paste(&quot;a&quot;, 1:2, sep=&quot;&quot;))&#xA;B &amp;lt;- gl(2, n/4, n, labels=paste(&quot;b&quot;, 1:2, sep=&quot;&quot;))&#xA;# generate fake data for a balanced two-way ANOVA&#xA;df &amp;lt;- data.frame(y=rnorm(n), A, B)&#xA;summary(lm1 &amp;lt;- lm(y~A+B, data=df)) # compare with summary.aov(...)&#xA;predict(lm1, expand.grid(A=levels(A), B=levels(B)), interval=&quot;confidence&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The latter command gives you predictions for each combination of the A and B factor levels (here, I didn't included the interaction), in the following order:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   A  B&#xA;1 a1 b1&#xA;2 a2 b1&#xA;3 a1 b2&#xA;4 a2 b2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Another option is to use the &lt;a href=&quot;http://cran.r-project.org/web/packages/effects/index.html&quot; rel=&quot;nofollow&quot;&gt;effects&lt;/a&gt; package.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-04-18T19:19:11.087" CommentCount="3" />
  <row Id="9699" PostTypeId="1" AcceptedAnswerId="9700" CreationDate="2011-04-18T19:26:12.323" Score="10" ViewCount="188" Body="&lt;p&gt;Is there a possibility to use R in a webinterface without the need to install it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have only one small script which I like to run but I just want to give it a shot without a long installation procedure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you.&lt;/p&gt;&#xA;" OwnerUserId="230" LastActivityDate="2011-04-18T21:02:49.140" Title="Using R online - without installing it" Tags="&lt;r&gt;" AnswerCount="3" CommentCount="6" FavoriteCount="5" />
  <row Id="9700" PostTypeId="2" ParentId="9699" CreationDate="2011-04-18T19:35:38.070" Score="12" ViewCount="" Body="&lt;p&gt;Yes, there are some Rweb interface, like &lt;a href=&quot;http://pbil.univ-lyon1.fr/Rweb/&quot; rel=&quot;nofollow&quot;&gt;this one&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note:&lt;/strong&gt; Installation of the R software is pretty straightforward and quick, on any platform.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-04-18T19:35:38.070" CommentCount="2" />
  <row Id="9701" PostTypeId="2" ParentId="9666" CreationDate="2011-04-18T19:42:11.797" Score="1" ViewCount="" Body="&lt;p&gt;I saw you said you prefer Python, but there are a bunch of R libraries for this, see Highest Density Region function: &lt;a href=&quot;http://cran.r-project.org/web/packages/hdrcde/hdrcde.pdf&quot; rel=&quot;nofollow&quot;&gt;http://cran.r-project.org/web/packages/hdrcde/hdrcde.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second iteration of your looking for the median wouldn't work, as your modes would balance each other. Better off calculating the steepest points of ascent in the cdf.&lt;/p&gt;&#xA;" OwnerUserId="1893" LastActivityDate="2011-04-18T19:42:11.797" />
  <row Id="9703" PostTypeId="2" ParentId="9662" CreationDate="2011-04-18T19:55:39.293" Score="0" ViewCount="" Body="&lt;p&gt;You can use logistic regression.  In SPSS the categorical variable (group A, B, or C) can be entered as a single variable using the contrast command, in which case one of the three will be designated as the reference category, or if you prefer you can create 2 dummy variables to account for the 3 groups.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You would run the regression hierarchically:  first enter those risk factors you want to control, then on a separate step enter Group and watch for the coefficients, odds ratios [in SPSS, &quot;Exp(B)&quot;], and/or p-values you obtain for each category as compared to the reference category.  For example, if C is the reference and if the odds ratio for A is 1.3, then A has 1.3 times the odds that C has of developing the disease.  (Just be careful to distinguish odds from probability.)&lt;/p&gt;&#xA;" OwnerUserId="2669" LastActivityDate="2011-04-18T19:55:39.293" />
  <row Id="9704" PostTypeId="2" ParentId="9666" CreationDate="2011-04-18T20:21:42.377" Score="1" ViewCount="" Body="&lt;p&gt;Had to change my answer because I had trouble with &lt;code&gt;strucchange&lt;/code&gt;, which doesn't seem to like hard changes. Maybe this code will help a bit.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library (robfilter)&#xA;&#xA;# Make phoney data...&#xA;clock &amp;lt;- ts (rnorm (1000, 1, 0.03) * approx (1:10, rgamma (10, 1, 1, 1), seq (0.01, 10, 0.01), method=&quot;constant&quot;)$y)&#xA;spike &amp;lt;- round (runif (12, 1, 1000))&#xA;clock[spike] &amp;lt;- clock[spike] * 10&#xA;&#xA;# Median filter, then difference...&#xA;&#xA;clock.med &amp;lt;- med.filter (clock, 50)$level[,1]&#xA;&#xA;clock.change &amp;lt;- abs (diff (clock.med))&#xA;&#xA;plot (clock.change, type=&quot;l&quot;)&#xA;&#xA;clock.med[clock.change &amp;gt; 0.15]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1764" LastEditorUserId="1764" LastEditorDisplayName="" LastEditDate="2011-04-25T20:13:49.233" LastActivityDate="2011-04-25T20:13:49.233" CommentCount="3" />
  <row Id="9705" PostTypeId="2" ParentId="9626" CreationDate="2011-04-18T20:28:17.657" Score="0" ViewCount="" Body="&lt;p&gt;Balanced designs have really just one goal, orthogonal treatment effects. Orthogonal design lowers the risk of unobservables sneaking into your effect estimates in an uneven way. See: &lt;a href=&quot;http://www1.umn.edu/statsoft/doc/statnotes/stat06.txt&quot; rel=&quot;nofollow&quot;&gt;http://www1.umn.edu/statsoft/doc/statnotes/stat06.txt&lt;/a&gt; for an excellent discussion of this topic.&lt;/p&gt;&#xA;" OwnerUserId="1893" LastActivityDate="2011-04-18T20:28:17.657" />
  <row Id="9706" PostTypeId="2" ParentId="9541" CreationDate="2011-04-18T20:33:07.297" Score="0" ViewCount="" Body="&lt;p&gt;The concept of numbers of parameters and hence df in the lmer model is kind of fuzzy. Don't bother with it, and use AICc; you stand on firmer theoretical ground: &lt;a href=&quot;http://warnercnr.colostate.edu/~anderson/PDF_files/TESTING.pdf&quot; rel=&quot;nofollow&quot;&gt;http://warnercnr.colostate.edu/~anderson/PDF_files/TESTING.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1893" LastActivityDate="2011-04-18T20:33:07.297" />
  <row Id="9707" PostTypeId="2" ParentId="9699" CreationDate="2011-04-18T20:38:17.917" Score="3" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://www.sagemath.org/&quot; rel=&quot;nofollow&quot;&gt;Sage&lt;/a&gt; also has R included with a Python interface. The Sage system is available as installable, but also &lt;a href=&quot;http://www.sagenb.org/&quot; rel=&quot;nofollow&quot;&gt;online&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;r.data(&quot;faithful&quot;)&#xA;r.lm(&quot;eruptions ~ waiting&quot;, data=r.faithful)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Output:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Call:&#xA;lm(formula = sage2, data = sage0)&#xA;&#xA;Coefficients:&#xA;(Intercept)      waiting  &#xA;   -1.87402      0.07563  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="3911" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-18T20:45:20.250" LastActivityDate="2011-04-18T20:45:20.250" CommentCount="1" />
  <row Id="9708" PostTypeId="2" ParentId="7775" CreationDate="2011-04-18T20:45:12.783" Score="0" ViewCount="" Body="&lt;p&gt;Well, r^2 is really just covariance squared over the product of the variances, so you could probably do something like cov(Yfull, Ytrue)/var(Ytrue)var(Yfull) - cov(YReduced, Ytrue)/var(Ytrue)var(YRed) regardless of model type; check to verify that gives you the same answer in the lm case though. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.stator-afm.com/image-files/r-squared.gif&quot; rel=&quot;nofollow&quot;&gt;http://www.stator-afm.com/image-files/r-squared.gif&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1893" LastActivityDate="2011-04-18T20:45:12.783" />
  <row Id="9709" PostTypeId="2" ParentId="9699" CreationDate="2011-04-18T21:02:49.140" Score="3" ViewCount="" Body="&lt;p&gt;Also, if you want to provide a solution to other users, you can set up a webserver with &lt;a href=&quot;http://rapache.net/&quot; rel=&quot;nofollow&quot;&gt;RApache&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="582" LastActivityDate="2011-04-18T21:02:49.140" CommentCount="2" />
  <row Id="9710" PostTypeId="2" ParentId="8147" CreationDate="2011-04-18T21:04:11.073" Score="1" ViewCount="" Body="&lt;p&gt;Sums of bernoullis are distributed exactly binomial, so one often would use logistic regression.&lt;/p&gt;&#xA;" OwnerUserId="1893" LastActivityDate="2011-04-18T21:04:11.073" CommentCount="2" />
  <row Id="9711" PostTypeId="2" ParentId="9695" CreationDate="2011-04-18T21:06:24.723" Score="0" ViewCount="" Body="&lt;p&gt;Not sure it gives a final answer to the question, but I would give a look at &lt;a href=&quot;http://cscs.umich.edu/~crshalizi/weblog/491.html&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt;. Especially point 2. See also the discussion in appendix A2 of the &lt;a href=&quot;http://arxiv.org/abs/0706.1062&quot; rel=&quot;nofollow&quot;&gt;paper&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="4220" LastActivityDate="2011-04-18T21:06:24.723" CommentCount="1" />
  <row Id="9712" PostTypeId="1" AcceptedAnswerId="9713" CreationDate="2011-04-18T21:13:05.013" Score="2" ViewCount="121" Body="&lt;p&gt;I'm in the process of learning R, in the hope of replacing everything I do in SPSS/Sigmplot with R. It's going well so far :) I've got to the point of running a repeated-measures ANOVA, but have come unstuck when trying to plot the results&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've worked out how to plot a set of means using ggplot2, but now I'm unsure of how to plot the standard error as error bars. I've seen a number of guides with different implementations, and none of them seem to be appropriate (or even agree with each other). Many people use standard deviations, which is not what I am after. Others have different methods of computing the standard error, so I'm unsure of the best way to proceed. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I have so far is this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;qplot(CATEGORIES, means, shape=factor(ANOTHER_CATEGORY), facets=MORE_CATEGORIES ~ ., data=alldata)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I was wondering if someone could point me in the right direction in terms of how to get the standard errors from a repeated-measures ANOVA in R, and then how to translate this into error bars in ggplot?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="4204" LastActivityDate="2011-04-18T21:50:00.160" Title="How to add standard error to plots in ggplot2 with R?" Tags="&lt;r&gt;&lt;anova&gt;&lt;ggplot2&gt;" AnswerCount="1" />
  <row Id="9713" PostTypeId="2" ParentId="9712" CreationDate="2011-04-18T21:50:00.160" Score="2" ViewCount="" Body="&lt;p&gt;The reason you're running into multiple methods is because the target variability to visualize in a repeated measures design is not necessarily that straightforward to determine.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you calculate the conventional SE then what you've done is give an estimate of how well you calculated the raw score.  However, generally in a repeated measures design that wasn't the goal of the study.  What you are typically looking to do is to calculate an effect.  The variability of that effect is much less.  I generally recommend plotting your effects only and the variability of your effect estimates (better as confidence intervals than SEs).  Then the error bar will represent something about what you actually attempted to study.  The effect SE will be the sqrt(MSe/n) where n is the number of measurements of the effect (not to be confused with number of S's).&lt;/p&gt;&#xA;" OwnerUserId="601" LastActivityDate="2011-04-18T21:50:00.160" CommentCount="4" />
  <row Id="9714" PostTypeId="2" ParentId="9693" CreationDate="2011-04-18T22:04:17.970" Score="0" ViewCount="" Body="&lt;p&gt;Hopefully, modelling the dynamics of tumor progression qualifies for this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anderson &amp;amp; Quaranta. &lt;a href=&quot;http://www.nature.com/nrc/journal/v8/n3/abs/nrc2329.html&quot; rel=&quot;nofollow&quot;&gt;Integrative mathematical oncology&lt;/a&gt;. Nature Reviews Cancer, 2008.&lt;/p&gt;&#xA;" OwnerUserId="3770" LastActivityDate="2011-04-18T22:04:17.970" CommunityOwnedDate="2011-04-18T22:04:17.970" />
  <row Id="9715" PostTypeId="1" CreationDate="2011-04-18T22:35:27.610" Score="4" ViewCount="166" Body="&lt;p&gt;I ran a multinomial logit model in JMP and got back results which included the AIC as well P&gt;ChiSq values for each parameter estimate. The model has one categorical outcome and 7 categorical explanatory vars.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I then fit what I thought would build the same model in R, using the &lt;code&gt;multinom&lt;/code&gt; function in the &lt;a href=&quot;http://cran.r-project.org/web/packages/nnet/index.html&quot; rel=&quot;nofollow&quot;&gt;nnet&lt;/a&gt; package.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The code was basically:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;fit1 &amp;lt;- multinom(y ~ x1+x2+...xn,data=mydata);&#xA;summary(fit1);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, the two give different results. With JMP the AIC is 2923.21, and with &lt;code&gt;nnet::multinom&lt;/code&gt; the AIC is 3116.588.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my &lt;strong&gt;first question&lt;/strong&gt; is: Is one of the models wrong?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second thing is, JMP gives P&gt;ChiSq values for each parameter estimate, which I need. Running summary on the multinom &lt;code&gt;fit1&lt;/code&gt; does not - it just gives the estimates, AIC and Deviance. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My &lt;strong&gt;second question&lt;/strong&gt; is thus: Is there a way to get the P values for the model and estimates when using &lt;code&gt;nnet::multinom&lt;/code&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know &lt;a href=&quot;http://cran.r-project.org/web/packages/mlogit/index.html&quot; rel=&quot;nofollow&quot;&gt;mlogit&lt;/a&gt; is another R package for this and it looks like its output includes the P values, however I have not been able to run &lt;code&gt;mlogit&lt;/code&gt; using my data. I think I had the data formatted right, but it said I had an invalid formula. I used the same formula that I used for &lt;code&gt;multinom&lt;/code&gt;, but it seems like it requires a different format using a pipe and I don't understand how that works.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="3984" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-19T14:41:35.720" LastActivityDate="2011-05-07T22:51:34.940" Title="How to setup and estimate a multinomial logit model in R?" Tags="&lt;r&gt;&lt;logistic-regression&gt;&lt;multinomial&gt;&lt;logit&gt;&lt;jmp&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="9718" PostTypeId="1" CreationDate="2011-04-19T01:49:14.960" Score="3" ViewCount="103" Body="&lt;p&gt;If the correlation between demographic dissimilarity and satisfaction is $r=.-14$ and the partial correlation, with career development partialled out, between demographic dissimilarity and satisfaction is $r=-.06$ across a very large sample of size $n$, what is the appropriate test to determine if these correlations are significantly different?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Steiger (1980) appears to be the authoritative article on this but the article and most other sources assume one is comparing the correlation between x/y and v/y with dependent groups.  This one has stumped our school's statistic resource center.&lt;/p&gt;&#xA;" OwnerUserId="4224" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-19T14:52:12.420" LastActivityDate="2011-04-20T14:01:56.807" Title="How to test whether correlation measures differ when controlling or not for a third variable? " Tags="&lt;correlation&gt;&lt;statistical-significance&gt;&lt;causal-inference&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="9719" PostTypeId="2" ParentId="4898" CreationDate="2011-04-19T02:15:25.713" Score="1" ViewCount="" Body="&lt;p&gt;And I think one could put it clear by noting (between), so people would know it's an experimental-control effect size. Because there is within-group effect size too. FYI. Good luck!&lt;/p&gt;&#xA;" OwnerUserId="4225" LastActivityDate="2011-04-19T02:15:25.713" />
  <row Id="9720" PostTypeId="2" ParentId="9627" CreationDate="2011-04-19T02:50:30.023" Score="2" ViewCount="" Body="&lt;p&gt;You cannot &quot;systemically avoid this problem in the future&quot;, because it should not be called a &quot;problem&quot;. If the reality of the material world features strong covariates, then we should accept it as fact and adjust our theories and models in consequence. I like the question very much, and hope that what follows will not sound too disappointing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are some adjustments that might work for you. You will need to review a regression handbook before proceeding.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Diagnose the issue, using correlation or post-estimation techniques like the Variance Inflation Factor (VIF). Use the tools mentioned by Peter Flom if you are using SAS or R. In Stata, use &lt;code&gt;pwcorr&lt;/code&gt; to build a correlation matrix, &lt;code&gt;gr matrix&lt;/code&gt; to build a scatterplot matrix, and &lt;code&gt;vif&lt;/code&gt; to detect problematic tolerance levels of 1/VIF &amp;lt; 0.1.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Measure the interaction effect by adding, for example, &lt;code&gt;var3*var4&lt;/code&gt; to the model. The coefficient will help you realise how much is at play between &lt;code&gt;var3&lt;/code&gt; and &lt;code&gt;var4&lt;/code&gt;. This will only bring you so far as partially measuring the interaction, but it will not rescue your model from its limitations.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Most importantly, if you detect strong multicollinearity or other issues like heteroscedasticity, you should ditch your model and start again. Model misspecification is the plague of regression analysis (and frequentist methods in general). Paul Schrodt has several excellent papers on the issue, including his recent &quot;&lt;a href=&quot;http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1661045&quot; rel=&quot;nofollow&quot;&gt;Seven Deadly Sins&lt;/a&gt;&quot; that I like a lot.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;This answers your point on multicollinearity, and a lot of this can be learnt from the &lt;a href=&quot;http://www.ats.ucla.edu/stat/stata/webbooks/reg/&quot; rel=&quot;nofollow&quot;&gt;regression handbook&lt;/a&gt; over at UCLA Stat Computing. It does not answer your question on causality. Briefly put, regression is never causal. Neither is any statistical model: causal and statistical information are separate species. Read selectively from Judea Pearl (&lt;a href=&quot;http://ftp.cs.ucla.edu/pub/stat_ser/r373-reprint.pdf&quot; rel=&quot;nofollow&quot;&gt;example&lt;/a&gt;) to learn more on the matter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All in all, this answer does not cancel out the value of regression analysis, or even of frequentist statistics (I happen to teach both). It does, however, reduce their scope of appropriateness, and also underlines the crucial role of your initial explanatory theory, which really determines the possibility of your model possessing causal properties.&lt;/p&gt;&#xA;" OwnerUserId="3582" LastActivityDate="2011-04-19T02:50:30.023" CommentCount="2" />
  <row Id="9721" PostTypeId="2" ParentId="9685" CreationDate="2011-04-19T02:54:55.040" Score="0" ViewCount="" Body="&lt;p&gt;I am going to assume that &quot;100% survival&quot; means that your sites only contained a single organism.  so 30 means 30 organisms died, and 31 means 31 organisms didn't.&#xA;Based on this the chi-square should be fine, but it will only tell which hypothesis are not supported by the data - it won't tell you if two reasonable hypothesis are better or not.  I present a probability analysis which does extract this information - it agrees with the chi-square test, but it gives you more information than the chi-square test, and a better way to present the results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The model is a bernouli model for the indicator of &quot;death&quot;, $Y_{ij}\sim Bin(1,\theta_{ij})$ ($i$ denotes the cell of the $2\times 3$ table, and $j$ denotes the individual unit within the cell).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are two global assumption underlying the chi-square test:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;within a given cell of the table, the $\theta_{ij}$ are all equal, that is $\theta_{ij}=\theta_{ik}=\theta_{i}$&lt;/li&gt;&#xA;&lt;li&gt;the $Y_{ij}$ are statistically independent, given $\theta_{i}$.  This means that the probability parameters tell you everything about $Y_{ij}$ - all other information is irrelevant if you know $\theta_{i}$&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Denote $X_{i}$ as the sum of $Y_{ij}$, (so $X_{1}=30,X_{2}=10,X_{3}=1$) and let $N_{i}$ be the group size (so $N_{1}=61,N_{2}=30,N_{3}=11$).  Now we have a hypothesis to test:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$H_{A}:\theta_{1}=\theta_{2},\theta_{1}=\theta_{3},\theta_{2}=\theta_{3}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But what are the alternatives?  I would say the other possible combinations of equal or not equal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$H_{B1}:\theta_{1}\neq\theta_{2},\theta_{1}\neq\theta_{3},\theta_{2}=\theta_{3}$$&#xA;$$H_{B2}:\theta_{1}\neq\theta_{2},\theta_{1}=\theta_{3},\theta_{2}\neq\theta_{3}$$&#xA;$$H_{B3}:\theta_{1}=\theta_{2},\theta_{1}\neq\theta_{3},\theta_{2}\neq\theta_{3}$$&#xA;$$H_{C}:\theta_{1}\neq\theta_{2},\theta_{1}\neq\theta_{3},\theta_{2}\neq\theta_{3}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of these hypothesis has to be true, given the &quot;global&quot; assumptions above.  But note that none of these specify specific values for the rates - so they must be integrated out.  Now given that $H_{A}$ is true, we only have one parameter (because all are equal), and the uniform prior is a conservative choice, denote this and the global assumptions by $I_{0}$.  so we have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(X_{1},X_{2},X_{3}|N_{1},N_{2},N_{3},H_{A},I_{0})=\int_{0}^{1}P(X_{1},X_{2},X_{3},\theta|N_{1},N_{2},N_{3},H_{A},I_{0})d\theta$$&#xA;$$={N_{1} \choose X_{1}}{N_{2} \choose X_{2}}{N_{3} \choose X_{3}}\int_{0}^{1}\theta^{X_{1}+X_{2}+X_{3}}(1-\theta)^{N_{1}+N_{2}+N_{3}-X_{1}-X_{2}-X_{3}}d\theta$$&#xA;$$=\frac{{N_{1} \choose X_{1}}{N_{2} \choose X_{2}}{N_{3} \choose X_{3}}}{(N_{1}+N_{2}+N_{3}+1){N_{1}+N_{2}+N_{3} \choose X_{1}+X_{2}+X_{3}}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which is a hypergeometric distribution divided by a constant.  Similarly for $H_{B1}$ we will have:&#xA;$$P(X_{1},X_{2},X_{3}|N_{1},N_{2},N_{3},H_{B1},I_{0})=\int_{0}^{1}P(X_{1},X_{2},X_{3},\theta_{1}\theta_{2}|N_{1},N_{2},N_{3},H_{B1},I_{0})d\theta_{1}d\theta_{2}$$&#xA;$$=\frac{{N_{2} \choose X_{2}}{N_{3} \choose X_{3}}}{(N_{1}+1)(N_{2}+N_{3}+1){N_{2}+N_{3} \choose X_{2}+X_{3}}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can see the pattern for the others.  We can calculate the odds for say $H_{A}\;vs\;H_{B1}$ by simply dividing the above two expressions.  The answer is about $4$, which means the data support $H_{A}$ over $H_{B1}$ by about a factor of $4$ - fairly weak evidence in favour of equal rates.  The other probabilities are given below.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\begin{array}{c|c} &#xD;&#xA;Hypothesis &amp;amp; probability \\ \hline&#xD;&#xA;(H_{A}|D) &amp;amp; 0.018982265 \\&#xD;&#xA;(H_{B1}|D) &amp;amp; 0.004790669 \\&#xD;&#xA;(H_{B2}|D) &amp;amp; 0.051620022 \\&#xD;&#xA;(H_{B3}|D) &amp;amp; 0.484155874 \\&#xD;&#xA;(H_{C}|D) &amp;amp; 0.440451171 \\&#xD;&#xA; \end{array} &#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is showing strong evidence against equal rates, but not in strong evidence favour of a defintie alternative.  It seems like there is strong evidence that the &quot;offshore&quot; rate is different to the other two rates, but inconclusive evidence as to whether &quot;inshore&quot; and &quot;mid-channel&quot; rates differ.  This is what the chi-square test won't tell you - it only tells you that hypothesis $A$ is &quot;crap&quot; but not what alternative to put in its place&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-04-19T02:54:55.040" />
  <row Id="9722" PostTypeId="2" ParentId="9664" CreationDate="2011-04-19T03:32:57.203" Score="10" ViewCount="" Body="&lt;p&gt;If the quantity of interest, usually a functional of a distribution, is reasonably smooth and your data are i.i.d., you're usually in pretty safe territory. Of course, there are other circumstances when the bootstrap will work as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What it means for the bootstrap to &quot;fail&quot;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Broadly speaking, the purpose of the bootstrap is to construct an approximate sampling distribution for the statistic of interest. It's not about actual estimation of the parameter. So, if the statistic of interest (under some rescaling and centering) is $\newcommand{\Xhat}{\hat{X}_n}\Xhat$ and $\Xhat \to X_\infty$ in distribution, we'd like our bootstrap distribution to converge to the distribution of $X_\infty$. If we don't have this, then we can't trust the inferences made.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;strong&gt;canonical&lt;/strong&gt; example of when the bootstrap can fail, even in an i.i.d. framework is when trying to approximate the sampling distribution of an extreme order statistic. Below is a brief discussion.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Maximum order statistic of a random sample from a $\;\mathcal{U}[0,\theta]$ distribution&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let $X_1, X_2, \ldots$ be a sequence of i.i.d. uniform random variables on $[0,\theta]$. Let $\newcommand{\Xmax}{X_{(n)}} \Xmax = \max_{1\leq k \leq n} X_k$. The distribution of $\Xmax$ is &#xA;$$&#xD;&#xA;\renewcommand{\Pr}{\mathbb{P}}\Pr(\Xmax \leq x) = (x/\theta)^n \&amp;gt;.&#xD;&#xA;$$ &#xA;(Note that by a very simple argument, this actually also shows that $\Xmax \to \theta$ in probability, and even, &lt;em&gt;almost surely&lt;/em&gt;, if the random variables are all defined on the same space.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An elementary calculation yields&#xA;$$&#xD;&#xA;\Pr( n(\theta - \Xmax) \leq x ) = 1 - \Big(1 - \frac{x}{\theta n}\Big)^n \to 1 - e^{-x/\theta} \&amp;gt;,&#xD;&#xA;$$&#xA;or, in other words, $n(\theta - \Xmax)$ converges in distribution to an exponential random variable with mean $\theta$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, we form a (naive) &lt;strong&gt;bootstrap&lt;/strong&gt; estimate of the distribution of $n(\theta - \Xmax)$ by resampling $X_1, \ldots, X_n$ with replacement to get $X_1^\star,\ldots,X_n^\star$ and using the distribution of $n(\Xmax - \Xmax^\star)$ conditional on $X_1,\ldots,X_n$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But, observe that $\Xmax^\star = \Xmax$ with probability $1 - (1-1/n)^n \to 1 - e^{-1}$, and so the bootstrap distribution has a point mass at zero &lt;strong&gt;even asymptotically&lt;/strong&gt; despite the fact that the actual limiting distribution is continuous.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More explicitly, though the true limiting distribution is exponential with mean $\theta$, the limiting bootstrap distribution places a &lt;strong&gt;point mass&lt;/strong&gt; at zero of size $1−e^{-1} \approx 0.632$ &lt;em&gt;independent of the actual value of $\theta$&lt;/em&gt;. By taking $\theta$ sufficiently large, we can make the probability of the true limiting distribution arbitrary small for any fixed interval $[0,\varepsilon)$, yet the bootstrap will (&lt;em&gt;still&lt;/em&gt;!) report that there is at least probability 0.632 in this interval! From this it should be clear that the bootstrap can behave &lt;strong&gt;arbitrarily badly&lt;/strong&gt; in this setting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In summary, the bootstrap fails (miserably) in this case. Things tend to go wrong when dealing with parameters at the edge of the parameter space.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;An example from a sample of normal random variables&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are other similar examples of the failure of the bootstrap in surprisingly simple circumstances.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider a sample $X_1, X_2, \ldots$ from $\mathcal{N}(\mu,1)$ where the parameter space for $\mu$ is restricted to $[0,\infty)$. The MLE in this case is $\newcommand{\Xbar}{\bar{X}}\Xhat = \max(\bar{X},0)$. Again, we use the bootstrap estimate $\Xhat^\star = \max(\Xbar^\star, 0)$. Again, it can be shown that the distribution of $\sqrt{n}(\Xhat^\star - \Xhat)$ (conditional on the observed sample) does not converge to the same limiting distribution as $\sqrt{n}(\Xhat - \mu)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Exchangeable arrays&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Perhaps one of the most dramatic examples is for an exchangeable array. Let $\newcommand{\bm}[1]{\mathbf{#1}}\bm{Y} = (Y_{ij})$ be an array of random variables such that, for &lt;em&gt;every&lt;/em&gt; pair of permutation matrices $\bm{P}$ and $\bm{Q}$, the arrays $\bm{Y}$ and $\bm{P} \bm{Y} \bm{Q}$ have the same joint distribution. That is, permuting rows and columns of $\bm{Y}$ keeps the distribution invariant. (You can think of a two-way random effects model with one observation per cell as an example, though the model is much more general.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose we wish to estimate a confidence interval for the mean $\mu = \mathbb{E}(Y_{ij}) = \mathbb{E}(Y_{11})$ (due to the exchangeability assumption described above the means of all the cells must be the same). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;McCullagh (2000) considered two different natural (i.e., naive) ways of bootstrapping such an array. Neither of them get the asymptotic variance for the sample mean correct. He also considers some examples of a one-way exchangeable array and linear regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;References&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unfortunately, the subject matter is nontrivial, so none of these are particularly easy reads.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;P. Bickel and D. Freedman, &lt;a href=&quot;http://projecteuclid.org/euclid.aos/1176345637&quot; rel=&quot;nofollow&quot;&gt;Some asymptotic theory for the bootstrap&lt;/a&gt;. &lt;em&gt;Ann. Stat.&lt;/em&gt;, vol. 9, no. 6 (1981), 1196&amp;ndash;1217.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;D. W. K. Andrews, &lt;a href=&quot;http://cowles.econ.yale.edu/P/cp/p09b/p0994.pdf&quot; rel=&quot;nofollow&quot;&gt;Inconsistency of the bootstrap when a parameter is on the boundary of the parameter space&lt;/a&gt;, &lt;em&gt;Econometrica&lt;/em&gt;, vol. 68, no. 2 (2000), 399&amp;ndash;405.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;P. McCullagh, &lt;a href=&quot;http://projecteuclid.org/euclid.bj/1081788029&quot; rel=&quot;nofollow&quot;&gt;Resampling and exchangeable arrays&lt;/a&gt;, &lt;em&gt;Bernoulli&lt;/em&gt;, vol. 6, no. 2 (2000), 285&amp;ndash;301.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;E. L. Lehmann and J. P. Romano, &lt;em&gt;Testing Statistical Hypotheses&lt;/em&gt;, 3rd. ed., Springer (2005). [Chapter 15: General Large Sample Methods]&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="2970" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-04-20T02:08:28.043" LastActivityDate="2011-04-20T02:08:28.043" CommentCount="14" />
  <row Id="9723" PostTypeId="2" ParentId="4111" CreationDate="2011-04-19T05:09:08.220" Score="0" ViewCount="" Body="&lt;p&gt;There are companies that specialize in counting people. For instance, &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[www.lynce.es][1]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(I am not affiliated nor have any interest whatsoever in such company).&#xA;They hung cameras over the groups they want to count, shoot pictures &#xA;and actually count heads. They only make small adjustaments when it comes to estimate people under trees or other objects which prevent direct vision. &lt;/p&gt;&#xA;" OwnerUserId="892" LastActivityDate="2011-04-19T05:09:08.220" />
  <row Id="9724" PostTypeId="1" CreationDate="2011-04-19T05:19:00.737" Score="2" ViewCount="56" Body="&lt;p&gt;I'm hoping to hear from someone who has worked on mouse models or similar biological analyses where there is a tendency to run 'replicates' of an experiment. I know multiple testing is a sizeable kettle of fish which is definitely relevant to this discussion. I have some applications for projects where they talk about running 3 replicates of an experiment, where each experiment has n = 3 to 7. However there seems to be no mention of what they will do with the multiple sets of results, as in how they will handle success, failure, success vs failure, failure, success etc. It seems like this 'replicates' approach is quite common practise in this field.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are your thoughts / experiences with this situation?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know there are different types of replication, technical vs biological, however I've found little useful reading on this issue.&lt;/p&gt;&#xA;" OwnerUserId="4226" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-19T14:12:19.783" LastActivityDate="2011-05-20T03:48:09.377" Title="Mouse models - 'replicates' and analysis" Tags="&lt;multiple-comparisons&gt;&lt;experiment-design&gt;&lt;biostatistics&gt;" AnswerCount="1" />
  <row Id="9725" PostTypeId="1" CreationDate="2011-04-19T05:39:14.750" Score="0" ViewCount="36" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;Possible Duplicate:&lt;/strong&gt;&lt;br&gt;&#xA;  &lt;a href=&quot;http://stats.stackexchange.com/questions/5934/odds-of-rolling-doubles-or-triples-given-n-6-sided-dice&quot;&gt;Odds of rolling doubles or triples given N 6-sided dice&lt;/a&gt;  &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&#xA;&#xA;&lt;p&gt;Three fair dice are rolled simultaneously. In 10 independent throws,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;let $X$: number of times all the dice show same face&lt;/p&gt;&#xA;&#xA;&lt;p&gt;let $Y$: number of times exactly $2$ dice show same face&lt;/p&gt;&#xA;&#xA;&lt;p&gt;let $p_1,p_2,p_3$ be corresponding probabilities of $X, Y, Z=n-X-Y$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;pdf of trinomial distribution,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p(X=3, Y=3)= 10C3\times (10-3)C3\times p_1^3\times p_2^3\times p_3^{10-3-3}$$&#xA;i understand $p_1= \frac{6}{6^3}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;but what are the values of $p_2,p_3 ?$&lt;/p&gt;&#xA;" OwnerUserId="3846" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-04-19T06:11:00.163" LastActivityDate="2011-04-19T06:11:00.163" ClosedDate="2011-04-19T06:43:09.043" Title="What will be the probabilities of this trinomial distribution?" Tags="&lt;probability&gt;&lt;homework&gt;" CommentCount="1" />
  <row Id="9727" PostTypeId="2" ParentId="9507" CreationDate="2011-04-19T07:10:28.757" Score="0" ViewCount="" Body="&lt;p&gt;To asnwer the first part of my question, does a flat initial guess lead to flat data, the answer would be &quot;yes&quot;. Not only does having a flat guess flatten the result, it also makes it unchanging (a fact I missed thanks to a small error in my algorithm).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's a proof:&lt;br&gt;&#xA;Assuming that $\langle R_{r\alpha} \rangle^{(t)} = x$ for all $r$ and $\alpha$.&lt;br&gt;&#xA;$\hat{\pi}_{\alpha}^{(t)} = \frac1{L}\sum_{r=1}^{L}\langle R_{r\alpha} \rangle^{(t)} = \frac{Lx}{L} = x$ for each $\alpha$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\hat{p}_{i|\alpha}^{(t)} = \frac1{L\hat{\pi}_\alpha^{(t)}} \sum_{r:i(r)=i} \langle R_{r\alpha} \rangle^{(t)} = \frac{n_ix}{Lx} = \frac{n_i}{L}$ for each $\alpha$, $n_i$ is the number of co-observations where $r:i(r)=i$ is true.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\hat{q}_{j|\alpha}^{(t)} = \frac1{L\hat{\pi}_\alpha^{(t)}} \sum_{r:j(r)=j} \langle R_{r\alpha} \rangle^{(t)} = \frac{n_jx}{Lx} = \frac{n_j}{L}$ for each $\alpha$, $n_j$ is the number of co-observations where $r:j(r)=j$ is true.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From these, we can calculate&lt;br&gt;&#xA;$\langle R_{r\alpha} \rangle^{(t+1)} = \frac{\pi_{\alpha}^{(t)}\hat{p}_{i(r)|\alpha}^{(t)}\hat{q}_{j(r)|\alpha}^{(t)}}{\sum_{v=1}^K\pi_{v}^{(t)}\hat{p}_{i(r)|v}^{(t)}\hat{q}_{j(r)|v}^{(t)}} = \frac{x(n_{i(r)}n_{j(r)})/L^2}{Kx(n_{i(r)}n_{j(r)})/L^2} = \frac1{K}$ for all $r$ and $\alpha$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And thus, $R_{r\alpha}$ is again a constat for the next round of iteration.&lt;/p&gt;&#xA;" OwnerUserId="4141" LastEditorUserId="4141" LastEditorDisplayName="" LastEditDate="2011-04-20T06:52:07.723" LastActivityDate="2011-04-20T06:52:07.723" />
  <row Id="9728" PostTypeId="1" CreationDate="2011-04-19T07:19:11.553" Score="0" ViewCount="193" Body="&lt;p&gt;Under what circumstances would using regression with two given variables not increase accuracy of prediction?&lt;/p&gt;&#xA;" OwnerUserId="4227" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-29T05:14:44.927" LastActivityDate="2011-04-29T05:14:44.927" Title="When is there no point in using regression?" Tags="&lt;regression&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="9729" PostTypeId="1" CreationDate="2011-04-19T08:27:19.990" Score="1" ViewCount="44" Body="&lt;p&gt;I was given the following question:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A survey found that 89% of a random sample of 1024 American adults approved of cloning endangered animals. Find the margin of error for this survey if we want 90% confidence in our estimate of the percent of American adults who approve of cloning endangered animals.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know that for 90% Confidence, $\text{ME}\sim 0.82/\sqrt{n}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I attempted using this formula with n equal to both 1024 and (.89)1024. I got 0.025625 and 0.02716, respectively. The answer given for the problem is 1.61%.  I do not understand where I went wrong. Perhaps I am using the Margin of Error formula incorrectly?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks. :)&lt;/p&gt;&#xA;" OwnerUserId="4228" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-19T12:50:04.660" LastActivityDate="2011-04-19T12:50:04.660" Title="How to compute margin of error with a given confidence interval?" Tags="&lt;homework&gt;&lt;sampling&gt;&lt;survey&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="9730" PostTypeId="1" AcceptedAnswerId="9732" CreationDate="2011-04-19T08:50:53.817" Score="2" ViewCount="114" Body="&lt;p&gt;Assume we have a dataframe &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x   y&#xA;1   1&#xA;2   4 &#xA;4   5 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;how can you add a new variable to the dataframe such that if x is less than or equal to 1 it returns &quot;good&quot; if x is between 3 and 5 it returns &quot;bad&quot; else returns &quot;fair&quot;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x   y  w&#xA;1   1  &quot;good&quot;&#xA;2   2   &quot;fair&quot;&#xA;5   5   &quot;bad&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Applied the method shown by ocram., however this one here does not work. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;d1 &amp;lt;- c(&quot;e&quot;, &quot;c&quot;, &quot;a&quot;)&#xA;d2 &amp;lt;- c(&quot;e&quot;, &quot;a&quot;, &quot;b&quot;)&#xA;&#xA;w &amp;lt;- ifelse(d1 == &quot;e&quot; &amp;amp; (d2==&quot;e&quot;), 1, ifelse((d1 == &quot;a&quot;) &amp;amp; (d2 ==&quot;b&quot;), 2, ifelse(d1 == &quot;e&quot;),3,99))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Any ideas?&#xA;Thanks&lt;/p&gt;&#xA;" OwnerUserId="201" LastEditorUserId="201" LastEditorDisplayName="" LastEditDate="2011-04-20T07:14:04.460" LastActivityDate="2011-04-20T08:17:44.813" Title="How to create a new variable in a data.frame based on a condition?" Tags="&lt;r&gt;&lt;dataframe&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="9731" PostTypeId="1" CreationDate="2011-04-19T08:55:24.803" Score="3" ViewCount="81" Body="&lt;p&gt;I am doing an exploratory study to investigate the relationship between a drug (actually measured in two ways - by direct and indirect methods) and 15 various parameters. There are three groups of individuals, on which I compute correlation statistics separately ($n=11$, $n=11$ and $n=24$ - the first two are paired). Obviously, the number of statistical tests is large, and I would like to control for multiple testing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using Spearman correlation as data are limited and do not (always) follow a normal distribution. I am working in R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How should I control for multiplicity of testing? This is an hypothesis generating study, and I would rather prefer to use less conservative methods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have been looking into Benjamini–Hochberg's procedure (known under R as &lt;code&gt;&quot;BH&quot;&lt;/code&gt; or &lt;code&gt;&quot;fdr&quot;&lt;/code&gt;) which allows to control the false discovery rate. However, I am not sure if I would violate dependence assumptions. And I am not sure if this could be used in correlation statistics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Perhaps I should not adjust for multiple testing at all... In my mind, if both direct and indirect methods give the same association, false positives are highly unlikely...&lt;/p&gt;&#xA;" OwnerUserId="4229" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-20T12:17:04.417" LastActivityDate="2011-04-20T12:17:04.417" Title="Adjust a large set of Spearman correlation analyses for multiple testing" Tags="&lt;correlation&gt;&lt;multiple-comparisons&gt;&lt;spearman&gt;" CommentCount="3" />
  <row Id="9732" PostTypeId="2" ParentId="9730" CreationDate="2011-04-19T08:55:53.547" Score="3" ViewCount="" Body="&lt;p&gt;One obvious and straightforward possibility is to use &quot;if-else conditions&quot;. In that example&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x &amp;lt;- c(1, 2, 4)&#xA;y &amp;lt;- c(1, 4, 5)&#xA;w &amp;lt;- ifelse(x &amp;lt;= 1, &quot;good&quot;, ifelse((x &amp;gt;= 3) &amp;amp; (x &amp;lt;= 5), &quot;bad&quot;, &quot;fair&quot;))&#xA;data.frame(x, y, w)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;** For the additional question in the edit**&#xA;Is that what you expect ?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; d1 &amp;lt;- c(&quot;e&quot;, &quot;c&quot;, &quot;a&quot;)&#xA;&amp;gt; d2 &amp;lt;- c(&quot;e&quot;, &quot;a&quot;, &quot;b&quot;)&#xA;&amp;gt; &#xA;&amp;gt; w &amp;lt;- ifelse((d1 == &quot;e&quot;) &amp;amp; (d2 == &quot;e&quot;), 1, &#xA;+    ifelse((d1==&quot;a&quot;) &amp;amp; (d2 == &quot;b&quot;), 2,&#xA;+    ifelse((d1 == &quot;e&quot;), 3, 99)))&#xA;&amp;gt;     &#xA;&amp;gt; data.frame(d1, d2, w)&#xA;  d1 d2  w&#xA;1  e  e  1&#xA;2  c  a 99&#xA;3  a  b  2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you do not feel comfortable with the &lt;code&gt;ifelse&lt;/code&gt; function, you can also work with the &lt;code&gt;if&lt;/code&gt; and &lt;code&gt;else&lt;/code&gt; statements for such applications.&lt;/p&gt;&#xA;" OwnerUserId="3019" LastEditorUserId="3019" LastEditorDisplayName="" LastEditDate="2011-04-20T08:17:44.813" LastActivityDate="2011-04-20T08:17:44.813" CommentCount="2" />
  <row Id="9733" PostTypeId="2" ParentId="9728" CreationDate="2011-04-19T09:43:04.257" Score="5" ViewCount="" Body="&lt;p&gt;When the model assumptions are valid, but the data are not correlated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When the model assumptions are invalid (e.g. the noise process is heteroskedastic) in which case a regression model may fit the data very well, but provide very poor out-of-sample predictions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See also the excellent point about extrapolation made by probabilityislogic.&lt;/p&gt;&#xA;" OwnerUserId="887" LastEditorUserId="887" LastEditorDisplayName="" LastEditDate="2011-04-19T14:18:17.637" LastActivityDate="2011-04-19T14:18:17.637" />
  <row Id="9734" PostTypeId="1" CreationDate="2011-04-19T09:47:27.687" Score="2" ViewCount="57" Body="&lt;p&gt;Is it possible to use a continuous predictor in Bugs? The simplest way of doing this would be turning the size variable in alligators example from discrete to continuous. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Both Winbugs and JAGS examples use combination of values of covariates as indices as in &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X[i,j,] ~ dmulti( p[i,j,] , n[i,j]  );&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where &lt;code&gt;i&lt;/code&gt; is the lake index (4 possible values), and &lt;code&gt;j&lt;/code&gt; is the size index (2 possible values). With this approach, a continuous size variable would mean an infinite amount of indices. There must be something I'm missing here.&lt;/p&gt;&#xA;" OwnerUserId="3280" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-19T12:53:18.977" LastActivityDate="2011-04-19T12:53:18.977" Title="How to model logistic regression with continuous predictor in Bugs?" Tags="&lt;bayesian&gt;&lt;logistic-regression&gt;&lt;bugs&gt;" />
  <row Id="9735" PostTypeId="1" AcceptedAnswerId="9866" CreationDate="2011-04-19T09:53:48.520" Score="4" ViewCount="162" Body="&lt;p&gt;... (optional) within the context of Google Web Optimizer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose you have two groups and a binary response variable. Now you get the following outcome:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;em&gt;Original&lt;/em&gt;: 401 trials, 125 successful trials &lt;/li&gt;&#xA;&lt;li&gt;&lt;em&gt;Combination16&lt;/em&gt;: 441 trials, 141 successful trials&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The difference is not statistically significant, however one can calculate a probability that Combination16 will beat Original.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To calculate &quot;Chance to beat Original&quot; I have used an bayesian approach, i.e. performing a two-dimensional monte carlo integration over the bayesian-style confidence intervals (beta-distribution, (0,0) prior). Here is the code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;trials &amp;lt;- 10000&#xA;resDat&amp;lt;-data.frame(&quot;orig&quot;=rbeta(trials,125+1,401-125+1),&#xA;                    &quot;opt&quot;=rbeta(trials,144+1,441-144+1))&#xA;length(which(resDat$opt&amp;gt;resDat$orig))/trials&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This results in 0.6764. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which technique would a frequentist use to calculate &quot;Chance to beat ...&quot; ? Maybe the power function of Fisher's exact test ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Optional: Context of Google Web Optimizer&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Google Web Optimizer is a tool for controlling multivariate Testing or A/B-Testing. This only as an introduction since this should not matter for the question itself.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The example presented above was taken from the explanation page of Google Web Optimizer (GWO), which you can find &lt;a href=&quot;https://www.google.com/analytics/siteopt/siteopt/help/techoverview.html#stats&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; (please scroll down to the section &quot;&lt;em&gt;Estimated Conversion Rate Ranges&lt;/em&gt;&quot;), specifically from figure 2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here GWO delivers 67.8% for &quot;Chance to beat Original&quot;, which slightly differs from my result. I guess Google uses a more frequentist-like approach and I wondered: What could it be ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT:&lt;/strong&gt; Since this question was close to disappear (I guess because of its too specific nature), I have rephrased it to be of general interest.&lt;/p&gt;&#xA;" OwnerUserId="264" LastEditorUserId="264" LastEditorDisplayName="" LastEditDate="2011-05-06T11:17:56.070" LastActivityDate="2011-05-06T11:17:56.070" Title="How does a frequentist calculate the chance that group A beats group B regarding binary response" Tags="&lt;bayesian&gt;&lt;ab-test&gt;" AnswerCount="1" CommentCount="6" FavoriteCount="1" />
  <row Id="9736" PostTypeId="1" CreationDate="2011-04-19T11:00:41.787" Score="4" ViewCount="49" Body="&lt;p&gt;Hi!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a time series (X) representing a natural phenomenon (wind speed, measured every 15 minutes) and I have to create similar time series (up to 20, Xdi, i=1,...,20) with the same structure (same average, same standard deviation, same percentiles distribution...) but with a predetermined correlation (about 0.7) between each other.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any defined method for this operation? Can you provide a link to a book, a paper, a page, or just the name of the method? Thank you very much,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Andrew&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;A couple of clarifications:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;for &quot;pre-defined correlation&quot; I mean that if I have a seed time series X and I want to create three derived time series Xd1, Xd2, Xd3, then the correlation among two of the time series must be equal (almost) to a value chosen (e.g.: 0.7).&#xA;For example Correlation(Xdi, Xdj) = 0.7&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The comment of charles.y.zheng answers the question (thanks!), but the resulting time series (AZ in the example) does not have necessarily the same autocorrelation of the original seed. (in the original time series 0 tends to be followed by 0 and 1 tends to be followed by 1, the values are clustered). Sorry to haven't specified also this requirement, but I noticed after trying the solution proposed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I guess that I can fix the problem by manipulating the values in Z. This adds a second section of the question:&#xA;Is there a defined method to create a time series with a pre determined autocorrelation?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;p.s.: sorry if the language is not correct, I'm not a statistician but a programmer, I try to do my best but if there is something not clear, just ask for details and clarifications.&lt;/p&gt;&#xA;" OwnerUserId="4230" LastEditorUserId="4230" LastEditorDisplayName="" LastEditDate="2011-04-21T13:57:37.540" LastActivityDate="2011-04-21T13:57:37.540" Title="How to create n time series characterised by a defined average and correlation?" Tags="&lt;time-series&gt;&lt;correlation&gt;" CommentCount="3" />
  <row Id="9737" PostTypeId="2" ParentId="9729" CreationDate="2011-04-19T11:39:35.177" Score="2" ViewCount="" Body="&lt;p&gt;Because you are dealing with proportions, the variance is given by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{p(1-p)}{n}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And so the 90% CI ME is equal to $1.645\times \sqrt{\frac{p(1-p)}{n}}=1.645\times \sqrt{\frac{0.89(1-0.89)}{1024}}=0.016$&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-04-19T11:39:35.177" CommentCount="0" />
  <row Id="9738" PostTypeId="1" CreationDate="2011-04-19T12:22:14.087" Score="2" ViewCount="85" Body="&lt;p&gt;I am attempting to build a Multinomial Logit model with dummy variables of the following form:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The dependent variable represents 0-8 discrete choices.&lt;/li&gt;&#xA;&lt;li&gt;Dummy Variable 1: 965 dummy vars&lt;/li&gt;&#xA;&lt;li&gt;Dummy Variable 2: 805 dummy vars&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The data set I am using has the dummy columns pre-created, so it's a table of 72,381 rows and 1770 columns.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first 965 columns represent the dummy columns for Variable 1; the next 805 columns represent the dummy columns for Variable 2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm on a Sun Grid Machine at my university, so memory won't be an issue...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have been able to generate the factors and generate &lt;code&gt;mlogit&lt;/code&gt; data using code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mldata&amp;lt;-mlogit.data(mydata, varying=NULL, choice=&quot;pitch_type_1&quot;, shape=&quot;wide&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;my &lt;code&gt;mlogit&lt;/code&gt; data looks like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&quot;dependent_var&quot;,&quot;A variable&quot;,&quot;B Var&quot;,&quot;chid&quot;,&quot;alt&quot;&#xA;FALSE,&quot;110&quot;,&quot;19&quot;,1,&quot;0&quot;&#xA;FALSE,&quot;110&quot;,&quot;19&quot;,1,&quot;1&quot;&#xA;FALSE,&quot;110&quot;,&quot;19&quot;,1,&quot;2&quot;&#xA;FALSE,&quot;110&quot;,&quot;19&quot;,1,&quot;3&quot;&#xA;FALSE,&quot;110&quot;,&quot;19&quot;,1,&quot;4&quot;&#xA;TRUE,&quot;110&quot;,&quot;19&quot;,1,&quot;5&quot;&#xA;FALSE,&quot;110&quot;,&quot;19&quot;,1,&quot;6&quot;&#xA;FALSE,&quot;110&quot;,&quot;19&quot;,1,&quot;7&quot;&#xA;FALSE,&quot;110&quot;,&quot;19&quot;,1,&quot;8&quot;&#xA;FALSE,&quot;110&quot;,&quot;19&quot;,2,&quot;0&quot;&#xA;FALSE,&quot;110&quot;,&quot;19&quot;,2,&quot;1&quot;&#xA;FALSE,&quot;110&quot;,&quot;19&quot;,2,&quot;2&quot;&#xA;FALSE,&quot;110&quot;,&quot;19&quot;,2,&quot;3&quot;&#xA;FALSE,&quot;110&quot;,&quot;19&quot;,2,&quot;4&quot;&#xA;FALSE,&quot;110&quot;,&quot;19&quot;,2,&quot;5&quot;&#xA;TRUE,&quot;110&quot;,&quot;19&quot;,2,&quot;6&quot;&#xA;FALSE,&quot;110&quot;,&quot;19&quot;,2,&quot;7&quot;&#xA;FALSE,&quot;110&quot;,&quot;19&quot;,2,&quot;8&quot;&#xA;TRUE,&quot;110&quot;,&quot;561&quot;,3,&quot;0&quot;&#xA;&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The &lt;code&gt;mldata&lt;/code&gt; contains 651,431 rows.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I try to run this full data set I get the following error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; mlogit.model&amp;lt;- mlogit(dependent_var~0|A+B, data = mldata, reflevel=&quot;0&quot;)&#xA;Error in model.matrix.default(formula, data) :&#xA;allocMatrix: too many elements specified&#xA;Calls: mlogit ... model.matrix.mFormula -&amp;gt; model.matrix -&amp;gt; model.matrix.default&#xA;Execution halted&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Smaller datasets (&lt;code&gt;mldata&lt;/code&gt; with only 595 rows) and &lt;code&gt;mlogit&lt;/code&gt; works fine and generates the expected regression output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a problem with &lt;code&gt;mlogit&lt;/code&gt; and huge datasets?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suppose this is perhaps not the best way to assess this kind of data, but I am trying to replicate a previous analysis that was completed on a similar amount of similar data. &lt;/p&gt;&#xA;" OwnerUserId="4231" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-20T12:24:23.947" LastActivityDate="2011-04-20T12:43:40.830" Title="Problem building multinomial logit model on huge data in R" Tags="&lt;r&gt;&lt;logistic-regression&gt;&lt;multinomial&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="9739" PostTypeId="1" CreationDate="2011-04-19T13:16:03.780" Score="5" ViewCount="179" Body="&lt;p&gt;Hi everyone&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a set of sea surface temperature (SST) monthly data and I want to apply some cluster methodology to detect regions with similar SST patterns. I have a set of monthly data files running from 1985 to 2009 and want to apply clustering to each month as a first step.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each file contains gridded data for 358416 points where approximately 50% are land and are marked with a 99.99 value that will be NA. Data format is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   lon     lat   sst&#xA;-10.042  44.979  12.38&#xA; -9.998  44.979  12.69&#xA; -9.954  44.979  12.90&#xA; -9.910  44.979  12.90&#xA; -9.866  44.979  12.54&#xA; -9.822  44.979  12.37&#xA; -9.778  44.979  12.37&#xA; -9.734  44.979  12.51&#xA; -9.690  44.979  12.39&#xA; -9.646  44.979  12.36&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have tried CLARA clustering method and got some apparently nice results but it also seems to me that is just smoothing (grouping) isolines. Then I am not sure this is the best clustering method to analyse spatial data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any other clustering method devoted to this type of datasets? Some reference would be good to start reading.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="4147" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-20T12:48:42.013" LastActivityDate="2011-05-12T10:29:20.660" Title="Clustering spatial data in R" Tags="&lt;r&gt;&lt;clustering&gt;&lt;spatial&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="2" />
  <row Id="9740" PostTypeId="2" ParentId="9728" CreationDate="2011-04-19T13:23:26.303" Score="5" ViewCount="" Body="&lt;p&gt;Considering the OLS case&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Y_{i}=\alpha+\beta X_{i}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One case is when you try to predict using values of $X_{i}$ outside your sample range (extrapolation).  Say if your data had $1&amp;lt;X_{i}&amp;lt;10$ in the sample, and you try to predict for when a new value is $X=100$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In OLS you have a prediction interval for a new value $X_{p}$ of:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\hat{Y}_{p}=\overline{Y}+\hat{\beta}(X_{p}-\overline{X})\pm t_{\alpha/2}^{(n-2)}\hat{\sigma}_{Y|X}\sqrt{1+\frac{1}{n}+\frac{X_{p}-\overline{X}}{\sum_{i=1}^{n}(X_{i}-\overline{X})^{2}}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Compare this to a prediction interval without using $X$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\hat{Y}_{p}=\overline{Y}\pm t_{\alpha/2}^{(n-1)}\hat{\sigma}_{Y}\sqrt{1+\frac{1}{n}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;on comparing the two, if the new covariate $X_{p}$ is far enough away from the mean value $\overline{X}$ the OLS prediction interval will get wider and wider, whereas the unconditional interval remains constant.  Further if the $X$ variable is uncorrelated with $Y$ so that $\hat{\beta}=0$, then $\hat{\sigma}_{Y|X}\geq \hat{\sigma}_{Y}$ and the prediction interval is always wider.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-04-19T13:23:26.303" CommentCount="1" />
  <row Id="9741" PostTypeId="1" AcceptedAnswerId="9782" CreationDate="2011-04-19T13:57:46.837" Score="2" ViewCount="95" Body="&lt;p&gt;I'm interested in assessing model performance on data with an ordinal categorical dependent variable. For my use case, the ideal metric would:  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Not assume equal intervals between classes or that recoding to a continuous scale is appropriate&lt;br&gt;&#xA;2) Be scale independent&lt;br&gt;&#xA;3) Give preference to models that &lt;em&gt;rank&lt;/em&gt; the outcomes accurately, with higher penalties for mis-ranking classes with a larger degree of difference (e.g., Excellent &gt; Poor &gt; Good is better than Excellent &gt; Very Poor &gt; Good)&lt;br&gt;&#xA;4) Accept continuous predictions and be indifferent to their distributions  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, suppose we have the following test set, where &quot;response&quot; is 5-category ordinal response and &quot;pred1&quot;, &quot;pred2&quot;, and &quot;pred3&quot; are predictions:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;id      response   pred1    pred2    pred3&#xA; 1     Excellent    1.00      150       10&#xA; 2          Good     .80       39        9&#xA; 3          Good     .85       12        5&#xA; 4          Fair     .40       11        4&#xA; 5          Poor     .39       10        3&#xA; 6     Very Poor     .20        3        2&#xA; .             .       .        .        .&#xA; .             .       .        .        .&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For my purposes, the ideal metric would score all three predictions as equally accurate since all three perfectly rank the response.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are my options and the benefits/drawbacks to each? Bonus points for references to R packages or functions. Thanks!&lt;/p&gt;&#xA;" OwnerUserId="1611" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-04-19T17:25:08.590" LastActivityDate="2011-04-20T13:03:50.337" Title="Model performance metrics for ordinal response" Tags="&lt;r&gt;&lt;model-selection&gt;&lt;predictive-models&gt;&lt;ordinal&gt;" AnswerCount="1" FavoriteCount="2" />
  <row Id="9742" PostTypeId="2" ParentId="4884" CreationDate="2011-04-19T14:14:37.803" Score="2" ViewCount="" Body="&lt;p&gt;If the treatment is randomly assigned the aggregation won't matter in determining the effect of the treatment (or the average treatment effect). I use lowercase in the following examples to refer to disaggregated items and uppercase to refer to aggregated items. Lets &lt;em&gt;a priori&lt;/em&gt; state a model of individual decision making, where $y$ is the outcome of interest, and $x$ represents when an observation recieved the treatment;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$y = \alpha + b_1(x) + b_2(z) + e$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When one aggregates, one is simply summing random variables. So one would observe;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\sum y = \sum\alpha + \beta_1(\sum x) + \beta_2(\sum z) + \sum e$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So what is to say that $\beta_1$ (divided by its total number of elements, $n$) will equal $b_1$? Because by the nature of random assignment all of the individual components of $x$ are orthogonal (i.e. the variance of $(\sum x)$ is simply the sum of the individual variances), and all of the individual components are uncorrelated with any of the $z$'s or $e$'s in the above equation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Perhaps using an example of summing two random variables will be more informative. So say we have a case where we aggregate two random variables from the first equation presented. So what we observe is;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$(y_i + y_j) = (\alpha_1 + \alpha_2) + \beta_1(x_i + x_j) + \beta_2(z_i + z_j) + (e_1 + e_2)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This can subsequently be broken down into its individual components;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$(y_i + y_j) = \alpha_1 + \alpha_2 + b_1(x_i) + b_2(x_j) + b_3(z_i) + b_4(z_j) + e_1 + e_2$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By the nature of random assignment we expect $x_i$ and $x_j$ in the above statement to be independent of all the other parameters ($z_i$, $z_j$, $e_1$, etc.) and &lt;em&gt;each other&lt;/em&gt;. Hence the effect of the aggregated data is equal to the effect of the data disaggregated (or $\beta_1$ equals the sum of $b_1$ and $b_2$). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This exercise is informative though to see where the aggregation bias will come into play. Anytime the components of that aggregated variable are not independent of the other components you are creating an inherent confound in the analysis (e.g. you can not independently identify the effects of each individual item). So going with your &quot;blue day&quot; scenario one might have a model of individual behavior;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$y = \alpha + b_1(x) + \beta_2(Z) + b_3(x*Z) + e$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $Z$ refers to whether the observation was taken on blue day and $x*Z$ is the interaction of the treatment effect with it being blue day. This should be fairly obvious why it would be problematic if you take all of your observations on one day. If treatment is randomly assigned $b_1(x)$ and $\beta_2(Z)$ should be independent, but $b_1(x)$ and $b_3(x*Z)$ are not. Hence you will not be able to uniquely identify $b_1$, and the research design is inherently confounded.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could potentially make a case for doing the data analysis on the aggregated items (aggregated values tend to be easier to work with and find correlations, less noisy and tend to have easier distributions to model). But if the real questions is to identify $b_1(x)$, then the research design should be structured to appropriately identify it. While I made an argument above for why it does not matter in a randomized experiment, in many settings the argument that all of the individual components are independent is violated. If you expect specific effects on specific days, aggregation of the observations will not help you identify the treatment effect (it is actually a good argument to prolong the observations to make sure no inherent confounds are present).&lt;/p&gt;&#xA;" OwnerUserId="1036" LastActivityDate="2011-04-19T14:14:37.803" CommentCount="4" />
  <row Id="9743" PostTypeId="2" ParentId="9629" CreationDate="2011-04-19T14:48:08.320" Score="3" ViewCount="" Body="&lt;p&gt;Using the extra information you gave (being that quite some of the values are 0), it's pretty obvious why your solution returns nothing. For one, you have a probability that is 0, so :&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;$e_i$ in the solution of Henry is 0 for at least one i&lt;/li&gt;&#xA;&lt;li&gt;$np_i$ in the solution of probabilityislogic is 0 for at least one i&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Which makes the divisions impossible. Now saying that $p=0$ means that it is impossible to have that outcome. If so, you might as well just erase it from the data (see comment of @cardinal). If you mean highly improbable, a first 'solution' might be to increase that 0 chance with a very small number.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X &amp;lt;- c(0,0,0,8,6,2,0,0)&#xA;p &amp;lt;- c(0.406197174,0.088746395,0.025193306,0.42041479,0.03192905,0.018328576,0.009190708,0)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You could do :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; p2 &amp;lt;- p + 1e-6&#xA;&amp;gt; chisq.test(X,p2)&#xA;&#xA;        Pearson's Chi-squared test&#xA;&#xA;data:  X and p2 &#xA;X-squared = 24, df = 21, p-value = 0.2931&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But this is not a correct result. In any case, one should avoid using the chi-square test in these borderline cases. A better approach is using a bootstrap approach, calculating an adapted test statistic and comparing the one from the sample with the distribution obtained by the bootstrap.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In R code this could be (step by step) : &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# The function to calculate the adapted statistic.&#xA;# We add 0.5 to the expected value to avoid dividing by 0&#xA;Statistic &amp;lt;- function(o,e){&#xA;    e &amp;lt;- e+0.5&#xA;    sum(((o-e)^2)/e)&#xA;}&#xA;&#xA;# Set up the bootstraps, based on the multinomial distribution&#xA;n &amp;lt;- 10000&#xA;bootstraps &amp;lt;- rmultinom(n,size=sum(X),p=p)&#xA;&#xA;# calculate the expected values&#xA;expected &amp;lt;- p*sum(X)&#xA;&#xA;# calculate the statistic for the sample and the bootstrap&#xA;ChisqSamp &amp;lt;- Statistic(X,expected)&#xA;ChisqDist &amp;lt;- apply(bootstraps,2,Statistic,expected)&#xA;&#xA;# calculate the p-value&#xA;p.value &amp;lt;- sum(ChisqSamp &amp;lt; sort(ChisqDist))/n&#xA;p.value&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This gives a p-value of 0, which is much more in line with the difference between observed and expected. Mind you, this method assumes your data is drawn from a multinomial distribution. If this assumption doesn't hold, the p-value doesn't hold either. &lt;/p&gt;&#xA;" OwnerUserId="1124" LastEditorUserId="1124" LastEditorDisplayName="" LastEditDate="2011-04-20T13:12:12.077" LastActivityDate="2011-04-20T13:12:12.077" CommentCount="3" />
  <row Id="9744" PostTypeId="1" AcceptedAnswerId="9793" CreationDate="2011-04-19T15:01:13.000" Score="4" ViewCount="68" Body="&lt;p&gt;I'm trying to understand the following claim:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;if the $t$-statistic is greater than&#xA;  zero, it indicates that the variable&#xA;  is explosive... but does that mean it&#xA;  has unit root?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In the context of &lt;strong&gt;Dickey Fuller&lt;/strong&gt; test.&lt;/p&gt;&#xA;" OwnerUserId="333" LastEditorUserId="2645" LastEditorDisplayName="" LastEditDate="2011-04-20T15:56:42.650" LastActivityDate="2011-04-20T15:56:42.650" Title="What is explosive variable?" Tags="&lt;hypothesis-testing&gt;&lt;stationarity&gt;" AnswerCount="1" />
  <row Id="9745" PostTypeId="1" CreationDate="2011-04-19T15:23:56.890" Score="1" ViewCount="80" Body="&lt;p&gt;How would you go about explainin &quot;Stambaugh Bias&quot; in simple relatively non-technical language?&lt;/p&gt;&#xA;" OwnerUserId="333" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-20T12:50:09.470" LastActivityDate="2011-04-20T12:50:09.470" Title="Stambaugh bias definition" Tags="&lt;time-series&gt;" AnswerCount="1" />
  <row Id="9746" PostTypeId="2" ParentId="9745" CreationDate="2011-04-19T15:37:25.683" Score="3" ViewCount="" Body="&lt;p&gt;I'm not sure you can explain this term without using some technical terms, unfortunately. I'll give it my best shot.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some definitions first:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Bias: the difference between the expectation of an estimator and the true value of the parameter you're estimating.&lt;/li&gt;&#xA;&lt;li&gt;OLS: Ordinary Least Squares; a method for solving a regression problem.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Autoregressive_process&quot; rel=&quot;nofollow&quot;&gt;Autoregressive process (AR): (via Wikipedia)&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Stambaugh bias&lt;/em&gt; occurs when you perform regression on a lagged stochastic input. Essentially, when you do this, you have to use an estimate for the input (regressor), which requires estimating autocorrelation coefficients. The bias in the autocorrelation coefficients is then proportional to the bias in the slope coefficient's estimate from the OLS. You can correct for this if you know that your method for computing autocorrelation coefficients is biased. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The original paper really isn't too complicated, so long as you know both what an AR process is and how OLS regression works: &lt;a href=&quot;http://finance.wharton.upenn.edu/~stambaugh/bias_reg.pdf&quot; rel=&quot;nofollow&quot;&gt;Paper&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="781" LastActivityDate="2011-04-19T15:37:25.683" CommentCount="5" />
  <row Id="9747" PostTypeId="2" ParentId="9667" CreationDate="2011-04-19T15:57:14.973" Score="6" ViewCount="" Body="&lt;p&gt;If you're coming from a mathematics background, and you want to learn time series, it's hard to go wrong with a combination of:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://books.google.com/books?id=qKzyAbdaDFAC&amp;amp;dq=chatfield%20time%20series&amp;amp;hl=en&amp;amp;ei=qq6tTajLKuTk0QHfzLH6Bw&amp;amp;sa=X&amp;amp;oi=book_result&amp;amp;ct=result&amp;amp;resnum=1&amp;amp;ved=0CDIQ6AEwAA&quot; rel=&quot;nofollow&quot;&gt;The Analysis of Time Series&lt;/a&gt; (Chatfield): introduction at the undergraduate level&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://books.google.com/books?id=zQsupRg5rrAC&amp;amp;printsec=frontcover&amp;amp;dq=time%20series%20bloomfield&amp;amp;hl=en&amp;amp;ei=aa-tTffiKMTX0QHKioC7Cw&amp;amp;sa=X&amp;amp;oi=book_result&amp;amp;ct=result&amp;amp;resnum=1&amp;amp;ved=0CDQQ6AEwAA#v=onepage&amp;amp;q&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;Fourier Analysis of Time Series&lt;/a&gt; (Bloomfield): introduction to Fourier methods at the undergraduate level&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;and after you've gone through those two and learned the basics, proceed to:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://books.google.com/books?id=_DcYu_EhVzUC&amp;amp;printsec=frontcover&amp;amp;dq=brockwell%20and%20davis%20time%20series&amp;amp;hl=en&amp;amp;src=bmrr&amp;amp;ei=4K6tTf36M-aP0QH4q-G2Cw&amp;amp;sa=X&amp;amp;oi=book_result&amp;amp;ct=result&amp;amp;resnum=1&amp;amp;ved=0CDQQ6AEwAA#v=onepage&amp;amp;q&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;Time Series: Theory and Methods&lt;/a&gt; (Brockwell &amp;amp; Davis): excellent high-level undergraduate / starting graduate-level book&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://books.google.com/books?id=olYZAQAAIAAJ&amp;amp;q=priestley%20time%20series&amp;amp;dq=priestley%20time%20series&amp;amp;hl=en&amp;amp;ei=gq6tTejIHIy10QGktYSxCw&amp;amp;sa=X&amp;amp;oi=book_result&amp;amp;ct=result&amp;amp;resnum=1&amp;amp;ved=0CDQQ6AEwAA&quot; rel=&quot;nofollow&quot;&gt;Spectral Analysis and Time Series&lt;/a&gt; (Priestley): excellent graduate-level text&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;and if you become interested in spectrum estimation, the best book I'm aware of is:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://books.google.com/books?id=FubniGJ0ECQC&amp;amp;printsec=frontcover&amp;amp;dq=percival%20and%20walden&amp;amp;hl=en&amp;amp;ei=V7CtTfvPFKnE0QHvgbW6Cw&amp;amp;sa=X&amp;amp;oi=book_result&amp;amp;ct=result&amp;amp;resnum=1&amp;amp;ved=0CCsQ6AEwAA#v=onepage&amp;amp;q&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;Spectral Analysis for Physical Applications&lt;/a&gt; (Percival and Walden): more of an engineering flavour, but lots of great examples and carefully written algorithms that you can turn into code.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;When I want to look up something I've seen before in classical time series methods, I mostly use Priestley. It's not an easy read by any means, but it's very well written, and you can go back to it and learn new things every time. Since you're coming from a mathematics background, you shouldn't have too much issue with any of the probabilistic notation, especially if you've had some measure theory. If I'm reviewing an algorithm for spectral methods, I use  Percival &amp;amp; Walden: it's the only good book I'm aware of that covers modern spectrum estimation techniques without diverging too strongly into wavelets or time-frequency methods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would encourage you to stay away from focused books on econometrics or any area of time series where the focus is on one particular area, as nonstandard notation and terminology tends to develop within these subfields. If it's your first approach to time series, start with a couple of good general undergraduate books (1 and 2 are decent, and have lots of examples that you can work through on your own with R). Only after you know the basics should you venture into the world of specific subfields and read books there.&lt;/p&gt;&#xA;" OwnerUserId="781" LastActivityDate="2011-04-19T15:57:14.973" CommentCount="4" />
  <row Id="9748" PostTypeId="1" AcceptedAnswerId="9750" CreationDate="2011-04-19T15:59:49.677" Score="3" ViewCount="73" Body="&lt;p&gt;First of all, I’m new to statistics and this is the first time I am trying to apply it to a real world problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am doing analysis of a series of observations of a variable over all weeks of a year. During certain weeks an event happened that I believe has impacted the variable and I want to check for this. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The values are technically a time series, however I only need to know that the event has an impact so I did an uncorrelated t-test. My reasoning is that, if I split the series of observations into a group of observations when the event happened and one where it didn’t happen, then I can compare the mean of each group using a t-test. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this reasoning valid?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks,&#xA;David&lt;/p&gt;&#xA;&#xA;&lt;p&gt;@GaBorgulya, I only have ONE value for each week.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;@Gavin Simpson No, I don't think that the values are autocorrelated. I am working with human behaviour and I believe that the event has an effect but it is not as deterministic as what it would be if the data were autocorrelated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;@Wayne, the event is random. &lt;/p&gt;&#xA;" OwnerUserId="4233" LastEditorUserId="4233" LastEditorDisplayName="" LastEditDate="2011-04-20T08:37:56.523" LastActivityDate="2011-04-20T08:37:56.523" Title="Can I split a series of observations of a variable over time into two groups instead of working with time series?" Tags="&lt;time-series&gt;&lt;statistical-significance&gt;&lt;mean&gt;&lt;t-test&gt;" AnswerCount="1" CommentCount="3" FavoriteCount="1" />
  <row Id="9749" PostTypeId="1" CreationDate="2011-04-19T16:14:33.530" Score="2" ViewCount="149" Body="&lt;p&gt;I have a set of data that are binomial, and am comparing them across 9 years. The first 5 years have low sample sizes ($~n=20$) and the last 4 have $n&amp;gt;100$. I've run a glm in R with the family set to &quot;binomial&quot;, and the results look reasonable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, when I did the multiple comparisons afterwards using the &lt;a href=&quot;http://cran.r-project.org/web/packages/multcomp/index.html&quot; rel=&quot;nofollow&quot;&gt;multcomp&lt;/a&gt; package, as shown below &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cs_comp&amp;lt;-glht(model2, linfct =  mcp(bin_Year= &quot;Tukey&quot;)) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I got some really &#xA;unexpected results that do not coincide with a bar graph I made of the means $\pm$ SE. I've been told that this is likely due to the unequal sample sizes. I've also had the suggestion that I use Wald's CIs to determine pairwise comparisons by looking for overlapping CI's. Does anyone know if this is a better approach to unequal samples sizes of this magnitude, or if there is another (even better) multiple comparison method for this? If so, any tips on how to approach it in R?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance!&lt;/p&gt;&#xA;" OwnerUserId="4238" LastEditorUserId="4238" LastEditorDisplayName="" LastEditDate="2011-05-14T23:39:19.180" LastActivityDate="2011-05-14T23:39:19.180" Title="Binomial GLM post-hoc tests for unequal sample sizes" Tags="&lt;r&gt;&lt;binomial&gt;&lt;glm&gt;&lt;post-hoc&gt;" AnswerCount="0" CommentCount="4" FavoriteCount="2" />
  <row Id="9750" PostTypeId="2" ParentId="9748" CreationDate="2011-04-19T16:25:56.253" Score="1" ViewCount="" Body="&lt;p&gt;Your reasoning sounds reasonable to me, although I have the feeling you are stretching the independence assumptions of t tests a little. Therefore, you should keep two things in mind.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, the size of both groups (weeks with event versus weeks without event) should be comparable. E.g., 20 versus 30. would be fine I guess.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second, your observations are not independent but follow a rule (weeks follow deterministically each other). Therefore, the occurrence of the events should be uncorrelated with the this rule (i.e., order of the weeks). This would be especially important if the dv (your measured variable) is influenced by the order of the week. But if you can negate both of these issues (correlation of event with order of weeks and of order with dv) you are good to go.&lt;/p&gt;&#xA;" OwnerUserId="442" LastActivityDate="2011-04-19T16:25:56.253" CommentCount="1" />
  <row Id="9751" PostTypeId="1" CreationDate="2011-04-19T16:51:22.190" Score="9" ViewCount="113" Body="&lt;p&gt;I often hear that post hoc tests after an ANOVA can only be used if the ANOVA itself was significant. However, post hoc tests adjust p values to keep the global type I error rate at 5%, don't they? So why do we need the global test first? If we don't need a global test is the terminology &quot;post hoc&quot; correct?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or are there multiple kinds of post hoc tests, some assuming a significant global test result and others without that assumption? &lt;/p&gt;&#xA;" OwnerUserId="4176" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-19T18:07:17.650" LastActivityDate="2011-04-20T18:35:24.807" Title="Do we need a global test before post hoc tests?" Tags="&lt;anova&gt;&lt;statistical-significance&gt;&lt;post-hoc&gt;" AnswerCount="2" FavoriteCount="2" />
  <row Id="9752" PostTypeId="1" AcceptedAnswerId="9754" CreationDate="2011-04-19T17:18:10.903" Score="2" ViewCount="100" Body="&lt;p&gt;I have a large set of customer data.  For these customers, I have devised a &lt;strong&gt;&lt;em&gt;customer loyalty score&lt;/em&gt;&lt;/strong&gt; which is a measure of the loyalty of the customer.  I want to find the features that are strongly associated/correlated with this score.  Features could be &lt;em&gt;number of purchases&lt;/em&gt; at various merchant types.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One obvious answer would to be just to calculate the &lt;strong&gt;&lt;em&gt;correlation&lt;/em&gt;&lt;/strong&gt; for each feature with the customer loyalty score and see which have the highest correlations.  Is this preferred way of doing this or are there better techniques?&lt;/p&gt;&#xA;" OwnerUserId="4235" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-20T12:50:21.000" LastActivityDate="2011-04-20T12:50:21.000" Title="Suggestions for identifying key features" Tags="&lt;correlation&gt;&lt;feature-selection&gt;" AnswerCount="4" CommentCount="0" FavoriteCount="2" />
  <row Id="9753" PostTypeId="2" ParentId="9751" CreationDate="2011-04-19T17:22:09.467" Score="7" ViewCount="" Body="&lt;p&gt;(1) &lt;em&gt;Post hoc&lt;/em&gt; tests might or might not achieve the nominal global Type I error rate, depending on (a) whether the analyst is adjusting for the number of tests and (b) to what extent the &lt;em&gt;post-hoc&lt;/em&gt; tests are independent of one another.  Applying a global test first is pretty solid protection against the risk of (even inadvertently) uncovering spurious &quot;significant&quot; results from &lt;em&gt;post-hoc&lt;/em&gt; data snooping.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(2) There is a problem of power. It is well known that a global ANOVA F test can detect a difference of means even in cases where no individual t-test of any of the pairs of means will yield a significant result.  In other words, in some cases the data can reveal that the true means likely differ but it cannot identify with sufficient confidence &lt;em&gt;which&lt;/em&gt; pairs of means differ.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-04-19T17:22:09.467" />
  <row Id="9754" PostTypeId="2" ParentId="9752" CreationDate="2011-04-19T17:58:00.073" Score="1" ViewCount="" Body="&lt;p&gt;I understand that the loyalty score is calculated on the strength of some data. If your features include components that are used in calculating the loyalty score they will prove evidently influential.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Multivariate techniques are probably more useful than pairwise correlations:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;they can detect weaker features that may be useful in combination with stronger ones&lt;/li&gt;&#xA;&lt;li&gt;they can reveal that some features have very similar information content.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The most simple way to start with could be multiple linear regression, although other methods may be better depending on many conditions.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastActivityDate="2011-04-19T17:58:00.073" CommentCount="2" />
  <row Id="9755" PostTypeId="2" ParentId="9752" CreationDate="2011-04-19T18:01:06.623" Score="1" ViewCount="" Body="&lt;p&gt;Sounds like Business Intelligence work (http://en.wikipedia.org/wiki/Business_intelligence). Could you confirm if it's a customer database or a survey that you ran? Both? Is it from a CRM database? Are customers segmented? Demographically/Physcographically? We need more detail as to what you have. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If it's a customer database, correlations tell a story about how features load upon your score but not the only story (cor != cause). If you have transactional information you can run survival analysis and calculate life time value (always useful). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We need to know a lot more about your variables in order to make recommendations of &quot;what to do with it&quot;&lt;/p&gt;&#xA;" OwnerUserId="776" LastActivityDate="2011-04-19T18:01:06.623" CommentCount="1" />
  <row Id="9756" PostTypeId="1" CreationDate="2011-04-19T18:10:25.063" Score="1" ViewCount="63" Body="&lt;p&gt;I have trained an &lt;strong&gt;SVM Regression model&lt;/strong&gt; using training data, $x_1,x_2,\dots,x_N$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to perform &lt;strong&gt;active learning&lt;/strong&gt; to improve the model; i.e., I want to add more samples to the training data and relearn a better model, and to choose these new samples in such a way as to maximize the resulting model performance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For an SVM classifier, a useful heuristic for active learning is to choose samples that fall close to the decision boundary; i.e., for a particular sample, the 'confidence' (&lt;em&gt;c&lt;/em&gt;) can be computed using the SVM. Samples that have small |c| are more likely to improve the decision boundary in the retrained SVM.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestions on how to do this for SVM regression?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(I can generate samples at will, but it is costly to label them, so I want to know if I can use the 'already-trained' regression-SVM to help me decide which ones to label)&lt;/p&gt;&#xA;" OwnerUserId="4218" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-20T13:00:49.357" LastActivityDate="2011-05-11T10:15:49.427" Title="Active learning using SVM Regression" Tags="&lt;regression&gt;&lt;cross-validation&gt;&lt;svm&gt;&lt;active-learning&gt;" AnswerCount="2" />
  <row Id="9757" PostTypeId="2" ParentId="9756" CreationDate="2011-04-19T18:20:25.197" Score="2" ViewCount="" Body="&lt;p&gt;Active learning requires a compromise between exploration and exploitation.  If the model you have so far is bad, if you exploit this model to determine the best place to label mode data, it will probably suggest bad places to label the data as your current hypothesis is poor.  It is a good idea to do some random exploration as well, as that is about the best way to ensure that eventually you will label the data that shows the current hypothesis to be incorrect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For regression models, I would suggest that Gaussian Process regression is a better bet for active learning, as it gives you predictive error bars, so you can query the labels for points where the model is most uncertain.  See for example &lt;a href=&quot;http://www.computer.org/portal/web/csdl/doi/10.1109/IJCNN.2000.861310&quot; rel=&quot;nofollow&quot;&gt;this paper&lt;/a&gt; looks an interesting place to start.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have worked on active learning in classification, and the results have been rather mixed for all strategies.  Often just picking points randomly (i.e. all exploration, no exploitation) works best.  I am looking into active learning for regression problems at the moment and intending to tse GPs, I'll add to my answer if I find out anything that seems to work better than exploration only.&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2011-04-19T18:20:25.197" />
  <row Id="9758" PostTypeId="2" ParentId="9752" CreationDate="2011-04-19T18:24:28.563" Score="1" ViewCount="" Body="&lt;p&gt;Hi!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In addition to the suggestions from the previous answers, I would suggest the &lt;code&gt;catdes&lt;/code&gt; function from the &lt;a href=&quot;ftp://ftp.ccu.edu.tw/pub/languages/CRAN/web/packages/FactoMineR/FactoMineR.pdf&quot; rel=&quot;nofollow&quot;&gt;FactoMineR&lt;/a&gt; package in R. It gives a &lt;em&gt;description of the categories of one factor by qualitative variables and/or by quantitative variables&lt;/em&gt;. The output is briefly explained in the manual but I think it would be worth to have a look at the reference mentioned there. The idea is that you get a list of the variables that characterise the most the factor along with a p-value to assess significance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note 1&lt;/strong&gt; I think that the function is particularly used in a &quot;cluster analysis&quot; context. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note 2&lt;/strong&gt; It requires to discretise your &quot;customer loyalty score&quot;...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By the way, about three years ago I used that function and I had a question about it. I wrote an email to the author (mentioned in the manual) and he kindly answered me! &lt;/p&gt;&#xA;" OwnerUserId="3019" LastEditorUserId="3019" LastEditorDisplayName="" LastEditDate="2011-04-19T18:37:16.410" LastActivityDate="2011-04-19T18:37:16.410" />
  <row Id="9759" PostTypeId="1" AcceptedAnswerId="9798" CreationDate="2011-04-19T18:46:18.587" Score="5" ViewCount="138" Body="&lt;p&gt;I'm new here, so I hope this hasn't been covered already, but my first few searches didn't find anything.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am about to dive into learning R and my learning project will entail applying mixed- or random-effects regression to a dataset in order to develop a predictive equation.  I share the concern of the writer in this post&#xA;&lt;a href=&quot;http://stats.stackexchange.com/questions/5344/how-to-choose-nlme-or-lme4-r-library-for-mixed-effects-models&quot;&gt;How to choose nlme or lme4 R library for mixed effects models?&lt;/a&gt; in wondering whether NLME or LME4 is the better package to familiarize myself with.  A more basic (hopefully not dumb) question is:  what's the difference between linear and nonlinear mixed-effects modeling?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For background, I applied M-E modeling in my MS research (in MATLAB, not R), so I'm familiar with how fixed vs. random variables are treated.  But I'm uncertain whether the work I did was considered linear or nonlinear M-E.  Is it simply the functional form of the equation used, or something else?&lt;/p&gt;&#xA;" OwnerUserId="4237" LastActivityDate="2011-04-20T17:21:42.293" Title="Can someone shed light on linear vs. nonlinear mixed-effects?" Tags="&lt;r&gt;&lt;regression&gt;&lt;random-effects-model&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="9760" PostTypeId="2" ParentId="9759" CreationDate="2011-04-19T20:00:02.607" Score="1" ViewCount="" Body="&lt;p&gt;For the linear-nonlinear part, see: &lt;a href=&quot;http://stats.stackexchange.com/questions/8689/what-does-linear-stand-for-in-linear-regression&quot;&gt;CrossValidated article on the topic&lt;/a&gt;, particularly the second-ranked answer by Charlie. I don't think there are any changes when dealing with mixed effects.&lt;/p&gt;&#xA;" OwnerUserId="1764" LastActivityDate="2011-04-19T20:00:02.607" CommentCount="1" />
  <row Id="9762" PostTypeId="1" CreationDate="2011-04-19T21:27:20.763" Score="0" ViewCount="66" Body="&lt;p&gt;I tried to transform my data (n=80) (for cluster analyses) using log, anti-log, square root, arcsin, arccosin, arctan, arcsinh, arccosinh, BUT none of the methods ended up with a normally distributed dataset (which is a requirement for cluster analyses).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there anything I can try? Thanks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S. All values in my data set are between 0 and 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S.S. The analysis I am doing is average linkage with Bray-Curtis similarity.&lt;/p&gt;&#xA;" OwnerUserId="2830" LastActivityDate="2011-04-19T21:27:20.763" Title="Transformations to normal for cluster analyses" Tags="&lt;clustering&gt;&lt;data-transformation&gt;&lt;normality&gt;" CommentCount="1" />
  <row Id="9763" PostTypeId="1" CreationDate="2011-04-19T22:41:27.670" Score="5" ViewCount="157" Body="&lt;p&gt;A typical image processing statistic is the use of &lt;a href=&quot;http://murphylab.web.cmu.edu/publications/boland/boland_node26.html&quot; rel=&quot;nofollow&quot;&gt;Haralick texture features&lt;/a&gt;, which are 14.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am wondering about the 14th of these features: Given an adjacency map $P$ (which we can simply view an the empirical distribution of two integers $i,j &amp;lt; 256$), it is defined as: &lt;em&gt;the square root of the second eigenvalue of $Q$&lt;/em&gt;, where $Q$ is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Q_{ij} = \sum_k \frac{ P(i,k) P(j,k)}{ [\sum_x P(x,i)] [\sum_y P(k, y)] }$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even after much googling, I could not find any references for this statistic. What are its properties? What is it representing?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(The value $P(i,j)$ above is the normalised number of times that a pixel of value $i$ is found next to a pixel of value $j$).&lt;/p&gt;&#xA;" OwnerUserId="2067" LastActivityDate="2011-04-28T18:12:12.330" Title="What is this &quot;maximum correlation coefficient&quot;?" Tags="&lt;probability&gt;&lt;computational-statistics&gt;" CommentCount="2" FavoriteCount="1" />
  <row Id="9764" PostTypeId="2" ParentId="9752" CreationDate="2011-04-19T23:17:10.557" Score="3" ViewCount="" Body="&lt;p&gt;One way to reformulate your problem is the following: you want to select a small set of features that predict well the loyalty score, using a linear model for example. This problem is called (best) subset selection. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose that you want to pick k features. The first way to do it is to test all the subsets of k features, by doing linear regression on each subset. But for large dataset, this is way too long.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another way to do it is in a greedy way. You start by picking the feature that is the most correlated with the score and add it to the (empty) subset. You compute the linear model associated to this subset (in this case, just a coefficient) to predict the loyalty score. Then, you pick the feature which is the most correlated with the residual (the difference between the value predicted by your linear model and the true score) and compute the linear model corresponding to your new subset. You do so, until you have k features in your set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are other methods, such as the lasso, to do subset selection. For a more complete introduction to subset selection, you should read the section 3.3 of &lt;a href=&quot;http://www-stat.stanford.edu/~tibs/ElemStatLearn/&quot; rel=&quot;nofollow&quot;&gt;The Elements of Statistical Learning&lt;/a&gt;, which is freely download-able on the authors' site.&lt;/p&gt;&#xA;" OwnerUserId="4241" LastEditorUserId="4241" LastEditorDisplayName="" LastEditDate="2011-04-19T23:22:28.073" LastActivityDate="2011-04-19T23:22:28.073" />
  <row Id="9765" PostTypeId="2" ParentId="9718" CreationDate="2011-04-20T00:20:29.863" Score="1" ViewCount="" Body="&lt;p&gt;For this particular case there's not much of a difference to work with in practical terms, but for the general case, I'm going to go out on a limb and guess that there is no way to conduct a strict test of significance.  The partial correlation will be a direct function of the correlations among the 3 variables.  Depending on 'career's &lt;em&gt;r&lt;/em&gt; with each of the others, we can specify the change from zero-order to partial &lt;em&gt;r&lt;/em&gt; for the 2 main variables.  So in a sense there's no room for variation and thus no place for a significance test.  I think.&lt;/p&gt;&#xA;" OwnerUserId="2669" LastActivityDate="2011-04-20T00:20:29.863" />
  <row Id="9766" PostTypeId="1" AcceptedAnswerId="9771" CreationDate="2011-04-20T02:46:09.323" Score="5" ViewCount="82" Body="&lt;p&gt;I have a question which asks:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Determine those values of the positive integer n for which a finite nth moment of X about zero exists.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;How should I approach this question? Does it depend on the numbers of variables in X? I think that a first moment exists if the mean exists, and the second moment exists if the variance can be determined. Is that general concept correct? &lt;/p&gt;&#xA;" OwnerUserId="3062" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-20T13:00:27.937" LastActivityDate="2011-05-20T20:03:52.177" Title="Determine whether a n-th finite moment of X exists" Tags="&lt;homework&gt;&lt;moments&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="9767" PostTypeId="1" CreationDate="2011-04-20T03:13:26.073" Score="5" ViewCount="61" Body="&lt;p&gt;Could anyone provide some suggestions on how to generate over-dispersed counts data with serial correlations? I am using R software to conduct a simulation study. Any references on this subject will be much appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for your help.&lt;/p&gt;&#xA;" OwnerUserId="2742" LastActivityDate="2011-04-20T13:03:57.170" Title="Generating over-dispersed counts data with serial correlation" Tags="&lt;r&gt;&lt;time-series&gt;&lt;distributions&gt;&lt;poisson&gt;&lt;simulation&gt;" AnswerCount="2" />
  <row Id="9768" PostTypeId="2" ParentId="9724" CreationDate="2011-04-20T03:18:45.590" Score="0" ViewCount="" Body="&lt;p&gt;The first thing that comes to my mind when I read of the approach that you describe is that there is a miss-match between the idea of replicating an experiment and the use of &quot;success&quot; and &quot;failure&quot; as descriptors of the outcomes. Presumably a success would be a result that is significant in the Neyman-Pearson paradigm (i.e. P &amp;lt; alpha). However, that paradigm allows control of type I errors and specification of power by assuming that the experimenter will act as if the null hypothesis is false when a significant result is observed. In that case there is no point in testing it again--the determination of type I error rates assumes that you discard the hypothesis and move on. Neyman called the process 'inductive behavior'.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A consequence of inductive behavior is that you can't integrate the information across multiple replications of the overall experiment within the Neyman-Pearson paradigm. There would be no sensible interpretation of any sequence of significant and not significant results from repeats of the experiment using the Neyman-Pearson paradigm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you use instead Fisher's approach of inductive inference, then you can combine P values from multiple experiments testing the same null hypothesis and obtain a composite value. The P values are indices of the strength of evidence against the null hypothesis (and NOT error rates!) and so, just as it is sensible to combine the evidence from multiple sources before judgement, you should combine the P values. However, while there are several (many?) way to combine the P values, it is perhaps the case that none is perfect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Arguably the best way to combine evidence from multiple experiments is to use a likelihood-based approach (see Royall 1997 for a very clear exposition) . The likelihood functions for each of the experiments can be combined exactly by multiplication to yield a composite function that can support both interval estimation and statement of the probability of the overall set of results having come from a true null hypothesis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Royall, R. M. (1997). Statistical evidence: a likelihood paradigm. Chapman &amp;amp; Hall.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;It is unfortunate that Fisher was frequently unclear in his expression, and rather inconsistent in his application of the ideas. However, he was undoubtedly in favor of repeating experiments and hated Neyman's idea of inductive behavior: &#xA;“...we may be able to validly apply a test of significance to discredit a hypothesis the expectations from which are widely at variance with the ascertained fact. If we use the term rejection for our attitude to such a hypothesis, it would be clearly understood that no irreversible decision has been taken; that, as rational beings, we are prepared to be convinced by future evidence that appearances were deceptive, and that a remarkable and exceptional coincidence had taken place.” Stat Meth &amp;amp; Sci Inf p.37 (Chapter II section 4).&lt;/p&gt;&#xA;" OwnerUserId="1679" LastActivityDate="2011-04-20T03:18:45.590" CommentCount="1" />
  <row Id="9770" PostTypeId="2" ParentId="9766" CreationDate="2011-04-20T07:14:14.310" Score="3" ViewCount="" Body="&lt;p&gt;It seems for me that the question is ill-posted if there is no additional context about $X$ distribution or at least the family of distribution it belongs to (Student $t$, &lt;a href=&quot;http://en.wikipedia.org/wiki/Pareto_distribution&quot; rel=&quot;nofollow&quot;&gt;Pareto&lt;/a&gt;, &lt;a href=&quot;http://en.wikipedia.org/wiki/Cauchy_distribution&quot; rel=&quot;nofollow&quot;&gt;Cauchy&lt;/a&gt;). For instance for normal distribution all moments exist, for Cauchy none. The topic is clearly related to the problem of &lt;a href=&quot;http://en.wikipedia.org/wiki/Heavy-tailed_distribution&quot; rel=&quot;nofollow&quot;&gt;heavy tails&lt;/a&gt;, therefore a &lt;em&gt;general approach&lt;/em&gt; would be:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;determine what distribution is relevant&lt;/li&gt;&#xA;&lt;li&gt;estimate the parameters of this distribution (consistent estimators will provide better parameter estimates with more numbers of $X$ available)&lt;/li&gt;&#xA;&lt;li&gt;decide on the number of finite moments that do exist&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;From practical point of view all you need is the behaviour of the tail of empirical distribution function. There is a great number of works and books on heavy tailed distributions, I can help to find some relevant methods if you think this approach is acceptable to you. &lt;/p&gt;&#xA;" OwnerUserId="2645" LastActivityDate="2011-04-20T07:14:14.310" />
  <row Id="9771" PostTypeId="2" ParentId="9766" CreationDate="2011-04-20T08:13:14.423" Score="6" ViewCount="" Body="&lt;p&gt;If you have the probability density function $f$ of the random variable, then it is a matter of checking for which $n$ the integral &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\int_{\mathbb{R}}x^nf(x)dx&amp;lt;\infty$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is then the standard exercise in real analysis. Alternatively if you know the &lt;a href=&quot;http://en.wikipedia.org/wiki/Characteristic_function_%28probability_theory%29&quot; rel=&quot;nofollow&quot;&gt;characteristic function&lt;/a&gt; of random variable $\phi$ then it is a matter of checking how many derivatives the function $\phi$ has.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Yet another alternative if you have the distribution function is to look for highest $n$ for which the the following limit is zero:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\lim_{t\to\infty}t^nP(|X|\ge t)=\lim_{t\to\infty}t^n(1-P(|X|&amp;lt;t))=\lim_{t\to\infty}(1-F(t)+F(-t))=0$$&lt;/p&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-20T20:03:52.177" LastActivityDate="2011-05-20T20:03:52.177" CommentCount="2" />
  <row Id="9772" PostTypeId="2" ParentId="9767" CreationDate="2011-04-20T08:51:38.153" Score="2" ViewCount="" Body="&lt;p&gt;This is one way to do it:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;v = rnorm(1, 30, 10)&#xA;for (i in 2:30) v = c(v, 0.5*v[i-1] + 0.5*rnorm(1, 30, 10))&#xA;round(v)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="3911" LastActivityDate="2011-04-20T08:51:38.153" CommentCount="1" />
  <row Id="9774" PostTypeId="1" CreationDate="2011-04-20T10:01:59.467" Score="6" ViewCount="81" Body="&lt;p&gt;For many years I have been conducting t-tests on response to mailing activity. Recently I was challenged that we should infact be conducting tests on profit rather than response.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, let me put this in context. If you have two groups of customers of sample size 10,000 each that you were mailing two different offers. One receives free delivery (group a) on goods and one receives 25% off their next purchase (group b). And assume the two groups responded as follows.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Group a - 10% of the 10,000 customers&#xA;mailed responded   &lt;/li&gt;&#xA;&lt;li&gt;Group b - 15% of the 10,000 customers&#xA;mailed responded&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I would conduct a t-test on the response rates (10% and 15%), which is correct.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have been asked to do a test on the profitability, which i am reluctant to do. Again to put this into context:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Group a - generated a profit of £1 per customer mailed (i.e. £10,000)&lt;/li&gt;&#xA;&lt;li&gt;Group b - generated a profit of £1.50 per customer mailed (i.e. £15,000)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I have been asked to do a test on the £1 and £1.50 profit, but my concern is that there are many variable factors that contribute to the final profit figures, eg, margins, cost of the mailing the customer received etc, that are not really considered when setting the groups up.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Am I correct in thinking that for this particular scenario that testing on profit is not the right thing to do, instead test on response as we are currently doing and then factor in all the other variable costs when making a final commercial decision?&lt;/p&gt;&#xA;" OwnerUserId="4248" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-04-20T16:40:37.660" LastActivityDate="2011-04-21T00:40:09.410" Title="Given two responses for two groups, how to decide what to test on response or profit?" Tags="&lt;t-test&gt;&lt;decision-theory&gt;" AnswerCount="1" CommentCount="4" />
  <row Id="9775" PostTypeId="1" AcceptedAnswerId="9780" CreationDate="2011-04-20T12:02:33.307" Score="0" ViewCount="57" Body="&lt;p&gt;Given a data-frame:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;d1 &amp;lt;-c(&quot;A&quot;,&quot;B&quot;,&quot;C&quot;,&quot;A&quot;)&#xA;d2 &amp;lt;-c(&quot;A&quot;,&quot;V&quot;,&quot;C&quot;,&quot;F&quot;)&#xA;d3 &amp;lt;-c(&quot;B&quot;,&quot;V&quot;,&quot;E&quot;,&quot;F&quot;)&#xA;d4 &amp;lt;-c(&quot;A&quot;,&quot;B&quot;,&quot;C&quot;,&quot;A&quot;)&#xA;data.frame(d1,d2,d3,d4)&#xA;  d1 d2 d3 d4&#xA;1  A  A  D  A&#xA;2  B  V  B  B&#xA;3  C  C  C  C&#xA;4  A  F  A  A&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Also given that each row may have a unique pattern such that the occurrence of the values A,D,A (first row) represents a unique pattern assigned to a class 1 and F,A,A last row also represents a unique pattern assigned a class 4. &#xA;I would like to manipulate the data-frame to search for rows that contain such 'unique patterns' and return a new column that classifies them such that, 0 represents rows that do not have any of the patterns. The pattern has to occur exactly as indicated. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   d1 d2 d3 d4  class&#xA;1  A  A  D  A     1&#xA;2  B  V  B  B     0&#xA;3  C  C  C  C     0&#xA;4  A  F  A  A     4&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I tried to use a select statement with a concat qualifier using package sqldf, but it does not provide a useful approach. &#xA;I would appreciate ideas on how to perform the search or if there are relevant packages to perform this type of search.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you&lt;/p&gt;&#xA;" OwnerUserId="201" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-04-20T12:13:27.853" LastActivityDate="2011-04-20T12:55:50.920" Title="Manipulating and searching data-frames" Tags="&lt;r&gt;&lt;dataframe&gt;" AnswerCount="1" CommentCount="4" />
  <row Id="9776" PostTypeId="1" CreationDate="2011-04-20T12:31:12.133" Score="1" ViewCount="34" Body="&lt;p&gt;I have a constant stream of geographical coordinates like so:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(lat1, long1),(lat2, long2),(lat3, long3)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know that device taking measurement can be off by 100 meters.&#xA;How do I filter out noise and get the actual measurements?&#xA;I mean if two positions are with in 100 meters of range then they can be considered as a single position.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I use Kalman filter to solve this problem?&lt;/p&gt;&#xA;" OwnerUserId="4251" LastActivityDate="2011-04-20T12:31:12.133" Title="How to apply Kalman filter to geographical coordinates?" Tags="&lt;kalman-filter&gt;" CommentCount="2" FavoriteCount="2" />
  <row Id="9777" PostTypeId="2" ParentId="9738" CreationDate="2011-04-20T12:43:40.830" Score="1" ViewCount="" Body="&lt;p&gt;Well, you are just exhausting RAM on your machine. Generally, you have four options:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Fetch a bigger computer (rather a bad idea, since it is rather impossible to push more than few hundred GB in one node).&lt;/li&gt;&#xA;&lt;li&gt;Limit your problem.&lt;/li&gt;&#xA;&lt;li&gt;Look for HPC version of multinomial logit, probably outside R -- using sparse matrices, parallelizable among multiple nodes, stuff.&lt;/li&gt;&#xA;&lt;li&gt;Switch to same better scalable algorithm.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;While you say that the problem was once solved, probably the way to go is option 3.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: I saw that the problem is in &lt;code&gt;model.matrix.default&lt;/code&gt;; this seems quite common while the formula (those statements with &lt;code&gt;~&lt;/code&gt;) interpretation algorithm in R is not written too well in terms of memory use. If there is a way to run your model without using formulas, try it.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2011-04-20T12:43:40.830" />
  <row Id="9778" PostTypeId="1" CreationDate="2011-04-20T12:44:27.060" Score="5" ViewCount="53" Body="&lt;p&gt;Most clustering algorithms I've seen start with creating a each-to-each distances among all points, which becomes problematic on larger datasets.  Is there one that doesn't do it? Or does it in some sort of partial/approximate/staggered approach?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which clustering algorithm/implemention takes less than O(n^2) space?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a list of algorithms and their Time and Space requirements somewhere?&lt;/p&gt;&#xA;" OwnerUserId="595" LastActivityDate="2011-04-20T13:38:13.817" Title="Space-efficient clustering" Tags="&lt;clustering&gt;&lt;algorithms&gt;&lt;large-data&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="1" />
  <row Id="9779" PostTypeId="1" AcceptedAnswerId="9819" CreationDate="2011-04-20T12:54:43.323" Score="1" ViewCount="146" Body="&lt;p&gt;I asked a question on StackOverflow for which I was suggested to use Kalman Filter. The question is as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://stackoverflow.com/questions/5726358/what-class-of-algorithms-reduce-margin-of-error-in-continuous-stream-of-input/5728373#5728373&quot;&gt;http://stackoverflow.com/questions/5726358/what-class-of-algorithms-reduce-margin-of-error-in-continuous-stream-of-input/5728373#5728373&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;A machine is taking measurements and&#xA;  giving me discrete numbers&#xA;  continuously like so:&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;1 2 5 7 8 10 11 12 13 14 18&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Let us say these measurements can be&#xA;  off by 2 points and a measurement is&#xA;  generated every 5 seconds. I want to&#xA;  ignore the measurements that may&#xA;  potentially be same&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Like continuous 2 and 3 could be same&#xA;  because margin of error is 2 so how do&#xA;  I partition the data such that I get&#xA;  only distinct measurements but I would&#xA;  also want to handle the situation in&#xA;  which the measurements are&#xA;  continuously increasing like so:&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;1 2 3 4 5 6 7 8 9 10&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;In this case if we keep ignoring the&#xA;  consecutive numbers with difference of&#xA;  less than 2 then we might lose actual&#xA;  measurements.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Now how do I apply Kalman Filter to solve this? All examples I see take multiple error estimations while I know a single thing that each measurement can be off by a value Q thats it and all the examples also work on multi dimensional vectors, that too multiple vectors.&lt;/p&gt;&#xA;" OwnerUserId="4251" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-20T18:36:22.290" LastActivityDate="2011-04-21T09:39:02.513" Title="How to apply Kalman filter to one dimensional data?" Tags="&lt;kalman-filter&gt;" AnswerCount="3" FavoriteCount="1" />
  <row Id="9780" PostTypeId="2" ParentId="9775" CreationDate="2011-04-20T12:55:50.920" Score="3" ViewCount="" Body="&lt;p&gt;Suppose the entries to data.frame contain single uppercase letters. Suppose that we have a vector containing the patterns and that only one pattern can be in one row.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;d1 &amp;lt;-c(&quot;A&quot;,&quot;B&quot;,&quot;C&quot;,&quot;A&quot;)&#xA;d2 &amp;lt;-c(&quot;A&quot;,&quot;V&quot;,&quot;C&quot;,&quot;F&quot;)&#xA;d3 &amp;lt;-c(&quot;B&quot;,&quot;V&quot;,&quot;E&quot;,&quot;F&quot;)&#xA;d4 &amp;lt;-c(&quot;A&quot;,&quot;B&quot;,&quot;C&quot;,&quot;A&quot;)&#xA;dd &amp;lt;- data.frame(d1,d2,d3,d4)&#xA;&amp;gt; dd&#xA;    d1 d2 d3 d4&#xA;1  A  A  B  A&#xA;2  B  V  V  B&#xA;3  C  C  E  C&#xA;4  A  F  F  A&#xA;&#xA;pats &amp;lt;- c(&quot;ABA&quot;,&quot;FFA&quot;)&#xA;pat.fun &amp;lt;- function(r,pats) {&#xA;     rr &amp;lt;- paste(r,collapse=&quot;&quot;)&#xA;     tmp &amp;lt;- sapply(pats,function(p)grep(p,rr))&#xA;     res &amp;lt;- which(tmp==1)&#xA;     if(length(res)==0) res &amp;lt;-0&#xA;     res&#xA; }&#xA;dd$class &amp;lt;- apply(dd,1,pats.fun,pats=pats)&#xA;&amp;gt; dd&#xA;&#xA;      d1 d2 d3 d4 class&#xA;1  A  A  B  A     1&#xA;2  B  V  V  B     0&#xA;3  C  C  E  C     0&#xA;4  A  F  F  A     2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is an example, the code certainly does not look like very efficient.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-04-20T12:55:50.920" CommentCount="2" />
  <row Id="9781" PostTypeId="2" ParentId="9573" CreationDate="2011-04-20T12:59:07.080" Score="8" ViewCount="" Body="&lt;p&gt;The central limit theorem is less useful than one might think in this context.  First, as someone pointed out already, one does not know if the current sample size is &quot;large enough&quot;.  Secondly, the CLT is more about achieving the desired type I error than about type II error.  In other words, the t-test can be uncompetitive power-wise.  That's why the Wilcoxon test is so popular.  If normality holds, it is 95% as efficient as the t-test.  If normality does not hold it can be arbitrarily more efficient than the t-test.&lt;/p&gt;&#xA;" OwnerUserId="4253" LastActivityDate="2011-04-20T12:59:07.080" CommentCount="2" />
  <row Id="9782" PostTypeId="2" ParentId="9741" CreationDate="2011-04-20T13:03:50.337" Score="4" ViewCount="" Body="&lt;p&gt;A good measure is Somers' Dxy rank correlation, a generalization of ROC area for ordinal or continuous Y.  It is computed for ordinal proportional odds regression in the lrm function in the rms package.&lt;/p&gt;&#xA;" OwnerUserId="4253" LastActivityDate="2011-04-20T13:03:50.337" CommentCount="1" />
  <row Id="9783" PostTypeId="2" ParentId="9767" CreationDate="2011-04-20T13:03:57.170" Score="4" ViewCount="" Body="&lt;p&gt;A standard way of generating overdispersed count data is to generate data from a Poisson distribution with a random mean: $Y_i\sim Poisson(\lambda_i)$, $\lambda_i \sim F$. For example, if $\lambda_i$ has a Gamma distribution, you will get the negative binomial distribution for $Y$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can easily impose serial correlation by imposing correlation on the $\lambda_i$'s. For example, you could have $\log\lambda_i \sim AR(1)$. Implemented in R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;N &amp;lt;- 100&#xA;rho &amp;lt;- 0.6&#xA;log.lambda &amp;lt;- 1 + arima.sim(model=list(ar=rho), n=N)&#xA;y &amp;lt;- rpois(N, lambda=exp(log.lambda))&#xA;&amp;gt; cor(head(y,-1), tail(y,-1))&#xA;[1] 0.4132512&#xA;&amp;gt; mean(y)&#xA;[1] 4.35&#xA;&amp;gt; var(y)&#xA;[1] 33.4015&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here $\lambda_i$'s come from a normal distribution, so the marginal distribution is not a classic distribution, but you could get more creative. Also note that the correlation of the $y$'s does not equal to &lt;code&gt;rho&lt;/code&gt;, but it is some function of it.&lt;/p&gt;&#xA;" OwnerUserId="279" LastActivityDate="2011-04-20T13:03:57.170" />
  <row Id="9784" PostTypeId="2" ParentId="9778" CreationDate="2011-04-20T13:38:13.817" Score="3" ViewCount="" Body="&lt;p&gt;K-Means and Mean-Shift use the raw sample descriptors (no need to pre-compute an affinity matrix).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Otherwise, for spectral clustering or power iteration clustering, you can use a sparse matrix representation (e.g. Compressed Sparse Rows) of the k-nearest-neighbours affinity matrix (for some distance or affinity metric). If k is small (let say 5 or 10). You will get a very space efficient representation (2 * n_samples * k * 8 bytes for double precision floating point values).&lt;/p&gt;&#xA;" OwnerUserId="2150" LastActivityDate="2011-04-20T13:38:13.817" />
  <row Id="9785" PostTypeId="1" CreationDate="2011-04-20T13:46:29.630" Score="3" ViewCount="90" Body="&lt;p&gt;Rob Tibshirani propose to use lasso with Cox&#xA;regression for variable selection in his 1997 paper&#xA;&quot;The lasso method for variable selection in the Cox&#xA;model&quot; published in Statistics In Medicine 16:385.&#xA;Does anyone know of any R package/function or syntax in R that&#xA;does lasso with a Cox model?&lt;/p&gt;&#xA;" OwnerUserId="4254" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-20T20:43:32.930" LastActivityDate="2011-04-20T20:43:32.930" Title="Cox model with LASSO" Tags="&lt;r&gt;&lt;regression&gt;&lt;survival&gt;&lt;lasso&gt;" AnswerCount="1" />
  <row Id="9786" PostTypeId="1" CreationDate="2011-04-20T13:47:25.053" Score="1" ViewCount="51" Body="&lt;p&gt;I want to test the hypothesis of a decreased level of vitamin D in diabetic subjects. For this I have recorded blood glucose and vitamin D levels in 40 cases and 40 controls. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What kind of statistical test can I use to the above hypothesis?&lt;/p&gt;&#xA;" OwnerUserId="4255" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-20T13:52:18.920" LastActivityDate="2011-04-20T15:28:56.520" Title="How to compare vitamin D and glucose levels between patients and controls?" Tags="&lt;hypothesis-testing&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="9787" PostTypeId="2" ParentId="9785" CreationDate="2011-04-20T14:01:49.567" Score="4" ViewCount="" Body="&lt;p&gt;Here are two suggestions. First, you can take a look at the &lt;a href=&quot;http://cran.r-project.org/web/packages/glmnet/index.html&quot; rel=&quot;nofollow&quot;&gt;glmnet&lt;/a&gt; package, from Friedman, Hastie and Tibshirani, but see their JSS 2010 (33) paper, &lt;a href=&quot;http://www.jstatsoft.org/v33/i01/paper&quot; rel=&quot;nofollow&quot;&gt;Regularization Paths for Generalized Linear Models via Coordinate Descent&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second, although I've never used this kind of penalized model, I know that the &lt;a href=&quot;http://cran.r-project.org/web/packages/penalized/index.html&quot; rel=&quot;nofollow&quot;&gt;penalized&lt;/a&gt; package implements L1/L2 penalties on GLM and the Cox model. What I found interesting in this package (this was with ordinary regression) was that you can include a set of unpenalized variables in the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The associated publication is now:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Goeman J.J. (2010). &lt;a href=&quot;http://onlinelibrary.wiley.com/doi/10.1002/bimj.200900028/abstract&quot; rel=&quot;nofollow&quot;&gt;L-1 Penalized&#xA;  Estimation in the Cox Proportional&#xA;  Hazards Model&lt;/a&gt;. &lt;em&gt;Biometrical&#xA;  Journal&lt;/em&gt; 52 (1) 70-84.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-04-20T14:01:49.567" />
  <row Id="9788" PostTypeId="2" ParentId="9718" CreationDate="2011-04-20T14:01:56.807" Score="2" ViewCount="" Body="&lt;p&gt;I don't doubt a particular test statistic aiming to accomplish what your asking for exists, but I will offer some alternatives that you may be interested in that offer different answers (but probably still interesting) given the nature of the question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Like Rolando already stated, the extent to which the partial correlation is moderated depends on the correlation between the three variables. Below I have inserted a picture of the situation in the form of a path diagram (and you can just pretend the box with X is demographic dissimilarity, Z is career development, and Y is satisfaction). The test for whether the direct correlation between X and Y is different than the partial correlation of X and Y controlling for Z amounts to stating whether the blue line &lt;em&gt;and&lt;/em&gt; the red line in the below picture are non-zero. So simply examining both of those correlations will be informative. If the product of those correlations is near zero, then the direct correlation between X and Y will be largely unchanged when partialling out the variance of Z. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/tQmHE.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another test you may be interested in would be a &lt;a href=&quot;http://en.wikipedia.org/wiki/Likelihood-ratio_test&quot; rel=&quot;nofollow&quot;&gt;likelihood ratio test&lt;/a&gt;. In this example you have a case of nested models, and hence you can test the model with only the direct correlation between X and Y and the model that includes both X and Z. This can be more easily extended to multiple Z's as well. But this is not the same as testing whether the effect of X is moderated by Z(s), but whether the model including X and Z is a better fit to the data than the model only including X.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some might interpret this question as asking about moderating and mediating effects. By the names of the variables though I wouldn't immediately suggest these are what you are after. &lt;a href=&quot;http://goo.gl/oBTBg&quot; rel=&quot;nofollow&quot;&gt;Kline (2010)&lt;/a&gt; has a few references regarding these, so those might be a good forray to search. Although to conduct those tests it appears you need to make much stronger assumptions about causality (&lt;a href=&quot;http://dx.doi.org/10.1037/0022-0167.51.1.115&quot; rel=&quot;nofollow&quot;&gt;Frazier, Barron, &amp;amp; Tix 2004&lt;/a&gt; &lt;a href=&quot;http://dionysus.psych.wisc.edu/Lit/Topics/Statistics/Mediation/frazier_barron_mediation_jcc.pdf&quot; rel=&quot;nofollow&quot;&gt;PDF&lt;/a&gt;). &lt;/p&gt;&#xA;" OwnerUserId="1036" LastActivityDate="2011-04-20T14:01:56.807" />
  <row Id="9789" PostTypeId="2" ParentId="9774" CreationDate="2011-04-20T14:05:00.977" Score="4" ViewCount="" Body="&lt;p&gt;The reason why you are conducting this test is to determine which policy is more valuable, and if value is measured in profitability, then it makes no sense to do statistical testing on any other variable.  A properly conducted test on profitability gives you all the information needed for your companies' decision: once you have the results of this test, results about other variables (e.g. response rate) provide no decision-relevant information.&lt;/p&gt;&#xA;" OwnerUserId="3567" LastEditorUserId="3567" LastEditorDisplayName="" LastEditDate="2011-04-21T00:40:09.410" LastActivityDate="2011-04-21T00:40:09.410" CommentCount="3" />
  <row Id="9790" PostTypeId="2" ParentId="7959" CreationDate="2011-04-20T14:39:35.683" Score="1" ViewCount="" Body="&lt;p&gt;I'd look at quantile regression. You can use it to determine a parametric estimate of whichever quantiles you want to look at. It make no assumption regarding normality, so it handles heteroskedasticity pretty well and can be used one a rolling window basis. It's basically an L1-Norm penalized regression, so it's not too numerically intensive and there's a pretty full featured R, SAS, and SPSS packages plus a few matlab implementations out there. Here's the &lt;a href=&quot;http://en.wikipedia.org/wiki/Quantile_regression&quot; rel=&quot;nofollow&quot;&gt;main&lt;/a&gt; and the &lt;a href=&quot;http://en.wikibooks.org/wiki/R_Programming/Quantile_Regression&quot; rel=&quot;nofollow&quot;&gt;R&lt;/a&gt; package wikis for more info.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edited:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Check out the math stack exchange crosslink: Someone sited a couple of papers that essentially lay out the very simple idea of just using a rolling window of order statistics to estimate quantiles. Literally all you have to do is sort the values from smallest to largest, select which quantile you want, and select the highest value within that quantile. You can obviously give more weight to the most recent observations if you believe they are more representative of actual current conditions. This will probably give rough estimates, but it's fairly simple to do and you don't have to go through the motions of quantitative heavy lifting. Just a thought.&lt;/p&gt;&#xA;" OwnerUserId="3737" LastEditorUserId="3737" LastEditorDisplayName="" LastEditDate="2011-04-20T19:09:57.173" LastActivityDate="2011-04-20T19:09:57.173" />
  <row Id="9791" PostTypeId="2" ParentId="9779" CreationDate="2011-04-20T15:00:21.363" Score="0" ViewCount="" Body="&lt;p&gt;I believe that for the Kalman Filter, you'll need to clarify your &quot;can be off by 2 points&quot; into something like, &quot;error is normal with mean 0 and standard deviation of 0.8&quot;. Also, I believe that the usual statement of the Kalman Filter assumes you have a model that would predict how the actual value changes over time. (Though I guess you can specify a naive model that basically says x_t = x_t-1.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found an interesting explanation at &lt;a href=&quot;http://www.convict.lu/htm/rob/imperfect_data_in_a_noisy_world.htm&quot; rel=&quot;nofollow&quot;&gt;this website&lt;/a&gt; that might be helpful.&lt;/p&gt;&#xA;" OwnerUserId="1764" LastActivityDate="2011-04-20T15:00:21.363" />
  <row Id="9792" PostTypeId="2" ParentId="9786" CreationDate="2011-04-20T15:28:56.520" Score="2" ViewCount="" Body="&lt;p&gt;Generally this sound like a simple &lt;a href=&quot;http://en.wikipedia.org/wiki/T-test&quot; rel=&quot;nofollow&quot;&gt;t-test&lt;/a&gt;. That is, you have two groups (diabetics and controls) and you measured 1 variable (Vitamin D).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, some more context/information about your data will lead to a lot better answers. For example, please answer chl's comment.&lt;br&gt;&#xA;Second, what is the idea behind measuring the glucose level regarding your hypothesized relation between diabetes and Vitamin D?&lt;/p&gt;&#xA;" OwnerUserId="442" LastActivityDate="2011-04-20T15:28:56.520" />
  <row Id="9793" PostTypeId="2" ParentId="9744" CreationDate="2011-04-20T15:31:31.433" Score="5" ViewCount="" Body="&lt;p&gt;First of all it could be useful to read a bit about the &lt;a href=&quot;http://en.wikipedia.org/wiki/Unit_root&quot; rel=&quot;nofollow&quot;&gt;unit root&lt;/a&gt; problem (you may start from the hypothesis section). So the nature of the explosiveness (exponential growth) is what matters. Roughly the growth could be explained either by deterministic part (for example linear trend) or by random walk with drift. Dickey Fuller (not the best unit root tests choice though, if you are familiar with $R$ I would suggest to go for &lt;code&gt;library(urca)&lt;/code&gt; Zivot and Andrews test &lt;code&gt;?ur.za&lt;/code&gt;, since this one also counts for possible structural breaks) is designed to distinguish between the too, thus $t$ statistic will help to decide if the nature of explosive behavior is deterministic or not. But bear in mind that DF test is very sensitive to deviations from the assumptions it was build on, even ADF would be more robust to apply in practice.&lt;/p&gt;&#xA;" OwnerUserId="2645" LastActivityDate="2011-04-20T15:31:31.433" CommentCount="2" />
  <row Id="9794" PostTypeId="1" AcceptedAnswerId="9816" CreationDate="2011-04-20T15:41:51.503" Score="5" ViewCount="57" Body="&lt;p&gt;Let's say I have two time series, one of which updates more frequently than the other:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$x_0,x_1,x_2,\dots,x_t,\dots$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$y_0,y_{10},y_{20},\dots,y_{10t},\dots$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to fit a model to this that predicts $y$ from $x$ (and possibly from previous values of $y$) at each of the values $1,2,3,\dots$, i.e. it gives a prediction even for values of $y$ for which we won't make an observation (equivalently, assume that there are true values for $y$ at every value of $t$, but we only observe it at $t=0,10,20,\dots$)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a canonical way to do this?&lt;/p&gt;&#xA;" OwnerUserId="2425" LastActivityDate="2011-04-21T06:50:58.667" Title="Time series factor model with one series more frequent" Tags="&lt;regression&gt;&lt;time-series&gt;&lt;predictive-models&gt;" AnswerCount="2" />
  <row Id="9795" PostTypeId="1" CreationDate="2011-04-20T16:16:38.850" Score="1" ViewCount="46" Body="&lt;p&gt;I am trying to predict diabetes using the &lt;a href=&quot;http://www.cdc.gov/brfss/&quot; rel=&quot;nofollow&quot;&gt;BRFSS dataset&lt;/a&gt; by using a supervised learning classification model. But I see that the target variable which is having diabetes or not is skewed. That is 90% of the records are non-diabetic and only 10% of the records are diabetic. How do I handle the skewness in the target variable? &lt;/p&gt;&#xA;" OwnerUserId="3897" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-04-20T18:49:53.570" LastActivityDate="2011-04-20T18:49:53.570" Title="How to handle skewed target variables?" Tags="&lt;machine-learning&gt;&lt;sampling&gt;&lt;unbalanced-classes&gt;" CommentCount="2" />
  <row Id="9796" PostTypeId="2" ParentId="9794" CreationDate="2011-04-20T16:56:56.410" Score="3" ViewCount="" Body="&lt;p&gt;I would cast the model in state-space form. Then there is no problem if one of the variables is observed more frequently than the other, or the observation times are irregular: the Kalman filter deals with missing and partially observed variables gracefully.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Without details on the exact kind of relationships you aim to model it is difficult to be more specific.&lt;/p&gt;&#xA;" OwnerUserId="892" LastActivityDate="2011-04-20T16:56:56.410" CommentCount="1" />
  <row Id="9797" PostTypeId="1" CreationDate="2011-04-20T17:20:17.850" Score="3" ViewCount="145" Body="&lt;p&gt;I have the following data, representing the binary state of four subjects at four times, note that it is only possible for each subject to transition $0\to 1$ but not $1\to 0$:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;testdata &amp;lt;- data.frame(id = c(1,2,3,4,1,2,3,4,1,2,3,4,1,2,3,4,1,2,3,4),&#xA;                       day = c(1,1,1,1,8,8,8,8,16,16,16,16,24,24,24,24,32,32,32,32),&#xA;                       obs = c(0,0,0,0,0,1,0,0,0,1,1,0,0,1,1,1,1,1,1,1))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I can model it with a logistic regression:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;testmodel &amp;lt;- glm(formula(obs~day, family=binomial), data=testdata)&#xA;&#xA;&amp;gt; summary(testmodel)&#xA;&#xA;&#xA;Coefficients:&#xA;              Estimate Std. Error t value Pr(&amp;gt;|t|)    &#xA;(Intercept) -0.018890   0.148077  -0.128 0.899907    &#xA;day          0.032030   0.007555   4.240 0.000493 ***&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;First, how can I account for repeated measures on the same individual within the model?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second, how can I estimate, with uncertainty, the day on which 1/2 of the subjects will have made the transition from $0\to 1$?&lt;/p&gt;&#xA;" OwnerUserId="1381" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-20T20:08:37.077" LastActivityDate="2011-05-27T16:50:47.630" Title="How can I use logistic regression to estimate the time at which 50% of a binomial variable will have transitioned?" Tags="&lt;logistic-regression&gt;&lt;censoring&gt;&lt;interval-censoring&gt;" AnswerCount="3" CommentCount="12" />
  <row Id="9798" PostTypeId="2" ParentId="9759" CreationDate="2011-04-20T17:21:42.293" Score="4" ViewCount="" Body="&lt;p&gt;There are several distinctions between linear and nonlinear regression models, but the primary mathematical one is that linear models are linear in the parameters, whereas nonlinear models are nonlinear in the parameters. Pinheiro and Bates (2000, pp. 284-285), authors of the &lt;code&gt;nlme&lt;/code&gt; R package, elegantly described the more substantive considerations in model selection:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;When choosing a regression model to describe how a response variable varies with covariates, one always has the option of using models, such as polynomial models, that are linear in the parameters. By increasing the order of a polynomial model, one can get increasingly accurate approximations to the true, usually nonlinear, regression function, within the observed range of the data. These empirical models are based only on the observed relationship between the response and the covariates and do not include any theoretical considerations about the underlying mechanism producing the data.&#xA;  Nonlinear models, on the other hand, are often mechanistic, i.e., based on a model for the mechanism producing the response. As a consequence, the model parameters in a nonlinear model generally have a natural physical interpretation. Even when derived empirically, nonlinear models usually incorporate known, theoretical characteristics of the data, such as asymptotes and monotonicity, and in these cases, can be considered as semi-mechanistic models. A nonlinear model generally uses fewer parameters than a competitor linear model, such as a polynomial, giving a more parsimonious description of the data. Nonlinear models also provide more reliable predictions for the response variable outside the observed range of the data than, say, polynomial models would.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;There are also some big differences between the nlme and lme4 packages that go beyond the linearity issue. For example, using nlme you can fit linear or nonlinear models and, for either type, specify the variance and correlation structures for within-group errors (e.g., autoregressive); lme4 can't do that. In addition, random effects can be fixed or crossed in either package, but it's much easier (and more computationally efficient) to specify and model crossed random effects in lme4.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would advise first considering a) whether you will need a nonlinear model, and b) whether you will need to specify either the within-group variance or correlation structures. If any of these answers is yes, then you have to use nlme (given that you're sticking with R). If you work a lot with linear models that have crossed random effects, or complicated combinations of nested and crossed random effects, then lme4 is probably a better choice. You may need to learn to use both packages. I learned lme4 first and then realized I had to use nlme because I almost always work with autoregressive error structures. However, I still prefer lme4 when I analyze data from experiments with crossed factors. The good news is that a great deal of what I learned about lme4 transferred well to nlme. Either way, Pinheiro and Bates (2000) is a great reference for mixed-effects models, and I'd say it's indispensable if you're using nlme.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;References&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Pinheiro, J.C., &amp;amp; Bates, D.M. (2000). &lt;em&gt;Mixed-effects models in S and S-PLUS&lt;/em&gt;. New York: Springer-Verlag.&lt;/p&gt;&#xA;" OwnerUserId="3964" LastActivityDate="2011-04-20T17:21:42.293" CommentCount="1" />
  <row Id="9799" PostTypeId="2" ParentId="9779" CreationDate="2011-04-20T18:21:57.773" Score="0" ViewCount="" Body="&lt;p&gt;I found a nice simple introductory example of a Kalman filter (coded in matlab) &lt;a href=&quot;http://www.mathworks.com/matlabcentral/fileexchange/5377-learning-the-kalman-filter&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;. The example the author provides in this code is on one dimensional  data. Hopefully this will at least give you a starting point for figuring out how to apply it to your specific problem.&lt;/p&gt;&#xA;" OwnerUserId="1913" LastActivityDate="2011-04-20T18:21:57.773" />
  <row Id="9800" PostTypeId="2" ParentId="9751" CreationDate="2011-04-20T18:35:24.807" Score="3" ViewCount="" Body="&lt;p&gt;Since multiple comparison tests are often called 'post tests', you'd think they logically follow the one-way ANOVA. In fact, this isn't so.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&quot;&lt;em&gt;An unfortunate common practice is to pursue multiple comparisons only when the hull hypothesis of homogeneity is rejected.&lt;/em&gt;&quot; (&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0412982811&quot; rel=&quot;nofollow&quot;&gt;Hsu, page 177&lt;/a&gt;)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Will the results of post tests be valid if the overall P value for the ANOVA is greater than 0.05?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Surprisingly, the answer is yes. With one exception, post tests are valid even if the overall ANOVA did not find a significant difference among means.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The exception is the first multiple comparison test invented, the protected Fisher Least Significant Difference (LSD) test. The first step of the protected LSD test is to check if the overall ANOVA rejects the null hypothesis of identical means. If it doesn't, individual comparisons should not be made.  But this protected LSD test is outmoded, and no longer recommended.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Is it possible to get a 'significant' result from a multiple comparisons test even when the overall ANOVA was not significant?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Yes it is possible. The exception is Scheffe's test. It is intertwined with the overall F test. If the overall ANOVA has a P value greater than 0.05, then the Scheffe's test won't find any significant post tests. In this case, performing post tests following an overall nonsignificant ANOVA is a waste of time but won't lead to invalid conclusions. But other multiple comparison tests can find significant differences (sometimes) even when the overall ANOVA showed no significant differences among groups.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How can I understand the apparent contradiction between an ANOVA saying, in effect, that all group means are identical and a post test finding differences?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The overall one-way ANOVA tests the null hypothesis that all the treatment groups have identical mean values, so any difference you happened to observe is due to random sampling. Each post test tests the null hypothesis that two particular groups have identical means.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The post tests are more focused, so have power to find differences between groups even when the overall ANOVA reports that the differences among the means are not statistically significant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Are the results of the overall ANOVA useful at all?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ANOVA tests the overall null hypothesis that all the data come from groups that have identical means. If that is your experimental question -- does the data provide convincing evidence that the means are not all identical -- then ANOVA is exactly what you want. More often, your experimental questions are more focused and answered by multiple comparison tests (post tests). In these cases, you can safely ignore the overall ANOVA results and jump right to the post test results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that the multiple comparison calculations all use the mean-square result from the ANOVA table. So even if you don't care about the value of F or the P value, the post tests still require that the ANOVA table be computed.&lt;/p&gt;&#xA;" OwnerUserId="25" LastActivityDate="2011-04-20T18:35:24.807" />
  <row Id="9801" PostTypeId="1" AcceptedAnswerId="9802" CreationDate="2011-04-20T18:39:30.667" Score="6" ViewCount="222" Body="&lt;p&gt;I'm trying to understand matrix notation, and working with vectors and matrices. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Right now I'd like to understand how the vector of coefficient estimates $\hat{\beta}$ in multiple regression is computed.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The basic equation seems to be  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;\frac{d}{d\boldsymbol{\beta}} (\boldsymbol{y}-\boldsymbol{X\beta})&amp;#39;(\boldsymbol{y}-\boldsymbol{X\beta}) = 0 \&amp;gt;.&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now how would I solve for a vector $\beta$ here?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit&lt;/strong&gt;: Wait, I'm stuck. I'm here now and don't know how to continue:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$    \frac{d}{d{\beta}} &#xD;&#xA;\left( &#xD;&#xA;\left(\begin{smallmatrix}&#xD;&#xA; y_1 \\&#xD;&#xA; y_2 \\&#xD;&#xA; \vdots \\&#xD;&#xA; y_n&#xD;&#xA;\end{smallmatrix}\right)&#xD;&#xA;-&#xD;&#xA;\left(\begin{smallmatrix}&#xD;&#xA; 1 &amp;amp; x_{11} &amp;amp; x_{12} &amp;amp; \dots &amp;amp; x_{1p} \\&#xD;&#xA; 1 &amp;amp; x_{21} &amp;amp; x_{22} &amp;amp; \dots &amp;amp; x_{2p} \\&#xD;&#xA; \vdots &amp;amp; &amp;amp; &amp;amp; &amp;amp; \vdots \\&#xD;&#xA; 1 &amp;amp; x_{n1} &amp;amp; x_{n2} &amp;amp; \dots &amp;amp; x_{np} \\&#xD;&#xA;\end{smallmatrix}\right)&#xD;&#xA;\left(\begin{smallmatrix}&#xD;&#xA; \beta_0 \\&#xD;&#xA; \beta_1 \\&#xD;&#xA; \vdots \\&#xD;&#xA; \beta_p&#xD;&#xA;\end{smallmatrix}\right)&#xD;&#xA;\right) &amp;#39;&#xD;&#xA;\left( &#xD;&#xA;\left(\begin{smallmatrix}&#xD;&#xA; y_1 \\&#xD;&#xA; y_2 \\&#xD;&#xA; \vdots \\&#xD;&#xA; y_n&#xD;&#xA;\end{smallmatrix}\right)&#xD;&#xA;-&#xD;&#xA;\left(\begin{smallmatrix}&#xD;&#xA; 1 &amp;amp; x_{11} &amp;amp; x_{12} &amp;amp; \dots &amp;amp; x_{1p} \\&#xD;&#xA; 1 &amp;amp; x_{21} &amp;amp; x_{22} &amp;amp; \dots &amp;amp; x_{2p} \\&#xD;&#xA; \vdots &amp;amp;  &amp;amp; &amp;amp; &amp;amp; \vdots \\&#xD;&#xA; 1 &amp;amp; x_{n1} &amp;amp; x_{n2} &amp;amp; \dots &amp;amp; x_{np} \\&#xD;&#xA;\end{smallmatrix}\right)&#xD;&#xA;\left(\begin{smallmatrix}&#xD;&#xA; \beta_0 \\&#xD;&#xA; \beta_1 \\&#xD;&#xA; \vdots \\&#xD;&#xA; \beta_p&#xD;&#xA;\end{smallmatrix}\right)&#xD;&#xA;\right)&#xD;&#xA;$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$&#xD;&#xA;\frac{d}{d{\beta}} \sum_{i=1}^n \left( y_i - \begin{pmatrix} 1 &amp;amp; x_{i1} &amp;amp; x_{i2} &amp;amp; \dots &amp;amp; x_{ip} \end{pmatrix} &#xD;&#xA;\begin{pmatrix}&#xD;&#xA; \beta_0 \\&#xD;&#xA; \beta_1 \\&#xD;&#xA; \vdots \\&#xD;&#xA; \beta_p&#xD;&#xA;\end{pmatrix} \right)^2$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With $x_{i0} = 1$ for all $i$ being the intercept:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$&#xD;&#xA;\frac{d}{d{\beta}} \sum_{i=1}^n \left( y_i - &#xD;&#xA;\sum_{k=0}^p x_{ik} \beta_k&#xD;&#xA; \right)^2&#xD;&#xA;$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can you point me in the right direction?&lt;/p&gt;&#xA;" OwnerUserId="2091" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-29T00:54:40.567" LastActivityDate="2011-04-29T00:54:40.567" Title="Analytical solution to linear-regression coefficient estimates" Tags="&lt;regression&gt;" AnswerCount="4" CommentCount="1" FavoriteCount="2" />
  <row Id="9802" PostTypeId="2" ParentId="9801" CreationDate="2011-04-20T19:04:57.233" Score="11" ViewCount="" Body="&lt;p&gt;We have&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\frac{d}{d\beta} (y - X \beta)&amp;#39; (y - X\beta) = -2 X&amp;#39; (y - X \beta)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It can be shown by writing the equation explicitly with components. For example, write $(\beta_{1}, \ldots, \beta_{p})&amp;#39;$ instead of $\beta$. Then take derivatives with respect to $\beta_{1}$, $\beta_{2}$, ..., $\beta_{p}$ and stack everything to get the answer. For a quick and easy illustration, you can start with $p = 2$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With experience one develops general rules, some of which are given, e.g., in &lt;a href=&quot;http://www.colorado.edu/engineering/cas/courses.d/IFEM.d/IFEM.AppD.d/IFEM.AppD.pdf&quot; rel=&quot;nofollow&quot;&gt;that document&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit to guide for the added part of the question&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With $p = 2$, we have&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$(y - X \beta)&amp;#39;(y - X \beta) = (y_1 - x_{11} \beta_1 - x_{12} \beta_2)^2 + &#xD;&#xA;(y_2 - x_{21}\beta_1 - x_{22} \beta_2)^2$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The derivative with respect to $\beta_1$ is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$-2x_{11}(y_1 - x_{11} \beta_1 - x_{12} \beta_2)-2x_{21}(y_2 - x_{21}\beta_1 - x_{22} \beta_2)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Similarly, the derivative with respect to $\beta_2$ is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$-2x_{12}(y_1 - x_{11} \beta_1 - x_{12} \beta_2)-2x_{22}(y_2 - x_{21}\beta_1 - x_{22} \beta_2)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hence, the derivative with respect to $\beta = (\beta_1, \beta_2)&amp;#39;$ is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$&#xD;&#xA;\left( &#xD;&#xA;\begin{array}{c}&#xD;&#xA;-2x_{11}(y_1 - x_{11} \beta_1 - x_{12} \beta_2)-2x_{21}(y_2 - x_{21}\beta_1 - x_{22} \beta_2) \\&#xD;&#xA;-2x_{12}(y_1 - x_{11} \beta_1 - x_{12} \beta_2)-2x_{22}(y_2 - x_{21}\beta_1 - x_{22} \beta_2)&#xD;&#xA;\end{array}&#xD;&#xA;\right)&#xD;&#xA;$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, observe you can rewrite the last expression as &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$-2\left( &#xD;&#xA;\begin{array}{cc}&#xD;&#xA;x_{11} &amp;amp; x_{21} \\&#xD;&#xA;x_{12} &amp;amp; x_{22}&#xD;&#xA;\end{array}&#xD;&#xA;\right)\left( &#xD;&#xA;\begin{array}{c}&#xD;&#xA;y_{1} - x_{11}\beta_{1} - x_{12}\beta_2 \\&#xD;&#xA;y_{2} - x_{21}\beta_{1} - x_{22}\beta_2&#xD;&#xA;\end{array}&#xD;&#xA;\right) = -2 X&amp;#39; (y - X \beta)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, everything is done in the same way for a larger $p$. &lt;/p&gt;&#xA;" OwnerUserId="3019" LastEditorUserId="2129" LastEditorDisplayName="" LastEditDate="2011-04-21T14:16:35.683" LastActivityDate="2011-04-21T14:16:35.683" CommentCount="8" />
  <row Id="9803" PostTypeId="2" ParentId="9797" CreationDate="2011-04-20T19:17:19.013" Score="0" ViewCount="" Body="&lt;p&gt;We know that the $t_1$ transition time (from state 0 to state 1) of subject &lt;code&gt;id=1&lt;/code&gt; was between two boundaries: $24&amp;lt;t_1&amp;lt;32$. &lt;strong&gt;An approximation is&lt;/strong&gt; to assume that $t_1$ may have taken values within this range with &lt;strong&gt;uniform&lt;/strong&gt; probability. Resampling the $t_i$ values we can get an approximate distribution of $\text{median}(t_i)$:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;t = replicate(10000, median(sample(c(runif(1, 24, 32),  # id=1&#xA;                                     runif(1,  1,  8),  # id=2&#xA;                                     runif(1,  8, 16),  # id=3&#xA;                                     runif(1, 16, 24)), # id=4&#xA;                                   replace=TRUE)))&#xA;c(quantile(t, c(.025, .25, .5, .75, .975)), mean=mean(t), sd=sd(t))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Result (repeated):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    2.5%       25%       50%       75%     97.5%      mean        sd &#xA;4.602999 11.428310 16.005289 20.549056 28.378774 16.085808  6.243129 &#xA;4.517058 11.717245 16.084075 20.898324 28.031452 16.201022  6.219094 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Thus an approximation with 95% confidence interval of this median is 16 (5 – 28).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: See whuber's comment on the limitation of this method when the number of observations is small (including n=4 itself).&lt;/p&gt;&#xA;" OwnerUserId="3911" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-20T23:40:21.120" LastActivityDate="2011-04-20T23:40:21.120" CommentCount="7" />
  <row Id="9804" PostTypeId="1" CreationDate="2011-04-20T19:51:45.117" Score="1" ViewCount="64" Body="&lt;p&gt;hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;has anyone tried to use ARIMA within RapidMiner?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If yes, which blocks were used?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please do not give me as answer R.&lt;/p&gt;&#xA;" OwnerUserId="4261" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-04-20T21:39:02.490" LastActivityDate="2011-04-20T21:39:02.490" Title="ARIMA &amp; RapidMiner" Tags="&lt;arima&gt;&lt;rapidminer&gt;" CommentCount="1" />
  <row Id="9805" PostTypeId="2" ParentId="8511" CreationDate="2011-04-20T20:08:26.750" Score="2" ViewCount="" Body="&lt;p&gt;if deviance were proportional to log likelihood, and one uses the definition (see for example McFadden's &lt;a href=&quot;http://www.ats.ucla.edu/stat/mult_pkg/faq/general/psuedo_rsquareds.htm&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pseudo R^2 = 1 - L(model) / L(intercept)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;then the psuedo-R^2 above would be 1 - 198.63 / 958.66 = 0.7928&lt;/p&gt;&#xA;" OwnerUserId="2849" LastActivityDate="2011-04-20T20:08:26.750" CommunityOwnedDate="2011-04-20T20:08:26.750" />
  <row Id="9806" PostTypeId="2" ParentId="9797" CreationDate="2011-04-20T21:47:11.837" Score="0" ViewCount="" Body="&lt;p&gt;Assuming that you will have more data of the same structure you will be able to use the &lt;a href=&quot;http://en.wikipedia.org/wiki/Life_table&quot; rel=&quot;nofollow&quot;&gt;actuarial (life table) method&lt;/a&gt; to estimate median survival.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-04-20T21:56:31.867" LastActivityDate="2011-04-20T21:56:31.867" CommentCount="1" />
  <row Id="9807" PostTypeId="1" CreationDate="2011-04-20T22:57:58.130" Score="0" ViewCount="266" Body="&lt;p&gt;I performed a survey using a Likert 1 to 5 scale (totally agree/agree/neutral/ disagree/totally disagree) on 12 questions which are split into 3 statements which the respondent places a value of between 1 to 5 dependent on how much they agree or disagree - there are 36 statements in total.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Respondents:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Group 1                          Group 2&#xA;architects UK   140 replied      Architects US   100 replied&#xA;engineers UK    140 replied      Engineers US    100 replied&#xA;Contractors Uk  140 replied      Contractors US  100 replied&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The data is in Excel.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Questions:&lt;/h3&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;How do I input them in SPSS?&lt;/li&gt;&#xA;&lt;li&gt;How do I work out the frequency of replies for each recipient?&lt;/li&gt;&#xA;&lt;li&gt;How do I work out frequency of replies i.e agrees/disagrees etc for each group?&lt;/li&gt;&#xA;&lt;li&gt;How can I rank each individual question (12 of them)? Remember, there are 3 individual statements to each question.&lt;/li&gt;&#xA;&lt;li&gt;How do I compare UK architects to US architects to show congruence or not?&lt;/li&gt;&#xA;&lt;li&gt;How would show correlation between the two groups UK and US?&lt;/li&gt;&#xA;&lt;li&gt;Will SPSS develop graphs etc for me showing frequency or correlation?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="4262" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-22T08:01:01.770" LastActivityDate="2011-04-22T08:19:58.057" Title="Working with Likert scales in SPSS" Tags="&lt;spss&gt;&lt;likert&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="9808" PostTypeId="2" ParentId="8511" CreationDate="2011-04-20T23:21:07.663" Score="1" ViewCount="" Body="&lt;p&gt;Don't forget the &lt;a href=&quot;http://cran.r-project.org/web/packages/rms/index.html&quot; rel=&quot;nofollow&quot;&gt;rms&lt;/a&gt; package, by Frank Harrell. You'll find everything you need for fitting and validating GLMs. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a toy example (with only one predictor):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(101)&#xA;n &amp;lt;- 200&#xA;x &amp;lt;- rnorm(n)&#xA;a &amp;lt;- 1&#xA;b &amp;lt;- -2&#xA;p &amp;lt;- exp(a+b*x)/(1+exp(a+b*x))&#xA;y &amp;lt;- factor(ifelse(runif(n)&amp;lt;p, 1, 0), levels=0:1)&#xA;mod1 &amp;lt;- glm(y ~ x, family=binomial)&#xA;summary(mod1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This yields:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Coefficients:&#xA;            Estimate Std. Error z value Pr(&amp;gt;|z|)    &#xA;(Intercept)   0.8959     0.1969    4.55 5.36e-06 ***&#xA;x            -1.8720     0.2807   -6.67 2.56e-11 ***&#xA;---&#xA;Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 &#xA;&#xA;(Dispersion parameter for binomial family taken to be 1)&#xA;&#xA;    Null deviance: 258.98  on 199  degrees of freedom&#xA;Residual deviance: 181.02  on 198  degrees of freedom&#xA;AIC: 185.02&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, using the &lt;code&gt;lrm&lt;/code&gt; function,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(rms)&#xA;mod1b &amp;lt;- lrm(y ~ x)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You soon get a lot of model fit indices, including Nagelkerke $R^2$, with &lt;code&gt;print(mod1b)&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Logistic Regression Model&#xA;&#xA;lrm(formula = y ~ x)&#xA;&#xA;                      Model Likelihood     Discrimination    Rank Discrim.    &#xA;                         Ratio Test            Indexes          Indexes       &#xA;&#xA;Obs           200    LR chi2      77.96    R2       0.445    C       0.852    &#xA; 0             70    d.f.             1    g        2.054    Dxy     0.705    &#xA; 1            130    Pr(&amp;gt; chi2) &amp;lt;0.0001    gr       7.801    gamma   0.705    &#xA;max |deriv| 2e-08                          gp       0.319    tau-a   0.322    &#xA;                                           Brier    0.150                     &#xA;&#xA;&#xA;          Coef    S.E.   Wald Z Pr(&amp;gt;|Z|)&#xA;Intercept  0.8959 0.1969  4.55  &amp;lt;0.0001 &#xA;x         -1.8720 0.2807 -6.67  &amp;lt;0.0001 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here, $R^2=0.445$ and it is computed as $\left(1-\exp(-\text{LR}/n)\right)/\left(1-\exp(-(-2L_0)/n)\right)$, where LR is the $\chi^2$ stat (comparing the two nested models you described), whereas the denominator is just the max value for $R^2$. For a perfect model, we would expect $\text{LR}=2L_0$, that is $R^2=1$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;By hand,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; mod0 &amp;lt;- update(mod1, .~.-x)&#xA;&amp;gt; lr.stat &amp;lt;- lrtest(mod0, mod1)&#xA;&amp;gt; (1-exp(-as.numeric(lr.stat$stats[1])/n))/(1-exp(2*as.numeric(logLik(mod0)/n)))&#xA;[1] 0.4445742&#xA;&amp;gt; mod1b$stats[&quot;R2&quot;]&#xA;       R2 &#xA;0.4445742 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Ewout W. Steyerberg discussed the use of $R^2$ with GLM, in his book &lt;em&gt;Clinical Prediction Models&lt;/em&gt; (Springer, 2009, § 4.2.2 pp. 58-60). Basically, the relationship between the LR statistic and Nagelkerke's $R^2$ is approximately linear (it will be more linear with low incidence). Now, as discussed on the earlier thread I linked to in my comment, you can use other measures like the $c$ statistic which is equivalent to the AUC statistic (there's also a nice illustration in the above reference, see Figure 4.6).&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-20T23:27:35.910" LastActivityDate="2011-04-20T23:27:35.910" />
  <row Id="9809" PostTypeId="1" CreationDate="2011-04-20T23:35:57.413" Score="2" ViewCount="131" Body="&lt;p&gt;Does anyone know of a good resource listing known tricks (with examples?) for calculating closed form expressions from messy expectations? (e.g., moment generating function, law of iterated expectations, change of measure, etc.) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a different setting, I've found &lt;a href=&quot;http://www.duke.edu/~rnau/arimrule.htm&quot; rel=&quot;nofollow&quot;&gt;Summary of Rules for Identifying ARIMA Models&lt;/a&gt; tremendously helpful. I was hoping a list of rules-of-thumb like this would also exist for calculating expectations...right? Unfortunately, I'm not finding anything.  &lt;/p&gt;&#xA;" OwnerUserId="3577" LastActivityDate="2011-04-26T10:29:53.403" Title="A list of tricks for calculating expectations?" Tags="&lt;expected-value&gt;" AnswerCount="2" CommentCount="3" />
  <row Id="9810" PostTypeId="1" CreationDate="2011-04-20T23:49:40.123" Score="0" ViewCount="115" Body="&lt;p&gt;I want to compare if three groups are different by a non-parametric test. Now..., the problem is that two groups are paired (prior to treatment and one-year after treatment), and the (reference) group is other healthy individuals.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a simple solution? I am using R,&lt;/p&gt;&#xA;" OwnerUserId="4229" LastActivityDate="2011-04-20T23:49:40.123" Title="Which non-parametric test for difference between three groups, of which two are paired?" Tags="&lt;multiple-comparisons&gt;&lt;nonparametric&gt;" CommentCount="5" />
  <row Id="9811" PostTypeId="2" ParentId="5396" CreationDate="2011-04-20T23:57:53.813" Score="1" ViewCount="" Body="&lt;p&gt;for your first question:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;this should be possible with &lt;code&gt;Table&lt;/code&gt;/ &lt;code&gt;Create Table&lt;/code&gt;&lt;br&gt;&#xA;&lt;em&gt;(I'm using the german version of &lt;code&gt;JMP7&lt;/code&gt;, so I'm not sure how the menu commands are called in the english version)&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="3760" LastActivityDate="2011-04-20T23:57:53.813" />
  <row Id="9813" PostTypeId="1" CreationDate="2011-04-21T04:00:49.700" Score="4" ViewCount="81" Body="&lt;p&gt;Consider data where each observation was generated as follows.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;We draw $Z_1,...,Z_m$ from some distribution. (Possibly they're independent or related in some other simple way.)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Next, based on the $Z_1,...,Z_m$, we choose a sequence $0=I_0 &amp;lt; I_1 &amp;lt; ... &amp;lt; I_N=m$ so that, for each $k$, (i) $I_k-I_{k-1}$ is not too small and (ii) the sample variance within $Z_{I_{k-1}+1},...,Z_{I_k}$ is small. (I am intentionally somewhat vague here - I am open to making various different assumptions along these lines.)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;We generate the observed variables $X_1,...,X_N$ as $X_k=$ the average of $Z_{I_{k-1}+1},...,Z_{I_k}$.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;For example, the hidden sequence $Z=(0.1, 0.3, 0.2, 1.3, 1.2, 0.1)$ might lead to the observed sequence $X=(0.2, 1.25, 0.1)$ [or perhaps to $X=(0.2,0.86)$ due to (i) above].&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone here know whether this type of setup has been studied before, and if so, what are some keywords to search for or papers/books to look at?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance for any answers!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Added on Apr 21: The motivation is as follows. Think of each $Z$ sequence as SNP data from a single patient. In order to anonymize the data for a public release a procedure like the one I described above can be performed. Based on the anonymized data $X$, I want to predict survival and/or to identify SNPs that are relevant to survival.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that $I$, $N$, and $X$ are all functions of $Z$, so they will be different for each patient. Also note that that $I$'s are observed, i.e., I know which $Z_j$'s were averaged to produce each $X_k$.&lt;/p&gt;&#xA;" OwnerUserId="3891" LastEditorUserId="3891" LastEditorDisplayName="" LastEditDate="2011-04-21T13:57:43.063" LastActivityDate="2011-04-22T09:29:57.620" Title="Averages of random subsets of variables" Tags="&lt;modeling&gt;&lt;censoring&gt;" AnswerCount="2" CommentCount="3" FavoriteCount="1" />
  <row Id="9814" PostTypeId="1" CreationDate="2011-04-21T05:33:27.247" Score="1" ViewCount="60" Body="&lt;p&gt;Hello Dear Members,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I got some nice data on multi-environmet trials (MET) data for genotype evaluation and would like to use some new developed techniques as discussed in &lt;a href=&quot;http://journals.cambridge.org/action/displayFulltext?type=1&amp;amp;fid=365462&amp;amp;jid=AGS&amp;amp;volumeId=143&amp;amp;issueId=06&amp;amp;aid=365461&amp;amp;bodyId=&amp;amp;membershipNumber=&amp;amp;societyETOCSession=&quot; rel=&quot;nofollow&quot;&gt;Smith et. al 2005&lt;/a&gt; &lt;a href=&quot;http://scholar.google.com/scholar?hl=en&amp;amp;q=The%20analysis%20of%20crop%20cultivar%20breeding%20and%20evaluation%20trials%3a%20an%20overview%20of%20current%20mixed%20model%20approaches&amp;amp;btnG=Search&amp;amp;as_sdt=0,28&amp;amp;as_ylo=&amp;amp;as_vis=0&quot; rel=&quot;nofollow&quot;&gt;Smith et. al 2005&lt;/a&gt;. I'm specifically interested in &lt;strong&gt;Factor Analytic (FA) structure&lt;/strong&gt;. Authors mentioned codes for these methods will be available on request.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;There are now several statistical packages (including ASReml, GENSTAT, S-language packages and SAS; Littel et al. 1996) that allow REML estimation of a range of mixed models. The present authors have found the packages ASReml and GENSTAT and the samm functions (through S-language environments) to be the most suitable for the analysis of MET data, both in terms of the generality of models that can be fitted and the ease with which predictions and inference about varietal effects can be made. All models in the current paper are easily fitted and summarized using these software (code is available from the authors on request).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Even after multiple requests to the authors I haven't heard back from the authors. I'm wondering if someone has tried these models and be kind enough to share worked examples, that would be great. I'm looking forward for some positive response.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Smith et al. (2001b) used the following mixed model version of multiplicative models&lt;/p&gt;&#xA;&#xA;&lt;p&gt;${\small  \begin{eqnarray*}&#xD;&#xA;(\boldsymbol{I}_{e}\otimes\boldsymbol{I}_{g})\boldsymbol{\eta}_{eg\times1} &amp;amp; = &amp;amp; (\boldsymbol{1}_{e}\otimes\boldsymbol{1}_{g})\mu+(\boldsymbol{I}_{e}\otimes\boldsymbol{1}_{g})\boldsymbol{E}_{e\times1}+(\boldsymbol{1}_{e}\otimes\boldsymbol{I}_{g})\boldsymbol{G}_{g\times1}+\underbrace{(\boldsymbol{I}_{e}\otimes\boldsymbol{I}_{g})(\boldsymbol{GE})_{eg\times1}}\\&#xD;&#xA; &amp;amp; = &amp;amp; (\boldsymbol{1}_{e}\otimes\boldsymbol{1}_{g})\mu+(\boldsymbol{I}_{e}\otimes\boldsymbol{1}_{g})\boldsymbol{E}_{e\times1}+(\boldsymbol{1}_{e}\otimes\boldsymbol{I}_{g})\boldsymbol{G}_{g\times1}\\&#xD;&#xA; &amp;amp;  &amp;amp; +\underbrace{(\boldsymbol{\Lambda}_{E_{e\times k}}\otimes\boldsymbol{I}_{g})\:\boldsymbol{f}_{G_{kg\times1}}+(\boldsymbol{I}_{e}\otimes\boldsymbol{I}_{g})\delta_{eg\times1}}\end{eqnarray*}&#xD;&#xA;}{\tiny }$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\boldsymbol{\Lambda}_{E_{e\times k}}$&#xA;is a matrix of environment loadings, $f_{G_{kg\times1}}$ is the associated&#xA;vector of genotype scores and &lt;em&gt;k&lt;/em&gt; is the number of components (multiplicative terms) included in the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The authors assumed that the environments are fixed and $\boldsymbol{G}_{g\times1}$, $\boldsymbol{f}_{G_{kg\times1}}$, and $\delta_{eg\times1}$ are random effects with&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\small  \left(\begin{array}{l}&#xD;&#xA;\boldsymbol{G}_{g\times1}\\&#xD;&#xA;\boldsymbol{f}_{G_{kg\times1}}\\&#xD;&#xA;\boldsymbol{\delta}_{eg\times1}\end{array}\right)\sim\mathcal{N}\left(\begin{array}{ccc}&#xD;&#xA;\left[\begin{array}{c}&#xD;&#xA;\boldsymbol{0}\\&#xD;&#xA;\boldsymbol{0}\\&#xD;&#xA;\boldsymbol{0}\end{array}\right] &amp;amp; , &amp;amp; \left[\begin{array}{ccc}&#xD;&#xA;\sigma_{g}^{2}\,\boldsymbol{I}_{g} &amp;amp; \boldsymbol{0} &amp;amp; \boldsymbol{0}\\&#xD;&#xA;\boldsymbol{0} &amp;amp; \boldsymbol{I}_{k}\otimes\boldsymbol{I}_{g} &amp;amp; \boldsymbol{0}\\&#xD;&#xA;\boldsymbol{0} &amp;amp; \boldsymbol{0} &amp;amp; \boldsymbol{\Psi}_{e}\otimes\boldsymbol{I}_{g}\end{array}\right]\end{array}\right)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\small E((\boldsymbol{I}_{e}\otimes\boldsymbol{I}_{g})\boldsymbol{\eta}_{eg\times1})=(\boldsymbol{1}_{e}\otimes\boldsymbol{1}_{g})\mu+(\boldsymbol{I}_{e}\otimes\boldsymbol{1}_{g})\boldsymbol{E}_{e\times1}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;${\small \begin{eqnarray*}&#xD;&#xA;\mathrm{var}((\boldsymbol{I}_{e}\otimes\boldsymbol{I}_{g})\boldsymbol{\eta}_{eg\times1}) &amp;amp; = &amp;amp; \sigma_{g}^{2}\,(\boldsymbol{J}_{e}\otimes\boldsymbol{I}_{g})+(\boldsymbol{\Lambda}_{E_{e\times k}}\otimes\boldsymbol{I}_{g})(\boldsymbol{I}_{k}\otimes\boldsymbol{I}_{g})(\boldsymbol{\Lambda}_{E_{e\times k}}\otimes\boldsymbol{I}_{g})^{T}+\boldsymbol{\Psi}_{e}\otimes\boldsymbol{I}_{g}\\&#xD;&#xA; &amp;amp; = &amp;amp; \sigma_{g}^{2}\,(\boldsymbol{J}_{e}\otimes\boldsymbol{I}_{g})+(\boldsymbol{\Lambda}_{E_{e\times k}}\boldsymbol{\Lambda}_{E_{e\times k}}^{T}\otimes\boldsymbol{I}_{g})+\boldsymbol{\Psi}_{e}\otimes\boldsymbol{I}_{g}\\&#xD;&#xA; &amp;amp; = &amp;amp; \sigma_{g}^{2}\,(\boldsymbol{J}_{e}\otimes\boldsymbol{I}_{g})+(\boldsymbol{\Lambda}_{E_{e\times k}}\boldsymbol{\Lambda}_{E_{e\times k}}^{T}+\boldsymbol{\Psi}_{e})\otimes\boldsymbol{I}_{g}\\&#xD;&#xA; &amp;amp; = &amp;amp; (\underbrace{\sigma_{g}^{2}\,\boldsymbol{J}_{e}+\boldsymbol{\Lambda}_{E_{e\times k}}\boldsymbol{\Lambda}_{E_{e\times k}}^{T}}+\boldsymbol{\Psi}_{e})\otimes\boldsymbol{I}_{g}\\&#xD;&#xA; &amp;amp; = &amp;amp; (\underbrace{\boldsymbol{\Lambda}_{E}^{*}\boldsymbol{\Lambda}_{E}^{*T}}+\boldsymbol{\Psi}_{e})\otimes\boldsymbol{I}_{g}\end{eqnarray*}&#xD;&#xA;}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\boldsymbol{\Psi}_{e}$ is a diagonal $e\times e$ matrix with elements commonly referred to as specific variances.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="3903" LastEditorUserId="3903" LastEditorDisplayName="" LastEditDate="2011-04-30T08:44:11.827" LastActivityDate="2011-04-30T08:44:11.827" Title="Multiplicative mixed models for analyzing variety-by-environment data" Tags="&lt;r&gt;&lt;mixed-model&gt;&lt;factor-analysis&gt;&lt;experiment-design&gt;&lt;sas&gt;" CommentCount="7" />
  <row Id="9815" PostTypeId="2" ParentId="9801" CreationDate="2011-04-21T06:26:04.143" Score="9" ViewCount="" Body="&lt;p&gt;You can also use formulas from &lt;a href=&quot;http://matrixcookbook.com/&quot; rel=&quot;nofollow&quot;&gt;Matrix cookbook&lt;/a&gt;. We have&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$(y-X\beta)&amp;#39;(y-X\beta)=y&amp;#39;y-\beta&amp;#39;X&amp;#39;y-y&amp;#39;X\beta+\beta&amp;#39;X&amp;#39;X\beta$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now take derivatives of each term. You might want to notice that $\beta&amp;#39;X&amp;#39;y=y&amp;#39;X\beta$. The derivative of term $y&amp;#39;y$ with respect to $\beta$ is zero. The remaining term &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\beta&amp;#39;X&amp;#39;X\beta-2y&amp;#39;X\beta$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;is of form of function &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$f(x)=x&amp;#39;Ax+b&amp;#39;x,$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;in formula (88) in the book in page 11,  with $x=\beta$, $A=X&amp;#39;X$ and $b=-2X&amp;#39;y$. The derivative is given in the formula (89):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{\partial f}{\partial x}=(A+A&amp;#39;)x+b$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;so &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{\partial}{\partial \beta}(y-X\beta)&amp;#39;(y-X\beta)=(X&amp;#39;X+(X&amp;#39;X)&amp;#39;)\beta-2X&amp;#39;y$$  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now since $(X&amp;#39;X)&amp;#39;=X&amp;#39;X$ we get the desired solution:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$X&amp;#39;X\beta=X&amp;#39;y$$&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-04-21T06:26:04.143" CommentCount="2" />
  <row Id="9816" PostTypeId="2" ParentId="9794" CreationDate="2011-04-21T06:50:58.667" Score="4" ViewCount="" Body="&lt;p&gt;The cannonical way is probably &lt;a href=&quot;http://en.wikipedia.org/wiki/Mixed_data_sampling&quot; rel=&quot;nofollow&quot;&gt;MIDAS&lt;/a&gt; regression. There is a Matlab toolbox for estimating, available upon request from the &lt;a href=&quot;http://www.unc.edu/~eghysels/&quot; rel=&quot;nofollow&quot;&gt;author Eric Ghysels&lt;/a&gt;. You might look into &lt;a href=&quot;http://www.unc.edu/~eghysels/papers/MIDAS_Usersguide_Version1.pdf&quot; rel=&quot;nofollow&quot;&gt;user guide&lt;/a&gt; of this toolbox, since it has a review of all literature on MIDAS.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The wikipedia page also talks about connection with Kalman filters, so @F. Tussel observation is spot on.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-04-21T06:50:58.667" CommentCount="1" />
  <row Id="9817" PostTypeId="1" AcceptedAnswerId="9830" CreationDate="2011-04-21T08:56:58.870" Score="1" ViewCount="49" Body="&lt;p&gt;I am trying to use Maximum Likelihood Estimation to learn the structure of a DAG, G.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How is the number of free parameters of G calculated to compare the complexity of different graphical models?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it based on one of the following, or something else?&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Number of edges in the graph&lt;/li&gt;&#xA;&lt;li&gt;Maximum number of possible edges in the graph&lt;/li&gt;&#xA;&lt;li&gt;Maximum number of parents (children)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="3595" LastActivityDate="2011-04-21T15:07:33.127" Title="What is the number of free parameters for a directed acyclic graph?" Tags="&lt;bayesian&gt;&lt;graphical-model&gt;" AnswerCount="1" CommentCount="8" />
  <row Id="9818" PostTypeId="2" ParentId="9801" CreationDate="2011-04-21T09:35:00.280" Score="5" ViewCount="" Body="&lt;p&gt;One way which may help you understand is to not use matrix algebra, and differentiate with each respect to each component, and then &quot;store&quot; the results in a column vector.  So we have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{\partial}{\partial \beta_{k}}\sum_{i=1}^{N}\left(Y_{i}-\sum_{j=1}^{p}X_{ij}\beta_{j}\right)^{2}=0$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now you have $p$ of these equations, one for each beta.  This is a simple application of the chain rule:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\sum_{i=1}^{N}2\left(Y_{i}-\sum_{j=1}^{p}X_{ij}\beta_{j}\right)^{1}\left(\frac{\partial}{\partial \beta_{k}}\left[Y_{i}-\sum_{j=1}^{p}X_{ij}\beta_{j}\right]\right)=0$$&#xA;$$-2\sum_{i=1}^{N}X_{ik}\left(Y_{i}-\sum_{j=1}^{p}X_{ij}\beta_{j}\right)=0$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now we can re-write the sum inside the bracket as $\sum_{j=1}^{p}X_{ij}\beta_{j}=\bf{x}_{i}^{T}\boldsymbol{\beta}$  So you get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\sum_{i=1}^{N}X_{ik}Y_{i}-\sum_{i=1}^{N}X_{ik}\bf{x}_{i}^{T}\boldsymbol{\beta}=0$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now we have $p$ of these equations, and we will &quot;stack them&quot; in a column vector.  Notice how $X_{ik}$ is the only term which depends on $k$, so we can stack this into the vector $\bf{x}_{i}$ and we get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\sum_{i=1}^{N}\bf{x}_{i}\rm{Y}_{i}=\sum_{i=1}^{N}\bf{x}_{i}\bf{x}_{i}^{T}\boldsymbol{\beta}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now we can take the beta outside the sum (but must stay on RHS of sum), and then take the invervse:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\left(\sum_{i=1}^{N}\bf{x}_{i}\bf{x}_{i}^{T}\right)^{-1}\sum_{i=1}^{N}\bf{x}_{i}\rm{Y}_{i}=\boldsymbol{\beta}$$&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-04-21T09:35:00.280" />
  <row Id="9819" PostTypeId="2" ParentId="9779" CreationDate="2011-04-21T09:39:02.513" Score="5" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;No Model&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Wayne points out that you do not say anything about the process generating the data.  In particular, you don't say whether the quantity you are tracking, the 'state' is fixed or moving. If it's fixed - the case Wayne considers - then you may as well keep a running average of all the observations and hope for the best because that's all a full-blown state space model with state estimated by KF would do for you anyway.  If it's moving then you need to ask yourself how it moves.  Is it a random walk?  Is it constantly increasing?  Does it have cyclical or other recurring structure?  You need those assumptions to define the state space model for which the KF supplies state estimates.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Off by 2&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you say 'let us say that measurements can be off by 2 points' you may think you are making things easier, either to explain or implement, but you aren't.  If we take the 'off by 2' idea literally, then Kalman filtering cannot do what you want (although I suppose it might possibly approximate it).  This is because the KF assumes your observations are conditionally Normally distributed. Your measurement error assumption would instead be Uniform.  This will lead to incorrect inferences about the state if you apply the KF directly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Thinking About the Problem Statistically&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You ask 'how do I partition the data such that I get only distinct measurements'.  That's a good question if all your data are either good measurements or bad ones.  However, when considering KF we assume rather that &lt;em&gt;all&lt;/em&gt; measurements have some error, about which we have a small theory - the state space model - that contains a sub-model of this error and another sub-model of the evolution of a underlying state that generates the measurements.  The KF makes inferences on the basis of this theory.  Consequently, in this framework you don't privilege any measurements but rather look to the estimated state for your answers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Suggestion&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you don't feel like or simply cannot specify as much detail as necessary for a complete state space model to which you can apply a KF, it might be better to back off to more 'empirical' (and easier to implement) methods based on smoothing and weighted averages of recent data.  Exponential smoothing might be a helpful place to start, e.g. as described fairly clearly here: &lt;a href=&quot;http://www.duke.edu/~rnau/411avg.htm&quot; rel=&quot;nofollow&quot;&gt;http://www.duke.edu/~rnau/411avg.htm&lt;/a&gt;  This approach has quite close connections to KF approaches, so you could return to them easily if necessary.&lt;/p&gt;&#xA;" OwnerUserId="1739" LastActivityDate="2011-04-21T09:39:02.513" />
  <row Id="9820" PostTypeId="1" AcceptedAnswerId="9823" CreationDate="2011-04-21T12:42:03.650" Score="1" ViewCount="61" Body="&lt;p&gt;Assume I have a trading system that I'm evaluating over a three-year period. The returns are 25%, -40% and 25%. Empirically, I can see that this system loses because at the end of three years, I have less than when I started. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Wikipedia defines &lt;code&gt;expected return&lt;/code&gt; as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;E(R)= Sum: probability (in scenario i) * the return (in scenario i)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If we insert our values into this formula, we get the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;E(R) = (.33 * .25) + (.33 * -.40) + (.33 * .25) = .033&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So a positive expected return for a system that loses every time, no matter if you re-arrange the order of the returns. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What's wrong here?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To further explain how the game works, consider an initial start value of 100. What can you expect at the end of the game? There are no re-investments, no withdrawls, no dividends, no broker fees nor SEC fees. It is a simple game. Here is some R code to illustrate the game.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;first  &amp;lt;- c(.25, .25, -.4)&#xA;second &amp;lt;- c(-.4, .25, .25)&#xA;third  &amp;lt;- c(.25, -.4, .25)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Pass any of the above sequence of returns into this function:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;game &amp;lt;- function(x){&#xA;&#xA;start &amp;lt;- 100&#xA;&#xA;for(i in 1:NROW(x))&#xA;&#xA;start &amp;lt;- start + start*x[i]&#xA;&#xA;return(start)   &#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;NOTE: I asked a similar question on quantexchange, but I'm interested here in the math behind the expected return equation. &lt;/p&gt;&#xA;" OwnerUserId="3306" LastEditorUserId="3306" LastEditorDisplayName="" LastEditDate="2011-04-21T13:53:43.563" LastActivityDate="2011-04-21T13:53:43.563" Title="Calculating expected return" Tags="&lt;expected-value&gt;" AnswerCount="2" />
  <row Id="9821" PostTypeId="2" ParentId="9813" CreationDate="2011-04-21T12:43:00.117" Score="2" ViewCount="" Body="&lt;p&gt;You could apply linear regression with regularization (&lt;a href=&quot;http://www-stat.stanford.edu/~tibs/lasso.html&quot; rel=&quot;nofollow&quot;&gt;Lasso&lt;/a&gt;) to solve this problem. The idea would be to fit the data with piece-wise constant functions adding a penalty for every jump that occurs. The objective you have to minimize is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$x^* = \arg\min_{x\in\mathbb{R}^m} \|z - x\|_2^2 + \lambda \|\nabla x\|_1$,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $(\nabla x)_i = x_{i-1} - x_{i}$ is the (backward) difference operator on the grid. The parameter $\lambda$ controls the tradeoff between small- vs. large intervals and low vs. high variance within the intervals.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the solution $x^*$ you can then reconstruct the intervals $I$.&lt;/p&gt;&#xA;" OwnerUserId="4272" LastActivityDate="2011-04-21T12:43:00.117" CommentCount="2" />
  <row Id="9822" PostTypeId="2" ParentId="9820" CreationDate="2011-04-21T13:25:33.207" Score="1" ViewCount="" Body="&lt;p&gt;Wikipedia is right. So are your calculations: this system should win.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However: it also assumes that your investments in each part are equal in size (because you give equal probability 1/3 to each). If this is not true, that may explain the difference with your empirical observations (perhaps you should share the numbers with us on those). e.g. If you invested twice as much in the part that has a negative return than in the two other parts, the probabilities become 1/4,1/2,1/4 and you'd have a losing system.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is also ignoring any extra costs, so if these extra costs are bigger than your meager (?) 1/30 return, this could be another explanation (I'm not familiar with the practicalities of trading).&lt;/p&gt;&#xA;" OwnerUserId="4257" LastActivityDate="2011-04-21T13:25:33.207" CommentCount="2" />
  <row Id="9823" PostTypeId="2" ParentId="9820" CreationDate="2011-04-21T13:39:55.973" Score="1" ViewCount="" Body="&lt;p&gt;The difference between the two ways to look at the return is whether you are reinvesting your gains. If you start with 100 dollars and reinvest all of the money the next year, your balances will be: 125, 75, 93.75 dollars - you lost money. However if you invest 100 dollars every year, then you get +25 - 40 +25 = +10 dollars gain. That's one way you could think about the Wikipedia formula. A better way to interpret it, though, is that you are investing in one of the three years chosen at random, and it gives you the expected return.&lt;/p&gt;&#xA;" OwnerUserId="279" LastActivityDate="2011-04-21T13:39:55.973" CommentCount="5" />
  <row Id="9824" PostTypeId="2" ParentId="9801" CreationDate="2011-04-21T14:10:55.193" Score="4" ViewCount="" Body="&lt;p&gt;Here is a technique for minimizing the sum of squares in regression that actually has applications to more general settings and which I find useful. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Let's try to avoid vector-matrix calculus altogether.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose we are interested in minimizing &#xA;$$&#xD;&#xA;\newcommand{\err}{\mathcal{E}}\newcommand{\my}{\mathbf{y}}\newcommand{\mX}{\mathbf{X}}\newcommand{\bhat}{\hat{\beta}}\newcommand{\reals}{\mathbb{R}}&#xD;&#xA;\err = (\my - \mX \beta)^T (\my - \mX \beta) = \|\my - \mX \beta\|_2^2 \&amp;gt; ,&#xD;&#xA;$$&#xA;where $\my \in \reals^n$, $\mX \in \reals^{n\times p}$ and $\beta \in \reals^p$. We assume for simplicity that $p \leq n$ and $\mathrm{rank}(\mX) = p$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For any $\bhat \in \reals^p$, we get&#xA;$$&#xD;&#xA;\err = \|\my - \mX \bhat + \mX \bhat - \mX \beta\|_2^2 = \|\my - \mX \bhat\|_2^2 + \|\mX(\beta-\bhat)\|_2^2 - 2(\beta - \bhat)^T \mX^T (\my - \mX \bhat) \&amp;gt;.&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we can &lt;strong&gt;choose&lt;/strong&gt; (find!) a vector $\bhat$ such that the last term on the right-hand side is zero for &lt;em&gt;every&lt;/em&gt; $\beta$, then we would be done, since that would imply that $\min_\beta \err \geq \|\my - \mX \bhat\|_2^2$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But, $(\beta - \bhat)^T \mX^T (\my - \mX \bhat) = 0$ for all $\beta$ if and only if $\mX^T (\my - \mX \bhat) = 0$ and this last equation is true if and only if $\mX^T \mX \bhat = \mX^T \my$. So $\err$ is minimized by taking $\bhat = (\mX^T \mX)^{-1} \mX^T \my$.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;While this may seem like a &quot;trick&quot; to avoid calculus, it actually has wider application and there is some interesting geometry at play.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One example where this technique makes a derivation &lt;em&gt;much&lt;/em&gt; simpler than any matrix-vector calculus approach is when we generalize to the matrix case. Let $\newcommand{\mY}{\mathbf{Y}}\newcommand{\mB}{\mathbf{B}}\mY \in \reals^{n \times p}$, $\mX \in \reals^{n \times q}$ and $\mB \in \reals^{q \times p}$. Suppose we wish to minimize&#xA;$$&#xD;&#xA;\err = \mathrm{tr}( (\mY - \mX \mB) \Sigma^{-1} (\mY - \mX \mB)^T )&#xD;&#xA;$$&#xA;over the entire &lt;em&gt;matrix&lt;/em&gt; $\mB$ of parameters. Here $\Sigma$ is a covariance matrix.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An entirely analogous approach to the above quickly establishes that the minimum of $\err$ is attained by taking&#xA;$$&#xD;&#xA;\hat{\mB} = (\mX^T \mX)^{-1} \mX^T \mY \&amp;gt;.&#xD;&#xA;$$&#xA;That is, in a regression setting where the response is a &lt;em&gt;vector&lt;/em&gt; with covariance $\Sigma$ and the observations are independent, then the OLS estimate is attained by doing $p$ separate linear regressions on the components of the response.&lt;/p&gt;&#xA;" OwnerUserId="2970" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-04-21T14:32:57.543" LastActivityDate="2011-04-21T14:32:57.543" CommentCount="8" />
  <row Id="9825" PostTypeId="1" CreationDate="2011-04-21T14:12:41.697" Score="8" ViewCount="106" Body="&lt;p&gt;I have some data that is highly correlated. If I run a linear regression I get a regression line with a slope close to one (= 0.93). What I'd like to do is test if this slope is significantly different from 1.0. My expectation is that it is not. In other words, I'd like to change the null hypothesis of the linear regression from a slope of zero to a slope of one. Is this a sensible approach? I'd also really appreciate it you could include some R code in your answer so I could implement this method (or a better one you suggest!).  Thanks.&lt;/p&gt;&#xA;" OwnerUserId="4274" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-21T14:16:00.223" LastActivityDate="2011-04-21T15:08:36.183" Title="Changing null hypothesis in linear regression" Tags="&lt;regression&gt;&lt;correlation&gt;&lt;hypothesis-testing&gt;" AnswerCount="4" FavoriteCount="2" />
  <row Id="9826" PostTypeId="2" ParentId="7505" CreationDate="2011-04-21T14:16:03.350" Score="1" ViewCount="" Body="&lt;p&gt;When you move from testing a difference in the conversion rate to testing the difference in volume, the characteristics of your underlying data is changing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The conversion rates you were testing were proportions based on the number of &quot;successes&quot; out of the number of &quot;trials&quot; which meant their intervals were [0,1]. That means those rates can be approximated to binomial distributions. As such, the sample variance/standard deviation required for confidence intervals and hypothesis testing could be simplified to a term related to the probability of a conversion, (p*(1-p)).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, comparing two sales volumes which would not lie on the [0,1] interval would mean you would not be using binomials. Therefore you would want to testing the difference between two means rather than two proportions. You would need to define the means you wish to test based on the business question you wanted to answer (for example: is the the test daily average of revenue, visitors, or some other metric statistically significant from the control?).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because the data you are using is not on the [0,1] interval, your variance will not be limited in the same manner as it is when using proportions. Therefore, you will need to  find the sample variances of the two groups using whatever data you were using to construct the means (both the data and the time period). Depending on the circumstance of your data acquisition, you may be able to continue to use excel or it may better to compute your variances through another package.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming the two groups have the same sample sizes, you should be able to average the two variances together to find the variance that would be used in either the confidence interval or the test statistic calculation.&lt;/p&gt;&#xA;" OwnerUserId="4260" LastEditorUserId="4260" LastEditorDisplayName="" LastEditDate="2011-04-21T15:05:07.710" LastActivityDate="2011-04-21T15:05:07.710" CommentCount="1" />
  <row Id="9827" PostTypeId="2" ParentId="9825" CreationDate="2011-04-21T14:25:14.817" Score="4" ViewCount="" Body="&lt;pre&gt;&lt;code&gt;set.seed(20); y = rnorm(20); x = y + rnorm(20, 0, 0.2) # generate correlated data&#xA;summary(lm(y ~ x))                  # original model&#xA;summary(lm(y ~ x, offset= 1.00*x))  # testing against slope=1&#xA;summary(lm(y-x ~ x))                # testing against slope=1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Outputs:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;            Estimate Std. Error t value Pr(&amp;gt;|t|)    &#xA;(Intercept)  0.01532    0.04728   0.324     0.75    &#xA;x            0.91424    0.04128  22.148 1.64e-14 ***&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&amp;nbsp;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;            Estimate Std. Error t value Pr(&amp;gt;|t|)  &#xA;(Intercept)  0.01532    0.04728   0.324   0.7497  &#xA;x           -0.08576    0.04128  -2.078   0.0523 .&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&amp;nbsp;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;            Estimate Std. Error t value Pr(&amp;gt;|t|)  &#xA;(Intercept)  0.01532    0.04728   0.324   0.7497  &#xA;x           -0.08576    0.04128  -2.078   0.0523 .&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="3911" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-21T14:38:38.310" LastActivityDate="2011-04-21T14:38:38.310" CommentCount="1" />
  <row Id="9828" PostTypeId="2" ParentId="9825" CreationDate="2011-04-21T14:43:27.883" Score="3" ViewCount="" Body="&lt;p&gt;Your hypothesis can be expressed as $R\beta=r$ where $\beta$ is your regression coefficients and $R$ is restriction matrix with $r$ the restrictions. If our model is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$y=\beta_0+\beta_1x+u$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;then for hypothesis $\beta_1=0$,  $R=[0,1]$ and $r=1$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For these type of hypotheses you can use &lt;code&gt;linearHypothesis&lt;/code&gt; function from package &lt;strong&gt;car&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(20); y = rnorm(20); x = y + rnorm(20, 0, 0.2) # generate correlated data&#xA;mod &amp;lt;- lm(y ~ x))                  # original model&#xA;&#xA;&#xA;&amp;gt; linearHypothesis(mod,matrix(c(0,1),nrow=1),rhs=c(1))&#xA;Linear hypothesis test&#xA;&#xA;Hypothesis:&#xA;x = 1&#xA;&#xA;Model 1: restricted model&#xA;Model 2: y ~ x&#xA;&#xA;  Res.Df     RSS Df Sum of Sq      F  Pr(&amp;gt;F)  &#xA;1     19 0.96022                              &#xA;2     18 0.77450  1   0.18572 4.3162 0.05234 .&#xA;---&#xA;Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-04-21T14:43:27.883" />
  <row Id="9829" PostTypeId="1" CreationDate="2011-04-21T14:52:56.543" Score="0" ViewCount="54" Body="&lt;h3&gt;Sample&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;n = 60&lt;/li&gt;&#xA;&lt;li&gt;10 men and 10 females from each of three job positions.&lt;/li&gt;&#xA;&lt;li&gt;Annual salary is recorded for each participant&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;Question&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;How can I test whether there is a statistically significant effect ($\alpha = .01$) of position and gender on salary?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="4276" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-22T11:00:55.157" LastActivityDate="2011-04-22T11:00:55.157" Title="Testing the effect of position and gender on salary" Tags="&lt;statistical-significance&gt;&lt;homework&gt;" CommentCount="8" />
  <row Id="9830" PostTypeId="2" ParentId="9817" CreationDate="2011-04-21T15:07:33.127" Score="1" ViewCount="" Body="&lt;p&gt;The answer depends on your likelihood for the data $X$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Joint Gaussian&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can fit the model by sequentially fitting the conditional distribution of each node $v$ given its parents $\mathrm{pa}(v)$, in this case:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$X_v | X_{\mathrm{pa}(v)} \sim N\big(\mu_v + \beta_v^\top [X_{\mathrm{pa}(v)} - \mu_{\mathrm{pa}(v)}], \sigma_v^2 \big)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then for each node you need 1 parameter $\mu_v$ for the mean, 1 parameter $\sigma_v^2$ for the conditional variance and a vector $\beta_v$ of length $|\mathrm{pa}(v)|$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the total number of parameters needed for a graph $G=(V,E)$ is $2|V| + |E|$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Discrete&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose the variable $X_v$ for each node $v$ is discrete with $n_v$ possible outcomes. Then for each possible outcome in the parent space, you require $n_v-1$ parameters. So for each node, you will need:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$(n_v-1) \prod_{u \in \mathrm{pa}(v)} n_u$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;parameters. If each node has 2 possible outcomes, you will need:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\sum_{v} 2^{|\mathrm{pa}(v)|}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;total parameters, unless you make some simplifying assumptions, such as proportional-odds (i.e. logistic regression).&lt;/p&gt;&#xA;" OwnerUserId="495" LastActivityDate="2011-04-21T15:07:33.127" />
  <row Id="9831" PostTypeId="2" ParentId="9825" CreationDate="2011-04-21T15:08:24.687" Score="1" ViewCount="" Body="&lt;p&gt;The point of testing is that you want to reject your null hypothesis, not confirm it. The fact that there is no significant difference, is in no way a proof of the absence of a significant difference. For that, you'll have to define what effect size you deem reasonable to reject the null.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Testing whether your slope is significantly different from 1 is not that difficult, you just test whether the difference $slope - 1$ differs significantly from zero. By hand this would be something like :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(20); y = rnorm(20); x = y + rnorm(20, 0, 0.2)&#xA;model &amp;lt;- lm(y~x)&#xA;&#xA;coefx &amp;lt;- coef(summary(model))[2,1]&#xA;seslope &amp;lt;- coef(summary(model))[2,2]&#xA;DF &amp;lt;- model$df.residual&#xA;&#xA;# normal test&#xA;p &amp;lt;- (1 - pt(coefx/seslope,DF) )*2&#xA;# test whether different from 1&#xA;p2 &amp;lt;- (1 - pt(abs(coefx-1)/seslope,DF) )*2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now you should be aware of the fact that the effect size for which a difference becomes significant, is &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; qt(0.975,DF)*seslope&#xA;[1] 0.08672358&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;provided that we have a decent estimator of the standard error on the slope. Hence, if you decide that a significant difference should only be detected from 0.1, you can calculate the necessary DF as follows : &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;optimize(&#xA;    function(x)abs(qt(0.975,x)*seslope - 0.1),&#xA;    interval=c(5,500)&#xA;) &#xA;$minimum&#xA;[1] 6.2593&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Mind you, this is pretty dependent on the estimate of the seslope. To get a better estimate on seslope, you could do a resampling of your data. A naive way would be :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;n &amp;lt;- length(y)&#xA;seslope2 &amp;lt;-&#xA;  mean(&#xA;    replicate(n,{&#xA;      id &amp;lt;- sample(seq.int(n),1)&#xA;      model &amp;lt;- lm(y[-id]~x[-id])&#xA;      coef(summary(model))[2,2]&#xA;    })&#xA;  )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;putting seslope2 in the optimization function, returns :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;$minimum&#xA;[1] 6.954609&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;All this will tell you that your dataset will return a significant result faster than you deem necessary, and that you only need 7 degrees of freedom (in this case 9 observations) if you want to be sure that non-significant means what you want it means.&lt;/p&gt;&#xA;" OwnerUserId="1124" LastActivityDate="2011-04-21T15:08:24.687" />
  <row Id="9832" PostTypeId="2" ParentId="9825" CreationDate="2011-04-21T15:08:36.183" Score="4" ViewCount="" Body="&lt;p&gt;It seems you're still trying to reject a null hypothesis.  There are loads of problems with that, not the least of which is that it's possible that you don't have enough power to see that you're different from 1.  It sounds like you don't care that the slope is 0.07 different from 1.  But what if you can't really tell?  What if you're actually estimating a slope that varies wildly and may actually be quite far from 1 with something like a confidence interval of ±0.4.  Your best tactic here is not changing the null hypothesis but actually speaking reasonably about an interval estimate.  If you apply the command confint() to your model you can get a 95% confidence interval around your slope.  Then you can use this to discuss the slope you did get.  If 1 is within the confidence interval you can state that it is within the range of values you believe likely to contain the true value.  But more importantly you can also state what that range of values is.&lt;/p&gt;&#xA;" OwnerUserId="601" LastActivityDate="2011-04-21T15:08:36.183" />
  <row Id="9833" PostTypeId="1" CreationDate="2011-04-21T15:11:29.673" Score="1" ViewCount="142" Body="&lt;p&gt;In my elementary statistics course, I learnt how to construct 95% confidence interval such as population mean, mu based on &lt;strong&gt;asymptotic normality&lt;/strong&gt; for &quot;large&quot; sample size. Apart from &lt;strong&gt;resampling methods&lt;/strong&gt;, there is another approach based on &lt;strong&gt;&quot;profile likelihood&quot;&lt;/strong&gt;. Could someone elucidate this approach? Under what situations, the constructed 95% CI based on asymptotic normality and profile likelihood  are comparable? I could not find any references on this topic, any suggested references, please? Why isn't it more widely used?&lt;/p&gt;&#xA;" OwnerUserId="2264" LastEditorUserId="2264" LastEditorDisplayName="" LastEditDate="2011-04-21T16:54:39.407" LastActivityDate="2011-05-21T17:48:47.173" Title="Constructing 95% confidence interval based on profile likelihood" Tags="&lt;r&gt;&lt;confidence-interval&gt;&lt;mathematical-statistics&gt;&lt;monte-carlo&gt;&lt;simulation&gt;" AnswerCount="1" />
  <row Id="9835" PostTypeId="1" AcceptedAnswerId="9847" CreationDate="2011-04-21T16:31:04.693" Score="2" ViewCount="80" Body="&lt;p&gt;Do there exist any systems for symbolically solving expectations?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is sort of a follow-up to my question &lt;a href=&quot;http://stats.stackexchange.com/q/9809/3577&quot;&gt;List of Tricks for Solving Messy Expectations?&lt;/a&gt;  Basically, I'm looking for ways to solve a messy expectation after I've exhausted all obvious routes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT: BACKGROUND&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm trying to solve the following for $\alpha$ (constrained to be greater than 0) as a function of $\sigma_X^2$, $\sigma_Y^2$, and $p$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$E\left[\ln(F) F^\alpha X^2 (X + Y)^2 + \ln(F) F^{2\alpha}(X + Y)^4\right] = 0$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Y \sim N(0,\sigma_Y^2)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ X = \left\{ \begin{array}{cc} &#xD;&#xA;           N(0,\sigma_X^2) &amp;amp; p \\ &#xD;&#xA;           0 &amp;amp; (1-p)&#xD;&#xA;          \end{array}\right.$ where it's assumed $\sigma_X^2 \gg \sigma_Y^2$ and that $p$ is very small. (i.e., $X$ is a jump process with most of its mass at 0)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$F = F_{|Z|}(|X+Y|)$ where $Z \sim N(0,\sigma_Y^2)$  (i.e., $F$ is the CDF for the absolute value of a normal)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT: EVEN MORE BACKGROUND&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The equation I'm trying to solve above is the first order condition for the min-MSE problem:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\min_{\alpha &amp;gt; 0} \left(X^2 - \widehat{X^2}\right)^2 $ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $ \widehat{X^2} = F_{|Z|}\left(|R|\right)^{\alpha}R^2$ and $R = X + Y\,$ is the only observed variable. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically, I'm trying to estimate the square of the jump, $X^2$ (given that I can only observe the aggregated process $R$) by smoothing down $R^2$. If $|R|$ is large, the smoothing function $F_{|Z|}\left(|R|\right)^{\alpha}$ should be close to 1 and the estimate of $X^2$ would be close to $R^2$. If $|R|$ is small, the estimate of $X^2$ would be close to 0. &lt;/p&gt;&#xA;" OwnerUserId="3577" LastEditorUserId="3577" LastEditorDisplayName="" LastEditDate="2011-04-27T18:52:38.367" LastActivityDate="2011-04-27T18:52:38.367" Title="Systems for symbolically solving expectations?" Tags="&lt;expected-value&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="9836" PostTypeId="1" CreationDate="2011-04-21T16:44:01.063" Score="1" ViewCount="62" Body="&lt;p&gt;Can somebody give me references (book/online resource) on using R for Marketing Mix Modelling?&lt;/p&gt;&#xA;" OwnerUserId="4278" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-21T17:22:25.290" LastActivityDate="2011-04-21T17:22:25.290" Title="Market mix modelling with R" Tags="&lt;r&gt;&lt;books&gt;" CommentCount="2" />
  <row Id="9837" PostTypeId="2" ParentId="9833" CreationDate="2011-04-21T17:12:45.203" Score="1" ViewCount="" Body="&lt;p&gt;In general, the confidence interval based on the standard error strongly depends on the assumption of normality for the estimator. The &quot;profile likelihood confidence interval&quot; provides an alternative.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am pretty sure you can find documentation for this. For instance, &lt;a href=&quot;http://people.upei.ca/hstryhn/stryhn208.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; and references therein.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a brief overview.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let us say the data depend upon two (vectors of) parameters, $\theta$ and $\delta$, where $\theta$ is of interest and $\delta$ is a nuisance parameter. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The profile likelihood of $\theta$ is defined by&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$L_p(\theta) = \max_{\delta} L(\theta, \delta)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $L(\theta, \delta)$ is the 'complete likelihood'. $L_p(\theta)$ does no longer depend on $\delta$ since it has been profiled out.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let a null hypothesis be $H_0 : \theta = \theta_0$ and the likelihood ratio statistic be&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$LR = 2 (\log L_p(\hat{\theta}) - \log L_p(\theta_0))$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\hat{\theta}$ is the value of $\theta$ that maximises the profile likelihood $L_p(\theta)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A &quot;profile likelihood confidence interval&quot; for $\theta$ consists of those values $\theta_0$ for which the test is not significant.&lt;/p&gt;&#xA;" OwnerUserId="3019" LastEditorUserId="3019" LastEditorDisplayName="" LastEditDate="2011-04-21T17:34:22.940" LastActivityDate="2011-04-21T17:34:22.940" CommentCount="1" />
  <row Id="9838" PostTypeId="2" ParentId="8784" CreationDate="2011-04-21T17:48:27.180" Score="3" ViewCount="" Body="&lt;p&gt;Why not? The models are estimating how much 1 unit of change in any model predictor will influence the probability of &quot;1&quot; for the outcome variable. I'll assume the models are the same-- that they have the same predictors in them. The most informative way to compare the relative magnitudes of any given predictor in the 2 models is to use the models to calculate (either deterministically or better by simulation) how much some meaningful increment of change (e.g., +/- 1 SD) in the predictor affects the probabilities of the respective outcome variables--&amp;amp; compare them! You'll want to determine confidence intervals for the two estimates as well as so you can satisfy yourself that the difference is &quot;significant,&quot; practically &amp;amp; statistically. &lt;/p&gt;&#xA;" OwnerUserId="2347" LastEditorUserId="2347" LastEditorDisplayName="" LastEditDate="2011-04-22T23:30:14.527" LastActivityDate="2011-04-22T23:30:14.527" CommentCount="2" />
  <row Id="9839" PostTypeId="1" CreationDate="2011-04-21T17:49:53.597" Score="2" ViewCount="78" Body="&lt;p&gt;Hi, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am trying to write a prediction algorithm for a set of temperature data. I settled on Holt-Winters since it seemed to be a simple time series prediction algorithm and I can easily code it up in python to understand what is going on with it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I am plotting the smoothing function as it learns this is how it looks. As you can see, it follows the original curve pretty well. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/bZtGz.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But when I try to plot a future curve for one year (365 days) -- it really falls down and dies. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/7mwJP.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And to me intuitively it makes sense why it dies like that. Because if you see the last prediction equation of Holt-Winters it really only makes use of the very very last point in both the curve smoothing and the trend smoothing. And we know that exponential smoothing has a very short memory because of the whole exponential thing. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I am wondering how does one actually go about using Holt-Winters for prediction (specifically for 365 day seasonal data like this).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you know of any other methods which I can look at for this domain (temperature prediction) please let me know. I come from python, so it would be very useful if you can point me to resources such as libraries etc to get the job done. &lt;/p&gt;&#xA;" OwnerUserId="4280" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-21T20:03:30.900" LastActivityDate="2011-04-21T20:03:30.900" Title="Predicting temperature time series with Holt-Winters" Tags="&lt;time-series&gt;" AnswerCount="1" />
  <row Id="9840" PostTypeId="2" ParentId="9835" CreationDate="2011-04-21T18:26:17.807" Score="1" ViewCount="" Body="&lt;p&gt;My favourite and free symbolic algebra system is &lt;a href=&quot;http://www.sagemath.org/&quot; rel=&quot;nofollow&quot;&gt;Sage&lt;/a&gt;, currently available as a Linux installer and via a &lt;a href=&quot;http://www.sagenb.org/&quot; rel=&quot;nofollow&quot;&gt;web interface&lt;/a&gt;. It is powerful, but I haven't tested how good it is in solving expectations.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-21T22:08:27.317" LastActivityDate="2011-04-21T22:08:27.317" CommentCount="2" />
  <row Id="9841" PostTypeId="1" AcceptedAnswerId="9902" CreationDate="2011-04-21T18:42:44.193" Score="4" ViewCount="70" Body="&lt;p&gt;We have data on the day in which a butterfly pupates (forms a cocoon) in the summer/fall of 2 different pairs of years (2005-2006 vs 2009-2010).  At the time that the pupa forms it can either be in diapause (suspended animation) or not.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Individual butterfly caterpillars were randomly selected from a wild population and observed in outdoor enclosures until they pupated.  At this time the pupa was determined to be either in diapause or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We would like to know if there is a shift in the probability that a butterfly pupa will be in diapause or not on a given day between years.  In other words, does a butterfly pupa have a significantly different probability of being in diapause later (or earlier) in 2005 vs 2006?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We have analyzed each pair of years using separate multiple logistic regressions with Day of pupation and Year as factors and are interpreting a significant Year effect as a significant shift in diapause probability on a given day between years. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this the appropriate way to approach this problem?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data Format:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Individual   Diapause Pupation_Day Year&#xA;1            1          200        2005&#xA;2            0          201        2005&#xA;.            .           .          .&#xA;.            .           .          .&#xA;.            .           .          . &#xA;n            1          300        2006&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="4048" LastEditorUserId="4048" LastEditorDisplayName="" LastEditDate="2011-04-21T20:11:02.477" LastActivityDate="2011-04-23T11:22:22.667" Title="What is the appropriate way to test for a shift in probability using multiple logistic regression?" Tags="&lt;logistic-regression&gt;" AnswerCount="1" CommentCount="8" />
  <row Id="9842" PostTypeId="1" AcceptedAnswerId="9869" CreationDate="2011-04-21T19:36:55.910" Score="4" ViewCount="125" Body="&lt;p&gt;Hi, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need some resources to get started on using neural networks for time series forecasting. I am wary of implementing some paper and then finding out that they have greatly over stated the potential of their methods. So if you have experience with the methods you are suggesting it will be even more awesome. &lt;/p&gt;&#xA;" OwnerUserId="4280" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-21T20:07:19.323" LastActivityDate="2011-04-22T13:18:17.590" Title="Getting started with neural networks for forecasting" Tags="&lt;time-series&gt;&lt;books&gt;&lt;neural-networks&gt;" AnswerCount="2" CommentCount="6" FavoriteCount="2" />
  <row Id="9843" PostTypeId="2" ParentId="9839" CreationDate="2011-04-21T19:40:45.750" Score="1" ViewCount="" Body="&lt;p&gt;Jason,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Holt-Winters is a particular model form, normally additive or multiplicative and apparently may not be applicable to your particular time series. In general a Transfer Function incorporating both stochastic and deterministic structure has been found to a powerful way of handling problems like this. The problem you face I believe may be better handled with a mixed frequency model that might include one or more trends and/or level shifts and perhaps either weekly or monthly effects to deal with the &quot;seasonal component&quot; . Furthermore the model might exclude any identifiable anomalies so as to get a more robust &quot;signal&quot;. I suggest that you post your original data to the web and have some of the readers use their methods/software to try to model the data. I will try and do the same. It might make for a very interesting comparison !&lt;/p&gt;&#xA;" OwnerUserId="3382" LastActivityDate="2011-04-21T19:40:45.750" CommentCount="2" />
  <row Id="9844" PostTypeId="1" CreationDate="2011-04-21T20:05:29.060" Score="3" ViewCount="41" Body="&lt;p&gt;While studying machine learning, I've read the following statement:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The kernel $K(x,y)=(x\cdot y+1)^d$ , for $x, y \in \mathbb{R}^p$, has $M={p+d \choose d}$ eigenfunctions that span the space of polynomials in $\mathbb{R}^p$ of total degree $d$.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I do not understand how does the $M={p+d \choose d}$ come from? What does exactly the degree $d$ mean here?&lt;/p&gt;&#xA;" OwnerUserId="3026" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-04-21T20:21:59.427" LastActivityDate="2011-04-21T20:21:59.427" Title="Number of eigenfunctions for kernel" Tags="&lt;machine-learning&gt;&lt;svm&gt;&lt;kernel&gt;" CommentCount="10" FavoriteCount="1" />
  <row Id="9845" PostTypeId="1" CreationDate="2011-04-21T21:22:26.710" Score="0" ViewCount="144" Body="&lt;p&gt;Hello All,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have 1000 data for two continuous variables (pressure and temperature). I'd like to calculate Bayesian probability between two variables. In other words, I would like to determine probability that temperature will increase/decrease if pressure is changed? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I appreciate any opinion &lt;/p&gt;&#xA;&#xA;&lt;p&gt;thanks&lt;/p&gt;&#xA;" OwnerUserId="4282" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-21T21:39:31.620" LastActivityDate="2011-04-23T14:37:22.967" Title="How to calculate Bayesian probability between two variables?" Tags="&lt;probability&gt;&lt;correlation&gt;&lt;bayesian&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="9846" PostTypeId="2" ParentId="9845" CreationDate="2011-04-21T21:54:04.843" Score="1" ViewCount="" Body="&lt;p&gt;Relationships between physical quantities (like pressure and temperature) often may be described using functions or equations, and sometimes the measurement error is small – this might be the case here as well. If so you could derive the type of relationship and the specific function from physics knowledge you could use statistical (including Bayesian) techniques to determine the values and uncertainties of the parameters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the error is large the relationship between the variables will be less apparent and (compared to the physical equation behind the phenomenon) a more simple function could be fitted on the data. Linear regression is a widely used, simple method.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You tagged your question “machine-learning”, so the relationship between your variables may be more complex, but you may not be able to derive it from physics. In this case you can choose from a wide spectrum of non-linear machine learning and statistical techniques.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In any case I suggest you to plot your data using a scatterplot, think about the possible mechanisms between the variables. Although machine learning can be Bayesian, too, the best Bayesian approach may be fitting plausible models on the data.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastActivityDate="2011-04-21T21:54:04.843" CommentCount="1" />
  <row Id="9847" PostTypeId="2" ParentId="9835" CreationDate="2011-04-21T23:13:24.443" Score="3" ViewCount="" Body="&lt;p&gt;Mathematica will do integrals (and simplify the results) like no tomorrow. You have to be a little careful specifying your assumptions - that is, you should specify all of them - but it works quite well. If you're a student then your university may well have a site license but if you're just using it for a couple of problems then it's probably not worth picking it up. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Wolfram Alpha (http://www.wolframalpha.com/) is free and has many of the same facilities but it wil choke on some heavier problems. You can also pose your problems in more natural language (&quot;integrate ... wrt x from 0 to infinity&quot;) so it's nice for one-off's.&lt;/p&gt;&#xA;" OwnerUserId="26" LastActivityDate="2011-04-21T23:13:24.443" CommentCount="5" />
  <row Id="9848" PostTypeId="2" ParentId="9845" CreationDate="2011-04-22T01:20:12.637" Score="2" ViewCount="" Body="&lt;p&gt;I think there is a relationship between pressure, temperature and volume (if my memory of high school chemistry serves me correctly - confirmed by &lt;a href=&quot;http://en.wikipedia.org/wiki/Gas_laws&quot; rel=&quot;nofollow&quot;&gt;wikipedia&lt;/a&gt;):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$T=\frac{PV}{kN}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\begin{array} \\&#xD;&#xA;P=\text{Pressure} \\&#xD;&#xA;V=\text{Volume} \\&#xD;&#xA;k=\text{Boltzman&amp;#39;s constant}\\&#xD;&#xA;N=\text{Number of molecules}\\&#xD;&#xA;T=\text{Temperature}&#xD;&#xA;\end{array}&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that this is an approximate relation &lt;a href=&quot;http://en.wikipedia.org/wiki/Real_gas&quot; rel=&quot;nofollow&quot;&gt;this page&lt;/a&gt; shows some more complex ways of modeling pressure on temperature.  The principle would be the same as below, just different (harder) equations.  To remind us that this is the information, an $I$ will be put as part of the conditions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now you have some data on part of this equation, but I'm sure you would also have some less precise information on Volume.  If you were to take the logarithm of both sides, you get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$log(T)=log(P)+log(V)-log(k)-log(N)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A very simple procedure is to assume that the volume and number of molecules is approximately a constant, and thus we can model this part as a normal distribution.  So we have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$log(T_{i})=log(P_{i})-log(k)+n_{i}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where we assume $n_{i}\sim N(\mu,\sigma)$.  You can look at the noise level estimate $\sigma$ to decide if its worthwhile in getting a more complex model.  So this model has two parameters which you need to estimate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So you have a new starting pressure $P_{old}$ and  starting temperature $T_{old}$ and the pressure then changes to $P_{new}=P_{old}(1+\delta_{P})$, with a corresponding change in temperature $T_{new}=T_{old}(1+\delta_{T})$.  You want the probability that $-1&amp;lt;\delta_{T}&amp;lt;0$.  Note that this would be conditional on $I$, that you are making predictions in similar Volumes and number of molecules to that which was present when you took the data.  The golden rule is to calculate this probability, conditional on what you know, integrating or averaging over what you don't know.  I'll add more details later.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;MORE DETAILS&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any $T_{new}$ will be greater than $T_{old}$ when $log(T_{new})&amp;gt;log(T_{old})$, substituting in the model equation and simplifying gives:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$log(1+\delta_{P})&amp;gt;n_{old}-n_{new}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus we require the probability &#xA;$$P(log(1+\delta_{P})&amp;gt;n_{old}-n_{new}|n_{1},\dots,n_{N},I)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Fitting the model is a straight forward exercise.  You may have reasonable prior knowledge about, but since you have lots of data, it isn't necessary and a non-informative prior will do fine.  This is $p(\mu,\sigma|I)\propto\frac{1}{\sigma}$.  And the likelihood of the data is just the probability that the noise will make up the difference:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p(n_{1},\dots,n_{N}|\mu,\sigma,I)\propto\frac{1}{\sigma^{N}}exp\left(-\frac{\sum_{i}n_{i}^{2}}{2\sigma^{2}}\right)=\frac{1}{\sigma^{N}}exp\left(-\frac{\sum_{i}\left(d_{i}-\mu\right)^{2}}{2\sigma^{2}}\right)$$&#xA;Where&#xA;$$d_{i}=log\left[\frac{T_{i}k}{P_{i}}\right]$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now you will only need 3 numbers from your data in this solution, the sample size, $N$, the mean of $d_{i}$ denoted by $\overline{d}=\frac{1}{N}\sum_{i}d_{i}$, and its standard deviation, $s_{d}^{2}=\frac{1}{N}\sum_{i}(d_{i}-\overline{d})^{2}=\overline{d^{2}}-\overline{d}^{2}$.  The posterior predictive distribution for the noise is given by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p(n_{old},n_{new}|N,\overline{d},s_{d}^{2},I)=\int_{0}^{\infty}\int_{-\infty}^{\infty}p(n_{old},n_{new}|\mu,\sigma,I)p(n_{1},\dots,n_{N}|\mu,\sigma,I)p(\mu,\sigma|I)d\mu d\sigma$$&#xA;$$=St_{2}\left(\begin{pmatrix}n_{new}\\n_{old}\end{pmatrix}|\begin{pmatrix}\overline{d}\\\overline{d}\end{pmatrix},\frac{s_{d}^{2}}{N-1}\begin{pmatrix} N+1 &amp;amp; 1\\ 1 &amp;amp; N+1 \end{pmatrix},N-1\right)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $St(x|\mu,\Sigma,\nu)$ is a &lt;a href=&quot;http://en.wikipedia.org/wiki/Multivariate_Student_distribution&quot; rel=&quot;nofollow&quot;&gt;bivariate student distribution&lt;/a&gt; with location $\mu$, scatter matrix $\Sigma$, and $\nu$ degrees of freedom.  Now we simply integrate over the region that we want the probability&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(log(1+\delta_{P})&amp;gt;n_{old}-n_{new}|n_{1},\dots,n_{N},I)$$&#xA;$$=\int_{-\infty}^{\infty}\int_{-\infty}^{log(1+\delta_{P})+t}St_{2}\left(\begin{pmatrix}t\\s\end{pmatrix}|\begin{pmatrix}\overline{d}\\\overline{d}\end{pmatrix},\frac{s_{d}^{2}}{N-1}\begin{pmatrix} N+1 &amp;amp; 1\\ 1 &amp;amp; N+1 \end{pmatrix},N-1\right)dsdt$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This integral can be done using the t-tables, but at the moment I can't remember exactly how to do it.  You get an 1-D integral over a regularised incomplete beta function.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastEditorUserId="2392" LastEditorDisplayName="" LastEditDate="2011-04-23T14:37:22.967" LastActivityDate="2011-04-23T14:37:22.967" CommentCount="2" />
  <row Id="9850" PostTypeId="1" CreationDate="2011-04-22T02:22:15.400" Score="1" ViewCount="128" Body="&lt;p&gt;HI,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried clustering a set of data ( a set of marks ) and got 2 clusters. I would like to graphically represent it. Bit confused about the representation, since I dont have the (x,y) cordinates.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also looking for matlab/python function for doing so.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think posting data make the question clearer. I have two clusters I made using kmeans clustering in python ( not using scipy). They are&lt;/p&gt;&#xA;&#xA;&lt;p&gt;class 1: a=[3222403552.0, 3222493472.0, 3222491808.0, 3222489152.0, 3222413632.0, 3222394528.0, 3222414976.0, 3222522768.0, 3222403552.0, 3222498896.0, 3222541408.0, 3222403552.0, 3222402816.0, 3222588192.0, 3222403552.0, 3222410272.0, 3222394560.0, 3222402704.0, 3222298192.0, 3222409264.0, 3222414688.0, 3222522512.0, 3222404096.0, 3222486720.0, 3222403968.0, 3222486368.0, 3222376320.0, 3222522896.0, 3222403552.0, 3222374480.0, 3222491648.0, 3222543024.0, 3222376848.0, 3222403552.0, 3222591616.0, 3222376944.0, 3222325568.0, 3222488864.0, 3222548416.0, 3222424176.0, 3222415024.0, 3222403552.0, 3222407504.0, 3222489584.0, 3222407872.0, 3222402736.0, 3222402032.0, 3222410208.0, 3222414816.0, 3222523024.0, 3222552656.0, 3222487168.0, 3222403728.0, 3222319440.0, 3222375840.0, 3222325136.0, 3222311568.0, 3222491984.0, 3222542032.0, 3222539984.0, 3222522256.0, 3222588336.0, 3222316784.0, 3222488304.0, 3222351360.0, 3222545536.0, 3222323728.0, 3222413824.0, 3222415120.0, 3222403552.0, 3222514624.0, 3222408000.0, 3222413856.0, 3222408640.0, 3222377072.0, 3222324304.0, 3222524016.0, 3222324000.0, 3222489808.0, 3222403552.0, 3223571920.0, 3222522384.0, 3222319712.0, 3222374512.0, 3222375456.0, 3222489968.0, 3222492752.0, 3222413920.0, 3222394448.0, 3222403552.0, 3222403552.0, 3222540576.0, 3222407408.0, 3222415072.0, 3222388272.0, 3222549264.0, 3222325280.0, 3222548208.0, 3222298608.0, 3222413760.0, 3222409408.0, 3222542528.0, 3222473296.0, 3222428384.0, 3222413696.0, 3222486224.0, 3222361280.0, 3222522640.0, 3222492080.0, 3222472144.0, 3222376560.0, 3222378736.0, 3222364544.0, 3222407776.0, 3222359872.0, 3222492928.0, 3222440496.0, 3222499408.0, 3222450272.0, 3222351904.0, 3222352480.0, 3222413952.0, 3222556416.0, 3222410304.0, 3222399984.0, 3222494736.0, 3222388288.0, 3222403552.0, 3222323824.0, 3222523616.0, 3222394656.0, 3222404672.0, 3222405984.0, 3222490432.0, 3222407296.0, 3222394720.0, 3222596624.0, 3222597520.0, 3222598048.0, 3222403552.0, 3222403552.0, 3222403552.0, 3222324448.0, 3222408976.0, 3222448160.0, 3222366320.0, 3222489344.0, 3222403552.0, 3222494480.0, 3222382032.0, 3222450432.0, 3222352000.0, 3222352528.0, 3222414032.0, 3222728448.0, 3222299456.0, 3222400016.0, 3222495056.0, 3222388848.0, 3222403552.0, 3222487568.0, 3222523744.0, 3222394624.0, 3222408112.0, 3222406496.0, 3222405616.0, 3222592160.0, 3222549360.0, 3222438560.0, 3222597024.0, 3222597616.0, 3222598128.0, 3222403552.0, 3222403552.0, 3222403552.0, 3222499056.0, 3222408512.0, 3222402064.0, 3222368992.0, 3222511376.0, 3222414624.0, 3222554816.0, 3222494608.0, 3222449792.0, 3222351952.0, 3222352272.0, 3222394736.0, 3222311856.0, 3222414288.0, 3222402448.0, 3222401056.0, 3222413568.0, 3222298848.0, 3222297184.0, 3222488000.0, 3222490528.0, 3222394688.0, 3222408224.0, 3222406672.0, 3222404896.0, 3222443120.0, 3222403552.0, 3222596400.0, 3222597120.0, 3222597712.0, 3222400896.0, 3222403552.0, 3222403552.0, 3222403552.0, 3222299200.0, 3222321296.0, 3222364176.0, 3222602208.0, 3222513040.0, 3222414656.0, 3222564864.0, 3222407904.0, 3222449984.0, 3222352096.0, 3222352432.0, 3222452832.0, 3222368560.0, 3222414368.0, 3222399376.0, 3222298352.0, 3222573152.0, 3222438080.0, 3222409168.0, 3222523488.0, 3222394592.0, 3222405136.0, 3222490624.0, 3222406928.0, 3222407104.0, 3222442464.0, 3222403552.0, 3222596512.0, 3222597216.0, 3222597968.0, 3222438208.0, 3222403552.0, 3222403552.0, 3222403552.0]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;class 2: b=[3498543128.0, 3498542920.0, 3498543252.0, 3498543752.0, 3498544872.0, 3498544528.0, 3498543024.0, 3498542548.0, 3498542232.0]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to plot it. I tried the follwoing and got the follwoing result when I plot a and b.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pylab.plot(a,'x')&#xA;pylab.plot(b,'o')&#xA;pylab.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/CHIWH.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;can I get a better visualization of clustering?&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="2721" LastEditorUserId="2721" LastEditorDisplayName="" LastEditDate="2011-04-28T07:13:48.970" LastActivityDate="2011-04-28T18:06:58.480" Title="How to plot data output of clustering?" Tags="&lt;data-visualization&gt;&lt;clustering&gt;" AnswerCount="3" CommentCount="4" FavoriteCount="1" />
  <row Id="9851" PostTypeId="1" CreationDate="2011-04-22T02:35:47.893" Score="0" ViewCount="50" Body="&lt;p&gt;Where can I obtain more weather data? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;NOAA has some: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://weather.noaa.gov/pub/SL.us008001/DF.an/DC.sflnd/DS.synop/&quot; rel=&quot;nofollow&quot;&gt;http://weather.noaa.gov/pub/SL.us008001/DF.an/DC.sflnd/DS.synop/&lt;/a&gt; &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://weather.noaa.gov/pub/SL.us008001/DF.an/DC.sflnd/DS.metar/&quot; rel=&quot;nofollow&quot;&gt;http://weather.noaa.gov/pub/SL.us008001/DF.an/DC.sflnd/DS.metar/&lt;/a&gt; &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://weather.noaa.gov/pub/SL.us008001/DF.an/DC.sfmar/DS.dbuoy/&quot; rel=&quot;nofollow&quot;&gt;http://weather.noaa.gov/pub/SL.us008001/DF.an/DC.sfmar/DS.dbuoy/&lt;/a&gt; &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://weather.noaa.gov/pub/SL.us008001/DF.an/DC.sfmar/DS.ships/&quot; rel=&quot;nofollow&quot;&gt;http://weather.noaa.gov/pub/SL.us008001/DF.an/DC.sfmar/DS.ships/&lt;/a&gt; &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;but I also want amateur observations. Specifically: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Where can I get APRS WX data? I know that &quot;telnet rotate.aprs.net &#xA;10152&quot; with &quot;user READONLY pass -1&quot; will give me lots of &#xA;information, but: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Can I filter it down to just weather reports? &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Is this all APRS WX data, or just a subset? &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;wunderground.com has over 17K stations &#xA;at &lt;a href=&quot;http://www.wunderground.com/weatherstation/index.asp&quot; rel=&quot;nofollow&quot;&gt;http://www.wunderground.com/weatherstation/index.asp&lt;/a&gt; reporting to &#xA;them. Is this data available publicly on some &quot;free weather exchange &#xA;network&quot;? I realize I can get it from wunderground.com, but would &#xA;prefer a &quot;legit&quot; source. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Are there other sources of weather data out there? Is there any &#xA;one source that conglomerates multiple other sources? &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="1566" LastActivityDate="2011-04-22T02:35:47.893" Title="Where can I obtain more weather data?" Tags="&lt;data-mining&gt;" CommentCount="3" />
  <row Id="9852" PostTypeId="1" AcceptedAnswerId="9953" CreationDate="2011-04-22T02:41:34.453" Score="4" ViewCount="93" Body="&lt;p&gt;Let's say I have two regression models, one with three variables and one with four.  Each spits out an adjusted r^2, which I can compare directly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Obviously, the model with the higher adjusted r^2 is the better fit, but is there way to test the difference between the two adjusted r^2 and get a p-value?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know you can do Chow test to test the difference between slopes, but this is variance, so I don't think that's what I'm looking for.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: One model does not simply contain a subset of variables from the other model, or else I would probably use stepwise regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In model 1, I have four variables: W, X, Y, and Z.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In model 2, I have three variables: W, X, and (Y+Z)/2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The idea is that if Y and Z are conceptually similar, the model may make better predictions by grouping these two variables together prior to entering them into the model.&lt;/p&gt;&#xA;" OwnerUserId="1977" LastEditorUserId="1977" LastEditorDisplayName="" LastEditDate="2011-04-22T03:40:03.617" LastActivityDate="2011-04-25T12:57:28.647" Title="Testing difference between two (adjusted) r^2" Tags="&lt;regression&gt;" AnswerCount="2" CommentCount="5" />
  <row Id="9853" PostTypeId="2" ParentId="9852" CreationDate="2011-04-22T02:51:35.427" Score="0" ViewCount="" Body="&lt;p&gt;Take a look at Mallow's Cp:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Mallows%27_Cp&quot; rel=&quot;nofollow&quot;&gt;Mallow's Cp&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's a related question:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://stats.stackexchange.com/questions/8918/is-there-a-way-to-optimize-regression-according-to-a-specific-criterion&quot;&gt;Is there a way to optimize regression according to a specific criterion?&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2775" LastActivityDate="2011-04-22T02:51:35.427" />
  <row Id="9854" PostTypeId="1" CreationDate="2011-04-22T03:26:07.057" Score="1" ViewCount="85" Body="&lt;p&gt;I tried &lt;code&gt;CorrelationFunction[Transpose[{data,data}]][[All,1,2]]&lt;/code&gt; but it doesn't work! I mean the results are identical with those if I run &lt;code&gt;CorrelationFunction[data]&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="4286" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-04-22T06:17:24.753" LastActivityDate="2011-04-22T07:23:44.970" Title="How can I calculate the autocorrelation of a signal in Mathematica environment?" Tags="&lt;autocorrelation&gt;&lt;mathematica&gt;" AnswerCount="1" />
  <row Id="9855" PostTypeId="5" CreationDate="2011-04-22T06:52:58.393" Score="0" ViewCount="" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditorDisplayName="" LastEditDate="2011-04-22T06:52:58.393" LastActivityDate="2011-04-22T06:52:58.393" />
  <row Id="9856" PostTypeId="4" CreationDate="2011-04-22T06:52:58.393" Score="0" ViewCount="" Body="Mathematica is a popular software package for mathematical computations. " OwnerUserId="-1" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-04-22T09:24:37.353" LastActivityDate="2011-04-22T09:24:37.353" />
  <row Id="9857" PostTypeId="3" CreationDate="2011-04-22T06:59:10.240" Score="0" ViewCount="" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditorDisplayName="" LastEditDate="2011-04-22T06:59:10.240" LastActivityDate="2011-04-22T06:59:10.240" />
  <row Id="9858" PostTypeId="3" CreationDate="2011-04-22T06:59:10.240" Score="0" ViewCount="" Body="Linear regression is a type of regression when regression function is linear. It is most widely used regression type." OwnerUserId="-1" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-04-22T09:24:38.193" LastActivityDate="2011-04-22T09:24:38.193" />
  <row Id="9859" PostTypeId="1" CreationDate="2011-04-22T07:01:16.563" Score="5" ViewCount="53" Body="&lt;p&gt;Consider a vector of parameters $(\theta_1, \theta_2)$, with $\theta_1$ the parameter of interest, and $\theta_2$ a nuisance parameter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If $L(\theta_1, \theta_2 ; x)$ is the likelihood constructed from the data $x$, the profile likelihood for $\theta_1$ is defined as $L_P(\theta_1 ; x) = L(\theta_1, \hat{\theta}_2(\theta_1) ; x)$ where $ \hat{\theta}_2(\theta_1)$ is the MLE of $\theta_2$ for a fixed value of $\theta_1$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\bullet$ Maximising the profile likelihood with respect to $\theta_1$ leads to same estimate $\hat{\theta}_1$ as the one obtained by maximising the likelihood simultaneously with respect to $\theta_1$ and $\theta_2$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\bullet$ I think the standard deviation of $\hat{\theta}_1$ may also be estimated from the second derivative of the profile likelihood.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\bullet$ The likelihood statistic for $H_0: \theta_1 = \theta_0$ can be written in terms of the profile likelihood: $LR = 2 \log( \tfrac{L_P(\hat{\theta}_1 ; x)}{L_P(\theta_0 ; x)})$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, it seems that the profile likelihood can be used exactly as if it was a genuine likelihood. Is it really the case ? What are the main drawbacks of that approach ? And what about the 'rumor' that the estimator obtained from the profile likelihood is biased (edit: even asymptotically) ? &lt;/p&gt;&#xA;" OwnerUserId="3019" LastEditorUserId="3019" LastEditorDisplayName="" LastEditDate="2011-04-22T07:24:18.283" LastActivityDate="2011-04-22T07:24:18.283" Title="What are the disadvantages of the profile likelihood?" Tags="&lt;maximum-likelihood&gt;&lt;likelihood-function&gt;&lt;profile-likelihood&gt;" CommentCount="4" FavoriteCount="1" />
  <row Id="9860" PostTypeId="5" CreationDate="2011-04-22T07:09:42.677" Score="0" ViewCount="" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditorDisplayName="" LastEditDate="2011-04-22T07:09:42.677" LastActivityDate="2011-04-22T07:09:42.677" />
  <row Id="9861" PostTypeId="4" CreationDate="2011-04-22T07:09:42.677" Score="0" ViewCount="" Body="Dynamic regression is a type of regression, where one of the independent variables is a lagged dependent variable." OwnerUserId="-1" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-04-22T07:24:23.333" LastActivityDate="2011-04-22T07:24:23.333" />
  <row Id="9862" PostTypeId="2" ParentId="9854" CreationDate="2011-04-22T07:15:33.557" Score="3" ViewCount="" Body="&lt;p&gt;It's been a long since I didn't play with Mathematica, and I just had a quick look on Google, but can't you just use (here with some fake data)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x = Table[Sin[x] + 0.2 RandomReal[], {x, -4, 4, .1}];&#xA;ListPlot[x, DataRange -&amp;gt; {-4, 4}]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/ckN4G.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;the function &lt;a href=&quot;http://reference.wolfram.com/mathematica/ref/ListCorrelate.html&quot; rel=&quot;nofollow&quot;&gt;ListCorrelate&lt;/a&gt;?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;acf = ListCorrelate[x, x, {1, 1}, 0]&#xA;ListPlot[acf, Filling -&amp;gt; Axis]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/bzHkD.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-22T07:23:44.970" LastActivityDate="2011-04-22T07:23:44.970" CommentCount="1" />
  <row Id="9863" PostTypeId="2" ParentId="9807" CreationDate="2011-04-22T07:55:35.147" Score="2" ViewCount="" Body="&lt;h3&gt;1. How do I input them in SPSS?&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;You can open an Excel file in SPSS.&#xA;Use the standard file open option, and select &lt;code&gt;file type = *xls&lt;/code&gt;.&#xA;Try to ensure that the first row has the variable names.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;2. How do I work out the frequency of replies for each recipient?&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Do you mean the frequency of responses for each question?&lt;/li&gt;&#xA;&lt;li&gt;Check out the menu &lt;code&gt;Descriptive Statistics - Frequencies&lt;/code&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;3. How do I work out frequency of replies i.e agrees/disagrees etc for each group?&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Check out &lt;code&gt;Descriptive Statistics - Crosstabs&lt;/code&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;4. How can I rank each individual question (12 of them)? Remember, there are 3 individual statements to each question.&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Rank them in terms of what?&lt;/li&gt;&#xA;&lt;li&gt;If you intend to rank each question in terms of their mean (e.g., on a one to five scale). One way would be to run &lt;code&gt;Descriptive Statistics - Descriptives&lt;/code&gt; and get the mean for each item. Then copy and paste the table of item means into Excel and sort by the &lt;code&gt;Mean&lt;/code&gt; column.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;5. How do I compare UK architects to US architects to show congruence or not?&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Check out &lt;code&gt;Descriptive Statistics - Explore&lt;/code&gt;; you could also look at some of the compare mean options.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;6. How would show correlation between the two groups UK and US?&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;These are different participants so I don't know what you mean by asking for correlations.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;7. Will SPSS develop graphs etc for me showing frequency or correlation?&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Yes, it will.&lt;/li&gt;&#xA;&lt;li&gt;Just have a play around with the Graphs menu (e.g., &lt;code&gt;Legacy - Scatter&lt;/code&gt; or &lt;code&gt;Legacy - Bar&lt;/code&gt;)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;General Suggestions&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;It sounds like you need a basic book explaining how to use SPSS.&#xA;A good one is the SPSS Survival Manual.&#xA;I also wrote a &lt;a href=&quot;http://web.psych.unimelb.edu.au/jkanglim/Anglim2006-SPSSIntroductionWorkshop.pdf&quot; rel=&quot;nofollow&quot;&gt;120 page PDF Introduction to SPSS&lt;/a&gt; several years back which explains all the things mentioned above with examples.&lt;/p&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-22T08:19:58.057" LastActivityDate="2011-04-22T08:19:58.057" />
  <row Id="9864" PostTypeId="1" AcceptedAnswerId="9873" CreationDate="2011-04-22T08:34:21.153" Score="4" ViewCount="106" Body="&lt;p&gt;I'm trying to use the EM cluster algorithm, provided by the software Weka, to classify my data and it only finds one cluster. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Could I interpret this as there are no ways to distinguish the instances in my sample? &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;This is a result that is coherent with others analysis  that I'm doing to the data, but I don't know if I could use a cluster analysis to state that.&lt;/p&gt;&#xA;" OwnerUserId="4203" LastEditorUserId="4203" LastEditorDisplayName="" LastEditDate="2011-04-24T18:02:58.863" LastActivityDate="2011-04-24T18:02:58.863" Title="Interpretation of a one cluster solution using the EM cluster algorithm" Tags="&lt;machine-learning&gt;&lt;clustering&gt;&lt;weka&gt;" AnswerCount="1" FavoriteCount="0" />
  <row Id="9865" PostTypeId="2" ParentId="9813" CreationDate="2011-04-22T09:29:57.620" Score="1" ViewCount="" Body="&lt;p&gt;Your problem would fall under the category of &quot;missing data.&quot;  Ultimately, one way or another you are going to have to infer the hidden variables $Z$.  This can be done using the &lt;a href=&quot;http://en.wikipedia.org/wiki/Expectation-maximization_algorithm&quot; rel=&quot;nofollow&quot;&gt;Expectation-Maximization Algorithm&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="3567" LastActivityDate="2011-04-22T09:29:57.620" CommentCount="1" />
  <row Id="9866" PostTypeId="2" ParentId="9735" CreationDate="2011-04-22T09:59:08.940" Score="1" ViewCount="" Body="&lt;p&gt;I will take this as an opportunity to explain some &lt;em&gt;fundamental&lt;/em&gt; issues regarding the difference between frequentist and Bayesian statistics, by interpreting frequentist practices from a Bayesian standpoint.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this example, we have observed data $D_1$ for the original and data $D_2$ for the combination case.  One assumes that these are generated by Bernoulli random variables with parameters $p_1$ and $p_2$, respectively, and that these parameters come from the priors, $f_i(p_i)$ (with cdfs $F_i(p_i)$).  The probability $p_1 &amp;gt; p_2$ can be calculated, as you pointed out.  It is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;P[p_1 &amp;gt; p_2;f_1,f_2] = \frac{\int_0^1 \int_0^1 I(p_1 &amp;gt; p_2) P[D_1|p_1] P[D_2|p_1] dF_1(p_1) dF_2(p_2)}{\int_0 ^1 \int_0^1 P[D_1|p_1] P[D_2|p_1] dF_1(p_1) dF_2(p_2) }&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here the Bayesian chooses priors $f_1(p_1)$ and $f_2(p_2)$ (and will usually choose the same prior for both, due to exchangeability) and proceeds with inference.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The frequentist takes a &quot;conservative&quot; approach when choosing a prior.  The possible values of $\theta$ are assumed to be known, but the frequentist has so little confidence in their ability to assign a meaningful prior, so that they effectively &lt;em&gt;look at all possible priors&lt;/em&gt; and then only make an inferential statement &lt;em&gt;when that inferential statement is true under all possible priors&lt;/em&gt;.  When no inference is valid under all possible priors, the frequentist remains silent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That is the situation in this case.  When one considers the priors $g_{\theta_i}(p_i)$ given by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;g_{\theta_i}(p_i) = \delta (\theta_i)&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;that is, the point mass concentrated at $\theta_i$, then one can easily see that the probability desired is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;P[p_1 &amp;gt; p_2;g_{\theta_1},g_{\theta_2}] = \delta_{\theta_1, \theta_2}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;that is, 1 when $\theta_1 = \theta_2$ and 0 otherwise.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus the frequentist remains silent. (Or, alternatively, makes the trivial statement: &quot;The probability is between 0 and 1...&quot;)&lt;/p&gt;&#xA;" OwnerUserId="3567" LastEditorUserId="3567" LastEditorDisplayName="" LastEditDate="2011-04-22T10:05:54.733" LastActivityDate="2011-04-22T10:05:54.733" CommentCount="4" />
  <row Id="9867" PostTypeId="1" CreationDate="2011-04-22T11:44:46.420" Score="5" ViewCount="85" Body="&lt;p&gt;I have performed a Cochran's Q test for a within-subjects experimental design with 3 conditions and 36 participants with a dichotomous dependent variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found a (just) statistically significant effect ($\chi^2$ = 6.00, df = 2, p = 0.04979) and would like to also report the effect size, but haven't been able to find any information as to which effect measure to use and how to calculate it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any pointers would be gratefully received - the domain is human factors (psychology).&lt;/p&gt;&#xA;" OwnerUserId="4258" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-22T16:16:14.697" LastActivityDate="2011-05-24T20:07:50.003" Title="Effect size of Cochran's Q" Tags="&lt;nonparametric&gt;&lt;effect-size&gt;&lt;cochran-q&gt;" AnswerCount="1" FavoriteCount="3" />
  <row Id="9868" PostTypeId="1" CreationDate="2011-04-22T12:00:12.613" Score="0" ViewCount="69" Body="&lt;p&gt;Hey, im a beginner to R and trying to run pvclust so as to test a cluster solution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've managed to load data and run the heirachical cluster, however the code i find online for running pvclust is constantly producing errors - just wondering if someone can point out where I'm going wrong...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;here is my code (data already transposed. my dataset is called &quot;transpose&quot; below..)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;##loaddata&#xA;&#xA;transpose &amp;lt;- &#xA;  read.table(&quot;C:/Users/Tim/University/Advanced Design and Data Analysis/Assignment/transposed.csv&quot;,&#xA;   header=TRUE, sep=&quot;,&quot;, na.strings=&quot;NA&quot;, dec=&quot;.&quot;, strip.white=TRUE)&#xA;&#xA;View(transpose)&#xA;hello &amp;lt;- hclust(dist(model.matrix(~-1 + &#xA;  var001+var002+var003+var004+var005+var006+var007+var008+var009+var010+var011+var012+var013+var014+var015+var016+var017+var018+var019+var020+var021+var022+var023+var024+var025+var026+var027+var028+var029+var030+var031+var032+var033+var034+var035+var036+var037+var038+var039+var040+var041+var042+var043+var044+var045+var046+var047+var048+var049+var050+var051+var052+var053+var054+var055+var056+var057+var058+var059+var060+var061+var062+var063+var064+var065+var066+var067+var068+var069+var070+var071+var072+var073+var074+var075+var076+var077+var078+var079+var080+var081+var082+var083+var084+var085+var086+var087+var088+var089+var090+var091+var092+var093+var094+var095+var096+var097+var098+var099+var100+var101+var102+var103+var104+var105+var106+var107+var108+var109+var110+var111+var112+var113+var114+var115+var116+var117+var118+var119+var120+var121+var122+var123+var124+var125+var126+var127+var128+var129+var130+var131+var132+var133+var134+var135+var136+var137+var138+var139+var140+var141+var142+var143+var144+var145+var146+var147+var148+var149+var150+var151+var152+var153+var154+var155+var156+var157+var158+var159+var160+var161+var162+var163+var164+var165+var166+var167+var168+var169+var170+var171+var172+var173+var174+var175+var176,&#xA;   transpose)) , method= &quot;ward&quot;)&#xA;&#xA;plot(hello, main= &quot;Cluster Dendrogram for Solution hello&quot;, xlab= &#xA;  &quot;Observation Number in Data Set transpose&quot;, sub=&quot;Method=ward; &#xA;  Distance=euclidian&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;all the above works fine, except the below where i try the pvclust&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(pvclust)&#xA;&#xA;fit &amp;lt;- pvclust(transpose, method.hclust=&quot;ward&quot;,&#xA;   method.dist=&quot;euclidean&quot;)&#xA;&#xA;plot(fit) &#xA;pvrect(fit, alpha=.95)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;the error comes back=&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(pvclust)&#xA;&amp;gt; fit &amp;lt;- pvclust(transpose, method.hclust=&quot;ward&quot;,&#xA;+    method.dist=&quot;euclidean&quot;)&#xA;Warning in dist(t(x), method) : NAs introduced by coercion&#xA;Error in hclust(distance, method = method.hclust) : &#xA;  NA/NaN/Inf in foreign function call (arg 11)&#xA;&amp;gt; plot(fit) &#xA;Error in plot(fit) : object 'fit' not found&#xA;&amp;gt; pvrect(fit, alpha=.95)&#xA;Error in nrow(x$edges) : object 'fit' not found&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="4287" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-22T16:05:37.807" LastActivityDate="2011-05-22T17:49:10.233" Title="Problem with pvclust in R" Tags="&lt;r&gt;&lt;clustering&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="9869" PostTypeId="2" ParentId="9842" CreationDate="2011-04-22T12:58:02.280" Score="4" ViewCount="" Body="&lt;p&gt;Here's a good quick introduction:&#xA;&lt;a href=&quot;http://arxiv.org/pdf/cs/0308031&quot; rel=&quot;nofollow&quot;&gt;intro to neural networks.&lt;/a&gt;&#xA;Note that R has neural-network functionality, so no need to spend any time implementing NN yourself until you've given it a spin and decided it looks promising for your application.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Neural networks are not obsolete, but they have gone through a couple of hype cycles, and then after realizing they don't do everything as was claimed, their reputation goes into a trough for a while (we're currently in one of those).  Neural networks are good at certain tasks, and generally are better for tasks in which a human can do a similar task, but cannot explain exactly how they do it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Neural networks do not give you much insight into the system you're using them to analyze, even after they are trained and operating well.  That is, they can predict what will happen (for some systems), but not tell you why.  In some cases, that is fine.  In others, that is not fine.  Generally, if you want or especially if you already have an understanding of the rules of how something works, you can use other techniques.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But, for certain tasks, they work well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For time-series in particular, see this question's discussion:&#xA;&lt;a href=&quot;http://stats.stackexchange.com/questions/8000/proper-way-of-using-recurrent-neural-network-for-time-series-analysis&quot;&gt;Proper way of using recurrent neural network for time series analysis&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2917" LastActivityDate="2011-04-22T12:58:02.280" />
  <row Id="9870" PostTypeId="2" ParentId="9842" CreationDate="2011-04-22T13:18:17.590" Score="3" ViewCount="" Body="&lt;p&gt;While it is focussed on statistical pattern recognition, rather than time series forecasting, I would strongly recommend Chris Bishop's book &lt;a href=&quot;http://www.oup.com/us/catalog/general/subject/Medicine/Neuroscience/?view=usa&amp;amp;ci=9780198538646&quot; rel=&quot;nofollow&quot;&gt;Neural Networks for Pattern Recognition&lt;/a&gt; becuase it is the best introduction to neural networks in general, and I think it would be a good idea to get to grips with the potential pitfalls in the use of neural networks in a more simple context, where the problems are more easily visualised an understood.  Then move on to the book on recurrent neural networks by &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0471495174&quot; rel=&quot;nofollow&quot;&gt;Mandic and Chambers&lt;/a&gt;.  The bishop book is a classic, nobody should use neural nets for anything until they feel confident that they understand the material contained in that book; ANN make it all too easy to shoot yourself in the foot!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also disagree with mbq, nn are not obsolete, while many problems are better solved with linear models or more modern machine learning techniques (e.g. kernel methods), there are some problems where they work well and other methods don't.  It is still a tool that should be in our toolboxes.&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2011-04-22T13:18:17.590" />
  <row Id="9871" PostTypeId="1" CreationDate="2011-04-22T13:28:43.467" Score="9" ViewCount="148" Body="&lt;p&gt;Hi, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a &quot;basic statistics&quot; concept question. As a student I would like to know if I'm thinking about this totally wrong and why, if so:     &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say I am hypothetically trying to look at the relationship between &quot;anger management issues&quot; and say divorce (yes/no) in a logistic regression and I have the option of using two different anger management scores -- both out of 100.&lt;br&gt;&#xA;Score 1 comes from questionnaire rating instrument 1 and my other choice; score 2 comes from a different questionnaire. Hypothetically, we have reason to believe from previous work that anger management issues give rise to divorce.&lt;br&gt;&#xA;If, in my sample of 500 people, the variance of score 1 is much higher than that of score 2, is there any reason to believe that score 1 would be a  better score to use as a predictor of divorce based on its variance? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To me, this instinctively seems right, but is it so?&lt;/p&gt;&#xA;" OwnerUserId="4054" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-22T16:11:54.387" LastActivityDate="2011-04-23T15:48:17.837" Title="Is a predictor with greater variance &quot;better&quot;? " Tags="&lt;regression&gt;&lt;logistic-regression&gt;" AnswerCount="3" FavoriteCount="1" />
  <row Id="9872" PostTypeId="1" CreationDate="2011-04-22T13:42:38.583" Score="3" ViewCount="92" Body="&lt;p&gt;I have a set of variables for building credit scorecards with logistic-regression.  I need to bin some variables, for e.g. years of credit history.  What is the method to determine how many bins and what is the interval for each bin?  Thanks&lt;/p&gt;&#xA;" OwnerUserId="4288" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-22T16:09:29.863" LastActivityDate="2011-04-22T16:09:29.863" Title="Binning raw data prior to building a logistic regression model" Tags="&lt;regression&gt;&lt;logistic-regression&gt;" AnswerCount="1" />
  <row Id="9873" PostTypeId="2" ParentId="9864" CreationDate="2011-04-22T13:58:04.903" Score="0" ViewCount="" Body="&lt;p&gt;Two assumptions here: 1) Weka's finding the number of clusters (k) without issues, and 2) I believe EM uses mixtures of Gaussians which means the clusters need to be round/elliptical.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, given that Weka's algorithm is finding the best k, the answer would be that using round/elliptical clusters, the most likely clustering is one group. That doesn't mean that your data doesn't cluster at all (using other shapes, essentially).&lt;/p&gt;&#xA;" OwnerUserId="1764" LastActivityDate="2011-04-22T13:58:04.903" />
  <row Id="9874" PostTypeId="2" ParentId="9871" CreationDate="2011-04-22T14:05:57.473" Score="1" ViewCount="" Body="&lt;p&gt;Always check the assumptions for the statistical test you're using!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of the assumptions of logistic regression is independence of errors which means that cases of data should not be related. Eg. you can't measure the same people at different points in time which I fear you may have done with your anger management surveys.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would also be worried that with 2 anger management surveys you're basically measuring the same thing and your analysis could suffer from multicollinearity.&lt;/p&gt;&#xA;" OwnerUserId="3597" LastActivityDate="2011-04-22T14:05:57.473" CommentCount="1" />
  <row Id="9875" PostTypeId="2" ParentId="9871" CreationDate="2011-04-22T14:11:05.953" Score="7" ViewCount="" Body="&lt;p&gt;A few quick points:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Variance can be arbitrarily increased or decreased by adopting a different scale for your variable. Multiplying a scale by a constant greater than one would increase the variance, but not change the predictive power of the variable. &lt;/li&gt;&#xA;&lt;li&gt;You may be confusing variance with reliability. All else being equal (and assuming that there is at least some true score prediction), increasing the reliability with which you measure a construct should increase its predictive power. Check out this discussion of &lt;a href=&quot;http://en.wikipedia.org/wiki/Correction_for_attenuation&quot; rel=&quot;nofollow&quot;&gt;correction for attenuation&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;li&gt;Assuming that both scales were made up of twenty 5-point items, and thus had total scores that ranged from 20 to 100, then the version with the greater variance would also be more reliable (at least in terms of internal consistency).&lt;/li&gt;&#xA;&lt;li&gt;Internal consistency reliability is not the only standard by which to judge a psychological test, and it is not the only factor that distinguishes the predictive power of one scale versus another for a given construct.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastEditorUserId="2669" LastEditorDisplayName="" LastEditDate="2011-04-23T15:48:17.837" LastActivityDate="2011-04-23T15:48:17.837" />
  <row Id="9876" PostTypeId="1" AcceptedAnswerId="9914" CreationDate="2011-04-22T14:31:04.797" Score="3" ViewCount="69" Body="&lt;p&gt;Good day all,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose that I am conducting a questionnaire study that is trying to measure level of awareness of subjects about a programming language and find the relation of those level of awareness to working conditions and methods etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To improve my precision I decided to go with stratified sampling. If I have 1 criteria for stratification such as geo-distribution (to make sure I don't over represent subjects from areas that have less programmers), then I end up with 6 distinct strata (country provinces).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know how to go about analysing these to find margin error, standard error etc but I realised that is not good enough and I need to introduce more criteria's for stratification, such as level of education (so i don't over represent a group who are not very present between programmers based on their education), level of seniority etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have the proportion (%) for all these criteria's but I don't know how I go about sampling when I have more than one criteria?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for your help :)  &lt;/p&gt;&#xA;" OwnerUserId="4043" LastActivityDate="2011-04-25T18:05:38.407" Title="Stratified sampling question" Tags="&lt;stratification&gt;" AnswerCount="2" />
  <row Id="9877" PostTypeId="2" ParentId="9872" CreationDate="2011-04-22T15:32:13.207" Score="4" ViewCount="" Body="&lt;p&gt;Binning will result in a more complex model, i.e., you will need more terms in the model to predict the outcome as well as a model that treats the predictors as continuous.  Bins also bring a degree of arbitrariness into the model.  Take a look at regression splines as an alternative.  Notes about this may be found at &lt;a href=&quot;http://biostat.mc.vanderbilt.edu/rms&quot; rel=&quot;nofollow&quot;&gt;http://biostat.mc.vanderbilt.edu/rms&lt;/a&gt;.  Also make sure that your outcome is truly dichotomous, i.e., that the time until the event is irrelevant and you have no censoring.&lt;/p&gt;&#xA;" OwnerUserId="4253" LastActivityDate="2011-04-22T15:32:13.207" CommentCount="3" />
  <row Id="9878" PostTypeId="2" ParentId="9868" CreationDate="2011-04-22T15:58:34.223" Score="0" ViewCount="" Body="&lt;p&gt;It's difficult to answer without seeing the data itself, but my best guess is that you have some non numerical entries in the matrix/dataframe (which is what is expected by &lt;code&gt;pvclust&lt;/code&gt;). For example, &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; as.numeric(c(1,2,&quot;NA&quot;))&#xA;[1]  1  2 NA&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;or &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; dist(c(1,2,&quot;NA&quot;))&#xA;   1  2&#xA;2  1   &#xA;3 NA NA&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;will produce the same warning message ('&lt;em&gt;NAs introduced by coercion&lt;/em&gt;'). I deliberately used &lt;code&gt;&quot;NA&quot;&lt;/code&gt;, but any element that is not numerical will result in the same warning message.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;A warning message is issued when trying to compute a distance matrix from the non-numerical input&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Then, &lt;code&gt;hclust&lt;/code&gt; failed when it is called within &lt;code&gt;pvclust&lt;/code&gt;. Again, &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; hclust(dist(c(1,2,&quot;NA&quot;)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;will throw the same error message.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In your first try, you called &lt;code&gt;hclust&lt;/code&gt; by using a matrix. Can't you just check that you use the same variables in both cases, or that there is no strange values in your data (e.g., &lt;code&gt;summary(transpose)&lt;/code&gt;), or no missing values coded as the character &lt;code&gt;&quot;NA&quot;&lt;/code&gt; instead of &lt;code&gt;NA&lt;/code&gt;, as below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; xx &amp;lt;- data.frame(replicate(3, sample(c(1,2,3), 3)))&#xA;&amp;gt; xx[2,3] &amp;lt;- &quot;NA&quot;&#xA;&amp;gt; is.na(xx[2,3])&#xA;[1] FALSE&#xA;&amp;gt; sapply(xx, is.character)&#xA;   X1    X2    X3 &#xA;FALSE FALSE  TRUE &#xA;&amp;gt; apply(xx, 2, function(x) sum(is.na(x)))&#xA;X1 X2 X3 &#xA; 0  0  0 &#xA;# Now if we had a true NA value, we would see&#xA;&amp;gt; xx[2,3] &amp;lt;- NA&#xA;&amp;gt; apply(xx, 2, function(x) sum(is.na(x)))&#xA;X1 X2 X3 &#xA; 0  0  1 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-04-22T15:58:34.223" />
  <row Id="9879" PostTypeId="1" AcceptedAnswerId="9882" CreationDate="2011-04-22T16:49:50.407" Score="3" ViewCount="83" Body="&lt;p&gt;Given two multivariate gaussian (say in 2D with mean $\mu$ as a 2D point and convariance marix $\Sigma$ as $2$x$2$ Matrix) $N_1(\mu_1,\Sigma_1)$ and $N_2(\mu_2,\Sigma_1)$, I would like to derive the pdf of $N_1+N_2$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can any one point me to the reference where i can find the pdf derivation of $N_1 + N_2$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance&lt;/p&gt;&#xA;" OwnerUserId="4290" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-23T08:07:08.500" LastActivityDate="2011-04-23T08:07:08.500" Title="Addition of multivariate gaussians" Tags="&lt;multivariate-analysis&gt;&lt;normal-distribution&gt;" AnswerCount="1" CommentCount="9" />
  <row Id="9880" PostTypeId="1" AcceptedAnswerId="9958" CreationDate="2011-04-22T17:40:49.883" Score="3" ViewCount="126" Body="&lt;p&gt;I'd like to model a set of processes. The processes in question are related to human-decision making, so the model will need a measure of input, processing, and then, finally, output.&#xA;Ideally, the model would be implemented in something like R (which I know quite well) or Python (which I know less well). &lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Questions:&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Where is the best place to start with something like this? &lt;/li&gt;&#xA;&lt;li&gt;What tools are available?&lt;/li&gt;&#xA;&lt;li&gt;What software or language is suited to writing the model in?&lt;/li&gt;&#xA;&lt;li&gt;What method would be suited to testing that model against data I've collected from real humans?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="4204" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-23T07:09:31.090" LastActivityDate="2011-04-25T16:55:53.633" Title="What models and software are suited to modelling human decision making?" Tags="&lt;r&gt;&lt;modeling&gt;" AnswerCount="2" CommentCount="4" FavoriteCount="1" />
  <row Id="9881" PostTypeId="2" ParentId="9868" CreationDate="2011-04-22T17:42:06.957" Score="0" ViewCount="" Body="&lt;p&gt;Hi&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Pvclust is a bit odd in that it expects your data to be organized with the observations in the rows, rather than the columns. This is what the documentation in &lt;code&gt;?pvclust&lt;/code&gt; tries to explain, just not very well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;try transposing your original data matrix in the call to &lt;code&gt;pvclust()&lt;/code&gt; and see if it runs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;i.e., &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;fit &amp;lt;- pvclust(t(transpose), method.hclust=&quot;ward&quot;,method.dist=&quot;euclidean&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;as an aside its not good R practice to give names to objects that are also the names of R functions e.g., 'transpose'&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good luck&lt;/p&gt;&#xA;" OwnerUserId="1475" LastActivityDate="2011-04-22T17:42:06.957" CommentCount="2" />
  <row Id="9882" PostTypeId="2" ParentId="9879" CreationDate="2011-04-22T19:46:13.887" Score="4" ViewCount="" Body="&lt;h2&gt;Method 1: characteristic functions&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Referring to (say) the Wikipedia article on the &lt;a href=&quot;http://en.wikipedia.org/wiki/Multivariate_normal_distribution&quot; rel=&quot;nofollow&quot;&gt;multivariate normal distribution&lt;/a&gt; and using the 1D technique to compute sums in the article on &lt;a href=&quot;http://en.wikipedia.org/wiki/Sum_of_normally_distributed_random_variables&quot; rel=&quot;nofollow&quot;&gt;sums of normal distributions&lt;/a&gt;, we find the log of its characteristic function is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$i t \mu - t&amp;#39; \Sigma t.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The cf of a sum is the product of the cfs, so the logarithms add.  This tells us the cf of the sum of two &lt;em&gt;independent&lt;/em&gt; MVN distributions (indexed by 1 and 2) has a logarithm equal to&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$i t (\mu_1 + \mu_2) - t&amp;#39; (\Sigma_1 + \Sigma_2) t.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because the cf uniquely determines the distribution we can immediately read off that the sum is MVN with mean $\mu_1 + \mu_2$ and variance $\Sigma_1 + \Sigma_2$.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Method 2: Linear combinations&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;View the pair of MVN distributions as being a single MVN with mean $(\mu_1, \mu_2)$ and covariance $\Sigma_1 \oplus \Sigma_2$.  The sum is given by a &lt;a href=&quot;http://en.wikipedia.org/wiki/Multivariate_normal_distribution#Affine_transformation&quot; rel=&quot;nofollow&quot;&gt;linear transformation&lt;/a&gt; and therefore is MVN.  (See p. 2 #4 &lt;a href=&quot;http://www.stat.lsu.edu/faculty/moser/exst7037/mvnprop.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.) The covariance again works out to $\Sigma_1 + \Sigma_2$.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-04-22T19:46:13.887" CommentCount="7" />
  <row Id="9883" PostTypeId="2" ParentId="9871" CreationDate="2011-04-22T20:15:55.110" Score="6" ViewCount="" Body="&lt;p&gt;A simple example helps us identify what is essential.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let &#xA;$$Y = C + \gamma X_1 + \varepsilon$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $C$ and $\gamma$ are parameters, $X_1$ is the score on the first instrument (or independent variable), and $\varepsilon$ represents unbiased iid error.  Let the score on the second instrument be related to the first one via &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$X_1 = \alpha X_2 + \beta.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, scores on the second instrument might range from 25 to 75 and scores on the first from 0 to 100, with $X_1 = 2 X_2 - 50$.  The variance of $X_1$ is $\alpha^2$ times the variance of $X_2$.  Nevertheless, we can rewrite&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Y = C + \gamma(\alpha X_2 + \beta) = (C + \beta \gamma)  + (\gamma \alpha) X_2 + \varepsilon = C&amp;#39; + \gamma&amp;#39; X_2 + \varepsilon.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The parameters change, &lt;em&gt;and the variance of the independent variable changes&lt;/em&gt;, yet the predictive capability of the model remains unchanged&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general the relationship between $X_1$ and $X_2$ may be nonlinear.  Which is a better predictor of $Y$ will depend on which has a closer linear relationship to $Y$.  Thus &lt;strong&gt;the issue is not one of scale&lt;/strong&gt; (as reflected by the variance of the $X_i$) but has to be decided by the relationships between the instruments and what they are being used to predict.  This idea is closely related to one explored in a recent question about &lt;a href=&quot;http://stats.stackexchange.com/q/9590/919&quot;&gt;selecting independent variables in regression&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There can be mitigating factors.  For instance, if $X_1$ and $X_2$ are discrete variables and both are equally well related to $Y$, then the one with larger variance &lt;em&gt;might&lt;/em&gt; (if it is sufficiently uniformly spread out) allow for finer distinctions among its values and thereby afford more precision.  &lt;em&gt;E.g.&lt;/em&gt;, if both instruments are questionnaires on a 1-5 Likert scale, both are equally well correlated with $Y$, and the answers to $X_1$ are all 2 and 3 and the answers to $X_2$ are spread among 1 through 5, $X_2$ might be favored on this basis.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-04-22T20:15:55.110" />
  <row Id="9884" PostTypeId="1" CreationDate="2011-04-22T21:52:28.570" Score="2" ViewCount="48" Body="&lt;h3&gt;Question on pairs()&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;I'd like to use &lt;code&gt;pairs()&lt;/code&gt; to choose a functional form for modeling a set of data.  I know that several of my independent and dependent variables are probably lognormally distributed, so I'd like to produce &lt;code&gt;pairs()&lt;/code&gt; plots where some of my variables are plotted on log axes and some are not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems like the par() command might hold the key, but I don't know how to implement it.  &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Can I specify log/linear axes behavior when I call pairs()?  &lt;/li&gt;&#xA;&lt;li&gt;Do I plot &lt;code&gt;pairs()&lt;/code&gt; first and then modify the resulting graph using par()?  Am I on the wrong track entirely?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I'm sure I could accomplish the task more easily by transforming some of my variables to log space, but I'm trying to learn something along the way here.&lt;/p&gt;&#xA;" OwnerUserId="4237" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-23T06:57:50.600" LastActivityDate="2011-04-23T06:57:50.600" Title="Selectively tweaking pairs() axes?" Tags="&lt;r&gt;&lt;data-visualization&gt;&lt;scatterplot&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="9885" PostTypeId="1" CreationDate="2011-04-22T22:27:24.467" Score="9" ViewCount="152" Body="&lt;p&gt;I have a &lt;strong&gt;Machine Learning&lt;/strong&gt; course this semester and the professor asked us to find a &lt;strong&gt;real-world problem&lt;/strong&gt; and solve it by one of machine learning methods introduced in the class, as:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Decision_tree_learning&quot; rel=&quot;nofollow&quot;&gt;Decision Trees&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Artificial_neural_network&quot; rel=&quot;nofollow&quot;&gt;Artificial Neural Networks&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Support_vector_machine&quot; rel=&quot;nofollow&quot;&gt;Support Vector Machines&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Instance-based_learning&quot; rel=&quot;nofollow&quot;&gt;Instance-based Learning&lt;/a&gt; (&lt;a href=&quot;http://en.wikipedia.org/wiki/K-nearest_neighbor_algorithm&quot; rel=&quot;nofollow&quot;&gt;kNN&lt;/a&gt;, &lt;a href=&quot;http://en.wikipedia.org/wiki/Locally_weighted_regression&quot; rel=&quot;nofollow&quot;&gt;LWL&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Bayesian_network&quot; rel=&quot;nofollow&quot;&gt;Bayesian Networks&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Reinforcement_learning&quot; rel=&quot;nofollow&quot;&gt;Reinforcement learning&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I am one of fans of &lt;strong&gt;stackoverflow&lt;/strong&gt; and &lt;strong&gt;stackexchange&lt;/strong&gt; and know &lt;a href=&quot;http://blog.stackoverflow.com/category/cc-wiki-dump/&quot; rel=&quot;nofollow&quot;&gt;database dumps&lt;/a&gt; of these websites are provided to the public because they are awesome! I hope I could find a good machine learning challenge about these databases and solve it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My idea&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One idea came to my mind is predicting tags for questions based on the entered words in question body. I think the Bayesian network is the right tool for learning tags for a question but need more research.&#xA;Anyway, after learning phase when user finishes entering the question some tags should be suggested to him. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Please tell me&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to ask the stats community as experienced people about ML two questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Do you think tag suggestion is at least a problem which has any chance to solve? Do you have any advice about it? I am a little worried because stackexchange does not implement such feature yet. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Do you have any other/better idea for the ML project that is based on stackexchange database? I find it really hard to find something to &lt;strong&gt;learn&lt;/strong&gt; from stackexchange databases. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Consideration about database errors:&lt;/strong&gt;&#xA;I would like to point that although the databases are huge and have many instances, they are not perfect and are prune to error. The obvious one is the age of users that is unreliable. Even selected tags for the question are not 100% correct. Anyway, we should consider the percent of correctness of data in selecting a problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Consideration about the problem itself:&lt;/strong&gt; My project should not be about &lt;code&gt;data-mining&lt;/code&gt; or something like this. It just should be an application of ML methods in real-world.&lt;/p&gt;&#xA;" OwnerUserId="2148" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-04-23T22:30:19.920" LastActivityDate="2011-04-25T23:08:59.523" Title="Application of machine learning methods in StackExchange websites" Tags="&lt;machine-learning&gt;" AnswerCount="2" FavoriteCount="4" />
  <row Id="9886" PostTypeId="1" CreationDate="2011-04-22T23:01:54.647" Score="4" ViewCount="72" Body="&lt;p&gt;Hi all.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a set of data that looks at the number of &quot;hits&quot; a specific program makes over the course of time.  The data goes back to September 2010, and includes data up to March 2011, so the data points are monthly.  What I want to see if the most recent data (March 2011) shows a statistically significant decrease in the number of &quot;hits&quot; this program makes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a feeling there might not be a test that would fit this perfectly, as the data is a bit limited.  I can also pull data weekly for the same time frame, which would build 31 points (at which point I would still want to look at the most recent unit for comparison).  There hasn't been a population mean built for this data as of yet, as the data can only be pulled as far back as Jan 2010 (but the data from then is not reliable).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For reference, just 9 weeks data (as i pulled that first) reveals&#xA;mean= 1013.67&#xA;n=9&#xA;st.dev= 53.57&#xA;Most recent week= 991&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just eyeballing it does not appear statistically significant as a drop in &quot;hits&quot;, however I'll need to perform this analysis every few weeks, and wondering if there's something reliable I can use.  Thanks ahead of time for the input!&lt;/p&gt;&#xA;" OwnerUserId="4292" LastActivityDate="2011-04-23T13:08:48.037" Title="Statistical test for a series of data over time" Tags="&lt;time-series&gt;&lt;statistical-significance&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="1" />
  <row Id="9887" PostTypeId="2" ParentId="9885" CreationDate="2011-04-22T23:16:59.540" Score="3" ViewCount="" Body="&lt;p&gt;I was thinking about tag prediction, too, I like the idea. I have the feeling that it is possible, but you may need to overcome many issues before you arrive to your final dataset. So I speculate the tag prediction may need a lot of time. In addition to incorrect tags the limit of max 5 tags may play a role. Also that some tags are subcategories of others (e.g. “multiple comparisons” can be viewed as a subcategory of “significance testing”).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I did not check if up-vote times are included in the downloadable database, but a more simple and still interesting project could be to predict the “final” number of votes (maybe after 5 months) on a question depending on the initial votes, and the timing of accepting an answer.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastActivityDate="2011-04-22T23:16:59.540" CommentCount="3" />
  <row Id="9889" PostTypeId="1" CreationDate="2011-04-23T00:37:14.603" Score="0" ViewCount="47" Body="&lt;p&gt;Hello folks, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am new to R and cointegration so please have patience with me as I try to explain what it is that I am trying to do. I am trying to find cointegrated variables among 1500-2000 voltage variables in the west power system in Canada/US. THe frequency is hourly (common in power) and cointegrated combinations can be as few as N variables and a maximum of M variables. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried to use ca.jo but here are issues that I ran into: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) ca.jo (Johansen) has a limit to the number of variables it can work with &lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) ca.jo appears to force the first variable in the y(t) vector to be the dependent variable (see below). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Eigenvectors, normalised to first column: &#xA;(These are the cointegration relations) &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;         V1.l2        V2.l2        V3.l2 &#xA;V1.l2    1.0000000    1.0000000    1.0000000   &#xA;V2.l2   -0.2597057   -2.3888060   -0.4181294   &#xA;V3.l2   -0.6443270   -0.6901678    0.5429844   &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As you can see ca.jo tries to find linear combinations of the 3 variables but by forcing the coefficient on the first variable (in this case V1) to be 1 (i.e. the dependent variable). My understanding was that ca.jo would try to find all combinations such that every variable is selected as a dependent variable. You can see the same treatment in the examples given in the documentation for ca.jo. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) ca.jo does not appear to find linear combinations of fewer than the number of variables in the y(t) vector. So if there were 5 variables and 3 of them are cointegrated (i.e. V1 ~ V2 + V3) then ca.jo fails to find this combination. Perhaps I am not using ca.jo correctly but my expectation was that a cointegrated combination where V1 ~ V2 + V3 is the same as V1 ~ V2 + V3 + 0 x V4 + 0 x V5. In other words the coefficient of the variable that are NOT cointegrated should be zero and ca.jo should find this type of combination. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would greatly appreciate some further insight as I am fairly new to R and cointegration and have spent the past 2 months teaching myself. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you.&lt;/p&gt;&#xA;" OwnerUserId="4294" LastEditorUserId="4294" LastEditorDisplayName="" LastEditDate="2011-04-23T01:01:34.980" LastActivityDate="2011-04-23T01:01:34.980" ClosedDate="2011-04-23T20:55:16.447" Title="Cointegration / multivariate case for 1500+ variables" Tags="&lt;multivariate-analysis&gt;&lt;cointegration&gt;" CommentCount="2" />
  <row Id="9890" PostTypeId="2" ParentId="9809" CreationDate="2011-04-23T02:12:41.907" Score="2" ViewCount="" Body="&lt;p&gt;If you are just looking to simplify an expression involving expectations, Economists’ Mathematical Manual has a nice concise list of identities. You can find a copy online &lt;a href=&quot;http://www.tbparis.com/Econ230/References%20from%20Web/Sydsaeter%20etal%20-%20Economists%27%20Mathematical%20Manual,%204th%20Edition,%20Springer%20%282005%29.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="4281" LastActivityDate="2011-04-23T02:12:41.907" CommentCount="2" />
  <row Id="9892" PostTypeId="1" CreationDate="2011-04-23T02:46:51.203" Score="3" ViewCount="122" Body="&lt;p&gt;I am trying to perform logistic regression with lasso. For the logistic regression part I am using &lt;code&gt;PROC LOGISTIC&lt;/code&gt; but I am not sure how to do lasso with &lt;code&gt;PROC LOGISTIC&lt;/code&gt;. I searched online  and found that &lt;code&gt;PROC GLMSELECT&lt;/code&gt; allows us to do lasso. But I am not sure how to do logistic regression with lasso using &lt;code&gt;PROC GLMSELECT&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note:&lt;/strong&gt; I posted this question in the &lt;a href=&quot;http://support.sas.com/forums/thread.jspa?threadID=13929&amp;amp;tstart=0&quot; rel=&quot;nofollow&quot;&gt;SAS Discussion Forum&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="3897" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-23T07:56:14.857" LastActivityDate="2011-04-23T07:56:14.857" Title="How to perform logistic regression with lasso using GLMSELECT?" Tags="&lt;logistic-regression&gt;&lt;sas&gt;&lt;lasso&gt;" CommentCount="2" />
  <row Id="9893" PostTypeId="1" AcceptedAnswerId="9896" CreationDate="2011-04-23T00:55:25.810" Score="2" ViewCount="76" Body="&lt;p&gt;Context: To recommend a minimum sample size when performing multivariate testing of a web page.  The sample size would vary based on the number of factors being tested (e.g. A heading, and an image) and the number of variations of a factor (e.g. Two different headings, and three different images).  The goal could be to see which combination caused the most purchases of a product.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Based on a recommended minimum sample size of 100 for a single factor with perhaps two variations, I'm trying to work out a formula that recommends a sample size with multiple factors and variations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The formula I first came up with is shown below where n is the number of variations for that factor.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ samplesize = &#xD;&#xA;100*\left((n_{f1}-1)^{n_{f1}-1}*(n_{f2}-1)^{n_{f2}-1}*\ldots\right)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does this seem reasonable and/or is there a simpler formula that would be as reasonable? The intended audience are online business owners who aren't necessarily strong at maths.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for reading!&lt;/p&gt;&#xA;" OwnerUserId="4346" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-23T06:48:32.757" LastActivityDate="2011-04-23T06:48:32.757" Title="Formula for recommended sample size for multivariate testing" Tags="&lt;multivariate-analysis&gt;&lt;sample-size&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="9895" PostTypeId="1" CreationDate="2011-04-23T03:51:41.710" Score="2" ViewCount="159" Body="&lt;p&gt;Dear All Members,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've tough luck with the use of  nls() in &lt;em&gt;R&lt;/em&gt; for the following model&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$N_e = N_o\{1-exp[\frac{(d+bN_o)(T_h N_e - T)}{(1+c N_o)}]\}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $b&amp;gt;0$, $c\geq 0$, $T_h&amp;gt;0$, and $T=72$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This code&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;T &amp;lt;- 72&#xA;NLS.Fit3 &amp;lt;- nls(Ne~No*(1-exp((d+b*No)*(Th*Ne-T)/(1+c*No))), data = Data,&#xA;            start = list(d = 0.01, b = 0.01, Th = 0.01, c = 0.01),&#xA;            control = nls.control(maxiter=50, tol=1e-05, minFactor=1/1024))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;gives the following error message: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Error in nls(Ne ~ No * (1 - exp((d + b * No) * (Th * Ne - T)/(1 + c *  : &#xA;  singular gradient&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And the following &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;NLS.Fit31 &amp;lt;- nls(Ne~No*(1-exp((d+b*No)*(Th*Ne-T)/(1+c*No))), data = Data,&#xA;            start = list(d = 0.01, b = 0.01, Th = 0.01, c = 0.01),&#xA;            control = nls.control(maxiter=50, tol=1e-05, minFactor=1/1024),&#xA;            algorithm = &quot;port&quot;, lower=c(0, 0, 0, 0))&#xA;summary(NLS.Fit31)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;code converges but provides the wrong results (drastically different from PROC NLIN)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Formula: Ne ~ No * (1 - exp((d + b * No) * (Th * Ne - T)/(1 + c * No)))&#xA;&#xA;Parameters:&#xA;Estimate Std. Error t value Pr(&amp;gt;|t|)  &#xA;d  0.008325   0.003488   2.387   0.0192 *&#xA;b  0.000000   0.000064   0.000   1.0000  &#xA;Th 0.000000   0.614220   0.000   1.0000  &#xA;c  0.020670   0.034439   0.600   0.5500  &#xA;---&#xA;Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 &#xA;&#xA;Residual standard error: 4.631 on 85 degrees of freedom&#xA;&#xA;Algorithm &quot;port&quot;, convergence message: relative convergence (4)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'd prefer to do this in &lt;em&gt;R&lt;/em&gt; rather than &lt;em&gt;SAS&lt;/em&gt; and how the constrains can be placed on only few paramters. Any help in this regard will be highly appreciated. Thanks&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data is here:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;No  Ne&#xA;5   0&#xA;5   1&#xA;5   1&#xA;5   2&#xA;5   2&#xA;5   2&#xA;5   2&#xA;5   3&#xA;7   0&#xA;7   0&#xA;7   1&#xA;7   1&#xA;7   2&#xA;7   2&#xA;7   2&#xA;7   3&#xA;10  1&#xA;10  1&#xA;10  2&#xA;10  2&#xA;10  3&#xA;10  3&#xA;10  3&#xA;10  4&#xA;10  7&#xA;15  1&#xA;15  1&#xA;15  3&#xA;15  3&#xA;15  4&#xA;15  5&#xA;15  5&#xA;15  5&#xA;20  3&#xA;20  4&#xA;20  7&#xA;20  7&#xA;20  8&#xA;20  8&#xA;20  9&#xA;20  11&#xA;25  4&#xA;25  5&#xA;25  6&#xA;25  7&#xA;25  9&#xA;25  9&#xA;25  13&#xA;25  14&#xA;30  5&#xA;30  8&#xA;30  10&#xA;30  11&#xA;30  11&#xA;30  12&#xA;30  14&#xA;30  20&#xA;45  4&#xA;45  7&#xA;45  8&#xA;45  10&#xA;45  11&#xA;45  14&#xA;45  15&#xA;45  19&#xA;60  9&#xA;60  14&#xA;60  14&#xA;60  16&#xA;60  18&#xA;60  21&#xA;60  24&#xA;60  26&#xA;80  7&#xA;80  11&#xA;80  12&#xA;80  15&#xA;80  17&#xA;80  12&#xA;80  21&#xA;80  23&#xA;100 7&#xA;100 8&#xA;100 10&#xA;100 11&#xA;100 15&#xA;100 24&#xA;100 26&#xA;100 33&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="3903" LastEditorUserId="3903" LastEditorDisplayName="" LastEditDate="2011-04-23T09:16:25.693" LastActivityDate="2011-04-25T00:02:07.613" Title="Having trouble with nls function in R" Tags="&lt;r&gt;&lt;nonlinear-regression&gt;" AnswerCount="2" CommentCount="7" />
  <row Id="9896" PostTypeId="2" ParentId="9893" CreationDate="2011-04-23T04:20:25.040" Score="1" ViewCount="" Body="&lt;p&gt;Commonly, the different values that a factor can attain in an experiment are called &quot;levels&quot;. So let's say there are $k$ factors, and factor $j$ has $n_j$ levels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are $n_{f1}\cdot n_{f2}\cdot \dots \cdot n_{fk}$ possible factor combinations, i.e. possible versions of web pages that could be viewed. To answer the question whether any one of these versions is better than any other one, each has to be viewed a certain number of times, let's say $N$ for simplicity, for a sample size of $2N$. (You assumed $N = 100$). So the total sample size required (the total number of pairs of eyes that you'll need for all versions) is &#xA;$$ N \cdot n_{f1}\cdot n_{f2}\cdot \dots \cdot n_{fk} $$&#xA;which can become pretty large, although it's generally smaller than your formula. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The size of $N$ in turn depends on the separation of the purchase probabilities that you want to distinguish. If all purchase probabilities are close to each other, then $N$ would have to be quite large to pick the larger probability reliably even in a simple pairwise comparison. Examples: If you use $N = 100$ and one particular page design has purchase probability $p = .5$ and you are using a test with significance level 0.95, then you'll have a better than even chance of correctly identifying another design as better only if that design has purchase probability at least $p = .62$ or so. If that other design has $p = .55$, you won't be able to tell with $N = 100$ ... although it means 10% more revenue. Paradoxically you would be forced to work with an even larger sample size if the differences in probabilities are smaller.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In practice, one would not use all possible level combinations for all factors, because experience shows that interactions between multiple factors rarely matter. For example if you have four factors (say number of headings, number of images, number of columns, background color), then it is likely that once two have been set (say number of headings and number of images), the other two factors don't matter that much any more. This can be used to reduce the total number of level combinations. Google &quot;fractional factorial design&quot;. &lt;/p&gt;&#xA;" OwnerUserId="4062" LastActivityDate="2011-04-23T04:20:25.040" CommentCount="1" />
  <row Id="9897" PostTypeId="2" ParentId="9884" CreationDate="2011-04-23T05:27:54.883" Score="1" ViewCount="" Body="&lt;p&gt;With pairs, you can set the upper, lower and diagonal panels to display differently by passing functions which setup the layout of the plot (so, use &lt;code&gt;text()&lt;/code&gt; or &lt;code&gt;points()&lt;/code&gt; or &lt;code&gt;lines()&lt;/code&gt; rather than &lt;code&gt;plot()&lt;/code&gt; which makes a new plot). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(123)&#xA;#fake some data&#xA;exp1 &amp;lt;- rnorm(1000,5,1)&#xA;exp2 &amp;lt;- rlnorm(1000,0.5,1)&#xA;response &amp;lt;- exp1+exp2+rnorm(1000)&#xA;&#xA;panel.loglog &amp;lt;- function(x,y, ...){&#xA;   #&quot;usr&quot; is a par attribute that sets the plot area&#xA;   par(usr = c(0, max(log(x)+3), 0, max(log(y)+3) ))&#xA;   points(log(x), log(y))&#xA;}&#xA;&#xA;pairs(response~exp1+exp2, panel.lower=panel.loglog)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But it seems you're more interested in the distribution of your variables than the relationship? In that case you could add histograms to the diagnoal (the panel.hist function is from the help for &lt;code&gt;pairs&lt;/code&gt;)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;panel.hist &amp;lt; function(x, ...){&#xA;    usr &amp;lt;- par(&quot;usr&quot;); on.exit(par(usr))&#xA;    par(usr = c(usr[1:2], 0, 1.5) )&#xA;    h &amp;lt;- hist(x, plot = FALSE)&#xA;    breaks &amp;lt;- h$breaks; nB &amp;lt;- length(breaks)&#xA;    y &amp;lt;- h$counts; y &amp;lt;- y/max(y)&#xA;    rect(breaks[-nB], 0, breaks[-1], y, col=&quot;cyan&quot;, ...)&#xA;}&#xA;&#xA;pairs(response~exp1+exp2, lower.panel=panel.loglog, diag.panel=panel.hist)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Which will give you this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/uRaME.png&quot; alt=&quot;pairs plot&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="3732" LastActivityDate="2011-04-23T05:27:54.883" />
  <row Id="9898" PostTypeId="1" AcceptedAnswerId="9900" CreationDate="2011-04-23T06:48:25.657" Score="2" ViewCount="229" Body="&lt;p&gt;Dear All Members,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could someone come up with &lt;em&gt;R&lt;/em&gt; code to plot an ellipse from the eigenvalues and the eigenvectors of the following matrix&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\begin{equation*}&#xD;&#xA;\mathbf{A} = \left(&#xD;&#xA;\begin{array} {cc}&#xD;&#xA;2.2 &amp;amp; 0.4\\&#xD;&#xA;0.4 &amp;amp; 2.8&#xD;&#xA;\end{array} \right)&#xD;&#xA;\end{equation*}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="3903" LastEditorUserId="3903" LastEditorDisplayName="" LastEditDate="2011-04-23T18:47:32.860" LastActivityDate="2011-04-23T18:47:32.860" Title="How to plot an ellipse from eigenvalues and eigenvectors in R?" Tags="&lt;r&gt;&lt;multivariate-analysis&gt;&lt;matrix&gt;&lt;matrix-decomposition&gt;" AnswerCount="2" FavoriteCount="2" />
  <row Id="9899" PostTypeId="2" ParentId="9898" CreationDate="2011-04-23T09:18:26.517" Score="2" ViewCount="" Body="&lt;p&gt;Hi MYaseen2008:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think this is the R code that you want. I borrowed the R-code from this &lt;a href=&quot;https://stat.ethz.ch/pipermail/r-help/2006-October/114652.html&quot; rel=&quot;nofollow&quot;&gt;thread&lt;/a&gt; on the r-mailing list. The idea basically is: the major and minor half-diameters are the two eigen values and you rotate the ellipse by the amount of angle between the first eigen vector and the x-axis  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mat &amp;lt;- matrix(c(2.2, 0.4, 0.4, 2.8), 2, 2)&#xA;eigens &amp;lt;- eigen(mat)&#xA;evs &amp;lt;- sqrt(eigens$values)&#xA;evecs &amp;lt;- eigens$vectors&#xA;&#xA;a &amp;lt;- evs[1]&#xA;b &amp;lt;- evs[2]&#xA;x0 &amp;lt;- 0&#xA;y0 &amp;lt;- 0&#xA;alpha &amp;lt;- atan(evecs[ , 1][2] / evecs[ , 1][1])&#xA;theta &amp;lt;- seq(0, 2 * pi, length=(1000))&#xA;&#xA;x &amp;lt;- x0 + a * cos(theta) * cos(alpha) - b * sin(theta) * sin(alpha)&#xA;y &amp;lt;- y0 + a * cos(theta) * sin(alpha) + b * sin(theta) * cos(alpha)&#xA;&#xA;&#xA;png(&quot;graph.png&quot;)&#xA;plot(x, y, type = &quot;l&quot;, main = expression(&quot;x = a cos &quot; * theta * &quot; + &quot; * x[0] * &quot; and y = b sin &quot; * theta * &quot; + &quot; * y[0]), asp = 1)&#xA;arrows(0, 0, a * evecs[ , 1][2], a * evecs[ , 1][2])&#xA;arrows(0, 0, b * evecs[ , 2][3], b * evecs[ , 2][2])&#xA;dev.off()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/F5tPk.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="1307" LastEditorUserId="1307" LastEditorDisplayName="" LastEditDate="2011-04-23T18:20:42.837" LastActivityDate="2011-04-23T18:20:42.837" CommentCount="5" />
  <row Id="9900" PostTypeId="2" ParentId="9898" CreationDate="2011-04-23T09:25:59.283" Score="3" ViewCount="" Body="&lt;p&gt;You could extract the eigenvectors and -values via &lt;code&gt;eigen(A)&lt;/code&gt;. However, it's simpler to use the Cholesky decomposition. Note that when plotting confidence ellipses for data, the ellipse-axes are usually scaled to have length = square-root of the corresponding eigenvalues, and this is what the Cholesky decomposition gives.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ctr    &amp;lt;- c(0, 0)                               # data centroid -&amp;gt; colMeans(dataMatrix)&#xA;A      &amp;lt;- matrix(c(2.2, 0.4, 0.4, 2.8), nrow=2) # covariance matrix -&amp;gt; cov(dataMatrix)&#xA;RR     &amp;lt;- chol(A)                               # Cholesky decomposition&#xA;angles &amp;lt;- seq(0, 2*pi, length.out=200)          # angles for ellipse&#xA;ell    &amp;lt;- 1 * cbind(cos(angles), sin(angles)) %*% RR  # ellipse scaled with factor 1&#xA;ellCtr &amp;lt;- sweep(ell, 2, ctr, &quot;+&quot;)               # center ellipse to the data centroid&#xA;plot(ellCtr, type=&quot;l&quot;, lwd=2, asp=1)            # plot ellipse&#xA;points(ctr[1], ctr[2], pch=4, lwd=2)            # plot data centroid&#xA;&#xA;library(car)  # verify with car's ellipse() function&#xA;ellipse(c(0, 0), shape=A, radius=0.98, col=&quot;red&quot;, lty=2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Edit: in order to plot the eigenvectors as well, you have to use the more complicated approach. This is equivalent to suncoolsu's answer, it just uses matrix notation to shorten the code.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;eigVal  &amp;lt;- eigen(A)$values&#xA;eigVec  &amp;lt;- eigen(A)$vectors&#xA;eigScl  &amp;lt;- eigVec %*% diag(sqrt(eigVal))  # scale eigenvectors to length = square-root&#xA;xMat    &amp;lt;- rbind(ctr[1] + eigScl[1, ], ctr[1] - eigScl[1, ])&#xA;yMat    &amp;lt;- rbind(ctr[2] + eigScl[2, ], ctr[2] - eigScl[2, ])&#xA;ellBase &amp;lt;- cbind(sqrt(eigVal[1])*cos(angles), sqrt(eigVal[2])*sin(angles)) # normal ellipse&#xA;ellRot  &amp;lt;- eigVec %*% t(ellBase)                                          # rotated ellipse&#xA;plot((ellRot+ctr)[1, ], (ellRot+ctr)[2, ], asp=1, type=&quot;l&quot;, lwd=2)&#xA;matlines(xMat, yMat, lty=1, lwd=2, col=&quot;green&quot;)&#xA;points(ctr[1], ctr[2], pch=4, col=&quot;red&quot;, lwd=3)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/aEJZ7.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="1909" LastEditorUserId="1909" LastEditorDisplayName="" LastEditDate="2011-04-23T11:04:09.037" LastActivityDate="2011-04-23T11:04:09.037" CommentCount="2" />
  <row Id="9901" PostTypeId="2" ParentId="9895" CreationDate="2011-04-23T10:40:49.003" Score="1" ViewCount="" Body="&lt;p&gt;I now think you have a problem with data. Why?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First of all, let's get rid of $e^x$, solving it out and taking $\log$ of both sides&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\log\left(\frac{N_0-N_e}{N_0}\right)=\frac{(d+bN_0)(T_h N_e -T)}{1+cN_0}.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, this log should be a linear function of $N_e$ for fixed $N_0$. As I understand, $T-T_h N_e $ is a time available to the predator to predate and this $-\frac{d+bN_0}{1+cN_0}$ (let's call it $\lambda$) is the frequency of attacks, which is told to be somehow related to the number of prey. Putting this $\lambda$ into the model, we have&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\log\left(\frac{N_0-N_e}{N_0}\right)=\lambda T- \lambda T_h N_e,$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;linear function of $N_e$. Is this in data? It seems so:&#xA;&lt;img src=&quot;http://i.imgur.com/njTeO.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, we fit lines, get line parameters&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;       lamT       -lamTh   No&#xA;1  0.05753546 -0.29739723   5&#xA;2  0.01408135 -0.18090769   7&#xA;3  0.13772614 -0.18005256  10&#xA;4  0.02065162 -0.08429763  15&#xA;5  0.09416886 -0.07751886  20&#xA;6  0.11148562 -0.06477383  25&#xA;7  0.19165200 -0.06134858  30&#xA;8  0.04328295 -0.03025566  45&#xA;9  0.06706140 -0.02399182  60&#xA;10 0.02236404 -0.01553757  80&#xA;11 0.01947130 -0.01247877 100&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and try to calculate the $\lambda$ and $T_h$; but something is wrong at this point:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/pFFZX.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$-\lambda T_h$ is a clearly $(\alpha N_0+\beta)^{-1}$ (what corresponds to $b=0$), but $T/T_h$ is a total mess; thus my guess is that $T$ was not equal to 72 for all samples and this is the main origin of your problems. &lt;/p&gt;&#xA;" OwnerUserId="88" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-24T18:04:48.237" LastActivityDate="2011-04-24T18:04:48.237" CommentCount="10" />
  <row Id="9902" PostTypeId="2" ParentId="9841" CreationDate="2011-04-23T11:22:22.667" Score="1" ViewCount="" Body="&lt;p&gt;Your initial logistic regression approach seems reasonable, assuming good diagnostics.  However, if I understand the scientific question right, you'll need an interaction between year and day to see &lt;em&gt;differences&lt;/em&gt; between years in the tendency to be in diapause as day increases.  The main effect of year only says that one year has a higher per day probability of being in diapause, which is not, I think, what you care about.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Alternatively&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The other way to think about the problem is to have days as a dependent variable.  This means you would reverse the conditional probability formulation that you implicitly started with by not asking 'are later pupae more likely to be in diapause?' i.e. P(diapause | day, year), but rather 'do pupae in dipause tend to occur later in the year' i.e. P(day | diapause, year).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In such a model the observations are Pupation_Day which is classified by Diapause status which is nested within Year.  The effect of interest is a difference of differences, specifically: the difference between the difference in mean Pupation_Day for Diapause==1 and Diapause==0 in Year==2005 versus Year=2006.  (I'm not still sure what role the 'pairs of years' play in your design though.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The only potentially tricky bits I see are remembering that Diapause is a random effect, and making a sensible assumption about the conditional distribution of Pupation_Day.  Frankly, I'd try Normal with shared variance first and then see if the model diagnostics object.  If they do, I'd crack open JAGS/BUGS, and just write out the full model, which would make inference about difference in differences easier too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This alternative approach may be too much machinery for the scientific question though.  If we could tell what the ideal inferential endpoint was it would be easier to recommend an approach.&lt;/p&gt;&#xA;" OwnerUserId="1739" LastActivityDate="2011-04-23T11:22:22.667" CommentCount="5" />
  <row Id="9903" PostTypeId="2" ParentId="9886" CreationDate="2011-04-23T12:39:47.857" Score="1" ViewCount="" Body="&lt;p&gt;As GaBorgulya pointed out one needs to have a model to detect the potential anomaly. This model needs to generate a &quot;white noise&quot; error series or be sufficient to separate signal and noise. With this model in hand based upon older data one could then compare the new value with the prediction interval. This is the classical , albeit limited approach called an &quot;out off model test&quot;. A more comprehensive approach is to to include a &quot;pulse variable&quot; i.e. zeros and a 1 for the new data point and to estimate coefficients for the augmented model using all of the data. The probability of observing what you observed before you observed it ( i.e. the new value&quot; ) is then available from the &quot;t value&quot; of the &quot;pulse variable&quot; in this augmented model. In general this approach is referred to as Intervention Detection which scans ( data mines ) the time periods to detect the points where Pulses , Level Shifts , Seasonal Pulses and Local Time Trends have been significantly evidented. In your case you are not searching for the null hypothesis but rather simply is there a potential change point at the last observation i.e. the last &quot;1&quot; period. Your question also suggests solutions that we have seen which detect a significant change in the mean of the last K periods alerting the analyst to the innovation.&lt;/p&gt;&#xA;" OwnerUserId="3382" LastEditorUserId="3382" LastEditorDisplayName="" LastEditDate="2011-04-23T13:08:48.037" LastActivityDate="2011-04-23T13:08:48.037" />
  <row Id="9904" PostTypeId="1" AcceptedAnswerId="9906" CreationDate="2011-04-23T13:28:08.003" Score="3" ViewCount="80" Body="&lt;p&gt;The location and scale of a normally distributed data can be estimated by sampling the data then taking the mean of the sample means and standard deviations, respectively. For non-normal (heavy-tailed) data, is it correct to take the median of the sample medians and IQR/MAD, instead? That is, is it correct to use the median of sample medians as a robust estimator of location similar to the mean of sample means for normal data?&lt;/p&gt;&#xA;" OwnerUserId="3074" LastActivityDate="2011-04-24T02:09:31.230" Title="Median of medians as robust mean of means?" Tags="&lt;robust&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="9905" PostTypeId="2" ParentId="9880" CreationDate="2011-04-23T13:36:38.763" Score="2" ViewCount="" Body="&lt;p&gt;Have you thought about agent based modeling/social simulation? It's not that clear from your question what exactly you're trying to achieve but ABM may be suitable for your purposes as it does lend itself to modelling human decision making as you can programme agents with different characteristics. It's also very good for spatial problems.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could either use the data you have already gained on the population under study and use that to programme your agents and then you can forecast into the future and compare to what happens in the real life population. This can be repeatedly tested with different environmental factors say with an economic model on the decision to invest with the environmental effect of different tax rates. Alternatively you could set up a model and compare it to your real life data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While chapter 26 of The R Book is on Simulation Models you may be better off with Netlogo than with R which is also free and especially designed for agent based modelling.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://ccl.northwestern.edu/netlogo/&quot; rel=&quot;nofollow&quot;&gt;http://ccl.northwestern.edu/netlogo/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Journal of Artificial Societies and Social Simulation is a great open source journal covering this area.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://jasss.soc.surrey.ac.uk/JASSS.html&quot; rel=&quot;nofollow&quot;&gt;http://jasss.soc.surrey.ac.uk/JASSS.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="3597" LastActivityDate="2011-04-23T13:36:38.763" CommentCount="4" />
  <row Id="9906" PostTypeId="2" ParentId="9904" CreationDate="2011-04-23T13:42:53.630" Score="1" ViewCount="" Body="&lt;p&gt;If all the samples come from the same distribution, then yes the median of the sample medians is a fairly robust estimate of the median of the underlying distribution (though this need not be the same as the mean), since the median of a sample from a continuous distribution has probability 0.5 of being below (or above) the population median.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Added&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is some illustrative R code.  It takes a sample from a normal distribution and a case with outliers where 1% of data is 10,000 time bigger than it should be.  It looks at the various statistics for the overall sample data (50,000 points) and then by the centre (mean or median) of the statistics of the 10,000 samples with 5 points in each sample.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(matrixStats)&#xA;&#xA;wholestats &amp;lt;- function(x,n) {&#xA;     mea &amp;lt;- sum(x)/n  &#xA;     var &amp;lt;- sum((x-mea)^2)/(n-1)&#xA;     sdv &amp;lt;- sqrt(var) &#xA;     qun &amp;lt;- quantile(x, c(0.25,0.5,0.75))&#xA;     mad &amp;lt;- median(abs(x-qun[2]))&#xA;     c(mean=mea, variance=var, st.dev=sdv, &#xA;       median=qun[2], IQR=qun[3]-qun[1], &#xA;       MAD=mad)&#xA;    }&#xA;&#xA;rowstats &amp;lt;- function(x,b) {&#xA;     rmea &amp;lt;- rowSums(x)/b     &#xA;     rvar &amp;lt;- rowSums((x-rmea)^2)/(b-1)&#xA;     rsdv &amp;lt;- sqrt(rvar)&#xA;     rqun &amp;lt;- rowQuantiles(x, c(0.25,0.5,0.75))  &#xA;     rmad &amp;lt;- rowMedians(abs(x-rqun[,2]))&#xA;     c(mean=mean(rmea), variance=mean(rvar), st.dev=mean(rsdv), &#xA;       median=median(rqun[,2]), IQR=median(rqun[,3]-rqun[,1]), &#xA;       MAD=median(rmad))&#xA;    }&#xA;&#xA;a &amp;lt;- 10000 # number of samples&#xA;b &amp;lt;- 5     # samplesize&#xA;&#xA;set.seed(1)&#xA;d &amp;lt;- array(rnorm(a*b), dim=c(a,b))&#xA;doutlier &amp;lt;- array(d * ifelse(runif(a*b)&amp;gt;0.99, 10000, 1) , dim=c(a,b))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The median based statistics as expected are more robust, though they fail to show that the heavy tailed outlier variant is heavy tailed.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; wholestats(d,a*b)&#xA;        mean     variance       st.dev   median.50%      IQR.75%          MAD &#xA;-0.002440456  1.011306552  1.005637386 -0.001610677  1.357029247  0.678706371 &#xA;&amp;gt; wholestats(doutlier,a*b) &#xA;         mean      variance        st.dev    median.50%       IQR.75%           MAD &#xA;-3.425664e+00  9.591583e+05  9.793663e+02 -1.610677e-03  1.373658e+00  6.871415e-01 &#xA;&amp;gt; rowstats(d,b)&#xA;        mean     variance       st.dev       median          IQR          MAD &#xA;-0.002440456  1.014611308  0.947630870  0.003460172  0.917642167  0.510115277 &#xA;&amp;gt; rowstats(doutlier,b) &#xA;         mean      variance        st.dev        median           IQR           MAD &#xA;-3.425664e+00  9.607212e+05  1.685929e+02  3.460172e-03  9.301795e-01  5.175084e-01 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2958" LastEditorUserId="2958" LastEditorDisplayName="" LastEditDate="2011-04-24T02:09:31.230" LastActivityDate="2011-04-24T02:09:31.230" CommentCount="2" />
  <row Id="9907" PostTypeId="1" CreationDate="2011-04-23T17:03:22.900" Score="1" ViewCount="45" Body="&lt;p&gt;This questions consists of two parts that are quite similar and concern conditional probability.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Firstly, I would like to confirm the calculation of the conditional probability when we know the value of a random variable. Let's assume that we have a conditional matrix which gives us the values for $P(D|A\&amp;amp;B)$ and that B can take the values $(b0, b1)$. If we know that $B = b0$, then if we have to calculate $P(D|A)$, it would just be $P(D|A\&amp;amp;b0)$, right?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's now assume that we have a conditional probability matrix giving us $P(F|G\&amp;amp;H)$. If we assume that $F$ is independent from $H$, how can we calculate $P(F|G)$ only using $P(F|G\&amp;amp;H)$?&lt;/p&gt;&#xA;" OwnerUserId="4297" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-23T18:04:16.397" LastActivityDate="2011-04-23T18:04:16.397" Title="Conditional probability and instantiation" Tags="&lt;conditional-probability&gt;" CommentCount="1" />
  <row Id="9908" PostTypeId="1" CreationDate="2011-04-23T18:40:20.073" Score="-1" ViewCount="33" Body="&lt;p&gt;I have a question about the &lt;a href=&quot;http://cran.r-project.org/web/packages/forecast/index.html&quot; rel=&quot;nofollow&quot;&gt;forecast&lt;/a&gt; package return values. I see it print the point estimate and then low and high values for confidence intervals. I just want to get this first column in a separate vector. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  Point       Forecast       Lo 80     Hi 80       Lo 95     Hi 95&#xA;  10229      -2.237542 -5.81116452  1.336080 -7.70292589  3.227841&#xA;  10230       1.683324 -3.19974731  6.566396 -5.78468918  9.151337&#xA;  10231       3.893685 -1.32257692  9.109948 -4.08389942 11.871270&#xA;  .......&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When I say &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; typeof(fc)&#xA;[1] &quot;list&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So I tryed &lt;code&gt;fc$Forecast&lt;/code&gt;, but it returns &lt;code&gt;NULL&lt;/code&gt;. This does look like a stackoverflow question, but I figured this community will be better aquainted with the output of this particular package.&lt;/p&gt;&#xA;" OwnerUserId="4280" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-23T20:09:46.783" LastActivityDate="2011-04-23T20:09:46.783" Title="How to print a single column in the output of forecast function in R forecast package?" Tags="&lt;r&gt;&lt;time-series&gt;&lt;forecasting&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="9909" PostTypeId="2" ParentId="9908" CreationDate="2011-04-23T20:07:49.607" Score="1" ViewCount="" Body="&lt;p&gt;Although I never used the &lt;code&gt;forecast&lt;/code&gt; package, looking at the output of &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ts1 &amp;lt;- ets(USAccDeaths)&#xA;fc &amp;lt;- forecast(ts1, h=48)&#xA;str(fc)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;it seems that what you are looking for is just stored as&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;fc$mean&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;while &lt;code&gt;fc$lower&lt;/code&gt; (resp. &lt;code&gt;fc$upper&lt;/code&gt;) will give you 80 and 95% lower (resp. upper) limits for prediction. Note that you'll have to replace &lt;code&gt;fc&lt;/code&gt; with the name of your variable, &lt;code&gt;fit&lt;/code&gt;. Note also that this is &lt;em&gt;fully documented&lt;/em&gt; in the on-line help for &lt;code&gt;forecast&lt;/code&gt; (see returned value):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;     An object of class ‘&quot;forecast&quot;’ is a list containing at least the&#xA;     following elements:&#xA;&#xA;   model: A list containing information about the fitted model    &#xA;  method: The name of the forecasting method as a character string&#xA;    mean: Point forecasts as a time series    &#xA;   lower: Lower limits for prediction intervals    &#xA;   upper: Upper limits for prediction intervals&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-04-23T20:07:49.607" />
  <row Id="9911" PostTypeId="1" CreationDate="2011-04-23T20:27:33.923" Score="1" ViewCount="87" Body="&lt;p&gt;Hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm new in stats, and maybe this is a duplicated question, but I could not find a similar one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm trying to reduce a dimension of my dataset. Maybe &lt;em&gt;reduce&lt;/em&gt; is not a good word. I need to sample some of my dimensions.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Setup&lt;/strong&gt;    &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example:  &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;I have $M$ events (let's say $M \tilde{} 60$). They are ALL labeled.  &lt;/li&gt;&#xA;&lt;li&gt;I have $K$ trials/repetitions (let's say $K \tilde{} 10$ or more). $K$ different sets of $M$ events.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Therefore, I have a matrix $K\times M$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I need to choose $N$ events among $M$ of them as my representatives. I don't need to reduce the dataset,&#xA;I just need to pick $N$ columns (events) from the matrix $K\times M$. These $N$ events should reflect the behavior of particular trial. I'm interesting in corresponding labels of $N$ chosen events. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know that events are quite different, and probably if there is some correlation it's safe to assume that is a nonlinear and in the best case maybe linear.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Further, I want the best representation. Meaning $N$ is not an input (of course $N &amp;lt; M$).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Discussion&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I considered PCA, kernel PCA and sampling.&lt;br&gt;&#xA;After some reading, I found the following:  &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;PCA&lt;/strong&gt; will reduce my dataset, but I just need to pinpoint the events. Even if I brake down the algorithm, just choose the appropriate eigenvalues, and reverse back to the original dataset, PCA still remains as linear projection.  &lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;kernel PCA&lt;/strong&gt; is basically nonlinear, but still it will reduce my dataset instead of choosing particular event.  &lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;sampling&lt;/strong&gt;, I went through different types (random, systematic, stratified, cluster ...) but I could not find the particular type. I need something to capture nonlinear connection between the events.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Any pointer or explanation of the problem would be much appreciated. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="1313" LastEditorUserId="2148" LastEditorDisplayName="" LastEditDate="2011-04-25T08:24:48.713" LastActivityDate="2011-04-25T18:15:06.373" Title="Sampling dataset, choosing among N dimensions" Tags="&lt;pca&gt;&lt;sampling&gt;&lt;large-data&gt;" AnswerCount="2" CommentCount="3" FavoriteCount="2" />
  <row Id="9913" PostTypeId="1" CreationDate="2011-04-23T18:08:25.280" Score="0" ViewCount="80" Body="&lt;p&gt;In the log-log regression case,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\log(Y) = B_0 + B_1 \log(X) + U \&amp;gt;,$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;can you show $B_1$ is the elasticity of $Y$ with respect to $X$, i.e., that $E_{yx} = \frac{dY}{dX}(\frac{Y}{X})$?&lt;/p&gt;&#xA;" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-04-24T23:06:07.013" LastActivityDate="2011-04-24T23:06:07.013" Title="Elasticity of log-log regression" Tags="&lt;regression&gt;&lt;homework&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="9914" PostTypeId="2" ParentId="9876" CreationDate="2011-04-23T21:47:19.023" Score="4" ViewCount="" Body="&lt;p&gt;There isn't going to be one best answer for this kind of sampling.  It depends on the observable covariates in your sampling frame, the variables you expect to be important determinants of survey response, and the analysis you want to run once the survey is complete.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With that said, there are a couple of general principles that can help guide your sampling strategy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For descriptive surveys, you generally want your sample to closely resemble the population of interest in as many ways as possible.  This will help &lt;a href=&quot;http://en.wikipedia.org/wiki/Weighted_mean#Dealing_with_variance&quot; rel=&quot;nofollow&quot;&gt;keep your weights even&lt;/a&gt;, in order to maximize your effective sample size.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you intend to do multivariate analysis, you may want to stratify on important variables of interest.  This will increase variance in your IVs and DVs, and can help increase your &lt;a href=&quot;http://en.wikipedia.org/wiki/Statistical_power&quot; rel=&quot;nofollow&quot;&gt;statistical power&lt;/a&gt; in later analysis. This is why some studies conduct oversamples of minority populations -- because race and ethnicity are important IVs in many analyses. &lt;a href=&quot;http://en.wikipedia.org/wiki/Case-control_study&quot; rel=&quot;nofollow&quot;&gt;Case-control studies&lt;/a&gt; follow a similar logic for stratifying on the dependent variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you intend to do description &lt;i&gt;and&lt;/i&gt; analysis, then these are goals will be partly at odds.  No matter what, you need to follow the basic principle of sampling and make sure that every individual in the population has a known, non-zero chance of being selected into the sample.  Advanced topics worth looking up in this area include &lt;a href=&quot;http://en.wikipedia.org/wiki/Propensity_score_matching&quot; rel=&quot;nofollow&quot;&gt;propensity scores&lt;/a&gt;, and sample weighting via raking.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Closing thought: these are general principles for sampling design and sample weighting.  You don't say much about your specific application, but I'm guessing that most of this is overkill.  The main reason to stratify a sample is if you have reason to believe a simple random sample will miss out on some important group of interest (geographic, demographic, or otherwise).  That is, the sampled population would be too small for useful analysis. If you don't have that problem, then you don't need to worry too much about stratification.&lt;/p&gt;&#xA;" OwnerUserId="4110" LastEditorUserId="4110" LastEditorDisplayName="" LastEditDate="2011-04-25T18:05:38.407" LastActivityDate="2011-04-25T18:05:38.407" />
  <row Id="9915" PostTypeId="2" ParentId="9885" CreationDate="2011-04-23T22:26:56.633" Score="7" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Yes&lt;/strong&gt;, I think tag prediction is an interesting one and one for which you have a good shot at &quot;success&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Below are some thoughts intended to potentially aid in brainstorming and further exploration of this topic. I think there are many potentially interesting directions that such a project could take. I would guess that a serious attempt at just one or two of the below would make for a more than adequate project and you're likely to come up with more interesting questions than those I've posed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm going to take a very wide view as to what is considered &lt;em&gt;machine learning&lt;/em&gt;. Undoubtedly some of my suggestions would be better classified as &lt;em&gt;exploratory data analysis&lt;/em&gt; and more traditional &lt;em&gt;statistical analysis&lt;/em&gt;. But, perhaps, it will help in some small way as you formulate your own interesting questions. You'll note, I try to address questions that I think would be interesting in terms of enhancing the functionality of the site. Of course, there are many other interesting questions as well that may not be that related to site friendliness.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;strong&gt;Basic descriptive analysis of user behavior&lt;/strong&gt;: I'm guessing there is a very clear cyclic weekly pattern to user participation on this site. When does the site get the most traffic? What does the graph of user participation on the site look like, say, stratified by hour over the week? You'd want to adjust for potential changes in overall popularity of the site over time. This leads to the question, how has the site's popularity changed since inception? How does the participation of a &quot;typical&quot; user vary with time since joining? I'm guessing it ramps up pretty quickly at the start, then plateaus, and probably heads south after a few weeks or so of joining.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Optimal submission of questions and answers&lt;/strong&gt;: Getting insight on the first question seems to naturally lead to some more interesting (in an ML sense) questions. Say I have a question I need an answer to. If I want to maximize my probability of getting a response, when should I submit it? If I am responding to a question and I want to maximize my vote count, when should I submit my answer? Maybe the answers to these two are very different. How does this vary by the topic of the question (say, e.g., as defined by the associated tags)?&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Biclustering of users and topics&lt;/strong&gt;: Which users are most alike in terms of their interests, again, perhaps as measured by tags? What topics are most similar according to which users participate? Can you come up with a nice visualization of these relationships? Offshoots of this would be to try to predict which user(s) is most likely to submit an answer to a particular question. (Imagine providing such technology to SE so that users could be notified of potentially interesting questions, not simply based on tags.)&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Clustering of answerers by behavior&lt;/strong&gt;: It seems that there are a few different basic behavioral patterns regarding how answerers use this site. Can you come up with features and a clustering algorithm to cluster answerers according to their behavior. Are the clusters interpretable?&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Suggesting new tags&lt;/strong&gt;: Can you come up with suggestions for &lt;em&gt;new&lt;/em&gt; tags based on inferring topics from the questions and answers currently in the database. For example, I believe the tag [mixture-model] was recently added because someone noticed we were getting a bunch of related questions. But, it seems an information-retrieval approach should be able to extract such topics directly and potentially suggest them to moderators.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Semisupervised learning of geographic locations&lt;/strong&gt;: (&lt;em&gt;This one may be a bit touchy from a privacy perspective.&lt;/em&gt;) Some users list where they are located. Others do not. Using usage patterns and potentially vocabulary, etc, can you put a geographic confidence region on the location of each user? Intuitively, it would seem that this would be (much) more accurate in terms of longitude than latitude.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Automated flagging of possible duplicates and highly related questions&lt;/strong&gt;: The site already has a similar sort of feature with the &lt;strong&gt;Related&lt;/strong&gt; bar in the right margin. Finding nearly exact duplicates and suggesting them could be useful to the moderators. Doing this &lt;em&gt;across&lt;/em&gt; sites in the SE community would seem to be new.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Churn prediction and user retention&lt;/strong&gt;: Using features from each user's history, can you predict the next time you expect to see them? Can you predict the probability they will return to the site conditional on how long they've been absent and features of their past behavior? This could be used, e.g., to try to notice when users are at risk of &quot;churn&quot; and engage them (say, via email) in an effort to retain them. A typical approach would shoot out an email after some fixed period of inactivity. But, each user is very different and there is lots of information about lots of users, so a more tailored approach could be developed.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="2970" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-04-25T23:08:59.523" LastActivityDate="2011-04-25T23:08:59.523" CommentCount="5" />
  <row Id="9916" PostTypeId="2" ParentId="9876" CreationDate="2011-04-23T23:48:58.730" Score="3" ViewCount="" Body="&lt;p&gt;You basically run a pilot study so you can guess which strata are more variable and then oversample on those strata.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Acquire a book on sampling. (Maybe check out the QA 278 section of a university library. I have Sampling of Populations by Levy and Lemeshow, but I'm sure others are fine.) Look at sample size estimation for stratified sampling. The simpler approaches are&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Equal sample size in each stratum&lt;/li&gt;&#xA;&lt;li&gt;Sample size proportional to stratum size&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You're more interested in these fancy sampling size estimation schemes&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Variance minimzation&lt;/li&gt;&#xA;&lt;li&gt;Cost minimization (in case it costs more money to sample one stratum than another)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="3874" LastEditorUserId="3874" LastEditorDisplayName="" LastEditDate="2011-04-24T00:05:45.970" LastActivityDate="2011-04-24T00:05:45.970" />
  <row Id="9917" PostTypeId="1" CreationDate="2011-04-24T03:17:39.473" Score="3" ViewCount="57" Body="&lt;p&gt;I was actually looking at this &lt;a href=&quot;http://math.mit.edu/~jsoto/localpapers/spams2010.pdf&quot; rel=&quot;nofollow&quot;&gt;problem&lt;/a&gt; on slide 12. I will write it here briefly: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Problem:&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unknown number of people arriving in a fixed time period and my goal is to maximize my probability of picking the best candidate&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Assumptions:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Assume people arrive in interval [0,1] independently&lt;/li&gt;&#xA;&lt;li&gt;May assume also uniformly&lt;/li&gt;&#xA;&lt;li&gt;Can NOT beat probability $\frac{1}{e}$ to win.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Proof:&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Fix a wall at time $T$ and select the best candidate after $T$. Let $t$ be the time at which the best candidate arrives, then:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Prob(Win) \geq E_{t}[1_{\{t&amp;gt;T\}}\frac{T}{t}] = \int_{T}^{1}\frac{T}{t}dt = -T~ln~T$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is optimum at $T=\frac{1}{e}$ with a $Prob(Win) = \frac{1}{e} = 36.7\%$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a few questions. Can someone please explain how to interpret the expectation equation? In addition, what is the significance of the statement &quot;&lt;em&gt;Assume people arrive in interval $[0,1]$ independently.&lt;/em&gt;&quot;? Does it mean that I cannot consider some other time interval, say $[0, \alpha]$ or do I need to just multiply $\alpha$ by $\frac{1}{e}$ to get the actual time of making the decision?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am just trying to figure out what I should do to convert this into an actual real-world implementation. Any suggestions?&lt;/p&gt;&#xA;" OwnerUserId="2164" LastEditorUserId="2164" LastEditorDisplayName="" LastEditDate="2011-04-24T03:25:03.007" LastActivityDate="2011-05-24T13:49:46.970" Title="Meaning of this expectation equation?" Tags="&lt;probability&gt;&lt;algorithms&gt;&lt;stochastic-processes&gt;&lt;expected-value&gt;" AnswerCount="1" />
  <row Id="9918" PostTypeId="1" AcceptedAnswerId="9944" CreationDate="2011-04-24T03:41:44.080" Score="3" ViewCount="211" Body="&lt;p&gt;I have a matrix of 1000 observations and 50 variables each measured on a 5-point scale.  These variables are organized into groups, but there aren't an equal number of variables in each group.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd like to calculate two types of correlations:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Correlation within groups of variables (among characteristics): some measure of whether the variables within the group of variables are measuring the same thing.&lt;/li&gt;&#xA;&lt;li&gt;Correlation between groups of variables: some measure, assuming that each group reflects one overall trait, of how each trait (group) is related to every other trait.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;These characteristics have been previously classified into groups.  I'm interested in finding the correlation between the groups - i.e. assuming that the characteristics within in group are measuring the same underlying trait (having completed #1 above - Cronbach's alpha), are the traits themselves related?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anybody have suggestions for where to start?&lt;/p&gt;&#xA;" OwnerUserId="4301" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-25T08:26:32.587" LastActivityDate="2011-04-25T08:26:32.587" Title="How to compute correlation between/within groups of variables?" Tags="&lt;correlation&gt;&lt;psychometrics&gt;&lt;scales&gt;" AnswerCount="4" CommentCount="2" />
  <row Id="9919" PostTypeId="2" ParentId="4609" CreationDate="2011-04-24T04:00:58.850" Score="0" ViewCount="" Body="&lt;p&gt;Double-click the output to obtain the test statistics box.&lt;/p&gt;&#xA;" OwnerUserId="2025" LastActivityDate="2011-04-24T04:00:58.850" CommentCount="1" />
  <row Id="9920" PostTypeId="1" CreationDate="2011-04-24T05:03:58.663" Score="2" ViewCount="76" Body="&lt;p&gt;My question concerns the assumption of additivity for intraclass correlation. I shall first explain what I have done and then end with my questions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to calculate inter-rater reliability using intra-class correlation so I can report an overall coefficient (as done in previous similar research), and perhaps replace a rater if their judgements correspond poorly to the other raters. I have five raters and they have each rated video recordings of facial and vocal expressions of the same (randomly sampled) 4 participants in an experiment where the participants watched different emotional films.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Raters make 18 ratings per film. These ratings are Likert-type (generally ranging from 1-6 but for some measures 1-4) of the intensity of different 6 facial emotional expressions (anger, fear etc), the intensity of facial expression overall, and the number (frequency ratings) and intensity (Likert ratings) of positive and negative words and sounds, and level of overall vocal expressiveness.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are 16 films, so there is a total of 288 variables per rater, per participant rated. I have organised my data into four files, one per participant being rated, with each rater as a column and the 288 variables as rows. As I am calculating inter-rater reliability, I am interested in the similarity of the raters overall, and not any other (e.g. film) effects.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have calculated the ICC using the mixed model because all judges rate all targets, which are a random sample (as per &lt;a href=&quot;http://faculty.chass.ncsu.edu/garson/PA765/reliab.htm#rater&quot; rel=&quot;nofollow&quot;&gt;http://faculty.chass.ncsu.edu/garson/PA765/reliab.htm#rater&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Questions:&lt;/strong&gt;&#xA;The assumption of additivity states that each item should be linearly related to the total score. However I don’t think that the concept of a total score really applies, although I may be wrong. Tukey’s test of non-additivity tests the null hypothesis that there is no multiplicative interaction between cases and items.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Could somebody please explain this to&#xA;me in simple terms?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I found a significant Tukey’s test value, so I tried removing the overall facial and vocal ratings for each film, as I thought this perhaps violated the requirement that each item contributes to the total score. However Tukey’s test remained significant. So just as a little experiment, I removed 282 variables, leaving me with ratings of the 6 possible facial emotions for a single film. Tukey’s test was still significant!&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Is Tukey’s test of non-additivity&#xA;relevant to my problem?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If yes, what should I do about it &#xA;being significant?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="2025" LastActivityDate="2011-05-02T13:55:43.327" Title="Assumption of additivity for intra-class correlation" Tags="&lt;reliability&gt;&lt;inter-rater&gt;&lt;intraclass-correlation&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="1" />
  <row Id="9922" PostTypeId="2" ParentId="856" CreationDate="2011-04-24T08:04:05.637" Score="0" ViewCount="" Body="&lt;p&gt;One quick thing about sample weights - they are usually a way to incorporate some information about the population that one is sampling from - but usually they are based on &quot;big sample&quot; type scenarios (typically constrained BLUP or BLUE prediction in disguise).  So I would imagine that sample weights will probably do no better than no weights.  What would be better I think is to use the information about the population that the sample design was based on directly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, on what basis were the selection probabilities calculated?  My bet is that you knew a population total or some kind of population break-down which does not involve A or B (say age by sex groups).  If this is not correct then I am about to waste some space, but if it is correct, and supposing you had population totals $R_{1},\dots,R_{k}$ for $k$ groups (or strata), and within each group you had a &quot;mini&quot; 2 by 2 contingency table.  So we can now write $R_{1;11},R_{1;12},R_{1;21},R_{1;22},\dots$ as the &quot;target&quot; of our inference.  Or perhaps it is the sum $\sum_{l=1}^{k}R_{l;ij}$ that is the target of inference (how many in the population give response N/N??).  You are then trying to reason about $R_{l;ij}$ from the sampled numbers $r_{l;ij}$ subject to the constraint that $\sum_{i,j}R_{l;ij}=R_{l}$ for $(l=1,\dots,k)$.  (maxent anyone?)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that if the sampling probabilities were based only on what data you were likely to receive, then they are irrelevant (and Fisher's exact test applies), because once you receive the data, you know what sample you received.  So the coherent thing to do is to update the sampling probability to $P(D_{m})=1$ if the mth unit is in the sample, and $P(D_{m})=0$ if they weren't in the sample.  However, usually the design is based in more information than just the data one is likely to observe.  but note that it is the information rather than the survey design per se that is important.  Design based inference is just a rather efficient way to incorporate all that information into your analysis.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-04-24T08:04:05.637" />
  <row Id="9923" PostTypeId="2" ParentId="9233" CreationDate="2011-04-24T11:34:08.887" Score="1" ViewCount="" Body="&lt;p&gt;The required number of observations to Identify a model depends on the ratio of signal to noise in the data and the form of the model. If I am given the numbers ,1,2,3,4,5 , I will predict 6,7,8,.... Box-Jenkins model identification is an approach to determine the underlying General Term much like the test for &quot;numerical intelligence&quot; that we give to children. If the signal is strong then we need fewer observations and vice-versa. If the observed frequency suggests a possible &quot;seasonal structure&quot; then we need repetitions of this phenomena e.g. at least 3 seasons ( preferably more ) as a rule of thumb to extract (identify this from the basic descriptive statistics (the acf/pacf ).&lt;/p&gt;&#xA;" OwnerUserId="3382" LastActivityDate="2011-04-24T11:34:08.887" />
  <row Id="9924" PostTypeId="2" ParentId="9917" CreationDate="2011-04-24T13:40:16.647" Score="2" ViewCount="" Body="&lt;p&gt;Rather than &quot;Fix a wall at time $T$ and select the best candidate after $T$&quot;, I think you may be using &quot;Fix a wall at time $T$ and select the first candidate to arrive at or after $T$ who is better than all those arriving before time $T$&quot;.  This is a continuous version of the &lt;a href=&quot;http://en.wikipedia.org/wiki/Secretary_problem&quot; rel=&quot;nofollow&quot;&gt;secretary problem&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$E_{t}[1_{\{t&amp;gt;T\}}\frac{T}{t}]$ can be interpreted in two parts: $1_{\{t&amp;gt;T\}}$ is an indicator that the best candidate arrives at or after time $T$, i.e. you have not already rejected the best overall candidate; $\frac{T}{t}$ is the probability that the best candidate to arrive strictly before time $t$ arrives before time $T$, i.e. that you have not selected a candidate who is not the best overall. The expectation of the indicator of an event is the probability of that event.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The interval $[0,1]$ is simply the time during which you see the candidates.  You can scale this to other units, so if you are looking at time $[0,\alpha]$ then your $T=\frac{1}{e}$ will be scaled to $\frac{\alpha}{e}$.  &lt;/p&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-04-24T13:40:16.647" CommentCount="1" />
  <row Id="9925" PostTypeId="2" ParentId="9913" CreationDate="2011-04-24T14:05:19.113" Score="2" ViewCount="" Body="&lt;p&gt;whuber has made the point in the comment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If $\log_e(Y) = B_0 + B_1\log_e(X) + U$ and $U$ is independent of $X$ then taking the partial derivative with respect to $X$ gives $\frac{\partial Y}{\partial X}\cdot\frac{1}{Y} = B_1\frac{1}{X}$, i.e. $B_1 = \frac{\partial Y}{\partial X}\cdot\frac{X}{Y}$.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$E_{y,x} = \lim_{X \rightarrow x} \frac { \Delta Y} { y} / \frac { \Delta X} { x}$, which is the same thing.  Take absolute values if you want to avoid negative elasticities.&lt;/p&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-04-24T14:05:19.113" />
  <row Id="9926" PostTypeId="1" AcceptedAnswerId="9975" CreationDate="2011-04-24T15:05:10.177" Score="4" ViewCount="149" Body="&lt;p&gt;Is there any use for the quantity&#xA;$$&#xD;&#xA;\int f(x)^2 dx&#xD;&#xA;$$&#xA;in statistics or information theory?&lt;/p&gt;&#xA;" OwnerUserId="3567" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-04-29T03:05:15.267" LastActivityDate="2011-04-29T03:05:15.267" Title="Is there any use for the quantity $\int f(x)^2 dx$ in statistics or information theory?" Tags="&lt;probability&gt;&lt;entropy&gt;&lt;information-theory&gt;" AnswerCount="1" CommentCount="6" FavoriteCount="0" />
  <row Id="9927" PostTypeId="2" ParentId="9918" CreationDate="2011-04-24T15:48:08.573" Score="2" ViewCount="" Body="&lt;p&gt;The way I read your terminology, what you want is first to assess internal consistency within each group of variables, and then to assess the correlations among the scale scores which constitute the average of each group of variables.  So the first can be done using Cronbach's alpha, and the second using Pearson correlation.  This assumes you have reasonably normal distributions and reasonably linear relationships.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A more involved method, and not necessarily a required one, would be to conduct an exploratory factor analysis.  You would try to establish which variables should be grouped together and then again to what degree those factors would be correlated.  If you try this method, make sure you use oblique rotation to allow those correlations to show up.  Whether you use principal components extraction or principal axis extraction would depend, respectively, on whether your variables are objective, error-free measurements or subjective ones such as survey items that contain a certain amount of error.&lt;/p&gt;&#xA;" OwnerUserId="2669" LastActivityDate="2011-04-24T15:48:08.573" CommentCount="1" />
  <row Id="9928" PostTypeId="1" AcceptedAnswerId="9936" CreationDate="2011-04-24T18:07:51.337" Score="2" ViewCount="93" Body="&lt;p&gt;What &quot;fat-tailed distributions&quot; $p(x)$, symmetric about zero, have the property&#xA;$$\newcommand{\e}{\mathbb{E}}\newcommand{\rd}{\mathrm{d}}&#xD;&#xA;\e e^X = \int_{-\infty}^{\infty} e^x p(x) \rd x &amp;lt; \infty \&amp;gt; ?&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Context&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm attempting to price financial options for $X$ without using the Black&amp;ndash;Scholes formula. It is usually easier to work with the log-price $Y = \log(X)$ and often it is assumed that $Y$ is normally distributed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Empirical observations (eg, the &quot;volatility smile&quot;) suggest that $Y$ isn't normal; the normal distribution decreases too rapidly away from 0. Thus, we need a fat-tailed distribution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The value of a call option increases exponentially as $Y$ increases linearly. Therefore, $p(x)$ must decline fast enough that $\e e^X &amp;lt; \infty$. In other words, the distribution must decline slower than the normal distribution, but still fast enough that $\e e^X$ converges. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried the Cauchy and Student's $t$ distributions, but $\e e^X$ diverges for both, regardless of parameters. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also realize I can create arbitrary distributions meeting my conditions (though I'm not exactly sure how), but I'm looking for a well-known (parametrized family of) distribution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even more details (for the masochist): &#xA;&lt;a href=&quot;https://github.com/barrycarter/bcapps/blob/master/bc-imp-vol.m&quot; rel=&quot;nofollow&quot;&gt;https://github.com/barrycarter/bcapps/blob/master/bc-imp-vol.m&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="1566" LastEditorUserId="1566" LastEditorDisplayName="" LastEditDate="2011-04-27T16:44:25.453" LastActivityDate="2011-04-27T16:44:25.453" Title="Symmetric fat-tailed distributions where $\mathbb{E} e^X &lt; \infty$" Tags="&lt;distributions&gt;" AnswerCount="2" CommentCount="8" />
  <row Id="9930" PostTypeId="1" CreationDate="2011-04-24T20:08:52.457" Score="11" ViewCount="184" Body="&lt;p&gt;This is probably demonstrating a fundamental lack of understanding of how partial correlations work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have 3 variables, x,y,z. When I control for z, the correlation between x and y increases over the correlation between x and y when z was not controlled for.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does this make sense? I tend to think that when one controls for the effect of a 3rd variable, the correlation should decreases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you for your help!&lt;/p&gt;&#xA;" OwnerUserId="4307" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-29T00:49:44.813" LastActivityDate="2011-04-29T07:20:04.600" Title="Does it make sense for a partial correlation to be larger than a zero-order correlation?" Tags="&lt;correlation&gt;" AnswerCount="4" FavoriteCount="3" />
  <row Id="9931" PostTypeId="1" AcceptedAnswerId="9955" CreationDate="2011-04-24T18:56:26.757" Score="3" ViewCount="125" Body="&lt;p&gt;I have a problem where I need to calculate linear regression as samples come in. Is there a formula that I can use to get the exponentially weighted moving linear regression? Not sure if that's what you would call it though.&lt;/p&gt;&#xA;" OwnerUserId="4306" LastEditorUserId="4306" LastEditorDisplayName="" LastEditDate="2011-04-24T20:24:05.063" LastActivityDate="2011-04-25T16:20:15.547" Title="Exponentially weighted moving linear regression" Tags="&lt;regression&gt;" AnswerCount="4" CommentCount="1" />
  <row Id="9932" PostTypeId="2" ParentId="9930" CreationDate="2011-04-24T20:53:18.513" Score="1" ViewCount="" Body="&lt;p&gt;I think you need to know about moderator and mediator variables. The classic paper is Baron and Kenny [cited 21,659 times]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A moderator variable &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&quot;In general terms, a moderator is a&#xA;  qualitative (e.g., sex, race, class)&#xA;  or quantitative (e.g., level of&#xA;  reward) variable that affects the&#xA;  direction and/or strength of the&#xA;  relation between an independent or&#xA;  predictor variable and a dependent or&#xA;  criterion variable. Specifically&#xA;  within a correlational analysis&#xA;  framework, a moderator is a third&#xA;  variable that affects the zero-order&#xA;  correlation between two other&#xA;  variables. ... In the more familiar&#xA;  analysis of variance (ANOVA) terms, a&#xA;  basic moderator effect can be&#xA;  represented as an interaction between&#xA;  a focal independent variable and a&#xA;  factor that specifies the appropriate&#xA;  conditions for its operation.&quot; &lt;a href=&quot;http://psych.wisc.edu/henriques/mediator.html&quot; rel=&quot;nofollow&quot;&gt;p.&#xA;  1174&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;A mediator variable &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&quot;In general, a given variable may be&#xA;  said to function as a mediator to the&#xA;  extent that it accounts for the&#xA;  relation between the predictor and the&#xA;  criterion. Mediators explain how&#xA;  external physical events take on&#xA;  internal psychological significance.&#xA;  Whereas moderator variables specify&#xA;  when certain effects will hold,&#xA;  mediators speak to how or why such&#xA;  effects occur.&quot; p. 1176&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="3597" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-25T05:19:38.197" LastActivityDate="2011-04-25T05:19:38.197" CommentCount="1" />
  <row Id="9933" PostTypeId="2" ParentId="9931" CreationDate="2011-04-24T21:27:08.933" Score="2" ViewCount="" Body="&lt;p&gt;Sure, just add a &lt;code&gt;weights=&lt;/code&gt; argument to &lt;code&gt;lm()&lt;/code&gt; (in case of &lt;a href=&quot;http://www.r-project.org&quot; rel=&quot;nofollow&quot;&gt;R&lt;/a&gt;):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;R&amp;gt; x &amp;lt;- 1:10    ## mean of this is 5.5&#xA;R&amp;gt; lm(x ~ 1)    ## regression on constant computes mean&#xA;&#xA;Call:&#xA;lm(formula = x ~ 1)&#xA;&#xA;Coefficients:&#xA;(Intercept)  &#xA;        5.5  &#xA;&#xA;R&amp;gt; lm(x ~ 1, weights=0.9^(seq(10,1,by=-1)))&#xA;&#xA;Call:&#xA;lm(formula = x ~ 1, weights = 0.9^(seq(10, 1, by = -1)))&#xA;&#xA;Coefficients:&#xA;(Intercept)  &#xA;       6.35  &#xA;&#xA;R&amp;gt; &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here is give 'more recent' (&lt;em&gt;i.e.&lt;/em&gt;, higher) values more weight and the mean shifts from 5.5 to 6.35.  The key, if any, is the $\lambda ^ \tau$ exponential weight I compute on the fly; you can change the weight factor to any value you choose and depending on how you order your data you can also have the exponent run the other way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can do the same with regression models involving whichever regressors you have.&lt;/p&gt;&#xA;" OwnerUserId="334" LastActivityDate="2011-04-24T21:27:08.933" CommentCount="3" />
  <row Id="9934" PostTypeId="2" ParentId="9918" CreationDate="2011-04-24T22:34:58.023" Score="1" ViewCount="" Body="&lt;p&gt;I would suggest using as a replacement for the notion of correlation, which is defined only for pair-wise, the notion of mutual information and integration in Gaussian models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In Gaussian models, &lt;em&gt;integration&lt;/em&gt; of a group of variables $G_1$ is defined as the entropy of the group:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$I_1 \propto log(|C_1|)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $C_1$ is the correlation matrix of the group of variables $G_1$. It is easy to see that if $G_1$ is comprised only of 2 variables, its integration is $log ( 1 - \rho^2)$, which directly relates to the pairwise correlation coefficient of the variables $\rho$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To compute interaction between two groups of variables, you can use mutual information, which is just cross-entropy between the groups:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$MU_{12} = I_{12} - I_{1} - I_{2}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found &lt;a href=&quot;http://www.imt.liu.se/~magnus/cca/tutorial/node16.html&quot; rel=&quot;nofollow&quot;&gt;a reference&lt;/a&gt; on these notions after a quick google that might be helpful.&lt;/p&gt;&#xA;" OwnerUserId="1265" LastActivityDate="2011-04-24T22:34:58.023" CommentCount="1" />
  <row Id="9935" PostTypeId="2" ParentId="9895" CreationDate="2011-04-25T00:02:07.613" Score="1" ViewCount="" Body="&lt;p&gt;I think this is is not a good question: first having $N_e$ on both sides of the model, and second I suspect it is ill conditioned.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are my explorations.  Let's use R to try to minimise the sum of squares of the difference between the right hand side and $N_e$ by setting up two functions with &lt;code&gt;param[1]=b&lt;/code&gt;, &lt;code&gt;param[2]=c&lt;/code&gt;, &lt;code&gt;param[3]=d&lt;/code&gt; and &lt;code&gt;param[4]=T&lt;/code&gt; &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;RHS &amp;lt;- function(par) { &#xA;        Data$No * (1 - exp( (par[3] + par[1]*Data$No) * (par[4]*Data$Ne - 72)&#xA;                           / (1 + par[2]*Data$No) ) )&#xA;     }&#xA;sumsq &amp;lt;- function(par) { &#xA;     sum( ( Data$Ne - RHS(par) )^2 )&#xA;     }&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;We now need to start from some initial estimate, and this one seems to work quite well&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;param &amp;lt;- c(2,3,4,11)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;We can then use &lt;code&gt;optim&lt;/code&gt;.  We could set up the constraints, either with &lt;code&gt;optim&lt;/code&gt; or with &lt;code&gt;constrOptim&lt;/code&gt;, but with luck we can find a solution which meets the constraints anyway.  We may need to run &lt;code&gt;optim&lt;/code&gt; several times until the estimates of the parameters stop changing. For example&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; o &amp;lt;- optim(param, sumsq) &#xA;&amp;gt; (param &amp;lt;- o$par)&#xA;[1] -0.200  3.825  4.825 11.825&#xA;&amp;gt; o &amp;lt;- optim(param, sumsq) &#xA;&amp;gt; (param &amp;lt;- o$par)&#xA;[1]  6.081155e-05  1.821490e+00 -2.820120e-02  9.624836e+01&#xA;&amp;gt; o &amp;lt;- optim(param, sumsq) &#xA;&amp;gt; (param &amp;lt;- o$par)&#xA;[1]  5.278305e-05  3.244559e+00 -2.819326e-02  1.631129e+02&#xA;&amp;gt; o &amp;lt;- optim(param, sumsq) &#xA;&amp;gt; (param &amp;lt;- o$par)&#xA;[1]  5.278179e-05  3.244649e+00 -2.819326e-02  1.631168e+02&#xA;&amp;gt; o &amp;lt;- optim(param, sumsq) &#xA;&amp;gt; (param &amp;lt;- o$par)&#xA;[1]  5.278179e-05  3.244649e+00 -2.819326e-02  1.631168e+02&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and looking at the details we can see the sum of squares of the differences have been reduced to under 30, which is not bad given there are 89 data points&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; o&#xA;$par&#xA;[1]  5.278179e-05  3.244649e+00 -2.819326e-02  1.631168e+02&#xA;&#xA;$value&#xA;[1] 29.79642&#xA;&#xA;$counts&#xA;function gradient &#xA;     195       NA &#xA;&#xA;$convergence&#xA;[1] 0&#xA;&#xA;$message&#xA;NULL &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Using these parameters we can plot the right hand side of the model against the left hand side with a line to show where $y=x$ &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plot(RHS(param) ~ Data$Ne)&#xA;abline(0,1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/RmXB7.png&quot; alt=&quot;model against actual&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and that does not look too bad, though perhaps it should curve a bit lower for high $N_e$. The biggest differences come with the points &lt;code&gt;(No=30,Ne=20)&lt;/code&gt; and &lt;code&gt;(No=100,Ne=33)&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem comes when starting with other initial values for the parameters in the optimisation: some fail completely, some fail the constraints, and most worryingly some settle down on good but different values.  For example starting with &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;param &amp;lt;- c(2,3,4,9)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;the repeated &lt;code&gt;optim&lt;/code&gt; calls settle down on the parameters&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[1]  8.291305e-06  9.521925e-01 -4.292615e-03  3.153295e+02&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which are very different to the values found earlier.  The sum of squares of the differences are still under 34 and the graph looks almost identical, but if the parameters are supposed to have physical meaning then this is not a good way to estimate them; $Th$ has almost doubled from about 163 to 315.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the parameters produced can not be trusted. &lt;/p&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-04-25T00:02:07.613" CommentCount="2" />
  <row Id="9936" PostTypeId="2" ParentId="9928" CreationDate="2011-04-25T00:24:40.020" Score="1" ViewCount="" Body="&lt;p&gt;The variance gamma process is a useful way to go.  It is an extension of the standard brownian motion process, $Z(t)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Y(t;\mu,\sigma,\nu)=\mu \Gamma(t;1,\nu) + \sigma Z(\Gamma[t;1,\nu])$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $\Gamma(t;1,\nu)$ is a gamma process, with independent gamma distributed increments.  So $\Gamma(t+s;1,\nu)-\Gamma(t;1,\nu)$ has a gamma distribution with mean $s$ and variance $s\nu$.  (you could replace the $1$ by another parameter $\mu$ if desired, to have mean $s\mu$, as is done &lt;a href=&quot;http://www.math.nyu.edu/research/carrp/papers/pdf/VGEFRpub.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;).  The intuition is that &quot;time&quot; is measured in some sort of trading volume rather than calendar time.  So it incorporates a sense of the market &quot;getting hot&quot; and &quot;getting cold&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This process has the &quot;fat tail&quot; property that you seek, and is commonly used as an option pricing model.  It can be used by averaging the Black&amp;ndash;Scholes option price with respect to gamma distribution for the time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The increments of this distribution $Y(t)-Y(s)$ follow a Laplace distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More generally one speaks of what are called Levy process, which have characteristic function $\varphi_{y}(\theta)=E[\exp(i\theta Y_{t})]$:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\varphi_{t}(\theta)=\exp\left(it\mu\theta - \frac{t}{2}\sigma^{2}\theta^{2}+t\int_{|x|&amp;lt;\epsilon}[e^{i\theta x}-1-i\theta x]d\Pi(x)+t\int_{|x|&amp;gt;\epsilon}[e^{i\theta x}-1]d\Pi(x)\right)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $d\Pi(x)$ is called the Levy measure, and governs discontinuities in the process (or the &quot;jumps&quot;).  The measure must satisfy:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\int_{|x|&amp;lt;\epsilon}x^{2}d\Pi(x)&amp;lt;\infty$$&#xA;$$\int_{|x|&amp;gt;\epsilon}d\Pi(x)&amp;lt;\infty$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus it can have a singularity at zero as long it is not &quot;too big&quot;.  Can show for the variance gamma process the Levy measure is given by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$d\Pi(x)=\frac{dx}{\nu|x|}\exp(-\frac{1}{\nu}|x|)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In regards to Cardinal's well directed comments (I have a tendency to &quot;wander-off&quot; when I'm doing some sort of maths) you can see that the only criterion for your &quot;fat tailed&quot; distribution to satisfy your criterion is that the moment generating function $m_{X}(t)=E[\exp(tX)]$ exists when evaluated at $t=1$.  Equivalently we require the characteristic function to exist when evaluated at $\theta=-i$.  Because the Levy-process is defined by its CF then it is always finite when the two integral equations above are satisfied.  The expectation is given by the above formula&lt;/p&gt;&#xA;" OwnerUserId="2392" LastEditorUserId="2392" LastEditorDisplayName="" LastEditDate="2011-04-26T12:14:57.870" LastActivityDate="2011-04-26T12:14:57.870" CommentCount="7" />
  <row Id="9937" PostTypeId="1" AcceptedAnswerId="9938" CreationDate="2011-04-25T00:44:09.760" Score="3" ViewCount="128" Body="&lt;p&gt;In a data frame, I would like to get the column's index by name.  For example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x &amp;lt;- data.frame(foo=c('a','b','c'),bar=c(4,5,6),quux=c(4,5,6))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to know the column index for &quot;bar&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I came up with the following but it seems inelegant.  Is there a more straightforward builtin that I am missing?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;seq(1,length(names(x)))[names(x) == &quot;bar&quot;]&#xA;[1] 2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1138" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-25T14:45:57.530" LastActivityDate="2011-04-25T14:45:57.530" Title="Finding the column index by its name in R" Tags="&lt;r&gt;" AnswerCount="2" CommentCount="3" FavoriteCount="1" />
  <row Id="9938" PostTypeId="2" ParentId="9937" CreationDate="2011-04-25T00:59:02.707" Score="10" ViewCount="" Body="&lt;p&gt;Hi, probably this is the simplest way:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;which(names(x)==&quot;bar&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2280" LastActivityDate="2011-04-25T00:59:02.707" />
  <row Id="9939" PostTypeId="2" ParentId="9930" CreationDate="2011-04-25T01:01:49.443" Score="6" ViewCount="" Body="&lt;p&gt;Looking at the wikipedia page we have the partial correlation between $X$ and $Y$ given $Z$ is given by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\rho_{XY|Z}=\frac{\rho_{XY}-\rho_{XZ}\rho_{YZ}}{\sqrt{1-\rho_{XZ}^{2}}\sqrt{1-\rho_{YZ}^{2}}}&amp;gt;\rho_{XY}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So we simply require&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\rho_{XY}&amp;gt;\frac{\rho_{XZ}\rho_{YZ}}{1-\sqrt{1-\rho_{XZ}^{2}}\sqrt{1-\rho_{YZ}^{2}}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The right hand side has a global minimum when $\rho_{XZ}=-\rho_{YZ}$.  This global minimum is $-1$.  I think this should explain what's going on.  If the correlation between $Z$ and $Y$ is the opposite sign to the correlation between $Z$ and $X$ (but same magnitude), then the partial correlation between $X$ and $Y$ given $Z$ will always be greater than or equal to the correlation between $X$ and $Y$.  In some sense the &quot;plus&quot; and &quot;minus&quot; conditional correlation tend to cancel out in the unconditional correlation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I did some mucking around with R, and here is some code to generate a few plots.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;partial.plot &amp;lt;- function(r){  &#xA;  r.xz&amp;lt;- as.vector(rep(-99:99/100,199))  &#xA;  r.yz&amp;lt;- sort(r.xz)  &#xA;  r.xy.z &amp;lt;- (r-r.xz*r.yz)/sqrt(1-r.xz^2)/sqrt(1-r.yz^2)  &#xA;  tmp2 &amp;lt;- ifelse(abs(r.xy.z)&amp;lt;1,ifelse(abs(r.xy.z)&amp;lt;abs(r),2,1),0)  &#xA;  r.all &amp;lt;-cbind(r.xz,r.yz,r.xy.z,tmp2)  &#xA;  mycol &amp;lt;- tmp2  &#xA;  mycol[mycol==0] &amp;lt;- &quot;red&quot;  &#xA;  mycol[mycol==1] &amp;lt;- &quot;blue&quot;  &#xA;  mycol[mycol==2] &amp;lt;- &quot;green&quot;  &#xA;  plot(r.xz,r.yz,type=&quot;n&quot;)  &#xA;  text(r.all[,1],r.all[,2],labels=r.all[,4],col=mycol)  &#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;so you submit partial.plot(0.5) to see when a marginal correlation of 0.5 corresponds to in partial correlation.  The plot is color coded so that red area represents the &quot;impossible&quot; partial correlation, blue area where $|\rho|&amp;lt;|\rho_{XY|Z}|&amp;lt;1$ and the green area where $1&amp;gt;|\rho|&amp;gt;|\rho_{XY|Z}|$  Below is an example for $\rho_{XY}=r=0.5$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/lTixS.jpg&quot; alt=&quot;Partial correlation when marginal correlation is 0.5&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="2392" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-29T01:13:22.797" LastActivityDate="2011-04-29T01:13:22.797" CommentCount="4" />
  <row Id="9940" PostTypeId="1" CreationDate="2011-04-25T05:40:10.820" Score="2" ViewCount="61" Body="&lt;p&gt;I have $n$ standard normal and independent random variables $X_i$ (In reality I have a large known number of them, but let's just say I have $n$). In my experiment I want to on average get exactly 3 random variables $X_i$ under a threshold $c$. To get that, I can compute $c$ having that property easily, because the average number of $X_i$ that are under a threshold $c$ is $n \Phi(c)$ where $\Phi$ is the cdf of the standard normal distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I choose $c = \Phi^{-1}(3/n)$ in this case. (Which is a negative number for large $n$.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But unfortunately I already know the value of two other standard normal random variables $Y$ and $Z$ which may depend on each other and on any number of the other $X_i$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my question is: if I know that $Y$ and $Z$ are under the threshold $c=\Phi^{-1}(3/n)$, is it then still true that on average at most a constant number of the other random variables $X_i$ are under the threshold $c$? So by knowing that $Y$ and $Z$ are under the threshold, they can't suddenly make many of the other random variables go under it too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am almost certain that they can't, but I don't know how to prove it. Any hints are welcome. Or books where you think this might be in.&lt;/p&gt;&#xA;" OwnerUserId="4312" LastEditorUserId="4312" LastEditorDisplayName="" LastEditDate="2011-04-25T18:23:15.007" LastActivityDate="2011-04-25T20:32:48.797" Title="Few random variables cannot influence $n$ independent others too much?" Tags="&lt;probability&gt;&lt;normal-distribution&gt;&lt;random-variable&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="9941" PostTypeId="2" ParentId="9918" CreationDate="2011-04-25T05:43:44.557" Score="2" ViewCount="" Body="&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;The standard tools, at least in psychology, in your situation would be exploratory and confirmatory factor analysis to assess the convergence of the inter-item correlation matrix with some proposed model of the relationship between factors and items. The way that you have phrased your question suggests that you might not be familiar with this literature.&#xA;For example, here are my notes on the &lt;a href=&quot;http://jeromyanglim.blogspot.com/2009/10/exploratory-factor-analysis-and-scale.html&quot; rel=&quot;nofollow&quot;&gt;scale construction and factor analysis&lt;/a&gt; and here is a &lt;a href=&quot;http://www.statmethods.net/advstats/factor.html&quot; rel=&quot;nofollow&quot;&gt;tutorial in R on factor analysis form Quick-R&lt;/a&gt;.&#xA;Thus, while it's worth answering your specific question, I think that your broader aims will be better served by examining factor analytic approaches to evaluating multi-item, multi-factor scales.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Another standard strategy would be to calculate total scores for each group of variables (what I would call a &quot;scale&quot;) and correlate the scales.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Many reliability analysis tools will report average inter-item correlation.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If you created the 50 by 50 matrix of correlations between items, you could write a function in R that averaged subsets based on combinations of groups of variables. You might not get what you want if you have a mixture of positive and negative items, as the negative correlations might cancel out the positive correlations.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastActivityDate="2011-04-25T05:43:44.557" />
  <row Id="9942" PostTypeId="1" AcceptedAnswerId="9943" CreationDate="2011-04-25T06:08:43.413" Score="1" ViewCount="118" Body="&lt;p&gt;In the one-dimensional case, if $X$ is $\mathcal{N}(\mu,\sigma^2)$, then $Y =\alpha X + \beta $ is $\mathcal{N}(\alpha \mu + \beta,\alpha^2\sigma^2)$ . We can prove this using the cumulative distribution function of of $Y$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$F_Y(a) = P\{Y \leq a\} = P\{\alpha X + \beta \leq a\} = P\{X \leq (a-\beta)/\alpha\}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Substituting $Y =\alpha X + \beta $ and change of variable gives us,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$F_Y(a) = \int_{-\infty}^{a} \frac{1}{\sqrt{2\pi}(\alpha\sigma)} \exp \{ \frac{-(v-(\alpha \mu + \beta))^2}{2(\alpha\sigma)^2}\} dv $  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hence&#xA;$f_Y(v) = \frac{1}{\sqrt{2\pi}(\alpha\sigma)} \exp \{ \frac{-(v-(\alpha \mu + \beta))^2}{2(\alpha\sigma)^2}\} $  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus $Y$ is $\mathcal{N}(\alpha \mu + \beta, \alpha^2 \sigma^2)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the multivariate case, if $X$ is $\mathcal{N}(\mu,\Sigma)$ and $Y=\alpha X + \beta$, is $Y \sim \mathcal{N}(\alpha \mu + \beta,\alpha^2\Sigma)$? If so, how do we prove it?&lt;/p&gt;&#xA;" OwnerUserId="4290" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-29T00:54:00.753" LastActivityDate="2011-04-29T00:54:00.753" Title="Does $Y=\alpha X + \beta$ hold for multivariate gaussian density?" Tags="&lt;multivariate-analysis&gt;" AnswerCount="1" />
  <row Id="9943" PostTypeId="2" ParentId="9942" CreationDate="2011-04-25T07:00:19.810" Score="5" ViewCount="" Body="&lt;p&gt;The method of characteristic functions (CF) will work here.  So we have the CF for $X$ as &#xA;$$\varphi_{X}(t)=\exp\left(it^{T}\mu_{X}-\frac{1}{2}t^{T}\Sigma_{X}t\right)$$&#xA;Now we make the substitution $Y=\alpha X + \beta$ in the CF and we get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\varphi_{Y}(t)=E\left[\exp(it^{T}Y)\right]=E\left[\exp(it^{T}\alpha X +it^{T}\beta)\right]=\exp(it^{T}\beta)\varphi_{X}(\alpha t)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then substitute in the CF expression for $X$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\varphi_{Y}(t)=\exp(it^{T}\beta)\exp\left(i(\alpha t)^{T}\mu_{X}-\frac{1}{2}(\alpha t)^{T}\Sigma_{X}(\alpha t)\right)$$&#xA;$$=\exp\left(it^{T}[\alpha\mu_{X}+\beta]-\frac{1}{2}t^{T}[\alpha^{2}\Sigma_{X}]t\right)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But this is the characteristic function of a new normal distribution with mean vector $\alpha\mu_{X}+\beta$ and covariance matrix $\alpha^{2}\Sigma_{X}$.  As characteristic functions are uniquely defined from a distribution function and vice versa, you have your proof.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To generalise to the case where $\alpha$ is an appropriately defined $c\times p$ matrix ($p$ is the dimension of $X$). we simply replace the covariance matrix $\alpha^{2}\Sigma_{X}$ with the $c\times c$ covariance matrix $\alpha\Sigma_{X}\alpha^{T}$.  Note that $\beta$ must be a $c\times 1$ vector for mean vector to make sense - but it is unchanged at $\alpha\mu_{X}+\beta$.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-04-26T11:59:13.590" LastActivityDate="2011-04-26T11:59:13.590" CommentCount="17" />
  <row Id="9944" PostTypeId="2" ParentId="9918" CreationDate="2011-04-25T08:25:35.240" Score="4" ViewCount="" Body="&lt;p&gt;What @rolando suggested looks like a good start, if not the whole response (IMO). Let me continue with the correlational approach, following the Classical Test Theory (CTT) framework. Here, as noted by @Jeromy, a summary measure for your group of characteristics might be considered as the totalled (or sum) score of all items (a characteristic, in your words) belonging to what I will now refer to as a scale. Under CTT, this allows us to formalize individual &quot;trait&quot; propensity or liability as one's location on a continuous scale reflecting an underlying construct (a latent trait), although here it is merely an ordinal scale (but this another debate in the psychometrics literature).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you described has to do with what is know as &lt;em&gt;convergent&lt;/em&gt; (to what extent items belonging to the same scale do correlate one with each other) and &lt;em&gt;discriminant&lt;/em&gt; (items belonging to different scales should not correlate to a great extent) validity in psychometrics. Classical techniques include multi-trait multi-method (MTMM) analysis (Campbell &amp;amp; Fiske, 1959). An illustration of how it works is shown below (three methods or instruments, three constructs or traits):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/xvSyM.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this MTMM matrix, the diagonal elements might be Cronbach's alpha or test-retest intraclass correlation; these are indicators of the &lt;em&gt;reliability&lt;/em&gt; of each measurement scale. The &lt;em&gt;validity&lt;/em&gt; of the hypothesized (shared) constructs is assessed by the correlation of scales scores when different instruments are used to assess the same trait; if these instrument were developed independently, high correlation ($&gt; 0.7$) would support the idea that the traits are defined in a consistent and objective manner. The remaining cells in this MTMM matrix summarize relations &lt;em&gt;between traits within method&lt;/em&gt;, and &lt;em&gt;between traits across methods&lt;/em&gt;, and are indicative of the way unique constructs are measured with different scales and what are the relations between each trait in a given scale. Assuming independent traits, we generally don't expect them to be high (a recommended threshold is $&amp;lt;.3$), but more formal test of hypothesis (on correlation point estimates) can be carried out. A subtlety is that we use so-called &quot;rest correlation&quot;, that is we compute correlation between an item (or trait) and its scale (or method) after removing the contribution of this item to the sum score of this scale (correction for overlap).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even if this method was initially developed to assess convergent and discriminant validity of a certain number of traits as studied by different measurement instruments, it can be applied for a single multi-scale instrument. The traits then becomes the items, and the methods are just the different scales. A generalization of this method to a single instrument is also known as &lt;em&gt;multitrait scaling&lt;/em&gt;. Items correlating as expected (i.e., with their own scale rather than a different scale) are counted as &lt;em&gt;scaling success&lt;/em&gt;. We generally assume, however, that the different scales are not correlated, that is they are targeting different hypothetical constructs. But averaging the within and between-scale correlations provide a quick way of summarizing the internal structure of your instrument. Another convenient way of doing so is to apply a cluster analysis on the matrix of pairwise correlations and see how your variables do hang together.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of note, in both cases, the usual caveats of working with correlation measures apply, that is you cannot account for measurement error, you need a large sample, instruments or tests are assumed to be &quot;parallel&quot; (tau-equivalence, uncorrelated errors, equal error variances).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second part addressed by @rolando is also interesting: If there's no theoretical or substantive indication that the already established grouping of items makes sense, then you'll have to find a way to highlight the structure of your data with e.g., exploratory factor analysis. But even if you trust those &quot;characteristics within a group&quot;, you can check that this is a valid assumption. Now, you might be using confirmatory factor analysis model to check that the pattern of items loadings (correlation of an item with its own scale) behaves as expected.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead of traditional factor analytic methods, you can also take a look at items clustering (Revelle, 1979) which relies on a Cronbach's alpha-based split-rule to group together items into homogeneous scales.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A final word: If you are using R, there are two very nice packages that will ease the aforementioned steps:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://cran.r-project.org/web/packages/psych/index.html&quot; rel=&quot;nofollow&quot;&gt;psych&lt;/a&gt;, provides you with everything you need for getting started with psychometrics methods, including factor analysis (&lt;code&gt;fa&lt;/code&gt;, &lt;code&gt;fa.parallel&lt;/code&gt;, &lt;code&gt;principal&lt;/code&gt;), items clustering (&lt;code&gt;ICLUST&lt;/code&gt; and related methods), Cronbach's alpha (&lt;code&gt;alpha&lt;/code&gt;); there's a nice overview available on William Revelle's website, especially &lt;a href=&quot;http://personality-project.org/r/book/&quot; rel=&quot;nofollow&quot;&gt;An introduction to psychometric theory with applications in R&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://cran.r-project.org/web/packages/psy/index.html&quot; rel=&quot;nofollow&quot;&gt;psy&lt;/a&gt;, also includes scree plot (via PCA + simulated datasets) visualization (&lt;code&gt;scree.plot&lt;/code&gt;) and MTMM (&lt;code&gt;mtmm&lt;/code&gt;).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;References&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Campbell, D.T. and Fiske, D.W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. &lt;em&gt;Psychological Bulletin&lt;/em&gt;, 56: 81–105.&lt;/li&gt;&#xA;&lt;li&gt;Hays, R.D. and Fayers, P. (2005). Evaluating multi-item scales. In &lt;em&gt;Assessing quality of life in clinical trials&lt;/em&gt;, (Fayers, P. and Hays, R., Eds.), pp. 41-53. Oxford.&lt;/li&gt;&#xA;&lt;li&gt;Revelle, W. (1979). Hierarchical Cluster Analysis and the Internal Structure of Tests. &lt;em&gt;Multivariate Behavioral Research&lt;/em&gt;, 14: 57-74.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-04-25T08:25:35.240" CommentCount="1" />
  <row Id="9945" PostTypeId="2" ParentId="7538" CreationDate="2011-04-25T08:47:06.100" Score="1" ViewCount="" Body="&lt;p&gt;Assuming your data are stored in the file &lt;code&gt;1.dat&lt;/code&gt;, stacked barcharts might be generated as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set style data histograms&#xA;set style histogram rowstacked&#xA;set boxwidth 1 relative&#xA;set style fill solid 1.0 border -1&#xA;set yrange [0:1.5]&#xA;set datafile separator &quot; &quot;&#xA;plot '1.dat' using 2 t &quot;Var 1&quot;, '' using 3:xticlabels(1) t &quot;Var 2&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As you can see, barcharts are no different from histograms (at least, from within Gnuplot). More information can be found on gnuplot &lt;a href=&quot;http://gnuplot.sourceforge.net/demo/histograms.html&quot; rel=&quot;nofollow&quot;&gt;demo page&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/mlWkX.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-25T10:45:42.037" LastActivityDate="2011-04-25T10:45:42.037" />
  <row Id="9946" PostTypeId="2" ParentId="9911" CreationDate="2011-04-25T08:54:17.140" Score="1" ViewCount="" Body="&lt;p&gt;This smells like archetypal analysis -- extracting some underlying prototypical objects. However, the vanilla AA will give you linear combination as PCA; thus I would suggest making something similar by first making some k-means-like clustering of the events and then selecting those which are closest to the centroids. &lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2011-04-25T08:54:17.140" />
  <row Id="9947" PostTypeId="1" CreationDate="2011-04-25T09:04:15.967" Score="6" ViewCount="74" Body="&lt;p&gt;For a Dataset $D$, we have gold standard centroids say $c_1, c_2, \cdots, c_n$. Now if we run k-means algorithm on $D$ with input $n$, we get k-means centroid $k_1, k_2, \cdots, k_n$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I just wanted to know, is there any algorithm/heuristic to match the centroids between $k_i$ and $c_j$ where $i, j= 1, \cdots, n$ (One to one mapping between $k$'s and $c$'s)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried to calculate the pairwise distance between $k_p$ and $c_j,\; j= 1, \cdots, n$, and match $k_p$ to $c_r$ where the distance between them is minimum. But in this case  $k_p$ and $k_q$ are assigned to $c_r$, which we dont need. &lt;/p&gt;&#xA;" OwnerUserId="4290" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-25T09:22:45.317" LastActivityDate="2011-04-26T04:29:33.870" Title="Centroid matching problem" Tags="&lt;clustering&gt;&lt;algorithms&gt;" AnswerCount="3" CommentCount="2" FavoriteCount="2" />
  <row Id="9948" PostTypeId="1" CreationDate="2011-04-25T09:12:07.153" Score="4" ViewCount="74" Body="&lt;p&gt;I need to generate cross variograms of images using moving windows. For that I use the following equation: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;\gamma_{jk}(h)=\frac{1}{2n(h)}\sum_{i=1}^{n(h)}\Big\{\big[dn_j(x_i)-dn_j(x_i+h)\big]\cdot\big[dn_k(x_i)-dn_k(x_i+h)\big]\Big\}&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first part stands for one band(j) and next part of band k. To illustrate with sample matrices, &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;j =  1     2     3     4&#xA;     5     6     7     8&#xA;     9    10    11    12&#xA;    13    14    15    16&#xA;&#xA;k = 17    18    19    20&#xA;    21    22    23    24&#xA;    25    26    27    28&#xA;    29    30    31    32&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In actual case I am using 7 X 7 windows for large satellite images.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also had to generate variograms from images for this work. For generation of variograms I had to consider only one band of data. For that case I used &lt;code&gt;nlfilter&lt;/code&gt; for moving window and created a function to select and calculate values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But for cross variograms, I am not able to decide upon what function to use. For this case the calculations will go like this: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;(1 - 2)(17 - 18) + (2 - 3)(18 - 19)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;and so on.  &lt;/p&gt;&#xA;" OwnerUserId="4313" LastEditorUserId="4313" LastEditorDisplayName="" LastEditDate="2011-04-25T17:13:52.510" LastActivityDate="2011-04-25T18:03:19.457" Title="Cross variogram with a moving window" Tags="&lt;variance&gt;&lt;matlab&gt;&lt;image-processing&gt;" AnswerCount="1" CommentCount="4" FavoriteCount="2" />
  <row Id="9949" PostTypeId="2" ParentId="9937" CreationDate="2011-04-25T10:02:14.867" Score="5" ViewCount="" Body="&lt;p&gt;just to add another possibility:&lt;br&gt;&#xA;You can usually use &lt;code&gt;grep&lt;/code&gt; and it's decedents (i.e., grepl,  to do these kind of jobs in a more sophisiticated way using regular expressions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On your example your could get the column index with:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;grep(&quot;^bar$&quot;, colnames(x))&lt;/code&gt; or &lt;code&gt;grep(&quot;^bar$&quot;, names(x))&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;code&gt;^&lt;/code&gt; and &lt;code&gt;$&lt;/code&gt; are meta characters for the beginning and end of a string, respectively.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Check ?grep and especially ?regex for more infos (i.e., you can grab only partial names/matches, or the return value is the string itself or a logical vector,...)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For me, &lt;code&gt;grep&lt;/code&gt; is more R-ish.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Strongly related is the recent package by Hadley Wickhem: &lt;a href=&quot;http://cran.r-project.org/web/packages/stringr/index.html&quot; rel=&quot;nofollow&quot;&gt;stringr&lt;/a&gt;, A package for &quot;modern, consistent string processing&quot; including grep like functions. He recently published &lt;a href=&quot;http://journal.r-project.org/archive/2010-2/RJournal_2010-2_Wickham.pdf&quot; rel=&quot;nofollow&quot;&gt;a paper on it in the R Journal.&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://stackoverflow.com/questions/4427234/get-column-index-from-label-in-a-data-frame/4427459#4427459&quot;&gt;See also my answer on stackoverflow on an identical issue.&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="442" LastActivityDate="2011-04-25T10:02:14.867" />
  <row Id="9950" PostTypeId="2" ParentId="9931" CreationDate="2011-04-25T10:49:36.277" Score="1" ViewCount="" Body="&lt;p&gt;If you form the Transfer Function Model y(t)=W(B)*X(t)+[THETA(B)/PHI(B)]*a(t) the operator [THETA(B)/PHI(B)] is the &quot;smoothing component&quot;. For examnple if PHI(B)=1.0 and THETA(B)=1-.5B this would imply a set of weights of .5,.25,.125,... . in this way you could provide the answer to optimizing the &quot;weighted moving linear regression&quot; rather than assuming it's form.&lt;/p&gt;&#xA;" OwnerUserId="3382" LastActivityDate="2011-04-25T10:49:36.277" />
  <row Id="9951" PostTypeId="1" CreationDate="2011-04-25T12:18:37.257" Score="1" ViewCount="81" Body="&lt;p&gt;I was wrestling with stationarity in my head for a while... Is this how you think about it? Any comments or further thoughts will be appreciated.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Stationary process is the one which&#xA;  generates time-series values such that&#xA;  distribution mean and variance is kept&#xA;  constant. Strictly speaking, this is&#xA;  known as weak form of stationarity or&#xA;  covariance/mean stationarity.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Weak form of stationarity is when the&#xA;  time-series has constant mean and&#xA;  variance throughout the time.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Let's put it simple, practitioners say&#xA;  that the stationary time-series is the&#xA;  one with no trend - fluctuates around&#xA;  the constant mean and has constant&#xA;  variance.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Covariance between different lags is&#xA;  constant, it doesn't depend on&#xA;  absolute location in time-series. For&#xA;  example, the covariance between t and&#xA;  t-1 (first order lag) should always be&#xA;  the same (for the period from&#xA;  1960-1970 same as for the period from&#xA;  1965-1975 or any other period).&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;In non-stationary processes there is&#xA;  no long-run mean to which the series&#xA;  reverts; so we say that non-stationary&#xA;  time series do not mean revert. In&#xA;  that case, the variance depends on&#xA;  absolute position in time-series and&#xA;  variance goes to infinity as time goes&#xA;  on. Technically speaking,&#xA;  auto-correlations to not decay with&#xA;  time, but in small samples they do&#xA;  disappear - although slowly.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;In stationary processes, shocks are&#xA;  temporary and dissipate (lose energy)&#xA;  over time. After a while, they do not&#xA;  contribute to the new time-series&#xA;  values. For example, something which&#xA;  happened log time ago (long enough)&#xA;  such as World War II, had an impact,&#xA;  but, it the time-series today is the&#xA;  same as if World War II never&#xA;  happened, we would say that shock lost&#xA;  its energy or dissipated. Stationarity&#xA;  is especially important as many&#xA;  classical econometric theories are&#xA;  derived under the assumptions of&#xA;  stationarity.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;A strong form of stationarity is when&#xA;  the distribution of a time-series is&#xA;  exactly the same trough time. In other&#xA;  words, the distribution of original&#xA;  time-series is exactly same as lagged&#xA;  time-series (by any number of lags) or&#xA;  even sub-segments of the time-series.&#xA;  For example, strong form also suggests&#xA;  that the distribution should be the&#xA;  same even for a sub-segments&#xA;  1950-1960, 1960-1970 or even&#xA;  overlapping periods such as 1950-1960&#xA;  and 1950-1980. This form of&#xA;  stationarity is called strong because&#xA;  it doesn't assume any distribution. It&#xA;  only says the probability distribution&#xA;  should be the same. In the case of&#xA;  weak stationarity, we defined&#xA;  distribution by its mean and variance.&#xA;  We could do this simplification&#xA;  because implicitly we assumed normal&#xA;  distribution, and normal distribution&#xA;  is fully defined by its mean and&#xA;  variance or standard deviation. This&#xA;  is nothing but saying that probability&#xA;  measure of the sequence (within&#xA;  time-series) is the same as that for&#xA;  lagged/shifted sequence of values&#xA;  within same time-series.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="333" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-25T14:41:32.600" LastActivityDate="2011-04-25T14:41:32.600" Title="Intuitive explanation of stationarity" Tags="&lt;time-series&gt;&lt;stationarity&gt;" CommentCount="2" FavoriteCount="1" />
  <row Id="9952" PostTypeId="2" ParentId="9947" CreationDate="2011-04-25T12:43:32.667" Score="0" ViewCount="" Body="&lt;p&gt;Sounds like you might want to consider using/writing an energy function.  More here: &lt;a href=&quot;http://en.wikipedia.org/wiki/Optimization_%28mathematics%29#Multi-objective_optimization&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Optimization_%28mathematics%29#Multi-objective_optimization&lt;/a&gt;  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suppose if your number of k centroids is &quot;small&quot; you can run a distance function for all c k  pairings and select the set which minimizes total distance as the 'best' solution.&lt;br&gt;&#xA;Hope that helps -&lt;br&gt;&#xA;Perry&lt;/p&gt;&#xA;" OwnerUserId="4316" LastActivityDate="2011-04-25T12:43:32.667" CommentCount="1" />
  <row Id="9953" PostTypeId="2" ParentId="9852" CreationDate="2011-04-25T12:57:28.647" Score="3" ViewCount="" Body="&lt;p&gt;As whuber stated this actually is a case of nested models, and hence one can apply a &lt;a href=&quot;http://en.wikipedia.org/wiki/Likelihood-ratio_test&quot; rel=&quot;nofollow&quot;&gt;likelihood-ratio test&lt;/a&gt;. Because it is still not exactly clear what models you are specifying I will just rewrite them in this example;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So model 1 can be:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Y = a_1 + B_{11}(X) + B_{12}(W) + B_{13}(Z) + e_1$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And model 2 can be (I ignore the division by 2, but this action has no consequence for your question):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Y = a_2 + B_{21}(X) + B_{22}(W+Z) + e_2$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which can be rewritten as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Y = a_2 + B_{21}(X) + B_{22}(W) + B_{22}(Z)+ e_2$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And hence model 2 is a specific case of model 1 in which $B_{12}$ and $B_{13}$ are equal. One can use the likelihood-ratio test between these two models to assign a p-value to the fit of model 1 compared to model 2. There are good reasons in practice to do this, especially if the correlation between W and Z are quite large (&lt;a href=&quot;http://stats.stackexchange.com/q/1149/1036&quot;&gt;multicollinearity&lt;/a&gt;). As I stated previously, whether you divide by two does not matter for testing the fit of the models, although if it is easier to interpret $\frac{W+Z}{2}$ then $W+Z$ by all means use the average of the two variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Model fit statistics (such as Mallow's CP already mentioned by bill_080, and other examples are &lt;a href=&quot;http://en.wikipedia.org/wiki/Akaike_information_criterion&quot; rel=&quot;nofollow&quot;&gt;AIC&lt;/a&gt; and &lt;a href=&quot;http://en.wikipedia.org/wiki/Bayesian_information_criterion&quot; rel=&quot;nofollow&quot;&gt;BIC&lt;/a&gt;), are frequently used to assess non-nested models. Those statistics do not follow known distributions (like the log-likelihood does, &lt;a href=&quot;http://en.wikipedia.org/wiki/Chi-square_distribution&quot; rel=&quot;nofollow&quot;&gt;Chi-square&lt;/a&gt;) and hence the differences in those statistics between models can not be given a p-value. &lt;/p&gt;&#xA;" OwnerUserId="1036" LastActivityDate="2011-04-25T12:57:28.647" />
  <row Id="9954" PostTypeId="2" ParentId="9948" CreationDate="2011-04-25T14:50:39.467" Score="5" ViewCount="" Body="&lt;p&gt;I prefer a slight change of notation due to the many $n$'s appearing in the original. Let $\alpha$ and $\beta$ designate the images.  Let $i$ and $j$ each designate &lt;em&gt;pairs&lt;/em&gt; of indexes into the image rows and columns.  (Indexing goes from $1$ to $m$ for rows and $1$ to $n$ for columns.)  Let $h$ designate a &lt;em&gt;relative&lt;/em&gt; index pair (so that its two entries are integers, either of which can be negative), also known as an &lt;em&gt;offset&lt;/em&gt;.  Then, by definition, the value of the experimental cross-variogram of these images at an offset $h$ is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\gamma_{\alpha,\beta}(h)=\frac{1}{2n(h)}\sum_{i}\left(\alpha[i+h]-\alpha[i]\right)\left(\beta[i+h] - \beta[i]\right).$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The sum ranges over all indexes $i$ for which both $i$ and $i+h$ are valid indexes into both images; $n(h)$ is the number of such indexes (easily computed in the same way by taking a similar sum of $1$'s).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By expanding the summand algebraically the calculation is reduced to the problem of obtaining&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\sum_{i}\alpha[i+h]\beta[i]$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;for various $h$, both positive and negative, ranging from $(1-m,1-n)$ through $(m-1,n-1)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let us say that the &lt;em&gt;reversal&lt;/em&gt; of an image negates the indexes; that is, the value of the reversal of $\alpha$ at the pixel $h$ is the value of $\alpha$ at $(m+1,n+1)-h$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Such a sum can be seen as the reversal of the convolution of the reversal of $\alpha$ with $\beta$.  It is best computed using &lt;a href=&quot;http://en.wikipedia.org/wiki/Discrete_Fourier_transform#Circular_convolution_theorem_and_cross-correlation_theorem&quot; rel=&quot;nofollow&quot;&gt;discrete Fourier transforms&lt;/a&gt; after first padding each image to the right and down with zeros.  The padding must extend to the range of the largest $h$ for which $\gamma$ needs to be computed.  Convolutions with Fourier transforms are obtained by taking the inverse Fourier transform (itself a scalar multiple of the FT) of the product of the FTs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Direct computation of the variogram via its definition for a pair of $m$ by $n$ images requires up to $m n$ products and sums for each value of $h$.  Typically $O(m n)$ values of $h$ are needed.  The direct algorithm therefore has $O(m^2 n^2)$ computational cost, which is ridiculously large for moderate (megapixel) images.  The discrete Fourier transform costs at most $O(2m 2n \log(2m 2n))$ (assuming the maximum range of offsets $h$) and has to be applied only a constant number of times (3).  The reversals and paddings cost $O(2m 2n)$.  Thus the total cost is still only $O(12 m n \log(4 m n))$, a huge improvement.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;As a simple example, take $\alpha$ and $\beta$ to be the matrices&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1 2&#xA;3 4&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;5 6&#xA;7 8&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;After padding with zeros to the right and down (by two columns and two rows) &lt;em&gt;and reversing $\alpha$&lt;/em&gt;, multiplying these two 4 by 4 matrices componentwise, and taking the inverse Fourier transform, we get &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; 8 0  14 23&#xA; 0 0   0  0&#xA;18 0  20 39&#xA;30 0  38 70&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Rotating this right by 2 columns and 2 rows and reversing gives&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;0  0  0  0&#xA;0  8 23 14&#xA;0 30 70 38&#xA;0 18 39 20&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you think of the new row and column indexes ranging $-2, -1, 0, 1$, this new matrix is exactly $\sum_{i}\alpha[i+h]\beta[i]$ (indexed by $h$).  For example, the $h = (0,1)$ entry is 38 and indeed &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\alpha[1,2]\beta[1,1] + \alpha[2,2]\beta[2,1] = 2 \times 5 + 4 \times 7 = 38.$$&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-04-25T18:03:19.457" LastActivityDate="2011-04-25T18:03:19.457" CommentCount="2" />
  <row Id="9955" PostTypeId="2" ParentId="9931" CreationDate="2011-04-25T15:18:03.097" Score="3" ViewCount="" Body="&lt;p&gt;Sounds like what you want to do is a two-stage model.  First transform your data into exponentially smoothed form using a specified smoothing factor, and then input the transformed data into your linear regression formula.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.jstor.org/pss/2627674&quot; rel=&quot;nofollow&quot;&gt;http://www.jstor.org/pss/2627674&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Exponential_smoothing&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Exponential_smoothing&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="3489" LastActivityDate="2011-04-25T15:18:03.097" CommentCount="4" />
  <row Id="9956" PostTypeId="2" ParentId="8696" CreationDate="2011-04-25T15:34:00.080" Score="2" ViewCount="" Body="&lt;p&gt;Try&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;computeFunction=function(onWhat,what,...){foreach(i=onWhat) %do% what(i,...)},&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="88" LastActivityDate="2011-04-25T15:34:00.080" />
  <row Id="9957" PostTypeId="2" ParentId="9931" CreationDate="2011-04-25T16:20:15.547" Score="0" ViewCount="" Body="&lt;p&gt;If you are looking for an equation of the form &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$y=\alpha_n + \beta_n x$$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;after $n$ pieces of data have come in, and you are using an exponential factor $k \ge 1$ then you could use &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\beta_n = \frac{\left(\sum_{i=1}^n k^i\right) \left(\sum_{i=1}^n k^i X_i Y_i\right) - \left(\sum_{i=1}^n k^i X_i\right)   \left(\sum_{i=1}^n k^i  Y_i\right) }{ \left(\sum_{i=1}^n k^i\right) \left(\sum_{i=1}^n k^i X_i^2\right) - \left(\sum_{i=1}^n k^i X_i \right)^2}$$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;and    &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\alpha_n = \frac{\left(\sum_{i=1}^n k^i Y_i\right)  - \beta_n \left(\sum_{i=1}^n k^i X_i\right)}{\sum_{i=1}^n k^i} .$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If rounding or speed become issues, this can be recast in other forms.  It may also be worth knowing that for $k&amp;gt;1$ you have $\sum_{i=1}^n k^i = \frac{k(k^n - 1)}{k-1}$.&lt;/p&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-04-25T16:20:15.547" />
  <row Id="9958" PostTypeId="2" ParentId="9880" CreationDate="2011-04-25T16:46:01.893" Score="2" ViewCount="" Body="&lt;p&gt;For models of speeded decision tasks, check out the diffusion model and the linear ballistic accumulator; Donkin et al (2011, &lt;a href=&quot;http://mypage.iu.edu/~cdonkin/pubs/pbr11.pdf&quot; rel=&quot;nofollow&quot;&gt;pdf&lt;/a&gt;) provide a good overview of these models and their different behaviours. There is R code out there for both these models. You might also do a literature search using the keyword &quot;&lt;a href=&quot;http://en.wikipedia.org/wiki/Decision_field_theory&quot; rel=&quot;nofollow&quot;&gt;Decision Field Theory&lt;/a&gt;&quot;, which seems to be a specific instantiation of the principles of diffusion models for high-level decisions like consumer choices, etc (in contrast, the diffusion model proper and linear ballistic accumulator are more typically used for simpler perceptual discrimination tasks). Finally, possibly related are &lt;a href=&quot;http://www.scholarpedia.org/article/Neural_fields&quot; rel=&quot;nofollow&quot;&gt;Neural Field models&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For models of semantics, check out &lt;a href=&quot;http://www.indiana.edu/~clcl/BEAGLE/&quot; rel=&quot;nofollow&quot;&gt;BEAGLE&lt;/a&gt;. For a related model of memory encoding/retrieval, check out Mehort &amp;amp; Johns (2005, &lt;a href=&quot;http://www.queensu.ca/psychology/hiplab/Publications/Recentpublications/Mewhort_RP_14.pdf&quot; rel=&quot;nofollow&quot;&gt;pdf&lt;/a&gt;)'s Iterative Resonance Model.&lt;/p&gt;&#xA;" OwnerUserId="364" LastEditorUserId="364" LastEditorDisplayName="" LastEditDate="2011-04-25T16:55:53.633" LastActivityDate="2011-04-25T16:55:53.633" CommentCount="5" />
  <row Id="9959" PostTypeId="1" CreationDate="2011-04-25T16:46:53.890" Score="5" ViewCount="110" Body="&lt;p&gt;In &lt;a href=&quot;http://www.cs.cornell.edu/~caruana/ctp/ct.papers/caruana.icml06.pdf&quot; rel=&quot;nofollow&quot;&gt;An Empirical Comparison of Supervised Learning Algorithms&lt;/a&gt; (ICML 2006) the authors (Rich Caruana and Alexandru Niculescu-Mizil) evaluated several classification algorithms (SVMs, ANN, KNN, Random Forests, Decision Trees, etc.), and reported that calibrated boosted trees ranked as the best learning algorithm overall across eight different metrics (F-score, ROC Area, average precision, cross-entropy, etc.).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to test &lt;strong&gt;calibrated boosted decision trees&lt;/strong&gt; in one of my projects, and was wondering if anybody could suggest a good R package or MATLAB library for this. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am relatively new to R, although I have large experience with MATLAB and Python. I have read about R's &lt;strong&gt;gbm&lt;/strong&gt;, &lt;strong&gt;tree&lt;/strong&gt;, and &lt;strong&gt;rpart&lt;/strong&gt; but I am not sure if these packages  implement calibrated boosted decision trees or if there are others that implement them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks &lt;/p&gt;&#xA;" OwnerUserId="2798" LastEditorUserId="2798" LastEditorDisplayName="" LastEditDate="2011-04-26T13:01:49.497" LastActivityDate="2011-04-26T13:01:49.497" Title="Calibrated boosted decision trees in R or MATLAB" Tags="&lt;r&gt;&lt;classification&gt;&lt;matlab&gt;" AnswerCount="1" FavoriteCount="2" />
  <row Id="9960" PostTypeId="2" ParentId="9959" CreationDate="2011-04-25T17:06:02.400" Score="3" ViewCount="" Body="&lt;p&gt;About R, I would vote for the &lt;a href=&quot;http://cran.r-project.org/web/packages/gbm/index.html&quot; rel=&quot;nofollow&quot;&gt;gbm&lt;/a&gt; package; there's a vignette that provides a good overview: &lt;a href=&quot;http://cran.r-project.org/web/packages/gbm/vignettes/gbm.pdf&quot; rel=&quot;nofollow&quot;&gt;Generalized Boosted Models: A guide to the gbm package&lt;/a&gt;. If you are looking for an unified interface to ML algorithms, I recommend the &lt;a href=&quot;http://caret.r-forge.r-project.org/Classification_and_Regression_Training.html&quot; rel=&quot;nofollow&quot;&gt;caret&lt;/a&gt; package which has built-in facilities for data preprocessing, resampling, and comparative assessment of model performance. Other packages for boosted trees are reported under Table 1 of one of its accompanying vignettes, &lt;a href=&quot;http://cran.r-project.org/web/packages/caret/vignettes/caretTrain.pdf&quot; rel=&quot;nofollow&quot;&gt;Model tuning, prediction and performance functions&lt;/a&gt;. There is also an example of parameters tuning for boosted trees in the &lt;a href=&quot;http://www.jstatsoft.org/v28/i05/paper&quot; rel=&quot;nofollow&quot;&gt;JSS paper&lt;/a&gt;, pp. 10-11.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note:&lt;/strong&gt; I didn't check, but you can also look into &lt;a href=&quot;http://www.cs.waikato.ac.nz/ml/weka/&quot; rel=&quot;nofollow&quot;&gt;Weka&lt;/a&gt; (there's an R interface, &lt;a href=&quot;http://cran.r-project.org/web/packages/RWeka/index.html&quot; rel=&quot;nofollow&quot;&gt;RWeka&lt;/a&gt;).&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-25T19:01:26.750" LastActivityDate="2011-04-25T19:01:26.750" />
  <row Id="9961" PostTypeId="1" AcceptedAnswerId="9973" CreationDate="2011-04-25T17:17:12.213" Score="2" ViewCount="46" Body="&lt;p&gt;I am new to evolutionary algorithm. I have studied Covariance Matrix Adaptation Evolution Strategy. I am not good at statistics. So could you please explain me in simple language (I mean not too many equations)&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;What is CMA-ES?&lt;/li&gt;&#xA;&lt;li&gt;How does it work?&lt;/li&gt;&#xA;&lt;li&gt;Why is it superior to other strategies?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="4319" LastActivityDate="2011-04-25T23:22:45.657" Title="What is covariance matrix adaptation evolution strategy?" Tags="&lt;variance-covariance&gt;" AnswerCount="2" />
  <row Id="9962" PostTypeId="1" AcceptedAnswerId="9974" CreationDate="2011-04-25T17:52:52.057" Score="2" ViewCount="91" Body="&lt;p&gt;I have a multi-class dataset like the following (a,b,c,d are features and e is the class (it can be 0,1 and 2)).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    a b c d e&#xA;1   1 1 2 2 1&#xA;2   1 2 4 2 0&#xA;3   1 2 4 2 0&#xA;4   2 2 2 2 0&#xA;5   2 1 2 2 2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am trying to use mlogit package in order to see which column is more important but I am having a difficulty to understand how to use it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I am doing is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dataset$e&amp;lt;-as.factor(dataset$e)&#xA;mldata&amp;lt;-mlogit.data(dataset, choice=&quot;e&quot;)&#xA;&#xA;&amp;gt; mldata[1:5,]&#xA;    a b c d     e chid alt&#xA;1.0 1 1 2 2 FALSE    1   0&#xA;1.1 1 1 2 2  TRUE    1   1&#xA;1.2 1 1 2 2 FALSE    1   2&#xA;2.0 1 2 4 2  TRUE    2   0&#xA;2.1 1 2 4 2 FALSE    2   1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, in order to see the coefficients, I am constructing the model like that: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;mlogit.model&amp;lt;- mlogit(e~1|a+b+c+d, data = mldata, reflevel=&quot;1&quot;)&#xA;  mlogit.model&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Call:&#xA;mlogit(formula = e ~ 1 | a + b + c + d, data = mldata, reflevel = &quot;1&quot;,     method = &quot;nr&quot;, print.level = 0)&#xA;&#xA;Coefficients:&#xA;     alt0       alt2     alt0:a     alt2:a     alt0:b     alt2:b     alt0:c  &#xA;-211.0953   -89.7558    27.9911    33.1440    37.7503     7.3585     8.5072  &#xA;   alt2:c     alt0:d     alt2:d  &#xA;   3.0950    37.1340     5.5584  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But now I don't understand what are alt0, alt2? What are the real coefficients of a,b,c and d?&lt;/p&gt;&#xA;" OwnerUserId="4320" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-25T19:21:31.530" LastActivityDate="2011-04-26T00:15:37.927" Title="Multiclass logistic regression with mlogit in R" Tags="&lt;r&gt;&lt;logistic-regression&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="9963" PostTypeId="2" ParentId="9947" CreationDate="2011-04-25T17:53:54.317" Score="4" ViewCount="" Body="&lt;p&gt;Because K-means minimizes variances, a good criterion is to &lt;strong&gt;minimize the sum of squared distances between the pairs of points&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is an &lt;a href=&quot;http://en.wikipedia.org/wiki/Linear_programming#Integer_unknowns&quot; rel=&quot;nofollow&quot;&gt;integral (0/1) linear program&lt;/a&gt;.  Specifically, the pairing can be specified by a matrix $\Lambda = (\lambda_{ij})$ where $\lambda_{ij} = 1$ if $c_i$ is paired with $k_j$ and $\lambda_{ij}=0$ otherwise.  We seek to minimize&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\sum_{i,j}\lambda_{ij}|c_i - k_j|^2$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;subject to the constraints (which enforce the one-to-one pairing)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\sum_{j}\lambda_{ij}=1$$&#xA;$$\sum_{i}\lambda_{ij}=1$$&#xA;$$\lambda_{ij} \in\{0,1\}.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Provided the centroids do not number more than a few hundred, this is quickly solved.  (The matrices involved in setting up the problem will quickly exhaust RAM with more than a few hundred centroids, because they scale as $O(n^3)$, and then you might have to be a little fussier with the programming.)  For instance, &lt;em&gt;Mathematica&lt;/em&gt; 8's `LinearProgramming' function takes no measurable time with fewer than $n=20$ centroids, escalating to about 5 seconds with 400 centroids.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/VLRMa.png&quot; alt=&quot;Solution with 20 points&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By means of line segments to show the pairings, this figure depicts an optimal solution with $n=20$ bivariate normal centroids $c_i$ and independent bivariate normal K-means solutions $k_i$.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-04-25T17:59:55.590" LastActivityDate="2011-04-25T17:59:55.590" CommentCount="1" />
  <row Id="9964" PostTypeId="2" ParentId="9911" CreationDate="2011-04-25T18:15:06.373" Score="0" ViewCount="" Body="&lt;p&gt;I think you may want to &lt;em&gt;reshape&lt;/em&gt; your data, not reduce it. This will let you change the structure of your data set so that you can use all of your observations.  You don't mention which statistical package you're using, but R, stata, and MATLAB all have a nice out-of-the-box reshape command you can use.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Side thought: you may need to adjust for clustered errors in the reshaped data, since it doesn't sound like your observations are completely independent.&lt;/p&gt;&#xA;" OwnerUserId="4110" LastActivityDate="2011-04-25T18:15:06.373" />
  <row Id="9965" PostTypeId="1" CreationDate="2011-04-25T18:52:46.150" Score="0" ViewCount="31" Body="&lt;p&gt;This is probably a simple question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm studying events which have N outcomes, of which exactly one is correct.  N is very large, more than a billion (and is known).  There are many possible events, some of which are tested multiple times.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to test the following model: a given event is tested correctly with probability 1 - p.  Otherwise, the test result is chosen uniformly at random from the N outcomes (hence, almost certainly incorrect).  I want to (1) see if this model is plausible and (2) if so, find p.  p should be small, just a few percent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data I have omits events where two or more tests have the same value.  (Because N is large, if two tests have the same value that is likely to be the correct result.)  So what I have is, for each k, the number of different events which have had k different tests, no two of which match.  Most of the events have had k = 1 tests but some have 2, 3, or even more tests.  In most cases one result is correct and the rest are incorrect; in some cases all are incorrect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I test my hypothesis?&lt;/p&gt;&#xA;" OwnerUserId="1378" LastActivityDate="2011-04-25T18:52:46.150" Title="Testing a model with truncated data" Tags="&lt;hypothesis-testing&gt;&lt;binomial&gt;&lt;censoring&gt;" CommentCount="3" />
  <row Id="9966" PostTypeId="1" CreationDate="2011-04-25T20:01:37.703" Score="0" ViewCount="63" Body="&lt;p&gt;Hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a dataset with between 10,000 and 100,000 feature values. The number of datapoints is between 1,000 and 10,000. I want to perform a LASSO on this dataset but can't really find any good software to do so. Does anyone have any suggestions?&lt;/p&gt;&#xA;" OwnerUserId="4322" LastActivityDate="2011-04-26T06:40:19.617" Title="Software for LASSO for high dimensional dataset" Tags="&lt;software&gt;&lt;lasso&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="9967" PostTypeId="2" ParentId="9961" CreationDate="2011-04-25T20:08:08.723" Score="3" ViewCount="" Body="&lt;p&gt;It's an optimization algorithm: it tries to find the minimum of a function. It is said to be amongst the best optimization algorithms for non-convex problems in high dimensions (above 5 or 10 parameters to optimize).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The term &lt;em&gt;Covariance&lt;/em&gt; in the name is a bit misleading to the statistics community: there is no statistics involved in the algorithm. As a result, this site might not be the best place to ask for help on it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can't give much more information on it, but the &lt;a href=&quot;http://en.wikipedia.org/wiki/CMA-ES&quot; rel=&quot;nofollow&quot;&gt;wikipedia&lt;/a&gt; page is quite clear.&lt;/p&gt;&#xA;" OwnerUserId="1265" LastActivityDate="2011-04-25T20:08:08.723" CommentCount="1" />
  <row Id="9968" PostTypeId="1" CreationDate="2011-04-25T20:12:32.520" Score="1" ViewCount="36" Body="&lt;p&gt;Hi!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am a new user of the Encog Framework and using Neural Network in general and I've a number of questions:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd like to build two standard feed-forward neural networks (first network and filter) with back-propagation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have my data organized in a csv file containing 211 columns where the first column contains a classification ('1', '2', '3' or '4'); the second to the 16th column contains flag values and from column 17th to 211th contains pattern values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyway, the first network consists of one input layer (210 input units - columns 2 to 211 in excel file), one hidden layer (66 hidden units) and one output layer (4 output units - 60 output representing the 15 (2nd to 16th column by 4 matrix). The second network (filter network) consists of the following data:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Filter network: 75 inputs, 75 hidden units, 4 output units Filter input network: columns 2 to 16 in excel file (15 data inputs) + 60 outputs from first network Filter network output: '1', '2','3' or '4' (final four state prediction)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is how can I implement the connection of those both networks?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Below is the source code I exported using the Encog workbench but I am not sure if that works correctly. I tried to connect the output layer with the input layer of the filter network using the direct synapse, is that the right approach? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In that case I'd need to add the additional 15 data inputs seperately, right? I'd really happy if someone could help me to get this implementation started. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you very much!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sue&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;-- Exported Source Code using Encog workbench-----  &#xA;&#xA;public class EncogGeneratedClass {&#xA;&#xA;  public static void main(final String args[]) {&#xA;&#xA;    BasicNetwork network = new BasicNetwork();&#xA;&#xA;    Layer hiddenLayer1 = new BasicLayer( new ActivationTANH(),true,4);&#xA;    Layer inputLayer = new BasicLayer( new ActivationTANH(),false,75);&#xA;    Layer hiddenLayer2 = new BasicLayer( new ActivationTANH(),true,75);&#xA;    Layer outputLayer = new BasicLayer( new ActivationTANH(),true,4);&#xA;    Layer hiddenLayer3 = new BasicLayer( new ActivationTANH(),true,66);&#xA;    Layer hiddenLayer4 = new BasicLayer( new ActivationTANH(),false,210);&#xA;&#xA;    Synapse synapse1 = new WeightedSynapse(hiddenLayer3,hiddenLayer1);&#xA;    Synapse synapse2 = new DirectSynapse(hiddenLayer1,inputLayer);&#xA;    Synapse synapse3 = new WeightedSynapse(inputLayer,hiddenLayer2);&#xA;    Synapse synapse4 = new WeightedSynapse(hiddenLayer2,outputLayer);&#xA;    Synapse synapse5 = new WeightedSynapse(hiddenLayer4,hiddenLayer3);&#xA;&#xA;    hiddenLayer1.addSynapse(synapse2);&#xA;    inputLayer.addSynapse(synapse3);&#xA;    hiddenLayer2.addSynapse(synapse4);&#xA;    hiddenLayer3.addSynapse(synapse1);&#xA;    hiddenLayer4.addSynapse(synapse5);&#xA;&#xA;    network.tagLayer(&quot;INPUT&quot;,inputLayer);&#xA;    network.tagLayer(&quot;OUTPUT&quot;,outputLayer);&#xA;&#xA;    network.getStructure().finalizeStructure();&#xA;    network.reset();&#xA;  }&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="4323" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-26T08:55:40.090" LastActivityDate="2011-04-26T08:55:40.090" Title="How to connect two neural networks with Encog?" Tags="&lt;neural-networks&gt;" />
  <row Id="9969" PostTypeId="2" ParentId="9940" CreationDate="2011-04-25T20:32:48.797" Score="1" ViewCount="" Body="&lt;p&gt;In the asymptotic sense seemingly suggested by the phasing of the question, it's not true, but the analysis might be revealing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We don't even need $Z$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let $p$ be the chance of a standard normal variable being $c$ or less; that is, $p = \Phi(c)$.  Then the chance that at least $k$ or more of the $X_i$ are less than or equal to $c$ is given by a Binomial distribution&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\sum_{i=0}^{n-k} \binom{n}{i} p^{n-i}(1-p)^i\text{.}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because this sum runs from $p^n \lt p$ to $1 \gt p$, there exists a $k$ between $1$ and $n-1$ where the sum is as large as possible but still less than $p$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For future reference, note that as $n$ grows large, $k$ is approximately equal to $p n$.  This is a consequence of the Central Limit Theorem (for Binomial variates), because the sum is approximately equal to $\Phi((n-k - (1-p) n) / \sqrt{n p (1-p)})$.  If eventually $k$ were less than $p n$, say $k \lt (p - \epsilon)n$ for $\epsilon \gt 0$, then the sum would approximate $\Phi(\epsilon \sqrt{n} / \sqrt{p(1-p)})$, which approaches $1$ as $n$ increases, but the sum is constructed to stay below $p$.  Similarly, if $k \gt (p + \epsilon)n$, the sum would go to zero, again contradicting the construction of $k$ (to be as large as possible).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Define $q$ (which depends implicitly on $c$ and $n$) to be the value of the sum for such a $k$.  Let $b = \Phi^{-1}(q) \le c$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Conditional on at least $k$ of the $X_i$ not exceeding $c$, let $Y$ have a &lt;em&gt;truncated&lt;/em&gt; standard normal distribution ranging from $-\infty$ to $b$.  This happens with probability $q$.  Otherwise, let $Y$ have a truncated standard normal distribution ranging from $b$ to $+\infty$.  This gives $Y$ a standard normal distribution but it depends on the $X_i$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If $Y \le c$, the chance that $Y \le b$ equals $q/p$.  With sufficiently large $n$, an easy estimate shows this value is close to $1$.  Given that $Y \le b$, we &lt;em&gt;know&lt;/em&gt; at least $k$ of the $X_i$ are below $c$, by construction of $Y$.  Therefore the expected number of such $X_i$ is at least $p n$ (asymptotically in $n$).  This quantity grows without bound, &lt;strong&gt;it is not limited by a constant&lt;/strong&gt; (independent of $n$).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can work this analysis in reverse: if the expected number of $X_i$ below the threshold $c$, conditional on $Y \le c$, is much larger than $p n$, then the probability that $Y \le c$ would have to be greater than $\Phi(c)$, implying $Y$ does not have a standard Normal distribution.  In this sense the preceding construction is a worst case: asymptotically, it achieves the largest possible expected number of $X_i$ below $c$ consistent with the assumptions on $X_i$ and $Y$.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-04-25T20:32:48.797" CommentCount="5" />
  <row Id="9971" PostTypeId="1" CreationDate="2011-04-25T21:38:23.780" Score="3" ViewCount="65" Body="&lt;h3&gt;Context:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;I am trying to analyze an experiment on plant community response to two treatments.  Here’s a simplified description of the experiment, there are a few extra complications in reality.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Treatments were applied to small patches of ground arranged in blocks with a mix of naturally occurring plant species present in each patch. I have measured biomass of the plants in each patch in years one and two as the response variables.  Analyzing total biomass seems simple enough with a repeated measures GLM or linear mixed model and block a random effect.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However I am also interested in breaking down total biomass into four response groups of interest: perennial grasses, perennial forbs, annual grasses and annual forbs (forbs are plants that aren't grasses).  &lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Question:&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Because these groups sum up to total biomass can I go a head and analyze these responses with four separate repeated measures ANOVAs, or will I be guilty of re-analyzing the same experiment?  &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;It’s been suggested to me that I do a MANOVA followed by “protected ANOVA’s” but this doesn’t make sense to me because I am not interested in the multivariate response, rather I am interested in testing whether these specific variables have been affected by the treatments. I also don’t know how to do a repeated measures MANOVA correctly and it seems unduly complicated.  Any general advice would be appreciated—I’ll try and design a more easily analyzable experiment next time.&lt;/p&gt;&#xA;" OwnerUserId="4326" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-26T03:21:51.807" LastActivityDate="2011-04-26T05:13:34.833" Title="Is MANOVA the correct way to handle multiple response variables that are additive?" Tags="&lt;multivariate-analysis&gt;&lt;repeated-measures&gt;&lt;manova&gt;" AnswerCount="1" />
  <row Id="9972" PostTypeId="1" CreationDate="2011-04-25T21:54:51.503" Score="1" ViewCount="44" Body="&lt;p&gt;I'm compiling a survey that will have several questions which would lend to the creation of an index of a main dependent variable (level of engagement in sucession planning). The questions will involve topics like: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;PROCESS: linking strategic planning to succession planning, identifying critical positions that need to be filled,identifying competencies, identifying high potential employees, coaching and mentoring, providing leadership development; &lt;/li&gt;&#xA;&lt;li&gt;ROLE IDENTIFICATION: identifying roles (several options); and &lt;/li&gt;&#xA;&lt;li&gt;RESULTS: zero people ready for each position, 1 or 2 people ready for each position, or a pool of people ready for each position. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I'm not sure how to start. Any suggestions?&lt;/p&gt;&#xA;" OwnerUserId="4327" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-26T05:18:32.050" LastActivityDate="2011-04-26T14:11:36.723" Title="Getting started with creating an index based on multiple survey items" Tags="&lt;survey&gt;&lt;scales&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="9973" PostTypeId="2" ParentId="9961" CreationDate="2011-04-25T23:22:45.657" Score="1" ViewCount="" Body="&lt;p&gt;Per the &lt;a href=&quot;http://en.wikipedia.org/wiki/CMA-ES&quot; rel=&quot;nofollow&quot;&gt;wikipedia page&lt;/a&gt; linked above to answer (1) this is another form of gradient descent (which if you need more information with lots of pictures there are many articles available if you google it -- sorry apparently new posters only get 2 urls so I'm having to have to tell you to search instead of pointing you to specific ones) so it is used for optimization of an objective function. Usually this means I want to find the maximum or minimum value for a function I'm interested in.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Not to split hairs with the other poster, but how statistical this algorithm is a question of semantics. It is proposed to converge (for our considerations that is to say this method works) based on a maximum likelihood argument and covariance (variance in higher dimension -- the spread of the sample in the space) is used as part of the update rule for this algorithm to determine how best to proceed for the next iteration. Also new points are sampled from a multivariate normal distribution. Which is a probability model. Similarly as the wikipedia points out this method is similar to, but not identical to, principal component analysis. So from my perspective it seems pretty statistical, but again that is perhaps subjective.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suppose the second wiki link and parts of the above address (2), but perhaps not adequately so if you need more information please feel free to follow up. The short of it is if there is an optimum this procedure searches for it in the problem space and will eventually find it under the model assumptions. You need to test for the assumptions, because deviation from them is likely a major consequence to and how well this method will work on your problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I do however agree with the other poster that you may find more if not better answers on a site like &lt;a href=&quot;http://metaoptimize.com/qa/&quot; rel=&quot;nofollow&quot;&gt;Meta Optimize&lt;/a&gt; which is a community like this one specific to machine learning, but many Statisticians do work on learning problems so I would expect others will weigh in as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for (3) it is not necessarily a better method. It is a method for a different set of assumptions than other methods (such as feedforward neural network with gradient descent for example) specifically this method is for non-linear or non-convex problems. Again you need to test if the problem you are working on is appropriate for this method before proceeding to use it. Otherwise you may get a less than optimal solution or worse yet garbage in garbage out. If your problem is non-linear or non-convex then go for it, otherwise you may need to look into using a different method. There is a part of the wikipedia article &quot;Performance in Practice&quot; may help you determine whether this is an appropriate method for the problem you are working on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You alluded to some mathphobia when not wanting to see too many expressions. So you may not know some properties of the data/problem you're working with. Are you willing to share some more specific details or are you just asking in general?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hopefully this helps. Please let me know if there are any follow ups. Good luck.&lt;/p&gt;&#xA;" OwnerUserId="4325" LastActivityDate="2011-04-25T23:22:45.657" CommentCount="1" />
  <row Id="9974" PostTypeId="2" ParentId="9962" CreationDate="2011-04-26T00:15:37.927" Score="1" ViewCount="" Body="&lt;p&gt;Multinomial logit assumes that you have a categorical dependent variable.  In your case, there are three categories, denoted 0, 1, and 2.  You've set 1 as the reference category, which means that mlogit is going to use 1 as the baseline category -- everything else is compared to 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The thing to keep in mind is that in the mlogit framework, each additional category is compared against the reference group.  Each has its own log-linear model, with k+1 parameters, where k is the number of predictors in your model, and the 1 accounts for the intercept term.  Since you have 2 additional categories and 4 predictors, you're going to be estimating 2*(4+1) = 10 parameters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, alt0 is the intercept term for the 0-against-1 comparison, and alt-a,b,c and d describe the marginal change in likelihood between categories 0 and 1.  The alt2 parameters describe the category 2-against-1 comparison model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See &lt;a href=&quot;http://elsa.berkeley.edu/books/train1201.pdf&quot; rel=&quot;nofollow&quot;&gt;Train's excellent and (last I checked) free pdf book&lt;/a&gt; on discrete choice models for more background.&lt;/p&gt;&#xA;" OwnerUserId="4110" LastActivityDate="2011-04-26T00:15:37.927" CommentCount="1" />
  <row Id="9975" PostTypeId="2" ParentId="9926" CreationDate="2011-04-26T01:36:18.780" Score="7" ViewCount="" Body="&lt;p&gt;Letting $f$ denote a probability density function (either with respect to Lebesgue or counting measure, respectively), the quantity $\newcommand{\rd}{\mathrm{d}}$&#xA;$$&#xD;&#xA;H_\alpha(f) = -\frac{1}{\alpha-1} \log(\textstyle\int f^\alpha \rd \mu)&#xD;&#xA;$$ &#xA;is known as the &lt;a href=&quot;http://en.wikipedia.org/wiki/R%C3%A9nyi_entropy&quot; rel=&quot;nofollow&quot;&gt;Renyi entropy&lt;/a&gt; of order $\alpha \geq 0$. It is a generalization of Shannon entropy that retains many of the same properties. For the case $\alpha = 1$, we interpret $H_1(f)$ as $\lim_{\alpha \to 1} H_{\alpha}(f)$, and this corresponds to the standard Shannon entropy $H(f)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Renyi introduced this in his paper&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;A. Renyi, &lt;a href=&quot;http://digitalassets.lib.berkeley.edu/math/ucb/text/math_s4_v1_article-27.pdf&quot; rel=&quot;nofollow&quot;&gt;On measures of information and entropy&lt;/a&gt;, &lt;em&gt;Proc. 4th Berkeley Symp. on Math., Stat. and Prob.&lt;/em&gt; (1960), pp. 547&amp;ndash;561.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;which is well worth reading, not only for the ideas but for the exemplary exposition style.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The case $\alpha = 2$ is one of the more common choices for $\alpha$ and this special case is (also) often referred to as the Renyi entropy. Here we see that&#xA;$$\newcommand{\e}{\mathbb{E}}&#xD;&#xA;H_2(f) = - \log( \textstyle\int f^2 \rd \mu ) = -\log( \e f(X) ) &#xD;&#xA;$$&#xA;for a random variable distributed with density $f$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that $- \log(x)$ is a convex function and so, by Jensen's inequality we have&#xA;$$&#xD;&#xA;H_2(f) = -\log( \e f(X) ) \leq \e( -\log f(X) ) = - \e \log f(X) = H(f)&#xD;&#xA;$$&#xA;where the right-hand side denotes the Shannon entropy. Hence the Renyi entropy provides a lower bound for the Shannon entropy and, in many cases, is easier to calculate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another natural instance in which the Renyi entropy arises is when considering a discrete random variable $X$ and an independent copy $X^\star$. In some scenarios we want to know the probability that $X = X^\star$, which by an elementary calculation is&#xA;$$\renewcommand{\Pr}{\mathbb{P}}&#xD;&#xA;\Pr(X = X^\star) = \sum_{i=1}^\infty \Pr(X = x_i, X^\star = x_i) = \sum_{i=1}^\infty \Pr(X = x_i) \Pr(X^\star = x_i) = e^{-H_2(f)} .&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here $f$ denotes the density with respect to counting measure on the set of values $\Omega = \{x_i: i \in \mathbb{N}\}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The (general) Renyi entropy is also apparently related to free energy of a system in thermal equilibrium, though I'm not personally up on that. A (very) recent paper on the subject is&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;J. C. Baez, &lt;a href=&quot;http://arxiv.org/abs/1102.2098&quot; rel=&quot;nofollow&quot;&gt;Renyi entropy and free energy&lt;/a&gt;, arXiv [quant-ph] 1101.2098 (Feb. 2011).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="2970" LastActivityDate="2011-04-26T01:36:18.780" CommentCount="3" />
  <row Id="9976" PostTypeId="2" ParentId="2914" CreationDate="2011-04-26T01:38:14.393" Score="1" ViewCount="" Body="&lt;p&gt;I would suggest that this is a problem with how the results are reported.  Not to &quot;beat the Bayesian drum&quot; but approaching model uncertainty from a Bayesian perspective &lt;em&gt;as an inference problem&lt;/em&gt; would greatly help here.  And it doesn't have to be a big change either.  If the report simply contained the &lt;em&gt;probability that the model is true&lt;/em&gt; this would be very helpful.  This is an easy quantity to approximate using BIC.  Call the BIC for the mth model $BIC_{m}$.  Then the probability that mth model is the &quot;true&quot; model, given that $M$ models were fit (and that one of the models is true) is given by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(\text{model m is true}|\text{one of the M models is true})\approx\frac{w_{m}\exp\left(-\frac{1}{2}BIC_{m}\right)}{\sum_{j=1}^{M}w_{j}\exp\left(-\frac{1}{2}BIC_{j}\right)}$$&#xA;$$=\frac{1}{1+\sum_{j\neq m}^{M}\frac{w_{j}}{w_{m}}\exp\left(-\frac{1}{2}(BIC_{j}-BIC_{m})\right)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $w_{j}$ is proportional to the prior probability for the jth model.  Note that this includes a &quot;penalty&quot; for trying to many models - and the penalty depends on how well the other models fit the data.  Usually you will set $w_{j}=1$, however, you may have some &quot;theoretical&quot; models within your class that you would expect to be better prior to seeing any data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now if somebody else doesn't report all the BIC's from all the models, then I would attempt to infer the above quantity from what you have been given.  Suppose you are given the BIC from the model - note that BIC is calculable from the mean square error of the regression model, so you can always get BIC for the reported model.  Now if we take the basic premise that the final model was chosen from the smallest BIC then we have $BIC_{final}&amp;lt;BIC_{j}$.  Now, suppose you were told that &quot;forward&quot; or &quot;forward stepwise&quot; model selection was used, starting from the intercept using $p$ potential variables.  If the final model is of dimension $d$, then the procedure must have tried at least &#xA;$$M\geq 1+p+(p-1)+\dots+(p-d+1)=1+\frac{p(p-1)-(p-d)(p-d-1)}{2}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;different models (exact for forward selection),  If the backwards selection was used, then we know at least &#xA;$$M\geq 1+p+(p-1)+\dots+(d+1)=1+\frac{p(p-1)-d(d-1)}{2}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Models were tried (the +1 comes from the null model or the full model).  Now we could try an be more specific, but these are &quot;minimal&quot; parameters which a standard model selection must satisfy.  We could specify a probability model for the number of models tried $M$ and the sizes of the $BIC_{j}$ - but simply plugging in some values may be useful here anyway.  For example suppose that all the BICs were $\lambda$ bigger than the one of the model chosen so that $BIC_{m}=BIC_{j}-\lambda$, then the probability becomes:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{1}{1+(M-1)\exp\left(-\frac{\lambda}{2}\right)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So what this means is that unless $\lambda$ is large or $M$ is small, the probability will be small also.  From an &quot;over-fitting&quot; perspective, this would occur when the BIC for the bigger model is not much bigger than the BIC for the smaller model - a non-neglible term appears in the denominator.  Plugging in the backward selection formula for $M$ we get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{1}{1+\frac{p(p-1)-d(d-1)}{2}\exp\left(-\frac{\lambda}{2}\right)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now suppose we invert the problem.  say $p=50$ and the backward selection gave $d=20$ variables, what would $\lambda$ have to be to make the probability of the model greater than some value $P_{0}$? we have&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\lambda &amp;gt; -2 log\left(\frac{2(1-P_{0})}{P_{0}[p(p-1)-d(d-1)]}\right)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Setting $P_{0}=0.9$ we get $\lambda &amp;gt; 18.28$ - so BIC of the winning model has to win by a lot for the model to be certain.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-04-26T01:38:14.393" />
  <row Id="9977" PostTypeId="1" CreationDate="2011-04-26T03:05:11.570" Score="4" ViewCount="34" Body="&lt;p&gt;I'm trying to estimate the design effect of a series of relatively small sample size surveys ($n\sim 70$) with multiple responses. Design effects roughly correspond to how much larger actual sample variance than would be expected from naive random sampling. The simplest way to parametrize this is for Effective Sample Size to be equal to SampleSize/D, where D is an unknown parameter. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Throwing it naively into BUGS is impossible due to the inability to handle multinomial models with unknown sample size. I tried to do a multinormal approximation with a structured covariance matrix, but I can't figure out/find a closed form for the precision matrix of a multinomial variable. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone have any ideas about the best way to proceed? This seems like it should be a common problem. &lt;/p&gt;&#xA;" OwnerUserId="996" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-26T12:00:01.143" LastActivityDate="2011-04-26T12:00:01.143" Title="Modeling multinomial problems with unknown sample size in BUGS" Tags="&lt;bayesian&gt;&lt;sampling&gt;&lt;mcmc&gt;&lt;sample-size&gt;&lt;bugs&gt;" FavoriteCount="1" />
  <row Id="9978" PostTypeId="2" ParentId="9947" CreationDate="2011-04-26T04:29:33.870" Score="4" ViewCount="" Body="&lt;p&gt;The problem you're trying to solve is a &lt;a href=&quot;http://en.wikipedia.org/wiki/Hungarian_algorithm&quot; rel=&quot;nofollow&quot;&gt;min-cost matching problem&lt;/a&gt;, specifically the problem of minimizing the functional &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$F(\pi) = \sum_i \|c_i - k_{\pi(i)}\|^2 $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\pi$ is over all permutations in $S_n$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This can be solved by the Hungarian algorithm (which is a primal-dual method in disguise) and takes $n^3$ time. &lt;/p&gt;&#xA;" OwnerUserId="139" LastActivityDate="2011-04-26T04:29:33.870" CommentCount="1" />
  <row Id="9979" PostTypeId="2" ParentId="9971" CreationDate="2011-04-26T05:13:34.833" Score="1" ViewCount="" Body="&lt;p&gt;I can see the merits in running four separate repeated measures ANOVAs. If your theoretical question concerns the four individual variables, then running the ANOVAs separately is more aligned with your theoretical question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I guess the main issue is the parsimony of your approach and controlling your Type I error rate. Here are a few possibilities that you could explore:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;You could apply a Bonferonni correction to your $\alpha$ level (e.g., $\alpha=.05 /4=.0125$) so that your family-wise $\alpha$ is kept at an acceptable level.&lt;/li&gt;&#xA;&lt;li&gt;You could distinguish analyses into confirmatory and more exploratory analyses. For example, you could frame the effect of time on overall biomass as confirmatory and interpret the effect of time on individual aspects of biomass as exploratory analyses.&lt;/li&gt;&#xA;&lt;li&gt;You might even include type of biomass as another independent variable in your ANOVA. Then you could see whether there was a time by &quot;type of biomass&quot; interaction. This might be a more parsimonious approach because it, in some senses, starts with the assumption that any effect of time on type of biomass is constant across types of biomass.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastActivityDate="2011-04-26T05:13:34.833" CommentCount="1" />
  <row Id="9980" PostTypeId="2" ParentId="9966" CreationDate="2011-04-26T06:40:19.617" Score="1" ViewCount="" Body="&lt;p&gt;Check the article by Wu Chen Hastie Sobel Lange - Genome-wide association analysis by lasso penalized logistic regression - 2009. They mention a 'swindle' that is not hard to implement + then you can simply work with glmnet (there is a new version out recently which promises a performance improvement but I haven't had a chance to check it out).&lt;/p&gt;&#xA;" OwnerUserId="4257" LastActivityDate="2011-04-26T06:40:19.617" />
  <row Id="9981" PostTypeId="1" AcceptedAnswerId="9986" CreationDate="2011-04-26T06:54:27.007" Score="4" ViewCount="46" Body="&lt;p&gt;In a technique that uses CUSUM for change-point detection in this &lt;a href=&quot;http://www.cs.utexas.edu/~mahimkar/MERCURY_sigcomm10.pdf&quot; rel=&quot;nofollow&quot;&gt;paper&lt;/a&gt;, the first step is given below:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Let $x_1, x_2,..., x_n$ be the $n$&#xA;  samples in an event-series. The&#xA;  samples are ranked in increasing order&#xA;  and the rank $r_i$ for each sample is&#xA;  calculated. In case of ties, we assign&#xA;  average rank to each sample. The&#xA;  cumulative sums are computed as: $S_i = S_{i-1} + (r_i - \bar{r})$.&#xA;  If the ranks are randomly distributed, then&#xA;  there is no change-point. However, if&#xA;  there is indeed a change-point in the&#xA;  event-series, then higher ranks should&#xA;  dominate in either the earlier or&#xA;  later part of the event-series.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I am not quite sure I understand the meaning of rank in this context. For instance, if n=10 and my data points are: 5, 2, 4, 1, 9, 2, 9, 2, 10, 1 can someone please clarify what is really being done here?&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Sort in increasing order: 1, 1, 2, 2, 2, 4, 5, 9, 9, 10&lt;/li&gt;&#xA;&lt;li&gt;Assign Average Ranks to break ties: What does this step mean?&lt;/li&gt;&#xA;&lt;li&gt;Assign Final Ranks: What does this step mean?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="2164" LastEditorUserId="1390" LastEditorDisplayName="" LastEditDate="2011-04-26T17:52:45.217" LastActivityDate="2011-04-26T17:52:45.217" Title="What is the meaning of rank in the context of change-detection?" Tags="&lt;statistical-significance&gt;&lt;nonparametric&gt;&lt;change-point&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="9982" PostTypeId="2" ParentId="9739" CreationDate="2011-04-26T07:13:33.340" Score="0" ViewCount="" Body="&lt;p&gt;A nicely documented python library for spatial analysis that has some clustering is &lt;a href=&quot;http://pysal.org/1.1/library/region/index.html&quot; rel=&quot;nofollow&quot;&gt;pySAL&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another python library in the development stage that is focused on spatial clustering is &lt;a href=&quot;http://www.rise-group.org/risem/clusterpy/clusterPy-pysrc.html&quot; rel=&quot;nofollow&quot;&gt;clusterPy&lt;/a&gt; &lt;a href=&quot;http://dl.dropbox.com/u/408103/ClusterPy-Slides.pdf&quot; rel=&quot;nofollow&quot;&gt;(pdf slide presentation)&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With a more limited choice of clustering algorithms but with nice mapping interface is the GUI software &lt;a href=&quot;http://geogrouper.appspot.com/&quot; rel=&quot;nofollow&quot;&gt;GeoGrouper&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="4329" LastActivityDate="2011-04-26T07:13:33.340" CommentCount="1" />
  <row Id="9983" PostTypeId="2" ParentId="277" CreationDate="2011-04-26T07:24:16.177" Score="2" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Non-spatial model&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My House Value is a function of my home Gardening Investment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;SAR model&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My House Value is a function of the House Values of my neighbours. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;CAR model&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My House Value is a function of the Gardening Investment of my neighbours.&lt;/p&gt;&#xA;" OwnerUserId="4329" LastActivityDate="2011-04-26T07:24:16.177" />
  <row Id="9984" PostTypeId="2" ParentId="9928" CreationDate="2011-04-26T07:53:57.130" Score="2" ViewCount="" Body="&lt;p&gt;The definition of fat-tail in &lt;a href=&quot;http://en.wikipedia.org/wiki/Fat_tail&quot; rel=&quot;nofollow&quot;&gt;wikipedia&lt;/a&gt; is that &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p(x)\sim x^{-(\alpha+1)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;as $x\to\infty$ for some $\alpha&amp;gt;0$. Now &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{e^x}{x^{\alpha+1}}\to\infty,$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;as $x\to\infty$, so the $Ee^X$ cannot exist for such type of distributions. So you need to precise what do you have in mind by saying fat-tailed.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-04-26T07:53:57.130" CommentCount="3" />
  <row Id="9985" PostTypeId="2" ParentId="9809" CreationDate="2011-04-26T10:07:42.067" Score="1" ViewCount="" Body="&lt;p&gt;Let's build it!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You mentioned:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1 moment generating function&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2 law of iterated expectations&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3 change of measure&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Adding:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4 Decompose random variable as a sum. Usually the sum of indicators of something.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;5 Build a reccurence relation for E(X) (or a set of linear equations). Useful in Markov Chains.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;6 Stopping time theorem for martingales: $E(X_{T})=E(X_{1})$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;6b Wald identity. $E(S_{T})=E(X_{1})E(T)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;7 Kolmogorov forward/backward equation&lt;/p&gt;&#xA;&#xA;&lt;p&gt;8 &lt;a href=&quot;http://web.mit.edu/urban_or_book/www/book/chapter3/3.5.html&quot; rel=&quot;nofollow&quot;&gt;Crofton&lt;/a&gt;'s method&lt;/p&gt;&#xA;&#xA;&lt;p&gt;9 General idea: Symmetry. Especially for &quot;$n$ points are chosen uniformly on the $[0;1]$&quot; problems.&lt;/p&gt;&#xA;" OwnerUserId="2043" LastEditorUserId="2043" LastEditorDisplayName="" LastEditDate="2011-04-26T10:29:53.403" LastActivityDate="2011-04-26T10:29:53.403" CommentCount="1" />
  <row Id="9986" PostTypeId="2" ParentId="9981" CreationDate="2011-04-26T10:42:52.393" Score="6" ViewCount="" Body="&lt;p&gt;Given your data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cp &amp;lt;- c(5, 2, 4, 1, 9, 2, 9, 2, 10, 1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;then the ranks, with ties being given average of the ranks, are:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; rank(cp)&#xA; [1]  7.0  4.0  6.0  1.5  8.5  4.0  8.5  4.0 10.0  1.5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What is being done here? If you sort the data in increasing order, then we have a &lt;code&gt;1&lt;/code&gt; in both rank order positions 1 &lt;em&gt;and&lt;/em&gt; 2. We could assign rank 1 to both &lt;code&gt;1&lt;/code&gt;s, or rank 2, or as stated above, the average of the rank orders (1/2) / 2 = 1.5. This is why the two &lt;code&gt;1&lt;/code&gt;s have been given rank of 1.5 in the above output from R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now look at the next values in the rank order, the &lt;code&gt;2&lt;/code&gt;s. The &lt;code&gt;2&lt;/code&gt;'s are in rank order positions 3, 4, and 5, therefore they all get rank 4 from (3+4+5) / 3 = 4, as this is the average of the tied ranks for these values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we initiate $S_0 = 0$, i.e. the zeroth cumulative sum is 0, we compute the $i$th cumulative sum ($S_i$) as the previous cumulative sum ($S_{i-1}$) plus the difference between the rank of the $i$th data point ($r_i$) and the average over all ranks $\bar{r}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the above data, the average rank is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; rcp &amp;lt;- rank(cp)&#xA;&amp;gt; mean(rcp)&#xA;[1] 5.5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The values $r_i - \bar{r}$ for this set of data are:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; rcp - mean(rcp)&#xA; [1]  1.5 -1.5  0.5 -4.0  3.0 -1.5  3.0 -1.5  4.5 -4.0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and the cumulative sums are:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; cumsum(rcp - mean(rcp))&#xA; [1]  1.5  0.0  0.5 -3.5 -0.5 -2.0  1.0 -0.5  4.0  0.0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1390" LastActivityDate="2011-04-26T10:42:52.393" CommentCount="7" />
  <row Id="9987" PostTypeId="1" CreationDate="2011-04-26T11:35:49.713" Score="3" ViewCount="148" Body="&lt;p&gt;I know that &quot;deviations in the data are devil&quot;, and when the distribution is highly skewed, it is better to consider median as average rather than mean, but how to decide these hard-limits.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;CASE 1:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Assume X = 10,20,30,40,50,60,70&lt;/li&gt;&#xA;&lt;li&gt;In this case, I think that it is better to use mean and that it will give very accurate results.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;CASE 2:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Assume X = 10,20,30,40,50,60,70,7000&lt;/li&gt;&#xA;&lt;li&gt;In this case, I think that it is better to use median instead of using the mean.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;CASE 3: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Assume X = 10,20,30,400,500,600,700&lt;/li&gt;&#xA;&lt;li&gt;In this case, I think it is better to use IQR (Inter Quartile Range)&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;But I'm stuck with how to decide these hard-limits i.e. which to use in which condition, in general. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've found a tool working on subjected principle, which takes context-less sample-distribution as input and determines whether mean is close/moderate or against the null-hypothesis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Find References:- &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://home.ubalt.edu/ntsbarsh/Business-stat/otherapplets/MeanTest.htm&quot; rel=&quot;nofollow&quot;&gt;http://home.ubalt.edu/ntsbarsh/Business-stat/otherapplets/MeanTest.htm&lt;/a&gt; (For Mean Test)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://home.ubalt.edu/ntsbarsh/Business-stat/otherapplets/MediansTest.htm&quot; rel=&quot;nofollow&quot;&gt;http://home.ubalt.edu/ntsbarsh/Business-stat/otherapplets/MediansTest.htm&lt;/a&gt; (For Median Test)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;What I'm really looking is a good answer which states how to derive these conclusions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help or learning pointers will be highly appreciated.&lt;/p&gt;&#xA;" OwnerUserId="4331" LastEditorUserId="4331" LastEditorDisplayName="" LastEditDate="2011-04-27T10:36:23.297" LastActivityDate="2011-04-27T10:36:23.297" Title="When does the amount of skew or prevalence of outliers make the median preferable to the mean?" Tags="&lt;normal-distribution&gt;&lt;mean&gt;&lt;median&gt;" AnswerCount="5" CommentCount="3" FavoriteCount="1" />
  <row Id="9988" PostTypeId="1" CreationDate="2011-04-26T11:37:30.257" Score="3" ViewCount="78" Body="&lt;p&gt;I received this question by email from a Neuroscience PhD student.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;I would greatly appreciate if you&#xA;  could please let me know whether&#xA;  Factor Analysis could load positively&#xA;  and inversely correlated variables&#xA;  onto the same latent factor, whereas&#xA;  Cluster Analysis can only cluster into&#xA;  the same factor either positively or&#xA;  inversely correlated variables (e.g.,&#xA;  they would be segregated into 2&#xA;  factors).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="183" LastActivityDate="2011-04-27T05:01:49.230" Title="Can cluster analysis cluster variables that both positively and negatively correlate with each other?" Tags="&lt;clustering&gt;&lt;factor-analysis&gt;" AnswerCount="1" CommentCount="8" FavoriteCount="1" />
  <row Id="9989" PostTypeId="2" ParentId="9987" CreationDate="2011-04-26T11:49:10.857" Score="3" ViewCount="" Body="&lt;p&gt;You can read about measures of central tendency here: &lt;a href=&quot;http://en.wikipedia.org/wiki/Central_tendency&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Central_tendency&lt;/a&gt; .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Generally, you analyse a sample in order to tell something about a (much larger) population. Often you know more about the population than merely the data in your sample, usually something that motivated you to take a sample in the first place. If you know that the population has normal distribution then the sample mean will be the best estimator of the expected value even if the sample does not look normal (using small sample sizes like the above you can't really characterise the distribution anyway). You can reliably estimate the mean if you have a lot of data even if the distribution is not normal (see &lt;a href=&quot;http://stats.stackexchange.com/questions/9573/t-test-for-non-normal-when-n50&quot;&gt;T-test for non normal when N&amp;gt;50?&lt;/a&gt;). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In cases of distributions that can not be described parametrically median and the IQR may tell much more. IQR is a dispersion measure as opposed to the mean and median that are location measures. You can read about dispersion parameters here: &lt;a href=&quot;http://en.wikipedia.org/wiki/Statistical_dispersion&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Statistical_dispersion&lt;/a&gt; .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A further aspect to consider is that some of your data may be outliers (see &lt;a href=&quot;http://stats.stackexchange.com/questions/7155/rigorous-definition-of-an-outlier&quot;&gt;Rigorous definition of an outlier?&lt;/a&gt;).&lt;/p&gt;&#xA;" OwnerUserId="3911" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-26T13:02:21.527" LastActivityDate="2011-04-26T13:02:21.527" CommentCount="3" />
  <row Id="9990" PostTypeId="1" CreationDate="2011-04-26T12:33:47.493" Score="5" ViewCount="62" Body="&lt;p&gt;I'm decently familiar with mixed effects models (MEM), but a colleague recently asked me how it compares to latent growth models (LGM). I did a bit of googling, and it seems that LGM is a variant of structural equation modelling that is applied to circumstances where repeated measures are obtained within each level of at least one random effect, thus making Time a fixed effect in the model. Otherwise, MEM and LGM seem pretty similar (eg. they both permit exploration of different covariance structures, etc).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Am I correct that LGM is conceptually a special case of MEM, or are there differences between the two approaches with respect to their assumptions or capacity to evaluate different types of theories?&lt;/p&gt;&#xA;" OwnerUserId="364" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-26T14:16:02.490" LastActivityDate="2011-04-26T14:16:02.490" Title="What are the differences between &quot;Mixed Effects Modelling&quot; and &quot;Latent Growth Modelling&quot;?" Tags="&lt;mixed-model&gt;&lt;longitudinal&gt;" CommentCount="1" FavoriteCount="1" />
  <row Id="9991" PostTypeId="2" ParentId="9987" CreationDate="2011-04-26T12:58:56.007" Score="1" ViewCount="" Body="&lt;p&gt;Be careful with medians: they are biased estimators and the degree of bias can change depending on the skew of the distribution and the sample size (see &lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pubmed/2971778&quot; rel=&quot;nofollow&quot;&gt;Miller, 1988&lt;/a&gt;). This means that if you are comparing two conditions that have either different skew or different sample sizes, you may find a difference that is in fact attributable to bias rather than a real difference, or you may fail to find a difference when there is a real one when the direction of the difference in bias is opposite to the direction of the real difference between the conditions.&lt;/p&gt;&#xA;" OwnerUserId="364" LastActivityDate="2011-04-26T12:58:56.007" CommentCount="4" />
  <row Id="9993" PostTypeId="2" ParentId="9987" CreationDate="2011-04-26T13:17:12.987" Score="0" ViewCount="" Body="&lt;p&gt;There are no hard and fast rules.  They convey different information and have different properties.  You select the statistic that best conveys what you want to convey.  Or better yet, select statistics that best describe the data.  Keep this same thing in mind when you're selecting a measure of central tendency to analyze.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(snipped a bunch of stuff repeating Mike Lawrence's answer)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that Mike Lawrence is referring to something that's surprising for a lot of people.  In the behavioural sciences there's a lot of folk wisdom that you use medians with small sample sizes.  But in actual fact that's exactly the wrong thing to do because the median quickly becomes more biased than the mean with small samples.&lt;/p&gt;&#xA;" OwnerUserId="601" LastActivityDate="2011-04-26T13:17:12.987" CommentCount="5" />
  <row Id="9994" PostTypeId="2" ParentId="9987" CreationDate="2011-04-26T14:02:40.367" Score="5" ViewCount="" Body="&lt;h3&gt;Framing the question&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;You are asking an applied and subjective question, and thus, any answer needs to be infused with applied and subjective considerations.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;From a purely statistical perspective, the mean and median both provide different information about the central tendency of a sample of data. Thus, neither is correct or incorrect by definition.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;From an applied perspective, we often want to say something meaningful about the central tendency of a sample, where central tendency maps onto some subjective notion of &quot;typical&quot;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;General thoughts&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;When summarising what is typical in a sample, observations that are many standard deviations away from the mean (perhaps 3 or 4 SD) will have a large influence on the mean, but not the median. Such observations may lead the mean to deviate from what we think of as the &quot;typical&quot;  value of the sample. This helps to explain the popularity of the median when it comes to reporting house prices and income, where a single island in the pacific or billionaire could dramatically influence the mean, but not the median. Such distributions can often include extreme outliers, and the distribution is positively skewed. In contrast, the median is robust.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The median can be problematic when the data takes on a limited number of values. &#xA;For example, the median of a 5-point Likert item lacks the nuance possessed by the mean. For example, means of 2.8, 3.0, and 3.3 might all have a median of 3.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;In general, the mean has the benefit of using more of the information from the data.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;When skewed distributions exist, it is also possible to transform the distribution and report the mean of the transformed distribution.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;When a distribution includes outliers, it is possible to use a trimmed mean, or remove the outliers, or adjust the value of the outlier to a less extreme value (e.g., 2 SD from the mean).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastActivityDate="2011-04-26T14:02:40.367" CommentCount="5" />
  <row Id="9995" PostTypeId="2" ParentId="9972" CreationDate="2011-04-26T14:11:36.723" Score="1" ViewCount="" Body="&lt;p&gt;Check out some of the following:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;the literature on &lt;strong&gt;&lt;a href=&quot;http://jeromyanglim.blogspot.com/2009/10/exploratory-factor-analysis-and-scale.html&quot; rel=&quot;nofollow&quot;&gt;scale construction&lt;/a&gt;&lt;/strong&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;discussions of &lt;strong&gt;formative and reflective&lt;/strong&gt; indicators (&lt;a href=&quot;http://neoacademic.com/2011/04/25/stats-and-methods-urban-legends-1-formative-measurement/&quot; rel=&quot;nofollow&quot;&gt;here's a discussion&lt;/a&gt;).&#xA;It sounds like your scale might be formative, in that it is driven by theoretical definition of a construct rather than some natural correlation between the items and facets that you mention.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://stats.stackexchange.com/questions/9358/creating-an-index-of-quality-from-multiple-variables-to-enable-rank-ordering/9361#9361&quot;&gt;whuber's comments on this earlier question on an index of soil quality&lt;/a&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastActivityDate="2011-04-26T14:11:36.723" CommentCount="3" />
  <row Id="9997" PostTypeId="1" CreationDate="2011-04-26T15:25:45.430" Score="3" ViewCount="62" Body="&lt;p&gt;I've been doing some Machine Learning, and have been using k-fold cross-validation to assess the generalisation performance of the algorithm. I've tried k-fold cross-validation with k = 5 and k = 200 and get very different results for Support Vector Machine classification.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;k    SVM accuracy&#xA;-----------------&#xA;5    75%&#xA;200  94%&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This seems like a huge difference in accuracy caused by changing the number of splits we're doing for the k-fold cross-validation. Is there any reason for this? I can't seem to find any references on studies that have been done investigating the effects of using different k values. Obviously, which k value I decide to use in my report gives completely different impressions of the quality of my classifier!&lt;/p&gt;&#xA;" OwnerUserId="261" LastActivityDate="2011-04-26T16:54:13.690" Title="Information on how value of k in k-fold cross-validation affects resulting accuracies" Tags="&lt;machine-learning&gt;&lt;cross-validation&gt;&lt;svm&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="9998" PostTypeId="2" ParentId="9987" CreationDate="2011-04-26T16:39:30.940" Score="1" ViewCount="" Body="&lt;p&gt;&quot;deviations in the data are the devil&quot; is just not true I think - well I don't agree with it at least.  I'd say its more like &quot;chilli&quot; than the &quot;devil&quot; - as much as you can reasonably handle is good, but it can get nasty if there is too much.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The most general procedure I know of to &quot;choose a statistic&quot; to report your data is a combination of two things&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Bayesian inference (describing what is known)&lt;/li&gt;&#xA;&lt;li&gt;Decision theory (taking actions under uncertainty)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;However, both of these methods are only partially &quot;algorithmic&quot; so to speak.  You have to supply the inputs though.  Perhaps the most important part of this stage is that &lt;em&gt;you have to ask a question&lt;/em&gt; that your procedure is going to answer.  Naturally, different questions get different answers.  As the saying goes &quot;I have just derived a very elegant and beautiful answer.  All I have to do now is figure out the question.&quot;  This is a common problem that I have seen with many statistical procedures, is that there is not always a clear statement of the class of problems that it is the best procedure to use.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bayesian inference requires you to specify your prior information in a mathematical framework.  This involves&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Specifying the hypothesis space - what possibilities am I going to consider?&lt;/li&gt;&#xA;&lt;li&gt;Assigning probabilities to each part of the space&lt;/li&gt;&#xA;&lt;li&gt;Using the rules of probability theory to manipulate the assigned probabilities&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;this is basically an open-ended problem (you can always analyse a given English statement more deeply, to extract more or different information from it).  Decision theory also requires you to specify a loss function - and there are basically no rules or principles by which to do this, at least as far as I know (computational simplicity is a key driver).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One useful question to ask yourself though is &quot;what information about the sample do I convey by presenting this statistic?&quot;  or &quot;how much of the complete data set can I recover from using just this set of statistics?&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One way you could use Bayesian statistics to help you here is to propose a hypothesis:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\begin{array}{l l}&#xD;&#xA;  H_{mean}:\text{The mean is the best statistic}&#xD;&#xA;\\H_{med}:\text{The median is the best statistic}&#xD;&#xA;\\H_{IQR}:\text{The IQR is the best statistic}&#xD;&#xA;\end{array}&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now these are not &quot;mathematically well posed&quot; hypothesis, but if we use them anyway, and see what parts of the maths are required to make it well posed.  The first part is the prior probabilities, without any data, how likely is each hypothesis?  The usual answer is equal probabilities (but not always - may have some theoretical reason to support one hypothesis being more likely - the CLT is perhaps one for $H_{mean}$ being higher than the others).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So we use Bayes theorem to update each probability ($I$=prior information, $D$= data set):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(H_{i}|D,I)=P(H_{i}|I)\frac{P(D|H_{i},I)}{P(D|I)}\implies \frac{P(H_{i}|D,I)}{P(H_{j}|D,I)}=\frac{P(H_{i}|I)}{P(H_{j}|I)}\frac{P(D|H_{i},I)}{P(D|H_{j},I)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So if the prior probabilities are equal, then the relative probabilities are given by the likelihood ratio.  So you also need to specify a probability distribution for what type of data sets you would be likely to see if the mean was the best statistic, etc.  Note that each hypothesis doesn't actually state what the specific value of the mean, median, or IQR actually is.  Therefore, the probability cannot depend on the exact value of the mean.  Hence in the likelihoods these must have been &quot;integrated out&quot; using the sum and product rules&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(D|H_{i},I)=\int P(\theta_{i}|H_{i},I)P(D|\theta_{i},H_{i},I)d\theta_{i}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So you have the prior $P(\theta_{i}|H_{i},I)$ which can be interpreted for i=mean as &quot;given that the mean is the best statistic, and prior to seeing the data, what values of the mean are we likely to see?&quot; and the likelihood $P(D|\theta_{i},H_{i},I)$ can be similarly interpreted as &quot;given the mean is best, and equal to $\theta_{mean}$ how likely is the data that was observed?&quot;.  This may help you come up with some kinds of features that your distribution should have.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This describes the inference - now it is time to apply decision theory.  This is particularly simple because your decision doesn't influence the state of nature - the statistic won't change if you do or don't use it.  So we can describe the decisions ($A$ for &quot;action&quot; because $D$ is already taken):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\begin{array}{l l}&#xD;&#xA;  A_{mean}:\text{The mean is the reported statistic}&#xD;&#xA;\\A_{med}:\text{The median is the reported statistic}&#xD;&#xA;\\A_{IQR}:\text{The IQR is the reported statistic}&#xD;&#xA;\end{array}&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And now you need to specify a loss matrix $L_{ij}$ which relates the action/decision $A_{i}$ to the state of nature $H_{j}$ - what is the loss if I report the mean, but the median is actually the best statistic?  In most cases the diagonal elements will be zero - taking the correct action means no loss.  You may also have that all non-diagonal elements are equal - how you are wrong doesn't matter, only whether or not you are wrong.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You then proceed by calculating the average loss for each action, weighted by their probabilities:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$L_{i}=\sum_{j}L_{ij}P(H_{j}|D,I)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And you then choose the action with the smallest average loss.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-04-26T16:39:30.940" />
  <row Id="9999" PostTypeId="2" ParentId="9997" CreationDate="2011-04-26T16:54:13.690" Score="2" ViewCount="" Body="&lt;p&gt;Not much of a &quot;proof&quot; but when k is small, you are removing a much larger chunk of your data, so you model has a much smaller amount of data to &quot;learn from&quot;.  For k=5 you are removing 20% of the data each time, whereas for k=200 you are only removing 0.5%.  You model has a much better chance of picking up all the relevant &quot;structure&quot; in the training part when k is large.  When k is small, the is a larger chance that the &quot;left out&quot; part will contain a structure which is absent from the &quot;left in&quot; bit - a bit like an &quot;un-representative&quot; sub-sample.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-04-26T16:54:13.690" CommentCount="3" />
  <row Id="10001" PostTypeId="1" CreationDate="2011-04-26T18:39:04.007" Score="4" ViewCount="161" Body="&lt;p&gt;I have noticed that there are a few implementations of random forest such as ALGLIB, Waffles and some R packages like 'randomForest'.  Can anybody tell me whether these libraries are highly optimized?  Are they basically equivalent to the random forests as detailed in Elements of Statistical Learning or have a lot of extra tricks been added?  I hope this question is specific enough.  As an illustration of the type of answer I am looking for, if somebody asked me whether the linear algebra package BLAS was highly optimized, I would say it was extremely highly optimized and mostly not worth trying to improve upon except in very specialized applications.&lt;/p&gt;&#xA;" OwnerUserId="847" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-26T20:12:58.453" LastActivityDate="2011-05-01T17:49:37.287" Title="Implementations of the Random Forest algorithm" Tags="&lt;algorithms&gt;&lt;random-forest&gt;" AnswerCount="2" FavoriteCount="2" />
  <row Id="10002" PostTypeId="1" CreationDate="2011-04-26T19:43:37.553" Score="4" ViewCount="71" Body="&lt;p&gt;Given a predicted variable (P), a random effect (R) and a fixed effect (F), one could fit two* mixed effects models (&lt;a href=&quot;http://cran.r-project.org/web/packages/lme4/&quot; rel=&quot;nofollow&quot;&gt;lme4&lt;/a&gt; syntax):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;m1 = lmer( P ~ (1|R) + F )&#xA;m2 = lmer( P ~ (1+F|R) + F)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As I understand it, the second model is the one that permits the fixed effect to vary across levels of the random effect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my research I typically employ mixed effects models to analyze data from experiments conducted across multiple human participants. I model participant as a random effect and experimental manipulations as fixed effects. I think it makes sense a priori to let the degree to which the fixed effects affect performance in the experiment vary across participants. However, I have trouble imagining circumstances under which I should nor permit the fixed effects to vary across levels of a random effect, so my question is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When should one &lt;strong&gt;not&lt;/strong&gt; permit a fixed effect to vary across levels of a random effect?&lt;/p&gt;&#xA;" OwnerUserId="364" LastActivityDate="2011-04-27T11:12:22.073" Title="When should I *not* permit a fixed effect to vary across levels of a random effect in a mixed effects model?" Tags="&lt;mixed-model&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="3" />
  <row Id="10003" PostTypeId="1" AcceptedAnswerId="11139" CreationDate="2011-04-26T20:09:18.910" Score="8" ViewCount="121" Body="&lt;h2&gt;Background&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I read about &lt;a href=&quot;http://statprob.com&quot; rel=&quot;nofollow&quot;&gt;StatProb.com&lt;/a&gt; from a comment on &lt;a href=&quot;http://www.stat.columbia.edu/~cook/movabletype/archives/2011/03/why_edit_wikipe.html#comment-2187431&quot; rel=&quot;nofollow&quot;&gt;Andrew Gelman's Blog&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;According to the website, StatProb is:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;StatProb: The Encyclopedia Sponsored&#xA;  by Statistics and Probability&#xA;  Societies&lt;/strong&gt; combines the advantages of&#xA;  traditional wikis (rapid and&#xA;  up-to-date publication, user-generated&#xA;  development, hyperlinking, and a saved&#xA;  history) with traditional publishing&#xA;  (quality assurance, review, credit to&#xA;  authors, and a structured information&#xA;  display). All contributions have been&#xA;  approved by an editorial board&#xA;  determined by leading statistical&#xA;  societies; the editorial board members&#xA;  are listed on the About page.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I am not a statistician, but I use statistics, and this site appears to offer an opportunity me to publish material that while potentially useful to others, would likely go unpublished unless I were to include it as an appendix or post it on a website. The option is appealing because the review process would boost my own confidence in the methods that I use and give it some credibility in the public sphere.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Despite the support of major statistics and probability societies, the site has not taken off. Indeed, &lt;a href=&quot;http://xianblog.wordpress.com/tag/jsm-2010/&quot; rel=&quot;nofollow&quot;&gt;one blogger asked 'R.I.P. StatProb?'&lt;/a&gt; and the frequency of contributions has been declining with time.  &lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Question:&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Is it worth the effort to publish through StatProb.com? &lt;/p&gt;&#xA;" OwnerUserId="2750" LastEditorUserId="2750" LastEditorDisplayName="" LastEditDate="2011-04-26T21:02:11.597" LastActivityDate="2011-05-23T02:51:24.220" Title="Is it worthwhile to publish at the refereed wiki StatProb.com?" Tags="&lt;probability&gt;&lt;books&gt;&lt;methodology&gt;" AnswerCount="1" CommentCount="4" />
  <row Id="10004" PostTypeId="2" ParentId="10001" CreationDate="2011-04-26T20:12:31.790" Score="2" ViewCount="" Body="&lt;p&gt;The &lt;a href=&quot;http://www-stat.stanford.edu/~tibs/ElemStatLearn/&quot; rel=&quot;nofollow&quot;&gt;ELSII&lt;/a&gt; used &lt;a href=&quot;http://cran.r-project.org/web/packages/randomForest/index.html&quot; rel=&quot;nofollow&quot;&gt;randomForest&lt;/a&gt; (see e.g., footnote 3 p.591), which is an R implementation of the Breiman and Cutler's &lt;a href=&quot;http://stat-www.berkeley.edu/users/breiman/RandomForests&quot; rel=&quot;nofollow&quot;&gt;Fortran code&lt;/a&gt; from Salford. Andy Liaw's code is in C.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There's another implementation of RFs proposed in the &lt;a href=&quot;http://cran.r-project.org/web/packages/party/index.html&quot; rel=&quot;nofollow&quot;&gt;party&lt;/a&gt; package (in C), which relies on R/Lapack, which has some dependencies on BLAS (see&lt;code&gt;/include/R_ext/Lapack.h&lt;/code&gt; in your base R directory). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as bagging is concerned, it should not be too hard to parallelize it, but I'll let more specialized users answer on this aspect.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-04-26T20:12:31.790" />
  <row Id="10005" PostTypeId="1" CreationDate="2011-04-26T20:15:47.633" Score="0" ViewCount="163" Body="&lt;p&gt;I have a table I want to convert into a graph (bar-graph or line-graph)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first column has fixed values. Twenty different values are simulated for these fixed values and kept in the next columns. I want to plot a graph of the fixed column against all the different twenty simulated columns. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I go about it? &lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Edit&lt;/h2&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;table name:bygrace&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;V1    V2   V3   V4   V5                 &#xA;100    16   11   -6    1                               &#xA;120   -17  -12    7   -2                               &#xA;140    18   13   -8    3                              &#xA;150   -19  -14    9   -4                              &#xA;210    20   15  -10   -5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Actually, my table looks like the one above.&#xA;The first column V1 represents Premiums charged by an insurance company,say last year (for 5 policyholders/policies)&#xA;After critical analysis of the portfolio, the company decides to increase or decrease the premium amount for some of the customers.&#xA;However, the company does not know exactly by how much it should increase of decrease the premiums for fear that it would lose customers.&#xA;For this reason, four different scenarios/simulations are considered which are represented by the next four columns(V2 through V5) respectively.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now the task is to plot the premium amounts in V1 against these four different scenarios(bar-graph or line-graph; I think bar will be better).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And my question is can this be done on one graph/at once? If yes, how shoul I go about it?&#xA;Or do I have to plot the premium against each column separately?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In fact, I have spent the whole of yesterday and today on this but I have not been able to get the desired result. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Someone gave me an answer to try.And I am going to do that because it has given me an idea and I am very thankful to the one!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks to everyone for their help&lt;/p&gt;&#xA;" OwnerUserId="4340" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-04-27T22:10:36.207" LastActivityDate="2011-04-28T00:23:08.120" Title="How to convert a table into a graph in R" Tags="&lt;r&gt;&lt;data-visualization&gt;" AnswerCount="2" CommentCount="4" FavoriteCount="1" />
  <row Id="10006" PostTypeId="1" CreationDate="2011-04-26T20:30:43.627" Score="3" ViewCount="36" Body="&lt;p&gt;Suppose $X\sim InvWishart(\nu, \Sigma_0)$. I'm interested in the marginal distribution of the diagonal elements $diag(X) = (x_{11}, \dots, x_{pp})$. There are a few simple results on the distribution of submatrices of $X$ (at least some listed at Wikipedia). From this I can figure that the marginal distribution of any single element on the diagonal is inverse Gamma. But I've been unable to deduce the joint distribution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I thought maybe it could be derived by composition, like &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$p(x_{11} | x_{ii}, i\gt 1)p(x_{22}|x_{ii}, i&amp;gt;2)\dots p(x_{(p-1)(p-1)}|x_{pp})p(x_{pp})$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;but I never got anywhere with it and further suspect that I'm missing something simple; it seems like this &quot;ought&quot; to be known but I haven't been able to find/show it. &lt;/p&gt;&#xA;" OwnerUserId="26" LastActivityDate="2011-04-26T20:30:43.627" Title="Marginal distribution of the diagonal of an inverse Wishart distributed matrix" Tags="&lt;distributions&gt;&lt;probability&gt;&lt;density-function&gt;" />
  <row Id="10007" PostTypeId="2" ParentId="6776" CreationDate="2011-04-26T20:47:11.920" Score="0" ViewCount="" Body="&lt;p&gt;It's a mixture model set up you've got.  So to start, put the mixture identifying variable in - you don't have it yet.  It's an indicator variable saying whether a case comes from one regression (say Z=0) or the other (say Z=1).  Probably it will enter the full model in the form of an interaction with a slope and/or intercept to allow these to change depending on which regression generates the point (although other more complex arrangements are possible).  Formulate that model carefully to ensure the mixture dependencies are what you want - there are a lot of possibilities.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, if Z was &lt;em&gt;observed&lt;/em&gt; you'd know how to fit the complete model and get betas from it because there would be nothing unobserved on the right hand side of it.  But assuming you see only the data and the covariates, you don't observe it.  However, you &lt;em&gt;have&lt;/em&gt; assumed a complete model for how data is generated for each value of Z.  So (E-step) use that to get a posterior distribution over the possible values of Z for each data point using the model with its parameters as they stand and some prior assumption about the distribution of Z (or you could estimate that too).  Recall that the posterior probability of Z=1 just is the expectation of Z.  Now (M-step) use that expected Z as if it was a real observation of Z to refit the whole model.  The complete data likelihood will, in normal circumstances, not go down.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternate this process until the likelihood of the data under the model stops rising, retrieve the final set of betas, hope you're not in a local minimum, and declare that you've estimated them.&lt;/p&gt;&#xA;" OwnerUserId="1739" LastActivityDate="2011-04-26T20:47:11.920" />
  <row Id="10008" PostTypeId="1" CreationDate="2011-04-26T20:48:43.323" Score="1" ViewCount="80" Body="&lt;p&gt;Hello,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a scenario where I have a User which likes 10 different sports and there is another user which likes 20 different sports. I need to find the correlation between them. What kind of correlations can be used in such a scenario. Any kind of guide would be helpful. I tried with Pearson correlation but was not helpful. I would like to program using MATLAB. Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="4341" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-27T13:34:20.870" LastActivityDate="2011-04-27T13:34:20.870" Title="Need to find correlation between two entities" Tags="&lt;correlation&gt;&lt;matlab&gt;" AnswerCount="2" CommentCount="4" FavoriteCount="1" />
  <row Id="10009" PostTypeId="2" ParentId="10005" CreationDate="2011-04-26T21:56:07.137" Score="1" ViewCount="" Body="&lt;p&gt;Try this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#Generate 100 x values.  Then generate 20 random walk y values for each x value&#xA;x &amp;lt;- seq(1, 100, 1)  &#xA;y &amp;lt;- matrix(20*100, nrow=100, ncol=20)  &#xA;for (i in 1:20) {  &#xA;  y[, i] &amp;lt;- cumsum(rnorm(100))  &#xA;}  &#xA;&#xA;#Build the table  &#xA;df &amp;lt;- data.frame(x=x, y=y)  &#xA;head(df)  &#xA;&#xA;#Plot the table  &#xA;matplot(df[, 1], df[, 2:21],  type=&quot;l&quot;, main=&quot;Twenty Random Walks&quot;, xlab=&quot;x&quot;, ylab=&quot;y&quot;)    &#xA;grid()  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/GnCMV.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="2775" LastEditorUserId="2775" LastEditorDisplayName="" LastEditDate="2011-04-26T22:01:25.727" LastActivityDate="2011-04-26T22:01:25.727" />
  <row Id="10010" PostTypeId="1" CreationDate="2011-04-26T22:27:58.763" Score="1" ViewCount="48" Body="&lt;p&gt;I am currently trying to do the following in R:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have thousands of &lt;strong&gt;measured spectra&lt;/strong&gt; (x,y; see below). Each spectra has one or two peaks. Also I have sets of &lt;strong&gt;&quot;training&quot; spectra&lt;/strong&gt; obtained in more controlled conditions and I would like to know which of my training spectra has the closest match to the measured spectra!? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was thinking that some sort of pattern recognition would be useful but I know too little to make an informed choice as this is a bit outside of my usual work-area&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What is the most promising way/function in R to do this kind of pattern recognition I want?&lt;/li&gt;&#xA;&lt;li&gt;In case pattern recognition (like PCA) is not the most promising way, what other options are there?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I am looking for sample bits of code or literature dealing with this kind of data analysis. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/5mpqU.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt;&#xA;The peak position will most probably always be the same, however the laser used to record the spectra is temperature controlled and slight variations are possible. The intensity will change depending on experimental conditions.&#xA;The two peaks sould be treated as independet peaks.&lt;/p&gt;&#xA;" OwnerUserId="4342" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-27T22:58:18.527" LastActivityDate="2011-04-27T22:58:18.527" Title="Spectral data pattern recognition" Tags="&lt;pattern-recognition&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="10011" PostTypeId="1" CreationDate="2011-04-26T22:36:32.657" Score="5" ViewCount="99" Body="&lt;p&gt;I have a set of samples in which I assume there are 2 definite subsets in it. I plotted their values in a histogram and found that there are two distinct distribution as shown in the figure below.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is how do I differentiate two groups. i.e how do I choose a value that differentiate two subsets? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/wbIvt.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="2725" LastEditorUserId="2725" LastEditorDisplayName="" LastEditDate="2011-04-27T19:28:55.653" LastActivityDate="2011-04-27T19:28:55.653" Title="How to differentiate two subgroups from a histogram?" Tags="&lt;normal-distribution&gt;&lt;mixture&gt;" AnswerCount="2" CommentCount="5" FavoriteCount="1" />
  <row Id="10012" PostTypeId="2" ParentId="10008" CreationDate="2011-04-27T00:13:44.980" Score="1" ViewCount="" Body="&lt;p&gt;This is like &lt;a href=&quot;http://en.wikipedia.org/wiki/Inter-rater_reliability&quot; rel=&quot;nofollow&quot;&gt;inter-rater reliability&lt;/a&gt;. The code &lt;a href=&quot;http://www.mathworks.com/matlabcentral/fileexchange/15365&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; should do it.&lt;/p&gt;&#xA;" OwnerUserId="3874" LastActivityDate="2011-04-27T00:13:44.980" CommentCount="2" />
  <row Id="10013" PostTypeId="1" CreationDate="2011-04-27T00:16:30.820" Score="5" ViewCount="38" Body="&lt;p&gt;I found the following posts interesting and I was wondering if any of you guys know of good academic papers that describe methods/relationships of exogenous variables in VECM models. If so could you kindly point them out to me as I am very interested in learning. Thank you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://stats.stackexchange.com/questions/4030/finding-coefficients-for-vecm-exogenous-variables&quot;&gt;Finding coefficients for VECM + exogenous variables&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://stats.stackexchange.com/questions/6487/lagged-exogenous-variables-in-vecm-with-r&quot;&gt;Lagged Exogenous Variables in VECM with R&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="4338" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-04-27T07:06:30.040" LastActivityDate="2011-04-27T07:06:30.040" Title="Exogenous variables in VECM" Tags="&lt;time-series&gt;" CommentCount="1" />
  <row Id="10014" PostTypeId="2" ParentId="9930" CreationDate="2011-04-27T01:35:34.270" Score="0" ViewCount="" Body="&lt;p&gt;I can't top what probabilityislogic has done, but for a light treatment that gives illustrative examples and requires no math, see &lt;a href=&quot;http://www.integrativestatistics.com/partial.htm&quot; rel=&quot;nofollow&quot;&gt;http://www.integrativestatistics.com/partial.htm&lt;/a&gt;  &lt;/p&gt;&#xA;" OwnerUserId="2669" LastActivityDate="2011-04-27T01:35:34.270" />
  <row Id="10015" PostTypeId="2" ParentId="8642" CreationDate="2011-04-27T01:51:25.523" Score="1" ViewCount="" Body="&lt;p&gt;Thanks for the question, I have two notes and one suggestion.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first note is that testing theory is typically done by setting an acceptable level where you would reject a true hypothesis (Type I error), then minimize the risk of accepting a false hypothesis (Type II error).  There are two reasons for this, first is that all your tests use this assumption, and second of all in almost all cases you can't minimize both errors simultaneously.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My second note is that the Wilcoxon Test hypothesis is actually H_0: F_0 = F_1, H_1: F_0 != F_1, where F_i are CDFs, the relationship of this test to the mean is a property of the class of CDFs you are considering and the conditions you are considering them under.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Under the data discussed I think bootstrapping would probably be appropriate if you think the sample is representative of the population of interest.  Other possible choices include deriving an empirical likelihood ratio test, or resampling t-tests and checking robustness.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good luck and let us know how the analysis turns out :)&lt;/p&gt;&#xA;" OwnerUserId="2339" LastActivityDate="2011-04-27T01:51:25.523" />
  <row Id="10016" PostTypeId="2" ParentId="10011" CreationDate="2011-04-27T02:14:35.040" Score="0" ViewCount="" Body="&lt;p&gt;If you are willing to assume the populations have the same variance you could use essentially LDA without the normality assumption (a.k.a. Fisher's Method or Fisher's Discriminant Function).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Without this assumption you could try an EM algorithm which is indirectly what Matt Suggested since this would be a mixture model approach. &lt;/p&gt;&#xA;" OwnerUserId="2339" LastActivityDate="2011-04-27T02:14:35.040" />
  <row Id="10017" PostTypeId="1" CreationDate="2011-04-27T02:34:11.373" Score="3" ViewCount="134" Body="&lt;p&gt;I am trying to understand standard error &quot;clustering&quot; and how to execute in R (it is trivial in Stata). In R I have been unsuccessful using either &lt;code&gt;plm&lt;/code&gt; or writing my own function. I'll use the &lt;code&gt;diamonds&lt;/code&gt; data from the &lt;code&gt;ggplot2&lt;/code&gt; package.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can do fixed effects with either dummy variables&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; library(plyr)&#xA;&amp;gt; library(ggplot2)&#xA;&amp;gt; library(lmtest)&#xA;&amp;gt; library(sandwich)&#xA;&amp;gt; # with dummies to create fixed effects&#xA;&amp;gt; fe.lsdv &amp;lt;- lm(price ~ carat + factor(cut) + 0, data = diamonds)&#xA;&amp;gt; ct.lsdv &amp;lt;- coeftest(fe.lsdv, vcov. = vcovHC)&#xA;&amp;gt; ct.lsdv&#xA;&#xA;t test of coefficients:&#xA;&#xA;                      Estimate Std. Error  t value  Pr(&amp;gt;|t|)    &#xA;carat                 7871.082     24.892  316.207 &amp;lt; 2.2e-16 ***&#xA;factor(cut)Fair      -3875.470     51.190  -75.707 &amp;lt; 2.2e-16 ***&#xA;factor(cut)Good      -2755.138     26.570 -103.692 &amp;lt; 2.2e-16 ***&#xA;factor(cut)Very Good -2365.334     20.548 -115.111 &amp;lt; 2.2e-16 ***&#xA;factor(cut)Premium   -2436.393     21.172 -115.075 &amp;lt; 2.2e-16 ***&#xA;factor(cut)Ideal     -2074.546     16.092 -128.920 &amp;lt; 2.2e-16 ***&#xA;---&#xA;Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;or by de-meaning both left- and right-hand sides (no time invariant regressors here) and correcting degrees of freedom.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; # by demeaning with degrees of freedom correction&#xA;&amp;gt; diamonds &amp;lt;- ddply(diamonds, .(cut), transform, price.dm = price - mean(price), carat.dm = carat  .... [TRUNCATED] &#xA;&amp;gt; fe.dm &amp;lt;- lm(price.dm ~ carat.dm + 0, data = diamonds)&#xA;&amp;gt; ct.dm &amp;lt;- coeftest(fe.dm, vcov. = vcovHC, df = nrow(diamonds) - 1 - 5)&#xA;&amp;gt; ct.dm&#xA;&#xA;t test of coefficients:&#xA;&#xA;         Estimate Std. Error t value  Pr(&amp;gt;|t|)    &#xA;carat.dm 7871.082     24.888  316.26 &amp;lt; 2.2e-16 ***&#xA;---&#xA;Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I can't replicate these results with &lt;code&gt;plm&lt;/code&gt;, because I don't have a &quot;time&quot; index (i.e., this isn't really a panel, just clusters that could have a common bias in their error terms).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; plm.temp &amp;lt;- plm(price ~ carat, data = diamonds, index = &quot;cut&quot;)&#xA;duplicate couples (time-id)&#xA;Error in pdim.default(index[[1]], index[[2]]) : &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I also tried to code my own covariance matrix with clustered standard error using Stata's explanation of their &lt;code&gt;cluster&lt;/code&gt; option (&lt;a href=&quot;http://www.stata.com/support/faqs/stat/cluster.html&quot; rel=&quot;nofollow&quot;&gt;explained here&lt;/a&gt;), which is to solve $$\hat V_{cluster} = (X&amp;#39;X)^{-1} \left( \sum_{j=1}^{n_c} u_j&amp;#39;u_j \right) (X&amp;#39;X)^{-1}$$ where $u_j = \sum_{cluster~j} e_i * x_i$, $n_c$ si the number of clusters, $e_i$ is the residual for the $i^{th}$ observation and $x_i$ is the row vector of predictors, including the constant (this also appears as equation (7.22) in Wooldridge's &lt;em&gt;Cross Section and Panel Data&lt;/em&gt;). But the following code gives very large covariance matrices. Are these very large values given the small number of clusters I have? Given that I can't get &lt;code&gt;plm&lt;/code&gt; to do clusters on one factor, I'm not sure how to benchmark my code.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; # with cluster robust se&#xA;&amp;gt; lm.temp &amp;lt;- lm(price ~ carat + factor(cut) + 0, data = diamonds)&#xA;&amp;gt; &#xA;&amp;gt; # using the model that Stata uses&#xA;&amp;gt; stata.clustering &amp;lt;- function(x, clu, res) {&#xA;+     x &amp;lt;- as.matrix(x)&#xA;+     clu &amp;lt;- as.vector(clu)&#xA;+     res &amp;lt;- as.vector(res)&#xA;+     fac &amp;lt;- unique(clu)&#xA;+     num.fac &amp;lt;- length(fac)&#xA;+     num.reg &amp;lt;- ncol(x)&#xA;+     u &amp;lt;- matrix(NA, nrow = num.fac, ncol = num.reg)&#xA;+     meat &amp;lt;- matrix(NA, nrow = num.reg, ncol = num.reg)&#xA;+     &#xA;+     # outer terms (X'X)^-1&#xA;+     outer &amp;lt;- solve(t(x) %*% x)&#xA;+ &#xA;+     # inner term sum_j u_j'u_j where u_j = sum_i e_i * x_i&#xA;+     for (i in seq(num.fac)) {&#xA;+         index.loop &amp;lt;- clu == fac[i]&#xA;+         res.loop &amp;lt;- res[index.loop]&#xA;+         x.loop &amp;lt;- x[clu == fac[i], ]&#xA;+         u[i, ] &amp;lt;- as.vector(colSums(res.loop * x.loop))&#xA;+     }&#xA;+     inner &amp;lt;- t(u) %*% u&#xA;+ &#xA;+     # &#xA;+     V &amp;lt;- outer %*% inner %*% outer&#xA;+     return(V)&#xA;+ }&#xA;&amp;gt; x.temp &amp;lt;- data.frame(const = 1, diamonds[, &quot;carat&quot;])&#xA;&amp;gt; summary(lm.temp)&#xA;&#xA;Call:&#xA;lm(formula = price ~ carat + factor(cut) + 0, data = diamonds)&#xA;&#xA;Residuals:&#xA;     Min       1Q   Median       3Q      Max &#xA;-17540.7   -791.6    -37.6    522.1  12721.4 &#xA;&#xA;Coefficients:&#xA;                     Estimate Std. Error t value Pr(&amp;gt;|t|)    &#xA;carat                 7871.08      13.98   563.0   &amp;lt;2e-16 ***&#xA;factor(cut)Fair      -3875.47      40.41   -95.9   &amp;lt;2e-16 ***&#xA;factor(cut)Good      -2755.14      24.63  -111.9   &amp;lt;2e-16 ***&#xA;factor(cut)Very Good -2365.33      17.78  -133.0   &amp;lt;2e-16 ***&#xA;factor(cut)Premium   -2436.39      17.92  -136.0   &amp;lt;2e-16 ***&#xA;factor(cut)Ideal     -2074.55      14.23  -145.8   &amp;lt;2e-16 ***&#xA;---&#xA;Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 &#xA;&#xA;Residual standard error: 1511 on 53934 degrees of freedom&#xA;Multiple R-squared: 0.9272, Adjusted R-squared: 0.9272 &#xA;F-statistic: 1.145e+05 on 6 and 53934 DF,  p-value: &amp;lt; 2.2e-16 &#xA;&#xA;&amp;gt; stata.clustering(x = x.temp, clu = diamonds$cut, res = lm.temp$residuals)&#xA;                        const diamonds....carat..&#xA;const                11352.64           -14227.44&#xA;diamonds....carat.. -14227.44            17830.22&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Can this be done in R? It is a fairly common technique in econometrics (there's a brief tutorial in &lt;a href=&quot;http://sekhon.berkeley.edu/causalinf/sp2010/section/week7.pdf&quot; rel=&quot;nofollow&quot;&gt;this lecture&lt;/a&gt;), but I can't figure it out in R. Thanks!&lt;/p&gt;&#xA;" OwnerUserId="1445" LastEditorUserId="1445" LastEditorDisplayName="" LastEditDate="2011-04-28T03:32:58.020" LastActivityDate="2011-05-28T15:51:10.227" Title="Standard error clustering in R (either manually or in plm)" Tags="&lt;r&gt;&lt;panel-data&gt;&lt;fixed-effects-model&gt;" AnswerCount="1" CommentCount="6" FavoriteCount="1" />
  <row Id="10019" PostTypeId="2" ParentId="9988" CreationDate="2011-04-27T05:01:49.230" Score="3" ViewCount="" Body="&lt;h3&gt;General interpration of question:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;The question is a bit confusing, but I interpret it as follows.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Factor Analysis&lt;/strong&gt;: When a survey has multiple items and some are positively worded (e.g., &quot;I am the life of the party&quot;) and others are negatively worded (e.g., &quot;I avoid social interaction&quot;), factor analysis often assigns such items to the same factor.&#xA;For example, extraversion items load positively on one factor and introversion items load negatively on the same factor.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Cluster Analysis&lt;/strong&gt;: It is possible to cluster by variables.&#xA;&lt;ul&gt;&#xA;&lt;li&gt;In &lt;strong&gt;R&lt;/strong&gt; you can use &lt;code&gt;dist&lt;/code&gt; to generate a distance matrix and then send it to &lt;code&gt;hclust&lt;/code&gt; to perform a hierarchical cluster analysis.&lt;/li&gt;&#xA;&lt;li&gt;In &lt;strong&gt;SPSS&lt;/strong&gt; the hierarchical cluster analysis procedure allows you to cluster by variables. The procedure uses the &lt;code&gt;proximities&lt;/code&gt; command to generate the distance matrix.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;Variables and clusters:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;When running a cluster analysis on variables, you need to think about what formula you will use to generate your distance matrix.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Common ways of calculating distances between variables (i.e., the items in your survey) include the &lt;code&gt;squared euclidean distance&lt;/code&gt; or &lt;code&gt;one minus the correlation&lt;/code&gt;.&#xA;In both the cases, the distance between negative and positively worded items on the same dimension will appear distant. Thus, when you run a cluster analysis, such items will not cluster together.&lt;/li&gt;&#xA;&lt;li&gt;However, if you want such items to cluster together, you can choose a measure of distance between items that sees negatively correlated items as similar. A few ideas include:&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Reverse the negatively worded items before entering them into the cluster analysis.&lt;/li&gt;&#xA;&lt;li&gt;Use &lt;code&gt;1 minus absolute correlation&lt;/code&gt; as the measure of distance&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In general, cluster analysis is an algorithmic and atheoretical way of examining groupings in your variables. You have a lot of freedom in how you define distances and how you aggregate based on those distances. The important consideration is that you align your conceptual definition of distance with your operational algorithm for measuring distance.&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2011-04-27T05:01:49.230" />
  <row Id="10020" PostTypeId="1" CreationDate="2011-04-27T05:35:46.223" Score="7" ViewCount="160" Body="&lt;p&gt;I was advising a research student with a particular problem, and I was keen to get the input of others on this site.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Context:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;The researcher had three types of  predictor variables. Each type contained a different number of predictor variables. Each predictor was a continuous variable:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Social: S1, S2, S3, S4 (i.e., four predictors)&lt;/li&gt;&#xA;&lt;li&gt;Cognitive: C1, C2 (i.e., two predictors)&lt;/li&gt;&#xA;&lt;li&gt;Behavioural: B1, B2, B3 (i.e., three predictors)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The outcome variable was also continuous.&#xA;The sample included around 60 participants.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The researcher wanted to comment about which type of predictors were more important in explaining the outcome variable. This was related to broader theoretical concerns about the relative importance of these types of predictors.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Questions&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What is a good way to assess the relative importance of one set of predictors relative to another set?&lt;/li&gt;&#xA;&lt;li&gt;What is a good strategy for dealing with the fact that there are different numbers of predictors in each set?&lt;/li&gt;&#xA;&lt;li&gt;What caveats in interpretation might you suggest?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Any references to examples or discussion of techniques would also be most welcome.&lt;/p&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-27T08:57:44.160" LastActivityDate="2011-04-30T03:11:51.177" Title="Comparing importance of different sets of predictors" Tags="&lt;regression&gt;&lt;predictor&gt;&lt;importance&gt;" AnswerCount="3" FavoriteCount="2" />
  <row Id="10021" PostTypeId="1" CreationDate="2011-04-27T10:07:39.523" Score="0" ViewCount="55" Body="&lt;p&gt;Hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given a Gaussian distribution $N(\mu_1,\sigma_1^2)$, i would like to choose another mean $\mu_2$ which is $2\sigma_1$ away from $\mu_1$. &#xA;In this case our new mean $\mu_2=\mu_1\pm 2\sigma_1$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do we calculate the new mean($\mu_2$) in multivariate case? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I mean to say, when your multivariate Gaussian distribution is $N(\mu_1,\Sigma_1)$ and my $\Sigma_1$ is symmetric positive definite matrix. i.e &#xA;$\left[ \begin{array}{cc} \sigma_x^2 &amp;amp; \sigma_{xy} \\ \sigma_{yx} &amp;amp; \sigma_y^2 \end{array} \right]$.&lt;/p&gt;&#xA;" OwnerUserId="4290" LastActivityDate="2011-05-27T11:50:44.173" Title="New mean calculation in Multivariate Gaussian" Tags="&lt;multivariate-analysis&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="2" />
  <row Id="10022" PostTypeId="2" ParentId="10020" CreationDate="2011-04-27T10:48:27.033" Score="4" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Importance&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First thing to do is operationalise 'importance of predictors'.  I shall assume that it means something like 'sensitivity of mean outcome to changes in predictor values'.  Since your predictors are grouped then sensitivity of the mean outcome to groups of predictors is more interesting than a variable by variable analysis.  I leave it open whether sensitivity is understood causally.  That issue is picked up later.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Three version of importance&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Lots of variance explained&lt;/em&gt;: I'm guessing that psychologists' first port of call is probably a variance decomposition leading to a measure of how much outcome variance is explained by the variance-covarance structure in each group of predictors.  Not being an experimentalist I can't suggest much here, except to note that the whole 'variance explained' concept is a bit ungrounded for my taste, even without the 'which sum of which squares' issue.  Others are welcome to disagree and develop it further.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Large standardized coefficients&lt;/em&gt;: SPSS offers the (misnamed) beta to measure impact in a way that is comparable across variable.  There are several reasons not to use this, discussed in Fox's regression textbook, &lt;a href=&quot;http://www.jerrydallal.com/LHSP/importnt.htm&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;, and elsewhere.  All apply here.  It also ignores group structure. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, I imagine that one &lt;em&gt;could&lt;/em&gt; standardise predictors in groups and use covariance information to judge the effect of a one standard deviation movement in all of them.  Personally the motto: &quot;if a things not worth doing, it's not worth doing well&quot; damps my interest in doing so.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Large marginal effects&lt;/em&gt;: The other approach is to stay on the scale of the measurements and calculate marginal effects between carefully chosen sample points.&#xA;Because you are interested in groups it is useful to choose points to vary groups of variables rather than single ones, e.g. manipulating both cognitive variables at once.  (Lots of opportunity for cool plots here).  Basic paper &lt;a href=&quot;http://gking.harvard.edu/gking/files/making.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.  The &lt;code&gt;effects&lt;/code&gt; package in R will do this nicely.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are two caveats here: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;If you do that you will want to watch out that you are not choosing two cognitive variables that while individually plausible, e.g. medians, are jointly far from any subject observation.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Some variables are not even theoretically manipulable, so the interpretation of marginal effects as causal is more delicate, though still useful.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Different numbers of predictors&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Issues arise due to the grouped variables covariance structure, which we normally try not to worry about but for this task should.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In particular when calculating marginal effects (or standardized coefficients for that matter) on groups rather than single variables the curse of dimensionality will for larger groups make it easier for comparisons to stray into regions where there are no cases.  More predictors in a group lead to a more sparsely populated space, so any importance measure will depend more on model assumptions and less on observations (but will not tell you that...)  But these are the same issues as in the model fitting phase really.  Certainly the same ones as would arise in a model-based causal impact assessment. &lt;/p&gt;&#xA;" OwnerUserId="1739" LastActivityDate="2011-04-27T10:48:27.033" />
  <row Id="10023" PostTypeId="2" ParentId="10002" CreationDate="2011-04-27T11:12:22.073" Score="1" ViewCount="" Body="&lt;p&gt;I am not an expert in mixed effect modelling, but the question is much easier to answer if it is rephrased in hierarchical regression modelling context. So our observations have two indexes $P_{ij}$ and $F_{ij}$ with index $i$ representing class and $j$ members of the class. The hierarchical models let us fit linear regression, where coefficients vary across classes:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Y_{ij}=\beta_{0i}+\beta_{1i}F_{ij}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is our first level regression. The second level regression is done on the first regression coefficients:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align*}&#xA;\beta_{0i}&amp;amp;=\gamma_{00}+u_{0i}\\&#xA;\beta_{1i}&amp;amp;=\gamma_{01}+u_{1i}&#xA;\end{align*}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;when we substitute this in first level regression we get&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align*}&#xA;Y_{ij}&amp;amp;=(\gamma_0+u_{0i})+(\gamma_{01}+u_{1i})F_{ij}\\&#xA;&amp;amp;=\gamma_0+u_{0i}+u_{1i}F_{ij}+\gamma_{01}F_{ij}&#xA;\end{align*}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here $\gamma$ are fixed effects and $u$ are random effects. Mixed model estimates $\gamma$ and variances of $u$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The model I've written down corresponds to &lt;code&gt;lmer&lt;/code&gt; syntax&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;P ~ (1+F|R) + F&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now if we put $\beta_{1i}=\gamma_{01}$ without the random term we get &lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align*}&#xA;Y_{ij}=\gamma_0+u_{0i}+\gamma_{01}F_{ij}&#xA;\end{align*}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;which corresponds to &lt;code&gt;lmer&lt;/code&gt; syntax&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;P ~ (1|R) + F&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So the question now becomes when can we exclude error term from the second level regression? The canonical answer is that when we are sure that the regressors (here we do not have any, but we can include them, they naturally are constant within classes) in the second level regression fully explain the variance of coefficients across classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So in this particular case if coefficient of $F_{ij}$ does not vary, or alternatively the variance of $u_{1i}$ is very small we should entertain idea that we are probably better of with the first model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note&lt;/strong&gt;. I've only gave algebraic explanation, but I think having it in mind it is much easier to think of particular applied example.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-04-27T11:12:22.073" />
  <row Id="10024" PostTypeId="1" AcceptedAnswerId="10041" CreationDate="2011-04-27T11:13:57.890" Score="5" ViewCount="65" Body="&lt;p&gt;According to &lt;a href=&quot;http://en.wikipedia.org/wiki/Cluster_analysis#Partitional_clustering&quot; rel=&quot;nofollow&quot;&gt;wiki&lt;/a&gt; the most widely used convergence criterion is &quot;assigment hasn't changed&quot;. I was wondering whether cycling can occur if we use such convergence criterion? I'd be pleased if anyone pointed a reference to an article that gives an example of cycling or proves that this is impossible.&lt;/p&gt;&#xA;" OwnerUserId="1643" LastActivityDate="2011-04-27T16:44:37.730" Title="Cycling in k-means algorithm" Tags="&lt;clustering&gt;&lt;algorithms&gt;&lt;k-means&gt;" AnswerCount="2" CommentCount="0" FavoriteCount="1" />
  <row Id="10025" PostTypeId="2" ParentId="10021" CreationDate="2011-04-27T11:35:54.217" Score="0" ViewCount="" Body="&lt;p&gt;In the bivariate case you can substitute the two points ($\mu_2=\mu_1\pm 2\sigma_1$) with an isodensity ellipse: &lt;a href=&quot;http://www.stat.psu.edu/online/courses/stat505/05_multnorm/06_multnorm_revist.html&quot; rel=&quot;nofollow&quot;&gt;http://www.stat.psu.edu/online/courses/stat505/05_multnorm/06_multnorm_revist.html&lt;/a&gt; .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your $2\sigma_1$ criterion seems a bit arbitrary, but it includes 95.44997% of the random variable. So you may want to use the 95.44997% isodensity ellipse in the bivariate case, too. The principle axes of this rotated ellipse are the eigenvectors of the covariance matrix, see &lt;a href=&quot;http://web.as.uky.edu/statistics/users/viele/sta601s08/multinorm.pdf&quot; rel=&quot;nofollow&quot;&gt;http://web.as.uky.edu/statistics/users/viele/sta601s08/multinorm.pdf&lt;/a&gt; . You can generalise this to more than 2 dimensions.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastActivityDate="2011-04-27T11:35:54.217" />
  <row Id="10026" PostTypeId="2" ParentId="10020" CreationDate="2011-04-27T11:43:16.597" Score="5" ViewCount="" Body="&lt;p&gt;Suppose that the first set of predictors requires a degrees of freedom (a $\geq$ 4 allowing for nonlinear terms), the second set requires b, and the third requires c (c $\geq$ 3) allowing for nonlinear terms.  Compute the likelihood ratio $\chi^2$ test for the combined partial effects of each set, yielding $L_{1}, L_{2}, L_{3}$.  The expected value of a $\chi^2$ random variable with d degrees of freedom is d, so subtract d to level the playing field.  I.e., compute $L_{1}-a, L_{2}-b, L_{3}-c$.  If using F-tests, multiple F by its numerator d.f. to get the $\chi^2$ scale.&lt;/p&gt;&#xA;" OwnerUserId="4253" LastActivityDate="2011-04-27T11:43:16.597" CommentCount="4" />
  <row Id="10027" PostTypeId="2" ParentId="10011" CreationDate="2011-04-27T11:59:18.457" Score="2" ViewCount="" Body="&lt;p&gt;I assume you are talking about Neonatal Behavioral Assessment Scale values in Hereditary Renal Adysplasia. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I often see in medical research that physicians want to have cut-offs and simple threshold based interpretations of their research results, based merely on the distribution of the measurements. Practice and applications however usually need high positive predictive value or high negative predictive value, so the characteristics of the future population tested have to be considered. My point of view is even if now you just want to &quot;differentiate two groups&quot; you probably want to apply this somehow in the future and thus you probably want to find the optimal threshold, optimising costs, risks and benefits (survival, quality of life etc.) in a practical setting. So I suggest that you to think these over in your application.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastActivityDate="2011-04-27T11:59:18.457" CommentCount="1" />
  <row Id="10028" PostTypeId="2" ParentId="10001" CreationDate="2011-04-27T12:02:47.953" Score="2" ViewCount="" Body="&lt;p&gt;I must admit I'm only using R implementations, so I can at least say that party is few times slower than randomForest (yet offers greater control over the algorithm).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Non-R implementations I have heard of are &lt;a href=&quot;http://www.stat.berkeley.edu/~breiman/RandomForests/cc_home.htm&quot; rel=&quot;nofollow&quot;&gt;The Original One&lt;/a&gt;, &lt;a href=&quot;http://www.alglib.net/&quot; rel=&quot;nofollow&quot;&gt;ALGLIB&lt;/a&gt;, &lt;a href=&quot;http://waffles.sourceforge.net/&quot; rel=&quot;nofollow&quot;&gt;Waffles&lt;/a&gt;, &lt;a href=&quot;http://code.google.com/p/fast-random-forest/&quot; rel=&quot;nofollow&quot;&gt;so-called WEKA rf&lt;/a&gt;, &lt;a href=&quot;http://www.randomjungle.org/rjungle/&quot; rel=&quot;nofollow&quot;&gt;Random Jungle&lt;/a&gt;, &lt;a href=&quot;https://sites.google.com/site/rtranking/home&quot; rel=&quot;nofollow&quot;&gt;rt-rank&lt;/a&gt; and &lt;a href=&quot;http://www.irb.hr/en/research/projects/it/2004/2004-111/&quot; rel=&quot;nofollow&quot;&gt;PARF&lt;/a&gt;. &#xA;Only two of them are parallel, PARF via MPI and Weka via multithreading. And well, judging from the source none of them was really written with performance in mind, so I don't expect than any of them would be rocket faster than randomForest (or at least fast enough to make me switch from R). Also note they differ in the tree building algorithm they use, so they are not all compatible with the original.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Reassuming, the answer is no, there is no highly-optimized RF implementation, however this is not a burning issue as in case of heavy-BLAS-using algorithms like SVM; RF is already quite fast, scales well, and even though it is trivially parallelizable it is usually easier to parallelize the analysis on higher level (CV, feature selection, repetitions, datasets, etc.). &lt;/p&gt;&#xA;" OwnerUserId="88" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-01T17:49:37.287" LastActivityDate="2011-05-01T17:49:37.287" CommentCount="2" />
  <row Id="10029" PostTypeId="2" ParentId="10008" CreationDate="2011-04-27T12:11:40.880" Score="0" ViewCount="" Body="&lt;p&gt;If your goal is to measure similarity between individual users or groups of users you may use similarity or distance measures used in cluster analysis, biclustering or multidimensional scaling. In situations where you need such a measure the above techniques themselves may be useful, too.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastActivityDate="2011-04-27T12:11:40.880" />
  <row Id="10030" PostTypeId="1" CreationDate="2011-04-27T12:34:12.893" Score="2" ViewCount="149" Body="&lt;p&gt;On the Internet there is an example of k-s test being applied relative to distribution of number of bird varieties over different five hour periods.&#xA;The observed distribution was:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;a=c(0,1,1,9,4)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The expected distribution (if there is no difference between the five hours) could be:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;b=c(3,3,3,3,3)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;After I found the two cumulate distributions, I calculated manually, D = 0,4667 (a value that is similar with the internet). &#xA;But if I try to use R, I find a different value of D:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; a=c(0,1,1,9,4)&#xA;&#xA;&amp;gt; b=c(3,3,3,3,3)&#xA;&#xA;&amp;gt; ks.test(a,b)&#xA;&#xA;Two-sample Kolmogorov-Smirnov test&#xA;&#xA;data: a and b &#xA;&#xA;D = 0.6, p-value = 0.3291&#xA;&#xA;alternative hypothesis: two-sided .......&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What is leading to the difference between my manual calculation and result that R gives?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="4345" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-02T13:55:08.293" LastActivityDate="2011-05-02T13:55:08.293" Title="Difference between K-S manual test and K-S test with R?" Tags="&lt;r&gt;&lt;kolmogorov-smirnov&gt;" AnswerCount="2" CommentCount="6" />
  <row Id="10031" PostTypeId="1" AcceptedAnswerId="10034" CreationDate="2011-04-27T13:19:43.057" Score="1" ViewCount="77" Body="&lt;p&gt;I have a multinomial logistic regression model. &#xA;One of the output categories is not observed in the data set that I'm using. &lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Example:&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;4 different diagnoses (response variable) in the population, but in the sample, Type 3 never occurred&lt;/li&gt;&#xA;&lt;li&gt;5 hormone level measurements (predictors) &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;Question&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What books/papers discuss the mathematics of handling this situation in logistic regression?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="3280" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-28T15:18:55.860" LastActivityDate="2011-04-28T15:52:05.907" Title="How to handle categorical dependent variable using logistic regression when one of the categories never occurs in the sample" Tags="&lt;logistic-regression&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="10032" PostTypeId="2" ParentId="10030" CreationDate="2011-04-27T14:08:24.743" Score="4" ViewCount="" Body="&lt;p&gt;You are testing a different thing.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;While you think &lt;code&gt;c(0,1,1,9,4)&lt;/code&gt; means you are looking at 0 values of one, 1 value of two, 1 value of three, 9 values of four, and 4 values of five, R thinks you are looking at one value of 0, two values of 1, one value of 9, and one value of 4.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To get D = 0.4667..., try the rather verbose&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ks.test( c(2,3,4,4,4,4,4,4,4,4,4,5,5,5,5), &#xA;         c(1,1,1,2,2,2,3,3,3,4,4,4,5,5,5) ) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;giving &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    Two-sample Kolmogorov-Smirnov test &#xA;&#xA;D = 0.4667, p-value = 0.07626 &#xA;alternative hypothesis: two-sided  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-04-27T14:08:24.743" CommentCount="7" />
  <row Id="10033" PostTypeId="2" ParentId="10030" CreationDate="2011-04-27T14:09:42.373" Score="1" ViewCount="" Body="&lt;p&gt;First of all, the Kolmogorov-Smirnoff test is probably incorrect for your situation. I assume you want to test whether the distribution of the number bird sightings is uniform (constant accross time). A chi-square goodness-of-fit test would be the simplest solution for that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second, the R-produced value of 0.6 seems correct to me: at $x=2.5$, $F_a(x)=0.6$, while $F_b(x)=0$.&lt;/p&gt;&#xA;" OwnerUserId="279" LastActivityDate="2011-04-27T14:09:42.373" />
  <row Id="10034" PostTypeId="2" ParentId="10031" CreationDate="2011-04-27T14:44:33.060" Score="1" ViewCount="" Body="&lt;p&gt;If you are talking about a missing value in the response, there are many available texts on imputation with specific tailoring to regressions. I have not read it, but &lt;a href=&quot;http://books.google.com/books?hl=en&amp;amp;lr=&amp;amp;id=kfHrF-bVcvQC&amp;amp;oi=fnd&amp;amp;pg=PR7&amp;amp;dq=imputation%20multinomial%20logistic%20regression&amp;amp;ots=34FWS2bhA2&amp;amp;sig=3Fyi5kO0NVQ5Ipx5dCHt0zvm6NI#v=onepage&amp;amp;q&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;Frank E. Harrell's text&lt;/a&gt; is tailored to logistic regressions and has a chapter on missing values.&lt;/p&gt;&#xA;" OwnerUserId="3542" LastActivityDate="2011-04-27T14:44:33.060" />
  <row Id="10035" PostTypeId="2" ParentId="10010" CreationDate="2011-04-27T14:44:49.657" Score="1" ViewCount="" Body="&lt;p&gt;Perhaps &lt;a href=&quot;http://en.wikipedia.org/wiki/Dynamic_time_warping&quot; rel=&quot;nofollow&quot;&gt;dynamic time warping&lt;/a&gt; is appropriate? Maybe you can use it to measure the distance between your &quot;query&quot; signals and your training signals (dtw actually calculates alignments between two signals, from which you can derive the distance). You'd then pick the signal from your training data that's the nearest neighbor to your query.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a &lt;a href=&quot;http://cran.r-project.org/web/packages/dtw&quot; rel=&quot;nofollow&quot;&gt;dtw&lt;/a&gt; package on CRAN that lets you do this for R, which also has a very nice &lt;a href=&quot;http://cran.r-project.org/web/packages/dtw/vignettes/dtw.pdf&quot; rel=&quot;nofollow&quot;&gt;vignette&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The alignments are a bit intensive to compute using dtw, there is a &lt;a href=&quot;http://code.google.com/p/fastdtw/&quot; rel=&quot;nofollow&quot;&gt;Java implementation&lt;/a&gt; that calculates a fast approximation to the alignments if you want to pursue this further.&lt;/p&gt;&#xA;" OwnerUserId="227" LastActivityDate="2011-04-27T14:44:49.657" CommentCount="1" />
  <row Id="10036" PostTypeId="1" CreationDate="2011-04-27T14:46:36.657" Score="0" ViewCount="105" Body="&lt;p&gt;Dearest all,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using R to replicate a study and obtain mostly the same results the author reported. At one point, however, I calculate marginal effects that seem to be unrealistically small. I would greatly appreciate if you could have a look at my reasoning and the code below and see if I am mistaken at one point or another.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My sample contains 24535 observations, the dependent variable &quot;x028bin&quot; is a binary variable taking on the values 0 and 1, and there are furthermore 10 explaining variables. Nine of those independent variables have numeric levels, the independent variable &quot;f025grouped&quot; is a factor consisting of different religious denominations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to run a probit regression including dummies for religious denomination and then compute marginal effects. In order to do so, I first eliminate missing values and use cross-tabs between the dependent and independent variables to verify that there are no small or 0 cells. Then I run the probit model which works fine and I also obtain reasonable results:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;probit4AKIE &amp;lt;- glm(x028bin ~ x003 + x003squ + x025secv2 + x025terv2 + x007bin + x04chief + x011rec + a009bin + x045mod + c001bin + f025grouped, family=binomial(link=&quot;probit&quot;), data=wvshm5red2delna, na.action=na.pass)&#xA;&#xA;summary(probit4AKIE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, when calculating marginal effects with all variables at their means from the probit coefficients and a scale factor, the marginal effects I obtain are much too small (e.g. 2.6042e-78). The code looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ttt &amp;lt;- cbind(wvshm5red2delna$x003,&#xA;wvshm5red2delna$x003squ,&#xA;wvshm5red2delna$x025secv2,&#xA;wvshm5red2delna$x025terv2,&#xA;wvshm5red2delna$x007bin,&#xA;wvshm5red2delna$x04chief,&#xA;wvshm5red2delna$x011rec,&#xA;wvshm5red2delna$a009bin,&#xA;wvshm5red2delna$x045mod,&#xA;wvshm5red2delna$c001bin,&#xA;wvshm5red2delna$f025grouped,&#xA;wvshm5red2delna$f025grouped,&#xA;wvshm5red2delna$f025grouped,&#xA;wvshm5red2delna$f025grouped,&#xA;wvshm5red2delna$f025grouped,&#xA;wvshm5red2delna$f025grouped,&#xA;wvshm5red2delna$f025grouped,&#xA;wvshm5red2delna$f025grouped,&#xA;wvshm5red2delna$f025grouped) #I put variable &quot;f025grouped&quot; 9 times because this variable consists of 9 levels&#xA;&#xA;ttt &amp;lt;- as.data.frame(ttt)&#xA;&#xA;xbar &amp;lt;- as.matrix(mean(cbind(1,ttt[1:19]))) #1:19 position of variables in dataframe ttt&#xA;&#xA;betaprobit4AKIE &amp;lt;- probit4AKIE$coefficients&#xA;&#xA;zxbar &amp;lt;- t(xbar) %*% betaprobit4AKIE&#xA;&#xA;scalefactor &amp;lt;- dnorm(zxbar)&#xA;&#xA;marginprobit4AKIE &amp;lt;- scalefactor * betaprobit4AKIE[2:20] #2:20 are the positions of variables in the output of the probit model 'probit4AKIE' (variables need to be in the same ordering as in data.frame ttt), the constant in the model occupies the first position&#xA;&#xA;marginprobit4AKIE #in this step I obtain values that are much too small&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I apologize that I can not provide you with a working example as my dataset is much too large. Any comment would be greatly appreciated. Thanks a lot.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Best,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tobias&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: I posted the same question earlier today at stackoverflow and was kindly advised that Crossvalidated might be the more appropriate place in my case. Sorry for any inconvenience.&lt;/p&gt;&#xA;" OwnerUserId="4348" LastActivityDate="2011-04-27T14:46:36.657" Title="R glm probit regression marginal effects" Tags="&lt;r&gt;&lt;glm&gt;" CommentCount="1" />
  <row Id="10037" PostTypeId="1" CreationDate="2011-04-27T14:57:23.747" Score="2" ViewCount="86" Body="&lt;p&gt;let v to be forecasted value for periods 1 through T and $v_{t}$ be its forecasted value at time $t$. We express $v_{t}$  as the sum of two terms, its mean at time $t$,  and its deviation from the mean at time $t$, $\epsilon_{t}$. In other words, $$ v_{t}= \overline{v_{t}} + \epsilon_{t} $$ &#xA;The $\overline{v_{t}}$ are chosen based on the arguments. The $\epsilon_{t}$  term is assumed to be a normally distributed random variable with mean zero and standard deviation $\sigma (ε_{t})=0.234$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The moving average formation of order q is choosen, MA(q) where q is the number of lagged terms in the moving average. We use the following moving average specification:&#xA;$$\epsilon_{t} = \sum^{q}_{i=0}{\alpha_{i} \mu_{t-i}} $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\mu_{t-i}$ are independently distributed standard normal random variables. To ensure that the standard deviation of $εt$ is equal to its pre-specified value, we set the&#xA;$$\alpha_{i}= \frac{\sigma(\epsilon_{t})}{\sqrt{q+1}}$$&#xA;Note that $\epsilon t$ depends on $q+1$ random terms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;the R-code that i have used for above model&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;q=31&#xA;iter=10000&#xA;for(i in 1:31){alpha_i[i] &amp;lt;- 0.234/(sqrt(31+1))}&#xA;err_pk &amp;lt;- array(0,c(iter,11))&#xA;for(i in 1:iter){err_pk[i,] &amp;lt;- arima.sim(list(order=c(0,0,31), ma=alpha_i), n=11, innov=rnorm(iter,0,1))}&#xA;ffe &amp;lt;- array(0,11)&#xA;for(i in 1:11){ffe[i] &amp;lt;- median(err_pk[,i])}`&#xA;plot.ts(ffe)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am wondering that, $\alpha$ is changing through time?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and also i never get the same output as in the figure 2 &lt;a href=&quot;http://www.nature.com/nature/journal/v412/n6846/extref/412543a0_S1.htm&quot; rel=&quot;nofollow&quot;&gt;The end of world Population growth Nature, v412, 543&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;the parameter for figure in the paper are:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: MA(30), (31 terms), $\sigma(\epsilon_{t})=0.234$, 31 initial values of $\mu=0$, 10,000 simulation&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Am I missing any thing?&lt;/p&gt;&#xA;" OwnerUserId="3084" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-27T16:33:35.423" LastActivityDate="2011-04-27T16:33:35.423" Title="Fitting the moving average model" Tags="&lt;r&gt;&lt;time-series&gt;&lt;arima&gt;" />
  <row Id="10038" PostTypeId="1" CreationDate="2011-04-27T14:58:52.013" Score="4" ViewCount="67" Body="&lt;p&gt;I am trying to discern the best way to calculate a correlation and perform a one-way ANOVA on data I am taking from a PostgreSQL database.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What tools should I use?  &lt;/li&gt;&#xA;&lt;li&gt;Can I do this using the SQL language itself? &lt;/li&gt;&#xA;&lt;li&gt;Is there an easy way to export the data?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="1514" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-28T10:29:17.357" LastActivityDate="2011-04-28T10:29:17.357" Title="Conducting correlation and one way ANOVA using data from a PostgreSQL database" Tags="&lt;correlation&gt;&lt;anova&gt;&lt;sql&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="10039" PostTypeId="2" ParentId="9533" CreationDate="2011-04-27T15:10:30.253" Score="1" ViewCount="" Body="&lt;p&gt;Would the nearest-neighbor distance distribution help ?&#xA;For each of the 150 observations, you have distances&#xA;$d_1\ d_2$ ... to its nearest, 2nd nearest ... neighbors,&#xA;and an averaged distance distribution, call it DD.&#xA;A query point gives you the distribution $d_1 .. d_{150}$:&#xA;compare that to DD.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The metric between points is crucial, but I have no recipe.&#xA;Try the fractional or near-Hamming metric $\sum |a_j - b_j|^q$&#xA;(with no outer $\frac{1}{q}$).&#xA;For small $q$, say .1,&#xA;this up-weights close matches in a few features, which make sense;&#xA;otherwise a sum of 500 terms is just normally distributed&#xA;with no contrast at all / distance whiteout.&#xA;(Yes near-Hamming is not a norm,&#xA;but it is a metric, satisfies the triangle inequality.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Take a look at Omercevic et al.,&#xA;&lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.137.5973&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;High-dimensional feature matching: employing the concept of meaningful nearest neighbors&lt;/a&gt; 2007 8p,&#xA;who do this:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;find ~ 100 nearest neighbors to a query point&lt;/li&gt;&#xA;&lt;li&gt;fit $\lambda$ to exponential background noise&lt;/li&gt;&#xA;&lt;li&gt;weight the 100 neighbors: don't understand this bit, looks ad hoc&lt;/li&gt;&#xA;&lt;li&gt;pick ~ 10 outliers as &quot;signal&quot;.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;(However they're matching 128-d SIFT vectors,&#xA;whose distance distribution and noise model may be very different from yours.)&lt;/p&gt;&#xA;" OwnerUserId="557" LastActivityDate="2011-04-27T15:10:30.253" />
  <row Id="10040" PostTypeId="2" ParentId="10038" CreationDate="2011-04-27T15:13:22.780" Score="3" ViewCount="" Body="&lt;p&gt;Although you can substitute the values into the ANOVA formulas and correlation formulas yourself, maybe even calculate everything using SQL you probably want to use the ANOVA and correlation features of a statistical software. All major statistical software (SAS, SPSS, Stata, R, Statistica etc) are able to connect to databases and get the data table with a simple SQL query. Besides direct connections you can use files as well to transfer the data: export the data into a file and import them into the statistical software. The comma separated values format is often used, but many other formats may work well.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastActivityDate="2011-04-27T15:13:22.780" CommentCount="2" />
  <row Id="10041" PostTypeId="2" ParentId="10024" CreationDate="2011-04-27T15:27:37.043" Score="3" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=4767478&amp;amp;tag=1&quot; rel=&quot;nofollow&quot;&gt;This paper&lt;/a&gt; appears to prove convergence in a finite number of steps.&lt;/p&gt;&#xA;" OwnerUserId="495" LastActivityDate="2011-04-27T15:27:37.043" CommentCount="1" />
  <row Id="10042" PostTypeId="2" ParentId="9797" CreationDate="2011-04-27T16:02:11.710" Score="0" ViewCount="" Body="&lt;p&gt;Here is a simple approach that does not use logistic regression, but does attempt to use the suggestions above. Calculation of summary stats assumes, perhaps naively, that the date is normally distributed. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Please pardon inelegant code&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;write a function to estimate the day of budbreak for each individual: use the day of year half way between the last observation of 0 and the first observation of 1 for each individual.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;budburst.day &amp;lt;- function(i){&#xA;   data.subset &amp;lt;- subset(testdata, subset =&#xA;                         id == i, &#xA;                         na.rm = TRUE)&#xA;   y1 &amp;lt;- data.subset$day[max(which(data.subset$obs==0))]&#xA;   y2 &amp;lt;- data.subset$day[min(which(data.subset$obs==1))]&#xA;   y &amp;lt;- mean(c(y1, y2), na.rm = TRUE)&#xA;   if(is.na(y) | y&amp;lt;0 | y &amp;gt; 180) y &amp;lt;- NA&#xA;   return(y)&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Calculate summary statistics&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#calculate mean&#xA;mean(unlist(lapply(1:4, budburst.day)))&#xA;[1] 16.125  &#xA;&#xA;#calculate SE = sd/sqrt(n)&#xA;sd(unlist(lapply(1:4, budburst.day)))/2&#xA;[1] 5.06777&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="1381" LastActivityDate="2011-04-27T16:02:11.710" />
  <row Id="10043" PostTypeId="2" ParentId="10038" CreationDate="2011-04-27T16:02:20.230" Score="2" ViewCount="" Body="&lt;p&gt;For simple correlation, in would use the statistical functions which are directly available in Postgre.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;e.g.  ----&gt; SELECT CORR(x, y) FROM YourTable;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For ANOVA it is a bit tougher.  You might be better off transforming your problem to one involving regression and use the regression formulas.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.postgresql.org/docs/8.2/static/functions-aggregate.html&quot; rel=&quot;nofollow&quot;&gt;http://www.postgresql.org/docs/8.2/static/functions-aggregate.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="3489" LastActivityDate="2011-04-27T16:02:20.230" CommentCount="2" />
  <row Id="10044" PostTypeId="1" CreationDate="2011-04-27T16:27:36.533" Score="8" ViewCount="100" Body="&lt;p&gt;I have 100 patients and each patient have 10 longitudinal serum creatinine measurements. The estimated glomerular filtration rates (eGFR) were calculated from a MDRD formula comprising gender, age and serum creatinine. eGFR is the dependent variable and time is the independent variable in linear regression for each patient.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Do linear regressions violate the &quot;independent X's&quot; assumption and linear mixed models should be used instead?&lt;/li&gt;&#xA;&lt;li&gt;Can eGFR slopes (which are estimates with uncertainties instead of measured numbers) estimated from each patient (in linear regressions for each patient or in random effects mixed models [how to estimate slopes for each individual patient in mixed models?]) be used as the independent or dependent variables in other regression models?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Thank you.&lt;/p&gt;&#xA;" OwnerUserId="4349" LastEditorUserId="71" LastEditorDisplayName="" LastEditDate="2011-04-27T18:31:47.960" LastActivityDate="2011-04-29T04:59:18.027" Title="Can slopes in linear regressions be used as the independent or dependent variables in other regression models?" Tags="&lt;regression&gt;&lt;mixed-model&gt;&lt;repeated-measures&gt;&lt;longitudinal&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="10045" PostTypeId="1" AcceptedAnswerId="10047" CreationDate="2011-04-27T16:39:04.570" Score="6" ViewCount="110" Body="&lt;p&gt;I have hourly and daily temperature reports for many stations at &#xA;&lt;a href=&quot;http://data.barrycarter.info/&quot; rel=&quot;nofollow&quot;&gt;http://data.barrycarter.info/&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I encourage people to download it, but, at 6.6G, it uses up a lot of &#xA;bandwidth. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a service that hosts &quot;public interest&quot; data for free? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know about &lt;a href=&quot;http://aws.amazon.com/publicdatasets&quot; rel=&quot;nofollow&quot;&gt;http://aws.amazon.com/publicdatasets&lt;/a&gt;, but you need an &#xA;Amazon EC2 account to access that data. &lt;/p&gt;&#xA;" OwnerUserId="1566" LastActivityDate="2011-04-27T20:16:08.833" Title="Free public interest data hosting?" Tags="&lt;dataset&gt;" AnswerCount="3" FavoriteCount="1" />
  <row Id="10046" PostTypeId="2" ParentId="10024" CreationDate="2011-04-27T16:44:37.730" Score="1" ViewCount="" Body="&lt;p&gt;The $k$-means objective function strictly decreases with each change of assignment, which automatically implies convergence without cycling. Moreover, the partitions produced in each step of $k$-means satisfy a &quot;Voronoi property&quot; in that each point is always assigned to its nearest center. This implies an upper bound on the total number of possible partitions, which yields a finite upper bound on the termination time for $k$-means. &lt;/p&gt;&#xA;" OwnerUserId="139" LastActivityDate="2011-04-27T16:44:37.730" CommentCount="2" />
  <row Id="10047" PostTypeId="2" ParentId="10045" CreationDate="2011-04-27T17:23:10.253" Score="6" ViewCount="" Body="&lt;p&gt;I did a quick search for google projects that may fit your needs, and I came up with two hits, which I have not tested:&#xA;&lt;a href=&quot;https://www.google.com/accounts/ServiceLogin?service=fusiontables&amp;amp;passive=1209600&amp;amp;continue=http://www.google.com/fusiontables/Home&amp;amp;followup=http://www.google.com/fusiontables/Home&quot; rel=&quot;nofollow&quot;&gt;Google Fusion Tables&lt;/a&gt; and &lt;a href=&quot;http://www.google.com/publicdata/admin&quot; rel=&quot;nofollow&quot;&gt;Google Public Data&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="3542" LastActivityDate="2011-04-27T17:23:10.253" CommentCount="2" />
  <row Id="10049" PostTypeId="1" CreationDate="2011-04-27T18:37:21.917" Score="5" ViewCount="140" Body="&lt;p&gt;I am working on the BRFSS dataset with the goal of predicting Diabetes. The dataset has 500,000 rows and 405 columns. It is a 0/1 classification problem, the ratio of 0 to 1 is 90:10. I tried using decision trees, logistic regression an ensemble of decision trees and logistic regression and my misclassification rate is almost 14% in all of these methods. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What should I do to increase the accuracy? &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I saw an earlier &lt;a href=&quot;http://stats.stackexchange.com/questions/9398/supervised-learning-with-rare-events-when-rarity-is-due-to-the-large-number-of&quot;&gt;post&lt;/a&gt; which says subsampling or assigning different weights helps. But I am not sure about the ratio. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What would be the best ratio to start off with? &lt;/li&gt;&#xA;&lt;li&gt;I am working using SAS. Is there a way to do subsampling in SAS? &lt;/li&gt;&#xA;&lt;li&gt;I am also interested in trying out the weighted approach. Is there a way to implement this in SAS?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;EDIT (28 Apr 2011)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried subsampling and my misclassification rate goes up from 14% to 23%. The ratio I used was 50:50 for classes 0 and 1. The original ratio in the data was 90:10 and using the data as it is gave 14% error. So I believe subsampling doesn't work for my data. Would you suggest any other  way to improve accuracy?&lt;/p&gt;&#xA;" OwnerUserId="3897" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-28T15:07:54.027" LastActivityDate="2011-05-03T13:25:29.633" Title="Improving accuracy of a binary classification when the target is unbalanced" Tags="&lt;machine-learning&gt;&lt;sas&gt;" AnswerCount="4" CommentCount="5" />
  <row Id="10050" PostTypeId="2" ParentId="10045" CreationDate="2011-04-27T18:44:38.870" Score="1" ViewCount="" Body="&lt;p&gt;You should also take a look at &lt;a href=&quot;http://www.infochimps.com/&quot; rel=&quot;nofollow&quot;&gt;Infochimps&lt;/a&gt;.  I've never used the site personally, but it's designed for precisely this.&lt;/p&gt;&#xA;" OwnerUserId="71" LastActivityDate="2011-04-27T18:44:38.870" />
  <row Id="10051" PostTypeId="1" AcceptedAnswerId="10061" CreationDate="2011-04-27T18:45:15.907" Score="2" ViewCount="41" Body="&lt;p&gt;Hello all,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am looking for a repeated measures version of the &lt;a href=&quot;http://en.wikipedia.org/wiki/Logrank_test&quot; rel=&quot;nofollow&quot;&gt;Logrank test&lt;/a&gt;.  If I am correct, I am looking for an equivalent of the Friedman test for survival data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestions on where to look? (and R code will always be welcomed :) )&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="253" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-27T22:56:56.187" LastActivityDate="2011-04-27T22:56:56.187" Title="Is there a repeated measures aware version of the logrank test?" Tags="&lt;r&gt;&lt;nonparametric&gt;&lt;survival&gt;&lt;logrank&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="1" />
  <row Id="10052" PostTypeId="2" ParentId="10044" CreationDate="2011-04-27T18:54:41.967" Score="5" ViewCount="" Body="&lt;p&gt;In effect, you are proposing to use linear regression as a mathematical procedure to condense a 10-variate observation into a single variable (the slope).  As such it's just another example of similar procedures like (say) using an average of repeated measurements as a regression variable or including principal components scores in a regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Specific comments follow.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(1) Linear regression does not require the X's (independent variables) to be &quot;independent.&quot;  Indeed, in the standard formulation the concept of independence does not even apply because the X's are fixed values, not realizations of a random variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(2) Yes, you can use the slopes as &lt;em&gt;dependent&lt;/em&gt; variables.  It would help to establish that they might behave like the dependent variable in linear regression.  For ordinary least squares this means that&lt;/p&gt;&#xA;&#xA;&lt;p&gt;a.  Slopes may depend on some of the patient attributes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;b.  The dependence is approximately linear, at least for the range of observed patient attributes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;c.  Any variation between an observed slope and the hypothesized slope can be considered random.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;d.  This random variation is (i) independent from patient to patient and (ii) has approximately the same distribution from patient to patient.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;e.  As before, the independent variables are not viewed as random but as fixed and measured without appreciable error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If all these conditions approximately hold, you should be ok.  Violations of (d) or (e) can be cured by using generalizations of ordinary least squares.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(2'). Because the slopes will exhibit uncertainty (as measured in the regression used to estimate the slopes), they might not be good candidates for &lt;em&gt;independent&lt;/em&gt; variables unless you are treating them as random in a mixed model or are using an errors-in-variables model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also cope with this situation by means of a &lt;a href=&quot;http://en.wikipedia.org/wiki/Hierarchical_Bayes_model&quot; rel=&quot;nofollow&quot;&gt;hierarchical Bayes model&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-04-27T18:54:41.967" CommentCount="2" />
  <row Id="10053" PostTypeId="1" CreationDate="2011-04-27T19:10:24.630" Score="5" ViewCount="84" Body="&lt;p&gt;I am attempting to test the goodness of fit for a vector of count data to a binomial.  To do so I am using the &lt;code&gt;goodfit()&lt;/code&gt; function in the &lt;code&gt;vcd&lt;/code&gt; package.  When I run the function, however, it returns &lt;code&gt;NaN&lt;/code&gt; for the p-value of the Chi-squared test.  In my setup, I have a vector of count data with 75 elements.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; library(vcd)&#xA;&amp;gt; counts &amp;lt;- c(32, 35, 44, 35, 41, 33, 42, 49, 36, 41, 42, 45, 38, 43, 36, &#xA;35, 40, 40, 43, 34, 39, 31, 40, 39, 36, 37, 37, 37, 32, 48, 41, &#xA;32, 37, 36, 49, 37, 41, 36, 34, 37, 41, 32, 36, 36, 30, 33, 33, &#xA;42, 39, 36, 36, 29, 31, 41, 36, 39, 40, 37, 39, 39, 31, 39, 37, &#xA;40, 33, 41, 34, 46, 35, 41, 44, 38, 44, 34, 42)&#xA;&amp;gt; test.gof &amp;lt;- goodfit(counts, type=&quot;binomial&quot;, &#xA;+                     par=list(size=length(counts), prob=0.5))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Everything works fine, but when I inspect the &lt;code&gt;goodfit()&lt;/code&gt; object I get the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; summary(test.gof)&#xA;&#xA; Goodness-of-fit test for binomial distribution&#xA;&#xA;                      X^2 df  P(&amp;gt; X^2)&#xA;Pearson               NaN 75       NaN&#xA;Likelihood Ratio 21.48322 19 0.3107244&#xA;Warning message:&#xA;In summary.goodfit(test.gof) : Chi-squared approximation may be incorrect&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I suspected it was a small sample size issue at first, but I also have a data set with 50 observations that does not return &lt;code&gt;NaN&lt;/code&gt; for the p-value.  I have also tried to switch the method in &lt;code&gt;goodfit()&lt;/code&gt; to ML with similar results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why would this function be producing &lt;code&gt;NaN&lt;/code&gt; in this case?  Is there an alternative function to calculate GOF on count data?&lt;/p&gt;&#xA;" OwnerUserId="302" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-28T22:50:54.383" LastActivityDate="2011-04-28T22:50:54.383" Title="NaN p-value when using R's goodfit on binomial data" Tags="&lt;binomial&gt;&lt;chi-squared&gt;&lt;goodness-of-fit&gt;" AnswerCount="3" CommentCount="5" FavoriteCount="1" />
  <row Id="10054" PostTypeId="2" ParentId="10049" CreationDate="2011-04-27T19:25:02.690" Score="1" ViewCount="" Body="&lt;p&gt;Regarding decision trees, I would suggest the following. Assume that you have 10 training examples from class $C_1$ and 90 training examples from class $C_2$. You can use an ensemble of $N$ decision trees, where each tree is trained on 10 examples from $C_1$ and 10 randomly chosen examples from $C_2$. The decision of the ensemble may be the majority vote. You can play with different $N$ to see how it works.&lt;/p&gt;&#xA;" OwnerUserId="4337" LastEditorUserId="4337" LastEditorDisplayName="" LastEditDate="2011-04-28T02:49:11.433" LastActivityDate="2011-04-28T02:49:11.433" CommentCount="2" />
  <row Id="10055" PostTypeId="1" CreationDate="2011-04-27T19:37:31.773" Score="0" ViewCount="51" Body="&lt;p&gt;I have a table I want to convert into a graph (bar-graph or line-graph)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first column has fixed values. Twenty different values are simulated for these fixed values and kept in the next columns. I want to plot a graph of the fixed column against all the different twenty simulated columns.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I go about it? I am using R2.12.1&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;table name:bygrace&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;V1    V2   V3   V4   V5&lt;br&gt;&#xA;  100    16   11   -6    1&lt;br&gt;&#xA;  120   -17  -12    7   -2&lt;br&gt;&#xA;  140    18   13   -8    3&lt;br&gt;&#xA;  150   -19  -14    9   -4&lt;br&gt;&#xA;  210    20   15  -10   -5&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; values for V1 are:100,120,140 &#xA;             V2:16,-17,18,-19,20&#xA;            V3:11,-12,13,-14,15&#xA;            V4:-6,7,-8,9,10&#xA;            V5.1,-2,3,-4,-5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Actually, my table looks like the one above.&#xA;The first column V1 represents Premiums charged by an insurance company,say last year (for 5 policyholders/policies)&#xA;After critical analysis of the portfolio, the company decides to increase or decrease the premium amount for some of the customers.&#xA;However, the company does not know exactly by how much it should increase of decrease the premiums for fear that it would lose customers.&#xA;For this reason, four different scenarios/simulations are considered which are represented by the next four columns(V2 through V5) respectively.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now the task is to plot the premium amounts in V1 against these four different scenarios(bar-graph or line-graph; I think bar will be better).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And my question is can this be done on one graph/at once? If yes, how shoul I go about it?&#xA;Or do I have to plot the premium against each column separately?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In fact, I have spent the whole of yesterday and today on this but I have not been able to get the desired result. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Someone gave me an answer to try.And I am going to do that because it has given me an idea and I am very thankful to the one!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks to everyone for their help&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Owusu Isaac&lt;/p&gt;&#xA;" OwnerUserId="4340" LastActivityDate="2011-04-27T19:37:31.773" Title="How to convert a table into a graph in R" Tags="&lt;graph-theory&gt;" AnswerCount="0" CommentCount="0" FavoriteCount="0" />
  <row Id="10056" PostTypeId="2" ParentId="10045" CreationDate="2011-04-27T20:16:08.833" Score="3" ViewCount="" Body="&lt;p&gt;How about the &lt;a href=&quot;http://archive.ics.uci.edu/ml/index.html&quot; rel=&quot;nofollow&quot;&gt;UCI Machine Learning Repository&lt;/a&gt;? Here is their &lt;a href=&quot;http://archive.ics.uci.edu/ml/donation_policy.html&quot; rel=&quot;nofollow&quot;&gt;data donation policy&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="4325" LastActivityDate="2011-04-27T20:16:08.833" />
  <row Id="10057" PostTypeId="1" CreationDate="2011-04-27T20:45:01.373" Score="1" ViewCount="63" Body="&lt;p&gt;Many academic papers focusing on statistical learning in the applied setting of finance train a model such that their parameter set, B,  defines the relationship &#xA;y(t) = B*x(t) + e(t)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;y could be a 0-1 coded response for an increase (or not) of the asset price at time t.  Ignoring the details of the model my question is as follows: &#xA;given that you are predicting the response at the SAME time that you record your features, how would you make use of this model?  Intuitively I think ok, my input space says we will &#xA;see an increase in y but since they are at the same time, I can actually go and see what y has in fact done.  Do I take the view of my model to be the truth and create a trading signal based on this information (for example)? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any practical insight would be greatly appreciated.&lt;/p&gt;&#xA;" OwnerUserId="4352" LastActivityDate="2011-05-27T20:50:51.843" Title="Classification in a applied financial setting" Tags="&lt;classification&gt;" AnswerCount="1" />
  <row Id="10058" PostTypeId="1" AcceptedAnswerId="10321" CreationDate="2011-04-27T20:45:05.570" Score="5" ViewCount="126" Body="&lt;p&gt;I need a reality check if you will. I have a data set, where I know how many individual butterflies of two species co-occur at one meadow (not always, though). I have additional variables, for instance wet/dry meadow, intensely cultivated/not cultivated, percent of area around the meadow covered by wet or dry meadows...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is the head of the dataset. 50 rows in total, 25 per species. Notice that all columns are identical except count and species, indicating that they come from the same sampling location.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; head(dej)&#xA;  count     type1 type2 perc.for.100m perc.dry.100m perc.wet.100m species&#xA;1     1 intensive   dry        13.836        22.724         0.000   reali&#xA;2     3 extensive   wet         6.877         1.613        52.213   reali&#xA;3     4 intensive   wet        22.770         0.537        44.901   reali&#xA;4     6 intensive   dry        17.346        42.322         6.359   reali&#xA;5     1 extensive   wet        34.854         9.091        11.950   reali&#xA;6     2 extensive   dry        50.387        19.245         0.000   reali&#xA;...&#xA;26     0 intensive   dry        13.836        22.724         0.000 sinapis&#xA;27     0 extensive   wet         6.877         1.613        52.213 sinapis&#xA;28     0 intensive   wet        22.770         0.537        44.901 sinapis&#xA;29     0 intensive   dry        17.346        42.322         6.359 sinapis&#xA;30     1 extensive   wet        34.854         9.091        11.950 sinapis&#xA;31     1 extensive   dry        50.387        19.245         0.000 sinapis&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm interested in knowing if any of these variables influence the species and their respective counts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And this is the result of the &quot;full&quot; model.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;glm(formula = count ~ type1 + type2 + perc.for.100m + perc.dry.100m + &#xA;    perc.wet.100m + species, family = poisson, data = dej)&#xA;&#xA;Deviance Residuals: &#xA;    Min       1Q   Median       3Q      Max  &#xA;-2.8458  -1.1414  -0.4546   0.8297   2.2145  &#xA;&#xA;Coefficients:&#xA;               Estimate Std. Error z value Pr(&amp;gt;|z|)   &#xA;(Intercept)    0.028129   0.523509   0.054  0.95715   &#xA;type1intensive 0.196699   0.191960   1.025  0.30551   &#xA;type2wet       0.071841   0.334286   0.215  0.82984   &#xA;perc.for.100m  0.003741   0.008277   0.452  0.65130   &#xA;perc.dry.100m  0.010952   0.010750   1.019  0.30829   &#xA;perc.wet.100m  0.007467   0.011596   0.644  0.51960   &#xA;speciessinapis 0.597837   0.187689   3.185  0.00145 **&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Does this sound like the correct approach at all?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Some additional information&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a side note, based on my exploration of the data, I would expect that the count would depend (at least) on the type2 variable, alas that's not what I got.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/9sYc5.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using the &quot;reverse logic&quot;, I tried if species can be predicted using my data, which presumably confirms the above results.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Call:&#xA;glm(formula = species ~ type1 + type2 + perc.for.100m + perc.dry.100m + &#xA;    perc.wet.100m + count, family = binomial, data = dej)&#xA;&#xA;Deviance Residuals: &#xA;    Min       1Q   Median       3Q      Max  &#xA;-1.6322  -1.0136  -0.1568   1.0592   1.6407  &#xA;&#xA;Coefficients:&#xA;                Estimate Std. Error z value Pr(&amp;gt;|z|)  &#xA;(Intercept)    -0.351192   1.658052  -0.212   0.8323  &#xA;type1intensive -0.170583   0.651611  -0.262   0.7935  &#xA;type2wet       -0.107377   1.078726  -0.100   0.9207  &#xA;perc.for.100m  -0.002806   0.026807  -0.105   0.9166  &#xA;perc.dry.100m  -0.010227   0.036982  -0.277   0.7821  &#xA;perc.wet.100m  -0.006486   0.038071  -0.170   0.8647  &#xA;count           0.345036   0.153811   2.243   0.0249 *&#xA;---&#xA;Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 &#xA;&#xA;(Dispersion parameter for binomial family taken to be 1)&#xA;&#xA;    Null deviance: 69.315  on 49  degrees of freedom&#xA;Residual deviance: 63.431  on 43  degrees of freedom&#xA;AIC: 77.431&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;EDIT 1&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Aniko noticed that there may be an interaction between the type2 and species. Indeed!&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Call:&#xA;glm(formula = count ~ type1 + type2 * species + perc.for.100m + &#xA;    perc.dry.100m + perc.wet.100m, family = poisson, data = dej)&#xA;&#xA;Deviance Residuals: &#xA;    Min       1Q   Median       3Q      Max  &#xA;-3.0859  -1.1350  -0.1947   0.7109   2.7470  &#xA;&#xA;Coefficients:&#xA;                         Estimate Std. Error z value Pr(&amp;gt;|z|)    &#xA;(Intercept)             -0.357165   0.559987  -0.638  0.52360    &#xA;type1intensive           0.196699   0.191960   1.025  0.30551    &#xA;type2wet                 0.704769   0.429087   1.642  0.10049    &#xA;speciessinapis           1.145132   0.306847   3.732  0.00019 ***&#xA;perc.for.100m            0.003741   0.008277   0.452  0.65130    &#xA;perc.dry.100m            0.010952   0.010750   1.019  0.30829    &#xA;perc.wet.100m            0.007467   0.011596   0.644  0.51960    &#xA;type2wet:speciessinapis -0.962811   0.394038  -2.443  0.01455 *  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;EDIT 2&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After removing the non significant terms (assuming I found the global maximum for the sake of data dredging) the story gets another twist into the right direction.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Call:&#xA;glm(formula = count ~ type2 * species, family = poisson, data = dej)&#xA;&#xA;Deviance Residuals: &#xA;    Min       1Q   Median       3Q      Max  &#xA;-2.7080  -1.1617  -0.1582   0.6979   3.1599  &#xA;&#xA;Coefficients:&#xA;                        Estimate Std. Error z value Pr(&amp;gt;|z|)    &#xA;(Intercept)               0.1542     0.2673   0.577  0.56408    &#xA;type2wet                  0.6821     0.3237   2.107  0.03508 *  &#xA;speciessinapis            1.1451     0.3068   3.732  0.00019 ***&#xA;type2wet:speciessinapis  -0.9628     0.3940  -2.443  0.01455 *&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="144" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-27T22:55:54.050" LastActivityDate="2011-05-04T16:50:48.563" Title="Reality check using glm" Tags="&lt;r&gt;&lt;glm&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="10059" PostTypeId="1" CreationDate="2011-04-27T20:48:00.120" Score="2" ViewCount="84" Body="&lt;ul&gt;&#xA;&lt;li&gt;Is it possible to do 2-stage cluster analysis in R?&lt;/li&gt;&#xA;&lt;li&gt;Can anybody provide me resource on it?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="4278" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-28T02:14:29.323" LastActivityDate="2011-04-30T08:12:03.110" Title="Two-stage clustering in R" Tags="&lt;r&gt;&lt;clustering&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="10060" PostTypeId="2" ParentId="10057" CreationDate="2011-04-27T20:49:05.533" Score="1" ViewCount="" Body="&lt;p&gt;If you want to predict the future, and then use that information to profit, you need to build a model like this:&#xA;&lt;code&gt;Next(y)~x&lt;/code&gt; or &lt;code&gt;y(t+1)=B*x(t)+e(t)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;i.e. specify your problem as &quot;what will happen tomorrow&quot; not &quot;what did happen today.&quot;&#xA;Look at the &lt;a href=&quot;http://cran.r-project.org/web/packages/quantmod/index.html&quot; rel=&quot;nofollow&quot;&gt;quantmod&lt;/a&gt; package for R for more examples.&lt;/p&gt;&#xA;" OwnerUserId="2817" LastActivityDate="2011-04-27T20:49:05.533" CommentCount="2" />
  <row Id="10061" PostTypeId="2" ParentId="10051" CreationDate="2011-04-27T21:08:56.193" Score="1" ViewCount="" Body="&lt;p&gt;You can probably use a so-called marginal survival model. It would be more like Cox-regression than a log-rank test, i.e. proportional hazards for the effect of task would be assumed. It is implemented in the &lt;code&gt;survival&lt;/code&gt; package:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mod &amp;lt;- coxph(Surv(time, censor) ~ task + cluster(id))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here &lt;code&gt;task&lt;/code&gt; would be a factor representing the task, and &lt;code&gt;id&lt;/code&gt; would identify the subject. There is a book that goes into lots of detail on multivariate survival data: T.Therneau, P. Grambsch. &lt;em&gt;Modeling Survival Data: Extending the Cox Model&lt;/em&gt;. Springer, 2000.&lt;/p&gt;&#xA;" OwnerUserId="279" LastActivityDate="2011-04-27T21:08:56.193" />
  <row Id="10062" PostTypeId="1" AcceptedAnswerId="10070" CreationDate="2011-04-27T21:10:13.713" Score="4" ViewCount="72" Body="&lt;p&gt;I would like to fit a mixture model to Monte Carlo generated data with probability densities which typically look like those in the attached image.&#xA;&lt;img src=&quot;http://i.imgur.com/TQw1z.png&quot; alt=&quot;typical densities&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It would seem from visual inspection that a normal mixture model might be applicable but on checking the CRAN task view I really do not know which package might be appropriate for my needs. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically what I would like to do is supply a vector of the data and then have the package function return the mean, variance and proportional weights for each component in the mixture model, and also perhaps identify how many components there are in the model. &lt;/p&gt;&#xA;" OwnerUserId="226" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-05-11T13:29:30.453" LastActivityDate="2011-05-11T13:57:14.200" Title="Which R package to use to calculate component parameters for a mixture model" Tags="&lt;r&gt;&lt;mixed-model&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="10063" PostTypeId="1" CreationDate="2011-04-27T18:17:13.837" Score="2" ViewCount="67" Body="&lt;p&gt;I am attempting to test the goodness of fit for a vector of count data to a binomial.  To do so I am using the &lt;code&gt;goodfit()&lt;/code&gt; function in the &lt;code&gt;vcd&lt;/code&gt; package.  When I run the function, however, it returns &lt;code&gt;NaN&lt;/code&gt; for the p-value of the Chi-squared test.  In my setup, I have a vector of count data with 75 elements.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; library(vcd)&#xA;&amp;gt; counts&#xA; [1] 32 35 44 35 41 33 42 49 36 41 42 45 38 43 36 35 40 40 43 34 39 31 40 39 36 37 37 37 32 48 41 32 37 36 49 37 41 36 34 37 41 32 36 36 30 33 33 42 39 36 36 29 31&#xA;[54] 41 36 39 40 37 39 39 31 39 37 40 33 41 34 46 35 41 44 38 44 34 42&#xA;&amp;gt; test.gof &amp;lt;- goodfit(counts, type=&quot;binomial&quot;, &#xA;+                     par=list(size=length(counts), prob=0.5))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Everything works fine, but when I inspect the &lt;code&gt;goodfit()&lt;/code&gt; object I get the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; summary(test.gof)&#xA;&#xA; Goodness-of-fit test for binomial distribution&#xA;&#xA;                      X^2 df  P(&amp;gt; X^2)&#xA;Pearson               NaN 75       NaN&#xA;Likelihood Ratio 21.48322 19 0.3107244&#xA;Warning message:&#xA;In summary.goodfit(test.gof) : Chi-squared approximation may be incorrect&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I suspected it was a small sample size issue at first, but I also have a data set with 50 observations that does not return &lt;code&gt;NaN&lt;/code&gt; for the p-value.  I have also tried to switch the method in &lt;code&gt;goodfit()&lt;/code&gt; to ML with similar results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why would this function be producing &lt;code&gt;NaN&lt;/code&gt; in this case?  Is there an alternative function to calculate GOF on count data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data used:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;counts &amp;lt;- c(32, 35, 44, 35, 41, 33, 42, 49, 36, 41, 42, 45, 38, 43, 36, &#xA;35, 40, 40, 43, 34, 39, 31, 40, 39, 36, 37, 37, 37, 32, 48, 41, &#xA;32, 37, 36, 49, 37, 41, 36, 34, 37, 41, 32, 36, 36, 30, 33, 33, &#xA;42, 39, 36, 36, 29, 31, 41, 36, 39, 40, 37, 39, 39, 31, 39, 37, &#xA;40, 33, 41, 34, 46, 35, 41, 44, 38, 44, 34, 42)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="302" LastActivityDate="2011-04-27T22:04:50.433" Title="p-value NaN when using goodfit() on binomial data" Tags="&lt;r&gt;" AnswerCount="0" CommentCount="0" FavoriteCount="0" />
  <row Id="10064" PostTypeId="2" ParentId="10053" CreationDate="2011-04-27T18:54:14.177" Score="3" ViewCount="" Body="&lt;p&gt;You have zero frequencies in observed counts. That explains &lt;code&gt;NaN&lt;/code&gt;s in your data. If you look at &lt;code&gt;test.gof&lt;/code&gt; object, you'll see that:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;table(test.gof$observed)&#xA;&#xA; 0  1  2  3  4  5  7  8 10 &#xA;56  5  3  2  5  1  1  2  1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;you have 56 zeros. Anyway, IMHO this question is for &lt;a href=&quot;http://stats.stackexchange.com&quot;&gt;http://stats.stackexchange.com&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1356" LastActivityDate="2011-04-27T18:54:14.177" CommentCount="2" />
  <row Id="10065" PostTypeId="1" CreationDate="2011-04-27T22:17:42.550" Score="2" ViewCount="41" Body="&lt;p&gt;I have a collection of $10^5$ essays, each of which on average contains $10^3$ distinct words. There are $10^6$ distinct words in the entire collection. If I index every word what is the mean and median size of the inverted index lists?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My guess is that median would be 1, but I have no clue how without the parameters can I calculate harmonic mean?&#xA;Can anybody help?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;UPDATE: I should have really mentioned that before: I am talking about English language.&lt;/p&gt;&#xA;" OwnerUserId="1371" LastEditorUserId="1371" LastEditorDisplayName="" LastEditDate="2011-04-28T07:10:04.853" LastActivityDate="2011-04-28T07:10:04.853" Title="Mean and median in Zipf's  distribution" Tags="&lt;homework&gt;&lt;mean&gt;&lt;median&gt;&lt;zipf&gt;" CommentCount="2" />
  <row Id="10066" PostTypeId="1" CreationDate="2011-04-27T22:40:23.897" Score="2" ViewCount="50" Body="&lt;p&gt;I am trying to perform a canonical correlation analysis to investigate the relationship between attitudes (14 variables), perceived consumer effectiveness (6 variables) AND Intention to dine (DV) 14 variables. However when SPSS generates the Manova Output there are no  tables on redundancy index and as far as I understand  this index is important to report as it explains how good are the IVs as the predictors of the variance in DV. &#xA;This is the script that  i am using for the test:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;manova Attitudes PCE with Intention&#xA;/ discrim all alpha(1) &#xA;/ print=sig(eigen dim).&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Thank you!&#xA;Oksana&lt;/p&gt;&#xA;" OwnerUserId="4356" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-28T09:12:41.800" LastActivityDate="2011-04-28T09:12:41.800" Title="How to get a redundancy index when performing canonical correlation analysis in SPSS?" Tags="&lt;correlation&gt;&lt;multivariate-analysis&gt;&lt;spss&gt;" AnswerCount="2" />
  <row Id="10067" PostTypeId="1" CreationDate="2011-04-27T23:16:13.090" Score="2" ViewCount="41" Body="&lt;p&gt;Hi, I am a software engineer by trade doing stats in my free time.  I am playing around with an implementation of &lt;a href=&quot;http://research.microsoft.com/en-us/projects/trueskill/&quot; rel=&quot;nofollow&quot;&gt;Microsoft's TrueSkill&lt;/a&gt; rating system for ranking players and openings in from a data set of ~700,000 Dominion games.  I want to measure the log loss of predictions that the system gives for multiplayer games so that I can optimize hyperparameters of the system as well as explicitly model the inherent turn order advantage in the game.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, the TrueSkill builds a Bayesian Graphical Model, assuming a normal distribution on player skills as well as a normal distribution for player luck per game, and produces a mean/variance for the independent performance of players.  For a two player game, the probability of one beating the other is encoded in the difference of the performance distributions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, predicting the probability of a player winning a 3 player game seems to requires estimating the probability that given player will exceed the maximum performance of the other two players.  Is the maximum of two Gaussian also Gaussian?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've read &lt;a href=&quot;http://ee162.caltech.edu/notes/lect8.pdf&quot; rel=&quot;nofollow&quot;&gt;these notes&lt;/a&gt; which seem to solve a more general version of problem that I want on page 27 with equation (8-47).   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically it says that for random variables X and Y, then if &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Z = \max(X, Y)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With X and Y independent,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(1) $F_z(z) = F_x(x) F_y(y)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and hence&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(2) $f_z(z) = F_x(z)f_y(z) + f_x(z)F_y(z)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should I then plug in the formulas for the cdf/pdfs and pray that I get something useful?&lt;/p&gt;&#xA;" OwnerUserId="4164" LastActivityDate="2011-04-27T23:16:13.090" Title="Maximum of two (or more) gaussian distributions with known and possibly different means/variances" Tags="&lt;normal-distribution&gt;&lt;predictive-models&gt;" />
  <row Id="10069" PostTypeId="2" ParentId="10005" CreationDate="2011-04-28T00:23:08.120" Score="3" ViewCount="" Body="&lt;p&gt;Here's a few options using the ggplot2 package. If you haven't started learning much about plotting with R, taking the time now to learn &lt;a href=&quot;http://had.co.nz/ggplot2/&quot; rel=&quot;nofollow&quot;&gt;ggplot2&lt;/a&gt; or lattice may be worth the effort. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;ggplot requires data to be in &quot;long&quot; format, so the first thing we'll do is reformat to long:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(ggplot2)&#xA;&#xA;dat &amp;lt;- data.frame(V1 = c(100, 120, 140, 150, 210), V2 = c(16, -17, 18, -19, 20)&#xA;, V3 = c(11, -12, 13, -14, 15), V4 = c(-6, 7, -8, 9, -10), V5 = c(1, -2, 3, -4, 5))&#xA;&#xA;dat.m &amp;lt;- melt(dat, &quot;V1&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It sounds like V1 represents discrete levels, and should be treated as categorical data. That is why I treat V1 as a factor. If that's not the case, simply use V1 in the x-axis.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#Bar chart&#xA;ggplot(dat.m, aes(x = factor(V1), y = value, fill = variable)) + &#xA;geom_bar(position = &quot;dodge&quot;)&#xA;&#xA;#Line chart&#xA;ggplot(dat.m, aes(x = factor(V1), y = value, group = variable, colour = variable)) + &#xA;geom_line()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/zLCVu.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="696" LastActivityDate="2011-04-28T00:23:08.120" CommentCount="2" />
  <row Id="10070" PostTypeId="2" ParentId="10062" CreationDate="2011-04-28T00:38:55.720" Score="4" ViewCount="" Body="&lt;p&gt;Try  &lt;a href=&quot;http://cran.r-project.org/web/packages/mixdist/index.html&quot; rel=&quot;nofollow&quot;&gt;mixdist&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's an example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(mixdist)  &#xA;&#xA;#Build data vector &quot;x&quot; as a mixture of data from 3 Normal Distributions  &#xA;x1 &amp;lt;- rnorm(1000, mean=0, sd=2.0)  &#xA;x2 &amp;lt;- rnorm(500, mean=9, sd=1.5)  &#xA;x3 &amp;lt;- rnorm(300, mean=13, sd=1.0)  &#xA;x &amp;lt;- c(x1, x2, x3)  &#xA;&#xA;#Plot a histogram (you'll play around with the value for &quot;breaks&quot; as    &#xA;#you zero-in on the fit).   Then build a data frame that has the  &#xA;#bucket midpoints and counts.  &#xA;breaks &amp;lt;- 30  &#xA;his &amp;lt;- hist(x, breaks=breaks)  &#xA;df &amp;lt;- data.frame(mid=his$mids, cou=his$counts)  &#xA;head(df)  &#xA;&#xA;#The above Histogram shows 3 peaks that might be represented by 3 Normal  &#xA;#Distributions.  Guess at the 3 Means in Ascending Order, with a guess for  &#xA;#the associated 3 Sigmas and fit the distribution.  &#xA;guemea &amp;lt;- c(3, 11, 14)  &#xA;guesig &amp;lt;- c(1, 1, 1)  &#xA;guedis &amp;lt;- &quot;norm&quot;  &#xA;(fitpro &amp;lt;- mix(as.mixdata(df), mixparam(mu=guemea, sigma=guesig), dist=guedis))  &#xA;&#xA;#Plot the results  &#xA;plot(fitpro, main=&quot;Fit a Probability Distribution&quot;)  &#xA;grid()  &#xA;legend(&quot;topright&quot;, lty=1, lwd=c(1, 1, 2), c(&quot;Original Distribution to be Fit&quot;, &quot;Individual Fitted Distributions&quot;, &quot;Fitted Distributions Combined&quot;), col=c(&quot;blue&quot;, &quot;red&quot;, rgb(0.2, 0.7, 0.2)), bg=&quot;white&quot;)  &#xA;&#xA;&#xA;===========================  &#xA;&#xA;&#xA;Parameters:  &#xA;      pi     mu  sigma  &#xA;1 0.5533 -0.565 1.9671  &#xA;2 0.2907  8.570 1.6169  &#xA;3 0.1561 12.725 0.9987  &#xA;&#xA;Distribution:  &#xA;[1] &quot;norm&quot;  &#xA;&#xA;Constraints:  &#xA;   conpi    conmu consigma   &#xA;  &quot;NONE&quot;   &quot;NONE&quot;   &quot;NONE&quot;   &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/BzSbw.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="2775" LastActivityDate="2011-04-28T00:38:55.720" />
  <row Id="10071" PostTypeId="2" ParentId="9850" CreationDate="2011-04-28T01:51:18.807" Score="1" ViewCount="" Body="&lt;p&gt;Perhaps try something like &lt;a href=&quot;http://portal.acm.org/citation.cfm?id=223812&quot; rel=&quot;nofollow&quot;&gt;Fastmap&lt;/a&gt; to plot your set of marks using their relative distances.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://gromgull.net/blog/2009/08/fastmap-in-python/&quot; rel=&quot;nofollow&quot;&gt;(still) nothing clever&lt;/a&gt; has written up Fastmap in python to plot strings and could be easily updated to handle lists of attributes if you wrote up your own distance metric.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Below is a standard euclidean distance I use that takes two lists of attributes as parameters.  If your lists have a class value, don't use it in the distance calculation.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def distance(vecone, vectwo, d=0.0):&#xA;    for i in range(len(vecone)):&#xA;        if isnumeric(vecone[i]):&#xA;            d = d + (vecone[i] - vectwo[i])**2&#xA;        elif vecone[i] is not vectwo[i]:&#xA;            d += 1.0&#xA;    return math.sqrt(d)&#xA;&#xA;def isnumeric(s):&#xA;    try:&#xA;        float(s)&#xA;        return True&#xA;    except ValueError:&#xA;        return False&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1856" LastActivityDate="2011-04-28T01:51:18.807" />
  <row Id="10072" PostTypeId="2" ParentId="220" CreationDate="2011-04-28T02:10:35.973" Score="1" ViewCount="" Body="&lt;p&gt;If the cdf of $X_i$ is denoted by $F(x)$, then the cdf of the minimum is given by $1-[1-F(x)]^n$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Reasoning :&lt;/strong&gt;&#xA;Given $n$ Random Var, the probability $P(Y\leq y) = P(\min(X_1\dots X_n)\leq y)$ &#xA;implies that at least one $X_i$ be smaller than $y$. The probability that at least one $X_i$ is smaller than $y$ is equivalent to one minus the probability that all $X_i$ are greater than $y$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the $X_i$'s are iid, then the probability that all $X_i$ is greater than $y = (1-F(y))^n$.&#xA;Therefore, the original prob is $1-(1-F(y))^n$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say $X_i \sim \text{Uniform} (0,1)$, then intuitively the probability $\min(X_1\dots X_n)\leq 1$ should be equivalent to 1 (as the minimum value would always be less than 1 since $0\leq X_i\leq 1$ for all $i$). In this case $F(1)=1$ thus the probability is always 1.&lt;/p&gt;&#xA;" OwnerUserId="4358" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-28T09:45:03.343" LastActivityDate="2011-04-28T09:45:03.343" CommentCount="1" />
  <row Id="10073" PostTypeId="2" ParentId="10066" CreationDate="2011-04-28T02:34:53.760" Score="1" ViewCount="" Body="&lt;p&gt;It's been a while since I've run canonical correlation in SPSS.&#xA;Perhaps have a look at the &lt;code&gt;cancorr&lt;/code&gt; macro. It's an alternative way of running a canonical correlation, and from memory it provides slightly different output.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Running Cancor&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Here's some &lt;a href=&quot;http://jeromyanglim.blogspot.com/2010/06/canonical-correlation-getting-started.html&quot; rel=&quot;nofollow&quot;&gt;info on running &lt;code&gt;cancorr&lt;/code&gt;&lt;/a&gt;, extracting the important bit:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Note that the location script required to run CANCORR changes between versions and installations of SPSS. David Garson sets out the following code template:&quot;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;INCLUDE 'c:\Program Files\SPSS\Canonical correlation.sps'.&#xA;CANCORR SET1=varlist/  SET2=varlist/.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h3&gt;Additional Reference&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Here's some &lt;a href=&quot;http://www.iasri.res.in/ebook/EBADAT/4-Applications%20of%20Multivariate%20Techniques/3-canonical%20correlation.pdf&quot; rel=&quot;nofollow&quot;&gt;more information from a book chapter by V.K. Bhatia&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastActivityDate="2011-04-28T02:34:53.760" CommentCount="1" />
  <row Id="10074" PostTypeId="1" AcceptedAnswerId="10097" CreationDate="2011-04-28T03:46:16.683" Score="0" ViewCount="39" Body="&lt;p&gt;After a little searching, I was able to find data from the 1990 US census on the frequency of male and female first names, and on surnames. That data is &lt;a href=&quot;http://www.census.gov/genealogy/names/names_files.html&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;. I'm not able to find similar data for any other census years, however. Does anyone know where I can get data for other years, both earlier and later?&lt;/p&gt;&#xA;" OwnerUserId="4359" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-28T09:28:40.950" LastActivityDate="2011-04-28T15:08:58.207" Title="Where can I find name data for US census years other than 1990?" Tags="&lt;dataset&gt;&lt;census&gt;" AnswerCount="3" FavoriteCount="0" />
  <row Id="10075" PostTypeId="1" AcceptedAnswerId="10081" CreationDate="2011-04-28T04:51:05.910" Score="2" ViewCount="73" Body="&lt;p&gt;I'm teaching myself probability theory, and I'm not sure I understand any use for variance, as opposed to standard deviation. In the practice situations I'm looking at, the variance is larger than the range, so it doesn't seem intuitively useful.&lt;/p&gt;&#xA;" OwnerUserId="4361" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-28T07:58:18.980" LastActivityDate="2011-04-28T07:58:18.980" Title="What is the practical application of variance?" Tags="&lt;variance&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="0" />
  <row Id="10076" PostTypeId="2" ParentId="10075" CreationDate="2011-04-28T06:13:12.620" Score="0" ViewCount="" Body="&lt;p&gt;Variance is the most basic of the two measures... stddev = sqrt(variance).  While exaggerated, it's good enough for a comparison and grows very large when there is mixed-up-ness in the distribution.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;variance(22, 25, 29, 30, 37) = 32.3&#xA;variance(22, 25, 29, 30, 900) = 152611.0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Standard deviation is used way more often because the result has the same units as the data, making standard deviation more appropriate for any sort of visual analysis.&lt;/p&gt;&#xA;" OwnerUserId="1856" LastActivityDate="2011-04-28T06:13:12.620" />
  <row Id="10077" PostTypeId="2" ParentId="10074" CreationDate="2011-04-28T06:25:09.087" Score="0" ViewCount="" Body="&lt;p&gt;Try calling the census bureau using a help line number from census.gov. It usually takes a few tries and patience with bureaucracy, but they've always been helpful to me.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The census doesn't release individual names in their data sets, so it will have to be a census bureau created aggregation. &lt;/p&gt;&#xA;" OwnerUserId="4361" LastEditorUserId="4361" LastEditorDisplayName="" LastEditDate="2011-04-28T08:06:46.293" LastActivityDate="2011-04-28T08:06:46.293" />
  <row Id="10078" PostTypeId="1" CreationDate="2011-04-28T06:25:59.307" Score="0" ViewCount="121" Body="&lt;p&gt;For calculating Pearson Coefficient, i am using the below formula in MATLAB:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;C=cov(x,y);&#xA;p=C(2)/(std(x)*std(y));&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But C(2) is zero in my case if &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x = [1 0 1 0 0 0];&#xA;y = [1 1 1 1 1 1];&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Any idea where I am going wrong&lt;/p&gt;&#xA;" OwnerUserId="4341" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-28T07:07:03.987" LastActivityDate="2011-04-28T07:07:03.987" Title="Pearson correlation coefficient using MATLAB when standard deviation of one variable is zero " Tags="&lt;matlab&gt;" CommentCount="5" />
  <row Id="10079" PostTypeId="1" CreationDate="2011-04-28T06:40:32.977" Score="8" ViewCount="195" Body="&lt;p&gt;Within the context of a research proposal in the social sciences, I was asked the following question:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;I have always gone by 100 + m (where m&#xA;  is the number of predictors) when&#xA;  determining minimum sample size for&#xA;  multiple regression. Is this&#xA;  appropriate?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I get similar questions a lot, often with different rules of thumb.&#xA;I've also read such rules of thumbs quite a lot in various textbooks.&#xA;I sometimes wonder whether popularity of a rule in terms of citations is based on how low the standard is set.&#xA;However, I'm also aware of the value of good heuristics in simplifying decision making.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Questions:&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What is the utility of simple rules of thumb for minimum sample sizes within the context of applied researchers designing research studies?&lt;/li&gt;&#xA;&lt;li&gt;Would you suggest an alternative rule of thumb for minimum sample size for multiple regression?&lt;/li&gt;&#xA;&lt;li&gt;Alternatively, what alternative strategies would you suggest for determining minimum sample size for multiple regression? In particular, it would be good if value is assigned to the degree to which any strategy can readily be applied by a non-statistician.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-28T06:49:08.300" LastActivityDate="2011-04-28T19:01:45.053" Title="Rules of thumb for minimum sample size for multiple regression" Tags="&lt;regression&gt;&lt;sample-size&gt;&lt;power-analysis&gt;" AnswerCount="2" FavoriteCount="2" />
  <row Id="10080" PostTypeId="2" ParentId="9850" CreationDate="2011-04-28T06:42:26.660" Score="3" ViewCount="" Body="&lt;p&gt;Usually you'd plot the original values in a scatterplot (or a matrix of scatterplots if you have many of them) and use colour to show your groups. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You asked for an answer in python, and you actually do all the clustering and plotting with scipy, numpy and matplotlib:&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Start by making some data&lt;/h2&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;from scipy import cluster&#xA;from matplotlib import pyplot&#xA;&#xA;np.random.seed(123)&#xA;tests = np.reshape( np.random.uniform(0,100,60), (30,2) )&#xA;#tests[1:4]&#xA;#array([[ 22.68514536,  55.13147691],&#xA;#       [ 71.94689698,  42.31064601],&#xA;#       [ 98.07641984,  68.48297386]])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h2&gt;How many clusters?&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;This is the hard thing about k-means, and there are lots of methods. Let's use &lt;a href=&quot;http://en.wikipedia.org/wiki/Determining_the_number_of_clusters_in_a_data_set#The_Elbow_Method&quot; rel=&quot;nofollow&quot;&gt;the elbow method&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#plot variance for each value for 'k' between 1,10&#xA;initial = [cluster.vq.kmeans(tests,i) for i in range(1,10)]&#xA;pyplot.plot([var for (cent,var) in initial])&#xA;pyplot.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/4Y1Ah.png&quot; alt=&quot;Elbow plot&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Assign your observations to classes, and plot them&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I reckon index 3 (i.e. 4 clusters) is as good as any so&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cent, var = initial[3]&#xA;#use vq() to get as assignment for each obs.&#xA;assignment,cdist = cluster.vq.vq(tests,cent)&#xA;pyplot.scatter(tests[:,0], tests[:,1], c=assignment)&#xA;pyplot.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/3JT8o.png&quot; alt=&quot;scatter plot&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just work out where you can stick whatever you've already done into that workdflow (and I hope you clusters are a bit nicer than the random ones!)&lt;/p&gt;&#xA;" OwnerUserId="3732" LastActivityDate="2011-04-28T06:42:26.660" CommentCount="1" />
  <row Id="10081" PostTypeId="2" ParentId="10075" CreationDate="2011-04-28T06:49:29.953" Score="1" ViewCount="" Body="&lt;p&gt;In practice, you calculate the SD through calculating the variance (as abutcher indicated). I believe the variance is used more often (apart from interpretation, as you indicated yourself) because it has a lot of statistically interesting properties: it has unbiased estimators in a lot of cases, leads to known distributions for hypothesis testing etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As to the variance being bigger: if the variance were 1/4, the SD would be 1/2. As soon as your variance/SD are smaller than 1, this order reverses.&lt;/p&gt;&#xA;" OwnerUserId="4257" LastActivityDate="2011-04-28T06:49:29.953" />
  <row Id="10082" PostTypeId="2" ParentId="10058" CreationDate="2011-04-28T07:01:33.323" Score="3" ViewCount="" Body="&lt;p&gt;You indicate yourself that your measurements are not independent (you measure both species' abundance from the same locations). As such, you should correct for repeated measurements.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Try lmer from the lme4 package.&lt;/p&gt;&#xA;" OwnerUserId="4257" LastActivityDate="2011-04-28T07:01:33.323" />
  <row Id="10083" PostTypeId="2" ParentId="10066" CreationDate="2011-04-28T09:10:40.600" Score="1" ViewCount="" Body="&lt;p&gt;As it is not really difficult to import SAV dataset in R nowadays, with e.g.,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(foreign)&#xA;df &amp;lt;- read.spss(&quot;yourfilename&quot;, to.data.frame=TRUE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;you can check your SPSS results against one of the R packages that allow to perform CCA (see the CRAN Task View on &lt;a href=&quot;http://cran.r-project.org/web/views/Multivariate.html&quot; rel=&quot;nofollow&quot;&gt;Multivariate&lt;/a&gt; or &lt;a href=&quot;http://cran.r-project.org/web/views/Psychometrics.html&quot; rel=&quot;nofollow&quot;&gt;Psychometrics&lt;/a&gt; analysis). In particular, the &lt;a href=&quot;http://cran.r-project.org/web/packages/vegan/index.html&quot; rel=&quot;nofollow&quot;&gt;vegan&lt;/a&gt; package offers an handy way to apply CCA and has nice graphical and numerical summary through the &lt;code&gt;CCorA()&lt;/code&gt; function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, note that redundancy indexes apply onto one block of variables, conditional on the other block (hence the distinction you'll find in the aforementioned function between Y|X and X|Y); they are intended to provide a measure of the variance of one set of variables predicted from the linear combination of the other set of variables. However, in essence CCA consider that you have two sets of measures that play a symmetrical role. They are both descriptions of the same individuals or statistical units. If your blocks really play an assymmetric role--that is you have a block of predictors and a block of response variable--then you're better using &lt;a href=&quot;http://en.wikipedia.org/wiki/Partial_least_squares_regression&quot; rel=&quot;nofollow&quot;&gt;PLS regression&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-04-28T09:10:40.600" />
  <row Id="10084" PostTypeId="1" AcceptedAnswerId="10087" CreationDate="2011-04-28T09:30:46.753" Score="2" ViewCount="47" Body="&lt;p&gt;What's it called when I'm trying to remove the effect of a variable on another variable? Am I in the right ballpark using the terms &quot;detrending&quot; or &quot;normalizing&quot; or &quot;controlling for Y on X&quot;? What's the general process for doing this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem I'm working with currently is as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I work at a mine that loads trains that are 160 wagons in length. After the trains are filled, they leave to get unloaded and then later return to the mine. Sometimes the wagons end up in different positions in the train. Our aim is to maximise tonnes in every wagon. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We've noticed two things &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;some wagons IDs load consistently low in tonnes&lt;/li&gt;&#xA;&lt;li&gt;wagons in poisitions near the start or end of the train load consistently low. i.e. tonnes go down as distance from middle position goes up&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I've got a linear model fitted in R like&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    &amp;gt; lm(df$tonnes ~ df$dist_fr_middle)&#xA;    Coefficients:&#xA;        (Intercept)    df$dist_fr_middle  &#xA;        113.92001     -0.03915  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How do I go about creating a new column in my dataframe for distance-from-middle-adjusted tonnes?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for your help!&lt;/p&gt;&#xA;" OwnerUserId="827" LastActivityDate="2011-04-28T11:02:40.190" Title="Accounting for biases in the data (normalizing? detrending?)" Tags="&lt;regression&gt;&lt;normalization&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="2" />
  <row Id="10086" PostTypeId="1" CreationDate="2011-04-28T10:42:16.010" Score="1" ViewCount="36" Body="&lt;p&gt;I have 6 random number generators. They are &quot;black boxes&quot;, i.e. I do not know if they are the same or different. For example I do not know if they provide the same arithmetic averages and/or root mean square deviations. My goal is to check if they are the same or different. The problem is that I can generate a limited set of numbers (every random number generator gives me only 20 numbers).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The procedure that I use is as following.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;I choose a pair of generators.&lt;/li&gt;&#xA;&lt;li&gt;I use each of them to generate 20 numbers.&lt;/li&gt;&#xA;&lt;li&gt;For every generator I use these numbers to calculate the mean.&lt;/li&gt;&#xA;&lt;li&gt;Then I calculate the difference between the means (mean1 - mean2). I will call this difference as &quot;reference difference&quot;.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Now I want to know if this difference is &quot;real&quot;. In other words I want to know if the observed difference in means is caused by the fact that I do have two different random number generators or I have identical generators and the mean is different just because of chance (small number of numbers).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To answer the last question I use the following procedure.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;I combine the two sets of numbers (20 numbers + 20 numbers).&lt;/li&gt;&#xA;&lt;li&gt;Then I randomly split this set into two subsets of equal size.&lt;/li&gt;&#xA;&lt;li&gt;For each subset I calculate means and, as before, I calculate the difference between the two means (I will call this difference as &quot;test difference&quot;).&lt;/li&gt;&#xA;&lt;li&gt;I compare this test difference with the reference difference.&lt;/li&gt;&#xA;&lt;li&gt;I repeat this procedure many times and to see in home many cases the test difference is smaller then the reference difference.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I thought that if the reference difference is larger than the test difference in most of the cases than it is conditioned not by chance but by the fact that the compared random number generators are different. In contrast, if the generators are identical than the test difference has 50% of chances to be larger (or smaller) then the reference one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I see that for the most of the considered pairs of the generators the reference difference between the averages is smaller than the test differences. How could it be? &lt;strong&gt;It looks like the the generators are more similar than identical generators?&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="2407" LastActivityDate="2011-04-28T11:19:41.957" Title="How different random number generators can be more similar than identical ones?" Tags="&lt;hypothesis-testing&gt;&lt;mean&gt;&lt;random-generation&gt;&lt;hypothesis&gt;" AnswerCount="1" CommentCount="5" />
  <row Id="10087" PostTypeId="2" ParentId="10084" CreationDate="2011-04-28T11:02:40.190" Score="1" ViewCount="" Body="&lt;p&gt;The simple solution:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df$locationAdjustedTonnes = resid(lm(df$tonnes ~ df$dist_fr_middle))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;A general way is tweaking this (The example below gives the same result as the one above):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df$locationAdjustedTonnes = df$tonnes - predict(lm(df$tonnes ~ df$dist_fr_middle))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="3911" LastActivityDate="2011-04-28T11:02:40.190" CommentCount="4" />
  <row Id="10088" PostTypeId="1" CreationDate="2011-04-28T11:10:57.593" Score="0" ViewCount="111" Body="&lt;p&gt;For a multilabel dataset, i would like to find the number of clusters involved in it. The below example gives more details about the problem:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Label_A: feature values&#xA;Label_B: feature values&#xA;Label_A, Label_C: feature values &#xA;Label_C: feature values ... etc&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;We have say $n$ datarecord. Label field may have single label/multilabel(as in the case of record 3). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to determine the number of cluster involved in the data. Assuming number of label as the number of cluster results in bad accuracy. This is because there may be case where single label can have multiple cluster. In this case, if we can find more cluster and assign two or more cluster to same label, we can increase the accuracy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hence, how do you find the number of cluster present in the multilabel data?&lt;/p&gt;&#xA;" OwnerUserId="4290" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-28T15:07:36.650" LastActivityDate="2011-04-28T19:12:57.390" Title="How to determine optimal number of clusters?" Tags="&lt;clustering&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="10089" PostTypeId="1" CreationDate="2011-04-28T11:18:50.257" Score="0" ViewCount="49" Body="&lt;p&gt;I have 3 arms in a trial. I want to compare results of a survey before during and on completion of a treatment. Data is not normally distributed. What test should I use?&lt;/p&gt;&#xA;" OwnerUserId="4367" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-28T12:52:54.167" LastActivityDate="2011-05-15T00:18:32.437" Title="How to handle multiple comparisons in a three-arm clinical trial?" Tags="&lt;multiple-comparisons&gt;&lt;survey&gt;&lt;clinical-trials&gt;" AnswerCount="1" CommentCount="4" />
  <row Id="10090" PostTypeId="2" ParentId="10086" CreationDate="2011-04-28T11:19:41.957" Score="1" ViewCount="" Body="&lt;p&gt;What you are doing is essentially the standard nonparametrical trick (a black box random generator can be seen as simply an unknown distribution, and your main hypothesis is that the distributions are the same). As such, it should work. However, you are using the difference of the means as a test statistic. Though it is not obvious to create an example, this could be very dependent on the shape of your unknown distributions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm guessing that using the Wilcoxon / Mann-Whitney type of test statistic will be less influenced by weird forms of the underlying distributions (but may also be less powerful).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regardless of the above: the idea should work (but is dependent on the fact that your samples are good representations of the true distributions). Maybe you should post some code?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a sidenote: you do not mention any particulars about these random number generators: are they continuous (over an interval?), discrete,... ? This may also be of influence, as your sample sizes are rather small.&lt;/p&gt;&#xA;" OwnerUserId="4257" LastActivityDate="2011-04-28T11:19:41.957" CommentCount="2" />
  <row Id="10091" PostTypeId="2" ParentId="10088" CreationDate="2011-04-28T11:40:10.407" Score="1" ViewCount="" Body="&lt;p&gt;You can convert the labels into features indicating if the label is present or not. After that you can use various clustering algorithms and their corresponding methods to find out the number of clusters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: I understood that your difficulty was handling the multiple labels and I suggested a solution for that. Your question did not mention that you wanted to use the k-means algorithm. The number of k-means clusters question has been answered here: &lt;a href=&quot;http://stats.stackexchange.com/questions/9016/how-to-define-number-of-clusters-in-k-means-clustering&quot;&gt;How to define number of clusters in K-means clustering?&lt;/a&gt;. For hierarchical clustering the answer is here: &lt;a href=&quot;http://stats.stackexchange.com/questions/3685/where-to-cut-a-dendrogram&quot;&gt;Where to cut a dendrogram?&lt;/a&gt;. But there are many other clustering methods available: &lt;a href=&quot;http://stats.stackexchange.com/questions/3713/choosing-clustering-method&quot;&gt;Choosing clustering method&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-28T19:12:57.390" LastActivityDate="2011-04-28T19:12:57.390" CommentCount="2" />
  <row Id="10092" PostTypeId="1" CreationDate="2011-04-28T11:54:46.047" Score="0" ViewCount="51" Body="&lt;p&gt;I am trying to understand Logistic Regression in relation to credit scoring model. I wish to understand the significance of &quot;20/ln(2)&quot; in logistic regression. Why and how is it used? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regards&lt;/p&gt;&#xA;" OwnerUserId="4368" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-28T12:45:10.103" LastActivityDate="2011-04-28T12:45:10.103" Title="What does &quot;20/ln(2)&quot; mean in logistic regression?" Tags="&lt;logistic-regression&gt;" CommentCount="2" />
  <row Id="10093" PostTypeId="2" ParentId="10079" CreationDate="2011-04-28T12:11:19.007" Score="3" ViewCount="" Body="&lt;p&gt;(+1) for indeed a crucial, in my opinion, question. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In macro-econometrics you usually have much smaller sample sizes than in micro, financial or sociological experiments. A researcher feels quite well when on can provide at least feasible estimations. My personal least possible rule of thumb is $4\cdot m$ ($4$ degrees of freedom on one estimated parameter). In other applied fields of studies you usually are more lucky with data (if it is not too expensive, just collect more data points) and you may ask what is the optimal size of a sample (not just minimum value for such). The latter issue comes from the fact that more low quality (noisy) data is not better than smaller sample of high quality ones.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most of the sample sizes are linked to the power of tests for the hypothesis you are going to test after you fit the multiple regression model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a nice &lt;a href=&quot;http://www.danielsoper.com/statcalc/calc01.aspx&quot; rel=&quot;nofollow&quot;&gt;calculator&lt;/a&gt; that could be useful for multiple regression models and some &lt;a href=&quot;http://www.danielsoper.com/statkb/topic01.aspx&quot; rel=&quot;nofollow&quot;&gt;formula&lt;/a&gt; behind the scenes. I think such a-priory calculator could be easily applied by non-statistician. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Probably K.Kelley and S.E.Maxwell &lt;a href=&quot;http://nd.edu/~kkelley/publications/articles/Kelley_Maxwell_2003.pdf&quot; rel=&quot;nofollow&quot;&gt;article&lt;/a&gt; may be useful to answer the other questions, but I need more time first to study the problem. &lt;/p&gt;&#xA;" OwnerUserId="2645" LastEditorUserId="2645" LastEditorDisplayName="" LastEditDate="2011-04-28T19:01:45.053" LastActivityDate="2011-04-28T19:01:45.053" />
  <row Id="10094" PostTypeId="1" CreationDate="2011-04-28T13:26:53.983" Score="3" ViewCount="40" Body="&lt;p&gt;I performed both a SVD decomposition and a multidimensional scaling of a 6-dimensional data matrix, in order to get a better understanding of the structure of the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unfortunately, all the singular values are of the same order, implying that the dimensionality of the data is indeed 6. However, I would like to be able to interpret the values of the singular vectors. For instance, the first one seems to be more or less equal in each dimension (ie &lt;code&gt;(1,1,1,1,1,1)&lt;/code&gt;), and the second also has an interesting structure (something like &lt;code&gt;(1,-1,1,-1,-1,1)&lt;/code&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How could I interpret these vectors? Could you point me to some literature on the subject?&lt;/p&gt;&#xA;" OwnerUserId="3699" LastActivityDate="2011-04-28T13:26:53.983" Title="How to interpret results of dimensionality reduction/multidimensional scaling?" Tags="&lt;pca&gt;&lt;interpretation&gt;&lt;dimensionality-reduction&gt;&lt;svd&gt;" />
  <row Id="10095" PostTypeId="1" CreationDate="2011-04-28T13:52:13.837" Score="3" ViewCount="127" Body="&lt;h3&gt;Assumptions:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;In an ANOVA where the normality assumptions are violated, the Box-Cox transformation can be applied to the response variable. The &lt;code&gt;lambda&lt;/code&gt; can be estimated by the using maximum likelihood to optimize the normality of the model residuals.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Question:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;When the estimates for &lt;code&gt;lambda&lt;/code&gt; in the null model and the full model differ, how should &lt;code&gt;lambda&lt;/code&gt; be estimated?&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;My Data:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;In my data the lambda estimate for the null model is &lt;code&gt;-2.3&lt;/code&gt; and the lambda estimate for the full model is &lt;code&gt;-2.8&lt;/code&gt;. Transforming the response using these different parameters and preforming the ANOVA leads to different F-statistics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have produced below a simplified version of the analysis using beta distributions with different parameters to simulate non-normality.  Unfortunately, in this example the results of the ANOVA are insensitive to the different estimates of &lt;code&gt;lambda&lt;/code&gt;. So, it doesn't fully illustrate the problem.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(ggplot2)&#xA;library(MASS)&#xA;library(car)&#xA;&#xA;&#xA;#Generating random beta-distributed data&#xA;n=200&#xA;df &amp;lt;- rbind(&#xA;  data.frame(x=factor(rep(&quot;a1&quot;,n)), y=rbeta(n,2,5)), # more left skewed&#xA;  data.frame(x=factor(rep(&quot;a2&quot;,n)), y=rbeta(n,2,2))) # less left skewed&#xA;&#xA;print(qplot(data=df, color=x, x=y, geom=&quot;density&quot;))&#xA;&#xA;print(&quot;Untransformed Analaysis of Variance:&quot;)&#xA;m.null &amp;lt;- lm(y ~ 1, df)&#xA;m.full &amp;lt;- lm(y ~ x, df)&#xA;print(anova(m.null, m.full))&#xA;&#xA;# Estimate Maximum Liklihood Box-Cox transform parameters for both models&#xA;bc.null &amp;lt;- boxcox(m.null); bc.null.opt &amp;lt;- bc.null$x[which.max(bc.null$y)]&#xA;bc.full &amp;lt;- boxcox(m.full); bc.full.opt &amp;lt;- bc.full$x[which.max(bc.full$y)]&#xA;&#xA;print(paste(&quot;ML Box-Cox estimate for null model:&quot;,bc.null.opt))&#xA;print(paste(&quot;ML Box-Cox estimate for full model:&quot;,bc.full.opt))&#xA;&#xA;df$y.bc.null &amp;lt;- bcPower(df$y, bc.null.opt)&#xA;df$y.bc.full &amp;lt;- bcPower(df$y, bc.full.opt)&#xA;&#xA;print(qplot(data=df, x=x, y=y.bc.null, geom=&quot;boxplot&quot;))&#xA;print(qplot(data=df, x=x, y=y.bc.full, geom=&quot;boxplot&quot;))&#xA;&#xA;&#xA;print(&quot;Analysis of Variance with optimial Box-Cox transform for null model&quot;)&#xA;m.bc_null.null &amp;lt;- lm(y.bc.null ~ 1, data=df)&#xA;m.bc_null.full &amp;lt;- lm(y.bc.null ~ x, data=df)&#xA;print(anova(m.bc_null.null, m.bc_null.full))&#xA;&#xA;print(&quot;Analysis of Variance with optimial Box-Cox transform for full model&quot;)&#xA;m.bc_full.null &amp;lt;- lm(y.bc.null ~ 1, data=df)&#xA;m.bc_full.full &amp;lt;- lm(y.bc.null ~ x, data=df)&#xA;print(anova(m.bc_full.null, m.bc_full.full))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="501" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-28T15:12:30.470" LastActivityDate="2011-04-28T16:16:40.647" Title="Estimating Lambda for Box Cox transformation for ANOVA" Tags="&lt;r&gt;&lt;anova&gt;&lt;data-transformation&gt;&lt;maximum-likelihood&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="10096" PostTypeId="1" CreationDate="2011-04-28T14:06:44.270" Score="2" ViewCount="56" Body="&lt;p&gt;I am currently monitoring the daily energy consumption of a house since January of this year. That daily consumption depends on several variables, including outdoor temperature, solar irradiance etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At regular intervals (once every two weeks) I am changing the algorithm of the heating controller, to see if I can improve the energy savings.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the methodology one should follow for testing whether such a change has any effect on a timeserie? I've found a short example in BHH2 (where they analyze the effect of a law on pollution levels) but I need a more thorough discussion.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a plot of the data I'm working with:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://farm4.static.flickr.com/3301/5694800429_70e184727c_d.jpg&quot; alt=&quot;Energy vs time&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This shows the daily energy consumption. The green line indicates when we introduced the change to the algorithm. As you can see, there already was a trend towards lower energy consumption even before we introduced the change, probably due to outdoor temperatures getting warmer. What I need to know is whether the introduction of the change has significantly accentuated that trend or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note also that since that time, we have collected much more data than what is shown on this plot.&lt;/p&gt;&#xA;" OwnerUserId="4370" LastEditorUserId="26" LastEditorDisplayName="" LastEditDate="2011-05-07T05:47:24.217" LastActivityDate="2011-05-07T05:47:24.217" Title="How to test the effect of various manipulations of an independent variable on a dependent variable in a timeseries?" Tags="&lt;time-series&gt;&lt;intervention-analysis&gt;" AnswerCount="1" CommentCount="5" />
  <row Id="10097" PostTypeId="2" ParentId="10074" CreationDate="2011-04-28T14:16:49.660" Score="1" ViewCount="" Body="&lt;p&gt;In &lt;a href=&quot;http://www.mapleprimes.com/posts/38183-Baby-Names&quot; rel=&quot;nofollow&quot;&gt;this post&lt;/a&gt;, my friend John May links to &lt;a href=&quot;http://www.ssa.gov/OACT/babynames/&quot; rel=&quot;nofollow&quot;&gt;http://www.ssa.gov/OACT/babynames/&lt;/a&gt;; I believe it only has the top 1000 most popular first names for both sexes, but for all years since 1879. (John's post &lt;a href=&quot;http://www.mapleprimes.com/posts/38177-Baby-Names--Continued&quot; rel=&quot;nofollow&quot;&gt;and the followup&lt;/a&gt; are interesting in their own right - he looks at popular unisex baby names.)&lt;/p&gt;&#xA;" OwnerUserId="2898" LastActivityDate="2011-04-28T14:16:49.660" CommentCount="1" />
  <row Id="10098" PostTypeId="2" ParentId="10017" CreationDate="2011-04-28T15:08:21.680" Score="1" ViewCount="" Body="&lt;p&gt;After a lot of reading, I found the solution for doing clustering within the &lt;code&gt;lm&lt;/code&gt; framework.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There's an excellent &lt;a href=&quot;http://people.su.se/~ma/clustering.pdf&quot; rel=&quot;nofollow&quot;&gt;white paper&lt;/a&gt; by Mahmood Arai that provides a tutorial on clustering in the &lt;code&gt;lm&lt;/code&gt; framework, which he does with degrees-of-freedom corrections instead of my messy attempts above. He provides his functions for both one- and two-way clustering covariance matrices &lt;a href=&quot;http://people.su.se/~ma/clmclx.R&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, although the content isn't available free, &lt;a href=&quot;http://www.mostlyharmlesseconometrics.com/book-contents/&quot; rel=&quot;nofollow&quot;&gt;Angrist and Pischke's &lt;em&gt;Mostly Harmless Econometrics&lt;/em&gt;&lt;/a&gt; has a section on clustering that was very helpful. &lt;/p&gt;&#xA;" OwnerUserId="1445" LastActivityDate="2011-04-28T15:08:21.680" />
  <row Id="10099" PostTypeId="2" ParentId="10074" CreationDate="2011-04-28T15:08:58.207" Score="1" ViewCount="" Body="&lt;p&gt;A search for a name on &lt;a href=&quot;http://www.wolframalpha.com/input/?i=Eric&quot; rel=&quot;nofollow&quot;&gt;Wolfram|Alpha&lt;/a&gt; gives some stats and a distribution graph of ages.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At the bottom of the output is a 'Source information' link, which results in a popup with a bunch of references.  One is the ssa.gov babynames (link above).  Most of the references are for other countries (mostly English speaking), but the &lt;strong&gt;&lt;a href=&quot;http://users.erols.com/dgalbi/names/agnames.htm&quot; rel=&quot;nofollow&quot;&gt;AGNAMES&lt;/a&gt; link  looks really good&lt;/strong&gt;, with &quot;Popular Given Names US, 1801-1999&quot; data and some normalization code.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At the bottom of the Wolfram|Alpha sources popup is this statement:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&quot;Requests by researchers for detailed&#xA;  information on the sources for&#xA;  individual Wolfram|Alpha results can&#xA;  be directed here.&quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;where the popup has the actual link.  Seeing that Wolfram|Alpha output is all images of text and not text, I am not sure how open they are to sharing, but you could inquire.  &lt;/p&gt;&#xA;" OwnerUserId="4372" LastActivityDate="2011-04-28T15:08:58.207" CommentCount="1" />
  <row Id="10100" PostTypeId="1" AcceptedAnswerId="10118" CreationDate="2011-04-28T15:11:57.650" Score="2" ViewCount="37" Body="&lt;p&gt;Online marketers often need to choose a &quot;winning&quot; variation among, say, 2 possible variations.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Example:&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;PageVariation1 had 1000 sessions with 20 conversions (2.0% response rate) and &lt;/li&gt;&#xA;&lt;li&gt;PageVariation2 had 800 sessions with 8 conversions (1.0% response rate).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I'm using the &quot;unpooled&quot; &lt;a href=&quot;http://apcentral.collegeboard.com/apc/members/courses/teachers_corner/49013.html&quot; rel=&quot;nofollow&quot;&gt;Two-Proportion z-Test&lt;/a&gt; for estimating the P-value. In the example above it is 0.9615 - so I might conclude that there's a 96% that PageVariation1 has a higher response rate. Let's say it's good enough to declare it as &quot;winner&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Problem is that some sessions and conversions weren't tracked to the PageVariation level - i.e. I know there were, say, additional 200 sessions and 3 conversions, but those can't be allocated to any of the above. Intuitively it means that there is higher uncertainty due to tracking/measurement issues. Consequently the P-value should be lower.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Question:&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Which test should be used (and how) in order to calculate the new P-value?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="4374" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-28T22:51:49.780" LastActivityDate="2011-04-29T05:13:18.330" Title="How to embed measurement uncertainty in a two-proportion z-test?" Tags="&lt;missing-data&gt;&lt;proportion&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="10101" PostTypeId="2" ParentId="10053" CreationDate="2011-04-28T15:14:00.030" Score="2" ViewCount="" Body="&lt;p&gt;Would you be happier with a surgically altered goodfit object?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; idx &amp;lt;- which(test.gof$observed != 0)&#xA;&amp;gt; idx&#xA; [1] 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 49 50&#xA;&amp;gt; test.gof$par$size &amp;lt;- length(  idx-1)&#xA;&amp;gt; test.gof$fitted &amp;lt;- test.gof$fitted[idx]&#xA;&amp;gt; test.gof$count &amp;lt;- test.gof$count[idx]&#xA;&amp;gt; test.gof$observed &amp;lt;- test.gof$observed[idx]&#xA;&amp;gt; summary(test.gof)&#xA;&#xA;     Goodness-of-fit test for binomial distribution&#xA;&#xA;                      X^2 df  P(&amp;gt; X^2)&#xA;Pearson               Inf 75 0.0000000&#xA;Likelihood Ratio 21.48322 19 0.3107244&#xA;Warning message:&#xA;In summary.goodfit(test.gof) : Chi-squared approximation may be incorrect&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2129" LastActivityDate="2011-04-28T15:14:00.030" CommentCount="1" />
  <row Id="10102" PostTypeId="2" ParentId="10100" CreationDate="2011-04-28T15:33:56.817" Score="2" ViewCount="" Body="&lt;p&gt;A few quick points:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;You could draw on the general literature on missing data.&lt;/li&gt;&#xA;&lt;li&gt;If the 200 sessions with missing data are known to be completely random, then you could choose to ignore these observations and just analyse the data where you have complete data.&lt;/li&gt;&#xA;&lt;li&gt;In general it is important to think through what mechanisms might explain the missing data.&lt;/li&gt;&#xA;&lt;li&gt;You may find it useful to read up on strategies for dealing with &lt;a href=&quot;http://www.stat.psu.edu/~jls/missing_data/&quot; rel=&quot;nofollow&quot;&gt;missing data&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-29T05:13:18.330" LastActivityDate="2011-04-29T05:13:18.330" />
  <row Id="10103" PostTypeId="2" ParentId="10031" CreationDate="2011-04-28T15:52:05.907" Score="0" ViewCount="" Body="&lt;p&gt;If a potential outcome has never been observed, then you have no information about the effect of covariates on it. So any outcome-specific covariate effects are unidentifiable. If you assume constant covariate effects, then this outcome has no effect on their estimation, so you might as well omit it from the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More generally, you do have some information about the frequency of that outcome, but logistic regression (multinomial or otherwise) cannot handle this at all, because if you insist on having this outcome, the corresponding intercept has to be $-\infty$.&lt;/p&gt;&#xA;" OwnerUserId="279" LastActivityDate="2011-04-28T15:52:05.907" />
  <row Id="10104" PostTypeId="1" CreationDate="2011-04-28T15:57:52.087" Score="4" ViewCount="55" Body="&lt;p&gt;Because all clinical and laboratory data are continuous random variables larger than zero (i.e. they are bounded above zero), is it impossible for them to be normally distributed? Can they only be log-normally distributed?&#xA;Thank you.&lt;/p&gt;&#xA;" OwnerUserId="4349" LastActivityDate="2011-04-28T18:23:22.037" Title="Is it impossible for clinical and laboratory data to be normally distributed?" Tags="&lt;distributions&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="1" />
  <row Id="10105" PostTypeId="2" ParentId="10079" CreationDate="2011-04-28T16:03:22.203" Score="4" ViewCount="" Body="&lt;p&gt;I'm not a fan of simple formulas for generating minimum sample sizes.&#xA;At the very least, any formula should consider effect size and the questions of interest.&#xA;And the difference between either side of a cut-off is minimal.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Sample size as optimisation problem&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Bigger samples are better.&lt;/li&gt;&#xA;&lt;li&gt;Sample size is often determined by pragmatic considerations.&lt;/li&gt;&#xA;&lt;li&gt;Sample size should be seen as one consideration in an optimisation problem where the cost in time, money, effort, and so on of obtaining additional participants is weighed against the benefits of having additional participants.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;A Rough Rule of Thumb&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;In terms of very rough rules of thumb within the typical context of observational psychological studies involving things like ability tests, attitude scales, personality measures, and so forth, I sometimes think of:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;n=100 as adequate &lt;/li&gt;&#xA;&lt;li&gt;n=200 as good &lt;/li&gt;&#xA;&lt;li&gt;n=400+ as great&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;These rules of thumb are grounded in the 95% confidence intervals associated with correlations at these respective levels and the degree of precision that I'd like to theoretically understand the relations of interest.&#xA;However, it is only a heuristic.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;G Power 3&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;I typically use G-Power 3 to calculate power based on various assumptions&#xA;&lt;a href=&quot;http://jeromyanglim.blogspot.com/2010/05/statistical-power-analysis-in-gpower-3.html&quot; rel=&quot;nofollow&quot;&gt;see my post&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;li&gt;See this &lt;a href=&quot;http://www.psycho.uni-duesseldorf.de/abteilungen/aap/gpower3/user-guide-by-distribution/f/linear_multiple_regression_r2_deviation_from_zweo&quot; rel=&quot;nofollow&quot;&gt;tutorial from the G Power 3 site specific to multiple regression&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;The &lt;a href=&quot;http://www.education.wisc.edu/elpa/academics/syllabi/2006/06Spring/825Borman/Cohen1992.pdf&quot; rel=&quot;nofollow&quot;&gt;Power Primer&lt;/a&gt; is also a useful tool for applied researchers.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;Multiple Regression tests multiple hypotheses&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Any power analysis question requires consideration of effect sizes.&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Power analysis for multiple regression is made more complicated by the fact that there are multiple effects including the overall r-squared and one for each individual coefficient.&#xA;Furthermore, most studies include more than one multiple regression.&#xA;For me, this is further reason to rely more on general heuristics, and thinking about the minimal effect size that you want to detect.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;In relation to multiple regression, I'll often think more in terms of the degree of precision in estimating the underlying correlation matrix.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;Accuracy in Parameter Estimation&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;I also like Ken Kelley and colleagues' discussion of Accuracy in Parameter Estimation.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;See &lt;a href=&quot;http://nd.edu/~kkelley/publications/articles/Kelley_Maxwell_2003.pdf&quot; rel=&quot;nofollow&quot;&gt;Ken Kelley's website&lt;/a&gt; for publications&lt;/li&gt;&#xA;&lt;li&gt;As mentioned by @Dmitrij, Kelley and Maxwell (2003) &lt;a href=&quot;http://nd.edu/~kkelley/publications/articles/Kelley_Maxwell_2003.pdf&quot; rel=&quot;nofollow&quot;&gt;FREE PDF&lt;/a&gt; have a useful article.&lt;/li&gt;&#xA;&lt;li&gt;Ken Kelley developed the &lt;code&gt;MBESS&lt;/code&gt; package in R to perform analyses relating sample size to precision in parameter estimation.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastActivityDate="2011-04-28T16:03:22.203" CommentCount="1" />
  <row Id="10106" PostTypeId="2" ParentId="10095" CreationDate="2011-04-28T16:04:21.660" Score="2" ViewCount="" Body="&lt;p&gt;The Box-Cox transformation tries to improve the normality of the residuals. Since that is the assumption of ANOVA as well, you should run it on the model that you are actually going to use, i.e. the full model. For example, if you have two well separated groups, the distribution of the response variable will be strongly bimodal and nowhere near normal even if within each group the distribution is normal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Additionally, you certainly want to take whuber's comment to heart, and check for outliers, missing predictors, etc to make sure that some artifact is not driving your transformation. Also consider the confidence interval around the optimal lambda, and whether a particular transformation within that interval does make applied sense. For example, if you have linear measurements, but the outcome would reasonably be related to a volume, then a lambda=3 or lambda=-3 might be meaningful. If, on the other hand, areas are involved, then 2 or -2 might be better choices.&lt;/p&gt;&#xA;" OwnerUserId="279" LastActivityDate="2011-04-28T16:04:21.660" CommentCount="1" />
  <row Id="10107" PostTypeId="2" ParentId="10104" CreationDate="2011-04-28T16:07:58.037" Score="5" ViewCount="" Body="&lt;p&gt;You are correct, a positive-only value cannot be &lt;em&gt;really&lt;/em&gt; normally distributed. However depending on the particular circumstances, its distribution could be sufficiently close to normal for statistical methodology assuming normality to apply. I want to note that very few methods actually require normality of the &lt;em&gt;data&lt;/em&gt;, often only model residuals, or sampling distributions have to be (approximately) normally distributed.&lt;/p&gt;&#xA;" OwnerUserId="279" LastActivityDate="2011-04-28T16:07:58.037" />
  <row Id="10108" PostTypeId="2" ParentId="10095" CreationDate="2011-04-28T16:16:40.647" Score="2" ViewCount="" Body="&lt;p&gt;It is not appropriate to do ordinary ANOVA after using the same dataset to fit lambda.  The analysis should be unified, penalizing for uncertainty in lambda (a parameter to be estimated, and included in the covariance matrix).&lt;/p&gt;&#xA;" OwnerUserId="4253" LastActivityDate="2011-04-28T16:16:40.647" />
  <row Id="10109" PostTypeId="1" CreationDate="2011-04-28T16:51:00.163" Score="0" ViewCount="64" Body="&lt;p&gt;I am trying to use R to develop a corporate financial model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The model includes various line items, X, of the following form with actual values for time period 1, 2.. n and projected values for periods n+1, n+2,.. n+k. g is the average growth rate for the forecast period.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to construct a vector of the following form:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  X=c(X1,X2,...,Xn,Xn+1=(1+g)Xn,Xn+2=(1+g)Xn+1,...,Xn+k=(1+g)Xn+k-1))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How would I do this in R?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have tried looking up the R literature on lagged variables but could not find a simple example which does what is required. I look forward to any guidance that can be given.&lt;/p&gt;&#xA;" OwnerUserId="4375" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-28T18:12:11.967" LastActivityDate="2011-04-28T18:47:02.267" Title="Lagged Variables in R" Tags="&lt;r&gt;&lt;time-series&gt;&lt;forecasting&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="10110" PostTypeId="1" CreationDate="2011-04-28T16:59:06.690" Score="2" ViewCount="173" Body="&lt;p&gt;100 periods have been collected from a 3 dimensional periodic signal. The wavelength slightly varies. The noise of the wavelength follows Gaussian distribution with zero mean. A good estimate of the wavelength is known, that is not an issue here. The noise of the amplitude may not be Gaussian and may be contaminated with outliers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How can I compute a single period that approximates 'best' all of the collected 100 periods?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have no idea how time-series models work. Are they prepared for varying wavelengths? Can they handle non-smooth true signals? If a time-series model is fitted, can I compute a 'best estimate' for a single period? How?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A related question is &lt;a href=&quot;http://stackoverflow.com/q/2572444/341970&quot;&gt;this&lt;/a&gt;. Speed is not an issue in my case. Processing is done off-line, after all periods have been collected.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Origin of the problem:&lt;/strong&gt; I am measuring acceleration during human steps at 200 Hz. After that I am trying to double integrate the data to get the vertical displacement of the center of gravity. Of course the noise introduces a HUGE error when you integrate twice. I would like to exploit periodicity to reduce this noise. Here is a crude graph of the actual data (y: acceleration in g, x: time in second) of 6 steps corresponding to 3 periods (1 left and 1 right step is a period):&#xA;&lt;img src=&quot;http://i.imgur.com/44q8d.png&quot; alt=&quot;Human steps&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My interest is now purely theoretical, as &lt;a href=&quot;http://jap.physiology.org/content/39/1/174.abstract&quot; rel=&quot;nofollow&quot;&gt;http://jap.physiology.org/content/39/1/174.abstract&lt;/a&gt; gives a pretty good recipe what to do. It does not address periodicity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note:&lt;/strong&gt; I have asked &lt;a href=&quot;http://stackoverflow.com/q/5702974/341970&quot;&gt;this question on stackoverflow&lt;/a&gt; but it seems to be off-topic there.&lt;/p&gt;&#xA;" OwnerUserId="4240" LastActivityDate="2011-05-01T23:30:01.290" Title="How to exploit periodicity to reduce noise of a signal?" Tags="&lt;r&gt;&lt;time-series&gt;&lt;autocorrelation&gt;&lt;arima&gt;&lt;kalman-filter&gt;" AnswerCount="3" />
  <row Id="10111" PostTypeId="1" AcceptedAnswerId="11195" CreationDate="2011-04-28T17:14:11.820" Score="3" ViewCount="110" Body="&lt;p&gt;I would like to compute the p-value of my two-way ANOVA. The score that I am using to detect significant samples is the eta score which is computed as SS(between)/SS(total).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In many sites I saw that F=Var(between)/Var(within). &#xA;I want to know if I can consider the eta score as the F-statistic value, and then compute the p-value? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What if I want to compute the p-value for other scores such as eta-partial, or omega?&lt;br&gt;&#xA;Is it meaningful if I calculate the p-value for them? Or is it only meaningful for the ratio Var(between)/Var(within)? What difference does it make if I use SS(between)/SS(within) instead of Var?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The formula that I am using to calculate p-value is: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pvalue=-log10(betai(0.5 * df2, 0.5* df1, df2 / (df2 + df1 * eta)));  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;based on the book &lt;em&gt;Numerical Recipes in C&lt;/em&gt;, where df1 and df2 are degrees of freedom. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for your help.  &lt;/p&gt;&#xA;" OwnerUserId="2885" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-04-28T18:34:07.647" LastActivityDate="2011-05-24T14:34:40.337" Title="Calculating p-value for a two-way ANOVA" Tags="&lt;anova&gt;&lt;p-values&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="10112" PostTypeId="2" ParentId="10110" CreationDate="2011-04-28T17:59:25.397" Score="3" ViewCount="" Body="&lt;p&gt;I'm not sure I totally understand what would be best, but I believe that looking at Dynamic Linear Models, State Space modeling, and the Kalman Filter would be helpful. In R, the package &lt;code&gt;dlm&lt;/code&gt; is fairly accessible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: Describe DLM's simply, eh? One explanation I've seen is that DLMs are regression where your coefficients are allowed to change with time. Doesn't really give me much intuition.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, here is my I-realize-I-don't-understand-DLMs-well-enough-to-do-this-well-but-you-asked answer, which I hope others will correct as necessary. Let me use a situation that's similar to how they were invented...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say you were controlling a remotely-piloted vehicle. You could summarize the vehicle's actual state (speed, direction, altitude, fuel, etc, etc) as a vector in a state space. You can't directly observe the actual state, but you do have sensors that observe linear combinations of the actual state, with some (gaussian) noise added in. That is, your sensors are not perfect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Further, the vehicle has a program that determines how it changes states, as a linear combination of the current state, also with some (gaussian) noise. That is, your controls are not perfect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say you want to know the most likely and most accurate path possible for the vehicle -- that is, the states it actually went through: how it actually moved. What you observe is noisy, but you can use a Kalman filter on this system you model and the filter models the variances in the system and outputs the most likely actual states. At each time step, it forecasts what it believes the state will be, then observes what the sensors say, calculates the variances of the different parts, and comes up with a final estimate of the state as a weighted sum of the forecast and the sensor readings. Then the process is repeated for the next time step.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The whole concept is basically a continuous Hidden Markov Model, if you're familiar with those. The model is several equations which involve several matrices that are multiplied together, and you can add columns to the matrix to reflect trend, seasonal, and other types of components of the time series.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The R package &lt;code&gt;dlm&lt;/code&gt; makes it particularly easy to define and combine components.&lt;/p&gt;&#xA;" OwnerUserId="1764" LastEditorUserId="1764" LastEditorDisplayName="" LastEditDate="2011-04-30T22:52:56.120" LastActivityDate="2011-04-30T22:52:56.120" CommentCount="5" />
  <row Id="10113" PostTypeId="2" ParentId="9850" CreationDate="2011-04-28T18:06:58.480" Score="0" ViewCount="" Body="&lt;p&gt;I'm not a python expert, but it is extremely helpful to plot the 1st 2 principal components against each other on the x,y axes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Not sure which packages you are using, but here is a sample link:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://pyrorobotics.org/?page=PyroModuleAnalysis&quot; rel=&quot;nofollow&quot;&gt;http://pyrorobotics.org/?page=PyroModuleAnalysis&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="3489" LastActivityDate="2011-04-28T18:06:58.480" CommentCount="2" />
  <row Id="10114" PostTypeId="2" ParentId="10104" CreationDate="2011-04-28T18:23:22.037" Score="0" ViewCount="" Body="&lt;p&gt;Many biological measurements take positive values only and to such variables applies the rule of thumb: &lt;em&gt;If $\text{mean} &amp;lt; 2\cdot\text{SD}$ then the distribution is not normal&lt;/em&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Explanation: for normally distributed values the majority of values are in the interval $[\text{mean}\pm2\cdot\text{SD}]$, and a normal distribution with $\text{mean} &amp;lt; 2\cdot\text{SD}$ would have a too large proportion of negative values, the normal approximation is poor. Of course be cautious like with any other rules of thumb.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastActivityDate="2011-04-28T18:23:22.037" />
  <row Id="10115" PostTypeId="1" CreationDate="2011-04-28T18:26:07.847" Score="0" ViewCount="64" Body="&lt;p&gt;Describe how a domestic fast food chain with plans for expanding into China would be able to use a forecasting model. &lt;/p&gt;&#xA;" OwnerUserId="4377" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-28T19:00:33.907" LastActivityDate="2011-04-28T19:00:33.907" ClosedDate="2011-04-29T08:55:47.810" Title="Describe how a domestic fast food chain with plans for expanding into China would be able to use a forecasting model." Tags="&lt;forecasting&gt;" CommentCount="2" />
  <row Id="10116" PostTypeId="2" ParentId="10109" CreationDate="2011-04-28T18:47:02.267" Score="2" ViewCount="" Body="&lt;p&gt;Does this do what you want?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x1 &amp;lt;- 1:10&#xA;g &amp;lt;- .1&#xA;x2 &amp;lt;- cumprod(c(tail(x1, 1), rep(1+g, 5)))&#xA;c(x1, x2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="3542" LastActivityDate="2011-04-28T18:47:02.267" />
  <row Id="10117" PostTypeId="1" CreationDate="2011-04-28T19:57:29.240" Score="2" ViewCount="93" Body="&lt;p&gt;I sincerely apologize if there is another thread already that will answer this question. I'm so incredibly out of my league here that I don't even know what keywords to search for :-).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm a computer programmer by trade, and while I have a basic background in math, statistics was never really my cup of tea. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I currently work at a school and just finished developing a basic set of tools to help automatically collect and analyze data on our student's behaviors (this is a school for children with autism and other disabilities). So, we have a couple of year's worth of data for things like: given Billy, how frequently did he have Aggressions, Self-Injurious Behaviors, Drop, etc. Probably 6 - 10 &quot;inputs&quot; (I think that's the correct term) per student. We'll be adding more as well in the future.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What I'm curious about is this:&lt;/strong&gt; Are there any beginning tutorials out there that might show me some interesting things to do with this data (besides just graphing it?) For example, it would be interesting to be able to predict when Billy is likely to have a long string of aggressions given that these &lt;code&gt;x&lt;/code&gt; other factors have been increasing lately. Or, there is an increasing trend of this behavior which is way out of whack with its previous values, that should raise a big red flag. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've been doing some basic Googling and this seems to be in the realm of &quot;Statistical Data Mining&quot;&amp;mdash;some brief tutorials were found on &lt;a href=&quot;http://www.autonlab.org/tutorials/&quot; rel=&quot;nofollow&quot;&gt;Andrew Moore's site&lt;/a&gt;, but these just aren't detailed enough for me to really learn anything. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I realize that this is akin to someone walking into Stack Overflow and saying &quot;Hey, tell me how to write the next Facebook.&quot; So, if these are the sorts of things that I can only do with years and years of statistical experience, just let me know and I'll be on my way. However, I also know that while someone couldn't walk into SO and write the next Facebook in a few weeks, we could probably point them in the right direction to create a basic site for their dad's business, even if it would be a pretty basic site. Likewise, I'm not looking to create a genius AI capable of predicting student behavior down to the millisecond; rather, I'm just curious if there's any low-hanging fruit that a guy like me could pick up in a few weeks or months of diligent reading that might make for some interesting uses of this new data we've unlocked.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;I'm open to online tutorials, books, textbooks, videos, open source programs and libraries, etc.&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="4379" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-04-28T22:55:58.053" LastActivityDate="2011-04-29T00:08:49.870" Title="Beginner to prediction/statistics: Where do I start?" Tags="&lt;forecasting&gt;&lt;longitudinal&gt;" AnswerCount="4" CommentCount="4" FavoriteCount="4" />
  <row Id="10118" PostTypeId="2" ParentId="10100" CreationDate="2011-04-28T20:53:13.500" Score="2" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;It can help to survey the scene&lt;/strong&gt;: that is, to understand the consequences for your decision of the possible states of affairs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The missing information can be described by two integers: $m$, the number of additional sessions for &quot;PageVariation2&quot;, and $k$, the number of additional conversions for PageVariation2.  For each possible combination of $m$ and $k$ ($k = 0, 1, 2, 3$ and $0 \le m \le 200-k$) we can compute what the p-value would be.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/TxsNA.png&quot; alt=&quot;Plot of p-values vs. m by k&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this plot the graph of the p-value versus $m$ for $k=3$ is shown in blue (the highest one), for $k=2$ in red, for $k=1$ in orange, and for $k=0$ in green.  For reference, a (two-sided) p-value of 0.05 is shown with a black line.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is evident that the p-values can range tremendously from indicating strong significance (for $m=197$ and $k=0$) to indicating no significance at all (for small $m$ and $k = 1, 2, 3$).  This tells us that &lt;strong&gt;your assumptions about the missingness completely determine the significance or lack of significance of the results.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you think it highly likely that $m$ is large--so that the majority of missing results are for PageVariation2--&lt;em&gt;and&lt;/em&gt; that $k$ is small--so that the majority of missing conversions are for PageVariation1--then you can conclude the differences in conversion rates are not likely due to chance and proceed accordingly.  If you do not have sufficient information to justify these assumptions, then the data do not provide strong evidence for a significant difference.  That is, what you have observed could be the result of random variation rather than due to some intrinsic difference between the two variations.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-04-28T20:53:13.500" />
  <row Id="10119" PostTypeId="2" ParentId="10117" CreationDate="2011-04-28T21:06:43.127" Score="0" ViewCount="" Body="&lt;p&gt;I understand that you want to model longitudinal data. A starting point can be this tutorial &lt;a href=&quot;http://www.ats.ucla.edu/stat/r/examples/alda/ch2.htm&quot; rel=&quot;nofollow&quot;&gt;http://www.ats.ucla.edu/stat/r/examples/alda/ch2.htm&lt;/a&gt;, it uses the open source R statistical package.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastActivityDate="2011-04-28T21:06:43.127" />
  <row Id="10120" PostTypeId="2" ParentId="10117" CreationDate="2011-04-28T21:23:38.963" Score="1" ViewCount="" Body="&lt;p&gt;&quot;anomaly detection&quot; -- This is usually called detection of outliers. You can find many references via googling.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;can more aggressions predict more self injurious behaviors&quot; -- You can try one of the basic things: correlation between different variables or features (you called them &quot;inputs&quot;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data will be easier to analyze if they will be in the format, say, &quot;number of xxx incidents per week&quot;, i.e., if your variables will be measured on the same timescale.&lt;/p&gt;&#xA;" OwnerUserId="4337" LastEditorUserId="4337" LastEditorDisplayName="" LastEditDate="2011-04-28T21:31:47.557" LastActivityDate="2011-04-28T21:31:47.557" />
  <row Id="10121" PostTypeId="1" CreationDate="2011-04-28T22:18:23.307" Score="8" ViewCount="136" Body="&lt;p&gt;I would like to generate a random correlation matrix such that the distribution of the entries (except 1) look approximately like normal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The motivation is this. For a set of n time series data, the correlation distribution often look quite close to normal. I would like to generate many &quot;normal&quot; correlation matrices to represent the general siutation and use them to calculate risk number. Do you think I answer your question?&lt;/p&gt;&#xA;" OwnerUserId="4383" LastEditorUserId="4383" LastEditorDisplayName="" LastEditDate="2011-05-07T10:10:05.410" LastActivityDate="2011-05-07T10:10:05.410" Title="Generate random correlation matrix that has approximately normal entries" Tags="&lt;correlation&gt;&lt;normal-distribution&gt;&lt;matrix&gt;" AnswerCount="2" CommentCount="28" />
  <row Id="10122" PostTypeId="1" CreationDate="2011-04-28T22:22:09.967" Score="2" ViewCount="44" Body="&lt;p&gt;Hi everyone,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm looking for a clustering implementation with the following features:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Support for high-dimensional data. Now I have approximately 160.000 dimensions/features.&lt;/li&gt;&#xA;&lt;li&gt;Be able to manage sparse matrix. That is, not only to read sparse matrices, but also capable of making operations in this format.&lt;/li&gt;&#xA;&lt;li&gt;Properly shows the centroid for each cluster.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I've tested some packages:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Rapidminer, which seems to be a memory eater, I suppose because although capable of reading a sparse matrix, it is not capable of working with them as they are. &lt;/li&gt;&#xA;&lt;li&gt;Cluto, which is very fast and low-memory consumption, but it is not able of show properly the centroid elements (source code not available). It shows descriptive features together with a percentage of how that feature contributes to the average similarity, but there is no clear info (&lt;a href=&quot;http://glaros.dtc.umn.edu/gkhome/node/262&quot; rel=&quot;nofollow&quot;&gt;here is a question about that, with no clear answer&lt;/a&gt;) about how is calculated that, and also I have clusters where there is 0.0% but it is not clear for me if this means the program is unable to show an upper precision or if that feature has nothing to do tho the average similarity.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I appreciate any comment or answer about it.&lt;/p&gt;&#xA;" OwnerUserId="4382" LastEditorUserId="4382" LastEditorDisplayName="" LastEditDate="2011-04-30T05:38:51.300" LastActivityDate="2011-04-30T05:38:51.300" Title="Looking for sparse and high-dimensional clustering implementation" Tags="&lt;clustering&gt;&lt;algorithms&gt;&lt;large-data&gt;" FavoriteCount="1" />
  <row Id="10123" PostTypeId="2" ParentId="1737" CreationDate="2011-04-28T22:36:25.187" Score="0" ViewCount="" Body="&lt;p&gt;I personally like &lt;code&gt;cast()&lt;/code&gt;, from the reshape package because of it's simplicity: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(reshape)&#xA;cast(melt(tips), sex ~ smoker | variable, c(sd,mean, length))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="776" LastActivityDate="2011-04-28T22:36:25.187" />
  <row Id="10124" PostTypeId="2" ParentId="10117" CreationDate="2011-04-28T22:38:17.013" Score="1" ViewCount="" Body="&lt;p&gt;It sounds like you have some really wonderful data to work with! One person suggested trying analyses in R, and that's definitely a powerful option. With your background in programming, it may be well-suited for you. I personally prefer a program like SPSS, which is built specifically user-friendly(ish) analysis of social science data. If you're new to the program, I'd suggest Julie Pallant's &quot;SPSS Survival Manual,&quot; which has basic how-to instructions for most common analyses. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regardless of the software, it sounds like using correlations, regressions, and some time-series work could help you investigate your variables. If it seems overwhelming to learn all the stats in a short period of time, I might suggest advertising the fact that you have data to work with. I'm certain that psychology undergrads or grad students at a nearby university would jump at the chance to help you do analysis and possibly publish any useful results. Best of luck!&lt;/p&gt;&#xA;" OwnerUserId="4384" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-04-28T22:52:08.073" LastActivityDate="2011-04-28T22:52:08.073" CommentCount="1" />
  <row Id="10125" PostTypeId="2" ParentId="10053" CreationDate="2011-04-28T22:44:44.650" Score="0" ViewCount="" Body="&lt;p&gt;Try plotting it. You'll get a better idea of what's going on. As mentioned before, you're getting NaN because you're passing 0 frequencies to chisq.test()&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;test.gof &amp;lt;- goodfit(counts, type=&quot;binomial&quot;, par=list(size=length(counts), prob=0.5)) &#xA;plot(test.gof)&#xA;## doesn't look so good &#xA;test.gof &amp;lt;- goodfit(counts, type=&quot;binomial&quot;, par=list(size=length(counts))) &#xA;plot(test.gof)&#xA;## looks a little more clear&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="776" LastActivityDate="2011-04-28T22:44:44.650" />
  <row Id="10126" PostTypeId="1" AcceptedAnswerId="10132" CreationDate="2011-04-28T22:57:24.707" Score="3" ViewCount="111" Body="&lt;p&gt;I ran a GEE model, with a dependent variable of &quot;percent of total students with an unexcused absence,&quot; using a binomial family. My dependent variable is basically a proportion, with range from 0 to 1 to obey the rules of the binomial family.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand that the difference between the non-event/event 0-1 is &quot;zero percent of students with an unexcused absence&quot; and &quot;100% of students with an unexcused absence.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Questions:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;How can I interpret the Odds Ratio for this in a way that makes policy-level, real-world sense?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;An interpretation of 1-unit change in independent variable X would be e.g. &quot;a school with a 1-unit increase in X has 0.9 times the odds of having 100% of students unexcused from class at least once, compared to a school without the increase in X.&quot; Seems clunky!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may ask why I used a binomial family rather than a Gaussian family, but according to &quot;Comparison of Logistic Regression and Linear Regression in Modeling Percentage Data&quot; by Zhao, Chen and Schaffner (2001), logistic regression can and should be used for any models with a dependent variable that's modeled as a percentage for various reasons.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If there's no other accurate way to interpret this coefficient through logistic regression and ORs, does anyone have any suggestions about how to model the data to be better interpretable?&lt;/p&gt;&#xA;" OwnerUserId="3309" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-28T23:18:41.013" LastActivityDate="2011-04-29T10:09:11.960" Title="Interpretation of odds ratio when outcome is a percentage" Tags="&lt;logistic-regression&gt;" AnswerCount="1" />
  <row Id="10127" PostTypeId="2" ParentId="10121" CreationDate="2011-04-28T23:12:04.403" Score="1" ViewCount="" Body="&lt;p&gt;You might be interested in some of the code at the following link:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://quant.stackexchange.com/questions/1027/correlation-and-cointegration-similarities-differences-relationships&quot;&gt;Correlation and Co-integration&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2775" LastActivityDate="2011-04-28T23:12:04.403" />
  <row Id="10128" PostTypeId="2" ParentId="6698" CreationDate="2011-04-28T23:29:00.073" Score="0" ViewCount="" Body="&lt;p&gt;Although this question has already been answered a useful way to remember for more general situations is the law of iterated expectations.  Note that the independence for prediction does not hold even if the &quot;true process&quot; is independent.  This is because the estimates are not independent, unless you have $Z^{T}Z$ and $Z_{new}Z_{new}^{T}$ both to be diagonal (&quot;new&quot; for the predictions)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So if you let $\hat{Y}_{ti}$ denote the estimated monthly values in year $t$ for month $i$, and $\hat{X}_{t}$ denote the estimated annual value, you have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\hat{X}_{t}=\sum_{i=1}^{12}\hat{Y}_{ti}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Var(\hat{X}_{t})=E[Var(\hat{X}_{t}|\hat{Y}_{t,1},\dots,\hat{Y}_{t,12})]+Var[E(\hat{X}_{t}|\hat{Y}_{t,1},\dots,\hat{Y}_{t,12})]$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(note sure if it should be an average or total, if average, then divide my final result for the standard error by $12$ and divide variance by $144$) Plugging one into the other we get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Var(\hat{X}_{t})=E[Var(\sum_{i=1}^{12}\hat{Y}_{ti}|\hat{Y}_{t,1},\dots,\hat{Y}_{t,12})]+Var[E(\sum_{i=1}^{12}\hat{Y}_{ti}|\hat{Y}_{t,1},\dots,\hat{Y}_{t,12})]$$&#xA;$$=Var[\sum_{i=1}^{12}\hat{Y}_{ti}]=\sum_{i=1}^{12}\sum_{j=1}^{12}Cov(\hat{Y}_{tj},\hat{Y}_{ti})$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now when you condition on something, it is a constant, so that's why the &quot;inner&quot; variance term disappears.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now you have a regression model for $Y_{ti}$ so we know that &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\begin{array}{l l}&#xD;&#xA;\hat{Y}_{ti}=Z_{ti,new}^{T}\hat{\beta} &amp;amp; Cov(\hat{Y}_{ti},\hat{Y}_{sj})=s^{2}Z_{ti,new}^{T}(Z^{T}Z)^{-1}Z_{sj,new} \\&#xD;&#xA;\hat{\beta}=(Z^{T}Z)^{-1}Z^{T}Y &amp;amp; s^{2}=\frac{1}{n-dim(\hat{\beta})}(Y-Z\hat{\beta})^{T}(Y-Z\hat{\beta})&#xD;&#xA;\end{array}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $Z$ and $Y$ are the matrix and vector that you used to actually fit the regression (I am assuming OLS regression here), $dim(\hat{\beta})$ is the number of betas that you have fitted (including the intercept).  $Z_{ti,new}$ is a new set of regression co-efficients to be used in the prediction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that for prediction, your estimates of $Y$ are &lt;em&gt;not independent&lt;/em&gt;, even if the &quot;true values&quot; are.  So the square root of $N$ rule doesn't apply, unless your $Z$ variables are orthogonal so that $(Z^{T}Z)^{-1}=I$ and $Z_{ti}^{T}Z_{sj}=0$ when $s\neq t$ or $i\neq j$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Plugging this into the variance formula for $\hat{X}_{t}$ we get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Var(\hat{X}_{t})=\sum_{i=1}^{12}\sum_{j=1}^{12}s^{2}Z_{ti,new}^{T}(Z^{T}Z)^{-1}Z_{tj,new}=s^{2}J^{T}Z_{t,new}(Z^{T}Z)^{-1}Z_{t,new}^{T}J$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $J$ is a column of 12 ones, and $Z_{t,new}$ is the twelve $Z_{ti}^{T}$ rows for prediction stacked on top of each other, of dimension $12\times dim(\hat{\beta})$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But note that we also have the &quot;true&quot; process $X_{t}$, assumed to be governed by the regression model, so we apply the law of iterated expectations again, but conditioning on $\hat{X}_{t}$ this time:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Var(X_{t})=E[Var(X_{t}|\hat{X}_{t})]+Var[E(X_{t}|\hat{X}_{t})]=E[Var(\sum_{i=1}^{12}Y_{t}|\hat{Y}_{ti})]+Var[\hat{X}_{t}]$$&#xA;$$=E[\sum_{i=1}^{12}Var(Y_{ti})]+Var[\hat{X}_{t}]=12s^{2}+s^{2}J^{T}Z_{t,new}(Z^{T}Z)^{-1}Z_{t,new}^{T}J$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I probably should put approx because this is a &quot;plug-in&quot; of $s^{2}$ for the &quot;true variance&quot; $\sigma^{2}$ - however I don't know of many people who don't just do this.  It is also justified on Bayesian grounds as the proper way to account for uncertainty in estimating $\sigma^{2}$ for the normal model, plus it is an unbiased estimator on frequentist grounds.  So the annual standard error should really be &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$s\sqrt{12+J^{T}Z_{t,new}(Z^{T}Z)^{-1}Z_{t,new}^{T}J}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So what the $\sqrt{12}$ rule is essentially doing here is ignoring the uncertainty in estimating the betas.  If you already estimate the betas pretty well, then this will make little difference to the $\sqrt{12}$ rule - probably something like $\sqrt{13}$.  If the betas are not estimated well, or you are close to multi-collinearity, then the extra term may be important.hb**&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;text&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;**&lt;/p&gt;&#xA;" OwnerUserId="2392" LastEditorUserId="2392" LastEditorDisplayName="" LastEditDate="2011-04-28T23:46:12.220" LastActivityDate="2011-04-28T23:46:12.220" />
  <row Id="10129" PostTypeId="2" ParentId="10117" CreationDate="2011-04-29T00:08:49.870" Score="1" ViewCount="" Body="&lt;p&gt;One of my colleagues at StatSoft put this series of videos together. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.statsoft.com/textbook/data-mining-techniques/&quot; rel=&quot;nofollow&quot;&gt;http://www.statsoft.com/textbook/data-mining-techniques/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The textbook links are also helpful. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good luck!!&lt;/p&gt;&#xA;" OwnerUserId="4385" LastActivityDate="2011-04-29T00:08:49.870" />
  <row Id="10130" PostTypeId="3" CreationDate="2011-04-29T00:18:02.670" Score="0" ViewCount="" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditorDisplayName="" LastEditDate="2011-04-29T00:18:02.670" LastActivityDate="2011-04-29T00:18:02.670" />
  <row Id="10131" PostTypeId="3" CreationDate="2011-04-29T00:18:02.670" Score="0" ViewCount="" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditorDisplayName="" LastEditDate="2011-04-29T00:18:02.670" LastActivityDate="2011-04-29T00:18:02.670" />
  <row Id="10132" PostTypeId="2" ParentId="10126" CreationDate="2011-04-29T01:26:50.757" Score="3" ViewCount="" Body="&lt;p&gt;In a school with a 1-unit higher X value a randomly picked student has 0.9 times the odds of having been unexcused from class at least once, compared to an other school.&#xA;(Assuming that all other student and school characteristics modelled are the same.)&lt;/p&gt;&#xA;" OwnerUserId="3911" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-29T10:09:11.960" LastActivityDate="2011-04-29T10:09:11.960" CommentCount="2" />
  <row Id="10133" PostTypeId="1" CreationDate="2011-04-29T01:57:42.573" Score="5" ViewCount="97" Body="&lt;p&gt;Question for the experienced data miners out there:&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Given this scenario:&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;There are N shopping carts&lt;/li&gt;&#xA;&lt;li&gt;Each shopping cart is filled with an arbitrary number of M items from an infinitely large set (with the current amount of data I have, that arbitrary number can hit numbers around 1500)&lt;/li&gt;&#xA;&lt;li&gt;The order in which each cart is filled is significant&lt;/li&gt;&#xA;&lt;li&gt;There are other attributes such as geolocation of shopper, but these can be (and currently are) thrown out in favor of making the algorithm simpler&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;I need to:&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;At a particular point in time, given only the ordered sets of items in each cart, identify 'similar' carts without prior knowledge of class labels&lt;/li&gt;&#xA;&lt;li&gt;After a certain amount of data has been collected and a drudge works through the data and assigns labels, create a classifier that can work quickly with future unseen data&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;Initial approach:&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;So far, my approach has been focused on the first point. My method uses k-means clustering and handles the sequential nature of the data by using a distance matrix generated by calculating the Hamming distance between carts. In this way, [apple, banana, pear] is different from [pear, apple, banana], but [apple, banana, pear] is less different from [apple, banana, antelope]. The appropriate value of k is determined through investigation of the silhouette coefficient. The clusters generated from this seem to make sense, but the runtime of my method will definitely be prohibitive as my dataset scales.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;Question:&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Would anyone happen to have any suggestions for a novice data miner for this problem?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;Edits with more info:&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;I've found suggestions that consider using n-gram features and comparing them pair-wise. A concern I have about this is order: will the order of the sequences be maintained if n-gram models are used? Also, I see performance issues being a larger possibility with this method.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="4386" LastEditorUserId="4386" LastEditorDisplayName="" LastEditDate="2011-04-29T17:04:04.173" LastActivityDate="2011-05-01T20:20:58.663" Title="Data mining approaches for analysis of sequential data with nominal attributes" Tags="&lt;clustering&gt;&lt;classification&gt;&lt;data-mining&gt;&lt;ordinal&gt;" AnswerCount="1" CommentCount="5" />
  <row Id="10135" PostTypeId="2" ParentId="10110" CreationDate="2011-04-29T05:35:19.293" Score="2" ViewCount="" Body="&lt;p&gt;You may want to check out &lt;a href=&quot;http://www.psych.mcgill.ca/misc/fda/&quot; rel=&quot;nofollow&quot;&gt;Functional Data Analysis&lt;/a&gt;, which permits characterization of functional (temporal) phenomena with noise in the wavelength, amplitude, etc.&lt;/p&gt;&#xA;" OwnerUserId="364" LastActivityDate="2011-04-29T05:35:19.293" CommentCount="1" />
  <row Id="10136" PostTypeId="2" ParentId="10020" CreationDate="2011-04-29T06:08:36.737" Score="3" ViewCount="" Body="&lt;h3&gt;Suggestions&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;You could perform &lt;strong&gt;individual multiple regressions&lt;/strong&gt; for each type of predictor, and compare across multiple regressions, adjusted r-square, generalised r-square, or some other parsimony adjusted measure of variance explained.&lt;/li&gt;&#xA;&lt;li&gt;You could alternatively explore the general literature on &lt;strong&gt;variable importance&lt;/strong&gt; (&lt;a href=&quot;http://jeromyanglim.blogspot.com/2009/09/variable-importance-and-multiple.html&quot; rel=&quot;nofollow&quot;&gt;see here for a discussion with links&lt;/a&gt;). This would encourage a focus on the importance of individual predictors.&lt;/li&gt;&#xA;&lt;li&gt;In some situations &lt;strong&gt;hierarchical regression&lt;/strong&gt; may provide a useful framework. You would enter one type of variable in one block (e.g., cognitive variables), and in the second block another type (e.g., social variables). This would help answer the question of whether one type of variable predicts over and above another type.&lt;/li&gt;&#xA;&lt;li&gt;As a side examination, you could run a &lt;strong&gt;factor analysis&lt;/strong&gt; on the predictor variables to examine whether the correlations between predictor variables map on to the assignment of variables to types.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;Caveats&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Types of variables such as cognitive, social, and behavioural are broad classes of variables. A given study will always include only a subset of the possible variables, and typically such a subset is small relative to the possible variables. &#xA;Furthermore, the measured variables may not be the most reliable or valid means of measuring the intended construct.&#xA;Thus, you need to be careful when drawing the broader inference about the relative importance of a given type of variable over and beyond what was actually measured.&lt;/li&gt;&#xA;&lt;li&gt;You also need to consider any bias in the way that the dependent variable was measured. Particularly in psychological studies, there is a tendency for self-report measures to correlate well with self-report, ability with ability, other-report with other report, and so on. The issue is that the mode of measurement has a large effect over and beyond the actual construct of interest. Thus, if the dependent variable is measured in a particular way (e.g., self-report), then don't over-interpret larger correlations with one type of predictor if that type also uses self-report.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-30T03:11:51.177" LastActivityDate="2011-04-30T03:11:51.177" CommentCount="1" />
  <row Id="10137" PostTypeId="1" CreationDate="2011-04-29T06:24:13.510" Score="1" ViewCount="90" Body="&lt;p&gt;Related to an &lt;a href=&quot;http://stats.stackexchange.com/questions/10079/rules-of-thumb-for-minimum-sample-size-for-multiple-regression&quot;&gt;earlier question on power analysis for multiple regression&lt;/a&gt;, a social science researcher asked me about power analysis for &lt;a href=&quot;http://www.davidakenny.net/cm/moderation.htm&quot; rel=&quot;nofollow&quot;&gt;moderator regression&lt;/a&gt; (i.e., an interaction effect).&#xA;The researcher asked me:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;I seem to recall that power of tests&#xA;  for moderation with two continuous&#xA;  predictor variables is low - do you know the&#xA;  minimum sample size requirement in&#xA;  this context?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;From the context, it can  further be assumed that this is an observational study (not an experimental study) and that the dependent variable is continuous.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Question&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What advice would you give regarding calculating the minimum sample size required?&lt;/li&gt;&#xA;&lt;li&gt;Are there any caveats that you would present?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-29T07:55:17.863" LastActivityDate="2011-04-29T08:34:31.240" Title="Power analysis for moderator effect in regression with two continuous predictors" Tags="&lt;regression&gt;&lt;sample-size&gt;&lt;power&gt;&lt;moderator&gt;" AnswerCount="1" />
  <row Id="10138" PostTypeId="2" ParentId="9930" CreationDate="2011-04-29T07:20:04.600" Score="0" ViewCount="" Body="&lt;p&gt;I think that variable &lt;em&gt;z&lt;/em&gt; in the question is a &lt;em&gt;suppresor variable&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suggest having a look at: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tzelgov, J., &amp;amp; Henik, A. (1991).Suppression situations in psychological research: Definitions, implications, and applications, &lt;em&gt;Psychological Bulletin, 109&lt;/em&gt; (3), 524-536. &lt;a href=&quot;http://doi.apa.org/psycinfo/1991-20289-001&quot; rel=&quot;nofollow&quot;&gt;http://doi.apa.org/psycinfo/1991-20289-001&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See also: &lt;a href=&quot;http://dionysus.psych.wisc.edu/lit/articles/TzelgovJ1991a.pdf&quot; rel=&quot;nofollow&quot;&gt;http://dionysus.psych.wisc.edu/lit/articles/TzelgovJ1991a.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;HTH,&#xA;dror&lt;/p&gt;&#xA;" OwnerUserId="4390" LastActivityDate="2011-04-29T07:20:04.600" />
  <row Id="10140" PostTypeId="2" ParentId="10137" CreationDate="2011-04-29T08:34:31.240" Score="3" ViewCount="" Body="&lt;p&gt;If I had to do this, I would use a simulation approach. This would involve making assumptions about the regression coefficients, predictor distributions, correlation between predictors, and error variance (with help from the researcher), generating data sets using the assumed model, and seeing what proportion of these give a significant p-value for the interaction. Then use trial and error to find the minimum sample size giving the required power.&lt;/p&gt;&#xA;" OwnerUserId="3835" LastActivityDate="2011-04-29T08:34:31.240" CommentCount="1" />
  <row Id="10141" PostTypeId="1" CreationDate="2011-04-29T09:44:30.163" Score="4" ViewCount="91" Body="&lt;p&gt;I know this question might have been around before like in this SO &lt;a href=&quot;http://stackoverflow.com/questions/2804001/panel-data-with-binary-dependent-variable-in-r&quot;&gt;thread&lt;/a&gt;, but maybe (hopefully) its answer has changed over time.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Is there any package in R or an outline how to do panel regressions with a discrete dependent variable? &lt;/li&gt;&#xA;&lt;li&gt;Is there any other open source package that does it and that would help coding something in R?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="704" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-29T10:45:19.777" LastActivityDate="2011-05-30T07:39:10.500" Title="Discrete choice panel models in R" Tags="&lt;r&gt;&lt;panel-data&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="1" />
  <row Id="10142" PostTypeId="1" CreationDate="2011-04-29T10:59:15.100" Score="3" ViewCount="97" Body="&lt;p&gt;How can the following tables be computed:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Chi-Square table&lt;/li&gt;&#xA;&lt;li&gt;Student t-table&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I'm looking for a formula or procedure used to make these tables.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;For example:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;If I have a given 'x'-value, as df(degree of freedom) with some confidence percentage 'y', I should be able to plug these x and y values into that formulae or procedure and should get results close to that in the standard tables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm pretty sure that there is some logic behind these tables, but I could not find any concrete pointers. how to reach them. My guess, about some logic got surety when I found some tool doing this online: &lt;a href=&quot;http://faculty.vassar.edu/lowry/csqsamp.html&quot; rel=&quot;nofollow&quot;&gt;http://faculty.vassar.edu/lowry/csqsamp.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="4331" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-29T11:33:37.130" LastActivityDate="2011-04-30T11:18:54.993" Title="Formula or procedure for computing standard statistical tables such as z table, Student's t-table, or chi-square table" Tags="&lt;normal-distribution&gt;&lt;chi-squared&gt;&lt;students-t&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="10143" PostTypeId="2" ParentId="10142" CreationDate="2011-04-29T11:11:55.250" Score="3" ViewCount="" Body="&lt;p&gt;On e.g. &lt;a href=&quot;http://en.wikipedia.org/wiki/Normal_distribution&quot; rel=&quot;nofollow&quot;&gt;wikipedia&lt;/a&gt; you can find the formula for the cdf / pdf of these distributions. Enter the values of the parameters and you're done.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want the reverse (I think you do from your question), simplest 'general' way of getting it done (not all cdfs have an analytic inverse) is use a univariate solver. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maybe you could just use R, that holds all these functions...&lt;/p&gt;&#xA;" OwnerUserId="4257" LastActivityDate="2011-04-29T11:11:55.250" CommentCount="4" />
  <row Id="10145" PostTypeId="1" CreationDate="2011-04-29T12:03:17.433" Score="3" ViewCount="98" Body="&lt;p&gt;I am writing my master´s thesis in finance on the topic of voluntary disclosure of financial targets in annual reports of manufacturing firms. &lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Context&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;I have created a dependent variable that is an index of level of disclosure. &lt;/li&gt;&#xA;&lt;li&gt;At the moment I am gathering firm characteristics as independent variables from around 10 years (unbalanced panel data).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;Questions&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Should the model be probit of logit?&lt;/li&gt;&#xA;&lt;li&gt;Should the model be fixed or random?&lt;/li&gt;&#xA;&lt;li&gt;How should it be implemented in Stata?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="4394" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-30T03:26:20.447" LastActivityDate="2011-04-30T22:24:09.737" Title="Predicting index from multiple predictors using panel data over 10 years: logit or probit? Fixed or random?" Tags="&lt;logistic-regression&gt;&lt;stata&gt;&lt;random-effects-model&gt;&lt;fixed-effects-model&gt;" AnswerCount="1" CommentCount="5" />
  <row Id="10146" PostTypeId="1" CreationDate="2011-04-29T14:34:35.197" Score="5" ViewCount="94" Body="&lt;p&gt;I have been doing multinomial logistic regression analysis using SPSS 19.&#xA;I have encountered the following problem when I run the analysis procedure:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&quot;Unexpected singularities in the&#xA;  Hessian matrix are encountered. This&#xA;  indicates that either some predictor&#xA;  variables should be excluded or some&#xA;  categories should be merged.&quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;A little background about my data used. I have four categorical predictors with two levels each, 1 or 2. The response variable in my model is a three-level categorical variable. I used the last level as the reference category. I tried to compare the coefficients of the intercept with that of the four predictors in the two logits so as to find which level of the response variable may cause this problem. The big differences in coefficients between the intercept and three of the predictors suggest that it might be the reference category that has the problem. However, I could not combine the levels of the response variable (which I'm not allowed for my research). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have also tried to exclude the predictors one by one, but still got the same problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could anyone please tell me what I should do to solve this problem?&lt;/p&gt;&#xA;" OwnerUserId="4380" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-29T15:23:42.813" LastActivityDate="2011-04-29T15:23:42.813" Title="Unexpected singularities in the Hessian matrix error in multinomial logistic regression" Tags="&lt;logistic-regression&gt;&lt;spss&gt;&lt;multinomial&gt;" CommentCount="2" />
  <row Id="10147" PostTypeId="1" CreationDate="2011-04-29T15:30:49.133" Score="0" ViewCount="111" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;Possible Duplicate:&lt;/strong&gt;&lt;br&gt;&#xA;  &lt;a href=&quot;http://stats.stackexchange.com/questions/10115/describe-how-a-domestic-fast-food-chain-with-plans-for-expanding-into-china-would&quot;&gt;Describe how a domestic fast food chain with plans for expanding into China would be able to use a forecasting model.&lt;/a&gt;  &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&#xA;&#xA;&lt;p&gt;Choose one of the forecasting methods and explain the rationale behind using it in real-life.&#xA;Describe how a domestic fast food chain with plans for expanding into China would be able to use a forecasting model. &lt;/p&gt;&#xA;" OwnerUserId="4395" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-04-29T16:59:37.080" LastActivityDate="2011-04-29T16:59:37.080" ClosedDate="2011-04-29T20:13:18.363" Title="Forecasting methods" Tags="&lt;statistical-significance&gt;" AnswerCount="0" CommentCount="5" />
  <row Id="10149" PostTypeId="1" AcceptedAnswerId="10151" CreationDate="2011-04-29T17:43:14.533" Score="3" ViewCount="55" Body="&lt;p&gt;Is there a function in R that given a vector of numbers, returns another vector with the standard units corresponding to each value?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where...&#xA;standard unit: how many SDs a value is + or - from the mean&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; x &amp;lt;- c(1,3,4,5,7)    # note: mean=4, sd=2&#xA; su(x) &#xA; [1]  -1.5  -0.5  0.0  0.5  1.5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is this func already included in a package?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="4400" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-29T19:23:38.993" LastActivityDate="2011-04-29T19:23:38.993" Title="Converting a vector to a vector of standard units in R" Tags="&lt;r&gt;" AnswerCount="1" />
  <row Id="10150" PostTypeId="1" CreationDate="2011-04-29T17:45:23.660" Score="3" ViewCount="126" Body="&lt;p&gt;If I have a list of values throughout time, say a list of values for every minute throughout an hour of monitoring something, can I somehow 'predict' or estimate what the value would probably be in the future, say another 20-30 minute after my last known value?  And if so, how can this be done?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I figure maybe I could do it with just a 'best-fit' to a quadratic and look at the value for x=90 (60 known values + 30 values into the future) but I feel like that probably isn't going to be the most accurate method.  Does anyone have any suggestions for a better way to do this? Just a note, I do have very little statistical experience, so please bare with me if this question is known to be easy (or impossible). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks! &lt;/p&gt;&#xA;" OwnerUserId="4399" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-04-29T22:27:13.157" LastActivityDate="2011-05-07T11:45:48.963" Title="Predicting a future data value with regression" Tags="&lt;time-series&gt;&lt;forecasting&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="2" />
  <row Id="10151" PostTypeId="2" ParentId="10149" CreationDate="2011-04-29T18:05:02.800" Score="3" ViewCount="" Body="&lt;p&gt;Try the scale function.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;?scale&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="26" LastActivityDate="2011-04-29T18:05:02.800" CommentCount="6" />
  <row Id="10152" PostTypeId="1" CreationDate="2011-04-29T18:06:24.857" Score="2" ViewCount="40" Body="&lt;p&gt;I learned about the zero-inflated negative binomial distribution a few months ago when I was trying to do regression on some discrete data. I have a different data set now, and it seems to be very similar except that the value &lt;code&gt;1&lt;/code&gt; seems to be over-represented (as opposed to &lt;code&gt;0&lt;/code&gt;). Is there such a thing as a one-inflated negative binomial distribution? How could I model these data?&lt;/p&gt;&#xA;" OwnerUserId="1973" LastActivityDate="2011-04-29T18:15:14.767" Title="One-inflated negative binomial?" Tags="&lt;distributions&gt;&lt;negative-binomial&gt;" AnswerCount="1" CommentCount="4" />
  <row Id="10153" PostTypeId="2" ParentId="10152" CreationDate="2011-04-29T18:15:14.767" Score="4" ViewCount="" Body="&lt;p&gt;Sure. You can write&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Pr(X=x) = p*f(x) + (1-p)1(x=1)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $f$ is the NB pmf, $1()$ is just an indicator function and $p$ is some probability. Of course in general the $x=1$ could be $x=k$ for any integer $k$, and $f$ could be any pmf. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In principle it shouldn't be harder to fit than a ZI model, but an off-the-shelf model-fitting solution may not exist. &lt;/p&gt;&#xA;" OwnerUserId="26" LastActivityDate="2011-04-29T18:15:14.767" />
  <row Id="10154" PostTypeId="1" CreationDate="2011-04-29T19:14:16.423" Score="0" ViewCount="35" Body="&lt;p&gt;First of all I have to mention, that I'm not a statician at all, I'm just a simple programmer and I have some curiosities... and the wrost of all, i don't know where to start from.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's assume the following working scenario:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A big company, a internet service provider (ISP) with unlimitd bandwith choose to change how the users will use and pay for the internet services: each user has to predict how much bandwidth they will consume in the next day for each hour. If the user predict that she will consume 0MB from 0.00 TO 18.00 she will pay nothing. If the user predict that she will see a HD movie from 18.00 to 20.00 they will comsume 10GB and they will pay just for that ammout of data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the user consume more that she predicted, she will pay more for that ammount of data (just for the differece). Predicted mount of data is cheapest. If they consume more that it was predicted, they will have to pay penalities. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The thing is that the users can build networks/groups with they freinds in order to optimize their costs. For examaple, if a user is not using his predicted amount of data, another friend from theyr network can use it, for free. If they wish the users of a group, can see (at every 30minutes) if their freinds are consuming traffic or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Idea is that each user has to predict how much internet they will comsume, based of theyr habits and schedules. The users will pay at the end of each month the trafice that they consumed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, the quest is to find the most appropiate way to represent the traffic prognoses for each user, group and for the ISP.&lt;br&gt;&#xA;- I'll use line charts to represent how much user has predicted and how much she consumed in each day for each hour.&lt;br&gt;&#xA;- I'll use bar charts to represent the difference between what was predicted and what was consumed in each hour. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is to represent those data in a graphical way for each month, for each user, group and the ISP:  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;-Can you give me some samples or resources which can help to find the best way of represeting my data costs?&lt;br&gt;&#xA;-Do you have an idea of what statistical approach to use in order to illustrate the prognoses, comsumes and errors?   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks &lt;/p&gt;&#xA;" OwnerUserId="3856" LastEditorUserId="3856" LastEditorDisplayName="" LastEditDate="2011-04-30T07:45:19.057" LastActivityDate="2011-04-30T07:45:19.057" Title="Representing traffic use forecasts graphically" Tags="&lt;data-visualization&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="10155" PostTypeId="1" CreationDate="2011-04-29T21:05:25.247" Score="1" ViewCount="33" Body="&lt;p&gt;When using a binomial family, logit link for GLM (or GEE in my case), I notice that my model estimates diverge when my response variables (which are continuous probabilities with range 0 to 1) include 0 or 1 (or 0 &amp;lt;= y &amp;lt;= 1) as observed values, but the models with response variables that don't include 0 or 1 (or 0 &amp;lt; y &amp;lt; 1) are able to converge just fine. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Question:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Why does this happen? &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;When running a logistic regression model (with 0 &amp;lt; y &amp;lt; 1) the model runs fine, as does the model when the response variable is dichotomous 0/1. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suspect the following: say I have observations 0 &amp;lt; y &amp;lt;= 1. In this case, the algorithm sees my ones but not any zeros, and then craps out saying &quot;some groups have fewer than x observations,&quot; the aforementioned group being the ones that are supposed to have zeros.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Secondary question:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;If I exclude observations that are 0 or 1 in order to fit my models, am I biasing my results?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Here's an example: my response variable is graduation rate expressed as a percentage. For the logistic regression models, there are apparently schools that have 100% graduation rate (seen as a 1 in my dataset). Would it be a valid strategy to drop these schools from the model, and what are the implications in interpretation? Is this akin to dropping outliers willy-nilly?&lt;/p&gt;&#xA;" OwnerUserId="3309" LastActivityDate="2011-04-30T10:54:03.897" Title="Estimates diverging using continuous probabilities in logistic regression" Tags="&lt;logistic-regression&gt;&lt;stata&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="10156" PostTypeId="2" ParentId="10150" CreationDate="2011-04-29T21:34:52.957" Score="1" ViewCount="" Body="&lt;p&gt;Yes, the problem occurs in variety of scenarios for time series prediction thing.&#xA;But I would suggest to avoid going through quadratic analysis just by estimating that it might fit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Go on hitting with Linear multivariate regression, further try a combinations of non-linear series. Plus I have also seen Certain problems of this sort like retweet prediction, stock prices predictions giving nearly accurate results with SOFNN- self of organizing fuzzy and neural network. Try scholar.google search, you would hit upon a previous good work on this.&lt;/p&gt;&#xA;" OwnerUserId="3994" LastActivityDate="2011-04-29T21:34:52.957" />
  <row Id="10157" PostTypeId="2" ParentId="10141" CreationDate="2011-04-29T21:35:09.960" Score="0" ViewCount="" Body="&lt;p&gt;What is wrong with using &lt;a href=&quot;http://www.jstatsoft.org/v27/i02/paper&quot; rel=&quot;nofollow&quot;&gt;plm&lt;/a&gt; or &lt;a href=&quot;http://lme4.r-forge.r-project.org/slides/2009-07-07-Rennes/3Longitudinal-4.pdf&quot; rel=&quot;nofollow&quot;&gt;lme4&lt;/a&gt; (&lt;a href=&quot;http://lme4.r-forge.r-project.org/book/Ch4.pdf&quot; rel=&quot;nofollow&quot;&gt;another link&lt;/a&gt;)? Particularly the &lt;code&gt;glmer&lt;/code&gt; function? &lt;/p&gt;&#xA;" OwnerUserId="1893" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-30T07:39:10.500" LastActivityDate="2011-05-30T07:39:10.500" CommentCount="1" />
  <row Id="10158" PostTypeId="2" ParentId="10154" CreationDate="2011-04-29T22:24:25.127" Score="2" ViewCount="" Body="&lt;p&gt;I would use &lt;a href=&quot;http://4.bp.blogspot.com/_V8g1rNtmHuM/SSMfczhjWoI/AAAAAAAAASk/mmf4jfkBHfY/s320/plot3.jpg&quot; rel=&quot;nofollow&quot;&gt;hanging bars&lt;/a&gt;, where actual consumptions are the bars, and errors are the differences between the lower ends of the bars and the horizontal axis. For aggregate data (multiple users in one group) the bars can be substituted with &lt;a href=&quot;http://www.java2s.com/Code/JavaImages/JFreeChartStackedBarChartDemo4.PNG&quot; rel=&quot;nofollow&quot;&gt;stacked bars&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastActivityDate="2011-04-29T22:24:25.127" CommentCount="1" />
  <row Id="10159" PostTypeId="1" AcceptedAnswerId="10161" CreationDate="2011-04-29T22:30:31.797" Score="5" ViewCount="124" Body="&lt;p&gt;I'm going to start out by saying this is a homework problem straight out of the book. I have spent a couple hours looking up how to find expected values and have determined I understand nothing.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Let $X$ have the cdf $F(x) = 1 - x^{-\alpha}, x\ge1$.&#xA;  Find $E(X)$ for those values of $\alpha$ for which $E(X)$ exists.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I have no idea how to even start this. How can I determine which values of $\alpha$ exist? I also don't know what to do with the cdf (I'm assuming this means Cumulative Distribution Function). There are forumlas for finding the expected value hwne you have a frequency function or density function. Wikipedia says the CDF of $X$ can be defined in terms of the probability density function ƒ as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$F(x) = \int_{-\infty}^x f(t)\,dt$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is as far as I got. Where do I go from here?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: I meant to put $x\ge1$.&lt;/p&gt;&#xA;" OwnerUserId="4401" LastEditorUserId="4401" LastEditorDisplayName="" LastEditDate="2011-04-30T17:52:27.237" LastActivityDate="2011-04-30T17:52:27.237" Title="Find expected value using cdf" Tags="&lt;homework&gt;&lt;expected-value&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="10160" PostTypeId="1" AcceptedAnswerId="10163" CreationDate="2011-04-29T23:11:29.087" Score="3" ViewCount="51" Body="&lt;p&gt;The ultimate goal is to show users, at a glance, if their data is normally distributed.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first attempt is a kludge that plots the data in a frequency graph.  Then, the observed mean and standard deviation are used to build a &quot;normal curve&quot; graph.  The frequency chart is laid over the the normal-curve chart and put next to some key statistics.  The frequency chart also colors positive bins green and negative bins red.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It looks like this:&#xA;&lt;img src=&quot;http://i.imgur.com/vPNKj.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand the fallacy of this approach, but for now it's practical.   What is a better way to approach this issue?&lt;/p&gt;&#xA;" OwnerUserId="933" LastActivityDate="2011-04-30T01:42:18.423" Title="What are some ways to graphically display non-normal distributions in Excel?" Tags="&lt;excel&gt;&lt;summary-statistics&gt;&lt;curve-fitting&gt;" AnswerCount="1" CommentCount="4" FavoriteCount="1" />
  <row Id="10161" PostTypeId="2" ParentId="10159" CreationDate="2011-04-29T23:21:56.283" Score="4" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Edited for the comment from probabilityislogic&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that $F(1)=0$ in this case so the distribution has probability $0$ of being less than $1$, so $x \ge 1$, and you will also need $\alpha &amp;gt; 0$ for an increasing cdf. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have the cdf then you want the anti-integral or derivative which with a continuous distribution like this &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$f(x) = \frac{dF(x)}{dx}$$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;and in reverse $F(x) = \int_{1}^x  f(t)\,dt$ for $x \ge 1$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then to find the expectation you need to find &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$E[X] = \int_{1}^{\infty}  x f(x)\,dx$$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;providing that this exists.  I will leave the calculus to you.&lt;/p&gt;&#xA;" OwnerUserId="2958" LastEditorUserId="2958" LastEditorDisplayName="" LastEditDate="2011-04-30T09:54:18.863" LastActivityDate="2011-04-30T09:54:18.863" CommentCount="6" />
  <row Id="10162" PostTypeId="1" AcceptedAnswerId="10196" CreationDate="2011-04-30T00:11:19.003" Score="3" ViewCount="141" Body="&lt;p&gt;I'm new to machine learning, and I have been trying to figure out how to apply neural network to time series forecasting. I have found resource related to my query, but I seem to still be a bit lost. I think a basic explanation without too much detail would help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say I have some price values for each month over a few years, and I want to predict new price values. I could get a list of prices for the last few months, and then try to find similar trends in the past using K-Nearest-Neighbor. I could them use the rate of change or some other property of the past trends to try and predict new prices. How I can apply neural network to this same problem is what I am trying to find out.&lt;/p&gt;&#xA;" OwnerUserId="4403" LastEditorUserId="4403" LastEditorDisplayName="" LastEditDate="2011-04-30T01:47:52.990" LastActivityDate="2011-05-01T06:36:43.740" Title="How to apply Neural Network to time series forecasting?" Tags="&lt;time-series&gt;&lt;forecasting&gt;&lt;neural-networks&gt;" AnswerCount="1" CommentCount="3" FavoriteCount="3" />
  <row Id="10163" PostTypeId="2" ParentId="10160" CreationDate="2011-04-30T01:42:18.423" Score="5" ViewCount="" Body="&lt;p&gt;A &lt;strong&gt;normal probability plot&lt;/strong&gt; is an excellent way to compare an empirical distribution to a normal distribution.  Its merits are that it clearly displays the &lt;em&gt;nature&lt;/em&gt; of any deviations from normality: ideally, the points lie along the diagonal; vertical deviations from the diagonal depict deviations from normality.  Its disadvantages are that many people do not know how to read it, so beware!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To create a normal probability plot in Excel, rank the data (with the RANK function) and convert them to a normal score via&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;NORMSINV((rank-1/2)/count)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where 'count' is the amount of data and 'rank' references a cell with the rank, as shown in the illustrations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/7tbPW.png&quot; alt=&quot;spreadsheet example&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The formulas in this spreadsheet are&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/78rkr.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-04-30T01:42:18.423" CommentCount="1" />
  <row Id="10164" PostTypeId="2" ParentId="577" CreationDate="2011-04-30T02:01:01.827" Score="4" ViewCount="" Body="&lt;p&gt;In my experience, BIC results in serious underfitting and AIC typically performs well, when the goal is to maximize predictive discrimination.&lt;/p&gt;&#xA;" OwnerUserId="4253" LastActivityDate="2011-04-30T02:01:01.827" />
  <row Id="10165" PostTypeId="2" ParentId="577" CreationDate="2011-04-30T05:49:44.777" Score="2" ViewCount="" Body="&lt;p&gt;An informative and accessible &quot;derivation&quot; of AIC and BIC by Brian Ripley can be found here:&#xA;&lt;a href=&quot;http://www.stats.ox.ac.uk/~ripley/Nelder80.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.stats.ox.ac.uk/~ripley/Nelder80.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ripley provides some remarks on the assumptions behind the mathematical results. Contrary to what some of the other answers indicate, Ripley emphasizes that AIC is based on assuming that the model is true. If the model is not true, a general computation will reveal that the &quot;number of parameters&quot; has to be replaced by a more complicated quantity. Some references are given in Ripleys slides. Note, however, that for linear regression (strictly speaking with a known variance) the, in general, more complicated quantity simplifies to be equal to the number of parameters.  &lt;/p&gt;&#xA;" OwnerUserId="4376" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-03T13:37:26.293" LastActivityDate="2011-05-03T13:37:26.293" />
  <row Id="10166" PostTypeId="2" ParentId="8570" CreationDate="2011-04-30T06:01:46.580" Score="2" ViewCount="" Body="&lt;p&gt;Nocedal and Wrights book&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://users.eecs.northwestern.edu/~nocedal/book/&quot; rel=&quot;nofollow&quot;&gt;http://users.eecs.northwestern.edu/~nocedal/book/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;is a good reference for optimization in general, and many things in their book are of interest to a statistician. There is also a whole chapter on non-linear least squares.&lt;/p&gt;&#xA;" OwnerUserId="4376" LastActivityDate="2011-04-30T06:01:46.580" />
  <row Id="10167" PostTypeId="1" CreationDate="2011-04-30T06:10:14.923" Score="6" ViewCount="88" Body="&lt;h3&gt;Context:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Within the context of structural equation modelling, I have non-normality according to the Mardia test but univariate indices of skewness and kurtosis are less than 2.0.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Questions:&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Should parameter estimates (coeficients estimates) be evaluated using bootstrapping (1000 replicates) with bias-corrected methods? &lt;/li&gt;&#xA;&lt;li&gt;In place of the traditional chi-square test, should the Bollen-Stine bootstrapped version be used?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="4406" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-04-30T07:12:45.743" LastActivityDate="2011-05-21T07:54:27.303" Title="Bootstrapped parameter and fit estimates with non-normality for structural equation models" Tags="&lt;bootstrap&gt;&lt;normality&gt;&lt;sem&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="1" />
  <row Id="10168" PostTypeId="2" ParentId="10159" CreationDate="2011-04-30T07:59:08.367" Score="4" ViewCount="" Body="&lt;p&gt;I think you actually mean $x\geq 1$, otherwise the CDF is vacuous, as $F(1)=1-1^{-\alpha}=1-1=0$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you &quot;know&quot; about CDFs is that they eventually approach zero as the argument $x$ decreases without bound and eventually approach one as $x \to \infty$.  They are also non-decreasing, so this means $0\leq F(y)\leq F(x)\leq 1$ for all $y\leq x$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So if we plug in the CDF we get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$0\leq 1-x^{-\alpha}\leq 1\implies 1\geq \frac{1}{x^{\alpha}}\geq 0\implies x^{\alpha}\geq 1 &amp;gt; 0\implies x\geq 1 \&amp;gt;.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From this we conclude that the support for $x$ is $x\geq 1$.  Now we also require $\lim_{x\to\infty} F(x)=1$ which implies that $\alpha&amp;gt;0$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To work out what values the expectation exists, we require:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\newcommand{\rd}{\mathrm{d}}E(X)=\int_{1}^{\infty}x\frac{\rd F(x)}{\rd x}\rd x=\alpha\int_{1}^{\infty}x^{-\alpha} \rd x$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And this last expression shows that for $E(X)$ to exist, we must have $-\alpha&amp;lt;-1$, which in turn implies $\alpha&amp;gt;1$.  This can easily be extended to determine the values of $\alpha$ for which the $r$'th raw moment $E(X^{r})$ exists.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-04-30T13:59:33.003" LastActivityDate="2011-04-30T13:59:33.003" CommentCount="4" />
  <row Id="10169" PostTypeId="1" AcceptedAnswerId="10173" CreationDate="2011-04-30T08:07:27.833" Score="4" ViewCount="120" Body="&lt;p&gt;I have a three-way contingency table in which the marginal totals for two sides are fixed and for the third are random. I'm wondering how to perform a chi-square test for homogeneity for such a three-way contingency table.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Example&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/r3QGt.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assume both Trt and Gender marginal totals are fixed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="3903" LastEditorUserId="3903" LastEditorDisplayName="" LastEditDate="2011-05-04T09:18:55.823" LastActivityDate="2011-05-04T23:50:30.257" Title="$\chi^2$ test of homogeneity for three-way contingency table" Tags="&lt;statistical-significance&gt;&lt;chi-squared&gt;&lt;categorical-data&gt;" AnswerCount="1" CommentCount="3" FavoriteCount="1" />
  <row Id="10170" PostTypeId="2" ParentId="10059" CreationDate="2011-04-30T08:12:03.110" Score="2" ViewCount="" Body="&lt;p&gt;The closest package that I can think of is &lt;a href=&quot;http://cran.r-project.org/src/contrib/Archive/birch/&quot; rel=&quot;nofollow&quot;&gt;birch&lt;/a&gt;, but it is not available on CRAN anymore so you have to get the source and install it yourself (&lt;code&gt;R CMD install birch_1.1-3.tar.gz&lt;/code&gt; works fine for me, OS X 10.6 with R version 2.13.0 (2011-04-13)). It implements the original algorithm described in &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Zhang, T. and Ramakrishnan, R. and&#xA;  Livny, M. (1997). &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.93.2882&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;BIRCH: A New Data&#xA;  Clustering Algorithm and Its&#xA;  Applications&lt;/a&gt;. &lt;em&gt;Data Mining and&#xA;  Knowledge Discovery&lt;/em&gt;, &lt;em&gt;1&lt;/em&gt;, 141-182.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;which relies on cluster feature tree, as does SPSS TwoStep (I cannot check, though). There's a possibility of using the k-means algorithm to perform clustering on &lt;code&gt;birch&lt;/code&gt; object (&lt;code&gt;kmeans.birch()&lt;/code&gt;), that is partition the subclusters into k groups such that the sum of squares of all the points in each subcluster to the assigned cluster centers is minimized.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-04-30T08:12:03.110" CommentCount="2" />
  <row Id="10171" PostTypeId="1" CreationDate="2011-04-30T08:52:26.243" Score="6" ViewCount="155" Body="&lt;p&gt;I am confused about permutation analysis for feature selection in a logistic regression context.&lt;br&gt;&#xA;Could you provide a clear explanation of the random permutation test and how does it applies to feature selection? Possibly with exact algorithm and examples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, How does it compare to other shrinkage methods like Lasso or LAR?&lt;/p&gt;&#xA;" OwnerUserId="3896" LastActivityDate="2011-05-13T17:42:00.410" Title="Random permutation test for feature selection" Tags="&lt;regression&gt;&lt;logistic-regression&gt;&lt;feature-selection&gt;&lt;permutation&gt;&lt;shrinkage&gt;" AnswerCount="1" CommentCount="4" FavoriteCount="2" />
  <row Id="10172" PostTypeId="2" ParentId="9920" CreationDate="2011-04-30T09:23:06.453" Score="2" ViewCount="" Body="&lt;p&gt;What you describes about Tukey's nonadditivity test sounds good to me. In effect, it allows to test for an item by rater interaction. Some words of caution, though:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Tukey's nonadditivity test effectively allows to test for a &lt;em&gt;linear-by-linear&lt;/em&gt; product of two factor main effects.&lt;/li&gt;&#xA;&lt;li&gt;The possibility of deriving a total score is irrelevant here, as this particular Tukey's test can be applied in any randomized block design, as described on &lt;a href=&quot;http://www.ats.ucla.edu/stat/stata/faq/nonadd.htm&quot; rel=&quot;nofollow&quot;&gt;Stata FAQ&lt;/a&gt;, for example.&lt;/li&gt;&#xA;&lt;li&gt;It applies in situation where you have a &lt;em&gt;single observation per cell&lt;/em&gt;, that is each rater assess only one item (no replicates).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You might recall that the interaction term is confounded with the error term when there're no replicates in an ANOVA design; in inter-rater studies, it means we have only one rating for each rater x item cell. Tukey's test in this case provide a 1-DF test for assessing any deviation from additivity, which is a common assumption to interpret a main effect in two-factor models. Here is a &lt;a href=&quot;http://www.plantsciences.ucdavis.edu/agr205/Supplemental%20Handouts/Tukey_NonAdd_Handout_PDF.pdf&quot; rel=&quot;nofollow&quot;&gt;tutorial&lt;/a&gt; describing how it works.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I must admit I never used it when computing ICC, and I spent some times trying to reproduce Dave Garson's results with R. This led me to the following two papers that showed that Tukey's nonadditivity test might not be the &quot;best&quot; test to use as it will fail to recover a true interaction effect (e.g., where some raters exhibit an opposite rating behavior compared to the rest of the raters) when there's no main effect of the target of the ratings (e.g., marks given to items):&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Lahey, M.A., Downey, R.G., and Saal, F.E. (1983). Intraclass Correlations: There's More There Than Meets the Eye. &lt;em&gt;Psychological Bulletin&lt;/em&gt;, &lt;em&gt;93(3)&lt;/em&gt;, 586-595.&lt;/li&gt;&#xA;&lt;li&gt;Johnson, D.E. and Graybill, F.A. (1972). An analysis of a two-way model with interaction and no replication. &lt;em&gt;Journal of the American Statistical Association&lt;/em&gt;, &lt;em&gt;67&lt;/em&gt;, 862-868.&lt;/li&gt;&#xA;&lt;li&gt;Hegemann, V. and Johnson, D.E. (1976). The power of two tests for nonadditivity. &lt;em&gt;Journal of the American Statistical Association&lt;/em&gt;, 71(356), 945-948.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;(I'm very sorry but I couldn't find ungated PDF version of those papers. The first one is really a must-read one.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;About your particular design, you considered raters as fixed effects (hence the use of Shrout and Fleiss's type 3 ICC, i.e. mixed model approach). In this case, &#xA;Lahey et al. (1) stated that you face a situation of nonorthogonal interaction components (i.e., the interaction is not independent of other effect) and a biased estimate of the rating effect -- but, this for the case where you have a single observation per cell (ICC(3,1)). With multiple ratings per items, estimating ICC(3,k) requires the &quot;assumption of nonsignificance of the interaction. In this case, the ANOVA effects are neither theoretically nor mathematically independent, and without adequate justification, the assumption of no interaction is very tenuous.&quot; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words, such an interaction test aims at offering you diagnostic information. &#xA;My opinion is that you can go on with you ICC, but be sure to check that (a) there's a significant effect for the target of ratings (otherwise, it would mean the reliability of measurements is low), (b) no rater systematically deviates from others' ratings (this can be done graphically, or based on the residuals of your ANOVA model).&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;More technical details are given below.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The alternative test that is proposed is called the &lt;em&gt;characteristic root test of the interaction&lt;/em&gt; (2,3). Consider a multiplicative interaction model of the form (here, as an effect model, that is we use parameters that summarize deviations from the grand mean):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\mu_{ij}=\mu + \tau_i + \beta_j + \lambda\alpha_i\gamma_j + \varepsilon_{ij}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;with $\tau$ ($i=1,\dots,t$) the effect due to targets/items, $\beta$ ($j=1,\dots,b$) the effect of raters, $\alpha\gamma$ the interaction targets x raters, and the usual assumptions for the distribution of errors and parameters constraints. We can compute the largest characteristic root of $Z&amp;#39;Z$ or $ZZ&amp;#39;$, where $Z=z_{ij}=y_{ij}-y_{i\cdot}-y_{\cdot j}+y_{\cdot\cdot}$ is the $t \times b$ matrix of residuals from an additive model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The test then relies on the idea of using $\lambda_1/\text{RSS}$ as a test statistic ($H_0:\, \lambda=0$) where $\lambda_1$ is the largest nonzero characteristic root of $ZZ&amp;#39;$ (or $Z'Z$), and RSS equals the residual sum of squares from an additive model (2).&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-02T13:55:43.327" LastActivityDate="2011-05-02T13:55:43.327" />
  <row Id="10173" PostTypeId="2" ParentId="10169" CreationDate="2011-04-30T09:41:13.747" Score="4" ViewCount="" Body="&lt;p&gt;I would approach this as a hypothesis test (but then I am a Bayesian, and we always tend to do things a bit differently).  I take it by homogeneous that you mean homogeneous in the &quot;way&quot; which did not have its totals fixed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we index the cell counts as $n_{ijk}$ for $i=1,\dots,I$ for the first way, and $j=1,\dots,J$ for the second way, and $k=1,\dots,K$ for the third way.  This gives a total of $IJK$ cells in your contingency table.  The &quot;saturated model&quot; is to assume that each cell is a multinomial distributed variable, with an individual long run frequency, so &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p(n_{111},\dots,n_{IJK}|\theta_{111},\dots,\theta_{IJK})=n_{\bullet\bullet\bullet}!\prod_{i=1}^{I}\prod_{j=1}^{J}\prod_{k=1}^{K}\frac{\theta_{ijk}^{n_{ijk}}}{n_{ijk}!}\propto\prod_{i=1}^{I}\prod_{j=1}^{J}\prod_{k=1}^{K}\theta_{ijk}^{n_{ijk}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If a row total had been fixed, then we would get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p(n_{111},\dots,n_{IJK}|\theta_{111},\dots,\theta_{IJK})=\prod_{i=1}^{I}n_{i\bullet\bullet}!\prod_{j=1}^{J}\prod_{k=1}^{K}\frac{\theta_{ijk}^{n_{ijk}}}{n_{ijk}!}\propto\prod_{i=1}^{I}\prod_{j=1}^{J}\prod_{k=1}^{K}\theta_{ijk}^{n_{ijk}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If a row and column totals had been fixed, then we would get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p(n_{111},\dots,n_{IJK}|\theta_{111},\dots,\theta_{IJK})=\prod_{i=1}^{I}n_{i\bullet\bullet}!\prod_{j=1}^{J}n_{\bullet j\bullet}!\prod_{k=1}^{K}\frac{\theta_{ijk}^{n_{ijk}}}{n_{ijk}!}\propto\prod_{i=1}^{I}\prod_{j=1}^{J}\prod_{k=1}^{K}\theta_{ijk}^{n_{ijk}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This shows that whether the row totals are fixed or random, is not relevant to inference about the $\theta_{ijk}$ parameters - as this will simply change the factorial in the denominator.  This is called the &quot;likelihood principle&quot;.  However it &lt;em&gt;is relevant&lt;/em&gt; if you are going to be predicting a new set of counts sampled in the same way that you sampled your data.  This is usually how the standard statistician will think though (in terms of &quot;if I did this experiment again...&quot;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But for a hypothesis which is not sharp, such as two parameters being equal, but not restricted to be equal to anything the &quot;prior predictive&quot; distribution is what matters (at least from a Bayesian perspective) - and this certainly depends on the sample design!  This is quite interesting for the &quot;likelihood principle&quot; is a bit subtle - your interest must be &lt;em&gt;in the particular values of the parameters&lt;/em&gt; and not whether two parameters are equal.  It is quite easy to see that the prior predictive distribution will depend on the sample design.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My answer to &lt;a href=&quot;http://stats.stackexchange.com/questions/9685/how-to-carry-out-multiple-post-hoc-chi-square-tests-on-a-2-x-3-table/9721#9721&quot;&gt;this question&lt;/a&gt; should be easily adapted to your case.  I can provide more details if you are uncertain how to adapt it to your case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also use a poisson GLM with a log-link function to do this test, and it amounts to testing that a certain interaction term in the glm is zero, so you can use the Pearson residuals from the output of the glm to do your chi-square test - you can also use the deviance table.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having denied the likelihood principle here is actually incorrect one my part, for making inference on the $\theta_{ijk}$ parameters.  This is easily seen by first computing odds ratios and then renormalising - the part related to the sampling design will cancel out.  What an ass I made of myself, talking about &quot;subtle&quot; and all that!  To show this more explicitly, I will write out $K=K(n_{111},\dots,n_{IJK})$ as the constant which depends on how the experiment was done and which totals were fixed and which were random (independent of the $\theta_{ijk}$ parameters).  I will show the final answer does not depend on $K$.  So we we have:&#xA;$$p(n_{111},\dots,n_{IJK}|\theta_{111},\dots,\theta_{IJK})=K\prod_{i=1}^{I}\prod_{j=1}^{J}\prod_{k=1}^{K}\theta_{ijk}^{n_{ijk}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the specific problem I will let $i$ denote gender, $j$ denote treatment, and $k$ denote the response, so we have $I=2,J=3,K=2$.  Now I am going to assume that these are exhaustive classes.  This can be re-phrased as restricting the domain of generality to those units which could be classified into this table - and not as &quot;something else&quot; which isn't in the table (e.g. a fourth treatment).  This places some restrictions on the values of $\theta_{ijk}$, namely that they sum to 1:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\sum_{i=1}^{2}\sum_{j=1}^{3}\sum_{k=1}^{2}\theta_{ijk}=1$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And we have a hypothesis to consider:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$H_{0}:\text{Response does not differentiate between treatments and gender}$$&#xA;Or in mathematical terms&#xA;$$H_{0}:\theta_{i_{1}j_{1}k}=\theta_{i_{2}j_{2}k}\;\;\;\forall i_{1},j_{1},i_{2},j_{2},k$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now this basically means that there is only one parameter in the table $\theta$, and we have an integral to do (letting $D\equiv n_{111},\dots,n_{IJK}$ and $I$ denote the prior information):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p(H_{0}|D,I)=\frac{p(H_{0}|I)p(D|H_{0},I)}{p(D|I)}=\frac{p(H_{0}|I)}{p(D|I)}\int_{0}^{1}p(\theta|H_{0},I)p(D|\theta,H_{0},I)d\theta$$&#xA;$$=\frac{p(H_{0}|I)}{p(D|I)}K\int_{0}^{1}p(\theta|H_{0},I)\prod_{i=1}^{I}\prod_{j=1}^{J}\theta^{n_{ij1}}(1-\theta)^{n_{ij2}}$$&#xA;$$=\frac{p(H_{0}|I)}{p(D|I)}K\int_{0}^{1}p(\theta|H_{0},I)\theta^{n_{\bullet\bullet 1}}(1-\theta)^{n_{\bullet\bullet 2}}d\theta$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now nothing in the hypothesis says what the actual rate is, so $p(\theta|H_{0},I)=p(\theta|I)$ - this must depend on the prior information.  A conservative choice is the uniform one, this will adopted for all priors, and we have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p(H_{0}|D,I)=\frac{p(H_{0}|I)}{p(D|I)}K \frac{\Gamma(n_{\bullet\bullet 1}+1)\Gamma(n_{\bullet\bullet 2}+1)}{\Gamma(n_{\bullet\bullet\bullet}+2)}&#xD;&#xA;=\frac{p(H_{0}|I)}{p(D|I)}K \frac{\Gamma(46)\Gamma(276)}{\Gamma(322)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And it looks like $K$ is relevant, but we have yet to calculate $P(D|I)$.  In order to do this, we must think about what the alternatives to $H_0$ are.  These are fairly obvious: that the response differs by treatment and not by gender; by gender and not by treatment; or by both gender and by treatment&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$H_{1T}:\theta_{ij_{1}k}=\theta_{ij_{2}k}\;\;\;\forall i,j_{1},j_{2},k$$&#xA;$$H_{1G}:\theta_{i_{1}jk}=\theta_{i_{2}jk}\;\;\;\forall i_{1},i_{2},j,k$$&#xA;$$H_{2}:\theta_{i_{1}j_{1}k}\neq\theta_{i_{2}j_{2}k}\;\;\;\forall i_{1},j_{1},i_{2},j_{2},k$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These could be further broken down into statements about the individual cells, but this is not necessary.  As before we will set uniform priors for the unrestricted parameters, and for $H_{1T}$ we get an integral&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\int_{\sum_{ik}\theta_{ik}=1}\prod_{i,k=1}^{2}\theta_{ik}^{n_{i\bullet k}}\prod_{i,k=1}^{2}d\theta_{ik}=\frac{\Gamma(n_{1\bullet 1}+1)\Gamma(n_{1\bullet 2}+1)\Gamma(n_{2\bullet 1}+1)\Gamma(n_{2\bullet 2}+1)}{\Gamma(n_{\bullet\bullet\bullet}+4)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now if you are like me, you will be able to see the pattern.  The probabilities of each are written below:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p(H_{0}|D,I)=\frac{p(H_{0}|I)}{p(D|I)}K \frac{\Gamma(n_{\bullet\bullet 1}+1)\Gamma(n_{\bullet\bullet 2}+1)}{\Gamma(n_{\bullet\bullet\bullet}+2)}$$&#xA;$$p(H_{1T}|D,I)=\frac{p(H_{1T}|I)}{p(D|I)}K \frac{\prod_{i,k}\Gamma(n_{i\bullet k}+1)}{\Gamma(n_{\bullet\bullet\bullet}+4)}$$&#xA;$$p(H_{1G}|D,I)=\frac{p(H_{1G}|I)}{p(D|I)}K \frac{\prod_{j,k}\Gamma(n_{\bullet jk}+1)}{\Gamma(n_{\bullet\bullet\bullet}+6)}$$&#xA;$$p(H_{2}|D,I)=\frac{p(H_{2}|I)}{p(D|I)}K \frac{\prod_{i,j,k}\Gamma(n_{ijk}+1)}{\Gamma(n_{\bullet\bullet\bullet}+12)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that all of these probabilities contain the factor $\frac{K}{p(D|I)}$.  So if we were to take odds ratios they would just cancel out.  That is both $K$ and $P(D|I)$ cannot change the relative relationships between the 4 hypothesis - thus $K$ is irrelevant for answering this question as is $P(D|I)$.  It does not matter whether you had fixed designs or random ones - it only matters that the likelihood function has the form I wrote at the start.  So I will give my results in odds ratio form, with $H_0$ the denominator in the odds.  Now if we assume that all hypothesis are equally likely, then these also drop out of the odds ratios.  To go from odds ratios to probabilities is easy&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(H_0|D,I)=\left(\sum_{h} \frac{P(H_h|D,I)}{P(H_0|D,I)}\right)^{-1}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And the results are given below, which shows that you would have to be crazy to reject $H_{0}$ in favour of the alternatives (the probability has first 60 decimal digits of 9 before the first non 9 digit).  I also did a quick $\chi^{2}$ test and this gave values of $\chi^{2}=5.49$ on $5$ degrees of freedom, for a p-value of 0.36 (note that this is inflated because of some cells below 10, so the fit is much better than what $\chi^{2}$ test indicates).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;\begin{array}{c|c} &#xD;&#xA;\text{Hypothesis} &amp;amp; \text{prior odds vs }H_{0} &amp;amp; \text{posterior odds vs }H_{0} \\ \hline&#xD;&#xA;H_{0} &amp;amp; 1 &amp;amp; 1 \\&#xD;&#xA;H_{1T} &amp;amp; 1 &amp;amp; 5\times 10^{-61} \\&#xD;&#xA;H_{1G} &amp;amp; 1 &amp;amp; 9\times 10^{-147} \\&#xD;&#xA;H_{2} &amp;amp; 1 &amp;amp; 4\times 10^{-214} \\&#xD;&#xA; \end{array}&#xD;&#xA;$$&lt;/p&gt;&#xA;" OwnerUserId="2392" LastEditorUserId="2392" LastEditorDisplayName="" LastEditDate="2011-05-04T23:50:30.257" LastActivityDate="2011-05-04T23:50:30.257" CommentCount="1" />
  <row Id="10174" PostTypeId="2" ParentId="10155" CreationDate="2011-04-30T10:54:03.897" Score="1" ViewCount="" Body="&lt;p&gt;It shouldn't happen, if you do the taylor series well - I'd suggest starting it at different initial values.  A good choice is to set the intercept equal to the logit of the total proportion in your sample, and all other betas to zero.  So you have $p_{i}=\frac{y_{i}}{n_{i}}$ as the observed proportions for each unit.  Just set &#xA;$$\beta_0=logit\left(\frac{\overline{y}}{\overline{n}}\right)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and all other betas equal to zero as your starting values.  This should stop the 0s and 1s giving you problems.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another way to stabilise your results is the good old (+1) and (+2) rule, which is similar to ridging in ordinary regression.  To do this you regress&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\tilde{p}_{i}=\frac{y_{i}+1}{n_{i}+2}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On X directly using ols regression (i.e. no iterations).  This is shown to be a generalised MLE in &lt;a href=&quot;http://www.samsi.info/sites/default/files/tr2007-08.pdf&quot; rel=&quot;nofollow&quot;&gt;this paper&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-04-30T10:54:03.897" />
  <row Id="10175" PostTypeId="2" ParentId="10142" CreationDate="2011-04-30T11:18:54.993" Score="0" ViewCount="" Body="&lt;p&gt;I would look into a few special functions, which continually &quot;pop-up&quot; in statistics - often in disguised forms:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;the confluent hypergeometric function $_{1}F_{1}(a;b;z)$.  heaps of functions are special cases of this one, such as erf, incomplete gamma, bessel.&lt;/li&gt;&#xA;&lt;li&gt;the incomplete gamma function (and regularised incomplete gamma function)&lt;/li&gt;&#xA;&lt;li&gt;the incomplete beta function (and regularised incomplete beta function)&lt;/li&gt;&#xA;&lt;li&gt;Gaussian hypergeometric function $_{2}F_{1}(a;b;c;z)$&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;2) and 3) are the CDFs for chi-square and t-distributions for certain values of the parameters.  I would invert these using the table backwards, rather than directly looking for an inverse function.  It is likely that the error in interpolation (for sufficiently close values) may be less than the error in numerically evaluating a complicated inverse function.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-04-30T11:18:54.993" CommentCount="1" />
  <row Id="10176" PostTypeId="2" ParentId="8570" CreationDate="2011-04-30T12:02:50.277" Score="2" ViewCount="" Body="&lt;p&gt;&lt;em&gt;Optimization&lt;/em&gt;, by Kenneth Lange (Springer, 2004), &lt;a href=&quot;http://pubs.amstat.org/doi/abs/10.1198/jasa.2005.s47&quot; rel=&quot;nofollow&quot;&gt;reviewed&lt;/a&gt; in JASA by Russell Steele. It's a good textbook with Gentle's &lt;em&gt;Matrix algebra&lt;/em&gt; for an introductory course on Matrix Calculus and Optimization, like the one by &lt;a href=&quot;https://public.me.com/jdeleeuw&quot; rel=&quot;nofollow&quot;&gt;Jan de Leeuw&lt;/a&gt; (courses/202B).&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-04-30T12:02:50.277" CommentCount="4" />
  <row Id="10177" PostTypeId="1" CreationDate="2011-04-30T13:56:28.013" Score="3" ViewCount="76" Body="&lt;p&gt;Is there a function that does smart date parsing in R?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know the &lt;code&gt;strftime&lt;/code&gt;/&lt;code&gt;as.POSIXct&lt;/code&gt;/&lt;code&gt;as.POSIXlt&lt;/code&gt; commands, but they require a date formatting string, or throw the error &quot;character string is not in a standard unambiguous format&quot;.  This happens even when I pass a string with more than enough information to be parseable, like &quot;Fri Apr 29 16:43:20 GMT 2011&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It would be really nice to not bother with reverse-engineering the format every time I import dates from a new source.  Is there code that already does this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="4110" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-04-30T15:52:11.647" LastActivityDate="2011-04-30T15:52:11.647" Title="Smart date parsing in R?" Tags="&lt;r&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="10178" PostTypeId="2" ParentId="10177" CreationDate="2011-04-30T15:09:15.863" Score="3" ViewCount="" Body="&lt;p&gt;Try &quot;lubridate&quot; package, from CRAN, install in the usual way. Might help!&lt;/p&gt;&#xA;" OwnerUserId="1549" LastActivityDate="2011-04-30T15:09:15.863" CommentCount="1" />
  <row Id="10179" PostTypeId="2" ParentId="10177" CreationDate="2011-04-30T15:12:02.103" Score="1" ViewCount="" Body="&lt;p&gt;I don't know of any, and I'm not sure how it would work. Does it use the first entry, figure a template, then use that for the rest of the entries? Does it parse each entry individually, so that there is no template and each entry can be different?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the latter case, I'd worry a bit that it would be too flexible. Maybe I'm not thinking clearly, but the reverse-engineering and fixed-template method would seem to catch more errors and would help me to be more aware of my data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(I was about to recommend lubridate, which doesn't do what you want, but does make many date tasks easier, and while I was typing, Spacedman beat me to the punch.)&lt;/p&gt;&#xA;" OwnerUserId="1764" LastActivityDate="2011-04-30T15:12:02.103" CommentCount="1" />
  <row Id="10180" PostTypeId="1" CreationDate="2011-04-30T15:50:33.363" Score="3" ViewCount="114" Body="&lt;p&gt;I work quite often with Google's Website Optimizer, essentially it allows you to make small changes to a site to determine if they have an effect on the ratio of clicks to a web page to sales.  The stats look like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Combination 1: 500 clicks  10 sales&#xA;Combination 2: 498 clicks  11 sales&#xA;Combination 3: 503 clicks  15 sales&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now obviously that part makes sense to me, but then it gives a value called &quot;Probability of being better&quot; which is based (among other factors) on the sample size (see also &lt;a href=&quot;http://stats.stackexchange.com/questions/9735/how-does-a-frequentist-calculate-the-chance-that-group-a-beats-group-b-regarding&quot;&gt;this question&lt;/a&gt;).  So if a particular experiment has less data it would rate it as having a 13% chance at being better where as at a larger dataset it would have a 90% chance at being correct.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Obviously a smaller sample size would be more prone to swings in variance but  I'm curious to know if there's a formula to determine when enough data would have been collected so that a high probability of being better can be calculated (which seems to be equivalent to a small overlapping of the corresponding confidence intervals and hence a high probability of rejecting the Null-Hypothesis). &lt;/p&gt;&#xA;" OwnerUserId="4412" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-03T11:18:21.667" LastActivityDate="2011-05-03T11:18:21.667" Title="How to calculate sample size for a one-sided test on a rxs contigency table" Tags="&lt;probability&gt;&lt;hypothesis-testing&gt;&lt;ab-test&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="10182" PostTypeId="1" CreationDate="2011-04-30T19:02:45.603" Score="2" ViewCount="73" Body="&lt;p&gt;I'm a little confused regarding the intraclass correlation coefficient and one-way ANOVA.  As I understand it, both tell you how similar observations within a group are, relative to observations in other groups.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could someone explain this a little better, and perhaps explain the situation(s) in which each method is more advantageous?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for your time.&lt;/p&gt;&#xA;" OwnerUserId="4301" LastActivityDate="2011-04-30T19:02:45.603" Title="Intraclass correlation coefficient vs. F-test (one-way ANOVA)?" Tags="&lt;anova&gt;&lt;psychometrics&gt;&lt;intraclass-correlation&gt;" CommentCount="4" />
  <row Id="10183" PostTypeId="1" CreationDate="2011-04-30T19:20:12.770" Score="0" ViewCount="39" Body="&lt;p&gt;I have two sets of points, all of which belong to a 60D space. The first set are simply a large collection of points (around 500,000 points: ~1000 images, each having between ~400-500 points), and the second set contains 100 centroids obtained from running kmeans on the first set of points.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wish to reduce each image to a single 100D histogram through vector quantisation. To do so, each point in a given image votes for its nearest centroid (histogram bin), and the votes are accumulated giving the histogram. However, I am trying to implement this using a soft-voting scheme, which was described in [1].&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Soft-voting places a Gaussian centered about each centroid, with all Gaussians having a common variance. Each point now votes (softly) by computing its Gaussian probability for each centroid. According to [1] the common variance is set empirically such that each point has 4-5 &quot;significant votes&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have played around with the variance in an ad-hoc manner. So far, given my data, I have not been able to find a value for the variance that produces 4-5 significant votes. Is there a systematic way to approach this? Is it possible that such a variance does not exist?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an alternative, could I simply determine the Gaussian probabilities, and only use the &quot;best 5&quot; (highest probabilities) and normalise the votes to sum to 1? Would this have any nasty side-effects?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[1] Ankur Agarwal, Bill Triggs: Recovering 3D Human Pose from Monocular Images. IEEE Trans. PAMI 28(1): 44-58 (2006)&lt;/p&gt;&#xA;" OwnerUserId="3052" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-04-30T20:30:55.620" LastActivityDate="2011-04-30T20:30:55.620" Title="Vector quantisation: empirically determining the variance" Tags="&lt;variance&gt;&lt;variance-covariance&gt;" CommentCount="1" />
  <row Id="10184" PostTypeId="2" ParentId="10145" CreationDate="2011-04-30T22:24:09.737" Score="2" ViewCount="" Body="&lt;p&gt;Just on the first of your questions: probit or logit?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In practice it usually makes little difference.  You will get different parameter estimates from the two methods, but that is because the parameters mean different things. When you then use these to model, the differences will then largely disappear.  The logistic density distribution is slightly leptokurtic (a sharper peak and fatter tails) compared with the normal distribution, and being comfortable with odds I personally find it slightly easier to explain logit methods, but you need not worry too much whichever you choose.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can find some discussion in &lt;a href=&quot;http://www.iasri.res.in/ebook/EBADAT/6-Other%20Useful%20Techniques/5-Logit%20and%20Probit%20Analysis%20Lecture.pdf&quot; rel=&quot;nofollow&quot;&gt;this lecture&lt;/a&gt;.   &lt;/p&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-04-30T22:24:09.737" CommentCount="4" />
  <row Id="10185" PostTypeId="1" CreationDate="2011-04-30T23:14:00.433" Score="3" ViewCount="61" Body="&lt;p&gt;I'm working on a not-too-fancy Bayesian model in R and JAGS.  The goal is to isolate coder errors in a content analysis task.  Code and output are given below.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The larger question is how to go about debugging JAGS. (I assume that the same advice would hold for BUGS as well.) What am I supposed to make of an error like &quot;Invalid initial values&quot; when there are nearly a dozen different initial values?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's my R code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(rjags)&#xA;library(R2jags)&#xA;&#xA;#Load the data&#xA;toy_data &amp;lt;- read.csv(&quot;toy_data.csv&quot;)&#xA;&#xA;#Rescale data&#xA;rescaled_data &amp;lt;- toy_data[,c(3:(2+K))]&#xA;for( k in c(1:K) ){&#xA;  col &amp;lt;- rescaled_data[,c(k)]&#xA;  rescaled_data[,c(k)] &amp;lt;- (col-min(col))/(max(col)-min(col))&#xA;}&#xA;&#xA;codes &amp;lt;- as.matrix(rescaled_data)&#xA;doc_ids &amp;lt;- toy_data$docid&#xA;coder_ids &amp;lt;- toy_data$coderid&#xA;&#xA;N &amp;lt;- nrow( toy_data ) #Number of document codings&#xA;D &amp;lt;- max(toy_data$docid) #Number of documents&#xA;I &amp;lt;- max(toy_data$coderid) #Number of coders&#xA;K &amp;lt;- dim(toy_data)[2]-2 #Number of attributes&#xA;&#xA;#Package info for jags&#xA;jags.data &amp;lt;- list( &quot;doc_ids&quot;, &quot;coder_ids&quot;, &quot;codes&quot;, &quot;N&quot;, &quot;D&quot;, &quot;I&quot;, &quot;K&quot; )&#xA;jags.params &amp;lt;- c(&quot;z&quot;,&quot;mu&quot;,&quot;sigma&quot;,&quot;sigma_i&quot;,&quot;sigma_k&quot;,&quot;mu_dk&quot;,&quot;alpha_k&quot;,&quot;beta_k&quot;,&quot;alpha_i&quot;,&quot;beta_i&quot;)&#xA;jags.inits &amp;lt;- list(&#xA;  &quot;z&quot; &amp;lt;- matrix(rnorm(N*K),nrow=N,ncol=K),&#xA;  &quot;mu&quot; &amp;lt;- runif(1)*10,&#xA;  &quot;sigma&quot; &amp;lt;- rgamma(1,10),&#xA;  &quot;sigma_i&quot; &amp;lt;- rgamma(I,10),&#xA;  &quot;sigma_k&quot; &amp;lt;- rgamma(K,10),&#xA;  &quot;mu_dk&quot; &amp;lt;- as.matrix(rnorm(D*K),nrow=D,ncol=K),&#xA;  &quot;alpha_k&quot; &amp;lt;- rgamma(1,10),&#xA;  &quot;alpha_i&quot; &amp;lt;- rgamma(1,10),&#xA;  &quot;alpha_i&quot; &amp;lt;- rgamma(1,10),&#xA;  &quot;beta_i&quot; &amp;lt;- rgamma(1,10)&#xA;  )&#xA;&#xA;#Fit the model&#xA;jagsfit &amp;lt;- jags(&#xA;  model.file=&quot;coder_model.txt&quot;,&#xA;  data=jags.data, &#xA;  inits=jags.inits,&#xA;  jags.params,&#xA;  n.iter=5000,&#xA;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here's the JAGS model:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model {&#xA;  for( n in 1:N ){   #Loop over codings&#xA;    for( k in 1:K ){ #Loop over attributes&#xA;      #d &amp;lt;- doc_ids[n]    #Get document index&#xA;      #i &amp;lt;- coder_ids[n]  #Get code index&#xA;      codes[n,k] ~ dbern(p[n,k])&#xA;      logit(p[n,k]) &amp;lt;- z[n,k]&#xA;      z[n,k] ~ dnorm( mu_dk[doc_ids[n],k], sigma_k[k]*(1+sigma_i[coder_ids[n]]) )&#xA;    }&#xA;  }&#xA;&#xA;  for( d in 1:D ){&#xA;    for( k in 1:K ){&#xA;      mu_dk[d,k] ~ dnorm( mu, sigma )&#xA;    }&#xA;  }&#xA;&#xA;  for( k in 1:K ){   #Loop over attributes&#xA;    sigma_k[k] ~ dgamma( alpha_k, beta_k )&#xA;  }&#xA;  for( i in 1:I ){   #Loop over coders&#xA;    sigma_i[i] ~ dgamma( alpha_i, beta_i )&#xA;  }&#xA;&#xA;  #Noninformative priors over alphas and betas&#xA;  mu ~ dnorm( 0, 10 )&#xA;  sigma ~ dgamma(10,8)&#xA;  alpha_k ~ dgamma(10,8)&#xA;  beta_k ~ dgamma(10,8)&#xA;  alpha_i ~ dgamma(10,8)&#xA;  beta_i ~ dgamma(10,8)&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here's the data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&quot;docid&quot;,&quot;coderid&quot;,&quot;Answer.1&quot;,&quot;Answer.2&quot;&#xA;1,1,3,3&#xA;1,2,4,1&#xA;1,3,7,2&#xA;2,1,3,3&#xA;2,2,4,4&#xA;2,4,3,1&#xA;3,1,3,3&#xA;3,2,4,3&#xA;3,3,3,4&#xA;4,4,5,1&#xA;4,5,6,2&#xA;4,2,4,3&#xA;5,2,5,4&#xA;5,3,3,1&#xA;5,4,7,2&#xA;6,1,3,3&#xA;6,5,4,1&#xA;6,2,5,2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And here's the R output:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Compiling model graph&#xA;   Resolving undeclared variables&#xA;   Allocating nodes&#xA;   Graph Size: 352&#xA;&#xA;Error in jags.model(model.file, data = data, inits = inits, n.chains = n.chains,  : &#xA;  Invalid initial values&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="4110" LastActivityDate="2011-05-01T00:32:48.660" Title="Debugging JAGS and BUGS" Tags="&lt;r&gt;&lt;bugs&gt;&lt;jags&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="2" />
  <row Id="10186" PostTypeId="1" AcceptedAnswerId="10197" CreationDate="2011-04-30T23:40:13.477" Score="4" ViewCount="100" Body="&lt;p&gt;The setup is that I'm trying to understand how a computer program works, so I'm capturing some numbers every time a function is called.  For example, I might be capturing the number of branches taken and the number of branches incorrectly predicted.  During the course of running the program, a particular function might get called twenty or thirty thousand times.  I have a fair amount of control over how many times a given function is called.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My initial plan was to calculate a mean and standard deviation using those 20-30k data points as my sample.  However, my (computer science) professor suggested that I needed to rerun the experiment several times in order to calculate a standard deviation.  So I would run my script five or six times, calculating a mean each time.  Then I would use those five or six values to calculate a standard deviation.  That doesn't make a lot of sense to me - it seems to me that if I want to understand how a given function is behaving, I should treat the data from each function call as a data point, and that the professor's method more or less throws away a lot of data.&lt;br&gt;&#xA;However, I'm thinking that I may be making an unwarranted assumption that one run of the program is like another.  In this case, I guess that running and presenting both sets of numbers would be a good, as I would capture how the functions behave at each call, and also see if there is behavior difference across calls.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So getting around to the question, is my initial impulse to use each function call as my data set correct/better, or should I calculate the data both ways and present both numbers?&lt;/p&gt;&#xA;" OwnerUserId="527" LastActivityDate="2011-05-01T09:10:52.967" Title="Repeating an experiment - more valuable than sample size?" Tags="&lt;homework&gt;&lt;experiment-design&gt;" AnswerCount="3" CommentCount="1" FavoriteCount="1" />
  <row Id="10188" PostTypeId="2" ParentId="10186" CreationDate="2011-05-01T00:28:02.547" Score="5" ViewCount="" Body="&lt;p&gt;I assume that you analyse a stochastic algorithm. Repeated runs of the program may have different sources of variability than repeated function evaluations within a single run.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An example: the program may initialise a random number generator with the same seed in every run, which will make the results of repeated runs identical, but the function evaluations pseudo-random.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastActivityDate="2011-05-01T00:28:02.547" />
  <row Id="10189" PostTypeId="2" ParentId="10185" CreationDate="2011-05-01T00:32:48.660" Score="1" ViewCount="" Body="&lt;p&gt;You could try running the same model in WinBugs or OpenBugs. They generally give slightly more detailed error messages, so you might get something more useful in this specific case, too.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastActivityDate="2011-05-01T00:32:48.660" />
  <row Id="10190" PostTypeId="2" ParentId="10186" CreationDate="2011-05-01T01:58:23.390" Score="5" ViewCount="" Body="&lt;p&gt;GaBorgulya said it quite well (+1), but I'd like to present a more conceptual perspective too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If GaBorgulya's assumptions are correct, you can treat the random seed as a blocking factor. Imagine that each run of the program is a person and that for each person you measure blood pressure over time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could increase your sample size by taking more measurements on each person (like looking at all those thousands of data points) or by looking at more persons (like running the program more times). The former will tell you about the error &lt;strong&gt;within&lt;/strong&gt; persons, and the latter will tell you about error &lt;strong&gt;between&lt;/strong&gt; persons.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because you are running a computer program and your program can be repeated if the seed is saved, there is no error within the &quot;person&quot; (the particular run of the program); whatever output you get is the true, complete output for that particular seed. You would like to know how much noise is created by the random seed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, your ability to adjust how many times a particular function is called blurs my definitions of error within and error between. It may help to discuss in a bit more detail what those functions are doing.&lt;/p&gt;&#xA;" OwnerUserId="3874" LastActivityDate="2011-05-01T01:58:23.390" />
  <row Id="10191" PostTypeId="1" CreationDate="2011-05-01T04:02:02.753" Score="4" ViewCount="65" Body="&lt;p&gt;It is said that if the plots of the hypothetical responses are not parallel, but crossed, there is interaction. Suppose we have two factors. Is it possible that the plots cross but we do not have interaction? That is more reasonable when the plots are close to each other.&lt;br&gt;&#xA;I noticed the converse is true. We may have interaction even if neither the curves of factor A on factor B nor factor B on factor A intersect. In a book I read this happens when the interaction is removable, which means that there is another cognate independent variable.&lt;/p&gt;&#xA;" OwnerUserId="3454" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-01T08:05:59.720" LastActivityDate="2011-05-01T10:25:13.650" Title="Counterexample for interaction and parallel curves?" Tags="&lt;anova&gt;&lt;interaction&gt;" AnswerCount="3" FavoriteCount="1" />
  <row Id="10192" PostTypeId="1" AcceptedAnswerId="10215" CreationDate="2011-05-01T04:36:57.863" Score="3" ViewCount="90" Body="&lt;p&gt;Wikipedia says that the name of concept comes from physics, but I cannot find any similarity between these two concepts.&lt;/p&gt;&#xA;" OwnerUserId="4416" LastActivityDate="2011-05-01T21:49:03.560" Title="How to understand moments for a random variable?" Tags="&lt;moments&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="2" />
  <row Id="10193" PostTypeId="1" AcceptedAnswerId="10203" CreationDate="2011-05-01T05:24:17.993" Score="4" ViewCount="65" Body="&lt;p&gt;I have a set of sessions and urls that have been accessed in each of these sessions and frequencies with which they have been accessed. I've put them in a matrix-like representation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Imagine I have the following &quot;Pageview matrix&quot;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;COLUMN HEADINGS&#xA;&#xA;books placement resources br aca&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Each row represents a session.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an example of the records:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;4 5 0 2 2&#xA;1 2 1 7 3&#xA;1 3 6 1 6&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;saved in a &lt;code&gt;.txt&lt;/code&gt; file&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can I give this as an input to a k-means program and obtain clusters based on the highest frequency of occurrence? How do I use it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If not k-means, what other cluster method can I use?&lt;/p&gt;&#xA;" OwnerUserId="4402" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-01T11:16:55.293" LastActivityDate="2011-05-01T11:16:55.293" Title="Clustering elements by access counts in sessions" Tags="&lt;clustering&gt;" AnswerCount="2" FavoriteCount="2" />
  <row Id="10194" PostTypeId="2" ParentId="10193" CreationDate="2011-05-01T06:28:12.233" Score="4" ViewCount="" Body="&lt;p&gt;Let me try to answer your questions in parts:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) You can do k-mean cluster analysis using the dataset. But how you use the result of cluster analysis will be based on the problem that you are trying to solve using the cluster analysis. I used cluster analysis using clickstream data. But my dataset was bit different from theta of yours. I took variables(columns) like pageviews,time on page, bounce rate, etc &amp;amp; urls as row variables. The idea was try to segment the urls into different clusters &amp;amp; then try to find the meaningful attributes of these particular clusters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) There are mainly 2 types of cluster analysis: hierarchical &amp;amp; partition clustering. K-mean falls within partition clustering. The &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0387781889&quot; rel=&quot;nofollow&quot;&gt;book&lt;/a&gt; gives details of different king of clustering techniques available.&lt;/p&gt;&#xA;" OwnerUserId="4278" LastActivityDate="2011-05-01T06:28:12.233" CommentCount="1" />
  <row Id="10195" PostTypeId="2" ParentId="10192" CreationDate="2011-05-01T06:36:16.847" Score="1" ViewCount="" Body="&lt;p&gt;Moments gives information about the statistical distribution. We judge one dataset over other based on moments of the dataset (e.g. difference between means(1st moment) of the 2 dataset)&lt;/p&gt;&#xA;" OwnerUserId="4278" LastActivityDate="2011-05-01T06:36:16.847" />
  <row Id="10196" PostTypeId="2" ParentId="10162" CreationDate="2011-05-01T06:36:43.740" Score="3" ViewCount="" Body="&lt;p&gt;Here is a simple recipe that may help you get started writing code and testing ideas...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's assume you have monthly data recorded over several years, so you have 36 values. Let's also assume that you only care about predicting one month (value) in advance.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Exploratory data analysis: Apply some of the traditional time series analysis methods to estimate the lag dependence in the data (e.g. auto-correlation and partial auto-correlation plots, transformations, differencing).&#xA;Let's say that you find a given month's value is correlated with the past three month's data but not much so beyond that.&lt;/li&gt;&#xA;&lt;li&gt;Partition your data into training and validation sets: Take the first 24 points as your training values and the remaining points as the validation set.&lt;/li&gt;&#xA;&lt;li&gt;Create the neural network layout: You'll take the past three month's values as inputs and you want to predict the next month's value. So, you need a neural network with an input layer containing three nodes and an output layer containing one node. You should probably have a hidden layer with at least a couple of nodes. Unfortunately, picking the number of hidden layers, and their respective number of nodes, is not something for which there are clear guidelines. I'd start small, like 3:2:1.&lt;/li&gt;&#xA;&lt;li&gt;Create the training patterns: Each training pattern will be four values, with the first three corresponding to the input nodes and the last one defining what the correct value is for the output node. For example, if your training data are values $$x_1,x_2\dots,x_{24}$$ then $$pattern 1:  x_1,x_2,x_3,x_4$$ $$pattern 2: x_2,x_3,x_4,x_5$$ $$\dots$$ $$pattern 21: x_{21},x_{22},x_{23},x_{24}$$ &lt;/li&gt;&#xA;&lt;li&gt;Train the neural network on these patterns&lt;/li&gt;&#xA;&lt;li&gt;Test the network on the validation set (months 25-36): Here you will pass in the three values the neural network needs for the input layer and see what the output node gets set to. So, to see how well the trained neural network can predict month 32's value you'll pass in values for months 29, 30, and 31&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;This recipe is obviously high level and you may scratch your head at first when trying to map your context into different software libraries/programs. But, hopefully this sketches out the main point: you need to create training patterns that reasonably contain the correlation structure of the series you are trying to forecast. And whether you do the forecasting with a neural network or an ARIMA model, the exploratory work to determine what that structure is is often the most time consuming and difficult part.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my experience, neural networks can provide great classification and forecasting functionality but setting them up can be time consuming. In the example above, you may find that 21 training patterns is not enough; different input data transformations lead to a better/worse forecasts; varying the number of hidden layers and hidden layer nodes greatly affects forecasts; etc.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I highly recommend looking at the &lt;a href=&quot;http://www.neural-forecasting-competition.com/index.htm&quot; rel=&quot;nofollow&quot;&gt;neural_forecasting&lt;/a&gt; website, which contains tons of information on neural network forecasting competitions. The &lt;a href=&quot;http://www.neural-forecasting-competition.com/motivation.htm&quot; rel=&quot;nofollow&quot;&gt;Motivations&lt;/a&gt; page is especially useful.&lt;/p&gt;&#xA;" OwnerUserId="1080" LastActivityDate="2011-05-01T06:36:43.740" CommentCount="2" />
  <row Id="10197" PostTypeId="2" ParentId="10186" CreationDate="2011-05-01T09:10:52.967" Score="6" ViewCount="" Body="&lt;p&gt;For every experiment, one has explicit or implicit assumptions about all the boundary conditions that should not influence the experiment's outcome. When you run your program, maybe time of day should have no effect, it should not matter who presses the keyboard key to start the program, same for CPU architecture etc. You assume these things are unimportant because you have a well-established theory about what goes on when a computer runs a program.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, some of these assumptions could be wrong because the theory is incomplete, only partly valid, or because you missed a connection between an unimportant boundary condition and a known influence (think of the &lt;a href=&quot;http://www.ibiblio.org/harris/500milemail.html&quot; rel=&quot;nofollow&quot; title=&quot;email only within 500 miles&quot;&gt;email only within 500 miles anecdote&lt;/a&gt;, also check its FAQ). Repeating your experiment allows you to vary at least some of the boundary conditions and verify empirically that they indeed have no effect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Quite often, replications reveal that a specific combination of circumstances that were not considered important can influence experiment's a great deal, see e.g., this &lt;a href=&quot;http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_lehrer&quot; rel=&quot;nofollow&quot; title=&quot;New Yorker: the truth wears off&quot;&gt;New Yorker: the truth wears off&lt;/a&gt; article.&lt;/p&gt;&#xA;" OwnerUserId="1909" LastActivityDate="2011-05-01T09:10:52.967" />
  <row Id="10198" PostTypeId="2" ParentId="10191" CreationDate="2011-05-01T09:15:10.617" Score="2" ViewCount="" Body="&lt;p&gt;Yes, if the true (hypothetical) responses are not parallel there is interaction. Not parallel, however, does not necessarily mean that the segments cross. When you investigate interaction the sampling error may lead to different results in the sample than in the population, so it's useful to calculate confidence intervals or credibility intervals for the extent of the possible interaction. The extent of the interaction depends on the scales of the variables, in special cases (removable interaction) there is a transformation when the effects are additive and there is no interaction.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-05-01T09:20:14.450" LastActivityDate="2011-05-01T09:20:14.450" />
  <row Id="10199" PostTypeId="2" ParentId="10191" CreationDate="2011-05-01T09:28:25.543" Score="3" ViewCount="" Body="&lt;p&gt;To me it seems like you (and many books probably) are confusing the empirical level with the theoretical level: The &lt;a href=&quot;http://stats.stackexchange.com/questions/5617/what-is-the-null-hypothesis-for-interaction-in-a-two-way-anova/5622#5622&quot; title=&quot;null hypothesis of interaction&quot;&gt;null hypothesis of an interaction effect&lt;/a&gt; in a two-way ANOVA is defined on the theoretical level using the cell expected values $\mu_{jk}$ (and not response values): there is an interaction if (and only if) the lines connecting the $\mu_{jk}$ in a diagram are exactly parallel. Note that &quot;not parallel&quot; is not the same as &quot;lines cross&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the empirical side, we do not have the $\mu_{jk}$, but can only plot their estimates, the cell means $M_{jk}$. Even if the null hypothesis is true, their connecting lines will almost never be exactly parallel due to measurement error. Conversely, even if the alternative hypothesis is true, they could be almost parallel for the same reason. A measure for the degree to which deviation from parallelity of the $M_{jk}$ indicates interaction is the ANOVA's corresponding F-value.&lt;/p&gt;&#xA;" OwnerUserId="1909" LastActivityDate="2011-05-01T09:28:25.543" />
  <row Id="10200" PostTypeId="1" CreationDate="2011-05-01T09:38:50.040" Score="6" ViewCount="68" Body="&lt;p&gt;I remember hearing an argument that if a certain population of people has a mean IQ of 110 rather than the typical 100, it will have far more people of IQ 150 than a similarly-sized group from the general population.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if both groups' IQs are normally distributed and have the same standard deviation of 15, we expect about 0.2% of people from the high-IQ group to be over 150, and 0.02% from the general population.  There are about ten times as many &quot;genius-level&quot; people from the high-IQ group.  This is despite there being only three times as many people of IQ 120 or more in the high-IQ group, and only twice as many people of IQ 110 or more.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, when we meet someone with IQ over 150, we can strongly suspect they're from the high-IQ group, even though on average that group does not have a huge advantage.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a special name for this effect?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Similarly, if two populations have the same mean, but population A has a higher variance than population B, there will be more data points above a certain high threshold from population A.  (I have heard this argument given to explain the high proportion of men as opposed to women in the highest tiers of mathematical achievement.  It was claimed that men and women had the same mean ability, but men had higher variance.)  Does this effect also have a name?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I apologize for the somewhat controversial nature of the examples.  I'm only interested in the names for the effects, not in IQ and mathematical ability in different sorts of groups.  I simply cited these examples because that's the context under which I heard these phenomena described.&lt;/p&gt;&#xA;" OwnerUserId="2665" LastActivityDate="2011-05-01T11:33:22.337" Title="Is there a name for the high sensitivity of frequency of extreme data points to the mean of a normal distribution?" Tags="&lt;normal-distribution&gt;&lt;outliers&gt;&lt;terminology&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="1" />
  <row Id="10201" PostTypeId="2" ParentId="10191" CreationDate="2011-05-01T09:55:03.493" Score="2" ViewCount="" Body="&lt;p&gt;This depends on what is meant by &quot;interaction&quot;.  If the data have no noise - the plot is literally just two parallel lines, then there is certainly no interaction, we know this deductively, without any need for statistics.  Secondly if the lines are not parallel, then we know deductively that there is interaction.  So there is no counter example if there is no noise.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But if there is noise (or error), then there is basically more than one possible place that the &quot;noiseless&quot; or &quot;true&quot; lines could be.  It is also possible for the true lines to be parallel but if the noise is big enough and you get an &quot;unlucky&quot; sample of noise, then the noisy lines will cross.  Just how unlucky depends on how &quot;non-parallel&quot; the two &quot;true lines&quot; are and how many units have been sampled.  Consider the OLS case, the lines are generated by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$y_{i}=x_{i}^{T}\beta_{true}+n_{i}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $\beta_{true}$ is a 4-D vector with the intercept for group 1, the offset for group 2, the slope for group 1 and the offset to the slope for group 2&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now you fit an OLS to the observed data, and you get&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\beta_{OLS}=(X^{T}X)^{-1}X^{T}Y=(X^{T}X)^{-1}X^{T}(X\beta_{true}+n)=\beta_{true}+(X^{T}X)^{-1}X^{T}n$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So by a careful choice of the noise we can make the OLS estimates be basically anything.  So I don't have to invert a $4\times 4$ matrix, I will specialise to the case where both intercepts are equal to zero, and we have&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$y_{ij}=\beta_{1}x_{ij}+\beta_{2}x_{i2}I(j=2)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And then &#xA;$$(X^{T}X)^{-1}=\frac{1}{\left(\sum_{i}x_{i1}^{2}\right)\left(\sum_{i}x_{i2}^{2}\right)}\begin{pmatrix} \sum_{i}x_{i1}^{2}+\sum_{i}x_{i2}^{2} &amp;amp; -\sum_{i}x_{i2}^{2} \\ -\sum_{i}x_{i2}^{2} &amp;amp; \sum_{i}x_{i2}^{2}&#xD;&#xA;\end{pmatrix}$$&#xA;$$=\frac{1}{\sum_{i}x_{i1}^{2}}\begin{pmatrix} 1 &amp;amp; -1 \\ -1 &amp;amp; 1\end{pmatrix}&#xD;&#xA;+\frac{1}{\sum_{i}x_{i2}^{2}}\begin{pmatrix} 1 &amp;amp; 0 \\ 0 &amp;amp; 0\end{pmatrix}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now for $X^{T}n$ we have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$X^{T}n=\sum_{i}x_{i2}n_{i2}\begin{pmatrix} 1\\1\end{pmatrix}&#xD;&#xA;+\sum_{i}x_{i1}n_{i1}\begin{pmatrix} 1\\0\end{pmatrix}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And so the total error from the regression is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{\sum_{i}x_{i2}n_{i2}+\sum_{i}x_{i1}n_{i1}}{\sum_{i}x_{i2}^{2}}\begin{pmatrix} 1\\0\end{pmatrix}&#xD;&#xA;+\frac{\sum_{i}x_{i1}n_{i1}}{\sum_{i}x_{i1}^{2}}\begin{pmatrix} 1\\-1\end{pmatrix}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now if the true slopes are parallel, so that $\beta_{2,true}=0$, then the OLS estimates will be:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\hat{\beta}_{1}=\beta_{1,true}+\frac{\sum_{i}x_{i2}n_{i2}+\sum_{i}x_{i1}n_{i1}}{\sum_{i}x_{i2}^{2}}+\frac{\sum_{i}x_{i1}n_{i1}}{\sum_{i}x_{i1}^{2}}$$&#xA;$$\hat{\beta}_{2}=-\frac{\sum_{i}x_{i1}n_{i1}}{\sum_{i}x_{i1}^{2}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now this shows that the OLS estimate can indeed lead to erroneous interactions, just choose the &quot;true&quot; noise such that it is highly correlated with $x_{i1}$ - essentially you need to violate one of the assumptions of OLS, non-heteroscedasticity of the noise.  So if you generate data according to:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$y_{i1}=x_{i1}(\beta_{1,true}+n_{i1})$$&#xA;$$y_{i2}=x_{i2}\beta_{1,true}+n_{i2}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And then try to fit an interaction model using OLS of $y$ on $x$ with an interaction, you will find a significant result, even though the true betas are the same.  The plots will cross because of the fanning in the first group.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One example data set (true beta is 2 and noise was generated from standard normal).  You get a t-statistic above 10 for the interaction effect:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\begin{array}{c|c}&#xD;&#xA;group	&amp;amp;	y	&amp;amp;	x	\\&#xD;&#xA;1	&amp;amp;	1.282817715	&amp;amp;	1	\\&#xD;&#xA;1	&amp;amp;	2.026032115	&amp;amp;	2	\\&#xD;&#xA;1	&amp;amp;	5.9786882	&amp;amp;	3	\\&#xD;&#xA;1	&amp;amp;	22.1588319	&amp;amp;	7	\\&#xD;&#xA;2	&amp;amp;	16.28587668	&amp;amp;	9	\\&#xD;&#xA;2	&amp;amp;	15.12007527	&amp;amp;	6	\\&#xD;&#xA;2	&amp;amp;	9.566273403	&amp;amp;	5	\\&#xD;&#xA;\end{array}$$&lt;/p&gt;&#xA;" OwnerUserId="2392" LastEditorUserId="2392" LastEditorDisplayName="" LastEditDate="2011-05-01T10:25:13.650" LastActivityDate="2011-05-01T10:25:13.650" />
  <row Id="10202" PostTypeId="2" ParentId="10200" CreationDate="2011-05-01T10:21:02.827" Score="1" ViewCount="" Body="&lt;p&gt;This is not an answer to the question, but may be of interest. The question gave three ratios of number of people in the mean=110 and mean=100 populations having IQ above a given threshold (ratio(IQ=150)≈10, ratio(120)≈3, ratio(110)≈2). The R code below plots the ratio as a function of IQ.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;IQ = seq(0, 200, length.out=100)&#xA;c100 = pnorm(IQ, mean=100, sd=15)&#xA;c110 = pnorm(IQ, mean=110, sd=15)&#xA;ratio = (1 - c110) / (1 - c100)&#xA;plot(ratio ~ IQ); abline(h=c(0, 10, 20, 30, 40, 50, 60))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/PdCvR.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="3911" LastActivityDate="2011-05-01T10:21:02.827" />
  <row Id="10203" PostTypeId="2" ParentId="10193" CreationDate="2011-05-01T11:15:14.063" Score="2" ViewCount="" Body="&lt;p&gt;So, taking your comment into consideration, you want to make clusters of entries which group those entries which were frequently co-accessed?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If so, well, you need to decide how to measure this co-access, i.e. transform it into a dissimilarity, and this is a fairly nontrivial task.&lt;br&gt;&#xA;The simple measure is to count, for each pair of entries, sessions in which they both were accessed and divide by the count of session in which any of them was accessed. The resulting matrix will be similarity one, so you can for instance subtract each cell from one and feed the result to the clustering algorithm of your choice. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course this measure does not take the access counts during session into account, so you'll probably need something more complex; one idea (simple extension of the trivial one) may be to sum minimal smaller of counts from each session when particular pair co-occurs and divide the whole by the sum of total number of accesses to the both entries.&lt;br&gt;&#xA;However, you should try to make this measure on your own taking into account the specificity of this particular problem.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2011-05-01T11:15:14.063" CommentCount="1" />
  <row Id="10204" PostTypeId="1" CreationDate="2011-05-01T11:20:48.103" Score="3" ViewCount="113" Body="&lt;p&gt;I have measured frequency of a certain behavior on 15 individuals. &#xA;I would like to create two groups based on the amount of this behaviour that was observed (i.e., a group exhibiting high levels of the behaviour and a group exhibiting low levels of the behaviour).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to see whether this new binary variable predicts a dependent variable that I have measured.&lt;/p&gt;&#xA;" OwnerUserId="4420" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-02T07:54:45.727" LastActivityDate="2011-05-02T07:54:45.727" Title="How to split a numeric variable into a binary low-high variable" Tags="&lt;distributions&gt;&lt;data-transformation&gt;" AnswerCount="2" CommentCount="8" />
  <row Id="10206" PostTypeId="1" AcceptedAnswerId="10212" CreationDate="2011-05-01T13:06:32.093" Score="3" ViewCount="101" Body="&lt;p&gt;Cumbersome technical assumptions (e.g., mixing properties) are used in the literature to prove Central Limit Theorems for dependent sequences. I sketched a proof that does not require any of these technical assumptions. Can you help me figure out what is wrong with this proof? The proof is at: &lt;a href=&quot;http://www.statlect.com/central_limit_theorem_for_correlated_sequences.htm&quot; rel=&quot;nofollow&quot;&gt;http://www.statlect.com/central_limit_theorem_for_correlated_sequences.htm&lt;/a&gt;. Thanks in advance to all those who will be so generous and patient to read it.&lt;/p&gt;&#xA;" OwnerUserId="4422" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-05-02T02:42:42.530" LastActivityDate="2011-05-02T02:42:42.530" Title="Conditions for Central Limit Theorem for dependent sequences" Tags="&lt;probability&gt;&lt;stochastic-processes&gt;&lt;central-limit-theorem&gt;&lt;stationarity&gt;" AnswerCount="1" CommentCount="3" FavoriteCount="1" />
  <row Id="10207" PostTypeId="1" CreationDate="2011-05-01T13:45:33.557" Score="2" ViewCount="76" Body="&lt;p&gt;How to find the number of runs in the following&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;aaaabbabbbaabba&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;bbaaaaaabbbbaaaaaaa&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="4423" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-05-01T17:17:39.257" LastActivityDate="2011-05-01T17:17:39.257" Title="How to find the number of runs?" Tags="&lt;homework&gt;&lt;algorithms&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="1" />
  <row Id="10208" PostTypeId="2" ParentId="10207" CreationDate="2011-05-01T14:08:24.853" Score="2" ViewCount="" Body="&lt;p&gt;Quoting &lt;a href=&quot;http://en.wikipedia.org/wiki/Wald-Wolfowitz_runs_test&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Wald-Wolfowitz_runs_test&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;A &quot;run&quot; of a sequence is a maximal non-empty segment of the sequence consisting of adjacent equal elements. For example, the sequence &quot;++++−−−+++−−++++++−−−−&quot; consists of six runs, three of which consist of +'s and the others of −'s.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In your case&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1) aaaabbabbbaabba      2) bbaaaaaabbbbaaaaaaa&#xA;   1   2 34  5 6 7         1 2     3   4&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="3911" LastActivityDate="2011-05-01T14:08:24.853" />
  <row Id="10209" PostTypeId="2" ParentId="10207" CreationDate="2011-05-01T14:35:51.453" Score="2" ViewCount="" Body="&lt;p&gt;If you want do this in R, check out the &lt;code&gt;rle&lt;/code&gt; function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; ext &amp;lt;- function(x) {strsplit(x, &quot;&quot;)[[1]]}&#xA;&amp;gt; x &amp;lt;- ext(&quot;aaaabbabbbaabba&quot;)&#xA;&amp;gt; y &amp;lt;- ext(&quot;bbaaaaaabbbbaaaaaaa&quot;)&#xA;&amp;gt; length(rle(x)$lengths)&#xA;[1] 7&#xA;&amp;gt; length(rle(y)$lengths)&#xA;[1] 4&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;See also this related question on &lt;a href=&quot;http://stackoverflow.com/questions/1502910/how-can-i-count-runs-in-r&quot;&gt;Stack Overflow on counting runs in R&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-01T14:44:20.403" LastActivityDate="2011-05-01T14:44:20.403" />
  <row Id="10210" PostTypeId="1" CreationDate="2011-05-01T16:08:12.030" Score="3" ViewCount="75" Body="&lt;p&gt;Let's say you want to cluster some objects, say documents, or sentences, or images.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the technical side, you first represent these object somehow so that you could calculate distance between them, and then you feed those representations to some clustering algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Externally, however, you just want to group similar (in &lt;em&gt;some sense&lt;/em&gt; -- and that's where things become pretty vague for me) objects together. For example, in case of sentences we want for clusters to contain sentences about similar topic/concept; we feel that sentences &quot;oh look at this pic of a cute lolcat&quot; and &quot;facebook revealed new shiny feature tonight&quot; should be in different clusters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are the usual approaches for measuring this &quot;external&quot; quality of clustering? I.e. we want to measure how well our clustering procedure groups initial objects (sentences, images); we're not interested in internal measures (like averaged cluster radius, clusters sparseness), since those measures deal with objects' representations, not with real objects. Meaning, the chosen representation may be awful, and even if internal measures is great, externally we'll end up with clusters that are complete junk from our vague, subjective, &quot;some sense&quot;-ish point of view.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S. Having limited knowledge in clustering domain, I suspect I may be asking about really obvious thing, or my terminology may sound strange to clustering experts. If so, please advice what should I read on the subject.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.P.S. Just in case, I asked the very same question on Quora: &lt;a href=&quot;http://www.quora.com/How-to-evaluate-external-quality-of-clustering&quot; rel=&quot;nofollow&quot;&gt;http://www.quora.com/How-to-evaluate-external-quality-of-clustering&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="4425" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-01T17:16:20.670" LastActivityDate="2011-05-01T17:16:20.670" Title="How to evaluate &quot;external&quot; quality of clustering?" Tags="&lt;clustering&gt;&lt;data-mining&gt;" CommentCount="3" />
  <row Id="10211" PostTypeId="1" CreationDate="2011-05-01T17:28:38.533" Score="8" ViewCount="94" Body="&lt;p&gt;Hello everyone,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am running a structural equation model (SEM) in Amos 18. I was looking for 100 participants for my experiment (used loosely), which was deemed to be probably not enough to conduct successful SEM. I've been told repeatedly that SEM (along with EFA, CFA) is a &quot;large sample&quot; statistical procedure. Long story short, I didn't make it to 100 participants (what a surprise!), and only have 42 after excluding two problematic data points. Out of interest, I tried the model anyway, and to my surprise, it seemed to fit very well! CFI &gt;.95, RMSEA &amp;lt; .09, SRMR &amp;lt;.08. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The model is not simple, in fact, I would say it is relatively complex. I have two latent variables, one with two observed and the other with 5 observed variables. I also have four additional observed variables in the model. There are numerous relationships between the variables, indirect and direct, with some variables being endogenous to four others, as an example. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am somewhat new to SEM; however, two individuals that I know who are quite familiar with SEM tell me that as long as the fit indicies are good, the effects are interpretable (as long as they are significant) and there is nothing significantly &quot;wrong&quot; with the model. I know some fit indicies are biased for or against small samples in terms of suggesting good fit, but the three that I mentioned earlier seem fine, and I believe not similarly biased. To test for indirect effects I am using bootstrapping (2000 samples or so), 90 percent bias corrected confidence, monte carlo. An additional note is that I am running three different SEMs for three different conditions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have two questions that I would like some of you to consider and please reply to if you have something to contribute:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Are there any significant weaknesses to my model that are not demonstrated by the fit indices? The small sample will be highlighted as a weakness of the study, but I am left wondering if there is some huge statistical problem that I am completely oblivious to. I plan on getting another 10-20 participants in the future, but this will still leave me with a relatively small sample for such analyses. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Are there any problems with my use of bootstrapping given my small sample, or the context in which I am using it?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I hope these questions are not too &quot;basic&quot; for this forum. I have read a number of chapters on SEM and related matters, but I find people are very dispersed in terms of opinions in this area! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cheers&lt;/p&gt;&#xA;" OwnerUserId="3262" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-01T20:17:22.737" LastActivityDate="2011-05-01T20:17:22.737" Title="Complications of having a very small sample in a structural equation model" Tags="&lt;modeling&gt;&lt;sample-size&gt;&lt;bootstrap&gt;&lt;sem&gt;" CommentCount="6" FavoriteCount="5" />
  <row Id="10212" PostTypeId="2" ParentId="10206" CreationDate="2011-05-01T18:31:30.110" Score="4" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Additional conditions are needed.&lt;/strong&gt; (&lt;em&gt;A near-proof of this fact is that many incredibly smart individuals have been thinking deeply about these issues for over 100 years. It is highly unlikely that something like this would have escaped all of them.&lt;/em&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First of all, note that the formula for $V$ that you give is part of the &lt;strong&gt;conclusion&lt;/strong&gt; of the associated central limit theorem. See, for example, &lt;strong&gt;Theorem 7.6&lt;/strong&gt; on pages 416&amp;ndash;417 of R. Durrett, &lt;em&gt;Probability: Theory and Examples&lt;/em&gt;, 3rd. ed., which based on your link, you appear to have access to.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At any rate, here is a simple counterexample to your claim.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Let $X_0$ equal $+1$ with probability $1/2$ and $-1$ with probability $1/2$. Define $X_n = (-1)^n X_0$. Then $\{X_n\}$ is a stationary ergodic process with mean 0 and variance 1, but the Central Limit Theorem fails.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The properties of stationarity and ergodicity should be pretty easy to see as we can construct this process by defining a function over the states of a two-state Markov chain with stationary probability measure $\pi(x) = 1/2$ for $x \in \{0,1\}$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Observe that this process yields a sequence of the form $-X_0, X_0, -X_0, \ldots$ and so &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Even without appealing to any notions about ergodicity, it is easy to see that $\newcommand{\e}{\mathbb{E}}\bar{X}_n \to \e X_0 = 0$ almost surely, and,&lt;/li&gt;&#xA;&lt;li&gt;$\newcommand{\Var}{\mathbb{V}\mathrm{ar}}\Var(S_n) = 0$ if $n$ is even and $1$ if $n$ is odd. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;This already is enough to conclude that there is &lt;strong&gt;no way&lt;/strong&gt; that any rescaling of $S_n$ can make it converge in distribution to a normal random variable. In fact, for &lt;strong&gt;every&lt;/strong&gt; function $f$ such that $f(n) \to \infty$, $S_n / f(n) \to 0$ almost surely &lt;em&gt;no matter how slowly $f$ diverges&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note also that this example should make it clear that the formula for $V$ is a conclusion of the theorem. Indeed, for the example above,&#xA;$$&#xD;&#xA;V_n = 1 + 2 \sum_{i = 1}^n \e X_0 X_i = \left\{&#xD;&#xA;\begin{array}{rl}&#xD;&#xA;-1, &amp;amp; n \text{ odd}, \\&#xD;&#xA;1, &amp;amp; n \text{ even},&#xD;&#xA;\end{array}&#xD;&#xA;\right.&#xD;&#xA;$$&#xA;which, of course, (a) makes no sense as a variance, (b) does not have a limit, and (c) is not asymptotically equivalent to $\Var(S_n)$. (&lt;strong&gt;NB&lt;/strong&gt;: I use a slightly different form for $V_n$ than you do where mine matches that given in Durrett.)&lt;/p&gt;&#xA;" OwnerUserId="2970" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-05-01T18:48:08.857" LastActivityDate="2011-05-01T18:48:08.857" CommentCount="3" />
  <row Id="10213" PostTypeId="1" AcceptedAnswerId="10216" CreationDate="2011-05-01T19:37:56.640" Score="2" ViewCount="204" Body="&lt;p&gt;Hi there! I'm doing some reading on topic modeling (with Latent Dirichlet Allocation) which makes use of Gibbs sampling. As a newbie in statistics -- well, I know things like binomials, multinomials, priors etc -- I find it difficult to grasp how Gibbs sampling works. Can someone please explain it in simple English and/or using simple examples? (If you are not familiar with topic modeling, any examples will do.) Thank you all in advance.&lt;/p&gt;&#xA;" OwnerUserId="4429" LastActivityDate="2011-05-01T23:36:49.803" Title="Can someone explain Gibbs sampling in very simple words?" Tags="&lt;modeling&gt;&lt;sampling&gt;&lt;conditional-probability&gt;&lt;gibbs&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="3" />
  <row Id="10214" PostTypeId="2" ParentId="10133" CreationDate="2011-05-01T20:20:58.663" Score="1" ViewCount="" Body="&lt;p&gt;I am a novice data miner as well, but may I suggest that exploratory data analysis is always a good first step?  I would see if items can be assigned some sort of 'priority value' which can serve to predict how early they appear in the cart, as such a result may allow you to use simpler models.    Something as simple as a linear regression on (#order in cart/#number of items in cart) for all carts possessing item X will give you an idea of whether this is possible.  Suppose you find that a certain proportion of items always appear early, or later, and some seem to be completely random: this would guide you in your later model-building.&lt;/p&gt;&#xA;" OwnerUserId="3567" LastActivityDate="2011-05-01T20:20:58.663" />
  <row Id="10215" PostTypeId="2" ParentId="10192" CreationDate="2011-05-01T20:24:10.717" Score="3" ViewCount="" Body="&lt;p&gt;If you have a linear rod, the center of gravity is the first moment (the expected value), and the moment of rotational inertia about the center of gravity is the variance.  (A rod with centrally located mass will have less inertia than a rod with heavy concentrations of mass at the tips.)&lt;/p&gt;&#xA;" OwnerUserId="3567" LastEditorUserId="3567" LastEditorDisplayName="" LastEditDate="2011-05-01T21:49:03.560" LastActivityDate="2011-05-01T21:49:03.560" />
  <row Id="10216" PostTypeId="2" ParentId="10213" CreationDate="2011-05-01T20:52:40.543" Score="16" ViewCount="" Body="&lt;p&gt;You are a dungeonmaster hosting Dungeons and Dragons and a player casts 'Spell of &#xA;Eldritch Chaotic Weather (SECW).  You've never heard of this spell before, but it turns out it is quite involved.  The player hands you a dense book and says, 'the effect of this spell is that one of the events in this book occurs.'  The book contains a whopping 1000 different effects, and what's more, the events have different 'relative probabilities.'  The book tells you that the most likely event is 'fireball'; all the probabilities of the other events are described relative to the probability of 'fireball'; for example: on page 155, it says that 'duck storm' is half as likely as 'fireball.'&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How are you, the dungeonmaster, to sample a random event from this book?  Here's how you can do it: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;The accept-reject algorithm:&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Roll a d1000 to decide a 'candidate' event.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Suppose the candidate event is 44% as likely as the most likely event, 'fireball'.  Then &lt;em&gt;accept&lt;/em&gt; the candidate with probability 44%.  (Roll a d100, and accept if the roll is 44 or lower.  Otherwise, go back to step 1 until you accept an event.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) The accepted event is your random sample.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The accept-reject algorithm is guaranteed to sample from the distribution with the specified relative probabilities.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After much dice rolling you finally end up accepting a candidate: 'summon frog'.  You breathe a sigh of relief as you now you can get back to the (routine in comparison) business of handling the battle between the troll-orcs and dragon-elves.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, not to be outdone, another player decides to cast 'Lv. 2 arcane cyber-effect storm.' For this spell, two different random effects occur: a randomly generated attack, and a randomly generated character buff.  The manual for this spell is so huge that it can only fit on a CD.  The player boots you up and shows you a page.  Your jaw drops: the &lt;em&gt;entry&lt;/em&gt; for each attack is about as large a the manual for the previous spell, because it lists a relative probability for each possible accompanying buff&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;'Cupric Blade'&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;The most likely buff accompanying this attack is 'Hotelling aura'&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;'Jackal Vision' is 33% as likely to accompany this attack as 'Hotelling aura'&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;'Toaster Ears' is 20% as likely to accompany this attack as 'Hotelling aura'&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Similarly, the probability of a particular attack spell occurring depends on the probability of the buff occurring.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It would be justified to wonder if a proper probability distribution can even be &lt;em&gt;defined&lt;/em&gt; given this information.  Well, it turns out that if there is one, it is uniquely specified by the conditional probabilities given in the manual.  But how to sample from it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Luckily for you, the CD comes with an automated Gibbs sample, because you would have to spend an eternity doing the following by hand.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Gibbs sampler algorithm&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Choose an attack spell randomly&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Use the accept-reject algorithm to choose the buff conditional on the attack&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) Forget the attack spell you chose in step 1.&#xA;Choose a new attack spell using the accept-reject algorithm conditional on the buff in step 2&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4) Go to step 2, repeat forever (though usually 10000 iterations will be enough)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;5) Whatever your algorithm has at the last iteration, is your sample.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You see, in general, MCMC samplers are only &lt;em&gt;asymptotically&lt;/em&gt; guaranteed to generate samples from a distribution with the specified conditional probabilities.  But in many cases, MCMC samplers are the only practical solution available.&lt;/p&gt;&#xA;" OwnerUserId="3567" LastEditorUserId="4360" LastEditorDisplayName="" LastEditDate="2011-05-01T23:36:49.803" LastActivityDate="2011-05-01T23:36:49.803" CommentCount="8" />
  <row Id="10217" PostTypeId="2" ParentId="10204" CreationDate="2011-05-01T23:17:09.337" Score="4" ViewCount="" Body="&lt;p&gt;Based on the post and the comments to date:  If you want to create two groups based on a single variable, you are faced with an arbitrary choice.  You can say that below x is &quot;low&quot; and at or above x is &quot;high&quot; but there is not going to be any statistical procedure (certainly not a significance test) that can make that determination for you.  In this situation some people would draw a histogram and look for what seems like a &quot;natural&quot; dividing point, which might simply mean one that would be convincing or defensible to one's particular audience.  Alternatively, one might divide so as to create two equal-sized groups.  There is no right or wrong answer.  But I question the need for dichotomization, for I suspect that whatever methods you plan to apply with two groups could be replaced by other methods at least as informative that preserve the original variable.  For example, rather than dichotomizing and running a &lt;em&gt;T&lt;/em&gt;-test using a dependent variable, why not correlate your independent and dependent variables, or create a scatterplot to show their relationship.&lt;/p&gt;&#xA;" OwnerUserId="2669" LastActivityDate="2011-05-01T23:17:09.337" CommentCount="2" />
  <row Id="10218" PostTypeId="2" ParentId="10110" CreationDate="2011-05-01T23:19:42.463" Score="3" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://www.youtube.com/watch?v=C7JQ7Rpwn2k&quot; rel=&quot;nofollow&quot;&gt;This video&lt;/a&gt; (especially the part starting at 23:20) describes the same problem you have with double integration, which amplifies low frequency noise to unbearable levels quickly. They solve the problem by sensor fusion, effectively using other sensors (like magnetic field sensors and gyroscopes) simultaneously to infer a more robust estimate of the acceleration coming from gravity alone and the acceleration coming from the movement of the sensor.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To help you with the drift from the double integration you could also try a particle filter to estimate the true position of the accelerometer over time. &#xA;There is an interesting &lt;a href=&quot;http://www.youtube.com/watch?v=b6gPXKfJA5g&quot; rel=&quot;nofollow&quot;&gt;Tech Talk&lt;/a&gt; about a more robust version of this idea.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Perhaps you could also use characteristic points in your time series as a kind of position anchor, e.g. if you can infer with some confidence the times when the pivot is lowest (or highest) and just assume a fixed height over ground for these times. Then, instead of an initial value problem resulting in onesided double integration, you would have a boundary value problem, where you can additionally integrate backwards from the next anchor position. This reduces the time where errors can grow down to half of a period.&lt;/p&gt;&#xA;" OwnerUserId="4360" LastEditorUserId="4360" LastEditorDisplayName="" LastEditDate="2011-05-01T23:30:01.290" LastActivityDate="2011-05-01T23:30:01.290" CommentCount="2" />
  <row Id="10219" PostTypeId="2" ParentId="421" CreationDate="2011-05-02T03:02:45.817" Score="1" ViewCount="" Body="&lt;p&gt;As a first introduction to the topic i liked &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0198568320&quot; rel=&quot;nofollow&quot;&gt;Data Analysis: A Bayesian Tutorial&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a deep and philosophical discussion of the underlying ideas of quantitative scientific reasoning i recommend &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0521592712&quot; rel=&quot;nofollow&quot;&gt;Probability Theory: The Logic of Science&lt;/a&gt;. This book does not serve as a good introduction, though. It's only recommended for persons who want to know, why bayesian statistics is the way it is and/or are interested in a historic review of bayesian statistics.&lt;/p&gt;&#xA;" OwnerUserId="4360" LastEditorUserId="4360" LastEditorDisplayName="" LastEditDate="2011-05-02T04:24:27.573" LastActivityDate="2011-05-02T04:24:27.573" CommunityOwnedDate="2011-05-02T03:02:45.817" CommentCount="2" />
  <row Id="10220" PostTypeId="1" AcceptedAnswerId="10222" CreationDate="2011-05-02T03:26:20.900" Score="2" ViewCount="125" Body="&lt;p&gt;This may sound like a noob question but I'm unable to find any 'good' resources/examples on the same. The basic question is this: Most variables, depending on the problem will follow certain types of distributions. Normal/Gaussian &lt;em&gt;may not&lt;/em&gt; be the most appropriate one for capturing certain types of phenomena.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Although I'm quite familiar with various distributions from a mathematical viewpoint I'm unable to understand some of them conceptually e.g.: Uniform distribution is when the occurrence of that event is equally likely over time, Normal when the occurrence 'tends' to be centered around the mean more often (like number of defects in samples or heights of citizens in a country etc.,) Similarly for triangular - I understand these &lt;em&gt;easy ones&lt;/em&gt; so to speak.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What type of distributions have you commonly encountered when using monte-carlo simulations? Examples would be helpful along with the rationale for choosing that distribution. Basically looking for a reference/pointer that would help me lay it out as a list for reference and understanding. I'd prefer a non-mathematical explanation since it'll be used for discussing with non-mathematical stakeholders to whom the monte-carlo simulations would be shown&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&amp;lt;&quot;Distribution name&quot;&gt; : &amp;lt;&quot;Most appropriate use&quot;&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I've heard of the power law but don't really know/understand what it is and how it could be used.&lt;/p&gt;&#xA;" OwnerUserId="4426" LastActivityDate="2011-05-03T01:11:31.460" Title="Most suitable distributions for modeling Monte Carlo Simulations" Tags="&lt;distributions&gt;&lt;random-variable&gt;&lt;monte-carlo&gt;" AnswerCount="3" CommentCount="6" />
  <row Id="10221" PostTypeId="2" ParentId="10204" CreationDate="2011-05-02T04:47:47.903" Score="2" ViewCount="" Body="&lt;p&gt;Assuming you have a single predictor variable that represents frequency of behaviour, I would make the following points&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Should you split a numeric variable into high-low groups&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;I quote the following from one of my &lt;a href=&quot;http://jeromyanglim.blogspot.com/2009/09/cluster-analysis-and-single-dominant.html&quot; rel=&quot;nofollow&quot;&gt;blog posts on creating clusters&lt;/a&gt;, where I use the term &quot;median split&quot; as a prototypical example of converting a numeric variable into a binary high-low variable.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Many researchers have heard the advice&#xA;  to not form median splits (see,&#xA;  &lt;a href=&quot;http://www.uvm.edu/~dhowell/gradstat/psych341/lectures/Factorial2Folder/Median-split.html&quot; rel=&quot;nofollow&quot;&gt;Howell&lt;/a&gt;&#xA;  for a discussion), or other kinds of&#xA;  binary splits for that matter. The&#xA;  same arguments also tend to apply with&#xA;  other forms of abrupt grouping into a&#xA;  small number of factors. &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Some&#xA;  arguments &lt;strong&gt;FOR&lt;/strong&gt; running median splits&#xA;  are:  1) it allows you to do an ANOVA&#xA;  or t-test and compare group means;  2)&#xA;  group differences are easier to&#xA;  communicate to a lay audience;  3) it&#xA;  reflects the important distinction in&#xA;  the underlying continuous variable.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Some arguments &lt;strong&gt;AGAINST&lt;/strong&gt; running median&#xA;  splits are: 1) you can always find an&#xA;  equivalent analysis that respects the&#xA;  continuous nature of the variable&#xA;  (e.g., regression); 2) when creating&#xA;  median splits, you lose a lot of&#xA;  information; 3) the cut-off tends to&#xA;  be relatively arbitrary and it varies&#xA;  between samples; 4) the resulting&#xA;  model based on a median split does not&#xA;  reflect the underlying nature of the&#xA;  variable; 5) in most cases a binary&#xA;  split will have less statistical&#xA;  power; 6) if the purpose is to&#xA;  communicate to a scientific audience,&#xA;  respecting the continuous nature of&#xA;  the variable is a necessary&#xA;  complexity.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;From the above you can see that there&#xA;  are generally more reasons in favour&#xA;  of maintaining the continuous version&#xA;  of the variable. The two occasions&#xA;  where splits are tolerable are where&#xA;  it makes it easy to communicate&#xA;  findings to a lay audience and where&#xA;  the underlying effect of interest&#xA;  occurs in a stepwise fashion. In the&#xA;  case of the latter, the presence of a&#xA;  stepwise effect can be tested&#xA;  empirically; a quick look at a scatter&#xA;  plot should give some sense if there&#xA;  is a point where the effect changes&#xA;  dramatically. Likewise decisions based&#xA;  on test scores are often based on&#xA;  pass-fail kinds of categories, and&#xA;  there is often a concrete desire to&#xA;  draw inferences about these specific&#xA;  groups.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Also, check out &lt;a href=&quot;http://www.unt.edu/rss/class/mike/5030/articles/makefriends.pdf&quot; rel=&quot;nofollow&quot;&gt;page 128 of Making Friends with Your Data&lt;/a&gt; for further discussion.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In summary, my advice would be to run a correlation or a regression predicting your outcome variable from the continuous version of your predictor. You may or may not want to perform an order preserving transformation of your predictor depending on its distribution.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Creating two groups based on numeric variable&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Putting aside the issues raised above, if you decide that you still want to split your predictor variable into high-low groups, the following are some options&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Use Statistical properties of your sample&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Median split&lt;/li&gt;&#xA;&lt;li&gt;Above or below the mean&lt;/li&gt;&#xA;&lt;li&gt;Take bottom 25% and top 25% and throw out the middle&lt;/li&gt;&#xA;&lt;li&gt;Take bottom third and top third and throw out the middle third&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;Use accepted or externally validated cut-offs&#xA;&lt;ul&gt;&#xA;&lt;li&gt;e.g., medical diagnoses are often based on certain cut-offs on a continuous scale&lt;/li&gt;&#xA;&lt;li&gt;Use your own understanding of the phenomena to define a cut-off&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;Examine a histogram or density plot and look for a natural split in the data (as mentioned by @rolando2)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastActivityDate="2011-05-02T04:47:47.903" CommentCount="2" />
  <row Id="10222" PostTypeId="2" ParentId="10220" CreationDate="2011-05-02T05:28:45.687" Score="1" ViewCount="" Body="&lt;p&gt;The books &quot;Continuous univariate distributions&quot; Vol 1 + Vol 2 by Johnson, Kotz and Balakrishnan (and there is a multivariate book too, I believe) are classical references, rich on the mathematical properties as well as giving examples of the usages of the different distributions they treat. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want details on a specific class of distributions, Wikipedia is always a good place to start, see&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Power_law&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Power_law&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The requested list is probably not easy to compile - the &quot;most appropriate use&quot; may be highly dependent upon context, but again the Wikipedia list of distributions &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/List_of_probability_distributions&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/List_of_probability_distributions&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;could be a place to start to find distributions appropriate for your project.&lt;/p&gt;&#xA;" OwnerUserId="4376" LastActivityDate="2011-05-02T05:28:45.687" CommentCount="1" />
  <row Id="10223" PostTypeId="1" CreationDate="2011-05-02T06:06:46.337" Score="1" ViewCount="74" Body="&lt;p&gt;Does anyone know an approach to performing model selection in Weka through cross validation for regression problems? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as I can tell, the cross validation is implemented in Weka just to assess the performance of the classifier. I guess that calling Weka API from Java might solve the problem, but is there a GUI-based approach?&lt;/p&gt;&#xA;" OwnerUserId="976" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-02T10:34:32.220" LastActivityDate="2011-05-02T10:34:32.220" Title="Model selection in Weka through cross validation for regression problems" Tags="&lt;regression&gt;&lt;model-selection&gt;&lt;cross-validation&gt;&lt;weka&gt;" CommentCount="3" FavoriteCount="1" />
  <row Id="10224" PostTypeId="1" CreationDate="2011-05-02T06:22:30.293" Score="2" ViewCount="67" Body="&lt;p&gt;I'm running experiements that record the time my algorithm takes to solve a set of problem instances on a particular benchmark. Each problem has an associated difficulty in the range [1, n]. Ideally these should be evenly distributed across the difficulty spectrum but this is not the case: the problem sample I have is skewed toward the easier end of the spectrum. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To account for this, I have grouped the problems by difficulty e.g. [1-10], [11-20], ... , [n-9, n]. Each interval usually contains contains at least 10 problems (usually more; 50+ is not uncommon) and I take the average time required to solve all problems in each interval. This gives me a clearer picture of how my algorithm performs on both easy and hard problems with the caveat that the data is somewhat less reliable for the harder end of the spectrum. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;First question: is this okay or are there some gotchas I haven't accounted for?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Next: For comparative purposes, I need to summarise performance on each benchmark as a single number. I am loath to simply take the average across all problems as this figure is skewed by too many easy problems. Which brings me to...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second question: can I take an average of all interval averages instead?&lt;/p&gt;&#xA;" OwnerUserId="4431" LastEditorUserId="4431" LastEditorDisplayName="" LastEditDate="2011-05-02T07:04:07.107" LastActivityDate="2011-05-02T15:28:22.937" Title="Taking the mean of a data set with a skewed distribution" Tags="&lt;distributions&gt;&lt;mean&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="1" />
  <row Id="10225" PostTypeId="1" AcceptedAnswerId="10227" CreationDate="2011-05-02T06:53:09.437" Score="1" ViewCount="47" Body="&lt;p&gt;If I have a matrix &lt;code&gt;M&lt;/code&gt; of 15 columns, what is R syntax to extract a matrix &lt;code&gt;M1&lt;/code&gt; consisting of 1,7,9,11,13 and 15 columns?&lt;/p&gt;&#xA;" OwnerUserId="4432" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-02T07:06:05.500" LastActivityDate="2011-05-02T07:07:58.430" Title="Extracting multiple columns from a matrix in R" Tags="&lt;r&gt;&lt;matrix&gt;" AnswerCount="1" />
  <row Id="10226" PostTypeId="2" ParentId="6890" CreationDate="2011-05-02T06:53:51.023" Score="1" ViewCount="" Body="&lt;p&gt;Assuming you also have the raw data, you can use function heatmap(). It can take one or two dendrograms as input, if you want to avoid calculating the distances and clustering the objects again.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's first simulate some data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(1)&#xA;dat&amp;lt;-matrix(ncol=4, nrow=10, data=rnorm(40))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then cluster the rows and columns: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;rd&amp;lt;-dist(dat)&#xA;rc&amp;lt;-hclust(rd)&#xA;cd&amp;lt;-dist(t(dat))&#xA;cc&amp;lt;-hclust(cd)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;After this we have &lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) the raw data (dat) &#xA;2) a distance matrix (rd) and a dendrogram (rc) for rows of the raw data matrix &#xA;3) a distance matrix (cd) and and a dendrogram (cc) for columns of the raw data&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Distance matrices are not actually needed for the further steps, but the raw data on which the clustering was performed, and the resulting dendrogram(s) are.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With the raw data these dendrograms can be used as input to the function heatmap(). If both row and column dendrograms are needed, use: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;heatmap(dat, Rowv=as.dendrogram(rc), Colv=as.dendrogram(cc))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If only row or column dendrogram is needed, use NA as an input for either Rowv or Colv parameter in heatmap():&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Dendrogram for rows only&#xA;heatmap(dat, Rowv=as.dendrogram(rc), Colv=NA)&#xA;# Dendrogram for columns only&#xA;heatmap(dat, Rowv=NA, Colv=as.dendrogram(cc))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="4433" LastEditorUserId="4433" LastEditorDisplayName="" LastEditDate="2011-05-02T09:05:12.570" LastActivityDate="2011-05-02T09:05:12.570" CommentCount="1" />
  <row Id="10227" PostTypeId="2" ParentId="10225" CreationDate="2011-05-02T06:55:24.757" Score="2" ViewCount="" Body="&lt;p&gt;Like this: &lt;code&gt;M[,c(1,7,9,11,13,15)]&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="4257" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-02T07:07:58.430" LastActivityDate="2011-05-02T07:07:58.430" />
  <row Id="10228" PostTypeId="1" AcceptedAnswerId="10229" CreationDate="2011-05-02T07:50:49.067" Score="1" ViewCount="63" Body="&lt;p&gt;I have a matrix &lt;code&gt;M&lt;/code&gt; of float values, how to shuffle &lt;code&gt;M&lt;/code&gt; line-wise?&lt;/p&gt;&#xA;" OwnerUserId="4432" LastActivityDate="2011-05-02T07:55:36.740" Title="How to shuffle matrix data in R?" Tags="&lt;r&gt;&lt;matrix&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="10229" PostTypeId="2" ParentId="10228" CreationDate="2011-05-02T07:55:36.740" Score="3" ViewCount="" Body="&lt;p&gt;something like: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;nr&amp;lt;-dim(M)[1]&#xA;M[sample.int(nr),]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="4257" LastActivityDate="2011-05-02T07:55:36.740" CommentCount="1" />
  <row Id="10230" PostTypeId="1" CreationDate="2011-05-02T07:58:16.427" Score="2" ViewCount="78" Body="&lt;p&gt;I use kmeans for clustering a set of data. However, I have to specify the number of clusters. The problem is that sometimes I need 2 and other times I need 3 clusters.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Is there a clustering algorithm that could incorporate that feature in it?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="2721" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-02T08:22:37.333" LastActivityDate="2011-05-02T23:11:30.600" Title="Automating determination of number of clusters from a kmeans cluster analysis" Tags="&lt;clustering&gt;&lt;data-mining&gt;&lt;k-means&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="1" />
  <row Id="10231" PostTypeId="2" ParentId="10230" CreationDate="2011-05-02T08:00:12.460" Score="1" ViewCount="" Body="&lt;p&gt;Simplest solution: do both and then check which gives best results...&lt;/p&gt;&#xA;" OwnerUserId="4257" LastActivityDate="2011-05-02T08:00:12.460" CommentCount="3" />
  <row Id="10232" PostTypeId="2" ParentId="10220" CreationDate="2011-05-02T08:40:58.573" Score="2" ViewCount="" Body="&lt;p&gt;I personally worry about the philosophy of deciding on a distribution with certain properties first and then defining the parameters for the distribution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My advice is normally to get the best data that you can first and then to process that data to find the best distribution and parameters to fit it. If you decide a distribution first to suit the input data then you are deciding on the way the data should behave before you have actually checked that it behaves how you have assumed in reality. A simple example would be a data set where you assume that all of the variations are due to measurement errors, in which case a normal distribution would probably be your chosen distribution. What you can often find though is that the data is not normally distributed and there is something more complex underlying the data you have. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of the biggest problems in the modern use of Monte-Carlo analysis is the use of assumed distributions based on the best guesses of the users rather than actual data. If you don't actually have the data on which to estimate the input distributions then I would argue that there are better ways of looking at the problem than just going for Monte-Carlo analysis in the first instance. &lt;/p&gt;&#xA;" OwnerUserId="210" LastActivityDate="2011-05-02T08:40:58.573" CommentCount="6" />
  <row Id="10233" PostTypeId="2" ParentId="6763" CreationDate="2011-05-02T10:00:17.367" Score="1" ViewCount="" Body="&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;As already stated (by @mpiktas), in order to do PCA, you need to transpose your data so that chemical batches are rows and &quot;measurements&quot; are columns.&#xA;You can then run a PCA on the data and plot the 80 chemical batches on axes derived from the first two components.&#xA;&lt;a href=&quot;http://www.statmethods.net/advstats/factor.html&quot; rel=&quot;nofollow&quot;&gt;Here's an example on Quick-R&lt;/a&gt; of doing this in R.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Also a small supplementary suggestion, you might want to have a look at &lt;strong&gt;Chernoff faces&lt;/strong&gt;.&#xA;They present a face where each of your eight variables would represent a feature on the face. The size or shape of the feature indicates something about the variable. Flowing data has &lt;a href=&quot;http://flowingdata.com/2010/08/31/how-to-visualize-data-with-cartoonish-faces/&quot; rel=&quot;nofollow&quot;&gt;a  tutorial in R with images&lt;/a&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-02T13:52:06.720" LastActivityDate="2011-05-02T13:52:06.720" />
  <row Id="10234" PostTypeId="1" CreationDate="2011-05-02T10:16:09.873" Score="3" ViewCount="72" Body="&lt;p&gt;What is the meaning between the notation $P(z;d,w)$ and $P(z|d,w)$ which is commonly used in many books and papers?&lt;/p&gt;&#xA;" OwnerUserId="4290" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-02T10:30:50.847" LastActivityDate="2011-05-02T15:03:17.627" Title="Meaning of probability notations $P(z;d,w)$ and $P(z|d,w)$" Tags="&lt;probability&gt;&lt;notation&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="10235" PostTypeId="2" ParentId="10234" CreationDate="2011-05-02T10:45:59.180" Score="4" ViewCount="" Body="&lt;p&gt;I believe the origin of this is the likelihood paradigm (though I have not checked the actual historical correctness of the below, it is a reasonable way of understanding how iot came to be).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say in a regression setting, you would have a distribution:&#xA;p(Y | x, beta)&#xA;Which means: the distribution of Y if you know (conditional on) the x and beta values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to estimate the betas, you want to maximize the likelihood:&#xA;L(beta; y,x) = p(Y | x, beta)&#xA;Essentially, you are now looking at the expression p(Y | x, beta) as a function of the beta's, but apart from that, there is no difference (for mathematical correct expressions that you can properly derive, this is a necessity --- although in practice noone bothers).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, in bayesian settings, the difference between parameters and other variables soon fades, so one started to you use both notations intermixedly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, in essence: there is no actual difference: they both indicate the conditional distribution of the thing on the left, conditional on the thing(s) on the right.&lt;/p&gt;&#xA;" OwnerUserId="4257" LastActivityDate="2011-05-02T10:45:59.180" />
  <row Id="10236" PostTypeId="1" CreationDate="2011-05-02T11:19:01.220" Score="10" ViewCount="127" Body="&lt;p&gt;I have a dataset in which the event rate is very low ( 40,000 out of $12\cdot10^5$).&#xA;I am applying logistic regression on this. I have had a discussion with someone where it came out that logistic regression would not give good confusion matrix on such low event rate data. But because of the business problem and the way it has been defined, I can't increase the number of events from 40,000 to any larger number though I agree that I can delete some nonevent population.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please tell me your views on this, specifically:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Does accuracy of logistic regression depend on event rate or is there any minimum event rate which is recommended ?&lt;/li&gt;&#xA;&lt;li&gt;Is there any special technique for low event rate data ?&lt;/li&gt;&#xA;&lt;li&gt;Would deleting my nonevent population would be good for the accuracy of my model ?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I am new to statistical modeling so forgive my ignorance and please address any associated issues that I could think about.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks,&lt;/p&gt;&#xA;" OwnerUserId="1763" LastEditorUserId="3911" LastEditorDisplayName="" LastEditDate="2011-05-02T11:44:44.540" LastActivityDate="2011-05-02T14:00:51.480" Title="Applying logistic regression with low event rate" Tags="&lt;logistic-regression&gt;" AnswerCount="1" CommentCount="4" FavoriteCount="1" />
  <row Id="10237" PostTypeId="2" ParentId="10224" CreationDate="2011-05-02T11:26:14.357" Score="1" ViewCount="" Body="&lt;p&gt;You have a hierarchy of measurements, the first level of  multiple time measurements on problem number $i$ ($1\leq i \leq n$), the second level is multiple problems of the same difficulty group.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Level 1. The measured times follow a distribution. This distribution may be normal (if the run time is influence by a large number of more or less independent factors), exponential distribution (if the algorithm waits for a random event to occur), or something complicated (e.g. multimodal, where run time strongly depends on initial decisions). The average is useful in case of the normal and exponential distributions, but may not be useful in the complicated cases without a large number of runs on the same problem. To determine the distribution of run times it may be useful to (a) pick a couple of problems, and measure the run time with a large number of repetitions, (b) to think over the mechanism, the details of the algorithm. You may find that a few repetitions are generally enough, or that many repetitions are needed and perhaps median may be a better statistic than mean.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Level 2. The difficulties within a difficulty group are not the same, but you think that they are similar. The differences in run times between groups may be small or large. If the times within difficulty groups are close to each other compared to the differences between adjacent difficulty groups it may not be very important to find a perfect summary measure to characterise a difficulty group, mean will probably do. If, however, the differences between difficulty groups are small you probably want to have the best summary measure of the difficulty of groups. In this case again, the distribution of problem times within a difficulty group decides which method to use.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I generally advice against using “a single number”, because usually expressing the level of uncertainty is almost as important as finding the most likely values.&lt;/p&gt;&#xA;" OwnerUserId="3911" LastActivityDate="2011-05-02T11:26:14.357" />
  <row Id="10238" PostTypeId="1" CreationDate="2011-05-02T13:43:46.493" Score="3" ViewCount="86" Body="&lt;p&gt;This follows on from the previous question on &lt;a href=&quot;http://stats.stackexchange.com/questions/10030/difference-between-k-s-manual-test-and-k-s-test-with-r&quot;&gt;differences between K-S manual test and K-S test with R&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My frequency sample was &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;a=c(0,1,1,4,9).&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then the observed sample is &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; obs=c(2,3,4,4,4,4,5,5,5,5,5,5,5,5,5)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The expected sample is then &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;exp=c(1,1,1,2,2,2,3,3,3,4,4,4,5,5,5)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I hope you agree.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, I use &lt;code&gt;ks.test&lt;/code&gt;, like another time:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ks.test(obs,exp)&#xA;&#xA;data:  oss and att&#xA;&#xA;D = 0.4667, p-value = 0.07626&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then, I use the ks.test the other way:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The expected distribution can be the uniform. Do you agree?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And then:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ks.test(obs, &quot;punif&quot;, 0,5)&#xA;&#xA;data:  obs &#xA;&#xA;D = 0.6667, p-value = 3.239e-06&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h3&gt;Question&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Why do the two approaches give different results?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="4345" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-05-02T16:17:59.077" LastActivityDate="2011-05-02T16:17:59.077" Title="Even more with the Kolmogorov-Smirnov test with R software" Tags="&lt;r&gt;&lt;multinomial&gt;&lt;kolmogorov-smirnov&gt;" AnswerCount="1" CommentCount="4" FavoriteCount="1" />
  <row Id="10239" PostTypeId="2" ParentId="10236" CreationDate="2011-05-02T14:00:51.480" Score="3" ViewCount="" Body="&lt;p&gt;There is a better alternative to deleting nonevents for temporal or spatial data: you can aggregate your data across time/space, and model the counts as Poisson.  For example, if your event is &quot;volcanic eruption happens on day X&quot;, then not many days will have a volcanic eruption.  However, if you group together the days into weeks or months, e.g. &quot;number of volcanic eruptions on month X&quot;, then you will have reduced the number of events, and more of the events will have nonzero values.&lt;/p&gt;&#xA;" OwnerUserId="3567" LastActivityDate="2011-05-02T14:00:51.480" CommentCount="3" />
  <row Id="10240" PostTypeId="2" ParentId="10234" CreationDate="2011-05-02T15:03:17.627" Score="1" ViewCount="" Body="&lt;p&gt;Although it hasn't always been this way, these days $P(z; d, w)$ is generally used when $d,w$ are not random variables (which isn't to say that they're known, necessarily). $P(z | d, w)$ indicates conditioning on values of $d,w$. Conditioning is an operation on random variables and as such using this notation when $d, w$ aren't random variables is confusing (and tragically common). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As @Nick Sabbe points out $p(y|X, \Theta)$ is a common notation for the sampling distribution of observed data $y$. Some frequentists will use this notation but insist that $\Theta$ isn't a random variable, which is an abuse IMO. But they have no monopoly there; I've seen Bayesians do it too, tacking fixed hyperparameters on at the end of the conditionals. &lt;/p&gt;&#xA;" OwnerUserId="26" LastActivityDate="2011-05-02T15:03:17.627" />
  <row Id="10241" PostTypeId="1" AcceptedAnswerId="10244" CreationDate="2011-05-02T15:06:14.833" Score="2" ViewCount="180" Body="&lt;p&gt;I have two variables for some districts in the UK:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Number of crimes per habitant&lt;/li&gt;&#xA;&lt;li&gt;Median yearly income&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Here's what it looks like:&#xA;&lt;img src=&quot;http://i.imgur.com/dDKM9.png&quot; alt=&quot;Plot of number of crimes per habitant in function of the median yearly income&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to determine if there is a dependence between the two. I ran a Pearson's correlation test, which gave me the following result:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    Pearson's product-moment correlation&#xA;&#xA;data:  income and nbCrimesPerHab &#xA;t = 1.3689, df = 315, p-value = 0.172&#xA;alternative hypothesis: true correlation is not equal to 0 &#xA;95 percent confidence interval:&#xA; -0.03353993  0.18548945 &#xA;sample estimates:&#xA;      cor &#xA;0.0769025 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As I understand it, the two are almost uncorrelated, but I am not sure how to interpret the p-value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What could I do next? I am interested only in the crimes data. How could I determine their distribution? What more tools could I use?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As suggested in the comments, here's the same plot but this time with the mean&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/uFL5O.png&quot; alt=&quot;Plot of crime rates VS mean income&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It doesn't seem to change much, but the correlation is a bit higer (and with a better p-value):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    Pearson's product-moment correlation&#xA;&#xA;data:  income and nbCrimesPerHab &#xA;t = 2.0986, df = 319, p-value = 0.03664&#xA;alternative hypothesis: true correlation is not equal to 0 &#xA;95 percent confidence interval:&#xA; 0.007318403 0.223310038 &#xA;sample estimates:&#xA;      cor &#xA;0.1166938 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;A scatter plot shows the structure better, as suggested:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/m3Am2.png&quot; alt=&quot;Scatter plot of crime rates VS mean income&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="3699" LastEditorUserId="3699" LastEditorDisplayName="" LastEditDate="2011-05-03T08:15:19.843" LastActivityDate="2011-05-03T19:34:53.553" Title="These two variables are almost uncorrelated. What else can I say?" Tags="&lt;correlation&gt;&lt;multivariate-analysis&gt;&lt;independence&gt;" AnswerCount="3" CommentCount="4" />
  <row Id="10242" PostTypeId="1" CreationDate="2011-05-02T15:06:17.487" Score="1" ViewCount="34" Body="&lt;p&gt;For example, I have a set of numbers (say 0 to 10) that are presented to 100 subjects.&#xA;Each subject is asked whether the number is a small or a large number. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The results are that 100 people think zero is a small number, 70 people think one is a small number, etc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I use a certain distribution, say, exponential to estimate the parameter of the distribution based on these data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I assume people give subjective opinions and that most of them think 0 is a small number so it may be assumed that the degree of truth of zero being a small number is 1. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, by multiplying the exponential probability density function (PDF) by a constant, I can scale the (PDF) such that the peak of it hits one and making it a membership function. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does this sound right?&lt;/p&gt;&#xA;" OwnerUserId="4440" LastEditorUserId="264" LastEditorDisplayName="" LastEditDate="2011-05-03T09:27:19.547" LastActivityDate="2011-05-03T09:27:19.547" Title="Modeling membership function given some survey data or empirical distribution" Tags="&lt;density-function&gt;&lt;fuzzy-set&gt;" CommentCount="3" />
  <row Id="10243" PostTypeId="2" ParentId="10224" CreationDate="2011-05-02T15:22:38.353" Score="1" ViewCount="" Body="&lt;p&gt;I would take a look at the median score and see if it is a better representation of what you want.  If not you can do two other things.   1)  Use the mean and standard deviation.  Yes, I know that is 2 numbers, but it would give you a better representation of the distribution.  2) Use the mean and standard deviation to come up with a composite score. e.g. you might want to divide the mean by the standard deviation, or come up with some other linear combination of these two factors depending upon what you feel is more important.  If you are &quot;loath&quot; to take the average, for example, you might want to give more weight to the standard deviation, e.g. 60% of the total score.  But even so, two numbers are better, and I would plot the scores to see if they made sense.&lt;/p&gt;&#xA;" OwnerUserId="3489" LastEditorUserId="3489" LastEditorDisplayName="" LastEditDate="2011-05-02T15:28:22.937" LastActivityDate="2011-05-02T15:28:22.937" />
  <row Id="10244" PostTypeId="2" ParentId="10241" CreationDate="2011-05-02T15:35:25.667" Score="3" ViewCount="" Body="&lt;p&gt;I think you need to consider two issues when interpreting correlations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) the p-value tells you the probability of observing an effect of this magnitude due to chance alone.  In your case, the probability that a correlation of this magnitude occurring solely by chance is 17.2%.  This is what it is.  Statistical convention typically dictates that we only accept as significant those tests in which this probability is &amp;lt; 5% (i.e., p-value &amp;lt;= 0.05) but I think it is better to consider the definition of the p-value as opposed to viewing 0.05 as a rigid wall.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) the second thing to consider is the correlation coefficient, which is the measure of how much of the variation in one random variable corresponds to the variation in another random variable.  This value can only range from 0 to 1 with a maximum at 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Based on your comment about being interested in the crimes data mainly, I think you may be more interested in regression since you are likely thinking about income as a fixed variable and are mostly interested in how crimes are &lt;em&gt;dependent&lt;/em&gt; on income.  Your correlation coefficient is rather low (close to 0) so regardless of the p-value not much of the variation in crimes corresponds to the variation in income. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Without knowing more about the questions of your study it is hard to suggest exactly where to go next.  The distribution of the crimes data can be visualized with a number of tools including boxplots, frequency histograms and scatter plots.    &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit (based on comments)&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since you are interested in exploring other predictor variables you may want to consider stepwise multiple regression. This analysis allows you to build a multiple regression model from a suite of explanatory variables by only including those that improve the model by certain per-determined criteria.  Multiple regression will also allow you to assess the effect of one variable on crime after taking into account the effect of a previous variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given your list of proposed variables (in the comment to @Leo) be sure to test for correlation among your explanatory variables (e.g., income and age) because the explanatory variables of a multiple regression (or even a series of independent regressions) should not be substantially correlated . If there is a lot of correlated between the predictor variables then you cannot effectively partition the influence of each predictor on the dependent variable.&lt;/p&gt;&#xA;" OwnerUserId="4048" LastEditorUserId="4048" LastEditorDisplayName="" LastEditDate="2011-05-03T19:34:53.553" LastActivityDate="2011-05-03T19:34:53.553" CommentCount="8" />
  <row Id="10245" PostTypeId="2" ParentId="10241" CreationDate="2011-05-02T15:40:32.120" Score="2" ViewCount="" Body="&lt;p&gt;So you don't want to add any other variables in your model (average age, level of education, ethnicity, etc.)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It may be useful to visualize these two variables on a geographical map. You may see some patterns from there (including correlation, which may be slightly &quot;shifted in space&quot;).&lt;/p&gt;&#xA;" OwnerUserId="4337" LastActivityDate="2011-05-02T15:40:32.120" CommentCount="1" />
  <row Id="10246" PostTypeId="2" ParentId="10180" CreationDate="2011-05-02T16:01:55.797" Score="4" ViewCount="" Body="&lt;p&gt;What you are looking for is called &quot;Determination of minimum sample size&quot; for a particular test which is an application of statistical &lt;a href=&quot;http://en.wikipedia.org/wiki/Statistical_power&quot; rel=&quot;nofollow&quot;&gt;power analysis&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your special cases one anaylses a rxs &lt;a href=&quot;http://en.wikipedia.org/wiki/Contingency_table&quot; rel=&quot;nofollow&quot;&gt;contigency table&lt;/a&gt;. However, I can only provide some details to the 2x2 case, which can be extended to the multiple test case using e.g. the so called &lt;a href=&quot;http://de.wikipedia.org/wiki/Bonferroni-Methode&quot; rel=&quot;nofollow&quot;&gt;Bonferroni correction&lt;/a&gt; (details below). One test which can be performed here is a so called chi2-test, e.g. &lt;a href=&quot;http://en.wikipedia.org/wiki/Fishers_exact_test&quot; rel=&quot;nofollow&quot;&gt;Fisher's exact&lt;/a&gt; test.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;conversion_rate =$\frac{sales}{clicks}$&lt;/li&gt;&#xA;&lt;li&gt;$p_i$ conversion-rate of Combination i&lt;/li&gt;&#xA;&lt;li&gt;$n_i$ sample size for Combination i (aka the number of clicks)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Now what you want to calculate is: What is the minimum sample required sample size $n_i$ and $n_j$ so that my preferred statistical test with significance level $\alpha$ detects the difference $p_i-p_j$ with probability $1-\beta$, where ...&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;$\alpha$ denotes the probability that one rejects the Null-Hypothesis although it is true (i.e. calls a difference significant which is not)&lt;/li&gt;&#xA;&lt;li&gt;$\beta$ denotes the probability that one does not reject the Null-Hypothesis although it is false (i.e. fails to identify a significant difference).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;One formula for the case of Fisher's exact test is from &lt;a href=&quot;http://www.jstor.org/pss/2530613&quot; rel=&quot;nofollow&quot;&gt;Casagrande et al.&lt;/a&gt; which is (according to my reference):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$n_i:=n_j:=\frac{A(1+\sqrt{1+4\delta/A})^2}{4\delta^2}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$A=\left(u_{1-\alpha}\sqrt{2\frac{p_i+p_j}{2}(1-\frac{p_i+p_j}{2})}-u_{\beta}\sqrt{p_i(1-p_i)+p_j(1-p_j)}\right)^2$ and  $\delta=p_i-p_j$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$u_{\alpha}$ is the quantile of the standard normal distribution for the probability $\alpha$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you have seen above, $n_i$ is set equivalent to $n_j$ here. But since you are performing an ABC-Test, this should not be a problem because the sample sizes for all combinations are roughly the same.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The Bonferroni correction:&lt;/strong&gt;&#xA;Since you have three combinations, you have to perform at least 3 tests (1 against 2, 2 against 3, 1 against 3), so the $\alpha$ value you should use here is:&#xA;$\alpha_{corrected}$=&lt;em&gt;your-desired-alpha&lt;/em&gt;/3 (same for $\beta$).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now let's perform an exemplary calculation: &#xA;Let's say you want to detect at least a difference reflecting a 10% increase (assuming that a lesser difference, although significant, would not be of interest (e.g. because of cost effectiveness)). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So we got:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;$p_1=\frac{10}{500}$&lt;/li&gt;&#xA;&lt;li&gt;$p_2=\frac{10}{500}*1.1$ (which is roughly equivalent to the &quot;true&quot; $p_2=\frac{11}{498}$)&lt;/li&gt;&#xA;&lt;li&gt;$p_3=\frac{15}{503}&amp;gt;p_2*1.1$ &lt;/li&gt;&#xA;&lt;li&gt;let's say $\alpha=0.05$ =&gt; $\alpha_{corrected}=0.05/3\approx 0.0167$&lt;/li&gt;&#xA;&lt;li&gt;let's say $\beta=0.2$ =&gt; $\beta_{corrected}=0.2/3\approx 0.0667$&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Hence:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;$n_{p_1vsp_2}=136382.6$&lt;/li&gt;&#xA;&lt;li&gt;$n_{p_1vsp_3}=6425.552$&lt;/li&gt;&#xA;&lt;li&gt;$n_{p_2vsp_3}=10608.72$&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You see that the main influence on the outcome is the difference between the ps, which is squared in the formula (see $\delta$ above). One can show greater differences faster (i.e. with a smaller sample). So if you want to e.g. show in an AB-Test that Combination3 is better than Combination1 (assuming that the measured conversion-rates are the actual true ones), one can do this with a sample size of 4481.86 (calculated without any $\alpha$ or $\beta$-correction). A week, if you create $\frac{4481.86*2}{7}\approx 1281$ clicks per day. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Final Note&lt;/strong&gt;: This so called &quot;probability of being better&quot; is assumedly calculated using a bayesian approach (I started a discussion about that &lt;a href=&quot;http://stats.stackexchange.com/questions/9735/how-does-a-frequentist-calculate-the-chance-that-group-a-beats-group-b-regarding&quot;&gt;here&lt;/a&gt;). I would not make a decision based on that number if it is not close enough to 0 or 1 (i.e. 0.95). One can also calculate the sample size bayesian style, but I am not done with it yet (I am also struggling with interpretation of GWO - results).&lt;/p&gt;&#xA;" OwnerUserId="264" LastEditorUserId="264" LastEditorDisplayName="" LastEditDate="2011-05-03T09:40:26.993" LastActivityDate="2011-05-03T09:40:26.993" />
  <row Id="10247" PostTypeId="2" ParentId="10238" CreationDate="2011-05-02T16:17:40.357" Score="5" ViewCount="" Body="&lt;p&gt;The first is a &lt;strong&gt;two-sample test&lt;/strong&gt;; the second is a &lt;strong&gt;one-sample test against a continuous distribution&lt;/strong&gt;.  Neither is used correctly:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;The two-sample test views both sets of data as being &lt;em&gt;data&lt;/em&gt;, but your &quot;expected sample&quot; is not data, it's a theoretical reference.  It is not subject to any variation.  The two-sample test thinks that it &lt;em&gt;can&lt;/em&gt; vary.  That's why the p-value is so large.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The reference distribution used in the one-sample test is a continuous uniform distribution between 0 and 5.  However, these data look discrete: from the way they are given, it appears they can attain only the values 1, 2, ..., 5.  Because the one-sample test doesn't know this, its p-value is probably too small.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;At least this lets us infer that the correct p-value should lie somewhere between 0.076 and 3.2e-06.  Because that doesn't settle the question, let's analyze further.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To get a sense of whether the data (0, 1, 1, 4, 9) differ significantly from the discrete uniform frequencies (3, 3, 3, 3, 3), view the latter as describing a five-sided die.  What are the chances that in 0+1+...+9 = 15 tosses of this die that at least one value would appear 9 or more times?  The events (1 appears 9 or more times), (2 appears 9 or more times), ..., (5 appears 9 or more times) are mutually exclusive--no two of them can hold at once--so their probabilities add.  Because the die is uniform each of these five events has the same probability.  We can compute the chance that a 5 comes up 9 or more times by viewing it like tosses of a biased coin: a 5 has a 1/5 chance; a non-5 has a 4/5 chance.  The chance of 9 or more 5's therefore equals&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\binom{15}{9}(1/5)^9(4/5)^6 + \binom{15}{10}(1/5)^{10}(4/5)^5 + \cdots + \binom{15}{15}(1/5)^{15}(1/4)^0.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This value is approximately 0.000785.  Multiplying by 5 gives .00392 = 0.39%, still quite small.  Thus this set of frequencies is unlikely to have arisen through a single experiment in which each of the values has an equal chance of arising.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-05-02T16:17:40.357" CommentCount="1" />
  <row Id="10248" PostTypeId="2" ParentId="10241" CreationDate="2011-05-02T17:55:00.947" Score="1" ViewCount="" Body="&lt;p&gt;Pearson correlation assumes linear relationships. It is possible that you have a curvilinear relationship going on.That can often be the case when income is involved.  A second issue is that you have defined yearly income in terms of the median rather than the mean.  That may truly be hiding some relationships that you need to see. I suggest starting with a plot of the raw values prior to performing a correlation test to see if it reasonably meets the assumptions.&lt;/p&gt;&#xA;" OwnerUserId="3489" LastActivityDate="2011-05-02T17:55:00.947" CommentCount="2" />
  <row Id="10249" PostTypeId="1" CreationDate="2011-05-02T18:04:51.363" Score="0" ViewCount="84" Body="&lt;p&gt;I was reading OkTrends and &lt;a href=&quot;http://blog.okcupid.com/index.php/page/3/&quot; rel=&quot;nofollow&quot;&gt;came across this&lt;/a&gt;: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In fact, 32% of successful couples&#xA;  agreed on all of them—which is 3.7×&#xA;  the rate of simple coincidence.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So, my question is: what is simple coincidence? How is it calculated?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can't find a Wikipedia article about &lt;em&gt;simple&lt;/em&gt; coincidence, and nowhere can I find a numerical definition. So, I'm confused about how the author got &quot;3.7x&quot; the rate of &quot;simple&quot; coincidence.&lt;/p&gt;&#xA;" OwnerUserId="4441" LastActivityDate="2011-05-02T18:04:51.363" Title="What is simple coincidence?" Tags="&lt;probability&gt;" CommentCount="4" />
  <row Id="10250" PostTypeId="1" CreationDate="2011-05-02T22:05:57.513" Score="6" ViewCount="88" Body="&lt;p&gt;I have data on the percent of organic matter in lake sediments from 0 cm (i.e., the sediment - water interface) down to 9 cm for approximately 25 lakes.  In each lake 2 cores were taken from each location so I have 2 replicate measures of organic matter percentage at each sediment depth for each lake.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am interested in comparing how lakes differ in the relationship between percent organic matter and sediment depth (i.e., slope).  In some lakes the relationship between percent organic matter and sediment depth appears linear but in other cases the relationship is more complex (see examples below).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My initial thoughts were to fit linear relationships where appropriate either to the whole curve or to a subset of the curve if it was &quot;mainly&quot; linear and only compare those lakes where a significant linear relationship was found.  However I am unhappy with this approach in that it requires eliminating data for no other reason than they do not fit the linear model and it ignores potentially interesting information about the relationship between percent organic matter and sediment depth.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What would be a good way to summarize and compare the curves from different lakes?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example curves:  In all cases the y-axis is the percent organic matter in the sediment and the x-axis is the sediment depth where 0 = the sedi&#xA;ment-water interface.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A nice linear example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/TxzGA.png&quot; alt=&quot;A nice linear example&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2 non-linear examples:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/6M4N8.png&quot; alt=&quot;nonlinear 1&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/ESuxB.png&quot; alt=&quot;enonlinear 2&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An example with no obvious relationship:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/4Cczp.png&quot; alt=&quot;no relationship&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="4048" LastActivityDate="2011-05-03T02:21:12.163" Title="How to summarize and compare non-linear relationships?" Tags="&lt;regression&gt;&lt;nonlinear-regression&gt;" AnswerCount="1" />
  <row Id="10251" PostTypeId="1" CreationDate="2011-05-02T23:10:16.580" Score="9" ViewCount="249" Body="&lt;p&gt;Principal component analysis &lt;em&gt;can use&lt;/em&gt; matrix decomposition, but that is just a tool to get there.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How would you find the principal components without the use of matrix algebra?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is your objective function (goal), and what are your constraints?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="74" LastEditorUserId="74" LastEditorDisplayName="" LastEditDate="2011-05-03T21:50:44.300" LastActivityDate="2011-05-04T07:49:25.030" Title="How to find principal components without matrix algebra?" Tags="&lt;pca&gt;" AnswerCount="3" CommentCount="10" FavoriteCount="4" />
  <row Id="10252" PostTypeId="2" ParentId="10230" CreationDate="2011-05-02T23:11:30.600" Score="0" ViewCount="" Body="&lt;p&gt;You can have a look at &quot;&lt;a href=&quot;http://www.jstatsoft.org/v33/i04/paper&quot; rel=&quot;nofollow&quot;&gt;clues&lt;/a&gt;&quot; method.&lt;/p&gt;&#xA;" OwnerUserId="4278" LastActivityDate="2011-05-02T23:11:30.600" />
  <row Id="10253" PostTypeId="2" ParentId="10250" CreationDate="2011-05-02T23:20:45.080" Score="2" ViewCount="" Body="&lt;p&gt;Check out &lt;a href=&quot;http://en.wikipedia.org/wiki/Generalized_additive_model&quot; rel=&quot;nofollow&quot;&gt;Generalized Additive Models&lt;/a&gt;, which permit fitting non-linear functions without a priori specification of the non-linear form. I'm not sure how one would go about comparing the subsequent fits however. Another similar (in that I believe they both employ cubic splines) approach is achieved by &lt;a href=&quot;http://www.psych.mcgill.ca/misc/fda/&quot; rel=&quot;nofollow&quot;&gt;Functional Data Analysis&lt;/a&gt;, where I understand there are methods for characterizing differences between fitted functions.&lt;/p&gt;&#xA;" OwnerUserId="364" LastEditorUserId="364" LastEditorDisplayName="" LastEditDate="2011-05-03T02:21:12.163" LastActivityDate="2011-05-03T02:21:12.163" CommentCount="1" />
  <row Id="10254" PostTypeId="2" ParentId="10220" CreationDate="2011-05-03T00:58:11.377" Score="1" ViewCount="" Body="&lt;p&gt;If you know the domain of the random variable and maybe have knowledge of some other properties like the mean, variance, etc. but want to be ignorant in a fair way about all other aspects of the distribution, you can find a distribution by applying the &lt;a href=&quot;http://en.wikipedia.org/wiki/Principle_of_maximum_entropy&quot; rel=&quot;nofollow&quot;&gt;principle of maximum entropy&lt;/a&gt;. Or put simply&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Given a collection of facts,&#xA;     choose a model which is consistent with all the facts,&#xA;     but otherwise as uniform as possible.&#xA;         (&lt;a href=&quot;http://www.cs.cmu.edu/afs/cs/user/aberger/www/html/tutorial/node2.html#SECTION00011000000000000000&quot; rel=&quot;nofollow&quot;&gt;Adam Berger&lt;/a&gt;)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Even if you don't want to derive these distributions yourself i think it's still good to know that there is such a general principle that may be used.&#xA;For many common cases people have already done this and you can just look up the solution, depending on your domain and given statistics: &lt;a href=&quot;http://en.wikipedia.org/wiki/Maximum_entropy_probability_distribution&quot; rel=&quot;nofollow&quot;&gt;Wikipedia&lt;/a&gt; lists some of them.&lt;/p&gt;&#xA;" OwnerUserId="4360" LastEditorUserId="4360" LastEditorDisplayName="" LastEditDate="2011-05-03T01:11:31.460" LastActivityDate="2011-05-03T01:11:31.460" />
  <row Id="10255" PostTypeId="2" ParentId="10251" CreationDate="2011-05-03T01:50:30.833" Score="2" ViewCount="" Body="&lt;p&gt;See NIPALS (&lt;a href=&quot;http://en.wikipedia.org/wiki/Non-linear_iterative_partial_least_squares&quot; rel=&quot;nofollow&quot;&gt;wiki&lt;/a&gt;) for one algorithm which doesn't explicitly use a matrix decomposition. I suppose that's what you mean when you say that you want to avoid matrix algebra since you really can't avoid matrix algebra here :)&lt;/p&gt;&#xA;" OwnerUserId="26" LastActivityDate="2011-05-03T01:50:30.833" />
  <row Id="10256" PostTypeId="2" ParentId="10251" CreationDate="2011-05-03T02:27:02.357" Score="11" ViewCount="" Body="&lt;p&gt;Without trying to give a full primer on PCA, from an optimization standpoint, the primary &lt;strong&gt;objective function&lt;/strong&gt; is the &lt;a href=&quot;http://en.wikipedia.org/wiki/Rayleigh_quotient&quot; rel=&quot;nofollow&quot;&gt;Rayleigh quotient&lt;/a&gt;. The matrix that figures in the quotient is (some multiple of) the sample covariance matrix&#xA;$$\newcommand{\m}[1]{\mathbf{#1}}\newcommand{\x}{\m{x}}\newcommand{\S}{\m{S}}\newcommand{\u}{\m{u}}\newcommand{\reals}{\mathbb{R}}\newcommand{\Q}{\m{Q}}\newcommand{\L}{\boldsymbol{\Lambda}}&#xD;&#xA;&#xD;&#xA;\S = \frac{1}{n} \sum_{i=1}^n \x_i \x_i^T = \m{X}^T \m{X} / n&#xD;&#xA;$$&#xA;where each $\x_i$ is a vector of $p$ features and $\m{X}$ is the matrix such that the $i$th row is $\x_i^T$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PCA seeks to solve a &lt;em&gt;sequence&lt;/em&gt; of optimization problems. The first in the sequence is the unconstrained problem&#xA;$$&#xD;&#xA;\begin{array}{ll}&#xD;&#xA;\text{maximize} &amp;amp; \frac{\u^T \S \u}{\u^T\u} \;, \u \in \reals^p \&amp;gt; .&#xD;&#xA;\end{array}&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since $\u^T \u = \|\u\|_2^2 = \|\u\| \|\u\|$, the above unconstrained problem is equivalent to the constrained problem&#xA;$$&#xD;&#xA;\begin{array}{ll}&#xD;&#xA;\text{maximize} &amp;amp; \u^T \S \u \\&#xD;&#xA;\text{subject to} &amp;amp; \u^T \u = 1 \&amp;gt;.&#xD;&#xA;\end{array}&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is where the matrix algebra comes in. Since $\S$ is a symmetric positive semidefinite matrix (by construction!) it has an eigenvalue decomposition of the form&#xA;$$&#xD;&#xA;\S = \Q \L \Q^T \&amp;gt;,&#xD;&#xA;$$&#xA;where $\Q$ is an orthogonal matrix (so $\Q \Q^T = \m{I}$) and $\L$ is a diagonal matrix with nonnegative entries $\lambda_i$ such that $\lambda_1 \geq \lambda_2 \geq \cdots \geq \lambda_p \geq 0$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hence, $\u^T \S \u = \u^T \Q \L \Q^T \u = \m{w}^T \L \m{w} = \sum_{i=1}^p \lambda_i w_i^2$. Since $\u$ is constrained in the problem to have a norm of one, then so is $\m{w}$ since $\|\m{w}\|_2 = \|\Q^T \u\|_2 = \|\u\|_2 = 1$, by virtue of $\Q$ being orthogonal. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But, if we want to maximize the quantity $\sum_{i=1}^p \lambda_i w_i^2$ under the constraints that $\sum_{i=1}^p w_i^2 = 1$, then the best we can do is to set $\m{w} = \m{e}_1$, that is, $w_1 = 1$ and $w_i = 0$ for $i &amp;gt; 1$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, backing out the corresponding $\u$, which is what we sought in the first place, we get that&#xA;$$&#xD;&#xA;\u^\star = \Q \m{e}_1 = \m{q}_1 &#xD;&#xA;$$&#xA;where $\m{q}_1$ denotes the first column of $\Q$, i.e., the eigenvector corresponding to the largest eigenvalue of $\S$. The value of the objective function is then also easily seen to be $\lambda_1$.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;The remaining principal component vectors are then found by solving the sequence (indexed by $i$) of optimization problems&#xA;$$&#xD;&#xA;\begin{array}{ll}&#xD;&#xA;\text{maximize} &amp;amp; \u_i^T \S \u_i \\&#xD;&#xA;\text{subject to} &amp;amp; \u_i^T \u_i = 1 \\&#xD;&#xA;                &amp;amp; \u_i^T \u_j = 0 \quad \forall 1 \leq j &amp;lt; i\&amp;gt;.&#xD;&#xA;\end{array}&#xD;&#xA;$$&#xA;So, the problem is the same, except that we add the additional constraint that the solution must be orthogonal to &lt;em&gt;all&lt;/em&gt; of the previous solutions in the sequence. It is not difficult to extend the argument above inductively to show that the solution of the $i$th problem is, indeed, $\m{q}_i$, the $i$th eigenvector of $\S$.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;The PCA solution is also often expressed in terms of the &lt;a href=&quot;http://en.wikipedia.org/wiki/Singular_value_decomposition&quot; rel=&quot;nofollow&quot;&gt;singular value decomposition&lt;/a&gt; of $\m{X}$. To see why, let $\m{X} = \m{U} \m{D} \m{V}^T$. Then $n \S = \m{X}^T \m{X} = \m{V} \m{D}^2 \m{V}^T$ and so $\m{V} = \m{Q}$ (strictly speaking, up to sign flips) and $\L = \m{D}^2 / n$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The principal components are found by projecting $\m{X}$ onto the principal component vectors. From the SVD formulation just given, it's easy to see that&#xA;$$&#xD;&#xA;\m{X} \m{Q} = \m{X} \m{V} = \m{U} \m{D} \m{V}^T \m{V} = \m{U} \m{D} \&amp;gt; .&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The simplicity of representation of both the principal component vectors and the principal components themselves in terms of the SVD of the matrix of features is one reason the SVD features so prominently in some treatments of PCA.&lt;/p&gt;&#xA;" OwnerUserId="2970" LastEditorUserId="4376" LastEditorDisplayName="" LastEditDate="2011-05-04T07:49:25.030" LastActivityDate="2011-05-04T07:49:25.030" CommentCount="2" />
  <row Id="10257" PostTypeId="2" ParentId="421" CreationDate="2011-05-03T02:40:27.687" Score="0" ViewCount="" Body="&lt;p&gt;So many wonderful recommendations! It's not quite what you asked for, but &lt;a href=&quot;http://en.wikipedia.org/wiki/How_to_Lie_with_Statistics&quot; rel=&quot;nofollow&quot;&gt;How to Lie with Statistics&lt;/a&gt; is short and quite wonderful. It doesn't directly teach the things you want, but it does help point out violation of assumptions and other flaws.&lt;/p&gt;&#xA;" OwnerUserId="3874" LastActivityDate="2011-05-03T02:40:27.687" CommunityOwnedDate="2011-05-03T02:40:27.687" />
  <row Id="10258" PostTypeId="2" ParentId="10049" CreationDate="2011-05-03T03:14:21.113" Score="0" ViewCount="" Body="&lt;p&gt;Based on the output you shared, Maximum # of branches from a node is set at 2.  It's possible that raising that limit would give you more options for branches, especially if SAS can take continuous variables and break them up into categories.  It's data dredgy, but that's the game we're in, and as long as you crossvalidate you're on solid moral ground :-)&lt;/p&gt;&#xA;" OwnerUserId="2669" LastActivityDate="2011-05-03T03:14:21.113" CommentCount="2" />
  <row Id="10259" PostTypeId="2" ParentId="10049" CreationDate="2011-05-03T03:56:31.520" Score="0" ViewCount="" Body="&lt;p&gt;If you are using tree-based methods, you can play around with the splitting criterion. For example, at each step, choose the split that gives the highest weighted accuracy (the average of the two classes' accuracies).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This can be used as the basis for a random forest too, which should give you a good classifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I once used a similar process to boost precision while sacrificing recall. It worked very well (better than thresholding the scores from the classification algorithm which were very noisy anyway).&lt;/p&gt;&#xA;" OwnerUserId="2067" LastActivityDate="2011-05-03T03:56:31.520" />
  <row Id="10260" PostTypeId="2" ParentId="10251" CreationDate="2011-05-03T04:20:44.130" Score="6" ViewCount="" Body="&lt;p&gt;The solution presented by &lt;em&gt;cardinal&lt;/em&gt; focuses on the sample covariance matrix. Another starting point is the &lt;em&gt;reconstruction error&lt;/em&gt; of the data by a &lt;em&gt;q&lt;/em&gt;-dimensional hyperplane. If the &lt;em&gt;p&lt;/em&gt;-dimensional data points are $x_1, \ldots, x_n$ the objective is to solve&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\min_{\mu, \lambda_1,\ldots, \lambda_n, \mathbf{V}_q} \sum_{i=1}^n ||x_i - \mu - \mathbf{V}_q \lambda_i||^2$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;for a $p \times q$ matrix $\mathbf{V}_q$ with orthonormal columns and $\lambda_i \in \mathbb{R}^q$. This gives the best rank &lt;em&gt;q&lt;/em&gt;-reconstruction as measured by the euclidean norm, and the columns of the $\mathbf{V}_q$ solution are the first &lt;em&gt;q&lt;/em&gt; principal component vectors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For fixed $\mathbf{V}_q$ the solution for $\mu$ and $\lambda_i$ (this is regression) are &#xA;$$\mu = \overline{x} = \frac{1}{n}\sum_{i=1}^n x_i \qquad \lambda_i = \mathbf{V}_q^T(x_i - \overline{x})$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For ease of notation lets assume that $x_i$ have been centered in the following computations. We then have to minimize    &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\sum_{i=1}^n ||x_i  - \mathbf{V}_q\mathbf{V}_q^T x_i||^2$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;over $\mathbf{V}_q$ with orthonormal columns. Note that $P = \mathbf{V}_q\mathbf{V}_q^T$ is the &lt;em&gt;projection&lt;/em&gt; onto the &lt;em&gt;q&lt;/em&gt;-dimensional column space. Hence the problem is equivalent to minimizing&lt;br&gt;&#xA;$$\sum_{i=1}^n ||x_i  - P x_i||^2 = \sum_{i=1}^n ||x_i||^2 - \sum_{i=1}^n||Px_i||^2$$ &#xA;over rank &lt;em&gt;q&lt;/em&gt; projections $P$. That is, we need to &lt;em&gt;maximize&lt;/em&gt; &#xA;$$\sum_{i=1}^n||Px_i||^2 = \sum_{i=1}^n x_i^TPx_i = \text{tr}(P \sum_{i=1}^n x_i x_i^T) = n \text{tr}(P \mathbf{S})$$&#xA;over rank &lt;em&gt;q&lt;/em&gt; projections $P$, where $\mathbf{S}$ is the sample covariance matrix. Now $$\text{tr}(P\mathbf{S}) = \text{tr}(\mathbf{V}_q^T\mathbf{S}\mathbf{V}_q) = \sum_{i=1}^q u_i^T \mathbf{S} u_i$$&#xA;where $u_1, \ldots, u_q$ are the $q$ (orthonormal) columns in $\mathbf{V}_q$, and the arguments presented in @cardinal's answer show that the maximum is obtained by taking the $u_i$'s to be $q$ eigenvectors for $\mathbf{S}$ with the $q$ largest eigenvalues. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reconstruction error suggests a number of useful generalizations, for instance sparse principal components or reconstructions by low-dimensional manifolds instead of hyperplanes. For details, see Section 14.5 in &lt;a href=&quot;http://www-stat.stanford.edu/~tibs/ElemStatLearn/&quot; rel=&quot;nofollow&quot;&gt;The Elements of Statistical Learning&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="4376" LastEditorUserId="4376" LastEditorDisplayName="" LastEditDate="2011-05-04T06:25:29.590" LastActivityDate="2011-05-04T06:25:29.590" CommentCount="7" />
  <row Id="10261" PostTypeId="1" CreationDate="2011-05-03T06:23:35.117" Score="1" ViewCount="75" Body="&lt;p&gt;While im going through the derivation of E step in EM algorithm for pLSA, i came across the following derivation &lt;a href=&quot;http://www.hongliangjie.com/2010/01/04/notes-on-probabilistic-latent-semantic-analysis-plsa/&quot; rel=&quot;nofollow&quot;&gt;at this page&lt;/a&gt;. Could anyone explain me how the following step is derived. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\sum_z q(z) log \frac{P(X|z,\theta)P(z|\theta)}{q(z)}  = \sum_z q(z) log \frac{P(z|X,\theta)P(X,\theta)}{q(z)} $&lt;/p&gt;&#xA;" OwnerUserId="4290" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-03T12:57:33.020" LastActivityDate="2011-05-03T12:57:33.020" Title="Derivation of E step in EM algorithm" Tags="&lt;em-algorithm&gt;&lt;plsa&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="10262" PostTypeId="2" ParentId="6705" CreationDate="2011-05-03T06:40:58.367" Score="0" ViewCount="" Body="&lt;p&gt;Moran's I statistic is used to explore a specific type of spatial clustering: whether high values are located in proximity to other high values and whether low values are located in proximity to other low values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The trick then is 1st to get a sense of what you mean by proximity and 2nd formulating this mathematically. This idea of proximity will depend on the what type of observations (attributes) you are working with and what type of questions you have in mind. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, for human beings proximity could mean the distance needed to have a chat. So, if you wanted to know whether high income people like to chat with other high income people at your cocktail party, you could formulate proximity by using binary weights where 1 is defined by 2 people being within 3 feet of each other. To see whether house prices are spatially correlated you could define proximity as when 2 houses are neighbors or perhaps if two houses are on the same block or if 2 houses are within sight of one another etc etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically, you need a hypothesis of proximity that is based on some of your prior common sense ideas or expert knowledge of why 2 objects that are close to one another are more associated than 2 objects that are far from one another. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Moran's I can then be seen as a test of your hypothesis of how your notion of proximity structures high values next to one another on the landscape.  &lt;/p&gt;&#xA;" OwnerUserId="4329" LastEditorUserId="4329" LastEditorDisplayName="" LastEditDate="2011-05-03T08:46:51.997" LastActivityDate="2011-05-03T08:46:51.997" />
  <row Id="10263" PostTypeId="2" ParentId="10261" CreationDate="2011-05-03T06:47:38.257" Score="2" ViewCount="" Body="&lt;p&gt;Hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It looks like &lt;a href=&quot;http://en.wikipedia.org/wiki/Bayes%27_theorem&quot; rel=&quot;nofollow&quot;&gt;Bayes' formula&lt;/a&gt; : &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\Pr[A \mid B] = \frac{\Pr[B \mid A] \Pr[A]}{\Pr[B]}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here, it gives:&#xA;$\Pr[X \mid z, \theta] = \frac{\Pr[z \mid X, \theta] \Pr[X \mid \theta]}{\Pr[z \mid \theta]}$&lt;/p&gt;&#xA;" OwnerUserId="3019" LastActivityDate="2011-05-03T06:47:38.257" CommentCount="2" />
  <row Id="10264" PostTypeId="2" ParentId="3268" CreationDate="2011-05-03T06:52:24.267" Score="1" ViewCount="" Body="&lt;p&gt;An alternative to multidimentional scaling is making a map of the each group's position to one another as a SOM (Self Organising Maps). Just like you see with a geographic map of the United States with Kansas in the middle, the groups that are positioned near the middle of your SOM map would be the groups that are most connected to other groups.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a &lt;a href=&quot;http://code.google.com/p/pysom-thesis/&quot; rel=&quot;nofollow&quot;&gt;python SOM module&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="4329" LastEditorUserId="4329" LastEditorDisplayName="" LastEditDate="2011-05-03T08:26:00.337" LastActivityDate="2011-05-03T08:26:00.337" />
  <row Id="10265" PostTypeId="1" CreationDate="2011-05-03T09:12:35.067" Score="2" ViewCount="74" Body="&lt;p&gt;In the paper of &lt;a href=&quot;http://www.cs.brown.edu/~th/papers/Hofmann-UAI99.pdf&quot; rel=&quot;nofollow&quot;&gt;Probabilistic Latent Semantic Analysis&lt;/a&gt; by Hofmann, the author fits the model for &lt;em&gt;document&lt;/em&gt; $\times$ &lt;em&gt;word&lt;/em&gt; matrix through EM Algorithm in section 3. I was able to follow the derivation and meaning of the model derived in it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However in the later section, the author mentioned about Tempered EM for improving the generalization. Could anyone explain or point me to the location where I can understand the actual meaning of Tempered EM Algorithm. &lt;/p&gt;&#xA;" OwnerUserId="4290" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-03T11:06:56.247" LastActivityDate="2011-05-03T15:03:25.993" Title="What is a &quot;tempered EM algorithm&quot;?" Tags="&lt;em-algorithm&gt;&lt;plsa&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="10267" PostTypeId="1" CreationDate="2011-05-03T12:08:37.473" Score="4" ViewCount="149" Body="&lt;p&gt;I am building a Box-Jenkins model in Excel using solver. The model is AR(2).&#xA;The data that I have contains trend and seasonality both. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know how to remove seasonality using seasonal indexes and add it back to the forecast. &#xA;But, how do I handle trend? If I remove trend from the data, how should I add it back to the forecast?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, is the excel solver best way to find the AR parameters?&lt;/p&gt;&#xA;" OwnerUserId="4445" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-03T12:39:36.563" LastActivityDate="2011-05-04T15:17:58.293" Title="Predicting forecasts for next 12 months using Box-Jenkins" Tags="&lt;time-series&gt;&lt;forecasting&gt;" AnswerCount="3" />
  <row Id="10268" PostTypeId="1" CreationDate="2011-05-03T10:58:28.040" Score="2" ViewCount="75" Body="&lt;p&gt;I've got a question concerning the estimation of a tobit model with the &lt;a href=&quot;http://cran.r-project.org/web/packages/AER/index.html&quot; rel=&quot;nofollow&quot;&gt;AER&lt;/a&gt; package in R. I observed t-distributed residuals, so that the assumption of normal distributed std errors is violated. Fortunately it's possible to choose &quot;t&quot; as the distribution when fitting with &lt;code&gt;tobit()&lt;/code&gt;. It is &lt;code&gt;tobit(&amp;lt;formula&amp;gt;, dist=&quot;t&quot;)&lt;/code&gt;. After changing the distribution from t to normal, the AIC increases, so this seems to be a good idea. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My problem is: Is it possible to flip the normal distribution that way? Is it still a tobit model and do I have to care about something when changing the distribution in a tobit model?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I appreciate any thought! &lt;/p&gt;&#xA;" LastEditorUserId="71" LastEditorDisplayName="" LastEditDate="2011-05-03T18:42:17.707" LastActivityDate="2011-05-03T18:42:17.707" Title="Tobit model with t-distribution" Tags="&lt;r&gt;&lt;regression&gt;&lt;tobit-regression&gt;" CommentCount="0" />
  <row Id="10269" PostTypeId="2" ParentId="10267" CreationDate="2011-05-03T12:28:47.897" Score="4" ViewCount="" Body="&lt;p&gt;Your approach suggests initially adjusting in a deterministic manner the impact of seasonality. This approach may or may not be applicable as the impact of seasonality may be auto-projective in form. The best way to answer this question is to evaluate alternative final models for adequacy in terms of separating the observed observations to signal and noise. There are a number of possible pitfalls awaiting you . One of them &quot; does the series have one or more trends AND/OR one or more level shifts &quot; ? Another possible issue &quot;does the series have a constant set of monthly indicators or have some months had a statistically significant change in the their effects ? In terms of a seasonal ARIMA model this question translates to &quot; have the model parameters changed over time &quot; ?. My experience with Excel Solver has not been very positive.&lt;/p&gt;&#xA;" OwnerUserId="3382" LastEditorUserId="3382" LastEditorDisplayName="" LastEditDate="2011-05-03T12:53:10.690" LastActivityDate="2011-05-03T12:53:10.690" CommentCount="2" />
  <row Id="10270" PostTypeId="2" ParentId="10049" CreationDate="2011-05-03T13:25:29.633" Score="1" ViewCount="" Body="&lt;p&gt;The problem is more with the choice of the accuracy scoring rule.  Make sure that the ultimate goal is classification as opposed to prediction.  The proportion classified correctly is a discontinuous improper scoring rule.  An improper scoring rule is one that is optimized by a bogus model.  With an improper scoring rule such things as addition of a highly important predictor making the model less accurate can happen.  The use of log likelihood (or deviance) or the Brier quadratic scoring rule will help.  The concordance index C (which happens to equal the ROC area, making ROCs appear more useful than they really are) is a useful measure of predictive discrimination once the model is finalized.&lt;/p&gt;&#xA;" OwnerUserId="4253" LastActivityDate="2011-05-03T13:25:29.633" />
  <row Id="10271" PostTypeId="1" CreationDate="2011-05-03T13:35:45.367" Score="5" ViewCount="206" Body="&lt;p&gt;I am working with a time series of &lt;strong&gt;anomaly scores&lt;/strong&gt; (the background is anomaly detection in computer networks). Every minute, I get an anomaly score $x_t \in [0, 5]$ which tells me how &quot;unexpected&quot; or abnormal the current state of the network is. The higher the score, the more abnormal the current state. Scores close to 5 are theoretically possible but occur almost never.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I want to come up with an algorithm or a formula which automatically determines a &lt;strong&gt;threshold&lt;/strong&gt; for this anomaly time series. As soon as an anomaly score exceeds this threshold, an alarm is triggered.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The frequency distribution below is an example for an anomaly time series over 1 day. However, it is &lt;strong&gt;not&lt;/strong&gt; safe to assume that every anomaly time series is going to look like that. In this special example, an anomaly threshold such as the .99-quantile would make sense since the few scores on the very right can be regarded as anomalies.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/Ieo7b.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And the same frequency distribution as time series (it only ranges from 0 to 1 since there are no higher anomaly scores in the time series):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/0ZPoe.jpg&quot; alt=&quot;time series&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unfortunately, the frequency distribution might have shapes, where the .99-quantile is &lt;strong&gt;not useful&lt;/strong&gt;. An example is below. The right tail is very low, so if the .99-quantile is used as threshold, this might result in many false positives. This frequency distribution &lt;strong&gt;does not seem to contain anomalies&lt;/strong&gt; so the threshold should lie outside the distribution at around 0.25.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/gZw9a.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Summing up, the difference between these two examples is that the first one seems to exhibit anomalies whereas the second one does not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From my naive point of view, the algorithm should consider these two cases:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;If the frequency distribution has a large right tail (i.e. a couple abnormal scores), then the .99-quantile can be a good threshold.&lt;/li&gt;&#xA;&lt;li&gt;If the frequency distribution has a very short right tail (i.e. no abnormal scores), then the threshold should lie outside the distribution.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;/edit: There is also no ground truth, i.e. labeled data sets available. So the algorithm is &quot;blind&quot; against the nature of the anomaly scores.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I am not sure how these observations can be expressed in terms of an algorithm or a formula. Does anyone have a suggestion how this problem could be solved? I hope that my explanations are sufficient since my statistical background is very limited.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for your help!&lt;/p&gt;&#xA;" OwnerUserId="4446" LastEditorUserId="4446" LastEditorDisplayName="" LastEditDate="2011-05-05T08:59:25.660" LastActivityDate="2011-05-19T18:56:20.710" Title="Automatic threshold determination for anomaly detection" Tags="&lt;time-series&gt;&lt;outliers&gt;&lt;threshold&gt;" AnswerCount="5" CommentCount="7" />
  <row Id="10272" PostTypeId="5" CreationDate="2011-05-03T13:39:20.200" Score="0" ViewCount="" Body="&lt;p&gt;&lt;em&gt;Linear&lt;/em&gt; regression models a variable (the &quot;dependent variable&quot;) as varying randomly with respect to a linear combination of other variables (the &quot;independent variables&quot;).  &lt;em&gt;Multiple&lt;/em&gt; regression includes two or more non-constant independent variables (whence, three or more variables &lt;em&gt;in toto&lt;/em&gt;).  This adds complications not present with only one independent variable, including &lt;a href=&quot;http://en.wikipedia.org/wiki/Multiple_correlation&quot; rel=&quot;nofollow&quot;&gt;complex forms of correlation&lt;/a&gt; and &lt;a href=&quot;http://en.wikipedia.org/wiki/Interaction_%28statistics%29&quot; rel=&quot;nofollow&quot;&gt;interaction effects&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Use this &lt;code&gt;multiple-regression&lt;/code&gt; tag instead of the more generic &lt;a href=&quot;http://stats.stackexchange.com/questions/tagged/regression&quot;&gt;regression&lt;/a&gt; tag when your question focuses on an issue specifically related to including two or more independent variables in a regression model.&lt;/p&gt;&#xA;" OwnerUserId="-1" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-05-03T13:39:20.200" LastActivityDate="2011-05-03T13:39:20.200" />
  <row Id="10273" PostTypeId="4" CreationDate="2011-05-03T13:39:20.200" Score="0" ViewCount="" Body="This tag is a signal that the question focuses on a problem particular to multivariate analysis, such as multiple correlations or interactions." OwnerUserId="-1" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-05-03T13:39:20.200" LastActivityDate="2011-05-03T13:39:20.200" />
  <row Id="10274" PostTypeId="1" CreationDate="2011-05-03T07:29:01.627" Score="0" ViewCount="38" Body="&lt;p&gt;Hi&#xA;In R, I'm trying lmrob, cor.mve and cor.mcd in R for robust correlation of two number series (of the same length) A and B. I get different results if I put A~B or A,B than if I put B~A, or B,A.&#xA;Might someone guide me as to why?&#xA;thanks&lt;/p&gt;&#xA;" LastActivityDate="2011-05-03T13:42:43.880" Title="Robust correlation in R, directionality" Tags="&lt;r&gt;&lt;correlation&gt;&lt;robust&gt;" CommentCount="2" />
  <row Id="10275" PostTypeId="2" ParentId="8373" CreationDate="2011-05-03T13:54:05.717" Score="1" ViewCount="" Body="&lt;p&gt;Here are a couple of articles that deal with this subject:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www2.warwick.ac.uk/fac/soc/wbs/subjects/finance/faculty1/anthony_neuberger/improved.pdf&quot; rel=&quot;nofollow&quot;&gt;Britten-Jones and Neuberger, Improved inference and estimation in regression with overlapping observations&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.qass.org.uk/2009/Vol_3/paper4.pdf&quot; rel=&quot;nofollow&quot;&gt;Harri &amp;amp; Brorsen, The Overlapping Data Problem&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2544" LastActivityDate="2011-05-03T13:54:05.717" />
  <row Id="10276" PostTypeId="1" AcceptedAnswerId="10281" CreationDate="2011-05-03T14:06:30.987" Score="3" ViewCount="90" Body="&lt;p&gt;I have a response variable measured at three time points per individual (week 0, 18, and 36).  I am interested in differences in the change of the response over the 36 weeks within some categorical variable X.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I see two ways of modeling this.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;One way ANOVA with response = &lt;code&gt;week_36_score - week_0_score&lt;/code&gt;  (this seems like the simplest option)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Repeated Measures ANCOVA with response = &lt;code&gt;week_18_score&lt;/code&gt; and &lt;code&gt;week_36_score&lt;/code&gt;.  Covariate = &lt;code&gt;Week_0_score&lt;/code&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;h3&gt;Question&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Which would you prefer (if any, maybe there is a better choice)?  &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I know there has been a lot of articles on this and they seem to say each has its own strengths and weaknesses.  I believe the second model would have more power.  I am not worried about a ceiling effect here.&lt;/p&gt;&#xA;" OwnerUserId="2310" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-03T15:38:06.650" LastActivityDate="2011-05-04T01:56:24.750" Title="Best model for change in scores over three time points" Tags="&lt;anova&gt;&lt;modeling&gt;&lt;repeated-measures&gt;&lt;longitudinal&gt;" AnswerCount="2" />
  <row Id="10277" PostTypeId="1" CreationDate="2011-05-03T14:46:00.920" Score="6" ViewCount="70" Body="&lt;p&gt;This is a rather general question (i.e. not necessarily specific to statistics), but I have noticed a trend in the machine learning and statistical literature where authors prefer to follow the following approach: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Approach 1&lt;/strong&gt;: Obtain a solution to a practical problem by formulating a cost function for which it is possible (e.g. from a computational standpoint) to find a globally optimal solution (e.g. by formulating a convex cost function).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;rather than:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Approach 2&lt;/strong&gt;: Obtain a solution to the same problem by formulating a cost function for which we may not be able to obtain a globally optimal solution (e.g. we can only get a locally optimal solution for it).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that rigorously speaking the two problems are different; the assumption is that we can find the globally optimal solution for the first one, but not for the second one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other considerations aside (i.e. speed, ease of implementation, etc.), I am looking for:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;An &lt;strong&gt;explanation&lt;/strong&gt; of this trend (e.g. mathematical or historical arguments)&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Benefits&lt;/strong&gt; (practical and/or theoretical) for following Approach 1 instead of 2 when solving a practical problem.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="2798" LastEditorUserId="2798" LastEditorDisplayName="" LastEditDate="2011-05-04T15:09:13.407" LastActivityDate="2011-05-04T16:12:42.777" Title="Advantages of approaching a problem by formulating a cost function that is globally optimizable" Tags="&lt;optimization&gt;&lt;best-practices&gt;" AnswerCount="1" FavoriteCount="2" />
  <row Id="10278" PostTypeId="2" ParentId="10265" CreationDate="2011-05-03T15:03:25.993" Score="2" ViewCount="" Body="&lt;p&gt;I found an answer via Google in a &lt;a href=&quot;http://www.ma.utexas.edu/users/zmccoy/report.pdf&quot; rel=&quot;nofollow&quot;&gt;UTexas Paper&lt;/a&gt;. As I suspected from the name, it combines a temperature that decreases ala Simulated Annealing, changing the E step of the algorithm slightly.&lt;/p&gt;&#xA;" OwnerUserId="1764" LastActivityDate="2011-05-03T15:03:25.993" />
  <row Id="10279" PostTypeId="1" CreationDate="2011-05-03T15:23:07.630" Score="0" ViewCount="41" Body="&lt;p&gt;I am analyzing a large of dataset (n&gt;100) of incident rates, with the aim of forming a normal distribution. Then I will know if a future incident rate (x%) is either close to a historical mean or not, and can score/rate it accordingly with an already created formula. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data is positively-skewed, as most data points cluster around or near zero percent. I HAVE to transform this data into a normal distribution, correct? What is the preferred method when dealing with percentages (these will always be between 0 and 100%)?  Are there alternative non-normalizing methods I can use to reach my desired output? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyway, let's say I've transformed the data and it follows a normal distribution. Now I can find the mean and std dev, then plot these in Excel using z-scores. Then I should be able to determine if incident rate x% is in the top 10%, top 20% of values, and score it with my formula accordingly. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any problems with this method?  &lt;/p&gt;&#xA;" OwnerUserId="4450" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-03T17:17:19.583" LastActivityDate="2011-05-03T17:17:19.583" Title="Analyzing historical incident rates and rating future performance" Tags="&lt;normal-distribution&gt;&lt;data-transformation&gt;&lt;skew&gt;" AnswerCount="1" />
  <row Id="10280" PostTypeId="2" ParentId="10279" CreationDate="2011-05-03T16:12:25.170" Score="4" ViewCount="" Body="&lt;p&gt;You don't need to transform to a normal distribution to see if a particular value is the top tenth or top fifth of observations.  All you need to do is sort your observations (and count them). &lt;/p&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-05-03T16:12:25.170" CommentCount="1" />
  <row Id="10281" PostTypeId="2" ParentId="10276" CreationDate="2011-05-03T16:12:50.613" Score="1" ViewCount="" Body="&lt;p&gt;In general, I would go with a repeated measures design. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is nothing technically wrong with the first option. However, you are essentially throwing away 1/3 of your data (and 1/2 of your non-baseline data!), which may result in a lost of power. Additionally, since you have a measurement in between baseline and 36 weeks, you cannot conclude anything about the shapes of the response profiles (i.e. test for a quadratic/cubic effects over time.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With that being said, for your repeated measures design, I would define a new variable called response_change. I am assuming that you are interested in testing if X has some effect on changes in the responses. At week 18, this variable would take value week_18_score - week_0_score. At week 36, this would take value week_36_score - week_0_score. Using response_change is more suited to this hypothesis.&lt;/p&gt;&#xA;" OwnerUserId="2144" LastActivityDate="2011-05-03T16:12:50.613" CommentCount="1" />
  <row Id="10282" PostTypeId="2" ParentId="10277" CreationDate="2011-05-03T17:45:30.897" Score="2" ViewCount="" Body="&lt;p&gt;My believe is that the goal should be to optimize the function you are interested in. If that happens to be the number of misclassifications - and not a binomial likelihood, say - then you should try minimizing the number of misclassifications. However, for the number of practical reasons mentioned (speed, implementation, instability etc), this may not be so easy and it may even be impossible. In that case, we choose to approximate the solution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know of basically two approximation strategies; either we come up with algorithms that attempt to directly approximate the solution of the original problem, or we reformulate the original problem as a more directly solvable problem (e.g. convex relaxations). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A &lt;strong&gt;mathematical&lt;/strong&gt; argument for preferring one approach over the other is whether we can understand a) the properties of the solution actually computed and b) how well the solution approximates the solution of the problem we are actually interested in. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know of many results in statistics where we can prove properties of a solution to a  optimization problem. To me it seems more difficult to analyze the solution of an&#xA;algorithm, where you don't have a mathematical formulation of what it computes (e.g. that it solves a given optimization problem). I certainly won't claim that you can't, but it seems to be a &lt;strong&gt;theoretical benefit&lt;/strong&gt;, if you can give a clear mathematical formulation of what you compute. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is unclear to me, if such mathematical arguments give any &lt;strong&gt;practical benefits&lt;/strong&gt; to Approach 1 over Approach 2. There are certainly somebody out there, &lt;a href=&quot;http://videolectures.net/eml07_lecun_wia/&quot; rel=&quot;nofollow&quot;&gt;who are not afraid of a non-convex loss function&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="4376" LastEditorUserId="4376" LastEditorDisplayName="" LastEditDate="2011-05-04T16:12:42.777" LastActivityDate="2011-05-04T16:12:42.777" CommentCount="1" />
  <row Id="10283" PostTypeId="2" ParentId="10271" CreationDate="2011-05-03T18:18:53.510" Score="1" ViewCount="" Body="&lt;p&gt;Do you have any 'labeled' examples of what constitutes an anomaly? i.e. values associated with a network failure, or something like that?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One idea you might consider applying is a ROC curve, which is useful for picking threshholds that meet a specific criteria, like maximizing true positives or minimizing false negatives.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, to use a ROC curve, you need to label your data in some way.&lt;/p&gt;&#xA;" OwnerUserId="2817" LastActivityDate="2011-05-03T18:18:53.510" CommentCount="3" />
  <row Id="10284" PostTypeId="2" ParentId="10267" CreationDate="2011-05-03T18:23:46.203" Score="2" ViewCount="" Body="&lt;p&gt;If you are at all familiar with &lt;a href=&quot;http://www.r-project.org/&quot; rel=&quot;nofollow&quot;&gt;R&lt;/a&gt; (if you're building time series models, you should be), check out the &lt;a href=&quot;http://cran.r-project.org/web/packages/forecast/index.html&quot; rel=&quot;nofollow&quot;&gt;forecast&lt;/a&gt; package. It's designed to choose parameters for Arima as well as exponential smoothing models, and uses a solid methodology to do so.  It will probably get you a lot farther than what you are building in excel, especially because it will also allow you to explore exponential smoothing models.  The two functions you are interested in are 'auto.arima' and 'ets.'&lt;/p&gt;&#xA;&#xA;&lt;p&gt;/Edit: auto.arima can also be used to fit ARMAX models, which (if properly specified) can solve many of the problems identified by IrishStat.&lt;/p&gt;&#xA;" OwnerUserId="2817" LastEditorUserId="2817" LastEditorDisplayName="" LastEditDate="2011-05-04T15:17:58.293" LastActivityDate="2011-05-04T15:17:58.293" CommentCount="4" />
  <row Id="10285" PostTypeId="1" CreationDate="2011-05-03T18:31:11.137" Score="1" ViewCount="151" Body="&lt;h3&gt;Context&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;I ran an experiment with &lt;code&gt;3 x 2&lt;/code&gt; design with three levels of within subjects factor (repeated measures) and two levels to the between subjects factors.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am interested in examining the changes from baseline and the interaction effect. &lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Question&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;How do I compute the required sample size for a &lt;code&gt;3 x 2&lt;/code&gt; design in order to achieve adequate statistical power?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="4453" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-04T04:03:35.047" LastActivityDate="2011-05-04T16:12:13.730" Title="Sample size required for mixed design ANOVA to achieve adequate statistical power" Tags="&lt;anova&gt;&lt;repeated-measures&gt;&lt;power-analysis&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="10289" PostTypeId="1" CreationDate="2011-05-03T20:26:45.730" Score="5" ViewCount="148" Body="&lt;p&gt;At work we were discussing this as my boss has never heard of normalization. In Linear Algebra, Normalization seems to refer to the dividing of a vector by its length. And in statistics, Standardization seems to refer to the subtraction of a mean then dividing by its SD. But they seem interchangeable with other possibilities as well. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When creating some kind of universal score, that makes up 2 different metrics, which have different means and different SD's, would you Normalize, Standardize, or something else? One person told me it's just a matter of taking each metric and dividing them by their SD, individually. Then summing the two. And that will result in a universal score that can be used to judge both metrics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, say you had the number of people who take the subway to work (in NYC) and the number of people who drove to work (in NYC). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Train ---&gt; x&#xA;Car ---&gt; y&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you wanted to create a universal score to quickly report traffic fluctuations, you can't just add mean(x) and mean(y) because there will be a LOT more people who ride the train. There's 8 million people living in NYC, plus tourists. That's millions of people taking the train everyday verse hundreds of thousands of people in cars. So they need to be transformed to a similar scale in order to be compared.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If mean(x) = 8,000,000&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and mean (y) = 800,000&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would you normalize x &amp;amp; y then sum? Would you standardize x &amp;amp; y then sum? Or would you divide each by their respective SD then sum?  In order to get to a number that when fluctuates, represents total traffic fluctuations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any article or chapters of books for reference would be much appreciated. THANKS!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also here's another example of what I'm trying to do. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Imagine you're a college dean, and you're discussing admission requirements. You may want students with at least a certain GPA and a certain test score. It'd be nice if they were both on the same scale because then you could just add the two together and say, &quot;anyone with at least a 7.0 can get admitted.&quot; That way, if a prospective student has a 4.0 GPA, they could get as low as a 3.0 test score and still get admitted.  Inversely, if someone had a 3.0 GPA, they could still get admitted with a 4.0 test score.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But it's not like that. The ACT is on a 36 point scale and most GPA's are on 4.0 (some are 4.3, yes annoying). Since I can't just add an ACT and GPA to get some kind of universal score, how can I transform them so they can be added, thus creating a universal admission score.  And then as a Dean, I could just automatically accept anyone with a score above a certain threshold.  Or even automatically accept everyone whose score is within the top 95%.... those sorts of things.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would that be normalization? standardization? or just dividing each by their SD then summing?&lt;/p&gt;&#xA;" OwnerUserId="4455" LastActivityDate="2011-05-04T03:51:50.550" Title="What's the difference between Normalization and Standardization?" Tags="&lt;normalization&gt;&lt;scores&gt;&lt;descriptive-statistics&gt;&lt;standarization&gt;" AnswerCount="3" CommentCount="1" />
  <row Id="10290" PostTypeId="2" ParentId="10121" CreationDate="2011-05-03T20:40:19.043" Score="2" ViewCount="" Body="&lt;p&gt;If you are trying to generate random correlation matrices, consider sampling from the Wishart distribution. This following question provides information the Wishart distribution as well as advice on how to sample:&#xA;&lt;a href=&quot;http://stats.stackexchange.com/questions/2746/how-to-efficiently-generate-positive-semi-definite-correlation-matrices&quot;&gt;How to efficiently generate Positive Semi Definite correlation matrices?&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2773" LastActivityDate="2011-05-03T20:40:19.043" />
  <row Id="10291" PostTypeId="2" ParentId="10289" CreationDate="2011-05-03T21:02:01.230" Score="4" ViewCount="" Body="&lt;p&gt;In the business world, &quot;normalization&quot; typically means that the range of values are &quot;normalized to be from 0.0 to 1.0&quot;.    &quot;Standardization&quot; typically means that the range of values are &quot;standardized&quot; to measure how many standard deviations the value is from its mean.   However, not everyone would agree with that.   It's best to explain &lt;strong&gt;your definitions&lt;/strong&gt; before you use them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In any case, your transformation needs to provide something useful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your train/car example, do you gain anything out of knowing how many standard deviations from their mean, each value lies?   If you plot those &quot;standardized&quot; measures against each other as an x-y plot, you might see a correlation (see the first graph on the right):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Correlation_and_dependence&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Correlation_and_dependence&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If so, does that mean anything to you?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as your second example goes, if you want to &quot;equate&quot; a GPA from one scale to another scale, what do these scales have in common?  In other words, how could you transform those minimums to be equivalent, and the maximums to be equivalent?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's an example of &quot;normalization&quot;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Normalization_%28image_processing%29&quot; rel=&quot;nofollow&quot;&gt;Normalization Link&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you get your GPA and ACT scores in an interchangeable form, does it make sense to weigh the ACT and GPA scores differently?  If so, what weighting means something to you?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit 1 (05/03/2011) ==========================================&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, I would check out the links suggested by &lt;strong&gt;whuber&lt;/strong&gt; above.  The bottom line is, in both of your two-variable problems, you are going to have to come up with an &quot;equivalence&quot; of one variable versus the other.  And, a way to differentiate one variable from the other.   In other words, even if you can simplify this to a simple linear relationship, you'll need &quot;weights&quot; to differentiate one variable from the other.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's an example of a two variable problem:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.doc.ic.ac.uk/~frk/frank/da/6.%20multiple%20utility.pdf&quot; rel=&quot;nofollow&quot;&gt;Multi-Attribute Utilities&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the last page, if you can say that standardized train traffic &lt;code&gt;U1(x)&lt;/code&gt; versus standardized car traffic &lt;code&gt;U2(y)&lt;/code&gt; is &quot;additively independent&quot;, then you might be able to get away with a simple equation such as:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;U(x, y) = k1*U1(x) + (1 - k1)*U2(y)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Where k1=0.5 means you're indifferent to standardized car/train traffic.   A higher k1 would mean train traffic &lt;code&gt;U1(x)&lt;/code&gt; is more important.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, if these two variables are not &quot;additively independent&quot;, then you'll have to use a more complicated equation.  One possibility is shown on page 1:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;U(x, y) = k1*U1(x) + k2*U2(y) + (1-k1-k2)*U1(x)*U2(y)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In either case, you'll have to come up with a utility &lt;code&gt;U(x, y)&lt;/code&gt; that makes sense.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The same general weighting/comparison concepts hold for your GPA/ACT problem.  Even if they are &quot;normalized&quot; rather than &quot;standardized&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One last issue.   I know you're not going to like this, but the definition of the term &quot;additively independent&quot; is on page 4 of the following link.   I looked for a less geeky definition, but I couldn't find one.   You might look around to find something better.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.cs.toronto.edu/~fbacchus/Papers/BGUAI95.pdf&quot; rel=&quot;nofollow&quot;&gt;Additively Independent&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Quoting the link:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Intuitively, the agent prefers being both healthy and wealthy&#xA;more than might be suggested by considering the two attributes&#xA;separately. It thus displays a preference for probability&#xA;distributions in which health and wealth are positively&#xA;correlated.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As suggested at the top of this response, if you plot standardized train traffic versus standardized car traffic on an x-y plot, you might see a correlation.  If so, then you're stuck with the above non-linear utility equation or something similar.&lt;/p&gt;&#xA;" OwnerUserId="2775" LastEditorUserId="2775" LastEditorDisplayName="" LastEditDate="2011-05-04T03:51:50.550" LastActivityDate="2011-05-04T03:51:50.550" CommentCount="3" />
  <row Id="10292" PostTypeId="2" ParentId="10289" CreationDate="2011-05-03T21:08:58.627" Score="0" ViewCount="" Body="&lt;p&gt;The answer is simple, but you're not going to like it: it depends. If you value 1 standard deviation from both scores equally, then standardization is the way to go (note: in fact, you're &lt;a href=&quot;http://en.wikipedia.org/wiki/Studentized_range&quot; rel=&quot;nofollow&quot;&gt;studentizing&lt;/a&gt;, because you're dividing by an &lt;em&gt;estimate&lt;/em&gt; of the SD of the population).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If not, it is likely that standardization will be a good first step, after which you can give more weight to one of the score by multiplying by a wellchosen factor.&lt;/p&gt;&#xA;" OwnerUserId="4257" LastActivityDate="2011-05-03T21:08:58.627" CommentCount="1" />
  <row Id="10293" PostTypeId="1" CreationDate="2011-05-03T21:47:17.023" Score="1" ViewCount="59" Body="&lt;p&gt;I have some data that a downstream system needs an optimized function of boolean logic for.  Essentially I have data similar to:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;User cat1 cat2 cat3 cat4  &#xA;1    0    0    0    1  &#xA;2    1    0    0    1  &#xA;3    0    1    1    0  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I must optimize cat4 as a function like this: &lt;code&gt;(cat1 or cat2 or cat3)&lt;/code&gt;.  &quot;Or&quot; is the only operation that I can use.  Does anyone know a particular technique / strategy for optimizing a union of categories in this way?  (Note, I will have many categories.)&lt;/p&gt;&#xA;" OwnerUserId="739" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-05-04T04:22:33.797" LastActivityDate="2011-05-04T04:30:36.277" Title="Optimize a boolean function" Tags="&lt;machine-learning&gt;&lt;optimization&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="10294" PostTypeId="2" ParentId="10276" CreationDate="2011-05-03T22:08:01.730" Score="2" ViewCount="" Body="&lt;p&gt;There are difficulties in computing change.  This doesn't work on ordinal repsonses, and for continuous responses makes a strong assumption of proper choice of transformations for the variables.  I recommend adjusting for baseline and modeling the 2nd and 3rd measurements as longitudinal measurements.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Repeated measures ANOVA is becoming obsolete in favor of generalized least squares or mixed effects models.&lt;/p&gt;&#xA;" OwnerUserId="4253" LastEditorUserId="4253" LastEditorDisplayName="" LastEditDate="2011-05-04T01:56:24.750" LastActivityDate="2011-05-04T01:56:24.750" />
  <row Id="10295" PostTypeId="1" CreationDate="2011-05-03T21:53:26.910" Score="16" ViewCount="195" Body="&lt;p&gt;I am trying to put together a data-mining package for StackExchange sites and in particular, I am stuck in trying to determine the &quot;most interesting&quot; questions. I would like to use the question score, but remove the bias due to the number of views, but I don't know how to approach this rigorously.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the ideal world, I could sort the questions by calculating $\frac{v}{n}$, where $v$ is the votes total and $n$ is the number of views. After all it would measure the percentage of people that upvote the question, minus the percentage of people that downvote the question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unfortunately, the voting pattern is much more complicated. Votes tend to &quot;plateau&quot; to a certain level and this has the effect of drastically underestimating wildly popular questions. In practice, a question with 1 view and 1 upvote would certainly score and be sorted higher than any other question with 10,000 views, but less than 10,000 votes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am currently using $\frac{v}{\log{n}+1}$ as an empirical formula, but I would like to be precise. How can I approach this problem with mathematical rigorousness?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to address some of the comments, I'll try to restate the problem in a better way:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say I have a question with $v_0$ votes total and $n_0$ views. I would like to be able to estimate what votes total $v_1$ is most likely when the views reach $n_1$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this way I could simply choose a nominal value for $n_1$ and order all the question according to the expected $v_1$ total.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;I've created two queries on the SO datadump to show better the effect I am talking about:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://data.stackexchange.com/stackoverflow/q/99605/&quot; rel=&quot;nofollow&quot;&gt;Average Views by Score&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Result:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/AjwgO.png&quot; alt=&quot;Views by Score&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://data.stackexchange.com/stackoverflow/q/99606/&quot; rel=&quot;nofollow&quot;&gt;Average Score by Views (100-views buckets)&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Result:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/2dVzr.png&quot; alt=&quot;Score by Views&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://data.stackexchange.com/stackoverflow/q/99608/&quot; rel=&quot;nofollow&quot;&gt;The two formulas compared&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Results, not sure if straighter is better: ($\frac{v}{n}$ in blue, $\frac{v}{log{n}+1}$ in red)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/v8RVc.png&quot; alt=&quot;Formulas&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="4456" LastEditorUserId="4456" LastEditorDisplayName="" LastEditDate="2011-05-05T00:13:04.400" LastActivityDate="2011-05-05T01:16:53.857" Title="&quot;Interestingness&quot; function for StackExchange questions" Tags="&lt;data-mining&gt;&lt;predictive-models&gt;" AnswerCount="2" CommentCount="14" FavoriteCount="3" />
  <row Id="10297" PostTypeId="2" ParentId="498" CreationDate="2011-05-03T22:59:01.330" Score="1" ViewCount="" Body="&lt;p&gt;I have had the exact same problem . . . in fact I'm having right now!  It seems to matter whether or not I include the blue-colored labels from the output window.  Try copying only the text in black (the table fillin's) and see if that does the trick.  It worked for me just now, and then when I go back and try again to copy the whole of it, it allows me to copy the blue label text too!  It seems to have woken SAS up to realize my desire to fill the clipboard.  I hate SAS . . . but I refuse to let it defeat me!&lt;/p&gt;&#xA;" OwnerUserId="4459" LastActivityDate="2011-05-03T22:59:01.330" />
  <row Id="10298" PostTypeId="2" ParentId="10289" CreationDate="2011-05-04T00:05:54.767" Score="0" ViewCount="" Body="&lt;p&gt;Normalization rescales the values from to a range of [0,1].  This might useful in some cases where all parameters need to have the same positive scale, but outliers from data set are lost.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Xchanged = (X - Xmin)/(Xmax-Xmin)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Standardization rescales data to have a mean of 0 and standard deviation of 1 (unit variance).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Xchanged = x-mean/sd&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For most applications standardization is recommended. &lt;/p&gt;&#xA;" OwnerUserId="2202" LastActivityDate="2011-05-04T00:05:54.767" />
  <row Id="10299" PostTypeId="2" ParentId="498" CreationDate="2011-05-04T02:39:22.600" Score="0" ViewCount="" Body="&lt;p&gt;I have the problem sometimes.  It seems as long as I do not highlight the whole output window but instead highlight all of it except the last line and leave a little extra space at the end of the line it always works.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are facing the problem try copying just the middle section and see if it works, if so then this can probably fix it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hopefully that made sense.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I just saw Dason's post and this sounds a lot like my solution.&lt;/p&gt;&#xA;" OwnerUserId="2310" LastActivityDate="2011-05-04T02:39:22.600" />
  <row Id="10300" PostTypeId="2" ParentId="7200" CreationDate="2011-05-04T03:37:21.730" Score="1" ViewCount="" Body="&lt;p&gt;I suppose I'm too late the hero, but I wanted to comment on cardinal's post, and this comment became too big for its intended box.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For this answer, I'm assuming $x &amp;gt;0$; appropriate reflection formulae can be used for negative $x$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm more used to dealing with the error function $\mathrm{erf}(x)$ myself, but I'll try to recast what I know in terms of Mills's ratio $R(x)$ (as defined in cardinal's answer).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are in fact alternative ways for computing the (complementary) error function apart from using Chebyshev approximations. Since the use of a Chebyshev approximation requires the storage of not a few coefficients, these methods might have an edge if array structures are a bit costly in your computing environment (you could inline the coefficients, but the resulting code would probably look like a baroque mess).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For &quot;small&quot; $|x|$, Abramowitz and Stegun give a nicely behaved series (at least better behaved than the usual Maclaurin series):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$R(x)=\sqrt{\frac{\pi}{2}}\exp\left(\frac{x^2}{2}\right)-x\sum_{j=0}^\infty\frac{2^j j!}{(2j+1)!}x^{2j}$$&#xA;(adapted from &lt;a href=&quot;http://people.math.sfu.ca/~cbm/aands/page_297.htm&quot; rel=&quot;nofollow&quot;&gt;formula 7.1.6&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that the coefficients of $x^{2j}$ in the series $c_j=\frac{2^j j!}{(2j+1)!}$ can be computed by starting with $c_0=1$ and then using the recursion formula $c_{j+1}=\frac{c_j}{2j+3}$. This is convenient when implementing the series as a summation loop.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;cardinal gave the Laplacian continued fraction as a way to bound Mills's ratio for large $|x|$; what is not as well-known is that the continued fraction is also useful for numerical evaluation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://dx.doi.org/10.1364/AO.15.000668&quot; rel=&quot;nofollow&quot;&gt;Lentz&lt;/a&gt;, &lt;a href=&quot;http://dx.doi.org/10.1016/0021-9991%2886%2990046-X&quot; rel=&quot;nofollow&quot;&gt;Thompson and Barnett&lt;/a&gt; derived an algorithm for numerically evaluating a continued fraction as an infinite product, which is more efficient than the usual approach of computing a continued fraction &quot;backwards&quot;. Instead of displaying the general algorithm, I'll show how it specializes to the computation of Mills's ratio:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\displaystyle Y_0=x,\,C_0=Y_0,\,D_0=0$&lt;br&gt;&#xA;$\text{repeat for }j=1,2,\dots$&lt;br&gt;&#xA;$$D_j=\frac1{x+jD_{j-1}}$$&#xA;$$C_j=x+\frac{j}{C_{j-1}}$$&#xA;$$H_j=C_j D_j$$&#xA;$$Y_j=H_j Y_{j-1}$$&#xA;$\text{until }|H_j-1| &amp;lt; \text{tol}$&lt;br&gt;&#xA;$\displaystyle R(x)=\frac1{Y_j}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\text{tol}$ determines the accuracy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The CF is useful where the previously mentioned series starts to converge slowly; you will have to experiment with determining the appropriate &quot;break point&quot; to switch from the series to the CF in your computing environment. There is also the alternative of using an asymptotic series instead of the Laplacian CF, but my experience is that the Laplacian CF is good enough for most applications.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Finally, if you don't need to compute the (complementary) error function very accurately (i.e., to only a few significant digits), there are &lt;a href=&quot;http://sites.google.com/site/winitzki/sergei-winitzkis-files/erf-approx.pdf&quot; rel=&quot;nofollow&quot;&gt;compact&lt;/a&gt; &lt;a href=&quot;http://sites.google.com/site/winitzki/top-index/Winitzki_2003_Uniform_approximations_for_transcendental_functions_LNCS_2667.pdf&quot; rel=&quot;nofollow&quot;&gt;approximations&lt;/a&gt; due to Serge Winitzki. Here is one of them:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$R(x)\approx \frac{\sqrt{2\pi}+x(\pi-2)}{2+x\sqrt{2\pi}+x^2(\pi-2)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This approximation has a maximum relative error of $1.84\times 10^{-2}$ and becomes more accurate as $x$ increases.&lt;/p&gt;&#xA;" OwnerUserId="830" LastEditorUserId="830" LastEditorDisplayName="" LastEditDate="2011-05-04T13:33:45.540" LastActivityDate="2011-05-04T13:33:45.540" />
  <row Id="10301" PostTypeId="2" ParentId="10293" CreationDate="2011-05-04T04:30:36.277" Score="1" ViewCount="" Body="&lt;p&gt;Think you mean something like which categories you should take into account to union them with the OR operator to get a good probability to predict the last variable? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On your training set, try to develop a model based on minimum binary integer programming (mBIP; as proposed here: &lt;a href=&quot;http://www.sce.carleton.ca/faculty/chinneck/po/Chapter13.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.sce.carleton.ca/faculty/chinneck/po/Chapter13.pdf&lt;/a&gt;). This should get you a good starting point for your prediction. Minimizing should be right for your disjunctive combination optimization.&#xA;Later on, you can process your test set and for each of the items either re-calculate the mBIP optimal set or try and inject it into the algorithm, no idea though at the moment how efficient that can be.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope it helps though.&lt;/p&gt;&#xA;" OwnerUserId="1158" LastActivityDate="2011-05-04T04:30:36.277" CommentCount="1" />
  <row Id="10302" PostTypeId="1" CreationDate="2011-05-04T06:04:26.560" Score="5" ViewCount="78" Body="&lt;p&gt;I came across term &lt;strong&gt;perplexity&lt;/strong&gt; which refers to the log-averaged inverse probability on unseen data. Wikipedia &lt;a href=&quot;http://en.wikipedia.org/wiki/Perplexity&quot; rel=&quot;nofollow&quot;&gt;article&lt;/a&gt; on &lt;strong&gt;perplexity&lt;/strong&gt; doesnt give a intuitive meaning for the same. This perplexity measure was used in &lt;a href=&quot;http://www.cs.brown.edu/~th/papers/Hofmann-UAI99.pdf&quot; rel=&quot;nofollow&quot;&gt;pLSA&lt;/a&gt; paper.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone explain the need and intuitive meaning of &lt;strong&gt;perplexity&lt;/strong&gt; measure? &lt;/p&gt;&#xA;" OwnerUserId="4290" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-05T07:18:45.300" LastActivityDate="2011-05-05T07:18:45.300" Title="What is perplexity?" Tags="&lt;measurement&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="2" />
  <row Id="10303" PostTypeId="1" AcceptedAnswerId="10304" CreationDate="2011-05-04T06:45:56.653" Score="3" ViewCount="45" Body="&lt;p&gt;Using OLS, I've estimated the following equation:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$y_i = \alpha_0 + \alpha_1 X_i + \epsilon_i$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know that theoretically, the following should be true: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$y_i = a + (1-e^{-\lambda 60}) X_i$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any way, having an estimate of $\alpha_1$ I can translate it to an estimate of $\lambda$? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a follow up, if this is not possible without some difficulty, If I knew the distribution of $\alpha_1$ was a normal distribution with some mean and variance, is there a way to describe what the form of the distribution of $\lambda$ would be? I feel like it would be a log-linear distribution, but I'm not sure what the mean/variance would be.&lt;/p&gt;&#xA;" OwnerUserId="726" LastActivityDate="2011-05-04T07:20:52.663" Title="Using an OLS coefficient to estimate a non-linear coefficient" Tags="&lt;distributions&gt;&lt;estimation&gt;&lt;normal-distribution&gt;&lt;log-linear&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="10304" PostTypeId="2" ParentId="10303" CreationDate="2011-05-04T07:20:52.663" Score="5" ViewCount="" Body="&lt;p&gt;Judging from your equations there is no reason for the OLS estimate of $\alpha_1$ not to be consistent and asymptotically normal. So you can use plug-in estimate for $\lambda$:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\hat{\lambda}=-\frac{1}{60}\log(1-\hat{\alpha}_1)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using &lt;a href=&quot;http://en.wikipedia.org/wiki/Delta_method&quot; rel=&quot;nofollow&quot;&gt;delta method&lt;/a&gt; it would be possible to show that this estimate is also consistent and  asymptotically normal. The only caveat is that the estimate of $\alpha_1$ can lie outside the interval $[0,1]$, but this would indicate that your postulated model is incorrect.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-05-04T07:20:52.663" />
  <row Id="10305" PostTypeId="1" CreationDate="2011-05-04T07:38:37.180" Score="2" ViewCount="84" Body="&lt;p&gt;When does/can one use the likelihood ratio significance test instead of Fisher's exact test or its Pearson $\chi^2$ approximation for comparing two binomial datasets?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given two binomial datasets (distributions), I'm seeing the LR test being used to compare one distribution against the global (combined) distribution. Usually I apply Fisher's test for comparing one dataset against the other. I realize that LR testing is of the Neyman-Pearson school, which assumes a fully specified alternative model as well as null model. E.g., in the &lt;a href=&quot;http://en.wikipedia.org/wiki/Likelihood-ratio_test#Examples&quot; rel=&quot;nofollow&quot;&gt;LR test Wikipedia page example&lt;/a&gt;, it's being used to compare two binomial datasets (# heads/tails for two coins).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why not use the $\chi^2$ test to compare the two samples against each other? What are the conceptual differences in these two approaches? When do I use which? And when is it appropriate to compare one sample against not its complement but the global dataset?&lt;/p&gt;&#xA;" OwnerUserId="1720" LastEditorUserId="1720" LastEditorDisplayName="" LastEditDate="2011-05-05T01:01:57.673" LastActivityDate="2011-05-05T01:01:57.673" Title="Likelihood ratio test vs. $\chi^2$/Z-test for comparing binomial datasets" Tags="&lt;hypothesis-testing&gt;&lt;likelihood-ratio&gt;&lt;fisher&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="10306" PostTypeId="2" ParentId="10267" CreationDate="2011-05-04T08:02:11.270" Score="1" ViewCount="" Body="&lt;p&gt;The time series are usually &lt;a href=&quot;http://en.wikipedia.org/wiki/Decomposition_of_time_series&quot; rel=&quot;nofollow&quot;&gt;decomposed&lt;/a&gt; into 3 parts, trend, seasonality and irregular. (The link gives 4 parts, but cyclical and seasonality are usually lumped together). Strictly speaking ARIMA type of models are only used for irregular part and by their design these model do not incorporate any trend (I am assuming that trend is some function which varies in time). So if you simply want to estimate AR(2) model no software will estimate the trend for you, since if it did, it would not be fitting AR(2) model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To forecast the trend you will first need to create some sort of model and test it, and only after you are confident that your model truly estimates the trend, then you can use it to forecast the trend. Without such model any forecasting is impossible. Sadly the majority of time series textbooks do not stress this when talking about forecasting. &lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-05-04T08:02:11.270" CommentCount="3" />
  <row Id="10307" PostTypeId="2" ParentId="10285" CreationDate="2011-05-04T08:25:15.447" Score="2" ViewCount="" Body="&lt;ul&gt;&#xA;&lt;li&gt;You need to decide &lt;strong&gt;what is acceptable statistical power&lt;/strong&gt; for a given significance test. The rule of thumb of 80% power being reasonable is often bandied about.&#xA;However, I think it is more sensible to see sample size selection as an optimisation problem, where statistical power is but one consideration, and the cost of collecting additional data is considered (&lt;a href=&quot;http://jeromyanglim.blogspot.com/2010/05/statistical-power-analysis-in-gpower-3.html&quot; rel=&quot;nofollow&quot;&gt;see here for further discussion of my thoughts&lt;/a&gt;).&lt;/li&gt;&#xA;&lt;li&gt;Mixed ANOVA involves &lt;strong&gt;multiple potential significance tests&lt;/strong&gt;. At the very least there are two main effects and an interaction. Potentially, there are also various comparisons. Each significance test can have different statistical power, and thus, literally, it does not make sense to speak of &lt;strong&gt;&quot;the&quot;&lt;/strong&gt; statistical power of a mixed design as if it was a singular property. When determining the minimum sample size, you may want to think about which of the possible significance tests on the mixed ANOVA are important. If all of them are important, then you may want to consider the sample size required by the least powerful significance test.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.psycho.uni-duesseldorf.de/abteilungen/aap/gpower3/&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;G Power 3&lt;/strong&gt;&lt;/a&gt; permits power analysis for all three types of significance tests in a mixed ANOVA (between subjects; within subjects; and interaction). &#xA;&lt;ul&gt;&#xA;&lt;li&gt;Download this free software and go to the &lt;code&gt;tests - means - Repeated Measures ...&lt;/code&gt; menu. &lt;/li&gt;&#xA;&lt;li&gt;The software permits &lt;code&gt;a priori&lt;/code&gt; (i.e., calculate sample size for given effect size, alpha, power, and design) or &lt;code&gt;post hoc&lt;/code&gt; (i.e., calculate power for given sample size, effect size, alpha, and design) power analysis.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-04T16:12:13.730" LastActivityDate="2011-05-04T16:12:13.730" CommentCount="2" />
  <row Id="10308" PostTypeId="1" CreationDate="2011-05-04T08:30:37.473" Score="5" ViewCount="88" Body="&lt;p&gt;I would like to explore the different ways one can &lt;strong&gt;detrend&lt;/strong&gt; a time series &lt;strong&gt;without look ahead bias&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wanted to use the Hodrick Prescott filter, which seems like a quite good frequency filter, but it is based on an optimization method, and I understand that it may give strange and volatile results at the border.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Wavelet smoothing on a rolling window would be another option, but again border effects can be huge (the data is copied by symmetry which is horrible for the precision of the technic at the edge).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any idea or comments?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: The subject has already been discussed here, I know. But I would like to dig a bit more on a more precise question.&lt;/p&gt;&#xA;" OwnerUserId="1709" LastEditorUserId="1709" LastEditorDisplayName="" LastEditDate="2011-05-04T11:02:09.977" LastActivityDate="2011-05-04T15:42:02.307" Title="How to remove trend with no look ahead bias?" Tags="&lt;time-series&gt;&lt;econometrics&gt;&lt;trend&gt;" AnswerCount="2" CommentCount="6" FavoriteCount="2" />
  <row Id="10309" PostTypeId="1" CreationDate="2011-05-04T09:07:08.330" Score="3" ViewCount="103" Body="&lt;p&gt;I am carrying out a statistical analysis for my research. I am using a Friedman's test with post hoc analysis. At present I am using the function &lt;code&gt;friedman.test.with.post.hoc&lt;/code&gt; available for R software. This function reports &quot;maxT&quot; value instead of the chi-square value. Someone can explain me what is maxT and its relationship with Chi-square?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you.&lt;/p&gt;&#xA;" OwnerUserId="4461" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-04T09:46:04.787" LastActivityDate="2011-05-13T09:55:18.950" Title="Friedman's test and post-hoc analysis" Tags="&lt;hypothesis-testing&gt;&lt;anova&gt;&lt;nonparametric&gt;&lt;repeated-measures&gt;&lt;permutation&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="10310" PostTypeId="2" ParentId="10308" CreationDate="2011-05-04T09:12:27.980" Score="1" ViewCount="" Body="&lt;p&gt;De-trending requires a pre-specification of of how many values do you require before declaring that a new trend has started. Given this specification , say n values then one has to be concerned with distinguishing between Level Shifts ( i.e. intercept changes ) and time trend changes. If you assume that there are no Level Shifts then simply search for different points in time and select those points which have been find to be statistically significant. For example if you have the series 1,2,3,4,5,7,9,11 ...this would suggest two points in time where the trend &quot;changed&quot; , period 1 and period 5. Alternatively if you have a series like 0,0,0,0,0,1,2,3,4,5,,,,,, there is only 1 point in time where a significant trend is evidenced i.e. period 5 . Outliers and ARIMA structure in a time series can lead to distortions in the identification of trend-point changes and may need to be incorporated prior to trend-point detection. A recent paper on tree-ring data &lt;a href=&quot;http://www.autobox.com/pdfs/forestdisturbance.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.autobox.com/pdfs/forestdisturbance.pdf&lt;/a&gt; discusses this issue.&lt;/p&gt;&#xA;" OwnerUserId="3382" LastActivityDate="2011-05-04T09:12:27.980" CommentCount="4" />
  <row Id="10311" PostTypeId="1" AcceptedAnswerId="10312" CreationDate="2011-05-04T11:03:30.273" Score="5" ViewCount="68" Body="&lt;p&gt;Hello,&#xA;I analyze a set of multivariate measurements. It is known that several pairs of independent variables show high linear correlation. The graph below shows a scatterplot of one such pair (X and Y, upper pane), as well the residuals as a function of Y (lower left pane) and the histogram of these residuals (lower right pane)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/7L5pf.png&quot; alt=&quot;data example&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As one can see, there is a strange peak in the residual histogram. Many of the remaining linearly-dependent variable pairs from the same data set have a similar peak. I have double checked, and I'm sure there are no duplicate records in the data set. What might be the reason for such a behaviour? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS Please don't ask me to elaborate on the problem domain, I'm not allowed to.&lt;/p&gt;&#xA;" OwnerUserId="1496" LastActivityDate="2011-05-04T12:03:26.133" Title="Wired residuals in linear regression" Tags="&lt;regression&gt;&lt;dataset&gt;&lt;outliers&gt;&lt;residuals&gt;" AnswerCount="1" CommentCount="3" FavoriteCount="1" />
  <row Id="10312" PostTypeId="2" ParentId="10311" CreationDate="2011-05-04T12:03:26.133" Score="5" ViewCount="" Body="&lt;p&gt;What is the value of the residual that shows such a high count? It does not appear to be zero (slightly to the right of 0), so maybe 1? In any case, there may be something about that value that may provide you with some insight about the underlying mechanism. For example, if X and Y are measurements taken by observers, some of them may have a tendency to follow a certain pattern (i.e., Joe thinks: &quot;everybody knows that Y is always 1 point higher than X&quot; and &quot;observes&quot; accordingly), leading to such results. Without domain-level knowledge though, it is difficult to guess what is going on here.&lt;/p&gt;&#xA;" OwnerUserId="1934" LastActivityDate="2011-05-04T12:03:26.133" CommentCount="1" />
  <row Id="10313" PostTypeId="2" ParentId="10305" CreationDate="2011-05-04T12:42:45.710" Score="1" ViewCount="" Body="&lt;p&gt;Generally speaking, the likelihood ratio and the ordinary Pearson $\chi^2$ tests are more accurate than Fisher's &quot;exact&quot; test.  But for your situation you need an extremely heavy multiplicity adjustment thrown in, not matter which statistical test is used.  Decision trees such as the one you are building require amazingly large datasets for their structure to validate.  In a quick look at the CN2 link you provided I could not tell if the algorithms incorporated shrinkage (panelization; regularization).  If not, watch for over-interpretation.&lt;/p&gt;&#xA;" OwnerUserId="4253" LastActivityDate="2011-05-04T12:42:45.710" CommentCount="6" />
  <row Id="10314" PostTypeId="2" ParentId="9040" CreationDate="2011-05-04T13:50:39.353" Score="3" ViewCount="" Body="&lt;p&gt;Hi, checkout Gephi, this software has some very good layout algorithms to handle the spaghetti problem: &lt;a href=&quot;http://gephi.org/features/&quot; rel=&quot;nofollow&quot;&gt;http://gephi.org/features/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Especially, try the ForceAtlas layout: &lt;a href=&quot;http://forum.gephi.org/viewtopic.php?f=26&amp;amp;t=926&quot; rel=&quot;nofollow&quot;&gt;http://forum.gephi.org/viewtopic.php?f=26&amp;amp;t=926&lt;/a&gt;&#xA;The software let you control the parameters in real time, and you can move the nodes manually.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(disclamer: I'm part of this community)&lt;/p&gt;&#xA;" OwnerUserId="4443" LastActivityDate="2011-05-04T13:50:39.353" CommentCount="2" />
  <row Id="10315" PostTypeId="2" ParentId="9435" CreationDate="2011-05-04T13:57:44.040" Score="1" ViewCount="" Body="&lt;p&gt;Gephi, an open source network visualization software, can do that: &lt;a href=&quot;http://gephi.org&quot; rel=&quot;nofollow&quot;&gt;http://gephi.org&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;See &lt;a href=&quot;http://forum.gephi.org/viewtopic.php?f=29&amp;amp;t=1016&quot; rel=&quot;nofollow&quot;&gt;this recent discussion&lt;/a&gt; on what a user was able to do with a bipartite graph, which is what you have. It is also called a bipartite network, or a two-mode network in Social Network Analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thought you'll need some trick, because Gephi doesn't have a layout to handle that directly. Feel free to ask the user how did I get that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(disclamer: I'm part of this community)&lt;/p&gt;&#xA;" OwnerUserId="4443" LastActivityDate="2011-05-04T13:57:44.040" CommentCount="1" />
  <row Id="10316" PostTypeId="1" CreationDate="2011-05-04T14:49:20.663" Score="6" ViewCount="150" Body="&lt;p&gt;I'm working on a multiple logistic regression in R using &lt;code&gt;glm&lt;/code&gt;. The predictor variables are continuous and categorical. An extract of the summary of the model shows the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Coefficients:&#xA;               Estimate Std. Error z value Pr(&amp;gt;|z|)&#xA;(Intercept)   2.451e+00  2.439e+00   1.005   0.3150&#xA;Age           5.747e-02  3.466e-02   1.658   0.0973 .&#xA;BMI          -7.750e-02  7.090e-02  -1.093   0.2743&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Confidence intervals:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;                  2.5 %       97.5 %&#xA;(Intercept)  0.10969506 1.863217e+03&#xA;Age          0.99565783 1.142627e+00&#xA;BMI          0.80089276 1.064256e+00&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Odd ratios:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;                 Estimate Std. Error   z value Pr(&amp;gt;|z|)&#xA;(Intercept)  1.159642e+01  11.464683 2.7310435 1.370327&#xA;Age          1.059155e+00   1.035269 5.2491658 1.102195&#xA;B            9.254228e-01   1.073477 0.3351730 1.315670&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The first output shows that $Age$ is significant. However, the confidence interval for $Age$ includes the value 1 and the odds ratio for $Age$ is very close to 1. What does the significant p-value from the first output mean? Is $Age$ a predictor of the outcome or not?&lt;/p&gt;&#xA;" OwnerUserId="2824" LastActivityDate="2011-05-04T15:15:26.640" Title="Interpreting logistic regression output in R" Tags="&lt;r&gt;&lt;logistic-regression&gt;&lt;interpretation&gt;&lt;p-value&gt;" AnswerCount="1" CommentCount="3" FavoriteCount="1" />
  <row Id="10317" PostTypeId="2" ParentId="10316" CreationDate="2011-05-04T15:15:26.640" Score="4" ViewCount="" Body="&lt;p&gt;There are a host of questions here on the site that will help with the interpretation of the models output (here are three different examples, &lt;a href=&quot;http://stats.stackexchange.com/q/3628/1036&quot;&gt;1&lt;/a&gt; &lt;a href=&quot;http://stats.stackexchange.com/q/8106/1036&quot;&gt;2&lt;/a&gt; &lt;a href=&quot;http://stats.stackexchange.com/q/6740/1036&quot;&gt;3&lt;/a&gt; , and I am sure there are more if you dig through the archive). Here is also a &lt;a href=&quot;http://www.ats.ucla.edu/stat/mult_pkg/faq/general/odds_ratio.htm&quot; rel=&quot;nofollow&quot;&gt;tutorial on the UCLA&lt;/a&gt;  stats website on how to interpret the coefficients for logistic regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Although the odds-ratio for the age coefficient is close to one it does not necessarily mean the effect is small (whether an effect is small or large is frequently as much a normative question as it is an empirical one). One would need to know the typical variation in age between observations to make a more informed opinion.&lt;/p&gt;&#xA;" OwnerUserId="1036" LastActivityDate="2011-05-04T15:15:26.640" CommentCount="2" />
  <row Id="10318" PostTypeId="2" ParentId="10308" CreationDate="2011-05-04T15:42:02.307" Score="2" ViewCount="" Body="&lt;p&gt;There is no way to get rid of the end effects.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Like any interpolation technique, the HP method depends on data before and after the current location to provide a filtered point/line for that location.   As you approach either end of the data series and drop below the required number of future (or past) points, you either don't provide the filtered line or the characteristics of the filtered line must change.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is dangerous to blindly extend the line and assume that it has the same properties at the ends of the series as it does in the middle.   The bottom line is, the HP filter has no predicting power.&lt;/p&gt;&#xA;" OwnerUserId="2775" LastActivityDate="2011-05-04T15:42:02.307" />
  <row Id="10319" PostTypeId="2" ParentId="9435" CreationDate="2011-05-04T15:51:07.560" Score="0" ViewCount="" Body="&lt;p&gt;Try &lt;a href=&quot;http://goo.gl/bGAXb&quot; rel=&quot;nofollow&quot;&gt;Pajek&lt;/a&gt;. It also supports plethora of export options, so you can easily modify figure in dedicated software (e.g. Inkscape).&lt;/p&gt;&#xA;" OwnerUserId="609" LastActivityDate="2011-05-04T15:51:07.560" />
  <row Id="10320" PostTypeId="1" CreationDate="2011-05-04T15:56:25.277" Score="4" ViewCount="50" Body="&lt;p&gt;&lt;strong&gt;Context&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a set of data that was collected from several inertial measurement units (orientation and acceleration data). I want to determine to what extent an inference method degrades when the data becomes noisy (or, should I say &quot;noisier&quot;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Questions&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I determine what type of noise to use? Is there a way in which I can find this out from the data itself?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What range of noise levels would be appropriate for testing the inference method's robustness? My guess is that it would depend on the answer to the question: &quot;how noisy would you expect the environment/system to get&quot;? Again, given only the collected data, what is the best approach in determining these levels?&lt;/p&gt;&#xA;" OwnerUserId="3052" LastActivityDate="2011-05-04T15:56:25.277" Title="Determing noise type and level of noise" Tags="&lt;regression&gt;&lt;white-noise&gt;" CommentCount="3" />
  <row Id="10321" PostTypeId="2" ParentId="10058" CreationDate="2011-05-04T16:50:48.563" Score="3" ViewCount="" Body="&lt;p&gt;Following Nick Sabbe's answer, here is the simplest GLMM solution I can come up with:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dej$location &amp;lt;- factor(rep(1:25,2))&#xA;library(lme4)&#xA;glmer(count ~ type1 + type2*species + &#xA;   perc.for.100m + perc.dry.100m + perc.wet.100m + &#xA;   (1|location), family = poisson, data = dej)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It would be a good idea to check for overdispersion too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For your picture, I would try&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(ggplot2)&#xA;ggplot(dej,aes(x=type2,y=count))+stat_sum(aes(size=..n..))+&#xA;   facet_grid(.~species)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;mainly for the advantage of &lt;code&gt;stat_sum&lt;/code&gt;, which will easily show where you have a lot of overplotting (more simply you could try jittering)&lt;/p&gt;&#xA;" OwnerUserId="2126" LastActivityDate="2011-05-04T16:50:48.563" />
  <row Id="10322" PostTypeId="1" AcceptedAnswerId="10327" CreationDate="2011-05-04T18:25:56.640" Score="3" ViewCount="166" Body="&lt;p&gt;Hi!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My girlfriend is an Actuarial Analist at Nationale-Nederlanden in the Netherlands and because we'll soon have our two year anniversary, I thought of gifts for her.&#xA;Her master's thesis title is &quot;Managing market risks for Dutch insurers&quot;, so she's pretty much into insurance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On &lt;a href=&quot;http://proofmathisbeautiful.tumblr.com/post/5104877044/baseln-statistical-distribution-plushies-am&quot; rel=&quot;nofollow&quot;&gt;Proof: Math is beautiful&lt;/a&gt; I discovered these &lt;a href=&quot;http://www.etsy.com/listing/71739287/collection-of-10-distribution-plushies&quot; rel=&quot;nofollow&quot;&gt;Distribution pluffies&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So here's my question: What distribution is of the most relevance in the field of an econometrician?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The available pluffies are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Standard Normal Distribution&lt;/li&gt;&#xA;&lt;li&gt;t Distribution&lt;/li&gt;&#xA;&lt;li&gt;Chi-Square Distribution&lt;/li&gt;&#xA;&lt;li&gt;Log Normal Distribution&lt;/li&gt;&#xA;&lt;li&gt;Continuous Uniform Distribution&lt;/li&gt;&#xA;&lt;li&gt;Weibull Distribution&lt;/li&gt;&#xA;&lt;li&gt;Cauchy Distribution&lt;/li&gt;&#xA;&lt;li&gt;Poisson Distribution&lt;/li&gt;&#xA;&lt;li&gt;Gumbel Distribution&lt;/li&gt;&#xA;&lt;li&gt;Erlang Distribution&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Any help much appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT:&lt;/strong&gt; Thanks a lot for all the suggestions despite this being just off-topic! I'll get her the t Distribution pluffy.&lt;/p&gt;&#xA;" OwnerUserId="4468" LastEditorUserId="4468" LastEditorDisplayName="" LastEditDate="2011-05-05T08:29:39.697" LastActivityDate="2011-05-05T08:29:39.697" CommunityOwnedDate="2011-05-04T20:12:48.133" Title="What distribution pluffy to buy for an aspiring econometrician?" Tags="&lt;distributions&gt;" AnswerCount="5" CommentCount="2" FavoriteCount="1" />
  <row Id="10323" PostTypeId="2" ParentId="10322" CreationDate="2011-05-04T18:39:13.447" Score="0" ViewCount="" Body="&lt;p&gt;From the list I would pick standard normal. After all regression is the main tool of econometrician and usually econometrician can rely only on asymptotic results, hence standard normal rules them all :) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having said that I would not like to get a standard normal distribution pluffy (I am not a girl, but can be considered econometrician) for standard normal is so widely used that it is a bit mundane. So you can choose log-normal, since econometricians use log-log regression extensively or Cauchy as a reminder, that not all distributions have finite moments.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-05-04T18:39:13.447" CommunityOwnedDate="2011-05-04T20:12:48.133" CommentCount="2" />
  <row Id="10324" PostTypeId="2" ParentId="10322" CreationDate="2011-05-04T19:10:31.520" Score="5" ViewCount="" Body="&lt;p&gt;You're in big trouble if you're asking us for gift advice.&lt;/p&gt;&#xA;" OwnerUserId="2775" LastActivityDate="2011-05-04T19:10:31.520" CommunityOwnedDate="2011-05-04T20:12:48.133" CommentCount="1" />
  <row Id="10325" PostTypeId="1" CreationDate="2011-05-04T19:24:03.320" Score="2" ViewCount="47" Body="&lt;p&gt;In the book “Programming Collective Intelligence” Segaran explains the Fisher method for categorizing text as an alternative to Naive Bayes classifier. The Fisher method uses inverse-chi-square-distribution, which I do not really understand.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I watched this video found on stats.stackexchange about chi-square-distribution to understand at least the “forward” function: &lt;a href=&quot;http://www.youtube.com/watch?v=dXB3cUGnaxQ&quot; rel=&quot;nofollow&quot;&gt;http://www.youtube.com/watch?v=dXB3cUGnaxQ&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Segaran explains in his book that they use inverse chi-square to somehow get a probability “that a random set of probabilities would return such a high number”. With high number he means that an item fitting a specific category has many features with high probabilities in that category. Somehow he also seems to take into account that “if the probabilities were independent and random, the result of this calculation would fit a chi-squared distribution”. But as he mentioned before the words are not independant (which is also a false assumption at naive bayes). So how does this then work?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And if I understand it right now, the inverse chi-square function somehow checks if many of my words have a high probability of being in the text and only if all words have such a high probability it returns a high over-all probability?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I’m sort of confused.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: The whole paragraph:&#xA;“Fisher showed that if the probabilities were independent and random, the result of this calculation would fit a chi-squared distribution. You would expect an item that doesn’t belong in a particular category to contain words of varying feature probabilities for that category (which would appear somewhat random), and an item that does belong in that category to have many features with high probabilities. By feeding the result of the Fisher calculation to the inverse chi-square function”, ou get the probability that a random set of probabilities would return such a high number.”&lt;/p&gt;&#xA;" OwnerUserId="4350" LastEditorUserId="4350" LastEditorDisplayName="" LastEditDate="2011-05-04T19:31:30.930" LastActivityDate="2011-05-04T19:31:30.930" Title="What does inverse-chi-square in Fisher method (classifying) exactly do?" Tags="&lt;chi-squared&gt;&lt;text-mining&gt;&lt;fisher&gt;" />
  <row Id="10326" PostTypeId="2" ParentId="10322" CreationDate="2011-05-04T19:31:35.703" Score="1" ViewCount="" Body="&lt;p&gt;Insurance is all about skewed distributions with long tails: think amount of loss. These also typically have only positive values. The log-normal distribution looks most like one of those. Another good option is the Gumbel distribution, which comes up in extreme value theory.&lt;/p&gt;&#xA;" OwnerUserId="279" LastActivityDate="2011-05-04T19:31:35.703" CommunityOwnedDate="2011-05-04T20:12:48.133" />
  <row Id="10327" PostTypeId="2" ParentId="10322" CreationDate="2011-05-04T20:10:51.810" Score="10" ViewCount="" Body="&lt;p&gt;You gotta get her one with some Kurtosis. Maybe the t-distribution. And be sure and write a loving note along the lines of, &quot;Baby, when I think of fat tails, I think of you. Your kurtosis makes you non-normal.&quot; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My wife digs it when I get sappy like that. I have the scars to prove it. &lt;/p&gt;&#xA;" OwnerUserId="29" LastActivityDate="2011-05-04T20:10:51.810" CommunityOwnedDate="2011-05-04T20:12:48.133" CommentCount="3" />
  <row Id="10328" PostTypeId="1" AcceptedAnswerId="10336" CreationDate="2011-05-04T15:13:26.520" Score="2" ViewCount="80" Body="&lt;p&gt;I know that I, and others, sometimes get confused by the hypergeometric distribution (HD) as it pertains to overlapping lists. This is because the HD is usually described with the &quot;balls in an urn&quot; metaphor and not using &quot;overlapping lists.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the proper way to calculate the p-value, according to the hypergeometric distribution, for the overlap of $B$ and $C$ in the lists below, ideally using the &lt;code&gt;phyper&lt;/code&gt; function in R, where&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;$A$ contains all of the genes in the genome&lt;/li&gt;&#xA;&lt;li&gt;$B$ is one subset of genes in the genome&lt;/li&gt;&#xA;&lt;li&gt;$C$ is another subset of genes in the genome?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="3561" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-05T07:29:10.630" LastActivityDate="2011-05-05T07:29:10.630" Title="Using R's phyper to get the probability of list overlap" Tags="&lt;r&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="10329" PostTypeId="1" CreationDate="2011-05-04T20:32:27.960" Score="5" ViewCount="94" Body="&lt;p&gt;What functionality should exist in a &lt;a href=&quot;http://en.wikipedia.org/wiki/Computer_algebra_system&quot; rel=&quot;nofollow&quot;&gt;CAS&lt;/a&gt; that was specifically geared toward Statistics?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Symbolic algebra systems like Mathematica and Maple are often used for calculus, logic, and physics problems but are rarely used for statistics. Why is this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What statistical constructs could be added to a symbolic algebra system to improve its use in this field? What are some specific code samples that many people would like to be able to do. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please think about the following three users: research statistician, non-statistics researcher using statistics in another field (such as biology), statistics student. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'll be working on &lt;a href=&quot;http://sympy.org/&quot; rel=&quot;nofollow&quot;&gt;SymPy's&lt;/a&gt; statistics code over the next few months and would like to solicit input for desired functionality. The things I use are not necessarily what the broader community uses. &lt;/p&gt;&#xA;" OwnerUserId="3830" LastEditorUserId="3830" LastEditorDisplayName="" LastEditDate="2011-05-06T04:12:27.167" LastActivityDate="2011-05-06T04:12:27.167" CommunityOwnedDate="2011-05-04T20:45:04.117" Title="Symbolic computer algebra for statistics" Tags="&lt;python&gt;&lt;computing&gt;&lt;computational-statistics&gt;" AnswerCount="1" CommentCount="5" />
  <row Id="10330" PostTypeId="2" ParentId="10322" CreationDate="2011-05-04T20:52:28.667" Score="2" ViewCount="" Body="&lt;p&gt;Aren't &lt;a href=&quot;http://en.wikipedia.org/wiki/What%27s_that_got_to_do_with_the...?&quot; rel=&quot;nofollow&quot;&gt;econometricians&lt;/a&gt; concerned with the &lt;a href=&quot;http://supertart.com/priceofteainchina/index.php&quot; rel=&quot;nofollow&quot;&gt;price of t (distributions) in China&lt;/a&gt;?  It has the large (on occasion, infinite) kurtosis recommended by @JD Long, too.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-05-04T20:52:28.667" CommunityOwnedDate="2011-05-04T20:52:28.667" />
  <row Id="10331" PostTypeId="1" AcceptedAnswerId="10332" CreationDate="2011-05-04T21:17:49.260" Score="4" ViewCount="67" Body="&lt;p&gt;I'm working on some practice test problems, and one of them says to design a rejection sampling algorithm to produce draws from a unit exponential using draws from a Gamma(2,1).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't understand how this is possible, because I am under the impression that the &quot;envelope function&quot; g(x) needs to be scalable in such a manner that for some constant $M$, $Mg(x)\geq f(x)\; \forall x$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can't see any way to do this, as the Gamma(2,1) is going to have little mass around 0, while the exponential function has most of its mass around 0.  What kind of transformation do I need to do to the Gamma function to allow it to function as an envelope?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using R, I tried flipping it to make it an inverse gamma, but that won't adequately capture the probability mass close to 0 and a K-S test confirmed that the points I generated did not arise from a unit exponential.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;edit:  I will include my code in which I tried to use an inverse-gamma(2,1) as an envelope:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x &amp;lt;- c()&#xA;for(i in 1:100000)&#xA;{&#xA;  g &amp;lt;- runif(1, 0, 1) &#xA;  h &amp;lt;- rigamma(1, 2, 1)&#xA;  M &amp;lt;- densigamma(h, 2, 1)&#xA;  crit &amp;lt;- dexp(h, 1)/M&#xA;  if(g &amp;lt; crit)&#xA;    x[i] = h&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2984" LastEditorUserId="8" LastEditorDisplayName="" LastEditDate="2011-05-05T21:09:49.177" LastActivityDate="2011-05-05T21:09:49.177" Title="How to use rejection sampling to generate draws from Unit Exponential" Tags="&lt;homework&gt;&lt;monte-carlo&gt;&lt;simulation&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="10332" PostTypeId="2" ParentId="10331" CreationDate="2011-05-04T23:17:36.520" Score="4" ViewCount="" Body="&lt;p&gt;Try a location shift on the Gamma(2,1)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: &#xA;&lt;a href=&quot;http://www.wolframalpha.com/input/?i=3%2a%28x%2b1%29%20%2a%20exp%28-%28x%2b1%29%29%20vs%20exp%28-x%29%20from%200%20to%206&quot; rel=&quot;nofollow&quot;&gt;Illustration&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="3567" LastEditorUserId="3567" LastEditorDisplayName="" LastEditDate="2011-05-05T17:07:45.637" LastActivityDate="2011-05-05T17:07:45.637" CommentCount="5" />
  <row Id="10333" PostTypeId="2" ParentId="10329" CreationDate="2011-05-04T23:29:40.427" Score="5" ViewCount="" Body="&lt;p&gt;Support for matrix algebra. The vast majority of practiced statistics is multivariate and involves matrices, and often simplifying matrix forms requires special rules that aren't easily translated from a univariate case, so good matrix support would be really helpful.&lt;/p&gt;&#xA;" OwnerUserId="2839" LastActivityDate="2011-05-04T23:29:40.427" CommunityOwnedDate="2011-05-04T23:29:40.427" CommentCount="3" />
  <row Id="10334" PostTypeId="2" ParentId="10295" CreationDate="2011-05-05T01:09:16.273" Score="1" ViewCount="" Body="&lt;p&gt;This is my theory. I think there are two kinds of questions: those that remain mostly within SE (which usually have fewer views), and those that are viewed by outsiders because it was linked from somewhere else (usually have more views). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the questions that remain mostly within SE, votes are a good measure of interesting questions. This is the point of votes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When a question is linked to outside the site the votes stop meaning as much. Some linking sites may have very few SE members, others may have more. The variance of the number of votes for these questions is probably high (as evidenced by your score vs view plot, where the right side of the curve blooms out). These questions will have more views, and views MAY be a better indicator of interesting questions. Or questions that a larger community happened to find more interesting. There are many variables in this situation, and I think it would be worth trying to find more information to differentiate these cases. Does SE publicize referral information? &lt;/p&gt;&#xA;" OwnerUserId="2965" LastActivityDate="2011-05-05T01:09:16.273" />
  <row Id="10335" PostTypeId="2" ParentId="10295" CreationDate="2011-05-05T01:16:53.857" Score="3" ViewCount="" Body="&lt;p&gt;One might define an interesting question as one that has received comparatively many votes given the number of views. To this end, you can create a baseline curve that reflects the expected number of votes given the views. Curves that attracted a lot more votes than the baseline were considered particularly interesting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To construct the baseline, you may want to calculate the median number of votes per 100-view bin. In addition, you could calculate the median absolute deviation (MAD) as a robust measure for the standard deviation per bin. Then, &quot;interestingness&quot; can be calculated as &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;interestingness(votes,views) = (votes-baselineVotes(views))/baselineMAD(views) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="198" LastActivityDate="2011-05-05T01:16:53.857" />
  <row Id="10336" PostTypeId="2" ParentId="10328" CreationDate="2011-05-05T01:29:25.800" Score="4" ViewCount="" Body="&lt;p&gt;Trying to translate this into a statistical question, it seems you have a population with $a$ members and you take two random samples without replacement sized $b$ and $c$, and you want the distribution of $X$, the number appearing in both samples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an illustration, suppose $a=5$, $b=2$ and $c=3$. There are 100 ways of taking the samples, of which 10 have none in common, 60 have one in common and 30 have two in common.  It the language of black and white balls in an urn, the urn has $b=2$ white balls and $a-b=3$ black balls, and we take $c=3$ balls out to inspect how many white balls come out.  In R we can effectively get these values with  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; totalpop &amp;lt;- 5 &#xA;&amp;gt; sample1  &amp;lt;- 2&#xA;&amp;gt; sample2  &amp;lt;- 3 &#xA;&amp;gt; dhyper(0:2, sample1, totalpop-sample1, sample2) &#xA;[1] 0.1 0.6 0.3&#xA;&amp;gt; phyper(-1:2, sample1, totalpop-sample1, sample2) &#xA;[1] 0.0 0.1 0.7 1.0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which confirms the earlier calculations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to test a number &lt;code&gt;overlap&lt;/code&gt;, then the probability of getting that number or smaller from this model is &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;phyper(overlap, sampleb, totala - sampleb, samplec) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and of getting that number or larger is  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1 - phyper(overlap - 1, sampleb, totala - sampleb, samplec)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-05-05T01:29:25.800" CommentCount="1" />
  <row Id="10337" PostTypeId="1" AcceptedAnswerId="10339" CreationDate="2011-05-05T03:42:06.087" Score="1" ViewCount="84" Body="&lt;p&gt;What would be the $Var(x^2)$, if $Var(x)=\sigma^2$?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="3903" LastActivityDate="2011-05-05T06:35:12.517" Title="$Var(x^2)$, if $Var(x)=\sigma^2$" Tags="&lt;mathematical-statistics&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="10338" PostTypeId="1" CreationDate="2011-05-05T05:47:30.180" Score="2" ViewCount="50" Body="&lt;p&gt;I want to perform bootstrapping for calculation of efficiency score from data envelopment analysis (DEA) using R.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Are there any examples of data and results for this type of analysis in R to enable me to check my results?&lt;/li&gt;&#xA;&lt;li&gt;Are there any online or other resources that might assist my task?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="4472" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-05T08:03:28.140" LastActivityDate="2011-05-05T08:03:28.140" Title="Bootstrapping data envelopment analysis efficiency score using R" Tags="&lt;r&gt;&lt;bootstrap&gt;&lt;efficiency&gt;" CommentCount="3" />
  <row Id="10339" PostTypeId="2" ParentId="10337" CreationDate="2011-05-05T06:35:12.517" Score="1" ViewCount="" Body="&lt;p&gt;As a simple example of the responses of @user2168 and @mpiktas:&#xA;The variance of the set of values 1,2,3 is 0.67, while the variance of its square is 10.89. On the other hand, the variance of 2,3,4 is also 0.67, but the variance of the squares is 24.22.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These are just variances for finite sets of data, but the idea extends to distributions.&lt;/p&gt;&#xA;" OwnerUserId="4257" LastActivityDate="2011-05-05T06:35:12.517" CommentCount="1" />
  <row Id="10340" PostTypeId="2" ParentId="10302" CreationDate="2011-05-05T07:07:56.453" Score="2" ViewCount="" Body="&lt;p&gt;You have looked at the &lt;a href=&quot;http://en.wikipedia.org/wiki/Perplexity&quot; rel=&quot;nofollow&quot;&gt;Wikipedia article on perplexity&lt;/a&gt;. It gives the perplexity of a discrete distribution as &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$2^{-\sum_x p(x)\log_2 p(x)}$$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;which could also be written as &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\exp\left({\sum_x p(x)\log_e \frac{1}{p(x)}}\right)$$  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;i.e. as a weighted geometric average of the inverses of the probabilities. For a continuous distribution, the sum would turn into a integral.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The article also gives a way of estimating perplexity for a model using $N$ pieces of test data &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$2^{-\sum_{i=1}^N \frac{1}{N} \log_2 q(x_i)}$$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;which could also be written &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\exp\left(\frac{{\sum_{i=1}^N \log_e \left(\dfrac{1}{q(x_i)}\right)}}{N}\right) \text{ or } \sqrt[N]{\prod_{i=1}^N  \frac{1}{q(x_i)}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;or in a variety of other ways, and this should make it even clearer where &quot;log-average inverse probability&quot; comes from. &lt;/p&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-05-05T07:07:56.453" />
  <row Id="10342" PostTypeId="1" CreationDate="2011-05-05T08:06:46.827" Score="2" ViewCount="77" Body="&lt;p&gt;Given a panel of countries over time, a fixed effects estimator makes sense to control for country-specific effects. My intuition tells me that if the dependent variable is correlated with lags of the independent variables, then bias will be introduced into the estimator. However, I'm having difficulty rigorously understanding why this would be the case. Additionally, is there any easy way to tell if the bias would be towards or away from zero?&lt;/p&gt;&#xA;" OwnerUserId="726" LastActivityDate="2011-05-05T08:06:46.827" Title="Panel Data: In a fixed effects model, does auto-correlation introduce bias?" Tags="&lt;autocorrelation&gt;&lt;panel-data&gt;&lt;fixed-effects-model&gt;" CommentCount="1" FavoriteCount="2" />
  <row Id="10343" PostTypeId="1" AcceptedAnswerId="10351" CreationDate="2011-05-05T08:11:02.280" Score="2" ViewCount="101" Body="&lt;p&gt;I am totally confused: On the one hand you can read all kinds of explanations why you have to divide by n-1 to get an unbiased estimator for the (unknown) population variance (degrees of freedom, not defined for sample size 1 etc.) - see e.g. &lt;a href=&quot;http://en.wikipedia.org/wiki/Bessel%27s_correction&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; or &lt;a href=&quot;http://stats.stackexchange.com/questions/3931/intuitive-explanation-for-dividing-in-n-1-when-calculating-sd&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand when it comes to variance estimation of a supposed normal distribution all of this doesn't seem to be true anymore. There it is said that the maximum likelihood estimator for variance includes only a division by n - see e.g. &lt;a href=&quot;http://en.wikipedia.org/wiki/Normal_distribution#Estimation_of_parameters&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, can anyone please enlighten me why it is true here but not there? I mean normality is what most models boil down to (not least due to the &lt;a href=&quot;http://en.wikipedia.org/wiki/Central_limit_theorem&quot; rel=&quot;nofollow&quot;&gt;CLT&lt;/a&gt;). So is the choice &quot;division by n&quot; yet the best choice for finding the best estimation for the true population variance after all?!?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you!&lt;/p&gt;&#xA;" OwnerUserId="230" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-05T09:06:42.860" LastActivityDate="2011-05-05T12:15:11.133" Title="When estimating variance, why do unbiased estimators divide by n-1 yet maximum likelihood estimates divide by n?" Tags="&lt;normal-distribution&gt;&lt;variance&gt;&lt;unbiased-estimator&gt;" AnswerCount="2" FavoriteCount="2" />
  <row Id="10344" PostTypeId="2" ParentId="10343" CreationDate="2011-05-05T08:18:56.053" Score="3" ViewCount="" Body="&lt;p&gt;The MLE is indeed found through division by n. However, MLE's are not guaranteed to be unbiased. So there is no contradiction in the fact that the unbiased estimator (divided by n-1) is used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In practice, for reasonable sample sizes, it should not make a big difference anyway.&lt;/p&gt;&#xA;" OwnerUserId="4257" LastActivityDate="2011-05-05T08:18:56.053" CommentCount="3" />
  <row Id="10345" PostTypeId="2" ParentId="423" CreationDate="2011-05-05T09:27:32.327" Score="1" ViewCount="" Body="&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/z55DL.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Found this one in the &lt;a href=&quot;http://www.stat.columbia.edu/~cook/movabletype/archives/2011/04/worst_statistic.html#comment-2512168&quot; rel=&quot;nofollow&quot;&gt;comments on Andrew Gelman's blog&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="442" LastActivityDate="2011-05-05T09:27:32.327" CommunityOwnedDate="2011-05-05T09:27:32.327" />
  <row Id="10346" PostTypeId="1" AcceptedAnswerId="10348" CreationDate="2011-05-05T09:34:46.440" Score="3" ViewCount="58" Body="&lt;p&gt;I am running a logistic regression with customer event data with multiple predictors.&#xA;However, one variable is extremely important, alone predicting 60% of the customers for the event. When this main predictor is included in the model, other predictors add very little to prediction over and above this main predictor. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This main predictor is not a post event variable. The variable has full business support to be in the model.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Given this, is it still okay to retain this main predictor variable in the model?&lt;/li&gt;&#xA;&lt;li&gt;Does this suggest that there is anything wrong with the model?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="1763" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-05T10:31:15.537" LastActivityDate="2011-05-05T10:32:03.557" Title="Is it problematic if one predictor in a set accounts for almost all the prediction?" Tags="&lt;logistic-regression&gt;&lt;modeling&gt;" AnswerCount="1" CommentCount="4" />
  <row Id="10347" PostTypeId="1" CreationDate="2011-05-05T09:39:26.173" Score="2" ViewCount="95" Body="&lt;p&gt;I have made a heatmap based upon a regular data matrix in R, the package I use is &lt;code&gt;pheatmap&lt;/code&gt;. Regular clustering of my samples is performed by the &lt;code&gt;distfun&lt;/code&gt; function within the package. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I want to attach a precomputed distance matrix (generated by Unifrac) to my previously generated matrix/heatmap. Is this possible?&lt;/p&gt;&#xA;" OwnerUserId="4473" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-05T11:18:43.750" LastActivityDate="2011-05-05T11:18:43.750" Title="Making a heatmap with a precomputed distance matrix and data matrix in R" Tags="&lt;r&gt;&lt;data-visualization&gt;" AnswerCount="1" CommentCount="6" />
  <row Id="10348" PostTypeId="2" ParentId="10346" CreationDate="2011-05-05T10:32:03.557" Score="1" ViewCount="" Body="&lt;p&gt;I understand your gut feeling. But depending on the type of response and predictor, this has not to be unsual (example: response = &quot;weight&quot;, predictors &quot;height&quot; and others with presumably less meaning like &quot;state&quot;, &quot;favorite movie&quot; etc.).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, you should check that for the creation of the predictor only information has been used which was available at that time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an example:&#xA;Suppose you want to predict an event of the next day. You have raw data from day 1 to 10. Now you create one predictor by calculating a statistic over all 10 days and other predictors using only &quot;local&quot; information (i.e. directly extracted from a single event on a single day, i.e. &quot;event start time&quot;). Now you build a model from day 1 to 5 using this predictors to predict the events on day 6. The model is flawed because it contains information from day 6-10, information which is never available when building the model for real (i.e. not simulation) usage.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If this check has been succesful (I guess you mean that with &quot;not a post-event variable&quot;), then it shouldn't be a problem. In one of my projects, something similar occurred. Checking on the meaning of the variables indeed revealed, that the detected relationship (i.e. variable importance) was trivial, but true ;).&lt;/p&gt;&#xA;" OwnerUserId="264" LastActivityDate="2011-05-05T10:32:03.557" CommentCount="3" />
  <row Id="10349" PostTypeId="2" ParentId="10347" CreationDate="2011-05-05T11:02:11.647" Score="1" ViewCount="" Body="&lt;p&gt;Ok, so you can just look at the code by typing the name of the function at the R prompt, or use &lt;code&gt;edit(pheatmap)&lt;/code&gt; to see it in your default editor.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Around line 14 and 23, you'll see that another function is called for computing the distance matrices (for rows and columns), given a distance function (R &lt;code&gt;dist&lt;/code&gt;) and a method (compatible with &lt;code&gt;hclust&lt;/code&gt; for hierarchical clustering in R). What does this function do? Use &lt;code&gt;getAnywhere(&quot;cluster_mat&quot;)&lt;/code&gt; to print it on screen, and you soon notice that it does nothing more than returning an &lt;code&gt;hclust&lt;/code&gt; object, that is your dendrogram computed from the specified distance and linkage options.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, if you already have your distance matrix, change line 14 (rows) or 23 (columns) so that it reads, e.g.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;tree_row = hclust(my.dist.mat, method=&quot;complete&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where &lt;code&gt;my.dist.mat&lt;/code&gt; is your own distance function, and &lt;code&gt;complete&lt;/code&gt; is one of the many methods available in &lt;code&gt;hclust&lt;/code&gt; (see &lt;code&gt;help(hclust)&lt;/code&gt;). Here, it is important to use &lt;code&gt;fix(pheatmap)&lt;/code&gt; and not &lt;code&gt;edit(pheatmap)&lt;/code&gt;; otherwise, the edited function will not be callable in the correct environment/namespace.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a quick and dirty hack that I would not recommend with larger package. It seems to work for me at least, that is I can use a custom distance matrix with complete linkage for the rows.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In sum, assuming your distance matrix is stored in a variable named &lt;code&gt;dd&lt;/code&gt;,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(pheatmap)&#xA;fix(pheatmap)&#xA;# 1. change the function as you see fit&#xA;# 2. save and go back to R&#xA;# 3. if your custom distance matrix was simply read as a matrix, make sure&#xA;#    it is read as a distance matrix&#xA;my.dist.map &amp;lt;- dd  # or as.dist(dd)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then, you can call &lt;code&gt;pheatmap&lt;/code&gt; as you did but now it will use the results of &lt;code&gt;hclust&lt;/code&gt; applied to &lt;code&gt;my.dist.map&lt;/code&gt; with &lt;code&gt;complete&lt;/code&gt; linkage. Please note that you just have to ensure that &lt;code&gt;cluster_rows=TRUE&lt;/code&gt; (which is the default). Now, you may be able to change&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;the linkage method&lt;/li&gt;&#xA;&lt;li&gt;choose between rows or columns&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;by editing the package function appropriately.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-05-05T11:02:11.647" CommentCount="2" />
  <row Id="10350" PostTypeId="1" AcceptedAnswerId="10352" CreationDate="2011-05-05T11:20:13.343" Score="4" ViewCount="62" Body="&lt;p&gt;I am working on insurance data in which a customer has a field named &lt;code&gt;customer_no_dependent&lt;/code&gt; (customer's number of dependent). Its coming out to be a significant variable( just that it has $p&amp;lt;0.0001$). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This variable has almost 20% missing values. For imputation, I thought to determine proxy indicators for number of dependents. I tried age (thinking a person more aged could be having more dependents). I correlated it with premium amount as well to think that a person who has more dependents could be having less disposable income. So low premium paying could be meaning more dependents. I do understand that a demographic variable can't be fully taken out from such logic. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, if somebody goes into detail, he can prove my imputation to be far from perfect. What should I do in such situation? Would deleting those 20% be a correct solution? 20% for my data would be close to 2 lakh rows which is large amount of information..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know, this question can have many possible answers. I would be grateful for any pointers how to proceed.&lt;/p&gt;&#xA;" OwnerUserId="1763" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-05T12:10:32.370" LastActivityDate="2011-05-05T12:24:24.350" Title="Advice on missing value imputation" Tags="&lt;data-imputation&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="10351" PostTypeId="2" ParentId="10343" CreationDate="2011-05-05T12:15:11.133" Score="3" ViewCount="" Body="&lt;p&gt;The answer to your question is contained within your question. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When choosing an estimator for a parameter, you should ask yourself,&#xA;what property would you like your estimator to have:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Robustness&lt;/li&gt;&#xA;&lt;li&gt;Unbiasedness&lt;/li&gt;&#xA;&lt;li&gt;Have the distributional properties of a MLE&lt;/li&gt;&#xA;&lt;li&gt;Consistency&lt;/li&gt;&#xA;&lt;li&gt;Asymptotically normal&lt;/li&gt;&#xA;&lt;li&gt;You know the population mean, but the variance is unknown&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If your estimator is the one that is divided by (n-1), then you want an&#xA;unbiased estimtor of the variance. If your estimator is the one that is&#xA;divided by n, then you have an MLE estimator. Of course, when n is large;&#xA;dividing by either (n-1) or n will give you approximately the same results&#xA;and the estimator will be approximately unbiased and have the properties&#xA;of all MLE estimators.&lt;/p&gt;&#xA;" OwnerUserId="3805" LastActivityDate="2011-05-05T12:15:11.133" />
  <row Id="10352" PostTypeId="2" ParentId="10350" CreationDate="2011-05-05T12:24:24.350" Score="4" ViewCount="" Body="&lt;p&gt;First of all: it is not clear from your explanation whether or not you have done multiple imputation. If not: please do so: single imputation could be worse than simple complete case analysis, and can both lead to severely biased results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Next, if I understand correctly, your problem is that you don't know which variables to use as covariates for you imputation model. If you number of possible covariates (I assume these are the other covariates in your model of interest) is limited, you could opt for the nonparametric kind of imputation that is offered by MICE (in R) and similar algorithms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another option is to use shrinkage (LASSO or alike) in a model predicting customer_no_dependent: this should give you a set of likely predictors. Be aware though, that this step induces even more uncertainty (you reuse the data yet again), and you should trust your confidence intervals and p-values somewhat less. The effect should be negligble if your association is truly as strong as you indicate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you do use the kind of parametric and commonsense induced imputation mechanism (like regressing on 'credible' predictors): simply make note of this fact, and mention that the obtained results are conditional on this extra set of assumptions.&lt;/p&gt;&#xA;" OwnerUserId="4257" LastActivityDate="2011-05-05T12:24:24.350" CommentCount="1" />
  <row Id="10353" PostTypeId="1" CreationDate="2011-05-05T12:26:43.317" Score="3" ViewCount="102" Body="&lt;h3&gt;Context:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;My question concerns a typical design in my area – a researcher takes a group of subjects (say 10) and then applies three different conditions to them to measure the change in a response variable, e.g. vertical jump height performed after drinking a glucose drink, coloured plain water, and fruit juice (say).  Every subject has every treatment, but in a random order with enough time between for effects to ‘wash out’.  &lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Quote:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Kuehl (2000) (Kuehl, R. O. (2009)&#xA; Design of Experiments: Statistical&#xA; principles of research design and&#xA; analysis, Duxbury Press, CA, p497 2nd&#xA;Ed.) states: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;When each of the&#xA;  treatments is administered in a random&#xA;  order to each subject... then subjects&#xA;  are random blocks in a randomised&#xA;  complete block design”, and then shows&#xA;  the corresponding analysis.  In this&#xA;  case, the subject is a random effect,&#xA;  but a nuisance or blocking factor, and&#xA;  although our statistical model will&#xA;  test the significance of the block&#xA;  factor, we aren’t really interested in&#xA;  its significance.  However, many&#xA;  researchers (and reviewers!) think&#xA;  that such a design should be analysed&#xA;  as a repeated measures design with a&#xA;  Mauchly test for the Huynh-Feldt&#xA;  condition (with the treatment as the&#xA;  repeated measure).  However, this&#xA;  seems more appropriate when a time&#xA;  factor is being analysed – for example&#xA;  when observations are taken at 0&#xA;  minutes, 10 minutes, 30 minutes and 60&#xA;  minutes, for example.  In this case&#xA;  the covariance between pairs of time&#xA;  points might reasonably be expected to&#xA;  change, particularly when unequal time&#xA;  intervals are used.  [In fact, I use&#xA;  SAS to model different covariance&#xA;  structures in this case (e.g.&#xA;  autoregressive) and use the AIC to&#xA;  choose the best structure, though this&#xA;  is not an approach that is well&#xA;  received by many reviewers.]  I&#xA;  understood that when the subject is a&#xA;  block factor, and the different&#xA;  treatments are administered in a&#xA;  random order that is different for&#xA;  different subjects, this means that&#xA;  the correlation between observations&#xA;  is different for each subject so&#xA;  compound symmetry can be assumed.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;h3&gt;Question:&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;How should repeated measures ANOVAs with 3 or more conditions presented in random order be analysed?&lt;/li&gt;&#xA;&lt;li&gt;Is it reasonable to assume compound symmetry?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="4474" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-17T06:39:20.567" LastActivityDate="2011-05-17T06:39:20.567" Title="How to analyse repeated measure ANOVA with three or more conditions presented in randomised order?" Tags="&lt;hypothesis-testing&gt;&lt;anova&gt;&lt;repeated-measures&gt;" CommentCount="2" FavoriteCount="1" />
  <row Id="10354" PostTypeId="2" ParentId="9653" CreationDate="2011-05-05T12:28:58.153" Score="2" ViewCount="" Body="&lt;p&gt;Another consequence of a small sample is the increase of type 2 error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Nunnally demonstrated in the paper &quot;The place of statistics in psychology&quot;, 1960, that small samples generally fail to reject a point null hypothesis. These hypothesis are hypothesis having some parameters equals zero, and are known to be false in the considered experience.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the opposite, too large samples increase the type 1 error because the p-value depends on the size of the sample, but the alpha level of significance is fixed. A test on such a sample will always reject the null hypothesis. Read &quot;The insignificance of statistical significance testing&quot; by Johnson and Douglas (1999) to have an overview of the issue.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is not a direct answer to the question but these considerations are complementary.&lt;/p&gt;&#xA;" OwnerUserId="4443" LastActivityDate="2011-05-05T12:28:58.153" CommentCount="1" />
  <row Id="10355" PostTypeId="1" CreationDate="2011-05-05T12:54:59.907" Score="0" ViewCount="45" Body="&lt;p&gt;Hi,&#xA;This is a pretty basic question as I'm new to stats, but hopefully someone can help!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a sample size of 30,000 records and so far 6500 have been checked to see if an error exists - of these 6500, 1124 have an error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I would like to be able to do is the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) See how many records I would need to check to find 97% of all error records (this is a deliberately fixed value)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Determine if by checking 6500 records I had actually checked enough to have found all of the (97% of the total) error cases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is similar to one found &lt;a href=&quot;http://stats.stackexchange.com/questions/7902/how-to-calculate-confidence-interval-when-only-a-part-of-the-samples-are-valid&quot;&gt;here&lt;/a&gt; but is different enough for me to be unsure of how to apply it!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;TIA, and hope it makes sense!&#xA;Davin&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;edit to include sample data&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data in question can be found &lt;a href=&quot;http://hotfile.com/dl/116850309/76b905b/Sample_Data.xlsx.html&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;:&lt;/p&gt;&#xA;" OwnerUserId="4475" LastEditorUserId="4475" LastEditorDisplayName="" LastEditDate="2011-05-05T14:41:17.897" LastActivityDate="2011-05-05T14:41:17.897" Title="Sample size needed to find all error values" Tags="&lt;confidence-interval&gt;&lt;sampling&gt;&lt;sample-size&gt;" CommentCount="7" />
  <row Id="10356" PostTypeId="1" CreationDate="2011-05-05T14:03:29.927" Score="7" ViewCount="168" Body="&lt;p&gt;I am building a propensity model using logistic regression for a utility client. &#xA;My concern is that out of the total sample my 'bad' accounts are just 5%,  and the rest are all good. &#xA;I am predicting 'bad'. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Will the result be biassed? &lt;/li&gt;&#xA;&lt;li&gt;What is optimal 'bad to good proportion' to build a good model?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="4478" LastEditorUserId="495" LastEditorDisplayName="" LastEditDate="2011-05-06T14:07:42.000" LastActivityDate="2011-05-06T16:16:21.933" Title="Is a logistic regression biased when the outcome variable is split 5% - 95%?" Tags="&lt;logistic-regression&gt;&lt;modeling&gt;" AnswerCount="4" FavoriteCount="0" />
  <row Id="10357" PostTypeId="2" ParentId="10356" CreationDate="2011-05-05T14:42:39.460" Score="0" ViewCount="" Body="&lt;p&gt;In theory, you will be able to discriminate better if the proportions of &quot;good&quot; and &quot;bad&quot; are roughly similar in size.   You might be able to move towards this by stratified sampling, oversampling bad cases and then reweighting to return to the true proportions later. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This carries some risks.  In particular your model is likely to be labelling individuals as &quot;potentially bad&quot; - presumably those who may not pay their utility bills when due.  It is important that the impact of errors when doing this are properly recognised: in particular how many &quot;good customers&quot; will be labelled &quot;potentially bad&quot; by the model, and you are less likely to get the reweighting wrong if you have not distorted your model by stratified sampling.   &lt;/p&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-05-05T14:42:39.460" CommentCount="5" />
  <row Id="10358" PostTypeId="2" ParentId="10271" CreationDate="2011-05-05T14:44:07.957" Score="1" ViewCount="" Body="&lt;p&gt;You might find &lt;a href=&quot;http://www.stat.duke.edu/~mw/Smith+West1983.pdf&quot; rel=&quot;nofollow&quot;&gt;this paper&lt;/a&gt; of interest. See also more detailed presentation of similar models in &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0387947256&quot; rel=&quot;nofollow&quot;&gt;West &amp;amp; Harrison&lt;/a&gt;. There are other examples of this sort of monitoring as well, many which are more recent, but this isn't exactly my wheelhouse :). Undoubtedly there are suitable implementations of these models, but I don't know what they might be offhand...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The basic idea is that you have a switching model where some observations/sequence of observations are attributed to abnormal network states while the rest are considered normal. A mixture like this could account for the long right tail in your first plot. A dynamic model could also alert you to abnormal jumps like at 8:00 and 4:00 in real-time by assigning high probability to new observations belonging to a problem state. It could also be easily extended to include things like predictors, periodic components (perhaps your score rises/falls a bit with activity) and that sort of thing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: I should also add, this kind of model is &quot;unsupervised&quot; in the sense that anomalies are caught either by showing a large mean shift or increase in variance. As you gather data you can improve the model with more informative prior distributions. But perhaps once you have enough data (and hard-won training examples by dealing with network problems!) you could devise some simple monitoring rules (thresholds, etc)&lt;/p&gt;&#xA;" OwnerUserId="26" LastEditorUserId="26" LastEditorDisplayName="" LastEditDate="2011-05-05T14:49:44.440" LastActivityDate="2011-05-05T14:49:44.440" />
  <row Id="10359" PostTypeId="1" CreationDate="2011-05-05T14:48:53.297" Score="4" ViewCount="73" Body="&lt;p&gt;How can I generate dependent time series from a given marginal distribution? I want to be able to adjust the level of dependence, to influence the predictability of the series, which will be given as input to a Monte Carlo simulation. The dependence parameter can be the correlation, the mutual information, or something along those lines.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may assume the distribution is Bernoulli for discussion purposes. MATLAB code is gratefully accepted.&lt;/p&gt;&#xA;" OwnerUserId="4479" LastEditorUserId="4479" LastEditorDisplayName="" LastEditDate="2011-05-06T14:42:03.297" LastActivityDate="2011-05-09T15:45:40.490" Title="Generating dependent time series from a given distribution?" Tags="&lt;time-series&gt;&lt;monte-carlo&gt;&lt;simulation&gt;&lt;non-independent&gt;&lt;prediction-limit&gt;" AnswerCount="2" CommentCount="6" FavoriteCount="1" />
  <row Id="10360" PostTypeId="2" ParentId="10271" CreationDate="2011-05-05T14:56:16.697" Score="2" ViewCount="" Body="&lt;p&gt;The graph of the &quot;original series&quot; does not have to exhibit any pre-defined structure. What is critical is that the graph of the &quot;residuals from a suitable model series&quot; need to exhibit either a gaussian structure . This &quot;gaussian structure&quot; can usually obtained by incorporating one or more of the following &quot;transformations&quot;&#xA;1. an arima MODEL&#xA;2. Adjustments for Local Level Shifts or Local Time Trends or Seasonal Pulses or Ordinary Pulses&#xA;3. a weighted analysis exploiting proven variance heterogeneity&#xA;4. a possible power transformation ( logs etc ) to deal with a specific variance heterogenity&#xA;5. the detection of points in time where the model/parameters may have changed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Intervention Detection will yield a statement about the statistical significance of the most recent event suggesting either normalcy or an anomaly&lt;/p&gt;&#xA;" OwnerUserId="3382" LastActivityDate="2011-05-05T14:56:16.697" />
  <row Id="10361" PostTypeId="1" CreationDate="2011-05-05T15:13:41.393" Score="4" ViewCount="89" Body="&lt;p&gt;My homework question:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;An inspector suspects that the food in the factory she is inspecting has been contaminated with a harmful chemical c. Such chemical contamination occurs in 5% of factories producing this food. The inspector has a test A for the chemical which registers positive with 100% certainty when the chemical is present, but the test also registers positive in 10% of cases where the chemical is not present. She decides to use this test to help her decide whether there is contamination.&lt;/p&gt;&#xA;  &#xA;  &lt;ol&gt;&#xA;  &lt;li&gt;Assume that the prior probability of contamination is equal to the base rate, and that the&#xA;  inspector’s test shows a positive result. Compute the posterior probability of contamination.&lt;/li&gt;&#xA;  &lt;li&gt;The inspector has another test B for chemical c which only registers positive 50% of the time when c is present, but has the advantage of never giving a false positive (i.e., if c is not present, the test will never say it is). The results of the two tests, A and B, are independent given the presence or absence of c. It turns out that when the inspector uses test B, the results are negative. In addition, the inspector knows that the factory is poorly maintained. The rate of contamination in factories with poor maintenance is twice as high as the rate in factories overall. Compute the posterior probability of contamination.&lt;/li&gt;&#xA;  &lt;/ol&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I know these should be rather basic, but I'm getting stuck. For #1 I've reached an answer, but I'm not sure it's correct:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using Bayes rule the probability should be&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(c|positive) = \frac{P(positive|c)P(c)}{P(positive)}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I think that $P(positive)$ should be:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(positive) = P(positive,c)+P(positive,\neg c) = P(positive|c)P(c) + P(positive|\neg c)P(\neg c)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(c|positive) = \frac{P(positive|c)P(c)}{P(positive|c) P(c) + P(positive|\neg c) P(\neg c)}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(c|positive) = \frac{1 * 0.05}{1 * 0.05 + 0.1 * 0.95} = 0.34$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Is this correct?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In part #2 I thought since they are supposed to be independent this must hold:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ P(A_{positive} \cap B_{negative}|c) = P(A_{positive}|c)*P(B_{negative}|c) $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First I adjusted the base rate and recalculated #1:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(c|A_{positive}) = \frac{1 * 0.1}{1 * 0.1 + 0.1 * 0.9} = 0.52$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The questions says: &quot;the results are negative&quot;. So B should be:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(c|B_{negative}) = \frac{P(B_{negative}|c)P(c)}{P(B_{negative}|c) P(c) + P(B_{negative}|\neg c) P(\neg c)} = \frac{0.5 * 0.1}{0.5 * 0.1 + 1 * 0.9} = 0.05$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ P(c|A_{positive} \cap B_{negative}) = \frac{P(A_{positive} \cap B_{negative}|c) * P(c)}{P(A_{positive} \cap B_{negative})} = \frac{P(A_{positive}|c) * P(B_{negative}|c) * P(c)}{P(A_{positive} \cap B_{negative})} $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What should I do next?&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="2261" LastEditorUserId="2261" LastEditorDisplayName="" LastEditDate="2011-05-05T18:12:21.797" LastActivityDate="2011-06-04T18:54:39.510" Title="Bayes rule and base rate" Tags="&lt;homework&gt;" AnswerCount="1" CommentCount="6" FavoriteCount="1" />
  <row Id="10362" PostTypeId="2" ParentId="10356" CreationDate="2011-05-05T15:15:12.283" Score="2" ViewCount="" Body="&lt;p&gt;Assymptotically, the ratio of positive to negative patterns is essentially irrelevant.  The problem arises principally when you have too few samples of the minority class to adequately describe its statistical distribution.  Making the dataset larger generally solves the problem (where that is possible).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If this is not possible, the best thing to do is to re-sample the data to get a balanced dataset, and then apply a multiplicative adjustment to the output of the classifier to compensate for the difference between training set and operational relative class frequencies.  While you can calculate the (assymptitically) optimal adjustment factor, in practice it is best to tune the adjustment using cross-validation (as we are dealing with a finite practical case rather than an assymtotic one).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this sort of situation, I often use a committe of models, where each is trained on all of the minority patters and a different random sample of the majority patterns of the same size as the minority patterns.  This guards against bad luck in the selection of a single subset of the majority patterns.&lt;/p&gt;&#xA;" OwnerUserId="887" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-05T15:40:08.780" LastActivityDate="2011-05-05T15:40:08.780" CommentCount="10" />
  <row Id="10363" PostTypeId="1" CreationDate="2011-05-05T16:26:00.037" Score="10" ViewCount="377" Body="&lt;p&gt;I'm curious about repeatable procedures that can be used to discover the functional form of the function &lt;code&gt;y = f(A, B, C) + error_term&lt;/code&gt; where my only input is a set of observations (&lt;code&gt;y&lt;/code&gt;, &lt;code&gt;A&lt;/code&gt;, &lt;code&gt;B&lt;/code&gt; and &lt;code&gt;C&lt;/code&gt;). Please note that the functional form of &lt;code&gt;f&lt;/code&gt;is unknown.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider the following dataset:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;AA  BB  CC  DD  EE  FF&#xA;==  ==  ==  ==  ==  ==&#xA;98  11  66  84  67  10500&#xA;71  44  48  12  47  7250&#xA;54  28  90  73  95  5463&#xA;34  95  15  45  75  2581&#xA;56  37  0   79  43  3221&#xA;68  79  1   65  9   4721&#xA;53  2   90  10  18  3095&#xA;38  75  41  97  40  4558&#xA;29  99  46  28  96  5336&#xA;22  63  27  43  4   2196&#xA;4   5   89  78  39  492&#xA;10  28  39  59  64  1178&#xA;11  59  56  25  5   3418&#xA;10  4   79  98  24  431&#xA;86  36  84  14  67  10526&#xA;80  46  29  96  7   7793&#xA;67  71  12  43  3   5411&#xA;14  63  2   9   52  368&#xA;99  62  56  81  26  13334&#xA;56  4   72  65  33  3495&#xA;51  40  62  11  52  5178&#xA;29  77  80  2   54  7001&#xA;42  32  4   17  72  1926&#xA;44  45  30  25  5   3360&#xA;6   3   65  16  87  288&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In this example, assume that we know that &lt;code&gt;FF = f(AA, BB, CC, DD, EE) + error term&lt;/code&gt;, but we're not sure about the functional form of &lt;code&gt;f(...)&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What procedure/what methods would you use to arrive at finding the functional form of &lt;code&gt;f(...)&lt;/code&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Bonus point: What is your best guess at the definiton of &lt;code&gt;f&lt;/code&gt; given the data above? :-) And yes, there is a &quot;correct&quot; answer that will yield an &lt;code&gt;R^2&lt;/code&gt; in excess of 0.99.)&lt;/p&gt;&#xA;" OwnerUserId="914" LastEditorUserId="914" LastEditorDisplayName="" LastEditDate="2011-05-06T09:57:11.100" LastActivityDate="2011-05-09T22:05:01.390" Title="Data mining: How should I go about finding the functional form?" Tags="&lt;regression&gt;&lt;machine-learning&gt;&lt;algorithms&gt;&lt;model-selection&gt;&lt;data-mining&gt;" AnswerCount="6" CommentCount="6" FavoriteCount="5" />
  <row Id="10364" PostTypeId="2" ParentId="10361" CreationDate="2011-05-05T18:09:20.020" Score="0" ViewCount="" Body="&lt;p&gt;From your independence statement,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{equation} &#xA;P( A_{+} \bigcap B_{+} ) = P(A_{+} ) P(B_{+})&#xA;\nonumber&#xA;\end{equation}&#xA;and the definitions&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{equation} &#xA;P(A_{+} \bigcap B_{+} | c) \equiv \frac{P(A_{+} \bigcap B_{+} \bigcap c)}{P(c)}&#xA;\nonumber&#xA;\end{equation}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{equation}&#xA;P(A_{+} | c) \equiv  \frac{P(A_{+} \bigcap c)}{P(c)}&#xA;\nonumber&#xA;\end{equation}&#xA;and&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{equation}&#xA;P(B_{+} | c) \equiv  \frac{P(B_{+} \bigcap c)}{P(c)}&#xA;\nonumber&#xA;\end{equation}&#xA;can you algebraically show that&#xA;\begin{equation} &#xA;P(A_{+} \bigcap B_{+} | c) \equiv  \frac{P(A_{+} \bigcap B_{+} \bigcap c)}{P(c)}&#xA; = \frac{P(A_{+} \bigcap c)}{P(c)} \frac{P(B_{+} \bigcap c)}{P(c)} \equiv   P(A_{+} | c) P(B_{+} | c)&#xA;\nonumber&#xA;\end{equation}&lt;/p&gt;&#xA;" OwnerUserId="3805" LastActivityDate="2011-05-05T18:09:20.020" />
  <row Id="10365" PostTypeId="1" CreationDate="2011-05-05T18:53:24.037" Score="1" ViewCount="37" Body="&lt;p&gt;I have written some cost functions to find the parameters of a distribution that has a particular confidence interval.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, I wrote the following that can be used to find the parameters &lt;code&gt;parms&lt;/code&gt; of a gamma distribution given a vector &lt;code&gt;x&lt;/code&gt; used to state the 95% CI: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cost.fn &amp;lt;- function(parms, x) {&#xA;   a &amp;lt;- abs(qgamma(0.025, parms[1], parms[2])-x[1])&#xA;   b &amp;lt;- abs(qgamma(0.975, parms[1], parms[2])-x[2])&#xA;   return(a+b)&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I would like to write these out as succinctly as possible, but the best that I can come up with is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\text{cost} = |Q_{0.025|\text{parms}} - x_1| + |Q_{0.975|\text{parms}} - x_2|$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;or perhaps:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\text{min}( |Q_{\text{G}(a,b)}(0.05) - x_1| + |Q_{\text{G}(a,b)}(0.95) - x_2| )$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In english: the cost equals the sum of the difference between the target lower confidence interval and the 2.5\% quantile of the gamma distribution given a set of parameters plus the difference between the target upper confidence interval and the 97.5th quantile of the gamma distribution given the set of parameters. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is a straightforward, concise way to express this?&lt;/p&gt;&#xA;" OwnerUserId="1381" LastEditorUserId="1381" LastEditorDisplayName="" LastEditDate="2011-05-05T22:27:15.533" LastActivityDate="2011-05-05T22:27:15.533" Title="Simple notation for quantiles used in a cost function?" Tags="&lt;optimization&gt;&lt;teaching&gt;&lt;notation&gt;" AnswerCount="0" CommentCount="3" />
  <row Id="10366" PostTypeId="1" CreationDate="2011-05-05T19:00:36.900" Score="1" ViewCount="78" Body="&lt;p&gt;I am having difficulty to get $\bar u$ (to denote average of values $u_i$) as label for the x-axis in the output eps figure.&#xA;Any help would be appreciated!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks! &lt;/p&gt;&#xA;" OwnerUserId="3172" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-06T12:11:09.423" LastActivityDate="2011-05-07T18:40:09.827" Title="Using LaTeX expression in gnuplot" Tags="&lt;gnuplot&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="10367" PostTypeId="2" ParentId="10363" CreationDate="2011-05-05T19:20:09.780" Score="1" ViewCount="" Body="&lt;p&gt;All Models are wrong but some are useful : G.E.P.Box&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Y(T)=  -  4709.7&lt;br&gt;&#xA;       +  102.60*AA(T)- 17.0707*AA(T-1)&lt;br&gt;&#xA;       +  62.4994*BB(T)&#xA;       +  41.7453*CC(T)&#xA;       +  965.70*ZZ(T)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where ZZ(T)=0 FOR T=1,10&#xA;           =1 OTHERWISE&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There appears to be a &quot;lagged relationship&quot; between Y and AA AND an explained shift in the mean for observations 11-25 .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Curious results if this is not chronological or spatial data.&lt;/p&gt;&#xA;" OwnerUserId="3382" LastEditorUserId="3382" LastEditorDisplayName="" LastEditDate="2011-05-05T19:48:41.600" LastActivityDate="2011-05-05T19:48:41.600" CommentCount="6" />
  <row Id="10368" PostTypeId="2" ParentId="10363" CreationDate="2011-05-05T19:21:25.287" Score="4" ViewCount="" Body="&lt;p&gt;Your question needs refining because the function &lt;code&gt;f&lt;/code&gt; is almost certainly not uniquely defined by the sample data.  There are many different functions which could generate the same data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That being said, Analysis of Variance (ANOVA) or a &quot;sensitivity study&quot; can tell you a lot about how your inputs (AA..EE) affect your output (FF).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I just did a quick ANOVA and found a reasonably good model: &lt;code&gt;FF = 101*A + 47*B + 49*C - 4484&lt;/code&gt;.&#xA;The function does not seem to depend on DD or EE linearly.  Of course, we could go further with the model and add quadratic and mixture terms.  Eventually you will have a perfect model that over-fits the data and has no predictive value.  :)&lt;/p&gt;&#xA;" OwnerUserId="2260" LastEditorUserId="2260" LastEditorDisplayName="" LastEditDate="2011-05-05T19:32:49.257" LastActivityDate="2011-05-05T19:32:49.257" CommentCount="6" />
  <row Id="10369" PostTypeId="1" CreationDate="2011-05-05T19:28:23.433" Score="8" ViewCount="145" Body="&lt;p&gt;Suppose I have paired observations drawn i.i.d. as $X_i \sim \mathcal{N}\left(0,\sigma_x^2\right), Y_i \sim \mathcal{N}\left(0,\sigma_y^2\right),$ for $i=1,2,\ldots,n$. Let $Z_i = X_i + Y_i,$ and denote by $Z_{i_j}$ the $j$th largest observed value of $Z$. What is the (conditional) distribution of $X_{i_j}$? (or equivalently, that of $Y_{i_j}$) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;That is, what is the distribution of $X_i$ conditional on $Z_i$ being the $j$th largest of $n$ observed values of $Z$?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am guessing that as $\rho = \frac{\sigma_x}{\sigma_y} \to 0$, the distribution of $X_{i_j}$ converges to just the unconditional distribution of $X$, while as $\rho \to \infty$, the distribution of $X_{i_j}$ converges to the unconditional distribution of the $j$th order statistic of $X$. In the middle, though, I am uncertain.&lt;/p&gt;&#xA;" OwnerUserId="795" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-05-05T21:22:53.483" LastActivityDate="2011-05-11T18:22:17.520" Title="Distribution of 'unmixed' parts based on order of the mix." Tags="&lt;distributions&gt;&lt;order-statistics&gt;&lt;shrinkage&gt;" AnswerCount="2" CommentCount="5" FavoriteCount="1" />
  <row Id="10370" PostTypeId="1" CreationDate="2011-05-05T19:33:30.583" Score="4" ViewCount="89" Body="&lt;p&gt;I want to train a classifier, say SVM, or random forest, or any other classifier. One of the features in the dataset is a categorical variable with 1000 levels. What is the best way to reduce the number of levels in this variable. In R there is a function called &lt;code&gt;combine.levels()&lt;/code&gt; in the &lt;strong&gt;Hmisc&lt;/strong&gt; package, which combines infrequent levels, but I was looking for other suggestions. &lt;/p&gt;&#xA;" OwnerUserId="616" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-06T14:55:11.243" LastActivityDate="2011-05-06T14:55:11.243" Title="Reducing number of levels of unordered categorical predictor variable" Tags="&lt;classification&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="1" />
  <row Id="10371" PostTypeId="2" ParentId="10363" CreationDate="2011-05-05T19:41:52.350" Score="13" ViewCount="" Body="&lt;p&gt;To find the best fitting functional form (so called free-form or symbolic regression) for the data try this tool - to all of my knowledge this is the best one available (at least I am very excited about it)...and its free :-)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://creativemachines.cornell.edu/eureqa&quot; rel=&quot;nofollow&quot;&gt;http://creativemachines.cornell.edu/eureqa&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt;: I gave it a shot with Eureqa and I would go for:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$AA + AA^2 + BB*CC$$ with $R^2=0.99988$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would call it a perfect fit (Eureqa gives other, better fitting solutions, but these are also a little bit more complicated. Eureqa favours this one, so I chose this one) - and Eureqa did everything for me in about a few seconds on a normal laptop ;-)&lt;/p&gt;&#xA;" OwnerUserId="230" LastEditorUserId="230" LastEditorDisplayName="" LastEditDate="2011-05-09T18:58:37.057" LastActivityDate="2011-05-09T18:58:37.057" CommentCount="2" />
  <row Id="10373" PostTypeId="2" ParentId="10370" CreationDate="2011-05-05T20:54:18.127" Score="4" ViewCount="" Body="&lt;p&gt;How best to do this is going to vary &lt;em&gt;tremendously&lt;/em&gt; depending on the task you're performing, so it's impossible to say what will be best in a task-independent way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are two easy things to try if your levels are ordinal:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Bin them. E.g., 0 = (0 250), 1 = (251 500), etc. You may want to select the limits so each bin has an equal number of items.&lt;/li&gt;&#xA;&lt;li&gt;You can also take a log transform of the levels. This will squish the range down.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;If the levels are not ordinal you can cluster the levels based on other features/variables in your dataset and substitute the cluster ids for the previous levels. There are as many ways to do this as there are clustering algorithms, so the field is wide open. As I read it, this is what &lt;code&gt;combine.levels()&lt;/code&gt; is doing. You could do similarly using &lt;code&gt;kmeans()&lt;/code&gt; or &lt;code&gt;prcomp()&lt;/code&gt;. (You could/should subsequently train a classifier to predict the clusters for new datapoints.)&lt;/p&gt;&#xA;" OwnerUserId="1876" LastActivityDate="2011-05-05T20:54:18.127" />
  <row Id="10374" PostTypeId="2" ParentId="423" CreationDate="2011-05-05T21:04:31.380" Score="2" ViewCount="" Body="&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/V2yCG.png&quot; alt=&quot;enter image description here&quot;&gt;&#xA;This is data analysis in the form of a cartoon, and I find it particularly poignant.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The universe is probably littered with the one-planet graves of cultures which made the sensible economic decision that there's no good reason to go into space--each discovered, studied, and remembered by the ones who made the irrational decision.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="2817" LastActivityDate="2011-05-05T21:04:31.380" CommunityOwnedDate="2011-05-05T21:04:31.380" />
  <row Id="10375" PostTypeId="1" CreationDate="2011-05-05T22:18:43.513" Score="3" ViewCount="128" Body="&lt;p&gt;I have two datasets from genome-wide association studies. The only &#xA;information available is the odds ratio and the p-value for the first &#xA;data set. For the second data set I have the Odds Ratio, p-value and allele frequencies (AFD= disease, AFC= controls) (e.g: 0.321). I'm trying to do a meta-analysis of these data but I &#xA;don't have the effect size parameter to perform this. Is there a &#xA;possibility to calculate the SE and OR confidence intervals for each of &#xA;these data only using the info that is provided??&lt;br&gt;&#xA;Thank you in advance&lt;/p&gt;&#xA;&#xA;&lt;p&gt;example: &#xA;Data available:  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    Study     SNP ID      P        OR    Allele   AFD    AFC&#xA;    1         rs12345    0.023    0.85&#xA;    2         rs12345    0.014    0.91     C      0.32   0.25&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;With these data can I calculate the SE and CI95% OR ?&#xA;Thanks &lt;/p&gt;&#xA;" OwnerUserId="4483" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-06T13:45:19.837" LastActivityDate="2011-05-06T13:45:19.837" Title="How to calculate Standard Error of Odds Ratios?" Tags="&lt;meta-analysis&gt;&lt;genetics&gt;" AnswerCount="1" />
  <row Id="10376" PostTypeId="2" ParentId="10366" CreationDate="2011-05-05T22:26:02.227" Score="1" ViewCount="" Body="&lt;p&gt;I have not tried the following, but they might work for you*&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could try the Baltic letter &lt;code&gt;ū&lt;/code&gt; (&lt;em&gt;u with macron&lt;/em&gt;) either directly or with Unicode &lt;code&gt;U+016B&lt;/code&gt; or html &lt;code&gt;&amp;amp;#363;&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or you could follow the &lt;a href=&quot;http://www.fnal.gov/docs/products/gnuplot/tutorial/&quot; rel=&quot;nofollow&quot;&gt;advice here&lt;/a&gt;, which seems to imply that something like &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set xlabel &quot;$\bar{u}$&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;might work.&lt;/p&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-05-05T22:26:02.227" CommentCount="3" />
  <row Id="10377" PostTypeId="1" CreationDate="2011-05-05T23:27:01.870" Score="6" ViewCount="93" Body="&lt;p&gt;I am currently working towards a Statistics BSc and am frequently having to organise my research.&#xA;I am currently using a mixture of a mediawiki, and a bibliography browser plugin, to make an archive of my online data sources, and references to pdfs with useful information. but its becoming a bit of a mess.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any research tool that is orientated towards statistics?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;would-likes include&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;//TODO, task tracking so I can have a central list of outstanding tasks&lt;/li&gt;&#xA;&lt;li&gt;some cloud storage, so I can access my archive from any machine&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;some suggestions have been&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;org-mode&lt;/li&gt;&#xA;&lt;li&gt;zotero&lt;/li&gt;&#xA;&lt;li&gt;mediawiki&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;(I guess that anything I could store the files as text, I can distribute my work using SVN/git etc)&lt;/p&gt;&#xA;" OwnerUserId="4484" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-06T05:49:44.827" LastActivityDate="2011-05-06T10:38:58.583" Title="What research tool to use when researching a project?" Tags="&lt;software&gt;" AnswerCount="1" CommentCount="4" FavoriteCount="1" />
  <row Id="10378" PostTypeId="1" AcceptedAnswerId="10422" CreationDate="2011-05-05T23:46:05.043" Score="3" ViewCount="147" Body="&lt;p&gt;[I first posted this question to Stack Overflow &lt;a href=&quot;http://stackoverflow.com/questions/5866850/using-holt-winters-for-forecasting-in-python&quot;&gt;here&lt;/a&gt; but didn't get any replies, so I thought I'd try over here. Apologies if reposting isn't allowed.]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've been trying to use &lt;a href=&quot;http://adorio-research.org/wordpress/?p=1230&quot; rel=&quot;nofollow&quot;&gt;this implementation of the Holt-Winters algorithm&lt;/a&gt; for time series forecasting in Python but have run into a roadblock... basically, for some series of (positive) inputs, it sometimes forecasts negative numbers, which should clearly not be the case. Even if the forecasts are not negative, they are sometimes wildly inaccurate - orders of magnitude higher/lower than they should be. Giving the algorithm more periods of data to work with does not appear to help, and in fact often makes the forecast worse. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data I'm using has the following characteristics, which might be problems:  &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Very frequently sampled (one data point every 15 minutes, as opposed to monthly data as the example uses) - but from what I've read, the Holt-Winters algorithm shouldn't have a problem with that. Perhaps that indicates a problem with the implementation?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Has multiple periodicities - there are daily peaks (i.e. every 96 data points) as well as a weekly cycle of weekend data being significantly lower than weekday data - for example weekdays can peak around 4000 but weekends peak at 1000 - but even when I only give it weekday data, I run into the negative-number problem. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Is there something I'm missing with either the implementation or my usage of the Holt-Winters algorithm in general? I'm not a statistician so I'm using the 'default' values of alpha, beta, and gamma indicated in the link above - is that likely to be the problem? What is a better way to calculate these values?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or ... is there a better algorithm to use here than Holt-Winters? Ultimately I just want to create sensible forecasts from historical data here. I've tried single- and double-exponential smoothing but (as far as I understand) neither support periodicity in data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have also looked into using the R &lt;a href=&quot;http://robjhyndman.com/software/forecast/&quot; rel=&quot;nofollow&quot;&gt;forecast&lt;/a&gt; package instead through rpy2 - would that give me better results? I imagine I would still have to calculate the parameters and so on, so it would only be a good idea if  my current problem lies in the implementation of the algorithm...?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help/input would be greatly appreciated!&lt;/p&gt;&#xA;" OwnerUserId="4470" LastActivityDate="2011-05-06T17:02:29.550" Title="Using Holt-Winters for forecasting in Python" Tags="&lt;forecasting&gt;&lt;python&gt;" AnswerCount="2" />
  <row Id="10379" PostTypeId="2" ParentId="10378" CreationDate="2011-05-06T00:25:11.470" Score="4" ViewCount="" Body="&lt;p&gt;The problem might be that Holt-Winters is a specific model form and may not be applicable to your data. The HW Model assumes among other things the following. &#xA;a) one and only one trend &#xA;b) no level shifts in the data i.e. no intercept changes &#xA;3) that seasonal parameters do not vary over time &#xA;4) no outliers &#xA;5) no autoregressive structure or adaptive model structure &#xA;6)model errors that have constant variance &#xA;And of course &#xA;7) that the history causes the future i.e. no incorporation of price/promotions.events etc as helping variables &lt;/p&gt;&#xA;&#xA;&lt;p&gt;From your description it appears to me that a mixed-frequency approach might be needed. I have seen time series problems where the hour-of-the-day effects and the day-of-the-week effects have significant interaction terms. You are trying to force your data into an inadequate i.e. not-generalized enough structure. Estimating parameters and choosing from a small set of models does not replace Model Identification. You might want to read a piece on the different approaches to Automatic Modeling at www.autobox.com/pdfs/catchword.pdf . In terms of a more general approach I would suggest that you consider an ARMAX model otherwise known as a Transfer Function which relaxes the afore-mentioned assumptions.&lt;/p&gt;&#xA;" OwnerUserId="3382" LastActivityDate="2011-05-06T00:25:11.470" CommentCount="2" />
  <row Id="10380" PostTypeId="2" ParentId="10375" CreationDate="2011-05-06T01:03:12.977" Score="5" ViewCount="" Body="&lt;p&gt;You to calculate/approximate the standard errors via the p-values. First, convert the two-sided p-values into one-sided p-values by dividing them by 2. So you get $p = .0115$ and $p = .007$. Then convert these p-values to the corresponding z-values. For $p = .0115$, this is $z = -2.273$ and for $p = .007$, this is $z = -2.457$ (they are negative, since the odds ratios are below 1). These z-values are actually the test statistics calculated by taking the log of the odds ratios divided by the corresponding standard errors (i.e., $z = log(OR) / SE$). So, it follows that $SE = log(OR) / z$, which yields $SE = 0.071$ for the first and $SE = .038$ for the second study. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now you have everything to do a meta-analysis. I'll illustrate how you can do the computations with R, using the metafor package:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(metafor)&#xA;yi  &amp;lt;- log(c(.85, .91))     ### the log odds ratios&#xA;sei &amp;lt;- c(0.071, .038)       ### the corresponding standard errors&#xA;res &amp;lt;- rma(yi=yi, sei=sei)  ### fit a random-effects model to these data&#xA;res&#xA;&#xA;Random-Effects Model (k = 2; tau^2 estimator: REML)&#xA;&#xA;tau^2 (estimate of total amount of heterogeneity): 0 (SE = 0.0046)&#xA;tau (sqrt of the estimate of total heterogeneity): 0&#xA;I^2 (% of total variability due to heterogeneity): 0.00%&#xA;H^2 (total variability / within-study variance):   1.00&#xA;&#xA;Test for Heterogeneity: &#xA;Q(df = 1) = 0.7174, p-val = 0.3970&#xA;&#xA;Model Results:&#xA;&#xA;estimate       se     zval     pval    ci.lb    ci.ub          &#xA; -0.1095   0.0335  -3.2683   0.0011  -0.1752  -0.0438       ** &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Note that the meta-analysis is done using the log odds ratios. So, $-0.1095$ is the estimated pooled log odds ratio based on these two studies. Let's convert this back to an odds ratio:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;predict(res, transf=exp, digits=2)&#xA;&#xA; pred  se ci.lb ci.ub cr.lb cr.ub&#xA; 0.90  NA  0.84  0.96  0.84  0.96&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So, the pooled odds ratio is .90 with 95% CI: .84 to .96.&lt;/p&gt;&#xA;" OwnerUserId="1934" LastActivityDate="2011-05-06T01:03:12.977" CommentCount="2" />
  <row Id="10381" PostTypeId="1" CreationDate="2011-05-06T01:08:11.353" Score="3" ViewCount="38" Body="&lt;p&gt;I'm really confused. I never took statistics but am trying to read up as much as I can. I didn't think much about statistical analysis until &lt;em&gt;after&lt;/em&gt; doing the experiment, unfortunately, but maybe someone can help me out.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have 4 birds. Each bird must fly in 7 different wind conditions. In each wind condition, I measure things like wing stroke amplitude, flapping frequency, etc. ~40 times. I am pretty sure that those things are normally distributed. I would like to test whether the wind condition affects any of the kinematic variables I measure. The wind conditions I use are winds of different magnitudes and directions, so that 4 m/s and 3 m/s forward will probably cause greater correlation in kinematic variables then 4m/s forward wind and 4 m/s backwards wind. I hope this makes sense! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, from what I can understand, it seems as if I should do a repeated measures multivariate analysis. However, I am concerned with my sample size of 4 birds. A plot of say, individual bird's stroke amplitude with wind condition shows an obvious effect (when it must fly faster, amplitude increases), but I just don't know how/if I can show that this is statistically significant. Any input would be appreciated!&lt;/p&gt;&#xA;" OwnerUserId="4486" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-05-06T01:48:28.767" LastActivityDate="2011-05-06T01:48:28.767" Title="Is repeated measures ANOVA appropriate for my experiment and is my sample size large enough?" Tags="&lt;repeated-measures&gt;&lt;sample-size&gt;" />
  <row Id="10382" PostTypeId="1" CreationDate="2011-05-06T01:11:14.787" Score="0" ViewCount="33" Body="&lt;p&gt;Hi,&#xA;I am going to make an environmental index to be used as an explanatory variable in a regression model.  For making this index, I asked the respondents some questions about their environmental attitudes.  Each question has 5 categories: (completely agree --to-- completely disagree) and I'm going to summarize each these categories as one index. May anyone help me?&lt;/p&gt;&#xA;" OwnerUserId="4487" LastActivityDate="2011-05-08T20:36:37.933" Title="Making environmental index" Tags="&lt;classification&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="10385" PostTypeId="1" CreationDate="2011-05-06T04:04:21.720" Score="3" ViewCount="38" Body="&lt;p&gt;I am trying to fit an ordinal regression model using the &lt;code&gt;logit&lt;/code&gt; link function in R using &lt;code&gt;ordinal&lt;/code&gt; package; the response variables have five levels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The number of explanatory variables is much larger than the number of samples ($p \gg n$) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could any one help me with the following problem:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Start with a model that contains only the intercept.&lt;/li&gt;&#xA;&lt;li&gt;For the current model, explore the improvement in fit by adding additional variables.&lt;/li&gt;&#xA;&lt;li&gt;Add the baseline for the variables that performed the best (using AIC, deviance, etc.)&lt;/li&gt;&#xA;&lt;li&gt;Go back to step 2 until the maximal number of variables in the model is reached.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Unfortunately, &lt;code&gt;glmnet&lt;/code&gt;, cannot handle ordinal regression otherwise it would have been great. Is there a way of reducing the ordinal regression problem to multinomial regression using indicator variables? This would be of great benefit as I could use &lt;code&gt;glmnet&lt;/code&gt; for variable selection.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is sample data (in my case $n \sim 100$, and $p \sim 10000$):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;structure(list(resp = structure(c(1L, 1L, 2L, 2L, 2L), .Label = c(&quot;a&quot;, &#xA;&quot;b&quot;), class = c(&quot;ordered&quot;, &quot;factor&quot;)), x1 = 1:5, x2 = c(0.1, &#xA;0.2, 0.3, 0.4, 0.5), x3 = c(0.01, 0.04, 0.09, 0.16, 0.25), x4 = c(1, &#xA;4, 9, 16, 25), x5 = c(0.001, 0.002, 0.003, 0.004, 0.005), x6 = c(-5, &#xA;-4, -3, -2, -1), x7 = c(-0.5, -0.4, -0.3, -0.2, -0.1), x8 = c(0.25, &#xA;0.16, 0.09, 0.04, 0.01), x9 = c(25, 16, 9, 4, 1), x10 = c(0.0316227766016838, &#xA;0.0447213595499958, 0.0547722557505166, 0.0632455532033676, 0.0707106781186548&#xA;)), .Names = c(&quot;resp&quot;, &quot;x1&quot;, &quot;x2&quot;, &quot;x3&quot;, &quot;x4&quot;, &quot;x5&quot;, &quot;x6&quot;, &quot;x7&quot;, &#xA;&quot;x8&quot;, &quot;x9&quot;, &quot;x10&quot;), row.names = c(NA, -5L), class = &quot;data.frame&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Thanks a lot for any help or pointers!&lt;/p&gt;&#xA;" OwnerUserId="1307" LastActivityDate="2011-05-06T04:04:21.720" Title="How to add variables sequentially in ordinal package in R" Tags="&lt;r&gt;" CommentCount="2" FavoriteCount="1" />
  <row Id="10386" PostTypeId="1" CreationDate="2011-05-06T04:51:28.657" Score="4" ViewCount="55" Body="&lt;p&gt;I would like to measure the amplitude of waves in a noisy time-series on-line.  I have a time-series that models a noisy wave function, that undergoes shifts in amplitude.  Say, for example, something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed &amp;lt;- 1001&#xA;x &amp;lt;- abs(sin(seq(from = 0, t = 100, by = 0.1)))&#xA;x &amp;lt;- x + (runif(1001, 0, 1) / 5)&#xA;x &amp;lt;- x * c(rep(1.0, 500), rep(2.0, 501))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The resulting data looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; head(x, n = 30)&#xA; [1] 0.1581530 0.1329728 0.3911897 0.4104984 0.4774424 0.5118123 0.6499325&#xA; [8] 0.6837706 0.8520770 0.8625692 0.8441520 0.9960601 1.1119514 1.1414032&#xA;[15] 1.1153601 1.1456799 1.0843497 1.1141201 1.1290904 0.9906415 0.9836052&#xA;[22] 0.9369836 0.9493608 0.7484588 0.7588435 0.6467422 0.5787302 0.4665009&#xA;[29] 0.4643982 0.3398427&#xA;&amp;gt; plot(x)&#xA;&amp;gt; lines(x)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/bVVj2.png&quot; alt=&quot;Plot of waves&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you can see, the because of the noise in the series the data does not increase monotonically between the waves' troughs and crests.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;I'm looking for a way to estimate the amplitude of each wave's peak &lt;em&gt;on-line&lt;/em&gt; in a computationally untaxing way.&lt;/strong&gt;  I can probably find a way to measure the maximum magnitude of the noise term.  I'm not sure if the frequency of the waves is constant, so I'd be interested both in answers that assume a constant (known) wave frequency or a variable wave frequency.  The real data is also sinusoidal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm sure that this is a common problem with well-known solutions, but I am so new to this that I don't even know what terms to search for.  Also, apologies if this question would be more appropriate to stackoverflow, I can ask there if that is preferred.&lt;/p&gt;&#xA;" OwnerUserId="179" LastEditorUserId="179" LastEditorDisplayName="" LastEditDate="2011-05-06T06:21:06.680" LastActivityDate="2011-05-06T22:16:54.750" Title="Online method for detecting wave amplitude" Tags="&lt;time-series&gt;&lt;signal-processing&gt;&lt;online&gt;" AnswerCount="2" CommentCount="4" />
  <row Id="10387" PostTypeId="1" AcceptedAnswerId="10393" CreationDate="2011-05-06T05:36:53.120" Score="2" ViewCount="49" Body="&lt;p&gt;I'm using supervised classification algorithms from &lt;a href=&quot;https://mlpy.fbk.eu/data/doc/classification.html&quot; rel=&quot;nofollow&quot;&gt;mlpy&lt;/a&gt; to classify things into two groups for a question-answering system. I don't really know how these algorithms work, but they seem to be doing vaguely what I want.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to get some measure of confidence out of the classifiers. I can get &quot;real-valued predictions&quot; from the classifiers. These appear to be values of what I would call a link function. Here's some sample output from my system.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Predictions as to whether an answer is correct&#xA;for random data from various models&#xA;&#xA;Results for one run&#xA;----------------------------------------------&#xA;Model Result Confidence? (&quot;Real value&quot;)&#xA;----- ------ ---------------------------------&#xA; SVM  [True, 0.10396502611075412]&#xA; FDA  [True, 3.3052963597375227]&#xA;SRDA  [False, 0.34205901959526142]&#xA; PDA  [True, 3.8857018468328794]&#xA;----------------------------------------------&#xA;&#xA;Results for another run&#xA;----------------------------------------------&#xA;Model Result Confidence? (&quot;Real value&quot;)&#xA;----- ------ ---------------------------------&#xA; SVM  [False, -0.0059697528841203369]&#xA; FDA  [False, -0.15660355802446979]&#xA;SRDA  [False, 1.2465697042600801]&#xA; PDA  [True, 0.23122963338708608]&#xA;----------------------------------------------&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The real values are generally more positive for classifications as &quot;True&quot; and more negative for classifications as &quot;False&quot;, but the link is a bit more complex than that. Can I turn these real values into confidence measures? If so, how?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And I'm playing with the following classifiers.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Support Vector Machines (SVMs)&lt;/li&gt;&#xA;&lt;li&gt;K Nearest Neighbor (KNN)&lt;/li&gt;&#xA;&lt;li&gt;Fisher Discriminant Analysis (FDA)&lt;/li&gt;&#xA;&lt;li&gt;Spectral Regression Discriminant Analysis (SRDA)&lt;/li&gt;&#xA;&lt;li&gt;Penalized Discriminant Analysis (PDA)&lt;/li&gt;&#xA;&lt;li&gt;Diagonal Linear Discriminant Analysis (DLDA)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt;: Having thought about this more, realize that I just need the rank of the confidence actually. Does it seem right the answers with the highest real values be the ones that are most confidently categorized in the True group? I'd still like to understand this a bit better, but an answer to that would be nice in the short term.&lt;/p&gt;&#xA;" OwnerUserId="3874" LastEditorUserId="3874" LastEditorDisplayName="" LastEditDate="2011-05-06T08:16:05.417" LastActivityDate="2011-05-06T11:16:24.200" Title="What do &quot;real values&quot; refer to in supervised classification?" Tags="&lt;machine-learning&gt;&lt;classification&gt;&lt;svm&gt;&lt;discriminant-analysis&gt;&lt;mlpy&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="10388" PostTypeId="1" AcceptedAnswerId="10402" CreationDate="2011-05-06T06:07:07.653" Score="2" ViewCount="82" Body="&lt;p&gt;I have a data set &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; head(data)&#xA;  id centre        u      time event x c1 c2 c3 c4 c5 c6 c7 c8 c9 c10 c11 c12 c13 c14&#xA;   1      1 0.729891 0.3300478     1 0  1  0  0  0  0  0  0  0  0   0   0   0   0   0&#xA;   2      1 0.729891 7.0100000     0 0  1  0  0  0  0  0  0  0  0   0   0   0   0   0&#xA;   3      1 0.729891 7.0150000     0 1  1  0  0  0  0  0  0  0  0   0   0   0   0   0&#xA;   4      1 0.729891 1.3616940     1 0  1  0  0  0  0  0  0  0  0   0   0   0   0   0&#xA;   5      1 0.729891 7.0250000     0 0  1  0  0  0  0  0  0  0  0   0   0   0   0   0&#xA;   6      1 0.729891 5.0824055     1 0  1  0  0  0  0  0  0  0  0   0   0   0   0   0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and I want to fit a proportional hazards model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;IN R&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I first defined &lt;code&gt;formula&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; formula&#xA;Surv(time, event) ~ x + c1 + c2 + c3 + c4 + c5 + c6 + c7 + c8 + &#xA;    c9 + c10 + c11 + c12 + c13 + c14&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and I used the &lt;code&gt;coxph&lt;/code&gt; function:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; library(survival)&#xA;&amp;gt; mod &amp;lt;- coxph(formula, data=data)&#xA;Erreur dans fitter(X, Y, strats, offset, init, control, weights = weights,  : &#xA;  NA/NaN/Inf dans un appel à une fonction externe (argument 6)&#xA;De plus : Message d'avis :&#xA;In fitter(X, Y, strats, offset, init, control, weights = weights,  :&#xA;  Ran out of iterations and did not converge&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;IN SAS&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I used &lt;code&gt;proc phreg&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;proc phreg data=data;&#xA;    model time*event(0) = x c1-c14 / ties=breslow;&#xA;run;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Partial output:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;                      Convergence Status&#xA;&#xA;                     Convergence criterion (GCONV=1E-8) satisfied.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h3&gt;Question:&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Do you have any suggestion to make it work in R? &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I have tried to increase &lt;code&gt;iter.max&lt;/code&gt; but the problem is still there... If needed, I can provide the data (600 rows).&lt;/p&gt;&#xA;" OwnerUserId="3019" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-06T06:25:18.100" LastActivityDate="2011-05-06T13:01:48.383" Title="How to get convergence using coxph (R) given that model converges using proc phreg (SAS)" Tags="&lt;r&gt;&lt;sas&gt;&lt;cox-model&gt;&lt;coxph&gt;&lt;phreg&gt;" AnswerCount="2" CommentCount="4" />
  <row Id="10389" PostTypeId="1" CreationDate="2011-05-06T06:11:52.817" Score="2" ViewCount="76" Body="&lt;p&gt;According to normal probability distribution theory which says that for $n$ independent,&#xA;identically distributed, standard, normal, random variables $\xi_j$ the expected absolute maximum is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$E(\max|\xi_j|)=\sqrt{2 \ln n}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regarding this, why do we need to multiple the above-mentioned estimate by $\sigma$ (Standard Deviation) in order to derive the expected absolute maximum for a normal, random variable with zero mean?&lt;/p&gt;&#xA;" OwnerUserId="4286" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-06T12:18:03.990" LastActivityDate="2011-05-06T13:47:26.630" Title="Calculating the distribution of maximal value of $n$ draws from a normal distribution" Tags="&lt;normal-distribution&gt;&lt;extreme-value&gt;" AnswerCount="2" CommentCount="7" />
  <row Id="10390" PostTypeId="2" ParentId="10389" CreationDate="2011-05-06T06:22:13.960" Score="1" ViewCount="" Body="&lt;p&gt;Intuitively: values of the standard normal distribution (including the absolute maximum) 'tend to be' 1 SD = 1 away from 0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In non-standard zero-mean normal, the data 'tend to be' 1 SD = sigma away from 0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could say that as long as you're doing linear stuff, all distances to zero will blow up by a factor sigma.&lt;/p&gt;&#xA;" OwnerUserId="4257" LastActivityDate="2011-05-06T06:22:13.960" CommentCount="1" />
  <row Id="10391" PostTypeId="1" CreationDate="2011-05-06T06:27:29.207" Score="0" ViewCount="45" Body="&lt;p&gt;I am trying to impute a variable &lt;code&gt;x&lt;/code&gt; in my dataset. I want to do it through SAS Enterprise Miner. I am using impute node on it but finally my imputed dataset is not getting created. Is there any body who has used SAS Enterprise Miner and has done imputation? Please tell me step by step how to do it successfully. I am missing something in between which is not letting me impute it.&lt;/p&gt;&#xA;" OwnerUserId="1763" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-06T09:56:36.000" LastActivityDate="2011-05-06T09:56:36.000" Title="SAS Enterprise Miner imputation" Tags="&lt;sas&gt;&lt;data-imputation&gt;" />
  <row Id="10392" PostTypeId="2" ParentId="10389" CreationDate="2011-05-06T07:27:01.087" Score="4" ViewCount="" Body="&lt;p&gt;If $\zeta_j = \sigma \xi_j $ for some $\sigma &amp;gt;0$ and some $\mu$ then &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$E[\max|\zeta_j|] =  E[\max|\sigma \xi_j |] = E[\sigma \max| \xi_j|]= \sigma E[ \max| \xi_j|]$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and this tells us how to move from a standard normal with mean $0$ and standard deviation $1$ to a normal distribution with mean $0$ and standard deviation $\sigma$.  &lt;/p&gt;&#xA;" OwnerUserId="2958" LastEditorUserId="-1" LastEditorDisplayName="" LastEditDate="2011-05-06T13:47:26.630" LastActivityDate="2011-05-06T13:47:26.630" CommentCount="3" />
  <row Id="10393" PostTypeId="2" ParentId="10387" CreationDate="2011-05-06T07:53:39.133" Score="1" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Regarding the &quot;&lt;em&gt;Real Values&lt;/em&gt;&quot;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &quot;&lt;em&gt;Real Values&lt;/em&gt;&quot; are better called &quot;&lt;em&gt;confidences&lt;/em&gt;&quot; or (from my pov the most common term) &quot;&lt;em&gt;scores&lt;/em&gt;&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Such scores are often normalized so that they sum up to 1 for all classes. They represent a measure how, well, confident the model is that the presented example belongs to a certain class. They are highly dependent on the general strategy and the properties of the algorithm. For example in KNN the score for a class i is calculated by averaging the distance to those examples which both belong to the k-nearest-neighbors and have class i. Then the score is sum-normalized across all classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Regarding your question&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suppose with &quot;converting into confidences&quot; you actually mean &quot;probability estimates&quot;. E.g. if an example has probability 0.3 for class &quot;1&quot;, then 30% of all examples with similar values should belong to class &quot;1&quot; and 70% should not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as I know, his task is called &quot;calibration&quot;. For this purpose some general methods exist (e.g. binning the scores and mapping them to the class-fraction of the corresponding bin) and some classifier-dependent (like e.g. &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.41.1639&quot; rel=&quot;nofollow&quot;&gt;Platt Scaling&lt;/a&gt; which has been invented for SVMs). A good point to start is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.13.7457&quot; rel=&quot;nofollow&quot;&gt;Bianca Zadrozny, Charles Elkan: Transforming Classifier Scores into Accurate Multiclass Probability Estimates&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT after Question-Edit:&lt;/strong&gt;&#xA;@Thomas wrote: &lt;em&gt;Does it seem right the answers with the highest real values be the ones that are most confidently categorized in the True group?&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Yes, in general this is correct (with the same argument as above). I suggest to create a ROC - plot to see if this also applies to mlpy - package. I suggest &lt;a href=&quot;http://rocr.bioinf.mpi-sb.mpg.de/&quot; rel=&quot;nofollow&quot;&gt;ROCR&lt;/a&gt; for this purpose. &lt;/p&gt;&#xA;" OwnerUserId="264" LastEditorUserId="264" LastEditorDisplayName="" LastEditDate="2011-05-06T11:16:24.200" LastActivityDate="2011-05-06T11:16:24.200" CommentCount="0" />
  <row Id="10394" PostTypeId="1" CreationDate="2011-05-06T08:33:03.437" Score="2" ViewCount="68" Body="&lt;p&gt;In case of robust estimators, What does &lt;strong&gt;Gaussian efficiency&lt;/strong&gt; means? For example $Q_{_n}$ has 82% Gaussian efficiency and 50% breakdown point.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reference is: Rousseeuw P.J., and Croux, C. (1993). “Alternatives to median absolute deviation.” J. American Statistical Assoc., 88, 1273-1283&lt;/p&gt;&#xA;" OwnerUserId="4286" LastEditorUserId="4286" LastEditorDisplayName="" LastEditDate="2011-05-16T09:38:37.290" LastActivityDate="2011-05-17T01:53:56.790" Title="What does Gaussian efficiency mean?" Tags="&lt;normal-distribution&gt;&lt;scales&gt;&lt;robust&gt;" AnswerCount="1" CommentCount="4" />
  <row Id="10395" PostTypeId="2" ParentId="10386" CreationDate="2011-05-06T09:07:19.593" Score="5" ViewCount="" Body="&lt;p&gt;More than a complete solution this is meant to be a very &lt;em&gt;rough&lt;/em&gt; series of &quot;hints&quot; on how to implement one using FFT, there are probably better methods but, if it works...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First of all let's generate the wave, with varying frequency and amplitude&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  freqs &amp;lt;- c(0.2, 0.05, 0.1)&#xA;  x &amp;lt;- NULL&#xA;  y &amp;lt;- NULL&#xA;  for (n in 1:length(freqs))&#xA;      {&#xA;      tmpx &amp;lt;- seq(n*100, (n+1)*100, 0.1)&#xA;      x &amp;lt;- c(x, tmpx)&#xA;      y &amp;lt;- c(y, sin(freqs[n] * 2*pi*tmpx))&#xA;      }&#xA;&#xA;  y &amp;lt;- y * c(rep(1:5, each=(length(x)/5)))&#xA;  y &amp;lt;- y + rnorm(length(x), 0, 0.2)&#xA;  plot(x, y, &quot;l&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Which gives us this&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/IvjjX.png&quot; alt=&quot;Test wave&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, if we calculate the FFT of the wave using &lt;code&gt;fft&lt;/code&gt; and then plot it (I used the &lt;code&gt;plotFFT&lt;/code&gt; function I posted &lt;a href=&quot;http://stackoverflow.com/questions/3485456/useful-little-functions-in-r/3486696#3486696&quot;&gt;here&lt;/a&gt;) we get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/mAFjd.png&quot; alt=&quot;Global FFT of the wave&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that I overplotted the 3 frequencies (0.05, 0.1 and 0.2) with which I generated the data. As the data is sinusoidal the FFT does a very good job in retrieving them. Note that this works best when the y-values are 0 centered.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now let's do a sliding FFT with a window of 50 we get&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/Syxin.png&quot; alt=&quot;Windowed FFT&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As expected, at the beginning we only get the 0.2 frequency (first 2 plot, so between 0 and 100), as we go on we get the 0.05 frequency (100-200) and finally the 0.1 frequency comes about (200-300).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The power of the FFT function is proportional to the amplitude of the wave. In fact, if we write down the maximum in each window we get:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1 Max frequency:  0.2  - power:  254&#xA;2 Max frequency:  0.2  - power:  452&#xA;3 Max frequency:  0.04  - power:  478&#xA;4 Max frequency:  0.04  - power:  606&#xA;5 Max frequency:  0.1  - power:  1053&#xA;6 Max frequency:  0.1  - power:  1253 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;===&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This can be also achieved using a STFT (short-time Fourier transform), which is basically the same thing I showed you before, but with overlapping windows. &#xA;This is implemented, for instance, by the &lt;code&gt;evolfft&lt;/code&gt; function of the &lt;code&gt;RSEIS&lt;/code&gt; package.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It would give you:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;stft &amp;lt;- evolfft(y, dt=0.1, Nfft=2048, Ns=100, Nov=90, fl=0, fh=0.5)&#xA;plotevol(stft) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/wdyic.png&quot; alt=&quot;STFT&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This, however, may be trickier to analyze, especially online.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;hope this helps somehow&lt;/p&gt;&#xA;" OwnerUserId="582" LastActivityDate="2011-05-06T09:07:19.593" CommentCount="5" />
  <row Id="10396" PostTypeId="1" AcceptedAnswerId="10399" CreationDate="2011-05-06T09:30:12.217" Score="4" ViewCount="98" Body="&lt;p&gt;I want to do a regression where my dependent variable has four categories &lt;code&gt;(1,2,3,4)&lt;/code&gt; which represents the number of dependents. Can I do this with logistic regression? I read somewhere that &lt;code&gt;link=glogit&lt;/code&gt; option is useful in this, can somebody please shed some light? I am new to this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Writing the syntax here would be very useful for me.&lt;/p&gt;&#xA;" OwnerUserId="1763" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-06T09:51:59.813" LastActivityDate="2011-05-06T13:23:06.260" Title="Regression for dependent variable with 4 categories" Tags="&lt;logistic-regression&gt;&lt;sas&gt;" AnswerCount="4" CommentCount="8" />
  <row Id="10397" PostTypeId="2" ParentId="10356" CreationDate="2011-05-06T10:11:27.033" Score="2" ViewCount="" Body="&lt;p&gt;I disagreed with the other answers in the comments, so it's only fair I give my own. Let $Y$ be the response (good/bad accounts), and $X$ be the covariates.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For logistic regression, the model is the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\log\left(\frac{p(Y=1|X=x)}{p(Y=0|X=x)}\right)= \alpha + \sum_{i=1}^k x_i \beta_i $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Think about how the data might be collected:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;You could select the observations randomly from some hypothetical &quot;population&quot;&lt;/li&gt;&#xA;&lt;li&gt;You could select the data based on $X$, and see what values of $Y$ occur.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Both of these are okay for the above model, as you are only modelling the distribution of $Y|X$. These would be called a &lt;em&gt;prospective study&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;You could select the observations based on $Y$ (say 100 of each), and see the relative prevalence of $X$ (i.e. you are stratifying on $Y$). This is called a &lt;em&gt;retrospective&lt;/em&gt; or &lt;em&gt;case-control study&lt;/em&gt;.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;(You could also select the data based on $Y$ and certain variables of $X$: this would be a stratified case-control study, and is much more complicated to work with, so I won't go into it here).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a nice result from epidemiology (see &lt;a href=&quot;http://biomet.oxfordjournals.org/content/66/3/403.short&quot; rel=&quot;nofollow&quot;&gt;Prentice and Pyke (1979)&lt;/a&gt;) that for a case-control study, the maximum likelihood estimates for $\beta$ can be found by logistic regression, that is using the prospective model for retrospective data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So how is this relevant to your problem?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Well, it means that if you are able to collect more data, you could just look at the bad accounts and still use logistic regression to estimate the $\beta_i$'s (but you would need to adjust the $\alpha$ to account for the over-representation). Say it cost $1 for each extra account, then this might be more cost effective then simply looking at all accounts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But on the other hand, if you already have ALL possible data, there is no point to stratifying: you would simply be throwing away data (giving worse estimates), and then be left with the problem of trying to estimate $\alpha$.&lt;/p&gt;&#xA;" OwnerUserId="495" LastActivityDate="2011-05-06T10:11:27.033" CommentCount="7" />
  <row Id="10398" PostTypeId="2" ParentId="10377" CreationDate="2011-05-06T10:38:58.583" Score="1" ViewCount="" Body="&lt;p&gt;Steffen's comment and link to the question above is very useful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would say that it depends. I am going to assume that you are using LaTeX, at least (for mathematical typesetting), and possibly R (cos its free, and great).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In that case, especially given that you have mentioned org-mode, I would suggest using Emacs to organise your statistical analysis. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The advantages are as follows:&#xA;unify your statistical analysis (Emacs Speaks Statistics), your paper writing (LaTeX) - Emacs sweave support is also the best I have found.&#xA;You also have Ebib to manage references, and this is very nice. &#xA;Emacs is so customizable that anything you need to do, can be done from within it. However, it has quite a steep learning curve, and you may need to unlearn some shortcuts and ways of dealing with programs that are commonplace elsewhere.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Emacs also has integrated version control. I cannot speak to their quality as I have not used it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On reference management, I personally didn't like Zotero, and use JabRef. JabRef is nice for the GUI and its simplicity, and could support you while you learn enough about Emacs to be productive within it. It also has a cite while you write plugin for open office, if you like that kind of thing. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;HTH.&lt;/p&gt;&#xA;" OwnerUserId="656" LastActivityDate="2011-05-06T10:38:58.583" CommentCount="3" />
  <row Id="10399" PostTypeId="2" ParentId="10396" CreationDate="2011-05-06T11:12:23.657" Score="1" ViewCount="" Body="&lt;p&gt;You'll want to look up the literature on multinomial logistic regression, a.k.a. nominal regression.  It's an expanded version of the usual logistic regression.  Coefficients and odds ratios obtained deal with the likelihood of the outcome being A, B, or C &lt;em&gt;as opposed to D&lt;/em&gt;, the reference category.  Thus for 4 levels of a dependent variable you'll have 3 tables with coefficients, odds ratios, etc.  You'll want to make sure you choose that reference category intentionally:  to which result will you and your readers most want to make comparisons?  Sorry I can't provide syntax but SAS documentation should be a help there.&lt;/p&gt;&#xA;" OwnerUserId="2669" LastActivityDate="2011-05-06T11:12:23.657" CommentCount="3" />
  <row Id="10400" PostTypeId="2" ParentId="10396" CreationDate="2011-05-06T11:25:06.253" Score="0" ViewCount="" Body="&lt;p&gt;I'd suggest to you to consider using decision trees (such as CART) for such problems.&#xA;I see that SAS Enterprise Miner has some functions for decision trees:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://sas-x.com/2011/01/decision-trees-in-sas-enterprise-miner-and-spss-clementine/&quot; rel=&quot;nofollow&quot;&gt;http://sas-x.com/2011/01/decision-trees-in-sas-enterprise-miner-and-spss-clementine/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="253" LastActivityDate="2011-05-06T11:25:06.253" CommentCount="1" />
  <row Id="10401" PostTypeId="2" ParentId="10396" CreationDate="2011-05-06T11:44:23.293" Score="2" ViewCount="" Body="&lt;p&gt;I think this document: &lt;a href=&quot;http://www.biostat.umn.edu/~melanie/PH7402/2009/NOTE/logistic.pdf&quot; rel=&quot;nofollow&quot;&gt;logistic&lt;/a&gt; holds all the information you need (with pointers to how you can do it in both R and SAS). It explains the concept of the proportional odds model and indicates that glogit is indeed the way to go.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you need more, just google for &quot;SAS proportional odds logistic regression&quot;...&lt;/p&gt;&#xA;" OwnerUserId="4257" LastActivityDate="2011-05-06T11:44:23.293" />
  <row Id="10402" PostTypeId="2" ParentId="10388" CreationDate="2011-05-06T12:08:22.810" Score="2" ViewCount="" Body="&lt;p&gt;What about changing of the starting values? Supply to &lt;code&gt;init&lt;/code&gt; similar values to the output of SAS and see what happens. This in general is not a good strategy to get the convergence, but it helps to see whether the problem is in algorithm, or just in starting values.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-05-06T12:08:22.810" />
  <row Id="10403" PostTypeId="2" ParentId="423" CreationDate="2011-05-06T12:13:40.790" Score="1" ViewCount="" Body="&lt;p&gt;Another one from &lt;a href=&quot;http://xkcd.org/892/&quot; rel=&quot;nofollow&quot;&gt;xkcd&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/hN4lW.png&quot; alt=&quot;Hell, my eighth grade science class managed to conclusively reject it just based on a classroom experiment. It's pretty sad to hear about million-dollar research teams who can't even manage that.&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="565" LastActivityDate="2011-05-06T12:13:40.790" CommunityOwnedDate="2011-05-06T12:13:40.790" />
  <row Id="10404" PostTypeId="2" ParentId="6630" CreationDate="2011-05-06T12:32:01.043" Score="1" ViewCount="" Body="&lt;p&gt;I believe that you are trying to use statistical methods that are appropriate for independent observations while you have correlated data, both temporarily and spatially. If you have observations say for 5 hours and decide to re-state this as 241 observations taken every minute, you really don't have 240 degrees of freedom in respect to the mean of these 241 values. Autocorrelation potentially yields an overstatement of the size of &quot;N&quot; and thusly creates false uncertainty statements. What you need to do is to find someone/some textbook/some web site/.... to teach you about time series data and it's analysis. One way to start is to GOOGLE &quot;help me understand time series&quot; and start to read/learn. There is a lot of material available on the web. One available  trove of time series information is something I helped create at &lt;a href=&quot;http://www.autobox.com/AFSUniversity/afsuFrameset.htm&quot; rel=&quot;nofollow&quot;&gt;http://www.autobox.com/AFSUniversity/afsuFrameset.htm&lt;/a&gt; . I mention this as I am still associated with this firm and it's products thus my comments are &quot;biased and opinionated&quot; but not solely self-serving.&lt;/p&gt;&#xA;" OwnerUserId="3382" LastActivityDate="2011-05-06T12:32:01.043" />
  <row Id="10405" PostTypeId="2" ParentId="10388" CreationDate="2011-05-06T13:01:48.383" Score="1" ViewCount="" Body="&lt;p&gt;How many events are in the dataset?  The fit from SAS may not be meaningful if there are fewer than a few multiples of the number of covariates (14).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general with Cox regression I have not had to specify starting values but have occasionally played around with the convergence criteria.&lt;/p&gt;&#xA;" OwnerUserId="4253" LastActivityDate="2011-05-06T13:01:48.383" />
  <row Id="10406" PostTypeId="2" ParentId="10096" CreationDate="2011-05-06T13:02:33.687" Score="1" ViewCount="" Body="&lt;p&gt;In my opinion , you might need to add/detect day-of-the-week ; week-of-the-year ; Holiday effects ( lead ,contemporaneous and lag effects ); possible level shifts and or Local Time Trends ; Possible fixed-days-of-the-month; Pulses/Outlier correction via Intervention Detection schemes ; AND then INTRODUCE a set of possible predictors (K-1) reflecting EACH of the K algorithmic change effects. Additionally you might need to incorporate an ARMA component to your predictors to render your error process Gaussian. Care should also be taken to ensure that the parameters of your final model did not significantly change over time and that the residuals from your final model have constant variance. Heterogenous error variance can be caused by structural changes in variance at particular time points , coupling of the error dispersion and the level of the output series and/or a pure stochastic variance change over time.&lt;/p&gt;&#xA;" OwnerUserId="3382" LastActivityDate="2011-05-06T13:02:33.687" />
  <row Id="10407" PostTypeId="1" AcceptedAnswerId="10830" CreationDate="2011-05-06T13:16:53.577" Score="6" ViewCount="220" Body="&lt;p&gt;Repeating an experiment with $n$ possible outcomes $t$ times independently, where all but one outcomes have probability $\frac{1}{n+1}$ and the other outcome has the double probability $\frac{2}{n+1}$, is there a good approximate formula for the probability that the outcome with the higher probability happens more often than any other one?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For me, $n$ is typically some hundreds, and $t$ is chosen depending on $n$ such that the probability that the most likely outcome occurs most often is between 10% and 99.999%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the moment I use a small program that calculates a crude approximation by assuming that the counts for how often each outcome shows up in $t$ trials are independent and approximate the counts using the Poisson distribution. How can I improve on this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT:&lt;/strong&gt; I'd strongly appreciate comments/votes on the two (maybe soon more) answers given.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT 2:&lt;/strong&gt; As none of the two answers is convincing me, but as I don't want to let the 100 points bounty to vanish (and as nobody voted for/against one of the two answers), I'll just pick one of the answers. I'd still appreciate other answers.&lt;/p&gt;&#xA;" OwnerUserId="565" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-05-15T19:28:58.837" LastActivityDate="2011-05-15T19:28:58.837" Title="Probability for finding a double-as-likely event" Tags="&lt;probability&gt;&lt;approximation&gt;" AnswerCount="3" CommentCount="22" FavoriteCount="1" />
  <row Id="10408" PostTypeId="2" ParentId="1459" CreationDate="2011-05-06T13:17:11.287" Score="1" ViewCount="" Body="&lt;p&gt;Originally the idea of examining pre-whitened cross-correlations was suggested by Box and Jenkins. In 1981, Liu and Hanssens published( L.-M. Liu and D.M. Hanssens (1982). &quot;Identification of Multiple-Input Transfer Function Models.&quot; Communications in Statistics A 11: 297-314.) a paper that suggested a common filter approach that would effectively deal with multiple inputs whose pre-whitened series exhibit cross-correlative structure. They even created a 2 input model data set to demonstrate their solution.&#xA;After we programmed that approach and then compared it to the Box-Jenkins pre-whitening approach iteratively implemented by us we decided to not to use either the Pankratz approach or the Liu-Hanssens approach.We would be glad to share the Liu-Hansens test data with you if you wish me to post it to the list.&lt;/p&gt;&#xA;" OwnerUserId="3382" LastActivityDate="2011-05-06T13:17:11.287" />
  <row Id="10409" PostTypeId="2" ParentId="10396" CreationDate="2011-05-06T13:23:06.260" Score="1" ViewCount="" Body="&lt;p&gt;If you need imputation (as your comment suggests), look into &lt;code&gt;PROC MI&lt;/code&gt;. It is specifically designed for this purpose. One of the many options it has is imputation of an ordinal outcome.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, the following code will use proportional odds regression to impute &lt;code&gt;ndependents&lt;/code&gt; based on &lt;code&gt;x1&lt;/code&gt;, &lt;code&gt;x2&lt;/code&gt; and their interaction.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;proc mi;&#xA;   class ndependents;&#xA;   var x1 x2;&#xA;   monotone logistic(ndependents= x1 x2 x1*x2);&#xA;run;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can do only one imputation, but multiple imputations are preferrable. After the analysis you can use &lt;code&gt;PROC MIANALYZE&lt;/code&gt; to combine the results.&lt;/p&gt;&#xA;" OwnerUserId="279" LastActivityDate="2011-05-06T13:23:06.260" CommentCount="2" />
  <row Id="10411" PostTypeId="1" AcceptedAnswerId="10413" CreationDate="2011-05-06T13:34:49.283" Score="2" ViewCount="123" Body="&lt;p&gt;I have a few datasets of &quot;interactions&quot; between pairs of elements like so:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;element1 element2 1&#xA;element2 element3 1&#xA;element4 element5 1&#xA;...&#xA;element505535 element4 2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where the value in the 3rd column is the &quot;strength&quot; of interaction. Almost all of these strengths are &quot;1.&quot; A strength of 1 means that this interaction was observed one time. A strength of 2 means 2x, etc. I have actually gone one step further and normalized all of my datasets by the total # of interactions observed in the dataset so that datasets interaction values can be compared.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are 5-6 million interactions listed in each file and each dataset is obviously under-sampled since there are ~500k elements (making a square matrix of ~250 trillion positions).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to cluster these datasets so that I can make statements about which types of elements tend to cluster with which other types elements. Obviously, robusticity of clustering will be a factor—but this is partially ameliorated by the fact that I will make biological replicates of the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have tried a few different &quot;naive&quot; clustering approaches just to see what I could do easily with the data. I fully realize that these are problematic ways of clustering, either because they are not robust or because they rely on the data being very undersampled, but here is what I've done:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Clustering elements together as long as there is at least one interaction between each element in the cluster and at least one other element in the cluster. When I do this, all the elements end up in a single cluster. This was important to do because it tells me that there are no pairs of elements that are totally isolated from the rest of the group.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Finding &quot;superclusters&quot;—that is, clusters where every member of the cluster interacts with every other member of the cluster (e.g. a triangle for a cluster of 3 and a box with an X in the middle for a cluster of 4, etc). This yields almost exclusively clusters with 2 and 3 elements after about 10% of the data has been analyzed (this is still running).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I would love to be able to do some sort of hierarchical clustering using my &quot;interaction strength&quot; values as the distance measure between each pair of elements (unobserved interactions have a strength of 0). Does anyone know of a way to do HC on this sort of large, sparse data—or know of a clustering method that might be more appropriate? I've used R up until now.&lt;/p&gt;&#xA;" OwnerUserId="3561" LastEditorUserId="3561" LastEditorDisplayName="" LastEditDate="2011-05-06T15:42:24.923" LastActivityDate="2011-05-06T15:42:24.923" Title="Clustering large and sparse datasets" Tags="&lt;r&gt;&lt;clustering&gt;&lt;bioinformatics&gt;" AnswerCount="1" CommentCount="0" />
  <row Id="10412" PostTypeId="2" ParentId="10150" CreationDate="2011-05-06T14:43:19.223" Score="2" ViewCount="" Body="&lt;p&gt;You might try using ARIMA models making sure that you incorporate any identifiable Level Shifts and/or Local Time Trends culminating in an ARMAX model. Changes in parameters/variance of the errors should also be tested and remedied if necessary. As compared to NN, these approaches challenge the data rather than simply believing/fitting the data.&lt;/p&gt;&#xA;" OwnerUserId="3382" LastEditorUserId="3382" LastEditorDisplayName="" LastEditDate="2011-05-07T11:45:48.963" LastActivityDate="2011-05-07T11:45:48.963" />
  <row Id="10413" PostTypeId="2" ParentId="10411" CreationDate="2011-05-06T15:01:22.680" Score="3" ViewCount="" Body="&lt;p&gt;Does this happen to be protein interaction data? Regardless, there are many algorithms you can try. I am the author of &lt;a href=&quot;http://micans.org/mcl/&quot; rel=&quot;nofollow&quot;&gt;mcl&lt;/a&gt;, a clustering algorithm fairly often used in bioinformatics. You could easily try it, after installing, by given it the data in exaxctly the format you describe; the command line would be &quot;mcl yourdata --abc&quot; (the abc option tells mcl to expect this particular format). You need not and cannot specify the number of clusters, as cluster structure arises from the process computed by mcl in an emergent fashion. Protein interaction data can be quite voluminous and noisy -- and other data sources as well of course. Especially the presence of nodes of very high-degree (say with hundreds to thousands of neighbours) may obscure cluster structure. In such a scenario, in the absence of weights, it can be really worthwhile to try to reduce the input graph by e.g. a k-NN transform taking into account the two-step graph incidence relation. mcl also provides means to do this, as well a tool that lists various graph traits under a series of ever more stringent transformations, so that it is possible to choose such a transformation in an informed manner. As for hierarchical clustering, it is possible to achieve clusterings at different levels of granularity by varying mcl's so-called inflation parameter. You could for example supply three values (e.g. 1.4, 2, and 5) to obtain clustering at different levels of granularity. It is also possible to arrange clusterings thus obtained into a strictly hierarchical data structure.&lt;/p&gt;&#xA;" OwnerUserId="4495" LastActivityDate="2011-05-06T15:01:22.680" CommentCount="2" />
  <row Id="10414" PostTypeId="2" ParentId="9385" CreationDate="2011-05-06T15:14:38.527" Score="0" ViewCount="" Body="&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/NjDSh.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://i.imgur.com/NjDSh.jpg&quot; rel=&quot;nofollow&quot;&gt;1&lt;/a&gt;: &lt;a href=&quot;http://i.imgur.com/kxU4t.jpg&quot; rel=&quot;nofollow&quot;&gt;http://i.imgur.com/kxU4t.jpg&lt;/a&gt; reflects a questioning of the highly unusual Oct 2009 value 130 Oct-09 2301.41 . Time series analysis actually challenges the data rather than fitting a presumed set of models. The residuals from the following model &lt;img src=&quot;http://i.imgur.com/NjDSh.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://i.imgur.com/NjDSh.jpg&quot; rel=&quot;nofollow&quot;&gt;1&lt;/a&gt;: &lt;a href=&quot;http://i.imgur.com/Q4W5h.jpg&quot; rel=&quot;nofollow&quot;&gt;http://i.imgur.com/Q4W5h.jpg&lt;/a&gt; more closely exhibit the required Gaussian Structure for T tests to be valid. &lt;img src=&quot;http://i.imgur.com/NjDSh.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I apologize in advance for the unnecessary repetition of the forecast graph. I will have to go to wiki school to learn how to include images in my posts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://i.imgur.com/NjDSh.jpg&quot; rel=&quot;nofollow&quot;&gt;1&lt;/a&gt;: &lt;a href=&quot;http://i.imgur.com/OUc5a.jpg&quot; rel=&quot;nofollow&quot;&gt;http://i.imgur.com/OUc5a.jpg&lt;/a&gt; . The forecasts for the next 24 months are then robust to identified anomalies &lt;img src=&quot;http://i.imgur.com/NjDSh.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="3382" LastActivityDate="2011-05-06T15:14:38.527" />
  <row Id="10415" PostTypeId="2" ParentId="9429" CreationDate="2011-05-06T15:27:14.023" Score="3" ViewCount="" Body="&lt;p&gt;I'll imagine a concrete example, with more context, to make things easy. Assume you measure the score on test of 3k students of 200 schools and you measured each student at 4 time points (say, at each quarter). You have a covariate at student level that doesn't vary by time (like sex), that you called pred1.obs and a covariate by school that vary by time (say the number of meetings between teachers and parents until that moment in time). If this example resembles your study, than I think you have to set up a three level model (individual level, group level and time level for the groups):&#xA;i = 1 ... 3000 individuals&#xA;t = 1... 4 periods&#xA;g = 1... 200 groups&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The model would be:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;y_i ~ N(a + b_[groups_g] + b.ind*pred.obs1_i, sigma^2) # 1st level&#xA;b_g = N(gamma + gamma_[time] + gamma.g[time_t]*pred2.grp, sigma.b^2) # 2nd level&#xA;gamma.g_t = N(0, sigma.gamma^2) # 3rd level&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Note that you would have the slope at the second level (group level) varying by time, which makes sense, since you expect that the effect of schools on the perfomance of students may vary by time, depending of the valu of the covariate at the level of schools.&#xA;I'm not that sure how to estimate this with lmer (I know how to estimate a Bayesian model using WinBugs or Jags, calling them by R). In any case, here is my suggestion.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In lme4, I'd try:&#xA;First, expand  pred2.grp (the covariate at group level that vary by time) to the individual level, then you would have repetead measures by individuals at the group and time level. Then:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;lmer(outcome ~ pred1.obs + pred2.grp + (1|group))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="3058" LastActivityDate="2011-05-06T15:27:14.023" />
  <row Id="10416" PostTypeId="2" ParentId="4089" CreationDate="2011-05-06T15:35:24.853" Score="2" ViewCount="" Body="&lt;p&gt;I highly recommend the function &lt;a href=&quot;http://braverock.com/brian/R/PerformanceAnalytics/html/chart.Correlation.html&quot; rel=&quot;nofollow&quot;&gt;chart.Correlations&lt;/a&gt; in the package &lt;a href=&quot;http://cran.r-project.org/web/packages/PerformanceAnalytics/index.html&quot; rel=&quot;nofollow&quot;&gt;PerformanceAnalytics&lt;/a&gt;.  It packs an amazing amount of information into a single chart: kernel-density plots and histograms for each variable, and scatterplots, lowess smoothers, and correlations for each variable pair. It's one of my favorite graphical data summary functions:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(PerformanceAnalytics)&#xA;chart.Correlation(iris[,1:4],col=iris$Species)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/pcFhX.png&quot; alt=&quot;I love this chart!&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="2817" LastEditorUserId="2817" LastEditorDisplayName="" LastEditDate="2011-05-06T15:49:38.020" LastActivityDate="2011-05-06T15:49:38.020" />
  <row Id="10417" PostTypeId="2" ParentId="10356" CreationDate="2011-05-06T16:16:21.933" Score="0" ViewCount="" Body="&lt;p&gt;There are many ways in which you can think of logistic regressions. My favorite way is to think that your response variable, y_i, follows a Bernoulli distribution with probability p_i. &#xA;An p_i, in turn, is a function of some predictors. More formally:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;y_i ~ Bernoulli (p_i)&#xA;p_i = logistic^-1(a + b1x1 + ... bnxn), where logistic^-1 = 1/(1+exp(-x)).&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now does it matter it you have low proportion of failures (bad accounts)? Not really, as long as your sample data is balanced, as some people already pointed. However, if your data is not balanced, then getting more data may be almost useless if there is some selection effects you are not taking into account. In this case, you should use matching, but the lack of balance may turn matching pretty useless. Another strategy is trying to find a natural experiment, so you can use instrumental variable or regression disconinuity design.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Last, but not least, if you have a balanced sample or there is no selection bias, you may be worried with the fact the bad account is rare. I don't think 5% is rare, but just in case, take a look at the paper by Gary King &lt;a href=&quot;http://gking.harvard.edu/files/abs/0s-abs.shtml&quot; rel=&quot;nofollow&quot;&gt;http://gking.harvard.edu/files/abs/0s-abs.shtml&lt;/a&gt; about running a rare event logistic. In the Zelig package,in R, you can run a rare event logistic.&lt;/p&gt;&#xA;" OwnerUserId="3058" LastActivityDate="2011-05-06T16:16:21.933" />
  <row Id="10418" PostTypeId="1" CreationDate="2011-05-06T16:28:31.677" Score="4" ViewCount="70" Body="&lt;p&gt;Why the exchangeability of random variables is essential for the hierarchical Bayesian modeling?&lt;/p&gt;&#xA;" OwnerUserId="3125" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-06T20:13:58.907" LastActivityDate="2011-05-18T09:16:25.957" Title="Why the exchangeability of random variables is essential in hierarchical bayesian models?" Tags="&lt;bayesian&gt;&lt;multilevel-analysis&gt;" AnswerCount="3" />
  <row Id="10419" PostTypeId="1" AcceptedAnswerId="10442" CreationDate="2011-05-06T16:32:49.483" Score="1" ViewCount="146" Body="&lt;p&gt;I've got a question concerning a negative binomial regression: Suppose that you have the following commands: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(MASS)&#xA;attach(cars)&#xA;mod.NB&amp;lt;-glm.nb(dist~speed)&#xA;summary(mod.NB)&#xA;detach(cars)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(Note that cars is a dataset which is available in R, and I don't really care if this model makes sense.) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I'd like to know is: How can I interpret the variable &lt;code&gt;theta&lt;/code&gt; (as returned at the bottom of a call to &lt;code&gt;summary&lt;/code&gt;). Is this the shape parameter of the negbin distribution and is it possible to interpret it as a measure of skewness? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I appreciate every thought!&lt;/p&gt;&#xA;" OwnerUserId="4496" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-06T20:29:05.660" LastActivityDate="2011-05-06T20:29:05.660" Title="What is theta in a negative binomial regression fitted with R?" Tags="&lt;regression&gt;&lt;glm&gt;&lt;negative-binomial&gt;" AnswerCount="1" />
  <row Id="10420" PostTypeId="1" CreationDate="2011-05-06T16:37:04.897" Score="2" ViewCount="163" Body="&lt;p&gt;I have a large simulated loss data (from catastrophic models developed at my school) to calculate some extreme quantiles. previously they used non-parametric methods to do this (find the point estimate for these extreme quantiles and CIs). I am using parametric models (extreme value theory ,fat tail distributions etc) to do it. I have been thinking about the pros and cons for these two methods. I appreciate all your help I would appreciate if someone can provide some summaries of parametric and non-parametric models, their advantages and disadvantages.  &lt;/p&gt;&#xA;" OwnerUserId="4497" LastEditorUserId="4497" LastEditorDisplayName="" LastEditDate="2011-05-12T17:14:15.733" LastActivityDate="2011-05-12T17:14:15.733" Title="Advantages and disadvantages of parametric and non-parametric models" Tags="&lt;nonparametric&gt;&lt;parametric&gt;" AnswerCount="2" CommentCount="7" FavoriteCount="2" />
  <row Id="10421" PostTypeId="1" CreationDate="2011-05-06T16:46:51.740" Score="2" ViewCount="15" Body="&lt;p&gt;Recently I've been working EM algorithms for MAP estimation in a problem where the expectation is intractable, but the maximization is easy. Further, draws from the distribution in the E-step are easily available through MCMC, so I've been experimenting with stochastic versions of EM. Let $X$ be the observed data, $Z$ be the missing data, and $\Theta$ be the parameters to be estimated. I'll use $\Theta_t$ for the current estimate of $\Theta$. Specifically I've looked at stochastic approximation EM and Monte Carlo EM, which estimate the E step like so:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) MCEM: The $Q$ function is approximated with an average of $m_t$ MC draws $\hat Q(\Theta; \Theta_t) = \frac{1}{m_t}\sum_{i=1}^{m_t}\log(p(X,Z_i|\Theta)$ where $Z_i$ is drawn from $p(Z|X,\Theta_t)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) SAEM: $\hat Q(\Theta; \Theta_t) = \gamma_t \hat Q(\Theta; \Theta_{t_1}) + (1-\gamma_t)\frac{1}{m_t}\sum_{i=1}^{m_t}\log(p(X,Z_i|\Theta)$. In the context of exponential families, this amounts to using a weighted average of the &quot;new&quot; and &quot;old&quot; sufficient statistics at each step, compared to MCEM which takes a completely new average (ie $\gamma_t=0$)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Both have the same M step, with $\Theta_{t+1}$ chosen to maximize the approximate $Q$ function. My question is when does it make sense to average over the sequence of $\Theta_t$'s? Presumably this would further reduce the Monte Carlo error, and my understanding is that the convergence is (theoretically) unaffected since the $\Theta_t$'s are converging to a stationary point with appropriate conditions on $m_t$ and $\gamma_t$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I'm looking for is more practical advice, since averaging too early will bias the (finite sample) estimates and averaging too late is wasteful. Also the related question of assessing convergence; if we consider convergence criteria like $|\Theta_t - \Theta_{t-1}| \leq \epsilon$ should the difference be taken instead between the two &lt;em&gt;averaged&lt;/em&gt; estimates? (Not the best criterion, I know, but super easy :) )&lt;/p&gt;&#xA;" OwnerUserId="26" LastActivityDate="2011-05-06T16:46:51.740" Title="Averaged estimators in stochastic versions of EM" Tags="&lt;monte-carlo&gt;&lt;em-algorithm&gt;" FavoriteCount="1" />
  <row Id="10422" PostTypeId="2" ParentId="10378" CreationDate="2011-05-06T17:02:29.550" Score="1" ViewCount="" Body="&lt;p&gt;I think the R forecast package you mentioned is a better fit for this problem than just using Holt-Winters.  The two functions you are interested in are &lt;a href=&quot;http://www.oga-lab.net/RGM2/func.php?rd_id=forecast%3aets&quot; rel=&quot;nofollow&quot;&gt;ets()&lt;/a&gt; and &lt;a href=&quot;http://www.oga-lab.net/RGM2/func.php?rd_id=forecast%3aauto.arima&quot; rel=&quot;nofollow&quot;&gt;auto.arima()&lt;/a&gt;.  ets() will fit an exponential smoothing model, including Holt-Winters and several other methods.  It will choose parameters (alpha, beta, and gama) for a variety of models and then return the one with the lowest AIC (or BIC if you prefer).  auto.arima() works similarly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, as IrishStat pointed out, these kinds of models may not be appropriate for your analysis.  In that case, try calculating some covariates, such as dummy variables for weekends, holidays, and their interactions.  Once you've specified covariates that make sense, use auto.arima() to find a ARMAX model, and then &lt;a href=&quot;http://www.oga-lab.net/RGM2/func.php?rd_id=forecast%3aforecast&quot; rel=&quot;nofollow&quot;&gt;forecast()&lt;/a&gt; to make predictions.  You will probably end up with something much better than a simple Holt-Winters model in python with default parameters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should also note that both ets() and auto.arima can fit seasonal models, but you need to format your data as a seasonal time series. Let me know if you need any help with that.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can read more about the forecast package &lt;a href=&quot;http://www.jstatsoft.org/v27/i03&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2817" LastActivityDate="2011-05-06T17:02:29.550" CommentCount="5" />
  <row Id="10423" PostTypeId="1" CreationDate="2011-05-06T17:12:19.433" Score="4" ViewCount="68" Body="&lt;p&gt;Are there any papers/books/ideas about the relationship between the number of features and the number of observations one needs to have to train a &quot;robust&quot; classifier?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, assume I have 1000 features and 10 observations from two classes as a training set, and 10 other observations as a testing set. I train some classifier X and it gives me 90% sensitivity and 90% specificity on the testing set. Let's say I am happy with this accuracy and based on that I can say it is a good classifier. On the other hand, I've approximated a function of 1000 variables using 10 points only, which may seem to be not very... robust?&lt;/p&gt;&#xA;" OwnerUserId="4337" LastActivityDate="2011-05-08T18:51:15.643" Title="Number of features vs. number of observations" Tags="&lt;machine-learning&gt;" AnswerCount="4" FavoriteCount="0" />
  <row Id="10424" PostTypeId="1" CreationDate="2011-05-06T17:13:49.510" Score="4" ViewCount="45" Body="&lt;p&gt;I'm a journalist turned developer who hobbies in APIs and analysis of web traffic. I've always enjoyed learning about stats but as I learned, I learned that I have misapplied some basic concepts in the past. I now know a bit better, and know to doublecheck my ideas with those are smarter than me -- I'm hoping to find a great answer to whether my idea is going to work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A while back I was using traffic data to do basic summary statistics and standard deviation as a basic measure of variability. Where I went wrong was that I tried to create a confidence interval -- wrong because I presumed my dataset was Gaussian when in fact I now believe it to resemble something more like a Power Law. So I could sample the data and create standard deviations of the data all day but and they wouldn't be good models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Recently I've been thinking about this problem. Given a limited set of data, how can I try to model the traffic for a website in question? I have built an API on top of a wonderful dataset, and that dataset is used in all kinds of pages around an actively used site. My idea: use the API access pattern as a small sample of my larger data set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're following, what I'd do is basically treat each datarequest as a sample, maybe grouping requests by hour and logging total, standard deviation, number of samples n, etc. Then take use standard deviation of those samples to create the model, under the assumption that the Central Limit Theorem says that distribution will be normal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As I understand it, even though my dataset is not normal the statistics I derive from that dataset should be. Is that the case? If so, can I create a confidence interval from that data?&lt;/p&gt;&#xA;" OwnerUserId="3198" LastEditorUserId="3198" LastEditorDisplayName="" LastEditDate="2011-05-06T17:22:16.637" LastActivityDate="2011-05-06T18:10:09.203" Title="Does the Central Limit Theorem allow one to create confidence intervals from a web traffic dataset?" Tags="&lt;confidence-interval&gt;&lt;central-limit-theorem&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="10425" PostTypeId="1" CreationDate="2011-05-06T17:17:52.510" Score="3" ViewCount="51" Body="&lt;p&gt;I use the &lt;a href=&quot;http://www.oga-lab.net/RGM2/func.php?rd_id=forecast%3aauto.arima&quot; rel=&quot;nofollow&quot;&gt;auto.arima()&lt;/a&gt; function in the &lt;a href=&quot;http://cran.r-project.org/web/packages/forecast/index.html&quot; rel=&quot;nofollow&quot;&gt;forecast&lt;/a&gt; package to fit ARMAX models with a variety of covariates. However, I often have a large number of variables to select from and usually end up with a final model that works with a subset of them.  I don't like ad-hoc techniques for variable selection because I am human and subject to bias, but &lt;a href=&quot;http://stats.stackexchange.com/questions/8807/cross-validating-time-series-analysis&quot;&gt;cross-validating time series is hard&lt;/a&gt;, so I haven't found a good way to automatically try different subsets of my available variables, and am stuck tuning my models using my own best judgement.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I fit glm models, I can use the elastic net or the lasso for regularization and variable selection, via the &lt;a href=&quot;http://cran.r-project.org/web/packages/glmnet/index.html&quot; rel=&quot;nofollow&quot;&gt;glmnet&lt;/a&gt; package. Is there a existing toolkit in R for using the elastic net on ARMAX models, or am I going to have to roll my own? Is this even a good idea?&lt;/p&gt;&#xA;" OwnerUserId="2817" LastActivityDate="2011-05-07T15:26:53.670" Title="Fitting an ARMAX model with regularization and variables selection" Tags="&lt;r&gt;&lt;time-series&gt;&lt;lasso&gt;&lt;regularization&gt;&lt;elastic-net&gt;" AnswerCount="1" CommentCount="0" FavoriteCount="1" />
  <row Id="10426" PostTypeId="2" ParentId="10423" CreationDate="2011-05-06T17:38:58.600" Score="2" ViewCount="" Body="&lt;p&gt;What you've hit on here is &lt;a href=&quot;http://en.wikipedia.org/wiki/Curse_of_dimensionality&quot; rel=&quot;nofollow&quot;&gt;the curse of dimensionality&lt;/a&gt; or the p&gt;&gt;n problem (where p is predictors and n is observations).  There have been many techniques developed over the years to solve this problem. You can use &lt;a href=&quot;http://en.wikipedia.org/wiki/Akaike_information_criterion&quot; rel=&quot;nofollow&quot;&gt;AIC&lt;/a&gt; or &lt;a href=&quot;http://en.wikipedia.org/wiki/Bayesian_information_criterion&quot; rel=&quot;nofollow&quot;&gt;BIC&lt;/a&gt; to penalize models with more predictors.  You can choose random sets of variables and asses their importance using &lt;a href=&quot;http://en.wikipedia.org/wiki/Cross-validation_%28statistics%29&quot; rel=&quot;nofollow&quot;&gt;cross-validation&lt;/a&gt;.  You can use &lt;a href=&quot;http://en.wikipedia.org/wiki/Tikhonov_regularization&quot; rel=&quot;nofollow&quot;&gt;ridge-regression&lt;/a&gt;, &lt;a href=&quot;http://en.wikipedia.org/wiki/Lasso_%28statistics%29#LASSO_method&quot; rel=&quot;nofollow&quot;&gt;the lasso&lt;/a&gt;, or the &lt;a href=&quot;http://www.stanford.edu/~hastie/TALKS/enet_talk.pdf&quot; rel=&quot;nofollow&quot;&gt;elastic net&lt;/a&gt; for &lt;a href=&quot;http://en.wikipedia.org/wiki/Regularization_%28mathematics%29&quot; rel=&quot;nofollow&quot;&gt;regularization&lt;/a&gt;.  Or you can choose a technique, such as a &lt;a href=&quot;http://en.wikipedia.org/wiki/Support_vector_machine&quot; rel=&quot;nofollow&quot;&gt;support vector machine&lt;/a&gt; or &lt;a href=&quot;http://en.wikipedia.org/wiki/Random_forest&quot; rel=&quot;nofollow&quot;&gt;random forest&lt;/a&gt; that deals well with a large number of predictors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Honestly, the solution depends on the specific nature of the problem you are trying to solve.&lt;/p&gt;&#xA;" OwnerUserId="2817" LastActivityDate="2011-05-06T17:38:58.600" />
  <row Id="10427" PostTypeId="1" CreationDate="2011-05-06T17:40:57.943" Score="2" ViewCount="203" Body="&lt;p&gt;I am using a fixed effect model for my panel data (9 years, 1000+ obs), since my Hausman test indicates a value $&amp;lt;0.05$. When I add dummy variables for industries that my firms included, they always get omitted. I know there is a big difference when it comes to the DV (disclosure index) among the different industry groups. But I am not able to get them in my model when using Stata.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestions how to solve this? and why are they omitted?&lt;/p&gt;&#xA;" OwnerUserId="4394" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-07T08:51:52.560" LastActivityDate="2011-05-07T09:05:21.043" Title="How to deal with omitted dummy variables in a fixed effect model?" Tags="&lt;stata&gt;&lt;panel-data&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="10428" PostTypeId="2" ParentId="10423" CreationDate="2011-05-06T17:49:09.447" Score="2" ViewCount="" Body="&lt;p&gt;I suspect that no such rules of thumb will be generally applicable.  Consider a problem with two gaussian classes centered on $\vec{+1}$ and $\vec{-1}$, both with covariance matrix of $0.000001*\vec{I}$.  In that case, you only need two samples, one from either class to get perfect classification, almost regardless of the number of features.  At the other end of the spectrum if both classes are centered on the origin with covariance $\vec{I}$, no amount of training data is going to give you a useful classifier. At the end of the day, the amount of samples you need for a given number of features depends on how the data are distributed, in general, the more features you have, the more data you will need to adequately describe the distribution of the data (exponential in the number of features if you are unlucky - see the curse of dimensionality mentioned by Zach).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you use regularisation, then in principal, (an upper bound on) the generalisation error is independent of the number of features (see Vapnik's work on the support vector machine).  However that leaves the problem of finding a good value for the regularisation parameter (cross-validation is handy).&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2011-05-06T17:49:09.447" />
  <row Id="10429" PostTypeId="1" CreationDate="2011-05-06T17:53:42.990" Score="1" ViewCount="107" Body="&lt;p&gt;I'm wondering how to fit multivariate linear mixed model and finding multivariate BLUP in R. I'd appreciate if someone come up with example and R code. Thanks&lt;/p&gt;&#xA;" OwnerUserId="3903" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-06T22:14:59.773" LastActivityDate="2011-05-06T22:14:59.773" Title="Fitting multivariate linear mixed model in R" Tags="&lt;r&gt;&lt;mixed-model&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="10430" PostTypeId="2" ParentId="10424" CreationDate="2011-05-06T18:10:09.203" Score="1" ViewCount="" Body="&lt;p&gt;There are a few variants on the Central limit theorem, however an important point is whether or not you have a independent and identically distributed sample. Referrer engines and SEO seems to always be changing the sample, even though your theoretical target population may remain the same. You cannot make this assumption when sampling from a web site blindly. There is some flexibility to the CLT, but at some point it will break.  You might be better off studying your population first, and hopefully you will be able to explaining who they are or where they come from.&lt;/p&gt;&#xA;" OwnerUserId="3489" LastActivityDate="2011-05-06T18:10:09.203" />
  <row Id="10431" PostTypeId="2" ParentId="10429" CreationDate="2011-05-06T18:24:23.917" Score="2" ViewCount="" Body="&lt;p&gt;Try the R package &lt;code&gt;nlme&lt;/code&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can find some examples, theory and further documentation in: &#xA;&lt;a href=&quot;http://cran.r-project.org/doc/contrib/Fox-Companion/appendix-mixed-models.pdf&quot; rel=&quot;nofollow&quot;&gt;http://cran.r-project.org/doc/contrib/Fox-Companion/appendix-mixed-models.pdf&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;code&gt;nlme&lt;/code&gt; package is able to calculate pooled estimates [or the so called BLUP= best linear unbiased predictor]. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you've downloaded the package, type in R console: &lt;code&gt;help(predict.lme)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For more information, look at page 17 in Fox's paper. There you can find an example on how to pool information across subjects. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps :)&lt;/p&gt;&#xA;" OwnerUserId="2902" LastActivityDate="2011-05-06T18:24:23.917" CommentCount="1" />
  <row Id="10432" PostTypeId="1" AcceptedAnswerId="10580" CreationDate="2011-05-06T19:10:28.690" Score="7" ViewCount="101" Body="&lt;p&gt;First, let me say that I am a bit out of my depth here, so if this question needs to be re-phrased or closed as a duplicate, please let me know. It may simply be that I don't have the proper vocabulary to express my question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am working on an image processing task in which I identify features in an image, and then classify them based on their properties, including shape, size, darkness, etc. I'm quite experienced with the image processing portion of this, but  think I could improve the methods I use for classification of the features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Right now, I set thresholds for each of the parameters measured, and then classify features according to some simple logic based on which thresholds the feature has crossed. For example (the actual properties and groupings are more complex, but I'm trying to simplify irrelevant portions of my project for this question), lets say I'm grouping features into the groups &quot;Big and Dark,&quot; &quot;Big and Light&quot; and &quot;Small&quot;. Then a feature $A$ will be in &quot;Big and Dark&quot; iff (size($A$)&gt;sizeThreshold) &amp;amp; (darkness($A$)&gt;darknessThreshold).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The goal is for the classification to agree with the classification done by an expert-level human, so I can set the thresholds to produce the best match between the groupings made by human and computer on some test set, and then hope that the classification works well with new data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is already working pretty well, but I see one particular failure mode which I think may be fixable. Let's say feature $A$ is known to belong to &quot;Big and Dark.&quot; The human classified it in this way because, while is was just barely big enough, it was very very dark, which made up somewhat for the lack of &quot;bigness.&quot; My algorithm would fail to classify this feature properly, because classification is based on rigid binary logic, and requires all thresholds to be crossed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to improve this failure by making my algorithm better mimic the human guided process, in which a deficiency in one parameter can be compensated by an abundance of another. To do this, I would like to take each of the base properties of my features, and convert them into some sort of score which would be a predictor of the group to which the feature belongs. I have thought of many ways of doing this, but they are mostly ad hoc ideas, based on my background in vector calculus and physics. For example, I've considered treating each feature as a vector in the N-D space of feature properties, and calculating the projection of each feature along certain vectors, each of which would measure the degree to which a feature belongs in the group.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am sure there is a more rigorous and better established technique for doing this sort of thing, but my background is relatively weak in statistical analysis, so I'm looking for a shove in the right direction. Even the name of a technique, or a link to a textbook would be helpful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;TL;DR:&lt;/strong&gt;&#xA;What techniques are useful in classifying objects based on a large number of descriptive parameters?&lt;/p&gt;&#xA;" OwnerUserId="2426" LastActivityDate="2011-05-09T22:58:25.560" Title="Categorization/Segmentation techniques" Tags="&lt;classification&gt;" AnswerCount="1" CommentCount="6" />
  <row Id="10433" PostTypeId="1" AcceptedAnswerId="10440" CreationDate="2011-05-06T19:32:08.850" Score="3" ViewCount="65" Body="&lt;p&gt;I hope you can&#xA;help me with a question regarding calculating mean age from grouped census data. If the&#xA;age categories used were   [0–4], [5–9], [10–14], and [15–19] years,&#xA;how would you calculate the midpoints? I initially assumed the midpoints would be 2, 7,&#xA;and so on. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I read in a worked example that the midpoint should be 2.5 when age range is 0 to 4. I am assuming this has something to do with the babies not actually being&#xA;zero years, but I am not exactly sure why the midpoint would be 2.5. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone assist?  Many thanks&lt;/p&gt;&#xA;" OwnerUserId="4498" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-06T20:02:14.477" LastActivityDate="2011-05-06T20:13:02.327" Title="Calculating mean age from grouped census data" Tags="&lt;mean&gt;" AnswerCount="2" />
  <row Id="10434" PostTypeId="1" CreationDate="2011-05-05T21:24:40.070" Score="1" ViewCount="44" Body="&lt;p&gt;I am not that familiar with statistics and was wondering about the best way to quantify the difference between a set of data and a function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My scenario is that I have a set with maximum heart rate data from around 60 people, with their age. I'm trying to see which of all &quot;mathematical models&quot; out there that describe the relation between maximum heart rate and age, is the least wrong. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm much better at understanding the biology behind this, not the mathematics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you for any assistance and pointers! &lt;/p&gt;&#xA;" LastActivityDate="2011-05-06T19:44:25.300" Title="What is the best way to measure goodness of fit between data and functions?" Tags="&lt;regression&gt;" CommentCount="5" />
  <row Id="10435" PostTypeId="2" ParentId="7535" CreationDate="2011-05-06T19:48:18.143" Score="1" ViewCount="" Body="&lt;p&gt;absolutely agreed to what Matt said, first you have to think about the background of the data... It doesn't make any sense to fit ZI models, when there are no Zero generating triggers in the population! The advantage of NB models are that they can display unobserved heterogenity in a gamma distributed random variable. Techniqually: The main reasons for overdispersion are unobs Heterogenity and Zero inflation. I do not believe that your fit is bad. Btw to get the goodness of fit you should always compare the Deviance with the degrees of freedom of your model. If the Deviance D is higher than n-(p+1) (this is df) than you should search a better model. Although there are mostly no better models than ZINB to get rid of overdispersion.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;if you want to fit a ZINB with R, get the package &lt;code&gt;pscl&lt;/code&gt; and try to use the command &lt;code&gt;zeroinfl(&amp;lt;model&amp;gt;, dist=negative)&lt;/code&gt;. For further information see &lt;code&gt;?zeroinfl&lt;/code&gt; after loading the required package!&lt;/p&gt;&#xA;" OwnerUserId="4496" LastActivityDate="2011-05-06T19:48:18.143" />
  <row Id="10436" PostTypeId="2" ParentId="10433" CreationDate="2011-05-06T19:53:46.040" Score="3" ViewCount="" Body="&lt;p&gt;The 0-4 years group refers to the following age interval: $0 \leq x &amp;lt; 5$, i.e. a child which is 4 years and 364 days old still belongs to this group. So, let's compute the midpoint for that range:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; ((365+365+365+365+364)/2)/365&#xA;[1] 2.49863  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="307" LastActivityDate="2011-05-06T19:53:46.040" CommentCount="0" />
  <row Id="10437" PostTypeId="2" ParentId="10418" CreationDate="2011-05-06T20:00:20.750" Score="0" ViewCount="" Body="&lt;p&gt;It isn't! I'm no expert here, but i'll give my two cents.&#xA;In general when you have a hierarchical model, say &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$y|\Theta_{1} \sim \text{N}(X\Theta_{1},\sigma^2)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\Theta_{1}|\Theta_{2} \sim\text{N}(W\Theta_{2},\sigma^2)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We make conditional independence assumptions, i.e., conditional on $\Theta_{2}$, the $\Theta_{1}$ are exchangeable. If the second level is not exchangeable, than you can incluce another level that makes it exchangeable. But even in the case that you can't make an assumption of exchaganbelity, the model may still be a good fit to your data at the first level. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Last, but not least, exchangeability is important only if you wanna think in terms of De Finetti's representation theorem. You might just think that priors are regularization tools that help you to fit your model. In this case, the exchangeability assumption is as good as it is your model fit to the data. In other words, if you think of Bayesian hierarchical model as way to get  abetter fit to your data, then exchangeability is not essential in any sense. &lt;/p&gt;&#xA;" OwnerUserId="3058" LastEditorUserId="3058" LastEditorDisplayName="" LastEditDate="2011-05-06T20:16:49.073" LastActivityDate="2011-05-06T20:16:49.073" CommentCount="2" />
  <row Id="10438" PostTypeId="2" ParentId="10418" CreationDate="2011-05-06T20:04:43.487" Score="1" ViewCount="" Body="&lt;p&gt;&quot;Essential&quot; is too vague. But surpressing the technicalities, if the sequence $X=\{X_i\}$ is exchangeable then the $X_i$ are conditionally independent given some unobserved parameter(s) $\Theta$ with a probability distribution $\pi$. That is, $p(X) = \int p(X_i|\Theta)d\pi(\Theta)$. $\Theta$ needn't be univariate or even finite dimensional and may be further represented as a mixture, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Exchangability is essential in the sense that these conditional independence relationships allow us to fit models we almost certainly couldn't otherwise.&lt;/p&gt;&#xA;" OwnerUserId="26" LastActivityDate="2011-05-06T20:04:43.487" />
  <row Id="10439" PostTypeId="1" AcceptedAnswerId="10453" CreationDate="2011-05-06T20:10:53.717" Score="5" ViewCount="92" Body="&lt;p&gt;Suppose I have a quadratic regression model &#xA;$$&#xD;&#xA;Y = \beta_0 + \beta_1 X + \beta_2 X^2 + \epsilon&#xD;&#xA;$$&#xA;with the errors $\epsilon$ satisfying the usual assumptions (independent, normal, independent of the $X$ values). Let $b_0, b_1, b_2$ be the least squares estimates.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have two new $X$ values $x_1$ and $x_2$, and I'm interested in getting a confidence interval for $v = E(Y|X = x_2) - E(Y|X=x_1) = \beta_1 (x_2 - x_1) + \beta_2 (x_2^2 - x_1^2)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The point estimate is $\hat{v} = b_1 (x_2 - x_1) + b_2 (x_2^2 - x_1^2)$, and (correct me if I'm wrong) I can estimate the variance by $$\hat{s}^2 = (x_2 - x_1)^2 \text{Var}(b_1) + (x_2^2 - x_1^2)^2 \text{Var}(b_2) + 2 (x_2 - x_1)(x^2 - x_1^2)\text{Cov}(b_1, b_2)$$ using the variance and covariance estimates of the coefficients provided by the software.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I could use a normal approximation and take $\hat{v} \pm 1.96 \hat{s}$ as a 95% confidence interval for $v$, or I could use a bootstrap confidence interval, but is there a way to work out the exact distribution and use that?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you.&lt;/p&gt;&#xA;" OwnerUserId="3835" LastActivityDate="2011-05-07T18:07:15.263" Title="Confidence interval for difference of means in regression" Tags="&lt;regression&gt;&lt;confidence-interval&gt;" AnswerCount="1" CommentCount="5" />
  <row Id="10440" PostTypeId="2" ParentId="10433" CreationDate="2011-05-06T20:13:02.327" Score="4" ViewCount="" Body="&lt;p&gt;As @Bernd has pointed out, 2.5 really is the midpoint of the 0 to 4 year age group, etc.  However, using midpoints at either end of the population distribution introduces bias.  For instance, the midpoint of the 80 - 90 year group is approximately 83, because most people in this group are nearer 80 than 90.  If this nicety matters (and it perhaps it does, if you are agonizing over a half-year difference), read on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Demographers make their estimates using various methods of monotonic interpolation.  A classic method is &lt;a href=&quot;http://books.google.com/books?id=SuXrAAAAMAAJ&amp;amp;pg=PA688&amp;amp;lpg=PA688&amp;amp;dq=sprague%27s%20formula&amp;amp;source=bl&amp;amp;ots=REEU1bw97W&amp;amp;sig=myjV4EznXpojTh5qnfOPel7rQ9Q&amp;amp;hl=en&amp;amp;ei=a1bETfK-Hsjc0QHJw8i1CA&amp;amp;sa=X&amp;amp;oi=book_result&amp;amp;ct=result&amp;amp;resnum=3&amp;amp;ved=0CCMQ6AEwAg#v=onepage&amp;amp;q=sprague%27s%20formula&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;Sprague's Formula&lt;/a&gt;.  This is well described in their literature; for an overview see Hubert Vaughan, &lt;em&gt;Symmetry in Central Polynomial Interpolation&lt;/em&gt;, &lt;a href=&quot;http://www.actuaries.org.uk/research-and-resources/documents/symmetry-central-polynomial-interpolation&quot; rel=&quot;nofollow&quot;&gt;JIA 80, 1954&lt;/a&gt;.  This method as published requires equally-spaced age groups but it can be adapted to variable spacings.  @Rob Hyndman was the co-author of a nice paper on monotonic splines (Smith, Hyndman, &amp;amp; Wood, &lt;em&gt;Spline Interpolation for Demographic Variables: The Monotonicity Problem,&lt;/em&gt; &lt;a href=&quot;http://www.google.com/url?sa=t&amp;amp;source=web&amp;amp;cd=7&amp;amp;ved=0CEMQFjAG&amp;amp;url=http://citeseer.ist.psu.edu/viewdoc/download;jsessionid=DBDD57BFA879A27200EB2AC370EABEDA?doi=10.1.1.154.8979&amp;amp;rep=rep1&amp;amp;type=pdf&amp;amp;rct=j&amp;amp;q=Smith,%20Hyndman,%20&amp;amp;%20Wood,%20%2aSpline%20Interpolation%20for%20Demographic%20Variables%3a%20The%20Monotonicity%20Problem,&amp;amp;ei=0FXETd63FYXd0QG9r7mKCA&amp;amp;usg=AFQjCNEE7eAGilKXJe3lCmt-BzZQKvLniA&amp;amp;sig2=87e74oR_76nJulTI3hLJbw&amp;amp;cad=rja&quot; rel=&quot;nofollow&quot;&gt;J. Pop. Res. 21 #1, 2004&lt;/a&gt;).  The paper mentions R code for the &quot;Hyman filter.&quot;  It is still available on &lt;a href=&quot;http://robjhyndman.com/Rfiles/interpcode.R&quot; rel=&quot;nofollow&quot;&gt;Rob's Web site&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you have an interpolated age distribution you can compute moments (and any other properties) according to the standard definitions.  For instance, the mean is estimated by numerically integrating the age with respect to the distribution.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-05-06T20:13:02.327" CommentCount="3" />
  <row Id="10441" PostTypeId="1" AcceptedAnswerId="10445" CreationDate="2011-05-06T20:27:01.463" Score="3" ViewCount="84" Body="&lt;p&gt;I'm running an experiment where I'm gathering (independent) samples in parallel, I compute the variance of each group of samples and now I want to combine then all to find the total variance of all the samples. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm having a hard time finding a derivation for this as I'm not sure of terminology. I think of it as a partition of one RV. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I want to find $Var(X)$ from $Var(X_1)$, $Var(X_2)$, ..., and $Var(X_n)$, where $X$ = $[X_1, X_2, \dots, X_n]$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: The partitions are not the same size/cardinality, but the sum of the partition sizes equal the number of samples in the overall sample set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT 2: There is a formula for a &lt;a href=&quot;http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Parallel_algorithm&quot; rel=&quot;nofollow&quot;&gt;parallel computation here&lt;/a&gt;, but it only covers the case of a partition into two sets, not $n$ sets. &lt;/p&gt;&#xA;" OwnerUserId="4499" LastEditorUserId="4499" LastEditorDisplayName="" LastEditDate="2011-05-09T12:50:54.850" LastActivityDate="2011-05-10T16:34:37.680" Title="How to calculate the variance of a partition of variables" Tags="&lt;variance&gt;" AnswerCount="1" CommentCount="4" />
  <row Id="10442" PostTypeId="2" ParentId="10419" CreationDate="2011-05-06T20:28:31.947" Score="0" ViewCount="" Body="&lt;p&gt;Yes, &lt;code&gt;theta&lt;/code&gt; is the shape parameter of the negative binomial distribution, and no, you cannot really interpret it as a measure of skewness. More precisely:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;skewness will depend on the value of &lt;code&gt;theta&lt;/code&gt;, but also on the mean&lt;/li&gt;&#xA;&lt;li&gt;there is no value of &lt;code&gt;theta&lt;/code&gt; that will guarantee you lack of skew&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If I did not mess it up, in the &lt;code&gt;mu&lt;/code&gt;/&lt;code&gt;theta&lt;/code&gt; parametrization used in negative binomial regression, the skewness is $$Skew(NB) = \frac{\theta+2\mu}{\sqrt{\theta\mu(\theta+\mu)}} = \frac{1 + 2\frac{\mu}{\theta}}{\sqrt{\mu(1+\frac{\mu}{\theta})}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this context, $\theta$ is usually interpreted as a measure of overdispersion with respect to the Poisson distribution. The variance of the negative binomial is $\mu + \mu^2/\theta$, so $\theta$ really controls the excess variability compared to Poisson (which would be $\mu$), and not the skew.&lt;/p&gt;&#xA;" OwnerUserId="279" LastActivityDate="2011-05-06T20:28:31.947" CommentCount="2" />
  <row Id="10444" PostTypeId="1" CreationDate="2011-05-06T20:46:27.680" Score="6" ViewCount="154" Body="&lt;p&gt;In general, I standardize my independent variables in regressions, in order to properly compare the coefficients (this way they have the same units: standard deviations). However, with panel/longitudinal data, I'm not sure how I should standardize my data, especially if I estimate a hierarchical model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To see why it can be a potential problem, assume you have $i = 1, \ldots, n$ individuals measured along $t=1,\ldots, T$ periods and you measured a dependent variable, $y_{i,t}$ and one independent variable $x_{i,t}$. If you run a complete pooling regression, then it's ok to standardize your data in this way: $x.z = (x- \text{mean}(x))/\text{sd}(x)$, since it will not change t-statistic. On the other hand, if you fit an unpooled regression, i.e., one regression for each individual, then you should standardize your data by individual only, not the whole dataset (in R code): &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for (i in 1:n) {&#xA;  for ( t in 1:T) x.z[i] =  (x[i,t] - mean(x[i,]))/sd(x[i,]) &#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, if you fit a simple hierarchical model with a varying intercept by individuals, then you are using a shrinkage estimator, i.e, you are estimating a model between pooled and unpooled regression. How should I standardize my data? Using the whole data like a pooled regression? Using only individuals, like in the unpooled case?&lt;/p&gt;&#xA;" OwnerUserId="3058" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-05-06T23:05:44.520" LastActivityDate="2011-05-07T13:23:49.177" Title="Is it good practice to standardize your data in a regression with panel/longitudinal data?" Tags="&lt;r&gt;&lt;regression&gt;&lt;standarization&gt;" AnswerCount="1" />
  <row Id="10445" PostTypeId="2" ParentId="10441" CreationDate="2011-05-06T20:50:07.443" Score="4" ViewCount="" Body="&lt;p&gt;The formula is fairly straightforward if all the sub-sample have the same sample size. If you had $g$ sub-samples of size $k$ (for a total of $gk$ samples), then the variance of the combined sample depends on the mean $E_j$ and variance $V_j$ of each sub-sample: &#xA;$$ Var(X_1,\ldots,X_{gk}) = \frac{k-1}{gk-1}(\sum_{j=1}^g V_j + \frac{k(g-1)}{k-1} Var(E_j)),$$ where by $Var(E_j)$ means the variance of the sample means.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A demonstration in R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; x &amp;lt;- rnorm(100)&#xA;&amp;gt; g &amp;lt;- gl(10,10)&#xA;&amp;gt; mns &amp;lt;- tapply(x, g, mean)&#xA;&amp;gt; vs &amp;lt;- tapply(x, g, var)&#xA;&amp;gt; 9/99*(sum(vs) + 10*var(mns))&#xA;[1] 1.033749&#xA;&amp;gt; var(x)&#xA;[1] 1.033749&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If the sample sizes are not equal, the formula is not so nice.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT: formula for unequal sample sizes&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If there are $g$ sub-samples, each with $k_j, j=1,\ldots,g$ elements for a total of $n=\sum{k_j}$ values, then&#xA;$$ Var(X_1,\ldots,X_{n}) = \frac{1}{n-1}\left(\sum_{j=1}^g (k_j-1) V_j + \sum_{j=1}^g k_j (\bar{X}_j - \bar{X})^2\right), $$&#xA;where $\bar{X} = (\sum_{j=1}^gk_j\bar{X}_j)/n$ is the weighted average of all the means (and equals to the mean of all values).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Again, a demonstration:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; k &amp;lt;- rpois(10, lambda=10)&#xA;&amp;gt; n &amp;lt;- sum(k)&#xA;&amp;gt; g &amp;lt;- factor(rep(1:10, k))&#xA;&amp;gt; x &amp;lt;- rnorm(n)&#xA;&amp;gt; mns &amp;lt;- tapply(x, g, mean)&#xA;&amp;gt; vs &amp;lt;- tapply(x, g, var)&#xA;&amp;gt; 1/(n-1)*(sum((k-1)*vs) + sum(k*(mns-weighted.mean(mns,k))^2))&#xA;[1] 1.108966&#xA;&amp;gt; var(x)&#xA;[1] 1.108966&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;By the way, these formulas are easy to derive by writing the desired variance as the scaled sum of $(X_{ji}-\bar{X})^2$, then introducing $\bar{X}_j$:  $[(X_{ji}-\bar{X}_j)-(\bar{X}_j-\bar{X})]^2$, using the square of difference formula, and simplifying.&lt;/p&gt;&#xA;" OwnerUserId="279" LastEditorUserId="4499" LastEditorDisplayName="" LastEditDate="2011-05-10T16:34:37.680" LastActivityDate="2011-05-10T16:34:37.680" CommentCount="3" />
  <row Id="10446" PostTypeId="2" ParentId="10423" CreationDate="2011-05-06T22:08:35.283" Score="1" ViewCount="" Body="&lt;p&gt;You are probably over impression from the classical modelling, which is vulnerable to the &lt;a href=&quot;http://en.wikipedia.org/wiki/Runge%27s_phenomenon&quot; rel=&quot;nofollow&quot;&gt;Runge paradox&lt;/a&gt;-like problems and thus require some parsimony tuning in post-processing.&lt;br&gt;&#xA;However, in case of machine learning, the idea of including robustness as an aim of model optimization is just the core of the whole domain (often expressed as accuracy on unseen data). So, well, as long as you know your model works good (for instance from CV) there is probably no point to bother. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The real problem with $p\gg n$ in case of ML are the irrelevant attributes -- mostly because some set of them may become more usable for regenerating decision than the truly relevant ones due to some random fluctuations. Obviously this issue has nothing to do with parsimony, but, same as in classical case, ends up in terrible loss of generalization power. How to solve it is a different story, called feature selection -- but the general idea is to pre-process the data to kick out the noise rather than putting constrains on the model.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2011-05-06T22:08:35.283" />
  <row Id="10447" PostTypeId="2" ParentId="4089" CreationDate="2011-05-06T22:08:51.820" Score="0" ViewCount="" Body="&lt;p&gt;I have found this function helpful... the &lt;a href=&quot;http://gossetsstudent.wordpress.com/2010/08/02/159/&quot; rel=&quot;nofollow&quot;&gt;author's handle is respiratoryclub&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/coOFz.png&quot; alt=&quot;Here is what it outputs&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;f_summary &amp;lt;- function(data_to_plot)&#xA;{&#xA;## univariate data summary&#xA;require(nortest)&#xA;#data &amp;lt;- as.numeric(scan (&quot;data.txt&quot;)) #commenting out by mike&#xA;data &amp;lt;- as.numeric(data_to_plot) #added by mike&#xA;&#xA;# first job is to save the graphics parameters currently used&#xA;def.par &amp;lt;- par(no.readonly = TRUE)&#xA;par(&quot;plt&quot; = c(.2,.95,.2,.8))&#xA;layout( matrix(c(1,1,2,2,1,1,2,2,4,5,8,8,6,7,9,10,3,3,9,10), 5, 4, byrow = TRUE))&#xA;&#xA;#histogram on the top left&#xA;h &amp;lt;- hist(data, breaks = &quot;Sturges&quot;, plot = FALSE)&#xA;xfit&amp;lt;-seq(min(data),max(data),length=100)&#xA;yfit&amp;lt;-yfit&amp;lt;-dnorm(xfit,mean=mean(data),sd=sd(data))&#xA;yfit &amp;lt;- yfit*diff(h$mids[1:2])*length(data)&#xA;plot (h, axes = TRUE, main = &quot;Sturges&quot;)&#xA;lines(xfit, yfit, col=&quot;blue&quot;, lwd=2)&#xA;leg1 &amp;lt;- paste(&quot;mean = &quot;, round(mean(data), digits = 4))&#xA;leg2 &amp;lt;- paste(&quot;sd = &quot;, round(sd(data),digits = 4))&#xA;legend(x = &quot;topright&quot;, c(leg1,leg2), bty = &quot;n&quot;)&#xA;&#xA;## normal qq plot&#xA;qqnorm(data, bty = &quot;n&quot;, pch = 20)&#xA;qqline(data)&#xA;p &amp;lt;- ad.test(data)&#xA;leg &amp;lt;- paste(&quot;Anderson-Darling p = &quot;, round(as.numeric(p[2]), digits = 4))&#xA;legend(x = &quot;topleft&quot;, leg, bty = &quot;n&quot;)&#xA;&#xA;## boxplot (bottom left)&#xA;boxplot(data, horizontal = TRUE)&#xA;leg1 &amp;lt;- paste(&quot;median = &quot;, round(median(data), digits = 4))&#xA;lq &amp;lt;- quantile(data, 0.25)&#xA;leg2 &amp;lt;- paste(&quot;25th quantile =  &quot;, round(lq,digits = 4))&#xA;uq &amp;lt;- quantile(data, 0.75)&#xA;leg3 &amp;lt;- paste(&quot;75th quantile = &quot;, round(uq,digits = 4))&#xA;legend(x = &quot;top&quot;, leg1, bty = &quot;n&quot;)&#xA;legend(x = &quot;bottom&quot;, paste(leg2, leg3, sep = &quot;; &quot;), bty = &quot;n&quot;)&#xA;&#xA;## the various histograms with different bins&#xA;h2 &amp;lt;- hist(data,  breaks = (0:12 * (max(data) - min (data))/12)+min(data), plot = FALSE)&#xA;plot (h2, axes = TRUE, main = &quot;12 bins&quot;)&#xA;&#xA;h3 &amp;lt;- hist(data,  breaks = (0:10 * (max(data) - min (data))/10)+min(data), plot = FALSE)&#xA;plot (h3, axes = TRUE, main = &quot;10 bins&quot;)&#xA;&#xA;h4 &amp;lt;- hist(data,  breaks = (0:8 * (max(data) - min (data))/8)+min(data), plot = FALSE)&#xA;plot (h4, axes = TRUE, main = &quot;8 bins&quot;)&#xA;&#xA;h5 &amp;lt;- hist(data,  breaks = (0:6 * (max(data) - min (data))/6)+min(data), plot = FALSE)&#xA;plot (h5, axes = TRUE,main = &quot;6 bins&quot;)&#xA;&#xA;## the time series, ACF and PACF&#xA;plot (data, main = &quot;Time series&quot;, pch = 20)&#xA;acf(data, lag.max = 20)&#xA;pacf(data, lag.max = 20)&#xA;&#xA;## reset the graphics display to default&#xA;par(def.par)&#xA;&#xA;#original code for f_summary by respiratoryclub&#xA;&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="3748" LastActivityDate="2011-05-06T22:08:51.820" />
  <row Id="10448" PostTypeId="2" ParentId="10386" CreationDate="2011-05-06T22:16:54.750" Score="1" ViewCount="" Body="&lt;p&gt;In answer to another question on this forum I referred the OP to &lt;a href=&quot;http://ta-lib.org/function.html&quot; rel=&quot;nofollow&quot;&gt;this site&lt;/a&gt; where there is open source code for a function called HT_PERIOD to measure the instantaneous period of a time series. There are also functions called HT_PHASE, HT_PHASOR and HT_SINE to respectively measure the phase, the phasor components of the sine wave/cyclic component and to extract the sine wave itself (scaled between -1 and 1) of a time series. As the calculations in these functions are causal they would be appropriate for an on-line function that updates as new data comes in. The code for these functions might be of help to you.&lt;/p&gt;&#xA;" OwnerUserId="226" LastActivityDate="2011-05-06T22:16:54.750" />
  <row Id="10449" PostTypeId="2" ParentId="10359" CreationDate="2011-05-06T23:23:47.537" Score="1" ViewCount="" Body="&lt;p&gt;One way is to use transformations of random variables.  It's easy to generate dependent Gaussians; then transform them to uniform variates with the CDF of the gaussian, and then transform the uniform variates to your distribution with the inverse CDF of your distribution.&lt;/p&gt;&#xA;" OwnerUserId="3567" LastActivityDate="2011-05-06T23:23:47.537" CommentCount="1" />
  <row Id="10450" PostTypeId="1" CreationDate="2011-05-07T00:30:21.807" Score="4" ViewCount="97" Body="&lt;p&gt;I'd like to know if anyone can suggest a method for generating random correlation matrix with 90% of the entries (except 1) between $[-0.3, 0.3]$. The other 10% should be larger than 0.3 or smaller than -0.3.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="4383" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-07T08:14:11.543" LastActivityDate="2011-05-12T22:23:22.400" Title="How to generate a bounded random correlation matrix?" Tags="&lt;correlation&gt;" AnswerCount="3" CommentCount="6" />
  <row Id="10451" PostTypeId="2" ParentId="10450" CreationDate="2011-05-07T00:41:16.913" Score="0" ViewCount="" Body="&lt;p&gt;&lt;strike&gt;If all you care about is the proportion of entries between $\pm 0.3$ then sure - generate a random correlation matrix, compute the proportion of entries which are greater than $0.3$ in absolute value, and if there are too many pick some at random and reassign them to random values between $\pm 0.3$. Similarly if there are too few.&lt;/strike&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: Never you mind, this won't work; see the comments...&lt;/p&gt;&#xA;" OwnerUserId="26" LastEditorUserId="26" LastEditorDisplayName="" LastEditDate="2011-05-07T01:54:17.850" LastActivityDate="2011-05-07T01:54:17.850" CommentCount="13" />
  <row Id="10452" PostTypeId="2" ParentId="2181" CreationDate="2011-05-07T03:55:02.787" Score="0" ViewCount="" Body="&lt;p&gt;&quot;An Introduction to Multivariate Statistical Analysis&quot;  Third edition by T. W. Anderson  . &#xA;Wiley series in Probability and Statistics.&lt;/p&gt;&#xA;" OwnerUserId="4116" LastEditorUserId="4116" LastEditorDisplayName="" LastEditDate="2011-05-07T04:00:18.503" LastActivityDate="2011-05-07T04:00:18.503" CommunityOwnedDate="2011-05-07T03:55:02.787" />
  <row Id="10453" PostTypeId="2" ParentId="10439" CreationDate="2011-05-07T07:30:36.927" Score="6" ViewCount="" Body="&lt;p&gt;The general result you are looking for (under the stated assumptions) looks like this:&#xA;For linear regression with $p$ predictor variables (you have two, $X$ and $X^2$) and an intercept, then with $n$ observations, $\mathbf{X}$ the $n \times (p+1)$ design matrix,  $\hat{\beta}$ the $p+1$ dimensional estimator and $a \in \mathbb{R}^{p+1}$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ \frac{a^T\hat{\beta} - a^T \beta}{\hat{\sigma}&#xD;&#xA;  \sqrt{a^T(\mathbf{X}^T\mathbf{X})^{-1}a}} \sim t_{n-p-1}.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The consequence is that you can construct confidence intervals for any linear combination of the $\beta$ vector using the same $t$-distribution you use to construct a confidence interval for one of the coordinates. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case, $p = 2$ and $a^T = (0, x_2 - x_1, x_2^2 - x_1^2)$. The denominator in the formula above is the square root of what you compute as the estimate of the standard error (provided that this is what the software computes ...). Note that the variance estimator, $\hat{\sigma}^2$, is supposed to be the (usual) unbiased estimator, where you divide by the degrees of freedom, $n-p-1$, and not the number of observations $n$.&lt;/p&gt;&#xA;" OwnerUserId="4376" LastEditorUserId="4376" LastEditorDisplayName="" LastEditDate="2011-05-07T18:07:15.263" LastActivityDate="2011-05-07T18:07:15.263" CommentCount="3" />
  <row Id="10454" PostTypeId="1" AcceptedAnswerId="10456" CreationDate="2011-05-07T08:48:57.553" Score="3" ViewCount="100" Body="&lt;p&gt;I usually use the &lt;code&gt;plot(lm())&lt;/code&gt; or &lt;code&gt;plot(glm())&lt;/code&gt; (combined with &lt;code&gt;par(mfrow=c(2,2)&lt;/code&gt;) to analyze residuals. Unfortunately this is not possible whenn estimating a &lt;code&gt;tobit()&lt;/code&gt; (package &lt;a href=&quot;http://cran.r-project.org/web/packages/AER/index.html&quot; rel=&quot;nofollow&quot;&gt;AER&lt;/a&gt;) or &lt;code&gt;zeroinfl()&lt;/code&gt; (package &lt;a href=&quot;http://cran.r-project.org/web/packages/pscl/index.html&quot; rel=&quot;nofollow&quot;&gt;pscl&lt;/a&gt;) model. Furthermore any Cooks distance command does not work. It's either not possible to make the residual analysis in a convienent way, or I'm just too dumb (this is probably the reason). Any help is appreciated! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a short example of what I am talking about:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(AER)&#xA;require(stats)&#xA;attach(cars)&#xA;tob&amp;lt;-tobit(dist~speed)&#xA;summary(tob)&#xA;par(mfrow=c(2,2))&#xA;plot(tob)           #doesn't work&#xA;cooks.distance(tob) #doesn't work&#xA;detach(cars)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="4496" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-06-02T08:09:54.840" LastActivityDate="2011-06-02T08:09:54.840" Title="How to get Cook's distance and carry out residual analysis for non-lm() and non-glm() models in R?" Tags="&lt;r&gt;&lt;outliers&gt;&lt;residuals&gt;" AnswerCount="1" />
  <row Id="10455" PostTypeId="2" ParentId="10427" CreationDate="2011-05-07T09:05:21.043" Score="3" ViewCount="" Body="&lt;p&gt;Fixed effect panel regression models involve subtracting group means from the regressors. This means that you can only include time-varying regressors in the model. Since firms usually belong to one industry the dummy variable for industry does not vary with time. Hence it is excluded from your model by Stata, since after subtracting the group mean from such variable you will get that it is equal to zero.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that Hausman test is a bit tricky, so you cannot solely base your model selection (fixed vs random effects) with it. Wooldridge explains it very nicely (in my opinion) in &lt;a href=&quot;http://books.google.com/books?id=cdBPOJUP4VsC&amp;amp;lpg=PP1&amp;amp;dq=wooldridge%20panel%20data&amp;amp;hl=fr&amp;amp;pg=PA289#v=onepage&amp;amp;q&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;his book&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-05-07T09:05:21.043" />
  <row Id="10456" PostTypeId="2" ParentId="10454" CreationDate="2011-05-07T09:25:35.793" Score="4" ViewCount="" Body="&lt;p&gt;As described in the on-line help, the &lt;code&gt;cooks.distance()&lt;/code&gt; function expects an object of class &lt;code&gt;lm&lt;/code&gt; or &lt;code&gt;glm&lt;/code&gt; so it is not possible to get it work with other type of models. It is defined in &lt;code&gt;src/library/stats/R/lm.influence.R&lt;/code&gt;, from R source, so you can browse the code directly and build your own function if nothing exits in other places. A quick way of seeing what it does is to type &lt;code&gt;stats:::cooks.distance.lm&lt;/code&gt; at the R prompt, though.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, as &lt;code&gt;tobit&lt;/code&gt; is nothing more than a wrapper for &lt;code&gt;survreg&lt;/code&gt;, all attached methods to the latter kind of R object might be used. For example, there's a &lt;code&gt;residuals.survreg&lt;/code&gt; (in the &lt;a href=&quot;http://cran.r-project.org/web/packages/survival/index.html&quot; rel=&quot;nofollow&quot;&gt;survival&lt;/a&gt; package) S3 method for extracting residuals from objects inheriting from class &lt;code&gt;survreg&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-07T09:50:17.383" LastActivityDate="2011-05-07T09:50:17.383" CommentCount="5" />
  <row Id="10457" PostTypeId="1" CreationDate="2011-05-06T14:41:22.950" Score="2" ViewCount="49" Body="&lt;p&gt;Hello,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've got a question concerning a negbin regression: suppose that you have the following commands &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(MASS)&#xA;attach(cars)&#xA;mod.NB&amp;lt;-glm.nb(dist~speed)&#xA;summary(mod.NB)&#xA;detach(cars)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(note that cars is a dataset which is available in R) and don't care if this model makes sense. What I'd like to know is, how can I interpret the variable theta (at the bottom of the summary). Is this the shape parameter of the negbin distribution and is it possible to interpret it as a measure of skewness? I appreciate every thought!&lt;/p&gt;&#xA;" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-07T11:46:52.683" LastActivityDate="2011-05-07T11:50:28.077" Title="Interpreting  negative binomial regression output in R" Tags="&lt;r&gt;&lt;regression&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="10458" PostTypeId="2" ParentId="10457" CreationDate="2011-05-06T20:11:54.580" Score="2" ViewCount="" Body="&lt;p&gt;If Y is NB with  mean  $\mu$  then $var(Y) = \mu  + \mu^2/\theta$. Skewness would not be an appropriate interpretation, but the departure from $\theta= 1$  is often taken as &quot;extra-Poisson&quot; dispersion.&lt;/p&gt;&#xA;" OwnerUserId="2129" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-07T11:50:28.077" LastActivityDate="2011-05-07T11:50:28.077" />
  <row Id="10459" PostTypeId="1" CreationDate="2011-05-07T11:13:51.243" Score="13" ViewCount="304" Body="&lt;p&gt;Hi all! Can you suggest some good movies which involve math, probabilities etc? One example is &lt;a href=&quot;http://en.wikipedia.org/wiki/21_%282008_film%29&quot; rel=&quot;nofollow&quot;&gt;21&lt;/a&gt;. I would also be interested in movies that involve algorithms (e.g. text decryption). In general &quot;geeky&quot; movies with famous scientific theories but no science fiction or documentaries. Thanks in advance!&lt;/p&gt;&#xA;" OwnerUserId="4504" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-07T12:34:34.433" LastActivityDate="2011-05-08T07:16:23.570" CommunityOwnedDate="2011-05-07T12:32:39.210" Title="Are there any good movies involving mathematics or probability?" Tags="&lt;probability&gt;&lt;video&gt;" AnswerCount="10" CommentCount="5" FavoriteCount="1" />
  <row Id="10460" PostTypeId="2" ParentId="10459" CreationDate="2011-05-07T13:05:04.590" Score="4" ViewCount="" Body="&lt;p&gt;'&lt;a href=&quot;http://www.imdb.com/title/tt0268978/&quot; rel=&quot;nofollow&quot;&gt;A Beautiful Mind&lt;/a&gt;' naturally has a bit of game theory in it.&lt;/p&gt;&#xA;" OwnerUserId="4360" LastActivityDate="2011-05-07T13:05:04.590" CommunityOwnedDate="2011-05-07T13:05:04.590" CommentCount="1" />
  <row Id="10462" PostTypeId="2" ParentId="10444" CreationDate="2011-05-07T13:23:49.177" Score="4" ViewCount="" Body="&lt;p&gt;I can't see that standardization is a good idea in ordinary regression or with a longitudinal model.  It makes predictions harder to obtain and doesn't solve a problem that needs solving, usually.  And what if you have $x$ and $x^2$ in the model.  How do you standardize $x^2$?  What if you have a continuous variable and a binary variable in the model?  How do you standardize the binary variable?  Certainly not by its standard deviation, which would cause low prevalence variables to have greater importance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general it's best to interpret model effects on the original scale of $x$.&lt;/p&gt;&#xA;" OwnerUserId="4253" LastActivityDate="2011-05-07T13:23:49.177" CommentCount="10" />
  <row Id="10463" PostTypeId="1" CreationDate="2011-05-07T13:59:28.740" Score="2" ViewCount="57" Body="&lt;p&gt;Possible &lt;a href=&quot;http://stats.stackexchange.com/questions/8130/sample-size-required-to-determine-which-of-a-set-of-advertisements-has-the-highes&quot;&gt;Duplicate&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, I am not a statistician (though I'd like be one) but I am trying to understand how different tests can be utilized to examine samples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say that I have five of Google AdWords ads and I'm trying to develop a test to show which has been more effective at getting users to click on that ad.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;     January   February&#xA; Ad A - 850       5000 &#xA;&#xA; Ad B - 900       5300&#xA;&#xA; Ad C - 880       5100&#xA;&#xA; Ad D - 880       5100  &#xA;&#xA; Ad E - 800       5000  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In terms of frequency, Ad B has the highest amount of user click throughs in both January and February. (Is it fine just to say that one has a higher freqency and is therefore the more effective ad without relying on any significance test)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, for each month, I want to use a statistical test to determine which ad was more effective. Initially, I assumed a chi-square (pearsons) test would be appropriate, but with five seperate ads, I'm not sure if that's the appropriate plan of action. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone help me understand what action to take and why, or is it enough just to rely on the frequency distribution of the five ads.&lt;/p&gt;&#xA;" OwnerUserId="3310" LastEditorUserId="3310" LastEditorDisplayName="" LastEditDate="2011-05-07T21:47:19.420" LastActivityDate="2011-05-07T21:47:19.420" Title="Determining which AdWords have the highest amount of user click throughs" Tags="&lt;statistical-significance&gt;&lt;chi-squared&gt;" CommentCount="1" FavoriteCount="1" />
  <row Id="10464" PostTypeId="1" AcceptedAnswerId="10467" CreationDate="2011-05-07T14:13:51.813" Score="7" ViewCount="84" Body="&lt;p&gt;I've been taught that binning a continuous variable into categories is almost never a good idea, because you lose information in the process. But now I'm facing a situation where I have an age variable that is &quot;mostly continuous&quot;, which is to say that about 90% of the values represent age in years, and the remaining 10% are recorded as an ordered factor with haphazard levels, e.g., &gt;18, 18-34, &gt;50 ...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to use age as a predictor of a continuous outcome variable, but I'm not sure how to proceed. Should I make this into an ordered factor, even though this will mean throwing away information in 90% of the cases? If not, what do I do with the 10% of the cases where age is already an ordered factor? Any suggestions will be appreciated.&lt;/p&gt;&#xA;" OwnerUserId="4506" LastActivityDate="2011-05-07T15:14:25.560" Title="What to do with almost-continuous variable in regression" Tags="&lt;regression&gt;&lt;categorical-data&gt;&lt;continuous-data&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="10465" PostTypeId="2" ParentId="10459" CreationDate="2011-05-07T14:19:42.887" Score="7" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://www.imdb.com/title/tt0138704/&quot; rel=&quot;nofollow&quot;&gt;Pi&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="226" LastActivityDate="2011-05-07T14:19:42.887" CommunityOwnedDate="2011-05-07T14:19:42.887" CommentCount="1" />
  <row Id="10466" PostTypeId="2" ParentId="10425" CreationDate="2011-05-07T15:02:08.107" Score="1" ViewCount="" Body="&lt;p&gt;I was challenged by a client to solve this problem in an automatic i.e. turnkey way. I implemented an approach that for each pair ( i.e. y and a candidate x ) , prewhiten , compute cross-correlations of the pre-whitened series, identify the PDL ( OR ADL AUTOREGRESSIVE DISTRIBUTED LAG MODEL including any DEAD TIME ) while incorporating Intervention Detection to yield robust estimates, develop a &quot;measure&quot; for this structure. After conducting this for ALL candidate regressors, rank them by the &quot;measure&quot; and then select the top K regressors based upon the &quot;measure&quot;. This is sometimes referred to as Linear Filtering. We successfully incorporated this heuristic into our commercially available time series package. You should be able to &quot;ROLL YOUR OWN&quot; if you have sufficient time and statistical programming skills and/or available modules to implement some of these tasks. &lt;/p&gt;&#xA;" OwnerUserId="3382" LastEditorUserId="3382" LastEditorDisplayName="" LastEditDate="2011-05-07T15:26:53.670" LastActivityDate="2011-05-07T15:26:53.670" />
  <row Id="10467" PostTypeId="2" ParentId="10464" CreationDate="2011-05-07T15:14:25.560" Score="4" ViewCount="" Body="&lt;p&gt;I'd say interact continuous age with a dummy &quot;continuous age is available&quot;, and categorical age with a dummy &quot;continuous age is not available&quot;. That way you'll be using as much of the information you have as possible.&#xA;Of course if the effect of age is something you'd like to be able to summarize with just one point estimate, you'll have to think a bit more (though the coefficient on continuous age should be a pretty good approximation for that).&lt;/p&gt;&#xA;" OwnerUserId="2044" LastActivityDate="2011-05-07T15:14:25.560" CommentCount="6" />
  <row Id="10468" PostTypeId="2" ParentId="10450" CreationDate="2011-05-07T16:01:26.687" Score="0" ViewCount="" Body="&lt;p&gt;Here's an older answer to a similar question on SO.   It has some code that you could try/modify:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://stackoverflow.com/questions/1040324/how-to-generate-pseudo-random-positive-definite-matrix-with-constraints-on-the-of&quot;&gt;Similar Question&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some other links:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.oxford-man.ox.ac.uk/documents/papers/2011OMI08_Sheppard.pdf&quot; rel=&quot;nofollow&quot;&gt;Forecasting Covariance Matrices&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.kevinsheppard.com/images/4/47/Chapter8.pdf&quot; rel=&quot;nofollow&quot;&gt;Various Matrix Techniques&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://arxiv.org/PS_cache/arxiv/pdf/1009/1009.5331v1.pdf&quot; rel=&quot;nofollow&quot;&gt;Matrix Shrinkage Technique&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2775" LastEditorUserId="2775" LastEditorDisplayName="" LastEditDate="2011-05-11T15:03:29.787" LastActivityDate="2011-05-11T15:03:29.787" />
  <row Id="10469" PostTypeId="2" ParentId="10459" CreationDate="2011-05-07T16:13:17.890" Score="2" ViewCount="" Body="&lt;p&gt;There are several movie versions of &lt;em&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Flatland&quot; rel=&quot;nofollow&quot;&gt;Flatland&lt;/a&gt;&lt;/em&gt;. And there's &lt;em&gt;The Great $\pi$/e Debate&lt;/em&gt;.&lt;/p&gt;&#xA;" OwnerUserId="3874" LastEditorUserId="264" LastEditorDisplayName="" LastEditDate="2011-05-07T19:40:38.563" LastActivityDate="2011-05-07T19:40:38.563" CommunityOwnedDate="2011-05-07T16:13:17.890" />
  <row Id="10470" PostTypeId="2" ParentId="10459" CreationDate="2011-05-07T16:26:44.237" Score="3" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://zalafilms.com/films/nisanumber.html&quot; rel=&quot;nofollow&quot;&gt;N Is a Number: A Portrait of Paul Erdős&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="22" LastActivityDate="2011-05-07T16:26:44.237" CommunityOwnedDate="2011-05-07T16:26:44.237" />
  <row Id="10471" PostTypeId="2" ParentId="10459" CreationDate="2011-05-07T17:20:21.900" Score="2" ViewCount="" Body="&lt;p&gt;Early in &lt;a href=&quot;http://en.wikipedia.org/wiki/The_Social_Network#Plot&quot; rel=&quot;nofollow&quot;&gt;The Social Network&lt;/a&gt; begins with a one night hackathon where Mark Zuckerberg uses the &lt;a href=&quot;http://en.wikipedia.org/wiki/Elo_rating_system&quot; rel=&quot;nofollow&quot;&gt;Elo rating system algorithm&lt;/a&gt; to&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;... create a website that rates the attractiveness of female students&#xA;  when compared to each other. ... in a few hours, using an algorithm for&#xA;  ranking chess players supplied by his best friend, Eduardo Saverin, he&#xA;  creates a website called &quot;FaceMash,&quot; where students can choose which of&#xA;  two girls presented at a time is more&#xA;  attractive.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;However, much of the rest of the movie is devoted to episodes of hacking, corporate politics, lawsuits, escapades, Zuckerberg's interpersonal problems, etc.  But, I found it quite fascinating, overall.  A great geek movie.&lt;/p&gt;&#xA;" OwnerUserId="4508" LastActivityDate="2011-05-07T17:20:21.900" CommunityOwnedDate="2011-05-07T17:20:21.900" />
  <row Id="10472" PostTypeId="2" ParentId="10459" CreationDate="2011-05-07T17:43:25.480" Score="3" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://www.imdb.com/title/tt0377107/&quot; rel=&quot;nofollow&quot;&gt;Proof&lt;/a&gt; was pretty good.&lt;/p&gt;&#xA;" OwnerUserId="3748" LastActivityDate="2011-05-07T17:43:25.480" CommunityOwnedDate="2011-05-07T17:43:25.480" />
  <row Id="10473" PostTypeId="2" ParentId="10366" CreationDate="2011-05-07T18:40:09.827" Score="1" ViewCount="" Body="&lt;p&gt;If your penultimate goal is to embed your plot in a Latex document, you might consider using the &lt;a href=&quot;ftp://ftp.dante.de/tex-archive/help/Catalogue/entries/gnuplottex.html&quot; rel=&quot;nofollow&quot;&gt;gnuplottex&lt;/a&gt; package (as an alternative to &lt;a href=&quot;http://ctan.org/pkg/pgfplots&quot; rel=&quot;nofollow&quot;&gt;pgfplots&lt;/a&gt; which is an awesome package). The idea is rather simple: you write your code chunk in your tex document directly (like you would do with Sweave), et voilà!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an example (grabbed from gnuplot demos):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;\documentclass{standalone}&#xA;\usepackage{gnuplottex}&#xA;&#xA;\begin{document}&#xA;\begin{gnuplot}[scale=0.95, terminal=epslatex]&#xA;set style fill transparent pattern 4 bo&#xA;set style function filledcurves y1=0&#xA;set clip two    &#xA;Gauss(x,mu,sigma) = 1./(sigma*sqrt(2*pi)) * exp( -(x-mu)**2 / (2*sigma**2) )&#xA;d1(x) = Gauss(x, 0.5, 0.5)&#xA;d2(x) = Gauss(x,  2.,  1.)&#xA;d3(x) = Gauss(x, -1.,  2.)&#xA;set xrange [-5:5]&#xA;set yrange [0:1]&#xA;set xlabel &quot;$\\bar x$ values&quot;&#xA;unset colorbox    &#xA;plot d1(x) fs solid 1.0 lc rgb &quot;forest-green&quot;, \&#xA;     d2(x) lc rgb &quot;gold&quot;, d3(x) lc rgb &quot;red&quot;&#xA;\end{gnuplot}&#xA;\end{document}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You'll need to compile with the &lt;code&gt;-shell-escape&lt;/code&gt; option to &lt;code&gt;pdflatex&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.stack.imgur.com/zhQyn.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-05-07T18:40:09.827" />
  <row Id="10474" PostTypeId="2" ParentId="10459" CreationDate="2011-05-07T18:40:18.220" Score="5" ViewCount="" Body="&lt;p&gt;Not a movie, but a TV series:&#xA;&lt;a href=&quot;http://en.wikipedia.org/wiki/Numb3rs&quot; rel=&quot;nofollow&quot;&gt;Numb3rs&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="264" LastActivityDate="2011-05-07T18:40:18.220" CommunityOwnedDate="2011-05-07T18:40:18.220" />
  <row Id="10475" PostTypeId="2" ParentId="7045" CreationDate="2011-05-07T18:54:02.790" Score="1" ViewCount="" Body="&lt;p&gt;I would suggest trying the &lt;a href=&quot;http://cran.r-project.org/web/packages/glmnet/index.html&quot; rel=&quot;nofollow&quot;&gt;glmnet package&lt;/a&gt; for feature selection.  glmnet uses the &lt;a href=&quot;http://citeseer.ist.psu.edu/viewdoc/download;jsessionid=6DD6392682DB8F43219A2C81171B64CF?doi=10.1.1.124.4696&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;elastic net&lt;/a&gt; for regularization and feature selection, and is a pretty solid method for choosing a subset of variables for a glm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to automatically choose transformations for your dependent variable, I suggest checking out the &lt;a href=&quot;http://www.oga-lab.net/RGM2/func.php?rd_id=caret%3apreProcess&quot; rel=&quot;nofollow&quot;&gt;preProcess&lt;/a&gt; function in the &lt;a href=&quot;http://cran.r-project.org/web/packages/caret/index.html&quot; rel=&quot;nofollow&quot;&gt;caret&lt;/a&gt; package.  preProcess can help you choose sensible transformations for both your dependent and independent variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, preProcess will select transformations for you, such as log(y).  Then glmnet will select a sensible subset of your variables to build a model (i.e. not all of your independent variables may be important for your model).  You can use the cv.glmnet function to fine-tune the hyper-parameters.  I hope I've answered all your questions!&lt;/p&gt;&#xA;" OwnerUserId="2817" LastActivityDate="2011-05-07T18:54:02.790" CommentCount="7" />
  <row Id="10476" PostTypeId="2" ParentId="10459" CreationDate="2011-05-07T18:56:38.453" Score="1" ViewCount="" Body="&lt;p&gt;&lt;em&gt;&lt;a href=&quot;http://www.imdb.com/title/tt0478087/&quot; rel=&quot;nofollow&quot;&gt;21&lt;/a&gt;&lt;/em&gt; - based on the book Bringing Down the House (MIT Blackjack team)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Near the beginning they discuss the Monty Hall Problem.  However after that there isn't much actual math/probability.&lt;/p&gt;&#xA;" OwnerUserId="2310" LastEditorUserId="4360" LastEditorDisplayName="" LastEditDate="2011-05-07T23:56:16.110" LastActivityDate="2011-05-07T23:56:16.110" CommunityOwnedDate="2011-05-07T18:56:38.453" />
  <row Id="10477" PostTypeId="2" ParentId="10459" CreationDate="2011-05-07T19:25:34.977" Score="1" ViewCount="" Body="&lt;p&gt;I have not seen this yet, but it seems somewhat geeky:&#xA;&lt;a href=&quot;http://www.imdb.com/title/tt1016301/&quot; rel=&quot;nofollow&quot; title=&quot;Fermat's Room&quot;&gt;Fermat's Room&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2011-05-07T19:25:34.977" CommunityOwnedDate="2011-05-07T19:25:34.977" CommentCount="1" />
  <row Id="10478" PostTypeId="1" CreationDate="2011-05-07T22:37:35.410" Score="7" ViewCount="170" Body="&lt;p&gt;Are there any analytical results or experimental papers regarding the &lt;em&gt;optimal&lt;/em&gt; choice of the coefficient of the $\ell_1$ penalty term. By &lt;em&gt;optimal&lt;/em&gt;, I mean a parameter that maximizes the probability of selecting the best model, or that minimizes the expected loss. I am asking because often it is impractical to choose the parameter by cross-validation or bootstrap, either because of a large number of instances of the problem, or because of the size of the problem at hand. The only positive result I am aware of is Candes and Plan, &lt;a href=&quot;http://www-stat.stanford.edu/~candes/papers/LassoPredict.pdf&quot; rel=&quot;nofollow&quot;&gt;Near-ideal model selection by $\ell_1$ minimization&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="30" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-07T23:57:03.037" LastActivityDate="2011-05-09T03:45:30.610" Title="Optimal penalty selection for lasso" Tags="&lt;model-selection&gt;&lt;lasso&gt;&lt;shrinkage&gt;" AnswerCount="2" CommentCount="3" FavoriteCount="3" />
  <row Id="10479" PostTypeId="2" ParentId="9715" CreationDate="2011-05-07T22:51:34.940" Score="0" ViewCount="" Body="&lt;p&gt;You could also try running a multinomial logit using the glmnet package.  I'm not sure how to force it to keep all variables, but I'm sure it's possible.&lt;/p&gt;&#xA;" OwnerUserId="2817" LastActivityDate="2011-05-07T22:51:34.940" />
  <row Id="10480" PostTypeId="1" AcceptedAnswerId="10485" CreationDate="2011-05-07T23:24:21.390" Score="4" ViewCount="95" Body="&lt;p&gt;This is a homework problem out of the book. It says&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;If $U$ is a uniform random variable on [0,1], what is the distribution of the  random variable $X = [nU]$, where [$t$] denotes the greatest integer less than or equal to $t$?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;There is a second part that says&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Do this for $n = 10$. True or false, and explain: “Random&#xA;  digit” is a good name for the random variable $X$&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I don't even know where to begin. All the book says about Uniform Distribution is that uniform density is $f(x) = 1, 0 \le x \le 1$. What does $t$ represent and how do I start this problem?&lt;/p&gt;&#xA;" OwnerUserId="4401" LastActivityDate="2011-05-08T04:51:12.043" Title="Uniform random variable distribution" Tags="&lt;distributions&gt;&lt;homework&gt;&lt;uniform&gt;" AnswerCount="2" CommentCount="3" />
  <row Id="10481" PostTypeId="2" ParentId="10480" CreationDate="2011-05-07T23:38:16.203" Score="2" ViewCount="" Body="&lt;p&gt;$t$ is just a placeholder name for a variable, the actual focus in that explanation is on the square brackets which refer to the &lt;em&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Floor_and_ceiling_functions&quot; rel=&quot;nofollow&quot;&gt;floor&lt;/a&gt;&lt;/em&gt; function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would start by plotting the function that maps from $U$ to $X$, that is $X(u)=[nu]$ in the range of all the values $U$ can assume. What does the set of possible function values (i.e. the &lt;a href=&quot;http://en.wikipedia.org/wiki/Image_%28mathematics%29&quot; rel=&quot;nofollow&quot;&gt;image&lt;/a&gt; of the function) look like? How big are their relative proportions regarding their preimages? That should give you a clue how to get the distribution function of $X$.&lt;/p&gt;&#xA;" OwnerUserId="4360" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-08T04:51:12.043" LastActivityDate="2011-05-08T04:51:12.043" />
  <row Id="10482" PostTypeId="1" CreationDate="2011-05-07T23:00:27.467" Score="3" ViewCount="61" Body="&lt;p&gt;I was impressed by the Reputation histogram on the SE sites allowing one to zoom in to any time interval.  How was it created?&lt;/p&gt;&#xA;" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-07T23:53:52.087" LastActivityDate="2011-05-07T23:53:52.087" Title="How to create histogram with &quot;zoom-in&quot; feature" Tags="&lt;data-visualization&gt;" AnswerCount="1" />
  <row Id="10483" PostTypeId="2" ParentId="10482" CreationDate="2011-05-07T23:51:42.400" Score="4" ViewCount="" Body="&lt;p&gt;This is a barplot rather than histogram... Anyway, judging from the page source, it is made with &lt;a href=&quot;http://www.highcharts.com/&quot; rel=&quot;nofollow&quot;&gt;Highcharts JS&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2011-05-07T23:51:42.400" />
  <row Id="10484" PostTypeId="1" AcceptedAnswerId="10489" CreationDate="2011-05-08T00:25:47.643" Score="7" ViewCount="117" Body="&lt;p&gt;I have some data in R, stored in a list. Think&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;d &amp;lt;- c(1,2,3,4) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;although this is not my data. If I then enter the command&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; plot(density(d, kernel=&quot;gaussian&quot;, width=1))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;then I get the kernel probability density estimate, where the kernel is standard normal. If I replace 1 with other numbers, of course the picture changes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I would like to do is create a video or animation in which each frame is such a plot, but the bandwidth of the kernel varies from frame to frame, thereby showing the effect of changing the bandwidth. How can I do this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(My apologies if this is not the right place to ask questions about R.)&lt;/p&gt;&#xA;" OwnerUserId="98" LastActivityDate="2011-05-08T10:44:42.037" Title="Animating the effect of changing kernel width in R" Tags="&lt;r&gt;&lt;kernel&gt;" AnswerCount="4" FavoriteCount="3" />
  <row Id="10485" PostTypeId="2" ParentId="10480" CreationDate="2011-05-08T00:45:37.190" Score="3" ViewCount="" Body="&lt;p&gt;$[t]$ is the floor function, and $t$ just represents a generic argument.  So for example $[0.5]=0$, $[0.9]=0$, $[1.01]=1$, $[1]=1$, $[23.567]=23$, and so on.  You simply ignore whats written after the decimal point (note: this is not the same thing as rounding, for $[0.9]=0$ whereas rounding would give $1$.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With non-smooth functions such as the floor function, the safest way to go is to use the cumulative distribution function, or CDF.  For the uniform distribution this is given by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$F_{U}(y)=\Pr(U&amp;lt;y)=\int_{0}^{y}f_{U}(t)dt=\int_{0}^{y}dt=y$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now the good thing about CDFs is that you can simply substitute the functional relation in, but only once you have inverted the floor function.  Now this inversion is not 1-to-1, so a standard change of variables using jacobian's doesn't apply.  For example, suppose $X=0$.  Then we know that $[nU]=0$, which means that $nU&amp;lt;1$, which implies that $U&amp;lt;n^{-1}$.  We can work out this probability directly from the CDF:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\Pr(X=0)=\Pr(U&amp;lt;n^{-1})=F_{U}(n^{-1})=n^{-1}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reason we can do this is that the two propositions &quot; $X=0$ &quot; and &quot; $U&amp;lt;n^{-1}$ &quot; are equivalent - one occurs if and only if the other occurs.  So they must have the same &quot;truth value&quot; and hence also the same probability.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is not too hard to continue on.  Suppose $X=1$, then we must have $nU&amp;lt;2$ (or else $X&gt;1$) and we must also have $nU&amp;gt;1$ (or else $X=0$ as we have just seen).  So the equivalent condition to $X=1$ in terms of $U$ is $1&amp;lt;nU&amp;lt;2$.  I'll stop my answer here so you can work out the general form of the probability mass function for $X$ ($\Pr(X=z)$ for general argument $z$).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One small hint is to note that $\Pr(a&amp;lt;U&amp;lt;b)=\Pr(U&amp;lt;b)-\Pr(U&amp;lt;a)=b-a$ for a uniform distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can post the full answer if you wish, but you may not learn as well compared to if you do it yourself.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-05-08T03:32:23.783" LastActivityDate="2011-05-08T03:32:23.783" CommentCount="6" />
  <row Id="10486" PostTypeId="1" CreationDate="2011-05-08T00:51:51.530" Score="4" ViewCount="86" Body="&lt;p&gt;Suppose I want to predict Amazon or Netflix demand, using demand data over the past year. For example, I might want to forecast the number of sales in the Electronics category on Amazon, or the number of times someone wants to rent Titanic on Netflix. My dataset consists of daily demand per item over the past couple of months, along with item metadata (tags and categories), split by things like customer demographics (age group, gender, location, browser, job -- some of these might be unknown).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To be concrete, let's suppose I want to forecast the number of times someone wants to rent a Comedy on Netflix, and I want to make this forecast at various levels (e.g., overall, by the state the customer lives in, by male/female, etc.). How would I go about this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My naive first thought is to form a time series at each level I care about (e.g., form a time series of comedy demand by all the males living in Florida), and build some kind of time series model on top of this (I guess an ARIMA model...?). But this seems wrong for a bunch of reasons (not only would I be building a ton of different models for all the different possible levels, but each level would be ignoring a lot of data from closely related levels).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestions? Surprisingly, I couldn't find any papers related to this problem when Googling, but I might just be using the wrong search terms. (I learned a smidgen of time series analysis a couple years ago, but I was incredibly bad at it.) Also, I'm interested in both methods (what algorithms to use) and particular statistical libraries that might be useful (e.g., R packages or Python libraries).&lt;/p&gt;&#xA;" OwnerUserId="1106" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-08T02:27:15.560" LastActivityDate="2011-05-09T15:48:15.743" Title="Forecasting Amazon or Netflix demand" Tags="&lt;time-series&gt;&lt;forecasting&gt;" AnswerCount="2" />
  <row Id="10487" PostTypeId="2" ParentId="10486" CreationDate="2011-05-08T01:17:36.197" Score="2" ViewCount="" Body="&lt;p&gt;When you have a number of Endogenous series possibly cross-related and a number of Exogenous series, this is referred to as a VECTOR ARIMA MODEL , a super-set of a VAR model and an ARIMA MODEL.  We have resolved Daily forecasts for a Family TYpe and Daily forecasts for &quot;children&quot; or &quot;subset categories&quot; by incorporating the Parent as a possible predictor variable to model each of the &quot;children&quot; speaking to your concern about &quot;ignoring data&quot;. VARIMA models are not really practical because  each series might have different dependencies on level shifts and local time trends. My suggestion is that you form &quot;reasonable families&quot; , model both the Parent and the Children using ARMAX models and then use a reconciliation strategy.&lt;/p&gt;&#xA;" OwnerUserId="3382" LastEditorUserId="3382" LastEditorDisplayName="" LastEditDate="2011-05-08T02:40:32.763" LastActivityDate="2011-05-08T02:40:32.763" CommentCount="2" />
  <row Id="10488" PostTypeId="2" ParentId="10484" CreationDate="2011-05-08T02:16:55.657" Score="6" ViewCount="" Body="&lt;p&gt;One way to go is to use the excellent &lt;a href=&quot;http://animation.yihui.name/animation%3astart&quot; rel=&quot;nofollow&quot;&gt;animation&lt;/a&gt; package by Yihui Xie. I uploaded a very simple example to my public dropbox account: &lt;a href=&quot;http://dl.dropbox.com/u/6973449/RtmpeKvc2B/index.html&quot; rel=&quot;nofollow&quot;&gt;densityplot&lt;/a&gt; (I will remove this example in 3 days). Is this what you are looking for? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The animation was created using the following R code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(animation)&#xA;density.ani &amp;lt;- function(){&#xA;    i &amp;lt;- 1  &#xA;    d &amp;lt;- c(1,2,3,4) &#xA;    while (i &amp;lt;= ani.options(&quot;nmax&quot;)) {&#xA;      plot(density(d, kernel=&quot;gaussian&quot;, bw = i), ylim = c(0, 0.25))&#xA;      ani.pause()&#xA;      i &amp;lt;- i + 1&#xA;    }&#xA;}&#xA;&#xA;saveHTML({&#xA;par(mar = c(5, 4, 1, 0.5))&#xA;density.ani()&#xA;}, nmax = 30, title = &quot;Changing kernel width&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="307" LastEditorUserId="307" LastEditorDisplayName="" LastEditDate="2011-05-08T10:44:42.037" LastActivityDate="2011-05-08T10:44:42.037" CommentCount="1" />
  <row Id="10489" PostTypeId="2" ParentId="10484" CreationDate="2011-05-08T02:17:18.763" Score="7" ViewCount="" Body="&lt;p&gt;It depends a little bit on what your end goal is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Quick and dirty hack for real-time demonstrations&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using &lt;code&gt;Sys.sleep(seconds)&lt;/code&gt; in a loop where &lt;code&gt;seconds&lt;/code&gt; indicates the number of seconds between frames is a viable option. You'll need to set the &lt;code&gt;xlim&lt;/code&gt; and &lt;code&gt;ylim&lt;/code&gt; parameters in your call to &lt;code&gt;plot&lt;/code&gt; to make things behave as expected.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's some simple demonstration code.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Just a quick test of Sys.sleep() animation&#xA;&#xA;x &amp;lt;- seq(0,2*pi, by=0.01)&#xA;y &amp;lt;- sin(x)&#xA;n &amp;lt;- 5&#xA;pause &amp;lt;- 0.5&#xA;&#xA;ybnds &amp;lt;- quantile(n*y, probs=c(0,1))&#xA;x11()&#xA;&#xA;# Draw successively taller sinewaves with a gradually changing color&#xA;for( i in 1:n )&#xA;{&#xA;    plot(x, i*y, type=&quot;l&quot;, lwd=2, ylim=ybnds, col=topo.colors(2*n)[i])&#xA;    Sys.sleep(pause)&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This works pretty well, especially using X-Windows as the windowing system. I've found that Mac's &lt;code&gt;quartz()&lt;/code&gt; does not play nice, unfortunately.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Animated GIFs&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you need something that can be redistributed, posted on a webpage, etc., look at the &lt;code&gt;write.gif&lt;/code&gt; function in the &lt;a href=&quot;http://cran.r-project.org/web/packages/caTools/index.html&quot; rel=&quot;nofollow&quot;&gt;caTools&lt;/a&gt; package. Displaying help on &lt;code&gt;write.gif&lt;/code&gt; gives several nice examples, including a couple of animations&amp;mdash;one with a quite nice example using the Mandelbrot set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See also &lt;a href=&quot;http://en.wikibooks.org/wiki/R_Programming/Graphics#Animated_plots&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;http://www.oga-lab.net/RGM2/func.php?rd_id=caTools%3aGIF&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;More fine-tuned control and fancier animations&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is an &lt;a href=&quot;http://cran.r-project.org/web/packages/animation/index.html&quot; rel=&quot;nofollow&quot;&gt;animation&lt;/a&gt; package that looks pretty capable. I haven't used it myself, though, so I can't give any real recommendations either way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I &lt;em&gt;have&lt;/em&gt; seen a few good examples of output from this package and they look pretty nice. Perhaps one of the &quot;highlights&quot; is the ability to embed an animation in a PDF.&lt;/p&gt;&#xA;" OwnerUserId="2970" LastActivityDate="2011-05-08T02:17:18.763" CommentCount="2" />
  <row Id="10490" PostTypeId="2" ParentId="6702" CreationDate="2011-05-08T02:23:47.900" Score="0" ViewCount="" Body="&lt;p&gt;To answer my own question, I've moved over to using the 'glmnet' package to fit my multinomial logits, which has the added advantage of using the lasso or elastic net to regularize my independent variables.  glmnet seems to be a much more 'finished' packaged than mlogit, complete with a 'predict' function.&lt;/p&gt;&#xA;" OwnerUserId="2817" LastActivityDate="2011-05-08T02:23:47.900" />
  <row Id="10491" PostTypeId="2" ParentId="73" CreationDate="2011-05-08T02:26:21.970" Score="1" ViewCount="" Body="&lt;p&gt;If you are doing any kind of predictive modeling, &lt;a href=&quot;http://cran.r-project.org/web/packages/caret/index.html&quot; rel=&quot;nofollow&quot;&gt;caret&lt;/a&gt; is a godsend.  Especially combined with the &lt;a href=&quot;http://cran.r-project.org/web/packages/multicore/index.html&quot; rel=&quot;nofollow&quot;&gt;multicore&lt;/a&gt; package, some pretty amazing things are possible.&lt;/p&gt;&#xA;" OwnerUserId="2817" LastActivityDate="2011-05-08T02:26:21.970" CommunityOwnedDate="2011-05-08T02:26:21.970" />
  <row Id="10492" PostTypeId="2" ParentId="3392" CreationDate="2011-05-08T02:32:10.377" Score="3" ViewCount="" Body="&lt;p&gt;Excel is no good for statistics, but it can be wonderful for exploratory data analysis.  &lt;a href=&quot;http://www.r-bloggers.com/getting-into-shape-for-the-sport-of-data-science-screencast-of-talk-by-jeremy-howard-at-melbourne-r-users/&quot; rel=&quot;nofollow&quot;&gt;Take a look at this video&lt;/a&gt; for some particularly interesting techniques.  Excel's ability to conditionally color your data and add in-cell bar charts can give great insight into the structure of your raw data.&lt;/p&gt;&#xA;" OwnerUserId="2817" LastActivityDate="2011-05-08T02:32:10.377" />
  <row Id="10493" PostTypeId="2" ParentId="10486" CreationDate="2011-05-08T02:49:16.147" Score="2" ViewCount="" Body="&lt;p&gt;If you do a good enough job modeling the important predictor variables, you probably will not need to worry as much about the time series aspects (You should probably still test for serial correlation and adjust for it if needed).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most of the times series style association you will see can easily be modeled by things like day of the week, holiday/vacation indicators, and time since the dvd release or some form of advertising or event that spurs rentals of a particular movie.&lt;/p&gt;&#xA;" OwnerUserId="4505" LastEditorUserId="4505" LastEditorDisplayName="" LastEditDate="2011-05-09T15:48:15.743" LastActivityDate="2011-05-09T15:48:15.743" CommentCount="2" />
  <row Id="10494" PostTypeId="2" ParentId="3392" CreationDate="2011-05-08T03:01:16.803" Score="1" ViewCount="" Body="&lt;p&gt;Another good reference source for why you might not want to use excel is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.burns-stat.com/pages/Tutor/spreadsheet_addiction.html&quot; rel=&quot;nofollow&quot;&gt;Spreadsheet addiction&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you find yourself in a situation where you really need to use excel (some accademic departments insist), then I would suggest using the &lt;a href=&quot;http://rcom.univie.ac.at/&quot; rel=&quot;nofollow&quot;&gt;Rexcel plugin&lt;/a&gt;.  This lets you interface using Excel, but uses the R program as the computational engine.  You don't need to know R to use it, you can use drop down menus and dialogs, but you can do a lot more if you do.  Since R is doing the computations they are a lot more trustworthy than Excel and you have much better graphs and boxplots and other graphs missing from excel.  It even works with the automatic cell updating in excel (though that can make things really slow if you have a lot of complex analyses to recompute every time).  It does not fix all the problems from the spreadsheet addiction page, but it is a huge improvement over using straight excel.&lt;/p&gt;&#xA;" OwnerUserId="4505" LastActivityDate="2011-05-08T03:01:16.803" />
  <row Id="10495" PostTypeId="2" ParentId="10484" CreationDate="2011-05-08T03:22:27.187" Score="3" ViewCount="" Body="&lt;p&gt;Here is another approach:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(TeachingDemos)&#xA;&#xA;d &amp;lt;- c(1,2,3,4)&#xA;&#xA;tmpfun &amp;lt;- function(width=1, kernel='gaussian'){&#xA;    plot(density(d, width=width, kernel=kernel))&#xA;}&#xA;&#xA;tmplst &amp;lt;- list( width=list('slider', init=1, from=.5, to=5, resolution=.1),&#xA;    kernel=list('radiobuttons', init='gaussian', values=c('gaussian',&#xA;        &quot;epanechnikov&quot;,&quot;rectangular&quot;,&quot;triangular&quot;,&quot;biweight&quot;,&quot;cosine&quot;,&#xA;        &quot;optcosine&quot;)))&#xA;&#xA;tkexamp( tmpfun, tmplst, plotloc='left' )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="4505" LastActivityDate="2011-05-08T03:22:27.187" />
  <row Id="10496" PostTypeId="2" ParentId="10484" CreationDate="2011-05-08T05:01:33.230" Score="3" ViewCount="" Body="&lt;p&gt;Just for the sake of completeness, if you need this for a class demonstration, I would also mention the &lt;code&gt;manipulate&lt;/code&gt; package which comes with &lt;a href=&quot;http://www.rstudio.org/&quot; rel=&quot;nofollow&quot;&gt;RStudio&lt;/a&gt;. Note that this package is dependent on RStudio interface, so it won't work outside of it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;manipulate&lt;/code&gt; is quite cool because it allows to quickly create some sliders to manipulate any element in the plot. This would allow to do some easy and &lt;em&gt;real-time&lt;/em&gt; demonstration in class.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;manipulate(&#xA;  plot(density(1:10, bw)),&#xA;  bw = slider(0, 10, step = 0.1, initial = 1)) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Other examples &lt;a href=&quot;http://www.rstudio.org/docs/advanced/manipulate&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="582" LastActivityDate="2011-05-08T05:01:33.230" />
  <row Id="10497" PostTypeId="1" AcceptedAnswerId="10548" CreationDate="2011-05-08T05:08:07.753" Score="5" ViewCount="193" Body="&lt;p&gt;I became interested in doing this in C# for my own amusement after reading the following papers:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.cs.washington.edu/homes/brun/pubs/pubs/Kiddon11.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.cs.washington.edu/homes/brun/pubs/pubs/Kiddon11.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also took a look at &lt;a href=&quot;http://www.cs.rpi.edu/academics/courses/fall03/ai/misc/naive-example.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.cs.rpi.edu/academics/courses/fall03/ai/misc/naive-example.pdf&lt;/a&gt; as a concrete example for my implementation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a working implementation now, but I wanted to make sure that I was approaching it properly.  I just want to have a solid Naive Bayes Classifier (unigram).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Problem statement and setup&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using two sets of data, a list of sentences that ARE &quot;that's what she said&quot; and a list of sentences that dont make sense with a &quot;that's what she said&quot; suffix.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Next I parse through all the words in all of the sentences and keep a tally on each word and how many times it was found in each of the two sets, so I might end up with data that looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Word         PositiveCount             NegativeCount&#xA;wet          23                        4&#xA;hard         30                        5&#xA;haiti        0                         20&#xA;to           60                        77&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The I iterate over all of the words and calculate individual $\Pr(\text{Positive}|\text{&amp;lt;word&amp;gt;})$ and $\Pr(\text{Negative}|\text{&amp;lt;word&amp;gt;})$ using the following formula which I found in the above example paper:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;P(Positive|wet) = (23 + p * m) / ((23 + 4) + m)&#xA;P(Negative|wet) = (4 + p * m) / ((23 + 4) + m)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Where &lt;strong&gt;m is the equivilent sample size&lt;/strong&gt; and &lt;strong&gt;p is the a priori estimate&lt;/strong&gt;.  Then in order to check an unknown sentence to see if it is a TWSS I iterate over each word in the sentence and multiply their positive [probibility distributions?] together and multiply all that by p. And do the same for the negative.  Then if the positive number is larger I say that the sentence is a &quot;that's what she said&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Questions&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Currently I am using $p = .5$ for both positive and negative.  I feel like I could be doing something better.  Is this what the Bayesian vs. frequentist thing is about?  How would I go about getting better numbers for $p$?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Also, I am using $m$-estimates for the $\Pr( \text{Yes/No} | \text{&amp;lt;word&amp;gt;} )$.  Should I be doing it this way and what should $m$ be?  What effects does it have to make $m$ larger/smaller?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Super Minor Question: Suggestions on where to get sample data from would be a bonus.&lt;/p&gt;&#xA;" OwnerUserId="4513" LastEditorUserId="4513" LastEditorDisplayName="" LastEditDate="2011-05-08T20:02:33.823" LastActivityDate="2011-05-09T19:59:17.390" Title="Naive Bayes classification for &quot;That's what she said&quot; problem" Tags="&lt;machine-learning&gt;&lt;naive-bayes&gt;" AnswerCount="3" FavoriteCount="2" />
  <row Id="10498" PostTypeId="2" ParentId="8807" CreationDate="2011-05-08T05:51:21.903" Score="0" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;I can recomend you 2 interesting papers to read that are online&lt;/strong&gt;&#xA;1.Streamed Learning: One-Pass SVMs, by Piyush Rai, Hal Daum´e III, Suresh Venkatasubramanian&#xA;2.Streaming k-means approximation, by Nir Ailon&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope it clarifies you a little your ideas&lt;/p&gt;&#xA;" OwnerUserId="1808" LastActivityDate="2011-05-08T05:51:21.903" />
  <row Id="10499" PostTypeId="2" ParentId="10478" CreationDate="2011-05-08T07:10:23.307" Score="3" ViewCount="" Body="&lt;p&gt;I take it that you are mostly interested in regression, as in the cited paper, and not other applications of the $\ell_1$-penalty (graphical lasso, say). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I then believe that some answers can be found in the paper &lt;a href=&quot;http://projecteuclid.org/DPubS?service=UI&amp;amp;version=1.0&amp;amp;verb=Display&amp;amp;handle=euclid.aos/1194461726&quot; rel=&quot;nofollow&quot;&gt;On the “degrees of freedom” of the lasso&lt;/a&gt; by Zou et al. Briefly, it gives an analytic formula for the &lt;em&gt;effective degrees of freedom&lt;/em&gt;, which for the squared error loss allows you to replace CV by an analytic $C_p$-type statistic, say. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another place to look is in &lt;a href=&quot;http://projecteuclid.org/DPubS?service=UI&amp;amp;version=1.0&amp;amp;verb=Display&amp;amp;handle=euclid.aos/1201012958&quot; rel=&quot;nofollow&quot;&gt;The Dantzig selector: Statistical estimation when p is much larger than n&lt;/a&gt; and the discussion papers in the same issue of Annals of Statistics. My understanding is that they solve a problem closely related to lasso regression but with a fixed choice of penalty coefficient. But please take a look at the discussion papers too. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are not interested in prediction, but in model selection, I am not aware of similar results. Prediction optimal models often result in too many selected variables in regression models. In the paper &lt;a href=&quot;http://onlinelibrary.wiley.com/doi/10.1111/j.1467-9868.2010.00740.x/abstract;jsessionid=C4D77523A208F52E5BB4344781600CB3.d03t04&quot; rel=&quot;nofollow&quot;&gt;Stability selection&lt;/a&gt; Meinshausen and Bühlmann presents a subsampling technique more useful for model selection, but it may be too computationally demanding for your needs.  &lt;/p&gt;&#xA;" OwnerUserId="4376" LastActivityDate="2011-05-08T07:10:23.307" CommentCount="1" />
  <row Id="10500" PostTypeId="2" ParentId="10459" CreationDate="2011-05-08T07:16:23.570" Score="2" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Cube_%28film%29&quot; rel=&quot;nofollow&quot;&gt;The Cube&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="4376" LastActivityDate="2011-05-08T07:16:23.570" CommunityOwnedDate="2011-05-08T07:16:23.570" CommentCount="2" />
  <row Id="10501" PostTypeId="1" CreationDate="2011-05-08T07:32:35.350" Score="1" ViewCount="39" Body="&lt;p&gt;It is easy to find a package calculating area under ROC, but is there a package that calculates the area under precision-recall curve?&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2011-05-08T08:17:15.270" Title="Calculating AUPR in R" Tags="&lt;r&gt;&lt;precision-recall&gt;" AnswerCount="1" />
  <row Id="10502" PostTypeId="2" ParentId="10501" CreationDate="2011-05-08T08:17:15.270" Score="1" ViewCount="" Body="&lt;p&gt;A little googling returns one bioc package, &lt;a href=&quot;http://www.bioconductor.org/packages/2.4/bioc/html/qpgraph.html&quot; rel=&quot;nofollow&quot;&gt;qpgraph&lt;/a&gt; (&lt;code&gt;qpPrecisionRecall&lt;/code&gt;), and a cran one, &lt;a href=&quot;http://cran.r-project.org/web/packages/minet/&quot; rel=&quot;nofollow&quot;&gt;minet&lt;/a&gt; (&lt;code&gt;auc.pr&lt;/code&gt;). I have no experience with them, though. Both have been devised to deal with biological networks.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-05-08T08:17:15.270" CommentCount="1" />
  <row Id="10503" PostTypeId="1" CreationDate="2011-05-08T08:44:12.670" Score="2" ViewCount="46" Body="&lt;p&gt;I have a set of data (positive real numbers &amp;lt; 1) in five categories. My aim is to show that the data in the last category is bigger than the other categories for a range of examples (the data set), but the last category isn't necessarily more important than the others.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I thought the best thing is to average each category and then plot the graph of it which works. But I feel this is too simple and is it good if I look at the standard deviation of each category to show that there isn't too much difference and the average-taking is valid?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, I was going to use a scatter plot, is this OK? What would you recommend?&lt;/p&gt;&#xA;" OwnerUserId="4515" LastEditorUserId="4515" LastEditorDisplayName="" LastEditDate="2011-05-08T10:03:27.937" LastActivityDate="2011-05-08T10:03:27.937" Title="Set of data and averaging/standard deviation" Tags="&lt;standard-deviation&gt;&lt;simulation&gt;&lt;averages&gt;" CommentCount="1" />
  <row Id="10504" PostTypeId="2" ParentId="73" CreationDate="2011-05-08T11:53:51.660" Score="1" ViewCount="" Body="&lt;p&gt;lattice, car, MASS, foreign, party.&lt;/p&gt;&#xA;" OwnerUserId="686" LastActivityDate="2011-05-08T11:53:51.660" CommunityOwnedDate="2011-05-08T11:53:51.660" />
  <row Id="10505" PostTypeId="2" ParentId="10497" CreationDate="2011-05-08T12:13:10.863" Score="1" ViewCount="" Body="&lt;p&gt;An interesting source of suggestions for this problem might be the &lt;a href=&quot;http://www.quora.com/How-would-you-programmatically-parse-a-sentence-and-decide-whether-to-answer-thats-what-she-said&quot; rel=&quot;nofollow&quot;&gt;That's What She Said Quora thread&lt;/a&gt;. Specifically, the first comment identifies that it might make sense to use Twitter streams as a source of statements to which there has been a response of &quot;That's What She Said&quot;. Another thing you could do would be to make a simple UI which would take input from twitter, and then ask a human annotator to decide whether &quot;That's What She Said&quot; is an appropriate response.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The UW paper mentions their data sources; I assume (though you didn't mention it specifically) that you're already using this data? (It looks like you can slightly increase the size of the training set from that paper, but not a lot.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another potential source of data to look at might be to analyze television show dialogue for positive examples. (The &quot;&lt;a href=&quot;http://blogs.forbes.com/alexknapp/2011/04/30/a-computer-that-knows-when-to-say-thats-what-she-said/&quot; rel=&quot;nofollow&quot;&gt;A Computer That Knows When To Say “That’s What She Said&lt;/a&gt;&quot; Forbes Article mentions The Office, for example.) Using Hulu's Captions search (&lt;a href=&quot;http://www.hulu.com/labs/caption-search?lang=en&amp;amp;query=that%27s%20what%20she%20said&quot; rel=&quot;nofollow&quot;&gt;search for That's What she said&lt;/a&gt;) identifies 179 examples, though it looks like in the first 10, only 2 are positive training examples, so that may be a somewhat noisy source of data. Officequotes.net appears to have a larger source of potential data, though again, cleaning it up and using it may take some work.&lt;/p&gt;&#xA;" OwnerUserId="1065" LastEditorUserId="1065" LastEditorDisplayName="" LastEditDate="2011-05-09T13:02:03.733" LastActivityDate="2011-05-09T13:02:03.733" CommentCount="2" />
  <row Id="10506" PostTypeId="2" ParentId="10382" CreationDate="2011-05-08T12:17:28.707" Score="0" ViewCount="" Body="&lt;p&gt;You need to check how others have built indexes with similar questions. My guess is that Inglehart and Norris, in &lt;em&gt;&lt;a href=&quot;http://www.hks.harvard.edu/fs/pnorris/Books/Rising%20tide.htm&quot; rel=&quot;nofollow&quot;&gt;Rising Tide&lt;/a&gt;&lt;/em&gt;, have built their Gender Equality/Empowerment index in a way that you can emulate (the construction of the index escapes me but I remember it's in the Technical Appendix).&lt;/p&gt;&#xA;" OwnerUserId="3582" LastActivityDate="2011-05-08T12:17:28.707" />
  <row Id="10507" PostTypeId="2" ParentId="10420" CreationDate="2011-05-08T12:31:33.807" Score="0" ViewCount="" Body="&lt;p&gt;The question is too vague as such, but any answer will depend on your object of study (&lt;a href=&quot;http://www.iq.harvard.edu/blog/sss/archives/2006/02/bayesian_vs_fre.shtml&quot; rel=&quot;nofollow&quot;&gt;example&lt;/a&gt;). You will find tons on the topic over at &lt;a href=&quot;http://www.stat.columbia.edu/~cook/movabletype/archives/2011/04/bayesian_statis_1.html&quot; rel=&quot;nofollow&quot;&gt;Andrew Gelman's blog&lt;/a&gt;, with an April's Fool somewhere in the middle if I recall correctly.&lt;/p&gt;&#xA;" OwnerUserId="3582" LastActivityDate="2011-05-08T12:31:33.807" />
  <row Id="10508" PostTypeId="2" ParentId="10497" CreationDate="2011-05-08T12:58:42.243" Score="1" ViewCount="" Body="&lt;p&gt;Here is a Web site that uses a classifier to determine the gender of an author of text:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;       http://bookblog.net/gender/genie.php&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;There are a number of articles on the subject at the Web site and the composer of the&#xA;site has written a number of articles on the subject as well. There are a number of&#xA;good methods that can be used in the classification&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Bayesian Logistic Regression&lt;/li&gt;&#xA;&lt;li&gt;Linear Discriminant Analysis&lt;/li&gt;&#xA;&lt;li&gt;Quadratic Discriminant Analysis&lt;/li&gt;&#xA;&lt;li&gt;Bayes Classifiers&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Have fun&lt;/p&gt;&#xA;" OwnerUserId="3805" LastActivityDate="2011-05-08T12:58:42.243" />
  <row Id="10509" PostTypeId="1" CreationDate="2011-05-08T13:32:48.573" Score="0" ViewCount="52" Body="&lt;p&gt;I'm not sure which figures to use below in a problem Im trying to solve that involves using the Pearson Correlation formula.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    A  B  C  D  E  F&#xA;Bob 4  5     4  2&#xA;Fra 2  2     2  3  2&#xA;Lee 2  4  3        5&#xA;Cha 5  4     4  1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&quot;Describe a neighbourhood based approach using Pearson Correlation and weighted averages to generate a prediction for whether Bob will like &quot;F&quot;. Your answer should outline the main steps in the neighbourhood based approach and show the steps required to generate the prediction&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I want to find out if Bob will like F... (5=Like 1=Dislike) I understand the Pearson Formula but I'm not sure what other 'User' to compare Bob with, or if I'm supposed to consider the views of all users.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help would be appreciated, thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="4516" LastActivityDate="2011-05-08T13:32:48.573" ClosedDate="2011-05-08T14:08:29.773" Title="Pearson Correlation problem" Tags="&lt;correlation&gt;&lt;homework&gt;" CommentCount="2" />
  <row Id="10510" PostTypeId="1" CreationDate="2011-05-08T16:09:04.040" Score="17" ViewCount="158" Body="&lt;p&gt;In the last few years I've read a number of papers arguing against the use of null hypothesis significance testing in science, but didn't think to keep a persistent list. A colleague recently asked me for such a list, so I thought I'd ask everyone here to help build it. To start things off, here's what I have so far:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pubmed/21077903&quot; rel=&quot;nofollow&quot;&gt;Johansson (2011) &quot;Hail the impossible: p-values, evidence, and likelihood.&quot;&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.dgps.de/fachgruppen/methoden/mpr-online/issue16/art1/article.html&quot; rel=&quot;nofollow&quot;&gt;Haller &amp;amp; Kraus (2002) &quot;Misinterpretation of significance: A problem students share with their teachers.&quot;&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pubmed/18087943&quot; rel=&quot;nofollow&quot;&gt;Wagenmakers (2007) &quot;A practical solution to the pervasive problem of p-values.&quot;&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.springerlink.com/content/al4r791253006174/&quot; rel=&quot;nofollow&quot;&gt;Rodgers (2010) &quot;The epistemology of mathematical and statistical modeling: A quiet methodological revolution.&quot;&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://psycnet.apa.org/journals/amp/65/1/1&quot; rel=&quot;nofollow&quot;&gt;Dixon (1998) &quot;Why scientists value p-values.&quot;&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pubmed/15732688&quot; rel=&quot;nofollow&quot;&gt;Glover &amp;amp; Dixon (2004) &quot;Likelihood ratios: a simple and flexible statistic for empirical psychologists.&quot;&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="364" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-05-08T19:14:10.147" LastActivityDate="2011-05-08T20:27:28.377" CommunityOwnedDate="2011-05-08T16:12:29.843" Title="What are good references containing arguments against null hypothesis significance testing?" Tags="&lt;hypothesis-testing&gt;&lt;statistical-significance&gt;&lt;big-list&gt;&lt;bibliography&gt;" AnswerCount="2" CommentCount="3" FavoriteCount="7" />
  <row Id="10511" PostTypeId="2" ParentId="10510" CreationDate="2011-05-08T16:52:17.817" Score="13" ViewCount="" Body="&lt;p&gt;R. Chris Fraley has taught &lt;a href=&quot;http://www.uic.edu/classes/psych/psych548/fraley/&quot; rel=&quot;nofollow&quot;&gt;a whole course on the history of the debate&lt;/a&gt;.  His summary/conclusion is &lt;a href=&quot;http://www.uic.edu/classes/psych/psych548/fraley/NHSTsummary.htm&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'll copy in the reading list in case the course page ever disappears:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Week 1&#xA;Introduction: What is a Null Hypothesis Significance Test? Facts, Myths, and the State of Our Science&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lyken, D. L. (1991). What’s wrong with psychology? In D. Cicchetti &amp;amp; W.M. Grove (eds.), Thinking Clearly about Psychology, vol. 1: Matters of Public Interest, Essays in honor of Paul E. Meehl (pp. 3 – 39). Minneapolis, MN: University of Minnesota Press.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Week 2&#xA;Early Criticisms of NHST&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Meehl, P. E. (1967). Theory-testing in psychology and physics: A methodological paradox. Philosophy of Science, 34, 103-115.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Meehl, P. E. (1978). Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. Journal of Consulting and Clinical Psychology, 46, 806-834.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rozeboom, W. W. (1960). The fallacy of the null hypothesis significance test. Psychological Bulletin, 57, 416-428.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bakan, D. (1966). The test of significance in psychological research. Psychological Bulletin, 66, 423-437. [optional]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Week 3&#xA;Contemporary Criticisms of NHST&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cohen, J. (1994). The earth is round (p &amp;lt; .05). American Psychologist, 49, 997-1003.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Gigerenzer, G. (1993). The superego, the ego, and the id in statistical reasoning. In G. Keren &amp;amp; C. Lewis (Eds.), A handbook for data analysis in the behavioral sciences: Methodological issues (pp. 311-339). Hillsdale, NJ: Lawrence Erlbaum Associates.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Schmidt, F. L. &amp;amp; Hunter, J. E. (1997). Eight common but false objections to the discontinuation of significance testing in the analysis of research data. In Lisa A. Harlow, Stanley A. Mulaik, and James H. Steiger (Eds.) What if there were no significance tests? (pp. 37-64). Mahwah, NJ: Lawrence Erlbaum Associates.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Oakes, M. (1986). Statistical inference: A commentary for the social and behavioral sciences. New York: Wiley. (Chapter 2 [A Critique of Significance Tests]) [optional]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Week 4&#xA;Rebuttal: Advocates of NHST Come to Its Defense&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Frick, R. W. (1996). The appropriate use of null hypothesis testing. Psychological Methods, 1, 379-390.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hagen, R. L. (1997). In praise of the null hypothesis statistical test. American Psychologist, 52, 15-24.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Wilkinson, L., &amp;amp; the Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594-604.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Wainer, H. (1999). One cheer for null hypothesis significance testing. Psychological Methods, 6, 212-213.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Mulaik, S. A., Raju, N. S., &amp;amp; Harshman, R. A. (1997). There is a time and place for significance testing. In Lisa A. Harlow, Stanley A. Mulaik, and James H. Steiger , Eds. What if there were no significance tests? (pp. 65-116). Mahwah, NJ: Lawrence Erlbaum Associates. [optional]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Week 5&#xA;Rebuttal: Advocates of NHST Come to Its Defense&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Abelson, R. P. (1997). On the surprising longevity of flogged horses: Why there is a case for the significance test. Psychological Science, 8, 12-15.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Krueger, J. (2001). Null hypothesis significance testing: On the survival of a flawed method. American Psychologist, 56, 16-26.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Scarr, S. (1997). Rules of evidence: A larger context for the statistical debate. Psychological Science, 8, 16-17.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Greenwald, A. G., Gonzalez, R., Harris, R. J., &amp;amp; Guthrie, D. (1996). Effect sizes and p values: What should be reported and what should be replicated? Psychophysiology, 33, 175-183.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Nickerson, R. S. (2000). Null hypothesis significance testing: A review of an old and continuing controversy. Psychological Methods, 5, 241-301. [optional]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Harris, R. J. (1997). Significance tests have their place. Psychological Science, 8, 8-11. [optional]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Week 6 &#xA;Effect Size&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rosenthal, R. (1984). Meta-analytic procedures for social research. Beverly Hills, CA: Sage. [Ch. 2, Defining Research Results]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Chow, S. L. (1988). Significance test or effect size? Psychological Bulletin, 103, 105-110.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Abelson, R. P. (1985). A variance explanation paradox: When a little is a lot. Psychological Bulletin, 97, 129-133. [optional]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Week 7 &#xA;Statistical Power&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hallahan, M., &amp;amp; Rosenthal, R. (1996). Statistical power: Concepts, procedures, and applications. Behaviour Research and Therapy, 34, 489-499.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sedlmeier, P., &amp;amp; Gigerenzer, G. (1989). Do studies of statistical power have an effect on the power of studies? Psychological Bulletin, 105, 309-316. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cohen, J. (1962). The statistical power of abnormal-social psychological research: A review. Journal of Abnormal and Social Psychology, 65, 145-153. [optional]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maddock, J. E., Rossi, J. S. (2001). Statistical power of articles published in three health-psychology related journals. Health Psychology, 20, 76-78. [optional]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thomas, L. &amp;amp; Juanes, F. (1996). The importance of statistical power analysis: An example from Animal Behaviour. Animal Behaviour, 52, 856-859. [optional]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rossi, J. S. (1990). Statistical power of psychological research: What have we gained in 20 years? Journal of Consulting and Clinical Psychology, 58, 646-656. [optional]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tukey, J. W. (1969). Analyzing data: Sanctification or detective work? American Psychologist, 24, 83-91. [optional]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Week 8&#xA;Confidence Intervals and Significance testing&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Gardner, M. J., &amp;amp; D. G. Altman. 1986. Confidence intervals rather than P values: Estimation rather than hypothesis testing. British Medical Journal, 292, 746-750.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cumming, G., &amp;amp; Finch, S. (2001). A primer on understanding, use, and calculation of confidence intervals that are based on central and noncentral distributions. Educational and Psychological Measurement, 61, 532-574.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Loftus, G. R., &amp;amp; Masson, M.E.J. (1994). Using confidence intervals in within-subject designs. Psychonomic Bulletin and Review, 1, 476-490.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Week 9 [note: we are skipping this section] &#xA;Theoretical Modeling: Developing Formal Models of Natural Phenomena&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Haefner, J. W. (1996). Modeling biological systems: Principles and applications. New York: International Thomson Publishing. (Chapters 1 [Models of Systems] &amp;amp; 2 [The Modeling Process])&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Loehlin, J. C. (1992). Latent variable models: An introduction to factor, path, and structural analysis. Hillsdale, NJ: Lawrence Erlbaum Associates. (Chapter 1 [Path models in factor, path and structural analysis], p. 1-18]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Grant, D. A. (1962). Testing the null hypothesis and the strategy of investigating theoretical models. Psychological Review, 69, 54-61. [optional]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Binder, A. (1963). Further considerations on testing the null hypothesis and the strategy and tactics of investigating theoretical models. Psychological Review, 70, 107-115. [optional]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edwards, W. (1965). Tactical note on the relations between scientific and statistical hypotheses. Psychological Bulletin, 63, 400-402. [optional]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Week 10&#xA;What is the Meaning of Probability? Controversy Concerning Relative Frequency and Subjective Probability&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Salsburg, D. (2001). The lady tasting tea: How statistics revolutionized science in the twentieth century. New York: W. H. Freeman. (Chapters 10, 11, &amp;amp; 12)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Oakes, M. (1986). Statistical inference: A commentary for the social and behavioral sciences. New York: Wiley. (Chapters 4, 5, &amp;amp; 6)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Pruzek, R. M. (1997). An introduction to Bayesian inference and its applications. In Lisa A. Harlow, Stanley A. Mulaik, and James H. Steiger , Eds. What if there were no significance tests? (pp. 287-318). Mahwah, NJ: Lawrence Erlbaum Associates.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rindskoph, D. M. (1997). Testing &quot;small,&quot; not null, hypothesis: Classical and Bayesian Approaches. In Lisa A. Harlow, Stanley A. Mulaik, and James H. Steiger (Eds). What if there were no significance tests? (pp. 319-332). Mahwah, NJ: Lawrence Erlbaum Associates.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edwards, W., Lindman, H., Savage, L. J. (1963). Bayesian statistical inference for psychological research. Psychological Review, 70, 193-242. [optional]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Week 11&#xA;Theory Appraisal: Philosophy of Science and the Testing and Amending of Theories&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Meehl, P. E. (1990). Appraising and amending theories: The strategy of Lakatosian defense and two principles that warrant it. Psychological Inquiry, 1, 108-141.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Roberts, S. &amp;amp; Pashler, H. (2000). How persuasive is a good fit? A comment on theory testing. Psychological Review, 107, 358-367.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Week 12&#xA;Theory Appraisal: Philosophy of Science and the Testing and Amending of Theories&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Urbach, P. (1974). Progress and degeneration in the &quot;IQ debate&quot; (I). British Journal of Philosophy of Science, 25, 99-125.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Serlin, R. C. &amp;amp; Lapsley, D. K. (1985). Rationality in psychological research: The good-enough principle. American Psychologist, 40, 73-83.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Dar, R. (1987). Another look at Meehl, Lakatos, and the scientific practices of psychologists. American Psychologist, 42, 145-151.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Gholson, B. &amp;amp; Barker, P. (1985). Kuhn, Lakatos, &amp;amp; Laudan: Applications in the history of physics and psychology. American Psychologist, 40, 755-769. [optional]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Faust, D., &amp;amp; Meehl, P. E. (1992). Using scientific methods to resolve questions in the history and philosophy of science: Some illustrations. Behavior Therapy, 23, 195-211. [optional]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Urbach, P. (1974). Progress and degeneration in the &quot;IQ debate&quot; (II). British Journal of Philosophy of Science, 25, 235-259. [optional]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Salmon, W. C. (1973, May). Confirmation. Scientific American, 228, 75-83. [optional]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Meehl, P. E. (1993). Philosophy of science: Help or hindrance? Psychological Reports, 72, 707-733. [optional]&#xA;Manicas. P. T., &amp;amp; Secord, P. F. (1983). Implications for psychology of the new philosophy of science. American Psychologist, 38, 399-413. [optional]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Week 13&#xA;Has the NHST Tradition Undermined a Non-Biased, Cumulative Knowledge Base in Psychology?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cooper, H., DeNeve, K., &amp;amp; Charlton, K. (1997). Finding the missing science: The fate of studies submitted for review by a human subjects committee. Psychological Methods, 2, 447-452.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Schmidt, F. L. (1996). Statistical significance testing and cumulative knowledge in psychology: Implications for training of researchers. Psychological Methods, 1, 115-129.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Greenwald, A. G. (1975). Consequences of prejudice against the null hypothesis. Psychological Bulletin, 82, 1-20.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Berger, J. O. &amp;amp; Berry, D. A. (1988). Statistical analysis and illusion of objectivity. American Scientist, 76, 159-165.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Week 14&#xA;Replication and Scientific Integrity&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Smith, N. C. (1970). Replication studies: A neglected aspect of psychological research. American Psychologist, 25, 970-975.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sohn, D. (1998). Statistical significance and replicability: Why the former does not presage the latter. Theory and Psychology, 8, 291-311.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Meehl, P. E. (1990). Why summaries of research on psychological theories are often uninterpretable. Psychological Reports, 66, 195-244.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Platt, J. R. (1964). Strong Inference. Science, 146, 347-353.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Feynman, R. L. (1997). Surely you’re joking, Mr. Feynman! New York: W. W. Norton. (Chapter: Cargo-cult science).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rorer, L. G. (1991). Some myths of science in psychology. In D. Cicchetti &amp;amp; W.M. Grove (eds.), Thinking Clearly about Psychology, vol. 1: Matters of Public Interest, Essays in honor of Paul E. Meehl (pp. 61 – 87). Minneapolis, MN: University of Minnesota Press. [optional]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lindsay, R. M. &amp;amp; Ehrenberg, A. S. C. (1993). The design of replicated studies. The American Statistician, 47, 217-228. [optional]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Week 15&#xA;Quantitative Thinking: Why We Need Mathematics (and not NHST per se) in Psychological Science&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Aiken, L. S., West, S. G., Sechrest, L., &amp;amp; Reno, R. R. (1990). Graduate training in statistics, methodology, and measurement in psychology: A survey of Ph.D. programs in North America. American Psychologist, 45, 721-734.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Meehl, P. E. (1998, May). The power of quantitative thinking. Invited address as recipient of the James McKeen Cattell Award at the annual meeting of the American Psychological Society, Washington, DC.&lt;/p&gt;&#xA;" OwnerUserId="3748" LastActivityDate="2011-05-08T16:52:17.817" CommunityOwnedDate="2011-05-08T16:52:17.817" CommentCount="2" />
  <row Id="10513" PostTypeId="2" ParentId="3719" CreationDate="2011-05-08T17:46:41.447" Score="1" ViewCount="" Body="&lt;p&gt;Pearson correlations can be converted into 'z' scores. Those z scores may be averaged and their median converted back to the composit correlation&lt;/p&gt;&#xA;" OwnerUserId="4518" LastActivityDate="2011-05-08T17:46:41.447" />
  <row Id="10514" PostTypeId="2" ParentId="10363" CreationDate="2011-05-08T17:51:27.797" Score="0" ViewCount="" Body="&lt;p&gt;r square of 97.2&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Estimation/Diagnostic Checking for Variable Y    Y&lt;br&gt;&#xA;                                            X1   AAS&lt;br&gt;&#xA;                                            X2   BB&lt;br&gt;&#xA;                                            X3   BBS&lt;br&gt;&#xA;                                            X4   CC                             &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Number of Residuals (R)        =n                         25&lt;br&gt;&#xA; Number of Degrees of Freedom   =n-m                       20&lt;br&gt;&#xA; Residual Mean                  =Sum R / n               -.141873E-05&lt;br&gt;&#xA; Sum of Squares                 =Sum R**2                 .775723E+07&lt;br&gt;&#xA; Variance                       =SOS/(n)                  310289.&lt;br&gt;&#xA; Adjusted Variance              =SOS/(n-m)                387861.&lt;br&gt;&#xA; Standard Deviation RMSE        =SQRT(Adj Var)            622.785&lt;br&gt;&#xA; Standard Error of the Mean     =Standard Dev/ (n-m)      139.259&lt;br&gt;&#xA; Mean / its Standard Error      =Mean/SEM                -.101877E-07&lt;br&gt;&#xA; Mean Absolute Deviation        =Sum(ABS(R))/n            455.684&lt;br&gt;&#xA; AIC Value ( Uses var )         =nln  +2m                 326.131&lt;br&gt;&#xA; SBC Value ( Uses var )         =nln  +m*lnn              332.226&lt;br&gt;&#xA; BIC Value ( Uses var )         =see Wei p153             340.388&lt;br&gt;&#xA; R Square                       =                         .972211&lt;br&gt;&#xA; Durbin-Watson Statistic        =[-A(T-1)]*&lt;em&gt;2/A&lt;/em&gt;*2        1.76580        &lt;/p&gt;&#xA;&#xA;&lt;p&gt;**&lt;br&gt;&#xA;          MODEL COMPONENT       LAG    COEFF     STANDARD      P       T&lt;br&gt;&#xA;  #                            (BOP)              ERROR      VALUE   VALUE      &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1CONSTANT                         -.381E+04   466.       .0000    -8.18&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;INPUT SERIES X1  AAS   AA SQUARED                                                       &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;2Omega (input) -Factor #  1    0   .983       .410E-01   .0000    23.98&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;INPUT SERIES X2  BB    BB AS GIVEN                                                       &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;3Omega (input) -Factor #  2    0   108.       14.9       .0000     7.27&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;INPUT SERIES X3  BBS   BB SQUARED                                                      &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;4Omega (input) -Factor #  3    0  -.577       .147       .0008    -3.93&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;INPUT SERIES X4  CC    CC AS GIVEN                                                       &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;5Omega (input) -Factor #  4    0   49.9       4.67       .0000    10.67&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.stack.imgur.com/W3Cnx.jpg&quot; alt=&quot; Residual PLOT&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="3382" LastActivityDate="2011-05-08T17:51:27.797" />
  <row Id="10515" PostTypeId="2" ParentId="10420" CreationDate="2011-05-08T17:52:31.057" Score="0" ViewCount="" Body="&lt;p&gt;In my experience, they are both useful in different situations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where you can confidently say that the data come from a specified probability model, then parametric statistics will usually give you more information. However, they can also lead to significantly biased conclusions if the wrong model is used. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Non-parametric statistics, on the other hand, require fewer assumptions about the data, and consequently will prove better in situations where the true distribution is unknown or cannot be easily approximated using a probability distribution.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;All in all, I prefer making as few assumptions as possible, so I tend to prefer non-parametric approaches. &lt;/p&gt;&#xA;" OwnerUserId="656" LastActivityDate="2011-05-08T17:52:31.057" CommentCount="2" />
  <row Id="10516" PostTypeId="2" ParentId="10423" CreationDate="2011-05-08T18:51:15.643" Score="0" ViewCount="" Body="&lt;p&gt;One of my absolutely most valuable books over the years has been &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0126913609&quot; rel=&quot;nofollow&quot;&gt;Tinsley and Brown's Handbook&lt;/a&gt;.  There are many places in the book where this topic is discussed, by different contributing authors.&lt;/p&gt;&#xA;" OwnerUserId="2669" LastActivityDate="2011-05-08T18:51:15.643" />
  <row Id="10517" PostTypeId="1" CreationDate="2011-05-08T18:56:08.370" Score="1" ViewCount="92" Body="&lt;p&gt;Given a sample data set of floating point numbers, how do we determine its probability distribution and prove it? &#xA;Also generate random numbers of the same distributions thereafter.&lt;/p&gt;&#xA;" OwnerUserId="4520" LastActivityDate="2011-05-11T13:06:21.467" Title="Identify probability distributions" Tags="&lt;distributions&gt;&lt;probability&gt;&lt;modeling&gt;&lt;dataset&gt;&lt;simulation&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="10518" PostTypeId="2" ParentId="10510" CreationDate="2011-05-08T19:03:21.840" Score="3" ViewCount="" Body="&lt;p&gt;These are excellent references.  I have a perhaps useful handout at &lt;a href=&quot;http://biostat.mc.vanderbilt.edu/wiki/pub/Main/FHHandouts/whyBayesian.pdf&quot; rel=&quot;nofollow&quot;&gt;http://biostat.mc.vanderbilt.edu/wiki/pub/Main/FHHandouts/whyBayesian.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="4253" LastActivityDate="2011-05-08T19:03:21.840" CommunityOwnedDate="2011-05-08T19:03:21.840" />
  <row Id="10519" PostTypeId="1" AcceptedAnswerId="10524" CreationDate="2011-05-08T19:31:29.433" Score="2" ViewCount="105" Body="&lt;p&gt;If I have many time series that I'd like to compare to see if there are relationships between the variables, (I have several dependent variables and many more independent variables) how might I go about doing this (I'm working in R, just fyi)? I haven't really found too many examples seeking to compare and explore the relationships between many variables. In particular, I'd like to see if the variation in my independent time series drives variation in the dependent time series, and I'm so new to statistics (and R) that I'm really not sure how to approach this problem. Here is some sample data (I know there are 2 missing values, and I may choose to treat Y2 as an independent variable since I think Y2 and Y1 may be correlated.  The column labeled &quot;covariate&quot; is years since last flood, because I think this might be important too):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;year Y1 Y2  X1  X2  X3      X4  X5  covariate&#xA;1    40  92 0   0   20.6    91  503 3&#xA;2    54  65 0   0   21.7    33  175 4&#xA;3    59  75 1   1   22.2    34  94  5&#xA;4    68  53 8   9   22.2    24  86  6&#xA;5           5   20  20.6    5   185 7&#xA;6    76  65 8   13  22.2    32  119 8&#xA;7    76  55 16  18  23.3    0   153 9&#xA;8    82  58 18  2   24.4    19  0   1&#xA;9    60  57 28  24  23.33   0   223 2&#xA;10   58  46 18  3   22.78   0   184 3&#xA;11   49  48 2   1   23.33   0   110 4&#xA;12   28  76 0   3   22.78   0   213 5&#xA;13   56  61 0   1   22.78   12  123 6&#xA;14   105 53 56  24  23.33   0   122 7&#xA;15   99  43 28  13  24.44   0   154 8&#xA;16   119 47 46  35  23.33   0   182 9&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Any guidance would be much appreciated as I'm not really sure how to proceed here.&lt;/p&gt;&#xA;" OwnerUserId="4521" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-08T20:43:07.527" LastActivityDate="2011-05-08T22:37:39.683" Title="Comparing multiple time series in R" Tags="&lt;r&gt;&lt;time-series&gt;&lt;multiple-comparisons&gt;" AnswerCount="1" CommentCount="0" FavoriteCount="1" />
  <row Id="10520" PostTypeId="2" ParentId="10363" CreationDate="2011-05-08T19:40:32.340" Score="10" ViewCount="" Body="&lt;p&gt;$R^2$ alone is not a good measure of goodness of fit, but let's not get into that here except to observe that &lt;em&gt;parsimony&lt;/em&gt; is valued in modeling.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To that end, note that standard techniques of EDA and regression (but &lt;em&gt;not&lt;/em&gt; stepwise or other automated procedures) suggest using a linear model in the form&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\sqrt{f} = a + b*c + a*b*c + \text{constant} + \text{error}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using OLS, this does achieve an $R^2$ above 0.99.  Heartened by such a result, one is tempted to square both sides and regress $f$ on $a$, $b*c$, $a*b*c$, &lt;em&gt;and&lt;/em&gt; all their squares and products.  This immediately produces a model&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$f = a^2 + b*c +  \text{constant} + \text{error}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;with a root MSE of under 34 and an &lt;strong&gt;adjusted $R^2$ of 0.9999&lt;/strong&gt;.  The estimated coefficients of 1.0112 and 0.988 suggest the data may be artificially generated with the formula&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$f = a^2 + b*c + 50$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;plus a little normally distributed error of SD approximately equal to 50.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Edit&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;In response to @knorv's hints, I continued the analysis.  To do so I used the techniques that had been successful so far, beginning with inspecting scatterplot matrices of the residuals against the original variables.  Sure enough, there was a clear indication of correlation between $a$ and the residuals (even though OLS regression of $f$ against $a$, $a^2$, and $b*c$ did &lt;em&gt;not&lt;/em&gt; indicate $a$ was &quot;significant&quot;).  Continuing in this vein I explored all correlations between the quadratic terms $a^2, \ldots, e^2, a*b, a*c, \ldots, d*e$ and the new residuals and found a tiny but highly significant relationship with $b^2$.  &quot;Highly significant&quot; means that all this snooping involved looking at some 20 different variables, so my criterion for significance on this fishing expedition was approximately 0.05/20 = 0.0025: anything less stringent could easily be an artifact of the probing for fits.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This has something of the flavor of a physical model in that we expect, and therefore search for, relationships with &quot;interesting&quot; and &quot;simple&quot; coefficients.  So, for instance, seeing that the estimated coefficient of $b^2$ was -0.0092 (between -0.005 and -0.013 with 95% confidence), I elected to use -1/100 for it.  If this were some other dataset, such as observations of a social or political system, I would make no such changes but just use the OLS estimates as-is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyway, an improved fit is given by&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$f = a + a^2 + b*c - b^2/100 + 30.5 + \text{error}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;with mean residual $0$, standard deviation 26.8, all residuals between -50 and +43, and no evidence of non-normality (although with such a small dataset the errors could even be uniformly distributed and one couldn't really tell the difference).  The reduction in residual standard deviation from around 50 to around 25 would often be expressed as &quot;explaining 75% of the residual variance.&quot;&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;I make no claim that this is &lt;em&gt;the&lt;/em&gt; formula used to generate the data&lt;/strong&gt;.  The residuals are large enough to allow some fairly large changes in a few of the coefficients.  For instance, 95% CIs for the coefficients of $a$, $b^2$, and the constant are [-0.4, 2.7], [-0.013, -0.003], and [-7, 61] respectively.  The point is that if any random error has actually been introduced in the data-generation procedure (and that is true of &lt;em&gt;all&lt;/em&gt; real-world data), that would preclude definitive identification of the coefficients (and even of all the variables that might be involved).  That's not a limitation of statistical methods: it's just a mathematical fact.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;BTW, using robust regression I can fit the model&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$f = 1.0103 a^2 + 0.99493 b*c - 0.007 b^2 + 46.78 + \text{error}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;with residual SD of 27.4 and all residuals between -51 and +47: essentially as good as the previous fit but with one less variable.  It is more parsimonious in that sense, but less parsimonious in the sense that I haven't rounded the coefficients to &quot;nice&quot; values.  Nevertheless, this is the form I would usually favor in a regression analysis absent any rigorous theories about what kinds of values the coefficients ought to have and which variables ought to be included.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;It is likely that additional strong relationships are lurking here&lt;/strong&gt;, but they would have to be fairly complicated.  Incidentally, taking data whose original SD is 3410 and reducing their variation to residual with an SD of 27 is a &lt;strong&gt;99.99384%&lt;/strong&gt; reduction (the $R^2$ of this new fit).  One would continue looking for additional effects only if the residual SD is too large for the intended purpose.  In the absence of any purpose besides second-guessing the OP, it's time to stop. &lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-05-09T22:05:01.390" LastActivityDate="2011-05-09T22:05:01.390" CommentCount="7" />
  <row Id="10522" PostTypeId="2" ParentId="10382" CreationDate="2011-05-08T20:36:37.933" Score="0" ViewCount="" Body="&lt;p&gt;People often use the phrase &quot;Likert scale&quot; erroneously, not realizing that it originally described the coherent method Rensis Likert developed to do just what you're describing.  Key steps are &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Seeing how well each item correlates with &quot;the whole&quot;--the average of all other items&lt;/li&gt;&#xA;&lt;li&gt;Checking variability, since an item with low variability is unlikely to contribute much information to such a scale&lt;/li&gt;&#xA;&lt;li&gt;Using steps 1 and 2, as well as tests of Cronbach's alpha (and sometimes factor analysis, as @Fr. has said), to narrow down the list of items to be kept&lt;/li&gt;&#xA;&lt;li&gt;Averaging the items ultimately selected to compute the scale score for each person. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;An excellent and very accessible guide to this process can be found in Paul Spector's Summated Rating Scale Construction, a little green Sage book that's available for about $18 new. &lt;/p&gt;&#xA;" OwnerUserId="2669" LastActivityDate="2011-05-08T20:36:37.933" />
  <row Id="10523" PostTypeId="1" AcceptedAnswerId="10525" CreationDate="2011-05-08T20:47:43.803" Score="1" ViewCount="38" Body="&lt;p&gt;Hello,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've been trying to replicate the results in this online calculator:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.raosoft.com/samplesize.html&quot; rel=&quot;nofollow&quot;&gt;http://www.raosoft.com/samplesize.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, it seems that I am missing something. Exactly how do I solve the margin of error when all the other variables (sample size, confidence level, distribution and population*) are known?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've tried to use the formulas on the page but I cannot arrive at the same result. Also using the formulas on bottom of &lt;a href=&quot;http://www.resolutions.co.nz/sample_sizes.htm&quot; rel=&quot;nofollow&quot;&gt;http://www.resolutions.co.nz/sample_sizes.htm&lt;/a&gt; gives me somewhat different results. What am I missing?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;*) = a small population so it cannot be assumed infinite.&lt;/p&gt;&#xA;" OwnerUserId="3401" LastEditorUserId="3401" LastEditorDisplayName="" LastEditDate="2011-05-08T20:56:19.740" LastActivityDate="2011-05-08T21:27:25.250" Title="Margin-of-error calculation in survey" Tags="&lt;confidence-interval&gt;&lt;standard-error&gt;" AnswerCount="1" />
  <row Id="10524" PostTypeId="2" ParentId="10519" CreationDate="2011-05-08T20:53:30.243" Score="4" ViewCount="" Body="&lt;p&gt;here goes ... These are the steps that are required to form your analysis. Simply reproduce them in R or whatever tools you have available. The reason the following exercise is daunting is because the statistical problem you are asking is &quot;daunting&quot; and one needs to &quot;up-armor&quot; their solutions skills/procedures.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;pre-whiten each of your 7 candidate regressors in order to identify an initial Tranfer Function model of the form&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  MODEL COMPONENT       LAG    COEFF     STANDARD      P       T        &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h1&gt;(BOP)              ERROR      VALUE   VALUE&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;1CONSTANT                         -258.       330.      0.4568    -0.78&lt;/p&gt;&#xA;&#xA;&lt;p&gt;INPUT SERIES X1  Y2                                                           &lt;/p&gt;&#xA;&#xA;&lt;p&gt;2Omega (input) -Factor #  1    0  0.635      0.583      0.3079     1.09&lt;/p&gt;&#xA;&#xA;&lt;p&gt;INPUT SERIES X2  X1                                                           &lt;/p&gt;&#xA;&#xA;&lt;p&gt;3Omega (input) -Factor #  2    1  0.544      0.563      0.3623     0.97&lt;/p&gt;&#xA;&#xA;&lt;p&gt;INPUT SERIES X3  X2                                                           &lt;/p&gt;&#xA;&#xA;&lt;p&gt;4Omega (input) -Factor #  3    0   1.60      0.485      0.0110     3.29&lt;/p&gt;&#xA;&#xA;&lt;p&gt;INPUT SERIES X4  X3                                                           &lt;/p&gt;&#xA;&#xA;&lt;p&gt;5Omega (input) -Factor #  4    0   11.7       14.8      0.4493     0.80&lt;/p&gt;&#xA;&#xA;&lt;p&gt;INPUT SERIES X5  X4                                                           &lt;/p&gt;&#xA;&#xA;&lt;p&gt;6Omega (input) -Factor #  5    0  0.491      0.699      0.5020     0.70&lt;/p&gt;&#xA;&#xA;&lt;p&gt;INPUT SERIES X6  X5                                                           &lt;/p&gt;&#xA;&#xA;&lt;p&gt;7Omega (input) -Factor #  6    0 -0.151      0.164      0.3852    -0.92&lt;/p&gt;&#xA;&#xA;&lt;p&gt;INPUT SERIES X7  X6                                                           &lt;/p&gt;&#xA;&#xA;&lt;p&gt;8Omega (input) -Factor #  7    0   2.40       1.57      0.1639     1.53&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Then do Interventon Detection to extract any anomalies in the data finding three &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;         :  NEWLY IDENTIFIED VARIABLE   X8   I~P00010       10    PULSE     &#xA;         :  NEWLY IDENTIFIED VARIABLE   X9   I~P00014       14    PULSE     &#xA;         :  NEWLY IDENTIFIED VARIABLE   X10  I~P00002        2    PULSE     &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This leads to an augmented model :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;      MODEL COMPONENT       LAG    COEFF     STANDARD      P       T        &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;#                            (BOP)              ERROR      VALUE   VALUE      &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1CONSTANT                         -33.9       191.      0.8662    -0.18&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;INPUT SERIES X1  Y2                                                           &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;2Omega (input) -Factor #  1    0   1.08      0.324      0.0207     3.34&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;INPUT SERIES X2  X1                                                           &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;3Omega (input) -Factor #  2    1  0.867      0.325      0.0446     2.67&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;INPUT SERIES X3  X2                                                           &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;4Omega (input) -Factor #  3    0   2.06      0.248      0.0004     8.29&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;INPUT SERIES X4  X3                                                           &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;5Omega (input) -Factor #  4    0   1.68       8.47      0.8503     0.20&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;INPUT SERIES X5  X4                                                           &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;6Omega (input) -Factor #  5    0 -0.239      0.455      0.6218    -0.53&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;INPUT SERIES X6  X5                                                           &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;7Omega (input) -Factor #  6    0 -0.363      0.102      0.0161    -3.56&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;INPUT SERIES X7  X6                                                           &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;8Omega (input) -Factor #  7    0   3.21      0.712      0.0064     4.50&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;INPUT SERIES X8  I~P00010       10    PULSE                                   &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;9Omega (input) -Factor #  8    0   30.4       6.52      0.0055     4.66&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;INPUT SERIES X9  I~P00014       14    PULSE                                   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;10Omega (input) -Factor #  9    0   14.7       6.42      0.0709     2.29&lt;/p&gt;&#xA;&#xA;&lt;p&gt;INPUT SERIES X 10 I~P00002        2    PULSE                                  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;11Omega (input) -Factor # 10    0   39.6       8.22      0.0048     4.81&lt;/p&gt;&#xA;&#xA;&lt;p&gt;which is over-specified thus we must step-down and obtain&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;      MODEL COMPONENT       LAG    COEFF     STANDARD      P       T        &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;#                            (BOP)              ERROR      VALUE   VALUE      &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1CONSTANT                         -2.32       7.25      0.7584    -0.32&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;INPUT SERIES X1  Y2                                                           &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;2Omega (input) -Factor #  1    0   1.10      0.872E-01  0.0000    12.58&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;INPUT SERIES X2  X1                                                           &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;3Omega (input) -Factor #  2    1   1.04      0.103      0.0000    10.10&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;INPUT SERIES X3  X2                                                           &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;4Omega (input) -Factor #  3    0   2.04      0.199      0.0000    10.22&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;INPUT SERIES X4  X5                                                           &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;5Omega (input) -Factor #  4    0 -0.335      0.301E-01  0.0000   -11.13&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;INPUT SERIES X5  X6                                                           &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;6Omega (input) -Factor #  5    0   2.84      0.663      0.0037     4.28&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;INPUT SERIES X6  I~P00010       10    PULSE                                   &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;7Omega (input) -Factor #  6    0   27.8       6.48      0.0036     4.29&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;INPUT SERIES X7  I~P00014       14    PULSE                                   &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;8Omega (input) -Factor #  7    0   21.3       6.17      0.0107     3.45&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;INPUT SERIES X8  I~P00002        2    PULSE                                   &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;9Omega (input) -Factor #  8    0   32.3       5.99      0.0010     5.39&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which now will suggest additional structure in the X's via cross-correlative tests between the current model residuals and the residuals from the pre-whitened X's .... that had previously remained unidentified.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;      MODEL COMPONENT       LAG    COEFF     STANDARD      P       T        &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;#                            (BOP)              ERROR      VALUE   VALUE      &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1CONSTANT                         -25.8       6.12      0.0083    -4.22&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;INPUT SERIES X1  Y2                                                           &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;2Omega (input) -Factor #  1    0   1.11      0.598E-01  0.0000    18.50&#xA;3                              1 -0.211      0.516E-01  0.0095    -4.09&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;INPUT SERIES X2  X1                                                           &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;4Omega (input) -Factor #  2    1   1.05      0.692E-01  0.0000    15.23&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;INPUT SERIES X3  X2                                                           &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;5Omega (input) -Factor #  3    0   2.28      0.143      0.0000    15.99&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;INPUT SERIES X4  X5                                                           &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;6Omega (input) -Factor #  4    0 -0.311      0.193E-01  0.0000   -16.17&#xA;7                              1 -0.693E-01  0.103E-01  0.0011    -6.72&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;INPUT SERIES X5  X6                                                           &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;8Omega (input) -Factor #  5    0   2.19      0.466      0.0054     4.70&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;INPUT SERIES X6  I~P00010       10    PULSE                                   &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;9Omega (input) -Factor #  6    0   19.8       4.31      0.0059     4.59&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;INPUT SERIES X7  I~P00014       14    PULSE                                   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;10Omega (input) -Factor #  7    0   18.6       4.21      0.0068     4.43&lt;/p&gt;&#xA;&#xA;&lt;p&gt;culminating in the final model&lt;/p&gt;&#xA;&#xA;&lt;p&gt;MODEL STATISTICS AND EQUATION FOR THE CURRENT EQUATION (DETAILS FOLLOW).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Estimation/Diagnostic Checking for Variable Y    Y1&lt;br&gt;&#xA;                                            X1   Y2&lt;br&gt;&#xA;                                            X2   X1&lt;br&gt;&#xA;                                            X3   X2&lt;br&gt;&#xA;                                            X4   X5&lt;br&gt;&#xA;                                            X5   X6&lt;br&gt;&#xA;             :  NEWLY IDENTIFIED VARIABLE   X6   I~P00010       10    PULSE&lt;br&gt;&#xA;             :  NEWLY IDENTIFIED VARIABLE   X7   I~P00014       14    PULSE     &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Number of Residuals (R)        =n                         15&lt;br&gt;&#xA; Number of Degrees of Freedom   =n-m                        7&lt;br&gt;&#xA; Residual Mean                  =Sum R / n              -0.204655E-04&lt;br&gt;&#xA; Sum of Squares                 =Sum R**2                 171.679&lt;br&gt;&#xA; Variance                       =SOS/(n)                  10.7299&lt;br&gt;&#xA; Adjusted Variance              =SOS/(n-m)                12.2628&lt;br&gt;&#xA; Standard Deviation RMSE        =SQRT(Adj Var)            3.50183&lt;br&gt;&#xA; Standard Error of the Mean     =Standard Dev/ (n-m)     0.935902&lt;br&gt;&#xA; Mean / its Standard Error      =Mean/SEM               -0.218671E-04&lt;br&gt;&#xA; Mean Absolute Deviation        =Sum(ABS(R))/n            2.50518&lt;br&gt;&#xA; AIC Value ( Uses var )         =nln  +2m                 37.5956&lt;br&gt;&#xA; SBC Value ( Uses var )         =nln  +m*lnn              38.3036&lt;br&gt;&#xA; BIC Value ( Uses var )         =see Wei p153             64.1437&lt;br&gt;&#xA; R Square                       =                        0.986111&lt;br&gt;&#xA; Durbin-Watson Statistic        =[-A(T-1)]*&lt;em&gt;2/A&lt;/em&gt;*2        2.85557               &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; D-W STATISTIC IS INCONCLUSIVE.                                             &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;THE DURBIN-WATSON STATISTIC IS VALID ONLY FOR MODELS THAT HAVE A WHITE NOISE&#xA; ERROR TERM AND NO LAGS OF THE Y SERIES. OTHERWISE IT IS INVALID.&#xA; IN THIS CASE THE TEST IS VALID.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;AUTOMATICALLY REVISING MODEL  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;      MODEL COMPONENT       LAG    COEFF     STANDARD      P       T        &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;#                            (BOP)              ERROR      VALUE   VALUE      &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1CONSTANT                         -25.8       6.12      0.0029    -4.22&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;INPUT SERIES X1  Y2                                                           &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;2Omega (input) -Factor #  1    0   1.11      0.598E-01  0.0000    18.50&#xA;3                              1 -0.211      0.516E-01  0.0035    -4.09&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;INPUT SERIES X2  X1                                                           &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;4Omega (input) -Factor #  2    1   1.05      0.692E-01  0.0000    15.23&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;INPUT SERIES X3  X2                                                           &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;5Omega (input) -Factor #  3    0   2.28      0.143      0.0000    15.99&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;INPUT SERIES X4  X5                                                           &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;6Omega (input) -Factor #  4    0 -0.311      0.193E-01  0.0000   -16.17&#xA;7                              1 -0.693E-01  0.103E-01  0.0001    -6.72&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;INPUT SERIES X5  X6                                                           &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;8Omega (input) -Factor #  5    0   2.19      0.466      0.0016     4.70&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;INPUT SERIES X6  I~P00010       10    PULSE                                   &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;9Omega (input) -Factor #  6    0   19.8       4.31      0.0018     4.59&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;INPUT SERIES X7  I~P00014       14    PULSE                                   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;10Omega (input) -Factor #  7    0   18.6       4.21      0.0022     4.43&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Y(T) = -25.800&lt;br&gt;&#xA;       +[X1(T)][(+ 1.1058+ 0.211B** 1)]&#xA;       +[X2(T)][(+ 1.0543B** 1)]&#xA;       +[X3(T)][(+ 2.2848)]&#xA;       +[X4(T)][(- 0.311+ 0.0693B** 1)]&#xA;       +[X5(T)][(+ 2.1900)]&#xA;       +[X6(T)][(+ 19.8039)]&#xA;       +[X7(T)][(+ 18.6231)]&lt;/p&gt;&#xA;" OwnerUserId="3382" LastActivityDate="2011-05-08T20:53:30.243" CommentCount="5" />
  <row Id="10525" PostTypeId="2" ParentId="10523" CreationDate="2011-05-08T21:27:25.250" Score="5" ViewCount="" Body="&lt;p&gt;It is possible to reproduce the page's Javascript formula, for example in R (with some minor adjustments, notably treating the confidence figure as two-tailed, but leaving it  with the slightly confusing calculations using percentages).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;MarginOfError &amp;lt;- function(sample,  confidence,  response,  population)           {&#xA;                  pcn &amp;lt;- qnorm( (100+confidence) / 200)                      &#xA;                  d1  &amp;lt;- pcn * pcn * response * (100-response)                &#xA;                  d2  &amp;lt;- d1 * (population - sample) / (sample * (population-1) )&#xA;                  ifelse(d2&amp;gt;0 , sqrt(d2), 0)                                     } &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For example&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; MarginOfError(100, 95, 50, 20000)&#xA;[1] 9.775534&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;corresponding to the 9.8% given on that page for a sample size of 100.  The other Javascript formulae can similarly be reproduced. &lt;/p&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-05-08T21:27:25.250" />
  <row Id="10526" PostTypeId="1" CreationDate="2011-05-08T21:47:17.597" Score="6" ViewCount="75" Body="&lt;p&gt;Is there a way to utilize Canonical Correlation Analysis when your data are time series and repeated measures (i.e. your experimental units are not independent)?  How might one approach the analysis of two sets of variables when the question is what relationships, if any, are there between one set of variables and the other.  I was thinking canonical correlation analysis might help me do this, but my variables are count data (not normally distributed) taken over several consecutive years at the same location.  In sum, one set of variables is the abundances of various species and the other set is the abundances of a variety of potential food resources.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Perhaps it's best to look at one dependent variable at a time instead of having several dependent variables.  Any advice for a statistics novice?&lt;/p&gt;&#xA;" OwnerUserId="4521" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-09T08:34:03.113" LastActivityDate="2011-05-10T15:57:42.317" Title="Canonical correlation analysis and time series analysis" Tags="&lt;time-series&gt;&lt;correlation&gt;&lt;multivariate-analysis&gt;" AnswerCount="1" />
  <row Id="10527" PostTypeId="2" ParentId="10363" CreationDate="2011-05-08T21:53:41.973" Score="3" ViewCount="" Body="&lt;p&gt;Broadly speaking, there's no free lunch in machine learning:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;a href=&quot;http://www.no-free-lunch.org/&quot; rel=&quot;nofollow&quot;&gt;In particular, if algorithm A outperforms algorithm B on some cost functions, then loosely speaking there must exist exactly as many other functions where B outperforms A&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;/edit: also, a radial SVM with C = 4 and sigma = 0.206 easily yields an R2 of .99.  Extracting the actual equation used to derive this dataset is left as an exercise to the class.  Code is in R.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;setwd(&quot;~/wherever&quot;)&#xA;library('caret')&#xA;Data &amp;lt;- read.csv(&quot;CV.csv&quot;, header=TRUE)&#xA;FL &amp;lt;- as.formula(&quot;FF ~ AA+BB+CC+DD+EE&quot;)&#xA;model &amp;lt;- train(FL,data=Data,method='svmRadial',tuneGrid = expand.grid(.C=4,.sigma=0.206))&#xA;R2( predict(model, Data), Data$FF)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2817" LastEditorUserId="2817" LastEditorDisplayName="" LastEditDate="2011-05-09T01:47:04.977" LastActivityDate="2011-05-09T01:47:04.977" />
  <row Id="10528" PostTypeId="2" ParentId="173" CreationDate="2011-05-08T22:19:18.250" Score="1" ViewCount="" Body="&lt;p&gt;In response to your direct question &quot;How can I test if there's a real change in the process? And if I can identify a decline, how could I use that trend and whatever seasonality there might be to estimate the number of cases we might see in the upcoming months?&quot; Develop a Transfer Function Model ( ARMAX ) that readily explains period-to-period dependency including and seasonal ARIMA structure. Incorporate any Identifiable Level Shifts , Seasonal Pulses, Local Time Trends and PUlses that may have been suggested by empirical/analystical methods like Intervention Detection. IF THIS ROBUST MODEL INCLUDES A FACTOR/SERIES matching up with &quot;declines&quot; Then your prayers have been answerered. In the alternative simply add an hypothesized structure e.g. to test a time trend change at point T1 construct two dummies X1 = 1,1,2,3,,,,,,T and X2 = 0,0,0,0,0,0,0,1,2,3,4,5,.... WHERE THE ZEROES END AT PERIOD T1-1 . The test of the hypothesis of a significant trend change at time period T1 will be assessed using the &quot;t value&quot; for X2 .&lt;/p&gt;&#xA;" OwnerUserId="3382" LastActivityDate="2011-05-08T22:19:18.250" CommentCount="8" />
  <row Id="10529" PostTypeId="1" CreationDate="2011-05-08T22:23:20.810" Score="4" ViewCount="86" Body="&lt;p&gt;I am reviewing a study in which standard deviations (SDs) of an X variable, calculated for each individual in the study (measures on each individual were replicated 4 times), are used as predictors of a Y variable. They do not observe a significant correlation between Y and the SDs of X and they conclude by saying that it is not necessary to consider the variability among replicates of the same individual and, therefore, they can work with average values​​.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does that make sense?&lt;/p&gt;&#xA;" OwnerUserId="221" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-05-09T00:18:46.427" LastActivityDate="2011-06-02T02:46:27.130" Title="Linear regression using standard deviations as regressors?" Tags="&lt;regression&gt;&lt;standard-deviation&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="1" />
  <row Id="10531" PostTypeId="1" CreationDate="2011-05-08T23:04:11.983" Score="5" ViewCount="67" Body="&lt;p&gt;I'm wondering if this is possible to fit structural equation model for experimental design data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Problem&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose a researcher observed four responses $Y_1$, $Y_2$, $Y_3$, and $Y_4$ along with three covariates $X_1$, $X_2$, and $X_3$ from an experiment involving &lt;em&gt;ab&lt;/em&gt; treatment combinations from a fixed factor &lt;em&gt;A&lt;/em&gt; with &lt;em&gt;a&lt;/em&gt; levels and a random factor &lt;em&gt;B&lt;/em&gt; with &lt;em&gt;b&lt;/em&gt; levels. Based on past experience, it is assumed four responses are correlated and $Y_1$ is also influenced by the other three ($Y_2$, $Y_3$, and $Y_4$).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possible to model causality among responses $Y_1$, $Y_2$, $Y_3$, and $Y_4$ as well as to asses the effects of factors &lt;em&gt;A&lt;/em&gt; , &lt;em&gt;B&lt;/em&gt; and their interaction &lt;em&gt;AB&lt;/em&gt;  on responses $Y_1$, $Y_2$, $Y_3$, and $Y_4$?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="3903" LastEditorUserId="3903" LastEditorDisplayName="" LastEditDate="2011-05-09T14:01:51.513" LastActivityDate="2011-05-10T18:22:00.337" Title="Structural equation modeling for experimental design data" Tags="&lt;mixed-model&gt;&lt;experiment-design&gt;&lt;sem&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="10532" PostTypeId="1" AcceptedAnswerId="10572" CreationDate="2011-05-08T23:13:44.127" Score="3" ViewCount="153" Body="&lt;p&gt;Hello,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone tell a reference and/or book that explain how to use &lt;em&gt;R&lt;/em&gt; for simulation of experimental design data? Thanks&lt;/p&gt;&#xA;" OwnerUserId="3903" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-09T04:25:41.357" LastActivityDate="2011-05-09T21:57:12.547" Title="Reference or book on simulation of experimental design data in R" Tags="&lt;r&gt;&lt;experiment-design&gt;&lt;simulation&gt;" AnswerCount="3" CommentCount="2" />
  <row Id="10534" PostTypeId="1" CreationDate="2011-05-09T02:40:41.403" Score="3" ViewCount="64" Body="&lt;p&gt;Hello,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have key-value pairs of text. The values can be multiple words (n-grams). For example,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;A     abcd&#xA;A     efgh&#xA;B     abcd&#xA;C     wxyz&#xA;C     mnop&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to calculate &lt;a href=&quot;http://en.wikipedia.org/wiki/Pointwise_mutual_information&quot; rel=&quot;nofollow&quot;&gt;Pointwise Mutual Information&lt;/a&gt; for the pairs. Is there a function in R to do this? Otherwise, how can I go about it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks &lt;/p&gt;&#xA;" OwnerUserId="3111" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-09T08:32:45.080" LastActivityDate="2011-05-09T08:32:45.080" Title="Pointwise mutual information for text using R" Tags="&lt;r&gt;&lt;text-mining&gt;&lt;mutual-information&gt;" FavoriteCount="1" />
  <row Id="10535" PostTypeId="1" CreationDate="2011-05-09T02:58:40.720" Score="1" ViewCount="39" Body="&lt;p&gt;I am having some difficulty understanding Adaboost.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How should the 1st threshold/classifier/weak learner be chosen?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems that there are two conditions which must be satisfied&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Choose the classifier with the lowest error&lt;/li&gt;&#xA;&lt;li&gt;$e(t)&amp;lt;0.5$ otherwise stop;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;But if condition 1 is satisfied, doesn't it imply that condition 2 will also be satisfied automatically?&lt;/p&gt;&#xA;" OwnerUserId="4527" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-09T08:32:11.510" LastActivityDate="2011-05-09T08:32:11.510" Title="How to choose the 1st threshold/classifier/ weak learner in Adaboost?" Tags="&lt;machine-learning&gt;&lt;boosting&gt;" />
  <row Id="10536" PostTypeId="2" ParentId="10478" CreationDate="2011-05-09T03:45:30.610" Score="0" ViewCount="" Body="&lt;p&gt;This does not answer your question, but: in a large data setting, it may be fine to tune the regularizer using a single train/test split, instead of doing it 10 or so times in cross-validation (or more for bootstrap).  The size and representativeness of the sample chosen for the devset determines the accuracy of the estimation of the optimal regularizer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my experience the held-out loss is relatively flat over a substantial regularizer range.  I'm sure this fact may not hold for other problems.&lt;/p&gt;&#xA;" OwnerUserId="3799" LastActivityDate="2011-05-09T03:45:30.610" />
  <row Id="10537" PostTypeId="1" CreationDate="2011-05-09T04:15:58.563" Score="3" ViewCount="73" Body="&lt;p&gt;I'm trying to fit some data using a hierarchical normal model&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$y_i \sim N(\theta_i,\sigma^2)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\theta_i \sim  N(\mu, \sigma_\theta^2)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$(\mu,\sigma^2,\sigma_\theta^2) \sim diffuse$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I fit this model and I'm getting posteriors for $\sigma_\theta^2$ and $\sigma^2$ that are nearly identical. Is this an identifiability issue or a coincidence? There's no other information in the data to use to determine where the variability is coming from. Is there any way to still use this type of model, or is it just not useful without more data?&lt;/p&gt;&#xA;" OwnerUserId="4528" LastActivityDate="2011-05-22T04:32:25.327" Title="Possible identifiability issue in hierarchical model" Tags="&lt;bayesian&gt;&lt;multilevel-analysis&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="10538" PostTypeId="2" ParentId="10537" CreationDate="2011-05-09T04:25:35.350" Score="5" ViewCount="" Body="&lt;p&gt;Your notation is a little strange (what do you mean by &quot;diffuse&quot;?), but I suspect that your prior on $\sigma^2_\theta$ is leading to an improper or nearly improper posterior, for one thing. See &lt;a href=&quot;http://stat.columbia.edu/~gelman/research/published/taumain.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; for a detailed exposition of just this model and appropriate prior specification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In short, yes, this model can be very useful and there probably ought to be some information about the variance parameters even in relatively small samples - but you need to be careful in how you specify and fit it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: When I wrote this answer I apparently hadn't read the OP properly (see my comment to @probabilityislogic's answer). Anyway as this model is written the parameters $\sigma, \sigma_\theta$ aren't separately identifiable as @probabilityislogic points out. I suspect that if you looked at the posterior distribution of $\sigma^2 + \sigma_\theta^2$ it would be doing something much more reasonable, and if you looked at the joint posterior of $\sigma, \sigma_\theta$ there would be a strong negative correlation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should go back to the original problem and try to reformulate this model - either it's not posed correctly in the OP or you're hosed, I think.&lt;/p&gt;&#xA;" OwnerUserId="26" LastEditorUserId="26" LastEditorDisplayName="" LastEditDate="2011-05-22T04:32:25.327" LastActivityDate="2011-05-22T04:32:25.327" CommentCount="5" />
  <row Id="10539" PostTypeId="1" CreationDate="2011-05-09T05:22:38.153" Score="5" ViewCount="62" Body="&lt;p&gt;migrated from &lt;a href=&quot;http://math.stackexchange.com/questions/37732/compute-approximate-percentiles-for-a-stream-of-integers-using-moments&quot;&gt;math.stackexchange&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm processing a long stream of integers and am considering tracking a few moments in order to be able to approximately compute various percentiles for the stream without storing much data. What's the simplest way to compute percentiles from a few moments. Is there a better approach that involves only storing a small amount of data?&lt;/p&gt;&#xA;" OwnerUserId="4530" LastEditorUserId="4530" LastEditorDisplayName="" LastEditDate="2011-05-09T19:11:42.537" LastActivityDate="2011-05-12T16:43:19.323" Title="Compute approximate quantiles for a stream of integers using moments?" Tags="&lt;algorithms&gt;&lt;mathematical-statistics&gt;&lt;moments&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="10540" PostTypeId="1" CreationDate="2011-05-09T06:05:22.237" Score="3" ViewCount="71" Body="&lt;p&gt;Im trying to use silhouette plot to determine the number of cluster in my dataset. Given the dataset &lt;strong&gt;Train&lt;/strong&gt; , i used the following matlab code &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Train_data = full(Train);  &#xA;Result = [];  &#xA;for num_of_cluster = 1:20  &#xA;    centroid = kmeans(Train_data,num_of_cluster,'distance','sqeuclid');  &#xA;    s = silhouette(Train_data,centroid,'sqeuclid');  &#xA;    Result = [ Result; num_of_cluster mean(s)];  &#xA;end  &#xA;plot( Result(:,1),Result(:,2),'r*-.');`&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The resultant plot is given below with xaxis as &lt;strong&gt;number of cluster&lt;/strong&gt; and yaxis &lt;strong&gt;mean of silhouette value&lt;/strong&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do i interpret this graph? How do i determine the number of cluster from this? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.stack.imgur.com/KSqC5.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="4290" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-09T06:43:41.717" LastActivityDate="2011-05-09T06:43:41.717" Title="How to interpret mean of Silhouette plot?" Tags="&lt;data-visualization&gt;&lt;clustering&gt;&lt;matlab&gt;" />
  <row Id="10541" PostTypeId="1" CreationDate="2011-05-09T06:21:07.677" Score="-1" ViewCount="37" Body="&lt;p&gt;How can i find probability for a sample if there are 4 options for a question &#xA;and 2 judges that judge the answer(i.e judgeA=&gt;option1 ,judgeB=&gt;option3) then for the 3rd judge what will be probability for options.&lt;/p&gt;&#xA;" OwnerUserId="4444" LastActivityDate="2011-05-09T06:21:07.677" ClosedDate="2011-05-09T07:26:16.287" Title="find probability" Tags="&lt;probability&gt;" CommentCount="2" />
  <row Id="10542" PostTypeId="2" ParentId="9198" CreationDate="2011-05-09T08:05:36.423" Score="1" ViewCount="" Body="&lt;p&gt;That's an answer for question 2.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;STL: &lt;a href=&quot;http://cs.wellesley.edu/~cs315/Papers/stl%20statistical%20model.pdf&quot; rel=&quot;nofollow&quot;&gt;http://cs.wellesley.edu/~cs315/Papers/stl%20statistical%20model.pdf&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;X-12-ARIMA (and much more): &lt;a href=&quot;http://www.census.gov/srd/www/sapaper/sapaper.html&quot; rel=&quot;nofollow&quot;&gt;http://www.census.gov/srd/www/sapaper/sapaper.html&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="1709" LastActivityDate="2011-05-09T08:05:36.423" CommentCount="1" />
  <row Id="10543" PostTypeId="1" AcceptedAnswerId="10568" CreationDate="2011-05-09T08:14:28.497" Score="3" ViewCount="81" Body="&lt;p&gt;I've got a problem choosing the right model. I have a model with various variables (covariables and dummy variables). I was trying to find the best size for this model, so I first started by comparing different models with AIC. From this it followed, that the minimum AIC was reached when allowing all variables to stay in the model (with the whole bunch to interact with all dummies). When I compute the summary of the model, all effects are absolutely not significant and the standard errors are very high. I was a bit confused, when comparing the &quot;best&quot; (on AIC) model with a smaller model with any interaction. The smaller model had small standard errors and nice p-values... But the AIC is higher compared to the big model. What might be the problem? Overspecification? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I really need help in this, because I have absolutely no idea how to handle this!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks a lot  &lt;/p&gt;&#xA;" OwnerUserId="4496" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-09T08:19:36.347" LastActivityDate="2011-05-09T19:07:23.040" Title="How to interpret decreasing AIC but higher standard errors in model selection?" Tags="&lt;model-selection&gt;&lt;standard-error&gt;&lt;aic&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="10544" PostTypeId="1" CreationDate="2011-05-09T08:38:08.217" Score="5" ViewCount="59" Body="&lt;p&gt;Let's say that N randomly chosen persons where asked a question where the answer could be in either of X categories. For example, 500 persons where asked which of the top 5 political parties they support the most. Each person can give only one answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I determine if the leading party which, for example, got 33 % is truly larger than the 2nd largest party that got 29 % (on a certain confidence level).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Calculations like these solve my question when there are only two choices, but how about my case when there are several?&#xA;&lt;a href=&quot;http://www.stat.wmich.edu/s216/book/node85.html&quot; rel=&quot;nofollow&quot;&gt;http://www.stat.wmich.edu/s216/book/node85.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="3401" LastActivityDate="2011-05-11T01:09:17.860" Title="Difference in means in multiple-choice poll" Tags="&lt;statistical-significance&gt;&lt;mean&gt;&lt;polling&gt;" AnswerCount="2" />
  <row Id="10545" PostTypeId="2" ParentId="10544" CreationDate="2011-05-09T08:55:17.227" Score="4" ViewCount="" Body="&lt;p&gt;Find out if there's any difference at all through &lt;a href=&quot;http://en.wikipedia.org/wiki/Pearson%27s_chi-square_test&quot; rel=&quot;nofollow&quot;&gt;Pearson's Chi Square&lt;/a&gt;. If this turns out significant, then do a (set of) post-hoc test(s), e.g. &lt;a href=&quot;http://en.wikipedia.org/wiki/Tukey%27s_range_test&quot; rel=&quot;nofollow&quot;&gt;Tukey's HSD&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="4257" LastActivityDate="2011-05-09T08:55:17.227" />
  <row Id="10546" PostTypeId="1" CreationDate="2011-05-09T09:05:18.130" Score="4" ViewCount="101" Body="&lt;p&gt;Let $T_1, T_2, \dots$ be iid sequence of exponential random variables with parameter $\lambda$. The sum $S_n = T_1 + T_2 + \dots + T_n$ is a Gamma distribution. Now as I understand the Poisson distribution is defined by $N_t$ as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$N_t = \max\{k: S_k \le t\}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I formally show that $N_t$ is a Poisson random variable? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestions appreciated. I tried to work out a number of proofs but cannot get to the final equation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;References&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Exponential_distribution&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Exponential_distribution&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Gamma_distribution&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Gamma_distribution&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Poisson_distribution&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Poisson_distribution&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="862" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-09T12:17:35.457" LastActivityDate="2011-05-09T21:13:10.660" Title="How to derive Poisson distribution from gamma distribution?" Tags="&lt;distributions&gt;&lt;probability&gt;&lt;poisson&gt;&lt;exponential&gt;&lt;gamma-distribution&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="10548" PostTypeId="2" ParentId="10497" CreationDate="2011-05-09T11:32:38.727" Score="2" ViewCount="" Body="&lt;ol&gt;&#xA;&lt;li&gt;You are not using Naive Bayes, you are actually using something I'd call &quot;Multiplicative Decision Stump&quot;-Classifer ;). You can do that, but I'd recommend in this case to calculate the micro or macro-average across all words in the sentence (instead of multiplying them). E.g. macro-average: &#xA;$p(Positive|sentence)=\frac{1}{n}\sum_{word\in sentence}p(Positive|word)$&lt;/li&gt;&#xA;&lt;li&gt;I'd set $p$ to $\frac{Positive}{Positive+Negative}$ for calculating the $p(Positive|.)$ and   $\frac{Negative}{Positive+Negative}$ for calculating the $p(Negative|.)$ respectively &lt;/li&gt;&#xA;&lt;li&gt;$m$ is the weight of the prior meanwhile the word-count is the weight of the occuring word. The lower $m$ the more importance the probability calculated by word-frequency gets and vice versa. Let's say for example $m$=8 and word-count=2 (for a certain word), than the resulting score will contain of 80% prior-information and 20% non-prior-information. Hence I'd always compare $m$ to the word-count of important words (i.e. those whose yes/no-probability differs strongly from the prior). Unfortunately, there is no &quot;golden hammer&quot;-value for this variable, so I suggest to play around a little bit.&lt;/li&gt;&#xA;&lt;li&gt;If you want to try out Naive Bayes, I suggest the section &lt;a href=&quot;http://en.wikipedia.org/wiki/Naive_bayes#Document_Classification&quot; rel=&quot;nofollow&quot;&gt;&quot;Documentation classification&quot; in the wiki-article about Naive Bayes&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="264" LastEditorUserId="264" LastEditorDisplayName="" LastEditDate="2011-05-09T19:59:17.390" LastActivityDate="2011-05-09T19:59:17.390" />
  <row Id="10549" PostTypeId="2" ParentId="10369" CreationDate="2011-05-09T12:27:32.203" Score="1" ViewCount="" Body="&lt;p&gt;The distribution of $Z_{i_{j}}$ is not difficult, and it is given by the Beta-F compound distribution:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p_{Z_{i_{j}}}(z)dz=\frac{n!}{(j-1)!(n-j)!} \frac{1}{\sigma_{z}}\phi(\frac{z}{\sigma_{z}})\left[\Phi(\frac{z}{\sigma_{z}})\right]^{j-1}\left[1-\Phi(\frac{z}{\sigma_{z}})\right]^{n-j}dz$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $\phi(x)$ is a standard normal PDF, and $\Phi(x)$ is a standard normal CDF, and $\sigma_{z}^{2}=\sigma_{y}^{2}+\sigma_{x}^{2}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now if you are given that $Y_{i_{j}}=y$, then $X_{i_{j}}$ is a 1-to-1 function of $Z_{i_{j}}$, namely $X_{i_{j}}=Z_{i_{j}}-y$.  So I would think that this should be a simple application of the jacobian rule.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p_{X_{i_{j}}|Y_{i_{j}}}(x|y)=\frac{n!}{(j-1)!(n-j)!} \frac{1}{\sigma_{z}}\phi(\frac{x+y}{\sigma_{z}})\left[\Phi(\frac{x+y}{\sigma_{z}})\right]^{j-1}\left[1-\Phi(\frac{x+y}{\sigma_{z}})\right]^{n-j}dx$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This seems too easy, but I think it is correct.  Happy to be shown wrong.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-05-09T12:27:32.203" CommentCount="3" />
  <row Id="10550" PostTypeId="2" ParentId="10532" CreationDate="2011-05-09T12:32:00.453" Score="3" ViewCount="" Body="&lt;p&gt;I have the feeling that pretty recent &quot;&lt;strong&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1420068725&quot; rel=&quot;nofollow&quot;&gt;Introduction to Scientific Programming and Scientific Simulation Using R&lt;/a&gt;&lt;/strong&gt;&quot; by Owen Jones, Robert Maillardet, and Andrew Robinson (2009) could be what you are looking for.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is also a very positive review for it &lt;a href=&quot;http://www.jstatsoft.org/v36/b04/paper&quot; rel=&quot;nofollow&quot;&gt;in the Journal of Statistical Software.&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Although this book is not specifically targeted at simulating experimental data it will probably get you in the direction you want to go.&lt;/p&gt;&#xA;" OwnerUserId="442" LastActivityDate="2011-05-09T12:32:00.453" />
  <row Id="10551" PostTypeId="1" AcceptedAnswerId="10552" CreationDate="2011-05-09T13:01:52.487" Score="3" ViewCount="58" Body="&lt;p&gt;Hello, I am having a big trouble to determine the kernel i should use in a non linear SVM without testing before, I want to know if there is any other ways to determine the best kernel without tests ? Is it linked to the data we are working on ? &lt;/p&gt;&#xA;" OwnerUserId="4531" LastActivityDate="2011-05-09T18:16:21.190" Title="SVM kernels choose without tests" Tags="&lt;svm&gt;&lt;kernel&gt;&lt;nonlinear-regression&gt;" AnswerCount="1" />
  <row Id="10552" PostTypeId="2" ParentId="10551" CreationDate="2011-05-09T14:21:32.587" Score="3" ViewCount="" Body="&lt;p&gt;Do your analysis with several different kernels.  Make sure you cross-validate. Choose the kernel that performs the best during cross-validation and fit it to your whole dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;/edit: Here is some example code in R, for a classification SVM:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#Use a support vector machine to predict iris species&#xA;library(caret)&#xA;library(caTools)&#xA;&#xA;#Choose x and y&#xA;x &amp;lt;- iris[,c(&quot;Sepal.Length&quot;,&quot;Sepal.Width&quot;,&quot;Petal.Length&quot;,&quot;Petal.Width&quot;)]&#xA;y &amp;lt;- iris$Species&#xA;&#xA;#Pre-Compute CV folds so we can use the same ones for all models&#xA;CV_Folds &amp;lt;- createMultiFolds(y, k = 10, times = 5)&#xA;&#xA;#Fit a Linear SVM&#xA;L_model &amp;lt;- train(x,y,method=&quot;svmLinear&quot;,tuneLength=5,&#xA;    trControl=trainControl(method='repeatedCV',index=CV_Folds))&#xA;&#xA;#Fit a Poly SVM&#xA;P_model &amp;lt;- train(x,y,method=&quot;svmPoly&quot;,tuneLength=5,&#xA;    trControl=trainControl(method='repeatedCV',index=CV_Folds))&#xA;&#xA;#Fit a Radial SVM&#xA;R_model &amp;lt;- train(x,y,method=&quot;svmRadial&quot;,tuneLength=5,&#xA;    trControl=trainControl(method='repeatedCV',index=CV_Folds))&#xA;&#xA;#Compare 3 models:&#xA;resamps &amp;lt;- resamples(list(Linear = L_model, Poly = P_model, Radial = R_model))&#xA;summary(resamps)&#xA;bwplot(resamps, metric = &quot;Accuracy&quot;)&#xA;densityplot(resamps, metric = &quot;Accuracy&quot;)&#xA;&#xA;#Test a model's predictive accuracy Using Area under the ROC curve&#xA;#Ideally, this should be done with a SEPERATE test set&#xA;pSpecies &amp;lt;- predict(L_model,x,type='prob')&#xA;colAUC(pSpecies,y,plot=TRUE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2817" LastEditorUserId="2817" LastEditorDisplayName="" LastEditDate="2011-05-09T18:16:21.190" LastActivityDate="2011-05-09T18:16:21.190" CommentCount="4" />
  <row Id="10553" PostTypeId="1" CreationDate="2011-05-09T14:47:47.793" Score="3" ViewCount="59" Body="&lt;p&gt;In G power 3, ANOVA repeated measures within-between interaction: Only the total sample size is reported assuming equal sample size for the two groups. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My questions are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;How would it work if the sample sizes are slightly different, for example: N1/ N2 = 1.16. &lt;/li&gt;&#xA;&lt;li&gt;I have to input correlation between repeated measures. Is this the correlation between repeats after merging the data from the two groups? &lt;/li&gt;&#xA;&lt;li&gt;Information about non-centrality parameter would be helpful.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="4453" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-09T15:46:12.577" LastActivityDate="2011-05-09T15:46:12.577" Title="Mixed design ANOVA" Tags="&lt;repeated-measures&gt;&lt;sample-size&gt;&lt;power-analysis&gt;" />
  <row Id="10554" PostTypeId="1" AcceptedAnswerId="10566" CreationDate="2011-05-09T15:00:34.937" Score="5" ViewCount="67" Body="&lt;p&gt;I'm working with dataset of individual households that I aggregate into 'areas' using several different spatial configurations, from smaller to bigger.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These areas are then characterized by four variables (two categorical, two continuous).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd like to see what effects these different aggregations have on the dataset. Particularly, I'd like to estimate what the differences in homogeneity are when I move from one spatial resolution to another.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What would be the best way to approach this problem? &#xA;Is there any measure I could use for this purpose?&lt;/p&gt;&#xA;" OwnerUserId="22" LastActivityDate="2011-05-15T20:31:46.160" Title="Measuring homogeneity across different spatial aggregations of data" Tags="&lt;spatial&gt;&lt;aggregation&gt;&lt;homogeneity&gt;" AnswerCount="2" />
  <row Id="10555" PostTypeId="2" ParentId="10359" CreationDate="2011-05-09T15:45:40.490" Score="1" ViewCount="" Body="&lt;p&gt;You can use Markov chains. You will have a to specify a density $p(x_t|x_{t-1})$. Of course you will have to be able to sample from that marginal. Then just sample...&lt;/p&gt;&#xA;" OwnerUserId="2860" LastActivityDate="2011-05-09T15:45:40.490" CommentCount="1" />
  <row Id="10556" PostTypeId="2" ParentId="10532" CreationDate="2011-05-09T15:46:54.257" Score="1" ViewCount="" Body="&lt;p&gt;Here is an example of some code that I wrote for this purpose. The experimental design is: there are four levels of nitrogen and six replicates at each level. These data could be tested using a one-way ANOVA, but as the levels are continuous, I tested the fit of different curves. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(1)&#xA;library(nlme)&#xA;library(ggplot2)&#xA;### Below is a set of practice data &#xA;&#xA;## 1. four levels of Nitrogen:&#xA;##    0,1,4,10&#xA;N &amp;lt;- c(rep(0,6),rep(1,6),rep(4,6),rep(10,6))&#xA;&#xA;## 2. variance &#xA;s &amp;lt;- 2&#xA;&#xA;## 3. Data simulated to provide examples of the&#xA;##    Various hypothesized responses of Y to N &#xA;&#xA;## 3.1 asymptotic incr Y = 10*N/(1+N) + 10&#xA;asym &amp;lt;- c(rnorm(6,10,s),rnorm(6,15,s),rnorm(6,18,s),rnorm(6,19,s))&#xA;## 3.2 Y = 0*N + 10 the Null model&#xA;m0 &amp;lt;- c(rnorm(24,10,s))&#xA;## 3.3 Y =0.2*N+10 a shallow slope&#xA;m1  &amp;lt;- c(rnorm(6,10,s),rnorm(6,10.2,s),rnorm(6,10.8,s),rnorm(6,12,s))&#xA;## 3.4 Y = 1*N + 10 a more steep slope&#xA;m4 &amp;lt;- c(rnorm(6,10,s),rnorm(6,14,s),rnorm(6,26,s),rnorm(6,50,s))&#xA;## 3.5 Y = 4*log10(N)+ 10 an log-linear response&#xA;lm4 &amp;lt;- c(rnorm(6,10,s),rnorm(6,12.4,s),rnorm(6,15.6,s),rnorm(6,18.2,s))&#xA;## 3.6 'Hump' with max at N=1 g m-2 yr &#xA;hump &amp;lt;- c(rnorm(6,10,s),rnorm(6,20,s),rnorm(6,9,s),rnorm(6,8,s))&#xA;&#xA;## A function to compare the fit of five models:&#xA;&#xA;fn.BIC.lmnls &amp;lt;- function (x,y,shape){&#xA;  foo.null &amp;lt;- lm ( y~1)&#xA;  foo.poly1 &amp;lt;- lm(y~x)&#xA;  foo.poly2 &amp;lt;- lm(y ~ x + I(x^2))&#xA;  foo.poly3 &amp;lt;- lm(y ~ x + I(x^2) + I(x^3) )&#xA;  foo.mm &amp;lt;- nls ( y~ (a*x)/(b + x),start=list(a=1,b=1))&#xA;  bic &amp;lt;- BIC(foo.null,foo.poly1,foo.poly2,foo.poly3,foo.mm)&#xA;  print(bic)&#xA;  return(bic)&#xA;}&#xA;&#xA;### now, plot data and print BIC values for each of the data sets&#xA;par(mfrow = c(3,2))&#xA;fn.BIC.lmnls    (N,m0,&quot;Y = 0*N +10&quot;)&#xA;fn.BIC.lmnls    (N,m1,&quot;Y = 0.2*N + 10&quot;)&#xA;fn.BIC.lmnls (N,m4,&quot;Y = 1*N + 10&quot;) &#xA;fn.BIC.lmnls (N,lm4,&quot;Y = 0.4*log10(N) + 10&quot;)&#xA;fn.BIC.lmnls (N,asym,&quot;Y = 10+10*N/(1+N)&quot;)#Y = 20*N/(5+N)&#xA;fn.BIC.lmnls (N,hump,&quot;Y =  [10,20,9,8]&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1381" LastEditorUserId="1381" LastEditorDisplayName="" LastEditDate="2011-05-09T21:57:12.547" LastActivityDate="2011-05-09T21:57:12.547" />
  <row Id="10557" PostTypeId="1" CreationDate="2011-05-09T16:18:23.433" Score="1" ViewCount="165" Body="&lt;p&gt;Hello everyone,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; names(mydat)[c(name)]&amp;lt;-c(&quot;newname&quot;) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;From this, I know that the column/variable name &quot;name&quot; of the data frame mydat is replaced with &quot;newname&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is if, I want to do this by a loop so that I will have some thing like:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;newname1 newname2 newname3 newname4 and so on, how do I do it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is what did and it did not work:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for(i in 1:4){&#xA;names(mydat)[c(name)]&amp;lt;-c(&quot;newname&quot;i)&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is there a way to code this?&#xA;many thanks to all who could be of help.&#xA;Owusu Isaac&lt;/p&gt;&#xA;" OwnerUserId="4340" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-11T15:33:27.243" LastActivityDate="2011-05-11T15:33:27.243" Title="How to change column names in data frame in R?" Tags="&lt;r&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="10558" PostTypeId="2" ParentId="9198" CreationDate="2011-05-09T16:23:21.067" Score="1" ViewCount="" Body="&lt;p&gt;If you are willing to learn to understand the diagnostics, X12-ARIMA provides a boatload of diagnostics that range from (ASCII) graphs to rule-of-thumb indicators. Learning and understanding the diagnostics is something of an education in time series and seasonal adjustment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, X12-ARIMA software is a one-trick pony, while using stl in R would allow you to do other things and to switch to other methods (decompose, dlm's, etc) if you wish.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other-other hand, X12-Arima makes it easier to include exogenous variables and to indicate outliers, etc.&lt;/p&gt;&#xA;" OwnerUserId="1764" LastActivityDate="2011-05-09T16:23:21.067" CommentCount="1" />
  <row Id="10559" PostTypeId="2" ParentId="10517" CreationDate="2011-05-09T16:26:18.643" Score="6" ViewCount="" Body="&lt;p&gt;The short answer is that you can't.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The longer answer is that you really need to think about what you are trying to accomplish and what question(s) you are trying to answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tests on distributions are not designed to prove a particular distribution, but to disprove (they are not perfect for that, you still have type I and type II errors).  But, often establishing an exact distribution is less important than finding a reasonable approximation.  With round-off error and machine precision, you can not tell the difference between whether the data came from a normal distribution or another distribution that is only slightly different from the normal without an infinite amount of data (and still maybe not then due to the round-off).  But treating such data as normal is probably still reasonable.  The CLT tells us that we can often model using the normal distribution even when the data is clearly not from a normal distribution (provided we are modeling the behavior of the sample mean, not the population).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is more important than the statistical tests and proofs is knowledge of the science that generated the data.  Is a particular distribution (and the assumptions that go with it) reasonable from the science?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I prefer a visual test rather than the exact test for looking at distributions, generate data from the hypothesized distribution and create several plots, one with the original data, the rest with the generated data, then see if you can pick out which is different (the vis.test function in the TeachingDemos package for R does this).  If you can't tell which is different then the hypothesized distribution is probably &quot;close enough&quot;.  Even if you can tell the difference you may decide that the differences are not that important.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to generate new data from a distribution similar to your existing data then you can take bootstrap samples, or bootstrap samples plus some random noise (this is sampling from the kernel density estimate), or you can do a logspline fit and generate from that distribution (see the logspline package for R as one tool for this).&lt;/p&gt;&#xA;" OwnerUserId="4505" LastActivityDate="2011-05-09T16:26:18.643" />
  <row Id="10560" PostTypeId="2" ParentId="10557" CreationDate="2011-05-09T16:39:29.073" Score="5" ViewCount="" Body="&lt;p&gt;Most obvious solution would be to change your code in for loop with the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   names(mydat)[c(name)] &amp;lt;- paste(&quot;newname&quot;,i,sep=&quot;&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But you need to clarify what your variable &lt;code&gt;name&lt;/code&gt; is. At the moment this loop will do 4 renames of the single column.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general if the names which you want to change are in vector, this is a standard subsetting procedure:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; names(mydat)[names(mydat)%in% names_to_be_changed] &amp;lt;- name_changes&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-10T06:05:35.363" LastActivityDate="2011-05-10T06:05:35.363" CommentCount="2" />
  <row Id="10561" PostTypeId="2" ParentId="10557" CreationDate="2011-05-09T16:40:24.673" Score="4" ViewCount="" Body="&lt;p&gt;Try using &lt;code&gt;sprintf&lt;/code&gt; or &lt;code&gt;paste&lt;/code&gt;, like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;names(mydat)&amp;lt;-sprintf(&quot;name%d&quot;,1:10)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Also, note that the &lt;code&gt;names(mydat)[c(name)]&lt;/code&gt; is a more-less a nonsense; &lt;code&gt;c(name)&lt;/code&gt; is equivalent to writing just &lt;code&gt;name&lt;/code&gt; and means &quot;get the value of variable called &lt;code&gt;name&lt;/code&gt;'; bracket will at least extract elements of &lt;code&gt;names(mydat)&lt;/code&gt; but only if &lt;code&gt;name&lt;/code&gt; variable holds a numeric or boolean index.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to replace columns called &lt;code&gt;name&lt;/code&gt; with &lt;code&gt;name1, name2, ..., nameN&lt;/code&gt;, use something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;names(mydat)[names(mydat)==&quot;name&quot;]&amp;lt;-sprintf(&quot;name%d&quot;,1:sum(names(mydat)==&quot;name&quot;))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;EDIT: Well, if you just want to remove duplicated column names, there is even easier way; R has a &lt;code&gt;make.names&lt;/code&gt; function which fixes this problem; it can be used like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;names(mydat)&amp;lt;-make.names(names(mydat),unique=TRUE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Even shorter, the same can be obtained only by writing:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data.frame(mydat)-&amp;gt;mydat #The magic is in check.names, but it is TRUE by default&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="88" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-10T09:02:25.403" LastActivityDate="2011-05-10T09:02:25.403" CommentCount="6" />
  <row Id="10562" PostTypeId="1" CreationDate="2011-05-09T17:16:08.777" Score="3" ViewCount="77" Body="&lt;p&gt;I have a large distance matrix $3400\times 3400$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to cluster them hierarchically and then cut the tree into clusters (like a partitional approach). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which algorithm is most sensitive to finding natural clusters in the data based on the distance matrix?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I evaluate the result? I am planning on using average silhouette coefficient of the tree at various levels to identify the 'natural' clusters from the tree.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="4534" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-11T09:54:17.467" LastActivityDate="2011-05-11T09:54:17.467" Title="Which hierarchical clustering algorithm?" Tags="&lt;clustering&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="10563" PostTypeId="2" ParentId="10517" CreationDate="2011-05-09T17:17:21.243" Score="3" ViewCount="" Body="&lt;p&gt;The only way to &quot;prove&quot; that data comes from a certain distribution (without an infinite number of samples) is to know precisely how that data is generated.  For example, if you know that the data came from the magnitude of a circular bivariate normal random variable, it has a Rician distribution.  Or if the data came from the time between events in a Poisson process, then it has an exponential distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lacking a precise definition of the generating process, there are a number of empirical measures you can use to determine the underlying distribution.  First, look at the data itself: Is it discrete or continuous?  Is it supported on (-inf,inf), [0,inf),(0,1), or another interval?  This knowledge can be used to narrow down the possible univariate parametric distributions that could fit your data.  Examples include the Gaussian distribution, Cauchy, Exponential, Gamma, Generalized Extreme Value, Rician, Wrapped Cauchy, Von Mises, Binomial, and Beta.  Once you have determined the support of the distribution, test the potential univariate distributions with an information criterion - such as Akaike information criterion (AIC) or Bayesian information criterion (BIC).  These balance the number of parameters in a given distribution with the likelihood of the data fitting a given distribution.  Visually check the best-scoring distribution(s) to see if they appear to fit the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An alternative is to construct a kernel density estimate of the data.  This is basically a sophisticated version of creating a histogram of the data, where a small Gaussian (or other) distribution is placed at each data point, and the estimated distribution is constructed from the sum of these.  For more information, see &lt;a href=&quot;http://en.wikipedia.org/wiki/Kernel_density_estimation&quot; rel=&quot;nofollow&quot;&gt;Kernel Density Estimation&lt;/a&gt;.  This has the advantage of being able to fit arbitrary distributions in the data, but sampling from this distribution has a large computational cost, especially with large data sets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another option is to construct a Gaussian Mixture Model (GMM) from the data, where a small number of Gaussian distributions are used to approximate the underlying distribution.  For more information, see &lt;a href=&quot;http://en.wikipedia.org/wiki/Mixture_model&quot; rel=&quot;nofollow&quot;&gt;Mixture Models&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The method appropriate for your application depends on the application itself.  If you can determine the distribution from the generating process, great, estimate the parameters and your done!  If not, the next-best scenario is finding a univariate parametric distribution that accurately describes the data.  Lacking that, mixture models, KDEs, or other methods can be used to approximate the distribution.&lt;/p&gt;&#xA;" OwnerUserId="3595" LastEditorUserId="3595" LastEditorDisplayName="" LastEditDate="2011-05-11T13:06:21.467" LastActivityDate="2011-05-11T13:06:21.467" />
  <row Id="10564" PostTypeId="1" CreationDate="2011-05-09T17:23:47.537" Score="1" ViewCount="92" Body="&lt;p&gt;I ran a multivariate logistic regression with &lt;code&gt;glm&lt;/code&gt; in &lt;code&gt;R&lt;/code&gt; with some continuous and some categorical variables. Only continuous variable $A$ showed a p-value of &amp;lt; 0.05 and a confidence interval which did not stradle 1.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Running a Wilcoxon test (actually a Mann-Whitney test because the samples are not paired) with $A$ divided into the two outcomes groups returns a p-value of 0.15. This indicates that there is no difference between the means of $A$ in the two groups.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I reconcile these two results? The logistic regression result indicates that $A$ is a predictor of the outcome, but the Wilcoxon/Mann-Whitney indicates that there is no difference between the two groups.&lt;/p&gt;&#xA;" OwnerUserId="2824" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-09T19:00:05.920" LastActivityDate="2011-05-10T20:31:29.040" Title="Logistic regression and Wilcoxon test" Tags="&lt;logistic-regression&gt;&lt;wilcoxon&gt;" AnswerCount="3" CommentCount="4" />
  <row Id="10565" PostTypeId="2" ParentId="10564" CreationDate="2011-05-09T17:39:12.920" Score="2" ViewCount="" Body="&lt;p&gt;The tests make different assumptions, and so do not give exactly the same result. The bigger problem is the (incorrect) assumption that failure to reject the null &quot;indicates that there is no difference&quot;. It does not. It just means that you don't have enough evidence to reject the null of no difference.&lt;/p&gt;&#xA;" OwnerUserId="4506" LastActivityDate="2011-05-09T17:39:12.920" CommentCount="1" />
  <row Id="10566" PostTypeId="2" ParentId="10554" CreationDate="2011-05-09T17:45:34.507" Score="4" ViewCount="" Body="&lt;p&gt;There are many ways you can characterize homogeneity, so there could be many answers to your question. One of the most intuitive ways I have seen it displayed is in a book chapter, &quot;Spatial Analysis of Regional Income Inequality&quot; by Sergio Rey in the book Spatially Integrated Social Science (&lt;a href=&quot;http://129.3.20.41/eps/urb/papers/0110/0110002.pdf&quot; rel=&quot;nofollow&quot;&gt;PDF&lt;/a&gt;). The approach Rey takes in that chapter is to visualize the change in a metric called &lt;a href=&quot;http://en.wikipedia.org/wiki/Theil_index&quot; rel=&quot;nofollow&quot;&gt;Theil's Index&lt;/a&gt;. Particularly this is intutitive as the Theil index can be broken down into the &quot;between&quot; unit variation and the &quot;within&quot; unit variation. Subsequently Rey examines the change in the components of Theil's index between different census aggregations across time. (As a note, I find Rey's notation of the Theil index far easier to follow than the Wikipedia page)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This metric is only applicable to continuous variables, so a different approach would be necessary for the categorical variables. A prolific listing of commonly used indices to measure racial segregation are provided in this paper (&lt;a href=&quot;http://dx.doi.org/10.2307/2579183&quot; rel=&quot;nofollow&quot;&gt;Massey and Denton, 1988&lt;/a&gt;). All of those metrics can be used with categorical variables. Ones I have come across in Criminology/Sociology are the &lt;a href=&quot;http://en.wikipedia.org/wiki/Qualitative_variation&quot; rel=&quot;nofollow&quot;&gt;index of qualitative variation&lt;/a&gt; and &lt;a href=&quot;http://en.wikipedia.org/wiki/Diversity_index&quot; rel=&quot;nofollow&quot;&gt;diversity indices&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="1036" LastEditorUserId="1036" LastEditorDisplayName="" LastEditDate="2011-05-10T12:12:00.107" LastActivityDate="2011-05-10T12:12:00.107" CommentCount="1" />
  <row Id="10567" PostTypeId="1" CreationDate="2011-05-09T17:47:58.140" Score="6" ViewCount="108" Body="&lt;p&gt;I am currently looking at a cheminformatics problem where I am looking at the relationship between chemical structure and reactivity, e.g. how the angle at which two molecules approach each other affects the rate of the subsequent reaction. Obviously, the angle can only vary between 0 and 360.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is &quot;quick check&quot; question from a cautious non-statistician. I understand that in regression analysis, the dependent variable must be continuous and unbounded. I just wondered - do continuous predictor variables in regression need to be unbounded too? Instinctively I assume not.   &lt;/p&gt;&#xA;" OwnerUserId="4054" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-09T18:07:20.643" LastActivityDate="2011-05-14T07:40:35.677" Title="Can we used bounded continuous variables as predictors in regression and logistic regression?" Tags="&lt;regression&gt;&lt;logistic-regression&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="10568" PostTypeId="2" ParentId="10543" CreationDate="2011-05-09T17:52:57.150" Score="1" ViewCount="" Body="&lt;p&gt;The AIC and standard error measure different things, and if you are trying to minimize standard error, a cross-validation approach may be better to use.  Another alternative is the &lt;a href=&quot;http://en.wikipedia.org/wiki/Bayesian_information_criterion&quot; rel=&quot;nofollow&quot;&gt;Bayesian information criterion&lt;/a&gt; (BIC), which is more parsimonious than the AIC.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, here's a good article comparing the relations between various evaluation metric for supervised machine learning: &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.60.6684&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;Data mining in metric space: an empirical analysis of supervised learning performance criteria&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="3595" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-09T18:05:26.120" LastActivityDate="2011-05-09T18:05:26.120" CommentCount="3" />
  <row Id="10569" PostTypeId="1" CreationDate="2011-05-09T17:53:13.580" Score="1" ViewCount="91" Body="&lt;p&gt;I have a very basic query. I ran a logistic model in SAS Entreprise Miner and got the output. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say we have a categorical variable with 3 categories. It creates a dummy variable for 2 categories and so gives us 2 coefficients for each. When we run the same logistic equation with Base SAS using &lt;code&gt;proc logistic&lt;/code&gt; we get only one coefficient for that categorical variable. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are these two output saying the same thing? How does Base SAS take into account the number of categories while giving just one coefficient for the categorical variable? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Further, is there any way of getting the same kind of output with Base SAS as I got in Entreprise Miner (basically, one with different coefficients for different category)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any additional pointers would be greatly appreciated.&lt;/p&gt;&#xA;" OwnerUserId="1763" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-09T17:53:55.920" LastActivityDate="2011-05-09T17:53:55.920" Title="Interpreting logistic regression output from SAS" Tags="&lt;logistic-regression&gt;&lt;sas&gt;" CommentCount="4" />
  <row Id="10570" PostTypeId="2" ParentId="10564" CreationDate="2011-05-09T18:03:19.990" Score="2" ViewCount="" Body="&lt;p&gt;Did you just fit one big glm model then look at the individual p-values?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Remember that those p-values are measuring the effect of that variable above and beyond all other variables in the model.  It is possible that more of your covariates are really contributing, but there is redundant information, so they don't show significance.  It could be the combination of A along with another covariate or 2 that shows the real difference, but A by itself is not meaningful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also look at the effect sizes and standard errors for all terms.  There is a paradox associated with the Wald tests/estimates (the individual p-values in the standard summary) that you can have a very important variable appear to be non-significant because the standard error is over estimated.  Likelihood ratio tests are much better because of this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can assess both the above by fitting a reduced model using only A, then use the anova function to compare the 2 models, if that is significant then it indicates that there is something important in your model beyond the A variable.&lt;/p&gt;&#xA;" OwnerUserId="4505" LastActivityDate="2011-05-09T18:03:19.990" CommentCount="2" />
  <row Id="10571" PostTypeId="1" CreationDate="2011-05-09T18:25:49.753" Score="0" ViewCount="129" Body="&lt;p&gt;&lt;strong&gt;How to do interpretation for Johansen Co-integration..(Results from Eviews).. For example..&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unrestricted Cointegration Rank Test (Trace)        &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hypothesized        Trace   0.05&lt;br&gt;&#xA;No. of CE(s)    Eigenvalue  Statistic   Critical Value  Prob.** &lt;/p&gt;&#xA;&#xA;&lt;p&gt;None *   0.644375    112.7175    69.81889    0.0000 &#xA;At most 1 *  0.541077    66.19295    47.85613    0.0004 &#xA;At most 2 *  0.446395    31.14371    29.79707    0.0348 &#xA;At most 3    0.080995    4.535061    15.49471    0.8560 &#xA;At most 4    0.016183    0.734208    3.841466    0.3915 &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Trace test indicates 3 cointegrating eqn(s) at the 0.05 level&lt;br&gt;&#xA; * denotes rejection of the hypothesis at the 0.05 level&lt;br&gt;&#xA; **MacKinnon-Haug-Michelis (1999) p-values      &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unrestricted Cointegration Rank Test (Maximum Eigenvalue)   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hypothesized        Max-Eigen   0.05&lt;br&gt;&#xA;No. of CE(s)    Eigenvalue  Statistic   Critical Value  Prob.** &lt;/p&gt;&#xA;&#xA;&lt;p&gt;None *   0.644375    46.52456    33.87687    0.0010 &#xA;At most 1 *  0.541077    35.04924    27.58434    0.0046 &#xA;At most 2 *  0.446395    26.60865    21.13162    0.0077 &#xA;At most 3    0.080995    3.800853    14.26460    0.8798 &#xA;At most 4    0.016183    0.734208    3.841466    0.3915 &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Max-eigenvalue test indicates 3 cointegrating eqn(s) at the 0.05 level &#xA; * denotes rejection of the hypothesis at the 0.05 level&lt;br&gt;&#xA; **MacKinnon-Haug-Michelis (1999) p-values      &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unrestricted Cointegrating Coefficients (normalized by b'*S11*b=I):    &lt;/p&gt;&#xA;&#xA;&lt;p&gt;GDP FDI TP  LF  GCF &#xA; 0.038737    0.224438    0.136168    0.778706    0.601672&lt;br&gt;&#xA;-0.355583   -0.902010    0.430649   -0.119028   -1.155575&lt;br&gt;&#xA;-0.414174    1.159984    0.070270   -0.232217   -0.318090&lt;br&gt;&#xA;-0.076818   -1.260897   -0.002244    0.422635   -0.249978&lt;br&gt;&#xA; 0.156241    2.456015   -0.080264    0.282948   -0.145782   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unrestricted Adjustment Coefficients (alpha):      &lt;/p&gt;&#xA;&#xA;&lt;p&gt;D(GDP)   0.884944    0.426792    1.982606    0.673585    0.189664&#xA;D(FDI)   0.096660    0.037420    0.025205    0.009629   -0.033736&#xA;D(TP)   -2.163443   -0.893406    0.551412    0.345264   -0.094556&#xA;D(LF)   -0.317541    0.075557    0.269763   -0.116193   -0.013982&#xA;D(GCF)  -0.109865    0.865984   -0.190618    0.024741   -0.050355&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1 Cointegrating Equation(s):    Log likelihood  -349.3343       &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Normalized cointegrating coefficients (standard error in parentheses)&lt;br&gt;&#xA;GDP FDI TP  LF  GCF &#xA; 1.000000    5.793883    3.515193    20.10236    15.53222&lt;br&gt;&#xA;     (11.1720)   (1.18384)   (3.31074)   (3.48248)  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Adjustment coefficients (standard error in parentheses)&lt;br&gt;&#xA;D(GDP)   0.034280&lt;br&gt;&#xA;     (0.03054)&lt;br&gt;&#xA;D(FDI)   0.003744&lt;br&gt;&#xA;     (0.00208)&lt;br&gt;&#xA;D(TP)   -0.083805&lt;br&gt;&#xA;     (0.01904)&lt;br&gt;&#xA;D(LF)   -0.012301&lt;br&gt;&#xA;     (0.00468)&lt;br&gt;&#xA;D(GCF)  -0.004256&lt;br&gt;&#xA;     (0.00937)              &lt;/p&gt;&#xA;&#xA;&lt;p&gt;2 Cointegrating Equation(s):    Log likelihood  -331.8097       &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Normalized cointegrating coefficients (standard error in parentheses)&lt;br&gt;&#xA;GDP FDI TP  LF  GCF &#xA; 1.000000    0.000000   -4.891974   -15.06039   -6.315809&lt;br&gt;&#xA;         (0.92638)   (2.69829)   (2.83156)&lt;br&gt;&#xA; 0.000000    1.000000    1.451042    6.068944    3.770879&lt;br&gt;&#xA;         (0.35213)   (1.02567)   (1.07633)  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Adjustment coefficients (standard error in parentheses)&lt;br&gt;&#xA;D(GDP)  -0.117480   -0.186356&lt;br&gt;&#xA;     (0.28048)   (0.72887)&lt;br&gt;&#xA;D(FDI)  -0.009562   -0.012059&lt;br&gt;&#xA;     (0.01903)   (0.04945)&lt;br&gt;&#xA;D(TP)    0.233875    0.320302&lt;br&gt;&#xA;     (0.16510)   (0.42904)&lt;br&gt;&#xA;D(LF)   -0.039168   -0.139422&lt;br&gt;&#xA;     (0.04287)   (0.11141)&lt;br&gt;&#xA;D(GCF)  -0.312185   -0.805784&lt;br&gt;&#xA;     (0.06371)   (0.16557)          &lt;/p&gt;&#xA;&#xA;&lt;p&gt;3 Cointegrating Equation(s):    Log likelihood  -318.5054       &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Normalized cointegrating coefficients (standard error in parentheses)&lt;br&gt;&#xA;GDP FDI TP  LF  GCF &#xA; 1.000000    0.000000    0.000000    3.100752    3.508476&lt;br&gt;&#xA;             (0.48329)   (0.52704)&lt;br&gt;&#xA; 0.000000    1.000000    0.000000    0.682044    0.856830&lt;br&gt;&#xA;             (0.14025)   (0.15295)&lt;br&gt;&#xA; 0.000000    0.000000    1.000000    3.712436    2.008246&lt;br&gt;&#xA;             (0.57761)   (0.62988)  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Adjustment coefficients (standard error in parentheses)&lt;br&gt;&#xA;D(GDP)  -0.938623    2.113434    0.443617&lt;br&gt;&#xA;     (0.37697)   (1.02393)   (0.31487)&lt;br&gt;&#xA;D(FDI)  -0.020001    0.017179    0.031048&lt;br&gt;&#xA;     (0.02900)   (0.07876)   (0.02422)&lt;br&gt;&#xA;D(TP)    0.005494    0.959931   -0.640589&lt;br&gt;&#xA;     (0.24608)   (0.66840)   (0.20554)&lt;br&gt;&#xA;D(LF)   -0.150896    0.173499    0.008256&lt;br&gt;&#xA;     (0.05937)   (0.16125)   (0.04959)&lt;br&gt;&#xA;D(GCF)  -0.233236   -1.026897    0.344581&lt;br&gt;&#xA;     (0.09546)   (0.25930)   (0.07974)      &lt;/p&gt;&#xA;&#xA;&lt;p&gt;4 Cointegrating Equation(s):    Log likelihood  -316.6049       &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Normalized cointegrating coefficients (standard error in parentheses)&lt;br&gt;&#xA;GDP FDI TP  LF  GCF &#xA; 1.000000    0.000000    0.000000    0.000000    1.268977&lt;br&gt;&#xA;                 (1.10846)&lt;br&gt;&#xA; 0.000000    1.000000    0.000000    0.000000    0.364228&lt;br&gt;&#xA;                 (0.23575)&lt;br&gt;&#xA; 0.000000    0.000000    1.000000    0.000000   -0.673039&lt;br&gt;&#xA;                 (1.31040)&lt;br&gt;&#xA; 0.000000    0.000000    0.000000    1.000000    0.722244&lt;br&gt;&#xA;                 (0.35981)  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Adjustment coefficients (standard error in parentheses)&lt;br&gt;&#xA;D(GDP)  -0.990367    1.264113    0.442105    0.462597&lt;br&gt;&#xA;     (0.37411)   (1.31957)   (0.30945)   (0.62528)&lt;br&gt;&#xA;D(FDI)  -0.020741    0.005037    0.031026    0.069032&lt;br&gt;&#xA;     (0.02926)   (0.10322)   (0.02420)   (0.04891)&lt;br&gt;&#xA;D(TP)   -0.021028    0.524588   -0.641364   -1.560472&lt;br&gt;&#xA;     (0.24586)   (0.86721)   (0.20337)   (0.41092)&lt;br&gt;&#xA;D(LF)   -0.141970    0.320007    0.008517   -0.368015&lt;br&gt;&#xA;     (0.05871)   (0.20708)   (0.04856)   (0.09812)&lt;br&gt;&#xA;D(GCF)  -0.235137   -1.058094    0.344525   -0.133908&lt;br&gt;&#xA;     (0.09636)   (0.33990)   (0.07971)   (0.16106)  &lt;/p&gt;&#xA;" OwnerUserId="4517" LastActivityDate="2011-05-09T18:25:49.753" ClosedDate="2011-05-09T18:55:54.037" Title="Johansen Co-integration Interpretation With Example" Tags="&lt;multivariate-analysis&gt;&lt;cointegration&gt;" CommentCount="2" />
  <row Id="10572" PostTypeId="2" ParentId="10532" CreationDate="2011-05-09T18:28:15.963" Score="2" ViewCount="" Body="&lt;p&gt;&lt;em&gt;Statistical models in S&lt;/em&gt;, by Chambers and Hastie (Chapmann and Hall, 1991; or the so-called &lt;em&gt;White Book&lt;/em&gt;), and to a lesser extent &lt;em&gt;Modern Applied Statistics with S&lt;/em&gt;, by Venables and Ripley (Springer, 2002, 4th ed.), include some material about DoE and the analysis of common designs in S and R. Vikneswaran wrote &lt;a href=&quot;http://cran.r-project.org/doc/contrib/Vikneswaran-ED_companion.pdf&quot; rel=&quot;nofollow&quot;&gt;An R companion to &quot;Experimental Design&quot;&lt;/a&gt;, although it is not very complete (IMHO), but there are a lot other textbooks in the &lt;a href=&quot;http://cran.r-project.org/other-docs.html&quot; rel=&quot;nofollow&quot;&gt;Contributed&lt;/a&gt; section on CRAN that might help you get started.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Apart from textbook, the CRAN Task View on &lt;a href=&quot;http://cran.r-project.org/web/views/ExperimentalDesign.html&quot; rel=&quot;nofollow&quot;&gt;Design of Experiments (DoE) &amp;amp; Analysis of Experimental Data&lt;/a&gt; has some good packages that ease the creation and analysis of various experimental designs; I can think of &lt;a href=&quot;http://cran.r-project.org/web/packages/dae/index.html&quot; rel=&quot;nofollow&quot;&gt;dae&lt;/a&gt;, &lt;a href=&quot;http://cran.r-project.org/web/packages/agricolae/index.html&quot; rel=&quot;nofollow&quot;&gt;agricolae&lt;/a&gt;, or &lt;a href=&quot;http://cran.r-project.org/web/packages/AlgDesign/index.html&quot; rel=&quot;nofollow&quot;&gt;AlgDesign&lt;/a&gt; (which comes with a nice &lt;a href=&quot;http://cran.r-project.org/web/packages/AlgDesign/vignettes/AlgDesign.pdf&quot; rel=&quot;nofollow&quot;&gt;vignette&lt;/a&gt;), to name a few.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-05-09T18:28:15.963" />
  <row Id="10573" PostTypeId="2" ParentId="10567" CreationDate="2011-05-09T18:33:42.420" Score="10" ViewCount="" Body="&lt;p&gt;The condition that &lt;em&gt;dependent&lt;/em&gt; variables must be &quot;continuous and unbounded&quot; is unusual: there is no mathematical or statistical requirement for either.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In most regression models we posit that the dependent variable be a linear combination of the independent variables plus an independent random error term of zero mean, &lt;em&gt;approximately&lt;/em&gt; and &lt;em&gt;within the ranges attained by, or potentially attained by, the independent variables.&lt;/em&gt;  For instance, it would be fine to regress the length of the Mississippi River on time for the period 1700 - 1850 but not to project the regression back, say, a million years or forward 700 years:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In the space of one hundred and seventy-six years the Lower Mississippi has shortened itself two hundred and forty-two miles. That is an average of a trifle over one mile and a third per year. Therefore, any calm person, who is not blind or idiotic, can see that in the Old Oolitic Silurian Period, just a million years ago next November, the Lower Mississippi River was upwards of one million three hundred thousand miles long, and stuck out over the Gulf of Mexico like a fishing-rod. And by the same token any person can see that seven hundred and forty-two years from now the Lower Mississippi will be only a mile and three-quarters long, and Cairo and New Orleans will have joined their streets together, and be plodding comfortably along under a single mayor and a mutual board of aldermen. There is something fascinating about science. One gets such wholesale returns of conjecture out of such a trifling investment of fact.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;(Mark Twain, &lt;em&gt;Life on the Mississippi&lt;/em&gt;.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the present case it sounds like the angle is an &lt;em&gt;independent&lt;/em&gt; variable, not the dependent one, so this question does not even arise.  The problem that arises is that the angle seems to be defined only modulo 360 degrees (actually mod 180).  Actually, the angle is really a latitude and varies from 0 to 180 (or -90 to 90) without &quot;wrapping around&quot; at all.  Really, then, all that matters is how best to express this angle: does the reaction rate vary linearly with the &lt;em&gt;angle&lt;/em&gt; or does it vary perhaps with its sine or cosine?  Or maybe its tangent, which &lt;em&gt;is&lt;/em&gt; unbounded?  But that matter is addressed with appropriate exploratory analysis, perhaps by some stereochemical considerations, and standard procedures to fit and check models.  Therefore &lt;strong&gt;this angular variable neither enjoys nor suffers from any special quality that would distinguish it from other independent variables.&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-05-09T18:33:42.420" CommentCount="2" />
  <row Id="10574" PostTypeId="1" CreationDate="2011-05-09T18:39:54.350" Score="5" ViewCount="80" Body="&lt;p&gt;I have two independent poisson random variables, $X_1$ and $X_2$, with $X_1 \sim \text{Pois}(\lambda_1)$ and $X_2 \sim \text{Pois}(\lambda_2)$. I want to test $H_0:\, \lambda_1 = \lambda_2$ versus the alternative $H_1:\, \lambda_1 \neq \lambda_2$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I already derived maximum likelihood estimates under null and alternate hypothesis (model), and based on those I calculated likelihood ratio test (LRT) statistic (R codes given below).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I am interested to calculate power of the test based on: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Fixed alpha (type 1 error) = 0.05.&lt;/li&gt;&#xA;&lt;li&gt;Using different sample sizes (n), say n = 5, 10, 20, 50, 100.&lt;/li&gt;&#xA;&lt;li&gt;Different combination of $\lambda_1$ and $\lambda_2$, which will change LRT statistics (computed as &lt;code&gt;LRTstat&lt;/code&gt; below).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Here is my R code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X1 = rpois(λ1); X2 = rpois(λ2)&#xA;Xbar = (X1+X2)/2&#xA;LLRNum = dpois(X1, X1) * dpois(X2, X2)&#xA;LLRDenom = dpois(X1, Xbar) * dpois(X2, Xbar)&#xA;LRTstat = 2*log(LLRNum/LLRDenom)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;From here, how could I proceed with power calculation (preferably in R)?&lt;/p&gt;&#xA;" OwnerUserId="4098" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-09T18:44:41.843" LastActivityDate="2011-05-09T20:18:48.010" Title="Power calculation for likelihood ratio test" Tags="&lt;poisson&gt;&lt;power&gt;&lt;likelihood-ratio&gt;" AnswerCount="1" />
  <row Id="10576" PostTypeId="2" ParentId="10574" CreationDate="2011-05-09T19:07:42.993" Score="7" ViewCount="" Body="&lt;p&gt;You can do this using simulation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Write a function that does your test and accepts the lambdas and sample size(s) as arguments (you have a good start above).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now for a given set of lambdas and sample size(s) run the function a bunch of times (the replicate function in R is great for that).  Then the power is just the proportion of times that you reject the null hypothesis, you can use the mean function to compute the proportion and prop.test to give a confidence interval on the power.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is some example code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;tmpfunc1 &amp;lt;- function(l1, l2=l1, n1=10, n2=n1) {&#xA;    x1 &amp;lt;- rpois(n1, l1)&#xA;    x2 &amp;lt;- rpois(n2, l2)&#xA;    m1 &amp;lt;- mean(x1)&#xA;    m2 &amp;lt;- mean(x2)&#xA;    m &amp;lt;- mean( c(x1,x2) )&#xA;&#xA;    ll &amp;lt;- sum( dpois(x1, m1, log=TRUE) ) + sum( dpois(x2, m2, log=TRUE) ) - &#xA;            sum( dpois(x1, m, log=TRUE) ) - sum( dpois(x2, m, log=TRUE) )&#xA;    pchisq(2*ll, 1, lower=FALSE)&#xA;}&#xA;&#xA;# verify under null n=10&#xA;&#xA;out1 &amp;lt;- replicate(10000, tmpfunc1(3))&#xA;mean(out1 &amp;lt;= 0.05)&#xA;hist(out1)&#xA;prop.test( sum(out1&amp;lt;=0.05), 10000 )$conf.int&#xA;&#xA;# power for l1=3, l2=3.5, n1=n2=10&#xA;out2 &amp;lt;- replicate(10000, tmpfunc1(3,3.5))&#xA;mean(out2 &amp;lt;= 0.05)&#xA;hist(out2)&#xA;&#xA;# power for l1=3, l2=3.5, n1=n2=50&#xA;out3 &amp;lt;- replicate(10000, tmpfunc1(3,3.5,n1=50))&#xA;mean(out3 &amp;lt;= 0.05)&#xA;hist(out3)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My results (your will differ with a different seed, but should be similar) showed a type I error rate (alpha) of 0.0496 (95% CI 0.0455-0.0541) which is close to 0.05, more precision can be obtained by increasing the 10000 in the replicate command.  The powers I computed were: 9.86% and 28.6%.  The histograms are not strictly necessary, but I like seeing the patterns.&lt;/p&gt;&#xA;" OwnerUserId="4505" LastEditorUserId="4505" LastEditorDisplayName="" LastEditDate="2011-05-09T20:18:48.010" LastActivityDate="2011-05-09T20:18:48.010" CommentCount="2" />
  <row Id="10577" PostTypeId="2" ParentId="10546" CreationDate="2011-05-09T19:53:46.250" Score="2" ViewCount="" Body="&lt;p&gt;I'm sure that Durrett's proof is nice. A straight forward solution to the question asked is as follows.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For $n \geq 1$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;\begin{array}{rcl}&#xD;&#xA;P(N_t = n) &amp;amp; = &amp;amp; \int_0^t P(S_{n+1} &amp;gt; t \mid S_n = s) P(S_n \in ds) \\&#xD;&#xA;&amp;amp; = &amp;amp; \int_0^t P(T_{n+1} &amp;gt; t-s) P(S_n \in ds) \\&#xD;&#xA;&amp;amp; = &amp;amp; \int_0^t e^{-\lambda(t-s)} \frac{\lambda^n s^{n-1} e^{-\lambda s}}{(n-1)!} \mathrm{d} s \\&#xD;&#xA;&amp;amp; = &amp;amp; e^{-\lambda t} \frac{\lambda^n }{(n-1)!} \int_0^t s^{n-1}  \mathrm{d} s \\&#xD;&#xA;&amp;amp; = &amp;amp; e^{-\lambda t} \frac{(\lambda t)^n}{n!}&#xD;&#xA;\end{array}&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For $n = 0$ we have $P(N_t = 0) = P(T_1 &amp;gt; t) = e^{-\lambda t}$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This does not prove that $(N_t)_{t \geq 0}$ is a Poisson process, which is harder, but it does show that the marginal distribution of $N_t$ is Poisson with mean $\lambda t$.&lt;/p&gt;&#xA;" OwnerUserId="4376" LastEditorUserId="4376" LastEditorDisplayName="" LastEditDate="2011-05-09T21:13:10.660" LastActivityDate="2011-05-09T21:13:10.660" CommentCount="1" />
  <row Id="10578" PostTypeId="1" CreationDate="2011-05-09T20:43:10.830" Score="5" ViewCount="154" Body="&lt;p&gt;I am not comfortable with Fisher information, what it measures and how and how is it helpful. Also it's relationship with the Cramer-Rao bound is not apparent to me.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone please give an intuitive explanation of these concepts?&lt;/p&gt;&#xA;" OwnerUserId="4101" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-10T05:53:19.687" LastActivityDate="2011-05-10T15:09:16.277" Title="Intuitive explanation of Fisher Information and Cramer-Rao bound" Tags="&lt;estimation&gt;" AnswerCount="2" CommentCount="6" FavoriteCount="5" />
  <row Id="10579" PostTypeId="1" CreationDate="2011-05-09T22:20:50.180" Score="1" ViewCount="113" Body="&lt;p&gt;I have 4 groups each consisting of 1000 recent college graduates, and I am studying employment immediately after graduation. 2 groups took a business class in college, and 2 groups took a math course in college. Group 1 is a control that has taken neither a business class nor a math course in college. I want to conduct a statistical analysis to find if either class was significant in increasing the graduate in obtaining a job immediately after school. Here's the data:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Group 1: No Math, No Business&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1000 subjects, 50 employed&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Group 2: No Math, Yes Business&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1000 subjects, 60 employed&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Group 3: Yes Math, No Business&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1000 subjects, 75 employed&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Group 4: Yes Math, Yes Business&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1000 subjects, 100 employed&lt;/p&gt;&#xA;" OwnerUserId="4538" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-11T15:29:29.850" LastActivityDate="2011-05-11T15:29:29.850" Title="Testing significant effect in 2 by 2 factor design on a binary outcome" Tags="&lt;statistical-significance&gt;" AnswerCount="5" />
  <row Id="10580" PostTypeId="2" ParentId="10432" CreationDate="2011-05-09T22:51:32.127" Score="3" ViewCount="" Body="&lt;p&gt;It sounds like any linear classifier will do what you need. Suppose you have $N$ features and the value of feature $i$ is $f_i$. Then a linear classifier will compute a score &#xA;$$s = \sum_i w_i f_i + o$$ (where $o$ is the offset). Then, if $s &amp;gt; t$ (where $t$ is some threshold), then the feature belongs to a class (a group), and if $s &amp;lt; t$, then it doesn't. Note that there is a single threshold applied to the entire score (rather than to individual feature values), so indeed a deficiency in one parameter can be compensated for by abundance in another. The weights are intuitively interpretable, in the sense that the higher the weight is, the more important (or more decisive) that feature is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are a lot of off-the-shelf linear classifiers that can do that, including SVM, LDA (linear discriminant analysis), linear neural networks, and many others. I'd start by running linear SVM because it works well in a lot of cases and can tolerate limited training data. There are also a lot of packages in many environments (like Matlab and R), so you can easily try it. The downside of SVM is that it can be computationally heavy, so if you need to learn a lot of classes, it might be less appropriate. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to preserve some of the threshold behavior you currently have, you can pass the feature values through a sigmoid with the threshold in the right location. E.g. for a feature $i$ for which you currently use a threshold of $t_i$, first compute &#xA;$$g_i = \frac{1}{1 + \exp(f_i - t_i)},$$&#xA;and then learn a linear classifier using $g$'s rather than $f$'s. This way, the compensating behavior will only happen near the threshold, and things that are too far away from the threshold cannot be compensated for (which is sometimes desirable).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another thing that you could try is to use probabilistic classifiers like Naive Bayes or TAN. Naive Bayes is almost like a linear classifier, except it computes&#xA;$$s = \sum_i w^i_{f_i}.$$&#xA;So there is still a sum of weights. These weights depend on the feature values $f_i$, but not by multiplication like in a usual linear classifier. The score in this case is the log-probability, and the weights are the contributions of the individual features into that log-probability. The disadvantage of using this in your case is that you will need many bins for your feature values, and then learning may become difficult. There are ways around that (for example, using priors), but since you have no experience with this, it might be more difficult.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regarding terminology: what you called 'test set' is usually called a 'training set' in this context, and what you called 'new data' is called the 'test set'.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a book, I'd read &quot;Pattern recognition&quot; by Duda, Hart, and Stork. The first chapter is a very good introduction for beginners.&lt;/p&gt;&#xA;" OwnerUserId="3369" LastEditorUserId="3369" LastEditorDisplayName="" LastEditDate="2011-05-09T22:58:25.560" LastActivityDate="2011-05-09T22:58:25.560" CommentCount="2" />
  <row Id="10581" PostTypeId="2" ParentId="10578" CreationDate="2011-05-09T23:09:17.680" Score="7" ViewCount="" Body="&lt;p&gt;Here I explain why the asymptotic variance of the &lt;em&gt;maximum likelihood estimator&lt;/em&gt; is the Cramer-Rao lower bound.  Hopefully this will provide some insight as to the relevance of the Fisher information.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Statistical inference proceeds with the use of a likelihood function $\mathcal{L}(\theta)$ which you construct from the data.  The point estimate $\hat{\theta}$ is the value which maximizes $\mathcal{L}(\theta)$. The estimator $\hat{\theta}$ is a random variable, but it helps to realize that the &lt;em&gt;likelihood function&lt;/em&gt; $\mathcal{L}(\theta)$ is a &quot;random curve&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here we assume iid data drawn from a distribution $f(x|\theta)$, and we define the likelihood&#xA;$$&#xD;&#xA;\mathcal{L}(\theta) = \frac{1}{n}\sum_{i=1}^n \log f(x_i|\theta)&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The parameter $\theta$ has the property that it maximizes the value of the &quot;true&quot; likelihood, $\mathbb{E}\mathcal{L}(\theta)$.  However, the &quot;observed&quot; likelihood function $\mathcal{L}(\theta)$ which is constructed from the data is slightly &quot;off&quot; from the true likelihood.  Yet as you can imagine, as the sample size increases, the &quot;observed&quot; likelihood converges to the shape of the true likelihood curve.  The same applies to the derivative of the likelihood with respect to the parameter, &lt;em&gt;the score function&lt;/em&gt; $\partial \mathcal{L}/\partial \theta$.  (Long story short, the Fisher information determines &lt;em&gt;how quickly&lt;/em&gt; the observed score function converges to the shape of the true score function.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At a large sample size, we assume that our maximum likelihood estimate $\hat{\theta}$ is very close to $\theta$.  We zoom into a small neighborhood around $\theta$ and $\hat{\theta}$ so that the likelihood function is &quot;locally quadratic&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There, $\hat{\theta}$ is the point at which the &lt;em&gt;score function&lt;/em&gt; $\partial \mathcal{L}/\partial \theta$ intersects the origin.  In this small region, we treat the score function as a &lt;em&gt;line&lt;/em&gt;, one with slope $a$ and  &lt;em&gt;random&lt;/em&gt; intercept $b$ at $\theta$.  We know from the equation for a line that&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$a(\hat{\theta} - \theta)  + b = 0$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;or&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;\hat{\theta} = \theta - b/a .&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the consistency of the MLE estimator, we know that&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;\mathbb{E}(\hat{\theta}) = \theta&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;in the limit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, asymptotically&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;nVar(\hat{\theta}) = nVar(b/a)&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It turns out that the slope varies much less than the intercept, and asymptotically, we can treat the score function as having a constant slope in a small neighborhood around $\theta$.  Thus we can write&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;nVar(\hat{\theta}) = \frac{1}{a^2}nVar(b)&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, what are the values of $a$ and $nVar(b)$?  It turns out that due to a marvelous mathematical coincidence, they are the very same quantity (modulo a minus sign), the Fisher information.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$-a = \mathbb{E}\left[-\frac{\partial^2 \mathcal{L}}{\partial \theta^2}\right] = I(\theta)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$nVar(b) = nVar\left[\frac{\partial \mathcal{L}}{\partial \theta}\right] = I(\theta)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;nVar(\hat{\theta}) = \frac{1}{a^2}nVar(b) = (1/I(\theta)^2)I(\theta) = 1/I(\theta)&#xD;&#xA;$$&#xA;asymptotically: the Cramer-Rao lower bound.  (Showing that $1/I(\theta)$ is a the lower bound on the variance of an unbiased estimator is another matter.)&lt;/p&gt;&#xA;" OwnerUserId="3567" LastEditorUserId="3567" LastEditorDisplayName="" LastEditDate="2011-05-10T15:09:16.277" LastActivityDate="2011-05-10T15:09:16.277" />
  <row Id="10582" PostTypeId="2" ParentId="10579" CreationDate="2011-05-09T23:28:35.240" Score="2" ViewCount="" Body="&lt;p&gt;This seems like a classic problem for &lt;a href=&quot;http://en.wikipedia.org/wiki/Logistic_regression&quot; rel=&quot;nofollow&quot;&gt;logistic regression&lt;/a&gt;.  Rather than specifying these groups, turn math and business coursework into predictors for employment status.  This is easy to code up in pretty much whatever statistical software you have around, although you'll want to look into different contrasts (it sounds like you'd want to use treatment contrasts with &quot;no class&quot; set as the reference level).&lt;/p&gt;&#xA;" OwnerUserId="71" LastActivityDate="2011-05-09T23:28:35.240" />
  <row Id="10583" PostTypeId="2" ParentId="10579" CreationDate="2011-05-09T23:52:24.900" Score="2" ViewCount="" Body="&lt;p&gt;Self-selection problem need to be addressed here. People who choose to take business classes might be more likely to get a job. So, the sample you have is not random, and the inferences you draw might be incorrect. Heckman procedure is used to correct to self-selection bias. I am not sure if it is applicable for discrete dependent variable though.&lt;/p&gt;&#xA;" OwnerUserId="4540" LastActivityDate="2011-05-09T23:52:24.900" CommentCount="3" />
  <row Id="10585" PostTypeId="2" ParentId="10579" CreationDate="2011-05-10T00:59:05.887" Score="1" ViewCount="" Body="&lt;p&gt;One of the more basic approaches you could take is a &lt;a href=&quot;http://udel.edu/~mcdonald/stattwoway.html&quot; rel=&quot;nofollow&quot;&gt;two-way ANOVA&lt;/a&gt; (Page from U Delaware), using Math Class = {Y, N} and Business Class = {Y, N} as your two treatments. You would then perform an analysis of variance on the dependent variable (number of people employed) to determine whether taking a math/business class has an impact on how many people were able to get jobs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One thing you may want to watch out for, however, are some of the assumptions of ANOVA, which may not be appropriate. You can find the assumptions on &lt;a href=&quot;http://en.wikipedia.org/wiki/Analysis_of_variance#Assumptions_of_ANOVA&quot; rel=&quot;nofollow&quot;&gt;this wikipedia page&lt;/a&gt;. Normality of the residuals may be questionable if there was no random assignment, in which case, there are &lt;a href=&quot;http://en.wikipedia.org/wiki/Kruskal%E2%80%93Wallis_test&quot; rel=&quot;nofollow&quot;&gt;non-parametric methods&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1118" LastActivityDate="2011-05-10T00:59:05.887" />
  <row Id="10586" PostTypeId="2" ParentId="10579" CreationDate="2011-05-10T00:59:45.170" Score="2" ViewCount="" Body="&lt;p&gt;If @EEE's concern can be addressed and you proceed with an hypothesis test, then rather than logistic regression I'd recommend a chi-square test.  For a person fairly new to statistical testing, it'll be dramatically easier to conduct, interpret, and explain to an audience.  Plus I think it'll give you just about as much information.&lt;/p&gt;&#xA;" OwnerUserId="2669" LastActivityDate="2011-05-10T00:59:45.170" />
  <row Id="10587" PostTypeId="1" CreationDate="2011-05-10T01:47:23.723" Score="3" ViewCount="49" Body="&lt;p&gt;I lack knowledge of statistical terminology, so I'll try to thoroughly explain my predicament and hope that it is understandable:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am programming a software for technical analysis of financial markets.&lt;br&gt;&#xA;This software will receive a variable who's value represents current market conditions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this particular case the variable represents a measurement, in standard deviations, of the financial vehicle's price in respect to its moving average. The exact nature of this variable is not relevant to my question - the important part is that it has non-uniform distribution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I mean by this is that at any given time the variable has a much greater chance of being somewhere around 0 than of being at one of the extremes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, the point of my software is to find and evaluate &lt;em&gt;similar conditions&lt;/em&gt; in the market history. If I look for the exact same value (ex. 2.0021) I will not find it; so what I must do is create an interval around 2.0021 and evaluate whatever values fall within the interval.    &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, when I receive a different input value, I will need to create an interval of a &lt;strong&gt;different size,&lt;/strong&gt; such that approximately &lt;strong&gt;the same amount of values will be found within this interval.&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Here is a concrete example:&lt;/h2&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;code&gt;CURRENT VALUE X&lt;/code&gt;&lt;sub&gt;a&lt;/sub&gt; &lt;code&gt;=  2.0021&lt;/code&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Lets say I found that &lt;code&gt;100 values of X&lt;/code&gt;&lt;sub&gt;a&lt;/sub&gt; fall within the interval of  &lt;code&gt;1.9992 &amp;lt; X &amp;lt; 2.0050&lt;/code&gt;...&lt;br&gt;&#xA;(The size of the interval (obviously) is &lt;code&gt;0.0058&lt;/code&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Next I receive another value for X:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;code&gt;CURRENT VALUE X&lt;/code&gt;&lt;sub&gt;b&lt;/sub&gt; &lt;code&gt;=  4.5010&lt;/code&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Values around 4.5 are much more rare than values around 2.08&lt;/strong&gt;. Therefore, the interval surrounding &lt;code&gt;4.5010&lt;/code&gt; must also be larger in order for my software to find the same number of results as  (close to)  X&lt;sub&gt;a&lt;/sub&gt; ;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am trying to create intervals in my data such that each interval contains the same number of samples within its bounds.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words:&#xA;The &lt;code&gt;CLOSER TO 0&lt;/code&gt;, the more data points are present, the &lt;code&gt;SMALLER&lt;/code&gt; the interval needs to be.&#xA;The farther from 0, the fewer data points are present, the greater the interval needs to be in order to contain the same amount of results. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;And now to the heart of my question:&#xA;How can I formulaically calculate the size of the interval around &lt;code&gt;X&lt;/code&gt; based on the value of &lt;code&gt;X&lt;/code&gt;? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is also crucial to mention that I have a ton (500,000) of real &lt;code&gt;X&lt;/code&gt; values to work from.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is what I tried - I feel like I'm close but not quite there:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I created a histogram out of my 1D data.&#xA;Since the data is symmetric, and centered on 0, I created another histogram based on the data's absolute value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://i.stack.imgur.com/oPKWc.png&quot; rel=&quot;nofollow&quot;&gt;My Histograms&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I do not know where to go from here :(&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried doing an exponential regression to the data; it fits very well but I do not know how to use the formula to create the sort of bounds within my data that I want.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I seriously regret not taking a Stats class in college...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone help me out please?&lt;/p&gt;&#xA;" OwnerUserId="4542" LastActivityDate="2011-05-10T01:58:11.990" Title="Uniform frequency from non-uniform (exponential) distribution?" Tags="&lt;distributions&gt;&lt;probability&gt;&lt;confidence-interval&gt;&lt;histogram&gt;" AnswerCount="1" />
  <row Id="10588" PostTypeId="2" ParentId="10587" CreationDate="2011-05-10T01:58:11.990" Score="3" ViewCount="" Body="&lt;p&gt;If your data really is &lt;a href=&quot;http://en.wikipedia.org/wiki/Exponential_random_variable&quot; rel=&quot;nofollow&quot;&gt;exponentially distributed&lt;/a&gt;, find the &lt;a href=&quot;http://en.wikipedia.org/wiki/Exponential_random_variable#Maximum_likelihood&quot; rel=&quot;nofollow&quot;&gt;maximum likelihood estimate of the rate&lt;/a&gt; $\lambda$, then transform the samples $X_n$ to a sequence $Y_n$ uniformly distributed in $[0,1]$ using the equation $Y_n=1-\mathrm e^{-\lambda X_n}$&lt;/p&gt;&#xA;" OwnerUserId="4479" LastActivityDate="2011-05-10T01:58:11.990" CommentCount="3" />
  <row Id="10589" PostTypeId="2" ParentId="10578" CreationDate="2011-05-10T07:19:38.177" Score="3" ViewCount="" Body="&lt;p&gt;One way that I understand the fisher information is by the following definition:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$I(\theta)=\int_{\cal{X}} \frac{\partial^{2}f(x|\theta)}{\partial \theta^{2}}dx-\int_{\cal{X}} f(x|\theta)\frac{\partial^{2}}{\partial \theta^{2}}log[f(x|\theta)]dx$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Fisher Information can be written this way whenever the density $f(x|\theta)$ is twice differentiable.  If the sample space $\cal{X}$ does not depend on the parameter $\theta$, then we can use the Leibniz integral formula to show that the first term is zero (differentiate both sides of $\int_{\cal{X}} f(x|\theta)dx=1$ twice and you get zero), and the second term is the &quot;standard&quot; definition.  I will take the case when the first term is zero.  The cases when it isn't zero aren't much use for understanding Fisher Information.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now when you do maximum likelihood estimation (insert &quot;regularity conditions&quot; here) you set&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{\partial}{\partial \theta}log[f(x|\theta)]=0$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And solve for $\theta$.  So the second derivative says how quickly the gradient is changing, and in a sense &quot;how far&quot; $\theta$ can depart from the MLE without making an appreciable change in the right hand side of the above equation.  Another way you can think of it is to imagine a &quot;mountain&quot; drawn on the paper - this is the log-likelihood function.  Solving the MLE equation above tells you where the peak of this mountain is located as a function of the random variable $x$.  The second derivative tells you how steep the mountain is - which in a sense tells you how easy it is to find the peak of the mountain.  Fisher information comes from taking the expected steepness of the peak, and so it has a bit of a &quot;pre-data&quot; interpretation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One thing that I still find curious is that its how steep the log-likelihood is and not how steep some other monotonic function of the likelihood is (perhaps related to &quot;proper&quot; scoring functions in decision theory? or maybe to the consistency axioms of entropy?).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Fisher information also &quot;shows up&quot; in many asymptotic analysis due to what is known as the Laplace approximation.  This basically due to the fact that any function with a &quot;well-rounded&quot; single maximum raise to a higher and higher power goes into a Gaussian function $\exp(-ax^{2})$ (similar to Central Limit Theorem, but slightly more general).  So when you have a large sample you are effectively in this position and you can write:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$f(data|\theta)=\exp(log[f(data|\theta)])$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And when you taylor expand the log-likelihood about the MLE:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$f(data|\theta)\approx [f(data|\theta)]_{\theta=\theta_{MLE}}\exp\left(-\frac{1}{2}\left[-\frac{\partial^{2}}{\partial \theta^{2}}log[f(data|\theta)]\right]_{\theta=\theta_{MLE}}(\theta-\theta_{MLE})^{2}\right)$$&#xA;and that second derivative of the log-likelihood shows up (but in &quot;observed&quot; instead of &quot;expected&quot; form).  What is usually done here is to make to further approximation:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$-\frac{\partial^{2}}{\partial \theta^{2}}log[f(data|\theta)]=n\left(-\frac{1}{n}\sum_{i=1}^{n}\frac{\partial^{2}}{\partial \theta^{2}}log[f(x_{i}|\theta)]\right)\approx nI(\theta)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which amounts to the usually good approximation of replacing a sum by an integral, but this requires that the data be independent.  So for large independent samples (given $\theta$) you can see that the Fisher information is how variable the MLE is, for various values of the MLE.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-05-10T07:19:38.177" CommentCount="1" />
  <row Id="10591" PostTypeId="1" AcceptedAnswerId="10661" CreationDate="2011-05-10T10:38:24.523" Score="6" ViewCount="76" Body="&lt;p&gt;The &lt;a href=&quot;http://en.wikipedia.org/wiki/Anscombe_transform&quot; rel=&quot;nofollow&quot;&gt;Anscombe transform&lt;/a&gt; is $a(x) = 2\sqrt{x+3/8}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone show me how to prove that an Anscombe-transformed version $Y = a(X)$ of a Poisson distributed random variable $X$ is approximately normal distributed (when $\lambda&gt;4$)? &lt;/p&gt;&#xA;" OwnerUserId="4496" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-05-12T13:02:02.343" LastActivityDate="2011-05-12T13:02:02.343" Title="Anscombe transform and normal approximation" Tags="&lt;distributions&gt;&lt;normal-distribution&gt;&lt;poisson&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="1" />
  <row Id="10592" PostTypeId="1" AcceptedAnswerId="10637" CreationDate="2011-05-10T10:38:49.923" Score="7" ViewCount="85" Body="&lt;p&gt;Given a data matrix $X$ of say 1000000 observations $\times$ 100 features,&#xA;is there a fast way to build a tridiagonal approximation&#xA;$A \approx cov(X)$ ?&lt;br&gt;&#xA;Then one could factor $A = L L^T$,&#xA;$L$ all 0 except $L_{i\ i-1}$ and $L_{i i}$,&#xA;and do fast decorrelation (whitening) by solving&#xA;$L x = x_{white}$. &#xA;(By &quot;fast&quot; I mean $O( size\  X )$.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Added, trying to clarify): I'm looking for a quick and dirty whitener&#xA;which is faster than full $cov(X)$ but better than diagonal.&#xA;Say that $X$ is $N$ data points $\times Nf$ features, e.g. 1000000$\times$ 100,&#xA;with features 0-mean.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) build $Fullcov = X^T X$, Cholesky factor it as $L L^T$,&#xA;solve $L x = x_{white}$ to whiten new $x$ s. &#xA;This is quadratic in the number of features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) diagonal: $x_{white} = x / \sigma(x)$&#xA; ignores cross-correlations completely.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One &lt;em&gt;could&lt;/em&gt; get a tridiagonal matrix from $Fullcov$&#xA;just by zeroing all entries outside the tridiagonal,&#xA;or not accumulating them in the first place.&#xA;And here I start sinking: there must be a better approximation,&#xA;perhaps hierarchical, block diagonal &amp;rarr; tridiagonal ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;hr&gt;&#xA;(Added 11 May): Let me split the question in two:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) is there a fast approximate $cov(X)$ ?&lt;br&gt;&#xA;No (whuber), one must look at all ${N \choose 2}$ pairs &#xA;(or have structure, or sample).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) given a $cov(X)$, how fast can one whiten new $x$ s ?&lt;br&gt;&#xA;Well, factoring $cov = L L^T$, $L$ lower triangular, once,&#xA;then solving $L x = x_{white}$ &#xA;is pretty fast; scipy.linalg.solve_triangular, for example, uses Lapack.&lt;br&gt;&#xA;I was looking for a yet faster whiten(), still looking.&lt;/p&gt;&#xA;" OwnerUserId="557" LastEditorUserId="557" LastEditorDisplayName="" LastEditDate="2011-05-15T09:34:53.137" LastActivityDate="2011-05-15T09:34:53.137" Title="How to calculate tridiagonal approximate covariance matrix, for fast decorrelation?" Tags="&lt;variance&gt;&lt;approximation&gt;&lt;variance-covariance&gt;" AnswerCount="2" CommentCount="5" />
  <row Id="10593" PostTypeId="2" ParentId="10594" CreationDate="2011-05-10T11:06:58.780" Score="2" ViewCount="" Body="&lt;p&gt;Suppose you have a cumulative distribution function $F$ of the variable in question. Suppose the value given is $x$, and the range is $[r_1,r_2]$ with $x\in[r_1,r_2]$. Then if you select the amount of values falling into that range $N$, the following should hold:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$F(r_1)-F(r_2)=\frac{N}{500 000}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is an equation with 2 unknown variables, so we need some restrictions to solve it. The popular one would be setting $r_1=x-\varepsilon/2$ and $r_2=x+\varepsilon/2$. This would give us the equation of one variable, which could be solved pretty easily, using any optimisation algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The only thing you need is the function $F$. You can either model it, or use some non-parametric estimate. In the latter case probably optimisation algorithm may be unnecessary, as it should be possible to work out the solution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note:&lt;/strong&gt; This is only one possible approach, depending on data it might not work.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-05-10T11:06:58.780" CommentCount="5" />
  <row Id="10594" PostTypeId="1" AcceptedAnswerId="10595" CreationDate="2011-05-10T09:23:10.113" Score="3" ViewCount="118" Body="&lt;p&gt;I have 500,000 values for a variable derived from financial markets. This variable has a arbitrary distribution. I need a formula that will allow me to select a range around any value of this variable such that an equal (or close to it) amount of values fall within that range. From what I understand, this means that I need to convert it from arbitrary distribution to uniform distribution. I have read (but barely understood) that what I am looking for is called &quot;probability integral transform.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone assist me with some code (Matlab preferred, but it doesn't really matter) to help me accomplish this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: I uploaded my dataset as a &lt;a href=&quot;http://furlender.com/data.csv&quot; rel=&quot;nofollow&quot;&gt;.csv file&lt;/a&gt; and &lt;a href=&quot;http://furlender.com/data.rar&quot; rel=&quot;nofollow&quot;&gt;compressed .rar file&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I used the empirical distribution function within MatLab and got the following plot:&lt;img src=&quot;http://i.stack.imgur.com/YErTM.png&quot; alt=&quot;empirical cumulative distribution&quot;&gt;&#xA;Does this look about right?&#xA;Here is a histogram of the raw data for reference: &#xA;&lt;img src=&quot;http://i.stack.imgur.com/Oyy1m.png&quot; alt=&quot;histogram of the raw data&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="4544" LastEditorUserId="4544" LastEditorDisplayName="" LastEditDate="2011-05-10T15:00:17.433" LastActivityDate="2011-05-10T15:00:17.433" Title="Converting arbitrary distribution to uniform one" Tags="&lt;probability&gt;&lt;matlab&gt;" AnswerCount="2" CommentCount="6" FavoriteCount="0" />
  <row Id="10595" PostTypeId="2" ParentId="10594" CreationDate="2011-05-10T10:11:43.367" Score="4" ViewCount="" Body="&lt;p&gt;If $X$ has the (cumulative) distribution function $F(x)=P(X&amp;lt;x)$, then $F(X)$ has a uniform distribution on $[0,1]$. You don't know what $F$ is, but with N = 500,000 data points you could simply use the empirical distribution function:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\hat{F}(x) = \frac{1}{N} \sum_{i=1}^N 1[x_i\leq x]$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $1[A]$ is the indicator function, $1[A]=1$ is $A$ is true and $1[A]=0$ if $A$ is false. The function $F$ is often also called the quantile function.&lt;/p&gt;&#xA;&#xA;&lt;hr /&gt;&#xA;&#xA;&lt;p&gt;In coding terms, once you've written your function &lt;code&gt;F&lt;/code&gt; you now have two objects, &lt;code&gt;x&lt;/code&gt; containing your data and &lt;code&gt;q&lt;/code&gt; containing the transformed data, so you could write a function &lt;code&gt;Finv&lt;/code&gt; which takes a number in [0,1] and returns the value of your sample distribution at that quantile (using linear interpolation or some other appropriate method for filling in the gaps).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now if you want to take e.g. 5% of the data either side of the value &lt;code&gt;x0&lt;/code&gt;, your range will be &lt;code&gt;Finv(F(x0) - 0.05)&lt;/code&gt; to &lt;code&gt;Finv(F(x0) + 0.05)&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2425" LastActivityDate="2011-05-10T10:26:19.683" CommentCount="5" />
  <row Id="10596" PostTypeId="1" CreationDate="2011-05-10T13:47:48.933" Score="0" ViewCount="19" Body="&lt;p&gt;n&amp;lt;-NULL&lt;/p&gt;&#xA;&#xA;&lt;p&gt;acr&amp;lt;-NULL&lt;/p&gt;&#xA;&#xA;&lt;p&gt;while((is.numeric(n)==F) &amp;amp; (acr&amp;lt;0 ¦acr&gt;1)){&lt;/p&gt;&#xA;&#xA;&lt;p&gt;print(&quot;enter a  positive integer and the average cancellation rate you want&quot;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;try(n&amp;lt;-scan(what=integer(),nmax=1), silent=TRUE);try(acr&amp;lt;-scan(what=double(),nmax=1), silent=TRUE)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would want the users of my program to enter a positive integer which I store in &quot;n&quot; and the second &#xA;entry which I keep in &quot;acr&quot; is a probability so it lies between 0 and 1.&#xA;(I don't want it to be exacly 0 or 1, though it could be according to probability theory).&#xA;So I want the user to keep on doing the entry until they are able to enter a positive interger for &quot;n&quot; and a probabilty value between 0 and 1 for &quot;acr&quot;.(using while with the AND, OR operators)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I am having a problem with the while statement/loop.&#xA; I have tried all other possibilities such as the one below, but it still doesn't work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;while(is.numeric(n)==F &amp;amp; acr&amp;lt;0 ¦acr&gt;1)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;AGAIN:question 2 &#xA;There is a proble with what=double() also in the scan function, I think.&#xA;I know that, for example, 0.5 is a double data type in other programming languages but &#xA;I cannot figure it out in R(I don't know what it is called in R).&#xA;what is the diffrence between integer() and double() in R? (I am not familiar with double)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would be highly appreciative to anyone who could come to my aid.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;many thanks to all.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Isaac Owusu&lt;/p&gt;&#xA;" OwnerUserId="4340" LastEditorUserId="4340" LastEditorDisplayName="" LastEditDate="2011-05-10T13:55:08.220" LastActivityDate="2011-05-10T13:55:08.220" ClosedDate="2011-05-10T14:22:17.423" Title="using while loop with logical operators (AND and OR)" Tags="&lt;r&gt;" />
  <row Id="10597" PostTypeId="1" AcceptedAnswerId="10622" CreationDate="2011-05-10T13:55:45.370" Score="1" ViewCount="51" Body="&lt;p&gt;Is there a reasonable way to quantify the amount of local correlations in an image?  For example, I want to justify the correlations between a neighbourhood of pixels is much higher than the correlations between pixels in entirely different regions of the image.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would showing the xcorr2(A,A) as the 2-d autocorrelation of the image be a valid way to show this?  ie: if if there are large values mostly located at the center of the matrix.&lt;/p&gt;&#xA;" OwnerUserId="4547" LastEditorUserId="1036" LastEditorDisplayName="" LastEditDate="2011-05-10T15:47:44.150" LastActivityDate="2011-05-10T21:03:06.407" Title="Valid method to analyze spatial correlations in images?" Tags="&lt;correlation&gt;&lt;spatial&gt;&lt;image-processing&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="10598" PostTypeId="1" CreationDate="2011-05-10T14:19:14.667" Score="7" ViewCount="71" Body="&lt;p&gt;Various forms of the correlation, e.g.,&#xA;$r =  \frac{\Sigma_i x_i * y_i}{\sigma_x  \sigma_y}$&#xA;or&#xA;$r = \frac{\Sigma_i (x_i-\bar{x}) * (y_i-\bar{y})}{\sigma_x  \sigma_y}$&#xA;are popular similarity measures in many applications.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a probabilistic interpretation for this such that either $r$ or $r^2$ is an approximate likelihood for x and y coming from the same or similar distribution? i.e., if we have some form of $P_{\theta_1}(x)$ and $P_{\theta_2}(y)$, then $r$ is related to $P(\theta_1=\theta_2 | x,y)$?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would guess that the correlation may be the first term in the approximation of some sort of a likelihood measure. But I am unable to arrive at such a model. Assuming $x$ and $y$ to come from a normal, and $\theta$ being the mean, it doesn't really derive that expression.&lt;/p&gt;&#xA;" OwnerUserId="2728" LastEditorUserId="2728" LastEditorDisplayName="" LastEditDate="2011-05-10T14:30:19.253" LastActivityDate="2011-05-10T14:30:19.253" Title="Correlation as a likelihood measure" Tags="&lt;probability&gt;&lt;correlation&gt;&lt;interpretation&gt;&lt;likelihood-function&gt;" CommentCount="2" FavoriteCount="2" />
  <row Id="10599" PostTypeId="2" ParentId="10597" CreationDate="2011-05-10T15:14:59.910" Score="1" ViewCount="" Body="&lt;p&gt;One straightforward way of doing this is to consider arbitrarily-sized patches of the image.  For example, let's say we are interested in all 9*9 regions of pixels that can be taken from the image.  Extract each of these image patches, and transform each image patch to a row vector.  Consider the entire set of image patches (8464 row vectors for a 100*100 pixel image) as a matrix M.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Compute the correlation (or covariance, corr(M) and cov(M) in Matlab) between each of the columns in M.  For your specific question, look at the three columns of the correlation/covariance matrix corresponding to the central pixel in the image.  Reshape these back to the size of the image patch, and plot these.  For natural images, you should find that the central pixel is highly correlated with adjacent pixels, and that the correlation decreases as distance from the central pixel increases across each of the three color channels.&lt;/p&gt;&#xA;" OwnerUserId="3595" LastActivityDate="2011-05-10T15:14:59.910" />
  <row Id="10600" PostTypeId="2" ParentId="10526" CreationDate="2011-05-10T15:57:42.317" Score="1" ViewCount="" Body="&lt;p&gt;I don't think that using CCA will help you. It appears to me that you have a number of endogenous series ( abundance of species n in number ) and a number of exogenous series ( variety of food resources m in number ). I would suggest constructing n transfer functions each one optimized to fully utilize the information content in the m supporting series and their lags if appropriate while incorporating and unspecified stochastic structure with ARMA and unspecified deterministic structure like Level Shifts/Local Time Trends etc.. Having these n equations unser a &quot;statistical microscope&quot; might illuminate &quot;commonalities&quot; suggesting further grouping of the n equations into subsets. &lt;/p&gt;&#xA;" OwnerUserId="3382" LastActivityDate="2011-05-10T15:57:42.317" CommentCount="2" />
  <row Id="10601" PostTypeId="2" ParentId="10564" CreationDate="2011-05-10T16:09:36.753" Score="1" ViewCount="" Body="&lt;p&gt;If I understand things correctly, you're testing if the mean value of predictor $A$ associated with outcome $1$ differ from the mean value of predictor A associated with outcome $2$. Even if they don't differ, this result says nothing to your research question. What it says is only that in your sample, the average value of the predictor $A$ is'n different among succes and failures of the dependent variable. I'll give an example so you can see why.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Imagine that i'm interesting to know if education affects the probability of an employee being manager in a given firm. I collect a random sample of employees, with $200$ employes that are not managers and $70$ which are actually manager. I ran a logistic regression and the effect of education is significant (usual caveats about p-value apply here as well). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, you may wonder if the sample is balanced, which is, if there is the same variation in education for managers and non-managers. One first test to see this is to test if their means differ. If their means don't differ, than you know that on average on your sample, managers and non-managers have the same level of education. You could check if there is roughly the same variation of educational level among managers and non-managers. But I think that it's pretty clear that the Wilcoxon test (or the Mann-Whitney) has nothing to say about the effect of your predicton on the probability of success (in my example, success means to be a manager).&lt;/p&gt;&#xA;" OwnerUserId="3058" LastEditorUserId="3058" LastEditorDisplayName="" LastEditDate="2011-05-10T20:31:29.040" LastActivityDate="2011-05-10T20:31:29.040" CommentCount="3" />
  <row Id="10602" PostTypeId="1" CreationDate="2011-05-10T16:28:07.593" Score="0" ViewCount="39" Body="&lt;p&gt;I have a design where birds of 1, 3, 5 week of age are placed in 2 chambers(independent) with 4 treatments( light technology) for 4 days. for example, x no of birds of 1 weeks of age are kept for 4 days and then new birds of 3 week age are placed in both chambers and so on...readings are taken for their position, feed consumption/day/bird etc. I need to calculate the sample size required. I am using Proc GLMpower in SAS, using twowayanova to calculate the sample size but I am not sure how to account for repeated measurement for 4 days. Also, Can you suggest any other way to do this type power analysis. I tried in G*Power as well but not quite sure how it works. Thank you.&lt;/p&gt;&#xA;" OwnerUserId="4550" LastActivityDate="2011-05-10T17:14:40.017" Title="Sample size determination for block design with repeated measurement in SAS" Tags="&lt;anova&gt;" AnswerCount="1" />
  <row Id="10603" PostTypeId="1" CreationDate="2011-05-10T16:55:47.630" Score="0" ViewCount="37" Body="&lt;p&gt;I have a very basic question on when a discrete distribution might be called a &lt;em&gt;symmetric&lt;/em&gt; distribution. Let say I have a r.v. $X$ that can take two possible values $(x1, x2)$ with $x1 \neq x2$ and corresponding probabilities $(0.4, 0.6)$. Then can I say that $X$ is symmetric?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks,&lt;/p&gt;&#xA;" OwnerUserId="4551" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-10T20:59:45.283" LastActivityDate="2011-05-10T20:59:45.283" Title="How to characterize symmetric discrete distribution?" Tags="&lt;distributions&gt;&lt;discrete-data&gt;" AnswerCount="1" />
  <row Id="10604" PostTypeId="1" AcceptedAnswerId="11499" CreationDate="2011-05-10T17:02:08.347" Score="3" ViewCount="151" Body="&lt;p&gt;I'm trying to forecast using ARIMAX with two exogenous (input) variables. I'm using PROC ARIMA, but I can't figure out from the SAS documentation whether my code is producing the parameterization I want.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to extend an ARI(12,1) model so that it also includes the last 12 terms of each of the two exogenous variables in my forecast. So, using &lt;code&gt;VariableX&lt;/code&gt; with the two exogenous variables &lt;code&gt;VariableY&lt;/code&gt; and &lt;code&gt;VariableZ&lt;/code&gt;, my best attempt at the code is:&lt;/p&gt;&#xA;&#xA;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;proc arima;&#xA;  identify var=VariableY(1) nlag=24;&#xA;  estimate p=12;&#xA;  identify var=VariableZ(1) nlag=24;&#xA;  estimate p=12;&#xA;  identify var=VariableX(1) nlag=24 crosscorr=( VariableY(1) VariableZ(1) );&#xA;  estimate p=12 input=( VariableY VariableZ );&#xA;  forecast id=MonthNumber interval=month alpha=.05 lead=24;&#xA;  run;&#xA;quit;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The documentation leads me to believe the first four lines of the procedure are required for setting up the forecast at the end. But when I run the procedure, the output appears to show a forecast using only the last term of each of the two exogenous variables.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;In summary, I'd like to be sure where each of the following are controlled:&lt;/h2&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The $p$ of $AR(p)$, and similarly for each of the exogenous variables&lt;/li&gt;&#xA;&lt;li&gt;The $d$ of $I(d)$, and similarly for each of the exogenous variables&lt;/li&gt;&#xA;&lt;li&gt;The $q$ of $MA(q)$, and similarly for each of the exogenous variables&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="1583" LastEditorUserId="1583" LastEditorDisplayName="" LastEditDate="2011-05-10T17:12:45.243" LastActivityDate="2011-06-03T07:32:50.997" Title="How do I ensure PROC ARIMA is performing the correct parameterization of input variables?" Tags="&lt;time-series&gt;&lt;sas&gt;&lt;dynamic-regression&gt;" AnswerCount="2" CommentCount="3" />
  <row Id="10605" PostTypeId="2" ParentId="10603" CreationDate="2011-05-10T17:11:08.470" Score="2" ViewCount="" Body="&lt;p&gt;No, it only would be symmetric if the corresponding probabilities were (0.5,0.5).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, with binary or categorical distributions, the concept of symmetry does not have much meaning.&lt;/p&gt;&#xA;" OwnerUserId="3595" LastEditorUserId="3595" LastEditorDisplayName="" LastEditDate="2011-05-10T20:46:21.960" LastActivityDate="2011-05-10T20:46:21.960" CommentCount="2" />
  <row Id="10606" PostTypeId="2" ParentId="10602" CreationDate="2011-05-10T17:14:40.017" Score="1" ViewCount="" Body="&lt;p&gt;When problems get more complicated than the simple cases that have nice canned sample size solutions I turn to simulation.  The basic steps:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Decide what you think your data will look like (including things you may want to change, e.g. sample size(s))&lt;/li&gt;&#xA;&lt;li&gt;Decide how you will analyze the data&lt;/li&gt;&#xA;&lt;li&gt;create some code that simulates the data and analyzes it then returns the p-value or other statistic of interest&lt;/li&gt;&#xA;&lt;li&gt;run the code in step 3 a bunch of times on a given set of conditions, then see how often the null hypothesis is rejected, this is your power (or an estimate of the power).&lt;/li&gt;&#xA;&lt;li&gt;change the conditions and run again, ...&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In SAS you could probably create the data in proc IML, then send it to proc GLM or such for the analysis, use ODS to save the p-value, then capture this back in IML, etc.  However, I think R is more straight forward for this (but I use R much more than SAS these days, so could be biased).&lt;/p&gt;&#xA;" OwnerUserId="4505" LastActivityDate="2011-05-10T17:14:40.017" />
  <row Id="10607" PostTypeId="1" AcceptedAnswerId="10620" CreationDate="2011-05-10T17:23:17.073" Score="2" ViewCount="147" Body="&lt;p&gt;I am running a logistic model. In SAS Entreprise Miner, I noticed there's a link function that has three possible options: &lt;code&gt;logit&lt;/code&gt;, &lt;code&gt;probit&lt;/code&gt; and &lt;code&gt;cll&lt;/code&gt; (complementary log-log).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can you please shed light on the following questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Can we use any of these link function to carry out a logistic regression?&lt;/li&gt;&#xA;&lt;li&gt;Are there situations where one would be better than others?&lt;/li&gt;&#xA;&lt;li&gt;Is it intuitively possible to get some insight about which kind of function could be useful in which situation? (By just looking at the formula, complementary log-log function might be good for normalization of data when data does not depart (too much) from a normal distribution.)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Any additional pointers would be greatly appreciated.&lt;/p&gt;&#xA;" OwnerUserId="1763" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-10T20:48:39.123" LastActivityDate="2011-05-11T01:37:41.867" Title="How to choose the link function when performing a logistic regression?" Tags="&lt;logistic-regression&gt;&lt;sas&gt;" AnswerCount="3" CommentCount="5" />
  <row Id="10608" PostTypeId="1" AcceptedAnswerId="10610" CreationDate="2011-05-10T17:29:14.633" Score="3" ViewCount="97" Body="&lt;p&gt;I have a question concerning feature selection and classification. I will be working with R. I should start by saying that I am not very familiar with data mining techniques, aside from a brief glimpse provided by an undergraduate course on multivariate analysis, so forgive me if I am lacking in details regarding my question. I will try my best to describe my problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, a little about my project:&#xA;I am working on an image cytometry project, and the dataset is composed of over 100 quantitative features of histological images of cellular nuclei. All of the variables are continuous variables describing features of the nucleus, such as size, DNA amount, etc. There is currently a manual process and an automatic process for obtaining these cellular images. The manual process is (very) slow, but is done by a technician and yields only images that are usable for further analysis. The automatic process is very fast, but introduces too many unusable images - only about 5% of the images are suitable for further analysis, and there are thousands of nuclear images per sample. As it turns out, cleaning the data obtained from the automatic process is actually more time consuming than the manual process.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My goal is to train a classification method, using R, to discriminate between good objects and bad objects from the data obtained from the automatic process. I have an already classified training set that was obtained from the automatic process. It consists of 150,000 rows, of which ~5% are good objects and ~95% are bad objects.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My first question deals with feature selection. There are over 100 continuous explanatory features, and I would like possibly get rid of noise variables to (hopefully) help with the classification. What methods are there for dimensionality reduction with the goal of improving classification? I understand that the need for variable reduction may vary depending on the classification technique used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which leads to my second question. I have been reading about different classification techniques, but I feel that I cannot adequately determine the most suitable method for my problem. My main concerns are having a low misclassification rate of good objects over bad objects, and the fact that the prior probability of the good objects is much lower than the prior probability of the bad objects. Having a bad object classified as good is less of a hassle than recovering a good object from the pool of bad objects, but it would be nice if not too many bad objects were classified as being good.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have read this &lt;a href=&quot;http://stats.stackexchange.com/questions/3458/alternatives-to-classification-trees-with-better-predictive-e-g-cv-performanc&quot;&gt;post&lt;/a&gt; and I am currently considering Random Forests as per chl's answer. I would like to explore other methods as well, and I would like to collect the suggestions of the good people here at CV. I also welcome any readings on the subject of classification that may be helpful, and suggestions for R packages to use.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please ask for more details if my post is lacking in details.&lt;/p&gt;&#xA;" OwnerUserId="2252" LastEditorUserId="2252" LastEditorDisplayName="" LastEditDate="2011-05-11T16:23:22.990" LastActivityDate="2011-05-11T16:23:22.990" Title="Questions about variable selection for classification, and different classification techniques" Tags="&lt;r&gt;&lt;machine-learning&gt;&lt;classification&gt;&lt;dimensionality-reduction&gt;" AnswerCount="2" CommentCount="6" FavoriteCount="1" />
  <row Id="10610" PostTypeId="2" ParentId="10608" CreationDate="2011-05-10T18:04:24.603" Score="7" ViewCount="" Body="&lt;p&gt;Feature selection does not necessarily improve the performance of modern classifier systems, and quite frequently makes performance worse.  Unless finding out which features are the most important is an objective of the analysis, it is often better not even to try and to use regularisation to avoid over-fitting (select regularisation parameters by e.g. cross-validation).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reason that feature selection is difficult is that it involves an optimisation problem with many degrees of freedom (essentially one per feature) where the criterion depends on a finite sample of data.  This means you can over-fit the feature selection criterion and end up with a set of features that works well for that particular sample of data, but not for any other (i.e. it generalises poorly).  Regularisation on the other hand, while also optimising a criterion based on a finite sample of data, involves fewer degrees of freedom (typically one), which means that over-fitting the criterion is more difficult.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems to me that the &quot;feature selection gives better performance&quot; idea has rather passed its sell by date.  For simple linear unregularised classifiers (e.g. logistic regression), the complexity of the model (VC dimension) grows with the number of features.  Once you bring in regularisation, the complexity of the model depends on the value of the regularisation parameter rather than the number of parameters.  That means that regularised classifiers are resistant to over-fitting (provided you tune the regularisation parameter properly) even in very high dimensional spaces.  In fact that is the basis of why the support vector machine works, use a kernel to tranform the data into a high (possibly infinite) dimensional space, and then use regularisation to control the complexity of the model and hence avoid over-fitting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having said which, there are no free lunches; your problem may be one where feature selection works well, and the only way to find out is to try it.  However, whatever you do, make sure you use something like nested cross-validation to get an unbiased estimate of performance.  The outer cross-validation is used for performance evaluation, but in each fold repeat every step in fitting the model (including feature selection) again independently.  A common error is to perform feature selection using all of the data and then cross-validate to estimate performance using the features identified.  IT should be obvious why that is not the right thing to do, but many have done it as the correct approach is computationally expensive.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My suggestion is to try SVMs or kernel logistic regression or LS-SVM etc. with various kernels, but no feature selection.  If nothing else it will give you a meaningfull baseline.&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2011-05-10T18:04:24.603" CommentCount="5" />
  <row Id="10611" PostTypeId="2" ParentId="10608" CreationDate="2011-05-10T18:04:58.020" Score="3" ViewCount="" Body="&lt;p&gt;On dimensionality reduction, a good first choice might be &lt;a href=&quot;http://en.wikipedia.org/wiki/Principal_components_analysis&quot; rel=&quot;nofollow&quot;&gt;principal components analysis&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Apart from that, i don't have too much to add, except that if you have any interest in data mining, I strongly recommend you read &lt;a href=&quot;http://www-stat.stanford.edu/~tibs/ElemStatLearn/&quot; rel=&quot;nofollow&quot;&gt;the elements of statistical learning&lt;/a&gt;. Its both rigorous and clear, and although I haven't finished it, it would probably give you much insight into the right way to approach your problem. Chapter 4, linear classifiers would almost certainly enough to get you started.&lt;/p&gt;&#xA;" OwnerUserId="656" LastActivityDate="2011-05-10T18:04:58.020" CommentCount="3" />
  <row Id="10612" PostTypeId="2" ParentId="10531" CreationDate="2011-05-10T18:22:00.337" Score="2" ViewCount="" Body="&lt;p&gt;There is no simple yes or no answer.  People constantly attempt to make inferences about causal relationships.  The question is what assumptions you have to make, and how sensitive your inferences are to changing those assumptions.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The causal effects you can identify with the fewest assumptions are the effects of the things you randomly assign: A, B, and the interaction A*B, on Y1, Y2, Y3, and Y4.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm likely to be skeptical of a claim to have identified the causal effect of any of the non-randomized variables on anything else.  The scientific context (which you have not provided) will shape what is considered a reasonable inference.&lt;/p&gt;&#xA;" OwnerUserId="3748" LastActivityDate="2011-05-10T18:22:00.337" />
  <row Id="10613" PostTypeId="1" CreationDate="2011-05-10T18:26:26.630" Score="11" ViewCount="245" Body="&lt;p&gt;Recently, I have found in a paper a statement that p-values should be uniformly distributed. I believe the authors, but cannot understand, why it is so. &lt;/p&gt;&#xA;" OwnerUserId="4552" LastActivityDate="2011-05-27T00:19:15.937" Title="Why p-values are uniformly distributed?" Tags="&lt;p-value&gt;&lt;uniform&gt;" AnswerCount="2" CommentCount="4" FavoriteCount="2" />
  <row Id="10614" PostTypeId="2" ParentId="9299" CreationDate="2011-05-10T18:59:18.107" Score="4" ViewCount="" Body="&lt;p&gt;You can use this module of the &lt;a href=&quot;http://pysal.org/1.1/library/spreg/ols.html&quot; rel=&quot;nofollow&quot;&gt;pysal&lt;/a&gt; python library for the spatial data analysis methods I discuss below.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your description of how each person's attitude is influenced by the attitudes of the people surrounding her can be represented by a &lt;a href=&quot;http://www.ecoevol.ufg.br/sam/wiki/Autoregressive_Models_%28SAR_/_CAR%29&quot; rel=&quot;nofollow&quot;&gt;spatial autoregressive model (SAR)&lt;/a&gt; (also see my simple SAR explanation from &lt;a href=&quot;http://stats.stackexchange.com/questions/277/spatial-statistics-models-car-vs-sar&quot;&gt;this SE answer&lt;/a&gt; &lt;a href=&quot;http://www.ecoevol.ufg.br/sam/wiki/Autoregressive_Models_%28SAR_/_CAR%29&quot; rel=&quot;nofollow&quot;&gt;2&lt;/a&gt;). The simplest approach is to ignore other factors, and estimate the strength of the influence how surrounding people affect one another's attitudes by using the &lt;a href=&quot;http://en.wikipedia.org/wiki/Moran%27s_I&quot; rel=&quot;nofollow&quot;&gt;Moran's I&lt;/a&gt; statistic.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to assess the importance of other factors while estimating the strength of the influence of surrounding people, a more complex task, then you can estimate the parameters of a regression: $y = bx + rhoWy + e$. See the docs &lt;a href=&quot;http://pysal.org/1.1/library/spreg/ols.html&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.(Methods of estimating this type of regression come from the field of spatial econometrics and can get much more sophisticated than the reference I gave.) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your challenge will be to build a spatial weights matrix ($W$). I think each element $w_{ij}$ of the matrix should be 1 or 0 based on whether the person $i$ is within some distance you feel that it is required to influence the other person $j$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To get an intuitive idea of the problem, below I illustrate how a spatial autoregressive data generating process (DGP) will make a pattern of values. For the 2 lattices of simulated values the white blocks represent high values and the dark blocks represent low values. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the first lattice below the grid values have been generated by a normally distributed random process (or Gaussian), where $rho$ is zero. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.stack.imgur.com/O2hMI.png&quot; alt=&quot;Random (Gaussian)&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the next lattice below the grid values have been generated by a spatial autoregressive process, where $rho$ has been set to something high, say .8.&#xA;&lt;img src=&quot;http://i.stack.imgur.com/KDeGc.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="4329" LastEditorUserId="4329" LastEditorDisplayName="" LastEditDate="2011-05-11T03:57:17.720" LastActivityDate="2011-05-11T03:57:17.720" CommentCount="3" />
  <row Id="10615" PostTypeId="1" CreationDate="2011-05-10T19:28:40.657" Score="2" ViewCount="80" Body="&lt;p&gt;I'd be too happy, if someone could post a code snippet, which explains how to compute mean and variance of a set of records, which contain frequencies?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose we have records like &lt;strong&gt;(FORMAT F)&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;- GroupA, 6 x Grade 1, 5 x Grade 2, 10 x Grade 3&#xA;- GroupB, 2 x Grade 1, 7 x Grade 2, 18 x Grade 3&#xA;- GroupA, 23 x Grade 1, 5 x Grade 2, 1 x Grade 3&#xA;- GroupA, 10 x Grade 1, 15 x Grade 2, 12 x Grade 3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Goal: Compute for each Group's mean and variance of grades using SPSS syntax.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Clarification:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using SPS, processing cases &lt;strong&gt;(FORMAT C)&lt;/strong&gt; would be simple (as colleagues told me):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Record number 1 (line 1 of the above data)&#xA;GroupA, Grade 1&#xA;   ... 6 times&#xA;GroupA, Grade 1&#xA;GroupA, Grade 2&#xA;   ... 5 times&#xA;GroupA, Grade 2&#xA;GroupA, Grade 3&#xA;   ... 10 times&#xA;GroupA, Grade 3&#xA;# Record number 2 (line 2 of the above data) &#xA;GroupB, Grade 1&#xA;GroupB, Grade 1&#xA;GroupB, Grade 2&#xA;   ... 78 times&#xA;GroupB, Grade 7&#xA;   ... and so on&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Unfortunately, we don't have cases of FORMAT C, but already aggregated data as in FORMAT F.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I probably should add this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I implemented a solution, which uses (user entered) FORMAT F data using PHP/SQL: The PHP program constructs an SQL query, which computes the mean, the variance and the standard deviation of each Group. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I even tested the solution using well-known sample data. The results are identical. Thus I'm sure the solution is correct.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since the computed statistical properties are sensible, I'd like to get a verification using a standard statistical package using the real FORMAT F data (not the test case data).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My colleague is going to use SPSS.&lt;/p&gt;&#xA;" OwnerUserId="4554" LastEditorUserId="4554" LastEditorDisplayName="" LastEditDate="2011-05-15T22:31:54.813" LastActivityDate="2011-05-15T22:31:54.813" Title="Variance based on given frequencies using SPSS" Tags="&lt;spss&gt;" AnswerCount="1" CommentCount="6" />
  <row Id="10617" PostTypeId="2" ParentId="10613" CreationDate="2011-05-10T19:45:10.343" Score="12" ViewCount="" Body="&lt;p&gt;To clarify a bit.  The p-value is uniformly distributed when the null hypothesis is true and all other assumptions are met.  The reason for this is really the definition of alpha as the probability of a type I error.  We want the probability of rejecting a true null hypothesis to be alpha, we reject when the observed $\text{p-value} &amp;lt; \alpha$, the only way this happens for any value of alpha is when the p-value comes from a uniform distribution.  The whole point of using the correct distribution (normal, t, f, chisq, etc.) is to tranform from the test statistic to a uniform p-value.  If the null hypothesis is false then the distribution of the p-value will (hopefully) be more weighted towards 0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;code&gt;Pvalue.norm.sim&lt;/code&gt; and &lt;code&gt;Pvalue.binom.sim&lt;/code&gt; functions in the &lt;a href=&quot;http://cran.r-project.org/web/packages/TeachingDemos/index.html&quot; rel=&quot;nofollow&quot;&gt;TeachingDemos&lt;/a&gt; package for R will simulate several data sets, compute the p-values and plot them to demonstrate this idea. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also see: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Murdock, D, Tsai, Y, and Adcock, J (2008). P-Values are Random&#xA;  Variables. &lt;em&gt;The American Statistician&lt;/em&gt;, &lt;em&gt;62&lt;/em&gt;, 242-245.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;for some more details.&lt;/p&gt;&#xA;" OwnerUserId="4505" LastEditorUserId="4505" LastEditorDisplayName="" LastEditDate="2011-05-11T00:38:41.177" LastActivityDate="2011-05-11T00:38:41.177" CommentCount="2" />
  <row Id="10618" PostTypeId="1" CreationDate="2011-05-10T20:18:01.967" Score="0" ViewCount="27" Body="&lt;p&gt;I thank everyone, in particular, Bernd! His answer works perfectly.&#xA;But I am having a problem with the execution of my program.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is the program execution and its results:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;n   &amp;lt;- -2&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;acr &amp;lt;- -2&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;while((n&amp;lt;=0) | (acr&amp;lt;=0) | (acr&gt;=1)) {&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;print(&quot;enter a  positive integer and the average cancellation rate between 0 and 1 you want&quot;)&#xA;&#xA;try(n&amp;lt;-scan(what=integer(),nmax=1), silent=TRUE);try(acr&amp;lt;-scan(what=double(),nmax=1), silent=TRUE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;THIS IS THE EXECUTION OF THE PROGRAM&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[1] &quot;enter a  positive integer and the average cancellation rate between 0 and 1 you want&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1: #We read it back into R&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1:  bygrace&amp;lt;-read.table(&quot;C:\MyRfolder\bygrace.txt&quot;,header=FALSE)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1: r&amp;lt;-nrow(bygrace)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[1] &quot;enter a  positive integer and the average cancellation rate between 0 and 1 you want&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1: c&amp;lt;-ncol(bygrace)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1: copybygrame&amp;lt;-array(bygrace,dim=c(6,6))&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;1: copybygrace&amp;lt;-bygrace[-((n+1):r),]# I WANT TO USE &quot;n&quot; HERE&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;1: write.table(copybygrace,file=&quot;C:\MyRfolder\copybygrace.txt&quot;,sep=&quot;\t&quot;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[1] &quot;enter a  positive integer and the average cancellation rate between 0 and 1 you want&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1: copybygrace&amp;lt;-read.table(&quot;C:\MyRfolder\bygrace.txt&quot;,header=FALSE)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1: 4&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1: 0.5&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Read 1 item&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[1] &quot;enter a  positive integer and the average cancellation rate between 0 and 1 you want&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1: 4&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Read 1 item&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1: 0.5&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Read 1 item&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;bygrace&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;V1 V2  V3  V4  V5  V6&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1 100  0   1  -6  11  16&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2 120  0  -2   7 -12 -17&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3 150  0   3  -8  13  18&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4 210  0  -4   9 -14 -19&lt;/p&gt;&#xA;&#xA;&lt;p&gt;5 140  0  -5 -10  15  20&lt;/p&gt;&#xA;&#xA;&lt;p&gt;6 300  0 100 150 200 250&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;copybygrace&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;V1 V2  V3  V4  V5  V6&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1 100  0   1  -6  11  16&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2 120  0  -2   7 -12 -17&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3 150  0   3  -8  13  18&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4 210  0  -4   9 -14 -19&lt;/p&gt;&#xA;&#xA;&lt;p&gt;5 140  0  -5 -10  15  20&lt;/p&gt;&#xA;&#xA;&lt;p&gt;6 300  0 100 150 200 250&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;n&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;[1] 4&lt;/p&gt;&#xA;&#xA;&lt;p&gt;QUESTIONS:&#xA;1. The print statement:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;print(&quot;enter a  positive integer and the average cancellation rate between 0 and 1 you want&quot;)&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;keeps on repeating in the code. it does not wait for the entry to be made.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;I want to use &quot;n&quot; in the line of the code highlited with the comment:&#xA;that is, in the line:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;copybygrace&amp;lt;-bygrace[-((n+1):r),]# I WANT TO USE &quot;n&quot; HERE&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In the execution of the program, I entered &quot;n&quot; to be four(4). This means that from the above line, we take the first 4 rows of bygrace and put it in the copybygrace.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, as can be seen in the output, both bygrace and copybygrace are the same; which means that the program is not able to use &quot;n&quot; in that line of code as I want. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please how do I go about this so that the program can wait for the data entry and that it would be able to use &quot;n&quot; in the said line!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks in advance to all who could help me.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Isaac&lt;/p&gt;&#xA;" OwnerUserId="4340" LastActivityDate="2011-05-10T20:18:01.967" ClosedDate="2011-05-10T20:21:51.577" Title="undesired output of program" Tags="&lt;r&gt;" CommentCount="1" />
  <row Id="10619" PostTypeId="1" AcceptedAnswerId="10626" CreationDate="2011-05-10T20:21:47.193" Score="1" ViewCount="93" Body="&lt;p&gt;I am using Matlab to try and find a good fit for this curve:&#xA;&lt;img src=&quot;http://i.stack.imgur.com/pF3lz.png&quot; alt=&quot;cumulative distribution&quot;&gt;&#xA;None of the built-in formulas seem to work well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestions?&lt;/p&gt;&#xA;" OwnerUserId="4544" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-05-10T20:23:56.783" LastActivityDate="2011-05-10T23:37:39.147" Title="What regression formula would best fit this curve?" Tags="&lt;distributions&gt;&lt;matlab&gt;" AnswerCount="2" CommentCount="4" FavoriteCount="1" />
  <row Id="10620" PostTypeId="2" ParentId="10607" CreationDate="2011-05-10T20:30:24.190" Score="3" ViewCount="" Body="&lt;p&gt;I don't know of SAS, so i'll just answer based on the statistics side of the question. About the software you mays ask at the sister site, stackoverflow.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;If the link function is different (logistic, probit or Clog-log), than you will get different results. For logistic, use logistic.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;About the real differences of these link functions.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Logistic and probit are pretty much the same. To see why they are pretty much the same, remember that in linear regression the link function is the identity. In logistic regression, the link function is the logistic and in the probit, the normal. &#xA;Formally, you can see this by noting that, in case your dependent variable is binary, you can think of it as following a Bernoulli distribution with a given probability of success.&#xA;$Y \sim Bernoulli(p_{i})$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$p_{i} = f(\mu)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\mu = XB$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here, thew probabitliy $p_{i}$ is a function of predictor, just like in linear regression. The real difference is the link function. In linear regression, the link function is just the identity, i.e., $f(\mu) = \mu$, so you can just plug-in the linear predictors.In the logistic regression, the link function is the cumulative logistic distribution, given by $1/(1+exp(-x)). In the probit regression, the link function is the (inverse) cumulative Normal distribution function. And in the Clog-log regression, the link function is the complementary log log distribution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I never used the Cloglog, so i'll abstein of coments about it here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can see that Normal and Logist are very similar in this blog post by John Cook, of Endeavour &lt;a href=&quot;http://www.johndcook.com/blog/2010/05/18/normal-approximation-to-logistic/&quot; rel=&quot;nofollow&quot;&gt;http://www.johndcook.com/blog/2010/05/18/normal-approximation-to-logistic/&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general I use the logistic because the coefficients are easier to interpret than in a probit regression. In some specific context I use probit (ideal point estimation or when I have to code my own Gibbs Sampler), but I guess they are not relevant to you. So, my advice is, whenever in doubt about probit or logistic, use logistic!&lt;/p&gt;&#xA;" OwnerUserId="3058" LastEditorUserId="3058" LastEditorDisplayName="" LastEditDate="2011-05-10T21:14:00.837" LastActivityDate="2011-05-10T21:14:00.837" />
  <row Id="10621" PostTypeId="1" CreationDate="2011-05-10T21:02:18.387" Score="2" ViewCount="49" Body="&lt;p&gt;I'm interested in fitting a conditional Poisson regression model using PROC GENMOD in SAS to analyze a matched cohort study. However, it's not quite clear to me how I should exactly go about it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My impression is that a REPEATED statement should be used along with the events/trials syntax, but if so then how does one account for continuous covariates in the model, such as age?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would be most grateful if anyone had any pointers. Thus far, Google and the internet have disappointed me. Cheers.&lt;/p&gt;&#xA;" OwnerUserId="4555" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-11T07:30:51.817" LastActivityDate="2011-05-11T07:30:51.817" Title="How does one fit conditional Poisson regression in SAS?" Tags="&lt;regression&gt;&lt;poisson&gt;&lt;sas&gt;&lt;matching&gt;" CommentCount="2" />
  <row Id="10622" PostTypeId="2" ParentId="10597" CreationDate="2011-05-10T21:03:06.407" Score="1" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://pysal.org/1.1/library/esda/moran.html&quot; rel=&quot;nofollow&quot;&gt;Local Moran's I&lt;/a&gt; statistic  &lt;/p&gt;&#xA;" OwnerUserId="4329" LastActivityDate="2011-05-10T21:03:06.407" />
  <row Id="10623" PostTypeId="1" AcceptedAnswerId="10624" CreationDate="2011-05-10T21:42:26.387" Score="2" ViewCount="42" Body="&lt;p&gt;If I have a sample of k successes and n-k failures, there are standard techniques (Agresti-Coull, Clopper, etc.) for finding a confidence interval of the probability of an individual success.  What if I want to find a confidence interval for the probability of getting at least k' out of n' instead?  Obviously it can be approximated by the usual&#xA;$$\sum_{j\ge k&amp;#39;}{n&amp;#39;\choose j}(k/n)^j(1-k/n)^{n&amp;#39;-j}$$&#xA;but this does not take into account the uncertainty of the initial sample, which may be important (the samples are small).&lt;/p&gt;&#xA;" OwnerUserId="1378" LastActivityDate="2011-05-11T00:46:47.260" Title="Binomial testing with probability estimated from sample" Tags="&lt;confidence-interval&gt;&lt;binomial&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="10624" PostTypeId="2" ParentId="10623" CreationDate="2011-05-10T21:59:57.487" Score="3" ViewCount="" Body="&lt;p&gt;I believe you are looking for the &lt;a href=&quot;https://secure.wikimedia.org/wikipedia/en/wiki/Beta-binomial_distribution&quot; rel=&quot;nofollow&quot;&gt;beta binomial distribution&lt;/a&gt;, which reduces the pdf of of interest ($\pi(k\prime)$) to&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\pi(k\prime|n\prime,n,k) = {n\prime \choose k\prime} \frac{B(k\prime+k+1,n\prime-k\prime+n-k+1)}{B(k+1,n-k+1)}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For motivation, remember that you do not know $p$ of the process, but you need to estimate $p$ as the posterior after making observation of $k$ successes out of $n$ trials. So&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\pi(p|k,n) = \frac{p^k(1-p)^{(n-k)}}{\int x^k(1-x)^{(n-k)} dx}$&lt;/p&gt;&#xA;" OwnerUserId="2728" LastActivityDate="2011-05-10T21:59:57.487" />
  <row Id="10625" PostTypeId="2" ParentId="10619" CreationDate="2011-05-10T22:13:21.433" Score="1" ViewCount="" Body="&lt;p&gt;Check out &lt;a href=&quot;http://creativemachines.cornell.edu/eureqa&quot; rel=&quot;nofollow&quot;&gt;Eureqa&lt;/a&gt; for a neat evolutionary approach to finding the mathematical form of an otherwise ambiguous function. It's native to Windows but works fine on Linux &amp;amp; Mac via Wine (in which case I'd suggest you use &lt;a href=&quot;http://winebottler.kronenberg.org&quot; rel=&quot;nofollow&quot;&gt;winebottler&lt;/a&gt;). &lt;/p&gt;&#xA;" OwnerUserId="364" LastActivityDate="2011-05-10T22:13:21.433" CommentCount="1" />
  <row Id="10626" PostTypeId="2" ParentId="10619" CreationDate="2011-05-10T23:37:39.147" Score="2" ViewCount="" Body="&lt;p&gt;Looking at the charts in your first question, this looks slightly like the absolute value of a standard normal distribution, what Wikipedia calls a &lt;a href=&quot;http://en.wikipedia.org/wiki/Half-normal_distribution&quot; rel=&quot;nofollow&quot;&gt;half-normal distribution&lt;/a&gt;, and your curve here looks like the top half of the cdf of a normal distribution.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One way to check is to look at various quantiles of your distribution, compared with those of the absolute value of a standard normal.  In R, using your data, this might look like &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; mydat &amp;lt;- read.csv(&quot;http://furlender.com/data.csv&quot;, header=FALSE)&#xA;&amp;gt; mean(mydat$V1)   # compare with sqrt(2/pi) = 0.7978846 for half normal&#xA;[1] 1.014772&#xA;&amp;gt; median(mydat$V1) #              compare with 0.6744898 for half-normal&#xA;[1] 0.9162514&#xA;&amp;gt; percents &amp;lt;- c(0.01, (1:19)/20, 0.99, 0.999, 0.9999, 0.99999, 0.999999)&#xA;&amp;gt; quantile(mydat$V1, percents) / qnorm(percents/2 + 0.5)&#xA;      1%       5%      10%      15%      20%      25%      30%      35% &#xA;1.479777 1.455286 1.443074 1.435248 1.429568 1.421616 1.409148 1.400204 &#xA;     40%      45%      50%      55%      60%      65%      70%      75% &#xA;1.386696 1.372990 1.358436 1.341848 1.323763 1.306209 1.287204 1.266550 &#xA;     80%      85%      90%      95%      99%    99.9%   99.99%  99.999% &#xA;1.245071 1.223053 1.198247 1.172505 1.148309 1.141927 1.220474 1.412236 &#xA;99.9999% &#xA;1.371697 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and this suggests that your distribution is slightly more dispersed than the absolute value of a standard normal, perhaps somewhere between 1.15 and 1.4 times. Even the tails fit this description.  Drawing the ecdf confirms this compared with scaled normals. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; plot.ecdf(mydat$V1, lwd=3)&#xA;&amp;gt; curve(2*pnorm(x/1.4 )-1, 0, 7, col=&quot;red&quot;  , add=TRUE)&#xA;&amp;gt; curve(2*pnorm(x/1.15)-1, 0, 7, col=&quot;green&quot;, add=TRUE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.stack.imgur.com/PpoKT.png&quot; alt=&quot;ecdf&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you wish, you could stop there, and choose some intermediate value (perhaps 1.2718 to give a similar mean).  Or you could look for some normal-like symmetric distribution which has even moments about 0 similar to your data, and then take the absolute value.  These moments are easily calculated: for example the second and fourth are:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; mean(mydat$V1^2) &#xA;[1] 1.516602&#xA;&amp;gt; mean(mydat$V1^4) &#xA;[1] 5.972706&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;suggesting that (before taking the absolute value) the symmetric distribution would be slightly platykurtic even though the extreme tail is normal-like.&lt;/p&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-05-10T23:37:39.147" CommentCount="1" />
  <row Id="10627" PostTypeId="2" ParentId="1676" CreationDate="2011-05-10T23:50:24.527" Score="2" ViewCount="" Body="&lt;p&gt;I use &lt;code&gt;lattice&lt;/code&gt;, &lt;code&gt;ggplot2&lt;/code&gt;, &lt;code&gt;lubridate&lt;/code&gt;, &lt;code&gt;reshape&lt;/code&gt;, &lt;code&gt;boot&lt;/code&gt;, &lt;code&gt;e1071&lt;/code&gt;, &lt;code&gt;car&lt;/code&gt;, &lt;code&gt;forecast&lt;/code&gt;, and &lt;code&gt;zoo&lt;/code&gt; a lot.&lt;/p&gt;&#xA;" OwnerUserId="1764" LastActivityDate="2011-05-10T23:50:24.527" CommunityOwnedDate="2011-05-10T23:50:24.527" />
  <row Id="10628" PostTypeId="1" CreationDate="2011-05-11T00:03:07.227" Score="0" ViewCount="40" Body="&lt;p&gt;Consider $\boldsymbol{x}= [x_1,x_2,...x_n]$ and $\boldsymbol{y}= [y_1,y_2,...y_n]$ to be two multivariate Gaussians with an isotropic diagonal variance structure and uninformative priors so that:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$p(x_i,\mu_{x_i},\sigma_x) = \frac{e^{\frac{-(x_i-\mu_{x_i})^2}{2\sigma_x^2}}}{\sqrt{2\pi \sigma_x}}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$p(y_i,\mu_{y_i},\sigma_y) = \frac{e^{\frac{-(y_i-\mu_{y_i})^2}{2\sigma_y^2}}}{\sqrt{2\pi \sigma_y}}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\sigma_x$ is a scalar, constant for all $i$, $\sigma_y$ is a scalar, constant for all $i$; and $\mu_{x_i}$ and $\mu_{y_i}$ are scalars for each $i$. We can define vectors $\boldsymbol{\mu_x} = [\mu_{x_1},...\mu_{x_n}]$, and $\boldsymbol{\mu_y} = [\mu_{y_1},...\mu_{y_n}]$. For simplicity, let us assume that we have unnormalized uninformative uniform priors of all $\mu$ and $\sigma$ so that,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$p(\boldsymbol{x},\boldsymbol{\mu_x},\sigma_x) =\Pi_i p(x_i,\mu_{x_i},\sigma_x) = \Pi_i \frac{e^{\frac{-(x_i-\mu_{x_i})^2}{2\sigma_x^2}}}{\sqrt{2\pi \sigma_x}}$&#xA;and&#xA;$p(\boldsymbol{y},\boldsymbol{\mu_y},\sigma_y) =\Pi_i p(x_i,\mu_{x_i},\sigma_x) = \Pi_i \frac{e^{\frac{-(y_i-\mu_{y_i})^2}{2\sigma_y^2}}}{\sqrt{2\pi \sigma_y}}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given this, can I evaluate&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$p(\boldsymbol{\mu_y} \propto \boldsymbol{\mu_x}|\boldsymbol{x},\boldsymbol{y})$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;i.e.,&#xA;$p(\boldsymbol{\mu_y}=m\boldsymbol{\mu_x}+c|\boldsymbol{x},\boldsymbol{y})$&#xA;for any scalars $m$ and $c$?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just to explain what I mean, since we already have expressions for $p(\boldsymbol{x},\boldsymbol{\mu_x},\sigma_x)$ and $p(\boldsymbol{y},\boldsymbol{\mu_y},\sigma_y)$, I   am interested in&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\frac{\int p(\boldsymbol{x},\boldsymbol{\mu_x},\sigma_x)p(\boldsymbol{y},\boldsymbol{\mu_y},\sigma_y)\mathbf{1}_{[\frac{\mu_{y_1}-\bar{\mu_y}}{\mu_{x_1}-\bar{\mu_x}}=\frac{\mu_{y_2}-\bar{\mu_y}}{\mu_{x_2}-\bar{\mu_x}}=...=\frac{\mu_{y_n}-\bar{\mu_y}}{\mu_{x_n}-\bar{\mu_x}}]}d\sigma_x d\sigma_y d\boldsymbol{\mu_x} d\boldsymbol{\mu_y}}&#xD;&#xA;{\int p(\boldsymbol{x},\boldsymbol{\mu_x},\sigma_x)p(\boldsymbol{y},\boldsymbol{\mu_y},\sigma_y)d\sigma_x d\sigma_y d\boldsymbol{\mu_x} d\boldsymbol{\mu_y}}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\bar{\mu_x} = \Sigma_i \mu_{x_i}/n$ and $\bar{\mu_y} = \Sigma_i \mu_{y_i}/n$, and $\mathbf{1}_{[P]}$ is the indicator function, $1$ if proposition $P$ is true, $0$ otherwise.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I thought of doing this by removing $\mu_y$ in the numerator, replacing it by $m\mu_x+c$, and integrating over $m$ and $c$, but I do not know what pdf to use for $m$ and $c$ for the uninformative $\mu$s, surely $m$ and $c$ are not simply uniform. &lt;/p&gt;&#xA;" OwnerUserId="2728" LastEditorUserId="2728" LastEditorDisplayName="" LastEditDate="2011-05-12T16:45:04.147" LastActivityDate="2011-05-12T16:45:04.147" Title="Help evaluating a posterior probability expression" Tags="&lt;probability&gt;&lt;estimation&gt;&lt;normal-distribution&gt;&lt;likelihood-function&gt;&lt;integral&gt;" CommentCount="3" />
  <row Id="10629" PostTypeId="2" ParentId="10607" CreationDate="2011-05-11T00:35:33.890" Score="1" ViewCount="" Body="&lt;p&gt;All 3 link functions are s-shaped and are not going to be that different.  Li and Duan showed that if the predictor variables are well behaved (elliptically symetric predictors are a subset of the well behaved group) then changing the link function will change the coefficients by a multiplicitive constant.  Even if the predictors are not perfectly well behaved the differences between similar link functions are unlikely to change the overall inference (the exact coefficients will change, but what is important or significant will still be under a different link function).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The logit allows you to interpret individual coefficients as log-odds, so it tends to be the most popular these days.&lt;/p&gt;&#xA;" OwnerUserId="4505" LastActivityDate="2011-05-11T00:35:33.890" CommentCount="2" />
  <row Id="10630" PostTypeId="2" ParentId="10623" CreationDate="2011-05-11T00:46:47.260" Score="2" ViewCount="" Body="&lt;p&gt;If you are happy with your confidence interval on the unknown proportion of a single success, then just plug both of those values into the above formula.  Since k and n are known constants (rather than random variables) and the probability of k or more out of n is monotone, you can just transform the ends of your confidence interval to get a confidence interval on the transformed parameter (and your probability above is just a transform of the parameter).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to get fancier you can do a full Bayesian analysis with the Beta-Binomial and get the posterior distribution of the next observation (I think there is a more formal name for it, but that could be what you want).&lt;/p&gt;&#xA;" OwnerUserId="4505" LastActivityDate="2011-05-11T00:46:47.260" />
  <row Id="10631" PostTypeId="1" CreationDate="2011-05-11T00:51:20.380" Score="3" ViewCount="80" Body="&lt;p&gt;Hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a data set with a distribution of one variable against the other resembling a cubic one (rises to some point and then falls to a steady level without a consequent rise). I know in which cases to use log-linear, log-lin, lin-log, and reciprocal or log reciprocal linear models, but I am not sure what to do here (I have checked all of the above and they not surprisingly turned out to be a bad fit). Is there any linear model that would help me in this case?&lt;/p&gt;&#xA;" OwnerUserId="4560" LastActivityDate="2011-05-11T13:31:27.083" Title="What is the best linear regression model to use when the shape of the data resembles a cubic distribution?" Tags="&lt;regression&gt;&lt;econometrics&gt;" AnswerCount="2" CommentCount="5" />
  <row Id="10632" PostTypeId="2" ParentId="10579" CreationDate="2011-05-11T00:58:52.337" Score="0" ViewCount="" Body="&lt;p&gt;Hi, &#xA;you could use the Agresti-Caffo simultaneous confidence interval or (Simultaneous Score Intervals for Difference of Proportions) to compare differences in proportions (Agresti et al. 2008. Simultaneous confidence intervals for comparing binomial parameters, Biometrics 64, 1270-1275). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The corresponding R code is available in &lt;a href=&quot;http://www.stat.ufl.edu/~aa/cda/software.html&quot; rel=&quot;nofollow&quot;&gt;http://www.stat.ufl.edu/~aa/cda/software.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="4559" LastActivityDate="2011-05-11T00:58:52.337" />
  <row Id="10633" PostTypeId="2" ParentId="10544" CreationDate="2011-05-11T01:09:17.860" Score="2" ViewCount="" Body="&lt;p&gt;Hi, &#xA;You would need to use the &quot;Simultaneous Score Intervals for Difference of Proportions&quot; to solve your question. The reference is &quot; Agresti et al. 2008. Simultaneous confidence interval for comparing binomial parameters. Biometrics 64, 1270-1275. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The corresponding R code is available in &lt;a href=&quot;http://www.stat.ufl.edu/~aa/cda/software.html&quot; rel=&quot;nofollow&quot;&gt;http://www.stat.ufl.edu/~aa/cda/software.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sincerely,&lt;/p&gt;&#xA;" OwnerUserId="4559" LastActivityDate="2011-05-11T01:09:17.860" CommentCount="0" />
  <row Id="10634" PostTypeId="2" ParentId="10562" CreationDate="2011-05-11T01:13:16.013" Score="0" ViewCount="" Body="&lt;p&gt;Hi,&#xA;You might want to try &quot;model-based clustering&quot;. This algorithm uses &quot;BIC&quot; to determine the number of clusters. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sincerely&lt;/p&gt;&#xA;" OwnerUserId="4559" LastActivityDate="2011-05-11T01:13:16.013" />
  <row Id="10636" PostTypeId="2" ParentId="10607" CreationDate="2011-05-11T01:37:41.867" Score="0" ViewCount="" Body="&lt;p&gt;I have a question/comment.  I thought that by definition, logistic regression uses the logit link.  If you are using the probit or complementary log-log link, then I do not think that is logistic regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you are doing is fitting generalized linear models on a binary outcome, which is assumed to follow a Bernoulli.  The 3 usual choices of link functions are the logit, probit, and complementary log-log.  If you are using the logit link, that is logistic regression.&lt;/p&gt;&#xA;" OwnerUserId="2312" LastActivityDate="2011-05-11T01:37:41.867" CommentCount="3" />
  <row Id="10637" PostTypeId="2" ParentId="10592" CreationDate="2011-05-11T03:37:27.523" Score="1" ViewCount="" Body="&lt;p&gt;Merely &lt;em&gt;computing&lt;/em&gt; the covariance matrix--which you're going to need to get started in any event--is $O((Nf)^2)$ so, asymptotically in $N$, nothing is gained by choosing a $O(Nf)$ algorithm for the whitening. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are approximations when the variables have additional structure, such as when they form a time series or realizations of a spatial stochastic process at various locations. These effectively rely on assumptions that let us relate the covariance between one pair of variables to that between other pairs of variables, such as between pairs separated by the same time lags.  This is the conventional reason for assuming a process is &lt;a href=&quot;http://en.wikipedia.org/wiki/Stationary_process&quot; rel=&quot;nofollow&quot;&gt;stationary&lt;/a&gt; or &lt;a href=&quot;http://www.nrcse.washington.edu/pdf/trs56_covar.pdf&quot; rel=&quot;nofollow&quot;&gt;intrinsically stationary&lt;/a&gt;, for instance.  Calculations can be $O(Nflog(Nf)$ in such cases (&lt;em&gt;e.g.&lt;/em&gt;, using the Fast Fourier Transform as in &lt;a href=&quot;http://www.springerlink.com/content/k2073q016076hk56/&quot; rel=&quot;nofollow&quot;&gt;Yao &amp;amp; Journel 1998&lt;/a&gt;). Absent such a model, I don't see how you can avoid computing all pairwise covariances. &lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-05-11T03:37:27.523" />
  <row Id="10638" PostTypeId="2" ParentId="10592" CreationDate="2011-05-11T04:00:43.317" Score="0" ViewCount="" Body="&lt;p&gt;On a whim, I decided to try computing (in R) the covariance matrix for a dataset of about the size mentioned in the OP:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;z &amp;lt;- rnorm(1e8)&#xA;dim(z) &amp;lt;- c(1e6, 100)&#xA;vcv &amp;lt;- cov(z)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This took less than a minute in total, on a fairly generic laptop running Windows XP 32-bit. It probably took longer to generate &lt;code&gt;z&lt;/code&gt; in the first place than to compute the matrix &lt;code&gt;vcv&lt;/code&gt;. And R isn't particularly optimised for matrix operations out of the box.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given this result, is speed that important? If N &gt;&gt; p, the time taken to compute your approximation is probably not going to be much less than to get the actual covariance matrix.&lt;/p&gt;&#xA;" OwnerUserId="1569" LastActivityDate="2011-05-11T04:00:43.317" />
  <row Id="10639" PostTypeId="1" AcceptedAnswerId="10722" CreationDate="2011-05-11T04:34:34.790" Score="2" ViewCount="99" Body="&lt;p&gt;According to &lt;a href=&quot;http://en.wikipedia.org/wiki/Wilks%27_lambda_distribution&quot; rel=&quot;nofollow&quot;&gt;Wikipedia&lt;/a&gt;, Wilks' Lambda distribution generalizes Hotelling's distribution. I am having some problems seeing how this works. I can see how Hotelling's distribution generalizes Student's t-distribution (a RV distributed as Hotelling's law with $p=1$ is just the square of a RV distribution as Student's t), but cannot see how to get to Wilks' Lambda. Is there some setting of the parameters $p,m,n$ such that a RV distributed as Wilks' lambda is some transform of a Hotelling RV?&lt;/p&gt;&#xA;" OwnerUserId="795" LastEditorUserId="2958" LastEditorDisplayName="" LastEditDate="2011-05-11T06:13:18.020" LastActivityDate="2011-05-12T16:49:57.787" Title="How exactly does Wilks' Lambda distribution generalize the Hotelling distribution?" Tags="&lt;distributions&gt;&lt;t-distribution&gt;&lt;hotelling&gt;&lt;wilks-lambda&gt;" AnswerCount="2" />
  <row Id="10640" PostTypeId="1" CreationDate="2011-05-11T07:09:00.243" Score="2" ViewCount="72" Body="&lt;p&gt;I'm trying to design an experiment where I measure a variable as a function of 5 two-level factors, labelled A, B, C, D and E.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm trying to understand how to best design this experiment so I can conduct it in 8 runs. I've tried to follow the guidance given in Box, Hunter &amp;amp; Hunter, and found two experimental $2^{5-2}$ designs that seem to contradict each others.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One is given in the table p. 272, where D=AB and E=AC. That would yield the following design:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ABCDE&#xA;+++++&#xA;++-+-&#xA;+-+-+&#xA;+----&#xA;-++--&#xA;-+--+&#xA;--++-&#xA;---++&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The other one is the example whose table is given on p. 236, defined by D=BC and E=ABC:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ABCDE&#xA;+++++&#xA;++---&#xA;+-+--&#xA;+--++&#xA;-+++-&#xA;-+--+&#xA;--+-+&#xA;---++&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;On what criteria should one choose one design over another? Sorry if this sounds like a newbie's question, but I'd really like to understand this issue.&lt;/p&gt;&#xA;" OwnerUserId="4370" LastEditorUserId="26" LastEditorDisplayName="" LastEditDate="2011-05-11T15:36:41.373" LastActivityDate="2011-05-12T14:51:54.927" Title="How to design an 8-run experiment in 5 factors?" Tags="&lt;experiment-design&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="1" />
  <row Id="10641" PostTypeId="2" ParentId="10639" CreationDate="2011-05-11T07:19:46.017" Score="1" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://faculty.chass.ncsu.edu/garson/PA765/manova.htm&quot; rel=&quot;nofollow&quot;&gt;These NCSU course notes&lt;/a&gt; say &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;em&gt;Multivariate tests&lt;/em&gt; in contrast to the overall F test, answer the&#xA;  question, &quot;Is each effect&#xA;  significant?&quot; or more specifically,&#xA;  &quot;Is each effect significant for at&#xA;  least one of the dependent variables?&quot;&#xA;  That is, where the F test focuses on&#xA;  the dependents, the multivariate tests&#xA;  focus on the independents and their&#xA;  interactions. These tests appear in&#xA;  the &quot;Multivariate Tests&quot; table of SPSS&#xA;  output. The multivariate formula for F&#xA;  is based not only on the sum of&#xA;  squares between and within groups, as&#xA;  in ANOVA, but also on the sum of&#xA;  crossproducts - that is, it takes&#xA;  covariance into account as well as&#xA;  group means....&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;&lt;em&gt;Hotelling's T-Square&lt;/em&gt; is the most common, traditional test where there&#xA;  are two groups formed by the&#xA;  independent variables.... &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;&lt;em&gt;Wilks' lambda&lt;/em&gt;, U. This is the most common, traditional test where there&#xA;  are more than two groups formed by the&#xA;  independent variables.... The t-test,&#xA;  Hotelling's T, and the F test are&#xA;  special cases of Wilks's lambda....&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So I presume that if you take Wilks' lambda, and reduce the number of groups formed by the independent variables to two, then you get something like Hotelling's T-Square.&lt;/p&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-05-11T07:19:46.017" CommentCount="1" />
  <row Id="10642" PostTypeId="2" ParentId="10640" CreationDate="2011-05-11T07:47:29.803" Score="1" ViewCount="" Body="&lt;p&gt;I do not have the book at hand, so I cannot comment on the reasoning to find these models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, it is reasonable to expect in this type of setting that:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Different designs might better achieve different optimality criteria: perhaps you want the design with 8 runs that has best overall predictive ability, perhaps you want least maximum variance,...&lt;/li&gt;&#xA;&lt;li&gt;There may be more than one design yielding the same optimality (given a criterion)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;As such, if you want help for your specific situation, you're going to have to enlighten us on what you want to use this design for (i.e. what is your optimality criterion).&lt;/p&gt;&#xA;" OwnerUserId="4257" LastActivityDate="2011-05-11T07:47:29.803" CommentCount="2" />
  <row Id="10643" PostTypeId="1" AcceptedAnswerId="10686" CreationDate="2011-05-11T09:01:45.637" Score="2" ViewCount="38" Body="&lt;p&gt;I'm measuring distances of various samples from a reference point. The distance is defined as a non-negative number, where $d=0$ means that the test case is identical to the reference. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My general question is: Given a set  of &quot;typical&quot; distances, what is the proper way to tell whether a given $d_1$ &quot;too large&quot;, compared to the &quot;typical&quot;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my particular case the distance distribution is shown on the following graph&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.stack.imgur.com/BcFYJ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I failed to transform these data to anything symmetrical, so I can't use normal approximation. Any suggestions?&lt;/p&gt;&#xA;" OwnerUserId="1496" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-11T09:50:04.673" LastActivityDate="2011-05-11T23:04:01.823" Title="How to properly analyze distance from a reference?" Tags="&lt;distributions&gt;&lt;hypothesis-testing&gt;&lt;distance-functions&gt;" AnswerCount="2" />
  <row Id="10644" PostTypeId="1" CreationDate="2011-05-11T09:27:52.943" Score="3" ViewCount="27" Body="&lt;p&gt;In PCA eigenvalues determine the order of components. In ICA I am using kurtosis to obtain the ordering. What are some accepted methods to assess the number, (given I have the order) of components that are singificant apart from prior knowledge about the signal?&lt;/p&gt;&#xA;" OwnerUserId="4563" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-11T09:49:21.693" LastActivityDate="2011-05-11T09:49:21.693" Title="Using kurtosis to assess significance of components from independent component analysis" Tags="&lt;pca&gt;&lt;kurtosis&gt;&lt;ica&gt;" />
  <row Id="10645" PostTypeId="2" ParentId="10643" CreationDate="2011-05-11T09:54:22.177" Score="0" ViewCount="" Body="&lt;p&gt;Can you not use the empirical distribution's 95% (or whichever you prefer) confidence limit? If your sample size is big enough, this ought to be a reasonable approximation.&lt;/p&gt;&#xA;" OwnerUserId="4257" LastActivityDate="2011-05-11T09:54:22.177" CommentCount="2" />
  <row Id="10646" PostTypeId="2" ParentId="1676" CreationDate="2011-05-11T10:03:37.470" Score="1" ViewCount="" Body="&lt;p&gt;Some packages are very useful in R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I will just recommand &lt;strong&gt;kernlab&lt;/strong&gt; for Kernel-based Machine Learning Lab and &lt;strong&gt;e1071&lt;/strong&gt; for SVM and &lt;strong&gt;ggplot2&lt;/strong&gt; for graphics&lt;/p&gt;&#xA;" OwnerUserId="4531" LastActivityDate="2011-05-11T10:03:37.470" CommunityOwnedDate="2011-05-11T10:03:37.470" />
  <row Id="10647" PostTypeId="2" ParentId="9756" CreationDate="2011-05-11T10:15:49.427" Score="0" ViewCount="" Body="&lt;p&gt;I have worked on active learning in classification and in SVM, that problem was same for me, if the boundary you found out by first model isn't that good the probability to have a good label for new points will decrease. If you have any other method to labelize your new generated points rather than using the boundary that can be a good way and your accuracy for the new generated boundary will be better.&lt;/p&gt;&#xA;" OwnerUserId="4531" LastActivityDate="2011-05-11T10:15:49.427" />
  <row Id="10648" PostTypeId="1" CreationDate="2011-05-11T10:50:48.933" Score="3" ViewCount="29" Body="&lt;p&gt;Hello,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm trying to model a situation of a system which reacts to inputs (deterministically) and according to an internal state generates an output. According to a feature the input (or the output, it's almost indifferent in this case) the system changes state. States are hidden, their number is unknown and one can only feed inputs and observe outputs.&#xA;The probability of the system being in any given state can be considered uniform.&#xA;If it weren't for the inputs I'd try to maximise the BIC by using Baum-Welch's algorithm and changing the number of hidden states, but given the inputs I couldn't find clear, relevant literature helping me to deal with the problem.&#xA;I've also been pointed to POMDPs but then how would I model the reward function?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help is appreciated&lt;/p&gt;&#xA;" OwnerUserId="4564" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-11T12:24:58.293" LastActivityDate="2011-05-11T12:24:58.293" Title="Input/output hidden Markov models" Tags="&lt;markov-process&gt;" />
  <row Id="10649" PostTypeId="1" CreationDate="2011-05-11T12:28:39.347" Score="2" ViewCount="74" Body="&lt;p&gt;How can one obtain confidence limits of predicted values in ARIMA?&lt;/p&gt;&#xA;" OwnerUserId="4427" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-05-11T16:51:38.367" LastActivityDate="2011-05-11T21:10:48.220" Title="How to obtain confidence limits of predicted values in ARIMA?" Tags="&lt;forecasting&gt;" AnswerCount="2" />
  <row Id="10650" PostTypeId="2" ParentId="73" CreationDate="2011-05-11T12:33:30.563" Score="1" ViewCount="" Body="&lt;p&gt;For me &#xA;I am using &lt;strong&gt;kernlab&lt;/strong&gt; for Kernel-based Machine Learning Lab and &lt;strong&gt;e1071&lt;/strong&gt; for SVM and &lt;strong&gt;ggplot2&lt;/strong&gt; for graphics&lt;/p&gt;&#xA;" OwnerUserId="4531" LastActivityDate="2011-05-11T12:33:30.563" CommunityOwnedDate="2011-05-11T12:33:30.563" />
  <row Id="10651" PostTypeId="2" ParentId="10631" CreationDate="2011-05-11T12:35:05.150" Score="2" ViewCount="" Body="&lt;p&gt;I would have thought a &quot;cubic regression&quot; would work well for a cubic relationship.  Call $Y_{i}$ the dependent variable, and $X_{i}$ the independent variable (or regressor).  You simply use a polynomial regression:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Y_{i}=\left(\sum_{j=0}^{p}\beta_{j}X_{i}^{j}\right)+e_{i}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would use BIC to select the value of $p$.  To do this is very easy - calculate the coefficient of determination $R_{p}^{2}$ from a standard OLS regression output.  Then a convenient form of BIC is given by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$BIC_{p}=n\log(1-R_{p}^{2})+p\log(n)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Although this is the standard form, with the natural logarithm's, a more convenient numerical form is given by &#xA;$$BIC10_{p}=-\frac{1}{2}\log_{10}(e)BIC_{p}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reason I say this is that in this form above, you get BIC expressed in based 10 log units, and this leads to a very quick interpretation of the actual number of the BIC.  If BIC is positive, then the current order $p$ polynomial is more supported by the data (compared to intercept only model), and the numerical value in odds form is $10^{BIC10_{p}}$.  So if $BIC10_{p}=1$, then the order $p$ polynomial is 10 times more likely than the intercept only model, if $BIC10_{p}=10$ then the order $p$ polynomial is 10 billion times more likely.  BIC10 tells you how many digits are in the odds ratio.  So a reasonable way to proceed is to continue to increase the order of a polynomial until $BIC10_{p}$ becomes sufficiently large.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One thing to be careful of though, is that this type of procedure is not likely to work well for extrapolation outside the range of the $X_{i}$ values.  This is mainly because this is a data driven procedure.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-05-11T12:35:05.150" />
  <row Id="10652" PostTypeId="2" ParentId="7224" CreationDate="2011-05-11T12:53:49.673" Score="0" ViewCount="" Body="&lt;p&gt;You need to put on an algorithm detecting which person that picture is referring to. You can build a model based on different portrait pictures of famous personality and use classifiers to ensure that this picture is referring to one of your database picture. You need to use a certain classifier based on different parameters liked to the face, like distance between eyes or other parameters to rise up the accuracy of your model. &#xA;There is also skin analysis.&#xA;The most important is to build a good classifier. This method can be vulnerable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But there is also a very good project working on face recognition &lt;a href=&quot;http://opencv-code.com/Opencv_Face_Detection&quot; rel=&quot;nofollow&quot;&gt;http://opencv-code.com/Opencv_Face_Detection&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="4531" LastActivityDate="2011-05-11T12:53:49.673" />
  <row Id="10653" PostTypeId="2" ParentId="10604" CreationDate="2011-05-11T13:26:05.713" Score="0" ViewCount="" Body="&lt;p&gt;I have reviewed the output and the forecast refelects an AR(12) in the error term which translates to a 12 period weighted forecast using the last 12 values of both your predictor series as the AR polynomial acts a multiplier across all series ( X,Y,Z ). Without getting into it in great detail , your model specification or rather lack of specification is in my opinion &quot;found wanting&quot;. Unfortunately the SAS procedure assumes that the differencing operators required to make the original series stationary is the same as the differencing operators in the Transfer Function. Furthermore the ARIMA component in the Transfer Function is the same as the ARIMA component for the Univariate Analysis of the dependent series. This structure should be identified form the residuals of a suitably formed Transfer Function that does not have an ARIMA structure. Finally your specification ( by default ) of the ARIMA component in the Transfer Function is a &quot;common factor&quot;. What you need to do is to identify the forms of differencing ( if any ) for all three series and the nature of the response (PDL/ADL/LAG STRUCTURE) for EACH of the two inputs. After estimating such a tentative model verify that there are no Level Shifts/Local Time Trends/Seasonal or 1 time Pulses in the tentative set of model errors via Interevention Detection schemes. Furthermore one must ensure the model errors have constant variance and that the parameters of the model haven't proved to have changed over time and that the acf of the model errors is free of any significant structure AND that these errors are uncorrelated with either of the two pre-whitened input series. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In summary you are getting what you want but you might not want what you are getting !You might consider posting the original data for the three series and have the list members ( including myself ) aid you in constructing a minimally sufficient model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: I found some material on the web that might be of help to you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For illustration, say the non-zero lags are 2 and 4. The process y might be estimated as follows using an x as an input. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;estimate input=( 2$(2) x ) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The input is of the form cB*&lt;em&gt;2+ dB&lt;/em&gt;*4  =   B*&lt;em&gt;2( c + dB&lt;/em&gt;*2). It is this latter form that gives the form of the input statement&lt;/p&gt;&#xA;" OwnerUserId="3382" LastEditorUserId="3382" LastEditorDisplayName="" LastEditDate="2011-05-12T22:10:04.407" LastActivityDate="2011-05-12T22:10:04.407" CommentCount="9" />
  <row Id="10654" PostTypeId="2" ParentId="10631" CreationDate="2011-05-11T13:31:27.083" Score="4" ViewCount="" Body="&lt;p&gt;Restricted cubic splines (natural splines) are an excellent choice.  These are piecewise cubic polynomials that can fit any shape given enough knots.  The following code in R shows how to fit such relationships and to plot the fit with confidence bands.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(rms)&#xA;dd &amp;lt;- datadist(mydata); options(datadist='dd')&#xA;f &amp;lt;- ols(y ~ rcs(x1, 5))  # 5 knots at default locations&#xA;f   # print model stats&#xA;plot(Predict(f))  # or plot(Predict(f, x1)) # plots over 10th smallest to 10th largest observation&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="4253" LastActivityDate="2011-05-11T13:31:27.083" />
  <row Id="10655" PostTypeId="1" AcceptedAnswerId="10706" CreationDate="2011-05-11T13:33:07.000" Score="2" ViewCount="64" Body="&lt;p&gt;I have a scatter plot of (for example) height against age.  How does one calculate for an individual point the percentile of the height for a given age?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suggestions in R would be most appreciated.  Thanks!&lt;/p&gt;&#xA;" OwnerUserId="1991" LastActivityDate="2011-05-12T21:10:18.593" Title="How to calculate a percentile of y for a given x given a series of (x, y)?" Tags="&lt;percentile&gt;" AnswerCount="2" />
  <row Id="10656" PostTypeId="1" CreationDate="2011-05-11T13:44:50.520" Score="2" ViewCount="63" Body="&lt;p&gt;Hi everyone, i just wanted to know if i could use factor scores and crosstab it with demographics (i.e. gender, age, etc.)? I have 69 likert-scale variables and run factor analysis on SPSS. It gave me 10 new variables (types of personality. i just wanted to see the demographics of each new variables. Thanks!!!&lt;/p&gt;&#xA;" OwnerUserId="4565" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-11T19:34:00.410" LastActivityDate="2011-05-11T19:34:00.410" Title="Crosstab factor scores generated by factor analysis" Tags="&lt;factor-analysis&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="10657" PostTypeId="1" CreationDate="2011-05-11T13:55:41.987" Score="3" ViewCount="59" Body="&lt;p&gt;In relation to web usage mining from a log file, can you cluster data without performing User and/or Session identification? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I mean,let's say I have these entries:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;123.234.324.122 [timestamp] &quot;GET /cars/sport/porsche.jpg&quot; 200 23432 &quot;http://topgear.com/cars&quot; &quot;Mozilladsfsd&quot;  &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;120.23.324.122 [timestamp] &quot;GET /bikes/sport/r1.jpg&quot; 200 23432 &quot;http://topgear.com/cars&quot; &quot;Mozilladsfsd&quot;  &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;13.234.324.122 [timestamp] &quot;GET /cars/utility/micra.jpg&quot; 200 23432&quot;http://topgear.com/cars&quot; &quot;Mozilladsfsd&quot;  &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So,in this scenario, I just need to cluster based on which cars have been viewed more frequently etc etc..Do I need user identification and session identification then? Or can I just consider the URLs and cluster on them?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because as far as the traditional Web Usage Mining approach goes and all the papers I've gone through suggest, you do preprocessing,then the pattern-discovery comes along..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is why not jump to the pattern discovery straight-away????&lt;/p&gt;&#xA;" OwnerUserId="4402" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-11T19:38:20.157" LastActivityDate="2011-05-11T20:07:20.180" Title="Can one cluster web log data without performing user or session identification?" Tags="&lt;clustering&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="10658" PostTypeId="2" ParentId="10062" CreationDate="2011-05-11T13:57:14.200" Score="3" ViewCount="" Body="&lt;p&gt;Package &lt;a href=&quot;http://cran.r-project.org/web/packages/mclust/mclust.pdf&quot; rel=&quot;nofollow&quot;&gt;Mclust&lt;/a&gt; is nice.  The &lt;em&gt;mclust&lt;/em&gt; function fits a mixture of normals distribution to data.  You can automatically choose the number of components based on BIC (&lt;em&gt;mclustmodel&lt;/em&gt;) or specify the number of components.  There is also no need to convert your data into a data frame.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, package &lt;a href=&quot;http://cran.r-project.org/web/packages/mixtools/mixtools.pdf&quot; rel=&quot;nofollow&quot;&gt;Mixtools&lt;/a&gt; and the function &lt;em&gt;normalmixEM&lt;/em&gt; fits a mixture of normals.&lt;/p&gt;&#xA;" OwnerUserId="2310" LastActivityDate="2011-05-11T13:57:14.200" />
  <row Id="10659" PostTypeId="2" ParentId="10655" CreationDate="2011-05-11T14:18:39.307" Score="1" ViewCount="" Body="&lt;p&gt;'The' percentile for a given age implies some sort of regression (i.e. you can find 'the' mean predicted height from a given age).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you have found this, the result depends on your assumptions: if you want no assumptions (besides the regression's), find how many of the heights in your original data are smaller than the predicted one for your age (=use empirical distribution).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Otherwise, you can fit whatever model you like to the marginal distribution of height and then find the percentile of the predicted height in that distribution.&lt;/p&gt;&#xA;" OwnerUserId="4257" LastActivityDate="2011-05-11T14:18:39.307" CommentCount="3" />
  <row Id="10660" PostTypeId="2" ParentId="10657" CreationDate="2011-05-11T14:58:55.840" Score="1" ViewCount="" Body="&lt;p&gt;If you don't identify your sessions/users, you are clustering different things: one user who is an insane adept of any given car and looks at its picture dayly could have a huge impact on your results, though you're probably not interested in this.&lt;/p&gt;&#xA;" OwnerUserId="4257" LastActivityDate="2011-05-11T14:58:55.840" />
  <row Id="10661" PostTypeId="2" ParentId="10591" CreationDate="2011-05-11T15:28:51.700" Score="3" ViewCount="" Body="&lt;p&gt;Here is a sketch of a proof which combines three ideas: &lt;strong&gt;(a)&lt;/strong&gt; the delta method, &lt;strong&gt;(b)&lt;/strong&gt; variance-stabilization transformations and &lt;strong&gt;(c)&lt;/strong&gt; the closure of the Poisson distribution under independent sums.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, let's consider a &lt;em&gt;sequence&lt;/em&gt; of iid Poisson random variables $X_1, X_2, \ldots$ with mean $\lambda &amp;gt; 0$. Then, the Central Limit Theorem asserts that&#xA;$$\newcommand{\barX}{\bar{X}_n}\newcommand{\convd}{\,\xrightarrow{\,d\,}\,}\newcommand{\Nml}{\mathcal{N}}&#xD;&#xA;\sqrt{n} (\barX - \lambda) \convd \Nml(0,\lambda) \&amp;gt; .&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Notice that the asymptotic variance depends on the (presumably unknown) parameter $\lambda$. It would be nice if we could find some function of the data other than $\bar{X}_n$ such that, after centering and rescaling, it had the same asymptotic variance no matter what the parameter $\lambda$ was.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;a href=&quot;http://en.wikipedia.org/wiki/Delta_method&quot; rel=&quot;nofollow&quot;&gt;delta method&lt;/a&gt; provides a handy way for determining the distribution of smooth functions of some statistic whose limiting distribution is already known. Let $g$ be a function with continuous first derivative such that $g&amp;#39;(\lambda) \neq 0$. Then, by the delta method (specialized to our particular case of interest),&#xA;$$&#xD;&#xA;\sqrt{n}\big(g(\barX) - g(\lambda)\big) \convd \Nml(0, \lambda g&amp;#39;(\lambda)^2) \&amp;gt;.&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, how can we make the asymptotic variance constant (say, the value $1$) for all possible $\lambda$? From the expression above, we know we need to solve&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$g&amp;#39;(\lambda) = \lambda^{-1/2} \&amp;gt;.$$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is not hard to see that the general antiderivative is $g(\lambda) = 2 \sqrt{\lambda} + c$ for any $c$, and the limiting distribution is invariant to the choice of $c$ (by subtraction), so we can set $c = 0$ without loss of generality. Such a function $g$ is called a &lt;strong&gt;variance-stabilizing transformation&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hence, by the delta method and our choice of $g$, we conclude that&#xA;$$&#xD;&#xA;\sqrt{n}\Big(2\sqrt{\barX} - 2\sqrt{\lambda}\Big) \convd \Nml(0, 1) \&amp;gt;.&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, the Poisson distribution is closed under independent sums. So, if $X$ is Poisson with mean $\lambda$, then there exist random variables $Z_1, \ldots, Z_n$ that are iid Poisson with mean $\lambda/n$ such that $\sum_{i=1}^n Z_i$ has the same distribution as $X$. This motivates the approximation in the case of a single Poisson random variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What &lt;a href=&quot;http://www.jstor.org/stable/2984159&quot; rel=&quot;nofollow&quot;&gt;Anscombe (1948)&lt;/a&gt; found was that modifying the transformation $g$ (slightly) to $\tilde{g}(\lambda) = 2\sqrt{\lambda + b}$ for some constant $b$ actually worked better for smaller $\lambda$. In this case, $b = 3/8$ is about optimal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that this modification &quot;destroys&quot; the true variance-stabilizing property of $g$, i.e., $\tilde{g}$ is &lt;em&gt;not&lt;/em&gt; variance-stabilizing in the strict sense. But, it is close and gives better results for smaller $\lambda$.&lt;/p&gt;&#xA;" OwnerUserId="2970" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-05-12T12:56:29.837" LastActivityDate="2011-05-12T12:56:29.837" CommentCount="1" />
  <row Id="10662" PostTypeId="1" AcceptedAnswerId="10663" CreationDate="2011-05-11T15:37:13.417" Score="3" ViewCount="98" Body="&lt;p&gt;I'm working with a lot of data that was collected by obstetricians regarding the health of infants (birth weight, gestational age at delivery, mother's BMI), and I am trying to connect this data with geometric measurements performed on microscopic slide scans for each associated placenta (area, perimeter, number of blood vessels). Each mother-infant-placenta trio is identified with a lab ID so it is possible to know which is which, but there are only 27 sets of mother-infant-placenta.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All the clinical data were taken before I arrived on the scene. I was pretty much given the placenta slide images, and an excel sheet of the clinical data. Then I performed the geometric measurements of the placentas. So the data was not taken with my purpose in mind.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is, what can I do with this data? I collected measurements with some clinical knowledge that the condition of a placenta is both an influence on and reflection of the infant health outcome. But I desperately need advice on which statistical/data mining techniques I can use to see how my measurements affect/are an indicator of infant health. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any hope for ad-hoc analysis on a small sample size?&lt;/p&gt;&#xA;" OwnerUserId="4566" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-11T19:43:51.520" LastActivityDate="2011-05-11T19:46:48.087" Title="What to do with a small (27) medical dataset?" Tags="&lt;hypothesis-testing&gt;&lt;data-mining&gt;&lt;small-sample&gt;&lt;eda&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="10663" PostTypeId="2" ParentId="10662" CreationDate="2011-05-11T15:57:01.233" Score="4" ViewCount="" Body="&lt;p&gt;If you're looking for statistical significance I wouldn't hold out hope unless you have a very targeted hypothesis and/or there is a very strong effect. But certainly you could &lt;em&gt;generate&lt;/em&gt; some new hypotheses with this data via some exploratory analysis. With 6 variables overall I'm not sure I'd start with any sophisticated modeling. Never underestimate the power of scatterplots and histograms :)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One really simple thing to do would be to run PCA and see if the scores on any of the components have an apparent relationship with the response(s) you're interested in. It's probably a good&amp;lt;\strike&gt; reasonable idea anyhow since your measurements are certainly correlated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: My thought on using PCA was basically to reduce the area/perimeter/number variables to a single dimension. Not strictly necessary but it might make visualizing the relationships easier.&lt;/p&gt;&#xA;" OwnerUserId="26" LastEditorUserId="26" LastEditorDisplayName="" LastEditDate="2011-05-11T19:46:48.087" LastActivityDate="2011-05-11T19:46:48.087" CommentCount="12" />
  <row Id="10664" PostTypeId="1" CreationDate="2011-05-11T16:17:56.080" Score="2" ViewCount="82" Body="&lt;p&gt;I am trying to analyse a group of 4 questions that are on a 5 point scale.  I need to group the answers for each question based on age.  There are three different ages.  How would I go about doing this?&lt;/p&gt;&#xA;" OwnerUserId="4567" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-12T01:47:22.627" LastActivityDate="2011-05-12T10:58:56.187" Title="How to examine group differences on several 5-point items using SPSS?" Tags="&lt;spss&gt;&lt;likert&gt;" AnswerCount="2" />
  <row Id="10665" PostTypeId="2" ParentId="10649" CreationDate="2011-05-11T16:40:47.500" Score="3" ViewCount="" Body="&lt;p&gt;One idea would be to use the &lt;a href=&quot;http://cran.r-project.org/web/packages/forecast/index.html&quot; rel=&quot;nofollow&quot;&gt;forecast&lt;/a&gt; package in &lt;a href=&quot;http://www.r-project.org/&quot; rel=&quot;nofollow&quot;&gt;R&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(forecast)&#xA;fit &amp;lt;- auto.arima(WWWusage)&#xA;fit&#xA;f &amp;lt;- forecast(fit,h=20)&#xA;f&#xA;plot(f)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can also give auto.arima parameters to use, rather than allowing it to fit its own.  I'm not sure how to obtain confidence intervals for the historic period-- you could try 'rolling' through the dataset, producing 1-step ahead forecasts+confidence intervals.&lt;/p&gt;&#xA;" OwnerUserId="2817" LastEditorUserId="2817" LastEditorDisplayName="" LastEditDate="2011-05-11T16:48:50.180" LastActivityDate="2011-05-11T16:48:50.180" CommentCount="5" />
  <row Id="10666" PostTypeId="2" ParentId="10656" CreationDate="2011-05-11T16:43:02.240" Score="3" ViewCount="" Body="&lt;p&gt;In SPSS, choose Analyze...Descriptive Statistics...Explore.  The factor scores will be your dependents, and a demographic grouping will be your &quot;factor&quot; in this procedure.  I'd choose Plots Only to start with and request boxplots.  You can get either factor levels together or dependents together:  you can experiment.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Later, when you want means and standard deviations and such, I'd use the Summarize command, located in the menus at Analyze...Reports...Case Summaries. You can also experiment with  &quot;means [var list] by [var list]/stat anova.&quot;  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I'd caution you that some of your 10 factors are likely to be marginal ones that have small eigenvalues and explain little of the total item variance.  Maybe consider making your criteria stricter for factor extraction, such as using a subcommand like &quot;/criteria mineigen (1.5)&quot; or 2.0 instead of 1.0 which is so commonly used.  It'll also be a plus to reduce your number of factors when it comes to displaying and explaining your results.&lt;/p&gt;&#xA;" OwnerUserId="2669" LastActivityDate="2011-05-11T16:43:02.240" CommentCount="2" />
  <row Id="10667" PostTypeId="1" AcceptedAnswerId="10670" CreationDate="2011-05-11T16:58:07.793" Score="1" ViewCount="58" Body="&lt;p&gt;I have to assess for each product p, the odds ratio associated (success/failure).  The data are in this table:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;N_success   N_trials    p1    p2   p3    p4   p5&#xA;5          310         n     n     n     n    n&#xA;17         700         n     y     n     n    y&#xA;12         650         y     y     y     n    n&#xA;27         214         n     y     n     n    y&#xA;0          87          n     n     n     y    n&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So I did that and I got the odds ratios for each plus the 95% asymptotic confidence intervals:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;p1= (0.2558322 0.48442194 0.91725993)&#xA;p2= (1.1584454 2.9114056 7.316946)&#xA;p3= (0.2558322 0.48442194 0.91725993)&#xA;p4= nil&#xA;p5= (1.738197 3.0642326 5.4018736)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now is this correct way to assess? There are at least two situations where the lower 95% CI of the OR is above 1 (p2 and p4) but should I not take into account multiple comparisons? What is the best way to do that with odds ratios (Bonferroni, others)? Is it better to use logistic regressions (and how)? Thanks&lt;/p&gt;&#xA;" OwnerUserId="4569" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-05-11T18:18:03.137" LastActivityDate="2011-05-11T18:52:39.843" Title="Odds ratios multiple comparisons" Tags="&lt;logistic-regression&gt;&lt;multiple-comparisons&gt;&lt;odds-ratio&gt;" AnswerCount="1" />
  <row Id="10668" PostTypeId="2" ParentId="10369" CreationDate="2011-05-11T18:22:17.520" Score="1" ViewCount="" Body="&lt;p&gt;Observe that the random variable $i_j$ is a function of $\mathbf{Z} = (Z_1, \ldots, Z_n)$ only. For an $n$-vector, $\mathbf{z}$, we write $i_j(\mathbf{z})$ for the index of the $j$th largest coordinate. Let also $P_z(A) = P(X_1 \in A \mid Z_1 = z)$ denote the conditional distribution of $X_1$ given $Z_1$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we break probabilities down according to the value of $i_j$ and desintegrate w.r.t. $\mathbf{Z}$ we get &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\begin{array}{rcl}&#xD;&#xA;P(X_{i_j} \in A) &amp;amp; = &amp;amp; \sum_{k} P(X_k \in A, i_j = k) \\&#xD;&#xA;&amp;amp; = &amp;amp;\sum_k \int_{(i_j(z) = k)} P(X_k \in A \mid \mathbf{Z} = \mathbf{z})  P(\mathbf{Z} \in d\mathbf{z}) \\&#xD;&#xA;&amp;amp; = &amp;amp; \sum_k \int_{(i_j(z) = k)} P(X_k \in A \mid Z_k = z_k) P(\mathbf{Z} \in d\mathbf{z}) \\&#xD;&#xA;&amp;amp; = &amp;amp; \sum_k \int_{(i_j(z) = k)} P_{z_k}(A) P(\mathbf{Z} \in d\mathbf{z}) \\&#xD;&#xA;&amp;amp; = &amp;amp; \int P_{z}(A) P(Z_{i_j} \in dz) \\&#xD;&#xA;\end{array}$$   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This argument is quite general and relies only on the stated i.i.d. assumptions, and $Z_k$ could be any given function of $(X_k, Y_k)$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Under the assumptions of normal distributions (taking $\sigma_y = 1$) and $Z_k$ being the sum, the conditional distribution of $X_1$ given $Z_1 = z$ is &#xA;$$N\left(\frac{\sigma_x^2}{1+\sigma_x^2} z, \sigma_x^2\left(1 - \frac{\sigma_x^2}{1+\sigma_x^2}\right)\right)$$&#xA;and @probabilityislogic shows how to compute the distribution of $Z_{i_j}$, hence we have explicit expressions for both the distributions that enter in the last integral above. Whether the integral can be computed analytically is another question. You might be able to, but off the top of my head I can't tell if it is possible. For asymptotic analysis when $\sigma_x \to 0$ or $\sigma_x \to \infty$ it might not be necessary.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The intuition behind the computation above is that this is a conditional independence argument. Given $Z_{k} = z$ the variables $X_{k}$ and $i_j$ are independent. &lt;/p&gt;&#xA;" OwnerUserId="4376" LastActivityDate="2011-05-11T18:22:17.520" />
  <row Id="10669" PostTypeId="1" AcceptedAnswerId="10685" CreationDate="2011-05-11T18:37:16.257" Score="1" ViewCount="54" Body="&lt;p&gt;I am running a model for which I am getting a very bad percentage detection of events in the confusion matrix (basically my true positives). &#xA;Obviously that implies my false negatives are too hight. &#xA;When I gave these dataset to a neural network node or a decision tree node, I see that the percentage detection by these two techniques is quite good (65% - 67% vis a vis mine 15%). &#xA;I went to the decision tree diagram and saw the various cuts under which it divides the population. &#xA;I obviously understand that the variable which falls on the root of the tree has highest importance and the leafs have lowest. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;How can the decision tree &quot;tree&quot; help me create categorical variables or treat continuous variables so that the accuracy of my model improves? &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;To clarify, if a decision tree can generate a matrix with 65% detection, it would have some rule inside it to get such accuracy.&#xA;These rules would display in the tree diagram we get as output. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Can we use this tree to create out variables in a different way and get close to the accuracy given by decision tree? &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="1763" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-12T13:51:27.850" LastActivityDate="2011-05-12T13:51:27.850" Title="Decision tree output -- learning" Tags="&lt;decision-trees&gt;" AnswerCount="1" />
  <row Id="10670" PostTypeId="2" ParentId="10667" CreationDate="2011-05-11T18:52:39.843" Score="0" ViewCount="" Body="&lt;p&gt;Hi, I believe you could use &quot;simultaneous score confidence interval for OR&quot; to analyze your question. The reference is Agresti et al. 2008 Simultaneous confidence intervals for comparing binomial parameters. Biometrics 64 1270-1275. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The R code is available in &lt;a href=&quot;http://www.stat.ufl.edu/~aa/cda/software.html&quot; rel=&quot;nofollow&quot;&gt;http://www.stat.ufl.edu/~aa/cda/software.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sincerely,&lt;/p&gt;&#xA;" OwnerUserId="4559" LastActivityDate="2011-05-11T18:52:39.843" />
  <row Id="10671" PostTypeId="2" ParentId="10662" CreationDate="2011-05-11T19:11:30.590" Score="0" ViewCount="" Body="&lt;p&gt;Hi,&#xA;I agree with JMS, you will need to plot each of your variable first because PCA requires the normality assumption. If your variables are not normally distributed then it is not appropriate to use PCA before transforming the variables. I think you will need to ask yourself, what you really want to know from this data set (set up your hypothesis) then you will be able to pick the right statistical tests.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is not good to dichotomize continuous variables into categorical variables because you will lose power to detect the difference. However, if this is the case, You could use &quot;odds ratio&quot;, &quot;risk difference&quot; etc to interpret your data sets. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sincerely,&lt;/p&gt;&#xA;" OwnerUserId="4559" LastActivityDate="2011-05-11T19:11:30.590" CommentCount="5" />
  <row Id="10672" PostTypeId="1" AcceptedAnswerId="10757" CreationDate="2011-05-11T19:57:03.987" Score="4" ViewCount="63" Body="&lt;p&gt;I've read about a number of algorithms for solving n-armed bandit problems like $\epsilon$-greedy, softmax, and UCB1, but I'm having some trouble sorting through what approach is best for minimizing regret.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a known optimal algorithm for solving the n-armed bandit problem? Is there a choice of algorithm that seems to perform best in practice?&lt;/p&gt;&#xA;" OwnerUserId="4281" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-05-12T02:38:49.657" LastActivityDate="2011-05-13T08:51:01.957" Title="Optimal algorithm for solving n-armed bandit problems?" Tags="&lt;machine-learning&gt;&lt;reinforcement-learning&gt;" AnswerCount="1" CommentCount="5" />
  <row Id="10673" PostTypeId="2" ParentId="10657" CreationDate="2011-05-11T20:07:20.180" Score="0" ViewCount="" Body="&lt;p&gt;Cluster analysis does not involve hypothesis testing per se, but is really just a collection of different similarity algorithms for exploratory analysis. You can force hypothesis testing somewhat but the results are often inconsistent, since cluster changes are very sensitive to changes in parameters. So the answer is yes, you can do it, but be careful about making specific statistical inference.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As to your other point. As in any data analysis, your data should be as clean and as representative as possible, so I would avoid jumping steps.&lt;/p&gt;&#xA;" OwnerUserId="3489" LastActivityDate="2011-05-11T20:07:20.180" />
  <row Id="10674" PostTypeId="1" CreationDate="2011-05-11T20:07:34.483" Score="0" ViewCount="48" Body="&lt;p&gt;I have a two dimensional data set that looks like (t, x) where t is a time in seconds when event X happened. X ranges from [0, 200]. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to visualize the frequency of each X at time t over some time period. I guess this would be a bar graph with x-axis being event #, y-axis being frequency, and z-axis of time t. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furthermore, I would like to group all events that happen within say a 5 second interval to count towards the same frequency bar on the y-axis. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If there is a way to do this with R that would be even better. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My goal is to get a sense how often some event occurs over the course of a day, and when certain events happen a lot or infrequently. If you know of a better way to understand this information, I am all ears. &lt;/p&gt;&#xA;" OwnerUserId="4571" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-12T06:38:30.490" LastActivityDate="2011-05-12T06:38:30.490" Title="Modeling frequency over time" Tags="&lt;data-visualization&gt;" CommentCount="2" />
  <row Id="10676" PostTypeId="1" AcceptedAnswerId="10677" CreationDate="2011-05-11T20:32:04.700" Score="2" ViewCount="99" Body="&lt;p&gt;Lets say I have a highly dimensional classification problem with a lot of noise, and I want to improve my results by removing some of the noisy variables.  I've &lt;a href=&quot;http://research.microsoft.com/pubs/69946/tr-2002-63.pdf&quot; rel=&quot;nofollow&quot;&gt;read&lt;/a&gt; &lt;a href=&quot;http://www.andrew.cmu.edu/user/minhhoan/papers/SVMFeatureWeight_PR.pdf&quot; rel=&quot;nofollow&quot;&gt;several&lt;/a&gt; &lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pubmed/16606446&quot; rel=&quot;nofollow&quot;&gt;papers&lt;/a&gt; on using SVMs for feature selection, but I'm at a loss as to how to implement this in R.  Are there pre-existing packages that do this, or am I going to have to roll my own?&lt;/p&gt;&#xA;" OwnerUserId="2817" LastActivityDate="2011-05-12T09:22:47.270" Title="Using an SVM for feature selection" Tags="&lt;r&gt;&lt;svm&gt;" AnswerCount="3" />
  <row Id="10677" PostTypeId="2" ParentId="10676" CreationDate="2011-05-11T20:47:54.837" Score="2" ViewCount="" Body="&lt;p&gt;As I understand them, SVMs have built-in &lt;em&gt;regularization&lt;/em&gt; because they tend to penalize large weights of predictors which amounts to favor simpler models. They're often used with &lt;a href=&quot;http://www.brainvoyager.com/bvqx/doc/UsersGuide/WebHelp/Content/MVPATools/Recursive_Feature_Elimination.htm&quot; rel=&quot;nofollow&quot;&gt;recursive feature elimination&lt;/a&gt; (in neuroimaging paradigms, at least).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;About R specifically, there's the &lt;a href=&quot;http://cran.r-project.org/web/packages/kernlab/index.html&quot; rel=&quot;nofollow&quot;&gt;kernlab&lt;/a&gt; package, by &lt;a href=&quot;http://alex.smola.org/&quot; rel=&quot;nofollow&quot;&gt;Alex Smola&lt;/a&gt; who co-authored &lt;em&gt;Learning with Kernels&lt;/em&gt; (2002, MIT Press), which implements SVM (in addition to &lt;a href=&quot;http://cran.r-project.org/web/packages/e1071/index.html&quot; rel=&quot;nofollow&quot;&gt;e1071&lt;/a&gt;). However, if you are after a dedicated framework, I would warmly recommend the &lt;a href=&quot;http://caret.r-forge.r-project.org/Classification_and_Regression_Training.html&quot; rel=&quot;nofollow&quot;&gt;caret&lt;/a&gt; package.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-05-11T20:47:54.837" CommentCount="4" />
  <row Id="10678" PostTypeId="2" ParentId="10649" CreationDate="2011-05-11T21:10:48.220" Score="3" ViewCount="" Body="&lt;p&gt;The confidence limits for an ARIMA forecast are based upon the PSI WEIGHTS . The PSI WEIGHTS are easily computed by representing the ARIMA MODEL as a pure MOVING AVERAGE MODEL. One should not be dependent upon software ( any software ! ) for answers.&lt;/p&gt;&#xA;" OwnerUserId="3382" LastActivityDate="2011-05-11T21:10:48.220" />
  <row Id="10679" PostTypeId="2" ParentId="10664" CreationDate="2011-05-11T22:19:51.593" Score="3" ViewCount="" Body="&lt;p&gt;Here are 3 options; hard brackets need to be filled in, while braces contain optional subcommands:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cross [varlist] by age {/cells count col row}.&#xA;&#xA;means [varlist] by age {/stat anova}.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;summarize command - best obtained through the menus via Analyze...Reports...Case Summaries.  Then you may need to double-click the resulting pivot table and right-click ... Pivoting Trays to rearrange.&lt;/p&gt;&#xA;" OwnerUserId="2669" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-12T01:55:53.997" LastActivityDate="2011-05-12T01:55:53.997" />
  <row Id="10680" PostTypeId="1" AcceptedAnswerId="10724" CreationDate="2011-05-11T17:47:50.800" Score="10" ViewCount="245" Body="&lt;p&gt;This is sort of an open ended question but I wanna be clear. Given a sufficient population you might be able to learn something (this is the open part) but whatever you learn about your population, when is it ever applicable to a member of the population?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From what I understand of statistics it's never applicable to a single member of a population, however, all to often I find myself in a discussion where the other person goes &quot;I read that 10% of the world population has this disease&quot; and continue to conclude that every tenth person in the room has this disease.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand that ten people in this room is not a big enough sample for the statistic to be relevant but apparently a lot don't.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then there's this thing about &lt;em&gt;large enough&lt;/em&gt; samples. You only need to probe a large enough population to get reliable statistics. This though, isn't it proportional to the complexity of the statistic? If I'm measuring something that's very rare, doesn't that mean I need a much bigger sample to be able to determine the relevance for such a statistic?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The thing is, I truly question the validity of any newspaper or article when statistics is involved, they way it's used to build confidence. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;That's a bit of background.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Back to the question, &lt;strong&gt;in what ways can you NOT or may you NOT use statistics to form an argument&lt;/strong&gt;. I negated the question because I'd like to find out more about common misconceptions regarding statistics.&lt;/p&gt;&#xA;" OwnerUserId="4576" LastEditorUserId="830" LastEditorDisplayName="" LastEditDate="2011-05-12T01:23:07.240" LastActivityDate="2011-05-12T20:17:49.997" Title="How to NOT use statistics" Tags="&lt;big-list&gt;" AnswerCount="7" CommentCount="6" FavoriteCount="4" />
  <row Id="10681" PostTypeId="2" ParentId="10680" CreationDate="2011-05-11T17:56:10.473" Score="7" ViewCount="" Body="&lt;p&gt;Unless the people in the room are a random sample of the world's population, any conclusions based on statistics about the world's population are going to be very suspect.  One out of every 5 people in the world is Chinese, but none of my five children are...&lt;/p&gt;&#xA;" LastActivityDate="2011-05-11T17:56:10.473" />
  <row Id="10682" PostTypeId="2" ParentId="10680" CreationDate="2011-05-11T17:57:05.260" Score="5" ViewCount="" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;To address overapplication of statistics to small samples, I recommend countering with well-known jokes (&quot;I am so excited, my mother is pregnant again and my baby sibling will be Chinese.&quot; &quot;Why?&quot; &quot;I have read that every fourth baby is Chinese.&quot;).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Actually, I recommend jokes to address all kinds of misconception in statistics, see &lt;a href=&quot;http://xkcd.com/552/&quot; rel=&quot;nofollow&quot;&gt;http://xkcd.com/552/&lt;/a&gt; for correlation and causation.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The problem with newspaper articles is rarely the fact that they treat a rare phenomenon.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Simpsons's paradox comes to mind as example that statistics can rarely be used without analysis of the causes.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" LastActivityDate="2011-05-11T17:57:05.260" CommentCount="1" />
  <row Id="10683" PostTypeId="2" ParentId="10680" CreationDate="2011-05-11T18:09:30.407" Score="3" ViewCount="" Body="&lt;p&gt;There is an interesting article by Mary Gray on misuse of statistics in court cases and things like that...  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Gray, Mary W.; Statistics and the Law. Math. Mag. 56 (1983), no. 2, 67–81&lt;/p&gt;&#xA;" LastActivityDate="2011-05-11T18:09:30.407" />
  <row Id="10684" PostTypeId="2" ParentId="10680" CreationDate="2011-05-11T20:27:27.100" Score="0" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Hypothesis:&lt;/strong&gt; $A$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;(Textbook) Result:&lt;/strong&gt; Do no reject $A$ ($\sigma = c$)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Your Statement:&lt;/strong&gt; $A$ holds with probability $\sigma$!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Correct would be: In this case, you know nothing. If you want to &quot;prove&quot; $A$, your hypothesis has to be $\neg A$; reject it with $\sigma$ to get the desired statement.&lt;/p&gt;&#xA;" LastActivityDate="2011-05-11T20:27:27.100" />
  <row Id="10685" PostTypeId="2" ParentId="10669" CreationDate="2011-05-11T22:48:58.137" Score="2" ViewCount="" Body="&lt;p&gt;I see that you've accepted answers to just 3 of 9 questions...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are you using the type of Decision Tree known as CHAID?  If so, you will obtain an indication of one main effect and then any number of so-called interaction effects.  You can try these effects in a regression, ANOVA, or general linear model.  You build in the main effect.  You build in the main effect for each variable involved in an interaction.  And you build in the interactions.  But before you do this, the interaction effects all need to be pre-tested, as I explained in a comment to&#xA;&lt;a href=&quot;http://stats.stackexchange.com/questions/7815/what-skills-are-required-to-perform-large-scale-statistical-analyses&quot;&gt;this post&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2669" LastActivityDate="2011-05-11T22:48:58.137" CommentCount="2" />
  <row Id="10686" PostTypeId="2" ParentId="10643" CreationDate="2011-05-11T23:04:01.823" Score="1" ViewCount="" Body="&lt;p&gt;My first instinct is to say that it would be silly to make such a determination absent any knowledge of the topic.  &quot;Too large&quot; for what, or for whom?  But perhaps what you're looking for is really a test for outliers in the distribution--not that you're likely to find any in the one you've shown.  Check out Dixon's Test for Outliers (sometimes called the Q-Test).  I'm not thrilled with what Wikipedia provides, so you might want to check around further than that.  Sorry I don't have a good web reference; I use the guidelines in the book &lt;em&gt;100 Statistical Tests&lt;/em&gt; by Gopal Kanji.&lt;/p&gt;&#xA;" OwnerUserId="2669" LastActivityDate="2011-05-11T23:04:01.823" CommentCount="2" />
  <row Id="10687" PostTypeId="1" CreationDate="2011-05-11T23:05:00.207" Score="7" ViewCount="198" Body="&lt;p&gt;I know correlation does not imply causation but instead the strength and direction of the relationship. Does simple linear regression imply causation? Or is an inferential (t-test, etc.) statistical test required for that?&lt;/p&gt;&#xA;" OwnerUserId="4572" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-12T06:44:00.000" LastActivityDate="2011-05-12T18:16:46.903" Title="Does simple linear regression imply causation?" Tags="&lt;regression&gt;&lt;correlation&gt;&lt;causal-inference&gt;" AnswerCount="4" CommentCount="3" FavoriteCount="1" />
  <row Id="10688" PostTypeId="2" ParentId="2770" CreationDate="2011-05-11T23:19:08.927" Score="0" ViewCount="" Body="&lt;p&gt;I was using &quot;Fundamentals of Clinical Trials&quot; when I was in PhD program.&lt;/p&gt;&#xA;" OwnerUserId="4559" LastActivityDate="2011-05-11T23:19:08.927" />
  <row Id="10689" PostTypeId="2" ParentId="10687" CreationDate="2011-05-11T23:44:00.907" Score="10" ViewCount="" Body="&lt;p&gt;The quick answer is, no.   You can easily come up with non-related data that when regressed, will pass all sorts of statistical tests.   Below is an old picture from Wikipedia (which, for some reason has recently been removed) that has been used to illustrate data-driven &quot;causality&quot;.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We need more pirates to cool the planet?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.stack.imgur.com/bDhy4.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For time series, there is a term called &quot;Granger Causality&quot; that has a very specific meaning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Granger_causality&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Granger_causality&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other than that, &quot;causality&quot; is in the eye of the beholder.&lt;/p&gt;&#xA;" OwnerUserId="2775" LastEditorUserId="2775" LastEditorDisplayName="" LastEditDate="2011-05-11T23:57:04.897" LastActivityDate="2011-05-11T23:57:04.897" CommentCount="5" />
  <row Id="10690" PostTypeId="2" ParentId="10676" CreationDate="2011-05-12T00:50:24.680" Score="1" ViewCount="" Body="&lt;p&gt;The &lt;a href=&quot;http://www.cs.waikato.ac.nz/ml/weka/&quot; rel=&quot;nofollow&quot;&gt;Weka&lt;/a&gt; &lt;a href=&quot;http://weka.sourceforge.net/doc/weka/attributeSelection/SVMAttributeEval.html&quot; rel=&quot;nofollow&quot;&gt;SVMAttributeEval&lt;/a&gt; package allows you to do feature selection using SVM.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It should be pretty easy to dump your R data frame to a csv file, import that into Weka, do the feature selection, and then pull it back into R.&lt;/p&gt;&#xA;" OwnerUserId="1876" LastActivityDate="2011-05-12T00:50:24.680" />
  <row Id="10691" PostTypeId="2" ParentId="10680" CreationDate="2011-05-12T01:03:26.873" Score="2" ViewCount="" Body="&lt;p&gt;When it comes to logic and common sense, be careful, those two are rare.   With certain &quot;discussions&quot; you might recognize something......the point of the argument is the argument.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.wired.com/wiredscience/2011/05/the-sad-reason-we-reason/&quot; rel=&quot;nofollow&quot;&gt;http://www.wired.com/wiredscience/2011/05/the-sad-reason-we-reason/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2775" LastEditorUserId="2775" LastEditorDisplayName="" LastEditDate="2011-05-12T01:09:31.147" LastActivityDate="2011-05-12T01:09:31.147" CommentCount="1" />
  <row Id="10692" PostTypeId="2" ParentId="10664" CreationDate="2011-05-12T01:55:31.253" Score="3" ViewCount="" Body="&lt;p&gt;This overlaps largely with what @rolando2 has already said.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;General ideas:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Regardless of the software you use, here are some things that you could do:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Compare means on each item across age groups&lt;/li&gt;&#xA;&lt;li&gt;Crosstabulate age group by item response (for each item); you are probably most interested in the proportion of each item response within an age group.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;Further considerations:&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Do the four items form a scale? If so, you may want to create a composite based on the items, and compare means for each age group&lt;/li&gt;&#xA;&lt;li&gt;Do you wish to run statistical tests of significant group differences? if so, you'll most likely be interested in ANOVAs and follow-up tests looking at the effect of group on item means (but be aware of debates about the appropriateness of using means on ordinal items).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-12T10:58:56.187" LastActivityDate="2011-05-12T10:58:56.187" />
  <row Id="10693" PostTypeId="1" CreationDate="2011-05-12T02:22:37.467" Score="1" ViewCount="60" Body="&lt;h3&gt;Scenario:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;An industrial/organizational psychologist is interested in determining whether adding 15-minute breaks increases worker productivity. &#xA;She selects a sample $n$ and measures productivity (on a continuous scale) before and after introducing the intervention.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The researcher runs a repeated measures t-test.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Question&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;How can I work out whether the intervention is effective?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="4573" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-12T03:58:41.083" LastActivityDate="2011-05-13T04:43:39.707" Title="Determining statistical significance of a repeated measures t-test" Tags="&lt;homework&gt;&lt;t-test&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="10695" PostTypeId="1" CreationDate="2011-05-12T05:15:57.730" Score="0" ViewCount="34" Body="&lt;p&gt;Does anybody have a code for calculating power spectrum and/or co-spectrum of a signal applying max entropy method in R or Mathematica environment?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My problem is that when I try to smooth the calculated spectrum in the &lt;strong&gt;time domain&lt;/strong&gt; using Mathematica, the final shape of spectrum looks distorted. On the other hand, smoothing in the &lt;strong&gt;frequency domain&lt;/strong&gt; doesn't remove some undesirable noises particularly in low frequencies.&#xA;I use &lt;em&gt;Daniell Window&lt;/em&gt; and &lt;em&gt;Batlett Window&lt;/em&gt; respectively for time and frequency domain filtering.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Nevertheless, shape of spectra obtained from the two smoothing approaches are different!    &lt;/p&gt;&#xA;" OwnerUserId="4286" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-12T06:32:34.173" LastActivityDate="2011-05-12T06:32:34.173" Title="How to smooth power spectrum applying max. entropy method using R or Mathematica?" Tags="&lt;r&gt;&lt;time-series&gt;&lt;mathematica&gt;" CommentCount="1" />
  <row Id="10696" PostTypeId="2" ParentId="10687" CreationDate="2011-05-12T05:15:58.993" Score="4" ViewCount="" Body="&lt;p&gt;There is nothing explicit in the mathematics of regression that state causal relationships, and hence one need not explicitly interpret the slope (strength and direction) nor the p-values (i.e. the probability a relation as strong as or stronger would have been observed if the relationship were zero in the population) in a causal manner.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That being said, I would say regression does have a much stronger connotation that one is estimating an explicit directional relationship than does estimating the correlation between two variables. Assuming by correlation you mean &lt;a href=&quot;http://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient&quot; rel=&quot;nofollow&quot;&gt;Pearson's r&lt;/a&gt;, it typically does not have an explicit causal interpretation as the metric is symmetrical (i.e. you can switch which variable is X and which is Y and you will still have the same measure). Also the colloquialism &quot;Correlation does not imply causation&quot; I would suspect is so well known that stating two variables are correlated the assumption is one is not making a causal statement.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Estimated effects in &lt;a href=&quot;http://en.wikipedia.org/wiki/Regression_analysis&quot; rel=&quot;nofollow&quot;&gt;regression&lt;/a&gt; analysis are not symetrical though, and so by choosing what variable is on the right hand side versus the left hand side one is making an implicit statement unlike that of the correlation. I suspect one intends to make some causal statement in the vast majority of circumstances in which regression is used (inference vs prediction aside). Even in cases of simply stating correlations I suspect people frequently have some implied goals of causal inference in mind. Given some constraints are met &lt;a href=&quot;http://stats.stackexchange.com/questions/534/under-what-conditions-does-correlation-imply-causation&quot;&gt;correlation can imply causation&lt;/a&gt;!&lt;/p&gt;&#xA;" OwnerUserId="1036" LastActivityDate="2011-05-12T05:15:58.993" />
  <row Id="10697" PostTypeId="1" CreationDate="2011-05-12T05:31:19.257" Score="3" ViewCount="49" Body="&lt;p&gt;I'm studying R package dlm. So far it seems very powerful and flexible package, with nice programming interfaces and good documentation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've been able to successfully use dlmMLE and dlmModARMA to estimate the parameters of AR(1) process:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;u &amp;lt;- arima.sim(list(ar = 0.3), 100)&#xA;fit &amp;lt;- dlmMLE(u, parm = c(0.5, sd(u)),&#xA;              build = function(x)&#xA;                dlmModARMA(ar = x[1], sigma2 = x[2]^2))&#xA;fit$par&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now I'm trying to use similar code to estimate the parameters of simple linear regression model:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;r &amp;lt;- rnorm(100)&#xA;u &amp;lt;- -1*r + 0.5*rnorm(100)&#xA;fit &amp;lt;- dlmMLE(u, parm = c(0, 1),&#xA;              build = function(x)&#xA;                dlmModReg(x[1]*r, FALSE, dV = x[2]^2))&#xA;fit$par&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I expect fit$par to be close to c(-1, 0.5), but I keep getting something like&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[1] -0.0002118851  0.4884367070&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The coefficient -1 is not estimated correctly. However, the strange thing is that the variance of the noise is returned correctly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand that max-likelihood estimation might fail given bad initial values, but I observed that the likelihood function returned by dlmLL is very flat in the first coordinate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I wonder: can such model be estimated at all using dlm? I believe the model is &quot;non-singular&quot;, however I'm not sure how the likelihood function is calculated inside the dlm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any hint greatly appreciated.&lt;/p&gt;&#xA;" OwnerUserId="4575" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-12T09:22:09.627" LastActivityDate="2011-05-13T15:26:56.350" Title="Maximum likelihood estimation of dlmModReg" Tags="&lt;r&gt;&lt;regression&gt;&lt;maximum-likelihood&gt;&lt;dlm&gt;" AnswerCount="3" FavoriteCount="1" />
  <row Id="10698" PostTypeId="1" CreationDate="2011-05-12T07:45:57.930" Score="0" ViewCount="8" Body="&lt;p&gt;Dear everyone,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The my program below (which is in two parts) works  if I run them separately – that is, if I paste the &lt;/p&gt;&#xA;&#xA;&lt;p&gt;first part into the R Console, run it and then paste the second  and run it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, that is not how I want it. I want to run the whole program at once. If I do that it gives me &lt;/p&gt;&#xA;&#xA;&lt;p&gt;errors. This is because I use and entry “n” in the first part in the second section of my code.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That is, I use “n” to take the first “n rows” of a data frame called “bygrace” and put it in “copybygrace”.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The idea is that, having created the data frame “bygrace”&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want a user of my program to be able to take any number of rows, “n”, he wants.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have tried to indentify the problem but I have not been able find the root cause.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would be more than glad, if someone could come to my aid.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks in advance to all those who help me!(PLEASE SEE BELOW)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Isaac Owusu&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#FIRST PART OF THE PROGRAM&#xA;n &amp;lt;- -2&#xA;acr &amp;lt;- -2&#xA;&#xA;while((n&amp;lt;=0) | (acr&amp;lt;=0) | (acr&amp;gt;=1)) {&#xA; print(&quot;enter a  positive integer and the average cancellation rate between 0 and 1 you want&quot;)&#xA; try(n&amp;lt;-scan(what=integer(),nmax=1), silent=TRUE);&#xA; try(acr&amp;lt;-scan(what=double(),nmax=1), silent=TRUE)&#xA;}&#xA;&#xA;#SECOND PART OF THE PROGRAM&#xA;bygrace&amp;lt;-read.table(&quot;C:\\MyRfolder\\bygrace.txt&quot;,header=FALSE)&#xA;r&amp;lt;-nrow(bygrace)&#xA;c&amp;lt;-ncol(bygrace)&#xA;copybygrace&amp;lt;-array(bygrace,dim=c(r,c))&#xA;copybygrace&amp;lt;-bygrace[-((n+1):r),]&#xA;write.table(copybygrace,file=&quot;C:\\MyRfolder\\copybygrace.txt&quot;,sep=&quot;\t&quot;)&#xA;copybygrace&amp;lt;-read.table(&quot;C:\\MyRfolder\\copybygrace.txt&quot;,header=TRUE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="4340" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-12T07:52:23.213" LastActivityDate="2011-05-12T07:52:23.213" ClosedDate="2011-05-12T07:52:47.797" Title="my program is not running as I want it" Tags="&lt;r&gt;" />
  <row Id="10699" PostTypeId="2" ParentId="1164" CreationDate="2011-05-12T08:11:59.857" Score="1" ViewCount="" Body="&lt;p&gt;I'm not statistician, my experience in statistics is fairly limited, I just use robust statistics in computer vision/3d reconstruction/pose estimation. Here is my take on the problem from the user point of view:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, robust statistics used a lot in engineering and science without calling it &quot;robust statistics&quot;. A lot of people use it intuitively, coming to it in the process of adjusting specific method to real-world problem. For example iterative reweighted least squares and trimmed means/trimmed least square used commonly, that just the user don't know  they used robust statistics - they just make method workable for real, non-synthetic data.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second, both &quot;intuitive&quot; and conscious robust statistics practically always used in the case where results are verifiable, or where exists clearly visible error metrics. If result obtained with normal distribution are obviously non-valid or wrong, people start tinkering with weights, trimming,sampling,  read some paper and end up using robust estimators, whether they know term or not. On the other hand if end result of research just some graphics and diagrams, and there is no insensitive to verify results, or if normal statistic produce reults good enough - people just don't bother. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;And last, about usefulness of robust statistics as a theory - while theory itself is very interesting it's not often give any practical advantages. Most of robust estimators are fairly trivial and intuitive, often people reinventing them without any statistical knowledge. Theory, like breakdown point estimation, asymptotics, data depth, heteroskedacity etc allow deeper understanding of data, but in most cases it's just unnecessary. &#xA;One big exception is intersection of robust statistics and compressive sensing, which produce some new practical methods such as &quot;cross-and-bouquet&quot;&lt;/p&gt;&#xA;" OwnerUserId="4578" LastActivityDate="2011-05-12T08:11:59.857" />
  <row Id="10700" PostTypeId="1" CreationDate="2011-05-12T08:35:17.713" Score="1" ViewCount="118" Body="&lt;p&gt;Does anyone know of an existing implementation of ordinal logistic regression in Excel?&lt;/p&gt;&#xA;" OwnerUserId="333" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-12T09:21:36.930" LastActivityDate="2011-05-13T10:55:57.083" Title="Excel spreadsheet for ordinal logistic regression" Tags="&lt;logistic-regression&gt;&lt;excel&gt;" AnswerCount="3" CommentCount="2" />
  <row Id="10701" PostTypeId="2" ParentId="10676" CreationDate="2011-05-12T09:22:47.270" Score="1" ViewCount="" Body="&lt;p&gt;For Recursive Feature Extraction (SVM-RFE) the packages &lt;strong&gt;e1071&lt;/strong&gt; and &lt;strong&gt;Kernlab&lt;/strong&gt; doesn't implement it i think. For the Weka SVMAttributeEval package is for Java i think, but the question was for R as i saw.&#xA;The best way is trying to implement the SVM-RFE using &lt;strong&gt;e1071 and LIBSVM&lt;/strong&gt; library &#xA;I found a good parper relating that &lt;a href=&quot;http://www.uccor.edu.ar/paginas/seminarios/Software/SVM_RFE_R_implementation.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="4531" LastActivityDate="2011-05-12T09:22:47.270" />
  <row Id="10702" PostTypeId="1" CreationDate="2011-05-12T09:40:56.987" Score="3" ViewCount="171" Body="&lt;p&gt;I have a question regarding the interpretation of resulting p-values of a two sample Kolmogorov Smirnov test. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basis of my analysis is to try to identify groups that show a difference in their distribution difference compared to totality. I used a two sample Kologorov Smirnov Test in R to do so.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sample sizes:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Full = 2409&#xA;Group_1 = 25&#xA;Group_2 = 26&#xA;Group_3 = 33&#xA;Group_4 = 43&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Dataset plots:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.stack.imgur.com/rQqU0.png&quot; alt=&quot;Boxplots of dataset distributions&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Other&quot; contains a collection of groups containing less than 20 datapoints.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The resulting p-values I get when I compare each &quot;Group&quot; against &quot;Full - Group&quot; are the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Group 1: 2.6155e-002&#xA;Group 2: 2.1126e-003&#xA;Group 3: 7.2113e-002&#xA;Group 4: 7.3466e-003&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can I interpret these results - especially with regards to the low number of datapoints per group as well as the difference in sample size for Full (N=2409) and Groups (N=25-43)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is the choice of a KS test good or might another test be more appropriate in this case?&lt;/p&gt;&#xA;" OwnerUserId="4579" LastEditorUserId="4579" LastEditorDisplayName="" LastEditDate="2011-05-13T14:09:30.737" LastActivityDate="2011-05-13T14:09:30.737" Title="Two sample Kolmogorov-Smirnov test and p-value interpretation" Tags="&lt;sample-size&gt;&lt;p-value&gt;&lt;kolmogorov-smirnov&gt;" AnswerCount="1" />
  <row Id="10703" PostTypeId="1" CreationDate="2011-05-12T10:08:34.017" Score="4" ViewCount="48" Body="&lt;p&gt;I've implemented QR factorization based on Householder reflections (for the purposes of computing the OLS fit). Mathematically, the $R$ matrix is upper triangular. However, due to floating-point issues I typically end up with small non-zero entries below the diagonal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What should I do with them - leave alone or set to zero, and why?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, the following input:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[[ 1  1 10]&#xA; [ 3  3  5]&#xA; [ 2  4 -6]&#xA; [ 1 10  8]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;produces the following $R$ matrix:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[[ -3.87298335e+00  -7.22956891e+00  -5.42217668e+00]&#xA; [ -8.88178420e-16   8.58681159e+00   4.86793026e+00]&#xA; [ -4.44089210e-16  -3.33066907e-16   1.31111882e+01]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Note the non-zeros below the main diagonal. For comparison, both &lt;code&gt;numpy&lt;/code&gt; and &lt;code&gt;R&lt;/code&gt; return exact zeros, so I am wondering if there are reasons for me to do the same.&lt;/p&gt;&#xA;" OwnerUserId="439" LastActivityDate="2011-05-12T11:08:02.840" Title="QR factorization: floating-point issues" Tags="&lt;matrix-decomposition&gt;" AnswerCount="1" CommentCount="8" />
  <row Id="10704" PostTypeId="2" ParentId="9739" CreationDate="2011-05-12T10:29:20.660" Score="2" ViewCount="" Body="&lt;p&gt;There is different approach for scalable clustering, divide and conquer approach, parallel clustering and incremental one. This is for general approach after you can use normal clustering methods.&#xA;There a good method of clustering i really appreciate is DBSCAN (Density-Based Spatial Clustering of Applications with Noise) it is one of the most used clustering algorithm.&lt;/p&gt;&#xA;" OwnerUserId="4531" LastActivityDate="2011-05-12T10:29:20.660" CommentCount="4" />
  <row Id="10705" PostTypeId="2" ParentId="10703" CreationDate="2011-05-12T11:08:02.840" Score="3" ViewCount="" Body="&lt;p&gt;It's safe to ignore those tiny entries, as long as they are less than some quantity like &quot;norm of the matrix times machine epsilon&quot;. FWIW, if you'll be doing backsubstitution with the triangular matrix you now have, the routine is not supposed to access those subdiagonal entries anyway.&lt;/p&gt;&#xA;" OwnerUserId="830" LastActivityDate="2011-05-12T11:08:02.840" CommentCount="2" />
  <row Id="10706" PostTypeId="2" ParentId="10655" CreationDate="2011-05-12T11:11:08.707" Score="3" ViewCount="" Body="&lt;p&gt;The dataset would have to be enormous for the empirical approaches to have sufficient precision, and it doesn't help very much to look at percentiles of the marginal distribution of height.  I suggest quantile regression, allowing age to be flexibly modeled (e.g., using restricted cubic splines).  Here is an example using R.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(rms)  # loads quantreg, Hmisc, SparseM packages too&#xA;dd &amp;lt;- datadist(mydata); options(datadist='dd')&#xA;f &amp;lt;- Rq(ht ~ rcs(age,5), tau=.25, data=mydata) # model 25th percentile&#xA;f&#xA;plot(Predict(f))  # shows confidence bands &#xA;nomogram(f) # make a nomogram to predict the quantile manually&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="4253" LastEditorUserId="4253" LastEditorDisplayName="" LastEditDate="2011-05-12T21:10:18.593" LastActivityDate="2011-05-12T21:10:18.593" CommentCount="2" />
  <row Id="10707" PostTypeId="2" ParentId="10697" CreationDate="2011-05-12T11:12:51.557" Score="2" ViewCount="" Body="&lt;p&gt;I think your setup is not correct. Try this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(1234)&#xA;r &amp;lt;- rnorm(100)&#xA;X &amp;lt;- r&#xA;u &amp;lt;- -1*X + 0.5*rnorm(100)&#xA;MyModel &amp;lt;- function(x)  dlmModReg(X, FALSE, dV = x[1]^2)&#xA;fit &amp;lt;- dlmMLE(u, parm = c(0.3), build = MyModel)&#xA;mod &amp;lt;- MyModel(fit$par)&#xA;dlmFilter(u,mod)$a&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You recover the estimate of the observation variance from the only element of &#xA;fit$par:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; fit&#xA;$par&#xA;[1] 0.4431803&#xA;&#xA;$value&#xA;[1] -20.69313&#xA;&#xA;$counts&#xA;function gradient &#xA;      17       17 &#xA;&#xA;$convergence&#xA;[1] 0&#xA;&#xA;$message&#xA;[1] &quot;CONVERGENCE: REL_REDUCTION_OF_F &amp;lt;= FACTR*EPSMCH&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;while your estimate of the coefficient (should be around -1 in your case) can be obtained as the last element of &lt;code&gt;dlmFilter(u,mod)$a&lt;/code&gt;, which gives the values&#xA;of the state as new observations are processed:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    &amp;gt; dlmFilter(u,mod)$m&#xA;  [1]  0.0000000 -1.1486921 -1.2123431 -1.1172783 -1.1231454 -1.1170222&#xA;  [7] -1.0974931 -1.1377114 -1.0378758 -1.0927136 -1.0955372 -1.0120210&#xA; [13] -0.9874791 -1.0036429 -1.0765513 -1.0678725 -1.0795124 -1.1568597&#xA; [19] -1.2044821 -1.2056687 -1.2102896 -1.2938958 -1.2922945 -1.2670604&#xA; [25] -1.1789594 -1.1570172 -1.1601590 -1.1417200 -1.1585501 -1.1608675&#xA; [31] -1.1616278 -1.1744861 -1.1717561 -1.1715025 -1.1568086 -1.1451311&#xA; [37] -1.1520867 -1.1379211 -1.1270897 -1.1048035 -1.1015793 -1.1054597&#xA; [43] -1.0621750 -1.0621218 -1.0696813 -1.0807651 -1.0816893 -1.0647963&#xA; [49] -1.0643440 -1.0667282 -1.0626404 -1.0623697 -1.0586265 -1.0571205&#xA; [55] -1.0569135 -1.0579224 -1.0607623 -1.0582257 -1.0495232 -1.0494288&#xA; [61] -1.0539632 -1.0555427 -1.0553468 -1.0491239 -1.0488604 -1.0491036&#xA; [67] -1.0510551 -1.0576294 -1.0611296 -1.0628612 -1.0626451 -1.0573650&#xA; [73] -1.0629577 -1.0647724 -1.0658052 -1.0823839 -1.0753808 -1.0747229&#xA; [79] -1.0747762 -1.0615243 -1.0630352 -1.0697431 -1.0666448 -1.0617227&#xA; [85] -1.0585460 -1.0583981 -1.0563544 -1.0567715 -1.0544349 -1.0573228&#xA; [91] -1.0588404 -1.0639155 -1.0625845 -1.0578004 -1.0571034 -1.0602645&#xA; [97] -1.0604838 -1.0586019 -1.0580891 -1.0587096 -1.0577559  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="892" LastEditorUserId="892" LastEditorDisplayName="" LastEditDate="2011-05-12T11:38:03.097" LastActivityDate="2011-05-12T11:38:03.097" CommentCount="1" />
  <row Id="10708" PostTypeId="2" ParentId="10700" CreationDate="2011-05-12T11:13:03.210" Score="3" ViewCount="" Body="&lt;p&gt;It's difficult to recommend Excel (which has shown itself to be unreliable for simpler problems than the one posed) when R has well worked-out packages for this.&lt;/p&gt;&#xA;" OwnerUserId="4253" LastActivityDate="2011-05-12T11:13:03.210" CommentCount="3" />
  <row Id="10709" PostTypeId="2" ParentId="10640" CreationDate="2011-05-12T11:45:28.677" Score="3" ViewCount="" Body="&lt;p&gt;Neither one of these designs contradicts each other, they are generated in different ways. A $2^{5-2}_{III}$ is not unique. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using a design with so many factors and so few runs, necessiates the fact that main effects and two factor interactions will be confounded, the question is how. Since you want to do an experiment with $8 = 2^{3}$ observations, the full factorial that will generate this design will be based on 3 factors; call them A, B, and C.&#xA;In the 3 factor design, you will have interactions&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;AB, AC, BC&lt;/li&gt;&#xA;&lt;li&gt;ABC&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;To expand this design to include two more factors, each of the new factors ;call them D and E; must be confounded with one of the above listed interactions. Two ways to do that without confounding two main effects is to either have&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;D = AB and E = AC which means DE = BC&lt;/li&gt;&#xA;&lt;li&gt;D = BC and E = ABC which means DE = A&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You can try other combinations, and see the results of your confounding structure. &lt;/p&gt;&#xA;" OwnerUserId="3805" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-12T14:51:54.927" LastActivityDate="2011-05-12T14:51:54.927" CommentCount="3" />
  <row Id="10710" PostTypeId="2" ParentId="10687" CreationDate="2011-05-12T11:56:45.823" Score="3" ViewCount="" Body="&lt;p&gt;Neither correlation nor regression can indicate causation (as is illustrated by @bill_080's answer) but as @Andy W indicates regression is often based on an explicitly fixed (i.e., independent) variable and an explicit (i.e., random) dependent variable.  These designations are not appropriate in correlation analysis.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To quote Sokal and Rohlf, 1969, p. 496&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&quot;In regression we intend to describe the dependence of a variable &lt;em&gt;Y&lt;/em&gt; on an independent variable &lt;em&gt;X&lt;/em&gt;... to lend support to hypotheses regarding the possible causation of changes in &lt;em&gt;Y&lt;/em&gt; by changes in &lt;em&gt;X&lt;/em&gt;...&quot;&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;&quot;In correlation, by contrast, we are concerned largely whether two variables are interdependent or &lt;em&gt;covary&lt;/em&gt; - that is, vary together.  We do not express one as a function of the other.&quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Sokal, R. R. and F. J. Rohlf, 1969. &lt;em&gt;Biometry.&lt;/em&gt; Freeman and Co.&lt;/p&gt;&#xA;" OwnerUserId="4048" LastActivityDate="2011-05-12T11:56:45.823" />
  <row Id="10711" PostTypeId="1" CreationDate="2011-05-12T13:23:45.673" Score="8" ViewCount="116" Body="&lt;p&gt;Stack Exchange, as we all know it, is a collection of Q&amp;amp;A sites with diversified topics. Assuming that each site is independent from each other, given the stats a user has, how to compute his &quot;well-roundedness&quot; as compared to the next guy? What is the statistical tool I should employ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To be honest, I don't quite know how to mathematically define the &quot;well-roundness&quot;, but it must have the following characteristics:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;All things being equal, the more rep a user has, the more well-rounded he is&lt;/li&gt;&#xA;&lt;li&gt;All things being equal, the more sites a user participates in, the more well-rounded he is.&lt;/li&gt;&#xA;&lt;li&gt;Whether answer or question doesn't affect the well-roundness&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="175" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-12T16:44:32.713" LastActivityDate="2011-05-15T19:20:38.847" Title="How to measure the &quot;well-roundedness&quot; of SE contributors?" Tags="&lt;ranking&gt;" AnswerCount="4" CommentCount="3" />
  <row Id="10712" PostTypeId="1" AcceptedAnswerId="10715" CreationDate="2011-05-12T14:11:20.500" Score="15" ViewCount="220" Body="&lt;p&gt;Hi, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm just reading the book &quot;R in a Nutshell&quot;.&#xA;And it seems as if I skipped the part where the &quot;.&quot; as in &quot;sample.formula&quot; was explained. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; sample.formula &amp;lt;- as.formula(y~x1+x2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is sample an object with a field formula as in other languages? And if so, how can I find out, what other fields/functions this object has? (Type declaration) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT:&#xA;I just found another confusing use of the &quot;.&quot;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; svm(formula = is_spam~., data = spambase.training)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(the dot between ~., )&lt;/p&gt;&#xA;" OwnerUserId="3541" LastEditorUserId="3541" LastEditorDisplayName="" LastEditDate="2011-05-12T19:41:19.033" LastActivityDate="2011-05-12T21:04:50.773" Title="What is the meaning of the &quot;.&quot; (dot) in R?" Tags="&lt;r&gt;" AnswerCount="3" CommentCount="4" FavoriteCount="1" />
  <row Id="10713" PostTypeId="2" ParentId="10711" CreationDate="2011-05-12T14:16:43.510" Score="3" ViewCount="" Body="&lt;p&gt;If you define 'well-roundedness' as 'contributing to many different Stack Exchange Sites,' I would compute some metric of contribution per site.  You could use total posts, or average posts per day, or perhaps reputation.  Then look at the distribution of this metric across all sites, and compute its skewness in some way that makes sense.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words, a 'well-rounded' person would be one who contributes to many different sites, while a 'not well-rounded' person would be one who primarily contributes to one site.  You could further improve this by scaling your metric with a user's total across all sites.  i.e. someone who's contributed a lot to many different sites should be considered more well-rounded than someone who's contributed nothing to any of the sites.  A person who's never used SE isn't very well rounded!&lt;/p&gt;&#xA;" OwnerUserId="2817" LastActivityDate="2011-05-12T14:16:43.510" />
  <row Id="10714" PostTypeId="2" ParentId="10712" CreationDate="2011-05-12T14:19:47.597" Score="4" ViewCount="" Body="&lt;p&gt;There are some exceptions (S3 method dispatch), but generally it is simply used as legibility aid, and as such has no special meaning.&lt;/p&gt;&#xA;" OwnerUserId="4257" LastActivityDate="2011-05-12T14:19:47.597" CommentCount="2" />
  <row Id="10715" PostTypeId="2" ParentId="10712" CreationDate="2011-05-12T14:27:47.707" Score="13" ViewCount="" Body="&lt;p&gt;The dot can be used as in normal name. It has however additional special interpretation. Suppose we have an object with specific class:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; a &amp;lt;- list(b=1)&#xA; class(a) &amp;lt;- &quot;myclass&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now declare &lt;code&gt;myfunction&lt;/code&gt; as standard generic in the following way:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; myfunction &amp;lt;- function(x,...) UseMethod(&quot;myfunction&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now declare the function &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; myfunction.myclass &amp;lt;- function(x,...) x$b+1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then the dot has special meaning. For all objects with class &lt;code&gt;myclass&lt;/code&gt; calling &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; myfunction(a)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;will actualy call function &lt;code&gt;myfunction.myclass&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; &amp;gt; myfunction(a)&#xA;  [1] 2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is used widely in R, the most apropriate example is function &lt;code&gt;summary&lt;/code&gt;. Each class has its own &lt;code&gt;summary&lt;/code&gt; function, so when you fit some model for example (which usually returns object with specific class), you need to invoke &lt;code&gt;summary&lt;/code&gt; and it will call appropriate summary function for that specific model.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-12T14:35:03.243" LastActivityDate="2011-05-12T14:35:03.243" />
  <row Id="10716" PostTypeId="2" ParentId="10711" CreationDate="2011-05-12T14:27:59.733" Score="4" ViewCount="" Body="&lt;p&gt;EXAMPLE: say there are three sites, and we want to compare the well-roundedness of the Users A, B, C.  We write the reputations of the users across the three sites in vector form:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;User A: [23, 23, 0]&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;User B: [15, 15, 0]&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;User C: [10, 10, 10]&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;We would consider A more well-rounded than B (their reputations are both spread out evenly across two sites, but A has more total reputation). Also, we would consider C more well-rounded than B (they have the same total reputation, but C has an even spread across more sites.)  It is undecided whether A should be considered more well-rounded than C, or vice-versa.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let $x_A$, $x_B$, $x_C$ be the above reputation vectors respectively.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We want to measure the &quot;well-roundedness&quot; of a user by a function of their reputation vector $f(x)$.  By the above, we would want our function $f$ to satisfy $f(x_A) &amp;gt; f(x_B)$, and $f(x_C) &amp;gt; f(x_B)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any $f(x)$ that is &lt;em&gt;concave and increasing&lt;/em&gt; will do the trick.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Two common examples of convex functions are the &lt;em&gt;'fractional norm'&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;f([x_1,...,x_m]) = \sum_i x_i^p &#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;for $0 &amp;lt; p &amp;lt; 1$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Taking $p = 1/2$, we calculate&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$f(x_A) = 2\sqrt{23} \approx 9.6$$&#xA;$$f(x_B) = 2\sqrt{15} \approx 7.7$$&#xA;$$f(x_C) = 3\sqrt{10} \approx 9.5$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;According to the $1/2$-norm, User A would be considered the most well-rounded of the three, by a narrow margin over User C.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another choice for $f$ is the (scaled) &lt;em&gt;Shannon entropy&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;f([x_1,...,x_m]) = -\sum_i x_i \log(x_i/c).&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $c = \sum_i x_i$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we take $f$ to be the scaled Shannon entropy, then we calculate&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$f(x_A) = 46 \log(2) \approx 31.9$$&#xA;$$f(x_B) = 30 \log(2) \approx 20.8$$&#xA;$$f(x_C) = 30 \log(3) \approx 33.0$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Measured according the scaled Shannon entropy, then, we would say C is the most well-rounded of the three, and A the second most well-rounded.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: I originally said the function $f(x)$ had to be convex; the opposite is true.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT2: Added an example in light of whuber's comment.&lt;/p&gt;&#xA;" OwnerUserId="3567" LastEditorUserId="3567" LastEditorDisplayName="" LastEditDate="2011-05-15T19:20:38.847" LastActivityDate="2011-05-15T19:20:38.847" CommentCount="2" />
  <row Id="10717" PostTypeId="2" ParentId="10702" CreationDate="2011-05-12T15:21:02.490" Score="2" ViewCount="" Body="&lt;p&gt;If you are using the traditional 0.05 alpha level cutoff then all but group 3 are significantly different from your full group.  It is a little easier to see this if the p-values are not in scientific notation ( you can use options(scipen=5) in R to make this less likely).  Also group 1 becomes non-significant for some adjustments for multiple tests.  You should consider whether that adjustment applies in your case or not.  Also note that the groups that are not significant could be different, just low power.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But that just means that any differences, however small, are not easily explained by chance.  It could be that your groups are close enough for practical purposes.  It is usualy more meaningful to plot the data to see how different the distributions are.  You could use the qqplot function as one approach.  The vis.test function in the TeachingDemos package for R gives another approach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One possible hitch is if your groups are part of the &quot;Full&quot; data set as well, then you don't have the independence assumed (but given the sample sizes, I am not sure how much this would affect things).  You could address this by taking random samples from the full data set and computing the KS-distance for each (ignore the p-value), then compare where your actual data falls relative to the random samples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most of this comes down to what question you really want answered, many of the exact distributional tests answer a different question than the researcher is really interested in.&lt;/p&gt;&#xA;" OwnerUserId="4505" LastActivityDate="2011-05-12T15:21:02.490" CommentCount="2" />
  <row Id="10718" PostTypeId="1" CreationDate="2011-05-12T15:22:27.747" Score="0" ViewCount="21" Body="&lt;p&gt;Where can I find details of Steel's method for nonparametric multiple comparison with control on line ... ?&lt;/p&gt;&#xA;" OwnerUserId="3539" LastActivityDate="2011-05-12T15:22:27.747" Title="Steel's method for nonparametric multiple comparison with control" Tags="&lt;nonparametric&gt;" />
  <row Id="10719" PostTypeId="1" AcceptedAnswerId="10741" CreationDate="2011-05-12T15:54:06.523" Score="3" ViewCount="32" Body="&lt;p&gt;Is there a standard name for a multinomial choice model where the observations are in the form of binary questions such as &quot;do you prefer A to B&quot; and &quot;do you prefer B to D&quot;?  This seems like a common occurance, and the likelihood is easy enough to write out by hand, but I'm having trouble searching for references.&lt;/p&gt;&#xA;" OwnerUserId="493" LastActivityDate="2011-05-12T21:35:07.073" Title="Multinomial choice with binary observations" Tags="&lt;maximum-likelihood&gt;&lt;discrete-data&gt;" AnswerCount="1" />
  <row Id="10720" PostTypeId="2" ParentId="10539" CreationDate="2011-05-12T16:34:10.220" Score="1" ViewCount="" Body="&lt;p&gt;You don't state this explicitly, but from your description of the problem it seems likely that you're after a high-biased set of quantiles (e.g., 50th, 90th, 95th and 99th percentiles).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If that's the case, I've had a lot of success with the method described in &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.91.853&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;&quot;Effective Computation of Biased Quantiles over Data Streams&quot;&lt;/a&gt; by Cormode et al. It's a fast algorithm that requires little memory and that's easy to implement.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The method is based on an earlier algorithm by Greenwald and Khanna that maintains a small sample of the input stream along with upper and lower bounds on the rank of the values in the sample. It requires more space than a collection of few moments, but will be much better at describing the interesting tail region of the distribution accurately.&lt;/p&gt;&#xA;" OwnerUserId="439" LastEditorUserId="439" LastEditorDisplayName="" LastEditDate="2011-05-12T16:43:19.323" LastActivityDate="2011-05-12T16:43:19.323" CommentCount="1" />
  <row Id="10721" PostTypeId="2" ParentId="10711" CreationDate="2011-05-12T16:37:16.473" Score="6" ViewCount="" Body="&lt;p&gt;You need to account for similarity between the sites, as well.  Someone who participates on StackOverflow and &lt;a href=&quot;http://cooking.stackexchange.com/&quot;&gt;Seasoned Advice&lt;/a&gt; is more well-rounded than someone who participates on SO and CrossValidated, who is in turn (I would argue) more well-rounded than someone who participates in SO and &lt;a href=&quot;http://programmers.stackexchange.com&quot;&gt;Programmers&lt;/a&gt;.  There are undoubtedly many ways to do that, but you could check overlapping registration to just get a feel for it.&lt;/p&gt;&#xA;" OwnerUserId="71" LastActivityDate="2011-05-12T16:37:16.473" CommentCount="8" />
  <row Id="10722" PostTypeId="2" ParentId="10639" CreationDate="2011-05-12T16:49:57.787" Score="1" ViewCount="" Body="&lt;p&gt;My question was about the distributions, not the test, and I think I've figured out the answer: a t-distribution squared has an F(1,n) distribution, which is a Hotelling distribution (up to rescaling by a constant determined by the parameters). I believe one can say that an F(m,n) distribution is the same as a Wilks' $\Lambda(1,m,n)$ distribution, which is the kind of 'generalization' I was looking for. (I guess I'm now enough of an 'expert' to edit the wikipedia page ;)&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2011-05-12T16:49:57.787" />
  <row Id="10723" PostTypeId="1" CreationDate="2011-05-12T16:50:08.270" Score="2" ViewCount="58" Body="&lt;p&gt;I am running a dlog-dlog (difference of logarithm*) regression and I want to convert the coefficients into marginal effects. I know that it's different from a log-log regression, in which the coefficients directly give us the elasticities. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can we interpret the coefficients from a dlog-dlog regression? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;sub&gt;* For example dlog (Y)= a + b dlog(X)+ error term.&lt;/sub&gt;&lt;/p&gt;&#xA;" OwnerUserId="4586" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-12T21:16:41.350" LastActivityDate="2011-05-13T07:32:40.563" Title="Interpreting coefficients of a dlog-dlog regression" Tags="&lt;regression&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="10724" PostTypeId="2" ParentId="10680" CreationDate="2011-05-12T16:58:13.247" Score="2" ViewCount="" Body="&lt;p&gt;To make conclusions about a group based on the population the group must be representative of the population and independent.  Others have discussed this, so I will not dwell on this piece.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One other thing to consider is the non-intuitivness of probabilities.  Let's assume that we have a group of 10 people who are independent and representative of the population (random sample) and that we know that in the population 10% have a particular characteristic.  Therefore each of the 10 people has a 10% chance of having the characteristic.  The common assumption is that it is fairly certain that at least 1 will have the characteristic.  But that is a simple binomial problem, we can calculate the probability that none of the 10 have the characteristic, it is about 35% (converges to 1/e for bigger group/smaller probability) which is much higher than most people would guess.  There is also a 26% chance that 2 or more people have the characteristic.&lt;/p&gt;&#xA;" OwnerUserId="4505" LastActivityDate="2011-05-12T16:58:13.247" />
  <row Id="10725" PostTypeId="2" ParentId="1195" CreationDate="2011-05-12T17:12:20.963" Score="-1" ViewCount="" Body="&lt;p&gt;Generalized Pareto distribution (GPD) is a good model to accomplish this&lt;/p&gt;&#xA;" OwnerUserId="4497" LastActivityDate="2011-05-12T17:12:20.963" CommentCount="1" />
  <row Id="10726" PostTypeId="1" CreationDate="2011-05-12T17:23:54.350" Score="7" ViewCount="121" Body="&lt;p&gt;I thought heavy tail = fat tail, but some articles I read gave me a sense that they aren't.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of them says: heavy tail means the distribution have infinite jth moment for some integer j. Additionally all the dfs in the pot-domain of attraction of a Pareto df are heavy-tailed. &#xA;If the density has a high central peak and long tails, then the kurtosis is typically large. A df with kurtosis larger than 3 is fat-tailed or leptokurtic.&#xA;I still don't have a concrete distinction between these two (heavy tail vs. fat tail). Any thoughts or pointers to relevant articles would be appreciated.&lt;/p&gt;&#xA;" OwnerUserId="4497" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-12T21:12:05.827" LastActivityDate="2011-05-14T18:14:19.110" Title="Differences between heavy tail and fat tail distributions" Tags="&lt;distributions&gt;" AnswerCount="1" FavoriteCount="2" />
  <row Id="10727" PostTypeId="1" CreationDate="2011-05-12T17:35:16.817" Score="3" ViewCount="65" Body="&lt;p&gt;I want to get a confidence interval of a function of some parameters. for example, from the data I estimate parameters of Pareto. Now I want to get 95% CI for 90th quantile (it's a function of parameters of Pareto), so I would need standard error. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know delta method is one option. For simulation method, I am wondering if it is legitimate to simulate 1000 samples of size 50 from Pareto, calculate each of the 90th quantiles and take the standard deviation of the 1000 data. Is the standard deviation I get equivalent to standard error? Thanks for your help! &lt;/p&gt;&#xA;" OwnerUserId="4497" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-12T21:08:58.433" LastActivityDate="2011-05-12T21:08:58.433" Title="How to get standard error of a function (delta method vs. simulation)?" Tags="&lt;simulation&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="10728" PostTypeId="1" CreationDate="2011-05-12T17:56:52.957" Score="5" ViewCount="62" Body="&lt;p&gt;Given N sampled values, what does the &quot;p-th quantile of the sampled values&quot; mean?&lt;/p&gt;&#xA;" OwnerUserId="3026" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-12T23:31:27.247" LastActivityDate="2011-05-12T23:31:27.247" Title="Definition of quantile" Tags="&lt;sampling&gt;" AnswerCount="1" />
  <row Id="10729" PostTypeId="2" ParentId="10727" CreationDate="2011-05-12T18:04:07.217" Score="6" ViewCount="" Body="&lt;p&gt;Generating data from a given distribution, then calculating the part of interest then redoing this a bunch of times to get the interval is sometimes called a parametric bootstrap.  You might learn more by reading up on this topic.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why a sample of 50 each time?  is the 50 meaningful? if not, then bigger samples are probably better.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One thing that your above method does not take into account is any uncertainties that you have in the parameters of the pareto distribution itself.  You may be able to take this into account by doing a 2 stage bootstrap, fit the parameters on a bootstrap sample, then generate your new data from that set of parameters and find the percentile.  Then repeat the entire process many times (starting with the bootstrap sample again).&lt;/p&gt;&#xA;" OwnerUserId="4505" LastActivityDate="2011-05-12T18:04:07.217" CommentCount="2" />
  <row Id="10730" PostTypeId="2" ParentId="10687" CreationDate="2011-05-12T18:16:46.903" Score="2" ViewCount="" Body="&lt;p&gt;From a semantic perspective, an alternative goal is to build evidence for a good predictive model instead of proving causation. A simple procedure for building evidence for the predictive value of a regression model is to divide your data in 2 parts and fit your regression with one part of the data and with the other part of the data test how well it predicts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The notion of &lt;a href=&quot;http://en.wikipedia.org/wiki/Granger_causality&quot; rel=&quot;nofollow&quot;&gt;Granger causality&lt;/a&gt; is interesting.&lt;/p&gt;&#xA;" OwnerUserId="4329" LastActivityDate="2011-05-12T18:16:46.903" />
  <row Id="10731" PostTypeId="2" ParentId="9220" CreationDate="2011-05-12T18:48:43.983" Score="5" ViewCount="" Body="&lt;p&gt;The answer to this question can be found in the book &lt;em&gt;Quadratic forms in random variables&lt;/em&gt; by Mathai and Provost (1992, Marcel Dekker, Inc.). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As the comments clarify, you need to find the distribution of $Q = z_1^2 + z_2^2$ where &#xA;$z = a - b$ follows a bivariate normal distribution with mean $\mu$ and covariance matrix $\Sigma$. This is a quadratic form in the bivariate random variable $z$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Briefly, one nice general result for the $p$-dimensional case where $z \sim N_p(\mu, \Sigma)$ and&lt;br&gt;&#xA;$$Q = \sum_{j=1}^p z_j^2$$ &#xA;is that the moment generating function is &#xA;$$E(e^{tQ}) = e^{t \sum_{j=1}^p \frac{b_j^2 \lambda_j}{1-2t\lambda_j}}\prod_{j=1}^p (1-2t\lambda_j)^{-1/2}$$&#xA;where $\lambda_1, \ldots, \lambda_p$ are the eigenvalues of $\Sigma$ and $b$ is a linear function of $\mu$. See Theorem 3.2a.2 (page 42) in the book cited above (we assume here that $\Sigma$ is non-singular). Another useful representation is 3.1a.1 (page 29) &#xA;$$Q = \sum_{j=1}^p \lambda_j(u_j + b_j)^2$$&#xA;where $u_1, \ldots, u_p$ are i.i.d. $N(0, 1)$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The entire Chapter 4 in the book is devoted to the representation and computation of densities and distribution functions, which is not at all trivial. I am only superficially familiar with the book, but my impression is that all the general representations are in terms of infinite series expansions.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So in a certain way the answer to the question is, yes, the distribution of the squared euclidean distance between two bivariate normal vectors belongs to a known (and well studied) class of distributions parametrized by the four parameters $\lambda_1, \lambda_2 &amp;gt; 0$ and $b_1, b_2 \in \mathbb{R}$. However, I am pretty sure you won't find this distribution in your standard textbooks. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note, moreover, that $a$ and $b$ do not need to be independent. Joint normality is enough (which is automatic if they are independent and each normal), then the difference $a-b$ follows a normal distribution. &lt;/p&gt;&#xA;" OwnerUserId="4376" LastEditorUserId="4376" LastEditorDisplayName="" LastEditDate="2011-05-12T23:43:36.363" LastActivityDate="2011-05-12T23:43:36.363" />
  <row Id="10732" PostTypeId="2" ParentId="10728" CreationDate="2011-05-12T18:51:03.867" Score="6" ViewCount="" Body="&lt;p&gt;In theory (with $0 \lt p \lt 1$) it means the point a fraction $p$ up the cumulative distribution.  In practice there are various definitions used, particularly in statistical computing.  For example in R there are &lt;a href=&quot;http://stat.ethz.ch/R-manual/R-devel/library/stats/html/quantile.html&quot; rel=&quot;nofollow&quot;&gt;nine different definitions&lt;/a&gt;, the first three for a discrete interpretation and the rest for a variety of continuous interpolations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an example: if your sample is {400, 1, 1000, 40}, and you are looking for the 0.6 quantile (60th centile) then the different calculation methods give &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; for (t in 1:9) { x[t] &amp;lt;- quantile(c(400, 1, 1000, 40), probs=0.6, type = t ) }&#xA;&amp;gt; x&#xA;60%                                 &#xA;400 400  40 184 364 400 328 376 373 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My personal view is that the correct figure is that 400 is correct since &#xA;$$Pr(X&amp;lt;400) = 0.5 &amp;lt; 0.6 \text{ and } Pr(X&amp;gt;400) = 0.25 &amp;lt; 1-0.6.$$  This comes from treating the sample as the population, and if the empirical CDF is drawn it will be a sequence of steps. There are opposing arguments for interpolating so the empirical CDF is continuous, as being likely to be a better or more useful approximation to the population, and the method of interpolation will affect the result.    &lt;/p&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-05-12T18:51:03.867" CommentCount="1" />
  <row Id="10733" PostTypeId="2" ParentId="2580" CreationDate="2011-05-12T19:02:32.260" Score="0" ViewCount="" Body="&lt;p&gt;This is very interesting discussion and related to what I am trying to explore. It will be appreciated if author of this question share more details here. Thanks&lt;/p&gt;&#xA;" OwnerUserId="4098" LastActivityDate="2011-05-12T19:02:32.260" />
  <row Id="10734" PostTypeId="2" ParentId="5207" CreationDate="2011-05-12T19:03:23.007" Score="0" ViewCount="" Body="&lt;p&gt;I think the order is correct, but the labels assigned to p(x) and p(y|x) were wrong. The original problem states p(y|x) is log-normal and p(x) is Singh-Maddala. So, it's &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Generate an X from a Singh-Maddala, and&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;generate a Y from a log-normal having a mean which is a fraction of the generated X.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;No?&lt;/p&gt;&#xA;" OwnerUserId="3437" LastActivityDate="2011-05-12T19:03:23.007" />
  <row Id="10735" PostTypeId="2" ParentId="10697" CreationDate="2011-05-12T19:23:50.573" Score="0" ViewCount="" Body="&lt;p&gt;After reading help for dlmFilter, I could come up with the following code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;r &amp;lt;- rnorm(100)&#xA;u &amp;lt;- -1*r + 0.5*rnorm(100)&#xA;fit &amp;lt;- dlmMLE(u, parm = c(1, sd(u)),&#xA;              build = function(x)&#xA;                dlmModReg(r, FALSE, dV = x[2]^2,&#xA;                          m0 = x[1], C0 = matrix(0)))&#xA;fit$par&#xA;&#xA;[1] -1.1330088  0.4788357&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="4575" LastActivityDate="2011-05-12T19:23:50.573" CommentCount="1" />
  <row Id="10736" PostTypeId="2" ParentId="10726" CreationDate="2011-05-12T19:57:41.420" Score="9" ViewCount="" Body="&lt;p&gt;I would say that the usual definition in applied probability theory is that a &lt;strong&gt;right heavy tailed distribution&lt;/strong&gt; is one with infinite moment generating function on $(0, \infty)$, that is, $X$ has right heavy tail if &#xA;$$E(e^{tX}) = \infty, \quad t &amp;gt; 0.$$&#xA;This is in agreement with &lt;a href=&quot;http://en.wikipedia.org/wiki/Heavy-tailed_distribution&quot; rel=&quot;nofollow&quot;&gt;Wikipedia&lt;/a&gt;, which does mention other used definitions such as the one you have (some moment is infinite). There are also important subclasses such as the &lt;em&gt;long-tailed distributions&lt;/em&gt; and the &lt;em&gt;subexponential distributions&lt;/em&gt;. The standard example of a heavy-tailed distribution, according to the definition above, with all moments finite is the log-normal distribution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It may very well be that some authors use fat tailed and heavy tailed interchangeably, and others distinguish between fat tailed and heavy tailed. I would say that &lt;strong&gt;fat tailed&lt;/strong&gt; can be used more vaguely to indicate &lt;em&gt;fatter than normal&lt;/em&gt; tails and is sometimes used in the sense of &lt;em&gt;leptokurtic&lt;/em&gt; (positive kurtosis) as you indicate. One example of such a distribution, which is &lt;em&gt;not&lt;/em&gt; heavy tailed according to the definition above, is the logistic distribution. However, this is &lt;strong&gt;not&lt;/strong&gt; in agreement with e.g. &lt;a href=&quot;http://en.wikipedia.org/wiki/Fat_tail&quot; rel=&quot;nofollow&quot;&gt;Wikipedia&lt;/a&gt;, which is much more restrictive and requires that the (right) tail has a &lt;em&gt;power law decay&lt;/em&gt;. The Wikipedia article also suggests that fat tail and heavy tail are equivalent concepts, even though power law decay is much stronger than the definition of heavy tails given above.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To avoid confusions, I would recommend to use the definition of a (right) heavy tail above and forget about fat tails whatever that is. The primary reason behind the definition above is that in the analysis of rare events there is a qualitative difference between distributions with finite moment generating function on a positive interval and those with infinite moment generating function on $(0, \infty)$.    &lt;/p&gt;&#xA;" OwnerUserId="4376" LastEditorUserId="4376" LastEditorDisplayName="" LastEditDate="2011-05-14T18:14:19.110" LastActivityDate="2011-05-14T18:14:19.110" CommentCount="2" />
  <row Id="10737" PostTypeId="2" ParentId="5054" CreationDate="2011-05-12T20:00:40.270" Score="0" ViewCount="" Body="&lt;p&gt;A simple start using historical tweets data: Create a weekly variable called popularity change based on week to week changes in tweets for a tag for the past 25 weeks from the current time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Calculate these 2 measures:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;strong&gt;Trend:&lt;/strong&gt; Mean of popularity change&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Volatility:&lt;/strong&gt; &lt;a href=&quot;http://en.wikipedia.org/wiki/Standard_deviation&quot; rel=&quot;nofollow&quot;&gt;Standard Deviation&lt;/a&gt; (square root of&#xA;variance) of popularity change.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Meaning: A change from the number of tweets in your current week from the past week that is greater than 2 standard deviations can be considered a big change, and a change of 3 standard deviations would be a very big change (a very rare occurrence assuming the distribution of popularity changes for the time span looks like a &lt;a href=&quot;http://en.wikipedia.org/wiki/Normal_distribution&quot; rel=&quot;nofollow&quot;&gt;normal distribution&lt;/a&gt;). &lt;/p&gt;&#xA;" OwnerUserId="4329" LastActivityDate="2011-05-12T20:00:40.270" />
  <row Id="10738" PostTypeId="2" ParentId="10680" CreationDate="2011-05-12T20:17:49.997" Score="1" ViewCount="" Body="&lt;p&gt;Statistical analysis or statistical data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think this example in your question relates to statistical data: &quot;I read that 10% of the world population has this disease&quot;. In other words, in this example some one is using numbers to help communicate quantity more effectively than just saying 'many people'. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My guess is that the answer to your question is hidden in the motivation of the speaker on why she is using numbers. It could be to communicate some notion better or it could be to show authority or it could be to dazzle the listener. The good thing about stating numbers rather than saying 'very big' is that people can refute the number. See &lt;a href=&quot;http://en.wikipedia.org/wiki/Karl_Popper&quot; rel=&quot;nofollow&quot;&gt;Popper's idea&lt;/a&gt; on refutation.&lt;/p&gt;&#xA;" OwnerUserId="4329" LastActivityDate="2011-05-12T20:17:49.997" />
  <row Id="10739" PostTypeId="2" ParentId="10712" CreationDate="2011-05-12T21:04:50.773" Score="4" ViewCount="" Body="&lt;p&gt;Look at the help page for &lt;code&gt;?formula&lt;/code&gt; with regard to &lt;code&gt;.&lt;/code&gt; Here's the relevant bits:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;There are two special interpretations of . in a formula. The usual one is in the&#xA;context of a data argument of model fitting functions and means ‘all columns not&#xA;otherwise in the formula’: see terms.formula. In the context of update.formula, &#xA;only, it means ‘what was previously in this part of the formula’.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Alternatively, the &lt;code&gt;reshape&lt;/code&gt; and &lt;code&gt;reshape2&lt;/code&gt; packages use &lt;code&gt;.&lt;/code&gt; and &lt;code&gt;...&lt;/code&gt; a bit differently (from &lt;code&gt;?cast&lt;/code&gt;):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;There are a couple of special variables: &quot;...&quot; represents all other variables not&#xA;used in the formula and &quot;.&quot; represents no variable&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="696" LastActivityDate="2011-05-12T21:04:50.773" />
  <row Id="10740" PostTypeId="1" CreationDate="2011-05-12T21:05:20.087" Score="2" ViewCount="44" Body="&lt;p&gt;I am not so experience to design a customized covariance matrix / kernel functions. I would like to get such a feeling that after looking data, figure out the covariance matrix.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, in my case, I have a dataset, $X$ contains many zeros and couple of points far from them close to hundred. $Y$ is like a normal distribution, $N(50,10)$. $X$ and $Y$ are limited from 0 to 100.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, I am try to regress $X$ on $Y$, using gaussian process method. The difficulty arise from those many zeros that makes my covariant matrix messy. So, I have a large standard deviation for whole estimation.&lt;/p&gt;&#xA;" OwnerUserId="4581" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-12T23:37:24.897" LastActivityDate="2011-05-13T07:26:15.260" Title="Designing covariance matrix and kernel function for a gaussian process" Tags="&lt;regression&gt;&lt;machine-learning&gt;&lt;gaussian-process&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="10741" PostTypeId="2" ParentId="10719" CreationDate="2011-05-12T21:35:07.073" Score="4" ViewCount="" Body="&lt;p&gt;Unless I misunderstood the question, this refer to paired preference (1) or &lt;a href=&quot;http://en.wikipedia.org/wiki/Pairwise_comparison&quot; rel=&quot;nofollow&quot;&gt;pair comparison&lt;/a&gt; data. A well-known example of such a model is the Bradley-Terry model (2), which shares some connections with item scaling in psychometrics (3). There is an R package, &lt;a href=&quot;http://cran.r-project.org/web/packages/BradleyTerry2/index.html&quot; rel=&quot;nofollow&quot;&gt;BradleyTerry2&lt;/a&gt;, described in the JSS (2005) 12(1), &lt;a href=&quot;http://www.jstatsoft.org/v12/i01/paper&quot; rel=&quot;nofollow&quot;&gt;Bradley-Terry Models in R&lt;/a&gt;,  and a detailed overview in Agresti's CDA, pp. 436-439, with R code available in Laura Thompson's textbook, &lt;a href=&quot;https://home.comcast.net/~lthompson221/Splusdiscrete2.pdf&quot; rel=&quot;nofollow&quot;&gt;R (and S-PLUS) Manual to Accompany Agresti’s Categorical Data Analysis (2002) 2nd edition&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;References&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Thurstone, L.L. (1927). A law of comparative judgment. &lt;em&gt;Psychological Review&lt;/em&gt;, &lt;em&gt;3&lt;/em&gt;, 273-286.&lt;/li&gt;&#xA;&lt;li&gt;Bradley, R.A. and Terry, M.E. (1952). Rank analysis of incomplete block designs I: The methods of paired comparisons. &lt;em&gt;Biometrika&lt;/em&gt;, &lt;em&gt;39&lt;/em&gt;, 324-345.&lt;/li&gt;&#xA;&lt;li&gt;Andrich, D. (1978). Relationships between the Thurstone and Rasch approaches to item scaling. &lt;em&gt;Applied Psychological Measurement&lt;/em&gt;, &lt;em&gt;2(3)&lt;/em&gt;, 451-462.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-05-12T21:35:07.073" />
  <row Id="10742" PostTypeId="2" ParentId="10723" CreationDate="2011-05-12T22:19:02.890" Score="1" ViewCount="" Body="&lt;p&gt;If this is indeed linear then I think your underlying model may be something like &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Y_j \approx k \, \exp(aj) X_j^b $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where your regression does not tell you about the value of the constant $k$, but you might perhaps be able to use it to pin the first and last points of your observed data. &lt;/p&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-05-12T22:19:02.890" />
  <row Id="10743" PostTypeId="2" ParentId="10450" CreationDate="2011-05-12T22:23:22.400" Score="1" ViewCount="" Body="&lt;p&gt;Here's a heuristic that I coded up quickly that seems to do quite well:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Initialize a matrix with 1 on the diagonals.&lt;/li&gt;&#xA;&lt;li&gt;Fill out the upper triangular sub-matrix according to your distribution (90% are uniform on (-.3,.3) and 10% outside that).&lt;/li&gt;&#xA;&lt;li&gt;Make the matrix symmetric.&lt;/li&gt;&#xA;&lt;li&gt;Now iterate between&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Project the matrix onto the PSD cone.&lt;/li&gt;&#xA;&lt;li&gt;Project the matrix onto the set of matrices with diagonal 1.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;Alternating projections converges, so we just hope that the matrix we get out has values according to your distribution (see simulation for the check).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;&lt;pre&gt;&lt;code&gt;   pickone &amp;lt;- function(x){&#xA;  if(runif(1)&amp;lt;.9){&#xA;    return(runif(1,-.3,.3))&#xA;  } else {&#xA;    return(sample(c(-1,1),1)*runif(1,.3,1))&#xA;  }&#xA;}&#xA;&#xA;generateMat &amp;lt;- function(x){&#xA;  X &amp;lt;- matrix(0,nrow=10,ncol=10)&#xA;  diag(X) &amp;lt;- rep(1,10)&#xA;  X[upper.tri(X)] &amp;lt;- sapply(1:45,pickone)&#xA;  X &amp;lt;- X + t(X)-diag(rep(1,10))&#xA;  Xnew &amp;lt;- X&#xA;&#xA;  for(i in 1:50){&#xA;    eig &amp;lt;- eigen(Xnew)&#xA;    ##project onto the PSD cone&#xA;    Xnew &amp;lt;- eig$vectors%*%diag(sapply(eig$values,max,0))%*%t(eig$vectors)&#xA;    ##project onto the set of matrices with diagonal 1&#xA;    diag(Xnew) &amp;lt;- rep(1,10)&#xA;  }&#xA;&#xA;  vals &amp;lt;- Xnew[upper.tri(Xnew)]&#xA;  return(mean(vals &amp;lt; .3 &amp;amp; vals &amp;gt; -.3))&#xA;}&#xA;&#xA;summary(sapply(1:100,generateMat))&#xA;&#xA;   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. &#xA; 0.7556  0.8667  0.8889  0.8960  0.9333  0.9778&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It seems like most of the values after simulating 100 times are close to 90% within (-.3,.3).&lt;/p&gt;&#xA;" OwnerUserId="1815" LastActivityDate="2011-05-12T22:23:22.400" CommentCount="4" />
  <row Id="10744" PostTypeId="1" CreationDate="2011-05-13T01:06:27.260" Score="3" ViewCount="61" Body="&lt;p&gt;In gene expression studies using microarrays, intensity data has to be normalized so that intensities can be compared between individuals, between genes. Conceptually, and algorithmically, how does &quot;quantile normalization&quot; work, and how would you explain this to a non-statistician?&lt;/p&gt;&#xA;" OwnerUserId="36" LastActivityDate="2011-05-13T05:05:47.527" Title="How does quantile normalization work?" Tags="&lt;r&gt;&lt;genetics&gt;&lt;normalization&gt;&lt;microarray&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="10745" PostTypeId="1" CreationDate="2011-05-13T01:20:32.237" Score="4" ViewCount="75" Body="&lt;p&gt;All,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm working on a project looking at cross-national public opinion across two different observational &quot;waves&quot;.  Many countries were surveyed in both waves, though some were surveyed in the first wave and not the other (and vice-versa).  With predictors at both the level of the individual and the level of the country, a mixed effects model is appropriate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question involves specifying the random effects, or the groupings.  I think there are two: the country and the wave.  That is, individuals in certain countries are going to be more like each other than individuals in other countries.  Further, observations in the first wave are going to be more similar than with observations in the second wave, but it is not entirely a repeated measures design.  Thus, I'm inclined to believe I should model the random effect component in lmer as &lt;code&gt;(1 | country/wave)&lt;/code&gt; [alternatively: &lt;code&gt;(1|country:wave) + (1|country)&lt;/code&gt;], where the country is nested within the wave.  The results are a little sensitive to changes in the random effects structure.  Nesting waves within countries and excluding the wave grouping altogether produce different results on key independent variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was wanting to know if you think that's right.  I could also be inviting trouble by treating wave as a random effect when there are only two waves.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for any input.&lt;/p&gt;&#xA;" OwnerUserId="4594" LastEditorUserId="279" LastEditorDisplayName="" LastEditDate="2011-05-13T13:28:05.307" LastActivityDate="2011-05-13T13:28:05.307" Title="Properly specifying mixed effects model in lmer" Tags="&lt;r&gt;&lt;mixed-model&gt;&lt;lmer&gt;" AnswerCount="1" />
  <row Id="10746" PostTypeId="2" ParentId="10700" CreationDate="2011-05-13T04:09:32.393" Score="2" ViewCount="" Body="&lt;p&gt;Since you just need it for demonstration, how about using &lt;a href=&quot;http://www.minitab.com/en-US/default.aspx&quot; rel=&quot;nofollow&quot;&gt;Minitab&lt;/a&gt;? It is similarly transparent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/RExcel&quot; rel=&quot;nofollow&quot;&gt;RExcel&lt;/a&gt; looks promising too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, both of these options are somewhat opaque because all of the software is proprietary, closed-source software. You could also try &lt;a href=&quot;http://wiki.services.openoffice.org/wiki/R_and_Calc&quot; rel=&quot;nofollow&quot;&gt;R and Calc&lt;/a&gt;, which uses open-source software. (I know that's not what you meant by transparent.)&lt;/p&gt;&#xA;" OwnerUserId="3874" LastActivityDate="2011-05-13T04:09:32.393" CommentCount="2" />
  <row Id="10747" PostTypeId="2" ParentId="10615" CreationDate="2011-05-13T04:22:02.233" Score="2" ViewCount="" Body="&lt;p&gt;I will to demonstrate how to do this without reshaping the data, as all it entails is simple arithmetic. If I am reading your question correctly, the data look something like this;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Data List Free / Group (A1) Grade1 Grade2 Grade3.&#xA;Begin Data&#xA;A 1 2 3&#xA;A 6 5 10&#xA;B 2 7 18&#xA;C 23 5 1&#xA;D 7 7 13&#xA;End Data.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To calculate any type of variance, you need to assign some type of numeric values to your grade variables. Here I will assume that a value for Grade1 is the equivalent of a 95, Grade 2 is equal to an 85, and Grade3 is equal to a 75. All that one needs to do after this is to essentially plug in the values for the calculation of the variance. The Wikipedia page for &lt;a href=&quot;http://en.wikipedia.org/wiki/Standard_deviation&quot; rel=&quot;nofollow&quot;&gt;standard deviation&lt;/a&gt; has an explicit example taking you through the steps. The only difference in the code below is that I need to multiply the squared differences by the number of observations within that specific grade variable.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;compute row_sum = (Grade1*95)+(Grade2*85)+(Grade3*75).&#xA;compute row_n = Grade1+Grade2+Grade3.&#xA;compute row_mean = row_sum/row_n.&#xA;execute.&#xA;&#xA;compute square_diff1 = (95 - row_mean)*(95 - row_mean).&#xA;compute square_diff2 = (85 - row_mean)*(85 - row_mean).&#xA;compute square_diff3 = (75 - row_mean)*(75 - row_mean).&#xA;execute.&#xA;&#xA;compute row_variance = ( (Grade1*square_diff1) + (Grade2*square_diff2) + (Grade3*square_diff3) ) / (row_n).&#xA;execute.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The resulting variable, &lt;code&gt;row_variance&lt;/code&gt; , is a calculation of the variance for the observations in the row. In your example given it suggested you would have groupings of variables, and hence would need to calculate not only the row variance, but the group variance. You can simply use the AGGREGATE command to sum the Grade variables within each group, and then follow the same steps above. An example would be;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;DATASET DECLARE Grouped.&#xA;AGGREGATE&#xA;  /OUTFILE='Grouped'&#xA;  /BREAK=Group&#xA;  /Grade1 =SUM(Grade1) &#xA;  /Grade2 =SUM(Grade2)&#xA;  /Grade3 =SUM(Grade3).&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This will produce a new dataset named Grouped, in which you can calculate the row variance using the exact same code above. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a note, you asked for the variance in your question. I suspect the &lt;em&gt;sample&lt;/em&gt; standard deviation is another statistic you might be interested in. One can not simply take the square root of the variance I gave above though, as one divides by the number of observations minus one for the sample standard deviation. So if you were interested in the sample standard deviation in the row you could use the code below;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;compute row_sd =SQRT( ( (Grade1*square_diff1) + (Grade2*square_diff2) + (Grade3*square_diff3) ) / (row_n - 1) ).&#xA;execute.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I doubt this is the simplest solution in terms of length of code. But I hope it is straightforward mathematically.&lt;/p&gt;&#xA;" OwnerUserId="1036" LastActivityDate="2011-05-13T04:22:02.233" CommentCount="3" />
  <row Id="10748" PostTypeId="2" ParentId="10693" CreationDate="2011-05-13T04:43:39.707" Score="1" ViewCount="" Body="&lt;p&gt;You just use a generic t-test with matched pairs. For each worker, measure before and after. Use a one-sample t-test to test whether the difference between these two measurements is zero.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've actually never before heard anyone use &quot;repeated measures&quot; to refer to fewer than three measurement times. Repeated measures gets more complicated when you have three measurement times, which may make the question's phrasing sort of scary.&lt;/p&gt;&#xA;" OwnerUserId="3874" LastActivityDate="2011-05-13T04:43:39.707" />
  <row Id="10749" PostTypeId="2" ParentId="10744" CreationDate="2011-05-13T05:05:47.527" Score="1" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://bioinformatics.oxfordjournals.org/content/19/2/185.abstract&quot; rel=&quot;nofollow&quot;&gt;A comparison of normalization methods for high density oligonucleotide array data based on variance and bias&lt;/a&gt; by Bolstad et al. introduces quantile normalization for array data and compares it to other methods. It has a pretty clear description of the algorithm. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The conceptual understanding is that it is a transformation of array $j$ using a function $\hat{F}^{-1} \circ \hat{G}_j$ where $\hat{G}_j$ is an estimated distribution function and $\hat{F}^{-1}$ is the inverse of an estimated distribution function. It has the consequence that the normalized distributions become identical for all the arrays. For quantile normalization $\hat{G}_j$ is the empirical distribution of array $j$ and $\hat{F}$ is the empirical distribution for the averaged quantiles across arrays. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;At the end of the day it is a method for transforming all the arrays to have a common distribution of intensities. &lt;/p&gt;&#xA;" OwnerUserId="4376" LastActivityDate="2011-05-13T05:05:47.527" />
  <row Id="10750" PostTypeId="1" CreationDate="2011-05-13T05:55:17.087" Score="4" ViewCount="102" Body="&lt;p&gt;I have a time series. I want to model it using ARMA, which will be used for forcasting. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In R I am using &lt;code&gt;arima()&lt;/code&gt; function to get the coefficients. But &lt;code&gt;arima()&lt;/code&gt; requires order(p,d,q) as input. What is the simplest way in R to arrive at a good value for p and q (with d = 0) so that I don't overfit?&lt;/p&gt;&#xA;" OwnerUserId="4596" LastEditorUserId="1390" LastEditorDisplayName="" LastEditDate="2011-05-13T07:06:54.123" LastActivityDate="2011-05-13T07:17:44.377" Title="ARMA modeling in R" Tags="&lt;r&gt;&lt;time-series&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="10751" PostTypeId="2" ParentId="10750" CreationDate="2011-05-13T07:16:27.647" Score="5" ViewCount="" Body="&lt;p&gt;Simplest way to arrive at values for $p$ and $q$ is using &lt;code&gt;auto.arima&lt;/code&gt; function from package &lt;strong&gt;forecast&lt;/strong&gt;. There is no simplest way in any statistical package to arrive at good values. The main reason for that is that there is no universal definition of good. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since you mention overfitting, one possible way is to fit arima models for different values of $p$ and $q$ and then pick the one which is the best according to your overfitting criteria (out of sample forecasting performance for example). &lt;code&gt;auto.arima&lt;/code&gt; does basically the same, you can choose between AIC, AICC and BIC to let &lt;code&gt;auto.arima&lt;/code&gt; pick the best model. &lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-05-13T07:16:27.647" CommentCount="4" />
  <row Id="10752" PostTypeId="2" ParentId="10750" CreationDate="2011-05-13T07:17:44.377" Score="5" ViewCount="" Body="&lt;p&gt;One option is to fit a series of ARMA models with combinations of $p$ and $q$ and work with the model that has the best &quot;fit&quot;. Here I evaluate &quot;fit&quot; using BIC to attempt to penalise overly complex fits. An example is shown below for the in-built Mauna Loa $\mathrm{CO}_2$ concentration data set&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;## load the data&#xA;data(co2)&#xA;## take only data up to end of 1990 - predict for remaining data later&#xA;CO2 &amp;lt;- window(co2, end = c(1990, 12))&#xA;&#xA;## Set up the parameter sets over which we want to operate&#xA;CO2.pars &amp;lt;- expand.grid(ar = 0:2, diff = 1, ma = 0:2, sar = 0:1,&#xA;                        sdiff = 1, sma = 0:1)&#xA;## As you are only wanting ARMA, then you would need something like&#xA;## pars &amp;lt;- expand.grid(ar = 0:4, diff = 0, ma = 0:4)&#xA;## and where you choose the upper and lower limits - here 0 and 4&#xA;&#xA;## A vector to hold the BIC values for each combination of model&#xA;CO2.bic &amp;lt;- rep(0, nrow(CO2.pars))&#xA;&#xA;## loop over the combinations, fitting an ARIMA model and recording the BIC&#xA;## for that model. Note we use AIC() with extra penalty given by `k`&#xA;for (i in seq(along = CO2.bic)) {&#xA;    CO2.bic[i] &amp;lt;- AIC(arima(CO2, unlist(CO2.pars[i, 1:3]), &#xA;                            unlist(CO2.pars[i, 4:6])),&#xA;                      k = log(length(CO2)))&#xA;}&#xA;&#xA;## identify the model with lowest BIC&#xA;CO2.pars[which.min(CO2.bic), ]&#xA;&#xA;## Refit the model with lowest BIC&#xA;CO2.mod &amp;lt;- arima(CO2, order = c(0, 1, 1), seasonal = c(0, 1, 1))&#xA;CO2.mod&#xA;## Diagnostics plots&#xA;tsdiag(CO2.mod, gof.lag = 36)&#xA;&#xA;## predict for the most recent data&#xA;pred &amp;lt;- predict(CO2.mod, n.ahead = 7 * 12)&#xA;upr &amp;lt;- pred$pred + (2 * pred$se) ## upper and lower confidence intervals&#xA;lwr &amp;lt;- pred$pred - (2 * pred$se) ## approximate 95% pointwise&#xA;&#xA;## plot what we have done&#xA;ylim &amp;lt;- range(co2, upr, lwr)&#xA;plot(co2, ylab = ylab, main = expression(bold(Mauna ~ Loa ~ CO[2])),&#xA;     xlab = &quot;Year&quot;, ylim = ylim)&#xA;lines(pred$pred, col = &quot;red&quot;)&#xA;lines(upr, col = &quot;red&quot;, lty = 2)&#xA;lines(lwr, col = &quot;red&quot;, lty = 2)&#xA;legend(&quot;topleft&quot;, legend = c(&quot;Observed&quot;, &quot;Predicted&quot;, &quot;95% CI&quot;),&#xA;       col = c(&quot;black&quot;, &quot;red&quot;, &quot;red&quot;), lty = c(1, 1, 2), bty = &quot;n&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1390" LastActivityDate="2011-05-13T07:17:44.377" CommentCount="7" />
  <row Id="10753" PostTypeId="2" ParentId="10740" CreationDate="2011-05-13T07:26:15.260" Score="0" ViewCount="" Body="&lt;p&gt;I've not worked through the details of GPs, so I cannot help you there.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, it seems like you have two groups of data in X: X=0 and X&gt;0.  You may get better results by first classifying X into these two groups based on Y, and then performing GP regression in the X&gt;0 class.&lt;/p&gt;&#xA;" OwnerUserId="3595" LastActivityDate="2011-05-13T07:26:15.260" CommentCount="1" />
  <row Id="10754" PostTypeId="2" ParentId="10723" CreationDate="2011-05-13T07:32:40.563" Score="3" ViewCount="" Body="&lt;p&gt;Since you have differences, this means that the data is time series and we can write&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Y_t=Y_0+\sum_{s=1}^t\Delta Y_s$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So if the true model is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\Delta Y_t=\alpha+\beta \Delta X_t$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;we have&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Y_t=Y_0+\sum_{s=1}^t(\alpha+\beta \Delta X_t)=Y_0-\beta X_0+\alpha t+\beta X_t$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So you can say that interpretation remains the same as in model with levels.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-05-13T07:32:40.563" CommentCount="2" />
  <row Id="10755" PostTypeId="5" CreationDate="2011-05-13T07:38:51.577" Score="0" ViewCount="" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditorDisplayName="" LastEditDate="2011-05-13T07:38:51.577" LastActivityDate="2011-05-13T07:38:51.577" />
  <row Id="10756" PostTypeId="4" CreationDate="2011-05-13T07:38:51.577" Score="0" ViewCount="" Body="Delta method is a method for showing that a function of an asymptotically normal statistical estimator is asymptotically normal." OwnerUserId="-1" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-13T08:30:56.507" LastActivityDate="2011-05-13T08:30:56.507" />
  <row Id="10757" PostTypeId="2" ParentId="10672" CreationDate="2011-05-13T08:40:46.693" Score="1" ViewCount="" Body="&lt;p&gt;Here are two survey papers I have found recently. I have not read them yet, but the abstracts sound promising.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.cs.nyu.edu/~mohri/pub/bandit.pdf&quot; rel=&quot;nofollow&quot;&gt;Joann`s Vermorel and Mehryar Mohri: Multi-Armed Bandit Algorithms and Empirical Evaluation&lt;/a&gt; (2005)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the abstract: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The multi-armed bandit problem for a gambler is to decide&#xA;  which arm of a K-slot machine to pull to maximize his total reward in&#xA;  a series of trials. Many real-world learning and optimization problems&#xA;  can be modeled in this way. Several strategies or algorithms have been&#xA;  proposed as a solution to this problem in the last two decades, but, to our&#xA;  knowledge, there has been no common evaluation of these algorithms.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.cs.mcgill.ca/~vkules/bandits.pdf&quot; rel=&quot;nofollow&quot;&gt;Volodymyr Kuleshov and Doina Precup: Algorithms for the multi-armed bandit problem&lt;/a&gt; (2000)&#xA;From the abstract: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Secondly, the performance of&#xA;  most algorithms varies dramatically with the parameters of the bandit problem. Our study&#xA;  identiﬁes for each algorithm the settings where it performs well, and the settings where it performs poorly. &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="264" LastEditorUserId="264" LastEditorDisplayName="" LastEditDate="2011-05-13T08:51:01.957" LastActivityDate="2011-05-13T08:51:01.957" />
  <row Id="10758" PostTypeId="1" CreationDate="2011-05-13T09:44:48.233" Score="0" ViewCount="67" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;Possible Duplicate:&lt;/strong&gt;&lt;br&gt;&#xA;  &lt;a href=&quot;http://stats.stackexchange.com/questions/577/is-there-any-reason-to-prefer-the-aic-or-bic-over-the-other&quot;&gt;Is there any reason to prefer the AIC or BIC over the other?&lt;/a&gt;  &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&#xA;&#xA;&lt;p&gt;Hello!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just very quick! What reason could I give my professor, why I'am using the AIC or BIC in a linear (or generalized linear) model? I know that BIC penalisation is higher, but I don't want to say that I choosed BIC because I'm a lazy guy that doesn't want to interpret a lot of interaction effects (even if this is the truth :D).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks Guys!&lt;/p&gt;&#xA;" OwnerUserId="4496" LastActivityDate="2011-05-13T09:44:48.233" ClosedDate="2011-05-13T10:29:06.827" Title="AIC versus BIC Criteria" Tags="&lt;model-selection&gt;&lt;aic&gt;&lt;bic&gt;" CommentCount="1" />
  <row Id="10759" PostTypeId="2" ParentId="10309" CreationDate="2011-05-13T09:55:18.950" Score="2" ViewCount="" Body="&lt;p&gt;As @caracal's said, this script implements a permutation-based approach to Friedman's test with the &lt;a href=&quot;http://cran.r-project.org/web/packages/coin/index.html&quot; rel=&quot;nofollow&quot;&gt;coin&lt;/a&gt; package. &#xA;The maxT procedure is rather complex and there is no relation with the traditional $\chi^2$ statistic you're probably used to get after a Friedman ANOVA. The general idea is to control the &lt;a href=&quot;http://en.wikipedia.org/wiki/Familywise_error_rate&quot; rel=&quot;nofollow&quot;&gt;FWER&lt;/a&gt;. Let's say you perform 1000 permutations, for every variable of interest, then you can derive not only pointwise empirical p-values for each variable (as you would do with a single permutation test) but also a value that accounts for the fact that you tested all those variables at the same time. The latter is achieved by comparing each observed test statistic against the maximum of permuted statistics over all variables. In other words, this p-value reflects the chance of seeing a test statistic as large as the one you observed, given you've performed as many tests.&#xA;More information (in a genomic context, and with algorithmic considerations) can be found in &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Dudoit, S., Shaffer, J.P., and&#xA;  Boldrick, J.C. (2003). &lt;a href=&quot;http://www.bepress.com/cgi/viewcontent.cgi?article=1014&amp;amp;context=ucbbiostat&quot; rel=&quot;nofollow&quot;&gt;Multiple&#xA;  Hypothesis Testing in Microarray&#xA;  Experiments&lt;/a&gt;. &lt;em&gt;Statistical&#xA;  Science&lt;/em&gt;, &lt;em&gt;18(1)&lt;/em&gt;, 71–103.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;(Here are some &lt;a href=&quot;http://www.epibiostat.ucsf.edu/biostat/cbmb/courses/multtest.pdf&quot; rel=&quot;nofollow&quot;&gt;slides&lt;/a&gt; from the same author with applications in R with the &lt;a href=&quot;http://www.bioconductor.org/packages/release/bioc/html/multtest.html&quot; rel=&quot;nofollow&quot;&gt;multtest&lt;/a&gt; package.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another good reference is &lt;em&gt;Multiple Testing Procedures with Applications to Genomics&lt;/em&gt;, by Dudoit and van der Laan (Springer, 2008).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, if you want to get more &quot;traditional&quot; statistic, you can use the &lt;a href=&quot;http://cran.r-project.org/web/packages/agricolae/index.html&quot; rel=&quot;nofollow&quot;&gt;agricolae&lt;/a&gt; package which has a &lt;code&gt;friedman()&lt;/code&gt; function that performs the overall Friedman's test followed by post-hoc comparisons.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The permutation method yields a maxT=3.24, p=0.003394, suggesting an overall effect of the target when accounting for the blocking factor. The post-hoc tests basically indicate that only results for Wine A vs. Wine C (p=0.003400) are statistically different at the 5% level. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using the non-parametric test, we have&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; library(agricolae)&#xA;&amp;gt; with(WineTasting, friedman(Taster, Wine, Taste, group=FALSE)) &#xA;Friedman's Test&#xA;===============&#xA;Adjusted for ties&#xA;Value: 11.14286&#xA;Pvalue chisq : 0.003805041&#xA;F value : 7.121739&#xA;Pvalue F: 0.002171298&#xA;&#xA;Alpha     : 0.05&#xA;t-Student : 2.018082&#xA;&#xA;Comparison between treatments&#xA;Sum of the ranks&#xA;&#xA;                Difference   pvalue sig   LCL   UCL&#xA;Wine A - Wine B          6 0.301210     -5.57 17.57&#xA;Wine A - Wine C         21 0.000692 ***  9.43 32.57&#xA;Wine B - Wine C         15 0.012282   *  3.43 26.57&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The two global tests agree and basically say there is a significant effect of Wine type. We would, however, reach different conclusions about the pairwise difference. It should be noted that the above pairwise tests (Fisher's LSD) are not really corrected for multiple comparisons, although the difference B-C would remain significant even after Holm's correction (which also provides strong control of the FWER).&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-05-13T09:55:18.950" />
  <row Id="10760" PostTypeId="2" ParentId="6298" CreationDate="2011-05-13T10:24:17.390" Score="0" ViewCount="" Body="&lt;p&gt;Google is using different machine learning techniques and algorithm for training and prediction. The strategies for large-scale supervised learning:&#xA;1. Sub-sample&#xA;2. Embarrassingly parallelize some algorithms&#xA;3. Distributed gradient descent&#xA;4. Majority Vote&#xA;5. Parameter mixture&#xA;6. Iterative parameter mixture&lt;/p&gt;&#xA;&#xA;&lt;p&gt;They should train and predict the model with the different machine learning techniques and using an algorithm to decide the best model and prediction to return.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Sub-sampling provides inferior performance&lt;/li&gt;&#xA;&lt;li&gt;Parameter mixture improves, but not as good as all data&lt;/li&gt;&#xA;&lt;li&gt;Distributed algorithms return better classifiers quicker&lt;/li&gt;&#xA;&lt;li&gt;Iterative parameter mixture achieves as good as all data&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;But of course it is not really clear in the API documentation.&lt;/p&gt;&#xA;" OwnerUserId="4531" LastActivityDate="2011-05-13T10:24:17.390" />
  <row Id="10761" PostTypeId="2" ParentId="10700" CreationDate="2011-05-13T10:55:57.083" Score="4" ViewCount="" Body="&lt;p&gt;It sounds like your goal is didactic; that you are trying to explain ordinal logistic to some group of people.  I have used Excel for this sort of thing when the topic is much simpler - e.g., crosstabs and chi-square - so that there is some intuition about the formulas.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't think that will be the case here.  Even if you could find (or create) an Excel spreadsheet that does this, I think the intermediate steps are so numerous that it will not be clear to any audience that could not follow a more usual explanation of ordinal logistic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would use some standard software and give different examples. I've written a talk doing this, using SAS, but it could be done using R or whatever.&lt;/p&gt;&#xA;" OwnerUserId="686" LastActivityDate="2011-05-13T10:55:57.083" />
  <row Id="10762" PostTypeId="2" ParentId="10745" CreationDate="2011-05-13T13:27:04.293" Score="2" ViewCount="" Body="&lt;p&gt;By nesting country within wave, you are cutting the connection between the repeated measurements within the same country. I would just use crossed random effects:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; (1|country) + (1|wave) + (1|country:wave)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="279" LastActivityDate="2011-05-13T13:27:04.293" CommentCount="1" />
  <row Id="10763" PostTypeId="2" ParentId="577" CreationDate="2011-05-13T14:06:44.390" Score="1" ViewCount="" Body="&lt;p&gt;From what I can tell, there isn't much difference between AIC and BIC.  They are both mathematically convenient &lt;em&gt;approximations&lt;/em&gt; one can make in order to efficiently compare models.  If they give you different &quot;best&quot; models, it probably means you have high model uncertainty, which is more important to worry about than whether you should use AIC or BIC.  I personally like BIC better because it asks more (less) of a model if it has more (less) data to fit its parameters - kind of like a teacher asking for a higher (lower) standard of performance if their student has more (less) time to learn about the subject.  To me this just seems like the intuitive thing to do.  But then I am certain there also exists equally intuitive and compelling arguments for AIC as well, given its simple form.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now any time you make an approximation, there will surely be some conditions when those approximations are rubbish.  This can be seen certainly for AIC, where there exist many &quot;adjustments&quot; (AICc) to account for certain conditions which make the original approximation bad.  This is also present for BIC, because various other more exact (but still efficient) methods exist, such as Fully Laplace Approximations to mixtures of Zellner's g-priors (BIC is an approximation to the Laplace approximation method for integrals).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One place where they are both crap is when you have substantial prior information about the parameters within any given model.  AIC and BIC unnecessarily penalise models where parameters are partially known compared to models which require parameters to be estimated from the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;one thing I think is important to note is that BIC does not assume a &quot;true&quot; model a) exists, or b) is contained in the model set.  BIC is simply an approximation to an integrated likelihood $P(D|M,A)$ (D=Data, M=model, A=assumptions).  Only by multiplying by a prior probability and then normalising can you get $P(M|D,A)$.  BIC simply represents how likely the data was if the proposition implied by the symbol $M$ is true.  So from a logical viewpoint, any proposition which would lead one to BIC as an approximation are equally supported by the data.  So if I state $M$ and $A$ to be the propositions&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\begin{array}{l|l}&#xD;&#xA;M_{i}:\text{the ith model is the best description of the data}&#xD;&#xA;\\&#xD;&#xA;A:\text{out of the set of K models being considered, one of them is the best}&#xD;&#xA;\end{array}&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And then continue to assign the same probability models (same parameters, same data, same approximations, etc.), I will get the same set of BIC values.  It is only by attaching some sort of unique meaning to the logical letter &quot;M&quot; that one gets drawn into irrelevant questions about &quot;the true model&quot; (echoes of &quot;the true religion&quot;).  The only thing that &quot;defines&quot; M is the mathematical equations which use it in their calculations - and this is hardly ever singles out one and only one definition.  I could equally put in a prediction proposition about M (&quot;the ith model will give the best predictions&quot;).  I personally can't see how this would change any of the likelihoods, and hence how good or bad BIC will be (AIC for that matter as well - although AIC is based on a different derivation)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And besides, what is wrong with the statement &lt;em&gt;If the true model is in the set I am considering, then there is a 57% probability that it is model B&lt;/em&gt;.  Seems reasonable enough to me, or you could go the more &quot;soft&quot; version &lt;em&gt;there is a 57% probability that model B is the best out of the set being considered&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One last comment: I think you will find about as many opinions about AIC/BIC as there are people who know about them.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-05-13T14:06:44.390" />
  <row Id="10764" PostTypeId="5" CreationDate="2011-05-13T14:19:52.103" Score="0" ViewCount="" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditorDisplayName="" LastEditDate="2011-05-13T14:19:52.103" LastActivityDate="2011-05-13T14:19:52.103" />
  <row Id="10765" PostTypeId="4" CreationDate="2011-05-13T14:19:52.103" Score="0" ViewCount="" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditorDisplayName="" LastEditDate="2011-05-13T14:19:52.103" LastActivityDate="2011-05-13T14:19:52.103" />
  <row Id="10766" PostTypeId="1" CreationDate="2011-05-13T14:45:17.463" Score="2" ViewCount="86" Body="&lt;p&gt;I have two cases:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Two random poisson variables $X_1 \sim \text{Pois}(\lambda_1)$, $X_2 \sim \text{Pois}(\lambda_2)$, and testing:&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Null Hypothesis: $\lambda_1 = \lambda_2$&lt;/li&gt;&#xA;&lt;li&gt;Alternate hyp: $\lambda_1 \neq \lambda_2$&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;Two random binomial variables $X_1 \sim \text{Binom}(n_1, p_1)$, $X_2 \sim \text{Binom}(n_2, p_2)$, where $n_1=n_2$:&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Null Hypothesis: $p_1 = p_2$&lt;/li&gt;&#xA;&lt;li&gt;Alternate hyp  : $p_1 \neq p_2$&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I started with likelihood ratio test for both cases, where I calculated maximum likelihood estimates for both cases, and then estimated likelihood ratio test statistics. But while reading some literature and discussion, I learned likelihood ratio test may not be a valid test in all cases, such as at low lambda values or one lambda low and other high, so for first case (poisson case), I could also explore the option of poisson random variable condition on sum of $X_1+X_2$ which will be $\text{Binom}(X_1+X_2, 0.5)$ and can be tested using Fisher exact test. Similarly for second case with two binom random variables a good choice will be McNamara test. And there are also some Bayesian tests which can be used (I do not know of any).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I am wondering how could I do Fisher exact test for two poisson r.v condition on sum of them, and McNamara test for two binom, and if there is any bayesian test which i could use.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Preferably, i will try these test in R as I am trying to learn R. In past I received very good help and would like to thank in advance. thanks&lt;/p&gt;&#xA;" OwnerUserId="4098" LastEditorUserId="4360" LastEditorDisplayName="" LastEditDate="2011-05-13T18:49:21.900" LastActivityDate="2011-05-13T19:06:40.390" Title="Two poisson random variables and likelihood ratio test" Tags="&lt;r&gt;&lt;maximum-likelihood&gt;" AnswerCount="2" />
  <row Id="10767" PostTypeId="1" CreationDate="2011-05-13T14:56:25.473" Score="3" ViewCount="50" Body="&lt;p&gt;I am overthinking this for sure, but I am stumped. I have a historical data set of projects with hours of contribution by various positions. There are six types of projects. How can I model the average contribution of each position for future forecasting purposes? Linear regression doesn't work because that models the independent contribution of each position. The end goal is to establish a formula that will allow me to say &quot;when working on project type 1, if position A contributes 40 hours, then positions B and C usually need X hours a piece.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data look like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Total Hours    Project Type    Position A    Position B    Position C&#xA;200                  1         100           40            60&#xA;140                  2         40            60            40&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And so on for about 700 projects.&lt;/p&gt;&#xA;" OwnerUserId="4600" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-13T16:29:02.787" LastActivityDate="2011-05-13T16:29:02.787" Title="Modeling relative contribution of a variable" Tags="&lt;modeling&gt;&lt;forecasting&gt;" CommentCount="3" />
  <row Id="10768" PostTypeId="1" AcceptedAnswerId="10814" CreationDate="2011-05-13T15:02:08.907" Score="8" ViewCount="159" Body="&lt;p&gt;If $F_Z$ is a CDF, it looks like $F_Z(z)^\alpha$ ($\alpha \gt 0$) is a CDF as well. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Q: Is this a standard result?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Q: Is there a good way to find a function $g$ with $X \equiv g(Z)$ s.t. $F_X(x) = F_Z(z)^\alpha$, where $ x \equiv g(z)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically, I have another CDF in hand, $F_Z(z)^\alpha$. In some reduced form sense I'd like to characterize the random variable that produces that CDF.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: I'd be happy if I could get an analytical result for the special case $Z \sim N(0,1)$. Or at least know that such a result is intractable.&lt;/p&gt;&#xA;" OwnerUserId="3577" LastEditorUserId="3577" LastEditorDisplayName="" LastEditDate="2011-05-14T15:50:10.710" LastActivityDate="2011-05-20T17:59:24.970" Title="CDF raised to a power?" Tags="&lt;data-transformation&gt;&lt;cdf&gt;" AnswerCount="3" CommentCount="6" FavoriteCount="1" />
  <row Id="10769" PostTypeId="2" ParentId="10697" CreationDate="2011-05-13T15:26:56.350" Score="0" ViewCount="" Body="&lt;p&gt;Below is code which implements my solution and Paramonov's solution (a slight edit: I have changed &lt;code&gt;dlmFilter(u,mod)$a&lt;/code&gt; in the orginally posted answer by &#xA;&lt;code&gt;dlmFilter(u,mod)$m&lt;/code&gt;).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(dlm)&#xA;set.seed(1234)&#xA;reps      &amp;lt;- 100&#xA;MyEstimates &amp;lt;- YourEstimates &amp;lt;- matrix(0,reps,2)&#xA;for (i in (1:reps) ) {&#xA;X &amp;lt;- r &amp;lt;- rnorm(100)&#xA;u &amp;lt;- -1*r + 0.5*rnorm(100)&#xA;#&#xA;fit &amp;lt;- dlmMLE(u, parm = c(1, sd(u)),&#xA;              build = function(x)&#xA;                dlmModReg(r, FALSE, dV = x[2]^2,&#xA;                          m0 = x[1], C0 = matrix(0)))&#xA;YourEstimates[i,] &amp;lt;- fit$par&#xA;#&#xA;MyModel &amp;lt;- function(x)  dlmModReg(X, FALSE, dV = x[1]^2)&#xA;fit &amp;lt;- dlmMLE(u, parm = c(0.3), build = MyModel)&#xA;mod &amp;lt;- MyModel(fit$par)&#xA;MyEstimates[i,] &amp;lt;- c(dlmFilter(u,mod)$m[101],fit$par[1])&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When I run the above code, this is what I get:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; summary(YourEstimates)&#xA;       V1                V2         &#xA; Min.   :-9.5284   Min.   :-0.5747  &#xA; 1st Qu.:-1.4280   1st Qu.: 0.4710  &#xA; Median :-0.9795   Median : 0.4937  &#xA; Mean   :-0.9737   Mean   : 0.4369  &#xA; 3rd Qu.:-0.5636   3rd Qu.: 0.5215  &#xA; Max.   : 4.5222   Max.   : 0.5980  &#xA;&amp;gt; summary(MyEstimates)&#xA;       V1                V2         &#xA; Min.   :-1.1099   Min.   :-0.6010  &#xA; 1st Qu.:-1.0266   1st Qu.: 0.4736  &#xA; Median :-0.9974   Median : 0.4961  &#xA; Mean   :-0.9938   Mean   : 0.4469  &#xA; 3rd Qu.:-0.9635   3rd Qu.: 0.5158  &#xA; Max.   :-0.8390   Max.   : 0.5776  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;While the first set of estimates gives similar estimates for the second parameter, it occasionally gives values well off the mark for the first. I think the reason is that &quot;tying&quot; the state to its initial value with &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;C0=matrix(0)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;leads to numerical instability, but I am not sure. In any case, you may want to look at the issue.&lt;/p&gt;&#xA;" OwnerUserId="892" LastActivityDate="2011-05-13T15:26:56.350" CommentCount="1" />
  <row Id="10770" PostTypeId="2" ParentId="10766" CreationDate="2011-05-13T16:36:38.337" Score="0" ViewCount="" Body="&lt;p&gt;Did your reading suggest that the liklihood ratio test statistic had problems? or that the chi-squared approximation was not very good?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I expect that most of the problems are more likely the latter, the test statistic is fine, but we don't know the distribution of it under the null hypothesis.  With modern computers we can estimate the distribution fairly easily and compare to that (R works great for this).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just generate data that looks like yours but assuming the null hypothesis to be true, compute the LR statistic (or other statistic) for each simulated dataset.  Now compare the same statistic for your real data to the set of simulated statistics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another option, depending on the nature of your data and question, is to use permutation tests where the null hypothesis is that there is no difference between your groups, they are just samples from the same population.  So you mix them together and recreate the samples over and over again and compare the test statistic of interest from the original samples to the permuted samples.  Fisher's exact test is actualy a permutation test.&lt;/p&gt;&#xA;" OwnerUserId="4505" LastActivityDate="2011-05-13T16:36:38.337" CommentCount="1" />
  <row Id="10771" PostTypeId="2" ParentId="10171" CreationDate="2011-05-13T17:42:00.410" Score="0" ViewCount="" Body="&lt;p&gt;(Don't have much time now so I'll answer briefly and then expand later)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say that we are considering a binary classification problem and have a training set of $m$ class 1 samples and $n$ class 2 samples.  A permutation test for feature selection looks at each feature individually.  A test statistic $\theta$, such as information gain or the normalized difference between the means, is calculated for the feature.  The data for the feature is then randomly permuted and partitioned into two sets, one of size $m$ and one of size $n$.  The test statistic $\theta_p$ is then calculated based on this new partition $p$.  Depending on the computational complexity of the problem, this is then repeated over all possible partitions of the feature into two sets of order $m$ and $n$, or a random subset of these.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now that we have established a distribution over $\theta_p$, we calculate the p-value that the observed test statistic $\theta$ arose from a random partition of the feature.  The null hypothesis is that samples from each class come from the same underlying distribution (the feature is irrelevant).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This process is repeated over all features, and then the subset of features used for classification can be selected in two ways:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The $N$ features with the lowest p-values&lt;/li&gt;&#xA;&lt;li&gt;All features with a p-value$&amp;lt;\epsilon$&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="3595" LastActivityDate="2011-05-13T17:42:00.410" />
  <row Id="10772" PostTypeId="2" ParentId="10567" CreationDate="2011-05-13T18:15:07.350" Score="0" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;With respect to the question in the header&lt;/strong&gt;&#xA;With logistic regression predicting posterior probabilities, the &lt;em&gt;dependent&lt;/em&gt; variable (outcome) is both bounded and continuous.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One train of thoughts to arrive at logistic regression in fact is thinking how to construct a regression with limits for the continuous outcome. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;You want e.g. to do a regression directly on the probability&lt;/li&gt;&#xA;&lt;li&gt;&quot;Common&quot; regression methods (e.g. linear regression) give you continuous output in the set of real numbers, $\mathbb R$.&lt;/li&gt;&#xA;&lt;li&gt;But probabilities are in [0, 1]&lt;/li&gt;&#xA;&lt;li&gt;So put a sigmoid transformation into your model to transform $\mathbb R \mapsto [0, 1]$&lt;/li&gt;&#xA;&lt;li&gt;If you choose the &lt;a href=&quot;http://en.wikipedia.org/wiki/Logistic_function&quot; rel=&quot;nofollow&quot;&gt;logistic function $\frac{1}{1 + e^{-x}}$&lt;/a&gt; (a standard choice of a sigmoid), you end up  with logistic regression.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;With respect to modeling angles in general&lt;/strong&gt; I'd like to follow up with another question: how to model cyclic behaviour, how would I tell a model that 359° is almost the same as 0° (regardless of whether the variable is dependent or independent)?&lt;/p&gt;&#xA;" OwnerUserId="4598" LastEditorUserId="2669" LastEditorDisplayName="" LastEditDate="2011-05-14T07:40:35.677" LastActivityDate="2011-05-14T07:40:35.677" CommentCount="8" />
  <row Id="10773" PostTypeId="1" CreationDate="2011-05-13T18:36:09.233" Score="1" ViewCount="58" Body="&lt;p&gt;Hi!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Although reading quite a bunch of books, I'm still not sure, which method to use and how to implement it, therefore I appreciate any help! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have 4 different groups (treatments) with 50 participants each. Each participant's action is observed 5 times under the same condition. The 5 different values are collected within one &quot;round&quot;, where each participant has to select 5 items at an arbitrary time. &#xA;So I know that the different values for each participant are not independent of each other. However, time effects are not of interest for me and I don't assume that learning effects are apparent. I just collect 5 values for each participant in order to get more data. All the data within one group (for all users and their choices) is then taken together and a complex aggregated measure (a number) is calculated for all groups. Now the question is, whether there are significant differences for this measure between the groups. As there is no unbiased estimator for the measure and as my data is not normal, I know that I need some form of sampling. So the Welch or Levene tests, for instance, would not work here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm just not sure whether bootstrapping or permutation tests seem to be more appropriate.&#xA;Second, could I &quot;ignore&quot; the repeated measures and permute all values together without regard to the participants who provided it? If not, am I supposed to create vectors for each participant and permute these?&#xA;What happens if I have an unbalanced design (in case some observations are missing)?&#xA;Do you know any software where such a method is already implemented?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you so much for your help!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Pat&lt;/p&gt;&#xA;" OwnerUserId="4602" LastActivityDate="2011-05-13T19:29:16.703" Title="Permutation tests with repeated measures" Tags="&lt;repeated-measures&gt;&lt;permutation&gt;" AnswerCount="2" />
  <row Id="10774" PostTypeId="1" AcceptedAnswerId="10865" CreationDate="2011-05-13T18:44:04.633" Score="4" ViewCount="126" Body="&lt;h2&gt;Background&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I have data from a field study in which there are four treatment levels and six replicates in each of two blocks. (4x6x2=48 observations)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The blocks are about 1 mile apart, and within the blocks, there is a grid of 42, 2m x 4m plots and a 1m wide walkway; my study only used 24 plots in each block.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to evaluate evaluate spatial covariance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an example analysis using the data from a single block, without accounting for spatial covariance. In the dataset, &lt;code&gt;plot&lt;/code&gt; is the plot id, &lt;code&gt;x&lt;/code&gt; is the x location and &lt;code&gt;y&lt;/code&gt; the y location of each plot with plot 1 centered on 0, 0. &lt;code&gt;level&lt;/code&gt; is the treatment level and &lt;code&gt;response&lt;/code&gt; is the response variable. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;layout &amp;lt;- structure(list(plot = c(1L, 3L, 5L, 7L, 8L, 11L, 12L, 15L, 16L, &#xA;17L, 18L, 22L, 23L, 26L, 28L, 30L, 31L, 32L, 35L, 36L, 37L, 39L, &#xA;40L, 42L), level = c(0L, 10L, 1L, 4L, 10L, 0L, 4L, 10L, 0L, 4L, &#xA;0L, 1L, 0L, 10L, 1L, 10L, 4L, 4L, 1L, 1L, 1L, 0L, 10L, 4L), response = c(5.93, &#xA;5.16, 5.42, 5.11, 5.46, 5.44, 5.78, 5.44, 5.15, 5.16, 5.17, 5.82, &#xA;5.75, 4.48, 5.25, 5.49, 4.74, 4.09, 5.93, 5.91, 5.15, 4.5, 4.82, &#xA;5.84), x = c(0, 0, 0, 3, 3, 3, 3, 6, 6, 6, 6, 9, 9, 12, 12, 12, &#xA;15, 15, 15, 15, 18, 18, 18, 18), y = c(0, 10, 20, 0, 5, 20, 25, &#xA;10, 15, 20, 25, 15, 20, 0, 15, 25, 0, 5, 20, 25, 0, 10, 20, &#xA;25)), .Names = c(&quot;plot&quot;, &quot;level&quot;, &quot;response&quot;, &quot;x&quot;, &quot;y&quot;), row.names = c(NA, &#xA;-24L), class = &quot;data.frame&quot;)&#xA;&#xA;model &amp;lt;- lm(response ~ level, data = layout)      &#xA;summary(model)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h2&gt;Questions&lt;/h2&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;How can I calculate a covariance matrix and include it in my regression?&lt;/li&gt;&#xA;&lt;li&gt;The blocks are very different, and there are strong treatment * block interactions. Is it appropriate to analyze them separately?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="1381" LastEditorUserId="1381" LastEditorDisplayName="" LastEditDate="2011-05-16T15:59:16.513" LastActivityDate="2011-05-16T19:56:57.893" Title="How can I account for spatial covariance in a linear model?" Tags="&lt;r&gt;&lt;spatial&gt;&lt;linear-model&gt;&lt;covariance&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="10775" PostTypeId="2" ParentId="10773" CreationDate="2011-05-13T19:04:56.143" Score="1" ViewCount="" Body="&lt;p&gt;If you permute the individual values then you are testing the combined null hypothesis that there is no difference between groups and that there is no structure within values from the same individual.  So if you reject the null you don't know if it is because the groups differ or because there is structure within an individual.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would suggest permuting individuals and keeping all values with the individual.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't know of a program that does this out of the box, but writing a permutation test in R is often easier/quicker than figuring out a precanned program.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a simple example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(nlme)&#xA;&#xA;Odist &amp;lt;- split(Orthodont$distance, Orthodont$Subject)&#xA;&#xA;out &amp;lt;- replicate( 1999, { tmp &amp;lt;- sample(Odist);&#xA;    mean( unlist( tmp[1:16] ) ) - mean( unlist( tmp[17:27] ) ) } )&#xA;&#xA;out &amp;lt;- c(out, mean(unlist(Odist[1:16])) - mean(unlist(Odist[17:27])) )&#xA;&#xA;hist(out)&#xA;abline( v=out[2000] )&#xA;&#xA;mean(out &amp;gt;= out[2000])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="4505" LastEditorUserId="4505" LastEditorDisplayName="" LastEditDate="2011-05-13T19:29:16.703" LastActivityDate="2011-05-13T19:29:16.703" CommentCount="2" />
  <row Id="10776" PostTypeId="2" ParentId="10766" CreationDate="2011-05-13T19:06:40.390" Score="1" ViewCount="" Body="&lt;p&gt;The Bayesian tests for your question is based on the integrated (rather than maximised) likelihood.  So for Poisson we have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\begin{array}{c|c}&#xD;&#xA;H_{1}:\lambda_{1}=\lambda_{2} &amp;amp; H_{2}:\lambda_{1}\neq\lambda_{2}&#xD;&#xA;\end{array}&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now neither hypothesis say what the parameters are, so the actual values are nuisance parameters to be integrated out with respect to their prior probabilities.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(H_{1}|D,I)=P(H_{1}|I)\frac{P(D|H_{1},I)}{P(D|I)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The model likelihood is given by:&#xA;$$P(D|H_{1},I)=\int_{0}^{\infty} P(D,\lambda|H_{1},I)d\lambda=\int_{0}^{\infty} P(\lambda|H_{1},I)P(D|\lambda,H_{1},I)d\lambda$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$=\int_{0}^{\infty} P(\lambda|H_{1},I)\frac{\lambda^{x_1+x_2}exp(-2\lambda)}{\Gamma(x_1+1)\Gamma(x_2+1)}d\lambda$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $P(\lambda|H_{1},I)$ is the prior for lambda.  A convenient mathematical choice is the gamma prior, which gives:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(D|H_{1},I)=\int_{0}^{\infty} \frac{\beta^{\alpha}}{\Gamma(\alpha)}\lambda^{\alpha-1}\exp(-\beta \lambda)\frac{\lambda^{x_1+x_2}exp(-2\lambda)}{\Gamma(x_1+1)\Gamma(x_2+1)}d\lambda$$&#xA;$$=\frac{\beta^{\alpha}\Gamma(x_1+x_2+\alpha)}{(2+\beta)^{x_1+x_2+\alpha}\Gamma(\alpha)\Gamma(x_1+1)\Gamma(x_2+1)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And for the alternative hypothesis we have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(D|H_{2},I)=\frac{\beta_{1}^{\alpha_{1}}\beta_{2}^{\alpha_{2}}\Gamma(x_1+\alpha_{1})\Gamma(x_2+\alpha_{2})}{(1+\beta_{1})^{x_1+\alpha_{1}}(1+\beta_{2})^{x_2+\alpha_{2}}\Gamma(\alpha_{1})\Gamma(\alpha_{2})\Gamma(x_1+1)\Gamma(x_2+1)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now if we assume that all hyper-parameters are equal (not an unreasonable assumption, given that you are testing for equality), then we have an integrated likelihood ratio of:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{P(D|H_{1},I)}{P(D|H_{2},I)}=&#xD;&#xA;\frac{(1+\beta)^{x_1+x_2+2\alpha}\Gamma(x_1+x_2+\alpha)\Gamma(\alpha)}&#xD;&#xA;{(2+\beta)^{x_1+x_2+\alpha}\beta^{\alpha}\Gamma(x_1+\alpha)\Gamma(x_2+\alpha)}&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which you can see that the prior information is still very important.  We can't set $\alpha$ or $\beta$ equal to zero (jeffrey's prior), or else $H_{1}$ will always be favored, regardless of the data.  One way to get values for them is to specify prior estimates for $E[\lambda]$ and $E[log(\lambda)]$ and solve for the parameters - this cannot be based on $x_1$ or $x_2$ but can be based on any other relevant information.  You can also put in a few different (reasonable) values for the parameters and see what difference it makes to the conclusion.  The numerical value of this statistic tells you how much the data and your prior information about the rates in each hypothesis support the hypothesis of equal rates.  This explains why the likelihood ratio test is not always reliable - because it essentially ignores prior information, which is usually equivalent to specifying jeffrey's prior.  Note that you could also specify upper and lower limits for the rate parameters (this is usually not too hard to do given some common sense thinking about the real world problem).  The you would use a prior of the form:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p(\lambda|I)=\frac{I(L&amp;lt;\lambda&amp;lt;U)}{log(\frac{U}{L})\lambda}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And you would be left with a similar equation to that above but in terms of incomplete, instead of complete gamma functions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the binomial case things are much simpler, because the non-informative prior (uniform) is proper.  The procedure is similar to that above, and the integrated likelihood for $H_{1}:p_{1}=p_{2}$ is given by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(D|H_{1},I)={n_1 \choose x_1}{n_2 \choose x_2}\int_{0}^{1}p^{x_1+x_2}(1-p)^{n_1+n_2-x_1-x_2}dp$$&#xA;$$={n_1 \choose x_1}{n_2 \choose x_2}B(x_1+x_2+1,n_1+n_2-x_1-x_2+1)$$&#xA;And similarly for $H_{2}:p_{1}\neq p_{2}$&#xA;$$P(D|H_{2},I)={n_1 \choose x_1}{n_2 \choose x_2}\int_{0}^{1}p_{1}^{x_1}p_{2}^{x_2}(1-p_{1})^{n_1-x_1}(1-p_{2})^{n_{2}-x_{2}}dp_{1}dp_{2}$$&#xA;$$={n_1 \choose x_1}{n_2 \choose x_2}B(x_1+1,n_1-x_1+1)B(x_2+1,n_2-x_2+1)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And so taking ratios gives:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{P(D|H_{1},I)}{P(D|H_{2},I)}=&#xD;&#xA;\frac{B(x_1+x_2+1,n_1+n_2-x_1-x_2+1)}&#xD;&#xA;{B(x_1+1,n_1-x_1+1)B(x_2+1,n_2-x_2+1)}&#xD;&#xA;$$&#xA;$$=\frac{{x_1+x_2 \choose x_1}{n_1+n_2-x_1-x_2 \choose n_1-x_1}(n_1+1)(n_2+1)}{{n_1+n_2 \choose n_1}(n_1+n_2+1)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And the choose functions can be calculated using the hypergeometric(r,n,R,N) distribution where $N=n_1+n_2$, $R=x_1+x_2$, $n=n_1$, $r=x_1$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And this tells you how much the data support the hypothesis of equal probabilities, given that you don't have much information about which particular value this may be.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-05-13T19:06:40.390" CommentCount="3" />
  <row Id="10777" PostTypeId="2" ParentId="10773" CreationDate="2011-05-13T19:23:07.920" Score="0" ViewCount="" Body="&lt;p&gt;You can do permutations tests of both your within and between tests.  Just make sure that you permute your values nested within your participants.  So, if I'm a participant you can permute all the within conditions I ran in, but it's still all my data.  Then you can take each participant and permute them through the between conditions.&lt;/p&gt;&#xA;" OwnerUserId="601" LastActivityDate="2011-05-13T19:23:07.920" CommentCount="1" />
  <row Id="10778" PostTypeId="1" AcceptedAnswerId="10796" CreationDate="2011-05-13T19:27:26.613" Score="0" ViewCount="72" Body="&lt;p&gt;I have a general understanding of the difference between a population (set of entities under study) and a sample (a subsection selected from the population). However, I've been doing some work in PPC (Pay-Per-Click) and AdWords recently, and can't seem to grasp the population/sample difference in regards to that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, let's say there are two Google AdWords ads. Users will click the ad and it takes them to a form which they can fill out if they choose to. Therefore, I have data on the number of clicks and the number of forms filled out. The question I'm trying to answer is which Ad was more effective at getting more clicks and forms filled out. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;        Clicks      Forms&#xA;Ad1 -    300         105&#xA;Ad2 -    320         100&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Initially, I thought that my sample was the two ads (Ad1 and Ad2), but that wouldn't be right as I'm really examining the number of clicks and forms filled out. So it would seem that the population that I'm examining is the clicks and forms associated with the two ads (Ad1 and Ad2) and my sample would be the number of clicks and number of forms filled out. Is that right/wrong? Thus, would clicks and forms be considered two seperate samples taken from the same population? Or is my population the same as my sample in this case?&lt;/p&gt;&#xA;" OwnerUserId="3310" LastEditorUserId="3310" LastEditorDisplayName="" LastEditDate="2011-05-13T19:40:08.083" LastActivityDate="2011-05-14T05:15:57.400" Title="Difference between population and sample" Tags="&lt;sample-size&gt;&lt;population&gt;" AnswerCount="1" />
  <row Id="10779" PostTypeId="1" CreationDate="2011-05-13T19:29:20.173" Score="2" ViewCount="30" Body="&lt;p&gt;Suppose I am given $n$ samples of sizes $N_1, \dots, N_n$ from a Dirichlet&amp;ndash;multinomial distribution: Fixed and given is a $k$-vector $\mathbf{\alpha}$ of positive real numbers. For each $i, \, 1 \le i \le n$, a random probability vector $\mathbf{p}_i$ is drawn from a Dirichlet distribution $\mathrm{Dir}(\mathbf{\alpha})$, and then a sample of size $N_i$ is drawn from a multinomial distribution on $\{1, \dots, k\}$ with probabilities given by $\mathbf{p}_i$. The observed frequencies are recorded in an $n \times k$ table. Assume that there are no problems with low cell counts. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What can one say about the value $X$ of the $\chi^2$-statistic that can be computed for this random table? We can always write $\mathbf{\alpha} = M \mathbf{p_o}$  for some probability vector $\mathbf{p_o}$, and if $M$ is large, then all rows in this random table come from approximately the same multinomial distribution, so approximately $X \sim \chi^2_{r}$ with $r = (n-1)(k-1)$. How large must $M$ be for this to happen? And what happens for small $M$?&lt;/p&gt;&#xA;" OwnerUserId="4062" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-05-14T14:38:51.870" LastActivityDate="2011-05-14T14:38:51.870" Title="$\chi^2$ test for data from Dirichlet-multinomial distribution" Tags="&lt;chi-squared&gt;&lt;multinomial&gt;&lt;dirichlet-distribution&gt;" />
  <row Id="10780" PostTypeId="2" ParentId="10768" CreationDate="2011-05-13T19:42:43.590" Score="9" ViewCount="" Body="&lt;h2&gt;Proof without words&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.stack.imgur.com/NHSjO.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The lower blue curve is $F$, the upper red curve is $F^\alpha$ (typifying the case $\alpha \lt 1$), and the arrows show how to go from $z$ to $x = g(z)$.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-05-13T19:42:43.590" CommentCount="3" />
  <row Id="10781" PostTypeId="1" CreationDate="2011-05-13T19:53:41.427" Score="2" ViewCount="82" Body="&lt;p&gt;I really enjoy this resource -- its been super helpful to learn and read along as I explore some concepts for a new venture.  But now I need real help -- in the short term contract, and in the long term statistical pros. What are the best sites were folks look for side projects and gigs? What is the best advice you have for someone hiring stats folks?&lt;/p&gt;&#xA;" OwnerUserId="4606" LastActivityDate="2011-05-13T19:53:41.427" ClosedDate="2011-05-13T20:14:36.357" Title="What is the best site to hire contract or permanent stats folks" Tags="&lt;r&gt;" />
  <row Id="10782" PostTypeId="1" CreationDate="2011-05-13T20:21:44.867" Score="4" ViewCount="105" Body="&lt;p&gt;I am analyzing data from an experiment in which treatment levels increase quadratically, e.g. the treatment levels are $0, 1, 4, 9$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When analyzing the response using regression, would it make sense to use the square root of the treatment level as a predictor? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If so, how would this affect interpretation?&lt;/p&gt;&#xA;" OwnerUserId="1381" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-05-13T22:22:18.320" LastActivityDate="2011-05-13T23:12:11.193" Title="When to transform predictors?" Tags="&lt;regression&gt;&lt;data-transformation&gt;" AnswerCount="2" />
  <row Id="10783" PostTypeId="2" ParentId="8514" CreationDate="2011-05-13T21:01:47.547" Score="1" ViewCount="" Body="&lt;p&gt;Your stated objective: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Compare the population of several&#xA;  states in a small country.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Your stated problem: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Since some states have a population of&#xA;  3000,000 and some a population of&#xA;  2,000. Is there an easy way to&#xA;  &quot;normalise&quot; or make the data&#xA;  comparable?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;h2&gt;Aim of normalising your data before mapping&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;This answer will be lacking since I am not sure of the context of why you are making the map.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Nevertheless, here are some thoughts to explore: Normalise your data so that the map provides interesting meaning to the map's potential readers, so they can link what they see on your map to some concept they normally think about. Basically, I think your new normalised numbers should be linked to some qualitative concept that the map readers find interesting to understand (random tidbit: Measure = Quantity x Quality, Hegel). &lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Two proposed ways to normalise your data&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;1. In order to give a sense of how much open space is in each state.&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Create a new state variable for population density by calculating the population divided by total state area.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;2. In order to make the coloring of the states contrast with one another.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Create a new state variable by calculating the deviation from the mean of each state. For example, say you have 3 states with populations as follows:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;State A is 100.&lt;/li&gt;&#xA;&lt;li&gt;State B is  50.&lt;/li&gt;&#xA;&lt;li&gt;State C is 1.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The mean will be be about 50.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The new variable's values for each state will be as follows:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;State A is +50 (color intense &lt;em&gt;green&lt;/em&gt;).&lt;/li&gt;&#xA;&lt;li&gt;State B is  0 (color &lt;em&gt;grey&lt;/em&gt;).&lt;/li&gt;&#xA;&lt;li&gt;State C is -49 (color intense &lt;em&gt;red&lt;/em&gt;).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You can use any color scheme where positive numbers contrast with negative numbers (google 'colorbrewer' for lots of examples of color schemes for maps). &lt;/p&gt;&#xA;" OwnerUserId="4329" LastActivityDate="2011-05-13T21:01:47.547" />
  <row Id="10784" PostTypeId="1" AcceptedAnswerId="10785" CreationDate="2011-05-13T21:14:57.233" Score="2" ViewCount="71" Body="&lt;p&gt;(Apologies if the notations are &quot;unusual&quot;, I'm not sure what the correct notations should be. I'm putting an example at the end of the question.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's assume there was an initial dataset of an n by m matrix $M=(x_{ij})$ with $1&amp;lt;=i&amp;lt;=n$ and $1&amp;lt;=j&amp;lt;=m$, from which the following two vectors have been calculated:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;the vector of the column-wise means, that is, a vector of length $n$ where each element of index $j$ is the mean of the $m$ elements in column $j$ of the matrix: $(\bar{x}_{*j})$ with $1&amp;lt;=i&amp;lt;=n$, where $\bar{x}_{*j} = \frac{1}{m}\sum_{i=1}^m{x_{ij}}$.&lt;/li&gt;&#xA;&lt;li&gt;the corresponding vector of the columns-wise standard deviations ($s_{*j}$).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I'd like to get the mean and standard deviation of the row-wise vector of means, that is, the column vector made of the mean of each row: $(\bar{x}_{i*})$ with $1&amp;lt;=j&amp;lt;=m$, where $\bar{x}_{i*} = \frac{1}{n}\sum_{j=1}^n{x_{ij}}$, assuming that the initial matrix has been lost.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The mean of the elements in the row-wise vector is also the mean of the elements in the column-wise vector (which is also the mean of all the elements in the matrix, with equal weight): $\frac{1}{m}\sum_{i=1}^m{\bar{x}_{i*}} = \frac{1}{m}\sum_{i=1}^m({\frac{1}{n}\sum_{j=1}^n{x_{ij}}}) = \frac{1}{n\times m}\sum_{i=1}^m\sum_{j=1}^n{x_{ij}}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any way to get the standard deviation within the row-wise vector of means without having the original matrix?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;M = \left(\begin{matrix}&#xD;&#xA;1 &amp;amp; 2 &amp;amp; 3\\&#xD;&#xA;7 &amp;amp; 5 &amp;amp; 4\\&#xD;&#xA;8 &amp;amp; 2 &amp;amp; 3\\&#xD;&#xA;5 &amp;amp; 2 &amp;amp; 4&#xD;&#xA;\end{matrix}\right)&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;(\bar{x}_{*j}) = \left(\begin{matrix}5.25 &amp;amp; 2.75 &amp;amp; 3.5\end{matrix}\right)&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;(s_{*j}) = \left(\begin{matrix} 2.68 &amp;amp; 1.29 &amp;amp; 0.5\end{matrix}\right)&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;(\bar{x}_{i*}) = \left(\begin{matrix} 2\\ 5.33 \\ 4.33 \\ 3.66\end{matrix}\right)&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The mean of $\{2, 5.33, 4.33, 3.66\}$ is $3.83$ (also the mean of ${5.25, 2.75, 3.5}$).&#xA;The standard deviation of $\{2, 5.33, 4.33, 3.66\}$ is $1.21$. Would there be any way of calculating this standard deviation, knowing $(\bar{x}_{*j})$, $(s_{*j})$, but without knowing $M$ nor $(\bar{x}_{i*})$?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there anything else that could have been calculated &quot;column-wise&quot; (that is, independently for each column, thus excluding the &quot;row-wise&quot; means themselves) that would help find out the (preferably sample) standard deviation of the &quot;row-wise&quot; means?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you.&lt;/p&gt;&#xA;" OwnerUserId="4607" LastActivityDate="2011-05-13T22:28:20.920" Title="Standard deviation of means over two dimensions" Tags="&lt;standard-deviation&gt;&lt;mean&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="10785" PostTypeId="2" ParentId="10784" CreationDate="2011-05-13T21:30:54.460" Score="1" ViewCount="" Body="&lt;p&gt;No, there isn't. In essence, having the columns-wise means is equivalent to having the sums along the columns. With that, you cannot get the sums along the rows. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, to recover the sum along the rows you'll need to recover the full matrix. Knowing the other sum (in your example, $(5.25 2.75 3.5)$ ) is not enough. This can be seen in matrix form. Essentially, you have $A U = M$  where A is the data matrix, U is a row column with ones (or 1/N) and M is the sums or means along the colums or rows. If you could augment U by adding other operations, so that U is square, you could invert U and recover A. For example, if you had not only the sum of the four rows, but the sum of the first three rows, and the first two rows ... etc... you'd have 4 vector of 3 components, and then you could recover your matrix.&lt;/p&gt;&#xA;" OwnerUserId="2546" LastActivityDate="2011-05-13T21:30:54.460" />
  <row Id="10786" PostTypeId="2" ParentId="10782" CreationDate="2011-05-13T21:43:50.037" Score="5" ViewCount="" Body="&lt;p&gt;When you don't know the functional form ahead of time (which is a common setting) and you have no reason to assume it's linear, it's best to be flexible.  If there were more levels of treatment you could fit a quadratic or restricted cubic spline shape, for example.  For only 4 levels it may be best to assign 3 degrees of freedom to treatment using 3 dummy variables.&lt;/p&gt;&#xA;" OwnerUserId="4253" LastActivityDate="2011-05-13T21:43:50.037" CommentCount="5" />
  <row Id="10787" PostTypeId="1" CreationDate="2011-05-13T21:49:07.063" Score="2" ViewCount="73" Body="&lt;p&gt;I am exploring different classification methods for a project I am working on, and am interested in trying Random Forests. I am trying to educate myself as I go along, and would appreciate any help provided by the CV community.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have split my data into training/test sets. From experimentation with random forests in R (using the randomForest package), I have been having trouble with a high misclassification rate for my smaller class. I have read &lt;a href=&quot;http://www.stat.berkeley.edu/tech-reports/666.pdf&quot; rel=&quot;nofollow&quot;&gt;this paper&lt;/a&gt; concerning the performance of random forests on imbalanced data, and the authors presented two methods with dealing with class imbalance when using random forests.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;1. Weighted Random Forests&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;2. Balanced Random Forests&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The R package does not allow weighting of the classes (from the R help forums, I have read  the classwt parameter is not performing properly and is scheduled as a future bug fix), so I am left with option 2. I am able to specify the number of objects sampled from each class for each iteration of the random forest. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I feel uneasy about setting equal sample sizes for random forests, as I feel like I would be losing too much information about the larger class leading to poor performance with future data. The misclassification rates when downsampling the larger class has shown to improve, but I was wondering if there were other ways to deal with imbalanced class sizes in random forests?&lt;/p&gt;&#xA;" OwnerUserId="2252" LastActivityDate="2011-05-13T21:49:07.063" Title="For classification with Random Forests in R, how should one adjust for imbalanced class sizes?" Tags="&lt;r&gt;&lt;machine-learning&gt;&lt;random-forest&gt;" FavoriteCount="2" />
  <row Id="10788" PostTypeId="1" CreationDate="2011-05-13T22:20:55.140" Score="3" ViewCount="47" Body="&lt;p&gt;Hello everyone,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was wondering if you can share your experiences on what you feel is the best method to test lead / lag relationships between I(1) time series variables (i.e stock prices) and advantages and disadvantages of your proposed method(s). Also if you have links to academic papers that further describe these methods I would greatly appreciate them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have read several papers that speak of VECMs, IRSUR, simple OLS, threshold regression etc..   but I'm not sure which to use for my study. I am trying to establish lead/lag relationships in intraday stock data returns (using 1 minute price time series) between stock prices of ~50 companies in one particular sector. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Appreciate the help.&lt;/p&gt;&#xA;" OwnerUserId="4338" LastActivityDate="2011-05-13T22:20:55.140" Title="Methods to best test lead/lag relationships" Tags="&lt;regression&gt;&lt;least-squares&gt;" CommentCount="1" FavoriteCount="2" />
  <row Id="10789" PostTypeId="1" AcceptedAnswerId="10791" CreationDate="2011-05-13T22:24:15.883" Score="7" ViewCount="143" Body="&lt;p&gt;I'm having problems understanding the concept of a random variable as a function.  I understand the mechanics (I think) but I do not understand the motivation...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say $(\Omega, B, P) $ is a probability triple, where $\Omega = [0,1]$, $B$ is the borel-$\sigma$-algebra on that interval and $P$ is the regular lebesgue measure.  Let $X$ be a random variable from $B$ to $\{1,2,3,4,5,6\}$ such that $X([0,1/6)) = 1$, $X([1/6,2/6)) = 2$, ..., $X([5/6,1]) = 6$, so $X$ has a discrete uniform distribution on the values 1 through 6.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;That's all good, but I do not understand the necessity of the original probability triple... we could have directly constructed something equivalent as $(\{1,2,3,4,5,6\}, S, P_x)$ where $S$ is all the appropriate $\sigma$-algebra of the space, and $P_x$ is a measure that assigns to each subset the measure (# of elements)/6.  Also, the choice of $\Omega=[0,1]$ was arbitrary-- it could've been $[0,2]$, or any other set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my question is, why bother constructing an arbitrary $\Omega$ with a $\sigma$-algebra and a measure, and define a random variable as a map from the $\sigma$-algebra to the real line?  &lt;/p&gt;&#xA;" OwnerUserId="4608" LastActivityDate="2011-05-14T17:19:49.897" Title="Why are random variables defined as functions?" Tags="&lt;random-variable&gt;" AnswerCount="2" CommentCount="1" FavoriteCount="1" />
  <row Id="10790" PostTypeId="2" ParentId="10784" CreationDate="2011-05-13T22:28:20.920" Score="1" ViewCount="" Body="&lt;p&gt;You can alter the standard deviation of the row-means by suitably reordering the the elements of each column, while leaving the statistics for each column unchanged.  For example, with &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;M_2 = \left(\begin{matrix}&#xD;&#xA;1 &amp;amp; 2 &amp;amp; 3\\&#xD;&#xA;5 &amp;amp; 2 &amp;amp; 3\\&#xD;&#xA;7 &amp;amp; 2 &amp;amp; 3\\&#xD;&#xA;8 &amp;amp; 5 &amp;amp; 4&#xD;&#xA;\end{matrix}\right)&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;you will get the same column statistics, but the row means will be 2.000, 3.333, 4.333, and 5.667, with a standard deviation of about 1.55, rather than the original 2.000, 5.333, 4.333, 3.667 with a standard deviation about 1.40.  So you cannot derive statistics for the rows (as opposed to the whole matrix) from the statistics for the columns.&lt;/p&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-05-13T22:28:20.920" CommentCount="1" />
  <row Id="10791" PostTypeId="2" ParentId="10789" CreationDate="2011-05-13T22:47:25.717" Score="5" ViewCount="" Body="&lt;p&gt;If you are wondering why all this machinery is used when something much simpler could suffice--you are right, for most common situations.  However, the measure-theoretic version of probability was developed by Kolmogorov for the purpose of establishing a theory of such generality that it could handle, in some cases, very abstract and complicated probability spaces.  In fact, Kolmogorov's measure theoretic foundations for probability ultimately allowed probabilistic tools to be applied far beyond their original intended domain of application into areas such as harmonic analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At first it does seem more straightforward to skip any &quot;underlying&quot; $\sigma$-algebra $\Omega$, and to simply assign probability masses to the events comprising the sample space directly, as you have proposed.  Indeed, probabilists effectively do the same thing whenever they choose to work with the &quot;induced-measure&quot; on the sample space defined by $P \circ X^{-1}$.  However, things start getting tricky when you start getting into infinite dimensional spaces.  Suppose you want to prove the Strong Law of Large Numbers for the specific case of flipping fair coins (that is, that the proportion of heads tends arbitrarily closely to 1/2 as the number of coin flips goes to infinity).  You could attempt to construct a $\sigma$-algebra on the set of infinite sequences of the form $(H,T,H,...)$.  But here can find that it is much more convenient to take the underlying space to be $\Omega = [0,1)$; and then use the binary representations of real numbers (e.g. $0.10100...$) to represent sequences of coin flips (1 being heads, 0 being tails.)  An illustration of this very example can be found in the first few chapters of Billingsley's &lt;em&gt;Probability and Measure&lt;/em&gt;.&lt;/p&gt;&#xA;" OwnerUserId="3567" LastActivityDate="2011-05-13T22:47:25.717" CommentCount="9" />
  <row Id="10792" PostTypeId="2" ParentId="10768" CreationDate="2011-05-13T22:55:51.613" Score="5" ViewCount="" Body="&lt;p&gt;Q1) Yes. It's also useful for generating variables which are stochastically ordered; you can see this from @whuber's pretty picture :). $\alpha&amp;gt;1$ swaps the stochastic order.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That it's a valid cdf is just a matter of verifying the requisite conditions: $F_z(z)^\alpha$ has to be &lt;a href=&quot;http://en.wikipedia.org/wiki/C%C3%A0dl%C3%A0g&quot; rel=&quot;nofollow&quot;&gt;cadlag&lt;/a&gt;, nondecreasing and limit to $1$ at infinity and $0$ at negative infinity. $F_z$ has these properties so these are all easy to show.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Q2) Seems like it would be pretty difficult analytically, unless $F_Z$ is special&lt;/p&gt;&#xA;" OwnerUserId="26" LastActivityDate="2011-05-13T22:55:51.613" CommentCount="12" />
  <row Id="10793" PostTypeId="2" ParentId="4909" CreationDate="2011-05-13T23:07:29.717" Score="1" ViewCount="" Body="&lt;p&gt;The empirical CDF is just one estimator for the CDF. It's consistent, converges pretty quickly in general, and is dead simple to understand. If you want something fancier you could certainly get a kernel density estimate for the PDF and integrate it to get another estimate for the CDF, which would do some kind of interpolation as you suggest. But if it ain't broke....&lt;/p&gt;&#xA;" OwnerUserId="26" LastActivityDate="2011-05-13T23:07:29.717" CommentCount="10" />
  <row Id="10794" PostTypeId="2" ParentId="10782" CreationDate="2011-05-13T23:12:11.193" Score="5" ViewCount="" Body="&lt;p&gt;Why not look at a bivariate X-Y scatterplot in advance of running a regression.  That'll show you the shape of the line or curve, especially if you have software that can give a lowess/loess fit (locally weighted smoothed fit).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As to interpretation, it'll no doubt be easier for you than for your audience, but if you do have a quadratic fit, then for each increment of one on the sq. rt. of X, Y will change by b, your coefficient.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you really only have 4 levels of X, I agree with @Frank's point and would add that you might make your job easier by running an ANOVA instead of regression.  Or, some software makes it easy to combine continuous and categorical predictors, fusing regression and anova into a general linear model without the need for dummy variables (if you use SPSS, search 'Unianova').&lt;/p&gt;&#xA;" OwnerUserId="2669" LastActivityDate="2011-05-13T23:12:11.193" CommentCount="8" />
  <row Id="10795" PostTypeId="1" CreationDate="2011-05-14T01:13:14.647" Score="6" ViewCount="241" Body="&lt;p&gt;I was wondering whether anyone could point me to some references that discuss the interpretation of the elements of the inverse covariance matrix, also known as the concentration matrix or the precision matrix.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have access to Cox and Wermuth's &lt;em&gt;Multivariate Dependencies&lt;/em&gt;, but what I'm looking for is an interpretation of each element in the inverse matrix.  Wikipedia &lt;a href=&quot;http://en.wikipedia.org/wiki/Covariance_matrix&quot; rel=&quot;nofollow&quot;&gt;states&lt;/a&gt;: &quot;The elements of the precision matrix have an interpretation in terms of partial correlations and partial variances,&quot; which leads me to &lt;a href=&quot;http://en.wikipedia.org/wiki/Partial_correlation&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; page.  Is there an interpretation without using linear regression?  IE, in terms of covariances or geometry?&lt;/p&gt;&#xA;" OwnerUserId="4610" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-16T20:47:12.393" LastActivityDate="2011-05-29T13:02:11.333" Title="How to interpret an inverse covariance or precision matrix?" Tags="&lt;interpretation&gt;&lt;variance-covariance&gt;" AnswerCount="2" CommentCount="3" />
  <row Id="10796" PostTypeId="2" ParentId="10778" CreationDate="2011-05-14T05:15:57.400" Score="2" ViewCount="" Body="&lt;p&gt;The phrase &quot;Population&quot; is an abstract concept you use to define what type of question you want to answer.  You could consider the &quot;population of ads&quot; or the &quot;population of clicks&quot; - they are just two different forms of inference (on about ads, one about clicks).  I would suggest that in either case the notion of &quot;population&quot; is not helpful to you from the way you have asked the question.  Suppose I write two hypothesis:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\begin{array}{c c}&#xD;&#xA;H_{1}:\text{Ad1 is more effective at getting clicks and forms filled out}&#xD;&#xA;\\&#xD;&#xA;H_{2}:\text{Ad2 is more effective at getting clicks and forms filled out}&#xD;&#xA;\end{array}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I also write out the data, denoted $D$ in your table, and your prior information by $I$.  Then we calculate:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(H_{1}|D,I)=P(H_{1}|I)\frac{P(D|H_{1},I)}{P(D|I)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(H_{1}|I)$ is the prior probability - how plausible is $H_{1}$ without considering the data?  Note that because you only have two ads, then their probabilities must sum to 1, under this prior information.  The main quantity to calculate is the likelihood $P(D|H_{1},I)$ - how plausible is the data you actually observed given that $H_{1}$ is true?  $P(D|I)$ is sometimes called the evidence, and is usually not explicitly calculated, but divided out by considering odds ratios $\frac{P(H_{1}|D,I)}{P(H_{2}|D,I)}$.  It can be thought of as asking the question how plausible is the data you actually observed, regardless of the hypothesis in question? (&quot;does any hypothesis being considered explain these data well?&quot;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You don't need a notion of the population to answer the question you have asked This can be seen quite clearly from the absence of such a notion in your question &lt;em&gt;which Ad was more effective at getting more clicks and forms filled out?&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've personally decided to call this &quot;the golden rule&quot; for inference using probability theory as extended logic.  You write a proposition which if true or false would answer your question (such as $H_{1}$ and $H_{2}$).  Then you simply calculate the probability of that proposition, conditional on whatever evidence you have (i.e. what you know), marginalising (or averaging) out what is unknown.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-05-14T05:15:57.400" />
  <row Id="10797" PostTypeId="2" ParentId="10789" CreationDate="2011-05-14T07:07:41.470" Score="2" ViewCount="" Body="&lt;p&gt;The issues regarding $\sigma$-algebras are mathematical subtleties, that do not really explain why or if we need a &lt;strong&gt;background space&lt;/strong&gt;. Indeed, I would say that there is no compelling evidence that the background space is a necessity. For any probabilistic setup &#xA;$(E, \mathbb{E}, \mu)$ where $E$ is the sample space, $\mathbb{E}$ the $\sigma$-algebra  and $\mu$ a probability measure, the interest is in $\mu$, and there is no abstract reason that we want $\mu$ to be the image measure of a measurable map $X : (\Omega, \mathbb{B}) \to (E, \mathbb{E})$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the use of an abstract background space gives &lt;strong&gt;mathematical convenience&lt;/strong&gt; that makes many results appear more natural and intuitive. The objective is always to say something about $\mu$, the &lt;em&gt;distribution&lt;/em&gt; of $X$, but it may be easier and more clearly expressed in terms of $X$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;An example is given by the central limit theorem. If $X_1, \ldots, X_n$ are i.i.d. real valued with mean $\mu$ and variance $\sigma^2$ the CLT says that &#xA;$$P\left(\frac{\sqrt{n}}{\sigma} \left(\frac{1}{n}\sum_{i=1}^n X_i - \xi\right) \leq x \right) \to \Phi(x)$$&#xA;where $\Phi$ is the distribution function for the standard normal distribution. &#xA;If the distribution of $X_i$ is $\mu$ the corresponding result in terms of the measure reads &#xA;$$\rho_{\sqrt{n}/\sigma} \circ \tau_{\xi} \circ \rho_{1/n}(\mu^{*n})((-\infty, x]) \to \Phi(x)$$&#xA;Some explanation of the terminology is needed. By $\mu^{*n}$ we mean the $n$-times convolution of $\mu$ (the distribution of the sum). The functions $\rho_c$ are the linear functions $\rho_c(x) = cx$ and $\tau_{\xi}$ is the translation $\tau_{\xi}(x) = x - \xi$. &#xA;One could probably get used to the second formulation, but it does a good job at hiding what it is all about. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What seems to be the issue is that the &lt;em&gt;arithmetic transformations&lt;/em&gt; involved in the CLT are quite clearly expressed in terms of random variables but they do not translate so well in terms of the measures. &lt;/p&gt;&#xA;" OwnerUserId="4376" LastEditorUserId="4376" LastEditorDisplayName="" LastEditDate="2011-05-14T17:19:49.897" LastActivityDate="2011-05-14T17:19:49.897" CommentCount="2" />
  <row Id="10798" PostTypeId="1" AcceptedAnswerId="10799" CreationDate="2011-05-14T07:58:36.470" Score="6" ViewCount="49" Body="&lt;p&gt;&lt;strong&gt;Direct Question:&lt;/strong&gt; Are there any measures of auto-correlation for a sequence of observations of an (unordered) categorical variable?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Background:&lt;/strong&gt;  I'm using MCMC to sample from a categorical variable and I'd like a measure of how well the sampling method I've developed is mixing across the posterior distribution.  I'm familiar with acf plots and auto-correlation for continuous variables, but I've been stuck looking at the transition probability matrix for this categorical variable...  Any thoughts?&lt;/p&gt;&#xA;" OwnerUserId="1499" LastEditorUserId="1499" LastEditorDisplayName="" LastEditDate="2011-05-14T08:04:11.110" LastActivityDate="2011-05-16T05:34:30.250" Title="Measures of autocorrelation in categorical values of a Markov Chain?" Tags="&lt;categorical-data&gt;&lt;mcmc&gt;&lt;autocorrelation&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="10799" PostTypeId="2" ParentId="10798" CreationDate="2011-05-14T08:26:49.843" Score="5" ViewCount="" Body="&lt;p&gt;You can always choose one or several real valued functions of the categorical variables and look at the auto-correlation for the resulting sequence(s). You can, for instance, consider indicators of some subsets of the variables. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, if I understood your question correctly, your sequence is obtained by an MCMC algorithm on the discrete space. In that case, it may be more interesting to look directly at the convergence rate for the Markov chain. Chapter 6 in &lt;a href=&quot;http://books.google.com/books?id=KF0LgxRCgQsC&amp;amp;printsec=frontcover&amp;amp;dq=bremaud+markov+chains&amp;amp;hl=en&amp;amp;ei=nDjOTcj5KZS-sAPavPizCw&amp;amp;sa=X&amp;amp;oi=book_result&amp;amp;ct=result&amp;amp;resnum=1&amp;amp;ved=0CEUQ6AEwAA#v=onepage&amp;amp;q=bremaud%20markov%20chains&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;this book&lt;/a&gt; by Brémaud treats this in details. The size of the second largest absolute value of the eigenvalues determines the convergence rate of the matrix of transition probabilities and thus the mixing of the process.  &lt;/p&gt;&#xA;" OwnerUserId="4376" LastEditorUserId="4376" LastEditorDisplayName="" LastEditDate="2011-05-16T05:34:30.250" LastActivityDate="2011-05-16T05:34:30.250" />
  <row Id="10800" PostTypeId="1" CreationDate="2011-05-14T08:53:36.253" Score="5" ViewCount="125" Body="&lt;p&gt;Hi guys and gals, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm just having a look through Hadley's very excellent book about his ggplot2 R package. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;He has some code to remove a linear trend in the diamonds dataset, like so:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;d &amp;lt;- subset(diamonds, carat &amp;lt; 2.5 &amp;amp; rbinom(nrow(diamonds), 1, 0.2) == 1)&#xA;d$lcarat &amp;lt;- log10(d$carat)&#xA;d$lprice &amp;lt;- log10(d$price)&#xA;&#xA;detrend &amp;lt;- lm(lprice ~ lcarat, data = d)&#xA;d$lprice2 &amp;lt;- resid(detrend)&#xA;&#xA;qplot(lcarat, lprice, data = d)&#xA;qplot(lcarat, lprice2, data = d)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Produces these graphs&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unadjusted...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.stack.imgur.com/OV174.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Detrended...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.stack.imgur.com/JWxiC.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd like to see what the actual values of lprice would be without the effect of lcarat. Plotting residuals vs lcarat shows the right shape, but the points are shifted toward y = 0 (look at the range of the y-axis units).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To get what I want, does it make sense to simply plot residuals + mean(lprice)? i.e. shift the previous graph up by mean(lprice).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;qplot(lcarat, lprice2 + mean(lprice), data = d)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.stack.imgur.com/hrrDr.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Does it make sense to do this? Is there a name for what I'm trying to do?&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="827" LastEditorUserId="827" LastEditorDisplayName="" LastEditDate="2011-05-14T23:10:52.190" LastActivityDate="2011-05-14T23:10:52.190" Title="How do I display detrended data from a linear regression?" Tags="&lt;data-visualization&gt;" AnswerCount="1" CommentCount="6" FavoriteCount="1" />
  <row Id="10801" PostTypeId="2" ParentId="10795" CreationDate="2011-05-14T09:07:25.183" Score="0" ViewCount="" Body="&lt;p&gt;Covariance matrix can represent relation between all variables while inverse covariance, shoes the relation of element with their neighbors (as wikipedia said partial/pair wise relation). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I borrow the following example from &lt;a href=&quot;http://videolectures.net/gpip06_mackay_gpb/&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; in 24:10, imagine 5 masses are connected together and vowelling around with 6 springs, covariance matrix would contain correlation of all masses, if one goes right, others can also goes right. but the inverse covariance matrix shoes the relation of those masses that are connected by same spring (neighbors) and it contains many zeros and its not necessary positive.&lt;/p&gt;&#xA;" OwnerUserId="4581" LastEditorUserId="4581" LastEditorDisplayName="" LastEditDate="2011-05-29T13:02:11.333" LastActivityDate="2011-05-29T13:02:11.333" CommentCount="2" />
  <row Id="10802" PostTypeId="2" ParentId="6883" CreationDate="2011-05-14T09:34:48.027" Score="0" ViewCount="" Body="&lt;p&gt;Download for free Spatial Econometrics TB by Le Sage. Nice and neat output may be obtained from therein. GT.&lt;/p&gt;&#xA;" OwnerUserId="4093" LastActivityDate="2011-05-14T09:34:48.027" CommentCount="1" />
  <row Id="10803" PostTypeId="2" ParentId="6883" CreationDate="2011-05-14T11:34:41.720" Score="0" ViewCount="" Body="&lt;p&gt;simply without using any toolbox box use &quot;polyfit&quot; and in consequence &quot;polyval&quot; function,&#xA;here is the example :&#xA;X could you you input data, and Y you response value or target value &#xA;You can make a n-degree polynomial by :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  p = polyfit(x,y,n)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and fit the polynomial and get the interest output yd, based on fitted value.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  yd = polyval(p,xd);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;the plot the data&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  plot(x,y)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you check these functions help document, you would get more comprehensive information. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also &lt;a href=&quot;http://www.uni-bonn.de/~mbrei/Data/regressionMatlab.pdf&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; documents talks about different types of regression and their implementation in Matlab &lt;/p&gt;&#xA;" OwnerUserId="4581" LastActivityDate="2011-05-14T11:34:41.720" />
  <row Id="10804" PostTypeId="2" ParentId="10800" CreationDate="2011-05-14T12:49:02.853" Score="0" ViewCount="" Body="&lt;p&gt;As for me, it is terribly confusing, especially while you can do much simpler thing -- calculate &lt;code&gt;price/carat&lt;/code&gt; to get a price of one carat, which would be way easier to interpret. &lt;/p&gt;&#xA;" OwnerUserId="88" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-14T16:13:40.960" LastActivityDate="2011-05-14T16:13:40.960" CommentCount="4" />
  <row Id="10805" PostTypeId="1" CreationDate="2011-05-14T14:05:14.403" Score="2" ViewCount="30" Body="&lt;p&gt;Hello,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there someone who knows an econmetric model to describe count datas with an upper bound? Thank you! &lt;/p&gt;&#xA;" OwnerUserId="4496" LastActivityDate="2011-05-14T14:05:14.403" Title="Count data regression for data with upper bound" Tags="&lt;poisson&gt;&lt;count-data&gt;" CommentCount="1" />
  <row Id="10806" PostTypeId="1" CreationDate="2011-05-14T15:38:30.307" Score="7" ViewCount="111" Body="&lt;p&gt;In practical application, I have witnessed often the following practice. One observes a pair  $(x_t, y_t)$ over time. Under the assumption that they are linearly related, we regress one against the other using geometric weights rather than uniform ones, i.e., the OLS minimizes &#xA;$$\sum_{t=0}^\infty k^{t} (y_{T-t}- a x_{T-t}-b)^2$$&#xA;for some $k\in (0,1)$. This is very intuitive: we weight less observations far in the past. Compared to a &quot;boxcar&quot; weighting scheme, it has also the advantage of producing estimates that are changing smoothly over time, because observations do not fall abruptly off the observation window. However, I wonder if there is a &lt;em&gt;probabilistic&lt;/em&gt; model underlying the relationship between $x_t$ and $y_t$ that justifies this choice.  &lt;/p&gt;&#xA;" OwnerUserId="30" LastEditorUserId="30" LastEditorDisplayName="" LastEditDate="2011-05-15T02:21:41.113" LastActivityDate="2011-05-16T08:33:33.177" Title="Justification for using geometric weights in linear regression" Tags="&lt;regression&gt;&lt;least-squares&gt;" AnswerCount="2" CommentCount="3" />
  <row Id="10807" PostTypeId="1" AcceptedAnswerId="10808" CreationDate="2011-05-14T19:05:37.203" Score="4" ViewCount="133" Body="&lt;p&gt;In R, what is the difference between:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;if(x&amp;gt;2 &amp;amp; x&amp;lt;3) ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;if(x&amp;gt;2 &amp;amp;&amp;amp; x&amp;lt;3) ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Similarly:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;if(x&amp;lt;2 | x&amp;gt;3) ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;if(x&amp;lt;2 || x&amp;gt;3) ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2310" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-15T16:59:09.517" LastActivityDate="2011-05-15T16:59:09.517" Title="Using and/or operators in R" Tags="&lt;r&gt;" AnswerCount="1" />
  <row Id="10808" PostTypeId="2" ParentId="10807" CreationDate="2011-05-14T19:25:01.863" Score="9" ViewCount="" Body="&lt;p&gt;See &lt;code&gt;?&quot;&amp;amp;&quot;&lt;/code&gt;: the single version does elementwise comparisons (for when you are doing logical operations on two vectors of the same length, e.g. if in your example x&amp;lt;-c(1.5,3.5). The other one works just like C++'s or java's &amp;amp;&amp;amp;: it only looks at the first element of each vector (this is typically an unexpected downside), but in a typically better performing way: it looks from left to right, and as soon as at least one of the values is false, it knows not to look at the rest anymore.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So if you know, in your examples x&amp;lt;-6 (in any case, just one value), you're better off using &amp;amp;&amp;amp;, otherwise, always use &amp;amp;.&lt;/p&gt;&#xA;" OwnerUserId="4257" LastActivityDate="2011-05-14T19:25:01.863" CommentCount="1" />
  <row Id="10809" PostTypeId="2" ParentId="10806" CreationDate="2011-05-14T19:26:33.187" Score="6" ViewCount="" Body="&lt;p&gt;&quot;Linearly related&quot; usually means &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$y_t = a x_t + b + \varepsilon_t$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;for constant $a$, $b$ and iid random errors $\varepsilon_t$, $t=0,1,\ldots,T$.  One reason one would make an exponentially weighted OLS estimate is the suspicion that $a$ and $b$ might themselves be (slowly) varying with time, too.  Thus we really think the correct model is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$y_t = \alpha(t) x_t + \beta(t) + \varepsilon_t$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;for unknown &lt;em&gt;functions&lt;/em&gt; $\alpha(t)$ and $\beta(t)$ which vary slowly (if at all) over time and we're interested in estimating their current values, $a = \alpha_T$ and $b = \beta_T$.  Let's assume these functions are smooth, so we can apply Taylor's Theorem.  This asserts that &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\alpha(t) = \alpha(T) + \alpha&amp;#39;(t_{\alpha,t})(t-T)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;for some $t_{\alpha,t}, 0 \le t_{\alpha,t} \lt T$, and similarly for $\beta(t)$.  We think of $a$ and $b$ as being the most recent values, $\alpha_T$ and $\beta_T$, respectively.  Use this to re-express the residuals:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$y_t - (a x_t + b) = \alpha&amp;#39;(t_{\alpha,t})(t-T)x_t + \beta&amp;#39;(t_{\beta,t})(t-T) + \varepsilon_t\text{.}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now a lot of hand-waving needs to occur.  We will consider the entire right hand side to be random.  Its variance is that of $\varepsilon_t$ plus $x_t^2(t-T)^2$ times the variance of $\alpha&amp;#39;(t_{\alpha,t})$ plus $(t-T)^2$ times the variance of $\beta&amp;#39;(t_{\beta,t})$.  Those two variances are completely unknown, but (&lt;em&gt;abracadabra&lt;/em&gt;) let's think of them as resulting from some kind of (stochastic) process in which &lt;em&gt;possibly systematic&lt;/em&gt; (not random, but still unknown) &quot;errors&quot; or &quot;variations&quot; are accumulated from one time to the other.  This would suggest an &lt;em&gt;exponential&lt;/em&gt; change in those variances over time.  Now just simplify the explicit (but essentially useless) expression for the right hand side, and absorb the quadratic terms $(t-T)^2$ into the exponential (since we're waving our hands so wildly about anyway), to obtain&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$y_t - (a x_t + b) = \delta_t$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;with the variance of $\delta_t$ equal to $\exp(\kappa(t-T))$ for some constant $\kappa$.  Ignoring possible temporal correlations among the $\delta_t$ and assuming they have Normal distributions gives a log likelihood for the data proportional to&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\sum_{t=0}^{T} k^{-t} (y_{T-t}- a x_{T-t}-b)^2$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(plus an irrelevant constant depending only on $k$) with $k = \exp{\kappa}$.  The exponentially weighted OLS procedure therefore maximizes the likelihood, &lt;em&gt;assuming we know the value of $k$&lt;/em&gt; (kind of like a profile likelihood procedure).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Although this entire derivation clearly is fanciful, it does show how, and approximately to what degree, the exponential weighting attempts to cope with possible changes in the linear parameters over time.  It relates the parameter $k$ to the temporal rate of change of those parameters.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-05-15T14:10:12.053" LastActivityDate="2011-05-15T14:10:12.053" CommentCount="10" />
  <row Id="10810" PostTypeId="1" CreationDate="2011-05-14T21:23:24.490" Score="4" ViewCount="65" Body="&lt;p&gt;Who created sampling distributions? I have looked everywhere and I'm writing a paper, but, so far, all I've been able to come up with is the theory and the definition. Please help me find the who, what, when,and where.&lt;/p&gt;&#xA;" OwnerUserId="4615" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-05-15T00:19:29.613" LastActivityDate="2011-05-15T00:19:29.613" Title="Sampling distributions" Tags="&lt;distributions&gt;&lt;sampling&gt;&lt;history&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="10811" PostTypeId="2" ParentId="10810" CreationDate="2011-05-14T22:14:06.040" Score="5" ViewCount="" Body="&lt;p&gt;If you refer to the &lt;em&gt;term&lt;/em&gt; &quot;sampling distribution,&quot; information is hard to find.  But the &lt;em&gt;concept&lt;/em&gt; is crucial to Jakob Bernoulli's (Switzerland) recognition that there is a distinction between that distribution and the population distribution itself, leading to his formulation (and proof) of a &lt;a href=&quot;http://en.wikipedia.org/wiki/Law_of_large_numbers&quot; rel=&quot;nofollow&quot;&gt;Law of Large Numbers&lt;/a&gt;.  (The Wikipedia article attributes the first statement of such a law to &lt;a href=&quot;http://en.wikipedia.org/wiki/Gerolamo_Cardano&quot; rel=&quot;nofollow&quot;&gt;Cardano&lt;/a&gt; (Italy), who--among many other things--was an avid gambler and mathematician in the first half of the 16th century.)  Bernoulli's seminal work was published posthumously in 1713 as his &lt;em&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Ars_Conjectandi&quot; rel=&quot;nofollow&quot;&gt;ars conjectandi&lt;/a&gt;&lt;/em&gt; but likely was developed in the late 1600's in response to &lt;a href=&quot;http://en.wikipedia.org/wiki/Problem_of_points#Pascal_and_Fermat&quot; rel=&quot;nofollow&quot;&gt;pioneering work by Pascal and Fermat&lt;/a&gt; (France) as recounted by Christian Huyghens (Netherlands) in an influential (and amazingly brief) 1657 textbook, &lt;em&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Christiaan_Huygens#Probability_theory&quot; rel=&quot;nofollow&quot;&gt;de ratiociniis in ludo aleae&lt;/a&gt;&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For more background you can read a &lt;a href=&quot;http://onlinelibrary.wiley.com/doi/10.1111/j.1539-6924.2009.01271.x/full&quot; rel=&quot;nofollow&quot;&gt;brief accounting of some of this history&lt;/a&gt; I wrote in the form of a review of a Keith Devlin book, &lt;a href=&quot;http://literati.net/Devlin/DevlinReviews.htm&quot; rel=&quot;nofollow&quot;&gt;The Unfinished Game&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-05-14T22:14:06.040" CommentCount="2" />
  <row Id="10812" PostTypeId="2" ParentId="10089" CreationDate="2011-05-14T22:46:19.800" Score="0" ViewCount="" Body="&lt;p&gt;You could use &lt;a href=&quot;http://en.wikipedia.org/wiki/Kruskal%E2%80%93Wallis_one-way_analysis_of_variance&quot; rel=&quot;nofollow&quot;&gt;Kruskal&amp;ndash;Wallis test&lt;/a&gt; to perform a test for &#xA;$$&#xD;&#xA;H_0: m_1 = m_2 = m_3 \&amp;gt; , &#xD;&#xA;$$&#xA;where $m_i$ is the median of the $i$th group.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the $p$-value is $&amp;lt; 0.05$, you could conduct a post hoc test using Bonferroni correction. &#xA;Take a look at &lt;a href=&quot;http://yatani.jp/HCIstats/KruskalWallis&quot; rel=&quot;nofollow&quot;&gt;this website&lt;/a&gt;. It might be helpful.&lt;/p&gt;&#xA;" OwnerUserId="4559" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-05-15T00:18:32.437" LastActivityDate="2011-05-15T00:18:32.437" CommentCount="1" />
  <row Id="10813" PostTypeId="2" ParentId="10806" CreationDate="2011-05-15T00:38:21.330" Score="0" ViewCount="" Body="&lt;p&gt;I think in that you actually mean $k^{t}$ as your weight, or that $k&amp;gt;1$.  If $0&amp;lt;k&amp;lt;1$ and we take $k^{-t}$ as the weight then $k^{-\infty}=\infty$.  So this actually weights the present observation the least.  For example, if we take $k=0.5$ then $k^{0}=1,\;k^{-1}=2,\;k^{-2}=4,\dots,k^{-20}\approx 10^{6}$, and so on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is just stating something that you know about the how the variance changes with each observation (it gets bigger as you mover further backward in time from time $T$):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$(y_{T-t}|x_{T-t},a,b,k,s) \sim Normal(ax_{T-t}+b,s^{2}k^{-t})$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Denoting $Y\equiv\{y_{T},y_{T-1},\dots,y_{1}\}$ and $X\equiv\{x_{T},x_{T-1},\dots,x_{1}\}$ we have a joint log-likelihood of:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\log\left[p(Y|X,a,b,k,s)\right]=-\frac{1}{2}\left(T\log(2\pi s^{2} k^{-t})+\sum_{t=0}^{T-1}\frac{(y_{T-t}-ax_{T-t}-b)^{2}}{s^{2}k^{-t}}\right)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So in order to get the maximum likelihood estimates of $a$ and $b$ you have the following objective function:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\sum_{t=0}^{T-1}k^{t}(y_{T-t}-ax_{T-t}-b)^{2}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which is the one you seek&lt;/p&gt;&#xA;" OwnerUserId="2392" LastEditorUserId="2392" LastEditorDisplayName="" LastEditDate="2011-05-16T08:33:33.177" LastActivityDate="2011-05-16T08:33:33.177" CommentCount="2" />
  <row Id="10814" PostTypeId="2" ParentId="10768" CreationDate="2011-05-15T00:52:55.247" Score="6" ViewCount="" Body="&lt;p&gt;I like the other answers, but nobody has mentioned the following yet.  The event $\{U \leq t,\ V\leq t \}$  occurs if and only if $\{\mathrm{max}(U,V)\leq t\}$, so if $U$ and $V$ are independent and $W = \mathrm{max}(U,V)$, then $F_{W}(t) = F_{U}(t)*F_{V}(t)$ so for $\alpha$ a positive integer (say, $\alpha = n$) take $X = \mathrm{max}(Z_{1},...Z_{n})$ where the $Z$'s are i.i.d. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For $\alpha = 1/n$ we can switcheroo to get $F_{Z} = F_{X}^n$, so $X$ would be that random variable such that the max of $n$ independent copies has the same distribution as $Z$ (and this would not be one of our familiar friends, in general).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The case of $\alpha$ a positive rational number (say, $\alpha = m/n$) follows from the previous since&#xA;$$&#xD;&#xA;\left(F_{Z}\right)^{m/n} = \left(F_{Z}^{1/n}\right)^{m}.&#xD;&#xA;$$  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For $\alpha$ an irrational, choose a sequence of positive rationals $a_{k}$ converging to $\alpha$; then the sequence $X_{k}$ (where we can use our above tricks for each $k$) will converge in distribution to the $X$ desired.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This might not be the characterization you are looking for, but it least gives some idea of how to think about $F_{Z}^{\alpha}$ for $\alpha$ suitably nice.  On the other hand, I'm not really sure how much nicer it can really get: you already have the CDF, so the chain rule gives you the PDF, and you can calculate moments till the sun sets...?  It's true that most $Z$'s won't have an $X$ that's familiar for $\alpha = \sqrt{2}$, but if I wanted to play around with an example to look for something interesting I might try $Z$ uniformly distributed on the unit interval with $F(z) = z$, $0&amp;lt;z&amp;lt;1$.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT:&lt;/strong&gt;  I wrote some comments in @JMS answer, and there was a question about my arithmetic, so I'll write out what I meant in the hopes that it's more clear.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;@cardinal correctly in the comment to @JMS answer wrote that the problem simplifies to &#xA;$$&#xD;&#xA;g^{-1}(y) = \Phi^{-1}(\Phi^{\alpha}(y)),&#xD;&#xA;$$&#xA;or more generally when $Z$ is not necessarily $N(0,1)$, we have&#xA;$$&#xD;&#xA;x = g^{-1}(y) = F^{-1}(F^{\alpha}(y)).&#xD;&#xA;$$&#xA;My point was that when $F$ has a nice inverse function we can just solve for the function $y = g(x)$ with basic algebra.  I wrote in the comment that $g$ should be &#xA;$$&#xD;&#xA;y = g(x) = F^{-1}(F^{1/\alpha}(x)).&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's take a special case, plug things in, and see how it works.  Let $X$ have an Exp(1) distribution, with CDF&#xA;$$&#xD;&#xA;F(x) = (1 - \mathrm{e}^{-x}),\ x &amp;gt; 0,&#xD;&#xA;$$&#xA;and inverse CDF&#xA;$$&#xD;&#xA;F^{-1}(y) = -\ln(1 - y).&#xD;&#xA;$$&#xA;It is easy to plug everything in to find $g$;  after we're done we get&#xA;$$&#xD;&#xA;y = g(x) = -\ln \left( 1 - (1 - \mathrm{e}^{-x})^{1/\alpha} \right)&#xD;&#xA;$$&#xA;So, in summary, my claim is that if $X \sim \mathrm{Exp}(1)$ and if we define&#xA;$$&#xD;&#xA;Y = -\ln \left( 1 - (1 - \mathrm{e}^{-X})^{1/\alpha} \right),&#xD;&#xA;$$&#xA;then $Y$ will have a CDF which looks like&#xA;$$&#xD;&#xA;F_{Y}(y) = \left( 1 - \mathrm{e}^{-y} \right)^{\alpha}.&#xD;&#xA;$$&#xA;We can prove this directly (look at $P(Y \leq y)$ and use algebra to get the expression, in the next to the last step we need the Probability Integral Transform).  Just in the (often repeated) case that I'm crazy, I ran some simulations to double-check that it works, ... and it does.  See below.  To make the code easier I used two facts:&#xA;$$&#xA;\mbox{If $X \sim F$ then $U = F(X) \sim \mathrm{Unif}(0,1)$.} &#xA;$$&#xD;&#xA;$$&#xA;\mbox{If $U \sim \mathrm{Unif}(0,1)$ then $U^{1/\alpha} \sim \mathrm{Beta}(\alpha,1)$.} &#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The plot of the simulation results follows.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.stack.imgur.com/K6XFB.png&quot; alt=&quot;ECDF and F to the alpha&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The R code used to generate the plot (minus labels) is&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;n &amp;lt;- 10000; alpha &amp;lt;- 0.7&#xA;z &amp;lt;- rbeta(n, shape1 = alpha, shape2 = 1)&#xA;y &amp;lt;- -log(1 - z)&#xA;plot(ecdf(y))&#xA;f &amp;lt;- function(x) (pexp(x, rate = 1))^alpha&#xA;curve(f, add = TRUE, lty = 2, lwd = 2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The fit looks pretty good, I think?  Maybe I'm not crazy (this time)?&lt;/p&gt;&#xA;" OwnerUserId="1108" LastEditorUserId="1108" LastEditorDisplayName="" LastEditDate="2011-05-20T17:59:24.970" LastActivityDate="2011-05-20T17:59:24.970" CommentCount="6" />
  <row Id="10815" PostTypeId="2" ParentId="9685" CreationDate="2011-05-15T01:12:35.080" Score="0" ViewCount="" Body="&lt;p&gt;Hi,&#xA;I believe you could use the &quot;simultaneous confidence intervals&quot; for doing multiple comparisons. The reference is Agresti et al. 2008 Simultaneous confidence intervals for comparing binomial parameters. Biometrics 64 1270-1275. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could find the corresponding R code in &lt;a href=&quot;http://www.stat.ufl.edu/~aa/cda/software.html&quot; rel=&quot;nofollow&quot;&gt;http://www.stat.ufl.edu/~aa/cda/software.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="4559" LastActivityDate="2011-05-15T01:12:35.080" />
  <row Id="10816" PostTypeId="1" CreationDate="2011-05-15T05:31:38.173" Score="5" ViewCount="60" Body="&lt;p&gt;Hello all,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On page 9 in &lt;a href=&quot;http://jenni.uchicago.edu/Oxford2005/four_param_all_2005-08-07_csh.pdf&quot; rel=&quot;nofollow&quot;&gt;http://jenni.uchicago.edu/Oxford2005/four_param_all_2005-08-07_csh.pdf&lt;/a&gt;&#xA;ATE - the average treatment effect is the expected gain from participation in a program for a random individual. For example, evaluate the impact of going to college on wages.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Due to selection bias estimation steps are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Probit, to model probability of an individual going to college. Two different inverse Mills ratios are calculated.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;for those who did go to college, do OLS of wage on explanatory variables (like gender, etc.) and  do the same for those who did not go to college. For each OLS regression include an appropriate inverse Mills ratio obtained from step 1 as an additional explanatory variables.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;ATE on is the average of the difference in predicted values using parameter estimates for college and non-college groups.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;My questions are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;on step 3, there is no need to use parameter estimates on inverse Mills ratios used in prediction? I just drop these coefficients calculating the ATE.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;do I need to keep the variables in the OLS same across college and non-college groups? If I fit OLS for college and non-college groups different variables are going to be significant in explaining variation in income. So, when I calculate the ATE some parameter estimates will be zero. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I had decided to split independent variables into two sets, one for probit and another for OLS. In the OLS, if I use the inverse Mills ratios together with variables used in the Probit, there is high multicollinearity. Even if unbiased estimates are obtained in the presence of multicollinearity, I am worried about the prediction and wide confidence interval due to inflated standard errors.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Please let me know,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&#xA;Mermex&lt;/p&gt;&#xA;" OwnerUserId="4617" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-16T05:53:49.550" LastActivityDate="2011-05-16T06:11:15.550" Title="Heckman sample selection" Tags="&lt;regression&gt;&lt;econometrics&gt;&lt;probit&gt;&lt;heckit&gt;" AnswerCount="1" />
  <row Id="10817" PostTypeId="1" CreationDate="2011-05-15T05:58:33.960" Score="4" ViewCount="94" Body="&lt;p&gt;Hello all,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Have anyone tried modeling number of phone calls using OLS?&#xA;The dataset is number of calls per months for each customer's account.&#xA;The dependent variable is number of calls or average number of calls, and the explanatory variables are customer specific variables including number of purchases, total spend and so on....&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do you deal with zero callers? Only small proportion of customers actually call, 5%. I am attempting to build a predictive model, so I want to keep zero callers in the model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I do not believe that number of call is bounded or censored random variable? I thought zero number of calls is a true zero and there is to account for it? Do I need to use Tobit for the estimation here?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;M&lt;/p&gt;&#xA;" OwnerUserId="4617" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-16T05:51:19.287" LastActivityDate="2011-05-16T05:51:19.287" Title="Modeling number of phone calls with OLS" Tags="&lt;regression&gt;&lt;least-squares&gt;" AnswerCount="2" />
  <row Id="10818" PostTypeId="2" ParentId="10817" CreationDate="2011-05-15T10:35:51.817" Score="3" ViewCount="" Body="&lt;p&gt;Just from knowing that there are many zeros in the data, this would suggest to me that you use a zero-inflated poisson model - &lt;a href=&quot;http://en.wikipedia.org/wiki/Generalized_linear_model&quot; rel=&quot;nofollow&quot;&gt;a generalised linear model&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="656" LastActivityDate="2011-05-15T10:35:51.817" CommentCount="3" />
  <row Id="10819" PostTypeId="2" ParentId="10711" CreationDate="2011-05-15T11:44:02.263" Score="2" ViewCount="" Body="&lt;p&gt;This is a really, really interesting question (indeed I'm somewhat in love with the idea of modelling the stack exchange sites in general).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the issue of well-roundedness, one way to assess this would be through the tags that particular users tend to answer, and their distribution across sites. Examples may make this clearer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am a member on TeX, StackOverflow, CrossValidated and AskUbuntu. Now, I really only contribute to here and StackOverflow, and only about R on Stackoverflow. So,to define well roundedness I would look at a) the amount of tags which two sites have in common (to define similarity across sites) and the extent to which a user answers questions on sites which have little or no tags in common. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If, for instance, someone contributes to Python tags on StackOverflow and cooking, that person is more well-rounded than someone who is answering questions statistical software questions (for instance) on Overflow and stats questions here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope this is somewhat helpful.&lt;/p&gt;&#xA;" OwnerUserId="656" LastActivityDate="2011-05-15T11:44:02.263" CommentCount="1" />
  <row Id="10820" PostTypeId="2" ParentId="10817" CreationDate="2011-05-15T12:26:30.877" Score="2" ViewCount="" Body="&lt;p&gt;•   Since this is time series data you would be well advised to include some form of &quot;time variable&quot; in the model. This could be accomplished by including seasonal dummies and/or seasonal autoregressive structure. You might also have one or more Level Shifts and/or one or more trends in the data. You might have changes in parameters or error variance over time that might need to be incorporated. Incorporating predictor variables would be important making sure that correct contemporaneous and lag effects were treated. Additionally you might want to detect anomalies/pulses so that your model parameters were robust to them via Intervention Detection. In general this is  referred to as a Box-Jenkins Model with Causal Variables (ARMAX or Transfer Functions).You might want to Google &quot;regression vs box-jenkins&quot; to find out more about the whys and wherefores of incorporating time series structure into your model. Be careful about some web content that incorrectly positions Box-Jenkins as being non-causative. The univariate (single series approach) is called ARIMA Modelling. This approach is only suggested when you don't want to include predictor variables. Lots of incorrect textbooks and web sites don't make this clear as they assert things like “Box-Jenkins ignores information that might be contained in a structural regression model” &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example google &quot;difference between box-jenkins and regression&quot; and you will get some other hits on this topic. The first hit leads to a typical misrepresentation of what Box-Jenkins models are. For example &quot;Box-Jenkins ignores information that might be contained in a structural regression model&quot; is a half-truth as what is more correct is to say “Box-Jenkins without Causal Variables etc.”&lt;/p&gt;&#xA;" OwnerUserId="3382" LastEditorUserId="3382" LastEditorDisplayName="" LastEditDate="2011-05-15T14:41:24.497" LastActivityDate="2011-05-15T14:41:24.497" CommentCount="2" />
  <row Id="10821" PostTypeId="1" CreationDate="2011-05-15T13:17:55.733" Score="1" ViewCount="73" Body="&lt;p&gt;Hi,&#xA;I am trying to evaluate the effect of an intervention. I have a group-randomised rct design and have established that I have a design effect and therefore need to control for school. So far I have managed to restructure my data so that it is now vertical and I have a variable called 'time' which represents my pre and post time points. Although I have randomised effectively in terms of SES, my recruitment method (inviting parents to take part in a parenting program versus a questionnaire study) has resulted in a significantly more distressed Intervention group when reported by parents (but not when reported by the child).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have used the book &lt;em&gt;Multilevel and longitudinal Modeling with IBM SPSS&lt;/em&gt; by Heck, Thomas &amp;amp; Tabata, 2010, which is great, but I can not figure out how to run a simple two level repeated measures model. Does anyone know of a reference or is able to help me with syntax? I seem to get an error message with what I have used and don't know where to put school, my group variable, so I can control for it.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;MIXED SDQPT WITH Time Int_Cont&#xA;  /CRITERIA=CIN(95) MXITER(100) MXSTEP(5) SCORING(1) SINGULAR(0.000000000001) HCONVERGE(0,ABSOLUTE) LCONVERGE(0, ABSOLUTE) PCONVERGE(0.000001, ABSOLUTE)&#xA;/FIXED=Time Int_Cont Time*Int_Cont | SSTYPE(3)&#xA;/METHOD=REML&#xA;/PRINT=G  SOLUTION TESTCOV&#xA;/RANDOM=INTERCEPT Time | SUBJECT(CodeNumb) COVTYPE(UN)&#xA;/REPEATED=Time | SUBJECT(CodeNumb) COVTYPE(DIAG).&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Thanks for your help!!!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this it? I have established that autoregressive fits the model best (SCASPT is my measure of anxiety; Int_Cont is my grouping variable, and Time is 0=time 1 and 1= time 2): &lt;/p&gt;&#xA;&#xA;&lt;p&gt;MIXED SCASPT BY Int_Cont Time&#xA;  /CRITERIA=CIN(95) MXITER(100) MXSTEP(5) SCORING(1) SINGULAR(0.000000000001) HCONVERGE(0, &#xA;    ABSOLUTE) LCONVERGE(0, ABSOLUTE) PCONVERGE(0.000001, ABSOLUTE)&#xA;  /FIXED=Int_Cont Time Int_Cont*Time | SSTYPE(3)&#xA;  /METHOD=REML&#xA;  /PRINT=G  R SOLUTION TESTCOV&#xA;  /REPEATED=Time | SUBJECT(School*CodeNumb) COVTYPE(AR1).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would really appreciate if you know of a reference that would explain this in a really simple way for someone with very limited stats knowledge. I am finding the Heck reference a little too hard to follow. Thanks!&lt;/p&gt;&#xA;" OwnerUserId="4269" LastEditorUserId="4269" LastEditorDisplayName="" LastEditDate="2011-05-18T03:08:34.217" LastActivityDate="2011-05-18T03:08:34.217" Title="How do I run a Mixed Model repeated measures analysis in SPSS to evaluate intervention effect while controlling for school?" Tags="&lt;mixed-model&gt;&lt;spss&gt;&lt;repeated-measures&gt;" CommentCount="1" />
  <row Id="10822" PostTypeId="1" AcceptedAnswerId="10826" CreationDate="2011-05-15T13:48:01.293" Score="3" ViewCount="49" Body="&lt;p&gt;I am not able to find an existing name for the comparison function I wrote.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have two lists of values which I want to compare. At first I used jaccard, but then I recognized I need to eliminate the length of the second list as it does not seem to have anything to say in my use-case. So I changed jaccard to:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;sim(L1,L2) = \frac{|L1 \cap L2|}{|L1|}&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One example might be:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;L1 = {name, test, somevalue}&#xA;L2 = {name, foo, bar, baz, some, random, other, values, which, would,&#xA;      flood, the, jaccard, comparison, as, they, are, too, much}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;sim(L1,L2) = \frac{|L1 \cap L2|}{|L1|} = \frac{|\{name\}|}{|\{name, test, somevalue\}|} = \frac{1}{3}&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To make a long story short, I only want to calculate how many percent of the values in L1 are in L2.&#xA;Does anybody know if there is a name for such a function so I can give it a good name within my source code?&lt;/p&gt;&#xA;" OwnerUserId="4350" LastActivityDate="2011-05-15T15:40:04.573" Title="Is there a name for the function | L1 sect L2 | / | L1 |" Tags="&lt;distance-functions&gt;" AnswerCount="1" />
  <row Id="10823" PostTypeId="2" ParentId="10407" CreationDate="2011-05-15T14:13:09.697" Score="0" ViewCount="" Body="&lt;p&gt;I agree with some comments, in that the Poisson approximation sounds nice here (not a 'crude' approximation). It should be asympotically exact, and it seems the most reasonable thing to do, as an exact analytic solutions seems difficult.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an intermediate alternative, (if you really need it) I suggest I fisrt order correction to the Poisson approximation, in the following way (I've done something similar some time ago, and it worked).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As suggested by a comment, your model is (not approximately but exactly) Poisson if we condition on the sum. That is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let $X_t$ ($t$ is a parameter here) be a vector of $n$ independent Poisson variables, the first one with $\lambda = 2t/(n+1)$, the others with $\lambda = t/(n+1)$. Let $s=\sum x$, so $E(s)=t$. It is clear that $X_t$ is not equivalent to other model (because our model is restricted to $s=t$), but it is a good approximation. Further, the distribution of $X_t | s$ is equivalent to our model. Indeed, we can write&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ \displaystyle P(X_t) = \sum_s P(X_t | s) P(s)$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This can also be writen for the event in consideration (that $x_1 $ is the maximum).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We know to compute the LHS, and $P(s)$, but we are interested in the other term. Our first order Poisson approximation comes from assuming that $P(s)$ concentrates about the mean so that it can be assimilated to a delta, and then $ P(X_t) \approx  P(X_t | s=t) $ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To refine the aprroximation, we can see the above as a convolution of two functions: our unknown $P(X_t | s)$, which we assume smooth around $s=t$, and a quasi delta function, say a gaussian with small variance. Now, we have our first order approximation (for continuous variables) :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$h(x) = g(x)  * N(x_0,\sigma^2)$ (convolution)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$h(x_0) \approx g(x_0)  + g(x_0)&amp;#39;&amp;#39;\sigma^2/2$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$g(x_0) \approx h(x_0)  - h&amp;#39;&amp;#39;(x_0)&amp;#39;&amp;#39;\sigma^2/2$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Applying this to the previous equation can lead to a refined approximation to our desired probability.&lt;/p&gt;&#xA;" OwnerUserId="2546" LastActivityDate="2011-05-15T14:13:09.697" CommentCount="6" />
  <row Id="10824" PostTypeId="2" ParentId="10774" CreationDate="2011-05-15T14:33:44.393" Score="2" ViewCount="" Body="&lt;p&gt;1) What is your spatial explaining variable? Looks like the x*y plane would be a poor model for the spatial effect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.stack.imgur.com/KDVrG.png&quot; alt=&quot;plot of treatments and responses&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;&lt;pre&gt;&lt;code&gt;i=c(1,3,5,7,8,11,14,15,16,17,18,22,23,25,28,30,31,32,35,36,39,39,41,42)&#xA;l=rep(NA,42)[i];l[i]=level&#xA;r=rep(NA,42)[i];r[i]=response&#xA;image(t(matrix(-l,6)));title(&quot;treatment&quot;)&#xA;image(t(matrix(-r,6)));title(&quot;response&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;2) Seeing as how the blocks are 1 mile apart and you expect to see effects for mere 30 meters, I would say it's entirely appropriate to analyze them separately.&lt;/p&gt;&#xA;" OwnerUserId="2456" LastEditorUserId="2456" LastEditorDisplayName="" LastEditDate="2011-05-15T20:09:18.353" LastActivityDate="2011-05-15T20:09:18.353" CommentCount="4" />
  <row Id="10825" PostTypeId="2" ParentId="10407" CreationDate="2011-05-15T15:05:05.490" Score="0" ViewCount="" Body="&lt;p&gt;Just a word of explanation: Part out of curiosity, part for lack of a better more theoretical method, i approached the problem in a completely empirical/inductive way. I'm aware that there is the risk of getting stuck in a dead end without gaining much insight, but i thought, i'll just present what i got so far anyway, in case it is useful to someone.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Starting by computing the exact probabilities for $n,t\in\{1,...,8\}$ we get&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.stack.imgur.com/FczWE.png&quot; alt=&quot;Table of the first few probabilities for low n,t&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Due to the underlying multinomial distribution, multiplying the entries in the table by $(n+1)^t$ leaves us with a purely integer table:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.stack.imgur.com/Nn20U.png&quot; alt=&quot;Table of integerified probabilities&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now we find that there is a polynomial in $n$ for every column which acts as the sequence function for that column:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.stack.imgur.com/RMLVI.png&quot; alt=&quot;Sequence functions for different t's&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Dividing the sequence functions by $(n+1)^t$ gives us sequence functions for the original probabilities for the first $t$'s. These rational polynomials can be simplified by decomposing them into partial fractions and substituting $x$ for $1/(n+1)$, leaving us with:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.stack.imgur.com/RxbR5.png&quot; alt=&quot;Sequence functions in x=1/(n+1)&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;or as coefficient table&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.stack.imgur.com/epKgZ.png&quot; alt=&quot;sequence function coefficients&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Starting with the $x^2$ column there are sequence functions for these coefficients again:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.stack.imgur.com/FtVpS.png&quot; alt=&quot;x^k coefficients sequence functions&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That's how far i got. There are definitely exploitable patterns here that allow sequence functions to occur, but i'm not sure if there is a nice closed form solution for these sequence functions.&lt;/p&gt;&#xA;" OwnerUserId="4360" LastActivityDate="2011-05-15T15:05:05.490" CommentCount="3" />
  <row Id="10826" PostTypeId="2" ParentId="10822" CreationDate="2011-05-15T15:40:04.573" Score="2" ViewCount="" Body="&lt;p&gt;From where I stand it looks like the &lt;em&gt;conditional relative frequency&lt;/em&gt; of $L2$ given $L1$.&lt;/p&gt;&#xA;" OwnerUserId="4376" LastActivityDate="2011-05-15T15:40:04.573" />
  <row Id="10827" PostTypeId="1" CreationDate="2011-05-15T15:48:13.567" Score="8" ViewCount="143" Body="&lt;p&gt;I'm investigating interplay between two variables ($x_1$ and $x_2$). There is a great deal of linear correlation between these variables with $r&amp;gt;0.9$. From the nature of the problem I cannot say anything about the causation (whether $x_1$ causes $x_2$ or the other way around). I would like to study deviations from the regression line, in order to detect outliers. In order to do this I can either build a linear regression of $x_1$ as a function of $x_2$, or the other way around. Can my choice of variable order influence my results?&lt;/p&gt;&#xA;" OwnerUserId="4622" LastActivityDate="2011-05-16T15:13:08.160" Title="Does the variable order matter in linear regression" Tags="&lt;regression&gt;&lt;outliers&gt;&lt;linear-model&gt;" AnswerCount="4" CommentCount="4" FavoriteCount="1" />
  <row Id="10828" PostTypeId="2" ParentId="10827" CreationDate="2011-05-15T15:53:39.430" Score="2" ViewCount="" Body="&lt;p&gt;It surely can (actually, it even matters with regard to the assumptions on your data - you only make assumptions about the distribution of the outcome given the covariate). In this light, you might look up a term like &quot;inverse prediction variance&quot;. Either way, linear regression says nothing about causation! At best, you can say something about causation through careful design.&lt;/p&gt;&#xA;" OwnerUserId="4257" LastActivityDate="2011-05-15T15:53:39.430" />
  <row Id="10829" PostTypeId="2" ParentId="10827" CreationDate="2011-05-15T16:53:06.540" Score="3" ViewCount="" Body="&lt;p&gt;To make the case symmetrical, one may regress the difference between the two variables ($\Delta x$) vs their average value. &lt;/p&gt;&#xA;" OwnerUserId="1496" LastEditorUserId="1496" LastEditorDisplayName="" LastEditDate="2011-05-15T17:01:17.343" LastActivityDate="2011-05-15T17:01:17.343" />
  <row Id="10830" PostTypeId="2" ParentId="10407" CreationDate="2011-05-15T19:07:36.967" Score="3" ViewCount="" Body="&lt;p&gt;Partition the outcomes by the frequency of occurrences $x$ of the &quot;double outcome&quot;, $0 \le x \le t$.  Conditional on this number, the distribution of the remaining $t-x$ outcomes is multinomial across $n-1$ &lt;em&gt;equiprobable&lt;/em&gt; bins.  Let $p(t-x, n-1, x)$ be the chance that no bin out of $n-1$ equally likely ones receives more than $x$ outcomes.  The sought-for probability therefore equals&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\sum_{x=0}^{t} \binom{t}{x}\left(\frac{2}{n+1}\right)^x \left(\frac{n-1}{n+1}\right)^{t-x} p(t-x,n-1,x).$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In &lt;a href=&quot;http://www.stat.purdue.edu/~dasgupta/mult.pdf&quot; rel=&quot;nofollow&quot;&gt;Exact Tail Probabilities and Percentiles of the Multinomial Maximum&lt;/a&gt;, Anirban DasGupta points out (after correcting typographical errors) that $p(n,K,x)K^n/n!$ equals the coefficient of $\lambda^n$ in the expansion of $\left(\sum_{j=0}^{x}\lambda^j/j!\right)^K$ (using his notation).  For the values of $t$ and $n$ involved here, this coefficient can be computed in at most a few seconds (making sure to discard all $O(\lambda^{n+1})$ terms while performing the successive convolutions needed to obtain the $K^{\text{th}}$ power).  (I checked the timing and corrected the typos by reproducing DasGupta's Table 4, which displays the complementary probabilities $1 - p(n,K,x)$, and extending it to values where $n$ and $K$ are both in the hundreds.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Quoting a theorem of Kolchin &lt;em&gt;et al.&lt;/em&gt;, DasGupta provides an approximation for the computationally intensive case where $t$ is substantially larger than $n$.  Between the exact computation and the approximation, it looks like all possibilities are covered.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-05-15T19:07:36.967" CommentCount="3" />
  <row Id="10831" PostTypeId="2" ParentId="10827" CreationDate="2011-05-15T19:50:27.713" Score="0" ViewCount="" Body="&lt;p&gt;Your x1 and x2 variables are collinear. In the presence of multicollinearity, your parameter estimates are still unbiased, but their variance is large, i.e., your inference on the significance of the parameter estimates is not valid, and your prediction will have large confidence intervals. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Interpretation of the parameter estimates is also difficult. In the linear regression framework, the parameter estimate on x1 is the change in Y for a unit change in x1 given every other exogeneous variable in the model is held constant. In your case, x1 and x2 are highly correlated, and you cannot hold x2 constant when x1 is changing.&lt;/p&gt;&#xA;" OwnerUserId="4617" LastActivityDate="2011-05-15T19:50:27.713" />
  <row Id="10832" PostTypeId="1" AcceptedAnswerId="10834" CreationDate="2011-05-15T19:51:12.777" Score="5" ViewCount="69" Body="&lt;p&gt;I've written a small hierarchical clustering algorithm (for better or for worse). I'd like a quick way of visualizing it, any tooling ideas?&lt;/p&gt;&#xA;" OwnerUserId="4623" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-15T20:18:15.013" LastActivityDate="2011-05-17T21:54:51.667" Title="Anyone know of a simple dendrogram visualizer?" Tags="&lt;data-visualization&gt;&lt;dendrogram&gt;" AnswerCount="3" />
  <row Id="10834" PostTypeId="2" ParentId="10832" CreationDate="2011-05-15T20:17:01.317" Score="4" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://taxonomy.zoology.gla.ac.uk/rod/treeview.html&quot; rel=&quot;nofollow&quot;&gt;TreeView&lt;/a&gt; -- it is not a statistical tool, but it is very light and I have a great sentiment to it; and it is easy to make output to &lt;a href=&quot;http://en.wikipedia.org/wiki/Newick_format&quot; rel=&quot;nofollow&quot;&gt;Newick format&lt;/a&gt;, which TV eats without problems.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More powerful solution is to use R, but here you would have to invest some time in making conversion to the &lt;code&gt;dendrogram&lt;/code&gt; object (basically list-of-lists). &lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2011-05-15T20:17:01.317" CommentCount="4" />
  <row Id="10835" PostTypeId="2" ParentId="10554" CreationDate="2011-05-15T20:20:23.437" Score="2" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Homogeneity Definition:&lt;/strong&gt; To start, let's define homogeneity as the degree to which households grouped in same area are like one another for some attribute.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;MAUP Approach&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Paraphrasing the stated problem:&lt;/strong&gt; We are uncertain how homogeneity changes as we decrease the spatial resolution of the design of how we group households into areas. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For this problem @AndyW answer is solid. In the field of geography, your problem can be classed within the modifiable areal unit problem (&lt;a href=&quot;http://en.wikipedia.org/wiki/Modifiable_areal_unit_problem&quot; rel=&quot;nofollow&quot;&gt;MAUP&lt;/a&gt;).You can search the index for 'MAUP' at &lt;a href=&quot;http://www.spatialanalysisonline.com/output/&quot; rel=&quot;nofollow&quot;&gt;this site&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Alternative Clustering Approach&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;An alternative problem:&lt;/strong&gt; Given that we want to maximise areal homogeneity when aggregating households, we are uncertain of the optimal spatial configuration of how we should group the households. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;With the &lt;a href=&quot;http://pysal.org/users/tutorials/region.html&quot; rel=&quot;nofollow&quot;&gt;p-regions clustering algorithm&lt;/a&gt;, you can visually explore different structures of homogeneity within your data by creating different maps of household areas by playing with these 2 parameters: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;changing the different attributes&#xA;for maximising area homogeneity&lt;/li&gt;&#xA;&lt;li&gt;changing the number of households&#xA;required to build an areal group&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="4329" LastEditorUserId="4329" LastEditorDisplayName="" LastEditDate="2011-05-15T20:31:46.160" LastActivityDate="2011-05-15T20:31:46.160" CommunityOwnedDate="2011-05-15T20:20:23.437" CommentCount="1" />
  <row Id="10836" PostTypeId="1" CreationDate="2011-05-15T21:02:28.460" Score="0" ViewCount="52" Body="&lt;p&gt;Hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have attached 2 documents and listed my questions below. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;hoop://www.mediafire.com/?9u3vc2dop16okna&lt;/p&gt;&#xA;&#xA;&lt;p&gt;hoop://www.mediafire.com/?4b994pl89ljdqql&#xA;(replace 'hoop' with 'http')&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I thought it would be easier to attach the question and solutions so you could see clearly what I’m asking. They are very short questions but they are things I am rather stuck on/confused about. A one line answer will be great! :) I have marked the relevant bits in red so it doesn’t waste your time looking for the relevant part! If you could reference the document/page you are answering I would be grateful!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Document 1:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Page 1:&#xA;- Shouldn’t it read min(Tm, z) not just Tm?&#xA;- Is the sum from m;Nm=1 of 1 = sum from m=1 to n of Nm?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Page 2: &#xA;- do go from state ‘a’ to state ‘m’ you can only stay in state ‘a’ and then go to ‘m’ or go to ‘m’ and stay in state ‘m’. I got the differential equation given in the solution apart from the red box highlighted bit. I don’t get how you can go from ‘a’ to state ‘im’ because there is no 'direct arrow'. I don’t get how you can go from ‘im’ to ‘a’ either....&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Page 3: &#xA;- I managed to derive the integral equation for Paa(s,t) but I am not getting that minus sign at all. I have checked and double checked my work but the minus sign seems to come from nowhere! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Page 4: &#xA;-   shouldn’t there be a (mu)Vd1(t) reserve and (v)Vd2(t) reserve added to the first differential equation? Or are they equal to zero and hence ignored? Is the reserve when you die always 0? Is this assumed always?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Document 2:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Page 1:&#xA;Question 1- I’m not at all sure how to start!&#xA;Question 2- What’s an asymptotic covariance matrix?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Page 2:&#xA;How do you find the side conditions to a Thieles differential equation like the one given? I know how to find the differential equation btw.&#xA;For part (b) I’m guessing the v(y+t)Vo(t) term vanishes from the first differential equation? Is that true? Why? &#xA;For part (d). Again I know Pii(0,t) is given by the integral of e^-int(0,t) mui.(t). But this only works when it’s the same state. I don’t get how to work out Poj(0,t) without deriving it…. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!!!! :)&lt;/p&gt;&#xA;" OwnerUserId="4624" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-05-15T21:06:34.343" LastActivityDate="2011-05-15T21:52:42.370" ClosedDate="2011-05-16T10:48:11.893" Title="Mixed Ques on Markov Chains, Kolmogrov Differential Equations, Thieles DE, MLE, Reserves" Tags="&lt;homework&gt;&lt;maximum-likelihood&gt;&lt;markov-process&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="10837" PostTypeId="2" ParentId="10836" CreationDate="2011-05-15T21:52:42.370" Score="0" ViewCount="" Body="&lt;p&gt;Didn't answer all, but here is what I got.  I might update my answer later, if nobody else does.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First doco&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For page 1, the answer is &quot;no it shouldn't&quot;.  This is because that part of the equation is representing the instantaneous probability of death - which you can't observe unless $T_m&amp;lt;z$.  The probability of surviving up to $min(T_m,z)$ is contained in the integral.  Note that the summation over the integrals are over all $m$, not just the ones who didn't die.  Note that the summation you have highlighted is indicating to &quot;sum over all the non-censored times of death&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Page 2 and page 4 was to involved for me - would take me a fairly long time to figure out what was going on (i.e. to understand the question).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Page 3 the minus sign comes from the fact that you want to &lt;em&gt;stay&lt;/em&gt; in state $a$ over the time period.  I think this is a standard result from Kolmogorov equations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second doco&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Question 1 - not enough information about what the notation means - you need to add this in.  I think it's some sort of survival notation (life insurance I think), but don't know it off the top of my head.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Question 2 - this is fairly standard.  Note that asymptotically, the MLEs have a multivariate normal distribution with covariance matrix equal to the (matrix) inverse of $n$ times the Expected Fisher Information, or the matrix inverse of the observed information, evaluated at the MLEs.  This just involves differentiating the log-likelihood twice, and then plugging in the MLEs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Page 2 - again, transition equations take a while for me to understand what's going on, so I need to have a read before answer.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-05-15T21:52:42.370" CommentCount="3" />
  <row Id="10838" PostTypeId="1" CreationDate="2011-05-16T00:17:52.007" Score="1" ViewCount="140" Body="&lt;p&gt;I wonder if there is a simple way to produce a list of variables using a for loop, and give its value.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for(i in 1:3)&#xA;{&#xA;  noquote(paste(&quot;a&quot;,i,sep=&quot;&quot;))=i&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In the above code, I try to create &lt;code&gt;a1&lt;/code&gt;, &lt;code&gt;a2&lt;/code&gt;, &lt;code&gt;a3&lt;/code&gt;, which assign to the values of 1, 2, 3. However, R gives an error message. Thanks for your help.&lt;/p&gt;&#xA;" OwnerUserId="4625" LastEditorUserId="159" LastEditorDisplayName="" LastEditDate="2011-05-16T04:18:00.890" LastActivityDate="2011-05-16T11:53:29.887" Title="Produce a list of variable name in a for loop, then assign values to them" Tags="&lt;r&gt;" AnswerCount="2" CommentCount="6" FavoriteCount="1" />
  <row Id="10839" PostTypeId="2" ParentId="10838" CreationDate="2011-05-16T00:51:17.090" Score="11" ViewCount="" Body="&lt;p&gt;Your are looking for &lt;code&gt;assign()&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for(i in 1:3){&#xA;  assign(paste(&quot;a&quot;, i, sep = &quot;&quot;), i)    &#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;gives&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; ls()&#xA;[1] &quot;a1&quot;          &quot;a2&quot;          &quot;a3&quot; &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; a1&#xA;[1] 1&#xA;&amp;gt; a2&#xA;[1] 2&#xA;&amp;gt; a3&#xA;[1] 3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I agree that using loops is (very often) bad R coding style (see discussion above). Using &lt;code&gt;list2env()&lt;/code&gt; (thanks to @mbq for mentioning it), this is another solution to @Han Lin Shang's question:  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x &amp;lt;- as.list(rnorm(10000))&#xA;names(x) &amp;lt;- paste(&quot;a&quot;, 1:length(x), sep = &quot;&quot;)&#xA;list2env(x , envir = .GlobalEnv)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="307" LastEditorUserId="307" LastEditorDisplayName="" LastEditDate="2011-05-16T11:53:29.887" LastActivityDate="2011-05-16T11:53:29.887" />
  <row Id="10840" PostTypeId="2" ParentId="206" CreationDate="2011-05-16T02:19:18.120" Score="0" ViewCount="" Body="&lt;p&gt;In the case of database, we would always store the data in discrete even the nature of the data is continuous. Why should I emphasize the nature of data? We should take the distribution of data that could help us to analyze the data. IF the nature of data is continuous, I suggest you to use them by continuous analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Take an example of continuous and discrete: MP3. Even the type of &quot;sound&quot; is analogy, if stored by digital format. We should analyze it always in a analogy way.   &lt;/p&gt;&#xA;" OwnerUserId="4588" LastActivityDate="2011-05-16T02:19:18.120" />
  <row Id="10841" PostTypeId="1" AcceptedAnswerId="10843" CreationDate="2011-05-16T02:23:47.057" Score="5" ViewCount="65" Body="&lt;p&gt;Hello, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're modelling a proportion response against numerous predictors that are also proportions, is it necessary to transform the response if the standard OLS model is seemingly well behaved?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By well behaved I mean:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;None of the fitted values are outside the range [0,1] (In fact they are fairly accurate) &lt;/li&gt;&#xA;&lt;li&gt;Residuals look good&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I believe arcsine transform is typically used in this scenario to make the data look normal, but what if this is not needed?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, say the data wasn't normal, would a transform still be necessary if one were modelling the proportions with the Random Forest technique?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cheers&lt;/p&gt;&#xA;" OwnerUserId="845" LastActivityDate="2011-05-16T04:33:27.030" Title="Is it necessary to perform a transformation on proportion data if it's reasonably well behaved?" Tags="&lt;modeling&gt;&lt;data-transformation&gt;&lt;proportion&gt;" AnswerCount="1" />
  <row Id="10842" PostTypeId="1" AcceptedAnswerId="10844" CreationDate="2011-05-16T04:32:57.500" Score="4" ViewCount="76" Body="&lt;p&gt;A simple question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If $Y=\frac{1}{X}$ and I know $f_X(x)$, is it true that $E(Y) = E(1/X) = \int_{-\infty}^\infty \frac{1}{x}f_X(x) dx$?&lt;/p&gt;&#xA;" OwnerUserId="4627" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-16T09:37:28.880" LastActivityDate="2011-05-16T15:28:28.480" Title="Expected value of a transformation" Tags="&lt;data-transformation&gt;&lt;expected-value&gt;" AnswerCount="2" />
  <row Id="10843" PostTypeId="2" ParentId="10841" CreationDate="2011-05-16T04:33:27.030" Score="4" ViewCount="" Body="&lt;p&gt;It depends. If your goal is prediction, then you may not need to do any gymnastics to get a more theoretically sound model if the one in hand does well. But of course you should be always be aware that a good-fitting model to present data may not perform well on new data. You can try to get a feel for that using cross validation, although you simply might not have important aspects of the distribution represented in your sample.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to make inferences using some of the parameters in the model then that model should be motivated by the problem at hand.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyway, a first step is to just look at the response. Is it roughly bell-shaped? Did you try the arcsine transform? Does the transformed distribution look (much) different? If the distribution of the proportions is fairly tight and located somewhere in the middle the transformation might not do much. And then, of course, does the transformation make a difference in the regression?&lt;/p&gt;&#xA;" OwnerUserId="26" LastActivityDate="2011-05-16T04:33:27.030" CommentCount="2" />
  <row Id="10844" PostTypeId="2" ParentId="10842" CreationDate="2011-05-16T04:37:24.950" Score="5" ViewCount="" Body="&lt;p&gt;Yes. In general if $X\sim f(x)$ then for a function $g(x)$ you have $E(g(X)) = \int g(x)f(x)dx$. You can verify this for simple cases by deriving the distribution of the transformed variable. The completely general result takes some more advanced math which you can probably safely avoid :)&lt;/p&gt;&#xA;" OwnerUserId="26" LastActivityDate="2011-05-16T04:37:24.950" CommentCount="3" />
  <row Id="10845" PostTypeId="2" ParentId="10838" CreationDate="2011-05-16T05:46:43.297" Score="6" ViewCount="" Body="&lt;p&gt;If the values are in vector, the loop is not necessary:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;vals &amp;lt;- rnorm(3)&#xA;n &amp;lt;-length(vals)&#xA;lhs &amp;lt;- paste(&quot;a&quot;,1:n,sep=&quot;&quot;)&#xA;rhs &amp;lt;- paste(&quot;vals[&quot;,1:n,&quot;]&quot;,sep=&quot;&quot;)&#xA;eq &amp;lt;- paste(paste(lhs,rhs,sep=&quot;&amp;lt;-&quot;),collapse=&quot;;&quot;)&#xA;eval(parse(text=eq))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As a side note, this is the reason why I love R.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-05-16T05:46:43.297" CommentCount="14" />
  <row Id="10846" PostTypeId="2" ParentId="10816" CreationDate="2011-05-16T06:11:15.550" Score="4" ViewCount="" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;The answer is yes, you do not need to use the parameters of inverse Mills ratios. But you must include them in the regression nevertheless, or your other parameters will be biased.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;According to the article yes. Although if different variables are statistically significant in different regression there is no problem. Just assume that coefficients for the non-significant regressors are zero. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Splitting is perfectly reasonable. Since you are fitting two models, one for decision whether to go to college or not and another for log-earnings, it is perfectly reasonable to assume that different variables will be important. I should investigate this further though, high multicolinearity when using the same variables in probit and ols regression is not a standard feature of Heckman model as far as I know.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-05-16T06:11:15.550" CommentCount="2" />
  <row Id="10847" PostTypeId="1" AcceptedAnswerId="10848" CreationDate="2011-05-16T07:22:55.747" Score="4" ViewCount="192" Body="&lt;p&gt;What kind of function is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$f_X(x) = 2 \lambda \pi x e^{-\lambda \pi x ^2}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this a common distribution? I am trying to find a confidence interval of $\lambda$ using the estimator $\hat{\lambda}=\frac{n}{\pi \sum^n_{i=1} X^2_i}$ and I am struggling to prove if this estimator has Asymptotic Normality.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="4627" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-17T03:53:29.227" LastActivityDate="2011-05-17T03:53:29.227" Title="What kind of distribution is $f_X(x) = 2 \lambda \pi x e^{-\lambda \pi x ^2}$ ?" Tags="&lt;distributions&gt;&lt;confidence-interval&gt;&lt;estimation&gt;&lt;homework&gt;" AnswerCount="2" CommentCount="5" />
  <row Id="10848" PostTypeId="2" ParentId="10847" CreationDate="2011-05-16T08:17:54.307" Score="9" ViewCount="" Body="&lt;p&gt;It is a square root of &lt;a href=&quot;http://en.wikipedia.org/wiki/Exponential_distribution&quot; rel=&quot;nofollow&quot;&gt;exponential distribution&lt;/a&gt; with rate $\pi\lambda$. This means that if $Y\sim\exp(\pi\lambda)$, then $\sqrt{Y}\sim f_X$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since your estimate is &lt;a href=&quot;http://en.wikipedia.org/wiki/Maximum_likelihood&quot; rel=&quot;nofollow&quot;&gt;maximum likelihood estimate&lt;/a&gt; it should be asymptotically normal. This follows immediately from the properties of maximum likelihood estimates. In this particular case:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\sqrt{n}(\hat\lambda-\lambda)\to N(0,\lambda^2)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;since &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$E\frac{\partial^2}{\partial \lambda^2}\log f_X(X)=-\frac{1}{\lambda^2}.$$&lt;/p&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-16T08:24:30.760" LastActivityDate="2011-05-16T08:24:30.760" />
  <row Id="10849" PostTypeId="2" ParentId="10847" CreationDate="2011-05-16T10:22:52.400" Score="8" ViewCount="" Body="&lt;p&gt;Why do you care about asymptotics when the exact answer is just as simple (and exact)?  I am assuming that you want asymptotic normality so that you can use the $Est\pm z_{\alpha}StdErr$ type of confidence interval&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you make the probability transformation $Y_{i}=X_{i}^{2}$ then you have an exponential sampling distribution (as @mpiktas has mentioned):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$f_{Y_{i}}(y_{i})=f_{X_{i}}(\sqrt{y_{i}})|\frac{\partial\sqrt{y_{i}}}{\partial y_{i}}|=2 \lambda \pi \sqrt{y_{i}} \exp(-\lambda \pi \sqrt{y_{i}} ^2)\frac{1}{2\sqrt{y_{i}}}=\lambda\pi\exp(-\lambda\pi y_{i})$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the joint log-likelihood in terms of $D\equiv\{y_{1},\dots,y_{N}\}$ becomes:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Log[f(D|\lambda)]=Nlog(\pi)+Nlog(\lambda)-\lambda\pi\sum_{i=1}^{N}y_{i}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now the only way the data enters the analysis is through the total $T_{N}=\sum_{i=1}^{N}y_{i}$ (and the sample size $N$).  Now it is an elementary sampling theory calculation to show that $T_{N}\sim Gamma(N,\pi\lambda)$, and further that $\pi N^{-1}T_{N}\sim Gamma(N,N\lambda)$.  We can further make this a &quot;pivotal&quot; quantity by taking $\lambda$ out of the equations (via the same way that I just put $N$ into them).  And we have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\lambda\pi N^{-1}T_{N}=\frac{\lambda}{\hat{\lambda}_{MLE}}\sim Gamma(N,N)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that thus we now have a distribution which involves the MLE and whose sampling distribution is independent of the parameter $\lambda$.  Now your MLE is equal to $\frac{1}{\pi N^{-1}T_{N}}$ And so writing quantities $L_{\alpha}$ and $U_{\alpha}$ such that the following holds:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Pr(L_{\alpha} &amp;lt; G &amp;lt; U_{\alpha})=1-\alpha\;\;\;\;\;\;\;G\sim Gamma(N,N)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And we then have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Pr(L_{\alpha} &amp;lt; \frac{\lambda}{\hat{\lambda}_{MLE}} &amp;lt; U_{\alpha})=Pr(L_{\alpha}\hat{\lambda}_{MLE} &amp;gt; \lambda &amp;gt; U_{\alpha}\hat{\lambda}_{MLE})=1-\alpha$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And you have an &lt;strong&gt;exact&lt;/strong&gt; $1-\alpha$ confidence interval for $\lambda$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;NOTE: The Gamma distribution I am using is the &quot;precision&quot; style, so that a $Gamma(N,N)$ density looks like:&#xA;$$f_{Gamma(N,N)}(g)=\frac{N^{N}}{\Gamma(N)}g^{N-1}\exp(-Ng)$$&lt;/p&gt;&#xA;" OwnerUserId="2392" LastEditorUserId="2392" LastEditorDisplayName="" LastEditDate="2011-05-16T13:03:22.043" LastActivityDate="2011-05-16T13:03:22.043" CommentCount="1" />
  <row Id="10850" PostTypeId="1" AcceptedAnswerId="10853" CreationDate="2011-05-16T11:28:46.337" Score="5" ViewCount="67" Body="&lt;p&gt;My current dataset has three conditions, and we've measured the activity levels of 10,000 genes in each condition. Replicated 8 times.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using 10,000 linear models, we determine for each pair of conditions (ie for each of three contrasts) the number of genes with significantly different activity levels. This is &lt;a href=&quot;http://bioinf.wehi.edu.au/limma/&quot; rel=&quot;nofollow&quot;&gt;standard procedure&lt;/a&gt; for this kind of microarray data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We find:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;2000 genes have significantly different activity levels between A and B&lt;/li&gt;&#xA;&lt;li&gt;1500 genes have significantly different activity levels between A and C&lt;/li&gt;&#xA;&lt;li&gt;100 genes have significantly different activity levels between B and C&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;This suggests that conditions B and C are more similar to each other than to C. PCA suggests the same result.&#xA;Is there any way for us to quantify the extent to which &quot;conditions B and C are more similar to each other than to C (ie to put a p-value on it?)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for your help, and apologies if this question is trivial. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Kind regards,&lt;/p&gt;&#xA;" OwnerUserId="3773" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-16T11:51:00.613" LastActivityDate="2011-05-16T12:12:57.250" Title="Comparing numbers of p-values from many linear models" Tags="&lt;genetics&gt;&lt;microarray&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="10851" PostTypeId="1" CreationDate="2011-05-16T11:42:56.027" Score="3" ViewCount="47" Body="&lt;p&gt;I am trying to devise a simple algorithm to create a Bayesian prior from measurements obtained from time series data. Firstly, I presuppose that the data can take on one of five possible &quot;shapes&quot; or &quot;patterns&quot; and then measure the Euclidean distance of each time series data point from its equivalent point in time in each of the five possible patterns, summed over a specified period of time for each pattern. This will give five distinct numbers with the lowest one indicating which pattern the data is most likely to resemble due to the fact that this distance metric is the smallest &quot;error measurement.&quot; However, this is not a sufficient measure in and of itself hence my desire to use it as a prior in a naive Bayesian classifier. My problem is how to use these numbers to form a prior?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am thinking of using some form of ratio i.e. the numbers are 1 2 3 4 5 for a total of 15, therefore 15/1=15, 15/2=7.5, 15/3=5, 15/4=3.75, 15/5=3 and so the prior for 1 would be 15/(15+7.5+5+3.75+3), prior for 2 would be 7.5/(15+7.5+5+3.75+3) etc. Is there any justification for using an approach such as this, or is there perhaps a better approach?&lt;/p&gt;&#xA;" OwnerUserId="226" LastActivityDate="2011-05-16T11:42:56.027" Title="Algorithm to create Bayesian priors from measurements" Tags="&lt;bayesian&gt;&lt;modeling&gt;" />
  <row Id="10852" PostTypeId="1" AcceptedAnswerId="10864" CreationDate="2011-05-16T12:02:12.167" Score="2" ViewCount="53" Body="&lt;p&gt;I have annual returns and standard deviations for two funds, $r_{a}$, $r_{b}$, $\mathrm{SD}_{a}$ and $\mathrm{SD}_{b}$ but I do not have individual data, just the annual data. The annual correlation between the prices of the funds is 0.7. If I had the individual data I could use&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\mathrm{Cov}(A,B) =\sum_i (\bar{r}_{a} - r_{i a}) (\bar{r}_{b} - r_{i b}) \&amp;gt;,$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;but now I am lost without the data. Are there some approximations to get the half annual data from the annual data or some formula to do it directly?&lt;/p&gt;&#xA;" OwnerUserId="2914" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-05-16T14:47:44.590" LastActivityDate="2011-05-16T15:59:16.300" Title="Half annual covariance $\mathrm{Cov}(X,Y)_{\text{6 months}}$ from annual covariance?" Tags="&lt;covariance&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="10853" PostTypeId="2" ParentId="10850" CreationDate="2011-05-16T12:12:57.250" Score="2" ViewCount="" Body="&lt;p&gt;Hi Yannick,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Prior to answering your question -  is the distribution of effect of the genes justify using a linear model (e.g: are they distributed more or less normally?)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now to your question - I might offer to go a different way about it.  It sounds like what you are asking for is to measure the correlation (e.g similarity of behavior) between the different conditions.  A simple way to do that is to take the mean (or maybe the trimmed mean or median) of the 8 replications and then you'd have 10K triplets you can use for creating a correlation matrix (between the 3 conditions).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second step would then be to answer the question if one correlation (say between A and B) is significantly higher then the other two correlations (say, between --A and C-- and --B and C--).  Here you can use the following nice &lt;a href=&quot;http://faculty.vassar.edu/lowry/rdiff.html&quot; rel=&quot;nofollow&quot;&gt;online tool&lt;/a&gt; or you can code it yourself in R using the information given &lt;a href=&quot;http://davidmlane.com/hyperstat/B8712.html&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cheers,&#xA;Tal&lt;/p&gt;&#xA;" OwnerUserId="253" LastActivityDate="2011-05-16T12:12:57.250" CommentCount="3" />
  <row Id="10855" PostTypeId="1" CreationDate="2011-05-16T12:59:12.657" Score="5" ViewCount="97" Body="&lt;p&gt;I have this long running experiment. Each time I run it I get a new goodness value, since the algorithm has random variables in it. So I need to report the mean and the std of some n runs. What should n be?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to be able to defend n based on some statistical ideas. Some kind of scientific reference (a book, a paper) would be wonderful, too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I provide more details as you say, thanks for the answers:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In computer vision, an important challenge is to recognize objects from images. Different computer algorithms are developed for this purpose. To see how good a new algorithm is, one sometimes constructs a test and a training set of images, say 1000 images for each, train the algorithm with the training images, and produce a success rate using the test set. If out of the 1000 objects in the test images, 800 is recognized by the algorithm, the success rate is said to be 80 percent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, my algorithm analyses, say, 1000 RANDOM points in the image, and using that analysis, tries to recognize the objects in the image. Each time I run the algorithm, I get a different success rate, since 1000 points are produced RANDOMLY. So I think its best to report some kind of summary statistics (e.g. the main and std deviation) of the success rate. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, one sometimes needs to say, &quot;well in addition to my algorithm, I tried these, say, 10 algorithms on the same dataset, and this table shows that mine is the best in this and this way...&quot; Some of these algorithms may need to run more than once, too. So one can really have a long experiment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, as I said before, at least how many times should I run the long running experiment?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thx.&lt;/p&gt;&#xA;" OwnerUserId="4629" LastEditorUserId="4629" LastEditorDisplayName="" LastEditDate="2011-05-17T12:27:13.450" LastActivityDate="2011-05-18T03:09:31.003" Title="Number of times to run a lengthy experiment" Tags="&lt;experiment-design&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="10856" PostTypeId="1" CreationDate="2011-05-16T13:52:03.117" Score="7" ViewCount="61" Body="&lt;p&gt;... and why ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming $X_1$,$X_2$ are independent random-variables with mean $\mu_1,\mu_2$ and variance $\sigma^2_1,\sigma^2_2$ respectively. My basic statistics book tells me that the distribution of the $X_1-X_2$ has the following properties: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;$E(X_1-X_2)=\mu_1-\mu_2$&lt;/li&gt;&#xA;&lt;li&gt;$Var(X_1-X_2)=\sigma^2_1 +\sigma^2_2$&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Now let's say $X_1$, $X_2$  are t-distributions with $n_1-1$, $n_2-2$ degrees of freedom. What is the distribution of $X_1-X_2$ ? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;This question has been edited:&lt;/strong&gt; The original question was &lt;em&gt;&quot;What are the degrees of freedom of the difference of two t-distributions ?&quot;&lt;/em&gt;. mpiktas has already pointed out that this makes no sense since $X_1-X_2$ is not t-distributed, no matter how approximately normal $X_1,X_2$ (i.e. high df) may be.&lt;/p&gt;&#xA;" OwnerUserId="264" LastEditorUserId="264" LastEditorDisplayName="" LastEditDate="2011-05-16T14:39:37.140" LastActivityDate="2011-05-16T14:39:37.140" Title="What is the distribution of the difference of two-t-distributions" Tags="&lt;distributions&gt;&lt;degrees-of-freedom&gt;&lt;students-t&gt;" AnswerCount="1" CommentCount="4" />
  <row Id="10857" PostTypeId="2" ParentId="10856" CreationDate="2011-05-16T13:58:53.843" Score="5" ViewCount="" Body="&lt;p&gt;The sum of two independent t-distributed random variables is not t-distributed. Hence you cannot talk about degrees of freedom of this distribution, since the resulting distribution does not have any degrees of freedom in a sense that t-distribution has.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-05-16T13:58:53.843" CommentCount="6" />
  <row Id="10858" PostTypeId="1" AcceptedAnswerId="10860" CreationDate="2011-05-16T14:01:19.187" Score="4" ViewCount="190" Body="&lt;p&gt;This is more of a general statistics question, though if it matters I'm writing PHP code.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say I'm trying to compute the average value of a toy that is commonly bought and sold on the secondary market, and I have a set of price values culled both from auctions and from user-entered &quot;price paid&quot; data. The data points that represent auctions are pretty reliable, but I also get the occasional &quot;garage sale&quot; type of data point, where someone may have paid a buck to buy something from Aunt Polly at a garage sale. The problem is that the &lt;code&gt;$1&lt;/code&gt; type of data points aren't really valuable to me, as they don't really indicate value--Aunt Polly didn't know any better, and didn't care. Similarly, I may occasionally get a data point coming from a jokester entering &lt;code&gt;$9000&lt;/code&gt; for a toy that is really only worth &lt;code&gt;$9&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, when computing value, what's the best way to factor these types of anomalies out of otherwise useful data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've read about outliers, and something about generally ignoring anything that is more than 2.5 standard deviations outside the rest of the data, but I'm looking for the full recipe, here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks so much!&lt;/p&gt;&#xA;" OwnerUserId="4631" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-17T17:02:35.693" LastActivityDate="2011-05-17T17:02:35.693" Title="Computing average value ignoring outliers" Tags="&lt;standard-deviation&gt;&lt;outliers&gt;" AnswerCount="5" CommentCount="3" />
  <row Id="10859" PostTypeId="2" ParentId="10858" CreationDate="2011-05-16T14:13:37.913" Score="2" ViewCount="" Body="&lt;p&gt;I originally posted this on SO before it was deleted:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://stats.stackexchange.com/&quot;&gt;http://stats.stackexchange.com/&lt;/a&gt; will probably help you better with this, and give a more comprehensive answer. I'm not a mathematician, but I suspect there are multiple ways to solve this issue.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a programmer this is how I would tackle the problem. I'm not skilled enough to tell you if this is sound, but for simple data it should be acceptable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Depending on the type of data, it might be acceptable to have cut off amounts. You will probably want a rolling average (often used in stock markets) that takes the average price over the last n months, this helps negate the impact of inflation, and then have a &lt;code&gt;$n&lt;/code&gt; cuttoff or a percentage based cutoff, that is, any value that deviates +-20% or +-&lt;code&gt;$n&lt;/code&gt; of the rolling average will be ignored.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This would work quite well for relatively stable markets, if your entity exists in a volatile market that fluctuates wildly then you probably want to find a different approach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You also need to seriously consider cutting data off, you mention granny's yard sale which is arguably a legitimate cut off, but you need to accept that you will probably be losing legitimate data points as well that could have a significant effect on your results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But again, there will be multiple ways to achieve this.&lt;/p&gt;&#xA;" OwnerUserId="4632" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-16T14:26:03.333" LastActivityDate="2011-05-16T14:26:03.333" CommentCount="1" />
  <row Id="10860" PostTypeId="2" ParentId="10858" CreationDate="2011-05-16T14:22:43.363" Score="5" ViewCount="" Body="&lt;p&gt;In boxplots, values that are more than 1.5 times the IQR (interquartile range, difference between quartile 1 and 3) away from (as in: in the direction away from the median) the quartiles are typically considered outliers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I cannot say whether this is an appropriate measure for your data, though...&lt;/p&gt;&#xA;" OwnerUserId="4257" LastActivityDate="2011-05-16T14:22:43.363" CommentCount="2" />
  <row Id="10861" PostTypeId="2" ParentId="10827" CreationDate="2011-05-16T15:13:08.160" Score="3" ViewCount="" Body="&lt;p&gt;Standard regression minimizes the vertical distance between the points and the line, so switching the 2 variables will now minimize the horizontal distance (given the same scatterplot).  Another option (which goes by several names) is to minimize the perpendicular distance, this can be done using principle components.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is some R code that shows the differences:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(MASS)&#xA;&#xA;tmp &amp;lt;- mvrnorm(100, c(0,0), rbind( c(1,.9),c(.9,1)) )&#xA;&#xA;plot(tmp, asp=1)&#xA;&#xA;fit1 &amp;lt;- lm(tmp[,1] ~ tmp[,2])  # horizontal residuals&#xA;segments( tmp[,1], tmp[,2], fitted(fit1),tmp[,2], col='blue' )&#xA;o &amp;lt;- order(tmp[,2])&#xA;lines( fitted(fit1)[o], tmp[o,2], col='blue' )&#xA;&#xA;fit2 &amp;lt;- lm(tmp[,2] ~ tmp[,1])  # vertical residuals&#xA;segments( tmp[,1], tmp[,2], tmp[,1], fitted(fit2), col='green' )&#xA;o &amp;lt;- order(tmp[,1])&#xA;lines( tmp[o,1], fitted(fit2)[o], col='green' )&#xA;&#xA;fit3 &amp;lt;- prcomp(tmp)&#xA;b &amp;lt;- -fit3$rotation[1,2]/fit3$rotation[2,2]&#xA;a &amp;lt;- fit3$center[2] - b*fit3$center[1]&#xA;abline(a,b, col='red')&#xA;segments(tmp[,1], tmp[,2], tmp[,1]-fit3$x[,2]*fit3$rotation[1,2], tmp[,2]-fit3$x[,2]*fit3$rotation[2,2], col='red')&#xA;&#xA;legend('bottomright', legend=c('Horizontal','Vertical','Perpendicular'), lty=1, col=c('blue','green','red'))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To look for outliers you can just plot the results of the principle components analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may also want to look at:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Bland and Altman (1986), Statistical&#xA;  Methods for Assessing Agreement&#xA;  Between Two Methods of CLinical&#xA;  Measurement. Lancet, pp 307-310&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="4505" LastActivityDate="2011-05-16T15:13:08.160" />
  <row Id="10862" PostTypeId="2" ParentId="10842" CreationDate="2011-05-16T15:28:28.480" Score="1" ViewCount="" Body="&lt;p&gt;Another approach if you are happy with a numerical estimate (as opposed to the theorectical exact value) is to generate a bunch of data from the distribution, do the transformation, then take the mean of the transformed data as the estimate of the expected value.  This avoids integration which can be nice in ugly cases, but does not give the theory, relationship, or exact value.&lt;/p&gt;&#xA;" OwnerUserId="4505" LastActivityDate="2011-05-16T15:28:28.480" />
  <row Id="10863" PostTypeId="1" CreationDate="2011-05-16T15:52:15.507" Score="-3" ViewCount="37" Body="&lt;p&gt;Am completely lost for the second half of my statistics assignment..&#xA;I haven't slept for days because of another project that is due. I am behind in lecture materials so i have no idea what is going on in this question:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://i1088.photobucket.com/albums/i330/andrewtran91/STATSQUESTION2.jpg&quot; rel=&quot;nofollow&quot;&gt;http://i1088.photobucket.com/albums/i330/andrewtran91/STATSQUESTION2.jpg&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;look, i know this is very lazy and i don't seem to be putting in any effort to try and learn etc.. but it has come to a point where i am so tired that i start to understand nothing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, this assignment needs to be done. I will try to understand it after a good night sleep.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If someone would be kind enough to email me a set of solutions to andrew.tran91@yahoo.com.au that would mean the world to me.. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;i know most would probably give me hints and tell me to do it myself etc. but i have no hope of understanding what's going on and i just thought it wouldn't hurt to ask.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;thank you in advance &lt;/p&gt;&#xA;" OwnerUserId="4633" LastActivityDate="2011-05-16T15:52:15.507" ClosedDate="2011-05-16T17:11:25.983" Title="Assignment questions on transformation/convergence/inference" Tags="&lt;data-transformation&gt;&lt;inference&gt;" CommentCount="2" />
  <row Id="10864" PostTypeId="2" ParentId="10852" CreationDate="2011-05-16T15:59:16.300" Score="1" ViewCount="" Body="&lt;p&gt;The &lt;a href=&quot;http://en.wikipedia.org/wiki/Covariance&quot; rel=&quot;nofollow&quot;&gt;Wikipedia entry for covariance&lt;/a&gt; is good for learning the rules for manipulating and calculating covariance. The &lt;a href=&quot;http://en.wikipedia.org/wiki/Variance&quot; rel=&quot;nofollow&quot;&gt;Wikipedia entry on variance&lt;/a&gt; is good for that special case. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your particular case, you can solve your problem fairly easily by using the rule (in the properties section of the covariance article)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ &#xD;&#xA;\operatorname{Cov}(aX+bY, cW+dV) = ac\,\operatorname{Cov}(X,W)+ad\,\operatorname{Cov}(X,V)+bc\,\operatorname{Cov}(Y,W)+bd\,\operatorname{Cov}(Y,V)\ &#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you think of annual returns as the sum of two semi-annual returns, and assume that the semi-annual returns are not correlated across periods and the correlations for the two periods are constant. I think it should turn out that you can simply divide the covariance in half.&lt;/p&gt;&#xA;" OwnerUserId="1146" LastActivityDate="2011-05-16T15:59:16.300" CommentCount="5" />
  <row Id="10865" PostTypeId="2" ParentId="10774" CreationDate="2011-05-16T16:09:29.857" Score="4" ViewCount="" Body="&lt;p&gt;1) You can model spatial correlation with the &lt;code&gt;nlme&lt;/code&gt; library; there are several possible models you might choose.  See pages 260-266 of Pinheiro/Bates.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A good first step is to make a variogram to see how the correlation depends on distance.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(nlme)&#xA;m0 &amp;lt;- gls(response ~ level, data = layout)  &#xA;plot(Variogram(m0, form=~x+y))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here the sample semivariogram increases with distance indicating that the observations are indeed spatially correlated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One option for the correlation structure is a spherical structure; that could be modeled in the following way.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;m1 &amp;lt;- update(m0, corr=corSpher(c(15, 0.25), form=~x+y, nugget=TRUE))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This model does seem to fit better than the model with no correlation structure, though it's entirely possible it too could be improved on with one of the other possible correlation structures.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; anova(m0, m1)&#xA;   Model df     AIC      BIC    logLik   Test  L.Ratio p-value&#xA;m0     1  3 46.5297 49.80283 -20.26485                        &#xA;m1     2  5 43.3244 48.77961 -16.66220 1 vs 2 7.205301  0.0273&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;2) You could also try including &lt;code&gt;x&lt;/code&gt; and &lt;code&gt;y&lt;/code&gt; directly in the model; this could be appropriate if the pattern of correlation depends on more than just distance.  In your case (looking at sesqu's pictures) it seems that for this block anyway, you may have a diagonal pattern.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here I'm updating the original model instead of m0 because I'm only changing the fixed effects, so the models should both be fit using maximum likelihood.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; model2 &amp;lt;- update(model, .~.+x*y)&#xA;&amp;gt; anova(model, model2)&#xA;Analysis of Variance Table&#xA;&#xA;Model 1: response ~ level&#xA;Model 2: response ~ level + x + y + x:y&#xA;  Res.Df    RSS Df Sum of Sq      F   Pr(&amp;gt;F)   &#xA;1     22 5.3809                                &#xA;2     19 2.7268  3    2.6541 6.1646 0.004168 **&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To compare all three models, you'd need to fit them all with &lt;code&gt;gls&lt;/code&gt; and the maximum likelihood method instead of the default method of REML.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; m0b &amp;lt;- update(m0, method=&quot;ML&quot;)&#xA;&amp;gt; m1b &amp;lt;- update(m1, method=&quot;ML&quot;)&#xA;&amp;gt; m2b &amp;lt;- update(m0b, .~x*y)&#xA;&amp;gt; anova(m0b, m1b, m2b, test=FALSE)&#xA;    Model df      AIC      BIC     logLik&#xA;m0b     1  3 38.22422 41.75838 -16.112112&#xA;m1b     2  5 35.88922 41.77949 -12.944610&#xA;m2b     3  5 29.09821 34.98847  -9.549103&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Remember that especially with your knowledge of the study, you might be able to come up with a model that is better than any of these.  That is, model &lt;code&gt;m2b&lt;/code&gt; shouldn't necessarily be considered to be the best yet.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: These calculations were performed after changing the x-value of plot 37 to 0.&lt;/p&gt;&#xA;" OwnerUserId="3601" LastEditorUserId="3601" LastEditorDisplayName="" LastEditDate="2011-05-16T19:56:57.893" LastActivityDate="2011-05-16T19:56:57.893" CommentCount="4" />
  <row Id="10867" PostTypeId="1" CreationDate="2011-05-16T17:36:32.373" Score="2" ViewCount="62" Body="&lt;p&gt;I have 3 experiments, where some quantity was measured 3 times. Thus, 3 biological replicas, 3 technical replicas in each biological replica, 9 measurements in total. I need to answer the following questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;For each biological replica: are my 3 measurements consistent? To do this, I'm using Dixon's Q test.&lt;/li&gt;&#xA;&lt;li&gt;For average measurements derived for each biological replica: are these 3 measurements consistent?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;So, how should I go about the second question? Is it a good idea to use Dixon's Q test here too? What other tests can I use to address the above questions? I guess ANOVA is not suitable because samples are too small, right?&lt;/p&gt;&#xA;" OwnerUserId="4337" LastActivityDate="2011-05-16T20:12:36.277" Title="Tests for consistent measurements and outliers" Tags="&lt;repeated-measures&gt;&lt;outliers&gt;&lt;reproducible-research&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="10868" PostTypeId="2" ParentId="10858" CreationDate="2011-05-16T18:31:08.690" Score="0" ViewCount="" Body="&lt;p&gt;Perhaps a robust estimator like &lt;a href=&quot;http://en.wikipedia.org/wiki/RANSAC&quot; rel=&quot;nofollow&quot;&gt;RANSAC&lt;/a&gt; could be used here.&lt;/p&gt;&#xA;" OwnerUserId="4360" LastActivityDate="2011-05-16T18:31:08.690" />
  <row Id="10869" PostTypeId="2" ParentId="10858" CreationDate="2011-05-16T18:58:16.157" Score="1" ViewCount="" Body="&lt;p&gt;hope this helps&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Simplistic approaches , as suggested here , often fail to their lack of generality. In general you may have a series that has multiple trends and/or multiple levels thus to detect anomalies one has to &quot;control&quot; for these effects. Additionally there may be a seasonal effect that may have started in the last k periods and not present in the first n-k values. Now let's get to the meat of the problem. Assume that there are no mean shifts/no trend changes/no seasonal pulse structure in the data. The data may be autocorrelated causing the simple standard deviation to be over or under estimated depending upon the nature of the autocorrelation. The possible existence of Pulses,Seasonal Pulses,Level Shifts and/or local time trends obfuscates the identification of the &quot;exceptions&quot;. Using a &quot;bad standard deviation&quot; to try to identify anomalies is flawed because it is an out of model test as compared to an &quot;in model test&quot; which ultimately is what is used to conclude about the statistical significance of the anomilies. You might Google &quot;how to do statistical intervention detection&quot; to help you find sources/software to do this. &lt;/p&gt;&#xA;" OwnerUserId="3382" LastEditorUserId="3382" LastEditorDisplayName="" LastEditDate="2011-05-16T20:39:41.933" LastActivityDate="2011-05-16T20:39:41.933" />
  <row Id="10870" PostTypeId="2" ParentId="10858" CreationDate="2011-05-16T19:10:27.770" Score="3" ViewCount="" Body="&lt;p&gt;You could consider using a &lt;a href=&quot;http://en.wikipedia.org/wiki/Truncated_mean&quot; rel=&quot;nofollow&quot;&gt;trimmed mean&lt;/a&gt;. This would involve discarding, say, the highest 10% of values and the lowest 10% of values, regardless of whether you consider them to be bad.&lt;/p&gt;&#xA;" OwnerUserId="3835" LastActivityDate="2011-05-16T19:10:27.770" CommentCount="4" />
  <row Id="10871" PostTypeId="1" AcceptedAnswerId="10879" CreationDate="2011-05-16T19:35:27.120" Score="7" ViewCount="101" Body="&lt;p&gt;Hello all,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am interested to know what level of statistics kids are learning in different countries around the world.  Could you please suggest data/links that shed light on what is happening in this regards?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'll start.  Israel: The students who are taking advance math study more or less - mean, sd, histogram, normal distribution, very basic probability.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="253" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-16T20:31:17.657" LastActivityDate="2011-05-16T22:32:27.220" CommunityOwnedDate="2011-05-16T20:26:08.733" Title="Children's statistical education in different countries?" Tags="&lt;dataset&gt;&lt;education&gt;" AnswerCount="2" CommentCount="5" FavoriteCount="1" />
  <row Id="10872" PostTypeId="2" ParentId="10867" CreationDate="2011-05-16T20:12:36.277" Score="1" ViewCount="" Body="&lt;p&gt;It seems highly unlikely that there would be a test that, based upon three observations, decides whether one is an outlier! The fact that 'the other two' are closer could just as well be an anomaly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At best, you could use Dixon's Q test to find out whether the largest/smallest value of your 9 observations is an outlier. Note that even the almighty &lt;a href=&quot;http://en.wikipedia.org/wiki/Dixon%27s_Q_test&quot; rel=&quot;nofollow&quot;&gt;Wikipedia&lt;/a&gt; advises to only use it once within a dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Either way: the terminology of '3 measurements being consistent' is confusing: in statistics, normally consistency points to big sample sizes...&lt;/p&gt;&#xA;" OwnerUserId="4257" LastActivityDate="2011-05-16T20:12:36.277" CommentCount="4" />
  <row Id="10873" PostTypeId="2" ParentId="10795" CreationDate="2011-05-16T20:24:14.100" Score="2" ViewCount="" Body="&lt;p&gt;There are basically two things to be said. The first is that if you look at the density for the multivariate normal distribution (with mean 0 here) it is proportional to &#xA;$$\exp\left(-\frac{1}{2}x^T P x\right)$$&#xA;where $P = \Sigma^{-1}$ is the inverse of the covariance matrix, also called the precision. This matrix is positive definite and defines via &#xA;$$(x,y) \mapsto x^T P y$$ &#xA;an &lt;em&gt;inner product&lt;/em&gt; on $\mathbb{R}^p$. The resulting geometry, which gives specific meaning to the concept of orthogonality and defines a norm related to the normal distribution, is important, and to understand, for instance, the geometric content of &lt;a href=&quot;http://en.wikipedia.org/wiki/Linear_discriminant_analysis&quot; rel=&quot;nofollow&quot;&gt;LDA&lt;/a&gt; you need to view things in the light of the geometry given by $P$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The other thing to be said is that the partial correlations can be read of directly from $P$, see &lt;a href=&quot;http://en.wikipedia.org/wiki/Partial_correlation#Using_matrix_inversion&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;. &#xA;The same Wikipedia page gives that the partial correlations, and thus the entries of $P$, have a geometrical interpretation in terms of cosine to an angle. What is, perhaps, more important in the context of partial correlations is that the partial correlation between $X_i$ and $X_j$ is 0 if and only if entry $i,j$ in $P$ is zero. For the normal distribution the variables $X_i$ and $X_j$ are then &lt;strong&gt;conditionally independent&lt;/strong&gt; given all other variables. This is what Steffens book, that I referred to in the comment above, is all about. Conditional independence and Graphical models. It has a fairly complete treatment of the normal distribution, but it may not be that easy to follow. &lt;/p&gt;&#xA;" OwnerUserId="4376" LastActivityDate="2011-05-16T20:24:14.100" />
  <row Id="10874" PostTypeId="1" CreationDate="2011-05-16T20:41:15.513" Score="5" ViewCount="100" Body="&lt;p&gt;What are important/notable publishing houses for books in statistics?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I come across a book published at O'Reilly or Springer I imagine it's quality will be high.  What other notable publishing houses are out there (for statistics books)?  Any recommendation on a way to find out? (I'd imagine we could check it if we had the database dump of amazon, but I don't imagine something like that is available anywhere...)&lt;/p&gt;&#xA;" OwnerUserId="253" LastEditorUserId="253" LastEditorDisplayName="" LastEditDate="2011-05-16T21:21:10.213" LastActivityDate="2011-05-16T21:21:10.213" CommunityOwnedDate="2011-05-16T21:48:29.193" Title="High quality publishing house for books in the field of statistics" Tags="&lt;books&gt;" CommentCount="6" />
  <row Id="10875" PostTypeId="2" ParentId="10871" CreationDate="2011-05-16T20:51:58.830" Score="4" ViewCount="" Body="&lt;p&gt;Good question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For my answer, I'll talk about Ireland. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In Senior Cycle (16-18 years) students study very basic statistics, mean, histograms, standard deviation. Basic probability is covered (completely seperately). Calculus, up to the level of integration by parts. Matrices (only 2*2) are an option on the Higher level paper, as is more statistics. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;That being said, less than 20% of the school population take the higher course, so the other 80% do basic statistics, some differentiation and very basic probability. &lt;/p&gt;&#xA;" OwnerUserId="656" LastActivityDate="2011-05-16T20:51:58.830" CommunityOwnedDate="2011-05-16T20:51:58.830" />
  <row Id="10879" PostTypeId="2" ParentId="10871" CreationDate="2011-05-16T22:32:27.220" Score="5" ViewCount="" Body="&lt;p&gt;Statistics eduction in the US is in flux, in no small part because we now expect even &lt;strong&gt;grade school students&lt;/strong&gt; (ages 5-12) to become proficient not only with fundamental concepts of statistical thinking, but also with techniques of data summary and presentation that many of their teachers do not even know!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For an authoritative overview of efforts being made at both the &lt;strong&gt;K-12 and college levels&lt;/strong&gt;, see the &lt;a href=&quot;http://www.amstat.org/education/gaise/&quot; rel=&quot;nofollow&quot;&gt;GAISE reports&lt;/a&gt; on the ASA Website.  At a high level, these documents expect that &lt;em&gt;all&lt;/em&gt; students graduating from U.S. high schools (age 18) will:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;ul&gt;&#xA;  &lt;li&gt;&lt;p&gt;formulate questions that can be addressed with&#xA;  data and collect, organize, and display relevant&#xA;  data to answer them;&lt;/p&gt;&lt;/li&gt;&#xA;  &lt;li&gt;&lt;p&gt;select and use appropriate statistical methods to&#xA;  analyze data;&lt;/p&gt;&lt;/li&gt;&#xA;  &lt;li&gt;&lt;p&gt;develop and evaluate inferences and predictions&#xA;  that are based on data; and&lt;/p&gt;&lt;/li&gt;&#xA;  &lt;li&gt;&lt;p&gt;understand and apply basic concepts of probability.&lt;/p&gt;&lt;/li&gt;&#xA;  &lt;/ul&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Notably, in my opinion, is an insistence that by virtue of &quot;variability in data,&quot; there is an important &quot;difference between statistics and mathematics.&quot;  The aim is to &quot;develop statistical thinking&quot; in students as opposed to teaching techniques or algorithms alone.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a &lt;strong&gt;college level&lt;/strong&gt; approach, a good resource is &lt;a href=&quot;http://www.causeweb.org/&quot; rel=&quot;nofollow&quot;&gt;CAUSEweb&lt;/a&gt; (Consortium for the Advancement of Undergraduate Statistics Education).&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-05-16T22:32:27.220" CommunityOwnedDate="2011-05-16T22:32:27.220" CommentCount="1" />
  <row Id="10880" PostTypeId="1" CreationDate="2011-05-16T23:24:57.510" Score="3" ViewCount="70" Body="&lt;p&gt;I hope this is a stats question :) There's a game (RftG) with cards, each of which has various attributes (generally bonuses of some kind) and a price. I'm wondering if there's some kind of technique for determining the average value of each attribute, and therefore calculating the expected cost of each card as a whole, and therefore finding &quot;undervalued&quot; and &quot;overpriced&quot; cards.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, say that each attribute is a letter (A, B, C...), an obvious and simple technique would be to look for minimal pairs:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ABC: $5&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ABCD: $7&lt;/p&gt;&#xA;&#xA;&lt;p&gt;E: $1&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ED: $2&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore D has an average value of $1.50&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But that seems far too simple and limited. For starters, not every attribute will turn up in a minimal pair. Can anyone point me in the right direction of the kind of technique that would work here? I'm afraid I don't have a stats background. Even some relevant stats terms would help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If interested, there's a card list here: &#xA;&lt;a href=&quot;http://boardgamegeek.com/file/download/4wl2eqaevk/RftG_%2B_exp1_%2B_exp2_%2B_Card_Reference_v1.0.xls&quot; rel=&quot;nofollow&quot;&gt;http://boardgamegeek.com/file/download/4wl2eqaevk/RftG_%2B_exp1_%2B_exp2_%2B_Card_Reference_v1.0.xls&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="4636" LastActivityDate="2011-05-17T01:03:57.033" Title="Calculating the value of attributes of cards in a card game" Tags="&lt;bayesian&gt;&lt;games&gt;&lt;parametric&gt;&lt;pairwise-comparison&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="10881" PostTypeId="2" ParentId="10880" CreationDate="2011-05-17T00:25:11.637" Score="0" ViewCount="" Body="&lt;p&gt;Regression analysis could be used for this sort of thing. But it wouldn't be perfect. For one reason, regression analysis is meant for random data, and the prices fixed to playing cards is definitely not random.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There's not really a standard method here. One reason is that certain abilities, in different contexts become more valuable, and a combination of abilities can be more powerful than if the two abilities were on separate cards.&lt;/p&gt;&#xA;" OwnerUserId="4637" LastEditorUserId="4637" LastEditorDisplayName="" LastEditDate="2011-05-17T01:03:57.033" LastActivityDate="2011-05-17T01:03:57.033" CommentCount="4" />
  <row Id="10882" PostTypeId="1" CreationDate="2011-05-17T01:32:50.910" Score="3" ViewCount="62" Body="&lt;p&gt;&lt;em&gt;I'm not a statician/mathematician, so please be gentle with me&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a cross-post &lt;a href=&quot;http://stackoverflow.com/questions/6021170/formula-to-discard-items-by-votes-lower-bound-of-wilson-score-confidence-interva&quot;&gt;from Stack Overflow&lt;/a&gt;. I'm working with a site that lets the users create 'sections'. Each section have multiple items which are voted. The best ones are shown in the main page.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found that a simplistic approach to rank would be bad idea. &lt;code&gt;positive_votes - negative_votes&lt;/code&gt; will rank the recent ones with less votes higher (100%, two votes) than the older ones that have way more votes but lower percentage (93%, 300 votes). The average isn't a solution either. I found &lt;a href=&quot;http://www.evanmiller.org/how-not-to-sort-by-average-rating.html&quot; rel=&quot;nofollow&quot;&gt;one article&lt;/a&gt; that explains these concepts, why they are a bad idea and how to fix it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I'm using the lower bound of Wilson score confidence interval for a Bernoulli parameter and it seems to be working just fine. However, I'd like to discard items that rank way too bad in that particular section.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think I require two things:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The minimum votes required to discard an item in each section&lt;/li&gt;&#xA;&lt;li&gt;The score's threshold that decides if an item will be discarded or not&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;It has to consider that while one section might have hundreds of items with thousands of votes, another might have less than 10 items with 50 or 60 votes, so while the minimum votes required for the popular one might be 100, it might be too high for the less popular ones.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;In the original question, &lt;a href=&quot;http://stackoverflow.com/questions/6021170/formula-to-discard-items-by-votes-lower-bound-of-wilson-score-confidence-interva/6021687#6021687&quot;&gt;somebody suggested&lt;/a&gt; to use the same formula. However, it seems that the Ruby implementation is missing some parts: the &lt;code&gt;alpha/2&lt;/code&gt; is not present anywhere. Also, the original formula has a +/- sign, while the implementation has just a minus sign.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="4638" LastActivityDate="2011-05-20T17:39:28.577" Title="Formula to discard items by votes (Lower bound of Wilson score confidence interval)" Tags="&lt;confidence-interval&gt;" AnswerCount="1" CommentCount="6" FavoriteCount="1" />
  <row Id="10883" PostTypeId="2" ParentId="10394" CreationDate="2011-05-17T01:53:56.790" Score="-1" ViewCount="" Body="&lt;p&gt;I guess Gaussian efficiency is something related to computation cost.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The efficiency of Gaussian adaptation relies on the theory of information due to Claude E. Shannon. When an event occurs with probability P, then the information −log(P) may be achieved. For instance, if the mean fitness is P, the information gained for each individual selected for survival will be −log(P) – on the average - and the work/time needed to get the information is proportional to 1/P. Thus, if efficiency, E, is defined as information divided by the work/time needed to get it we have:&#xA;E = −P log(P).&#xA;This function attains its maximum when P = 1/e = 0.37. The same result has been obtained by Gaines with a different method.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I may simply conclude that the higher the Gaussian Efficiency is, less resources (RAM) is needed for computing something like a robust scale estimator of a large sample. Since CPUs are much faster than the rest of computer we prefer to run a trial/error algorithm for times rather doing it at once with saying 128GB of RAM. when the Gaussian Efficiency is high the job will be done in a shorter time. &lt;/p&gt;&#xA;" OwnerUserId="4286" LastActivityDate="2011-05-17T01:53:56.790" CommentCount="2" />
  <row Id="10884" PostTypeId="1" CreationDate="2011-05-17T05:26:40.100" Score="4" ViewCount="112" Body="&lt;p&gt;I received this elementary question by email:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In a regression equation am I correct&#xA;  in thinking that if the beta value is&#xA;  positive the dependent variable has&#xA;  increased in response to greater use&#xA;  of the independent variable, and if&#xA;  negative the dependent variable has&#xA;  decreased in response to an increase&#xA;  in the independent variable - similar&#xA;  to the way you read correlations?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="183" LastActivityDate="2011-05-18T12:14:06.123" Title="Interpretation of positive and negative beta weights in regression equation" Tags="&lt;regression&gt;&lt;causal-inference&gt;" AnswerCount="2" CommentCount="6" FavoriteCount="0" />
  <row Id="10885" PostTypeId="2" ParentId="10884" CreationDate="2011-05-17T05:36:19.520" Score="3" ViewCount="" Body="&lt;p&gt;In explaining the meaning of regression coefficient I found that the following explanation very useful. Suppose we have the regression&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Y=a+bX$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say  $X$ changes by $\Delta X$  and $Y$ changes by $\Delta Y$.  Since we have the linear relationship  we have&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Y+\Delta Y= a+ b(X+\Delta X)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since $Y=a+bX$ we get that&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\Delta Y = b \Delta X.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having this is easy to see that if $b$ positive, then positive change in $X$ will result in positive change in $Y$. If $b$ is negative then positive change in $X$ will result in negative change in $Y$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note:&lt;/strong&gt; I treated this question as a pedagogical one, i.e. provide simple explanation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note 2:&lt;/strong&gt; As pointed out by @whuber this explanation has an important assumption that &#xA;the relationship holds for all possible values of $X$ and $Y$. In reality this is a very restricting assumption, on the other hand the the explanation is valid for small values of $\Delta X$, since Taylor theorem says that relationships which can be expressed as differentiable functions (and this is a reasonable assumption to make) are linear locally.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-17T06:03:25.247" LastActivityDate="2011-05-17T06:03:25.247" CommentCount="5" />
  <row Id="10886" PostTypeId="1" CreationDate="2011-05-17T06:24:48.690" Score="4" ViewCount="54" Body="&lt;p&gt;I'm wondering how to proceed to perform Canonical Correspondence Analysis and Multiple Correspondence Analysis in &lt;em&gt;R&lt;/em&gt; on the following three-way contingency table&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Eco_region3  Eco_region4  Species Freq&#xA;    A           A1          S1      10&#xA;    A           A1          S2      12&#xA;    A           A1          S3       8&#xA;    A           A2          S1      10&#xA;    A           A2          S2       6&#xA;    A           A2          S3      11&#xA;    A           A3          S1       2&#xA;    A           A3          S2       9&#xA;    A           A3          S3      13&#xA;&#xA;    B           B1          S1      13&#xA;    B           B1          S2      15&#xA;    B           B1          S3       7&#xA;    B           B2          S1       9&#xA;    B           B2          S2       8&#xA;    B           B2          S3      13&#xA;    B           B3          S1      15&#xA;    B           B3          S2      12&#xA;    B           B3          S3      13&#xA;&#xA;    C           C1          S1      12&#xA;    C           C1          S2      18&#xA;    C           C1          S3      20&#xA;    C           C2          S1      12&#xA;    C           C2          S2       0&#xA;    C           C2          S3      11&#xA;    C           C3          S1      18&#xA;    C           C3          S2      10&#xA;    C           C3          S3      16&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Eco_region4 is nested within Eco_region3.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="3903" LastEditorUserId="3903" LastEditorDisplayName="" LastEditDate="2011-05-17T17:10:47.323" LastActivityDate="2011-05-17T17:10:47.323" Title="Correspondence analysis for a three-way contingency table" Tags="&lt;r&gt;&lt;contingency-tables&gt;&lt;correspondence&gt;" />
  <row Id="10887" PostTypeId="2" ParentId="10884" CreationDate="2011-05-17T06:33:30.290" Score="3" ViewCount="" Body="&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Beta weights:&lt;/strong&gt; As @whuber mentioned, &quot;beta weights&quot; are by convention standardised regression coefficients (see &lt;a href=&quot;http://en.wikipedia.org/wiki/Standardized_coefficient&quot; rel=&quot;nofollow&quot;&gt;wikipedia on standardised coefficient&lt;/a&gt;).&#xA;In this context, $b$ is often used for unstandardised coefficients and $\beta$ is often used for standardised coefficients.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Basic interpretation&lt;/strong&gt;: A beta weight for a given predictor variable is the predicted difference in the outcome variable in standard units for a one standard deviation increase on the given predictor variable holding all other predictors constant.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;General resource on multiple regression:&lt;/strong&gt; The question is elementary and implies that you should read some general material on multiple regression (&lt;a href=&quot;http://www.statisticshell.com/multireg.pdf&quot; rel=&quot;nofollow&quot;&gt;here is an elementary description by Andy Field&lt;/a&gt;).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Causality:&lt;/strong&gt; Be careful of language like &quot;the dependent variable has increased in response to greater use of the independent variable&quot;. Such language has causal connotations. Beta weights by themselves are not enough to justify a causal interpretation. You would require additional evidence to justify a causal interpretation.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-18T12:14:06.123" LastActivityDate="2011-05-18T12:14:06.123" />
  <row Id="10889" PostTypeId="1" CreationDate="2011-05-17T08:42:53.303" Score="2" ViewCount="29" Body="&lt;p&gt;Say I want to classify my data into two categories. I am pretty sure that my data has been generated by two mixtures of Gaussians -- on has a bimodal and one a trimodal form.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I then train the generating models $p(x|c_1)$ and $p(x|c_2)$ and combine them via Bayes' theorem into a classifier of the form $p(c_i|x)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since both models are obtained by an optimization process, I don't have any guarantee that they fit their class equally well. Actually, it might be that one of the models assigns too much probability mass to points from the other class &lt;em&gt;solely&lt;/em&gt; due to the fact that I let the optimization process run too long, chose the wrong model or whatever.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you can see, I just chose the mixture of Gaussians example only for the sake of the optimization. It might as well be hidden Markov models or even wrong modeling assumptions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does this have a name? Are there principled ways to overcome this?&lt;/p&gt;&#xA;" OwnerUserId="2860" LastActivityDate="2011-05-17T08:42:53.303" Title="Generative modelling: what if the generating models have very different &quot;quality of fit&quot;" Tags="&lt;classification&gt;&lt;generative-models&gt;" />
  <row Id="10890" PostTypeId="1" AcceptedAnswerId="11133" CreationDate="2011-05-17T09:38:43.160" Score="4" ViewCount="95" Body="&lt;p&gt;What is the difference between &lt;a href=&quot;http://en.wikipedia.org/wiki/Endogeneity_%28economics%29&quot; rel=&quot;nofollow&quot;&gt;endogeneity&lt;/a&gt; and unobserved heterogeneity? I know that endogeneity comes for example from omitted variables? But as far as I understand, unobserved heterogeneity causes the same problem. But where exactly lays the difference between these two notions?&lt;/p&gt;&#xA;" OwnerUserId="4496" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-17T10:16:50.513" LastActivityDate="2011-05-26T18:19:51.773" Title="Endogeneity versus unobserved heterogeneity" Tags="&lt;regression&gt;&lt;assumptions&gt;" AnswerCount="2" CommentCount="3" />
  <row Id="10891" PostTypeId="2" ParentId="10271" CreationDate="2011-05-17T14:09:26.937" Score="-1" ViewCount="" Body="&lt;p&gt;After a friend of mine pointed me into the direction of &lt;strong&gt;clustering algorithms&lt;/strong&gt;, I stumbled across &lt;a href=&quot;http://en.wikipedia.org/wiki/DBSCAN&quot; rel=&quot;nofollow&quot;&gt;DBSCAN&lt;/a&gt; which builds clusters in n-dimensional space according to two predefined parameters. The basic idea is density-based clustering, i.e. dense regions form clusters. Outliers are returned separately by the algorithm. So, when applied to my 1-dimensional histogram, DBSCAN is able to tell me, whether my anomaly scores feature any outliers. Note: In DBSCAN, an outlier is just a point which does not belong to any cluster. During normal operations, I expect the algorithm to yield only a single cluster (and no outliers).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After some experimenting, I found out that the parameters $\epsilon \approx 0.1$ works well. This means that points have to exhibit a distance of at least 0.1 to the &quot;normal&quot; cluster in order to be seen as outlier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After being able to identify outliers, finding the threshold boils down to simple rules such as:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;If the set exhibits outliers, set the threshold between the &quot;normal&quot; and &quot;outlier&quot; cluster so that the margin to both is maximized.&lt;/li&gt;&#xA;&lt;li&gt;If the set does not exhibit any outliers, set the threshold one standard deviation away from the outmost right point.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Anyway, thanks for all the helpful replies!&lt;/p&gt;&#xA;" OwnerUserId="4446" LastEditorUserId="4446" LastEditorDisplayName="" LastEditDate="2011-05-18T08:32:24.623" LastActivityDate="2011-05-18T08:32:24.623" CommentCount="8" />
  <row Id="10892" PostTypeId="1" CreationDate="2011-05-17T14:16:14.383" Score="5" ViewCount="59" Body="&lt;p&gt;I'm trying to run R's multidimensional scaling algorithm, &lt;code&gt;cmdscale&lt;/code&gt;, on roughly 2,200 variables, i.e. a 2,200x2,200 distance matrix.  It's taking forever (about a day so far).  I don't know much about the algorithm used under the hood.  What's its big-O notation?  Is a more efficient, even if only approximate, algorithm available that's easy to set up and use?&lt;/p&gt;&#xA;" OwnerUserId="1347" LastActivityDate="2011-05-17T18:24:11.260" Title="Big-O Scaling of R's cmdscale()" Tags="&lt;r&gt;&lt;algorithms&gt;&lt;multidimensional-scaling&gt;&lt;big-o-notation&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="10893" PostTypeId="1" CreationDate="2011-05-16T17:39:58.170" Score="2" ViewCount="17" Body="&lt;p&gt;I'm developing an application in which users can create 'sections' (à la subreddit in reddit), in which items/posts can be created and voted with a thumbs-up/down system.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.evanmiller.org/how-not-to-sort-by-average-rating.html&quot; rel=&quot;nofollow&quot;&gt;A great article&lt;/a&gt; guided me on how to sort these votes so that an item with a 100% positive response but with few votes won't get ranked over one with hundreds of votes and an acceptance of 80%. The article describes it pretty well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I'd like to discard the lowest-ranked items and this is where it gets tricky:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;How could I know the minimum number of votes in order to discard it?&lt;/li&gt;&#xA;&lt;li&gt;What is the score's threshold required to discard the item?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;As I said, there are sections, and each one has items (which are the ones voted). The formula has to take into consideration the fact that one section may have 100 items with thousands of votes and another might have 3 or 4 items with 20 votes, so a minimum of 40 votes required might be optimal for the first case but totally out of bounds for the second.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(I was tempted in posting this to MathOverflow, but I'm not really sure since this also involves some programming)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="4638" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-06-02T22:10:16.550" LastActivityDate="2011-06-02T22:10:16.550" Title="Formula to discard items by votes (Lower bound of Wilson score confidence interval)" Tags="&lt;confidence-interval&gt;" AnswerCount="0" CommentCount="0" FavoriteCount="0" />
  <row Id="10894" PostTypeId="2" ParentId="10882" CreationDate="2011-05-16T18:27:42.933" Score="2" ViewCount="" Body="&lt;p&gt;Perhaps you could extend the idea of using the lower bound of the confidence interval for sorting: you could throw away items that have a low &lt;em&gt;upper&lt;/em&gt; bound. The items with only a few votes will have pretty high upper bounds; the lowest upper bounds will correspond to the lowest &quot;quality&quot; items.&lt;/p&gt;&#xA;" OwnerUserId="279" LastActivityDate="2011-05-16T18:27:42.933" CommentCount="5" />
  <row Id="10895" PostTypeId="1" CreationDate="2011-05-17T15:57:51.927" Score="0" ViewCount="28" Body="&lt;p&gt;Can I get some help with these ques from the following document:&#xA;hoop://www.mediafire.com/?5yfym8mchcyej14&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Page 1&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I’m not sure how to calculate the side conditions and/or what they mean. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I’m not sure how to show that as t goes to n the contact gets negative. If you replace t with n don’t you get Va’(n) = c because the Va(n) and Vi(n) = 0 according to the side conditions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Page 2&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Part b – I know that the v(t)/2 bit from the first differential equation will vanish. But how would you show the lump sum, s, payable to the children?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Page 3&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To go from state ‘a’ to state ‘m’ you can only stay in state ‘a’ and then go to ‘m’ or go to ‘m’ and stay in state ‘m’. I got the differential equation given in the solution apart from the red box highlighted bit. I don’t get how you can go from ‘a’ to state ‘im’ because there is no 'direct arrow'. I don’t get how you can go from ‘im’ to ‘a’ either....&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="4624" LastActivityDate="2011-05-17T15:57:51.927" ClosedDate="2011-05-17T16:13:19.810" Title="Thieles Differential Equations" Tags="&lt;mathematical-statistics&gt;" AnswerCount="0" CommentCount="2" />
  <row Id="10896" PostTypeId="1" CreationDate="2011-05-17T16:03:35.043" Score="0" ViewCount="35" Body="&lt;p&gt;Can someone please help me with the questions below from this document: hoop://www.mediafire.com/?5yfym8mchcyej14&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can calculate the likelihood function and the MLE equations for the estimators. I’m not sure how to calculate the following:&#xA;- the asymptotic covariance matrix&#xA;- the individual information matrix&#xA;- the total information matrix&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know the definitions but have never seen example (and cant find one). So if you could please guide me I would be grateful. Feel free to use the question provided or another one of your own. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why doesn’t the distribution have a density? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="4624" LastActivityDate="2011-05-17T16:03:35.043" ClosedDate="2011-05-17T16:13:57.517" Title="MLE (Actuarial Stats)" Tags="&lt;maximum-likelihood&gt;" CommentCount="1" />
  <row Id="10897" PostTypeId="1" CreationDate="2011-05-17T16:06:19.143" Score="5" ViewCount="77" Body="&lt;p&gt;I would like to compute the probability distribution for the length of the fragments which I would obtain by fragmenting a linear rod of length $L$ in the following way:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;I choose at random (uniformly) $n$ breakpoints &lt;/li&gt;&#xA;&lt;li&gt;I cut the rod at those breakpoints, creating $(n+1)$ fragments.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Now, while it is easy to see that the probability that a stretch of length $x$ does not contain any breakpoint goes like a negative exponential, I don't know how to throw in the information about the length of the rod.&lt;/p&gt;&#xA;" OwnerUserId="4642" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-05-17T16:38:47.773" LastActivityDate="2011-05-18T16:41:17.197" Title="Probability distribution of fragment lengths" Tags="&lt;probability&gt;" AnswerCount="2" CommentCount="7" />
  <row Id="10898" PostTypeId="1" CreationDate="2011-05-17T16:10:15.277" Score="0" ViewCount="33" Body="&lt;p&gt;Page 4/5 of this document are relelvant:&#xA;hoop://www.mediafire.com/?5yfym8mchcyej14&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Part d – I can write down the equations. Not sure why the forward equation is more useful then the backward equation. Is it because time is continuous and looking into the future will be more practical then looking at the past? Is there some link with the Markov Property? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can get the integral question through calculations but can’t verbally argue each term. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My try:&#xA;Delta(ij) means 1 when i=j and 0 when i not equal to j. So for Pig(s,t) its 0 x Pjj(s,t) basically. So that term vanishes.&#xA;Pig(s,q) means you go to some state q (q different from i) from time (s,q) and then mu(q,j) means you transition from q to j and then you stay in state j from (q,t). So basically you have gone from i to j between (s,t) (with an intermediary step). So we take the sum because q can be any state different from i. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don’t get why we take the integral!!! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Part f – when I substituted and stuff I got&#xA;Poo(s,t) = Poo(s,t) and Poj(s,t) = 0 (because the mu(g,0) = 0 because once your dead you cant go back). But my answers do not coincide with the statement. Only if you don’t take mu(g,0) = 0 does it look closer. Then that makes no sense because you cannot go to state 0 once you leave it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;3 part c – I don’t really understand this part. What are they asking? What contracts are being compared? &lt;/p&gt;&#xA;" OwnerUserId="4624" LastActivityDate="2011-05-17T16:10:15.277" ClosedDate="2011-05-17T16:14:46.877" Title="Kolmogorov Differential Equations" Tags="&lt;homework&gt;" CommentCount="1" />
  <row Id="10899" PostTypeId="2" ParentId="10897" CreationDate="2011-05-17T16:17:19.443" Score="0" ViewCount="" Body="&lt;p&gt;Let $\{X_i\}$ be the locations of the cuts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd approach this problem by finding the order statistics $\{Y_i\}$ so that $Y_1$ would be the location of the leftmost cut. Then I'd calculate the probability distributions of the differences between the variables $Y_i-Y_{i-1}$. Don't forget to also calculate $Y_1-0$ and $L-Y_n$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone think of a better way?&lt;/p&gt;&#xA;" OwnerUserId="4637" LastActivityDate="2011-05-17T16:17:19.443" CommentCount="2" />
  <row Id="10900" PostTypeId="1" CreationDate="2011-05-17T17:13:30.487" Score="1" ViewCount="54" Body="&lt;p&gt;I am a newbie in stat. I have to generate random numbers for my algorithm through probability distributions. I am concentrating on distributions which have heavy tails and unskewed, whose can produce numbers far away from location parameter. There should be a parameter to control the tail heaviness, like levy distribution where alpha determines tail heaviness. Less central tendency in distribution will be better. I have found t-distribution (for smaller degrees of freedom) and laplace distribution. Which one is better for my purpose and why? Apart from t-distribution and laplace distribution, is there any distribution except cauchy or levy which satisfies what i need.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please give your answer as simple as possible. Thanks in advance&lt;/p&gt;&#xA;" OwnerUserId="4319" LastEditorUserId="4319" LastEditorDisplayName="" LastEditDate="2011-05-17T17:34:38.630" LastActivityDate="2011-05-17T17:34:38.630" Title="Long tailed distribution for generating random numbers" Tags="&lt;distributions&gt;&lt;randomness&gt;" CommentCount="1" />
  <row Id="10901" PostTypeId="2" ParentId="10892" CreationDate="2011-05-17T17:33:50.177" Score="3" ViewCount="" Body="&lt;p&gt;You should be able to test this on your system via Monte Carlo. For example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;try.cmdscale &amp;lt;- function(n) {&#xA;    x &amp;lt;- matrix(rnorm(n*n) ** 2,nrow=n,ncol=n);&#xA;    took.time &amp;lt;- system.time(cmdscale(x))&#xA;}&#xA;&#xA;#repeat multiple times and take the median of the user.self + sys.self&#xA;#probably a better way to do timing?&#xA;multi.cmdscale &amp;lt;- function(n,ntrial = 11) {&#xA;    all.times &amp;lt;- replicate(ntrial,try.cmdscale(n))&#xA;    median(all.times[1,] + all.times[2,])&#xA;}&#xA;&#xA;nvals &amp;lt;- c(8,16,32,64,128,256,512,1024)&#xA;tvals &amp;lt;- sapply(nvals,multi.cmdscale)&#xA;#ack! wish I could do a fit plot in logspace&#xA;plot(nvals,tvals,log=&quot;xy&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;not so informative yet: If I could do some kind of fit on this plot, I could estimate the big O. (not so proficient in R yet, but learning).&#xA;&lt;img src=&quot;http://i.stack.imgur.com/UbBtf.jpg&quot; alt=&quot;scaling of time versus sample size&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2011-05-17T17:33:50.177" CommentCount="1" />
  <row Id="10902" PostTypeId="2" ParentId="10855" CreationDate="2011-05-17T17:50:45.363" Score="3" ViewCount="" Body="&lt;p&gt;If we assume an underlying success probability $\theta$ for a specific training set and number of points used for image analysis, we can expect a binomial distribution for the number of successful object recognitions $k$ out of $n$ runs:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(k|\theta,n)=\left(n\atop k\right)\theta^k (1-\theta)^{n-k}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What we're actually interested in, is the value and uncertainty of $\theta$ for given $n,k$. We get a probability distribution for $\theta$ by applying &lt;a href=&quot;http://en.wikipedia.org/wiki/Bayes%27_theorem&quot; rel=&quot;nofollow&quot;&gt;Bayes Theorem&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$p(\theta|k,n) \propto p(k|\theta,n)*p(\theta|n)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first term on the right side is the likelihood, for which we can use the formula from above. The second term is the &lt;a href=&quot;http://en.wikipedia.org/wiki/Prior_probability&quot; rel=&quot;nofollow&quot;&gt;prior distribution&lt;/a&gt; for $\theta$. If we want to be neutral about $\theta$ before any experiment is done we can choose the &lt;a href=&quot;http://en.wikipedia.org/wiki/Jeffreys_prior&quot; rel=&quot;nofollow&quot;&gt;Jeffreys prior&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$p(\theta|n) \propto \theta^{-1/2}(1-\theta)^{-1/2}.$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Putting our likelihood and prior distributions together and normalizing the resulting distribution we end up with&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$p(\theta|k,n) = \frac{\Gamma(n+1)}{\Gamma(k+\frac{1}{2})\Gamma(n-k+\frac{1}{2})} \theta^{k-1/2}(1-\theta)^{n-k-1/2},$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;which is the &lt;a href=&quot;http://en.wikipedia.org/wiki/Beta_distribution&quot; rel=&quot;nofollow&quot;&gt;Beta distribution&lt;/a&gt; with parameters $k+\frac{1}{2}$ and $n-k+\frac{1}{2}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we e.g. have $n=5$ runs and $k\in\{1,3\}$ successes, the distribution for probability of different underlying $\theta$ values looks like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.stack.imgur.com/cCy6R.png&quot; alt=&quot;true theta pdf&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From this probability distribution we can get all the information we need about $\theta$ in order to decide to do further runs or not, e.g. we can compute &lt;a href=&quot;http://en.wikipedia.org/wiki/Credible_interval&quot; rel=&quot;nofollow&quot;&gt;credible intervals&lt;/a&gt; for the true value of $\theta$. By computing expected values for $\theta$ and $\theta^2$ we can get formulas for the mean $\theta$ and the variance of $\theta$:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$E\left[\theta\right]=\int_{\theta=0}^1 \theta \; p(\theta|k,n) = \frac{2k+1}{2n+2}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$E\left[\theta^2\right]=\int_{\theta=0}^1 \theta^2 p(\theta|k,n) = \frac{(2k+1)(2k+3)}{4(n+1)(n+2)}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$V\left[\theta\right]=E\left[\theta^2\right] - E\left[\theta\right]^2=\frac{4k(n-k)+2n+1}{4(n+1)^2(n+2)}=\frac{(k+\frac{1}{2})(n-k+\frac{1}{2})}{(n+1)^2(n+2)}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now we could run experiments and compute the mean and variance of the expected $\theta$ and stop when the variance is smaller than a predefined value that reflects our desired certainty for $\theta$. For example, let's say we want the standard deviation of our $\theta$-distribution be smaller than $\sigma$, then we should do runs until the condition&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\frac{(k+\frac{1}{2})(n-k+\frac{1}{2})}{(n+1)^2(n+2)} &amp;lt; \sigma^2$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;is satisfied, with a resulting estimate for ${\tilde \theta}=\frac{2k+1}{2n+2}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If e.g. $\sigma=0.05$ we have to do runs until we end up in the orange region in this plot:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.stack.imgur.com/lrL88.png&quot; alt=&quot;&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can see that we need a lot more runs to be as certain about $\theta$ in case of $\theta \approx 0.5$ (about a hundred in this example) than if we have a $\theta$ near zero or one (where perhaps 30 runs could be sufficient).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A nice introductory textbook which contains examples like this is &lt;a href=&quot;http://amzn.com/0198568320&quot; rel=&quot;nofollow&quot;&gt;Data Analysis: A Bayesian Tutorial&lt;/a&gt;. It also contains a small chapter on experiment design.&lt;/p&gt;&#xA;" OwnerUserId="4360" LastEditorUserId="4360" LastEditorDisplayName="" LastEditDate="2011-05-18T03:09:31.003" LastActivityDate="2011-05-18T03:09:31.003" CommentCount="4" />
  <row Id="10903" PostTypeId="2" ParentId="10892" CreationDate="2011-05-17T18:24:11.260" Score="2" ViewCount="" Body="&lt;p&gt;Ironically, the MDS actually wasn't my problem.  The preprocessing I was doing was the issue.  I'm used to coding in lower-level languages and forgot how slow looping is in R.  I rewrote the preprocessing code using vector ops and the MDS actually only takes a few seconds.&lt;/p&gt;&#xA;" OwnerUserId="1347" LastActivityDate="2011-05-17T18:24:11.260" />
  <row Id="10904" PostTypeId="1" AcceptedAnswerId="10922" CreationDate="2011-05-17T18:24:27.733" Score="10" ViewCount="106" Body="&lt;p&gt;What are the pros and cons of learning about a distribution's properties algorithmically (via computer simulations) versus mathematically?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems like computer simulations can be an alternative learning method, especially for those new students who do not feel strong in calculus.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also it seems that coding simulations can offer an earlier and more intuitive grasp of the concept of a distribution.  &lt;/p&gt;&#xA;" OwnerUserId="4329" LastActivityDate="2011-05-17T21:57:27.727" Title="What are the pros and cons of learning about a distribution algorithmically (simulations) versus mathematically?" Tags="&lt;distributions&gt;&lt;algorithms&gt;&lt;education&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="10905" PostTypeId="1" CreationDate="2011-05-17T18:39:01.290" Score="1" ViewCount="97" Body="&lt;p&gt;Hi, &#xA;I have searched for this online for hours but none of online posts is what I am looking for. My question is very easy to implement in SAS Proc mixed procedure but I am not sure how to do it in lme and/or lmer packages. Assume, I have a model,&#xA;$y = \mu + \alpha + \beta +\alpha\beta + e$, &#xA;where $\alpha$ is fixed but $\beta$ and $\alpha\beta$ are random. My R code is&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; f1 = lme(y ~ factor(a), data = mydata,&#xA;     random = list(factor(b) = ~ 1, factor(a):factor(b) = ~ 1))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Error: unexpected &lt;code&gt;=&lt;/code&gt; in:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; f1 = lme(y ~ factor(a), data = mydata,&#xA;          random = list(factor(a) =  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Could someone please tell me how to specify these random effects in lme?&#xA;many thanks in advance&lt;/p&gt;&#xA;" OwnerUserId="4559" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-17T18:40:30.710" LastActivityDate="2011-05-17T20:15:56.673" Title="How to specify random effects in lme?" Tags="&lt;r&gt;&lt;mixed-model&gt;&lt;cross-validation&gt;&lt;biostatistics&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="10906" PostTypeId="2" ParentId="10905" CreationDate="2011-05-17T19:01:07.637" Score="0" ViewCount="" Body="&lt;p&gt;It would help a lot if you provided a data.frame. Now it is not clear what is a grouping factor. I judge that it is $\beta$. Then in &lt;code&gt;lme&lt;/code&gt; notation your model should be written as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;lme(y~a,random=~a|b, data=mydata)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-05-17T19:01:07.637" CommentCount="1" />
  <row Id="10907" PostTypeId="1" AcceptedAnswerId="10944" CreationDate="2011-05-17T19:14:12.547" Score="4" ViewCount="69" Body="&lt;p&gt;I have data with standard error, included below for clarity,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    X           Y           Error in Y&#xA;    0.0105574   -28.831027  0.04422&#xA;    0.0070382   -27.800385  0.04225&#xA;    0.0052787   -27.314088  0.04209&#xA;    0.0042229   -27.054207  0.04185&#xA;    0.0035191   -27.000188  0.04143&#xA;    0.0030164   -26.891275  0.04108&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have obtained parameters &lt;strong&gt;a&lt;/strong&gt; and &lt;strong&gt;b&lt;/strong&gt; of the expression &lt;strong&gt;y=a*x*x + b&lt;/strong&gt; from a weighted least squares regression using this data (&lt;em&gt;fit&lt;/em&gt; in gnuplot). The regression returned what was called &quot;Asymptotic Standard Error&quot; associated with these parameters. I believe this error was calculated using the deviation from fitted point to actual points (&lt;a href=&quot;http://mathworld.wolfram.com/LeastSquaresFitting.html&quot; rel=&quot;nofollow&quot;&gt;Equation 34/35 here&lt;/a&gt;) and is used to assess the quality of a fit. However, this is not the error that I'm interested in.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm looking to determine the value of the data point at X=0.0 from my fitted function &lt;em&gt;with standard error&lt;/em&gt; like my other values. The output of the regression was:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    Final set of parameters            Asymptotic Standard Error&#xA;    a              = -19389.1         +/- 752          (3.878%)        &#xA;    b              = -26.7951         +/- 0.03915      (0.1461%)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So, to be quite specific, &lt;strong&gt;how might I determine the standard error at the point (X,Y)=(0.0, -26.7951) using my fitted function&lt;/strong&gt;? I expect the error in this calculated point to be much larger than the errors of the values reported in the Y values of my table above.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can see how gnuplot is not the right tool for this, as it only weights my data points using the standard error in my input. What I need to do is propagate the error in my original data points to obtain the error on the regression line. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This seems like a pretty basic exercise, sorry for my statistics ignorance. Thanks!&lt;/p&gt;&#xA;" OwnerUserId="4643" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-17T19:56:14.250" LastActivityDate="2011-05-18T15:34:41.480" Title="Obtaining standard error on a data point obtained from linear regression" Tags="&lt;regression&gt;&lt;basic-concepts&gt;&lt;error&gt;&lt;least-squares&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="1" />
  <row Id="10909" PostTypeId="2" ParentId="10905" CreationDate="2011-05-17T20:15:56.673" Score="1" ViewCount="" Body="&lt;p&gt;Try this, it's a standard way to do a split plot.  The notation &lt;code&gt;/&lt;/code&gt; means that method is nested in day.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;lme(level~method, random=~1|day/method, data=d)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="3601" LastActivityDate="2011-05-17T20:15:56.673" CommentCount="3" />
  <row Id="10910" PostTypeId="1" CreationDate="2011-05-17T20:50:11.410" Score="1" ViewCount="46" Body="&lt;p&gt;I have a 37-question scale, where each item belongs to one of four scales.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The items for each scale are interspersed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I compute a scale score for each of the four scales?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to be able to analyse each subscale independently and then compare scale responses on and independent variable.&lt;/p&gt;&#xA;" OwnerUserId="4644" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-18T04:59:26.770" LastActivityDate="2011-05-18T05:03:38.017" Title="How do I create subscales based on sets of items in SPSS" Tags="&lt;spss&gt;&lt;scales&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="10911" PostTypeId="1" CreationDate="2011-05-17T21:04:57.280" Score="-1" ViewCount="39" Body="&lt;p&gt;Q1. &#xA;Suppose we have $Va&amp;#39;(t) = (r + \mu(x+t) + \theta(x+t))Va(t) + c – \theta(x+t)Vi(t)$.&#xA;Side condition : $Va(n-)=0$. What does this side condition mean and how do you calculate it if it's not given? Can you please explain what each term means in the Thieles differential equation (in words)? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Q2. &#xA;Show that the differential equation in Q1 goes negative towards the term of the contract (for $t$ close to $n$). When I tried I got $Va&amp;#39;(t) = c$ which is positive! What modification to the contract that would eliminate this problem?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Q3.&#xA;A company provides a married couple with the following insurance:&#xA;The contract starts in $a$ in which both husband and wife are alive.&#xA;A premium with rate $c$ is paid in state $a$. &#xA;In the case the husband dies by a job related accident, upon which the policy moves to state $h_1$, his wife receives a term insurance with sum $t_i$ and an annuity with rate $a_1$ as long as she is alive.&#xA;In case he dies by any other cause, upon which the policy moves to state $h_2$, his wife receives just an annuity with rate $a_2$. &#xA;In the case the wife dies before the husband (state $w$), half of the reserve is paid back to the husband. &#xA;No payments are made upon transition to state $b$ where both are dead, or afterwards. We assume a constant interest rate $r$. &#xA;The transition rates are:&#xA;$a-h_1 a(t)$; $a-h_2 b(t)$; $a-w c(t)$; $w-d d(t)$; $h_1-d e(t)$ ; $h_2-d f(t)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Write down the general Thieles Differential Equations for the prospective reserves in state $a$ and $h_1$. (This part was fine)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider a modified contract where in addition a lump sum $s$ is paid to the children if both wife and husband are dead, but no reserve is paid back to the husband in the case his wife has died. Write down the new differential equations in state $a$ and $h_1$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know that the $0.5V_i(t)$ term vanishes and you just have $V_i(t)$ instead (in the first bracket). But I don’t really understand why. Why was it $0.5V_i(t)$ in the first place? &#xA;I’m also not sure how to include the second change? Is $Vd(t) = s$? Is it a transition payment?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sorry if its not formatted correctly. I'm not really sure how to use the maths symbols etc.&lt;/p&gt;&#xA;" OwnerUserId="4645" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-05-17T21:54:02.797" LastActivityDate="2011-05-17T21:54:02.797" Title="Thieles Differential Equation &amp; Reserves" Tags="&lt;homework&gt;" CommentCount="5" />
  <row Id="10912" PostTypeId="2" ParentId="10910" CreationDate="2011-05-17T21:07:35.910" Score="0" ViewCount="" Body="&lt;p&gt;This code divides the sum of scores (or of weighted scores) by&#xA; the number of valid values a given case has for that set of variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this example, if a case has values only for verbal and quant, then &#xA; the score will be simply the sum of those divided by 2.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Compute SCORE = sum (.05* jobs, .15* exper, verbal, quant, intvw, college, gpa)/&#xA;nvalid (jobs, exper, verbal, quant, intvw, college, gpa).&#xA;Execute.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The benefit of using this over simpler syntax is that cases will not be left out of the SCORE even if they are missing one or more values on the component variables.&lt;/p&gt;&#xA;" OwnerUserId="2669" LastActivityDate="2011-05-17T21:07:35.910" />
  <row Id="10913" PostTypeId="2" ParentId="10832" CreationDate="2011-05-17T21:13:53.457" Score="0" ViewCount="" Body="&lt;p&gt;While it's not a tool per-se, ascii art is a fairly safe option actually, and not as hard as it seems at first. It's not pretty, but it gets the point accross.&lt;/p&gt;&#xA;" OwnerUserId="4623" LastActivityDate="2011-05-17T21:13:53.457" />
  <row Id="10914" PostTypeId="1" CreationDate="2011-05-17T21:14:03.417" Score="0" ViewCount="37" Body="&lt;p&gt;hey! can someone help me out with this question please?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let mu(t) = alpha + beta(e^ delta t) – Gompertz-Makeham form&#xA;Where alpha, beta, delta are all greater then zero.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The parameters alpha and beta are to be estimated form data on n independent lives observed from birth until age z or death, whichever comes first. Therefore, if Tm is the life length of person m, the observations are right-censored life lengths, min(Tm, z), m=1….n. What does right-censored mean? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I write down the distribution of min(Tm,z). Why does this distribution not have a density? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand how to find the log-likelihood functions and derive the likelihood equations that determine the MLE of alpha and beta (hat). What I am not sure about it how to rearrange those equations to get alpha/beta or work out the following:&#xA;- the asymptotic covariance matrix&#xA;- the individual information matrix&#xA;- the total information matrix&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know the definitions but have never seen example (and can’t find one). So if you could please guide me I would be grateful. Feel free to use the question provided or another one of your own. &lt;/p&gt;&#xA;" OwnerUserId="4645" LastActivityDate="2011-05-17T21:14:03.417" Title="MLE (Actuarial Stats)" Tags="&lt;homework&gt;&lt;maximum-likelihood&gt;" CommentCount="1" />
  <row Id="10915" PostTypeId="1" CreationDate="2011-05-17T21:24:35.407" Score="0" ViewCount="24" Body="&lt;p&gt;Hey! can someone help me out with the following questions please? sorry if its set out incorrectly. Im not sure how to use the maths symbols and did try but miserably failed! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Q1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why are the forward differential equations more useful then the backward equations. What does each actually show?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Q2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can you please verbally argue each term in the integral equation:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Pij(s,t) = delta(i,j) Pjj(s,t) + [sum (all g not equal to j) {integral (s,t) Pig(s,t) mugj(theta) pjj(theta,t) d(theta) } ]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My try:&#xA;Delta(ij) means 1 when i=j and 0 when i not equal to j. So for Pig(s,t) its 0 x Pjj(s,t) basically. So that term vanishes.&#xA;Pig(s,q) means you go to some state q (q different from i) from time (s,q) and then mu(q,j) means you transition from q to j and then you stay in state j from (q,t). So basically you have gone from i to j between (s,t) (with an intermediary step). So we take the sum because q can be any state different from i. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don’t get why we take the integral!!! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Q3.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose we have a model. 0=Alive. 1=Dead (Cause 1). 2=Dead (Cause 2) etc&#xA;Intensity from 0-1 is given by mu(0,1)(t). Intensity from 0-2 is given by mu(0,2)(t). etc&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Write down the expressions for Poo(s,t) and Poj(s,t), j=1….J&#xA;Use them to underpin the following statement&#xA;‘The introduction of a new effective treatment of Alzheimer’s disease would (if everything else remained unchanged) be to the effect that the survival probability would increase, the probability of dying from any other specific cause (e.g. cancer) would increase, and the probability of dying from Alzheimer would decrease”. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I substituted, I got:&#xA;Poo(s,t) = Poo(s,t) and Poj(s,t) = 0 (because the mu(g,0) = 0 because once your dead you cant go back). However, my answers do not coincide with the statement. Only if you don’t take mu(g,0) = 0 does it look closer. Then that makes no sense because you cannot go to state 0 once you leave it. &lt;/p&gt;&#xA;" OwnerUserId="4645" LastActivityDate="2011-05-17T21:24:35.407" Title="Kolmogorov Differential Equations" Tags="&lt;homework&gt;" />
  <row Id="10916" PostTypeId="1" CreationDate="2011-05-17T21:30:36.680" Score="-2" ViewCount="29" Body="&lt;p&gt;Q1. (Difficult)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This insurance covers losses suffered by firms during cessation of production caused by fire or other accidental events. Suppose that a firm is either in state a (active) or I (inactive). The benefit is paid continuously at rate 1 in inactive state, and premium is paid as a single amount at time 0. The contract terminated after n years. Let Z(t) denote the state of the firm at time t.&#xA;We assume that Z is time-homogenous Markov Chain and that the intensities are constant: alpha: a-I and beta: i-a&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The total benefit is the time spent in inactive state during the term of the contact,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tn = integral (0,n) I (Z(p) = i) dp (thats an 'I' not a line)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming an interest rate of zero, find the single premium pi(n) for the benefit using the principle of equivalence. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Q2. (Easier) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The policyholder can be in three states, a (active), i (inactive), r (recovery). &#xA;Transitions are: a-i: alpha; i-r: beta; r-a: gamma. Assume all transition rates are constant/homogenous so p(t) = p(0,t)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using the principle of equivalence (i.e. take the expectation of future discounted benefits) what is the fair premium for the following contract: the insured person received a lump sum of x in case she gets inactive and an annuity with rate y as long as she received recovery treatment. The contract starts in state a at time 0 and terminates after n years. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I got pi = alpha/y as my answer. Not sure if its right though. &lt;/p&gt;&#xA;" OwnerUserId="4645" LastActivityDate="2011-05-17T21:30:36.680" Title="Principle of Equivalence (Actuarial Stats)" Tags="&lt;homework&gt;" CommentCount="4" />
  <row Id="10917" PostTypeId="1" CreationDate="2011-05-17T21:36:29.437" Score="0" ViewCount="34" Body="&lt;p&gt;Im really really stuck on this integral calculation...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assume the interest rate $r$ is constant, and that the conditional hazard rate $\mu(x+t)$ is increasing in $t \in [0,n]$. Prove that:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$a(x+t,n-t) = \int_t^n \text{exp}(-\int_t^p r + \mu(x+s) \text{d}s) \text{d}p$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;($a(x+t,n-t)$ means a continous life annuity of an $x+t$ year old person contact lasts $n-t$ years)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;is decreasing in $t$. Show first that&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\frac{\text{d}}{\text{d}t} a(x+t,n-t) = (r + \mu(x+t)) a(x+t,n-t) – 1.$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know what to do from this result. But I’m not sure how to differentiate that integral.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sorry its not in proper maths symbols but i really dont understand how to use it! My exams on Thursday... have some mercy!!! :)&lt;/p&gt;&#xA;" OwnerUserId="4645" LastEditorUserId="4360" LastEditorDisplayName="" LastEditDate="2011-05-18T01:44:56.140" LastActivityDate="2011-05-18T01:44:56.140" ClosedDate="2011-05-18T04:17:07.887" Title="Stats Integration question" Tags="&lt;homework&gt;" AnswerCount="0" CommentCount="1" />
  <row Id="10918" PostTypeId="1" AcceptedAnswerId="10919" CreationDate="2011-05-17T19:55:21.100" Score="3" ViewCount="32" Body="&lt;p&gt;I'm trying to do a mixture discriminant analysis for a mid-sized data.frame, and bumped into a problem: all my predictions are NA.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After tracing through way too much code, I figured it had something to do with the fact that some of the coefficients in the mda turn out to be NA. I've created a smaller data.frame that still has the problem:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dfr&amp;lt;-structure(list(min_GCs_last_3_bases = structure(c(1L, 2L, 1L, &#xA;1L, 2L, 1L, 2L, 1L, 2L, 2L, 1L, 1L, 2L, 2L, 2L, 2L, 1L, 1L, 2L, &#xA;1L), .Label = c(&quot;zero&quot;, &quot;one&quot;), class = &quot;factor&quot;), cq = c(-0.334138707578632, &#xA;-0.586150643906373, -0.474578712720667, -0.220268433139143, -0.486876877353103, &#xA;-0.0912154554410563, 0.00341593805213764, 0.713424582672338, &#xA;-0.448914652233824, 2.94156773625266, -0.0954835698859817, -0.238125375419562, &#xA;-0.448914652233824, -0.0290299100261503, -0.479666688891261, &#xA;-0.596316272919155, -0.0919665779183363, -0.394427887135795, &#xA;-0.396749521580081, -0.338156015653477), Coverage = structure(c(2L, &#xA;2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 1L, &#xA;1L, 2L, 2L), .Label = c(&quot;failure&quot;, &quot;success&quot;), class = &quot;factor&quot;)), .Names = c(&quot;min_GCs_last_3_bases&quot;, &#xA;&quot;cq&quot;, &quot;Coverage&quot;), class = &quot;data.frame&quot;, row.names = c(&quot;SMURF2_16_1&quot;, &#xA;&quot;MI0000437&quot;, &quot;SMAD6_4_1&quot;, &quot;SMAD2_8_1&quot;, &quot;MI0003561&quot;, &quot;SKP1_6_1&quot;, &#xA;&quot;MI0003556&quot;, &quot;LTBP1_19_1&quot;, &quot;MI0006350&quot;, &quot;MI0006351&quot;, &quot;TGFBR1_6_1&quot;, &#xA;&quot;MI0000772_1&quot;, &quot;MI0003176&quot;, &quot;MI0005758&quot;, &quot;MI0003639&quot;, &quot;MI0003641&quot;, &#xA;&quot;CUL1_5_1&quot;, &quot;APC_3_1&quot;, &quot;MI0005769&quot;, &quot;KPNB1_5_1&quot;))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When I use this data.frame like so (after loading the package mda):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mda(Coverage~.,data=dfr,subclasses=2)$fit$coef&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I get this result:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;                                [,1]        [,2]&#xA;Intercept                -0.90574883 -0.42382446&#xA;min_GCs_last_3_baseszero  1.80667985  0.85786905&#xA;min_GCs_last_3_basesone  -0.02302343  0.04884007&#xA;cq                                NA          NA&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What troubles me about this, is:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Why are there coefficients for both&#xA;the values of the factor&#xA;min_GCs_last_3_bases? &lt;/li&gt;&#xA;&lt;li&gt;Why are the    coefficients of cq NA?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I'm willing to assume I have completely misunderstood mda. Can someone please enlighten me? Or did I just bump into a bug?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(note: I've tried if the names of the columns had an influence: they don't)&lt;/p&gt;&#xA;" OwnerUserId="4257" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-18T04:52:58.743" LastActivityDate="2011-05-18T04:52:58.743" Title="Problem with mixture discriminant analysis in R returning NA for predictions" Tags="&lt;r&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="1" />
  <row Id="10919" PostTypeId="2" ParentId="10918" CreationDate="2011-05-17T20:09:15.070" Score="1" ViewCount="" Body="&lt;p&gt;You only have 2 failures Why were you thinking you could estimate more than two coefficients?&lt;/p&gt;&#xA;" OwnerUserId="2129" LastActivityDate="2011-05-17T20:09:15.070" CommentCount="2" />
  <row Id="10920" PostTypeId="1" CreationDate="2011-05-17T21:49:53.323" Score="1" ViewCount="88" Body="&lt;p&gt;I am looking for a statistical method to define the variance/ diversity / inequality in a set of observations. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example: &#xA;If I have following (n=4) observations using 4 data points, here the diversity/variance/inequality is zero. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;A-B-C-D&#xA;A-B-C-D&#xA;A-B-C-D&#xA;A-B-C-D &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In the following  (n=4) observations using 7 data points, there is x amount of diversity / variance / inequality in the observations. What method I can use to derive a diversity / variance score ?  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;A-B-B-D&#xA;Q-D-C-B&#xA;B-C-B-A&#xA;B-Z-F-A&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My real data is derived from a database of 10K data points and observations are usually &gt;= 2 - 1000s. Currently am looking at &lt;a href=&quot;http://en.wikipedia.org/wiki/Gini_coefficient#Advantages_of_Gini_coefficient_as_a_measure_of_inequality&quot; rel=&quot;nofollow&quot;&gt;Gini Impurity&lt;/a&gt; as a potential method, Do you think Gini Impurity is better for this type data ? Do you know about any better method which can derive a score by considering the order of the data ? Looking forward for your suggestions. &lt;/p&gt;&#xA;" OwnerUserId="529" LastEditorUserId="529" LastEditorDisplayName="" LastEditDate="2011-05-18T21:39:52.873" LastActivityDate="2011-05-19T06:10:19.527" Title="Statistical method to quantify  diversity / variance / inequality" Tags="&lt;variance&gt;&lt;diversity&gt;" AnswerCount="1" CommentCount="4" />
  <row Id="10921" PostTypeId="2" ParentId="10832" CreationDate="2011-05-17T21:54:51.667" Score="0" ViewCount="" Body="&lt;p&gt;You can use PhyFi web server for generating dendrograms from Newick files. &#xA;Sample output using your data from PhyFi: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.stack.imgur.com/wcoj8.png&quot; alt=&quot;PhyFi figure&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="529" LastActivityDate="2011-05-17T21:54:51.667" />
  <row Id="10922" PostTypeId="2" ParentId="10904" CreationDate="2011-05-17T21:57:27.727" Score="11" ViewCount="" Body="&lt;p&gt;This is an important question that I have given some thoughts over the years in my own teaching, and not only regarding distributions but also many other probabilistic and mathematical concepts. I don't know of any research that actually targets this question so the following is based on experience, reflection and discussions with colleagues. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;First it is important to realize that what motivates students to understand a fundamentally mathematical concept, such as a distribution and its mathematical properties, may depend on a lot of things and vary from student to student. Among math students in general I find that mathematically precise statements are appreciated and too much beating around the bush can be confusing and frustrating (hey, get to point man). That is &lt;strong&gt;not&lt;/strong&gt; to say that you shouldn't use, for example, computer simulations. On the contrary, they can be very illustrative of the mathematical concepts, and I know of many examples where computational illustrations of key mathematical concepts could help the understanding, but where the teaching is still old-fashioned math oriented. It is important, though, for math students that the precise math gets through.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, your question suggests that you are not so much interested in math students. If the students have some kind of computational emphasis, computer simulations and algorithms are really good for quickly getting an intuition about what a distribution is and what kind of properties it can have. The students need to have good tools for programming and visualizing, and I use R. This implies that you need to teach some R (or another preferred language), but if this is part of the course anyway, that is not really a big deal. If the students are not expected to work rigorously with the math afterwords, I feel comfortable if they get most of their understanding from algorithms and simulations. I teach bioinformatics students like that. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then for the students who are neither computationally oriented nor math students, it may be better to have a range of real and relevant data sets that illustrate how different kinds of distributions occur in their field. If you teach survival distributions to medical doctors, say, the best way to get their attention is to have a range of real survival data. To me, it is an open question whether a subsequent mathematical treatment or a simulation based treatment is best. If you haven't done any programming before, the practical problems of doing so can easily overshadow the expected gain in understanding. The students may end up learning how to write if-then-else statements but fail to relate this to the real life distributions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a general remark, I find that one of the really important points to investigate with  simulations is how distributions transform. In particular, in relation to test statistics. It is quite a challenge to understand that this single number you computed, the $t$-test statistic, say, from your entire data set has anything to do with a distribution. Even if you understand the math quite well. As a curious side effect of having to deal with multiple testing for microarray data, it has actually become much easier to show the students how the distribution of the test statistic pops up in real life situations. &lt;/p&gt;&#xA;" OwnerUserId="4376" LastActivityDate="2011-05-17T21:57:27.727" CommentCount="2" />
  <row Id="10923" PostTypeId="1" CreationDate="2011-05-18T00:38:18.920" Score="1" ViewCount="53" Body="&lt;p&gt;I am using few data sets available online and trying to visualize tree. However, it does not let me visualize tree option at all. Could anyone please guide me how to get the tree diagram in weka by using data sets available online?&lt;/p&gt;&#xA;" OwnerUserId="4641" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-18T07:29:55.577" LastActivityDate="2011-05-18T07:29:55.577" Title="Visualizing Weka classification tree" Tags="&lt;data-visualization&gt;&lt;classification&gt;&lt;decision-trees&gt;&lt;weka&gt;" AnswerCount="0" CommentCount="1" />
  <row Id="10925" PostTypeId="2" ParentId="10920" CreationDate="2011-05-18T03:43:24.807" Score="3" ViewCount="" Body="&lt;p&gt;I can't give a specific answer - mainly because the question isn't specific enough (happy to edit in due course though, given more information).  If the values A, B, C, D, E, etc. have a definite ordering that is meaningful to you (such as $A&gt;C&gt;B&gt;E&gt;Z&gt;\dots$) and you only want to describe the diversity that exists within your observations, then gini coefficient makes sense.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, if the values A, B, C, etc. are just arbitrary labels (i.e. they carry no information apart from distinguishing one type from another type) then a gini coefficient makes less sense - what is the relevant analogy to &quot;rich&quot; and &quot;poor&quot; if being &quot;rich&quot; can't be seen to be greater than being &quot;poor&quot;.  For Gini coefficient to work, you have to be able to order the observations&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A further consideration is if you wish to make inference about something which exists outside your sampled values.  For example, if the first set of 4 observations (which are the same) are part of a larger group which was not observed and you want to make a statement about the larger group.  Such a statement might be &lt;em&gt;the sample is not diverse&lt;/em&gt; &lt;strong&gt;therefore&lt;/strong&gt; &lt;em&gt;the group that the sample came from is not diverse&lt;/em&gt;.  Gini co-efficient may or may not be appropriate in this case, depending on what you know about how the larger group is related to the sample, and on how big your sample is compared to the larger group.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One way to think about the problem which may help is to consider the following scenarios.  If you were told that a set of observations A is diverse, but not given any observations from A, what would you predict them to be?  What if you were given the first observation only?  If you were told that the set A is more diverse than another set B, and you were given the observations from set B, what would you predict the observations in set A to be?  Thinking about the problem this way will help you to describe the features that your diversity measure should have (and features that it shouldn't have).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your data are categorical, then you could use tests based on a multinomial distribution with the number of trials equal to the number of categories per observation (4 in your examples) and the number of probability parameters equal to the number of data points (4 in your first example, and 7 in your second).  So, taking the second example we have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$x_{i}\sim Multinomial(4,\theta_{1},\theta_{2},\dots,\theta_{7})$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And a homogeneity score can be created by calculating the probability that all of the $\theta_{j}$ are equal.  However, in calculating this probability (which is not difficult) you will also end up calculate the probability of several other hypothesis (which are also useful), such as $\theta_{1}$ is different, all other $\theta_{j}$ are equal, that 2 are different, 3 are different, and so on up to all 7 are different.  I can post how you would do this, but I want to make sure this is something that you actually want first!  If you also cared about positioning (so that $A-B-C-D$ is considered different to $A-C-B-D$), then this can be incorporated by creating $28$ $\theta$ pararemters, and doing hypothesis tests about them.  So you would have:&#xA;$$x_{ik}\sim Multinomial(1,\theta_{1k},\theta_{2k},\dots,\theta_{7k})\;\;\;\;\;\;\;\;k=1,2,3,4$$&#xA;Admittedly you will need a reasonably sized data set in order to do this kind of test.  And you would have hypothesis about the $k$ index indicating &quot;ordering&quot; diversity, and hypothesis about the $j$ index indicating &quot;composition&quot; diversity.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-05-18T03:43:24.807" CommentCount="1" />
  <row Id="10926" PostTypeId="1" CreationDate="2011-05-18T04:44:30.650" Score="0" ViewCount="78" Body="&lt;p&gt;As question, I have found something similar &lt;a href=&quot;http://www.graphpad.com/quickcalcs/ConfInterval1.cfm&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;, but how to do it in R?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="588" LastActivityDate="2011-05-18T07:25:38.310" Title="How to calculate confidence interval for count data in R?" Tags="&lt;r&gt;&lt;confidence-interval&gt;&lt;count-data&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="10927" PostTypeId="2" ParentId="10910" CreationDate="2011-05-18T05:03:38.017" Score="1" ViewCount="" Body="&lt;p&gt;Here are two posts where I describe the process of computing scale scores for multiple-item, multiple-scale tests:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://jeromyanglim.blogspot.com/2009/10/scale-construction-item-reversal-scale.html&quot; rel=&quot;nofollow&quot;&gt;Scale construction and item reversal for multiple item scale in SPSS&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://jeromyanglim.blogspot.com/2009/10/calculating-scale-scores-for.html&quot; rel=&quot;nofollow&quot;&gt;Calculating scale scores in SPSS&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;There are many things to consider when creating scales (e.g., should the items be equally weighted? do you have missing data? do you want a mean or sum? etc.), and there are several tricks for doing it efficiently and reliably (e.g., using loops, using syntax, automatically generating syntax from metadata), but the posts above describe this in detail.&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2011-05-18T05:03:38.017" />
  <row Id="10928" PostTypeId="1" CreationDate="2011-05-18T07:17:28.760" Score="1" ViewCount="70" Body="&lt;p&gt;If i test two hypotheses, one of which has a null that is -in fact- false and the other has a null that is -in fact- true, I want to know the probability that the first test will obtain a p value less than that of the second, given other parameters such as delta, sigma, and sample size. I am not interested in whether one, or the other. or both, or neither are above some threshold, I'm interested in the probability that one is larger than the other.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can simulate the situation in R, and come up with a reasonable estimate that way, but I want to know how I can impute the answer exactly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using the program R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; a=vector()  &#xA;&amp;gt; b=vector()  &#xA;&amp;gt; for (i in 1:1000) {  &#xA;+ ai = rnorm(108, mean = 7, sd = 20)  &#xA;+ a = c(a,t.test(ai)$p.value)    &#xA;+ bi = rnorm(108)  &#xA;+ b = c(b,t.test(bi)$p.value)  &#xA;+ }  &#xA;&amp;gt; q=rep(0,length=length(a))  &#xA;&amp;gt; q[a &amp;lt; b]=1  &#xA;&amp;gt; mean(q)  &#xA;[1] 0.978&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So, I find that approximately 98% of the time, the truly alternative hypothesis yields a lower p value, but how can I impute this result. I want to prove it mathematically.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for your help,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Nick &lt;/p&gt;&#xA;&#xA;&lt;p&gt;@Nick Sabbe&#xA;What I think you are saying is that if the p value from the false null hypothesis equals a, then the probability that the p value from the true null hypothesis is less than a, is a. I get that, but what if I don't know the p value from the false null hypothesis test? What if I have two p values, and I know that exactly one of them comes from a hypothesis test in which the null is -in fact- false, what is the probability that it is the lower of the two p values? Assume all I know are the population parameters: delta, sigma, and N.&lt;/p&gt;&#xA;" OwnerUserId="4647" LastEditorUserId="4647" LastEditorDisplayName="" LastEditDate="2011-05-19T17:50:06.617" LastActivityDate="2011-05-19T20:48:57.070" Title="Probability that X&lt;=Y when X and Y are p values from two different hypotheses" Tags="&lt;probability&gt;&lt;testing&gt;&lt;hypothesis&gt;" AnswerCount="2" />
  <row Id="10929" PostTypeId="2" ParentId="10926" CreationDate="2011-05-18T07:25:38.310" Score="1" ViewCount="" Body="&lt;p&gt;You are looking for a confidence interval around the count from a Poisson process.  If you put for example 42 into your linked example you get &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;You observed 42 objects in a certain&#xA;  volume or 42 events in a certain time&#xA;  period.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Exact Poisson confidence interval: &lt;/p&gt;&#xA;  &#xA;  &lt;ul&gt;&#xA;  &lt;li&gt;The 90% confidence interval extends from 31.94 to 54.32 &lt;/li&gt;&#xA;  &lt;li&gt;The 95% confidence interval extends from 30.27 to 56.77&lt;/li&gt;&#xA;  &lt;li&gt;The 99% confidence interval extends from 27.18 to 61.76&lt;/li&gt;&#xA;  &lt;/ul&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;You can get this in R using &lt;code&gt;poisson.test&lt;/code&gt;.  For example&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; poisson.test(42, conf.level = 0.9 )&#xA;&#xA;        Exact Poisson test&#xA;&#xA;data:  42 time base: 1 &#xA;number of events = 42, time base = 1, p-value &amp;lt; 2.2e-16&#xA;alternative hypothesis: true event rate is not equal to 1 &#xA;90 percent confidence interval:&#xA; 31.93813 54.32395 &#xA;sample estimates:&#xA;event rate &#xA;        42  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and similarly the other values by changing &lt;code&gt;conf.level&lt;/code&gt;.  If you do not want all the background information, try something like &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; poisson.test(42, conf.level = 0.95 )$conf.int&#xA;[1] 30.26991 56.77180&#xA;attr(,&quot;conf.level&quot;)&#xA;[1] 0.95&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-05-18T07:25:38.310" />
  <row Id="10931" PostTypeId="2" ParentId="10537" CreationDate="2011-05-18T07:37:57.803" Score="1" ViewCount="" Body="&lt;p&gt;Because you are dealing with normal-normal model, its not to hard to work out analytically whats going on.  Now the standard argument for &quot;diffuse&quot; priors is usually $\frac{1}{\sigma}$ for variance parameters (&quot;jeffreys&quot; prior).  But you will be able to see that if you were to use jeffreys prior for both parameters, you would have an improper posterior.  But note that the main justification for using jeffreys prior is that it is a scale parameter.  However you can show for your model, that neither parameter sets the scale of the problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we consider the marginal model, with $\theta_{i}$ integrated out.  It is a well-known result that if you integrate a normal with another normal, you get a normal.  So we can skip the integration, and just work out the expectation and variance.  We then get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$E(y_{i}|\mu\sigma\sigma_{\theta})=E\left[E(y_{i}|\mu\sigma\sigma_{\theta}\theta_{i})\right]=E\left[\theta_{i}|\mu\sigma\sigma_{\theta}\right]=\mu$$&#xA;$$V(y_{i}|\mu\sigma\sigma_{\theta})=E\left[V(y_{i}|\mu\sigma\sigma_{\theta}\theta_{i})\right]+V\left[E(y_{i}|\mu\sigma\sigma_{\theta}\theta_{i})\right]=\sigma^{2}+\sigma_{\theta}^{2}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And hence we have the marginal model:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$(y_{i}|\mu\sigma\sigma_{\theta})\sim N(\mu,\sigma^{2}+\sigma_{\theta}^{2})$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And this does show an identifiability problem with this model - so the data cannot distinguish between the two variances, it can only give information about their sum.  You may have been able to see this intuitively.  For example, we can always take $\theta_{i}=y_{i}$ for all $i$ and hence this will set $\sigma=0$.  Alternatively we can set $\theta_{i}=\mu$ for all $i$ and this will set $\sigma_{\theta}=0$.  Both of these scenarios will be indistinguishable by the data - in the sense that if I was to generate two data sets, one from the first case, and one from the second (but ensured that $\sigma^{2}+\sigma_{\theta}^{2}$ was the same in both cases), you would not be able to tell which data set came from which case.  This suggests that it is fundamentally the sum that sets the scale and so we should apply jeffreys prior to the parameter $\tau^{2}=\sigma^{2}+\sigma_{\theta}^{2}$.  Now suppose that $\tau^{2}$ was known, I would have thought a non-informative choice of prior for $\sigma^{2}$ would be uniform between $0$ and $\tau^{2}$ (for a more informative choice I would use a re-scaled beta distribution over this range).  So we have the prior:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p(\tau^{2},\sigma^{2})\propto\frac{1}{\tau^{2}}\frac{I(0&amp;lt;\sigma^{2}&amp;lt;\tau^{2})}{\tau^{2}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we make the change of variables to $\sigma^{2},\tau^{2}\to\sigma,\sigma_{\theta}$ so that.  We then get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p(\sigma_{\theta},\sigma)\propto\frac{1}{(\sigma^{2}+\sigma_{\theta}^{2})^{2}}|\frac{\partial\sigma^{2}}{\partial\sigma}\frac{\partial\tau^{2}}{\partial\sigma_{\theta}}-\frac{\partial\sigma^{2}}{\partial\sigma_{\theta}}\frac{\partial\tau^{2}}{\partial\sigma}|&#xD;&#xA;=\frac{2\sigma\sigma_{\theta}}{(\sigma^{2}+\sigma_{\theta}^{2})^{2}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that the non-identifiability is preserved in this prior because it is symmetric in its arguments.  Another not so obvious symmetry is that if you were to integrate out either one of the variance parameters you would be left with the jeffreys prior for the other one:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\int_{0}^{\infty}\frac{2\sigma\sigma_{\theta}}{(\sigma^{2}+\sigma_{\theta}^{2})^{2}}d\sigma=\frac{1}{\sigma_{\theta}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hence, all you are required to input is the prior range for one of the parameters, as this will stop you from getting into trouble with improper priors.  Call this $0&amp;lt;L_{\sigma}&amp;lt;\sigma&amp;lt;U_{\sigma}&amp;lt;\infty$.  It is then easy to sample from the joint density using the inverse CDF method, for we have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$F_{\sigma}(x)=\frac{\log\left(\frac{x}{L_{\sigma}}\right)}{\log\left(\frac{U_{\sigma}}{L_{\sigma}}\right)}\implies F^{-1}_{\sigma}(p)=\frac{U_{\sigma}^{p}}{L_{\sigma}^{p-1}}$$&#xA;$$F_{\sigma_{\theta}|\sigma}(y|x)=1-\frac{x^{2}}{y^{2}+x^{2}}\implies F^{-1}_{\sigma_{\theta}|\sigma}(p|x)=x\sqrt{\frac{p}{1-p}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So you sample two independent uniform random variables $q_{1b},q_{2b}$, and then your random value of $\sigma^{(b)}=U_{\sigma}^{q_{1b}}L_{\sigma}^{1-q_{1b}}$ and your random value of $\sigma^{(b)}_{\theta}=U_{\sigma}^{q_{1b}}L_{\sigma}^{1-q_{1b}}\sqrt{\frac{q_{2b}}{1-q_{2b}}}$.  Combine this with the usual flat prior for $-\infty&amp;lt;L_{\mu}&amp;lt;\mu&amp;lt;U_{\mu}&amp;lt;\infty$ generated by a third random uniform variable $\mu^{(b)}=L_{\mu}+q_{3b}(U_{\mu}-L_{\mu})$ and you have all the ingredients to do monte carlo posterior simulation - note that this is much better than &quot;Gibbs sampling&quot; because each simulation is independent, so no need to wait for convergence (and also less need for a large number of simulations) - and you are dealing with proper priors - so divergence is impossible (however some moments may or may not exist, but all quantiles exist).&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-05-18T07:37:57.803" CommentCount="4" />
  <row Id="10932" PostTypeId="2" ParentId="10928" CreationDate="2011-05-18T07:45:35.023" Score="2" ViewCount="" Body="&lt;p&gt;The p-values for the true null hypothesis (Ha) should be uniformly distributed (see amongst others &lt;a href=&quot;http://stats.stackexchange.com/questions/10613/why-p-values-are-uniformly-distributed&quot;&gt;q10613&lt;/a&gt;). If your two tests are independent (which they seem to be from your example), the chance of the p-value of Hb, given that for Ha's (the non-true one) is a, is simply a.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, if you know the distribution of a, you may be able to integrate this out to find an analytical solution. But this depends upon your alternative, and upon which test you are using (for the false null hypothesis).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Extending the comment by @Henry and abusing notation somewhat:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$p(a&amp;lt;b) = \int p(a&amp;lt;b) da = \int (1-a) da = E(1-a)$&lt;/p&gt;&#xA;" OwnerUserId="4257" LastEditorUserId="4257" LastEditorDisplayName="" LastEditDate="2011-05-18T08:21:47.847" LastActivityDate="2011-05-18T08:21:47.847" CommentCount="2" />
  <row Id="10933" PostTypeId="2" ParentId="1252" CreationDate="2011-05-18T08:10:16.340" Score="2" ViewCount="" Body="&lt;p&gt;Here is a collection page:&#xA;&lt;a href=&quot;http://sasdataguru.blogspot.com/2011/05/online-statistics-cheat-sheet.html&quot; rel=&quot;nofollow&quot;&gt;http://sasdataguru.blogspot.com/2011/05/online-statistics-cheat-sheet.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="4648" LastActivityDate="2011-05-18T08:10:16.340" />
  <row Id="10934" PostTypeId="2" ParentId="10418" CreationDate="2011-05-18T09:16:25.957" Score="4" ViewCount="" Body="&lt;p&gt;Exchangeability is not an essential feature of a hierarchical model (at least not at the observational level).  It is basically a Bayesian analogue of &quot;independent and identically distributed&quot; from the standard literature.  It is simply a way of describing what you know about the situation at hand.  This is namely that &quot;shuffling&quot; does not alter your problem.  One way I like to think of this is to consider the case where you were given $x_{j}=5$ but you were not told the value of $j$.  If learning that $x_{j}=5$ would lead you to suspect particular values of $j$ more than others, then the sequence is not exchangeable.  If it tells you nothing about $j$, then the sequence is exchangeable.  Note that exhcangeability is &quot;in the information&quot; rather than &quot;in reality&quot; - it depends on what you know.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While exchangeability is not essential in terms of the observed variables, it would probably be quite difficult to fit any model without some notion of exchangeability, because without exchangeability you basically have no justification for pooling observations together.  So my guess is that your inferences will be much weaker if you don't have exchangeability somewhere in the model.  For example, consider $x_{i}\sim N(\mu_{i},\sigma_{i})$ for $i=1,\dots,N$.  If $x_{i}$ are completely exchangeable then this means $\mu_{i}=\mu$ and $\sigma_{i}=\sigma$.  If $x_{i}$ are conditionally exchangeable given $\mu_{i}$ then this means $\sigma_{i}=\sigma$.  If $x_{i}$ are conditionally exchangeable given $\sigma_{i}$ then this means $\mu_{i}=\mu$.  But note that in either of these two &quot;conditionally exchangeable&quot; cases, the quality of inference is reduced compared to the first, because there are an extra $N$ parameters that get introduced into the problem. If we have no exchangeability, then we basically have $N$ unrelated problems.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically exchangeability means we can make the inference $x_{i}\to \text{parameters}\to x_{j}$ for any $i$ and $j$ which are partly exchangeable&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-05-18T09:16:25.957" CommentCount="1" />
  <row Id="10935" PostTypeId="1" CreationDate="2011-05-18T10:28:52.960" Score="0" ViewCount="36" Body="&lt;p&gt;I have many (hundreds) excel documents with around 10 columns and 10 rows each.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My objective is to create &lt;em&gt;separate&lt;/em&gt; txt files containing the first column and the second column, then another file containing the first and the third and so on... and the same for the rest of excel files.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any way of doing this in Excel? Rather, would it be possible to apply a batch command in R to get into the Excel files (previously exported to CSV or of the kind) to produce separate txt files containing the pairing of columns?&lt;/p&gt;&#xA;" OwnerUserId="4651" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-18T10:39:11.333" LastActivityDate="2011-05-18T12:32:35.880" ClosedDate="2011-05-18T14:28:13.783" Title="Extracting columns from table Excel and merging them into another table" Tags="&lt;r&gt;&lt;excel&gt;&lt;csv-file&gt;" AnswerCount="0" CommentCount="1" />
  <row Id="10936" PostTypeId="1" AcceptedAnswerId="10940" CreationDate="2011-05-18T10:42:55.837" Score="2" ViewCount="72" Body="&lt;p&gt;I have some data in Stata which look like the first two columns of:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;group_id   var_to_rank  desired_rank&#xA;____________________________________&#xA;&#xA;1           10          1&#xA;1           20          2&#xA;1           30          3&#xA;1           40          4&#xA;2           10          1&#xA;2           20          2&#xA;2           20          2&#xA;2           30          3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'd like to create a rank of each observation within group (group_id) according to one variable (var_to_rank). Usually, for this purpose I used:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;gen id = _n&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However some of my observations (group_id = 2 in my small example) have the same values of ranking variable and this approach doesn't work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have also tried using:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;egen rank&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;command with different options, but cannot make my rank variables make to look like desired_rank.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could you point me to a solution to this problem?&lt;/p&gt;&#xA;" OwnerUserId="22" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-18T11:08:06.903" LastActivityDate="2011-05-18T13:41:19.027" Title="How can I rank observations within groups in Stata?" Tags="&lt;stata&gt;&lt;ranks&gt;" AnswerCount="2" />
  <row Id="10939" PostTypeId="2" ParentId="10936" CreationDate="2011-05-18T11:40:15.377" Score="4" ViewCount="" Body="&lt;p&gt;The following works for me:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;bysort group_id: egen desired_rank=rank(var_to_rank)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.stack.imgur.com/DU7Sq.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-05-18T11:40:15.377" CommentCount="5" />
  <row Id="10940" PostTypeId="2" ParentId="10936" CreationDate="2011-05-18T13:41:19.027" Score="3" ViewCount="" Body="&lt;p&gt;Stumbled upon such solution on the &lt;a href=&quot;http://statalist.1588530.n2.nabble.com/st-Ranking-observations-within-groups-td1643494.html&quot; rel=&quot;nofollow&quot;&gt;Statalist&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;bysort group_id (var_to_rank) : gen rank = var_to_rank != var_to_rank[_n-1]&#xA;by group_id : replace rank = sum(rank)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Seems to sort out this issue.&lt;/p&gt;&#xA;" OwnerUserId="22" LastActivityDate="2011-05-18T13:41:19.027" CommentCount="1" />
  <row Id="10941" PostTypeId="1" CreationDate="2011-05-18T14:47:27.453" Score="0" ViewCount="29" Body="&lt;p&gt;I am fighting with the installation of RMysql on R2.13.0 on Windows 7 64bit.  The version of my mysql 5.5. Here are the steps, which led me to an error:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Install RTools&lt;/li&gt;&#xA;&lt;li&gt;Set up proxy by calling: &lt;code&gt;setInternet2(TRUE)&lt;/code&gt; after &lt;a href=&quot;http://cran.r-project.org/bin/windows/base/rw-FAQ.html#The-Internet-download-functions-fail_002e&quot; rel=&quot;nofollow&quot;&gt;FAQ@CRAN&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Set MYSQL_HOME: &lt;code&gt;Sys.setenv(&quot;MYSQL_HOME&quot;=&quot;C:\\Program Files\\MySQL\\MySQL Server 5.5\\&quot;)&lt;/code&gt; after &lt;a href=&quot;http://stackoverflow.com/questions/4785933/adding-rmysql-package-to-r-fails/4792884#4792884&quot;&gt;this post on SO&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Call in R Console: &lt;code&gt;install.packages(&quot;RMySQL&quot;, type=&quot;source&quot;)&lt;/code&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;After 4, I got the following error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Installing package(s) into ‘C:/Users/Foo/Documents/R/win-library/2.13’&#xA;(as ‘lib’ is unspecified)&#xA;Warning: unable to access index for repository http://artfiles.org/cran/src/contrib&#xA;Warning message:&#xA;In getDependencies(pkgs, dependencies, available, lib) :&#xA;  package ‘RMySQL’ is not available (for R version 2.13.0)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Alternative to 4 is just to install the package: &lt;code&gt;install.packages(&quot;RMySQL&quot;)&lt;/code&gt;. In this case I got:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Installing package(s) into ‘C:/Users/foo/Documents/R/win-library/2.13’&#xA;(as ‘lib’ is unspecified)&#xA;Warning: unable to access index for repository http://artfiles.org/cran/bin/windows/contrib/2.13&#xA;Warning message:&#xA;In getDependencies(pkgs, dependencies, available, lib) :&#xA;  package ‘RMySQL’ is not available (for R version 2.13.0)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am sure that I do not have problems with connection, because I can call R update. &#xA;Could anybody help me to find out how I can access mysql db from R?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ps. Here is output from sessionInfo()&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;R version 2.13.0 (2011-04-13)&#xA;Platform: i386-pc-mingw32/i386 (32-bit)&#xA;&#xA;locale:&#xA;[1] LC_COLLATE=German_Germany.1252  LC_CTYPE=German_Germany.1252   &#xA;[3] LC_MONETARY=German_Germany.1252 LC_NUMERIC=C                   &#xA;[5] LC_TIME=German_Germany.1252    &#xA;&#xA;attached base packages:&#xA;[1] stats     graphics  grDevices utils     datasets  methods   base     &#xA;&#xA;loaded via a namespace (and not attached):&#xA;[1] tools_2.13.0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1389" LastActivityDate="2011-05-18T14:47:27.453" ClosedDate="2011-05-18T14:55:08.410" Title="How to install RMysql on Windows 7 with proxy settings" Tags="&lt;r&gt;" CommentCount="1" />
  <row Id="10942" PostTypeId="2" ParentId="173" CreationDate="2011-05-18T14:52:56.227" Score="0" ViewCount="" Body="&lt;p&gt;Escape from traditional enumerative statistics as Deming would suggest and venture into traditional analytical statistics - in this case, control charts.  See any books by Donald Wheeler PhD, particularly his &quot;Advanced Topics in SPC&quot; for more info.&lt;/p&gt;&#xA;" OwnerUserId="4652" LastActivityDate="2011-05-18T14:52:56.227" />
  <row Id="10943" PostTypeId="1" CreationDate="2011-05-18T14:59:03.320" Score="5" ViewCount="29" Body="&lt;p&gt;I have a mixed effect model (in fact a generalized additive mixed model) that gives me predictions for a timeseries. To counter the autocorrelation, I use a corCAR1 model, given the fact I have missing data. The data is supposed to give me a total load, so I need to sum over the whole prediction interval. But I should also get an estimate of the standard error on that total load.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If all predictions would be independent, this could be easily solved by :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Var(\sum^{n}_{i=1}E[X_i]) = \sum^{n}_{i=1}Var(E[X_i])$&#xA;with $Var(E[X_i]) = SE(E[X_i])^2$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Problem is, the predicted values are coming from a model, and the original data has autocorrelation. The whole problem leads to following questions :&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Am I correct in assuming that the SE on the calculated predictions can be interpreted as the root of the variance on the expected value of that prediction? I tend to interprete the predictions as &quot;mean predictions&quot;, and hence sum a whole set of means.&lt;/li&gt;&#xA;&lt;li&gt;How do I incorporate the autocorrelation in this problem, or can I safely assume that it wouldn't influence the results too much?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;This is an example in R. My real dataset has about 34.000 measurements, so scaleability is a problem. That's the reason why I model the autocorrelation within each month, otherwise the calculations aren't possible any more. It's not the most correct solution, but the most correct one isn't feasible.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(12)&#xA;require(mgcv)&#xA;&#xA;Data &amp;lt;- data.frame(&#xA;    dates = seq(as.Date(&quot;2011-1-1&quot;),as.Date(&quot;2011-12-31&quot;),by=&quot;day&quot;)&#xA;)&#xA;&#xA;Data &amp;lt;- within(Data,{&#xA;X &amp;lt;- abs(rnorm(nrow(Data),3))&#xA;Y &amp;lt;- 2*X + X^2 + scale(Data$dates)^2&#xA;month &amp;lt;- as.POSIXlt(dates)$mon+1&#xA;mday &amp;lt;- as.POSIXlt(dates)$mday&#xA;})&#xA;&#xA;model &amp;lt;- gamm(Y~s(X)+s(as.numeric(dates)),correlation=corCAR1(form=~mday|month),data=Data)&#xA;&#xA;preds &amp;lt;- predict(model$gam,se=T)&#xA;&#xA;Total &amp;lt;- sum(preds$fit)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Edit : &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lesson to learn : first go through all the samples in all the help files before panicking. In the help files of predict.gam, I can find :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#########################################################&#xA;## now get variance of sum of predictions using lpmatrix&#xA;#########################################################&#xA;&#xA;Xp &amp;lt;- predict(b,newd,type=&quot;lpmatrix&quot;) &#xA;&#xA;## Xp %*% coef(b) yields vector of predictions&#xA;&#xA;a &amp;lt;- rep(1,31)&#xA;Xs &amp;lt;- t(a) %*% Xp ## Xs %*% coef(b) gives sum of predictions&#xA;var.sum &amp;lt;- Xs %*% b$Vp %*% t(Xs)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Which seems to be close to what I want to do. This still doesn't tell me exactly how it is done. I could get as far as the fact that it's based on the linear predictor matrix. Any insights are still welcome.&lt;/p&gt;&#xA;" OwnerUserId="1124" LastEditorUserId="1124" LastEditorDisplayName="" LastEditDate="2011-05-19T14:12:19.330" LastActivityDate="2011-05-19T14:12:19.330" Title="Variance on the sum of predicted values from a mixed effect model on a timeseries" Tags="&lt;mixed-model&gt;&lt;variance&gt;&lt;random-variable&gt;" />
  <row Id="10944" PostTypeId="2" ParentId="10907" CreationDate="2011-05-18T15:34:41.480" Score="1" ViewCount="" Body="&lt;p&gt;The basic idea that you want is either the confidence interval on a predicted mean, or the prediction interval on an individual point.  Both formulas are found in any standard regression textbook and probably many places on the web.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Though deriving the correct pieces that you need for those formulas is probably a lot more work than is worth it.  Gnuplot is a fine plotting program, but is not a full statistics package.  A statistics package will give you the predictions fairly straight forward.  The &lt;a href=&quot;http://www.r-project.org/&quot; rel=&quot;nofollow&quot;&gt;R statistical package&lt;/a&gt; is the same general price as gnuplot.  In R you can fit your regression using the lm (linear model) function, then use the predict function to get either confidence or prediction intervals.  You could generate the intervals for a whole sequence of values, then either plot them directly in R, or transfer the predictions back to gnuplot and add them to your plot there.&lt;/p&gt;&#xA;" OwnerUserId="4505" LastActivityDate="2011-05-18T15:34:41.480" CommentCount="1" />
  <row Id="10945" PostTypeId="1" CreationDate="2011-05-18T16:20:16.897" Score="6" ViewCount="69" Body="&lt;p&gt;If an exploratory factor analysis is done with some 1-5 agreement items and some 0/1 &quot;choose all that apply&quot; items, theoretically how much of a spurious tendency would there be for the 1-5 items to load on one or two factors and the 0/1 items to load on a separate set of one or two factors?  (I've heard arguments both for and against the idea that correlations tend to be much higher among items of a similar scale.  My own experimentation/simulation has not found much of an effect.)&lt;/p&gt;&#xA;" OwnerUserId="2669" LastActivityDate="2011-05-28T12:45:24.507" Title="Should factor loadings be dominated by items' ranges of answer options?" Tags="&lt;factor-analysis&gt;" FavoriteCount="1" />
  <row Id="10946" PostTypeId="2" ParentId="10897" CreationDate="2011-05-18T16:41:17.197" Score="1" ViewCount="" Body="&lt;p&gt;Let the rod have length $L$ and fix a segment of length $x$.  The chance that any single breakpoint misses the segment equals the proportion of the rod not occupied by the segment, $1−x/L$.  Because the breakpoints are independent, the chance that all of them miss it is the product of $n$ such chances, $(1 - x/L)^n$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;From comments following the question, it appears that $x$ is intended to be small compared to the rod's length: $x/L \ll 1$.  Let $\xi = L/x$ (assumed to be large) and rewrite $n = \xi(n/\xi)$, leading (purely via substitutions) to&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\Pr(\text{all miss}) =  (1 - x/L)^n = (1 - 1/\xi)^{\xi(n/\xi)} = \left((1-1/\xi)^\xi\right)^{n/\xi}\text{.}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Asymptotically $\xi \to \infty$.  If we assume that $n$ varies in a way that makes $n/\xi$ converge to a constant, this probability approaches a computable limit.  Let this constant be some value $\lambda$ times $x$.  It is the limiting value of $n/\xi/x = n/L$: notice how the length of the rod is involved here and effectively is incorporated in $\lambda$.  Because $\exp(-1) = 1/e$ is the limiting value of $(1-1/\xi)^\xi$ and raising to (positive) powers is a continuous function, it follows readily that the limit is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\Pr(\text{all miss}) \to e^{-\lambda x}.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One application is when $n$ is a constant, entailing $\lambda = n/L$, and $x \ll L$.  We obtain $$e^{-nx/L}$$ as a good approximation for the probability that all breaks miss the segment.  This analysis shows that the approximation fails as $x$ grows large: the approximation is only as good as the approximation $1/e \sim (1-1/\xi)^\xi$.  Finally, if you set $x = L$, the approximation is clearly wrong because it gives $e^{-n}$ instead of the correct answer, $0$.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-05-18T16:41:17.197" CommentCount="3" />
  <row Id="10947" PostTypeId="1" AcceptedAnswerId="10948" CreationDate="2011-05-18T17:00:10.830" Score="6" ViewCount="115" Body="&lt;p&gt;I am trying to figure out how R computes lag-k autocorrelation (apparently, it is the same formula used by Minitab and SAS), so that I can compare it to using Excel's CORREL function applied to the series and its k-lagged version. R and Excel (using CORREL) give slightly different autocorrelation values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd also be interested to find out whether one computation is more correct than the other.&lt;/p&gt;&#xA;" OwnerUserId="1945" LastActivityDate="2011-05-18T18:42:20.133" Title="Formula for autocorrelation in R vs. Excel" Tags="&lt;r&gt;&lt;sas&gt;&lt;autocorrelation&gt;&lt;excel&gt;" AnswerCount="2" FavoriteCount="2" />
  <row Id="10948" PostTypeId="2" ParentId="10947" CreationDate="2011-05-18T17:23:01.437" Score="11" ViewCount="" Body="&lt;p&gt;The exact equation is given in: Venables, W. N. and Ripley, B. D. (2002) Modern Applied Statistics with S. Fourth Edition. Springer-Verlag. I'll give you an example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;### simulate some data with AR(1) where rho = .75&#xA;xi &amp;lt;- 1:50&#xA;yi &amp;lt;- arima.sim(model=list(ar=.75), n=50)&#xA;&#xA;### get residuals&#xA;res &amp;lt;- resid(lm(yi ~ xi))&#xA;&#xA;### acf for lags 1 and 2&#xA;cor(res[1:49], res[2:50])      ### not quite how this is calculated by R&#xA;cor(res[1:48], res[3:50])      ### not quite how this is calculated by R&#xA;&#xA;### how R calculates these&#xA;acf(res, lag.max=2, plot=F)&#xA;&#xA;### how this is calculated by R&#xA;### note: mean(res) = 0 for this example, so technically not needed here&#xA;c0 &amp;lt;- 1/50 * sum( (res[1:50] - mean(res)) * (res[1:50] - mean(res)) ) &#xA;c1 &amp;lt;- 1/50 * sum( (res[1:49] - mean(res)) * (res[2:50] - mean(res)) ) &#xA;c2 &amp;lt;- 1/50 * sum( (res[1:48] - mean(res)) * (res[3:50] - mean(res)) ) &#xA;c1/c0&#xA;c2/c0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And so on (e.g., &lt;code&gt;res[1:47]&lt;/code&gt; and &lt;code&gt;res[4:50]&lt;/code&gt; for lag 3).&lt;/p&gt;&#xA;" OwnerUserId="1934" LastActivityDate="2011-05-18T17:23:01.437" CommentCount="1" />
  <row Id="10949" PostTypeId="1" AcceptedAnswerId="11130" CreationDate="2011-05-18T18:36:14.577" Score="3" ViewCount="60" Body="&lt;p&gt;I want to estimate the impact of having health insurance on health care expenditures and have a hunch that it is best to use a two-part model to first estimate the probability of using any health care and then estimating the amount spent by those who used health care. However, I am not clear on the advantage of using a two-part model over just running ordinary least squares regression on the subsample of people who used any health care. Any insight into the advantages of the 2-part model over OLS on the subsample of users, and the differences in these two approaches would be much appreciated!&lt;/p&gt;&#xA;" OwnerUserId="834" LastActivityDate="2011-05-22T19:19:30.557" Title="What is the added value of using a 2-part model over an OLS model on a subsample when estimating health care expenditures?" Tags="&lt;regression&gt;&lt;probability&gt;&lt;logistic-regression&gt;" AnswerCount="3" CommentCount="3" />
  <row Id="10950" PostTypeId="2" ParentId="10947" CreationDate="2011-05-18T18:42:20.133" Score="6" ViewCount="" Body="&lt;p&gt;The naive way to calculate the auto correlation (and possibly what Excel uses) is to create 2 copies of the vector then remove the 1st n elements from the first copy and the last n elements from the second copy (where n is the lag that you are computing from).  Then pass those 2 vectors to the function to calculate the correlation.  This method is OK and will give a reasonable answer, but it ignores the fact that the 2 vectors being compared are really measures of the same thing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The improved version (as shown by Wolfgang) is a similar function to the regular correlation, except that it uses the entire vector for computing the mean and variance.&lt;/p&gt;&#xA;" OwnerUserId="4505" LastActivityDate="2011-05-18T18:42:20.133" CommentCount="4" />
  <row Id="10951" PostTypeId="1" CreationDate="2011-05-18T19:36:32.787" Score="5" ViewCount="84" Body="&lt;p&gt;I have a question concerning random variables. Let us assume that we have two random variables $X$ and $Y$. Let's say $X$ is Poisson distributed with parameter $\lambda_1$, and $Y$ is Poisson distributed with parameter $\lambda_2$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you build the fracture from $X/Y$ and call this a random variable $Z$, how is this distributed and what is the mean? Is it $\lambda_1/\lambda_2$?&lt;/p&gt;&#xA;" OwnerUserId="4496" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-18T20:06:01.823" LastActivityDate="2011-05-20T01:32:39.790" Title="What is the distribution of the ratio of two Poisson random variables?" Tags="&lt;random-variable&gt;&lt;poisson&gt;" AnswerCount="1" />
  <row Id="10952" PostTypeId="2" ParentId="10951" CreationDate="2011-05-18T19:47:22.720" Score="6" ViewCount="" Body="&lt;p&gt;I think you're going to have a problem with that.   Because variable Y will have zero's, X/Y will have some undefined values such that you won't get a distribution.&lt;/p&gt;&#xA;" OwnerUserId="2775" LastActivityDate="2011-05-18T19:47:22.720" CommentCount="6" />
  <row Id="10953" PostTypeId="1" AcceptedAnswerId="10971" CreationDate="2011-05-18T20:18:52.783" Score="2" ViewCount="62" Body="&lt;p&gt;I have a data set consisting of interactions between male-female dyads within a group in two conditions. The values range from -1 to 1 and indicate how responsible the female was for maintaining the relationship (Hinde's index). What I want to see is if the female responsibility changes in the two conditions. My understanding is that the Wilcoxon test is inappropriate because I have both paired and repeated measurements: each individual is part of several dyads.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Female1-Male1&lt;br&gt;&#xA;F1-M2&lt;br&gt;&#xA;F1-M3&lt;br&gt;&#xA;F2-M1....etc  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My other option was to run a permutation test, and I've been searching for one in R (I'm not conversant enough with R to create one myself), but so far the only ones I've found want my data to be integers, which is most definitely not the case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I suppose my two questions are&lt;br&gt;&#xA;1) Am I even on the right track here or should I be looking for a different test?&lt;br&gt;&#xA;2) Can anybody suggest a permutation test that can deal with decimal places?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="4655" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-18T20:27:06.850" LastActivityDate="2011-05-19T13:00:42.123" Title="Paired permutation test for repeated measures" Tags="&lt;repeated-measures&gt;&lt;permutation&gt;" AnswerCount="3" CommentCount="2" />
  <row Id="10954" PostTypeId="2" ParentId="10949" CreationDate="2011-05-18T20:47:46.283" Score="1" ViewCount="" Body="&lt;p&gt;Sarah, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was a little puzzled to think about whether there is a selection bias here. It seems that health insurance is bundled with other job benefits, and there is no self selection. However, those people who do not have health insurance are very different from those who do.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If there is a selection bias, OLS is biased and inconsistent. Selection bias is interpreted as a problem of omitted variables which can be corrected with the inverse mills ratio in your OLS. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Heckman's procedure gets unbiased OLS parameter estimates. Testing their significance requires additional work, since the estimates of the covariance matrix of the parameter estimates are inconsistent - use bootstrapping to derive the appropriate asymptotic standard errors and test statistics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See &lt;a href=&quot;http://en.wikipedia.org/wiki/Heckman_correction&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Heckman_correction&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;M&lt;/p&gt;&#xA;" OwnerUserId="4617" LastActivityDate="2011-05-18T20:47:46.283" CommentCount="3" />
  <row Id="10955" PostTypeId="2" ParentId="10953" CreationDate="2011-05-18T22:41:35.020" Score="0" ViewCount="" Body="&lt;p&gt;Check out the &lt;code&gt;ezPerm&lt;/code&gt; function from the &lt;a href=&quot;http://cran.r-project.org/web/packages/ez/index.html&quot; rel=&quot;nofollow&quot;&gt;ez package&lt;/a&gt; for R. For example, assuming your data is in &quot;long format&quot; (see the &lt;a href=&quot;http://cran.r-project.org/web/packages/reshape2/index.html&quot; rel=&quot;nofollow&quot;&gt;reshape2&lt;/a&gt; package):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ezPerm(&#xA;    data = my_data&#xA;    , dv = .(the_dv)&#xA;    , wid = .(dyad_num)&#xA;    , within = .(sex)&#xA;    , perms = 1e3&#xA;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="364" LastActivityDate="2011-05-18T22:41:35.020" />
  <row Id="10957" PostTypeId="2" ParentId="10949" CreationDate="2011-05-19T04:26:13.363" Score="1" ViewCount="" Body="&lt;p&gt;I think you are assuming something like this. People use health care if they are sick. If they use health care, than they spend more money on it. But, sick people will see more value on insurance and will be more likey to be insured. So, you will find out that insured people spends more money on health care, but this may be so due to selection bias.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And you seem to think that, if you use only the subsample of people who used health care, than you don't have this problem. However, think of this example. Assume you have a woman who wants to be pregnant and a man who doesn't want to have a child, both on their 30's. The woman than gets insured, but the man not. Both get sick and use health care, so they both are in your subsample. However, they have different probabilities of being insured and they will spend different amount of money if she is pregnant. Thus selection bias will occur anyway.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, what I'm saying is that restricting your subsample will not solve the selection bias problem. You still have to account for the different probabilities of people getting insurance.&lt;/p&gt;&#xA;" OwnerUserId="3058" LastActivityDate="2011-05-19T04:26:13.363" CommentCount="2" />
  <row Id="10958" PostTypeId="1" AcceptedAnswerId="11002" CreationDate="2011-05-19T06:01:21.953" Score="3" ViewCount="76" Body="&lt;p&gt;I am trying to increase my model accuracy by taking into account interaction effect of relevant variables. &#xA;I am choosing variables to interact based more on the common sense than on trying every combination. &#xA;So far most interaction effects with good p value ( p&amp;lt;.0001) and chi-square have resulted in an increase of true positives.&#xA;However, the last interaction effect, while having good p values and chi square values,  resulted in decreased true positives by 10. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Should this happen?&lt;/li&gt;&#xA;&lt;li&gt;How do I interpret this?&lt;/li&gt;&#xA;&lt;li&gt;Shouldnt a variable which is significant always increase my true positives?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="1763" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-19T06:58:31.413" LastActivityDate="2011-05-19T21:55:58.340" Title="Why does adding statistically significant interaction reduce true positives?" Tags="&lt;interaction&gt;" AnswerCount="2" />
  <row Id="10959" PostTypeId="2" ParentId="10958" CreationDate="2011-05-19T06:51:10.817" Score="6" ViewCount="" Body="&lt;p&gt;No it should not. e.g. for logistic regression, which appears to be the case here, it may be that the greater p-value (I'm assuming you mean the right kind of p-value here, like from a likelihood ratio test) comes from increasing the odds for (correctly predicted) observations that are already relatively extreme (e.g. all observations that had predicted probability 0.8 become 0.9 and all those that had 0.2 become 0.1), while the ones that were only just on the right side of the 50% threshold are now just on the other side. As a result, the extremities are now predicted with more confidence, but there are more misclassifications.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, good fit does not guarantee good prediction (or the other way around) - even though that's the way most 'scientific' publications work these days :-(&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would advise you to look into a more evolved technique like LASSO or elastic net for variable selection... It will also easily allow you to optimize for some predictive measure like missclassification. In R, try glmnet.&lt;/p&gt;&#xA;" OwnerUserId="4257" LastActivityDate="2011-05-19T06:51:10.817" CommentCount="1" />
  <row Id="10960" PostTypeId="2" ParentId="10953" CreationDate="2011-05-19T07:41:35.067" Score="0" ViewCount="" Body="&lt;p&gt;Since you say each pair was observed in both conditions, you can use the basic trick of the paired t-test: subtract the values for the same pair in both conditions and then test for this to be zero (or if that's more relevant for this kind of measure: divide them and test for this to be 1).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At the least, this reduces the complexity of your problem.&lt;/p&gt;&#xA;" OwnerUserId="4257" LastActivityDate="2011-05-19T07:41:35.067" />
  <row Id="10961" PostTypeId="1" CreationDate="2011-05-19T08:23:46.003" Score="2" ViewCount="40" Body="&lt;p&gt;I am trying to model &lt;code&gt;steel&lt;/code&gt; prices using &lt;code&gt;brent&lt;/code&gt; prices with following model:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$steel_t=\alpha + \beta steel_{t-1}+\gamma brent_t + \epsilon_t$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have monthly data. I fitted the parameter values with &lt;code&gt;lm&lt;/code&gt; (R, is that reasonable?).  Now I want to see the effect of a &quot;shock event&quot; in one year, e.g. what happens to steel prices  if brent prices go to USD 50 in one year (it is about USD 100 nowadays). How can I do it? Does it make sense to use an autoregressive model for such analysis?&lt;/p&gt;&#xA;" OwnerUserId="1443" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-19T08:26:25.030" LastActivityDate="2011-05-19T11:55:52.667" Title="Autoregressive model and shock events" Tags="&lt;r&gt;&lt;time-series&gt;" AnswerCount="1" CommentCount="4" FavoriteCount="1" />
  <row Id="10962" PostTypeId="2" ParentId="9334" CreationDate="2011-05-19T08:55:18.997" Score="0" ViewCount="" Body="&lt;p&gt;I'm also interested in distinguishing by coefficient not by heart between linear and exponential growth.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My book inquiries resulted in least square fitting using the regression coefficient to determine the quality of fit.&#xA;I haven't found a method for differing between linear and exponential growth.&#xA;The least square fitting and regression coefficient seems to be biased, as in the case of a true/real exponential relationship in the data set, the residuum of the big values dominates the one of the small values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To my humble opinion the most clear criterion for the distinction between exponential and linear growth is the change of the derivative.&#xA;By splitting the data set in two or more segments a linear fit performed on each of these segments will answer the question about the change of derivative.&#xA;A criterion for the least significant exponential growth rate has to be specified depending on the problem and data quality.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If there is a better method, please let me know&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Johannes&lt;/p&gt;&#xA;" OwnerUserId="4662" LastActivityDate="2011-05-19T08:55:18.997" />
  <row Id="10963" PostTypeId="1" AcceptedAnswerId="10967" CreationDate="2011-05-19T10:14:30.060" Score="1" ViewCount="39" Body="&lt;p&gt;I have data like Person $A$ like movies &lt;code&gt;['X','Y', 'Z']&lt;/code&gt; and he dislikes &lt;code&gt;['V']&lt;/code&gt;. Person $B$ like movies &lt;code&gt;['X','L','V']&lt;/code&gt; and  dislikes &lt;code&gt;['Y']&lt;/code&gt;. like wise so many users. What could be a good algorithm to find mean difference of users' tastes?&lt;/p&gt;&#xA;" OwnerUserId="4665" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-19T10:33:22.753" LastActivityDate="2011-05-19T11:45:45.760" Title="Algorithm to calculate difference in users' tastes" Tags="&lt;algorithms&gt;&lt;matching&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="10964" PostTypeId="1" AcceptedAnswerId="10972" CreationDate="2011-05-19T10:43:13.270" Score="6" ViewCount="86" Body="&lt;p&gt;I was presenting proofs of WLLN and a version of SLLN (assuming bounded 4th central moment) when somebody asked which measure is the probability with respect too and I realised that, on reflection, I wasn't quite sure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems that it is straightforward, since in both laws we have a sequence of $X_{i}$'s, independent RVs with identical mean and finite variance. There is only one random variable in sight, namely the $X_{i}$, so the probability must be w.r.t the distribution of the $X_{i}$, right? But then that doesn't seem quite right for the strong law since the typical proof technique is then to define a new RV $S_{n} := \sum_{i=1}^{n} X_{i}$ and work with that, and the limit is &lt;em&gt;inside&lt;/em&gt; the probability:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$&#xD;&#xA;Pr \left[\lim_{n \rightarrow \infty}\frac{1}{n}\sum_{i=1}^{n} X_{i} = E[X_{i}]\right]=1&#xD;&#xA;$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So now it looks as if the RV is the sums over $n$ terms, so the probability is over the distribution of the sums $S_{n}$, where $n$ is no longer fixed. Is that correct? If it is, how would we go about constructing a suitable probability measure on the sequences of partial sums?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Happy to receive intuitive responses as to what is going on as well as formal ones using e.g. real or complex analysis, undergrad probability/statistics, basic measure theory. I've read &lt;a href=&quot;http://stats.stackexchange.com/questions/2230&quot;&gt;Convergence in probability vs. almost sure convergence&lt;/a&gt; and associated links, but find no help there.&lt;/p&gt;&#xA;" OwnerUserId="3248" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-19T12:46:17.573" LastActivityDate="2011-05-19T13:28:30.873" Title="In convergence in probability or a.s. convergence w.r.t which measure is the probability?" Tags="&lt;probability&gt;&lt;random-variable&gt;&lt;probability-theory&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="2" />
  <row Id="10965" PostTypeId="1" CreationDate="2011-05-19T11:03:36.530" Score="0" ViewCount="87" Body="&lt;p&gt;I'm using the &lt;code&gt;ets&lt;/code&gt; forecast function in R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I fit a model to some timeseries t1:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   model&amp;lt;-ets(t1) [36 periods]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and the calculate forecasts from that model:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    f1 &amp;lt;- forecast(model,10)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;so i get 10 forecasts for periods 37-48&lt;/p&gt;&#xA;&#xA;&lt;p&gt;so my question is, are these 10 point-forecast one-step-ahead forecasts wich&#xA;have their seeds in $t,t+1,t+2$ with $t=37$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;or are these forecasts with their seed only in $t=37$ with forecast horizon $h=1,2,3,4,...$&lt;/p&gt;&#xA;" OwnerUserId="4666" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-21T18:45:41.890" LastActivityDate="2011-05-21T18:45:41.890" Title="Problem with ets from R forecast package" Tags="&lt;r&gt;&lt;forecasting&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="10966" PostTypeId="2" ParentId="10963" CreationDate="2011-05-19T11:07:59.960" Score="0" ViewCount="" Body="&lt;p&gt;If you represent each movie as a categorical variable with 3 levels (like, unspecified, dislike) you can do any type of clustering analysis on your users with these covariates.&lt;/p&gt;&#xA;" OwnerUserId="4257" LastActivityDate="2011-05-19T11:07:59.960" CommentCount="2" />
  <row Id="10967" PostTypeId="2" ParentId="10963" CreationDate="2011-05-19T11:28:41.813" Score="0" ViewCount="" Body="&lt;p&gt;What you want to do is called &quot;Collaborative Filtering&quot;. Searching the web will offer you a tremendous amount of resources for this topic, but I truly recommend this paper:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.hindawi.com/journals/aai/2009/421425/&quot; rel=&quot;nofollow&quot;&gt;Xiaoyuan Su &amp;amp; Taghi M. Khoshgoftaar: A Survey of Collaborative Filtering Techniques&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In section &lt;em&gt;3. Memory-Based Collaborative Filtering Techniques&lt;/em&gt; you'll find the basic  techniques to find users with similar taste. They generally consist of selecting a metric for a nearest-neighor-approach plus some modifications on the user-item-matrix (how to deal with missing values / how to treat items which have not been rated this often etc.).This is a good point to start. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;After grasping the basic ideas you may want to try out some sophisticated techniques like Singular Value Decomposition, which has been successfully applied in the &lt;a href=&quot;http://en.wikipedia.org/wiki/Netflix_Prize&quot; rel=&quot;nofollow&quot;&gt;Netflix-Price&lt;/a&gt; (you'll find a link to this and other techniques in section &lt;em&gt;4. Model-Based Collaborative Filtering Techniques of the recommended paper&lt;/em&gt;). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have some bucks to spend, I also recommend &quot;Programming Collective Intelligence&quot; by Toby Segaran, which approaches this topic in a very very practical way. &lt;/p&gt;&#xA;" OwnerUserId="264" LastEditorUserId="264" LastEditorDisplayName="" LastEditDate="2011-05-19T11:45:45.760" LastActivityDate="2011-05-19T11:45:45.760" />
  <row Id="10968" PostTypeId="2" ParentId="10965" CreationDate="2011-05-19T11:28:45.260" Score="3" ViewCount="" Body="&lt;p&gt;They can't possibly be one-step forecasts because you haven't provided any data for t&gt;36. They are forecasts of times 37,...,46 based on data up to time 36 (i.e., horizons 1,2,3,...,10).&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2011-05-19T11:28:45.260" />
  <row Id="10970" PostTypeId="2" ParentId="10961" CreationDate="2011-05-19T11:55:52.667" Score="1" ViewCount="" Body="&lt;p&gt;First of all, I'd suggest using the &lt;code&gt;arima&lt;/code&gt; function instead and explicitly fitting a AR(1) model. I don't think it will change your result, but it should properly handle the error correlations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you have that, you can set up your predicted values of &lt;code&gt;brent&lt;/code&gt; and just run the model on that.&lt;/p&gt;&#xA;" OwnerUserId="1397" LastActivityDate="2011-05-19T11:55:52.667" />
  <row Id="10971" PostTypeId="2" ParentId="10953" CreationDate="2011-05-19T13:00:42.123" Score="1" ViewCount="" Body="&lt;p&gt;There is the possibility of using the &lt;code&gt;coin&lt;/code&gt; package for this type of stuff. &lt;a href=&quot;http://cran.r-project.org/web/packages/coin/index.html&quot; rel=&quot;nofollow&quot;&gt;See its webpage&lt;/a&gt; and &lt;a href=&quot;http://stats.stackexchange.com/questions/6127/which-permutation-test-implementation-in-r-to-use-instead-of-t-tests-paired-and&quot;&gt;the accepted answer to this question&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An implementation for this type of stuff would be the following.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#load the package &#xA;require(coin)&#xA;&#xA;# Some toy data:&#xA;s.data &amp;lt;- data.frame(dyad = c(&quot;F1-M1&quot;, &quot;F1-M2&quot;,&quot;F2-M1&quot;,&quot;F2-M2&quot;), condition = c(rep(&quot;A&quot;, 4), rep(&quot;B&quot;, 4)), dv = runif(8,0,1)) &#xA;&#xA;# Make sure the factors are really factors!&#xA;str(s.data)&#xA;&#xA;# here goes the permutation test. &#xA;oneway_test(dv ~ condition | dyad, distribution = approximate(B=10000), data = s.data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(Note that you need to use an high number for B in your real example)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furthermore, you could also use a paired t.test. I don't see any reasons against it:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;t.test(subset(s.data, condition == &quot;A&quot;, &quot;dv&quot;, drop = TRUE), subset(s.data, condition == &quot;B&quot;, &quot;dv&quot;, drop = TRUE), paired = TRUE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;I think there are is one import thing you haven't discussed so far: With this implementation you would have controlled for the dyads, but not for the individuals (e.g., F1, independent of her male interaction partner). This is a serious threat to your analysis.&lt;br&gt;&#xA;I know that there is a bunch of stuff an dyad analyses in psychology and related fields. Unfortunately I cannot give you a real pointer. But you should definitely check this stuff out before finishing your analyses. A quick search on rseek.org returns at least a package called &lt;code&gt;dyad&lt;/code&gt; and &lt;a href=&quot;http://www.davidakenny.net/dyad.htm&quot; rel=&quot;nofollow&quot;&gt;this webpage&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="442" LastActivityDate="2011-05-19T13:00:42.123" CommentCount="1" />
  <row Id="10972" PostTypeId="2" ParentId="10964" CreationDate="2011-05-19T13:28:30.873" Score="7" ViewCount="" Body="&lt;p&gt;The probability measure is the same in both cases, but the question of interest is different between the two.  In both cases we have a (countably) infinite sequence of random variables defined on a the single probability space $(\Omega,\mathscr{F},P)$.  We take $\Omega$, $\mathscr{F}$ and $P$ to be the infinite products in each case (care is needed, here, that we are talking about only probability measures because we can run into troubles otherwise). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the SLLN, what we care about is the probability (or measure) of the set of all $\omega = (\omega_{1},\omega_{2},\ldots)$ where the scaled partial sums DO NOT converge.  This set has measure zero (w.r.t. $P$), says the SLLN.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the WLLN, what we care about is the behavior of the sequence of projection measures $\left(P_{n}\right)_{n=1}^{\infty}$, where for each $n$, $P_{n}$ is the projection of $P$ onto the finite measureable space $\Omega_{n} = \prod_{i=1}^{n} \Omega_{i}$.  The WLLN says that the (projected) probability of the cylinders (that is, events involving $X_{1},\ldots,X_{n}$), on which the scaled partial sums do not converge, goes to zero in the limit as $n$ goes to infinity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the WLLN we are calculating probabilities which appear removed from the infinite product space, but it never actually went away - it was there all along.  All we were doing was projecting onto the subspace from 1 to $n$ and then taking the limit afterward.  That such a thing is possible, that it is possible to construct a probability measure on an infinite product space such that the projections for each $n$ match what we think they should, and do what they're supposed to do, is one of the consequences of &lt;a href=&quot;http://en.wikipedia.org/wiki/Kolmogorov_extension_theorem&quot; rel=&quot;nofollow&quot;&gt;Kolmogorov's Extension Theorem&lt;/a&gt; .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you'd like to read more, I've found the most detailed discussion of subtle points like these in &quot;Probability and Measure Theory&quot; by Ash, Doleans-Dade.  There are a couple others, but Ash/D-D is my favorite.&lt;/p&gt;&#xA;" OwnerUserId="1108" LastActivityDate="2011-05-19T13:28:30.873" CommentCount="6" />
  <row Id="10973" PostTypeId="2" ParentId="3" CreationDate="2011-05-19T13:31:00.567" Score="1" ViewCount="" Body="&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://code.google.com/p/clusterpy/&quot; rel=&quot;nofollow&quot;&gt;clusterPy&lt;/a&gt; for analytical&#xA;regionalization or geospatial&#xA;clustering&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://geodacenter.asu.edu/pysal&quot; rel=&quot;nofollow&quot;&gt;PySal&lt;/a&gt; for spatial data analysis.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="4329" LastEditorUserId="4329" LastEditorDisplayName="" LastEditDate="2011-06-02T05:25:33.257" LastActivityDate="2011-06-02T05:25:33.257" CommunityOwnedDate="2011-05-19T13:31:00.567" />
  <row Id="10974" PostTypeId="1" AcceptedAnswerId="10980" CreationDate="2011-05-19T13:34:08.113" Score="4" ViewCount="29" Body="&lt;p&gt;For an upcoming study of about 200 (rare) cancer cases, we would like to determine the power of detection for a hypothetical marker, present in 10% of cases, i.e. 20 cases with a hazard ratio of 3.0. Cases will be followed up for at least 5 years. We will have the option to validate any identified markers in an independent additional 200 member cohort. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the past, I have used the program &lt;a href=&quot;http://biostat.mc.vanderbilt.edu/twiki/bin/view/Main/PowerSampleSize&quot; rel=&quot;nofollow&quot;&gt;PS&lt;/a&gt; to run power calculations, which is fairly idiot-proof for a biologist like me. However, the cancer we are studying tends to have an overall survival rate of 60%, so the calculation of a median survival time doesn't really make sense (at least to me).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I wanted to know is whether there are approaches available for the calculation of power which can overcome this limitation, and if so, how can I easily implement them in, ideally, R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have googled around to find an answer, but haven't met with success so far.&lt;/p&gt;&#xA;" OwnerUserId="3429" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-19T14:59:26.263" LastActivityDate="2011-05-19T15:15:25.510" Title="Calculation of power of survival study" Tags="&lt;survival&gt;&lt;median&gt;&lt;power-analysis&gt;" AnswerCount="1" />
  <row Id="10975" PostTypeId="1" AcceptedAnswerId="10979" CreationDate="2011-05-19T13:48:08.743" Score="6" ViewCount="89" Body="&lt;p&gt;Is there a (stronger?) alternative to the arcsin square root transformation for percentage/proportion data? In the data set I'm working on at the moment, marked &#xA;heteroscedasticity remains after I apply this transformation, i.e. the plot of residuals vs. fitted values is still very much rhomboid. Thanks in advance for any suggestions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edited to respond to comments: the data are investment decisions by experimental participants who may invest 0-100% of an endowment in multiples of 10%.  I have also looked at these data using ordinal logistic regression, but would like to see what a valid glm would produce.  Plus I could see the answer being useful for future work, as arcsin square root seems to be used as a one-size-fits all solution in my field and I hadn't come across any alternatives being employed.  &lt;/p&gt;&#xA;" OwnerUserId="266" LastEditorUserId="266" LastEditorDisplayName="" LastEditDate="2011-05-20T11:08:26.173" LastActivityDate="2011-05-20T11:08:26.173" Title="Transforming proportion data: when arcsin square root is not enough" Tags="&lt;data-transformation&gt;&lt;glm&gt;&lt;heteroscedasticity&gt;" AnswerCount="2" CommentCount="3" FavoriteCount="1" />
  <row Id="10976" PostTypeId="2" ParentId="6314" CreationDate="2011-05-19T14:06:47.853" Score="5" ViewCount="" Body="&lt;p&gt;I don't have a full answer to this question, but I can give a partial answer on some of the analytical aspects. Warning: I've been working on other problems since the first paper below, so it's very likely there is other good stuff out there I'm not aware of.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First I think it's worth noting that despite the title of their paper &quot;When is `nearest neighbor' meaningful&quot;, Beyer et al actually answered a different question, namely when is NN &lt;em&gt;not&lt;/em&gt; meaningful. We proved the converse to their theorem, under some additional mild assumptions on the size of the sample, in &lt;a href=&quot;http://www.cs.bham.ac.uk/~durranrj/JCOM_article.pdf&quot; rel=&quot;nofollow&quot;&gt;When Is 'Nearest Neighbor' Meaningful: A Converse Theorem and Implications. Journal of Complexity, 25(4), August 2009, pp 385-397.&lt;/a&gt; and showed that there are situations when (in theory) the concentration of distances will not arise (we give examples, but in essence the number of non-noise features needs to grow with the dimensionality so of course they seldom arise in practice).&#xA;The references 1 and 7 cited in our paper give some examples of ways in which the distance concentration can be mitigated in practice.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A paper by my supervisor, Ata Kaban, looks at whether these distance concentration issues persist despite applying dimensionality reduction techniques in  &lt;a href=&quot;http://www.cs.bham.ac.uk/~axk/dred2.pdf&quot; rel=&quot;nofollow&quot;&gt;On the Distance Concentration Awareness of Certain Data Reduction Techniques. Pattern Recognition. Vol. 44, Issue 2, Feb 2011, pp.265-277.&lt;/a&gt;. There's some nice discussion in there too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A recent paper by Radovanovic et al &lt;a href=&quot;http://jmlr.csail.mit.edu/papers/volume11/radovanovic10a/radovanovic10a.pdf&quot; rel=&quot;nofollow&quot;&gt;Hubs in Space: Popular Nearest Neighbors in High-Dimensional Data. JMLR, 11(Sep), September 2010, pp:2487−2531. &lt;/a&gt; discusses the issue of &quot;hubness&quot;, that is when a small subset of points belong to the $k$ nearest neighbours of many of the labelled observations. See also the first author's PhD thesis, which is on the web.&lt;/p&gt;&#xA;" OwnerUserId="3248" LastActivityDate="2011-05-19T14:06:47.853" CommentCount="4" />
  <row Id="10977" PostTypeId="1" AcceptedAnswerId="10978" CreationDate="2011-05-19T14:23:29.600" Score="0" ViewCount="45" Body="&lt;p&gt;I am looking for an easy to use stand alone software that is able to construct &lt;a href=&quot;http://en.wikipedia.org/wiki/Bayesian_network&quot; rel=&quot;nofollow&quot;&gt;Bayesian belief networks&lt;/a&gt; out of data. The software should (of course ;-) be free.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anybody recommend something? Thank you!&lt;/p&gt;&#xA;" OwnerUserId="230" LastActivityDate="2011-05-19T14:46:47.357" Title="Software for learning Bayesian belief networks" Tags="&lt;bayesian&gt;&lt;software&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="1" />
  <row Id="10978" PostTypeId="2" ParentId="10977" CreationDate="2011-05-19T14:46:47.357" Score="1" ViewCount="" Body="&lt;p&gt;Your own link contains a load of free tools (near the bottom: software resources), and you can check the bayesian task view at &lt;a href=&quot;http://cran.r-project.org/&quot; rel=&quot;nofollow&quot;&gt;CRAN&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="4257" LastActivityDate="2011-05-19T14:46:47.357" />
  <row Id="10979" PostTypeId="2" ParentId="10975" CreationDate="2011-05-19T14:49:19.280" Score="6" ViewCount="" Body="&lt;p&gt;Sure.  John Tukey describes a family of transformations in &lt;em&gt;&lt;a href=&quot;http://www.amazon.com/Exploratory-Data-Analysis-Wilder-Tukey/dp/B0007347RW&quot; rel=&quot;nofollow&quot;&gt;EDA&lt;/a&gt;&lt;/em&gt;.  It is based on these ideas:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;To be able to extend the tails (towards 0 and 1) as controlled by a parameter.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Nevertheless, to match the original (untransformed) values near the middle (0.5), which makes the transformation easier to interpret.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;To make the re-expression symmetric about 0.5.  That is, if $p$ is re-expressed as $f(p)$, then $1-p$ will be re-expressed as $-f(p)$.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;If you begin with any increasing monotonic function $g: (0,1) \to \mathbb{R}$ differentiable at $1/2$ you can adjust it to meet the second and third criteria: just define&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$f(p) = \frac{g(p) - g(1-p)}{2g&amp;#39;(1/2)}.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tukey calls this the &quot;folded&quot; version of $g$.  His family consists of the power and log transformations $g(p) = p^\lambda$ where, when $\lambda=0$, we consider $g(p) = \log(p)$.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's look at some examples.  When $\lambda = 1/2$ we get the folded root, or &quot;froot,&quot; $f(p) = \sqrt{1/2}\left(\sqrt{p} - \sqrt{1-p}\right)$.  When $\lambda = 0$ we have the folded logarithm, or &quot;flog,&quot; $f(p) = (\log(p) - \log(1-p))/4.$  Evidently this is just a constant multiple of the &lt;em&gt;logit&lt;/em&gt; transformation, $\log(\frac{p}{1-p})$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.stack.imgur.com/5fn3k.png&quot; alt=&quot;Graphs for lambda=1, 1/2, 0, and arcsin&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;In this graph the blue line corresponds to $\lambda=1$, the intermediate red line to $\lambda=1/2$, and the extreme green line to $\lambda=0$.  The dashed gold line is the arcsine transformation, $\arcsin(2p-1)/2 = \arcsin(\sqrt{p}) - \arcsin(\sqrt{1/2})$.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The most useful values of the parameter $\lambda$ lie between $1$ and $0$.  (You can make the tails even heavier with negative values of $\lambda$, but this use is rare.)  $\lambda=1$ doesn't do anything at all except recenter the values ($f(p) = p-1/2$).  As $\lambda$ shrinks towards zero, the tails get pulled further towards $\pm \infty$.  This satisfies criterion #1.  Thus, by choosing an appropriate value of $\lambda$, you can control the &quot;strength&quot; of this re-expression in the tails.&lt;/p&gt;&#xA;" OwnerUserId="919" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-05-19T16:36:55.463" LastActivityDate="2011-05-19T16:36:55.463" CommentCount="4" />
  <row Id="10980" PostTypeId="2" ParentId="10974" CreationDate="2011-05-19T15:15:25.510" Score="5" ViewCount="" Body="&lt;p&gt;In the R Hmisc package see the cpower and spower functions.  spower does simulations for complex situations (late treatment effect, drop-in, drop-out, etc.) whereas cpower using normal approximations for simpler cases such as yours.&lt;/p&gt;&#xA;" OwnerUserId="4253" LastActivityDate="2011-05-19T15:15:25.510" CommentCount="1" />
  <row Id="10981" PostTypeId="1" CreationDate="2011-05-19T15:21:04.487" Score="6" ViewCount="105" Body="&lt;p&gt;I'm re-posting a question from &lt;a href=&quot;http://math.stackexchange.com/questions/32569/a-question-about-linear-regression&quot;&gt;math.stackexchange.com&lt;/a&gt;, I think the current answer in math.se is not right.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Select $n$ numbers from a set $\{1,2,...,U\}$, $y_i$ is the $i$th number selected, and $x_i$ is the rank of $y_i$ in the $n$ numbers. The selection is without replacement. $n$ is always smaller than $U$. The rank is the order of the a number after the $n$ numbers are sorted in ascending order.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can get $n$ data points $(x_1, y_1), (x_2, y_2), ..., (x_n, y_n)$, And a best fit line for these data points can be found by linear regression.  $r_{xy}$ (correlation coefficient) is the goodness of the fit line, I want to calculate $\mathbb{E}(r_{xy})$ or $\mathbb{E}(r_{xy}^2)$ (correlation of determination).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the $\mathbb{E}[r_{xy}]$ cannot be calculated, an estimation or lower bound is still OK.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Updated:&#xA;By calculating the sample correlation coefficient using randomly generated data, we can see that $r_{xy}$ is quite closed to 1, so I want to proof it from the theoretical view, or theoretically say the data generated by the above method is very linear.&lt;/p&gt;&#xA;" OwnerUserId="4670" LastEditorUserId="4670" LastEditorDisplayName="" LastEditDate="2011-05-19T17:47:47.600" LastActivityDate="2011-05-19T18:52:43.147" Title="Computing mathematical expectation of the correlation coefficient or $R^2$ in linear regression" Tags="&lt;regression&gt;&lt;correlation&gt;" AnswerCount="1" CommentCount="12" />
  <row Id="10982" PostTypeId="1" CreationDate="2011-05-19T15:21:22.930" Score="2" ViewCount="44" Body="&lt;p&gt;I am a newbie in stat. I am working on the Laplace distribution for my algorithm. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Could tell me the first what the four moments of the Laplace distribution are? &lt;/li&gt;&#xA;&lt;li&gt;Does it have infinite tail like the Cauchy distribution? &lt;/li&gt;&#xA;&lt;li&gt;What is the empirical rule?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="4319" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-19T22:10:05.820" LastActivityDate="2011-05-19T22:10:05.820" Title="Moments of Laplace distribution" Tags="&lt;distributions&gt;&lt;moments&gt;" AnswerCount="0" CommentCount="6" />
  <row Id="10984" PostTypeId="1" CreationDate="2011-05-19T16:11:28.713" Score="2" ViewCount="73" Body="&lt;p&gt;I'm having a hard time explaining this (hence the weird and long title), also I'm not a mathematician, I have this data lying around in a database and was wondering how I could visualise it (&lt;strike&gt;and predict the future&lt;/strike&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say I gave you the following data for one user, it can show anything, for example say it shows whether someone ran on a certain day (or completed any other task):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Date            Done?&#xA;19/05/2011      yes &#xA;18/05/2011      no&#xA;17/05/2011      no&#xA;16/05/2011      no&#xA;15/05/2011      no&#xA;14/05/2011      yes&#xA;13/05/2011      yes&#xA;12/05/2011      yes&#xA;11/05/2011      yes&#xA;10/05/2011      no&#xA;9/05/2011       yes&#xA;8/05/2011       no&#xA;7/05/2011       no&#xA;6/05/2011       yes&#xA;5/05/2011       no&#xA;4/05/2011       yes&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have a couple of questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;What data could you derive from this? How can this be represented (other than pie charts, bar charts and totals).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Is there anyway to predict what they would do tomorrow?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I used &lt;a href=&quot;http://www.evanmiller.org/how-not-to-sort-by-average-rating.html&quot; rel=&quot;nofollow&quot;&gt;Lower bound of Wilson score confidence interval for a Bernoulli parameter&lt;/a&gt; to figure out a score... what does this mean, is this useful in anyway?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-19T20:56:03.663" LastActivityDate="2011-05-20T00:16:26.300" Title="How can you predict the likelihood of someone doing something given previous data?" Tags="&lt;time-series&gt;&lt;binomial&gt;&lt;predictive-models&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="10985" PostTypeId="1" AcceptedAnswerId="11042" CreationDate="2011-05-19T16:23:47.003" Score="0" ViewCount="45" Body="&lt;p&gt;Can anybody show me, why the Dispersionparameter of the negbin distribution is taken to be one? In the Poisson case you can show that E(y)/var(y)=$\mu/$\mu=1 which is called Equidispersion. But how can you show that the Dispersionparameter of the negbin distribution is 1 as well?&lt;/p&gt;&#xA;" OwnerUserId="4496" LastActivityDate="2011-05-20T14:27:28.050" Title="Dispersionparameter of negbin distribution" Tags="&lt;negative-binomial&gt;&lt;proof&gt;&lt;overdispersion&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="10986" PostTypeId="1" CreationDate="2011-05-19T17:36:23.553" Score="1" ViewCount="62" Body="&lt;p&gt;I am trying to find some help with something that is called an &quot;Adjusted Analysis&quot; (or also Covariate Adjusted Logistic Regression);  a typical response has been that I might just want multivariable logistic regression, but this is not quite what I am looking for. The trouble I have is with what exactly an &quot;adjusted&quot; analysis is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an example, I have at my disposal a software suite that performs this type of adjusted analysis. We have some genes and various clinical variables from patients; what the method seems to do is adjust the p-values of the genes. But I can't figure out why, or how.  So I am trying to move outside of this software suite to truly understand what the underlying mathematics of this statistical technique is. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I've posted this question in &lt;a href=&quot;http://stackoverflow.com/questions/6061305/using-r-for-covariate-adjusted-logistic-regression&quot;&gt;other places&lt;/a&gt; the response has been that I should just take more courses in statistics. So while acknowledging my short comings, I would like to please ask if anyone can point me in a somewhat correct direction. I have been trying to find resources to help however I think I am not posing my question correctly enough.  As an aside I have a background in computer science and  more recently I am branching into biostatistics and I don't like using black box software so I would eventually like to re-implement this technique in R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you for any help that can be offered. Please let me know if there is a way I can pose my question clearer.&lt;/p&gt;&#xA;" OwnerUserId="4673" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-19T22:04:42.313" LastActivityDate="2011-05-19T22:04:42.313" Title="Covariate Adjusted Logistic Regression (&quot;Adjusted Analysis&quot;)" Tags="&lt;r&gt;&lt;logistic-regression&gt;&lt;genetics&gt;&lt;clinical-trials&gt;" CommentCount="7" />
  <row Id="10987" PostTypeId="1" AcceptedAnswerId="10998" CreationDate="2011-05-19T17:42:19.817" Score="2" ViewCount="107" Body="&lt;p&gt;I am looking for input on how others organize their R code and output.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My current practice is to write code in blocks in a text file as such:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#=================================================&#xA;# 19 May 2011&#xA;date()&#xA;# Correlation analysis of variables in sed summary&#xA;load(&quot;/media/working/working_files/R_working/sed_OM_survey.RData&quot;)&#xA;# correlation between estimated surface and mean perc.OM in epi samples&#xA;cor.test(survey$mean.perc.OM[survey$Depth == &quot;epi&quot;], &#xA;    survey$est.surf.OM[survey$Depth   == &quot;epi&quot;]))&#xA;#==================================================&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I then paste the output into another text file, usually with some annotation.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problems with this method are:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) The code and the output are not explicitly linked other than by date.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) The code and output are organized chronologically and thus can be hard to search.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have considered making one Sweave document with everything since I could then make a table of contents but this seems like it may be more hassle than the benefits it would provide.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please let me know of any effective routines you have for organizing your R code and output that would allow for efficient searching and editing the analysis.&lt;/p&gt;&#xA;" OwnerUserId="4048" LastActivityDate="2011-05-19T20:53:08.020" Title="What are efficient ways to organize R code and output?" Tags="&lt;r&gt;&lt;best-practices&gt;&lt;sweave&gt;" AnswerCount="3" CommentCount="2" />
  <row Id="10988" PostTypeId="2" ParentId="6314" CreationDate="2011-05-19T18:10:44.623" Score="2" ViewCount="" Body="&lt;p&gt;You might as well be interested in &lt;a href=&quot;http://www.google.de/url?sa=t&amp;amp;source=web&amp;amp;cd=1&amp;amp;ved=0CB0QFjAA&amp;amp;url=http://eprints.pascal-network.org/archive/00001570/01/nca6.pdf&amp;amp;ei=jFzVTf_xAczPsgaurv2QDA&amp;amp;usg=AFQjCNG-rlschiAWiAEtgyL-dhdsi2H3ng&amp;amp;sig2=CV-NnOB_Obf6Et_4EWijdQ&quot; rel=&quot;nofollow&quot;&gt;neighbourhood components analysis&lt;/a&gt; by Goldberger et al.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here, a linear transformation is learned to maximize the expected correctly classified points via a stochastic nearest neighbourhood selection.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a side effect the (expected) number of neighbours is determined from the data.&lt;/p&gt;&#xA;" OwnerUserId="2860" LastActivityDate="2011-05-19T18:10:44.623" CommentCount="2" />
  <row Id="10989" PostTypeId="2" ParentId="10271" CreationDate="2011-05-19T18:17:05.830" Score="1" ViewCount="" Body="&lt;p&gt;In the OP's reponse to my prior answer he has posted his data to the web. &lt;a href=&quot;http://i.stack.imgur.com/XhSQH.jpg&quot; rel=&quot;nofollow&quot;&gt;60 readings per hour for 24 hours for 6 days&lt;/a&gt; . Since this is time series cross-sectional tools like DBSCAN have limited relevance as the data has temporal dependence. With data like this one normally looks for intra-hour and intra-day structure. In addition to these kinds of structure one can pursue the detection of anomalies which can be either one time only (pulse) or systematic in nature (level shift) using methods that are well documented (see the literature of Tsay,Tiao,Chen et.al.) These procedures yielded the following &quot;anomalies'.Note that a level shift is essentially suggestive of separate &quot;clusters&quot;. &lt;img src=&quot;http://i.stack.imgur.com/irUvj.jpg&quot; alt=&quot; top identified anomalies &quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;                                HOUR/MINUTE     TIME&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="3382" LastEditorUserId="3382" LastEditorDisplayName="" LastEditDate="2011-05-19T18:56:20.710" LastActivityDate="2011-05-19T18:56:20.710" />
  <row Id="10990" PostTypeId="2" ParentId="10987" CreationDate="2011-05-19T18:31:02.617" Score="1" ViewCount="" Body="&lt;p&gt;I for one organize everything into 4 files for every project or analysis.&#xA;(1) 'code' Where I store text files of R functions.&#xA;(2) 'sql' Where I keep the queries used to gather my data.&#xA;(3) 'dat' Where I keep copies (usually csv) of my raw and processed data.&#xA;(4) 'rpt' Where I store the reports I've distributed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ALL of my files are named using very verbose names such as 'analysis_of_network_abc_for_research_on_modified_buffer_19May2011'&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also write detailed documentation up front where I organize the hypothesis, any assumptions, inclusion and exclusion criteria, and steps I intend to take to reach my deliverable. All of this is invaluable for repeatable research and makes my annual goal setting process easier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;HTH,&#xA;Will&lt;/p&gt;&#xA;" OwnerUserId="4675" LastActivityDate="2011-05-19T18:31:02.617" />
  <row Id="10991" PostTypeId="2" ParentId="10928" CreationDate="2011-05-19T18:33:15.893" Score="0" ViewCount="" Body="&lt;p&gt;Found the answer. Seems to work well except when power is low.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ P(X\le Y) = \int_0^1 \text{pnorm}((\delta\sqrt{N})/\sigma - \text{qnorm}(x))dx $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for your help guys, couldn't have gotten there without you.&lt;/p&gt;&#xA;" OwnerUserId="4647" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-05-19T20:48:57.070" LastActivityDate="2011-05-19T20:48:57.070" CommentCount="4" />
  <row Id="10992" PostTypeId="2" ParentId="10985" CreationDate="2011-05-19T18:37:53.050" Score="2" ViewCount="" Body="&lt;p&gt;Based on your previous question using &lt;code&gt;glm.nb&lt;/code&gt;, I'll take a wild guess that you are referring to the text in the output of that function:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; library(MASS)&#xA;&amp;gt; a &amp;lt;- glm.nb(Days ~ Eth + Sex, data=quine)&#xA;&amp;gt; summary(a)&#xA;&#xA;Call:&#xA;glm.nb(formula = Days ~ Eth + Sex, data = quine, init.theta = 1.171409701, &#xA;    link = log)&#xA;&#xA;Deviance Residuals: &#xA;    Min       1Q   Median       3Q      Max  &#xA;-2.5901  -1.0235  -0.3985   0.3414   2.3438  &#xA;&#xA;Coefficients:&#xA;            Estimate Std. Error z value Pr(&amp;gt;|z|)    &#xA;(Intercept)   2.9630     0.1351  21.936  &amp;lt; 2e-16 ***&#xA;EthN         -0.5738     0.1588  -3.613 0.000303 ***&#xA;SexM          0.2135     0.1594   1.340 0.180380    &#xA;---&#xA;Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 &#xA;&#xA;(Dispersion parameter for Negative Binomial(1.1714) family taken to be 1)&#xA;&#xA;    Null deviance: 182.14  on 145  degrees of freedom&#xA;Residual deviance: 168.12  on 143  degrees of freedom&#xA;AIC: 1112.9&#xA;&#xA;Number of Fisher Scoring iterations: 1&#xA;&#xA;&#xA;              Theta:  1.171 &#xA;          Std. Err.:  0.145 &#xA;&#xA; 2 x log-likelihood:  -1104.855 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In this output, the sentence about the dispersion parameter is an artifact of the approach that &lt;code&gt;glm.nb&lt;/code&gt; is using for fitting the negative binomial distribution. It repeatedly calls &lt;code&gt;glm&lt;/code&gt; for fixed values of the shape parameter &lt;code&gt;theta&lt;/code&gt;, then updates &lt;code&gt;theta&lt;/code&gt; based on the results of the fit. The sentence means that the distribution used by the final model is not overdispersed compared to the NegBin(1.1714) distribution, which is what you want, since that is the final fitted distribution. It says nothing about the overdispersion of NegBin(1.1714) compared to anything else.&lt;/p&gt;&#xA;" OwnerUserId="279" LastEditorUserId="279" LastEditorDisplayName="" LastEditDate="2011-05-20T13:46:09.087" LastActivityDate="2011-05-20T13:46:09.087" CommentCount="2" />
  <row Id="10993" PostTypeId="2" ParentId="10981" CreationDate="2011-05-19T18:52:43.147" Score="3" ViewCount="" Body="&lt;p&gt;If you only want to show $r^2_{xy}$ must be close to 1, and compute a lower bound for it, it's straightforward, because that means for given $U$ and $n$ you only need to maximize the variance of the residuals.  This can be done in exactly four symmetric ways.  The two extremes (lowest and highest possible correlations) are illustrated for $U=20, n=9$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.stack.imgur.com/BVAkr.png&quot; alt=&quot;Extreme correlation plots for U=20, n=9&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For large values of $U$ and appropriate values of $n$, $r^2_{xy}$ can actually get close to 0.  For example, with $n=100$ and very large values of $U \gg n$, $r^2_{xy} \sim 0.03$ in the worst case.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-05-19T18:52:43.147" CommentCount="6" />
  <row Id="10994" PostTypeId="2" ParentId="9659" CreationDate="2011-05-19T19:02:59.720" Score="0" ViewCount="" Body="&lt;p&gt;How accurate does your posterior cdf need to be? You might consider replacing the continuous prior with a discrete approximation:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$p^*(\theta) \propto p(\theta) 1(\theta\in t_1, \dots, t_k)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $p(\theta)$ is your original continuous prior.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then to compute the posterior you just calculate likelihood x prior &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$p(\theta|x) \propto p^*(\theta)p(x|\theta)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;over the support of the prior $t_1, \dots, t_k$ and renormalize.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is called &quot;griddy Gibbs&quot; by some. It can be quite effective if you have an informative prior in which case you can choose the grid points non-uniformly (and, of course, if you can live with a discrete approximation coarse enough to be computationally feasible).&lt;/p&gt;&#xA;" OwnerUserId="26" LastActivityDate="2011-05-19T19:02:59.720" />
  <row Id="10995" PostTypeId="2" ParentId="10987" CreationDate="2011-05-19T20:02:44.443" Score="2" ViewCount="" Body="&lt;p&gt;Now that I've made the switch to Sweave I never want to go back.  Especially if you have plots as output, it's so much easier to keep track of the code used to create each plot.  It also makes it much easier to correct one minor thing at the beginning and have it ripple through the output without having to rerun anything manually.&lt;/p&gt;&#xA;" OwnerUserId="3601" LastActivityDate="2011-05-19T20:02:44.443" CommentCount="4" />
  <row Id="10996" PostTypeId="1" AcceptedAnswerId="10999" CreationDate="2011-05-19T20:33:12.903" Score="7" ViewCount="72" Body="&lt;p&gt;I want to test a multi-stage path model (e.g., A predicts B, B predicts C, C predicts D) where all of my variables are individual observations nested within groups. So far I've been doing this through multiple unique multilevel analysis in R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would prefer to use a technique like SEM that lets me test multiple paths at the same time (A -&gt; B -&gt; C -&gt; D) and still properly handle the 2-levels (individuals in groups).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand that MPLUS can handle this. Is there an R package I can use?&lt;/p&gt;&#xA;" OwnerUserId="4677" LastEditorUserId="307" LastEditorDisplayName="" LastEditDate="2011-05-20T02:13:11.003" LastActivityDate="2011-05-20T04:06:40.047" Title="R package for multilevel structural equation modeling?" Tags="&lt;r&gt;&lt;multilevel-analysis&gt;&lt;sem&gt;&lt;path-model&gt;" AnswerCount="2" />
  <row Id="10997" PostTypeId="1" CreationDate="2011-05-19T20:35:50.280" Score="0" ViewCount="15" Body="&lt;p&gt;I have a large set of data (&gt;300 MB) of time series data consisting of groups of values intermixed with constants [new user, so I can't post example plot].&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to produce an average of just the clusters, while eliminating the periods of constant, unvarying data, between the clusters. Too much to do manually, obviously.  Perhaps a modified moving average, but I'm not sure how to evaluate the end point of the cluster on the fly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd like to do this in excel, as that's my main number handling platform.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="4676" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-05-19T21:08:16.710" LastActivityDate="2011-05-19T21:08:16.710" Title="Averaging data clusters amongst constants" Tags="&lt;time-series&gt;&lt;clustering&gt;&lt;excel&gt;&lt;averages&gt;" CommentCount="1" />
  <row Id="10998" PostTypeId="2" ParentId="10987" CreationDate="2011-05-19T20:53:08.020" Score="5" ViewCount="" Body="&lt;p&gt;You are not the first person to ask this question.  &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.r-statistics.com/2010/09/managing-a-statistical-analysis-project-guidelines-and-best-practices/&quot; rel=&quot;nofollow&quot;&gt;Managing a statistical analysis project – guidelines and best practices&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://blog.revolutionanalytics.com/2010/10/a-workflow-for-r.html&quot; rel=&quot;nofollow&quot;&gt;A workflow for R&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://jeromyanglim.blogspot.com/2010/12/r-workflow-slides-from-talk-at.html&quot; rel=&quot;nofollow&quot;&gt;R Workflow: Slides from a Talk at Melbourne R Users&lt;/a&gt; by Jeromy Anglim (including another much longer list of webpages dedicated to R Workflow)&lt;/li&gt;&#xA;&lt;li&gt;My own stuff: &lt;a href=&quot;https://github.com/berndweiss/dynamic_documents_with_r&quot; rel=&quot;nofollow&quot;&gt;Dynamic documents with R and LATEX as an important part of reproducible research&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="307" LastActivityDate="2011-05-19T20:53:08.020" CommentCount="4" />
  <row Id="10999" PostTypeId="2" ParentId="10996" CreationDate="2011-05-19T20:59:58.840" Score="10" ViewCount="" Body="&lt;p&gt;It seems that &lt;a href=&quot;http://openmx.psyc.virginia.edu/&quot; rel=&quot;nofollow&quot;&gt;OpenMx&lt;/a&gt; (based on Mx but it's now an R package) can do what you are looking for: &lt;a href=&quot;http://openmx.psyc.virginia.edu/thread/485&quot; rel=&quot;nofollow&quot;&gt;&quot;Multi Level Analysis&quot;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="307" LastActivityDate="2011-05-19T20:59:58.840" CommentCount="3" />
  <row Id="11000" PostTypeId="1" AcceptedAnswerId="11028" CreationDate="2011-05-19T21:03:54.640" Score="4" ViewCount="109" Body="&lt;p&gt;I'd like to regress a vector B against each of the columns in a matrix A. This is trivial if there are no missing data, but if matrix A contains missing values, then my regression against A is constrained to include only rows where all values are present (the default &lt;em&gt;na.omit&lt;/em&gt; behavior). This produces incorrect results for columns with no missing data. I can regress the column matrix B against individual columns of the matrix A, but I have thousands of regressions to do, and this is prohibitively slow and inelegant. The &lt;em&gt;na.exclude&lt;/em&gt; function seems to be designed for this case, but I can't make it work. What am I doing wrong here? Using R 2.13 on OSX, if it matters.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;A = matrix(1:20, nrow=10, ncol=2)&#xA;B = matrix(1:10, nrow=10, ncol=1)&#xA;dim(lm(A~B)$residuals)&#xA;# [1] 10 2 (the expected 10 residual values)&#xA;&#xA;# Missing value in first column; now we have 9 residuals&#xA;A[1,1] = NA  &#xA;dim(lm(A~B)$residuals)&#xA;#[1]  9 2 (the expected 9 residuals, given na.omit() is the default)&#xA;&#xA;# Call lm with na.exclude; still have 9 residuals&#xA;dim(lm(A~B, na.action=na.exclude)$residuals)&#xA;#[1]  9 2 (was hoping to get a 10x2 matrix with a missing value here)&#xA;&#xA;A.ex = na.exclude(A)&#xA;dim(lm(A.ex~B)$residuals)&#xA;# Throws an error because dim(A.ex)==9,2&#xA;#Error in model.frame.default(formula = A.ex ~ B, drop.unused.levels = TRUE) : &#xA;#  variable lengths differ (found for 'B')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1699" LastEditorUserId="1699" LastEditorDisplayName="" LastEditDate="2011-05-20T19:00:34.397" LastActivityDate="2011-05-20T22:46:53.610" Title="How does R handle missing values in lm?" Tags="&lt;r&gt;&lt;missing-data&gt;&lt;linear-model&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="11001" PostTypeId="2" ParentId="4840" CreationDate="2011-05-19T21:20:30.913" Score="2" ViewCount="" Body="&lt;p&gt;Since all you have is the data for the series you are trying to predict , the approach should be to construct a Robust Arima Model. A Robust Arima Model can reflect not only auto-projective structure (arima component) but changes in Levels and or Trends over time. The parameters of this model should be proven to not have changed over time and this is also true for the variance of the errors. There may be a change in any required &quot;seasonal Dummies&quot; to reflect a deterministic component as compared to a memory component. If you load your data up on the web and share it with the list , perhaps you can get understanding of the analytics and the analytical tools required. Similar work as been done in forecasting a highly seasonal monthly series entitled &quot;air line passengers&quot;. You might google &quot;Airline Passenger series BOX-JENKINS&quot; to get some clues as to how a similar series was handled and severely mishandled by NOT CORRECTLY identifying omitted deterministic structure. &lt;/p&gt;&#xA;" OwnerUserId="3382" LastActivityDate="2011-05-19T21:20:30.913" />
  <row Id="11002" PostTypeId="2" ParentId="10958" CreationDate="2011-05-19T21:55:58.340" Score="2" ViewCount="" Body="&lt;p&gt;&quot;True positives&quot;, like proportion classified correctly, requires arbitrary and information losing categorization of the predicted values.  These are improper scoring rules.  An improper scoring rule is a criterion that when optimized leads to a bogus model.  Also, watch out when using P-values in any way to guide model selection.  This will greatly distort the inference from the final model.&lt;/p&gt;&#xA;" OwnerUserId="4253" LastActivityDate="2011-05-19T21:55:58.340" CommentCount="2" />
  <row Id="11003" PostTypeId="1" CreationDate="2011-05-19T22:55:08.337" Score="0" ViewCount="23" Body="&lt;p&gt;Hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a mixed ANOVA with one between and one within factor:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;between factor: control versus treament within factor: ad1,ad2 (measures click rates on 2 ads)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;aov(repdat~(within*between)+Error(subjcts/(withincontrasts))+(between),data.rep)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However my outcome repdat has a Poisson distribution and 0s (since sometimes users just dont click). How do I model this in R?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I considered:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1.) using Zelig package with model Poisson, but I dont know how to properly write the mixed model formula here. Is it the same as for the mixed ANOVA?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2.) using lme4 package with model Poisson. Can I use both a between and within factor or even treat &quot;ads&quot; as a random variable?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would be great if someone could quickly show me an example code for both 1. and 2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;thanks&lt;/p&gt;&#xA;" OwnerUserId="4679" LastActivityDate="2011-05-19T22:55:08.337" Title="R statistics Mixed ANOVA but outcome Poisson distributed - which model in R?" Tags="&lt;r&gt;&lt;poisson&gt;&lt;lmer&gt;" FavoriteCount="1" />
  <row Id="11004" PostTypeId="1" AcceptedAnswerId="11005" CreationDate="2011-05-19T23:04:30.417" Score="2" ViewCount="34" Body="&lt;p&gt;I was wondering how to get arguments for &lt;em&gt;R&lt;/em&gt; functions. Is there any &lt;em&gt;R&lt;/em&gt; function which can be used to get all arguments for a certain function?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As &lt;em&gt;R&lt;/em&gt; function glm has the following arguments&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;glm(formula, family = gaussian, data, weights, subset,&#xA;na.action, start = NULL, etastart, mustart, offset,&#xA;control = list(...), model = TRUE, method = &quot;glm.fit&quot;,&#xA;x = FALSE, y = TRUE, contrasts = NULL, ...)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How these arguments can be obtained on &lt;em&gt;R&lt;/em&gt; console? Thanks&lt;/p&gt;&#xA;" OwnerUserId="3903" LastEditorUserId="3903" LastEditorDisplayName="" LastEditDate="2011-05-19T23:19:32.777" LastActivityDate="2011-05-19T23:44:51.157" Title="Checking $R$ functions arguments" Tags="&lt;r&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="11005" PostTypeId="2" ParentId="11004" CreationDate="2011-05-19T23:44:51.157" Score="4" ViewCount="" Body="&lt;p&gt;You could use something like &lt;code&gt;formals()&lt;/code&gt; or &lt;code&gt;args()&lt;/code&gt;, e.g. &lt;code&gt;formals(glm)&lt;/code&gt; gives:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; formals(glm)&#xA;$formula&#xA;&#xA;&#xA;$family&#xA;gaussian&#xA;&#xA;$data&#xA;&#xA;&#xA;$weights&#xA;&#xA;&#xA;$subset&#xA;&#xA;&#xA;$na.action&#xA;&#xA;&#xA;$start&#xA;NULL&#xA;&#xA;$etastart&#xA;&#xA;&#xA;$mustart&#xA;&#xA;&#xA;$offset&#xA;&#xA;&#xA;$control&#xA;list(...)&#xA;&#xA;$model&#xA;[1] TRUE&#xA;&#xA;$method&#xA;[1] &quot;glm.fit&quot;&#xA;&#xA;$x&#xA;[1] FALSE&#xA;&#xA;$y&#xA;[1] TRUE&#xA;&#xA;$contrasts&#xD;&#xA;    NULL&#xD;&#xA;    &#xD;&#xA;    $...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="307" LastActivityDate="2011-05-19T23:44:51.157" CommentCount="1" />
  <row Id="11006" PostTypeId="2" ParentId="10984" CreationDate="2011-05-20T00:16:26.300" Score="0" ViewCount="" Body="&lt;p&gt;You can have a look at &lt;a href=&quot;http://seed.ucsd.edu/~mindreader/&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; &quot;Mind Reading&quot; game and at the details of its implementation. I think it is very relevant to your second question.&lt;/p&gt;&#xA;" OwnerUserId="4337" LastActivityDate="2011-05-20T00:16:26.300" />
  <row Id="11007" PostTypeId="1" CreationDate="2011-05-20T00:30:20.703" Score="0" ViewCount="20" Body="&lt;p&gt;I am performing this simple experiment: I have one variety of grass and 8 different fungi (say #1 to #8). I am going to put 10-20 grass plants in each one of 18 containers, and then I will put each one of the fungi in 2 containers and let 2 with no fungus (&quot;negative control&quot;). After some time I will count how many plants have been killed by each one of the fungi in each one of two replicates for that fungus.&#xA;I want to compare the results for each one of the fungi with respect to the negative control. I think I can do that using the Fisher exact test, but I don't know if I have to combine the counts of the two replicates into a single count (like using a single container with all the plants together) or if I can keep those count separated. Also, I would like to know if I have to run the test doing pairwise comparisons (controls vs #n) or if I should run a single test considering all the results at the same time. May I do that also with the Fisher exact test? Do I have to make a multiple comparison adjustment?&#xA;Thank you for any help. I am very confused about this.&lt;/p&gt;&#xA;" OwnerUserId="4680" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-20T03:35:39.637" LastActivityDate="2011-05-20T03:35:39.637" Title="Test for biological replicates" Tags="&lt;statistical-significance&gt;" FavoriteCount="1" />
  <row Id="11008" PostTypeId="1" CreationDate="2011-05-20T00:39:14.580" Score="0" ViewCount="68" Body="&lt;p&gt;I made a comparison of hatch success between 2 populations of birds using R's &lt;code&gt;prop.test()&lt;/code&gt; function:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;prop.test(c(#hatched_site1, #hatched_site2),c(#laid_site1, #laid_site2))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It gave me the proportions of each site as part of the summary. How can I calculate the standard error for each proportion? &lt;/p&gt;&#xA;" OwnerUserId="4238" LastEditorUserId="307" LastEditorDisplayName="" LastEditDate="2011-05-20T11:06:57.907" LastActivityDate="2011-05-20T11:06:57.907" Title="How can I calculate the standard error of a proportion?" Tags="&lt;r&gt;&lt;standard-deviation&gt;&lt;proportion&gt;" CommentCount="5" FavoriteCount="1" />
  <row Id="11009" PostTypeId="1" CreationDate="2011-05-20T01:19:45.107" Score="9" ViewCount="220" Body="&lt;p&gt;Is it ever valid to include a two-way interaction in a model without including the main effects?  What if your hypothesis is only about the interaction, do you still need to include the main effects?&lt;/p&gt;&#xA;" OwnerUserId="2310" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-20T09:20:21.180" LastActivityDate="2011-05-22T14:26:28.640" Title="Including the interaction but not the main effects in a model" Tags="&lt;modeling&gt;&lt;interaction&gt;" AnswerCount="8" CommentCount="4" FavoriteCount="1" />
  <row Id="11011" PostTypeId="2" ParentId="10996" CreationDate="2011-05-20T01:52:07.213" Score="0" ViewCount="" Body="&lt;p&gt;Try searching for &quot;structural equation modeling&quot; on &lt;a href=&quot;http://rseek.org&quot; rel=&quot;nofollow&quot;&gt;http://rseek.org&lt;/a&gt;.  You'll find several helpful links, including links to several possible packages. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might also check out the Task View for the social sciences, there's a section for structural equation modeling maybe a third of the way down.  See &lt;a href=&quot;http://cran.r-project.org/web/views/SocialSciences.html&quot; rel=&quot;nofollow&quot;&gt;http://cran.r-project.org/web/views/SocialSciences.html&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One package in particular you might find helpful is John Fox's &lt;code&gt;sem&lt;/code&gt; package.&#xA;&lt;a href=&quot;http://cran.r-project.org/web/packages/sem/index.html&quot; rel=&quot;nofollow&quot;&gt;http://cran.r-project.org/web/packages/sem/index.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="3601" LastEditorUserId="3601" LastEditorDisplayName="" LastEditDate="2011-05-20T04:06:40.047" LastActivityDate="2011-05-20T04:06:40.047" CommentCount="4" />
  <row Id="11012" PostTypeId="1" CreationDate="2011-05-20T02:43:38.253" Score="0" ViewCount="27" Body="&lt;p&gt;I understand that SPSS can create a contingency table, and at the same time perform the chi-square test. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, is it possible that when we already have the contingency table, to have SPSS do the chi-square test?&lt;/p&gt;&#xA;" OwnerUserId="1663" LastActivityDate="2011-05-20T02:43:38.253" Title="Can SPSS perform chi-square test on an existing contingency table?" Tags="&lt;spss&gt;&lt;contingency-tables&gt;" CommentCount="1" />
  <row Id="11013" PostTypeId="2" ParentId="2230" CreationDate="2011-05-20T02:47:18.010" Score="4" ViewCount="" Body="&lt;p&gt;I know this question has already been answered (and quite well, in my view), but there was a different question &lt;a href=&quot;http://stats.stackexchange.com/questions/10964/in-convergence-in-probability-or-a-s-convergence-w-r-t-which-measure-is-the-prob&quot;&gt;here&lt;/a&gt; which had a comment @NRH that mentioned the graphical explanation, and rather than put the pictures &lt;a href=&quot;http://stats.stackexchange.com/questions/10964/in-convergence-in-probability-or-a-s-convergence-w-r-t-which-measure-is-the-prob&quot;&gt;there&lt;/a&gt; it would seem more fitting to put them here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, here goes.  It's not as cool as an R package.  But it's self-contained and doesn't require a subscription to JSTOR.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the following we're talking about a simple random walk, $X_{i}= \pm 1$ with equal probability, and we are calculating running averages,&#xA;$$&#xD;&#xA;\frac{S_{n}}{n} = \frac{1}{n}\sum_{i = 1}^{n}X_{i},\quad n=1,2,\ldots.&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.stack.imgur.com/QwNgS.png&quot; alt=&quot;Strong Law of Large Numbers&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The SLLN (convergence almost surely) says that we can be 100% sure that &lt;em&gt;this&lt;/em&gt; curve stretching off to the right will eventually, at some finite time, fall entirely within the bands forever afterward (to the right).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The R code used to generate this graph is below (plot labels omitted for brevity).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;n &amp;lt;- 1000;  m &amp;lt;- 50; e &amp;lt;- 0.05&#xA;s &amp;lt;- cumsum(2*(rbinom(n, size=1, prob=0.5) - 0.5))&#xA;plot(s/seq.int(n), type = &quot;l&quot;, ylim = c(-0.4, 0.4))&#xA;abline(h = c(-e,e), lty = 2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.stack.imgur.com/rtcqE.png&quot; alt=&quot;Weak Law of Large Numbers&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The WLLN (convergence in probability) says that a large proportion of the sample paths will be in the bands on the right-hand side, at time $n$ (for the above it looks like around 48 or 9 out of 50).  We can never be sure that any &lt;em&gt;particular&lt;/em&gt; curve will be inside at any finite time, but looking at the mass of noodles above it'd be a pretty safe bet.  The WLLN also says that we can make the proportion of noodles inside as close to 1 as we like by making the plot sufficiently wide.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The R code for the graph follows (again, skipping labels).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x &amp;lt;- matrix(2*(rbinom(n*m, size=1, prob=0.5) - 0.5), ncol = m)&#xA;y &amp;lt;- apply(x, 2, function(z) cumsum(z)/seq_along(z))&#xA;matplot(y, type = &quot;l&quot;, ylim = c(-0.4,0.4))&#xA;abline(h = c(-e,e), lty = 2, lwd = 2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1108" LastEditorUserId="1108" LastEditorDisplayName="" LastEditDate="2011-05-20T12:06:18.283" LastActivityDate="2011-05-20T12:06:18.283" />
  <row Id="11014" PostTypeId="1" CreationDate="2011-05-20T03:08:08.677" Score="2" ViewCount="21" Body="&lt;p&gt;Hello.  I need to forecast a large set of values given a bell curve where a small set of given points define the contraints of the curve between -1 &amp;amp; 0.  HELP!  Eventually i will be coding this in javascript.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Specifically, i am forecasting sales.  I know what the approximate total sales will be for each of the first five years and that sales follow a bell curve.  i need to forecast by month.  I'm not to concearned that the annual forecast is exatly on the correct month of the curve just that the curve is contrained by the given numbers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas how to do this?&lt;/p&gt;&#xA;" OwnerUserId="4684" LastActivityDate="2011-05-20T03:08:08.677" Title="Calculate gaussian function" Tags="&lt;normal-distribution&gt;&lt;curve-fitting&gt;&lt;javascript&gt;" CommentCount="1" />
  <row Id="11015" PostTypeId="1" CreationDate="2011-05-20T03:29:33.120" Score="2" ViewCount="26" Body="&lt;p&gt;Hello everyone,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The latest general framework I know in MCMC-based wrapper method(doing variable selection and clustering simultaneously) are the paper &quot;Bayesian variable selection in clustering high-dimensional data&quot; of Tadesse et al.(2005) and the paper &quot;Variable selection in clustering via Dirichlet process mixture models&quot; of Kim et al.(2006).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wonder if there are any new developments in this area? In particular, has anyone tried to extend the model of Tadesse?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance!&lt;/p&gt;&#xA;" OwnerUserId="4683" LastEditorUserId="4683" LastEditorDisplayName="" LastEditDate="2011-05-20T03:35:29.480" LastActivityDate="2011-05-20T03:35:29.480" Title="New development in variable selection in clustering using MCMC?" Tags="&lt;clustering&gt;&lt;mcmc&gt;&lt;feature-selection&gt;" FavoriteCount="1" />
  <row Id="11016" PostTypeId="2" ParentId="11009" CreationDate="2011-05-20T03:42:34.577" Score="2" ViewCount="" Body="&lt;p&gt;Arguably, it depends on what you're using your model for.  But I've never seen a reason not to run and describe models with main effects, even in cases where the hypothesis is only about the interaction.&lt;/p&gt;&#xA;" OwnerUserId="3748" LastActivityDate="2011-05-20T03:42:34.577" CommentCount="2" />
  <row Id="11017" PostTypeId="2" ParentId="10975" CreationDate="2011-05-20T03:46:23.310" Score="3" ViewCount="" Body="&lt;p&gt;One way to include is to include an indexed transformation.  One general way is to use any symmetric (inverse) cumulative distribution function, so that $F(0)=0.5$ and $F(x)=1-F(-x)$.  One example is the standard student t distribution, with $\nu$ degrees of freedom.  The parameter $v$ controls how quickly the transformed variable wanders off to infinity.  If you set $v=1$ then you have the arctan transform:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$x=arctan\left(\frac{\pi[2p-1]}{2}\right)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is much more extreme than arcsine, and more extreme than logit transform.  Note that logit transform can be roughly approximated by using the t-distribution with $\nu\approx 8$.  SO in some way it provides an approximate link between logit and probit ($\nu=\infty$) transforms, and an extension of them to more extreme transformations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem with these transforms is that they give $\pm\infty$ when the observed proportion is equal to $1$ or $0$.  So you need to somehow shrink these somehow - the simplest way being to add $+1$ &quot;successes&quot; and $+1$ &quot;failures&quot;.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-05-20T03:46:23.310" CommentCount="1" />
  <row Id="11018" PostTypeId="1" CreationDate="2011-05-20T03:54:39.277" Score="4" ViewCount="39" Body="&lt;p&gt;Let's aim for some at an introductory level, some articles and some textbooks.  Applied is more helpful, including R code is great.  Thanks!&lt;/p&gt;&#xA;" OwnerUserId="3748" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-20T09:21:52.803" LastActivityDate="2011-05-20T10:56:14.390" CommunityOwnedDate="2011-05-20T09:23:56.153" Title="Recommend references on survey sample weighting" Tags="&lt;sampling&gt;&lt;books&gt;&lt;weighted-sampling&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="11019" PostTypeId="1" CreationDate="2011-05-20T04:07:25.990" Score="1" ViewCount="81" Body="&lt;p&gt;I am doing text classification, and have been playing around with different classifiers. However I have a pretty basic question: what if a new unseen document comes in and it happens to not belong to any of the pre-existing classes? The classifiers that I have seen (in WEKA, libsvm etc.) still go ahead and put the unseen document in one of the existing classes anyway.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This situation comes up pretty frequently in my work. How can I handle it sensibly?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On a related note, is there a way I could get a sense of how confident a classifier is regarding it's classification decision?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;================================Edit 1======================================&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After reading the comments here, I think I am going to try one of the two things:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(1) Use the approach described in &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.13.7457&quot; rel=&quot;nofollow&quot;&gt;Zadrozny,Elkan&lt;/a&gt; paper that Steffen pointed me to, in order to get the probability estimates. If the probabilities are less than a magic threshold, I could then simply discard the unseen instance as noise. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(2) I am increasingly starting to think that I could instead handle this as &lt;em&gt;n&lt;/em&gt; 1-class problems. let's say that I have &lt;em&gt;n&lt;/em&gt; classes in my data. I could build and train &lt;em&gt;n&lt;/em&gt; 1-class classifiers that give a yes/no decision (whether an instance belongs to a particular class or not). So that way when a new instance comes in, I could just pass it through each of there 1-class classifiers. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thoughts? Any implementations/packages anyone is aware of that would let me do (2)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;===========================================================================&lt;/p&gt;&#xA;" OwnerUserId="3301" LastEditorUserId="3301" LastEditorDisplayName="" LastEditDate="2011-05-20T14:37:42.537" LastActivityDate="2011-05-20T15:25:44.807" Title="How does a classifier handle unseen documents that do not belong to any of the pre-existing classes?" Tags="&lt;machine-learning&gt;&lt;classification&gt;&lt;text-mining&gt;" AnswerCount="3" CommentCount="3" />
  <row Id="11020" PostTypeId="2" ParentId="11009" CreationDate="2011-05-20T04:51:11.040" Score="4" ViewCount="" Body="&lt;p&gt;The reason to keep the main effects in the model is for identifiability. Hence, if the purpose is statistical inference about each of the effects, you should keep the main effects in the model. However, if your modeling purpose is solely to predict new values, then it is perfectly legitimate to include only the interaction if that improves predictive accuracy.&lt;/p&gt;&#xA;" OwnerUserId="1945" LastActivityDate="2011-05-20T04:51:11.040" CommentCount="2" />
  <row Id="11021" PostTypeId="1" AcceptedAnswerId="11158" CreationDate="2011-05-20T04:57:27.690" Score="5" ViewCount="175" Body="&lt;p&gt;It has come to my attention that companies will rank resumes based on buzz words and only look at those that have high scores assuming enough people submit for the position.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't like this system but still want a job, so instead of inserting these meaningless terms into my actual resume I've compiled a list of such terms and am going to incorporate the list into my resume in a text box with white text.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is that I don't know how those scoring engines work (is there an application most employers use?) and so I can't be sure if I should maximize the terms I put in, if I should repeat terms, or if I should maximize the effective terms put in because number of words matter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have entered my resume on &lt;a href=&quot;http://rezscore.com/&quot; rel=&quot;nofollow&quot;&gt;Rezscore&lt;/a&gt; with and without the buzz word metadata and found my resume score significantly lowered with buzz words because it was so verbose.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone have more knowledge of these resume ranking systems?&lt;/p&gt;&#xA;" OwnerUserId="4685" LastActivityDate="2011-05-23T18:12:47.270" Title="Resume buzz words" Tags="&lt;data-mining&gt;&lt;job&gt;" AnswerCount="1" CommentCount="7" FavoriteCount="2" />
  <row Id="11022" PostTypeId="2" ParentId="11009" CreationDate="2011-05-20T05:31:47.087" Score="1" ViewCount="" Body="&lt;p&gt;This one is tricky and happened to me in my last project..I would explain it this way..lets say you had variables A and B which came out significant independently and by a business sense you thought that an interaction of A and B seems good. You included the interaction which came out to be significant but B lost its significance. You would explain your model initially by showing two results. The results would show that initially B was significant but when seen in light of A it lost its sheen. So B is a good variable but only when seen in light of various levels of A ( if A is a categorical variable)..Its like saying Obama is a good leader when seen in the light of its SEAL army..so Obama*seal will be a significant variable. But Obama when seen alone might not be as important..( no offense to Obama, just an example)&lt;/p&gt;&#xA;" OwnerUserId="1763" LastActivityDate="2011-05-20T05:31:47.087" CommentCount="1" />
  <row Id="11023" PostTypeId="2" ParentId="3484" CreationDate="2011-05-20T06:18:32.570" Score="0" ViewCount="" Body="&lt;p&gt;Hi..when i joined analytics industry( just out of my own interest) after serving software for 5 yrs..I didnt know SAS either..I got some version from somewhere and started writing codes on my own. Yes, I had programming background before that..I knew SQL, I knew general programming. I would suggest you visit tutorials and start writing codes yourself. The version is something you should get first but. Read SQL. Know everything from select * to joins to merge..tommorow if some interviewer gives you a loop or a join(left,right, full)..or some function like 1) contains 2) coalesce 3) sum, min, max, average&#xA;4) merge( in=a) (in=b) ..bla bla..you should be in decent condition that you are gonna ace it. These are just some bits from my side..apart from this you could also focus on reading things like regression analysis, MLE and OLS methods..this would show the interviewer that though this guy didnt have SAS facility he is good on general concepts..All I am preaching here is what i practiced.&lt;/p&gt;&#xA;" OwnerUserId="1763" LastActivityDate="2011-05-20T06:18:32.570" CommentCount="3" />
  <row Id="11024" PostTypeId="2" ParentId="11019" CreationDate="2011-05-20T07:01:08.583" Score="1" ViewCount="" Body="&lt;p&gt;In a lot of the classification algorithms (note: I do mean &lt;a href=&quot;http://en.wikipedia.org/wiki/Statistical_classification&quot; rel=&quot;nofollow&quot;&gt;classification&lt;/a&gt; in the 'classical' sense here), it is silently assumed that the classes are complete, i.e. every observation &lt;em&gt;must&lt;/em&gt; belong to one of the classes. So in that sense, the situation you are describing should not occur.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A kind of solution if your classes are not complete, is to provide a 'garbage' class. This may or may not work, depending on your situation, because the items that get assigned to this class could be of diverse nature, so if you take this class along in your classification algorithm, it may not work as expected. If your 'real' classes are strongly predicted, it could still work. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you roll your own algorithm, you &lt;em&gt;could&lt;/em&gt; exclude this class from the algorithm and just throw the nonfitting observations 'in the garbage', but I guess you would be violating some assumptions then (so guarantees provided by some methods will no longer be valid).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A relatively easy way to assess the confidence of a classifier is to do &lt;a href=&quot;http://en.wikipedia.org/wiki/Cross-validation_%28statistics%29&quot; rel=&quot;nofollow&quot;&gt;crossvalidation&lt;/a&gt; with missclassification.&lt;/p&gt;&#xA;" OwnerUserId="4257" LastActivityDate="2011-05-20T07:01:08.583" CommentCount="1" />
  <row Id="11025" PostTypeId="2" ParentId="11000" CreationDate="2011-05-20T07:28:22.837" Score="1" ViewCount="" Body="&lt;p&gt;I can think of two ways. One is combine the data use the &lt;code&gt;na.exclude&lt;/code&gt; and then separate data again:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;A = matrix(1:20, nrow=10, ncol=2)&#xA;colnames(A) &amp;lt;- paste(&quot;A&quot;,1:ncol(A),sep=&quot;&quot;)&#xA;&#xA;B = matrix(1:10, nrow=10, ncol=1)&#xA;colnames(B) &amp;lt;- paste(&quot;B&quot;,1:ncol(B),sep=&quot;&quot;)&#xA;&#xA;C &amp;lt;- cbind(A,B)&#xA;&#xA;C[1,1] &amp;lt;- NA&#xA;C.ex &amp;lt;- na.exclude(C)&#xA;&#xA;A.ex &amp;lt;- C[,colnames(A)]&#xA;B.ex &amp;lt;- C[,colnames(B)]&#xA;&#xA;lm(A.ex~B.ex)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Another way is to use the &lt;code&gt;data&lt;/code&gt; argument and create a formula.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Cd &amp;lt;- data.frame(C)&#xA;fr &amp;lt;- formula(paste(&quot;cbind(&quot;,paste(colnames(A),collapse=&quot;,&quot;),&quot;)~&quot;,paste(colnames(B),collapse=&quot;+&quot;),sep=&quot;&quot;))&#xA;&#xA;lm(fr,data=Cd)&#xA;&#xA;Cd[1,1] &amp;lt;-NA&#xA;&#xA;lm(fr,data=Cd,na.action=na.exclude)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you are doing a lot of regression the first way should be faster, since less background magic is performed. Although if you need only coefficients and residuals I suggest using &lt;code&gt;lsfit&lt;/code&gt;, which is much faster than &lt;code&gt;lm&lt;/code&gt;. The second way is a bit nicer, but on my laptop trying to do summary on the resulting regression throws an error. I will try to see whether this is a bug.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-05-20T07:28:22.837" CommentCount="2" />
  <row Id="11026" PostTypeId="1" CreationDate="2011-05-20T08:02:31.720" Score="1" ViewCount="20" Body="&lt;p&gt;My problem is to figure out an algorithm for summarising wind statistics over an area. The statistics are (24x) hourly but I want to end up with 2-3 'blocks' that summarise the conditions for the whole day.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My data is in the form of separate magnitude and direction NumPy arrays but I could convert them to u,v or polar coordinates or whatever if it made the solution easier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are other conditions like: each block must be at least 3 hours long, but the general problem is the trade off between accuracy and precision. And in a way, to maximise the similarity of the statistics inside each 'block' and the difference between each 'block'.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyway I don't have a background in statistics and am wondering if there is an array-based approach I could use, maybe involving calculating minimum bounding box/convex hull, that kind of thing?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not looking for a complete algorithm but some hints as to what I should be searching for to do further research.&lt;/p&gt;&#xA;" OwnerUserId="4686" LastActivityDate="2011-05-20T08:02:31.720" Title="Summarising wind vector statistics over time" Tags="&lt;algorithms&gt;&lt;python&gt;&lt;summary-statistics&gt;" />
  <row Id="11028" PostTypeId="2" ParentId="11000" CreationDate="2011-05-20T09:30:15.167" Score="2" ViewCount="" Body="&lt;p&gt;Edit: I misunderstood your question. There are two aspects:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;a) &lt;code&gt;na.omit&lt;/code&gt; and &lt;code&gt;na.exclude&lt;/code&gt; both do casewise deletion with respect to both predictors and criterions. They only differ in that extractor functions like &lt;code&gt;residuals()&lt;/code&gt; or &lt;code&gt;fitted()&lt;/code&gt; will pad their output with &lt;code&gt;NA&lt;/code&gt;s for the omitted cases with &lt;code&gt;na.exclude&lt;/code&gt;, thus having an output of the same length as the input variables.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; N    &amp;lt;- 20                               # generate some data&#xA;&amp;gt; y1   &amp;lt;- rnorm(N, 175, 7)                 # criterion 1&#xA;&amp;gt; y2   &amp;lt;- rnorm(N,  30, 8)                 # criterion 2&#xA;&amp;gt; x    &amp;lt;- 0.5*y1 - 0.3*y2 + rnorm(N, 0, 3) # predictor&#xA;&amp;gt; y1[c(1, 3,  5)] &amp;lt;- NA                    # some NA values&#xA;&amp;gt; y2[c(7, 9, 11)] &amp;lt;- NA                    # some other NA values&#xA;&amp;gt; Y    &amp;lt;- cbind(y1, y2)                    # matrix for multivariate regression&#xA;&amp;gt; fitO &amp;lt;- lm(Y ~ x, na.action=na.omit)     # fit with na.omit&#xA;&amp;gt; dim(residuals(fitO))                     # use extractor function&#xA;[1] 14  2&#xA;&#xA;&amp;gt; fitE &amp;lt;- lm(Y ~ x, na.action=na.exclude)  # fit with na.exclude&#xA;&amp;gt; dim(residuals(fitE))                     # use extractor function -&amp;gt; = N&#xA;[1] 20  2&#xA;&#xA;&amp;gt; dim(fitE$residuals)                      # access residuals directly&#xA;[1] 14  2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;b) The real issue is not with this difference between &lt;code&gt;na.omit&lt;/code&gt; and &lt;code&gt;na.exclude&lt;/code&gt;, you don't seem to want casewise deletion that takes criterion variables into account, which both do.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; X &amp;lt;- model.matrix(fitE)                  # design matrix&#xA;&amp;gt; dim(X)                                   # casewise deletion -&amp;gt; only 14 complete cases&#xA;[1] 14  2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The regression results depend on the matrices $X^{+} = (X&amp;#39; X)^{-1} X&amp;#39;$ (pseudoinverse of design matrix $X$, coefficients $\hat{\beta} = X^{+} Y$) and the hat matrix $H = X X^{+}$, fitted values $\hat{Y} = H Y$). If you don't want casewise deletion, you need a different design matrix $X$ for each column of $Y$, so there's no way around fitting separate regressions for each criterion. You can try to avoid the overhead of &lt;code&gt;lm()&lt;/code&gt; by doing something along the lines of the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; Xf &amp;lt;- model.matrix(~ x)                    # full design matrix (all cases)&#xA;# function: manually calculate coefficients and fitted values for single criterion y&#xA;&amp;gt; getFit &amp;lt;- function(y) {&#xA;+     idx   &amp;lt;- !is.na(y)                     # throw away NAs&#xA;+     Xsvd  &amp;lt;- svd(Xf[idx , ])               # SVD decomposition of X&#xA;+     # get X+ but note: there might be better ways&#xA;+     Xplus &amp;lt;- tcrossprod(Xsvd$v %*% diag(Xsvd$d^(-2)) %*% t(Xsvd$v), Xf[idx, ])&#xA;+     list(coefs=(Xplus %*% y[idx]), yhat=(Xf[idx, ] %*% Xplus %*% y[idx]))&#xA;+ }&#xA;&#xA;&amp;gt; res &amp;lt;- apply(Y, 2, getFit)    # get fits for each column of Y&#xA;&amp;gt; res$y1$coefs&#xA;                   [,1]&#xA;(Intercept) 113.9398761&#xA;x             0.7601234&#xA;&#xA;&amp;gt; res$y2$coefs&#xA;                 [,1]&#xA;(Intercept) 91.580505&#xA;x           -0.805897&#xA;&#xA;&amp;gt; coefficients(lm(y1 ~ x))      # compare with separate results from lm()&#xA;(Intercept)           x &#xA;113.9398761   0.7601234 &#xA;&#xA;&amp;gt; coefficients(lm(y2 ~ x))&#xA;(Intercept)           x &#xA;  91.580505   -0.805897&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Note that there might be numerically better ways to caculate $X^{+}$ and $H$, you could check a $QR$-decomposition instead. The SVD-approach is explained &lt;a href=&quot;http://stats.stackexchange.com/questions/6731/using-singular-value-decomposition-to-compute-variance-covariance-matrix-from-lin/6732#6732&quot;&gt;here on SE&lt;/a&gt;. I have not timed the above approach with big matrices $Y$ against actually using &lt;code&gt;lm()&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1909" LastEditorUserId="1909" LastEditorDisplayName="" LastEditDate="2011-05-20T22:46:53.610" LastActivityDate="2011-05-20T22:46:53.610" CommentCount="4" />
  <row Id="11029" PostTypeId="2" ParentId="2377" CreationDate="2011-05-20T09:56:36.523" Score="0" ViewCount="" Body="&lt;p&gt;I would rather model this data set using a leptokurtic distribution instead of using data-transformations.  I like the sinh-arcsinh distribution from Jones and Pewsey (2009), Biometrika.&lt;/p&gt;&#xA;" OwnerUserId="4688" LastActivityDate="2011-05-20T09:56:36.523" />
  <row Id="11030" PostTypeId="2" ParentId="11018" CreationDate="2011-05-20T10:56:14.390" Score="3" ViewCount="" Body="&lt;p&gt;I guess one could start with Thomas Lumley's webpage &quot;&lt;a href=&quot;http://faculty.washington.edu/tlumley/survey/&quot; rel=&quot;nofollow&quot;&gt;Survey analysis in R&lt;/a&gt;&quot;. He is the author of an R package called &lt;code&gt;survey&lt;/code&gt; and he has recently published a book about &quot;&lt;a href=&quot;http://faculty.washington.edu/tlumley/svybook/&quot; rel=&quot;nofollow&quot;&gt;Complex Surveys: a guide to analysis using R&lt;/a&gt;&quot;.&lt;/p&gt;&#xA;" OwnerUserId="307" LastActivityDate="2011-05-20T10:56:14.390" CommunityOwnedDate="2011-05-20T10:56:14.390" />
  <row Id="11031" PostTypeId="1" CreationDate="2011-05-20T11:10:16.710" Score="1" ViewCount="82" Body="&lt;p&gt;I just read a fascinating paper: &lt;a href=&quot;http://www.psy.cmu.edu/~ckemp/Papers/kempt08.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.psy.cmu.edu/~ckemp/Papers/kempt08.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my opinion it takes the whole area of machine learning to a completely new level because it flexibly discovers the structure of data (and doesn't only try to find a best fit for an existing structure).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The code is also available: &lt;a href=&quot;http://www.psy.cmu.edu/~ckemp/code/formdiscovery.html&quot; rel=&quot;nofollow&quot;&gt;http://www.psy.cmu.edu/~ckemp/code/formdiscovery.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried to do a few experiments of my own - but unfortunately I don't possess Matlab. I tried it with Octave but it only produced all kinds of error messages, which I don't understand (I am no expert on these programs).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could anybody perhaps have a quick look if these problems can be solved easily (or at all)? Perhaps the solution will be an easy one (this is my hope after all).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This would really be a big help! I am very much looking forward to trying a few data sets of my own. &lt;/p&gt;&#xA;" OwnerUserId="230" LastActivityDate="2011-05-20T11:10:16.710" ClosedDate="2011-05-26T06:53:48.303" Title="Estimating effort to port code from Matlab to Octave" Tags="&lt;matlab&gt;&lt;code&gt;&lt;octave&gt;" CommentCount="7" FavoriteCount="2" />
  <row Id="11032" PostTypeId="1" CreationDate="2011-05-20T11:10:50.573" Score="5" ViewCount="43" Body="&lt;p&gt;What are the benefits of giving certain initial values to transition probabilities in a Hidden Markov Model? Eventually system will learn them, so what is the point of giving values other than random ones? Does underlying algorithm make a difference such as Baum–Welch? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I know the transition probabilities at the beginning very accurately, and my main purpose is to predict output probabilities from hidden state to observations, what would you advise me?&lt;/p&gt;&#xA;" OwnerUserId="4001" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-20T11:57:38.207" LastActivityDate="2011-05-21T08:51:14.170" Title="Significance of initial transition probabilites in a hidden markov model" Tags="&lt;machine-learning&gt;&lt;em-algorithm&gt;&lt;hmm&gt;" AnswerCount="1" />
  <row Id="11033" PostTypeId="1" AcceptedAnswerId="11035" CreationDate="2011-05-20T11:29:45.040" Score="5" ViewCount="117" Body="&lt;p&gt;I have a model fitted (from the literature). I also have the raw data for the predictive variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What's the equation I should be using to get probabilities? Basically, how do I combine raw data and coefficients to get probabilities?&lt;/p&gt;&#xA;" OwnerUserId="333" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-27T07:01:11.133" LastActivityDate="2011-05-27T07:01:11.133" Title="How can I use logistic regression betas + raw data to get probabilities" Tags="&lt;regression&gt;&lt;logistic-regression&gt;" AnswerCount="2" FavoriteCount="2" />
  <row Id="11034" PostTypeId="2" ParentId="11033" CreationDate="2011-05-20T11:39:22.537" Score="11" ViewCount="" Body="&lt;p&gt;The link function of a logistic model is $f: x \mapsto \log \tfrac{x}{1 - x}$. Its inverse is $g: x \mapsto \tfrac{\exp x}{1 + \exp x}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a logistic model, the left-hand side is the logit of $\pi$, the probability of success:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$f(\pi) = \beta_0 + x_1 \beta_1 + x_2 \beta_2 + \ldots$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, if you want $\pi$ you need to evaluate $g$ at the right-hand side:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\pi = g( \beta_0 + x_1 \beta_1 + x_2 \beta_2 + \ldots)$.&lt;/p&gt;&#xA;" OwnerUserId="3019" LastActivityDate="2011-05-20T11:39:22.537" CommentCount="4" />
  <row Id="11035" PostTypeId="2" ParentId="11033" CreationDate="2011-05-20T12:30:31.097" Score="7" ViewCount="" Body="&lt;p&gt;Here is the applied researcher's answer (using the statistics package R).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, let's create some data, i.e. I am simulating data for a simple bivariate logistic regression model $log(\frac{p}{1-p})=\beta_0 + \beta_1 \cdot x$:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; set.seed(3124)&#xA;&amp;gt; &#xA;&amp;gt; ## Formula for converting logit to probabilities &#xA;&amp;gt; ## Source: http://www.statgun.com/tutorials/logistic-regression.html&#xA;&amp;gt; logit2prop &amp;lt;- function(l){exp(l)/(1+exp(l))}&#xA;&amp;gt; &#xA;&amp;gt; ## Make up some data&#xA;&amp;gt; y &amp;lt;- rbinom(100, 1, 0.2)&#xA;&amp;gt; x &amp;lt;- rbinom(100, 1, 0.5)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The predictor &lt;code&gt;x&lt;/code&gt; is a dichotomous variable:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; x&#xA;  [1] 0 1 1 1 1 1 0 1 0 1 0 1 0 0 1 1 1 0 1 1 1 1 1 1 0 0 1 1 1 1 0 0 1 0 0 1 0 0 0 1 1 1 0 1 1 1 1 &#xA; [48] 1 1 0 1 0 0 0 0 1 0 0 1 1 0 0 0 0 1 0 0 1 1 1 0 0 1 0 0 0 0 1 1 0 1 0 1 0 1 1 1 1 1 0 1 0 0 0&#xA; [95] 1 1 1 1 1 0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Second, estimate the intercept ($\beta_0$) and the slope ($\beta_1$). As you can see, the intercept is $\beta_0 = -0.8690$ and the slope is $\beta_1 = -1.0769 $.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; ## Run the model&#xA;&amp;gt; summary(glm.mod &amp;lt;- glm(y ~ x, family = &quot;binomial&quot;))&#xA;&#xA;[...]&#xA;&#xA;    Coefficients:&#xA;            Estimate Std. Error z value Pr(&amp;gt;|z|)   &#xA;(Intercept)  -0.8690     0.3304  -2.630  0.00854 **&#xA;x            -1.0769     0.5220  -2.063  0.03910 * &#xA;---&#xA;Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 &#xA;&#xA;(Dispersion parameter for binomial family taken to be 1)&#xA;&#xA;[...]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Third, R, like most statistical packages, can compute the fitted values, i.e. the probabilities. I will use these values as reference. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; ## Save the fitted values&#xA;&amp;gt; glm.fitted &amp;lt;- fitted(glm.mod)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Fourth, this step directly refers to your question: We have the raw data (here: $x$) and we have the coefficients ($\beta_0$ and $\beta_1$). Now, let's compute the logits and save these fitted values in &lt;code&gt;glm.rcdm&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; ## &quot;Raw data + coefficients&quot; method (RDCM)&#xA;## logit = -0.8690 + (-1.0769) * x&#xA;glm.rdcm &amp;lt;- -0.8690 + (-1.0769)*x&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The final step is a comparison of the fitted values based on R's &lt;code&gt;fitted&lt;/code&gt;-function (&lt;code&gt;glm.fitted&lt;/code&gt;) and my &quot;hand-made&quot; approach (&lt;code&gt;logit2prop.glm.rdcm&lt;/code&gt;). My own function &lt;code&gt;logit2prop&lt;/code&gt; (see first step) converts logits to probabilities:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; ## Compare fitted values and RDCM&#xA;&amp;gt; df &amp;lt;- data.frame(glm.fitted, logit2prop(glm.rdcm))&#xA;&amp;gt; df[10:25,]&#xA;&amp;gt; df[10:25,]&#xA;   glm.fitted logit2prop.glm.rdcm.&#xA;10  0.1250000            0.1250011&#xA;11  0.2954545            0.2954624&#xA;12  0.1250000            0.1250011&#xA;13  0.2954545            0.2954624&#xA;14  0.2954545            0.2954624&#xA;15  0.1250000            0.1250011&#xA;16  0.1250000            0.1250011&#xA;17  0.1250000            0.1250011&#xA;18  0.2954545            0.2954624&#xA;19  0.1250000            0.1250011&#xA;20  0.1250000            0.1250011&#xA;21  0.1250000            0.1250011&#xA;22  0.1250000            0.1250011&#xA;23  0.1250000            0.1250011&#xA;24  0.1250000            0.1250011&#xA;25  0.2954545            0.2954624&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="307" LastEditorUserId="307" LastEditorDisplayName="" LastEditDate="2011-05-20T13:42:54.187" LastActivityDate="2011-05-20T13:42:54.187" CommentCount="3" />
  <row Id="11036" PostTypeId="2" ParentId="10965" CreationDate="2011-05-20T13:22:43.093" Score="1" ViewCount="" Body="&lt;p&gt;A point of clarification but in my opinion all ARIMA forecasts are &quot;one-step ahead forecasts&quot;. In the &quot;fitting area t&amp;lt;37&quot; the fitted values are 1 period/step ahead forecasts. In the absence of data beyond t&gt;36 the procedure is to use the PREDICTED VALUE as if it were the ACTUAL VALUE thus bootstrapping the forecast. For example the forecast for period 38 will be a one-step ahead forecast as it will assume that the predicted value for period 37 is the &quot;actual&quot; for period 37. This is precisely why forecast accuracy measured from a single origin is tainted because the forecasts are correlated as they are obtained from a recursive scheme. If a one period out forecast is &quot;wrong&quot; then subsequent &quot;wrong forecasts&quot; may ensue. One is always better off by looking at forecast accuracies from a number of origins in order to assess model performance.&lt;/p&gt;&#xA;" OwnerUserId="3382" LastEditorUserId="3382" LastEditorDisplayName="" LastEditDate="2011-05-20T13:40:57.760" LastActivityDate="2011-05-20T13:40:57.760" />
  <row Id="11037" PostTypeId="2" ParentId="11019" CreationDate="2011-05-20T13:42:11.990" Score="1" ViewCount="" Body="&lt;p&gt;I find this interesting, you say that you somehow &quot;know&quot; that it does not belong to the classes given.  Yet you do not then describe what precisely it is about that item that makes you think this.  As soon as you suspect a class coming from &quot;something else&quot; (called SE from now on) you have basically begun to describe a model for predicting that class - if only intuitively.  The task then is to &quot;weed out&quot; the model that is hiding in your intuition.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is always done by describing what distinguishes SE from the rest of your data (this always exists, or else you wouldn't suspect anything).  The main problem with the SE is that this distinguishing feature must be based on the prior information alone - no data can possibly be used to train a classifier for SE.  For if it could, then SE would effectively not exist as you explain, it would just be one of the classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The best way to incorporate SE is to make a prediction about what kind of document you would see if it came from SE.  Given that the situation comes up pretty frequently, this should not be too difficult to do (as you would have some idea of the type of features that you would predict).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are using a Bayesian posterior probability based approach this is fairly simple to incorporate into a standard analysis.  We just add one extra likelihood into the denominator, so you have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(C_{i}|D_{new}D_{train}I)=\frac{P(C_{i}|D_{train}I)P(D_{new}|C_{i}D_{train}I)}{P(D_{new}|D_{train}I)}$$&#xA;$$=\frac{P(C_{i}|D_{train}I)P(D_{new}|C_{i}D_{train}I)}{\left[\sum_{j=1}^{r}P(C_{j}|D_{train}I)P(D_{new}|C_{j}D_{train}I)\right]+P(C_{SE}|D_{train}I)P(D_{new}|C_{SE}D_{train}I)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $C_{j}$ is the jth class (plus something else), $D_{new}$ is the data from the document you are trying to classify, $D_{train}$ is the training data, and $I$ is your prior information.  Note that the general procedure is not any different in principle from adding any of the other classes.  But it is different in practice because the phrase &quot;something else&quot; is vague and does not make any obvious predictions about future data.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-05-20T13:42:11.990" CommentCount="2" />
  <row Id="11039" PostTypeId="1" CreationDate="2011-05-20T10:58:57.617" Score="2" ViewCount="59" Body="&lt;p&gt;Hi, I want to run a multiple regression analysis measuring the effect of various independent variables on a continuous dependent variable measuring the strength of a political institution. The problem is that the data is measured in different years. Thus I might have data for countries A and B only for the year 2009 and for countries C and D only for the year 2010. In order to increase my sample, I would like to use all countries. How can I do that, ie which statistical method is most adequate? Thank you sooo much!&lt;/p&gt;&#xA;" LastActivityDate="2011-05-21T05:08:42.717" Title="How to use cross-section country samples of different years in multiple regression analysis?" Tags="&lt;r&gt;&lt;regression&gt;&lt;stata&gt;" AnswerCount="2" CommentCount="4" />
  <row Id="11040" PostTypeId="2" ParentId="11009" CreationDate="2011-05-20T14:07:16.690" Score="5" ViewCount="" Body="&lt;p&gt;You ask whether it's ever valid.  Let me provide a common example, whose elucidation may suggest additional analytical approaches for you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The simplest example of an interaction is a model with one dependent variable $Z$ and two independent variables $X$, $Y$ in the form&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Z = \alpha + \beta&amp;#39; X + \gamma&amp;#39; Y + \delta&amp;#39; X Y + \varepsilon,$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;with $\varepsilon$ a random term variable having zero expectation, and using parameters $\alpha, \beta&amp;#39;, \gamma&amp;#39;,$ and $\delta&amp;#39;$.  It's often worthwhile checking whether $\delta&amp;#39;$ approximates $\beta&amp;#39; \gamma&amp;#39;$, because an algebraically equivalent expression of the same model is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Z = \alpha \left(1 + \beta X + \gamma Y + \delta X Y \right) + \varepsilon$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$= \alpha \left(1 + \beta X \right) \left(1 + \gamma Y \right) + \alpha \left( \delta - \beta \gamma \right) X Y + \varepsilon$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(where $\beta&amp;#39; = \alpha \beta$, etc).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Whence, if there's a reason to suppose $\left( \delta - \beta \gamma \right) \sim 0$, we can absorb it in the error term $\varepsilon$.  Not only does this give a &quot;pure interaction&quot;, it does so &lt;em&gt;without a constant term.&lt;/em&gt;  This in turn strongly suggests taking logarithms.  Some heteroscedasticity in the residuals--that is, a tendency for residuals associated with larger values of $Z$ to be larger in absolute value than average--would also point in this direction.  We would then want to explore an alternative formulation&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\log(Z) = \log(\alpha) + \log(1 + \beta X) + \log(1 + \gamma Y) + \tau$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;with iid random error $\tau$.  Furthermore, if we expect $\beta X$ and $\gamma Y$ to be large compared to $1$, we would instead just propose the model&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\log(Z) = \left(\log(\alpha) + \log(\beta) + \log(\gamma)\right) + \log(X) + \log(Y) + \tau$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$= \eta + \log(X) + \log(Y) + \tau.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This new model has just a single parameter $\eta$ instead of four parameters ($\alpha$, $\beta&amp;#39;$, etc.) subject to a quadratic relation ($\delta' = \beta' \gamma'$), a considerable simplification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not saying that this is a necessary or even the only step to take, but I am suggesting that this kind of algebraic rearrangement of the model is usually worth considering whenever interactions alone appear to be significant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some excellent ways to explore models with interaction, especially with just two and three independent variables, appear in chapters 10 - 13 of Tukey's &lt;a href=&quot;http://www.amazon.com/Exploratory-Data-Analysis-Wilder-Tukey/dp/B0007347RW&quot; rel=&quot;nofollow&quot;&gt;EDA&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-05-20T14:07:16.690" />
  <row Id="11041" PostTypeId="2" ParentId="7385" CreationDate="2011-05-20T14:15:43.860" Score="1" ViewCount="" Body="&lt;p&gt;I had the same issue in my dissertation.  In Stata, I just built myself a custom .ado program with two calls to xtgee.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For this, I found the &lt;a href=&quot;http://harrisschool.uchicago.edu/faculty/web-pages/willard-manning.asp&quot; rel=&quot;nofollow&quot;&gt;&quot;Modeling Health Care Costs and Counts&quot;&lt;/a&gt; slides/programs by Partha Deb, Willard Manning, and Edward Norton to be useful.  They don't talk about longitudinal data, but it's a useful starting point.&lt;/p&gt;&#xA;" OwnerUserId="4691" LastActivityDate="2011-05-20T14:15:43.860" />
  <row Id="11042" PostTypeId="2" ParentId="10985" CreationDate="2011-05-20T14:27:28.050" Score="0" ViewCount="" Body="&lt;p&gt;For a general exponential family, we have the variance in the following form:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Var(Y_{i})=\phi h(E[Y_{i}])$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;for some function $h(.)$.  Using the &lt;a href=&quot;http://en.wikipedia.org/wiki/Negative_binomial_distribution&quot; rel=&quot;nofollow&quot;&gt;wikipedia definition of negative binomial&lt;/a&gt; we have a pdf of:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p(Y_{i}=y|r,p)={r+y-1 \choose y}p^{y}(1-p)^{r}\;\;\;\;\;\;y=0,1,2,\dots$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And this has expectation $E[Y_{i}]=\frac{pr}{1-p}$ and a variance equal to $Var[Y_{i}]=\frac{pr}{(1-p)^{2}}$.  Note that this cannot be written in the usual form for a generalised linear model, but has the form:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Var[Y_{i}]=E[Y_{i}]+\frac{1}{r}E[Y_{i}]^{2}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And as such it can be seen as taking the function $h(x)=x+\frac{1}{r}x^{2}$.  hence dispersion equal to &quot;1&quot;.  although neg binomial technically not a member of exponential family (as it is a mixture of exponential family, similar to student distribution).&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-05-20T14:27:28.050" CommentCount="2" />
  <row Id="11043" PostTypeId="1" CreationDate="2011-05-20T14:55:36.447" Score="1" ViewCount="54" Body="&lt;p&gt;Hello,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a good random generator (Lecuyer et al. from  &lt;a href=&quot;http://www.iro.umontreal.ca/~simardr/ssj/doc/html/index.html&quot; rel=&quot;nofollow&quot;&gt;http://www.iro.umontreal.ca/~simardr/ssj/doc/html/index.html&lt;/a&gt;) which help me to get a number with lognormal distribution into a random stream of double (a Well1024 random generator) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to normalize this number between [ 0 - 1 ], a friend tell me to build sigmoid function to normalize my numbers, but i'm not sure it's the best option.. Could you enligth me about this option ? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks !&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A small example of code in java/scala :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import umontreal.iro.lecuyer.randvar._&#xA;import umontreal.iro.lecuyer.rng.WELL1024&#xA;import scala.Math._&#xA;object testRandomLog extends App {&#xA;&#xA;  var logNormal = new LognormalGen(new WELL1024() )&#xA;  (0 until 10000).map{x =&amp;gt; {&#xA;    var d = logNormal.nextDouble()&#xA;      println(&quot;d &amp;gt;&quot; + d)&#xA;      println(&quot;sigm &amp;gt; &quot;+ 1/ (1+ exp (-d)))&#xA;    }}&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Edit 1 : &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm using genetic algorithm to optimize a fitness function. In each individual i have a chromosome with 7 gene, one for each varying parameter into my simulation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to create random population a) at start, b) and after when i make crossover and mutation with best individual, at each generation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of my parameter is probability between 0.0000000001 and 0.1, with a step precision of 0.000000001. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My problem is double : &lt;/p&gt;&#xA;&#xA;&lt;p&gt;1 - Random generator give me poor precision in my case, i have number like A = 0.25614548  . I can't fix precision, so i have a second random generator between 1 and 100.000.000 which help me to divide the previous number A. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;2 - I want to use LogNormal random generator because i want to have more big and medium  value 10E-4 -&gt; 10E-8 than little value 10E-9 / 10E-10 Problem in my case i need to standardize to obtain a probability value for my parameter between [0;1]&lt;/p&gt;&#xA;" OwnerUserId="4693" LastEditorUserId="4693" LastEditorDisplayName="" LastEditDate="2011-05-20T20:55:35.187" LastActivityDate="2011-05-20T20:55:35.187" Title="Normalize random numbers picked up into lognormal distribution" Tags="&lt;normalization&gt;&lt;random-generation&gt;&lt;lognormal&gt;" CommentCount="9" />
  <row Id="11044" PostTypeId="2" ParentId="11009" CreationDate="2011-05-20T15:07:08.627" Score="4" ViewCount="" Body="&lt;p&gt;While it is often stated in textbooks that one should never include an interaction in a model without the corresponding main effects, there are certainly examples where this would make perfect sense. I'll give you the simplest example I can imagine. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose subjects randomly assigned to two groups are measured twice, once at baseline (i.e., right after the randomization) and once after group T received some kind of treatment, while group C did not. Then a repeated-measures model for these data would include a main effect for measurement occasion (a dummy variable that is 0 for baseline and 1 for the follow-up) and an interaction term between the group dummy (0 for C, 1 for T) and the time dummy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The model intercept then estimates the average score of the subjects at baseline (regardless of the group they are in). The coefficient for the measurement occasion dummy indicates the change in the control group between baseline and the follow-up. And the coefficient for the interaction term indicates how much bigger/smaller the change was in the treatment group compared to the control group.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here, it is not necessary to include the main effect for group, because at baseline, the groups are equivalent by definition due to the randomization.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One could of course argue that the main effect for group should still be included, so that, in case the randomization failed, this will be revealed by the analysis. However, that is equivalent to testing the baseline means of the two groups against each other. And there are plenty of people who frown upon testing for baseline differences in randomized studies (of course, there are also plenty who find it useful, but this is another issue).&lt;/p&gt;&#xA;" OwnerUserId="1934" LastActivityDate="2011-05-20T15:07:08.627" CommentCount="6" />
  <row Id="11045" PostTypeId="1" CreationDate="2011-05-20T15:19:52.550" Score="1" ViewCount="26" Body="&lt;p&gt;I am trying to study the public opinion concept linked to quantitative measurement of opinion distributions and i am trying to define major parameters and methods that can help me starting from a certain finite number of people replying to a survey to generalize the results of that survey to say that this result can be considered as a public opinion.&#xA;Is there any statistical methods and parameters linked to country that i should integrate to define the concept of a public opinion ? what minimum number of people that must reply to a survey ? is there any intrinsic parameters linked to the survey it self ?&lt;/p&gt;&#xA;" OwnerUserId="4531" LastActivityDate="2011-05-20T15:53:13.237" Title="How to define specific population for Public Opinion" Tags="&lt;mathematical-statistics&gt;&lt;finite-population&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="11046" PostTypeId="2" ParentId="11019" CreationDate="2011-05-20T15:25:44.807" Score="5" ViewCount="" Body="&lt;p&gt;It sounds to me that the problem is one of &quot;novelty detection&quot;, you want to identify test patterns of a type not seen in the training data.  This can be achieved using the one-class support vector machine, which IIRC tries to construct a small volume in a kernel induced feature space that contains all (or a large fraction) of the training set, so any novel patterns encountered in operation are likely to fall outside this boundary.  There are loads of papers on uses of one-class SVM indexed on Google Scholar, so it should be easy to find something relevant to your application.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another approach would be to build a classifier by constructing a density estimator for each class, and then combine using Bayes rule for the classification.  If the likelihood of a test observation for the winning class is low, you can then reject the pattern as a possible novelty.  &lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2011-05-20T15:25:44.807" CommentCount="5" />
  <row Id="11047" PostTypeId="2" ParentId="11045" CreationDate="2011-05-20T15:53:13.237" Score="1" ViewCount="" Body="&lt;p&gt;The population is the sampling frame. If you're using some non-random sampling scheme like convenience sampling, the population is harder to define.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Random samples of the same size from populations of different sizes have different margins of error; samples from smaller populations have smaller margins of error. Statistical methods often assume infinite population sizes. Read about &quot;finite population correction&quot; to see how to adjust for the population size.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you have a large, heterogeneous population, you'll need to be concerned as to whether everyone will interpret the questionnaire the way you expect them to, so you might want to put more time into questionnaire development and pilot testing.&lt;/p&gt;&#xA;" OwnerUserId="3874" LastActivityDate="2011-05-20T15:53:13.237" />
  <row Id="11048" PostTypeId="1" CreationDate="2011-05-20T17:48:11.040" Score="0" ViewCount="18" Body="&lt;p&gt;I want to determine the amount of food stamp fraud a retailer perpetrated based on food stamp sales in other stores. The retailer had both legitimate food stamps sales, and illegitimate sales. The illegitimate sales consisted of food stamps used to buy ineligible items or whee food stamps were illegally redeemed for cash.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know the total amount of food stamps sales, but that figure reflects the total of all legitimate and illegitimate sales. Can an average of food stamp sales from other stores be compared to the retailer in a way that is statistically accurate? If so how big must that sample of other stores be?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks,&#xA;John&lt;/p&gt;&#xA;" OwnerUserId="4694" LastActivityDate="2011-05-20T18:36:29.503" Title="Market share based on comparison of competitors' average sales" Tags="&lt;sample-size&gt;" AnswerCount="1" />
  <row Id="11049" PostTypeId="1" CreationDate="2011-05-20T17:53:41.003" Score="3" ViewCount="42" Body="&lt;p&gt;I have a dataset of 380 samples of 6 variables. These variables are counts of different types of events in each of the 380 defined regions. These counts are per month, which means that I have several of these datasets (for now, I only have four months).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When looking at the data, I can clearly see that there is some missing (or incomplete) data. For instance, the counts for one given region are about the same for all months, except one (where it's close to zero). However, I does not seem likely that there were actually that few events during this period.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I would like is to be able, given data for a few months, to detect missing or incomplete values, and possibly to correct/complete them. Detecting missing values is not as obvious as looking for zeros, because it may happen that no event occurred in some regions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I read a few things about matrix factorization, but I'm not sure it would apply to my case. It seems suited for the cases where you know what data is missing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I assume this kind of problem is be quite common, for instance in biology for population estimation.&lt;/p&gt;&#xA;" OwnerUserId="3699" LastActivityDate="2011-05-22T19:06:50.440" Title="How to detect (and possibly estimate/interpolate) missing or incomplete data?" Tags="&lt;missing-data&gt;&lt;count-data&gt;" AnswerCount="2" />
  <row Id="11050" PostTypeId="1" CreationDate="2011-05-20T18:18:51.617" Score="11" ViewCount="115" Body="&lt;p&gt;I find that simple data analysis exercises can often help to illustrate and clarify statistical concepts.  What data analysis exercises do you use to teach statistical concepts?  &lt;/p&gt;&#xA;" OwnerUserId="485" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-20T18:22:22.097" LastActivityDate="2011-05-21T06:17:49.747" CommunityOwnedDate="2011-05-20T18:21:06.153" Title="Learning statistical concepts through data analysis exercises" Tags="&lt;teaching&gt;" AnswerCount="2" FavoriteCount="4" />
  <row Id="11051" PostTypeId="2" ParentId="11039" CreationDate="2011-05-20T18:32:50.783" Score="0" ViewCount="" Body="&lt;p&gt;I think that a Bayesian approach may solve your problem. But it depends on how many data you have. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I ned did it, but I guesst's possible to estimate a hierarchical model, in which data are nested by country or time. For example, assume you have $i = 1, 2, ..., n$ countries and $t = 1, 2, ..., T$ years. For country $j$ you don't have information for year $l$ and for country $k$ you don't have information for year $m$. So, you would have a model like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$y_{it} \sim N(a + b_{t}*x_{it})$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$b_{t} \sim N(0, sigma^{2})$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$sigma^{2} \sim Unif(0, 100)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$x_{jk} \sim N(0, 100)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$x_{lm} \sim N(0, 100)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The details of the priors will depend of your theory and your data. It is fairly simple to estimate this model using WinBugs or Jags (so you don't have to worry about the MCMC machinery).&lt;/p&gt;&#xA;" OwnerUserId="3058" LastActivityDate="2011-05-20T18:32:50.783" />
  <row Id="11052" PostTypeId="2" ParentId="11048" CreationDate="2011-05-20T18:36:29.503" Score="1" ViewCount="" Body="&lt;p&gt;If the proportion of illegitimate sales were known to be the same in all stores, then your problem would be strictly a statistical one.  As it is, it's first and foremost a matter of content knowledge:  on what bases are you able to estimate the proportion in this store as compared to in others?  Do you have any other evidence to bring to bear on this situation?  Others more skilled in Bayesian methods will be able to give you specific steps to take so that you can create a credible interval for the proportion in the store in question.&lt;/p&gt;&#xA;" OwnerUserId="2669" LastActivityDate="2011-05-20T18:36:29.503" />
  <row Id="11053" PostTypeId="2" ParentId="11050" CreationDate="2011-05-20T18:39:55.037" Score="4" ViewCount="" Body="&lt;p&gt;&lt;em&gt;&lt;strong&gt;Multiple Regression Coefficients and the Expected Sign Fallacy&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of my favorite illustrations of a statistical concept through a data analysis exercise is the deconstruction of a multiple regression into multiple bivariate regressions.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Objectives&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;To clarify the meaning of regression&#xA;coefficients in the presence of&#xA;multiple predictors.&lt;/li&gt;&#xA;&lt;li&gt;To illustrate why it is incorrect to&#xA;“expect” a multiple regression&#xA;coefficient to have a particular sign&#xA;based on its bivariate relationship&#xA;with Y when the predictors are&#xA;correlated.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Concept&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The regression coefficients in a multiple regression model represent the relationship between a) the part of a given predictor variable (x1) that is not related to all of the other predictor variables (x2...xN)  in the model;  and 2) the part of the response variable (Y) that is not related to all of the other predictor variables (x2...xN) in the model. When there is correlation among the predictors, the signs associated with the predictor coefficients represent the relationships among those residuals.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Exercise&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Generate some random data for two&#xA;predictors (x1, x2) and a response&#xA;(y).&lt;/li&gt;&#xA;&lt;li&gt;Regress y on x2 and store the&#xA;residuals.&lt;/li&gt;&#xA;&lt;li&gt;Regress x1 on x2 and store the&#xA;residuals.&lt;/li&gt;&#xA;&lt;li&gt;Regress the residuals of step 2 (r1)&#xA;on the residuals of step 3 (r2).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The coefficient for step 4 for r2 will be the coefficient of x1 for the multiple regression model with x1 and x2.  You could do the same for x2 by partialing out x1 for both y and x2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's some R code for this exercise.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(3338)&#xA;x1 &amp;lt;- rnorm(100)&#xA;x2 &amp;lt;- rnorm(100)&#xA;y &amp;lt;- 0 + 2*x1 + 5*x2 + rnorm(100)&#xA;lm(y ~ x1 + x2)  # Multiple regression Model&#xA;ry1 &amp;lt;- residuals(  lm( y ~ x2)  )  # The part of y not related to x2&#xA;rx1 &amp;lt;- residuals(  lm(x1 ~ x2)  ) # The part of x1 not related to x2&#xA;lm( ry1  ~ rx1) &#xA;ry2 &amp;lt;- residuals(  lm( y ~ x1)  ) # The part of y not related to x1&#xA;rx2 &amp;lt;- residuals(  lm(x2 ~ x1)  ) # The part of x2 not related to x1&#xA;lm( ry2 ~ rx2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here are the relevant outputs and results.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Call:&#xA;lm(formula = y ~ x1 + x2)&#xA;&#xA;Coefficients:&#xA;&#xA;(Intercept)           ***x1***           ***x2***  &#xA;   -0.02410      ***1.89527***      ***5.07549*** &#xA;&#xA;Call:&#xA;lm(formula = ry1 ~ rx1)&#xA;&#xA;Coefficients:&#xA;&#xA;(Intercept)          ***rx1***  &#xA; -2.854e-17    ***1.895e+00*** &#xA;&#xA;Call:&#xA;lm(formula = ry2 ~ rx2)&#xA;&#xA;Coefficients:&#xA;&#xA;(Intercept)          ***rx2***  &#xA;  3.406e-17    ***5.075e+00*** &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="485" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-21T06:17:49.747" LastActivityDate="2011-05-21T06:17:49.747" CommunityOwnedDate="2011-05-20T18:39:55.037" />
  <row Id="11054" PostTypeId="1" AcceptedAnswerId="11057" CreationDate="2011-05-20T18:43:03.960" Score="3" ViewCount="71" Body="&lt;p&gt;For my classification problem, I am trying to classify an object as Good or Bad. I have been able to create a good first classification step that separates the data into 2 groups using SVM. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;After tuning the parameters for the SVM using a training/holdout set (75% training, 25% holdout), I obtained the following results from the holdout set: Group 1 (model classified as Bad) consisted of 99% Bad objects, and Group 2 (model classified as Good) consisted of about 45% Good objects and 55% Bad objects. I verified the performance of the model using k-fold CV (k=5) and found the model to be stable and perform relatively consistently in terms of misclassification rates.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, I want to pass these objects through another round of classification by training another model (may or may not be SVM) on my group 2 of maybe good/maybe bad objects to try and correctly classify this second group now that I have gotten rid of the obviously bad objects.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I had a couple of thoughts, but am unsure of how to proceed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(1) My first idea was to use the data from the classified objects from the Holdout set to train another model. I was able to train another classification model from the results of the holdout set. The problem is I am using less than 25% of the original data, and I am worried of overfitting on a very small subset of my data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(2) My second idea was to gather the results of the 5-fold CV to create another dataset. My reasoning is that since the data is partitioned into 5 parts, and each part is classified into two groups from a model trained by the other 4 parts, I thought that I could aggregate the predicted results of the 5 parts to obtain a classified version of my original dataset and continue from there.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The only problem is, I have a sinking feeling that both methods are no good. Could CV shed some light on some possible next steps?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sorry, my question was badly worded. Let me try to clarify what I am trying to do. It can be thought of like a tree...&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Let me call the original dataset Node 0. &lt;/li&gt;&#xA;&lt;li&gt;I used classification method 1 to split Node 0 into Node 1 and Node 2. &#xA;&lt;ul&gt;&#xA;&lt;li&gt;Node 1 has low misclassification rate (Mostly consists of bad objects)&lt;/li&gt;&#xA;&lt;li&gt;Node 2 has high misclassification rate (Roughly even mix of good and bad objects)&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;I now want to use classification method 2 to split Node 2 into Node 3 and 4&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The &quot;classification method&quot; can be anything (LDA, QDA, SVM, CART, Random Forest, etc). So I guess what I am trying to achieve here is a &quot;classification&quot; tree (&lt;em&gt;not&lt;/em&gt; CART), where each node is subjected to a &lt;em&gt;different&lt;/em&gt; classification method to obtain an overall high &quot;class purity&quot;. Basically, I want to use a mix of different classification methods to obtain reasonable results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My problem lies in the loss of training data after the first split. I run out of usable data after I run it through &quot;classification method 1&quot;, which was SVM in my case.&lt;/p&gt;&#xA;" OwnerUserId="2252" LastEditorUserId="2252" LastEditorDisplayName="" LastEditDate="2011-05-20T21:21:45.350" LastActivityDate="2011-05-24T18:00:35.007" Title="Training multiple models for classification using the same dataset" Tags="&lt;machine-learning&gt;" AnswerCount="2" CommentCount="8" FavoriteCount="1" />
  <row Id="11055" PostTypeId="2" ParentId="11049" CreationDate="2011-05-20T19:34:15.570" Score="3" ViewCount="" Body="&lt;p&gt;I think you are really looking for outliers in your data (small values where most 'similar' values, i.e. values where a set of covariates hold the same values, are 'bigger').&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could look at &lt;a href=&quot;http://stats.stackexchange.com/questions/213/what-is-the-best-way-to-identify-outliers-in-multivariate-data&quot;&gt;outliers in MV data&lt;/a&gt; for this for starters, although you may be able to utilize some particularities of your situation (you are only interested in outliers in one variable, and only in one direction).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your sample size is big enough for that, you can simply use the outlier definition as used in most boxplots (1.5 IQR away from the outer quartiles). If not, you should apply some truly parametric way of detecting outliers (e.g. residuals in a regression).&lt;/p&gt;&#xA;" OwnerUserId="4257" LastActivityDate="2011-05-20T19:34:15.570" />
  <row Id="11056" PostTypeId="2" ParentId="11054" CreationDate="2011-05-20T19:38:54.947" Score="4" ViewCount="" Body="&lt;p&gt;I would use the same training dataset for both models, and use the same CV-folds for tuning.  Don't use ANY of the 25% hold-out for training or tuning.  Once you've fit your 2 models on the 75% training sample, evaluate your performance using the holdout.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are using R, the caret package has functions for creating folds on a dataset that you can re-use to tune multiple models and then evaluate their predictive accuracy.  If you would like I can help you with some example code.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit:&#xA;Here is the promised code, modified from the &lt;a href=&quot;http://cran.r-project.org/web/packages/caret/vignettes/caretTrain.pdf&quot; rel=&quot;nofollow&quot;&gt;vignette&lt;/a&gt; for the package &lt;a href=&quot;http://cran.r-project.org/web/packages/caret/index.html&quot; rel=&quot;nofollow&quot;&gt;caret&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#Setup&#xA;rm(list = ls(all = TRUE)) #CLEAR WORKSPACE&#xA;set.seed(123)&#xA;&#xA;#Pretend we only care about virginica&#xA;Data &amp;lt;- iris&#xA;virginica &amp;lt;- Data$Species=='virginica'&#xA;Data$Species &amp;lt;- NULL&#xA;&#xA;#Look at the variable relationships&#xA;library(PerformanceAnalytics)&#xA;chart.Correlation(Data,col=ifelse(virginica,1,2))&#xA;&#xA;#Create cross-validation folds to use for multiple models&#xA;#Use 10-fold CV, repeat 5 times&#xA;library(caret)&#xA;MyFolds &amp;lt;- createMultiFolds(virginica, k = 10, times = 5)&#xA;MyControl &amp;lt;- trainControl(method = &quot;repeatedCV&quot;, index = MyFolds,&#xA;                summaryFunction = twoClassSummary,&#xA;                classProbs = TRUE)&#xA;&#xA;#Define Equation for Models&#xA;fmla &amp;lt;- as.formula(paste(&quot;virginica ~ &quot;, paste(names(Data), collapse= &quot;+&quot;)))&#xA;&#xA;#Fit some models&#xA;Data$virginica &amp;lt;- as.factor(ifelse(virginica,'Yes','No'))&#xA;&#xA;svmModel &amp;lt;- train(fmla,Data,method='svmRadial',&#xA;    tuneLength=3,metric='ROC',trControl=MyControl)&#xA;&#xA;rfModel &amp;lt;- train(fmla,Data,method='rf',&#xA;    tuneLength=3,metric='ROC',trControl=MyControl)&#xA;&#xA;#Compare Models&#xA;resamps &amp;lt;- resamples(list(&#xA;    SVM = svmModel,&#xA;    RandomForest = rfModel&#xA;    ))&#xA;summary(resamps)&#xA;densityplot(resamps,auto.key = TRUE, metric='ROC')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2817" LastEditorUserId="2817" LastEditorDisplayName="" LastEditDate="2011-05-24T18:00:35.007" LastActivityDate="2011-05-24T18:00:35.007" CommentCount="4" />
  <row Id="11057" PostTypeId="2" ParentId="11054" CreationDate="2011-05-20T19:51:32.450" Score="3" ViewCount="" Body="&lt;p&gt;Just to make sure that we are on the same page, I take it from your description that you consider a supervised learning problem where you know the Good/Bad status of your objects and where you have a vector of features for each object that you want to use to classify the object as either Good or Bad. Moreover, the result of training an SVM is to give a classifier, which, on the holdout data, gives almost no false Bad predictions, but 55% false Good predictions. I have not personally worked with problems with such a huge difference in error rates on the two groups. It suggests to me that the distribution of features in the two groups overlap, but that the distribution of features in the Bad group is more spread out. Like two Gaussian distributions with almost the same mean but larger variance for the group of Bad objects. If that is the case, I would imagine that it will be difficult, if not impossible, to improve much on the error rate for the Good predictions. There may be other explanations that I am not aware of.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having said that, I think it is a sensible strategy to combine classification procedures in a hierarchical way as you suggest. First, one classifier splits the full training set into two groups, and then other classifiers split each of the groups into two groups etc. In fact, that is what classification trees do, but typically using very simple splits in each step. I see no formal problem in training whatever model you like on the training data that is classified as being Good by the SVM. You don't need to use the holdout data. In fact, you shouldn't, if you need the holdout data for assessment of the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your second suggestion is closely related to just using the group classified as Good from your training data to train a second model. I don't see any particular reason to use CV-based classifications to obtain this group. Just remember, that if you are going to use CV, then the entire training procedure must be carried out each time. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My suggestion is to first get a better understanding of what the feature distributions look like in the two groups from low-dimensional projections and exploratory visualizations. It might shed some light on why the error rate on the Good classifications is so large.&lt;/p&gt;&#xA;" OwnerUserId="4376" LastEditorUserId="4376" LastEditorDisplayName="" LastEditDate="2011-05-20T20:03:08.883" LastActivityDate="2011-05-20T20:03:08.883" CommentCount="2" />
  <row Id="11058" PostTypeId="2" ParentId="3104" CreationDate="2011-05-20T19:55:14.197" Score="4" ViewCount="" Body="&lt;p&gt;At risk of sounding too simplistic, I think the best problem to introduce depends on who you are talking to.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example my arts friends freak out when I talk about math and stats, but then I tell them they shouldn't be afraid because they speak math all the time. So I give them examples such as &quot;What are the odds it will rain today?&quot;, you don't acknowledge you're doing the computation but you are assessing some probability in your mind. So for them I like to pick very relateable problems dealing with weather and emotions (&quot;For example, given you are depressed, how likely is it do be raining outside?&quot;) and show them the math behind how we might answer that. Then later after they have discovered an intuition for mathematical problem solving I tell them what the terminology is for it. AND yes I have gotten my arts friends to sit willing through that!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I personally learned statistics better when I had a problem in my domain I understood very well.  I find when you understand a problem very well it becomes easier to understand the math. I think too often people just learn by rote and look to fit problems they've already seen onto new ones rather than trying to understand each problem.&lt;/p&gt;&#xA;" OwnerUserId="4673" LastActivityDate="2011-05-20T19:55:14.197" CommunityOwnedDate="2011-05-20T19:55:14.197" />
  <row Id="11059" PostTypeId="2" ParentId="11050" CreationDate="2011-05-20T20:15:32.913" Score="4" ViewCount="" Body="&lt;p&gt;As I have to explain variable selection methods quite often, not in a teaching context, but for non-statisticians requesting aid with their research, I love this extremely simple example that illustrates why single variable selection is not necessarily a good idea.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have this dataset:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;y      X1     x2&#xA;1       1      1&#xA;1       0      0&#xA;0       1      0&#xA;0       0      1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It doesn't take long to realize that both X1 and X2 individually are completely noninformative for y (when they are the same, y is 'certain' to be 1 - I'm ignoring sample size issues here, just assume these four observations to be the whole universe). However, the combination of the two variables is &lt;em&gt;completely&lt;/em&gt; informative. As such, it is more easy for people to understand why it is not a good idea to (e.g.) only check the p-value for models with each individual variable as a regressor.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my experience, this really gets the message through.&lt;/p&gt;&#xA;" OwnerUserId="4257" LastActivityDate="2011-05-20T20:15:32.913" CommunityOwnedDate="2011-05-20T20:15:32.913" CommentCount="1" />
  <row Id="11060" PostTypeId="1" AcceptedAnswerId="11063" CreationDate="2011-05-20T20:57:49.180" Score="4" ViewCount="70" Body="&lt;p&gt;Hi,&#xA;I'm hoping someone can help me sort out how the proportion comparisons using a GLM works in R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm comparing hatch success among multiple years (and later, sites). I've used a GLM to compare among years by making a txt file where the success column contains the number of chicks that hatched, and failures are the number of eggs in each nest that didn't hatch. Each row corresponds to a nest. (I did this according to descriptions in &quot;The R Book&quot; by Michael Crawley)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The code looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;y&amp;lt;-cbind(success,fail)&#xA;hsmodel1&amp;lt;-glm(y~year,binomial)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This tests for differences in hatch success among years as #of chicks hatched/eggs laid, correct? Not # of chicks hatched /nest?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Secondly, if my species lays up to 2 eggs, is using this proportional method still valid, since there could be 0, 1 or 2 successes or failures? I'm pretty sure it is, but I am starting to doubt myself from a question a colleague asked today.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&#xA;Mog &lt;/p&gt;&#xA;" OwnerUserId="4238" LastEditorUserId="4238" LastEditorDisplayName="" LastEditDate="2011-05-21T01:06:56.623" LastActivityDate="2011-05-21T01:06:56.623" Title="GLM for proportional data" Tags="&lt;r&gt;&lt;glm&gt;&lt;proportion&gt;" AnswerCount="1" />
  <row Id="11061" PostTypeId="2" ParentId="3104" CreationDate="2011-05-20T21:19:25.467" Score="0" ViewCount="" Body="&lt;p&gt;For a gentle introduction, I like examples using 2x2 contingency tables.  The diagnostic testing example as mentioned above, where the Probability of a positive test result given disease is not equal to the Probability of disease given a positive test result.  Also, one can use designs with different sampling schemes, such as the cohort study vs. the case-control study, to illustrate how that affects what probabilities can be estimated.&lt;/p&gt;&#xA;" OwnerUserId="2312" LastActivityDate="2011-05-20T21:19:25.467" CommunityOwnedDate="2011-05-20T21:19:25.467" />
  <row Id="11063" PostTypeId="2" ParentId="11060" CreationDate="2011-05-20T22:08:44.360" Score="2" ViewCount="" Body="&lt;p&gt;Logistic regression, like this is, assumes a binomial distribution, or, as I prefer, a Bernoulli distribution per event. I know of &lt;em&gt;no&lt;/em&gt; case nor reason where this should not be safely assumed by itself (either it happens or it doesn't, and in a population you can always assign a probability to this). There is no reason the upper limit on your number of events per nest should influence this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That distribution, by linearity, is assumed conditionally on the year, where the logodds are linear in year. This could be faulty, but that has nothing to do with the possible number of events, just the fact that any model can be wrong.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can (with &lt;code&gt;predict(type=&quot;response&quot;)&lt;/code&gt;) get the probability of an egg hatching, conditional on the year from this type of model (technically that is not exactly the same as a rate, but for most practical purposes, it is).&lt;/p&gt;&#xA;" OwnerUserId="4257" LastActivityDate="2011-05-20T22:08:44.360" CommentCount="3" />
  <row Id="11064" PostTypeId="1" AcceptedAnswerId="11065" CreationDate="2011-05-20T22:18:14.793" Score="11" ViewCount="173" Body="&lt;p&gt;In a number of statistical packages including SAS, SPSS and maybe more, there is an option to &quot;suppress the intercept&quot;. Why would you want to do that?&lt;/p&gt;&#xA;" OwnerUserId="333" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-05-21T22:51:01.893" LastActivityDate="2011-05-21T22:51:01.893" Title="Why would one suppress the intercept in linear regression?" Tags="&lt;regression&gt;" AnswerCount="3" />
  <row Id="11065" PostTypeId="2" ParentId="11064" CreationDate="2011-05-20T22:30:41.580" Score="9" ViewCount="" Body="&lt;p&gt;If for some reason you &lt;em&gt;know&lt;/em&gt; the intercept (particularly if it is zero), you can avoid wasting the variance in your data for estimating something you already know, and have more confidence in the values you &lt;em&gt;do&lt;/em&gt; have to estimate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A somewhat oversimplified example is if you already know (from domain knowledge) that one variable is (on average) a multiple of another, and you are trying to find that multiple.&lt;/p&gt;&#xA;" OwnerUserId="4257" LastActivityDate="2011-05-20T22:30:41.580" CommentCount="3" />
  <row Id="11066" PostTypeId="1" AcceptedAnswerId="11067" CreationDate="2011-05-20T22:31:13.453" Score="0" ViewCount="48" Body="&lt;p&gt;I'm estimating a GLM with a bunch of parameters in R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I run this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  M &amp;lt;- glm( Y ~ factor(X1) + factor(X2) )   &#xA;  summary(M)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;R only gives me part of the table, then cuts out with the message:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[ reached getOption(&quot;max.print&quot;) -- omitted 621 rows ]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The summary table will be big, but I want the whole thing.  How do I change the max.print option?  I've tried several different versions and nothing works.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: Here's another attempt which gives me a different error.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;summary(M, options(max.print=100000))&#xA;Error in dispersion * covmat.unscaled : &#xA;  non-numeric argument to binary operator&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="4110" LastEditorUserId="4110" LastEditorDisplayName="" LastEditDate="2011-05-20T22:46:20.803" LastActivityDate="2011-05-21T06:14:51.543" Title="How do I change the max.print option in R's summary?" Tags="&lt;r&gt;&lt;glm&gt;&lt;summary-statistics&gt;" AnswerCount="2" />
  <row Id="11067" PostTypeId="2" ParentId="11066" CreationDate="2011-05-20T22:38:56.957" Score="2" ViewCount="" Body="&lt;p&gt;I just googled &quot;R getOption(&quot;max.print&quot;)&quot;, and found: &lt;code&gt;options(max.print=5.5E5)&lt;/code&gt;...&lt;/p&gt;&#xA;" OwnerUserId="4257" LastActivityDate="2011-05-20T22:38:56.957" CommentCount="2" />
  <row Id="11068" PostTypeId="2" ParentId="11064" CreationDate="2011-05-20T23:48:34.010" Score="7" ViewCount="" Body="&lt;p&gt;Consider the case of a 3-level categorical covariate.  If one has an intercept, that would require 2 indicator variables.  Using the usual coding for indicator variables, the coefficient for either indicator variable is the mean difference compared to the reference group.  By suppressing the intercept, you would have 3 variables representing the categorical covariate, instead of just 2.  A coefficient is then the mean estimate for that group.  A more concrete example of where to do this is in political science where one may be studying the 50 states of the United States.  Instead of having an intercept and 49 indicator variables for the states, it is often preferable to suppress the intercept and instead have 50 variables.&lt;/p&gt;&#xA;" OwnerUserId="2312" LastActivityDate="2011-05-20T23:48:34.010" CommentCount="1" />
  <row Id="11069" PostTypeId="1" CreationDate="2011-05-21T00:58:27.447" Score="0" ViewCount="38" Body="&lt;p&gt;I am trying to estimate time series model using gls method.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data is monthly from sep 1997 to april 2011 &lt;/p&gt;&#xA;&#xA;&lt;p&gt;First I estimate the model and know that the erorr are IMA(1,1). For that I use the code :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  tsdata=ts(data, start=c(1997,9), frequency=12)&#xA;&#xA;  difftsdata=diff(tsdata)&#xA;&#xA;  trend = time(difftsdata)   &#xA;&#xA;  model= Arima(difftsdata, order=c(0,0,1), xreg=trend)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My questions are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Is the code above is apropriate?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;How to forecast from this model?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I tried using the following code:  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  nobs=length(difftsdata)                &#xA;&#xA;  fore=predict(model , 10, newxreg=(nobs+1):(nobs+10))&#xA;&#xA;  ts.plot(tsdata,fore$pred,col=1:2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;but the forecast is larger than expected.&lt;/p&gt;&#xA;" OwnerUserId="4698" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-21T05:34:45.183" LastActivityDate="2011-05-21T05:34:45.183" Title="Estimation of time series regression using GLS" Tags="&lt;time-series&gt;&lt;arima&gt;&lt;gls&gt;" CommentCount="6" />
  <row Id="11070" PostTypeId="1" CreationDate="2011-05-21T03:05:59.953" Score="4" ViewCount="61" Body="&lt;p&gt;The &quot;Linear Ballistic Accumulator&quot; model (LBA) is a rather successful model for human behaviour in speeded simple decision tasks. &lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pubmed/19897817&quot; rel=&quot;nofollow&quot;&gt;Donkin et al&lt;/a&gt; (2009, &lt;a href=&quot;http://mypage.iu.edu/~cdonkin/pubs/brm09b.pdf&quot; rel=&quot;nofollow&quot;&gt;PDF&lt;/a&gt;) provide code that permits estimating the parameters of the model given human behavioural data, and I've copied this code (with some minor formatting changes) to a gist &lt;a href=&quot;https://gist.github.com/941314&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;. However, I'd like to make a seemingly minor modification to the model but I'm not sure how to achieve this modification in the code.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To start with the canonical model, LBA represents each response alternative as a competitor in a rather strange race such that the competitors can differ in the following characteristics:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Starting position: this varies from race to race according to a uniform distribution bounded by U(0,X1).&lt;/li&gt;&#xA;&lt;li&gt;Speed: this is kept constant within a given race (no acceleration) but varies from race to race according to a Gaussian distribution defined by N(X2,X3)&lt;/li&gt;&#xA;&lt;li&gt;Finish line position (X4)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Thus, each competitor has its own set of values for X1, X2, X3 and X4.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The race is repeated many times, with the winner and their time recorded after each race. A constant of X5 is added to every winning time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, the modification I want to make is to swap the variability in the starting point to the finish line. That is, I want the start point to be zero for all competitors and all races, thereby eliminating X1, but I want to add a parameter, X6, that specifies the size of the range of a uniform distribution centered on X4 from which each competitor's finish line is sampled for each race. In this model, then, each competitor will have values for X2, X3, X4, and X6, and we still have the across-competitor value for X5.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd be very grateful if anyone is willing to help with this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Oh, and to provide a mapping from the &quot;X&quot; named parameters described above to the variable names used by the LBA code I linked: X1 = x0max; X2 = driftrate; X3 = sddrift; X4 = chi; X5 = Ter.&lt;/p&gt;&#xA;" OwnerUserId="364" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-21T12:27:24.327" LastActivityDate="2011-05-21T12:27:24.327" Title="Modifying linear ballistic accumulator (LBA) simulation in  R" Tags="&lt;stochastic-processes&gt;" />
  <row Id="11071" PostTypeId="1" CreationDate="2011-05-21T03:11:44.490" Score="0" ViewCount="26" Body="&lt;p&gt;I wonder how to get LATEX table using xtable function from the following &lt;em&gt;R&lt;/em&gt; code.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Block &amp;lt;- gl(8, 4)&#xA;A &amp;lt;- factor(c(0,1,0,1,0,1,0,1,0,1,0,1,0,1,0,1,0,1,0,1,&#xA;          0,1,0,1,0,1,0,1,0,1,0,1))&#xA;B &amp;lt;- factor(c(0,0,1,1,0,0,1,1,0,1,0,1,1,0,1,0,0,0,1,1,&#xA;          0,0,1,1,0,0,1,1,0,0,1,1))&#xA;C &amp;lt;- factor(c(0,1,1,0,1,0,0,1,0,0,1,1,0,0,1,1,0,1,0,1,&#xA;          1,0,1,0,0,0,1,1,1,1,0,0))&#xA;Yield &amp;lt;- c(101, 373, 398, 291, 312, 106, 265, 450, 106, 306, 324, 449,&#xA;       272, 89, 407, 338, 87, 324, 279, 471, 323, 128, 423, 334,&#xA;       131, 103, 445, 437, 324, 361, 302, 272)&#xA;aovdat &amp;lt;- data.frame(Block, A, B, C, Yield)&#xA;&#xA;summary(aov(Yield~Block+A*B+Error(A*Block), data=aovdat))&#xA;&#xA;&#xA;Error: A&#xA;  Df Sum Sq Mean Sq&#xA;A  1 3465.3  3465.3&#xA;&#xA;Error: Block&#xA;      Df Sum Sq Mean Sq&#xA;Block  7   4499  642.71&#xA;&#xA;Error: A:Block&#xA;          Df Sum Sq Mean Sq F value Pr(&amp;gt;F)&#xA;B          1  41616   41616  3.5354 0.1091&#xA;Residuals  6  70628   11771               &#xA;&#xA;Error: Within&#xA;          Df Sum Sq Mean Sq F value  Pr(&amp;gt;F)  &#xA;B          1 119568  119568  7.3751 0.01673 *&#xA;A:B        1     28      28  0.0017 0.96734  &#xA;Residuals 14 226975   16213                  &#xA;---&#xA;Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Using the following code&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;xtable(summary(aov(Yield~Block+A*B+Error(A*Block), data=aovdat)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;producing the following error&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Error in rbind(deparse.level, ...) : &#xA;  numbers of columns of arguments do not match&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="3903" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-21T05:59:20.980" LastActivityDate="2011-05-21T06:51:23.910" ClosedDate="2011-05-21T09:10:29.893" Title="LaTeX table through xtable" Tags="&lt;r&gt;&lt;latex&gt;" AnswerCount="0" CommentCount="1" />
  <row Id="11072" PostTypeId="1" CreationDate="2011-05-21T03:22:11.233" Score="1" ViewCount="88" Body="&lt;p&gt;Hi &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is perhaps basic but I couldn't find a suitable reference. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a regression model with a rather complicated link function. &#xA;So $\vec{x}$ is a vector of continuous predictors, and $z$ is a binary variable such &#xA;that according to the model: $Pr(z=1)  = f(\vec{x})$ for some (known) function $f$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I observe data of the form $(\vec{x}^{(1)}, z^{(1)}), (\vec{x}^{(2)}, z^{(2)}), (\vec{x}^{(n)}, z^{(n)})$ and want to test the null hypothesis that the above model is the one generating the data - that is compute a statistic and reject the model if the statistic is too extreme. What would be a good goodness-of-fit test for this case? is there a 'standard' way to test for this? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One possibility is binning the data points by the value of  $f(\vec{x})$, (say to $10$ bins: $([0,0.1], ..[0.9,1])$ and performing a chi-square test for expected vs. observed proportion of $z$'s in each bin. Another is to bin the multidimensional space of the $\vec{x}$'s (say if $\vec{x}$ is two-dimensional, we can divide $R^2$ to $100$ squares and compute a chi-square for observed vs. expected for each square). Yet another one is not binning at all but just computing $\sum_i (z^{(i)} - f(\vec{x}^{(i)}))^2/f(\vec{x}^{(i)})$&#xA;but this seems to cause numerical issues since sometimes $f(\vec{x}^{(i)})$ is very small.&#xA;Are there other known approaches? which test would be the most appropriate?&lt;/p&gt;&#xA;" OwnerUserId="3036" LastEditorUserId="3036" LastEditorDisplayName="" LastEditDate="2011-05-21T05:25:53.687" LastActivityDate="2011-05-26T18:25:22.063" Title="Goodness of fit for a regression with multiple predictors" Tags="&lt;regression&gt;&lt;goodness-of-fit&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="11073" PostTypeId="2" ParentId="11039" CreationDate="2011-05-21T04:29:57.587" Score="1" ViewCount="" Body="&lt;p&gt;You originally asked this in reference to R (on stackoverflow), so I'll answer with reference to R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not sure exactly what your goal is, but I'd guess that you want to estimate a panel data model.   In that context, you have an &quot;unbalanced panel&quot; and if you want to stick with R,  I'd recommend the package plm.  The package documentation describes how it deals with unbalanced panels. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You will need to sort out how to model panel data and interpret the results--I'd suggest looking at an econometrics textbook (whichever one is at the appropriate level for you; these models are covered in almost all econometrics texts).  The book &quot;Applied Econometrics with R&quot; also gives some examples and a short, fairly accessible (if terse) introduction to these and other models.&lt;/p&gt;&#xA;" OwnerUserId="4699" LastEditorUserId="4699" LastEditorDisplayName="" LastEditDate="2011-05-21T05:08:42.717" LastActivityDate="2011-05-21T05:08:42.717" />
  <row Id="11074" PostTypeId="2" ParentId="11066" CreationDate="2011-05-21T06:14:51.543" Score="1" ViewCount="" Body="&lt;p&gt;You may want to use &lt;code&gt;sink&lt;/code&gt;. This will divert output to a file, which then you can inspect. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sink(file=&quot;output.txt&quot;)&#xA;summary(M)&#xA;sink(NULL)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The error you get in your code is because the second argument of function &lt;code&gt;summary.glm&lt;/code&gt; is &lt;code&gt;dispersion&lt;/code&gt;, which according to help page should be either numeric or NULL. You supply the output of &lt;code&gt;options(max.print=100000)&lt;/code&gt; which is not what R expects, hence the error.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-05-21T06:14:51.543" />
  <row Id="11076" PostTypeId="2" ParentId="11064" CreationDate="2011-05-21T06:57:45.463" Score="3" ViewCount="" Body="&lt;p&gt;To illustrate @Nick Sabbe's point with a specific example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I once saw a researcher present a model of the age of a tree as a function of its width. It can be assumed that when the tree is at age zero, it effectively has a width of zero. Thus, an intercept is not required.&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2011-05-21T06:57:45.463" CommentCount="4" />
  <row Id="11077" PostTypeId="2" ParentId="10167" CreationDate="2011-05-21T07:31:43.063" Score="2" ViewCount="" Body="&lt;p&gt;The following are just a few points:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;If you have departure from normality then bootstrapping is often a good idea. &lt;/li&gt;&#xA;&lt;li&gt;You mention using &quot;1000&quot; replicates. Increasing the number of replicates increases computational time and accuracy. Thus, sometimes when first setting up your model, you'll set the number of replicates at a level that is relatively quick to run. However, for your final model that you report, you may want to push up the number of replicates to 10,000 or more.&lt;/li&gt;&#xA;&lt;li&gt;If the departure of your data from normality is mild, then coefficient and model fit tests that assume normality are often a reasonable approximation. In particular when you have a big sample, as is often the case with structural equation modelling, assumption tests that perform a significant test with the null hypothesis as normality often are overly sensitive for the purpose of deciding whether to persist with methods that  assume normality. I would pay more attention to the actual indices of non-normality like skewness and kurtosis values (or if your intuition is sufficiently trained, check out histograms of the variables).&lt;/li&gt;&#xA;&lt;li&gt;If the departure from normality is mild, I would expect that both standard and bootstrapped approaches should yield similar results. Showing that your results are robust to such analytic decisions may provide you with greater confidence in your results.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-21T07:54:27.303" LastActivityDate="2011-05-21T07:54:27.303" CommentCount="1" />
  <row Id="11078" PostTypeId="2" ParentId="11032" CreationDate="2011-05-21T08:51:14.170" Score="2" ViewCount="" Body="&lt;p&gt;Baum-Welch is an optimization algorithm for computing the maximum-likelihood estimator. For hidden Markov models the likelihood surface may be quite ugly, and it is certainly not concave. With good starting points the algorithm may converge faster and towards the MLE. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you already know the transition probabilities and want to predict hidden states by the Viterbi algorithm, you need the transition probabilities. If you already know them, there is no need to re-estimate them using Baum-Welch. The re-estimation is computationally more expensive than the prediction.  &lt;/p&gt;&#xA;" OwnerUserId="4376" LastActivityDate="2011-05-21T08:51:14.170" />
  <row Id="11079" PostTypeId="1" CreationDate="2011-05-21T12:24:06.390" Score="1" ViewCount="127" Body="&lt;p&gt;Hello everybody,&#xA;I need an help because I don´t know if the command for the ANOVA analysis I am &#xA;performing in R is correct. Indeed using the function aov I get the following error: &lt;code&gt;In aov (......) Error() model is singular&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The structure of my table is the following: subject, stimulus, condition, sex, response&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;subject  stimulus condition sex    response&#xA;subject1    gravel  EXP1    M      59.8060&#xA;subject2    gravel  EXP1    M      49.9880&#xA;subject3    gravel  EXP1    M      73.7420&#xA;subject4    gravel  EXP1    M      45.5190&#xA;subject5    gravel  EXP1    M      51.6770&#xA;subject6    gravel  EXP1    M      42.1760&#xA;subject7    gravel  EXP1    M      56.1110&#xA;subject8    gravel  EXP1    M      54.9500&#xA;subject9    gravel  EXP1    M      62.6920&#xA;subject10   gravel  EXP1    M      50.7270&#xA;subject1    gravel  EXP2    M      70.9270&#xA;subject2    gravel  EXP2    M      61.3200&#xA;subject3    gravel  EXP2    M      70.2930&#xA;subject4    gravel  EXP2    M      49.9880&#xA;subject5    gravel  EXP2    M      69.1670&#xA;subject6    gravel  EXP2    M      62.2700&#xA;subject7    gravel  EXP2    M      70.9270&#xA;subject8    gravel  EXP2    M      63.6770&#xA;subject9    gravel  EXP2    M      72.4400&#xA;subject10   gravel  EXP2    M      58.8560&#xA;subject11   gravel  EXP1    F      46.5750&#xA;subject12   gravel  EXP1    F      58.1520&#xA;subject13   gravel  EXP1    F      57.4490&#xA;subject14   gravel  EXP1    F      59.8770&#xA;subject15   gravel  EXP1    F      55.5480&#xA;subject16   gravel  EXP1    F      46.2230&#xA;subject17   gravel  EXP1    F      63.3260&#xA;subject18   gravel  EXP1    F      60.6860&#xA;subject19   gravel  EXP1    F      59.4900&#xA;subject20   gravel  EXP1    F      52.6630&#xA;subject11   gravel  EXP2    F      55.7240&#xA;subject12   gravel  EXP2    F      66.4220&#xA;subject13   gravel  EXP2    F      65.9300&#xA;subject14   gravel  EXP2    F      61.8120&#xA;subject15   gravel  EXP2    F      62.5160&#xA;subject16   gravel  EXP2    F      65.5780&#xA;subject17   gravel  EXP2    F      59.5600&#xA;subject18   gravel  EXP2    F      63.8180&#xA;subject19   gravel  EXP2    F      61.4250&#xA;.....&#xA;.....&#xA;.....&#xA;.....&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As you can notice each subject repeated the evaluation in 2 conditions (EXP1 and EXP2).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I am interested in is to know if there are significant differences between &#xA;the evaluations of the males and the females.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is the command I used to perform the ANOVA with repeated measures:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;aov1 = aov(response ~ stimulus*sex + Error(subject/(stimulus*sex)), data=scrd)&#xA;summary(aov1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I get the following error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; aov1 = aov(response ~ stimulus*sex + Error(subject/(stimulus*sex)), data=scrd)&#xA;Warning message:&#xA;In aov(response ~ stimulus * sex + Error(subject/(stimulus * sex)),  :&#xA;Error() model is singular&#xA;&amp;gt; summary(aov1)&#xA;&#xA;Error: subject&#xA;          Df  Sum Sq Mean Sq F value Pr(&amp;gt;F)&#xA;sex        1  166.71  166.72   1.273  0.274&#xA;Residuals 18 2357.29  130.96               &#xA;&#xA;Error: subject:stimulus&#xA;              Df Sum Sq Mean Sq F value Pr(&amp;gt;F)    &#xA;stimulus       6 7547.9 1257.98 35.9633 &amp;lt;2e-16 ***&#xA;stimulus:sex   6   94.2   15.70  0.4487 0.8445    &#xA;Residuals    108 3777.8   34.98                   &#xA;---&#xA;Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 &#xA;&#xA;Error: Within&#xA;           Df Sum Sq Mean Sq F value Pr(&amp;gt;F)&#xA;Residuals 420 9620.6  22.906               &#xA;&amp;gt; &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The thing is that looking at the data it is evident for me that there is a &#xA;difference between male and females, because for each stimulus I always get&#xA;a mean higher for the males rather than the females. &#xA;Therefore the ANOVA should indicate significant differences....&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there anyone who can suggest me where I am wrong?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, I know that in R there are two libraries on linear mixed models called &#xA;nlme and lme4, but I have never used it so far and I don´t know if I have to utilize it for my case.&#xA;Is it the case to utilize it? If yes, could you please provide a quick R example&#xA;of a command which could solve my problem?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Best regards&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Dear all, &#xA;I am stuck now ;-( Indeed I understood everything you suggested me but still I don´t get significance in the ANOVA results, and definitively there is an error, because results cannot be non-significant. Indeed looking at the means for each stimulus, it is possible to notice that males gave always higher evaluations than females.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To prove this I discarded for a moment the effect of the repeated measures, and I performed an ANOVA separately on both the two conditions (EXP1 and EXP2) during which the evaluations were given.&#xA;What I get is significant differences between males and female, in both EXP1 and EXP2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, why when I perform the ANOVA with repeated measures I don´t get the same behavior?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My design is the following:&#xA;-sex is a between-subjects factor (with two levels)&#xA;-stimulus is a within-subjects factor (with 3 assumed levels)&#xA;-condition is a within-subjects factor (with 2 levels)&#xA;-all factors are fully crossed&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried, both the ways suggested but without achieving significance: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mDf &amp;lt;- aggregate(response ~ subject + sex, data=scrd, FUN=mean)&#xA;summary(aov(response ~ sex, data=mDf))     # ANOVA with just the between-effect&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;aov1 = aov(response ~ sex*stimulus*condition + Error(subject/(stimulus*condition)), data=scrd)&#xA;summary(aov1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Instead if I perform the ANOVA on the two subtables of EXP 1 and 2 I get significant differences. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;table_EXP1 &amp;lt;- subset(scrd, condition == &quot;EXP1&quot;)&#xA;table_EXP2 &amp;lt;- subset(scrd, condition == &quot;EXP2&quot;)&#xA;&#xA;&#xA;fit_table_EXP1 &amp;lt;- lm(response ~ stimulus*sex, data=table_EXP1) &#xA;summary(fit_table_EXP1 )&#xA;anova(fit_table_EXP1 )&#xA;&#xA;&#xA;fit_table_EXP2 &amp;lt;- lm(response ~ stimulus*sex, data=table_EXP2) &#xA;summary(fit_table_EXP2)&#xA;anova(fit_table_EXP2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;....how can this be possible?...it is a contraddiction....&lt;/p&gt;&#xA;&#xA;&lt;p&gt;HELP!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please enlighten me!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cheers&lt;/p&gt;&#xA;" OwnerUserId="4701" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-05-28T20:12:23.450" LastActivityDate="2011-05-28T20:12:23.450" Title="Problem with ANOVA repeated measures: &quot;Error() model is singular&quot;" Tags="&lt;r&gt;&lt;anova&gt;&lt;mixed-model&gt;" AnswerCount="2" CommentCount="8" />
  <row Id="11080" PostTypeId="2" ParentId="11009" CreationDate="2011-05-21T12:31:20.447" Score="3" ViewCount="" Body="&lt;p&gt;In my experience, not only is it necessary to have all lower order effects in the model when they are connected to higher order effects, but it is also important to properly model (e.g., allowing to be nonlinear) main effects that are seemingly unrelated to the factors in the interactions of interest.  That's because interactions between x1 and x2 can be stand-ins for main effects of x3 and x4.  I.e. sometimes &lt;em&gt;seem&lt;/em&gt; to be needed because they are collinear with omitted variables or omitted nonlinear (e.g., spline) terms.&lt;/p&gt;&#xA;" OwnerUserId="4253" LastActivityDate="2011-05-21T12:31:20.447" />
  <row Id="11081" PostTypeId="2" ParentId="11009" CreationDate="2011-05-21T14:49:49.257" Score="1" ViewCount="" Body="&lt;p&gt;I would suggest it is simply a special case of model uncertainty.  From a Bayesian perspective, you simply treat this in exactly the same way you would treat any other kind of uncertainty, by either:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Calculating its probability, if it is the object of interest&lt;/li&gt;&#xA;&lt;li&gt;Integrating or averaging it out, if it is not of interest, but may still affect your conclusions&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;This is exactly what people do when testing for &quot;significant effects&quot; by using t-quantiles instead of normal quantiles.  Because you have uncertainty about the &quot;true noise level&quot; you take this into account by using a more spread out distribution in testing.  So from your perspective the &quot;main effect&quot; is actually a &quot;nuisance parameter&quot; in relation to the question that you are asking.  So you simply average out the two cases (or more generally, over the models you are considering).  So I would have the (vague) hypothesis:&#xA;$$H_{int}:\text{The interaction between A and B is significant}$$&#xA;I would say that although not precisely defined, this is the question you want to answer here.  And note that it is not the verbal statements such as above which &quot;define&quot; the hypothesis, but the mathematical equations as well.  We have some data $D$, and prior information $I$, then we simply calculate:&#xA;$$P(H_{int}|DI)=P(H_{int}|I)\frac{P(D|H_{int}I)}{P(D|I)}$$&#xA;(small note: no matter how many times I write out this equation, it always helps me understand the problem better. weird).  The main quantity to calculate is the likelihood $P(D|H_{int}I)$,  this makes no reference to the model, so the model must have been removed using the law of total probability:&#xA;$$P(D|H_{int}I)=\sum_{m=1}^{N_{M}}P(DM_{m}|H_{int}I)=\sum_{m=1}^{N_{M}}P(M_{m}|H_{int}I)P(D|M_{m}H_{int}I)$$&#xA;Where $M_{m}$ indexes the mth model, and $N_{M}$ is the number of models being considered.  The first term is the &quot;model weight&quot; which says how much the data and prior information support the mth model.  The second term indicates how much the mth model supports the hypothesis.  Plugging this equation back into the original Bayes theorem gives:&#xA;$$P(H_{int}|DI)=\frac{P(H_{int}|I)}{P(D|I)}\sum_{m=1}^{N_{M}}P(M_{m}|H_{int}I)P(D|M_{m}H_{int}I)$$&#xA;$$=\frac{1}{P(D|I)}\sum_{m=1}^{N_{M}}P(DM_{m}|I)\frac{P(M_{m}H_{int}D|I)}{P(DM_{m}|I)}=\sum_{m=1}^{N_{M}}P(M_{m}|DI)P(H_{int}|DM_{m}I)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And you can see from this that $P(H_{int}|DM_{m}I)$ is the &quot;conditional conclusion&quot; of the hypothesis under the mth model (this is usually all that is considered, for a chosen &quot;best&quot; model).  Note that this standard analysis is justified whenever $P(M_{m}|DI)\approx 1$ - an &quot;obviously best&quot; model - or whenever $P(H_{int}|DM_{j}I)\approx P(H_{int}|DM_{k}I)$ - all models give the same/similar conclusions.  However if neither are met, then Bayes Theorem says the best procedure is to average out the results, placing higher weights on the models which are most supported by the data and prior information.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-05-21T14:49:49.257" />
  <row Id="11083" PostTypeId="2" ParentId="3814" CreationDate="2011-05-21T16:04:11.060" Score="1" ViewCount="" Body="&lt;p&gt;Using causal language to describe associations in observational data when omitted variables are almost certainly a serious concern.&lt;/p&gt;&#xA;" OwnerUserId="3748" LastActivityDate="2011-05-21T16:04:11.060" CommunityOwnedDate="2011-05-21T16:04:11.060" CommentCount="1" />
  <row Id="11084" PostTypeId="1" AcceptedAnswerId="11086" CreationDate="2011-05-21T16:36:36.803" Score="5" ViewCount="72" Body="&lt;p&gt;I've received a results from a Mann-Whitney rank test that I don't understand. &#xA;The median of the 2 populations is identical (6.9). The uppper and lower quantiles of each population are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;6.64 &amp;amp; 7.2&lt;/li&gt;&#xA;&lt;li&gt;6.60 &amp;amp; 7.1&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The p-value resulting from the test comparing these populations is 0.007. How can these populations be significantly different? Is it due to the spread about the median? A boxplot comparing the 2 shows that the second one has far more outliers than the first.&#xA;Thanks for any suggestions.&lt;/p&gt;&#xA;" OwnerUserId="4238" LastEditorUserId="307" LastEditorDisplayName="" LastEditDate="2011-05-21T18:43:02.763" LastActivityDate="2011-05-21T18:43:02.763" Title="Why is the Mann–Whitney U test significant when the medians are equal?" Tags="&lt;nonparametric&gt;&lt;median&gt;&lt;ranks&gt;&lt;mann-whitney-u-test&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="11085" PostTypeId="1" CreationDate="2011-05-21T16:39:00.437" Score="1" ViewCount="28" Body="&lt;p&gt;Hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have conducted a search for genetic interactions using a simple dosage model:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Y ~ A + B + AB&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where Y is the phenotype, in this case, gene expression values and A and B are vectors of genotype information for ~500 samples. I wish to determine a signficance threshold using permutation testing in order to correct for multiple testing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To date, I have recalculated the p-values for the interaction term (AB) for 100 permutations (I permuted the phenotype values) and am unsure how to proceed in order to derive a false discovery rate (FDR).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestions?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks, D.&lt;/p&gt;&#xA;" OwnerUserId="2842" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-21T17:01:38.293" LastActivityDate="2011-05-21T17:01:38.293" Title="False discovery rate from permutation testing?" Tags="&lt;genetics&gt;&lt;permutation&gt;&lt;multiple-hypothesis&gt;" />
  <row Id="11086" PostTypeId="2" ParentId="11084" CreationDate="2011-05-21T16:50:26.763" Score="7" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://www.ats.ucla.edu/stat/mult_pkg/faq/general/mann-whitney.htm&quot; rel=&quot;nofollow&quot;&gt;FAQ: Why is the Mann-Whitney significant when the medians are equal?&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="307" LastActivityDate="2011-05-21T16:50:26.763" CommentCount="5" />
  <row Id="11087" PostTypeId="1" AcceptedAnswerId="11089" CreationDate="2011-05-21T18:25:00.283" Score="6" ViewCount="192" Body="&lt;p&gt;I was just wondering why regression problems are called &quot;regression&quot; problems. What is the story behind the name? &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;One definition for regression:&#xA;  &quot;Relapse to a less perfect or&#xA;  developed state.&quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="3541" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-21T18:44:13.433" LastActivityDate="2011-05-21T18:54:18.390" Title="Why are regression problems called &quot;regression&quot; problems?" Tags="&lt;regression&gt;&lt;terminology&gt;" AnswerCount="1" />
  <row Id="11088" PostTypeId="1" CreationDate="2011-05-21T18:53:13.727" Score="4" ViewCount="161" Body="&lt;p&gt;I am a newbie in stat. I am completing my thesis in &lt;a href=&quot;http://en.wikipedia.org/wiki/Evolutionary_algorithm&quot; rel=&quot;nofollow&quot;&gt;Evolutionary algorithm&lt;/a&gt;. I have to generate some random numbers from &lt;a href=&quot;http://en.wikipedia.org/wiki/Student%27s_t-distribution&quot; rel=&quot;nofollow&quot;&gt;T-distribution&lt;/a&gt; or &lt;a href=&quot;http://en.wikipedia.org/wiki/Laplace_distribution&quot; rel=&quot;nofollow&quot;&gt;Laplace distribution&lt;/a&gt;. How can I do this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An easy and simple explanation would be appreciated. Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="4319" LastEditorUserId="4319" LastEditorDisplayName="" LastEditDate="2011-05-23T08:05:20.197" LastActivityDate="2011-05-23T17:29:06.840" Title="Random number generation using t-distribution or laplace distribution" Tags="&lt;distributions&gt;&lt;matlab&gt;&lt;random-generation&gt;" AnswerCount="4" CommentCount="6" FavoriteCount="1" />
  <row Id="11089" PostTypeId="2" ParentId="11087" CreationDate="2011-05-21T18:54:18.390" Score="9" ViewCount="" Body="&lt;p&gt;The term &quot;regression&quot; was used by Francis Galton in his 1886 paper &quot;Regression towards mediocrity in hereditary stature&quot;. To my knowledge he only used the term in the context of &lt;a href=&quot;http://en.wikipedia.org/wiki/Regression_toward_the_mean&quot; rel=&quot;nofollow&quot;&gt;regression toward the mean&lt;/a&gt;. The term was then adopted by others to get more or less the meaning it has today as a general statistical method. &lt;/p&gt;&#xA;" OwnerUserId="4376" LastActivityDate="2011-05-21T18:54:18.390" CommentCount="1" />
  <row Id="11090" PostTypeId="2" ParentId="11088" CreationDate="2011-05-21T19:05:29.677" Score="5" ViewCount="" Body="&lt;p&gt;Easy answer: Use R and get &lt;code&gt;n&lt;/code&gt; variables for a $t$-distribution with &lt;code&gt;df&lt;/code&gt; degrees of freedom by &lt;code&gt;rt(n, df)&lt;/code&gt;. If you don't use R, maybe you can write what language you use, and others may be able to tell precisely what to do. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you don't use R or another language with a built in random number generator for the $t$-distribution, but you have access to the &lt;em&gt;quantile function&lt;/em&gt;, $Q$, for the $t$-distribution and you can generate a uniform random variable $U$ on $[0,1]$ then $Q(U)$ follows a $t$-distribution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Else take a look at &lt;a href=&quot;http://en.wikipedia.org/wiki/Student%27s_t-distribution#Monte_Carlo_sampling&quot; rel=&quot;nofollow&quot;&gt;this brief section&lt;/a&gt; in the Wikipedia page.&lt;/p&gt;&#xA;" OwnerUserId="4376" LastActivityDate="2011-05-21T19:05:29.677" CommentCount="3" />
  <row Id="11091" PostTypeId="1" AcceptedAnswerId="11094" CreationDate="2011-05-21T19:13:02.953" Score="5" ViewCount="49" Body="&lt;p&gt;I believe that independent variables $X_1,X_2$ affect the dependent variable $Y$ through a latent variable $Z$ such that &#xA;$$&#xD;&#xA;\begin{align}&#xD;&#xA;Y &amp;amp;= \beta_0 + \beta_1Z \\&#xD;&#xA;Z &amp;amp;= \operatorname{Logit}^{-1}(\beta_2X_1 + \beta_3X_2) \\&#xD;&#xA;\\&#xD;&#xA;Y &amp;amp;= \beta_0 + \beta_1\operatorname{Logit}^{-1}(\beta_2X_1 + \beta_3X_2)&#xD;&#xA;\end{align}&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possible to estimate $\beta_2$ and $\beta_3$, given $Y$?&lt;/p&gt;&#xA;" OwnerUserId="82" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-05-21T21:25:11.250" LastActivityDate="2011-05-21T21:25:11.250" Title="Estimating effect of latent variable in regression" Tags="&lt;regression&gt;&lt;logit&gt;&lt;binary-data&gt;" AnswerCount="1" CommentCount="5" />
  <row Id="11092" PostTypeId="2" ParentId="11088" CreationDate="2011-05-21T20:47:23.670" Score="8" ViewCount="" Body="&lt;p&gt;Here's how to do this in Matlab using &lt;a href=&quot;http://www.mathworks.com/help/toolbox/stats/tinv.html&quot; rel=&quot;nofollow&quot;&gt;TINV&lt;/a&gt; from that statistics toolbox:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;%# choose the degree of freedom&#xA;df = 4; %# note you can also choose an array of df's if necessary&#xA;&#xA;%# create a vector of 100,000 uniformly distributed random varibles&#xA;uni = rand(100000,1);&#xA;&#xA;%# look up the corresponding t-values&#xA;out = tinv(uni,df);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;With a more recent version of Matlab, you can also simply use &lt;a href=&quot;http://www.mathworks.com/help/toolbox/stats/trnd.html&quot; rel=&quot;nofollow&quot;&gt;TRND&lt;/a&gt; to create the random numbers directly.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;out = trnd(100000,df);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here's the histogram of &lt;code&gt;out&lt;/code&gt; &lt;img src=&quot;http://i.stack.imgur.com/HVi6Y.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT Re:merged question&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Matlab has no built-in function for drawing numbers from a Laplace distribution. However, there is the function &lt;a href=&quot;http://www.mathworks.com/matlabcentral/fileexchange/13705-laplacian-random-number-generator&quot; rel=&quot;nofollow&quot;&gt;LAPRND&lt;/a&gt; from the Matlab File Exchange that provides a well-written implementation. &lt;/p&gt;&#xA;" OwnerUserId="198" LastEditorUserId="198" LastEditorDisplayName="" LastEditDate="2011-05-22T17:35:16.307" LastActivityDate="2011-05-22T17:35:16.307" />
  <row Id="11093" PostTypeId="1" CreationDate="2011-05-21T21:07:11.437" Score="2" ViewCount="39" Body="&lt;p&gt;Apologies for what is probably a very basic question. I have looked around both here and in the usual places and haven't had any luck.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have read that there are at least two methods for linearly transforming data so that you can give your distribution a certain desired standard deviation. What are they and are there cases where you'd want to use one method rather than another?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just for concreteness's sake, let's say you have test scores from 0-50, a mean of 35 and sd of 10, and you wanted to rescale so the sd is 15.&lt;/p&gt;&#xA;" OwnerUserId="52" LastActivityDate="2011-05-21T22:38:27.183" Title="Rescaling for desired standard deviation" Tags="&lt;data-transformation&gt;&lt;standard-deviation&gt;" AnswerCount="2" />
  <row Id="11094" PostTypeId="2" ParentId="11091" CreationDate="2011-05-21T21:19:28.657" Score="7" ViewCount="" Body="&lt;p&gt;One answer is &quot;no.&quot;  Another is, &quot;of course.&quot;&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;No&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;To simplify notation, let $\lambda(x) = 1/(1 + \exp(-x))$, the inverse logit.  Because $\lambda(x) = 1 - \lambda(-x)$,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\beta_0 + \beta_1 \lambda(x) = (\beta_0 + \beta_1) - \beta_1 \lambda(-x)).$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore it is impossible to distinguish the parameters $(\beta_0, \beta_1, \beta_2, \beta_3)$ from $(\beta_0+\beta_1, -\beta_1, -\beta_2, -\beta_3)$.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Of course&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Let us stipulate that the first nonzero element of $(\beta_1, \beta_2, \beta_3)$ must be positive.  That resolves the indeterminacy.  We still need a model for the errors.  If we suppose, for instance, that $Y - \left(\beta_0 + \beta_1 \lambda(\beta_2 X_2 + \beta_3 X_3)\right)$ has a Normal distribution and the various $Y$'s are independent, then we can use least squares to estimate the parameters.  There is no exact solution to this nonlinear optimization problem, but it is straightforward to do numerically.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.stack.imgur.com/a0yvg.png&quot; alt=&quot;50 data points and their fit as a surface&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This graphic shows 50 points generated with standard Normal values for $X_1$ and $X_2$, parameter $\beta = (1,2,1/2,-1)$, with iid Normal errors of standard deviation 1/2.  The surface shows the fit, $\hat{\beta} = (2.68, -1.23, -0.89, 1.75) \sim (1.45, 1.23, 0.89, -1.75)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Least squares is the maximum likelihood with iid Normal errors.  With another error distribution, use MLE directly.  You can obtain asymptotic confidence intervals for the parameters in the standard ways.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-05-21T21:19:28.657" CommentCount="1" />
  <row Id="11095" PostTypeId="2" ParentId="11093" CreationDate="2011-05-21T21:30:00.263" Score="4" ViewCount="" Body="&lt;p&gt;The SD is directly proportional to the data.  Therefore, to change it from 10 to 15 = 1.5 * 10, multiply all scores by 1.5.  The other way is to multiply all scores by -1.5, because negating all values does not change the SD.  Of course you can also add an arbitrary constant to all the scores, too, without changing the SD.  That is an exhaustive description of the &lt;em&gt;linear&lt;/em&gt; transformations of the data that change the SD to the desired value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You would use the negative multiple when you want to reverse the order of the data.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-05-21T21:30:00.263" />
  <row Id="11096" PostTypeId="1" CreationDate="2011-05-21T15:10:15.500" Score="2" ViewCount="138" Body="&lt;p&gt;How can I interpret the main effects (coefficients for dummy-coded factor) in a Poisson regression?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assume the following example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;treatment &amp;lt;- factor(rep(c(1, 2), c(43, 41)), &#xA;                    levels = c(1, 2),&#xA;                    labels = c(&quot;placebo&quot;, &quot;treated&quot;))&#xA;improved &amp;lt;- factor(rep(c(1, 2, 3, 1, 2, 3), c(29, 7, 7, 13, 7, 21)),&#xA;                   levels = c(1, 2, 3),&#xA;                   labels = c(&quot;none&quot;, &quot;some&quot;, &quot;marked&quot;))    &#xA;numberofdrugs &amp;lt;- rpois(84, 10)+1    &#xA;healthvalue &amp;lt;- rpois(84, 5)   &#xA;y &amp;lt;- data.frame(healthvalue, numberofdrugs, treatment, improved)&#xA;test &amp;lt;- glm(healthvalue ~ numberofdrugs+treatment+improved, y, family=poisson)&#xA;summary(test)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The output is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Coefficients:&#xA;                 Estimate Std. Error z value Pr(&amp;gt;|z|)    &#xA;(Intercept)       1.88955    0.19243   9.819   &amp;lt;2e-16 ***&#xA;numberofdrugs    -0.02303    0.01624  -1.418    0.156    &#xA;treatmenttreated -0.01271    0.10861  -0.117    0.907   MAIN EFFECT  &#xA;improvedsome     -0.13541    0.14674  -0.923    0.356   MAIN EFFECT &#xA;improvedmarke    -0.10839    0.12212  -0.888    0.375   MAIN EFFECT &#xA;---&#xA;Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I know that the incident rate for &lt;code&gt;numberofdrugs&lt;/code&gt; is exp(-0.023)=0.977. But how do I interpret the main effects for the dummy variables? &lt;/p&gt;&#xA;" LastActivityDate="2011-05-22T03:17:12.347" Title="How to interpret coefficients in a Poisson regression?" Tags="&lt;r&gt;&lt;poisson&gt;&lt;glm&gt;" AnswerCount="1" CommentCount="4" />
  <row Id="11097" PostTypeId="2" ParentId="11096" CreationDate="2011-05-21T15:28:42.660" Score="5" ViewCount="" Body="&lt;p&gt;The exponentiated 'numberofdrugs' coefficient is the multiplicative term to use to calculate the estimated &quot;healthvalue&quot; when 'numberofdrugs' increases by 1 unit. In the case of categorical (factor) variables, the exponentiated coefficient is the multiplicative term relative to the base (first factor) level for that variable (since R uses treatment contrasts by default). The exp(Intercept) is the baseline rate, and all other estimates would be relative to it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your example the estimated 'healthvalue' for someone with 2 drugs, &quot;placebo&quot; and improvement==&quot;none&quot; would be (using addition inside exp as the equivalent of multiplication): &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; exp( 1.88955  + 2 *-0.02303 + 0 +0 )&#xA; [1] 6.318552&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;While someone on 4 drugs, 'treated', and 'some' improvement would have an estimated 'healthvalue' of &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;exp( 1.88955  + 4 *-0.02303  + -0.01271 + -0.13541)&#xA;[1] 38.44813&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;ADDENDUM: The way to return coefficients from regression objects is generally to use the &lt;code&gt;coef()&lt;/code&gt; extractor function (done with a different random realization below):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; coef(test)&#xA;  #   (Intercept)    numberofdrugs treatmenttreated     improvedsome   improvedmarked &#xA;  #   1.18561313       0.03272109       0.05544510      -0.09295549       0.06248684 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So the calculation of the estimate for a subject with 4 drugs, treated, with moderate improvement would be:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; exp( sum( coef(test)[ c(1,2,3,4) ]* c(1,4,1,1) ) ) &#xA; [1] 3.592999&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And the linear predictor for that case should be the sum of:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; coef(test)[c(1,2,3,4)]*c(1,4,1,1) &#xA; #    (Intercept)    numberofdrugs treatmenttreated     improvedsome &#xA; #     1.18561313       0.13088438       0.05544510      -0.09295549&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2129" LastEditorUserId="2129" LastEditorDisplayName="" LastEditDate="2011-05-22T03:17:12.347" LastActivityDate="2011-05-22T03:17:12.347" CommentCount="7" />
  <row Id="11098" PostTypeId="2" ParentId="11079" CreationDate="2011-05-21T21:47:32.110" Score="0" ViewCount="" Body="&lt;p&gt;Clearly sex is a between condition.  You've stated below in the comments that stimulus is within subjects and condition is as well.  You are only supposed to put your within conditions in the error term.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, ...&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;aov(response ~ stimulus * sex * condition + Error(subject/(stimulus * condition))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Or, if as you've done it in your example it looks like maybe you don't actually want to test condition analyzed, in which case it would be...&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;a &amp;lt;- aggregate(response ~ stimulus + sex + subject, myData, mean)&#xA;aov(response ~ stimulus * sex + Error(subject/stimulus), a)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="601" LastEditorUserId="601" LastEditorDisplayName="" LastEditDate="2011-05-22T21:37:31.003" LastActivityDate="2011-05-22T21:37:31.003" CommentCount="5" />
  <row Id="11099" PostTypeId="2" ParentId="11093" CreationDate="2011-05-21T22:38:27.183" Score="3" ViewCount="" Body="&lt;p&gt;If you have a random variable (or observed data) $X$ with mean $\mu_x$ and standard deviation $\sigma_x$, and then apply any linear transformation $$Y=a+bX$$ then you will find the mean of $Y$ is $$\mu_y = a + b \mu_x$$ and the standard deviation of $Y$ is $$\sigma_y = |b|\; \sigma_x.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So for example, as whuber says, to multiply the standard deviation by 1.5, the two possibilities are $b=1.5$ or $b=-1.5$, while $a$ can have any value. &lt;/p&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-05-21T22:38:27.183" CommentCount="1" />
  <row Id="11100" PostTypeId="2" ParentId="11079" CreationDate="2011-05-21T23:33:03.443" Score="2" ViewCount="" Body="&lt;p&gt;Assuming your design is the following:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;code&gt;sex&lt;/code&gt; is a between-subjects IV (with two levels)&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;stimulus&lt;/code&gt; is a within-subjects IV (with 3 assumed levels)&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;condition&lt;/code&gt; is a within-subjects IV (with 2 levels)&lt;/li&gt;&#xA;&lt;li&gt;all IVs are fully crossed&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Then this is what you can do to run the full analysis, or to just test for a main effect of &lt;code&gt;sex&lt;/code&gt; (generating some data first):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Nj        &amp;lt;- 10                               # number of subjects per sex&#xA;P         &amp;lt;- 2                                # number of levels for IV sex&#xA;Q         &amp;lt;- 3                                # number of levels for IV stimulus&#xA;R         &amp;lt;- 2                                # number of levels for IV condition&#xA;subject   &amp;lt;- factor(rep(1:(P*Nj), times=Q*R)) # subject id&#xA;sex       &amp;lt;- factor(rep(1:P, times=Q*R*Nj), labels=c(&quot;F&quot;, &quot;M&quot;)) # IV sex&#xA;stimulus  &amp;lt;- factor(rep(1:Q, each=P*R*Nj))    # IV stimulus&#xA;condition &amp;lt;- factor(rep(rep(1:R, each=P*Nj), times=Q), labels=c(&quot;EXP1&quot;, &quot;EXP2&quot;))&#xA;DV_t11    &amp;lt;- round(rnorm(P*Nj,  8, 2), 2)     # responses for stimulus=1 and condition=1&#xA;DV_t21    &amp;lt;- round(rnorm(P*Nj, 13, 2), 2)     # responses for stimulus=2 and condition=1&#xA;DV_t31    &amp;lt;- round(rnorm(P*Nj, 13, 2), 2)&#xA;DV_t12    &amp;lt;- round(rnorm(P*Nj, 10, 2), 2)&#xA;DV_t22    &amp;lt;- round(rnorm(P*Nj, 15, 2), 2)&#xA;DV_t32    &amp;lt;- round(rnorm(P*Nj, 15, 2), 2)&#xA;response  &amp;lt;- c(DV_t11, DV_t12, DV_t21, DV_t22, DV_t31, DV_t32)       # all responses&#xA;dfL       &amp;lt;- data.frame(subject, sex, stimulus, condition, response) # long format&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now with the data set up, you can use &lt;code&gt;aov()&lt;/code&gt;, but you won't get the $\hat{\epsilon}$ corrections for the within-effects.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; summary(aov(response ~ sex*stimulus*condition&#xA;+                        + Error(subject/(stimulus*condition)), data=dfL))&#xA;Error: subject&#xA;          Df Sum Sq Mean Sq F value Pr(&amp;gt;F)&#xA;sex        1  2.803  2.8030    0.51 0.4843   # ... snip ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can also use the &lt;code&gt;Anova()&lt;/code&gt; function from the &lt;code&gt;car&lt;/code&gt; package, which gives you the $\hat{\epsilon}$ corrections. However, it requires your data to be in wide format. You have to use multivariate notation for your model formula.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; sexW  &amp;lt;- factor(rep(1:P, Nj), labels=c(&quot;F&quot;, &quot;M&quot;))     # factor sex for wide format&#xA;&amp;gt; dfW   &amp;lt;- data.frame(sexW, DV_t11, DV_t21, DV_t31, DV_t12, DV_t22, DV_t32) # wide format&#xA;&amp;gt; # between-model in multivariate notation&#xA;&amp;gt; fit   &amp;lt;- lm(cbind(DV_t11, DV_t21, DV_t31, DV_t12, DV_t22, DV_t32) ~ sexW, data=dfW)&#xA;&amp;gt; # dataframe describing the columns of the data matrix&#xA;&amp;gt; intra &amp;lt;- expand.grid(stimulus=gl(Q, 1), condition=gl(R, 1))&#xA;&amp;gt; library(car)                    # for Anova()&#xA;&amp;gt; summary(Anova(fit, idata=intra, idesign=~stimulus*condition),&#xA;+         multivariate=FALSE, univariate=TRUE)&#xA;Univariate Type II Repeated-Measures ANOVA Assuming Sphericity&#xA;                   SS num Df Error SS den Df         F    Pr(&amp;gt;F)    &#xA;(Intercept)   17934.1      1   98.930     18 3263.0403 &amp;lt; 2.2e-16 ***&#xA;sexW              2.8      1   98.930     18    0.5100 0.4843021  # ... snip ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Using the &lt;code&gt;ez&lt;/code&gt; package and the command suggested by @Mike Lawrence gives the same result:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; library(ez)              # for ezANOVA()&#xA;&amp;gt; ezANOVA(data=dfL, wid=.(subject), dv=.(response),&#xA;+         within=.(stimulus, condition), between=.(sex), observed=.(sex))&#xA;$ANOVA&#xA;     Effect DFn DFd          F            p p&amp;lt;.05         ges&#xA;2       sex   1  18  0.5099891 4.843021e-01       0.004660043      # ... snip ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Finally, if the main effect for &lt;code&gt;sex&lt;/code&gt; is really all you're interested in, it's equivalent to just average for each person across all the conditions created by the combinations of &lt;code&gt;stimulus&lt;/code&gt; and &lt;code&gt;condition&lt;/code&gt;, and then run a between-subjects ANOVA for the aggregated data.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# average per subject across all repeated measures&#xA;&amp;gt; mDf &amp;lt;- aggregate(response ~ subject + sex, data=dfL, FUN=mean)&#xA;&amp;gt; summary(aov(response ~ sex, data=mDf))     # ANOVA with just the between-effect&#xA;            Df  Sum Sq Mean Sq F value Pr(&amp;gt;F)&#xA;sex          1  0.4672 0.46716    0.51 0.4843&#xA;Residuals   18 16.4884 0.91602&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1909" LastEditorUserId="1909" LastEditorDisplayName="" LastEditDate="2011-05-21T23:57:52.530" LastActivityDate="2011-05-21T23:57:52.530" CommentCount="2" />
  <row Id="11101" PostTypeId="1" CreationDate="2011-05-22T02:36:46.997" Score="2" ViewCount="73" Body="&lt;p&gt;I was trying to do a test of reliability for my survey items. In addition to Cronbach's alpha I'm looking at communalities. My criteria is that survey items with communality below 0.4 will be dropped. But when I looked at my communality table, I saw that some items had .99 for communality. Is this problematic? What should I do with these? &lt;/p&gt;&#xA;" OwnerUserId="4702" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-23T01:43:04.013" LastActivityDate="2011-05-23T15:54:10.807" Title="Implications of communalities close to 1.00 for reliability analysis and survey design" Tags="&lt;factor-analysis&gt;&lt;reliability&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="11102" PostTypeId="1" AcceptedAnswerId="11103" CreationDate="2011-05-22T03:56:56.093" Score="4" ViewCount="84" Body="&lt;p&gt;I'm having trouble performing factor analysis on my dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I perform the factor analysis in SPSS (default settings), it works fine.  Problem is, I need to do it programmatically (in Python).  When I try using Python (MDP library) to do factor analysis on the same dataset, I get this error:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;The covariance matrix of the data is singular. Redundant dimensions need to be removed&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Upon looking into the MDP documentation, it says &quot;...returns the Maximum A Posteriori estimate of the latent variables.&quot;  Being a factor analysis newbie, I wasn't too clear on what this meant, but I tried changing the default extraction method in SPSS from &quot;principal components&quot; to &quot;maximum likelihood&quot;.  Then, in SPSS, I get the error:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;This matrix is not positive definite.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are these two errors the same thing?  Regardless, what can I do to fix my dataset so that the covariance matrix is not singular?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;edit:&lt;/strong&gt; OK, so I was trying to keep things simplified, but perhaps its better to just explain everything from the start.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a series of documents.  Yes, I'm only using 9 documents as a simple test case, but my final objective will be to use it on a much larger corpus.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've built a term-document matrix, performed tf-idf, and did SVD-- mostly with the help of blog.josephwilk.net/.../latent-semantic-analysis-in-python.html&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I have a reconstructed matrix, and I want to sort the documents into categories.  So, I tried using factor analysis.  In fact, it seems to work-- when I put it in SPSS, the factor loadings indicate that the documents are grouped the way I thought they should be, and the loading are higher than if I hadn't performed SVD. (Although I think technically, SPSS is doing PCA even though its under the 'Factor Analysis' heading).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried using MDP's PCANode, but that doesn't seem to give me anything close to what I want.  Strangely, if I transpose my matrix, the factor analysis does work (it will group the terms, instead of the documents).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hopefully this all makes a little more sense now...&lt;/p&gt;&#xA;" OwnerUserId="1977" LastEditorUserId="1977" LastEditorDisplayName="" LastEditDate="2011-05-23T00:13:58.417" LastActivityDate="2011-05-23T00:13:58.417" Title="Factor analysis problem -- singular covariance matrix?" Tags="&lt;spss&gt;&lt;factor-analysis&gt;&lt;python&gt;" AnswerCount="1" CommentCount="7" />
  <row Id="11103" PostTypeId="2" ParentId="11102" CreationDate="2011-05-22T04:08:51.723" Score="3" ViewCount="" Body="&lt;p&gt;Yes, the two errors amount to the same thing. They're telling you (roughly) that two or more of your manifest variables are linearly dependent (like $y_1 = ay_2 + b$ for scalars $a, b$). These two variables (dimensions) would be &quot;redundant&quot;, meaning that the sample covariance matrix is not invertible (ie is singular) and therefore not positive definite either.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for what you ought to do about it, that depends. First I would try to find out which variables are giving you the trouble; a scatterplot matrix might be enough to tell you that. Then you can decide what to do from there - most likely dropping some redundant variables.&lt;/p&gt;&#xA;" OwnerUserId="26" LastActivityDate="2011-05-22T04:08:51.723" CommentCount="8" />
  <row Id="11104" PostTypeId="2" ParentId="1995" CreationDate="2011-05-22T04:40:26.447" Score="3" ViewCount="" Body="&lt;p&gt;Multi-level modelling is appropriate, as the name suggests, when your data have influences occurring at different levels (individual, over time, over domains, etc).  Single level modeling assumes everything is occurring at the lowest level.  Another thing that a multi-level model does is to introduce correlations among nested units.  So level-1 units within the same level-2 unit will be correlated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In some sense you can think of multi-level modelling as finding the middle ground between the &quot;individualist fallacy&quot; and the &quot;ecological fallacy&quot;.  Individualist fallacy is when &quot;community effects&quot; are ignored such as the compatibility of a teacher's style with a student's learning style, for example (the effect is assumed to come from the individual alone, so just do regression at level 1).  whereas &quot;ecological fallacy&quot; is the opposite, and would be like supposing the best teacher had the students with the best grades (and so that the level-1 is not needed, just do regression entirely at level 2).  In most settings, neither is appropriate (the student-teacher is a &quot;classical&quot; example).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that in the school example, there was a &quot;natural&quot; clustering or structure in the data.  But this is not an essential feature of multi-level/hierachical modeling.  However, the natural clustering makes the mathematics and computations easier.  The key ingredient is the prior information which says that there are processes happening at different levels.  In fact you can devise clustering algorithms by imposing a multi-level structure on your data with uncertainty about which unit is in which higher level.  So you have $y_{ij}$ with the subscript $j$ being unknown.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-05-22T04:40:26.447" />
  <row Id="11105" PostTypeId="1" CreationDate="2011-05-22T05:11:51.350" Score="0" ViewCount="50" Body="&lt;p&gt;I am a newbie in stat. I am completing my thesis in &lt;a href=&quot;http://en.wikipedia.org/wiki/Evolutionary_algorithm&quot; rel=&quot;nofollow&quot;&gt;Evolutionary algorithm&lt;/a&gt;. I have to generate some random numbers from &lt;a href=&quot;http://en.wikipedia.org/wiki/Laplace_distribution&quot; rel=&quot;nofollow&quot;&gt;Laplace distribution&lt;/a&gt;. How can I do this using matlab?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An easy and simple explanation would be appreciated. Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="4319" LastActivityDate="2011-05-22T06:22:48.647" Title="Random number generation using laplace distribution" Tags="&lt;distributions&gt;&lt;matlab&gt;&lt;random-generation&gt;" AnswerCount="0" CommentCount="0" FavoriteCount="0" />
  <row Id="11106" PostTypeId="2" ParentId="11088" CreationDate="2011-05-22T06:22:48.647" Score="2" ViewCount="" Body="&lt;p&gt;You can use the same approach that was described in response to your question about generating  random numbers from a t-distribution. First generate uniformly distributed random numbers from (0,1) and then apply the inverse cumulative distribution function of the Laplace distribution, which is given in the Wikipedia article you linked to.&lt;/p&gt;&#xA;" OwnerUserId="3835" LastActivityDate="2011-05-22T06:22:48.647" CommentCount="2" />
  <row Id="11107" PostTypeId="1" CreationDate="2011-05-22T07:14:51.037" Score="1" ViewCount="151" Body="&lt;p&gt;I need to do a logistic regression using R on my data. My response variable (&lt;code&gt;y&lt;/code&gt;) is survival at weaning (&lt;code&gt;surv=0&lt;/code&gt;; did not &lt;code&gt;surv=1&lt;/code&gt;) and I have several independent variables which are binary and categoricals in nature.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am following some examples on this website &lt;a href=&quot;http://www.ats.ucla.edu/stat/r/dae/logit.htm&quot; rel=&quot;nofollow&quot;&gt;http://www.ats.ucla.edu/stat/r/dae/logit.htm&lt;/a&gt; and trying to run some models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Running the model: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; mysurv2 &amp;lt;- glm(surv~as.factor(PTEM)+as.factor(pshiv)+as.factor(presp)+as.factor(pmtone),family=binomial(link=&quot;logit&quot;),data=ap)&#xA;&amp;gt; summary(mysurv2)&#xA;&#xA;Call:&#xA;glm(formula = surv ~ as.factor(PTEM) + as.factor(pshiv) + as.factor(presp) + &#xA;    as.factor(pmtone), family = binomial(link = &quot;logit&quot;), data = ap)&#xA;&#xA;Deviance Residuals: &#xA;    Min       1Q   Median       3Q      Max  &#xA;-2.2837  -0.5121  -0.5121  -0.5058   2.0590  &#xA;&#xA;Coefficients:&#xA;                   Estimate Std. Error z value Pr(&amp;gt;|z|)    &#xA;(Intercept)        -0.01135    0.23613  -0.048  0.96166    &#xA;as.factor(PTEM)2   -0.74642    0.24482  -3.049  0.00230 ** &#xA;as.factor(PTEM)3   -1.95401    0.23259  -8.401  &amp;lt; 2e-16 ***&#xA;as.factor(pshiv)2  -0.02638    0.06784  -0.389  0.69738    &#xA;as.factor(presp)2   0.74549    0.10532   7.079 1.46e-12 ***&#xA;as.factor(presp)3   0.66793    0.66540   1.004  0.31547    &#xA;as.factor(pmtone)2  0.54699    0.09678   5.652 1.58e-08 ***&#xA;as.factor(pmtone)3  1.82337    0.75409   2.418  0.01561 *  &#xA;---&#xA;Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 &#xA;&#xA;(Dispersion parameter for binomial family taken to be 1)&#xA;&#xA;    Null deviance: 7892.6  on 8791  degrees of freedom&#xA;Residual deviance: 7252.8  on 8784  degrees of freedom&#xA;  (341 observations deleted due to missingness)&#xA;AIC: 7268.8&#xA;&#xA;Number of Fisher Scoring iterations: 4&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Adding the &lt;code&gt;na.action=na.pass&lt;/code&gt; at the end of the model gave me an error message. I thought that this would take care NA's in my independent variables.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; mysurv &amp;lt;- glm(surv~as.factor(PTEM)+as.factor(pshiv)+as.factor(presp)+as.factor(pmtone),family=binomial(link=&quot;logit&quot;),na.action=na.pass,data=ap)&#xA;Error: NA/NaN/Inf in foreign function call (arg 1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Since this is my first time to venture into logistic regression, I am wondering whether there is any package in R that would be more suitable?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am also tryng to understand the regression coefficients. The independent variables used in the model are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;rectal temperature: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;(PTEM)1&lt;/code&gt; = newborns with rectal temp. below 35.4 0C&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;(PTEM)2&lt;/code&gt; = newborns with rectal temp. between 35.4 to 36.9 0C&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;(PTEM)3&lt;/code&gt; = newborns with rectal temp. above 37.0 0C&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;shivering:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;(pshiv)1&lt;/code&gt; = newborns that were not shivering&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;(pshiv)2&lt;/code&gt; = newborns that were shivering&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;respiration:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;(presp)1&lt;/code&gt; = newborns with normal respiration&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;(presp)2&lt;/code&gt; = newborns with slight respiration problem&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;(presp)3&lt;/code&gt; = newborns with poor respiration&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;muscle tone:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;(pmtone)1&lt;/code&gt; = newborns with normal muscle tone&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;(pmtone)2&lt;/code&gt; = newborns with moderate muscle tone&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;(pmtone)1&lt;/code&gt; = newborns with poor muscle tone&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Looking at the coefficients, I got the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;                   Estimate Std. Error z value Pr(&amp;gt;|z|)    &#xA;(Intercept)        -0.01135    0.23613  -0.048  0.96166    &#xA;as.factor(PTEM)2   -0.74642    0.24482  -3.049  0.00230 ** &#xA;as.factor(PTEM)3   -1.95401    0.23259  -8.401  &amp;lt; 2e-16 ***&#xA;as.factor(pshiv)2  -0.02638    0.06784  -0.389  0.69738    &#xA;as.factor(presp)2   0.74549    0.10532   7.079 1.46e-12 ***&#xA;as.factor(presp)3   0.66793    0.66540   1.004  0.31547    &#xA;as.factor(pmtone)2  0.54699    0.09678   5.652 1.58e-08 ***&#xA;as.factor(pmtone)3  1.82337    0.75409   2.418  0.01561 *  &#xA;---&#xA;Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In my other analysis, I found that newborns: &#xA;a) with higher rectal temperature &#xA;b) do not shiver&#xA;c) good respiration and&#xA;d) good muscle tone at birth were more likely to survive.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am a bit confused with the coefficients I am getting above. I am wondering whether whether I am not interpreting the results correctly or is it something else?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help would be highly appreciated.&lt;/p&gt;&#xA;" OwnerUserId="4263" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-22T10:37:25.620" LastActivityDate="2011-05-24T15:46:47.447" Title="Doing logistic regression using R" Tags="&lt;r&gt;&lt;logistic-regression&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="11108" PostTypeId="1" CreationDate="2011-05-21T22:15:36.050" Score="4" ViewCount="85" Body="&lt;p&gt;Take a look at this photo:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.stack.imgur.com/J93c2.png&quot; alt=&quot;data&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It depicts a &lt;a href=&quot;http://en.wikipedia.org/wiki/Box_plot&quot; rel=&quot;nofollow&quot;&gt;box plot&lt;/a&gt; of series of identical runs for successive &lt;strong&gt;i&lt;/strong&gt; values.  (AFAIK it's the standard Min/Max and 1rst, 2nd, 3rd quartiles.) So the x-axis of 1 represents 1000 runs where i=1; and the second plot shows 1000 runs where i=2; and so on. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's easy to eye-ball and see that there's a split between the i=1,2 and i=3-19.  The values for i=2 are on 'average' larger, by a little bit.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I aim to do is given the input that produced this graph, programmatically find that split (between 2 and 3) where there's the sudden consistent change.  (Step 1)  It was also be awesome if there was some sort of confidence score to go along with it - just for user feedback.  The change may be up or down, but I know that on both sides of the split the values will be consistent (just like for i &gt; 2 the box plots stay pretty even and don't return to i&amp;lt;2 values).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, after that, I want to take a measurement for an unknown i and decide which 'side' of the split it falls on.  Now I know I could never know that answer conclusively from a single measurement, so I plan to take several (5? 50? 100?) measurements for this unknown-but-unchanging i value.  Then using those measurements know which side of the split the i falls on (Step 2).  Again, it would be awesome if there was a confidence value associated with this decision.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm working in python so if there's a library awesome, but I'm cool with implementing an algorithm/equation myself.  What's the techniques/equations/papers I should read up on to learn how to do this?&lt;/p&gt;&#xA;" OwnerUserId="4703" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-05-22T15:40:30.043" LastActivityDate="2011-05-22T19:26:42.680" Title="Automatically detecting sudden change of mean" Tags="&lt;time-series&gt;&lt;python&gt;&lt;change-point&gt;" AnswerCount="1" CommentCount="4" FavoriteCount="1" />
  <row Id="11109" PostTypeId="1" AcceptedAnswerId="11110" CreationDate="2011-05-22T10:37:08.303" Score="5" ViewCount="79" Body="&lt;p&gt;If you have variable which perfecly separates zeroes and ones in target variable, R will yield the following &quot;perfect or quasi perfect separation&quot; warning message:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Warning message:&#xA;glm.fit: fitted probabilities numerically 0 or 1 occurred &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;We still get the model but the coefficient estimates are inflated. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do you deal with this in practice?&lt;/p&gt;&#xA;" OwnerUserId="333" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-22T10:38:38.443" LastActivityDate="2011-05-23T00:00:20.187" Title="How to deal with perfect separation in logistic regression?" Tags="&lt;r&gt;&lt;logistic-regression&gt;" AnswerCount="2" />
  <row Id="11110" PostTypeId="2" ParentId="11109" CreationDate="2011-05-22T11:14:19.717" Score="3" ViewCount="" Body="&lt;p&gt;A solution to this is to utilize a form of penalized regression. In fact, this is the original reason some of the penalized regression forms were developed (although they turned out to have other interesting properties.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;install and load package glmnet in R and you're mostly ready to go. One of the less user-friendly aspects of glmnet is that you can only feed it matrices, not formulas as we're used to. However, you can look at model.matrix and the like to construct this matrix from a data.frame and a formula...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, when you expect that this perfect separation is not just a byproduct of your sample, but could be true in the population, you specificly &lt;em&gt;don't&lt;/em&gt; want to handle this: use this separating variable simply as the sole predictor for your outcome, not employing a model of any kind.&lt;/p&gt;&#xA;" OwnerUserId="4257" LastActivityDate="2011-05-22T11:14:19.717" />
  <row Id="11112" PostTypeId="1" CreationDate="2011-05-22T14:03:41.867" Score="3" ViewCount="48" Body="&lt;p&gt;I have some data acquired by an acoustic sensor with 1 Hz sampling rate. Due to some inevitable issues, I have some noise in my signal, saying 10% pollution.&#xA;I'm looking for a reliable method for replacing the outliers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to find a suitable approach I manipulated a clean record such that it contains 9% spurious data. I replace outliers with some different methods like Kalman Predictor, Linear Time Series Modeling, and the Local Mean.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, I want to compare them together with. Can you suggest criteria to show which method restores fluctuations better if compared to the clean original signal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the figure below shows the cross-correlation between the original signal and the restored ones, how can I interpret the large negative peak in lag19?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.stack.imgur.com/PeHaK.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furthermore, is it correct if I say: as the ACF of the signal resored by wavelet-LTS tracks that of original signal, this method could mimic the original signal better than the others?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.stack.imgur.com/k96KC.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="4286" LastEditorUserId="4286" LastEditorDisplayName="" LastEditDate="2011-05-26T02:51:52.030" LastActivityDate="2011-05-26T02:51:52.030" Title="What is the best way to compare fluctuations of two signals?" Tags="&lt;correlation&gt;&lt;autocorrelation&gt;&lt;signal-processing&gt;&lt;variance-covariance&gt;" CommentCount="4" FavoriteCount="0" />
  <row Id="11113" PostTypeId="1" CreationDate="2011-05-22T14:20:49.473" Score="1" ViewCount="79" Body="&lt;p&gt;I am using ANOVA with repeated measures to test significance between males and females results of an experiment during which participants had to evaluate 7 stimuli in 2 conditions (EXP1 and EXP2).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is that even if from results it is clear that there are significant differences between males and females, I don´t get significance in the ANOVA results.&#xA;Definitively there is an error, because results cannot be non-significant. Indeed looking at the means for each stimulus, it is possible to notice that males gave always higher evaluations than females.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To prove this I discarded for a moment the effect of the repeated measures, and I performed an ANOVA separately on both the two conditions (EXP1 and EXP2) during which the evaluations were given.&#xA;What I get is significant differences between males and female, in both EXP1 and EXP2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, why when I perform the ANOVA with repeated measures I don´t get the same behavior?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The structure of my table is the following: subject, stimulus, condition, sex, response. The design is the following:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;sex is a between-subjects factor (with two levels)&lt;/li&gt;&#xA;&lt;li&gt;stimulus is a within-subjects factor (with 3 assumed levels)&lt;/li&gt;&#xA;&lt;li&gt;condition is a within-subjects factor (with 2 levels)&lt;/li&gt;&#xA;&lt;li&gt;all factors are fully crossed&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;subject  stimulus condition sex        response&#xA;subject1    gravel  EXP1    M      59.8060&#xA;subject2    gravel  EXP1    M      49.9880&#xA;subject3    gravel  EXP1    M      73.7420&#xA;subject4    gravel  EXP1    M      45.5190&#xA;subject5    gravel  EXP1    M      51.6770&#xA;subject6    gravel  EXP1    M      42.1760&#xA;subject7    gravel  EXP1    M      56.1110&#xA;subject8    gravel  EXP1    M      54.9500&#xA;subject9    gravel  EXP1    M      62.6920&#xA;subject10   gravel  EXP1    M      50.7270&#xA;subject1    gravel  EXP2    M      70.9270&#xA;subject2    gravel  EXP2    M      61.3200&#xA;subject3    gravel  EXP2    M      70.2930&#xA;subject4    gravel  EXP2    M      49.9880&#xA;subject5    gravel  EXP2    M      69.1670&#xA;subject6    gravel  EXP2    M      62.2700&#xA;subject7    gravel  EXP2    M      70.9270&#xA;subject8    gravel  EXP2    M      63.6770&#xA;subject9    gravel  EXP2    M      72.4400&#xA;subject10   gravel  EXP2    M      58.8560&#xA;subject11   gravel  EXP1    F      46.5750&#xA;subject12   gravel  EXP1    F      58.1520&#xA;subject13   gravel  EXP1    F      57.4490&#xA;subject14   gravel  EXP1    F      59.8770&#xA;subject15   gravel  EXP1    F      55.5480&#xA;subject16   gravel  EXP1    F      46.2230&#xA;subject17   gravel  EXP1    F      63.3260&#xA;subject18   gravel  EXP1    F      60.6860&#xA;subject19   gravel  EXP1    F      59.4900&#xA;subject20   gravel  EXP1    F      52.6630&#xA;subject11   gravel  EXP2    F      55.7240&#xA;subject12   gravel  EXP2    F      66.4220&#xA;subject13   gravel  EXP2    F      65.9300&#xA;subject14   gravel  EXP2    F      61.8120&#xA;subject15   gravel  EXP2    F      62.5160&#xA;subject16   gravel  EXP2    F      65.5780&#xA;subject17   gravel  EXP2    F      59.5600&#xA;subject18   gravel  EXP2    F      63.8180&#xA;subject19   gravel  EXP2    F      61.4250&#xA;.....&#xA;.....&#xA;.....&#xA;.....&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As you can notice each subject repeated the evaluation in 2 conditions (EXP1 and EXP2).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I am interested in is to know if there are significant differences between &#xA;the evaluations of the males and the females (both at global level and for each stimulus).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is the command I used to perform the ANOVA with repeated measures:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;aov1 = aov(response ~ sex*stimulus*condition + Error(subject/(stimulus*condition)), data=scrd)&#xA;summary(aov1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Doing so I don´t get significance for the differences between males and females.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead if I perform the ANOVA on the two subtables of EXP 1 and 2 I get significant differences. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;table_EXP1 &amp;lt;- subset(scrd, condition == &quot;EXP1&quot;)&#xA;table_EXP2 &amp;lt;- subset(scrd, condition == &quot;EXP2&quot;)&#xA;&#xA;&#xA;fit_table_EXP1 &amp;lt;- lm(response ~ stimulus*sex, data=table_EXP1) &#xA;anova(fit_table_EXP1)&#xA;&#xA;&#xA;fit_table_EXP2 &amp;lt;- lm(response ~ stimulus*sex, data=table_EXP2) &#xA;anova(fit_table_EXP2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can this be possible? Is it a contradiction?&lt;/p&gt;&#xA;" OwnerUserId="4701" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-22T14:51:56.993" LastActivityDate="2011-05-22T14:51:56.993" Title="Wrong results using ANOVA with repeated measures" Tags="&lt;r&gt;&lt;anova&gt;&lt;repeated-measures&gt;" CommentCount="14" />
  <row Id="11114" PostTypeId="2" ParentId="11009" CreationDate="2011-05-22T14:26:28.640" Score="0" ViewCount="" Body="&lt;p&gt;this is implicit in many of answers others have given but the simple point is that models w/ a product term but w/ &amp;amp; w/o the moderator &amp;amp; predictor are just different models. Figure out &lt;em&gt;what each means&lt;/em&gt; given the process you are modeling and whether a model w/o the moderator &amp;amp; predictor makes more sense given your theory or hypothesis. The observation that the product term is significant but only when moderator &amp;amp; predictor are not included doesn't tell you anything (except maybe that you are fishing around for &quot;significance&quot;) w/o a cogent explanation of &lt;em&gt;why&lt;/em&gt; it makes sense to leave them out.&lt;/p&gt;&#xA;" OwnerUserId="2347" LastActivityDate="2011-05-22T14:26:28.640" />
  <row Id="11115" PostTypeId="1" CreationDate="2011-05-22T14:33:20.660" Score="1" ViewCount="42" Body="&lt;p&gt;Does anybody know how to plot all AIC values for different size models, when using the command &lt;code&gt;regsubsets&lt;/code&gt; from the package &lt;code&gt;leaps&lt;/code&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assume you have the following variables:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;treatment &amp;lt;- factor(rep(c(1, 2), c(43, 41)), levels = c(1, 2),labels = c(&quot;placebo&quot;, &quot;treated&quot;))&#xA;&#xA;improved &amp;lt;- factor(rep(c(1, 2, 3, 1, 2, 3), c(29, 7, 7, 13, 7, 21)),levels = c(1, 2, 3),labels = c(&quot;none&quot;, &quot;some&quot;, &quot;marked&quot;))&#xA;&#xA;numberofdrugs&amp;lt;-rpois(84, 5)+1&#xA;&#xA;healthvalue&amp;lt;-rpois(84,5)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And now you want to select variables. Then you can use the following commands&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(leaps)&#xA;require(faraway)&#xA;&#xA;d&amp;lt;-regsubsets(healthvalue~numberofdrugs*improved*treatment,x, nvmax=10)&#xA;rs&amp;lt;-summary(d)&#xA;plot(rs$bic, xlab=&quot;Parameter&quot;, ylab=&quot;BIC&quot;) #where is AIC (bic works)?&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It works when using &lt;code&gt;rs$bic&lt;/code&gt;, but why isn't there a way to use &lt;code&gt;rs$aic&lt;/code&gt;? When looking at the help &lt;code&gt;?regsubsets&lt;/code&gt;, it seems to be not available. Do I understand the help wrong? And if I don't understand it wrong, how can I plot the above code when using AIC?&lt;/p&gt;&#xA;" OwnerUserId="4496" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-22T17:14:58.163" LastActivityDate="2011-05-24T08:08:48.100" Title="How to plot AIC values when using the leaps package?" Tags="&lt;r&gt;&lt;aic&gt;&lt;stepwise-regression&gt;&lt;validation&gt;&lt;bic&gt;" AnswerCount="1" />
  <row Id="11116" PostTypeId="2" ParentId="11107" CreationDate="2011-05-22T14:44:36.060" Score="1" ViewCount="" Body="&lt;p&gt;The lrm function in the R rms package is devoted to binary and ordinal logistic regression, and my help, once you understand the rms documentation.  Detailed case studies using rms may be found in course notes at &lt;a href=&quot;http://biostat.mc.vanderbilt.edu/rms&quot; rel=&quot;nofollow&quot;&gt;http://biostat.mc.vanderbilt.edu/rms&lt;/a&gt;.  However there are more important issues.  Categorizing continuous variables leads to erroneous conclusions, underfitting, and residual confounding (here, temperature and perhaps respiratory rate).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One R coding convention suggestion: Set up your data frame the way you want to treat the variables in later modeling steps.  For example if variables are really factors, give them descriptive levels and make them factors once and for all.  The rms package almost requires this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you do use rms you will find its summary, Predict, plot, contrast, and nomogram functions handy for interpreting the model, once the model properly uses continuous variables.&lt;/p&gt;&#xA;" OwnerUserId="4253" LastActivityDate="2011-05-22T14:44:36.060" CommentCount="4" />
  <row Id="11117" PostTypeId="2" ParentId="11115" CreationDate="2011-05-22T15:04:26.447" Score="4" ViewCount="" Body="&lt;p&gt;The relevant excerpt from the &lt;code&gt;regsubsets&lt;/code&gt; help pages is the following:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Since this function returns separate&#xA;  best models of all sizes up to nvmax&#xA;  and since different model selection&#xA;  criteria such as AIC, BIC, CIC, DIC,&#xA;  ... differ only in how models of&#xA;  different sizes are compared, the&#xA;  results do not depend on the choice of&#xA;  cost-complexity tradeoff.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This means that it does not matter which criteria you use, the end result will be the same. Hence the graph with AIC instead of BIC will be similar, i.e. will convey the same information.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-05-22T15:04:26.447" CommentCount="2" />
  <row Id="11118" PostTypeId="1" AcceptedAnswerId="11119" CreationDate="2011-05-22T16:37:29.693" Score="5" ViewCount="69" Body="&lt;p&gt;Given a data frame in R is there a way to export it in &lt;em&gt;R syntax&lt;/em&gt; such that executing this code would re-create the data frame? I would find this useful to store results in R files along with calculations without depending on external files.&lt;/p&gt;&#xA;" OwnerUserId="4195" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-06-03T22:30:59.033" LastActivityDate="2011-06-03T22:30:59.033" Title="How to export data in R syntax?" Tags="&lt;r&gt;&lt;dataframe&gt;" AnswerCount="1" />
  <row Id="11119" PostTypeId="2" ParentId="11118" CreationDate="2011-05-22T17:06:59.327" Score="7" ViewCount="" Body="&lt;p&gt;You can use &lt;code&gt;dput()&lt;/code&gt; to get a &lt;code&gt;structure()&lt;/code&gt; that can be used later.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; #Build the original data frame&#xA;&amp;gt; x &amp;lt;- seq(1, 10, 1)  &#xA;&amp;gt; y &amp;lt;- seq(10, 100, 10)  &#xA;&amp;gt; df &amp;lt;- data.frame(x=x, y=y)   &#xA;&amp;gt; df   &#xA;    x   y&#xA;1   1  10&#xA;2   2  20&#xA;3   3  30&#xA;4   4  40&#xA;5   5  50&#xA;6   6  60&#xA;7   7  70&#xA;8   8  80&#xA;9   9  90&#xA;10 10 100&#xA;&#xA;&amp;gt; #Use the dput() statement to print out the structure of df&#xA;&amp;gt; dput(df)   &#xA;structure(list(x = c(1, 2, 3, 4, 5, 6, 7, 8, 9, 10), y = c(10, &#xA;20, 30, 40, 50, 60, 70, 80, 90, 100)), .Names = c(&quot;x&quot;, &quot;y&quot;), row.names = c(NA, &#xA;-10L), class = &quot;data.frame&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The above &lt;code&gt;structure&lt;/code&gt; statement is the output of &lt;code&gt;dput(df)&lt;/code&gt;.   If you copy/paste that into your R text file, you can use it later.   Here's how.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; #Build a new dataframe from the structure() statement&#xA;&amp;gt; newdf &amp;lt;- structure(list(x = c(1, 2, 3, 4, 5, 6, 7, 8, 9, 10), y = c(10, &#xA;20, 30, 40, 50, 60, 70, 80, 90, 100)), .Names = c(&quot;x&quot;, &quot;y&quot;), row.names = c(NA, &#xA;-10L), class = &quot;data.frame&quot;)&#xA;&amp;gt; newdf&#xA;    x   y&#xA;1   1  10&#xA;2   2  20&#xA;3   3  30&#xA;4   4  40&#xA;5   5  50&#xA;6   6  60&#xA;7   7  70&#xA;8   8  80&#xA;9   9  90&#xA;10 10 100&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2775" LastActivityDate="2011-05-22T17:06:59.327" />
  <row Id="11120" PostTypeId="1" AcceptedAnswerId="11128" CreationDate="2011-05-22T17:21:46.187" Score="4" ViewCount="58" Body="&lt;p&gt;Hello,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have 5 emerging market foreign exchange total return series, for which I am forecasting single period future returns (1 year). I would like to construct a Markowitz mean variance optimized portfolio of the 5 series, using historical variances and covariances (1) and my own forecast expected returns. Does R have an (easy) way/library to do this? In addition how would I go about calculating (1) is there a built in function?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For interest sake my currencies are USDTRY, USDZAR, USDRUB, USDHUF and USDPLN.&lt;/p&gt;&#xA;" OwnerUserId="4705" LastActivityDate="2011-05-22T18:59:54.430" Title="Markowitz portfolio mean variance optimization in R" Tags="&lt;r&gt;" AnswerCount="1" FavoriteCount="0" />
  <row Id="11121" PostTypeId="2" ParentId="11108" CreationDate="2011-05-22T18:00:54.037" Score="1" ViewCount="" Body="&lt;p&gt;If I understand you you correctly, you might need to learn about multiple comparisons:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Multiple_comparisons&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Multiple_comparisons&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The choice of a particular procedure is a different question, i.e., Shauffe's vs. Turkey vs. Bonferonni.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At least in this framework, there is a clear and straightforward way to have hypothesis testing as well as confidence interval estimation.&lt;/p&gt;&#xA;" OwnerUserId="4617" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-05-22T19:26:42.680" LastActivityDate="2011-05-22T19:26:42.680" CommentCount="1" />
  <row Id="11122" PostTypeId="2" ParentId="11107" CreationDate="2011-05-22T18:05:50.277" Score="3" ViewCount="" Body="&lt;p&gt;I think you're confused because you defined survival at weaning as surv=0 rather than surv=1. In your model, &lt;strong&gt;negative&lt;/strong&gt; coefficients indicate &lt;strong&gt;high&lt;/strong&gt; odds of survival (low odds of surv=1).&lt;/p&gt;&#xA;" OwnerUserId="3874" LastEditorUserId="3874" LastEditorDisplayName="" LastEditDate="2011-05-24T15:46:47.447" LastActivityDate="2011-05-24T15:46:47.447" CommentCount="1" />
  <row Id="11123" PostTypeId="1" CreationDate="2011-05-22T18:09:43.557" Score="1" ViewCount="30" Body="&lt;p&gt;Hi all,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've been using R to run GLMs with the logit link to compare clutch sizes (a binomial data set) and proportional data among several years. I now need to compare 2 groups of years (good years and poor years) between 2 sites for the same data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not interested in any interactions between year-type (good or poor) and site, but really am just interested in comparing good vs. good and poor vs. poor between sites. If my data were continuous, I'd use a 2-way ANOVA followed up with t-tests for each comparisons, using the Holm–Bonferroni method for these multiple (2) comparisons. However, I'm not entirely sure how to approach these comparisons with binomial data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To start, I'm considering clutch sizes (proportional data will come next). I'm thinking that this code (below) is akin to the 2-way ANOVA for binomial data?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model&amp;lt;-glm(clutchsize~Site+Year_type, binomial)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then, since I got a significant result here, I can go forward and make direct comparisons between the good years and the poor 'independently' of each other (using the Holm–Bonferroni method). But how can I make those comparisons? Can I use the &lt;code&gt;glm&lt;/code&gt; model for that as well, like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model&amp;lt;-glm(clsz_gd~Site_gd,binomial)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where &lt;code&gt;clsz_gd&lt;/code&gt; is binomial and &lt;code&gt;Site_gd&lt;/code&gt; is a factor with only 2 levels? I presume I can't use t-tests!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this the way to approach these data?&#xA;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="4238" LastActivityDate="2011-05-22T18:09:43.557" Title="GLM for comparing 2 populations with binomial data?" Tags="&lt;binomial&gt;&lt;glm&gt;" CommentCount="1" />
  <row Id="11124" PostTypeId="1" CreationDate="2011-05-22T18:20:22.577" Score="0" ViewCount="9" Body="&lt;p&gt;Hi. I have a question on Time-Weighted Rate of Return (TWRR) and then a question on the links between MWRR and TWRR,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An investor invested £100 in a fund on Jan 1st 1998 and another £100 on Jan 1st 1999. The following gives the price of a unit in the fund on Jan 1st:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Year   Unit Price&#xA;1998   100&#xA;1998   125&#xA;2000   130&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Calculate TWRR and MWRR for the period 1998-2000 (1st Jan).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am aware of the definition, but when I looked at the solution to this question, it was rather bizarre. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For MWRR they got:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;100 + 100(1+i)^-1 = 234(1+i)^-2 so i=10.93%&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For TWRR they got:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;125/100 x 234/225 = (1+i)^2 so i=14.02%&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The ‘234’ comes from a ratio I think. They had these calculations: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Year   Unit Price  Invest&#xA;1998   100            100          1                   100&#xA;1998   125            100          1+ 100/125 = 1.8    225&#xA;2000   130                         1.8                 234&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Can you please explain this to me? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, the last part was a general proof. It asked ‘when is MWRR &gt; TWRR’. Therefore, I am guessing we need to find a general interest rate (It) to show this is true. The questions hint is: Assume you invest 1 unit at time 0 and 1 unit at time 1. What’s the accumulation at time 2? This is guess but is the accumulation 1(1+I1)(1+I2) + 1(1+I2). But what’s this got to do with MWRR/TWRR? And how does it help? I am not sure how to do this and hope someone can help me! :)&lt;/p&gt;&#xA;" OwnerUserId="4645" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-05-22T20:52:35.087" LastActivityDate="2011-05-22T20:52:35.087" ClosedDate="2011-05-22T20:54:01.277" Title="Money-Weighted and Time-Weighted Rate of Return" Tags="&lt;homework&gt;" CommentCount="1" />
  <row Id="11125" PostTypeId="1" CreationDate="2011-05-22T18:25:26.570" Score="0" ViewCount="11" Body="&lt;p&gt;Can someone help me with this please:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose you have a decreasing continuous perpetuity p(t) = alpha [e^ -beta (t)] (an asset) and a fixed liability at time n. Can you immunise this? How?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I’m not sure how to do this as we have only been taught the discrete case. Can you provide an example please? &lt;/p&gt;&#xA;" OwnerUserId="4645" LastActivityDate="2011-05-22T18:25:26.570" ClosedDate="2011-05-22T20:56:42.787" Title="Reddington Theory (Immunisation of Assets and Liabilities etc)" Tags="&lt;homework&gt;" CommentCount="1" />
  <row Id="11126" PostTypeId="1" CreationDate="2011-05-22T18:30:34.947" Score="0" ViewCount="12" Body="&lt;p&gt;Can someone please help me with this question? I’ve really tried but I’m not understanding what to do at all. Suppose an investor is subject to income tax and CGT. A 10-year bond is redeemable at par (at100) and its price is, say, X.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At time 7.6 (no coupon) he sells and buys back the bond at price Y. What is the price Y? Is this a good idea? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The solution says it is not a good idea but does not expand. We have to split it up into cases, i.e. X100 and compare the capital gain to the tax I think but I am not sure how to do this. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The solution says that ‘it will only be a good idea when offsetting is allowed’. What does this mean and why? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.s. this is bookwork not homework but i didnt know how else to label it&lt;/p&gt;&#xA;" OwnerUserId="4645" LastActivityDate="2011-05-22T18:30:34.947" ClosedDate="2011-05-22T21:07:35.880" Title="Capital Gain Tax / Bond Price" Tags="&lt;homework&gt;" CommentCount="1" />
  <row Id="11127" PostTypeId="1" AcceptedAnswerId="11132" CreationDate="2011-05-22T18:33:57.020" Score="5" ViewCount="113" Body="&lt;p&gt;I have 2 dependent variables (DVs) each of whose score may be influenced by the set of 7 independent variables (IVs). DVs are continuous, while the set of IVs consists of a mix of continuous and binary coded variables. (In code below continuous variables are written in upper case letters and binary variables in lower case letters.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The aim of the study is to uncover how these DVs are influenced by IVs variables. I proposed the following multivariate multiple regression (MMR) model:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;my.model &amp;lt;- lm(cbind(A, B) ~ c + d + e + f + g + H + I)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To interpret the results I call two statements:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;code&gt;summary(manova(my.model))&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;Manova(my.model)&lt;/code&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Outputs from both calls are pasted below and are significantly different. Can somebody please explain which statement among the two should be picked to properly summarize the results of MMR, and why? Any suggestion would be greatly appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Output using &lt;code&gt;summary(manova(my.model))&lt;/code&gt; statement:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; summary(manova(my.model))&#xA;           Df   Pillai approx F num Df den Df    Pr(&amp;gt;F)    &#xA;c           1 0.105295   5.8255      2     99  0.004057 ** &#xA;d           1 0.085131   4.6061      2     99  0.012225 *  &#xA;e           1 0.007886   0.3935      2     99  0.675773    &#xA;f           1 0.036121   1.8550      2     99  0.161854    &#xA;g           1 0.002103   0.1043      2     99  0.901049    &#xA;H           1 0.228766  14.6828      2     99 2.605e-06 ***&#xA;I           1 0.011752   0.5887      2     99  0.556999    &#xA;Residuals 100                                              &#xA;---&#xA;Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Output using &lt;code&gt;Manova(my.model)&lt;/code&gt; statement:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; library(car)&#xA;&amp;gt; Manova(my.model)&#xA;&#xA;Type II MANOVA Tests: Pillai test statistic&#xA;  Df test stat approx F num Df den Df    Pr(&amp;gt;F)    &#xA;c  1  0.030928   1.5798      2     99   0.21117    &#xA;d  1  0.079422   4.2706      2     99   0.01663 *  &#xA;e  1  0.003067   0.1523      2     99   0.85893    &#xA;f  1  0.029812   1.5210      2     99   0.22355    &#xA;g  1  0.004331   0.2153      2     99   0.80668    &#xA;H  1  0.229303  14.7276      2     99 2.516e-06 ***&#xA;I  1  0.011752   0.5887      2     99   0.55700    &#xA;---&#xA;Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="609" LastActivityDate="2011-05-22T20:14:46.023" Title="Multivariate multiple regression in R" Tags="&lt;r&gt;&lt;multivariate-analysis&gt;&lt;manova&gt;&lt;multiple-regression&gt;&lt;multivariate-regression&gt;" AnswerCount="1" FavoriteCount="2" />
  <row Id="11128" PostTypeId="2" ParentId="11120" CreationDate="2011-05-22T18:59:54.430" Score="4" ViewCount="" Body="&lt;p&gt;You might look at the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://cran.r-project.org/web/packages/tawny/index.html&quot; rel=&quot;nofollow&quot;&gt;http://cran.r-project.org/web/packages/tawny/index.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.rinfinance.com/RinFinance2009/presentations/yollin_slides.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.rinfinance.com/RinFinance2009/presentations/yollin_slides.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://nurometic.com/quantitative-finance/tawny/portfolio-optimization-with-tawny&quot; rel=&quot;nofollow&quot;&gt;http://nurometic.com/quantitative-finance/tawny/portfolio-optimization-with-tawny&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://quantivity.wordpress.com/2011/04/17/minimum-variance-portfolios/&quot; rel=&quot;nofollow&quot;&gt;http://quantivity.wordpress.com/2011/04/17/minimum-variance-portfolios/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2775" LastActivityDate="2011-05-22T18:59:54.430" CommentCount="1" />
  <row Id="11129" PostTypeId="2" ParentId="11049" CreationDate="2011-05-22T19:06:50.440" Score="1" ViewCount="" Body="&lt;p&gt;Dealing with missing observation is a never ending problem.&#xA;There are a few approaches i will mention, but I am sure others will add more.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Make sure there is no any problems in recording the data, such as systematic omitting certain values. Given you have no control for it, then other options need to be considered.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Based on the proportion of missing values, you have different options.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;If there are less than 5% missing observations:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2.1 I would not care, and use either average or median based on the shape of the sample's distribution. Some use this approach even when there are more than 5% missing observations. It is a very quick solution when time pressed. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If there are more than 5% missing observations, some advanced techniques can used: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;2.2  Data imputation methods is a wide class of techniques to impute missing values based on the observed values of the variable/or other variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, in time series framework, some values could be forecast based on the observed time series data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In regression framework, knowledge of other variables as well can be used to impute missing variables, i.e., build a regression model for the variable of interest (the one with missing values) as a function of other variables available (probably excluding the original dependent variable Y) and then predict the missing values using the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A more advanced technique is Expectation Maximization method.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, a dummy variable can be used just to control for the missing values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;M  &lt;/p&gt;&#xA;" OwnerUserId="4617" LastActivityDate="2011-05-22T19:06:50.440" CommentCount="1" />
  <row Id="11130" PostTypeId="2" ParentId="10949" CreationDate="2011-05-22T19:19:30.557" Score="1" ViewCount="" Body="&lt;p&gt;Sara, to understand the difference in the two models is to ask you what question you are trying to find an answer to:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;What is the health care expenditure of those who have it? &lt;/li&gt;&#xA;&lt;li&gt;What is the impact on having health insurance on health care expenditure? &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;To answer the first question, you have a subsample (as you refer to it) of those who have health insurance and there is observed health care expenditures as well as other explanatory variables (age, occupation, etc.). In this case, you are trying to explain variation in health care expenditures of people who choose to have health insurance. There is no way two answer your fist question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second question is more general. You need to have two groups of people with and without insurance. Also, if you just use OLS here on the subsample of insured people, you will get biased and inconsistent OLS parameter estimates due to self-selection bias, and you would not be able to correctly estimate the impact of having the insurance on the expenditure. So, Heckman's!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope that helps.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;M &lt;/p&gt;&#xA;" OwnerUserId="4617" LastActivityDate="2011-05-22T19:19:30.557" CommentCount="4" />
  <row Id="11131" PostTypeId="1" CreationDate="2011-05-22T19:34:27.523" Score="4" ViewCount="62" Body="&lt;p&gt;I am wondering if there is a sample size formula like Lehr's formula that applies to an F-test? Lehr's formula for t-tests is $n = 16 / \Delta^2$, where $\Delta$ is the effect size (&lt;em&gt;e.g.&lt;/em&gt; $\Delta = (\mu_1 - \mu_2) / \sigma$). This can be generalized to $n = c / \Delta^2$ where $c$ is a constant that depends on the type I rate, the desired power, and whether one is performing a one-sided or two sided test.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am looking for a similar formula for an F-test. My test statistic is distributed, under the alternative, as a non-central F with $k,n$ degrees of freedom and non-centrality parameter $n \lambda$, where $\lambda$ depends only on population parameters, which are unknown but posited to take some value. The parameter $k$ is fixed by the experiment, and $n$ is the sample size. Ideally I am looking for a (preferably well-known) formula of the form&#xA;$$n = \frac{c}{g(k,\lambda)}$$&#xA;where $c$ depends only on the type I rate and the power. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The sample size should satisfy&#xA;$$&#xD;&#xA;F(F^{-1}(1-\alpha;k,n,0);k,n,n\lambda) = \beta,$$&#xA;where $F(x;k,n,\delta)$ is the CDF of a non-central F with $k,n$ d.o.f. and non-centrality parameter $\delta$, and $\alpha, \beta$ are the type I and type II rates. We can assume $k \ll n$, &lt;em&gt;i.e.&lt;/em&gt; $n$ need be 'sufficiently large.'&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My attempts at fiddling with this in R have not been fruitful. I have seen $g(k,\lambda) = \lambda / sqrt(k+1)$ suggested but the fits have not looked very good.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;edit:&lt;/strong&gt; originally I had vaguely stated that the non-centrality parameter 'depends' on the sample size. On second thought, I found that too confusing, so made the relationship clear. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, I can compute the value of $n$ exactly by solving the implicit equation via a root finder (&lt;em&gt;e.g.&lt;/em&gt; Brent's method). I am looking for an equation to guide my intuition and for use as a rule of thumb.&lt;/p&gt;&#xA;" OwnerUserId="795" LastEditorUserId="795" LastEditorDisplayName="" LastEditDate="2011-05-23T02:57:07.730" LastActivityDate="2011-05-23T02:57:07.730" Title="Sample size formula for an F-test?" Tags="&lt;sample-size&gt;&lt;power&gt;&lt;non-central&gt;&lt;f-test&gt;" CommentCount="1" FavoriteCount="1" />
  <row Id="11132" PostTypeId="2" ParentId="11127" CreationDate="2011-05-22T19:42:34.220" Score="12" ViewCount="" Body="&lt;p&gt;Briefly stated, this is because base-R's &lt;code&gt;manova(lm())&lt;/code&gt; uses sequential model comparisons for so-called Type I sum of squares, whereas &lt;code&gt;car&lt;/code&gt;'s &lt;code&gt;Manova()&lt;/code&gt; by default uses model comparisons for Type II sum of squares.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I assume you're familiar with the model-comparison approach to ANOVA or regression analysis. This approach defines these tests by comparing a restricted model (corresponding to a null hypothesis) to an unrestricted model (corresponding to the alternative hypothesis). If you're not familiar with this idea, I recommend Maxwell &amp;amp; Delaney's excellent &quot;Designing experiments and analyzing data&quot; (2004).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For type I SS, the restricted model in a regression analysis for your first predictor &lt;code&gt;c&lt;/code&gt; is the null-model which only uses the absolute term: &lt;code&gt;lm(Y ~ 1)&lt;/code&gt;, where &lt;code&gt;Y&lt;/code&gt; in your case would be the multivariate DV defined by &lt;code&gt;cbind(A, B)&lt;/code&gt;. The unrestricted model then adds predictor &lt;code&gt;c&lt;/code&gt;, i.e. &lt;code&gt;lm(Y ~ c + 1)&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For type II SS, the unrestricted model in a regression analysis for your first predictor &lt;code&gt;c&lt;/code&gt; is the full model which includes all predictors except for their interactions, i.e., &lt;code&gt;lm(Y ~ c + d + e + f + g + H + I)&lt;/code&gt;. The restricted model removes predictor &lt;code&gt;c&lt;/code&gt; from the unrestricted model, i.e., &lt;code&gt;lm(Y ~ d + e + f + g + H + I)&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since both functions rely on different model comparisons, they lead to different results. The question which one is preferable is hard to answer - it really depends on your hypotheses.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What follows assumes you're familiar with how multivariate test statistics like the Pillai-Bartlett Trace are calculated based on the null-model, the full model, and the pair of restricted-unrestricted models. For brevity, I only consider predictors &lt;code&gt;c&lt;/code&gt; and &lt;code&gt;H&lt;/code&gt;, and only test for &lt;code&gt;c&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; N &amp;lt;- 100                             # generate some data: number of subjects&#xA;&amp;gt; c &amp;lt;- rbinom(N, 1, 0.2)               # dichotomous predictor c&#xA;&amp;gt; H &amp;lt;- rnorm(N, -10, 2)                # metric predictor H&#xA;&amp;gt; A &amp;lt;- -1.4*c + 0.6*H + rnorm(N, 0, 3) # DV A&#xA;&amp;gt; B &amp;lt;-  1.4*c - 0.6*H + rnorm(N, 0, 3) # DV B&#xA;&amp;gt; Y &amp;lt;- cbind(A, B)                     # DV matrix&#xA;&amp;gt; my.model &amp;lt;- lm(Y ~ c + H)            # the multivariate model&#xA;&amp;gt; summary(manova(my.model))            # from base-R: SS type I&#xA;          Df  Pillai approx F num Df den Df  Pr(&amp;gt;F)    &#xA;c          1 0.06835   3.5213      2     96 0.03344 *  &#xA;H          1 0.32664  23.2842      2     96 5.7e-09 ***&#xA;Residuals 97                                           &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For comparison, the result from &lt;code&gt;car&lt;/code&gt;'s &lt;code&gt;Manova()&lt;/code&gt; function using SS type II.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; library(car)                           # for Manova()&#xA;&amp;gt; Manova(my.model, type=&quot;II&quot;)&#xA;Type II MANOVA Tests: Pillai test statistic&#xA;  Df test stat approx F num Df den Df  Pr(&amp;gt;F)    &#xA;c  1   0.05904   3.0119      2     96 0.05387 .  &#xA;H  1   0.32664  23.2842      2     96 5.7e-09 ***&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now manually verify both results. Build the design matrix $X$ first and compare to R's design matrix.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; X  &amp;lt;- cbind(1, c, H)&#xA;&amp;gt; XR &amp;lt;- model.matrix(~ c + H)&#xA;&amp;gt; all.equal(X, XR, check.attributes=FALSE)&#xA;[1] TRUE&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now define the orthogonal projection for the full model ($P_{f} = X (X'X)^{-1} X'$, using all predictors). This gives us the matrix $W = Y&amp;#39; (I-P_{f}) Y$.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; Pf  &amp;lt;- X %*% solve(t(X) %*% X) %*% t(X)&#xA;&amp;gt; Id  &amp;lt;- diag(N)&#xA;&amp;gt; WW  &amp;lt;- t(Y) %*% (Id - Pf) %*% Y&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Restricted and unrestricted models for SS type I plus their projections $P_{rI}$ and $P_{uI}$, leading to matrix $B_{I} = Y&amp;#39; (P_{uI} - P_{PrI}) Y$.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; XrI &amp;lt;- X[ , 1]&#xA;&amp;gt; PrI &amp;lt;- XrI %*% solve(t(XrI) %*% XrI) %*% t(XrI)&#xA;&amp;gt; XuI &amp;lt;- X[ , c(1, 2)]&#xA;&amp;gt; PuI &amp;lt;- XuI %*% solve(t(XuI) %*% XuI) %*% t(XuI)&#xA;&amp;gt; Bi  &amp;lt;- t(Y) %*% (PuI - PrI) %*% Y&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Restricted and unrestricted models for SS type II plus their projections $P_{rI}$ and $P_{uII}$, leading to matrix $B_{II} = Y&amp;#39; (P_{uII} - P_{PrII}) Y$.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; XrII &amp;lt;- X[ , -2]&#xA;&amp;gt; PrII &amp;lt;- XrII %*% solve(t(XrII) %*% XrII) %*% t(XrII)&#xA;&amp;gt; PuII &amp;lt;- Pf&#xA;&amp;gt; Bii  &amp;lt;- t(Y) %*% (PuII - PrII) %*% Y&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Pillai-Bartlett trace for both types of SS: trace of $(B + W)^{-1} B$.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; (PBTi  &amp;lt;- sum(diag(solve(Bi  + WW) %*% Bi)))   # SS type I&#xA;[1] 0.0683467&#xA;&#xA;&amp;gt; (PBTii &amp;lt;- sum(diag(solve(Bii + WW) %*% Bii)))  # SS type II&#xA;[1] 0.05904288&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Note that the calculations for the orthogonal projections mimic the mathematical formula, but are a bad idea numerically. One should really use QR-decompositions or SVD in combination with &lt;code&gt;crossprod()&lt;/code&gt; instead.&lt;/p&gt;&#xA;" OwnerUserId="1909" LastEditorUserId="1909" LastEditorDisplayName="" LastEditDate="2011-05-22T20:14:46.023" LastActivityDate="2011-05-22T20:14:46.023" CommentCount="2" />
  <row Id="11133" PostTypeId="2" ParentId="10890" CreationDate="2011-05-22T19:44:47.340" Score="3" ViewCount="" Body="&lt;p&gt;The terms endogeneity and unobserved heterogeneity often refer to the same thing but usage varies somewhat, even within economics, the discipline I most associate with the terms.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a regression equation, an explanatory variable is &lt;a href=&quot;http://en.wikipedia.org/wiki/Endogeneity&quot; rel=&quot;nofollow&quot;&gt;endogenous&lt;/a&gt; if it is correlated with the error term.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Endogeneity is often described as having three sources: omitted variables, measurement error, and simultaneity.  Though it is often helpful to mention these &quot;sources&quot; separately, confusion sometimes arises because they are not truly distinct.  Imagine a regression predicting the effect of education on wages.  Perhaps our measure of education is simply the number of years someone spent in formal education, regardless of the type of education.  If I have a clear idea of what type of education affects wages, I might describe this situation as measurement error in the education variable.  Alternatively, I could describe the situation as an omitted variables problem (the variables indicating type of education).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Perhaps wages also affect education decisions.  If wages and education are measured at the same time this is an example of simultaneity, but it too, might be reframed in terms of omitted variables.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unobserved heterogeneity is simply variation/differences among cases which are not measured.  If you understand endogeneity, I think you understand the implications of unobserved heterogeneity in a regression context.&lt;/p&gt;&#xA;" OwnerUserId="3748" LastActivityDate="2011-05-22T19:44:47.340" />
  <row Id="11134" PostTypeId="1" AcceptedAnswerId="11140" CreationDate="2011-05-22T20:31:12.243" Score="3" ViewCount="56" Body="&lt;p&gt;I was playing around with writing a code for Montecarlo integration of a function defined in spherical coordinates. As a first simple rapid test I decided to write a test code to obtain  the solid angle under an angle $\theta_m$. For two random number $u$ and $v$ in $[0,1)$. I generate a homogeneous random sampling of the spherical angle using&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\phi=2\pi u$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\theta = \arccos(1-2v)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For $N$ generated points, I have $M$ points for which $\theta &amp;lt; \theta_m$. My first idea was that since I have an homogeneous sampling  I should have obtained the correct solid angle $\Omega=2 \pi (1-\cos (\theta_m))$ simply as $4\pi\times M/N$. &#xA;Actually it looks like I the correct result comes out only if I use:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\Omega=\sum_{i=1}^M \frac{4\pi}{N} 2 \cos(\theta_i)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can not see the reason why this should be correct. &#xA;The probability distribution function in $\theta$ is $PDF=1/2 \sin(\theta)$ so I would rather expect I should normalize each point of the sum by this function but this doesn't work. What am I doing wrong and how could I justify the cosine? Many thanks!&lt;/p&gt;&#xA;" OwnerUserId="4706" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-23T06:19:34.313" LastActivityDate="2011-05-23T06:19:34.313" Title="Monte carlo integration in spherical coordinates" Tags="&lt;sampling&gt;&lt;monte-carlo&gt;&lt;integral&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="11135" PostTypeId="1" CreationDate="2011-05-22T21:16:30.673" Score="4" ViewCount="52" Body="&lt;p&gt;I am doing a meta analysis for the first time and have a few basic questions regarding statistical analysis. &#xA;Let's say I have one study where the primary outcome (thrombosis) in the 2 treatment groups (intervention v. placebo) was compared using the Mantel-Haenszel chi-square test.  It does not report df in the article (is this something I can deduce?). Then it tells me that &quot;Treatment effects were expressed as a weighted average of the strata-specific relative risks.&quot; Is this something I need to consider for the meta-analysis statistical calculation? If so, what does it mean?&#xA;In the results we learn &quot;Among the 866 participants who had patency assessed, the primary outcome of fistula thrombosis at 6 weeks occurred in 53 participants (12.2%) in the clopidogrel group compared with 84 participants (19.5%) in the placebo group (relative risk, 0.63; 95% CI, 0.46-0.97;P=.018).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What do I do with all this information? I tried using a computerized program, but I didn't know what to fill all the blanks in with...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the second study, we have the following:&#xA;&quot;The hazard ratio was 0.81 (95% CI, 0.47 to 1.40) in favor of aspirin and clopidogrel therapy, but the reduction was not significant (P= 0.45). Although the event rates in the two treatment arms converged, there was no evidence that the proportional-hazards assumption was violated (P 0.53). The annual hazard rate for thrombosis among  articipants receiving placebos was 0.59 (95% CI, 0.39 to 0.87), compared with 0.47 (95% CI, 0.31 to 0.71) among participants receiving aspirin and clopidogrel. The absolute&#xA;risk reduction for the group receiving aspirin and clopidogrel was 0.03 (95% CI,-0.19 to 0.07), and the number needed to treat was 33.3 (95% CI, 14.9 to &gt;/=1000).&quot;&#xA;Regarding analysis, it states: &quot;The cumulative incidence of the first episode of thrombosis was estimated with the Kaplan-Meier method, and differences in rates between treatment groups were tested with the log rank test. The cumulative incidence of the first bleeding event was analyzed in a similar manner.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What do I do with all this information? Thank you for any help you can provide.&lt;/p&gt;&#xA;" OwnerUserId="4707" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-23T01:31:35.270" LastActivityDate="2011-05-23T01:31:35.270" Title="How to extract data from published articles (RCTs) to do a meta-analysis?" Tags="&lt;meta-analysis&gt;&lt;clinical-trials&gt;" CommentCount="1" />
  <row Id="11136" PostTypeId="1" CreationDate="2011-05-22T21:59:09.607" Score="3" ViewCount="46" Body="&lt;h3&gt;Context&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;I have a multivariate dataset with a test group and three control groups.&#xA;I was thinking that the best way to determine if and how the test group differed from all of the control groups would be to perform a MANOVA, then perform contrast analysis between the test group and all of the control groups.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Main Question&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;I cannot seem to find a good description of how contrast analysis for MANOVA is performed.&lt;br&gt;&#xA;Every tutorial/online class I can find, just talks about throwing the contrast into SPSS, or the univariate case.  &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Can anyone point me to a good tutorial on contrasts for MANOVA?  &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;Questions&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Is MANOVA with contrasts the appropriate way of answering my research question?&lt;/li&gt;&#xA;&lt;li&gt;Would it be better to perform repeated Hotelling's tests on the test group vs each control group, and just use a Bonferroni correction?  &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="3629" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-23T01:27:54.783" LastActivityDate="2011-05-23T01:27:54.783" Title="Tutorial for performing Contrasts for MANOVA" Tags="&lt;manova&gt;&lt;contrasts&gt;&lt;hotelling&gt;" FavoriteCount="1" />
  <row Id="11137" PostTypeId="2" ParentId="11109" CreationDate="2011-05-23T00:00:20.187" Score="0" ViewCount="" Body="&lt;p&gt;Be careful with this warning messenger from R. Take a look at this &lt;a href=&quot;http://www.stat.columbia.edu/~cook/movabletype/archives/2011/05/whassup_with_gl.html&quot; rel=&quot;nofollow&quot;&gt;blog post&lt;/a&gt; , by Andrew Gelman, and you will see that it is not always a problem of perfect separation, but sometimes a bug with glm. It seems that if the starting values are too far from the maximum likelihood estimate, it blows up. So, check first with another software, like Stata.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you really have this problem, you may try to use Bayesian modeling, with informative priors. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But in practice I just get rid of the predictors causing the trouble, because I don't know how to pick an informative prior. But I guess there is a paper by Gelman about using informative prior when you have this problem of perfect separation problem. Just google it. Maybe you should give it a try.&lt;/p&gt;&#xA;" OwnerUserId="3058" LastActivityDate="2011-05-23T00:00:20.187" />
  <row Id="11138" PostTypeId="1" CreationDate="2011-05-23T00:40:54.910" Score="2" ViewCount="54" Body="&lt;p&gt;Given an hierarchical clustering of data points, some of which are labeled, are there good ways to use the tree/dendrogram to make predictions for the unlabeled points?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One approach might be to find the &quot;best&quot; place to cut the tree so that clusters match labels.&#xA;I'd be especially interested in efficient ways to cut each cluster at a different height. But I'd also be interested in approaches that don't use &quot;hard&quot; cuts to make the predictions.&lt;/p&gt;&#xA;" OwnerUserId="4711" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-23T01:28:43.287" LastActivityDate="2011-05-23T18:45:34.503" Title="Using hierarchical clustering to classify?" Tags="&lt;clustering&gt;&lt;classification&gt;" AnswerCount="2" CommentCount="2" FavoriteCount="1" />
  <row Id="11139" PostTypeId="2" ParentId="10003" CreationDate="2011-05-23T01:11:57.523" Score="6" ViewCount="" Body="&lt;p&gt;As long as the sponsors of the site are committed to keeping the site running, it would be premature to declare it 'dead.'  It is not out of the question that StatProb.com may experience a revival in the future.  In judging the longevity of a resource like StatProb.com, the short-term trends are irrelevant.  Instead, the right questions to ask are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Is the &lt;em&gt;principle&lt;/em&gt; behind a site like StatProb.com a sound one?  Is the idea of a free access peer-reviewed encyclopedia an idea that will grow in relevance over time, or diminish?&lt;/li&gt;&#xA;&lt;li&gt;If the answer to the first question is &quot;Yes&quot;, then is it likely that an &lt;em&gt;alternative&lt;/em&gt; to the site will arise?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I think the answer to the first question is Yes.  The field of statistics is rapidly growing and the demand for online statistical answers is growing, as evidenced by this site (stats.SE).  The value of online encyclopedias has been proven by the success of Wikipedia.  Yet because Wikipedia is open to everyone, peer-reviewed alternatives to Wikipedia will eventually be needed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a site like StatProb.com gains more articles, it will gain more users, and as it gains more users, it will increase its public profile.  As it increases its public profile, more researchers will be interesting in contributing to the site.  That StatProb.com is off to a slow start gives no indication of where it may one day end up.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think the answer to the second question is No, because Springer.com has taken the lead in the online academic publishing world and it seems unlikely that it will give up that lead.  Any prospective competitor to StatProb.com will need a strong advantage to compensate for the brand-name recognition that Springer possesses.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I checked the site, and recently (5/11) a new article has appeared on 'Strong Mixing Conditions.'  As long as the site has the name of Springer attached to it, it will have some credibility in the academic world (whether it deserves it or not!) and a smart researcher can take advantage of this credibility.  I imagine it would be a useful place to write background information for you or a colleague to cite in your own papers.  I will keep the site StatProb in mind as a potential resource to that end, and I upvoted this question for making me aware of the site as a potential resource for my own academic career.&lt;/p&gt;&#xA;" OwnerUserId="3567" LastEditorUserId="3567" LastEditorDisplayName="" LastEditDate="2011-05-23T02:51:24.220" LastActivityDate="2011-05-23T02:51:24.220" CommentCount="1" />
  <row Id="11140" PostTypeId="2" ParentId="11134" CreationDate="2011-05-23T01:15:42.030" Score="4" ViewCount="" Body="&lt;p&gt;Your calculations are correct and you should get the right Monte Carlo estimate from your formula $4\pi M/N$ with the mapping $(u,v)\to(\phi,\theta)$ you presented. So I would guess it's an error in your code and not in your reasoning. You shouldn't need the cosine weighting factor.&#xA;Perhaps if you show your code we can spot the error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, on a related note, there's a really nice paper by James Arvo called &lt;a href=&quot;http://www.ics.uci.edu/~arvo/papers/notes2001.pdf&quot; rel=&quot;nofollow&quot;&gt;Stratified Sampling of 2-Manifolds&lt;/a&gt; which explains how to construct an area-preserving mapping from the unit-square to a 2-manifold in general, but also specifically for a (hemi-)sphere.&lt;/p&gt;&#xA;" OwnerUserId="4360" LastActivityDate="2011-05-23T01:15:42.030" CommentCount="1" />
  <row Id="11141" PostTypeId="1" AcceptedAnswerId="11144" CreationDate="2011-05-23T01:59:04.253" Score="1" ViewCount="57" Body="&lt;p&gt;Hi all,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know I'm asking a lot of questions these days! Sorry about that, but I'm trying to work through my grad thesis data collected over the last 4 years, and am repeatedly tripping on my beginner's grasp on stats.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The background:&lt;/strong&gt;&#xA;My basic question is the same as described in &lt;a href=&quot;http://stats.stackexchange.com/questions/11123/glm-for-comparing-2-populations-with-binomial-data&quot;&gt;another question on this site&lt;/a&gt;. In brief, I'm trying to sort out a reasonable way to compare good years vs. poor between 2 sites for a bunch of variables (breeding metrics of birds). I'm only interested in comparing good years between sites and poor years between sites. I'm not interested (much) in whether good years at one site are better than poor years at the other site (so I guess that means I'm not interested in interactions? Although what I want to get at is whether Site within Year-type is sig. different, so perhaps interactions &lt;em&gt;are&lt;/em&gt; important?). Though the metrics are related in that they're all from the same 2 populations, the variables I'm looking at are independent otherwise. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;This question&lt;/strong&gt; is about comparing nest initiation date (converted to ordinal day and is therefore ordinal data) between these 2 sites and 2 year-types. The data are skewed and the variances heterogeneous from a glance at a histogram and boxplots. My sample size ranges (for good within-site and poor within-site) from n= 69 to 864. I thought I should use a non-parametric 2-way ANOVAs and did some searching on those. I've seen Friedmans' test offered up as a non-parametric alternative to 2-way ANOVAs, but from what I gather, it's only for repeated measures data. I found some who recommended that you can rank-transform data and run the 2-way ANOVA on that. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To do this, I just converted the days within each site and within each year-type to ranks by making the first day (say, 154) = 1, day 155 = 2 etc. , but this really doesn't change anything in my data since I have ordinal dates anyway. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, what can I do? I've run the 2-way ANOVA on ranks and on the original centred data. While the numbers aren't the same from each test, the ultimate result is: the Holm-Sidak mulitple comparisons show that within each year-type of good and poor, the sites are significantly different (p&amp;lt;0.001). Can I accept these results? Or is there a better way of analyzing these data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm currently using Sigmaplot 11 (with built-in Sigmastat functionality) for the basic stuff, and turning to R whenever it can't handle what I need to do.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for reading, and for any tips you can provide!&#xA;Mog&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt;: I'm also interested in what I can do with &lt;em&gt;continuous&lt;/em&gt; data that is non-normal (skewed) and has unequal variances. Is rank-transforming inappropriate for that as well?&lt;/p&gt;&#xA;" OwnerUserId="4238" LastEditorUserId="4238" LastEditorDisplayName="" LastEditDate="2011-05-23T16:51:00.203" LastActivityDate="2011-05-23T16:51:00.203" Title="Rank transformed 2-way ANOVA" Tags="&lt;anova&gt;&lt;continuous-data&gt;&lt;ordinal&gt;&lt;ranks&gt;" AnswerCount="1" />
  <row Id="11142" PostTypeId="1" AcceptedAnswerId="11148" CreationDate="2011-05-23T02:47:10.030" Score="6" ViewCount="115" Body="&lt;p&gt;I participate in predictive modeling competitions on &lt;a href=&quot;http://www.kaggle.com/&quot; rel=&quot;nofollow&quot;&gt;Kaggle&lt;/a&gt; and &lt;a href=&quot;http://tunedit.org/&quot; rel=&quot;nofollow&quot;&gt;TunedIt&lt;/a&gt;.  I find that these sites are a good way to &quot;work-out&quot; for statistics/machine learning. Are there any other sites I should know about?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do you all feel about competitions where the host intends to profit from competitors' submissions?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;edit: Given the silence, I'm assuming kaggle and TuneEdit are the only sites out there that host predictive analytics competitions?&lt;/p&gt;&#xA;" OwnerUserId="2817" LastEditorUserId="2817" LastEditorDisplayName="" LastEditDate="2011-06-01T17:42:46.887" LastActivityDate="2011-06-01T17:42:46.887" Title="Sites for predictive modeling competitions" Tags="&lt;machine-learning&gt;&lt;predictive-models&gt;" AnswerCount="1" FavoriteCount="4" />
  <row Id="11144" PostTypeId="2" ParentId="11141" CreationDate="2011-05-23T03:42:57.647" Score="5" ViewCount="" Body="&lt;p&gt;The proportional odds (PO) ordinal logistic model is a generalization of the Wilcoxon and Kruskal-Wallis tests, allowing for covariates, interactions, and anything else you can do in a regression model for a univariate response.  A two-way ANOVA on ranks is not based on strong statistical principles.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of many computational tools for the PO model is the &lt;code&gt;lrm&lt;/code&gt; function in the R &lt;strong&gt;rms&lt;/strong&gt; package.  The rms package's &lt;code&gt;contrast&lt;/code&gt;, &lt;code&gt;anova&lt;/code&gt;, &lt;code&gt;summary&lt;/code&gt;, &lt;code&gt;plot&lt;/code&gt;, and &lt;code&gt;nomogram&lt;/code&gt; functions can help.&lt;/p&gt;&#xA;" OwnerUserId="4253" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-23T06:21:18.493" LastActivityDate="2011-05-23T06:21:18.493" CommentCount="1" />
  <row Id="11145" PostTypeId="1" CreationDate="2011-05-23T03:43:22.740" Score="1" ViewCount="93" Body="&lt;p&gt;Suppose I thought that ingesting greater than 100 mg of chemical X annually noticeably decreased one's weight. Also, I had data (from a &quot;natural&quot; experiment) from 100 people (some male and some female) measuring how much of chemical X they had eaten, and their weights at the time of the experiment. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What is the best way to test whether or not eating said amount of X leads to weight loss?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="4713" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-23T07:21:07.450" LastActivityDate="2011-05-25T01:35:52.533" Title="Determining causality from a natural experiment" Tags="&lt;hypothesis-testing&gt;" AnswerCount="3" CommentCount="1" />
  <row Id="11146" PostTypeId="2" ParentId="11145" CreationDate="2011-05-23T04:57:46.807" Score="8" ViewCount="" Body="&lt;p&gt;By a &quot;natural&quot; experiment you mean that you do &lt;em&gt;not&lt;/em&gt; control, by randomization, say, the amount of chemical X that each subject takes. This is also often called an &lt;em&gt;observational study&lt;/em&gt;. Do you know the difficulties in drawing conclusions about cause and effect from such data? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's really not a question about statistical tests - they are blind to causality. You can use standard methods to test if intake of chemical X is &lt;em&gt;associated&lt;/em&gt; with weight loss correcting for gender if you like, but that does not in itself prove that intake of X causes weight loss. An association might be viewed as evidence in the direction of a causal effect, but how strong the evidence is, and how serious it will be taken, is much more a question of understanding the subject matter than the statistical test.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a literature on causal inference, which I am only superficially familiar with, which gives a more nuanced picture on what you can say about causal effects and how you can do it, but a basic premise is a set of &lt;em&gt;untestable assumptions&lt;/em&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you can provide more details about what you know and what you want, I might be able to give you some appropriate references. There are also other, related, questions with answers &lt;a href=&quot;http://stats.stackexchange.com/questions/534/under-what-conditions-does-correlation-imply-causation&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;http://stats.stackexchange.com/questions/3400/from-a-statistical-perspective-can-one-infer-causality-using-propensity-scores-w&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="4376" LastEditorUserId="4376" LastEditorDisplayName="" LastEditDate="2011-05-23T05:03:41.510" LastActivityDate="2011-05-23T05:03:41.510" />
  <row Id="11147" PostTypeId="2" ParentId="11145" CreationDate="2011-05-23T05:51:29.287" Score="4" ViewCount="" Body="&lt;p&gt;It sounds like everyone in your sample has ingested X.  If your hypothesis is that ingesting X causes you to lose weight, you need a sample of people, some of whom have ingested X and some of whom haven't.  If your hypothesis is that the more X you ingest, the more weight you lose, then, of course, it's okay if your entire sample has ingested X.  You can also split up your sample into two groups if you thought that ONLY those who ingested &gt; 100mg will lose weight, but there are a lot of potential problems there and I wouldn't recommend it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the sound of it, what you probably want to start with is a simple regression to see if there is any connection between X and weight.  You can simply graph your sample, with weight lost (or gained) on one axis and amount of X ingested on the other.  Using a stat package like SPSS or R will tell you the correlation between X and losing weight (r), and how much of the variance (of weight loss) is accounted for by X (called r^2).  It's always a good idea to have a control group, though, to make sure the weight loss could not be attributed to something else.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are other things you can do depending on what data you have, but this is a good place to start I think.&lt;/p&gt;&#xA;" OwnerUserId="1977" LastActivityDate="2011-05-23T05:51:29.287" />
  <row Id="11148" PostTypeId="2" ParentId="11142" CreationDate="2011-05-23T06:30:18.670" Score="3" ViewCount="" Body="&lt;p&gt;Here are some nice datasets: &lt;a href=&quot;http://archive.ics.uci.edu/ml/&quot; rel=&quot;nofollow&quot;&gt;http://archive.ics.uci.edu/ml/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2860" LastActivityDate="2011-05-23T06:30:18.670" />
  <row Id="11149" PostTypeId="1" CreationDate="2011-05-23T11:33:59.683" Score="6" ViewCount="163" Body="&lt;p&gt;Is the method of mean substitution for replacing missing data out of date? Are there more sophisticated models that should be used? If so, what are they?&lt;/p&gt;&#xA;" OwnerUserId="4716" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-23T11:54:25.253" LastActivityDate="2011-05-24T10:59:39.857" Title="Is the method of mean substitution for replacing missing data out of date? " Tags="&lt;missing-data&gt;" AnswerCount="4" CommentCount="1" />
  <row Id="11150" PostTypeId="2" ParentId="11149" CreationDate="2011-05-23T11:53:59.070" Score="10" ViewCount="" Body="&lt;p&gt;Barring the fact that it's not necessary to shoot mosquitoes with a cannon (i.e. if you have one missing value in a million data points, just drop it), using the mean could be suboptimal to say the least: the result can be biased, and you should at least correct the result for the uncertainty.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are some other options, but the one easiest to explain is multiple imputation. The concept is simple: based upon a model for your data itself (e.g. obtained from the complete cases, though other options are available, like MICE), draw values from the associated distribution to 'complete' your dataset. Then in this completed dataset you don't have anymore missing data, and you can run your analysis of interest.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you did this only once (in fact, replacing the missing values with the mean is a very contorted form of this), it would be called single imputation, and there is no reason why it would perform better than mean replacement.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However: the trick is to do this repeatedly (hence Multiple Imputation), and each time do your analysis on each completed (=imputed) dataset. The result is typically a set of parameter estimates or similar for each completed dataset. Under relatively loose conditions, it is OK to average your parameter estimates over all these imputed datasets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The advantage is that there also exists a simple formula to adjust the standard error for the uncertainty caused by the missing data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to know more, you probably want to read Little and Rubin's 'Statistical Analysis with Missing Data'. This also holds other methods (EM,...) and more explanation on how/why/when they work.&lt;/p&gt;&#xA;" OwnerUserId="4257" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-23T12:21:03.900" LastActivityDate="2011-05-23T12:21:03.900" CommentCount="1" />
  <row Id="11151" PostTypeId="2" ParentId="11149" CreationDate="2011-05-23T11:54:42.270" Score="10" ViewCount="" Body="&lt;p&gt;You did not tell us very much about the nature of your missing data. Did you check for MCAR (&lt;a href=&quot;http://en.wikipedia.org/wiki/Missing_completely_at_random&quot; rel=&quot;nofollow&quot;&gt;Missing Completely at Random&lt;/a&gt;)? Given that you cannot assume MCAR, mean substitution can lead to biased estimators. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a non-mathematical starting point, I can recommend the following two references:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.stats.ox.ac.uk/~snijders/Graham2009.pdf&quot; rel=&quot;nofollow&quot;&gt;Graham, Hohn W. (2009): Missing Data Analysis: Making It Work in the Real World.&lt;/a&gt; &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://books.google.com/books?id=ZtYArHXjpB8C&amp;amp;lpg=PP1&amp;amp;hl=de&amp;amp;pg=PA11#v=onepage&amp;amp;q&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;Allison, Paul (2002): Missing data.&lt;/a&gt; (see section &quot;Imputation&quot;, p. 11)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="307" LastActivityDate="2011-05-23T11:54:42.270" CommentCount="1" />
  <row Id="11152" PostTypeId="1" AcceptedAnswerId="11154" CreationDate="2011-05-23T13:12:51.430" Score="3" ViewCount="67" Body="&lt;p&gt;I am going to be hosting a number (~10) of &lt;a href=&quot;http://en.wikipedia.org/wiki/Potluck&quot; rel=&quot;nofollow&quot;&gt;potluck meals&lt;/a&gt; over the course of the summer, my pool of people to invite is about 40 people with about 10-15 coming to each meal. So I figure this would be a good opportunity to record data over time about the meals/people. The issue I am having is I am not sure what information to keep track of and what format to record it in.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are some examples of trends I think would be interesting:  &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;How many meals I have invited people to&lt;/li&gt;&#xA;&lt;li&gt;On average which round of invites did people get invited to (some people rsvp as no in the beginning and so there is another 'round' of invites)&#xA;&lt;li&gt;How many meals people have attended&lt;/li&gt;&#xA;&lt;li&gt;What items people have brought&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I have started a spreadsheet where each page is a meal, the first few columns of the page represent different rounds of invites, I input a persons name in the column that corresponds to the round of their invite. The last two columns are the ultimate rsvp from any round of invitation and the item brought if applicable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To summarize I am looking for an efficient and concise way of recording the data associated with these meals for the trends mentioned. Additionally I am looking for other trends I can keep track of, I am doing a lot of this communication via email so timestamps would potentially be available for other interesting trends.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Help with good tags for this question would be appreciated.&lt;/p&gt;&#xA;" OwnerUserId="4717" LastEditorUserId="307" LastEditorDisplayName="" LastEditDate="2011-05-23T22:49:21.180" LastActivityDate="2011-05-23T22:49:21.180" Title="Statistics of events and invitations" Tags="&lt;dataset&gt;&lt;multilevel-analysis&gt;&lt;trend&gt;&lt;data-collection&gt;" AnswerCount="1" />
  <row Id="11153" PostTypeId="2" ParentId="11149" CreationDate="2011-05-23T15:03:30.203" Score="1" ViewCount="" Body="&lt;p&gt;If your missing values are randomly distributed, or your sample size is small, you might be better off just using the mean.  I would first split the data into two parts: 1 with the missing values and the other without and then test for the difference in means of some key variables between the two samples.  If there is no difference, you have some support for substituting the  mean, or just deleting the observations entirely.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-Ralph Winters&lt;/p&gt;&#xA;" OwnerUserId="3489" LastActivityDate="2011-05-23T15:03:30.203" CommentCount="3" />
  <row Id="11154" PostTypeId="2" ParentId="11152" CreationDate="2011-05-23T15:06:28.850" Score="3" ViewCount="" Body="&lt;p&gt;Yoel, great question! I will address your question of what can be an &quot;efficient and concise way of recording the data&quot;. Given your small data set, the following thoughts are more of theoretical nature than of practical use.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You have (what social scientists call) a multilevel data set, e.g. students (level 1) are nested in classes (level 2) which are nested in schools (level 3). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unfortunately, your case is more complicated because each meal can be attended by more than one person &lt;em&gt;and&lt;/em&gt; each person can attend more than one meal. So, there is no easy way to handle $1:n$ relationship between meals and persons. Furthermore, it is unrealistic to assume that each person brings unique items to each meal, i.e. many will bring salad, bread, cheese, etc. (that's at least my experience ;-) Again, there is no easy $1:n$ relationship between person and item(s). The following picture may be helpful to illustrate what I mean:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.stack.imgur.com/Envvw.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you had more data, I would recommand using a relational database management system (MS-Access, MySQL, SQLite etc.) and you would need the following 5 tables (or relations). Each table needs at a minimum the following identifier variables:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Meal&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;meal.id&lt;/code&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Person&lt;/strong&gt;: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;person.id&lt;/code&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Item&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;item.id&lt;/code&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Since you do not have $1:n$-relations, you also need auxiliary-tables which help you to establish $n:m$ relations:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;meal-person&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;meal.id&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;person.id&lt;/code&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;person-item&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;person.id&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;item.id&lt;/code&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;By the way, if you intend to do more complicated (regression) analyzes than those mentioned in your question, then you will need to run &quot;Multilevel Models With Crossed Random Effects&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Yoel asked for some resources to get started with multilevel analysis:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;My all time favourite intro text is &quot;&lt;a href=&quot;http://www.ats.ucla.edu/stat/paperexamples/singer/default.htm&quot; rel=&quot;nofollow&quot;&gt;Using SAS Proc Mixed to Fit Multilevel Models, Hierarchical Models, and Individual Growth Models&lt;/a&gt;&quot; by Judith Singer. This is a basic introduction to multilevel models. The text seems, however, currently not available. &lt;/li&gt;&#xA;&lt;li&gt;The University of Bristol Centre for Multilevel Modelling offers &quot;&lt;a href=&quot;http://www.bristol.ac.uk/cmm/learning/&quot; rel=&quot;nofollow&quot;&gt;Training materials and online information about multilevel modelling and MLwiN&lt;/a&gt;&quot;. Lean back and enjoy the videos!&lt;/li&gt;&#xA;&lt;li&gt;Joop Hox has written one of the best textbooks and a lot of articles. Some of them can be found on his &lt;a href=&quot;http://joophox.net/&quot; rel=&quot;nofollow&quot;&gt;homepage&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;li&gt;The UCLA Academic Technology Service offers &lt;a href=&quot;http://www.ats.ucla.edu/stat/examples/&quot; rel=&quot;nofollow&quot;&gt;textbook examples&lt;/a&gt;, see the section about Multilevel Modeling.&lt;/li&gt;&#xA;&lt;li&gt;Douglas Bates is currently working on a book called &quot;&lt;a href=&quot;http://lme4.r-forge.r-project.org/&quot; rel=&quot;nofollow&quot;&gt;lme4: Mixed-effects Modeling with R&lt;/a&gt;&quot;. Chapter two is about &quot;Models With Multiple Random-effects Terms&quot;.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;However, I was not successful in finding a good intro text which covers multilevel models with crossed random effects. Maybe, this &lt;a href=&quot;http://www.analysisfactor.com/statchat/multilevel-models-with-crossed-random-effects/&quot; rel=&quot;nofollow&quot;&gt;blog post&lt;/a&gt; can get you started.&lt;/p&gt;&#xA;" OwnerUserId="307" LastEditorUserId="307" LastEditorDisplayName="" LastEditDate="2011-05-23T22:44:00.780" LastActivityDate="2011-05-23T22:44:00.780" CommentCount="5" />
  <row Id="11155" PostTypeId="2" ParentId="11101" CreationDate="2011-05-23T15:54:10.807" Score="1" ViewCount="" Body="&lt;p&gt;With a sample size of 104, any factor analysis is going to be shaky at best. The best approach is probably to collect more data (not really that useful an answer, but its true). &lt;a href=&quot;http://www.technion.ac.il/docs/sas/stat/chap26/sect21.htm&quot; rel=&quot;nofollow&quot;&gt;This page&lt;/a&gt; gives some useful advice.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.statpower.net/Content/312/Handout/Fabrigar1999.pdf&quot; rel=&quot;nofollow&quot;&gt;Fabrigar et al (1999)&lt;/a&gt; indicate that large communalities can often lower the sample size required, but almost 1 is probably way too high. I would drop the offending items and re-run the analysis (using ML, principal axis and another method of your choice) and see what the results are. If they still produce heywood cases, then FA is probably not the right approach.&lt;/p&gt;&#xA;" OwnerUserId="656" LastActivityDate="2011-05-23T15:54:10.807" />
  <row Id="11156" PostTypeId="2" ParentId="11088" CreationDate="2011-05-23T17:29:06.840" Score="3" ViewCount="" Body="&lt;p&gt;The best (fastest to run, not fastest to code;) free solution I have found in Matlab was to wrap R's MATLHLIB_STANDALONE c library with a mex function. This gives you access to R's t-distribution PRNG. One advantage of this approach is that you also can use the same trick to get variates from a non-central t distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second best free solution was to use octave's implementation of &lt;a href=&quot;http://www.gnu.org/software/octave/doc/interpreter/Random-Number-Generation.html&quot; rel=&quot;nofollow&quot;&gt;trnd&lt;/a&gt;. Porting from octave turned out to be more work than wrapping c code for me. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For my tastes, using uniform generation via &lt;code&gt;rand&lt;/code&gt; and inverting via &lt;code&gt;tinv&lt;/code&gt; was much too slow. YMMV.&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2011-05-23T17:29:06.840" />
  <row Id="11157" PostTypeId="1" AcceptedAnswerId="11162" CreationDate="2011-05-23T18:01:24.313" Score="4" ViewCount="88" Body="&lt;p&gt;I have time series data that represent dates/times of trades taken in a financial market. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to assign a score to this data that represents whether the trades are &lt;code&gt;mostly clustered&lt;/code&gt; around particular time values or if they are  &lt;code&gt;mostly spread out&lt;/code&gt; evenly. I am going to have about 1000+ results per dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Example situation one (High degree of &quot;clustering&quot; ):&lt;/strong&gt;  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1. 01/01/01 : 13:00 &#xA;2. 01/01/01 : 13:10&#xA;3. 01/01/01 : 13:15&#xA;4. 01/01/01 : 13:25&#xA;5. 03/05/01 : 17:20&#xA;6. 03/05/01 : 17:35&#xA;7. 03/05/01 : 17:40&#xA;8. 03/05/01 : 17:45&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Example situation two( Low degree of &quot;clustering)&quot;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1. 01/01/01 : 13:00&#xA;2. 01/05/01 : 02:30&#xA;4. 02/12/01 : 06:40&#xA;5. 02/25/01 : 02:30&#xA;6. 03/30/01 : 21:10&#xA;7. 04/12/01 : 02:20&#xA;8. 05/02/01 : 03:25&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I can of course convert all the timestamps to posix time or whatnot so doing calculation with the time values won't be a problem.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was thinking possibly standard error?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(For those who want more background info: I am using backtest results to modulate the size of my entry position in a complex manner. If the results contain trades that are clustered together, then they don't really count as 1 trade each (more like one big trade). This means that such results are untrustworthy and I should not act on them.)&#xA;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="4544" LastActivityDate="2011-05-24T21:34:01.523" Title="What statistical test can I use to detect clumping?" Tags="&lt;time-series&gt;&lt;statistical-significance&gt;&lt;standard-error&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="11158" PostTypeId="2" ParentId="11021" CreationDate="2011-05-23T18:12:47.270" Score="1" ViewCount="" Body="&lt;p&gt;I contacted Sean at RezScore and he clarified some things for me. In a nutshell, inserting buzzwords into a hidden text box seems to be a good idea if you don't want to put them in your actual resume. However, you should be selective about which words you include because many of the algorithms penalize verbosity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maybe RezScore will include a feature to do just this for specific industries, I'd bet it would be cheaper than a resume rewrite.&lt;/p&gt;&#xA;" OwnerUserId="4685" LastActivityDate="2011-05-23T18:12:47.270" />
  <row Id="11159" PostTypeId="2" ParentId="11138" CreationDate="2011-05-23T18:21:27.423" Score="0" ViewCount="" Body="&lt;p&gt;I'm answering about another approach that doesn't use hard cuts on the dendrogram. I would suggest you to use something like linear discriminant analysis (LDA) or any other technique that allows you to predict the class of the unlabeled points. (There are many techniques that can do the job, but I find LDA the easiest)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;LAD is used when you have a set of multivariate observations, and those observations belong to a particular class. This dataset is composed by the labeled points. On the other hand you may have a set of points in the same variables but the class is unknown. These are the unlabeled points. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The goal of LDA is to classify the unknown points in the given classes. It is important to notice that in your case, the classes are defined by the hierarchical clustering you've already performed. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Discriminant analysis tries to define linear boundaries between the classes, creating some sort of &quot;territories&quot; (or regions) for each class. For any unlabeled point, you must check to which region it belongs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can check this &lt;a href=&quot;http://research.cs.tamu.edu/prism/lectures/pr/pr_l10.pdf&quot; rel=&quot;nofollow&quot;&gt;lecture on LDA&lt;/a&gt;. It is simple and sufficiently explained.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you need software to accomplish this task, you might check &lt;a href=&quot;http://stat.ethz.ch/R-manual/R-patched/library/MASS/html/lda.html&quot; rel=&quot;nofollow&quot;&gt;R documentation&lt;/a&gt; on the functions &lt;code&gt;lda()&lt;/code&gt; and &lt;code&gt;predict.lda()&lt;/code&gt; in &lt;code&gt;MASS&lt;/code&gt; package. Check &lt;a href=&quot;http://www.statmethods.net/advstats/discriminant.html&quot; rel=&quot;nofollow&quot;&gt;Quick-R&lt;/a&gt; for additional help. SPSS has a very easy implementation of LDA.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In fact, I think it is very odd to &quot;force&quot; the dendrogram to include the unlabeled points. Normally, the dendrogram is built using all of the unlabeled points in a dataset.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this is useful :)&lt;/p&gt;&#xA;" OwnerUserId="2902" LastActivityDate="2011-05-23T18:21:27.423" />
  <row Id="11160" PostTypeId="2" ParentId="11138" CreationDate="2011-05-23T18:45:34.503" Score="0" ViewCount="" Body="&lt;p&gt;Just my two cents, but I would look at decision trees or using your initial cluster analysis to determine a suitable number of clusters, and then use kmeans to refine.  From there, you can get the cluster centers and reclassify new cases based on those centers.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;HTH.&lt;/p&gt;&#xA;" OwnerUserId="569" LastActivityDate="2011-05-23T18:45:34.503" />
  <row Id="11161" PostTypeId="1" CreationDate="2011-05-23T19:24:32.360" Score="1" ViewCount="46" Body="&lt;p&gt;I have a sequence of integers that represent total sales of my product for each day. From time to time, we have large press or marketing events that increase sales on the day of the event and for a few days after that but then eventually taper down to the long-run average. Here's some made-up numbers showing what I mean:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;34, 40, 35, 36, 150, 110, 140, 107, 80, 68, 75, 50, 45, 35, 38, 41, 42,...&#xA;                ^^^                                   ^^^&#xA;          event occurs here                        reversion to mean&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have limited experience with statistics, so I'm looking for some guidance on statistical methods that I could use to determine:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;how much of the sales on the event day and the following days should be attributed to the event&lt;/li&gt;&#xA;&lt;li&gt;whether the press or marketing event contributes to a permanently higher sales average, even after its initial effect has worn off.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="4718" LastActivityDate="2011-05-24T01:52:15.880" Title="How can I isolate the effect of an event on a sequence of sales numbers?" Tags="&lt;mean&gt;" AnswerCount="1" />
  <row Id="11162" PostTypeId="2" ParentId="11157" CreationDate="2011-05-23T19:35:59.183" Score="1" ViewCount="" Body="&lt;p&gt;I would simply calculate a rolling window of the number of trades (or dollar volume) per hour, day, week, or whatever time frame that makes sense.   For example, you might use 1 day as the rolling window.  If 1 trade per day is a low degree of clustering then 10 trades per day might be a high degree of clustering.   If so, then a linear &quot;y&quot; scale for a &quot;clustering plot&quot; is probably reasonable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's an example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit 2 ===========================================&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Below is the updated version of the plot.   Just like the previous plots, the gray line is from a &quot;window&quot; of 1 day where the &quot;cluster number&quot; is the number of trades for the previous day.   The new blue line is from a &quot;window&quot; of 5 days where the &quot;cluster number&quot; is the sum of the trades for the previous 5 days divided by 5 (the divide by 5 is to scale the result so it can be directly compared to a 1 day &quot;window&quot;).    The new purple line is from a &quot;window&quot; that sums the trades for 10 days and then divides by 10, and the new green line is from a &quot;window&quot; for 20 days, divided by 20.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The last day in the plot (far right hand side) is for the day 2010-07-02 where the values are:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1 day window = 0&lt;/p&gt;&#xA;&#xA;&lt;p&gt;5 day window = 2&lt;/p&gt;&#xA;&#xA;&lt;p&gt;10 day window = 1.5&lt;/p&gt;&#xA;&#xA;&lt;p&gt;20 day window = 1.25&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you had chosen a &quot;window&quot; of 5 days, then before you trade on 2010-07-03 (assuming that's the next trading day), your &quot;cluster number&quot; would be 2 (averaging 2 trades per day for the previous 5 days).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just like any moving average, the longer the &quot;window&quot;, the smoother the plot.  However, this smoothing delays the peaks and valleys.  Compare the gray peak in early April with the blue peak, then the purple peak, and then the green peak.   This may not be a big issue for the current use, but  I thought it was a good idea to point it out.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The bottom line is, you'll have to play around with different &quot;windows&quot; to zero-in on your desired smoothness and timeliness.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.stack.imgur.com/UoejR.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="2775" LastEditorUserId="2775" LastEditorDisplayName="" LastEditDate="2011-05-24T02:24:52.420" LastActivityDate="2011-05-24T02:24:52.420" CommentCount="9" />
  <row Id="11163" PostTypeId="1" CreationDate="2011-05-23T19:43:20.070" Score="4" ViewCount="51" Body="&lt;p&gt;I would like to test that two difference/distance/dissimilarity matrices are not the same. i.e. the rows and columns between the two matrices represent the same features, but the distances are obtained from 2 populations and I'm interested in whether the difference matrices &quot;look different&quot; between the populations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm think I'm looking for something similar to a Mantel's test, but with the null hypothesis flipped. Whereas (as I understand it) the Mantel test tests for a linear correlation between two dissimilarity matrices against a null hypothesis of no linear correlation, the null in my case is that the two dissimilarity matrices are the same, and I'm interested in rejecting that null when two dissimilarity matrices differ significantly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a follow up to this question, once I have some sort of omnibus test for difference, what would be the best way to decompose the differences to contributions from individual cells.&lt;/p&gt;&#xA;" OwnerUserId="4720" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-23T23:19:53.857" LastActivityDate="2011-05-24T00:53:14.377" Title="How to test whether two distance/difference matrices are different?" Tags="&lt;clustering&gt;" AnswerCount="1" />
  <row Id="11164" PostTypeId="1" CreationDate="2011-05-23T19:50:26.923" Score="2" ViewCount="17" Body="&lt;p&gt;I administered a questionnaire a bunch of people people. The questionnaire asked about demographics, which I think I should model as random effects, and a bunch of related questions that could be thought of as a factorial layout, which I think I should model as fixed effects. I currently have a model where person is the only random effect, and I would like to include the demographics in the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There were 173 participants, with a different &quot;id&quot; value assigned to each person. This questionnaire included questions on demographics and 12 questions that can be thought of as a $2\times2\times2$ factorial layout. I thus have $2076=173\times12$ experimental units. The demographics are all categorical except for &quot;height&quot; and &quot;age&quot;. The data look something like this.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  id school county  building country height age  sex     task   space cleanliness posture   sit hover&#xA;1  4     GR      0 house_apt       1     58  24 male    urine  public unspecified   stand FALSE FALSE&#xA;2  4     GR      0 house_apt       1     58  24 male    urine private unspecified   stand FALSE FALSE&#xA;3  4     GR      0 house_apt       1     58  24 male defecate  public unspecified   hover FALSE  TRUE&#xA;4  4     GR      0 house_apt       1     58  24 male defecate private unspecified     sit  TRUE FALSE&#xA;5  4     GR      0 house_apt       1     58  24 male    urine  public       dirty   stand FALSE FALSE&#xA;6  4     GR      0 house_apt       1     58  24 male    urine  public       clean   stand FALSE FALSE&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The three factors in the factorial layout are &quot;task&quot;, &quot;space&quot; and&#xA;&quot;cleanliness&quot;. The response variable is the binary variable &quot;sit&quot;.&#xA;This model allows me to block by participant (&quot;id&quot;). In R, the code is&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; &amp;gt; glmer(sit ~ space*task*cleanliness + (1|id), family = binomial, data = posture.df)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I would like to model the effects of demographics, such as &quot;sex&quot; on both intercepts and slopes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The demographics were measured at the person level. If I wanted to&#xA;look at the effects of these demographics for only one of the 12&#xA;treatment combinations, I could do thus something like this.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  &amp;gt; glm(sit ~ sex*height+school, family = binomial, data = subset(posture.df,space=='public'&amp;amp;cleanliness=='unspecified'&amp;amp;task=='defecate'))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I would prefer, of course, to look at these effects on all 12&#xA;treatment combinations simultaneously. So how do I include the&#xA;demographics? I don't understand all of the relevant theory, so&#xA;I'd like to know more about the relevant theory and about how to&#xA;do this in R.&lt;/p&gt;&#xA;" OwnerUserId="3874" LastActivityDate="2011-05-23T19:50:26.923" Title="How do you model block-level attributes in a complete block design?" Tags="&lt;r&gt;&lt;mixed-model&gt;&lt;questionnaires&gt;&lt;glmer&gt;&lt;complete-block&gt;" />
  <row Id="11165" PostTypeId="1" AcceptedAnswerId="11173" CreationDate="2011-05-23T20:08:50.817" Score="4" ViewCount="90" Body="&lt;p&gt;Take the task of fitting an a priori distribution like the ex-Gaussian&#xA;to a collection of observed human response times (RT). One method is to compute the sum log liklihood of each observed RT given a set of candidate ex-Gaussian parameters, then try to find the set of parameters that maximizes this sum log likelihood. I wonder if this alternative approach might also be reasonable:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Select a set of equidistant quantile probabilities. eg:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;qps = seq( .1 , .9 , .1 )&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;For a given set of candidate ex-Gaussian parameters, estimate the&#xA;quantile RT values corresponding to qps. eg:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sim_dat = rnorm( 1e5 , mu , sigma ) + rexp( 1e5 , 1/tau )&#xA;qrt = quantile( sim_dat , prob = qps )&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;For each sequential interval between the thus-generated quantile RT&#xA;values, count the number of observations falling into that interval.&#xA;eg:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;obs_counts = rep( NA , length(qrt)-1 )&#xA;for( i in 1:(length(qrt)-1) ){&#xA;    obs_counts[i] = length( obs_rt[ (obs_rt&amp;gt;qrt[i]) &amp;amp; (obs_rt&amp;lt;=qrt[i+1]) ] )&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Compare these observed counts to the expected counts:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;exp_counts = diff(range(qps)) * diff(qps)[1] * length(obs_rt)&#xA;chi_sq = sum( (( obs_counts - exp_counts )^2 )/exp_counts )&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Repeat steps 2-4, searching for candidate parameter values that&#xA;minimize chi_sq.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Is this approach a reasonable alternative to the more standard maximum likelihood estimation procedure? Does this approach already have a name?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that I use the example of an ex-Gaussian purely for illustrative purposes; in practice I'm playing with using the above approach in a rather more complicated context (eg. fitting data to a stochastic model that yields multiple distributions, each with a different number of expected observation count). The purpose of this question is to ascertain whether I've re-invented the wheel as well as if anyone can pick out any problematic features of the method.&lt;/p&gt;&#xA;" OwnerUserId="364" LastEditorUserId="364" LastEditorDisplayName="" LastEditDate="2011-05-23T22:57:28.080" LastActivityDate="2011-05-24T02:13:03.453" Title="Is this a reasonable approach to fitting distributions?" Tags="&lt;distributions&gt;&lt;fitting&gt;" AnswerCount="3" FavoriteCount="1" />
  <row Id="11166" PostTypeId="2" ParentId="11165" CreationDate="2011-05-23T20:19:23.160" Score="0" ViewCount="" Body="&lt;p&gt;Take a look at the QQ-Plot (under my answer) in the following link:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://stats.stackexchange.com/questions/8662/need-help-identifying-a-distribution-by-its-histogram/8674#8674&quot;&gt;Need help identifying a distribution by its histogram&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2775" LastActivityDate="2011-05-23T20:19:23.160" CommentCount="1" />
  <row Id="11167" PostTypeId="2" ParentId="11165" CreationDate="2011-05-23T22:21:40.967" Score="4" ViewCount="" Body="&lt;p&gt;What you are proposing is called &lt;em&gt;quantile matching&lt;/em&gt;, though the way you propose to do it will be exhausting. The ex-Gaussian distribution can be found in the package &lt;code&gt;gamlss.dist&lt;/code&gt; with quantiles as &lt;code&gt;qexGAUS&lt;/code&gt; etc.; it uses &lt;code&gt;nu&lt;/code&gt; where you use &lt;code&gt;tau&lt;/code&gt;.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A similar quantile matching method can be used in the function &lt;code&gt;fitdist&lt;/code&gt; in the package &lt;code&gt;fitdistrplus&lt;/code&gt;  using &lt;code&gt;method=&quot;qme&quot;&lt;/code&gt;.  The package is mentioned in the answer linked by bill_080.  One difference is that it only matches as many quantiles as there are parameters (three in this case).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The following seems to work more or less: it simulates some data points from a particular ex-Gaussian distribution and then tries to estimate the parameters using quantile matching, and then draws some graphs.  It needs a rough estimate of the parameters to work.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(fitdistrplus)&#xA;library(gamlss.dist)&#xA;&#xA;set.seed(1)&#xA;sim_size &amp;lt;- 1000&#xA;Gm &amp;lt;- 10 # mean of Gaussian   &#xA;Gs &amp;lt;- 2  # sd of Gaussian&#xA;Em &amp;lt;- 5  # mean of exponential&#xA;sim_dat &amp;lt;- rnorm( sim_size , Gm , Gs ) + rexp( sim_size , 1/Em )&#xA;&#xA;fit_qme &amp;lt;- fitdist(sim_dat, &quot;exGAUS&quot;, method=&quot;qme&quot;, &#xA;                   start=c(mu=15, sigma=1, nu=3),&#xA;                   probs=c(0.2,0.5,0.8)               )&#xA;fit_qme&#xA;plot(fit_qme) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In this example and with this seed, the estimates are &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; fit_qme&#xA;Fitting of the distribution ' exGAUS ' by matching quantiles &#xA;Parameters:&#xA;      estimate&#xA;mu    9.859207&#xA;sigma 1.753703&#xA;nu    5.049785&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;By comparison a maximum likelihood estimate method using the same function might look something like &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;fit_mle &amp;lt;- fitdist(sim_dat, &quot;exGAUS&quot;, method=&quot;mle&quot;, &#xA;                   start=c(mu=15, sigma=1, nu=3)      )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and produce something like &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; fit_mle&#xA;Fitting of the distribution ' exGAUS ' by maximum likelihood &#xA;Parameters:&#xA;      estimate Std. Error&#xA;mu    9.938870  0.1656315&#xA;sigma 2.034017  0.1253632&#xA;nu    5.007996  0.2199171&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-05-23T22:21:40.967" CommentCount="3" />
  <row Id="11168" PostTypeId="1" CreationDate="2011-05-23T23:59:06.663" Score="2" ViewCount="65" Body="&lt;p&gt;Should the n for sample size be capitalized?  Is there a difference between n and N?&lt;/p&gt;&#xA;" OwnerUserId="4722" LastEditorUserId="2902" LastEditorDisplayName="" LastEditDate="2011-05-24T01:13:30.937" LastActivityDate="2011-05-24T03:03:16.627" Title="Capitalization of n for sample size." Tags="&lt;notation&gt;" AnswerCount="2" />
  <row Id="11169" PostTypeId="2" ParentId="11168" CreationDate="2011-05-24T00:03:19.040" Score="6" ViewCount="" Body="&lt;p&gt;There is actually a difference in some textbooks: $N$ generally means population size and $n$ sample size. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, this is not always the case. You should check in your textbook.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;:)&lt;/p&gt;&#xA;" OwnerUserId="2902" LastActivityDate="2011-05-24T00:03:19.040" />
  <row Id="11170" PostTypeId="2" ParentId="11163" CreationDate="2011-05-24T00:53:14.377" Score="0" ViewCount="" Body="&lt;p&gt;I am not sure I understand what you mean by difference/distance/dissimilarity matrix. Assuming that $D_{i,j}^2 = (v_i - v_j)^{\top}(v_i - v_j)$ for some vectors $v_i, v_j$, if you can accept a transformation to the crossproduct matrix $G_{i,j} = -2 v_i^{\top}v_j$ (say for example the vectors are normalized so $v_i^{\top}v_i = 1 = v_j^{\top}v_j$, and so you can subtract out the 2 from $D_{i,j}^2$). Then you can compare the two cross product matrices, $G$ and $H$ call them, by a Wilks' lambda test, I think. I'm not sure, but I think they would both have to be the same rotation away from Wishart matrices. The Wilks' lambda distribution would then describe the ratio of the products of the eigenvalues of the two matrices under the null.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This may not be applicable to your problem, though...&lt;/p&gt;&#xA;" OwnerUserId="795" LastActivityDate="2011-05-24T00:53:14.377" />
  <row Id="11171" PostTypeId="2" ParentId="11161" CreationDate="2011-05-24T01:19:16.530" Score="2" ViewCount="" Body="&lt;p&gt;To answer your question , one would be advised to build a single equation model which captured day-of-the-week effects (6 dummy indicators) and an indicator for the &quot;event&quot;. Software exists to capture any lead, contemporaneous and.or lag effects around known event. In the absence of such software you might try and &quot;roll your own&quot; in order to identify the &quot;window of response&quot; around your event variable. In addition you might want to include week-of-the-year and/or month-of-the-year variables to handle annual seasonality. Furthermore since this is daily data you might want to include other events such as the Holidays and also incorporating any required &quot;window of response&quot; around these Holidays. You should also try to identify &quot;particular days of the month&quot; irrespective of what day of the week they fall on as important contributors to explaining daily demand such as the end-of-the-month or the day seniors get their social security checks et al . Other factors that might be important in the &quot;discovery model phase&quot; might be the empirical identification of mean shifts and/or local time trends. Before congratulating yourself for either writing such a &quot;tour de force&quot; you might (should !) validate that neither the model parameters of the error variance have changed over time. By sharing your data and inviting the list to provide quality analysis of your data we could all learn what others are doing in a precise manner. Upon completion of the modelling phase one could then eliminate the event variables effect , essentially scrubbing the data by obtaining a realization of the daily sales series without the events coefficients/effects being incorporated. The difference between the originally observed series and this scrubbed series provides an estimate of the events effects.This recommended approach uses all of the data as compared to the incorrect approach of trying to forecast data values at a point prior to the events effects which of course is the unknown ( but to be found ) point in time where the effect &quot;starts&quot;. &lt;/p&gt;&#xA;" OwnerUserId="3382" LastEditorUserId="3382" LastEditorDisplayName="" LastEditDate="2011-05-24T01:52:15.880" LastActivityDate="2011-05-24T01:52:15.880" />
  <row Id="11172" PostTypeId="1" CreationDate="2011-05-24T01:49:43.733" Score="4" ViewCount="55" Body="&lt;p&gt;I'm trying to figure out how to calculate the standard error of a mean correlation coefficient.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have 6 bilateral correlation coefficients for 4 countries. I have transformed them using the Fisher z transformation in order to calculate their mean correlation coefficient. I'm trying to figure out what the standard error of this statistic is in order to create a confidence interval to test for significance against other mean correlation coefficients. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think I'll need to use the Fisher std error formula 1/SQRT(N-3) but I'm not sure what the N here should be. The sample size, or the number of bilateral pairs used in the mean correlation coefficient statistic? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I use the N=number of bilateral pairs, so that the standard error is the cross-sectional dispersion, my 95% confidence interval doesn't make sense. e.g. r = 0.7040 (-0.2510,0.9645). Using the n=number of observations would mean that the standard error is constant across all mean correlation coefficients. Would it make sense to bootstrap the errors instead?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It would be great if I could create a confidence interval for each of these mean statistics and report it in one table so that the reader is able to compare across these regional correlations as they like.&lt;/p&gt;&#xA;" OwnerUserId="4724" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-24T04:41:24.997" LastActivityDate="2011-05-24T10:38:39.853" Title="How to calculate standard error of the mean of a set of correlation coefficients" Tags="&lt;correlation&gt;&lt;confidence-interval&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="11173" PostTypeId="2" ParentId="11165" CreationDate="2011-05-24T02:13:03.453" Score="4" ViewCount="" Body="&lt;p&gt;One problematic feature is that there may be a continuum of optimal solutions.  In most settings the quantiles are continuous functions of the parameters.  When the distributions are continuous, almost surely there will be positive intervals between the data values.  Suppose your objective function is optimized by a particular parameter value whose quantiles do not coincide exactly with any of the data: that is, they lie in the interiors of the intervals determined by the nearby data values.  (This is an extremely likely event.)  Then small changes in the parameter value will move the quantiles slightly, to remain within the same intervals, thereby leaving the chi-squared value unchanged because none of the counts changes. Thus the procedure doesn't even pick out a definite set of parameter values!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another problematic feature is that this procedure apparently provides no way to obtain estimation errors for the parameters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another problem is that you do not know even the most basic properties of this estimator, such as its amount of bias.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-05-24T02:13:03.453" />
  <row Id="11174" PostTypeId="2" ParentId="11168" CreationDate="2011-05-24T03:03:16.627" Score="0" ViewCount="" Body="&lt;p&gt;In terms of ANOVA small n (usually subscripted) could mean the sample size of a particular group while capital N might mean the total sample size.  It depends on context.&lt;/p&gt;&#xA;" OwnerUserId="2310" LastActivityDate="2011-05-24T03:03:16.627" />
  <row Id="11175" PostTypeId="1" CreationDate="2011-05-24T04:43:59.053" Score="2" ViewCount="75" Body="&lt;p&gt;It is mentioned in &lt;a href=&quot;http://en.wikipedia.org/wiki/Determining_the_number_of_clusters_in_a_data_set#The_Elbow_Method&quot; rel=&quot;nofollow&quot;&gt;wiki page&lt;/a&gt; that one of the method to determine the optimal number of cluster in the dataset is &quot;elbow method&quot;. Here the percentage of variance is calculated as the ratio of the between-group variance to the total variance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I felt difficult in understanding this calculation. Can any one let me know how to calculate the percentage of variance for a dataset represented as feature matrix $F \in \mathbf{R}^{m \times n}$, where $m$ is the feature dimension and $n$ is the number of data points. I use k-means algorithm for clustering.&lt;/p&gt;&#xA;" OwnerUserId="4290" LastActivityDate="2011-05-24T13:53:07.530" Title="Elbow criteria to determine number of cluster" Tags="&lt;clustering&gt;&lt;k-means&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="11176" PostTypeId="1" CreationDate="2011-05-24T05:22:53.910" Score="3" ViewCount="43" Body="&lt;p&gt;&lt;a href=&quot;http://www.springerlink.com/content/p7241345l444446n/&quot; rel=&quot;nofollow&quot;&gt;Heathcote, Brown &amp;amp; Mewhort&lt;/a&gt; (2002, &lt;a href=&quot;http://www.springerlink.com/content/p7241345l444446n/fulltext.pdf&quot; rel=&quot;nofollow&quot;&gt;PDF&lt;/a&gt;) present Quantile Maximum Probability Estimation (originally termed Quantile Maximum Likelihood Estimation but later corrected) as a method of fitting distributional data, and find that it outperforms the more traditional Continuous Maximum Likelihood Estimation approach, at least in the case of fitting data to the ex-Gaussian (though see also &lt;a href=&quot;http://www.springerlink.com/content/wk1707q2k83513tg/&quot; rel=&quot;nofollow&quot;&gt;this paper&lt;/a&gt;, which shows that this benefit generalizes to other distributions as well).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm trying to understand the actual steps to achieving QMPE. I understand that one first specifies increasing and equidistant quantile probabilities, then uses these to obtain the quantile values (&lt;code&gt;q&lt;/code&gt;) in the observed data corresponding to these probabilities. I also understand that these observed quantile values are then used in counting the number of observations between each quantile (&lt;code&gt;N&lt;/code&gt;). But this is where I'm stuck. Presumably one searches through the parameter space of whatever a priori model one assumes generated the data, searching for a parameter set that maximizes the joint probability of &lt;code&gt;q&lt;/code&gt; and &lt;code&gt;N&lt;/code&gt;.  However, I don't know how, given a set of candidate parameters, this joint probability is computed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Not having a strong math background, I think much better in code, so if someone could help me figure out what comes next, I'd be greatly appreciative. Here's the beginning of an attempt to fit some data to an ex-Gaussian:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    #generate some data to fit&#xA;    true_mu = 300&#xA;    true_sigma = 50&#xA;    true_tau = 100&#xA;    my_data = rnorm(100,true_mu,true sigma)+rexp(100,1/true_tau)&#xA;&#xA;    #select some quantile probabilities; estimate quantiles and inter-quantile counts from the observed data&#xA;    quantile_probs = seq(.1,.9,.1) #or does it have to be seq(0,1,.1) ?&#xA;    q = quantile( my_data, probs = quantile_probs, type = 5 ) #Heathcote et al apparently use type=5 given their example&#xA;    N = rep( NA , length(q)-1 )&#xA;    for( i in 1:( length(q)-1 ) ){&#xA;        N = length( my_data[ (my_data&amp;gt;q[i]) &amp;amp; (my_data&amp;lt;=q[i+1]) ] )&#xA;    }&#xA;&#xA;    #specify some candidate parameter values to assess (normally done as part of an iterative search  using an optimizer like optim)&#xA;    candidate_mu = 350&#xA;    candidate_sigma = 25&#xA;    candidate_tau = 30&#xA;    #given these candidates, what next?&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="364" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-24T09:59:57.253" LastActivityDate="2011-05-24T09:59:57.253" Title="Can anyone explain quantile maximum probability estimation (QMPE)?" Tags="&lt;distributions&gt;&lt;fitting&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="11177" PostTypeId="2" ParentId="11176" CreationDate="2011-05-24T06:22:09.193" Score="1" ViewCount="" Body="&lt;p&gt;Just a small suggestion:&#xA;Have you checked out the &lt;a href=&quot;http://www.newcl.org/?q=node/10&quot; rel=&quot;nofollow&quot;&gt;Newcastle Cognition Lab's page on QMPE&lt;/a&gt;?&#xA;It has source code, a getting started guide, and a few other resources.&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2011-05-24T06:22:09.193" CommentCount="2" />
  <row Id="11178" PostTypeId="1" CreationDate="2011-05-24T07:17:41.610" Score="0" ViewCount="81" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;Possible Duplicate:&lt;/strong&gt;&lt;br&gt;&#xA;  &lt;a href=&quot;http://stats.stackexchange.com/questions/8661/logistic-regression-in-r-odds-ratio&quot;&gt;Logistic Regression in R (Odds Ratio)&lt;/a&gt;  &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&#xA;&#xA;&lt;p&gt;I need to do a logistic regression in R. My response variable is &lt;code&gt;surv=0&lt;/code&gt;; &lt;code&gt;surv=1&lt;/code&gt; and I have about 18 predictor variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After reading my model, I got the table of Coefficients below and I need to go through some steps, which I am not familiar with, until I get to the odds ratios.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is my first time to do a logistic regression in R and your help would be appreciated.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Call:&#xA;glm(formula = surv ~ as.factor(tdate) + as.factor(line) + as.factor(wt) + &#xA;    as.factor(crump) + as.factor(pind) + as.factor(pcscore) + &#xA;    as.factor(ptem) + as.factor(pshiv) + as.factor(pincis) + &#xA;    as.factor(presp) + as.factor(pmtone) + as.factor(pscolor) + &#xA;    as.factor(ppscore) + as.factor(pmstain) + as.factor(pbse) + &#xA;    as.factor(psex) + as.factor(pgf), family = binomial(link = &quot;logit&quot;), &#xA;    data = ap)&#xA;&#xA;Deviance Residuals: &#xA;    Min       1Q   Median       3Q      Max  &#xA;-1.9772  -0.5896  -0.4419  -0.3154   2.8264  &#xA;&#xA;Coefficients:&#xA;                           Estimate Std. Error z value Pr(&amp;gt;|z|)    &#xA;(Intercept)                -0.59796    0.27024  -2.213 0.026918 *  &#xA;as.factor(tdate)2009-09-08  0.43918    0.19876   2.210 0.027130 *  &#xA;as.factor(tdate)2009-09-11  0.27613    0.20289   1.361 0.173514    &#xA;as.factor(tdate)2009-09-15  0.58733    0.19232   3.054 0.002259 ** &#xA;as.factor(tdate)2009-09-18  0.52823    0.20605   2.564 0.010360 *  &#xA;as.factor(tdate)2009-09-22  0.45661    0.19929   2.291 0.021954 *  &#xA;as.factor(tdate)2009-09-25 -0.09189    0.21740  -0.423 0.672526    &#xA;as.factor(tdate)2009-09-29 -0.15696    0.28369  -0.553 0.580076    &#xA;as.factor(tdate)2010-01-26  1.39260    0.21049   6.616 3.69e-11 ***&#xA;as.factor(tdate)2010-01-29  1.67827    0.21099   7.954 1.80e-15 ***&#xA;as.factor(tdate)2010-02-02  1.35442    0.21292   6.361 2.00e-10 ***&#xA;as.factor(tdate)2010-02-05  1.36856    0.21439   6.383 1.73e-10 ***&#xA;as.factor(tdate)2010-02-09  1.18159    0.21951   5.383 7.33e-08 ***&#xA;as.factor(tdate)2010-02-12  1.40457    0.22001   6.384 1.73e-10 ***&#xA;as.factor(tdate)2010-02-16  1.01063    0.21783   4.639 3.49e-06 ***&#xA;as.factor(tdate)2010-02-19  1.54992    0.21535   7.197 6.14e-13 ***&#xA;as.factor(tdate)2010-02-23  0.85695    0.33968   2.523 0.011641 *  &#xA;as.factor(line)2           -0.26311    0.07257  -3.625 0.000288 ***&#xA;as.factor(line)5            0.06766    0.11162   0.606 0.544387    &#xA;as.factor(line)6           -0.30409    0.12130  -2.507 0.012176 *  &#xA;as.factor(wt)2             -0.33904    0.10708  -3.166 0.001544 ** &#xA;as.factor(wt)3             -0.28976    0.13217  -2.192 0.028359 *  &#xA;as.factor(wt)4             -0.50470    0.16264  -3.103 0.001915 ** &#xA;as.factor(wt)5             -0.74870    0.20067  -3.731 0.000191 ***&#xA;as.factor(crump)2           0.07537    0.10751   0.701 0.483280    &#xA;as.factor(crump)3          -0.14050    0.13217  -1.063 0.287768    &#xA;as.factor(crump)4          -0.20131    0.16689  -1.206 0.227724    &#xA;as.factor(crump)5          -0.23963    0.20778  -1.153 0.248803    &#xA;as.factor(pind)2           -0.29893    0.10752  -2.780 0.005434 ** &#xA;as.factor(pind)3           -0.40828    0.12436  -3.283 0.001027 ** &#xA;as.factor(pind)4           -0.73021    0.14947  -4.885 1.03e-06 ***&#xA;as.factor(pind)5           -0.68878    0.17650  -3.902 9.52e-05 ***&#xA;as.factor(pcscore)2        -0.52667    0.13606  -3.871 0.000108 ***&#xA;as.factor(ptem)2           -0.72600    0.08964  -8.099 5.52e-16 ***&#xA;as.factor(ptem)3           -0.79145    0.10503  -7.536 4.86e-14 ***&#xA;as.factor(ptem)4           -0.89956    0.10331  -8.707  &amp;lt; 2e-16 ***&#xA;as.factor(ptem)5           -0.90181    0.10721  -8.412  &amp;lt; 2e-16 ***&#xA;as.factor(pshiv)2           0.25236    0.07713   3.272 0.001068 ** &#xA;as.factor(pincis)2          0.02327    0.07216   0.323 0.747041    &#xA;as.factor(presp)2           0.43746    0.11598   3.772 0.000162 ***&#xA;as.factor(pmtone)2          0.34515    0.11178   3.088 0.002016 ** &#xA;as.factor(pscolor)2         0.53469    0.26851   1.991 0.046443 *  &#xA;as.factor(ppscore)2         0.25664    0.08751   2.933 0.003361 ** &#xA;as.factor(pmstain)2        -0.48619    0.84408  -0.576 0.564611    &#xA;as.factor(pbse)2           -0.28248    0.07335  -3.851 0.000117 ***&#xA;as.factor(psex)2           -0.18240    0.06385  -2.857 0.004280 ** &#xA;as.factor(pgf)12            0.10329    0.14314   0.722 0.470554    &#xA;as.factor(pgf)21           -0.06481    0.10772  -0.602 0.547388    &#xA;as.factor(pgf)22            0.39584    0.12740   3.107 0.001890 ** &#xA;as.factor(pgf)31            0.18820    0.10082   1.867 0.061936 .  &#xA;as.factor(pgf)32            0.39662    0.13963   2.841 0.004504 ** &#xA;as.factor(pgf)41            0.09178    0.10413   0.881 0.378106    &#xA;as.factor(pgf)42            0.21056    0.14906   1.413 0.157787    &#xA;---&#xA;Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 &#xA;&#xA;(Dispersion parameter for binomial family taken to be 1)&#xA;&#xA;    Null deviance: 7812.9  on 8714  degrees of freedom&#xA;Residual deviance: 6797.4  on 8662  degrees of freedom&#xA;  (418 observations deleted due to missingness)&#xA;AIC: 6903.4&#xA;&#xA;Number of Fisher Scoring iterations: 5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="4263" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-24T07:51:34.850" LastActivityDate="2011-05-24T07:58:08.240" ClosedDate="2011-05-24T16:26:12.197" Title="How to interpret table of logistic regression coefficients using glm function in R" Tags="&lt;r&gt;&lt;logistic-regression&gt;" AnswerCount="1" CommentCount="4" />
  <row Id="11179" PostTypeId="2" ParentId="11175" CreationDate="2011-05-24T07:53:03.603" Score="3" ViewCount="" Body="&lt;p&gt;The idea underlying the k-means algorithm is to try to find clusters that minimize the within-cluster variance (or up to a constant the corresponding sum of squares or SS), which amounts to maximize the between-cluster SS because the total variance is fixed. As mentioned on the wiki, you can directly use the within SS and look at its variation when increasing the number of clusters (like we would do in Factor Analysis with a screeplot): an abrupt change in how SS evolve is suggestive of an optimal solution, although this merely stands from visual appreciation. As the total variance is fixed, it is equivalent to study how the ratio of the between and total SS, also called the percentage of variance explained, evolves, because in this the case it will present a large gap from one k to the next k+1. (Note that the between/within ratio is not distributed as an F-distribution because k is not fixed; so, test are meaningless.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In sum, you just have to compute the squared distance between each data point and their respective center (or centroid), for each cluster--this gives you the within SS, and the total within SS is just the sum of the cluster-specific WSS (transforming them to variance is just a matter of dividing by the corresponding degrees of freedom); the between SS is obtained by substracting the total WSS from the total SS, the latter being obtained by considering k=1 for example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By the way, with k=1, WSS=TSS and BSS=0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're after determining the number of clusters or where to stop with the k-means, you might consider the Gap statistic as an alternative to the elbow criteria:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Tibshirani, R., Walther, G., and&#xA;  Hastie, T. (2001). &lt;a href=&quot;http://gremlin1.gdcb.iastate.edu/MIP/gene/MicroarrayData/gapstatistics.pdf&quot; rel=&quot;nofollow&quot;&gt;Estimating the&#xA;  numbers of clusters in a data set via&#xA;  the gap statistic&lt;/a&gt;. &lt;em&gt;J. R. Statist.&#xA;  Soc. B&lt;/em&gt;, 63(2): 411-423.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-24T13:53:07.530" LastActivityDate="2011-05-24T13:53:07.530" />
  <row Id="11180" PostTypeId="2" ParentId="11178" CreationDate="2011-05-24T07:58:08.240" Score="2" ViewCount="" Body="&lt;p&gt;Call&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;exp(your.model$coefficients)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where &lt;code&gt;your.model&lt;/code&gt; is your R object with &lt;code&gt;glm&lt;/code&gt; class. Similar question was ask previously; detailed answer is &lt;a href=&quot;http://stats.stackexchange.com/questions/8661/logistic-regression-in-r-odds-ratio&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="609" LastActivityDate="2011-05-24T07:58:08.240" />
  <row Id="11182" PostTypeId="1" AcceptedAnswerId="11183" CreationDate="2011-05-24T08:12:01.783" Score="5" ViewCount="76" Body="&lt;p&gt;Does anybody know why offset in a Poisson regression is used? What do you achieve by this? &lt;/p&gt;&#xA;" OwnerUserId="4496" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-24T08:30:58.793" LastActivityDate="2011-05-24T09:03:34.040" Title="When to use an offset in a Poisson regression?" Tags="&lt;regression&gt;&lt;poisson&gt;" AnswerCount="1" />
  <row Id="11183" PostTypeId="2" ParentId="11182" CreationDate="2011-05-24T09:03:34.040" Score="3" ViewCount="" Body="&lt;p&gt;Here is an example of application.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Poisson regression is typically used to model count data. But, sometimes, it is more relevant to model rates instead of counts. This is relevant when, e.g., individuals are not followed the same amount of time. For example, six cases over 1 year should not amount to the same as six cases over 10 years. So, instead of having &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\log \mu_x = \beta_0 + \beta_1 x$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(where $\mu_x$ is the expected count for those with covariate $x$), you have&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\log \tfrac{\mu_x}{t_x} = \beta&amp;#39;_0 + \beta&amp;#39;_1 x$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(where $t_x$ is the exposure time for those with covariate $x$). Now, the last equation could be rewritten&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\log \mu_x = \log t_x + \beta&amp;#39;_0 + \beta&amp;#39;_1 x$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and $\log t_x$ plays the role of an offset.&lt;/p&gt;&#xA;" OwnerUserId="3019" LastActivityDate="2011-05-24T09:03:34.040" CommentCount="2" />
  <row Id="11184" PostTypeId="2" ParentId="11172" CreationDate="2011-05-24T10:38:39.853" Score="4" ViewCount="" Body="&lt;p&gt;Just a few thoughts: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;n is the sample size for &lt;em&gt;each&lt;/em&gt; bivariate correlation, i.e. $n \neq 6$. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I am not sure if this makes sense but you could run a small &lt;a href=&quot;http://www.statisticshell.com/meta.pdf&quot; rel=&quot;nofollow&quot;&gt;meta-analysis&lt;/a&gt; (based on the Fisher's transformed correlations). This would give you a pooled standard error (see page 4). &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Whatever you do, your effect sizes (correlations) are not independent because each country is part of three bilateral correlations. Using dependent effect sizes will probably lead to a biased pooled standard error (which means that running a simple meta-analysis isn't such a good idea but you could do a &lt;a href=&quot;http://www.northwestern.edu/ipr/qcenter/RVE-meta-analysis.html&quot; rel=&quot;nofollow&quot;&gt;robust variance estimation&lt;/a&gt;).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I do not understand the last paragraph of your post/question: &quot;It would be great if I could create a confidence interval for each of these mean statistics and report it in one table so that the reader is able to compare across these regional correlations as they like.&quot; Can you give us an example?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="307" LastActivityDate="2011-05-24T10:38:39.853" CommentCount="1" />
  <row Id="11185" PostTypeId="2" ParentId="11149" CreationDate="2011-05-24T10:59:39.857" Score="0" ViewCount="" Body="&lt;p&gt;Missing data is one big issue everywhere. I wish you'd answer the following question first. 1) what %age of the data is missing ? -- if its more than 10% of the data you'd not risk imputing it with mean. Because imputing such missing with mean is equivalent to telling the LR box that look ..this variable has mean most of the places( so draw some conclusion) and you dont want LR box to draw conclusions upon your suggestions.do you ?? Now, the least you can do if you dont want much is you can try to relate this variables available values with different predictors value or use a business sense where ever possible..example..if I have a missing for marriage_ind , one of the ways could be seeing the median age of the people married, (lets say it comes out to be 29), I can assume that generally people(in India) get married by 30 and 29 suggests so. PROC MI also does thing internally for you but in a far more sofisticated way..so my 2 cents..see atleast 4-5 variables which are linked to your missings and try to form a correlation..This can be better than mean.&lt;/p&gt;&#xA;" OwnerUserId="1763" LastActivityDate="2011-05-24T10:59:39.857" />
  <row Id="11186" PostTypeId="1" CreationDate="2011-05-24T11:08:06.330" Score="1" ViewCount="52" Body="&lt;p&gt;Suppose we have  time-series $ X_t $  and it has the following decomposition&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$X_t=\mu + \varepsilon_t,$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\mu$ is a mean and $\varepsilon_t$ - the error term.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The model complexity will increase if we divide this time-series in to some segments,say $k$, and repeat above process. As the model complexity increases the approximation accuracy also increases. So I want to introduce a regularisation term here which will help in deciding the number of segments $k$ in which we need to divide the time-series. The error in approximation can be defined as &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ \epsilon_t= \frac{1}{k} \sum_{i=1}^k (\mu_{1}-X_{i})+\frac{1}{n-k} \sum_{i=n-k}^n (\mu_{2}-X_{i}), $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;here I have divided the time-series in 2 segments and $\mu_{1}, \mu_{2}$ are their respective means. Now I want to find out the optimal number of segments in general. Please note, that here I want to introduce a &quot;regularisation&quot; term which will help in deciding optimal number of segments.          &lt;/p&gt;&#xA;" OwnerUserId="3722" LastEditorUserId="3722" LastEditorDisplayName="" LastEditDate="2011-05-24T13:36:02.940" LastActivityDate="2011-05-24T13:36:02.940" Title="Number of segments to divide a time-series" Tags="&lt;time-series&gt;&lt;regularization&gt;&lt;change-point&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="11187" PostTypeId="2" ParentId="11186" CreationDate="2011-05-24T11:45:53.530" Score="1" ViewCount="" Body="&lt;p&gt;Seems that you have a &lt;a href=&quot;http://en.wikipedia.org/wiki/Structural_break&quot; rel=&quot;nofollow&quot;&gt;change point problem&lt;/a&gt;. Also look at &lt;a href=&quot;http://stats.stackexchange.com/questions/tagged/change-point&quot;&gt;change-point tag&lt;/a&gt; for related questions in this site. For fitting these type of models &lt;strong&gt;R&lt;/strong&gt; for example has the packages &lt;strong&gt;segmented&lt;/strong&gt; and &lt;strong&gt;strucchange&lt;/strong&gt;. The relevant function to find the optimal number of segments in package strucchange is &lt;code&gt;breakpoints&lt;/code&gt;. Here is the simple example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;e&amp;lt;-rnorm(100) ##errror term&#xA;&#xA;##the mean, value 10 for first 20 observations, &#xA;##then 30 for next 30 and 3 for last 50.&#xA;mu&amp;lt;-c(rep(10,20),rep(5,30),rep(3,50)) &#xA;&#xA;##generate time series X&#xA;x&amp;lt;-mu+e&#xA;&#xA;&amp;gt; breakpoints(x~1,data=data.frame(x=x))&#xA;&#xA;     Optimal 3-segment partition: &#xA;&#xA;Call:&#xA;breakpoints.formula(formula = y ~ 1, data = data.frame(y = y))&#xA;&#xA;Breakpoints at observation number:&#xA;20 50 &#xA;&#xA;Corresponding to breakdates:&#xA;0.2 0.5 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-05-24T11:45:53.530" CommentCount="5" />
  <row Id="11188" PostTypeId="1" CreationDate="2011-05-24T13:49:53.553" Score="0" ViewCount="26" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;Possible Duplicate:&lt;/strong&gt;&lt;br&gt;&#xA;  &lt;a href=&quot;http://stats.stackexchange.com/questions/2675/how-to-calculate-average-time-in-service&quot;&gt;How to calculate average time in service&lt;/a&gt;  &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&#xA;&#xA;&lt;p&gt;I run a subscription-based service, and I'm trying to refine my estimate for the lifetime value of a customer.  I can look at my user database and determine the average duration between signup and cancel, but that approach seems obviously flawed since it doesn't take into account users that are still active.  It seems like I should have enough data to run some more advanced statistics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, given a list of customers, their dates of signup, and (if applicable) their dates of cancellation, how would one go about predicting the expected subscription duration of a typical user?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance!&lt;/p&gt;&#xA;" OwnerUserId="4730" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-05-24T15:28:38.947" LastActivityDate="2011-05-24T15:28:38.947" ClosedDate="2011-05-24T15:30:04.577" Title="Predicting expected subscription durations based on past user data" Tags="&lt;survival&gt;&lt;expected-value&gt;" CommentCount="0" />
  <row Id="11189" PostTypeId="1" CreationDate="2011-05-24T14:02:48.443" Score="3" ViewCount="64" Body="&lt;p&gt;I received the following question by email:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;I was wondering should I use tick the&#xA;  option for pairwise exclusion of&#xA;  missing data when I carry out&#xA;  regression analyses (or any analyses&#xA;  for that matter) rather than using [some other missing values replacement strategy].&#xA;  Julie Pallant recommends pairwise&#xA;  exclusion of missing data in her SPSS&#xA;  textbook.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I have a few thoughts, but I was interested in first hearing your thoughts.&lt;/p&gt;&#xA;" OwnerUserId="183" LastActivityDate="2011-05-24T20:41:41.547" Title="When, if ever, to use pairwise deletion in multiple regression?" Tags="&lt;regression&gt;&lt;missing-data&gt;" AnswerCount="2" />
  <row Id="11190" PostTypeId="1" CreationDate="2011-05-24T14:08:15.747" Score="1" ViewCount="51" Body="&lt;p&gt;I am working on 4 different species of tomatoes. From the data I had, I looked at the occurrence of a particular &quot;event&quot; in certain intervals of their genome (this interval is identical in all 4 plants) and I have a file for each of the species with their probability of occurrence. The file looks something like this:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;ch01:a1-b1&lt;br&gt;&#xA;  p = 0.45&lt;br&gt;&#xA;  ch01:a2-b2&lt;br&gt;&#xA;  p = 0.005&lt;br&gt;&#xA;  ...&lt;br&gt;&#xA;  ...  &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I have 4 such files (one for each species) where the chromosomal locations/intervals are identical and the probability value differs. Now I would like to find out those intervals where the occurrence of this event stands out.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What do I mean by that? If I find, for example, over a particular interval, the event is more significant in plant/file 1 and 3, but not the others, then this interval is of interest to me. I would like to find all such intervals where &quot;the occurrence of event is not all &lt;code&gt;equally&lt;/code&gt; significant/insignificant&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is how should I go about this? Which parametric/non-parametric test can I use? Can I use the test on each interval separately? What are the usual issues to look for and overcome.&lt;/p&gt;&#xA;" OwnerUserId="4731" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-24T14:15:12.237" LastActivityDate="2011-05-24T14:15:12.237" Title="R: statistical test" Tags="&lt;r&gt;" />
  <row Id="11191" PostTypeId="1" AcceptedAnswerId="11197" CreationDate="2011-05-24T14:12:45.140" Score="5" ViewCount="48" Body="&lt;p&gt;This question came up in a consulting context, and I was interested in your thoughts.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Context&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;One strategy for dealing with occasional missing data when calculating scale means looks like this in the language of SPSS:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;COMPUTE depmean =  mean.4(dep1, dep2, dep3, dep4, dep5, dep6).&#xA;EXECUTE.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I.e., calculate the mean of a psychological scale such as depression by taking the mean of six items. &#xA;If a participant has four or more non-missing items, return the mean of the non-missing items. &#xA;If the participant has three or fewer non-missing items, return missing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course the number of items in the scale and the threshold number items for calculating the mean can vary.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Question&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;In general, under what conditions, would you see this method of dealing with missing data to be appropriate?&lt;/li&gt;&#xA;&lt;li&gt;If you perceive it to be inappropriate, what alternative procedure would you recommend?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-25T06:43:02.630" LastActivityDate="2011-05-25T13:31:15.813" Title="Appropriateness of calculating scale means based on available non-missing responses (i.e., person-mean imputation)" Tags="&lt;scales&gt;&lt;missing-data&gt;" AnswerCount="3" FavoriteCount="1" />
  <row Id="11193" PostTypeId="1" AcceptedAnswerId="11224" CreationDate="2011-05-24T14:23:45.017" Score="4" ViewCount="119" Body="&lt;p&gt;I have a data frame that contains some duplicate ids.  I want to remove records with duplicate ids, keeping only the row with the maximum value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So for structured like this (other variables not shown):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;id var_1&#xA;1 2&#xA;1 4&#xA;2 1&#xA;2 3&#xA;3 5&#xA;4 2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to generate this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;id var_1&#xA;1 4&#xA;2 3&#xA;3 5&#xA;4 2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I know about unique() and duplicated(), but I can't figure out how to incorporate the maximization rule...&lt;/p&gt;&#xA;" OwnerUserId="4110" LastActivityDate="2011-05-25T19:40:10.713" Title="How do I remove all but one specific duplicate record in an R data frame?" Tags="&lt;r&gt;&lt;dataframe&gt;" AnswerCount="5" FavoriteCount="2" />
  <row Id="11194" PostTypeId="2" ParentId="11193" CreationDate="2011-05-24T14:33:45.407" Score="5" ViewCount="" Body="&lt;p&gt;You actualy want to select the maximum element from the elements with the same id. For that you can use &lt;code&gt;ddply&lt;/code&gt; from package &lt;strong&gt;plyr&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; dt&amp;lt;-data.frame(id=c(1,1,2,2,3,4),var=c(2,4,1,3,4,2))&#xA;&amp;gt; ddply(dt,.(id),summarise,var_1=max(var))&#xA;   id var_1&#xA;1  1   4&#xA;2  2   3&#xA;3  3   4&#xA;4  4   2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;unique&lt;/code&gt; and &lt;code&gt;duplicated&lt;/code&gt; is for removing duplicate records, in your case you only have duplicate ids, not records.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update:&lt;/strong&gt; Here is the code when there are additional variables:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; dt&amp;lt;-data.frame(id=c(1,1,2,2,3,4),var=c(2,4,1,3,4,2),bu=rnorm(6))&#xA;&amp;gt; ddply(dt,~id,function(d)d[which.max(d$var),])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-24T19:43:38.453" LastActivityDate="2011-05-24T19:43:38.453" CommentCount="3" />
  <row Id="11195" PostTypeId="2" ParentId="10111" CreationDate="2011-05-24T14:34:40.337" Score="1" ViewCount="" Body="&lt;p&gt;The eta-square ($\eta^2$) value you are describing is intended to be used as a measure of &lt;em&gt;effect size&lt;/em&gt; in the observed data (i.e., your sample), as it amounts to quantify how much of the total variance can be explained by the factor considered in the analysis (that is what you wrote in fact, BSS/TSS). With more than one factor, you can also compute partial $\eta^2$ that reflect the percentage of variance explained by one factor when holding constant the remaining ones.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The F-ratio (BSS/WSS) is the right &lt;em&gt;test statistic&lt;/em&gt; to use if you want to test the null hypothesis ($H_0$) that there is no effect of your factor (all group means are equal), that is your factor of interest doesn't account for a sufficient amount of variance compared to the residual (unexplained) variance. In other words, we test whether the added explained variance (BSS=TSS-RSS) is large enough to be considered as a &quot;significant quantity&quot;. The distribution of the ratio of these two sum of squares (scaled by their corresponding degrees of freedom--this answers one of your question, about why we don't use directly SSs), which individually follow a $\chi^2$ distribution, is known as the &lt;a href=&quot;http://en.wikipedia.org/wiki/F-distribution&quot; rel=&quot;nofollow&quot;&gt;Fisher-Snedecor&lt;/a&gt; distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't know which software you are using, but &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;If you have R, everything you need for basic modeling is given in the &lt;code&gt;aov()&lt;/code&gt; base function ($\eta^2$ might be computed with &lt;code&gt;etasq&lt;/code&gt; from the &lt;a href=&quot;http://cran.r-project.org/web/packages/heplots/index.html&quot; rel=&quot;nofollow&quot;&gt;heplots&lt;/a&gt; package; and there's a lot more to see for diagnostics and plotting in other packages).&lt;/li&gt;&#xA;&lt;li&gt;If you're more versed into C programming, you may have a look at the &lt;a href=&quot;http://apophenia.sf.net/&quot; rel=&quot;nofollow&quot;&gt;apophenia&lt;/a&gt; library which features a nice set of statistical functions with bindings for MySQL and Python.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-05-24T14:34:40.337" CommentCount="1" />
  <row Id="11196" PostTypeId="2" ParentId="9671" CreationDate="2011-05-24T15:05:40.330" Score="2" ViewCount="" Body="&lt;p&gt;One generally consider that a &quot;good partitioning&quot; must satisfy one or more of the following criteria: (a) &lt;em&gt;compactness&lt;/em&gt; (small within-cluster variation), &lt;em&gt;connectedness&lt;/em&gt; (neighbouring data belong to the same cluster), and &lt;em&gt;spatial separation&lt;/em&gt; (must be combined with other criteria like compactness or balance of cluster sizes). As part of a large battery of internal measures of cluster validity (where we do not use additional knowledge about the data, like some a priori on class labeling), they can be complemented with so-called &lt;em&gt;combination measures&lt;/em&gt; (for example, assessing intra-cluster homogeneity and inter-cluster separation), like Dunn or Davies–Bouldin index, silhouette width, SD-validity index, etc., but also estimates of predictive power (self-consistency and stability of a partitioning), how well distance information are reproduced in the resulting partitions (e.g., cophenetic correlation and Hubert's Gamma statistic). &#xA;A more complete review, and simulation results, are available in &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Handl, J., Knowles, J., and Kell, D.B.&#xA;  (2005). &lt;a href=&quot;http://bioinformatics.oxfordjournals.org/content/21/15/3201.full&quot; rel=&quot;nofollow&quot;&gt;Computational cluster&#xA;  validation in post-genomic data&#xA;  analysis&lt;/a&gt;. &lt;em&gt;Bioinformatics&lt;/em&gt;,&#xA;  21(15): 3201-3212.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I guess you could rely on some of them for comparing your different cluster solutions and choose the features set that yields the better indices. You can even use bootstrap to get an estimate of the variability of those indices (e.g., cophenetic correlation, Dunn's index, silhouette width), as was done by Tom Nichols and coll. in a neuroimaging study, &lt;a href=&quot;http://www2.warwick.ac.uk/fac/sci/statistics/staff/academic-research/nichols/presentations/ohbm2010/nichols-ohbm2010-CoheritabilityClustering.pdf&quot; rel=&quot;nofollow&quot;&gt;Finding Distinct Genetic Factors that Influence Cortical Thickness&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are using R, I warmly recommend taking a look at the &lt;a href=&quot;http://cran.r-project.org/web/packages/fpc/index.html&quot; rel=&quot;nofollow&quot;&gt;fpc&lt;/a&gt; package, by &lt;a href=&quot;http://www.homepages.ucl.ac.uk/~ucakche/&quot; rel=&quot;nofollow&quot;&gt;Christian Hennig&lt;/a&gt;, which provides almost all statistical indices described above (&lt;code&gt;cluster.stats()&lt;/code&gt;) as well as a bootstrap procedure (&lt;code&gt;clusterboot()&lt;/code&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;About the use of mutual information in clustering, I have no experience with it but here is a paper that discusses its use in a genomic context (with comparison to k-means):&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Priness, I., Maimon, O., and Ben-Gal,&#xA;  I. (2007). &lt;a href=&quot;http://www.biomedcentral.com/1471-2105/8/111&quot; rel=&quot;nofollow&quot;&gt;Evaluation of&#xA;  gene-expression clustering via mutual&#xA;  information distance measure&lt;/a&gt;. &lt;em&gt;BMC&#xA;  Bioinformatics&lt;/em&gt;, 8: 111.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-05-24T15:05:40.330" />
  <row Id="11197" PostTypeId="2" ParentId="11191" CreationDate="2011-05-24T15:05:55.980" Score="5" ViewCount="" Body="&lt;p&gt;Some years ago, I thought it might be a good idea to apply person-mean imputation (person-mean substitution or case-mean imputation) in case of item non-response. Nowadays, however, it seems obvious to me that this approach assumes that all scale items share similar characteristics (similar variance, standard deviation, item difficulty, etc.). In other words, I would be concerned if some respondents do not answer difficult/sensitive/... items. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.sciencedirect.com/science/article/pii/S155174110600043X&quot; rel=&quot;nofollow&quot;&gt;Bono et al (2007: 7)&lt;/a&gt; are less concerned about this approach: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&quot;Person-mean imputation requires&#xA;  substitution of the mean of all of an&#xA;  individual’s completed items for those&#xA;  items that were not completed on a&#xA;  given scale. This differs from&#xA;  item-mean where the mean response of&#xA;  the whole sample that responded to the&#xA;  item is substituted. Person-mean&#xA;  imputation could result in different&#xA;  substitutions for each person with&#xA;  missing items. On the plus side,&#xA;  because it does not substitute a&#xA;  constant value, it does not&#xA;  artificially reduce the measure’s&#xA;  variability and is less likely to&#xA;  attenuate the correlation. A&#xA;  disadvantage is that it tends to&#xA;  inflate the reliability estimates as&#xA;  the number of missing items increases.&#xA;  However, when the numbers of either&#xA;  respondents with missing items or&#xA;  items missing within scales are 20% or&#xA;  less, both item-mean imputation and&#xA;  person-mean imputation provide good&#xA;  estimates of the reliability of&#xA;  measures.&quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;You also might want to check &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Craig K. Enders (2010): &lt;a href=&quot;http://books.google.com/books?id=MN8ruJd2tvgC&amp;amp;lpg=PA51&amp;amp;ots=dHfCpWZcoa&amp;amp;dq=%22person-mean%20imputation%22&amp;amp;hl=de&amp;amp;pg=PA51#v=onepage&amp;amp;q=%22person-mean%20imputation%22&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;Applied Missing Data Analysis&lt;/a&gt;. (Google books link)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Downey RG, King C. (1998): &lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pubmed/9935342&quot; rel=&quot;nofollow&quot;&gt;Missing data in Likert ratings: A comparison of replacement methods&lt;/a&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="307" LastEditorUserId="307" LastEditorDisplayName="" LastEditDate="2011-05-24T15:20:22.497" LastActivityDate="2011-05-24T15:20:22.497" />
  <row Id="11198" PostTypeId="2" ParentId="11189" CreationDate="2011-05-24T15:25:33.190" Score="1" ViewCount="" Body="&lt;p&gt;I think it depends on the situation at hand.  If you're missing a couple values out of several hundred or thousand observations, sure, delete them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If one of your important variables is 10% missing, you may need to think up a strategy for dealing with this.&lt;/p&gt;&#xA;" OwnerUserId="2817" LastActivityDate="2011-05-24T15:25:33.190" />
  <row Id="11199" PostTypeId="2" ParentId="11193" CreationDate="2011-05-24T15:35:27.920" Score="6" ViewCount="" Body="&lt;p&gt;The base-R solution would involve &lt;code&gt;split&lt;/code&gt;, like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;z&amp;lt;-data.frame(id=c(1,1,2,2,3,4),var=c(2,4,1,3,4,2))&#xA;do.call(rbind,lapply(split(z,z$id),function(chunk) chunk[which.max(chunk$var),]))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;split&lt;/code&gt; splits the data frame into a list of chunks, on which we perform cutting to the single row with max value and then &lt;code&gt;do.call(rbind,...)&lt;/code&gt; reduces the list of single rows into a data frame again.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2011-05-24T15:35:27.920" CommentCount="7" />
  <row Id="11200" PostTypeId="1" AcceptedAnswerId="11201" CreationDate="2011-05-24T15:53:05.780" Score="3" ViewCount="45" Body="&lt;p&gt;Today I opened two STATA windows and ran the following command in both:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set obs 100&#xA;gen x = rnormal()&#xA;sort x&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(the difference is that on the second window I generated a variable called y).  Summing up: I asked STATA to give me 100 pseudo-random numbers taken from a standard normal distribution, then I sorted it.  To my surprise, the numbers of the x and y vectors are the same!  I did this at home, and then at work, and my impression is that all of these vectors are the same.  Is there an explanation for this, to me, strange behavior?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If this is a problem in STATA, does R have a better pseudo-random number generator procedure?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A side question.  I came up to this &quot;problem&quot; because I was trying to generate two pseudo-random columns in Stata (x and y, say), and then sort then separately.  But the two commands I know for sorting (sort and gsort) sort the whole database, not separate columns.  Would you know of a Stata command that allows me to sort a column while keeping the other columns fixed?&lt;/p&gt;&#xA;" OwnerUserId="2929" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-05-24T16:07:17.587" LastActivityDate="2011-05-24T16:07:17.587" Title="Generating sorted pseudo-random numbers in Stata" Tags="&lt;stata&gt;&lt;random-generation&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="11201" PostTypeId="2" ParentId="11200" CreationDate="2011-05-24T16:05:08.697" Score="6" ViewCount="" Body="&lt;p&gt;The help for &lt;code&gt;set_seed&lt;/code&gt; states&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The sequences these functions produce are determined by the seed, which is just a number and which is set to 123456789 every time Stata is launched. &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Stata's philosophy emphasizes reproducibility, so this consistency is not surprising.  Of course you can set the seed yourself.  See the help page for more information.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One way to sort a column separately from all others is to preserve your data, keep only the column to sort, sort it, save the results in a temporary file, restore your data, and merge the temporary file:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;gen y = rnormal()&#xA;preserve&#xA;keep y&#xA;sort y&#xA;tempfile out&#xA;save `out'&#xA;restore&#xA;merge 1:1 _n using `out', nogen&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-05-24T16:05:08.697" CommentCount="2" />
  <row Id="11202" PostTypeId="1" AcceptedAnswerId="11205" CreationDate="2011-05-24T16:39:54.080" Score="1" ViewCount="35" Body="&lt;p&gt;Hello&#xA;I am trying to forecast using different exponential smoothing methods(Linear and Winter's). For the optimal parameters, I am getting negative values of the forecasats. &#xA;I am assuming it means that the values will be zero, since it is a sales forecast. &#xA;I wanted to know if negative values denote something wrong with the model or is it possible to have negative values in forecast. Since it is just a model, I think we can get negative values for certain type of time series.&lt;/p&gt;&#xA;" OwnerUserId="4445" LastActivityDate="2011-05-24T17:15:30.827" Title="Can the forecasts using exponential smoothing be negative in value?" Tags="&lt;forecasting&gt;" AnswerCount="2" />
  <row Id="11203" PostTypeId="1" AcceptedAnswerId="11228" CreationDate="2011-05-24T16:46:48.120" Score="3" ViewCount="49" Body="&lt;p&gt;Hi&#xA;I am using Linear and exponential forecasting models to do sales forecasting. In the model itself, we use the forecasts of period t to get next forecast and so on.&#xA;While analyzing the accuracy of the forecast using Mean Absolute Percentage Error, I get good results. But when I compare the intermediate forecast values of the model against the actual time series data, I can see some big Percentage Errors like MAPE might be 15%, but some of the intermediate percentage errors might be as high as 80% and some will be pretty low. So the MAPE averages out to low.&#xA;I wanted to know &#xA;1) Is it wise to compare these intermediate forecasts with actual values?&#xA;2) Can we use these high intermediate forecast values to say that forecasts for so and so month might be unreliable?&lt;/p&gt;&#xA;" OwnerUserId="4445" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-25T06:53:45.830" LastActivityDate="2011-05-25T07:54:30.247" Title="Should we compare the individual monthly forecasts with actual values?" Tags="&lt;time-series&gt;&lt;forecasting&gt;" AnswerCount="2" />
  <row Id="11204" PostTypeId="1" CreationDate="2011-05-24T11:46:21.430" Score="0" ViewCount="43" Body="&lt;p&gt;Hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have 4 files, each containing the observations for a different plant. The observations are the probability of occurrence of a particular &quot;event&quot; over many intervals over the plant's genome.&#xA;That is, the observation file looks like this:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;file/plant 1:&lt;br&gt;&#xA;  ch01:a1-b1&lt;br&gt;&#xA;  p = 0.41&lt;br&gt;&#xA;  ch01:a2-b2&lt;br&gt;&#xA;  p = 0.005&lt;br&gt;&#xA;  ...&lt;br&gt;&#xA;  ...&lt;br&gt;&#xA;  ...  &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;And in the same format, over the same intervals, I have the probability of occurrence of this event for other 3 plants/files. There are a total of 15,000 observations in each file over identical intervals of chromosomal locations. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am interested in finding those intervals where the event is &quot;significant&quot; in 1 or more plants, but NOT ALL; i.e., events that stand out (or different among the plants). If the event is equally likely in all 4, then it of course is not interesting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, it would be great if I could get some insight on how to go about this? The things I have managed to find out so far are this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) I could perform a two-sample t-test on all pairs, but maybe it will be biased (?) &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;So I decided to skip this. &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;2) I could perform a F-test, but this is (also) a parametric and so I will have to find out if the distribution of residuals is normal (I do not know why the residuals have to be normally distributed, but was told to do this. So it would be nice if someone could explain the logic behind this or share a link that explains it).&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;So, I decided to find out the distribution. But I am not sure how. I use R. If I understand it right, for each of the chromosomal interval (or observation) over all 4 plants, lets say, for example, for observation ch01:a1-b1, p = (0.004, 0.33, 0.19, 0.021).&lt;br&gt;&#xA;  Then my understanding is that, the residuals of these values must be normally distributed. If not, a parametric test is not possible... I am not used to R. I was told to use the &quot;lm&quot; syntax followed by &quot;anova&quot;. I would like to know if my understanding and this approach is right? If so, how do I find the distribution of the residuals? And if this is the way, then I have to find the distribution for all 15000 observations??&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;3) If this isn't the method to proceed, then I guess a non-parametric method is the way to go? In this case, which test should I adopt to test the significance of each event over all 4-plants. I guess its multiple (hypothesis) testing??&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you very much!&lt;/p&gt;&#xA;" OwnerUserId="4751" LastActivityDate="2011-05-24T16:54:48.647" Title="R: statistical test and residuals" Tags="&lt;r&gt;" CommentCount="3" />
  <row Id="11205" PostTypeId="2" ParentId="11202" CreationDate="2011-05-24T17:14:00.257" Score="4" ViewCount="" Body="&lt;p&gt;Holt's or Winter-Holt's exponential smoothing methods can give negative values for purely non-negative input values because of the &lt;em&gt;trend factor&lt;/em&gt; which acts as a kind of inertia, which can drive the time series below zero. Normal exponential smoothing doesn't have this problem, it's always smoothing &lt;em&gt;inwards&lt;/em&gt;, it never overshoots.&lt;/p&gt;&#xA;" OwnerUserId="4360" LastActivityDate="2011-05-24T17:14:00.257" />
  <row Id="11206" PostTypeId="2" ParentId="11202" CreationDate="2011-05-24T17:15:30.827" Score="0" ViewCount="" Body="&lt;p&gt;You can get negative values for certain kinds of models.  You might want to explore more complicated models than simple exponential smoothing.&lt;/p&gt;&#xA;" OwnerUserId="2817" LastActivityDate="2011-05-24T17:15:30.827" />
  <row Id="11207" PostTypeId="2" ParentId="11203" CreationDate="2011-05-24T17:20:16.913" Score="5" ViewCount="" Body="&lt;p&gt;Yes, you should absolutely compare your predicted values with actual values.  This is good practice with any kind of statistical modeling, not just time series analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If certain months are consistently off, you should use a seasonal model.&lt;/p&gt;&#xA;" OwnerUserId="2817" LastActivityDate="2011-05-24T17:20:16.913" />
  <row Id="11208" PostTypeId="1" CreationDate="2011-05-24T18:19:51.373" Score="5" ViewCount="74" Body="&lt;p&gt;Problem is that government wants to close electronic roulette and they claim that roulette failed at statistical test.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sorry for my language but this is translated from Slovenian law as good as possible&#xA;Official (by law) requirements are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;frequency of each event should not differ from expected frequency by more than 3 sigma&lt;/li&gt;&#xA;&lt;li&gt;chi square test of normal distribution has to be within risk level of 0.025&lt;/li&gt;&#xA;&lt;li&gt;test of consecutive correlation has to pass 3 sigma test and chi squared test&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I have tested first 2 requirements and they pass tests, but I have problems whith understanding 3rd requirement. (keep in mind that this is translated and &quot;consecutive correlation&quot; can be something else)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How should I test 3rd requirement?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data if somebody is interested:&lt;br&gt;&#xA;&lt;a href=&quot;http://pastebin.com/ffbSKpr1&quot; rel=&quot;nofollow&quot;&gt;http://pastebin.com/ffbSKpr1&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT:&#xA;chi squared fails 2% of the time (what I expect that is expected due to fact that alpha is 0.025) and sigma3 test fails 5%  where i expect 9% failure for 3sigma (it looks like that frequencies are not distributed according to normal distribution even for random numbers)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I might not understand this law correctly, but it is almost 0% probability to pass 3sigma test for all autocorrelation vectors, since it is 9% probability to fail in single run and 2.5 for chi squared test.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Python code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from math import sqrt&#xA;from itertools import *&#xA;import random&#xA;&#xA;#uncoment for python 2.x &#xA;    #zip = izip &#xA;    #range = xrange&#xA;&#xA;&#xA;#with open(&quot;rng.txt&quot;,&quot;r&quot;) as wr:&#xA;#   n = [int(i) for i in wr]&#xA;n = [random.randint(0,36) for i in range(44000)] &#xA;&#xA;&#xA;def get_freq(n):&#xA;    r=[0 for i in range(37)]&#xA;    for i in n:&#xA;        r[i] += 1&#xA;    return r&#xA;&#xA;def trisigmatest(freq):&#xA;    Ef = 1.0*sum(freq)/37&#xA;    sigma = sqrt(sum(i**2 for i in freq)/37-Ef**2)&#xA;    return all((abs(i - Ef )&amp;lt; sigma*3) for i in freq)&#xA;&#xA;&#xA;def chiquaretest(freq):&#xA;    Ef = 1.0*sum(freq)/37&#xA;    chi2 = sum((i-Ef)**2 / Ef for i in freq)&#xA;    # values are from http://itl.nist.gov/div898/handbook/eda/section3/eda3674.htm&#xA;    # (EDIT) I recaluclated these valuse from inverse cdf chi2 &#xA;    # distribution for interval (0.025/2,1-0.025/2) (alpha = 0.025)&#xA;    return 20.4441 &amp;lt; chi2 &amp;lt; 58.8954   &#xA;&#xA;&#xA;#whitout autocorelation  &#xA;gf = get_freq(n)&#xA;if not trisigmatest(gf):&#xA;    print(&quot;failed&quot;)&#xA;    raise&#xA;if not chiquaretest(gf):&#xA;    print(&quot;failed&quot;)&#xA;    raise&#xA;&#xA;&#xA;actests = 1000&#xA;trifailed = 0;&#xA;chifailed = 0;&#xA;for i in range(1,actests + 1):&#xA;    f=((b-a+37) % 37 for (a,b) in zip(n,n[i:]))&#xA;    gf = get_freq(f)&#xA;    if not trisigmatest(gf):&#xA;        trifailed += 1;&#xA;    if not chiquaretest(gf):&#xA;        chifailed += 1;&#xA;print(&quot;trisigmatest failed &quot;, 1.0 * trifailed / actests )&#xA;print(&quot;chiquaretest failed &quot;, 1.0 * chifailed / actests )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="4738" LastEditorUserId="4738" LastEditorDisplayName="" LastEditDate="2011-05-25T06:18:33.933" LastActivityDate="2011-05-25T06:18:33.933" Title="Statistics for gambling machine validation" Tags="&lt;correlation&gt;&lt;statistical-significance&gt;&lt;chi-squared&gt;" CommentCount="3" />
  <row Id="11209" PostTypeId="1" AcceptedAnswerId="11230" CreationDate="2011-05-24T19:12:20.207" Score="2" ViewCount="86" Body="&lt;p&gt;Hi,&#xA;I have written 3way-anova in c++. I have 3 factors, lets say a,b and c and my aim to check the strength of all possible interactions and main effects.The result of my code is the same as MATLAB when I use type I sum of squares.&lt;br&gt;&#xA;But when I change the data so that the number of replicates is so high in some cells and too low in others: I don't get the same results as MATLAB. (to be precise, only SSt, SSe and SSa are the same as MATLAB) .&#xA;My question is that is it possible that since I have a large difference in the number of replicates, I should use type III sum of squares? Or is there is special way that matlab treats the data in such cases so its results differs from mine?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for your help. &#xA;Pegah&lt;/p&gt;&#xA;" OwnerUserId="2885" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-24T22:34:46.483" LastActivityDate="2011-05-25T08:56:55.523" Title="The effect of the number of replicates in different cells on the results of ANOVA" Tags="&lt;anova&gt;&lt;matlab&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="2" />
  <row Id="11210" PostTypeId="1" AcceptedAnswerId="11217" CreationDate="2011-05-24T19:53:26.753" Score="9" ViewCount="105" Body="&lt;p&gt;I appreciate the usefulness of the bootstrap in obtaining uncertainty estimates, but one thing that's always bothered me about it is that the distribution corresponding to those estimates is the distribution defined by the sample. In general, it seems like a bad idea to believe that our sample frequencies look exactly like the underlying distribution, so why is it sound/acceptable to derive uncertainty estimates based on a distribution where the sample frequencies define the underlying distribution?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, this may be no worse (possibly better) than other distributional assumptions we typically make, but I'd still like to understand the justification a bit better.&lt;/p&gt;&#xA;" OwnerUserId="4733" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-24T22:34:07.543" LastActivityDate="2011-05-24T23:06:16.357" Title="Assumptions regarding bootstrap estimates of uncertainty" Tags="&lt;bootstrap&gt;&lt;uncertainty&gt;" AnswerCount="4" CommentCount="1" FavoriteCount="2" />
  <row Id="11211" PostTypeId="2" ParentId="9867" CreationDate="2011-05-24T20:07:50.003" Score="1" ViewCount="" Body="&lt;p&gt;I found this paper with Google but I cannot access it, so I don't really know what it is about really:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Berry KJ, Johnston JE, Mielke PW Jr.&#xA;  &lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pubmed/17879655&quot; rel=&quot;nofollow&quot;&gt;An alternative measure of effect size&#xA;  for Cochran's Q test for related&#xA;  proportions&lt;/a&gt;. Percept Mot Skills.&#xA;  2007 Jun;104(3 Pt 2):1236-42.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I initially thought that using pairwise multiple comparisons with Cochran or McNemar test* (if the overall test is significant) would give you further indication of where the differences lie, while reporting simple difference for your binary outcome would help asserting the magnitude of the observed difference. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;sub&gt;* I found an &lt;a href=&quot;http://yatani.jp/HCIstats/Cochran&quot; rel=&quot;nofollow&quot;&gt;online tutorial with R&lt;/a&gt;.&lt;/sub&gt;&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-05-24T20:07:50.003" CommentCount="2" />
  <row Id="11212" PostTypeId="2" ParentId="11189" CreationDate="2011-05-24T20:41:41.547" Score="3" ViewCount="" Body="&lt;p&gt;Pairwise is a dangerous method in this case, IMO. If you delete pairwise then you'll end up with different numbers of observations contributing to different parts of your model, which can make interpretation difficult.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That being said, casewise deletion tends to discard lots and lots of information, so I suppose it depends on both the proportion of missing responses, and your sample size.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Personally, I would probably use the multiple imputation procedure in SPSS and run the analyses for each dataset, then combine if nothing looks odd.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This would be my strategy of choice with a high proportion of missing values, whereas if the number is small, case-wise would probably be my first choice. &lt;/p&gt;&#xA;" OwnerUserId="656" LastActivityDate="2011-05-24T20:41:41.547" />
  <row Id="11213" PostTypeId="2" ParentId="11210" CreationDate="2011-05-24T21:01:51.687" Score="2" ViewCount="" Body="&lt;p&gt;The main trick (and sting) of bootstrapping is that it is an asymptotic theory: if you have an infinite sample to start with, the empirical distribution is going to be so close to the actual distribution that the difference is negligible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unfortunately, bootstrapping is often applied in small sample sizes. The common feel is that bootstrapping has shown itself to work in some very non-asymptotic situations, but be careful nonetheless. If your samplesize is too small, you are in fact working conditionally on your sample being a 'good representation' of the true distribution, which leads very easily to reasoning in circles :-)&lt;/p&gt;&#xA;" OwnerUserId="4257" LastActivityDate="2011-05-24T21:01:51.687" CommentCount="2" />
  <row Id="11214" PostTypeId="2" ParentId="11157" CreationDate="2011-05-24T21:34:01.523" Score="0" ViewCount="" Body="&lt;p&gt;Maybe use an adaptation of &lt;a href=&quot;http://www.investopedia.com/articles/technical/04/060204.asp&quot; rel=&quot;nofollow&quot;&gt;J-Charts&lt;/a&gt; and/or &lt;a href=&quot;http://daytrading.about.com/od/indicators/a/MarketProfile.htm&quot; rel=&quot;nofollow&quot;&gt;Market Profile charts&lt;/a&gt;, but instead of plotting price (y-axis) vs volume (x-axis) you could plot time of trade (y-axis) vs no. of trades (x-axis) and use colours to delineate different trading days or averages of no. of trades at these times for different look back periods.&lt;/p&gt;&#xA;" OwnerUserId="226" LastActivityDate="2011-05-24T21:34:01.523" />
  <row Id="11215" PostTypeId="2" ParentId="11210" CreationDate="2011-05-24T21:55:49.077" Score="2" ViewCount="" Body="&lt;p&gt;I would argue not from the perspective of &quot;asymptotically, the empirical distribution will be close to the actual distribution&quot; (which, of course, is very true), but from a &quot;long run perspective&quot;. In other words, in any particular case, the empirical distribution derived by bootstrapping will be off (sometimes shifted too far this way, sometimes shifted too far that way, sometimes too skewed this way, sometimes too skewed that way), but &lt;strong&gt;on average&lt;/strong&gt; it will be a good approximation to the actual distribution. Similarly, your uncertainty estimates derived from the bootstrap distribution will be off in any particular case, but again, on average, they will be (approximately) right.&lt;/p&gt;&#xA;" OwnerUserId="1934" LastEditorUserId="1934" LastEditorDisplayName="" LastEditDate="2011-05-24T22:30:27.953" LastActivityDate="2011-05-24T22:30:27.953" />
  <row Id="11216" PostTypeId="2" ParentId="11193" CreationDate="2011-05-24T22:39:14.203" Score="5" ViewCount="" Body="&lt;p&gt;I prefer using &lt;code&gt;ave&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dt&amp;lt;-data.frame(id=c(1,1,2,2,3,4),var=c(2,4,3,3,4,2))&#xA;## use unique if you want to exclude duplicate maxima&#xA;unique(subset(dt, var==ave(var, id, FUN=max)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="375" LastActivityDate="2011-05-24T22:39:14.203" CommentCount="1" />
  <row Id="11217" PostTypeId="2" ParentId="11210" CreationDate="2011-05-24T22:48:41.360" Score="7" ViewCount="" Body="&lt;p&gt;There are several ways that one can conceivably apply the bootstrap. The two most basic approaches are what are deemed the &quot;nonparametric&quot; and &quot;parametric&quot; bootstrap. The second one assumes that the model you're using is (essentially) correct.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's focus on the first one. We'll assume that you have a random sample $X_1, X_2, \ldots, X_n$ distributed according the the distribution function $F$. (Assuming otherwise requires modified approaches.) Let $\hat{F}_n(x) = n^{-1} \sum_{i=1}^n \mathbf{1}(X_i \leq x)$ be the empirical cumulative distribution function. Much of the motivation for the bootstrap comes from a couple of facts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Dvoretzky&amp;ndash;Kiefer&amp;ndash;Wolfowitz inequality&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;\renewcommand{\Pr}{\mathbb{P}}&#xD;&#xA;\Pr\big( \textstyle\sup_{x \in \mathbb{R}} \,|\hat{F}_n(x) - F(x)| &amp;gt; \varepsilon \big) \leq 2 e^{-2n \varepsilon^2} \&amp;gt; .&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What this shows is that the empirical distribution function converges &lt;em&gt;uniformly&lt;/em&gt; to the true distribution function &lt;em&gt;exponentially fast&lt;/em&gt; in probability. Indeed, this inequality coupled with the Borel&amp;ndash;Cantelli lemma shows immediately that $\sup_{x \in \mathbb{R}} \,|\hat{F}_n(x) - F(x)| \to 0$ almost surely.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are no additional conditions on the form of $F$ in order to guarantee this convergence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Heuristically, then, if we are interested in some functional $T(F)$ of the distribution function that is &lt;em&gt;smooth&lt;/em&gt;, then we expect $T(\hat{F}_n)$ to be close to $T(F)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;(Pointwise) Unbiasedness of $\hat{F}_n(x)$&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By simple linearity of expectation and the definition of $\hat{F}_n(x)$, for each $x \in \mathbb{R}$,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;\newcommand{\e}{\mathbb{E}}&#xD;&#xA;\e_F \hat{F}_n(x) = F(x) \&amp;gt;.&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose we are interested in the mean $\mu = T(F)$. Then the unbiasedness of the empirical measure extends to the unbiasedness of linear functionals of the empirical measure. So,&#xA;$$&#xD;&#xA;\e_F T(\hat{F}_n) = \e_F \bar{X}_n = \mu = T(F) \&amp;gt; .&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So $T(\hat{F}_n)$ is correct on average and since $\hat{F_n}$ is rapidly approaching $F$, then (heuristically), $T(\hat{F}_n)$ rapidly approaches $T(F)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To construct a confidence interval (&lt;em&gt;which is, essentially, what the bootstrap is all about&lt;/em&gt;), we can use the central limit theorem, the consistency of empirical quantiles and the delta method as tools to move from simple linear functionals to more complicated statistics of interest.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good references are&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;B. Efron, &lt;a href=&quot;http://projecteuclid.org/euclid.aos/1176344552&quot; rel=&quot;nofollow&quot;&gt;Bootstrap methods: Another look at the jackknife&lt;/a&gt;, &lt;em&gt;Ann. Stat.&lt;/em&gt;, vol. 7, no. 1, 1&amp;ndash;26.&lt;/li&gt;&#xA;&lt;li&gt;B. Efron and R. Tibshirani, &lt;em&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0412042312&quot; rel=&quot;nofollow&quot;&gt;An Introduction to the Bootstrap&lt;/a&gt;&lt;/em&gt;, Chapman&amp;ndash;Hall, 1994.&lt;/li&gt;&#xA;&lt;li&gt;G. A. Young and R. L. Smith, &lt;em&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0521548667&quot; rel=&quot;nofollow&quot;&gt;Essentials of Statistical Inference&lt;/a&gt;&lt;/em&gt;, Cambridge University Press, 2005, &lt;strong&gt;Chapter 11&lt;/strong&gt;.&lt;/li&gt;&#xA;&lt;li&gt;A. W. van der Vaart, &lt;em&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0521784506&quot; rel=&quot;nofollow&quot;&gt;Asymptotic Statistics&lt;/a&gt;&lt;/em&gt;, Cambridge University Press, 1998, &lt;strong&gt;Chapter 23&lt;/strong&gt;.&lt;/li&gt;&#xA;&lt;li&gt;P. Bickel and D. Freedman, &lt;a href=&quot;http://projecteuclid.org/euclid.aos/1176345637&quot; rel=&quot;nofollow&quot;&gt;Some asymptotic theory for the bootstrap&lt;/a&gt;. &lt;em&gt;Ann. Stat.&lt;/em&gt;, vol. 9, no. 6 (1981), 1196–1217.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="2970" LastEditorUserId="2970" LastEditorDisplayName="" LastEditDate="2011-05-24T23:06:16.357" LastActivityDate="2011-05-24T23:06:16.357" CommentCount="3" />
  <row Id="11218" PostTypeId="2" ParentId="11210" CreationDate="2011-05-24T23:00:19.693" Score="2" ViewCount="" Body="&lt;p&gt;Here is a different approach to thinking about it:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Start with the theory where we know the true distribution, we can discover properties of sample statistics by simulating from the true distribution.  This is how Gosset developed the t-distribution and t-test, by sampling from known normals and computing the statistic.  This is actually a form of the parametric bootstrap.  Note that we are simulating to discover the behavior of the statistics (sometimes relative to the parameters).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, what if we do not know the population distribution, we have an estimate of the distribution in the empirical distribution and we can sample from that.  By sampling from the empirical distribution (which is known) we can see the relationship between the bootstrap samples and the empirical distribution (the population for the bootstrap sample).  Now we infer that the relationship from bootstrap samples to empirical distribution is the same as from the sample to the unknown population.  Of course how well this relationship translates will depend on how representative the sample is of the population.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Remember that we are not using the means of the bootstrap samples to estimate the population mean, we use the sample mean for that (or whatever the statistic of interest is).  But we are using the bootstrap samples to estimate properties (spread, bias) of the sampling proccess.  And using sampling from a know population (that we hope is representative of the population of interest) to learn the effects of sampling makes sense and is much less circular. &lt;/p&gt;&#xA;" OwnerUserId="4505" LastActivityDate="2011-05-24T23:00:19.693" />
  <row Id="11219" PostTypeId="1" CreationDate="2011-05-25T00:29:48.363" Score="2" ViewCount="58" Body="&lt;p&gt;I have been reading about appropriate measures of central tendency for ordinal level data. &#xA;So far I have learned that the median and mode can be used but that the latter can only be used in some cases. Some sources state that the median can only be used with Likert questions when there is an odd number of scores. It is not clear to me what this means and also which cases the median cannot be used. &lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Example:&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;An example may illustrate. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;If there was a question: &quot;Climate change is England’s most serious environmental problem&quot; on a response scale: 1=strongly agree 2=agree 3=unsure 4=disagree 5=strongly disagree. Would the median be 3=unsure?&lt;/li&gt;&#xA;&lt;li&gt;What if no respondents stated disagree or strongly disagree and all 100 respondents stated either 1, 2, or 3, is the median then 2? &lt;/li&gt;&#xA;&lt;li&gt;what if respondents only stated 2 or 3. In this case is it not possible to identify the median?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="4498" LastEditorUserId="4498" LastEditorDisplayName="" LastEditDate="2011-05-25T03:22:21.527" LastActivityDate="2011-05-25T05:41:03.343" Title="Median value on ordinal scales" Tags="&lt;median&gt;" AnswerCount="2" />
  <row Id="11220" PostTypeId="1" CreationDate="2011-05-25T00:56:40.383" Score="7" ViewCount="99" Body="&lt;p&gt;I have data from a load test of a web site with several thousand data points spread out over roughly 30 minutes (the values are the response time of the site in milliseconds). The values are spread out among the 30 minute range, but not at a constant rate (i.e. there may be a few milliseconds between some points, other points maybe at the same timestamp, etc). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd like to present this data visually and chart it, but I'm not sure of the best method for doing so - there is a good amount of variance around any sort of concept of average values or a trend line.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any generally accepted best practices for methods on how to graph data of this type? I'm concerned about choosing a poor method for averaging/smoothing out the data and misrepresenting the data - such as underweighting some outlier values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've played around with a line chart with the timestamps on the x-axis and the average of samples in the same minute on the y-axis. I'd also like to consider graphing a moving average of the data, but I'm unsure if I should be averaging datapoints in the same N minutes or a window of the last N points. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd like to make sure that whatever choice I make would appear to be a rigorous representation of the data and not too amateur-ish.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update:&lt;/strong&gt; below is a sample of what I have produced so far, each point on the chart is taken as the mean/median of all of the samples within the same minute (i.e. within 11:12:00.000 and 11:12:59.999). I included the number of samples per minute as a bar chart in the second half of the image to be able to show if any single points in the line chart look as outliers due to a small amount of samples, although aesthetically speaking I think the bar chart takes up way too much real estate for the amount of information it gives.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.imgur.com/4t1sT.png&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="4739" LastEditorUserId="4739" LastEditorDisplayName="" LastEditDate="2011-05-25T13:24:19.233" LastActivityDate="2011-05-25T16:02:58.067" Title="Preferred methods for graphing time-series data to present &quot;averages&quot;?" Tags="&lt;time-series&gt;&lt;data-visualization&gt;" AnswerCount="2" />
  <row Id="11221" PostTypeId="2" ParentId="11145" CreationDate="2011-05-25T01:35:52.533" Score="1" ViewCount="" Body="&lt;p&gt;As far as your statistical test, it might be a choice between 1) ancova with pretest weight as the covariate and 2) anova with change scores as the outcome.  You'd use ancova if you believed posttest weight would naturally be different from pretest weight even without the treatment, and that posttest weight would be a linear function of pretest weight with a correlation of at least .3.  You'd use anova on change scores if you believed that absent the treatment there would be no expected change in mean weight.  Cook and Campbell's Quasi-Experimentation has a particularly thought-provoking section on these issues.&lt;/p&gt;&#xA;" OwnerUserId="2669" LastActivityDate="2011-05-25T01:35:52.533" />
  <row Id="11222" PostTypeId="2" ParentId="11191" CreationDate="2011-05-25T01:49:14.433" Score="3" ViewCount="" Body="&lt;p&gt;Person-mean imputation with an minimum-item threshold is a simple strategy for retaining scale scores where participants miss the occasional response. &lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Some general principles&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;If missing data is minimal (e.g., less than 5% of participants are missing 1 item on a 10 item scale), the method of dealing with missing data is unlikely to make a difference to substantive conclusions.&lt;/li&gt;&#xA;&lt;li&gt;From a first principle perspective, imputation methods should provide more robust estimates of missing item responses as they incorporate both item and person characteristics into estimating the missing response.&lt;/li&gt;&#xA;&lt;li&gt;Design studies to avoid sporadic item-missing data.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;Conditions where person-mean imputation is more reasonable:&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Item means are all about the same&lt;/li&gt;&#xA;&lt;li&gt;The threshold number of missing items is low relative to the total number of items in the scale (e.g., a requirement for 19 out of 20 items is more appropriate than 10 out of 20 items)&lt;/li&gt;&#xA;&lt;li&gt;There is generally very little missing data; at the extreme level, there is no missing data, and person-mean imputation does not change the data at all&lt;/li&gt;&#xA;&lt;li&gt;the cause of missing data is due to random processes such as accidentally skipping items, not clearly indicating the response, and so on.&lt;/li&gt;&#xA;&lt;li&gt;A simple and standardised rule for calculating scale means is desired (e.g., a rule might be required for a test manual that can be applied in a standardised way across studies and samples)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;Avoiding sporadic missing data for items in scales&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;At a broader level, person-mean imputation of item responses is a response to a problem that can often be avoided using various study design strategies:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Computerised administration of questionnaires can prevent (where this is ethically permitted) participants skipping or missing items.&lt;/li&gt;&#xA;&lt;li&gt;If questionnaires are administered on paper and in person, the experimenter can review the questionnaire booklet to check for missing data before the participant leaves the room.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-25T02:07:08.603" LastActivityDate="2011-05-25T02:07:08.603" />
  <row Id="11223" PostTypeId="2" ParentId="11219" CreationDate="2011-05-25T01:57:05.730" Score="1" ViewCount="" Body="&lt;p&gt;No, the median is the value where half the data is less than or equal to that value and half the data is greater than or equal to that value.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So if your ordinal scale had 100 respondents then find the value that has at least 50 less or equal and 50 greater than or equal.  It would only be 3 if half the responses were to either side.  If 1 person said 1, 2 people said 2, 3 said 3, 4 said 4, and the remaining 90 said 5, then 5 would be the median.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The median works when the data is ordered, but would not make sense for nominal/unordered data, like what is your favorite color?&lt;/p&gt;&#xA;" OwnerUserId="4505" LastActivityDate="2011-05-25T01:57:05.730" CommentCount="2" />
  <row Id="11224" PostTypeId="2" ParentId="11193" CreationDate="2011-05-25T02:59:08.417" Score="5" ViewCount="" Body="&lt;p&gt;One way is to reverse-sort the data and use &lt;code&gt;duplicated&lt;/code&gt; to drop all the duplicates.&#xA;For me, this method is conceptually simpler than those that use apply. I think it should be very fast as well.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Some data to start with:&#xA;z &amp;lt;- data.frame(id=c(1,1,2,2,3,4),var=c(2,4,1,3,5,2))&#xA;# id var&#xA;#  1   2&#xA;#  1   4&#xA;#  2   1&#xA;#  2   3&#xA;#  3   5&#xA;#  4   2&#xA;&#xA;# Reverse sort&#xA;z &amp;lt;- z[order(z$id, z$var, decreasing=TRUE),]&#xA;# id var&#xA;#  4   2&#xA;#  3   5&#xA;#  2   3&#xA;#  2   1&#xA;#  1   4&#xA;#  1   2&#xA;&#xA;# Keep only the first row for each duplicate of z$id; this row will have the&#xA;# largest value for z$var&#xA;z &amp;lt;- z[!duplicated(z$id),]&#xA;&#xA;# Sort so it looks nice&#xA;z &amp;lt;- z[order(z$id, z$var),]&#xA;# id var&#xA;#  1   4&#xA;#  2   3&#xA;#  3   5&#xA;#  4   2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Edit: I just realized that the reverse sort above doesn't even need to sort on &lt;code&gt;id&lt;/code&gt; at all. You could just use &lt;code&gt;z[order(z$var, decreasing=TRUE),]&lt;/code&gt; instead and it will work just as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One more thought... If the &lt;code&gt;var&lt;/code&gt; column is numeric, then there's a simple way to sort so that &lt;code&gt;id&lt;/code&gt; is ascending, but &lt;code&gt;var&lt;/code&gt; is descending. This eliminates the need for the sort at the end (assuming you even wanted it to be sorted).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;z &amp;lt;- data.frame(id=c(1,1,2,2,3,4),var=c(2,4,1,3,5,2))&#xA;&#xA;# Sort: id ascending, var descending&#xA;z &amp;lt;- z[order(z$id, -z$var),]&#xA;&#xA;# Remove duplicates&#xA;z &amp;lt;- z[!duplicated(z$id),]&#xA;# id var&#xA;#  1   4&#xA;#  2   3&#xA;#  3   5&#xA;#  4   2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="4740" LastEditorUserId="4740" LastEditorDisplayName="" LastEditDate="2011-05-25T19:40:10.713" LastActivityDate="2011-05-25T19:40:10.713" />
  <row Id="11225" PostTypeId="1" CreationDate="2011-05-25T04:43:24.693" Score="2" ViewCount="17" Body="&lt;p&gt;I need to calculate an exponential moving average for a series of data. The intended sampling interval is fixed (say 1s) but the data stream has varying intervals (data intervals vary from 0.01s to 10s or so). The data is somewhat noisy (a random data sample would virtually never be on the average).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My impression is that I cannot thus just take the most recent data sample at each interval, as that could easily lead to a misleading stat. I reckon I can somehow just calculate the average for each period and take that as the sample, but I'm not positive.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a standard algorithm that manages an exponential moving average on a time-variable data stream?&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;I need to program this for a real-time system where it won't be possible to store a sample history. Nonetheless, I'm sure I can adapt any non-streaming algorithm to the streaming form.&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="4741" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-25T06:55:33.350" LastActivityDate="2011-05-25T06:55:33.350" Title="Exponential moving average with sub-interval relevance / varying timeframe" Tags="&lt;time-series&gt;&lt;sampling&gt;&lt;exponential-smoothing&gt;" FavoriteCount="1" />
  <row Id="11226" PostTypeId="2" ParentId="11219" CreationDate="2011-05-25T05:41:03.343" Score="1" ViewCount="" Body="&lt;h3&gt;Definitional issues:&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;The median is the middle value of the data; it is not by definition the middle value of the scale.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;When the sample size is even, then the median is the mean of the values either side of middle most point after rank ordering all values (&lt;a href=&quot;http://en.wikipedia.org/wiki/Median#For_an_even_number_of_values&quot; rel=&quot;nofollow&quot;&gt;see wikipedia description&lt;/a&gt;).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;When to use median on ordinal data&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;In theory the median can be used on data from any variable where the values can be ordered.&lt;/li&gt;&#xA;&lt;li&gt;In practice, the median is  often not the most useful summary of central tendency with ordinal variables. &#xA;This partially depends on what you want to get out of your measure of central tendency.&#xA;When you are describing the central tendency of data on an ordinal variable with only a small number of response options (i.e., perhaps less than 20 or 50 or 100), the median can be quite gross (e.g., 1,1,3,3,3 and 1,3,3,5,5 both have a median of 3, but the second example would have a higher mean). &#xA;When it comes to summarising the central tendency of Likert items, I find the mean to be much more useful and sensitive to meaningful differences.&#xA;Ordinal variables that are ranks do not suffer from this problem of &quot;grossness&quot;.&lt;/li&gt;&#xA;&lt;li&gt;Interpolated medians are another way of overcoming the gross nature of the median on ordinal data with few values.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastActivityDate="2011-05-25T05:41:03.343" CommentCount="1" />
  <row Id="11227" PostTypeId="1" CreationDate="2011-05-25T05:52:01.007" Score="3" ViewCount="40" Body="&lt;p&gt;Say I have 2 sets, $A$ and $B$ with $n_{A}$ and $n_{B}$ elements respectively, which I assume is known.  I would like to estimate $| A \bigcup B |$ using samples of $\tilde{A} \subset A$ and $ \tilde{B} \subset B$.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;That is if $\tilde{A}$'s elements are uniformly sampled from $A$, and likewise for $\tilde{B}$, will $ \frac{| \tilde{A} \bigcup \tilde{B} |}{| \tilde{A} | + |  \tilde{B} |}$ be an unbiased estimate for  $\frac{| A \bigcup B |}{| A |+|B |}$?  If not, is there some another estimator that will allow me estimate $| A \bigcup B |$ without bias?&lt;/p&gt;&#xA;" OwnerUserId="4742" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-25T08:40:23.070" LastActivityDate="2011-05-25T08:40:23.070" Title="Bias in sampling for set intersections" Tags="&lt;sampling&gt;&lt;unbiased-estimator&gt;&lt;bias&gt;" CommentCount="2" FavoriteCount="1" />
  <row Id="11228" PostTypeId="2" ParentId="11203" CreationDate="2011-05-25T06:52:17.017" Score="3" ViewCount="" Body="&lt;p&gt;MAPE is known to have problems, when the time series have values close to zero. Check whether this is the case, since high MAPEs may be the problem of time series values close to zero, not of model accuracy. For a discussion on accuracy measures I recommend &lt;a href=&quot;http://www.buseco.monash.edu.au/ebs/pubs/wpapers/2005/wp13-05.pdf&quot; rel=&quot;nofollow&quot;&gt;this article&lt;/a&gt; by Rob Hyndman and Anne Koehler. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If it is not the problem with MAPE, @Zach advice is spot on, you should always  compare the forecasts with actual values, that is how you know how good your forecasts are.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-25T07:54:30.247" LastActivityDate="2011-05-25T07:54:30.247" CommentCount="3" />
  <row Id="11229" PostTypeId="2" ParentId="11220" CreationDate="2011-05-25T07:22:42.653" Score="5" ViewCount="" Body="&lt;p&gt;I suggest adding an example or two of what you are presently doing so we can better see what you are dealing with.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you are concerned with is an important issue: how do you convey the &quot;overall&quot; pattern in the time series data while also not misleading viewers by showing just average values? One way I have dealt with this situation is plotting an average or median line along with surrounding quantile bands. For example,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.stack.imgur.com/ol3CF.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here, the time series data are from a bootstrap-based simulation so there are hundreds of values associated with each time point. The actual data are plotted in the black line with colored bands showing the variability of values from the simulation. This particular plot is maybe not the best example to show, but you can see that some points have much more variability than others, and you can also assess how the variability is skewed above/below the actual values depending on the position in the series. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE&lt;/strong&gt;:&#xA;Given your update here are some additional questions and thoughts... What decisions, if any, are made from this visualization? For example, are you looking for specific points in time where there is very slow response time, perhaps above a specific threshold? If so, it may be better to simply plot all of the points as a scatter plot, and then also plot a time series line showing the average value, as well as some lines delineating the bounds you are concerned about. This recommendation is not appropriate if you have numerous observations at some time points (too much clutter), or if your time measurement is not sufficiently coarse (in which case you can bin response data into minute-wide time of day intervals). But the visualization recommendation will certainly be affected by what decision(s) will be supported with it. In my example, I was looking at such plots side by side, one from one simulation and the other from another simulation (each simulation using different parameters) so I could assess the variability of the underlying model due to sampling error.&lt;/p&gt;&#xA;" OwnerUserId="1080" LastEditorUserId="1080" LastEditorDisplayName="" LastEditDate="2011-05-25T15:50:44.233" LastActivityDate="2011-05-25T15:50:44.233" CommentCount="5" />
  <row Id="11230" PostTypeId="2" ParentId="11209" CreationDate="2011-05-25T07:53:11.060" Score="5" ViewCount="" Body="&lt;p&gt;I don't have Matlab but from what I've read in the on-line help for &lt;a href=&quot;http://www.mathworks.com/help/toolbox/stats/anovan.html&quot; rel=&quot;nofollow&quot;&gt;N-way analysis of variance&lt;/a&gt; it's not clear to me whether Matlab would automatically adapt the &lt;code&gt;type&lt;/code&gt; (1--3) depending on your design. My best guess is that yes you got different results because the tests were not designed in the same way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Generally, with an imbalanced design it is recommended to use Type III sum of squares (SS), where each term is tested after all other (the difference with Type II sum of squares is only apparent when an interaction term is present), while with an incomplete design it might be interesting to compare Type III and Type IV SS. Note that the use of type III vs. Type II in the case of unbalanced data is subject to discussion in the literature.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(The following is based on a French tutorial that I cannot found anymore on the original website. Here is a &lt;a href=&quot;http://www.aliquote.org/cours/2006_cogmaster_A4/ressources/ab-deseq.pdf&quot; rel=&quot;nofollow&quot;&gt;personal copy&lt;/a&gt;, and here is another paper that discussed the different ways to compute SS in factorial ANOVAs: &lt;a href=&quot;http://www.matstat.com/ss/easleaao.pdf&quot; rel=&quot;nofollow&quot;&gt;Which Sums of Squares Are Best In Unbalanced Analysis of Variance?&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The difference between Type I/II and Type III (also called Yates's weighted squares of means) lies in the model that serves as a reference model when computing SS, and whether factors are treated in the order they enter the model or not. Let's say we have two factors, A and B, and their interaction A*B, and a model like y ~ A + B + A:B (Wilkinson's notation). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;With Type I SS, we first compute SS associated to A, then B, and finally A*B. Those SS are computed as the difference in residual SS (RSS) between the largest model omitting the term of interest and the smallest one including it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For Type II and III, SS are computed in a sequantial manner, starting with those associated to A*B, then B, and finally A. For A*B, it is simply the difference between the RSS in the full model and the RSS in the model without interaction. The SS associated to B is computed as the difference between RSS for a model where B is omitted and a model where B is included (reference model); with Type III SS, the reference model is the full model (A+B+A*B), whereas for Type I and II SS, it is the additive model (A+B). This explains why Type II and III will be identical when no interaction is present in the full model. However, to obtain the first SS, we need to use dummy variables to code the levels of the factor, or more precisely difference between those dummy-coded levels (which also means that the reference level considered for a given factor matters; e.g., SAS consider the last level, whereas R consider the first one, in a lexicographic order). To compute SS for the A term, we follow the same idea: we consider the difference between the RSS for the model A+B+A*B and that for the reduced model B+A*B (A omitted), in case of Type III SS; with Type II SS, we consider A+B vs. B.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that in a complete balanced design, all SS will be equal. Moreover, with Type I SS, the sum of all SS will equal that of the full model, whatever the order of the terms in the model is. (This is not true for Type II and Type III SS.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A detailed and concrete overview of the different methods is available in one of Howell's handout: &lt;a href=&quot;http://www.uvm.edu/~dhowell/StatPages/More_Stuff/Type1-3.pdf&quot; rel=&quot;nofollow&quot;&gt;Computing Type I, Type II, and Type III Sums of Squares directly using the general linear model&lt;/a&gt;. That might help you check your code. You can also use R with the &lt;a href=&quot;http://cran.r-project.org/web/packages/car/index.html&quot; rel=&quot;nofollow&quot;&gt;car&lt;/a&gt; package, by John Fox who dicussed the use of incremental sum of squares in his textbook, &lt;em&gt;Applied Regression Analysis, Linear Models, and Related Methods&lt;/em&gt; (Sage Publications, 1997, § 8.2.4--8.2.6). An example of use can be found on &lt;a href=&quot;http://www.uni-kiel.de/psychologie/dwoll/r/ssTypes.php&quot; rel=&quot;nofollow&quot;&gt;Daniel Wollschläger&lt;/a&gt; website.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, the following paper offers a good discussion on the use of Type III SS (§ 5.1):&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Venables, W.N. (2000). &lt;a href=&quot;http://www.stats.ox.ac.uk/pub/MASS3/Exegeses.pdf&quot; rel=&quot;nofollow&quot;&gt;Exegeses on&#xA;  Linear Models&lt;/a&gt;. Paper presented to&#xA;  the S-PLUS User’s Conference&#xA;  Washington, DC, 8-9th October, 1998.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;(See also this &lt;a href=&quot;http://r.789695.n4.nabble.com/Type-I-v-s-Type-III-Sum-Of-Squares-in-ANOVA-td1573657.html&quot; rel=&quot;nofollow&quot;&gt;R-help thread&lt;/a&gt;, references therein, and the following post &lt;a href=&quot;http://mcfromnz.wordpress.com/2011/03/02/anova-type-iiiiii-ss-explained/&quot; rel=&quot;nofollow&quot;&gt;Anova – Type I/II/III SS explained&lt;/a&gt;.)&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-25T08:56:55.523" LastActivityDate="2011-05-25T08:56:55.523" CommentCount="3" />
  <row Id="11231" PostTypeId="1" CreationDate="2011-05-25T09:46:14.150" Score="2" ViewCount="79" Body="&lt;p&gt;I am investigating many different kinds of PCA versions, I am trying to find out whether PCR will apply to my analysis thus the question on use of PCR.&lt;/p&gt;&#xA;" OwnerUserId="4747" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-25T11:27:21.693" LastActivityDate="2011-05-27T06:59:20.680" Title="Applications of principal component analysis versus principal component regression?" Tags="&lt;pca&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="11232" PostTypeId="1" CreationDate="2011-05-25T11:30:44.473" Score="3" ViewCount="70" Body="&lt;p&gt;I am trying to compare the difference between two means with two pairwise samples. Unfortunately, my data are very far of being normal. What test would you recommend to use in this situation? Should I revert to a nonparametric test?&lt;/p&gt;&#xA;" OwnerUserId="4748" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-25T11:40:42.647" LastActivityDate="2011-05-25T13:44:31.233" Title="Testing difference between two means with pairwise data and absence of normality" Tags="&lt;hypothesis-testing&gt;" AnswerCount="3" FavoriteCount="1" />
  <row Id="11233" PostTypeId="1" CreationDate="2011-05-25T11:51:00.517" Score="0" ViewCount="74" Body="&lt;p&gt;I have a question about simulating data sets for a linear mixed model. I google this online and found a post &lt;a href=&quot;https://stat.ethz.ch/pipermail/r-sig-mixed-models/2007q3/000293.html&quot; rel=&quot;nofollow&quot;&gt;https://stat.ethz.ch/pipermail/r-sig-mixed-models/2007q3/000293.html&lt;/a&gt;.&#xA;It seems a great answer for me but I am unable to figure it out because information is not complete. Assume, I have a data set, which is similar to &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(nlme) &#xA;?Orthodont &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and my model is &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;fm2 &amp;lt;- lme(distance ~ age + Sex, data = Orthodont, random = ~ 1) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can I use this &lt;code&gt;fm2&lt;/code&gt; information to generate several data sets, which have sample sizes 300, 400, 500, ... ?&lt;/p&gt;&#xA;" OwnerUserId="4559" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-25T12:22:13.020" LastActivityDate="2011-05-25T12:22:13.020" Title="Simulating data sets for a linear mixed model in R" Tags="&lt;r&gt;&lt;mixed-model&gt;&lt;simulation&gt;" CommentCount="1" />
  <row Id="11234" PostTypeId="2" ParentId="11232" CreationDate="2011-05-25T12:15:55.230" Score="3" ViewCount="" Body="&lt;p&gt;Sounds like a job for the &lt;a href=&quot;http://en.wikipedia.org/wiki/Wilcoxon_signed-rank_test&quot; rel=&quot;nofollow&quot;&gt;paired Wilcoxon test&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that this method compares the medians of the two samples, not their means. In any case, the mean is often not a good estimator when the distributions are not normally distributed, as it is easily biased by extremely low or high values. &lt;/p&gt;&#xA;" OwnerUserId="656" LastActivityDate="2011-05-25T12:15:55.230" />
  <row Id="11235" PostTypeId="2" ParentId="11232" CreationDate="2011-05-25T13:16:56.460" Score="3" ViewCount="" Body="&lt;p&gt;A paired t-test assumes that the &lt;em&gt;differences&lt;/em&gt; are normal: the original values could have any distribution. More precisely, just like with a t-test, the differences don't even have to be normal, just the sampling distribution of the mean. This usually means that with a large enough sample you can use a t-test even without normality because the central limit theorem will kick in.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, one can always use a non-parametric test with not too much loss in efficiency.&lt;/p&gt;&#xA;" OwnerUserId="279" LastActivityDate="2011-05-25T13:16:56.460" CommentCount="1" />
  <row Id="11236" PostTypeId="1" AcceptedAnswerId="11291" CreationDate="2011-05-25T13:29:57.467" Score="3" ViewCount="51" Body="&lt;p&gt;I've been using R's &lt;code&gt;lm&lt;/code&gt; to do some linear regression, but decided to give &lt;code&gt;MCMCregress&lt;/code&gt; a try to get a feel for how it works. As expected, I got basically the same coefficients, but the extra &lt;code&gt;sigma2&lt;/code&gt; value puzzles me.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I do a &lt;code&gt;qqmath&lt;/code&gt; plot of the coefficients, I get the following graph, and I'm puzzled by the sigma2 plot. It's obviously not linear, but I'm not sure if that's meaningful in this context. I assume it's sigma &lt;em&gt;squared&lt;/em&gt;, and when I took the square root and plotted it, the line was straighter, but still curved.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I guess my question boils down to: what is sigma2 telling me about the MCMC regression fit, and is a graph of it useful or should I ignore the graph and focus on something else? (All of the diagnostics and graphs I've done on my original &lt;code&gt;lm&lt;/code&gt; fit seem to indicate that the fit is good, so I'm also wondering if the MCMC regression gives me more information or not.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.stack.imgur.com/wayi4.png&quot; alt=&quot;qqmath plot of MCMCregress results&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(If I need to provide the actual data, I can. I'm hoping that an answer depends more on what sigma2 is rather than on specific values.)&lt;/p&gt;&#xA;" OwnerUserId="1764" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-25T19:35:48.040" LastActivityDate="2011-05-26T20:15:53.020" Title="QQ plot of sigma2 from an MCMC regression?" Tags="&lt;r&gt;&lt;regression&gt;&lt;mcmc&gt;" AnswerCount="1" />
  <row Id="11237" PostTypeId="2" ParentId="11191" CreationDate="2011-05-25T13:31:15.813" Score="1" ViewCount="" Body="&lt;p&gt;one more piece of advice: make sure the full 6-item composite scale is reliable &amp;amp; that none of the included items &lt;em&gt;reduces&lt;/em&gt; scale reliability. If those conditions aren't satisfied, you shouldn't be averaging them even in cases where data are complete. If these conditions are satisfied, then using a subset of items for cases w/ missing data isn't going to bias your result (assuming you are averaging or adding z-score transformation of items, as you always should in forming aggregate likert scale); it is just going to make it noisier than it should be (b/c you are relying on fewer items &amp;amp; thus cancelling out less of the random measurement error associated with each individual item). &#xA;(Best solution, though, is multiple imputation, again assuming composite scale is reliable.)&lt;/p&gt;&#xA;" OwnerUserId="2347" LastActivityDate="2011-05-25T13:31:15.813" />
  <row Id="11238" PostTypeId="2" ParentId="11232" CreationDate="2011-05-25T13:44:31.233" Score="4" ViewCount="" Body="&lt;p&gt;Your description of your design is not too precise as it allows two interpretations.&lt;br&gt;&#xA;First, it is possible that you have a 2 (between) x 2 (within) design (i.e., two &lt;strong&gt;groups&lt;/strong&gt; with two pairwise samples).&lt;br&gt;&#xA;Second, it is possible that you have a simple design with one group which was measured two times.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Only in the second case, the answers here apply.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furthermore, the question if it is &lt;strong&gt;really&lt;/strong&gt; inappropriate to use a t-test for our data is the crucial part of your question. Sometimes this question is difficult in the sense that one may confuse normality of the data with normality of the residuals (crucial in the first interpretations of your design, see &lt;a href=&quot;http://stats.stackexchange.com/questions/6350/anova-assumption-normality-normal-distribution-of-residuals&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;http://stats.stackexchange.com/questions/1637/if-the-t-test-and-the-anova-for-two-groups-are-equivalent-why-arent-their-assum&quot;&gt;here&lt;/a&gt;) and/or normality of the differences (crucial for the second interpretation, see Aniko's answer).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the deviation from normality are that serious that you do not want to use a t-test, you should think about using a permutation test instead of the wilcoxon. See the answers to the following two questions for how to make permutations tests using the &lt;code&gt;coin&lt;/code&gt; package for r with dependent samples:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://stats.stackexchange.com/questions/6127/which-permutation-test-implementation-in-r-to-use-instead-of-t-tests-paired-and&quot;&gt;Which permutation test implementation in R to use instead of t-tests (paired and non-paired)?&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://stats.stackexchange.com/questions/10953/paired-permutation-test-for-repeated-measures&quot;&gt;Paired permutation test for repeated measures&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="442" LastActivityDate="2011-05-25T13:44:31.233" CommentCount="1" />
  <row Id="11239" PostTypeId="2" ParentId="11193" CreationDate="2011-05-25T14:34:17.263" Score="1" ViewCount="" Body="&lt;p&gt;Yet another way to do this with base:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dt&amp;lt;-data.frame(id=c(1,1,2,2,3,4),var=c(2,4,1,3,4,2))&#xA;&#xA;data.frame(id=sort(unique(dt$var)),max=tapply(dt$var,dt$id,max))&#xA;  id max&#xA;1  1   4&#xA;2  2   3&#xA;3  3   4&#xA;4  4   2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I prefer mpiktas&#xA;' plyr solution though.&lt;/p&gt;&#xA;" OwnerUserId="3094" LastActivityDate="2011-05-25T14:34:17.263" />
  <row Id="11240" PostTypeId="2" ParentId="11220" CreationDate="2011-05-25T16:02:58.067" Score="2" ViewCount="" Body="&lt;p&gt;Have you considered a scatterplot of the data themselves?  &lt;a href=&quot;http://stats.stackexchange.com/questions/173/time-series-for-count-data-with-counts-20&quot;&gt;That's an approach I really like&lt;/a&gt;.  It lets the viewer make their own conclusions about the presence and significance of trends, and it doesn't conceal variability or outliers.  Alpha-blending the points will help if you have serious overplotting (which it sounds like you might).  You can also overlay whatever trend you like and take comfort in knowing that the data are still present to speak for themselves.&lt;/p&gt;&#xA;" OwnerUserId="71" LastActivityDate="2011-05-25T16:02:58.067" CommentCount="6" />
  <row Id="11241" PostTypeId="1" CreationDate="2011-05-25T16:30:02.967" Score="0" ViewCount="18" Body="&lt;p&gt;I'm reverse-engineering how Mathematica does list interpolation: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(* Fortunately, Mathematica WILL interpolate an arbitrary list *) &#xA;&#xA;tab = Table[a[i], {i,1,100}] &#xA;&#xA;f = Interpolation[tab] &#xA;&#xA;(* get the coefficient of each term by setting others to zero *) &#xA;&#xA;Plot[{f[42+x] /. {a[42] -&amp;gt; 0, a[43] -&amp;gt;0, a[44] -&amp;gt; 0, a[41] -&amp;gt; 1}}, &#xA; {x,0,1}] &#xA;&#xA;Plot[{f[42+x] /. {a[41] -&amp;gt; 0, a[43] -&amp;gt;0, a[44] -&amp;gt; 0, a[42] -&amp;gt; 1}}, &#xA; {x,0,1}] &#xA;&#xA;Plot[{f[42+x] /. {a[42] -&amp;gt; 0, a[41] -&amp;gt;0, a[44] -&amp;gt; 0, a[43] -&amp;gt; 1}}, &#xA; {x,0,1}] &#xA;&#xA;Plot[{f[42+x] /. {a[42] -&amp;gt; 0, a[43] -&amp;gt;0, a[41] -&amp;gt; 0, a[44] -&amp;gt; 1}}, &#xA; {x,0,1}] &#xA;&#xA;(* above is neither Hermite, nor linear, though some look close *) &#xA;&#xA;(* these are available at oneoff.barrycarter.info/STACK/ *) &#xA;&#xA;Table[f[42+x] /. {a[42] -&amp;gt; 0, a[43] -&amp;gt;0, a[44] -&amp;gt; 0, a[41] -&amp;gt; 1}, &#xA; {x,0,1, 1/100}] &amp;gt;&amp;gt; /home/barrycarter/BCINFO/ONEOFF/STACK/coeff41.txt &#xA;&#xA;Table[f[42+x] /. {a[41] -&amp;gt; 0, a[43] -&amp;gt;0, a[44] -&amp;gt; 0, a[42] -&amp;gt; 1}, &#xA; {x,0,1, 1/100}] &amp;gt;&amp;gt; /home/barrycarter/BCINFO/ONEOFF/STACK/coeff42.txt &#xA;&#xA;Table[f[42+x] /. {a[41] -&amp;gt; 0, a[42] -&amp;gt;0, a[44] -&amp;gt; 0, a[43] -&amp;gt; 1}, &#xA; {x,0,1, 1/100}] &amp;gt;&amp;gt; /home/barrycarter/BCINFO/ONEOFF/STACK/coeff43.txt &#xA;&#xA;Table[f[42+x] /. {a[41] -&amp;gt; 0, a[42] -&amp;gt;0, a[43] -&amp;gt; 0, a[44] -&amp;gt; 1}, &#xA; {x,0,1, 1/100}] &amp;gt;&amp;gt; /home/barrycarter/BCINFO/ONEOFF/STACK/coeff44.txt&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The graphs are below: how is Mathematica computing the coefficients of each term?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.stack.imgur.com/4zgKe.jpg&quot; alt=&quot;coefficient of 41st term&quot;&gt;&#xA;&lt;img src=&quot;http://i.stack.imgur.com/StKvL.jpg&quot; alt=&quot;coefficient of 42nd term&quot;&gt;&#xA;&lt;img src=&quot;http://i.stack.imgur.com/TPVm5.jpg&quot; alt=&quot;coefficient of 43rd term&quot;&gt;&#xA;&lt;img src=&quot;http://i.stack.imgur.com/lWe8e.jpg&quot; alt=&quot;coefficient of 44th term&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="1566" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-05-25T18:37:06.033" LastActivityDate="2011-05-25T18:37:06.033" ClosedDate="2011-05-25T18:38:05.050" Title="Identify Mathematica interpolation functions from graphs (not Hermite)" Tags="&lt;splines&gt;&lt;interpolation&gt;&lt;mathematica&gt;" AnswerCount="0" CommentCount="1" />
  <row Id="11242" PostTypeId="1" CreationDate="2011-05-25T14:45:39.393" Score="1" ViewCount="76" Body="&lt;p&gt;This is a question strongly related to Cauchy &quot;characters&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm constructing a 4 question canvassing questionnaire that will tell the likely voter being contacted which of the presidential candidates most closely matches them.  The advantage of this approach for a dark horse presidential candidate is obvious, presuming, of course, that most of the likely voters match him.  I do have the verbal interest in this from the state-level executive director for such a presidential candidate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One might be entitled to think that this work has been done umpteen times by the thousands of political science PhDs and/or major polling organizations -- at least using the General Social Survey data if nothing else -- and one would be wrong.  Moreover, we don't have much time to deploy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ideally the questionnaire construction would result in a kind of decision tree where the door-to-door canvassing volunteer could have a mobile device app providing the next question to ask based on the answers to prior questions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also ideally, the construction process, itself, would minimize the likely-voter contact as this drives the expense.  Using GSS data zeros out that cost and would be optimal if we could get access to the raw GSS data, but we can't.  We have to do a survey to gather the data for construction of the 4-deep tree of questions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the result side, as a practical compromise, I've proposed falling back to finding just 4 questions rather than finding a 4-deep tree of questions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the construction side, as a practical compromise, I've proposed a prize-fund backing a tournament where:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The contestants each submit 4 questions.   &lt;/li&gt;&#xA;&lt;li&gt;The submitted questionnaires are paired up for the contests.&lt;/li&gt;&#xA;&lt;li&gt;Campaign volunteers each get a pair of contested questionnaires, and get ten likely voters to completely answer all 8 questions.&lt;/li&gt;&#xA;&lt;li&gt;The winning questionnaire of a contest is the one whose author can, from the answers to his own questionnaire, best-guess the answers to the  opponent's questionnaire.&lt;/li&gt;&#xA;&lt;li&gt;Award prizes after the log2(N) contests have selected a winner of that tournament.&lt;/li&gt;&#xA;&lt;li&gt;Publish the rankings of the questionnaires and, by permission, their authors.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;As resources permit, this tournament is iterated for multiple rounds.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We really have to place weight on the value of up-front volunteer time, so minimizing the construction labor is crucial.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know this is fairly far from a pure mathematics question but I've brought it as close as I can to some kind of weighted figure of merit involving high-value labor in the construction phase and the expected accuracy of the resulting 4 questions answered by likely voters contacted during blind poll canvassing by lower value labor.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The question:  About how far from optimal would be the proposed practical compromise of the 4-question questionnaire (constructed as described) from the ideal 4-deep decision tree of questions constructed from an infinite number of samples during the construction phase?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A secondary question:  Is there a better way to make use of the same up-front volunteer time?&lt;/p&gt;&#xA;" OwnerUserId="4753" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-25T19:38:19.693" LastActivityDate="2011-05-29T18:01:20.593" Title="Optimal blind poll construction" Tags="&lt;survey&gt;&lt;experimental-design&gt;" AnswerCount="2" CommentCount="4" FavoriteCount="1" />
  <row Id="11243" PostTypeId="1" CreationDate="2011-05-25T17:11:02.417" Score="2" ViewCount="57" Body="&lt;p&gt;&lt;strong&gt;Context&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a regression framework and two sets of data. Using leave-one-out cross-validation, the first set gives very good performance and the second set gives rather poor performance. I need to explain the reason for this difference in performance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having looked at the data, it is clear that the first set is a much easier test case. The data in the first set does not vary as much as in the second set. In addition, the test and training data of the first set are strikingly similar, which is not the case in the second dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To explain the reason for the performance difference, I can show these differences in data qualitatively using pictures of the physical phenomena being modeled. However, I would also like to quantify these dataset differences and relate it to the performance discrepancy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Questions&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What metrics can I use to show that the first dataset is a trivial test case and that the second set is more challenging?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a statistician, what else would you want to know in order to be confident that the performance discrepancy was caused by these differences in data and not something else?&lt;/p&gt;&#xA;" OwnerUserId="3052" LastActivityDate="2011-05-25T17:11:02.417" Title="Explaining regression performance differences" Tags="&lt;regression&gt;&lt;cross-validation&gt;&lt;ridge-regression&gt;" CommentCount="9" />
  <row Id="11246" PostTypeId="1" AcceptedAnswerId="11265" CreationDate="2011-05-25T18:44:42.107" Score="3" ViewCount="59" Body="&lt;p&gt;The last line is an example of what I'm looking for:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data(airquality)&#xA;attach(airquality)&#xA;&#xA;lm1 &amp;lt;- lm(Ozone ~ Solar.R+Wind)&#xA;lm2 &amp;lt;- lm(Ozone ~ Solar.R+Wind+Temp)&#xA;anova(lm1 , lm2)&#xA;&#xA;require(rpart)&#xA;rp1 &amp;lt;- rpart(Ozone ~ Solar.R+Wind)&#xA;rp2 &amp;lt;- rpart(Ozone ~ Solar.R+Wind+Temp)&#xA;anova(rp1 , rp2) # this doesn't exist - is there something like it? some sort of anova.rpart function?&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="253" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-25T19:42:03.047" LastActivityDate="2011-05-26T07:49:23.200" Title="Is there an ANOVA table generalization for two nested CART models?" Tags="&lt;anova&gt;&lt;cart&gt;" AnswerCount="2" CommentCount="2" />
  <row Id="11247" PostTypeId="2" ParentId="11242" CreationDate="2011-05-25T18:46:50.477" Score="0" ViewCount="" Body="&lt;p&gt;EDIT in response to last comments.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is my suggestion for how to run the contest.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The contest holder should decide on a list of &quot;test questions&quot;.  The 4-item questionnaires will be scored on how well they allow the guesser to guess the voter's responses to these &quot;test questions&quot;.  These test questions will be made public, and there will be a call for submissions for 4-test questionnaires.  There will also be a call for participants to compete in the &quot;guessing&quot; contest.  No participant is allowed to compete in both contests.&lt;/li&gt;&#xA;&lt;li&gt;The contest holder decides on a list of the (e.g.) 10 most promising questionnaires.&lt;/li&gt;&#xA;&lt;li&gt;The questionnaires are randomly assigned to volunteers.  The volunteers interview potential voters.&lt;/li&gt;&#xA;&lt;li&gt;The survey that a voter completes consists of (i.) the complete list of &quot;test questions&quot; (ii.) plus one of the competing 4-item questionnaires. &lt;/li&gt;&#xA;&lt;li&gt;The completed surveys are assembled.  A test is created for the guessers to complete.  Each test question corresponds to an survey that a voter completed.  The test question gives the voter's responses to the 4-item questionnaire.  The guessers attempt to guess the voter's responses to the &quot;test questions&quot; based on that information.&lt;/li&gt;&#xA;&lt;li&gt;Compute a &quot;guesser score&quot; based on how well each guesser did overall, and compute a &quot;questionnaire score&quot; by taking a weighted average for each questionnaire weighted by the guesser score.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="3567" LastEditorUserId="3567" LastEditorDisplayName="" LastEditDate="2011-05-25T23:37:05.250" LastActivityDate="2011-05-25T23:37:05.250" CommentCount="7" />
  <row Id="11248" PostTypeId="1" CreationDate="2011-05-25T21:05:29.583" Score="3" ViewCount="67" Body="&lt;p&gt;I'm using Gibbs sampling to learn the distributions of coefficients for a multinomial logistic regression model. At the end, I end up using the mean values of distributions of coefficients, and the resulting logistic regression is used as a classifier. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm trying to find out advantages of having probability distributions for coefficients and the response variable, but I can't really see the way to leverage credibility intervals. What can I do with these distributions that I end up with, other than using their mean values?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ok, I think I've failed to express my question clearly. Here is the update&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's assume that I have y = b1*x1 + b2*x2 + b3*x3 in my hands. for all b1,b2,b3 I have normal distributions, with mean and quantiles. These distributions come from the mcmc results. I can use the means of b1,b2,b3 and that'd give me a classifier. I can also use %2.5 quantile values and 97.5 quantile values for b1,b2,b3 and so on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What would be the probabilistic interpretation of these other equations? Can I produce a smoother classifier this way, rather than using only means? &#xA;The use of credibility intervals is quite clear to me when they are used for a single variable, but in this case, I'm talking about N variables (5 actually) each with their own credibility intervals. I'm having trouble getting the semantics of this setup. I have not seen any papers etc that discusses this, and any pointers would be appreciated. &lt;/p&gt;&#xA;" OwnerUserId="3280" LastEditorUserId="3280" LastEditorDisplayName="" LastEditDate="2011-05-29T11:33:36.090" LastActivityDate="2011-05-31T22:49:06.153" Title="How can I use credibility intervals in Bayesian logistic regression?" Tags="&lt;logistic-regression&gt;&lt;bayesian&gt;&lt;credible-interval&gt;" AnswerCount="3" CommentCount="0" />
  <row Id="11249" PostTypeId="1" CreationDate="2011-05-25T21:32:13.980" Score="4" ViewCount="138" Body="&lt;p&gt;I have a large data set which is in .dbf format right now and what I would like to do is be able to manipulate it easily in Excel and do something like subtotal and calculate stdev and ratios.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Details of the data set;&#xA;This data set contains shopper information. It has 1.2 million rows and 20 columns where the rows are each a unique shopper and the columns hold their shopping data (what they bought).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using Office 2007 programs, I know Excel the best but was wondering what alternatives I could use to accomplish my goals (subtotal, calculate stdev, and ratio's).&lt;/p&gt;&#xA;" OwnerUserId="4755" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-26T06:08:23.393" LastActivityDate="2011-05-26T17:33:25.187" Title="What would be a good way to work with a large data set in Excel?" Tags="&lt;excel&gt;&lt;large-data&gt;" AnswerCount="2" CommentCount="7" />
  <row Id="11250" PostTypeId="2" ParentId="11231" CreationDate="2011-05-25T21:45:02.643" Score="2" ViewCount="" Body="&lt;p&gt;When doing a PCA, you are effectively choosing a new set of 'variables' that you know for all your observations. Their main property is that they maximize the variance-content in one dimension (first PC has the most,...), while being linear combinations of the original covariates. This is the way it works like a dimension reduction: if 3 PCs contain 99% of the variance delivered by 100 covariates, there is not much reason, it seems, to keep the 100 covariates.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PCR essentially does regression on a set of principal components. Initially it makes sense, and in quite a few cases it does work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, in this regard, it is useful to look at Fisher's interpretation of discriminant analysis: he poses the problem as finding the direction(s) where the between-classes variance is maximal wrt the within-class variance. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is where PCA fails somewhat (or could fail): it finds the direction where the 'overall' variance in the covariates is maximal (a much simpler problem), and then hopes this discriminates well. So, there is some criticism on the method, but that must not stop it from working :-)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, doing a clustering style algorithm on your covariates first, and then using the results for classification is not a practice I'd recommend: perhaps the strongest structure in the covariates alone is not the most efficient one for prediction of another variable.&lt;/p&gt;&#xA;" OwnerUserId="4257" LastActivityDate="2011-05-25T21:45:02.643" />
  <row Id="11251" PostTypeId="2" ParentId="11249" CreationDate="2011-05-25T22:39:07.793" Score="13" ViewCount="" Body="&lt;p&gt;If you feel you may start more of such very large Excel type projects in the future, then you should consider installing and spending 10 hours learning the basics of R (free), which will let you do what you mention in your question, in a much more efficient manner than Excel.&lt;br&gt;&#xA;&lt;a href=&quot;http://cran.r-project.org/doc/contrib/Paradis-rdebuts_en.pdf&quot; rel=&quot;nofollow&quot;&gt;R for Beginners PDF&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can ask questions about R on &lt;a href=&quot;http://stackexchange.com/&quot;&gt;StackOverflow&lt;/a&gt; and here.&lt;/p&gt;&#xA;" OwnerUserId="4329" LastEditorUserId="4329" LastEditorDisplayName="" LastEditDate="2011-05-26T17:33:25.187" LastActivityDate="2011-05-26T17:33:25.187" CommentCount="4" />
  <row Id="11252" PostTypeId="1" CreationDate="2011-05-25T23:57:37.597" Score="2" ViewCount="147" Body="&lt;p&gt;I received a question today that I wasn't exactly sure how to answer.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have built a predictive model using a fairly basic logistic regression that works pretty well and fits our business needs.  Recently, we purchased a CRM tool that allows us to build &quot;probability&quot; scores, but only allows the end users to give integer weights to various factors.  Said differently, one can arbitrarily assign a weight of 10 points to one factor and -5 points to another with the sum of all weights representing the &quot;probability&quot; for a given entity in our database.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I am looking to do is translate my model to this new format such that the resulting score equals the calculated probability from my logistic model. This is not out of desire, but business needs.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Admittedly I am not sure how to use the calculated coefficients and &quot;adjust&quot; them to these requirements. What is the best approach, if any?  General thoughts on how to assign statistically valid integer weights to business criteria given these constraints?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any thoughts or insight will be very much appreciated.&lt;/p&gt;&#xA;" OwnerUserId="569" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-26T09:21:28.103" LastActivityDate="2011-06-03T00:03:52.680" Title="Weight variables for predictive model" Tags="&lt;logistic-regression&gt;&lt;predictive-models&gt;&lt;validation&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="11253" PostTypeId="1" AcceptedAnswerId="11278" CreationDate="2011-05-26T02:40:02.887" Score="5" ViewCount="70" Body="&lt;p&gt;If I take a set of measurements and test correlation of variable $A$ vs variable $B$ and get a significant correlation, that makes sense to me.  But what if further analysis reveals that of those factors, there is only a significant positive correlation within one group, and that group is over-represented.  Is the global correlation still valid, or is it, upon more detailed inspection a sample-bias effect?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is some graphs to explain:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The global correlation&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.stack.imgur.com/9tFJx.png&quot; alt=&quot;global correlation&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The group separated correlations&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.stack.imgur.com/cTPwA.png&quot; alt=&quot;grouping separated correlation&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="1327" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-27T06:51:02.347" LastActivityDate="2011-05-27T06:51:02.347" Title="Factor dependent correlation" Tags="&lt;correlation&gt;" AnswerCount="3" CommentCount="2" />
  <row Id="11254" PostTypeId="2" ParentId="11246" CreationDate="2011-05-26T03:12:00.933" Score="4" ViewCount="" Body="&lt;p&gt;Recursive partitioning does not provide such inferential statistics.  It is a highly exploratory method that would require an enormous multiplicity adjustment should you compute regression and error sum of squares from the result.  Better would be to do formal but flexible modeling of the two predictors, e.g., using regression splines, and if allowing for interaction, tensor splines (by adding products of spline terms).  There is a good reason there is no anova.rpart function in the R rpart package.&lt;/p&gt;&#xA;" OwnerUserId="4253" LastActivityDate="2011-05-26T03:12:00.933" CommentCount="3" />
  <row Id="11255" PostTypeId="1" CreationDate="2011-05-26T03:20:04.987" Score="8" ViewCount="85" Body="&lt;p&gt;I've noticed this issue coming up a lot in statistical consulting settings and i was keen to get your thoughts.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Context&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;I often speak to research students that have conducted a study approximately as follows: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Observational study&lt;/li&gt;&#xA;&lt;li&gt;Sample size might be 100, 200, 300, etc.&lt;/li&gt;&#xA;&lt;li&gt;Multiple psychological scales have been measured (e.g., perhaps anxiety, depression, personality, attitudes, other clinical scales, perhaps intelligence, etc.)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The researchers have read the relevant literature and have some thoughts about possible causal processes.&#xA;Often there will be some general conceptualisation of variables into antecedents, process variables, and outcome variables.&#xA;They have also often heard that structural equation modelling is more appropriate for testing overall models of the relationships between the set of variables that they are studying.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Question&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Under what conditions do you think structural equation modelling is an appropriate technique for analysing such studies?&lt;/li&gt;&#xA;&lt;li&gt;If you would not recommend structural equation modelling, what alternative techniques would you recommend?&lt;/li&gt;&#xA;&lt;li&gt;What advice would you give to researchers considering using structural equation modelling in such cases?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-26T08:12:19.967" LastActivityDate="2011-05-27T08:08:38.927" Title="Whether to use structural equation modelling to analyse observational studies in psychology" Tags="&lt;scales&gt;&lt;causal-inference&gt;&lt;sem&gt;&lt;observational-study&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="11256" PostTypeId="1" AcceptedAnswerId="11260" CreationDate="2011-05-26T04:36:57.507" Score="6" ViewCount="97" Body="&lt;p&gt;I need to do a regression with a non-normal DV for which no proper non-linear transformation (that I know of) exists:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.stack.imgur.com/7fPjE.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is a score ranging from 10 to 50, with a high peak at 10, a drop at 11 and a regular decline from 11 to 50. The distribution of residuals is not normal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.stack.imgur.com/pSjZ3.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are more or less 500 subjects in the study, and the covariates are all dichotomous.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I thought of using n-tiles and performing an ordinal regression on the resulting variable, but then I end up with a high proportion of empty cells -- not because of a low sample size, but rather because of the several covariates that need to be in the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The best I could do for now is to remove covariates and use tertiles (instead of, say, quintiles) to minimize the number of empty cells. I am not sure, though, to what extent avoiding empty cells is THAT important. Compared to the original analysis on a raw score, regressing on tertiles with less covariates seems like a lot of sacrifices.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Based on the decent sample size, and given the fact that the distribution of residuals is not THAT far from normal, I am wondering if the results of the regression would be reliable as they are.&lt;/p&gt;&#xA;" OwnerUserId="4754" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-26T06:03:45.123" LastActivityDate="2011-05-26T12:42:05.990" Title="Regression on a non-normal dependent variable" Tags="&lt;regression&gt;&lt;ordinal&gt;" AnswerCount="2" CommentCount="4" FavoriteCount="2" />
  <row Id="11257" PostTypeId="1" CreationDate="2011-05-26T04:38:23.123" Score="2" ViewCount="31" Body="&lt;h3&gt;Context:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;I am analysing some impact assessment data (measuring invertebrate richness in response to pollution), but they are unbalanced - there are not data for every site at sampling occasion, and there were more datapoints recorded &lt;em&gt;after&lt;/em&gt; the impact than &lt;em&gt;before&lt;/em&gt; the impact. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am a new user to R, and have gathered through reading on this site and others that the standard anova packages &lt;code&gt;aov()&lt;/code&gt; and &lt;code&gt;ezAnova()&lt;/code&gt; can't deal with unbalanced designs. I assume I should instead be using a package like &lt;code&gt;lme4&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I am not sure how to structure my data, or program the analysis. One of the problems is that I'm not sure how to incorporate sampling dates as the repeated measures aspect of my design. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My data has 5 columns Site code,   Date,   BeforeAfter,   ControlImpact,   Richness. &lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Questions:&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;How should I set up my data for conducting repeated measures analysis with unbalanced data in R?&lt;/li&gt;&#xA;&lt;li&gt;Should I use &lt;code&gt;lme4&lt;/code&gt; or some other package?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="4758" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-26T04:52:43.310" LastActivityDate="2011-05-26T11:05:46.140" Title="How do I set up an unbalanced repeated measures analysis (MBACI) in R?" Tags="&lt;r&gt;&lt;mixed-model&gt;&lt;repeated-measures&gt;" AnswerCount="1" />
  <row Id="11258" PostTypeId="2" ParentId="11253" CreationDate="2011-05-26T05:12:45.137" Score="7" ViewCount="" Body="&lt;p&gt;Are you familiar with &lt;a href=&quot;http://en.wikipedia.org/wiki/Simpson%27s_paradox&quot; rel=&quot;nofollow&quot;&gt;Simpson's paradox&lt;/a&gt;? This would seem to be what you're observing here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: I didn't answer your question :) What exactly you should do is to some degree context dependent (Are the groups meaningful? Does this represent a problem in the study design? etc). At the very least you should report both results IMO.&lt;/p&gt;&#xA;" OwnerUserId="26" LastActivityDate="2011-05-26T05:12:45.137" />
  <row Id="11259" PostTypeId="1" CreationDate="2011-05-26T05:17:10.670" Score="3" ViewCount="55" Body="&lt;p&gt;I need to perform regression on a skewed dependent variable. Since the residuals weren't normally distributed, I was able to transform the DV non-linearly in a way that it now approaches normality. Residuals are normal when using this transformed variable as a dependent variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the two models, The p-values for the various predictors are very much alike, and the relative sizes of the coefficients are very similar as well. I now wonder to what extent those two facts are indicators (or not) of the reliability of the coefficients and p-values obtained in the first model (using raw data). Note that the sample size is around 500.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you (and may I add that even though similar to my &lt;a href=&quot;http://stats.stackexchange.com/questions/11256/regression-on-a-non-normal-dependent-variable&quot;&gt;previous one&lt;/a&gt;, is not about the same analysis).&lt;/p&gt;&#xA;" OwnerUserId="4754" LastEditorUserId="4754" LastEditorDisplayName="" LastEditDate="2011-05-26T15:04:27.877" LastActivityDate="2011-05-26T15:04:27.877" Title="Regression on transformed and untransformed dependent variable" Tags="&lt;regression&gt;" CommentCount="6" />
  <row Id="11260" PostTypeId="2" ParentId="11256" CreationDate="2011-05-26T05:26:42.400" Score="4" ViewCount="" Body="&lt;p&gt;The &lt;strong&gt;normality&lt;/strong&gt; assumption is a &lt;em&gt;convenient&lt;/em&gt; property of model's residuals, since it enables correct inferences about the estimated parameters and critical values of many other tests are also dependent on this assumption (therefore some corrections should be made, or you may roughly take more strict rule-of-thumb criteria, increasing the acceptable range of your tests), however it &lt;em&gt;doesn't ruin&lt;/em&gt; the regression estimators. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus it &lt;em&gt;may&lt;/em&gt; (you still need to check the other assumptions) produce well behaved &lt;em&gt;predictions&lt;/em&gt;, but &lt;em&gt;data-mining&lt;/em&gt; and &lt;em&gt;hypothesis testing&lt;/em&gt; would be a bit more difficult. At this point I do agree with Huber that you need to clarify the purpose of the model.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Regarding some &lt;strong&gt;tips&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At first glance it seems that your distribution after $Y-10$ transformation, could be approximated by some truncated versions of continuous distributions: exponential (&lt;a href=&quot;http://en.wikipedia.org/wiki/Gamma_distribution&quot; rel=&quot;nofollow&quot;&gt;Gamma&lt;/a&gt;), &lt;a href=&quot;http://en.wikipedia.org/wiki/Log-normal_distribution&quot; rel=&quot;nofollow&quot;&gt;log-normal&lt;/a&gt;, &lt;a href=&quot;http://en.wikipedia.org/wiki/Pareto_distribution&quot; rel=&quot;nofollow&quot;&gt;Pareto&lt;/a&gt; or some other. So in log-normal case you still may move to something close to normality.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another option could be to try something like fitting the combo of &lt;a href=&quot;http://en.wikipedia.org/wiki/Generalised_logistic_function&quot; rel=&quot;nofollow&quot;&gt;generalized logistic function&lt;/a&gt; and &lt;a href=&quot;http://en.wikipedia.org/wiki/Logistic_regression&quot; rel=&quot;nofollow&quot;&gt;logistic regression&lt;/a&gt;. Since you DO know the upper and lower limits it seems feasible.   &lt;/p&gt;&#xA;" OwnerUserId="2645" LastActivityDate="2011-05-26T05:26:42.400" CommentCount="5" />
  <row Id="11262" PostTypeId="1" CreationDate="2011-05-26T05:50:38.113" Score="3" ViewCount="13" Body="&lt;p&gt;This question is motivated by a issue regarding network motifs.  To determine if a (connected, induced) subgraph $H$ occurs with significantly high frequency in an input network $G$, we generate an ensemble of comparison networks similar to $G$ and count the number of occurences of $H$ in them.  Thus we obtain the number of copies of $H$ in $G$ (call this frequency $f$), and the number of copies of $H$ in the comparison networks (call these frequencies $x=(x_1,x_2,...,x_n)$).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A trade-off arises:  We want $n$ to be large in order to give a better comparison, and $n$ to be small to be able to actually perform the computation.  I'm concerned, however, that sometimes $n$ is chosen too small.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question&lt;/strong&gt;: How can we determine if $n$ is too small?  I.e. how can we determine if we have sufficiently many comparisons?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My feeling is to compare $f$ with the mean of datasets $x$ and &lt;code&gt;c(x,ceiling(mean(x)))&lt;/code&gt; [in R notation].  I.e. in the second case, we add an artificial point to the dataset, which is close to the mean of $x$, integral and non-zero.  The idea is, if we instead chose to use $n+14 comparison networks, and the new comparison graph turned out to have &lt;code&gt;ceiling(mean(x))&lt;/code&gt; copies of $H$, would we have come to a different conclusion?  If so, this would indicate that $n$ is too small.&lt;/p&gt;&#xA;" OwnerUserId="386" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-27T06:53:27.230" LastActivityDate="2011-05-27T06:53:27.230" Title="Methods for checking if the number of comparisons is sufficient" Tags="&lt;statistical-significance&gt;" />
  <row Id="11263" PostTypeId="1" CreationDate="2011-05-26T06:28:43.387" Score="0" ViewCount="43" Body="&lt;p&gt;I'm sure this is a pretty standard statistics question, but I'm no expert...  I'm running an A/B test on my website to see if a change results in users adding more content.  So there are 2 basic things I'm looking at; the # of users adding at least 1 piece of content and the total # of pieces of content added by all users.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I really care much more about the total # of pieces of content added by all users.  I'll make the change permanent if I know it's at least not worse than the existing site.  So I need to know how many samples (users logging in) I need to have a 95% confidence level.  Normally I can use &lt;a href=&quot;http://www.prconline.com/education/tools/statsignificance/index.asp&quot; rel=&quot;nofollow&quot;&gt;one of the many web A/B test calculators&lt;/a&gt; that use chi-square and similar tests to figure out if my test is statistically significant or to figure out the sample size I need.  In the first case of seeing how many users added content, I can do this.  But to see the total pieces of content added among all users, I can't use those tests as their isn't a &quot;conversion&quot; event.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So what's the best way to see what sample size I need to be statistically significant to a 95% confidence level?  And how can I see if my test shows whether there was a difference?  Again I just want to make sure the new change isn't worse (or isn't &quot;much&quot; worse, do I need to and how should I define &quot;much&quot;?).&lt;/p&gt;&#xA;" OwnerUserId="4759" LastActivityDate="2011-05-26T21:39:40.440" Title="Statistical significance when A/B test has multiple values" Tags="&lt;confidence-interval&gt;&lt;statistical-significance&gt;&lt;sample-size&gt;&lt;chi-squared&gt;&lt;ab-test&gt;" AnswerCount="2" />
  <row Id="11264" PostTypeId="2" ParentId="11263" CreationDate="2011-05-26T07:27:25.430" Score="0" ViewCount="" Body="&lt;p&gt;Applying binomial distribution you can calculate variance out of number of events.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Formula for standard deviation from binomial distribution is&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sigma =  sqrt( n * p * (1-p) )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where n is number of all events and p is probability of event you observing. &#xA;To be sure 95% you have to be at lest 2 sigma away from previous result.&lt;/p&gt;&#xA;" OwnerUserId="4738" LastEditorUserId="4738" LastEditorDisplayName="" LastEditDate="2011-05-26T07:33:00.320" LastActivityDate="2011-05-26T07:33:00.320" CommentCount="2" />
  <row Id="11265" PostTypeId="2" ParentId="11246" CreationDate="2011-05-26T07:49:23.200" Score="6" ViewCount="" Body="&lt;p&gt;Just to add to Frank's points and paint a somewhat finer picture: CART/RPART is indeed highly exploratory and adding a p-value is difficult. I have seen some rare cases where people tried to use bootstrapping to get such p-value but I agree with Frank that it's not worth the effort.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for combining statistical inference with recursive partitioning more generally: The CTree and MOB methods implemented in the &quot;party&quot; package as well as several other tools outside R (especially the work of Loh &amp;amp; co-authors) tries to combine standard statistical tests (nonparametric in case of CTree, parametric for MOB) for growing the trees. They also control their error level in the sense of a closed testing procedure. However, the inference for the final fitted tree is still not trivial. The situation is similar to a regression model where you have selected interactions between your regressors in a forward search (and we know how Frank feels about that). Hence, also the &quot;party&quot; package has no anova() methods. We do provide AIC() for &quot;mob&quot; objects, though. It is not strictly valid (because we haven't done full optimization of the log-likelihood but only forward search) but it would be conservative. So the error is into the preferable direction.&lt;/p&gt;&#xA;" OwnerUserId="4760" LastActivityDate="2011-05-26T07:49:23.200" CommentCount="4" />
  <row Id="11266" PostTypeId="1" CreationDate="2011-05-26T08:32:34.833" Score="10" ViewCount="82" Body="&lt;p&gt;Hi all,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like (in the distant future) to teach statistics to kids.  For that matter, I'd be happy to know of software (obviously I am tending towards FOSS), or webapps, that are helpful in explaining statistical/probabilistic ideas to kids (or adults for that matter).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This can be used either by the instructor, the kids, or both.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suggested format of the answer: Software name, what it help teach, who should use it, link.&lt;/p&gt;&#xA;" OwnerUserId="253" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-26T09:34:12.713" LastActivityDate="2011-05-26T12:05:30.677" CommunityOwnedDate="2011-05-26T08:46:13.517" Title="Software (or webapps) for teaching kids statistics or probability?" Tags="&lt;software&gt;&lt;teaching&gt;&lt;learning&gt;" AnswerCount="4" FavoriteCount="7" />
  <row Id="11267" PostTypeId="2" ParentId="11266" CreationDate="2011-05-26T08:34:15.173" Score="3" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;GGobi&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Help teach interactive data visualization.  Including - histograms, scatter plots (2d, 3d, multi-d), with brushing/linking etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Mostly for the teacher - less for the children (but still possible)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.ggobi.org/&quot; rel=&quot;nofollow&quot;&gt;http://www.ggobi.org/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="253" LastActivityDate="2011-05-26T08:34:15.173" CommunityOwnedDate="2011-05-26T08:34:15.173" />
  <row Id="11268" PostTypeId="2" ParentId="11266" CreationDate="2011-05-26T08:42:20.983" Score="5" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;RcmdrPlugin&lt;/strong&gt;.TeachingDemos: Rcmdr Teaching Demos Plug-In&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Extending R with Rcmdr and give demos for probability and statistics ideas.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Interactive: correlation and linear regression.&lt;/li&gt;&#xA;&lt;li&gt;Static: power of a test, confidence interval, central limit theorem.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Mostly for the teacher - less for the children&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://cran.r-project.org/web/packages/RcmdrPlugin.TeachingDemos/index.html&quot; rel=&quot;nofollow&quot;&gt;http://cran.r-project.org/web/packages/RcmdrPlugin.TeachingDemos/index.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="253" LastEditorUserId="253" LastEditorDisplayName="" LastEditDate="2011-05-26T09:06:15.867" LastActivityDate="2011-05-26T09:06:15.867" CommunityOwnedDate="2011-05-26T08:46:13.517" />
  <row Id="11269" PostTypeId="2" ParentId="11266" CreationDate="2011-05-26T08:43:59.467" Score="2" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;animation&lt;/strong&gt;: A Gallery of Animations in Statistics and Utilities to Create Animations&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An R package.  Enables the teacher to create many animation that can be made into webapps.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Great for the teacher to create a children webapp.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://cran.r-project.org/web/packages/animation/index.html&quot; rel=&quot;nofollow&quot;&gt;http://cran.r-project.org/web/packages/animation/index.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Examples:&#xA;&lt;a href=&quot;http://animation.yihui.name/&quot; rel=&quot;nofollow&quot;&gt;http://animation.yihui.name/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="253" LastActivityDate="2011-05-26T08:43:59.467" CommunityOwnedDate="2011-05-26T08:46:13.517" />
  <row Id="11270" PostTypeId="1" CreationDate="2011-05-26T10:13:55.363" Score="2" ViewCount="57" Body="&lt;p&gt;My study is about customers’ perception of the effectiveness for a Malaysia corporate Weblog. In my study, customers’ perception of the effectiveness defined as perceived ease of use, perceived interactivity, and perceived trustworthiness; whereas the corporate Weblog defined as Weblog publishing software, Weblog comment system, and Weblog blogroll and hyperlink.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my study, I want to determine whether there is a relationships between:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;a)      Customers’ perception of ease of use and Weblog publishing software;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;b)      Customers’ perception of interactivity and Weblog comment system; and&lt;/p&gt;&#xA;&#xA;&lt;p&gt;c)      Customers’ perception of trustworthiness and Weblog blogroll and hyperlink.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the questionnaire, each section (perceived easy of use; perceived interactivity; perceived trustworthiness; Weblog publishing software; Weblog comment system; and Weblog blogroll and hyperlink) consist of 5 questions. As the questionnaire design in 7-likert scale (ordinal measurement), therefore I’m using non-parametric correlation (Spearman’s correlation) to analyzed the data. Besides that, as in my dissertation chapter 1, there have some research hypothesis; therefore I’m selected one-tailed significance.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Questions:&lt;/h3&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Is Spearman Correlation the appropriate method to analyse my data?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Because each section has 5 questions (eg: Perceived ease of use and Weblog publishing software), the statistical output has 10 correlation coefficients, but how should I interpret these 10 correlation coefficient into one conclusion as to whether there is a  relationship between customers’ perception of ease of use and Weblog publishing software?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If I wanted to do a regression for customers’ perceived ease of use (5 questions) and Weblog publishing software (5 questions), which regression should I use? &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="4762" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-26T11:04:02.463" LastActivityDate="2011-05-26T12:38:05.190" Title="How to analyse a study looking at relationship between one set of five items (predictors) and a second set of five items (outcomes)" Tags="&lt;regression&gt;&lt;spearman&gt;&lt;rank-correlation&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="11271" PostTypeId="2" ParentId="11257" CreationDate="2011-05-26T11:05:46.140" Score="0" ViewCount="" Body="&lt;p&gt;I believe that your scenario is generally described as one of missing data, not as an unbalanced design, which is usually reserved for cases of unequal numbers of observations between independent groups. &lt;code&gt;ezANOVA()&lt;/code&gt; from the &lt;a href=&quot;http://cran.r-project.org/web/packages/ez/index.html&quot; rel=&quot;nofollow&quot;&gt;ez package&lt;/a&gt; can handle unbalanced designs, but cannot handle missing data. &lt;code&gt;lmer()&lt;/code&gt; fromthe &lt;a href=&quot;http://cran.r-project.org/web/packages/lme4/index.html&quot; rel=&quot;nofollow&quot;&gt;lme4 package&lt;/a&gt; can handle missing data. You can either use &lt;code&gt;lmer()&lt;/code&gt; directly, as in:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;my_lmer = lmer(&#xA;    formula = richness ~ (1|site) + BeforeAfter*ControlImpact&#xA;    , data = my_data_frame&#xA;    , family = gaussian #or change to whatever model of error is appropriate&#xA;)&#xA;anova(my_lmer)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Or you could check out the ezMixed function from the ez package, which wraps lmer and produces likelihood ratios (a superior metric of evidence):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;my_mix = ezMixed(&#xA;    data = my_data_frame&#xA;    , dv = .(richness)&#xA;    , random = .(site)&#xA;    , fixed = .(BeforeAfter,ControlImpact)&#xA;    , family = gaussian&#xA;)&#xA;print(my_mix$summary)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Finally, be careful with Date as a predictor; it sounds like it may be confounded with BeforeAfter.&lt;/p&gt;&#xA;" OwnerUserId="364" LastActivityDate="2011-05-26T11:05:46.140" />
  <row Id="11272" PostTypeId="2" ParentId="11266" CreationDate="2011-05-26T12:05:30.677" Score="2" ViewCount="" Body="&lt;p&gt;&lt;a href=&quot;http://understandinguncertainty.org/view/videos&quot; rel=&quot;nofollow&quot;&gt;Videos&lt;/a&gt; and &lt;a href=&quot;http://understandinguncertainty.org/view/animations&quot; rel=&quot;nofollow&quot;&gt;animations&lt;/a&gt; from Understanding Uncertainty website.&lt;/p&gt;&#xA;" OwnerUserId="22" LastActivityDate="2011-05-26T12:05:30.677" CommunityOwnedDate="2011-05-26T12:05:30.677" CommentCount="1" />
  <row Id="11273" PostTypeId="1" CreationDate="2011-05-26T12:31:52.523" Score="2" ViewCount="40" Body="&lt;p&gt;In principle a simple question: What is the pull distribution?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(All I could find out is that it is the error-weighted distribution of estimators around the true value.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd be interested in the precise mathematical definition, how, why, and when to use it, what is it expected to look like, and if both estimator values and &quot;true&quot; value have errors associated to them, do these add up in quadrature.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: Feel free to add / change tags, I didn't find any good ones.&lt;/p&gt;&#xA;" OwnerUserId="1512" LastEditorUserId="1512" LastEditorDisplayName="" LastEditDate="2011-05-26T12:54:14.570" LastActivityDate="2011-05-26T12:54:14.570" Title="What is the pull distribution?" Tags="&lt;distributions&gt;&lt;data-mining&gt;" CommentCount="4" />
  <row Id="11274" PostTypeId="2" ParentId="11270" CreationDate="2011-05-26T12:38:05.190" Score="3" ViewCount="" Body="&lt;p&gt;About your first question (using Spearman rank correlation with ordinal scales), I think you will find useful responses on this site (search for spearman, likert, ordinal or scale).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;About your second question: As I understand the situation, for each dimension (what you call a &quot;section&quot;), you have a set of five questions scored on a 7-point Likert-type scale. If those five questions all define a single construct, that is if we can consider they form an unidimensional scale (such an assumption might be checked, anyway), why don't you use a summated scale score (add up the individuals response to the five responses)? This way, your problem would vanish because then you only have one single estimate for the correlation between, say Perceived ease of use and Weblog publishing. Another option is to use &lt;a href=&quot;http://en.wikipedia.org/wiki/Canonical_correlation&quot; rel=&quot;nofollow&quot;&gt;Canonical Correlation Analysis&lt;/a&gt; (CCA) which allows to build maximally correlated linear combinations of two-block data structure, as described in &lt;a href=&quot;http://stats.stackexchange.com/questions/8300/correlation-analysis-and-correcting-p-values-for-multiple-testing/8323#8323&quot;&gt;this response&lt;/a&gt;. The pattern of loadings on the two blocks will help you to summarize which item contribute the most information in each block and how they relate each other (under the constraint imposed by CCA). The canonical correlation itself will give you a single number to summarize the association between any two section (again, when considering a linear combination of the 5 questions that compose a section).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For your third question, I would suggest considering &lt;a href=&quot;http://en.wikipedia.org/wiki/Partial_least_squares_regression&quot; rel=&quot;nofollow&quot;&gt;PLS regression&lt;/a&gt;, where you define one block of variables as outcomes and the other one as predictors. The idea of PLS regression is to build successive linear combinations of the variables belonging to each block such that their covariance (instead of the correlation like in CCA) is maximal, within a regression approach (because there's an asymmetric deflation process when constructing the next linear combination). In other words, you build &quot;latent variables&quot; that account for a maximum of information included in the block of predictors while allowing to predict the block of outcomes with minimal error. As you are working with ordinal data, you can even preprocess each variable with optimal scaling if you want, see for example the &lt;a href=&quot;http://cran.r-project.org/web/packages/homals/index.html&quot; rel=&quot;nofollow&quot;&gt;homals&lt;/a&gt; package in R, and the papers referenced in the dcocumentation.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-05-26T12:38:05.190" CommentCount="2" />
  <row Id="11275" PostTypeId="2" ParentId="11256" CreationDate="2011-05-26T12:42:05.990" Score="3" ViewCount="" Body="&lt;p&gt;Ordinal regression is not affected by empty cells of Y.  Quantile grouping is not required unless you just want to reduce computational burden.  Proportional odds or continuation ratio ordinal logistic models are likely to be able to handle the distribution of Y you plotted (with no grouping of Y).&lt;/p&gt;&#xA;" OwnerUserId="4253" LastActivityDate="2011-05-26T12:42:05.990" CommentCount="3" />
  <row Id="11276" PostTypeId="2" ParentId="6723" CreationDate="2011-05-26T12:43:37.610" Score="2" ViewCount="" Body="&lt;p&gt;Stepwise regression in the absence of penalization is frought with so many difficulties that I'm surprised people are still using it.  The web has long lists of problems, starting with the extremely low probability of finding the &quot;right&quot; model.&lt;/p&gt;&#xA;" OwnerUserId="4253" LastActivityDate="2011-05-26T12:43:37.610" />
  <row Id="11277" PostTypeId="1" CreationDate="2011-05-26T12:51:55.790" Score="4" ViewCount="49" Body="&lt;p&gt;In a nutshell, here's what I have:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Annual population estimates for the State&lt;/li&gt;&#xA;&lt;li&gt;Periodical (5 years) age, population, and basic census data per zones&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Here's what I want to do:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Create a simplistic model to generate the data for the missing years between the period for each zone, and have the total sums add up to where the yearly state population estimate is.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;All in all, I'm looking for a non complicated statistical model that is able to generate values with an acceptable (doesn't have to be super high) precision.&lt;/p&gt;&#xA;" OwnerUserId="59" LastEditorUserId="59" LastEditorDisplayName="" LastEditDate="2011-05-26T13:09:38.747" LastActivityDate="2011-05-27T23:41:14.597" Title="Is there a simplistic model to disaggregate census data based on years and smaller zones?" Tags="&lt;estimation&gt;&lt;census&gt;&lt;disaggregation&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="11278" PostTypeId="2" ParentId="11253" CreationDate="2011-05-26T13:45:16.910" Score="6" ViewCount="" Body="&lt;p&gt;I agree with JMS advice, that the answer is totally context dependent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But what you are looking at may also be considered a &lt;a href=&quot;http://en.wikipedia.org/wiki/Moderation_%28statistics%29&quot; rel=&quot;nofollow&quot;&gt;moderation effect&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In statistics, moderation occurs when&#xA;  the relationship between two variables&#xA;  depends on a third variable.  &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;(quoted from &lt;a href=&quot;http://en.wikipedia.org/wiki/Moderation_%28statistics%29&quot; rel=&quot;nofollow&quot;&gt;wikipedia&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A moderation is statistically significant if in a multiple regression analyses the interaction of the predictor with the third variable is significant.&lt;/p&gt;&#xA;" OwnerUserId="442" LastActivityDate="2011-05-26T13:45:16.910" />
  <row Id="11279" PostTypeId="2" ParentId="11249" CreationDate="2011-05-26T13:55:58.253" Score="8" ViewCount="" Body="&lt;p&gt;Although I would always recommend to use R, you could nevertheless achieve what you want with python.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is at least a package for reading &lt;a href=&quot;http://pypi.python.org/pypi/dbf/&quot; rel=&quot;nofollow&quot;&gt;dbf files&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furthermore, &lt;a href=&quot;http://www.scipy.org/&quot; rel=&quot;nofollow&quot;&gt;scipy&lt;/a&gt; offers a great range of functions for statistical analysis. For example the library &lt;a href=&quot;http://dirac.cnrs-orleans.fr/plone/software/scientificpython/&quot; rel=&quot;nofollow&quot;&gt;ScientifyPython&lt;/a&gt; probably contains &lt;a href=&quot;http://dirac.cnrs-orleans.fr/ScientificPython/ScientificPythonManual/toc-Scientific.Statistics-module.html&quot; rel=&quot;nofollow&quot;&gt;the functions&lt;/a&gt; you need.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The best idea is to check &lt;a href=&quot;http://www.scipy.org/&quot; rel=&quot;nofollow&quot;&gt;scipy.org&lt;/a&gt;. There you will find what you want.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(But learning R is a great idea!!)&lt;/p&gt;&#xA;" OwnerUserId="442" LastActivityDate="2011-05-26T13:55:58.253" CommentCount="1" />
  <row Id="11280" PostTypeId="1" CreationDate="2011-05-26T14:51:26.587" Score="2" ViewCount="57" Body="&lt;p&gt;I am new to R and some help would be of great use to me.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically, I need to perform a GLM analysis with negative binomial errors and with fixed factors, no covariates and no random effects. My factors are of type: year (1-4), site (1-3), sex (1-2), age (1-3), with a sample size of around 5000.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently I am fitting GLMs with negative binomial errors, using the &lt;code&gt;glm.nb&lt;/code&gt; routine in the MASS library of R. I then need to remove all nonsignificant interactions by hand (using the R console). Can someone point me into the right direction of how can I rather automate the process, in such a way that R will choose desired factors by trying different combinations and ignoring nonsignificant ones?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(I am sorry if the question lack details, I can add more info if you need it.)&lt;/p&gt;&#xA;" OwnerUserId="4766" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-26T16:02:15.300" LastActivityDate="2011-05-27T09:30:47.783" Title="Automatisation of GLM analysis with negative binomial errors" Tags="&lt;r&gt;&lt;model-selection&gt;&lt;glm&gt;" AnswerCount="2" />
  <row Id="11281" PostTypeId="2" ParentId="4753" CreationDate="2011-05-26T15:05:23.787" Score="1" ViewCount="" Body="&lt;p&gt;One has to be careful about the meaning of the word sparse. Your matrix contains many zeroes and one may represent such a matrix in a sparse way (to save on storage). But since the figures represent co-occurrences these zeroes are still to be considered informative (they are not missing; they are not structurally zero) and should therefore be taken into account when modeling the content of the matrix. The many zeroes and the skewness (approximately geometric) would suggest to use generalized forms of bilinear models (see de Falguerolles/Gabriel : Generalized Linear-Bilinear Models). The R-package gnm supports this type of models. The sparse variants of PCA/SVD you are referring to rather relate to L1-regularisations of the factorial representation such that estimated loadings come out as sparse (many zeroes).&lt;/p&gt;&#xA;" OwnerUserId="4767" LastActivityDate="2011-05-26T15:05:23.787" />
  <row Id="11282" PostTypeId="2" ParentId="11280" CreationDate="2011-05-26T15:41:30.293" Score="2" ViewCount="" Body="&lt;p&gt;The &lt;code&gt;stepAIC&lt;/code&gt; function in MASS can perform the kinds of variable selection you are looking for.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In addition, the &lt;code&gt;leaps&lt;/code&gt; package appears to have similar capacities. That being said, I have not used it, so cannot speak directly on its efficacy.&lt;/p&gt;&#xA;" OwnerUserId="656" LastActivityDate="2011-05-26T15:41:30.293" />
  <row Id="11283" PostTypeId="1" AcceptedAnswerId="11285" CreationDate="2011-05-26T16:26:07.750" Score="2" ViewCount="53" Body="&lt;p&gt;I am running a logistic model on insurance data. I have a field agent gender which matters for a channel A (say) and doesn't matter for B. I want to put null values in case of B. The only thing I risk is exclusion by SAS (as SAS excludes every missing case by default). I heard that pairwise exclusion can solve my problem. Please tell me the following:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;What is the difference between normal exclusion and pairwise exclusion?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;How to do this is SAS?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;What is listwise exclusion?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="1763" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-27T06:45:52.127" LastActivityDate="2011-05-27T06:45:52.127" Title="Pairwise exclusions" Tags="&lt;modeling&gt;&lt;dataset&gt;&lt;sas&gt;&lt;missing&gt;" AnswerCount="2" FavoriteCount="1" />
  <row Id="11284" PostTypeId="2" ParentId="11283" CreationDate="2011-05-26T16:51:15.040" Score="4" ViewCount="" Body="&lt;p&gt;There are two main types of traditional treatments of missing data.&#xA;These are:&#xA;1) listwise&#xA;2) pairwise&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Listwise is (from what you have said) the default in SAS. It means that you exclude any observation that has missing values in any of the terms in your model.&#xA;The advantage of this is that it ensures that all variables have the same n in the model. The disadvantage is that if you have one variable with a high proportion of missing data, much of the rest of the data can be ignored. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Pairwise, by contrast, only excludes cases where the value is missing for that particular variable. This has the advantage of keeping more data in the model that listwise does. However, the disadvantage is that some of your results will be based on different subsets of the data, and this can cause problems with p values and confidence intervals. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A better approach is probably multiple imputation where you attempt to predict the missing values using all of the data you have. This normally involves simulating a number of datasets with the missing data filled in, and then performing all analyses on each of the datasets individually, and averaging the results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A good paper on treatment of missing data is Graham 2009 &lt;a href=&quot;http://www.iapsych.com/articles/graham2009.pdf&quot; rel=&quot;nofollow&quot;&gt;Missing Data: Making it work in the real world&lt;/a&gt; which goes into much more detail than I have here. You also need to determine if the missingness in your data is random or not, as if the probability of missingness depends on exogenous variables to your dataset, then none of these approaches will work correctly. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A good resource on multiple imputation is &lt;a href=&quot;http://www.multiple-imputation.com/&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.ats.ucla.edu/stat/sas/modules/missing.htm&quot; rel=&quot;nofollow&quot;&gt;This&lt;/a&gt; article on missing data in SAS may prove helpful, or someone else may answer as to how to do this in SAS. &lt;/p&gt;&#xA;" OwnerUserId="656" LastActivityDate="2011-05-26T16:51:15.040" CommentCount="1" />
  <row Id="11285" PostTypeId="2" ParentId="11283" CreationDate="2011-05-26T18:11:50.257" Score="6" ViewCount="" Body="&lt;p&gt;This doesn't sound like a missing data problem to me: it sounds like a question of model structure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Distilling it to its essence, it seems you have two independent categorical variables gender ($X$, say) and &quot;channel&quot; ($Y$) and a binary response ($Z$).  Conceptually the model is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$logit(\Pr(Z=1)) = \beta_0 + \beta_1 X + \beta_2 Y + \varepsilon$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;when $Y$ = &quot;A&quot; and otherwise&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$logit(\Pr(Z=1)) = \beta_0 + \beta_2 Y + \varepsilon$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;when $Y \ne $ &quot;A&quot; (with parameters $\beta_0$, $\beta_1$, and $\beta_2$ and zero-mean random error $\varepsilon$).  If this interpretation is correct, just set $X = 0$ in the data whenever $Y \ne $ &quot;A&quot;.  This will cause such cases to have no effect on the value of $\beta_1$, which will be estimated solely from the other cases where $Y$ = &quot;A&quot;.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-05-26T18:11:50.257" CommentCount="4" />
  <row Id="11286" PostTypeId="2" ParentId="10890" CreationDate="2011-05-26T18:19:51.773" Score="3" ViewCount="" Body="&lt;p&gt;I agree with @Michael's description of endogeneity---this is about a problem with the variables that you include and their relationship to the variables that you do not (i.e., the stuff in the error term). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unobserved heterogeneity is typically about unobservable componenents of the effects that you are estimating. Continuing with @Michael's education example, unobserved heterogeneity might be that some people have higher returns (e.g., increases in wages) from going to school than others. Let the returns for person $i$ be $\beta + b_i$ with $\mathbb{E}(b_i) = 0$. We have&#xA;$$\begin{equation*} y_i = x_i (\beta + b_i) + w^\prime_i \gamma + \epsilon_i, \end{equation*}$$&#xA;where $y_i$ is (typically, log) income, $x_i$ is years of education, and $w_i$ is a set of other controls. An example of endogeneity is when $x_i$ is correlated with $\epsilon_i$ (e.g., education is correlated with IQ, which is not among our other predictors).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we estimate a single coefficient, we have&#xA;$$\begin{equation*} y_i = x_i \beta + w^\prime_i \gamma + (\epsilon_i + b x_i) = x_i \beta + w^\prime_i \gamma + \tilde{\epsilon}_i \end{equation*}$$&#xA;See that the included variable $x_i$ is correlated with the error term $\tilde{\epsilon}_i$, inducing the same problems as the case of endogeneity.&lt;/p&gt;&#xA;" OwnerUserId="401" LastActivityDate="2011-05-26T18:19:51.773" />
  <row Id="11287" PostTypeId="2" ParentId="11072" CreationDate="2011-05-26T18:25:22.063" Score="2" ViewCount="" Body="&lt;p&gt;The $R^2$ in regression is given that name because it is the correlation between $y_i$ and $\hat{y}_i$ squared. You could calculate the correlation between $z_i$ and $\hat{z}_i$ in your case, square it, and use that as a measure of goodness-of-fit. I can't say what the statistical properties of this measure will be for your particular case, however.&lt;/p&gt;&#xA;" OwnerUserId="401" LastActivityDate="2011-05-26T18:25:22.063" CommentCount="1" />
  <row Id="11288" PostTypeId="2" ParentId="10613" CreationDate="2011-05-26T18:50:27.493" Score="2" ViewCount="" Body="&lt;p&gt;Under the null hypothesis, your test statistic $T$ has the distribution $F(t)$ (e.g., standard normal). We show that the p-value $P=F(T)$ has a probability distribution&#xA;$$\begin{equation*} \Pr(P &amp;lt; p) = \Pr(F^{-1}(P) &amp;lt; F^{-1}(p)) = \Pr(T &amp;lt; t) \equiv p; \end{equation*}$$&#xA;in other words, $P$ is distributed uniformly. This holds so long as $F(\cdot)$ is invertible, a necessary condition of which is that $T$ is not a discrete random variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This result is general: the distribution of an invertible CDF of a random variable is uniform on $[0,1]$.&lt;/p&gt;&#xA;" OwnerUserId="401" LastEditorUserId="401" LastEditorDisplayName="" LastEditDate="2011-05-27T00:19:15.937" LastActivityDate="2011-05-27T00:19:15.937" CommentCount="1" />
  <row Id="11289" PostTypeId="1" CreationDate="2011-05-26T19:27:05.283" Score="6" ViewCount="97" Body="&lt;p&gt;Hi everyone,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am looking for some probability inequalities for sums of unbounded random variables. I would really appreciate it if anyone can provide me some thoughts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My problem is to find an exponential upper bound over the probability that the sum of unbounded i.i.d. random variables, which are in fact the multiplication of two i.i.d. Gaussian, exceeds some certain value, i.e., $\mathrm{Pr}[ X  \geq \epsilon\sigma^2 N] \leq \exp(?)$, where $X = \sum_{i=1}^{N} w_iv_i$, $w_i$ and $v_i$ are generated i.i.d. from $\mathcal{N}(0, \sigma)$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried to use the Chernoff bound using moment generating function (MGF), the derived bound is given by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\begin{eqnarray}&#xD;&#xA;\mathrm{Pr}[ X  \geq \epsilon\sigma^2 N] &amp;amp;\leq&amp;amp; \min\limits_s \exp(-s\epsilon\sigma^2 N)g_X(s) \\&#xD;&#xA; &amp;amp;=&amp;amp; \exp\left(-\frac{N}{2}\left(\sqrt{1+4\epsilon^2} -1 + \log(\sqrt{1+4\epsilon^2}-1) - \log(2\epsilon^2)\right)\right)&#xD;&#xA;\end{eqnarray}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $g_X(s) = \left(\frac{1}{1-\sigma^4 s^2}\right)^{\frac{N}{2}}$ is the MGF of $X$. But the bound is not so tight. The main issue in my problem is that the random variables are unbounded, and unfortunately I can not use the bound of Hoeffding inequality.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I will be to happy if you help me find some tight exponential bound .&#xA;Thanks in advance&lt;/p&gt;&#xA;" OwnerUserId="4770" LastEditorUserId="4770" LastEditorDisplayName="" LastEditDate="2011-05-27T13:58:25.183" LastActivityDate="2011-05-27T13:58:25.183" Title="Probability inequalities" Tags="&lt;probability&gt;&lt;mathematical-statistics&gt;" CommentCount="13" FavoriteCount="1" />
  <row Id="11290" PostTypeId="1" AcceptedAnswerId="11314" CreationDate="2011-05-26T20:06:04.120" Score="2" ViewCount="47" Body="&lt;p&gt;Hi,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I came across sampling method &quot;Propensity Weighting Sampling/RIM&quot;. But I have not much idea on these survey methods. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anybody please refer me some literature on this topic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regards,&#xA;Ari&lt;/p&gt;&#xA;" OwnerUserId="4278" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-27T14:03:33.900" LastActivityDate="2011-05-27T16:14:16.013" Title="What is a propensity weighting sampling / RIM?" Tags="&lt;sampling&gt;" AnswerCount="1" />
  <row Id="11291" PostTypeId="2" ParentId="11236" CreationDate="2011-05-26T20:15:53.020" Score="3" ViewCount="" Body="&lt;p&gt;From the &lt;a href=&quot;http://stat.ethz.ch/R-manual/R-devel/library/lattice/html/qqmath.html&quot; rel=&quot;nofollow&quot;&gt;documentation for qqmath&lt;/a&gt; it seems that the default behavior is to compare the empirical quantiles to those of a normal distribution. So what the QQ plot for $\sigma^2$ (which is the error variance) is telling you is that its marginal posterior distribution is not normal, which you would fully expect. So it's not really informative or meaningful. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The remaining panels are telling you that the marginal posterior distributions of the regression coefficients are basically normal, perhaps with slightly heavier tails because they are marginalized over $\sigma^2$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The MCMC regression does give you more information in that you have a full posterior distribution over all the parameters, so you can answer questions like &quot;Given the data I've seen and my prior information, what is the probability that the tempwarmer coefficient is positve&quot;. You can also look at joint distributions of two or more parameters, or functions of the coefficients, and on and on. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I should mention that these QQ plots aren't really giving you an indication of goodness of fit. For that you would want to look at some other measures - posterior predictive distributions, for example, or MCMCregress may be able to do formal Bayesian model comparisons for you too. I'm not familiar with that particular function but a cursory Googling turned up some potential leads.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All that said, if the lm() fit is good I wouldn't loose much sleep over it. But I would most wholeheartedly encourage any forays into Bayesian analysis :)&lt;/p&gt;&#xA;" OwnerUserId="26" LastActivityDate="2011-05-26T20:15:53.020" CommentCount="3" />
  <row Id="11292" PostTypeId="1" CreationDate="2011-05-26T21:13:05.773" Score="3" ViewCount="66" Body="&lt;p&gt;I am performing a retrospective study on patients looking at the size of their nostril (continuous variable measured in millimetres) and the need for treatment which is either conservative or surgical (this is a categorical variable).&lt;br&gt;&#xA;Sample size is only 15.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What would be the right test to compare groups to determine if size of nostril was a significant factor in predicting method of treatment?&lt;/p&gt;&#xA;" OwnerUserId="4772" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-27T12:59:13.753" LastActivityDate="2011-05-27T12:59:13.753" Title="How to compare two groups of patients with a continuous outcome?" Tags="&lt;statistical-significance&gt;" AnswerCount="1" />
  <row Id="11293" PostTypeId="2" ParentId="11253" CreationDate="2011-05-26T21:28:02.027" Score="3" ViewCount="" Body="&lt;p&gt;The previous comments are all good, but with group sample sizes of 5, 7, and 11, I wouldn't trust any of their correlations as far as I could throw them.  You'll need to give the overall &lt;em&gt;r&lt;/em&gt; a wide confidence interval as well.  btw Nice job on the graph.&lt;/p&gt;&#xA;" OwnerUserId="2669" LastActivityDate="2011-05-26T21:28:02.027" CommentCount="1" />
  <row Id="11294" PostTypeId="2" ParentId="11292" CreationDate="2011-05-26T21:36:45.680" Score="7" ViewCount="" Body="&lt;p&gt;Note that the sample size is very small and it is therefore higly likely that you run into power problems if you get a nonsignificant result. Therefore, definitely report the effect size whatever the result of your test will be (i.e., the differences in nostril size between the two treatment groups, in relation to the standard deviations = &lt;a href=&quot;http://en.wikipedia.org/wiki/Effect_size&quot; rel=&quot;nofollow&quot;&gt;cohen's d et al.&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a test, the &lt;a href=&quot;http://en.wikipedia.org/wiki/Student%27s_t-test&quot; rel=&quot;nofollow&quot;&gt;t-test&lt;/a&gt; for independent samples seems to be appropriate. &lt;/p&gt;&#xA;" OwnerUserId="442" LastActivityDate="2011-05-26T21:36:45.680" CommentCount="1" />
  <row Id="11295" PostTypeId="2" ParentId="11263" CreationDate="2011-05-26T21:39:40.440" Score="1" ViewCount="" Body="&lt;p&gt;The perspective Ralu is using is basically p is the probability of A and for the binomial he's saying you have the events A and not A which for you is B and that's your event space. Since you don't know your actual value for P(A) and assuming you don't have a good guess for it you'll want to use a conservative estimate of .5 plugging that into the equation in the other answer is going to imply you need 16 observations in your sample. However I'm not sure binomial is the best choice in this case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When determining sample size there are two things you're going to want to decide. First is your confidence level (which you have as 95%) the next is you're going to want to decide what margin of error is acceptable for your analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It might be worth considering &lt;a href=&quot;http://books.google.com/books?id=ZvPKTemPsY4C&amp;amp;lpg=PP1&amp;amp;dq=wackerly%20mathematical%20statistics&amp;amp;pg=PA423#v=onepage&amp;amp;q=Example%208.10&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;example 8.10 in Wackerly et al&lt;/a&gt;. Since it actually looks at determining sample size for two sample groups which is your situation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The explanation in the example seems thorough enough (if you have questions though please ask), but in case you don't click to take a look it will result ( 2n = 1/[$(1/1.96){}^{2}$/8] = 31 so n = 62 ) in requiring 31 in each group so 62 in total. Notice that this is nearly four times the size of what the other method suggested it is also larger than the 40 samples usually cited for the Central Limit Theorem which gives it good properties. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However that has a fairly large margin of error (1 = 100% margin of error). Let's say instead you wanted a very small margin of error such as 4% &lt;/p&gt;&#xA;&#xA;&lt;p&gt;2n = 1/[$(.04/1.96){}^{2}$/8] = 19208 so n = 38416&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Remember though in this example we assumed the range was 8 and used 4*Sigma is approximately equal to the range. Which may or may not make sense for your problem. Range is Max - Min. So if in your current data you see a very different range you may want to use that value instead to recalculate accordingly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for determining whether there was a difference you're going to want to use a hypothesis test. In particular you're going to want to use a &lt;a href=&quot;http://www.acastat.com/Statbook/ttest2.htm&quot; rel=&quot;nofollow&quot;&gt;two sample T-Test&lt;/a&gt;. Your null hypothesis is that the means are equal. In your case you want to know if the new one is greater than the old one so you'll want a one tail (also called a directional hypothesis) alternative hypothesis. Once you've calculated the T statistic using the formulas in that link you'll need to find the corresponding critical value &lt;a href=&quot;http://www.statsoft.com/textbook/distribution-tables/#t&quot; rel=&quot;nofollow&quot;&gt;from a table&lt;/a&gt;. For your confidence level you'll want to know if the test statistic is greater than this value: 1.644854 if it is than the new mean which we're going to consider as mu1 is greater than the old one mu2. If it is not greater than that value you fail to reject because your evidence isn't strong enough.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hopefully this helps!&lt;/p&gt;&#xA;" OwnerUserId="4325" LastActivityDate="2011-05-26T21:39:40.440" CommunityOwnedDate="2011-05-26T21:39:40.440" CommentCount="1" />
  <row Id="11296" PostTypeId="1" CreationDate="2011-05-27T00:39:52.903" Score="3" ViewCount="111" Body="&lt;p&gt;I have a table  with four groups (4 BMI groups) as the independent variable (factor). I have a dependent variable that is &quot;percent mother smoking in pregnancy&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it permissible to use ANOVA for this or do I have to use chi-square or some other test? &lt;/p&gt;&#xA;" OwnerUserId="4774" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-27T02:16:45.603" LastActivityDate="2011-05-29T13:18:28.267" Title="Using ANOVA on percentages?" Tags="&lt;anova&gt;" AnswerCount="3" />
  <row Id="11297" PostTypeId="2" ParentId="11296" CreationDate="2011-05-27T01:05:52.183" Score="5" ViewCount="" Body="&lt;p&gt;It depends on how close the responses within different groups are to 0 or 100%.  If there are a lot of extreme values (i.e. many values piled up on 0 or 100%) this will be difficult.  (If you don't know the &quot;denominators&quot;, i.e. the numbers of subjects from which the percentages are calculated, then you can't use contingency table approaches anyway.) If the values within groups are more reasonable, then you can transform the response variable (e.g. classical arcsine-square-root or perhaps logit transform). There are a variety of graphical (preferred) and null-hypothesis testing (less preferred) approaches for deciding whether your transformed data meet the assumptions of ANOVA adequately (homogeneity of variance and normality, the former more important than the latter). Graphical tests: boxplots (homogeneity of variance) and Q-Q plots (normality) [the latter should be done within groups, or on residuals].  Null-hypothesis tests: e.g. Bartlett or Fligner test (homogeneity of variance), Shapiro-Wilk, Jarque-Bera, etc.&lt;/p&gt;&#xA;" OwnerUserId="2126" LastActivityDate="2011-05-27T01:05:52.183" />
  <row Id="11298" PostTypeId="2" ParentId="11296" CreationDate="2011-05-27T02:23:47.557" Score="8" ViewCount="" Body="&lt;p&gt;There is a difference between having a binary variable as your dependent variable and having a proportion as your dependent variable.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Binary dependent variable&lt;/strong&gt;: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;This sounds like what you have. (i.e., each mother either smoked or she did not smoke)&lt;/li&gt;&#xA;&lt;li&gt;In this case I would not use ANOVA. Logistic regression with some form of coding (perhaps dummy coding) for the categorical predictor variable is the obvious choice if you are conceptualising the binary variable as the dependent variable (otherwise you could do chi-square).&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Proportion as dependent variable&lt;/strong&gt;: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;This does not sound like what you have. (i.e., you don't have data on the proportion of total waking time that a mother was smoking during pregnancy in a sample of smoking pregnant women).&lt;/li&gt;&#xA;&lt;li&gt;In this case, ANOVA and standard linear model approaches in general may or may not be  reasonable for your purposes. See @Ben Bolker's answer for a discussion of the issues.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastActivityDate="2011-05-27T02:23:47.557" />
  <row Id="11299" PostTypeId="1" CreationDate="2011-05-27T04:08:42.267" Score="6" ViewCount="46" Body="&lt;h3&gt;Context:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;I'm investigating behaviour in a clinical study involving children. I had both parents and teachers completing  questionnaires to inform an understanding of the same underlying constructs, for example reactive aggression. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;At the conclusion of data collection I have parent data in all cases, n=55, and teacher data for 41 cases - therefore, I have 14 cases where I only have the parent data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the purposes of our study it makes sense to aggregate parent and teacher observations for each case. As I have a small sample and I do have parent data for every case, deleting cases in a pairwise manner does not seem a viable option. There are several variables for which I need to do this, all of which seem to correlate well. Before aggregating data, I thought it would make sense to address the missing data issue.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With regards to 'substituting' I have done some brief reading and am familiar with the basic 'within variable' substitution options, however, I thought there may be a more powerful method available where I could use: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(a) the parent score we have in every case for every variable, n=55, 100% of cases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(b) our understanding of the relationship between parent and teacher scores in cases where we have both, n=41, 74.5% of cases&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Question:&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Does the general idea outlined above seem reasonable?&lt;/li&gt;&#xA;&lt;li&gt;What would be a good algorithm for implementing it in detail?&lt;/li&gt;&#xA;&lt;li&gt;How could it be implemented in SPSS?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="4775" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-27T06:25:27.220" LastActivityDate="2011-05-27T10:02:06.747" Title="Getting an average measurement based on two raters for cases where data is missing for one rater" Tags="&lt;spss&gt;&lt;missing-data&gt;&lt;data-imputation&gt;" AnswerCount="1" FavoriteCount="1" />
  <row Id="11300" PostTypeId="1" CreationDate="2011-05-27T04:59:18.913" Score="2" ViewCount="29" Body="&lt;p&gt;I have a particular semiparametric model which I'm fitting via MCMC. One of the model parameters I have &quot;semiparametric'ed&quot; away (say $\alpha$) is known to lie between two other parameters, $\theta_1$ and $\theta_2$. Since I have a series of samples $(\theta_1^t, \theta_2^t)$ I also have a series of interval estimates for $\alpha$. What is a reasonable way to summarize these? I can think of doing something like putting down a grid and just recording for each gridpoint whether each sample covers it or not. But I worry about efficiency since I don't have a great sense of what a plausible range is, and I actually have hundreds to thousands of $\alpha$'s. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(I'm also welcoming retags, since I can't seem to find any I like...)&lt;/p&gt;&#xA;" OwnerUserId="26" LastActivityDate="2011-05-27T09:12:29.650" Title="Summarizing samples of an interval" Tags="&lt;estimation&gt;" AnswerCount="1" CommentCount="5" />
  <row Id="11302" PostTypeId="2" ParentId="8903" CreationDate="2011-05-27T06:31:45.607" Score="0" ViewCount="" Body="&lt;p&gt;According to &lt;a href=&quot;http://en.wikipedia.org/wiki/Tf-idf&quot; rel=&quot;nofollow&quot;&gt;Wikipedia's article of tf-idf&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The term count in the given document is simply the number of times a given term appears in &#xA;  that document. This count is usually normalized to prevent a bias towards longer documents&#xA;  (which may have a higher term count regardless of the actual importance of that term in the &#xA;  document) to give a measure of the importance of the term t within the particular document d&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So, normalize the frequency of a term t by the length of the document d in which it occurs.  Then you can compute cosine similarity between your tf-idf vectors.&lt;/p&gt;&#xA;" OwnerUserId="4777" LastActivityDate="2011-05-27T06:31:45.607" />
  <row Id="11303" PostTypeId="2" ParentId="11231" CreationDate="2011-05-27T06:59:20.680" Score="1" ViewCount="" Body="&lt;p&gt;The classic problem with PCR is that principal components corresponding to small eigenvalues (and hence discarded) can be significant for explaining the dependent variable. One of the solutions to this problem is to use &lt;a href=&quot;http://en.wikipedia.org/wiki/Partial_least_squares_regression&quot; rel=&quot;nofollow&quot;&gt;PLS regression&lt;/a&gt;. In PLS regression the principal components are picked  to have maximal correlation with dependent variable.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-05-27T06:59:20.680" CommentCount="1" />
  <row Id="11304" PostTypeId="2" ParentId="11255" CreationDate="2011-05-27T08:08:38.927" Score="6" ViewCount="" Body="&lt;p&gt;Disclaimer: I consider myself an experiemtal psychologist with an emphasis on experimental. Hence, I have a natural unease with designs like this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To answer your first and second question: I think for a design like this a SEM or, depending on the number of variables involved, mediation or moderation analyses is the natural way of dealing with the data. I have no good idea what else to recommend. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For your third question: I think the main advantage with a design like this is it's main disadvantage. Namely that you (given enough variables) &lt;strong&gt;will&lt;/strong&gt; find significant results. The question is, how you interpret these results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That is, you can look at so many hypotheses (some more some less inspired by the relevant literature) that you will probably find something significant (not in the literal sense of rejecting a SEM) that will be interpratable in a psychological sense. Therefore, my advice to anyone doing this would be twofold:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Stress the problem with causal interpretation of these designs. I am not an expert in this but know, that a fully cross-sectional design can hardly be interpreted causal, independent of how intuitively plausible that may sound. More advanced designs like cross-lagged pnael designs or stuff like this is needed for causal interpetations. I think the work by Shadish, Cook &amp;amp; Campbell (or at least some of them) are a good ressource for further discussion of these topics.&lt;/li&gt;&#xA;&lt;li&gt;Stress the individual responsibility and scientific ethics. If you see that your initial idea is not supported by the data, it is the natural next step to inspect the data further. However, you shall never rely on &lt;a href=&quot;http://psr.sagepub.com/content/2/3/196.abstract?ijkey=083445b69bb4d27671d6370069e53b88e6e702f1&amp;amp;keytype2=tf_ipsecsha&quot; rel=&quot;nofollow&quot;&gt;HARKing&lt;/a&gt; (Hypothesizing After the Results are Known; &lt;a href=&quot;http://psr.sagepub.com/content/2/3/196.abstract?ijkey=083445b69bb4d27671d6370069e53b88e6e702f1&amp;amp;keytype2=tf_ipsecsha&quot; rel=&quot;nofollow&quot;&gt;Kerr, 1998&lt;/a&gt;, see also &lt;a href=&quot;http://psych.colorado.edu/~willcutt/res_meth/Maxwell_2004.pdf&quot; rel=&quot;nofollow&quot;&gt;Maxwell, 2004&lt;/a&gt;). That is, you should stress that there is a thin line between a reasonable adaption of your hypotheses given the data and cherry picking of significant results.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="442" LastActivityDate="2011-05-27T08:08:38.927" CommentCount="2" />
  <row Id="11305" PostTypeId="2" ParentId="726" CreationDate="2011-05-27T08:11:49.203" Score="4" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;&quot;The first time I was in a statistics course, I was there to teach it&quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;John Tukey (&lt;a href=&quot;http://www.stat.berkeley.edu/~brill/Papers/life.pdf&quot; rel=&quot;nofollow&quot;&gt;link&lt;/a&gt;)&lt;/p&gt;&#xA;" OwnerUserId="74" LastActivityDate="2011-05-27T08:11:49.203" CommunityOwnedDate="2011-05-27T08:11:49.203" />
  <row Id="11306" PostTypeId="2" ParentId="11300" CreationDate="2011-05-27T09:12:29.650" Score="0" ViewCount="" Body="&lt;p&gt;Let's see:&#xA;$$&#xD;&#xA;P(\alpha) = \int P(\alpha, \theta_1, \theta_2) d\theta_1 d\theta_2&#xD;&#xA; = \int P(\alpha | \theta_1, \theta_2) P(\theta_1, \theta_2) d\theta_1 d\theta_2&#xD;&#xA;$$&#xA;With the good help of Monte Carlo, we can approximate this as&#xA;$$&#xD;&#xA;\frac{1}{n} \sum_t P(\alpha | \theta_1^t, \theta_2^t)&#xD;&#xA;$$&#xA;With the grid trick, you are doing something like this, but then you are implicitly assuming that $P(\alpha | \theta_1, \theta_2)$ is uniform. Is that an OK assumption for you? If so, it's a good technique if you want to get a view on the distribution of $\alpha$. If you are only out for confidence intervals for $\alpha$, you can probably do better by ordering your intervalsbase on $\theta_1$, and then finding the last value that is only covered by the first $2.5\%$ of you intervals (if you're aiming for $95\%$ confidence), and then similar for the other side.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An OK range of values for your grid would be from the minimum $\theta_1$ up till the maximum $\theta_2$, no? If you mean by 'hundreds of thousands of $\alpha$s that you have that many parameters that you wish to get information about simultaneously, then yes, you are in trouble: you cannot abstract out those parameters, and then hope to automagically and easily get this kind of information back. Univariately though, you're safe.&lt;/p&gt;&#xA;" OwnerUserId="4257" LastActivityDate="2011-05-27T09:12:29.650" CommentCount="2" />
  <row Id="11307" PostTypeId="2" ParentId="11280" CreationDate="2011-05-27T09:30:47.783" Score="1" ViewCount="" Body="&lt;p&gt;See also the &lt;a href=&quot;http://cran.r-project.org/web/packages/glmulti/index.html&quot; rel=&quot;nofollow&quot;&gt;glmulti&lt;/a&gt; package on CRAN and the accompanying &lt;a href=&quot;http://www.jstatsoft.org/v34/i12/paper&quot; rel=&quot;nofollow&quot;&gt;JSS paper&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;glmulti&lt;/code&gt; provides a wrapper for &lt;code&gt;glm&lt;/code&gt; and similar functions (&lt;code&gt;glm.nb&lt;/code&gt;, etc.), automatically generating all possible models (under constraints set by the user) with the specified response and explanatory variables, and finding the best models in terms of some Information Criterion (AIC, AICc or BIC). It can handle very large numbers of candidate models and features a Genetic Algorithm to find the best models when an exhaustive screening of the candidates is not feasible.&lt;/p&gt;&#xA;" OwnerUserId="103" LastActivityDate="2011-05-27T09:30:47.783" />
  <row Id="11308" PostTypeId="2" ParentId="11299" CreationDate="2011-05-27T10:02:06.747" Score="4" ViewCount="" Body="&lt;p&gt;The idea above sounds rather like single imputation. This is a better idea when faced with missing data than either list-wise or pair-wise deletion.  However, its still not a good approach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A better approach could be multiple imputation. Essentially, you simulate from 3-10 datasets conditional on your observed data. You then perform all of your analyses on each of these datasets, and combine the results at the end. The purposes in the simulation of multiple datasets is to ensure that the uncertainty in the imputation process is accounted for. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This can be done using the Multiple Imputation procedure in SPSS (i believe its on the analyze menu). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, while multiple imputation has been shown to be valid with large datasets, there is not as much information on its use in small samples. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A good introduction (from an educational perspective) can be found &lt;a href=&quot;http://www.csos.jhu.edu/contact/staff/jwayman_pub/wayman_multimp_aera2003.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A paper on its use in small samples (in a longitudinal context) can be found &lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pubmed/16220515&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="656" LastActivityDate="2011-05-27T10:02:06.747" />
  <row Id="11309" PostTypeId="1" CreationDate="2011-05-27T12:17:11.043" Score="1" ViewCount="53" Body="&lt;p&gt;I am wondering what the proper term for this is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Turning this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;________________&#xA;| Length | Count |&#xA;|      1 |     5 |&#xA;|      3 |     2 |&#xA;|      6 |    12 |&#xA;|________|_______|&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Into this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;________________&#xA;| Length | Count |&#xA;|      1 |     5 |&#xA;|      2 |     0 |&#xA;|      3 |     2 |&#xA;|      4 |     0 |&#xA;|      5 |     0 |&#xA;|      6 |    12 |&#xA;|________|_______|&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance!&lt;/p&gt;&#xA;" OwnerUserId="4781" LastEditorUserId="26" LastEditorDisplayName="" LastEditDate="2011-05-27T14:57:28.583" LastActivityDate="2011-05-27T15:27:04.487" Title="What do you call adding zeros to a table of frequency counts of consecutive integers where the given integer does not occur" Tags="&lt;data-transformation&gt;" AnswerCount="1" CommentCount="6" />
  <row Id="11310" PostTypeId="1" AcceptedAnswerId="11311" CreationDate="2011-05-27T13:30:12.590" Score="2" ViewCount="43" Body="&lt;p&gt;I've found critical values for the Anderson Darling test for a Normal Distribution at 1%, 2.5%, 5%, 10% and 15% significance levels from various sources, including wikipedia:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Anderson%E2%80%93Darling_test&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Anderson%E2%80%93Darling_test&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd really like a critical value for a 0.1% significance level (in &quot;case 4&quot; - neither mean nor variance known). I couldn't find it by searching the web, and I am not sure how I should calculate it. Can anyone help?&lt;/p&gt;&#xA;" OwnerUserId="4780" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-27T14:02:45.477" LastActivityDate="2011-05-27T15:04:37.847" Title="Critical values for Anderson-Darling test" Tags="&lt;distributions&gt;&lt;hypothesis-testing&gt;&lt;normal-distribution&gt;" AnswerCount="1" />
  <row Id="11311" PostTypeId="2" ParentId="11310" CreationDate="2011-05-27T15:04:37.847" Score="2" ViewCount="" Body="&lt;p&gt;You can use simulation (this is not a new idea, it is how Gosset/Student derived the original t table (but we have faster tools than he did)).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Generate a psuedo random sample from a normal distribution (or at least as close as the computer can come) of the sample size of interest and compute the Anderson Darling Statistic for that sample.  Now repeate this process a few million times (or maybe more than a few depending on how precise you want to be).  The 0.1% critical value will be the 0.1% or 99.9% percentile.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I have a hard time imagining a useful question that would be answered by an Anderson-Darling test at 0.1% significance.  What is the question that you are trying to answer?   &lt;/p&gt;&#xA;" OwnerUserId="4505" LastActivityDate="2011-05-27T15:04:37.847" CommentCount="3" />
  <row Id="11312" PostTypeId="2" ParentId="11309" CreationDate="2011-05-27T15:27:04.487" Score="1" ViewCount="" Body="&lt;p&gt;Mapping a set of observed value onto the observable values expected for a given variable? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;That is, a variable is characterized by all hypothetical values that can be observed when using it, but observed values may not reflect the full range of possible values. For example, when collecting n=100 discrete scores on a 0-20 point scale, you might end up with some scores that were observed more often than other, while some never occurred.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And if the number of observable (distinct) value is so small, I would suggest some kind of bar graph (or a dot chart) rather than an histogram, which for a random sample might look like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.stack.imgur.com/wtczl.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-05-27T15:27:04.487" CommentCount="2" />
  <row Id="11313" PostTypeId="2" ParentId="726" CreationDate="2011-05-27T15:30:01.647" Score="1" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;&quot;He who loves practice without theory&#xA;  is like the sailor who boards ship&#xA;  without a rudder and compass and never&#xA;  knows where he may be cast.&quot;&#xA;  - Leonardo da Vinci, 1452-1519 &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Found &lt;a href=&quot;http://socserv.mcmaster.ca/jfox/&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="253" LastActivityDate="2011-05-27T15:30:01.647" CommunityOwnedDate="2011-05-27T15:30:01.647" />
  <row Id="11314" PostTypeId="2" ParentId="11290" CreationDate="2011-05-27T16:14:16.013" Score="1" ViewCount="" Body="&lt;p&gt;You may know that weighting generally aims at ensuring that a given sample is representative of its target population. If in your sample some attributes (e.g., gender, SES, type of medication) are less well represented than in the population from which the sample comes from, then we may adjust the weights of the incriminated statistical units to better reflect the hypothetical target population.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;RIM weighting (or raking) means that we will equate the sample marginal distribution to the theoretical marginal distribution. It bears some idea with post-stratification, but allows to account for many covariates. I found a good overview in this handout about &lt;a href=&quot;http://sekhon.berkeley.edu/causalinf/sp2010/section/week9.pdf&quot; rel=&quot;nofollow&quot;&gt;Weighting Methods&lt;/a&gt;, and here is an example of its use in a real study: &lt;a href=&quot;http://www.fcsm.gov/01papers/Greene.pdf&quot; rel=&quot;nofollow&quot;&gt;Raking Fire Data&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Propensity weighting is used to compensate for unit non-response in a survey, for example, by increasing the sampling weights of the respondents in the sample using estimates of the probabilities that they responded to the survey. This is in spirit the same idea than the use of propensity scores to adjust for treatment selection bias in observational clinical studies: based on external information, we estimate the probability of patients being included in a given treatment group and compute weights based on factors hypothesized to influence treatment selection. Here are some pointers I found to go further: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.statistics.su.se/modernsurveys/publ/11.pdf&quot; rel=&quot;nofollow&quot;&gt;The propensity score and estimation in nonrandom surveys - an overview&lt;/a&gt; &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.amstat.org/sections/srms/proceedings/y2009/Files/304345.pdf&quot; rel=&quot;nofollow&quot;&gt;A Simulation Study to Compare Weighting Methods for Nonresponses in the National Survey of Recent College Graduates&lt;/a&gt; &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.jds-online.com/file_download/94/JDS-233.pdf&quot; rel=&quot;nofollow&quot;&gt;A Comparison of Propensity Score and Linear Regression Analysis of Complex Survey Data&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;As for a general reference, I would suggest &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Kalton G, Flores-Cervantes I.&#xA;  Weighting Methods. J. Off. Stat.&#xA;  (2003) 19: 81-97. Available on&#xA;  &lt;a href=&quot;http://www.jos.nu/&quot; rel=&quot;nofollow&quot;&gt;http://www.jos.nu/&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-05-27T16:14:16.013" CommentCount="1" />
  <row Id="11315" PostTypeId="1" AcceptedAnswerId="11323" CreationDate="2011-05-27T16:14:56.817" Score="5" ViewCount="126" Body="&lt;p&gt;So when I assume that the error terms are normally distributed in a linear regression, what does it mean for the response variable, $y$?&lt;/p&gt;&#xA;" OwnerUserId="4496" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-27T18:37:16.203" LastActivityDate="2011-05-28T23:34:21.947" Title="How does the distribution of the error term affect the distribution of the response?" Tags="&lt;regression&gt;&lt;distributions&gt;" AnswerCount="4" />
  <row Id="11316" PostTypeId="2" ParentId="11315" CreationDate="2011-05-27T16:36:35.590" Score="8" ViewCount="" Body="&lt;p&gt;The short answer is that you cannot conclude anything about the distribution of $y$, because it depends on the distribution of the $x$'s and the strength and shape of the relationship. More formally, $y$ will have a &quot;mixture of normals&quot; distribution, which in practice can be pretty much anything.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are two extreme examples to illustrate this:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Suppose there are only two possible $x$ values, 0 an 1, and $y = 10x + N(0,1)$. Then $y$ will have a strongly bimodal distribution with bumps at 0 and 10.&lt;/li&gt;&#xA;&lt;li&gt;Now assume the same relationship, but let $x$ be uniformly distributed on the 0-1 interval with lots of values. Then $y$ will be almost uniformly distributed over the 0-10 interval (with some half-normal tails at the edges).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In fact, since every distribution can be approximated arbitrarily well with mixture of normals, you can really get any distribution for $y$.&lt;/p&gt;&#xA;" OwnerUserId="279" LastEditorUserId="279" LastEditorDisplayName="" LastEditDate="2011-05-28T02:52:27.773" LastActivityDate="2011-05-28T02:52:27.773" CommentCount="2" />
  <row Id="11318" PostTypeId="2" ParentId="11315" CreationDate="2011-05-27T16:54:10.607" Score="4" ViewCount="" Body="&lt;p&gt;We invent the error term by imposing a fictitious model on real data; the distribution of the error term does not affect the distribution of the response.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We often assume that the error is distributed normally and thus try to construct the model such that our estimated residuals are normally distributed. This can be difficult for some distributions of $y$. In these cases, I suppose you could say that the distribution of the response affects the error term.&lt;/p&gt;&#xA;" OwnerUserId="3874" LastEditorUserId="3874" LastEditorDisplayName="" LastEditDate="2011-05-28T19:05:10.253" LastActivityDate="2011-05-28T19:05:10.253" CommentCount="2" />
  <row Id="11319" PostTypeId="1" CreationDate="2011-05-27T16:57:08.000" Score="3" ViewCount="45" Body="&lt;p&gt;I have an experiment where people click on different ads online. My measure is click counts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I end up finding that I should use models for count data such as Poisson, Quasi-Poisson, or Negative Binomial regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a standard in marketing regarding what model should be used for click counts?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="4679" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-27T20:18:44.890" LastActivityDate="2011-05-28T17:17:05.713" Title="Is there a standard procedure or regression model in marketing for explaining click rates on ads?" Tags="&lt;poisson&gt;&lt;count-data&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="11320" PostTypeId="2" ParentId="726" CreationDate="2011-05-27T18:19:55.340" Score="3" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;The Earth is round.  p &amp;lt; .05&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Jacob Cohen&lt;/p&gt;&#xA;" OwnerUserId="686" LastActivityDate="2011-05-27T18:19:55.340" CommunityOwnedDate="2011-05-27T18:19:55.340" />
  <row Id="11321" PostTypeId="2" ParentId="726" CreationDate="2011-05-27T18:21:27.587" Score="4" ViewCount="" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;When I see articles with lots of&#xA;  significance tests, I say that the&#xA;  statisticians are p-ing on the&#xA;  research.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Herman Friedmann (by recollection, he said this in class)&lt;/p&gt;&#xA;" OwnerUserId="686" LastActivityDate="2011-05-27T18:21:27.587" CommunityOwnedDate="2011-05-27T18:21:27.587" />
  <row Id="11322" PostTypeId="1" CreationDate="2011-05-27T20:07:41.690" Score="3" ViewCount="84" Body="&lt;p&gt;Here is a recent Google correlate query:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://correlate.googlelabs.com/search?e=internet+usage&amp;amp;t=weekly#&quot; rel=&quot;nofollow&quot;&gt;http://correlate.googlelabs.com/search?e=internet+usage&amp;amp;t=weekly#&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you can see in the search box at that link, I entered &quot;internet usage&quot; and Google did the rest.   It shows a value of 0.9298 as the &quot;correlation&quot; with the query &quot;data mining&quot;.   However, when I read page 2 of Google's white paper (click the &quot;Whitepaper&quot; selection on the left side of the linked page), it says:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&quot;....The objective of Google Correlate is to surface the queries in&#xA;the database whose spatial or temporal pattern is most highly correlated&#xA;with a target pattern. Google Correlate employs a novel approximate nearest&#xA;neighbor (ANN) algorithm over millions of candidate queries in an online&#xA;search tree to produce results similar to the batch-based approach employed&#xA;by Google Flu Trends but in a fraction of a second. For additional details,&#xA;please see the Methods section below....&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So, my questions are:  Is Google using a normal Pearson or Spearman Correlation to find this stuff or are they using something else?   If so, can you explain the general technique?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;==================&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, notice in the plot that the search for &quot;internet usage&quot; (and &quot;data mining&quot;) drops during the summer months and really dives around Christmas.  I would guess that kids and their homework have something to do with this.&lt;/p&gt;&#xA;" OwnerUserId="2775" LastActivityDate="2011-05-27T20:07:41.690" Title="What method is used in Google's correlate?" Tags="&lt;time-series&gt;&lt;correlation&gt;" CommentCount="3" />
  <row Id="11323" PostTypeId="2" ParentId="11315" CreationDate="2011-05-27T23:07:30.837" Score="4" ViewCount="" Body="&lt;p&gt;Maybe I'm off but I think we ought to be wondering about $f(y|\beta, X)$, which is how I read the OP. In the very simplest case of linear regression if your model is $y=X\beta + \epsilon$ then the only stochastic component in your model is the error term. As such it determines the sampling distribution of $y$. If $\epsilon\sim N(0, \sigma^2I)$ then $y|X, \beta\sim N(X\beta, \sigma^2I)$. What @Aniko says is certainly true of $f(y)$ (marginally over $X, \beta$), however. So as it stands the question is slightly vague.&lt;/p&gt;&#xA;" OwnerUserId="26" LastActivityDate="2011-05-27T23:07:30.837" CommentCount="1" />
  <row Id="11324" PostTypeId="2" ParentId="11277" CreationDate="2011-05-27T23:41:14.597" Score="3" ViewCount="" Body="&lt;p&gt;About the simplest thing you can do is interpolate normalized counts over time and (almost) the simplest form of interpolation is linear.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Specifically, suppose $y_i$ is the state population at time $i$ and $x_i$ is some other count (by age, tract, or whatever).  Define $\xi_i = x_i/y_i$.  Suppose $i$ is a year for which you do not have the periodic data.  Let $i_{-}$ and $i_{+}$ be the years immediately preceding and following $i$, respectively, for which $x_i$ is available.  The linearly interpolated estimate of $\xi_i$ is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\hat{\xi}_i = \frac{\xi_{i_{-}} (i_{+} - i) + \xi_{i_{+}} (i - i_{-})} {i_{+} - i_{-}} \text{.}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The estimate of $x_i$ is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\hat{x}_i = \hat{\xi}_i y_i.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The sums will come out correctly because this estimator is linear with weights summing to unity.  For example, suppose you are tracking two variables $x$ and $z$ which count complementary parts of the population (such as males and females), so that $x_i+z_i = y_i$ whenever you have all three counts.  Defining $\xi_i = x_i/y_i$ as before and, similarly, $\zeta_i = z_i/y_i$, the two fractions sum to unity: $\xi_i + \zeta_i = y_i/y_i = 1$ for all $i$.  Therefore the interpolated fractions also sum to unity:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\hat{\xi}_i + \hat{\zeta}_i = \frac{\xi_{i_{-}} (i_{+} - i) + \xi_{i_{+}} (i - i_{-})} {i_{+} - i_{-}} +  \frac{\zeta_{i_{-}} (i_{+} - i) + \zeta_{i_{+}} (i - i_{-})} {i_{+} - i_{-}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$= \frac{(\xi_{i_{-}} + \zeta_{i_{-}}) (i_{+} - i) + (\xi_{i_{+}} + \zeta_{i_{+}}) (i - i_{-})} {i_{+} - i_{-}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$= \frac{(i_{+} - i) + (i - i_{-})} {i_{+} - i_{-}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$= 1.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Whence $\hat{x}_i + \hat{z}_i = y_i(\hat{\xi}_i + \hat{\zeta}_i) = y_i$ as desired.  This generalizes to population partitions of any size, such as age distributions.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-05-27T23:41:14.597" CommentCount="9" />
  <row Id="11325" PostTypeId="2" ParentId="726" CreationDate="2011-05-28T13:02:46.950" Score="0" ViewCount="" Body="&lt;p&gt;Statistics' real contribution to society is primarily moral, not technical. &#xA;Steve Vardeman and Max Morris&lt;/p&gt;&#xA;" OwnerUserId="2669" LastActivityDate="2011-05-28T13:02:46.950" CommunityOwnedDate="2011-05-28T13:02:46.950" />
  <row Id="11326" PostTypeId="2" ParentId="11315" CreationDate="2011-05-28T13:14:20.353" Score="1" ViewCount="" Body="&lt;p&gt;If you write out the response as&#xA;$$\bf{y}=m+e$$&#xA;Where $\bf{m}$ is the &quot;model&quot; (the prediction for $\bf{y}$) and $\bf{e}$ is the &quot;errors&quot;, then this can be re-arranged to indicate $\bf{y}-m=e$.  So assigning a distribution for the errors is the same thing as indicating the ways your model is incomplete.  To put it another way is that it indicates to what extent you don't know why the observed response was the value that it actually was, and not what the model predicted.  If you knew your model was perfect, then you would assign a probability distribution with all of its mass on zero for the errors.  Assigning a $N(0,\sigma^{2})$ basically says that the errors are small in units of $\sigma$.  The idea is that the model predictions tend to be &quot;wrong&quot; by similar amounts for different observations, and is &quot;about right&quot; on the scale of $\sigma$.  As a contrast, an alternative assignment is $Cauchy(0,\gamma)$ which says that most of the errors are small, but some errors are quite large - the model has the occasional &quot;blunder&quot; or &quot;shocker&quot; in terms of predicting the response.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a sense the error distribution is more closely linked to the model than to the response.  This can be seen from the non-identifiability of the above equation, for if both $\bf{m}$ and $\bf{e}$ are unknown then adding an arbitrary vector to $\bf{m}$ and subtracting it from $\bf{e}$ leads to the same value of $\bf{y}$, $\bf{y}=m+e=(m+b)+(e-b)=m&amp;#39;+e&amp;#39;$.  The assignment of an error distribution and a model equation basically says which arbitrary vectors are more plausible than others.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastEditorUserId="2392" LastEditorDisplayName="" LastEditDate="2011-05-28T23:34:21.947" LastActivityDate="2011-05-28T23:34:21.947" CommentCount="4" />
  <row Id="11327" PostTypeId="1" AcceptedAnswerId="11330" CreationDate="2011-05-28T13:37:12.280" Score="3" ViewCount="78" Body="&lt;p&gt;We've created a survey asking students, among other things, their GPA (=weighted average of grades) and their marks in some specific courses (which count towards GPA).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We wanted to see which regressors influence the GPA using a simple OLS model. &lt;strong&gt;Is it sensible to use a formula like this?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;GPA ~ grade_maths + grade_statistics + grade_privatelaw + ... + {other regressors, like study habits or origin}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Of course, the grade regressors turn out to be highly significant (some more than others, and &lt;em&gt;not directly related to the weight they have in GPA&lt;/em&gt;), while few of the other ones are...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this a case of endogeneity, i.e. does a regression like this violate strict exogeneity? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;With this regression we want to get an quick overview of which variables will likely be useful in the regressions that follow, which for example try to find out whether being good in quantitative courses tends to help get good grades in law and other ideas like this...&lt;/p&gt;&#xA;" OwnerUserId="4788" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-30T07:44:27.810" LastActivityDate="2011-05-30T08:16:18.460" Title="Dependent variable is a function of independent variables; can I sensibly include them in a regression?" Tags="&lt;regression&gt;&lt;least-squares&gt;" AnswerCount="3" CommentCount="4" />
  <row Id="11328" PostTypeId="1" CreationDate="2011-05-28T13:41:53.917" Score="0" ViewCount="48" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;Possible Duplicate:&lt;/strong&gt;&lt;br&gt;&#xA;  &lt;a href=&quot;http://stats.stackexchange.com/questions/11113/wrong-results-using-anova-with-repeated-measures&quot;&gt;Wrong results using ANOVA with repeated measures&lt;/a&gt;  &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&#xA;&#xA;&lt;p&gt;Hello everybody,&#xA;I did an experiment and I need to understand how to detect, by means of an ANOVA (repeated measures), the differences between males and females evaluations at stimulus level.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the experiment, participants had to evaluate 7 stimuli in 2 conditions (EXP1 and EXP2).&#xA;The structure of the table is the following: subject, stimulus, condition, sex, response. The design is the following:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;sex is a between-subjects factor (with two levels)&lt;/li&gt;&#xA;&lt;li&gt;stimulus is a within-subjects factor (with 3 assumed levels)&lt;/li&gt;&#xA;&lt;li&gt;condition is a within-subjects factor (with 2 levels)&lt;/li&gt;&#xA;&lt;li&gt;all factors are fully crossed&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;For example now I want to detect the difference between males and females evaluation for stimulus 1.&#xA;So far the only way that I found is to use a t-test.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I used the following R command for conducting the ANOVA:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;aov1 = aov(response ~ sex*stimulus*condition + Error(subject/(stimulus*condition)), data=scrd)&#xA;&#xA;summary(aov1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;but I don´t see the way to understand if evaluataions between males and females for stimulus 1 are significant.&#xA;I can only see if there is a difference at global level between males and females for all the stimuli. But I am interested in discovering it for each stimulus.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I reach this goal?....I don´t think that the interactions of the previous ANOVA code can help me.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I used the t-test in this way&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;table_gravel_M &amp;lt;- subset(my_table, stimulus == &quot;gravel&quot; &amp;amp; sex == &quot;M&quot;)&#xA;table_gravel_F &amp;lt;- subset(my_table, stimulus == &quot;gravel&quot; &amp;amp; sex == &quot;F&quot;)&#xA;&#xA;t.test(table_gravel_M$response,table_gravel_F$response)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Any suggestion?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here an example of the table I used:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;subject  stimulus condition sex        response&#xA;&#xA;subject1    gravel  EXP1    M      59.8060&#xA;subject2    gravel  EXP1    M      49.9880&#xA;subject3    gravel  EXP1    M      73.7420&#xA;subject4    gravel  EXP1    M      45.5190&#xA;subject5    gravel  EXP1    M      51.6770&#xA;subject6    gravel  EXP1    M      42.1760&#xA;subject7    gravel  EXP1    M      56.1110&#xA;subject8    gravel  EXP1    M      54.9500&#xA;subject9    gravel  EXP1    M      62.6920&#xA;subject10   gravel  EXP1    M      50.7270&#xA;subject1    gravel  EXP2    M      70.9270&#xA;subject2    gravel  EXP2    M      61.3200&#xA;subject3    gravel  EXP2    M      70.2930&#xA;subject4    gravel  EXP2    M      49.9880&#xA;subject5    gravel  EXP2    M      69.1670&#xA;subject6    gravel  EXP2    M      62.2700&#xA;subject7    gravel  EXP2    M      70.9270&#xA;subject8    gravel  EXP2    M      63.6770&#xA;subject9    gravel  EXP2    M      72.4400&#xA;subject10   gravel  EXP2    M      58.8560&#xA;subject11   gravel  EXP1    F      46.5750&#xA;subject12   gravel  EXP1    F      58.1520&#xA;subject13   gravel  EXP1    F      57.4490&#xA;subject14   gravel  EXP1    F      59.8770&#xA;subject15   gravel  EXP1    F      55.5480&#xA;subject16   gravel  EXP1    F      46.2230&#xA;subject17   gravel  EXP1    F      63.3260&#xA;subject18   gravel  EXP1    F      60.6860&#xA;subject19   gravel  EXP1    F      59.4900&#xA;subject20   gravel  EXP1    F      52.6630&#xA;subject11   gravel  EXP2    F      55.7240&#xA;subject12   gravel  EXP2    F      66.4220&#xA;subject13   gravel  EXP2    F      65.9300&#xA;subject14   gravel  EXP2    F      61.8120&#xA;subject15   gravel  EXP2    F      62.5160&#xA;subject16   gravel  EXP2    F      65.5780&#xA;subject17   gravel  EXP2    F      59.5600&#xA;subject18   gravel  EXP2    F      63.8180&#xA;subject19   gravel  EXP2    F      61.4250&#xA;.....&#xA;.....&#xA;.....&#xA;.....&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="4701" LastActivityDate="2011-05-28T13:41:53.917" ClosedDate="2011-05-28T20:11:23.140" Title="Detecting significant differences for each stimulus using ANOVA repeated measures" Tags="&lt;anova&gt;&lt;repeated-measures&gt;&lt;t-test&gt;" CommentCount="6" />
  <row Id="11329" PostTypeId="1" CreationDate="2011-05-28T13:56:54.953" Score="5" ViewCount="49" Body="&lt;p&gt;Suppose we have a set $S$ consisting of $p$ features, and a subset $S_+$ of the features are positive.  If $Q$ is any subset of $S$, define the false positive rate as the proportion of features in $Q$ which are not positive:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$FPR[Q] = 1 - \frac{|Q \cap S_+|}{|Q|}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $|\cdot|$ denotes cardinality.  If $Q$ is a function of the data, $d$, then we can define the false discovery rate as the expected false positive rate:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$FDR[Q(d)] = \mathbb{E}_d[FPR[Q(d)]].$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now suppose that I have a method for ranking the features in $S$ by likelihood of significance.  I will report the top $r$ features most likely to be significant, based on my data, $Q_r(d)$.  Formally, I have a family of set-valued functions&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Q_1(d) \subset Q_2(d) \subset Q_3(d) \subset \cdots \subset Q_p(d)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $|Q_r(d)| = r.$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I want to know is the maximum $r$ such that the set $Q_r$ has a false discovery rate less than a certain critical value, $q_{crit}$.  That is, I want to know what is the value&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$IFDR_{q_{crit}} = \max_r \{r \in \{1,...,d\}: FDR[Q_r] \leq q_{crit}\}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a name for this 'inverse false discovery rate' function?  If not, can you suggest a name better than 'inverse false discovery rate'?&lt;/p&gt;&#xA;" OwnerUserId="3567" LastEditorUserId="3567" LastEditorDisplayName="" LastEditDate="2011-05-29T01:03:15.980" LastActivityDate="2011-05-29T01:03:15.980" Title="Inverse of false discovery rate (FDR)" Tags="&lt;error&gt;&lt;multiple-hypothesis&gt;" CommentCount="3" FavoriteCount="2" />
  <row Id="11330" PostTypeId="2" ParentId="11327" CreationDate="2011-05-28T15:38:07.043" Score="1" ViewCount="" Body="&lt;p&gt;I see no problem with fitting the regression.  We do regressions because we believe that the predictors may be related to the response, you just have more knowledge to begin with.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But what questions are you actually trying to answer?  The fact that certain coefficients are significant is not surprising, so those were not really interesting questions to begin with.  What may be interesting is if they differ from a value other than 0 (the weight in the GPA).  That could tell you if they have an indirect effect in addition to the known effect, e.g. math score could be related to science score which is not in your model, but contributes to the GPA.&lt;/p&gt;&#xA;" OwnerUserId="4505" LastActivityDate="2011-05-28T15:38:07.043" CommentCount="1" />
  <row Id="11331" PostTypeId="1" CreationDate="2011-05-28T16:04:40.950" Score="2" ViewCount="98" Body="&lt;p&gt;Hello,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I asked this on the mathematics site, but now I think this is a better place.  Sorry for the cross-post.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given any line graph, is there a reliable way to identify any sort of regular oscillation?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's assume I'm charting the prevalence of different species of animals in a single location, over the span of several years. What sort of algorithm could I apply that would identify a regular migration instead of a steady increase or decline?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope this makes sense.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for the help!&lt;/p&gt;&#xA;" OwnerUserId="4792" LastEditorUserId="4792" LastEditorDisplayName="" LastEditDate="2011-05-28T22:28:15.937" LastActivityDate="2011-06-01T03:18:46.037" Title="Identifying oscillation in a time series" Tags="&lt;time-series&gt;&lt;data-visualization&gt;" AnswerCount="2" CommentCount="1" />
  <row Id="11332" PostTypeId="2" ParentId="11319" CreationDate="2011-05-28T17:17:05.713" Score="3" ViewCount="" Body="&lt;p&gt;You can use Poisson regression and in more general form, Poisson process when data is following a Poisson distribution. In terms of Bayesian inference, you can make your own likelihood model, and then by conjugating your Poisson prior by likelihood, derive your posterior distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here I borrow an example from &lt;em&gt;glm&lt;/em&gt; function in R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; ## Dobson (1990) Page 93: Randomized Controlled Trial :&#xA; counts &amp;lt;- c(18,17,15,20,10,20,25,13,12)&#xA; outcome &amp;lt;- gl(3,1,9)&#xA; treatment &amp;lt;- gl(3,3)&#xA; print(d.AD &amp;lt;- data.frame(treatment, outcome, counts))&#xA; glm.D93 &amp;lt;- glm(counts ~ outcome + treatment, family=poisson())&#xA; anova(glm.D93)&#xA; summary(glm.D93)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Also, some slides about Poission Rregression with some example are &lt;a href=&quot;http://www.stat.wisc.edu/courses/st572-larget/Spring2007/handouts24-2.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;http://www.statpower.net/Content/MLRM/Lecture%20Slides/PoissonRegression.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="4581" LastActivityDate="2011-05-28T17:17:05.713" />
  <row Id="11333" PostTypeId="2" ParentId="11248" CreationDate="2011-05-28T18:12:47.033" Score="1" ViewCount="" Body="&lt;p&gt;I don't know if I am understanding correctly your question. But I guess you may use the posterior density to assess the uncertainty around point estimates like the mean. You may plot a histogram, calculate standard deviations. This is easy to do, if you have the MCMC output. Just take the values sampled (after a burnin period) and compute the means, standard deviations and plot histograms or densities.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another advantage of a posterior distribution is that you can assess your uncertainty on tails of the distribution and also if the distribution is simetric around the mode/mean. Confidence intervals in general assume that the distribution is simetric and that outliers are rare...&lt;/p&gt;&#xA;" OwnerUserId="3058" LastEditorUserId="3058" LastEditorDisplayName="" LastEditDate="2011-05-31T22:49:06.153" LastActivityDate="2011-05-31T22:49:06.153" />
  <row Id="11334" PostTypeId="2" ParentId="11248" CreationDate="2011-05-28T18:57:08.913" Score="3" ViewCount="" Body="&lt;p&gt;Don't use the mean of the sampled coefficients for making predictions, instead compute the predictions for logistic regression models with all of the sampled coefficient vectors and take the mean of those predictions (or better still treat the predictions for all sampled coefficient vectors as the posterior distribution of the probability of class membership - the spread of that distribution is a useful indicator of how confident the classifier is about the probabilistic classification).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The distribution of the sampled coeffcient vector gives an impression of how well the training data constrain the value of that parameter, so if the distribution is broad, we can't be confident of the &quot;true&quot; value of that coefficient (as explained by Manoel Galdino).  However, the key advantage of having a distribution of plausible coefficient vectors is that it provides you with a rational way to get a distribution of plausible values for the probability of class membership, which is what we really want.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Often using a Bayesian approach, we are not really interested in the coefficients of the model, but in the function implemented by the model.&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2011-05-28T18:57:08.913" CommentCount="1" />
  <row Id="11335" PostTypeId="2" ParentId="11327" CreationDate="2011-05-28T20:45:28.527" Score="2" ViewCount="" Body="&lt;p&gt;Another point to consider:  what enables a student to do well in one course is related to what enables him/her to do well in another.  There are overarching factors (cognitive, personality, circumstances) that play some role in determining each of the individual course grades.  So to use regression--to see how X1 relates to GPA while controlling for X2, X3, X4, etc.--, will &quot;cannibalize&quot; each X-Y relationship, partialling a portion of the relationship right out of itself.  The coefficients you obtain will be, in Tukey and Mosteller's words, &quot;arbitrary nonsense.&quot; Here's how Elazar Pedhazur puts it (Multiple Regression in Behavioral Research, 3rd Ed., 170-2):&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;“Partial correlation is not an all-purpose method of control […] Controlling variables without regard to the theoretical considerations about the pattern of relations among them may yield misleading or meaningless results […] It makes no sense to control for one measure of mental ability, say, while correlating another measure of mental ability with academic achievement when the aim is to study the relation between mental ability and academic achievement. […] This is tantamount to partialling a relation out of itself and may lead to the fallacious conclusion that mental ability and academic achievement are not correlated.” &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I'd recommend studying bivariate correlations among the variables over a regression predicting GPA.  Predicting each individual course grade (@probabilityislogic's idea) also seems very worth doing.   @Manoel's factor analysis idea makes me pause because you may not have all the variables necessary to map out the key underlying factors.&lt;/p&gt;&#xA;" OwnerUserId="2669" LastEditorUserId="2669" LastEditorDisplayName="" LastEditDate="2011-05-29T12:07:32.897" LastActivityDate="2011-05-29T12:07:32.897" CommentCount="2" />
  <row Id="11336" PostTypeId="1" CreationDate="2011-05-28T23:03:15.690" Score="2" ViewCount="54" Body="&lt;p&gt;I have a multinomial model estimated with the zelig package in R. Whenever I try to use the setx() command, I get an error message saying there is more than one mode. So in stead of using Zelig, I thought I would do it the hard way. I used the instructions &lt;a href=&quot;http://www.ats.ucla.edu/stat/r/dae/mlogit.htm&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;, but I am not sure if I trust these instructions as they give me some weird results. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Could anyone explain how to get predicted probabilities from a multinomial logit in R?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;hr&gt;&#xA;PS: here are the coefficients from a simple model (there are 6 choice catagories):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; structure(c(3.68021133487111, -0.903496528862169, -1.56339830041814, &#xA;-1.13238307296064, -1.67706243532044, -0.177585202845615, 0.0611115470557421, &#xA;-0.0458373863009504, 0.0881133593132653, -0.0686190052488972, &#xA;0.0163917121907627, 0.0165232098847022, 0.0373815294869855, -0.0353209839724262, &#xA;-0.00698911507852077), .Names = c(&quot;(Intercept):1&quot;, &quot;(Intercept):2&quot;, &#xA;&quot;(Intercept):3&quot;, &quot;(Intercept):4&quot;, &quot;(Intercept):5&quot;, &quot;Deviance:1&quot;, &#xA;&quot;Deviance:2&quot;, &quot;Deviance:3&quot;, &quot;Deviance:4&quot;, &quot;Deviance:5&quot;, &quot;Votes:1&quot;, &#xA;&quot;Votes:2&quot;, &quot;Votes:3&quot;, &quot;Votes:4&quot;, &quot;Votes:5&quot;))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and here are some values for the independent variables:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;structure(c(0.71847390030784, 1.01838748408701, 1.01838748408701, &#xA;1.20499277373001, 0.71847390030784, 1.20499277373001, 0.56393315893118, &#xA;1.20499277373001, 0.71847390030784, 0.56393315893118, 2, 5, 4, &#xA;7, 10, 27, 5, 9, 17, 16), .Dim = c(10L, 2L))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2704" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-30T02:45:08.173" LastActivityDate="2011-05-30T02:45:08.173" Title="Predicted probabilities from a multinomial regression model using zelig and R" Tags="&lt;r&gt;&lt;probability&gt;&lt;multinomial&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="11337" PostTypeId="1" AcceptedAnswerId="11340" CreationDate="2011-05-28T23:35:40.667" Score="3" ViewCount="146" Body="&lt;p&gt;A six-sided die is rolled 100 times. Using the normal approximation, find the probability that the face showing six turns up between 15 and 20 times. Find the probability that the sum of the face values of the 100 trials is less than 300.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the first part of the question, I did the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(15 \le X \le 20) = \sum_{15 \le i \le 20} C(100,i)(\frac{1}{6})^i(\frac{5}{6})^{100-i}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where X is the number of sixes rolled. My answer was about &lt;code&gt;0.56&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have no idea how to do the second part. I know I have to do something like&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(Y&amp;lt;300|N=100)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where Y is the sum and N is the number of times rolled. But I don't know the probability of the sum so I'm stuck.&lt;/p&gt;&#xA;" OwnerUserId="4401" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-31T05:07:31.587" LastActivityDate="2011-05-31T05:07:31.587" Title="Probability of a certain sum of values from a set of dice rolls" Tags="&lt;homework&gt;&lt;binomial&gt;&lt;dice&gt;" AnswerCount="2" CommentCount="3" />
  <row Id="11338" PostTypeId="2" ParentId="11337" CreationDate="2011-05-29T00:02:10.430" Score="2" ViewCount="" Body="&lt;p&gt;Due to the CLT, a sum of i.i.d. random variables is distributed:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xD;&#xA;\sum_{i=1}^nX_i \sim N\left(\mu =n\cdot\mu_{X_i},\sigma^2 = n\cdot\sigma^2_{X_i}\right)&#xD;&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The mean of a single dice roll ($X_i$) is 3.5 and the variance is 35/12.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That should help you find the answer.&lt;/p&gt;&#xA;" OwnerUserId="2310" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-30T07:10:59.470" LastActivityDate="2011-05-30T07:10:59.470" CommentCount="12" />
  <row Id="11339" PostTypeId="2" ParentId="11331" CreationDate="2011-05-29T07:17:39.377" Score="5" ViewCount="" Body="&lt;p&gt;You may want to look at spectral analysis techniques. Look at,&#xA;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/144197864X&quot; rel=&quot;nofollow&quot;&gt;Shumway &amp;amp; Stoffer&lt;/a&gt; (among many other books which treat the subject) or look &quot;Spectral analysis&quot; in the Wikipedia for some pointers.&lt;/p&gt;&#xA;" OwnerUserId="892" LastActivityDate="2011-05-29T07:17:39.377" CommentCount="2" />
  <row Id="11340" PostTypeId="2" ParentId="11337" CreationDate="2011-05-29T10:52:08.007" Score="4" ViewCount="" Body="&lt;p&gt;In the comments to Glen's answer you seem to have used a normal approximation &lt;code&gt;pnorm(300, 350, sqrt(3500/12))&lt;/code&gt; to get 0.001707396.  This is not a bad answer, though you can do better.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you used the continuity correction the continuity correction &lt;code&gt;pnorm(299.5, 350, sqrt(3500/12))&lt;/code&gt; you would get &lt;code&gt;0.001553355&lt;/code&gt;.  I suspect this is what was being asked for.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is in fact possible to calculate this more precisely. The following R code does so (yes, I know it has &lt;code&gt;for&lt;/code&gt; loops).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sides &amp;lt;-  6   &#xA;throws &amp;lt;- 100 &#xA;&#xA;## p[j,i] is probability of exactly (j+sides) after (i+1) throws &#xA;p &amp;lt;- matrix(rep(0, sides*(throws+1)^2 ), ncol=throws+1 )&#xA;p[sides,1] &amp;lt;- 1 # probability 1 of score of 0 after 0 throws  &#xA;&#xA;for (i in 2:(throws+1) ){&#xA;  for (j in (sides+1):(sides*(throws+1)) ){&#xA;     p[j,i] &amp;lt;-  sum(p[(j-sides):(j-1), i-1]) / sides&#xA;                                          } &#xA;                            }&#xA;sum( p[0:(299+sides), throws+1] ) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This gives the result &lt;code&gt;0.001505810&lt;/code&gt;.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The normal approximation with continuity correction is within 0.00005, which looks good, though the relative error is about 3%, which looks slightly less impressive; this often happens using the normal approximation in the tail of the distribution.  &lt;/p&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-05-29T10:52:08.007" CommentCount="4" />
  <row Id="11341" PostTypeId="1" CreationDate="2011-05-29T12:00:36.773" Score="2" ViewCount="49" Body="&lt;p&gt;Is there a way to give (in R or Minitab or Statgraphics) a fractional factorial design like that and inspect the generators and the complete defining relation ($2^4 - 1$ relations)?    &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;A   B   C   D   E   F   G   H&#xA;-1  1   1   1   -1  -1  1   -1&#xA;-1  -1  -1  1   1   1   1   -1&#xA;-1  1   1   -1  1   1   -1  -1&#xA;-1  -1  -1  -1  -1  -1  -1  -1&#xA;-1  1   -1  -1  1   -1  1   1&#xA;-1  -1  1   -1  -1  1   1   1&#xA;-1  1   -1  1   -1  1   -1  1&#xA;-1  -1  1   1   1   -1  -1  1&#xA;1   1   -1  -1  -1  1   1   -1&#xA;1   -1  1   -1  1   -1  1   -1&#xA;1   1   -1  1   1   -1  -1  -1&#xA;1   -1  1   1   -1  1   -1  -1&#xA;1   1   1   1   1   1   1   1&#xA;1   -1  -1  1   -1  -1  1   1&#xA;1   1   1   -1  -1  -1  -1  1&#xA;1   -1  -1  -1  1   1   -1  1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Thank you&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(This is not an answer. It's just a workaround)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df&amp;lt;- read.table(textConnection(&quot;&#xA;A B C D E F G H&#xA;-1  1   1   1   -1  -1  1   -1&#xA;-1  -1  -1  1   1   1   1   -1&#xA;-1  1   1   -1  1   1   -1  -1&#xA;-1  -1  -1  -1  -1  -1  -1  -1&#xA;-1  1   -1  -1  1   -1  1   1&#xA;-1  -1  1   -1  -1  1   1   1&#xA;-1  1   -1  1   -1  1   -1  1&#xA;-1  -1  1   1   1   -1  -1  1&#xA;1   1   -1  -1  -1  1   1   -1&#xA;1   -1  1   -1  1   -1  1   -1&#xA;1   1   -1  1   1   -1  -1  -1&#xA;1   -1  1   1   -1  1   -1  -1&#xA;1   1   1   1   1   1   1   1&#xA;1   -1  -1  1   -1  -1  1   1&#xA;1   1   1   -1  -1  -1  -1  1&#xA;1   -1  -1  -1  1   1   -1  1&#xA;&quot;)-&amp;gt;con,header=T)&#xA;close(con)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;ABCDEFGH gives a column of ones, so I=ABCDEFGH. We need the 14 remaining defining relations. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# start with four way interactions&#xA;four.way &amp;lt;- combn(c(&quot;A&quot;,&quot;B&quot;,&quot;C&quot;,&quot;D&quot;,&quot;E&quot;,&quot;F&quot;,&quot;G&quot;,&quot;H&quot;),4)&#xA;&#xA;res &amp;lt;- apply(four.way,2,function(x) {&#xA;apply(df[,x],1,prod)})&#xA;colnames(res) &amp;lt;- apply(four.way,2,paste,collapse=&quot;&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The res matrix has 70 columns of the products of the 4-combinations. Get the column names with a colSum of either 16 or -16 (they are defining relations)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def &amp;lt;- colnames(res[,colSums(res) == 16 | colSums(res) == -16])&#xA;[1] &quot;ABCH&quot; &quot;ABDE&quot; &quot;ABFG&quot; &quot;ACDF&quot; &quot;ACEG&quot; &quot;ADGH&quot; &quot;AEFH&quot; &quot;BCDG&quot; &quot;BCEF&quot; &quot;BDFH&quot; &quot;BEGH&quot; &quot;CDEH&quot; &quot;CFGH&quot; &quot;DEFG&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So I got another 14 defining relations (including the generalized interactions). No need to look for other &quot;words&quot; since I already got 15. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The resolution is IV (min word length). It's easy to observe that the BGHA columns form the typical $2^4$ design (with opposite signs for B and G). Using the above defining relations you can get the generators: C=ABH, D=AGH, E=BGH and F=ABG. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To get the alias structure I did the following (I applied it only for the main effects and 2-way interactions, but you can get whatever you ask for)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# two way interactions&#xA;two.way &amp;lt;- apply(combn(c(&quot;A&quot;,&quot;B&quot;,&quot;C&quot;,&quot;D&quot;,&quot;E&quot;,&quot;F&quot;,&quot;G&quot;,&quot;H&quot;),2),2,paste,collapse=&quot;&quot;)&#xA;&#xA;test &amp;lt;- c(&quot;A&quot;,&quot;B&quot;,&quot;C&quot;,&quot;D&quot;,&quot;E&quot;,&quot;F&quot;,&quot;G&quot;,&quot;H&quot;,two.way)&#xA;gen &amp;lt;- c(def,&quot;ABCDEFGH&quot;)&#xA;&#xA;mat &amp;lt;- character(length(test)*(length(gen)+1))&#xA;dim(mat) &amp;lt;- c(length(test),length(gen)+1)&#xA;colnames(mat) &amp;lt;- c(&quot;Effect&quot;,paste(&quot;I=&quot;,gen,sep=&quot;&quot;))&#xA;for (j in 1:length(test)){&#xA;mat[j,1] &amp;lt;- test[j]&#xA;for (i in 1:length(gen)) {&#xA;res &amp;lt;- paste(sort(c(unlist(strsplit(gen[i],&quot;&quot;)),unlist(strsplit(test[j],&quot;&quot;)))),collapse=&quot;&quot;)&#xA;        le &amp;lt;- rle(unlist(strsplit(res,&quot;&quot;)))$lengths&#xA;        va &amp;lt;- rle(unlist(strsplit(res,&quot;&quot;)))$values&#xA;        mat[j,i+1] &amp;lt;- paste(sort(va[le %% 2 == 1]),collapse=&quot;&quot;)&#xA;}}&#xA;noquote(mat)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And here is the result&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;      Effect I=ABCH I=ABDE I=ABFG I=ACDF I=ACEG I=ADGH I=AEFH I=BCDG I=BCEF I=BDFH I=BEGH I=CDEH I=CFGH I=DEFG I=ABCDEFGH&#xA; [1,] A      BCH    BDE    BFG    CDF    CEG    DGH    EFH    ABCDG  ABCEF  ABDFH  ABEGH  ACDEH  ACFGH  ADEFG  BCDEFGH   &#xA; [2,] B      ACH    ADE    AFG    ABCDF  ABCEG  ABDGH  ABEFH  CDG    CEF    DFH    EGH    BCDEH  BCFGH  BDEFG  ACDEFGH   &#xA; [3,] C      ABH    ABCDE  ABCFG  ADF    AEG    ACDGH  ACEFH  BDG    BEF    BCDFH  BCEGH  DEH    FGH    CDEFG  ABDEFGH   &#xA; [4,] D      ABCDH  ABE    ABDFG  ACF    ACDEG  AGH    ADEFH  BCG    BCDEF  BFH    BDEGH  CEH    CDFGH  EFG    ABCEFGH   &#xA; [5,] E      ABCEH  ABD    ABEFG  ACDEF  ACG    ADEGH  AFH    BCDEG  BCF    BDEFH  BGH    CDH    CEFGH  DFG    ABCDFGH   &#xA; [6,] F      ABCFH  ABDEF  ABG    ACD    ACEFG  ADFGH  AEH    BCDFG  BCE    BDH    BEFGH  CDEFH  CGH    DEG    ABCDEGH   &#xA; [7,] G      ABCGH  ABDEG  ABF    ACDFG  ACE    ADH    AEFGH  BCD    BCEFG  BDFGH  BEH    CDEGH  CFH    DEF    ABCDEFH   &#xA; [8,] H      ABC    ABDEH  ABFGH  ACDFH  ACEGH  ADG    AEF    BCDGH  BCEFH  BDF    BEG    CDE    CFG    DEFGH  ABCDEFG   &#xA; [9,] AB     CH     DE     FG     BCDF   BCEG   BDGH   BEFH   ACDG   ACEF   ADFH   AEGH   ABCDEH ABCFGH ABDEFG CDEFGH    &#xA;[10,] AC     BH     BCDE   BCFG   DF     EG     CDGH   CEFH   ABDG   ABEF   ABCDFH ABCEGH ADEH   AFGH   ACDEFG BDEFGH    &#xA;[11,] AD     BCDH   BE     BDFG   CF     CDEG   GH     DEFH   ABCG   ABCDEF ABFH   ABDEGH ACEH   ACDFGH AEFG   BCEFGH    &#xA;[12,] AE     BCEH   BD     BEFG   CDEF   CG     DEGH   FH     ABCDEG ABCF   ABDEFH ABGH   ACDH   ACEFGH ADFG   BCDFGH    &#xA;[13,] AF     BCFH   BDEF   BG     CD     CEFG   DFGH   EH     ABCDFG ABCE   ABDH   ABEFGH ACDEFH ACGH   ADEG   BCDEGH    &#xA;[14,] AG     BCGH   BDEG   BF     CDFG   CE     DH     EFGH   ABCD   ABCEFG ABDFGH ABEH   ACDEGH ACFH   ADEF   BCDEFH    &#xA;[15,] AH     BC     BDEH   BFGH   CDFH   CEGH   DG     EF     ABCDGH ABCEFH ABDF   ABEG   ACDE   ACFG   ADEFGH BCDEFG    &#xA;[16,] BC     AH     ACDE   ACFG   ABDF   ABEG   ABCDGH ABCEFH DG     EF     CDFH   CEGH   BDEH   BFGH   BCDEFG ADEFGH    &#xA;[17,] BD     ACDH   AE     ADFG   ABCF   ABCDEG ABGH   ABDEFH CG     CDEF   FH     DEGH   BCEH   BCDFGH BEFG   ACEFGH    &#xA;[18,] BE     ACEH   AD     AEFG   ABCDEF ABCG   ABDEGH ABFH   CDEG   CF     DEFH   GH     BCDH   BCEFGH BDFG   ACDFGH    &#xA;[19,] BF     ACFH   ADEF   AG     ABCD   ABCEFG ABDFGH ABEH   CDFG   CE     DH     EFGH   BCDEFH BCGH   BDEG   ACDEGH    &#xA;[20,] BG     ACGH   ADEG   AF     ABCDFG ABCE   ABDH   ABEFGH CD     CEFG   DFGH   EH     BCDEGH BCFH   BDEF   ACDEFH    &#xA;[21,] BH     AC     ADEH   AFGH   ABCDFH ABCEGH ABDG   ABEF   CDGH   CEFH   DF     EG     BCDE   BCFG   BDEFGH ACDEFG    &#xA;[22,] CD     ABDH   ABCE   ABCDFG AF     ADEG   ACGH   ACDEFH BG     BDEF   BCFH   BCDEGH EH     DFGH   CEFG   ABEFGH    &#xA;[23,] CE     ABEH   ABCD   ABCEFG ADEF   AG     ACDEGH ACFH   BDEG   BF     BCDEFH BCGH   DH     EFGH   CDFG   ABDFGH    &#xA;[24,] CF     ABFH   ABCDEF ABCG   AD     AEFG   ACDFGH ACEH   BDFG   BE     BCDH   BCEFGH DEFH   GH     CDEG   ABDEGH    &#xA;[25,] CG     ABGH   ABCDEG ABCF   ADFG   AE     ACDH   ACEFGH BD     BEFG   BCDFGH BCEH   DEGH   FH     CDEF   ABDEFH    &#xA;[26,] CH     AB     ABCDEH ABCFGH ADFH   AEGH   ACDG   ACEF   BDGH   BEFH   BCDF   BCEG   DE     FG     CDEFGH ABDEFG    &#xA;[27,] DE     ABCDEH AB     ABDEFG ACEF   ACDG   AEGH   ADFH   BCEG   BCDF   BEFH   BDGH   CH     CDEFGH FG     ABCFGH    &#xA;[28,] DF     ABCDFH ABEF   ABDG   AC     ACDEFG AFGH   ADEH   BCFG   BCDE   BH     BDEFGH CEFH   CDGH   EG     ABCEGH    &#xA;[29,] DG     ABCDGH ABEG   ABDF   ACFG   ACDE   AH     ADEFGH BC     BCDEFG BFGH   BDEH   CEGH   CDFH   EF     ABCEFH    &#xA;[30,] DH     ABCD   ABEH   ABDFGH ACFH   ACDEGH AG     ADEF   BCGH   BCDEFH BF     BDEG   CE     CDFG   EFGH   ABCEFG    &#xA;[31,] EF     ABCEFH ABDF   ABEG   ACDE   ACFG   ADEFGH AH     BCDEFG BC     BDEH   BFGH   CDFH   CEGH   DG     ABCDGH    &#xA;[32,] EG     ABCEGH ABDG   ABEF   ACDEFG AC     ADEH   AFGH   BCDE   BCFG   BDEFGH BH     CDGH   CEFH   DF     ABCDFH    &#xA;[33,] EH     ABCE   ABDH   ABEFGH ACDEFH ACGH   ADEG   AF     BCDEGH BCFH   BDEF   BG     CD     CEFG   DFGH   ABCDFG    &#xA;[34,] FG     ABCFGH ABDEFG AB     ACDG   ACEF   ADFH   AEGH   BCDF   BCEG   BDGH   BEFH   CDEFGH CH     DE     ABCDEH    &#xA;[35,] FH     ABCF   ABDEFH ABGH   ACDH   ACEFGH ADFG   AE     BCDFGH BCEH   BD     BEFG   CDEF   CG     DEGH   ABCDEG    &#xA;[36,] GH     ABCG   ABDEGH ABFH   ACDFGH ACEH   AD     AEFG   BCDH   BCEFGH BDFG   BE     CDEG   CF     DEFH   ABCDEF &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="339" LastEditorUserId="339" LastEditorDisplayName="" LastEditDate="2011-05-30T11:38:12.837" LastActivityDate="2011-05-30T11:38:12.837" Title="Inspect generators and defining relations of a fractional factorial design" Tags="&lt;r&gt;&lt;experiment-design&gt;" AnswerCount="1" CommentCount="5" />
  <row Id="11342" PostTypeId="2" ParentId="11341" CreationDate="2011-05-29T12:25:29.513" Score="2" ViewCount="" Body="&lt;p&gt;&lt;strong&gt;Disclaimer:&lt;/strong&gt; Not really a positive answer...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Take a look at the &lt;a href=&quot;http://cran.r-project.org/web/packages/FrF2/index.html&quot; rel=&quot;nofollow&quot;&gt;FrF2&lt;/a&gt; package, for example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;des.24 &amp;lt;- FrF2(16,8)         &#xA;design.info(des.24)$aliased  # look at the alias structure&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;create a randomized fractional design with 8 factors, 16 runs. To print all designs, &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;print(catlg, nfactor=8, nruns=16)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For example, we have design 8-4.1 for a $2_{IV}^{8-4}$ design, whose generators are $E=ABC$, $F=ABD$, $G=ACD$, and $H=BCD$ (with defining relations in e.g., Montgomery 5ed Appendix X p. 629):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;summary(FrF2(design=&quot;8-4.1&quot;))&#xA;FrF2:::generators.from.design(FrF2(design=&quot;8-4.1&quot;))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Yet I found no way to update the design matrix. It seems we can update a response vector (there's an example of use with &lt;code&gt;design()&lt;/code&gt;/&lt;code&gt;undesign()&lt;/code&gt;), but AFAIK there's no function that would import a matrix of contrasts and allow to match it in the catalog or find the generators.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S. Apart from Minitab and StatGraphics, &lt;a href=&quot;http://www.reliasoft.com/doe/&quot; rel=&quot;nofollow&quot;&gt;DOE++&lt;/a&gt; seems to offer many facilities to work with two-level fractional designs, but I cannot test it unfortunately.&lt;/p&gt;&#xA;" OwnerUserId="930" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-30T08:39:55.627" LastActivityDate="2011-05-30T08:39:55.627" CommentCount="3" />
  <row Id="11343" PostTypeId="2" ParentId="11248" CreationDate="2011-05-29T12:42:17.257" Score="3" ViewCount="" Body="&lt;p&gt;I wouldn't use the means at all for the classifier.  You don't need to apply &quot;corrections&quot; or to &quot;smooth out&quot; a Bayesian solution, it is the optimal one for the prior information and data that you have actually used.  But the means can be useful for giving you a feel for which combinations of regressor variables are likely to lead to classifying towards a particular category.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However this can be a horrendously complicated beast for multinomial regression, as you have a matrix of betas to interpret (one column for each category, except for the reference, which can be thought of as having all betas &quot;estimated&quot; as zero with zero standard error).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given that this seems to be an attempt at a intuitive way to understand what your classifier is doing, let me propose another.  I will delete this section if this is not what you were intending.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You have your MCMC samples of the beta matrix: call this $\beta_{ij}^{(b)}$ where $i=1,\dots,R$ denotes the multinomial category, $j=1,\dots,p$ denotes the regressor variable (the $X$), and $b=1,\dots,B$ denotes the $bth$ MCMC sampled value.  If the categories have different $X$ variables, then simply set those excluded variables' betas to zero in the matrix: $\beta_{ik}^{(b)}=0$ for all $b$ if variable $k$ was not part of the model fit to the ith category, and $\beta_{Rj}^{(b)}=0$ for all $j$ and $b$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first thing you need is a set of covariates to use $X_{mj}\;\;\;\;m=1,\dots,M$, where $m$ is the &quot;observation number&quot; and $M$ is the number of predictions you are going to make.  The data used to fit your model should do for this purpose, so $M=\text{sample size}$.  You now calculate the linear predictor for each category for each prediction for each MCMC sample:&#xA;$$y_{im}^{(b)}=\sum_{j=1}^{p}X_{mj}\beta_{ij}^{(b)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(this may be quicker to code up as a matrix/array operation).  Note that $y_{Rm}^{(b)}=0$ for all $b$ and $m$.  Then convert this into a probability for the $mth$ observation belonging to the $ith$ category/class, call this quantity $Classify(m,i)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Classify(m,i)=\frac{1}{B}\sum_{b=1}^{B}\frac{\exp\left(y_{im}^{(b)}\right)}{\sum_{l=1}^{R}\exp\left(y_{lm}^{(b)}\right)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now you plot the value of $Classify(i,m)$ against $X_{mj}$, so you will have a total of $R\times p$ plots.  Looking at these should give you a feel for what the classifier is doing in relation to the regressor variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that when it comes to actually classifying a new variable, you only need $Classify(i,m)$ in order to do this - all other quantities from the MCMC are irrelevant for the purpose of classification.  What you do need though is a loss matrix which describes the loss incurred from classifying into category $i_{est}$ when the true category is actually $i_{true}$,  this will be a $R\times R$ matrix, usually zero on the diagonal and positive everywhere else.  This can be very important if correctly identifying &quot;rare&quot; classes is crucial compared to correctly identifying &quot;common&quot; classes.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-05-29T12:42:17.257" CommentCount="2" />
  <row Id="11344" PostTypeId="2" ParentId="11296" CreationDate="2011-05-29T13:18:28.267" Score="1" ViewCount="" Body="&lt;p&gt;You need to have the raw data, so that the response variable is 0/1 (not smoke, smoke).  Then you can use binary logistic regression.  It is not correct to group BMI into intervals.  The cutpoints are not correct, probably don't exist, and you are not officially testing whether BMI is associated with smoking.  You are currently testing whether BMI with much of its information discarded is associated with smoking.  You'll find that especially the outer BMI intervals are quite heterogeneous.&lt;/p&gt;&#xA;" OwnerUserId="4253" LastActivityDate="2011-05-29T13:18:28.267" CommentCount="12" />
  <row Id="11345" PostTypeId="2" ParentId="11336" CreationDate="2011-05-29T15:32:53.610" Score="1" ViewCount="" Body="&lt;p&gt;The first thing to do is to construct the &quot;linear predictors&quot; or &quot;logits&quot; for each category for each prediction.  So you have your model equation:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\eta_{ir}=\sum_{j=1}^{p}X_{ij}\hat{\beta}_{jr}\;\; (i=1,\dots,m\;\; r=1,\dots,R)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where for notational convenience, the above is to be understood to have $\hat{\beta}_{jR}=\eta_{iR}=0$ (as this is the reference category), and $\hat{\beta}_{jr}=0$ if variable $j$ was not included in the linear predictor for class $r$.  So you will have an $m\times R$ matrix of logits.  You then exponentiate to form predicted odds ratios and renormalise to form predicted probabilities.  Note that the predicted odds ratios can be calculated by a simple matrix operation if your data is sufficiently organised:&#xA;$$\bf{O}=\exp(\boldsymbol{\eta})=\exp(\bf{X}\boldsymbol{\beta})$$&#xA; of the $m\times p$ prediction matrix $\bf{X}\equiv\it{\{X_{ij}\}}$ with the $p\times R$ estimated co-efficient matrix $\boldsymbol{\beta}\equiv\it{\{\hat{\beta}_{jr}\}}$, and $\exp(.)$ is defined component wise (i.e. not the matrix exponential).  The matrix $\bf{O}$ is an &quot;odds ratio&quot; matrix, with the last column should be all ones.  If we take the $m\times 1$ vector $\bf{T}=O1_{R}$  This gives the normalisation constant for each prediction &quot;row&quot; of odds ratios.  Now create the $(m\times m)$ diagonal matrix defined by $W_{kk}\equiv T_{k}^{-1}$, and the predicted probability matrix is given by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\bf{P}=\bf{W}\bf{O}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So in the example you give the matrix $\boldsymbol{\beta}$ would look like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\begin{array}{c|c}&#xD;&#xA;Int:1 &amp;amp; Int:2 &amp;amp; Int:3 &amp;amp; Int:4 &amp;amp; Int:5 &amp;amp; 0 \\&#xD;&#xA;\hline&#xD;&#xA;Dev:1 &amp;amp; Dev:2 &amp;amp; Dev:3 &amp;amp; Dev:4 &amp;amp; Dev:5 &amp;amp; 0 \\&#xD;&#xA;\hline&#xD;&#xA;Vote:1 &amp;amp; Vote:2 &amp;amp; Vote:3 &amp;amp; Vote:4 &amp;amp; Vote:5 &amp;amp; 0 \\&#xD;&#xA;\hline&#xD;&#xA;\end{array}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With the (roughly rounded) values plugged in we get:&#xA;$$\boldsymbol{\beta}=\begin{array}{c|c}&#xD;&#xA;3.68 &amp;amp; -0.90 &amp;amp; -1.56 &amp;amp; -1.13 &amp;amp; -1.68 &amp;amp; 0 \\&#xD;&#xA;\hline&#xD;&#xA;-0.18 &amp;amp; 0.06 &amp;amp; -0.04 &amp;amp; 0.08 &amp;amp; -0.07 &amp;amp; 0 \\&#xD;&#xA;\hline&#xD;&#xA;0.02 &amp;amp; 0.02 &amp;amp; 0.04 &amp;amp; -0.04 &amp;amp; -0.01 &amp;amp; 0 \\&#xD;&#xA;\hline&#xD;&#xA;\hline&#xD;&#xA;\end{array}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And the $\bf{X}$ matrix would look like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\bf{X}=\begin{array}{c|c}&#xD;&#xA;1 &amp;amp; 0.72 &amp;amp; 2\\ \hline 1 &amp;amp; 1.02 &amp;amp; 5\\ \hline 1 &amp;amp; 1.02 &amp;amp; 4\\ \hline 1 &amp;amp; 1.20 &amp;amp; 7\\&#xD;&#xA;\hline 1 &amp;amp; 0.72 &amp;amp; 10\\ \hline 1 &amp;amp; 1.20 &amp;amp; 27\\ \hline 1 &amp;amp; 0.56 &amp;amp; 5\\&#xD;&#xA;\hline 1 &amp;amp; 1.20 &amp;amp; 9\\ \hline 1 &amp;amp; 0.72 &amp;amp; 17\\ \hline 1 &amp;amp; 0.56 &amp;amp; 16\\&#xD;&#xA;\hline&#xD;&#xA;\end{array}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So some R-code to do this would simply be (with the matrices $\bf{X}$ and $\boldsymbol{\beta}$ defined as above).  The main parts are reading in the data, and padding it with 1s and 0s for $\bf{X}$ and $\boldsymbol{\beta}$:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;code&gt;&#xA;  beta&amp;lt;-cbind(as.matrix( structure(c(3.68021133487111,&#xA;   -0.903496528862169, -1.56339830041814, &#xA;   -1.13238307296064, -1.67706243532044, -0.177585202845615, 0.0611115470557421, &#xA;   -0.0458373863009504, 0.0881133593132653, -0.0686190052488972, &#xA;   0.0163917121907627, 0.0165232098847022, 0.0373815294869855, -0.0353209839724262, &#xA;   -0.00698911507852077), .Names = c(&quot;(Intercept):1&quot;, &quot;(Intercept):2&quot;, &#xA;   &quot;(Intercept):3&quot;, &quot;(Intercept):4&quot;,&#xA;   &quot;(Intercept):5&quot;, &quot;Deviance:1&quot;, &#xA;   &quot;Deviance:2&quot;, &quot;Deviance:3&quot;,&#xA;   &quot;Deviance:4&quot;, &quot;Deviance:5&quot;, &quot;Votes:1&quot;,&#xA;   &quot;Votes:2&quot;, &quot;Votes:3&quot;, &quot;Votes:4&quot;,&#xA;   &quot;Votes:5&quot;) &#xA;   , .Dim=c(3L,5L))  ),0)&lt;br&gt;&#xA;   X&amp;lt;-cbind(1,as.matrix(&#xA;   structure(c(0.71847390030784,&#xA;   1.01838748408701, 1.01838748408701, &#xA;   1.20499277373001, 0.71847390030784, 1.20499277373001, 0.56393315893118, &#xA;   1.20499277373001, 0.71847390030784, 0.56393315893118, 2, 5, 4,  7, 10, 27, 5, 9, 17, 16), .Dim = c(10L, 2L)) ))&lt;br&gt;&#xA;   P&amp;lt;-diag(as.vector(exp(X %*% beta) %*%&#xA;   as.matrix(rep(1,ncol(beta))))^-1) %*%&#xA;   exp(X %*% beta)&#xA;  &lt;/code&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-05-29T15:32:53.610" />
  <row Id="11346" PostTypeId="1" CreationDate="2011-05-29T17:20:28.353" Score="2" ViewCount="69" Body="&lt;p&gt;This is probably a pretty simple question, but I have been having some trouble interpreting the documentation for the &lt;code&gt;predict&lt;/code&gt; function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am generating a simple linear model from a data frame containing (X, Y) pairs, which I would then like to use to predict Y given new X. My code looks something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;my_lm = lm(Y ~ X, data=my_data)&#xA;new_Y = predict(my_lm, new_data, interval=&quot;confidence&quot;, type=&quot;response&quot;)&#xA;print(new_Y[,'fit'])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This does seem to give me values already, but I am a bit confused about the &lt;code&gt;type&lt;/code&gt; parameter. The documentation only mentions that &lt;code&gt;type&lt;/code&gt; specifies &quot;response or model term&quot;, but it doesn't say what that actually means. Searching the web for examples I have seen both methods mentioned on different sites, with no clear explanation why they were used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone tell me what the right way to solve my problem is and where I can read up on the actual meaning of the type?&lt;/p&gt;&#xA;" OwnerUserId="3031" LastActivityDate="2011-05-29T18:13:04.837" Title="Obtaining predictions from linear model" Tags="&lt;r&gt;&lt;linear-model&gt;" AnswerCount="1" />
  <row Id="11347" PostTypeId="1" AcceptedAnswerId="11349" CreationDate="2011-05-29T17:35:12.357" Score="0" ViewCount="39" Body="&lt;p&gt;I want to jointly estimate a very simple MV-Normal two-dimensional AR[1] process, &#xA;$[x_t,y_t]=[x_{t-1},y_{t-1}]+\text{[Bivariate Gaussian error]}$, in BUGS. But the syntax has been impossible to figure out. Here's the problem part of the code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; ## transition model (aka random walk prior)&#xA; for(i in 2:NPERIODS1){         &#xA;    mu.vector[i,1:2]&amp;lt;-vector[i-1,1:2]&#xA;    vector[i,1:2]~dmnorm(mu.vector[i,1:2], omega[1:2,1:2])&#xA;    }&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The compiler throws up a &quot;Expected a multivariate node&quot; error. Looking through some examples, there doesn't seem to be any easy way to introduce a structured mean or covariance variables for the multivariate normal function. How should I proceed?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt;  Changed omega[,] to omega[1:2,1:2] for clarity. &lt;/p&gt;&#xA;" OwnerUserId="996" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-05-29T18:57:29.510" LastActivityDate="2011-05-29T18:57:29.510" Title="Multivariate random walks in BUGS" Tags="&lt;time-series&gt;&lt;multivariate-analysis&gt;&lt;mcmc&gt;&lt;bugs&gt;" AnswerCount="1" />
  <row Id="11348" PostTypeId="2" ParentId="11242" CreationDate="2011-05-29T18:01:20.593" Score="1" ViewCount="" Body="&lt;p&gt;First: Why can't you get the raw data from the GSS? It's easily available. Fail that, you can work with ANES or with the US sample of the World Value Survey. Or raw exit poll data. If you need academic access to get the files, contact me. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second: The poly-sci way to do this is to run the Ideal or OC to construct a d-dimensional &quot;Issue-space&quot;, figure out where the candidates are in issue space(Pretty easy, either by interpreting item parameters of the &quot;Supports Candidate X&quot; question or just by looking at the coordinates of candidate supporters, and then find which 4 questions are maximally informative with regards to issue space. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I actually just finished working through a similar problem. &lt;/p&gt;&#xA;" OwnerUserId="996" LastActivityDate="2011-05-29T18:01:20.593" CommentCount="7" />
  <row Id="11349" PostTypeId="2" ParentId="11347" CreationDate="2011-05-29T18:11:00.007" Score="1" ViewCount="" Body="&lt;p&gt;Have you tried replacing omega[,] with omega[1:2,1:2]? I haven't got BUGS here but IIRC that's what it expects inside dmnorm.&lt;/p&gt;&#xA;" OwnerUserId="26" LastActivityDate="2011-05-29T18:11:00.007" CommentCount="3" />
  <row Id="11350" PostTypeId="2" ParentId="11346" CreationDate="2011-05-29T18:13:04.837" Score="4" ViewCount="" Body="&lt;p&gt;The &lt;code&gt;type&lt;/code&gt; argument specifies if you want predictions of the response (the $Y$ variable) or if you want predictions for the individual terms in the model. In combination with the &lt;code&gt;terms&lt;/code&gt; argument you can get predictions for some or all (default) of the terms. &#xA;In your example, there is just one term in addition to the intercept. Your model is &#xA;$$Y_i = \alpha + \beta X_i + \epsilon_i$$ &#xA;hence, predictions of the terms would give you $\hat{\beta}X^{\text{new}}$ while predictions of the response would give you $\hat{\alpha} + \hat{\beta}X^{\text{new}}$.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that &lt;code&gt;type = &quot;response&quot;&lt;/code&gt; is the default for &lt;code&gt;predict.lm&lt;/code&gt;, so you get predictions of responses by default, which I guess is what you would expect from a predict function. If you look at &lt;code&gt;predict.glm&lt;/code&gt; instead the default is &lt;code&gt;&quot;link&quot;&lt;/code&gt;, which gives the linear predictors by default, so for generalized linear models you need to be explicit if you want predictions of the response. &lt;/p&gt;&#xA;" OwnerUserId="4376" LastActivityDate="2011-05-29T18:13:04.837" />
  <row Id="11351" PostTypeId="1" AcceptedAnswerId="11352" CreationDate="2011-05-29T20:14:17.173" Score="0" ViewCount="45" Body="&lt;p&gt;This is pretty hard for me to describe, but I'll try to make my problem understandable. So first you have to know that I've done a very simple linear regression so far. Before I estimated the coefficient, I watched the distribution of my $y$. It is heavy left skewed. After I estimated the model, I was quite sure to observe a left-skewed residual in a QQ-Plot as wel, but I absolutely did not. What might be the reason for this solution? Where is the mistake? Or has the distribution $y$ nothing to do with the distribution of the error term? &lt;/p&gt;&#xA;" OwnerUserId="4496" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-29T20:39:36.307" LastActivityDate="2011-05-29T21:06:10.660" Title="Left skewed vs. symmetric distribution observed" Tags="&lt;regression&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="11352" PostTypeId="2" ParentId="11351" CreationDate="2011-05-29T21:06:10.660" Score="3" ViewCount="" Body="&lt;p&gt;To answer your question, let's take a very simple example. The simple regression model is given by $y_i = \beta_0 + \beta_1 x_i + \epsilon_i$, where $\epsilon_i \sim N(0,\sigma^2)$. Now suppose that $x_i$ is dichotomous. If $\beta_1$ is not equal to zero, then the distribution of $y_i$ will not be normal, but actually a mixture of two normal distributions, one with mean $\beta_0$ and one with mean $\beta_0 + \beta_1$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If $\beta_1$ is large enough and $\sigma^2$ is small enough, then a histogram of $y_i$ will look bimodal. However, one can also get a histogram of $y_i$ that looks like a &quot;single&quot; skewed distribution. Here is one example (using R):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;xi &amp;lt;- rbinom(10000, 1, .2)&#xA;yi &amp;lt;- 0 + 3 * xi + rnorm(10000, .7)&#xA;hist(yi, breaks=20)&#xA;qqnorm(yi); qqline(yi)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It's not the distribution of $y_i$ that matters -- but the distribution of the error terms.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;res &amp;lt;- lm(yi ~ xi)&#xA;hist(resid(res), breaks=20)&#xA;qqnorm(resid(res)); qqline(resid(res))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And that looks perfectly normal -- not only figuratively speaking =)&lt;/p&gt;&#xA;" OwnerUserId="1934" LastActivityDate="2011-05-29T21:06:10.660" />
  <row Id="11353" PostTypeId="1" CreationDate="2011-05-29T23:16:54.367" Score="3" ViewCount="68" Body="&lt;p&gt;I have hundreds of explanatory variables and under 100 observations (saturated data set).   I'd like to create a linear model in which I have two or so composite variables made up of a dozen of the explanatory variables each.  How do I find the best variables to use for the composites without going through every combination?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently my $R^2 = .64$, I'd like to improve that. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The model looks something like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Y = B_1(v_1 + v_2 + \dots + v_{12}) + B_2(v_{13} + v_{14} + \dots +v_{24})$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $B_1$ and $B_2$ are the coefficients and the entirety of ($v_1 + v_2 + \dots + v_{12}$) is acting as one variable.&lt;/p&gt;&#xA;" OwnerUserId="4798" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-30T06:07:01.963" LastActivityDate="2011-05-31T13:35:40.367" Title="How do I find the best model with a saturated dataset?" Tags="&lt;regression&gt;&lt;modeling&gt;" AnswerCount="3" FavoriteCount="1" />
  <row Id="11354" PostTypeId="1" CreationDate="2011-05-30T03:50:28.957" Score="0" ViewCount="18" Body="&lt;p&gt;Short version:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have streaming data that would be infeasible to hold in memory, so I'm limited to a constant number of quantities I can keep. I would like to know if there are a small number of quantities I can accumulate in order to run Levene's test on 2 data sets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For more context...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm trying to compare the variances of possibly very large and currently unknown data sets (streaming data, effectively -- only one pass is feasible). I can't assume the data is normal, so I figured I'd use Levene's test to compare them. Based on &quot;&gt;wikipedia's notation, it looks like I have to have all the data in memory to compute Z_ij, since it relies on knowing the mean of each group, which I won't have until after I process the stream. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For comparison, I'm also doing a Student's t test to compare the means -- for this test, I only have to keep the count, the sum, and the sum of the squares (3 numbers) for each data set I want to test. From there, I can derive the variance, plug in the numbers, and I'm good to go.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, is there a set of quantities I can keep that let me derive (&lt;code&gt;Zi. - Z..)^2&lt;/code&gt; and &lt;code&gt;(Zij - Zi.)^2&lt;/code&gt;, without storing every value, or even a significant fraction of the values?  &lt;/p&gt;&#xA;" OwnerUserId="4800" LastActivityDate="2011-05-30T03:50:28.957" Title="Efficient storage of information necessary to compute Levene's test of variance" Tags="&lt;variance&gt;&lt;computing&gt;&lt;efficiency&gt;&lt;levenes-test&gt;" />
  <row Id="11355" PostTypeId="2" ParentId="11353" CreationDate="2011-05-30T07:03:04.490" Score="1" ViewCount="" Body="&lt;p&gt;How about factor analysis over your variables ? That will give you the list of variables which behave similarly to give you a latent variable. Apart from this you should also run multicollinearity diagonistics to detect collinearity in your present list of 100 variables. Chances are high that many of your variables would be correlated.&lt;/p&gt;&#xA;" OwnerUserId="1763" LastActivityDate="2011-05-30T07:03:04.490" CommentCount="1" />
  <row Id="11356" PostTypeId="2" ParentId="11353" CreationDate="2011-05-30T07:08:04.773" Score="2" ViewCount="" Body="&lt;p&gt;The standard answer when determining the &quot;best&quot; linear combinations of variables is &lt;a href=&quot;http://en.wikipedia.org/wiki/Principal_component_analysis&quot; rel=&quot;nofollow&quot;&gt;principal component analysis&lt;/a&gt;. Its regression counterparts are &lt;a href=&quot;http://en.wikipedia.org/wiki/Principal_component_regression&quot; rel=&quot;nofollow&quot;&gt;principal components regression&lt;/a&gt; and &lt;a href=&quot;http://en.wikipedia.org/wiki/Partial_least_squares_regression&quot; rel=&quot;nofollow&quot;&gt;partial least squares regression&lt;/a&gt;. Both methods will select several linear combinations of &lt;strong&gt;all&lt;/strong&gt; the variables which should be the &quot;best&quot; predictors of the dependent variable. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suggest starting with PCR, since it looks like more suited for your analysis as it is stated now. It would help if you would elaborate more on what are you trying to model. Goal of raising $R^2$ is a bit suspicious, as it should never be the sole goal in any kind of modeling.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-05-30T07:08:04.773" CommentCount="2" />
  <row Id="11357" PostTypeId="1" CreationDate="2011-05-30T08:12:21.220" Score="0" ViewCount="39" Body="&lt;p&gt;Question was originally posted &lt;a href=&quot;http://stackoverflow.com/questions/6172746/calculate-variance-of-3-or-more-attributes-with-known-frequecies&quot;&gt;on Stack Overflow&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to calculate variance of data at a given time where I know the frequency of the conditional and decision attributes. &#xA;At time $t$ every conditional attribute can have 3 possible values while decsion attribute consists of two classes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;See the picture below for more details:&#xA;&lt;img src=&quot;http://i.stack.imgur.com/XZnue.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;How can I calculate the variance of this data snapshot?&lt;/li&gt;&#xA;&lt;li&gt;Sample code or an algorithm would be most appreciated.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="4802" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-30T08:21:28.197" LastActivityDate="2011-05-30T08:21:28.197" Title="How to calculate variance of three or more attributes with known frequencies" Tags="&lt;machine-learning&gt;&lt;variance&gt;&lt;data-mining&gt;" CommentCount="7" />
  <row Id="11358" PostTypeId="2" ParentId="11327" CreationDate="2011-05-30T08:16:18.460" Score="2" ViewCount="" Body="&lt;p&gt;Strong exogeneity is a term related to dynamic models, i.e. when there is time-series data involved. Since you are doing one-time survey, this term does not apply. What might be the problem with the regression though is &lt;a href=&quot;http://en.wikipedia.org/wiki/Omitted-variable_bias&quot; rel=&quot;nofollow&quot;&gt;omitted variable bias&lt;/a&gt;. Since GPA is a weighted average purely arithmetical formula applies:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$GPA = w_1 G_1+...+ w_n G_n$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $w_i$ are the weights and $G_i$ are grades. This equation is not stochastic. However we can say that each grade is determined by student's ability plus stochastic term:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$G_i=f_i(\mathbf{A})+\varepsilon_i$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\mathbf{A}$ is vector of variables which determine students ability, and $f$ is the functional form of the relationship. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When viewed in this light it makes no sense including grades in the regression.  What might be of interest how different functional relationships $f_i$ aggregate into functional relationship $f$:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$GPA=f(\mathbf{A})+\varepsilon$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But this does not answer the questions in your given example. So as other's suggested it is better to use &lt;a href=&quot;http://en.wikipedia.org/wiki/Factor_analysis&quot; rel=&quot;nofollow&quot;&gt;factor analysis&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-05-30T08:16:18.460" />
  <row Id="11359" PostTypeId="1" AcceptedAnswerId="11373" CreationDate="2011-05-30T08:47:46.340" Score="4" ViewCount="143" Body="&lt;p&gt;Is there any reason of what I can think of, to transform the data with a square root? I mean what I always observe is that the R^2 increases. But this is probably just because of centering the data! Any thought is appreciated!&lt;/p&gt;&#xA;" OwnerUserId="4496" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-30T08:50:11.490" LastActivityDate="2011-05-30T19:20:54.643" Title="What could be the reason for using square root transformation on data?" Tags="&lt;regression&gt;&lt;data-transformation&gt;" AnswerCount="4" FavoriteCount="2" />
  <row Id="11360" PostTypeId="1" CreationDate="2011-05-29T03:57:40.317" Score="3" ViewCount="25" Body="&lt;p&gt;I have a multivariate dataset with a test group and three control groups. I was thinking that the best way to determine if and how the test group differed from all of the control groups would be to perform a MANOVA, then perform contrast analysis between the test group and all of the control groups.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I cannot seem to find a good description of how contrast analysis for MANOVA is performed.&#xA;Every tutorial/online class I can find, just talks about throwing the contrast into SPSS, or the univariate case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone point me to a good tutorial on contrasts for MANOVA?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is MANOVA with contrasts the appropriate way of answering my research question?&#xA;Would it be better to perform repeated Hotelling's T-squared tests on the test group vs each control group, and just use a Bonferroni correction?&lt;/p&gt;&#xA;" OwnerUserId="3629" LastEditorUserId="88" LastEditorDisplayName="" LastEditDate="2011-05-30T11:24:59.163" LastActivityDate="2011-05-30T11:24:59.163" Title="Questions about contrasts for MANOVA" Tags="&lt;spss&gt;&lt;manova&gt;" CommentCount="1" />
  <row Id="11361" PostTypeId="2" ParentId="7249" CreationDate="2011-05-30T09:13:13.920" Score="4" ViewCount="" Body="&lt;p&gt;Try &lt;a href=&quot;http://orange.biolab.si/&quot; rel=&quot;nofollow&quot;&gt;Orange Canvas&lt;/a&gt;, it will give you option to build interactive decision tree.&lt;/p&gt;&#xA;" OwnerUserId="4802" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-31T06:39:19.290" LastActivityDate="2011-05-31T06:39:19.290" />
  <row Id="11362" PostTypeId="2" ParentId="11359" CreationDate="2011-05-30T09:58:02.853" Score="1" ViewCount="" Body="&lt;p&gt;Taking the square root is sometimes advocated to make a non-normal variable appear like a normal variable in regression problems. The logarithm is another common possible transformation.&lt;/p&gt;&#xA;" OwnerUserId="3019" LastActivityDate="2011-05-30T09:58:02.853" />
  <row Id="11363" PostTypeId="2" ParentId="11359" CreationDate="2011-05-30T10:42:59.890" Score="4" ViewCount="" Body="&lt;p&gt;The square-root transformation is just a special case of Box-Cox power transformation (a nice overview by Pengfi Li, could be useful reading and is found &lt;a href=&quot;http://www.stat.uconn.edu/~studentjournal/index_files/pengfi_s05.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;), with $\lambda = 0.5$ and omitting some centering. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The aim of the Box-Cox transformations is to ensure the usual&#xA;  assumptions for Linear Model hold. That is, $y\sim N(X\beta, \sigma^2 I_n)$.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;However this a priori fixed value could be (and probably is) not optimal. In R you may consider a function from &lt;code&gt;car&lt;/code&gt; library &lt;code&gt;powerTransform&lt;/code&gt; that helps to estimate an optimal value for Box-Cox transformations for each of the variables participated in linear regression or any data you work with (see the &lt;code&gt;example(powerTransform)&lt;/code&gt; for further details).&lt;/p&gt;&#xA;" OwnerUserId="2645" LastEditorUserId="2645" LastEditorDisplayName="" LastEditDate="2011-05-30T10:54:55.480" LastActivityDate="2011-05-30T10:54:55.480" />
  <row Id="11364" PostTypeId="1" CreationDate="2011-05-30T13:00:39.147" Score="0" ViewCount="58" Body="&lt;p&gt;I will be to grateful if help me find a lower bound $g(x)$ over the following concave function:&#xA; $$f(x) = \sqrt{1+4x} -1 + \log(\sqrt{1+4x}-1) - \log(2x) \geq g(x),$$&#xA;where $x \geq 0$.&#xA;The taylor expansion around the point ($x = 0$) of this function is given by:&#xA;$$f(x) = x - \frac{x^2}{2} - + \frac{2x^3}{3} - \frac{5x^4}{4} + \frac{14x^{5}}{5} - \ldots$$&lt;/p&gt;&#xA;" OwnerUserId="4770" LastEditorUserId="4770" LastEditorDisplayName="" LastEditDate="2011-05-30T15:01:51.507" LastActivityDate="2011-05-30T15:01:51.507" Title="Lower bound over a concave function" Tags="&lt;bounds&gt;" AnswerCount="1" CommentCount="3" />
  <row Id="11365" PostTypeId="2" ParentId="6656" CreationDate="2011-05-30T13:29:35.353" Score="0" ViewCount="" Body="&lt;p&gt;The &quot;expected entropy&quot; at a particular time step given a particular starting point is a well-defined quantity and you could certainly study it.  There is no particular reason why you should favor the &quot;median&quot; entropy without knowing anything else about the system.  You should do experiments with as diverse of a set of starting configurations as possible to get a better understanding of your system.&lt;/p&gt;&#xA;" OwnerUserId="3567" LastActivityDate="2011-05-30T13:29:35.353" />
  <row Id="11366" PostTypeId="2" ParentId="11359" CreationDate="2011-05-30T14:00:01.927" Score="1" ViewCount="" Body="&lt;p&gt;When the variable follows a Poisson distribution, the results of the square root transform will be much closer to Gaussian.&lt;/p&gt;&#xA;" OwnerUserId="25" LastActivityDate="2011-05-30T14:00:01.927" />
  <row Id="11367" PostTypeId="2" ParentId="11364" CreationDate="2011-05-30T14:11:17.350" Score="0" ViewCount="" Body="&lt;p&gt;You should be able to show that for positive $x$,  $f(x) \lt 2x$. &lt;/p&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2011-05-30T14:11:17.350" CommentCount="3" />
  <row Id="11368" PostTypeId="1" CreationDate="2011-05-30T14:18:25.663" Score="9" ViewCount="78" Body="&lt;p&gt;Suppose I have the following model &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$y_i=f(x_i,\theta)+\varepsilon_i$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $y_i\in \mathbb{R}^K$ ,  $x_i$ is a vector of explanatory variables, $\theta$ is the parameters of non-linear function $f$ and $\varepsilon_i\sim N(0,\Sigma)$, where $\Sigma$ naturally is $K\times K$ matrix. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The goal is the usual to estimate $\theta$ and $\Sigma$. The obvious choice is maximum likelihood method. Log-likelihood for this model (assuming we have a sample $(y_i,x_i),i=1,...,n$) looks like&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$l(\theta,\Sigma)=-\frac{n}{2}\log(2\pi)-\frac{n}{2} \log\det\Sigma-\sum_{i=1}^n(y_i-f(x_i,\theta))&amp;#39;\Sigma^{-1}(y-f(x_i,\theta)))$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now this seems simple, the log-likelihood is specified, put in data, and use some algorithm for non-linear optimisation. The problem is how to ensure that $\Sigma$ is positive definite. Using for example &lt;code&gt;optim&lt;/code&gt; in R (or any other non-linear optimisation algorithm) will not guarantee me that $\Sigma$ is positive definite. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the question is how to ensure that $\Sigma$ stays positive definite? I see two possible solutions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Reparametrise $\Sigma$ as  $RR&amp;#39;$ where $R$ is upper-triangular or symmetric matrix. Then $\Sigma$ will always be positive-definite and $R$ can be unconstrained.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Use profile likelihood. Derive the formulas for $\hat\theta(\Sigma)$ and $\hat{\Sigma}(\theta)$. Start with some $\theta_0$ and iterate $\hat{\Sigma}_j=\hat\Sigma(\hat\theta_{j-1})$, $\hat{\theta}_j=\hat\theta(\hat\Sigma_{j-1})$ until convergence.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Is there some other way and what about these 2 approaches, will they work, are they standard?  This seems pretty standard problem, but quick search did not give me any pointers. I know that Bayesian estimation would be also possible, but for the moment I would not want to engage in it.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-31T06:30:58.367" LastActivityDate="2011-06-02T19:42:33.137" Title="How to ensure properties of covariance matrix when fitting multivariate normal model using maximum likelihood?" Tags="&lt;maximum-likelihood&gt;&lt;optimization&gt;&lt;covariance&gt;" AnswerCount="1" CommentCount="1" />
  <row Id="11370" PostTypeId="1" CreationDate="2011-05-30T14:36:08.930" Score="2" ViewCount="50" Body="&lt;p&gt;I have data from an experiment that is testing how the order of two studying methods (visual or auditory) affects word recall. For analysis a multi-factor anova with a repeated measure is appropriate, but I am not sure if I am structuring my data correctly. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is the command I'm using:&#xA;&lt;code&gt;aov(recalled_items~task*order)+Error(subject/task)+(order))&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an example of the data structure:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Subject    Task    Order    Recalled Items&#xA;A        Visual    First       13&#xA;A       Auditory   Second      22&#xA;B        Visual    First       14&#xA;B       Auditory   Second      28&#xA;C        Visual    Second      10&#xA;C       Auditory   First       15&#xA;D        Visual    Second      14&#xA;D       Auditory   First       29&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Does R know to compare Visual 1 and Visual 2 recall values and Auditory 1 and Auditory 2 recall values? &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I am worried that because of the way I structured my data R is just comparing Visual 1 and Auditory 1 and as a result I am getting no effect.&lt;/p&gt;&#xA;" OwnerUserId="4804" LastEditorUserId="307" LastEditorDisplayName="" LastEditDate="2011-05-30T15:11:25.247" LastActivityDate="2011-06-02T15:46:52.483" Title="Testing a 2 by 2 mixed ANOVA in R" Tags="&lt;r&gt;&lt;anova&gt;&lt;dataset&gt;" AnswerCount="2" CommentCount="3" FavoriteCount="1" />
  <row Id="11371" PostTypeId="2" ParentId="11370" CreationDate="2011-05-30T15:12:41.203" Score="5" ViewCount="" Body="&lt;p&gt;Your method won't work because it's going to treat order within.  Try this...&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Subject    Task    Order    Recalled Items&#xA;A        Visual    vFirst      13&#xA;A       Auditory   vFirst      22&#xA;D        Visual    aFirst      14&#xA;D       Auditory   aFirst      28&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Or something like that.&lt;/p&gt;&#xA;" OwnerUserId="601" LastActivityDate="2011-05-30T15:12:41.203" />
  <row Id="11372" PostTypeId="1" CreationDate="2011-05-30T15:49:10.737" Score="3" ViewCount="85" Body="&lt;p&gt;I have to run a factor analysis on a dataset made up of dichotomous variables (0=yes, 1= no) and I don´t know if I'm on the right track.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using &lt;code&gt;tetrachoric()&lt;/code&gt; I create a correlation matrix, on which I run &lt;code&gt;fa(data,factors=1)&lt;/code&gt;. &#xA;The result is quite near to the results i receive when using &lt;a href=&quot;https://typo3.univie.ac.at/index.php?id=78913&quot; rel=&quot;nofollow&quot;&gt;MixFactor&lt;/a&gt;, but it´s not the same. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Is this ok or would you recommend another procedure?&lt;/li&gt;&#xA;&lt;li&gt;Why does &lt;code&gt;fa()&lt;/code&gt; work and &lt;code&gt;factanal()&lt;/code&gt; produce an error? (&lt;code&gt;Fehler in solve.default(cv) : &#xA;System ist für den Rechner singulär: reziproke Konditionszahl = 4.22612e-18&lt;/code&gt;)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="4805" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-31T19:29:11.803" LastActivityDate="2011-05-31T19:29:11.803" Title="Recommended procedure for factor analysis on dichotomous data with R" Tags="&lt;r&gt;&lt;factor-analysis&gt;" CommentCount="11" />
  <row Id="11373" PostTypeId="2" ParentId="11359" CreationDate="2011-05-30T16:01:00.770" Score="3" ViewCount="" Body="&lt;p&gt;In general, parametric regression / GLM assume that the relationship between the $Y$ variable and each $X$ variable is linear, that the residuals once you've fitted the model follow a normal distribution and that the size of the residuals stays about the same all the way along your fitted line(s).  When your data don't conform to these assumptions, transformations can help.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It should be intuitive that if $Y$ is proportional to $X^2$ then square-rooting $Y$ linearises this relationship, leading to a model that better fits the assumptions and that explains more variance (has higher $R^2$).  Square rooting $Y$ also helps when you have the problem that the size of your residuals progressively increases as your values of $X$ increase (i.e. the scatter of data points around the fitted line gets more marked as you move along it).  Think of the shape of a square root function: it increases steeply at first but then saturates.  So applying a square root transform inflates smaller numbers but stabilises bigger ones.  So you can think of it as pushing small residuals at low $X$ values away from the fitted line and squishing large residuals at high $X$ values towards the line.  (This is mental shorthand not proper maths!) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As Dmitrij and ocram say, this is just one possible transformation which will help in certain circumstances, and tools like the Box-Cox formula can help you to pick the most useful one.  I would advise getting into the habit of always looking at a plots of residuals against fitted values (and also a normal probability plot or histogram of residuals) when you fit a model.  You'll find you'll often end up being able to see from these what sort of transformation will help.&lt;/p&gt;&#xA;" OwnerUserId="266" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-30T19:20:54.643" LastActivityDate="2011-05-30T19:20:54.643" CommentCount="2" />
  <row Id="11374" PostTypeId="1" AcceptedAnswerId="11379" CreationDate="2011-05-30T16:12:20.290" Score="3" ViewCount="140" Body="&lt;p&gt;I'm a statistics newbie and I would like to make sense of the p-value with ANOVA, hopefully with a visual presentation. All online visual tools I've found so far only have MSB, MSE and F values visually presented (such as &lt;a href=&quot;http://www.psych.utah.edu/aoce/tools/Anova/anovatool.html&quot; rel=&quot;nofollow&quot;&gt;this here&lt;/a&gt;. I'm a very visual person and I would like to see how it works before I learn more about it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you have any suggestions for me? It could be a Matlab program as well, something where I play with the distributions and see the p value change. &lt;/p&gt;&#xA;" OwnerUserId="4806" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-31T03:31:51.527" LastActivityDate="2011-05-31T03:53:48.550" Title="Visual representations of the p-value in ANOVA to assist intuitive understanding" Tags="&lt;hypothesis-testing&gt;&lt;anova&gt;" AnswerCount="2" CommentCount="4" FavoriteCount="1" />
  <row Id="11375" PostTypeId="1" CreationDate="2011-05-30T17:50:09.890" Score="0" ViewCount="92" Body="&lt;p&gt;I have a question for adjusted $R^2$  given a specific regression model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am doing a project on January effect and I have a model from some journal using&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$R_i = a_0 + a_1D_{Jan} + \varepsilon_i$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;$R_i$ is daily return of portfolio/index,&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;$a_0$  is non-January daily returns, &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;$a_1$  is January returns over non January returns, &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;$D_{Jan}$ is  dummy variable (1 for Jan, 0 otherwise). &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;With this model I tried to test whether January return is significantly greater than non-Jan returns, especially in small capitalization stocks. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I have returns for size sorted portfolio 1 to 4, where Portfolio 1 (P1) consist of smallest cap stocks, P4 consist of largest cap stocks. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I did was using regression program in excel and spss, input all daily returns of the portfolio from Jan to Dec for 10 years as dependent variable and the dummy (1 for january and 0 for others) as independent variable into the regression program and I get &lt;strong&gt;negative adjusted $R^2$&lt;/strong&gt; for all portfolio (P1-P4) mostly about -0.05. The value of $R^2$ itself was also very small at about 0.0001&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The journal I am basing my model was using monthly returns as $R_i$, but I modified it into daily return. I still get negative adjusted $R^2$ when I use the monthly returns.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone please help me point out what was wrong in the model? Did I use the wrong input based on the given model? If yes, what should be the correct input? Or if the model was wrong, what model should I use? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here the result of my test, the p-value of the dummy variable and the adj $R^2$. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;For P1 0.54, adj $R^2$ -0.0002. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;P2 0.36, adj $R^2$ 0.0004. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;P3 0.68, Adj $R^2$ -0.0003, &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;P4 0.14, adj $R^2$ 0.0005.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I only understand that the variables are all insignificant. But I am confused in interpreting the $R^2$&lt;/p&gt;&#xA;" OwnerUserId="4808" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-31T06:28:56.573" LastActivityDate="2011-05-31T06:28:56.573" Title="Why is a regression model of portfolio return giving smaller adjusted r-square (i.e., negative) than expected?" Tags="&lt;regression&gt;&lt;homework&gt;&lt;spss&gt;&lt;model-selection&gt;&lt;r-squared&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="1" />
  <row Id="11376" PostTypeId="1" CreationDate="2011-05-30T19:46:51.073" Score="0" ViewCount="20" Body="&lt;p&gt;Hi,&#xA;I'm using the following commands to plot a set of points on a 2d map in matlab&lt;/p&gt;&#xA;&#xA;&lt;p&gt;labels =  {'label1', ' label2', 'label3',...)&#xA;plot(score(:,1),score(:,2), '*')&#xA;gname(lables)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a way to choose the color for each label?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="4809" LastActivityDate="2011-05-30T19:46:51.073" ClosedDate="2011-05-30T22:07:54.217" Title="Display colored labels in plots with matlab" Tags="&lt;data-visualization&gt;&lt;matlab&gt;" CommentCount="1" />
  <row Id="11377" PostTypeId="2" ParentId="11375" CreationDate="2011-05-30T21:17:59.833" Score="2" ViewCount="" Body="&lt;p&gt;You've included an interaction term without including both of the main effects that are the components of that interaction.  According to standard practice, you need a term for January returns.  Exceptions to this rule are rare though they have been discussed on this site recently at &lt;a href=&quot;http://stats.stackexchange.com/questions/11009/including-the-interaction-but-not-the-main-effects-in-a-model&quot;&gt;Including the interaction but not the main effects in a model&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2669" LastActivityDate="2011-05-30T21:17:59.833" CommentCount="3" />
  <row Id="11378" PostTypeId="1" CreationDate="2011-05-30T21:43:35.950" Score="3" ViewCount="25" Body="&lt;p&gt;I am attempting to perform a two-group confirmatory factor analysis (CFA) of one continuous factor on six ordinal predictors in &lt;code&gt;OpenMx&lt;/code&gt; (&lt;code&gt;R&lt;/code&gt;) using (robust) Weighted Least Squares (WLS) estimation. While I am completely new to OpenMx (I only used &lt;code&gt;sem&lt;/code&gt; and &lt;code&gt;lavaan&lt;/code&gt; ) I think I have most things now: I have a way of estimating polychoric correlations/covariances and estimating the latent means in one group and a good idea on how to implement this model in &lt;code&gt;OpenMx&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I just have one problem, I need a weight matrix in WLS estimation. This should be an asymptotic covariance matrix of the elements of the observed correlations/covariances.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is this exactly and how can I compute such a matrix?&lt;/p&gt;&#xA;" OwnerUserId="3094" LastEditorUserId="3094" LastEditorDisplayName="" LastEditDate="2011-05-30T21:56:35.773" LastActivityDate="2011-05-30T21:56:35.773" Title="How to compute the weight matrix for WLS estimation of a multi-group ordinal CFA model" Tags="&lt;r&gt;&lt;factor-analysis&gt;&lt;computing&gt;" />
  <row Id="11379" PostTypeId="2" ParentId="11374" CreationDate="2011-05-30T21:56:25.460" Score="3" ViewCount="" Body="&lt;p&gt;Here is a toy example for simulating a one-way ANOVA in R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, I just defined a general function that expect an effect size (&lt;code&gt;es&lt;/code&gt;), which is simply the ratio MSB/MSW (between/within mean squares), a value for the MSB, the number of groups, which might or not be of equal sizes:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sim.exp &amp;lt;- function(es=0.25, msb=10, groups=5, n=NULL, verbose=FALSE) {&#xA;  msw &amp;lt;- msb/es&#xA;  N &amp;lt;- ifelse(is.null(n), sample(10:40, groups), groups*n)&#xA;  means &amp;lt;- rnorm(n=groups, mean=0, sd=sqrt(msb)) &#xA;  my.df &amp;lt;- data.frame(grp=gl(groups, 1, N), &#xA;                      y=rnorm(N, means, sqrt(msw))) &#xA;  aov.res &amp;lt;- aov(y ~ grp, my.df)&#xA;  if (verbose) print(summary(aov.res))&#xA;  ave &amp;lt;- with(my.df, tapply(y, grp, function(x) c(mean(x), sd(x))))&#xA;  invisible(list(ave=ave, p.value=summary(aov.res)[[1]][1,5]))&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This function returns the p-value associated to the F-test, as well as the sample means and SDs. We can use it as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; sim.exp(verbose=TRUE)&#xA;            Df Sum Sq Mean Sq F value Pr(&amp;gt;F)&#xA;grp          4  32.71   8.176  0.1875 0.9418&#xA;Residuals   18 784.93  43.607               &#xA;&amp;gt; sim.exp(es=2, verbose=TRUE)&#xA;            Df Sum Sq Mean Sq F value    Pr(&amp;gt;F)    &#xA;grp          4 555.66 138.915  33.567 1.653e-09 ***&#xA;Residuals   24  99.32   4.138                      &#xA;---&#xA;Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 &#xA;&amp;gt; sim.exp(es=.5, n=30, groups=3, verbose=TRUE)&#xA;            Df  Sum Sq Mean Sq F value   Pr(&amp;gt;F)    &#xA;grp          2  639.12  319.56  16.498 8.42e-07 ***&#xA;Residuals   87 1685.13   19.37                     &#xA;---&#xA;Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then, I created a grid of values for &lt;code&gt;es&lt;/code&gt; and &lt;code&gt;msb&lt;/code&gt;, that is I want to check whether varying these parameters has an effect on the estimated p-value.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;my.design &amp;lt;- expand.grid(es=seq(.2, 2.4, by=.2), msb=seq(2, 10, by=2))&#xA;n.sim &amp;lt;- nrow(my.design)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Finally, let's use it. First, with a single replicate of each condition:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for (i in 1:n.sim)&#xA;  my.design$p[i] &amp;lt;- sim.exp(my.design[i,1], my.design[i,2], n=20)$p.value &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As can be seen, when increasing the effect size we are more likely to reject the null (averaged over MSB):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; with(my.design, aggregate(p, list(es=es), mean))&#xA;    es            x&#xA;1  0.2 1.178042e-01&#xA;2  0.4 1.315028e-02&#xA;3  0.6 5.765548e-02&#xA;4  0.8 5.742882e-02&#xA;5  1.0 8.940993e-05&#xA;6  1.2 9.199611e-09&#xA;7  1.4 9.115640e-06&#xA;8  1.6 8.100427e-10&#xA;9  1.8 2.656848e-07&#xA;10 2.0 3.577391e-05&#xA;11 2.2 5.477981e-14&#xA;12 2.4 1.219156e-04&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The results are shown below, although for clarity I took the log of the p-value. The horizontal dashed line shows the 5% limit for type I risk.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.stack.imgur.com/zrLD4.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ok, it's somewhat noisy. So, let's try to average p-values for 500 replicates in each conditions:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for (i in 1:n.sim)&#xA;  my.design$p[i] &amp;lt;- mean(unlist(replicate(500, &#xA;                      sim.exp(my.design[i,1], my.design[i,2], n=20))[2,]))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and the results are:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.stack.imgur.com/eGLA5.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can play with &lt;code&gt;es&lt;/code&gt; only as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;k &amp;lt;- 10000&#xA;es1 &amp;lt;- sample(seq(.1, 5, by=.1), k, rep=T)&#xA;pp &amp;lt;- numeric(k)&#xA;for (i in 1:k) &#xA;  pp[i] &amp;lt;- sim.exp(groups=3, es=es1[i])$p.value&#xA;plot(es1, -log10(pp), pch=19, col=&quot;#FF737350&quot;, cex=.6, xlab=&quot;Effect size (MSB=10)&quot;)&#xA;xx &amp;lt;- seq(.1, 5, by=.1)&#xA;lines(xx, predict(loess(-log10(pp) ~ es1), data.frame(es1=xx)), &#xA;      col=&quot;green&quot;, lwd=2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.stack.imgur.com/E682Y.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many other experiments are possible, and probably a better code too.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-05-30T21:56:25.460" CommentCount="4" />
  <row Id="11380" PostTypeId="1" CreationDate="2011-05-31T00:11:36.817" Score="-3" ViewCount="44" Body="&lt;p&gt;1.) Gender: male or female&#xA;2.) Age:&#xA;3.) Do you have a good home life?&#xA;4.) Is your family affectionate with each other?&#xA;5.) What is the most important thing to you in a relationship?&#xA;6.) If a significant other cheated on you, would you end the relationship?&#xA;7.) If it was guarunteed that no one would ever find out, would you cheat on a significant other?&#xA;8.) When something bothers you, do you tell your significant other?&#xA;9.) What is something that a significant other could do that would cause you to end the relationship?&#xA;10.) Have you ever told someone you loved them (and didn’t mean it) in order to get something you wanted?&#xA;11.) Have you ever cried in front of a significant other?&#xA;12.) Do you share the personal details of your love life with others?&#xA;13.) Would you date someone who made a higher salary than you?&#xA;14.) What is more important to you: looks or personality?&#xA;15.) Public displays of affection: yes or no?&lt;/p&gt;&#xA;" OwnerUserId="4811" LastActivityDate="2011-05-31T00:11:36.817" ClosedDate="2011-05-31T04:14:21.693" Title="Guy and girl differences on love/relationships: quick survey" Tags="&lt;survey&gt;&lt;polling&gt;&lt;psychology&gt;" />
  <row Id="11381" PostTypeId="1" AcceptedAnswerId="11382" CreationDate="2011-05-31T00:43:26.973" Score="5" ViewCount="61" Body="&lt;p&gt;I've calculated the proportion of chicks fledged out of the number of eggs hatched in each year using &lt;code&gt;prop.test()&lt;/code&gt; in R. I see that it gives me the proportion fledged, but also the 95% confidence interval, which is what I'm after. &#xA;Having read the excellent information from another question on this site &lt;a href=&quot;http://stats.stackexchange.com/questions/4713/binomial-confidence-interval-estimation-why-is-it-not-symmetric&quot;&gt;here&lt;/a&gt;, I understand why I don't have symmetry in my 95% CIs! &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;However, how should I report this in a paper? &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I've seen people report values as 38% (±0.2%), with an indication that the value in brackets is 95% CI. Obviously this won't work for asymmetrical CIs. Must I report the upper and lower values in these cases?&lt;/p&gt;&#xA;" OwnerUserId="4238" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-31T03:16:18.480" LastActivityDate="2011-05-31T03:16:18.480" Title="How to report asymmetrical confidence intervals of a proportion" Tags="&lt;r&gt;&lt;confidence-interval&gt;&lt;binomial&gt;" AnswerCount="1" CommentCount="2" />
  <row Id="11382" PostTypeId="2" ParentId="11381" CreationDate="2011-05-31T01:18:08.367" Score="4" ViewCount="" Body="&lt;p&gt;You should report the lower and upper intervals and also the method used to calculate the interval. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It turns out that there is no 'right' way to calculate confidence intervals for proportions, but instead many competing methods, each with advantages and disadvantages. The lack of a universally correct method stands in contrast to many statistical things that you might put numbers to, like means and standard deviations. For your interval to be fully specified you have to say how you calculated it.&lt;/p&gt;&#xA;" OwnerUserId="1679" LastActivityDate="2011-05-31T01:18:08.367" CommentCount="3" />
  <row Id="11383" PostTypeId="2" ParentId="11374" CreationDate="2011-05-31T03:53:48.550" Score="2" ViewCount="" Body="&lt;p&gt;There could possibly be a good reason why the MSB, MSE, and F values are only shown.  These are what is &quot;actually important&quot; in the analysis so to speak.  A p-value is just a sampling probability of some function of these three quantities (not sure of the specific function).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You don't need a p-value if you understand how to interpret these quantities.  p-value doesn't provide you with &quot;extra&quot; information, and it is so easy to mis-interpret (such as: probability for hypothesis, probability of eroneous conclusion, probability of type 1 error, etc.).  It can be quite fun to have a &quot;p-value bash&quot; every now and then :)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Much more useful and informative to look at effect sizes.  For if they are all pretty close - to within say plus/minus one standard error - then you already know without any need for hypothesis testing, that the data do provide support in favour of the hypothesis of equal means $H_0:\;\mu_0=\mu_1=\dots=\mu_k$.  You only need a hypothesis test, in the formal setting when it isn't &quot;obvious&quot; whether or not to accept or reject.  Say if one or two out of 20 means were between 2 and 3 standard errors away from the rest.  This is when the hypothesis test will help.  And also, if you see 10 out of the 20 groups over 3 standard errors apart, then you don't need the test: you know the null is not supported by the data.&lt;/p&gt;&#xA;" OwnerUserId="2392" LastActivityDate="2011-05-31T03:53:48.550" CommentCount="2" />
  <row Id="11384" PostTypeId="1" AcceptedAnswerId="11404" CreationDate="2011-05-31T07:02:42.717" Score="7" ViewCount="75" Body="&lt;p&gt;I want to apply a PCA on a dataset, which consists of mixed type variables (continuous and binary). To illustrate the procedure, I paste a minimal reproducible example in R below.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Generate synthetic dataset&#xA;set.seed(12345)&#xA;n &amp;lt;- 100&#xA;x1 &amp;lt;- rnorm(n)&#xA;x2 &amp;lt;- runif(n, -2, 2)&#xA;x3 &amp;lt;- x1 + x2 + rnorm(n)&#xA;x4 &amp;lt;- rbinom(n, 1, 0.5)&#xA;x5 &amp;lt;- rbinom(n, 1, 0.6)&#xA;data &amp;lt;- data.frame(x1, x2, x3, x4, x5)&#xA;&#xA;# Correlation matrix with appropriate coefficients&#xA;# Pearson product-moment: 2 continuous variables&#xA;# Point-biserial: 1 continuous and 1 binary variable&#xA;# Phi: 2 binary variables&#xA;# For testing purposes use hetcor function&#xA;library(polycor)&#xA;C &amp;lt;- as.matrix(hetcor(data=data))&#xA;&#xA;# Run PCA&#xA;pca &amp;lt;- princomp(covmat=C)&#xA;L &amp;lt;- loadings(pca)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, I wonder how to calculate component scores (i.e., raw variables weighted by component loadings). When dataset consists of continuous variables, component scores are simply obtained by multiplying (scaled) raw data and eigenvectors stored in loading matrix (L in the example above). Any pointers would be greatly appreciated.&lt;/p&gt;&#xA;" OwnerUserId="609" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-31T07:45:14.223" LastActivityDate="2011-05-31T18:12:02.890" Title="PCA and component scores based on a mix of continuous and binary variables" Tags="&lt;r&gt;&lt;pca&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="1" />
  <row Id="11385" PostTypeId="1" AcceptedAnswerId="11386" CreationDate="2011-05-31T07:05:36.300" Score="6" ViewCount="53" Body="&lt;p&gt;I am trying to test whether my regression has an issue of heteroscedasticity. After running a regression, I can clearly see that the residual plot has a pattern. After taking a log of the dependent variable the pattern is much, much reduced. The White's test on the original formula returns a p-value of 0.0004 before the transformation (the model with strong pattern in residuals), and a p-value of 0.08 after the log transformation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can see that the second model has less heteroscedasticity on the plot, but how do I interpret the results of white's test? Does the first value mean that we can reject that there is heteroscedasticity at (100-0.0004)% significance, while in the second model, we can reject it at, say, 95% confidence? &lt;/p&gt;&#xA;" OwnerUserId="4814" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-31T07:15:27.240" LastActivityDate="2011-05-31T07:26:43.670" Title="Linear regression, heteroscedasticity, White's test interpretation?" Tags="&lt;regression&gt;&lt;econometrics&gt;&lt;heteroscedasticity&gt;&lt;heteroskedasticity&gt;" AnswerCount="1" />
  <row Id="11386" PostTypeId="2" ParentId="11385" CreationDate="2011-05-31T07:26:43.670" Score="4" ViewCount="" Body="&lt;p&gt;The &lt;a href=&quot;http://www.jstor.org/stable/1912934&quot; rel=&quot;nofollow&quot;&gt;original White paper&lt;/a&gt; where the test statistic was proposed is an enlightening read. This excerpt I think is of interest here:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;...the null hypothesis maintains not&#xA;  only that the errors are&#xA;  homoskedastic, but also that they are&#xA;  independent of the regressors, &lt;em&gt;and&lt;/em&gt;&#xA;  that the model is correctly&#xA;  specified... Failure of any of these&#xA;  conditions cal lead to a statistically&#xA;  significant test statistic.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Assuming that the model is correctly specified your results indicate that for non-transformed case there is a clear presence of heteroskedasticity, and in the log case there is no heteroskedasticity at 5% significance level, but there is at 10%. This means that in the log case further tests should be made, since the test &quot;barely&quot; accepts the null hypothesis of no heteroskedasticity. For me personally this would be an indication that maybe model specification is not correct and other heteroskedasticity  tests should be made. Incidentally White gives an overview of alternative tests in its article: Godfrey, Goldfeld-Quandt, etc.&lt;/p&gt;&#xA;" OwnerUserId="2116" LastActivityDate="2011-05-31T07:26:43.670" />
  <row Id="11387" PostTypeId="1" CreationDate="2011-05-31T08:55:38.270" Score="3" ViewCount="43" Body="&lt;p&gt;Let's say i have sequences of symbols which can have five values : A, B, C, X, Y. The average length of sequences is around 7.&#xA;It is important that the symbols A, B, C have &lt;strong&gt;a bigger importance&lt;/strong&gt; than X and Y which may be consider as 'whatever different from A, B or C'&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to classify those data among two classes : positives and negatives. The positive class is composed of sequences generally well aligned like &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X X A B Y C&#xA;A B C X X&#xA;A Y A X B C X X&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Note that positive examples have generally symbols A,B and C in that order.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Negatives examples look like more 'messy' like&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;B X A X X X X C&#xA;C A Y Y X X B&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My first thought was that entropy was the key of that problem. I checked various papers but nothing was really satisfying. So my question is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which features would you use for a classification purpose?&lt;/p&gt;&#xA;" OwnerUserId="2505" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-31T09:10:44.403" LastActivityDate="2011-05-31T15:16:16.697" Title="Classification of sequences of symbols" Tags="&lt;classification&gt;" AnswerCount="4" />
  <row Id="11388" PostTypeId="2" ParentId="11387" CreationDate="2011-05-31T09:06:53.763" Score="2" ViewCount="" Body="&lt;p&gt;I would not use any feature detectors but recurrent neural networks. They are very good for symbolic sequences: for example, they are able to recognize context sensitive languages.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Check out &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.10.5339&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;Biologically Phoneme Classification with LSTM neural nets (Graves, Schmidhuber)&lt;/a&gt; for an explanation of how to use RNNs for classification. See &lt;a href=&quot;http://www.google.de/url?sa=t&amp;amp;source=web&amp;amp;cd=1&amp;amp;ved=0CCIQFjAA&amp;amp;url=http://www.icml-2011.org/papers/524_icmlpaper.pdf&amp;amp;ei=b6_kTcXmLMPPsga8wrSABg&amp;amp;usg=AFQjCNHGCGk9fYM9owhkFX6VahYzLadLAw&amp;amp;sig2=isZS8pi_fMoRm99Q4tzGmw&quot; rel=&quot;nofollow&quot;&gt;Generating Text with Recurrent Neural Networks (Martens, Sutskever, Hinton)&lt;/a&gt; for an impressive symbolic application of RNNs.&lt;/p&gt;&#xA;" OwnerUserId="2860" LastActivityDate="2011-05-31T09:06:53.763" />
  <row Id="11389" PostTypeId="1" CreationDate="2011-05-31T10:13:58.237" Score="5" ViewCount="93" Body="&lt;p&gt;I am working on a new webpage for my part-time job as a methodological/statistical consultant for (psychology) students at my university. On this website I would like to place several links to online recourses for clients to consult themselves.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I am looking for links to websites that offer a lot of statistical information. Preferably written in a way that is easy to comprehend. Most students use SPSS, but information on other programs is welcome too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far I have:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;www.crossvalidated.com&lt;/li&gt;&#xA;&lt;li&gt;www.statmethods.net&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="3094" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-31T12:11:46.573" LastActivityDate="2011-05-31T14:22:50.303" CommunityOwnedDate="2011-05-31T11:04:18.993" Title="Internet statistics resources suitable for psychology  students doing research" Tags="&lt;spss&gt;&lt;psychology&gt;&lt;web&gt;" AnswerCount="4" CommentCount="5" FavoriteCount="1" />
  <row Id="11390" PostTypeId="2" ParentId="11387" CreationDate="2011-05-31T10:36:03.260" Score="0" ViewCount="" Body="&lt;p&gt;Can we do it programitically ?? Meaning that writing a peace of code that does that you saying ..ex giving &quot;A x X B Y c&quot; more importance than &quot;B X X X A C&quot;.&#xA;agree that this code should be bit complex though.? &lt;/p&gt;&#xA;" OwnerUserId="1763" LastActivityDate="2011-05-31T10:36:03.260" CommentCount="2" />
  <row Id="11391" PostTypeId="2" ParentId="11389" CreationDate="2011-05-31T10:44:46.417" Score="4" ViewCount="" Body="&lt;p&gt;Copy N' Paste from my Google Reader: &lt;a href=&quot;http://jeromyanglim.blogspot.com/&quot; rel=&quot;nofollow&quot;&gt;http://jeromyanglim.blogspot.com/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="609" LastActivityDate="2011-05-31T10:44:46.417" CommunityOwnedDate="2011-05-31T11:04:18.993" />
  <row Id="11392" PostTypeId="2" ParentId="11389" CreationDate="2011-05-31T11:05:52.783" Score="2" ViewCount="" Body="&lt;p&gt;The &lt;a href=&quot;http://www.ats.ucla.edu/stat/&quot; rel=&quot;nofollow&quot;&gt;UCLA&lt;/a&gt; server has a lot of ressources for statistical computing, including annotated output from various statistical packages.&lt;/p&gt;&#xA;" OwnerUserId="930" LastActivityDate="2011-05-31T11:05:52.783" CommunityOwnedDate="2011-05-31T11:05:52.783" />
  <row Id="11393" PostTypeId="2" ParentId="11389" CreationDate="2011-05-31T11:07:04.313" Score="3" ViewCount="" Body="&lt;ul&gt;&#xA;&lt;li&gt;There's a &lt;em&gt;correct&lt;/em&gt; answer here! &#xA;&lt;a href=&quot;http://faculty.chass.ncsu.edu/garson/PA765/statnote.htm&quot; rel=&quot;nofollow&quot;&gt;http://faculty.chass.ncsu.edu/garson/PA765/statnote.htm&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Also good:&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://statcomp.ats.ucla.edu/&quot; rel=&quot;nofollow&quot;&gt;http://statcomp.ats.ucla.edu/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://dss.princeton.edu/online_help/&quot; rel=&quot;nofollow&quot;&gt;http://dss.princeton.edu/online_help/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.psych.cornell.edu/darlington/&quot; rel=&quot;nofollow&quot;&gt;http://www.psych.cornell.edu/darlington/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I know you didn't ask, probably because you know, the answer, but absolutely best statistics tests (for multivariate analysis) for psychologists (&amp;amp; for most other social scientists, although they don't all realize it) are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Cohen, J., Cohen, P., West, S.G. &amp;amp; Aiken, L.S. Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences, (L. Erlbaum Associates, Mahwah, N.J., 2003).&lt;/li&gt;&#xA;&lt;li&gt;Judd, C.M., McClelland, G.H. &amp;amp; Ryan, C.S. Data analysis : a model comparison approach, (Routledge/Taylor and Francis, New York, NY, 2008).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="2347" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-31T11:29:37.487" LastActivityDate="2011-05-31T11:29:37.487" CommunityOwnedDate="2011-05-31T11:07:04.313" />
  <row Id="11394" PostTypeId="2" ParentId="11389" CreationDate="2011-05-31T11:07:15.030" Score="7" ViewCount="" Body="&lt;p&gt;In general, encouraging research students to use Google and sites like &lt;a href=&quot;http://stats.stackexchange.com/&quot;&gt;Cross Validated&lt;/a&gt; to ask and answer their own questions is important.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Specific Sites&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.statisticshell.com/apf.html&quot; rel=&quot;nofollow&quot;&gt;Andy Field&lt;/a&gt; is famous for making statistics more palatable for psychology students. He provides many online resources generally with a focus on SPSS.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.ats.ucla.edu/stat/&quot; rel=&quot;nofollow&quot;&gt;UCLA Statistics Consulting&lt;/a&gt; has many useful resources.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.aliquote.org/&quot; rel=&quot;nofollow&quot;&gt;@chl has many good statistics resources&lt;/a&gt; with a psychology flavour, such as this one on &lt;a href=&quot;http://aliquote.org/memos/2011/02/24/psychoco-2011&quot; rel=&quot;nofollow&quot;&gt;psychometrics and R&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://faculty.chass.ncsu.edu/garson/PA765/statnote.htm&quot; rel=&quot;nofollow&quot;&gt;G. David Garson&lt;/a&gt; provides extensive notes on most techniques with a focus on SPSS generally suitable to a psychology research audience.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://davidakenny.net/kenny.htm&quot; rel=&quot;nofollow&quot;&gt;David Kenny&lt;/a&gt; has lots of resources particularly on SEM, mediation, moderation, and dyadic data analysis.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.psychology.org/links/Resources/Statistics/&quot; rel=&quot;nofollow&quot;&gt;Encyclopedia - Psychology and Statistics&lt;/a&gt; has an extensive set of links to resources&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.reddit.com/r/mathpsych/&quot; rel=&quot;nofollow&quot;&gt;mathpsych on Reddit&lt;/a&gt; is a small but interesting Reddit community.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;A little self-promotion&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;One of my main aims over the last few years has been to develop resources designed to assist psychology students perform the data analysis for their research. Thus, I hope you'll forgive the self-promotion. The following links may be relevant:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://jeromyanglim.blogspot.com/p/site-map-jeromy-anglims-blog-psychology.html&quot; rel=&quot;nofollow&quot;&gt;Sitemap&lt;/a&gt; of the blog. Most of the blog is devoted to saying what I find myself saying to psychology research students in consultation settings. Thus, there's a fair bit of SPSS content in addition to my own preference for R.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://jeromyanglim.blogspot.com/2009/05/statistics-for-psychology-thesis.html&quot; rel=&quot;nofollow&quot;&gt;Advice on completing data analysis for a thesis in psychology&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://jeromyanglim.blogspot.com/2009/09/teaching-resources.html&quot; rel=&quot;nofollow&quot;&gt;General teaching resources with an SPSS manual and some multivariate course notes&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://jeromyanglim.blogspot.com/2011/03/how-to-ask-me-statistics-question.html&quot; rel=&quot;nofollow&quot;&gt;General thoughts on encouraging students to use sites like Cross Validated&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;R in Psychology&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;I also have a post on &lt;a href=&quot;http://jeromyanglim.blogspot.com/2009/06/learning-r-for-researchers-in.html&quot; rel=&quot;nofollow&quot;&gt;getting started with R&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The following quotes the section of that post listing specific resources for researchers in psychology.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Task Views particularly relevant to psychology&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://cran.r-project.org/web/views/Psychometrics.html&quot; rel=&quot;nofollow&quot;&gt;Psychometric Models and Methods&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://cran.r-project.org/web/views/SocialSciences.html&quot; rel=&quot;nofollow&quot;&gt;Social Sciences&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://cran.r-project.org/web/views/Multivariate.html&quot; rel=&quot;nofollow&quot;&gt;Multivariate&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://artsweb.uwaterloo.ca/~jalockli/R_exp_psy.pdf&quot; rel=&quot;nofollow&quot;&gt;R Notes for Experimental Psychology&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.personality-project.org/R/&quot; rel=&quot;nofollow&quot;&gt;William Revelle's&#xA;Psychology R Site&lt;/a&gt; ;also see the package ;&lt;code&gt;pscyh&lt;/code&gt;, ;and the ;&lt;a&#xA;href=&quot;http://personality-project.org/r/book/&quot; rel=&quot;nofollow&quot;&gt;online book and workshop&#xA;resources&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.psych.upenn.edu/~baron/rpsych/rpsych.html&quot; rel=&quot;nofollow&quot;&gt;Jonathan&#xA;Baron and Yelin Li's R for Psychology Experiments&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.drewconway.com/zia/?p=1614&quot; rel=&quot;nofollow&quot;&gt;Drew Conway suggests a list of must have R packages for the social&#xA;scientsist&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://jeromyanglim.blogspot.com/2009/09/structural-equation-modelling-in-r.html&quot; rel=&quot;nofollow&quot;&gt;SEM in R&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=PSYCH-R&quot; rel=&quot;nofollow&quot;&gt;Mailing&#xA;list for Psychology and R&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://psy-ed.wikidot.com/&quot; rel=&quot;nofollow&quot;&gt;Edinburgh Psychology R-users&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://artsweb.uwaterloo.ca/~jalockli/R_exp_psy.pdf&quot; rel=&quot;nofollow&quot;&gt;Jason&#xA;Locklin's notes on standard experimental analyses in psychology&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://jeromyanglim.blogspot.com/search/label/R&quot; rel=&quot;nofollow&quot;&gt;My posts with the R tag&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="183" LastEditorUserId="183" LastEditorDisplayName="" LastEditDate="2011-05-31T14:22:50.303" LastActivityDate="2011-05-31T14:22:50.303" CommunityOwnedDate="2011-05-31T11:07:15.030" CommentCount="4" />
  <row Id="11396" PostTypeId="1" CreationDate="2011-05-31T12:55:39.020" Score="1" ViewCount="30" Body="&lt;p&gt;Hello, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm trying to implement logNormal distribution into my java program because lognormal dist doesn't exist into apache commons math library.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have no problem to re-write density and cumulative probability function, extending abstract classes of apache commons math library, like this : &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; public double cumulativeProbability(double mu, double sigma, double x) {&#xA;&#xA;        if (sigma &amp;lt;= 0.0) {&#xA;            throw new IllegalArgumentException(&quot;sigma  &amp;lt;= 0&quot;);&#xA;        }&#xA;        if (x &amp;lt;= 0.0) {&#xA;            return 0.0;&#xA;        }&#xA;&#xA;        return this.cumulativeProbability((Math.log(x) - mu) / sigma);&#xA;    }&#xA;&#xA; public double cumulativeProbability(double x){&#xA;&#xA;        double var = 0.0;&#xA;        NormalDistributionImpl normalDist = new NormalDistributionImpl();&#xA;&#xA;        try {&#xA;            var = normalDist.cumulativeProbability(x);&#xA;        } catch (MathException e) {}&#xA;&#xA;        return var;  &#xA;    }&#xA;&#xA;&#xA;  public double density (double x) {&#xA;      return density (mu, sigma, x);&#xA;  }&#xA;&#xA;&#xA;  public static double density (double mu, double sigma, double x) {&#xA;&#xA;      if (sigma &amp;lt;= 0)&#xA;         throw new IllegalArgumentException (&quot;sigma &amp;lt;= 0&quot;);&#xA;      if (x &amp;lt;= 0)&#xA;         return 0;&#xA;      double diff = Math.log (x) - mu;&#xA;      return Math.exp (-diff*diff/(2*sigma*sigma))/&#xA;              (Math.sqrt (2*Math.PI)*sigma*x);&#xA;   }&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Apache commons math need implementation of a list of utilies function to compute the inverse cumulative probability, and i don't know which upper/lowerbound/initial domain my lognormal distribution can take ... &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suppose it's something like this, but i'm not sure : x between [0 ; +infinity] ? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks a lot for your help&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;p = Desired probability for the critical value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Access the domain value lower bound, based on p, used to bracket a CDF root. This method is used by inverseCumulativeProbability(double) to find critical values.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  public double getDomainLowerBound(double p){&#xA;    return ?;&#xA;  }&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Access the domain value upper bound, based on p, used to bracket a CDF root. This method is used by inverseCumulativeProbability(double) to find critical values.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  public double getDomainUpperBound(double p){&#xA;    return ?;&#xA;  }&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Access the initial domain value, based on p, used to bracket a CDF root. This &#xA;method is used by inverseCumulativeProbability(double) to find critical values. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  public double getInitialDomain(double p){&#xA;    return ?;&#xA;  }&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Access the lower bound of the support.&#xA;Returns:&#xA;lower bound of the support (might be Double.NEGATIVE_INFINITY)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  public double getSupportLowerBound(){&#xA;    return ?;&#xA;  }&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Access the upper bound of the support.&#xA;Returns:&#xA;upper bound of the support (might be Double.POSITIVE_INFINITY)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  public double getSupportUpperBound(){&#xA;    return ?;&#xA;  }&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Use this method to get information about whether the lower bound of the support is inclusive or not.&#xA;Returns:&#xA;whether the lower bound of the support is inclusive or not&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  public  boolean isSupportLowerBoundInclusive(){&#xA;      return ;&#xA;  }&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Use this method to get information about whether the upper bound of the support is inclusive or not.&#xA;Returns:&#xA;whether the upper bound of the support is inclusive or not&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  public boolean isSupportUpperBoundInclusive(){&#xA;      return ;&#xA;  }&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="4693" LastActivityDate="2011-05-31T16:08:04.300" Title="Upper/lower bound and initial domain for lognormal distribution" Tags="&lt;distributions&gt;&lt;lognormal&gt;&lt;bounds&gt;" AnswerCount="1" CommentCount="4" />
  <row Id="11397" PostTypeId="2" ParentId="11353" CreationDate="2011-05-31T13:35:40.367" Score="1" ViewCount="" Body="&lt;p&gt;I would strongly recommend you use some form of &lt;a href=&quot;http://en.wikipedia.org/wiki/Regularization_%28mathematics%29&quot; rel=&quot;nofollow&quot;&gt;regularization&lt;/a&gt;.  The package '&lt;a href=&quot;http://cran.r-project.org/web/packages/glmnet/index.html&quot; rel=&quot;nofollow&quot;&gt;glmnet&lt;/a&gt;' in R is very good and will do variable selection and regularization for the linear model using the &lt;a href=&quot;http://www.stanford.edu/~hastie/TALKS/enet_talk.pdf&quot; rel=&quot;nofollow&quot;&gt;elastic net&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let me know if you'd like to see some examples of how to use glmnet.&lt;/p&gt;&#xA;" OwnerUserId="2817" LastActivityDate="2011-05-31T13:35:40.367" />
  <row Id="11398" PostTypeId="1" AcceptedAnswerId="11401" CreationDate="2011-05-31T15:02:15.027" Score="4" ViewCount="30" Body="&lt;p&gt;In case you want to compare the average income of a group of male employees against the average income of a group of female employees, the observations are clearly independent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, I have a network of a certain number of nodes. These nodes are linked by edges and I can characterize each node by the number of links it has to other nodes. (this is $k$: called the degree)&#xA;I can also characterize the nodes by their average nearest neighbour degree. That is the sum of the degree of all nodes, to which one node is linked to. (this is $k_{nn}$; $k_{nn}$ of node i = $\sum k_j$ for any node $j$ that is linked to $i$).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I create a scatter plot of these nodes ($k$ vs $k_{nn}$) I can clearly distinguish two groups of nodes by a certain threshold value for $k$ and $k_{nn}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My nodes also have a color. Now I want to test if a certain color in these two groups is overrepresented.&#xA;I can do that using the Wilcoxon rank test, because the color is an independent observation. Fine.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But is the color really an independent observation?&#xA;Implicitly the association to a group is not only based on the node's own property, but also on the properties of the other nodes (because of $k_{nn}$).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So can I really use the Wilcoxon rank test here?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Actually, my question is:&#xA;Does the Wilcoxon rank test require only an independent observation.&#xA;Or does it also require an association to a group that is based on independent observations?&lt;/p&gt;&#xA;" OwnerUserId="4819" LastEditorUserId="2116" LastEditorDisplayName="" LastEditDate="2011-05-31T19:28:07.607" LastActivityDate="2011-05-31T19:28:07.607" Title="Should the group decisions be independent in Wilcoxon rank sum test?" Tags="&lt;nonparametric&gt;&lt;independence&gt;&lt;wilcoxon&gt;" AnswerCount="1" />
  <row Id="11399" PostTypeId="2" ParentId="11387" CreationDate="2011-05-31T15:05:15.183" Score="2" ViewCount="" Body="&lt;p&gt;It sounds like this question is asking for a way to quantify the sense of &quot;generally well aligned&quot; strings.  Of course there are many ways to do this, but the examples and the description suggest that any solution meet two criteria:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1.  The X's and Y's should play no role in the result.&#xA;&#xA;2.  The strings in which the A, B, and C's appear in order are the most &quot;well aligned.&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This suggests basing the classification on an &lt;a href=&quot;http://en.wikipedia.org/wiki/Edit_distance&quot; rel=&quot;nofollow&quot;&gt;edit distance&lt;/a&gt; among the {A,B,C} substrings or on a &lt;a href=&quot;http://www.emis.de/journals/NSJOM/Papers/37_2/NSJOM_37_2_073_092.pdf&quot; rel=&quot;nofollow&quot;&gt;partial ordering of multiset permutations&lt;/a&gt;.  To provide more focused advice, we would need more information about the purpose of the intended importance ordering and about how these strings are generated.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-05-31T15:05:15.183" />
  <row Id="11400" PostTypeId="2" ParentId="11387" CreationDate="2011-05-31T15:16:16.697" Score="2" ViewCount="" Body="&lt;p&gt;Kernel methods (such as the support vector machine) are likely to be quite good for this kind of problem as you can use kernel functions that operate directly on strings of symbols of variable length.  Examples include the &lt;a href=&quot;http://psb.stanford.edu/psb-online/proceedings/psb02/leslie.pdf&quot; rel=&quot;nofollow&quot;&gt;spectrum kernel&lt;/a&gt; (which projects the strings into an implicit feature space where each dimension records the number of ocurrences of all possible substrings of a given length - or less) and the &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.78.9441&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;mismatch kernel&lt;/a&gt;, which is similar, but the counting of sub-strings allows a certain amount of mis-matches.  There is also the &lt;a href=&quot;http://bioinformatics.oxfordjournals.org/content/19/15/1964.short&quot; rel=&quot;nofollow&quot;&gt;sequence  alignment&lt;/a&gt; kernel,  which might be of interest.&lt;/p&gt;&#xA;" OwnerUserId="887" LastActivityDate="2011-05-31T15:16:16.697" />
  <row Id="11401" PostTypeId="2" ParentId="11398" CreationDate="2011-05-31T15:20:53.820" Score="5" ViewCount="" Body="&lt;p&gt;Regardless of the graph's coloring, associated with each of its nodes is a pair $(k, k_{nn})$.  You have used these pairs to classify the nodes into two groups.  The coloring separately classifies the nodes by color.  This is the situation of a $c$ by $2$ contingency table with fixed margins.  To assess whether color is associated with node group, use an appropriate test of association: &lt;a href=&quot;http://en.wikipedia.org/wiki/Fisher%27s_exact_test&quot; rel=&quot;nofollow&quot;&gt;Fisher's Exact Test&lt;/a&gt; if the counts are not too large; otherwise a chi-squared test.&lt;/p&gt;&#xA;" OwnerUserId="919" LastActivityDate="2011-05-31T15:20:53.820" />
  <row Id="11402" PostTypeId="1" AcceptedAnswerId="11409" CreationDate="2011-05-31T15:59:34.210" Score="9" ViewCount="88" Body="&lt;p&gt;My nonparametric text, &lt;a href=&quot;http://www.wiley.com/WileyCDA/WileyTitle/productCd-0471160687.html&quot; rel=&quot;nofollow&quot;&gt;Practical Nonparametric Statistics&lt;/a&gt;, often gives clean formulas for expectations, variances, test statistics, and the like, but includes the caveat that this only works if we ignore ties. When calculating the Mann-Whitney U Statistic, it is encouraged that you throw out tied pairs when comparing which is bigger.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I get that ties don't really tell us much about which population is bigger (if that's what we're interested in) since neither group is bigger than the other, but it doesn't seem like that would matter when developing asymptotic distributions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why then is it such a quandary dealing with ties in some nonparametric procedures? Is there a way of extracting any useful information from ties, rather than simply throwing them away?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: In regards to @whuber's comment, I checked my sources again, and some procedures use an average of ranks instead of dropping the tied values completely. While this seems more sensible in reference to retaining information, it also seems to me that it lacks rigor. The spirit of the question still stands, however.&lt;/p&gt;&#xA;" OwnerUserId="1118" LastEditorUserId="930" LastEditorDisplayName="" LastEditDate="2011-05-31T21:40:00.740" LastActivityDate="2011-05-31T21:40:00.740" Title="Why are ties so difficult in nonparametric statistics?" Tags="&lt;nonparametric&gt;" AnswerCount="1" CommentCount="4" FavoriteCount="1" />
  <row Id="11403" PostTypeId="2" ParentId="11396" CreationDate="2011-05-31T16:08:04.300" Score="2" ViewCount="" Body="&lt;p&gt;The support for the log normal distribution is the open interval from 0 to infinity. This should give you enough information to implement all methods that have 'Bound' in their name.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand that the methods with 'Domain' in their name are used to provide bounds and estimates for the inverse CDF that are easy to compute; for this you can easily follow @whuber's suggestion and return the $\exp$ of the corresponding values for the normal distribution. It's instructive to see what the implementors use for that: if $p \not= \frac{1}{2}$, then the infinite interval on the appropriate side of $\mu$ provides the bounds and the initial estimate is $\mu \pm \sigma$. So you could also directly return the $\exp$ of these values.&lt;/p&gt;&#xA;" OwnerUserId="2898" LastActivityDate="2011-05-31T16:08:04.300" />
  <row Id="11404" PostTypeId="2" ParentId="11384" CreationDate="2011-05-31T17:39:23.990" Score="5" ViewCount="" Body="&lt;p&gt;I think Insanodag is right. I quote Jollife's Principal Component Analysis:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;When PCA is used as a descriptive&#xA;  technique, there is no reason for the&#xA;  variables in the analysis to be of any&#xA;  particular type. [...] the basic&#xA;  objective of PCA - to summarize most&#xA;  of the 'variation' that is present in&#xA;  the original set of $p$ variables&#xA;  using smaller number of derived&#xA;  varaibles - can be achieved regardless&#xA;  of the nature of the original&#xA;  variables.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Multiplying the data matrix with the loadings matrix will give the desired result. However, I've had some problems with &lt;code&gt;princomp()&lt;/code&gt; function so I used &lt;code&gt;prcomp()&lt;/code&gt; instead.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of the return values of the function &lt;code&gt;prcomp()&lt;/code&gt; is &lt;code&gt;x&lt;/code&gt;, which is activated using &lt;code&gt;retx=TRUE&lt;/code&gt;. This x is the multiplication of the data matrix by the loadings matrix as stated in the R Documentation:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    rotation: the matrix of variable&#xA;              loadings (i.e., a matrix whose columns&#xA;              contain the eigenvectors).  The function ‘princomp’ returns&#xA;              this in the element ‘loadings’.&#xA;&#xA;           x: if ‘retx’ is true the value of the rotated data (the centred&#xA;              (and scaled if requested) data multiplied by the ‘rotation’&#xA;              matrix) is returned.  Hence, ‘cov(x)’ is the diagonal matrix&#xA;              ‘diag(sdev^2)’.  For the formula method, ‘napredict()’ is&#xA;              applied to handle the treatment of values omitted by the&#xA;              ‘na.action’.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Let me know if this was useful, or if it needs further corrections.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;--&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I.T. Jollife. Principal Component Analysis. Springer. Second Edition. 2002. pp 339-343.&lt;/p&gt;&#xA;" OwnerUserId="2902" LastEditorUserId="2902" LastEditorDisplayName="" LastEditDate="2011-05-31T18:12:02.890" LastActivityDate="2011-05-31T18:12:02.890" CommentCount="2" />
  <row Id="11405" PostTypeId="1" CreationDate="2011-05-31T17:49:46.537" Score="0" ViewCount="42" Body="&lt;p&gt;Does anybody know where to find good application and examples (besides the manual and the book applied econometrics with R) using the tobit model with the packages AER?&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Edit&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;I'm searching for a command to compute the marginal effects for y (not for the latent variable y*). It seems to be $\phi(x\beta/\sigma)\beta$, where $\phi$ is the std.normal cumulative distribution function. But how can I compute those effects with R?&lt;/p&gt;&#xA;" OwnerUserId="4496" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-06-01T16:57:40.790" LastActivityDate="2011-06-01T16:57:40.790" Title="Tobit model with R" Tags="&lt;r&gt;&lt;tobit-regression&gt;" AnswerCount="0" CommentCount="0" FavoriteCount="0" />
  <row Id="11406" PostTypeId="1" AcceptedAnswerId="11407" CreationDate="2011-05-31T18:53:09.813" Score="3" ViewCount="79" Body="&lt;p&gt;I am very new to R and to any packages in R. I looked at the ggplot2 documentation but could not find this. I want a box plot of variable &lt;code&gt;boxthis&lt;/code&gt; with respect to two factors &lt;code&gt;f1&lt;/code&gt; and &lt;code&gt;f2&lt;/code&gt;. That is suppose both &lt;code&gt;f1&lt;/code&gt; and &lt;code&gt;f2&lt;/code&gt; are factor variables and each of them takes two values and &lt;code&gt;boxthis&lt;/code&gt; is a continuous variable. I want to get 4 boxplots on a graph, each corresponding to one combination from the possible combinations that &lt;code&gt;f1&lt;/code&gt; and &lt;code&gt;f2&lt;/code&gt; can take. I think using the basic functionality in R, this can be done by&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; boxplot(boxthis ~ f1 * f2 , data = datasetname) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance for any help.&lt;/p&gt;&#xA;" OwnerUserId="4820" LastEditorUserId="919" LastEditorDisplayName="" LastEditDate="2011-06-01T03:44:48.610" LastActivityDate="2011-06-01T03:44:48.610" Title="Boxplot with respect to two factors using ggplot2 in R" Tags="&lt;r&gt;&lt;boxplot&gt;&lt;ggplot2&gt;" AnswerCount="1" CommentCount="2" FavoriteCount="2" />
  <row Id="11407" PostTypeId="2" ParentId="11406" CreationDate="2011-05-31T19:23:14.877" Score="6" ViewCount="" Body="&lt;p&gt;I can think of two ways to accomplish this: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;1. Create all combinations of &lt;code&gt;f1&lt;/code&gt; and &lt;code&gt;f2&lt;/code&gt; outside of the &lt;code&gt;ggplot&lt;/code&gt;-function&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(ggplot2)&#xA;&#xA;df &amp;lt;- data.frame(f1=factor(rbinom(100, 1, 0.45), label=c(&quot;m&quot;,&quot;w&quot;)), &#xA;                 f2=factor(rbinom(100, 1, 0.45), label=c(&quot;young&quot;,&quot;old&quot;)),&#xA;                 boxthis=rnorm(100))&#xA;&#xA;df$f1f2 &amp;lt;- interaction(df$f1, df$f2)&#xA;&#xA;ggplot(aes(y = boxthis, x = f1f2), data = df) + geom_boxplot()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.stack.imgur.com/RzBr4.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;2. use colour/fill/etc.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ggplot(aes(y = boxthis, x = f2, fill = f1), data = df) + geom_boxplot()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;http://i.stack.imgur.com/w7rF1.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="307" LastActivityDate="2011-05-31T19:23:14.877" CommentCount="2" />
  <row Id="11408" PostTypeId="2" ParentId="396" CreationDate="2011-05-31T19:35:02.157" Score="1" ViewCount="" Body="&lt;p&gt;Don't use dynamite plots:&#xA;&lt;a href=&quot;http://pablomarin-garcia.blogspot.com/2010/02/why-dynamite-plots-are-bad.html&quot; rel=&quot;nofollow&quot;&gt;http://pablomarin-garcia.blogspot.com/2010/02/why-dynamite-plots-are-bad.html&lt;/a&gt;, use violin plots or similar (boxplots family)&lt;/p&gt;&#xA;" OwnerUserId="2343" LastEditorUserId="2343" LastEditorDisplayName="" LastEditDate="2011-05-31T19:57:14.673" LastActivityDate="2011-05-31T19:57:14.673" CommunityOwnedDate="2011-06-01T03:42:00.250" />
  <row Id="11409" PostTypeId="2" ParentId="11402" CreationDate="2011-05-31T20:00:46.263" Score="7" ViewCount="" Body="&lt;p&gt;Most of the work on non-parametrics was originally done assuming that there was an underlying continuous distribution in which ties would be impossible (if measured accurately enough).  The theory can then be based on the distributions of order statistics (which are a lot simpler without ties) or other formulas.  In some cases the statistic works out to be approximately normal which makes things really easy.  When ties are introduced either because the data was rounded or is naturally discrete, then the standard assumptions do not hold.  The approximation may still be good enough in some cases, but not in others, so often the easiest thing to do is just give a warning that these formulas don't work with ties.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are tools for some of the standard non-parametric tests that have worked out the exact distribution when ties are present.  The exactRankTests package for R is one example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One simple way to deal with ties is to use randomization tests like permutation tests or bootstrapping.  These don't worry about asymptotic distributions, but use the data as it is, ties and all (note that with a lot of ties, even these techniques may have low power).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There was an article a few years back (I thought in the American Statistician, but I am not finding it) that discussed the ideas of ties and some of the things that you can do with them.  One point is that it depends on what question you are asking, what to do with ties can be very different in a superiority test vs. a non-inferiority test.&lt;/p&gt;&#xA;" OwnerUserId="4505" LastActivityDate="2011-05-31T20:00:46.263" />
  <row Id="11411" PostTypeId="1" CreationDate="2011-05-31T20:57:49.140" Score="1" ViewCount="38" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;Possible Duplicate:&lt;/strong&gt;&lt;br&gt;&#xA;  &lt;a href=&quot;http://stats.stackexchange.com/questions/3115/cross-correlation-significance-in-r&quot;&gt;Cross-correlation significance in R&lt;/a&gt;  &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&#xA;&#xA;&lt;p&gt;How can I test the significance of the correlation coefficient? I have two time series and I want to test if they are cross correlation. Should I do prewhitening  the two series before comuputing the ccf or there are an easy way?&lt;/p&gt;&#xA;" OwnerUserId="4823" LastEditorUserId="307" LastEditorDisplayName="" LastEditDate="2011-05-31T21:03:51.863" LastActivityDate="2011-05-31T21:03:51.863" ClosedDate="2011-05-31T21:06:46.090" Title="How can I test the significance of the correlation coefficient?" Tags="&lt;time-series&gt;&lt;cross-correlation&gt;" CommentCount="0" />
  <row Id="11412" PostTypeId="1" CreationDate="2011-05-31T22:03:47.097" Score="4" ViewCount="35" Body="&lt;p&gt;I've used a wide array of tests for my thesis data, from parametric ANOVAs and t-tests to non-parametric Kruskal-Wallis tests and Mann-Whitneys, as well as rank-transformed 2-way ANOVAs, and GzLMs with binary, poisson and proportional data. Now I need to report everything as I write all of this up in my results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've already asked &lt;a href=&quot;http://stats.stackexchange.com/questions/11381/how-to-report-asymmetrical-confidence-intervals-of-a-proportion&quot;&gt;here&lt;/a&gt; how to report asymmetrical confidence intervals for proportion data. I know that standard deviation, standard error or confidence intervals are appropriate for means, which is what I'd report if all my tests were nicely parametric. However, for my non-parametric tests, should I be reporting medians and not means? If so, what error would I report with it?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Associated with this is how best to present non-parametric test results graphically. Since I largely have continuous or interval data within categories, I'm generally using bar graphs, with the top of the bar being the mean and error bars showing 95% CI. For NP tests, can I still use bar graphs, but have the top of the bar represent the median?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for your suggestions!&lt;/p&gt;&#xA;" OwnerUserId="4238" LastEditorUserId="4238" LastEditorDisplayName="" LastEditDate="2011-06-03T00:52:37.473" LastActivityDate="2011-06-03T00:52:37.473" Title="Error to report with median and graphical representations?" Tags="&lt;data-visualization&gt;&lt;median&gt;&lt;error&gt;" CommentCount="2" />
</posts>